feat: introduce MCP server, cross-platform runtime, and Showcase UI reorganization

- Showcase UI Reorganization:
  * Move web UI dashboard from root directory into apps/showcase_ui/ for cleaner project structure
  * Add 'artemis ui' CLI command (artemis/interfaces/cli/commands/ui.py) and launcher scripts for the interactive showcase console
- Standalone MCP Server & Notifiers (mcp_server/):
  * Implement dedicated MCP server package with mobile automation tools (device_state, inspect_trace, task_manager, task_runner)
  * Add multi-channel notifiers (AgentAPI, Desktop notifications, Webhooks, File output, and Composite routing)
  * Introduce background task runner and trace store for autonomous mobile execution monitoring
- Cross-Platform Runtime & Toolchain Support:
  * Add artemis/platform/ with OS-specific implementations for Linux, macOS (Darwin), and Windows
  * Add artemis/runtime/supervisor.py for resilient process and agent runtime supervision
  * Add artemis/toolchain/ with dependency resolution and SDK descriptors
  * Add cross-platform launcher scripts (start.sh, start.bat) and installer scripts (scripts/install_deps.*)
- Core Enhancements & Diagnostics:
  * Enhance coordinate normalization, OCR API fallback, and perception pipeline
  * Add credential validation (credentials_validator.py) and system diagnostics
  * Update bilingual documentation (README.md, README_CN.md) and architecture diagrams
- Test Suite Expansion:
  * Add integration tests for cloud gateway, remote ADB, and Artemis login
  * Add unit tests covering core diagnostics, platform detection, toolchain, credentials validator, optional OCR, and MCP tools
This commit is contained in:
somew1nd committed 2026-08-18 13:59:27 -07:00
1 parent 78e478f502
commit 9a5aaf99b7
249 files changed
+37226 -9339

No files matched your search

-51
View File
@@ -1,51 +0,0 @@
# 🛠️ Artemis Admin & Trace Console
A comprehensive developer/administrator control plane and trace inspection console for the Latte/Artemis mobile automation system.
---
## 🎯 Key Features
1. **Full Trace Inspection (`/api/sessions`, `/api/steps`)**:
- Hierarchical Call Tree exploration (LLM thoughts, tool calls, agent motivations).
- Step-by-step pre/post screenshots and action overlays (tap circles, swipe paths).
- Streaming logs and video recording playback.
2. **Task Control Center (`/api/run`)**:
- Directly launch autonomous tasks with custom goals and model profiles (Flash / Pro).
- Pause, resume, and manage FIFO task execution queues.
3. **Step Replay & Debugging Engine (`/api/replay`)**:
- Replay individual steps with isolated inputs, state inspection, and live device execution.
---
## 🚀 How to Start
### Option 1: Using `uv` (Recommended)
From the repository root:
```bash
uv run uvicorn apps.admin_console.server:app --host 0.0.0.0 --port 8000 --reload
```
Or directly using Python module:
```bash
uv run python -m apps.admin_console.server
```
### Option 2: From the `apps/admin_console` directory
```bash
cd apps/admin_console
uv run uvicorn server:app --host 0.0.0.0 --port 8000 --reload
```
Once started, open your browser and navigate to:
👉 **`http://localhost:8000`**
---
## 🎥 Video Debugging Mode
By default, `VideoAnalyzer` cleans up local temporary video files after execution. To retain video recordings for full playback in the console:
```bash
KEEP_VIDEOS=1 artemis run "your task goal"
```
Or set `ARTEMIS_DEBUG=1` in your environment.
+7 -1
View File
@@ -63,6 +63,12 @@ except ImportError:
REPLAY_BASE_DIR = TRACES_PATH / "replay"
TEST_DATA_DIR = REPLAY_BASE_DIR / "data"
TEST_OUTPUTS_DIR = REPLAY_BASE_DIR / "outputs"
IMAGES_DIR = TRACES_PATH / "images"
PAUSE_FILE = WORKSPACE_ROOT / ".artemis_paused"
# Ensure all essential runtime directories exist automatically
for _d in (TRACES_PATH, REPLAY_BASE_DIR, TEST_DATA_DIR, TEST_OUTPUTS_DIR, IMAGES_DIR):
try:
_d.mkdir(parents=True, exist_ok=True)
except Exception:
pass
+81 -13
View File
@@ -36,25 +36,96 @@ class ServerState:
self.active_connections: dict[str, dict[str, Any]] = {}
self.active_session_id: str | None = None
self.was_stopped_manually: bool = False
self.cancelled_session_ids: set[str] = set()
# FIFO Task queue
self.task_queue: asyncio.Queue = asyncio.Queue()
self.queue_tasks: list[dict[str, Any]] = []
# Unified single source of truth for task queue
self.queue_items: list[dict[str, Any]] = []
self._wake_event: asyncio.Event | None = None
self.worker_task: asyncio.Task | None = None
@property
def wake_event(self) -> asyncio.Event:
try:
loop = asyncio.get_running_loop()
except RuntimeError:
loop = None
if self._wake_event is None or (
loop
and getattr(self._wake_event, "_loop", None) is not None
and self._wake_event._loop != loop
):
self._wake_event = asyncio.Event()
return self._wake_event
@property
def task_queue(self) -> list[dict[str, Any]]:
"""Backward compatibility alias for queue_items."""
return self.queue_items
@property
def queue_tasks(self) -> list[dict[str, Any]]:
"""Returns all currently pending tasks in the queue."""
return [t for t in self.queue_items if isinstance(t, dict) and t.get("status") == "pending"]
@queue_tasks.setter
def queue_tasks(self, val: list[dict[str, Any]]):
# Compatibility setter
self.queue_items = list(val)
@property
def queue_goals(self) -> list[str]:
"""Backward compatibility helper for queue goals."""
return [t.get("goal", "") for t in self.queue_tasks if isinstance(t, dict)]
return [t.get("goal", "") for t in self.queue_tasks]
@queue_goals.setter
def queue_goals(self, val: list[Any]):
# Allow setting if legacy code updates it
pass
@property
def is_running(self) -> bool:
return self.current_process is not None and self.current_process.returncode is None
has_proc = False
if self.current_process is not None:
if self.current_process.returncode is not None:
has_proc = False
self.current_process = None
else:
pid = getattr(self.current_process, "pid", None)
if pid:
try:
import os
os.kill(pid, 0)
has_proc = True
except (ProcessLookupError, PermissionError, OSError):
has_proc = False
self.current_process = None
else:
has_proc = False
self.current_process = None
has_running_item = any(
isinstance(t, dict) and t.get("status") == "running" for t in self.queue_items
)
has_live_connection = False
for sid, conn in list(self.active_connections.items()):
c_pid = conn.get("pid")
if c_pid:
try:
import os
os.kill(c_pid, 0)
has_live_connection = True
except (ProcessLookupError, PermissionError, OSError):
self.active_connections.pop(sid, None)
if not has_proc and not has_running_item and not has_live_connection:
self.active_session_id = None
self.current_process = None
return False
return True
@property
def is_paused(self) -> bool:
@@ -69,13 +140,10 @@ class ServerState:
self.ipc_subscribers.remove(callback)
def clear_queue(self):
while not self.task_queue.empty():
try:
self.task_queue.get_nowait()
self.task_queue.task_done()
except (asyncio.QueueEmpty, ValueError):
break
self.queue_tasks.clear()
"""Clears all pending items from the task queue."""
self.queue_items = [t for t in self.queue_items if t.get("status") == "running"]
if self._wake_event:
self._wake_event.set()
# Global shared instance
+18 -3
View File
@@ -15,7 +15,6 @@
from contextlib import contextmanager
from pathlib import Path
import sqlite3
from fastapi import HTTPException
try:
from admin_console.core.config import DB_PATH
@@ -23,13 +22,29 @@ except ImportError:
from apps.admin_console.core.config import DB_PATH
_initialized_dbs = set()
def get_db(db_path: Path | None = None) -> sqlite3.Connection:
"""Returns a SQLite connection with timeout and Row row_factory."""
path = db_path or DB_PATH
if not path.exists():
raise HTTPException(status_code=500, detail=f"Database not found at {path}")
path_key = str(path)
if path_key not in _initialized_dbs:
path.parent.mkdir(parents=True, exist_ok=True)
try:
from artemis.data_engine.storage import StorageManager
StorageManager(db_path=path, base_trace_dir=path.parent)
_initialized_dbs.add(path_key)
except Exception:
pass
conn = sqlite3.connect(path, timeout=10.0)
conn.row_factory = sqlite3.Row
try:
conn.execute("PRAGMA journal_mode=WAL")
conn.execute("PRAGMA busy_timeout=10000")
except Exception:
pass
return conn
@@ -10,4 +10,4 @@
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# limitations under the License.
@@ -33,13 +33,6 @@ class SessionRepository:
cursor.execute("SELECT * FROM sessions ORDER BY start_time DESC")
return [dict(r) for r in cursor.fetchall()]
def get_session_by_id(self, session_id: str) -> dict[str, Any] | None:
with db_session(self.db_path) as conn:
cursor = conn.cursor()
cursor.execute("SELECT * FROM sessions WHERE session_id = ?", (str(session_id),))
row = cursor.fetchone()
return dict(row) if row else None
def get_video_recordings_map(self) -> dict[str, str]:
with db_session(self.db_path) as conn:
cursor = conn.cursor()
@@ -13,6 +13,7 @@
# limitations under the License.
import json
import re
from typing import Any
try:
@@ -183,9 +184,27 @@ class StepRepository:
)
args = payload_obj.get("args", payload_obj)
if isinstance(args, dict):
return {"args": self._clean_value(args)}
return {"args": self._clean_value(args)}
cleaned_args = (
self._clean_value(args) if isinstance(args, dict) else self._clean_value(args)
)
result_payload: dict[str, Any] = {"args": cleaned_args}
# Extract structured action execution results
res = payload_obj.get("result")
if isinstance(res, dict):
if res.get("post_image_name"):
result_payload["post_image_name"] = res["post_image_name"]
if res.get("pre_image_name"):
result_payload["pre_image_name"] = res["pre_image_name"]
if res.get("outcome"):
result_payload["outcome"] = res["outcome"]
elif res:
res_str = str(res)
match = re.search(r"([a-f0-9]{64})", res_str)
if match:
result_payload["post_image_name"] = match.group(1)
return result_payload
def get_session_steps(self, session_id: str, client: str | None = None) -> list[dict[str, Any]]:
with db_session(self.db_path) as conn:
@@ -221,7 +240,7 @@ class StepRepository:
if pre_img and step_dict.get("post_image_name") == pre_img:
step_dict["post_image_name"] = None
# Fetch generic tool traces for this step
# Fetch action and generic tool traces for this step
step_id = step_dict.get("step_id")
if step_id:
cursor.execute(
@@ -229,7 +248,7 @@ class StepRepository:
" t1.type, t1.name, t1.timestamp, t1.duration, t1.status,"
" t1.payload, t2.name as agent_name FROM traces t1 LEFT"
" JOIN traces t2 ON t1.parent_trace_id = t2.trace_id WHERE"
" t1.step_id = ? AND (t1.type = 'tool' OR (t1.type ="
" t1.step_id = ? AND (t1.type = 'tool' OR t1.type = 'action' OR (t1.type ="
" 'llm_call' AND t1.status = 'failed')) ORDER BY"
" t1.timestamp ASC",
(step_id,),
+123 -21
View File
@@ -20,6 +20,11 @@
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Artemis Debug UI</title>
<link rel="icon" type="image/png" sizes="192x192" href="/logo.png?v=2">
<link rel="icon" type="image/png" sizes="32x32" href="/logo.png?v=2">
<link rel="icon" type="image/svg+xml" href="/logo.svg?v=2">
<link rel="shortcut icon" href="/favicon.ico?v=2">
<link rel="apple-touch-icon" href="/logo.png?v=2">
<style>
pre.light-code {
background: transparent !important;
@@ -385,21 +390,33 @@
marker.style.top = `${pctY}%`;
marker.style.transform = 'translate(-50%, -50%)';
marker.style.zIndex = '10';
marker.style.pointerEvents = 'none';
const dotSize = sequenceIndex !== undefined ? 20 : 14;
const dot = document.createElement('div');
dot.style.width = '14px';
dot.style.height = '14px';
dot.style.width = `${dotSize}px`;
dot.style.height = `${dotSize}px`;
dot.style.borderRadius = '50%';
dot.style.backgroundColor = '#ef4444';
dot.style.border = '2px solid white';
dot.style.boxShadow = '0 0 6px rgba(0,0,0,0.4)';
if (sequenceIndex !== undefined) {
dot.style.display = 'flex';
dot.style.alignItems = 'center';
dot.style.justifyContent = 'center';
dot.style.fontSize = '11px';
dot.style.fontWeight = 'bold';
dot.style.color = 'white';
dot.innerText = String(sequenceIndex);
}
const pulseSize = dotSize * 2;
const pulse = document.createElement('div');
pulse.style.position = 'absolute';
pulse.style.top = '-9px';
pulse.style.left = '-9px';
pulse.style.width = '28px';
pulse.style.height = '28px';
pulse.style.top = `-${dotSize / 2 + 2}px`;
pulse.style.left = `-${dotSize / 2 + 2}px`;
pulse.style.width = `${pulseSize}px`;
pulse.style.height = `${pulseSize}px`;
pulse.style.borderRadius = '50%';
pulse.style.border = '2px solid #ef4444';
pulse.style.animation = 'pulse-animation 1.5s infinite ease-out';
@@ -409,14 +426,18 @@
marker.appendChild(dot);
const label = document.createElement('div');
label.innerText = actionType.toUpperCase();
if (sequenceIndex !== undefined) {
label.innerText = totalSequence && totalSequence > 1 ? `#${sequenceIndex} TAP` : actionType.toUpperCase();
} else {
label.innerText = actionType.toUpperCase();
}
label.style.position = 'absolute';
label.style.top = '16px';
label.style.top = `${dotSize + 2}px`;
label.style.left = '50%';
label.style.transform = 'translateX(-50%)';
label.style.backgroundColor = 'rgba(15, 23, 42, 0.8)';
label.style.backgroundColor = 'rgba(15, 23, 42, 0.9)';
label.style.color = 'white';
label.style.padding = '1px 5px';
label.style.padding = '1.5px 5px';
label.style.borderRadius = '3px';
label.style.fontSize = '8px';
label.style.fontWeight = 'bold';
@@ -427,6 +448,66 @@
overlay.appendChild(marker);
}
function createSequenceConnector(x1, y1, x2, y2, overlay, isNormalized) {
let pctX1 = x1 / 10;
let pctY1 = y1 / 10;
let pctX2 = x2 / 10;
let pctY2 = y2 / 10;
const container = overlay.parentElement;
const img = container ? container.querySelector('img') : null;
if (!isNormalized && img && img.naturalWidth && img.naturalHeight) {
if (x1 > 1000 || y1 > 1000 || x2 > 1000 || y2 > 1000 || (x1 > 100 && y1 > 100 && img.naturalWidth > 500)) {
pctX1 = (x1 / img.naturalWidth) * 100;
pctY1 = (y1 / img.naturalHeight) * 100;
pctX2 = (x2 / img.naturalWidth) * 100;
pctY2 = (y2 / img.naturalHeight) * 100;
}
}
let svg = overlay.querySelector('svg');
if (!svg) {
svg = document.createElementNS('http://www.w3.org/2000/svg', 'svg');
svg.setAttribute('width', '100%');
svg.setAttribute('height', '100%');
svg.style.position = 'absolute';
svg.style.top = '0';
svg.style.left = '0';
svg.style.pointerEvents = 'none';
const defs = document.createElementNS('http://www.w3.org/2000/svg', 'defs');
const marker = document.createElementNS('http://www.w3.org/2000/svg', 'marker');
marker.setAttribute('id', 'seq-arrow');
marker.setAttribute('viewBox', '0 0 10 10');
marker.setAttribute('refX', '5');
marker.setAttribute('refY', '5');
marker.setAttribute('markerWidth', '6');
marker.setAttribute('markerHeight', '6');
marker.setAttribute('orient', 'auto');
const path = document.createElementNS('http://www.w3.org/2000/svg', 'path');
path.setAttribute('d', 'M 0 0 L 10 5 L 0 10 z');
path.setAttribute('fill', '#ef4444');
marker.appendChild(path);
defs.appendChild(marker);
svg.appendChild(defs);
overlay.appendChild(svg);
}
const line = document.createElementNS('http://www.w3.org/2000/svg', 'line');
line.setAttribute('x1', `${pctX1}%`);
line.setAttribute('y1', `${pctY1}%`);
line.setAttribute('x2', `${pctX2}%`);
line.setAttribute('y2', `${pctY2}%`);
line.setAttribute('stroke', '#ef4444');
line.setAttribute('stroke-width', '2.5');
line.setAttribute('marker-end', 'url(#seq-arrow)');
line.setAttribute('stroke-dasharray', '4,4');
svg.appendChild(line);
}
function createLineMarker(x1, y1, x2, y2, overlay, isNormalized) {
let pctX1 = x1 / 10;
let pctY1 = y1 / 10;
@@ -518,9 +599,37 @@
return;
}
// 1. Multi-point click sequence
const isSequenceAction = actionName.includes('sequence') || Boolean(act.sequence || args.sequence || act.normalized_sequence || args.normalized_sequence);
let rawSeq = act.normalized_sequence || args.normalized_sequence || act.sequence || args.sequence || act.targets || args.targets;
if (typeof rawSeq === 'string') {
try { rawSeq = JSON.parse(rawSeq); } catch (e) {}
}
if (Array.isArray(rawSeq) && rawSeq.length > 0) {
const validPoints = [];
rawSeq.forEach(pt => {
if (typeof pt === 'string') {
try { pt = JSON.parse(pt); } catch (e) {}
}
if (Array.isArray(pt) && pt.length >= 2 && typeof pt[0] === 'number' && typeof pt[1] === 'number') {
validPoints.push([pt[0], pt[1]]);
}
});
if (validPoints.length > 0 && (isSequenceAction || validPoints.length > 1)) {
const isNormSeq = Boolean(act.normalized_sequence || args.normalized_sequence) || validPoints.every(p => p[0] <= 1000 && p[1] <= 1000);
for (let i = 0; i < validPoints.length - 1; i++) {
createSequenceConnector(validPoints[i][0], validPoints[i][1], validPoints[i+1][0], validPoints[i+1][1], overlayElement, isNormSeq);
}
validPoints.forEach((pt, idx) => {
createPointMarker(pt[0], pt[1], actionName, overlayElement, isNormSeq, idx + 1, validPoints.length);
});
return;
}
}
const isSwipeLike = actionName.includes('swipe') || actionName.includes('scroll') || actionName.includes('drag') || actionName.includes('slide');
// 1. Separate start and end coordinates
// 2. Separate start and end coordinates
const normStart = act.normalized_start_coordinates || args.normalized_start_coordinates;
const normEnd = act.normalized_end_coordinates || args.normalized_end_coordinates;
const startCoords = normStart || act.start_coordinates || act.start_point || act.start || act.from ||
@@ -538,11 +647,11 @@
return;
}
// 2. 4-coordinate array
// 3. 4-coordinate array
const normCoords = act.normalized_coordinates || args.normalized_coordinates;
const isNorm = Boolean(normCoords);
const rawCoords = normCoords || act.coordinates || act.coords || act.target ||
args.target || args.coordinates || args.coords || args.sequence ||
args.target || args.coordinates || args.coords ||
(isSwipeLike && Array.isArray(args.action) ? args.action : null) ||
(isSwipeLike && Array.isArray(act.action) ? act.action : null) ||
(isSwipeLike && Array.isArray(args.gesture) ? args.gesture : null) ||
@@ -557,17 +666,10 @@
createPointMarker(rawCoords[0], rawCoords[1], actionName, overlayElement, isNorm);
return;
}
} else if (Array.isArray(rawCoords[0]) && rawCoords[0].length === 2) {
rawCoords.forEach(pt => {
if (Array.isArray(pt) && pt.length === 2 && typeof pt[0] === 'number') {
createPointMarker(pt[0], pt[1], actionName, overlayElement, isNorm);
}
});
return;
}
}
// 3. Directional swipe / scroll ONLY for swipe-like actions
// 4. Directional swipe / scroll ONLY for swipe-like actions
if (isSwipeLike) {
const rawDir = (typeof args.action === 'string' ? args.action : null) ||
(typeof act.action === 'string' && act.action !== actionName ? act.action : null) ||
+1 -1
View File
@@ -1493,7 +1493,7 @@ class ReplayManager:
f" offset={start_offset:.3f}s, duration={duration:.3f}s"
)
try:
res = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
res = subprocess.run(cmd, capture_output=True, text=True)
if res.returncode != 0:
print(
f"Warning: ffmpeg failed with exit code {res.returncode}. stderr: {res.stderr}"
+1 -2
View File
@@ -26,10 +26,9 @@ except ImportError:
router = APIRouter(tags=["media"])
@router.get("/", response_class=HTMLResponse)
@router.get("/admin", response_class=HTMLResponse)
@router.get("/debug", response_class=HTMLResponse)
async def get_index():
async def get_admin_index():
admin_index = Path(__file__).resolve().parent.parent / "index.html"
if admin_index.exists():
return admin_index.read_text(encoding="utf-8")
+348
View File
@@ -0,0 +1,348 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""System Readiness & Diagnostics Router for Artemis Admin Console."""
from fastapi import APIRouter, HTTPException
from pydantic import BaseModel, Field
from artemis.core.diagnostics import readiness_engine
from artemis.core.diagnostics.schema import SystemReadinessReport
router = APIRouter(prefix="/api/system", tags=["system"])
class SelectDeviceRequest(BaseModel):
"""Payload to select an active target Android device."""
serial: str = Field(description="Serial number or identifier of the Android device to select")
@router.get("/readiness", response_model=SystemReadinessReport)
async def get_system_readiness() -> SystemReadinessReport:
"""Execute all diagnostic probes and return a comprehensive system readiness report."""
return await readiness_engine.run_all()
@router.post("/devices/select")
async def select_active_device(request: SelectDeviceRequest):
"""Select the active Android device or emulator for subsequent automated tasks."""
serial = request.serial.strip()
if not serial:
raise HTTPException(status_code=400, detail="Device serial cannot be empty.")
readiness_engine.set_active_device_serial(serial)
# Return updated readiness
report = await readiness_engine.run_all()
return {
"status": "success",
"selected_serial": serial,
"report": report,
}
@router.post("/adb/restart")
async def restart_adb_server():
"""Restart local ADB server and return an updated readiness check."""
restart_result = await readiness_engine.restart_adb_server()
updated_report = await readiness_engine.run_all()
return {
"restart_result": restart_result,
"report": updated_report,
}
class ConnectAdbRequest(BaseModel):
"""Payload to connect to an Android device over Wi-Fi."""
host: str = Field(description="IP address of Android device")
port: int = Field(default=5555, description="Port number")
@router.post("/adb/connect")
async def connect_wireless_adb(request: ConnectAdbRequest):
"""Connect to a device over Wi-Fi and return updated readiness."""
connect_result = await readiness_engine.connect_wireless_adb(request.host, request.port)
updated_report = await readiness_engine.run_all()
return {
"connect_result": connect_result,
"report": updated_report,
}
class LaunchEmulatorRequest(BaseModel):
"""Payload to launch a local Android Virtual Device (AVD)."""
avd_name: str = Field(
description="Name of the installed AVD emulator to launch (e.g. Android_2)"
)
@router.post("/emulator/launch")
async def launch_emulator(request: LaunchEmulatorRequest):
"""Launch an Android emulator in the background and return initiation status."""
avd_name = request.avd_name.strip()
if not avd_name:
raise HTTPException(status_code=400, detail="AVD name cannot be empty.")
launch_res = await readiness_engine.launch_emulator(avd_name)
return launch_res
@router.get("/emulator/status")
async def get_emulator_status():
"""Query real-time progress and logs of background emulator launch."""
return readiness_engine.get_emulator_status()
@router.post("/emulator/stop")
async def stop_emulator():
"""Stop active emulator process."""
return await readiness_engine.stop_emulator()
@router.post("/emulator/dismiss")
async def dismiss_emulator():
"""Dismiss emulator launch tracking state."""
return readiness_engine.dismiss_emulator()
class UpdateCredentialsRequest(BaseModel):
"""Payload to update and configure LLM or Vision OCR API credentials."""
provider: str = Field(
default="google",
description="Provider identifier (e.g. google, gemini, openai, anthropic, openrouter, ocr)",
)
api_key: str = Field(description="The secret API key string to configure")
persist_to_env: bool = Field(
default=True, description="Whether to persist the key to .env file"
)
class ValidateCredentialsRequest(BaseModel):
"""Payload to test and verify LLM or Vision OCR API credentials without saving."""
provider: str = Field(
default="google",
description="Provider identifier (e.g. google, gemini, openai, anthropic, openrouter, ocr)",
)
api_key: str = Field(description="The secret API key string to test")
base_url: str | None = Field(default=None, description="Optional custom base URL")
@router.get("/credentials")
async def get_credentials():
"""Retrieve current configured API keys for management."""
from artemis.config import settings
gemini_k = settings.get_api_key("google")
openai_k = settings.get_api_key("openai")
anthropic_k = settings.get_api_key("anthropic")
openrouter_k = settings.get_api_key("openrouter")
ocr_k = settings.get_api_key("ocr")
return {
"google": gemini_k.get_secret_value() if gemini_k else None,
"gemini": gemini_k.get_secret_value() if gemini_k else None,
"openai": openai_k.get_secret_value() if openai_k else None,
"anthropic": anthropic_k.get_secret_value() if anthropic_k else None,
"openrouter": openrouter_k.get_secret_value() if openrouter_k else None,
"ocr": ocr_k.get_secret_value() if ocr_k else None,
}
@router.post("/credentials/test")
async def test_credentials(request: ValidateCredentialsRequest):
"""Test and verify whether an API key is valid and usable with the corresponding provider endpoint."""
from artemis.utils.credentials_validator import validate_api_key
provider = request.provider.strip().lower()
key = request.api_key.strip()
if not key:
raise HTTPException(status_code=400, detail="API key cannot be empty.")
is_valid, message = await validate_api_key(
provider=provider,
api_key=key,
base_url=request.base_url,
)
if not is_valid:
raise HTTPException(status_code=400, detail=message)
return {
"valid": True,
"provider": provider,
"message": message,
}
@router.post("/credentials")
async def update_credentials(request: UpdateCredentialsRequest):
"""Dynamically configure and persist LLM or Vision API key, returning updated readiness report."""
from artemis.utils.credentials_validator import validate_api_key
provider = request.provider.strip().lower()
key = request.api_key.strip()
# If a non-empty key is provided, verify it before saving
if key:
is_valid, validation_msg = await validate_api_key(provider=provider, api_key=key)
if not is_valid:
raise HTTPException(
status_code=400,
detail=f"API key verification failed: {validation_msg}",
)
try:
from artemis.config import settings
settings.set_api_key(provider, key, persist_to_env=request.persist_to_env)
# Re-run all diagnostic probes to build updated report
updated_report = await readiness_engine.run_all()
action_desc = (
"successfully verified, updated, and applied" if key else "successfully cleared"
)
return {
"status": "success",
"message": f"API key for {provider} {action_desc}.",
"provider": provider,
"report": updated_report,
}
except HTTPException:
raise
except Exception as exc:
raise HTTPException(status_code=500, detail=f"Failed to update credentials: {exc}")
@router.get("/model-config-env")
async def get_model_config_and_env():
"""Retrieve the current active artemis.jsonc configuration and .env status for custom setup."""
import os
from artemis.config.paths import ROOT_DIR, get_config_path
from artemis.config import settings
from artemis.utils.file import load_jsonc
from artemis.config.settings import is_placeholder_key
# 1. Config file resolution
config_path = None
config_content = ""
parsed_config = {}
try:
config_path_obj = get_config_path("artemis.jsonc")
config_path = str(config_path_obj)
config_content = config_path_obj.read_text(encoding="utf-8")
with open(config_path_obj, encoding="utf-8") as f:
parsed_config = load_jsonc(f)
except Exception as e:
config_content = f"// Error reading config: {e}"
# 2. Env file resolution
env_path = ROOT_DIR / ".env"
# 3. Relevant Env Variables status
def get_real_env_value(key_name: str, provider: str) -> str | None:
k = settings.get_api_key(provider)
if k and not is_placeholder_key(k):
return k.get_secret_value()
val = os.environ.get(key_name)
if val and val.strip() and not is_placeholder_key(val):
return val.strip()
return None
def mask_key(k: str | None) -> str | None:
if not k:
return None
if len(k) <= 8:
return "****"
return f"{k[:4]}...{k[-4:]}"
gemini_real = get_real_env_value("GEMINI_API_KEY", "google") or get_real_env_value(
"GOOGLE_API_KEY", "google"
)
openai_real = get_real_env_value("OPENAI_API_KEY", "openai")
anthropic_real = get_real_env_value("ANTHROPIC_API_KEY", "anthropic")
openrouter_real = get_real_env_value("OPEN_ROUTER_API_KEY", "openrouter")
xai_real = get_real_env_value("XAI_API_KEY", "xai")
base_url_val = settings.OPENAI_BASE_URL or os.environ.get("OPENAI_BASE_URL")
if base_url_val and is_placeholder_key(base_url_val):
base_url_val = None
ocr_real = get_real_env_value("OCR_API_KEY", "ocr") or get_real_env_value(
"VISION_API_KEY", "ocr"
)
env_vars = [
{
"name": "GEMINI_API_KEY",
"provider": "google",
"is_set": bool(gemini_real),
"preview": mask_key(gemini_real),
"description": "Google Gemini multimodal vision API key (free tier available)",
},
{
"name": "OPENAI_API_KEY",
"provider": "openai",
"is_set": bool(openai_real),
"preview": mask_key(openai_real),
"description": "OpenAI API key (GPT-4o, GPT-4o-mini)",
},
{
"name": "ANTHROPIC_API_KEY",
"provider": "anthropic",
"is_set": bool(anthropic_real),
"preview": mask_key(anthropic_real),
"description": "Anthropic Claude API key (Claude 3.5 Sonnet, Claude 3.7)",
},
{
"name": "OPEN_ROUTER_API_KEY",
"provider": "openrouter",
"is_set": bool(openrouter_real),
"preview": mask_key(openrouter_real),
"description": "OpenRouter unified API gateway key",
},
{
"name": "XAI_API_KEY",
"provider": "xai",
"is_set": bool(xai_real),
"preview": mask_key(xai_real),
"description": "xAI Grok vision API key",
},
{
"name": "OPENAI_BASE_URL",
"provider": "custom",
"is_set": bool(base_url_val),
"preview": base_url_val,
"description": "Custom API endpoint (for local Ollama, vLLM, DeepSeek, or proxies)",
},
{
"name": "VISION_API_KEY",
"provider": "ocr",
"is_set": bool(ocr_real),
"preview": mask_key(ocr_real),
"description": "Google Cloud Vision OCR key for screen text detection (optional)",
},
]
return {
"config_path": config_path or "config/artemis.jsonc",
"config_filename": "artemis.jsonc",
"config_content": config_content,
"default_model": parsed_config.get("default", {}),
"presets": parsed_config.get("presets", {}),
"env_path": str(env_path),
"env_filename": ".env",
"env_vars": env_vars,
}
+83 -41
View File
@@ -23,6 +23,7 @@ try:
from admin_console.schemas.task_schema import RunRequest
from admin_console.services.ipc_service import ipc_service
from admin_console.services.model_service import model_service
from admin_console.services.task_preset_catalog import task_recommendation_engine
from admin_console.services.task_queue_service import task_queue_service
except ImportError:
from apps.admin_console.core.state import state
@@ -30,12 +31,37 @@ except ImportError:
from apps.admin_console.schemas.task_schema import RunRequest
from apps.admin_console.services.ipc_service import ipc_service
from apps.admin_console.services.model_service import model_service
from apps.admin_console.services.task_preset_catalog import task_recommendation_engine
from apps.admin_console.services.task_queue_service import task_queue_service
router = APIRouter(tags=["tasks"])
@router.get("/api/tasks/presets")
async def get_task_presets(
category: str = "recommended",
packages: str | None = None,
limit: int = 24,
):
"""Retrieve smart task recommendations optionally tailored to detected device packages."""
pkg_list = [p.strip() for p in packages.split(",") if p.strip()] if packages else None
return task_recommendation_engine.recommend_tasks(
installed_packages=pkg_list,
category=category,
limit=limit,
)
@router.get("/api/tasks/catalog")
async def get_task_catalog():
"""Retrieve full catalog of predefined tasks and app package registry."""
return {
"tasks": [t.model_dump() for t in task_recommendation_engine.get_all_tasks()],
"app_registry": task_recommendation_engine.get_app_registry(),
}
@router.post("/api/run")
async def run_task(request: RunRequest):
incoming_goals = []
@@ -51,7 +77,12 @@ async def run_task(request: RunRequest):
)
return await task_queue_service.enqueue_tasks(
incoming_goals, profile=request.profile or "flash"
incoming_goals,
profile=request.profile or "flash",
expected_output=request.expected_output,
enable_outputter=request.enable_outputter,
locked_app_package=request.locked_app_package,
app_path=request.app_path,
)
@@ -73,15 +104,33 @@ async def resume_task():
@router.get("/api/status")
async def get_status():
# Watchdog check to ensure background worker is alive
task_queue_service.ensure_worker_running()
latest_session = session_repo.get_latest_session()
latest_session_id = latest_session.get("session_id") if latest_session else None
bg_tasks = session_repo.get_background_tasks(latest_session_id) if latest_session_id else []
is_running = state.is_running
running_sid = state.active_session_id or (
session_repo.get_running_session_id() if is_running else None
running_task = next(
(t for t in state.queue_items if isinstance(t, dict) and t.get("status") == "running"), None
)
if not running_task and is_running:
running_task = next(
(t for t in state.queue_items if isinstance(t, dict) and t.get("status") == "pending"),
None,
)
running_sid = (
state.active_session_id
or (running_task.get("session_id") if running_task else None)
or (session_repo.get_running_session_id() if is_running else None)
)
running_goal = state.current_goal or (running_task.get("goal") if running_task else None)
active_profile = state.current_profile or (
running_task.get("profile") if running_task else None
)
active_profile = state.current_profile
if not active_profile and (running_sid or latest_session_id):
check_sid = running_sid or latest_session_id
sess_row = session_repo.get_session_by_id(check_sid)
@@ -95,30 +144,17 @@ async def get_status():
model_info = model_service.get_active_model_info(active_profile)
queue_data = state.queue_tasks
if state.current_process:
if state.current_process.returncode is None:
is_paused = state.is_paused
return {
"status": "paused" if is_paused else "running",
"goal": state.current_goal,
"pid": state.current_process.pid,
"session_id": running_sid,
"background_tasks": bg_tasks,
"queue": queue_data,
"model_info": model_info,
}
else:
completed_status = {
"status": "completed",
"goal": state.current_goal,
"returncode": state.current_process.returncode,
"session_id": latest_session_id,
"background_tasks": bg_tasks,
"queue": queue_data,
"model_info": model_info,
}
state.current_process = None
return completed_status
if is_running:
is_paused = state.is_paused
return {
"status": "paused" if is_paused else "running",
"goal": running_goal,
"pid": state.current_process.pid if state.current_process else None,
"session_id": running_sid,
"background_tasks": bg_tasks,
"queue": queue_data,
"model_info": model_info,
}
if latest_session_id and str(latest_session_id) in state.active_connections:
conn_info = state.active_connections[str(latest_session_id)]
@@ -151,21 +187,27 @@ async def stream_events(session_id: str, client: str | None = None):
def callback(event_type, data):
try:
# Filter events by session_id when subscribed to a specific session
if session_id and session_id not in ("all", "active"):
evt_session_id = None
if isinstance(data, dict):
evt_session_id = data.get("session_id")
# Global queue lifecycle events should always be delivered
if event_type not in (
"session_started",
"session_ended",
"background_tasks_updated",
):
# Filter events by session_id when subscribed to a specific session
if session_id and session_id not in ("all", "active"):
evt_session_id = None
if isinstance(data, dict):
evt_session_id = data.get("session_id")
if evt_session_id and str(evt_session_id) != str(session_id):
return
if evt_session_id and str(evt_session_id) != str(session_id):
return
if (
not evt_session_id
and state.active_session_id
and str(state.active_session_id) != str(session_id)
):
return
if (
not evt_session_id
and state.active_session_id
and str(state.active_session_id) != str(session_id)
):
return
sanitized_data = ipc_service.sanitize_event_data(event_type, data)
loop = asyncio.get_running_loop()
@@ -19,6 +19,10 @@ class RunRequest(BaseModel):
goal: str | None = None
goals: list[str] | None = None
profile: str | None = "flash"
expected_output: str | None = None
enable_outputter: bool | None = None
locked_app_package: str | None = None
app_path: str | None = None
class ReplayRequest(BaseModel):
+53 -2
View File
@@ -42,6 +42,7 @@ for _p in (str(_workspace_root), str(_apps_dir), str(_admin_console_dir), str(_c
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import FileResponse, HTMLResponse
try:
from admin_console.core.config import (
@@ -59,7 +60,7 @@ try:
from admin_console.database.repositories.session_repository import session_repo
from admin_console.database.repositories.step_repository import step_repo
from admin_console.database.repositories.trace_repository import trace_repo
from admin_console.routers import media, replay, sessions, steps, stream, tasks
from admin_console.routers import media, replay, sessions, steps, stream, system, tasks
from admin_console.routers.replay import replay_manager
from admin_console.services.ipc_service import ipc_service
from admin_console.services.media_service import media_service
@@ -71,7 +72,7 @@ except ImportError:
)
from apps.admin_console.core.state import state
from apps.admin_console.database.repositories.session_repository import session_repo
from apps.admin_console.routers import media, replay, sessions, steps, stream, tasks
from apps.admin_console.routers import media, replay, sessions, steps, stream, system, tasks
from apps.admin_console.routers.replay import replay_manager
from apps.admin_console.services.ipc_service import ipc_service
from apps.admin_console.services.model_service import model_service
@@ -116,6 +117,7 @@ app.include_router(sessions.router)
app.include_router(steps.router)
app.include_router(tasks.router)
app.include_router(replay.router)
app.include_router(system.router)
# Mount cloud gateway router for Frappe / Cloud integration if present
try:
@@ -131,6 +133,55 @@ except ImportError:
pass
# ------------------------------------------------------------------------------
# Showcase UI (Angular 19) Unified Single-Port SPA Static Hosting
# ------------------------------------------------------------------------------
_showcase_dist = _workspace_root / "apps" / "showcase_ui" / "dist" / "frontend" / "browser"
if not _showcase_dist.exists():
_showcase_dist = _workspace_root / "apps" / "showcase_ui" / "dist"
if _showcase_dist.is_dir() and (_showcase_dist / "index.html").exists():
@app.get("/{full_path:path}", include_in_schema=False)
async def serve_showcase_spa(full_path: str):
# Do not intercept API, media, or replay paths
if (
full_path.startswith("api/")
or full_path.startswith("images/")
or full_path.startswith("videos/")
or full_path.startswith("local_file")
or full_path == "docs"
or full_path == "openapi.json"
or full_path.startswith("redoc")
):
from fastapi import HTTPException
raise HTTPException(status_code=404, detail="Endpoint not found")
# Admin / Debug Console routes
if full_path in ("admin", "debug"):
admin_index = _admin_console_dir / "index.html"
if admin_index.exists():
return HTMLResponse(admin_index.read_text(encoding="utf-8"))
# Exact static file match (js, css, images, fonts, favicon, logo)
target_file = _showcase_dist / full_path
if full_path and target_file.is_file():
return FileResponse(target_file)
# Default fallback to Angular SPA index.html
index_file = _showcase_dist / "index.html"
return HTMLResponse(index_file.read_text(encoding="utf-8"))
else:
@app.get("/", response_class=HTMLResponse, include_in_schema=False)
async def fallback_root():
admin_index = _admin_console_dir / "index.html"
if admin_index.exists():
return HTMLResponse(admin_index.read_text(encoding="utf-8"))
return HTMLResponse("<h1>Artemis Console</h1>")
# ------------------------------------------------------------------------------
# Backward compatibility exports
# ------------------------------------------------------------------------------
@@ -24,6 +24,8 @@ import subprocess
import time
from collections.abc import AsyncGenerator
from artemis.toolchain import find_adb
logger = logging.getLogger("artemis.stream_service")
@@ -41,8 +43,9 @@ class DeviceStreamService:
async def get_device_serial(self) -> str | None:
"""Find the currently connected active ADB device serial."""
try:
adb_bin = find_adb()
proc = await asyncio.create_subprocess_exec(
"adb",
adb_bin,
"devices",
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
@@ -64,10 +67,11 @@ class DeviceStreamService:
try:
start_t = time.time()
serial = await self.get_device_serial()
adb_bin = find_adb()
cmd = (
["adb", "-s", serial, "exec-out", "screencap", "-p"]
[adb_bin, "-s", serial, "exec-out", "screencap", "-p"]
if serial
else ["adb", "exec-out", "screencap", "-p"]
else [adb_bin, "exec-out", "screencap", "-p"]
)
proc = await asyncio.create_subprocess_exec(
*cmd,
+12 -3
View File
@@ -78,7 +78,7 @@ class IPCService:
except Exception:
pass
if tr_type == "tool" and isinstance(payload, dict):
if tr_type in ("tool", "action") and isinstance(payload, dict):
args = payload.get("args", payload)
data["payload"] = {"args": cls._clean_value(args)}
elif tr_type == "llm_call" and isinstance(payload, dict):
@@ -144,8 +144,17 @@ class IPCService:
data = payload.get("data")
if event_type == "session_started" and data:
state.active_session_id = data.get("session_id")
current_session_id = data.get("session_id")
incoming_sid = data.get("session_id")
if incoming_sid and str(incoming_sid) in getattr(
state, "cancelled_session_ids", set()
):
print(
f"[IPC] Ignoring session_started for cancelled session: {incoming_sid}"
)
continue
state.active_session_id = incoming_sid
current_session_id = incoming_sid
pid = data.get("pid")
goal = data.get("initial_goal")
device_info = data.get("device_info")
+5 -4
View File
@@ -26,18 +26,18 @@ class ModelService:
if "pro" in p_lower:
return {
"name": "Pro",
"id": "gemini-3.6-pro",
"id": "gemini-3.7-pro",
"provider": "google",
}
elif "flash" in p_lower:
return {
"name": "Flash",
"id": "gemini-3.6-flash",
"id": "gemini-3.7-flash",
"provider": "google",
}
model_name = "Flash"
model_id = "gemini-3.6-flash"
model_id = "gemini-3.7-flash"
provider = "google"
try:
from artemis.config import parse_llm_config
@@ -118,7 +118,8 @@ class ModelService:
m_str = str(p_obj.get("model") or p_obj.get("model_name") or "").lower()
if (
"gemini-3.6-pro" in m_str
"gemini-3.7-pro" in m_str
or "gemini-3.6-pro" in m_str
or "gemini-1.5-pro" in m_str
or "gpt-4" in m_str
or "claude" in m_str
@@ -0,0 +1,514 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Preset Task Catalog & Dynamic Recommendation Engine for Artemis."""
from typing import Any, Literal
from pydantic import BaseModel, Field
class AppInfo(BaseModel):
"""Application metadata."""
name: str
icon: str
pkg: str
category: str = "general"
class TaskPreset(BaseModel):
"""Smart suggestion task definition."""
id: str
title: str
description: str
goal: str
profile: Literal["flash", "pro"]
category: Literal["flash", "pro", "cross_app", "monitor"]
tag: str
apps: list[AppInfo]
required_packages: list[str] = Field(default_factory=list)
match_mode: Literal["any", "all"] = "any"
priority: int = 50
# ============================================================================
# APP PACKAGE REGISTRY
# ============================================================================
APP_REGISTRY: dict[str, dict[str, str]] = {
# Google Suite & System
"com.google.android.apps.maps": {"name": "Maps", "icon": "explore", "category": "navigation"},
"com.google.android.gm": {"name": "Gmail", "icon": "mail", "category": "productivity"},
"com.android.chrome": {"name": "Chrome", "icon": "public", "category": "browser"},
"com.google.android.youtube": {
"name": "YouTube",
"icon": "smart_display",
"category": "entertainment",
},
"com.android.settings": {"name": "Settings", "icon": "settings", "category": "system"},
"com.google.android.deskclock": {"name": "Clock", "icon": "timer", "category": "utility"},
"com.android.deskclock": {"name": "Clock", "icon": "timer", "category": "utility"},
"com.google.android.calculator": {
"name": "Calculator",
"icon": "calculate",
"category": "utility",
},
"com.android.calculator2": {"name": "Calculator", "icon": "calculate", "category": "utility"},
"com.google.android.apps.photos": {
"name": "Photos",
"icon": "photo_library",
"category": "media",
},
"com.google.android.calendar": {
"name": "Calendar",
"icon": "calendar_month",
"category": "productivity",
},
"com.google.android.keep": {
"name": "Keep Notes",
"icon": "note_alt",
"category": "productivity",
},
"com.android.vending": {"name": "Play Store", "icon": "storefront", "category": "tools"},
"com.google.android.apps.messaging": {
"name": "Messages",
"icon": "chat",
"category": "communication",
},
# Popular Ecosystem Apps
"com.tencent.mm": {"name": "WeChat", "icon": "forum", "category": "social"},
"com.xingin.xhs": {"name": "Xiaohongshu", "icon": "auto_stories", "category": "social"},
"com.sankuai.meituan": {"name": "Meituan", "icon": "restaurant", "category": "lifestyle"},
"com.dianping.v1": {"name": "Dianping", "icon": "star", "category": "lifestyle"},
"tv.danmaku.bili": {"name": "Bilibili", "icon": "video_library", "category": "entertainment"},
"com.eg.android.AlipayGphone": {
"name": "Alipay",
"icon": "account_balance_wallet",
"category": "finance",
},
"com.netease.cloudmusic": {
"name": "NetEase Music",
"icon": "headphones",
"category": "entertainment",
},
"com.spotify.music": {"name": "Spotify", "icon": "music_note", "category": "entertainment"},
}
# ============================================================================
# CURATED TASK PRESET LIBRARY (1-2 curated tasks per common app)
# ============================================================================
PRESET_TASK_CATALOG: list[TaskPreset] = [
# 1. Google Maps
TaskPreset(
id="maps_coffee",
title="Find Specialty Coffee",
description="Search nearby top-rated cafes in Google Maps",
goal="Open Google Maps, search for top-rated specialty coffee shops nearby, and view the top result details.",
profile="flash",
category="flash",
tag="Maps",
apps=[
AppInfo(
name="Maps",
icon="explore",
pkg="com.google.android.apps.maps",
category="navigation",
)
],
required_packages=["com.google.android.apps.maps"],
priority=95,
),
TaskPreset(
id="pro_commute_share",
title="Commute ETA & Message Draft",
description="Check transit time on Maps and draft arrival ETA in Messages",
goal="Open Google Maps to check commute time to the International Airport, calculate arrival time, then open Messages and draft an ETA text message.",
profile="pro",
category="cross_app",
tag="Maps + Messages",
apps=[
AppInfo(
name="Maps",
icon="explore",
pkg="com.google.android.apps.maps",
category="navigation",
),
AppInfo(
name="Messages",
icon="chat",
pkg="com.google.android.apps.messaging",
category="communication",
),
],
required_packages=["com.google.android.apps.maps", "com.google.android.apps.messaging"],
match_mode="all",
priority=92,
),
# 2. Gmail
TaskPreset(
id="gmail_receipts",
title="Search Order Receipts",
description="Find recent flight or delivery confirmation emails in Gmail",
goal="Open Gmail and search for recent flight or package delivery confirmation emails.",
profile="flash",
category="flash",
tag="Gmail",
apps=[
AppInfo(name="Gmail", icon="mail", pkg="com.google.android.gm", category="productivity")
],
required_packages=["com.google.android.gm"],
priority=90,
),
TaskPreset(
id="pro_email_to_calendar",
title="Email Itinerary to Calendar",
description="Extract flight or event dates from Gmail and schedule in Calendar",
goal="Open Gmail to find the latest event invitation or itinerary, extract dates and location, then open Google Calendar and create a corresponding calendar event.",
profile="pro",
category="cross_app",
tag="Gmail + Calendar",
apps=[
AppInfo(
name="Gmail", icon="mail", pkg="com.google.android.gm", category="productivity"
),
AppInfo(
name="Calendar",
icon="calendar_month",
pkg="com.google.android.calendar",
category="productivity",
),
],
required_packages=["com.google.android.gm"],
priority=94,
),
# 3. Chrome
TaskPreset(
id="chrome_research",
title="Search AI News Breakthroughs",
description="Search latest multimodal AI developments in Chrome browser",
goal="Open Chrome browser and search for latest breakthroughs in multimodal mobile AI agents.",
profile="flash",
category="flash",
tag="Chrome",
apps=[AppInfo(name="Chrome", icon="public", pkg="com.android.chrome", category="browser")],
required_packages=["com.android.chrome"],
priority=88,
),
TaskPreset(
id="pro_research_keep",
title="Product Research & Notes Note",
description="Compare top 3 headphones on Chrome and record comparison in Keep",
goal="Open Chrome, research top 3 noise-cancelling headphones comparing price and battery life, then write a structured comparison summary note in Keep Notes.",
profile="pro",
category="pro",
tag="Chrome + Keep",
apps=[
AppInfo(name="Chrome", icon="public", pkg="com.android.chrome", category="browser"),
AppInfo(
name="Keep Notes",
icon="note_alt",
pkg="com.google.android.keep",
category="productivity",
),
],
required_packages=["com.android.chrome"],
priority=91,
),
# 4. YouTube
TaskPreset(
id="youtube_lofi",
title="Play Lo-Fi Music Radio",
description="Search and play a Lo-Fi hip hop live stream on YouTube",
goal='Open YouTube, search for "Lofi hip hop beats relaxing radio" and tap on the live stream.',
profile="flash",
category="flash",
tag="YouTube",
apps=[
AppInfo(
name="YouTube",
icon="smart_display",
pkg="com.google.android.youtube",
category="entertainment",
)
],
required_packages=["com.google.android.youtube"],
priority=85,
),
# 5. Settings
TaskPreset(
id="settings_display_wifi",
title="Dark Mode & Wi-Fi Check",
description="Toggle dark theme and verify network connection in Settings",
goal="Open Settings app, navigate to Display settings, ensure Dark theme is enabled, and check Wi-Fi connection status.",
profile="flash",
category="flash",
tag="Settings",
apps=[
AppInfo(name="Settings", icon="settings", pkg="com.android.settings", category="system")
],
required_packages=["com.android.settings"],
priority=87,
),
TaskPreset(
id="pro_settings_qa",
title="Subsystem Health & Crash Probe",
description="Traverse Settings submenus to verify screens and check for crash dialogs",
goal="Explore Settings submenus (Network, Connected devices, Apps, Battery, Storage), verify each screen loads properly without ANR or crash dialogs, and summarize results.",
profile="pro",
category="monitor",
tag="Settings QA",
apps=[
AppInfo(name="Settings", icon="settings", pkg="com.android.settings", category="system")
],
required_packages=["com.android.settings"],
priority=93,
),
# 6. Clock
TaskPreset(
id="clock_timer",
title="25-Min Pomodoro Timer",
description="Start a 25-minute focus countdown timer in Clock app",
goal="Open Clock app, switch to Timer tab, set 25 minutes and start the countdown timer.",
profile="flash",
category="flash",
tag="Clock",
apps=[
AppInfo(
name="Clock", icon="timer", pkg="com.google.android.deskclock", category="utility"
)
],
required_packages=["com.google.android.deskclock", "com.android.deskclock"],
priority=86,
),
# 7. Calculator
TaskPreset(
id="calc_gratuity",
title="Split Bill & Calculate Tip",
description="Calculate 18% gratuity on $186.40 for 3 people in Calculator",
goal="Open Calculator and calculate 18% tip on a bill of $186.40, then divide by 3 people.",
profile="flash",
category="flash",
tag="Calculator",
apps=[
AppInfo(
name="Calculator",
icon="calculate",
pkg="com.google.android.calculator",
category="utility",
)
],
required_packages=["com.google.android.calculator", "com.android.calculator2"],
priority=84,
),
# 8. Photos
TaskPreset(
id="photos_inspect",
title="Inspect Recent Screenshot",
description="Open Google Photos and review the latest screenshot taken",
goal="Open Google Photos and view the most recent screenshot in the screenshots album.",
profile="flash",
category="flash",
tag="Photos",
apps=[
AppInfo(
name="Photos",
icon="photo_library",
pkg="com.google.android.apps.photos",
category="media",
)
],
required_packages=["com.google.android.apps.photos"],
priority=82,
),
# 9. WeChat (微信)
TaskPreset(
id="wechat_browse",
title="Check WeChat Messages",
description="Open WeChat and view top recent chat conversations",
goal="Open WeChat (微信) and view the top recent chat messages.",
profile="flash",
category="flash",
tag="WeChat",
apps=[AppInfo(name="WeChat", icon="forum", pkg="com.tencent.mm", category="social")],
required_packages=["com.tencent.mm"],
priority=89,
),
TaskPreset(
id="pro_wechat_to_calendar",
title="WeChat Notice to Calendar",
description="Extract meeting notice from WeChat chat and add to Calendar",
goal="Open WeChat (微信), locate the latest meeting announcement or event message in the top chat, extract the time and topic, then open Calendar and schedule an event.",
profile="pro",
category="cross_app",
tag="WeChat + Calendar",
apps=[
AppInfo(name="WeChat", icon="forum", pkg="com.tencent.mm", category="social"),
AppInfo(
name="Calendar",
icon="calendar_month",
pkg="com.google.android.calendar",
category="productivity",
),
],
required_packages=["com.tencent.mm"],
priority=93,
),
# 10. Xiaohongshu (小红书)
TaskPreset(
id="xhs_coffee_guide",
title="RED Cafe Guide Search",
description="Search trending specialty cafe reviews on Xiaohongshu",
goal="Open Xiaohongshu (小红书), search for top-rated specialty coffee shops, and view the top post.",
profile="flash",
category="flash",
tag="Xiaohongshu",
apps=[
AppInfo(
name="Xiaohongshu", icon="auto_stories", pkg="com.xingin.xhs", category="social"
)
],
required_packages=["com.xingin.xhs"],
priority=87,
),
# 11. Meituan / Dianping (美团 / 大众点评)
TaskPreset(
id="meituan_ramen_search",
title="Meituan Food Search",
description="Search top-rated Ramen nearby on Meituan or Dianping",
goal="Open Meituan (美团) or Dianping (大众点评), search for top-rated Ramen nearby, and view top restaurant rating.",
profile="flash",
category="flash",
tag="Meituan",
apps=[
AppInfo(
name="Meituan", icon="restaurant", pkg="com.sankuai.meituan", category="lifestyle"
)
],
required_packages=["com.sankuai.meituan", "com.dianping.v1"],
priority=86,
),
# 12. Bilibili (哔哩哔哩)
TaskPreset(
id="bilibili_stream",
title="Bilibili Tech Video",
description="Search and play an AI Agent tutorial video on Bilibili",
goal='Open Bilibili (哔哩哔哩), search for "AI Agent 架构实战", and play the top matching video.',
profile="flash",
category="flash",
tag="Bilibili",
apps=[
AppInfo(
name="Bilibili",
icon="video_library",
pkg="tv.danmaku.bili",
category="entertainment",
)
],
required_packages=["tv.danmaku.bili"],
priority=85,
),
# 13. Play Store
TaskPreset(
id="pro_playstore_review",
title="Play Store App Review Study",
description="Compare top task management apps and ratings on Google Play",
goal="Open Google Play Store, search for top rated task management apps, compare ratings and latest user reviews of the top 2 candidates, and record recommendations.",
profile="pro",
category="pro",
tag="Play Store",
apps=[
AppInfo(
name="Play Store", icon="storefront", pkg="com.android.vending", category="tools"
)
],
required_packages=["com.android.vending"],
priority=88,
),
]
# ============================================================================
# RECOMMENDATION ENGINE
# ============================================================================
class TaskRecommendationEngine:
"""Intelligent recommendation engine matching device capabilities."""
def __init__(self, catalog: list[TaskPreset] | None = None):
self.catalog = catalog or PRESET_TASK_CATALOG
def get_all_tasks(self) -> list[dict[str, Any]]:
return [task.model_dump() for task in self.catalog]
def recommend_tasks(
self, installed_packages: list[str] | set[str], category: str = "all", limit: int = 12
) -> list[dict[str, Any]]:
pkgs_set = (
set(installed_packages) if isinstance(installed_packages, list) else installed_packages
)
scored_tasks: list[tuple[int, TaskPreset, bool]] = []
for task in self.catalog:
# Check package presence
is_matched = False
if pkgs_set:
if not task.required_packages:
is_matched = True
elif task.match_mode == "all":
is_matched = all(pkg in pkgs_set for pkg in task.required_packages)
else:
is_matched = any(pkg in pkgs_set for pkg in task.required_packages)
else:
is_matched = False
# Calculate score
score = task.priority
if pkgs_set:
if is_matched:
score += 100
if len(task.required_packages) > 1 and task.match_mode == "all":
score += 30
else:
score -= 40
# Filter by category
if category == "flash" and task.profile != "flash":
continue
elif category == "pro" and task.profile != "pro":
continue
elif category == "cross_app" and task.category != "cross_app":
continue
elif category == "monitor" and task.category != "monitor":
continue
scored_tasks.append((score, task, is_matched))
# Sort descending by score
scored_tasks.sort(key=lambda item: item[0], reverse=True)
results: list[dict[str, Any]] = []
for _, task, matched in scored_tasks[:limit]:
d = task.model_dump()
d["is_device_matched"] = matched
results.append(d)
return results
task_recommendation_engine = TaskRecommendationEngine()
+254 -65
View File
@@ -16,7 +16,6 @@ import asyncio
from datetime import datetime
import os
import shutil
import signal
import sys
import time
from typing import Any
@@ -31,6 +30,7 @@ try:
)
from admin_console.core.state import state
from admin_console.database.repositories.session_repository import session_repo
from artemis.runtime import process_supervisor
except ImportError:
from apps.admin_console.core.config import (
PAUSE_FILE,
@@ -40,6 +40,7 @@ except ImportError:
)
from apps.admin_console.core.state import state
from apps.admin_console.database.repositories.session_repository import session_repo
from artemis.runtime import process_supervisor
class TaskQueueService:
@@ -48,51 +49,114 @@ class TaskQueueService:
"""
@classmethod
async def queue_worker(cls):
while True:
goal = None
sess_id = None
def _broadcast_event(cls, event_type: str, data: Any):
"""Broadcasts an event safely to all registered subscribers."""
for cb in list(state.ipc_subscribers):
try:
item = await state.task_queue.get()
if isinstance(item, dict):
goal = item.get("goal")
profile = item.get("profile", "flash")
sess_id = item.get("session_id")
else:
goal = item
profile = "flash"
sess_id = None
cb(event_type, data)
except Exception:
pass
@classmethod
def _get_next_pending_task(cls) -> dict[str, Any] | None:
"""Finds and returns the first pending task from queue_items."""
for item in state.queue_items:
if isinstance(item, dict) and item.get("status") == "pending":
return item
return None
@classmethod
def _remove_task(cls, session_id: str | None):
"""Removes a task from queue_items by session_id."""
if not session_id:
return
state.queue_items = [
t
for t in state.queue_items
if not (isinstance(t, dict) and t.get("session_id") == session_id)
]
@classmethod
def ensure_worker_running(cls):
"""Guarantees that the background queue worker task is active and running."""
try:
loop = asyncio.get_running_loop()
if state.worker_task is None or state.worker_task.done():
state.worker_task = loop.create_task(cls.queue_worker())
try:
state.wake_event.set()
except Exception:
pass
except RuntimeError:
pass
@classmethod
async def queue_worker(cls):
"""Persistent, self-healing background worker loop processing tasks in FIFO order."""
print("[QueueWorker] Background worker loop initialized and running.")
while True:
task_item = None
sess_id = None
goal = None
profile = "flash"
try:
if state.is_running:
await asyncio.sleep(0.2)
continue
# 1. Fetch the next pending task
task_item = cls._get_next_pending_task()
if task_item is None:
await asyncio.sleep(0.3)
continue
# 2. Extract and initialize task execution metadata
sess_id = task_item.get("session_id")
if sess_id and str(sess_id) in getattr(state, "cancelled_session_ids", set()):
cls._remove_task(sess_id)
continue
goal = task_item.get("goal")
profile = task_item.get("profile", "flash")
expected_output = task_item.get("expected_output")
enable_outputter = task_item.get("enable_outputter")
locked_app = task_item.get("locked_app_package") or task_item.get("locked_app")
app_path = task_item.get("app_path")
task_item["status"] = "running"
task_item["start_time"] = time.time()
state.current_goal = goal
state.current_profile = profile
if sess_id:
state.active_session_id = sess_id
state.active_session_id = sess_id
state.was_stopped_manually = False
# Remove from state.queue_tasks
state.queue_tasks = [
t
for t in state.queue_tasks
if not (
isinstance(t, dict)
and (t.get("session_id") == sess_id or t.get("goal") == goal)
)
]
# Broadcast session_started so all connected clients know the task has started
cls._broadcast_event(
"session_started",
{
"session_id": sess_id,
"initial_goal": goal,
"profile": profile,
},
)
test_name = f"web_{int(time.time())}"
env = os.environ.copy()
pythonpath_parts = [
str(WORKSPACE_ROOT),
str(WORKSPACE_ROOT / "apps/admin_console"),
str(WORKSPACE_ROOT / "apps/cloud_service"),
str(WORKSPACE_ROOT / "apps" / "admin_console"),
str(WORKSPACE_ROOT / "apps" / "cloud_service"),
env.get("PYTHONPATH", ""),
]
env["PYTHONPATH"] = os.pathsep.join([p for p in pythonpath_parts if p])
env["PYTHONUTF8"] = "1"
env["PYTHONUNBUFFERED"] = "1"
if state.ipc_port is not None:
env["ARTEMIS_IPC_PORT"] = str(state.ipc_port)
if sess_id:
env["ARTEMIS_SESSION_ID"] = str(sess_id)
print(f"[QueueWorker] Starting task: {goal} with profile: {profile}")
state.current_process = await asyncio.create_subprocess_exec(
cmd = [
sys.executable,
"-m",
"artemis.main",
@@ -101,48 +165,98 @@ class TaskQueueService:
profile,
"--test-name",
test_name,
]
if expected_output:
cmd.extend(["--output-description", str(expected_output)])
if enable_outputter is not None:
cmd.append("--enable-outputter" if enable_outputter else "--disable-outputter")
if locked_app:
cmd.extend(["--locked-app", str(locked_app)])
if app_path:
cmd.extend(["--app-path", str(app_path)])
print(
f"[QueueWorker] Starting task [{sess_id}]: '{goal}' (profile: {profile}, outputter: {bool(expected_output or enable_outputter)})"
)
proc = await asyncio.create_subprocess_exec(
*cmd,
cwd=str(WORKSPACE_ROOT),
env=env,
)
await state.current_process.wait()
print(
f"[QueueWorker] Task finished with code "
f"{state.current_process.returncode}: {goal}"
)
state.current_process = proc
task_item["pid"] = proc.pid
# Fallback status update
if state.active_session_id:
curr_status = session_repo.get_session_status(state.active_session_id)
if curr_status == "running":
if state.was_stopped_manually:
new_status = "cancelled"
else:
new_status = (
"completed" if state.current_process.returncode == 0 else "failed"
)
session_repo.update_session_status(
state.active_session_id, new_status, time.time()
)
print(
f"[QueueWorker] Fallback: Updated session "
f"{state.active_session_id} status to {new_status}"
)
state.active_session_id = None
if sess_id and (
str(sess_id) in getattr(state, "cancelled_session_ids", set())
or state.was_stopped_manually
):
print(
f"[QueueWorker] Task [{sess_id}] was cancelled during launch. Terminating."
)
try:
proc.kill()
except Exception:
pass
state.current_profile = None
state.was_stopped_manually = False
# 3. Await subprocess completion
returncode = await proc.wait()
print(f"[QueueWorker] Task [{sess_id}] exited with returncode {returncode}")
# 4. Perform fallback database status update and notification
if sess_id:
if state.was_stopped_manually:
new_status = "cancelled"
elif returncode == 0:
new_status = "completed"
else:
new_status = "failed"
session_repo.update_session_status(sess_id, new_status, time.time())
state.active_connections.pop(sess_id, None)
print(f"[QueueWorker] Updated session {sess_id} status to '{new_status}'")
cls._broadcast_event(
"session_ended",
{
"session_id": sess_id,
"status": new_status,
"was_stopped_manually": state.was_stopped_manually,
},
)
except asyncio.CancelledError:
print("[QueueWorker] Background worker task was cancelled.")
print("[QueueWorker] Task received cancellation signal.")
if state.current_process and state.current_process.returncode is None:
try:
state.current_process.kill()
except Exception:
pass
break
except Exception as e:
print(f"[QueueWorker] Error executing task: {e}")
print(f"[QueueWorker] Unexpected error executing task: {e}")
await asyncio.sleep(0.5)
finally:
if goal is not None:
state.task_queue.task_done()
# 5. Clean up completed task from queue and reset execution state
if sess_id:
cls._remove_task(sess_id)
state.current_process = None
state.active_session_id = None
state.current_goal = None
state.current_profile = None
state.was_stopped_manually = False
state.wake_event.set()
@classmethod
async def enqueue_tasks(cls, goals: list[str], profile: str = "flash") -> dict[str, Any]:
async def enqueue_tasks(
cls,
goals: list[str],
profile: str = "flash",
expected_output: str | None = None,
enable_outputter: bool | None = None,
locked_app_package: str | None = None,
app_path: str | None = None,
) -> dict[str, Any]:
"""Enqueues one or more goals and wakes up the background worker."""
cls.ensure_worker_running()
enqueued_tasks = []
now = time.time()
for i, goal in enumerate(goals):
@@ -151,14 +265,20 @@ class TaskQueueService:
"session_id": sess_id,
"goal": goal,
"profile": profile or "flash",
"expected_output": expected_output,
"enable_outputter": enable_outputter,
"locked_app_package": locked_app_package,
"app_path": app_path,
"status": "pending",
"created_at": now + i * 0.001,
"start_time": now + i * 0.001,
}
await state.task_queue.put(task_item)
state.queue_tasks.append(task_item)
state.queue_items.append(task_item)
enqueued_tasks.append(task_item)
# Wake worker immediately
state.wake_event.set()
return {
"status": "queued" if state.is_running else "started",
"tasks": enqueued_tasks,
@@ -168,36 +288,105 @@ class TaskQueueService:
@classmethod
def stop_tasks(cls, clear_all: bool = False) -> bool:
"""Stops the currently executing task and optionally clears pending tasks."""
stopped = False
stopped_session_id = state.active_session_id
if not stopped_session_id:
running_item = next(
(
t
for t in state.queue_items
if isinstance(t, dict) and t.get("status") == "running"
),
None,
)
if running_item:
stopped_session_id = running_item.get("session_id")
elif state.queue_items and not clear_all:
stopped_session_id = state.queue_items[0].get("session_id")
if stopped_session_id:
getattr(state, "cancelled_session_ids", set()).add(str(stopped_session_id))
for t in state.queue_items:
if isinstance(t, dict) and t.get("status") == "running" and t.get("session_id"):
getattr(state, "cancelled_session_ids", set()).add(str(t["session_id"]))
if clear_all:
state.clear_queue()
stopped = True
state.was_stopped_manually = True
if state.is_running and state.current_process:
if state.current_process:
pid = getattr(state.current_process, "pid", None)
if pid:
try:
process_supervisor.terminate_tree(pid)
except Exception:
pass
try:
state.current_process.kill()
stopped = True
except Exception:
pass
stopped = True
for _, conn_info in list(state.active_connections.items()):
pid = conn_info.get("pid")
if pid:
try:
os.kill(pid, signal.SIGINT)
process_supervisor.terminate_tree(pid)
stopped = True
except Exception as e:
print(f"Failed to kill external process {pid}: {e}")
# Always reset process and active states cleanly
state.current_process = None
state.active_connections.clear()
session_repo.mark_all_running_cancelled()
# Broadcast session_ended event so all SSE streams and subscribers update immediately
if stopped_session_id:
cls._broadcast_event(
"session_ended",
{
"session_id": stopped_session_id,
"status": "cancelled",
"was_stopped_manually": True,
},
)
stopped = True
state.active_session_id = None
state.current_goal = None
state.current_profile = None
# Ensure any running or stopped task is removed from queue_items so worker_loop is unblocked
if clear_all:
state.queue_items = []
else:
state.queue_items = [
t
for t in state.queue_items
if not (
isinstance(t, dict)
and (
t.get("status") == "running"
or (stopped_session_id and t.get("session_id") == stopped_session_id)
)
)
]
if PAUSE_FILE.exists():
try:
PAUSE_FILE.unlink()
except Exception:
pass
return stopped
# Wake up worker to process the next pending task or transition cleanly
cls.ensure_worker_running()
state.wake_event.set()
return stopped or True
@classmethod
def resume_task(cls) -> bool: