feat: add step-by-step visual replay and harden process lifecycle

- UI: Implement dual-mode playback in FloatingVideoPlayer supporting video and step-by-step frame replay with scrub controls and auto-fallback
- Runtime: Prevent asyncio event loop hangs on Darwin and Linux by avoiding psutil.wait_procs on direct child processes
- Task Queue: Add worker exit watchdog and automatic recording recovery for completed or interrupted sessions
- Recording: Add ffmpeg ultrafast transcode fallback, shield recording finalization against cancellation, and handle on-the-fly MP4 conversion
- Docs: Polish README_CN styling and streamline flash runner agent prompt
This commit is contained in:
somew1nd committed 2026-08-26 00:53:24 -07:00
1 parent 06643e955b
commit a36a66ce6d
26 files changed
+1152 -117

No files matched your search

+18 -18
View File
@@ -28,21 +28,21 @@
<p align="center"> <p align="center">
<img src="./docs/assets/demo.gif" alt="Artemis 演示效果" width="100%" /> <img src="./docs/assets/demo.gif" alt="Artemis 演示效果" width="100%" />
<br> <br>
<em><b>实机演示</b>:在 Google Maps 中设置驾车路线并计算总耗时,随后打开 YouTube 播放 Coldplay 的歌曲。</em> <em>实机演示:在 Google Maps 中设置驾车路线并计算总耗时,随后打开 YouTube 播放 Coldplay 的歌曲。</em>
</p> </p>
## ✨ 核心亮点 ## 核心亮点
* 🤖 **跨 App 复杂自动化与 AI 个人助理**:不仅能执行严格的自动化测试用例,更具备接管真机的自主决策与长流程操作能力,一句话处理跨应用复杂业务与任务; * **跨 App 复杂自动化与 AI 个人助理**:不仅能执行严格的自动化测试用例,更具备接管真机的自主决策与长流程操作能力,一句话处理跨应用复杂业务与任务;
* 🧪 **零脚本维护自动化测试**:基于“动态优先、坐标兜底”的多模态语义定位,彻底告别传统 XPath / 控件 ID 频繁失效的痛点,无惧 App 改版、分辨率差异与系统升级; * **零脚本维护自动化测试**:基于“动态优先、坐标兜底”的多模态语义定位,彻底告别传统 XPath / 控件 ID 频繁失效的痛点,无惧 App 改版、分辨率差异与系统升级;
* 🐞 **IDE 内一键 Bug 复现与 Logcat 诊断**:原生支持 **MCP 协议**,可在 **Antigravity、Claude Code、Windsurf** 中直接用自然语言驱动真机复现缺陷,自动抓取关键帧截图与 **Logcat 崩溃堆栈**,完成研发测试闭环; * **IDE 内一键 Bug 复现与 Logcat 诊断**:原生支持 **MCP 协议**,可在 **Antigravity、Claude Code、Windsurf** 中直接用自然语言驱动真机复现缺陷,自动抓取关键帧截图与 **Logcat 崩溃堆栈**,完成研发测试闭环;
* ⚡ **极速执行吞吐(单步 3–5 秒)**:首创**全链路乐观异步流水线 (Optimistic Async Pipeline)**,解耦重度推理与页面操作,在 Flash 模式下实现丝滑的高频交互与快速回归验证; * **极速执行吞吐(单步 3–5 秒)**:首创**全链路乐观异步流水线 (Optimistic Async Pipeline)**,解耦重度推理与页面操作,在 Flash 模式下实现丝滑的高频交互与快速回归验证;
* 🛡️ **干扰弹窗自愈与超长程巡检**:独创 **Safety Net** 执行前校验机制,自动识别并清除系统权限弹窗、通知遮挡等异常;Pro 模式支持连续 **10+ 小时** 无人值守稳定性巡检与探索性测试 (Exploratory / Monkey-plus Testing); * **干扰弹窗自愈与超长程巡检**:独创 **Safety Net** 执行前校验机制,自动识别并清除系统权限弹窗、通知遮挡等异常;Pro 模式支持连续 **10+ 小时** 无人值守稳定性巡检与探索性测试 (Exploratory / Monkey-plus Testing);
* **业界顶尖 SOTA**:在 Google Research **AndroidWorld** 基准评测(100+ 复杂长程系统与应用交互任务)中取得 **99%+ 任务完成率**。 * **业界顶尖 SOTA**:在 Google Research **AndroidWorld** 基准评测(100+ 复杂长程系统与应用交互任务)中取得 **99%+ 任务完成率**。
<a id="workflow-showcase"></a> <a id="workflow-showcase"></a>
<a id="全流程演示"></a> <a id="全流程演示"></a>
## 🤝 Antigravity × ARTEMIS:全流程自主测试演示 ## Antigravity × ARTEMIS:全流程自主测试演示
通过原生 MCP 协议,**Antigravity** 与 **ARTEMIS** 深度协同——只需一句自然语言指令,即可自动完成从需求理解、测试规划、真机执行到深度报告输出的完整闭环: 通过原生 MCP 协议,**Antigravity** 与 **ARTEMIS** 深度协同——只需一句自然语言指令,即可自动完成从需求理解、测试规划、真机执行到深度报告输出的完整闭环:
@@ -62,7 +62,7 @@
<tr> <tr>
<td width="50%" align="center"> <td width="50%" align="center">
<b>3. 自主执行测试 (Autonomous Test Execution)</b><br> <b>3. 自主执行测试 (Autonomous Test Execution)</b><br>
<sub>接管真机自动化操作,跳过广告并实时采集性能指标</sub><br><br> <sub>驱动真机操作、界面导航并实时分析性能指标</sub><br><br>
<img src="./docs/assets/workflow-3-exec.png" width="100%" alt="步骤三:自主执行测试" /> <img src="./docs/assets/workflow-3-exec.png" width="100%" alt="步骤三:自主执行测试" />
</td> </td>
<td width="50%" align="center"> <td width="50%" align="center">
@@ -78,8 +78,8 @@
## 快速上手 ## 快速上手
确保电脑已连接 Android 实体机(已开启 **USB 调试**)或 Android 模拟器。一键启动脚本将会自动完成以下配置: 确保电脑已连接 Android 实体机(已开启 **USB 调试**)或 Android 模拟器。一键启动脚本将会自动完成以下配置:
- 🛠️ **安装系统环境依赖**:自动检测并安装 ADB、scrcpy、FFmpeg 与 Python(`uv`)运行时及项目依赖。 - **安装系统环境依赖**:自动检测并安装 ADB、scrcpy、FFmpeg 与 Python(`uv`)运行时及项目依赖。
- 🔌 **全局挂载 MCP 服务与测试准则 (Rules)**:主动引导并自动将全局 MCP 服务与 **Artemis 移动端测试思维准则 (`rules.md`)** 挂载至你使用的 AI IDE(支持 **Antigravity**、**Cursor**、**Claude Code**、**Codex**、**Windsurf**、**VS Code**、**Cline/Roo**、**OpenClaw**)。 - **全局挂载 MCP 服务与测试准则 (Rules)**:主动引导并自动将全局 MCP 服务与 **Artemis 移动端测试思维准则 (`rules.md`)** 挂载至你使用的 AI IDE(支持 **Antigravity**、**Cursor**、**Claude Code**、**Codex**、**Windsurf**、**VS Code**、**Cline/Roo**、**OpenClaw**)。
### macOS 与 Linux ### macOS 与 Linux
@@ -109,7 +109,7 @@ cd artemis
<a id="mcp-setup"></a> <a id="mcp-setup"></a>
<a id="mcp"></a> <a id="mcp"></a>
<details> <details>
<summary><b>🔌 接入 Codex / Antigravity / Claude Code / Windsurf (MCP)(点击展开)</b></summary> <summary><b>Codex / Antigravity / Claude Code / Windsurf MCP 配置(点击展开)</b></summary>
<br> <br>
@@ -192,17 +192,17 @@ PYTHONPATH = "/path/to/artemis"
* **Codex**:将内容添加至 `~/.codex/AGENTS.md`(或当前生效的 `AGENTS.override.md`)。 * **Codex**:将内容添加至 `~/.codex/AGENTS.md`(或当前生效的 `AGENTS.override.md`)。
* **Windsurf / OpenClaw**:将内容添加到工作区规则或全局 System Prompt 中。 * **Windsurf / OpenClaw**:将内容添加到工作区规则或全局 System Prompt 中。
> 💡 更多规范设计细节与 MCP 架构说明,请参阅 [MCP Server 文档](./mcp_server/README.md)。 > 更多规范设计细节与 MCP 架构说明,请参阅 [MCP Server 文档](./mcp_server/README.md)。
### 4. 在 IDE 中体验真机协同 ### 4. 在 IDE 中体验真机协同
在 Codex / Antigravity / Claude Code 对话框中直接输入: 在 Codex / Antigravity / Claude Code 对话框中直接输入:
> 💬 *"请帮我把刚刚修改的代码编译成 APK 并安装到手机上,打开登录页面输入测试账号,验证登录后是否有异常弹窗,并把最终页面截图回传。"* > *"请帮我把刚刚修改的代码编译成 APK 并安装到手机上,打开登录页面输入测试账号,验证登录后是否有异常弹窗,并把最终页面截图回传。"*
</details> </details>
<a id="python-sdk"></a> <a id="python-sdk"></a>
<details> <details>
<summary><b>🐍 Python SDK 极简集成(点击展开)</b></summary> <summary><b>Python SDK 集成(点击展开)</b></summary>
<br> <br>
@@ -238,12 +238,12 @@ if __name__ == "__main__":
<p align="center"> <p align="center">
<img src="./docs/assets/artemis-ui-showcase.png" alt="Artemis 可视化控制台" width="100%" /> <img src="./docs/assets/artemis-ui-showcase.png" alt="Artemis 可视化控制台" width="100%" />
<br /> <br />
<sub><b>控制台功能布局</b>:<b>① 顶栏视图切换</b>(主页与工作区) · <b>② 运行模式与录屏回放</b>(Flash/Pro 状态与视频回放) · <b>③ 实时感知推理流</b>(动作分解、点击坐标与结构化总结) · <b>④ 自然语言下发胶囊</b>(自然语言驱动真机) · <b>⑤ 任务队列看板</b>(状态流转与历史回溯)</sub> <sub><b>控制台功能概览</b>:<b>① 视图切换</b>(主页与工作区) · <b>② 运行模式与回放</b>(Flash/Pro 状态与视频回放) · <b>③ 实时感知推理流</b>(动作感知、目标坐标与结构化总结) · <b>④ 提示词输入坞 (Prompt Dock)</b>(自然语言下发) · <b>⑤ 任务队列看板</b>(生命周期与历史回溯)</sub>
</p> </p>
* **Web 可视化测试控制台 (`uv run artemis ui`)**:集成设备实时投屏与交互面板,支持通过自然语言下发测试用例,实时观测推理步骤、操作轨迹、截图留存与异常状态回放; * **Web 可视化测试控制台 (`uv run artemis ui`)**:集成设备实时投屏与交互面板,支持通过自然语言下发测试用例,实时观测推理步骤、操作轨迹、截图留存与异常状态回放;
* **MCP 协议集成 (AI IDE 协同)**:作为标准 MCP 服务器无缝接入 **Antigravity、Claude Code、Windsurf** 等开发环境,在 IDE 中直接驱动真机完成自动化测试与 Bug 复现验证; * **原生 MCP 协议 (IDE 协同)**:作为标准 MCP 服务器无缝接入 **Antigravity、Claude Code、Windsurf** 等开发环境,在 IDE 中直接驱动真机完成自动化测试与 Bug 复现验证;
* **命令行工具 (`uv run artemis run`)**:支持通过 CLI 直接执行自动化用例、稳定性巡检或 AndroidWorld 基准评测,提供高保真结构化终端输出; * **开发者命令行 CLI (`uv run artemis run`)**:支持通过终端直接执行自动化测试用例、探索性稳定性巡检或 AndroidWorld 基准评测,提供高保真结构化终端输出;
* **Python SDK**:作为标准 Python 库无缝集成至现有自动化测试框架(如 pytest)或 CI/CD 流水线,提供基于 Pydantic 的强类型结构化结果与断言支持。 * **Python SDK**:作为标准 Python 库无缝集成至现有自动化测试框架(如 pytest)或 CI/CD 流水线,提供基于 Pydantic 的强类型结构化结果与断言支持。
<a id="benchmarks"></a> <a id="benchmarks"></a>
+6
View File
@@ -204,3 +204,9 @@ class ServerState:
# Global shared instance # Global shared instance
state = ServerState() state = ServerState()
import sys
if __name__ == "admin_console.core.state":
sys.modules["apps.admin_console.core.state"] = sys.modules[__name__]
elif __name__ == "apps.admin_console.core.state":
sys.modules["admin_console.core.state"] = sys.modules[__name__]
@@ -141,6 +141,22 @@ class SessionRepository:
except Exception: except Exception:
return False return False
def mark_recording_ready(self, session_id: str, local_video_path: str) -> bool:
"""Mark a recording as ready with its finalized video path."""
try:
with db_session(self.db_path) as conn:
cursor = conn.cursor()
cursor.execute(
"UPDATE video_recordings SET status = 'ready', error = NULL, "
"local_video_path = ?, end_time = COALESCE(end_time, ?) "
"WHERE session_id = ?",
(str(local_video_path), time.time(), str(session_id)),
)
conn.commit()
return cursor.rowcount > 0
except Exception:
return False
def get_llm_traces_for_profile(self, session_id: str, limit: int = 3) -> list[str]: def get_llm_traces_for_profile(self, session_id: str, limit: int = 3) -> list[str]:
with db_session(self.db_path) as conn: with db_session(self.db_path) as conn:
cursor = conn.cursor() cursor = conn.cursor()
+12 -8
View File
@@ -102,14 +102,18 @@ async def get_session_video(session_id: str):
} }
if recording_status == "failed": if recording_status == "failed":
return { v_url = media_service.resolve_video_url(row_dict, video_rec_map, video_idx)
"session_id": session_id, if not v_url:
"status": "failed", return {
"has_video": False, "session_id": session_id,
"video_url": None, "status": "failed",
"video_segments": [], "has_video": False,
"message": recording.get("error") or "Recording finalization failed", "video_url": None,
} "video_segments": [],
"message": recording.get("error") or "Recording finalization failed",
}
# If a video was recovered/found, update DB to ready and proceed to serve it
session_repo.mark_recording_ready(session_id, v_url)
v_url = media_service.resolve_video_url(row_dict, video_rec_map, video_idx) v_url = media_service.resolve_video_url(row_dict, video_rec_map, video_idx)
video_segments = media_service.resolve_video_segments(v_url) video_segments = media_service.resolve_video_segments(v_url)
+8 -3
View File
@@ -63,12 +63,17 @@ async def list_sessions():
orphaned_ids.append(s_id) orphaned_ids.append(s_id)
recording_status = str((recording or {}).get("status") or "unavailable") recording_status = str((recording or {}).get("status") or "unavailable")
row_dict["recording_status"] = recording_status resolved_v_url = (
row_dict["video_url"] = (
media_service.resolve_video_url(row_dict, video_rec_map, video_idx) media_service.resolve_video_url(row_dict, video_rec_map, video_idx)
if recording_status in ("ready", "unavailable") if recording_status != "recording"
else None else None
) )
if resolved_v_url:
recording_status = "ready"
row_dict["video_url"] = resolved_v_url
else:
row_dict["video_url"] = None
row_dict["recording_status"] = recording_status
agent_names = agent_names_by_session.get(s_id, []) agent_names = agent_names_by_session.get(s_id, [])
sess_profile = model_service.resolve_session_profile( sess_profile = model_service.resolve_session_profile(
+74 -1
View File
@@ -17,6 +17,7 @@ import hashlib
import json import json
import os import os
from pathlib import Path from pathlib import Path
import subprocess
from typing import Any from typing import Any
import urllib.parse import urllib.parse
from fastapi import HTTPException from fastapi import HTTPException
@@ -30,6 +31,72 @@ except ImportError:
class MediaService: class MediaService:
"""Service handling media indexing, local file security, payload unwrapping and notes/plans.""" """Service handling media indexing, local file security, payload unwrapping and notes/plans."""
@classmethod
def ensure_browser_playable_video(cls, p: Path) -> Path:
"""Ensure the video file is in a browser-supported container (MP4).
If given an MKV, check if a converted MP4 already exists or convert it.
"""
try:
if not p.exists():
return p
suffix = p.suffix.lower()
if suffix == ".mp4":
return p
if suffix in (".mkv", ".webm"):
mp4_cand = p.with_suffix(".mp4")
if mp4_cand.exists() and mp4_cand.stat().st_size > 0:
return mp4_cand
from artemis.utils.video import get_ffmpeg_path
ffmpeg = get_ffmpeg_path()
# 1. Attempt fast copy remux
subprocess.run(
[
ffmpeg,
"-y",
"-fflags",
"+genpts",
"-i",
str(p),
"-c",
"copy",
"-movflags",
"+faststart",
str(mp4_cand),
],
capture_output=True,
timeout=15,
)
if mp4_cand.exists() and mp4_cand.stat().st_size > 0:
return mp4_cand
# 2. If copy remux failed, fallback to ultrafast re-encode
subprocess.run(
[
ffmpeg,
"-y",
"-fflags",
"+genpts+discardcorrupt",
"-i",
str(p),
"-c:v",
"libx264",
"-preset",
"ultrafast",
"-pix_fmt",
"yuv420p",
"-movflags",
"+faststart",
str(mp4_cand),
],
capture_output=True,
timeout=30,
)
if mp4_cand.exists() and mp4_cand.stat().st_size > 0:
return mp4_cand
except Exception:
pass
return p
@staticmethod @staticmethod
def path_to_video_url(p: Path) -> str: def path_to_video_url(p: Path) -> str:
resolved = p.resolve() resolved = p.resolve()
@@ -56,12 +123,14 @@ class MediaService:
for ext in [".mp4", ".mkv", ".webm"]: for ext in [".mp4", ".mkv", ".webm"]:
vfile = item / f"recording{ext}" vfile = item / f"recording{ext}"
if vfile.exists(): if vfile.exists():
vfile = cls.ensure_browser_playable_video(vfile)
url = cls.path_to_video_url(vfile) url = cls.path_to_video_url(vfile)
idx[item.name] = url idx[item.name] = url
prefix = item.name.split("_PASS_")[0].split("_FAIL_")[0] prefix = item.name.split("_PASS_")[0].split("_FAIL_")[0]
idx[prefix] = url idx[prefix] = url
break break
elif item.suffix in [".mp4", ".mkv", ".webm"]: elif item.suffix in [".mp4", ".mkv", ".webm"]:
item = cls.ensure_browser_playable_video(item)
url = cls.path_to_video_url(item) url = cls.path_to_video_url(item)
idx[item.stem] = url idx[item.stem] = url
idx[item.name] = url idx[item.name] = url
@@ -85,12 +154,14 @@ class MediaService:
v_url = None v_url = None
v_fp = row_dict.get("video_filepath") v_fp = row_dict.get("video_filepath")
if v_fp and os.path.exists(v_fp): if v_fp and os.path.exists(v_fp):
v_url = cls.path_to_video_url(Path(v_fp)) p = cls.ensure_browser_playable_video(Path(v_fp))
v_url = cls.path_to_video_url(p)
orig_rec = video_rec_map.get(s_id) orig_rec = video_rec_map.get(s_id)
if not v_url and orig_rec: if not v_url and orig_rec:
p = Path(orig_rec) p = Path(orig_rec)
if p.exists(): if p.exists():
p = cls.ensure_browser_playable_video(p)
v_url = cls.path_to_video_url(p) v_url = cls.path_to_video_url(p)
else: else:
folder_name = p.parent.name folder_name = p.parent.name
@@ -105,12 +176,14 @@ class MediaService:
for ext in [".mp4", ".webm", ".mkv"]: for ext in [".mp4", ".webm", ".mkv"]:
vfile = sess_dir / f"recording{ext}" vfile = sess_dir / f"recording{ext}"
if vfile.exists(): if vfile.exists():
vfile = cls.ensure_browser_playable_video(vfile)
v_url = cls.path_to_video_url(vfile) v_url = cls.path_to_video_url(vfile)
break break
if not v_url: if not v_url:
try: try:
for item in sess_dir.iterdir(): for item in sess_dir.iterdir():
if item.is_file() and item.suffix.lower() in [".mp4", ".webm", ".mkv"]: if item.is_file() and item.suffix.lower() in [".mp4", ".webm", ".mkv"]:
item = cls.ensure_browser_playable_video(item)
v_url = cls.path_to_video_url(item) v_url = cls.path_to_video_url(item)
break break
except Exception: except Exception:
@@ -32,6 +32,7 @@ try:
) )
from admin_console.core.state import state from admin_console.core.state import state
from admin_console.database.repositories.session_repository import session_repo from admin_console.database.repositories.session_repository import session_repo
from admin_console.services.media_service import media_service
from artemis.runtime import DeviceExecutionLock, process_supervisor from artemis.runtime import DeviceExecutionLock, process_supervisor
except ImportError: except ImportError:
from apps.admin_console.core.config import ( from apps.admin_console.core.config import (
@@ -42,6 +43,7 @@ except ImportError:
) )
from apps.admin_console.core.state import state from apps.admin_console.core.state import state
from apps.admin_console.database.repositories.session_repository import session_repo from apps.admin_console.database.repositories.session_repository import session_repo
from apps.admin_console.services.media_service import media_service
from artemis.runtime import DeviceExecutionLock, process_supervisor from artemis.runtime import DeviceExecutionLock, process_supervisor
@@ -199,6 +201,28 @@ class TaskQueueService:
except ProcessLookupError: except ProcessLookupError:
pass pass
@staticmethod
async def _wait_for_worker_process(proc: asyncio.subprocess.Process) -> int:
"""Wait for worker process to exit, with watchdog fallback if PID was reaped externally."""
while True:
try:
return await asyncio.wait_for(proc.wait(), timeout=1.0)
except asyncio.TimeoutError:
pid = getattr(proc, "pid", None)
if pid:
try:
import psutil
p = psutil.Process(pid)
if not p.is_running() or p.status() == psutil.STATUS_ZOMBIE:
return proc.returncode if proc.returncode is not None else -15
except (psutil.NoSuchProcess, ProcessLookupError):
return proc.returncode if proc.returncode is not None else -15
except Exception:
pass
else:
return proc.returncode if proc.returncode is not None else -15
@classmethod @classmethod
def ensure_worker_running(cls): def ensure_worker_running(cls):
"""Guarantees that the background queue worker task is active and running.""" """Guarantees that the background queue worker task is active and running."""
@@ -341,7 +365,7 @@ class TaskQueueService:
await cls._terminate_worker_process(proc) await cls._terminate_worker_process(proc)
# 3. Await subprocess completion # 3. Await subprocess completion
returncode = await proc.wait() returncode = await cls._wait_for_worker_process(proc)
print(f"[QueueWorker] Task [{sess_id}] exited with returncode {returncode}") print(f"[QueueWorker] Task [{sess_id}] exited with returncode {returncode}")
# 4. Perform fallback database status update and notification # 4. Perform fallback database status update and notification
@@ -360,12 +384,36 @@ class TaskQueueService:
f"[QueueWorker] Preserved authoritative session {sess_id} " f"[QueueWorker] Preserved authoritative session {sess_id} "
f"status '{new_status}'" f"status '{new_status}'"
) )
recording_error = "Task worker exited before recording finalization completed" rec_info = session_repo.get_video_recording_for_session(sess_id)
if session_repo.mark_recording_failed_if_pending(sess_id, recording_error): rec_status = (rec_info or {}).get("status")
cls._broadcast_event( recovered_video_path = None
"recording_failed", if rec_status != "ready":
{"session_id": sess_id, "error": recording_error}, try:
) video_rec_map = session_repo.get_video_recordings_map()
video_idx = await asyncio.to_thread(media_service.build_video_index)
recovered_url = await asyncio.to_thread(
media_service.resolve_video_url,
{"session_id": sess_id},
video_rec_map,
video_idx,
)
if recovered_url:
session_repo.mark_recording_ready(sess_id, recovered_url)
cls._broadcast_event(
"recording_ready",
{"session_id": sess_id, "video_url": recovered_url},
)
recovered_video_path = recovered_url
except Exception as rec_err:
print(f"[QueueWorker] Error attempting recording recovery: {rec_err}")
if not recovered_video_path and rec_status != "ready":
recording_error = "Task worker exited before recording finalization completed"
if session_repo.mark_recording_failed_if_pending(sess_id, recording_error):
cls._broadcast_event(
"recording_failed",
{"session_id": sess_id, "error": recording_error},
)
state.active_connections.pop(sess_id, None) state.active_connections.pop(sess_id, None)
cls._broadcast_event( cls._broadcast_event(
"session_ended", "session_ended",
@@ -263,28 +263,41 @@
[class.recording]="agentService.isCurrentSessionRunning()" [class.recording]="agentService.isCurrentSessionRunning()"
[class.processing]="!agentService.isCurrentSessionRunning() && agentService.currentSessionRecordingStatus() === 'processing'" [class.processing]="!agentService.isCurrentSessionRunning() && agentService.currentSessionRecordingStatus() === 'processing'"
[class.has-video]="!agentService.isCurrentSessionRunning() && !!agentService.currentSessionVideoUrl()" [class.has-video]="!agentService.isCurrentSessionRunning() && !!agentService.currentSessionVideoUrl()"
[class.has-steps]="!agentService.isCurrentSessionRunning() && !agentService.currentSessionVideoUrl() && agentService.hasCurrentSessionStepFrames()"
[class.failed]="!agentService.isCurrentSessionRunning() && !agentService.currentSessionVideoUrl() && !agentService.hasCurrentSessionStepFrames() && agentService.currentSessionRecordingStatus() === 'failed'"
(click)="agentService.toggleVideoPlayer()" (click)="agentService.toggleVideoPlayer()"
[title]="agentService.isCurrentSessionRunning() ? 'Task is currently running (Recording screen)' : (agentService.currentSessionRecordingStatus() === 'processing' ? 'Preparing screen recording' : (agentService.currentSessionVideoUrl() ? 'Play Screen Recording' : 'Check Device Screen Recording'))" [title]="getScreenRecordingButtonTitle()"
> >
<span class="material-symbols-outlined video-icon" *ngIf="agentService.isCurrentSessionRunning()"> <span class="material-symbols-outlined video-icon" *ngIf="agentService.isCurrentSessionRunning()">
videocam videocam
</span> </span>
<span class="material-symbols-outlined video-icon" *ngIf="!agentService.isCurrentSessionRunning() && agentService.currentSessionRecordingStatus() === 'processing'"> <span class="material-symbols-outlined video-icon spin-animation" *ngIf="!agentService.isCurrentSessionRunning() && agentService.currentSessionRecordingStatus() === 'processing'">
progress_activity progress_activity
</span> </span>
<span class="material-symbols-outlined video-icon play-icon" *ngIf="!agentService.isCurrentSessionRunning() && agentService.currentSessionRecordingStatus() !== 'processing' && agentService.currentSessionVideoUrl()"> <span class="material-symbols-outlined video-icon play-icon" *ngIf="!agentService.isCurrentSessionRunning() && agentService.currentSessionRecordingStatus() !== 'processing' && agentService.currentSessionVideoUrl()">
smart_display smart_display
</span> </span>
<span class="material-symbols-outlined video-icon" *ngIf="!agentService.isCurrentSessionRunning() && agentService.currentSessionRecordingStatus() !== 'processing' && !agentService.currentSessionVideoUrl()"> <span class="material-symbols-outlined video-icon steps-icon" *ngIf="!agentService.isCurrentSessionRunning() && agentService.currentSessionRecordingStatus() !== 'processing' && !agentService.currentSessionVideoUrl() && agentService.hasCurrentSessionStepFrames()">
slideshow
</span>
<span class="material-symbols-outlined video-icon error-icon" *ngIf="!agentService.isCurrentSessionRunning() && agentService.currentSessionRecordingStatus() !== 'processing' && !agentService.currentSessionVideoUrl() && !agentService.hasCurrentSessionStepFrames() && agentService.currentSessionRecordingStatus() === 'failed'">
videocam_off
</span>
<span class="material-symbols-outlined video-icon" *ngIf="!agentService.isCurrentSessionRunning() && agentService.currentSessionRecordingStatus() !== 'processing' && !agentService.currentSessionVideoUrl() && !agentService.hasCurrentSessionStepFrames() && agentService.currentSessionRecordingStatus() !== 'failed'">
videocam videocam
</span> </span>
<span class="btn-text" *ngIf="agentService.isCurrentSessionRunning()">Recording...</span> <span class="btn-text" *ngIf="agentService.isCurrentSessionRunning()">Recording...</span>
<span class="btn-text" *ngIf="!agentService.isCurrentSessionRunning() && agentService.currentSessionRecordingStatus() === 'processing'">Preparing...</span> <span class="btn-text" *ngIf="!agentService.isCurrentSessionRunning() && agentService.currentSessionRecordingStatus() === 'processing'">Preparing...</span>
<span class="btn-text" *ngIf="!agentService.isCurrentSessionRunning() && agentService.currentSessionRecordingStatus() !== 'processing' && agentService.currentSessionVideoUrl()">Play Recording</span> <span class="btn-text" *ngIf="!agentService.isCurrentSessionRunning() && agentService.currentSessionRecordingStatus() !== 'processing' && agentService.currentSessionVideoUrl()">Play Recording</span>
<span class="btn-text" *ngIf="!agentService.isCurrentSessionRunning() && agentService.currentSessionRecordingStatus() !== 'processing' && !agentService.currentSessionVideoUrl()">Screen Recording</span> <span class="btn-text" *ngIf="!agentService.isCurrentSessionRunning() && agentService.currentSessionRecordingStatus() !== 'processing' && !agentService.currentSessionVideoUrl() && agentService.hasCurrentSessionStepFrames()">Step Replay</span>
<span class="btn-text" *ngIf="!agentService.isCurrentSessionRunning() && agentService.currentSessionRecordingStatus() !== 'processing' && !agentService.currentSessionVideoUrl() && !agentService.hasCurrentSessionStepFrames() && agentService.currentSessionRecordingStatus() === 'failed'">Recording Failed</span>
<span class="btn-text" *ngIf="!agentService.isCurrentSessionRunning() && agentService.currentSessionRecordingStatus() !== 'processing' && !agentService.currentSessionVideoUrl() && !agentService.hasCurrentSessionStepFrames() && agentService.currentSessionRecordingStatus() !== 'failed'">Screen Recording</span>
<span class="video-pulse-dot red" *ngIf="agentService.isCurrentSessionRunning()"></span> <span class="video-pulse-dot red" *ngIf="agentService.isCurrentSessionRunning()"></span>
<span class="steps-count-badge" *ngIf="!agentService.isCurrentSessionRunning() && !agentService.currentSessionVideoUrl() && agentService.hasCurrentSessionStepFrames()">
{{ agentService.currentSessionStepFrames().length }}
</span>
</button> </button>
</div> </div>
@@ -936,6 +936,56 @@ $stream-row-height: 25px;
box-shadow: 0 1px 3px rgba(15, 23, 42, 0.08); box-shadow: 0 1px 3px rgba(15, 23, 42, 0.08);
} }
&.has-steps {
background: rgba(240, 253, 250, 0.9);
border-color: rgba(94, 234, 212, 0.7);
box-shadow: 0 2px 6px -1px rgba(13, 148, 136, 0.12), inset 0 1px 1px rgba(255, 255, 255, 0.9);
.steps-icon {
color: #0d9488;
}
.btn-text {
color: #0f766e;
}
&:hover:not(.active) {
background: rgba(204, 251, 241, 0.95);
border-color: rgba(45, 212, 191, 0.85);
.steps-icon {
color: #0f766e;
}
}
}
&.failed {
background: rgba(254, 242, 242, 0.8);
border-color: rgba(254, 202, 202, 0.9);
.error-icon {
color: #ef4444;
}
.btn-text {
color: #dc2626;
}
}
.steps-count-badge {
display: inline-flex;
align-items: center;
justify-content: center;
padding: 1px 5px;
border-radius: 10px;
background: #0d9488;
color: #ffffff;
font-size: 10px;
font-weight: 700;
line-height: 1;
margin-left: 2px;
}
&.active { &.active {
background: rgba(26, 115, 232, 0.95); background: rgba(26, 115, 232, 0.95);
border-color: rgba(26, 115, 232, 0.95); border-color: rgba(26, 115, 232, 0.95);
@@ -949,6 +999,11 @@ $stream-row-height: 25px;
background: #ffffff; background: #ffffff;
box-shadow: 0 0 6px rgba(255, 255, 255, 0.9); box-shadow: 0 0 6px rgba(255, 255, 255, 0.9);
} }
.steps-count-badge {
background: #ffffff;
color: #1a73e8;
}
} }
} }
} }
@@ -936,6 +936,27 @@ export class AgentStreamComponent implements AfterViewInit {
} }
} }
public getScreenRecordingButtonTitle(): string {
if (this.agentService.isCurrentSessionRunning()) {
return 'Task is currently running (Recording screen)';
}
if (this.agentService.currentSessionRecordingStatus() === 'processing') {
return 'Preparing screen recording...';
}
if (this.agentService.currentSessionVideoUrl()) {
return 'Play Screen Recording Video';
}
if (this.agentService.hasCurrentSessionStepFrames()) {
return this.agentService.currentSessionRecordingStatus() === 'failed'
? 'Video recording failed — Click to replay step-by-step screenshots'
: 'Play step-by-step screenshots replay';
}
if (this.agentService.currentSessionRecordingStatus() === 'failed') {
return 'Screen recording generation failed';
}
return 'Screen Recording';
}
public isDeviceActionTool(tool: any): boolean { public isDeviceActionTool(tool: any): boolean {
return isDeviceActionTool(tool); return isDeviceActionTool(tool);
} }
@@ -33,16 +33,38 @@
<header class="window-header" (mousedown)="startDrag($event)"> <header class="window-header" (mousedown)="startDrag($event)">
<div class="header-left"> <div class="header-left">
<span class="material-symbols-outlined header-icon"> <span class="material-symbols-outlined header-icon">
{{ agentService.recordingPlaybackStatus() === 'live' ? 'sensors' : (agentService.recordingPlaybackStatus() === 'processing' ? 'progress_activity' : 'smart_display') }} {{ agentService.recordingPlaybackStatus() === 'live' ? 'sensors' : (agentService.playerMode() === 'steps' ? 'slideshow' : (agentService.recordingPlaybackStatus() === 'processing' ? 'progress_activity' : 'smart_display')) }}
</span> </span>
<span class="header-title" [title]="agentService.activeVideoTitle()"> <span class="header-title" [title]="agentService.activeVideoTitle()">
{{ agentService.recordingPlaybackStatus() === 'live' ? 'Live Device Screen' : (agentService.recordingPlaybackStatus() === 'processing' ? 'Preparing Recording' : (agentService.activeVideoTitle() || 'Screen Recording')) }} {{ agentService.recordingPlaybackStatus() === 'live' ? 'Live Device Screen' : (agentService.playerMode() === 'steps' ? ('Step Replay (' + (totalStepFrames() > 0 ? (activeStepIndex() + 1) + '/' + totalStepFrames() : '0') + ')') : (agentService.recordingPlaybackStatus() === 'processing' ? 'Preparing Recording' : (agentService.activeVideoTitle() || 'Screen Recording'))) }}
</span> </span>
</div> </div>
<div class="header-actions" (mousedown)="$event.stopPropagation()"> <div class="header-actions" (mousedown)="$event.stopPropagation()">
<!-- Playback Speed Pills (Shown only when playback video is active) --> <!-- Mode Toggle Group: Video vs. Step Replay -->
<div class="speed-pill-group" *ngIf="agentService.recordingPlaybackStatus() === 'ready' && !agentService.isVideoMinimized() && !videoLoadError() && agentService.activeVideoUrl()"> <div class="mode-toggle-group" *ngIf="agentService.recordingPlaybackStatus() === 'ready' && agentService.hasCurrentSessionStepFrames() && !agentService.isVideoMinimized()">
<button
class="mode-btn"
[class.active]="agentService.playerMode() === 'video'"
(click)="agentService.setPlayerMode('video')"
title="Video Recording"
>
<span class="material-symbols-outlined mode-icon">videocam</span>
<span>Video</span>
</button>
<button
class="mode-btn"
[class.active]="agentService.playerMode() === 'steps'"
(click)="agentService.setPlayerMode('steps')"
title="Step Screenshot Replay"
>
<span class="material-symbols-outlined mode-icon">slideshow</span>
<span>Steps</span>
</button>
</div>
<!-- Playback Speed Pills (Shown when video is active OR when step replay is active) -->
<div class="speed-pill-group" *ngIf="!agentService.isVideoMinimized() && (agentService.recordingPlaybackStatus() === 'ready' || agentService.playerMode() === 'steps') && (agentService.activeVideoUrl() || totalStepFrames() > 0)">
<button <button
*ngFor="let speed of speedOptions" *ngFor="let speed of speedOptions"
class="speed-btn" class="speed-btn"
@@ -121,16 +143,107 @@
</div> </div>
</ng-container> </ng-container>
<!-- 2. RECORDING PLAYBACK (AUTOMATICALLY ACTIVE WHEN TEST FINISHED) --> <!-- 2. RECORDING PLAYBACK / STEP REPLAY (AUTOMATICALLY ACTIVE WHEN TEST FINISHED) -->
<ng-container *ngIf="agentService.recordingPlaybackStatus() !== 'live'"> <ng-container *ngIf="agentService.recordingPlaybackStatus() !== 'live'">
<!-- Loading State --> <!-- Loading State (when in video mode and processing) -->
<div class="video-loading-state" *ngIf="agentService.recordingPlaybackStatus() === 'processing' || agentService.isVideoLoading()"> <div class="video-loading-state" *ngIf="agentService.playerMode() === 'video' && (agentService.recordingPlaybackStatus() === 'processing' || agentService.isVideoLoading())">
<span class="material-symbols-outlined spinning-icon">progress_activity</span> <span class="material-symbols-outlined spinning-icon">progress_activity</span>
<span>{{ agentService.recordingPlaybackMessage() || 'Preparing screen recording...' }}</span> <span>{{ agentService.recordingPlaybackMessage() || 'Preparing screen recording...' }}</span>
</div> </div>
<!-- Active Video Playback View --> <!-- Option A: STEP REPLAY VIEW (ACTIVE IN STEP MODE) -->
<ng-container *ngIf="agentService.recordingPlaybackStatus() === 'ready' && !agentService.isVideoLoading() && agentService.activeVideoUrl() as videoUrl"> <ng-container *ngIf="agentService.playerMode() === 'steps' && totalStepFrames() > 0">
<div class="step-replay-container">
<!-- Fallback info banner if recording had failed -->
<div class="step-replay-notice" *ngIf="agentService.recordingPlaybackStatus() === 'failed' || videoLoadError()">
<span class="material-symbols-outlined notice-icon">info</span>
<span>Screen recording unavailable. Replaying step screenshots.</span>
</div>
<div class="step-image-viewport">
<img
class="step-frame-img"
[src]="currentStepFrame()?.imageUrl"
[alt]="currentStepFrame()?.title"
(click)="openImageInNewTab()"
title="Click to view image in new tab"
/>
<!-- Step Overlay Badge -->
<div class="step-frame-badge" *ngIf="currentStepFrame() as frame">
<span class="badge-step-no">Step {{ frame.stepNumber }}</span>
<span class="badge-action-desc">{{ frame.actionText || frame.title }}</span>
</div>
</div>
<!-- Step Replay Controls Footer -->
<div class="video-controls-footer">
<!-- Timeline Scrubber Bar -->
<div class="scrubber-bar-container">
<input
type="range"
class="scrubber-range"
[min]="0"
[max]="totalStepFrames() - 1"
[step]="1"
[value]="activeStepIndex()"
(input)="onStepScrub($event)"
/>
<div class="scrubber-progress-fill" [style.width.%]="stepProgressPercent()"></div>
</div>
<!-- Bottom Controls Row -->
<div class="controls-row">
<div class="controls-left">
<!-- Play / Pause Button -->
<button class="ctrl-btn play-btn" (click)="toggleStepPlay()" [title]="isStepPlaying() ? 'Pause' : 'Play Slideshow'">
<span class="material-symbols-outlined">{{ isStepPlaying() ? 'pause' : 'play_arrow' }}</span>
</button>
<!-- Previous Frame Button -->
<button class="ctrl-btn" (click)="prevStepFrame()" [disabled]="activeStepIndex() <= 0" title="Previous Frame">
<span class="material-symbols-outlined">skip_previous</span>
</button>
<!-- Next Frame Button -->
<button class="ctrl-btn" (click)="nextStepFrame()" [disabled]="activeStepIndex() >= totalStepFrames() - 1" title="Next Frame">
<span class="material-symbols-outlined">skip_next</span>
</button>
<!-- Step Replay / Reset -->
<button class="ctrl-btn" (click)="restartStep()" title="Replay from start">
<span class="material-symbols-outlined">replay</span>
</button>
<!-- Frame Counter Display -->
<span class="time-display">
Frame {{ activeStepIndex() + 1 }} / {{ totalStepFrames() }}
</span>
</div>
<div class="controls-right">
<!-- Loop Toggle -->
<button
class="ctrl-btn"
[class.active]="isLooping()"
(click)="toggleLoop()"
[title]="isLooping() ? 'Loop enabled' : 'Enable loop'"
>
<span class="material-symbols-outlined">repeat</span>
</button>
<!-- Open in new tab -->
<button class="ctrl-btn" (click)="openImageInNewTab()" title="Open image in new tab">
<span class="material-symbols-outlined">open_in_new</span>
</button>
</div>
</div>
</div>
</div>
</ng-container>
<!-- Option B: ACTIVE VIDEO PLAYBACK VIEW -->
<ng-container *ngIf="agentService.playerMode() === 'video' && agentService.recordingPlaybackStatus() === 'ready' && !agentService.isVideoLoading() && agentService.activeVideoUrl() as videoUrl">
<div class="video-container" *ngIf="!videoLoadError()"> <div class="video-container" *ngIf="!videoLoadError()">
<video <video
#videoRef #videoRef
@@ -146,7 +259,7 @@
></video> ></video>
</div> </div>
<!-- Dedicated Bottom Controls Bar (Outside Video: ZERO Obscurity / Zero Overlap) --> <!-- Dedicated Bottom Controls Bar -->
<div class="video-controls-footer" *ngIf="!videoLoadError()"> <div class="video-controls-footer" *ngIf="!videoLoadError()">
<!-- Timeline Scrubber Bar --> <!-- Timeline Scrubber Bar -->
<div class="scrubber-bar-container"> <div class="scrubber-bar-container">
@@ -206,8 +319,8 @@
</div> </div>
</ng-container> </ng-container>
<!-- Clean, High-Contrast Minimal Empty / Fallback State --> <!-- Option C: EMPTY / FALLBACK STATE (Only when no video AND no step frames) -->
<div class="empty-video-fallback" *ngIf="!agentService.isVideoLoading() && (agentService.recordingPlaybackStatus() === 'failed' || agentService.recordingPlaybackStatus() === 'unavailable' || videoLoadError())"> <div class="empty-video-fallback" *ngIf="!agentService.isVideoLoading() && (agentService.playerMode() === 'video' || totalStepFrames() === 0) && (agentService.recordingPlaybackStatus() === 'failed' || agentService.recordingPlaybackStatus() === 'unavailable' || videoLoadError()) && totalStepFrames() === 0">
<div class="fallback-icon-wrap"> <div class="fallback-icon-wrap">
<span class="material-symbols-outlined fallback-icon">videocam_off</span> <span class="material-symbols-outlined fallback-icon">videocam_off</span>
</div> </div>
@@ -176,6 +176,48 @@
gap: 4px; gap: 4px;
flex-shrink: 0; flex-shrink: 0;
.mode-toggle-group {
display: flex;
align-items: center;
background: rgba(241, 245, 249, 0.7);
backdrop-filter: blur(12px);
border: 1px solid rgba(226, 232, 240, 0.85);
border-radius: 8px;
padding: 2px;
margin-right: 4px;
gap: 2px;
.mode-btn {
display: inline-flex;
align-items: center;
gap: 3px;
border: none;
background: transparent;
color: #475569;
font-size: 11px;
font-weight: 600;
padding: 2px 7px;
border-radius: 6px;
cursor: pointer;
transition: all 0.15s ease;
.mode-icon {
font-size: 13px;
}
&:hover:not(.active) {
color: #0f172a;
background: rgba(255, 255, 255, 0.8);
}
&.active {
background: #0284c7;
color: #ffffff;
box-shadow: 0 1px 3px rgba(2, 132, 199, 0.3);
}
}
}
.speed-pill-group { .speed-pill-group {
display: flex; display: flex;
align-items: center; align-items: center;
@@ -185,8 +227,6 @@
border-radius: 8px; border-radius: 8px;
padding: 2px; padding: 2px;
margin-right: 4px; margin-right: 4px;
padding: 2px;
margin-right: 4px;
.speed-btn { .speed-btn {
border: none; border: none;
@@ -370,6 +410,98 @@
} }
} }
.step-replay-container {
display: flex;
flex-direction: column;
background: #000000;
position: relative;
width: 100%;
overflow: hidden;
.step-replay-notice {
display: flex;
align-items: center;
gap: 6px;
background: rgba(254, 243, 199, 0.95);
color: #92400e;
padding: 6px 12px;
font-size: 11.5px;
font-weight: 500;
border-bottom: 1px solid rgba(245, 158, 11, 0.3);
.notice-icon {
font-size: 14px;
color: #d97706;
flex-shrink: 0;
}
}
.step-image-viewport {
display: flex;
align-items: center;
justify-content: center;
position: relative;
background: #0a0a0c;
min-height: 240px;
max-height: 520px;
cursor: pointer;
overflow: hidden;
.step-frame-img {
max-width: 100%;
max-height: 520px;
width: auto;
height: auto;
object-fit: contain;
display: block;
user-select: none;
transition: opacity 0.15s ease;
&:hover {
opacity: 0.96;
}
}
.step-frame-badge {
position: absolute;
bottom: 12px;
left: 12px;
right: 12px;
display: flex;
align-items: center;
gap: 6px;
background: rgba(15, 23, 42, 0.84);
backdrop-filter: blur(10px);
border: 1px solid rgba(255, 255, 255, 0.18);
border-radius: 8px;
padding: 6px 10px;
color: #ffffff;
box-shadow: 0 4px 12px rgba(0, 0, 0, 0.45);
pointer-events: none;
animation: fadeIn 0.2s ease;
.badge-step-no {
font-size: 11px;
font-weight: 700;
background: #0284c7;
padding: 1px 6px;
border-radius: 4px;
letter-spacing: 0.2px;
flex-shrink: 0;
}
.badge-action-desc {
font-size: 11.5px;
font-weight: 500;
color: #f8fafc;
white-space: nowrap;
overflow: hidden;
text-overflow: ellipsis;
}
}
}
}
.video-container { .video-container {
position: relative; position: relative;
width: 100%; width: 100%;
@@ -22,11 +22,13 @@ import {
ElementRef, ElementRef,
ViewChild, ViewChild,
HostListener, HostListener,
effect effect,
OnDestroy
} from '@angular/core'; } from '@angular/core';
import { CommonModule } from '@angular/common'; import { CommonModule } from '@angular/common';
import { FormsModule } from '@angular/forms'; import { FormsModule } from '@angular/forms';
import { AgentService } from '../../services/agent.service'; import { AgentService } from '../../services/agent.service';
import { StepReplayFrame } from '../../core/models/stream.model';
@Component({ @Component({
selector: 'app-floating-video-player', selector: 'app-floating-video-player',
@@ -35,7 +37,7 @@ import { AgentService } from '../../services/agent.service';
templateUrl: './floating-video-player.component.html', templateUrl: './floating-video-player.component.html',
styleUrl: './floating-video-player.component.scss' styleUrl: './floating-video-player.component.scss'
}) })
export class FloatingVideoPlayerComponent { export class FloatingVideoPlayerComponent implements OnDestroy {
public agentService = inject(AgentService); public agentService = inject(AgentService);
@ViewChild('videoRef') videoRef?: ElementRef<HTMLVideoElement>; @ViewChild('videoRef') videoRef?: ElementRef<HTMLVideoElement>;
@@ -60,6 +62,24 @@ export class FloatingVideoPlayerComponent {
private pendingAutoplay = false; private pendingAutoplay = false;
private pendingAbsoluteSeek: number | null = null; private pendingAbsoluteSeek: number | null = null;
// Step Replay state signals
public activeStepIndex = signal<number>(0);
public isStepPlaying = signal<boolean>(false);
private stepTimer: any = null;
public stepFrames = computed(() => this.agentService.currentSessionStepFrames());
public totalStepFrames = computed(() => this.stepFrames().length);
public currentStepFrame = computed<StepReplayFrame | null>(() => {
const frames = this.stepFrames();
const idx = this.activeStepIndex();
return frames[idx] || null;
});
public stepProgressPercent = computed(() => {
const total = this.totalStepFrames();
if (total <= 1) return 0;
return (this.activeStepIndex() / (total - 1)) * 100;
});
// Available speed options // Available speed options
public speedOptions = [0.5, 1.0, 1.5, 2.0, 4.0]; public speedOptions = [0.5, 1.0, 1.5, 2.0, 4.0];
@@ -92,6 +112,16 @@ export class FloatingVideoPlayerComponent {
{ allowSignalWrites: true } { allowSignalWrites: true }
); );
// Reset step playback state when session changes
effect(
() => {
const _ = this.stepFrames();
this.activeStepIndex.set(0);
this.pauseStepPlay();
},
{ allowSignalWrites: true }
);
effect(() => { effect(() => {
const request = this.agentService.videoSeekRequest(); const request = this.agentService.videoSeekRequest();
if (!request) return; if (!request) return;
@@ -327,6 +357,9 @@ export class FloatingVideoPlayerComponent {
if (v) { if (v) {
v.playbackRate = rate; v.playbackRate = rate;
} }
if (this.isStepPlaying()) {
this.scheduleNextStepFrame();
}
} }
public toggleLoop(): void { public toggleLoop(): void {
@@ -361,4 +394,89 @@ export class FloatingVideoPlayerComponent {
window.open(url, '_blank'); window.open(url, '_blank');
} }
} }
/**
* Step Replay Controls
*/
public toggleStepPlay(): void {
if (this.isStepPlaying()) {
this.pauseStepPlay();
} else {
this.startStepPlay();
}
}
public startStepPlay(): void {
if (this.totalStepFrames() <= 0) return;
if (this.activeStepIndex() >= this.totalStepFrames() - 1) {
this.activeStepIndex.set(0);
}
this.isStepPlaying.set(true);
this.scheduleNextStepFrame();
}
public pauseStepPlay(): void {
this.isStepPlaying.set(false);
if (this.stepTimer) {
clearTimeout(this.stepTimer);
this.stepTimer = null;
}
}
private scheduleNextStepFrame(): void {
if (this.stepTimer) clearTimeout(this.stepTimer);
const delay = Math.max(300, 1500 / this.playbackRate());
this.stepTimer = setTimeout(() => {
if (!this.isStepPlaying()) return;
if (this.activeStepIndex() < this.totalStepFrames() - 1) {
this.activeStepIndex.update((i) => i + 1);
this.scheduleNextStepFrame();
} else {
if (this.isLooping()) {
this.activeStepIndex.set(0);
this.scheduleNextStepFrame();
} else {
this.pauseStepPlay();
}
}
}, delay);
}
public prevStepFrame(): void {
this.pauseStepPlay();
this.activeStepIndex.update((i) => Math.max(0, i - 1));
}
public nextStepFrame(): void {
this.pauseStepPlay();
this.activeStepIndex.update((i) => Math.min(this.totalStepFrames() - 1, i + 1));
}
public seekStepFrame(index: number): void {
this.activeStepIndex.set(Math.max(0, Math.min(this.totalStepFrames() - 1, index)));
}
public onStepScrub(event: Event): void {
const input = event.target as HTMLInputElement;
this.seekStepFrame(parseInt(input.value, 10));
}
public restartStep(): void {
this.seekStepFrame(0);
this.startStepPlay();
}
public openImageInNewTab(): void {
const url = this.currentStepFrame()?.imageUrl;
if (url) {
window.open(url, '_blank');
}
}
public ngOnDestroy(): void {
if (this.stepTimer) {
clearTimeout(this.stepTimer);
this.stepTimer = null;
}
}
} }
@@ -98,3 +98,14 @@ export interface CheckerResult {
reason: string; reason: string;
} }
export interface StepReplayFrame {
index: number;
stepNumber: number;
title: string;
imageUrl: string;
actionText?: string;
isPost?: boolean;
timestamp?: number;
phaseId?: string;
}
@@ -1,4 +1,4 @@
import { signal } from '@angular/core'; import { signal, computed } from '@angular/core';
import { of } from 'rxjs'; import { of } from 'rxjs';
import { AgentService } from './agent.service'; import { AgentService } from './agent.service';
@@ -198,6 +198,9 @@ describe('AgentService recording finalization lifecycle', () => {
service.recordingPlaybackStatus = signal('idle'); service.recordingPlaybackStatus = signal('idle');
service.recordingPlaybackMessage = signal(''); service.recordingPlaybackMessage = signal('');
service.shouldAutoplayVideo = signal(true); service.shouldAutoplayVideo = signal(true);
service.playerMode = signal<'video' | 'steps'>('video');
(service as any).hasCurrentSessionStepFrames = computed(() => false);
(service as any).currentSessionStepFrames = computed(() => []);
return service; return service;
} }
@@ -19,8 +19,9 @@ import { HttpClient } from '@angular/common/http';
import { Observable } from 'rxjs'; import { Observable } from 'rxjs';
import { Session, ModelInfo, TaskQueueItem, AgentStatusResponse } from '../core/models/session.model'; import { Session, ModelInfo, TaskQueueItem, AgentStatusResponse } from '../core/models/session.model';
import { StepItemData } from '../core/models/stream.model'; import { StepItemData, StepReplayFrame } from '../core/models/stream.model';
export type { Session, ModelInfo, TaskQueueItem, AgentStatusResponse, StepItemData }; import { extractStepReplayFrames } from '../utils/action-formatter.util';
export type { Session, ModelInfo, TaskQueueItem, AgentStatusResponse, StepItemData, StepReplayFrame };
const SESSION_CACHE_KEY = 'artemis.sessions.v1'; const SESSION_CACHE_KEY = 'artemis.sessions.v1';
@@ -158,12 +159,32 @@ export class AgentService {
public recordingPlaybackMessage = signal<string>(''); public recordingPlaybackMessage = signal<string>('');
public shouldAutoplayVideo = signal<boolean>(false); public shouldAutoplayVideo = signal<boolean>(false);
public videoSeekRequest = signal<{ seconds: number; requestId: number } | null>(null); public videoSeekRequest = signal<{ seconds: number; requestId: number } | null>(null);
public playerMode = signal<'video' | 'steps'>('video');
private activeVideoSessionId: string | null = null; private activeVideoSessionId: string | null = null;
private videoSeekRequestId = 0; private videoSeekRequestId = 0;
private videoRequestGeneration = 0; private videoRequestGeneration = 0;
private videoRetryTimer: ReturnType<typeof setTimeout> | null = null; private videoRetryTimer: ReturnType<typeof setTimeout> | null = null;
private videoWaitStartedAt = 0; private videoWaitStartedAt = 0;
/**
* Computed step replay frames from current session logs
*/
public currentSessionStepFrames = computed<StepReplayFrame[]>(() => {
return extractStepReplayFrames(this.sessionLogs());
});
public hasCurrentSessionStepFrames = computed<boolean>(() => {
return this.currentSessionStepFrames().length > 0;
});
public setPlayerMode(mode: 'video' | 'steps'): void {
this.playerMode.set(mode);
}
public togglePlayerMode(): void {
this.playerMode.update((mode) => (mode === 'video' ? 'steps' : 'video'));
}
/** /**
* Clear user-pinned selection so subsequent runs automatically follow active runner * Clear user-pinned selection so subsequent runs automatically follow active runner
*/ */
@@ -1284,6 +1305,13 @@ export class AgentService {
this.isVideoLoading.set(true); this.isVideoLoading.set(true);
this.recordingPlaybackStatus.set('processing'); this.recordingPlaybackStatus.set('processing');
this.recordingPlaybackMessage.set('Loading screen recording...'); this.recordingPlaybackMessage.set('Loading screen recording...');
if (targetUrl) {
this.playerMode?.set('video');
} else if (this.hasCurrentSessionStepFrames?.()) {
this.playerMode?.set('steps');
} else {
this.playerMode?.set('video');
}
this.requestSessionVideo(targetSessionId, this.videoRequestGeneration); this.requestSessionVideo(targetSessionId, this.videoRequestGeneration);
} }
@@ -1341,6 +1369,7 @@ export class AgentService {
this.recordingPlaybackMessage.set(''); this.recordingPlaybackMessage.set('');
this.activeVideoUrl.set(res.video_url); this.activeVideoUrl.set(res.video_url);
this.activeVideoSegments.set(res.video_segments || []); this.activeVideoSegments.set(res.video_segments || []);
this.playerMode?.set('video');
this.rawSessions.update((list) => this.rawSessions.update((list) =>
list.map((s) => s.session_id === sessionId list.map((s) => s.session_id === sessionId
? { ...s, video_url: res.video_url || undefined, recording_status: 'ready' } ? { ...s, video_url: res.video_url || undefined, recording_status: 'ready' }
@@ -1368,6 +1397,9 @@ export class AgentService {
? 'Recording finalization failed.' ? 'Recording finalization failed.'
: 'No screen recording is available for this task.') : 'No screen recording is available for this task.')
); );
if (this.hasCurrentSessionStepFrames?.()) {
this.playerMode?.set('steps');
}
}, },
error: () => { error: () => {
if (generation !== this.videoRequestGeneration || sessionId !== this.activeVideoSessionId) { if (generation !== this.videoRequestGeneration || sessionId !== this.activeVideoSessionId) {
@@ -1380,6 +1412,9 @@ export class AgentService {
this.isVideoLoading.set(false); this.isVideoLoading.set(false);
this.recordingPlaybackStatus.set('failed'); this.recordingPlaybackStatus.set('failed');
this.recordingPlaybackMessage.set('Unable to load the screen recording.'); this.recordingPlaybackMessage.set('Unable to load the screen recording.');
if (this.hasCurrentSessionStepFrames?.()) {
this.playerMode?.set('steps');
}
} }
}); });
} }
@@ -1,6 +1,7 @@
import { import {
getStepPreImageUrl, getStepPreImageUrl,
getStepPostImageUrl getStepPostImageUrl,
extractStepReplayFrames
} from './action-formatter.util'; } from './action-formatter.util';
describe('action-formatter.util screenshot chaining', () => { describe('action-formatter.util screenshot chaining', () => {
@@ -109,4 +110,100 @@ describe('action-formatter.util screenshot chaining', () => {
expect(preUrl).toBe('/images/pre123'); expect(preUrl).toBe('/images/pre123');
expect(postUrl).toBe('/images/post456'); expect(postUrl).toBe('/images/post456');
}); });
it('should resolve post screenshot for primary action when generic_tools contains action traces', () => {
const stepData = {
step_id: 'step-02',
pre_image_name: 'pre_screen_hash',
post_image_name: 'post_screen_hash',
action_taken: {
action: 'click',
coordinates: [500, 928],
args: { target: [500, 928] },
timestamp: 1000
},
generic_tools: [
{
trace_id: 'trace-click-internal',
type: 'action',
name: 'click',
timestamp: 1001,
payload: {
args: { target: '[500, 928]', times: '1', delay_ms: '100' },
result: {
outcome: 'Clicked successfully.',
post_image_name: 'post_screen_hash',
status: 'success'
}
}
}
]
};
const preUrl = getStepPreImageUrl(stepData, stepData.action_taken);
const postUrl = getStepPostImageUrl(stepData, stepData.action_taken);
expect(preUrl).toBe('/images/pre_screen_hash');
expect(postUrl).toBe('/images/post_screen_hash');
});
});
describe('extractStepReplayFrames', () => {
it('should return an empty array for empty or invalid logs', () => {
expect(extractStepReplayFrames([])).toEqual([]);
expect(extractStepReplayFrames(null as any)).toEqual([]);
});
it('should extract ordered StepReplayFrames and deduplicate consecutive identical images', () => {
const logs = [
{
type: 'step_updated',
data: {
step_id: 's-1',
step_number: 1,
pre_image_name: 'img_screen_1',
post_image_name: 'img_screen_2',
action_taken: { action: 'tap', coordinates: [500, 1000] },
timestamp: 100
}
},
{
type: 'step_updated',
data: {
step_id: 's-2',
step_number: 2,
pre_image_name: 'img_screen_2',
post_image_name: 'img_screen_3',
action_taken: { action: 'input_text', text: 'hello' },
timestamp: 200
}
}
];
const frames = extractStepReplayFrames(logs);
expect(frames.length).toBe(3);
expect(frames[0].title).toBe('Step 1 (Before Action)');
expect(frames[0].imageUrl).toBe('/images/img_screen_1');
expect(frames[1].title).toContain('Step 1');
expect(frames[1].imageUrl).toBe('/images/img_screen_2');
expect(frames[2].title).toContain('Step 2');
expect(frames[2].imageUrl).toBe('/images/img_screen_3');
});
it('should handle terminal step with only pre_image (e.g. cancelled / failed task)', () => {
const logs = [
{
step_number: 1,
pre_image_name: 'img_start',
post_image_name: null,
action_taken: null,
timestamp: 100
}
];
const frames = extractStepReplayFrames(logs);
expect(frames.length).toBe(1);
expect(frames[0].imageUrl).toBe('/images/img_start');
expect(frames[0].stepNumber).toBe(1);
});
}); });
@@ -14,7 +14,7 @@
* limitations under the License. * limitations under the License.
*/ */
import { ActionParam } from '../core/models/stream.model'; import { ActionParam, StepReplayFrame } from '../core/models/stream.model';
import { extractNumbersFromCoordinateValue, isPureDirectionString, parseSequenceCoordinates } from './image-overlay.util'; import { extractNumbersFromCoordinateValue, isPureDirectionString, parseSequenceCoordinates } from './image-overlay.util';
import { cleanErrorMessage } from './tool-formatter.util'; import { cleanErrorMessage } from './tool-formatter.util';
@@ -698,7 +698,7 @@ export function resolveStepImageMap(stepData: any): Map<any, ResolvedImageEntry>
if (explicitPost) { if (explicitPost) {
resolvedPost = explicitPost; resolvedPost = explicitPost;
hasExplicitPost = true; hasExplicitPost = true;
} else if (e.isPrimary && !hadPrimaryFailure && !hasSubsequentActions) { } else if (e.isPrimary && !hadPrimaryFailure) {
resolvedPost = stepData.post_image_name || stepData.post_screenshot || null; resolvedPost = stepData.post_image_name || stepData.post_screenshot || null;
} else if (e.item === lastActionItem && !hadPrimaryFailure) { } else if (e.item === lastActionItem && !hadPrimaryFailure) {
resolvedPost = stepData.post_image_name || stepData.post_screenshot || null; resolvedPost = stepData.post_image_name || stepData.post_screenshot || null;
@@ -765,3 +765,86 @@ export function getStepPostImageUrl(stepData: any, actionData?: any): string | n
} }
return postUrl; return postUrl;
} }
/**
* Extract an ordered array of visual StepReplayFrames from session logs or step blocks
*/
export function extractStepReplayFrames(logsOrSteps: any[]): StepReplayFrame[] {
if (!Array.isArray(logsOrSteps) || logsOrSteps.length === 0) {
return [];
}
// Extract raw step data objects
const rawSteps: any[] = [];
for (const item of logsOrSteps) {
if (!item) continue;
if (item.type === 'step_updated' && item.data) {
rawSteps.push(item.data);
} else if (item.type === 'step' && item.data) {
rawSteps.push(item.data);
} else if (item.step_number !== undefined || item.step_id !== undefined) {
rawSteps.push(item);
}
}
// Deduplicate by step_id or step_number, keeping latest
const stepMap = new Map<any, any>();
for (const step of rawSteps) {
const key = step.step_id || step.step_number;
if (key !== undefined) {
stepMap.set(key, step);
}
}
const steps = Array.from(stepMap.values()).sort((a, b) => {
const numA = Number(a.step_number ?? 0);
const numB = Number(b.step_number ?? 0);
if (numA !== numB) return numA - numB;
return Number(a.timestamp ?? 0) - Number(b.timestamp ?? 0);
});
const frames: StepReplayFrame[] = [];
let lastImageUrl: string | null = null;
for (const stepData of steps) {
const stepNum = Number(stepData.step_number ?? (frames.length + 1));
const act = stepData.action_taken;
const title = act ? (getActionTitle(act) || 'Action') : 'Step';
const coords = act ? getActionCoords(act) : '';
const actionDesc = coords ? `${title} (${coords})` : title;
const preUrl = getStepPreImageUrl(stepData, act);
const postUrl = getStepPostImageUrl(stepData, act);
// If pre-image exists and is new, add it
if (preUrl && preUrl !== lastImageUrl) {
frames.push({
index: frames.length,
stepNumber: stepNum,
title: `Step ${stepNum} (Before Action)`,
imageUrl: preUrl,
actionText: `Preparing: ${actionDesc}`,
isPost: false,
timestamp: stepData.timestamp
});
lastImageUrl = preUrl;
}
// If post-image exists and is different from current screen, add it
if (postUrl && postUrl !== lastImageUrl) {
frames.push({
index: frames.length,
stepNumber: stepNum,
title: `Step ${stepNum}: ${actionDesc}`,
imageUrl: postUrl,
actionText: actionDesc,
isPost: true,
timestamp: stepData.timestamp
});
lastImageUrl = postUrl;
}
}
return frames;
}
+1 -1
View File
@@ -1,5 +1,5 @@
# ROLE & OBJECTIVE # ROLE & OBJECTIVE
You are **ARTEMIS Flash Runner**, an autonomous and highly efficient Android Device Execution Agent. Your goal is to accomplish the user's objective on the device. You are an autonomous and highly efficient Android Device Execution Agent. Your goal is to accomplish the user's objective on the device.
**Objective: {{ goal }}** **Objective: {{ goal }}**
+32 -16
View File
@@ -697,7 +697,10 @@ class UnifiedMobileController:
session.is_active = False session.is_active = False
if session.watchdog_task and not session.watchdog_task.done(): if session.watchdog_task and not session.watchdog_task.done():
session.watchdog_task.cancel() session.watchdog_task.cancel()
await session.watchdog_task try:
await session.watchdog_task
except (asyncio.CancelledError, Exception):
pass
try: try:
process = session.process process = session.process
@@ -708,7 +711,14 @@ class UnifiedMobileController:
logger.warning(f"Error terminating scrcpy process: {proc_e}") logger.warning(f"Error terminating scrcpy process: {proc_e}")
output_path = session.local_video_path output_path = session.local_video_path
if not output_path or not output_path.exists(): has_existing_recording = (
(output_path and output_path.exists())
or any(
Path(r.get("output_path", "")).exists() or Path(r.get("path", "")).exists()
for r in session.android_segment_records
)
)
if not has_existing_recording:
message = "Recording file not found on disk" message = "Recording file not found on disk"
self._record_recording_failure(session, message) self._record_recording_failure(session, message)
remove_active_session(device_id) remove_active_session(device_id)
@@ -726,31 +736,37 @@ class UnifiedMobileController:
self._remux_segment_record(record) self._remux_segment_record(record)
) )
) )
conversion_results = []
if session.android_conversion_tasks: if session.android_conversion_tasks:
conversion_results = await asyncio.gather( await asyncio.gather(
*session.android_conversion_tasks, return_exceptions=True *session.android_conversion_tasks, return_exceptions=True
) )
mp4_paths = [ mp4_paths = [
Path(record["output_path"]) Path(record["output_path"])
for record in session.android_segment_records for record in session.android_segment_records
if Path(record["output_path"]).exists() if Path(record["output_path"]).exists() and Path(record["output_path"]).stat().st_size > 0
] ]
manifest_path = await write_recording_manifest(output_path.parent, mp4_paths)
conversion_failed = ( # Emergency fallback: if no segment MP4 was produced, attempt to remux local_video_path directly
len(mp4_paths) != len(session.android_segment_records) if not mp4_paths and output_path and output_path.exists() and output_path.stat().st_size > 0:
or any(result is not True for result in conversion_results) fallback_mp4 = (
or manifest_path is None output_path.parent / "recording.mp4"
) if output_path.name != "recording.mp4"
if conversion_failed: else output_path.with_name("recording_converted.mp4")
)
if await remux_recording_to_mp4(output_path, fallback_mp4):
mp4_paths.append(fallback_mp4)
if not mp4_paths:
message = "Recording finalization failed; no complete browser-safe video was produced" message = "Recording finalization failed; no complete browser-safe video was produced"
self._record_recording_failure(session, message) self._record_recording_failure(session, message)
remove_active_session(device_id) remove_active_session(device_id)
return VideoRecordingResult(success=False, message=message) return VideoRecordingResult(success=False, message=message)
remove_active_session(device_id)
final_video_path = mp4_paths[0] final_video_path = mp4_paths[0]
output_dir = final_video_path.parent
manifest_path = await write_recording_manifest(output_dir, mp4_paths)
remove_active_session(device_id)
# Persist update to local database if Data Engine is active # Persist update to local database if Data Engine is active
if self.ctx and self.ctx.data_engine: if self.ctx and self.ctx.data_engine:
@@ -764,12 +780,12 @@ class UnifiedMobileController:
return VideoRecordingResult( return VideoRecordingResult(
success=True, success=True,
message=f"Recording stopped, saved {len(mp4_paths)} orientation-safe segments", message=f"Recording stopped, saved {len(mp4_paths)} video segments",
video_path=mp4_paths[0], video_path=final_video_path,
video_id=session.video_id, video_id=session.video_id,
generation=session.generation, generation=session.generation,
sealed_until=session.sealed_until, sealed_until=session.sealed_until,
source_revision=f"{session.video_id}:{session.generation}:sealed", source_revision=f"{session.video_id}:{session.generation}:ready",
) )
except Exception as e: except Exception as e:
+15
View File
@@ -29,6 +29,7 @@ from artemis.sdk.types.task import AgentProfile
from artemis.utils.cli_helpers import display_device_status from artemis.utils.cli_helpers import display_device_status
from artemis.utils.logger import get_logger from artemis.utils.logger import get_logger
from artemis.utils.video import check_ffmpeg_available from artemis.utils.video import check_ffmpeg_available
import signal
from rich.console import Console from rich.console import Console
from rich.panel import Panel from rich.panel import Panel
import typer import typer
@@ -292,6 +293,15 @@ def run_command(
display_device_status(console, adb_client=adb_client) display_device_status(console, adb_client=adb_client)
cancelled = False cancelled = False
original_sigterm = None
try:
if hasattr(signal, "SIGTERM"):
original_sigterm = signal.signal(
signal.SIGTERM, lambda s, f: (_ for _ in ()).throw(KeyboardInterrupt())
)
except Exception:
pass
try: try:
asyncio.run( asyncio.run(
execute_task( execute_task(
@@ -335,5 +345,10 @@ def run_command(
console.print(f"[bold red]Task execution failed:[/bold red] {e}") console.print(f"[bold red]Task execution failed:[/bold red] {e}")
raise raise
finally: finally:
try:
if original_sigterm is not None and hasattr(signal, "SIGTERM"):
signal.signal(signal.SIGTERM, original_sigterm)
except Exception:
pass
if cancelled: if cancelled:
raise SystemExit(130) raise SystemExit(130)
+42 -6
View File
@@ -19,6 +19,7 @@ from pathlib import Path
import shutil import shutil
import signal import signal
import tempfile import tempfile
import time
from artemis.platform.base import IPlatform, IPlatformPaths, IPlatformProcess, OSType from artemis.platform.base import IPlatform, IPlatformPaths, IPlatformProcess, OSType
@@ -98,12 +99,47 @@ class DarwinPlatformProcess(IPlatformProcess):
pass pass
parent.send_signal(signal.SIGTERM) parent.send_signal(signal.SIGTERM)
gone, alive = psutil.wait_procs(children + [parent], timeout=timeout_seconds)
for p in alive: # Avoid calling wait_procs on direct children of the current process.
try: # In POSIX, psutil.Process.wait() on a child of os.getpid() invokes
p.kill() # os.waitpid(), which reaps the zombie and prevents asyncio / uvloop
except (psutil.NoSuchProcess, psutil.AccessDenied): # event loops from detecting that the child exited, causing proc.wait()
pass # to hang indefinitely.
current_pid = os.getpid()
external_procs = [
p
for p in children + [parent]
if p.pid != current_pid and getattr(p, "ppid", lambda: None)() != current_pid
]
direct_children = [
p
for p in children + [parent]
if getattr(p, "ppid", lambda: None)() == current_pid
]
if external_procs:
gone, alive = psutil.wait_procs(external_procs, timeout=timeout_seconds)
for p in alive:
try:
p.kill()
except (psutil.NoSuchProcess, psutil.AccessDenied):
pass
deadline = time.time() + timeout_seconds
for p in direct_children:
while time.time() < deadline:
try:
if not p.is_running() or p.status() == psutil.STATUS_ZOMBIE:
break
except (psutil.NoSuchProcess, psutil.AccessDenied):
break
time.sleep(0.05)
else:
try:
p.kill()
except (psutil.NoSuchProcess, psutil.AccessDenied):
pass
return True return True
except Exception: except Exception:
try: try:
+42 -6
View File
@@ -19,6 +19,7 @@ from pathlib import Path
import shutil import shutil
import signal import signal
import tempfile import tempfile
import time
from artemis.platform.base import IPlatform, IPlatformPaths, IPlatformProcess, OSType from artemis.platform.base import IPlatform, IPlatformPaths, IPlatformProcess, OSType
@@ -113,12 +114,47 @@ class LinuxPlatformProcess(IPlatformProcess):
pass pass
parent.send_signal(signal.SIGTERM) parent.send_signal(signal.SIGTERM)
gone, alive = psutil.wait_procs(children + [parent], timeout=timeout_seconds)
for p in alive: # Avoid calling wait_procs on direct children of the current process.
try: # In POSIX, psutil.Process.wait() on a child of os.getpid() invokes
p.kill() # os.waitpid(), which reaps the zombie and prevents asyncio / uvloop
except (psutil.NoSuchProcess, psutil.AccessDenied): # event loops from detecting that the child exited, causing proc.wait()
pass # to hang indefinitely.
current_pid = os.getpid()
external_procs = [
p
for p in children + [parent]
if p.pid != current_pid and getattr(p, "ppid", lambda: None)() != current_pid
]
direct_children = [
p
for p in children + [parent]
if getattr(p, "ppid", lambda: None)() == current_pid
]
if external_procs:
gone, alive = psutil.wait_procs(external_procs, timeout=timeout_seconds)
for p in alive:
try:
p.kill()
except (psutil.NoSuchProcess, psutil.AccessDenied):
pass
deadline = time.time() + timeout_seconds
for p in direct_children:
while time.time() < deadline:
try:
if not p.is_running() or p.status() == psutil.STATUS_ZOMBIE:
break
except (psutil.NoSuchProcess, psutil.AccessDenied):
break
time.sleep(0.05)
else:
try:
p.kill()
except (psutil.NoSuchProcess, psutil.AccessDenied):
pass
return True return True
except Exception: except Exception:
try: try:
+27 -16
View File
@@ -645,23 +645,34 @@ class Agent:
return output return output
finally: finally:
if recording_started: if recording_started:
try: async def _safe_stop_recording():
logger.info(f"[{task_name}] Stopping automated screen recording...") try:
logger.info(f"[{task_name}] Stopping automated screen recording...")
controller = get_controller(context)
return await controller.stop_video_recording()
except Exception as e:
logger.error(f"[{task_name}] Error stopping screen recording: {e}")
return None
controller = get_controller(context) stop_task = asyncio.create_task(_safe_stop_recording())
stop_result = await controller.stop_video_recording() try:
if stop_result.success and stop_result.video_path: stop_result = await asyncio.shield(stop_task)
logger.info( except (asyncio.CancelledError, BaseException):
f"[{task_name}] Screen recording saved" try:
f" to: {stop_result.video_path}" stop_result = await asyncio.wait_for(stop_task, timeout=15.0)
) except Exception:
else: stop_result = None
logger.warning(
f"[{task_name}] Failed to save screen" if stop_result and stop_result.success and stop_result.video_path:
f" recording: {stop_result.message}" logger.info(
) f"[{task_name}] Screen recording saved"
except Exception as e: f" to: {stop_result.video_path}"
logger.error(f"[{task_name}] Error stopping screen recording: {e}") )
elif stop_result:
logger.warning(
f"[{task_name}] Failed to save screen"
f" recording: {stop_result.message}"
)
except asyncio.CancelledError: except asyncio.CancelledError:
err = f"[{task_name}] Task cancelled" err = f"[{task_name}] Task cancelled"
logger.warning(err) logger.warning(err)
+46 -9
View File
@@ -297,6 +297,36 @@ async def remux_recording_to_mp4(source_path: Path, output_path: Path) -> bool:
if valid: if valid:
temporary_path.replace(output_path) temporary_path.replace(output_path)
return True return True
logger.warning(
f"Direct remux failed for {source_path} (code {process.returncode}), "
"attempting transcode fallback with ffmpeg..."
)
# Fallback to ultrafast re-encoding in case MKV container was truncated or has timestamp irregularities
fallback_proc = await asyncio.create_subprocess_exec(
get_ffmpeg_path(),
"-y",
"-fflags",
"+genpts+discardcorrupt",
"-i",
str(source_path),
"-c:v",
"libx264",
"-preset",
"ultrafast",
"-pix_fmt",
"yuv420p",
"-movflags",
"+faststart",
str(temporary_path),
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
_fout, _ferr = await fallback_proc.communicate()
if fallback_proc.returncode == 0 and temporary_path.exists() and temporary_path.stat().st_size > 0:
temporary_path.replace(output_path)
return True
logger.error( logger.error(
f"Failed to remux recording segment (code {process.returncode}): " f"Failed to remux recording segment (code {process.returncode}): "
f"{stderr.decode(errors='replace')[-2000:]}" f"{stderr.decode(errors='replace')[-2000:]}"
@@ -314,10 +344,8 @@ async def probe_video_segment(video_path: Path) -> dict[str, float | int]:
get_ffprobe_path(), get_ffprobe_path(),
"-v", "-v",
"error", "error",
"-select_streams",
"v:0",
"-show_entries", "-show_entries",
"stream=width,height:format=duration", "stream=width,height,duration,codec_type:format=duration",
"-of", "-of",
"json", "json",
str(video_path), str(video_path),
@@ -329,11 +357,20 @@ async def probe_video_segment(video_path: Path) -> dict[str, float | int]:
return {} return {}
try: try:
payload = json.loads(stdout) payload = json.loads(stdout)
stream = (payload.get("streams") or [{}])[0] streams = payload.get("streams") or []
video_stream = next(
(s for s in streams if s.get("width") and s.get("height")),
next((s for s in streams if s.get("codec_type") == "video"), streams[0] if streams else {})
)
duration = float(
(payload.get("format") or {}).get("duration")
or video_stream.get("duration")
or 0
)
return { return {
"duration": float((payload.get("format") or {}).get("duration") or 0), "duration": duration,
"width": int(stream.get("width") or 0), "width": int(video_stream.get("width") or 0),
"height": int(stream.get("height") or 0), "height": int(video_stream.get("height") or 0),
} }
except (TypeError, ValueError, json.JSONDecodeError): except (TypeError, ValueError, json.JSONDecodeError):
return {} return {}
@@ -380,8 +417,8 @@ async def write_recording_manifest(output_dir: Path, mp4_paths: list[Path]) -> P
width = int(metadata.get("width", 0)) width = int(metadata.get("width", 0))
height = int(metadata.get("height", 0)) height = int(metadata.get("height", 0))
if duration <= 0 or width <= 0 or height <= 0: if duration <= 0 or width <= 0 or height <= 0:
logger.error(f"Recording segment failed validation: {path}") logger.warning(f"Recording segment skipped due to invalid metadata: {path}")
return None continue
segments.append( segments.append(
{ {
"file": path.name, "file": path.name,
@@ -553,3 +553,54 @@ async def test_immediate_cancel_ignores_stale_ipc_and_runs_next_task():
await task await task
except (asyncio.CancelledError, Exception): except (asyncio.CancelledError, Exception):
pass pass
@pytest.mark.asyncio
async def test_wait_for_worker_process_watchdog_handles_reaped_process():
"""Verify that _wait_for_worker_process never hangs even if the process was already reaped."""
mock_proc = MagicMock()
mock_proc.pid = 999999
mock_proc.returncode = None
async def hanging_wait():
await asyncio.sleep(10)
return 0
mock_proc.wait = AsyncMock(side_effect=hanging_wait)
with patch("psutil.Process") as mock_psutil_proc:
mock_p = MagicMock()
mock_p.is_running.return_value = False
mock_psutil_proc.return_value = mock_p
rc = await TaskQueueService._wait_for_worker_process(mock_proc)
assert rc == -15
def test_darwin_terminate_process_tree_preserves_direct_child_for_asyncio():
"""Verify darwin terminate_process_tree does not pass direct children to psutil.wait_procs."""
import os
from artemis.platform.darwin import DarwinPlatformProcess
process = DarwinPlatformProcess()
current_pid = os.getpid()
with (
patch("psutil.Process") as mock_psutil_proc,
patch("psutil.wait_procs") as mock_wait_procs,
):
parent = MagicMock()
parent.pid = 12345
parent.ppid.return_value = current_pid
parent.children.return_value = []
parent.is_running.return_value = False
mock_psutil_proc.return_value = parent
mock_wait_procs.return_value = ([], [])
success = process.terminate_process_tree(12345, timeout_seconds=0.1)
assert success is True
parent.send_signal.assert_called_once()
for call_args in mock_wait_procs.call_args_list:
procs_waited = call_args[0][0]
assert parent not in procs_waited