feat: add step-by-step visual replay and harden process lifecycle
- UI: Implement dual-mode playback in FloatingVideoPlayer supporting video and step-by-step frame replay with scrub controls and auto-fallback - Runtime: Prevent asyncio event loop hangs on Darwin and Linux by avoiding psutil.wait_procs on direct child processes - Task Queue: Add worker exit watchdog and automatic recording recovery for completed or interrupted sessions - Recording: Add ffmpeg ultrafast transcode fallback, shield recording finalization against cancellation, and handle on-the-fly MP4 conversion - Docs: Polish README_CN styling and streamline flash runner agent prompt
This commit is contained in:
1 parent
06643e955b
commit
a36a66ce6d
26 files changed
+1152
-117
No files matched your search
+18
-18
@@ -28,21 +28,21 @@
|
|||||||
<p align="center">
|
<p align="center">
|
||||||
<img src="./docs/assets/demo.gif" alt="Artemis 演示效果" width="100%" />
|
<img src="./docs/assets/demo.gif" alt="Artemis 演示效果" width="100%" />
|
||||||
<br>
|
<br>
|
||||||
<em><b>实机演示</b>:在 Google Maps 中设置驾车路线并计算总耗时,随后打开 YouTube 播放 Coldplay 的歌曲。</em>
|
<em>实机演示:在 Google Maps 中设置驾车路线并计算总耗时,随后打开 YouTube 播放 Coldplay 的歌曲。</em>
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
## ✨ 核心亮点
|
## 核心亮点
|
||||||
|
|
||||||
* 🤖 **跨 App 复杂自动化与 AI 个人助理**:不仅能执行严格的自动化测试用例,更具备接管真机的自主决策与长流程操作能力,一句话处理跨应用复杂业务与任务;
|
* **跨 App 复杂自动化与 AI 个人助理**:不仅能执行严格的自动化测试用例,更具备接管真机的自主决策与长流程操作能力,一句话处理跨应用复杂业务与任务;
|
||||||
* 🧪 **零脚本维护自动化测试**:基于“动态优先、坐标兜底”的多模态语义定位,彻底告别传统 XPath / 控件 ID 频繁失效的痛点,无惧 App 改版、分辨率差异与系统升级;
|
* **零脚本维护自动化测试**:基于“动态优先、坐标兜底”的多模态语义定位,彻底告别传统 XPath / 控件 ID 频繁失效的痛点,无惧 App 改版、分辨率差异与系统升级;
|
||||||
* 🐞 **IDE 内一键 Bug 复现与 Logcat 诊断**:原生支持 **MCP 协议**,可在 **Antigravity、Claude Code、Windsurf** 中直接用自然语言驱动真机复现缺陷,自动抓取关键帧截图与 **Logcat 崩溃堆栈**,完成研发测试闭环;
|
* **IDE 内一键 Bug 复现与 Logcat 诊断**:原生支持 **MCP 协议**,可在 **Antigravity、Claude Code、Windsurf** 中直接用自然语言驱动真机复现缺陷,自动抓取关键帧截图与 **Logcat 崩溃堆栈**,完成研发测试闭环;
|
||||||
* ⚡ **极速执行吞吐(单步 3–5 秒)**:首创**全链路乐观异步流水线 (Optimistic Async Pipeline)**,解耦重度推理与页面操作,在 Flash 模式下实现丝滑的高频交互与快速回归验证;
|
* **极速执行吞吐(单步 3–5 秒)**:首创**全链路乐观异步流水线 (Optimistic Async Pipeline)**,解耦重度推理与页面操作,在 Flash 模式下实现丝滑的高频交互与快速回归验证;
|
||||||
* 🛡️ **干扰弹窗自愈与超长程巡检**:独创 **Safety Net** 执行前校验机制,自动识别并清除系统权限弹窗、通知遮挡等异常;Pro 模式支持连续 **10+ 小时** 无人值守稳定性巡检与探索性测试 (Exploratory / Monkey-plus Testing);
|
* **干扰弹窗自愈与超长程巡检**:独创 **Safety Net** 执行前校验机制,自动识别并清除系统权限弹窗、通知遮挡等异常;Pro 模式支持连续 **10+ 小时** 无人值守稳定性巡检与探索性测试 (Exploratory / Monkey-plus Testing);
|
||||||
* **业界顶尖 SOTA**:在 Google Research **AndroidWorld** 基准评测(100+ 复杂长程系统与应用交互任务)中取得 **99%+ 任务完成率**。
|
* **业界顶尖 SOTA**:在 Google Research **AndroidWorld** 基准评测(100+ 复杂长程系统与应用交互任务)中取得 **99%+ 任务完成率**。
|
||||||
|
|
||||||
<a id="workflow-showcase"></a>
|
<a id="workflow-showcase"></a>
|
||||||
<a id="全流程演示"></a>
|
<a id="全流程演示"></a>
|
||||||
## 🤝 Antigravity × ARTEMIS:全流程自主测试演示
|
## Antigravity × ARTEMIS:全流程自主测试演示
|
||||||
|
|
||||||
通过原生 MCP 协议,**Antigravity** 与 **ARTEMIS** 深度协同——只需一句自然语言指令,即可自动完成从需求理解、测试规划、真机执行到深度报告输出的完整闭环:
|
通过原生 MCP 协议,**Antigravity** 与 **ARTEMIS** 深度协同——只需一句自然语言指令,即可自动完成从需求理解、测试规划、真机执行到深度报告输出的完整闭环:
|
||||||
|
|
||||||
@@ -62,7 +62,7 @@
|
|||||||
<tr>
|
<tr>
|
||||||
<td width="50%" align="center">
|
<td width="50%" align="center">
|
||||||
<b>3. 自主执行测试 (Autonomous Test Execution)</b><br>
|
<b>3. 自主执行测试 (Autonomous Test Execution)</b><br>
|
||||||
<sub>接管真机自动化操作,跳过广告并实时采集性能指标</sub><br><br>
|
<sub>驱动真机操作、界面导航并实时分析性能指标</sub><br><br>
|
||||||
<img src="./docs/assets/workflow-3-exec.png" width="100%" alt="步骤三:自主执行测试" />
|
<img src="./docs/assets/workflow-3-exec.png" width="100%" alt="步骤三:自主执行测试" />
|
||||||
</td>
|
</td>
|
||||||
<td width="50%" align="center">
|
<td width="50%" align="center">
|
||||||
@@ -78,8 +78,8 @@
|
|||||||
## 快速上手
|
## 快速上手
|
||||||
|
|
||||||
确保电脑已连接 Android 实体机(已开启 **USB 调试**)或 Android 模拟器。一键启动脚本将会自动完成以下配置:
|
确保电脑已连接 Android 实体机(已开启 **USB 调试**)或 Android 模拟器。一键启动脚本将会自动完成以下配置:
|
||||||
- 🛠️ **安装系统环境依赖**:自动检测并安装 ADB、scrcpy、FFmpeg 与 Python(`uv`)运行时及项目依赖。
|
- **安装系统环境依赖**:自动检测并安装 ADB、scrcpy、FFmpeg 与 Python(`uv`)运行时及项目依赖。
|
||||||
- 🔌 **全局挂载 MCP 服务与测试准则 (Rules)**:主动引导并自动将全局 MCP 服务与 **Artemis 移动端测试思维准则 (`rules.md`)** 挂载至你使用的 AI IDE(支持 **Antigravity**、**Cursor**、**Claude Code**、**Codex**、**Windsurf**、**VS Code**、**Cline/Roo**、**OpenClaw**)。
|
- **全局挂载 MCP 服务与测试准则 (Rules)**:主动引导并自动将全局 MCP 服务与 **Artemis 移动端测试思维准则 (`rules.md`)** 挂载至你使用的 AI IDE(支持 **Antigravity**、**Cursor**、**Claude Code**、**Codex**、**Windsurf**、**VS Code**、**Cline/Roo**、**OpenClaw**)。
|
||||||
|
|
||||||
### macOS 与 Linux
|
### macOS 与 Linux
|
||||||
|
|
||||||
@@ -109,7 +109,7 @@ cd artemis
|
|||||||
<a id="mcp-setup"></a>
|
<a id="mcp-setup"></a>
|
||||||
<a id="mcp"></a>
|
<a id="mcp"></a>
|
||||||
<details>
|
<details>
|
||||||
<summary><b>🔌 接入 Codex / Antigravity / Claude Code / Windsurf (MCP)(点击展开)</b></summary>
|
<summary><b>Codex / Antigravity / Claude Code / Windsurf MCP 配置(点击展开)</b></summary>
|
||||||
|
|
||||||
<br>
|
<br>
|
||||||
|
|
||||||
@@ -192,17 +192,17 @@ PYTHONPATH = "/path/to/artemis"
|
|||||||
* **Codex**:将内容添加至 `~/.codex/AGENTS.md`(或当前生效的 `AGENTS.override.md`)。
|
* **Codex**:将内容添加至 `~/.codex/AGENTS.md`(或当前生效的 `AGENTS.override.md`)。
|
||||||
* **Windsurf / OpenClaw**:将内容添加到工作区规则或全局 System Prompt 中。
|
* **Windsurf / OpenClaw**:将内容添加到工作区规则或全局 System Prompt 中。
|
||||||
|
|
||||||
> 💡 更多规范设计细节与 MCP 架构说明,请参阅 [MCP Server 文档](./mcp_server/README.md)。
|
> 更多规范设计细节与 MCP 架构说明,请参阅 [MCP Server 文档](./mcp_server/README.md)。
|
||||||
|
|
||||||
### 4. 在 IDE 中体验真机协同
|
### 4. 在 IDE 中体验真机协同
|
||||||
在 Codex / Antigravity / Claude Code 对话框中直接输入:
|
在 Codex / Antigravity / Claude Code 对话框中直接输入:
|
||||||
> 💬 *"请帮我把刚刚修改的代码编译成 APK 并安装到手机上,打开登录页面输入测试账号,验证登录后是否有异常弹窗,并把最终页面截图回传。"*
|
> *"请帮我把刚刚修改的代码编译成 APK 并安装到手机上,打开登录页面输入测试账号,验证登录后是否有异常弹窗,并把最终页面截图回传。"*
|
||||||
|
|
||||||
</details>
|
</details>
|
||||||
|
|
||||||
<a id="python-sdk"></a>
|
<a id="python-sdk"></a>
|
||||||
<details>
|
<details>
|
||||||
<summary><b>🐍 Python SDK 极简集成(点击展开)</b></summary>
|
<summary><b>Python SDK 集成(点击展开)</b></summary>
|
||||||
|
|
||||||
<br>
|
<br>
|
||||||
|
|
||||||
@@ -238,12 +238,12 @@ if __name__ == "__main__":
|
|||||||
<p align="center">
|
<p align="center">
|
||||||
<img src="./docs/assets/artemis-ui-showcase.png" alt="Artemis 可视化控制台" width="100%" />
|
<img src="./docs/assets/artemis-ui-showcase.png" alt="Artemis 可视化控制台" width="100%" />
|
||||||
<br />
|
<br />
|
||||||
<sub><b>控制台功能布局</b>:<b>① 顶栏视图切换</b>(主页与工作区) · <b>② 运行模式与录屏回放</b>(Flash/Pro 状态与视频回放) · <b>③ 实时感知推理流</b>(动作分解、点击坐标与结构化总结) · <b>④ 自然语言下发胶囊</b>(自然语言驱动真机) · <b>⑤ 任务队列看板</b>(状态流转与历史回溯)</sub>
|
<sub><b>控制台功能概览</b>:<b>① 视图切换</b>(主页与工作区) · <b>② 运行模式与回放</b>(Flash/Pro 状态与视频回放) · <b>③ 实时感知推理流</b>(动作感知、目标坐标与结构化总结) · <b>④ 提示词输入坞 (Prompt Dock)</b>(自然语言下发) · <b>⑤ 任务队列看板</b>(生命周期与历史回溯)</sub>
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
* **Web 可视化测试控制台 (`uv run artemis ui`)**:集成设备实时投屏与交互面板,支持通过自然语言下发测试用例,实时观测推理步骤、操作轨迹、截图留存与异常状态回放;
|
* **Web 可视化测试控制台 (`uv run artemis ui`)**:集成设备实时投屏与交互面板,支持通过自然语言下发测试用例,实时观测推理步骤、操作轨迹、截图留存与异常状态回放;
|
||||||
* **MCP 协议集成 (AI IDE 协同)**:作为标准 MCP 服务器无缝接入 **Antigravity、Claude Code、Windsurf** 等开发环境,在 IDE 中直接驱动真机完成自动化测试与 Bug 复现验证;
|
* **原生 MCP 协议 (IDE 协同)**:作为标准 MCP 服务器无缝接入 **Antigravity、Claude Code、Windsurf** 等开发环境,在 IDE 中直接驱动真机完成自动化测试与 Bug 复现验证;
|
||||||
* **命令行工具 (`uv run artemis run`)**:支持通过 CLI 直接执行自动化用例、稳定性巡检或 AndroidWorld 基准评测,提供高保真结构化终端输出;
|
* **开发者命令行 CLI (`uv run artemis run`)**:支持通过终端直接执行自动化测试用例、探索性稳定性巡检或 AndroidWorld 基准评测,提供高保真结构化终端输出;
|
||||||
* **Python SDK**:作为标准 Python 库无缝集成至现有自动化测试框架(如 pytest)或 CI/CD 流水线,提供基于 Pydantic 的强类型结构化结果与断言支持。
|
* **Python SDK**:作为标准 Python 库无缝集成至现有自动化测试框架(如 pytest)或 CI/CD 流水线,提供基于 Pydantic 的强类型结构化结果与断言支持。
|
||||||
|
|
||||||
<a id="benchmarks"></a>
|
<a id="benchmarks"></a>
|
||||||
|
|||||||
@@ -204,3 +204,9 @@ class ServerState:
|
|||||||
|
|
||||||
# Global shared instance
|
# Global shared instance
|
||||||
state = ServerState()
|
state = ServerState()
|
||||||
|
|
||||||
|
import sys
|
||||||
|
if __name__ == "admin_console.core.state":
|
||||||
|
sys.modules["apps.admin_console.core.state"] = sys.modules[__name__]
|
||||||
|
elif __name__ == "apps.admin_console.core.state":
|
||||||
|
sys.modules["admin_console.core.state"] = sys.modules[__name__]
|
||||||
@@ -141,6 +141,22 @@ class SessionRepository:
|
|||||||
except Exception:
|
except Exception:
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
def mark_recording_ready(self, session_id: str, local_video_path: str) -> bool:
|
||||||
|
"""Mark a recording as ready with its finalized video path."""
|
||||||
|
try:
|
||||||
|
with db_session(self.db_path) as conn:
|
||||||
|
cursor = conn.cursor()
|
||||||
|
cursor.execute(
|
||||||
|
"UPDATE video_recordings SET status = 'ready', error = NULL, "
|
||||||
|
"local_video_path = ?, end_time = COALESCE(end_time, ?) "
|
||||||
|
"WHERE session_id = ?",
|
||||||
|
(str(local_video_path), time.time(), str(session_id)),
|
||||||
|
)
|
||||||
|
conn.commit()
|
||||||
|
return cursor.rowcount > 0
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
|
||||||
def get_llm_traces_for_profile(self, session_id: str, limit: int = 3) -> list[str]:
|
def get_llm_traces_for_profile(self, session_id: str, limit: int = 3) -> list[str]:
|
||||||
with db_session(self.db_path) as conn:
|
with db_session(self.db_path) as conn:
|
||||||
cursor = conn.cursor()
|
cursor = conn.cursor()
|
||||||
|
|||||||
@@ -102,14 +102,18 @@ async def get_session_video(session_id: str):
|
|||||||
}
|
}
|
||||||
|
|
||||||
if recording_status == "failed":
|
if recording_status == "failed":
|
||||||
return {
|
v_url = media_service.resolve_video_url(row_dict, video_rec_map, video_idx)
|
||||||
"session_id": session_id,
|
if not v_url:
|
||||||
"status": "failed",
|
return {
|
||||||
"has_video": False,
|
"session_id": session_id,
|
||||||
"video_url": None,
|
"status": "failed",
|
||||||
"video_segments": [],
|
"has_video": False,
|
||||||
"message": recording.get("error") or "Recording finalization failed",
|
"video_url": None,
|
||||||
}
|
"video_segments": [],
|
||||||
|
"message": recording.get("error") or "Recording finalization failed",
|
||||||
|
}
|
||||||
|
# If a video was recovered/found, update DB to ready and proceed to serve it
|
||||||
|
session_repo.mark_recording_ready(session_id, v_url)
|
||||||
|
|
||||||
v_url = media_service.resolve_video_url(row_dict, video_rec_map, video_idx)
|
v_url = media_service.resolve_video_url(row_dict, video_rec_map, video_idx)
|
||||||
video_segments = media_service.resolve_video_segments(v_url)
|
video_segments = media_service.resolve_video_segments(v_url)
|
||||||
|
|||||||
@@ -63,12 +63,17 @@ async def list_sessions():
|
|||||||
orphaned_ids.append(s_id)
|
orphaned_ids.append(s_id)
|
||||||
|
|
||||||
recording_status = str((recording or {}).get("status") or "unavailable")
|
recording_status = str((recording or {}).get("status") or "unavailable")
|
||||||
row_dict["recording_status"] = recording_status
|
resolved_v_url = (
|
||||||
row_dict["video_url"] = (
|
|
||||||
media_service.resolve_video_url(row_dict, video_rec_map, video_idx)
|
media_service.resolve_video_url(row_dict, video_rec_map, video_idx)
|
||||||
if recording_status in ("ready", "unavailable")
|
if recording_status != "recording"
|
||||||
else None
|
else None
|
||||||
)
|
)
|
||||||
|
if resolved_v_url:
|
||||||
|
recording_status = "ready"
|
||||||
|
row_dict["video_url"] = resolved_v_url
|
||||||
|
else:
|
||||||
|
row_dict["video_url"] = None
|
||||||
|
row_dict["recording_status"] = recording_status
|
||||||
|
|
||||||
agent_names = agent_names_by_session.get(s_id, [])
|
agent_names = agent_names_by_session.get(s_id, [])
|
||||||
sess_profile = model_service.resolve_session_profile(
|
sess_profile = model_service.resolve_session_profile(
|
||||||
|
|||||||
@@ -17,6 +17,7 @@ import hashlib
|
|||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
import subprocess
|
||||||
from typing import Any
|
from typing import Any
|
||||||
import urllib.parse
|
import urllib.parse
|
||||||
from fastapi import HTTPException
|
from fastapi import HTTPException
|
||||||
@@ -30,6 +31,72 @@ except ImportError:
|
|||||||
class MediaService:
|
class MediaService:
|
||||||
"""Service handling media indexing, local file security, payload unwrapping and notes/plans."""
|
"""Service handling media indexing, local file security, payload unwrapping and notes/plans."""
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def ensure_browser_playable_video(cls, p: Path) -> Path:
|
||||||
|
"""Ensure the video file is in a browser-supported container (MP4).
|
||||||
|
|
||||||
|
If given an MKV, check if a converted MP4 already exists or convert it.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
if not p.exists():
|
||||||
|
return p
|
||||||
|
suffix = p.suffix.lower()
|
||||||
|
if suffix == ".mp4":
|
||||||
|
return p
|
||||||
|
if suffix in (".mkv", ".webm"):
|
||||||
|
mp4_cand = p.with_suffix(".mp4")
|
||||||
|
if mp4_cand.exists() and mp4_cand.stat().st_size > 0:
|
||||||
|
return mp4_cand
|
||||||
|
from artemis.utils.video import get_ffmpeg_path
|
||||||
|
ffmpeg = get_ffmpeg_path()
|
||||||
|
# 1. Attempt fast copy remux
|
||||||
|
subprocess.run(
|
||||||
|
[
|
||||||
|
ffmpeg,
|
||||||
|
"-y",
|
||||||
|
"-fflags",
|
||||||
|
"+genpts",
|
||||||
|
"-i",
|
||||||
|
str(p),
|
||||||
|
"-c",
|
||||||
|
"copy",
|
||||||
|
"-movflags",
|
||||||
|
"+faststart",
|
||||||
|
str(mp4_cand),
|
||||||
|
],
|
||||||
|
capture_output=True,
|
||||||
|
timeout=15,
|
||||||
|
)
|
||||||
|
if mp4_cand.exists() and mp4_cand.stat().st_size > 0:
|
||||||
|
return mp4_cand
|
||||||
|
# 2. If copy remux failed, fallback to ultrafast re-encode
|
||||||
|
subprocess.run(
|
||||||
|
[
|
||||||
|
ffmpeg,
|
||||||
|
"-y",
|
||||||
|
"-fflags",
|
||||||
|
"+genpts+discardcorrupt",
|
||||||
|
"-i",
|
||||||
|
str(p),
|
||||||
|
"-c:v",
|
||||||
|
"libx264",
|
||||||
|
"-preset",
|
||||||
|
"ultrafast",
|
||||||
|
"-pix_fmt",
|
||||||
|
"yuv420p",
|
||||||
|
"-movflags",
|
||||||
|
"+faststart",
|
||||||
|
str(mp4_cand),
|
||||||
|
],
|
||||||
|
capture_output=True,
|
||||||
|
timeout=30,
|
||||||
|
)
|
||||||
|
if mp4_cand.exists() and mp4_cand.stat().st_size > 0:
|
||||||
|
return mp4_cand
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
return p
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def path_to_video_url(p: Path) -> str:
|
def path_to_video_url(p: Path) -> str:
|
||||||
resolved = p.resolve()
|
resolved = p.resolve()
|
||||||
@@ -56,12 +123,14 @@ class MediaService:
|
|||||||
for ext in [".mp4", ".mkv", ".webm"]:
|
for ext in [".mp4", ".mkv", ".webm"]:
|
||||||
vfile = item / f"recording{ext}"
|
vfile = item / f"recording{ext}"
|
||||||
if vfile.exists():
|
if vfile.exists():
|
||||||
|
vfile = cls.ensure_browser_playable_video(vfile)
|
||||||
url = cls.path_to_video_url(vfile)
|
url = cls.path_to_video_url(vfile)
|
||||||
idx[item.name] = url
|
idx[item.name] = url
|
||||||
prefix = item.name.split("_PASS_")[0].split("_FAIL_")[0]
|
prefix = item.name.split("_PASS_")[0].split("_FAIL_")[0]
|
||||||
idx[prefix] = url
|
idx[prefix] = url
|
||||||
break
|
break
|
||||||
elif item.suffix in [".mp4", ".mkv", ".webm"]:
|
elif item.suffix in [".mp4", ".mkv", ".webm"]:
|
||||||
|
item = cls.ensure_browser_playable_video(item)
|
||||||
url = cls.path_to_video_url(item)
|
url = cls.path_to_video_url(item)
|
||||||
idx[item.stem] = url
|
idx[item.stem] = url
|
||||||
idx[item.name] = url
|
idx[item.name] = url
|
||||||
@@ -85,12 +154,14 @@ class MediaService:
|
|||||||
v_url = None
|
v_url = None
|
||||||
v_fp = row_dict.get("video_filepath")
|
v_fp = row_dict.get("video_filepath")
|
||||||
if v_fp and os.path.exists(v_fp):
|
if v_fp and os.path.exists(v_fp):
|
||||||
v_url = cls.path_to_video_url(Path(v_fp))
|
p = cls.ensure_browser_playable_video(Path(v_fp))
|
||||||
|
v_url = cls.path_to_video_url(p)
|
||||||
|
|
||||||
orig_rec = video_rec_map.get(s_id)
|
orig_rec = video_rec_map.get(s_id)
|
||||||
if not v_url and orig_rec:
|
if not v_url and orig_rec:
|
||||||
p = Path(orig_rec)
|
p = Path(orig_rec)
|
||||||
if p.exists():
|
if p.exists():
|
||||||
|
p = cls.ensure_browser_playable_video(p)
|
||||||
v_url = cls.path_to_video_url(p)
|
v_url = cls.path_to_video_url(p)
|
||||||
else:
|
else:
|
||||||
folder_name = p.parent.name
|
folder_name = p.parent.name
|
||||||
@@ -105,12 +176,14 @@ class MediaService:
|
|||||||
for ext in [".mp4", ".webm", ".mkv"]:
|
for ext in [".mp4", ".webm", ".mkv"]:
|
||||||
vfile = sess_dir / f"recording{ext}"
|
vfile = sess_dir / f"recording{ext}"
|
||||||
if vfile.exists():
|
if vfile.exists():
|
||||||
|
vfile = cls.ensure_browser_playable_video(vfile)
|
||||||
v_url = cls.path_to_video_url(vfile)
|
v_url = cls.path_to_video_url(vfile)
|
||||||
break
|
break
|
||||||
if not v_url:
|
if not v_url:
|
||||||
try:
|
try:
|
||||||
for item in sess_dir.iterdir():
|
for item in sess_dir.iterdir():
|
||||||
if item.is_file() and item.suffix.lower() in [".mp4", ".webm", ".mkv"]:
|
if item.is_file() and item.suffix.lower() in [".mp4", ".webm", ".mkv"]:
|
||||||
|
item = cls.ensure_browser_playable_video(item)
|
||||||
v_url = cls.path_to_video_url(item)
|
v_url = cls.path_to_video_url(item)
|
||||||
break
|
break
|
||||||
except Exception:
|
except Exception:
|
||||||
|
|||||||
@@ -32,6 +32,7 @@ try:
|
|||||||
)
|
)
|
||||||
from admin_console.core.state import state
|
from admin_console.core.state import state
|
||||||
from admin_console.database.repositories.session_repository import session_repo
|
from admin_console.database.repositories.session_repository import session_repo
|
||||||
|
from admin_console.services.media_service import media_service
|
||||||
from artemis.runtime import DeviceExecutionLock, process_supervisor
|
from artemis.runtime import DeviceExecutionLock, process_supervisor
|
||||||
except ImportError:
|
except ImportError:
|
||||||
from apps.admin_console.core.config import (
|
from apps.admin_console.core.config import (
|
||||||
@@ -42,6 +43,7 @@ except ImportError:
|
|||||||
)
|
)
|
||||||
from apps.admin_console.core.state import state
|
from apps.admin_console.core.state import state
|
||||||
from apps.admin_console.database.repositories.session_repository import session_repo
|
from apps.admin_console.database.repositories.session_repository import session_repo
|
||||||
|
from apps.admin_console.services.media_service import media_service
|
||||||
from artemis.runtime import DeviceExecutionLock, process_supervisor
|
from artemis.runtime import DeviceExecutionLock, process_supervisor
|
||||||
|
|
||||||
|
|
||||||
@@ -199,6 +201,28 @@ class TaskQueueService:
|
|||||||
except ProcessLookupError:
|
except ProcessLookupError:
|
||||||
pass
|
pass
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
async def _wait_for_worker_process(proc: asyncio.subprocess.Process) -> int:
|
||||||
|
"""Wait for worker process to exit, with watchdog fallback if PID was reaped externally."""
|
||||||
|
while True:
|
||||||
|
try:
|
||||||
|
return await asyncio.wait_for(proc.wait(), timeout=1.0)
|
||||||
|
except asyncio.TimeoutError:
|
||||||
|
pid = getattr(proc, "pid", None)
|
||||||
|
if pid:
|
||||||
|
try:
|
||||||
|
import psutil
|
||||||
|
|
||||||
|
p = psutil.Process(pid)
|
||||||
|
if not p.is_running() or p.status() == psutil.STATUS_ZOMBIE:
|
||||||
|
return proc.returncode if proc.returncode is not None else -15
|
||||||
|
except (psutil.NoSuchProcess, ProcessLookupError):
|
||||||
|
return proc.returncode if proc.returncode is not None else -15
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
else:
|
||||||
|
return proc.returncode if proc.returncode is not None else -15
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def ensure_worker_running(cls):
|
def ensure_worker_running(cls):
|
||||||
"""Guarantees that the background queue worker task is active and running."""
|
"""Guarantees that the background queue worker task is active and running."""
|
||||||
@@ -341,7 +365,7 @@ class TaskQueueService:
|
|||||||
await cls._terminate_worker_process(proc)
|
await cls._terminate_worker_process(proc)
|
||||||
|
|
||||||
# 3. Await subprocess completion
|
# 3. Await subprocess completion
|
||||||
returncode = await proc.wait()
|
returncode = await cls._wait_for_worker_process(proc)
|
||||||
print(f"[QueueWorker] Task [{sess_id}] exited with returncode {returncode}")
|
print(f"[QueueWorker] Task [{sess_id}] exited with returncode {returncode}")
|
||||||
|
|
||||||
# 4. Perform fallback database status update and notification
|
# 4. Perform fallback database status update and notification
|
||||||
@@ -360,12 +384,36 @@ class TaskQueueService:
|
|||||||
f"[QueueWorker] Preserved authoritative session {sess_id} "
|
f"[QueueWorker] Preserved authoritative session {sess_id} "
|
||||||
f"status '{new_status}'"
|
f"status '{new_status}'"
|
||||||
)
|
)
|
||||||
recording_error = "Task worker exited before recording finalization completed"
|
rec_info = session_repo.get_video_recording_for_session(sess_id)
|
||||||
if session_repo.mark_recording_failed_if_pending(sess_id, recording_error):
|
rec_status = (rec_info or {}).get("status")
|
||||||
cls._broadcast_event(
|
recovered_video_path = None
|
||||||
"recording_failed",
|
if rec_status != "ready":
|
||||||
{"session_id": sess_id, "error": recording_error},
|
try:
|
||||||
)
|
video_rec_map = session_repo.get_video_recordings_map()
|
||||||
|
video_idx = await asyncio.to_thread(media_service.build_video_index)
|
||||||
|
recovered_url = await asyncio.to_thread(
|
||||||
|
media_service.resolve_video_url,
|
||||||
|
{"session_id": sess_id},
|
||||||
|
video_rec_map,
|
||||||
|
video_idx,
|
||||||
|
)
|
||||||
|
if recovered_url:
|
||||||
|
session_repo.mark_recording_ready(sess_id, recovered_url)
|
||||||
|
cls._broadcast_event(
|
||||||
|
"recording_ready",
|
||||||
|
{"session_id": sess_id, "video_url": recovered_url},
|
||||||
|
)
|
||||||
|
recovered_video_path = recovered_url
|
||||||
|
except Exception as rec_err:
|
||||||
|
print(f"[QueueWorker] Error attempting recording recovery: {rec_err}")
|
||||||
|
|
||||||
|
if not recovered_video_path and rec_status != "ready":
|
||||||
|
recording_error = "Task worker exited before recording finalization completed"
|
||||||
|
if session_repo.mark_recording_failed_if_pending(sess_id, recording_error):
|
||||||
|
cls._broadcast_event(
|
||||||
|
"recording_failed",
|
||||||
|
{"session_id": sess_id, "error": recording_error},
|
||||||
|
)
|
||||||
state.active_connections.pop(sess_id, None)
|
state.active_connections.pop(sess_id, None)
|
||||||
cls._broadcast_event(
|
cls._broadcast_event(
|
||||||
"session_ended",
|
"session_ended",
|
||||||
|
|||||||
@@ -263,28 +263,41 @@
|
|||||||
[class.recording]="agentService.isCurrentSessionRunning()"
|
[class.recording]="agentService.isCurrentSessionRunning()"
|
||||||
[class.processing]="!agentService.isCurrentSessionRunning() && agentService.currentSessionRecordingStatus() === 'processing'"
|
[class.processing]="!agentService.isCurrentSessionRunning() && agentService.currentSessionRecordingStatus() === 'processing'"
|
||||||
[class.has-video]="!agentService.isCurrentSessionRunning() && !!agentService.currentSessionVideoUrl()"
|
[class.has-video]="!agentService.isCurrentSessionRunning() && !!agentService.currentSessionVideoUrl()"
|
||||||
|
[class.has-steps]="!agentService.isCurrentSessionRunning() && !agentService.currentSessionVideoUrl() && agentService.hasCurrentSessionStepFrames()"
|
||||||
|
[class.failed]="!agentService.isCurrentSessionRunning() && !agentService.currentSessionVideoUrl() && !agentService.hasCurrentSessionStepFrames() && agentService.currentSessionRecordingStatus() === 'failed'"
|
||||||
(click)="agentService.toggleVideoPlayer()"
|
(click)="agentService.toggleVideoPlayer()"
|
||||||
[title]="agentService.isCurrentSessionRunning() ? 'Task is currently running (Recording screen)' : (agentService.currentSessionRecordingStatus() === 'processing' ? 'Preparing screen recording' : (agentService.currentSessionVideoUrl() ? 'Play Screen Recording' : 'Check Device Screen Recording'))"
|
[title]="getScreenRecordingButtonTitle()"
|
||||||
>
|
>
|
||||||
<span class="material-symbols-outlined video-icon" *ngIf="agentService.isCurrentSessionRunning()">
|
<span class="material-symbols-outlined video-icon" *ngIf="agentService.isCurrentSessionRunning()">
|
||||||
videocam
|
videocam
|
||||||
</span>
|
</span>
|
||||||
<span class="material-symbols-outlined video-icon" *ngIf="!agentService.isCurrentSessionRunning() && agentService.currentSessionRecordingStatus() === 'processing'">
|
<span class="material-symbols-outlined video-icon spin-animation" *ngIf="!agentService.isCurrentSessionRunning() && agentService.currentSessionRecordingStatus() === 'processing'">
|
||||||
progress_activity
|
progress_activity
|
||||||
</span>
|
</span>
|
||||||
<span class="material-symbols-outlined video-icon play-icon" *ngIf="!agentService.isCurrentSessionRunning() && agentService.currentSessionRecordingStatus() !== 'processing' && agentService.currentSessionVideoUrl()">
|
<span class="material-symbols-outlined video-icon play-icon" *ngIf="!agentService.isCurrentSessionRunning() && agentService.currentSessionRecordingStatus() !== 'processing' && agentService.currentSessionVideoUrl()">
|
||||||
smart_display
|
smart_display
|
||||||
</span>
|
</span>
|
||||||
<span class="material-symbols-outlined video-icon" *ngIf="!agentService.isCurrentSessionRunning() && agentService.currentSessionRecordingStatus() !== 'processing' && !agentService.currentSessionVideoUrl()">
|
<span class="material-symbols-outlined video-icon steps-icon" *ngIf="!agentService.isCurrentSessionRunning() && agentService.currentSessionRecordingStatus() !== 'processing' && !agentService.currentSessionVideoUrl() && agentService.hasCurrentSessionStepFrames()">
|
||||||
|
slideshow
|
||||||
|
</span>
|
||||||
|
<span class="material-symbols-outlined video-icon error-icon" *ngIf="!agentService.isCurrentSessionRunning() && agentService.currentSessionRecordingStatus() !== 'processing' && !agentService.currentSessionVideoUrl() && !agentService.hasCurrentSessionStepFrames() && agentService.currentSessionRecordingStatus() === 'failed'">
|
||||||
|
videocam_off
|
||||||
|
</span>
|
||||||
|
<span class="material-symbols-outlined video-icon" *ngIf="!agentService.isCurrentSessionRunning() && agentService.currentSessionRecordingStatus() !== 'processing' && !agentService.currentSessionVideoUrl() && !agentService.hasCurrentSessionStepFrames() && agentService.currentSessionRecordingStatus() !== 'failed'">
|
||||||
videocam
|
videocam
|
||||||
</span>
|
</span>
|
||||||
|
|
||||||
<span class="btn-text" *ngIf="agentService.isCurrentSessionRunning()">Recording...</span>
|
<span class="btn-text" *ngIf="agentService.isCurrentSessionRunning()">Recording...</span>
|
||||||
<span class="btn-text" *ngIf="!agentService.isCurrentSessionRunning() && agentService.currentSessionRecordingStatus() === 'processing'">Preparing...</span>
|
<span class="btn-text" *ngIf="!agentService.isCurrentSessionRunning() && agentService.currentSessionRecordingStatus() === 'processing'">Preparing...</span>
|
||||||
<span class="btn-text" *ngIf="!agentService.isCurrentSessionRunning() && agentService.currentSessionRecordingStatus() !== 'processing' && agentService.currentSessionVideoUrl()">Play Recording</span>
|
<span class="btn-text" *ngIf="!agentService.isCurrentSessionRunning() && agentService.currentSessionRecordingStatus() !== 'processing' && agentService.currentSessionVideoUrl()">Play Recording</span>
|
||||||
<span class="btn-text" *ngIf="!agentService.isCurrentSessionRunning() && agentService.currentSessionRecordingStatus() !== 'processing' && !agentService.currentSessionVideoUrl()">Screen Recording</span>
|
<span class="btn-text" *ngIf="!agentService.isCurrentSessionRunning() && agentService.currentSessionRecordingStatus() !== 'processing' && !agentService.currentSessionVideoUrl() && agentService.hasCurrentSessionStepFrames()">Step Replay</span>
|
||||||
|
<span class="btn-text" *ngIf="!agentService.isCurrentSessionRunning() && agentService.currentSessionRecordingStatus() !== 'processing' && !agentService.currentSessionVideoUrl() && !agentService.hasCurrentSessionStepFrames() && agentService.currentSessionRecordingStatus() === 'failed'">Recording Failed</span>
|
||||||
|
<span class="btn-text" *ngIf="!agentService.isCurrentSessionRunning() && agentService.currentSessionRecordingStatus() !== 'processing' && !agentService.currentSessionVideoUrl() && !agentService.hasCurrentSessionStepFrames() && agentService.currentSessionRecordingStatus() !== 'failed'">Screen Recording</span>
|
||||||
|
|
||||||
<span class="video-pulse-dot red" *ngIf="agentService.isCurrentSessionRunning()"></span>
|
<span class="video-pulse-dot red" *ngIf="agentService.isCurrentSessionRunning()"></span>
|
||||||
|
<span class="steps-count-badge" *ngIf="!agentService.isCurrentSessionRunning() && !agentService.currentSessionVideoUrl() && agentService.hasCurrentSessionStepFrames()">
|
||||||
|
{{ agentService.currentSessionStepFrames().length }}
|
||||||
|
</span>
|
||||||
</button>
|
</button>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
|||||||
@@ -936,6 +936,56 @@ $stream-row-height: 25px;
|
|||||||
box-shadow: 0 1px 3px rgba(15, 23, 42, 0.08);
|
box-shadow: 0 1px 3px rgba(15, 23, 42, 0.08);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
&.has-steps {
|
||||||
|
background: rgba(240, 253, 250, 0.9);
|
||||||
|
border-color: rgba(94, 234, 212, 0.7);
|
||||||
|
box-shadow: 0 2px 6px -1px rgba(13, 148, 136, 0.12), inset 0 1px 1px rgba(255, 255, 255, 0.9);
|
||||||
|
|
||||||
|
.steps-icon {
|
||||||
|
color: #0d9488;
|
||||||
|
}
|
||||||
|
|
||||||
|
.btn-text {
|
||||||
|
color: #0f766e;
|
||||||
|
}
|
||||||
|
|
||||||
|
&:hover:not(.active) {
|
||||||
|
background: rgba(204, 251, 241, 0.95);
|
||||||
|
border-color: rgba(45, 212, 191, 0.85);
|
||||||
|
|
||||||
|
.steps-icon {
|
||||||
|
color: #0f766e;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
&.failed {
|
||||||
|
background: rgba(254, 242, 242, 0.8);
|
||||||
|
border-color: rgba(254, 202, 202, 0.9);
|
||||||
|
|
||||||
|
.error-icon {
|
||||||
|
color: #ef4444;
|
||||||
|
}
|
||||||
|
|
||||||
|
.btn-text {
|
||||||
|
color: #dc2626;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
.steps-count-badge {
|
||||||
|
display: inline-flex;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: center;
|
||||||
|
padding: 1px 5px;
|
||||||
|
border-radius: 10px;
|
||||||
|
background: #0d9488;
|
||||||
|
color: #ffffff;
|
||||||
|
font-size: 10px;
|
||||||
|
font-weight: 700;
|
||||||
|
line-height: 1;
|
||||||
|
margin-left: 2px;
|
||||||
|
}
|
||||||
|
|
||||||
&.active {
|
&.active {
|
||||||
background: rgba(26, 115, 232, 0.95);
|
background: rgba(26, 115, 232, 0.95);
|
||||||
border-color: rgba(26, 115, 232, 0.95);
|
border-color: rgba(26, 115, 232, 0.95);
|
||||||
@@ -949,6 +999,11 @@ $stream-row-height: 25px;
|
|||||||
background: #ffffff;
|
background: #ffffff;
|
||||||
box-shadow: 0 0 6px rgba(255, 255, 255, 0.9);
|
box-shadow: 0 0 6px rgba(255, 255, 255, 0.9);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
.steps-count-badge {
|
||||||
|
background: #ffffff;
|
||||||
|
color: #1a73e8;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -936,6 +936,27 @@ export class AgentStreamComponent implements AfterViewInit {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public getScreenRecordingButtonTitle(): string {
|
||||||
|
if (this.agentService.isCurrentSessionRunning()) {
|
||||||
|
return 'Task is currently running (Recording screen)';
|
||||||
|
}
|
||||||
|
if (this.agentService.currentSessionRecordingStatus() === 'processing') {
|
||||||
|
return 'Preparing screen recording...';
|
||||||
|
}
|
||||||
|
if (this.agentService.currentSessionVideoUrl()) {
|
||||||
|
return 'Play Screen Recording Video';
|
||||||
|
}
|
||||||
|
if (this.agentService.hasCurrentSessionStepFrames()) {
|
||||||
|
return this.agentService.currentSessionRecordingStatus() === 'failed'
|
||||||
|
? 'Video recording failed — Click to replay step-by-step screenshots'
|
||||||
|
: 'Play step-by-step screenshots replay';
|
||||||
|
}
|
||||||
|
if (this.agentService.currentSessionRecordingStatus() === 'failed') {
|
||||||
|
return 'Screen recording generation failed';
|
||||||
|
}
|
||||||
|
return 'Screen Recording';
|
||||||
|
}
|
||||||
|
|
||||||
public isDeviceActionTool(tool: any): boolean {
|
public isDeviceActionTool(tool: any): boolean {
|
||||||
return isDeviceActionTool(tool);
|
return isDeviceActionTool(tool);
|
||||||
}
|
}
|
||||||
|
|||||||
+125
-12
@@ -33,16 +33,38 @@
|
|||||||
<header class="window-header" (mousedown)="startDrag($event)">
|
<header class="window-header" (mousedown)="startDrag($event)">
|
||||||
<div class="header-left">
|
<div class="header-left">
|
||||||
<span class="material-symbols-outlined header-icon">
|
<span class="material-symbols-outlined header-icon">
|
||||||
{{ agentService.recordingPlaybackStatus() === 'live' ? 'sensors' : (agentService.recordingPlaybackStatus() === 'processing' ? 'progress_activity' : 'smart_display') }}
|
{{ agentService.recordingPlaybackStatus() === 'live' ? 'sensors' : (agentService.playerMode() === 'steps' ? 'slideshow' : (agentService.recordingPlaybackStatus() === 'processing' ? 'progress_activity' : 'smart_display')) }}
|
||||||
</span>
|
</span>
|
||||||
<span class="header-title" [title]="agentService.activeVideoTitle()">
|
<span class="header-title" [title]="agentService.activeVideoTitle()">
|
||||||
{{ agentService.recordingPlaybackStatus() === 'live' ? 'Live Device Screen' : (agentService.recordingPlaybackStatus() === 'processing' ? 'Preparing Recording' : (agentService.activeVideoTitle() || 'Screen Recording')) }}
|
{{ agentService.recordingPlaybackStatus() === 'live' ? 'Live Device Screen' : (agentService.playerMode() === 'steps' ? ('Step Replay (' + (totalStepFrames() > 0 ? (activeStepIndex() + 1) + '/' + totalStepFrames() : '0') + ')') : (agentService.recordingPlaybackStatus() === 'processing' ? 'Preparing Recording' : (agentService.activeVideoTitle() || 'Screen Recording'))) }}
|
||||||
</span>
|
</span>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<div class="header-actions" (mousedown)="$event.stopPropagation()">
|
<div class="header-actions" (mousedown)="$event.stopPropagation()">
|
||||||
<!-- Playback Speed Pills (Shown only when playback video is active) -->
|
<!-- Mode Toggle Group: Video vs. Step Replay -->
|
||||||
<div class="speed-pill-group" *ngIf="agentService.recordingPlaybackStatus() === 'ready' && !agentService.isVideoMinimized() && !videoLoadError() && agentService.activeVideoUrl()">
|
<div class="mode-toggle-group" *ngIf="agentService.recordingPlaybackStatus() === 'ready' && agentService.hasCurrentSessionStepFrames() && !agentService.isVideoMinimized()">
|
||||||
|
<button
|
||||||
|
class="mode-btn"
|
||||||
|
[class.active]="agentService.playerMode() === 'video'"
|
||||||
|
(click)="agentService.setPlayerMode('video')"
|
||||||
|
title="Video Recording"
|
||||||
|
>
|
||||||
|
<span class="material-symbols-outlined mode-icon">videocam</span>
|
||||||
|
<span>Video</span>
|
||||||
|
</button>
|
||||||
|
<button
|
||||||
|
class="mode-btn"
|
||||||
|
[class.active]="agentService.playerMode() === 'steps'"
|
||||||
|
(click)="agentService.setPlayerMode('steps')"
|
||||||
|
title="Step Screenshot Replay"
|
||||||
|
>
|
||||||
|
<span class="material-symbols-outlined mode-icon">slideshow</span>
|
||||||
|
<span>Steps</span>
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Playback Speed Pills (Shown when video is active OR when step replay is active) -->
|
||||||
|
<div class="speed-pill-group" *ngIf="!agentService.isVideoMinimized() && (agentService.recordingPlaybackStatus() === 'ready' || agentService.playerMode() === 'steps') && (agentService.activeVideoUrl() || totalStepFrames() > 0)">
|
||||||
<button
|
<button
|
||||||
*ngFor="let speed of speedOptions"
|
*ngFor="let speed of speedOptions"
|
||||||
class="speed-btn"
|
class="speed-btn"
|
||||||
@@ -121,16 +143,107 @@
|
|||||||
</div>
|
</div>
|
||||||
</ng-container>
|
</ng-container>
|
||||||
|
|
||||||
<!-- 2. RECORDING PLAYBACK (AUTOMATICALLY ACTIVE WHEN TEST FINISHED) -->
|
<!-- 2. RECORDING PLAYBACK / STEP REPLAY (AUTOMATICALLY ACTIVE WHEN TEST FINISHED) -->
|
||||||
<ng-container *ngIf="agentService.recordingPlaybackStatus() !== 'live'">
|
<ng-container *ngIf="agentService.recordingPlaybackStatus() !== 'live'">
|
||||||
<!-- Loading State -->
|
<!-- Loading State (when in video mode and processing) -->
|
||||||
<div class="video-loading-state" *ngIf="agentService.recordingPlaybackStatus() === 'processing' || agentService.isVideoLoading()">
|
<div class="video-loading-state" *ngIf="agentService.playerMode() === 'video' && (agentService.recordingPlaybackStatus() === 'processing' || agentService.isVideoLoading())">
|
||||||
<span class="material-symbols-outlined spinning-icon">progress_activity</span>
|
<span class="material-symbols-outlined spinning-icon">progress_activity</span>
|
||||||
<span>{{ agentService.recordingPlaybackMessage() || 'Preparing screen recording...' }}</span>
|
<span>{{ agentService.recordingPlaybackMessage() || 'Preparing screen recording...' }}</span>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<!-- Active Video Playback View -->
|
<!-- Option A: STEP REPLAY VIEW (ACTIVE IN STEP MODE) -->
|
||||||
<ng-container *ngIf="agentService.recordingPlaybackStatus() === 'ready' && !agentService.isVideoLoading() && agentService.activeVideoUrl() as videoUrl">
|
<ng-container *ngIf="agentService.playerMode() === 'steps' && totalStepFrames() > 0">
|
||||||
|
<div class="step-replay-container">
|
||||||
|
<!-- Fallback info banner if recording had failed -->
|
||||||
|
<div class="step-replay-notice" *ngIf="agentService.recordingPlaybackStatus() === 'failed' || videoLoadError()">
|
||||||
|
<span class="material-symbols-outlined notice-icon">info</span>
|
||||||
|
<span>Screen recording unavailable. Replaying step screenshots.</span>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="step-image-viewport">
|
||||||
|
<img
|
||||||
|
class="step-frame-img"
|
||||||
|
[src]="currentStepFrame()?.imageUrl"
|
||||||
|
[alt]="currentStepFrame()?.title"
|
||||||
|
(click)="openImageInNewTab()"
|
||||||
|
title="Click to view image in new tab"
|
||||||
|
/>
|
||||||
|
|
||||||
|
<!-- Step Overlay Badge -->
|
||||||
|
<div class="step-frame-badge" *ngIf="currentStepFrame() as frame">
|
||||||
|
<span class="badge-step-no">Step {{ frame.stepNumber }}</span>
|
||||||
|
<span class="badge-action-desc">{{ frame.actionText || frame.title }}</span>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Step Replay Controls Footer -->
|
||||||
|
<div class="video-controls-footer">
|
||||||
|
<!-- Timeline Scrubber Bar -->
|
||||||
|
<div class="scrubber-bar-container">
|
||||||
|
<input
|
||||||
|
type="range"
|
||||||
|
class="scrubber-range"
|
||||||
|
[min]="0"
|
||||||
|
[max]="totalStepFrames() - 1"
|
||||||
|
[step]="1"
|
||||||
|
[value]="activeStepIndex()"
|
||||||
|
(input)="onStepScrub($event)"
|
||||||
|
/>
|
||||||
|
<div class="scrubber-progress-fill" [style.width.%]="stepProgressPercent()"></div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Bottom Controls Row -->
|
||||||
|
<div class="controls-row">
|
||||||
|
<div class="controls-left">
|
||||||
|
<!-- Play / Pause Button -->
|
||||||
|
<button class="ctrl-btn play-btn" (click)="toggleStepPlay()" [title]="isStepPlaying() ? 'Pause' : 'Play Slideshow'">
|
||||||
|
<span class="material-symbols-outlined">{{ isStepPlaying() ? 'pause' : 'play_arrow' }}</span>
|
||||||
|
</button>
|
||||||
|
|
||||||
|
<!-- Previous Frame Button -->
|
||||||
|
<button class="ctrl-btn" (click)="prevStepFrame()" [disabled]="activeStepIndex() <= 0" title="Previous Frame">
|
||||||
|
<span class="material-symbols-outlined">skip_previous</span>
|
||||||
|
</button>
|
||||||
|
|
||||||
|
<!-- Next Frame Button -->
|
||||||
|
<button class="ctrl-btn" (click)="nextStepFrame()" [disabled]="activeStepIndex() >= totalStepFrames() - 1" title="Next Frame">
|
||||||
|
<span class="material-symbols-outlined">skip_next</span>
|
||||||
|
</button>
|
||||||
|
|
||||||
|
<!-- Step Replay / Reset -->
|
||||||
|
<button class="ctrl-btn" (click)="restartStep()" title="Replay from start">
|
||||||
|
<span class="material-symbols-outlined">replay</span>
|
||||||
|
</button>
|
||||||
|
|
||||||
|
<!-- Frame Counter Display -->
|
||||||
|
<span class="time-display">
|
||||||
|
Frame {{ activeStepIndex() + 1 }} / {{ totalStepFrames() }}
|
||||||
|
</span>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="controls-right">
|
||||||
|
<!-- Loop Toggle -->
|
||||||
|
<button
|
||||||
|
class="ctrl-btn"
|
||||||
|
[class.active]="isLooping()"
|
||||||
|
(click)="toggleLoop()"
|
||||||
|
[title]="isLooping() ? 'Loop enabled' : 'Enable loop'"
|
||||||
|
>
|
||||||
|
<span class="material-symbols-outlined">repeat</span>
|
||||||
|
</button>
|
||||||
|
|
||||||
|
<!-- Open in new tab -->
|
||||||
|
<button class="ctrl-btn" (click)="openImageInNewTab()" title="Open image in new tab">
|
||||||
|
<span class="material-symbols-outlined">open_in_new</span>
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</ng-container>
|
||||||
|
|
||||||
|
<!-- Option B: ACTIVE VIDEO PLAYBACK VIEW -->
|
||||||
|
<ng-container *ngIf="agentService.playerMode() === 'video' && agentService.recordingPlaybackStatus() === 'ready' && !agentService.isVideoLoading() && agentService.activeVideoUrl() as videoUrl">
|
||||||
<div class="video-container" *ngIf="!videoLoadError()">
|
<div class="video-container" *ngIf="!videoLoadError()">
|
||||||
<video
|
<video
|
||||||
#videoRef
|
#videoRef
|
||||||
@@ -146,7 +259,7 @@
|
|||||||
></video>
|
></video>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<!-- Dedicated Bottom Controls Bar (Outside Video: ZERO Obscurity / Zero Overlap) -->
|
<!-- Dedicated Bottom Controls Bar -->
|
||||||
<div class="video-controls-footer" *ngIf="!videoLoadError()">
|
<div class="video-controls-footer" *ngIf="!videoLoadError()">
|
||||||
<!-- Timeline Scrubber Bar -->
|
<!-- Timeline Scrubber Bar -->
|
||||||
<div class="scrubber-bar-container">
|
<div class="scrubber-bar-container">
|
||||||
@@ -206,8 +319,8 @@
|
|||||||
</div>
|
</div>
|
||||||
</ng-container>
|
</ng-container>
|
||||||
|
|
||||||
<!-- Clean, High-Contrast Minimal Empty / Fallback State -->
|
<!-- Option C: EMPTY / FALLBACK STATE (Only when no video AND no step frames) -->
|
||||||
<div class="empty-video-fallback" *ngIf="!agentService.isVideoLoading() && (agentService.recordingPlaybackStatus() === 'failed' || agentService.recordingPlaybackStatus() === 'unavailable' || videoLoadError())">
|
<div class="empty-video-fallback" *ngIf="!agentService.isVideoLoading() && (agentService.playerMode() === 'video' || totalStepFrames() === 0) && (agentService.recordingPlaybackStatus() === 'failed' || agentService.recordingPlaybackStatus() === 'unavailable' || videoLoadError()) && totalStepFrames() === 0">
|
||||||
<div class="fallback-icon-wrap">
|
<div class="fallback-icon-wrap">
|
||||||
<span class="material-symbols-outlined fallback-icon">videocam_off</span>
|
<span class="material-symbols-outlined fallback-icon">videocam_off</span>
|
||||||
</div>
|
</div>
|
||||||
|
|||||||
+134
-2
@@ -176,6 +176,48 @@
|
|||||||
gap: 4px;
|
gap: 4px;
|
||||||
flex-shrink: 0;
|
flex-shrink: 0;
|
||||||
|
|
||||||
|
.mode-toggle-group {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
background: rgba(241, 245, 249, 0.7);
|
||||||
|
backdrop-filter: blur(12px);
|
||||||
|
border: 1px solid rgba(226, 232, 240, 0.85);
|
||||||
|
border-radius: 8px;
|
||||||
|
padding: 2px;
|
||||||
|
margin-right: 4px;
|
||||||
|
gap: 2px;
|
||||||
|
|
||||||
|
.mode-btn {
|
||||||
|
display: inline-flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 3px;
|
||||||
|
border: none;
|
||||||
|
background: transparent;
|
||||||
|
color: #475569;
|
||||||
|
font-size: 11px;
|
||||||
|
font-weight: 600;
|
||||||
|
padding: 2px 7px;
|
||||||
|
border-radius: 6px;
|
||||||
|
cursor: pointer;
|
||||||
|
transition: all 0.15s ease;
|
||||||
|
|
||||||
|
.mode-icon {
|
||||||
|
font-size: 13px;
|
||||||
|
}
|
||||||
|
|
||||||
|
&:hover:not(.active) {
|
||||||
|
color: #0f172a;
|
||||||
|
background: rgba(255, 255, 255, 0.8);
|
||||||
|
}
|
||||||
|
|
||||||
|
&.active {
|
||||||
|
background: #0284c7;
|
||||||
|
color: #ffffff;
|
||||||
|
box-shadow: 0 1px 3px rgba(2, 132, 199, 0.3);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
.speed-pill-group {
|
.speed-pill-group {
|
||||||
display: flex;
|
display: flex;
|
||||||
align-items: center;
|
align-items: center;
|
||||||
@@ -185,8 +227,6 @@
|
|||||||
border-radius: 8px;
|
border-radius: 8px;
|
||||||
padding: 2px;
|
padding: 2px;
|
||||||
margin-right: 4px;
|
margin-right: 4px;
|
||||||
padding: 2px;
|
|
||||||
margin-right: 4px;
|
|
||||||
|
|
||||||
.speed-btn {
|
.speed-btn {
|
||||||
border: none;
|
border: none;
|
||||||
@@ -370,6 +410,98 @@
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
.step-replay-container {
|
||||||
|
display: flex;
|
||||||
|
flex-direction: column;
|
||||||
|
background: #000000;
|
||||||
|
position: relative;
|
||||||
|
width: 100%;
|
||||||
|
overflow: hidden;
|
||||||
|
|
||||||
|
.step-replay-notice {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 6px;
|
||||||
|
background: rgba(254, 243, 199, 0.95);
|
||||||
|
color: #92400e;
|
||||||
|
padding: 6px 12px;
|
||||||
|
font-size: 11.5px;
|
||||||
|
font-weight: 500;
|
||||||
|
border-bottom: 1px solid rgba(245, 158, 11, 0.3);
|
||||||
|
|
||||||
|
.notice-icon {
|
||||||
|
font-size: 14px;
|
||||||
|
color: #d97706;
|
||||||
|
flex-shrink: 0;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
.step-image-viewport {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: center;
|
||||||
|
position: relative;
|
||||||
|
background: #0a0a0c;
|
||||||
|
min-height: 240px;
|
||||||
|
max-height: 520px;
|
||||||
|
cursor: pointer;
|
||||||
|
overflow: hidden;
|
||||||
|
|
||||||
|
.step-frame-img {
|
||||||
|
max-width: 100%;
|
||||||
|
max-height: 520px;
|
||||||
|
width: auto;
|
||||||
|
height: auto;
|
||||||
|
object-fit: contain;
|
||||||
|
display: block;
|
||||||
|
user-select: none;
|
||||||
|
transition: opacity 0.15s ease;
|
||||||
|
|
||||||
|
&:hover {
|
||||||
|
opacity: 0.96;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
.step-frame-badge {
|
||||||
|
position: absolute;
|
||||||
|
bottom: 12px;
|
||||||
|
left: 12px;
|
||||||
|
right: 12px;
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 6px;
|
||||||
|
background: rgba(15, 23, 42, 0.84);
|
||||||
|
backdrop-filter: blur(10px);
|
||||||
|
border: 1px solid rgba(255, 255, 255, 0.18);
|
||||||
|
border-radius: 8px;
|
||||||
|
padding: 6px 10px;
|
||||||
|
color: #ffffff;
|
||||||
|
box-shadow: 0 4px 12px rgba(0, 0, 0, 0.45);
|
||||||
|
pointer-events: none;
|
||||||
|
animation: fadeIn 0.2s ease;
|
||||||
|
|
||||||
|
.badge-step-no {
|
||||||
|
font-size: 11px;
|
||||||
|
font-weight: 700;
|
||||||
|
background: #0284c7;
|
||||||
|
padding: 1px 6px;
|
||||||
|
border-radius: 4px;
|
||||||
|
letter-spacing: 0.2px;
|
||||||
|
flex-shrink: 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
.badge-action-desc {
|
||||||
|
font-size: 11.5px;
|
||||||
|
font-weight: 500;
|
||||||
|
color: #f8fafc;
|
||||||
|
white-space: nowrap;
|
||||||
|
overflow: hidden;
|
||||||
|
text-overflow: ellipsis;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
.video-container {
|
.video-container {
|
||||||
position: relative;
|
position: relative;
|
||||||
width: 100%;
|
width: 100%;
|
||||||
|
|||||||
+120
-2
@@ -22,11 +22,13 @@ import {
|
|||||||
ElementRef,
|
ElementRef,
|
||||||
ViewChild,
|
ViewChild,
|
||||||
HostListener,
|
HostListener,
|
||||||
effect
|
effect,
|
||||||
|
OnDestroy
|
||||||
} from '@angular/core';
|
} from '@angular/core';
|
||||||
import { CommonModule } from '@angular/common';
|
import { CommonModule } from '@angular/common';
|
||||||
import { FormsModule } from '@angular/forms';
|
import { FormsModule } from '@angular/forms';
|
||||||
import { AgentService } from '../../services/agent.service';
|
import { AgentService } from '../../services/agent.service';
|
||||||
|
import { StepReplayFrame } from '../../core/models/stream.model';
|
||||||
|
|
||||||
@Component({
|
@Component({
|
||||||
selector: 'app-floating-video-player',
|
selector: 'app-floating-video-player',
|
||||||
@@ -35,7 +37,7 @@ import { AgentService } from '../../services/agent.service';
|
|||||||
templateUrl: './floating-video-player.component.html',
|
templateUrl: './floating-video-player.component.html',
|
||||||
styleUrl: './floating-video-player.component.scss'
|
styleUrl: './floating-video-player.component.scss'
|
||||||
})
|
})
|
||||||
export class FloatingVideoPlayerComponent {
|
export class FloatingVideoPlayerComponent implements OnDestroy {
|
||||||
public agentService = inject(AgentService);
|
public agentService = inject(AgentService);
|
||||||
|
|
||||||
@ViewChild('videoRef') videoRef?: ElementRef<HTMLVideoElement>;
|
@ViewChild('videoRef') videoRef?: ElementRef<HTMLVideoElement>;
|
||||||
@@ -60,6 +62,24 @@ export class FloatingVideoPlayerComponent {
|
|||||||
private pendingAutoplay = false;
|
private pendingAutoplay = false;
|
||||||
private pendingAbsoluteSeek: number | null = null;
|
private pendingAbsoluteSeek: number | null = null;
|
||||||
|
|
||||||
|
// Step Replay state signals
|
||||||
|
public activeStepIndex = signal<number>(0);
|
||||||
|
public isStepPlaying = signal<boolean>(false);
|
||||||
|
private stepTimer: any = null;
|
||||||
|
|
||||||
|
public stepFrames = computed(() => this.agentService.currentSessionStepFrames());
|
||||||
|
public totalStepFrames = computed(() => this.stepFrames().length);
|
||||||
|
public currentStepFrame = computed<StepReplayFrame | null>(() => {
|
||||||
|
const frames = this.stepFrames();
|
||||||
|
const idx = this.activeStepIndex();
|
||||||
|
return frames[idx] || null;
|
||||||
|
});
|
||||||
|
public stepProgressPercent = computed(() => {
|
||||||
|
const total = this.totalStepFrames();
|
||||||
|
if (total <= 1) return 0;
|
||||||
|
return (this.activeStepIndex() / (total - 1)) * 100;
|
||||||
|
});
|
||||||
|
|
||||||
// Available speed options
|
// Available speed options
|
||||||
public speedOptions = [0.5, 1.0, 1.5, 2.0, 4.0];
|
public speedOptions = [0.5, 1.0, 1.5, 2.0, 4.0];
|
||||||
|
|
||||||
@@ -92,6 +112,16 @@ export class FloatingVideoPlayerComponent {
|
|||||||
{ allowSignalWrites: true }
|
{ allowSignalWrites: true }
|
||||||
);
|
);
|
||||||
|
|
||||||
|
// Reset step playback state when session changes
|
||||||
|
effect(
|
||||||
|
() => {
|
||||||
|
const _ = this.stepFrames();
|
||||||
|
this.activeStepIndex.set(0);
|
||||||
|
this.pauseStepPlay();
|
||||||
|
},
|
||||||
|
{ allowSignalWrites: true }
|
||||||
|
);
|
||||||
|
|
||||||
effect(() => {
|
effect(() => {
|
||||||
const request = this.agentService.videoSeekRequest();
|
const request = this.agentService.videoSeekRequest();
|
||||||
if (!request) return;
|
if (!request) return;
|
||||||
@@ -327,6 +357,9 @@ export class FloatingVideoPlayerComponent {
|
|||||||
if (v) {
|
if (v) {
|
||||||
v.playbackRate = rate;
|
v.playbackRate = rate;
|
||||||
}
|
}
|
||||||
|
if (this.isStepPlaying()) {
|
||||||
|
this.scheduleNextStepFrame();
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
public toggleLoop(): void {
|
public toggleLoop(): void {
|
||||||
@@ -361,4 +394,89 @@ export class FloatingVideoPlayerComponent {
|
|||||||
window.open(url, '_blank');
|
window.open(url, '_blank');
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Step Replay Controls
|
||||||
|
*/
|
||||||
|
public toggleStepPlay(): void {
|
||||||
|
if (this.isStepPlaying()) {
|
||||||
|
this.pauseStepPlay();
|
||||||
|
} else {
|
||||||
|
this.startStepPlay();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public startStepPlay(): void {
|
||||||
|
if (this.totalStepFrames() <= 0) return;
|
||||||
|
if (this.activeStepIndex() >= this.totalStepFrames() - 1) {
|
||||||
|
this.activeStepIndex.set(0);
|
||||||
|
}
|
||||||
|
this.isStepPlaying.set(true);
|
||||||
|
this.scheduleNextStepFrame();
|
||||||
|
}
|
||||||
|
|
||||||
|
public pauseStepPlay(): void {
|
||||||
|
this.isStepPlaying.set(false);
|
||||||
|
if (this.stepTimer) {
|
||||||
|
clearTimeout(this.stepTimer);
|
||||||
|
this.stepTimer = null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private scheduleNextStepFrame(): void {
|
||||||
|
if (this.stepTimer) clearTimeout(this.stepTimer);
|
||||||
|
const delay = Math.max(300, 1500 / this.playbackRate());
|
||||||
|
this.stepTimer = setTimeout(() => {
|
||||||
|
if (!this.isStepPlaying()) return;
|
||||||
|
if (this.activeStepIndex() < this.totalStepFrames() - 1) {
|
||||||
|
this.activeStepIndex.update((i) => i + 1);
|
||||||
|
this.scheduleNextStepFrame();
|
||||||
|
} else {
|
||||||
|
if (this.isLooping()) {
|
||||||
|
this.activeStepIndex.set(0);
|
||||||
|
this.scheduleNextStepFrame();
|
||||||
|
} else {
|
||||||
|
this.pauseStepPlay();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}, delay);
|
||||||
|
}
|
||||||
|
|
||||||
|
public prevStepFrame(): void {
|
||||||
|
this.pauseStepPlay();
|
||||||
|
this.activeStepIndex.update((i) => Math.max(0, i - 1));
|
||||||
|
}
|
||||||
|
|
||||||
|
public nextStepFrame(): void {
|
||||||
|
this.pauseStepPlay();
|
||||||
|
this.activeStepIndex.update((i) => Math.min(this.totalStepFrames() - 1, i + 1));
|
||||||
|
}
|
||||||
|
|
||||||
|
public seekStepFrame(index: number): void {
|
||||||
|
this.activeStepIndex.set(Math.max(0, Math.min(this.totalStepFrames() - 1, index)));
|
||||||
|
}
|
||||||
|
|
||||||
|
public onStepScrub(event: Event): void {
|
||||||
|
const input = event.target as HTMLInputElement;
|
||||||
|
this.seekStepFrame(parseInt(input.value, 10));
|
||||||
|
}
|
||||||
|
|
||||||
|
public restartStep(): void {
|
||||||
|
this.seekStepFrame(0);
|
||||||
|
this.startStepPlay();
|
||||||
|
}
|
||||||
|
|
||||||
|
public openImageInNewTab(): void {
|
||||||
|
const url = this.currentStepFrame()?.imageUrl;
|
||||||
|
if (url) {
|
||||||
|
window.open(url, '_blank');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public ngOnDestroy(): void {
|
||||||
|
if (this.stepTimer) {
|
||||||
|
clearTimeout(this.stepTimer);
|
||||||
|
this.stepTimer = null;
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
@@ -98,3 +98,14 @@ export interface CheckerResult {
|
|||||||
reason: string;
|
reason: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export interface StepReplayFrame {
|
||||||
|
index: number;
|
||||||
|
stepNumber: number;
|
||||||
|
title: string;
|
||||||
|
imageUrl: string;
|
||||||
|
actionText?: string;
|
||||||
|
isPost?: boolean;
|
||||||
|
timestamp?: number;
|
||||||
|
phaseId?: string;
|
||||||
|
}
|
||||||
|
|
||||||
@@ -1,4 +1,4 @@
|
|||||||
import { signal } from '@angular/core';
|
import { signal, computed } from '@angular/core';
|
||||||
import { of } from 'rxjs';
|
import { of } from 'rxjs';
|
||||||
|
|
||||||
import { AgentService } from './agent.service';
|
import { AgentService } from './agent.service';
|
||||||
@@ -198,6 +198,9 @@ describe('AgentService recording finalization lifecycle', () => {
|
|||||||
service.recordingPlaybackStatus = signal('idle');
|
service.recordingPlaybackStatus = signal('idle');
|
||||||
service.recordingPlaybackMessage = signal('');
|
service.recordingPlaybackMessage = signal('');
|
||||||
service.shouldAutoplayVideo = signal(true);
|
service.shouldAutoplayVideo = signal(true);
|
||||||
|
service.playerMode = signal<'video' | 'steps'>('video');
|
||||||
|
(service as any).hasCurrentSessionStepFrames = computed(() => false);
|
||||||
|
(service as any).currentSessionStepFrames = computed(() => []);
|
||||||
return service;
|
return service;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -19,8 +19,9 @@ import { HttpClient } from '@angular/common/http';
|
|||||||
import { Observable } from 'rxjs';
|
import { Observable } from 'rxjs';
|
||||||
|
|
||||||
import { Session, ModelInfo, TaskQueueItem, AgentStatusResponse } from '../core/models/session.model';
|
import { Session, ModelInfo, TaskQueueItem, AgentStatusResponse } from '../core/models/session.model';
|
||||||
import { StepItemData } from '../core/models/stream.model';
|
import { StepItemData, StepReplayFrame } from '../core/models/stream.model';
|
||||||
export type { Session, ModelInfo, TaskQueueItem, AgentStatusResponse, StepItemData };
|
import { extractStepReplayFrames } from '../utils/action-formatter.util';
|
||||||
|
export type { Session, ModelInfo, TaskQueueItem, AgentStatusResponse, StepItemData, StepReplayFrame };
|
||||||
|
|
||||||
const SESSION_CACHE_KEY = 'artemis.sessions.v1';
|
const SESSION_CACHE_KEY = 'artemis.sessions.v1';
|
||||||
|
|
||||||
@@ -158,12 +159,32 @@ export class AgentService {
|
|||||||
public recordingPlaybackMessage = signal<string>('');
|
public recordingPlaybackMessage = signal<string>('');
|
||||||
public shouldAutoplayVideo = signal<boolean>(false);
|
public shouldAutoplayVideo = signal<boolean>(false);
|
||||||
public videoSeekRequest = signal<{ seconds: number; requestId: number } | null>(null);
|
public videoSeekRequest = signal<{ seconds: number; requestId: number } | null>(null);
|
||||||
|
public playerMode = signal<'video' | 'steps'>('video');
|
||||||
private activeVideoSessionId: string | null = null;
|
private activeVideoSessionId: string | null = null;
|
||||||
private videoSeekRequestId = 0;
|
private videoSeekRequestId = 0;
|
||||||
private videoRequestGeneration = 0;
|
private videoRequestGeneration = 0;
|
||||||
private videoRetryTimer: ReturnType<typeof setTimeout> | null = null;
|
private videoRetryTimer: ReturnType<typeof setTimeout> | null = null;
|
||||||
private videoWaitStartedAt = 0;
|
private videoWaitStartedAt = 0;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Computed step replay frames from current session logs
|
||||||
|
*/
|
||||||
|
public currentSessionStepFrames = computed<StepReplayFrame[]>(() => {
|
||||||
|
return extractStepReplayFrames(this.sessionLogs());
|
||||||
|
});
|
||||||
|
|
||||||
|
public hasCurrentSessionStepFrames = computed<boolean>(() => {
|
||||||
|
return this.currentSessionStepFrames().length > 0;
|
||||||
|
});
|
||||||
|
|
||||||
|
public setPlayerMode(mode: 'video' | 'steps'): void {
|
||||||
|
this.playerMode.set(mode);
|
||||||
|
}
|
||||||
|
|
||||||
|
public togglePlayerMode(): void {
|
||||||
|
this.playerMode.update((mode) => (mode === 'video' ? 'steps' : 'video'));
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Clear user-pinned selection so subsequent runs automatically follow active runner
|
* Clear user-pinned selection so subsequent runs automatically follow active runner
|
||||||
*/
|
*/
|
||||||
@@ -1284,6 +1305,13 @@ export class AgentService {
|
|||||||
this.isVideoLoading.set(true);
|
this.isVideoLoading.set(true);
|
||||||
this.recordingPlaybackStatus.set('processing');
|
this.recordingPlaybackStatus.set('processing');
|
||||||
this.recordingPlaybackMessage.set('Loading screen recording...');
|
this.recordingPlaybackMessage.set('Loading screen recording...');
|
||||||
|
if (targetUrl) {
|
||||||
|
this.playerMode?.set('video');
|
||||||
|
} else if (this.hasCurrentSessionStepFrames?.()) {
|
||||||
|
this.playerMode?.set('steps');
|
||||||
|
} else {
|
||||||
|
this.playerMode?.set('video');
|
||||||
|
}
|
||||||
this.requestSessionVideo(targetSessionId, this.videoRequestGeneration);
|
this.requestSessionVideo(targetSessionId, this.videoRequestGeneration);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1341,6 +1369,7 @@ export class AgentService {
|
|||||||
this.recordingPlaybackMessage.set('');
|
this.recordingPlaybackMessage.set('');
|
||||||
this.activeVideoUrl.set(res.video_url);
|
this.activeVideoUrl.set(res.video_url);
|
||||||
this.activeVideoSegments.set(res.video_segments || []);
|
this.activeVideoSegments.set(res.video_segments || []);
|
||||||
|
this.playerMode?.set('video');
|
||||||
this.rawSessions.update((list) =>
|
this.rawSessions.update((list) =>
|
||||||
list.map((s) => s.session_id === sessionId
|
list.map((s) => s.session_id === sessionId
|
||||||
? { ...s, video_url: res.video_url || undefined, recording_status: 'ready' }
|
? { ...s, video_url: res.video_url || undefined, recording_status: 'ready' }
|
||||||
@@ -1368,6 +1397,9 @@ export class AgentService {
|
|||||||
? 'Recording finalization failed.'
|
? 'Recording finalization failed.'
|
||||||
: 'No screen recording is available for this task.')
|
: 'No screen recording is available for this task.')
|
||||||
);
|
);
|
||||||
|
if (this.hasCurrentSessionStepFrames?.()) {
|
||||||
|
this.playerMode?.set('steps');
|
||||||
|
}
|
||||||
},
|
},
|
||||||
error: () => {
|
error: () => {
|
||||||
if (generation !== this.videoRequestGeneration || sessionId !== this.activeVideoSessionId) {
|
if (generation !== this.videoRequestGeneration || sessionId !== this.activeVideoSessionId) {
|
||||||
@@ -1380,6 +1412,9 @@ export class AgentService {
|
|||||||
this.isVideoLoading.set(false);
|
this.isVideoLoading.set(false);
|
||||||
this.recordingPlaybackStatus.set('failed');
|
this.recordingPlaybackStatus.set('failed');
|
||||||
this.recordingPlaybackMessage.set('Unable to load the screen recording.');
|
this.recordingPlaybackMessage.set('Unable to load the screen recording.');
|
||||||
|
if (this.hasCurrentSessionStepFrames?.()) {
|
||||||
|
this.playerMode?.set('steps');
|
||||||
|
}
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
import {
|
import {
|
||||||
getStepPreImageUrl,
|
getStepPreImageUrl,
|
||||||
getStepPostImageUrl
|
getStepPostImageUrl,
|
||||||
|
extractStepReplayFrames
|
||||||
} from './action-formatter.util';
|
} from './action-formatter.util';
|
||||||
|
|
||||||
describe('action-formatter.util screenshot chaining', () => {
|
describe('action-formatter.util screenshot chaining', () => {
|
||||||
@@ -109,4 +110,100 @@ describe('action-formatter.util screenshot chaining', () => {
|
|||||||
expect(preUrl).toBe('/images/pre123');
|
expect(preUrl).toBe('/images/pre123');
|
||||||
expect(postUrl).toBe('/images/post456');
|
expect(postUrl).toBe('/images/post456');
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it('should resolve post screenshot for primary action when generic_tools contains action traces', () => {
|
||||||
|
const stepData = {
|
||||||
|
step_id: 'step-02',
|
||||||
|
pre_image_name: 'pre_screen_hash',
|
||||||
|
post_image_name: 'post_screen_hash',
|
||||||
|
action_taken: {
|
||||||
|
action: 'click',
|
||||||
|
coordinates: [500, 928],
|
||||||
|
args: { target: [500, 928] },
|
||||||
|
timestamp: 1000
|
||||||
|
},
|
||||||
|
generic_tools: [
|
||||||
|
{
|
||||||
|
trace_id: 'trace-click-internal',
|
||||||
|
type: 'action',
|
||||||
|
name: 'click',
|
||||||
|
timestamp: 1001,
|
||||||
|
payload: {
|
||||||
|
args: { target: '[500, 928]', times: '1', delay_ms: '100' },
|
||||||
|
result: {
|
||||||
|
outcome: 'Clicked successfully.',
|
||||||
|
post_image_name: 'post_screen_hash',
|
||||||
|
status: 'success'
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
};
|
||||||
|
|
||||||
|
const preUrl = getStepPreImageUrl(stepData, stepData.action_taken);
|
||||||
|
const postUrl = getStepPostImageUrl(stepData, stepData.action_taken);
|
||||||
|
|
||||||
|
expect(preUrl).toBe('/images/pre_screen_hash');
|
||||||
|
expect(postUrl).toBe('/images/post_screen_hash');
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('extractStepReplayFrames', () => {
|
||||||
|
it('should return an empty array for empty or invalid logs', () => {
|
||||||
|
expect(extractStepReplayFrames([])).toEqual([]);
|
||||||
|
expect(extractStepReplayFrames(null as any)).toEqual([]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('should extract ordered StepReplayFrames and deduplicate consecutive identical images', () => {
|
||||||
|
const logs = [
|
||||||
|
{
|
||||||
|
type: 'step_updated',
|
||||||
|
data: {
|
||||||
|
step_id: 's-1',
|
||||||
|
step_number: 1,
|
||||||
|
pre_image_name: 'img_screen_1',
|
||||||
|
post_image_name: 'img_screen_2',
|
||||||
|
action_taken: { action: 'tap', coordinates: [500, 1000] },
|
||||||
|
timestamp: 100
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
type: 'step_updated',
|
||||||
|
data: {
|
||||||
|
step_id: 's-2',
|
||||||
|
step_number: 2,
|
||||||
|
pre_image_name: 'img_screen_2',
|
||||||
|
post_image_name: 'img_screen_3',
|
||||||
|
action_taken: { action: 'input_text', text: 'hello' },
|
||||||
|
timestamp: 200
|
||||||
|
}
|
||||||
|
}
|
||||||
|
];
|
||||||
|
|
||||||
|
const frames = extractStepReplayFrames(logs);
|
||||||
|
expect(frames.length).toBe(3);
|
||||||
|
expect(frames[0].title).toBe('Step 1 (Before Action)');
|
||||||
|
expect(frames[0].imageUrl).toBe('/images/img_screen_1');
|
||||||
|
expect(frames[1].title).toContain('Step 1');
|
||||||
|
expect(frames[1].imageUrl).toBe('/images/img_screen_2');
|
||||||
|
expect(frames[2].title).toContain('Step 2');
|
||||||
|
expect(frames[2].imageUrl).toBe('/images/img_screen_3');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('should handle terminal step with only pre_image (e.g. cancelled / failed task)', () => {
|
||||||
|
const logs = [
|
||||||
|
{
|
||||||
|
step_number: 1,
|
||||||
|
pre_image_name: 'img_start',
|
||||||
|
post_image_name: null,
|
||||||
|
action_taken: null,
|
||||||
|
timestamp: 100
|
||||||
|
}
|
||||||
|
];
|
||||||
|
|
||||||
|
const frames = extractStepReplayFrames(logs);
|
||||||
|
expect(frames.length).toBe(1);
|
||||||
|
expect(frames[0].imageUrl).toBe('/images/img_start');
|
||||||
|
expect(frames[0].stepNumber).toBe(1);
|
||||||
|
});
|
||||||
});
|
});
|
||||||
@@ -14,7 +14,7 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import { ActionParam } from '../core/models/stream.model';
|
import { ActionParam, StepReplayFrame } from '../core/models/stream.model';
|
||||||
import { extractNumbersFromCoordinateValue, isPureDirectionString, parseSequenceCoordinates } from './image-overlay.util';
|
import { extractNumbersFromCoordinateValue, isPureDirectionString, parseSequenceCoordinates } from './image-overlay.util';
|
||||||
import { cleanErrorMessage } from './tool-formatter.util';
|
import { cleanErrorMessage } from './tool-formatter.util';
|
||||||
|
|
||||||
@@ -698,7 +698,7 @@ export function resolveStepImageMap(stepData: any): Map<any, ResolvedImageEntry>
|
|||||||
if (explicitPost) {
|
if (explicitPost) {
|
||||||
resolvedPost = explicitPost;
|
resolvedPost = explicitPost;
|
||||||
hasExplicitPost = true;
|
hasExplicitPost = true;
|
||||||
} else if (e.isPrimary && !hadPrimaryFailure && !hasSubsequentActions) {
|
} else if (e.isPrimary && !hadPrimaryFailure) {
|
||||||
resolvedPost = stepData.post_image_name || stepData.post_screenshot || null;
|
resolvedPost = stepData.post_image_name || stepData.post_screenshot || null;
|
||||||
} else if (e.item === lastActionItem && !hadPrimaryFailure) {
|
} else if (e.item === lastActionItem && !hadPrimaryFailure) {
|
||||||
resolvedPost = stepData.post_image_name || stepData.post_screenshot || null;
|
resolvedPost = stepData.post_image_name || stepData.post_screenshot || null;
|
||||||
@@ -765,3 +765,86 @@ export function getStepPostImageUrl(stepData: any, actionData?: any): string | n
|
|||||||
}
|
}
|
||||||
return postUrl;
|
return postUrl;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Extract an ordered array of visual StepReplayFrames from session logs or step blocks
|
||||||
|
*/
|
||||||
|
export function extractStepReplayFrames(logsOrSteps: any[]): StepReplayFrame[] {
|
||||||
|
if (!Array.isArray(logsOrSteps) || logsOrSteps.length === 0) {
|
||||||
|
return [];
|
||||||
|
}
|
||||||
|
|
||||||
|
// Extract raw step data objects
|
||||||
|
const rawSteps: any[] = [];
|
||||||
|
for (const item of logsOrSteps) {
|
||||||
|
if (!item) continue;
|
||||||
|
if (item.type === 'step_updated' && item.data) {
|
||||||
|
rawSteps.push(item.data);
|
||||||
|
} else if (item.type === 'step' && item.data) {
|
||||||
|
rawSteps.push(item.data);
|
||||||
|
} else if (item.step_number !== undefined || item.step_id !== undefined) {
|
||||||
|
rawSteps.push(item);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Deduplicate by step_id or step_number, keeping latest
|
||||||
|
const stepMap = new Map<any, any>();
|
||||||
|
for (const step of rawSteps) {
|
||||||
|
const key = step.step_id || step.step_number;
|
||||||
|
if (key !== undefined) {
|
||||||
|
stepMap.set(key, step);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const steps = Array.from(stepMap.values()).sort((a, b) => {
|
||||||
|
const numA = Number(a.step_number ?? 0);
|
||||||
|
const numB = Number(b.step_number ?? 0);
|
||||||
|
if (numA !== numB) return numA - numB;
|
||||||
|
return Number(a.timestamp ?? 0) - Number(b.timestamp ?? 0);
|
||||||
|
});
|
||||||
|
|
||||||
|
const frames: StepReplayFrame[] = [];
|
||||||
|
let lastImageUrl: string | null = null;
|
||||||
|
|
||||||
|
for (const stepData of steps) {
|
||||||
|
const stepNum = Number(stepData.step_number ?? (frames.length + 1));
|
||||||
|
const act = stepData.action_taken;
|
||||||
|
const title = act ? (getActionTitle(act) || 'Action') : 'Step';
|
||||||
|
const coords = act ? getActionCoords(act) : '';
|
||||||
|
const actionDesc = coords ? `${title} (${coords})` : title;
|
||||||
|
|
||||||
|
const preUrl = getStepPreImageUrl(stepData, act);
|
||||||
|
const postUrl = getStepPostImageUrl(stepData, act);
|
||||||
|
|
||||||
|
// If pre-image exists and is new, add it
|
||||||
|
if (preUrl && preUrl !== lastImageUrl) {
|
||||||
|
frames.push({
|
||||||
|
index: frames.length,
|
||||||
|
stepNumber: stepNum,
|
||||||
|
title: `Step ${stepNum} (Before Action)`,
|
||||||
|
imageUrl: preUrl,
|
||||||
|
actionText: `Preparing: ${actionDesc}`,
|
||||||
|
isPost: false,
|
||||||
|
timestamp: stepData.timestamp
|
||||||
|
});
|
||||||
|
lastImageUrl = preUrl;
|
||||||
|
}
|
||||||
|
|
||||||
|
// If post-image exists and is different from current screen, add it
|
||||||
|
if (postUrl && postUrl !== lastImageUrl) {
|
||||||
|
frames.push({
|
||||||
|
index: frames.length,
|
||||||
|
stepNumber: stepNum,
|
||||||
|
title: `Step ${stepNum}: ${actionDesc}`,
|
||||||
|
imageUrl: postUrl,
|
||||||
|
actionText: actionDesc,
|
||||||
|
isPost: true,
|
||||||
|
timestamp: stepData.timestamp
|
||||||
|
});
|
||||||
|
lastImageUrl = postUrl;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return frames;
|
||||||
|
}
|
||||||
|
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
# ROLE & OBJECTIVE
|
# ROLE & OBJECTIVE
|
||||||
You are **ARTEMIS Flash Runner**, an autonomous and highly efficient Android Device Execution Agent. Your goal is to accomplish the user's objective on the device.
|
You are an autonomous and highly efficient Android Device Execution Agent. Your goal is to accomplish the user's objective on the device.
|
||||||
|
|
||||||
**Objective: {{ goal }}**
|
**Objective: {{ goal }}**
|
||||||
|
|
||||||
|
|||||||
@@ -697,7 +697,10 @@ class UnifiedMobileController:
|
|||||||
session.is_active = False
|
session.is_active = False
|
||||||
if session.watchdog_task and not session.watchdog_task.done():
|
if session.watchdog_task and not session.watchdog_task.done():
|
||||||
session.watchdog_task.cancel()
|
session.watchdog_task.cancel()
|
||||||
await session.watchdog_task
|
try:
|
||||||
|
await session.watchdog_task
|
||||||
|
except (asyncio.CancelledError, Exception):
|
||||||
|
pass
|
||||||
|
|
||||||
try:
|
try:
|
||||||
process = session.process
|
process = session.process
|
||||||
@@ -708,7 +711,14 @@ class UnifiedMobileController:
|
|||||||
logger.warning(f"Error terminating scrcpy process: {proc_e}")
|
logger.warning(f"Error terminating scrcpy process: {proc_e}")
|
||||||
|
|
||||||
output_path = session.local_video_path
|
output_path = session.local_video_path
|
||||||
if not output_path or not output_path.exists():
|
has_existing_recording = (
|
||||||
|
(output_path and output_path.exists())
|
||||||
|
or any(
|
||||||
|
Path(r.get("output_path", "")).exists() or Path(r.get("path", "")).exists()
|
||||||
|
for r in session.android_segment_records
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if not has_existing_recording:
|
||||||
message = "Recording file not found on disk"
|
message = "Recording file not found on disk"
|
||||||
self._record_recording_failure(session, message)
|
self._record_recording_failure(session, message)
|
||||||
remove_active_session(device_id)
|
remove_active_session(device_id)
|
||||||
@@ -726,31 +736,37 @@ class UnifiedMobileController:
|
|||||||
self._remux_segment_record(record)
|
self._remux_segment_record(record)
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
conversion_results = []
|
|
||||||
if session.android_conversion_tasks:
|
if session.android_conversion_tasks:
|
||||||
conversion_results = await asyncio.gather(
|
await asyncio.gather(
|
||||||
*session.android_conversion_tasks, return_exceptions=True
|
*session.android_conversion_tasks, return_exceptions=True
|
||||||
)
|
)
|
||||||
mp4_paths = [
|
mp4_paths = [
|
||||||
Path(record["output_path"])
|
Path(record["output_path"])
|
||||||
for record in session.android_segment_records
|
for record in session.android_segment_records
|
||||||
if Path(record["output_path"]).exists()
|
if Path(record["output_path"]).exists() and Path(record["output_path"]).stat().st_size > 0
|
||||||
]
|
]
|
||||||
manifest_path = await write_recording_manifest(output_path.parent, mp4_paths)
|
|
||||||
|
|
||||||
conversion_failed = (
|
# Emergency fallback: if no segment MP4 was produced, attempt to remux local_video_path directly
|
||||||
len(mp4_paths) != len(session.android_segment_records)
|
if not mp4_paths and output_path and output_path.exists() and output_path.stat().st_size > 0:
|
||||||
or any(result is not True for result in conversion_results)
|
fallback_mp4 = (
|
||||||
or manifest_path is None
|
output_path.parent / "recording.mp4"
|
||||||
)
|
if output_path.name != "recording.mp4"
|
||||||
if conversion_failed:
|
else output_path.with_name("recording_converted.mp4")
|
||||||
|
)
|
||||||
|
if await remux_recording_to_mp4(output_path, fallback_mp4):
|
||||||
|
mp4_paths.append(fallback_mp4)
|
||||||
|
|
||||||
|
if not mp4_paths:
|
||||||
message = "Recording finalization failed; no complete browser-safe video was produced"
|
message = "Recording finalization failed; no complete browser-safe video was produced"
|
||||||
self._record_recording_failure(session, message)
|
self._record_recording_failure(session, message)
|
||||||
remove_active_session(device_id)
|
remove_active_session(device_id)
|
||||||
return VideoRecordingResult(success=False, message=message)
|
return VideoRecordingResult(success=False, message=message)
|
||||||
|
|
||||||
remove_active_session(device_id)
|
|
||||||
final_video_path = mp4_paths[0]
|
final_video_path = mp4_paths[0]
|
||||||
|
output_dir = final_video_path.parent
|
||||||
|
manifest_path = await write_recording_manifest(output_dir, mp4_paths)
|
||||||
|
|
||||||
|
remove_active_session(device_id)
|
||||||
|
|
||||||
# Persist update to local database if Data Engine is active
|
# Persist update to local database if Data Engine is active
|
||||||
if self.ctx and self.ctx.data_engine:
|
if self.ctx and self.ctx.data_engine:
|
||||||
@@ -764,12 +780,12 @@ class UnifiedMobileController:
|
|||||||
|
|
||||||
return VideoRecordingResult(
|
return VideoRecordingResult(
|
||||||
success=True,
|
success=True,
|
||||||
message=f"Recording stopped, saved {len(mp4_paths)} orientation-safe segments",
|
message=f"Recording stopped, saved {len(mp4_paths)} video segments",
|
||||||
video_path=mp4_paths[0],
|
video_path=final_video_path,
|
||||||
video_id=session.video_id,
|
video_id=session.video_id,
|
||||||
generation=session.generation,
|
generation=session.generation,
|
||||||
sealed_until=session.sealed_until,
|
sealed_until=session.sealed_until,
|
||||||
source_revision=f"{session.video_id}:{session.generation}:sealed",
|
source_revision=f"{session.video_id}:{session.generation}:ready",
|
||||||
)
|
)
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
|
|||||||
@@ -29,6 +29,7 @@ from artemis.sdk.types.task import AgentProfile
|
|||||||
from artemis.utils.cli_helpers import display_device_status
|
from artemis.utils.cli_helpers import display_device_status
|
||||||
from artemis.utils.logger import get_logger
|
from artemis.utils.logger import get_logger
|
||||||
from artemis.utils.video import check_ffmpeg_available
|
from artemis.utils.video import check_ffmpeg_available
|
||||||
|
import signal
|
||||||
from rich.console import Console
|
from rich.console import Console
|
||||||
from rich.panel import Panel
|
from rich.panel import Panel
|
||||||
import typer
|
import typer
|
||||||
@@ -292,6 +293,15 @@ def run_command(
|
|||||||
display_device_status(console, adb_client=adb_client)
|
display_device_status(console, adb_client=adb_client)
|
||||||
|
|
||||||
cancelled = False
|
cancelled = False
|
||||||
|
original_sigterm = None
|
||||||
|
try:
|
||||||
|
if hasattr(signal, "SIGTERM"):
|
||||||
|
original_sigterm = signal.signal(
|
||||||
|
signal.SIGTERM, lambda s, f: (_ for _ in ()).throw(KeyboardInterrupt())
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
try:
|
try:
|
||||||
asyncio.run(
|
asyncio.run(
|
||||||
execute_task(
|
execute_task(
|
||||||
@@ -335,5 +345,10 @@ def run_command(
|
|||||||
console.print(f"[bold red]Task execution failed:[/bold red] {e}")
|
console.print(f"[bold red]Task execution failed:[/bold red] {e}")
|
||||||
raise
|
raise
|
||||||
finally:
|
finally:
|
||||||
|
try:
|
||||||
|
if original_sigterm is not None and hasattr(signal, "SIGTERM"):
|
||||||
|
signal.signal(signal.SIGTERM, original_sigterm)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
if cancelled:
|
if cancelled:
|
||||||
raise SystemExit(130)
|
raise SystemExit(130)
|
||||||
@@ -19,6 +19,7 @@ from pathlib import Path
|
|||||||
import shutil
|
import shutil
|
||||||
import signal
|
import signal
|
||||||
import tempfile
|
import tempfile
|
||||||
|
import time
|
||||||
|
|
||||||
from artemis.platform.base import IPlatform, IPlatformPaths, IPlatformProcess, OSType
|
from artemis.platform.base import IPlatform, IPlatformPaths, IPlatformProcess, OSType
|
||||||
|
|
||||||
@@ -98,12 +99,47 @@ class DarwinPlatformProcess(IPlatformProcess):
|
|||||||
pass
|
pass
|
||||||
|
|
||||||
parent.send_signal(signal.SIGTERM)
|
parent.send_signal(signal.SIGTERM)
|
||||||
gone, alive = psutil.wait_procs(children + [parent], timeout=timeout_seconds)
|
|
||||||
for p in alive:
|
# Avoid calling wait_procs on direct children of the current process.
|
||||||
try:
|
# In POSIX, psutil.Process.wait() on a child of os.getpid() invokes
|
||||||
p.kill()
|
# os.waitpid(), which reaps the zombie and prevents asyncio / uvloop
|
||||||
except (psutil.NoSuchProcess, psutil.AccessDenied):
|
# event loops from detecting that the child exited, causing proc.wait()
|
||||||
pass
|
# to hang indefinitely.
|
||||||
|
current_pid = os.getpid()
|
||||||
|
external_procs = [
|
||||||
|
p
|
||||||
|
for p in children + [parent]
|
||||||
|
if p.pid != current_pid and getattr(p, "ppid", lambda: None)() != current_pid
|
||||||
|
]
|
||||||
|
direct_children = [
|
||||||
|
p
|
||||||
|
for p in children + [parent]
|
||||||
|
if getattr(p, "ppid", lambda: None)() == current_pid
|
||||||
|
]
|
||||||
|
|
||||||
|
if external_procs:
|
||||||
|
gone, alive = psutil.wait_procs(external_procs, timeout=timeout_seconds)
|
||||||
|
for p in alive:
|
||||||
|
try:
|
||||||
|
p.kill()
|
||||||
|
except (psutil.NoSuchProcess, psutil.AccessDenied):
|
||||||
|
pass
|
||||||
|
|
||||||
|
deadline = time.time() + timeout_seconds
|
||||||
|
for p in direct_children:
|
||||||
|
while time.time() < deadline:
|
||||||
|
try:
|
||||||
|
if not p.is_running() or p.status() == psutil.STATUS_ZOMBIE:
|
||||||
|
break
|
||||||
|
except (psutil.NoSuchProcess, psutil.AccessDenied):
|
||||||
|
break
|
||||||
|
time.sleep(0.05)
|
||||||
|
else:
|
||||||
|
try:
|
||||||
|
p.kill()
|
||||||
|
except (psutil.NoSuchProcess, psutil.AccessDenied):
|
||||||
|
pass
|
||||||
|
|
||||||
return True
|
return True
|
||||||
except Exception:
|
except Exception:
|
||||||
try:
|
try:
|
||||||
|
|||||||
@@ -19,6 +19,7 @@ from pathlib import Path
|
|||||||
import shutil
|
import shutil
|
||||||
import signal
|
import signal
|
||||||
import tempfile
|
import tempfile
|
||||||
|
import time
|
||||||
|
|
||||||
from artemis.platform.base import IPlatform, IPlatformPaths, IPlatformProcess, OSType
|
from artemis.platform.base import IPlatform, IPlatformPaths, IPlatformProcess, OSType
|
||||||
|
|
||||||
@@ -113,12 +114,47 @@ class LinuxPlatformProcess(IPlatformProcess):
|
|||||||
pass
|
pass
|
||||||
|
|
||||||
parent.send_signal(signal.SIGTERM)
|
parent.send_signal(signal.SIGTERM)
|
||||||
gone, alive = psutil.wait_procs(children + [parent], timeout=timeout_seconds)
|
|
||||||
for p in alive:
|
# Avoid calling wait_procs on direct children of the current process.
|
||||||
try:
|
# In POSIX, psutil.Process.wait() on a child of os.getpid() invokes
|
||||||
p.kill()
|
# os.waitpid(), which reaps the zombie and prevents asyncio / uvloop
|
||||||
except (psutil.NoSuchProcess, psutil.AccessDenied):
|
# event loops from detecting that the child exited, causing proc.wait()
|
||||||
pass
|
# to hang indefinitely.
|
||||||
|
current_pid = os.getpid()
|
||||||
|
external_procs = [
|
||||||
|
p
|
||||||
|
for p in children + [parent]
|
||||||
|
if p.pid != current_pid and getattr(p, "ppid", lambda: None)() != current_pid
|
||||||
|
]
|
||||||
|
direct_children = [
|
||||||
|
p
|
||||||
|
for p in children + [parent]
|
||||||
|
if getattr(p, "ppid", lambda: None)() == current_pid
|
||||||
|
]
|
||||||
|
|
||||||
|
if external_procs:
|
||||||
|
gone, alive = psutil.wait_procs(external_procs, timeout=timeout_seconds)
|
||||||
|
for p in alive:
|
||||||
|
try:
|
||||||
|
p.kill()
|
||||||
|
except (psutil.NoSuchProcess, psutil.AccessDenied):
|
||||||
|
pass
|
||||||
|
|
||||||
|
deadline = time.time() + timeout_seconds
|
||||||
|
for p in direct_children:
|
||||||
|
while time.time() < deadline:
|
||||||
|
try:
|
||||||
|
if not p.is_running() or p.status() == psutil.STATUS_ZOMBIE:
|
||||||
|
break
|
||||||
|
except (psutil.NoSuchProcess, psutil.AccessDenied):
|
||||||
|
break
|
||||||
|
time.sleep(0.05)
|
||||||
|
else:
|
||||||
|
try:
|
||||||
|
p.kill()
|
||||||
|
except (psutil.NoSuchProcess, psutil.AccessDenied):
|
||||||
|
pass
|
||||||
|
|
||||||
return True
|
return True
|
||||||
except Exception:
|
except Exception:
|
||||||
try:
|
try:
|
||||||
|
|||||||
+27
-16
@@ -645,23 +645,34 @@ class Agent:
|
|||||||
return output
|
return output
|
||||||
finally:
|
finally:
|
||||||
if recording_started:
|
if recording_started:
|
||||||
try:
|
async def _safe_stop_recording():
|
||||||
logger.info(f"[{task_name}] Stopping automated screen recording...")
|
try:
|
||||||
|
logger.info(f"[{task_name}] Stopping automated screen recording...")
|
||||||
|
controller = get_controller(context)
|
||||||
|
return await controller.stop_video_recording()
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"[{task_name}] Error stopping screen recording: {e}")
|
||||||
|
return None
|
||||||
|
|
||||||
controller = get_controller(context)
|
stop_task = asyncio.create_task(_safe_stop_recording())
|
||||||
stop_result = await controller.stop_video_recording()
|
try:
|
||||||
if stop_result.success and stop_result.video_path:
|
stop_result = await asyncio.shield(stop_task)
|
||||||
logger.info(
|
except (asyncio.CancelledError, BaseException):
|
||||||
f"[{task_name}] Screen recording saved"
|
try:
|
||||||
f" to: {stop_result.video_path}"
|
stop_result = await asyncio.wait_for(stop_task, timeout=15.0)
|
||||||
)
|
except Exception:
|
||||||
else:
|
stop_result = None
|
||||||
logger.warning(
|
|
||||||
f"[{task_name}] Failed to save screen"
|
if stop_result and stop_result.success and stop_result.video_path:
|
||||||
f" recording: {stop_result.message}"
|
logger.info(
|
||||||
)
|
f"[{task_name}] Screen recording saved"
|
||||||
except Exception as e:
|
f" to: {stop_result.video_path}"
|
||||||
logger.error(f"[{task_name}] Error stopping screen recording: {e}")
|
)
|
||||||
|
elif stop_result:
|
||||||
|
logger.warning(
|
||||||
|
f"[{task_name}] Failed to save screen"
|
||||||
|
f" recording: {stop_result.message}"
|
||||||
|
)
|
||||||
except asyncio.CancelledError:
|
except asyncio.CancelledError:
|
||||||
err = f"[{task_name}] Task cancelled"
|
err = f"[{task_name}] Task cancelled"
|
||||||
logger.warning(err)
|
logger.warning(err)
|
||||||
|
|||||||
+46
-9
@@ -297,6 +297,36 @@ async def remux_recording_to_mp4(source_path: Path, output_path: Path) -> bool:
|
|||||||
if valid:
|
if valid:
|
||||||
temporary_path.replace(output_path)
|
temporary_path.replace(output_path)
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
logger.warning(
|
||||||
|
f"Direct remux failed for {source_path} (code {process.returncode}), "
|
||||||
|
"attempting transcode fallback with ffmpeg..."
|
||||||
|
)
|
||||||
|
# Fallback to ultrafast re-encoding in case MKV container was truncated or has timestamp irregularities
|
||||||
|
fallback_proc = await asyncio.create_subprocess_exec(
|
||||||
|
get_ffmpeg_path(),
|
||||||
|
"-y",
|
||||||
|
"-fflags",
|
||||||
|
"+genpts+discardcorrupt",
|
||||||
|
"-i",
|
||||||
|
str(source_path),
|
||||||
|
"-c:v",
|
||||||
|
"libx264",
|
||||||
|
"-preset",
|
||||||
|
"ultrafast",
|
||||||
|
"-pix_fmt",
|
||||||
|
"yuv420p",
|
||||||
|
"-movflags",
|
||||||
|
"+faststart",
|
||||||
|
str(temporary_path),
|
||||||
|
stdout=asyncio.subprocess.PIPE,
|
||||||
|
stderr=asyncio.subprocess.PIPE,
|
||||||
|
)
|
||||||
|
_fout, _ferr = await fallback_proc.communicate()
|
||||||
|
if fallback_proc.returncode == 0 and temporary_path.exists() and temporary_path.stat().st_size > 0:
|
||||||
|
temporary_path.replace(output_path)
|
||||||
|
return True
|
||||||
|
|
||||||
logger.error(
|
logger.error(
|
||||||
f"Failed to remux recording segment (code {process.returncode}): "
|
f"Failed to remux recording segment (code {process.returncode}): "
|
||||||
f"{stderr.decode(errors='replace')[-2000:]}"
|
f"{stderr.decode(errors='replace')[-2000:]}"
|
||||||
@@ -314,10 +344,8 @@ async def probe_video_segment(video_path: Path) -> dict[str, float | int]:
|
|||||||
get_ffprobe_path(),
|
get_ffprobe_path(),
|
||||||
"-v",
|
"-v",
|
||||||
"error",
|
"error",
|
||||||
"-select_streams",
|
|
||||||
"v:0",
|
|
||||||
"-show_entries",
|
"-show_entries",
|
||||||
"stream=width,height:format=duration",
|
"stream=width,height,duration,codec_type:format=duration",
|
||||||
"-of",
|
"-of",
|
||||||
"json",
|
"json",
|
||||||
str(video_path),
|
str(video_path),
|
||||||
@@ -329,11 +357,20 @@ async def probe_video_segment(video_path: Path) -> dict[str, float | int]:
|
|||||||
return {}
|
return {}
|
||||||
try:
|
try:
|
||||||
payload = json.loads(stdout)
|
payload = json.loads(stdout)
|
||||||
stream = (payload.get("streams") or [{}])[0]
|
streams = payload.get("streams") or []
|
||||||
|
video_stream = next(
|
||||||
|
(s for s in streams if s.get("width") and s.get("height")),
|
||||||
|
next((s for s in streams if s.get("codec_type") == "video"), streams[0] if streams else {})
|
||||||
|
)
|
||||||
|
duration = float(
|
||||||
|
(payload.get("format") or {}).get("duration")
|
||||||
|
or video_stream.get("duration")
|
||||||
|
or 0
|
||||||
|
)
|
||||||
return {
|
return {
|
||||||
"duration": float((payload.get("format") or {}).get("duration") or 0),
|
"duration": duration,
|
||||||
"width": int(stream.get("width") or 0),
|
"width": int(video_stream.get("width") or 0),
|
||||||
"height": int(stream.get("height") or 0),
|
"height": int(video_stream.get("height") or 0),
|
||||||
}
|
}
|
||||||
except (TypeError, ValueError, json.JSONDecodeError):
|
except (TypeError, ValueError, json.JSONDecodeError):
|
||||||
return {}
|
return {}
|
||||||
@@ -380,8 +417,8 @@ async def write_recording_manifest(output_dir: Path, mp4_paths: list[Path]) -> P
|
|||||||
width = int(metadata.get("width", 0))
|
width = int(metadata.get("width", 0))
|
||||||
height = int(metadata.get("height", 0))
|
height = int(metadata.get("height", 0))
|
||||||
if duration <= 0 or width <= 0 or height <= 0:
|
if duration <= 0 or width <= 0 or height <= 0:
|
||||||
logger.error(f"Recording segment failed validation: {path}")
|
logger.warning(f"Recording segment skipped due to invalid metadata: {path}")
|
||||||
return None
|
continue
|
||||||
segments.append(
|
segments.append(
|
||||||
{
|
{
|
||||||
"file": path.name,
|
"file": path.name,
|
||||||
|
|||||||
@@ -553,3 +553,54 @@ async def test_immediate_cancel_ignores_stale_ipc_and_runs_next_task():
|
|||||||
await task
|
await task
|
||||||
except (asyncio.CancelledError, Exception):
|
except (asyncio.CancelledError, Exception):
|
||||||
pass
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_wait_for_worker_process_watchdog_handles_reaped_process():
|
||||||
|
"""Verify that _wait_for_worker_process never hangs even if the process was already reaped."""
|
||||||
|
mock_proc = MagicMock()
|
||||||
|
mock_proc.pid = 999999
|
||||||
|
mock_proc.returncode = None
|
||||||
|
|
||||||
|
async def hanging_wait():
|
||||||
|
await asyncio.sleep(10)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
mock_proc.wait = AsyncMock(side_effect=hanging_wait)
|
||||||
|
|
||||||
|
with patch("psutil.Process") as mock_psutil_proc:
|
||||||
|
mock_p = MagicMock()
|
||||||
|
mock_p.is_running.return_value = False
|
||||||
|
mock_psutil_proc.return_value = mock_p
|
||||||
|
|
||||||
|
rc = await TaskQueueService._wait_for_worker_process(mock_proc)
|
||||||
|
assert rc == -15
|
||||||
|
|
||||||
|
|
||||||
|
def test_darwin_terminate_process_tree_preserves_direct_child_for_asyncio():
|
||||||
|
"""Verify darwin terminate_process_tree does not pass direct children to psutil.wait_procs."""
|
||||||
|
import os
|
||||||
|
from artemis.platform.darwin import DarwinPlatformProcess
|
||||||
|
|
||||||
|
process = DarwinPlatformProcess()
|
||||||
|
current_pid = os.getpid()
|
||||||
|
|
||||||
|
with (
|
||||||
|
patch("psutil.Process") as mock_psutil_proc,
|
||||||
|
patch("psutil.wait_procs") as mock_wait_procs,
|
||||||
|
):
|
||||||
|
parent = MagicMock()
|
||||||
|
parent.pid = 12345
|
||||||
|
parent.ppid.return_value = current_pid
|
||||||
|
parent.children.return_value = []
|
||||||
|
parent.is_running.return_value = False
|
||||||
|
|
||||||
|
mock_psutil_proc.return_value = parent
|
||||||
|
mock_wait_procs.return_value = ([], [])
|
||||||
|
|
||||||
|
success = process.terminate_process_tree(12345, timeout_seconds=0.1)
|
||||||
|
assert success is True
|
||||||
|
parent.send_signal.assert_called_once()
|
||||||
|
for call_args in mock_wait_procs.call_args_list:
|
||||||
|
procs_waited = call_args[0][0]
|
||||||
|
assert parent not in procs_waited
|
||||||
Reference in new issue
Block a user