Compare commits

...
3 Commits
25 changed files with 573 additions and 38 deletions

No files matched your search

+12 -2
View File
@@ -21,7 +21,7 @@
| 探索、示范、用例提炼与复跑 | [探索说明](docs/exploration-mode.md) | | 探索、示范、用例提炼与复跑 | [探索说明](docs/exploration-mode.md) |
| 插件协议、适配或市场协同 | [插件开发规范](docs/plugin-development.md)及插件仓库自身的 AGENTS.md、DESIGN.md | | 插件协议、适配或市场协同 | [插件开发规范](docs/plugin-development.md)及插件仓库自身的 AGENTS.md、DESIGN.md |
| 远程 Python SDK | [SDK 设计](packages/artemis-client/DESIGN.md),保持标准库薄客户端边界 | | 远程 Python SDK | [SDK 设计](packages/artemis-client/DESIGN.md),保持标准库薄客户端边界 |
| 继续历史开发、确认版本或发布 | [迁入时的开发快照](docs/development-handoff.md)、[2.1.0 发行说明](docs/releases/2.1.0.md),再核对实时 Git 状态;`docs/plans/` 保留各自阶段的状态 | | 继续历史开发、确认版本或发布 | [README 当前发行入口](README.md)、对应发行说明及实时 Git 状态;[开发快照](docs/development-handoff.md)与 `docs/plans/` 仅作为历史线索,不替代当前源码与发布包核验 |
## Quick Commands ## Quick Commands
@@ -60,7 +60,7 @@ Windows 首次安装使用 `start.bat`,国内镜像使用 `start-cn.bat`。启
## Conventions ## Conventions
- Python 使用绝对导入、4 空格缩进和双引号,由 Ruff 检查;不要放宽质量基线来掩盖新增问题。 - Python 使用绝对导入、4 空格缩进和双引号,由 Ruff 检查;不要放宽质量基线来掩盖新增问题。
- 新增可调参数集中进入配置层,并更新环境示例、网页/MCP 入口和必要文档;默认值同步 `config/artemis.jsonc` 与 `artemis/resources/config/artemis.jsonc`。 - 新增可调参数集中进入配置层,按作用域更新环境示例、网页/MCP 入口和文档;明确仅供环境配置的参数不增加前端入口。JSON 默认值同步 `config/artemis.jsonc` 与 `artemis/resources/config/artemis.jsonc`。
- MCP 工具增删时同步工具注册、CLI 安装白名单、测试、Skill 和能力文档。 - MCP 工具增删时同步工具注册、CLI 安装白名单、测试、Skill 和能力文档。
- 同步 Codex MCP/Skill 时保留已有工具白名单、禁用状态和审批策略;异常时保留原配置并告警,不生成非法 TOML 或阻断网页启动。 - 同步 Codex MCP/Skill 时保留已有工具白名单、禁用状态和审批策略;异常时保留原配置并告警,不生成非法 TOML 或阻断网页启动。
- Windows PowerShell 启动脚本保持兼容 Windows PowerShell 5.1,避免编码导致解析失败。升级只做 Git 快进,保护已有配置和本地修改。 - Windows PowerShell 启动脚本保持兼容 Windows PowerShell 5.1,避免编码导致解析失败。升级只做 Git 快进,保护已有配置和本地修改。
@@ -75,6 +75,16 @@ Windows 首次安装使用 `start.bat`,国内镜像使用 `start-cn.bat`。启
- 设备验证沿用用户指定设备;多设备且未指定时先确认。编写应用交互测试前,先通过 ARTEMIS 探索实际页面和操作路径。 - 设备验证沿用用户指定设备;多设备且未指定时先确认。编写应用交互测试前,先通过 ARTEMIS 探索实际页面和操作路径。
- 提交前检查 `git diff --check` 和暂存范围,只暂存本次具体路径;报告实际执行的验证及未完成项,不将未运行的检查写成通过。 - 提交前检查 `git diff --check` 和暂存范围,只暂存本次具体路径;报告实际执行的验证及未完成项,不将未运行的检查写成通过。
## Release Gates
- 主程序、SDK、插件和第三方依赖独立版本化。主程序升版只修改 `pyproject.toml` 的项目版本、`uv.lock` 的 artemis 条目及前端应用的根版本字段,发行文档另行更新。禁止对仓库或整个锁文件全局替换版本字符串。
- 单独审查锁文件差异。第三方条目的版本、下载地址、完整性校验和依赖声明必须相互匹配;主程序升版不应改变这些条目。真正升级依赖时由包管理器生成并核对上游声明;发生 EINTEGRITY 先核验来源,禁止关闭校验或只照错误提示改哈希。
- 发布前必须在隔离目录解压最终 ZIP,使用独立空 npm 缓存完成 `npm ci` 和生产构建,覆盖本次发布的官方/国内安装路径。包内安装或构建输入变化后必须重跑;不得清空用户全局缓存或改动用户运行目录来制造干净环境。
- 记录实际 Node/npm 版本、镜像、源码提交、ZIP SHA256 和验证范围。已有 node_modules、构建指纹、缓存命中或大量单元测试通过,均不能证明全新安装成功;某个 Node 版本通过也不能冒充其他版本实测。测试和构建命令见前端局部指南。
- 发布顺序:固定提交并打包 → 验证最终包 → 推送标签 → 创建草稿并上传包及校验文件 → 回读校验 → 正式发布 → 无认证下载核验。标签必须指向打包提交;推送标签后检查 Release 状态,本仓库曾出现 Gitea 将已有草稿自动发布的情况。失败时修复并重新验证,不提前宣布完成。
- 候选包与正式包目录明确区分。候选包失效后移出可分发路径并标明替代版本,不能留下同名旧 ZIP 供用户误取;对外只给已核验的正式发布链接或准确文件路径。正式版本不覆盖原标签或附件,后续修复发补丁版。
- 安装或市场版本不符时,先核对用户实际 ZIP 的 SHA256、运行源码提交及生效市场地址;不凭界面版本号判断,也不把未经验证的缓存假设当成原因。主程序与插件版本、索引兼容性分别验证。
## Boundaries ## Boundaries
- 始终保护用户任务、历史库、环境配置和凭据。不要把个人设备序列号、绝对目录或 API Key 写入源码。 - 始终保护用户任务、历史库、环境配置和凭据。不要把个人设备序列号、绝对目录或 API Key 写入源码。
+1 -1
View File
@@ -2,7 +2,7 @@
基于 [ARTEMIS](https://github.com/google/artemis) 扩展的 Android AI 自动化测试平台。用自然语言描述任务,即可通过简体中文网页或 MCP 驱动真机、模拟器执行操作并收集结果。模型统一使用 **OpenAI Responses API**。 基于 [ARTEMIS](https://github.com/google/artemis) 扩展的 Android AI 自动化测试平台。用自然语言描述任务,即可通过简体中文网页或 MCP 驱动真机、模拟器执行操作并收集结果。模型统一使用 **OpenAI Responses API**。
**2.4.1**:修复全新安装时的 npm 校验失败,准确检查 Node.js 版本要求。人工评审插件继续使用 1.1.0,市场使用兼容旧宿主的独立索引。见 [2.4.1 发行说明](docs/releases/2.4.1.md);功能更新见 [2.4.0](docs/releases/2.4.0.md)。 **2.4.3**:异常模型输出耗尽重试后只失败当前任务,避免触发全局暂停;补充请求关联和有界参数摘要,便于定位流式异常。见 [2.4.3 发行说明](docs/releases/2.4.3.md)。人工评审插件保持 1.1.0;列表优化见 [2.4.2](docs/releases/2.4.2.md),安装修复见 [2.4.1](docs/releases/2.4.1.md)。
## 主要功能 ## 主要功能
+3 -1
View File
@@ -12,7 +12,7 @@ from apps.admin_console.services.exploration_listing import EXPLORATION_RUNS_CTE
from apps.admin_console.services.automation_store import identifier, store from apps.admin_console.services.automation_store import identifier, store
from apps.admin_console.services import automation_service as service from apps.admin_console.services import automation_service as service
from apps.admin_console.services import batch_archive as archive_service from apps.admin_console.services import batch_archive as archive_service
from apps.admin_console.services import report_export, task_import from apps.admin_console.services import report_export, task_extensions, task_import
from artemis.config.settings import settings from artemis.config.settings import settings
from apps.admin_console.services.case_review import read_reviewed_result from apps.admin_console.services.case_review import read_reviewed_result
@@ -131,7 +131,9 @@ def batch_by_request(request_id: str):
@router.get("/batches/{batch_id}") @router.get("/batches/{batch_id}")
def batch_detail(batch_id: str): def batch_detail(batch_id: str):
jobs = store.jobs(batch_id) jobs = store.jobs(batch_id)
reviewed = task_extensions.reviewed_batch_sessions(batch_id, store)
for job in jobs: for job in jobs:
job["human_reviewed"] = job["session_id"] in reviewed
job["test_result"] = read_reviewed_result( job["test_result"] = read_reviewed_result(
job["session_id"], job["status"], job.get("error") or "", storage=store job["session_id"], job["status"], job.get("error") or "", storage=store
) )
@@ -36,6 +36,32 @@ def installation_signature() -> str:
) )
def reviewed_batch_sessions(batch_id: str, store: AutomationStore) -> set[str]:
"""Read current human review values once per batch, without loading history."""
manifest = next((item for item in active_manifests() if item.id == "task-review"), None)
if manifest is None or manifest.workspace is None:
return set()
with store.connection() as conn:
rows = conn.execute(
"""SELECT j.session_id, json_extract(r.payload, '$."task-review".values') AS review_values
FROM automation_jobs j JOIN automation_records r
ON r.kind='task_extensions' AND r.id=j.session_id
WHERE j.batch_id=?""",
(batch_id,),
).fetchall()
reviewed = set()
for row in rows:
values = json.loads(row["review_values"] or "{}")
if any(
isinstance(value := values.get(field.id), str)
and value.strip()
and (field.kind == "text" or value in field.options)
for field in manifest.workspace.fields
):
reviewed.add(row["session_id"])
return reviewed
def task_context(session_id: str, store: AutomationStore) -> dict: def task_context(session_id: str, store: AutomationStore) -> dict:
identifier(session_id) identifier(session_id)
session = SessionRepository(store.path).get_session_by_id(session_id) or {} session = SessionRepository(store.path).get_session_by_id(session_id) or {}
+3
View File
@@ -30,6 +30,9 @@ npm run build
## Local Conventions ## Local Conventions
- 发版遵守根指南的 Release Gates。前端应用版本只改 package.json 顶层 `version` 和 package-lock.json 顶层及 `packages[""].version`;禁止把相同数字的第三方依赖一并替换。
- Node 支持范围以锁定的 Angular CLI/构建依赖实际 `engines` 为准,核对启动脚本的主、次、补丁版本边界和提示;不要只判断“Node 22/24/26”。
- 安装或依赖变更需在隔离目录、独立空 npm 缓存下运行 `npm ci`,再运行上述测试和构建;国内入口同时核验镜像安装。不得以关闭完整性校验、强制安装或删除锁文件掩盖不一致,也不得用已装依赖的构建结果宣称全新安装通过。
- 页面组件在 src/app/components,服务在 src/app/services;先复用已有组件、样式及 API 服务。 - 页面组件在 src/app/components,服务在 src/app/services;先复用已有组件、样式及 API 服务。
- 保持现有简体中文文案、颜色、间距和控件样式,菜单、弹窗和表单必须清晰可读。 - 保持现有简体中文文案、颜色、间距和控件样式,菜单、弹窗和表单必须清晰可读。
- 修改任务或配置交互时覆盖加载、保存、失败和空状态;不要将执行中误标成录屏中或把完成误标成验证通过。 - 修改任务或配置交互时覆盖加载、保存、失败和空状态;不要将执行中误标成录屏中或把完成误标成验证通过。
+2 -2
View File
@@ -1,12 +1,12 @@
{ {
"name": "artemis-showcase-ui", "name": "artemis-showcase-ui",
"version": "2.4.1", "version": "2.4.3",
"lockfileVersion": 3, "lockfileVersion": 3,
"requires": true, "requires": true,
"packages": { "packages": {
"": { "": {
"name": "artemis-showcase-ui", "name": "artemis-showcase-ui",
"version": "2.4.1", "version": "2.4.3",
"dependencies": { "dependencies": {
"@angular/cdk": "^22.2.4", "@angular/cdk": "^22.2.4",
"@angular/common": "^22.2.3", "@angular/common": "^22.2.3",
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "artemis-showcase-ui", "name": "artemis-showcase-ui",
"version": "2.4.1", "version": "2.4.3",
"scripts": { "scripts": {
"ng": "ng", "ng": "ng",
"start": "ng serve --proxy-config proxy.conf.json", "start": "ng serve --proxy-config proxy.conf.json",
@@ -108,6 +108,19 @@
</div> </div>
} }
@if(b.report_error){<p class="error">报告生成失败:{{b.report_error}}</p>} @if(b.report_error){<p class="error">报告生成失败:{{b.report_error}}</p>}
@for(j of b.jobs; track j.session_id){<div class="job" [class.selected]="agent.currentSessionId() === j.session_id"><app-favorite-button [sourceId]="j.session_id" /><button (click)="openBatchTask(j.session_id)" [title]="j.case.goal" [attr.aria-pressed]="agent.currentSessionId() === j.session_id">{{j.case.case_id ? j.case.case_id + ' ' : ''}}{{j.case.name}}@if(j.repeat_count > 1){ · 第 {{j.iteration}} / {{j.repeat_count}} 轮}</button><span>{{label(j.status)}}</span><app-test-result-badge [result]="j.test_result" [executionStatus]="j.status" [compact]="true" />@if(j.error){<small>{{j.error}}</small>}</div>} @for(j of b.jobs; track j.session_id){
<div class="job" [class.selected]="agent.currentSessionId() === j.session_id">
<div class="job-heading">
<app-favorite-button [sourceId]="j.session_id" />
<button (click)="openBatchTask(j.session_id)" [title]="j.case.goal" [attr.aria-pressed]="agent.currentSessionId() === j.session_id">{{j.case.case_id ? j.case.case_id + ' ' : ''}}{{j.case.name}}@if(j.repeat_count > 1){ · 第 {{j.iteration}} / {{j.repeat_count}} 轮}</button>
</div>
<div class="job-statuses">
<span>{{label(j.status)}}</span>
<app-test-result-badge [result]="j.test_result" [executionStatus]="j.status" [compact]="true" />
@if(j.human_reviewed){<span class="human-review-badge" title="人工评审已有保存内容,不代表测试通过">人工已审</span>}
</div>
@if(j.error){<small>{{j.error}}</small>}
</div>
}
</section>} </section>}
} }
@@ -37,7 +37,15 @@ input[type=checkbox]{width:15px;height:15px;accent-color:var(--app-primary-text,
@keyframes archive-spin{to{transform:rotate(360deg);}} @keyframes archive-spin{to{transform:rotate(360deg);}}
.code{font-family:Consolas,monospace;font-size:12px;line-height:1.6;resize:vertical;background:var(--app-surface-soft, #f8fafc);}.table-wrap{max-height:260px;overflow:auto;border:1px solid var(--app-border, #e2e8f0);border-radius:8px;margin-bottom:14px;}table{width:100%;border-collapse:collapse;}th,td{padding:9px 10px;border-bottom:1px solid var(--app-border, #e2e8f0);text-align:left;overflow-wrap:anywhere;}th{background:var(--app-surface-soft, #f8fafc);color:var(--app-text-secondary, #475569);font-weight:500;} .code{font-family:Consolas,monospace;font-size:12px;line-height:1.6;resize:vertical;background:var(--app-surface-soft, #f8fafc);}.table-wrap{max-height:260px;overflow:auto;border:1px solid var(--app-border, #e2e8f0);border-radius:8px;margin-bottom:14px;}table{width:100%;border-collapse:collapse;}th,td{padding:9px 10px;border-bottom:1px solid var(--app-border, #e2e8f0);text-align:left;overflow-wrap:anywhere;}th{background:var(--app-surface-soft, #f8fafc);color:var(--app-text-secondary, #475569);font-weight:500;}
.preview{white-space:pre-wrap;max-height:400px;overflow:auto;font:inherit;color:var(--app-text-control, #334155);background:var(--app-surface-soft, #f8fafc);border-radius:8px;padding:12px;border:1px solid var(--app-border, #e2e8f0);} .preview{white-space:pre-wrap;max-height:400px;overflow:auto;font:inherit;color:var(--app-text-control, #334155);background:var(--app-surface-soft, #f8fafc);border-radius:8px;padding:12px;border:1px solid var(--app-border, #e2e8f0);}
.job{display:flex;flex-wrap:wrap;gap:8px;align-items:center;padding:10px 0;border-bottom:1px solid var(--app-border, #f1f5f9);}.job:last-child{border-bottom:0;}.job button{flex:1;justify-content:flex-start;text-align:left;min-width:150px;}.job span{font-size:12px;background:var(--app-surface-soft, #f1f5f9);color:var(--app-text-secondary, #475569);border-radius:5px;padding:3px 7px;}.job small{width:100%;}details{margin:14px 0;}summary{color:var(--app-text-secondary, #475569);cursor:pointer;padding:8px 0;} .job{display:flex;flex-direction:column;gap:8px;padding:12px 0;border-bottom:1px solid var(--app-border, #f1f5f9);}
.job:last-child{border-bottom:0;}
.job-heading{position:relative;min-width:0;}
.job-heading>app-favorite-button{position:absolute;left:6px;top:4px;z-index:1;}
.job-heading>button{box-sizing:border-box;width:100%;min-width:0;padding-left:40px;justify-content:flex-start;text-align:left;overflow-wrap:anywhere;}
.job-statuses{display:flex;flex-wrap:wrap;align-items:center;gap:8px;}
.job-statuses>span{display:inline-flex;align-items:center;box-sizing:border-box;height:20px;padding:0 6px;border:1px solid transparent;border-radius:5px;font-size:11px;line-height:14px;background:var(--app-surface-soft, #f1f5f9);color:var(--app-text-secondary, #475569);white-space:nowrap;}
.job small{overflow-wrap:anywhere;}
details{margin:14px 0;}summary{color:var(--app-text-secondary, #475569);cursor:pointer;padding:8px 0;}
.task-guidance{margin-top:16px;padding:14px;border:1px solid var(--app-primary-border, #bfdbfe);border-radius:10px;background:var(--app-primary-soft, #f8fbff);} .task-guidance{margin-top:16px;padding:14px;border:1px solid var(--app-primary-border, #bfdbfe);border-radius:10px;background:var(--app-primary-soft, #f8fbff);}
.task-guidance label{margin:8px 0 4px;} .task-guidance label{margin:8px 0 4px;}
.report-overview{padding:14px;border:1px solid var(--app-primary-border, #dbeafe);border-radius:10px;background:var(--app-primary-soft, #f8fbff);} .report-overview{padding:14px;border:1px solid var(--app-primary-border, #dbeafe);border-radius:10px;background:var(--app-primary-soft, #f8fbff);}
@@ -58,3 +66,5 @@ input[type=checkbox]{width:15px;height:15px;accent-color:var(--app-primary-text,
table{width:100%;border-collapse:collapse;font-size:12px;}th,td{padding:7px 9px;border:1px solid var(--app-border, #e2e8f0);vertical-align:top;}th{background:var(--app-primary-soft, #eff6ff);color:var(--app-primary-text, #1e3a8a);} table{width:100%;border-collapse:collapse;font-size:12px;}th,td{padding:7px 9px;border:1px solid var(--app-border, #e2e8f0);vertical-align:top;}th{background:var(--app-primary-soft, #eff6ff);color:var(--app-primary-text, #1e3a8a);}
.md-para-gap{height:8px;} .md-para-gap{height:8px;}
} }
.job .human-review-badge{white-space:nowrap;border:1px solid var(--app-primary-border, #bfdbfe);background:var(--app-primary-soft, #eff6ff);color:var(--app-primary-text, #1d4ed8);font-weight:500;}
@@ -90,6 +90,30 @@ describe('AutomationPanelComponent batch deletion', () => {
expect(http.post).not.toHaveBeenCalled(); expect(http.post).not.toHaveBeenCalled();
}); });
it('shows only saved human review badges and removes them on a batch refresh', async () => {
const fixture = TestBed.createComponent(AutomationPanelComponent);
const component = fixture.componentInstance;
await new Promise<void>(resolve => setTimeout(resolve, 0));
component.tab.set('batches');
const batch = { id: 'review-batch', name: '评审批次', status: 'completed', jobs: [
{ session_id: 'reviewed', status: 'completed', human_reviewed: true, case: { name: '已评用例' } },
{ session_id: 'empty', status: 'completed', human_reviewed: false, case: { name: '空白用例' } },
{ session_id: 'legacy', status: 'completed', case: { name: '旧版任务' } }
] };
component.batch.set(batch);
fixture.detectChanges();
const rows = fixture.nativeElement.querySelectorAll('.job');
expect(rows[0].querySelector('.human-review-badge').textContent).toBe('人工已审');
expect(rows[1].querySelector('.human-review-badge')).toBeNull();
expect(rows[2].querySelector('.human-review-badge')).toBeNull();
expect(fixture.nativeElement.textContent).not.toContain('人工未审');
(http.get.and as any).callFake((url: string) => of(url.endsWith('/review-batch')
? { ...batch, jobs: batch.jobs.map(job => ({ ...job, human_reviewed: false })) } : []));
await component.refresh();
fixture.detectChanges();
expect(fixture.nativeElement.querySelector('.human-review-badge')).toBeNull();
});
it('confirms deletion and removes the batch from the visible list', async () => { it('confirms deletion and removes the batch from the visible list', async () => {
const fixture = TestBed.createComponent(AutomationPanelComponent); const fixture = TestBed.createComponent(AutomationPanelComponent);
fixture.componentInstance.tab.set('batches'); fixture.componentInstance.tab.set('batches');
@@ -129,7 +153,7 @@ describe('AutomationPanelComponent batch deletion', () => {
{session_id:'task-b',status:'completed',case:{name:'用例 B',goal:'观察主页'}} {session_id:'task-b',status:'completed',case:{name:'用例 B',goal:'观察主页'}}
]}); ]});
fixture.detectChanges(); fixture.detectChanges();
const jobs: HTMLButtonElement[] = [...fixture.nativeElement.querySelectorAll('.job > button')]; const jobs: HTMLButtonElement[] = [...fixture.nativeElement.querySelectorAll('.job-heading > button')];
jobs[0].click(); jobs[1].click(); jobs[0].click(); jobs[1].click();
expect(agent.selectSession.calls.allArgs()).toEqual([['task-a',true],['task-b',true]]); expect(agent.selectSession.calls.allArgs()).toEqual([['task-a',true],['task-b',true]]);
expect(component.tab()).toBe('batches'); expect(component.tab()).toBe('batches');
+53
View File
@@ -0,0 +1,53 @@
"""Bounded fingerprints of received argument deltas, without retaining text."""
import hashlib
PREFIX_BYTES = 4096
MAX_PREFIXES = 16
MAX_TRACKED_TOOLS = 64
class ArgumentFingerprint:
def __init__(self):
self.characters = 0
self.bytes = 0
self.deltas = 0
self.encoding_valid = True
self.digest = hashlib.sha256()
self.prefixes: list[dict] = []
def observe(self, arguments: str) -> None:
self.characters += len(arguments)
if not arguments:
return
self.deltas += 1
if not self.encoding_valid:
return
try:
encoded = arguments.encode("utf-8")
except UnicodeEncodeError:
# Diagnostics must not repair, reject or otherwise alter the stream.
self.encoding_valid = False
return
offset = 0
while offset < len(encoded) and len(self.prefixes) < MAX_PREFIXES:
size = min(PREFIX_BYTES - self.bytes % PREFIX_BYTES, len(encoded) - offset)
self.digest.update(encoded[offset : offset + size])
self.bytes += size
offset += size
if self.bytes % PREFIX_BYTES == 0:
self.prefixes.append({"bytes": self.bytes, "sha256": self.digest.hexdigest()})
if offset < len(encoded):
self.digest.update(encoded[offset:])
self.bytes += len(encoded) - offset
def snapshot(self) -> dict:
return {
"argument_chars": self.characters,
"argument_bytes": self.bytes if self.encoding_valid else None,
"argument_sha256": self.digest.hexdigest() if self.encoding_valid else None,
"argument_deltas": self.deltas,
"argument_prefixes": [dict(prefix) for prefix in self.prefixes],
"argument_encoding_valid": self.encoding_valid,
}
+35 -10
View File
@@ -13,6 +13,7 @@ from uuid import uuid4
from PIL import Image from PIL import Image
from artemis.data_engine.context_vars import CURRENT_NODE_NAME, CURRENT_TRACE_ID from artemis.data_engine.context_vars import CURRENT_NODE_NAME, CURRENT_TRACE_ID
from artemis.llm.argument_fingerprint import ArgumentFingerprint, MAX_TRACKED_TOOLS
from artemis.llm.final_review import ACTIVE_FINAL_REVIEW, FinalReviewBudgetExceeded from artemis.llm.final_review import ACTIVE_FINAL_REVIEW, FinalReviewBudgetExceeded
from artemis.llm.reliability import LLMExhaustedError from artemis.llm.reliability import LLMExhaustedError
from artemis.llm.usage import usage_from_message from artemis.llm.usage import usage_from_message
@@ -107,6 +108,7 @@ class RequestAttempt:
self.clock = time.monotonic() self.clock = time.monotonic()
self.last_publish = self.clock self.last_publish = self.clock
self.tools: dict[int | str, dict] = {} self.tools: dict[int | str, dict] = {}
self.tool_fingerprints: dict[int | str, ArgumentFingerprint] = {}
self.review = ACTIVE_FINAL_REVIEW.get() self.review = ACTIVE_FINAL_REVIEW.get()
if self.review: if self.review:
self.review.request_started(request_id, attempt) self.review.request_started(request_id, attempt)
@@ -126,6 +128,9 @@ class RequestAttempt:
"usage": None, "usage": None,
"output_chunks": 0, "output_chunks": 0,
"tool_argument_chars": 0, "tool_argument_chars": 0,
"tool_observation_layer": "langchain_tool_call_chunks",
"argument_fingerprint_version": 1,
"tool_stats_truncated": False,
"tools": [], "tools": [],
"step_id": str(self.step_id) if self.step_id else None, "step_id": str(self.step_id) if self.step_id else None,
"parent_trace_id": str(self.parent_id) if self.parent_id else None, "parent_trace_id": str(self.parent_id) if self.parent_id else None,
@@ -167,9 +172,10 @@ class RequestAttempt:
self.data.update( self.data.update(
headers_at=time.time(), headers_at=time.time(),
http_status=response.status_code, http_status=response.status_code,
server_request_id=_identifier( server_request_id=_identifier(response.headers.get("x-oneapi-request-id"))
response.headers.get("x-request-id") or response.headers.get("request-id") or _identifier(response.headers.get("x-request-id"))
), or _identifier(response.headers.get("request-id")),
response_x_request_id=_identifier(response.headers.get("x-request-id")),
server_trace_id=_identifier(response.headers.get("x-trace-id")), server_trace_id=_identifier(response.headers.get("x-trace-id")),
phase="waiting_output", phase="waiting_output",
) )
@@ -196,18 +202,31 @@ class RequestAttempt:
self.data["usage"] = usage self.data["usage"] = usage
tool_chunks = getattr(chunk, "tool_call_chunks", None) or [] tool_chunks = getattr(chunk, "tool_call_chunks", None) or []
# Responses exposes the same argument delta in content and tool_call_chunks. # Responses exposes the same argument delta in content and tool_call_chunks.
# Count only the latter; retain lengths/names, never argument bodies. # Count only the latter; retain bounded fingerprints, never argument bodies.
for tool in tool_chunks: for tool in tool_chunks:
arguments = tool.get("args")
if isinstance(arguments, str):
self.data["tool_argument_chars"] += len(arguments)
index = tool.get("index") index = tool.get("index")
key = index if index is not None else tool.get("id") or "unknown" call_id = _identifier(tool.get("id"))
stats = self.tools.setdefault(key, {"name": None, "argument_chars": 0}) key = index if index is not None else call_id or "unknown"
if key not in self.tools and len(self.tools) >= MAX_TRACKED_TOOLS:
self.data["tool_stats_truncated"] = True
continue
stats = self.tools.setdefault(key, {"name": None})
if call_id:
stats["call_id"] = call_id
if index is not None:
stats["chunk_index"] = index
name = tool.get("name") name = tool.get("name")
if isinstance(name, str) and re.fullmatch(r"[\w.:-]{1,128}", name): if isinstance(name, str) and re.fullmatch(r"[\w.:-]{1,128}", name):
stats["name"] = name stats["name"] = name
arguments = tool.get("args") fingerprint = self.tool_fingerprints.get(key)
if fingerprint is None:
fingerprint = self.tool_fingerprints[key] = ArgumentFingerprint()
if isinstance(arguments, str): if isinstance(arguments, str):
stats["argument_chars"] += len(arguments) fingerprint.observe(arguments)
self.data["tool_argument_chars"] += len(arguments) stats.update(fingerprint.snapshot())
self.data["tools"] = list(self.tools.values()) self.data["tools"] = list(self.tools.values())
content = getattr(chunk, "content", None) content = getattr(chunk, "content", None)
substantive = isinstance(content, str) and bool(content) substantive = isinstance(content, str) and bool(content)
@@ -252,7 +271,9 @@ class RequestAttempt:
if isinstance(status, int): if isinstance(status, int):
self.data["http_status"] = status self.data["http_status"] = status
if request_id := _identifier(getattr(error, "request_id", None)): if request_id := _identifier(getattr(error, "request_id", None)):
self.data["server_request_id"] = request_id self.data["error_request_id"] = request_id
if not self.data.get("server_request_id"):
self.data["server_request_id"] = request_id
if category == "timeout": if category == "timeout":
self.data.setdefault("timeout_scope", "transport") self.data.setdefault("timeout_scope", "transport")
if response is not None: if response is not None:
@@ -301,6 +322,7 @@ async def observe_request_headers(request):
if attempt is None: if attempt is None:
return return
request.headers.setdefault("X-Client-Request-Id", attempt.data["request_id"]) request.headers.setdefault("X-Client-Request-Id", attempt.data["request_id"])
request.headers.setdefault("X-Client-Attempt-Id", attempt.data["attempt_id"])
routing = {} routing = {}
for name in ("thread-id", "thread_id", "session-id", "session_id"): for name in ("thread-id", "thread_id", "session-id", "session_id"):
value = request.headers.get(name) value = request.headers.get(name)
@@ -309,9 +331,12 @@ async def observe_request_headers(request):
attempt.data.update( attempt.data.update(
request_sent_at=time.time(), request_sent_at=time.time(),
client_request_id=_identifier(request.headers.get("X-Client-Request-Id")), client_request_id=_identifier(request.headers.get("X-Client-Request-Id")),
client_attempt_id=_identifier(request.headers.get("X-Client-Attempt-Id")),
routing_header_fingerprints=routing, routing_header_fingerprints=routing,
client_request_id_matches=request.headers.get("X-Client-Request-Id") client_request_id_matches=request.headers.get("X-Client-Request-Id")
== attempt.data["request_id"], == attempt.data["request_id"],
client_attempt_id_matches=request.headers.get("X-Client-Attempt-Id")
== attempt.data["attempt_id"],
) )
attempt.publish() attempt.publish()
+1 -1
View File
@@ -153,7 +153,7 @@ class LLMPermanentError(LLMCallError):
class LLMExhaustedError(LLMCallError): class LLMExhaustedError(LLMCallError):
"""Retryable failures exhausted every attempt and the pause deadline.""" """Retryable failures exhausted attempts and any applicable pause deadline."""
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
+1 -1
View File
@@ -23,7 +23,7 @@ class ToolArgumentOverflow(ValueError):
"limit_chars": SMALL_ACTION_ARGUMENT_LIMIT, "limit_chars": SMALL_ACTION_ARGUMENT_LIMIT,
} }
super().__init__( super().__init__(
f"模型生成的 {name} 参数异常膨胀({characters} 字符," f"收到的 {name} 参数流超出安全限制({characters} 字符,"
f"保护上限 {SMALL_ACTION_ARGUMENT_LIMIT});已关闭输出流,未执行本次动作。" f"保护上限 {SMALL_ACTION_ARGUMENT_LIMIT});已关闭输出流,未执行本次动作。"
) )
+13 -5
View File
@@ -381,7 +381,7 @@ async def _run_with_recovery[T](
immediately: retrying cannot help and pausing would hang the task. immediately: retrying cannot help and pausing would hang the task.
- When retryable attempts are exhausted: if a fallback model is waiting - When retryable attempts are exhausted: if a fallback model is waiting
(with_fallback), raise LLMExhaustedError so it takes over immediately; (with_fallback), raise LLMExhaustedError so it takes over immediately;
final acceptance also raises instead of pausing an already-finished task. final acceptance and invalid output also raise instead of globally pausing.
Other calls pause (bounded by settings.LLM_PAUSE_TIMEOUT_SECONDS) and Other calls pause (bounded by settings.LLM_PAUSE_TIMEOUT_SECONDS) and
retry the same request on resume. retry the same request on resume.
""" """
@@ -470,7 +470,7 @@ async def _run_with_recovery[T](
) )
if failure.category is FailureCategory.INVALID_OUTPUT: if failure.category is FailureCategory.INVALID_OUTPUT:
# One fresh response can recover a generation loop. Repeated # One fresh response can recover a generation loop. Repeated
# runaway output must reach the existing fallback/pause path, # runaway output must reach fallback or fail this request,
# even when ordinary network retries are configured higher. # even when ordinary network retries are configured higher.
max_attempts = min( max_attempts = min(
max_attempts, total_attempts + max(0, policy.max_attempts - attempt) max_attempts, total_attempts + max(0, policy.max_attempts - attempt)
@@ -505,7 +505,11 @@ async def _run_with_recovery[T](
ACTIVE_ATTEMPT.reset(observation_token) ACTIVE_ATTEMPT.reset(observation_token)
# Retryable attempts exhausted. # Retryable attempts exhausted.
if _FALLBACK_AVAILABLE.get() or ACTIVE_FINAL_REVIEW.get() is not None: if (
_FALLBACK_AVAILABLE.get()
or ACTIVE_FINAL_REVIEW.get() is not None
or last_failure.category is FailureCategory.INVALID_OUTPUT
):
if failed_endpoints is not None and endpoint_key: if failed_endpoints is not None and endpoint_key:
failed_endpoints[endpoint_key] = last_error failed_endpoints[endpoint_key] = last_error
_record_llm_event( _record_llm_event(
@@ -514,7 +518,11 @@ async def _run_with_recovery[T](
"error": str(last_error)[:1000], "error": str(last_error)[:1000],
"category": last_failure.category.value, "category": last_failure.category.value,
"retryable": True, "retryable": True,
"handover": "fallback" if _FALLBACK_AVAILABLE.get() else "final_review_end", "handover": "fallback"
if _FALLBACK_AVAILABLE.get()
else "final_review_end"
if ACTIVE_FINAL_REVIEW.get() is not None
else "task_failed",
}, },
status="failed", status="failed",
) )
@@ -1253,7 +1261,7 @@ async def _with_fallback[T](
) )
# The contextvar must be reset BEFORE fallback_call runs: the fallback has # The contextvar must be reset BEFORE fallback_call runs: the fallback has
# no further fallback behind it, so its own exhaustion should pause. # no further fallback behind it, so apply its category's terminal policy.
fallback_token = _FALLBACK_AVAILABLE.set(True) fallback_token = _FALLBACK_AVAILABLE.set(True)
try: try:
result = await main_call() result = await main_call()
+39
View File
@@ -0,0 +1,39 @@
# 模型流式参数诊断
小型内置动作的参数流仍受 16384 字符保护。超限时关闭输出流、丢弃不完整参数,不执行该次动作;一次新响应重试和可用备用端点耗尽后,只结束当前任务,不进入共享的模型暂停。批次仍遵循“失败即停止”、人工暂停和取消设置。网络故障、认证错误和最终验收保留各自的恢复规则。
## 请求关联
每个模型请求尝试沿用现有 `llm_diagnostic` 记录,不另建日志数据库。
| 字段 / 请求头 | 含义 |
| --- | --- |
| `request_id` / `X-Client-Request-Id` | 同一次恢复循环内保持稳定的逻辑请求编号;切换备用调用或恢复暂停后可能建立新的逻辑请求 |
| `attempt_id` / `X-Client-Attempt-Id` | 本次客户端尝试编号;重试时变化,`previous_attempt_id` 关联前次尝试 |
| `client_*_id_matches` | 实际发送的请求头是否与本地编号一致;调用方已指定的头保留原值 |
| `server_request_id` | 首个有效的 `X-Oneapi-Request-Id`、`X-Request-Id` 或 `Request-Id`,按此前后顺序选择 |
| `response_x_request_id` | 响应中的原始 `X-Request-Id`,不推定它来自模型上游 |
| `error_request_id` | SDK 异常带回的编号,不覆盖已经从响应头读取的编号 |
| `response_id` | SDK 实际返回的 Responses 响应编号 |
编号只接收有长度限制的可打印白名单字符。请求头不控制会话亲和性,不替换 `thread-id` / `session-id`。网关自身的重试应使用独立编号或序号,不能假装是一次新的客户端尝试。
## 参数指纹 v1
`tool_observation_layer=langchain_tool_call_chunks` 明确表示客户端观测的是 SDK 转换后的参数片段,不是原始 SSE 帧。每工具记录实际可得的 `call_id`、`chunk_index`、工具名,以及:
- `argument_chars`:参数片段的 Unicode 字符数之和。
- `argument_bytes` / `argument_sha256`:按原顺序连接片段后,其 UTF-8 字节长度和 SHA256;不解析、重排参数 JSON,也不规范化空白或 Unicode。
- `argument_deltas`:非空参数片段数,不能直接与网关原始事件数比较。
- `argument_prefixes`:每 4096 字节的累计 SHA256,最多保留前 16 个。可以在分片大小不同、上游预读或中途取消时比较相同字节位置。
- `argument_encoding_valid`:遇到不能编码为 UTF-8 的片段时为 false,当前总字节数和总哈希留空,避免伪造可比结果;诊断本身不改变执行流程。
最多记录 64 个工具;超过时设置 `tool_stats_truncated=true`,总参数字符计数仍继续。诊断不保留参数正文。哈希仅证明已收到的字节前缀,不能证明输出完整或动作成功;完整性仍由响应终止状态和解析校验决定。
跨三端优先使用 `response_id + call_id` 关联。LangChain 的 `chunk_index` 可能因文本或推理子块不同于原始 `output_index`,缺失编号时不能按数组位置猜测。正常的 `arguments.done`、`output_item.done`、`response.completed` 快照不重复计入参数;自定义工具转换、非空初始参数或协议异常需要先核对 SDK 映射,不能仅凭跨层哈希不同认定某一端重复发送。
历史日志没有保存的原始帧无法由新增诊断恢复。HTTP 200 只说明响应头已到达;客户端保护后出现的 499 或 Broken pipe 也不能证明最初原因。网关写入本机下游 writer 成功不等于客户端已消费或执行。
## 离线验证
`tests/unit/test_llm_stream_guard.py` 验证保护、有限重试、相同/不同备用端点,以及不会创建或清除其他请求的全局暂停。`tests/unit/test_request_diagnostics.py` 验证请求编号、超时重试、响应头及单次参数计数。`tests/unit/test_argument_fingerprint.py` 验证不同分片、中文和 emoji、固定字节前缀、有界统计与异常编码。这些测试不调用真实模型或设备。
+24
View File
@@ -0,0 +1,24 @@
# ARTEMIS 2.4.2
本版为批次任务增加人工评审标记,并调整任务列表布局。人工评审插件保持 **1.1.0**,第三方依赖版本不变。
## 更新
- 安装并启用人工评审插件后,任务存在已保存的有效评审内容时,批次列表显示“人工已审”。评审结论或非空备注均计入;清空全部内容后隐藏,禁用或卸载插件后也不显示。
- 标记仅说明已有人工评审内容,不代表测试通过,不覆盖 AI 判定或改变任务执行流程。没有安装插件的用户不显示该标记。
- 批次任务标题与状态分行显示:收藏星标位于标题框内,执行状态、AI 判定和人工已审标签在下方等高、左对齐排列。此布局适用于所有用户。
- 按批次一次读取当前评审内容,沿用列表自动刷新,不逐条增加网络请求;重跑任务不继承旧任务的已审标记。
- 补充发版约束:应用与第三方依赖独立版本化,检查锁文件差异,并验证最终源码包的全新安装。
## 验证
- 默认 Python 回归:3269 通过,2 跳过,8 按标记排除;Ruff、质量基线和核心类型检查通过。
- 前端 ChromeHeadless:348 项通过;生产构建通过。
- 内置浏览器使用隔离数据检查收藏星标、两行布局及有评审/无评审标签显示。未写入真实任务评审数据。
- 最终源码包另行执行独立目录、空 npm 缓存的官方源和国内镜像安装及生产构建,结果随发布附件提供。
## 升级
已有安装请等待任务结束,运行 `update-cn.bat` 或 `update.bat` 后重启。新安装下载 `artemis-2.4.2-source.zip`,按 `SHA256SUMS.txt` 校验后启动。请勿直接覆盖旧安装目录。
人工评审插件仍需单独安装并启用;本次无需升级已安装的 1.1.0。发布过程不构建 Docker,不重启用户服务,不执行设备或真实业务任务。
+25
View File
@@ -0,0 +1,25 @@
# ARTEMIS 2.4.3
本版修复异常模型输出影响整个批次的问题,并增加流式参数诊断。人工评审插件保持 **1.1.0**,SDK、其他插件及第三方依赖版本不变。
## 更新
- 已识别的异常输出(包括工具参数溢出、响应完整性校验失败)在有限重试和可用备用端点耗尽后,只结束当前任务,不再创建全局模型暂停。批次未开启“失败即停止”时可继续下一条;人工暂停、失败即停止、网络恢复及最终验收的原有规则保留。
- 保留 16384 字符工具参数保护,关闭异常输出流,丢弃未完成内容,不执行本次异常动作。报错改为描述“收到的参数流”,避免在证据不足时直接归责模型。
- 增加单次尝试请求头,区分逻辑请求、客户端尝试、网关响应与 SDK 异常中的请求编号,便于跨服务核对。
- 对收到的工具参数片段记录字符数、UTF-8 字节数及有界 SHA256 前缀摘要,默认不新增参数正文记录。最多跟踪 64 个工具,每工具最多保留 16 个固定位置前缀。SDK 的 chunk_index 与原始协议 output_index 分开命名。
诊断字段及边界见 [流式参数诊断](../llm-stream-diagnostics.md)。本次主程序安装不会部署或升级 NEWAPI、CodexManager;服务端修复独立交付。
## 验证
- 默认 Python 回归:3284 通过,6 跳过,8 按标记排除;Ruff 格式和静态检查、质量基线、核心类型检查通过。
- 五类合成输入经过实际 CodexManager、NEWAPI 转发函数及原 OpenAI/LangChain SDK 的离线文件串接:正常输入有效;超长、累计全文、重复及半关闭输入被拒绝;各层参数摘要一致,设备动作执行数为 0。
- 最终源码 ZIP 的隔离解压、独立空 npm 缓存、官方源与国内镜像安装和生产构建结果随发布附件 `INSTALL-VERIFICATION.md` 提供。
- 未重跑用户原业务批次、未做高并发压力测试或真实模型端到端验证。历史原始参数帧未保存,不能据此认定原事故参数膨胀首先发生在哪一层。
## 升级
已有安装请等待任务结束,运行 `update-cn.bat` 或 `update.bat` 后重启。新安装下载 `artemis-2.4.3-source.zip`,按 `SHA256SUMS.txt` 校验后启动。请勿直接覆盖旧安装目录。
人工评审插件 1.1.0 无需更新。本次发布不构建 Docker,不重启用户服务,不执行设备或真实业务任务。
+1 -1
View File
@@ -14,7 +14,7 @@
[project] [project]
name = "artemis" name = "artemis"
version = "2.4.1" version = "2.4.3"
description = "Autonomous AI Assistant for Mobile Automation" description = "Autonomous AI Assistant for Mobile Automation"
authors = [ authors = [
@@ -57,15 +57,16 @@ def manifest_payload():
} }
def install(plugins, original_case=False): def install(plugins, original_case=False, plugin_id="review"):
package = BytesIO() package = BytesIO()
manifest = manifest_payload() manifest = manifest_payload()
manifest["id"] = plugin_id
if original_case: if original_case:
manifest["workspace"]["panels"][0]["kind"] = "original_case" manifest["workspace"]["panels"][0]["kind"] = "original_case"
with zipfile.ZipFile(package, "w") as archive: with zipfile.ZipFile(package, "w") as archive:
archive.writestr("plugin.json", json.dumps(manifest)) archive.writestr("plugin.json", json.dumps(manifest))
install_archive(plugins, package.getvalue()) install_archive(plugins, package.getvalue())
plugins.set_enabled("review", True) plugins.set_enabled(plugin_id, True)
@pytest.fixture @pytest.fixture
@@ -108,6 +109,71 @@ def save(client, values, revision=0, request_id=None):
) )
def test_batch_review_badge_tracks_current_values_and_plugin_lifecycle(context, monkeypatch):
from apps.admin_console.routers import automation
store, plugins, client = context
monkeypatch.setattr(automation, "store", store)
result = {"status": "inconclusive", "source": "execution"}
monkeypatch.setattr(automation, "read_reviewed_result", lambda *args, **kwargs: result)
client.app.include_router(automation.router)
store.create_batch({"id": "review-batch", "status": "completed"}, [], {})
store.save_job(
{"session_id": "batch-task", "batch_id": "review-batch", "status": "completed", "case": {}}
)
store.save_job(
{"session_id": "new-run", "batch_id": "review-batch", "status": "completed", "case": {}}
)
def badge():
response = client.get("/api/automation/batches/review-batch")
assert response.status_code == 200
jobs = {job["session_id"]: job for job in response.json()["jobs"]}
assert "batch-task" in jobs, response.json()
assert jobs["batch-task"]["test_result"] == result
assert jobs["new-run"]["human_reviewed"] is False
return jobs["batch-task"]["human_reviewed"]
assert badge() is False
install(plugins)
assert (
client.put(
"/api/task-extensions/batch-task/review",
json={
"values": {"accuracy": "Pass"},
"expected_revision": 0,
"request_id": str(uuid4()),
},
).status_code
== 200
)
assert badge() is False # Other workspace plugins are not the human review plugin.
install(plugins, plugin_id="task-review")
assert badge() is False
for revision, (values, expected) in enumerate(
[
({}, False),
({"accuracy_note": " \n"}, False),
({"accuracy": "Pass"}, True),
({"accuracy": "Failed"}, True),
({"accuracy": "", "accuracy_note": ""}, False),
({"accuracy_note": "人工核对了截图"}, True),
]
):
response = client.put(
"/api/task-extensions/batch-task/task-review",
json={"values": values, "expected_revision": revision, "request_id": str(uuid4())},
)
assert response.status_code == 200
assert badge() is expected
plugins.set_enabled("task-review", False)
assert badge() is False
plugins.set_enabled("task-review", True)
assert badge() is True
plugins.uninstall("task-review")
assert badge() is False
def test_original_case_import_requires_enabled_plugin_and_keeps_execution_prompt(context): def test_original_case_import_requires_enabled_plugin_and_keeps_execution_prompt(context):
from apps.admin_console.services import task_import from apps.admin_console.services import task_import
+59
View File
@@ -0,0 +1,59 @@
import hashlib
import json
import pytest
from langchain_core.messages import AIMessageChunk
from artemis.llm.argument_fingerprint import ArgumentFingerprint, MAX_PREFIXES, PREFIX_BYTES
from artemis.llm.diagnostics import RequestAttempt
@pytest.mark.parametrize("fragment_size", [1, 37, 4096, 100000])
def test_fingerprints_match_at_byte_offsets_despite_unicode_and_different_fragmentation(
fragment_size,
):
arguments = '{"target":"' + "私人内容🙂" * 8000 + '"}'
expected = arguments.encode("utf-8")
fingerprint = ArgumentFingerprint()
for offset in range(0, len(arguments), fragment_size):
fingerprint.observe(arguments[offset : offset + fragment_size])
result = fingerprint.snapshot()
assert result["argument_chars"] == len(arguments)
assert result["argument_bytes"] == len(expected)
assert result["argument_sha256"] == hashlib.sha256(expected).hexdigest()
assert result["argument_encoding_valid"] is True
assert result["argument_prefixes"] == [
{"bytes": size, "sha256": hashlib.sha256(expected[:size]).hexdigest()}
for size in range(PREFIX_BYTES, PREFIX_BYTES * MAX_PREFIXES + 1, PREFIX_BYTES)
]
assert "私人" not in json.dumps(result, ensure_ascii=False)
def test_invalid_utf8_is_marked_without_failing_or_forging_a_digest():
fingerprint = ArgumentFingerprint()
for value in ["safe", "\ud800", "later"]:
fingerprint.observe(value)
assert fingerprint.snapshot() == {
"argument_chars": 10,
"argument_bytes": None,
"argument_sha256": None,
"argument_deltas": 3,
"argument_prefixes": [],
"argument_encoding_valid": False,
}
def test_tool_tracking_is_bounded_but_total_count_keeps_all_received_arguments():
attempt = RequestAttempt(
None, request_id="bounded", attempt=1, timeout=300, max_retries=0, metadata={}
)
for index in range(65):
attempt.chunk(
AIMessageChunk(
content=[],
tool_call_chunks=[{"name": "click", "args": "{}", "index": index}],
)
)
assert len(attempt.data["tools"]) == len(attempt.tool_fingerprints) == 64
assert attempt.data["tool_stats_truncated"] is True
assert attempt.data["tool_argument_chars"] == 130
+29
View File
@@ -2,6 +2,7 @@
import asyncio import asyncio
import json import json
import hashlib
from types import SimpleNamespace from types import SimpleNamespace
from unittest.mock import AsyncMock from unittest.mock import AsyncMock
@@ -15,6 +16,7 @@ import pytest_asyncio
from artemis.llm.reliability import FailureCategory, LLMExhaustedError from artemis.llm.reliability import FailureCategory, LLMExhaustedError
from artemis.llm.response_guard import ResponseIntegrityError, ensure_response_complete from artemis.llm.response_guard import ResponseIntegrityError, ensure_response_complete
from artemis.llm.router import ModelEndpoint, ModelFactory from artemis.llm.router import ModelEndpoint, ModelFactory
from artemis.llm.diagnostics import RequestAttempt
from artemis.services import llm as llm_service from artemis.services import llm as llm_service
from artemis.services.llm import RobustChatModelWrapper, acomplete from artemis.services.llm import RobustChatModelWrapper, acomplete
@@ -195,6 +197,33 @@ async def test_completed_stream_returns_original_valid_arguments(transport_case)
assert result.tool_calls[0]["args"] == {"passed": True} assert result.tool_calls[0]["args"] == {"passed": True}
@pytest.mark.asyncio
@pytest.mark.parametrize("transport_case", ["fragmented"], indirect=True)
async def test_real_sdk_fingerprint_counts_received_arguments_once(transport_case):
case = transport_case
arguments = json.dumps({"passed": True, "evidence": "中文🙂" * 1000}, ensure_ascii=False)
case.outputs.append(("completed", arguments))
observation = RequestAttempt(
None, request_id="sdk-test", attempt=1, timeout=300, max_retries=0, metadata={}
)
merged = None
async for chunk in case.model.astream([]):
observation.chunk(chunk)
merged = chunk if merged is None else merged + chunk
observation.finish("completed", response=merged)
assert len(observation.data["tools"]) == 1
tool = observation.data["tools"][0]
assert tool["call_id"] == "call_test"
assert tool["argument_chars"] == len(arguments)
assert tool["argument_bytes"] == len(arguments.encode())
assert tool["argument_sha256"] == hashlib.sha256(arguments.encode()).hexdigest()
assert tool["argument_prefixes"][0] == {
"bytes": 4096,
"sha256": hashlib.sha256(arguments.encode()[:4096]).hexdigest(),
}
assert tool["argument_deltas"] == 1
@pytest.mark.asyncio @pytest.mark.asyncio
@pytest.mark.parametrize("transport_case", ["fragmented"], indirect=True) @pytest.mark.parametrize("transport_case", ["fragmented"], indirect=True)
@pytest.mark.parametrize("status", ["completed", None]) @pytest.mark.parametrize("status", ["completed", None])
+47 -1
View File
@@ -124,7 +124,9 @@ async def test_gateway_closes_bad_stream_discards_partial_and_bounds_retry(
with pytest.raises(LLMExhaustedError) as caught: with pytest.raises(LLMExhaustedError) as caught:
await wrapper.complete([]) await wrapper.complete([])
assert caught.value.failure.category is FailureCategory.INVALID_OUTPUT assert caught.value.failure.category is FailureCategory.INVALID_OUTPUT
llm._wait_for_resume.assert_awaited_once() llm._wait_for_resume.assert_not_awaited()
assert not llm.PAUSE_FILE.exists()
assert any(name == "llm_gave_up" and p["handover"] == "task_failed" for name, p in resets)
assert model.calls == 2 assert model.calls == 2
assert closed == [1, 2] assert closed == [1, 2]
failure = next( failure = next(
@@ -139,3 +141,47 @@ async def test_gateway_closes_bad_stream_discards_partial_and_bounds_retry(
assert failure["usage"] is None assert failure["usage"] is None
assert "private" not in json.dumps(records + resets, default=str) assert "private" not in json.dumps(records + resets, default=str)
assert any(name == "llm_stream_reset" and p["action"] == "discard" for name, p in resets) assert any(name == "llm_stream_reset" and p["action"] == "discard" for name, p in resets)
@pytest.mark.asyncio
@pytest.mark.parametrize("fallback_kind", ["same", "failing", "working"])
@pytest.mark.parametrize("already_paused", [False, True])
async def test_invalid_output_fallback_does_not_pause_or_clear_another_request(
fallback_kind, already_paused
):
calls = []
if already_paused:
llm.PAUSE_FILE.write_text("another request or user pause", encoding="utf-8")
async def main():
calls.append("main")
raise ToolArgumentOverflow("click", SMALL_ACTION_ARGUMENT_LIMIT + 1)
async def fallback():
calls.append("fallback")
if fallback_kind == "working":
return "valid response"
raise ToolArgumentOverflow("click", SMALL_ACTION_ARGUMENT_LIMIT + 2)
async def run():
return await llm.with_fallback(
lambda: llm._run_with_recovery(main, endpoint_key="main", max_retries=5),
lambda: llm._run_with_recovery(
fallback,
endpoint_key="main" if fallback_kind == "same" else "fallback",
max_retries=5,
),
)
if fallback_kind == "working":
assert await run() == "valid response"
else:
with pytest.raises(LLMExhaustedError) as caught:
await run()
assert caught.value.failure.category is FailureCategory.INVALID_OUTPUT
assert calls.count("main") == 2
assert calls.count("fallback") == {"same": 0, "failing": 2, "working": 1}[fallback_kind]
llm._wait_for_resume.assert_not_awaited()
assert llm.PAUSE_FILE.exists() is already_paused
if already_paused:
assert llm.PAUSE_FILE.read_text(encoding="utf-8") == "another request or user pause"
+79 -6
View File
@@ -1,6 +1,7 @@
import asyncio import asyncio
from copy import deepcopy from copy import deepcopy
import json import json
import hashlib
from types import SimpleNamespace from types import SimpleNamespace
from uuid import uuid4 from uuid import uuid4
@@ -36,6 +37,46 @@ class Engine:
pass pass
@pytest.mark.parametrize(
("headers", "expected"),
[
({"X-Oneapi-Request-Id": "gateway-request"}, "gateway-request"),
(
{"X-Oneapi-Request-Id": "gateway-request", "X-Request-Id": "provider-request"},
"gateway-request",
),
({"X-Request-Id": "provider-request"}, "provider-request"),
({"Request-Id": "legacy-request"}, "legacy-request"),
(
{"X-Oneapi-Request-Id": "invalid header value", "X-Request-Id": "standard-request"},
"standard-request",
),
],
)
def test_gateway_request_id_preserves_the_standard_header_without_assuming_its_source(
headers, expected
):
attempt = RequestAttempt(
Engine(), request_id="local-request", attempt=1, timeout=300, max_retries=1, metadata={}
)
attempt.headers(httpx.Response(200, headers=headers))
assert attempt.data["server_request_id"] == expected
assert attempt.data["response_x_request_id"] == headers.get("X-Request-Id")
def test_sdk_error_cannot_overwrite_the_observed_gateway_request_id():
class RequestError(Exception):
request_id = "sdk-request"
attempt = RequestAttempt(
Engine(), request_id="local-request", attempt=1, timeout=300, max_retries=1, metadata={}
)
attempt.headers(httpx.Response(400, headers={"X-Oneapi-Request-Id": "gateway-request"}))
attempt.finish("failed", error=RequestError("failed"))
assert attempt.data["server_request_id"] == "gateway-request"
assert attempt.data["error_request_id"] == "sdk-request"
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_request_hook_correlates_api_calls_without_logging_header_values(): async def test_request_hook_correlates_api_calls_without_logging_header_values():
engine = Engine() engine = Engine()
@@ -55,13 +96,18 @@ async def test_request_hook_correlates_api_calls_without_logging_header_values()
) )
await observe_request_headers(request) await observe_request_headers(request)
assert request.headers["x-client-request-id"] == "local-request" assert request.headers["x-client-request-id"] == "local-request"
assert request.headers["x-client-attempt-id"] == str(attempt.trace_id)
assert attempt.data["client_attempt_id_matches"] is True
assert attempt.data["client_request_id_matches"] is True assert attempt.data["client_request_id_matches"] is True
assert len(attempt.data["routing_header_fingerprints"]) == 2 assert len(attempt.data["routing_header_fingerprints"]) == 2
assert "private" not in json.dumps(engine.records, default=str) assert "private" not in json.dumps(engine.records, default=str)
request.headers["X-Client-Request-Id"] = "caller-id" request.headers["X-Client-Request-Id"] = "caller-id"
request.headers["X-Client-Attempt-Id"] = "caller-attempt"
await observe_request_headers(request) await observe_request_headers(request)
assert request.headers["x-client-request-id"] == "caller-id" assert request.headers["x-client-request-id"] == "caller-id"
assert attempt.data["client_request_id_matches"] is False assert attempt.data["client_request_id_matches"] is False
assert request.headers["X-Client-Attempt-Id"] == "caller-attempt"
assert attempt.data["client_attempt_id_matches"] is False
finally: finally:
ACTIVE_ATTEMPT.reset(token) ACTIVE_ATTEMPT.reset(token)
@@ -75,6 +121,7 @@ async def test_headers_without_body_timeout_then_retry_keeps_evidence(monkeypatc
) )
llm._ENDPOINT_BREAKER._states.clear() llm._ENDPOINT_BREAKER._states.clear()
calls = 0 calls = 0
wire_ids = []
class StalledBody(httpx.AsyncByteStream): class StalledBody(httpx.AsyncByteStream):
async def __aiter__(self): async def __aiter__(self):
@@ -84,6 +131,9 @@ async def test_headers_without_body_timeout_then_retry_keeps_evidence(monkeypatc
async def transport(request): async def transport(request):
nonlocal calls nonlocal calls
calls += 1 calls += 1
wire_ids.append(
(request.headers["X-Client-Request-Id"], request.headers["X-Client-Attempt-Id"])
)
headers = {"x-request-id": f"server-{calls}", "set-cookie": "secret-cookie"} headers = {"x-request-id": f"server-{calls}", "set-cookie": "secret-cookie"}
return httpx.Response( return httpx.Response(
200, headers=headers, stream=StalledBody() if calls == 1 else httpx.ByteStream(b"ok") 200, headers=headers, stream=StalledBody() if calls == 1 else httpx.ByteStream(b"ok")
@@ -91,7 +141,7 @@ async def test_headers_without_body_timeout_then_retry_keeps_evidence(monkeypatc
async with httpx.AsyncClient( async with httpx.AsyncClient(
transport=httpx.MockTransport(transport), transport=httpx.MockTransport(transport),
event_hooks={"response": [observe_response_headers]}, event_hooks={"request": [observe_request_headers], "response": [observe_response_headers]},
) as client: ) as client:
class Model: class Model:
@@ -123,6 +173,10 @@ async def test_headers_without_body_timeout_then_retry_keeps_evidence(monkeypatc
assert final["request_id"] == failed["request_id"] assert final["request_id"] == failed["request_id"]
assert final["attempt_id"] != failed["attempt_id"] and final["attempt"] == 2 assert final["attempt_id"] != failed["attempt_id"] and final["attempt"] == 2
assert final["previous_attempt_id"] == failed["attempt_id"] assert final["previous_attempt_id"] == failed["attempt_id"]
assert wire_ids == [
(failed["request_id"], failed["attempt_id"]),
(final["request_id"], final["attempt_id"]),
]
retry = next(r["payload"] for r in engine.records if r["name"] == "llm_retry") retry = next(r["payload"] for r in engine.records if r["name"] == "llm_retry")
assert retry["request_id"] == failed["request_id"] assert retry["request_id"] == failed["request_id"]
assert retry["attempt_id"] == failed["attempt_id"] assert retry["attempt_id"] == failed["attempt_id"]
@@ -163,10 +217,26 @@ def test_tool_deltas_are_counted_once_without_retaining_arguments():
attempt.finish("completed", response=combined) attempt.finish("completed", response=combined)
assert attempt.data["output_chunks"] == 4 assert attempt.data["output_chunks"] == 4
assert attempt.data["tool_argument_chars"] == len('{"secret":"private"}{"seconds":1}') assert attempt.data["tool_argument_chars"] == len('{"secret":"private"}{"seconds":1}')
assert attempt.data["tools"] == [ for tool, name, arguments, call_id, index, count in zip(
{"name": "click", "argument_chars": len('{"secret":"private"}')}, attempt.data["tools"],
{"name": "wait", "argument_chars": len('{"seconds":1}')}, ["click", "wait"],
] ['{"secret":"private"}', '{"seconds":1}'],
["c1", "c2"],
[0, 1],
[2, 1],
strict=True,
):
assert tool == {
"name": name,
"call_id": call_id,
"chunk_index": index,
"argument_chars": len(arguments),
"argument_bytes": len(arguments.encode()),
"argument_sha256": hashlib.sha256(arguments.encode()).hexdigest(),
"argument_deltas": count,
"argument_prefixes": [],
"argument_encoding_valid": True,
}
assert attempt.data["response_status"] == "completed" assert attempt.data["response_status"] == "completed"
assert "private" not in json.dumps(engine.records, default=str) assert "private" not in json.dumps(engine.records, default=str)
assert "secret" not in json.dumps(engine.records, default=str) assert "secret" not in json.dumps(engine.records, default=str)
@@ -205,7 +275,10 @@ async def test_timeout_during_tool_generation_keeps_last_output_and_tool_name(mo
) )
assert failed["failure_phase"] == "generating_tool_arguments" assert failed["failure_phase"] == "generating_tool_arguments"
assert failed["last_output_at"] >= failed["first_output_at"] assert failed["last_output_at"] >= failed["first_output_at"]
assert failed["tools"] == [{"name": "click", "argument_chars": 5}] assert len(failed["tools"]) == 1
tool = failed["tools"][0]
assert tool["name"] == "click" and tool["argument_chars"] == tool["argument_bytes"] == 5
assert tool["argument_sha256"] == hashlib.sha256(b'{"x":').hexdigest()
assert failed["usage"] is None assert failed["usage"] is None
assert failed["max_output_tokens"] == 12000 assert failed["max_output_tokens"] == 12000
Generated
+1 -1
View File
@@ -248,7 +248,7 @@ wheels = [
[[package]] [[package]]
name = "artemis" name = "artemis"
version = "2.4.1" version = "2.4.3"
source = { editable = "." } source = { editable = "." }
dependencies = [ dependencies = [
{ name = "adbutils" }, { name = "adbutils" },