合入运行质量修复:结构化抽取截断重试、写作目录按模型窗口定篇幅并按来源均衡、缺失断言交回重写、审批卡标明运行类别并附报告核对概况、交给审批的报告探索运行也在节点里裁判、裁判小批分波地问、流式用量回传、金额估不出时写「金额无法估算」、证据目录省略警告照实说、日期时间按可读格式显示、助手写 SQL 用中文别名、大库没有数据目录时提示先起草

This commit is contained in:
ThearchyHelios
2026-10-04 19:09:21 +08:00
46 changed files with 3485 additions and 196 deletions
+66 -4
View File
@@ -152,6 +152,11 @@ NODE_REFERENCE = """\
查不到的记空值、不兜底成 0。assign_to 拿到的就是核对过的对象,口径卡直接写 vars.<assign_to>.字段。
只写 output_schema 不开 cite_fields 不生效。**不要用 transform 解析 agent / llm 的文字**:模型写的 JSON
常夹着没转义的引号,解析一失败整个节点就失败,也核对不了出处
**agent 查库时**,prompt 里写明查询结果列用中文别名(AS 月份、AS 曝光量):报告用 [[table:]] 生成的表格表头、
[[v:]] 引用、口径卡的 cell() 都用这个列名,写成英文缩写的原始列名读的人看不懂。中文别名按数据源的方言
加标识符引号:MySQL / MariaDB 用反引号(AS `月份`),PostgreSQL、Oracle、SQLite 用双引号(AS "月份"),不要用
单引号(那是字符串)。只给最外层 SELECT 的结果列起别名,WHERE、JOIN、GROUP BY 里照旧写原列名;Oracle 旧版本
的标识符最长 30 字节,别名不超过 10 个字
**默认不要写 max_steps、budget_tokens、budget_usd**。平台用的是护栏而不是固定步数:
步数只是很高的兜底(跟随设置,默认 100),重复调用、连续几步没有新信息、预算用完、
上下文快满时,它会按查到的部分收尾。写死一个小数字(见过 8)会让它查完表结构就没额度回答了。
@@ -193,6 +198,10 @@ NODE_REFERENCE = """\
写进去再也读不回来——真发生过:一轮写进 scope"user",下一轮没写 scope 落回
default,于是「我叫张三」存下来了,问「我是谁」却答不上来
- human:人工审批。config: {mode: approve|input|edit, title, message}
title、message 说批了会怎样时写「批准后输出」「批准后继续」,**不要写「正式出具」「批准后才会出具」**:出具只发生在
成果节点配了出具契约、并且从已发布版本发起的正式运行里;画布上试运行、问数据都是探索运行,批准后只是输出。
这次是探索运行还是正式运行,审批卡会自己标明。审的是报告时把 human 接在 report 后面(message 里可以引用
{{ nodes.报告节点id.text }}),审批卡会附上这份报告的核对概况
- validate:JSON Schema 校验,可自动让模型修复。config: {schema, source, max_retries}
- transform:数据整形。config: {mode: expression|template|json, expression/template, assign_to}
- subgraph:嵌套另一个工作流。config: {workflow_id, input}
@@ -228,10 +237,15 @@ NODE_REFERENCE = """\
挂引用,没挂的计入缺口、出具降档)/ judge(在 require_citation 之外,再请另一个模型按每句挂的依据判断支不支持:
正式运行在节点里判,证据相矛盾的按 judge.on_unsupported 处置——degrade 降档(默认)或 withhold 不予出具,
证据不足的(摘录里没有判断所需的信息)只降档;
探索运行不在节点里花钱,读的人点开哪句再按需判哪句,只标注)。judge 的子配置写在节点的 judge 里:judge.model、
探索运行不在节点里花钱,读的人点开哪句再按需判哪句,只标注;要交给人工审批的报告例外,探索运行也在节点里判,
审批人批准前就能看到判了几句、几句证据相矛盾)。judge 的子配置写在节点的 judge 里:judge.model、
judge.provider 指定裁判模型(建议和写作模型不同,同一个模型等于自己审自己;不写就用设置里的「证据裁判模型」);
judge.max_cost_usd、judge.max_claims、judge.timeout_s 是每份报告的裁判金额、句数、时长上限,写 null 表示不限,
不写取设置里的默认;judge.rewrite_once 默认 false,打开后把证据相矛盾、证据不足的句子交回写作者只改一次。
**多个来源汇总成的分析报告**(几个 agent 分头查库,或者几次查询的结果合在一起写成的分析、结论)默认写 claims: "judge",
不写 judge 的预算(judge 整个不写,金额、句数、时长上限都取设置里的默认):正式运行在节点里自动判,探索运行由读的人
点开哪句判哪句,不在节点里花钱;报告要交给人工审批的,探索运行也在节点里判。用户明确说不要核对结论句的不加;只回答一个问题、一次查询的不必加。要按受管级别
发布的模板不适用这条默认,按下面受管级别的规则写预算
entities 默认 link
(核对表名、字段名),报告里根本不提表名时才写 off
- output 的出具契约:config.contract = {report_from: 报告节点id, metrics_from:[口径卡id],
@@ -241,7 +255,7 @@ NODE_REFERENCE = """\
(问数据的典型图),契约不写 metrics_from,也不写 required / expected
要按受管级别发布的模板(发布前检查不过就发不出去):报告节点写明 numbers: "strict"、on_violation: "fail"、
claims: "require_citation"(或者更严的 claims: "judge",这时 judge 里必须写 judge.max_cost_usd——写一个金额,或者
写 null 表示不限,预算要用户定,拿不准就问);契约用 report_from(受管级别不收 narrative),成果字段只取报告节点的正文;给口径卡
写 null 表示不限,预算要用户定,拿不准就问;只有受管级别要这样,普通草稿里默认开的 judge 不写预算、不为预算问人);契约用 report_from(受管级别不收 narrative),成果字段只取报告节点的正文;给口径卡
供数的 agent 配 output_schema 和 cite_fields: true,code 节点喂口径卡的必须是真取数(evidence_role: "source"),
否则把计算写进口径卡;llm、agent、协作团队写的文字不能绕过报告节点直接进出口,llm 也不能喂口径卡
@@ -351,7 +365,8 @@ def _user_message(payload: GenerateIn, *, patch: bool) -> str:
"- report 提到表名、字段名时写 [[t:表名]] / [[c:表名.字段名]](英文名也可以放进反引号,中文名不行:"
"反引号里的中文名不核对),只写 agent 真查过的名字,"
"编造的会被标出来;要引用知识库原话就在 report 前接 retrieve,用 [[q:K1|原话]] 逐字引用。问数是探索运行,"
"report 的 claims 不用写:结论句支不支持,由读答案的人点开时按需请模型判断\n"
"report 的 claims 不用写:结论句支不支持,由读答案的人点开时按需请模型判断。节点说明里「多个来源汇总成的"
"分析报告默认开结论句裁判」那条在问数里不适用,几个 agent 分头查的也一样\n"
"- 用户要的是周报、要三档出具时,按搭图规则配口径卡和 report_from 契约\n"
"- 不要用 transform 解析 agent / llm 的文字:要把 agent 查到的数交给下游,就给它配 output_schema + "
"cite_fields\n"
@@ -1323,6 +1338,7 @@ def authored_issues(spec: GraphSpec, sources: list[Any], baseline: list[dict[str
changed = _changed(spec, baseline)
out: list[dict[str, Any]] = []
out += _unsourced_reports(spec, changed)
out += _issuance_promises(spec, changed)
for issue in text_parse_issues(spec):
if issue.node_id in changed:
# 模型的文字放在模板引号里的那种,改法是 | json,不是换结构化输出
@@ -1356,6 +1372,36 @@ def authored_issues(spec: GraphSpec, sources: list[Any], baseline: list[dict[str
return out
#: 人工审批的标题、说明里不该出现的说法:出具要成果节点配了出具契约、从已发布版本发起的正式运行才谈得上
_ISSUANCE_PROMISE = "正式出具"
def _issuance_promises(spec: GraphSpec, changed: set[str]) -> list[dict[str, Any]]:
"""这一轮写的人工审批节点在标题、说明里许诺「正式出具」(human_formal_issuance)。
真实运行里一次探索运行的审批写着「批准后才会正式出具」,审批人以为自己在签发正式的东西。画布上试运行、问数据
都是探索运行,成果节点也常常没有出具契约;同一张图发布后发起的正式运行才谈得上出具。审批说明写「批准后输出」,
这次是哪一类运行由审批卡标明。"""
out: list[dict[str, Any]] = []
for node in spec.nodes:
if node.id not in changed or node.type != NodeType.HUMAN:
continue
field = next((key for key in ("title", "message")
if _ISSUANCE_PROMISE in str(node.config.get(key) or "")), None)
if field is None:
continue
out.append({
"level": "error", "node_id": node.id, "edge_id": None, "field": field, "code": "human_formal_issuance",
"message": f"人工审批「{node.title}」的{'标题' if field == 'title' else '说明'}写了「{_ISSUANCE_PROMISE}」。"
"只有成果节点配了出具契约、从已发布版本发起的正式运行才会出具,画布上试运行和问数据都是探索运行,"
"批准后只是输出。请改成「批准后输出」这类说法",
"for_model": f"人工审批的 {field} 里不要写「{_ISSUANCE_PROMISE}」:只有成果节点配了出具契约、从已发布版本发起的"
"正式运行才会出具,这张图在画布上试运行、问数据时都是探索运行,批准后只是输出。改成「批准后输出」"
"「批准后继续」这类说法,运行类别由审批卡标明",
})
return out
def _catalog_issues(node_id: str, sql: str, checker: Any) -> list[dict[str, Any]]:
"""一个调用工具节点的 SQL 对照数据目录查出来的问题,形状和别的自查问题一样(node_id、field 都在),
另带 table、column、relation_id、sql_excerpt 给界面定位。"""
@@ -1423,6 +1469,22 @@ _DATEPART_ARG = re.compile(
_WINDOW_NAME = re.compile(r"(?:\bwindow|,)\s*([A-Za-z_][\w$#]*)\s+as\s*\(", re.I)
def _blank_unicode_identifiers(sql: str) -> str:
"""带引号、名字里有非 ASCII 字符的标识符(中文别名 AS "月份"、AS `曝光量`)连引号一起换成空格。
这项核对只认 ASCII 的列名,中文名本来就不判;不抹掉的话,SQL 里一出现带引号的标识符就整条放弃核对,照「结果列
起中文别名」写的 SQL 会把猜错的英文列名一起放过去。英文的带引号标识符("StatDate")大小写、空格都可能,照旧不判。"""
from app.data.guard import _GENERIC, _tokens
out = []
for token in _tokens(sql, _GENERIC):
if token.kind == "ident" and not token.text.isascii():
out.append(" " * len(token.text))
else:
out.append(token.text)
return "".join(out)
def sql_unknown_columns(sql: str, source: Any) -> tuple[list[str], list[str]] | None:
"""SQL 里用到、数据源结构里查不到的列:(查不到的列, 这几张表的全部列)。判断不了、或者都查得到时返回 None。"""
from app.data import introspect as _introspect
@@ -1431,7 +1493,7 @@ def sql_unknown_columns(sql: str, source: Any) -> tuple[list[str], list[str]] |
cache = getattr(source, "schema_cache", None) or {}
if not cache.get("tables") or cache.get("failed") or first_verb(sql) != "select":
return None
text = re.sub(r"\{\{.*?\}\}", " 0 ", strip_comments(sql), flags=re.S)
text = _blank_unicode_identifiers(re.sub(r"\{\{.*?\}\}", " 0 ", strip_comments(sql), flags=re.S))
if '"' in text.replace("''", "") and _blank_quoted(text) != _blank_quoted(text, identifiers=False):
return None # 引号括起来的标识符:大小写、空格都可能,不猜
text = _blank_quoted(text)
+18 -5
View File
@@ -99,6 +99,10 @@ SQL_RULES_PROMPT = (
"去对结构,对不上会打回来让你改\n"
"- 比率、增幅、占比这类派生计算进口径卡(metrics 节点);查询结果里原样的数,报告撰写节点(report)"
"可以直接用 [[v:Q1.r0.列名]] 引用。别让 llm 节点直接对数字做算术\n"
# 真实运行里报告表格的表头全是英文缩写的原始列名:[[table:]] 原样用查询结果的列名
"- 结果列用中文别名(AS 月份、AS 曝光量),报告里的表格表头、引用直接可读。按上面标的方言加标识符引号:"
"MySQL / MariaDB 用反引号(AS `月份`),PostgreSQL、Oracle、SQLite 用双引号(AS \"月份\"),不要用单引号"
"(那是字符串);只给最外层 SELECT 的结果列起别名,WHERE、JOIN、GROUP BY 里照旧写原列名\n"
)
PICK_SYSTEM = (
@@ -210,6 +214,11 @@ class SourcePlan:
"""给模型看的目录:去掉驳回项。"""
return {t: catalog.visible_notes(n) for t, n in self.raw_notes.items()}
@property
def catalog_tables(self) -> int:
"""有数据目录的表数:去掉驳回项之后还有内容的。整份被驳回的不算——挑表、SQL 检查都用不上它。"""
return sum(1 for n in self.raw_notes.values() if catalog.visible_notes(n))
def _modes(sources: list[Any]) -> list[Literal["all", "pick"]]:
"""每个源给详细摘要还是先挑表。
@@ -561,16 +570,20 @@ class DatasourceContext:
def op(self) -> dict[str, Any] | None:
"""告诉前端这一轮参考了哪些表:{"op": "context", "sources": [...], "elapsed_ms"?}。没有可查的表时为 None。
每个源一项:source 源名,tables 给了全字段的表(全名),selected_by,total 这个源一共几张表(表结构缓存里
记的真实总数);探查截断了的再带 explored(探查到几张),界面写「共 N 张,只探查了 200 张」;挑表失败的
带上 reason。只列了名字的表不进 tables:界面上说的「参考了 N 张表」是模型看得到字段的那些。
每个源一项:source 源名,source_id 源的 id(界面据此给去数据目录页的入口),tables 给了全字段的表(全名),
selected_by,total 这个源一共几张表(表结构缓存里记的真实总数),catalog_tables 有数据目录的表数(没有目录的
大库,界面提示先去起草目录:挑表和 SQL 检查都靠它);探查截断了的再带 explored(探查到几张),界面写「共 N 张,
只探查了 200 张」;挑表失败的带上 reason。只列了名字的表不进 tables:界面上说的「参考了 N 张表」是模型看得到
字段的那些。
"""
rows = []
for s in self.sources:
if not s.explored:
continue
row: dict[str, Any] = {"source": s.source.name, "tables": [s.qualified(t) for t in s.tables],
"selected_by": s.selected_by, "total": s.total}
row: dict[str, Any] = {"source": s.source.name, "source_id": getattr(s.source, "id", None),
"tables": [s.qualified(t) for t in s.tables],
"selected_by": s.selected_by, "total": s.total,
"catalog_tables": s.plan.catalog_tables}
if s.plan.unexplored:
row["explored"] = s.explored
if s.selected_by == "fallback" and self.reason:
+6
View File
@@ -56,6 +56,12 @@ class NodeContext:
writer = get_stream_writer()
except Exception: # noqa: BLE001 - 不在图执行上下文里(比如单元测试)
return
if str(event_type) == EventType.LLM_END and "unpriced" not in data:
from app.providers.catalog import unpriced_call
if unpriced_call(data):
# 金额估不出来(模型不在价格目录里):标出来,界面不把它的 0 当成「没花钱」
data = {**data, "unpriced": True}
writer(
{
"__agentlab__": True,
+429 -37
View File
@@ -47,7 +47,9 @@ import json
import math
import re
import unicodedata
from collections import Counter
from collections.abc import Callable
from dataclasses import dataclass, field
from decimal import ROUND_HALF_UP, Context, Decimal
from typing import Any, Literal, TypedDict
@@ -57,6 +59,7 @@ from app.data.names import FILLERS, name_key
from app.data.provenance_types import DOC_PROVENANCE
from app.data.tabular import UNSHAPED_NOTE
from app.engine.expressions import CellError, cell_value, column_kind, locate_cell
from app.engine.guards import FALLBACK_CONTEXT
from app.engine.issuance import _tolerance, extract_numbers, number_allowance
from app.engine.labels import option_label
@@ -69,6 +72,15 @@ DOC_SCHEMA = "agentlab.report/1"
#: 和记下的状态对不上,会被当成文档被改过(state_mismatch)。
ENTITY_SYNTAX = 2
ENTITY_SYNTAXES = frozenset({1, 2})
#: 单元格显示规则的版本(文档顶层 cell_display):
#: - 1:照快照里的原样显示,ISO 日期时间带 T(2026-06-30T08:21:39)。没有这个字段的老文档按它复核
#: - 2:ISO 日期时间显示成「2026-06-30 08:21:39」(小数秒、时区照留)。只改显示:快照里的值、引用记下的 value、
#: eid、证据下钻都不变
#: 新组装的文档里真有日期时间被改写时才记 cell_display: 2(参与内容哈希),没有的文档和以前一字不差。复核必须按
#: 文档生成时的那一版:老文档存的是带 T 的字,拿新规则重新渲染对不上,会被当成文档被改过(render_mismatch)
CELL_DISPLAY = 2
CELL_DISPLAYS = frozenset({1, 2})
_ISO_DATETIME = re.compile(r"^(\d{4}-\d{2}-\d{2})T(\d{2}:\d{2}(?::\d{2}(?:\.\d+)?)?(?:Z|[+-]\d{2}:?\d{2})?)$")
# --------------------------------------------------------------------------
@@ -1270,7 +1282,8 @@ class _Snapshots:
compose / verify / 流式渲染里:出口复核是另一次调用,照样重新取、重新验。
"""
def __init__(self, loader: Loader | None = None, *, entities: bool = True, syntax: int | None = None) -> None:
def __init__(self, loader: Loader | None = None, *, entities: bool = True, syntax: int | None = None,
display: int | None = None) -> None:
self.loader = loader or load_artifact
self.memo: dict[str, tuple[Any, str | None]] = {}
self._index: tuple[dict[str, Any], _EntityIndex] | None = None
@@ -1278,6 +1291,17 @@ class _Snapshots:
self.entities_on = entities
#: 这一次解析按哪一版实体语法:组装、流式渲染用当前版,复核按文档记下的版本(verify_doc)
self.syntax = ENTITY_SYNTAX if syntax is None else syntax
#: 单元格按哪一版显示规则渲染(CELL_DISPLAY):组装、流式渲染用当前版,复核按文档记下的版本
self.display = CELL_DISPLAY if display is None else display
#: 这一次解析里有没有哪一格因为显示规则改了写法:组装时据此决定文档记不记 cell_display
self.reformatted = False
def shown(self, raw: Any) -> Any:
"""按显示规则要显示的原值:2 版起 ISO 日期时间的 T 换成空格。没改就是原值本身。"""
if self.display >= 2 and isinstance(raw, str) and (m := _ISO_DATETIME.match(raw.strip())):
self.reformatted = True
return f"{m.group(1)} {m.group(2)}"
return raw
def entities(self, catalog: dict[str, Any]) -> _EntityIndex:
if self._index is None or self._index[0] is not catalog:
@@ -1286,11 +1310,12 @@ class _Snapshots:
self._index = (catalog, _EntityIndex(shown, syntax=self.syntax))
return self._index[1]
def with_syntax(self, syntax: int) -> "_Snapshots":
"""换一版实体语法的同一次解析。取过的快照共用:快照内容和语法版本无关,不必再取、再验一遍。"""
if syntax == self.syntax:
def with_syntax(self, syntax: int, display: int | None = None) -> "_Snapshots":
"""换一版实体语法(和单元格显示规则)的同一次解析。取过的快照共用:快照内容和版本无关,不必再取、再验一遍。"""
display = self.display if display is None else display
if syntax == self.syntax and display == self.display:
return self
twin = _Snapshots(self.loader, entities=self.entities_on, syntax=syntax)
twin = _Snapshots(self.loader, entities=self.entities_on, syntax=syntax, display=display)
twin.memo = self.memo
return twin
@@ -1336,11 +1361,11 @@ class _Snapshots:
def _snapshots(loader: Loader | _Snapshots | None, *, entities: bool = True,
syntax: int | None = None) -> _Snapshots:
"""loader 已经是 _Snapshots 时沿用它(entities 不看);给了 syntax 就换成那一版。"""
syntax: int | None = None, display: int | None = None) -> _Snapshots:
"""loader 已经是 _Snapshots 时沿用它(entities 不看);给了 syntax、display 就换成那一版。"""
if isinstance(loader, _Snapshots):
return loader if syntax is None else loader.with_syntax(syntax)
return _Snapshots(loader, entities=entities, syntax=syntax)
return loader if syntax is None and display is None else loader.with_syntax(syntax or loader.syntax, display)
return _Snapshots(loader, entities=entities, syntax=syntax, display=display)
_CELL_REF = re.compile(r"^(?P<alias>[QK]\d+)\.r(?P<row>\d+)\.(?P<column>.+)$")
@@ -1390,7 +1415,7 @@ def _resolve_cell(marker: dict[str, Any], catalog: dict[str, Any], snaps: _Snaps
try:
raw, column = locate_cell(snapshot, parsed["row"], parsed["column"])
kind = column_kind(snapshot, column)
rendered = render_cell(raw, conv, kind=kind)
rendered = render_cell(snaps.shown(raw), conv, kind=kind)
except CellError as e:
return _unresolved(marker, alias, "cell", f"{alias} {e}")
except RenderError as e:
@@ -2471,11 +2496,13 @@ def compose_doc(
cells_allowed: bool = True,
loader: Loader | None = None,
entities: bool = True,
absence: bool = False,
) -> dict[str, Any]:
"""模型写的原文(带标记)→ 报告文档:渲染后的 markdown、块、句、片段、统计、违规。
entities=False(报告节点 entities: off):表名、字段名整层不管——不链接、不核对反引号,
[[t:]] / [[c:]] 解析不了,原因是 ENTITIES_OFF_REASON。
absence=True(写作目录有省略):另查缺失断言(absence_claims),和句中的 [[see:]] 一样只在组装时查。
stats / violations 由 verify_doc 算出,和出口契约复核用的是同一个函数。
整表标记先展开成每格一个 [[v:]] 的 Markdown 表,之后和模型自己写的表格走同一条路;
@@ -2625,14 +2652,18 @@ def compose_doc(
# entity_syntax:这份文档按哪一版实体语法组装,复核照它走(verify_doc)。在文档顶层,参与内容哈希
# provenance:期 4 的文档标记(P4-SPEC 1.4)。带它的文档才有查询步骤的提示字段、推断来源、裁判摘录的新行;
# 没有它的(期 4 之前组装的)一律按原样展示。写在顶层、参与内容哈希,事后补不上:补了哈希就变,封存链就断
# cell_display:有单元格因为显示规则改了写法(ISO 日期时间)才记,复核照它渲染;没有的文档和以前一字不差
doc: dict[str, Any] = {
"schema": DOC_SCHEMA, "entity_syntax": snaps.syntax, "provenance": DOC_PROVENANCE,
**({"cell_display": snaps.display} if snaps.reformatted and snaps.display != 1 else {}),
"run_id": run_id, "node_id": node_id,
"markdown": markdown, "source": source, "catalog": kept, "blocks": out_blocks,
}
checked = verify_doc(doc, catalog, allow_numbers=allow_numbers, cells_allowed=cells_allowed, loader=snaps)
# 句中的 [[see:]] 只在组装时查(交回写作者重写):出口复核看的是数字和出处,句子通不通不影响出具
doc["stats"], doc["violations"] = checked["stats"], checked["violations"] + inline_see(source)
# 句中的 [[see:]]、目录有省略时的缺失断言只在组装时查(交回写作者重写):出口复核看的是数字和出处,
# 句子通不通、结论有没有越过证据不影响出具
doc["stats"], doc["violations"] = checked["stats"], \
checked["violations"] + inline_see(source) + (absence_claims(doc) if absence else [])
return doc
@@ -2778,7 +2809,8 @@ def verify_doc(
有知识库检索时才有(它们才可能不是 0):升级前的运行统计的键和以前一模一样。
"""
entity_syntax = doc_entity_syntax(doc)
snaps = _snapshots(loader, entities=entities, syntax=entity_syntax or ENTITY_SYNTAX)
display = doc_cell_display(doc)
snaps = _snapshots(loader, entities=entities, syntax=entity_syntax or ENTITY_SYNTAX, display=display or CELL_DISPLAY)
index = snaps.entities(catalog)
violations: list[dict[str, Any]] = []
uncited: list[dict[str, Any]] = []
@@ -2796,6 +2828,8 @@ def verify_doc(
if entity_syntax is None:
violations.append(_violation("bad_schema",
f"报告文档记录的表名、字段名规则版本「{doc.get('entity_syntax')}」无法识别"))
if display is None:
violations.append(_violation("bad_schema", f"报告文档记录的单元格显示规则版本「{doc.get('cell_display')}」无法识别"))
markdown = str(doc.get("markdown") or "")
masks: list[tuple[int, int]] = [] # 引用渲染出来的字
@@ -2942,6 +2976,13 @@ def verify_doc(
return {"ok": not violations, "violations": violations, "stats": stats, "uncited": uncited}
def doc_cell_display(doc: Any) -> int | None:
"""文档按哪一版单元格显示规则组装:没有 cell_display 字段的是 1 版(老文档、没有日期时间被改写的文档),
记了认不出的值返回 None。"""
raw = doc.get("cell_display", 1) if isinstance(doc, dict) else 1
return raw if type(raw) is int and raw in CELL_DISPLAYS else None
def doc_entity_syntax(doc: Any) -> int | None:
"""文档按哪一版实体语法组装:没有 entity_syntax 字段的是老文档(1 版),记了认不出的值返回 None。"""
raw = doc.get("entity_syntax", 1) if isinstance(doc, dict) else 1
@@ -3032,6 +3073,58 @@ def inline_see(source: str) -> list[dict[str, Any]]:
return [_violation("inline_see", INLINE_SEE_MESSAGE, text=m.group(0)) for m in _INLINE_SEE.finditer(source or "")]
#: 写作目录有省略、或者有来源的结构化结论为空时,拼在写作规则后面(给模型看)
ABSENCE_RULE = (
"这次你看不到全部证据(证据目录有省略,或者有来源的结构化结论为空):不得断言「没有查到」「未查到」「不存在」"
"「缺失」「没有数据」「无数据」这类结论。某个方向的证据没有展开或为空时,只能写「本报告未覆盖……」,并说明依据不足"
"(比如「目录有省略,相关查询未展开」),句末可以挂 [[see:Qn]] 指向相关查询。"
)
#: 缺失断言:断言「没有数据」的几种说法。只认拿得准的,不泛化:「不存在明显差异」「缺少口径说明」这类说的是看到的
#: 数据里的情况,不算;「不存在」「为空」「缺失」只认跟在数据、记录、结果后面(或者前面)的
_ABSENCE = re.compile(
r"(?:没有|没|未|未能|没能)(?:查询|查)到|查不到|查无"
r"|(?:没有|暂无|并无)(?:任何)?(?:相关的?)?(?:数据|记录|结果)"
r"|无(?:任何)?(?:相关的?)?(?:数据|记录)(?![库源集表权字项列])"
r"|不存在(?:任何)?(?:相关的?)?(?:数据|记录|结果)|(?:数据|记录|结果)(?:均|都|全部)?(?:不存在|为空|缺失|空缺)"
)
#: 前面是这些词的不算:「不代表没有数据」「并非查不到」说的恰恰是不能下这个结论
_ABSENCE_NEGATED = re.compile(r"(?:不代表|并不代表|不等于|不意味着|并非|不是|不能说明|不能断言|不得断言|不应断言)[^。!?\n]{0,6}$")
ABSENCE_MESSAGE = _Reason(
"证据目录有省略时写了缺失断言:写作模型没有看到全部证据,「没有查到」「不存在」「缺失」这类结论依据不足",
"证据目录有省略(没展开的查询只列了编号和摘要),这一句却断言了没有数据。看不到的证据不能当成不存在:"
"改成「本报告未覆盖……」,并说明依据不足(目录有省略、相关查询未展开),不要写「没有查到」「不存在」「缺失」「没有数据」",
)
def absence_claims(doc: dict[str, Any]) -> list[dict[str, Any]]:
"""正文里的缺失断言(_ABSENCE),每处一条违规 absence_claim。只在写作目录有省略时由报告节点要求查(compose_doc 的
absence):目录全给了的时候,「本周没有退款记录」可能就是看了数据说的。
在渲染后的正文里找(标记已经换成了字),代码块不查。带着 span、所在片段和句子:证据面板能定位、能显示原句。
"""
markdown = str(doc.get("markdown") or "")
found: list[dict[str, Any]] = []
for block in doc.get("blocks") or []:
if block.get("type") == "code":
continue
for unit in block.get("units") or []:
span = unit.get("span")
if not (isinstance(span, list) and len(span) == 2):
continue
sentence = markdown[span[0]:span[1]]
for m in _ABSENCE.finditer(sentence):
if _ABSENCE_NEGATED.search(sentence[:m.start()]):
continue
at = span[0] + m.start()
seg = next((g for g in unit.get("segments") or [] if g.get("kind") != "structural"
and isinstance(g.get("span"), list) and g["span"][0] <= at < g["span"][1]), None)
found.append(_violation("absence_claim", ABSENCE_MESSAGE, text=m.group(0),
span=[at, span[0] + m.end()], context=sentence.strip()[:120],
segment=seg.get("id") if seg else None, unit=unit.get("id")))
return found
#: 引用查询结果的写法。只跟着目录里的查询出现(catalog_prompt):写作规则本身保持一期那份,升级前
#: 发起的运行跑到报告节点时提示一字不差;只有口径卡的目录也不该教写作者去引用不存在的 Q1
CELL_RULES = (
@@ -3094,10 +3187,12 @@ def _sql_check_note(artifact: Any, snaps: _Snapshots) -> str | None:
"引用时须如实说明,不要当作确定的结论)")
def _query_prompt(entry: dict[str, Any], snaps: _Snapshots, cells_allowed: bool) -> list[str]:
def _query_prompt(entry: dict[str, Any], snaps: _Snapshots, cells_allowed: bool,
shown_rows: int = _PROMPT_ROWS) -> list[str]:
"""一次查询在写作目录里的样子:编号、来源、列,以及能引用的行(行号 + 渲染后的值)。
值按 [[v:]] 的渲染规则写出来:写作者在这里看到的,就是引用之后报告里显示的字。
值按 [[v:]] 的渲染规则写出来:写作者在这里看到的,就是引用之后报告里显示的字。shown_rows 是列出几行:
目录放不下时先给每次查询列前几行(_PROMPT_COMPACT_ROWS),再补到 _PROMPT_ROWS。
"""
alias = entry["alias"]
cols = entry.get("columns") or []
@@ -3115,13 +3210,13 @@ def _query_prompt(entry: dict[str, Any], snaps: _Snapshots, cells_allowed: bool)
columns = [str(c) for c in snapshot.get("columns") or []][:_PROMPT_COLS]
rows = snapshot["rows"]
example = None
for r, record in enumerate(rows[:_PROMPT_ROWS]):
for r, record in enumerate(rows[:shown_rows]):
cells = []
for c in columns:
try:
raw, _ = locate_cell(snapshot, r, c)
kind = column_kind(snapshot, c)
shown, numeric = render_cell(raw, kind=kind), is_number(cell_value(raw, kind))
shown, numeric = render_cell(snaps.shown(raw), kind=kind), is_number(cell_value(raw, kind))
except (CellError, RenderError):
shown, numeric = MISSING, False
cells.append(f"{c}={shown}")
@@ -3130,22 +3225,85 @@ def _query_prompt(entry: dict[str, Any], snaps: _Snapshots, cells_allowed: bool)
lines.append(f" r{r}:" + ",".join(cells))
if not rows:
lines.append(" (0 行,没有可以引用的格)")
elif len(rows) > _PROMPT_ROWS:
lines.append(f" …共 {len(rows)} 行,这里只列出前 {_PROMPT_ROWS} 行;后面的行也能按行号引用")
elif len(rows) > shown_rows:
lines.append(f" …共 {len(rows)} 行,这里只列出前 {shown_rows} 行;后面的行也能按行号引用")
if rows and columns:
lines.append(f" 例:[[v:{alias}.r0.{example or columns[0]}]]")
return lines
def catalog_prompt(catalog: dict[str, Any], *, budget: int = 12000, cells_allowed: bool = True,
loader: Loader | None = None) -> str:
"""证据目录的文字版,放进写作者的 prompt。超过预算就截断,并照实说截断了。
#: 写作目录的字数预算(catalog_budget)按写作模型的上下文窗口定,不再写死 12000:
#: - 占窗口的 40%。其余留给写作规则和指令(约 4 千字)、重写轮次(每轮把上一稿和违规清单追加进对话,最多 3 轮)、
#: 推理模型的思考和报告正文(输出额度)。字数按 1 字 ≈ 1 token 算:中文一个字约 0.6–1 个 token,目录里的数字、
#: 列名、标点每个字更少,按 1:1 算偏保守,实际占用低于 40%
#: - 拿不到窗口时按 128K(和 Agent 护栏的 FALLBACK_CONTEXT 同一个保守值),即 51200 字。原来的 12000 字只放得下
#: 20 多次查询:2026-10 一次上百次查询的运行,十万字以上的目录被截到一万两千字以内,写作者只看到前二十来次
#: - 下限 12000:窗口很小(32K 以下)的本地模型维持原来的篇幅;上限 160000:百万窗口的模型每写一稿、每重写一次
#: 都要重发整份目录,花费随之上涨,而 160000 字已经放得下那次运行的查询全部展开
CATALOG_WINDOW_SHARE = 0.4
CATALOG_MIN_CHARS = 12_000
CATALOG_MAX_CHARS = 160_000
#: 目录放不下时,先给每次查询列几行
_PROMPT_COMPACT_ROWS = 3
#: 放不下时表和字段、知识库检索两节合起来最多占预算的几分之一(剩下的给查询)
_PROMPT_TAIL_SHARE = 4
_PROMPT_CUT = "\n…(目录太长,后面的省略了)"
查询结果会列出能引用的行(要读快照);cells_allowed=False 时只列编号,说明只能当依据。
沙箱代码节点不列:它的产出不能引用。
def catalog_budget(window: int | None) -> int:
"""写作目录的字数预算:写作模型上下文窗口(token)的 40%,夹在上下限之间;拿不到窗口按 FALLBACK_CONTEXT。"""
size = window if isinstance(window, int) and not isinstance(window, bool) and window > 0 else FALLBACK_CONTEXT
return max(CATALOG_MIN_CHARS, min(CATALOG_MAX_CHARS, int(size * CATALOG_WINDOW_SHARE)))
DEFAULT_CATALOG_BUDGET = catalog_budget(None)
#: 目录有省略时,开头跟在省略情况后面的约定(给模型看)。和数字核对一致:没展开的查询只能当依据,数照样逐字核对
OMITTED_RULES = (
"没展开的查询照样在证据台账里,只是这里没有列出它的数据:\n"
"- 不要据此断言「没有查到」「不存在」「缺失」「没有数据」,只能写「本报告未覆盖」,并说明依据不足;\n"
"- 要用没展开的查询,只能在句末写 [[see:Qn]] 作依据(Qn 换成它的编号);不要写 [[v:]]、[[table:]] 去猜它的行和列,"
"也不要凭摘要写任何数字——数字照样逐字核对,没有出处的数会被打回重写。结构化结论里写好的引用标记可以照抄。"
)
#: 结构化结论为空的来源后面跟的那句(给模型看)
EMPTY_SOURCE_PROMPT = ("结构化结论为空(字段都记为空值)。这不代表没有数据:不得据此断言没有查到、不存在或缺失,"
"只能写本报告未覆盖")
@dataclass
class WriterCatalog:
"""给写作者的出处清单(writer_catalog):文字,以及省略了什么。
- trimmed:放不下、有省略;nodes 是各来源的查询数 [{node_id, title, total, full, compact, omitted}],
full 列出前 20 行、compact 只列前几行、omitted 只列编号和摘要;omitted 是只列摘要的查询编号
- empty_sources:结构化结论为空的来源(Agent 的名字)
- compact_rows:只列前几行的查询列了几行
清单有省略、或有来源的结构化结论为空时,报告节点加写作规则(ABSENCE_RULE);有省略时组装文档另查缺失断言
"""
snaps = _Snapshots(loader)
lines = ["可引用的证据(写数字只能用下面的引用标记,系统会换成真实数值):"]
text: str
budget: int
queries: int = 0
trimmed: bool = False
nodes: list[dict[str, Any]] = field(default_factory=list)
omitted: list[str] = field(default_factory=list)
empty_sources: list[str] = field(default_factory=list)
compact_rows: int = _PROMPT_COMPACT_ROWS
@property
def compact(self) -> int:
"""列出了数据、但只列了前 compact_rows 行的查询次数。"""
return sum(int(r.get("compact") or 0) for r in self.nodes)
@property
def guarded(self) -> bool:
"""写作者看不全证据:缺失断言没有依据。"""
return self.trimmed or bool(self.empty_sources)
def _metric_prompt(catalog: dict[str, Any]) -> list[str]:
"""口径卡指标和运行输入在写作目录里的样子。"""
lines: list[str] = []
groups: dict[tuple[str, str, str], list[dict[str, Any]]] = {}
for entry in catalog.values():
if entry.get("kind") == "metric":
@@ -3172,23 +3330,257 @@ def catalog_prompt(catalog: dict[str, Any], *, budget: int = 12000, cells_allowe
if inputs:
lines.append("\n运行输入:")
lines.extend(f"- [[{e['alias']}]] = {e.get('rendered')}" for e in inputs)
return lines
def _dig_path(data: Any, path: str) -> Any:
for key in path.split("."):
data = data.get(key) if isinstance(data, dict) else None
return data
def _field_prompt(path: str, value: Any, ev: dict[str, Any], aliases: dict[str, str], snaps: _Snapshots,
cells_allowed: bool) -> str:
"""一个结构化字段在写作目录里的那一行:值(按 [[v:]] 的渲染规则)和引用它的标记。"""
status = ev.get("status")
if status == "missing":
return f"- {path}:抽取时没有找到这个值(记为空,不代表库里没有)"
if status not in ("verified", "mismatch"):
return f"- {path}:出处核对不了(记为空)"
locator = ev.get("locator") if isinstance(ev.get("locator"), dict) else {}
alias = aliases.get(str(ev.get("artifact") or ""))
if alias is None:
return f"- {path}:所依据的查询不在这份目录里,不能引用"
if not cells_allowed:
return f"- {path}:出自 {alias},只能在句末写 [[see:{alias}]] 作依据"
if isinstance(locator.get("rows"), list) and len(locator["rows"]) == 2:
first, last = locator["rows"]
named = locator.get("columns") if isinstance(locator.get("columns"), dict) else {"": locator.get("column")}
cols = ",".join(dict.fromkeys(str(c) for c in named.values() if c))
many = f"(共 {last - first + 1} 行,整表最多 {TABLE_MAX_ROWS} 行,要分成几张表)" \
if last - first + 1 > TABLE_MAX_ROWS else ""
return (f"- {path}:数组,出自 {alias} 第 {first}–{last} 行,可以单独一行写 "
f"[[table:{alias} cols={cols} rows={first}-{last}]]{many}")
row, column = locator.get("row"), locator.get("column")
shown = MISSING
snapshot, why = snaps.get(alias, ev.get("artifact"))
try:
if why:
raise CellError(why)
raw, column = locate_cell(snapshot, row, column)
shown = render_cell(snaps.shown(raw), kind=column_kind(snapshot, column))
except (CellError, RenderError):
try:
shown = render_cell(value)
except RenderError:
pass
return f"- {path} = {shown}:[[v:{alias}.r{row}.{column}]]"
def _fields_prompt(sources: list[dict[str, Any]], catalog: dict[str, Any], snaps: _Snapshots,
cells_allowed: bool) -> tuple[list[str], list[str], list[str]]:
"""各 Agent 经按出处核对交来的结构化字段:(写作目录里的几行, 结构化结论为空的来源, 字段引用到的查询编号)。
字段的出处记的是 Agent 自己的编号(它的第几次查询),这里换成报告目录里的全局编号:按快照工件认,同一个节点的
优先(工件内容寻址,两次一模一样的查询共用一份快照,指哪一次都是同一格)。
"""
mine: dict[tuple[str, str], str] = {}
anyone: dict[str, str] = {}
for e in catalog.values():
if e.get("kind") == "query":
mine.setdefault((str(e.get("node_id") or ""), str(e.get("artifact") or "")), e["alias"])
anyone.setdefault(str(e.get("artifact") or ""), e["alias"])
lines: list[str] = []
empty: list[str] = []
cited: list[str] = []
for source in sources:
evidence = source.get("evidence")
if not isinstance(evidence, dict) or not evidence:
continue
node_id = str(source.get("node_id") or "")
title = str(source.get("title") or node_id)
data = source.get("data") if isinstance(source.get("data"), dict) else {}
if all(_dig_path(data, path) is None for path in evidence):
empty.append(title)
lines.append(f"「{title}」:{EMPTY_SOURCE_PROMPT}")
continue
aliases = {artifact: mine.get((node_id, artifact)) or alias for artifact, alias in anyone.items()}
lines.append(f"「{title}」:")
for path, ev in evidence.items():
ev = ev if isinstance(ev, dict) else {}
lines.append(_field_prompt(path, _dig_path(data, path), ev, aliases, snaps, cells_allowed))
if ev.get("status") in ("verified", "mismatch") and (alias := aliases.get(str(ev.get("artifact") or ""))):
cited.append(alias)
if not lines:
return [], [], []
return (["\n各 Agent 交来的结构化结论(系统已逐格核对到查询快照;要写这些数,用后面的引用标记):", *lines],
empty, list(dict.fromkeys(cited)))
def _query_brief(entry: dict[str, Any], tables: list[str], snaps: _Snapshots) -> str:
"""没展开的查询在目录里的一行摘要:编号、查的表、行数。
表先看目录里的表条目(带表结构快照的查询才有),没有就从查询快照里存的 SQL 认(sql_tables),
再没有写工具名。
"""
if not tables:
snapshot, why = snaps.get(entry["alias"], entry.get("artifact"))
if not why and isinstance(snapshot.get("sql"), str):
tables = sql_tables(snapshot["sql"])
rows = entry.get("rows")
what = ("、".join(tables[:3]) + (f" 等 {len(tables)} 张表" if len(tables) > 3 else "")) if tables \
else str(entry.get("tool") or entry.get("source") or "查询")
size = f"{rows} 行" if isinstance(rows, int) else "行数不明"
return f" - {entry['alias']}:{what},{size}" + ("(结果已截断)" if entry.get("truncated") else "")
def _cut_lines(lines: list[str], limit: int, note: str) -> list[str]:
"""从头取到放得下 limit 个字为止,后面换成一句 note。"""
out: list[str] = []
used = len(note) + 1
for line in lines:
if used + len(line) + 1 > limit:
return [*out, note]
out.append(line)
used += len(line) + 1
return out
def writer_catalog(catalog: dict[str, Any], *, budget: int = DEFAULT_CATALOG_BUDGET, cells_allowed: bool = True,
loader: Loader | None = None, fields: list[dict[str, Any]] | None = None,
titles: dict[str, str] | None = None) -> WriterCatalog:
"""证据目录的文字版,放进写作者的 prompt,连同省略了什么(WriterCatalog)。
查询结果会列出能引用的行(要读快照);cells_allowed=False 时只列编号,说明只能当依据。
沙箱代码节点不列:它的产出不能引用。fields 是上游 Agent 经按出处核对交来的结构化字段
[{node_id, title, data, evidence}],titles 是节点在画布上的名字(分组标题用)。
放得下就整份给(没有结构化字段时和以前一字不差)。放不下时按优先级、按来源均衡(_trimmed_catalog):
口径卡指标、运行输入、结构化字段一律列出;查询按产出它的节点分组、轮流展开;没展开的列编号和摘要;
开头照实说目录有省略、各来源省略了几次。
"""
snaps = _Snapshots(loader)
head = ["可引用的证据(写数字只能用下面的引用标记,系统会换成真实数值):"]
field_lines, empty, cited = _fields_prompt(fields or [], catalog, snaps, cells_allowed)
first = [*_metric_prompt(catalog), *field_lines]
queries = [e for e in catalog.values() if e.get("kind") == "query"]
if queries:
lines.append(f"\n{CELL_RULES}" if cells_allowed else
"\n查询结果(这次出具不允许直接引用单元格,只能写在 [[see:Q1]] 这样的依据里):")
for e in queries:
lines.extend(_query_prompt(e, snaps, cells_allowed))
rules = ([f"\n{CELL_RULES}" if cells_allowed else
"\n查询结果(这次出具不允许直接引用单元格,只能写在 [[see:Q1]] 这样的依据里):"] if queries else [])
tail: list[str] = []
if (index := snaps.entities(catalog)).active:
lines.extend(_entity_prompt(catalog, partial=index.partial, snaps=snaps))
tail.extend(_entity_prompt(catalog, partial=index.partial, snaps=snaps))
retrievals = [e for e in catalog.values() if e.get("kind") == "retrieval"]
if retrievals:
lines.append(f"\n{QUOTE_RULES}")
tail.append(f"\n{QUOTE_RULES}")
for e in retrievals:
lines.extend(_retrieval_prompt(e, snaps))
text = "\n".join(lines)
tail.extend(_retrieval_prompt(e, snaps))
full = {e["alias"]: _query_prompt(e, snaps, cells_allowed) for e in queries}
text = "\n".join([*head, *first, *rules, *(line for e in queries for line in full[e["alias"]]), *tail])
if len(text) <= budget:
return WriterCatalog(text=text, budget=budget, queries=len(queries), empty_sources=empty)
return _trimmed_catalog(catalog, head, first, rules, queries, full, tail, budget=budget, snaps=snaps,
cells_allowed=cells_allowed, titles=titles or {}, empty=empty, pinned=cited)
def _trimmed_catalog(catalog: dict[str, Any], head: list[str], first: list[str], rules: list[str],
queries: list[dict[str, Any]], full: dict[str, list[str]], tail: list[str], *, budget: int,
snaps: _Snapshots, cells_allowed: bool, titles: dict[str, str], empty: list[str],
pinned: list[str]) -> WriterCatalog:
"""放不下时的目录。以前从头截断:查询按编号排,前面的节点占满预算,后面的节点一次都不露面。
1. 口径卡指标、运行输入、结构化字段一律列出(第一优先)
2. 表和字段、知识库检索两节合起来最多占预算的 1/_PROMPT_TAIL_SHARE
3. 每次查询先只占一行摘要;结构化字段引用到的查询先展开(pinned),再按产出它的节点分组、轮流展开:
第一轮每次列前 _PROMPT_COMPACT_ROWS 行,第二轮补到 _PROMPT_ROWS 行,放不下为止
4. 开头写明目录有省略、各来源省略了几次,以及没展开的查询怎么用(OMITTED_RULES)
最后整份量一次:超了就把最后展开的退回去;指标和字段本身就超(极少见)才从尾部截断。
"""
if tail and len("\n".join(tail)) > budget // _PROMPT_TAIL_SHARE:
tail = _cut_lines(tail, budget // _PROMPT_TAIL_SHARE, "…(表和字段、知识库检索太长,后面的省略了)")
tables: dict[str, list[str]] = {}
for e in catalog.values():
if e.get("kind") == "table":
for alias in e.get("queries") or []:
tables.setdefault(alias, []).append(str(e.get("name")))
groups: dict[str, list[dict[str, Any]]] = {}
for e in queries:
groups.setdefault(str(e.get("node_id") or ""), []).append(e)
compact = {e["alias"]: _query_prompt(e, snaps, cells_allowed, _PROMPT_COMPACT_ROWS) for e in queries}
brief = {e["alias"]: [_query_brief(e, tables.get(e["alias"], []), snaps)] for e in queries}
shapes = {"brief": brief, "compact": compact, "full": full}
tier = {e["alias"]: "brief" for e in queries}
def rows() -> list[dict[str, Any]]:
out = []
for nid, entries in groups.items():
got = Counter(tier[e["alias"]] for e in entries)
out.append({"node_id": nid, "title": titles.get(nid) or nid or "其他", "total": len(entries),
"full": got["full"], "compact": got["compact"], "omitted": got["brief"]})
return out
def assemble() -> str:
stats = rows()
per = "、".join(f"{r['title']} {r['total']} 次(展开 {r['full'] + r['compact']} 次,省略 {r['omitted']} 次)"
for r in stats)
n = {k: sum(r[k] for r in stats) for k in ("full", "compact", "omitted")}
notice = (f"注意:证据太多,这份目录有省略(篇幅上限约 {budget} 字)。共 {len(queries)} 次查询:"
f"{n['full']} 次列出前 {_PROMPT_ROWS} 行,{n['compact']} 次只列前 {_PROMPT_COMPACT_ROWS} 行,"
f"{n['omitted']} 次只列编号和摘要。各来源:{per}。")
body = [*head, notice, OMITTED_RULES, *first, *rules]
for r, entries in zip(stats, groups.values()):
body.append(f"\n来自「{r['title']}」的查询(共 {r['total']} 次,展开 {r['full'] + r['compact']} 次):")
for e in entries:
if tier[e["alias"]] != "brief":
body.extend(shapes[tier[e["alias"]]][e["alias"]])
if r["omitted"]:
body.append(f" 没展开({r['omitted']} 次,只列编号和摘要):")
body.extend(brief[e["alias"]][0] for e in entries if tier[e["alias"]] == "brief")
return "\n".join([*body, *tail])
def size(lines: list[str]) -> int:
return sum(len(line) + 1 for line in lines)
# 省略情况那句里的数字会变长几位:留一点余量,最后再整份量一次
remaining = budget - len(assemble()) - 8 * (len(groups) + 3)
upgraded: list[tuple[str, str]] = []
for alias in pinned:
if tier.get(alias) == "brief" and (delta := size(compact[alias]) - size(brief[alias])) <= remaining:
tier[alias] = "compact"
remaining -= delta
upgraded.append((alias, "brief"))
for target, source in (("compact", "brief"), ("full", "compact")):
moved = True
while moved:
moved = False
for entries in groups.values():
for e in entries:
alias = e["alias"]
if tier[alias] != source:
continue
delta = size(shapes[target][alias]) - size(shapes[source][alias])
if delta <= remaining:
tier[alias] = target
remaining -= delta
upgraded.append((alias, source))
moved = True
break
text = assemble()
while len(text) > budget and upgraded:
alias, source = upgraded.pop()
tier[alias] = source
text = assemble()
if len(text) > budget:
text = text[:budget].rsplit("\n", 1)[0] + "\n…(目录太长,后面的省略了)"
return text
text = text[:budget - len(_PROMPT_CUT)].rsplit("\n", 1)[0] + _PROMPT_CUT
return WriterCatalog(text=text, budget=budget, queries=len(queries), trimmed=True, nodes=rows(),
omitted=[e["alias"] for e in queries if tier[e["alias"]] == "brief"], empty_sources=empty)
def catalog_prompt(catalog: dict[str, Any], *, budget: int = DEFAULT_CATALOG_BUDGET, cells_allowed: bool = True,
loader: Loader | None = None, fields: list[dict[str, Any]] | None = None,
titles: dict[str, str] | None = None) -> str:
"""证据目录的文字版(writer_catalog 的文字)。超过预算时分优先级、按来源均衡,并照实说省略了什么。"""
return writer_catalog(catalog, budget=budget, cells_allowed=cells_allowed, loader=loader, fields=fields,
titles=titles).text
def unshaped_table(entry: dict[str, Any], fetch: Callable[[Any], Any]) -> bool:
+96 -70
View File
@@ -98,8 +98,12 @@ MISSING_MAX = 30
#: 判定的规则版本:新判的判定带着它(rule_version),也进 request_key。报告节点的断点里换了规则的旧判定不复用;
#: 按需裁判遇到早于当前规则的判定(没有版本、版本更小、旧取值 unsupported)重判一次
RULES_VERSION = 2
#: 一次调用最多判几句:再多,4096 个输出令牌装不下每句一条判定加理由
BATCH_SIZE = 40
#: 一次调用最多判几句。以前是 40:真实运行里推理模型拿着 40 句的摘录 30 秒没回来,整份报告一句都没判成
#: (同一个模型 5 句一批 3 秒多就回来了)。批小了,超时最多丢掉没回来的那几批
BATCH_SIZE = 8
#: 一波最多几批同时问:几批共用剩下的时长,句数上限内的句子一两波就判完。第一批单独问:接入调不通(鉴权、
#: 网关挂了)时只撞这一次,不让一整波一起失败
PARALLEL_BATCHES = 4
MAX_TOKENS = 4096
#: 证据摘录:SQL 前几个字、每次查询最多几行几列、每段知识库片段最多几个字、整份引用的检索给几段。
#: 列数上限只管没被引用的列:句子引用的格子所在的列一律给(遮罩的除外),再用别的列补满到这么多列
@@ -1180,80 +1184,102 @@ async def run_request(request: JudgeRequest, *, spec: ModelSpec, budget: Budget,
emit(EventType.LOG, level="warn", code="judge_unpriced", message=note)
spent = 0.0
width = 1
while pending:
remaining = None if budget.timeout_s is None else budget.timeout_s - (time.monotonic() - started)
if remaining is not None and remaining <= 0:
stop(pending, "timeout_s")
break
batch = pending[:BATCH_SIZE]
cost = request.cost(model_id, batch) if priced else None
if cost is not None:
caps = []
if budget.max_cost_usd is not None:
caps.append((budget.max_cost_usd - spent, "max_cost_usd"))
if budget.daily_max_usd is not None:
caps.append((budget.daily_max_usd - (await daily_spend())["usd"], "daily_max_usd"))
if caps:
cap, binding = min(caps, key=lambda x: x[0])
trimmed = False
while batch and (cost := request.cost(model_id, batch) or 0.0) > cap + 1e-12:
batch, trimmed = batch[:-1], True # 按优先级从后往前丢:先判挂了依据的
if not batch:
stop(pending, binding)
break
if trimmed:
# 这一批已经是剩下的钱装得下的最多句子,后面的句子更装不下
stop(pending[len(batch):], binding)
pending = pending[:len(batch)]
pending = pending[len(batch):]
charged = cost or 0.0
day = await _charge(charged, limit=budget.daily_max_usd if priced else None, priced=priced)
if day is None: # 估算之后、记账之前别的裁判先花掉了今天剩下的钱
stop([*batch, *pending], "daily_max_usd")
break
estimated += charged
began = time.perf_counter()
emit(EventType.LLM_START, model=model_id, structured=True, purpose="judge", message_count=2,
units=len(batch))
items: list[Any] | None = None
raws: list[BaseMessage] = []
problem: str | None = None
calls = 1
try:
async with asyncio.timeout(remaining):
items, raws, problem, calls = await _ask(model, request.messages(batch))
except TimeoutError:
problem = "timeout_s"
tokens = [_usage_of(r, model_id) for r in raws]
inp, out = sum(t["input_tokens"] for t in tokens), sum(t["output_tokens"] for t in tokens)
# 回复没带用量(有的网关不给、调用超时)就按估算记:宁可多记,不能让每日上限漏掉
actual = (pricing.estimate_cost(model_id, inp, out) if inp or out else charged) if priced else 0.0
await _settle(day, actual - charged)
spent += actual
usage["input_tokens"] += inp
usage["output_tokens"] += out
usage["calls"] += calls
usage["cost_usd"] += actual
end = {"model": model_id, "purpose": "judge", "duration_ms": int((time.perf_counter() - began) * 1000),
"input_tokens": inp, "output_tokens": out, "total_tokens": inp + out, "cost_usd": round(actual, 6),
"calls": calls, "units": len(batch)}
if problem is not None and problem != "format":
reason = "timeout_s" if problem == "timeout_s" else "error"
detail = problem.partition(":")[2]
emit(EventType.LLM_END, **end, error=detail or _why_unjudged(reason, label, click=click))
if reason == "error":
# 调用本身失败(鉴权、网关挂了):后面几批多半一样,不再一次次去撞
details["error"] = detail
stop([*batch, *pending], reason, detail)
# 一波最多 PARALLEL_BATCHES 批:逐批按金额上限定下来(这一波已经定下的按估算先占着),再一起问
wave: list[tuple[list[Candidate], float, Any]] = []
reserved = 0.0
while pending and len(wave) < width:
batch = pending[:BATCH_SIZE]
cost = request.cost(model_id, batch) if priced else None
if cost is not None:
caps = []
if budget.max_cost_usd is not None:
caps.append((budget.max_cost_usd - spent - reserved, "max_cost_usd"))
if budget.daily_max_usd is not None:
caps.append((budget.daily_max_usd - (await daily_spend())["usd"], "daily_max_usd"))
if caps:
cap, binding = min(caps, key=lambda x: x[0])
trimmed = False
while batch and (cost := request.cost(model_id, batch) or 0.0) > cap + 1e-12:
batch, trimmed = batch[:-1], True # 按优先级从后往前丢:先判挂了依据的
if not batch:
stop(pending, binding)
pending = []
break
if trimmed:
# 这一批已经是剩下的钱装得下的最多句子,后面的句子更装不下
stop(pending[len(batch):], binding)
pending = pending[:len(batch)]
pending = pending[len(batch):]
charged = cost or 0.0
day = await _charge(charged, limit=budget.daily_max_usd if priced else None, priced=priced)
if day is None: # 估算之后、记账之前别的裁判先花掉了今天剩下的钱
stop([*batch, *pending], "daily_max_usd")
pending = []
break
stop(batch, reason, detail)
continue
emit(EventType.LLM_END, **end)
if items is None:
stop(batch, "format")
continue
_take(items, batch, request, verdicts, judge=judge, post_seal=post_seal)
stop(batch, "missing")
estimated += charged
reserved += charged
wave.append((batch, charged, day))
if not wave:
break
began = time.perf_counter()
for batch, _, _ in wave:
emit(EventType.LLM_START, model=model_id, structured=True, purpose="judge", message_count=2,
units=len(batch))
tasks = [asyncio.create_task(_ask(model, request.messages(batch))) for batch, _, _ in wave]
done, late = await asyncio.wait(tasks, timeout=remaining)
for task in late:
task.cancel()
if late:
await asyncio.gather(*late, return_exceptions=True)
failed: str | None = None
for (batch, charged, day), task in zip(wave, tasks):
items: list[Any] | None = None
raws: list[BaseMessage] = []
problem: str | None = None
calls = 1
if task in done:
items, raws, problem, calls = task.result()
else:
problem = "timeout_s"
tokens = [_usage_of(r, model_id) for r in raws]
inp, out = sum(t["input_tokens"] for t in tokens), sum(t["output_tokens"] for t in tokens)
# 回复没带用量(有的网关不给、调用超时)就按估算记:宁可多记,不能让每日上限漏掉
actual = (pricing.estimate_cost(model_id, inp, out) if inp or out else charged) if priced else 0.0
await _settle(day, actual - charged)
spent += actual
usage["input_tokens"] += inp
usage["output_tokens"] += out
usage["calls"] += calls
usage["cost_usd"] += actual
end = {"model": model_id, "purpose": "judge", "duration_ms": int((time.perf_counter() - began) * 1000),
"input_tokens": inp, "output_tokens": out, "total_tokens": inp + out,
"cost_usd": round(actual, 6), "calls": calls, "units": len(batch)}
if problem is not None and problem != "format":
reason = "timeout_s" if problem == "timeout_s" else "error"
detail = problem.partition(":")[2]
emit(EventType.LLM_END, **end, error=detail or _why_unjudged(reason, label, click=click))
if reason == "error":
# 调用本身失败(鉴权、网关挂了):后面几批多半一样,不再一次次去撞
details["error"] = failed = detail
stop(batch, reason, detail)
continue
emit(EventType.LLM_END, **end)
if items is None:
stop(batch, "format")
continue
_take(items, batch, request, verdicts, judge=judge, post_seal=post_seal)
stop(batch, "missing")
if failed is not None:
stop(pending, "error", failed)
break
width = PARALLEL_BATCHES
for c in request.cands: # 兜底:每个候选都有判定
verdicts.setdefault(c.unit, _unjudged("missing", _why_unjudged("missing", label), judge=judge,
+67
View File
@@ -33,6 +33,8 @@ async def run_human(state: GraphState, ctx: NodeContext) -> dict[str, Any]:
}
if mode == "edit":
payload["draft"] = ctx.render(ctx.cfg("draft", ""), state)
if reports := _reviewed_reports(state, ctx):
payload["reports"] = reports
response = await ask(ctx, payload, **payload)
# 谁批的跟着事件走:时间线要写得出"张工驳回了",没署名就是 None,不再默认写"你"
@@ -71,6 +73,71 @@ async def run_human(state: GraphState, ctx: NodeContext) -> dict[str, Any]:
return updates
#: 模板里对某个节点产出的引用:{{ nodes.write.text }}、{{ vars.weekly }}
_NODE_REF = re.compile(r"\bnodes\s*\.\s*([\w-]+)")
_VAR_REF = re.compile(r"\bvars\s*\.\s*([\w-]+)")
#: 审批卡要的裁判摘要:判了几句、各判成什么、有没有判完。其余(上限、费用、模型名)留在证据面板里看
_JUDGE_KEYS = ("mode", "counts", "complete", "unjudged", "limits_hit")
def reviewed_report_ids(spec: Any, human: Any) -> list[str]:
"""一个人工审批节点审的是哪几份报告:上游直接连着的报告撰写节点,加上标题、说明里引用了正文的
({{ nodes.<id>.text }},或者报告的 assign_to 变量)。只看图,不看运行状态。
审批卡(_reviewed_reports)和报告节点(reviewed_by_human:要交给人审的报告,探索运行也在节点里判)
用的是同一套认法,两边说的「这份报告要人审」不会对不上。"""
from app.engine.schema import NodeType
config = human.config or {}
text = " ".join(json.dumps(config.get(key), ensure_ascii=False, default=str)
for key in ("title", "message", "context", "draft") if config.get(key))
# 报告节点的 assign_to 也能把正文交给下游:{{ vars.<assign_to> }} 和 {{ nodes.<id>.text }} 一样算引用
by_var = {str(n.config.get("assign_to")): n.id for n in spec.nodes
if n.type == NodeType.REPORT and n.config.get("assign_to")}
nodes = spec.node_map()
wanted = ([e.source for e in spec.edges if e.target == human.id] + _NODE_REF.findall(text)
+ [by_var[v] for v in _VAR_REF.findall(text) if v in by_var])
return [rid for rid in dict.fromkeys(wanted) if (n := nodes.get(rid)) is not None and n.type == NodeType.REPORT]
def reviewed_by_human(spec: Any, report_id: str) -> bool:
"""这份报告会不会交给人工审批(有审批节点审它,认法同 reviewed_report_ids)。"""
from app.engine.schema import NodeType
return any(report_id in reviewed_report_ids(spec, n) for n in spec.nodes if n.type == NodeType.HUMAN)
def _reviewed_reports(state: GraphState, ctx: NodeContext) -> list[dict[str, Any]]:
"""这次审批审的报告各自的核对概况(审的是哪几份见 reviewed_report_ids)。
真实运行里报告有一句结论和证据相矛盾,审批人批准时完全不知道:审批卡只有正文。这里带上报告节点产出里的
stats(同 report.checked)、结论句策略、裁判摘要和报告文档的工件 id,审批卡据此写核对概况、给打开证据的入口。
只认已经跑完、带着核对统计的报告;重放时 state 一样,算出来的也一样。"""
from app.engine.schema import NodeType
spec = ctx.run.spec
nodes = spec.node_map()
outputs = state.get("nodes") or {}
out: list[dict[str, Any]] = []
for rid in reviewed_report_ids(spec, ctx.node):
node = nodes.get(rid)
result = outputs.get(rid)
if node is None or node.type != NodeType.REPORT or not isinstance(result, dict) \
or not isinstance(result.get("stats"), dict):
continue
item: dict[str, Any] = {
"node_id": rid, "label": node.title,
"stats": {k: v for k, v in result["stats"].items() if isinstance(v, int) and not isinstance(v, bool)},
"claims": result.get("claims") or "off",
}
if isinstance(result.get("doc_artifact"), str) and result["doc_artifact"]:
item["doc_artifact"] = result["doc_artifact"]
if isinstance(result.get("judge"), dict):
item["judge"] = {k: result["judge"][k] for k in _JUDGE_KEYS if k in result["judge"]}
out.append(item)
return out
# --------------------------------------------------------------------------
# 结构化校验
# --------------------------------------------------------------------------
+3 -3
View File
@@ -467,10 +467,10 @@ def _claims(out: dict[str, Any], contract: dict[str, Any], payload: dict[str, An
里 ignore 一律当 degrade:没有哪道门禁查契约里的 claims,不能让它把用户要的底线拉低。
报告节点写的是 judge 时,判定在文档里(按哈希取回的那份:判定是模型给的,出口复算不了,只能信封存的
文档)。正式运行在节点里判过:证据相矛盾的(contradicted,旧文档里的 unsupported 同样算)按 on_unsupported
文档)。节点里判过的(正式运行,以及交给人工审批的探索运行):证据相矛盾的(contradicted,旧文档里的 unsupported 同样算)按 on_unsupported
判档(报告节点和契约取更严的);证据不足的(insufficient)记一条缺口——最多降档,不会不予出具,也不会
完整出具;没判完的(裁判没跑成、触顶)把裁判摘要里的缺口照抄进来——不能判完整出具,也不当成「矛盾」。
探索运行按需裁判,这里只标注。契约要 judge、报告节点却没开的,照实记缺口:出口替不了报告节点去裁判。
其余探索运行按需裁判,这里只标注。契约要 judge、报告节点却没开的,照实记缺口:出口替不了报告节点去裁判。
数的是这里重新核对出来的 uncited,不信文档里记的 cites。升级前写的报告(节点产出里没有 claims 这个键,
包括跨着升级还没跑完的运行)一律不管,契约写了也只记一条缺口:报告生成于升级前,结论句未核对。
@@ -520,7 +520,7 @@ def _claims(out: dict[str, Any], contract: dict[str, Any], payload: dict[str, An
inline = summary.get("mode") == "inline"
flagged: dict[str, list[dict[str, Any]]] = {s: [] for s in ("contradicted", "insufficient", UNSUPPORTED, "partial")}
if inline:
# 探索运行按需裁判的判定不在文档里(在封存之后追加的 evidence.judged 事件里),这里只数正式运行的
# 按需裁判的判定不在文档里(在封存之后追加的 evidence.judged 事件里),这里只数节点里判过的
markdown = str((doc or {}).get("markdown") or "")
for _, unit in iter_units(doc or {}):
verdict = unit.get("verdict")
+127 -22
View File
@@ -541,25 +541,66 @@ async def run_agent(state: GraphState, ctx: NodeContext) -> dict[str, Any]:
async def extract_step(payload: list[BaseMessage], schema: dict[str, Any]) -> dict[str, Any]:
# cite_fields 的结构化抽取:进 checkpoint,节点重放时不再调用、不再计费。失败也在这里收住、
# 作为结果返回——agent 的结论还在,字段记空值、发警告,而不是把整个节点赔进去。
# thinking 关掉:结构化输出靠强制工具调用,和思考模式不兼容,抽取也用不着思考
spec = model_spec.model_copy(update={
"thinking": "off", "max_tokens": max(int(ctx.cfg("max_tokens") or 0), 4096)})
started = time.perf_counter()
ctx.emit(EventType.LLM_START, model=model_id, structured=True, purpose="cite_fields",
message_count=len(payload))
try:
async with SessionLocal() as session:
extractor, _ = await get_chat_model(session, spec)
got = await extractor.with_structured_output(schema, include_raw=True).ainvoke(payload)
except Exception as e: # noqa: BLE001
# thinking 关掉:结构化输出靠强制工具调用,和思考模式不兼容,抽取也用不着思考。但经 OpenAI 兼容
# 接口接入的推理模型关不掉思考,推理照样算在额度里:额度见 EXTRACT_MAX_TOKENS,被截断时加大重试一次
budget = max(int(ctx.cfg("max_tokens") or 0), EXTRACT_MAX_TOKENS)
spent = {"input_tokens": 0, "output_tokens": 0, "cost_usd": 0.0, "calls": 0}
rejected: int | None = None # 服务方嫌太大、拒收的额度:之后重试不再超过它
retried = False
def count(usage: dict[str, Any]) -> None:
for key in ("input_tokens", "output_tokens", "calls"):
spent[key] += int(usage.get(key) or 0)
spent["cost_usd"] = round(spent["cost_usd"] + float(usage.get("cost_usd") or 0), 6)
while True:
spec = model_spec.model_copy(update={"thinking": "off", "max_tokens": budget})
started = time.perf_counter()
ctx.emit(EventType.LLM_START, model=model_id, structured=True, purpose="cite_fields",
message_count=len(payload), max_tokens=budget)
failure: Exception | None = None
raw: BaseMessage | None = None
parsed: Any = None
try:
async with SessionLocal() as session:
extractor, _ = await get_chat_model(session, spec)
got = await extractor.with_structured_output(schema, include_raw=True).ainvoke(payload)
raw, parsed = _split_structured(got)
usage = _usage_of(raw if raw is not None else AIMessage(content=""), model_id)
except Exception as e: # noqa: BLE001
failure, usage = e, _error_usage(e, model_id)
count(usage)
cut = truncated_error(failure) if failure is not None else (parsed is None and truncated_reply(raw))
elapsed = int((time.perf_counter() - started) * 1000)
if failure is not None and not cut and _budget_rejected(failure) \
and (lower := next((b for b in _EXTRACT_FALLBACKS if b < budget), None)) is not None:
# 服务方的输出上限比默认额度小(DeepSeek chat 系只收 8K):退一档再来,别因为额度提高反而抽不成
ctx.emit(EventType.LLM_END, agent=ctx.node.title, model=model_id, purpose="cite_fields",
duration_ms=elapsed, **usage,
error=f"模型接入不接受 {budget} token 的输出额度,已改用 {lower} token 重试")
rejected, budget = budget, lower
continue
bigger = min(budget * 2, EXTRACT_RETRY_CAP)
if cut and not retried and bigger > budget and (rejected is None or bigger < rejected):
ctx.emit(EventType.LLM_END, agent=ctx.node.title, model=model_id, purpose="cite_fields",
duration_ms=elapsed, **usage,
error=f"输出被截断:{budget} token 的输出额度用完了,结构化结论没有写完,"
f"已把额度加大到 {bigger} token 重试一次")
retried, budget = True, bigger
continue
if cut:
ctx.emit(EventType.LLM_END, agent=ctx.node.title, model=model_id, purpose="cite_fields",
duration_ms=elapsed, **usage,
error=f"输出被截断:{budget} token 的输出额度用完了,结构化结论没有写完")
return {"error": TRUNCATED_EXTRACT.format(budget=budget), "truncated": True,
"max_tokens": budget, **spent}
if failure is not None:
ctx.emit(EventType.LLM_END, agent=ctx.node.title, model=model_id, purpose="cite_fields",
duration_ms=elapsed, **usage, error=describe_exception(failure))
return {"error": f"结构化抽取失败:{explain_model_error(failure, model_id)}", **spent}
ctx.emit(EventType.LLM_END, agent=ctx.node.title, model=model_id, purpose="cite_fields",
duration_ms=int((time.perf_counter() - started) * 1000), error=describe_exception(e))
return {"error": f"结构化抽取失败:{explain_model_error(e, model_id)}"}
raw, parsed = _split_structured(got)
usage = _usage_of(raw if raw is not None else AIMessage(content=""), model_id)
ctx.emit(EventType.LLM_END, agent=ctx.node.title, model=model_id, purpose="cite_fields",
duration_ms=int((time.perf_counter() - started) * 1000), **usage)
return {"parsed": parsed, **usage}
duration_ms=elapsed, **usage)
return {"parsed": parsed, **spent}
final_text = ""
#: 模型把工具调用写成了文字:纠正过一次了没有
@@ -778,8 +819,10 @@ async def run_agent(state: GraphState, ctx: NodeContext) -> dict[str, Any]:
total_usage[key] += int(extracted.get(key) or 0)
total_usage["cost_usd"] = round(total_usage["cost_usd"] + float(extracted.get("cost_usd") or 0), 6)
cited = verify_cited_fields(extracted, fields, queries)
for code, message, names in _field_warnings(*cited, extracted.get("error")):
await once(ctx, EventType.LOG, level="warn", code=code, message=message, fields=names)
truncated = extracted.get("truncated") is True
for code, message, names in _field_warnings(*cited, extracted.get("error"), truncated=truncated):
extra = {"max_tokens": extracted.get("max_tokens")} if code == "agent_field_truncated" else {}
await once(ctx, EventType.LOG, level="warn", code=code, message=message, fields=names, **extra)
total_usage["total_tokens"] = total_usage["input_tokens"] + total_usage["output_tokens"]
result = {
@@ -941,6 +984,60 @@ def cited_schema(schema: dict[str, Any]) -> dict[str, Any]:
return {"title": "cited_fields", **rewrite(schema)}
#: cite_fields 结构化抽取的输出额度。thinking: off 只对 Anthropic 有用:经 OpenAI 兼容接口接入的推理模型
#: (DeepSeek 系等)关不掉思考,推理也算在这个额度里。以前是 4096:2026-10 一次运行 4 个 Agent 的抽取全部在
#: 推理阶段就用完了额度(completion_tokens=4096),一个字段都没写出来。节点配了更大的「最大输出 token」就用节点的
EXTRACT_MAX_TOKENS = 16384
#: 被截断时额度翻倍重试一次,最多到这里
EXTRACT_RETRY_CAP = 32768
#: 服务方嫌额度太大、拒收时依次退到这几档(DeepSeek chat 系只收 8K)。拒收发生在请求刚发出时,不产生用量
_EXTRACT_FALLBACKS = (8192, 4096)
#: 服务方嫌额度太大时报错里会出现的字样(和 api/copilot.py 改图时退回小额度是同一个判断)
_BUDGET_WORDS = ("max_tokens", "max_completion_tokens", "max_output_tokens", "maximum context length")
#: 截断类报错的原话:openai 的 LengthFinishReasonError(结构化输出解析时 finish_reason=length)
_LENGTH_WORDS = ("length limit was reached", "finish_reason=length", "finish_reason='length'")
#: 两次都被截断时的原因(给人看)。重试、去设置页测试模型都不解决额度问题,不能照通用的模型报错那样说
TRUNCATED_EXTRACT = ("结构化抽取没有完成:模型的输出额度在推理阶段就用完了(最后一次的额度为 {budget} token),"
"结构化结论没能写出来")
#: agent_field_truncated 的下一步(接在「N 个字段都记为空值」后面)
TRUNCATED_NEXT = ("可以为该 Agent 换用非推理模型(抽取沿用 Agent 的模型),或者调大该 Agent 的"
"「最大输出 token」后重新运行")
def truncated_error(error: BaseException | None) -> bool:
"""这个异常是不是在说「输出被截断」:openai 的 LengthFinishReasonError,或者 langchain、网关转述的同一件事。
服务方嫌额度太大的 400(报错里也有 max_tokens)不算:那是请求被拒,模型一个字都没写。
"""
if error is None:
return False
if type(error).__name__ == "LengthFinishReasonError":
return True
text = str(error).lower()
return any(word in text for word in _LENGTH_WORDS)
def truncated_reply(raw: Any) -> bool:
"""这条回复是不是因为额度用完停下的:OpenAI 系是 finish_reason=length,Anthropic 是 stop_reason=max_tokens。"""
meta = getattr(raw, "response_metadata", None) or {}
reason = meta.get("finish_reason") or meta.get("stop_reason")
return reason in ("length", "max_tokens")
def _budget_rejected(error: BaseException) -> bool:
status = getattr(error, "status_code", None) or getattr(getattr(error, "response", None), "status_code", None)
return status in (400, 422) and any(word in str(error).lower() for word in _BUDGET_WORDS)
def _error_usage(error: BaseException, model_id: str) -> dict[str, Any]:
"""失败的那次调用花掉的用量。截断时 openai 的异常带着 completion.usage:额度是真花出去了,要记账。"""
usage = getattr(getattr(error, "completion", None), "usage", None)
inp = int(getattr(usage, "prompt_tokens", 0) or 0)
out = int(getattr(usage, "completion_tokens", 0) or 0)
return {"input_tokens": inp, "output_tokens": out, "total_tokens": inp + out,
"cost_usd": round(catalog.estimate_cost(model_id, inp, out), 6), "calls": int(bool(inp or out))}
EXTRACT_SYSTEM = (
"你是数据抽取员:把智能体查到的数据按给定结构填好,每个字段都写出处。\n"
"- value 是字段的值;from 写它在哪次查询的哪一格:call 是【证据 Qn】里的编号,row 是行号(从 0 数),"
@@ -1184,9 +1281,12 @@ def _shown(value: Any) -> str:
def _field_warnings(data: dict[str, Any], evidence: dict[str, Any],
failed: str | None) -> list[tuple[str, str, list[str]]]:
failed: str | None, *, truncated: bool = False) -> list[tuple[str, str, list[str]]]:
"""核对结果里要让人看见的:(code, message, 字段)。mismatch 和 unresolved 各一条;missing 是模型
照实说没查到,值已经是空的,不另发警告。"""
照实说没查到,值已经是空的,不另发警告。
抽取两次都被截断(truncated)时 code 是 agent_field_truncated:原因是输出额度在推理阶段就用完了,
下一步是换非推理模型或调大最大输出 token,不是「稍后重试」——和别的抽取失败分开说。"""
out: list[tuple[str, str, list[str]]] = []
mismatched = [(p, e) for p, e in evidence.items() if e.get("status") == "mismatch"]
if mismatched:
@@ -1196,6 +1296,11 @@ def _field_warnings(data: dict[str, Any], evidence: dict[str, Any],
f"有 {len(mismatched)} 个字段与查询快照不一致,已按快照取值:{detail}", [p for p, _ in mismatched]))
unresolved = [(p, e) for p, e in evidence.items() if e.get("status") == "unresolved"]
if unresolved:
if failed and truncated:
out.append(("agent_field_truncated",
f"{failed}。{len(unresolved)} 个字段都记为空值(未以 0 代替)。{TRUNCATED_NEXT}",
[p for p, _ in unresolved]))
return out
if failed:
message = f"{failed}。{len(unresolved)} 个字段都记为空值(未以 0 代替)"
else:
+102 -12
View File
@@ -44,17 +44,20 @@ from app.engine import judge as judging
from app.engine.context import NodeContext, NodeError
from app.engine.labels import field_label, option_label
from app.engine.evidence import (
ABSENCE_RULE,
CELLS_REASON,
CLAIMS_RULE,
ENTITY_KINDS,
MARKER_RULES,
SEE_AND_CODE_RULES,
StreamRenderer,
WriterCatalog,
build_catalog,
catalog_prompt,
catalog_budget,
compose_doc,
describe_violations,
ledger_enabled,
writer_catalog,
)
from app.engine.nodes.llm import _invoke_streaming, _model_spec, _report_call
from app.engine.schema import (
@@ -70,7 +73,8 @@ from app.engine.schema import (
)
from app.engine.state import GraphState, message_text, thinking_text
from app.engine.toolcalls import leaked_markup, markup_warning
from app.providers.factory import ProviderNotConfigured, get_chat_model
from app.providers import catalog as model_catalog
from app.providers.factory import ModelSpec, ProviderNotConfigured, get_chat_model, resolve_provider
ON_VIOLATION = ("fail", "flag")
NUMBERS = ("strict", "off")
@@ -81,8 +85,10 @@ CLAIMS = CLAIMS_VALUES
#: 只标注、不让写作者重写也不判失败的违规:可疑实体(受管的正式出具按缺口降档)、表结构快照不全时
#: 核对不了的名字(出口只标注)
SOFT = frozenset({"unknown_entity", "unverified_entity"})
#: 交回写作者重写、重写不好也不判失败的违规:句中的 [[see:]](渲染后留下断句,数字和出处都没问题)
REWRITE_ONLY = frozenset({"inline_see"})
#: 交回写作者重写一次、重写不好也不判失败的违规:句中的 [[see:]](渲染后留下断句,数字和出处都没问题);
#: 写作目录有省略时的缺失断言(「没有查到」「数据缺失」:写作者没看全证据,结论依据不足,但数字和出处都核对过)。
#: 只剩这几类时不按 max_repairs 反复重写:重写一次没改掉,再写也多半一样,留在核对结果里标注
REWRITE_ONLY = frozenset({"inline_see", "absence_claim"})
#: 重写一次就是多一次整篇的模型调用,写不对的模型多给几次也大多写不对
MAX_REPAIRS = 3
@@ -140,6 +146,66 @@ def report_catalog(state: GraphState, spec: GraphSpec, node: GraphNode) -> dict[
return catalog
def report_fields(state: GraphState, spec: GraphSpec, node: GraphNode) -> list[dict[str, Any]]:
"""上游 Agent 经按出处核对交来的结构化字段:[{node_id, title, data, evidence}],按画布上的顺序。
范围和 report_catalog 收查询的范围一致:祖先节点,写了 evidence_from 就只收那几个。只进写作目录(第一优先,
放不下也列出),不进目录本身:出口复核按目录重建,字段的引用落到的是查询单元格,和以前一样核对。
"""
sources = _setting(spec, node, "evidence_from", "ancestors")
if isinstance(sources, str) and sources != "ancestors":
sources = [sources]
keep = {str(s) for s in sources} if isinstance(sources, list) else None
allowed = _ancestors(spec, node.id)
outputs = state.get("nodes") or {}
found: list[dict[str, Any]] = []
for other in spec.nodes:
result = outputs.get(other.id)
if other.id not in allowed or (keep is not None and other.id not in keep) or not isinstance(result, dict):
continue
if isinstance(result.get("data_evidence"), dict) and result["data_evidence"]:
found.append({"node_id": other.id, "title": other.title or other.id, "data": result.get("data"),
"evidence": result["data_evidence"]})
return found
async def model_window(session: Any, spec: ModelSpec, model_id: str) -> int | None:
"""写作模型的上下文窗口(token):模型接入里给这个模型登记的 context 优先,其次模型目录
(providers/catalog.py);都没有返回 None,由 catalog_budget 取保守值。"""
try:
provider = await resolve_provider(session, spec.provider, spec.model or model_id)
except ProviderNotConfigured:
provider = None
for entry in (provider.models or []) if provider is not None else []:
if isinstance(entry, dict) and entry.get("id") == model_id:
raw = entry.get("context")
try:
value = int(raw) if not isinstance(raw, bool) and raw not in (None, "") else 0
except (TypeError, ValueError):
value = 0
if value > 0:
return value
return model_catalog.find_context(model_id)
def _trimmed_message(writer: WriterCatalog) -> str:
"""写作目录有省略时的那条警告(给人看):省略的是什么、各来源省略了几次,写作模型看不到哪些明细。
放不下不一定省略了查询:查询全都列出、只是有的只列前几行,或者只省略了表和字段的说明。以前一律按「省略查询」写,
全部列出时成了「0 次只列编号和摘要()。写作模型看不到这些查询的明细」,说的不是实情。"""
head = f"证据目录超出篇幅(约 {writer.budget} 字,按写作模型的上下文窗口确定):"
if writer.omitted:
per = "、".join(f"{r['title']}省略 {r['omitted']} 次" for r in writer.nodes if r["omitted"])
shown = writer.queries - len(writer.omitted)
return (f"{head}{writer.queries} 次查询中 {shown} 次列出了数据,{len(writer.omitted)} 次只列编号和摘要({per})。"
"写作模型看不到这些查询的明细,报告不得据此断言没有数据")
if writer.compact:
per = "、".join(f"{r['title']} {r['compact']} 次" for r in writer.nodes if r.get("compact"))
return (f"{head}{writer.queries} 次查询都列出了数据,其中 {writer.compact} 次只列前 {writer.compact_rows} 行"
f"({per})。写作模型看不到这些查询其余的行,报告不得据此断言没有数据")
return f"{head}{writer.queries} 次查询都列出了数据,表和字段、知识库检索的说明有省略"
def report_entities(spec: GraphSpec, node: GraphNode) -> str:
"""报告节点的实体层:link(默认)/ off。出口复核调 verify_doc 时传 entities=(这个值 == "link")。
@@ -351,19 +417,21 @@ def _repair_request(violations: list[dict[str, Any]]) -> str:
)
def _messages(ctx: NodeContext, state: GraphState, catalog: dict[str, Any], cells: bool = True,
claims: str = "off") -> list[BaseMessage]:
def _messages(ctx: NodeContext, state: GraphState, writer: WriterCatalog, claims: str = "off") -> list[BaseMessage]:
system = ctx.render_str(ctx.cfg("system", ""), state)
instructions = ctx.render_str(ctx.cfg("instructions", ""), state).strip() \
or "根据下面的证据写一份简洁的报告,先总后分。"
rules = f"{MARKER_RULES}\n{SEE_AND_CODE_RULES}"
if writer.guarded:
# 写作者看不全证据(目录有省略、有来源的结构化结论为空):「没有查到」这类结论没有依据
rules = f"{rules}\n{ABSENCE_RULE}"
if claims in ("require_citation", CLAIMS_JUDGE):
# 裁判只看写作者挂的依据:要判,就更得每句都挂上
rules = f"{rules}\n{CLAIMS_RULE}" + (f"\n{judging.JUDGE_RULE}" if claims == CLAIMS_JUDGE else "")
return [
SystemMessage(content="\n\n".join(p for p in (system, ROLE) if p)),
HumanMessage(content=(
f"{instructions}\n\n{catalog_prompt(catalog, cells_allowed=cells)}\n\n{rules}\n\n"
f"{instructions}\n\n{writer.text}\n\n{rules}\n\n"
"只输出报告正文(Markdown),不要写撰写说明。"
)),
]
@@ -392,6 +460,8 @@ async def run_report(state: GraphState, ctx: NodeContext) -> dict[str, Any]:
model, model_id = await get_chat_model(session, _model_spec(ctx))
except ProviderNotConfigured as e:
raise NodeError(ctx.node.id, not_configured(e)) from e
# 写作目录的篇幅按这个模型的上下文窗口定(evidence.catalog_budget)
window = await model_window(session, _model_spec(ctx), model_id)
spec = ctx.run.spec
catalog = report_catalog(state, spec, ctx.node)
@@ -408,7 +478,15 @@ async def run_report(state: GraphState, ctx: NodeContext) -> dict[str, Any]:
if warning := _no_evidence(catalog, cells):
ctx.emit(EventType.LOG, level="warn", code="report_no_evidence", message=warning)
messages = _messages(ctx, state, catalog, cells, claims)
writer = writer_catalog(catalog, budget=catalog_budget(window), cells_allowed=cells,
fields=report_fields(state, spec, ctx.node),
titles={n.id: n.title or n.id for n in spec.nodes})
if writer.trimmed:
ctx.emit(EventType.LOG, level="warn", code="report_catalog_trimmed", message=_trimmed_message(writer),
budget=writer.budget, window=window, queries=writer.queries, omitted=len(writer.omitted),
compact=writer.compact, compact_rows=writer.compact_rows,
nodes=[{k: r[k] for k in ("node_id", "title", "total", "omitted", "compact")} for r in writer.nodes])
messages = _messages(ctx, state, writer, claims)
started = time.perf_counter()
spent: list[dict[str, Any]] = []
nudged = False
@@ -438,12 +516,13 @@ async def run_report(state: GraphState, ctx: NodeContext) -> dict[str, Any]:
def compose(raw: str) -> dict[str, Any]:
return compose_doc(raw, catalog, node_id=ctx.node.id, run_id=ctx.run.run_id, allow_numbers=allow,
cells_allowed=cells, entities=entities)
cells_allowed=cells, entities=entities, absence=writer.trimmed)
response, raw = await draft(messages)
doc = compose(raw)
repairs = 0
while (blocking := _blocking(doc["violations"], numbers)) and repairs < max_repairs:
while (blocking := _blocking(doc["violations"], numbers)) and repairs < max_repairs \
and not (repairs and not _fatal(blocking)):
repairs += 1
ctx.emit(EventType.LOG, level="warn", code="report_repair",
message=f"报告有 {len(blocking)} 处未通过核对({_named(blocking)}),"
@@ -454,9 +533,13 @@ async def run_report(state: GraphState, ctx: NodeContext) -> dict[str, Any]:
judgement: dict[str, Any] | None = None
if claims == CLAIMS_JUDGE:
from app.engine.nodes.human import reviewed_by_human
judge_cfg = report_judge(spec, ctx.node)
if await _run_class(ctx.run.run_id) != "formal":
# 探索运行默认按需裁判:点开哪句才判哪句,节点里不花这笔钱
if await _run_class(ctx.run.run_id) != "formal" and not reviewed_by_human(spec, ctx.node.id):
# 探索运行默认按需裁判:点开哪句才判哪句,节点里不花这笔钱。要交给人工审批的报告例外:审批人据此
# 拍板,批准前就该知道哪句结论站不住;而按需裁判要等运行封存之后才能判,封存前放开会动证据清单的
# 完整性约束,所以这种报告探索运行也在节点里当场判(和正式运行同一条路)
judgement = judging.on_demand(doc, on_unsupported=judge_cfg["on_unsupported"],
rewrite_once=judge_cfg["rewrite_once"], writer_model=model_id)
elif not (_fatal(blocking) and on_violation == "fail"):
@@ -466,6 +549,13 @@ async def run_report(state: GraphState, ctx: NodeContext) -> dict[str, Any]:
draft=draft, compose=compose, numbers=numbers, blocking=blocking)
spent.extend(usages)
if absent := [v for v in doc["violations"] if v.get("code") == "absence_claim"]:
# 重写之后还在(或者没让重写):报告照常产出,这几处标注出来,时间线上单独说清为什么依据不足
texts = list(dict.fromkeys(str(v.get("text") or "") for v in absent))
tried = f"已要求模型重写 {repairs} 次仍未改掉" if repairs else "未要求重写"
ctx.emit(EventType.LOG, level="warn", code="report_absence_claim", count=len(absent), texts=texts,
message=f"报告在证据目录有省略时写了 {len(absent)} 处缺失断言({_named(absent)}),{tried}:"
"被省略的查询仍在证据台账里,这类结论依据不足,应写作本报告未覆盖。报告照常产出,这几处已标注")
doc_artifact = await _store(doc, ctx)
ctx.emit(EventType.REPORT_CHECKED, doc_artifact=doc_artifact, ok=not doc["violations"],
stats=doc["stats"], violations=doc["violations"][:20], repairs=repairs,
+6
View File
@@ -89,12 +89,18 @@ WARN_CODES: dict[str, tuple[str, str]] = {
"report_repair": ("report_repair", DEGRADED),
# 报告撰写节点的上游没有可引用的口径卡指标:报告里的数都没有出处
"report_no_evidence": ("report_no_evidence", DEGRADED),
# 证据目录超出篇幅,有的查询只列了编号和摘要:写作模型没看全证据,报告的覆盖面有缺口
"report_catalog_trimmed": ("report_catalog_trimmed", DEGRADED),
# 目录有省略时写了「没有查到」这类缺失断言、重写后仍在:报告照常产出并标注,这几句结论依据不足
"report_absence_claim": ("report_absence_claim", DEGRADED),
# 口径卡的指标集、报告文档没能落进工件库:答案照常,只是点不开证据
"evidence_store_failed": ("evidence_store_failed", DEGRADED),
# agent 的结构化字段:模型报的值和查询快照对不上(已按快照取值),或者出处核对不了
# (记成空值,不兜底成 0)。答案能用,缺口要说清是哪一种
"agent_field_mismatch": ("agent_field_mismatch", DEGRADED),
"agent_field_unverified": ("agent_field_unverified", DEGRADED),
# 结构化抽取两次都被截断(输出额度在推理阶段就用完了):字段都记空值,答案能用,缺口要说清是额度问题
"agent_field_truncated": ("agent_field_truncated", DEGRADED),
# 结论句裁判(claims: judge):触顶、没跑成的句子记为未裁判,出具不能判完整——是缺口,要说明
"judge_limit": ("judge_limit", DEGRADED),
"judge_failed": ("judge_failed", DEGRADED),
+14 -1
View File
@@ -837,11 +837,15 @@ class RunManager:
run = await session.get(Run, run_id)
if run:
timing = await _timing(session, run, elapsed_ms, now)
events_usage = await _usage_of_events(session, run_id)
if status != "succeeded":
# 状态里的用量只算跑完了的节点。失败、取消、停在审批上的那个节点已经
# 花出去的调用(校验修复、agent 的前几步)只在 llm.end 里——以前这部分
# 在运行记录上凭空消失,失败的运行看着像没花钱
usage = _spent(usage, await _usage_of_events(session, run_id))
usage = _spent(usage, events_usage)
elif events_usage.get("unpriced_calls"):
# 金额估不出来的调用只有 llm.end 数得出(状态里的用量只有金额,0 分不出是免费还是没价)
usage = {**usage, "unpriced_calls": events_usage["unpriced_calls"]}
# duration_ms 是各段执行时长之和(= active_ms)。以前用这一段的耗时
# 覆盖它:87 秒、中间等过一次审批的运行,列表上写着 22ms
usage.update(duration_ms=timing["active_ms"], **timing)
@@ -1251,13 +1255,20 @@ async def _usage_of_events(session: Any, run_id: str) -> dict[str, Any]:
rows = await session.execute(
select(RunEvent.data).where(RunEvent.run_id == run_id, RunEvent.type == EventType.LLM_END)
)
from app.providers.catalog import unpriced_call
total: dict[str, Any] = dict.fromkeys(_USAGE_KEYS, 0)
unpriced = 0
for (data,) in rows:
for key in _USAGE_KEYS:
value = (data or {}).get(key, 1 if key == "calls" else 0)
if isinstance(value, (int, float)) and not isinstance(value, bool):
total[key] += value
unpriced += unpriced_call(data)
total["cost_usd"] = round(total["cost_usd"], 6)
if unpriced:
# 有几次调用的金额估不出来(模型不在价格目录里):cost_usd 只是估得出的那部分,界面据此不写 $0
total["unpriced_calls"] = unpriced
return total
@@ -1269,6 +1280,8 @@ def _spent(state_usage: dict[str, Any], events_usage: dict[str, Any]) -> dict[st
mine, theirs = out.get(key) or 0, events_usage.get(key) or 0
if isinstance(mine, (int, float)) and theirs > mine:
out[key] = theirs
if events_usage.get("unpriced_calls"):
out["unpriced_calls"] = events_usage["unpriced_calls"]
if "input_tokens" in out or "output_tokens" in out:
out["total_tokens"] = int(out.get("input_tokens") or 0) + int(out.get("output_tokens") or 0)
return out
+16
View File
@@ -229,6 +229,22 @@ def find_context(model: str) -> int | None:
return None
def unpriced_call(data: Any) -> bool:
"""一次模型调用(llm.end 的数据)的金额估不出来:用了 token,模型却不在价格目录里,estimate_cost 只能给 0。
这时写 $0 像是没花钱(真实运行里上百次调用、上百万 token,运行页写着 $0)。本地模型在目录里标的是 0 价,
不算估不出来。事件自己标了 unpriced 的以它为准。"""
if not isinstance(data, dict):
return False
if isinstance(data.get("unpriced"), bool):
return data["unpriced"]
tokens = sum(v for v in (data.get("input_tokens"), data.get("output_tokens"))
if isinstance(v, (int, float)) and not isinstance(v, bool))
model = data.get("model")
return tokens > 0 and isinstance(model, str) and bool(model) and not data.get("cost_usd") \
and find_pricing(model) is None
def estimate_cost(model: str, input_tokens: int, output_tokens: int) -> float:
"""按目录价估算一次调用的花费(美元)。"""
pricing = find_pricing(model)
+64
View File
@@ -1,5 +1,6 @@
from __future__ import annotations
import logging
import os
from typing import Any
@@ -14,6 +15,8 @@ from app.db.models import Provider
from app.providers import catalog
from app.providers.mock_model import MockChatModel
logger = logging.getLogger(__name__)
class ModelSpec(BaseModel):
"""节点上对模型的请求。省略的字段会回退到 provider 默认值。"""
@@ -221,6 +224,10 @@ def build_chat_model(provider: Provider, spec: ModelSpec) -> BaseChatModel:
kwargs["base_url"] = base_url
if extra.get("headers"):
kwargs["default_headers"] = extra["headers"]
# 流式调用要服务方在流尾补一块用量(stream_options.include_usage)。ChatOpenAI 只在没填 base_url 时默认开,
# 兼容接口都填了 base_url,于是一次运行上百次调用记成 0 token、0 美元——llm.end 读的 usage_metadata
# 就来自这一块。不认 stream_options 的服务由 ReasoningAwareChatOpenAI 去掉它重发,不让调用失败
kwargs["stream_usage"] = True
return _ReasoningAwareChatOpenAI(**kwargs)
@@ -260,9 +267,66 @@ def _reasoning_aware_openai() -> type:
gen.message.additional_kwargs["reasoning_content"] = text
return gen
async def _astream(self, messages: Any, stop: Any = None, run_manager: Any = None, **kwargs: Any):
"""流式调用;服务方不认 stream_options 时去掉它重发一次,之后这个接口不再带它。
只在还没收到任何一块时重发:拒绝都发生在请求刚发出去的时候,重发不会重复输出。"""
usage = not self._use_responses_api({**kwargs, **self.model_kwargs}) \
and self._should_stream_usage(kwargs.get("stream_usage"), **kwargs)
key = _endpoint_key(self)
first: Exception | None = None
if usage and key not in _NO_STREAM_USAGE:
started = False
try:
async for chunk in super()._astream(messages, stop, run_manager, **dict(kwargs)):
started = True
yield chunk
return
except Exception as e: # noqa: BLE001 - 认得出的拒绝才重发,其余原样抛出
if started or not _rejects_stream_options(e):
raise
first = e
if usage:
kwargs = {k: v for k, v in kwargs.items() if k != "stream_options"} | {"stream_usage": False}
try:
async for chunk in super()._astream(messages, stop, run_manager, **kwargs):
if first is not None and key not in _NO_STREAM_USAGE:
# 去掉它就成了:记下这个接口不认,之后不再每次先失败一次
_NO_STREAM_USAGE.add(key)
logger.warning("模型接入 %s 不支持流式用量回传(%s),已改为不请求用量", key, str(first)[:200])
yield chunk
except Exception as e:
if first is None:
raise
raise e from first
return ReasoningAwareChatOpenAI
#: 认定不支持 stream_options 的接口(base_url)。进程内有效:重启后再试一次,服务方可能已经升级
_NO_STREAM_USAGE: set[str] = set()
def _endpoint_key(model: Any) -> str:
return str(getattr(model, "openai_api_base", None) or "https://api.openai.com/v1")
def _rejects_stream_options(exc: Exception) -> bool:
"""这个报错是不是服务方不认 stream_options。
点名了 stream_options / include_usage 的一定是;只回 400 / 422、说不清哪个参数的(有的网关只回一句
Bad Request)也去掉它试一次——重发也失败就说明不是它的事,原样报重发的错。上下文超长同样是 400,
重发也没用,不算。"""
from langchain_core.exceptions import ContextOverflowError
if isinstance(exc, ContextOverflowError):
return False
text = str(exc).lower()
if "stream_options" in text or "include_usage" in text:
return True
return getattr(exc, "status_code", None) in (400, 422)
class _LazyReasoningChatOpenAI:
"""延迟导入 langchain_openai:它加载得慢,而 mock/anthropic 路径用不上。"""
+106
View File
@@ -0,0 +1,106 @@
"""合成的「4 个专项 Agent 并行查库 → 报告」场景:查询多、分布不均,每个 Agent 都交了按出处核对的字段。
形状仿一次真实运行(4 个 Agent 并行、查询次数悬殊,合计上百次),次数和内容全部虚构:门店、会员、卡券、活动。
不连库、不调模型:台账条目、查询快照、表结构快照、Agent 的 data / data_evidence 都在内存里拼好,快照经 loader 取。
台账按节点跑完的先后排(state["evidence"] 是各节点结束时追加的),所以全局编号 Q1… 前 26 个都是会员专项的。
"""
from __future__ import annotations
from typing import Any
from app.engine.evidence import cell_eid, make_eid
#: (节点 id, 画布上的名字, 查询次数, 查的表)
AGENTS: list[tuple[str, str, int, str]] = [
("agent_member", "会员专项", 26, "member_daily"),
("agent_store", "门店专项", 17, "store_daily"),
("agent_coupon", "卡券核销", 79, "coupon_redemption"),
("agent_campaign", "活动内容", 42, "campaign_view"),
]
COLUMNS = ["store_name", "biz_time", "cnt", "amount"]
TYPES = {"store_name": "text", "biz_time": "datetime", "cnt": "number", "amount": "number"}
ROWS = 30
#: 结构化抽取失败(两次都被截断)的那个 Agent:字段全是空值
EMPTY_AGENT = "agent_coupon"
FIELDS = ["total_cnt", "top_store", "latest_time", "top_stores", "churn_rate"]
def artifact_of(node_id: str, i: int) -> str:
return f"{node_id}-q{i:03d}".ljust(64, "0")
def snapshot(node_id: str, i: int, rows: int = ROWS) -> dict[str, Any]:
"""第 i 次查询的快照:每行一个门店,时间是 ISO 写法(快照里存的就是这样)。"""
seed = sum(map(ord, node_id)) + i * 7
return {
"columns": list(COLUMNS),
"column_types": dict(TYPES),
"rows": [[f"门店{(seed + r) % 90 + 1:02d}", f"2026-06-{r % 28 + 1:02d}T08:{(seed + r) % 60:02d}:39",
(seed * 13 + r * 17) % 900 + 10, f"{(seed * 31 + r * 7) % 9000 + 100}.50"] for r in range(rows)],
"truncated": False,
}
def schema_snapshot(table: str) -> dict[str, Any]:
return {"source": "shop", "tables": {table: {"columns": [{"name": c, "type": TYPES[c]} for c in COLUMNS]}}}
def build() -> tuple[list[dict[str, Any]], dict[str, Any], dict[str, Any], dict[str, str]]:
"""(台账, 快照 {工件 id: 内容}, state["nodes"], {节点 id: 名字})。"""
ledger: list[dict[str, Any]] = []
store: dict[str, Any] = {}
nodes: dict[str, Any] = {}
titles: dict[str, str] = {}
for node_id, title, count, table in AGENTS:
titles[node_id] = title
schema_id = f"schema-{table}".ljust(64, "0")
store[schema_id] = schema_snapshot(table)
ledger.append({"kind": "schema", "node_id": node_id, "exec": 1, "artifact": schema_id, "source": "shop",
"tables": {table: list(COLUMNS)}})
for i in range(1, count + 1):
aid = artifact_of(node_id, i)
store[aid] = snapshot(node_id, i)
ledger.append({"kind": "query", "node_id": node_id, "exec": 1, "artifact": aid,
"via": f"tool-{aid}"[:64], "call_id": f"call_{i}", "tool": "db_query__shop", "source": "shop",
"columns": list(COLUMNS), "rows": ROWS, "truncated": False,
"schema_artifact": schema_id, "tables": [table]})
nodes[node_id] = agent_output(node_id, store)
return ledger, store, nodes, titles
def agent_output(node_id: str, store: dict[str, Any]) -> dict[str, Any]:
"""Agent 交出的结构化字段:本节点的 Q3 那一格、Q5 的前 5 行;EMPTY_AGENT 抽取失败,字段全空。"""
if node_id == EMPTY_AGENT:
reason = "结构化抽取没有完成:模型的输出额度在推理阶段就用完了(最后一次的额度为 32768 token),结构化结论没能写出来"
return {"text": "查到了大量核销和兑换记录。", "data": {f: None for f in FIELDS},
"data_evidence": {f: {"status": "unresolved", "reason": reason} for f in FIELDS}}
q3, q5 = artifact_of(node_id, 3), artifact_of(node_id, 5)
first = store[q3]["rows"][0]
def cell(column: str, index: int) -> dict[str, Any]:
return {"ref": f"Q3.r0.{column}", "call": "Q3", "artifact": q3, "via": f"tool-{q3}"[:64],
"locator": {"row": 0, "column": column}, "eid": cell_eid(q3, 0, column), "status": "verified"}
rows_locator = {"rows": [0, 4], "column": "store_name"}
return {
"text": "结论见字段。",
"data": {"total_cnt": first[2], "top_store": first[0], "latest_time": first[1],
"top_stores": [r[0] for r in store[q5]["rows"][:5]], "churn_rate": None},
"data_evidence": {
"total_cnt": cell("cnt", 2), "top_store": cell("store_name", 0), "latest_time": cell("biz_time", 1),
"top_stores": {"ref": "Q5.r0-4.store_name", "call": "Q5", "artifact": q5, "via": f"tool-{q5}"[:64],
"locator": rows_locator, "eid": make_eid("rows", q5, rows_locator), "status": "verified"},
"churn_rate": {"ref": None, "status": "missing"},
},
}
def global_alias(node_id: str, local: int) -> str:
"""本节点第 local 次查询在报告目录里的全局编号(台账按节点先后排)。"""
offset = 0
for nid, _, count, _ in AGENTS:
if nid == node_id:
return f"Q{offset + local}"
offset += count
raise KeyError(node_id)
+165
View File
@@ -0,0 +1,165 @@
"""人工审批审的是报告时,审批卡附上这份报告的核对概况(human 节点的请求里带 reports)。
真实运行里:报告里有一句结论和证据相矛盾,审批人批准时完全不知道——报告节点只把正文交给了审批卡,几个数字有出处、
结论句判没判过,一个字都没有。用户事后在证据面板点开那句,裁判才判它相矛盾。
现在 human 节点挂起时,上游的报告节点(直接连着的,或者标题、说明里引用了它正文的)各带一份核对概况:
数字核对的统计(report.checked 的 stats)、结论句策略、裁判摘要(几句判了、几句相矛盾),以及报告文档的工件 id,
审批卡据此写「N 个数字都有出处」「结论句已裁判 X 句,证据相矛盾 Y 句」或「结论句尚未核对」,并给出打开证据的入口。
表名、指标、数字都是虚构的。
"""
from __future__ import annotations
import asyncio
import json
import re
import pytest
from langchain_core.messages import AIMessage
from sqlalchemy import delete, select
from app.db.base import SessionLocal
from app.db.models import Approval, Run, RunEvent, Setting
from app.engine import judge as judge_mod
from app.engine.judge import JUDGE_ROLE, SPEND_KEY
from app.engine.runner import run_manager
from app.providers.mock_model import MockChatModel
PRICED = "claude-sonnet-5"
METRICS = [{"id": "gmv", "name": "销售额", "unit": "元", "decimals": 1, "expression": "vars.kpi.gmv"}]
TEXT = "本周销售额 [[m:gmv]]。[[see:m:gmv]]增长主要来自新客首单。"
@pytest.fixture(autouse=True)
async def engine_up():
await run_manager.setup()
async with SessionLocal() as session:
await session.execute(delete(Setting).where(Setting.key.in_([SPEND_KEY, "judge", "copilot"])))
await session.commit()
yield
await run_manager.shutdown()
@pytest.fixture(autouse=True)
def models(monkeypatch):
"""写作者回 TEXT;裁判把「新客」那句判成证据相矛盾,其余有依据。"""
def decide(model, messages):
if JUDGE_ROLE in str(messages[0].content):
request = str(messages[-1].content)
items = [{"unit": uid, "verdict": "contradicted" if "新客" in line else "supported",
"rationale": "虚构的理由", "used": []}
for uid, line in re.findall(r"^\[(u\d+)\]([^\n]*)$", request, re.M)]
return AIMessage(content=json.dumps({"items": items}, ensure_ascii=False))
return AIMessage(content=TEXT)
monkeypatch.setattr(MockChatModel, "_decide", decide)
async def judge_model_of(session, spec):
return MockChatModel(model_name=PRICED), PRICED
monkeypatch.setattr(judge_mod, "get_chat_model", judge_model_of)
def node(nid, ntype, label=None, **config):
return {"id": nid, "type": ntype, "position": {"x": 0, "y": 0}, "data": {"label": label or nid, "config": config}}
def graph(*, gate: dict, wire_report_to_gate: bool = True, **report):
nodes = [
node("start", "input"),
node("fetch", "transform", mode="expression", expression="{'gmv': 45678.5}", assign_to="kpi"),
node("caliber", "metrics", caliber="周报口径", caliber_version="v1", metrics=METRICS),
node("write", "report", "写周报", instructions="写周报", **report),
node("note", "transform", mode="expression", expression="'ok'", assign_to="note"),
node("gate", "human", "主管审批", mode="approve", **gate),
node("out", "output", fields=[{"name": "周报", "value": "{{ nodes.write.text }}"}]),
]
chain = ["start", "fetch", "caliber", "write"] + (["gate"] if wire_report_to_gate else ["note", "gate"])
edges = [{"source": a, "target": b} for a, b in zip(chain, chain[1:])]
edges.append({"source": "gate", "target": "out", "sourceHandle": "approved"})
return {"nodes": nodes, "edges": edges}
async def paused(g, *, run_class: str) -> tuple[Run, Approval]:
started = await run_manager.start(graph=g, input_payload={}, run_class=run_class)
for _ in range(400):
await asyncio.sleep(0.05)
async with SessionLocal() as session:
row = await session.get(Run, started.id)
if row.status in ("interrupted", "failed", "succeeded"):
approval = (await session.execute(select(Approval).where(Approval.run_id == row.id))).scalar_one_or_none()
if row.status != "interrupted" or approval is not None:
break
assert row.status == "interrupted", row.error
assert approval is not None
return row, approval
async def test_formal_run_carries_the_judged_claims_to_the_approval():
row, approval = await paused(graph(gate={"title": "审核周报", "message": "{{ nodes.write.text }}"},
claims="judge"), run_class="formal")
[report] = approval.payload["reports"]
assert report["node_id"] == "write" and report["label"] == "写周报"
assert isinstance(report["doc_artifact"], str) and report["doc_artifact"]
assert report["stats"]["numbers"] == report["stats"]["numbers_cited"] == 1
assert report["claims"] == "judge"
judge = report["judge"]
assert judge["mode"] == "inline"
assert judge["counts"]["contradicted"] == 1 and judge["counts"]["supported"] == 1
assert judge["counts"]["unjudged"] == 0
# 和 report.checked 是同一份:审批卡写的和时间线那一行对得上
async with SessionLocal() as session:
[checked] = list((await session.execute(select(RunEvent).where(
RunEvent.run_id == row.id, RunEvent.type == "report.checked"))).scalars())
assert checked.data["doc_artifact"] == report["doc_artifact"]
assert {k: checked.data["stats"][k] for k in report["stats"]} == report["stats"]
assert checked.data["judge"]["counts"] == judge["counts"]
# human.requested 事件里也带着,回放时界面一样读得到
async with SessionLocal() as session:
[requested] = list((await session.execute(select(RunEvent).where(
RunEvent.run_id == row.id, RunEvent.type == "human.requested"))).scalars())
assert requested.data["reports"] == approval.payload["reports"]
async def test_exploratory_report_under_approval_is_judged_in_the_node():
"""要交给人审批的报告,探索运行也在节点里判:审批人据此拍板,批准前就该知道哪句结论站不住。
真实运行里探索运行的报告有一句和证据相矛盾,审批卡只能写「结论句尚未核对」——按需裁判要等封存
(运行结束)之后才能判,而封存前放开按需裁判会动证据清单的完整性约束,所以改在节点里当场判。
没有审批的探索运行照旧按需判,不在节点里花这笔钱(test_contract_judge.test_explore_runs_only_annotate)。"""
_, approval = await paused(graph(gate={"title": "审核周报"}, claims="judge"), run_class="exploratory")
[report] = approval.payload["reports"]
assert report["claims"] == "judge" and report["judge"]["mode"] == "inline"
assert report["judge"]["counts"]["contradicted"] == 1 and report["judge"]["counts"]["unjudged"] == 0
async def test_report_quoted_in_the_approval_message_is_judged_in_the_node_too():
_, approval = await paused(graph(gate={"title": "审核", "message": "请看:{{ nodes.write.text }}"},
wire_report_to_gate=False, claims="judge"), run_class="exploratory")
[report] = approval.payload["reports"]
assert report["judge"]["mode"] == "inline" and report["judge"]["counts"]["contradicted"] == 1
async def test_report_without_claims_policy_still_reports_numbers():
_, approval = await paused(graph(gate={"title": "审核周报"}), run_class="exploratory")
[report] = approval.payload["reports"]
assert report["claims"] == "off" and "judge" not in report
assert report["stats"]["numbers"] == 1
async def test_a_report_quoted_in_the_message_counts_even_when_not_wired_directly():
_, approval = await paused(graph(gate={"title": "审核", "message": "请看:{{ nodes.write.text }}"},
wire_report_to_gate=False), run_class="exploratory")
assert [r["node_id"] for r in approval.payload["reports"]] == ["write"]
async def test_a_report_quoted_through_its_variable_counts_too():
_, approval = await paused(graph(gate={"title": "审核", "message": "{{ vars.weekly }}"}, wire_report_to_gate=False,
assign_to="weekly"), run_class="exploratory")
assert [r["node_id"] for r in approval.payload["reports"]] == ["write"]
async def test_approvals_unrelated_to_a_report_carry_nothing():
_, approval = await paused(graph(gate={"title": "确认", "message": "继续吗"}, wire_report_to_gate=False),
run_class="exploratory")
assert "reports" not in approval.payload
@@ -0,0 +1,142 @@
"""助手写人工审批的说明时不说「正式出具」。
真实运行里:助手搭了「几个 Agent 查库 → 报告 → 人工审批 → 成果」,审批节点的标题和说明写着「请审核后正式出具」
「批准后才会正式出具」。可那是一次探索运行,成果节点也没有配出具契约——根本不存在「正式出具」。审批人照着这句
话批准,以为自己在签发一份正式的东西。
出具只发生在成果节点配了出具契约、从已发布版本发起的正式运行里;画布上试运行、问数据都是探索运行。审批说明
写「批准后输出」这类说法,这次是哪一类运行由审批卡自己标明。
这组守的是:
- 节点手册里 human 那一条写明这条规矩和替代说法
- 助手写出来的审批标题、说明里有「正式出具」时,自查交回去改(给人看的那句不带键名,给模型的那句说怎么改)
- 这一轮没碰过的审批节点不替人改
"""
from __future__ import annotations
import json
import pytest
from httpx import ASGITransport, AsyncClient
from app.api.copilot import NODE_REFERENCE, authored_issues
from app.engine.schema import GraphSpec
from app.main import app
def _human_rule() -> str:
start = NODE_REFERENCE.index("- human:人工审批")
return NODE_REFERENCE[start:NODE_REFERENCE.index("\n- validate:", start)]
def test_node_reference_tells_the_assistant_not_to_promise_formal_issuance():
rule = _human_rule()
assert "不要写「正式出具」" in rule
assert "批准后输出" in rule
# 说清什么时候才谈得上出具:配了出具契约、从已发布版本发起的正式运行
assert "出具契约" in rule and "已发布版本" in rule and "正式运行" in rule and "探索运行" in rule
def _node(nid, ntype, **config):
return {"id": nid, "type": ntype, "data": {"label": nid, "config": config}}
def _spec(*nodes):
return GraphSpec.model_validate({"nodes": list(nodes), "edges": [
{"source": a["id"], "target": b["id"]} for a, b in zip(nodes, nodes[1:])]})
def _graph(**human):
return _spec(_node("start", "input"),
_node("write", "report", instructions="写周报"),
_node("gate", "human", mode="approve", **human),
_node("out", "output", fields=[{"name": "周报", "value": "{{ nodes.write.text }}"}]))
@pytest.mark.parametrize("human, field", [
({"title": "请审核后正式出具", "message": "请审核本周报告"}, "title"),
({"title": "审批", "message": "请审核本周报告,批准后才会正式出具。\n\n{{ nodes.write.text }}"}, "message"),
])
def test_promising_formal_issuance_is_sent_back(human, field):
issues = [i for i in authored_issues(_graph(**human), []) if i["code"] == "human_formal_issuance"]
assert len(issues) == 1, issues
[issue] = issues
assert issue["level"] == "error" and issue["node_id"] == "gate" and issue["field"] == field
assert "正式出具" in issue["message"] and "批准后输出" in issue["message"]
assert "出具契约" in issue["for_model"] and "批准后输出" in issue["for_model"]
def test_accurate_wording_passes():
spec = _graph(title="请审核本周报告", message="批准后输出这份报告。\n\n{{ nodes.write.text }}")
assert not [i for i in authored_issues(spec, []) if i["code"] == "human_formal_issuance"]
def test_approvals_this_round_did_not_touch_are_left_alone():
spec = _graph(title="请审核后正式出具")
gate = next(n for n in spec.model_dump()["nodes"] if n["id"] == "gate")
assert not [i for i in authored_issues(spec, [], baseline=[gate]) if i["code"] == "human_formal_issuance"]
class Scripted:
"""第一轮吐 first;收到自查请求(human 里有「上线前自查」)时吐 fix。"""
def __init__(self, first: list[dict], fix: list[dict]) -> None:
self.first = [json.dumps(o, ensure_ascii=False) for o in first]
self.fix = [json.dumps(o, ensure_ascii=False) for o in fix]
self.calls: list[str] = []
async def astream(self, messages):
from langchain_core.messages import AIMessageChunk
human = next(text for role, text in messages if role == "human")
self.calls.append(human)
for line in (self.fix if "上线前自查" in human else self.first):
yield AIMessageChunk(content=line + "\n")
def _add(nid, ntype, label, **config):
return {"op": "add_node", "node": {"id": nid, "type": ntype, "label": label, "config": config}}
def _link(a, b, handle=None):
return {"op": "add_edge", "edge": {"source": a, "target": b, **({"sourceHandle": handle} if handle else {})}}
async def test_the_self_check_rewrites_the_approval_before_delivery(monkeypatch):
import app.api.copilot as copilot
first = [
_add("start", "input", "输入", fields=[{"name": "week"}]),
_add("fetch", "transform", "取数", mode="expression", expression="{'gmv': 1}", assign_to="kpi"),
_add("card", "metrics", "口径卡", metrics=[{"id": "gmv", "name": "销售额", "expression": "vars.kpi.gmv"}]),
_add("write", "report", "写周报", instructions="写周报"),
_add("gate", "human", "审批", mode="approve", title="请审核后正式出具",
message="批准后才会正式出具。\n\n{{ nodes.write.text }}"),
_add("out", "output", "成果", fields=[{"name": "周报", "value": "{{ nodes.write.text }}"}]),
_link("start", "fetch"), _link("fetch", "card"), _link("card", "write"), _link("write", "gate"),
_link("gate", "out", "approved"),
{"op": "done", "explanation": "周报", "run": False},
]
fix = [{"op": "update_node", "id": "gate", "config": {
"mode": "approve", "title": "请审核本周报告", "message": "批准后输出这份报告。\n\n{{ nodes.write.text }}"}},
{"op": "done"}]
model = Scripted(first, fix)
async def _model(*_a, **_k):
return model, "mock-fast"
monkeypatch.setattr(copilot, "get_chat_model", _model)
events: list[dict] = []
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
async with client.stream("POST", "/api/copilot/generate-stream", json={"instruction": "写周报并审批"}) as r:
async for line in r.aiter_lines():
if line.startswith("data: "):
events.append(json.loads(line[6:]))
checks = [e for e in events if e.get("op") == "check"]
assert checks[0]["status"] == "repairing"
assert any(i.get("code") == "human_formal_issuance" for i in checks[0]["issues"])
assert "出具契约" in model.calls[-1], "交回模型的修正请求里写明为什么不能这么说"
assert checks[-1]["status"] == "passed"
final = next(e for e in events if e.get("op") == "final")
gate = next(n for n in final["graph"]["nodes"] if n["id"] == "gate")
assert "正式出具" not in json.dumps(gate, ensure_ascii=False)
@@ -0,0 +1,53 @@
"""助手搭多来源汇总的分析报告时,默认给报告撰写节点开结论句裁判(claims: judge)。
真实运行里:四个 Agent 分头查库、汇总成一份分析报告,报告节点没写 claims。有一句结论和证据相矛盾,审批人批准时
完全不知道,事后在证据面板点开那句才判出来。分析报告的结论句恰恰最需要核对:几个来源拼在一起,最容易写出
证据撑不住的归因和判断。
取舍:默认写 claims: "judge"、不写 judge 的预算(取设置里的默认)——正式运行在节点里自动判,探索运行不在节点里
花钱、由读的人按需判。和「受管级别的 judge 必须写预算、拿不准就问」不冲突:那条只管要按受管级别发布的模板,
普通草稿不问预算;发布前修复里缺预算要问的规矩不变。用户明确说不要核对结论句的不加。问数据那条路径原本定了
不替人打开结论句策略(一律探索运行,读的人按需判),那条保留,并写明节点说明里的默认在问数里不适用。
这组守的是节点手册和问数据那条路径里的说法,以及它们和受管级别规则、发布前修复规则不打架。
"""
from __future__ import annotations
from app.api.copilot import NODE_REFERENCE, GenerateIn, _publish_fix_request, _user_message
def _report_rule() -> str:
start = NODE_REFERENCE.index("- report:报告撰写")
return NODE_REFERENCE[start:NODE_REFERENCE.index("\n- output 的出具契约", start)]
def test_multi_source_analysis_reports_default_to_the_judge():
rule = _report_rule()
assert '默认写 claims: "judge"' in rule
# 什么算多来源:几个 Agent 分头查、或者几次查询的结果合在一起写
assert "多个来源汇总成的分析报告" in rule and "几个 agent" in rule and "几次查询" in rule
# 不写预算,交给设置里的默认值
assert "不写 judge 的预算" in rule and "设置里的默认" in rule
# 正式运行在节点里自动判,探索运行由读的人按需判
assert "正式运行在节点里自动判" in rule and "探索运行由读的人" in rule
# 用户明确不要的不加
assert "用户明确说不要" in rule
def test_the_managed_level_budget_rule_still_applies_and_says_so():
# 受管级别仍然要写预算、拿不准就问;默认开裁判的那一条说明它让位给受管级别的规则
assert "judge 里必须写 judge.max_cost_usd" in NODE_REFERENCE
assert "拿不准就问" in NODE_REFERENCE
rule = _report_rule()
assert "受管级别" in rule
# 发布前修复(受管级别)缺预算时照旧问人,不替人定
request = _publish_fix_request({"nodes": [], "edges": []}, [], "governed")
assert "judge 缺预算(judge.max_cost_usd)时输出 question 问写多少" in request
def test_answer_path_keeps_its_own_rule_and_says_which_one_wins():
"""问数据那条路径原本就定了不替人打开结论句策略(test_copilot_entity_guide 守着):问数一律是探索运行,读答案的人
点开哪句判哪句。节点说明里新加的默认和它说法相反,所以这里写明那条默认在问数里不适用,免得模型两头为难。"""
text = _user_message(GenerateIn(instruction="各渠道上月表现怎么样", intent="answer"), patch=False)
assert "report 的 claims 不用写" in text
assert "分析报告默认开结论句裁判」那条在问数里不适用" in text
+67
View File
@@ -0,0 +1,67 @@
"""助手写查库 SQL 时给结果列起中文别名,报告里的表格表头直接可读。
真实运行里:报告节点用 [[table:…]] 把查询结果原样生成表格,表头就是 Agent 写的 SQL 里的列名——一串英文缩写的
原始列名,读报告的人得猜。结果列起中文别名(AS 月份、AS 曝光量)以后,表头、单元格引用、口径卡的
cell() 都直接用中文名。
中文别名要按方言加标识符引号:MySQL / MariaDB 用反引号,PostgreSQL、Oracle、SQLite 用双引号;单引号是字符串。
这组守的是:
- 节点手册里 Agent 查库的说明、写 SQL 的硬性要求(搭查库节点时的规则)都有这一条,按方言给出示例
- 加了引号的中文别名不让自查放弃列名核对:以前 SQL 里只要有带引号的标识符就整条不判,照这条规矩写的 SQL
会把猜错的英文列名一起放过去
"""
from __future__ import annotations
from types import SimpleNamespace
import pytest
from app.api.copilot import NODE_REFERENCE, sql_unknown_columns
from app.api.copilot_context import SQL_RULES_PROMPT
def _agent_rule() -> str:
start = NODE_REFERENCE.index("- agent:带工具循环的 agent")
return NODE_REFERENCE[start:NODE_REFERENCE.index("\n- supervisor:", start)]
@pytest.mark.parametrize("text", [_agent_rule(), SQL_RULES_PROMPT], ids=["agent", "sql-rules"])
def test_guidance_asks_for_chinese_aliases_with_dialect_quotes(text):
assert "中文别名" in text
assert "AS `月份`" in text and "MySQL" in text # MySQL / MariaDB:反引号
assert 'AS "月份"' in text and "PostgreSQL" in text and "Oracle" in text and "SQLite" in text
assert "单引号" in text # 单引号是字符串,不是别名
def _source(**tables):
return SimpleNamespace(name="shop", schema_cache={"tables": {
name: {"qualified": name, "columns": [{"name": c} for c in cols]} for name, cols in tables.items()}})
SHOP = _source(daily_stats=["id", "stat_date", "impressions", "clicks", "region"])
@pytest.mark.parametrize("sql", [
'SELECT stat_date AS "日期", SUM(impressions) AS "曝光量" FROM daily_stats GROUP BY stat_date ORDER BY "日期"',
"SELECT stat_date AS `日期`, SUM(impressions) AS `曝光量` FROM daily_stats GROUP BY stat_date ORDER BY `曝光量` DESC",
"SELECT stat_date AS 日期, SUM(clicks) AS 点击量 FROM daily_stats GROUP BY stat_date",
'SELECT region AS "区域", COUNT(*) AS "天数" FROM daily_stats WHERE region = \'华东\' GROUP BY region',
])
def test_quoted_chinese_aliases_are_fine(sql):
assert sql_unknown_columns(sql, SHOP) is None, sql_unknown_columns(sql, SHOP)
@pytest.mark.parametrize("sql, unknown", [
('SELECT stat_date AS "日期", SUM(impresions) AS "曝光量" FROM daily_stats GROUP BY stat_date', ["impresions"]),
("SELECT stat_month AS `月份`, SUM(impressions) AS `曝光量` FROM daily_stats GROUP BY stat_month",
["stat_month"]),
])
def test_guessed_columns_next_to_quoted_chinese_aliases_are_still_caught(sql, unknown):
found = sql_unknown_columns(sql, SHOP)
assert found is not None and found[0] == unknown, found
def test_quoted_ascii_identifiers_are_still_not_guessed():
# 带引号的英文标识符:大小写、空格都可能,照旧不判
assert sql_unknown_columns('SELECT "StatDate", impresions FROM daily_stats', SHOP) is None
+31 -6
View File
@@ -322,8 +322,9 @@ async def test_small_source_keeps_detail_and_carries_the_catalog(small_source):
assert f"中文名:订单{INFERRED_MARK}" in text and "单位:元" in text
assert f"orders.customer_id = customers.id(多对一){INFERRED_MARK}" in text
op = ctx.op()
assert op["sources"] == [{"source": small_source.name, "tables": ["customers", "orders"], "selected_by": "all",
"total": 2}]
assert op["sources"] == [{"source": small_source.name, "source_id": small_source.id,
"tables": ["customers", "orders"], "selected_by": "all", "total": 2,
"catalog_tables": 1}]
# ---------------------------------------------------------------- 回退:绝不阻断生成
@@ -346,8 +347,9 @@ async def test_pick_failures_fall_back_to_names_only(client, monkeypatch, scenic
"instruction": NEED, "intent": "answer", "datasource_ids": [scenic_source.id]})
op = _context_op(events)
assert op["sources"] == [{"source": scenic_source.name, "tables": [], "selected_by": "fallback",
"total": scenic.TABLE_COUNT + len(scenic.VIEWS), "reason": op["sources"][0]["reason"]}]
assert op["sources"] == [{"source": scenic_source.name, "source_id": scenic_source.id, "tables": [],
"selected_by": "fallback", "total": scenic.TABLE_COUNT + len(scenic.VIEWS),
"catalog_tables": 0, "reason": op["sources"][0]["reason"]}]
assert op["sources"][0]["reason"]
system = model.systems[0]
# 和以前一样只列表名、提醒先查字段
@@ -371,8 +373,9 @@ async def test_stream_reports_the_context_and_prompt_carries_selected_tables(cli
"instruction": NEED, "intent": "answer", "datasource_ids": [scenic_source.id]})
op = _context_op(events)
assert op["sources"] == [{"source": scenic_source.name, "tables": ["visits", "channels", "channel_visits"],
"selected_by": "model", "total": scenic.TABLE_COUNT + len(scenic.VIEWS)}]
assert op["sources"] == [{"source": scenic_source.name, "source_id": scenic_source.id,
"tables": ["visits", "channels", "channel_visits"], "selected_by": "model",
"total": scenic.TABLE_COUNT + len(scenic.VIEWS), "catalog_tables": 0}]
assert isinstance(op["elapsed_ms"], int) and op["elapsed_ms"] >= 0
# context 在模型首帧之后、第一条图操作之前
kinds = [e["op"] for e in events]
@@ -423,6 +426,28 @@ async def test_self_check_repair_reuses_the_same_context(client, monkeypatch, sc
assert len(picker.calls) == 1
async def test_context_op_says_how_much_of_each_source_has_a_catalog(scenic_source):
"""context 操作里每个源带上有数据目录的表数(catalog_tables)和源的 id:没有目录的大库,界面提示先去起草目录。
真实运行里数据源一张目录都没有,挑表和 SQL 检查都用不上目录,用户也不知道该先去起草。只算有可见项的表:
整份被驳回的不算有目录。"""
from app.api import copilot_context
async def op() -> dict:
plan = await copilot_context.plan_context([scenic_source])
return (await plan.resolve(Picker(PICKED), need=NEED)).op()["sources"][0]
first = await op()
assert first["catalog_tables"] == 0 and first["source_id"] == scenic_source.id
async with SessionLocal() as session:
await catalog.write_entry(session, scenic_source.id, "visits", {
"label": make_item("入园记录", "llm")}, if_version=0, actor="王敏")
rejected = make_item("门店", "llm")
rejected["status"] = "rejected"
await catalog.write_entry(session, scenic_source.id, "stores", {"label": rejected}, if_version=0, actor="王敏")
assert (await op())["catalog_tables"] == 1
async def test_small_sources_do_not_call_the_picker(client, monkeypatch, small_source):
picker = Picker(error=AssertionError("小库不该挑表"))
_use_picker(monkeypatch, picker)
+32
View File
@@ -17,6 +17,7 @@
"""
from __future__ import annotations
import asyncio
import json
import re
@@ -464,6 +465,37 @@ async def test_cost_truncation_follows_priority(monkeypatch, stored):
assert {status_of(outcome, doc, t)["status"] for t in (CITED, CITED2)} == {"supported"}
async def test_a_slow_batch_times_out_alone_and_the_others_keep_their_verdicts(monkeypatch, stored):
"""真实运行里推理模型拿着 40 句一批,30 秒没回来,整份报告一句都没判成(同一个模型 5 句一批 3 秒多就回来)。
现在小批:第一批单独问,之后几批同时问、共用剩下的时长;超时只丢没回来的那一批,回来的判定照样留下。"""
Judge(monkeypatch)
catalog = make_catalog(stored)
raw = "".join(f"第 {i} 个区域的销售额增长来自新客。[[see:m:gmv]]" for i in range(BATCH_SIZE * 3))
doc = compose_doc(raw, catalog)
real_ask = judge_mod._ask
asked: list[int] = []
running = peak = 0
async def ask(model, messages):
nonlocal running, peak
asked.append(len(re.findall(r"^\[(u\d+)\]", str(messages[-1].content), re.M)))
running += 1
peak = max(peak, running)
try:
await asyncio.sleep(5 if len(asked) == 3 else 0.05) # 第三批卡住
return await real_ask(model, messages)
finally:
running -= 1
monkeypatch.setattr(judge_mod, "_ask", ask)
outcome = await judge_doc(doc, catalog, spec=SPEC,
budget=Budget(max_claims=None, max_cost_usd=None, timeout_s=1, daily_max_usd=None))
assert asked == [BATCH_SIZE] * 3
assert peak == 2, "第一批单独问,后两批同时问"
assert outcome["counts"]["supported"] == BATCH_SIZE * 2
assert outcome["unjudged"] == {"timeout_s": BATCH_SIZE} and outcome["limits_hit"] == ["timeout_s"]
async def test_unlimited_never_truncates(monkeypatch, stored):
"""每一项都写 null:句数再多也全判(按每批上限分批),金额、时长、每日都不拦。"""
judge = Judge(monkeypatch)
+5 -1
View File
@@ -374,11 +374,15 @@ async def test_formal_runs_are_judged_inside_the_node_not_on_demand(client, monk
async def test_a_run_paused_at_an_approval_is_not_judged(client, monkeypatch):
"""停在审批上的运行还没封存,按需裁判不接。交给审批的报告探索运行也在节点里判过了
(test_approval_report_check),这里只看点击:不再多一次裁判调用,也不追加判定事件。"""
models = Models(monkeypatch)
row = await finish(weekly(claims="judge", gate=True), statuses=("interrupted",))
judged_in_node = list(models.judges)
assert judged_in_node
res = await judge(client, row.id, ["u1"])
assert res.status_code == 409 and res.json()["code"] == "evidence_judge_unsealed"
assert models.judges == [] and await judged_events(row.id) == []
assert models.judges == judged_in_node and await judged_events(row.id) == []
async def test_runs_from_before_the_upgrade_are_not_judged(client, monkeypatch):
+208
View File
@@ -0,0 +1,208 @@
"""cite_fields 的结构化抽取:输出被截断时加大额度重试,仍截断就照实说是额度在推理阶段用完了。
2026-10 一次运行里 4 个开了 cite_fields 的 Agent 抽取全部失败:抽取用 thinking: off、额度 4096,而经 OpenAI 兼容接口
接入的推理模型关不掉思考,4096 在推理阶段就用完了,openai 抛 LengthFinishReasonError(「Could not parse response
content as the length limit was reached」)。日志却写「请稍后重试;如持续失败,请到设置测试该模型」——重试、测试都没用。
- 额度默认至少 16384,节点配了更大的用节点的
- 认出「输出被截断」:openai 的 length limit 类错误、finish_reason=length、Anthropic 的 stop_reason=max_tokens;
被截断时额度翻倍(最多 32768)重试一次
- 仍然截断:日志 code agent_field_truncated,说清是额度在推理阶段用完了,给出能做的下一步;字段记空值,不以 0 代替
- 服务方嫌额度太大(400,报错里有 max_tokens 一类字样):退回小一档的额度重来,不能因为额度提高反而抽不成
- 非截断类的失败照旧(agent_field_unverified)
夹具全部合成:门店、会员这类虚构的表。
"""
from __future__ import annotations
import json
import pytest
from langchain_core.messages import AIMessage
from openai import LengthFinishReasonError
from openai.types import CompletionUsage
from openai.types.chat import ChatCompletion
from app.engine.nodes import llm as llm_mod
from app.engine.runner import run_manager
from tests.test_agent_cited_fields import ( # noqa: F401 - 夹具按名字引入
EXTRACTED,
FINAL,
SCHEMA,
SQLS,
TOOL,
Script,
engine_up,
events,
graph,
logs,
output_of,
shop,
wait,
)
def length_error(prompt: int = 12000, completion: int = 4096) -> LengthFinishReasonError:
"""openai 的结构化输出在 finish_reason=length 时抛的那个错(和真实运行里的原话一样)。"""
usage = CompletionUsage(completion_tokens=completion, prompt_tokens=prompt, total_tokens=prompt + completion)
return LengthFinishReasonError(completion=ChatCompletion.model_construct(
id="x", choices=[], created=0, model="m", object="chat.completion", usage=usage))
class BudgetRejected(Exception):
"""服务方嫌输出额度太大时的 400(DeepSeek chat 系只收 8K)。"""
status_code = 400
class Extraction:
"""记下每次抽取调用的额度(thinking 关掉的那次 get_chat_model),并按额度决定这次怎么回。
outcome(budget) 返回 None 表示照常回 EXTRACTED,抛异常就是这次调用失败。
"""
def __init__(self, monkeypatch, outcome):
from app.providers import mock_model
self.budgets: list[int | None] = []
real = llm_mod.get_chat_model
async def spy(session, spec):
if spec.thinking == "off":
self.budgets.append(spec.max_tokens)
return await real(session, spec)
monkeypatch.setattr(llm_mod, "get_chat_model", spy)
base = Script(monkeypatch)
loop = mock_model.MockChatModel._decide
def decide(model, messages):
if model.response_format:
outcome(self.budgets[-1])
return AIMessage(content=json.dumps(EXTRACTED, ensure_ascii=False))
return loop(model, messages)
monkeypatch.setattr(mock_model.MockChatModel, "_decide", decide)
self.script = base
# --------------------------------------------------------------------------
# 认出「输出被截断」
# --------------------------------------------------------------------------
def test_length_errors_are_recognised():
assert llm_mod.truncated_error(length_error())
assert llm_mod.truncated_error(RuntimeError("Could not parse response content as the length limit was reached"))
assert not llm_mod.truncated_error(RuntimeError("网关超时"))
assert not llm_mod.truncated_error(BudgetRejected("max_tokens is too large: 16384"))
def test_cut_off_replies_are_recognised_for_both_families():
assert llm_mod.truncated_reply(AIMessage(content="", response_metadata={"finish_reason": "length"}))
assert llm_mod.truncated_reply(AIMessage(content="", response_metadata={"stop_reason": "max_tokens"}))
assert not llm_mod.truncated_reply(AIMessage(content="", response_metadata={"finish_reason": "stop"}))
assert not llm_mod.truncated_reply(None)
# --------------------------------------------------------------------------
# 额度与重试
# --------------------------------------------------------------------------
async def test_the_default_budget_leaves_room_for_reasoning(monkeypatch):
spy = Extraction(monkeypatch, lambda budget: None)
row = await wait((await run_manager.start(graph=graph(card=False), input_payload={})).id)
assert row.status == "succeeded", row.error
assert spy.budgets == [16384], "推理模型的思考也算在额度里:默认至少 16384"
async def test_a_larger_node_budget_is_used(monkeypatch):
spy = Extraction(monkeypatch, lambda budget: None)
row = await wait((await run_manager.start(graph=graph(card=False, max_tokens=24000), input_payload={})).id)
assert row.status == "succeeded", row.error
assert spy.budgets == [24000]
async def test_a_cut_off_extraction_is_retried_with_a_larger_budget(monkeypatch):
def outcome(budget):
if budget < 32768:
raise length_error(completion=budget)
spy = Extraction(monkeypatch, outcome)
row = await wait((await run_manager.start(graph=graph(card=False), input_payload={})).id)
assert row.status == "succeeded", row.error
assert spy.budgets == [16384, 32768], "截断后翻倍重试一次"
bot = await output_of(row.id, "bot")
assert bot["data"]["gmv"] == 600.5 and bot["data_evidence"]["gmv"]["status"] == "verified"
evts = await events(row.id)
assert not logs(evts, "agent_field_truncated")
# EXTRACTED 里 region 指向本节点没有的 Q9:只有它核对不了,别的字段都照常核对
assert [w["fields"] for w in logs(evts, "agent_field_unverified")] == [["region"]]
# 两次调用各记一行:第一次写明为什么重来
ends = [e.data for e in evts if e.type == "llm.end" and e.data.get("purpose") == "cite_fields"]
assert len(ends) == 2 and "32768" in ends[0]["error"] and not ends[1].get("error")
# 被截断那次的用量照样记账(花出去的 token 是真的)
assert ends[0]["output_tokens"] == 16384
async def test_cut_off_twice_is_reported_as_truncation_with_fields_empty(monkeypatch):
def outcome(budget):
raise length_error(completion=budget)
spy = Extraction(monkeypatch, outcome)
row = await wait((await run_manager.start(graph=graph(card=False), input_payload={})).id)
assert row.status == "succeeded", row.error
assert spy.budgets == [16384, 32768], "只重试一次"
bot = await output_of(row.id, "bot")
assert bot["data"] == {k: None for k in SCHEMA["properties"]}, "记空值,不以 0 代替"
assert all(e["status"] == "unresolved" for e in bot["data_evidence"].values())
evts = await events(row.id)
assert not logs(evts, "agent_field_unverified")
[warn] = logs(evts, "agent_field_truncated")
assert warn["level"] == "warn" and warn["fields"] == list(SCHEMA["properties"])
message = warn["message"]
assert "推理阶段就用完了" in message and "结构化结论没能写出来" in message
assert "32768" in message and "非推理模型" in message and "最大输出 token" in message
assert "请稍后重试" not in message and "测试该模型" not in message, "重试、测试模型都不解决额度问题"
assert "未以 0 代替" in message
assert warn.get("max_tokens") == 32768
async def test_a_node_budget_at_the_cap_is_not_retried(monkeypatch):
def outcome(budget):
raise length_error(completion=budget)
spy = Extraction(monkeypatch, outcome)
row = await wait((await run_manager.start(graph=graph(card=False, max_tokens=40000), input_payload={})).id)
assert row.status == "succeeded", row.error
assert spy.budgets == [40000], "额度已经超过重试上限,再调一次也一样"
[warn] = logs(await events(row.id), "agent_field_truncated")
assert "40000" in warn["message"]
async def test_a_budget_the_service_rejects_falls_back(monkeypatch):
def outcome(budget):
if budget > 8192:
raise BudgetRejected(f"Invalid max_tokens value, the valid range of max_tokens is [1, 8192]: {budget}")
spy = Extraction(monkeypatch, outcome)
row = await wait((await run_manager.start(graph=graph(card=False), input_payload={})).id)
assert row.status == "succeeded", row.error
assert spy.budgets == [16384, 8192]
bot = await output_of(row.id, "bot")
assert bot["data_evidence"]["gmv"]["status"] == "verified"
async def test_other_failures_are_reported_as_before(monkeypatch):
def outcome(budget):
raise RuntimeError("网关超时")
spy = Extraction(monkeypatch, outcome)
row = await wait((await run_manager.start(graph=graph(card=False), input_payload={})).id)
assert row.status == "succeeded", row.error
assert spy.budgets == [16384], "不是截断就不重试"
evts = await events(row.id)
assert not logs(evts, "agent_field_truncated")
[warn] = logs(evts, "agent_field_unverified")
assert "网关超时" in warn["message"] and "未以 0 代替" in warn["message"]
+246
View File
@@ -0,0 +1,246 @@
"""OpenAI 兼容接入的流式用量回传(stream_options.include_usage)。
真实运行里:模型走 LiteLLM 的 OpenAI 兼容接口,上百次模型调用,运行用量却是 0 token、0 美元。ChatOpenAI
只在没填 base_url 时默认开流式用量回传;填了 base_url(所有兼容接口都填)就不开,流里没有 usage 那一块,
llm.end 记下的全是 0,运行的用量是 llm.end 逐次加总的,于是也是 0。
这组守的是:
- 兼容接入构造出来的模型开着流式用量回传,请求里带 stream_options.include_usage
- 流里最后那块 usage 合进响应,llm.end 和运行的用量里 token、金额都对
- 不认 stream_options 的服务:同一次调用去掉它重发一次,照常流式输出,不让调用失败;同一个接口之后不再带它
- 和用量回传无关的报错不重发、原样抛出
假的是 HTTP:openai SDK 发出去的请求在它的 HTTP 库的传输层被接住,按请求体回 SSE。模型名、内容都是虚构的。
"""
from __future__ import annotations
import asyncio
import json
import uuid
import pytest
from langchain_core.messages import HumanMessage
from sqlalchemy import delete, select
from app.db.base import SessionLocal
from app.db.models import Provider, Run, RunEvent
from app.engine.runner import run_manager
from app.providers import catalog, factory
from app.providers.factory import ModelSpec, build_chat_model
try: # openai SDK 3 起换成了 httpx2(和 httpx 同一套接口):假服务要装在 SDK 真正用的那个库上
from langchain_openai._compat import httpx
except ImportError: # pragma: no cover - 老版本的 langchain_openai
import httpx
PRICED = "gpt-5.2-mini"
USAGE = {"prompt_tokens": 1200, "completion_tokens": 300, "total_tokens": 1500}
def _chunk(delta: dict | None = None, *, finish: str | None = None, usage: dict | None = None) -> str:
body = {"id": "c-1", "object": "chat.completion.chunk", "created": 1, "model": PRICED,
"choices": [] if delta is None else [{"index": 0, "delta": delta, "finish_reason": finish}]}
if usage is not None:
body["usage"] = usage
return f"data: {json.dumps(body, ensure_ascii=False)}\n\n"
class FakeServer:
"""OpenAI 兼容服务:带了 stream_options.include_usage 才在流尾补 usage 那一块。
rejects:不认 stream_options 的服务,带了就回 status(400 / 422)和 message;
fail:每个请求都回这个错误(和用量回传无关的报错)。"""
def __init__(self, *, rejects: tuple[int, str] | None = None, fail: tuple[int, str] | None = None) -> None:
self.rejects, self.fail = rejects, fail
self.bodies: list[dict] = []
async def handle(self, request: httpx.Request) -> httpx.Response:
body = json.loads(request.content or b"{}")
self.bodies.append(body)
wants = bool((body.get("stream_options") or {}).get("include_usage"))
if self.fail or (self.rejects and "stream_options" in body):
status, message = self.fail or self.rejects
return httpx.Response(status, json={"error": {"message": message, "type": "invalid_request_error"}},
request=request)
if not body.get("stream"):
reply = {"id": "c-1", "object": "chat.completion", "created": 1, "model": PRICED,
"choices": [{"index": 0, "message": {"role": "assistant", "content": "上周到店 120 人。"},
"finish_reason": "stop"}], "usage": USAGE}
return httpx.Response(200, json=reply, request=request)
sse = (_chunk({"role": "assistant", "content": "上周"}) + _chunk({"content": "到店 120 人。"})
+ _chunk({}, finish="stop") + (_chunk(usage=USAGE) if wants else "") + "data: [DONE]\n\n")
return httpx.Response(200, headers={"content-type": "text/event-stream"}, content=sse.encode(),
request=request)
@pytest.fixture
def server(monkeypatch):
def install(**kwargs) -> FakeServer:
fake = FakeServer(**kwargs)
monkeypatch.setattr(httpx.AsyncHTTPTransport, "handle_async_request", fake.handle)
return fake
# 记住「这个接口不认 stream_options」的表是进程级的:每个用例从空表开始
monkeypatch.setattr(factory, "_NO_STREAM_USAGE", set(), raising=False)
return install
def _provider(base_url: str | None = None, *, kind: str = "openai_compatible") -> Provider:
return Provider(name=f"compat-{uuid.uuid4().hex[:8]}", kind=kind,
base_url=base_url or f"http://llm-{uuid.uuid4().hex[:6]}.example.test/v1",
api_key=None, models=[{"id": PRICED}], default_model=PRICED, enabled=True, extra={})
async def _stream(model) -> tuple[str, dict | None, int]:
"""流式调用一次:(正文, 合并后的 usage_metadata, 块数)。"""
merged, count = None, 0
async for chunk in model.astream([HumanMessage(content="上周到店多少人")]):
merged = chunk if merged is None else merged + chunk
count += 1
return str(merged.content), merged.usage_metadata, count
# ---------------------------------------------------------------- 构造
@pytest.mark.parametrize("kind", ["openai_compatible", "openai"])
def test_compatible_providers_turn_on_streaming_usage(kind):
model = build_chat_model(_provider(kind=kind), ModelSpec())
assert model.stream_usage is True
# ---------------------------------------------------------------- 流式调用
async def test_usage_chunk_is_requested_and_merged(server):
fake = server()
model = build_chat_model(_provider(), ModelSpec())
text, usage, _ = await _stream(model)
assert text == "上周到店 120 人。"
assert fake.bodies[0]["stream_options"] == {"include_usage": True}
assert usage and usage["input_tokens"] == 1200 and usage["output_tokens"] == 300
@pytest.mark.parametrize("status,message", [
(400, "Unrecognized request argument supplied: stream_options"),
(422, "body -> stream_options: extra fields not permitted"),
(400, "Bad Request"),
], ids=["named-400", "named-422", "generic-400"])
async def test_service_rejecting_stream_options_falls_back_and_is_remembered(server, status, message):
fake = server(rejects=(status, message))
provider = _provider()
model = build_chat_model(provider, ModelSpec())
text, usage, count = await _stream(model)
# 照常流式输出:去掉 stream_options 重发了一次,正文分块到达
assert text == "上周到店 120 人。" and count >= 2
assert ["stream_options" in b for b in fake.bodies] == [True, False]
assert usage is None or not usage.get("input_tokens"), "这个服务给不了流式用量,不编"
# 同一个接口之后不再带 stream_options(新构造的模型也一样),不再每次先失败一次
again = build_chat_model(provider, ModelSpec())
await _stream(again)
assert ["stream_options" in b for b in fake.bodies] == [True, False, False]
async def test_other_errors_are_not_retried(server):
fake = server(fail=(401, "invalid api key"))
model = build_chat_model(_provider(), ModelSpec())
with pytest.raises(Exception, match="invalid api key"):
await _stream(model)
assert len(fake.bodies) == 1
async def test_a_failing_retry_is_not_remembered(server):
"""去掉 stream_options 重发也失败:说明不是它的事,报重发的那个错,也不记成这个接口不认它。"""
fake = server(fail=(400, "max_tokens is too large"))
provider = _provider()
model = build_chat_model(provider, ModelSpec())
with pytest.raises(Exception, match="max_tokens is too large"):
await _stream(model)
assert ["stream_options" in b for b in fake.bodies] == [True, False]
assert provider.base_url not in factory._NO_STREAM_USAGE
# ---------------------------------------------------------------- 运行的用量
@pytest.fixture
async def engine_up():
await run_manager.setup()
yield
await run_manager.shutdown()
async def _run_once(model: str) -> tuple[Run, list[RunEvent]]:
"""兼容接入上用 model 跑一个 llm 节点,返回运行行和 llm.end 事件。"""
provider = _provider()
async with SessionLocal() as session:
session.add(provider)
await session.commit()
await session.refresh(provider)
session.expunge(provider)
try:
graph = {
"nodes": [
{"id": "in", "type": "input", "data": {"config": {}}},
{"id": "ask", "type": "llm", "data": {"label": "问答", "config": {
"prompt": "上周到店多少人", "provider": provider.name, "model": model}}},
{"id": "out", "type": "output", "data": {"config": {
"fields": [{"name": "回答", "value": "{{ nodes.ask.text }}"}]}}},
],
"edges": [{"source": "in", "target": "ask"}, {"source": "ask", "target": "out"}],
}
started = await run_manager.start(graph=graph, input_payload={})
for _ in range(200):
await asyncio.sleep(0.05)
async with SessionLocal() as session:
row = await session.get(Run, started.id)
if row.status in ("succeeded", "failed"):
break
assert row.status == "succeeded", row.error
async with SessionLocal() as session:
ends = list((await session.execute(select(RunEvent).where(
RunEvent.run_id == started.id, RunEvent.type == "llm.end"))).scalars())
return row, ends
finally:
async with SessionLocal() as session:
await session.execute(delete(Provider).where(Provider.id == provider.id))
await session.commit()
async def test_run_usage_counts_streamed_tokens(server, engine_up):
"""整条链:兼容接入 → llm 节点流式调用 → llm.end → 运行的用量。修之前这里是 0 token、0 美元。"""
server()
row, ends = await _run_once(PRICED)
assert [(e.data["input_tokens"], e.data["output_tokens"]) for e in ends] == [(1200, 300)]
assert row.usage["input_tokens"] == 1200 and row.usage["output_tokens"] == 300
assert row.usage["total_tokens"] == 1500
expected = round(catalog.estimate_cost(PRICED, 1200, 300), 6)
assert expected > 0 and row.usage["cost_usd"] == pytest.approx(expected)
assert "unpriced" not in ends[0].data and "unpriced_calls" not in row.usage
async def test_a_model_missing_from_the_price_catalog_is_marked_unpriced(server, engine_up):
"""token 记到了,模型却不在价格目录里:金额估不出来,不能让运行页写 $0 像是没花钱。
llm.end 标 unpriced(运行中的实时统计据此数),运行用量记 unpriced_calls(界面据此写「无法估算」)。"""
server()
row, ends = await _run_once("vendor-unlisted-model")
assert [(e.data["input_tokens"], e.data["output_tokens"]) for e in ends] == [(1200, 300)]
assert ends[0].data["unpriced"] is True and ends[0].data["cost_usd"] == 0
assert row.usage["total_tokens"] == 1500 and row.usage["cost_usd"] == 0
assert row.usage["unpriced_calls"] == 1
@pytest.mark.parametrize(("data", "unpriced"), [
({"model": "vendor-unlisted-model", "input_tokens": 10, "output_tokens": 2, "cost_usd": 0}, True),
({"model": PRICED, "input_tokens": 10, "output_tokens": 2, "cost_usd": 0.0001}, False),
# 目录里标 0 价的本地模型是真不花钱,不是估不出来
({"model": "qwen3:8b", "input_tokens": 10, "output_tokens": 2, "cost_usd": 0}, False),
# 没用 token 的调用(失败、被取消)谈不上金额
({"model": "vendor-unlisted-model", "input_tokens": 0, "output_tokens": 0}, False),
# 老事件没有 unpriced 键:按模型名补判;事件自己标了的以它为准
({"model": PRICED, "input_tokens": 10, "unpriced": True}, True),
(None, False),
])
def test_unpriced_call(data, unpriced):
assert catalog.unpriced_call(data) is unpriced
@@ -0,0 +1,82 @@
"""缺失断言(absence_claim):写作目录有省略时,「没有查到」「不存在」「缺失」「没有数据」这类结论依据不足。
2026-10 一次运行,写作者只看到上百次查询里的前二十来次,写出「某类数据缺失:本轮没有查到任何结果」这样的断言,
可那个 Agent 实际查到了上百条记录。
- 写作规则:清单有省略、或有来源的结构化结论为空时,不得断言没有数据,只能写「本报告未覆盖」并说明依据不足
- 组装文档:清单有省略时按一组明确的说法认出缺失断言,记 absence_claim,交回重写一次;重写后仍有的只标注、不判失败
(report.REWRITE_ONLY,和句中的 [[see:]] 同一种处理)
- 只认拿得准的说法:「不存在明显差异」「本报告未覆盖」「不代表没有数据」都不算
"""
from __future__ import annotations
import pytest
from app.engine.evidence import ABSENCE_RULE, build_catalog, compose_doc
from app.engine.nodes.report import REWRITE_ONLY
@pytest.fixture
def catalog():
card = {"kind": "metric_set", "caliber": "周报口径", "caliber_version": "v1",
"metrics": [{"id": "gmv", "name": "销售额", "value": 45678.5, "unit": "元", "format": "thousands"}]}
return build_catalog(nodes={"card": card}, ledger=[{"kind": "metric_set", "node_id": "card", "artifact": "c" * 64}])
def claims(doc):
return [v for v in doc["violations"] if v["code"] == "absence_claim"]
def test_the_rule_tells_the_writer_what_to_write_instead():
for phrase in ("没有查到", "不存在", "缺失", "没有数据", "本报告未覆盖", "依据不足"):
assert phrase in ABSENCE_RULE
assert "absence_claim" in REWRITE_ONLY and "inline_see" in REWRITE_ONLY
@pytest.mark.parametrize("text, found", [
("卡券数据缺失:本轮没有查到核销、兑换、退券的任何结果。", ["数据缺失", "没有查到"]),
("卡券核销没有任何记录。", ["没有任何记录"]),
("本周门店客流数据为空。", ["数据为空"]),
("活动内容无数据。", ["无数据"]),
("未查询到会员流失的情况。", ["未查询到"]),
("库里不存在相关记录。", ["不存在相关记录"]),
("兑换记录均不存在。", ["记录均不存在"]),
("暂无数据,查不到核销明细。", ["暂无数据", "查不到"]),
("| 方向 | 结论 |\n| --- | --- |\n| 卡券 | 没有查到 |", ["没有查到"]),
])
def test_absence_claims_are_recognised(catalog, text, found):
doc = compose_doc(text, catalog, absence=True)
got = claims(doc)
assert [v["text"] for v in got] == found, doc["violations"]
for v in got:
assert doc["markdown"][v["span"][0]:v["span"][1]] == v["text"], "span 指着正文里的那几个字"
assert v.get("unit") and v.get("context")
assert "依据不足" in v["message"] and "本报告未覆盖" in v["for_model"]
@pytest.mark.parametrize("text", [
"本报告未覆盖卡券核销的明细:证据目录有省略,相关查询未展开,依据不足。",
"两家门店的客单价不存在明显差异。",
"目录里没展开的查询不代表没有数据。",
"这并非没有查到,而是没有展开。",
"本周无数据库变更,无数据源切换。",
"本周销售额 [[m:gmv]],缺少口径说明的部分已注明。",
"```sql\nSELECT * FROM t -- 没有查到\n```",
])
def test_other_wording_is_left_alone(catalog, text):
assert not claims(compose_doc(text, catalog, absence=True))
def test_only_checked_when_the_catalog_was_trimmed(catalog):
text = "卡券核销数据缺失:没有查到任何记录。"
assert not claims(compose_doc(text, catalog))
assert claims(compose_doc(text, catalog, absence=True))
def test_the_claim_can_be_located_in_the_document(catalog):
doc = compose_doc("## 卡券\n\n本周销售额 [[m:gmv]]。核销数据缺失。", catalog, absence=True)
[v] = claims(doc)
segments = {s["id"]: s for b in doc["blocks"] for u in b["units"] for s in u["segments"]}
seg = segments[v["segment"]]
assert seg["kind"] == "text" and seg["span"][0] <= v["span"][0] < seg["span"][1]
assert v["context"] == "核销数据缺失。"
+251
View File
@@ -0,0 +1,251 @@
"""写作者的出处清单:预算按写作模型的上下文窗口定;超出预算时分优先级、按来源均衡,省略了什么照实说。
2026-10 一次运行:证据台账里上百次查询(4 个 Agent 查询次数悬殊),出处清单写死 12000 字,十万字以上的目录
被截到一万两千字以内,只有前二十来次查询进了写作者的视野,后面只有一句「目录太长,后面的省略了」。查询按编号排,前面的
Agent 占满了预算,有的 Agent 一次都没露面;写作者于是写出「本轮没有查到任何核销结果」,可那个 Agent 查到了上百条。
- 预算:写作模型上下文窗口的 40%(字数按 1 字 ≈ 1 token 保守算),拿不到窗口按 128K;下限 12000、上限 160000
- 第一优先:各 Agent 经按出处核对交来的结构化字段、口径卡指标,一律列出
- 查询按产出它的节点分组、轮流取:先给每次查询前几行,再补到 20 行;每个节点都有代表
- 没展开的查询列出编号和一行摘要(表、行数);约定只能写 [[see:Qn]] 作依据,数字照样逐字核对
- 清单开头明说「目录有省略」,各节点各省略了多少条
夹具全部合成(tests/fixtures/evidence/store_run.py)。
"""
from __future__ import annotations
import re
import pytest
import app.db.models # noqa: F401 - 建表(conftest 的 create_all)前要先登记模型
from app.engine.evidence import (
CATALOG_MAX_CHARS,
CATALOG_MIN_CHARS,
DEFAULT_CATALOG_BUDGET,
build_catalog,
catalog_budget,
catalog_prompt,
writer_catalog,
)
from tests.fixtures.evidence import store_run
LEDGER, STORE, NODES, TITLES = store_run.build()
def load(artifact):
return STORE.get(artifact)
@pytest.fixture
def catalog():
return build_catalog(nodes=NODES, ledger=LEDGER)
def sources():
return [{"node_id": nid, "title": TITLES[nid], "data": NODES[nid]["data"],
"evidence": NODES[nid]["data_evidence"]} for nid, *_ in store_run.AGENTS]
def written(catalog, budget=DEFAULT_CATALOG_BUDGET, **kw):
return writer_catalog(catalog, budget=budget, loader=load, fields=sources(), titles=TITLES, **kw)
def group_of(text: str, title: str) -> str:
"""清单里「来自「<名字>」的查询」那一段(到下一个来源或下一节为止)。"""
start = text.index(f"来自「{title}」的查询")
rest = text[start + 1:]
ends = [m.start() for m in re.finditer(r"\n(?:来自「|\n)", rest)]
return text[start:start + 1 + (ends[0] if ends else len(rest))]
# --------------------------------------------------------------------------
# 预算
# --------------------------------------------------------------------------
def test_budget_follows_the_model_window():
assert DEFAULT_CATALOG_BUDGET == catalog_budget(None) == 51_200, "拿不到窗口按 128K 的 40%"
assert catalog_budget(128_000) == 51_200
assert catalog_budget(200_000) == 80_000
assert catalog_budget(32_000) == 12_800
assert catalog_budget(8_000) == CATALOG_MIN_CHARS == 12_000, "很小的窗口维持原来的篇幅"
assert catalog_budget(1_000_000) == CATALOG_MAX_CHARS == 160_000, "百万窗口也有上限:每写一稿都要重发整份目录"
assert catalog_budget(0) == catalog_budget(-5) == DEFAULT_CATALOG_BUDGET
def test_the_default_is_far_above_the_old_cut():
assert DEFAULT_CATALOG_BUDGET >= 4 * 12_000
# --------------------------------------------------------------------------
# 放得下:照旧
# --------------------------------------------------------------------------
def test_a_catalog_that_fits_is_not_trimmed():
small = build_catalog(nodes={}, ledger=[e for e in LEDGER if e.get("node_id") == "agent_store"])
got = writer_catalog(small, loader=load)
assert not got.trimmed and not got.omitted
assert "目录有省略" not in got.text
assert got.text == catalog_prompt(small, loader=load), "没有结构化字段时和 catalog_prompt 一字不差"
assert "Q17:" in got.text and " r19:" in got.text
# --------------------------------------------------------------------------
# 放不下:分优先级、按来源均衡(那次运行同样形状的场景)
# --------------------------------------------------------------------------
@pytest.mark.parametrize("budget", [DEFAULT_CATALOG_BUDGET, 12_000])
def test_every_agent_stays_in_view_within_the_budget(catalog, budget):
got = written(catalog, budget)
text = got.text
assert got.trimmed and len(text) <= budget, len(text)
for nid, title, count, table in store_run.AGENTS:
group = group_of(text, title)
assert f"共 {count} 次" in group
assert re.search(r"\n {2}r0:", group), f"「{title}」一次查询都没展开"
# 每次查询都在清单里:展开的有行,没展开的有编号和摘要
total = sum(c for *_, c, _ in store_run.AGENTS)
for n in range(1, total + 1):
assert re.search(rf"(?:^|\n)\s*- Q{n}:", text), f"Q{n} 不见了"
def test_omitted_queries_are_listed_with_a_summary(catalog):
got = written(catalog)
assert got.omitted, "这个场景放不下全部展开"
for alias in got.omitted:
node = catalog[alias]["node_id"]
table = next(t for nid, _, _, t in store_run.AGENTS if nid == node)
assert re.search(rf"\n {{2}}- {alias}:{table},30 行", got.text), alias
# 没展开的查询没有行
first = got.omitted[0]
block = got.text[got.text.index(f"- {first}:"):].split("\n", 2)
assert not block[1].lstrip().startswith("r0:")
def test_the_head_says_what_was_left_out_per_source(catalog):
got = written(catalog)
head = got.text[:800]
assert "目录有省略" in head
for row in got.nodes:
assert f"{row['title']} {row['total']} 次" in head and f"省略 {row['omitted']} 次" in head, row
assert sum(r["omitted"] for r in got.nodes) == len(got.omitted)
assert [r["node_id"] for r in got.nodes] == [nid for nid, *_ in store_run.AGENTS]
def test_how_to_use_an_omitted_query_is_spelled_out(catalog):
text = written(catalog).text
assert "[[see:Qn]]" in text and "本报告未覆盖" in text
assert "不要凭摘要写" in text and "数字照样逐字核对" in text
for phrase in ("没有查到", "不存在", "缺失", "没有数据"):
assert phrase in text
def test_structured_fields_come_first_and_point_at_global_cells(catalog):
got = written(catalog, 12_000)
text = got.text
fields_at = text.index("各 Agent 交来的结构化结论")
assert fields_at < text.index("来自「"), "结构化结论排在查询前面"
member_q3 = store_run.global_alias("agent_member", 3)
store_q3 = store_run.global_alias("agent_store", 3)
campaign_q5 = store_run.global_alias("agent_campaign", 5)
assert f"[[v:{member_q3}.r0.cnt]]" in text and f"[[v:{store_q3}.r0.store_name]]" in text
assert f"[[table:{campaign_q5} cols=store_name rows=0-4]]" in text
assert "churn_rate:抽取时没有找到这个值" in text
# 抽取失败的那个 Agent:结构化结论为空,不能据此断言没有数据
block = text[text.index("「卡券核销」:", fields_at):]
assert block.startswith("「卡券核销」:结构化结论为空") and "本报告未覆盖" in block.split("\n", 1)[0]
assert got.empty_sources == ["卡券核销"]
def test_queries_the_fields_cite_are_expanded_first(catalog):
"""结构化字段引用到的那几次查询先展开:写作者照抄字段的引用标记时,看得到那几行。"""
got = written(catalog, 12_000)
for nid in ("agent_member", "agent_store", "agent_campaign"):
for local in (3, 5):
alias = store_run.global_alias(nid, local)
assert alias not in got.omitted, alias
assert re.search(rf"\n- {alias}:.*\n {{2}}r0:", got.text), alias
def test_a_field_value_reads_like_the_cell_it_cites(catalog):
"""字段后面写的值和引用之后报告里显示的一样([[v:]] 的渲染规则)。"""
text = written(catalog).text
q3 = STORE[store_run.artifact_of("agent_member", 3)]
assert f"total_cnt = {q3['rows'][0][2]:,}:[[v:Q3.r0.cnt]]" in text
# --------------------------------------------------------------------------
# 报告节点:窗口从哪来
# --------------------------------------------------------------------------
async def test_the_window_comes_from_the_provider_entry_then_the_catalog():
"""模型接入里给这个模型登记的 context 优先,其次模型目录;都没有返回 None(预算按保守值)。"""
from sqlalchemy import delete
from app.db.base import SessionLocal
from app.db.models import Provider
from app.engine.nodes.report import model_window
from app.providers.factory import ModelSpec
async with SessionLocal() as session:
row = Provider(name="fa-window-test", kind="mock", enabled=True, default_model="mock-wide",
models=[{"id": "mock-wide", "context": 64_000}, {"id": "mock-text", "context": "96000"},
{"id": "mock-bare"}])
session.add(row)
await session.commit()
try:
assert await model_window(session, ModelSpec(model="mock-wide"), "mock-wide") == 64_000
assert await model_window(session, ModelSpec(model="mock-text"), "mock-text") == 96_000
# 接入里没登记窗口:按模型目录(deepseek-chat 登记了 128K);目录里也没有就是 None
assert await model_window(session, ModelSpec(model="deepseek-chat"), "deepseek-chat") == 128_000
assert await model_window(session, ModelSpec(model="mock-bare"), "mock-bare") is None
assert catalog_budget(await model_window(session, ModelSpec(model="mock-wide"), "mock-wide")) == 25_600
finally:
await session.execute(delete(Provider).where(Provider.id == row.id))
await session.commit()
# ---------------------------------------------------------------- 警告说的是实情
def _writer(nodes, omitted=()):
from app.engine.evidence import WriterCatalog
return WriterCatalog(text="", budget=51200, queries=sum(r["total"] for r in nodes), trimmed=True, nodes=nodes,
omitted=list(omitted))
def _row(title, total, *, full=0, compact=0, omitted=0):
return {"node_id": title, "title": title, "total": total, "full": full, "compact": compact, "omitted": omitted}
def test_the_warning_names_omitted_queries_when_there_are_some():
from app.engine.nodes.report import _trimmed_message
text = _trimmed_message(_writer([_row("会员专项", 26, full=26), _row("卡券核销", 79, full=20, omitted=59)],
omitted=[f"Q{i}" for i in range(1, 60)]))
assert "105 次查询中 46 次列出了数据,59 次只列编号和摘要(卡券核销省略 59 次)" in text
assert "看不到这些查询的明细" in text
def test_all_queries_listed_but_some_only_first_rows_is_not_called_omitted():
"""放不下但查询全都列出,只是有的只列前几行:不写「0 次只列编号和摘要()」,写清是行数有省略。"""
from app.engine.nodes.report import _trimmed_message
writer = _writer([_row("会员专项", 26, full=26), _row("卡券核销", 79, full=30, compact=49)])
text = _trimmed_message(writer)
assert writer.compact == 49
assert f"105 次查询都列出了数据,其中 49 次只列前 {writer.compact_rows} 行(卡券核销 49 次)" in text
assert "看不到这些查询其余的行" in text
assert "只列编号和摘要" not in text and "()" not in text
def test_only_the_table_notes_were_cut():
from app.engine.nodes.report import _trimmed_message
text = _trimmed_message(_writer([_row("会员专项", 26, full=26)]))
assert "26 次查询都列出了数据,表和字段、知识库检索的说明有省略" in text and "()" not in text
@@ -0,0 +1,118 @@
"""报告里单元格的 ISO 日期时间显示成「2026-06-30 08:21:39」,不再是「2026-06-30T08:21:39」。
只改显示:快照里的值、引用记下的 value、eid、证据下钻都不变。复核要按文档生成时的规则:老文档里存的是带 T 的字,
拿新规则重新渲染会对不上,被当成文档被改过(render_mismatch)——所以新文档在顶层记 cell_display: 2(只在真有日期
时间被改写时记,没有的文档和以前一字不差),没有这个字段的老文档按 1 版(原样)复核。期 4 的金样
(test_provenance_unchanged)照旧逐字不变。
夹具合成:门店、卡券。
"""
from __future__ import annotations
import pytest
from app.core.artifact_store import canonical_json
from app.engine import evidence as ev
from app.engine.evidence import (
CELL_DISPLAY,
StreamRenderer,
build_catalog,
cell_eid,
compose_doc,
doc_cell_display,
iter_segments,
verify_doc,
writer_catalog,
)
ART = "d" * 64
SNAP = {"columns": ["store_name", "redeemed_at", "biz_date", "amount", "note"],
"column_types": {"store_name": "text", "redeemed_at": "datetime", "biz_date": "date", "amount": "number",
"note": "text"},
"rows": [["门店01", "2026-06-30T08:21:39", "2026-06-30", 120, "T 开头的备注"],
["门店02", "2026-06-29T23:05:00.250000+08:00", "2026-06-29", 80, "2026-06-29T10:00:00Z"]],
"truncated": False}
STORE = {ART: SNAP}
def load(artifact):
return STORE.get(artifact)
@pytest.fixture
def catalog():
ledger = [{"kind": "query", "node_id": "fetch", "exec": 1, "artifact": ART, "tool": "db_query__shop",
"columns": SNAP["columns"], "rows": 2, "truncated": False}]
return build_catalog(nodes={}, ledger=ledger)
def cells(doc):
return [s for s in iter_segments(doc) if s.get("ref", "").startswith("v:")]
def test_iso_datetimes_read_with_a_space(catalog):
doc = compose_doc("最近一笔核销在 [[v:Q1.r0.redeemed_at]],门店 [[v:Q1.r0.store_name]]。[[see:Q1]]", catalog, loader=load)
[when, store] = cells(doc)
assert when["text"] == "2026-06-30 08:21:39" and "2026-06-30 08:21:39" in doc["markdown"]
# 只改显示:引用记下的值、证据标识照旧
assert when["cite"]["value"] == "2026-06-30T08:21:39"
assert when["cite"]["eid"] == cell_eid(ART, 0, "redeemed_at")
assert store["text"] == "门店01"
assert doc["cell_display"] == CELL_DISPLAY == 2 and doc_cell_display(doc) == 2
assert '"cell_display":2' in canonical_json(doc), "在文档顶层,参与内容哈希"
assert verify_doc(doc, catalog, loader=load)["ok"]
def test_fractions_and_offsets_are_kept_and_dates_are_untouched(catalog):
doc = compose_doc("时间 [[v:Q1.r1.redeemed_at]],日期 [[v:Q1.r1.biz_date]],备注 [[v:Q1.r1.note]]。", catalog,
loader=load)
assert [s["text"] for s in cells(doc)] == ["2026-06-29 23:05:00.250000+08:00", "2026-06-29", "2026-06-29 10:00:00Z"]
assert verify_doc(doc, catalog, loader=load)["ok"]
def test_tables_read_the_same_way(catalog):
doc = compose_doc("[[table:Q1 cols=store_name,redeemed_at]]", catalog, loader=load)
assert "| 门店01 | 2026-06-30 08:21:39 |" in doc["markdown"]
assert "T08:21:39" not in doc["markdown"]
assert verify_doc(doc, catalog, loader=load)["ok"]
def test_documents_without_datetimes_are_unchanged(catalog):
doc = compose_doc("门店 [[v:Q1.r0.store_name]],金额 [[v:Q1.r0.amount]]。", catalog, loader=load)
assert "cell_display" not in doc, "没有被改写的日期时间:文档和以前一字不差"
assert doc_cell_display(doc) == 1
def test_old_documents_still_verify_as_they_were(catalog, monkeypatch):
"""改动前组装的文档:存的是带 T 的字、没有 cell_display。按 1 版复核,不报 render_mismatch。"""
monkeypatch.setattr(ev, "CELL_DISPLAY", 1)
old = compose_doc("最近一笔核销在 [[v:Q1.r0.redeemed_at]]。", catalog, loader=load)
monkeypatch.undo()
assert "cell_display" not in old and "2026-06-30T08:21:39" in old["markdown"]
checked = verify_doc(old, catalog, loader=load)
assert checked["ok"], checked["violations"]
def test_a_document_edited_back_is_caught(catalog):
doc = compose_doc("最近一笔核销在 [[v:Q1.r0.redeemed_at]]。", catalog, loader=load)
[seg] = cells(doc)
start = seg["span"][0]
doc["markdown"] = doc["markdown"].replace("2026-06-30 08:21:39", "2026-06-30T08:21:39")
seg["text"] = "2026-06-30T08:21:39"
seg["span"] = [start, start + len(seg["text"])]
codes = {v["code"] for v in verify_doc(doc, catalog, loader=load)["violations"]}
assert "render_mismatch" in codes
def test_an_unknown_display_version_is_reported(catalog):
doc = compose_doc("最近一笔核销在 [[v:Q1.r0.redeemed_at]]。", catalog, loader=load)
doc["cell_display"] = 9
assert "bad_schema" in {v["code"] for v in verify_doc(doc, catalog, loader=load)["violations"]}
def test_the_writer_and_the_stream_see_what_the_report_will_show(catalog):
text = writer_catalog(catalog, loader=load).text
assert "redeemed_at=2026-06-30 08:21:39" in text and "T08:21:39" not in text
stream = StreamRenderer(catalog, loader=load)
shown = stream.feed("最近一笔在 [[v:Q1.r0.redeemed_at]]。") + stream.flush()
assert "2026-06-30 08:21:39" in shown
@@ -0,0 +1,316 @@
"""回归:「4 个专项 Agent 并行查库 → 报告」整条跑一遍,复现 2026-10 那次运行的形状。
那次运行状态是 succeeded,报告却不可靠:4 个 Agent 的结构化抽取全部被截断(推理模型的额度在推理阶段就用完了);
写作者的出处清单写死 12000 字,上百次查询只有前二十来次露了面;写作者于是在清单被截断的情况下写了「数据缺失:
本轮没有查到任何核销结果」,可那个 Agent 查到了上百条。
这里用合成的门店、会员、卡券、活动库(SQLite)真跑:4 个 Agent 分别查 26、17、79、42 次(虚构的次数,一次并行发完),都配了
output_schema 和按出处核对字段;写作模型、抽取模型都是脚本。验证:
- 抽取被截断后加大额度重试成功;两次都截断的记 agent_field_truncated,字段为空值
- 出处清单在预算内,每个 Agent 都有代表,被省略的查询有编号和摘要,开头说明各来源省略了几次
- 清单有省略时的缺失断言被认出来(absence_claim)、交回重写;重写后仍有的只标注、不判失败
"""
from __future__ import annotations
import asyncio
import json
import re
import sqlite3
import pytest
from langchain_core.messages import AIMessage, HumanMessage, SystemMessage, ToolMessage
from openai import LengthFinishReasonError
from openai.types import CompletionUsage
from openai.types.chat import ChatCompletion
from sqlalchemy import select
from app.core import artifact_store
from app.data.engine import engines
from app.db.base import SessionLocal
from app.db.models import DataSource, Run, RunEvent
from app.engine.evidence import ABSENCE_RULE, DEFAULT_CATALOG_BUDGET, MARKER_RULES
from app.engine.nodes import llm as llm_mod
from app.engine.nodes import report as report_mod
from app.engine.runner import run_manager
from tests.fixtures.sources import drop_source
SOURCE = "fa_store"
TOOL = f"db_query__{SOURCE}"
#: (节点 id, 画布上的名字, 记号, 查询次数, 表, 列)
AGENTS = [
("agent_member", "会员专项", "member", 26, "member_daily", ["store_name", "biz_date", "new_members", "active_members"]),
("agent_store", "门店专项", "store", 17, "store_daily", ["store_name", "biz_date", "visits", "sales"]),
("agent_coupon", "卡券核销", "coupon", 79, "coupon_redemption", ["store_name", "redeemed_at", "coupon_type", "amount"]),
("agent_campaign", "活动内容", "campaign", 42, "campaign_view", ["campaign", "biz_date", "views", "clicks"]),
]
SCHEMA = {"type": "object", "properties": {
"first_store": {"type": "string", "description": "第一次查询第一行的门店或活动"},
"first_value": {"type": "number", "description": "第一次查询第一行的第三列"},
"churn_rate": {"type": "number", "description": "流失率(要计算,填 null)"},
}}
ROWS_PER_QUERY = 5
ABSENT = ("## 概览\n\n各专项都已完成查询。[[see:Q1]]\n\n"
"## 卡券核销\n\n卡券数据缺失:本轮没有查到核销、兑换的任何结果。")
COVERED = ("## 概览\n\n各专项都已完成查询。[[see:Q1]]\n\n"
"## 卡券核销\n\n本报告未覆盖卡券核销的明细:证据目录有省略,相关查询未展开,依据不足。[[see:Q1]]")
def length_error(completion: int) -> LengthFinishReasonError:
usage = CompletionUsage(completion_tokens=completion, prompt_tokens=14000, total_tokens=14000 + completion)
return LengthFinishReasonError(completion=ChatCompletion.model_construct(
id="x", choices=[], created=0, model="m", object="chat.completion", usage=usage))
@pytest.fixture(autouse=True)
async def engine_up():
await run_manager.setup()
yield
await run_manager.shutdown()
@pytest.fixture
async def store_db(tmp_path):
"""合成库:每张表 120 行,第 i 次查询取 OFFSET i 的 5 行,次次结果不同。"""
path = tmp_path / "store.db"
db = sqlite3.connect(path)
for _, _, _, _, table, cols in AGENTS:
db.execute(f"CREATE TABLE {table} ({cols[0]} TEXT, {cols[1]} TEXT, {cols[2]} {'TEXT' if table == 'coupon_redemption' else 'INTEGER'}, {cols[3]} REAL)")
rows = []
for r in range(120):
when = f"2026-06-{r % 28 + 1:02d}" + (f"T08:{r % 60:02d}:39" if table == "coupon_redemption" else "")
third = f"券型{r % 4 + 1}" if table == "coupon_redemption" else (r * 7) % 300 + 5
rows.append((f"门店{r % 30 + 1:02d}", when, third, round((r * 13) % 900 + 0.5, 1)))
db.executemany(f"INSERT INTO {table} VALUES (?, ?, ?, ?)", rows)
db.commit()
db.close()
async with SessionLocal() as session:
row = DataSource(name=SOURCE, kind="sqlite", database=str(path), readonly=True, enabled=True)
session.add(row)
await session.commit()
source_id = row.id
await engines.invalidate(source_id)
yield path
await drop_source(source_id)
def sql_of(table: str, cols: list[str], i: int) -> str:
return f"SELECT {', '.join(cols)} FROM {table} ORDER BY rowid LIMIT {ROWS_PER_QUERY} OFFSET {i}"
def node(nid, ntype, label, **config):
return {"id": nid, "type": ntype, "position": {"x": 0, "y": 0}, "data": {"label": label, "config": config}}
def graph(counts: dict[str, int] | None = None, **report):
nodes = [node("start", "input", "开始")]
for nid, title, key, count, table, cols in AGENTS:
nodes.append(node(nid, "agent", title, prompt=f"【{key}】按{title}的口径查数,每次查一段", tools=[TOOL],
approval="never", parallel_tools=True, output_schema=SCHEMA, cite_fields=True,
assign_to=f"{key}_findings"))
nodes.append(node("write", "report", "专项报告", instructions="汇总各专项的发现", **report))
nodes.append(node("out", "output", "成果", fields=[{"name": "报告", "value": "{{ nodes.write.text }}"}]))
edges = [{"source": "start", "target": a[0]} for a in AGENTS] + [{"source": a[0], "target": "write"} for a in AGENTS]
edges.append({"source": "write", "target": "out"})
return {"nodes": nodes, "edges": edges}
class World:
"""脚本化的模型:Agent 一步并行发完自己的查询再给结论;抽取按 outcome 决定;写作者依次回 drafts。"""
def __init__(self, monkeypatch, db_path, *, counts: dict[str, int], drafts: list[str], outcome):
from app.providers import mock_model
self.counts = counts
self.drafts = drafts
self.writer: list[list] = []
self.extract_calls: dict[str, list[int | None]] = {a[2]: [] for a in AGENTS}
self.budgets: list[int | None] = []
db = sqlite3.connect(db_path)
self.first_rows = {key: db.execute(sql_of(table, cols, 1)).fetchone() for _, _, key, _, table, cols in AGENTS}
db.close()
real = llm_mod.get_chat_model
async def spy(session, spec):
if spec.thinking == "off":
self.budgets.append(spec.max_tokens)
return await real(session, spec)
monkeypatch.setattr(llm_mod, "get_chat_model", spy)
world = self
def text(messages) -> str:
return "\n".join(m.content if isinstance(m.content, str) else json.dumps(m.content, ensure_ascii=False)
for m in messages)
def who(messages) -> str:
found = re.search(r"【(member|store|coupon|campaign)】", text(messages))
assert found, "认不出是哪个 Agent"
return found.group(1)
def decide(model, messages):
if model.response_format:
key = who(messages)
world.extract_calls[key].append(world.budgets[-1])
outcome(key, len(world.extract_calls[key]), world.budgets[-1])
return AIMessage(content=json.dumps(world.extracted(key), ensure_ascii=False))
if any(isinstance(m, SystemMessage) and "报告撰写人" in str(m.content) for m in messages):
world.writer.append(list(messages))
return AIMessage(content=world.drafts[min(len(world.writer), len(world.drafts)) - 1])
key = who(messages)
if model.tools and not any(isinstance(m, ToolMessage) for m in messages):
_, _, _, _, table, cols = next(a for a in AGENTS if a[2] == key)
return AIMessage(content="分段查询。", tool_calls=[
{"name": TOOL, "args": {"sql": sql_of(table, cols, i)}, "id": f"{key}_{i}"}
for i in range(1, world.counts[key] + 1)])
return AIMessage(content=f"【{key}】查完了,结论见各次查询。")
monkeypatch.setattr(mock_model.MockChatModel, "_decide", decide)
def extracted(self, key: str) -> dict:
first = self.first_rows[key]
_, _, _, _, _, cols = next(a for a in AGENTS if a[2] == key)
return {"first_store": {"value": first[0], "from": {"call": "Q1", "row": 0, "column": cols[0]}},
"first_value": {"value": first[3], "from": {"call": "Q1", "row": 0, "column": cols[3]}},
"churn_rate": {"value": None, "from": None}}
async def finish(graph_json) -> Run:
run = await run_manager.start(graph=graph_json, input_payload={})
for _ in range(1200):
await asyncio.sleep(0.05)
async with SessionLocal() as session:
row = await session.get(Run, run.id)
if row.status in ("succeeded", "failed") and row.manifest_seq is not None:
return row
raise AssertionError(f"run {run.id} 没跑完:{row.status}")
async def events(run_id: str, etype: str | None = None) -> list[RunEvent]:
async with SessionLocal() as session:
q = select(RunEvent).where(RunEvent.run_id == run_id)
if etype:
q = q.where(RunEvent.type == etype)
return list((await session.execute(q.order_by(RunEvent.seq))).scalars())
def logs(evts, code):
return [e.data for e in evts if e.type == "log" and e.data.get("code") == code]
async def output_of(run_id: str, node_id: str) -> dict:
[finished] = [e for e in await events(run_id, "node.finished") if e.node_id == node_id]
return artifact_store.load(finished.data["artifact"])
def catalog_text(messages) -> str:
"""写作者第一次收到的提示里出处清单那一段(从「可引用的证据」到写作规则之前)。"""
prompt = next(m.content for m in messages if isinstance(m, HumanMessage))
return prompt[prompt.index("可引用的证据"):prompt.index(MARKER_RULES)].rstrip()
# --------------------------------------------------------------------------
# 和那次运行同样的形状:上百次查询、分布不均
# --------------------------------------------------------------------------
async def test_the_reported_run_shape_end_to_end(monkeypatch, store_db):
counts = {key: count for _, _, key, count, _, _ in AGENTS}
def outcome(key, attempt, budget):
if key == "coupon":
raise length_error(budget) # 两次都截断
if key == "member" and attempt == 1:
raise length_error(budget) # 截断一次,加大额度后成功
world = World(monkeypatch, store_db, counts=counts, drafts=[ABSENT, COVERED], outcome=outcome)
row = await finish(graph())
assert row.status == "succeeded", row.error
evts = await events(row.id)
# 1. 抽取:会员专项截断后重试成功;卡券核销两次都截断,字段为空值
assert world.extract_calls["member"] == [16384, 32768]
assert world.extract_calls["coupon"] == [16384, 32768]
assert world.extract_calls["store"] == world.extract_calls["campaign"] == [16384]
member = await output_of(row.id, "agent_member")
assert member["data"]["first_store"] == world.first_rows["member"][0]
assert member["data_evidence"]["first_store"]["status"] == "verified"
coupon = await output_of(row.id, "agent_coupon")
assert coupon["data"] == {"first_store": None, "first_value": None, "churn_rate": None}
[cut] = logs(evts, "agent_field_truncated")
assert cut["fields"] == ["first_store", "first_value", "churn_rate"] and "推理阶段就用完了" in cut["message"]
assert not logs(evts, "agent_field_unverified")
# 2. 出处清单:预算内,每个 Agent 都有代表,被省略的有编号和摘要,开头说明各来源省略了几次
first = world.writer[0]
text = catalog_text(first)
assert len(text) <= DEFAULT_CATALOG_BUDGET, len(text)
head = text[:900]
assert "目录有省略" in head
for _, title, _, count, table, _ in AGENTS:
assert f"{title} {count} 次" in head
group = text[text.index(f"来自「{title}」的查询"):]
group = group[:group.index("\n\n", 1)] if "\n\n" in group[1:] else group
assert f"共 {count} 次" in group and re.search(r"\n {2}r0:", group), f"「{title}」没有展开的查询"
briefs = re.findall(r"\n {2}- (Q\d+):(\w+),5 行", text)
assert briefs, "没有列出被省略的查询"
assert {t for _, t in briefs} <= {a[4] for a in AGENTS}
assert all(f"Q{n}:" in text for n in range(1, 165)), "每次查询都要在清单里(展开的或摘要)"
# 结构化结论排在查询前面;抽取失败的卡券核销写明结构化结论为空、不代表没有数据
assert text.index("各 Agent 交来的结构化结论") < text.index("来自「")
assert "「卡券核销」:结构化结论为空" in text
# 写作规则里有缺失断言那一条
assert ABSENCE_RULE in next(m.content for m in first if isinstance(m, HumanMessage))
[trimmed] = logs(evts, "report_catalog_trimmed")
assert trimmed["queries"] == 164 and trimmed["budget"] == DEFAULT_CATALOG_BUDGET
assert [n["title"] for n in sorted(trimmed["nodes"], key=lambda n: n["title"])] == \
sorted(a[1] for a in AGENTS)
assert trimmed["omitted"] == sum(n["omitted"] for n in trimmed["nodes"]) > 0
# 3. 缺失断言:认出来、交回重写一次;改成「本报告未覆盖」后通过
assert len(world.writer) == 2
request = str(world.writer[1][-1].content)
assert "本报告未覆盖" in request and "没有查到" in request
[repair] = logs(evts, "report_repair")
assert "数据缺失" in repair["message"] or "没有查到" in repair["message"]
[checked] = [e.data for e in evts if e.type == "report.checked"]
assert not [v for v in checked["violations"] if v["code"] == "absence_claim"]
assert not logs(evts, "report_absence_claim")
# --------------------------------------------------------------------------
# 重写后仍然写缺失断言:只标注、不判失败
# --------------------------------------------------------------------------
SMALL = {"member": 6, "store": 4, "coupon": 12, "campaign": 8}
async def test_an_absence_claim_that_survives_the_rewrite_is_flagged_not_failed(monkeypatch, store_db):
monkeypatch.setattr(report_mod, "catalog_budget", lambda window: 4000)
world = World(monkeypatch, store_db, counts=SMALL, drafts=[ABSENT], outcome=lambda *a: None)
row = await finish(graph(on_violation="fail", max_repairs=3))
assert row.status == "succeeded", row.error
assert len(world.writer) == 2, "缺失断言只交回重写一次,不按最多重写次数反复重写"
evts = await events(row.id)
[checked] = [e.data for e in evts if e.type == "report.checked"]
assert checked["failed"] is False
claims = [v for v in checked["violations"] if v["code"] == "absence_claim"]
assert {v["text"] for v in claims} == {"数据缺失", "没有查到"}
assert all(v.get("segment") and v.get("context") for v in claims), "带着片段和原句:证据面板能定位"
[flag] = logs(evts, "report_absence_claim")
assert flag["level"] == "warn" and flag["count"] == 2 and flag["texts"] == ["数据缺失", "没有查到"]
assert "本报告未覆盖" in flag["message"] or "依据不足" in flag["message"]
async def test_without_omissions_an_absence_statement_is_left_alone(monkeypatch, store_db):
"""目录全给了:「没有查到」可能就是看了数据说的,不查。"""
counts = {"member": 1, "store": 1, "coupon": 1, "campaign": 1}
world = World(monkeypatch, store_db, counts=counts, drafts=[ABSENT], outcome=lambda *a: None)
row = await finish(graph())
assert row.status == "succeeded", row.error
assert len(world.writer) == 1
evts = await events(row.id)
assert not logs(evts, "report_catalog_trimmed") and not logs(evts, "report_absence_claim")
prompt = next(m.content for m in world.writer[0] if isinstance(m, HumanMessage))
assert ABSENCE_RULE not in prompt and "目录有省略" not in prompt
+29 -3
View File
@@ -488,11 +488,35 @@ function Expanding({ dense, n }: { dense: boolean; n: number }) {
const APPROVAL = {
id: 'ap-1', run_id: 'syn-mixed', node_id: 'review', mode: 'approve' as const,
title: '出勤结论可以发出吗?', status: 'pending', response: {},
payload: { message: '早班出勤率 **94.23%**,晚班 **73.08%**。' },
payload: { kind: 'human_node', message: '早班出勤率 **94.23%**,晚班 **73.08%**。' },
created_at: new Date(Date.now() - 8 * 86_400_000 - 3_600_000).toISOString().replace('Z', ''),
workflow_name: '出勤日报', node_label: '班长复核', run_class: 'formal' as const,
}
/**
* ?approval=report:探索运行里审一份报告。核对概况取 evidence-doc 夹具(7/12 数字有出处、1 处引用解析不了),
* 报告没开结论句裁判;?approval=report-formal:正式运行,节点里判过,1 句证据相矛盾
*/
const APPROVAL_REPORT = {
id: 'ap-2', run_id: evidenceFixture.run_id, node_id: 'review', mode: 'approve' as const,
title: '本周周报可以发出吗?', status: 'pending', response: {},
payload: {
kind: 'human_node', message: '请审核本周周报。',
reports: [{ node_id: 'write', label: '写周报', stats: evidenceFixture.doc.stats, claims: 'off',
doc_artifact: evidenceFixture.doc_artifact }],
},
created_at: new Date(Date.now() - 120_000).toISOString().replace('Z', ''),
workflow_name: '周报', node_label: '主管审批', run_class: 'exploratory' as const,
}
const APPROVAL_REPORT_FORMAL = {
...APPROVAL_REPORT, id: 'ap-3', run_class: 'formal' as const,
payload: {
...APPROVAL_REPORT.payload,
reports: [{ ...APPROVAL_REPORT.payload.reports[0], claims: 'judge', stats: { ...evidenceFixture.doc.stats, numbers_cited: 12, uncited_numbers: 0, unresolved: 0 },
judge: { mode: 'inline', counts: { supported: 5, contradicted: 1, unjudged: 0 }, complete: true } }],
},
}
function Preview() {
// ?case=db 只看一个用例,?theme=light 直接出浅色——都是为了截图可复现,
// 不用靠点按钮
@@ -525,11 +549,13 @@ function Preview() {
if (md) return <div className="h-full overflow-y-auto"><MarkdownCases /></div>
if (params.get('grow') === '1') return <Growing dense={dense} />
if (params.get('expand')) return <Expanding dense={dense} n={Number(params.get('expand')) || 1} />
if (params.get('approval') === '1') {
const approvalCase = params.get('approval')
if (approvalCase === '1' || approvalCase === 'report' || approvalCase === 'report-formal') {
const approval = approvalCase === 'report' ? APPROVAL_REPORT : approvalCase === 'report-formal' ? APPROVAL_REPORT_FORMAL : APPROVAL
return (
<div className={dense ? 'w-[360px] border-r' : 'mx-auto max-w-3xl p-4'}>
<div className="overflow-hidden rounded-lg border" style={{ borderColor: 'var(--warn)' }}>
<ApprovalCard approval={APPROVAL} onResolved={() => undefined} />
<ApprovalCard approval={approval} onResolved={() => undefined} />
</div>
</div>
)
+64 -1
View File
@@ -46,7 +46,7 @@ import type {
CatalogCardinality, CatalogMeasure, CatalogProfileNumberKey, CatalogSource, CatalogStatus, CatalogTableKind, NodeType, SnapshotReasonCode,
SqlCheckCode, SqlCheckLevel, ToolTrust,
} from '../types'
import { formatNumber } from './format'
import { formatCost, formatNumber } from './format'
export const TERMS = {
workflow: '工作流',
@@ -87,6 +87,35 @@ export function runClassLabel(runClass: 'formal' | 'exploratory' | null | undefi
return '探索运行'
}
/**
* 审批卡上人工审批的运行类别和审的报告的核对概况。出具只发生在成果节点配了出具契约、从已发布版本发起的正式运行里:
* 探索运行批准后只是输出,卡上要说破,不能让审批人以为自己在签发正式的东西
*/
export const APPROVAL_TEXT = {
/** 人工审批批了会怎样,按运行类别说 */
outcome: (runClass: 'formal' | 'exploratory' | null | undefined, approve: boolean) => (runClass === 'formal'
? `从已发布版本发起,${approve ? '批准' : '提交'}后继续运行`
: `${approve ? '批准后输出' : '提交后继续运行'},这是探索运行,不是正式出具`),
reportHead: (label: string) => `报告「${label}」的核对`,
noNumbers: '报告中没有需要核对的数字',
/** 节点里当场裁判过(正式运行开了结论句裁判):判了几句、几句证据相矛盾、几句没判成 */
claimsJudged: (judged: number, contradicted: number, unjudged: number) =>
`结论句已裁判 ${formatNumber(judged)} 句,证据相矛盾 ${formatNumber(contradicted)} 句`
+ (unjudged ? `,未裁判 ${formatNumber(unjudged)} 句` : ''),
claimsNone: '报告中没有需要裁判的结论句',
claimsPending: '结论句尚未核对',
/** 探索运行、报告没开结论句裁判:按需裁判要等运行结束封存之后。开了裁判的报告交给审批时会在节点里当场判,不走这句 */
claimsLater: '这份报告没有开启结论句裁判,运行结束后可在证据中逐句请模型判断;开启后,交给审批的报告会在批准前判完',
/** 开了结论句裁判、却没在节点里判(这条规则之前的探索运行):按需裁判要等运行结束封存之后 */
claimsAfterRun: '这次没有在节点内裁判,运行结束后可在证据中逐句请模型判断',
/** 正式运行、报告没开结论句裁判 */
claimsOff: '这份报告没有开启结论句裁判',
openEvidence: '查看逐段证据',
closeEvidence: '收起证据',
docLoading: '正在加载报告文档…',
docMissing: '无法获取报告文档',
}
/**
* 没挂在工作流上的运行:画布或问数据当场搭的,从没存成工作流。后端给它的名字老数据是
* 「临时图」、新的是「未保存的工作流」,也可能是空的;界面上一律叫「未保存的工作流」。
@@ -203,6 +232,19 @@ export function evidenceTally(cited: number, total: number, other = 0): string {
return parts.join(' · ')
}
/**
* 缺失断言:证据目录有省略时,报告写了「没有查到」「不存在」「缺失」这类结论(违规 absence_claim,重写后仍有时
* 报告节点另发日志 report_absence_claim)。证据面板的违规清单和运行时间线共用这一套说法
*/
export const ABSENCE_CLAIM_TEXT = {
label: '缺失断言',
hint: '写作模型没有看到全部证据:被省略的查询仍在证据台账里,看不到不等于不存在,这类结论应写作「本报告未覆盖」。'
+ '报告照常产出,这一处只标注、不判失败',
title: (n?: number) => `报告写了${n ? ` ${formatNumber(n)} 处` : ''}缺失断言,已标注(证据目录有省略)`,
next: '被省略的查询仍在证据台账里,可在证据面板中查看。请在报告撰写节点的指令中要求未展开的方向写作「本报告未覆盖」,'
+ '或换用上下文窗口更大的写作模型后重新运行',
}
/** 证据面板和证据条的文案 */
export const EVIDENCE_TEXT = {
allCited: (n: number) => `${formatNumber(n)} 个数字都有出处`,
@@ -609,6 +651,24 @@ export const JUDGE_ON_UNSUPPORTED_LABEL: Record<string, string> = {
}
/** 设置页「证据裁判」一组(后端 engine/judge.py 的 JUDGE_DEFAULTS) */
/** 运行的金额。有调用的模型不在价格目录中时金额估不出来,不写「$0」(看着像没花钱) */
export const COST_TEXT = {
unpriced: '金额无法估算',
/** 悬停说明,n 是估不出金额的调用次数 */
unpricedTitle: (n: number) => `${formatNumber(n)} 次调用的模型不在价格目录中,金额无法估算,未计入`,
} as const
/**
* 运行级金额的写法。unpriced 是估不出金额的调用次数:一分都没估出来写「金额无法估算」,估出了一部分写「≥ $x」;
* 运行中(approx)只算已上报的调用,也写「≥」。没有金额返回 null,由调用方写占位
*/
export function runCostText(usd: number | null | undefined, unpriced: number, approx = false): string | null {
const known = typeof usd === 'number' && Number.isFinite(usd)
if (unpriced > 0 && !(known && usd > 0)) return COST_TEXT.unpriced
if (!known) return null
return `${(approx || unpriced > 0) && usd > 0 ? '≥ ' : ''}${formatCost(usd)}`
}
export const JUDGE_SETTING_TEXT = {
label: '证据裁判',
hint: '报告中的结论句(如「增长主要来自新客首单」)由另一个模型按证据逐句判断:正式运行由报告撰写节点当场裁判,探索运行在点击时逐句裁判。判断由模型给出,不是系统核对,界面上一律标注「非确定」。',
@@ -2657,4 +2717,7 @@ export const COPILOT_CONTEXT_TEXT = {
withFields: (n: number) => `参考了 ${formatNumber(n)} 张表的字段`,
namesOnly: (n: number) => `另有 ${formatNumber(n)} 张表只提供了表名`,
notPicked: '未能按需求挑选',
/** 没有数据目录的大库:只有一个数据源时说「这个数据源」,几个时点名 */
noCatalog: (who: string | null) => `${who ?? '这个数据源'}还没有数据目录,起草后挑表和 SQL 检查会更准`,
toCatalog: '去数据目录起草',
}
+5 -2
View File
@@ -10,7 +10,7 @@ import { type StatusCode } from '../../lib/status'
import { isExecuting, liveAt, project, type Projection, type RunPhase, type Trace } from '../../run/trace'
import { useRunClock } from '../../run/useRunClock'
import type { Approval, Run } from '../../types'
import { UNSAVED_HINT, isUnsaved, runName } from '../../lib/terms'
import { COST_TEXT, UNSAVED_HINT, isUnsaved, runCostText, runName } from '../../lib/terms'
import {
LONG_WAIT_MS, ageMs, cancelReason, runScope, runTiming, type RunTiming,
} from './model'
@@ -92,6 +92,9 @@ export function Telemetry({ run, trace, phase, code, streaming, pending, labelOf
const tokens = settled || !streaming ? tokensSettled ?? (liveTokens || undefined) : liveTokens
const cost = settled || !streaming ? (isNum(usage.cost_usd) ? usage.cost_usd : trace.costUsd || undefined) : trace.costUsd
const approx = streaming && !settled
// 金额估不出来的调用(模型不在价格目录中):有的话不写 $0
const unpriced = (settled || !streaming ? (isNum(usage.unpriced_calls) ? usage.unpriced_calls : undefined) : undefined)
?? trace.unpricedCalls
const running = Object.values(trace.nodes).filter(isExecuting)
const waitingId = trace.waitingNodeId ?? pending?.[0]?.node_id
@@ -156,7 +159,7 @@ export function Telemetry({ run, trace, phase, code, streaming, pending, labelOf
<Value data="tokens">
{tokens ? <>{approx && '≥ '}{formatTokens(tokens, { compact: true })}</> : NONE}
</Value>
<Sub>{isNum(cost) && cost > 0 ? `${approx ? '≥ ' : ''}${formatCost(cost)}` : isNum(cost) ? formatCost(cost) : ' '}</Sub>
<Sub title={unpriced ? COST_TEXT.unpricedTitle(unpriced) : undefined}>{runCostText(cost, unpriced, approx) ?? ' '}</Sub>
</Cell>
</div>
)
+4 -2
View File
@@ -11,7 +11,7 @@ import { explainRunError } from '../../lib/explain'
import {
NONE, formatCost, formatDateTime, formatNumber, formatOffset, formatSpan, formatTokens, shortId,
} from '../../lib/format'
import { runClassLabel, nodeTypeLabel } from '../../lib/terms'
import { COST_TEXT, runClassLabel, nodeTypeLabel, runCostText } from '../../lib/terms'
import { topology, type GraphLike } from '../../run/derive'
import {
isActivePhase, isSettled, liveAt, project, type NodeState, type Projection, type RunPhase, type Trace,
@@ -245,7 +245,9 @@ function Readout({
sub={tokens ? `输入 ${formatNumber(proj.tokensIn)} · 输出 ${formatNumber(proj.tokensOut)}` : undefined}>
{tokens == null ? NONE : formatTokens(tokens, { compact: true })}
</Metric>
<Metric label="成本" data="cost">{!metered || proj.costUsd == null ? NONE : formatCost(proj.costUsd)}</Metric>
<Metric label="成本" data="cost" title={trace.unpricedCalls ? COST_TEXT.unpricedTitle(trace.unpricedCalls) : undefined}>
{!metered || proj.costUsd == null ? NONE : runCostText(proj.costUsd, trace.unpricedCalls) ?? NONE}
</Metric>
</div>
<div className="runs-trace-cols">
+130
View File
@@ -0,0 +1,130 @@
import { useEffect, useId, useState } from 'react'
import { ChevronRight, ShieldCheck } from 'lucide-react'
import { EVIDENCE_STATE, claimCountsOf, claimProblems, statsTally } from '../lib/evidence'
import { formatNumber } from '../lib/format'
import { APPROVAL_TEXT, EVIDENCE_TEXT, JUDGE_TEXT, evidenceTally, runClassLabel } from '../lib/terms'
import { useEvidence } from '../store/evidence'
import type { ApprovalReport } from '../types'
import { EvidenceDoc } from './EvidenceDoc'
type RunClass = 'formal' | 'exploratory' | null | undefined
/**
* 人工审批这次是探索运行还是正式运行,批了会怎样。
*
* 真实运行里审批说明写着「批准后才会正式出具」,那却是一次探索运行:出具只发生在成果节点配了出具契约、从已发布
* 版本发起的正式运行里。审批卡自己标明运行类别,不靠模型写的说明
*/
export function ApprovalRunClass({ runClass, approve }: { runClass: RunClass; approve: boolean }) {
const formal = runClass === 'formal'
return (
<div className="mb-2 flex flex-wrap items-center gap-x-1.5 gap-y-0.5 text-2xs leading-relaxed"
data-approval-run-class={formal ? 'formal' : 'exploratory'}>
<span className="chip shrink-0" style={formal ? { color: 'var(--accent)' } : undefined}>{runClassLabel(runClass)}</span>
<span className="text-dim">{APPROVAL_TEXT.outcome(runClass, approve)}</span>
</div>
)
}
/** 结论句那一句:节点里判过的说判了几句、几句相矛盾;没判过的说尚未核对,并说什么时候能核对 */
function claimsLine(report: ApprovalReport, runClass: RunClass): { text: string; sub?: string; warn: boolean } {
const judge = report.judge
if (judge?.mode === 'inline' && judge.counts && typeof judge.counts === 'object') {
const c = claimCountsOf(judge.counts)
if (!c.total) return { text: APPROVAL_TEXT.claimsNone, warn: false }
const contradicted = c.contradicted + c.unsupported
return {
text: APPROVAL_TEXT.claimsJudged(c.total - c.unjudged, contradicted, c.unjudged),
warn: claimProblems(c) > 0 || c.unjudged > 0,
}
}
return {
text: APPROVAL_TEXT.claimsPending,
sub: runClass === 'formal' ? (report.claims === 'judge' ? undefined : APPROVAL_TEXT.claimsOff)
// 开了裁判却没在节点里判的,只会是这条规则之前的探索运行(现在交给审批的报告探索运行也当场判)
: report.claims === 'judge' ? APPROVAL_TEXT.claimsAfterRun : APPROVAL_TEXT.claimsLater,
warn: true,
}
}
/**
* 审批审的报告的核对概况:几个数字有出处(同证据条的说法),结论句判没判过、几句证据相矛盾,再给一个查看逐段证据
* 的入口——点开是和成果里同一份可逐段点开的报告文档,点数字、句末徽标打开证据面板。
*
* 真实运行里报告有一句结论和证据相矛盾,审批人批准时完全不知道:卡上只有正文
*/
export function ApprovalReports({ reports, runId, runClass }: {
reports: ApprovalReport[]
runId: string
runClass: RunClass
}) {
if (!reports.length) return null
return (
<>
{reports.map((r) => <ReportCheck key={r.node_id} report={r} runId={runId} runClass={runClass} />)}
</>
)
}
function ReportCheck({ report, runId, runClass }: { report: ApprovalReport; runId: string; runClass: RunClass }) {
const [open, setOpen] = useState(false)
const docId = useId()
const counts = statsTally(report.stats)
const suspect = typeof report.stats?.unknown_entities === 'number' ? report.stats.unknown_entities : 0
const numbers = counts ? evidenceTally(counts.cited, counts.total, counts.other) : ''
const claims = claimsLine(report, runClass)
const gaps = !!counts && (counts.none > 0 || counts.other > 0)
return (
<div className="mb-2 rounded border bg-bg p-2" data-approval-report={report.node_id}>
<div className="mb-1 flex items-center gap-1 text-2xs font-medium text-faint">
<ShieldCheck size={11} aria-hidden /> {APPROVAL_TEXT.reportHead(report.label || report.node_id)}
</div>
<div className="flex flex-wrap items-center gap-x-2 gap-y-1 text-2xs">
<span className="tnum" data-approval-report-numbers=""
style={{ color: !counts ? 'var(--text-dim)' : gaps ? 'var(--st-waiting)' : 'var(--st-done)' }}>
{numbers || APPROVAL_TEXT.noNumbers}
</span>
{suspect > 0 && (
<span className="tnum" style={{ color: 'var(--st-waiting)' }}>
· {EVIDENCE_STATE.suspect.glyph} {EVIDENCE_TEXT.suspectTag} {formatNumber(suspect)}
</span>
)}
{/* 判定是模型给的:全都有依据也不用确定性的绿,有问题、没核对的用提醒色 */}
<span className="tnum" data-approval-report-claims="" title={report.judge?.mode === 'inline' ? JUDGE_TEXT.badge(null) : undefined}
style={{ color: claims.warn ? 'var(--st-waiting)' : 'var(--text-dim)' }}>
· {claims.text}
</span>
</div>
{claims.sub && <div className="mt-0.5 text-2xs leading-relaxed text-dim">{claims.sub}</div>}
{report.doc_artifact && (
<>
<button type="button" className="btn btn-xs mt-1.5" data-approval-evidence="" aria-expanded={open}
aria-controls={docId} onClick={() => setOpen((v) => !v)}>
<ChevronRight size={11} aria-hidden style={{ transform: open ? 'rotate(90deg)' : 'none', transition: 'transform .15s' }} />
{open ? APPROVAL_TEXT.closeEvidence : APPROVAL_TEXT.openEvidence}
</button>
{open && (
<div id={docId} className="mt-1.5">
<ReportDoc artifact={report.doc_artifact} runId={runId} runClass={runClass} />
</div>
)}
</>
)}
</div>
)
}
/** 报告文档:和成果里同一份可逐段点开的文档,自带证据条(定位下一处、违规清单),点开的证据面板在卡内展开 */
function ReportDoc({ artifact, runId, runClass }: { artifact: string; runId: string; runClass: RunClass }) {
const slot = useEvidence((s) => s.docs[artifact])
const loadDoc = useEvidence((s) => s.loadDoc)
useEffect(() => { void loadDoc(artifact) }, [artifact, loadDoc])
if (slot?.status === 'error') return <div className="text-2xs text-faint">{APPROVAL_TEXT.docMissing}</div>
const doc = slot?.status === 'ok' ? slot.data : undefined
if (!doc) return <div className="text-2xs text-faint">{APPROVAL_TEXT.docLoading}</div>
return (
<div className="max-h-96 overflow-y-auto rounded border p-2" data-approval-evidence-doc="">
<EvidenceDoc doc={doc} artifact={artifact} runId={runId} dense tally runClass={runClass ?? undefined} />
</div>
)
}
+10 -2
View File
@@ -15,7 +15,9 @@ import { humanizeError } from '../lib/errors'
import { explainRunError } from '../lib/explain'
import { formatClock, formatDuration, formatNumber, NONE, shortId } from '../lib/format'
import { statusLabel } from '../lib/status'
import { EVIDENCE_TEXT, JUDGE_TEXT, MERGE_TEXT, RUN_SQL_CHECK_TEXT, evidenceTally, issuanceLabel, nodeTypeLabel } from '../lib/terms'
import {
COPILOT_CONTEXT_TEXT, EVIDENCE_TEXT, JUDGE_TEXT, MERGE_TEXT, RUN_SQL_CHECK_TEXT, evidenceTally, issuanceLabel, nodeTypeLabel,
} from '../lib/terms'
import { SqlCheckList } from './SqlChecks'
import {
ISSUANCE_HINT, childrenByExec, compactSteps, parseQueryResult, progressOf, spread, teamVerdictOf,
@@ -901,6 +903,12 @@ function FixAction({ fix, nodeId, label, to, first, field }: {
if (fix === 'settings') {
return <Link to={to ?? '/settings/providers'} className={cls} data-fix="settings"><Settings2 size={10} aria-hidden /> 去模型接入</Link>
}
if (fix === 'catalog' && to) {
// 助手参考的大库还没有数据目录:直达那个数据源的目录页去起草
return (
<Link to={to} className={cls} data-fix="catalog"><Database size={10} aria-hidden /> {COPILOT_CONTEXT_TEXT.toCatalog}</Link>
)
}
if (fix === 'tools') {
// 数据源工具(db_query__ 这类)不在工具库里接入,在数据页
const data = !!to?.startsWith('/data')
@@ -1170,7 +1178,7 @@ function StepLine({ step, depth, turnMs }: { step: Step; depth: number; turnMs?:
</span>
{/* 要改的是另一个节点(指标的问题出在来源查询的 SQL 上):入口落到那个节点的那一栏 */}
<FixAction fix={step.fix} nodeId={step.fixNode ?? step.nodeId} field={step.fixField}
label={step.fixNode ? step.fixLabel : undefined} />
label={step.fixNode ? step.fixLabel : undefined} to={step.fixTo} />
</div>
)}
+9 -3
View File
@@ -18,7 +18,7 @@ import {
} from '../lib/evidence'
import { humanizeError } from '../lib/errors'
import { formatDateTime, formatNumber, NONE, shortId } from '../lib/format'
import { EVIDENCE_TEXT, JUDGE_TEXT, MERGE_TEXT, NODE_TYPE_LABEL, RUN_SQL_CHECK_TEXT, SQL_CHECK_TEXT, nodeTypeLabel } from '../lib/terms'
import { ABSENCE_CLAIM_TEXT, EVIDENCE_TEXT, JUDGE_TEXT, MERGE_TEXT, NODE_TYPE_LABEL, RUN_SQL_CHECK_TEXT, SQL_CHECK_TEXT, nodeTypeLabel } from '../lib/terms'
import { useCatalog } from '../store/catalog'
import { askKey, segmentKey, useEvidence, useExplore, useVerdicts } from '../store/evidence'
import type {
@@ -1552,6 +1552,7 @@ const CODE_LABEL: Record<string, string> = {
unresolved_ref: '引用无法解析',
unknown_entity: '疑似不存在的名称',
unverified_entity: '无法核实的名称',
absence_claim: ABSENCE_CLAIM_TEXT.label,
}
/** 违规清单:报告撰写节点核对出来的全部问题,画得出线的能定位,画不出的说清在哪 */
@@ -1563,7 +1564,9 @@ function ViolationList({ doc, onLocate }: { doc: EvidenceDocData; onLocate: (seg
<p className="text-2xs text-faint">{EVIDENCE_TEXT.violationsHint}</p>
<ul className="space-y-2" data-ev-violations="">
{list.map((v, i) => {
const seg = locatable(doc, v)
// 缺失断言说的是一整句话,正文里普通文字不是可聚焦的片段:不给「定位」,原句写在下面
const sentence = v.code === 'absence_claim'
const seg = sentence ? null : locatable(doc, v)
return (
<li key={i} className="rounded border px-2 py-1.5" data-ev-violation={v.code}
data-ev-locatable={seg ? 'yes' : 'no'}>
@@ -1576,8 +1579,11 @@ function ViolationList({ doc, onLocate }: { doc: EvidenceDocData; onLocate: (seg
)}
</div>
{v.message && <p className="mt-1 text-dim">{v.message}</p>}
{v.code === 'absence_claim' && (
<p className="mt-0.5 text-2xs text-dim" data-ev-absence-hint="">{ABSENCE_CLAIM_TEXT.hint}</p>
)}
{v.context && <p className="mono mt-0.5 text-2xs text-faint [overflow-wrap:anywhere]">「{v.context.trim()}」</p>}
{!seg && (
{!seg && !sentence && (
<p className="mt-0.5 text-2xs" style={{ color: 'var(--st-waiting)' }}>
{v.segment ? EVIDENCE_TEXT.structural : EVIDENCE_TEXT.noSegment}
</p>
+9 -4
View File
@@ -8,7 +8,7 @@ import { confirmDialog, Kbd, Spinner, StatusBadge, toast } from '../components/u
import { formatCost, formatDuration, formatLapse, formatOffset, formatSpan, formatTokens, NONE } from '../lib/format'
import { isTypingTarget, matchShortcut } from '../lib/keys'
import { statusMeta, STATUS, type StatusCode } from '../lib/status'
import { issuanceLabel, runClassLabel } from '../lib/terms'
import { COST_TEXT, issuanceLabel, runClassLabel, runCostText } from '../lib/terms'
import { lastStampOf, projectSettled, useDock, waitedMs } from '../canvas/RunTimeline'
import { structureSig, useStudio } from '../store/studio'
import type { RunEvent } from '../types'
@@ -91,6 +91,8 @@ export interface RunGlance {
series: [number, number][]
tokens: number | null
cost: number | null
/** 金额估不出来的调用次数(模型不在价格目录中);有的话金额不写 $0 */
unpriced: number
/** 这一段执行还剩多少时间;null = 不知道上限或眼下不在计时 */
remainingMs: number | null
limitMs: number | null
@@ -220,6 +222,7 @@ export function useRunGlance(active: boolean): RunGlance {
const replayTok = proj.tokensIn != null && proj.tokensOut != null ? proj.tokensIn + proj.tokensOut : null
const tokens = replayAt != null ? replayTok : backendTok ?? (seen ? usage.tokensIn + usage.tokensOut : null)
const cost = replayAt != null ? proj.costUsd ?? null : backendCost ?? (seen ? usage.costUsd : null)
const unpriced = (settledUsage ? num(runUsage.unpriced_calls) : null) ?? trace.unpricedCalls
// 时限按"这一段执行"计:等人工审批时执行已经结束,那段不算(和后端一致)
const drive = trace.drives[trace.drives.length - 1]
@@ -253,7 +256,7 @@ export function useRunGlance(active: boolean): RunGlance {
headline: headlineOf(trace, phase, states, at, label, runError, replayAt == null ? cancelled : ''),
executing: phase === 'running' && cells.some((c) => c.state === 'running' && !trace.nodes[c.id]?.looping),
cells, done, visited, parallelNow: proj.parallelNow, series: trace.parallelSeries,
tokens, cost, remainingMs, limitMs, attention,
tokens, cost, unpriced, remainingMs, limitMs, attention,
failedNodeId: trace.failedNodeId, waitingNodeId: trace.waitingNodeId,
}
}
@@ -591,7 +594,7 @@ export function RunCapsule() {
<span className="sf-cap-nodes tnum hidden lg:inline" title="完成的节点 / 全部节点">{nodesText}</span>
<span className="sf-cap-sep hidden xl:block" />
<span className="sf-cap-usage tnum hidden xl:inline">
{tokens}{g.cost ? ` · ${formatCost(g.cost)}` : ''}
{tokens}{g.cost || g.unpriced ? ` · ${runCostText(g.cost, g.unpriced)}` : ''}
</span>
</>
)}
@@ -654,7 +657,9 @@ export function RunCapsule() {
<div className="sf-metric">
<div className="sf-metric-k">用量</div>
<div className="sf-metric-v tnum">{tokens}</div>
<div className="sf-metric-s tnum">{g.cost != null ? `≈ ${formatCost(g.cost)}` : NONE}</div>
<div className="sf-metric-s tnum" title={g.unpriced ? COST_TEXT.unpricedTitle(g.unpriced) : undefined}>
{g.unpriced ? runCostText(g.cost, g.unpriced) : g.cost != null ? `≈ ${formatCost(g.cost)}` : NONE}
</div>
</div>
<div className="sf-metric">
<div className="sf-metric-k">时限</div>
+14 -2
View File
@@ -11,7 +11,8 @@ import { formatDateTime, formatSpan, formatTime, parseServerTime } from '../lib/
import { runClassLabel, runName, TOOL_TRUST_TEXT } from '../lib/terms'
import { useRunClock } from './useRunClock'
import { Markdown } from './Markdown'
import type { Approval } from '../types'
import { ApprovalReports, ApprovalRunClass } from './ApprovalReport'
import type { Approval, ApprovalReport } from '../types'
// 发起运行的表单搬去了 run/RunControl.tsx 的工具栏控件里。它原先常驻在
// 助手栏顶部,和 Copilot 输入框两个"主要动作"互相压着——而它们是两种不同的
@@ -99,6 +100,12 @@ export function ApprovalCard({ approval, onResolved, showWorkflow = true }: {
}
const where = approval.node_label || approval.node_id
// 人工审批节点挂起的审批(工具、代码的审批另有 kind):卡上标明运行类别、批了会怎样,审的是报告时附上核对概况
const humanNode = payload.kind === 'human_node'
const reports: ApprovalReport[] = Array.isArray(payload.reports)
? payload.reports.filter((r: unknown): r is ApprovalReport =>
!!r && typeof r === 'object' && typeof (r as ApprovalReport).node_id === 'string')
: []
return (
<div className="fade-up border-b p-3 last:border-b-0" data-approval={approval.id}
style={{ background: 'color-mix(in srgb, var(--warn) 7%, transparent)' }}>
@@ -109,7 +116,7 @@ export function ApprovalCard({ approval, onResolved, showWorkflow = true }: {
<span>人工审批</span>
{showWorkflow && approval.workflow_name && <><span aria-hidden>·</span><span>{runName(approval)}</span></>}
{where && <><span aria-hidden>·</span><span>节点「{where}」</span></>}
{approval.run_class && <><span aria-hidden>·</span><span>{runClassLabel(approval.run_class)}</span></>}
{approval.run_class && !humanNode && <><span aria-hidden>·</span><span>{runClassLabel(approval.run_class)}</span></>}
{created && (
<>
<span aria-hidden>·</span>
@@ -126,6 +133,9 @@ export function ApprovalCard({ approval, onResolved, showWorkflow = true }: {
)}
</div>
{/* 老后端的审批没有运行类别:不猜,不画这一行 */}
{humanNode && approval.run_class && <ApprovalRunClass runClass={approval.run_class} approve={approveMode} />}
{payload.message && (
// 待审的往往是一整段模型写的文案,同样是 Markdown。要人家判断"能不能
// 发",却让他对着一堆星号看,是在给审批这件事添难度
@@ -134,6 +144,8 @@ export function ApprovalCard({ approval, onResolved, showWorkflow = true }: {
</div>
)}
<ApprovalReports reports={reports} runId={approval.run_id} runClass={approval.run_class} />
{payload.tool && (
<div className="mb-2 rounded border bg-bg p-2">
<div className="mb-1 flex items-center gap-1.5 text-2xs">
+7 -3
View File
@@ -503,8 +503,10 @@ export const COPILOT_CONTEXT = [
{ op: 'model', model: 'claude-sonnet-4' },
{ op: 'heartbeat', phase: 'planning', elapsed_ms: 3000 },
{ op: 'context', elapsed_ms: 3420, sources: [
{ source: 'scenic', tables: ['visits', 'channels', 'channel_visits'], selected_by: 'model', total: 51 },
{ source: 'shop', tables: ['orders', 'customers'], selected_by: 'all', total: 2 },
{ source: 'scenic', source_id: 'src-scenic', tables: ['visits', 'channels', 'channel_visits'], selected_by: 'model', total: 51,
catalog_tables: 12 },
// 小库全部表都带字段:没有数据目录也不提示
{ source: 'shop', source_id: 'src-shop', tables: ['orders', 'customers'], selected_by: 'all', total: 2, catalog_tables: 0 },
] },
{ op: 'heartbeat', phase: 'planning', elapsed_ms: 6000 },
{ op: 'thinking', delta: '入园记录不带渠道,要经 channel_visits 连到渠道表,再按渠道汇总上月的入园人次。' },
@@ -539,7 +541,9 @@ export const COPILOT_CONTEXT_FALLBACK = [
{ op: 'model', model: 'claude-sonnet-4' },
{ op: 'heartbeat', phase: 'planning', elapsed_ms: 18000 },
{ op: 'context', elapsed_ms: 20004, sources: [
{ source: 'scenic', tables: [], selected_by: 'fallback', total: 51, reason: '挑选数据表超过 20 秒未完成' },
// 大库一张目录都没有:过程里提示先去起草目录
{ source: 'scenic', source_id: 'src-scenic', tables: [], selected_by: 'fallback', total: 51, catalog_tables: 0,
reason: '挑选数据表超过 20 秒未完成' },
] },
{ op: 'plan', summary: '先查表结构,再取数' },
{ op: 'add_node', node: { id: 's', type: 'tool', label: '查看表结构' } },
+87 -4
View File
@@ -2,7 +2,7 @@ import type { CatalogDriftTable, RunEvent, SqlCheckItem, TeamMember, TeamRound,
import type { NodeState, RunPhase } from './trace'
import { formatDuration, formatNumber } from '../lib/format'
import {
CATALOG_DRIFT_TEXT, CATALOG_PATCH_TEXT, COPILOT_CONTEXT_TEXT, JUDGE_TEXT, MERGE_TEXT, RUN_SQL_CHECK_TEXT, SCHEMA_PARTIAL_TEXT, SQL_CHECK_TEXT, TYPE_LABEL,
ABSENCE_CLAIM_TEXT, CATALOG_DRIFT_TEXT, CATALOG_PATCH_TEXT, COPILOT_CONTEXT_TEXT, JUDGE_TEXT, MERGE_TEXT, RUN_SQL_CHECK_TEXT, SCHEMA_PARTIAL_TEXT, SQL_CHECK_TEXT, TYPE_LABEL,
claimTally,
evidenceTally, issuanceLabel, nodeTypeLabel,
} from '../lib/terms'
@@ -48,7 +48,8 @@ export const teamVerdictOf = (team: TeamRun | undefined | null): TeamVerdict | u
(team as TeamRunEx | undefined | null)?.verdict
/** 下一步该去哪儿改。界面据此给一个直达的入口,而不只是一句话 */
export type FixKind = 'canvas' | 'settings' | 'tools'
/** catalog:去数据目录页(fixTo 指到哪个数据源),助手参考的大库还没有数据目录时给 */
export type FixKind = 'canvas' | 'settings' | 'tools' | 'catalog'
/**
* 事件 → 人能看懂的步骤。
@@ -147,6 +148,8 @@ export interface Step {
next?: string
/** 下一步该去的地方 */
fix?: FixKind
/** fix 要去的页面地址(catalog:/data/catalog/:sourceId) */
fixTo?: string
/** 工具的时限(秒,tool.start.timeout_s)。进行中的行超过它就要说破 */
limitS?: number
/** 出具那一行的档位(formal / degraded / withheld) */
@@ -756,6 +759,47 @@ function explainLog(code: string | undefined, message: string, d: Record<string,
sub: message.match(/(?:没|未)通过核对((.+?))/)?.[1],
}
}
case 'report_absence_claim': {
// 证据目录有省略时报告写了缺失断言,重写后仍在(report.py):读结构化字段 count、texts,不从原话里认
const n = typeof d.count === 'number' && d.count > 0 ? d.count : undefined
const texts = strs(d.texts)
return {
title: ABSENCE_CLAIM_TEXT.title(n),
sub: texts.length ? texts.map((t) => `「${t}」`).join('、') : undefined,
next: ABSENCE_CLAIM_TEXT.next,
}
}
case 'report_catalog_trimmed': {
// 写作目录超出篇幅(report.py _trimmed_message)。读结构化字段(queries、omitted、nodes),不从原话里认
const total = typeof d.queries === 'number' ? d.queries : undefined
const omitted = typeof d.omitted === 'number' ? d.omitted : undefined
const compact = typeof d.compact === 'number' ? d.compact : undefined
const rows: any[] = Array.isArray(d.nodes) ? d.nodes : []
const per = rows.filter((r) => typeof r?.omitted === 'number' && r.omitted > 0 && str(r?.title))
.map((r) => `${r.title} 省略 ${formatNumber(r.omitted)}/${formatNumber(r.total)} 次`)
const all = total != null ? `${formatNumber(total)} 次查询都列出了数据` : '查询都列出了数据'
if (omitted === 0) {
// 放不下但一次查询都没省略:有的只列了前几行,或者只省略了表和字段的说明。不写「0 次只列了编号和摘要」
const rowsPer = rows.filter((r) => typeof r?.compact === 'number' && r.compact > 0 && str(r?.title))
.map((r) => `${r.title} ${formatNumber(r.compact)}/${formatNumber(r.total)} 次只列前几行`)
return {
title: compact ? `证据目录超出篇幅,${all},${formatNumber(compact)} 次只列了前几行`
: compact === 0 ? `证据目录超出篇幅,${all},表和字段的说明有省略`
: `证据目录超出篇幅,${all},部分明细有省略`,
sub: rowsPer.length ? rowsPer.join('、') : undefined,
next: '写作模型看到了每次查询,但有的只看到前几行;其余的行仍在证据台账里,报告不得据此断言没有数据。'
+ '需要完整明细时,请换用上下文窗口更大的写作模型,或让 Agent 在 SQL 中先汇总再交出结果',
}
}
return {
title: omitted != null
? `证据目录超出篇幅,${total != null ? `${formatNumber(total)} 次查询中 ` : ''}${formatNumber(omitted)} 次只列了编号和摘要`
: '证据目录超出篇幅,部分查询只列了编号和摘要',
sub: per.length ? per.join('、') : undefined,
next: '写作模型按来源均衡地看到了一部分查询的明细,被省略的查询仍在证据台账里,报告不得据此断言没有数据。'
+ '需要完整明细时,请换用上下文窗口更大的写作模型,或让 Agent 在 SQL 中先汇总再交出结果',
}
}
case 'agent_field_mismatch': {
// 「有 1 个字段和查询快照对不上,已按快照取值:order_cnt 模型报 1240,快照是 1234」(llm.py _field_warnings)
const n = message.match(/有\s*(\d+)\s*个字段/)?.[1]
@@ -784,6 +828,22 @@ function explainLog(code: string | undefined, message: string, d: Record<string,
next: '模型给出的出处在查询结果中不存在。请让模型先查询到这些字段再提交,查询不到的字段记为空值',
}
}
case 'agent_field_truncated': {
// 结构化抽取两次都被截断(llm.py TRUNCATED_EXTRACT):「结构化抽取没有完成:模型的输出额度在推理阶段就用完了
// (最后一次的额度为 32768 token),结构化结论没能写出来。5 个字段都记为空值(未以 0 代替)。可以……」。
// 字段数、额度读结构化字段(fields、max_tokens),不从原话里认;不说「稍后重试」——重试解决不了额度问题
const n = strs(d.fields).length || Number(message.match(/(\d+)\s*个字段都?记为空值/)?.[1] ?? 0)
const budget = typeof d.max_tokens === 'number' && d.max_tokens > 0 ? d.max_tokens : undefined
return {
title: `模型的输出额度在推理阶段用完,${n ? `${n} 个` : ''}字段记为空值(未以 0 代替)`,
sub: `按出处抽取字段时输出被截断${budget ? `(已加大额度重试,最后一次为 ${formatNumber(budget)} token)` : ''},`
+ '结构化结论没能写出来',
next: '这些字段以空值传给下游,口径卡按缺少输入处理,报告不得据此断言没有数据。请为该 Agent 换用非推理模型'
+ '(抽取沿用 Agent 的模型),或调大它的「最大输出 token」后重新运行',
fix: 'canvas',
fixField: 'max_tokens',
}
}
case 'metric_incomplete': {
// 结构化字段(metrics.py):metric_name、reason(「基于被截断的查询结果计算(只取回了前 1000 行),结果不完整」)。
// 老运行的日志没有这两个字段:只给不点名的标题,原话留在展开区,不从中文里解析
@@ -2649,6 +2709,10 @@ export interface CopilotContextSource {
explored?: number
/** 没能挑出来的原因 */
reason?: string
/** 数据源 id:去数据目录页的入口用(老服务端不给) */
sourceId?: string
/** 有数据目录的表数(老服务端不给:不知道就不提示) */
catalogTables?: number
}
/** 挑表失败时的说法:模型这一轮只看到表名,字段要它自己去查 */
@@ -2668,10 +2732,24 @@ function contextSources(op: CopilotOp): CopilotContextSource[] {
...(total != null ? { total } : {}),
...(total != null && explored != null && explored < total ? { explored } : {}),
...(typeof x.reason === 'string' && x.reason ? { reason: x.reason } : {}),
...(typeof x.source_id === 'string' && x.source_id ? { sourceId: x.source_id } : {}),
...(num(x.catalog_tables) != null ? { catalogTables: num(x.catalog_tables) } : {}),
}
})
}
/**
* 没有数据目录的大库(要按需求挑表的那种):挑表只能照表名猜,SQL 检查也没有目录可对照。真实运行里数据源一张目录
* 都没有,用户也不知道该先去起草。只认服务端明说了 0 张的;小库全部表都带字段,不提示
*/
function noCatalog(groups: CopilotContextSource[]): { next: string; fixTo?: string } | null {
const bare = groups.filter((g) => g.catalogTables === 0 && g.selectedBy !== 'all')
if (!bare.length) return null
const who = groups.length === 1 ? null : `「${bare.map((g) => g.source).join('」「')}」`
const to = bare.find((g) => g.sourceId)?.sourceId
return { next: COPILOT_CONTEXT_TEXT.noCatalog(who), ...(to ? { fixTo: `/data/catalog/${encodeURIComponent(to)}` } : {}) }
}
/**
* 一个库在展开区里的那一行:怎么来的 + 表名。探查结构截断了的库写明「共 205 张,只探查了 200 张」:
* 模型只看得到探查到的那些,用户说的表不在里面时,先得知道是没探查到,不是库里没有
@@ -2702,7 +2780,8 @@ function contextLine(g: CopilotContextSource): string {
* 先得知道模型当时看到的是哪几张。标题只说几张(「参考了 8 张表」),展开按数据源分组列表名;挑表失败时
* 标题直说只给了表名,原因放在展开区。认不出的形状返回 null:不出这一行,也不报错。
*/
export function copilotContext(op: CopilotOp): { title: string; sub?: string; detail: string; ms?: number } | null {
export function copilotContext(op: CopilotOp):
{ title: string; sub?: string; detail: string; ms?: number; next?: string; fixTo?: string } | null {
// 展开区里按需求挑过表的库排在前面:请求里点名的那个库往往就是它,排在一串小库后面得在内层滚动框里翻才看得到
const order: Record<CopilotContextSource['selectedBy'], number> = { model: 0, all: 1, fallback: 2 }
const groups = contextSources(op).map((g, i) => ({ g, i }))
@@ -2721,11 +2800,13 @@ export function copilotContext(op: CopilotOp): { title: string; sub?: string; de
const reasons = [...new Set(failed.map((g) => g.reason).filter(Boolean))]
// 挑过表才有 elapsed_ms:挑表是生成之前多出来的一次模型调用,没挑成(比如超时)也要让人看到等了多久
const ms = num(op.elapsed_ms)
const hint = noCatalog(groups)
return {
title: count ? (namesOnly ? COPILOT_CONTEXT_TEXT.withFields(count) : `参考了 ${count} 张表`) : CONTEXT_FALLBACK_TEXT,
...(sub ? { sub } : {}),
detail: [...groups.map(contextLine), ...reasons.map((r) => `原因:${r}`)].join('\n'),
...(ms != null && ms >= 10 ? { ms } : {}),
...(hint ?? {}),
}
}
@@ -2848,7 +2929,9 @@ export function decodeCopilot(ops: CopilotOp[], opts?: {
add({ id: `cx-${i}`, seq: i, kind: 'schema', status: 'done', code: 'copilot_context',
title: ctx.title, detail: ctx.detail,
...(ctx.sub ? { sub: ctx.sub } : {}),
...(ctx.ms != null ? { ms: ctx.ms, meta: formatDuration(ctx.ms) } : {}) })
...(ctx.ms != null ? { ms: ctx.ms, meta: formatDuration(ctx.ms) } : {}),
// 没有数据目录的大库:下一步去起草目录,入口直达那个数据源的目录页
...(ctx.next ? { next: ctx.next, ...(ctx.fixTo ? { fix: 'catalog' as const, fixTo: ctx.fixTo } : {}) } : {}) })
break
}
case 'catalog_patch': {
+5 -1
View File
@@ -96,6 +96,8 @@ export interface Trace {
tokensIn: number
tokensOut: number
costUsd: number
/** 金额估不出来的调用次数(llm.end 标了 unpriced:模型不在价格目录中)。这时 costUsd 的 0 不是「没花钱」 */
unpricedCalls: number
failedNodeId?: string
waitingNodeId?: string
issuance?: {
@@ -238,7 +240,7 @@ export const lastEventAt = (t: Trace): number => t.book.clock
export function emptyTrace(): Trace {
return {
phase: 'idle', waitMs: 0, nodes: {}, tokensIn: 0, tokensOut: 0, costUsd: 0,
phase: 'idle', waitMs: 0, nodes: {}, tokensIn: 0, tokensOut: 0, costUsd: 0, unpricedCalls: 0,
lastSeq: 0, parallelSeries: [], usageSeries: [], timed: true, drives: [], waits: [], phases: [],
lastReplay: false,
book: { clock: 0, awaiting: false, dispatchFrom: {}, exactDispatch: {}, asking: {} },
@@ -673,6 +675,7 @@ export function foldEvent(t: Trace, ev: RunEvent): Trace {
next.tokensIn += inTok
next.tokensOut += outTok
next.costUsd += cost
if (d.unpriced === true) next.unpricedCalls += 1
next.usageSeries = [...next.usageSeries, [at, next.tokensIn, next.tokensOut, next.costUsd]]
if (nodeId) {
const n = node(nodeId)
@@ -816,6 +819,7 @@ export function foldEvent(t: Trace, ev: RunEvent): Trace {
next.tokensIn = num(u.input_tokens) ?? next.tokensIn
next.tokensOut = num(u.output_tokens) ?? next.tokensOut
next.costUsd = num(u.cost_usd) ?? next.costUsd
next.unpricedCalls = num(u.unpriced_calls) ?? next.unpricedCalls
}
const timing = d.timing
if (timing && num(timing.wall_ms) != null) {
+21 -3
View File
@@ -110,6 +110,8 @@ export interface RunUsage {
input_tokens?: number
output_tokens?: number
cost_usd?: number
/** 金额估不出来的调用次数(模型不在价格目录中),cost_usd 只是估得出的那部分。没有这种调用时不给 */
unpriced_calls?: number
[key: string]: any
}
@@ -173,6 +175,22 @@ export interface Approval {
resolved_at?: string | null
}
/**
* 人工审批审的报告的核对概况(human 节点请求里的 payload.reports,服务端 nodes/human._reviewed_reports):
* 上游直接连着的、或者标题和说明里引用了正文的报告撰写节点,各一项。stats 同 report.checked,judge 是裁判摘要
* 的几项(mode:inline 节点里当场判的——交给审批的报告探索运行也当场判 / on_demand 按需判,老服务端的探索运行)。
* 老后端、审的不是报告时没有
*/
export interface ApprovalReport {
node_id: string
label?: string
stats?: EvidenceStats
/** 结论句策略:off / require_citation / judge */
claims?: string
doc_artifact?: string
judge?: { mode?: string; counts?: Record<string, unknown>; complete?: boolean; unjudged?: unknown; limits_hit?: unknown }
}
/**
* 后端记着的最近一次测连接(模型接入、数据源、MCP 的列表接口都平铺这四项)。
* 没测过、或者连接配置改过之后全是 null。用 lib/health 的 healthFromServer 转成
@@ -1782,8 +1800,8 @@ export interface EvidenceUnit {
/** 列表项的缩进层级 */
depth?: number
/**
* 结论句裁判的判定(四期):只有送了裁判的候选句有。正式运行里节点当场判的进封存(post_seal false);
* 探索运行里候选句先标 unjudged + reason on_demand,点开再判
* 结论句裁判的判定(四期):只有送了裁判的候选句有。正式运行(以及交给人工审批的探索运行)里节点当场判的
* 进封存(post_seal false);其余探索运行里候选句先标 unjudged + reason on_demand,点开再判
*/
verdict?: EvidenceVerdict
}
@@ -1820,7 +1838,7 @@ export interface EvidenceVerdict {
/** 文档(和 report.checked、节点产出)里的裁判摘要:键的顺序同后端 judge.SUMMARY_KEYS */
export interface EvidenceJudgeSummary {
/** inline:正式运行节点里当场判;on_demand:探索运行按需 */
/** inline:节点里当场判(正式运行,以及交给人工审批的探索运行);on_demand:其余探索运行按需 */
mode?: 'inline' | 'on_demand' | string
model?: string | null
on_unsupported?: 'degrade' | 'withhold' | string
+10 -1
View File
@@ -487,8 +487,10 @@ async function fakeApi(route) {
const ops = [
{ op: 'heartbeat', phase: 'planning', elapsed_ms: 800 },
// 服务端按需求挑了表(copilot_context):过程里多一行「参考了 N 张表」
// 这个库还没有数据目录(catalog_tables: 0):那一行下面提示先去起草目录
{ op: 'context', elapsed_ms: 1200, sources: [
{ source: 'shop', tables: ['orders', 'order_items'], selected_by: 'model', total: 48 }] },
{ source: 'shop', source_id: 'src-shop', tables: ['orders', 'order_items'], selected_by: 'model', total: 48,
catalog_tables: 0 }] },
{ op: 'plan', summary: '查一下再回答' },
...GRAPH.nodes.map((n) => ({ op: 'add_node', node: n })),
{ op: 'done', explanation: '一张三步的图' },
@@ -1069,6 +1071,13 @@ for (const theme of THEMES) {
await ctxRow.locator('button').first().click()
check('再点开看到按数据源分组的表名', await ctxRow.getByText('「shop」按需求从 48 张表中挑出 2 张:orders、order_items')
.waitFor({ timeout: 3000 }).then(() => true, () => false))
// 真实运行里数据源一张目录都没有,挑表和 SQL 检查都用不上目录:这一行下面提示起草,并给去目录页的入口
const noCatalog = ctxRow.locator('[data-step-next]')
check('没有数据目录的大库:提示先起草目录', await noCatalog.count() === 1
&& (await noCatalog.innerText()).includes('这个数据源还没有数据目录,起草后挑表和 SQL 检查会更准'),
await noCatalog.count() ? await noCatalog.innerText() : '没有这一句')
check('……入口直达这个数据源的目录页', await ctxRow.locator('a[data-fix="catalog"]').getAttribute('href').catch(() => null)
=== '/data/catalog/src-shop')
check('没有运行时报错', errors.length === 0, errors[0] ?? '')
await ctx.close()
})
+3 -2
View File
@@ -230,12 +230,13 @@ const MODEL_ONLY = {
'NODE_REFERENCE', 'ROLE', 'MARKUP_NUDGE', 'JUDGE_RULES', 'JUDGE_ROLE', 'JUDGE_RULE', 'REVIEW_SYSTEM', 'MARKER_RULES',
'GATE_SYSTEM', 'TOOL_MARKUP_NUDGE', '_UNKNOWN_ENTITY_FOR_MODEL', 'NODE_FIELD_FIX', '_FIX_TAIL',
'_STREAM_PROTOCOL', '_RULES_HEAD', '_NO_LOWERING', '_ASK_HUMAN', '_UPGRADE_TASK', 'CELL_RULES', 'ENTITY_RULES',
'EXTRACT_SYSTEM', 'GRAPH_SCHEMA', 'REVIEW_SCHEMA', '_CN_N',
'EXTRACT_SYSTEM', 'GRAPH_SCHEMA', 'REVIEW_SCHEMA', '_CN_N', 'OMITTED_RULES', 'ABSENCE_RULE',
]),
// 拼提示词的函数:生成 / 修正 / 升级请求、抽取消息、交给抽取模型的输出结构;
// 助手的数据源上下文(api/copilot_context.py 的各段渲染、挑表请求,copilot.py 里挑表用的需求)
funcs: new Set(['_user_message', '_repair_request', '_publish_fix_request', '_upgrade_assist_request',
'_extract_messages', '_from_rows', 'catalog_prompt',
'_extract_messages', '_from_rows', 'catalog_prompt', 'writer_catalog', '_trimmed_catalog', '_metric_prompt',
'_fields_prompt', '_field_prompt', '_query_brief',
'_pick_messages', '_pick_need', '_publish_fix_need', '_upgrade_need',
'_detail_block', '_selected_block', '_join_lines', '_route_line', '_source_head']),
// 第二个参数是交回模型改写的指令(第一个是给人看的那句):engine/evidence.py 的 _Reason(text, fix)、fail(text, fix)
+116
View File
@@ -682,6 +682,28 @@ await section('Copilot:这一轮参考了哪些表(context 操作)', async
check('挑表失败:原因和库的规模放在展开区', !!fb?.detail?.includes('原因:挑选数据表超过 20 秒未完成')
&& fb.detail.includes('「scenic」共 51 张表'), fb?.detail)
check('挑表失败也说等了多久', fb?.ms === 20004, String(fb?.ms))
// 真实运行里数据源一张目录都没有,挑表、SQL 检查都用不上目录,用户也不知道该先去起草
check('没有数据目录的大库:下一步提示起草目录,入口直达那个源的目录页',
fb?.next === '这个数据源还没有数据目录,起草后挑表和 SQL 检查会更准' && fb?.fix === 'catalog'
&& fb?.fixTo === '/data/catalog/src-scenic', JSON.stringify({ next: fb?.next, fix: fb?.fix, fixTo: fb?.fixTo }))
check('有目录的大库、没有目录的小库:不提示', !row?.next && !row?.fix && !row?.fixTo, JSON.stringify({ next: row?.next }))
const bare = mod.copilotContext({ op: 'context', sources: [
{ source: 'a', source_id: 'id-a', tables: ['x'], selected_by: 'model', total: 60, catalog_tables: 0 },
{ source: 'b', source_id: 'id-b', tables: ['y'], selected_by: 'all', total: 3, catalog_tables: 0 },
{ source: 'c', source_id: 'id c', tables: [], selected_by: 'fallback', total: 90, catalog_tables: 0, reason: 'r' }] })
check('几个数据源:点名没有目录的大库,入口去第一个', bare?.next === '「a」「c」还没有数据目录,起草后挑表和 SQL 检查会更准'
&& bare?.fixTo === '/data/catalog/id-a', JSON.stringify({ next: bare?.next, fixTo: bare?.fixTo }))
const partly = mod.copilotContext({ op: 'context', sources: [
{ source: 'a', source_id: 'id-a', tables: ['x'], selected_by: 'model', total: 60, catalog_tables: 3 }] })
check('起草过一部分目录:不再提示', !partly?.next, partly?.next)
const old = mod.copilotContext({ op: 'context', sources: [{ source: 'a', tables: ['x'], selected_by: 'model', total: 60 }] })
check('老服务端不给目录表数:不猜,不提示', !old?.next && !old?.fixTo, old?.next)
const noId = mod.copilotContext({ op: 'context', sources: [
{ source: 'a', tables: [], selected_by: 'fallback', total: 60, catalog_tables: 0 }] })
check('没给数据源 id:照样提示,只是不给入口', !!noId?.next && !noId?.fixTo, JSON.stringify(noId))
const step = mod.decodeCopilot([{ op: 'context', sources: [
{ source: 'a', tables: [], selected_by: 'fallback', total: 60, catalog_tables: 0 }] }]).find((s) => s.code === 'copilot_context')
check('没有入口时不带 fix:不画一个点了没处去的按钮', !!step?.next && !step?.fix, JSON.stringify(step))
const seeded = mod.copilotContext({ op: 'context', sources: [
{ source: 'scenic', tables: ['stores'], selected_by: 'fallback', total: 51, reason: '模型未挑出任何表' }] })
@@ -1300,6 +1322,59 @@ await section('报告核对 report.checked、口径卡的台账(可点击证
`${flagged?.status} ${flagged?.level}`)
const bare = flatten(mod.decodeRun([E(1, 'report.checked', null, {})])).find((s) => s.code === 'report_checked')
check('载荷缺字段也不崩,只说核对了报告', bare?.title === '核对报告' && !bare.artifact, bare?.title)
// 写作目录超出篇幅(2026-10 一次运行:上百次查询只有前二十来次进了写作者的视野,界面上一个字都没说):
// 现在按来源均衡地展开,没展开的列编号和摘要,并发一条警告。界面读结构化字段(queries、omitted、nodes)
check('后端发目录有省略的警告,带着结构化字段', reportPy.includes('code="report_catalog_trimmed"')
&& reportPy.includes('queries=writer.queries, omitted=len(writer.omitted)') && reportPy.includes('"node_id", "title", "total", "omitted"'),
'改了 report.py 的字段就同步改 decode.ts 的 explainLog 和这里的夹具')
const TRIMMED = '证据目录超出篇幅(约 51200 字,按写作模型的上下文窗口确定):164 次查询中 90 次列出了数据,74 次只列编号和摘要'
+ '(卡券核销省略 52 次、活动内容省略 22 次)。写作模型看不到这些查询的明细,报告不得据此断言没有数据'
const trimmed = flatten(mod.decodeRun([E(1, 'node.started', 'w', { node_type: 'report', label: '写报告' }),
E(2, 'log', 'w', { level: 'warn', code: 'report_catalog_trimmed', message: TRIMMED, budget: 51200, window: null, queries: 164,
omitted: 74, nodes: [{ node_id: 'a', title: '会员专项', total: 26, omitted: 0 }, { node_id: 'b', title: '卡券核销', total: 79, omitted: 52 },
{ node_id: 'c', title: '活动内容', total: 42, omitted: 22 }] })])).find((s) => s.code === 'report_catalog_trimmed')
check('report_catalog_trimmed:标题说几次查询只列了编号和摘要', trimmed?.title === '证据目录超出篇幅,164 次查询中 74 次只列了编号和摘要', trimmed?.title)
check('……副标题按来源列省略了几次(没省略的来源不列)', trimmed?.sub === '卡券核销 省略 52/79 次、活动内容 省略 22/42 次', trimmed?.sub)
// 放不下但查询全都列出、只是有的只列前几行:不写「0 次只列了编号和摘要」
const rowsOnly = flatten(mod.decodeRun([E(1, 'node.started', 'w', { node_type: 'report', label: '写报告' }),
E(2, 'log', 'w', { level: 'warn', code: 'report_catalog_trimmed', message: '证据目录超出篇幅', budget: 51200, window: null,
queries: 140, omitted: 0, compact: 38, compact_rows: 3, nodes: [{ node_id: 'a', title: '会员专项', total: 60, omitted: 0, compact: 0 },
{ node_id: 'b', title: '卡券核销', total: 80, omitted: 0, compact: 38 }] })])).find((s) => s.code === 'report_catalog_trimmed')
check('……查询全都列出、有的只列前几行:标题照实说,不写「0 次只列了编号和摘要」',
rowsOnly?.title === '证据目录超出篇幅,140 次查询都列出了数据,38 次只列了前几行' && !rowsOnly?.title?.includes('编号和摘要'), rowsOnly?.title)
check('……副标题按来源列只列了前几行的次数', rowsOnly?.sub === '卡券核销 38/80 次只列前几行', rowsOnly?.sub)
check('……下一步说其余的行仍在台账里', !!rowsOnly?.next?.includes('只看到前几行') && !!rowsOnly?.next?.includes('不得据此断言没有数据'), rowsOnly?.next)
const notesOnly = flatten(mod.decodeRun([E(1, 'node.started', 'w', { node_type: 'report', label: '写报告' }),
E(2, 'log', 'w', { level: 'warn', code: 'report_catalog_trimmed', message: '证据目录超出篇幅', queries: 12, omitted: 0, compact: 0,
nodes: [{ node_id: 'a', title: '会员专项', total: 12, omitted: 0, compact: 0 }] })])).find((s) => s.code === 'report_catalog_trimmed')
check('……只省略了表和字段的说明:照实说', notesOnly?.title === '证据目录超出篇幅,12 次查询都列出了数据,表和字段的说明有省略', notesOnly?.title)
check('……下一步说被省略的仍在台账里、不得据此断言没有数据,以及怎么看全', !!trimmed?.next?.includes('不得据此断言没有数据')
&& !!trimmed?.next?.includes('上下文窗口更大') && trimmed?.level === 'warn' && trimmed?.detail === TRIMMED, trimmed?.next)
const trimmedBare = flatten(mod.decodeRun([E(1, 'log', 'w', { level: 'warn', code: 'report_catalog_trimmed', message: '目录有省略' })]))
.find((s) => s.code === 'report_catalog_trimmed')
check('……缺字段时标题照样说人话,不出 undefined', trimmedBare?.title === '证据目录超出篇幅,部分查询只列了编号和摘要'
&& trimmedBare?.sub === undefined, `${trimmedBare?.title} / ${trimmedBare?.sub}`)
// 目录有省略时报告写了缺失断言(「没有查到」「数据缺失」),交回重写一次仍在:只标注、不判失败,时间线单独一行说清
check('后端发缺失断言的警告,带着 count、texts', reportPy.includes('code="report_absence_claim"')
&& reportPy.includes('count=len(absent), texts=texts'), '改了 report.py 的字段就同步改 decode.ts 的 explainLog 和这里的夹具')
const ABSENT = '报告在证据目录有省略时写了 2 处缺失断言(「数据缺失」、「没有查到」),已要求模型重写 1 次仍未改掉:'
+ '被省略的查询仍在证据台账里,这类结论依据不足,应写作本报告未覆盖。报告照常产出,这几处已标注'
const absentRun = flatten(mod.decodeRun([E(1, 'node.started', 'w', { node_type: 'report', label: '写报告' }),
E(2, 'log', 'w', { level: 'warn', code: 'report_absence_claim', message: ABSENT, count: 2, texts: ['数据缺失', '没有查到'] }),
E(3, 'report.checked', 'w', { doc_artifact: 'abc', repairs: 1, ok: false, on_violation: 'fail', failed: false,
stats: { numbers: 0, numbers_cited: 0, uncited_numbers: 0, unresolved: 0 },
violations: [{ code: 'absence_claim', text: '数据缺失', message: '证据目录有省略时写了缺失断言:写作模型没有看到全部证据,「没有查到」「不存在」「缺失」这类结论依据不足' }] }),
]))
const ab = absentRun.find((s) => s.code === 'report_absence_claim')
check('report_absence_claim:标题说几处缺失断言、已标注、为什么(目录有省略)', ab?.title === '报告写了 2 处缺失断言,已标注(证据目录有省略)', ab?.title)
check('……副标题点名那几处说法', ab?.sub === '「数据缺失」、「没有查到」', ab?.sub)
check('……下一步说被省略的仍在台账里、应写「本报告未覆盖」;是提醒(warn),原话留在展开区',
!!ab?.next?.includes('本报告未覆盖') && !!ab?.next?.includes('证据台账') && ab?.level === 'warn' && ab?.detail === ABSENT, ab?.next)
const rcAbsent = absentRun.find((s) => s.code === 'report_checked')
check('……核对那一行照常完成(不判失败),违规清单里写着这条的说明', rcAbsent?.status === 'done'
&& !!rcAbsent?.detail?.includes('缺失断言') && !!rcAbsent?.detail?.includes('依据不足'), `${rcAbsent?.status} ${rcAbsent?.detail}`)
})
await section('工具门控 tool.gated(工具信任三档)', async () => {
@@ -1481,6 +1556,47 @@ await section('agent 字段按出处核对(可点击证据第二期):抽
check('「抽取结果格式有误(不是对象)」:同样说「失败,N 个字段记为空值」,原因放副标题,不说「出处不存在」',
notObject?.title === '按出处抽取字段失败,2 个字段记为空值(未以 0 代替)' && notObject?.sub === '抽取结果格式有误(不是对象)'
&& !notObject?.next?.includes('出处在查询结果中不存在'), `${notObject?.title} / ${notObject?.sub} / ${notObject?.next}`)
// 两次抽取都被截断(2026-10 一次运行:推理模型关不掉思考,4096 的额度在推理阶段就用完了):单独的 code,
// 说清是额度问题、给出能做的下一步,不说「稍后重试」「去设置测试该模型」(llm.py TRUNCATED_EXTRACT、TRUNCATED_NEXT)
const TRUNCATED = '结构化抽取没有完成:模型的输出额度在推理阶段就用完了(最后一次的额度为 32768 token),结构化结论没能写出来。'
+ '5 个字段都记为空值(未以 0 代替)。可以为该 Agent 换用非推理模型(抽取沿用 Agent 的模型),或者调大该 Agent 的「最大输出 token」后重新运行'
check('后端发截断警告的说法还是夹具里这一种', llmPy.includes('"agent_field_truncated"')
&& llmPy.includes('结构化抽取没有完成:模型的输出额度在推理阶段就用完了(最后一次的额度为 {budget} token),"')
&& llmPy.includes('"结构化结论没能写出来"') && llmPy.includes('可以为该 Agent 换用非推理模型(抽取沿用 Agent 的模型),或者调大该 Agent 的"')
&& llmPy.includes('已把额度加大到 {bigger} token 重试一次'), '改了 llm.py 的说法就同步改 decode.ts 的 explainLog 和这里的夹具')
const FIELDS5 = ['members', 'stores', 'coupons', 'redeemed', 'rate']
const cut = flatten(mod.decodeRun([
E(1, 'node.started', 'fetch', { node_type: 'agent', label: '会员专项' }),
E(2, 'llm.start', 'fetch', { model: 'm1', structured: true, purpose: 'cite_fields', message_count: 5, max_tokens: 16384 }),
E(3, 'llm.end', 'fetch', { agent: '会员专项', model: 'm1', purpose: 'cite_fields', duration_ms: 40000, output_tokens: 16384,
error: '输出被截断:16384 token 的输出额度用完了,结构化结论没有写完,已把额度加大到 32768 token 重试一次' }),
E(4, 'llm.start', 'fetch', { model: 'm1', structured: true, purpose: 'cite_fields', message_count: 5, max_tokens: 32768 }),
E(5, 'llm.end', 'fetch', { agent: '会员专项', model: 'm1', purpose: 'cite_fields', duration_ms: 80000, output_tokens: 32768,
error: '输出被截断:32768 token 的输出额度用完了,结构化结论没有写完' }),
E(6, 'log', 'fetch', { level: 'warn', code: 'agent_field_truncated', message: TRUNCATED, fields: FIELDS5, max_tokens: 32768 }),
E(7, 'node.finished', 'fetch', { duration_ms: 125000 }),
]))
const tr = cut.find((s) => s.code === 'agent_field_truncated')
check('agent_field_truncated:标题说额度在推理阶段用完、几个字段记为空值', tr?.title === '模型的输出额度在推理阶段用完,5 个字段记为空值(未以 0 代替)', tr?.title)
check('……副标题说抽取时输出被截断、重试到的额度(读 max_tokens 字段)',
tr?.sub === '按出处抽取字段时输出被截断(已加大额度重试,最后一次为 32,768 token),结构化结论没能写出来', tr?.sub)
check('……下一步:换非推理模型或调大最大输出 token;不说稍后重试,不叫人去测试模型',
!!tr?.next?.includes('非推理模型') && !!tr?.next?.includes('「最大输出 token」') && !tr?.next?.includes('稍后重试')
&& !tr?.next?.includes('测试') && !!tr?.next?.includes('不得据此断言没有数据'), tr?.next)
check('……入口打开该 Agent 的设置、落到最大输出 token;是提醒(warn),原话留在展开区',
tr?.fix === 'canvas' && tr?.fixField === 'max_tokens' && tr?.level === 'warn' && tr?.detail === TRUNCATED,
`${tr?.fix} ${tr?.fixField} ${tr?.level}`)
const tries = cut.filter((s) => s.kind === 'llm')
check('……两次抽取各成一行,第一次的展开区写明已加大额度重试', tries.length === 2
&& !!tries[0].detail?.includes('已把额度加大到 32768 token 重试一次') && !!tries[1].detail?.includes('结构化结论没有写完'),
tries.map((s) => s.detail).join(' | '))
check('……节点本身照样成功', cut.find((s) => s.kind === 'node')?.status === 'done')
const bareCut = flatten(mod.decodeRun([E(1, 'node.started', 'x', { node_type: 'agent' }),
E(2, 'log', 'x', { level: 'warn', code: 'agent_field_truncated', message: '结构化抽取没有完成。3 个字段都记为空值(未以 0 代替)' })]))
.find((s) => s.code === 'agent_field_truncated')
check('……缺 fields、max_tokens 字段时字段数从原话里认,不出 undefined',
bareCut?.title === '模型的输出额度在推理阶段用完,3 个字段记为空值(未以 0 代替)'
&& bareCut?.sub === '按出处抽取字段时输出被截断,结构化结论没能写出来', `${bareCut?.title} / ${bareCut?.sub}`)
const node = mod.decodeRun([E(1, 'node.started', 'fetch', { node_type: 'agent' }), E(2, 'node.finished', 'fetch', { duration_ms: 1 })])
check('对照:普通 agent 节点不多出任何行', flatten(node).filter((s) => s.kind !== 'node').length === 0)
const odd = flatten(mod.decodeRun([E(1, 'node.started', 'x', { node_type: 'agent' }),
+28 -1
View File
@@ -1094,7 +1094,34 @@ await section('sqlcheck', 'SQL 检查(数据目录阶段 4B):查询步骤
await bad.ctx.close()
})
await section('q-narrow', '查询步骤在 360px:栏内展开、底部抽屉都不横向滚动;减少动效', async () => {
await section('absence', '缺失断言:证据目录有省略时写了「没有查到」,违规清单写明是什么、为什么依据不足,带着原句', async () => {
// 违规的形状:backend/app/engine/evidence.py 的 absence_claims(code、message、text、span、segment、unit、context)
const MESSAGE = '证据目录有省略时写了缺失断言:写作模型没有看到全部证据,「没有查到」「不存在」「缺失」这类结论依据不足'
const r = await probeQ((id, body) => body, {
patchQuery: (f) => {
const units = f.doc.blocks.flatMap((b) => b.units ?? [])
const unit = units.find((u) => (u.segments ?? []).some((s) => s.kind === 'text' && s.text.trim().length > 2))
const seg = unit.segments.find((s) => s.kind === 'text' && s.text.trim().length > 2)
f.doc.violations = [...(f.doc.violations ?? []), { code: 'absence_claim', message: MESSAGE, text: '没有查到',
span: [seg.span[0], seg.span[0] + 1], segment: seg.id, unit: unit.id, context: '核销数据缺失:本轮没有查到任何结果。' }]
return f
},
})
await openQ(r.page, '1,288')
await r.page.locator('[data-evidence-panel] [data-ev-open-violations]').click()
await r.page.waitForSelector('[data-ev-violations]')
const item = r.page.locator('[data-ev-violation="absence_claim"]')
const text = await item.innerText().catch(() => '')
check('违规清单里有这一条,标「缺失断言」,不是笼统的「文档核对未通过」', text.includes('缺失断言') && !text.includes('文档核对未通过'), text)
check('……写明为什么依据不足、该怎么写(文案在 terms.ts)', text.includes(MESSAGE)
&& (await item.locator('[data-ev-absence-hint]').innerText().catch(() => '')).includes('本报告未覆盖'), text.replace(/\s+/g, ' '))
check('……带着原句;说的是一整句,不给「定位」,也不说「正文里没有对应文本」', text.includes('本轮没有查到任何结果')
&& await item.getByRole('button', { name: '定位' }).count() === 0 && !text.includes('没有对应文本'), text.replace(/\s+/g, ' '))
check('没有运行时报错', r.errors.length === 0, r.errors.join(' | '))
await r.ctx.close()
})
await section('q-narrow','查询步骤在 360px:栏内展开、底部抽屉都不横向滚动;减少动效', async () => {
const overflow = () => page.evaluate((sel) => {
const el = document.querySelector(sel)
return { sw: el.scrollWidth, cw: el.clientWidth }
+69 -1
View File
@@ -774,12 +774,24 @@ await section('参考了哪些表:画布右栏和问数据页(context 操作
&& expanded.includes('channel_visits') && expanded.includes('「shop」全部 2 张表'), expanded.slice(0, 160))
const overflow = await c.page.evaluate(() => document.documentElement.scrollWidth - document.documentElement.clientWidth)
check('展开后不撑破 380px 窄栏', overflow <= 0, `${overflow}px`)
check('有数据目录的大库:不提示起草目录', await row.locator('[data-step-next]').count() === 0
&& await row.locator('[data-fix="catalog"]').count() === 0)
check('没有运行时报错', c.errors.length === 0, c.errors.join(' | '))
await c.page.close()
const f = await open('syn=copilot-fallback&dense=1', { w: 380, h: 800, name: 'copilot-fallback' })
const fb = f.page.locator('[data-step-code="copilot_context"]')
check('挑表失败:写明只给了表名', await fb.count() === 1 && (await fb.innerText()).includes('未能按需求挑选,已提供全部表名'))
// 真实运行里数据源一张目录都没有:这一行下面常显下一步,入口去那个源的目录页
const hint = fb.locator('[data-step-next]')
check('没有数据目录的大库:常显「还没有数据目录」,不收进展开区', await hint.count() === 1
&& (await hint.innerText()).includes('这个数据源还没有数据目录,起草后挑表和 SQL 检查会更准'),
await hint.count() ? await hint.innerText() : '没有这一句')
const go = fb.locator('a[data-fix="catalog"]')
check('……附去数据目录页的入口', await go.count() === 1 && await go.getAttribute('href') === '/data/catalog/src-scenic'
&& (await go.innerText()).includes('去数据目录起草'), await go.count() ? await go.getAttribute('href') : '没有入口')
const narrow = await f.page.evaluate(() => document.documentElement.scrollWidth - document.documentElement.clientWidth)
check('……不撑破 380px 窄栏', narrow <= 0, `${narrow}px`)
await fb.locator('button').first().click()
await f.page.waitForTimeout(150)
check('挑表失败:展开看到原因', (await fb.innerText()).includes('原因:挑选数据表超过 20 秒未完成'))
@@ -911,10 +923,66 @@ await section('审批卡的上下文', async () => {
check('没署名时提醒并给去处', body.includes('未署名') && await page.locator('a[href="/settings/prefs"]').count() === 1)
await page.close()
const s = await open('approval=1&actor=张工', { w: 820, h: 500, name: 'approval-signed' })
check('署了名就写明以谁的名义', (await s.page.locator('body').innerText()).includes('将以「张工」签批'))
const signed = await s.page.locator('body').innerText()
check('署了名就写明以谁的名义', signed.includes('将以「张工」签批'))
// 人工审批自己标明运行类别:正式运行从已发布版本发起
check('人工审批:正式运行标明类别和批了会怎样', await s.page.locator('[data-approval-run-class="formal"]').count() === 1
&& signed.includes('从已发布版本发起,批准后继续运行'), signed.replace(/\n/g, ' ').slice(0, 200))
await s.page.close()
})
await section('审批卡:审的报告的核对概况', async () => {
// 真实运行里审批说明写着「批准后才会正式出具」,那却是一次探索运行;报告里一句结论和证据相矛盾,审批人批准时完全
// 不知道。审批卡自己标明运行类别,附上报告的核对概况和查看逐段证据的入口。文档、片段、证据图都用夹具伪造,非 GET 拦掉
const { readFileSync } = await import('node:fs')
const fx = JSON.parse(readFileSync(new URL('../frontend/src/run/__tests__/evidence-doc.json', import.meta.url), 'utf8'))
const page = await browser.newPage({ viewport: { width: 420, height: 900 } })
const errors = []
page.on('pageerror', (e) => errors.push(e.message))
await page.route((u) => new URL(u).pathname.startsWith('/api/'), (r) => {
const url = new URL(r.request().url())
if (r.request().method() !== 'GET') return r.abort()
const seg = url.pathname.match(/\/evidence\/segments\/([^/]+)$/)
if (seg) return r.fulfill({ json: fx.segments[seg[1]] ?? {} })
if (url.pathname === `/api/runs/${fx.run_id}/evidence`) return r.fulfill({ json: fx.graph })
if (url.pathname === `/api/artifacts/${fx.doc_artifact}`) return r.fulfill({ json: { id: fx.doc_artifact, content: fx.doc } })
return r.continue()
})
await page.goto(`${WEB}/preview.html?approval=report&dense=1&actor=张工`, { waitUntil: 'networkidle' })
await page.waitForSelector('[data-approval]', { timeout: 6000 })
const card = page.locator('[data-approval]')
const text = await card.innerText()
check('探索运行:标明类别,写明批准后输出、不是正式出具', await card.locator('[data-approval-run-class="exploratory"]').count() === 1
&& text.includes('批准后输出,这是探索运行,不是正式出具'), text.replace(/\n/g, ' ').slice(0, 160))
const report = card.locator('[data-approval-report="write"]')
check('报告的核对概况:标题写哪份报告', (await report.innerText()).includes('报告「写周报」的核对'))
check('数字和证据条同一种说法', (await report.locator('[data-approval-report-numbers]').innerText())
.includes('7/12 数字有出处 · 无证据 5 · 另有 1 处引用无法解析'), await report.locator('[data-approval-report-numbers]').innerText())
check('探索运行、报告没开裁判:结论句尚未核对,说明运行结束后可逐句判断、开启后会在批准前判完',
(await report.locator('[data-approval-report-claims]').innerText()).includes('结论句尚未核对')
&& (await report.innerText()).includes('没有开启结论句裁判') && (await report.innerText()).includes('交给审批的报告会在批准前判完'))
await report.locator('[data-approval-evidence]').click()
await page.waitForSelector('[data-approval-evidence-doc] [data-evidence-doc]', { timeout: 6000 })
check('查看逐段证据:卡里展开可点的报告文档,自带证据条', await report.locator('[data-evidence-doc] [data-seg]').count() > 0
&& await report.locator('[data-evidence-tally]').count() === 1)
await report.locator('[data-seg="s6"]').click()
await page.waitForSelector('[data-approval] [data-evidence-panel]', { timeout: 6000 })
check('点数字打开证据面板,在卡内展开', await page.locator('[data-approval] [data-evidence-panel]').getAttribute('data-evidence-panel') === 'inline')
const overflow = await page.evaluate(() => document.documentElement.scrollWidth - document.documentElement.clientWidth)
check('展开后不撑破窄栏', overflow <= 0, `${overflow}px`)
await report.locator('[data-approval-evidence]').click()
check('再点一下收起', await report.locator('[data-approval-evidence-doc]').count() === 0)
check('没有运行时报错', errors.length === 0, errors.join(' | '))
await page.close()
const f = await open('approval=report-formal&dense=1&actor=张工', { w: 420, h: 700, name: 'approval-report-formal' })
const formal = await f.page.locator('[data-approval-report="write"]').innerText()
check('正式运行:节点里判过的写判了几句、几句证据相矛盾', formal.includes('结论句已裁判 6 句,证据相矛盾 1 句'), formal.replace(/\n/g, ' '))
check('正式运行:数字都有出处', formal.includes('12 个数字都有出处'))
check('正式运行:卡上标明正式运行', await f.page.locator('[data-approval-run-class="formal"]').count() === 1)
await f.page.close()
})
await section('画布右栏:运行视图', async () => {
// 真页面、真 store:往 window.__studio 灌事件(同 check-canvas-fx)。工作流、会话、审批、
// 运行全用 page.route 伪造,写操作一律回 409——检查脚本不写库
+29
View File
@@ -540,6 +540,35 @@ await section('右栏收尾(decodeRun)', async () => {
fold([start('甲', 1), start('乙', 2), end('甲', 3, 100), end('乙', 4, 300)]).savedMs === 100)
})
await section('金额估不出来的调用', async () => {
// 模型不在价格目录中:token 记到了,金额只能是 0。运行页写 $0 像是没花钱,改写「金额无法估算」
const ev = (seq, type, node_id, data = {}) => ({ seq, type, node_id, ts: 5500 + seq, data })
const head = [
ev(1, 'run.started', null, { nodes: 1 }),
ev(2, 'node.started', 'agent'),
ev(3, 'llm.end', 'agent', { model: 'vendor-unlisted', input_tokens: 900, output_tokens: 90, cost_usd: 0, unpriced: true }),
ev(4, 'llm.end', 'agent', { model: 'vendor-unlisted', input_tokens: 700, output_tokens: 70, cost_usd: 0, unpriced: true }),
]
const gr = g(['agent'], [])
const live = run(head, gr)
check('运行中:llm.end 标了 unpriced 的逐次数进来', live.unpricedCalls === 2 && live.costUsd === 0, `${live.unpricedCalls}`)
const done = run([...head, ev(5, 'node.finished', 'agent', { duration_ms: 10 }),
ev(6, 'run.finished', null, { usage: { input_tokens: 1600, output_tokens: 160, cost_usd: 0, unpriced_calls: 3 } })], gr)
check('终态以后端总数为准(老后端少发的也算上)', done.unpricedCalls === 3, `${done.unpricedCalls}`)
const plain = run([ev(1, 'run.started', null, { nodes: 1 }), ev(2, 'node.started', 'agent'),
ev(3, 'llm.end', 'agent', { input_tokens: 10, output_tokens: 1, cost_usd: 0.001 })], gr)
check('目录里有价的调用不算', plain.unpricedCalls === 0)
const { runCostText, COST_TEXT } = await import(await load('/src/lib/terms.ts'))
check('一分都没估出来:写「金额无法估算」,不写 $0', runCostText(0, 3) === COST_TEXT.unpriced, runCostText(0, 3))
check('……没拿到金额时也一样', runCostText(null, 1) === COST_TEXT.unpriced)
check('估出了一部分:写「≥」', runCostText(0.012, 2) === '≥ $0.012', runCostText(0.012, 2))
check('全都有价:照常写', runCostText(0.012, 0) === '$0.012' && runCostText(0, 0) === '$0', `${runCostText(0.012, 0)} / ${runCostText(0, 0)}`)
check('运行中只算已上报的:写「≥」', runCostText(0.012, 0, true) === '≥ $0.012')
check('没有金额也没有估不出的:交给调用方写占位', runCostText(undefined, 0) === null)
check('悬停说明写清几次调用、为什么', COST_TEXT.unpricedTitle(3).includes('3 次调用') && COST_TEXT.unpricedTitle(3).includes('价格目录'))
})
await section('回放时的用量和工具数(逐刻度)', async () => {
// 回放到半路,卡片和 HUD 要写「那一刻」的 tokens 和工具数,而不是拿终值冒充
const ev = (seq, type, node_id, data = {}) => ({ seq, type, node_id, ts: 5000 + seq, data })