可点击证据第四期:结论句由模型裁判;只引单元格的问数据图能发成受管

一、结论句裁判

- 报告节点新增 claims: judge:写完、核对完之后,把结论句和它引用的证据摘录交给裁判模型,
  判为支持 / 部分支持 / 不支持 / 不是结论,附一句理由;判定只标注,不改写正文
- 送裁判的句子按固定规则预筛:带引用的、含方向词或因果词的送去判,短句、过渡句不送
- 证据摘录:指标写名称、值和原式;查询给 SQL 前 300 字和被引用的行,被引用的列一定给到;遮罩列一律不给;
  规则里写明「数字已经由系统核对过,不要复算」
- 一份报告合并成一次调用,超出句数就分批;结构化输出失败时退回 JSON 解析,再失败记为未裁判
- 裁判模型的优先级:节点上写的 > 设置里的「证据裁判模型」> Copilot 的模型;思考关闭;
  和写作模型相同时给警告(等于自己审自己)
- rewrite_once(默认关):把不支持的句子和理由交回写作者只改这几句,再裁一次;
  改写稿冒出原稿没有的问题就不采用,保留原稿和原来的判定

二、成本上限

- 每份报告的句数、金额、时长,探索运行每次点击的金额,全局每日金额,每一项都可以自定义,也都可以设成不限;
  默认 40 句、$0.05、30 秒,每次点击 $0.01,每天 $2
- 调用前先估算,超出上限按优先级截断:先判带引用的,再判含方向词、因果词的,其余记为未裁判并写明「已到上限」;
  不报错,也不让报告失败
- 每日花费按本地日期计数,持久化在设置表里;重跑同一次运行不重复计费;价格目录里没有的模型只计次数,并写明估不出金额

三、正式运行与探索运行

- 正式运行在报告节点内裁判,判定写进报告文档、随文档封存;不支持的按 on_unsupported(degrade / withhold)降档或不予出具,
  裁判没跑成记缺口,不能判完整出具
- 探索运行按需裁判:POST /api/runs/{id}/evidence/judge,点开哪句判哪句;判定作为 evidence.judged 事件追加在封存之后,
  标「封存后追加」,封存核对仍然一致;只读封存范围内的工件;同一句不重复判;运行在裁判期间被接着跑时不记录
- 正式运行调这个接口返回 409,写明「正式运行的裁判在节点内完成」

四、治理与自动修复

- G3 放开 judge:受管级别写 judge 必须写预算(judge.max_cost_usd,或显式写 null 表示不限)
- 发布前自动修复:缺预算时给选项(默认金额 / 不限),不替人选;
  从 require_citation 改成 judge 不算降低要求,judge 改回 off、withhold 放宽成 degrade 算降低要求
- Copilot 手册:受管模板可以写 judge 并带预算;问数据按需裁判,不替人打开结论句策略

五、界面

- 结论句句末放徽标:支持 ◆、部分支持 ◇、不支持 ! 整句浅底、未裁判 ?;不用表示「已核对」的绿色;n / N 也跳到不支持的句子
- 证据面板里「模型的解释」:带「模型判断 · 模型名 · 非确定」,封存后追加的另外标出;
  「请模型判断这句」只在探索运行、还没判过时出现,触顶时写明已到上限和怎么调
- 出具横幅多一段结论句计数;设置页新增「证据裁判」一组(裁判模型、各项上限、今日花费),设成不限时写明费用受什么约束;
  报告节点可以配 judge;运行轨迹里裁判相关的事件都有中文标题

六、门禁修正

- 引用模式下,图里没有口径卡(只引查询单元格)的,出具契约不再要求 metrics_from,受管门禁也不再要求 required:
  strict 加 cells 已经保证每个数都点得开、对不上就不予出具;有口径卡的照旧要求。
  以前问数据的图怎么都发不成受管,Copilot 补上契约也会被判「冒出新的问题」作废
This commit is contained in:
ThearchyHelios
2026-09-29 11:10:53 +08:00
parent d071bf119f
commit 9727c21d8d
47 changed files with 12367 additions and 263 deletions
+16 -5
View File
@@ -207,14 +207,21 @@ NODE_REFERENCE = """\
- 句末的 [[see:m:指标id,Q1,K1]]:这句结论的依据,不显示
metrics_from 不写就取所有上游口径卡;on_violation 不写时探索运行是 flag(标出来、照常产出),正式运行是 fail;
numbers 默认 strict(裸数字算违规)。claims 是结论句策略:off(默认,不管)/ require_citation(每句结论都要
挂引用,没挂的计入缺口、出具降档);judge(模型裁判)在后续版本支持,现在写了整张图跑不起来。entities 默认 link
挂引用,没挂的计入缺口、出具降档)/ judge(在 require_citation 之外,再请另一个模型按每句挂的依据判断支不支持:
正式运行在节点里判,证据不支持的按 judge.on_unsupported 处置——degrade 降档(默认)或 withhold 不予出具;
探索运行不在节点里花钱,读的人点开哪句再按需判哪句,只标注)。judge 的子配置写在节点的 judge 里:judge.model、
judge.provider 指定裁判模型(建议和写作模型不同,同一个模型等于自己审自己;不写就用设置里的「证据裁判模型」);
judge.max_cost_usd、judge.max_claims、judge.timeout_s 是每份报告的裁判金额、句数、时长上限,写 null 表示不限,
不写取设置里的默认;judge.rewrite_once 默认 false,打开后把证据不支持的句子交回写作者只改一次。entities 默认 link
(核对表名、字段名),报告里根本不提表名时才写 off
- output 的出具契约:config.contract = {report_from: 报告节点id, metrics_from:[口径卡id],
required:[必需指标id], expected:[期望指标id], strict: true}。成果字段写 {{ nodes.报告节点id.text }},
前后不要拼别的字——拼了就没法逐段对应证据,出具会降档。受管模板的报告要直接引用查询单元格
([[v:]] / [[table:]])的,契约里写 cells: true,否则这些引用按解析不了算
([[v:]] / [[table:]])的,契约里写 cells: true,否则这些引用按解析不了算;只引单元格、图里没有口径卡的
(问数据的典型图),契约不写 metrics_from,也不写 required / expected
要按受管级别发布的模板(发布前检查不过就发不出去):报告节点写明 numbers: "strict"、on_violation: "fail"、
claims: "require_citation";契约用 report_from(受管级别不收 narrative),成果字段只取报告节点的正文;给口径卡
claims: "require_citation"(或者更严的 claims: "judge",这时 judge 里必须写 judge.max_cost_usd——写一个金额,或者
写 null 表示不限,预算要用户定,拿不准就问);契约用 report_from(受管级别不收 narrative),成果字段只取报告节点的正文;给口径卡
供数的 agent 配 output_schema 和 cite_fields: true,code 节点喂口径卡的必须是真取数(evidence_role: "source"),
否则把计算写进口径卡;llm、agent、协作团队写的文字不能绕过报告节点直接进出口,llm 也不能喂口径卡
@@ -321,7 +328,7 @@ def _user_message(payload: GenerateIn, *, patch: bool) -> str:
# 三期起报告会核对写出来的表名、字段名:instructions 里让它「列出用到的表」时,编出来的名字会被标出来
"- report 提到表名、字段名时写 [[t:表名]] / [[c:表名.字段名]] 或放进反引号,只写 agent 真查过的名字,"
"编造的会被标出来;要引用知识库原话就在 report 前接 retrieve,用 [[q:K1|原话]] 逐字引用。问数是探索运行,"
"report 的 claims 不用写\n"
"report 的 claims 不用写:结论句支不支持,由读答案的人点开时按需请模型判断\n"
"- 用户要的是周报、要三档出具时,按搭图规则配口径卡和 report_from 契约\n"
"- 不要用 transform 解析 agent / llm 的文字:要把 agent 查到的数交给下游,就给它配 output_schema + "
"cite_fields\n"
@@ -1633,7 +1640,11 @@ _ASSIST_RULES = """\
审批策略只能往严里改(每次调用都审批 > 仅危险工具需要审批 > 全部自动放行),不放宽、不删掉,更不改成 never;
不取消子工作流钉住的版本;发布级别不是你能改的。结构要动(比如换掉协作团队),输出 question 说明怎么拆,不要自己动手;
报告撰写节点的 numbers、on_violation、claims 只能往严里改(numbers: strict、on_violation: fail、
claims: require_citation),不放宽、不删掉;契约里的 claims 也一样;不关 agent 的 cite_fields,不删它的 output_schema
claims: require_citation),不放宽、不删掉;claims 从宽到严是 off < require_citation < judge,已经是 judge 的不改回
require_citation;judge 缺预算(judge.max_cost_usd)时输出 question 问写多少、还是写 null 不限,不替人定;
judge.on_unsupported 是 withhold 的不改回 degrade、不删掉;update_node 会把 judge 整个换掉,补预算时把原来的
judge(包括从全图默认继承来的)整份带上,只加 max_cost_usd;契约里的 claims 也一样;
不关 agent 的 cite_fields,不删它的 output_schema
- 要人拿主意的,不替人选、不编:required 该包括哪些指标、几张口径卡该用哪张、钉哪个版本、协作团队拆成哪几个
固定步骤、沙箱代码节点是在取数还是在计算(不许自己把 evidence_role 设成 source)……每一处输出一行
{"op":"question","node_id":"节点 id","text":"要问的话,把候选列出来"},那一处不改
+340 -10
View File
@@ -20,23 +20,30 @@ artifacts 表可以事后插行,run.output 是可以改写的一列,封存
查询步骤只给被引用的行和前后各 2 行,完整快照仍走 /api/artifacts/{id}。数据源
options.mask_columns 里的列换成「已遮罩」——在有身份体系之前,这只减少暴露,不是
安全边界:知道工件 id 的人照样能取到整份快照。
结论句的判定(四期)是模型给的,不是证据:正式运行的判定在报告文档里、随文档封存;探索运行
按需裁判(POST …/evidence/judge),判定作为 evidence.judged 事件追加在封存之后,只填补没判过的
句子。交给裁判的摘录同样只取封存范围内的工件。
"""
from __future__ import annotations
import asyncio
import contextlib
import csv
import io
import re
from collections import OrderedDict
from collections.abc import Callable
from dataclasses import dataclass, field
from typing import Any
from fastapi import APIRouter, Query
from fastapi import APIRouter, Body, Query
from fastapi.responses import JSONResponse, Response
from sqlalchemy import select
from sqlalchemy import select, update
from app.api.coded import CodedHTTPException
from app.core.artifact_store import load
from app.core.events import EventType
from app.data.engine import masked_columns
from app.db.base import SessionLocal
from app.db.models import DataSource, Run, RunEvent, Workflow
@@ -54,6 +61,7 @@ from app.engine.evidence import (
guess_sources,
input_eid,
iter_units,
ledger_enabled,
make_eid,
normalize_quote,
query_entry_fields,
@@ -61,6 +69,7 @@ from app.engine.evidence import (
sql_tables,
uncited_claims,
)
from app.engine.judge import JUDGED, VERDICTS, candidates
from app.engine.toolcalls import QUERY_PREFIX
router = APIRouter(prefix="/api/runs", tags=["evidence"])
@@ -120,8 +129,17 @@ class _Sealed:
retrievals: dict[str, dict[str, Any]] = field(default_factory=dict)
#: 封存时记下的清单哈希(没封存的是 None):旧答案的猜测按 (run_id, 它) 缓存
manifest_hash: str | None = None
#: 运行记录上的 manifest_seq 原值(老运行是 None):按需裁判追加判定前核对封存没换过
run_manifest_seq: int | None = None
#: verify_manifest 的完整结论(审计表照实附上)
verdict: dict[str, Any] = field(default_factory=dict)
#: 运行记录上的几样:按需裁判只给跑完、封存了的探索运行,升级前发起的不给
run_class: str | None = None
status: str | None = None
ledger_on: bool = False
#: 按需裁判的判定(evidence.judged 事件),按先后:(seq, 节点, 载荷)。不管落在封存前后都收——
#: 失败后接着跑的运行会把早先追加的判定封进新的清单里,它仍然是按需裁判的批注
judged: list[tuple[int, str | None, dict[str, Any]]] = field(default_factory=list)
@property
def sealed(self) -> bool:
@@ -161,14 +179,18 @@ async def _sealed(run_id: str) -> _Sealed:
select(RunEvent.seq, RunEvent.type, RunEvent.node_id, RunEvent.data)
.where(RunEvent.run_id == run_id).order_by(RunEvent.seq))]
graph = run.graph if isinstance(run.graph, dict) else {}
manifest = run.manifest_hash
manifest, manifest_seq = run.manifest_hash, run.manifest_seq
run_class, status, ledger_on = run.run_class, run.status, ledger_enabled(run)
verdict = await verify_manifest(run_id)
bound = verdict.get("sealed_at") if verdict.get("sealed") else None
# 没封存(还在跑、停在审批)的运行没有「封存之后」:现有的事件都算,但每件证据都标未封存
events = [r for r in rows if bound is None or r[0] <= bound]
out = _Sealed(run_id=run_id, graph=graph, events=events, seal={
"sealed": bool(verdict.get("sealed")), "ok": verdict.get("ok"), "manifest_seq": bound,
"legacy": bool(verdict.get("legacy", False))}, manifest_hash=manifest, verdict=verdict)
"legacy": bool(verdict.get("legacy", False))}, manifest_hash=manifest, run_manifest_seq=manifest_seq,
verdict=verdict, run_class=run_class, status=status, ledger_on=ledger_on,
judged=[(seq, nid, data) for seq, etype, nid, data in rows
if etype == EventType.EVIDENCE_JUDGED and isinstance(data, dict)])
for _, etype, node_id, data in events:
if not node_id or not isinstance(data, dict):
continue
@@ -306,6 +328,7 @@ async def evidence_graph(run_id: str) -> dict[str, Any]:
"node_id": report.node_id, "doc_artifact": report.doc_artifact,
"doc_sealed": sealed.trusted, "hash_ok": report.hash_ok, "ok": report.ok,
"repairs": report.repairs, "fields": report.fields, "stats": report.stats,
**_judgement_of(report, sealed),
})
if report.doc is not None:
evidence, edges = _graph_of(report, sealed, inputs)
@@ -666,13 +689,16 @@ async def evidence_segment(run_id: str, segment_id: str, report: str | None = No
chain = await _chain(seg, doc, sealed, chosen.node_id)
covered = sealed.trusted and all(step.get("sealed") for step in chain if "sealed" in step)
masked = list(dict.fromkeys(c for step in chain if step.get("step") == "query" for c in step.get("masked") or []))
verdict = _effective_verdicts(chosen, sealed)[0].get(str(unit.get("id")))
return {
"report": {"node_id": chosen.node_id, "doc_artifact": chosen.doc_artifact},
"segment": {"id": seg["id"], "text": seg.get("text"), "kind": seg.get("kind"), "state": seg.get("state"),
"span": seg.get("span"), "unit": unit.get("id"),
**{k: seg[k] for k in ("ref", "issue", "strong", "cite") if k in seg}},
"unit": {"id": unit.get("id"), "kind": unit.get("kind"), "text": markdown[start:end],
"span": unit.get("span"), "cites": unit.get("cites") or []},
"span": unit.get("span"), "cites": unit.get("cites") or [],
**({"verdict": verdict} if verdict is not None else {}),
"on_demand": _on_demand(sealed, doc, str(unit.get("id")), verdict)},
"block": {"id": block.get("id"), "type": block.get("type")},
"chain": chain,
"note": _note(seg, unit),
@@ -1151,8 +1177,304 @@ def _note(seg: dict[str, Any], unit: dict[str, Any]) -> str:
cites = [str(c) for c in unit.get("cites") or []]
if cites:
return (f"这是结论句里的文字,这句挂了依据:{'、'.join(cites[:4])}{' 等' if len(cites) > 4 else ''}。"
"依据支不支持这句话,要到后续版本由模型判断")
return "这是结论句里的文字,这句没有挂依据。句子本身有没有依据要到后续版本由模型判断"
"依据支不支持这句话,系统核对不了,只能由模型判断(模型的判断不是确定性的)")
return "这是结论句里的文字,这句没有挂依据。句子本身有没有依据,系统核对不了,只能由模型判断(模型的判断不是确定性的)"
# --------------------------------------------------------------------------
# 结论句的判定:封存内的(正式运行在节点里判的)和封存之后按需追加的
# --------------------------------------------------------------------------
JUDGE_FORMAL = "evidence_judge_formal"
JUDGE_UNSEALED = "evidence_judge_unsealed"
JUDGE_LEGACY = "evidence_judge_legacy"
JUDGE_BAD_UNITS = "evidence_judge_bad_units"
SEAL_BROKEN_CODE = "evidence_seal_broken"
#: 一次最多点名几句。一份报告通常几十句,再多就是请求写错了
JUDGE_MAX_UNITS = 200
#: 跑完了的几种终态。还在跑、停在审批、正在接着跑的运行,追加的判定会被封进下一次的清单,不判
_TERMINAL = ("succeeded", "failed", "cancelled")
FORMAL_JUDGED = "正式运行的裁判在节点内完成(claims: judge),判定随报告文档一起封存,不能再按需裁判"
UNSEALED_JUDGE = "运行还没跑完封存(还在跑、停在人工审批,或者正在接着跑),跑完再请模型判断"
LEGACY_JUDGE = "这次运行发起于证据台账上线之前,不支持按需裁判"
BROKEN_JUDGE = "封存核对没通过:封存之后有事件被改过、删过或插过,报告和证据都不能再当真,不再请模型判断"
JUDGED_ALREADY = "这句已经判过了"
#: 裁判期间运行被接着跑了(失败的运行可以继续),或者接着跑完重新封存了:判定照样交回,不记进运行记录
NOT_RECORDED = "运行已经接着跑了,这次的判定没有记进运行记录(只在这里看得到,刷新就没了);等它跑完封存,再请模型判断"
NOT_A_CLAIM = "不是结论句(标题、连接性的话、表格单元格、代码),不需要模型判断"
BAD_UNITS = f'units 要写成句子编号的列表,比如 {{"units": ["u4"]}},一次最多 {JUDGE_MAX_UNITS} 句;report 要写报告节点的 id'
#: 触顶时告诉人怎么调:设置页「证据裁判」一组的各项上限都能调高,也都能设成不限
JUDGE_ADJUST = {
"max_cost_usd": "到设置 → 证据裁判里调高「每次点击」的金额上限,或者设成不限",
"daily_max_usd": "今天的裁判花费到了全局每日上限:到设置 → 证据裁判里调高「每日」上限或者设成不限,也可以明天再判",
"max_claims": "一次最多判这么多句:分几次点,或者到设置 → 证据裁判里调高「每份报告的句数」"
"(报告节点上写了 judge.max_claims 的按节点,写 null 是不限)",
"timeout_s": "裁判超时了:到设置 → 证据裁判里调高「时长」上限或者设成不限,再点一次",
}
def _verdict_ok(verdict: Any) -> bool:
return isinstance(verdict, dict) and verdict.get("status") in VERDICTS
def _effective_verdicts(report: _Report, sealed: _Sealed
) -> tuple[dict[str, dict[str, Any]], dict[str, dict[str, Any]]]:
"""这份报告每句的判定:(生效的, 其中封存之后按需追加的)。
文档里的判定随文档封存(正式运行在节点里判的;探索运行的候选句记「未裁判 · 按需」)。按需裁判的判定在
evidence.judged 事件里,只认指向这份文档、文档里确有其句的;它只填补没判过的句子——封存内判过的,后面
追加什么都盖不掉,一条追加的事件改写不了封存的结论。同一句按需判过几次(先触顶、后判成),判成的为准,
其次取最后一次。追加的判定一律标 post_seal。
"""
doc = report.doc or {}
ids = {str(u.get("id")) for _, u in iter_units(doc)}
out = {str(u.get("id")): u["verdict"] for _, u in iter_units(doc) if _verdict_ok(u.get("verdict"))}
sealed_judged = {uid for uid, v in out.items() if v.get("status") in JUDGED}
later: dict[str, dict[str, Any]] = {}
for _, _, data in sealed.judged:
if data.get("report") != report.node_id or data.get("doc_artifact") != report.doc_artifact:
continue
verdicts = data.get("verdicts") if isinstance(data.get("verdicts"), dict) else {}
for uid, verdict in verdicts.items():
uid = str(uid)
if uid not in ids or uid in sealed_judged or not _verdict_ok(verdict):
continue
if later.get(uid, {}).get("status") in JUDGED and verdict.get("status") not in JUDGED:
continue
later[uid] = out[uid] = {**verdict, "post_seal": True}
return out, later
def _judgement_of(report: _Report, sealed: _Sealed) -> dict[str, Any]:
"""证据图、审计表里每份报告带的结论句情况:写作时的策略、封存的裁判摘要、封存之后按需追加的判定。"""
doc = report.doc or {}
return {"claims": report.claims, "judge": doc.get("judge") if isinstance(doc.get("judge"), dict) else None,
"post_seal_verdicts": _effective_verdicts(report, sealed)[1] if report.doc is not None else {}}
def _on_demand(sealed: _Sealed, doc: dict[str, Any], uid: str, verdict: dict[str, Any] | None) -> dict[str, Any]:
"""这句能不能「请模型判断」:{available, reason, message}。和 POST …/evidence/judge 同一套条件。"""
def no(reason: str, message: str) -> dict[str, Any]:
return {"available": False, "reason": reason, "message": message}
if sealed.run_class == "formal":
return no("formal", FORMAL_JUDGED)
if not sealed.ledger_on:
return no("legacy", LEGACY_JUDGE)
if not candidates(doc, [uid]):
return no("not_a_claim", NOT_A_CLAIM)
if isinstance(verdict, dict) and verdict.get("status") in JUDGED:
return no("judged", JUDGED_ALREADY)
if not sealed.sealed or sealed.status not in _TERMINAL:
return no("unsealed", UNSEALED_JUDGE)
if not sealed.trusted:
return no("seal_broken", BROKEN_JUDGE)
return {"available": True, "reason": None, "message": None}
def _sealed_loader(sealed: _Sealed) -> Callable[[str], Any]:
"""给裁判取证据的 loader:只认封存范围内的事件交回过的工件(查询、表结构、检索快照和台账里记过的),
取回时照旧复验哈希。别的一律当取不到——人为插进工件表的行、封存之后才追加的事件引用的快照,
都不能送进裁判的摘录。报告文档里的目录也不例外:目录写着哪件工件,不等于它在封存链上。"""
allowed = {*sealed.queries, *sealed.schemas, *sealed.retrievals, *(str(a) for _, a in sealed.ledger)}
def loader(artifact: str) -> Any:
return load(artifact) if sealed.trusted and str(artifact) in allowed else None
return loader
def _node_judge(sealed: _Sealed, node_id: str) -> dict[str, Any]:
"""这次运行的图上报告节点写的 judge(裁判模型、句数和时长上限);图读不出来就当没写。"""
from app.engine.nodes.report import report_judge
from app.engine.schema import GraphSpec
try:
spec = GraphSpec.model_validate(sealed.graph)
node = spec.node_map().get(node_id)
except Exception: # noqa: BLE001 - 老运行的图可能已经不合现在的格式
node = None
return report_judge(spec, node) if node is not None else {}
class _RunLocks:
"""每次运行一把锁:同一次运行的按需裁判排队,连点两下时后一下看得到前一下记下的判定,不重复花钱。"""
def __init__(self) -> None:
self._locks: dict[str, asyncio.Lock] = {}
self._users: dict[str, int] = {}
@contextlib.asynccontextmanager
async def hold(self, key: str): # type: ignore[no-untyped-def]
lock = self._locks.setdefault(key, asyncio.Lock())
self._users[key] = self._users.get(key, 0) + 1
try:
async with lock:
yield
finally:
self._users[key] -= 1
if not self._users[key]:
del self._users[key], self._locks[key]
_JUDGING = _RunLocks()
def _judge_request(payload: Any, report: str | None) -> tuple[list[str], str | None]:
if not isinstance(payload, dict):
raise CodedHTTPException(422, BAD_UNITS, JUDGE_BAD_UNITS)
units = payload.get("units")
wanted = report if payload.get("report") is None else payload["report"]
if not isinstance(units, list) or not units or len(units) > JUDGE_MAX_UNITS \
or not all(isinstance(u, str) and u.strip() for u in units) \
or (wanted is not None and not isinstance(wanted, str)):
raise CodedHTTPException(422, BAD_UNITS, JUDGE_BAD_UNITS)
return list(dict.fromkeys(u.strip() for u in units)), wanted or None
@router.post("/{run_id}/evidence/judge")
async def judge_units(run_id: str, payload: Any = Body(None), report: str | None = None) -> dict[str, Any]:
"""探索运行里按需裁判几句结论:请求体 {units: ["u4"], report?: 报告节点 id}。
判过的句子直接给已有的判定,不再调用;没判过的按设置里每次点击的金额上限、每份报告的句数和时长上限、
全局每日上限去判,触顶照实说「已到上限」,不是报错。问过模型的这一次记一条 evidence.judged 事件,
落在封存之后(post_seal: true)——封存核对只覆盖 manifest_seq 之前的事件,追加它 verify 照旧一致;
封存的报告文档一个字都不改。交给裁判的证据只取封存范围内的工件,数据源遮罩的列不给。
"""
units, wanted = _judge_request(payload, report)
async with _JUDGING.hold(run_id):
sealed = await _sealed(run_id)
if sealed.run_class == "formal":
raise CodedHTTPException(409, FORMAL_JUDGED, JUDGE_FORMAL)
if not sealed.ledger_on:
raise CodedHTTPException(409, LEGACY_JUDGE, JUDGE_LEGACY)
if not sealed.sealed or sealed.status not in _TERMINAL:
raise CodedHTTPException(409, UNSEALED_JUDGE, JUDGE_UNSEALED)
if not sealed.trusted:
raise CodedHTTPException(409, BROKEN_JUDGE, SEAL_BROKEN_CODE)
reports = _reports(sealed)
if not reports:
raise CodedHTTPException(404, "这次运行没有封存的报告文档,没有可以判断的句子", REPORT_NOT_FOUND)
chosen = _choose(reports, wanted, sealed)
if chosen.hash_ok is False:
raise CodedHTTPException(409, "报告文档和它的哈希对不上,疑似被改过,不能再当证据展示", DOC_TAMPERED)
if chosen.doc is None:
raise CodedHTTPException(404, "报告文档在工件库里取不回来", DOC_MISSING)
return await _judge_now(sealed, chosen, units)
async def _judge_now(sealed: _Sealed, report: _Report, units: list[str]) -> dict[str, Any]:
from app.engine import judge as judging
doc = report.doc or {}
catalog = doc.get("catalog") or {}
known = _effective_verdicts(report, sealed)[0]
reused = [u for u in units if known.get(u, {}).get("status") in JUDGED]
pending = [u for u in units if u not in reused]
verdicts = {u: known[u] for u in reused}
node_judge = _node_judge(sealed, report.node_id)
async with SessionLocal() as session:
settings = await judging.judge_settings(session)
budget = judging.click_budget(settings, node_judge)
outcome: dict[str, Any] | None = None
skipped: dict[str, str] = {}
event = None
lost = False
if pending:
loader = _sealed_loader(sealed)
masked = await judging.source_masks(catalog, loader=loader)
request = judging.prepare(doc, catalog, units=pending, loader=loader, masked=masked)
skipped = dict(request.skipped)
if request.cands:
async with SessionLocal() as session:
spec = await judging.judge_model_spec(session, node_judge, settings=settings)
outcome = await judging.run_request(request, spec=spec, budget=budget, post_seal=True, click=True)
verdicts.update(outcome["verdicts"])
if outcome["calls"] or outcome["cost_usd"]:
# 没问模型(触顶、模型没配好)就没有判定可记:说给点的人听,不往运行记录里追加
event = await _append_judged(sealed, report, pending, outcome)
lost = event is None
limits = list(outcome["limits_hit"]) if outcome else []
message = _judge_message(outcome, budget, skipped)
if lost:
message = ";".join(filter(None, [NOT_RECORDED, message]))
return {
"run_id": sealed.run_id, "report": {"node_id": report.node_id, "doc_artifact": report.doc_artifact},
"units": units, "verdicts": {u: verdicts[u] for u in units if u in verdicts}, "reused": reused,
"judged": [u for u in pending if (outcome or {}).get("verdicts", {}).get(u, {}).get("status") in JUDGED],
"skipped": skipped, "unjudged": dict(outcome["unjudged"]) if outcome else {},
"limits_hit": limits, "limited": bool(limits), "message": message,
"adjust": [JUDGE_ADJUST[r] for r in limits if r in JUDGE_ADJUST],
"gaps": list(outcome["gaps"]) if outcome else [], "notes": list(outcome["notes"]) if outcome else [],
"model": (outcome or {}).get("model"), "priced": (outcome or {}).get("priced"),
"cost_usd": (outcome or {}).get("cost_usd", 0.0), "calls": (outcome or {}).get("calls", 0),
"duration_ms": (outcome or {}).get("duration_ms", 0),
"budget": budget.as_dict() if outcome else None, "event": event, "post_seal": True,
"spend": {**await judging.daily_spend(), "daily_max_usd": settings.get("daily_max_usd")},
}
def _judge_message(outcome: dict[str, Any] | None, budget: Any, skipped: dict[str, str]) -> str | None:
"""这次点击的一句话结论:触顶、没跑成、点的不是结论句;判成了(或者全是判过的)就没什么要说的。"""
from app.engine.judge import _limit_label
if outcome and outcome["limits_hit"]:
labels = "、".join(_limit_label(r, budget, click=True) for r in outcome["limits_hit"])
n = sum(outcome["unjudged"].get(r, 0) for r in outcome["limits_hit"])
return f"已到上限({labels}):{n} 句没判,记为未裁判;已判的保留"
if outcome and outcome["gaps"]:
return "裁判没跑完:" + ";".join(outcome["gaps"])
parts = []
if others := [u for u, why in skipped.items() if why == "not_a_claim"]:
parts.append(f"{'、'.join(others)} {NOT_A_CLAIM}")
if missing := [u for u, why in skipped.items() if why == "not_found"]:
parts.append(f"报告里没有 {'、'.join(missing)}")
return ";".join(parts) or None
async def _append_judged(sealed: _Sealed, report: _Report, units: list[str],
outcome: dict[str, Any]) -> dict[str, Any] | None:
"""记一条 evidence.judged:问过模型的这一次(连同没判成的句子),接在运行记录的最后一条后面。
只接在这次点击开始时的那份封存后面:裁判要花几秒(时长不限时更久),这中间有人点了「继续运行」
(失败的运行可以接着跑),再追加就落进一次活着的运行中间,还会被封进下一份清单;接着跑完、重新封存了
也一样,这次的判定依据的已经不是最新的封存。这两种都不记,返回 None(判定照样交回给点的人)。
核对和写入是同一个事务里的条件更新,和接着跑占住运行(runner._claim)抢的是同一行:谁先提交算谁的。
"""
from app.core.bus import bus
from app.engine.runner import _stage, run_manager
same_seal = (Run.manifest_seq.is_(None) if sealed.run_manifest_seq is None
else Run.manifest_seq == sealed.run_manifest_seq,
Run.manifest_hash.is_(None) if sealed.manifest_hash is None
else Run.manifest_hash == sealed.manifest_hash)
async with SessionLocal() as session:
run = await session.get(Run, sealed.run_id)
if run is None:
return None
# 进程重启过的话内存里的序号从 0 起:接在库里最后一条后面,不和已有的撞号
bus.set_seq(sealed.run_id, run.last_seq or 0)
event = run_manager._event(
sealed.run_id, EventType.EVIDENCE_JUDGED, node_id=report.node_id, data={
"report": report.node_id, "doc_artifact": report.doc_artifact, "units": units,
"verdicts": outcome["verdicts"],
"keys": {u: outcome["keys"][u] for u in outcome["verdicts"] if u in outcome["keys"]},
"model": outcome["model"], "priced": outcome["priced"], "cost_usd": outcome["cost_usd"],
"calls": outcome["calls"], "duration_ms": outcome["duration_ms"], "budget": outcome["budget"],
"limits_hit": outcome["limits_hit"], "unjudged": outcome["unjudged"], "gaps": outcome["gaps"],
"notes": outcome["notes"], "skipped": outcome["skipped"], "post_seal": True})
claimed = await session.execute(
update(Run).where(Run.id == sealed.run_id, Run.status.in_(_TERMINAL), *same_seal)
.values(last_seq=event.seq))
if claimed.rowcount != 1:
# 运行已经不是那份封存下的终态了。烧掉一个序号不要紧:不落库的事件(令牌流)本来就占序号,
# 序号只要求递增、不撞号
await session.rollback()
return None
await _stage(session, event, run)
await session.commit()
await bus.publish(event)
return {"seq": event.seq}
# --------------------------------------------------------------------------
@@ -1172,6 +1494,11 @@ UNCITED_CLAIM_COUNTED = "这句结论没有挂依据:这份报告要求结论
UNCITED_CLAIM_WITHHELD = "这句结论没有挂依据:出具契约要求结论句挂依据,没挂的按契约不予出具(on_uncited: withhold)"
UNCITED_CLAIM_REQUIRED = ("这句结论没有挂依据:这份报告要求结论句挂依据(claims: require_citation),不过没有出具契约"
"按它判档(没配契约,或者还没到出具那一步)")
#: claims: judge 在挂依据这件事上和 require_citation 一样严(出口按 on_uncited 计入缺口)
UNCITED_CLAIM_JUDGED = ("这句结论没有挂依据:这份报告的结论句由模型裁判(claims: judge),挂依据的要求和 "
"require_citation 一样,出具时计入缺口")
UNCITED_CLAIM_JUDGE_REQUIRED = ("这句结论没有挂依据:这份报告的结论句由模型裁判(claims: judge),要求结论句挂依据,"
"不过没有出具契约按它判档(没配契约,或者还没到出具那一步)")
LEGACY_UNMATCHED = "旧版出具里这个数字对不上任何指标"
#: CSV 的列:(表头, 行里的键)
@@ -1239,7 +1566,7 @@ async def evidence_audit(run_id: str, fmt: str | None = Query(None, alias="forma
"run_id": run_id, "schema": AUDIT_SCHEMA, "mode": mode,
"seal": {**sealed.seal, "events": sealed.verdict.get("events"), "message": sealed.verdict.get("message")},
"reports": [{"node_id": r.node_id, "doc_artifact": r.doc_artifact, "hash_ok": r.hash_ok,
"doc_sealed": sealed.trusted, "fields": r.fields, "claims": r.claims, "stats": r.stats}
"doc_sealed": sealed.trusted, "fields": r.fields, "stats": r.stats, **_judgement_of(r, sealed)}
for r in reports],
"groups": grouped, "counts": {g["key"]: g["count"] for g in grouped},
"total": sum(g["count"] for g in grouped),
@@ -1299,11 +1626,14 @@ def _claim_note(report: _Report, issuance: dict[str, Any] | None) -> str:
if isinstance(issuance, dict) and issuance.get("mode") == "citations" \
and (issuance.get("report") or {}).get("node_id") == report.node_id:
claims = issuance.get("claims")
if not isinstance(claims, dict) or claims.get("policy") != "require_citation":
if not isinstance(claims, dict) or claims.get("policy") not in ("require_citation", "judge"):
return UNCITED_CLAIM
on_uncited = claims.get("on_uncited") or "degrade"
return UNCITED_CLAIM_WITHHELD if on_uncited == "withhold" else \
UNCITED_CLAIM if on_uncited == "ignore" else UNCITED_CLAIM_COUNTED
UNCITED_CLAIM if on_uncited == "ignore" else \
UNCITED_CLAIM_JUDGED if claims.get("policy") == "judge" else UNCITED_CLAIM_COUNTED
if report.claims == "judge":
return UNCITED_CLAIM_JUDGE_REQUIRED
return UNCITED_CLAIM_REQUIRED if report.claims == "require_citation" else UNCITED_CLAIM
+26 -2
View File
@@ -14,6 +14,9 @@ from app.core.config import settings as app_settings
from app.core.crypto import encrypt, mask
from app.db.base import get_session
from app.db.models import Provider, Setting
from app.engine.judge import JUDGE_DEFAULTS, SETTING_GROUP as JUDGE_GROUP, SPEND_KEY as JUDGE_SPEND
from app.engine.judge import daily_spend, judge_settings
from app.engine.judge import sanitize_settings as judge_values, settings_problem as judge_problem
from app.providers import catalog
from app.providers.factory import ModelSpec, ProviderNotConfigured, _resolve_key, build_chat_model
@@ -434,6 +437,9 @@ DEFAULT_SETTINGS: dict[str, Any] = {
"max_agent_steps": app_settings.max_agent_steps,
"max_run_seconds": app_settings.max_run_seconds,
},
# 结论句裁判(engine/judge.py):证据裁判模型和各项上限,上限存 null 表示不限。不放进 limits:
# 那一组是环境变量定的、只读,这一组是用户自己定的
JUDGE_GROUP: dict(JUDGE_DEFAULTS),
}
@@ -482,7 +488,11 @@ async def get_settings(session: AsyncSession = Depends(get_session)) -> dict[str
# limits 是服务端配置(环境变量),只读:库里可能存着旧版设置页写进去的一份
#(见过 max_agent_steps: 25),它从来不生效,却会盖住真实的上限,界面照着错的数许诺
stored.pop("limits", None)
# 裁判的每日计数是引擎记的账,不是用户设置:设置页整组回写时带着一份旧的,会把今天的花费清零
stored.pop(JUDGE_SPEND, None)
merged = {k: {**v, **(stored.get(k) or {})} for k, v in DEFAULT_SETTINGS.items()}
# 裁判设置按引擎实际生效的样子给:存坏了的项显示默认,存了 null 的上限显示不限
merged[JUDGE_GROUP] = judge_values(stored.get(JUDGE_GROUP))
for key, value in stored.items():
merged.setdefault(key, value)
return merged
@@ -498,10 +508,14 @@ async def put_settings(
) -> dict[str, Any]:
from app.tools.trust import SETTING_KEY as TRUST_KEY
if JUDGE_GROUP in payload.values and (problem := judge_problem(payload.values[JUDGE_GROUP])):
# 先查后写:一组里有一项不对就整个请求不落库,免得存下半组
raise HTTPException(422, problem)
for key, value in payload.values.items():
if key in (TRUST_KEY, "limits"):
if key in (TRUST_KEY, "limits", JUDGE_SPEND):
# 信任三档只走 PUT /api/tools/trust:设置页整组回写时带着一份旧的,
# 会把刚在工具页改的档位冲掉。limits 是环境变量定的,写进库也不生效
# 会把刚在工具页改的档位冲掉。limits 是环境变量定的,写进库也不生效。
# 裁判的每日计数是引擎记的账,只有裁判自己写
continue
row = await session.get(Setting, key)
if row:
@@ -512,6 +526,16 @@ async def put_settings(
return await get_settings(session)
@router.get("/settings/judge/spend")
async def get_judge_spend(session: AsyncSession = Depends(get_session)) -> dict[str, Any]:
"""今天(本地日期)结论句裁判花了多少,设置页放在每日上限旁边。只读:计数只有裁判自己记。
{date, usd, calls, unpriced_calls, daily_max_usd}。unpriced_calls 是价格目录里没有的模型的调用次数,
这些调用估不出金额、没有算进 usd。daily_max_usd 为 null 表示不限。
"""
return {**await daily_spend(), "daily_max_usd": (await judge_settings(session))["daily_max_usd"]}
@router.get("/system")
async def system_info() -> dict[str, Any]:
"""给设置页顶部展示的运行环境概览。"""
+3
View File
@@ -69,6 +69,9 @@ class EventType(StrEnum):
# 报告撰写节点的自查结果:报告文档工件 id、统计、违规清单。落在封存范围内,
# 证据接口靠它找到文档——不去翻不受封存保护的 artifacts 表
REPORT_CHECKED = "report.checked"
# 探索运行里按需裁判的结论句判定:点开哪句才判哪句,追加在封存之后(post_seal),是批注不是证据。
# 封存核对只覆盖 manifest_seq 之前的事件,追加它不影响 verify;封存的报告文档一个字都不改
EVIDENCE_JUDGED = "evidence.judged"
# 其他
LOG = "log"
+138 -27
View File
@@ -23,8 +23,23 @@ from typing import Any, Callable
from pydantic import BaseModel
from app.engine.governance import REPORT_POLICY, exit_text_fields, publish_issues
from app.engine.schema import REPORT_CLAIMS, GraphNode, GraphSpec, NodeType, _ancestors, claims_problem
from app.engine.governance import (
JUDGE_UNLIMITED,
REPORT_POLICY,
REPORT_POLICY_ALSO,
exit_text_fields,
publish_issues,
)
from app.engine.schema import (
CLAIMS_JUDGE,
JUDGE_ON_UNSUPPORTED,
REPORT_CLAIMS,
GraphNode,
GraphSpec,
NodeType,
_ancestors,
claims_problem,
)
#: 被选中的修复按这个顺序应用:先补契约骨架,再补契约里的各项,最后是节点上的开关。
#: 前面的修复可能改变后面的候选(metrics_from 决定 required 从哪几张卡里挑,report_from 决定
@@ -34,7 +49,7 @@ _ORDER = (
"contract.report_from_missing", "contract.report_from_invalid", "governed.report_from_required",
"governed.exit_text_source", "report.metrics_from_invalid",
"contract.required_missing", "contract.strict_off", "contract.claims_ignored", "report.claims_invalid",
"governed.report_policy",
"governed.report_policy", "governed.judge_budget",
"governed.caliber_agent_cite_fields", "governed.caliber_compute_input", "governed.agent_approval_never",
"governed.default_approval_never", "governed.subgraph_unpinned", "governed.supervisor", "contract.not_object",
"governed.text_bypass", "governed.caliber_agent_schema", "governed.caliber_model_input",
@@ -210,7 +225,8 @@ def _plan_exit_fields(ctx: _Ctx, issue: dict[str, Any], node: GraphNode | None)
def _plan_report_policy(ctx: _Ctx, issue: dict[str, Any], node: GraphNode | None) -> dict[str, Any] | None:
"""G3:报告撰写节点写明最严的三项。答案唯一,只往严里改;写了不支持的 claims 由 claims_invalid 管。"""
"""G3:报告撰写节点写明最严的三项。答案唯一,只往严里改;写了不认识的 claims 由 claims_invalid 管。
已经写了更严的写法(claims: judge)的不动——改回 require_citation 就是替人放宽了。"""
if node is None:
return None
writes: dict[str, Any] = {}
@@ -218,7 +234,9 @@ def _plan_report_policy(ctx: _Ctx, issue: dict[str, Any], node: GraphNode | None
for key, want, _ in REPORT_POLICY:
value = node.config.get(key)
effective = ctx.spec.defaults.get(key) if value in (None, "") else value
if effective != want and not (key == "claims" and claims_problem(effective)):
if effective in (want, *REPORT_POLICY_ALSO.get(key, ())):
continue
if not (key == "claims" and claims_problem(effective)):
writes[key], before[key] = want, copy.deepcopy(value)
if not writes:
return None
@@ -230,18 +248,38 @@ def _plan_report_policy(ctx: _Ctx, issue: dict[str, Any], node: GraphNode | None
def _plan_claims(ctx: _Ctx, issue: dict[str, Any], node: GraphNode | None) -> dict[str, Any] | None:
"""claims 写了不支持的值。受管级别只认 require_citation,答案唯一;已发布级别 off 和 require_citation
都行,要人选——写的是 judge(还不支持的模型裁判)时建议 require_citation:它离裁判最近。"""
"""claims 写了不认识的值。受管级别改成 require_citation:确定性的两档里只有它合规,答案唯一(judge 要花钱、
要写预算,不替人开);已发布级别 off 和 require_citation 都行,要人选。"""
if node is None:
return None
before = node.config.get("claims")
if ctx.level == "governed":
return _auto(f"把「{node.title}」的 claims 改成 require_citation:受管出具要求每句结论挂引用"
+ ("(结论句裁判在后续版本支持)" if before == "judge" else ""), "claims", before, "require_citation")
return _auto(f"把「{node.title}」的 claims 改成 require_citation:受管出具要求每句结论挂引用", "claims",
before, "require_citation")
hints = {"off": "结论句不参与判档,和以前一样", "require_citation": "没挂引用的结论句计入缺口,按出具档位降档"}
options = [{"value": v, "label": v, "hint": hints[v]} for v in REPORT_CLAIMS]
return _choice(f"选「{node.title}」的结论句策略(claims)", "claims", options,
default="require_citation" if before == "judge" else None)
return _choice(f"选「{node.title}」的结论句策略(claims)", "claims", options)
def _plan_judge_budget(ctx: _Ctx, issue: dict[str, Any], node: GraphNode | None) -> dict[str, Any] | None:
"""G3:claims: judge 没写每份报告的裁判预算。写多少钱只有作者能定:候选是默认金额和「不限」,不替人选、
不给建议。judge 原来写在图级 defaults 里的,写到节点上时带上 defaults 那一整份——节点写了 judge 就整个
用节点的,只补一个 max_cost_usd 会把 defaults 里选的裁判模型、上限悄悄丢掉。"""
from app.engine.judge import JUDGE_DEFAULTS
if node is None:
return None
judge = node.config.get("judge")
if judge in (None, ""):
judge = ctx.spec.defaults.get("judge")
base = copy.deepcopy(judge) if isinstance(judge, dict) else {}
amount = JUDGE_DEFAULTS["report_max_cost_usd"]
options = [
{"value": amount, "label": f"每份报告最多 ${amount:g}", "hint": "系统默认的每份报告上限;写进节点以后,改设置不影响这份报告"},
{"value": None, "label": "不限(写 null)", "hint": JUDGE_UNLIMITED},
]
return _choice(f"给「{node.title}」的结论句裁判写每份报告的预算(judge.max_cost_usd)", "judge.max_cost_usd",
options, _judge=base)
def _plan_compute_input(ctx: _Ctx, issue: dict[str, Any], node: GraphNode | None) -> dict[str, Any] | None:
@@ -419,6 +457,7 @@ _PLANNERS: dict[str, Callable[[_Ctx, dict[str, Any], GraphNode | None], dict[str
"governed.text_bypass": _plan_text_bypass,
"governed.report_policy": _plan_report_policy,
"report.claims_invalid": _plan_claims,
"governed.judge_budget": _plan_judge_budget,
"governed.caliber_compute_input": _plan_compute_input,
"governed.caliber_agent_cite_fields": _plan_cite_fields,
"governed.caliber_agent_schema": _plan_agent_schema,
@@ -580,12 +619,25 @@ def _edit_exit_fields(graph: dict[str, Any], fix: dict[str, Any], value: Any) ->
return out
def _edit_judge_budget(graph: dict[str, Any], fix: dict[str, Any], value: Any) -> list[tuple[str, str, Any, Any]]:
"""G3 的裁判预算:节点的 judge 写成(原来生效的那份 judge + 选中的 max_cost_usd)。选「不限」写显式的 null,
和没写不是一回事。变更按整份 judge 记:前后都是 null 的 max_cost_usd 在预览里分不出写没写。"""
node = _node(graph, fix["node_id"])
if node is None:
return []
config = _config(node)
after = {**copy.deepcopy(fix.get("_judge") or {}), "max_cost_usd": value}
before = _write(config, "judge", after)
return [(str(node["id"]), "judge", before, copy.deepcopy(after))]
#: 每类问题怎么改图。大多是同一种「按节点字段写值」,改全图默认、改成果字段的各走一格;留着这张表是为了
#: 某类修复要特殊处理时只换它自己的那一格(测试里也借它模拟一条出错的修复规则)
_EDITORS: dict[str, Callable[[dict[str, Any], dict[str, Any], Any], list[tuple[str | None, str, Any, Any]]]] = {
code: (_edit_defaults if code in _GRAPH_FIXES else _edit) for code in _PLANNERS
}
_EDITORS["governed.exit_text_source"] = _edit_exit_fields
_EDITORS["governed.judge_budget"] = _edit_judge_budget
def _handoff(fix: dict[str, Any], value: Any) -> bool:
@@ -811,25 +863,34 @@ def _contract_loosened(name: str, prior: Any, contract: Any) -> list[str]:
if contract.get("cells") is True and prior.get("cells") is not True:
out.append(f"替你打开了「{name}」出具契约的 cells(允许报告直接引用查询单元格),这要你自己决定")
was, now = _claims_rank(prior.get("claims")), _claims_rank(contract.get("claims"))
if was is not None and now is not None and now < was:
out.append(f"把「{name}」出具契约的 claims 往宽里改了(没挂依据的结论句{_CLAIMS_WORDS[was]} → "
f"{_CLAIMS_WORDS[now]})")
if was is not None and now is not None:
looser = [f"{what}{words[a]} → {words[b]}" for (what, words), a, b in zip(_CLAIMS_AXES, was, now) if b < a]
if looser:
out.append(f"把「{name}」出具契约的 claims 往宽里改了({';'.join(looser)})")
return out
#: 契约 claims 的严格程度:不查 < 查了不算缺口(ignore)< 计入缺口降档(degrade)< 不予出具(withhold)
_CLAIMS_WORDS = ("不检查", "只标出来", "计入缺口", "不予出具")
#: 契约 claims 的严格程度分两样,各自从宽到严:没挂依据的结论句(不查 < 查了不算缺口 ignore < 计入缺口降档
#: degrade < 不予出具 withhold),证据不支持的结论句(不裁判 < 裁判、降档 < 裁判、不予出具)。judge 两样都管:
#: 挂依据的要求和 require_citation 一样,另外按裁判的判定判档
_CLAIMS_AXES = (("没挂依据的结论句", ("不检查", "只标出来", "计入缺口", "不予出具")),
("证据不支持的结论句", ("不裁判", "降档出具", "不予出具")))
def _claims_rank(value: Any) -> int | None:
"""契约里 claims 的严格程度,越大越严。judge 这类还不支持的写法不比较(validate 另报)。"""
def _claims_rank(value: Any) -> tuple[int, int] | None:
"""契约里 claims 的严格程度:(没挂依据的, 证据不支持的),各自越大越严,任何一样变小都算放宽。
认不出的写法不比较(出口按缺口处理,不会因此放松)。"""
name = value.get("policy") if isinstance(value, dict) else value
if name in (None, "", "off"):
return 0
if name != "require_citation":
return 0, 0
if name not in ("require_citation", CLAIMS_JUDGE):
return None
on_uncited = value.get("on_uncited") if isinstance(value, dict) else None
return {"ignore": 1, "withhold": 3}.get(on_uncited, 2)
uncited = {"ignore": 1, "withhold": 3}.get(on_uncited, 2)
if name != CLAIMS_JUDGE:
return uncited, 0
on_unsupported = value.get("on_unsupported") if isinstance(value, dict) else None
return uncited, 2 if on_unsupported == "withhold" else 1
def forbidden_changes(before: dict[str, Any], after: dict[str, Any],
@@ -844,7 +905,8 @@ def forbidden_changes(before: dict[str, Any], after: dict[str, Any],
- 审批策略往宽里改(每次调用都审批 > 仅危险工具需要审批 > 全部自动放行)、删掉写明的审批策略,
把审批写成(或新加一个)「全部自动放行」;全图默认同理
- 取消子工作流钉住的版本
- 报告撰写节点的 numbers、on_violation、claims 往宽里改;关掉 agent 的 cite_fields
- 报告撰写节点的 numbers、on_violation、claims 往宽里改(claims 改成认不出的写法也算),claims 一直是
judge 时 judge.on_unsupported 从 withhold 退回去;关掉 agent 的 cite_fields
- 替人把沙箱代码标成取数(evidence_role: source):它在取数还是在算只有作者知道。chosen 里的
(节点 id, 字段) 是人在 choice 里亲手选的,不算
- 改发布级别——级别不在图里,修复接口也从不写工作流的 status,所以只可能以一个不认识的
@@ -928,12 +990,17 @@ def forbidden_changes(before: dict[str, Any], after: dict[str, Any],
#: 报告撰写节点三项设置从严到宽的次序:没写时运行时的缺省值(on_violation 没写时探索运行按 flag)
_REPORT_STRICT = {"numbers": ("strict", "strict"), "on_violation": ("fail", "flag"),
"claims": ("require_citation", "off")}
#: claims 从宽到严:off < require_citation < judge(在挂引用之外再请模型逐句裁判,不支持的按 on_unsupported
#: 判档)。不在这里的写法不比较(validate 报 report.claims_invalid)
_CLAIMS_ORDER = ("off", "require_citation", CLAIMS_JUDGE)
def _report_loosened(name: str, prev: dict[str, Any], cfg: dict[str, Any], old_defaults: Any,
new_defaults: Any) -> list[str]:
"""报告撰写节点的三项设置有没有往宽里改:原来(按节点、再按全图默认)是最严的值,改完不是了。
claims 改成 judge 不算放宽——那是还不支持的更严的写法,validate 会报。"""
"""报告撰写节点的三项设置有没有往宽里改:numbers、on_violation 原来(按节点、再按全图默认)是最严的值,
改完不是了;claims 按 off < require_citation < judge 排,往下走一格也算(require_citation → judge 是收紧),
从 require_citation、judge 改成认不出的写法也算(原来就认不出的不比较)。claims 前后都是 judge 的,
再看 judge.on_unsupported 有没有从 withhold 退回去。"""
def effective(config: dict[str, Any], defaults: Any, key: str, fallback: str) -> Any:
value = config.get(key)
if value in (None, ""):
@@ -943,12 +1010,56 @@ def _report_loosened(name: str, prev: dict[str, Any], cfg: dict[str, Any], old_d
out = []
for key, (strict, fallback) in _REPORT_STRICT.items():
was, now = effective(prev, old_defaults, key, fallback), effective(cfg, new_defaults, key, fallback)
if was == strict and now != strict and not (key == "claims" and now == "judge"):
out.append(f"把「{name}」的 {key} 从 {strict} 放宽成了 {now}"
+ ("(去掉了写明的值,运行时退回缺省)" if cfg.get(key) in (None, "") else ""))
if key == "claims":
if was not in _CLAIMS_ORDER or (was == "off" and now not in _CLAIMS_ORDER):
continue
if now not in _CLAIMS_ORDER:
out.append(f"把「{name}」的 claims 从 {was} 改成了不认识的 {now}")
continue
if _CLAIMS_ORDER.index(now) >= _CLAIMS_ORDER.index(was):
continue
elif not (was == strict and now != strict):
continue
out.append(f"把「{name}」的 {key} 从 {was} 放宽成了 {now}"
+ ("(去掉了写明的值,运行时退回缺省)" if cfg.get(key) in (None, "") else ""))
if effective(prev, old_defaults, "claims", "off") == effective(cfg, new_defaults, "claims", "off") \
== CLAIMS_JUDGE and (why := _judge_loosened(name, prev, cfg, old_defaults, new_defaults)):
out.append(why)
return out
def _judge_loosened(name: str, prev: dict[str, Any], cfg: dict[str, Any], old_defaults: Any,
new_defaults: Any) -> str | None:
"""claims 一直是 judge 时,judge.on_unsupported 从 withhold(证据不支持的结论句不予出具)退回 degrade
(只降档)。judge 和运行时(report.report_judge)同一个取法:节点上写了就整个用节点的,没写取图级
defaults;on_unsupported 没写、写了认不出的都按 degrade。所以补预算时把 judge 整个换成
{max_cost_usd: …}、节点另写一份不带 withhold 的 judge 盖掉全图默认,都算放宽。"""
def block(config: dict[str, Any], defaults: Any) -> tuple[Any, bool]:
own = config.get("judge")
if own not in (None, ""):
return own, False
return (defaults if isinstance(defaults, dict) else {}).get("judge"), True
def written(judge: Any) -> Any:
return judge.get("on_unsupported") if isinstance(judge, dict) else None
(old_block, old_inherited), (new_block, new_inherited) = block(prev, old_defaults), block(cfg, new_defaults)
if written(old_block) != "withhold" or (raw := written(new_block)) == "withhold":
return None
was = "withhold" + ("(跟随全图默认)" if old_inherited else "")
if raw in JUDGE_ON_UNSUPPORTED:
now = f"{raw}" + ("(跟随全图默认)" if new_inherited else "")
elif raw not in (None, ""):
now = f"认不出的 {raw}(运行时按 degrade)"
elif new_inherited:
now = "degrade(跟随全图默认,那里没写 on_unsupported,运行时退回缺省)"
elif old_inherited:
now = "degrade(节点上另写了 judge,整个用节点的、不再跟随全图默认,没写 on_unsupported 就退回缺省)"
else:
now = "degrade(去掉了写明的值,运行时退回缺省)"
return f"把「{name}」的 judge.on_unsupported 从 {was}{'' if old_inherited else ' '}放宽成了 {now}"
def _type_word(node_type: Any, type_label: Callable[[Any], str]) -> str:
try:
return f"「{type_label(NodeType(node_type))}」"
+40 -6
View File
@@ -8,12 +8,15 @@ from sqlalchemy import func, select
from sqlalchemy.ext.asyncio import AsyncSession
from app.engine.schema import (
CLAIMS_JUDGE,
GraphNode,
GraphSpec,
NodeType,
ValidationResult,
_caliber_inputs,
claims_problem,
contract_needs_metrics,
judge_problems,
type_label,
validate_graph,
)
@@ -158,7 +161,7 @@ def _lint_contract(
if isinstance(sources, str):
sources = [sources]
if not sources:
if not sources and contract_needs_metrics(contract):
flag("出具契约没有声明 metrics_from(指标来自哪个「口径卡」节点),叙述里的数字无从回指",
code="contract.metrics_from_missing", node_id=node_id, hard=True, field="contract.metrics_from")
for src in sources:
@@ -177,7 +180,11 @@ def _lint_contract(
"(叙述),数字回指校验不会执行", code="contract.report_from_missing", node_id=node_id, hard=True,
field="contract.report_from")
if strict and not (contract.get("required") or []):
# 引用模式下整张图没有口径卡(只引查询单元格的问数据图):没有指标可列,strict + cells 已经保证
# 每个数点得开出处、对不上就不予出具(用户 2026-09-29 拍板)。有口径卡的照旧要列必需指标——
# 按图里有没有口径卡判,不按契约写没写 metrics_from,免得不写 metrics_from 就绕过去
cells_only = contract.get("report_from") not in (None, "") and not metric_nodes
if strict and not (contract.get("required") or []) and not cells_only:
flag(
"受管模板的出具契约必须声明 required(必需指标),否则「不予出具」这一档永远触发不了",
code="contract.required_missing", node_id=node_id, hard=True, field="contract.required",
@@ -202,7 +209,7 @@ def _lint_contract(
#
# G1 带契约的出口核对报告撰写节点的文档,成果字段的文字只能来自报告撰写节点
# G2 模型写的文字不绕过报告撰写节点直接流进出口(从图上追)
# G3 报告撰写节点写明 numbers: strict、on_violation: fail、claims: require_citation
# G3 报告撰写节点写明 numbers: strict、on_violation: fail、claims: require_citation(或 judge,带上预算)
# G4 口径卡的输入不来自计算角色的沙箱代码
# G5 给口径卡供数的 agent 配 output_schema 并开 cite_fields(模型调用节点不能给口径卡供数)
#
@@ -216,6 +223,11 @@ REPORT_POLICY = (
("on_violation", "fail", "重写之后还违规就让节点失败,不带着问题出具"),
("claims", "require_citation", "没挂引用的结论句计入缺口,按出具档位降档"),
)
#: 除了 REPORT_POLICY 里写的那个值,还认的更严的写法。claims: judge 在挂引用之外再请另一个模型按证据逐句
#: 裁判,不支持的按 on_unsupported 判档——比 require_citation 严,但要写预算(见 judge_budget_missing)
REPORT_POLICY_ALSO: dict[str, tuple[str, ...]] = {"claims": (CLAIMS_JUDGE,)}
#: 不限的写法:judge.max_cost_usd 写 null。说给人听的时候照 engine/judge 设置项的说法
JUDGE_UNLIMITED = "不设上限,费用只受句数、时长上限和每日上限约束"
#: 产出本身就是证据、或者由系统确定地算出来的节点。成果字段取它们的值,不算「别的节点写的字」
_TRACEABLE = (NodeType.REPORT, NodeType.METRICS, NodeType.INPUT, NodeType.TOOL, NodeType.RETRIEVE)
@@ -289,6 +301,21 @@ def _effective(spec: GraphSpec, node: GraphNode, key: str, default: Any = None)
return spec.defaults.get(key, default) if value in (None, "") else value
def judge_budget_missing(spec: GraphSpec, node: GraphNode) -> bool:
"""claims 是 judge、生效的 judge 配置里却没有 max_cost_usd 这个键(G3)。
写 null 是显式的不限,算写了;没写的键运行时取设置里的默认——受管出具要作者自己定。judge 本身写坏了
(不是对象、max_cost_usd 写成了别的东西)由 validate 的 report.judge_invalid 报,这里不重复。
judge 和 claims 一样按运行时的规则取:节点上写了就整个用节点的,不和图级 defaults 拼。
"""
if _effective(spec, node, "claims") != CLAIMS_JUDGE:
return False
judge = _effective(spec, node, "judge")
if any(field in ("judge", "judge.max_cost_usd") for field, _ in judge_problems(judge)):
return False
return not (isinstance(judge, dict) and "max_cost_usd" in judge)
class _Graph:
"""一张图上反复要用的查找:谁产出了某个值、最后一条消息是谁写的。"""
@@ -489,13 +516,20 @@ def _lint_evidence(spec: GraphSpec, flag: Callable[..., None], *, strict: bool)
for key, want, why in REPORT_POLICY:
value = _effective(spec, node, key)
if key == "claims" and claims_problem(value):
continue # 写了不支持的值(比如 judge),validate 的 report.claims_invalid 报
if value != want:
continue # 写了不认识的值,validate 的 report.claims_invalid 报
also = REPORT_POLICY_ALSO.get(key, ())
if value != want and value not in also:
now = "现在没写" if value in (None, "") else f"现在是 {value}"
loose.append((key, f"{key}: {want}({why};{now})"))
more = "".join(f",或者更严的 {v}" for v in also)
loose.append((key, f"{key}: {want}({why}{more};{now})"))
if loose:
flag(f"{_who(node)}要写明 {';'.join(text for _, text in loose)}。受管模板的报告撰写节点按最严的规则自查",
code="governed.report_policy", node_id=node.id, hard=True, field=loose[0][0])
if judge_budget_missing(spec, node):
flag(f"{_who(node)}的结论句由模型裁判(claims: judge),却没有写每份报告的裁判预算:在 judge 里写 "
"max_cost_usd——写一个金额(美元),或者显式写 null 表示不限(" + JUDGE_UNLIMITED + ")。"
"受管出具不让裁判的花费悄悄按设置里的默认值走",
code="governed.judge_budget", node_id=node.id, hard=True, field="judge.max_cost_usd")
# G4、G5
feeders: dict[str, tuple[GraphNode, list[str]]] = {}
File diff suppressed because it is too large Load Diff
+72 -17
View File
@@ -237,6 +237,7 @@ def _apply_contract(
strict=bool(contract.get("strict")),
gaps=gaps,
unresolved=cited.get("unresolved"),
unsupported=cited.get("unsupported"),
uncited_claims=cited.get("uncited"),
claims_policy=cited.get("claims_policy"),
)
@@ -298,7 +299,7 @@ def _check_citations(
reports: list[dict[str, Any]], *, cells: bool = True, governed: bool = False,
) -> dict[str, Any]:
"""独立复核报告文档:{matched, unmatched, unresolved, gaps, doc_artifact, stats, claims_policy, uncited,
claims, entities}。
unsupported, claims, entities}。
不信报告节点自己的统计:文档按 id 从工件库取回(取回时复验哈希),目录按报告节点
同一套参数从状态里重建(不用文档里存的那份),每个带引用的片段重新解析、重新渲染、
@@ -313,7 +314,7 @@ def _check_citations(
out: dict[str, Any] = {"matched": [], "unmatched": [], "unresolved": [], "gaps": [],
"doc_artifact": None, "stats": None, "claims_policy": None, "uncited": None,
"claims": None, "entities": None}
"unsupported": None, "claims": None, "entities": None}
gaps = out["gaps"]
spec = ctx.run.spec
node = spec.node_map().get(report_from)
@@ -403,7 +404,7 @@ def _check_citations(
"span": [start, end], "eid": cite["eid"]})
_claims(out, contract, payload, evidence_on=ledger_enabled(ctx.run), governed=governed,
uncited=checked.get("uncited") or [])
uncited=checked.get("uncited") or [], doc=doc, title=node.title)
mine = next((r for r in reports if r["node"].id == report_from), None)
edited = mine["edited"] if mine else []
@@ -416,20 +417,29 @@ def _check_citations(
#: 没挂依据的结论句怎么处置,从松到严。认不出的写法按 degrade 算:写错一个词不能让缺口悄悄消失
_ON_UNCITED = ("ignore", "degrade", "withhold")
#: 证据不支持的结论句怎么处置(claims: judge),从松到严。认不出的写法同样按 degrade 算
_ON_UNSUPPORTED = ("degrade", "withhold")
#: 要求结论句挂依据的策略:judge 在这一点上和 require_citation 一样严,另外再按裁判的判定判档
_CITING = ("require_citation", "judge")
def _claims(out: dict[str, Any], contract: dict[str, Any], payload: dict[str, Any], *, evidence_on: bool,
governed: bool = False, uncited: list[dict[str, Any]]) -> None:
"""结论句策略:没挂依据的结论句按 claims_policy 交给 decide_tier 判档。
governed: bool = False, uncited: list[dict[str, Any]], doc: dict[str, Any] | None = None,
title: str = "") -> None:
"""结论句策略:没挂依据的、裁判判为证据不支持的结论句按 claims_policy 交给 decide_tier 判档。
两处可以要求挂依据:报告节点写作时记下的 claims(节点产出里的,写作提示按它要求过),和契约自己写的
require_citation(对象形式可以另写 on_uncited)。两处都写了取更严的那个——契约只能收紧、不能放松:
require_citation / judge(对象形式可以另写 on_uncited)。两处都写了取更严的那个——契约只能收紧、不能放松:
报告节点要求了,契约写 ignore 或 off 也照旧计入缺口,写 withhold 就收紧成不予出具。受管级别的正式运行
里 ignore 一律当 degrade:没有哪道门禁查契约里的 claims,不能让它把用户要的底线拉低。
报告节点写的是 judge 时,判定在文档里(按哈希取回的那份:判定是模型给的,出口复算不了,只能信封存的
文档)。正式运行在节点里判过:证据不支持的按 on_unsupported 判档(报告节点和契约取更严的),没判完的
(裁判没跑成、触顶)把裁判摘要里的缺口照抄进来——不能判完整出具,也不当成「不支持」。探索运行按需裁判,
这里只标注。契约要 judge、报告节点却没开的,照实记缺口:出口替不了报告节点去裁判。
数的是这里重新核对出来的 uncited,不信文档里记的 cites。升级前写的报告(节点产出里没有 claims 这个键,
包括跨着升级还没跑完的运行)一律不管,契约写了也照旧记「这一版还不支持」。judge(裁判模型)是后续版本
的事,契约写了照实记缺口。
包括跨着升级还没跑完的运行)一律不管,契约写了也照旧记「这一版还不支持」。
"""
written = payload.get("claims") if isinstance(payload, dict) else None
# 报告节点升级后才在产出里记 claims(off 也记):没有这个键就是升级前写的报告
@@ -437,23 +447,66 @@ def _claims(out: dict[str, Any], contract: dict[str, Any], payload: dict[str, An
declared = contract.get("claims")
name = declared.get("policy") if isinstance(declared, dict) else declared
asks: list[str] = []
if upgraded and written == "require_citation":
if upgraded and written in _CITING:
asks.append("degrade")
if upgraded and name == "require_citation":
if upgraded and name in _CITING:
wanted = declared.get("on_uncited") if isinstance(declared, dict) else None
asks.append(wanted if wanted in _ON_UNCITED else "degrade")
elif name not in (None, "", "off"):
out["gaps"].append("契约声明了结论句检查(claims),这一版还不支持,结论句没有核对")
judged = upgraded and written == "judge"
if upgraded and name == "judge" and not judged:
out["gaps"].append(f"契约要求结论句由模型裁判(claims: judge),报告「{title}」没有开 claims: judge,"
"结论句没有裁判")
if not asks:
return
on_uncited = max(asks, key=_ON_UNCITED.index)
if governed and on_uncited == "ignore":
on_uncited = "degrade"
policy: Any = "require_citation" if on_uncited == "degrade" else \
{"policy": "require_citation", "on_uncited": on_uncited}
out["claims_policy"], out["uncited"] = policy, uncited
out["claims"] = {"policy": "require_citation", "uncited_claims": len(uncited), "uncited": uncited[:_LISTED],
**({"on_uncited": on_uncited} if on_uncited != "degrade" else {})}
extra = {"on_uncited": on_uncited} if on_uncited != "degrade" else {}
if not judged:
policy: Any = "require_citation" if on_uncited == "degrade" else \
{"policy": "require_citation", "on_uncited": on_uncited}
out["claims_policy"], out["uncited"] = policy, uncited
out["claims"] = {"policy": "require_citation", "uncited_claims": len(uncited), "uncited": uncited[:_LISTED],
**extra}
return
from app.engine.evidence import iter_units
summary = (doc or {}).get("judge")
if not isinstance(summary, dict):
# 报告节点开了 judge、文档里却没有裁判摘要(不该发生):当作没判,不能判完整出具
summary = {"mode": "inline", "counts": {}, "unjudged": {}, "limits_hit": [], "complete": False,
"gaps": ["开了 claims: judge,文档里却没有裁判结果,结论句没有裁判"]}
asked = [summary.get("on_unsupported")]
if name == "judge" and isinstance(declared, dict):
asked.append(declared.get("on_unsupported"))
on_unsupported = max((a if a in _ON_UNSUPPORTED else "degrade" for a in asked), key=_ON_UNSUPPORTED.index)
inline = summary.get("mode") == "inline"
flagged: dict[str, list[dict[str, Any]]] = {"unsupported": [], "partial": []}
if inline:
# 探索运行按需裁判的判定不在文档里(在封存之后追加的 evidence.judged 事件里),这里只数正式运行的
markdown = str((doc or {}).get("markdown") or "")
for _, unit in iter_units(doc or {}):
verdict = unit.get("verdict")
status = verdict.get("status") if isinstance(verdict, dict) else None
if status in flagged:
span = unit.get("span") or [0, 0]
flagged[status].append({"unit": unit.get("id"), "span": span, "text": markdown[span[0]:span[1]],
"rationale": verdict.get("rationale") or ""})
out["gaps"].extend(f"报告「{title}」{gap}" for gap in summary.get("gaps") or [])
counts = {k: int((summary.get("counts") or {}).get(k) or 0)
for k in ("supported", "partial", "unsupported", "not_a_claim", "unjudged")}
out["claims_policy"] = {"policy": "judge", "on_unsupported": on_unsupported, "on_uncited": on_uncited}
out["uncited"], out["unsupported"] = uncited, flagged["unsupported"]
out["claims"] = {
"policy": "judge", "uncited_claims": len(uncited), "uncited": uncited[:_LISTED], **extra,
"on_unsupported": on_unsupported, "mode": "inline" if inline else "on_demand", "counts": counts,
"unsupported": flagged["unsupported"][:_LISTED], "partial": flagged["partial"][:_LISTED],
"unjudged": dict(summary.get("unjudged") or {}), "limits_hit": list(summary.get("limits_hit") or []),
"complete": bool(summary.get("complete")), "model": summary.get("model"),
}
def _declare_citations(
@@ -497,8 +550,10 @@ def _declare_citations(
metrics_checked=len(metrics),
matched_numbers=len(cited["matched"]),
matched=cited["matched"][:50],
# on_uncited 只在不是默认的 degrade 时才有(收紧成 withhold、探索运行的契约写了 ignore)
**({"claims": {k: cited["claims"][k] for k in ("policy", "uncited_claims", "on_uncited")
# on_uncited 只在不是默认的 degrade 时才有(收紧成 withhold、探索运行的契约写了 ignore);
# judge 另带不支持时怎么判档、在哪判的和各判定的句数(名单只在出具声明里)
**({"claims": {k: cited["claims"][k] for k in ("policy", "uncited_claims", "on_uncited", "on_unsupported",
"mode", "counts")
if k in cited["claims"]}} if cited.get("claims") else {}),
# 计数取复核的统计(声明里的名单最多列 _LISTED 个)
**({"entities": {"unknown": (cited["stats"] or {}).get("unknown_entities", len(entities["unknown"])),
+224 -11
View File
@@ -15,20 +15,32 @@
写了个哪里都找不到的名字)只标注:不重写、不失败,受管级别的正式出具按缺口降档;表结构快照
不全时核对不了的名字(unverified_entity)只标注。entities: off 整层不管表名、字段名。
结论句策略 claims:off(默认)不管;require_citation 在写作提示里要求结论句挂依据(自动链接的
名字不算依据),没挂的计入缺口、出口降档;judge(模型裁判)是后续版本的事,现在写了直接报错。
名字不算依据),没挂的计入缺口、出口降档;judge 另外请一个模型按证据逐句判断(engine/judge.py):
- 正式运行在节点里判:写完、核对完之后判,判定写进文档的 unit.verdict,随文档一起封存。判定只标注、
不改写正文;裁判没跑成、触顶了照样产出,没判的记未裁判并留下缺口,出口按 on_unsupported 判档
- 探索运行不在节点里花这笔钱:候选句标「未裁判 · 按需」,点开哪句再判哪句(证据接口)
- rewrite_once(默认关):证据不支持的句子连同理由交回写作者只改这几句,再判一次,最多一轮;
改写稿冒出原稿没有的违规(只标注的可疑实体也算)就不采用
- 裁判调用包在 @task 里,排在节点所有已有调用之后:「接着跑」时直接取 checkpoint 里的判定,
不重复调用、不重复计费。写作调用不进 checkpoint,接着跑时会重写一稿——所以判定带着输入指纹,
对不上这一稿的不用
出口契约复核(io.py)用 report_catalog 按同一套参数重建目录、独立重算,不信这里的自查。
"""
from __future__ import annotations
import time
from collections import Counter
from typing import Any
from langchain_core.messages import AIMessage, BaseMessage, HumanMessage, SystemMessage
from langgraph.func import task
from app.core.errors import describe_exception
from app.core.events import EventType
from app.db.base import SessionLocal
from app.engine import judge as judging
from app.engine.context import NodeContext, NodeError
from app.engine.evidence import (
CELLS_REASON,
@@ -43,7 +55,17 @@ from app.engine.evidence import (
ledger_enabled,
)
from app.engine.nodes.llm import _invoke_streaming, _model_spec, _report_call
from app.engine.schema import REPORT_CLAIMS, GraphNode, GraphSpec, NodeType, _ancestors, claims_problem
from app.engine.schema import (
CLAIMS_JUDGE,
CLAIMS_VALUES,
JUDGE_ON_UNSUPPORTED,
GraphNode,
GraphSpec,
NodeType,
_ancestors,
claims_problem,
judge_problems,
)
from app.engine.state import GraphState, message_text, thinking_text
from app.engine.toolcalls import leaked_markup, markup_warning
from app.providers.factory import ProviderNotConfigured, get_chat_model
@@ -52,8 +74,8 @@ ON_VIOLATION = ("fail", "flag")
NUMBERS = ("strict", "off")
#: 实体层:link(默认)核对表名、字段名并自动链接;off 整层不管(目录里不收表和字段)
ENTITIES = ("link", "off")
#: 结论句策略,和画布校验同一份(schema.REPORT_CLAIMS);judge 是后续版本的事
CLAIMS = REPORT_CLAIMS
#: 结论句策略,和画布校验同一份(schema.CLAIMS_VALUES):off / require_citation / judge
CLAIMS = CLAIMS_VALUES
#: 只标注、不让写作者重写也不判失败的违规:可疑实体(受管的正式出具按缺口降档)、表结构快照不全时
#: 核对不了的名字(出口只标注)
SOFT = frozenset({"unknown_entity", "unverified_entity"})
@@ -67,6 +89,8 @@ MARKUP_NUDGE = ("你上一条回复把工具调用写成了文字。报告撰写
"证据目录里,直接用引用标记写报告正文。")
MARKUP_ERROR = ("模型输出了工具调用的原始标记,而报告撰写节点不调用工具——数据已经在证据目录里。"
"换一个模型再试")
#: 接着跑时 checkpoint 里缓存的判定最多翻几条去找这一稿的(每条都是先前某一稿的,对不上就往后翻)
_STALE_LIMIT = 16
# --------------------------------------------------------------------------
@@ -120,14 +144,26 @@ def report_entities(spec: GraphSpec, node: GraphNode) -> str:
def report_claims(spec: GraphSpec, node: GraphNode) -> str:
"""报告节点的结论句策略:off / require_citation。出口判档按它(和节点执行时同一个取值规则)。
"""报告节点的结论句策略:off / require_citation / judge。出口判档按它(和节点执行时同一个取值规则)。
写错的值(包括 judge)画布校验就拦了,执行时节点也会报错,这里只把没写当成 off。
写错的值画布校验就拦了,执行时节点也会报错,这里只把没写当成 off。
"""
value = _setting(spec, node, "claims", "off")
return value if isinstance(value, str) and value in CLAIMS else "off"
def report_judge(spec: GraphSpec, node: GraphNode) -> dict[str, Any]:
"""claims: judge 的子配置:节点上的 judge(没写取图级 defaults),写了的键原样保留——上限写 null 就是
不限,没写的键运行时取设置里的默认(judge.report_budget)。rewrite_once、on_unsupported 补上缺省
(false、degrade):出口判档读 on_unsupported。写错的值画布校验就拦了,执行时节点也会报错。
"""
raw = _setting(spec, node, "judge")
raw = raw if isinstance(raw, dict) else {}
return {**raw, "rewrite_once": raw.get("rewrite_once") is True,
"on_unsupported": raw.get("on_unsupported") if raw.get("on_unsupported") in JUDGE_ON_UNSUPPORTED
else "degrade"}
def report_allowance(spec: GraphSpec, node: GraphNode) -> list[Any]:
"""报告自查放行的字面量:出口契约 report_from 指向它时,契约的 allow_numbers。
@@ -209,13 +245,18 @@ class _RenderedStream:
self._ctx.emit(EventType.LLM_TOKEN, delta=tail)
async def _default_on_violation(run_id: str) -> str:
"""探索运行默认标注、照常产出;正式运行默认失败——封存的发布版不能带着裸数字出具。"""
async def _run_class(run_id: str) -> str | None:
"""按运行记录上的 run_class 取:子图、审批之后续跑都是另起的执行上下文,只有运行记录靠得住。"""
from app.db.models import Run
async with SessionLocal() as session:
run = await session.get(Run, run_id)
return "fail" if run is not None and run.run_class == "formal" else "flag"
return run.run_class if run is not None else None
async def _default_on_violation(run_id: str) -> str:
"""探索运行默认标注、照常产出;正式运行默认失败——封存的发布版不能带着裸数字出具。"""
return "fail" if await _run_class(run_id) == "formal" else "flag"
def _options(ctx: NodeContext) -> tuple[str, int]:
@@ -225,6 +266,8 @@ def _options(ctx: NodeContext) -> tuple[str, int]:
# 画布校验也拦这两项;执行时再守一道,报的是同一句话,免得写错的值被悄悄当成默认
if problem := claims_problem(ctx.cfg("claims")):
raise NodeError(ctx.node.id, problem)
if ctx.cfg("claims") == CLAIMS_JUDGE and (problems := judge_problems(ctx.cfg("judge"))):
raise NodeError(ctx.node.id, ";".join(message for _, message in problems))
entities = ctx.cfg("entities", "link") or "link"
if entities not in ENTITIES:
raise NodeError(ctx.node.id, f"entities 只能是 {' / '.join(ENTITIES)},写的是 {entities!r}")
@@ -248,6 +291,29 @@ def _blocking(violations: list[dict[str, Any]], numbers: str) -> list[dict[str,
and not (numbers == "off" and v.get("code") == "uncited_number")]
def _violation_key(violation: dict[str, Any]) -> tuple[str, str]:
"""认一处违规:(code, 点名的字)。点名的字按 text、ref、message 的顺序取第一个有的。"""
named = violation.get("text") or violation.get("ref") or violation.get("message") or ""
return str(violation.get("code") or ""), str(named)
def _new_violations(before: list[dict[str, Any]], after: list[dict[str, Any]]) -> list[dict[str, Any]]:
"""after 里 before 没有的违规,按 (code, 点名的字) 逐个抵消(同一个裸数字多写一处也算新的)。
全部违规都比,包括只标注、不让重写的那几类(SOFT):改写稿冒出一个编造的表名,受管的正式出具就会
因此降档;去掉一处、又冒出另一处,总数没变,冒出来的也还是新的。
"""
left = Counter(_violation_key(v) for v in before)
fresh: list[dict[str, Any]] = []
for violation in after:
key = _violation_key(violation)
if left[key] > 0:
left[key] -= 1
else:
fresh.append(violation)
return fresh
def _named(violations: list[dict[str, Any]], limit: int = 3) -> str:
"""违规里点名的那几个字:「45678」「m:nope」。"""
names = [str(v.get("text") or v.get("ref") or "") for v in violations]
@@ -269,7 +335,10 @@ def _messages(ctx: NodeContext, state: GraphState, catalog: dict[str, Any], cell
system = ctx.render_str(ctx.cfg("system", ""), state)
instructions = ctx.render_str(ctx.cfg("instructions", ""), state).strip() \
or "根据下面的证据写一份简洁的报告,先总后分。"
rules = f"{MARKER_RULES}\n{CLAIMS_RULE}" if claims == "require_citation" else MARKER_RULES
rules = MARKER_RULES
if claims in ("require_citation", CLAIMS_JUDGE):
# 裁判只看写作者挂的依据:要判,就更得每句都挂上
rules = f"{rules}\n{CLAIMS_RULE}" + (f"\n{judging.JUDGE_RULE}" if claims == CLAIMS_JUDGE else "")
return [
SystemMessage(content="\n\n".join(p for p in (system, ROLE) if p)),
HumanMessage(content=(
@@ -362,11 +431,26 @@ async def run_report(state: GraphState, ctx: NodeContext) -> dict[str, Any]:
response, raw = await draft(messages)
doc = compose(raw)
judgement: dict[str, Any] | None = None
if claims == CLAIMS_JUDGE:
judge_cfg = report_judge(spec, ctx.node)
if await _run_class(ctx.run.run_id) != "formal":
# 探索运行默认按需裁判:点开哪句才判哪句,节点里不花这笔钱
judgement = judging.on_demand(doc, on_unsupported=judge_cfg["on_unsupported"],
rewrite_once=judge_cfg["rewrite_once"])
elif not (blocking and on_violation == "fail"):
# 反正要判失败的报告不值得再花钱判(节点紧接着就报错)
doc, response, blocking, judgement, usages = await _judge_inline(
ctx, doc, catalog, judge_cfg, model_id=model_id, messages=[*messages, response],
draft=draft, compose=compose, numbers=numbers, blocking=blocking)
spent.extend(usages)
doc_artifact = await _store(doc, ctx)
ctx.emit(EventType.REPORT_CHECKED, doc_artifact=doc_artifact, ok=not doc["violations"],
stats=doc["stats"], violations=doc["violations"][:20], repairs=repairs,
on_violation=on_violation, failed=bool(blocking) and on_violation == "fail",
**({"claims": claims} if evidence_on else {}))
**({"claims": claims} if evidence_on else {}),
**({"judge": judgement} if judgement is not None else {}))
if blocking and on_violation == "fail":
head = ";".join(v["message"] for v in blocking[:5])
more = f";…另有 {len(blocking) - 5} 处" if len(blocking) > 5 else ""
@@ -388,6 +472,7 @@ async def run_report(state: GraphState, ctx: NodeContext) -> dict[str, Any]:
"model": model_id,
"duration_ms": int((time.perf_counter() - started) * 1000),
**({"claims": claims} if evidence_on else {}),
**({"judge": judgement} if judgement is not None else {}),
}
updates: dict[str, Any] = {"nodes": {ctx.node.id: result}, "usage": usage}
if ctx.cfg("emit_message", True):
@@ -396,3 +481,131 @@ async def run_report(state: GraphState, ctx: NodeContext) -> dict[str, Any]:
if var_name:
updates["vars"] = {var_name: text}
return updates
# --------------------------------------------------------------------------
# claims: judge——正式运行在节点里裁判
# --------------------------------------------------------------------------
def _rewrite_request(sentences: list[tuple[str, str]]) -> str:
listed = "\n".join(f"{i}.「{text}」——{why or '证据不支持这句话'}" for i, (text, why) in enumerate(sentences, 1))
return (
"另一个模型按证据逐句核对了你的报告,下面这几句它认为证据不支持:\n"
f"{listed}\n\n"
"请只改这几句:补上证据目录里真有的依据 [[see:…]],或者把说法收窄到证据能支持的程度,或者删掉;"
"其余部分一字不改。数字仍然只能用引用标记写。不要解释改了什么,只输出改好的整篇报告正文。"
)
def _merge(first: dict[str, Any], second: dict[str, Any], budget: judging.Budget) -> dict[str, Any]:
"""改写前后两轮裁判合成一份:判定、句数、缺口按改写后这一稿,花费、调用、用时两轮相加。"""
merged = dict(second)
for key in ("cost_usd", "estimated_usd"):
merged[key] = round(first[key] + second[key], 6)
for key in ("calls", "duration_ms"):
merged[key] = first[key] + second[key]
merged["notes"] = list(dict.fromkeys([*first["notes"], *second["notes"]]))
merged["budget"] = budget.as_dict()
merged["model"] = second.get("model") or first.get("model")
merged["priced"] = first.get("priced")
return merged
async def _judge_inline(ctx: NodeContext, doc: dict[str, Any], catalog: dict[str, Any], judge_cfg: dict[str, Any], *,
model_id: str, messages: list[BaseMessage], draft: Any, compose: Any, numbers: str,
blocking: list[dict[str, Any]]
) -> tuple[dict[str, Any], BaseMessage, list[dict[str, Any]], dict[str, Any], list[dict[str, Any]]]:
"""写完、核对完之后请裁判判一遍,判定写进文档。返回 (文档, 写作者最后一稿, 要拦的违规, 裁判摘要, 用量)。
messages 是写作的对话、最后一条是写作者这一稿。rewrite_once 时把证据不支持的句子交回去只改这几句,
再判一次(没改的句子直接用第一次的判定),最多一轮;改写稿冒出原稿没有的违规就不采用,保留原稿。
摘要里的 rewrite = {units, sentences, applied, reason, changed}:
- sentences:交回去的句子(原稿里的原文)
- units:这几句在封存文档里的编号。没采用时封存的是原稿,就是原稿的编号;采用了封存的是改写稿,
编号按改写稿重排,按原文认:原句原样留在改写稿里的给新编号,改掉了的是 null
- changed:采用时改写稿里新出来的句子(原稿里没有这段原文)在封存文档里的编号;没采用是 []
"""
from app.engine.replay import _in_graph
async with SessionLocal() as session:
settings = await judging.judge_settings(session)
judge_spec = await judging.judge_model_spec(session, judge_cfg, settings=settings)
budget = judging.report_budget(settings, judge_cfg)
masks = await judging.source_masks(catalog)
@task
async def judge_step(request: judging.JudgeRequest, key: str, limits: judging.Budget, known: dict[str, Any],
first: bool) -> dict[str, Any]:
# 裁判进 checkpoint:「接着跑」时直接取这里的判定,不再调用、不再计费(每日计数也记在这里面)。
# 事件都在这里面发:取缓存时不会再发一遍。第二轮 limits 是剩下的钱和时间,说给人看的仍是配置的上限
outcome = await judging.run_request(request, spec=judge_spec, budget=limits, known=known, emit=ctx.emit,
shown=budget)
if first and outcome.get("model") and outcome["model"] == model_id:
outcome["same_model"] = True
ctx.emit(EventType.LOG, level="warn", code="judge_same_model",
message=f"裁判模型和写作模型都是「{model_id}」:等于自己审自己,模型写错的地方它多半也看不出来。"
"到设置里选一个不同的「证据裁判模型」,或者在节点的 judge.model 里指定")
return {**outcome, "key": key}
async def judge_round(round_doc: dict[str, Any], limits: judging.Budget, known: dict[str, Any],
first: bool) -> dict[str, Any]:
request = judging.prepare(round_doc, catalog, masked=masks)
key = judging.request_key(request, judge_spec, limits, known)
if _in_graph():
# 写作不进 checkpoint,接着跑时写作者会重写一稿:缓存里先前那一稿的判定对不上指纹,往后翻
for _ in range(_STALE_LIMIT):
outcome = await judge_step(request, key, limits, known, first)
if outcome.get("key") == key:
return outcome
return await judging.run_request(request, spec=judge_spec, budget=limits, known=known, emit=ctx.emit,
shown=budget)
outcome = await judge_round(doc, budget, {}, True)
same_model = bool(outcome.get("same_model"))
usages = [outcome["usage"]]
response = messages[-1]
rewrite: dict[str, Any] | None = None
bad = [uid for uid, v in outcome["verdicts"].items() if v["status"] == "unsupported"]
if judge_cfg["rewrite_once"] and bad:
texts = judging.unit_texts(doc)
sentences = [(texts.get(uid, ""), outcome["verdicts"][uid].get("rationale") or "") for uid in bad]
ctx.emit(EventType.LOG, level="warn", code="report_rewrite",
message=f"裁判认为 {len(bad)} 句结论证据不支持({_named([{'text': t} for t, _ in sentences])}),"
"已交回写作者只改这几句")
again, raw = await draft([*messages, HumanMessage(content=_rewrite_request(sentences))])
rewritten = compose(raw)
worse = _blocking(rewritten["violations"], numbers)
fresh = _new_violations(doc["violations"], rewritten["violations"])
rewrite = {"units": bad, "sentences": [t for t, _ in sentences], "applied": False, "reason": None,
"changed": []}
if fresh:
rewrite["reason"] = (f"改写稿冒出 {len(fresh)} 处原稿没有的问题({_named(fresh)}),没有采用,"
"保留原稿和原来的判定")
ctx.emit(EventType.LOG, level="warn", code="report_rewrite_rejected", message=rewrite["reason"])
else:
known = {outcome["keys"][u]: v for u, v in outcome["verdicts"].items() if v["status"] in judging.JUDGED}
# 第二轮用第一轮剩下的钱和时间:每份报告的上限管的是整份报告
left = judging.Budget(
max_claims=budget.max_claims,
max_cost_usd=None if budget.max_cost_usd is None else max(0.0, budget.max_cost_usd - outcome["cost_usd"]),
timeout_s=None if budget.timeout_s is None else max(0.0, budget.timeout_s - outcome["duration_ms"] / 1000),
daily_max_usd=budget.daily_max_usd)
second = await judge_round(rewritten, left, known, False)
usages.append(second["usage"])
outcome = _merge(outcome, second, budget)
doc, response, blocking = rewritten, again, worse
# 封存的是改写稿,编号按它重排:按原文认回去,改掉了的句子在改写稿里没有编号
now = judging.unit_texts(rewritten)
at: dict[str, str] = {}
for uid, text in now.items():
at.setdefault(text, uid)
before = set(texts.values())
rewrite.update(applied=True, units=[at.get(texts.get(uid, "")) for uid in bad],
changed=[uid for uid, text in now.items() if text not in before])
summary = judging.summarize(outcome, mode="inline", on_unsupported=judge_cfg["on_unsupported"],
rewrite_once=judge_cfg["rewrite_once"], same_model=same_model, rewrite=rewrite)
judging.apply_judgement(doc, outcome, summary)
return doc, response, blocking, summary, usages
+10
View File
@@ -95,6 +95,16 @@ WARN_CODES: dict[str, tuple[str, str]] = {
# (记成空值,不兜底成 0)。答案能用,缺口要说清是哪一种
"agent_field_mismatch": ("agent_field_mismatch", DEGRADED),
"agent_field_unverified": ("agent_field_unverified", DEGRADED),
# 结论句裁判(claims: judge):触顶、没跑成的句子记为未裁判,出具不能判完整——是缺口,要说明
"judge_limit": ("judge_limit", DEGRADED),
"judge_failed": ("judge_failed", DEGRADED),
# 证据不支持的句子交回写作者改过一次(和 report_repair 一样只记缺口);改写稿冒出新问题没采用的,
# 原稿里那几句仍是证据不支持
"report_rewrite": ("report_rewrite", DEGRADED),
"report_rewrite_rejected": ("report_rewrite_rejected", DEGRADED),
# 估不出裁判的金额、裁判和写作是同一个模型:配置上的提醒,答案本身没受影响
"judge_unpriced": ("judge_unpriced", BENIGN),
"judge_same_model": ("judge_same_model", BENIGN),
}
# 这些 kind 重跑一次大概率能好:原因说得清,且不是「问题本身没问明白」。
+83 -12
View File
@@ -1,5 +1,6 @@
from __future__ import annotations
import math
import re
from collections import defaultdict
from enum import StrEnum
@@ -423,31 +424,101 @@ def _check_report_sources(node: GraphNode, spec: GraphSpec, result: ValidationRe
#: 报告撰写节点的结论句策略(claims)。off:结论句不参与判档(默认,保持以前的行为);
#: require_citation:没挂引用的结论句计入缺口、按出具档位降档。judge(模型裁判)是后续版本的
#: require_citation:没挂引用的结论句计入缺口、按出具档位降档。这两档是确定性的(发布前自动修复
#: 给人选的也只有这两档)
REPORT_CLAIMS = ("off", "require_citation")
CLAIMS_LATER = "结论句裁判在后续版本支持"
#: judge:结论句由另一个模型按证据逐句判断(engine/judge.py),子配置写在节点的 judge 里
CLAIMS_JUDGE = "judge"
CLAIMS_VALUES = (*REPORT_CLAIMS, CLAIMS_JUDGE)
#: judge 子配置能写的键。三个上限写 null 表示不限;没写的键运行时取设置里的默认(设置 → judge 一组)
JUDGE_KEYS = ("provider", "model", "max_claims", "max_cost_usd", "timeout_s", "rewrite_once", "on_unsupported")
#: 证据不支持时正式运行怎么判档:degrade 降档(默认)/ withhold 不予出具。探索运行只标注
JUDGE_ON_UNSUPPORTED = ("degrade", "withhold")
def claims_problem(value: Any) -> str | None:
"""claims 的取值有什么问题,没问题返回 None。校验和报告撰写节点执行时报的是同一句话。"""
if value in (None, "") or (isinstance(value, str) and value in REPORT_CLAIMS):
if value in (None, "") or (isinstance(value, str) and value in CLAIMS_VALUES):
return None
if value == "judge":
return f"claims 写的是 judge:{CLAIMS_LATER},现在只能写 off 或 require_citation"
return f"claims 只能是 {' / '.join(REPORT_CLAIMS)},写的是 {value!r}"
return f"claims 只能是 {' / '.join(CLAIMS_VALUES)},写的是 {value!r}"
def _positive(value: Any, *, integer: bool = False) -> bool:
if isinstance(value, bool) or not isinstance(value, (int, float)) or not math.isfinite(value) or value <= 0:
return False
return not integer or int(value) == value
def judge_problems(value: Any) -> list[tuple[str, str]]:
"""claims: judge 的子配置有什么问题:[(字段, 人话)],没问题返回 []。校验和节点执行时报的是同一句话。
上限写 null 就是不限(和没写不一样:没写取设置里的默认)。键写错了也报:写成 max_cost 的预算会被
悄悄当成没写、按默认值花钱。
"""
if value in (None, ""):
return []
if not isinstance(value, dict):
return [("judge", f'judge 要写成一个对象,比如 {{"max_cost_usd": 0.05, "max_claims": 40}},写的是 {value!r}')]
out: list[tuple[str, str]] = []
for key, item in value.items():
field = f"judge.{key}"
if key not in JUDGE_KEYS:
out.append((field, f"judge 里不认识「{key}」:能写的是 {'、'.join(JUDGE_KEYS)}"))
elif key in ("provider", "model") and item is not None and not isinstance(item, str):
what = "模型接入的名字" if key == "provider" else "模型 id"
out.append((field, f"judge.{key} 要写{what}(字符串),不写或写 null 就用设置里的「证据裁判模型」,"
f"写的是 {item!r}"))
elif key == "max_claims" and item is not None and not _positive(item, integer=True):
out.append((field, f"judge.max_claims 要写正整数(每份报告最多判几句),写 null 表示不限,写的是 {item!r}"))
elif key == "max_cost_usd" and item is not None and not _positive(item):
out.append((field, "judge.max_cost_usd 要写大于 0 的金额(美元,每份报告的裁判花费上限),"
f"写 null 表示不限,写的是 {item!r}"))
elif key == "timeout_s" and item is not None and not _positive(item):
out.append((field, f"judge.timeout_s 要写大于 0 的秒数(每份报告的裁判时长上限),写 null 表示不限,"
f"写的是 {item!r}"))
elif key == "rewrite_once" and not isinstance(item, bool):
out.append((field, f"judge.rewrite_once 只能是 true 或 false,写的是 {item!r}"))
elif key == "on_unsupported" and item not in JUDGE_ON_UNSUPPORTED:
out.append((field, f"judge.on_unsupported 只能是 {' / '.join(JUDGE_ON_UNSUPPORTED)},写的是 {item!r}"))
return out
def _check_report_claims(node: GraphNode, spec: GraphSpec, result: ValidationResult) -> None:
"""claims 写错了(包括还不支持的 judge)挡住运行:执行时报告节点照样会报这一句,
与其跑完取数、算完口径卡才死在写报告那一步,不如在画图时说破。
"""claims 写错了挡住运行:执行时报告节点照样会报这一句,与其跑完取数、算完口径卡才死在写报告
那一步,不如在画图时说破。claims 是 judge 时再查 judge 子配置的形状;裁判模型和写作模型写成
同一个的给警告——等于自己审自己。
和运行时同一个取值规则:节点上没写就取图级 defaults(NodeContext.cfg)。
"""
value = node.config.get("claims")
if value in (None, ""):
value = spec.defaults.get("claims")
def effective(key: str) -> Any:
value = node.config.get(key)
return spec.defaults.get(key) if value in (None, "") else value
value = effective("claims")
if problem := claims_problem(value):
result.add(problem, node_id=node.id, field="claims", code="report.claims_invalid")
if value != CLAIMS_JUDGE:
return
judge = effective("judge")
for field, message in judge_problems(judge):
result.add(message, node_id=node.id, field=field, code="report.judge_invalid")
judge_model = judge.get("model") if isinstance(judge, dict) else None
if isinstance(judge_model, str) and judge_model and judge_model == effective("model"):
result.add(f"裁判模型和写作模型都是「{judge_model}」:等于自己审自己,模型写错的地方它多半也看不出来。"
"给 judge.model 换一个不同的模型", level="warning", node_id=node.id, field="judge.model",
code="report.judge_same_model")
def contract_needs_metrics(contract: dict[str, Any]) -> bool:
"""出具契约要不要写 metrics_from(指标来自哪几张口径卡)。
旧写法按数值回指叙述,没有口径卡就无从回指,一定要写。引用模式(report_from)下,数字回指靠
报告里的引用标记:只引查询单元格(cells)的问数据图根本没有口径卡,不该逼它写;契约里列了
required / expected 指标的,指标总得有个来处,照样要写。运行时 io.py 也是这么判的。
"""
if contract.get("report_from") in (None, ""):
return True
return bool(contract.get("required") or contract.get("expected"))
def _check_report_from(node: GraphNode, report_from: Any, spec: GraphSpec,
@@ -542,7 +613,7 @@ def validate_graph(spec: GraphSpec) -> ValidationResult:
# 报成 error,两档发布和运行前都挡住
result.add("出具契约必须是一个 JSON 对象", node_id=node.id, field="contract",
code="contract.not_object")
elif contract and not contract.get("metrics_from"):
elif contract and not contract.get("metrics_from") and contract_needs_metrics(contract):
result.add("出具契约缺 metrics_from(指标来自哪个「口径卡」节点)",
node_id=node.id, field="contract.metrics_from", code="contract.metrics_from_missing")
if isinstance(contract, dict) and contract.get("report_from") not in (None, ""):
+4 -5
View File
@@ -118,22 +118,21 @@ def test_published_level_warnings_get_no_fix():
# --------------------------------------------------------------------------
def test_judge_becomes_require_citation_under_governed():
graph = traceable(report={**STRICT, "claims": "judge"})
def test_an_invalid_claims_value_becomes_require_citation_under_governed():
graph = traceable(report={**STRICT, "claims": "sometimes"})
fid = "report.claims_invalid:write"
fix = plan(graph)[fid]
assert fix["kind"] == "auto" and fix["preview"] == {"field": "claims", "before": "judge",
assert fix["kind"] == "auto" and fix["preview"] == {"field": "claims", "before": "sometimes",
"after": "require_citation"}
out = apply_fixes(graph, [fid], level="governed")
assert out["ok"] is True and config(out["graph"], "write")["claims"] == "require_citation"
def test_an_invalid_claims_value_is_a_choice_under_published():
graph = traceable(report={"claims": "judge"})
graph = traceable(report={"claims": "sometimes"})
fid = "report.claims_invalid:write"
fix = plan(graph, "published")[fid]
assert fix["kind"] == "choice" and [o["value"] for o in fix["options"]] == ["off", "require_citation"]
assert fix["default"] == "require_citation" # 离裁判最近的是「每句结论挂引用」,仍然要人确认
assert apply_fixes(graph, [fid], level="published")["applied"] == []
out = apply_fixes(graph, [fid], {fid: "off"}, level="published")
assert out["applied"] == [fid] and config(out["graph"], "write")["claims"] == "off" and out["ok"] is True
+111
View File
@@ -0,0 +1,111 @@
"""结论句预筛:哪些句子送裁判模型、按什么优先级,哪些直接判为连接性。
裁判按调用计费,一份报告里的过渡话、标题、表格里的一格、代码都不值得花钱判。预筛是确定性的
规则(同一份文档永远得到同一个结果),而且只看文档本身:
- 优先级 1:写作者给这句挂了依据(行内引用或 [[see:…]])
- 优先级 2:没挂依据,但有方向词、因果词(增长、主要来自、导致…)
- 优先级 3:其余有数字、实体这类实质内容、又不是短句的
- 不送:连接性的话、标题、代码、表格单元格;只有过渡词的(「原因如下:」);短句
预算不够时按优先级截断:先判 1,再判 2,剩下的记未裁判。
"""
from __future__ import annotations
from app.engine.evidence import build_catalog, compose_doc, iter_units
from app.engine.judge import SHORT_CHARS, candidates, claim_priority
def catalog():
return build_catalog(nodes={"caliber": {
"kind": "metric_set", "caliber": "周报口径", "caliber_version": "v2", "artifact": "f" * 64,
"metrics": [{"id": "gmv", "name": "销售额", "value": 45678.5, "unit": "元", "decimals": 1},
{"id": "orders", "name": "订单数", "value": 1234, "unit": "单"}]}})
RAW = """## 本周概览
本周销售额 [[m:gmv]]。增长主要来自新客首单。[[see:m:gmv]]原因如下:
下面看细节。订单一共是 1234 单,整体平稳。共 3 单。增长明显。
- 列表里订单 [[m:orders]]
- 过渡说明
| 区域 | 销售额 |
|---|---|
| 华东 | [[m:gmv]] |
```sql
SELECT 1
```
"""
def by_text(doc):
"""{句子原文: unit},测试按字找句子,不依赖编号。"""
md = doc["markdown"]
return {md[u["span"][0]:u["span"][1]].strip(): (b, u) for b, u in iter_units(doc)}
def priorities(doc):
return {text: claim_priority(u, b, doc["markdown"]) for text, (b, u) in by_text(doc).items()}
def test_the_rules():
got = priorities(compose_doc(RAW, catalog()))
assert got["本周销售额 45,678.5元。"] == 1 # 行内引用
assert got["增长主要来自新客首单。"] == 1 # 句末 [[see:]] 挂的依据
assert got["列表里订单 1,234单"] == 1 # 列表项也是句子
assert got["增长明显。"] == 2 # 没挂依据,有方向词:短也要判
assert got["订单一共是 1234 单,整体平稳。"] == 3 # 有数字、不是短句
assert got["原因如下:"] is None # 有「原因」但只是过渡
assert got["共 3 单。"] is None # 短句
assert got["下面看细节。"] is None # 连接性
assert got["过渡说明"] is None
assert got["本周概览"] is None # 标题
assert got["45,678.5元"] is None # 表格单元格:一格值不是一句结论
assert got["SELECT 1"] is None # 代码
def test_it_is_deterministic():
one, two = compose_doc(RAW, catalog()), compose_doc(RAW, catalog())
assert priorities(one) == priorities(two)
assert [c.unit for c in candidates(one)] == [c.unit for c in candidates(two)]
def test_candidates_come_in_priority_order_then_document_order():
doc = compose_doc(RAW, catalog())
texts = {u["id"]: t for t, (_, u) in by_text(doc).items()}
got = [(c.priority, texts[c.unit]) for c in candidates(doc)]
assert got == [
(1, "本周销售额 45,678.5元。"), (1, "增长主要来自新客首单。"), (1, "列表里订单 1,234单"),
(2, "增长明显。"),
(3, "订单一共是 1234 单,整体平稳。"),
]
def test_short_means_visible_characters_not_bytes():
"""短句按看得见的字算(标点、空白不算),数字的字也算进去。"""
assert SHORT_CHARS == 6
doc = compose_doc("订单 1234 单。订单总数 1234 单。", catalog())
got = priorities(doc)
assert got["订单 1234 单。"] == 3 and got["订单总数 1234 单。"] == 3
doc = compose_doc("共 12 单。", catalog())
assert priorities(doc)["共 12 单。"] is None
def test_an_explicit_request_overrides_the_screen_but_not_the_kind():
"""有人点开一句、明确要求判它:预筛放掉的结论句照判(按最低优先级);标题、连接性的话不是结论,不判。"""
doc = compose_doc(RAW, catalog())
units = {t: u["id"] for t, (_, u) in by_text(doc).items()}
picked = candidates(doc, units=[units["共 3 单。"], units["下面看细节。"], units["本周概览"], "u999"])
assert [(c.unit, c.priority) for c in picked] == [(units["共 3 单。"], 3)]
def test_cites_are_the_authors_own():
"""自动链接的名字不算依据(三期 D2):只提到表名的句子不会因此排进优先级 1。"""
ledger = [{"kind": "schema", "node_id": "fetch", "exec": 1, "artifact": "a" * 64, "source": "shop",
"schema": None, "synced_at": None, "tables": {"orders": ["amount", "region"]}}]
cat = build_catalog(nodes={}, ledger=ledger)
doc = compose_doc("明细在 `orders` 表里,按区域汇总之后东区最高。", cat)
[(text, prio)] = [(t, p) for t, p in priorities(doc).items()]
assert prio == 2, (text, prio)
+71
View File
@@ -326,3 +326,74 @@ async def test_the_report_node_and_the_exit_check_agree_when_the_status_changes_
assert seen == [True, True], seen
issuance = row.output["_issuance"]
assert issuance["tier"] == "withheld" and CELLS_REASON in issuance["unresolved"][0]["message"], issuance
# --------------------------------------------------------------------------
# 只引单元格、没有口径卡:契约不用写 metrics_from
#
# 问数据的典型图是 input → 取数 → 报告 → 出口,报告直接引用查过的单元格。受管级别在契约里
# 写 cells: true 就够了(用户拍板的第 5 条),以前校验和门禁无条件要 metrics_from,
# 这张图怎么都发不成受管,Copilot 补上契约也会被判「冒出新的问题」作废
# --------------------------------------------------------------------------
def cells_only(**contract):
contract = {"report_from": "write", "strict": True, "cells": True, **contract}
nodes = [
node("start", "input"),
node("fetch", "tool", tool="db_query__shop",
args={"sql": "SELECT SUM(amount) AS gmv, COUNT(*) AS orders FROM orders"}),
node("write", "report", instructions="写周报", numbers="strict", on_violation="fail",
claims="require_citation"),
node("out", "output", fields=[{"name": "周报", "value": "{{ nodes.write.text }}"}], contract=contract),
]
return {"nodes": nodes, "edges": [{"source": a["id"], "target": b["id"]} for a, b in zip(nodes, nodes[1:])]}
def contract_codes(graph: dict, level: str = "governed") -> tuple[list[str], list[str]]:
from app.engine.governance import lint_for_publish
from app.engine.schema import GraphSpec, validate_graph
spec = GraphSpec.model_validate(graph)
return ([i.code for i in validate_graph(spec).issues if i.level == "error"],
[i.code for i in lint_for_publish(spec, level=level).issues if i.level == "error"])
def test_a_cells_only_contract_needs_no_metrics_from():
validate, gate = contract_codes(cells_only())
assert validate == [], validate
assert gate == [], gate # 受管门禁整张图一条 error 都没有
def test_a_cited_contract_that_lists_required_metrics_still_needs_metrics_from():
validate, gate = contract_codes(cells_only(required=["gmv"]))
assert "contract.metrics_from_missing" in validate and "contract.metrics_from_missing" in gate
validate, gate = contract_codes(cells_only(expected=["gmv"]))
assert "contract.metrics_from_missing" in validate and "contract.metrics_from_missing" in gate
def test_a_narrative_contract_still_needs_metrics_from():
graph = cells_only(report_from=None, narrative="{{ nodes.write.text }}")
graph["nodes"][-1]["data"]["config"]["contract"].pop("report_from")
validate, _ = contract_codes(graph, level="published") # 受管级别不收 narrative,只看校验
assert "contract.metrics_from_missing" in validate
async def test_a_governed_formal_run_over_cells_only_issues_formally(client, monkeypatch):
Writer(monkeypatch, text="本周销售额 [[v:Q1.r0.gmv]]。")
row = await formal(client, cells_only(), status="governed")
assert row.status == "succeeded", row.error
issuance = row.output["_issuance"]
assert issuance["mode"] == "citations" and issuance["tier"] == "formal", issuance
assert issuance["unresolved"] == [] and issuance["gaps"] == [], issuance
[cell] = [m for m in issuance["matched"] if m.get("cell")]
assert cell["cell"] == "Q1.r0.gmv" and cell["token"] == "300.5"
def test_a_graph_with_a_caliber_card_still_needs_required_even_without_metrics_from():
graph = weekly(cells=True)
contract = graph["nodes"][-1]["data"]["config"]["contract"]
contract.pop("metrics_from"), contract.pop("required")
validate, gate = contract_codes(graph)
assert "contract.required_missing" in gate, gate # 不写 metrics_from 绕不过必需指标
assert validate == [], validate
+323
View File
@@ -0,0 +1,323 @@
"""出口契约(引用模式)按结论句裁判的判定判档(四期)。
- 报告节点 claims: judge 在正式运行里当场裁判,判定随文档封存。出口按文档里的判定判档:证据不支持的结论句
按 on_unsupported 降档(degrade,默认)或不予出具(withhold);裁判没跑成、触顶了没判完的,裁判摘要里的缺口
照抄进出具声明——不能判完整出具,也不因此不予出具
- 判定是模型给的,出口复算不了:信按哈希取回的文档
- judge 在挂引用这件事上和 require_citation 一样严:没挂依据的结论句照样计入缺口
- 探索运行按需裁判:出口只标注(候选句都是「未裁判 · 按需」),不因判定改档位
- 契约也能写 claims: judge:只能收紧(on_unsupported 取更严的);报告节点没开 judge 的,照实记缺口
- 升级前的运行照旧:契约写了也记「这一版还不支持」
"""
from __future__ import annotations
import asyncio
import json
import re
import uuid
import pytest
from httpx import ASGITransport, AsyncClient
from langchain_core.messages import AIMessage
from sqlalchemy import delete, select
from app.db.base import SessionLocal
from app.db.models import Run, RunEvent, Setting, Workflow, WorkflowVersion
from app.engine import judge as judge_mod
from app.engine import runner as runner_mod
from app.engine.judge import JUDGE_ROLE, SPEND_KEY
from app.engine.runner import run_manager
from app.main import app
from app.providers.mock_model import MockChatModel
JUDGE_MODEL = "claude-sonnet-5"
METRICS = [{"id": "gmv", "name": "销售额", "unit": "元", "decimals": 1, "expression": "vars.kpi.gmv"}]
#: 两句结论都挂了依据、一句连接性的话
CITED = "本周销售额 [[m:gmv]]。[[see:m:gmv]]增长主要来自新客首单。[[see:m:gmv]]下面看细节。"
#: 第二句结论没挂依据
UNCITED = "本周销售额 [[m:gmv]]。[[see:m:gmv]]增长主要来自新客首单。下面看细节。"
@pytest.fixture(autouse=True)
async def engine_up():
await run_manager.setup()
async with SessionLocal() as session:
await session.execute(delete(Setting).where(Setting.key.in_([SPEND_KEY, "judge", "copilot"])))
await session.commit()
yield
await run_manager.shutdown()
@pytest.fixture
async def client():
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as c:
yield c
def node(nid, ntype, **config):
return {"id": nid, "type": ntype, "position": {"x": 0, "y": 0}, "data": {"label": nid, "config": config}}
def weekly(*, report=None, contract=None) -> dict:
nodes = [
node("start", "input", fields=[{"name": "week", "default": "2026-W37"}]),
node("fetch", "transform", mode="expression", expression="{'gmv': 45678.5}", assign_to="kpi"),
node("caliber", "metrics", caliber="周报口径", caliber_version="v2", metrics=METRICS),
node("write", "report", instructions="写周报", **{"claims": "judge", "judge": {"max_cost_usd": 0.05},
**(report or {})}),
node("out", "output", fields=[{"name": "周报", "value": "{{ nodes.write.text }}"}],
contract={"report_from": "write", "metrics_from": ["caliber"], "required": ["gmv"], "strict": True,
**(contract or {})}),
]
ids = [n["id"] for n in nodes]
return {"nodes": nodes, "edges": [{"source": a, "target": b} for a, b in zip(ids, ids[1:])]}
class Models:
"""写作者回 text;裁判按句子里的字给判定(verdicts:{句中片段: 判定},没对上的是 supported)。"""
def __init__(self, monkeypatch, text=CITED, verdicts=None, *, fail_judge=None):
self.judges: list[str] = []
verdicts = verdicts or {}
def decide(model, messages):
if JUDGE_ROLE in str(messages[0].content):
request = str(messages[-1].content)
self.judges.append(request)
if fail_judge:
raise RuntimeError(fail_judge)
items = [{"unit": uid,
"verdict": next((s for frag, s in verdicts.items() if frag in line), "supported"),
"rationale": "摘录里有对应的指标" if "销售额" in line else "证据里没有新客维度", "used": []}
for uid, line in re.findall(r"^\[(u\d+)\]([^\n]*)$", request, re.M)]
return AIMessage(content=json.dumps({"items": items}, ensure_ascii=False))
return AIMessage(content=text)
monkeypatch.setattr(MockChatModel, "_decide", decide)
async def judge_model_of(session, spec):
return MockChatModel(model_name=JUDGE_MODEL), JUDGE_MODEL
monkeypatch.setattr(judge_mod, "get_chat_model", judge_model_of)
async def wait(run_id: str) -> Run:
for _ in range(400):
await asyncio.sleep(0.05)
async with SessionLocal() as session:
row = await session.get(Run, run_id)
if row.status in ("succeeded", "failed") and row.manifest_seq is not None:
return row
raise AssertionError(f"run {run_id} 没跑完:{row.status}")
async def issued(graph, *, run_class="formal") -> dict:
run = await run_manager.start(graph=graph, input_payload={}, run_class=run_class)
row = await wait(run.id)
assert row.status == "succeeded", row.error
return row.output["_issuance"]
async def governed(client, graph) -> Run:
"""把这张图发布成受管级别,从发布版本发起正式运行。"""
async with SessionLocal() as session:
wf = Workflow(name=f"judge-{uuid.uuid4().hex[:6]}", graph=graph, status="governed", published_version=1)
session.add(wf)
await session.flush()
session.add(WorkflowVersion(workflow_id=wf.id, version=1, graph=graph))
await session.commit()
wf_id = wf.id
res = await client.post("/api/runs", json={"workflow_id": wf_id, "run_class": "formal"})
assert res.status_code == 201, res.text
row = await wait(res.json()["id"])
assert row.status == "succeeded", row.error
return row
async def issuance_event(run_id: str) -> dict:
async with SessionLocal() as session:
[event] = (await session.execute(select(RunEvent).where(
RunEvent.run_id == run_id, RunEvent.type == "issuance"))).scalars()
return event.data
# --------------------------------------------------------------------------
# 正式运行:按封存的判定判档
# --------------------------------------------------------------------------
async def test_every_claim_supported_is_a_full_issuance(monkeypatch):
Models(monkeypatch)
issuance = await issued(weekly())
assert issuance["tier"] == "formal" and issuance["gaps"] == [], issuance
claims = issuance["claims"]
assert claims["policy"] == "judge" and claims["mode"] == "inline" and claims["complete"] is True
assert claims["counts"] == {"supported": 2, "partial": 0, "unsupported": 0, "not_a_claim": 0, "unjudged": 0}
assert claims["on_unsupported"] == "degrade" and claims["uncited_claims"] == 0
assert claims["unsupported"] == [] and claims["model"] == JUDGE_MODEL
async def test_an_unsupported_claim_degrades_by_default(monkeypatch):
Models(monkeypatch, verdicts={"新客": "unsupported"})
issuance = await issued(weekly())
assert issuance["tier"] == "degraded", "不支持按缺口降档,不是不予出具(契约是 strict 的)"
assert issuance["gaps"] == [], "判了就不是「没核」:档位来自判定,不来自缺口"
[flagged] = issuance["claims"]["unsupported"]
assert flagged["text"] == "增长主要来自新客首单。" and flagged["rationale"] == "证据里没有新客维度"
assert flagged["unit"] and len(flagged["span"]) == 2
assert issuance["claims"]["counts"]["unsupported"] == 1
async def test_withhold_withholds(monkeypatch):
Models(monkeypatch, verdicts={"新客": "unsupported"})
issuance = await issued(weekly(report={"judge": {"max_cost_usd": 0.05, "on_unsupported": "withhold"}}))
assert issuance["tier"] == "withheld" and issuance["claims"]["on_unsupported"] == "withhold"
async def test_partial_support_is_listed_but_does_not_cost_a_grade(monkeypatch):
Models(monkeypatch, verdicts={"新客": "partial"})
issuance = await issued(weekly(report={"judge": {"max_cost_usd": 0.05, "on_unsupported": "withhold"}}))
assert issuance["tier"] == "formal"
[item] = issuance["claims"]["partial"]
assert item["text"] == "增长主要来自新客首单。" and issuance["claims"]["counts"]["partial"] == 1
async def test_a_failed_judge_is_a_gap_not_a_withheld_report(monkeypatch):
Models(monkeypatch, fail_judge="网关 502")
issuance = await issued(weekly(report={"judge": {"max_cost_usd": 0.05, "on_unsupported": "withhold"}}))
assert issuance["tier"] == "degraded", "没判成不能盖完整出具,也不能当成「不支持」不予出具"
[gap] = [g for g in issuance["gaps"] if "裁判" in g]
assert gap.startswith("报告「write」") and "没裁判" in gap and "网关 502" in gap
assert issuance["claims"]["complete"] is False and issuance["claims"]["counts"]["unjudged"] == 2
async def test_hitting_a_limit_is_a_gap(monkeypatch):
Models(monkeypatch)
issuance = await issued(weekly(report={"judge": {"max_cost_usd": 0.05, "max_claims": 1}}))
assert issuance["tier"] == "degraded"
[gap] = [g for g in issuance["gaps"] if "裁判" in g]
assert "已到上限" in gap and "1 句" in gap
assert issuance["claims"]["limits_hit"] == ["max_claims"] and issuance["claims"]["counts"]["unjudged"] == 1
async def test_an_uncited_claim_still_counts_under_judge(monkeypatch):
"""judge 在挂引用这件事上和 require_citation 一样严:从 require_citation 换成 judge 不能让没挂依据的句子过关。"""
Models(monkeypatch, text=UNCITED)
issuance = await issued(weekly())
assert issuance["tier"] == "degraded" and issuance["claims"]["uncited_claims"] == 1
assert issuance["claims"]["uncited"][0]["text"] == "增长主要来自新客首单。"
async def test_the_issuance_event_carries_the_counts(monkeypatch):
Models(monkeypatch, verdicts={"新客": "unsupported"})
run = await run_manager.start(graph=weekly(), input_payload={}, run_class="formal")
row = await wait(run.id)
event = await issuance_event(row.id)
assert event["tier"] == "degraded"
assert event["claims"] == {"policy": "judge", "uncited_claims": 0, "on_unsupported": "degrade",
"mode": "inline", "counts": {"supported": 1, "partial": 0, "unsupported": 1,
"not_a_claim": 0, "unjudged": 0}}
async def test_a_governed_formal_run_grades_by_the_verdicts(client, monkeypatch):
Models(monkeypatch, verdicts={"新客": "unsupported"})
graph = weekly(report={"numbers": "strict", "on_violation": "fail",
"judge": {"max_cost_usd": None, "on_unsupported": "withhold"}})
row = await governed(client, graph)
issuance = row.output["_issuance"]
assert issuance["tier"] == "withheld" and issuance["claims"]["unsupported"][0]["text"] == "增长主要来自新客首单。"
# --------------------------------------------------------------------------
# 探索运行:只标注
# --------------------------------------------------------------------------
async def test_explore_runs_only_annotate(monkeypatch):
models = Models(monkeypatch, verdicts={"新客": "unsupported"})
issuance = await issued(weekly(report={"judge": {"max_cost_usd": 0.05, "on_unsupported": "withhold"}}),
run_class="exploratory")
assert models.judges == [], "探索运行节点里不花这笔钱"
assert issuance["tier"] == "formal" and issuance["gaps"] == []
claims = issuance["claims"]
assert claims["mode"] == "on_demand" and claims["complete"] is False
assert claims["counts"]["unjudged"] == 2 and claims["unjudged"] == {"on_demand": 2}
assert claims["unsupported"] == []
async def test_explore_runs_still_count_uncited_claims(monkeypatch):
Models(monkeypatch, text=UNCITED)
issuance = await issued(weekly(), run_class="exploratory")
assert issuance["tier"] == "degraded" and issuance["claims"]["uncited_claims"] == 1
# --------------------------------------------------------------------------
# 契约里的 claims: judge
# --------------------------------------------------------------------------
async def test_the_contract_can_tighten_on_unsupported(monkeypatch):
Models(monkeypatch, verdicts={"新客": "unsupported"})
issuance = await issued(weekly(contract={"claims": {"policy": "judge", "on_unsupported": "withhold"}}))
assert issuance["tier"] == "withheld" and issuance["claims"]["on_unsupported"] == "withhold"
assert not any("不支持" in g and "这一版" in g for g in issuance["gaps"])
async def test_the_contract_cannot_loosen_on_unsupported(monkeypatch):
Models(monkeypatch, verdicts={"新客": "unsupported"})
issuance = await issued(weekly(report={"judge": {"max_cost_usd": 0.05, "on_unsupported": "withhold"}},
contract={"claims": {"policy": "judge", "on_unsupported": "degrade"}}))
assert issuance["tier"] == "withheld"
async def test_a_contract_asking_for_judge_on_a_report_without_it_is_a_gap(monkeypatch):
Models(monkeypatch)
issuance = await issued(weekly(report={"claims": "require_citation"}, contract={"claims": "judge"}))
assert issuance["tier"] == "degraded"
[gap] = [g for g in issuance["gaps"] if "judge" in g]
assert "「write」" in gap and "没有开" in gap and "这一版" not in gap
assert issuance["claims"]["policy"] == "require_citation"
def test_a_run_in_flight_across_the_upgrade_keeps_the_old_gap():
"""升级前写的报告(产出里没有 claims)碰上写了 judge 的契约:照旧记「这一版还不支持」,不判档。"""
from app.engine.nodes.io import _claims
out = {"gaps": [], "claims_policy": None, "uncited": None, "claims": None, "unsupported": None}
_claims(out, {"claims": {"policy": "judge", "on_unsupported": "withhold"}}, {"text": "x"}, evidence_on=True,
uncited=[{"unit": "u1", "span": [0, 4], "text": "增长很快"}])
assert out["claims_policy"] is None and out["claims"] is None and out["unsupported"] is None
assert len(out["gaps"]) == 1 and "这一版还不支持" in out["gaps"][0]
async def test_runs_from_before_the_upgrade_ignore_judge(monkeypatch):
async def no_snapshot(session):
return None
monkeypatch.setattr(runner_mod, "_agent_limits", no_snapshot)
models = Models(monkeypatch, text=UNCITED, verdicts={"新客": "unsupported"})
issuance = await issued(weekly())
assert models.judges == [] and issuance["tier"] == "formal" and "claims" not in issuance
# --------------------------------------------------------------------------
# 答案复核(review.scan)认得裁判的几条警告
# --------------------------------------------------------------------------
@pytest.mark.parametrize(("code", "shown"), [
("judge_limit", True), ("judge_failed", True), ("report_rewrite", True), ("report_rewrite_rejected", True),
("judge_unpriced", False), ("judge_same_model", False),
])
def test_review_knows_the_judge_warnings(code, shown):
"""没判完、没跑成、改写过的是缺口(要说明);估不出金额、自己审自己只是配置上的提醒,轨迹里有就够了,
不值得为它再叫一次复核模型。"""
from app.engine.review import DEGRADED, scan
signals = scan([{"type": "log", "data": {"level": "warn", "code": code, "message": f"{code} 的原话"}}],
{"answer": "本周销售额 45,678.5元。"})
if shown:
[signal] = signals
assert signal.kind == code and signal.severity == DEGRADED and signal.detail == f"{code} 的原话"
else:
assert signals == []
+31 -9
View File
@@ -1,4 +1,5 @@
"""Copilot 的节点手册要教三期的写法:表名字段名、知识库原话、结论句策略,以及受管模板的报告节点怎么配。
"""Copilot 的节点手册要教三期、四期的写法:表名字段名、知识库原话、结论句策略(含模型裁判 judge),以及受管模板的
报告节点怎么配。
手册里没写的配置,Copilot 就不会用;写错的写法,报告节点会一直被打回。所以这里不只查关键词,
还把手册里举的标记例子交给真正的解析器,确认写法和系统认的是同一套。
@@ -7,9 +8,9 @@ from __future__ import annotations
import re
from app.api.copilot import NODE_REFERENCE, GenerateIn, _user_message
from app.api.copilot import _ASSIST_RULES, NODE_REFERENCE, GenerateIn, _user_message
from app.engine.evidence import QUOTE_MIN, SUPPORTED_KINDS, parse_markers
from app.engine.schema import REPORT_CLAIMS
from app.engine.schema import CLAIMS_VALUES, JUDGE_KEYS, JUDGE_ON_UNSUPPORTED
def _report_section() -> str:
@@ -22,12 +23,22 @@ def test_the_report_entry_teaches_entities_quotes_and_claims():
for words in (
["[[t:", "[[c:", "反引号", "可能是编造的名字"], # 表名、字段名
["[[q:K1|", "逐字", f"至少 {QUOTE_MIN} 个字", "[[see:K1]]"], # 知识库原话
["claims", "require_citation", "off", "judge", "后续版本"], # 结论句策略
["claims", "require_citation", "off", "judge"], # 结论句策略
["judge.max_cost_usd", "null", "不限", "rewrite_once", "on_unsupported", "按需"], # 模型裁判怎么配
["entities", "link"], # 实体层开关
):
assert all(w in report for w in words), [w for w in words if w not in report]
# 结论句策略的取值和校验认的是同一份
assert all(value in report for value in REPORT_CLAIMS)
# 结论句策略的取值和校验认的是同一份;judge 已经能用,不能再说「后续版本」「跑不起来」
assert all(value in report for value in CLAIMS_VALUES)
assert all(value in report for value in JUDGE_ON_UNSUPPORTED)
assert "后续版本" not in report and "跑不起来" not in report
# 裁判模型建议和写作模型不同:否则等于自己审自己
assert "不同" in report and "自己审自己" in report
def test_the_judge_keys_in_the_guide_are_the_ones_the_validator_knows():
named = set(re.findall(r"judge\.(\w+)", NODE_REFERENCE))
assert named and named <= set(JUDGE_KEYS), named - set(JUDGE_KEYS)
def test_the_markers_in_the_guide_parse_as_the_system_reads_them():
@@ -40,15 +51,26 @@ def test_the_markers_in_the_guide_parse_as_the_system_reads_them():
def test_governed_templates_spell_out_the_report_settings():
ref = NODE_REFERENCE
governed = ref[ref.index("受管"):]
governed = ref[ref.index("要按受管级别发布的模板"):]
for words in (['numbers: "strict"', 'on_violation: "fail"', 'claims: "require_citation"'],
['claims: "judge"', "judge.max_cost_usd", "null"],
["report_from", "narrative"], ["cite_fields: true"], ["evidence_role", "source"]):
assert all(w in governed for w in words), [w for w in words if w not in governed]
def test_the_publish_assist_knows_judge_is_the_stricter_one():
"""发布前的 Copilot 兜底只许往严里改:它要知道 judge 比 require_citation 严,不能为了过门禁把 judge 改回去。"""
assert "judge" in _ASSIST_RULES and "max_cost_usd" in _ASSIST_RULES
# update_node 把 judge 整个换掉:补预算只写 {max_cost_usd} 会把 withhold 连同裁判模型一起丢了
rules = " ".join(_ASSIST_RULES.split())
assert re.search(r"on_unsupported.{0,20}withhold.{0,20}不改回\s*degrade", rules), "withhold 不许改回 degrade"
assert re.search(r"补预算.{0,40}整份", rules), "补预算时 judge 要整份带上原来的键"
def test_the_answer_branch_mentions_names_and_quotes():
text = _user_message(GenerateIn(instruction="上周哪张表的退款最多", intent="answer"), patch=False)
assert "[[t:" in text and "反引号" in text and "编造" in text
assert "[[q:K1|" in text and "逐字" in text
# 问数是探索运行:不要替人打开结论句策略
assert not re.search(r"claims\s*[:=]\s*['\"]?require_citation", text)
# 问数是探索运行:不要替人打开结论句策略(结论句由读的人点开时按需请模型判断)
assert not re.search(r"claims\s*[:=]\s*['\"]?(require_citation|judge)", text)
assert "按需" in text
+554
View File
@@ -0,0 +1,554 @@
"""结论句裁判的内核(engine/judge.py):证据摘录、裁判调用、结构化输出兜底、预算与每日计数。
数字、实体是确定性的,系统自己能核对;「增长主要来自华东」这种话说得对不对,只能交给另一个模型按
证据判断。这是概率性的判断,所以每一步都要留余地、说实话:
- 一份报告合并成一次调用(超过每批的句数就分批),每句给 supported / partial / unsupported / not_a_claim
- 模型不支持结构化输出、或者给的不是合法结构时退回 JSON 解析,再失败就记「未裁判」,不编判定
- 裁判模型关掉 thinking,max_tokens 4096
- 预算(每份报告的句数、金额、时长,全局每日金额)每一项都能写 null 表示不限;触顶就停、已判的保留、
没判的记未裁判并说明是哪个上限——不报错、不让报告失败。金额超了按优先级截断:先判挂了依据的,
再判有方向词、因果词的
- 每日计数落库,按本地日期滚动
- 给裁判的证据摘录里没有数据源遮罩的列
所有模型调用都是 mock 剧本,不碰真接口。
"""
from __future__ import annotations
import json
import re
import pytest
from langchain_core.messages import AIMessage
from sqlalchemy import delete
from app.core import artifact_store
from app.core.events import EventType
from app.db.base import SessionLocal
from app.db.models import Setting
from app.engine import judge as judge_mod
from app.engine.evidence import build_catalog, compose_doc, iter_units
from app.engine.judge import (
BATCH_SIZE,
CHUNK_CHARS,
EXCERPT_COLS,
EXCERPT_ROWS,
JUDGE_ROLE,
JUDGE_RULES,
MAX_TOKENS,
RATIONALE_MAX,
SPEND_KEY,
SQL_CHARS,
Budget,
daily_spend,
judge_doc,
prepare,
source_masks,
)
from app.providers.factory import ModelSpec
from app.providers.mock_model import MockChatModel
#: 目录里有价格的模型:金额上限对它起作用
PRICED = "claude-sonnet-5"
UNLIMITED = Budget(max_claims=None, max_cost_usd=None, timeout_s=None, daily_max_usd=None)
SPEC = ModelSpec(model=PRICED)
# --------------------------------------------------------------------------
# 夹具:一张口径卡、一次查询(带手机号列)、一次知识库检索、一个运行输入
# --------------------------------------------------------------------------
def _store(content) -> str:
"""按 put_json 的落盘方式写进测试用的临时工件库:取回走真实的 load,复验哈希。"""
text = artifact_store.canonical_json(content)
artifact = artifact_store.content_hash(text)
path = artifact_store._path_of(artifact)
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(text, encoding="utf-8")
return artifact
CARD = {"kind": "metric_set", "caliber": "周报口径", "caliber_version": "v2", "metrics": [
{"id": "gmv", "name": "销售额", "unit": "元", "value": 45678.5, "decimals": 1, "format": "thousands",
"status": "ok", "expression": "vars.kpi.gmv", "rendered": "45,678.5元"},
{"id": "wow", "name": "环比增幅", "unit": "%", "value": 8.7, "decimals": 1, "format": "plain", "status": "ok",
"expression": "round((vars.kpi.gmv - vars.kpi.gmv_prev) / vars.kpi.gmv_prev * 100, 1)", "rendered": "8.7%"},
]}
LONG_SQL = "SELECT region, SUM(amount) AS amount, MAX(phone) AS phone FROM orders WHERE " + " AND ".join(
f"flag_{i} = 1" for i in range(60)) + " GROUP BY region ORDER BY amount DESC"
ROWS = [[f"区域{i}", 1000.0 - i, f"PHONE-{i}"] for i in range(30)]
SNAP = {"columns": ["region", "amount", "phone"], "rows": ROWS, "row_count": len(ROWS), "truncated": False,
"elapsed_ms": 3, "sql": LONG_SQL, "source": "shop"}
HITS = {"query": "退款", "collection": "docs", "hits": [
{"content": "退款主要集中在物流延误的订单。" * 60, "title": "售后周报", "document": "doc-1", "chunk": 0},
{"content": "第二段原文。", "title": "售后周报", "document": "doc-1", "chunk": 1}]}
@pytest.fixture
def stored():
card = {**CARD}
art = _store(card)
return {"card": art, "snap": _store(SNAP), "masked": _store({**SNAP, "mask_columns": ["phone"]}),
"hits": _store(HITS)}
def make_catalog(stored, snap="snap"):
ledger = [
{"kind": "metric_set", "node_id": "caliber", "exec": 1, "artifact": stored["card"], "caliber": "周报口径",
"version": "v2", "metrics": ["gmv", "wow"]},
{"kind": "query", "node_id": "fetch", "exec": 1, "artifact": stored[snap], "via": "b" * 64,
"call_id": "call_1", "tool": "db_query__shop", "source": "shop", "columns": SNAP["columns"],
"rows": len(ROWS), "truncated": False},
{"kind": "retrieval", "node_id": "kb", "exec": 1, "artifact": stored["hits"], "source": "docs", "rows": 2},
]
return build_catalog(nodes={"caliber": {**CARD, "artifact": stored["card"]}}, ledger=ledger,
inputs={"week": "2026-W37"})
RAW = ("## 本周概览\n\n"
"本周销售额 [[m:gmv]]。[[see:m:gmv]]"
"环比 [[m:wow]],增长主要来自华东。[[see:m:wow,Q1]]"
"增长主要来自新客首单。"
"订单一共是 1234 单,整体平稳。"
"下面看细节。")
def units_of(doc) -> dict[str, str]:
"""{句子原文: unit id}"""
md = doc["markdown"]
return {md[u["span"][0]:u["span"][1]].strip(): u["id"] for _, u in iter_units(doc)}
CITED, CITED2, CAUSAL, NUMBER = ("本周销售额 45,678.5元。", "环比 8.7%,增长主要来自华东。", "增长主要来自新客首单。",
"订单一共是 1234 单,整体平稳。")
@pytest.fixture(autouse=True)
async def clean_settings():
"""每日计数和裁判设置都落在设置表里:每个测试从干净的一天开始。"""
async with SessionLocal() as session:
await session.execute(delete(Setting).where(Setting.key.in_([SPEND_KEY, "judge", "copilot"])))
await session.commit()
yield
class Judge:
"""裁判模型的剧本。结构化调用和退回的纯文本调用分开记;verdicts 按 unit id 给判定,没写的都是 supported。"""
def __init__(self, monkeypatch, verdicts=None, *, model_id=PRICED, structured=None, plain=None,
latency=0.0, fail=None):
self.calls: list[dict] = []
self.specs: list[ModelSpec] = []
verdicts = verdicts or {}
def answer(units):
return json.dumps({"items": [{"unit": u, "verdict": verdicts.get(u, "supported"),
"rationale": f"{u} 的依据", "used": ["m:gmv"]} for u in units]},
ensure_ascii=False)
def decide(model, messages):
assert JUDGE_ROLE in str(messages[0].content), "这里只该有裁判调用"
units = re.findall(r"^\[(u\d+)\]", str(messages[-1].content), re.M)
kind = "structured" if model.response_format else "plain"
self.calls.append({"kind": kind, "units": units, "messages": list(messages)})
if fail:
raise RuntimeError(fail)
custom = structured if kind == "structured" else plain
if custom is not None:
return AIMessage(content=custom(units) if callable(custom) else custom)
return AIMessage(content=answer(units))
monkeypatch.setattr(MockChatModel, "_decide", decide)
async def fake_model(session, spec):
self.specs.append(spec)
return MockChatModel(model_name=model_id, latency=latency), model_id
monkeypatch.setattr(judge_mod, "get_chat_model", fake_model)
@property
def judged(self) -> list[list[str]]:
return [c["units"] for c in self.calls if c["kind"] == "structured"]
def status_of(outcome, doc, text):
return outcome["verdicts"][units_of(doc)[text]]
# --------------------------------------------------------------------------
# 调用与输出
# --------------------------------------------------------------------------
async def test_one_call_for_the_whole_report_and_the_verdict_shape(monkeypatch, stored):
judge = Judge(monkeypatch, {"u3": "unsupported"})
catalog = make_catalog(stored)
doc = compose_doc(RAW, catalog)
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
assert len(judge.calls) == 1, "一份报告合并成一次调用"
assert sorted(judge.judged[0]) == sorted(units_of(doc)[t] for t in (CITED, CITED2, CAUSAL, NUMBER))
v = status_of(outcome, doc, CITED)
assert v["status"] == "supported" and v["judge"] == PRICED and v["post_seal"] is False
assert v["rationale"] and v["used"] == ["m:gmv"]
assert outcome["counts"] == {"supported": 3, "partial": 0, "unsupported": 1, "not_a_claim": 0, "unjudged": 0}
assert outcome["model"] == PRICED and outcome["calls"] == 1 and outcome["gaps"] == []
assert units_of(doc)["下面看细节。"] not in outcome["verdicts"], "连接性的话不送裁判"
async def test_the_judge_model_runs_without_thinking(monkeypatch, stored):
judge = Judge(monkeypatch)
catalog = make_catalog(stored)
await judge_doc(compose_doc(RAW, catalog), catalog, spec=ModelSpec(model=PRICED, thinking="adaptive",
max_tokens=64000),
budget=Budget(max_claims=None, max_cost_usd=None, timeout_s=30, daily_max_usd=None))
[spec] = judge.specs
assert spec.thinking == "off" and spec.max_tokens == MAX_TOKENS == 4096
assert spec.model == PRICED and spec.timeout and spec.timeout <= 30
async def test_structured_output_failure_falls_back_to_json(monkeypatch, stored):
"""结构化输出失败(模型不支持、给的不是 JSON):退回纯文本调用,从回复里把 JSON 抠出来。"""
def fenced(units):
body = {"items": [{"unit": u, "verdict": "partial", "rationale": "只对了一半", "used": []} for u in units]}
return "判定如下:\n```json\n" + json.dumps(body, ensure_ascii=False) + "\n```"
judge = Judge(monkeypatch, structured="这不是 JSON", plain=fenced)
catalog = make_catalog(stored)
doc = compose_doc(RAW, catalog)
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
assert [c["kind"] for c in judge.calls] == ["structured", "plain"]
assert {v["status"] for v in outcome["verdicts"].values()} == {"partial"}
assert outcome["gaps"] == []
async def test_an_array_written_as_json_text_is_still_read(monkeypatch, stored):
"""有的模型走工具调用时把嵌套的数组写成一段 JSON 文本({"items": "[…]"}):照样认,不为它再花一次调用。"""
def stringified(units):
items = [{"unit": u, "verdict": "unsupported", "rationale": "对不上", "used": []} for u in units]
return json.dumps({"items": json.dumps(items, ensure_ascii=False)}, ensure_ascii=False)
judge = Judge(monkeypatch, structured=stringified)
catalog = make_catalog(stored)
doc = compose_doc(RAW, catalog)
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
assert [c["kind"] for c in judge.calls] == ["structured"]
assert {v["status"] for v in outcome["verdicts"].values()} == {"unsupported"}
async def test_a_broken_provider_is_not_called_batch_after_batch(monkeypatch, stored):
"""第一批就调不通(鉴权、网关挂了):后面几批不再一次次去撞,统一记未裁判。"""
judge = Judge(monkeypatch, fail="401 鉴权失败")
catalog = make_catalog(stored)
raw = "".join(f"第 {i} 个区域的销售额增长来自新客。[[see:m:gmv]]" for i in range(BATCH_SIZE + 5))
doc = compose_doc(raw, catalog)
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
assert len(judge.calls) == 2, "结构化一次、退回纯文本一次,第二批不再调用"
assert outcome["unjudged"] == {"error": BATCH_SIZE + 5} and "401 鉴权失败" in outcome["gaps"][0]
async def test_both_paths_failing_leaves_the_claims_unjudged(monkeypatch, stored):
Judge(monkeypatch, structured="不是 JSON", plain="还是不是")
catalog = make_catalog(stored)
doc = compose_doc(RAW, catalog)
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
assert {(v["status"], v["reason"]) for v in outcome["verdicts"].values()} == {("unjudged", "format")}
assert outcome["counts"]["unjudged"] == 4 and outcome["unjudged"] == {"format": 4}
assert outcome["gaps"] and "4 句" in outcome["gaps"][0]
async def test_bad_items_are_not_trusted(monkeypatch, stored):
"""判定不认识、理由太长、编出目录里没有的依据、漏判的句子:各按各的处理,不编判定。"""
catalog = make_catalog(stored)
doc = compose_doc(RAW, catalog)
ids = units_of(doc)
def messy(units):
return json.dumps({"items": [
{"unit": ids[CITED], "verdict": "supported", "rationale": "长" * 100, "used": ["m:gmv", "m:nope", "Q9"]},
{"unit": ids[CITED2], "verdict": "maybe", "rationale": "不认识的判定", "used": []},
{"unit": ids[CAUSAL], "verdict": "unsupported", "rationale": "没有新客维度", "used": []},
{"unit": "u999", "verdict": "supported", "rationale": "不存在的句子", "used": []},
]}, ensure_ascii=False)
Judge(monkeypatch, structured=messy)
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
v = outcome["verdicts"]
assert v[ids[CITED]]["rationale"] == "长" * RATIONALE_MAX and v[ids[CITED]]["used"] == ["m:gmv"]
assert (v[ids[CITED2]]["status"], v[ids[CITED2]]["reason"]) == ("unjudged", "format")
assert v[ids[CAUSAL]]["status"] == "unsupported"
assert (v[ids[NUMBER]]["status"], v[ids[NUMBER]]["reason"]) == ("unjudged", "missing")
assert "u999" not in v
async def test_a_failed_call_is_unjudged_not_an_error(monkeypatch, stored):
Judge(monkeypatch, fail="网关超时")
catalog = make_catalog(stored)
doc = compose_doc(RAW, catalog)
events: list = []
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED,
emit=lambda kind, **data: events.append((str(kind), data)))
assert {v["reason"] for v in outcome["verdicts"].values()} == {"error"}
assert "网关超时" in outcome["gaps"][0]
ends = [d for k, d in events if k == EventType.LLM_END]
assert ends and ends[-1]["purpose"] == "judge" and "网关超时" in ends[-1]["error"]
assert any(k == EventType.LOG and d["code"] == "judge_failed" for k, d in events)
# --------------------------------------------------------------------------
# 证据摘录
# --------------------------------------------------------------------------
def request_text(judge) -> str:
return "\n".join(str(m.content) for m in judge.calls[0]["messages"])
async def test_excerpts_follow_the_rules(monkeypatch, stored):
judge = Judge(monkeypatch)
catalog = make_catalog(stored)
raw = ("本周销售额 [[m:gmv]]。[[see:m:gmv]]"
"第 25 行的区域排在后面。[[see:Q1.r25]]"
"整体看华东领先。[[see:Q1]]"
"退款集中在物流延误的订单。[[see:K1]]"
"本周是 [[i:week]],增长来自新客。")
doc = compose_doc(raw, catalog)
await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
text = request_text(judge)
assert "数字已经由系统核对过,不要复算" in JUDGE_RULES and JUDGE_RULES in text
# 指标:名称 = 值,加原式
assert "销售额 = 45,678.5元" in text and "vars.kpi.gmv" in text
# 查询:SQL 前 300 字
assert LONG_SQL[:SQL_CHARS] in text and LONG_SQL[:SQL_CHARS + 20] not in text
# 被引用的行最多 20 行:点名的第 25 行先给,整份引用的再从头补满
rows = re.findall(r"^r(\d+):", text, re.M)
assert len(rows) == EXCERPT_ROWS and "25" in rows and "0" in rows and "22" not in rows
# 片段:最多 600 字
first = HITS["hits"][0]["content"]
assert first[:CHUNK_CHARS] in text and first[:CHUNK_CHARS + 1] not in text
assert "week = 2026-W37" in text
async def test_masked_columns_never_reach_the_judge(monkeypatch, stored):
judge = Judge(monkeypatch)
catalog = make_catalog(stored)
doc = compose_doc("整体看华东领先。[[see:Q1]]", catalog)
await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
assert "PHONE-0" in request_text(judge), "对照:没设遮罩时原值在摘录里"
# 数据源现在设的遮罩(列名不分大小写)
judge = Judge(monkeypatch)
await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED, masked={"shop": ["Phone"]})
text = request_text(judge)
assert "PHONE-" not in text and "区域0" in text
[header] = re.findall(r"^列:(.*)$", text, re.M)
assert "phone" not in header.lower() and "region" in header, "表头里也不给遮罩的列"
# 查询当时记下的遮罩(快照里的 mask_columns)
judge = Judge(monkeypatch)
catalog = make_catalog(stored, snap="masked")
doc = compose_doc("整体看华东领先。[[see:Q1]]", catalog)
await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
assert "PHONE-" not in request_text(judge)
#: 宽表:前 13 列是填充,被引用的列排在第 14、15 列,最后一列是手机号
WIDE_COLS = [f"c{i}" for i in range(13)] + ["refund_rate", "region", "phone"]
WIDE = {"columns": WIDE_COLS, "row_count": 5, "truncated": False, "elapsed_ms": 3, "source": "shop",
"sql": "SELECT * FROM wide_orders",
"rows": [[i * 100 + j for j in range(13)] + [f"RATE-{i}", f"区域{i}", f"PHONE-{i}"] for i in range(5)]}
def query_catalog(artifact: str, columns: list[str], *, source: str | None = "shop"):
ledger = [{"kind": "query", "node_id": "fetch", "exec": 1, "artifact": artifact, "tool": "db_query__shop",
**({"source": source} if source else {}), "columns": columns, "rows": 5, "truncated": False}]
return build_catalog(nodes={}, ledger=ledger)
def header_of(excerpt: str) -> tuple[list[str], str]:
"""(摘录给出的列, 表头那一整行)"""
[header] = re.findall(r"^列:(.*)$", excerpt, re.M)
return [c for c in header.split(" | ") if not c.startswith("…")], header
async def test_cited_columns_of_a_wide_table_always_reach_the_judge():
"""宽表只给前几列时,句子引用的列排在后面也一定给:裁判看不到被引用的格子,就只能判不支持。"""
catalog = query_catalog(_store(WIDE), WIDE_COLS)
doc = compose_doc("[[v:Q1.r2.region]] 的退款率最高,达到 [[v:Q1.r2.refund_rate]]。"
"手机尾号 [[v:Q1.r2.phone]] 的客户下单最多。", catalog)
excerpt = prepare(doc, catalog, masked={"shop": ["phone"]}).excerpts["Q1"]
shown, header = header_of(excerpt)
assert "region" in shown and "refund_rate" in shown, header
[row] = re.findall(r"^r2:(.*)$", excerpt, re.M)
cells = dict(zip(shown, row.split(" | "), strict=True))
assert cells["region"] == "区域2" and cells["refund_rate"] == "RATE-2"
# 遮罩的列被引用了也不给
assert "phone" not in header.lower() and "PHONE-" not in excerpt
# 没遮罩的 15 列给了 EXCERPT_COLS 列,「另有」按实际没给出的列数计
assert len(shown) == EXCERPT_COLS and header.endswith(f"…另有 {15 - EXCERPT_COLS} 列")
# 被引用的列比上限还多:一律保留
cited = "".join(f"[[v:Q1.r1.{c}]]、" for c in WIDE_COLS[:-1])
doc = compose_doc(f"这一行各列依次是 {cited}都比上周高。", catalog)
shown, header = header_of(prepare(doc, catalog).excerpts["Q1"])
assert set(shown) == set(WIDE_COLS[:-1]) and "另有 1 列" in header
async def test_current_masks_apply_when_the_entry_does_not_name_its_source(stored):
"""旧形状的台账条目没记数据源:按快照自己记的数据源找遮罩(和证据面板同一个口径),摘录里照样不给。"""
catalog = query_catalog(stored["snap"], SNAP["columns"], source=None)
doc = compose_doc("整体看华东领先。[[see:Q1]]", catalog)
excerpt = prepare(doc, catalog, masked={"shop": ["phone"]}).excerpts["Q1"]
assert "PHONE-" not in excerpt and "区域0" in excerpt and "数据源 shop" in excerpt
# 数据源现在设的遮罩也按快照里的数据源去查
from app.db.models import DataSource
catalog = query_catalog(_store({**SNAP, "source": "judge_mask_src"}), SNAP["columns"], source=None)
async with SessionLocal() as session:
session.add(DataSource(name="judge_mask_src", kind="sqlite", database="unused.db",
options={"mask_columns": "Phone"}))
await session.commit()
try:
assert await source_masks(catalog) == {"judge_mask_src": ["Phone"]}
finally:
async with SessionLocal() as session:
await session.execute(delete(DataSource).where(DataSource.name == "judge_mask_src"))
await session.commit()
# --------------------------------------------------------------------------
# 预算
# --------------------------------------------------------------------------
async def test_claims_over_max_claims_are_unjudged(monkeypatch, stored):
judge = Judge(monkeypatch)
catalog = make_catalog(stored)
doc = compose_doc(RAW, catalog)
outcome = await judge_doc(doc, catalog, spec=SPEC,
budget=Budget(max_claims=2, max_cost_usd=None, timeout_s=None, daily_max_usd=None))
ids = units_of(doc)
assert sorted(judge.judged[0]) == sorted([ids[CITED], ids[CITED2]]), "先判挂了依据的"
for text in (CAUSAL, NUMBER):
v = status_of(outcome, doc, text)
assert (v["status"], v["reason"]) == ("unjudged", "max_claims") and "已到上限" in v["rationale"]
assert outcome["limits_hit"] == ["max_claims"] and outcome["unjudged"] == {"max_claims": 2}
assert outcome["gaps"] and "已到上限" in outcome["gaps"][0]
async def test_cost_truncation_follows_priority(monkeypatch, stored):
catalog = make_catalog(stored)
doc = compose_doc(RAW, catalog)
request = prepare(doc, catalog)
by_text = {t: c for c in request.cands for t, u in units_of(doc).items() if u == c.unit}
without_p3 = request.cost(PRICED, [by_text[CITED], by_text[CITED2], by_text[CAUSAL]])
everything = request.cost(PRICED, request.cands)
assert 0 < without_p3 < everything
judge = Judge(monkeypatch)
budget = Budget(max_claims=None, max_cost_usd=(without_p3 + everything) / 2, timeout_s=None, daily_max_usd=None)
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=budget)
assert status_of(outcome, doc, NUMBER)["reason"] == "max_cost_usd"
assert all(status_of(outcome, doc, t)["status"] == "supported" for t in (CITED, CITED2, CAUSAL))
assert outcome["limits_hit"] == ["max_cost_usd"] and len(judge.judged[0]) == 3
cited_only = request.cost(PRICED, [by_text[CITED], by_text[CITED2]])
judge = Judge(monkeypatch)
budget = Budget(max_claims=None, max_cost_usd=cited_only * 1.001, timeout_s=None, daily_max_usd=None)
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=budget)
assert {status_of(outcome, doc, t)["reason"] for t in (CAUSAL, NUMBER)} == {"max_cost_usd"}
assert {status_of(outcome, doc, t)["status"] for t in (CITED, CITED2)} == {"supported"}
async def test_unlimited_never_truncates(monkeypatch, stored):
"""每一项都写 null:句数再多也全判(按每批上限分批),金额、时长、每日都不拦。"""
judge = Judge(monkeypatch)
catalog = make_catalog(stored)
raw = "".join(f"第 {i} 个区域的销售额增长来自新客。[[see:m:gmv]]" for i in range(BATCH_SIZE + 5))
doc = compose_doc(raw, catalog)
async with SessionLocal() as session: # 今天已经花掉很多:不限就是不限
session.add(Setting(key=SPEND_KEY, value={"date": judge_mod._today(), "usd": 999.0, "calls": 9}))
await session.commit()
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
assert [len(units) for units in judge.judged] == [BATCH_SIZE, 5]
assert outcome["counts"]["supported"] == BATCH_SIZE + 5 and outcome["counts"]["unjudged"] == 0
assert outcome["limits_hit"] == [] and outcome["gaps"] == []
async def test_the_daily_cap_resets_the_next_day(monkeypatch, stored):
catalog = make_catalog(stored)
doc = compose_doc(RAW, catalog)
cost = prepare(doc, catalog).cost(PRICED, prepare(doc, catalog).cands)
budget = Budget(max_claims=None, max_cost_usd=None, timeout_s=None, daily_max_usd=cost * 1.05)
day = {"today": "2026-09-28"}
monkeypatch.setattr(judge_mod, "_today", lambda: day["today"])
judge = Judge(monkeypatch)
first = await judge_doc(doc, catalog, spec=SPEC, budget=budget)
assert first["counts"]["unjudged"] == 0
spent = await daily_spend()
assert spent["date"] == "2026-09-28" and spent["usd"] == pytest.approx(cost) and spent["calls"] == 1
second = await judge_doc(doc, catalog, spec=SPEC, budget=budget)
assert {v["reason"] for v in second["verdicts"].values()} == {"daily_max_usd"}
assert second["limits_hit"] == ["daily_max_usd"] and len(judge.calls) == 1, "触顶就不再调用"
day["today"] = "2026-09-29"
third = await judge_doc(doc, catalog, spec=SPEC, budget=budget)
assert third["counts"]["unjudged"] == 0 and len(judge.calls) == 2
assert (await daily_spend())["date"] == "2026-09-29"
async def test_the_time_limit_stops_a_slow_judge(monkeypatch, stored):
Judge(monkeypatch, latency=2.0)
catalog = make_catalog(stored)
doc = compose_doc(RAW, catalog)
outcome = await judge_doc(doc, catalog, spec=SPEC,
budget=Budget(max_claims=None, max_cost_usd=None, timeout_s=0.2, daily_max_usd=None))
assert {v["reason"] for v in outcome["verdicts"].values()} == {"timeout_s"}
assert outcome["limits_hit"] == ["timeout_s"] and outcome["gaps"]
async def test_a_model_without_a_price_ignores_money_limits(monkeypatch, stored):
"""目录里没有价格的模型估不出金额:金额上限对它不起作用,并照实说「按令牌估不出金额」。"""
judge = Judge(monkeypatch, model_id="house-judge-model")
catalog = make_catalog(stored)
doc = compose_doc(RAW, catalog)
events: list = []
outcome = await judge_doc(doc, catalog, spec=ModelSpec(model="house-judge-model"),
budget=Budget(max_claims=None, max_cost_usd=1e-9, timeout_s=None, daily_max_usd=1e-9),
emit=lambda kind, **data: events.append((str(kind), data)))
assert len(judge.calls) == 1 and outcome["counts"]["unjudged"] == 0
assert outcome["priced"] is False and any("按令牌估不出金额" in n for n in outcome["notes"])
[log] = [d for k, d in events if k == EventType.LOG and d.get("code") == "judge_unpriced"]
assert "按令牌估不出金额" in log["message"]
# --------------------------------------------------------------------------
# 复用与按需
# --------------------------------------------------------------------------
async def test_known_sentences_are_not_judged_again(monkeypatch, stored):
judge = Judge(monkeypatch, {"u2": "unsupported"})
catalog = make_catalog(stored)
doc = compose_doc(RAW, catalog)
first = await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
known = {first["keys"][u]: v for u, v in first["verdicts"].items()}
again = await judge_doc(compose_doc(RAW, catalog), catalog, spec=SPEC, budget=UNLIMITED, known=known)
assert len(judge.calls) == 1 and again["reused"] == 4 and again["calls"] == 0
assert again["verdicts"] == first["verdicts"]
async def test_on_demand_judges_just_the_requested_sentence_after_the_seal(monkeypatch, stored):
judge = Judge(monkeypatch)
catalog = make_catalog(stored)
doc = compose_doc(RAW + "共 3 单。", catalog)
ids = units_of(doc)
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED, post_seal=True,
units=[ids[CAUSAL], ids["共 3 单。"], ids["本周概览"]])
assert sorted(judge.judged[0]) == sorted([ids[CAUSAL], ids["共 3 单。"]]), "点名要判的短句照判"
assert all(v["post_seal"] is True for v in outcome["verdicts"].values())
assert outcome["skipped"] == {ids["本周概览"]: "not_a_claim"}
+696
View File
@@ -0,0 +1,696 @@
"""按需裁判接口 POST /api/runs/{id}/evidence/judge(四期)。
探索运行不在节点里花裁判的钱:点开哪句才判哪句。
- 判定作为批注追加在封存之后(evidence.judged 事件,post_seal: true):封存核对只覆盖 manifest_seq 之前的
事件,所以追加之后 verify 照旧一致;封存的报告文档一个字都不改
- 只在探索运行里可用:正式运行的裁判在节点内完成,按需裁判返回 409;还没封存的、升级前发起的、封存链断了的
也不判
- 每次点击的上限、全局每日上限按设置控制;触顶不是报错,照实说「已到上限」和怎么调
- 同一次运行里同一句判过就直接给已有的判定,不再调用、不再花钱(连点两下也一样)
- 裁判只看得到封存范围内的证据:人为插进工件表的、封存之后才追加事件引用的快照,一行都不进摘录;
数据源遮罩的列同样不给
- 片段接口和证据图带上判定,封存内的和封存后追加的分得开
所有裁判都用 mock 模型给剧本,不调真接口。
"""
from __future__ import annotations
import asyncio
import json
import re
import sqlite3
from types import SimpleNamespace
import pytest
from httpx import ASGITransport, AsyncClient
from langchain_core.messages import AIMessage
from sqlalchemy import delete, select, update
from app.core import artifact_store
from app.core.events import EventType
from app.data.engine import engines
from app.db.base import SessionLocal
from app.db.models import DataSource, Run, RunEvent, Setting
from app.engine import judge as judge_mod
from app.engine import runner as runner_mod
from app.engine.evidence import iter_segments, iter_units
from app.engine.judge import JUDGE_ROLE, SPEND_KEY, daily_spend
from app.engine.runner import run_manager, verify_manifest
from app.main import app
from app.providers.mock_model import MockChatModel
PRICED = "claude-sonnet-5"
METRICS = [{"id": "gmv", "name": "销售额", "unit": "元", "decimals": 1, "expression": "vars.kpi.gmv"}]
#: 一句挂了依据的结论、一句没挂依据的归因、一句连接性的话
TEXT = "## 本周\n\n本周销售额 [[m:gmv]]。[[see:m:gmv]]增长主要来自新客首单。下面看细节。"
CITED, CAUSAL, LINK = "本周销售额 45,678.5元。", "增长主要来自新客首单。", "下面看细节。"
@pytest.fixture(autouse=True)
async def engine_up():
await run_manager.setup()
async with SessionLocal() as session:
await session.execute(delete(Setting).where(Setting.key.in_([SPEND_KEY, "judge", "copilot"])))
await session.commit()
yield
await run_manager.shutdown()
@pytest.fixture
async def client():
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as c:
yield c
def node(nid, ntype, **config):
return {"id": nid, "type": ntype, "position": {"x": 0, "y": 0}, "data": {"label": nid, "config": config}}
def chain(*nodes) -> dict:
ids = [n["id"] for n in nodes]
return {"nodes": list(nodes), "edges": [{"source": a, "target": b} for a, b in zip(ids, ids[1:])]}
def weekly(*, gate=False, **report) -> dict:
return chain(
node("start", "input", fields=[{"name": "week", "default": "2026-W37"}]),
node("fetch", "transform", mode="expression", expression="{'gmv': 45678.5}", assign_to="kpi"),
node("caliber", "metrics", caliber="周报口径", caliber_version="v2", metrics=METRICS),
node("write", "report", instructions="写周报", **report),
*([node("gate", "human", mode="approve", title="确认")] if gate else []),
node("out", "output", fields=[{"name": "周报", "value": "{{ nodes.write.text }}"}]),
)
class Models:
"""写作者回 text;裁判按句子里的字给判定(verdicts:{句中片段: 判定},没对上的是 supported)。
judges 记下每次交给裁判的请求原文,specs 记下每次要的裁判模型。"""
def __init__(self, monkeypatch, text=TEXT, verdicts=None, *, model=PRICED, fail_judge=None, delay=0.0):
self.judges: list[str] = []
self.specs: list = []
verdicts = verdicts or {}
def decide(chat, messages):
if JUDGE_ROLE in str(messages[0].content):
request = str(messages[-1].content)
self.judges.append(request)
if fail_judge:
raise RuntimeError(fail_judge)
items = [{"unit": uid,
"verdict": next((s for frag, s in verdicts.items() if frag in line), "supported"),
"rationale": f"判的是:{line.strip()[:16]}", "used": []}
for uid, line in re.findall(r"^\[(u\d+)\]([^\n]*)$", request, re.M)]
return AIMessage(content=json.dumps({"items": items}, ensure_ascii=False))
return AIMessage(content=text)
monkeypatch.setattr(MockChatModel, "_decide", decide)
async def judge_model_of(session, spec):
self.specs.append(spec)
if delay:
await asyncio.sleep(delay)
return MockChatModel(model_name=spec.model or model), spec.model or model
monkeypatch.setattr(judge_mod, "get_chat_model", judge_model_of)
async def wait(run_id: str, statuses=("succeeded", "failed")) -> Run:
for _ in range(400):
await asyncio.sleep(0.05)
async with SessionLocal() as session:
row = await session.get(Run, run_id)
if row.status in statuses and (row.status == "interrupted" or row.manifest_seq is not None):
return row
raise AssertionError(f"run {run_id} 没跑完:{row.status}")
async def finish(graph, *, run_class="exploratory", statuses=("succeeded", "failed")) -> Run:
run = await run_manager.start(graph=graph, input_payload={}, run_class=run_class)
row = await wait(run.id, statuses)
return row
def doc_of(row: Run) -> dict:
return artifact_store.load(row.output["_evidence"]["doc_artifact"])
def unit_ids(doc: dict) -> dict[str, str]:
md = doc["markdown"]
return {md[u["span"][0]:u["span"][1]].strip(): u["id"] for _, u in iter_units(doc)}
def unit_starting(doc: dict, prefix: str) -> str:
return next(uid for text, uid in unit_ids(doc).items() if text.startswith(prefix))
def seg_in(doc: dict, uid: str) -> str:
return next(s["id"] for _, u in iter_units(doc) if u["id"] == uid for s in u["segments"] if s["kind"] == "text")
async def judged_events(run_id: str) -> list[RunEvent]:
async with SessionLocal() as session:
return list((await session.execute(select(RunEvent).where(
RunEvent.run_id == run_id, RunEvent.type == EventType.EVIDENCE_JUDGED).order_by(RunEvent.seq))).scalars())
async def judge(client, run_id: str, units, **extra):
return await client.post(f"/api/runs/{run_id}/evidence/judge", json={"units": units, **extra})
async def save_setting(key: str, value) -> None:
async with SessionLocal() as session:
row = await session.get(Setting, key)
if row is None:
session.add(Setting(key=key, value=value))
else:
row.value = value
await session.commit()
# --------------------------------------------------------------------------
# 判定落在封存之后,封存核对照旧一致
# --------------------------------------------------------------------------
async def test_an_on_demand_verdict_lands_after_the_seal_and_the_seal_still_holds(client, monkeypatch):
models = Models(monkeypatch, verdicts={"新客": "unsupported"})
row = await finish(weekly(claims="judge"))
assert row.status == "succeeded" and models.judges == [], "探索运行节点里不判"
doc = doc_of(row)
ids = unit_ids(doc)
res = await judge(client, row.id, [ids[CAUSAL]])
assert res.status_code == 200, res.text
body = res.json()
verdict = body["verdicts"][ids[CAUSAL]]
assert verdict["status"] == "unsupported" and verdict["post_seal"] is True and verdict["judge"] == PRICED
assert verdict["rationale"] and body["judged"] == [ids[CAUSAL]] and body["reused"] == []
assert body["report"] == {"node_id": "write", "doc_artifact": row.output["_evidence"]["doc_artifact"]}
assert body["post_seal"] is True and body["limited"] is False and body["calls"] == 1
assert len(models.judges) == 1 and CAUSAL in models.judges[0] and CITED not in models.judges[0]
[event] = await judged_events(row.id)
assert event.seq > row.manifest_seq and event.node_id == "write" and body["event"] == {"seq": event.seq}
data = event.data
assert data["post_seal"] is True and data["report"] == "write"
assert data["doc_artifact"] == body["report"]["doc_artifact"]
assert data["verdicts"] == {ids[CAUSAL]: verdict} and data["model"] == PRICED and data["calls"] == 1
assert set(data["keys"]) == {ids[CAUSAL]} and data["units"] == [ids[CAUSAL]]
check = await verify_manifest(row.id)
assert check["ok"] is True and check["sealed_at"] == row.manifest_seq
assert (await client.get(f"/api/runs/{row.id}/verify")).json()["ok"] is True
async with SessionLocal() as session:
after = await session.get(Run, row.id)
assert after.manifest_seq == row.manifest_seq and after.last_seq == event.seq
assert doc_of(row) == doc, "封存的文档一个字都不改"
csv = await client.get(f"/api/runs/{row.id}/evidence/audit", params={"format": "csv"})
assert csv.headers["x-evidence-seal"] == "ok"
async def test_any_claim_in_an_explore_report_can_be_judged(client, monkeypatch):
"""报告节点没开 claims: judge 也能按需判:结论句都是候选。"""
models = Models(monkeypatch)
row = await finish(weekly())
ids = unit_ids(doc_of(row))
body = (await judge(client, row.id, [ids[CITED], ids[CAUSAL]])).json()
assert {u: v["status"] for u, v in body["verdicts"].items()} == {ids[CITED]: "supported", ids[CAUSAL]: "supported"}
assert len(models.judges) == 1, "一次点击的几句合成一次调用"
assert "【m:gmv】" in models.judges[0], "挂的依据摘给裁判"
async def test_sequence_numbers_continue_after_a_restart(client, monkeypatch):
"""进程重启后内存里的序号从 0 起:追加的事件要接在库里最后一条后面,不能和已有的撞号。"""
from app.core.bus import bus
Models(monkeypatch)
row = await finish(weekly())
ids = unit_ids(doc_of(row))
bus._seq.pop(row.id, None)
await judge(client, row.id, [ids[CAUSAL]])
[event] = await judged_events(row.id)
async with SessionLocal() as session:
seqs = [s for (s,) in await session.execute(select(RunEvent.seq).where(RunEvent.run_id == row.id))]
assert event.seq == max(seqs) and seqs.count(event.seq) == 1 and event.seq > row.manifest_seq
# --------------------------------------------------------------------------
# 同一句不重复判
# --------------------------------------------------------------------------
async def test_asking_again_returns_the_verdict_without_another_call(client, monkeypatch):
models = Models(monkeypatch, verdicts={"新客": "partial"})
row = await finish(weekly(claims="judge"))
ids = unit_ids(doc_of(row))
first = (await judge(client, row.id, [ids[CAUSAL]])).json()
spent = (await daily_spend())["calls"]
again = (await judge(client, row.id, [ids[CAUSAL]])).json()
assert len(models.judges) == 1 and (await daily_spend())["calls"] == spent, "不再调用、不再计费"
assert again["verdicts"] == first["verdicts"] and again["reused"] == [ids[CAUSAL]] and again["judged"] == []
assert again["calls"] == 0 and again["event"] is None and again["message"] is None
assert len(await judged_events(row.id)) == 1
# 判过的和没判过的一起点:只判没判过的
both = (await judge(client, row.id, [ids[CAUSAL], ids[CITED]])).json()
assert both["reused"] == [ids[CAUSAL]] and both["judged"] == [ids[CITED]]
assert len(models.judges) == 2 and CAUSAL not in models.judges[1]
async def test_a_double_click_is_judged_once(client, monkeypatch):
models = Models(monkeypatch, delay=0.05)
row = await finish(weekly(claims="judge"))
uid = unit_ids(doc_of(row))[CAUSAL]
one, two = await asyncio.gather(judge(client, row.id, [uid]), judge(client, row.id, [uid]))
assert one.status_code == two.status_code == 200
assert len(models.judges) == 1 and len(await judged_events(row.id)) == 1
assert sorted([one.json()["reused"], two.json()["reused"]]) == [[], [uid]]
async def test_a_failed_attempt_can_be_retried(client, monkeypatch):
Models(monkeypatch, fail_judge="网关 502")
row = await finish(weekly(claims="judge"))
uid = unit_ids(doc_of(row))[CAUSAL]
failed = (await judge(client, row.id, [uid])).json()
assert failed["verdicts"][uid]["status"] == "unjudged" and failed["verdicts"][uid]["reason"] == "error"
assert "网关 502" in failed["message"] and failed["limited"] is False
models = Models(monkeypatch, verdicts={"新客": "unsupported"})
retried = (await judge(client, row.id, [uid])).json()
assert retried["verdicts"][uid]["status"] == "unsupported" and retried["reused"] == [] and len(models.judges) == 1
# --------------------------------------------------------------------------
# 上限
# --------------------------------------------------------------------------
async def test_hitting_the_click_limit_is_a_result_not_an_error(client, monkeypatch):
models = Models(monkeypatch)
await save_setting("judge", {"click_max_cost_usd": 1e-9})
row = await finish(weekly(claims="judge"))
uid = unit_ids(doc_of(row))[CAUSAL]
res = await judge(client, row.id, [uid])
assert res.status_code == 200
body = res.json()
assert body["limited"] is True and body["limits_hit"] == ["max_cost_usd"]
assert body["unjudged"] == {"max_cost_usd": 1}
assert body["message"].startswith("已到上限") and "这次点击" in body["message"]
assert any("设置" in how and "不限" in how for how in body["adjust"])
verdict = body["verdicts"][uid]
assert verdict["status"] == "unjudged" and verdict["reason"] == "max_cost_usd" and verdict["post_seal"] is True
assert models.judges == [] and body["calls"] == 0 and body["budget"]["max_cost_usd"] == 1e-9
assert body["event"] is None and await judged_events(row.id) == [], "没问模型就没有判定可记"
async def test_the_daily_limit_applies_to_clicks_too(client, monkeypatch):
models = Models(monkeypatch)
today = (await daily_spend())["date"]
await save_setting(SPEND_KEY, {"date": today, "usd": 2.0, "calls": 9, "unpriced_calls": 0})
row = await finish(weekly(claims="judge"))
body = (await judge(client, row.id, [unit_ids(doc_of(row))[CAUSAL]])).json()
assert body["limits_hit"] == ["daily_max_usd"] and "每日" in body["message"] and models.judges == []
assert body["spend"]["usd"] == 2.0 and body["spend"]["daily_max_usd"] == 2.0
async def test_unlimited_click_budget_judges(client, monkeypatch):
models = Models(monkeypatch)
await save_setting("judge", {"click_max_cost_usd": None, "daily_max_usd": None})
row = await finish(weekly(claims="judge"))
body = (await judge(client, row.id, [unit_ids(doc_of(row))[CAUSAL]])).json()
assert body["limited"] is False and len(models.judges) == 1
assert body["budget"]["max_cost_usd"] is None and body["budget"]["daily_max_usd"] is None
assert body["spend"]["calls"] == 1 and body["spend"]["daily_max_usd"] is None
async def test_the_sentence_limit_keeps_what_was_judged(client, monkeypatch):
models = Models(monkeypatch)
row = await finish(weekly(claims="judge", judge={"max_claims": 1}))
ids = unit_ids(doc_of(row))
body = (await judge(client, row.id, [ids[CITED], ids[CAUSAL]])).json()
statuses = {u: v["status"] for u, v in body["verdicts"].items()}
assert statuses == {ids[CITED]: "supported", ids[CAUSAL]: "unjudged"} and len(models.judges) == 1
assert body["limits_hit"] == ["max_claims"] and body["verdicts"][ids[CAUSAL]]["reason"] == "max_claims"
[event] = await judged_events(row.id)
assert set(event.data["verdicts"]) == {ids[CITED], ids[CAUSAL]}, "问过模型的这一次连同没判的一起记下"
# 没判的那句还能再点
again = (await judge(client, row.id, [ids[CAUSAL]])).json()
assert again["judged"] == [ids[CAUSAL]] and again["reused"] == []
# --------------------------------------------------------------------------
# 用哪个模型
# --------------------------------------------------------------------------
async def test_the_judge_model_follows_node_then_settings(client, monkeypatch):
models = Models(monkeypatch)
await save_setting("judge", {"model": "judge-from-settings"})
row = await finish(weekly(claims="judge"))
await judge(client, row.id, [unit_ids(doc_of(row))[CAUSAL]])
[spec] = models.specs
assert spec.model == "judge-from-settings" and spec.thinking == "off" and spec.max_tokens == 4096
row = await finish(weekly(claims="judge", judge={"model": "judge-from-node"}))
body = (await judge(client, row.id, [unit_ids(doc_of(row))[CAUSAL]])).json()
assert models.specs[-1].model == "judge-from-node" and body["model"] == "judge-from-node"
# --------------------------------------------------------------------------
# 什么时候不判
# --------------------------------------------------------------------------
async def test_formal_runs_are_judged_inside_the_node_not_on_demand(client, monkeypatch):
models = Models(monkeypatch)
row = await finish(weekly(claims="judge", judge={"max_cost_usd": 0.05}), run_class="formal")
assert row.status == "succeeded", row.error
calls = len(models.judges)
res = await judge(client, row.id, [unit_ids(doc_of(row))[CAUSAL]])
assert res.status_code == 409 and res.json()["code"] == "evidence_judge_formal"
assert "正式运行的裁判在节点内完成" in res.json()["detail"]
assert len(models.judges) == calls and await judged_events(row.id) == []
async def test_a_run_paused_at_an_approval_is_not_judged(client, monkeypatch):
models = Models(monkeypatch)
row = await finish(weekly(claims="judge", gate=True), statuses=("interrupted",))
res = await judge(client, row.id, ["u1"])
assert res.status_code == 409 and res.json()["code"] == "evidence_judge_unsealed"
assert models.judges == [] and await judged_events(row.id) == []
async def test_runs_from_before_the_upgrade_are_not_judged(client, monkeypatch):
async def no_snapshot(session):
return None
monkeypatch.setattr(runner_mod, "_agent_limits", no_snapshot)
models = Models(monkeypatch)
row = await finish(weekly())
res = await judge(client, row.id, [unit_ids(doc_of(row))[CAUSAL]])
assert res.status_code == 409 and res.json()["code"] == "evidence_judge_legacy"
assert models.judges == [] and await judged_events(row.id) == []
async def test_a_broken_seal_is_not_judged(client, monkeypatch):
models = Models(monkeypatch)
row = await finish(weekly(claims="judge"))
async with SessionLocal() as session:
event = (await session.execute(select(RunEvent).where(
RunEvent.run_id == row.id, RunEvent.type == "node.finished", RunEvent.node_id == "fetch"))).scalar_one()
event.data = {**event.data, "tampered": True}
await session.commit()
res = await judge(client, row.id, [unit_ids(doc_of(row))[CAUSAL]])
assert res.status_code == 409 and res.json()["code"] == "evidence_seal_broken" and models.judges == []
@pytest.mark.parametrize("meanwhile", [
# 裁判的这几秒里有人点了「继续运行」(失败的运行可以接着跑):运行又活了
{"status": "running", "error": None},
# 接着跑完、重新封存了:manifest_seq 和清单哈希都换了,这次点击依据的那份封存已经不是最新的
{"manifest_seq": 10_000, "manifest_hash": "0" * 64},
])
async def test_a_run_continued_during_the_judge_call_records_no_event(client, monkeypatch, meanwhile):
"""判定只接在点击开始时的那份封存后面。运行在裁判期间被接着跑了,再追加就落进一次活着的运行中间,
还会被封进下一份清单:不记进运行记录,判定照样交回给点的人,并说明为什么没记。"""
models = Models(monkeypatch, verdicts={"新客": "unsupported"})
row = await finish(weekly(claims="judge"))
uid = unit_ids(doc_of(row))[CAUSAL]
ask_model = judge_mod.get_chat_model
async def continued_while_judging(session, spec):
async with SessionLocal() as other:
await other.execute(update(Run).where(Run.id == row.id).values(**meanwhile))
await other.commit()
return await ask_model(session, spec)
monkeypatch.setattr(judge_mod, "get_chat_model", continued_while_judging)
res = await judge(client, row.id, [uid])
assert res.status_code == 200, res.text
body = res.json()
assert len(models.judges) == 1 and body["calls"] == 1, "模型问过了"
assert body["verdicts"][uid]["status"] == "unsupported" and body["judged"] == [uid], "判定照样交回"
assert body["event"] is None and "接着跑" in body["message"] and "没有记进运行记录" in body["message"]
assert await judged_events(row.id) == []
async with SessionLocal() as session:
after = await session.get(Run, row.id)
assert after.last_seq == row.last_seq, "运行记录一条都没多"
async def test_a_continue_that_wins_the_row_between_check_and_write_gets_no_event(client, monkeypatch):
"""核对过状态、还没写入的那一下,接着跑的条件更新(runner._claim)先提交了:追加判定的条件更新落空,
照样不记——核对和写入是同一行上的条件更新,不是先读后写。"""
from app.core.bus import bus
from app.core.config import settings
Models(monkeypatch, verdicts={"新客": "unsupported"})
row = await finish(weekly(claims="judge"))
uid = unit_ids(doc_of(row))[CAUSAL]
set_seq = bus.set_seq
def claimed_in_between(run_id, value):
# _append_judged 读完运行、分配序号之前正好调用它:在这里用另一条连接把运行占成 running
if run_id == row.id:
db = sqlite3.connect(settings.db_path, timeout=5)
try:
db.execute("UPDATE runs SET status = 'running' WHERE id = ?", (row.id,))
db.commit()
finally:
db.close()
set_seq(run_id, value)
monkeypatch.setattr(bus, "set_seq", claimed_in_between)
body = (await judge(client, row.id, [uid])).json()
assert body["verdicts"][uid]["status"] == "unsupported" and body["event"] is None
assert "没有记进运行记录" in body["message"] and await judged_events(row.id) == []
async with SessionLocal() as session:
after = await session.get(Run, row.id)
assert after.status == "running" and after.last_seq == row.last_seq
async def test_missing_runs_and_reports(client, monkeypatch):
Models(monkeypatch)
res = await judge(client, "nope", ["u1"])
assert res.status_code == 404 and res.json()["code"] == "run_not_found"
row = await finish(chain(node("start", "input"), node("out", "output", fields=[{"name": "a", "value": "x"}])))
res = await judge(client, row.id, ["u1"])
assert res.status_code == 404 and res.json()["code"] == "evidence_report_not_found"
@pytest.mark.parametrize("payload", [{}, {"units": []}, {"units": "u1"}, {"units": [1]}, {"units": [""]},
{"units": [f"u{i}" for i in range(201)]}, {"units": ["u1"], "report": 3}])
async def test_bad_requests_are_422(client, monkeypatch, payload):
Models(monkeypatch)
row = await finish(weekly())
res = await client.post(f"/api/runs/{row.id}/evidence/judge", json=payload)
assert res.status_code == 422 and res.json()["code"] == "evidence_judge_bad_units"
async def test_sentences_that_are_not_claims_are_skipped(client, monkeypatch):
models = Models(monkeypatch)
row = await finish(weekly(claims="judge"))
ids = unit_ids(doc_of(row))
body = (await judge(client, row.id, [ids["本周"], ids[LINK], "u999"])).json()
assert body["skipped"] == {ids["本周"]: "not_a_claim", ids[LINK]: "not_a_claim", "u999": "not_found"}
assert body["verdicts"] == {} and models.judges == [] and body["event"] is None
assert "不是结论句" in body["message"]
# --------------------------------------------------------------------------
# 片段接口、证据图带上判定
# --------------------------------------------------------------------------
async def test_the_segment_shows_the_verdict_and_whether_it_can_still_be_asked(client, monkeypatch):
Models(monkeypatch, verdicts={"新客": "unsupported"})
row = await finish(weekly(claims="judge"))
doc = doc_of(row)
ids = unit_ids(doc)
before = (await client.get(f"/api/runs/{row.id}/evidence/segments/{seg_in(doc, ids[CAUSAL])}")).json()
assert before["unit"]["verdict"]["status"] == "unjudged" and before["unit"]["verdict"]["reason"] == "on_demand"
assert before["unit"]["on_demand"] == {"available": True, "reason": None, "message": None}
await judge(client, row.id, [ids[CAUSAL]])
after = (await client.get(f"/api/runs/{row.id}/evidence/segments/{seg_in(doc, ids[CAUSAL])}")).json()
assert after["unit"]["verdict"]["status"] == "unsupported" and after["unit"]["verdict"]["post_seal"] is True
assert after["unit"]["on_demand"]["available"] is False and after["unit"]["on_demand"]["reason"] == "judged"
assert "模型" in after["note"] and "后续版本" not in after["note"]
other = (await client.get(f"/api/runs/{row.id}/evidence/segments/{seg_in(doc, ids[CITED])}")).json()
assert other["unit"]["on_demand"]["available"] is True
assert "挂了依据" in other["note"] and "模型" in other["note"] and "后续版本" not in other["note"]
link = (await client.get(f"/api/runs/{row.id}/evidence/segments/{seg_in(doc, ids[LINK])}")).json()
assert link["unit"]["on_demand"]["reason"] == "not_a_claim" and "verdict" not in link["unit"]
graph = (await client.get(f"/api/runs/{row.id}/evidence")).json()
[report] = graph["reports"]
assert report["claims"] == "judge" and report["judge"]["mode"] == "on_demand"
assert report["post_seal_verdicts"] == {ids[CAUSAL]: after["unit"]["verdict"]}
audit = (await client.get(f"/api/runs/{row.id}/evidence/audit")).json()
assert audit["reports"][0]["post_seal_verdicts"] == report["post_seal_verdicts"]
async def test_sealed_verdicts_of_a_formal_run_are_shown_as_sealed(client, monkeypatch):
Models(monkeypatch, verdicts={"新客": "unsupported"})
row = await finish(weekly(claims="judge", judge={"max_cost_usd": 0.05}), run_class="formal")
doc = doc_of(row)
body = (await client.get(f"/api/runs/{row.id}/evidence/segments/{seg_in(doc, unit_ids(doc)[CAUSAL])}")).json()
assert body["unit"]["verdict"]["status"] == "unsupported" and body["unit"]["verdict"]["post_seal"] is False
assert body["unit"]["on_demand"]["reason"] == "formal"
graph = (await client.get(f"/api/runs/{row.id}/evidence")).json()
assert graph["reports"][0]["post_seal_verdicts"] == {} and graph["reports"][0]["judge"]["mode"] == "inline"
async def test_a_forged_event_cannot_overwrite_a_sealed_verdict(client, monkeypatch):
"""封存后追加的判定只填补没判过的句子:封存内判过的,后面追加什么都盖不掉。"""
Models(monkeypatch, verdicts={"新客": "unsupported"})
row = await finish(weekly(claims="judge", judge={"max_cost_usd": 0.05}), run_class="formal")
doc = doc_of(row)
uid = unit_ids(doc)[CAUSAL]
doc_artifact = row.output["_evidence"]["doc_artifact"]
await run_manager.note(row.id, EventType.EVIDENCE_JUDGED, node_id="write", report="write",
doc_artifact=doc_artifact, post_seal=True,
verdicts={uid: {"status": "supported", "rationale": "改好了", "judge": "x", "post_seal": True}})
body = (await client.get(f"/api/runs/{row.id}/evidence/segments/{seg_in(doc, uid)}")).json()
assert body["unit"]["verdict"]["status"] == "unsupported"
async def test_appended_verdicts_always_read_as_post_seal(client, monkeypatch):
"""追加的判定不管载荷里怎么写,展示出来都是「封存后追加」:只有随文档封存的判定才算封存内的。"""
Models(monkeypatch)
row = await finish(weekly(claims="judge"))
doc = doc_of(row)
uid = unit_ids(doc)[CAUSAL]
await run_manager.note(row.id, EventType.EVIDENCE_JUDGED, node_id="write", report="write",
doc_artifact=row.output["_evidence"]["doc_artifact"],
verdicts={uid: {"status": "supported", "rationale": "x", "judge": "x", "post_seal": False},
"u999": {"status": "supported", "rationale": "x", "judge": "x"}})
body = (await client.get(f"/api/runs/{row.id}/evidence/segments/{seg_in(doc, uid)}")).json()
assert body["unit"]["verdict"]["status"] == "supported" and body["unit"]["verdict"]["post_seal"] is True
graph = (await client.get(f"/api/runs/{row.id}/evidence")).json()
assert set(graph["reports"][0]["post_seal_verdicts"]) == {uid}, "文档里没有的句子不认"
# --------------------------------------------------------------------------
# 裁判只看得到封存范围内的证据
# --------------------------------------------------------------------------
SOURCE = "warehouse"
TOOL = f"db_query__{SOURCE}"
SQL = "SELECT region, SUM(amount) AS gmv, COUNT(*) AS order_cnt FROM orders GROUP BY region ORDER BY gmv DESC"
EAST, WEST = "东区销售额 [[v:Q1.r0.gmv]]。[[see:Q1]]", "西区销售额 [[v:Q1.r1.gmv]]。[[see:Q1]]"
@pytest.fixture
async def warehouse(tmp_path):
from app.data.introspect import introspect
path = tmp_path / "warehouse.db"
db = sqlite3.connect(path)
db.executescript("CREATE TABLE orders (id INTEGER PRIMARY KEY, region TEXT, amount REAL);")
db.executemany("INSERT INTO orders VALUES (?, ?, ?)", [(1, "east", 100.5), (2, "west", 200.0), (3, "east", 300.0)])
db.commit()
db.close()
probe = SimpleNamespace(id=f"probe-{path.name}", name=SOURCE, kind="sqlite", database=str(path), host=None,
port=None, username=None, password=None, options={}, readonly=True, schema_cache={})
cache = await introspect(probe)
await engines.invalidate(probe.id)
async with SessionLocal() as session:
row = (await session.execute(select(DataSource).where(DataSource.name == SOURCE))).scalar_one_or_none()
if row is None:
row = DataSource(name=SOURCE, kind="sqlite", database=str(path), readonly=True, enabled=True)
session.add(row)
row.database, row.kind, row.enabled, row.readonly, row.schema_cache = str(path), "sqlite", True, True, cache
row.options = {}
await session.commit()
source_id = row.id
await engines.invalidate(source_id)
yield source_id
await engines.invalidate(source_id)
def queried() -> dict:
return chain(node("start", "input"), node("fetch", "tool", tool=TOOL, args={"sql": SQL}),
node("write", "report", instructions="写周报"),
node("out", "output", fields=[{"name": "周报", "value": "{{ nodes.write.text }}"}]))
async def reseal(run_id: str) -> None:
from app.core.artifact_store import manifest_hash
async with SessionLocal() as session:
run = await session.get(Run, run_id)
rows = [tuple(r) for r in await session.execute(
select(RunEvent.seq, RunEvent.type, RunEvent.node_id, RunEvent.data)
.where(RunEvent.run_id == run_id, RunEvent.seq <= run.manifest_seq).order_by(RunEvent.seq))]
run.manifest_hash = manifest_hash(rows)
await session.commit()
async def unseal_queries(run_id: str) -> str:
"""把查询快照从封存的事件里摘掉、按改过的事件重新封存:快照还在工件库和工件表里,封存链完好,
只是没有哪条封存的事件引用它(等于「人为插进工件表」)。返回那件快照的 id。"""
async with SessionLocal() as session:
rows = list((await session.execute(select(RunEvent).where(
RunEvent.run_id == run_id, RunEvent.type.in_(["tool.end", "node.finished"])))).scalars())
artifact = None
for event in rows:
data = dict(event.data or {})
if event.type == "tool.end":
artifact = data.pop("query_artifact", None) or artifact
data.pop("schema_artifact", None)
elif data.get("evidence"):
data["evidence"] = [e for e in data["evidence"] if e.get("kind") not in ("query", "schema")]
event.data = data
await session.commit()
await reseal(run_id)
return artifact
def excerpt_lines(request: str) -> tuple[str | None, list[str]]:
header = re.search(r"^列:(.*)$", request, re.M)
return (header.group(1) if header else None), re.findall(r"^r\d+:.*$", request, re.M)
async def test_only_sealed_snapshots_reach_the_judge(client, monkeypatch, warehouse):
models = Models(monkeypatch, text=EAST + WEST)
control = await finish(queried())
assert control.status == "succeeded", control.error
await judge(client, control.id, [unit_starting(doc_of(control), "东区")])
assert "【Q1】查询结果" in models.judges[0] and "SQL:SELECT region" in models.judges[0]
assert excerpt_lines(models.judges[0])[1], "封存链完好时照常摘被引用的行"
row = await finish(queried())
doc = doc_of(row)
artifact = await unseal_queries(row.id)
assert artifact and artifact_store.load(artifact) is not None, "快照本身还在工件库里"
assert (await verify_manifest(row.id))["ok"] is True
res = await judge(client, row.id, [unit_starting(doc, "东区")])
assert res.status_code == 200, res.text
assert "【Q1】" not in models.judges[1] and "SELECT" not in models.judges[1]
assert excerpt_lines(models.judges[1]) == (None, [])
# 封存之后才追加的事件引用了它,也不算
async with SessionLocal() as session:
run = await session.get(Run, row.id)
seq = run.last_seq
from app.core.bus import bus
bus.set_seq(row.id, seq)
await run_manager.note(row.id, "tool.end", node_id="fetch", tool=TOOL, query_artifact=artifact)
await judge(client, row.id, [unit_starting(doc, "西区")])
assert "【Q1】" not in models.judges[2] and "SELECT" not in models.judges[2]
async def test_masked_columns_never_reach_the_judge(client, monkeypatch, warehouse):
models = Models(monkeypatch, text=EAST)
row = await finish(queried())
async with SessionLocal() as session:
source = await session.get(DataSource, warehouse)
source.options = {"mask_columns": ["order_cnt"]}
await session.commit()
await judge(client, row.id, [unit_starting(doc_of(row), "东区")])
header, lines = excerpt_lines(models.judges[0])
assert header is not None and "order_cnt" not in header and "gmv" in header
assert lines and all(line.count("|") == 1 for line in lines), lines
assert "遮罩" in models.judges[0]
+344
View File
@@ -0,0 +1,344 @@
"""四期:受管模板放开 claims: judge(结论句由另一个模型按证据逐句裁判),但要写明预算。
- G3:报告撰写节点的 claims 在受管级别认 require_citation 或 judge;写 judge 的,生效的 judge 配置里
必须有 max_cost_usd 这个键——写金额,或者显式写 null 表示不限。没写就是 error(已发布级别只给警告):
受管出具不能让每份报告的裁判花费悄悄按默认值走,也不能悄悄不设上限
- 发布前自动修复:缺预算是 choice,候选是默认金额和「不限」,不替人选;G3 另外那几项的一键修复不再把
judge 改回 require_citation
- 降低要求的判断:require_citation → judge 是往严里改;judge → require_citation / off / 删掉都算放宽,
报告节点上和出具契约里一样
"""
from __future__ import annotations
import copy
import pytest
from app.engine.autofix import NEED_CHOICE, apply_fixes, forbidden_changes, plan_fixes, public
from app.engine.governance import lint_for_publish, publish_issues
from app.engine.judge import JUDGE_DEFAULTS
from app.engine.schema import GraphSpec, validate_graph
STRICT = {"numbers": "strict", "on_violation": "fail", "claims": "require_citation"}
JUDGED = {**STRICT, "claims": "judge", "judge": {"max_cost_usd": 0.05}}
BUDGET = "governed.judge_budget"
def node(nid, ntype, label=None, **config):
return {"id": nid, "type": ntype, "position": {"x": 10, "y": 20},
"data": {"label": label or nid, "config": config}}
def traceable(*, report=None, defaults=None, **contract_overrides) -> dict:
"""input → 查库 → 口径卡 → 报告撰写 → 出具:受管要求(含 G1–G5)全部满足。"""
contract = {"report_from": "write", "metrics_from": ["card"], "required": ["gmv"], "strict": True}
contract.update(contract_overrides)
nodes = [
node("start", "input", "周期"),
node("fetch", "tool", "取数", tool="db_query__shop",
args={"sql": "SELECT SUM(amount) AS gmv FROM orders"}),
node("card", "metrics", "周报口径卡", caliber="周报口径", metrics=[
{"id": "gmv", "name": "销售额", "unit": "元", "expression": "cell(nodes.fetch, 0, 'gmv')"}]),
node("write", "report", "报告撰写", instructions="写周报", **(JUDGED if report is None else report)),
node("done", "output", "出具", fields=[{"name": "周报", "value": "{{ nodes.write.text }}"}],
contract={k: v for k, v in contract.items() if v is not None}),
]
edges = [{"id": f"e{i}", "source": a["id"], "target": b["id"]} for i, (a, b) in enumerate(zip(nodes, nodes[1:]))]
graph = {"nodes": nodes, "edges": edges}
if defaults is not None:
graph["defaults"] = defaults
return graph
def config(graph: dict, nid: str) -> dict:
return next(n for n in graph["nodes"] if n["id"] == nid)["data"]["config"]
def by_code(graph: dict, code: str, level: str = "governed") -> list:
return [i for i in lint_for_publish(GraphSpec.model_validate(graph), level=level).issues if i.code == code]
def errors(graph: dict, level: str = "governed") -> list:
return [i for i in publish_issues(GraphSpec.model_validate(graph), level=level) if i.level == "error"]
def plan(graph: dict, level: str = "governed") -> dict[str, dict]:
spec = GraphSpec.model_validate(graph)
return {f["id"]: f for f in plan_fixes(spec, publish_issues(spec, level=level), level=level)}
# --------------------------------------------------------------------------
# G3:judge 可以发布成受管,但要写预算
# --------------------------------------------------------------------------
@pytest.mark.parametrize("budget", [0.05, 1, None])
def test_judge_with_a_budget_publishes_as_governed(budget):
graph = traceable(report={**JUDGED, "judge": {"max_cost_usd": budget, "max_claims": 20}})
assert errors(graph) == [], [i.message for i in errors(graph)]
assert by_code(graph, "governed.report_policy") == [] and by_code(graph, BUDGET) == []
@pytest.mark.parametrize("judge", [None, {}, {"max_claims": 20, "model": "judge-model"}])
def test_judge_without_a_budget_is_an_error_at_governed(judge):
report = {**STRICT, "claims": "judge", **({"judge": judge} if judge is not None else {})}
graph = traceable(report=report)
[issue] = by_code(graph, BUDGET)
assert issue.level == "error" and issue.node_id == "write" and issue.field == "judge.max_cost_usd"
assert "max_cost_usd" in issue.message and "不限" in issue.message and "null" in issue.message
assert by_code(graph, "governed.report_policy") == [], "judge 本身合规,只缺预算"
assert [i.level for i in by_code(graph, BUDGET, "published")] == ["warning"]
def test_the_budget_follows_the_same_config_rule_as_run_time():
"""和运行时同一个取值规则:节点没写 judge 取图级 defaults;节点写了就整个用节点的(不和 defaults 拼)。"""
graph = traceable(report={**STRICT, "claims": "judge"}, defaults={"judge": {"max_cost_usd": 0.02}})
assert by_code(graph, BUDGET) == []
graph = traceable(report={**STRICT, "claims": "judge", "judge": {"max_claims": 5}},
defaults={"judge": {"max_cost_usd": 0.02}})
assert [i.field for i in by_code(graph, BUDGET)] == ["judge.max_cost_usd"]
# claims 本身也可以来自图级 defaults
graph = traceable(report={"numbers": "strict", "on_violation": "fail"},
defaults={"claims": "judge", "judge": {"max_cost_usd": None}})
assert by_code(graph, BUDGET) == [] and by_code(graph, "governed.report_policy") == []
def test_a_malformed_budget_is_left_to_validate():
graph = traceable(report={**STRICT, "claims": "judge", "judge": {"max_cost_usd": "cheap"}})
assert by_code(graph, BUDGET) == [], "写错的值由 validate 的 report.judge_invalid 报,同一处不说两遍"
[bad] = [i for i in validate_graph(GraphSpec.model_validate(graph)).issues if i.code == "report.judge_invalid"]
assert bad.field == "judge.max_cost_usd" and bad.level == "error"
@pytest.mark.parametrize("claims", [None, "off"])
def test_a_loose_claims_setting_names_both_accepted_values(claims):
report = {"numbers": "strict", "on_violation": "fail", **({"claims": claims} if claims else {})}
[issue] = by_code(traceable(report=report), "governed.report_policy")
assert issue.field == "claims" and "claims: require_citation" in issue.message and "judge" in issue.message
def test_judge_needs_no_other_report_policy_fix():
graph = traceable(report={**JUDGED, "numbers": "off"})
[issue] = by_code(graph, "governed.report_policy")
assert issue.field == "numbers" and "claims" not in issue.message
# --------------------------------------------------------------------------
# 发布前自动修复
# --------------------------------------------------------------------------
def test_the_report_policy_fix_leaves_judge_alone():
graph = traceable(report={**JUDGED, "numbers": "off", "on_violation": "flag"})
fix = plan(graph)["governed.report_policy:write"]
assert fix["kind"] == "auto" and fix["preview"]["after"] == {"numbers": "strict", "on_violation": "fail"}
out = apply_fixes(graph, [fix["id"]], level="governed")
assert out["ok"] is True and config(out["graph"], "write")["claims"] == "judge"
assert config(out["graph"], "write")["judge"] == {"max_cost_usd": 0.05}
def test_a_missing_budget_is_a_choice_between_the_default_and_unlimited():
graph = traceable(report={**STRICT, "claims": "judge", "judge": {"model": "judge-model"}})
fid = f"{BUDGET}:write"
fix = public(plan(graph)[fid])
assert fix["kind"] == "choice" and fix["node_id"] == "write" and fix["field"] == "judge.max_cost_usd"
assert [o["value"] for o in fix["options"]] == [JUDGE_DEFAULTS["report_max_cost_usd"], None]
assert "default" not in fix, "花多少钱要人定:不替人选,也不给建议"
unlimited = fix["options"][1]
assert "不限" in unlimited["label"] and "不设上限" in unlimited["hint"] and "约束" in unlimited["hint"]
assert all(not k.startswith("_") for k in fix), "内部键不交给前端"
[refused] = apply_fixes(graph, [fid], level="governed")["rejected"]
assert refused["reason"] == NEED_CHOICE.format(label=fix["label"])
out = apply_fixes(graph, [fid], {fid: 0.05}, level="governed")
assert out["applied"] == [fid] and out["ok"] is True
assert config(out["graph"], "write")["judge"] == {"model": "judge-model", "max_cost_usd": 0.05}
[change] = out["changes"]
assert change["field"] == "judge" and change["before"] == {"model": "judge-model"}
assert out["ops"] == [{"op": "update_node", "id": "write",
"config": {"judge": {"model": "judge-model", "max_cost_usd": 0.05}}}]
out = apply_fixes(graph, [fid], {fid: None}, level="governed")
judge = config(out["graph"], "write")["judge"]
assert out["ok"] is True and "max_cost_usd" in judge and judge["max_cost_usd"] is None, "不限要显式写成 null"
def test_the_budget_fix_keeps_what_the_graph_defaults_said():
"""judge 写在图级 defaults 里:修复写到节点上时带上 defaults 里的整份 judge,节点写了就不再跟随 defaults,
不能因为补一个预算把 defaults 里选的裁判模型丢了。"""
graph = traceable(report={**STRICT, "claims": "judge"},
defaults={"judge": {"model": "judge-model", "rewrite_once": True}})
fid = f"{BUDGET}:write"
out = apply_fixes(graph, [fid], {fid: None}, level="governed")
assert out["ok"] is True
assert config(out["graph"], "write")["judge"] == {"model": "judge-model", "rewrite_once": True,
"max_cost_usd": None}
assert out["graph"]["defaults"] == {"judge": {"model": "judge-model", "rewrite_once": True}}
def test_a_choice_outside_the_options_is_refused():
graph = traceable(report={**STRICT, "claims": "judge"})
fid = f"{BUDGET}:write"
out = apply_fixes(graph, [fid], {fid: 99}, level="governed")
assert out["applied"] == [] and "不在候选里" in out["rejected"][0]["reason"]
def test_published_level_budget_warnings_get_no_fix():
assert f"{BUDGET}:write" not in plan(traceable(report={**STRICT, "claims": "judge"}), "published")
def test_invalid_claims_fixes_no_longer_mention_a_later_version():
graph = traceable(report={**STRICT, "claims": "sometimes"})
governed = plan(graph)["report.claims_invalid:write"]
published = plan(graph, "published")["report.claims_invalid:write"]
assert "后续版本" not in governed["label"] and "default" not in published
assert [o["value"] for o in published["options"]] == ["off", "require_citation"]
# --------------------------------------------------------------------------
# 降低要求的判断
# --------------------------------------------------------------------------
def _changed(report_before: dict, report_after: dict | None, **graph_kw) -> list[str]:
old = traceable(report=report_before, **graph_kw)
new = copy.deepcopy(old)
if report_after is None:
config(new, "write").pop("claims")
else:
config(new, "write").clear()
config(new, "write").update({"instructions": "写周报", **report_after})
return forbidden_changes(old, new)
def test_moving_up_to_judge_is_not_loosening():
assert _changed(STRICT, JUDGED) == []
assert _changed({**STRICT, "claims": "off"}, JUDGED) == []
assert _changed({"numbers": "strict", "on_violation": "fail"}, JUDGED) == []
@pytest.mark.parametrize("after", ["off", "require_citation", None])
def test_moving_down_from_judge_is_loosening(after):
report_after = None if after is None else {**JUDGED, "claims": after}
[reason] = _changed(JUDGED, report_after)
assert "「报告撰写」" in reason and "claims" in reason and "judge" in reason
def test_dropping_a_judge_set_in_the_graph_defaults_is_loosening():
[reason] = _changed({"numbers": "strict", "on_violation": "fail"}, {**STRICT, "claims": "off"},
defaults={"claims": "judge", "judge": {"max_cost_usd": 0.05}})
assert "claims" in reason and "judge" in reason
def test_changing_the_budget_is_not_loosening():
assert _changed(JUDGED, {**JUDGED, "judge": {"max_cost_usd": None}}) == []
# claims 一直是 judge 时,judge.on_unsupported 从 withhold 往回改也是放宽:不支持的结论句从「不予出具」变成
# 「降档出具」。Copilot 改图的 update_node 会把 judge 整个盖掉,补预算时写 judge={max_cost_usd: 0.05}
# 就悄悄丢了 withhold——这一类要和契约里的 on_unsupported 一样拒掉
WITHHOLD = {**JUDGED, "judge": {"max_cost_usd": 0.05, "on_unsupported": "withhold"}}
@pytest.mark.parametrize("judge, dropped", [
({"max_cost_usd": 0.05, "on_unsupported": "degrade"}, False),
({"max_cost_usd": 0.05}, True),
({"max_cost_usd": 0.05, "on_unsupported": "sometimes"}, False),
])
def test_relaxing_on_unsupported_on_the_node_is_loosening(judge, dropped):
[reason] = _changed(WITHHOLD, {**WITHHOLD, "judge": judge})
assert "「报告撰写」" in reason and "judge.on_unsupported" in reason
assert "withhold" in reason and "degrade" in reason
assert ("去掉了写明的值" in reason) is dropped, reason
def test_replacing_the_whole_judge_block_to_add_a_budget_is_loosening():
"""补预算把 judge 整个换成 {max_cost_usd}:连同裁判模型、withhold 一起丢了。"""
before = {**STRICT, "claims": "judge", "judge": {"model": "judge-model", "on_unsupported": "withhold"}}
[reason] = _changed(before, {**before, "judge": {"max_cost_usd": 0.05}})
assert "judge.on_unsupported" in reason and "withhold" in reason
assert _changed(before, {**before, "judge": {**before["judge"], "max_cost_usd": 0.05}}) == [], \
"带着原来的键补预算不算放宽"
def test_a_withhold_from_the_graph_defaults_lost_by_a_node_judge_is_loosening():
"""judge 按运行时的规则取:节点写了自己的 judge 就整个用节点的,图级 defaults 里的 withhold 不再生效。"""
defaults = {"judge": {"on_unsupported": "withhold", "max_cost_usd": 0.05}}
report = {**STRICT, "claims": "judge"}
[reason] = _changed(report, {**report, "judge": {"max_cost_usd": 0.05}}, defaults=defaults)
assert "judge.on_unsupported" in reason and "从 withhold(跟随全图默认)" in reason, reason
assert "不再跟随全图默认" in reason, "说清楚为什么全图默认里的 withhold 不生效了"
assert _changed(report, {**report, "judge": {"max_cost_usd": 0.05, "on_unsupported": "withhold"}},
defaults=defaults) == []
def test_relaxing_on_unsupported_in_the_graph_defaults_is_loosening():
old = traceable(report={**STRICT, "claims": "judge"},
defaults={"judge": {"max_cost_usd": 0.05, "on_unsupported": "withhold"}})
new = copy.deepcopy(old)
new["defaults"]["judge"]["on_unsupported"] = "degrade"
[reason] = forbidden_changes(old, new)
assert "「报告撰写」" in reason and "judge.on_unsupported" in reason
@pytest.mark.parametrize("before, after", [
(JUDGED, {**JUDGED, "judge": {"max_cost_usd": 0.05, "on_unsupported": "withhold"}}),
(WITHHOLD, {**WITHHOLD, "judge": {**WITHHOLD["judge"], "max_cost_usd": None, "rewrite_once": True}}),
(WITHHOLD, WITHHOLD),
# 升到 judge 本身是收紧:不管带不带 withhold
({**STRICT, "judge": {"on_unsupported": "withhold"}}, JUDGED),
])
def test_tightening_or_keeping_on_unsupported_is_fine(before, after):
assert _changed(before, after) == []
def test_dropping_withhold_along_with_judge_is_one_reason():
"""从 judge 退回 require_citation:claims 那一条已经说了,不再为 on_unsupported 另记一条。"""
[reason] = _changed(WITHHOLD, {**STRICT, "judge": {"max_cost_usd": 0.05}})
assert "claims" in reason and "on_unsupported" not in reason
@pytest.mark.parametrize("before", ["require_citation", "judge"])
def test_moving_claims_to_an_unknown_value_is_loosening(before):
"""require_citation、judge 改成认不出的写法:出口按认不出的走,要求没了——照样算放宽,别只落个
「没让图变好」。原来就认不出的不比较。"""
report = {**STRICT, "claims": before, **({"judge": {"max_cost_usd": 0.05}} if before == "judge" else {})}
[reason] = _changed(report, {**report, "claims": "sometimes"})
assert "「报告撰写」" in reason and before in reason and "sometimes" in reason and "不认识" in reason
assert _changed({**STRICT, "claims": "sometimes"}, {**STRICT, "claims": "off"}) == []
assert _changed({**STRICT, "claims": "off"}, {**STRICT, "claims": "sometimes"}) == []
def _contract_changed(before, after) -> list[str]:
old = traceable(claims=before)
new = copy.deepcopy(old)
contract = config(new, "done")["contract"]
if after is None:
contract.pop("claims", None)
else:
contract["claims"] = after
return forbidden_changes(old, new)
@pytest.mark.parametrize("before, after", [
("judge", "off"),
("judge", None),
("judge", "require_citation"),
({"policy": "judge", "on_unsupported": "withhold"}, "judge"),
({"policy": "judge", "on_unsupported": "withhold"}, {"policy": "judge", "on_unsupported": "degrade"}),
({"policy": "require_citation", "on_uncited": "withhold"}, "judge"),
])
def test_loosening_a_judge_contract_is_refused(before, after):
[reason] = _contract_changed(before, after)
assert "「出具」" in reason and "claims" in reason
@pytest.mark.parametrize("before, after", [
("require_citation", "judge"),
("off", {"policy": "judge", "on_unsupported": "withhold"}),
("judge", {"policy": "judge", "on_unsupported": "withhold"}),
({"policy": "require_citation", "on_uncited": "withhold"}, {"policy": "judge", "on_uncited": "withhold"}),
])
def test_tightening_a_contract_towards_judge_is_fine(before, after):
assert _contract_changed(before, after) == []
+8 -9
View File
@@ -20,7 +20,7 @@ from app.db.base import SessionLocal
from app.db.models import Run, Workflow, WorkflowVersion
from app.engine.governance import lint_for_publish, publish_issues
from app.engine.runner import run_manager
from app.engine.schema import CLAIMS_LATER, GraphSpec, validate_graph
from app.engine.schema import GraphSpec, validate_graph
from app.main import app
from app.seed import TEMPLATES
@@ -281,10 +281,10 @@ def test_graph_defaults_count_like_they_do_at_run_time():
def test_an_unsupported_claims_value_is_left_to_validate():
graph = traceable(report={**STRICT, "claims": "judge"})
graph = traceable(report={**STRICT, "claims": "sometimes"})
assert by_code(graph, "governed.report_policy") == []
[bad] = [i for i in validate_graph(GraphSpec.model_validate(graph)).issues if i.code == "report.claims_invalid"]
assert bad.level == "error" and CLAIMS_LATER in bad.message
assert bad.level == "error" and "off / require_citation" in bad.message
# --------------------------------------------------------------------------
@@ -442,7 +442,7 @@ async def test_an_already_governed_version_keeps_running_formally(client, engine
# --------------------------------------------------------------------------
# validate:报告撰写节点的 claims 只接受 off / require_citation
# validate:报告撰写节点的 claims 只接受 off / require_citation / judge(四期放开 judge)
# --------------------------------------------------------------------------
@@ -458,10 +458,9 @@ def test_supported_claims_values_pass(value):
assert claims_issues(value) == []
def test_judge_is_a_later_version():
[issue] = claims_issues("judge")
assert issue.level == "error" and issue.node_id == "write" and issue.field == "claims"
assert CLAIMS_LATER in issue.message and "off" in issue.message and "require_citation" in issue.message
def test_judge_is_supported_now():
assert claims_issues("judge") == []
assert claims_issues("judge", on_defaults=True) == []
@pytest.mark.parametrize("value", ["strict", "Require_Citation", 1, ["off"], {"policy": "judge"}])
@@ -471,5 +470,5 @@ def test_other_claims_values_are_errors(value):
def test_claims_from_graph_defaults_are_checked_too():
[issue] = claims_issues("judge", on_defaults=True)
[issue] = claims_issues("sometimes", on_defaults=True)
assert issue.node_id == "write"
+262
View File
@@ -0,0 +1,262 @@
"""结论句裁判的设置与画布校验。
- 设置里单开一组 judge:证据裁判模型、每份报告的句数 / 金额 / 时长上限、探索运行每次点击的金额上限、
全局每日金额上限。每一项都能存成 null,表示不限——存了 null 就是不限,不会被默认值悄悄盖回去
- limits 组是环境变量定的、只读,裁判设置不放进去;每日计数存在设置表里,但不从设置接口进出
- 裁判模型的优先级:节点上的 judge.model > 设置里的证据裁判模型 > Copilot 模型 > 默认 provider
- 画布校验放开 claims: judge,并校验 judge 子配置的形状:上限可以写 null(不限),rewrite_once 是布尔值,
on_unsupported 取 degrade / withhold
"""
from __future__ import annotations
import pytest
from httpx import ASGITransport, AsyncClient
from sqlalchemy import delete
from app.db.base import SessionLocal
from app.db.models import Setting
from app.engine.judge import (
JUDGE_DEFAULTS,
SETTING_GROUP,
SPEND_KEY,
Budget,
click_budget,
judge_model_spec,
judge_settings,
report_budget,
)
from app.engine.schema import CLAIMS_VALUES, JUDGE_KEYS, GraphSpec, claims_problem, validate_graph
from app.main import app
@pytest.fixture(autouse=True)
async def clean():
async with SessionLocal() as session:
await session.execute(delete(Setting).where(Setting.key.in_([SPEND_KEY, SETTING_GROUP, "copilot"])))
await session.commit()
yield
def client() -> AsyncClient:
return AsyncClient(transport=ASGITransport(app=app), base_url="http://test")
# --------------------------------------------------------------------------
# 设置
# --------------------------------------------------------------------------
def test_the_defaults_are_the_agreed_ones():
assert SETTING_GROUP == "judge" and SETTING_GROUP != "limits"
assert JUDGE_DEFAULTS == {
"provider": None, "model": None,
"report_max_claims": 40, "report_max_cost_usd": 0.05, "report_timeout_s": 30,
"click_max_cost_usd": 0.01, "daily_max_usd": 2.0,
}
async def test_settings_expose_the_judge_group_with_defaults():
async with client() as c:
got = (await c.get("/api/settings")).json()
assert got["judge"] == JUDGE_DEFAULTS
assert SPEND_KEY not in got
async def test_null_is_stored_as_unlimited_and_stays_unlimited():
async with client() as c:
r = await c.put("/api/settings", json={"values": {"judge": {
**JUDGE_DEFAULTS, "model": "judge-model", "report_max_cost_usd": None, "daily_max_usd": None,
"report_timeout_s": None, "report_max_claims": None, "click_max_cost_usd": None}}})
assert r.status_code == 200, r.text
got = (await c.get("/api/settings")).json()["judge"]
assert got["model"] == "judge-model"
assert all(got[k] is None for k in ("report_max_cost_usd", "daily_max_usd", "report_timeout_s",
"report_max_claims", "click_max_cost_usd"))
async with SessionLocal() as session:
settings = await judge_settings(session)
assert report_budget(settings, None) == Budget(max_claims=None, max_cost_usd=None, timeout_s=None,
daily_max_usd=None)
assert click_budget(settings).max_cost_usd is None
async def test_a_partial_group_keeps_the_other_defaults():
async with client() as c:
await c.put("/api/settings", json={"values": {"judge": {"daily_max_usd": 5}}})
got = (await c.get("/api/settings")).json()["judge"]
assert got == {**JUDGE_DEFAULTS, "daily_max_usd": 5}
@pytest.mark.parametrize("key,value", [
("daily_max_usd", -1), ("report_max_cost_usd", 0), ("report_max_cost_usd", "0.05"),
("report_max_claims", 2.5), ("report_max_claims", True), ("report_timeout_s", 0),
("click_max_cost_usd", float("nan")), ("model", 3), ("unknown_key", 1),
])
async def test_bad_values_are_refused_with_a_reason(key, value):
async with client() as c:
r = await c.put("/api/settings", json={"values": {"judge": {key: value}}}) if value == value else \
await c.put("/api/settings", content='{"values": {"judge": {"%s": NaN}}}' % key,
headers={"content-type": "application/json"})
assert r.status_code == 422, r.text
assert key in r.json()["detail"]
assert (await c.get("/api/settings")).json()["judge"] == JUDGE_DEFAULTS, "整组都没写进去"
async def test_the_daily_counter_does_not_travel_through_the_settings_api():
async with SessionLocal() as session:
session.add(Setting(key=SPEND_KEY, value={"date": "2026-09-29", "usd": 1.5, "calls": 3}))
await session.commit()
async with client() as c:
assert SPEND_KEY not in (await c.get("/api/settings")).json()
await c.put("/api/settings", json={"values": {SPEND_KEY: {"date": "2026-09-29", "usd": 0, "calls": 0}}})
async with SessionLocal() as session:
row = await session.get(Setting, SPEND_KEY)
assert row.value["usd"] == 1.5, "设置页整组回写不能把今天的花费清零"
async def test_todays_spend_is_readable_next_to_the_daily_cap(monkeypatch):
from app.engine import judge as judge_mod
monkeypatch.setattr(judge_mod, "_today", lambda: "2026-09-29")
async with SessionLocal() as session:
session.add(Setting(key=SPEND_KEY, value={"date": "2026-09-28", "usd": 1.9, "calls": 7}))
await session.commit()
async with client() as c:
got = (await c.get("/api/settings/judge/spend")).json()
assert got == {"date": "2026-09-29", "usd": 0.0, "calls": 0, "unpriced_calls": 0, "daily_max_usd": 2.0}, \
"昨天的花费不算今天的"
async with SessionLocal() as session:
row = await session.get(Setting, SPEND_KEY)
row.value = {"date": "2026-09-29", "usd": 0.42, "calls": 3, "unpriced_calls": 1}
await session.commit()
await c.put("/api/settings", json={"values": {"judge": {"daily_max_usd": None}}})
got = (await c.get("/api/settings/judge/spend")).json()
assert got == {"date": "2026-09-29", "usd": 0.42, "calls": 3, "unpriced_calls": 1, "daily_max_usd": None}
async def test_bad_rows_already_in_the_database_fall_back_to_defaults():
"""库里存坏了的值(旧版本、手改)读出来按默认,不让裁判拿着一个负数上限跑。"""
async with SessionLocal() as session:
session.add(Setting(key=SETTING_GROUP, value={"daily_max_usd": -3, "report_max_claims": "many",
"report_timeout_s": None}))
await session.commit()
settings = await judge_settings(session)
assert settings["daily_max_usd"] == JUDGE_DEFAULTS["daily_max_usd"]
assert settings["report_max_claims"] == JUDGE_DEFAULTS["report_max_claims"]
assert settings["report_timeout_s"] is None
def test_the_node_overrides_key_by_key():
settings = {**JUDGE_DEFAULTS, "report_max_claims": 10}
assert report_budget(settings, None) == Budget(max_claims=10, max_cost_usd=0.05, timeout_s=30, daily_max_usd=2.0)
assert report_budget(settings, {"max_cost_usd": None, "timeout_s": 5}) == \
Budget(max_claims=10, max_cost_usd=None, timeout_s=5, daily_max_usd=2.0)
# 探索运行每次点击:金额按点击的上限,句数、时长按每份报告的
assert click_budget(settings) == Budget(max_claims=10, max_cost_usd=0.01, timeout_s=30, daily_max_usd=2.0)
# --------------------------------------------------------------------------
# 裁判模型的优先级
# --------------------------------------------------------------------------
async def test_the_judge_model_priority():
async with SessionLocal() as session:
spec = await judge_model_spec(session, None)
assert (spec.provider, spec.model) == (None, None), "什么都没配:默认 provider"
assert spec.thinking == "off" and spec.max_tokens == 4096
session.add(Setting(key="copilot", value={"provider": "", "model": "copilot-model"}))
await session.commit()
assert (await judge_model_spec(session, None)).model == "copilot-model"
session.add(Setting(key=SETTING_GROUP, value={"provider": "house", "model": "judge-model"}))
await session.commit()
spec = await judge_model_spec(session, None)
assert (spec.provider, spec.model) == ("house", "judge-model")
spec = await judge_model_spec(session, {"model": "node-model"})
assert (spec.provider, spec.model) == (None, "node-model"), "节点上的写法整组生效,不和设置里的拼"
spec = await judge_model_spec(session, {"model": None, "provider": None, "max_claims": 3})
assert spec.model == "judge-model"
# --------------------------------------------------------------------------
# 画布校验
# --------------------------------------------------------------------------
def node(nid, ntype, **config):
return {"id": nid, "type": ntype, "position": {"x": 0, "y": 0}, "data": {"label": nid, "config": config}}
def graph(report: dict, defaults: dict | None = None) -> dict:
nodes = [node("start", "input"),
node("caliber", "metrics", caliber="周报口径", metrics=[{"id": "gmv", "expression": "1"}]),
node("write", "report", instructions="写周报", **report),
node("out", "output", fields=[{"name": "周报", "value": "{{ nodes.write.text }}"}])]
ids = [n["id"] for n in nodes]
g = {"nodes": nodes, "edges": [{"source": a, "target": b} for a, b in zip(ids, ids[1:])]}
if defaults:
g["defaults"] = defaults
return g
def issues(g, code=None):
return [i for i in validate_graph(GraphSpec.model_validate(g)).issues
if i.code in ((code,) if code else ("report.claims_invalid", "report.judge_invalid"))]
def test_judge_is_a_supported_policy_now():
assert CLAIMS_VALUES == ("off", "require_citation", "judge")
assert claims_problem("judge") is None
assert issues(graph({"claims": "judge"})) == []
assert issues(graph({}, defaults={"claims": "judge"})) == []
def test_every_limit_can_be_unlimited():
judge = {"provider": None, "model": None, "max_claims": None, "max_cost_usd": None, "timeout_s": None,
"rewrite_once": True, "on_unsupported": "withhold"}
assert set(judge) == set(JUDGE_KEYS)
assert issues(graph({"claims": "judge", "judge": judge})) == []
assert issues(graph({"claims": "judge", "judge": {"max_claims": 40, "max_cost_usd": 0.05, "timeout_s": 30,
"model": "judge-model", "on_unsupported": "degrade"}})) == []
@pytest.mark.parametrize("judge,field,words", [
("fast", "judge", "对象"),
({"max_cost": 1}, "judge.max_cost", "max_cost_usd"),
({"max_claims": 0}, "judge.max_claims", "正整数"),
({"max_claims": 2.5}, "judge.max_claims", "正整数"),
({"max_claims": True}, "judge.max_claims", "正整数"),
({"max_cost_usd": -1}, "judge.max_cost_usd", "大于 0"),
({"max_cost_usd": "0.05"}, "judge.max_cost_usd", "大于 0"),
({"timeout_s": 0}, "judge.timeout_s", "大于 0"),
({"rewrite_once": "yes"}, "judge.rewrite_once", "true 或 false"),
({"on_unsupported": "ignore"}, "judge.on_unsupported", "degrade / withhold"),
({"model": 3}, "judge.model", "模型 id"),
])
def test_a_malformed_judge_block_is_an_error(judge, field, words):
[issue] = issues(graph({"claims": "judge", "judge": judge}), "report.judge_invalid")
assert issue.level == "error" and issue.node_id == "write" and issue.field == field
assert words in issue.message, issue.message
if field in ("judge.max_claims", "judge.max_cost_usd", "judge.timeout_s", "judge.model"):
assert "null" in issue.message, "上限、模型都说清写 null 是什么意思"
def test_graph_defaults_are_checked_like_at_run_time():
[issue] = issues(graph({"claims": "judge"}, defaults={"judge": {"max_claims": -1}}), "report.judge_invalid")
assert issue.field == "judge.max_claims"
def test_the_judge_block_only_matters_when_claims_is_judge():
assert issues(graph({"claims": "require_citation", "judge": {"max_claims": -1}})) == []
def test_judging_with_the_writing_model_is_a_warning():
[warn] = [i for i in validate_graph(GraphSpec.model_validate(graph(
{"claims": "judge", "model": "writer-x", "judge": {"model": "writer-x"}}))).issues
if i.code == "report.judge_same_model"]
assert warn.level == "warning" and warn.field == "judge.model" and "自己审自己" in warn.message
assert not [i for i in validate_graph(GraphSpec.model_validate(graph(
{"claims": "judge", "model": "writer-x", "judge": {"model": "judge-y"}}))).issues
if i.code == "report.judge_same_model"]
+10 -10
View File
@@ -1,10 +1,10 @@
"""报告节点的结论句策略 claims:off(默认,和以前一样)/ require_citation(没挂依据的结论句计入缺口)。
结论句要靠裁判模型判断支不支持,那是后续版本的事;这一版只做确定性的一半:
这里测确定性的一半(模型裁判 judge 见 test_report_judge.py):
- require_citation 时写作提示明确要求每句结论挂 [[see:…]];没挂的照常产出、计数,
出口按出具档位降档(io.py),报告节点自己不为它失败、不为它重写
- report.checked 事件和节点产出都带上 claims,出口据此判档
- 写 judge 时节点直接报错「结论句裁判在后续版本支持」,不能悄悄当成 off
- 写了不认识的值节点直接报错,不能悄悄当成 off
- 升级前发起的运行一个字都不变
"""
from __future__ import annotations
@@ -21,7 +21,7 @@ from app.db.models import Run, RunEvent
from app.engine import runner as runner_mod
from app.engine.evidence import CLAIMS_RULE, build_catalog, compose_doc, uncited_claims
from app.engine.runner import run_manager
from app.engine.schema import CLAIMS_LATER, REPORT_CLAIMS, claims_problem
from app.engine.schema import CLAIMS_VALUES, claims_problem
@pytest.fixture(autouse=True)
@@ -131,14 +131,14 @@ async def test_require_citation_asks_for_support_but_does_not_block(monkeypatch)
assert flagged["text"] == "增长主要来自新客。" and doc["markdown"][slice(*flagged["span"])] == flagged["text"]
async def test_judge_is_refused_until_it_exists(monkeypatch):
async def test_a_bad_value_is_refused_before_any_model_call(monkeypatch):
seen = script(monkeypatch)
with pytest.raises(ValueError) as info: # 画布校验先拦(E3-gov 的 validate)
await run_manager.start(graph=weekly(claims="judge"), input_payload={})
assert CLAIMS_LATER in str(info.value)
with pytest.raises(ValueError) as info: # 画布校验先拦
await run_manager.start(graph=weekly(claims="sometimes"), input_payload={})
assert claims_problem("sometimes") in str(info.value)
lenient(monkeypatch)
row = await finish(weekly(claims="judge"))
assert row.status == "failed" and claims_problem("judge") in (row.error or ""), row.error
row = await finish(weekly(claims="sometimes"))
assert row.status == "failed" and claims_problem("sometimes") in (row.error or ""), row.error
assert seen == [], "不支持的策略不该先花一次模型调用"
@@ -173,7 +173,7 @@ def test_the_supported_policies_match_the_validator():
from app.engine.nodes.report import CLAIMS, report_claims
from app.engine.schema import GraphSpec
assert CLAIMS == REPORT_CLAIMS == ("off", "require_citation")
assert CLAIMS == CLAIMS_VALUES == ("off", "require_citation", "judge")
spec = GraphSpec.model_validate(weekly(claims="require_citation"))
assert report_claims(spec, spec.node_map()["write"]) == "require_citation"
spec = GraphSpec.model_validate(weekly())
+495
View File
@@ -0,0 +1,495 @@
"""报告撰写节点的 claims: judge:结论句由另一个模型按证据逐句判断(四期)。
- 正式运行在节点里裁判:写完、核对完之后判,判定写进报告文档的 unit.verdict,随文档一起封存
- 探索运行不在节点里花这笔钱:候选句标「未裁判 · 按需」,点开哪句再判哪句(接口在 B 段)
- 判定只标注、不改写正文;裁判没跑成、触顶了,照样产出报告,没判的记未裁判并留下缺口
(出口据此不能判完整出具,接线在 B 段)
- rewrite_once(默认关):把证据不支持的句子和理由交回写作者只改这几句,再判一次,最多一轮;
改出了新的违规就不采用
- 裁判调用包在 @task 里、追加在节点已有的调用之后:节点「接着跑」时不重复调用、不重复计费;
升级前发起的运行一样都不加,停在审批上的运行恢复后照常
"""
from __future__ import annotations
import asyncio
import json
import re
import pytest
from langchain_core.messages import AIMessage
from sqlalchemy import delete, select
from app.core import artifact_store
from app.db.base import SessionLocal
from app.db.models import Run, RunEvent, Setting
from app.engine import judge as judge_mod
from app.engine import replay
from app.engine import runner as runner_mod
from app.engine.evidence import CLAIMS_RULE, iter_units
from app.engine.judge import JUDGE_ROLE, JUDGE_RULE, SPEND_KEY, SUMMARY_KEYS, daily_spend
from app.engine.nodes import report as report_mod
from app.engine.runner import run_manager, verify_manifest
from app.providers.mock_model import MockChatModel
PRICED = "claude-sonnet-5"
METRICS = [{"id": "gmv", "name": "销售额", "unit": "元", "decimals": 1, "expression": "vars.kpi.gmv"}]
#: 一句挂了依据的结论、一句没挂依据的归因、一句连接性的话
TEXT = "本周销售额 [[m:gmv]]。[[see:m:gmv]]增长主要来自新客首单。下面看细节。"
CITED, CAUSAL, LINK = "本周销售额 45,678.5元。", "增长主要来自新客首单。", "下面看细节。"
@pytest.fixture(autouse=True)
async def engine_up():
await run_manager.setup()
async with SessionLocal() as session:
await session.execute(delete(Setting).where(Setting.key.in_([SPEND_KEY, "judge", "copilot"])))
await session.commit()
yield
await run_manager.shutdown()
def node(nid, ntype, **config):
return {"id": nid, "type": ntype, "position": {"x": 0, "y": 0}, "data": {"label": nid, "config": config}}
def weekly(*, gate=False, **report):
nodes = [
node("start", "input", fields=[{"name": "week", "default": "2026-W37"}]),
node("fetch", "transform", mode="expression", expression="{'gmv': 45678.5}", assign_to="kpi"),
node("caliber", "metrics", caliber="周报口径", caliber_version="v2", metrics=METRICS),
*([node("gate", "human", mode="approve", title="确认口径")] if gate else []),
node("write", "report", instructions="写周报", **report),
node("out", "output", fields=[{"name": "周报", "value": "{{ nodes.write.text }}"}]),
]
ids = [n["id"] for n in nodes]
return {"nodes": nodes, "edges": [{"source": a, "target": b} for a, b in zip(ids, ids[1:])]}
class Models:
"""写作者和裁判两份剧本。写作者依次回 writer 里的几段(用完了一直回最后一段);裁判按句子里的字给判定
(verdicts:{句中片段: 判定},没对上的都是 supported)。"""
def __init__(self, monkeypatch, writer=(TEXT,), verdicts=None, *, judge_model=PRICED, fail_judge=None):
self.writes: list[list] = []
self.judges: list[str] = []
verdicts = verdicts or {}
def decide(model, messages):
if JUDGE_ROLE in str(messages[0].content):
request = str(messages[-1].content)
self.judges.append(request)
if fail_judge:
raise RuntimeError(fail_judge)
items = []
for uid, line in re.findall(r"^\[(u\d+)\]([^\n]*)$", request, re.M):
status = next((s for frag, s in verdicts.items() if frag in line), "supported")
items.append({"unit": uid, "verdict": status, "rationale": f"判的是:{line[:20]}", "used": []})
return AIMessage(content=json.dumps({"items": items}, ensure_ascii=False))
self.writes.append(list(messages))
return AIMessage(content=writer[min(len(self.writes), len(writer)) - 1])
monkeypatch.setattr(MockChatModel, "_decide", decide)
async def judge_model_of(session, spec):
return MockChatModel(model_name=judge_model), judge_model
monkeypatch.setattr(judge_mod, "get_chat_model", judge_model_of)
def spy_tasks(monkeypatch) -> list[str]:
"""报告节点里每调用一次 @task 记一笔名字:证明加进来的 task 只在该出现的地方出现。"""
from langgraph.func import task as real
calls: list[str] = []
def spying(func=None, **kwargs):
def wrap(fn):
inner = real(fn, **kwargs)
def call(*args, **kw):
calls.append(fn.__name__)
return inner(*args, **kw)
return call
return wrap(func) if func is not None else wrap
monkeypatch.setattr(report_mod, "task", spying)
return calls
async def finish(graph, *, run_class="formal", statuses=("succeeded", "failed")) -> Run:
run = await run_manager.start(graph=graph, input_payload={}, run_class=run_class)
return await wait(run.id, statuses)
async def wait(run_id, statuses=("succeeded", "failed")) -> Run:
for _ in range(400):
await asyncio.sleep(0.05)
async with SessionLocal() as session:
row = await session.get(Run, run_id)
if row.status in statuses and (row.status == "interrupted" or row.manifest_seq is not None):
return row
raise AssertionError(f"run {run_id} 没跑完:{row.status}")
async def events(run_id: str, etype: str, node_id: str | None = "write") -> list[RunEvent]:
async with SessionLocal() as session:
q = select(RunEvent).where(RunEvent.run_id == run_id, RunEvent.type == etype)
if node_id:
q = q.where(RunEvent.node_id == node_id)
return list((await session.execute(q.order_by(RunEvent.seq))).scalars())
async def report_of(run_id: str) -> tuple[dict, dict, dict]:
"""(report.checked 的载荷, 报告文档, 节点产出)"""
[checked] = await events(run_id, "report.checked")
[finished] = await events(run_id, "node.finished")
return checked.data, artifact_store.load(checked.data["doc_artifact"]), artifact_store.load(finished.data["artifact"])
def verdicts(doc) -> dict[str, dict | None]:
md = doc["markdown"]
return {md[u["span"][0]:u["span"][1]].strip(): u.get("verdict") for _, u in iter_units(doc)}
def unit_ids(doc) -> dict[str, str]:
md = doc["markdown"]
return {md[u["span"][0]:u["span"][1]].strip(): u["id"] for _, u in iter_units(doc)}
async def judge_ends(run_id: str) -> list[dict]:
return [e.data for e in await events(run_id, "llm.end") if e.data.get("purpose") == "judge"]
# --------------------------------------------------------------------------
# 正式运行:节点内裁判,判定随文档封存
# --------------------------------------------------------------------------
async def test_a_formal_run_judges_inside_the_node_and_seals_the_verdicts(monkeypatch):
models = Models(monkeypatch, verdicts={"新客": "unsupported"})
row = await finish(weekly(claims="judge", judge={"on_unsupported": "withhold"}))
assert row.status == "succeeded", row.error
assert len(models.writes) == 1 and len(models.judges) == 1
checked, doc, output = await report_of(row.id)
got = verdicts(doc)
assert got[CITED]["status"] == "supported" and got[CITED]["judge"] == PRICED
assert got[CITED]["post_seal"] is False and got[CITED]["rationale"]
assert got[CAUSAL]["status"] == "unsupported"
assert got[LINK] is None, "连接性的话不送裁判"
assert CITED in models.judges[0] and CAUSAL in models.judges[0] and LINK not in models.judges[0]
summary = checked["judge"]
assert tuple(summary) == SUMMARY_KEYS, "B 段照着这份键读:不能悄悄少一个、多一个"
assert summary["mode"] == "inline" and summary["model"] == PRICED and summary["on_unsupported"] == "withhold"
assert summary["counts"] == {"supported": 1, "partial": 0, "unsupported": 1, "not_a_claim": 0, "unjudged": 0}
assert summary["complete"] is True and summary["gaps"] == []
assert checked["claims"] == output["claims"] == "judge" and output["judge"] == summary
assert doc["judge"]["mode"] == "inline" and doc["judge"]["counts"] == summary["counts"]
assert {k: doc["stats"][k] for k in ("supported", "partial", "unsupported", "not_a_claim", "unjudged")} \
== summary["counts"]
[end] = await judge_ends(row.id)
assert end["model"] == PRICED and end["cost_usd"] >= 0
verdict = await verify_manifest(row.id)
assert verdict["ok"] is True
[event] = await events(row.id, "report.checked")
assert event.seq <= row.manifest_seq, "判定在封存范围内"
async def test_the_writer_is_told_that_claims_will_be_judged(monkeypatch):
models = Models(monkeypatch)
await finish(weekly(claims="judge"))
prompt = "\n".join(str(m.content) for m in models.writes[0])
assert CLAIMS_RULE in prompt and JUDGE_RULE in prompt
async def test_explore_runs_leave_it_to_the_reader(monkeypatch):
"""探索运行默认按需裁判:节点里不调用,候选句标「未裁判 · 按需」,连接性的话什么都不标。"""
models = Models(monkeypatch)
row = await finish(weekly(claims="judge"), run_class="exploratory")
assert row.status == "succeeded", row.error
assert models.judges == []
checked, doc, output = await report_of(row.id)
got = verdicts(doc)
for text in (CITED, CAUSAL):
assert got[text]["status"] == "unjudged" and got[text]["reason"] == "on_demand"
assert got[text]["judge"] is None and got[text]["post_seal"] is False
assert got[LINK] is None
assert checked["judge"]["mode"] == "on_demand" and checked["judge"]["gaps"] == []
assert tuple(checked["judge"]) == SUMMARY_KEYS and checked["judge"]["limits"] is None
assert output["judge"] == checked["judge"] and await judge_ends(row.id) == []
async def test_a_failed_judge_leaves_a_gap_not_a_failed_report(monkeypatch):
Models(monkeypatch, fail_judge="网关超时")
row = await finish(weekly(claims="judge"))
assert row.status == "succeeded", row.error
checked, doc, output = await report_of(row.id)
assert {verdicts(doc)[t]["reason"] for t in (CITED, CAUSAL)} == {"error"}
summary = checked["judge"]
assert summary["complete"] is False and summary["counts"]["unjudged"] == 2
assert summary["gaps"] and "网关超时" in summary["gaps"][0]
assert output["judge"]["gaps"] == summary["gaps"]
[end] = await judge_ends(row.id)
assert "网关超时" in end["error"]
async def test_hitting_a_limit_is_reported_not_raised(monkeypatch):
models = Models(monkeypatch)
row = await finish(weekly(claims="judge", judge={"max_claims": 1}))
assert row.status == "succeeded", row.error
checked, doc, _ = await report_of(row.id)
assert verdicts(doc)[CITED]["status"] == "supported", "先判挂了依据的"
assert (verdicts(doc)[CAUSAL]["status"], verdicts(doc)[CAUSAL]["reason"]) == ("unjudged", "max_claims")
assert checked["judge"]["limits_hit"] == ["max_claims"] and "已到上限" in checked["judge"]["gaps"][0]
assert CAUSAL not in models.judges[0]
async def test_the_node_budget_overrides_the_settings_and_null_means_unlimited(monkeypatch):
"""节点 judge 里写了的键盖过设置;写 null 就是不限(不是「没写」)。"""
async with SessionLocal() as session:
session.add(Setting(key="judge", value={"report_max_claims": 1, "report_max_cost_usd": 1e-9}))
await session.commit()
Models(monkeypatch)
row = await finish(weekly(claims="judge", judge={"max_cost_usd": None}))
checked, doc, _ = await report_of(row.id)
assert verdicts(doc)[CITED]["status"] == "supported"
assert verdicts(doc)[CAUSAL]["reason"] == "max_claims", "max_claims 没写,取设置里的 1"
assert checked["judge"]["limits"]["max_cost_usd"] is None and checked["judge"]["limits"]["max_claims"] == 1
async def test_judging_yourself_is_called_out(monkeypatch):
Models(monkeypatch, judge_model="mock-fast")
row = await finish(weekly(claims="judge"))
assert row.status == "succeeded", row.error
[warn] = [e.data for e in await events(row.id, "log") if e.data.get("code") == "judge_same_model"]
assert "mock-fast" in warn["message"] and "自己审自己" in warn["message"]
# --------------------------------------------------------------------------
# rewrite_once
# --------------------------------------------------------------------------
REWRITTEN = "本周销售额 [[m:gmv]]。[[see:m:gmv]]新客首单的贡献要等新客维度的数据再看。下面看细节。"
async def test_rewrite_once_hands_back_only_the_unsupported_sentences(monkeypatch):
models = Models(monkeypatch, writer=(TEXT, REWRITTEN),
verdicts={"主要来自新客": "unsupported", "要等新客维度": "not_a_claim"})
row = await finish(weekly(claims="judge", judge={"rewrite_once": True}))
assert row.status == "succeeded", row.error
assert len(models.writes) == 2 and len(models.judges) == 2
request = str(models.writes[1][-1].content)
assert CAUSAL in request and "判的是:" in request and CITED not in request, "只交回不支持的那几句和理由"
# 第二次只判改过的句子:没改的那句直接用第一次的判定
assert "要等新客维度" in models.judges[1] and CITED not in models.judges[1]
checked, doc, output = await report_of(row.id)
assert "新客首单的贡献要等新客维度的数据再看。" in doc["markdown"] and CAUSAL not in doc["markdown"]
got = verdicts(doc)
assert got[CITED]["status"] == "supported" and got["新客首单的贡献要等新客维度的数据再看。"]["status"] == "not_a_claim"
rewrite = checked["judge"]["rewrite"]
assert rewrite["applied"] is True and rewrite["sentences"] == [CAUSAL] and checked["judge"]["reused"] == 1
# 封存的是改写稿:units 按句子认封存文档里的编号,交回去的那句已经改掉了,对不上就是 null——
# 不能拿原稿的编号去指改写稿里碰巧同号的另一句;改写稿里新出来的句子在 changed 里
ids = unit_ids(doc)
assert rewrite["units"] == [None]
assert rewrite["changed"] == [ids["新客首单的贡献要等新客维度的数据再看。"]]
assert output["text"] == doc["markdown"]
[log] = [e.data for e in await events(row.id, "log") if e.data.get("code") == "report_rewrite"]
assert "1 句" in log["message"]
assert len(await judge_ends(row.id)) == 2
async def test_rewrite_once_is_off_by_default(monkeypatch):
models = Models(monkeypatch, writer=(TEXT, REWRITTEN), verdicts={"主要来自新客": "unsupported"})
row = await finish(weekly(claims="judge"))
assert row.status == "succeeded", row.error
assert len(models.writes) == 1 and len(models.judges) == 1
checked, doc, _ = await report_of(row.id)
assert verdicts(doc)[CAUSAL]["status"] == "unsupported" and checked["judge"].get("rewrite") is None
async def test_a_rewrite_that_breaks_the_numbers_is_not_used(monkeypatch):
models = Models(monkeypatch, writer=(TEXT, "本周销售额 [[m:gmv]]。[[see:m:gmv]]新客贡献了 45678 元。下面看细节。"),
verdicts={"主要来自新客": "unsupported"})
row = await finish(weekly(claims="judge", judge={"rewrite_once": True}))
assert row.status == "succeeded", row.error
assert len(models.writes) == 2 and len(models.judges) == 1, "改坏了的那一稿不再花钱判"
checked, doc, output = await report_of(row.id)
assert CAUSAL in doc["markdown"] and "45678" not in doc["markdown"]
assert verdicts(doc)[CAUSAL]["status"] == "unsupported"
rewrite = checked["judge"]["rewrite"]
assert rewrite["applied"] is False and "45678" in rewrite["reason"]
# 封存的是原稿:units 就是原稿(也就是封存文档)里的编号
assert rewrite["units"] == [unit_ids(doc)[CAUSAL]] and rewrite["changed"] == []
def with_schema(monkeypatch):
"""报告的目录里加一张表:实体层有了核对的底,反引号里的名字才会被认出是编造的。"""
from app.engine.evidence import _entity_entries
real = report_mod.report_catalog
def catalog_with_schema(state, spec, node):
return {**real(state, spec, node), **_entity_entries(
[{"artifact": "d" * 64, "source": "shop", "tables": {"orders": ["order_id", "amount"]}}], [])}
monkeypatch.setattr(report_mod, "report_catalog", catalog_with_schema)
async def test_a_rewrite_that_makes_up_a_name_is_not_used(monkeypatch):
"""改写稿冒出一个编造的表名(只标注、不拦的那类违规):同样是原稿没有的新违规,不采用。"""
with_schema(monkeypatch)
models = Models(monkeypatch, writer=(TEXT, "本周销售额 [[m:gmv]]。[[see:m:gmv]]新客首单的贡献要看 `refund_detail` 表。"
"下面看细节。"),
verdicts={"主要来自新客": "unsupported"})
row = await finish(weekly(claims="judge", judge={"rewrite_once": True}))
assert row.status == "succeeded", row.error
assert len(models.writes) == 2 and len(models.judges) == 1
checked, doc, _ = await report_of(row.id)
assert CAUSAL in doc["markdown"] and "refund_detail" not in doc["markdown"]
rewrite = checked["judge"]["rewrite"]
assert rewrite["applied"] is False and "refund_detail" in rewrite["reason"]
[log] = [e.data for e in await events(row.id, "log") if e.data.get("code") == "report_rewrite_rejected"]
assert "refund_detail" in log["message"]
async def test_a_rewrite_that_trades_one_violation_for_another_is_not_used(monkeypatch):
"""去掉一处违规、又冒出另一处:总数没变,可冒出来的是原稿没有的,一样不采用。"""
draft = "本周销售额 [[m:gmv]]。[[see:m:gmv]]增长主要来自新客首单,共 1200 单。下面看细节。"
swapped = "本周销售额 [[m:gmv]]。[[see:m:gmv]]新客首单约占 3500 单。下面看细节。"
models = Models(monkeypatch, writer=(draft, draft, swapped), verdicts={"主要来自新客": "unsupported"})
row = await finish(weekly(claims="judge", on_violation="flag", max_repairs=1, judge={"rewrite_once": True}))
assert row.status == "succeeded", row.error
assert len(models.writes) == 3 and len(models.judges) == 1
checked, doc, _ = await report_of(row.id)
assert "1200" in doc["markdown"] and "3500" not in doc["markdown"]
rewrite = checked["judge"]["rewrite"]
assert rewrite["applied"] is False and "3500" in rewrite["reason"] and "1200" not in rewrite["reason"]
def test_new_violations_are_counted_one_by_one():
"""改写稿的违规逐个和原稿抵消:原样留着的不算新的,同一个裸数字多写一处、只标注的那类,都算新的。"""
number = {"code": "uncited_number", "text": "1200", "message": "数字「1200」没有出处"}
name = {"code": "unknown_entity", "text": "refund_detail", "message": "可能是编造的名字"}
see = {"code": "unresolved_ref", "ref": "Q9", "message": "依据 [[see:Q9]] 解析不了"}
assert report_mod._new_violations([number, name, see], [see, number, name]) == []
assert report_mod._new_violations([number], [number, number]) == [number]
assert report_mod._new_violations([number], [name]) == [name]
assert report_mod._new_violations([number, see], []) == []
async def test_a_second_round_that_hits_a_limit_names_the_configured_limit(monkeypatch):
"""第二轮用的是第一轮剩下的钱,可说给人看的上限是节点上配的那个数,不是剩下的零头。"""
Models(monkeypatch, writer=(TEXT, REWRITTEN), verdicts={"主要来自新客": "unsupported"})
# 每句估 1 分钱、回复不带用量(实际就按估算记):第一轮两句花掉 2 分,第二轮只剩半分,一句也装不下
monkeypatch.setattr(judge_mod.JudgeRequest, "cost", lambda self, model_id, units: 0.01 * len(units))
monkeypatch.setattr(judge_mod, "_usage_of", lambda message, model_id: {"input_tokens": 0, "output_tokens": 0})
row = await finish(weekly(claims="judge", judge={"rewrite_once": True, "max_cost_usd": 0.025}))
assert row.status == "succeeded", row.error
checked, doc, _ = await report_of(row.id)
v = verdicts(doc)["新客首单的贡献要等新客维度的数据再看。"]
assert (v["status"], v["reason"]) == ("unjudged", "max_cost_usd")
assert "$0.025" in v["rationale"] and "0.005" not in v["rationale"], v["rationale"]
summary = checked["judge"]
assert summary["rewrite"]["applied"] is True and summary["limits"]["max_cost_usd"] == 0.025
assert "$0.025" in summary["gaps"][0] and "0.005" not in summary["gaps"][0], summary["gaps"]
[log] = [e.data for e in await events(row.id, "log") if e.data.get("code") == "judge_limit"]
assert "$0.025" in log["message"] and "0.005" not in log["message"]
# --------------------------------------------------------------------------
# 重放安全
# --------------------------------------------------------------------------
async def test_continuing_after_a_failure_does_not_judge_or_charge_twice(monkeypatch):
models = Models(monkeypatch)
real = report_mod._store
broken = {"on": True}
async def flaky(doc, ctx):
if broken["on"]:
raise RuntimeError("落盘时断了")
return await real(doc, ctx)
monkeypatch.setattr(report_mod, "_store", flaky)
run = await run_manager.start(graph=weekly(claims="judge"), input_payload={}, run_class="formal")
row = await wait(run.id)
assert row.status == "failed" and "落盘时断了" in (row.error or ""), row.error
assert len(models.judges) == 1
before = await daily_spend()
await run_manager.wait_idle(run.id)
broken["on"] = False
await run_manager.continue_failed(run.id)
row = await wait(run.id)
assert row.status == "succeeded", row.error
assert len(models.judges) == 1, "接着跑时裁判又调了一次:裁判的结果没进 checkpoint"
assert await daily_spend() == before, "接着跑时每日计数又记了一次"
assert len(await judge_ends(row.id)) == 1
_, doc, _ = await report_of(row.id)
assert verdicts(doc)[CITED]["status"] == "supported"
async def test_a_changed_draft_is_judged_afresh_when_continuing(monkeypatch):
"""接着跑时写作者重写了一稿(写作调用不进 checkpoint),缓存的判定是上一稿的:不能拿来用。"""
models = Models(monkeypatch, writer=(TEXT, "本周销售额 [[m:gmv]]。[[see:m:gmv]]增长主要来自老客复购。下面看细节。"))
real = report_mod._store
broken = {"on": True}
async def flaky(doc, ctx):
if broken["on"]:
raise RuntimeError("落盘时断了")
return await real(doc, ctx)
monkeypatch.setattr(report_mod, "_store", flaky)
run = await run_manager.start(graph=weekly(claims="judge"), input_payload={}, run_class="formal")
assert (await wait(run.id)).status == "failed"
await run_manager.wait_idle(run.id)
broken["on"] = False
await run_manager.continue_failed(run.id)
row = await wait(run.id)
assert row.status == "succeeded", row.error
assert len(models.judges) == 2 and "老客复购" in models.judges[1]
_, doc, _ = await report_of(row.id)
assert verdicts(doc)["增长主要来自老客复购。"]["status"] == "supported"
async def test_runs_from_before_the_upgrade_never_judge(monkeypatch):
"""升级前发起的运行(没有护栏快照):claims 一律按 off,节点里一个 task 都不多调用,产出一个键都不多。"""
async def no_snapshot(session):
return None
monkeypatch.setattr(runner_mod, "_agent_limits", no_snapshot)
models = Models(monkeypatch)
calls = spy_tasks(monkeypatch)
row = await finish(weekly(claims="judge"))
assert row.status == "succeeded", row.error
assert models.judges == [] and calls == []
checked, doc, output = await report_of(row.id)
assert "judge" not in checked and "claims" not in checked and "judge" not in output and "judge" not in doc
assert all(v is None for v in verdicts(doc).values())
async def test_runs_paused_at_an_approval_resume_with_tasks_in_place(monkeypatch):
"""停在审批上的运行恢复:审批节点照旧只问一次;报告节点在恢复之后才第一次执行,裁判 task 只调一次、
排在节点所有已有调用之后。没开裁判的报告节点一个 task 都不调用(和升级前一样)。"""
assert replay.PROTOCOL == 2, "没有改动已有 task 的顺序,重放协议不升版"
for claims, expected in (("require_citation", []), ("judge", ["judge_step"])):
models = Models(monkeypatch)
calls = spy_tasks(monkeypatch)
paused = await finish(weekly(gate=True, claims=claims), statuses=("interrupted", "failed"))
assert paused.status == "interrupted", paused.error
await run_manager.resume(paused.id, {"approved": True})
row = await wait(paused.id)
assert row.status == "succeeded", row.error
assert calls == expected, (claims, calls)
assert len(models.judges) == len(expected)
assert len(await events(row.id, "human.requested", "gate")) == 1
checked, _, _ = await report_of(row.id)
assert ("judge" in checked) is (claims == "judge")