mirror of
https://github.com/ThearchyHelios/AgentLab.git
synced 2026-11-18 08:51:58 +00:00
可点击证据第四期:结论句由模型裁判;只引单元格的问数据图能发成受管
一、结论句裁判
- 报告节点新增 claims: judge:写完、核对完之后,把结论句和它引用的证据摘录交给裁判模型,
判为支持 / 部分支持 / 不支持 / 不是结论,附一句理由;判定只标注,不改写正文
- 送裁判的句子按固定规则预筛:带引用的、含方向词或因果词的送去判,短句、过渡句不送
- 证据摘录:指标写名称、值和原式;查询给 SQL 前 300 字和被引用的行,被引用的列一定给到;遮罩列一律不给;
规则里写明「数字已经由系统核对过,不要复算」
- 一份报告合并成一次调用,超出句数就分批;结构化输出失败时退回 JSON 解析,再失败记为未裁判
- 裁判模型的优先级:节点上写的 > 设置里的「证据裁判模型」> Copilot 的模型;思考关闭;
和写作模型相同时给警告(等于自己审自己)
- rewrite_once(默认关):把不支持的句子和理由交回写作者只改这几句,再裁一次;
改写稿冒出原稿没有的问题就不采用,保留原稿和原来的判定
二、成本上限
- 每份报告的句数、金额、时长,探索运行每次点击的金额,全局每日金额,每一项都可以自定义,也都可以设成不限;
默认 40 句、$0.05、30 秒,每次点击 $0.01,每天 $2
- 调用前先估算,超出上限按优先级截断:先判带引用的,再判含方向词、因果词的,其余记为未裁判并写明「已到上限」;
不报错,也不让报告失败
- 每日花费按本地日期计数,持久化在设置表里;重跑同一次运行不重复计费;价格目录里没有的模型只计次数,并写明估不出金额
三、正式运行与探索运行
- 正式运行在报告节点内裁判,判定写进报告文档、随文档封存;不支持的按 on_unsupported(degrade / withhold)降档或不予出具,
裁判没跑成记缺口,不能判完整出具
- 探索运行按需裁判:POST /api/runs/{id}/evidence/judge,点开哪句判哪句;判定作为 evidence.judged 事件追加在封存之后,
标「封存后追加」,封存核对仍然一致;只读封存范围内的工件;同一句不重复判;运行在裁判期间被接着跑时不记录
- 正式运行调这个接口返回 409,写明「正式运行的裁判在节点内完成」
四、治理与自动修复
- G3 放开 judge:受管级别写 judge 必须写预算(judge.max_cost_usd,或显式写 null 表示不限)
- 发布前自动修复:缺预算时给选项(默认金额 / 不限),不替人选;
从 require_citation 改成 judge 不算降低要求,judge 改回 off、withhold 放宽成 degrade 算降低要求
- Copilot 手册:受管模板可以写 judge 并带预算;问数据按需裁判,不替人打开结论句策略
五、界面
- 结论句句末放徽标:支持 ◆、部分支持 ◇、不支持 ! 整句浅底、未裁判 ?;不用表示「已核对」的绿色;n / N 也跳到不支持的句子
- 证据面板里「模型的解释」:带「模型判断 · 模型名 · 非确定」,封存后追加的另外标出;
「请模型判断这句」只在探索运行、还没判过时出现,触顶时写明已到上限和怎么调
- 出具横幅多一段结论句计数;设置页新增「证据裁判」一组(裁判模型、各项上限、今日花费),设成不限时写明费用受什么约束;
报告节点可以配 judge;运行轨迹里裁判相关的事件都有中文标题
六、门禁修正
- 引用模式下,图里没有口径卡(只引查询单元格)的,出具契约不再要求 metrics_from,受管门禁也不再要求 required:
strict 加 cells 已经保证每个数都点得开、对不上就不予出具;有口径卡的照旧要求。
以前问数据的图怎么都发不成受管,Copilot 补上契约也会被判「冒出新的问题」作废
This commit is contained in:
@@ -207,14 +207,21 @@ NODE_REFERENCE = """\
|
||||
- 句末的 [[see:m:指标id,Q1,K1]]:这句结论的依据,不显示
|
||||
metrics_from 不写就取所有上游口径卡;on_violation 不写时探索运行是 flag(标出来、照常产出),正式运行是 fail;
|
||||
numbers 默认 strict(裸数字算违规)。claims 是结论句策略:off(默认,不管)/ require_citation(每句结论都要
|
||||
挂引用,没挂的计入缺口、出具降档);judge(模型裁判)在后续版本支持,现在写了整张图跑不起来。entities 默认 link
|
||||
挂引用,没挂的计入缺口、出具降档)/ judge(在 require_citation 之外,再请另一个模型按每句挂的依据判断支不支持:
|
||||
正式运行在节点里判,证据不支持的按 judge.on_unsupported 处置——degrade 降档(默认)或 withhold 不予出具;
|
||||
探索运行不在节点里花钱,读的人点开哪句再按需判哪句,只标注)。judge 的子配置写在节点的 judge 里:judge.model、
|
||||
judge.provider 指定裁判模型(建议和写作模型不同,同一个模型等于自己审自己;不写就用设置里的「证据裁判模型」);
|
||||
judge.max_cost_usd、judge.max_claims、judge.timeout_s 是每份报告的裁判金额、句数、时长上限,写 null 表示不限,
|
||||
不写取设置里的默认;judge.rewrite_once 默认 false,打开后把证据不支持的句子交回写作者只改一次。entities 默认 link
|
||||
(核对表名、字段名),报告里根本不提表名时才写 off
|
||||
- output 的出具契约:config.contract = {report_from: 报告节点id, metrics_from:[口径卡id],
|
||||
required:[必需指标id], expected:[期望指标id], strict: true}。成果字段写 {{ nodes.报告节点id.text }},
|
||||
前后不要拼别的字——拼了就没法逐段对应证据,出具会降档。受管模板的报告要直接引用查询单元格
|
||||
([[v:]] / [[table:]])的,契约里写 cells: true,否则这些引用按解析不了算
|
||||
([[v:]] / [[table:]])的,契约里写 cells: true,否则这些引用按解析不了算;只引单元格、图里没有口径卡的
|
||||
(问数据的典型图),契约不写 metrics_from,也不写 required / expected
|
||||
要按受管级别发布的模板(发布前检查不过就发不出去):报告节点写明 numbers: "strict"、on_violation: "fail"、
|
||||
claims: "require_citation";契约用 report_from(受管级别不收 narrative),成果字段只取报告节点的正文;给口径卡
|
||||
claims: "require_citation"(或者更严的 claims: "judge",这时 judge 里必须写 judge.max_cost_usd——写一个金额,或者
|
||||
写 null 表示不限,预算要用户定,拿不准就问);契约用 report_from(受管级别不收 narrative),成果字段只取报告节点的正文;给口径卡
|
||||
供数的 agent 配 output_schema 和 cite_fields: true,code 节点喂口径卡的必须是真取数(evidence_role: "source"),
|
||||
否则把计算写进口径卡;llm、agent、协作团队写的文字不能绕过报告节点直接进出口,llm 也不能喂口径卡
|
||||
|
||||
@@ -321,7 +328,7 @@ def _user_message(payload: GenerateIn, *, patch: bool) -> str:
|
||||
# 三期起报告会核对写出来的表名、字段名:instructions 里让它「列出用到的表」时,编出来的名字会被标出来
|
||||
"- report 提到表名、字段名时写 [[t:表名]] / [[c:表名.字段名]] 或放进反引号,只写 agent 真查过的名字,"
|
||||
"编造的会被标出来;要引用知识库原话就在 report 前接 retrieve,用 [[q:K1|原话]] 逐字引用。问数是探索运行,"
|
||||
"report 的 claims 不用写\n"
|
||||
"report 的 claims 不用写:结论句支不支持,由读答案的人点开时按需请模型判断\n"
|
||||
"- 用户要的是周报、要三档出具时,按搭图规则配口径卡和 report_from 契约\n"
|
||||
"- 不要用 transform 解析 agent / llm 的文字:要把 agent 查到的数交给下游,就给它配 output_schema + "
|
||||
"cite_fields\n"
|
||||
@@ -1633,7 +1640,11 @@ _ASSIST_RULES = """\
|
||||
审批策略只能往严里改(每次调用都审批 > 仅危险工具需要审批 > 全部自动放行),不放宽、不删掉,更不改成 never;
|
||||
不取消子工作流钉住的版本;发布级别不是你能改的。结构要动(比如换掉协作团队),输出 question 说明怎么拆,不要自己动手;
|
||||
报告撰写节点的 numbers、on_violation、claims 只能往严里改(numbers: strict、on_violation: fail、
|
||||
claims: require_citation),不放宽、不删掉;契约里的 claims 也一样;不关 agent 的 cite_fields,不删它的 output_schema
|
||||
claims: require_citation),不放宽、不删掉;claims 从宽到严是 off < require_citation < judge,已经是 judge 的不改回
|
||||
require_citation;judge 缺预算(judge.max_cost_usd)时输出 question 问写多少、还是写 null 不限,不替人定;
|
||||
judge.on_unsupported 是 withhold 的不改回 degrade、不删掉;update_node 会把 judge 整个换掉,补预算时把原来的
|
||||
judge(包括从全图默认继承来的)整份带上,只加 max_cost_usd;契约里的 claims 也一样;
|
||||
不关 agent 的 cite_fields,不删它的 output_schema
|
||||
- 要人拿主意的,不替人选、不编:required 该包括哪些指标、几张口径卡该用哪张、钉哪个版本、协作团队拆成哪几个
|
||||
固定步骤、沙箱代码节点是在取数还是在计算(不许自己把 evidence_role 设成 source)……每一处输出一行
|
||||
{"op":"question","node_id":"节点 id","text":"要问的话,把候选列出来"},那一处不改
|
||||
|
||||
+340
-10
@@ -20,23 +20,30 @@ artifacts 表可以事后插行,run.output 是可以改写的一列,封存
|
||||
查询步骤只给被引用的行和前后各 2 行,完整快照仍走 /api/artifacts/{id}。数据源
|
||||
options.mask_columns 里的列换成「已遮罩」——在有身份体系之前,这只减少暴露,不是
|
||||
安全边界:知道工件 id 的人照样能取到整份快照。
|
||||
|
||||
结论句的判定(四期)是模型给的,不是证据:正式运行的判定在报告文档里、随文档封存;探索运行
|
||||
按需裁判(POST …/evidence/judge),判定作为 evidence.judged 事件追加在封存之后,只填补没判过的
|
||||
句子。交给裁判的摘录同样只取封存范围内的工件。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import contextlib
|
||||
import csv
|
||||
import io
|
||||
import re
|
||||
from collections import OrderedDict
|
||||
from collections.abc import Callable
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, Query
|
||||
from fastapi import APIRouter, Body, Query
|
||||
from fastapi.responses import JSONResponse, Response
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy import select, update
|
||||
|
||||
from app.api.coded import CodedHTTPException
|
||||
from app.core.artifact_store import load
|
||||
from app.core.events import EventType
|
||||
from app.data.engine import masked_columns
|
||||
from app.db.base import SessionLocal
|
||||
from app.db.models import DataSource, Run, RunEvent, Workflow
|
||||
@@ -54,6 +61,7 @@ from app.engine.evidence import (
|
||||
guess_sources,
|
||||
input_eid,
|
||||
iter_units,
|
||||
ledger_enabled,
|
||||
make_eid,
|
||||
normalize_quote,
|
||||
query_entry_fields,
|
||||
@@ -61,6 +69,7 @@ from app.engine.evidence import (
|
||||
sql_tables,
|
||||
uncited_claims,
|
||||
)
|
||||
from app.engine.judge import JUDGED, VERDICTS, candidates
|
||||
from app.engine.toolcalls import QUERY_PREFIX
|
||||
|
||||
router = APIRouter(prefix="/api/runs", tags=["evidence"])
|
||||
@@ -120,8 +129,17 @@ class _Sealed:
|
||||
retrievals: dict[str, dict[str, Any]] = field(default_factory=dict)
|
||||
#: 封存时记下的清单哈希(没封存的是 None):旧答案的猜测按 (run_id, 它) 缓存
|
||||
manifest_hash: str | None = None
|
||||
#: 运行记录上的 manifest_seq 原值(老运行是 None):按需裁判追加判定前核对封存没换过
|
||||
run_manifest_seq: int | None = None
|
||||
#: verify_manifest 的完整结论(审计表照实附上)
|
||||
verdict: dict[str, Any] = field(default_factory=dict)
|
||||
#: 运行记录上的几样:按需裁判只给跑完、封存了的探索运行,升级前发起的不给
|
||||
run_class: str | None = None
|
||||
status: str | None = None
|
||||
ledger_on: bool = False
|
||||
#: 按需裁判的判定(evidence.judged 事件),按先后:(seq, 节点, 载荷)。不管落在封存前后都收——
|
||||
#: 失败后接着跑的运行会把早先追加的判定封进新的清单里,它仍然是按需裁判的批注
|
||||
judged: list[tuple[int, str | None, dict[str, Any]]] = field(default_factory=list)
|
||||
|
||||
@property
|
||||
def sealed(self) -> bool:
|
||||
@@ -161,14 +179,18 @@ async def _sealed(run_id: str) -> _Sealed:
|
||||
select(RunEvent.seq, RunEvent.type, RunEvent.node_id, RunEvent.data)
|
||||
.where(RunEvent.run_id == run_id).order_by(RunEvent.seq))]
|
||||
graph = run.graph if isinstance(run.graph, dict) else {}
|
||||
manifest = run.manifest_hash
|
||||
manifest, manifest_seq = run.manifest_hash, run.manifest_seq
|
||||
run_class, status, ledger_on = run.run_class, run.status, ledger_enabled(run)
|
||||
verdict = await verify_manifest(run_id)
|
||||
bound = verdict.get("sealed_at") if verdict.get("sealed") else None
|
||||
# 没封存(还在跑、停在审批)的运行没有「封存之后」:现有的事件都算,但每件证据都标未封存
|
||||
events = [r for r in rows if bound is None or r[0] <= bound]
|
||||
out = _Sealed(run_id=run_id, graph=graph, events=events, seal={
|
||||
"sealed": bool(verdict.get("sealed")), "ok": verdict.get("ok"), "manifest_seq": bound,
|
||||
"legacy": bool(verdict.get("legacy", False))}, manifest_hash=manifest, verdict=verdict)
|
||||
"legacy": bool(verdict.get("legacy", False))}, manifest_hash=manifest, run_manifest_seq=manifest_seq,
|
||||
verdict=verdict, run_class=run_class, status=status, ledger_on=ledger_on,
|
||||
judged=[(seq, nid, data) for seq, etype, nid, data in rows
|
||||
if etype == EventType.EVIDENCE_JUDGED and isinstance(data, dict)])
|
||||
for _, etype, node_id, data in events:
|
||||
if not node_id or not isinstance(data, dict):
|
||||
continue
|
||||
@@ -306,6 +328,7 @@ async def evidence_graph(run_id: str) -> dict[str, Any]:
|
||||
"node_id": report.node_id, "doc_artifact": report.doc_artifact,
|
||||
"doc_sealed": sealed.trusted, "hash_ok": report.hash_ok, "ok": report.ok,
|
||||
"repairs": report.repairs, "fields": report.fields, "stats": report.stats,
|
||||
**_judgement_of(report, sealed),
|
||||
})
|
||||
if report.doc is not None:
|
||||
evidence, edges = _graph_of(report, sealed, inputs)
|
||||
@@ -666,13 +689,16 @@ async def evidence_segment(run_id: str, segment_id: str, report: str | None = No
|
||||
chain = await _chain(seg, doc, sealed, chosen.node_id)
|
||||
covered = sealed.trusted and all(step.get("sealed") for step in chain if "sealed" in step)
|
||||
masked = list(dict.fromkeys(c for step in chain if step.get("step") == "query" for c in step.get("masked") or []))
|
||||
verdict = _effective_verdicts(chosen, sealed)[0].get(str(unit.get("id")))
|
||||
return {
|
||||
"report": {"node_id": chosen.node_id, "doc_artifact": chosen.doc_artifact},
|
||||
"segment": {"id": seg["id"], "text": seg.get("text"), "kind": seg.get("kind"), "state": seg.get("state"),
|
||||
"span": seg.get("span"), "unit": unit.get("id"),
|
||||
**{k: seg[k] for k in ("ref", "issue", "strong", "cite") if k in seg}},
|
||||
"unit": {"id": unit.get("id"), "kind": unit.get("kind"), "text": markdown[start:end],
|
||||
"span": unit.get("span"), "cites": unit.get("cites") or []},
|
||||
"span": unit.get("span"), "cites": unit.get("cites") or [],
|
||||
**({"verdict": verdict} if verdict is not None else {}),
|
||||
"on_demand": _on_demand(sealed, doc, str(unit.get("id")), verdict)},
|
||||
"block": {"id": block.get("id"), "type": block.get("type")},
|
||||
"chain": chain,
|
||||
"note": _note(seg, unit),
|
||||
@@ -1151,8 +1177,304 @@ def _note(seg: dict[str, Any], unit: dict[str, Any]) -> str:
|
||||
cites = [str(c) for c in unit.get("cites") or []]
|
||||
if cites:
|
||||
return (f"这是结论句里的文字,这句挂了依据:{'、'.join(cites[:4])}{' 等' if len(cites) > 4 else ''}。"
|
||||
"依据支不支持这句话,要到后续版本由模型判断")
|
||||
return "这是结论句里的文字,这句没有挂依据。句子本身有没有依据要到后续版本由模型判断"
|
||||
"依据支不支持这句话,系统核对不了,只能由模型判断(模型的判断不是确定性的)")
|
||||
return "这是结论句里的文字,这句没有挂依据。句子本身有没有依据,系统核对不了,只能由模型判断(模型的判断不是确定性的)"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 结论句的判定:封存内的(正式运行在节点里判的)和封存之后按需追加的
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
JUDGE_FORMAL = "evidence_judge_formal"
|
||||
JUDGE_UNSEALED = "evidence_judge_unsealed"
|
||||
JUDGE_LEGACY = "evidence_judge_legacy"
|
||||
JUDGE_BAD_UNITS = "evidence_judge_bad_units"
|
||||
SEAL_BROKEN_CODE = "evidence_seal_broken"
|
||||
#: 一次最多点名几句。一份报告通常几十句,再多就是请求写错了
|
||||
JUDGE_MAX_UNITS = 200
|
||||
#: 跑完了的几种终态。还在跑、停在审批、正在接着跑的运行,追加的判定会被封进下一次的清单,不判
|
||||
_TERMINAL = ("succeeded", "failed", "cancelled")
|
||||
|
||||
FORMAL_JUDGED = "正式运行的裁判在节点内完成(claims: judge),判定随报告文档一起封存,不能再按需裁判"
|
||||
UNSEALED_JUDGE = "运行还没跑完封存(还在跑、停在人工审批,或者正在接着跑),跑完再请模型判断"
|
||||
LEGACY_JUDGE = "这次运行发起于证据台账上线之前,不支持按需裁判"
|
||||
BROKEN_JUDGE = "封存核对没通过:封存之后有事件被改过、删过或插过,报告和证据都不能再当真,不再请模型判断"
|
||||
JUDGED_ALREADY = "这句已经判过了"
|
||||
#: 裁判期间运行被接着跑了(失败的运行可以继续),或者接着跑完重新封存了:判定照样交回,不记进运行记录
|
||||
NOT_RECORDED = "运行已经接着跑了,这次的判定没有记进运行记录(只在这里看得到,刷新就没了);等它跑完封存,再请模型判断"
|
||||
NOT_A_CLAIM = "不是结论句(标题、连接性的话、表格单元格、代码),不需要模型判断"
|
||||
BAD_UNITS = f'units 要写成句子编号的列表,比如 {{"units": ["u4"]}},一次最多 {JUDGE_MAX_UNITS} 句;report 要写报告节点的 id'
|
||||
#: 触顶时告诉人怎么调:设置页「证据裁判」一组的各项上限都能调高,也都能设成不限
|
||||
JUDGE_ADJUST = {
|
||||
"max_cost_usd": "到设置 → 证据裁判里调高「每次点击」的金额上限,或者设成不限",
|
||||
"daily_max_usd": "今天的裁判花费到了全局每日上限:到设置 → 证据裁判里调高「每日」上限或者设成不限,也可以明天再判",
|
||||
"max_claims": "一次最多判这么多句:分几次点,或者到设置 → 证据裁判里调高「每份报告的句数」"
|
||||
"(报告节点上写了 judge.max_claims 的按节点,写 null 是不限)",
|
||||
"timeout_s": "裁判超时了:到设置 → 证据裁判里调高「时长」上限或者设成不限,再点一次",
|
||||
}
|
||||
|
||||
|
||||
def _verdict_ok(verdict: Any) -> bool:
|
||||
return isinstance(verdict, dict) and verdict.get("status") in VERDICTS
|
||||
|
||||
|
||||
def _effective_verdicts(report: _Report, sealed: _Sealed
|
||||
) -> tuple[dict[str, dict[str, Any]], dict[str, dict[str, Any]]]:
|
||||
"""这份报告每句的判定:(生效的, 其中封存之后按需追加的)。
|
||||
|
||||
文档里的判定随文档封存(正式运行在节点里判的;探索运行的候选句记「未裁判 · 按需」)。按需裁判的判定在
|
||||
evidence.judged 事件里,只认指向这份文档、文档里确有其句的;它只填补没判过的句子——封存内判过的,后面
|
||||
追加什么都盖不掉,一条追加的事件改写不了封存的结论。同一句按需判过几次(先触顶、后判成),判成的为准,
|
||||
其次取最后一次。追加的判定一律标 post_seal。
|
||||
"""
|
||||
doc = report.doc or {}
|
||||
ids = {str(u.get("id")) for _, u in iter_units(doc)}
|
||||
out = {str(u.get("id")): u["verdict"] for _, u in iter_units(doc) if _verdict_ok(u.get("verdict"))}
|
||||
sealed_judged = {uid for uid, v in out.items() if v.get("status") in JUDGED}
|
||||
later: dict[str, dict[str, Any]] = {}
|
||||
for _, _, data in sealed.judged:
|
||||
if data.get("report") != report.node_id or data.get("doc_artifact") != report.doc_artifact:
|
||||
continue
|
||||
verdicts = data.get("verdicts") if isinstance(data.get("verdicts"), dict) else {}
|
||||
for uid, verdict in verdicts.items():
|
||||
uid = str(uid)
|
||||
if uid not in ids or uid in sealed_judged or not _verdict_ok(verdict):
|
||||
continue
|
||||
if later.get(uid, {}).get("status") in JUDGED and verdict.get("status") not in JUDGED:
|
||||
continue
|
||||
later[uid] = out[uid] = {**verdict, "post_seal": True}
|
||||
return out, later
|
||||
|
||||
|
||||
def _judgement_of(report: _Report, sealed: _Sealed) -> dict[str, Any]:
|
||||
"""证据图、审计表里每份报告带的结论句情况:写作时的策略、封存的裁判摘要、封存之后按需追加的判定。"""
|
||||
doc = report.doc or {}
|
||||
return {"claims": report.claims, "judge": doc.get("judge") if isinstance(doc.get("judge"), dict) else None,
|
||||
"post_seal_verdicts": _effective_verdicts(report, sealed)[1] if report.doc is not None else {}}
|
||||
|
||||
|
||||
def _on_demand(sealed: _Sealed, doc: dict[str, Any], uid: str, verdict: dict[str, Any] | None) -> dict[str, Any]:
|
||||
"""这句能不能「请模型判断」:{available, reason, message}。和 POST …/evidence/judge 同一套条件。"""
|
||||
def no(reason: str, message: str) -> dict[str, Any]:
|
||||
return {"available": False, "reason": reason, "message": message}
|
||||
|
||||
if sealed.run_class == "formal":
|
||||
return no("formal", FORMAL_JUDGED)
|
||||
if not sealed.ledger_on:
|
||||
return no("legacy", LEGACY_JUDGE)
|
||||
if not candidates(doc, [uid]):
|
||||
return no("not_a_claim", NOT_A_CLAIM)
|
||||
if isinstance(verdict, dict) and verdict.get("status") in JUDGED:
|
||||
return no("judged", JUDGED_ALREADY)
|
||||
if not sealed.sealed or sealed.status not in _TERMINAL:
|
||||
return no("unsealed", UNSEALED_JUDGE)
|
||||
if not sealed.trusted:
|
||||
return no("seal_broken", BROKEN_JUDGE)
|
||||
return {"available": True, "reason": None, "message": None}
|
||||
|
||||
|
||||
def _sealed_loader(sealed: _Sealed) -> Callable[[str], Any]:
|
||||
"""给裁判取证据的 loader:只认封存范围内的事件交回过的工件(查询、表结构、检索快照和台账里记过的),
|
||||
取回时照旧复验哈希。别的一律当取不到——人为插进工件表的行、封存之后才追加的事件引用的快照,
|
||||
都不能送进裁判的摘录。报告文档里的目录也不例外:目录写着哪件工件,不等于它在封存链上。"""
|
||||
allowed = {*sealed.queries, *sealed.schemas, *sealed.retrievals, *(str(a) for _, a in sealed.ledger)}
|
||||
|
||||
def loader(artifact: str) -> Any:
|
||||
return load(artifact) if sealed.trusted and str(artifact) in allowed else None
|
||||
return loader
|
||||
|
||||
|
||||
def _node_judge(sealed: _Sealed, node_id: str) -> dict[str, Any]:
|
||||
"""这次运行的图上报告节点写的 judge(裁判模型、句数和时长上限);图读不出来就当没写。"""
|
||||
from app.engine.nodes.report import report_judge
|
||||
from app.engine.schema import GraphSpec
|
||||
|
||||
try:
|
||||
spec = GraphSpec.model_validate(sealed.graph)
|
||||
node = spec.node_map().get(node_id)
|
||||
except Exception: # noqa: BLE001 - 老运行的图可能已经不合现在的格式
|
||||
node = None
|
||||
return report_judge(spec, node) if node is not None else {}
|
||||
|
||||
|
||||
class _RunLocks:
|
||||
"""每次运行一把锁:同一次运行的按需裁判排队,连点两下时后一下看得到前一下记下的判定,不重复花钱。"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._locks: dict[str, asyncio.Lock] = {}
|
||||
self._users: dict[str, int] = {}
|
||||
|
||||
@contextlib.asynccontextmanager
|
||||
async def hold(self, key: str): # type: ignore[no-untyped-def]
|
||||
lock = self._locks.setdefault(key, asyncio.Lock())
|
||||
self._users[key] = self._users.get(key, 0) + 1
|
||||
try:
|
||||
async with lock:
|
||||
yield
|
||||
finally:
|
||||
self._users[key] -= 1
|
||||
if not self._users[key]:
|
||||
del self._users[key], self._locks[key]
|
||||
|
||||
|
||||
_JUDGING = _RunLocks()
|
||||
|
||||
|
||||
def _judge_request(payload: Any, report: str | None) -> tuple[list[str], str | None]:
|
||||
if not isinstance(payload, dict):
|
||||
raise CodedHTTPException(422, BAD_UNITS, JUDGE_BAD_UNITS)
|
||||
units = payload.get("units")
|
||||
wanted = report if payload.get("report") is None else payload["report"]
|
||||
if not isinstance(units, list) or not units or len(units) > JUDGE_MAX_UNITS \
|
||||
or not all(isinstance(u, str) and u.strip() for u in units) \
|
||||
or (wanted is not None and not isinstance(wanted, str)):
|
||||
raise CodedHTTPException(422, BAD_UNITS, JUDGE_BAD_UNITS)
|
||||
return list(dict.fromkeys(u.strip() for u in units)), wanted or None
|
||||
|
||||
|
||||
@router.post("/{run_id}/evidence/judge")
|
||||
async def judge_units(run_id: str, payload: Any = Body(None), report: str | None = None) -> dict[str, Any]:
|
||||
"""探索运行里按需裁判几句结论:请求体 {units: ["u4"], report?: 报告节点 id}。
|
||||
|
||||
判过的句子直接给已有的判定,不再调用;没判过的按设置里每次点击的金额上限、每份报告的句数和时长上限、
|
||||
全局每日上限去判,触顶照实说「已到上限」,不是报错。问过模型的这一次记一条 evidence.judged 事件,
|
||||
落在封存之后(post_seal: true)——封存核对只覆盖 manifest_seq 之前的事件,追加它 verify 照旧一致;
|
||||
封存的报告文档一个字都不改。交给裁判的证据只取封存范围内的工件,数据源遮罩的列不给。
|
||||
"""
|
||||
units, wanted = _judge_request(payload, report)
|
||||
async with _JUDGING.hold(run_id):
|
||||
sealed = await _sealed(run_id)
|
||||
if sealed.run_class == "formal":
|
||||
raise CodedHTTPException(409, FORMAL_JUDGED, JUDGE_FORMAL)
|
||||
if not sealed.ledger_on:
|
||||
raise CodedHTTPException(409, LEGACY_JUDGE, JUDGE_LEGACY)
|
||||
if not sealed.sealed or sealed.status not in _TERMINAL:
|
||||
raise CodedHTTPException(409, UNSEALED_JUDGE, JUDGE_UNSEALED)
|
||||
if not sealed.trusted:
|
||||
raise CodedHTTPException(409, BROKEN_JUDGE, SEAL_BROKEN_CODE)
|
||||
reports = _reports(sealed)
|
||||
if not reports:
|
||||
raise CodedHTTPException(404, "这次运行没有封存的报告文档,没有可以判断的句子", REPORT_NOT_FOUND)
|
||||
chosen = _choose(reports, wanted, sealed)
|
||||
if chosen.hash_ok is False:
|
||||
raise CodedHTTPException(409, "报告文档和它的哈希对不上,疑似被改过,不能再当证据展示", DOC_TAMPERED)
|
||||
if chosen.doc is None:
|
||||
raise CodedHTTPException(404, "报告文档在工件库里取不回来", DOC_MISSING)
|
||||
return await _judge_now(sealed, chosen, units)
|
||||
|
||||
|
||||
async def _judge_now(sealed: _Sealed, report: _Report, units: list[str]) -> dict[str, Any]:
|
||||
from app.engine import judge as judging
|
||||
|
||||
doc = report.doc or {}
|
||||
catalog = doc.get("catalog") or {}
|
||||
known = _effective_verdicts(report, sealed)[0]
|
||||
reused = [u for u in units if known.get(u, {}).get("status") in JUDGED]
|
||||
pending = [u for u in units if u not in reused]
|
||||
verdicts = {u: known[u] for u in reused}
|
||||
node_judge = _node_judge(sealed, report.node_id)
|
||||
async with SessionLocal() as session:
|
||||
settings = await judging.judge_settings(session)
|
||||
budget = judging.click_budget(settings, node_judge)
|
||||
outcome: dict[str, Any] | None = None
|
||||
skipped: dict[str, str] = {}
|
||||
event = None
|
||||
lost = False
|
||||
if pending:
|
||||
loader = _sealed_loader(sealed)
|
||||
masked = await judging.source_masks(catalog, loader=loader)
|
||||
request = judging.prepare(doc, catalog, units=pending, loader=loader, masked=masked)
|
||||
skipped = dict(request.skipped)
|
||||
if request.cands:
|
||||
async with SessionLocal() as session:
|
||||
spec = await judging.judge_model_spec(session, node_judge, settings=settings)
|
||||
outcome = await judging.run_request(request, spec=spec, budget=budget, post_seal=True, click=True)
|
||||
verdicts.update(outcome["verdicts"])
|
||||
if outcome["calls"] or outcome["cost_usd"]:
|
||||
# 没问模型(触顶、模型没配好)就没有判定可记:说给点的人听,不往运行记录里追加
|
||||
event = await _append_judged(sealed, report, pending, outcome)
|
||||
lost = event is None
|
||||
limits = list(outcome["limits_hit"]) if outcome else []
|
||||
message = _judge_message(outcome, budget, skipped)
|
||||
if lost:
|
||||
message = ";".join(filter(None, [NOT_RECORDED, message]))
|
||||
return {
|
||||
"run_id": sealed.run_id, "report": {"node_id": report.node_id, "doc_artifact": report.doc_artifact},
|
||||
"units": units, "verdicts": {u: verdicts[u] for u in units if u in verdicts}, "reused": reused,
|
||||
"judged": [u for u in pending if (outcome or {}).get("verdicts", {}).get(u, {}).get("status") in JUDGED],
|
||||
"skipped": skipped, "unjudged": dict(outcome["unjudged"]) if outcome else {},
|
||||
"limits_hit": limits, "limited": bool(limits), "message": message,
|
||||
"adjust": [JUDGE_ADJUST[r] for r in limits if r in JUDGE_ADJUST],
|
||||
"gaps": list(outcome["gaps"]) if outcome else [], "notes": list(outcome["notes"]) if outcome else [],
|
||||
"model": (outcome or {}).get("model"), "priced": (outcome or {}).get("priced"),
|
||||
"cost_usd": (outcome or {}).get("cost_usd", 0.0), "calls": (outcome or {}).get("calls", 0),
|
||||
"duration_ms": (outcome or {}).get("duration_ms", 0),
|
||||
"budget": budget.as_dict() if outcome else None, "event": event, "post_seal": True,
|
||||
"spend": {**await judging.daily_spend(), "daily_max_usd": settings.get("daily_max_usd")},
|
||||
}
|
||||
|
||||
|
||||
def _judge_message(outcome: dict[str, Any] | None, budget: Any, skipped: dict[str, str]) -> str | None:
|
||||
"""这次点击的一句话结论:触顶、没跑成、点的不是结论句;判成了(或者全是判过的)就没什么要说的。"""
|
||||
from app.engine.judge import _limit_label
|
||||
|
||||
if outcome and outcome["limits_hit"]:
|
||||
labels = "、".join(_limit_label(r, budget, click=True) for r in outcome["limits_hit"])
|
||||
n = sum(outcome["unjudged"].get(r, 0) for r in outcome["limits_hit"])
|
||||
return f"已到上限({labels}):{n} 句没判,记为未裁判;已判的保留"
|
||||
if outcome and outcome["gaps"]:
|
||||
return "裁判没跑完:" + ";".join(outcome["gaps"])
|
||||
parts = []
|
||||
if others := [u for u, why in skipped.items() if why == "not_a_claim"]:
|
||||
parts.append(f"{'、'.join(others)} {NOT_A_CLAIM}")
|
||||
if missing := [u for u, why in skipped.items() if why == "not_found"]:
|
||||
parts.append(f"报告里没有 {'、'.join(missing)}")
|
||||
return ";".join(parts) or None
|
||||
|
||||
|
||||
async def _append_judged(sealed: _Sealed, report: _Report, units: list[str],
|
||||
outcome: dict[str, Any]) -> dict[str, Any] | None:
|
||||
"""记一条 evidence.judged:问过模型的这一次(连同没判成的句子),接在运行记录的最后一条后面。
|
||||
|
||||
只接在这次点击开始时的那份封存后面:裁判要花几秒(时长不限时更久),这中间有人点了「继续运行」
|
||||
(失败的运行可以接着跑),再追加就落进一次活着的运行中间,还会被封进下一份清单;接着跑完、重新封存了
|
||||
也一样,这次的判定依据的已经不是最新的封存。这两种都不记,返回 None(判定照样交回给点的人)。
|
||||
核对和写入是同一个事务里的条件更新,和接着跑占住运行(runner._claim)抢的是同一行:谁先提交算谁的。
|
||||
"""
|
||||
from app.core.bus import bus
|
||||
from app.engine.runner import _stage, run_manager
|
||||
|
||||
same_seal = (Run.manifest_seq.is_(None) if sealed.run_manifest_seq is None
|
||||
else Run.manifest_seq == sealed.run_manifest_seq,
|
||||
Run.manifest_hash.is_(None) if sealed.manifest_hash is None
|
||||
else Run.manifest_hash == sealed.manifest_hash)
|
||||
async with SessionLocal() as session:
|
||||
run = await session.get(Run, sealed.run_id)
|
||||
if run is None:
|
||||
return None
|
||||
# 进程重启过的话内存里的序号从 0 起:接在库里最后一条后面,不和已有的撞号
|
||||
bus.set_seq(sealed.run_id, run.last_seq or 0)
|
||||
event = run_manager._event(
|
||||
sealed.run_id, EventType.EVIDENCE_JUDGED, node_id=report.node_id, data={
|
||||
"report": report.node_id, "doc_artifact": report.doc_artifact, "units": units,
|
||||
"verdicts": outcome["verdicts"],
|
||||
"keys": {u: outcome["keys"][u] for u in outcome["verdicts"] if u in outcome["keys"]},
|
||||
"model": outcome["model"], "priced": outcome["priced"], "cost_usd": outcome["cost_usd"],
|
||||
"calls": outcome["calls"], "duration_ms": outcome["duration_ms"], "budget": outcome["budget"],
|
||||
"limits_hit": outcome["limits_hit"], "unjudged": outcome["unjudged"], "gaps": outcome["gaps"],
|
||||
"notes": outcome["notes"], "skipped": outcome["skipped"], "post_seal": True})
|
||||
claimed = await session.execute(
|
||||
update(Run).where(Run.id == sealed.run_id, Run.status.in_(_TERMINAL), *same_seal)
|
||||
.values(last_seq=event.seq))
|
||||
if claimed.rowcount != 1:
|
||||
# 运行已经不是那份封存下的终态了。烧掉一个序号不要紧:不落库的事件(令牌流)本来就占序号,
|
||||
# 序号只要求递增、不撞号
|
||||
await session.rollback()
|
||||
return None
|
||||
await _stage(session, event, run)
|
||||
await session.commit()
|
||||
await bus.publish(event)
|
||||
return {"seq": event.seq}
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
@@ -1172,6 +1494,11 @@ UNCITED_CLAIM_COUNTED = "这句结论没有挂依据:这份报告要求结论
|
||||
UNCITED_CLAIM_WITHHELD = "这句结论没有挂依据:出具契约要求结论句挂依据,没挂的按契约不予出具(on_uncited: withhold)"
|
||||
UNCITED_CLAIM_REQUIRED = ("这句结论没有挂依据:这份报告要求结论句挂依据(claims: require_citation),不过没有出具契约"
|
||||
"按它判档(没配契约,或者还没到出具那一步)")
|
||||
#: claims: judge 在挂依据这件事上和 require_citation 一样严(出口按 on_uncited 计入缺口)
|
||||
UNCITED_CLAIM_JUDGED = ("这句结论没有挂依据:这份报告的结论句由模型裁判(claims: judge),挂依据的要求和 "
|
||||
"require_citation 一样,出具时计入缺口")
|
||||
UNCITED_CLAIM_JUDGE_REQUIRED = ("这句结论没有挂依据:这份报告的结论句由模型裁判(claims: judge),要求结论句挂依据,"
|
||||
"不过没有出具契约按它判档(没配契约,或者还没到出具那一步)")
|
||||
LEGACY_UNMATCHED = "旧版出具里这个数字对不上任何指标"
|
||||
|
||||
#: CSV 的列:(表头, 行里的键)
|
||||
@@ -1239,7 +1566,7 @@ async def evidence_audit(run_id: str, fmt: str | None = Query(None, alias="forma
|
||||
"run_id": run_id, "schema": AUDIT_SCHEMA, "mode": mode,
|
||||
"seal": {**sealed.seal, "events": sealed.verdict.get("events"), "message": sealed.verdict.get("message")},
|
||||
"reports": [{"node_id": r.node_id, "doc_artifact": r.doc_artifact, "hash_ok": r.hash_ok,
|
||||
"doc_sealed": sealed.trusted, "fields": r.fields, "claims": r.claims, "stats": r.stats}
|
||||
"doc_sealed": sealed.trusted, "fields": r.fields, "stats": r.stats, **_judgement_of(r, sealed)}
|
||||
for r in reports],
|
||||
"groups": grouped, "counts": {g["key"]: g["count"] for g in grouped},
|
||||
"total": sum(g["count"] for g in grouped),
|
||||
@@ -1299,11 +1626,14 @@ def _claim_note(report: _Report, issuance: dict[str, Any] | None) -> str:
|
||||
if isinstance(issuance, dict) and issuance.get("mode") == "citations" \
|
||||
and (issuance.get("report") or {}).get("node_id") == report.node_id:
|
||||
claims = issuance.get("claims")
|
||||
if not isinstance(claims, dict) or claims.get("policy") != "require_citation":
|
||||
if not isinstance(claims, dict) or claims.get("policy") not in ("require_citation", "judge"):
|
||||
return UNCITED_CLAIM
|
||||
on_uncited = claims.get("on_uncited") or "degrade"
|
||||
return UNCITED_CLAIM_WITHHELD if on_uncited == "withhold" else \
|
||||
UNCITED_CLAIM if on_uncited == "ignore" else UNCITED_CLAIM_COUNTED
|
||||
UNCITED_CLAIM if on_uncited == "ignore" else \
|
||||
UNCITED_CLAIM_JUDGED if claims.get("policy") == "judge" else UNCITED_CLAIM_COUNTED
|
||||
if report.claims == "judge":
|
||||
return UNCITED_CLAIM_JUDGE_REQUIRED
|
||||
return UNCITED_CLAIM_REQUIRED if report.claims == "require_citation" else UNCITED_CLAIM
|
||||
|
||||
|
||||
|
||||
@@ -14,6 +14,9 @@ from app.core.config import settings as app_settings
|
||||
from app.core.crypto import encrypt, mask
|
||||
from app.db.base import get_session
|
||||
from app.db.models import Provider, Setting
|
||||
from app.engine.judge import JUDGE_DEFAULTS, SETTING_GROUP as JUDGE_GROUP, SPEND_KEY as JUDGE_SPEND
|
||||
from app.engine.judge import daily_spend, judge_settings
|
||||
from app.engine.judge import sanitize_settings as judge_values, settings_problem as judge_problem
|
||||
from app.providers import catalog
|
||||
from app.providers.factory import ModelSpec, ProviderNotConfigured, _resolve_key, build_chat_model
|
||||
|
||||
@@ -434,6 +437,9 @@ DEFAULT_SETTINGS: dict[str, Any] = {
|
||||
"max_agent_steps": app_settings.max_agent_steps,
|
||||
"max_run_seconds": app_settings.max_run_seconds,
|
||||
},
|
||||
# 结论句裁判(engine/judge.py):证据裁判模型和各项上限,上限存 null 表示不限。不放进 limits:
|
||||
# 那一组是环境变量定的、只读,这一组是用户自己定的
|
||||
JUDGE_GROUP: dict(JUDGE_DEFAULTS),
|
||||
}
|
||||
|
||||
|
||||
@@ -482,7 +488,11 @@ async def get_settings(session: AsyncSession = Depends(get_session)) -> dict[str
|
||||
# limits 是服务端配置(环境变量),只读:库里可能存着旧版设置页写进去的一份
|
||||
#(见过 max_agent_steps: 25),它从来不生效,却会盖住真实的上限,界面照着错的数许诺
|
||||
stored.pop("limits", None)
|
||||
# 裁判的每日计数是引擎记的账,不是用户设置:设置页整组回写时带着一份旧的,会把今天的花费清零
|
||||
stored.pop(JUDGE_SPEND, None)
|
||||
merged = {k: {**v, **(stored.get(k) or {})} for k, v in DEFAULT_SETTINGS.items()}
|
||||
# 裁判设置按引擎实际生效的样子给:存坏了的项显示默认,存了 null 的上限显示不限
|
||||
merged[JUDGE_GROUP] = judge_values(stored.get(JUDGE_GROUP))
|
||||
for key, value in stored.items():
|
||||
merged.setdefault(key, value)
|
||||
return merged
|
||||
@@ -498,10 +508,14 @@ async def put_settings(
|
||||
) -> dict[str, Any]:
|
||||
from app.tools.trust import SETTING_KEY as TRUST_KEY
|
||||
|
||||
if JUDGE_GROUP in payload.values and (problem := judge_problem(payload.values[JUDGE_GROUP])):
|
||||
# 先查后写:一组里有一项不对就整个请求不落库,免得存下半组
|
||||
raise HTTPException(422, problem)
|
||||
for key, value in payload.values.items():
|
||||
if key in (TRUST_KEY, "limits"):
|
||||
if key in (TRUST_KEY, "limits", JUDGE_SPEND):
|
||||
# 信任三档只走 PUT /api/tools/trust:设置页整组回写时带着一份旧的,
|
||||
# 会把刚在工具页改的档位冲掉。limits 是环境变量定的,写进库也不生效
|
||||
# 会把刚在工具页改的档位冲掉。limits 是环境变量定的,写进库也不生效。
|
||||
# 裁判的每日计数是引擎记的账,只有裁判自己写
|
||||
continue
|
||||
row = await session.get(Setting, key)
|
||||
if row:
|
||||
@@ -512,6 +526,16 @@ async def put_settings(
|
||||
return await get_settings(session)
|
||||
|
||||
|
||||
@router.get("/settings/judge/spend")
|
||||
async def get_judge_spend(session: AsyncSession = Depends(get_session)) -> dict[str, Any]:
|
||||
"""今天(本地日期)结论句裁判花了多少,设置页放在每日上限旁边。只读:计数只有裁判自己记。
|
||||
|
||||
{date, usd, calls, unpriced_calls, daily_max_usd}。unpriced_calls 是价格目录里没有的模型的调用次数,
|
||||
这些调用估不出金额、没有算进 usd。daily_max_usd 为 null 表示不限。
|
||||
"""
|
||||
return {**await daily_spend(), "daily_max_usd": (await judge_settings(session))["daily_max_usd"]}
|
||||
|
||||
|
||||
@router.get("/system")
|
||||
async def system_info() -> dict[str, Any]:
|
||||
"""给设置页顶部展示的运行环境概览。"""
|
||||
|
||||
@@ -69,6 +69,9 @@ class EventType(StrEnum):
|
||||
# 报告撰写节点的自查结果:报告文档工件 id、统计、违规清单。落在封存范围内,
|
||||
# 证据接口靠它找到文档——不去翻不受封存保护的 artifacts 表
|
||||
REPORT_CHECKED = "report.checked"
|
||||
# 探索运行里按需裁判的结论句判定:点开哪句才判哪句,追加在封存之后(post_seal),是批注不是证据。
|
||||
# 封存核对只覆盖 manifest_seq 之前的事件,追加它不影响 verify;封存的报告文档一个字都不改
|
||||
EVIDENCE_JUDGED = "evidence.judged"
|
||||
|
||||
# 其他
|
||||
LOG = "log"
|
||||
|
||||
+138
-27
@@ -23,8 +23,23 @@ from typing import Any, Callable
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
from app.engine.governance import REPORT_POLICY, exit_text_fields, publish_issues
|
||||
from app.engine.schema import REPORT_CLAIMS, GraphNode, GraphSpec, NodeType, _ancestors, claims_problem
|
||||
from app.engine.governance import (
|
||||
JUDGE_UNLIMITED,
|
||||
REPORT_POLICY,
|
||||
REPORT_POLICY_ALSO,
|
||||
exit_text_fields,
|
||||
publish_issues,
|
||||
)
|
||||
from app.engine.schema import (
|
||||
CLAIMS_JUDGE,
|
||||
JUDGE_ON_UNSUPPORTED,
|
||||
REPORT_CLAIMS,
|
||||
GraphNode,
|
||||
GraphSpec,
|
||||
NodeType,
|
||||
_ancestors,
|
||||
claims_problem,
|
||||
)
|
||||
|
||||
#: 被选中的修复按这个顺序应用:先补契约骨架,再补契约里的各项,最后是节点上的开关。
|
||||
#: 前面的修复可能改变后面的候选(metrics_from 决定 required 从哪几张卡里挑,report_from 决定
|
||||
@@ -34,7 +49,7 @@ _ORDER = (
|
||||
"contract.report_from_missing", "contract.report_from_invalid", "governed.report_from_required",
|
||||
"governed.exit_text_source", "report.metrics_from_invalid",
|
||||
"contract.required_missing", "contract.strict_off", "contract.claims_ignored", "report.claims_invalid",
|
||||
"governed.report_policy",
|
||||
"governed.report_policy", "governed.judge_budget",
|
||||
"governed.caliber_agent_cite_fields", "governed.caliber_compute_input", "governed.agent_approval_never",
|
||||
"governed.default_approval_never", "governed.subgraph_unpinned", "governed.supervisor", "contract.not_object",
|
||||
"governed.text_bypass", "governed.caliber_agent_schema", "governed.caliber_model_input",
|
||||
@@ -210,7 +225,8 @@ def _plan_exit_fields(ctx: _Ctx, issue: dict[str, Any], node: GraphNode | None)
|
||||
|
||||
|
||||
def _plan_report_policy(ctx: _Ctx, issue: dict[str, Any], node: GraphNode | None) -> dict[str, Any] | None:
|
||||
"""G3:报告撰写节点写明最严的三项。答案唯一,只往严里改;写了不支持的 claims 由 claims_invalid 管。"""
|
||||
"""G3:报告撰写节点写明最严的三项。答案唯一,只往严里改;写了不认识的 claims 由 claims_invalid 管。
|
||||
已经写了更严的写法(claims: judge)的不动——改回 require_citation 就是替人放宽了。"""
|
||||
if node is None:
|
||||
return None
|
||||
writes: dict[str, Any] = {}
|
||||
@@ -218,7 +234,9 @@ def _plan_report_policy(ctx: _Ctx, issue: dict[str, Any], node: GraphNode | None
|
||||
for key, want, _ in REPORT_POLICY:
|
||||
value = node.config.get(key)
|
||||
effective = ctx.spec.defaults.get(key) if value in (None, "") else value
|
||||
if effective != want and not (key == "claims" and claims_problem(effective)):
|
||||
if effective in (want, *REPORT_POLICY_ALSO.get(key, ())):
|
||||
continue
|
||||
if not (key == "claims" and claims_problem(effective)):
|
||||
writes[key], before[key] = want, copy.deepcopy(value)
|
||||
if not writes:
|
||||
return None
|
||||
@@ -230,18 +248,38 @@ def _plan_report_policy(ctx: _Ctx, issue: dict[str, Any], node: GraphNode | None
|
||||
|
||||
|
||||
def _plan_claims(ctx: _Ctx, issue: dict[str, Any], node: GraphNode | None) -> dict[str, Any] | None:
|
||||
"""claims 写了不支持的值。受管级别只认 require_citation,答案唯一;已发布级别 off 和 require_citation
|
||||
都行,要人选——写的是 judge(还不支持的模型裁判)时建议 require_citation:它离裁判最近。"""
|
||||
"""claims 写了不认识的值。受管级别改成 require_citation:确定性的两档里只有它合规,答案唯一(judge 要花钱、
|
||||
要写预算,不替人开);已发布级别 off 和 require_citation 都行,要人选。"""
|
||||
if node is None:
|
||||
return None
|
||||
before = node.config.get("claims")
|
||||
if ctx.level == "governed":
|
||||
return _auto(f"把「{node.title}」的 claims 改成 require_citation:受管出具要求每句结论挂引用"
|
||||
+ ("(结论句裁判在后续版本支持)" if before == "judge" else ""), "claims", before, "require_citation")
|
||||
return _auto(f"把「{node.title}」的 claims 改成 require_citation:受管出具要求每句结论挂引用", "claims",
|
||||
before, "require_citation")
|
||||
hints = {"off": "结论句不参与判档,和以前一样", "require_citation": "没挂引用的结论句计入缺口,按出具档位降档"}
|
||||
options = [{"value": v, "label": v, "hint": hints[v]} for v in REPORT_CLAIMS]
|
||||
return _choice(f"选「{node.title}」的结论句策略(claims)", "claims", options,
|
||||
default="require_citation" if before == "judge" else None)
|
||||
return _choice(f"选「{node.title}」的结论句策略(claims)", "claims", options)
|
||||
|
||||
|
||||
def _plan_judge_budget(ctx: _Ctx, issue: dict[str, Any], node: GraphNode | None) -> dict[str, Any] | None:
|
||||
"""G3:claims: judge 没写每份报告的裁判预算。写多少钱只有作者能定:候选是默认金额和「不限」,不替人选、
|
||||
不给建议。judge 原来写在图级 defaults 里的,写到节点上时带上 defaults 那一整份——节点写了 judge 就整个
|
||||
用节点的,只补一个 max_cost_usd 会把 defaults 里选的裁判模型、上限悄悄丢掉。"""
|
||||
from app.engine.judge import JUDGE_DEFAULTS
|
||||
|
||||
if node is None:
|
||||
return None
|
||||
judge = node.config.get("judge")
|
||||
if judge in (None, ""):
|
||||
judge = ctx.spec.defaults.get("judge")
|
||||
base = copy.deepcopy(judge) if isinstance(judge, dict) else {}
|
||||
amount = JUDGE_DEFAULTS["report_max_cost_usd"]
|
||||
options = [
|
||||
{"value": amount, "label": f"每份报告最多 ${amount:g}", "hint": "系统默认的每份报告上限;写进节点以后,改设置不影响这份报告"},
|
||||
{"value": None, "label": "不限(写 null)", "hint": JUDGE_UNLIMITED},
|
||||
]
|
||||
return _choice(f"给「{node.title}」的结论句裁判写每份报告的预算(judge.max_cost_usd)", "judge.max_cost_usd",
|
||||
options, _judge=base)
|
||||
|
||||
|
||||
def _plan_compute_input(ctx: _Ctx, issue: dict[str, Any], node: GraphNode | None) -> dict[str, Any] | None:
|
||||
@@ -419,6 +457,7 @@ _PLANNERS: dict[str, Callable[[_Ctx, dict[str, Any], GraphNode | None], dict[str
|
||||
"governed.text_bypass": _plan_text_bypass,
|
||||
"governed.report_policy": _plan_report_policy,
|
||||
"report.claims_invalid": _plan_claims,
|
||||
"governed.judge_budget": _plan_judge_budget,
|
||||
"governed.caliber_compute_input": _plan_compute_input,
|
||||
"governed.caliber_agent_cite_fields": _plan_cite_fields,
|
||||
"governed.caliber_agent_schema": _plan_agent_schema,
|
||||
@@ -580,12 +619,25 @@ def _edit_exit_fields(graph: dict[str, Any], fix: dict[str, Any], value: Any) ->
|
||||
return out
|
||||
|
||||
|
||||
def _edit_judge_budget(graph: dict[str, Any], fix: dict[str, Any], value: Any) -> list[tuple[str, str, Any, Any]]:
|
||||
"""G3 的裁判预算:节点的 judge 写成(原来生效的那份 judge + 选中的 max_cost_usd)。选「不限」写显式的 null,
|
||||
和没写不是一回事。变更按整份 judge 记:前后都是 null 的 max_cost_usd 在预览里分不出写没写。"""
|
||||
node = _node(graph, fix["node_id"])
|
||||
if node is None:
|
||||
return []
|
||||
config = _config(node)
|
||||
after = {**copy.deepcopy(fix.get("_judge") or {}), "max_cost_usd": value}
|
||||
before = _write(config, "judge", after)
|
||||
return [(str(node["id"]), "judge", before, copy.deepcopy(after))]
|
||||
|
||||
|
||||
#: 每类问题怎么改图。大多是同一种「按节点字段写值」,改全图默认、改成果字段的各走一格;留着这张表是为了
|
||||
#: 某类修复要特殊处理时只换它自己的那一格(测试里也借它模拟一条出错的修复规则)
|
||||
_EDITORS: dict[str, Callable[[dict[str, Any], dict[str, Any], Any], list[tuple[str | None, str, Any, Any]]]] = {
|
||||
code: (_edit_defaults if code in _GRAPH_FIXES else _edit) for code in _PLANNERS
|
||||
}
|
||||
_EDITORS["governed.exit_text_source"] = _edit_exit_fields
|
||||
_EDITORS["governed.judge_budget"] = _edit_judge_budget
|
||||
|
||||
|
||||
def _handoff(fix: dict[str, Any], value: Any) -> bool:
|
||||
@@ -811,25 +863,34 @@ def _contract_loosened(name: str, prior: Any, contract: Any) -> list[str]:
|
||||
if contract.get("cells") is True and prior.get("cells") is not True:
|
||||
out.append(f"替你打开了「{name}」出具契约的 cells(允许报告直接引用查询单元格),这要你自己决定")
|
||||
was, now = _claims_rank(prior.get("claims")), _claims_rank(contract.get("claims"))
|
||||
if was is not None and now is not None and now < was:
|
||||
out.append(f"把「{name}」出具契约的 claims 往宽里改了(没挂依据的结论句{_CLAIMS_WORDS[was]} → "
|
||||
f"{_CLAIMS_WORDS[now]})")
|
||||
if was is not None and now is not None:
|
||||
looser = [f"{what}{words[a]} → {words[b]}" for (what, words), a, b in zip(_CLAIMS_AXES, was, now) if b < a]
|
||||
if looser:
|
||||
out.append(f"把「{name}」出具契约的 claims 往宽里改了({';'.join(looser)})")
|
||||
return out
|
||||
|
||||
|
||||
#: 契约 claims 的严格程度:不查 < 查了不算缺口(ignore)< 计入缺口降档(degrade)< 不予出具(withhold)
|
||||
_CLAIMS_WORDS = ("不检查", "只标出来", "计入缺口", "不予出具")
|
||||
#: 契约 claims 的严格程度分两样,各自从宽到严:没挂依据的结论句(不查 < 查了不算缺口 ignore < 计入缺口降档
|
||||
#: degrade < 不予出具 withhold),证据不支持的结论句(不裁判 < 裁判、降档 < 裁判、不予出具)。judge 两样都管:
|
||||
#: 挂依据的要求和 require_citation 一样,另外按裁判的判定判档
|
||||
_CLAIMS_AXES = (("没挂依据的结论句", ("不检查", "只标出来", "计入缺口", "不予出具")),
|
||||
("证据不支持的结论句", ("不裁判", "降档出具", "不予出具")))
|
||||
|
||||
|
||||
def _claims_rank(value: Any) -> int | None:
|
||||
"""契约里 claims 的严格程度,越大越严。judge 这类还不支持的写法不比较(validate 另报)。"""
|
||||
def _claims_rank(value: Any) -> tuple[int, int] | None:
|
||||
"""契约里 claims 的严格程度:(没挂依据的, 证据不支持的),各自越大越严,任何一样变小都算放宽。
|
||||
认不出的写法不比较(出口按缺口处理,不会因此放松)。"""
|
||||
name = value.get("policy") if isinstance(value, dict) else value
|
||||
if name in (None, "", "off"):
|
||||
return 0
|
||||
if name != "require_citation":
|
||||
return 0, 0
|
||||
if name not in ("require_citation", CLAIMS_JUDGE):
|
||||
return None
|
||||
on_uncited = value.get("on_uncited") if isinstance(value, dict) else None
|
||||
return {"ignore": 1, "withhold": 3}.get(on_uncited, 2)
|
||||
uncited = {"ignore": 1, "withhold": 3}.get(on_uncited, 2)
|
||||
if name != CLAIMS_JUDGE:
|
||||
return uncited, 0
|
||||
on_unsupported = value.get("on_unsupported") if isinstance(value, dict) else None
|
||||
return uncited, 2 if on_unsupported == "withhold" else 1
|
||||
|
||||
|
||||
def forbidden_changes(before: dict[str, Any], after: dict[str, Any],
|
||||
@@ -844,7 +905,8 @@ def forbidden_changes(before: dict[str, Any], after: dict[str, Any],
|
||||
- 审批策略往宽里改(每次调用都审批 > 仅危险工具需要审批 > 全部自动放行)、删掉写明的审批策略,
|
||||
把审批写成(或新加一个)「全部自动放行」;全图默认同理
|
||||
- 取消子工作流钉住的版本
|
||||
- 报告撰写节点的 numbers、on_violation、claims 往宽里改;关掉 agent 的 cite_fields
|
||||
- 报告撰写节点的 numbers、on_violation、claims 往宽里改(claims 改成认不出的写法也算),claims 一直是
|
||||
judge 时 judge.on_unsupported 从 withhold 退回去;关掉 agent 的 cite_fields
|
||||
- 替人把沙箱代码标成取数(evidence_role: source):它在取数还是在算只有作者知道。chosen 里的
|
||||
(节点 id, 字段) 是人在 choice 里亲手选的,不算
|
||||
- 改发布级别——级别不在图里,修复接口也从不写工作流的 status,所以只可能以一个不认识的
|
||||
@@ -928,12 +990,17 @@ def forbidden_changes(before: dict[str, Any], after: dict[str, Any],
|
||||
#: 报告撰写节点三项设置从严到宽的次序:没写时运行时的缺省值(on_violation 没写时探索运行按 flag)
|
||||
_REPORT_STRICT = {"numbers": ("strict", "strict"), "on_violation": ("fail", "flag"),
|
||||
"claims": ("require_citation", "off")}
|
||||
#: claims 从宽到严:off < require_citation < judge(在挂引用之外再请模型逐句裁判,不支持的按 on_unsupported
|
||||
#: 判档)。不在这里的写法不比较(validate 报 report.claims_invalid)
|
||||
_CLAIMS_ORDER = ("off", "require_citation", CLAIMS_JUDGE)
|
||||
|
||||
|
||||
def _report_loosened(name: str, prev: dict[str, Any], cfg: dict[str, Any], old_defaults: Any,
|
||||
new_defaults: Any) -> list[str]:
|
||||
"""报告撰写节点的三项设置有没有往宽里改:原来(按节点、再按全图默认)是最严的值,改完不是了。
|
||||
claims 改成 judge 不算放宽——那是还不支持的更严的写法,validate 会报。"""
|
||||
"""报告撰写节点的三项设置有没有往宽里改:numbers、on_violation 原来(按节点、再按全图默认)是最严的值,
|
||||
改完不是了;claims 按 off < require_citation < judge 排,往下走一格也算(require_citation → judge 是收紧),
|
||||
从 require_citation、judge 改成认不出的写法也算(原来就认不出的不比较)。claims 前后都是 judge 的,
|
||||
再看 judge.on_unsupported 有没有从 withhold 退回去。"""
|
||||
def effective(config: dict[str, Any], defaults: Any, key: str, fallback: str) -> Any:
|
||||
value = config.get(key)
|
||||
if value in (None, ""):
|
||||
@@ -943,12 +1010,56 @@ def _report_loosened(name: str, prev: dict[str, Any], cfg: dict[str, Any], old_d
|
||||
out = []
|
||||
for key, (strict, fallback) in _REPORT_STRICT.items():
|
||||
was, now = effective(prev, old_defaults, key, fallback), effective(cfg, new_defaults, key, fallback)
|
||||
if was == strict and now != strict and not (key == "claims" and now == "judge"):
|
||||
out.append(f"把「{name}」的 {key} 从 {strict} 放宽成了 {now}"
|
||||
+ ("(去掉了写明的值,运行时退回缺省)" if cfg.get(key) in (None, "") else ""))
|
||||
if key == "claims":
|
||||
if was not in _CLAIMS_ORDER or (was == "off" and now not in _CLAIMS_ORDER):
|
||||
continue
|
||||
if now not in _CLAIMS_ORDER:
|
||||
out.append(f"把「{name}」的 claims 从 {was} 改成了不认识的 {now}")
|
||||
continue
|
||||
if _CLAIMS_ORDER.index(now) >= _CLAIMS_ORDER.index(was):
|
||||
continue
|
||||
elif not (was == strict and now != strict):
|
||||
continue
|
||||
out.append(f"把「{name}」的 {key} 从 {was} 放宽成了 {now}"
|
||||
+ ("(去掉了写明的值,运行时退回缺省)" if cfg.get(key) in (None, "") else ""))
|
||||
if effective(prev, old_defaults, "claims", "off") == effective(cfg, new_defaults, "claims", "off") \
|
||||
== CLAIMS_JUDGE and (why := _judge_loosened(name, prev, cfg, old_defaults, new_defaults)):
|
||||
out.append(why)
|
||||
return out
|
||||
|
||||
|
||||
def _judge_loosened(name: str, prev: dict[str, Any], cfg: dict[str, Any], old_defaults: Any,
|
||||
new_defaults: Any) -> str | None:
|
||||
"""claims 一直是 judge 时,judge.on_unsupported 从 withhold(证据不支持的结论句不予出具)退回 degrade
|
||||
(只降档)。judge 和运行时(report.report_judge)同一个取法:节点上写了就整个用节点的,没写取图级
|
||||
defaults;on_unsupported 没写、写了认不出的都按 degrade。所以补预算时把 judge 整个换成
|
||||
{max_cost_usd: …}、节点另写一份不带 withhold 的 judge 盖掉全图默认,都算放宽。"""
|
||||
def block(config: dict[str, Any], defaults: Any) -> tuple[Any, bool]:
|
||||
own = config.get("judge")
|
||||
if own not in (None, ""):
|
||||
return own, False
|
||||
return (defaults if isinstance(defaults, dict) else {}).get("judge"), True
|
||||
|
||||
def written(judge: Any) -> Any:
|
||||
return judge.get("on_unsupported") if isinstance(judge, dict) else None
|
||||
|
||||
(old_block, old_inherited), (new_block, new_inherited) = block(prev, old_defaults), block(cfg, new_defaults)
|
||||
if written(old_block) != "withhold" or (raw := written(new_block)) == "withhold":
|
||||
return None
|
||||
was = "withhold" + ("(跟随全图默认)" if old_inherited else "")
|
||||
if raw in JUDGE_ON_UNSUPPORTED:
|
||||
now = f"{raw}" + ("(跟随全图默认)" if new_inherited else "")
|
||||
elif raw not in (None, ""):
|
||||
now = f"认不出的 {raw}(运行时按 degrade)"
|
||||
elif new_inherited:
|
||||
now = "degrade(跟随全图默认,那里没写 on_unsupported,运行时退回缺省)"
|
||||
elif old_inherited:
|
||||
now = "degrade(节点上另写了 judge,整个用节点的、不再跟随全图默认,没写 on_unsupported 就退回缺省)"
|
||||
else:
|
||||
now = "degrade(去掉了写明的值,运行时退回缺省)"
|
||||
return f"把「{name}」的 judge.on_unsupported 从 {was}{'' if old_inherited else ' '}放宽成了 {now}"
|
||||
|
||||
|
||||
def _type_word(node_type: Any, type_label: Callable[[Any], str]) -> str:
|
||||
try:
|
||||
return f"「{type_label(NodeType(node_type))}」"
|
||||
|
||||
@@ -8,12 +8,15 @@ from sqlalchemy import func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.engine.schema import (
|
||||
CLAIMS_JUDGE,
|
||||
GraphNode,
|
||||
GraphSpec,
|
||||
NodeType,
|
||||
ValidationResult,
|
||||
_caliber_inputs,
|
||||
claims_problem,
|
||||
contract_needs_metrics,
|
||||
judge_problems,
|
||||
type_label,
|
||||
validate_graph,
|
||||
)
|
||||
@@ -158,7 +161,7 @@ def _lint_contract(
|
||||
if isinstance(sources, str):
|
||||
sources = [sources]
|
||||
|
||||
if not sources:
|
||||
if not sources and contract_needs_metrics(contract):
|
||||
flag("出具契约没有声明 metrics_from(指标来自哪个「口径卡」节点),叙述里的数字无从回指",
|
||||
code="contract.metrics_from_missing", node_id=node_id, hard=True, field="contract.metrics_from")
|
||||
for src in sources:
|
||||
@@ -177,7 +180,11 @@ def _lint_contract(
|
||||
"(叙述),数字回指校验不会执行", code="contract.report_from_missing", node_id=node_id, hard=True,
|
||||
field="contract.report_from")
|
||||
|
||||
if strict and not (contract.get("required") or []):
|
||||
# 引用模式下整张图没有口径卡(只引查询单元格的问数据图):没有指标可列,strict + cells 已经保证
|
||||
# 每个数点得开出处、对不上就不予出具(用户 2026-09-29 拍板)。有口径卡的照旧要列必需指标——
|
||||
# 按图里有没有口径卡判,不按契约写没写 metrics_from,免得不写 metrics_from 就绕过去
|
||||
cells_only = contract.get("report_from") not in (None, "") and not metric_nodes
|
||||
if strict and not (contract.get("required") or []) and not cells_only:
|
||||
flag(
|
||||
"受管模板的出具契约必须声明 required(必需指标),否则「不予出具」这一档永远触发不了",
|
||||
code="contract.required_missing", node_id=node_id, hard=True, field="contract.required",
|
||||
@@ -202,7 +209,7 @@ def _lint_contract(
|
||||
#
|
||||
# G1 带契约的出口核对报告撰写节点的文档,成果字段的文字只能来自报告撰写节点
|
||||
# G2 模型写的文字不绕过报告撰写节点直接流进出口(从图上追)
|
||||
# G3 报告撰写节点写明 numbers: strict、on_violation: fail、claims: require_citation
|
||||
# G3 报告撰写节点写明 numbers: strict、on_violation: fail、claims: require_citation(或 judge,带上预算)
|
||||
# G4 口径卡的输入不来自计算角色的沙箱代码
|
||||
# G5 给口径卡供数的 agent 配 output_schema 并开 cite_fields(模型调用节点不能给口径卡供数)
|
||||
#
|
||||
@@ -216,6 +223,11 @@ REPORT_POLICY = (
|
||||
("on_violation", "fail", "重写之后还违规就让节点失败,不带着问题出具"),
|
||||
("claims", "require_citation", "没挂引用的结论句计入缺口,按出具档位降档"),
|
||||
)
|
||||
#: 除了 REPORT_POLICY 里写的那个值,还认的更严的写法。claims: judge 在挂引用之外再请另一个模型按证据逐句
|
||||
#: 裁判,不支持的按 on_unsupported 判档——比 require_citation 严,但要写预算(见 judge_budget_missing)
|
||||
REPORT_POLICY_ALSO: dict[str, tuple[str, ...]] = {"claims": (CLAIMS_JUDGE,)}
|
||||
#: 不限的写法:judge.max_cost_usd 写 null。说给人听的时候照 engine/judge 设置项的说法
|
||||
JUDGE_UNLIMITED = "不设上限,费用只受句数、时长上限和每日上限约束"
|
||||
|
||||
#: 产出本身就是证据、或者由系统确定地算出来的节点。成果字段取它们的值,不算「别的节点写的字」
|
||||
_TRACEABLE = (NodeType.REPORT, NodeType.METRICS, NodeType.INPUT, NodeType.TOOL, NodeType.RETRIEVE)
|
||||
@@ -289,6 +301,21 @@ def _effective(spec: GraphSpec, node: GraphNode, key: str, default: Any = None)
|
||||
return spec.defaults.get(key, default) if value in (None, "") else value
|
||||
|
||||
|
||||
def judge_budget_missing(spec: GraphSpec, node: GraphNode) -> bool:
|
||||
"""claims 是 judge、生效的 judge 配置里却没有 max_cost_usd 这个键(G3)。
|
||||
|
||||
写 null 是显式的不限,算写了;没写的键运行时取设置里的默认——受管出具要作者自己定。judge 本身写坏了
|
||||
(不是对象、max_cost_usd 写成了别的东西)由 validate 的 report.judge_invalid 报,这里不重复。
|
||||
judge 和 claims 一样按运行时的规则取:节点上写了就整个用节点的,不和图级 defaults 拼。
|
||||
"""
|
||||
if _effective(spec, node, "claims") != CLAIMS_JUDGE:
|
||||
return False
|
||||
judge = _effective(spec, node, "judge")
|
||||
if any(field in ("judge", "judge.max_cost_usd") for field, _ in judge_problems(judge)):
|
||||
return False
|
||||
return not (isinstance(judge, dict) and "max_cost_usd" in judge)
|
||||
|
||||
|
||||
class _Graph:
|
||||
"""一张图上反复要用的查找:谁产出了某个值、最后一条消息是谁写的。"""
|
||||
|
||||
@@ -489,13 +516,20 @@ def _lint_evidence(spec: GraphSpec, flag: Callable[..., None], *, strict: bool)
|
||||
for key, want, why in REPORT_POLICY:
|
||||
value = _effective(spec, node, key)
|
||||
if key == "claims" and claims_problem(value):
|
||||
continue # 写了不支持的值(比如 judge),validate 的 report.claims_invalid 报
|
||||
if value != want:
|
||||
continue # 写了不认识的值,validate 的 report.claims_invalid 报
|
||||
also = REPORT_POLICY_ALSO.get(key, ())
|
||||
if value != want and value not in also:
|
||||
now = "现在没写" if value in (None, "") else f"现在是 {value}"
|
||||
loose.append((key, f"{key}: {want}({why};{now})"))
|
||||
more = "".join(f",或者更严的 {v}" for v in also)
|
||||
loose.append((key, f"{key}: {want}({why}{more};{now})"))
|
||||
if loose:
|
||||
flag(f"{_who(node)}要写明 {';'.join(text for _, text in loose)}。受管模板的报告撰写节点按最严的规则自查",
|
||||
code="governed.report_policy", node_id=node.id, hard=True, field=loose[0][0])
|
||||
if judge_budget_missing(spec, node):
|
||||
flag(f"{_who(node)}的结论句由模型裁判(claims: judge),却没有写每份报告的裁判预算:在 judge 里写 "
|
||||
"max_cost_usd——写一个金额(美元),或者显式写 null 表示不限(" + JUDGE_UNLIMITED + ")。"
|
||||
"受管出具不让裁判的花费悄悄按设置里的默认值走",
|
||||
code="governed.judge_budget", node_id=node.id, hard=True, field="judge.max_cost_usd")
|
||||
|
||||
# G4、G5
|
||||
feeders: dict[str, tuple[GraphNode, list[str]]] = {}
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -237,6 +237,7 @@ def _apply_contract(
|
||||
strict=bool(contract.get("strict")),
|
||||
gaps=gaps,
|
||||
unresolved=cited.get("unresolved"),
|
||||
unsupported=cited.get("unsupported"),
|
||||
uncited_claims=cited.get("uncited"),
|
||||
claims_policy=cited.get("claims_policy"),
|
||||
)
|
||||
@@ -298,7 +299,7 @@ def _check_citations(
|
||||
reports: list[dict[str, Any]], *, cells: bool = True, governed: bool = False,
|
||||
) -> dict[str, Any]:
|
||||
"""独立复核报告文档:{matched, unmatched, unresolved, gaps, doc_artifact, stats, claims_policy, uncited,
|
||||
claims, entities}。
|
||||
unsupported, claims, entities}。
|
||||
|
||||
不信报告节点自己的统计:文档按 id 从工件库取回(取回时复验哈希),目录按报告节点
|
||||
同一套参数从状态里重建(不用文档里存的那份),每个带引用的片段重新解析、重新渲染、
|
||||
@@ -313,7 +314,7 @@ def _check_citations(
|
||||
|
||||
out: dict[str, Any] = {"matched": [], "unmatched": [], "unresolved": [], "gaps": [],
|
||||
"doc_artifact": None, "stats": None, "claims_policy": None, "uncited": None,
|
||||
"claims": None, "entities": None}
|
||||
"unsupported": None, "claims": None, "entities": None}
|
||||
gaps = out["gaps"]
|
||||
spec = ctx.run.spec
|
||||
node = spec.node_map().get(report_from)
|
||||
@@ -403,7 +404,7 @@ def _check_citations(
|
||||
"span": [start, end], "eid": cite["eid"]})
|
||||
|
||||
_claims(out, contract, payload, evidence_on=ledger_enabled(ctx.run), governed=governed,
|
||||
uncited=checked.get("uncited") or [])
|
||||
uncited=checked.get("uncited") or [], doc=doc, title=node.title)
|
||||
|
||||
mine = next((r for r in reports if r["node"].id == report_from), None)
|
||||
edited = mine["edited"] if mine else []
|
||||
@@ -416,20 +417,29 @@ def _check_citations(
|
||||
|
||||
#: 没挂依据的结论句怎么处置,从松到严。认不出的写法按 degrade 算:写错一个词不能让缺口悄悄消失
|
||||
_ON_UNCITED = ("ignore", "degrade", "withhold")
|
||||
#: 证据不支持的结论句怎么处置(claims: judge),从松到严。认不出的写法同样按 degrade 算
|
||||
_ON_UNSUPPORTED = ("degrade", "withhold")
|
||||
#: 要求结论句挂依据的策略:judge 在这一点上和 require_citation 一样严,另外再按裁判的判定判档
|
||||
_CITING = ("require_citation", "judge")
|
||||
|
||||
|
||||
def _claims(out: dict[str, Any], contract: dict[str, Any], payload: dict[str, Any], *, evidence_on: bool,
|
||||
governed: bool = False, uncited: list[dict[str, Any]]) -> None:
|
||||
"""结论句策略:没挂依据的结论句按 claims_policy 交给 decide_tier 判档。
|
||||
governed: bool = False, uncited: list[dict[str, Any]], doc: dict[str, Any] | None = None,
|
||||
title: str = "") -> None:
|
||||
"""结论句策略:没挂依据的、裁判判为证据不支持的结论句按 claims_policy 交给 decide_tier 判档。
|
||||
|
||||
两处可以要求挂依据:报告节点写作时记下的 claims(节点产出里的,写作提示按它要求过),和契约自己写的
|
||||
require_citation(对象形式可以另写 on_uncited)。两处都写了取更严的那个——契约只能收紧、不能放松:
|
||||
require_citation / judge(对象形式可以另写 on_uncited)。两处都写了取更严的那个——契约只能收紧、不能放松:
|
||||
报告节点要求了,契约写 ignore 或 off 也照旧计入缺口,写 withhold 就收紧成不予出具。受管级别的正式运行
|
||||
里 ignore 一律当 degrade:没有哪道门禁查契约里的 claims,不能让它把用户要的底线拉低。
|
||||
|
||||
报告节点写的是 judge 时,判定在文档里(按哈希取回的那份:判定是模型给的,出口复算不了,只能信封存的
|
||||
文档)。正式运行在节点里判过:证据不支持的按 on_unsupported 判档(报告节点和契约取更严的),没判完的
|
||||
(裁判没跑成、触顶)把裁判摘要里的缺口照抄进来——不能判完整出具,也不当成「不支持」。探索运行按需裁判,
|
||||
这里只标注。契约要 judge、报告节点却没开的,照实记缺口:出口替不了报告节点去裁判。
|
||||
|
||||
数的是这里重新核对出来的 uncited,不信文档里记的 cites。升级前写的报告(节点产出里没有 claims 这个键,
|
||||
包括跨着升级还没跑完的运行)一律不管,契约写了也照旧记「这一版还不支持」。judge(裁判模型)是后续版本
|
||||
的事,契约写了照实记缺口。
|
||||
包括跨着升级还没跑完的运行)一律不管,契约写了也照旧记「这一版还不支持」。
|
||||
"""
|
||||
written = payload.get("claims") if isinstance(payload, dict) else None
|
||||
# 报告节点升级后才在产出里记 claims(off 也记):没有这个键就是升级前写的报告
|
||||
@@ -437,23 +447,66 @@ def _claims(out: dict[str, Any], contract: dict[str, Any], payload: dict[str, An
|
||||
declared = contract.get("claims")
|
||||
name = declared.get("policy") if isinstance(declared, dict) else declared
|
||||
asks: list[str] = []
|
||||
if upgraded and written == "require_citation":
|
||||
if upgraded and written in _CITING:
|
||||
asks.append("degrade")
|
||||
if upgraded and name == "require_citation":
|
||||
if upgraded and name in _CITING:
|
||||
wanted = declared.get("on_uncited") if isinstance(declared, dict) else None
|
||||
asks.append(wanted if wanted in _ON_UNCITED else "degrade")
|
||||
elif name not in (None, "", "off"):
|
||||
out["gaps"].append("契约声明了结论句检查(claims),这一版还不支持,结论句没有核对")
|
||||
judged = upgraded and written == "judge"
|
||||
if upgraded and name == "judge" and not judged:
|
||||
out["gaps"].append(f"契约要求结论句由模型裁判(claims: judge),报告「{title}」没有开 claims: judge,"
|
||||
"结论句没有裁判")
|
||||
if not asks:
|
||||
return
|
||||
on_uncited = max(asks, key=_ON_UNCITED.index)
|
||||
if governed and on_uncited == "ignore":
|
||||
on_uncited = "degrade"
|
||||
policy: Any = "require_citation" if on_uncited == "degrade" else \
|
||||
{"policy": "require_citation", "on_uncited": on_uncited}
|
||||
out["claims_policy"], out["uncited"] = policy, uncited
|
||||
out["claims"] = {"policy": "require_citation", "uncited_claims": len(uncited), "uncited": uncited[:_LISTED],
|
||||
**({"on_uncited": on_uncited} if on_uncited != "degrade" else {})}
|
||||
extra = {"on_uncited": on_uncited} if on_uncited != "degrade" else {}
|
||||
if not judged:
|
||||
policy: Any = "require_citation" if on_uncited == "degrade" else \
|
||||
{"policy": "require_citation", "on_uncited": on_uncited}
|
||||
out["claims_policy"], out["uncited"] = policy, uncited
|
||||
out["claims"] = {"policy": "require_citation", "uncited_claims": len(uncited), "uncited": uncited[:_LISTED],
|
||||
**extra}
|
||||
return
|
||||
|
||||
from app.engine.evidence import iter_units
|
||||
|
||||
summary = (doc or {}).get("judge")
|
||||
if not isinstance(summary, dict):
|
||||
# 报告节点开了 judge、文档里却没有裁判摘要(不该发生):当作没判,不能判完整出具
|
||||
summary = {"mode": "inline", "counts": {}, "unjudged": {}, "limits_hit": [], "complete": False,
|
||||
"gaps": ["开了 claims: judge,文档里却没有裁判结果,结论句没有裁判"]}
|
||||
asked = [summary.get("on_unsupported")]
|
||||
if name == "judge" and isinstance(declared, dict):
|
||||
asked.append(declared.get("on_unsupported"))
|
||||
on_unsupported = max((a if a in _ON_UNSUPPORTED else "degrade" for a in asked), key=_ON_UNSUPPORTED.index)
|
||||
inline = summary.get("mode") == "inline"
|
||||
flagged: dict[str, list[dict[str, Any]]] = {"unsupported": [], "partial": []}
|
||||
if inline:
|
||||
# 探索运行按需裁判的判定不在文档里(在封存之后追加的 evidence.judged 事件里),这里只数正式运行的
|
||||
markdown = str((doc or {}).get("markdown") or "")
|
||||
for _, unit in iter_units(doc or {}):
|
||||
verdict = unit.get("verdict")
|
||||
status = verdict.get("status") if isinstance(verdict, dict) else None
|
||||
if status in flagged:
|
||||
span = unit.get("span") or [0, 0]
|
||||
flagged[status].append({"unit": unit.get("id"), "span": span, "text": markdown[span[0]:span[1]],
|
||||
"rationale": verdict.get("rationale") or ""})
|
||||
out["gaps"].extend(f"报告「{title}」{gap}" for gap in summary.get("gaps") or [])
|
||||
counts = {k: int((summary.get("counts") or {}).get(k) or 0)
|
||||
for k in ("supported", "partial", "unsupported", "not_a_claim", "unjudged")}
|
||||
out["claims_policy"] = {"policy": "judge", "on_unsupported": on_unsupported, "on_uncited": on_uncited}
|
||||
out["uncited"], out["unsupported"] = uncited, flagged["unsupported"]
|
||||
out["claims"] = {
|
||||
"policy": "judge", "uncited_claims": len(uncited), "uncited": uncited[:_LISTED], **extra,
|
||||
"on_unsupported": on_unsupported, "mode": "inline" if inline else "on_demand", "counts": counts,
|
||||
"unsupported": flagged["unsupported"][:_LISTED], "partial": flagged["partial"][:_LISTED],
|
||||
"unjudged": dict(summary.get("unjudged") or {}), "limits_hit": list(summary.get("limits_hit") or []),
|
||||
"complete": bool(summary.get("complete")), "model": summary.get("model"),
|
||||
}
|
||||
|
||||
|
||||
def _declare_citations(
|
||||
@@ -497,8 +550,10 @@ def _declare_citations(
|
||||
metrics_checked=len(metrics),
|
||||
matched_numbers=len(cited["matched"]),
|
||||
matched=cited["matched"][:50],
|
||||
# on_uncited 只在不是默认的 degrade 时才有(收紧成 withhold、探索运行的契约写了 ignore)
|
||||
**({"claims": {k: cited["claims"][k] for k in ("policy", "uncited_claims", "on_uncited")
|
||||
# on_uncited 只在不是默认的 degrade 时才有(收紧成 withhold、探索运行的契约写了 ignore);
|
||||
# judge 另带不支持时怎么判档、在哪判的和各判定的句数(名单只在出具声明里)
|
||||
**({"claims": {k: cited["claims"][k] for k in ("policy", "uncited_claims", "on_uncited", "on_unsupported",
|
||||
"mode", "counts")
|
||||
if k in cited["claims"]}} if cited.get("claims") else {}),
|
||||
# 计数取复核的统计(声明里的名单最多列 _LISTED 个)
|
||||
**({"entities": {"unknown": (cited["stats"] or {}).get("unknown_entities", len(entities["unknown"])),
|
||||
|
||||
@@ -15,20 +15,32 @@
|
||||
写了个哪里都找不到的名字)只标注:不重写、不失败,受管级别的正式出具按缺口降档;表结构快照
|
||||
不全时核对不了的名字(unverified_entity)只标注。entities: off 整层不管表名、字段名。
|
||||
结论句策略 claims:off(默认)不管;require_citation 在写作提示里要求结论句挂依据(自动链接的
|
||||
名字不算依据),没挂的计入缺口、出口降档;judge(模型裁判)是后续版本的事,现在写了直接报错。
|
||||
名字不算依据),没挂的计入缺口、出口降档;judge 另外请一个模型按证据逐句判断(engine/judge.py):
|
||||
|
||||
- 正式运行在节点里判:写完、核对完之后判,判定写进文档的 unit.verdict,随文档一起封存。判定只标注、
|
||||
不改写正文;裁判没跑成、触顶了照样产出,没判的记未裁判并留下缺口,出口按 on_unsupported 判档
|
||||
- 探索运行不在节点里花这笔钱:候选句标「未裁判 · 按需」,点开哪句再判哪句(证据接口)
|
||||
- rewrite_once(默认关):证据不支持的句子连同理由交回写作者只改这几句,再判一次,最多一轮;
|
||||
改写稿冒出原稿没有的违规(只标注的可疑实体也算)就不采用
|
||||
- 裁判调用包在 @task 里,排在节点所有已有调用之后:「接着跑」时直接取 checkpoint 里的判定,
|
||||
不重复调用、不重复计费。写作调用不进 checkpoint,接着跑时会重写一稿——所以判定带着输入指纹,
|
||||
对不上这一稿的不用
|
||||
|
||||
出口契约复核(io.py)用 report_catalog 按同一套参数重建目录、独立重算,不信这里的自查。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from collections import Counter
|
||||
from typing import Any
|
||||
|
||||
from langchain_core.messages import AIMessage, BaseMessage, HumanMessage, SystemMessage
|
||||
from langgraph.func import task
|
||||
|
||||
from app.core.errors import describe_exception
|
||||
from app.core.events import EventType
|
||||
from app.db.base import SessionLocal
|
||||
from app.engine import judge as judging
|
||||
from app.engine.context import NodeContext, NodeError
|
||||
from app.engine.evidence import (
|
||||
CELLS_REASON,
|
||||
@@ -43,7 +55,17 @@ from app.engine.evidence import (
|
||||
ledger_enabled,
|
||||
)
|
||||
from app.engine.nodes.llm import _invoke_streaming, _model_spec, _report_call
|
||||
from app.engine.schema import REPORT_CLAIMS, GraphNode, GraphSpec, NodeType, _ancestors, claims_problem
|
||||
from app.engine.schema import (
|
||||
CLAIMS_JUDGE,
|
||||
CLAIMS_VALUES,
|
||||
JUDGE_ON_UNSUPPORTED,
|
||||
GraphNode,
|
||||
GraphSpec,
|
||||
NodeType,
|
||||
_ancestors,
|
||||
claims_problem,
|
||||
judge_problems,
|
||||
)
|
||||
from app.engine.state import GraphState, message_text, thinking_text
|
||||
from app.engine.toolcalls import leaked_markup, markup_warning
|
||||
from app.providers.factory import ProviderNotConfigured, get_chat_model
|
||||
@@ -52,8 +74,8 @@ ON_VIOLATION = ("fail", "flag")
|
||||
NUMBERS = ("strict", "off")
|
||||
#: 实体层:link(默认)核对表名、字段名并自动链接;off 整层不管(目录里不收表和字段)
|
||||
ENTITIES = ("link", "off")
|
||||
#: 结论句策略,和画布校验同一份(schema.REPORT_CLAIMS);judge 是后续版本的事
|
||||
CLAIMS = REPORT_CLAIMS
|
||||
#: 结论句策略,和画布校验同一份(schema.CLAIMS_VALUES):off / require_citation / judge
|
||||
CLAIMS = CLAIMS_VALUES
|
||||
#: 只标注、不让写作者重写也不判失败的违规:可疑实体(受管的正式出具按缺口降档)、表结构快照不全时
|
||||
#: 核对不了的名字(出口只标注)
|
||||
SOFT = frozenset({"unknown_entity", "unverified_entity"})
|
||||
@@ -67,6 +89,8 @@ MARKUP_NUDGE = ("你上一条回复把工具调用写成了文字。报告撰写
|
||||
"证据目录里,直接用引用标记写报告正文。")
|
||||
MARKUP_ERROR = ("模型输出了工具调用的原始标记,而报告撰写节点不调用工具——数据已经在证据目录里。"
|
||||
"换一个模型再试")
|
||||
#: 接着跑时 checkpoint 里缓存的判定最多翻几条去找这一稿的(每条都是先前某一稿的,对不上就往后翻)
|
||||
_STALE_LIMIT = 16
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
@@ -120,14 +144,26 @@ def report_entities(spec: GraphSpec, node: GraphNode) -> str:
|
||||
|
||||
|
||||
def report_claims(spec: GraphSpec, node: GraphNode) -> str:
|
||||
"""报告节点的结论句策略:off / require_citation。出口判档按它(和节点执行时同一个取值规则)。
|
||||
"""报告节点的结论句策略:off / require_citation / judge。出口判档按它(和节点执行时同一个取值规则)。
|
||||
|
||||
写错的值(包括 judge)画布校验就拦了,执行时节点也会报错,这里只把没写当成 off。
|
||||
写错的值画布校验就拦了,执行时节点也会报错,这里只把没写当成 off。
|
||||
"""
|
||||
value = _setting(spec, node, "claims", "off")
|
||||
return value if isinstance(value, str) and value in CLAIMS else "off"
|
||||
|
||||
|
||||
def report_judge(spec: GraphSpec, node: GraphNode) -> dict[str, Any]:
|
||||
"""claims: judge 的子配置:节点上的 judge(没写取图级 defaults),写了的键原样保留——上限写 null 就是
|
||||
不限,没写的键运行时取设置里的默认(judge.report_budget)。rewrite_once、on_unsupported 补上缺省
|
||||
(false、degrade):出口判档读 on_unsupported。写错的值画布校验就拦了,执行时节点也会报错。
|
||||
"""
|
||||
raw = _setting(spec, node, "judge")
|
||||
raw = raw if isinstance(raw, dict) else {}
|
||||
return {**raw, "rewrite_once": raw.get("rewrite_once") is True,
|
||||
"on_unsupported": raw.get("on_unsupported") if raw.get("on_unsupported") in JUDGE_ON_UNSUPPORTED
|
||||
else "degrade"}
|
||||
|
||||
|
||||
def report_allowance(spec: GraphSpec, node: GraphNode) -> list[Any]:
|
||||
"""报告自查放行的字面量:出口契约 report_from 指向它时,契约的 allow_numbers。
|
||||
|
||||
@@ -209,13 +245,18 @@ class _RenderedStream:
|
||||
self._ctx.emit(EventType.LLM_TOKEN, delta=tail)
|
||||
|
||||
|
||||
async def _default_on_violation(run_id: str) -> str:
|
||||
"""探索运行默认标注、照常产出;正式运行默认失败——封存的发布版不能带着裸数字出具。"""
|
||||
async def _run_class(run_id: str) -> str | None:
|
||||
"""按运行记录上的 run_class 取:子图、审批之后续跑都是另起的执行上下文,只有运行记录靠得住。"""
|
||||
from app.db.models import Run
|
||||
|
||||
async with SessionLocal() as session:
|
||||
run = await session.get(Run, run_id)
|
||||
return "fail" if run is not None and run.run_class == "formal" else "flag"
|
||||
return run.run_class if run is not None else None
|
||||
|
||||
|
||||
async def _default_on_violation(run_id: str) -> str:
|
||||
"""探索运行默认标注、照常产出;正式运行默认失败——封存的发布版不能带着裸数字出具。"""
|
||||
return "fail" if await _run_class(run_id) == "formal" else "flag"
|
||||
|
||||
|
||||
def _options(ctx: NodeContext) -> tuple[str, int]:
|
||||
@@ -225,6 +266,8 @@ def _options(ctx: NodeContext) -> tuple[str, int]:
|
||||
# 画布校验也拦这两项;执行时再守一道,报的是同一句话,免得写错的值被悄悄当成默认
|
||||
if problem := claims_problem(ctx.cfg("claims")):
|
||||
raise NodeError(ctx.node.id, problem)
|
||||
if ctx.cfg("claims") == CLAIMS_JUDGE and (problems := judge_problems(ctx.cfg("judge"))):
|
||||
raise NodeError(ctx.node.id, ";".join(message for _, message in problems))
|
||||
entities = ctx.cfg("entities", "link") or "link"
|
||||
if entities not in ENTITIES:
|
||||
raise NodeError(ctx.node.id, f"entities 只能是 {' / '.join(ENTITIES)},写的是 {entities!r}")
|
||||
@@ -248,6 +291,29 @@ def _blocking(violations: list[dict[str, Any]], numbers: str) -> list[dict[str,
|
||||
and not (numbers == "off" and v.get("code") == "uncited_number")]
|
||||
|
||||
|
||||
def _violation_key(violation: dict[str, Any]) -> tuple[str, str]:
|
||||
"""认一处违规:(code, 点名的字)。点名的字按 text、ref、message 的顺序取第一个有的。"""
|
||||
named = violation.get("text") or violation.get("ref") or violation.get("message") or ""
|
||||
return str(violation.get("code") or ""), str(named)
|
||||
|
||||
|
||||
def _new_violations(before: list[dict[str, Any]], after: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
"""after 里 before 没有的违规,按 (code, 点名的字) 逐个抵消(同一个裸数字多写一处也算新的)。
|
||||
|
||||
全部违规都比,包括只标注、不让重写的那几类(SOFT):改写稿冒出一个编造的表名,受管的正式出具就会
|
||||
因此降档;去掉一处、又冒出另一处,总数没变,冒出来的也还是新的。
|
||||
"""
|
||||
left = Counter(_violation_key(v) for v in before)
|
||||
fresh: list[dict[str, Any]] = []
|
||||
for violation in after:
|
||||
key = _violation_key(violation)
|
||||
if left[key] > 0:
|
||||
left[key] -= 1
|
||||
else:
|
||||
fresh.append(violation)
|
||||
return fresh
|
||||
|
||||
|
||||
def _named(violations: list[dict[str, Any]], limit: int = 3) -> str:
|
||||
"""违规里点名的那几个字:「45678」「m:nope」。"""
|
||||
names = [str(v.get("text") or v.get("ref") or "") for v in violations]
|
||||
@@ -269,7 +335,10 @@ def _messages(ctx: NodeContext, state: GraphState, catalog: dict[str, Any], cell
|
||||
system = ctx.render_str(ctx.cfg("system", ""), state)
|
||||
instructions = ctx.render_str(ctx.cfg("instructions", ""), state).strip() \
|
||||
or "根据下面的证据写一份简洁的报告,先总后分。"
|
||||
rules = f"{MARKER_RULES}\n{CLAIMS_RULE}" if claims == "require_citation" else MARKER_RULES
|
||||
rules = MARKER_RULES
|
||||
if claims in ("require_citation", CLAIMS_JUDGE):
|
||||
# 裁判只看写作者挂的依据:要判,就更得每句都挂上
|
||||
rules = f"{rules}\n{CLAIMS_RULE}" + (f"\n{judging.JUDGE_RULE}" if claims == CLAIMS_JUDGE else "")
|
||||
return [
|
||||
SystemMessage(content="\n\n".join(p for p in (system, ROLE) if p)),
|
||||
HumanMessage(content=(
|
||||
@@ -362,11 +431,26 @@ async def run_report(state: GraphState, ctx: NodeContext) -> dict[str, Any]:
|
||||
response, raw = await draft(messages)
|
||||
doc = compose(raw)
|
||||
|
||||
judgement: dict[str, Any] | None = None
|
||||
if claims == CLAIMS_JUDGE:
|
||||
judge_cfg = report_judge(spec, ctx.node)
|
||||
if await _run_class(ctx.run.run_id) != "formal":
|
||||
# 探索运行默认按需裁判:点开哪句才判哪句,节点里不花这笔钱
|
||||
judgement = judging.on_demand(doc, on_unsupported=judge_cfg["on_unsupported"],
|
||||
rewrite_once=judge_cfg["rewrite_once"])
|
||||
elif not (blocking and on_violation == "fail"):
|
||||
# 反正要判失败的报告不值得再花钱判(节点紧接着就报错)
|
||||
doc, response, blocking, judgement, usages = await _judge_inline(
|
||||
ctx, doc, catalog, judge_cfg, model_id=model_id, messages=[*messages, response],
|
||||
draft=draft, compose=compose, numbers=numbers, blocking=blocking)
|
||||
spent.extend(usages)
|
||||
|
||||
doc_artifact = await _store(doc, ctx)
|
||||
ctx.emit(EventType.REPORT_CHECKED, doc_artifact=doc_artifact, ok=not doc["violations"],
|
||||
stats=doc["stats"], violations=doc["violations"][:20], repairs=repairs,
|
||||
on_violation=on_violation, failed=bool(blocking) and on_violation == "fail",
|
||||
**({"claims": claims} if evidence_on else {}))
|
||||
**({"claims": claims} if evidence_on else {}),
|
||||
**({"judge": judgement} if judgement is not None else {}))
|
||||
if blocking and on_violation == "fail":
|
||||
head = ";".join(v["message"] for v in blocking[:5])
|
||||
more = f";…另有 {len(blocking) - 5} 处" if len(blocking) > 5 else ""
|
||||
@@ -388,6 +472,7 @@ async def run_report(state: GraphState, ctx: NodeContext) -> dict[str, Any]:
|
||||
"model": model_id,
|
||||
"duration_ms": int((time.perf_counter() - started) * 1000),
|
||||
**({"claims": claims} if evidence_on else {}),
|
||||
**({"judge": judgement} if judgement is not None else {}),
|
||||
}
|
||||
updates: dict[str, Any] = {"nodes": {ctx.node.id: result}, "usage": usage}
|
||||
if ctx.cfg("emit_message", True):
|
||||
@@ -396,3 +481,131 @@ async def run_report(state: GraphState, ctx: NodeContext) -> dict[str, Any]:
|
||||
if var_name:
|
||||
updates["vars"] = {var_name: text}
|
||||
return updates
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# claims: judge——正式运行在节点里裁判
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _rewrite_request(sentences: list[tuple[str, str]]) -> str:
|
||||
listed = "\n".join(f"{i}.「{text}」——{why or '证据不支持这句话'}" for i, (text, why) in enumerate(sentences, 1))
|
||||
return (
|
||||
"另一个模型按证据逐句核对了你的报告,下面这几句它认为证据不支持:\n"
|
||||
f"{listed}\n\n"
|
||||
"请只改这几句:补上证据目录里真有的依据 [[see:…]],或者把说法收窄到证据能支持的程度,或者删掉;"
|
||||
"其余部分一字不改。数字仍然只能用引用标记写。不要解释改了什么,只输出改好的整篇报告正文。"
|
||||
)
|
||||
|
||||
|
||||
def _merge(first: dict[str, Any], second: dict[str, Any], budget: judging.Budget) -> dict[str, Any]:
|
||||
"""改写前后两轮裁判合成一份:判定、句数、缺口按改写后这一稿,花费、调用、用时两轮相加。"""
|
||||
merged = dict(second)
|
||||
for key in ("cost_usd", "estimated_usd"):
|
||||
merged[key] = round(first[key] + second[key], 6)
|
||||
for key in ("calls", "duration_ms"):
|
||||
merged[key] = first[key] + second[key]
|
||||
merged["notes"] = list(dict.fromkeys([*first["notes"], *second["notes"]]))
|
||||
merged["budget"] = budget.as_dict()
|
||||
merged["model"] = second.get("model") or first.get("model")
|
||||
merged["priced"] = first.get("priced")
|
||||
return merged
|
||||
|
||||
|
||||
async def _judge_inline(ctx: NodeContext, doc: dict[str, Any], catalog: dict[str, Any], judge_cfg: dict[str, Any], *,
|
||||
model_id: str, messages: list[BaseMessage], draft: Any, compose: Any, numbers: str,
|
||||
blocking: list[dict[str, Any]]
|
||||
) -> tuple[dict[str, Any], BaseMessage, list[dict[str, Any]], dict[str, Any], list[dict[str, Any]]]:
|
||||
"""写完、核对完之后请裁判判一遍,判定写进文档。返回 (文档, 写作者最后一稿, 要拦的违规, 裁判摘要, 用量)。
|
||||
|
||||
messages 是写作的对话、最后一条是写作者这一稿。rewrite_once 时把证据不支持的句子交回去只改这几句,
|
||||
再判一次(没改的句子直接用第一次的判定),最多一轮;改写稿冒出原稿没有的违规就不采用,保留原稿。
|
||||
|
||||
摘要里的 rewrite = {units, sentences, applied, reason, changed}:
|
||||
- sentences:交回去的句子(原稿里的原文)
|
||||
- units:这几句在封存文档里的编号。没采用时封存的是原稿,就是原稿的编号;采用了封存的是改写稿,
|
||||
编号按改写稿重排,按原文认:原句原样留在改写稿里的给新编号,改掉了的是 null
|
||||
- changed:采用时改写稿里新出来的句子(原稿里没有这段原文)在封存文档里的编号;没采用是 []
|
||||
"""
|
||||
from app.engine.replay import _in_graph
|
||||
|
||||
async with SessionLocal() as session:
|
||||
settings = await judging.judge_settings(session)
|
||||
judge_spec = await judging.judge_model_spec(session, judge_cfg, settings=settings)
|
||||
budget = judging.report_budget(settings, judge_cfg)
|
||||
masks = await judging.source_masks(catalog)
|
||||
|
||||
@task
|
||||
async def judge_step(request: judging.JudgeRequest, key: str, limits: judging.Budget, known: dict[str, Any],
|
||||
first: bool) -> dict[str, Any]:
|
||||
# 裁判进 checkpoint:「接着跑」时直接取这里的判定,不再调用、不再计费(每日计数也记在这里面)。
|
||||
# 事件都在这里面发:取缓存时不会再发一遍。第二轮 limits 是剩下的钱和时间,说给人看的仍是配置的上限
|
||||
outcome = await judging.run_request(request, spec=judge_spec, budget=limits, known=known, emit=ctx.emit,
|
||||
shown=budget)
|
||||
if first and outcome.get("model") and outcome["model"] == model_id:
|
||||
outcome["same_model"] = True
|
||||
ctx.emit(EventType.LOG, level="warn", code="judge_same_model",
|
||||
message=f"裁判模型和写作模型都是「{model_id}」:等于自己审自己,模型写错的地方它多半也看不出来。"
|
||||
"到设置里选一个不同的「证据裁判模型」,或者在节点的 judge.model 里指定")
|
||||
return {**outcome, "key": key}
|
||||
|
||||
async def judge_round(round_doc: dict[str, Any], limits: judging.Budget, known: dict[str, Any],
|
||||
first: bool) -> dict[str, Any]:
|
||||
request = judging.prepare(round_doc, catalog, masked=masks)
|
||||
key = judging.request_key(request, judge_spec, limits, known)
|
||||
if _in_graph():
|
||||
# 写作不进 checkpoint,接着跑时写作者会重写一稿:缓存里先前那一稿的判定对不上指纹,往后翻
|
||||
for _ in range(_STALE_LIMIT):
|
||||
outcome = await judge_step(request, key, limits, known, first)
|
||||
if outcome.get("key") == key:
|
||||
return outcome
|
||||
return await judging.run_request(request, spec=judge_spec, budget=limits, known=known, emit=ctx.emit,
|
||||
shown=budget)
|
||||
|
||||
outcome = await judge_round(doc, budget, {}, True)
|
||||
same_model = bool(outcome.get("same_model"))
|
||||
usages = [outcome["usage"]]
|
||||
response = messages[-1]
|
||||
rewrite: dict[str, Any] | None = None
|
||||
bad = [uid for uid, v in outcome["verdicts"].items() if v["status"] == "unsupported"]
|
||||
if judge_cfg["rewrite_once"] and bad:
|
||||
texts = judging.unit_texts(doc)
|
||||
sentences = [(texts.get(uid, ""), outcome["verdicts"][uid].get("rationale") or "") for uid in bad]
|
||||
ctx.emit(EventType.LOG, level="warn", code="report_rewrite",
|
||||
message=f"裁判认为 {len(bad)} 句结论证据不支持({_named([{'text': t} for t, _ in sentences])}),"
|
||||
"已交回写作者只改这几句")
|
||||
again, raw = await draft([*messages, HumanMessage(content=_rewrite_request(sentences))])
|
||||
rewritten = compose(raw)
|
||||
worse = _blocking(rewritten["violations"], numbers)
|
||||
fresh = _new_violations(doc["violations"], rewritten["violations"])
|
||||
rewrite = {"units": bad, "sentences": [t for t, _ in sentences], "applied": False, "reason": None,
|
||||
"changed": []}
|
||||
if fresh:
|
||||
rewrite["reason"] = (f"改写稿冒出 {len(fresh)} 处原稿没有的问题({_named(fresh)}),没有采用,"
|
||||
"保留原稿和原来的判定")
|
||||
ctx.emit(EventType.LOG, level="warn", code="report_rewrite_rejected", message=rewrite["reason"])
|
||||
else:
|
||||
known = {outcome["keys"][u]: v for u, v in outcome["verdicts"].items() if v["status"] in judging.JUDGED}
|
||||
# 第二轮用第一轮剩下的钱和时间:每份报告的上限管的是整份报告
|
||||
left = judging.Budget(
|
||||
max_claims=budget.max_claims,
|
||||
max_cost_usd=None if budget.max_cost_usd is None else max(0.0, budget.max_cost_usd - outcome["cost_usd"]),
|
||||
timeout_s=None if budget.timeout_s is None else max(0.0, budget.timeout_s - outcome["duration_ms"] / 1000),
|
||||
daily_max_usd=budget.daily_max_usd)
|
||||
second = await judge_round(rewritten, left, known, False)
|
||||
usages.append(second["usage"])
|
||||
outcome = _merge(outcome, second, budget)
|
||||
doc, response, blocking = rewritten, again, worse
|
||||
# 封存的是改写稿,编号按它重排:按原文认回去,改掉了的句子在改写稿里没有编号
|
||||
now = judging.unit_texts(rewritten)
|
||||
at: dict[str, str] = {}
|
||||
for uid, text in now.items():
|
||||
at.setdefault(text, uid)
|
||||
before = set(texts.values())
|
||||
rewrite.update(applied=True, units=[at.get(texts.get(uid, "")) for uid in bad],
|
||||
changed=[uid for uid, text in now.items() if text not in before])
|
||||
summary = judging.summarize(outcome, mode="inline", on_unsupported=judge_cfg["on_unsupported"],
|
||||
rewrite_once=judge_cfg["rewrite_once"], same_model=same_model, rewrite=rewrite)
|
||||
judging.apply_judgement(doc, outcome, summary)
|
||||
return doc, response, blocking, summary, usages
|
||||
|
||||
|
||||
@@ -95,6 +95,16 @@ WARN_CODES: dict[str, tuple[str, str]] = {
|
||||
# (记成空值,不兜底成 0)。答案能用,缺口要说清是哪一种
|
||||
"agent_field_mismatch": ("agent_field_mismatch", DEGRADED),
|
||||
"agent_field_unverified": ("agent_field_unverified", DEGRADED),
|
||||
# 结论句裁判(claims: judge):触顶、没跑成的句子记为未裁判,出具不能判完整——是缺口,要说明
|
||||
"judge_limit": ("judge_limit", DEGRADED),
|
||||
"judge_failed": ("judge_failed", DEGRADED),
|
||||
# 证据不支持的句子交回写作者改过一次(和 report_repair 一样只记缺口);改写稿冒出新问题没采用的,
|
||||
# 原稿里那几句仍是证据不支持
|
||||
"report_rewrite": ("report_rewrite", DEGRADED),
|
||||
"report_rewrite_rejected": ("report_rewrite_rejected", DEGRADED),
|
||||
# 估不出裁判的金额、裁判和写作是同一个模型:配置上的提醒,答案本身没受影响
|
||||
"judge_unpriced": ("judge_unpriced", BENIGN),
|
||||
"judge_same_model": ("judge_same_model", BENIGN),
|
||||
}
|
||||
|
||||
# 这些 kind 重跑一次大概率能好:原因说得清,且不是「问题本身没问明白」。
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
import re
|
||||
from collections import defaultdict
|
||||
from enum import StrEnum
|
||||
@@ -423,31 +424,101 @@ def _check_report_sources(node: GraphNode, spec: GraphSpec, result: ValidationRe
|
||||
|
||||
|
||||
#: 报告撰写节点的结论句策略(claims)。off:结论句不参与判档(默认,保持以前的行为);
|
||||
#: require_citation:没挂引用的结论句计入缺口、按出具档位降档。judge(模型裁判)是后续版本的
|
||||
#: require_citation:没挂引用的结论句计入缺口、按出具档位降档。这两档是确定性的(发布前自动修复
|
||||
#: 给人选的也只有这两档)
|
||||
REPORT_CLAIMS = ("off", "require_citation")
|
||||
CLAIMS_LATER = "结论句裁判在后续版本支持"
|
||||
#: judge:结论句由另一个模型按证据逐句判断(engine/judge.py),子配置写在节点的 judge 里
|
||||
CLAIMS_JUDGE = "judge"
|
||||
CLAIMS_VALUES = (*REPORT_CLAIMS, CLAIMS_JUDGE)
|
||||
#: judge 子配置能写的键。三个上限写 null 表示不限;没写的键运行时取设置里的默认(设置 → judge 一组)
|
||||
JUDGE_KEYS = ("provider", "model", "max_claims", "max_cost_usd", "timeout_s", "rewrite_once", "on_unsupported")
|
||||
#: 证据不支持时正式运行怎么判档:degrade 降档(默认)/ withhold 不予出具。探索运行只标注
|
||||
JUDGE_ON_UNSUPPORTED = ("degrade", "withhold")
|
||||
|
||||
|
||||
def claims_problem(value: Any) -> str | None:
|
||||
"""claims 的取值有什么问题,没问题返回 None。校验和报告撰写节点执行时报的是同一句话。"""
|
||||
if value in (None, "") or (isinstance(value, str) and value in REPORT_CLAIMS):
|
||||
if value in (None, "") or (isinstance(value, str) and value in CLAIMS_VALUES):
|
||||
return None
|
||||
if value == "judge":
|
||||
return f"claims 写的是 judge:{CLAIMS_LATER},现在只能写 off 或 require_citation"
|
||||
return f"claims 只能是 {' / '.join(REPORT_CLAIMS)},写的是 {value!r}"
|
||||
return f"claims 只能是 {' / '.join(CLAIMS_VALUES)},写的是 {value!r}"
|
||||
|
||||
|
||||
def _positive(value: Any, *, integer: bool = False) -> bool:
|
||||
if isinstance(value, bool) or not isinstance(value, (int, float)) or not math.isfinite(value) or value <= 0:
|
||||
return False
|
||||
return not integer or int(value) == value
|
||||
|
||||
|
||||
def judge_problems(value: Any) -> list[tuple[str, str]]:
|
||||
"""claims: judge 的子配置有什么问题:[(字段, 人话)],没问题返回 []。校验和节点执行时报的是同一句话。
|
||||
|
||||
上限写 null 就是不限(和没写不一样:没写取设置里的默认)。键写错了也报:写成 max_cost 的预算会被
|
||||
悄悄当成没写、按默认值花钱。
|
||||
"""
|
||||
if value in (None, ""):
|
||||
return []
|
||||
if not isinstance(value, dict):
|
||||
return [("judge", f'judge 要写成一个对象,比如 {{"max_cost_usd": 0.05, "max_claims": 40}},写的是 {value!r}')]
|
||||
out: list[tuple[str, str]] = []
|
||||
for key, item in value.items():
|
||||
field = f"judge.{key}"
|
||||
if key not in JUDGE_KEYS:
|
||||
out.append((field, f"judge 里不认识「{key}」:能写的是 {'、'.join(JUDGE_KEYS)}"))
|
||||
elif key in ("provider", "model") and item is not None and not isinstance(item, str):
|
||||
what = "模型接入的名字" if key == "provider" else "模型 id"
|
||||
out.append((field, f"judge.{key} 要写{what}(字符串),不写或写 null 就用设置里的「证据裁判模型」,"
|
||||
f"写的是 {item!r}"))
|
||||
elif key == "max_claims" and item is not None and not _positive(item, integer=True):
|
||||
out.append((field, f"judge.max_claims 要写正整数(每份报告最多判几句),写 null 表示不限,写的是 {item!r}"))
|
||||
elif key == "max_cost_usd" and item is not None and not _positive(item):
|
||||
out.append((field, "judge.max_cost_usd 要写大于 0 的金额(美元,每份报告的裁判花费上限),"
|
||||
f"写 null 表示不限,写的是 {item!r}"))
|
||||
elif key == "timeout_s" and item is not None and not _positive(item):
|
||||
out.append((field, f"judge.timeout_s 要写大于 0 的秒数(每份报告的裁判时长上限),写 null 表示不限,"
|
||||
f"写的是 {item!r}"))
|
||||
elif key == "rewrite_once" and not isinstance(item, bool):
|
||||
out.append((field, f"judge.rewrite_once 只能是 true 或 false,写的是 {item!r}"))
|
||||
elif key == "on_unsupported" and item not in JUDGE_ON_UNSUPPORTED:
|
||||
out.append((field, f"judge.on_unsupported 只能是 {' / '.join(JUDGE_ON_UNSUPPORTED)},写的是 {item!r}"))
|
||||
return out
|
||||
|
||||
|
||||
def _check_report_claims(node: GraphNode, spec: GraphSpec, result: ValidationResult) -> None:
|
||||
"""claims 写错了(包括还不支持的 judge)挡住运行:执行时报告节点照样会报这一句,
|
||||
与其跑完取数、算完口径卡才死在写报告那一步,不如在画图时说破。
|
||||
"""claims 写错了挡住运行:执行时报告节点照样会报这一句,与其跑完取数、算完口径卡才死在写报告
|
||||
那一步,不如在画图时说破。claims 是 judge 时再查 judge 子配置的形状;裁判模型和写作模型写成
|
||||
同一个的给警告——等于自己审自己。
|
||||
|
||||
和运行时同一个取值规则:节点上没写就取图级 defaults(NodeContext.cfg)。
|
||||
"""
|
||||
value = node.config.get("claims")
|
||||
if value in (None, ""):
|
||||
value = spec.defaults.get("claims")
|
||||
def effective(key: str) -> Any:
|
||||
value = node.config.get(key)
|
||||
return spec.defaults.get(key) if value in (None, "") else value
|
||||
|
||||
value = effective("claims")
|
||||
if problem := claims_problem(value):
|
||||
result.add(problem, node_id=node.id, field="claims", code="report.claims_invalid")
|
||||
if value != CLAIMS_JUDGE:
|
||||
return
|
||||
judge = effective("judge")
|
||||
for field, message in judge_problems(judge):
|
||||
result.add(message, node_id=node.id, field=field, code="report.judge_invalid")
|
||||
judge_model = judge.get("model") if isinstance(judge, dict) else None
|
||||
if isinstance(judge_model, str) and judge_model and judge_model == effective("model"):
|
||||
result.add(f"裁判模型和写作模型都是「{judge_model}」:等于自己审自己,模型写错的地方它多半也看不出来。"
|
||||
"给 judge.model 换一个不同的模型", level="warning", node_id=node.id, field="judge.model",
|
||||
code="report.judge_same_model")
|
||||
|
||||
|
||||
def contract_needs_metrics(contract: dict[str, Any]) -> bool:
|
||||
"""出具契约要不要写 metrics_from(指标来自哪几张口径卡)。
|
||||
|
||||
旧写法按数值回指叙述,没有口径卡就无从回指,一定要写。引用模式(report_from)下,数字回指靠
|
||||
报告里的引用标记:只引查询单元格(cells)的问数据图根本没有口径卡,不该逼它写;契约里列了
|
||||
required / expected 指标的,指标总得有个来处,照样要写。运行时 io.py 也是这么判的。
|
||||
"""
|
||||
if contract.get("report_from") in (None, ""):
|
||||
return True
|
||||
return bool(contract.get("required") or contract.get("expected"))
|
||||
|
||||
|
||||
def _check_report_from(node: GraphNode, report_from: Any, spec: GraphSpec,
|
||||
@@ -542,7 +613,7 @@ def validate_graph(spec: GraphSpec) -> ValidationResult:
|
||||
# 报成 error,两档发布和运行前都挡住
|
||||
result.add("出具契约必须是一个 JSON 对象", node_id=node.id, field="contract",
|
||||
code="contract.not_object")
|
||||
elif contract and not contract.get("metrics_from"):
|
||||
elif contract and not contract.get("metrics_from") and contract_needs_metrics(contract):
|
||||
result.add("出具契约缺 metrics_from(指标来自哪个「口径卡」节点)",
|
||||
node_id=node.id, field="contract.metrics_from", code="contract.metrics_from_missing")
|
||||
if isinstance(contract, dict) and contract.get("report_from") not in (None, ""):
|
||||
|
||||
@@ -118,22 +118,21 @@ def test_published_level_warnings_get_no_fix():
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_judge_becomes_require_citation_under_governed():
|
||||
graph = traceable(report={**STRICT, "claims": "judge"})
|
||||
def test_an_invalid_claims_value_becomes_require_citation_under_governed():
|
||||
graph = traceable(report={**STRICT, "claims": "sometimes"})
|
||||
fid = "report.claims_invalid:write"
|
||||
fix = plan(graph)[fid]
|
||||
assert fix["kind"] == "auto" and fix["preview"] == {"field": "claims", "before": "judge",
|
||||
assert fix["kind"] == "auto" and fix["preview"] == {"field": "claims", "before": "sometimes",
|
||||
"after": "require_citation"}
|
||||
out = apply_fixes(graph, [fid], level="governed")
|
||||
assert out["ok"] is True and config(out["graph"], "write")["claims"] == "require_citation"
|
||||
|
||||
|
||||
def test_an_invalid_claims_value_is_a_choice_under_published():
|
||||
graph = traceable(report={"claims": "judge"})
|
||||
graph = traceable(report={"claims": "sometimes"})
|
||||
fid = "report.claims_invalid:write"
|
||||
fix = plan(graph, "published")[fid]
|
||||
assert fix["kind"] == "choice" and [o["value"] for o in fix["options"]] == ["off", "require_citation"]
|
||||
assert fix["default"] == "require_citation" # 离裁判最近的是「每句结论挂引用」,仍然要人确认
|
||||
assert apply_fixes(graph, [fid], level="published")["applied"] == []
|
||||
out = apply_fixes(graph, [fid], {fid: "off"}, level="published")
|
||||
assert out["applied"] == [fid] and config(out["graph"], "write")["claims"] == "off" and out["ok"] is True
|
||||
|
||||
@@ -0,0 +1,111 @@
|
||||
"""结论句预筛:哪些句子送裁判模型、按什么优先级,哪些直接判为连接性。
|
||||
|
||||
裁判按调用计费,一份报告里的过渡话、标题、表格里的一格、代码都不值得花钱判。预筛是确定性的
|
||||
规则(同一份文档永远得到同一个结果),而且只看文档本身:
|
||||
|
||||
- 优先级 1:写作者给这句挂了依据(行内引用或 [[see:…]])
|
||||
- 优先级 2:没挂依据,但有方向词、因果词(增长、主要来自、导致…)
|
||||
- 优先级 3:其余有数字、实体这类实质内容、又不是短句的
|
||||
- 不送:连接性的话、标题、代码、表格单元格;只有过渡词的(「原因如下:」);短句
|
||||
预算不够时按优先级截断:先判 1,再判 2,剩下的记未裁判。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from app.engine.evidence import build_catalog, compose_doc, iter_units
|
||||
from app.engine.judge import SHORT_CHARS, candidates, claim_priority
|
||||
|
||||
|
||||
def catalog():
|
||||
return build_catalog(nodes={"caliber": {
|
||||
"kind": "metric_set", "caliber": "周报口径", "caliber_version": "v2", "artifact": "f" * 64,
|
||||
"metrics": [{"id": "gmv", "name": "销售额", "value": 45678.5, "unit": "元", "decimals": 1},
|
||||
{"id": "orders", "name": "订单数", "value": 1234, "unit": "单"}]}})
|
||||
|
||||
|
||||
RAW = """## 本周概览
|
||||
|
||||
本周销售额 [[m:gmv]]。增长主要来自新客首单。[[see:m:gmv]]原因如下:
|
||||
下面看细节。订单一共是 1234 单,整体平稳。共 3 单。增长明显。
|
||||
|
||||
- 列表里订单 [[m:orders]]
|
||||
- 过渡说明
|
||||
|
||||
| 区域 | 销售额 |
|
||||
|---|---|
|
||||
| 华东 | [[m:gmv]] |
|
||||
|
||||
```sql
|
||||
SELECT 1
|
||||
```
|
||||
"""
|
||||
|
||||
|
||||
def by_text(doc):
|
||||
"""{句子原文: unit},测试按字找句子,不依赖编号。"""
|
||||
md = doc["markdown"]
|
||||
return {md[u["span"][0]:u["span"][1]].strip(): (b, u) for b, u in iter_units(doc)}
|
||||
|
||||
|
||||
def priorities(doc):
|
||||
return {text: claim_priority(u, b, doc["markdown"]) for text, (b, u) in by_text(doc).items()}
|
||||
|
||||
|
||||
def test_the_rules():
|
||||
got = priorities(compose_doc(RAW, catalog()))
|
||||
assert got["本周销售额 45,678.5元。"] == 1 # 行内引用
|
||||
assert got["增长主要来自新客首单。"] == 1 # 句末 [[see:]] 挂的依据
|
||||
assert got["列表里订单 1,234单"] == 1 # 列表项也是句子
|
||||
assert got["增长明显。"] == 2 # 没挂依据,有方向词:短也要判
|
||||
assert got["订单一共是 1234 单,整体平稳。"] == 3 # 有数字、不是短句
|
||||
assert got["原因如下:"] is None # 有「原因」但只是过渡
|
||||
assert got["共 3 单。"] is None # 短句
|
||||
assert got["下面看细节。"] is None # 连接性
|
||||
assert got["过渡说明"] is None
|
||||
assert got["本周概览"] is None # 标题
|
||||
assert got["45,678.5元"] is None # 表格单元格:一格值不是一句结论
|
||||
assert got["SELECT 1"] is None # 代码
|
||||
|
||||
|
||||
def test_it_is_deterministic():
|
||||
one, two = compose_doc(RAW, catalog()), compose_doc(RAW, catalog())
|
||||
assert priorities(one) == priorities(two)
|
||||
assert [c.unit for c in candidates(one)] == [c.unit for c in candidates(two)]
|
||||
|
||||
|
||||
def test_candidates_come_in_priority_order_then_document_order():
|
||||
doc = compose_doc(RAW, catalog())
|
||||
texts = {u["id"]: t for t, (_, u) in by_text(doc).items()}
|
||||
got = [(c.priority, texts[c.unit]) for c in candidates(doc)]
|
||||
assert got == [
|
||||
(1, "本周销售额 45,678.5元。"), (1, "增长主要来自新客首单。"), (1, "列表里订单 1,234单"),
|
||||
(2, "增长明显。"),
|
||||
(3, "订单一共是 1234 单,整体平稳。"),
|
||||
]
|
||||
|
||||
|
||||
def test_short_means_visible_characters_not_bytes():
|
||||
"""短句按看得见的字算(标点、空白不算),数字的字也算进去。"""
|
||||
assert SHORT_CHARS == 6
|
||||
doc = compose_doc("订单 1234 单。订单总数 1234 单。", catalog())
|
||||
got = priorities(doc)
|
||||
assert got["订单 1234 单。"] == 3 and got["订单总数 1234 单。"] == 3
|
||||
doc = compose_doc("共 12 单。", catalog())
|
||||
assert priorities(doc)["共 12 单。"] is None
|
||||
|
||||
|
||||
def test_an_explicit_request_overrides_the_screen_but_not_the_kind():
|
||||
"""有人点开一句、明确要求判它:预筛放掉的结论句照判(按最低优先级);标题、连接性的话不是结论,不判。"""
|
||||
doc = compose_doc(RAW, catalog())
|
||||
units = {t: u["id"] for t, (_, u) in by_text(doc).items()}
|
||||
picked = candidates(doc, units=[units["共 3 单。"], units["下面看细节。"], units["本周概览"], "u999"])
|
||||
assert [(c.unit, c.priority) for c in picked] == [(units["共 3 单。"], 3)]
|
||||
|
||||
|
||||
def test_cites_are_the_authors_own():
|
||||
"""自动链接的名字不算依据(三期 D2):只提到表名的句子不会因此排进优先级 1。"""
|
||||
ledger = [{"kind": "schema", "node_id": "fetch", "exec": 1, "artifact": "a" * 64, "source": "shop",
|
||||
"schema": None, "synced_at": None, "tables": {"orders": ["amount", "region"]}}]
|
||||
cat = build_catalog(nodes={}, ledger=ledger)
|
||||
doc = compose_doc("明细在 `orders` 表里,按区域汇总之后东区最高。", cat)
|
||||
[(text, prio)] = [(t, p) for t, p in priorities(doc).items()]
|
||||
assert prio == 2, (text, prio)
|
||||
@@ -326,3 +326,74 @@ async def test_the_report_node_and_the_exit_check_agree_when_the_status_changes_
|
||||
assert seen == [True, True], seen
|
||||
issuance = row.output["_issuance"]
|
||||
assert issuance["tier"] == "withheld" and CELLS_REASON in issuance["unresolved"][0]["message"], issuance
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 只引单元格、没有口径卡:契约不用写 metrics_from
|
||||
#
|
||||
# 问数据的典型图是 input → 取数 → 报告 → 出口,报告直接引用查过的单元格。受管级别在契约里
|
||||
# 写 cells: true 就够了(用户拍板的第 5 条),以前校验和门禁无条件要 metrics_from,
|
||||
# 这张图怎么都发不成受管,Copilot 补上契约也会被判「冒出新的问题」作废
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def cells_only(**contract):
|
||||
contract = {"report_from": "write", "strict": True, "cells": True, **contract}
|
||||
nodes = [
|
||||
node("start", "input"),
|
||||
node("fetch", "tool", tool="db_query__shop",
|
||||
args={"sql": "SELECT SUM(amount) AS gmv, COUNT(*) AS orders FROM orders"}),
|
||||
node("write", "report", instructions="写周报", numbers="strict", on_violation="fail",
|
||||
claims="require_citation"),
|
||||
node("out", "output", fields=[{"name": "周报", "value": "{{ nodes.write.text }}"}], contract=contract),
|
||||
]
|
||||
return {"nodes": nodes, "edges": [{"source": a["id"], "target": b["id"]} for a, b in zip(nodes, nodes[1:])]}
|
||||
|
||||
|
||||
def contract_codes(graph: dict, level: str = "governed") -> tuple[list[str], list[str]]:
|
||||
from app.engine.governance import lint_for_publish
|
||||
from app.engine.schema import GraphSpec, validate_graph
|
||||
|
||||
spec = GraphSpec.model_validate(graph)
|
||||
return ([i.code for i in validate_graph(spec).issues if i.level == "error"],
|
||||
[i.code for i in lint_for_publish(spec, level=level).issues if i.level == "error"])
|
||||
|
||||
|
||||
def test_a_cells_only_contract_needs_no_metrics_from():
|
||||
validate, gate = contract_codes(cells_only())
|
||||
assert validate == [], validate
|
||||
assert gate == [], gate # 受管门禁整张图一条 error 都没有
|
||||
|
||||
|
||||
def test_a_cited_contract_that_lists_required_metrics_still_needs_metrics_from():
|
||||
validate, gate = contract_codes(cells_only(required=["gmv"]))
|
||||
assert "contract.metrics_from_missing" in validate and "contract.metrics_from_missing" in gate
|
||||
validate, gate = contract_codes(cells_only(expected=["gmv"]))
|
||||
assert "contract.metrics_from_missing" in validate and "contract.metrics_from_missing" in gate
|
||||
|
||||
|
||||
def test_a_narrative_contract_still_needs_metrics_from():
|
||||
graph = cells_only(report_from=None, narrative="{{ nodes.write.text }}")
|
||||
graph["nodes"][-1]["data"]["config"]["contract"].pop("report_from")
|
||||
validate, _ = contract_codes(graph, level="published") # 受管级别不收 narrative,只看校验
|
||||
assert "contract.metrics_from_missing" in validate
|
||||
|
||||
|
||||
async def test_a_governed_formal_run_over_cells_only_issues_formally(client, monkeypatch):
|
||||
Writer(monkeypatch, text="本周销售额 [[v:Q1.r0.gmv]]。")
|
||||
row = await formal(client, cells_only(), status="governed")
|
||||
assert row.status == "succeeded", row.error
|
||||
issuance = row.output["_issuance"]
|
||||
assert issuance["mode"] == "citations" and issuance["tier"] == "formal", issuance
|
||||
assert issuance["unresolved"] == [] and issuance["gaps"] == [], issuance
|
||||
[cell] = [m for m in issuance["matched"] if m.get("cell")]
|
||||
assert cell["cell"] == "Q1.r0.gmv" and cell["token"] == "300.5"
|
||||
|
||||
|
||||
def test_a_graph_with_a_caliber_card_still_needs_required_even_without_metrics_from():
|
||||
graph = weekly(cells=True)
|
||||
contract = graph["nodes"][-1]["data"]["config"]["contract"]
|
||||
contract.pop("metrics_from"), contract.pop("required")
|
||||
validate, gate = contract_codes(graph)
|
||||
assert "contract.required_missing" in gate, gate # 不写 metrics_from 绕不过必需指标
|
||||
assert validate == [], validate
|
||||
|
||||
@@ -0,0 +1,323 @@
|
||||
"""出口契约(引用模式)按结论句裁判的判定判档(四期)。
|
||||
|
||||
- 报告节点 claims: judge 在正式运行里当场裁判,判定随文档封存。出口按文档里的判定判档:证据不支持的结论句
|
||||
按 on_unsupported 降档(degrade,默认)或不予出具(withhold);裁判没跑成、触顶了没判完的,裁判摘要里的缺口
|
||||
照抄进出具声明——不能判完整出具,也不因此不予出具
|
||||
- 判定是模型给的,出口复算不了:信按哈希取回的文档
|
||||
- judge 在挂引用这件事上和 require_citation 一样严:没挂依据的结论句照样计入缺口
|
||||
- 探索运行按需裁判:出口只标注(候选句都是「未裁判 · 按需」),不因判定改档位
|
||||
- 契约也能写 claims: judge:只能收紧(on_unsupported 取更严的);报告节点没开 judge 的,照实记缺口
|
||||
- 升级前的运行照旧:契约写了也记「这一版还不支持」
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import re
|
||||
import uuid
|
||||
|
||||
import pytest
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
from langchain_core.messages import AIMessage
|
||||
from sqlalchemy import delete, select
|
||||
|
||||
from app.db.base import SessionLocal
|
||||
from app.db.models import Run, RunEvent, Setting, Workflow, WorkflowVersion
|
||||
from app.engine import judge as judge_mod
|
||||
from app.engine import runner as runner_mod
|
||||
from app.engine.judge import JUDGE_ROLE, SPEND_KEY
|
||||
from app.engine.runner import run_manager
|
||||
from app.main import app
|
||||
from app.providers.mock_model import MockChatModel
|
||||
|
||||
JUDGE_MODEL = "claude-sonnet-5"
|
||||
METRICS = [{"id": "gmv", "name": "销售额", "unit": "元", "decimals": 1, "expression": "vars.kpi.gmv"}]
|
||||
#: 两句结论都挂了依据、一句连接性的话
|
||||
CITED = "本周销售额 [[m:gmv]]。[[see:m:gmv]]增长主要来自新客首单。[[see:m:gmv]]下面看细节。"
|
||||
#: 第二句结论没挂依据
|
||||
UNCITED = "本周销售额 [[m:gmv]]。[[see:m:gmv]]增长主要来自新客首单。下面看细节。"
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
async def engine_up():
|
||||
await run_manager.setup()
|
||||
async with SessionLocal() as session:
|
||||
await session.execute(delete(Setting).where(Setting.key.in_([SPEND_KEY, "judge", "copilot"])))
|
||||
await session.commit()
|
||||
yield
|
||||
await run_manager.shutdown()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
async def client():
|
||||
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as c:
|
||||
yield c
|
||||
|
||||
|
||||
def node(nid, ntype, **config):
|
||||
return {"id": nid, "type": ntype, "position": {"x": 0, "y": 0}, "data": {"label": nid, "config": config}}
|
||||
|
||||
|
||||
def weekly(*, report=None, contract=None) -> dict:
|
||||
nodes = [
|
||||
node("start", "input", fields=[{"name": "week", "default": "2026-W37"}]),
|
||||
node("fetch", "transform", mode="expression", expression="{'gmv': 45678.5}", assign_to="kpi"),
|
||||
node("caliber", "metrics", caliber="周报口径", caliber_version="v2", metrics=METRICS),
|
||||
node("write", "report", instructions="写周报", **{"claims": "judge", "judge": {"max_cost_usd": 0.05},
|
||||
**(report or {})}),
|
||||
node("out", "output", fields=[{"name": "周报", "value": "{{ nodes.write.text }}"}],
|
||||
contract={"report_from": "write", "metrics_from": ["caliber"], "required": ["gmv"], "strict": True,
|
||||
**(contract or {})}),
|
||||
]
|
||||
ids = [n["id"] for n in nodes]
|
||||
return {"nodes": nodes, "edges": [{"source": a, "target": b} for a, b in zip(ids, ids[1:])]}
|
||||
|
||||
|
||||
class Models:
|
||||
"""写作者回 text;裁判按句子里的字给判定(verdicts:{句中片段: 判定},没对上的是 supported)。"""
|
||||
|
||||
def __init__(self, monkeypatch, text=CITED, verdicts=None, *, fail_judge=None):
|
||||
self.judges: list[str] = []
|
||||
verdicts = verdicts or {}
|
||||
|
||||
def decide(model, messages):
|
||||
if JUDGE_ROLE in str(messages[0].content):
|
||||
request = str(messages[-1].content)
|
||||
self.judges.append(request)
|
||||
if fail_judge:
|
||||
raise RuntimeError(fail_judge)
|
||||
items = [{"unit": uid,
|
||||
"verdict": next((s for frag, s in verdicts.items() if frag in line), "supported"),
|
||||
"rationale": "摘录里有对应的指标" if "销售额" in line else "证据里没有新客维度", "used": []}
|
||||
for uid, line in re.findall(r"^\[(u\d+)\]([^\n]*)$", request, re.M)]
|
||||
return AIMessage(content=json.dumps({"items": items}, ensure_ascii=False))
|
||||
return AIMessage(content=text)
|
||||
|
||||
monkeypatch.setattr(MockChatModel, "_decide", decide)
|
||||
|
||||
async def judge_model_of(session, spec):
|
||||
return MockChatModel(model_name=JUDGE_MODEL), JUDGE_MODEL
|
||||
|
||||
monkeypatch.setattr(judge_mod, "get_chat_model", judge_model_of)
|
||||
|
||||
|
||||
async def wait(run_id: str) -> Run:
|
||||
for _ in range(400):
|
||||
await asyncio.sleep(0.05)
|
||||
async with SessionLocal() as session:
|
||||
row = await session.get(Run, run_id)
|
||||
if row.status in ("succeeded", "failed") and row.manifest_seq is not None:
|
||||
return row
|
||||
raise AssertionError(f"run {run_id} 没跑完:{row.status}")
|
||||
|
||||
|
||||
async def issued(graph, *, run_class="formal") -> dict:
|
||||
run = await run_manager.start(graph=graph, input_payload={}, run_class=run_class)
|
||||
row = await wait(run.id)
|
||||
assert row.status == "succeeded", row.error
|
||||
return row.output["_issuance"]
|
||||
|
||||
|
||||
async def governed(client, graph) -> Run:
|
||||
"""把这张图发布成受管级别,从发布版本发起正式运行。"""
|
||||
async with SessionLocal() as session:
|
||||
wf = Workflow(name=f"judge-{uuid.uuid4().hex[:6]}", graph=graph, status="governed", published_version=1)
|
||||
session.add(wf)
|
||||
await session.flush()
|
||||
session.add(WorkflowVersion(workflow_id=wf.id, version=1, graph=graph))
|
||||
await session.commit()
|
||||
wf_id = wf.id
|
||||
res = await client.post("/api/runs", json={"workflow_id": wf_id, "run_class": "formal"})
|
||||
assert res.status_code == 201, res.text
|
||||
row = await wait(res.json()["id"])
|
||||
assert row.status == "succeeded", row.error
|
||||
return row
|
||||
|
||||
|
||||
async def issuance_event(run_id: str) -> dict:
|
||||
async with SessionLocal() as session:
|
||||
[event] = (await session.execute(select(RunEvent).where(
|
||||
RunEvent.run_id == run_id, RunEvent.type == "issuance"))).scalars()
|
||||
return event.data
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 正式运行:按封存的判定判档
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_every_claim_supported_is_a_full_issuance(monkeypatch):
|
||||
Models(monkeypatch)
|
||||
issuance = await issued(weekly())
|
||||
assert issuance["tier"] == "formal" and issuance["gaps"] == [], issuance
|
||||
claims = issuance["claims"]
|
||||
assert claims["policy"] == "judge" and claims["mode"] == "inline" and claims["complete"] is True
|
||||
assert claims["counts"] == {"supported": 2, "partial": 0, "unsupported": 0, "not_a_claim": 0, "unjudged": 0}
|
||||
assert claims["on_unsupported"] == "degrade" and claims["uncited_claims"] == 0
|
||||
assert claims["unsupported"] == [] and claims["model"] == JUDGE_MODEL
|
||||
|
||||
|
||||
async def test_an_unsupported_claim_degrades_by_default(monkeypatch):
|
||||
Models(monkeypatch, verdicts={"新客": "unsupported"})
|
||||
issuance = await issued(weekly())
|
||||
assert issuance["tier"] == "degraded", "不支持按缺口降档,不是不予出具(契约是 strict 的)"
|
||||
assert issuance["gaps"] == [], "判了就不是「没核」:档位来自判定,不来自缺口"
|
||||
[flagged] = issuance["claims"]["unsupported"]
|
||||
assert flagged["text"] == "增长主要来自新客首单。" and flagged["rationale"] == "证据里没有新客维度"
|
||||
assert flagged["unit"] and len(flagged["span"]) == 2
|
||||
assert issuance["claims"]["counts"]["unsupported"] == 1
|
||||
|
||||
|
||||
async def test_withhold_withholds(monkeypatch):
|
||||
Models(monkeypatch, verdicts={"新客": "unsupported"})
|
||||
issuance = await issued(weekly(report={"judge": {"max_cost_usd": 0.05, "on_unsupported": "withhold"}}))
|
||||
assert issuance["tier"] == "withheld" and issuance["claims"]["on_unsupported"] == "withhold"
|
||||
|
||||
|
||||
async def test_partial_support_is_listed_but_does_not_cost_a_grade(monkeypatch):
|
||||
Models(monkeypatch, verdicts={"新客": "partial"})
|
||||
issuance = await issued(weekly(report={"judge": {"max_cost_usd": 0.05, "on_unsupported": "withhold"}}))
|
||||
assert issuance["tier"] == "formal"
|
||||
[item] = issuance["claims"]["partial"]
|
||||
assert item["text"] == "增长主要来自新客首单。" and issuance["claims"]["counts"]["partial"] == 1
|
||||
|
||||
|
||||
async def test_a_failed_judge_is_a_gap_not_a_withheld_report(monkeypatch):
|
||||
Models(monkeypatch, fail_judge="网关 502")
|
||||
issuance = await issued(weekly(report={"judge": {"max_cost_usd": 0.05, "on_unsupported": "withhold"}}))
|
||||
assert issuance["tier"] == "degraded", "没判成不能盖完整出具,也不能当成「不支持」不予出具"
|
||||
[gap] = [g for g in issuance["gaps"] if "裁判" in g]
|
||||
assert gap.startswith("报告「write」") and "没裁判" in gap and "网关 502" in gap
|
||||
assert issuance["claims"]["complete"] is False and issuance["claims"]["counts"]["unjudged"] == 2
|
||||
|
||||
|
||||
async def test_hitting_a_limit_is_a_gap(monkeypatch):
|
||||
Models(monkeypatch)
|
||||
issuance = await issued(weekly(report={"judge": {"max_cost_usd": 0.05, "max_claims": 1}}))
|
||||
assert issuance["tier"] == "degraded"
|
||||
[gap] = [g for g in issuance["gaps"] if "裁判" in g]
|
||||
assert "已到上限" in gap and "1 句" in gap
|
||||
assert issuance["claims"]["limits_hit"] == ["max_claims"] and issuance["claims"]["counts"]["unjudged"] == 1
|
||||
|
||||
|
||||
async def test_an_uncited_claim_still_counts_under_judge(monkeypatch):
|
||||
"""judge 在挂引用这件事上和 require_citation 一样严:从 require_citation 换成 judge 不能让没挂依据的句子过关。"""
|
||||
Models(monkeypatch, text=UNCITED)
|
||||
issuance = await issued(weekly())
|
||||
assert issuance["tier"] == "degraded" and issuance["claims"]["uncited_claims"] == 1
|
||||
assert issuance["claims"]["uncited"][0]["text"] == "增长主要来自新客首单。"
|
||||
|
||||
|
||||
async def test_the_issuance_event_carries_the_counts(monkeypatch):
|
||||
Models(monkeypatch, verdicts={"新客": "unsupported"})
|
||||
run = await run_manager.start(graph=weekly(), input_payload={}, run_class="formal")
|
||||
row = await wait(run.id)
|
||||
event = await issuance_event(row.id)
|
||||
assert event["tier"] == "degraded"
|
||||
assert event["claims"] == {"policy": "judge", "uncited_claims": 0, "on_unsupported": "degrade",
|
||||
"mode": "inline", "counts": {"supported": 1, "partial": 0, "unsupported": 1,
|
||||
"not_a_claim": 0, "unjudged": 0}}
|
||||
|
||||
|
||||
async def test_a_governed_formal_run_grades_by_the_verdicts(client, monkeypatch):
|
||||
Models(monkeypatch, verdicts={"新客": "unsupported"})
|
||||
graph = weekly(report={"numbers": "strict", "on_violation": "fail",
|
||||
"judge": {"max_cost_usd": None, "on_unsupported": "withhold"}})
|
||||
row = await governed(client, graph)
|
||||
issuance = row.output["_issuance"]
|
||||
assert issuance["tier"] == "withheld" and issuance["claims"]["unsupported"][0]["text"] == "增长主要来自新客首单。"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 探索运行:只标注
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_explore_runs_only_annotate(monkeypatch):
|
||||
models = Models(monkeypatch, verdicts={"新客": "unsupported"})
|
||||
issuance = await issued(weekly(report={"judge": {"max_cost_usd": 0.05, "on_unsupported": "withhold"}}),
|
||||
run_class="exploratory")
|
||||
assert models.judges == [], "探索运行节点里不花这笔钱"
|
||||
assert issuance["tier"] == "formal" and issuance["gaps"] == []
|
||||
claims = issuance["claims"]
|
||||
assert claims["mode"] == "on_demand" and claims["complete"] is False
|
||||
assert claims["counts"]["unjudged"] == 2 and claims["unjudged"] == {"on_demand": 2}
|
||||
assert claims["unsupported"] == []
|
||||
|
||||
|
||||
async def test_explore_runs_still_count_uncited_claims(monkeypatch):
|
||||
Models(monkeypatch, text=UNCITED)
|
||||
issuance = await issued(weekly(), run_class="exploratory")
|
||||
assert issuance["tier"] == "degraded" and issuance["claims"]["uncited_claims"] == 1
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 契约里的 claims: judge
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_the_contract_can_tighten_on_unsupported(monkeypatch):
|
||||
Models(monkeypatch, verdicts={"新客": "unsupported"})
|
||||
issuance = await issued(weekly(contract={"claims": {"policy": "judge", "on_unsupported": "withhold"}}))
|
||||
assert issuance["tier"] == "withheld" and issuance["claims"]["on_unsupported"] == "withhold"
|
||||
assert not any("不支持" in g and "这一版" in g for g in issuance["gaps"])
|
||||
|
||||
|
||||
async def test_the_contract_cannot_loosen_on_unsupported(monkeypatch):
|
||||
Models(monkeypatch, verdicts={"新客": "unsupported"})
|
||||
issuance = await issued(weekly(report={"judge": {"max_cost_usd": 0.05, "on_unsupported": "withhold"}},
|
||||
contract={"claims": {"policy": "judge", "on_unsupported": "degrade"}}))
|
||||
assert issuance["tier"] == "withheld"
|
||||
|
||||
|
||||
async def test_a_contract_asking_for_judge_on_a_report_without_it_is_a_gap(monkeypatch):
|
||||
Models(monkeypatch)
|
||||
issuance = await issued(weekly(report={"claims": "require_citation"}, contract={"claims": "judge"}))
|
||||
assert issuance["tier"] == "degraded"
|
||||
[gap] = [g for g in issuance["gaps"] if "judge" in g]
|
||||
assert "「write」" in gap and "没有开" in gap and "这一版" not in gap
|
||||
assert issuance["claims"]["policy"] == "require_citation"
|
||||
|
||||
|
||||
def test_a_run_in_flight_across_the_upgrade_keeps_the_old_gap():
|
||||
"""升级前写的报告(产出里没有 claims)碰上写了 judge 的契约:照旧记「这一版还不支持」,不判档。"""
|
||||
from app.engine.nodes.io import _claims
|
||||
|
||||
out = {"gaps": [], "claims_policy": None, "uncited": None, "claims": None, "unsupported": None}
|
||||
_claims(out, {"claims": {"policy": "judge", "on_unsupported": "withhold"}}, {"text": "x"}, evidence_on=True,
|
||||
uncited=[{"unit": "u1", "span": [0, 4], "text": "增长很快"}])
|
||||
assert out["claims_policy"] is None and out["claims"] is None and out["unsupported"] is None
|
||||
assert len(out["gaps"]) == 1 and "这一版还不支持" in out["gaps"][0]
|
||||
|
||||
|
||||
async def test_runs_from_before_the_upgrade_ignore_judge(monkeypatch):
|
||||
async def no_snapshot(session):
|
||||
return None
|
||||
|
||||
monkeypatch.setattr(runner_mod, "_agent_limits", no_snapshot)
|
||||
models = Models(monkeypatch, text=UNCITED, verdicts={"新客": "unsupported"})
|
||||
issuance = await issued(weekly())
|
||||
assert models.judges == [] and issuance["tier"] == "formal" and "claims" not in issuance
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 答案复核(review.scan)认得裁判的几条警告
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize(("code", "shown"), [
|
||||
("judge_limit", True), ("judge_failed", True), ("report_rewrite", True), ("report_rewrite_rejected", True),
|
||||
("judge_unpriced", False), ("judge_same_model", False),
|
||||
])
|
||||
def test_review_knows_the_judge_warnings(code, shown):
|
||||
"""没判完、没跑成、改写过的是缺口(要说明);估不出金额、自己审自己只是配置上的提醒,轨迹里有就够了,
|
||||
不值得为它再叫一次复核模型。"""
|
||||
from app.engine.review import DEGRADED, scan
|
||||
|
||||
signals = scan([{"type": "log", "data": {"level": "warn", "code": code, "message": f"{code} 的原话"}}],
|
||||
{"answer": "本周销售额 45,678.5元。"})
|
||||
if shown:
|
||||
[signal] = signals
|
||||
assert signal.kind == code and signal.severity == DEGRADED and signal.detail == f"{code} 的原话"
|
||||
else:
|
||||
assert signals == []
|
||||
@@ -1,4 +1,5 @@
|
||||
"""Copilot 的节点手册要教三期的写法:表名字段名、知识库原话、结论句策略,以及受管模板的报告节点怎么配。
|
||||
"""Copilot 的节点手册要教三期、四期的写法:表名字段名、知识库原话、结论句策略(含模型裁判 judge),以及受管模板的
|
||||
报告节点怎么配。
|
||||
|
||||
手册里没写的配置,Copilot 就不会用;写错的写法,报告节点会一直被打回。所以这里不只查关键词,
|
||||
还把手册里举的标记例子交给真正的解析器,确认写法和系统认的是同一套。
|
||||
@@ -7,9 +8,9 @@ from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
from app.api.copilot import NODE_REFERENCE, GenerateIn, _user_message
|
||||
from app.api.copilot import _ASSIST_RULES, NODE_REFERENCE, GenerateIn, _user_message
|
||||
from app.engine.evidence import QUOTE_MIN, SUPPORTED_KINDS, parse_markers
|
||||
from app.engine.schema import REPORT_CLAIMS
|
||||
from app.engine.schema import CLAIMS_VALUES, JUDGE_KEYS, JUDGE_ON_UNSUPPORTED
|
||||
|
||||
|
||||
def _report_section() -> str:
|
||||
@@ -22,12 +23,22 @@ def test_the_report_entry_teaches_entities_quotes_and_claims():
|
||||
for words in (
|
||||
["[[t:", "[[c:", "反引号", "可能是编造的名字"], # 表名、字段名
|
||||
["[[q:K1|", "逐字", f"至少 {QUOTE_MIN} 个字", "[[see:K1]]"], # 知识库原话
|
||||
["claims", "require_citation", "off", "judge", "后续版本"], # 结论句策略
|
||||
["claims", "require_citation", "off", "judge"], # 结论句策略
|
||||
["judge.max_cost_usd", "null", "不限", "rewrite_once", "on_unsupported", "按需"], # 模型裁判怎么配
|
||||
["entities", "link"], # 实体层开关
|
||||
):
|
||||
assert all(w in report for w in words), [w for w in words if w not in report]
|
||||
# 结论句策略的取值和校验认的是同一份
|
||||
assert all(value in report for value in REPORT_CLAIMS)
|
||||
# 结论句策略的取值和校验认的是同一份;judge 已经能用,不能再说「后续版本」「跑不起来」
|
||||
assert all(value in report for value in CLAIMS_VALUES)
|
||||
assert all(value in report for value in JUDGE_ON_UNSUPPORTED)
|
||||
assert "后续版本" not in report and "跑不起来" not in report
|
||||
# 裁判模型建议和写作模型不同:否则等于自己审自己
|
||||
assert "不同" in report and "自己审自己" in report
|
||||
|
||||
|
||||
def test_the_judge_keys_in_the_guide_are_the_ones_the_validator_knows():
|
||||
named = set(re.findall(r"judge\.(\w+)", NODE_REFERENCE))
|
||||
assert named and named <= set(JUDGE_KEYS), named - set(JUDGE_KEYS)
|
||||
|
||||
|
||||
def test_the_markers_in_the_guide_parse_as_the_system_reads_them():
|
||||
@@ -40,15 +51,26 @@ def test_the_markers_in_the_guide_parse_as_the_system_reads_them():
|
||||
|
||||
def test_governed_templates_spell_out_the_report_settings():
|
||||
ref = NODE_REFERENCE
|
||||
governed = ref[ref.index("受管"):]
|
||||
governed = ref[ref.index("要按受管级别发布的模板"):]
|
||||
for words in (['numbers: "strict"', 'on_violation: "fail"', 'claims: "require_citation"'],
|
||||
['claims: "judge"', "judge.max_cost_usd", "null"],
|
||||
["report_from", "narrative"], ["cite_fields: true"], ["evidence_role", "source"]):
|
||||
assert all(w in governed for w in words), [w for w in words if w not in governed]
|
||||
|
||||
|
||||
def test_the_publish_assist_knows_judge_is_the_stricter_one():
|
||||
"""发布前的 Copilot 兜底只许往严里改:它要知道 judge 比 require_citation 严,不能为了过门禁把 judge 改回去。"""
|
||||
assert "judge" in _ASSIST_RULES and "max_cost_usd" in _ASSIST_RULES
|
||||
# update_node 把 judge 整个换掉:补预算只写 {max_cost_usd} 会把 withhold 连同裁判模型一起丢了
|
||||
rules = " ".join(_ASSIST_RULES.split())
|
||||
assert re.search(r"on_unsupported.{0,20}withhold.{0,20}不改回\s*degrade", rules), "withhold 不许改回 degrade"
|
||||
assert re.search(r"补预算.{0,40}整份", rules), "补预算时 judge 要整份带上原来的键"
|
||||
|
||||
|
||||
def test_the_answer_branch_mentions_names_and_quotes():
|
||||
text = _user_message(GenerateIn(instruction="上周哪张表的退款最多", intent="answer"), patch=False)
|
||||
assert "[[t:" in text and "反引号" in text and "编造" in text
|
||||
assert "[[q:K1|" in text and "逐字" in text
|
||||
# 问数是探索运行:不要替人打开结论句策略
|
||||
assert not re.search(r"claims\s*[:=]\s*['\"]?require_citation", text)
|
||||
# 问数是探索运行:不要替人打开结论句策略(结论句由读的人点开时按需请模型判断)
|
||||
assert not re.search(r"claims\s*[:=]\s*['\"]?(require_citation|judge)", text)
|
||||
assert "按需" in text
|
||||
|
||||
@@ -0,0 +1,554 @@
|
||||
"""结论句裁判的内核(engine/judge.py):证据摘录、裁判调用、结构化输出兜底、预算与每日计数。
|
||||
|
||||
数字、实体是确定性的,系统自己能核对;「增长主要来自华东」这种话说得对不对,只能交给另一个模型按
|
||||
证据判断。这是概率性的判断,所以每一步都要留余地、说实话:
|
||||
|
||||
- 一份报告合并成一次调用(超过每批的句数就分批),每句给 supported / partial / unsupported / not_a_claim
|
||||
- 模型不支持结构化输出、或者给的不是合法结构时退回 JSON 解析,再失败就记「未裁判」,不编判定
|
||||
- 裁判模型关掉 thinking,max_tokens 4096
|
||||
- 预算(每份报告的句数、金额、时长,全局每日金额)每一项都能写 null 表示不限;触顶就停、已判的保留、
|
||||
没判的记未裁判并说明是哪个上限——不报错、不让报告失败。金额超了按优先级截断:先判挂了依据的,
|
||||
再判有方向词、因果词的
|
||||
- 每日计数落库,按本地日期滚动
|
||||
- 给裁判的证据摘录里没有数据源遮罩的列
|
||||
|
||||
所有模型调用都是 mock 剧本,不碰真接口。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
|
||||
import pytest
|
||||
from langchain_core.messages import AIMessage
|
||||
from sqlalchemy import delete
|
||||
|
||||
from app.core import artifact_store
|
||||
from app.core.events import EventType
|
||||
from app.db.base import SessionLocal
|
||||
from app.db.models import Setting
|
||||
from app.engine import judge as judge_mod
|
||||
from app.engine.evidence import build_catalog, compose_doc, iter_units
|
||||
from app.engine.judge import (
|
||||
BATCH_SIZE,
|
||||
CHUNK_CHARS,
|
||||
EXCERPT_COLS,
|
||||
EXCERPT_ROWS,
|
||||
JUDGE_ROLE,
|
||||
JUDGE_RULES,
|
||||
MAX_TOKENS,
|
||||
RATIONALE_MAX,
|
||||
SPEND_KEY,
|
||||
SQL_CHARS,
|
||||
Budget,
|
||||
daily_spend,
|
||||
judge_doc,
|
||||
prepare,
|
||||
source_masks,
|
||||
)
|
||||
from app.providers.factory import ModelSpec
|
||||
from app.providers.mock_model import MockChatModel
|
||||
|
||||
#: 目录里有价格的模型:金额上限对它起作用
|
||||
PRICED = "claude-sonnet-5"
|
||||
UNLIMITED = Budget(max_claims=None, max_cost_usd=None, timeout_s=None, daily_max_usd=None)
|
||||
SPEC = ModelSpec(model=PRICED)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 夹具:一张口径卡、一次查询(带手机号列)、一次知识库检索、一个运行输入
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _store(content) -> str:
|
||||
"""按 put_json 的落盘方式写进测试用的临时工件库:取回走真实的 load,复验哈希。"""
|
||||
text = artifact_store.canonical_json(content)
|
||||
artifact = artifact_store.content_hash(text)
|
||||
path = artifact_store._path_of(artifact)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(text, encoding="utf-8")
|
||||
return artifact
|
||||
|
||||
|
||||
CARD = {"kind": "metric_set", "caliber": "周报口径", "caliber_version": "v2", "metrics": [
|
||||
{"id": "gmv", "name": "销售额", "unit": "元", "value": 45678.5, "decimals": 1, "format": "thousands",
|
||||
"status": "ok", "expression": "vars.kpi.gmv", "rendered": "45,678.5元"},
|
||||
{"id": "wow", "name": "环比增幅", "unit": "%", "value": 8.7, "decimals": 1, "format": "plain", "status": "ok",
|
||||
"expression": "round((vars.kpi.gmv - vars.kpi.gmv_prev) / vars.kpi.gmv_prev * 100, 1)", "rendered": "8.7%"},
|
||||
]}
|
||||
LONG_SQL = "SELECT region, SUM(amount) AS amount, MAX(phone) AS phone FROM orders WHERE " + " AND ".join(
|
||||
f"flag_{i} = 1" for i in range(60)) + " GROUP BY region ORDER BY amount DESC"
|
||||
ROWS = [[f"区域{i}", 1000.0 - i, f"PHONE-{i}"] for i in range(30)]
|
||||
SNAP = {"columns": ["region", "amount", "phone"], "rows": ROWS, "row_count": len(ROWS), "truncated": False,
|
||||
"elapsed_ms": 3, "sql": LONG_SQL, "source": "shop"}
|
||||
HITS = {"query": "退款", "collection": "docs", "hits": [
|
||||
{"content": "退款主要集中在物流延误的订单。" * 60, "title": "售后周报", "document": "doc-1", "chunk": 0},
|
||||
{"content": "第二段原文。", "title": "售后周报", "document": "doc-1", "chunk": 1}]}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def stored():
|
||||
card = {**CARD}
|
||||
art = _store(card)
|
||||
return {"card": art, "snap": _store(SNAP), "masked": _store({**SNAP, "mask_columns": ["phone"]}),
|
||||
"hits": _store(HITS)}
|
||||
|
||||
|
||||
def make_catalog(stored, snap="snap"):
|
||||
ledger = [
|
||||
{"kind": "metric_set", "node_id": "caliber", "exec": 1, "artifact": stored["card"], "caliber": "周报口径",
|
||||
"version": "v2", "metrics": ["gmv", "wow"]},
|
||||
{"kind": "query", "node_id": "fetch", "exec": 1, "artifact": stored[snap], "via": "b" * 64,
|
||||
"call_id": "call_1", "tool": "db_query__shop", "source": "shop", "columns": SNAP["columns"],
|
||||
"rows": len(ROWS), "truncated": False},
|
||||
{"kind": "retrieval", "node_id": "kb", "exec": 1, "artifact": stored["hits"], "source": "docs", "rows": 2},
|
||||
]
|
||||
return build_catalog(nodes={"caliber": {**CARD, "artifact": stored["card"]}}, ledger=ledger,
|
||||
inputs={"week": "2026-W37"})
|
||||
|
||||
|
||||
RAW = ("## 本周概览\n\n"
|
||||
"本周销售额 [[m:gmv]]。[[see:m:gmv]]"
|
||||
"环比 [[m:wow]],增长主要来自华东。[[see:m:wow,Q1]]"
|
||||
"增长主要来自新客首单。"
|
||||
"订单一共是 1234 单,整体平稳。"
|
||||
"下面看细节。")
|
||||
|
||||
|
||||
def units_of(doc) -> dict[str, str]:
|
||||
"""{句子原文: unit id}"""
|
||||
md = doc["markdown"]
|
||||
return {md[u["span"][0]:u["span"][1]].strip(): u["id"] for _, u in iter_units(doc)}
|
||||
|
||||
|
||||
CITED, CITED2, CAUSAL, NUMBER = ("本周销售额 45,678.5元。", "环比 8.7%,增长主要来自华东。", "增长主要来自新客首单。",
|
||||
"订单一共是 1234 单,整体平稳。")
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
async def clean_settings():
|
||||
"""每日计数和裁判设置都落在设置表里:每个测试从干净的一天开始。"""
|
||||
async with SessionLocal() as session:
|
||||
await session.execute(delete(Setting).where(Setting.key.in_([SPEND_KEY, "judge", "copilot"])))
|
||||
await session.commit()
|
||||
yield
|
||||
|
||||
|
||||
class Judge:
|
||||
"""裁判模型的剧本。结构化调用和退回的纯文本调用分开记;verdicts 按 unit id 给判定,没写的都是 supported。"""
|
||||
|
||||
def __init__(self, monkeypatch, verdicts=None, *, model_id=PRICED, structured=None, plain=None,
|
||||
latency=0.0, fail=None):
|
||||
self.calls: list[dict] = []
|
||||
self.specs: list[ModelSpec] = []
|
||||
verdicts = verdicts or {}
|
||||
|
||||
def answer(units):
|
||||
return json.dumps({"items": [{"unit": u, "verdict": verdicts.get(u, "supported"),
|
||||
"rationale": f"{u} 的依据", "used": ["m:gmv"]} for u in units]},
|
||||
ensure_ascii=False)
|
||||
|
||||
def decide(model, messages):
|
||||
assert JUDGE_ROLE in str(messages[0].content), "这里只该有裁判调用"
|
||||
units = re.findall(r"^\[(u\d+)\]", str(messages[-1].content), re.M)
|
||||
kind = "structured" if model.response_format else "plain"
|
||||
self.calls.append({"kind": kind, "units": units, "messages": list(messages)})
|
||||
if fail:
|
||||
raise RuntimeError(fail)
|
||||
custom = structured if kind == "structured" else plain
|
||||
if custom is not None:
|
||||
return AIMessage(content=custom(units) if callable(custom) else custom)
|
||||
return AIMessage(content=answer(units))
|
||||
|
||||
monkeypatch.setattr(MockChatModel, "_decide", decide)
|
||||
|
||||
async def fake_model(session, spec):
|
||||
self.specs.append(spec)
|
||||
return MockChatModel(model_name=model_id, latency=latency), model_id
|
||||
|
||||
monkeypatch.setattr(judge_mod, "get_chat_model", fake_model)
|
||||
|
||||
@property
|
||||
def judged(self) -> list[list[str]]:
|
||||
return [c["units"] for c in self.calls if c["kind"] == "structured"]
|
||||
|
||||
|
||||
def status_of(outcome, doc, text):
|
||||
return outcome["verdicts"][units_of(doc)[text]]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 调用与输出
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_one_call_for_the_whole_report_and_the_verdict_shape(monkeypatch, stored):
|
||||
judge = Judge(monkeypatch, {"u3": "unsupported"})
|
||||
catalog = make_catalog(stored)
|
||||
doc = compose_doc(RAW, catalog)
|
||||
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
|
||||
assert len(judge.calls) == 1, "一份报告合并成一次调用"
|
||||
assert sorted(judge.judged[0]) == sorted(units_of(doc)[t] for t in (CITED, CITED2, CAUSAL, NUMBER))
|
||||
v = status_of(outcome, doc, CITED)
|
||||
assert v["status"] == "supported" and v["judge"] == PRICED and v["post_seal"] is False
|
||||
assert v["rationale"] and v["used"] == ["m:gmv"]
|
||||
assert outcome["counts"] == {"supported": 3, "partial": 0, "unsupported": 1, "not_a_claim": 0, "unjudged": 0}
|
||||
assert outcome["model"] == PRICED and outcome["calls"] == 1 and outcome["gaps"] == []
|
||||
assert units_of(doc)["下面看细节。"] not in outcome["verdicts"], "连接性的话不送裁判"
|
||||
|
||||
|
||||
async def test_the_judge_model_runs_without_thinking(monkeypatch, stored):
|
||||
judge = Judge(monkeypatch)
|
||||
catalog = make_catalog(stored)
|
||||
await judge_doc(compose_doc(RAW, catalog), catalog, spec=ModelSpec(model=PRICED, thinking="adaptive",
|
||||
max_tokens=64000),
|
||||
budget=Budget(max_claims=None, max_cost_usd=None, timeout_s=30, daily_max_usd=None))
|
||||
[spec] = judge.specs
|
||||
assert spec.thinking == "off" and spec.max_tokens == MAX_TOKENS == 4096
|
||||
assert spec.model == PRICED and spec.timeout and spec.timeout <= 30
|
||||
|
||||
|
||||
async def test_structured_output_failure_falls_back_to_json(monkeypatch, stored):
|
||||
"""结构化输出失败(模型不支持、给的不是 JSON):退回纯文本调用,从回复里把 JSON 抠出来。"""
|
||||
def fenced(units):
|
||||
body = {"items": [{"unit": u, "verdict": "partial", "rationale": "只对了一半", "used": []} for u in units]}
|
||||
return "判定如下:\n```json\n" + json.dumps(body, ensure_ascii=False) + "\n```"
|
||||
|
||||
judge = Judge(monkeypatch, structured="这不是 JSON", plain=fenced)
|
||||
catalog = make_catalog(stored)
|
||||
doc = compose_doc(RAW, catalog)
|
||||
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
|
||||
assert [c["kind"] for c in judge.calls] == ["structured", "plain"]
|
||||
assert {v["status"] for v in outcome["verdicts"].values()} == {"partial"}
|
||||
assert outcome["gaps"] == []
|
||||
|
||||
|
||||
async def test_an_array_written_as_json_text_is_still_read(monkeypatch, stored):
|
||||
"""有的模型走工具调用时把嵌套的数组写成一段 JSON 文本({"items": "[…]"}):照样认,不为它再花一次调用。"""
|
||||
def stringified(units):
|
||||
items = [{"unit": u, "verdict": "unsupported", "rationale": "对不上", "used": []} for u in units]
|
||||
return json.dumps({"items": json.dumps(items, ensure_ascii=False)}, ensure_ascii=False)
|
||||
|
||||
judge = Judge(monkeypatch, structured=stringified)
|
||||
catalog = make_catalog(stored)
|
||||
doc = compose_doc(RAW, catalog)
|
||||
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
|
||||
assert [c["kind"] for c in judge.calls] == ["structured"]
|
||||
assert {v["status"] for v in outcome["verdicts"].values()} == {"unsupported"}
|
||||
|
||||
|
||||
async def test_a_broken_provider_is_not_called_batch_after_batch(monkeypatch, stored):
|
||||
"""第一批就调不通(鉴权、网关挂了):后面几批不再一次次去撞,统一记未裁判。"""
|
||||
judge = Judge(monkeypatch, fail="401 鉴权失败")
|
||||
catalog = make_catalog(stored)
|
||||
raw = "".join(f"第 {i} 个区域的销售额增长来自新客。[[see:m:gmv]]" for i in range(BATCH_SIZE + 5))
|
||||
doc = compose_doc(raw, catalog)
|
||||
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
|
||||
assert len(judge.calls) == 2, "结构化一次、退回纯文本一次,第二批不再调用"
|
||||
assert outcome["unjudged"] == {"error": BATCH_SIZE + 5} and "401 鉴权失败" in outcome["gaps"][0]
|
||||
|
||||
|
||||
async def test_both_paths_failing_leaves_the_claims_unjudged(monkeypatch, stored):
|
||||
Judge(monkeypatch, structured="不是 JSON", plain="还是不是")
|
||||
catalog = make_catalog(stored)
|
||||
doc = compose_doc(RAW, catalog)
|
||||
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
|
||||
assert {(v["status"], v["reason"]) for v in outcome["verdicts"].values()} == {("unjudged", "format")}
|
||||
assert outcome["counts"]["unjudged"] == 4 and outcome["unjudged"] == {"format": 4}
|
||||
assert outcome["gaps"] and "4 句" in outcome["gaps"][0]
|
||||
|
||||
|
||||
async def test_bad_items_are_not_trusted(monkeypatch, stored):
|
||||
"""判定不认识、理由太长、编出目录里没有的依据、漏判的句子:各按各的处理,不编判定。"""
|
||||
catalog = make_catalog(stored)
|
||||
doc = compose_doc(RAW, catalog)
|
||||
ids = units_of(doc)
|
||||
|
||||
def messy(units):
|
||||
return json.dumps({"items": [
|
||||
{"unit": ids[CITED], "verdict": "supported", "rationale": "长" * 100, "used": ["m:gmv", "m:nope", "Q9"]},
|
||||
{"unit": ids[CITED2], "verdict": "maybe", "rationale": "不认识的判定", "used": []},
|
||||
{"unit": ids[CAUSAL], "verdict": "unsupported", "rationale": "没有新客维度", "used": []},
|
||||
{"unit": "u999", "verdict": "supported", "rationale": "不存在的句子", "used": []},
|
||||
]}, ensure_ascii=False)
|
||||
|
||||
Judge(monkeypatch, structured=messy)
|
||||
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
|
||||
v = outcome["verdicts"]
|
||||
assert v[ids[CITED]]["rationale"] == "长" * RATIONALE_MAX and v[ids[CITED]]["used"] == ["m:gmv"]
|
||||
assert (v[ids[CITED2]]["status"], v[ids[CITED2]]["reason"]) == ("unjudged", "format")
|
||||
assert v[ids[CAUSAL]]["status"] == "unsupported"
|
||||
assert (v[ids[NUMBER]]["status"], v[ids[NUMBER]]["reason"]) == ("unjudged", "missing")
|
||||
assert "u999" not in v
|
||||
|
||||
|
||||
async def test_a_failed_call_is_unjudged_not_an_error(monkeypatch, stored):
|
||||
Judge(monkeypatch, fail="网关超时")
|
||||
catalog = make_catalog(stored)
|
||||
doc = compose_doc(RAW, catalog)
|
||||
events: list = []
|
||||
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED,
|
||||
emit=lambda kind, **data: events.append((str(kind), data)))
|
||||
assert {v["reason"] for v in outcome["verdicts"].values()} == {"error"}
|
||||
assert "网关超时" in outcome["gaps"][0]
|
||||
ends = [d for k, d in events if k == EventType.LLM_END]
|
||||
assert ends and ends[-1]["purpose"] == "judge" and "网关超时" in ends[-1]["error"]
|
||||
assert any(k == EventType.LOG and d["code"] == "judge_failed" for k, d in events)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 证据摘录
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def request_text(judge) -> str:
|
||||
return "\n".join(str(m.content) for m in judge.calls[0]["messages"])
|
||||
|
||||
|
||||
async def test_excerpts_follow_the_rules(monkeypatch, stored):
|
||||
judge = Judge(monkeypatch)
|
||||
catalog = make_catalog(stored)
|
||||
raw = ("本周销售额 [[m:gmv]]。[[see:m:gmv]]"
|
||||
"第 25 行的区域排在后面。[[see:Q1.r25]]"
|
||||
"整体看华东领先。[[see:Q1]]"
|
||||
"退款集中在物流延误的订单。[[see:K1]]"
|
||||
"本周是 [[i:week]],增长来自新客。")
|
||||
doc = compose_doc(raw, catalog)
|
||||
await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
|
||||
text = request_text(judge)
|
||||
assert "数字已经由系统核对过,不要复算" in JUDGE_RULES and JUDGE_RULES in text
|
||||
# 指标:名称 = 值,加原式
|
||||
assert "销售额 = 45,678.5元" in text and "vars.kpi.gmv" in text
|
||||
# 查询:SQL 前 300 字
|
||||
assert LONG_SQL[:SQL_CHARS] in text and LONG_SQL[:SQL_CHARS + 20] not in text
|
||||
# 被引用的行最多 20 行:点名的第 25 行先给,整份引用的再从头补满
|
||||
rows = re.findall(r"^r(\d+):", text, re.M)
|
||||
assert len(rows) == EXCERPT_ROWS and "25" in rows and "0" in rows and "22" not in rows
|
||||
# 片段:最多 600 字
|
||||
first = HITS["hits"][0]["content"]
|
||||
assert first[:CHUNK_CHARS] in text and first[:CHUNK_CHARS + 1] not in text
|
||||
assert "week = 2026-W37" in text
|
||||
|
||||
|
||||
async def test_masked_columns_never_reach_the_judge(monkeypatch, stored):
|
||||
judge = Judge(monkeypatch)
|
||||
catalog = make_catalog(stored)
|
||||
doc = compose_doc("整体看华东领先。[[see:Q1]]", catalog)
|
||||
await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
|
||||
assert "PHONE-0" in request_text(judge), "对照:没设遮罩时原值在摘录里"
|
||||
|
||||
# 数据源现在设的遮罩(列名不分大小写)
|
||||
judge = Judge(monkeypatch)
|
||||
await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED, masked={"shop": ["Phone"]})
|
||||
text = request_text(judge)
|
||||
assert "PHONE-" not in text and "区域0" in text
|
||||
[header] = re.findall(r"^列:(.*)$", text, re.M)
|
||||
assert "phone" not in header.lower() and "region" in header, "表头里也不给遮罩的列"
|
||||
|
||||
# 查询当时记下的遮罩(快照里的 mask_columns)
|
||||
judge = Judge(monkeypatch)
|
||||
catalog = make_catalog(stored, snap="masked")
|
||||
doc = compose_doc("整体看华东领先。[[see:Q1]]", catalog)
|
||||
await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
|
||||
assert "PHONE-" not in request_text(judge)
|
||||
|
||||
|
||||
#: 宽表:前 13 列是填充,被引用的列排在第 14、15 列,最后一列是手机号
|
||||
WIDE_COLS = [f"c{i}" for i in range(13)] + ["refund_rate", "region", "phone"]
|
||||
WIDE = {"columns": WIDE_COLS, "row_count": 5, "truncated": False, "elapsed_ms": 3, "source": "shop",
|
||||
"sql": "SELECT * FROM wide_orders",
|
||||
"rows": [[i * 100 + j for j in range(13)] + [f"RATE-{i}", f"区域{i}", f"PHONE-{i}"] for i in range(5)]}
|
||||
|
||||
|
||||
def query_catalog(artifact: str, columns: list[str], *, source: str | None = "shop"):
|
||||
ledger = [{"kind": "query", "node_id": "fetch", "exec": 1, "artifact": artifact, "tool": "db_query__shop",
|
||||
**({"source": source} if source else {}), "columns": columns, "rows": 5, "truncated": False}]
|
||||
return build_catalog(nodes={}, ledger=ledger)
|
||||
|
||||
|
||||
def header_of(excerpt: str) -> tuple[list[str], str]:
|
||||
"""(摘录给出的列, 表头那一整行)"""
|
||||
[header] = re.findall(r"^列:(.*)$", excerpt, re.M)
|
||||
return [c for c in header.split(" | ") if not c.startswith("…")], header
|
||||
|
||||
|
||||
async def test_cited_columns_of_a_wide_table_always_reach_the_judge():
|
||||
"""宽表只给前几列时,句子引用的列排在后面也一定给:裁判看不到被引用的格子,就只能判不支持。"""
|
||||
catalog = query_catalog(_store(WIDE), WIDE_COLS)
|
||||
doc = compose_doc("[[v:Q1.r2.region]] 的退款率最高,达到 [[v:Q1.r2.refund_rate]]。"
|
||||
"手机尾号 [[v:Q1.r2.phone]] 的客户下单最多。", catalog)
|
||||
excerpt = prepare(doc, catalog, masked={"shop": ["phone"]}).excerpts["Q1"]
|
||||
shown, header = header_of(excerpt)
|
||||
assert "region" in shown and "refund_rate" in shown, header
|
||||
[row] = re.findall(r"^r2:(.*)$", excerpt, re.M)
|
||||
cells = dict(zip(shown, row.split(" | "), strict=True))
|
||||
assert cells["region"] == "区域2" and cells["refund_rate"] == "RATE-2"
|
||||
# 遮罩的列被引用了也不给
|
||||
assert "phone" not in header.lower() and "PHONE-" not in excerpt
|
||||
# 没遮罩的 15 列给了 EXCERPT_COLS 列,「另有」按实际没给出的列数计
|
||||
assert len(shown) == EXCERPT_COLS and header.endswith(f"…另有 {15 - EXCERPT_COLS} 列")
|
||||
|
||||
# 被引用的列比上限还多:一律保留
|
||||
cited = "".join(f"[[v:Q1.r1.{c}]]、" for c in WIDE_COLS[:-1])
|
||||
doc = compose_doc(f"这一行各列依次是 {cited}都比上周高。", catalog)
|
||||
shown, header = header_of(prepare(doc, catalog).excerpts["Q1"])
|
||||
assert set(shown) == set(WIDE_COLS[:-1]) and "另有 1 列" in header
|
||||
|
||||
|
||||
async def test_current_masks_apply_when_the_entry_does_not_name_its_source(stored):
|
||||
"""旧形状的台账条目没记数据源:按快照自己记的数据源找遮罩(和证据面板同一个口径),摘录里照样不给。"""
|
||||
catalog = query_catalog(stored["snap"], SNAP["columns"], source=None)
|
||||
doc = compose_doc("整体看华东领先。[[see:Q1]]", catalog)
|
||||
excerpt = prepare(doc, catalog, masked={"shop": ["phone"]}).excerpts["Q1"]
|
||||
assert "PHONE-" not in excerpt and "区域0" in excerpt and "数据源 shop" in excerpt
|
||||
|
||||
# 数据源现在设的遮罩也按快照里的数据源去查
|
||||
from app.db.models import DataSource
|
||||
|
||||
catalog = query_catalog(_store({**SNAP, "source": "judge_mask_src"}), SNAP["columns"], source=None)
|
||||
async with SessionLocal() as session:
|
||||
session.add(DataSource(name="judge_mask_src", kind="sqlite", database="unused.db",
|
||||
options={"mask_columns": "Phone"}))
|
||||
await session.commit()
|
||||
try:
|
||||
assert await source_masks(catalog) == {"judge_mask_src": ["Phone"]}
|
||||
finally:
|
||||
async with SessionLocal() as session:
|
||||
await session.execute(delete(DataSource).where(DataSource.name == "judge_mask_src"))
|
||||
await session.commit()
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 预算
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_claims_over_max_claims_are_unjudged(monkeypatch, stored):
|
||||
judge = Judge(monkeypatch)
|
||||
catalog = make_catalog(stored)
|
||||
doc = compose_doc(RAW, catalog)
|
||||
outcome = await judge_doc(doc, catalog, spec=SPEC,
|
||||
budget=Budget(max_claims=2, max_cost_usd=None, timeout_s=None, daily_max_usd=None))
|
||||
ids = units_of(doc)
|
||||
assert sorted(judge.judged[0]) == sorted([ids[CITED], ids[CITED2]]), "先判挂了依据的"
|
||||
for text in (CAUSAL, NUMBER):
|
||||
v = status_of(outcome, doc, text)
|
||||
assert (v["status"], v["reason"]) == ("unjudged", "max_claims") and "已到上限" in v["rationale"]
|
||||
assert outcome["limits_hit"] == ["max_claims"] and outcome["unjudged"] == {"max_claims": 2}
|
||||
assert outcome["gaps"] and "已到上限" in outcome["gaps"][0]
|
||||
|
||||
|
||||
async def test_cost_truncation_follows_priority(monkeypatch, stored):
|
||||
catalog = make_catalog(stored)
|
||||
doc = compose_doc(RAW, catalog)
|
||||
request = prepare(doc, catalog)
|
||||
by_text = {t: c for c in request.cands for t, u in units_of(doc).items() if u == c.unit}
|
||||
without_p3 = request.cost(PRICED, [by_text[CITED], by_text[CITED2], by_text[CAUSAL]])
|
||||
everything = request.cost(PRICED, request.cands)
|
||||
assert 0 < without_p3 < everything
|
||||
|
||||
judge = Judge(monkeypatch)
|
||||
budget = Budget(max_claims=None, max_cost_usd=(without_p3 + everything) / 2, timeout_s=None, daily_max_usd=None)
|
||||
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=budget)
|
||||
assert status_of(outcome, doc, NUMBER)["reason"] == "max_cost_usd"
|
||||
assert all(status_of(outcome, doc, t)["status"] == "supported" for t in (CITED, CITED2, CAUSAL))
|
||||
assert outcome["limits_hit"] == ["max_cost_usd"] and len(judge.judged[0]) == 3
|
||||
|
||||
cited_only = request.cost(PRICED, [by_text[CITED], by_text[CITED2]])
|
||||
judge = Judge(monkeypatch)
|
||||
budget = Budget(max_claims=None, max_cost_usd=cited_only * 1.001, timeout_s=None, daily_max_usd=None)
|
||||
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=budget)
|
||||
assert {status_of(outcome, doc, t)["reason"] for t in (CAUSAL, NUMBER)} == {"max_cost_usd"}
|
||||
assert {status_of(outcome, doc, t)["status"] for t in (CITED, CITED2)} == {"supported"}
|
||||
|
||||
|
||||
async def test_unlimited_never_truncates(monkeypatch, stored):
|
||||
"""每一项都写 null:句数再多也全判(按每批上限分批),金额、时长、每日都不拦。"""
|
||||
judge = Judge(monkeypatch)
|
||||
catalog = make_catalog(stored)
|
||||
raw = "".join(f"第 {i} 个区域的销售额增长来自新客。[[see:m:gmv]]" for i in range(BATCH_SIZE + 5))
|
||||
doc = compose_doc(raw, catalog)
|
||||
async with SessionLocal() as session: # 今天已经花掉很多:不限就是不限
|
||||
session.add(Setting(key=SPEND_KEY, value={"date": judge_mod._today(), "usd": 999.0, "calls": 9}))
|
||||
await session.commit()
|
||||
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
|
||||
assert [len(units) for units in judge.judged] == [BATCH_SIZE, 5]
|
||||
assert outcome["counts"]["supported"] == BATCH_SIZE + 5 and outcome["counts"]["unjudged"] == 0
|
||||
assert outcome["limits_hit"] == [] and outcome["gaps"] == []
|
||||
|
||||
|
||||
async def test_the_daily_cap_resets_the_next_day(monkeypatch, stored):
|
||||
catalog = make_catalog(stored)
|
||||
doc = compose_doc(RAW, catalog)
|
||||
cost = prepare(doc, catalog).cost(PRICED, prepare(doc, catalog).cands)
|
||||
budget = Budget(max_claims=None, max_cost_usd=None, timeout_s=None, daily_max_usd=cost * 1.05)
|
||||
day = {"today": "2026-09-28"}
|
||||
monkeypatch.setattr(judge_mod, "_today", lambda: day["today"])
|
||||
|
||||
judge = Judge(monkeypatch)
|
||||
first = await judge_doc(doc, catalog, spec=SPEC, budget=budget)
|
||||
assert first["counts"]["unjudged"] == 0
|
||||
spent = await daily_spend()
|
||||
assert spent["date"] == "2026-09-28" and spent["usd"] == pytest.approx(cost) and spent["calls"] == 1
|
||||
|
||||
second = await judge_doc(doc, catalog, spec=SPEC, budget=budget)
|
||||
assert {v["reason"] for v in second["verdicts"].values()} == {"daily_max_usd"}
|
||||
assert second["limits_hit"] == ["daily_max_usd"] and len(judge.calls) == 1, "触顶就不再调用"
|
||||
|
||||
day["today"] = "2026-09-29"
|
||||
third = await judge_doc(doc, catalog, spec=SPEC, budget=budget)
|
||||
assert third["counts"]["unjudged"] == 0 and len(judge.calls) == 2
|
||||
assert (await daily_spend())["date"] == "2026-09-29"
|
||||
|
||||
|
||||
async def test_the_time_limit_stops_a_slow_judge(monkeypatch, stored):
|
||||
Judge(monkeypatch, latency=2.0)
|
||||
catalog = make_catalog(stored)
|
||||
doc = compose_doc(RAW, catalog)
|
||||
outcome = await judge_doc(doc, catalog, spec=SPEC,
|
||||
budget=Budget(max_claims=None, max_cost_usd=None, timeout_s=0.2, daily_max_usd=None))
|
||||
assert {v["reason"] for v in outcome["verdicts"].values()} == {"timeout_s"}
|
||||
assert outcome["limits_hit"] == ["timeout_s"] and outcome["gaps"]
|
||||
|
||||
|
||||
async def test_a_model_without_a_price_ignores_money_limits(monkeypatch, stored):
|
||||
"""目录里没有价格的模型估不出金额:金额上限对它不起作用,并照实说「按令牌估不出金额」。"""
|
||||
judge = Judge(monkeypatch, model_id="house-judge-model")
|
||||
catalog = make_catalog(stored)
|
||||
doc = compose_doc(RAW, catalog)
|
||||
events: list = []
|
||||
outcome = await judge_doc(doc, catalog, spec=ModelSpec(model="house-judge-model"),
|
||||
budget=Budget(max_claims=None, max_cost_usd=1e-9, timeout_s=None, daily_max_usd=1e-9),
|
||||
emit=lambda kind, **data: events.append((str(kind), data)))
|
||||
assert len(judge.calls) == 1 and outcome["counts"]["unjudged"] == 0
|
||||
assert outcome["priced"] is False and any("按令牌估不出金额" in n for n in outcome["notes"])
|
||||
[log] = [d for k, d in events if k == EventType.LOG and d.get("code") == "judge_unpriced"]
|
||||
assert "按令牌估不出金额" in log["message"]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 复用与按需
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_known_sentences_are_not_judged_again(monkeypatch, stored):
|
||||
judge = Judge(monkeypatch, {"u2": "unsupported"})
|
||||
catalog = make_catalog(stored)
|
||||
doc = compose_doc(RAW, catalog)
|
||||
first = await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
|
||||
known = {first["keys"][u]: v for u, v in first["verdicts"].items()}
|
||||
again = await judge_doc(compose_doc(RAW, catalog), catalog, spec=SPEC, budget=UNLIMITED, known=known)
|
||||
assert len(judge.calls) == 1 and again["reused"] == 4 and again["calls"] == 0
|
||||
assert again["verdicts"] == first["verdicts"]
|
||||
|
||||
|
||||
async def test_on_demand_judges_just_the_requested_sentence_after_the_seal(monkeypatch, stored):
|
||||
judge = Judge(monkeypatch)
|
||||
catalog = make_catalog(stored)
|
||||
doc = compose_doc(RAW + "共 3 单。", catalog)
|
||||
ids = units_of(doc)
|
||||
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED, post_seal=True,
|
||||
units=[ids[CAUSAL], ids["共 3 单。"], ids["本周概览"]])
|
||||
assert sorted(judge.judged[0]) == sorted([ids[CAUSAL], ids["共 3 单。"]]), "点名要判的短句照判"
|
||||
assert all(v["post_seal"] is True for v in outcome["verdicts"].values())
|
||||
assert outcome["skipped"] == {ids["本周概览"]: "not_a_claim"}
|
||||
@@ -0,0 +1,696 @@
|
||||
"""按需裁判接口 POST /api/runs/{id}/evidence/judge(四期)。
|
||||
|
||||
探索运行不在节点里花裁判的钱:点开哪句才判哪句。
|
||||
|
||||
- 判定作为批注追加在封存之后(evidence.judged 事件,post_seal: true):封存核对只覆盖 manifest_seq 之前的
|
||||
事件,所以追加之后 verify 照旧一致;封存的报告文档一个字都不改
|
||||
- 只在探索运行里可用:正式运行的裁判在节点内完成,按需裁判返回 409;还没封存的、升级前发起的、封存链断了的
|
||||
也不判
|
||||
- 每次点击的上限、全局每日上限按设置控制;触顶不是报错,照实说「已到上限」和怎么调
|
||||
- 同一次运行里同一句判过就直接给已有的判定,不再调用、不再花钱(连点两下也一样)
|
||||
- 裁判只看得到封存范围内的证据:人为插进工件表的、封存之后才追加事件引用的快照,一行都不进摘录;
|
||||
数据源遮罩的列同样不给
|
||||
- 片段接口和证据图带上判定,封存内的和封存后追加的分得开
|
||||
|
||||
所有裁判都用 mock 模型给剧本,不调真接口。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import re
|
||||
import sqlite3
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
from langchain_core.messages import AIMessage
|
||||
from sqlalchemy import delete, select, update
|
||||
|
||||
from app.core import artifact_store
|
||||
from app.core.events import EventType
|
||||
from app.data.engine import engines
|
||||
from app.db.base import SessionLocal
|
||||
from app.db.models import DataSource, Run, RunEvent, Setting
|
||||
from app.engine import judge as judge_mod
|
||||
from app.engine import runner as runner_mod
|
||||
from app.engine.evidence import iter_segments, iter_units
|
||||
from app.engine.judge import JUDGE_ROLE, SPEND_KEY, daily_spend
|
||||
from app.engine.runner import run_manager, verify_manifest
|
||||
from app.main import app
|
||||
from app.providers.mock_model import MockChatModel
|
||||
|
||||
PRICED = "claude-sonnet-5"
|
||||
METRICS = [{"id": "gmv", "name": "销售额", "unit": "元", "decimals": 1, "expression": "vars.kpi.gmv"}]
|
||||
#: 一句挂了依据的结论、一句没挂依据的归因、一句连接性的话
|
||||
TEXT = "## 本周\n\n本周销售额 [[m:gmv]]。[[see:m:gmv]]增长主要来自新客首单。下面看细节。"
|
||||
CITED, CAUSAL, LINK = "本周销售额 45,678.5元。", "增长主要来自新客首单。", "下面看细节。"
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
async def engine_up():
|
||||
await run_manager.setup()
|
||||
async with SessionLocal() as session:
|
||||
await session.execute(delete(Setting).where(Setting.key.in_([SPEND_KEY, "judge", "copilot"])))
|
||||
await session.commit()
|
||||
yield
|
||||
await run_manager.shutdown()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
async def client():
|
||||
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as c:
|
||||
yield c
|
||||
|
||||
|
||||
def node(nid, ntype, **config):
|
||||
return {"id": nid, "type": ntype, "position": {"x": 0, "y": 0}, "data": {"label": nid, "config": config}}
|
||||
|
||||
|
||||
def chain(*nodes) -> dict:
|
||||
ids = [n["id"] for n in nodes]
|
||||
return {"nodes": list(nodes), "edges": [{"source": a, "target": b} for a, b in zip(ids, ids[1:])]}
|
||||
|
||||
|
||||
def weekly(*, gate=False, **report) -> dict:
|
||||
return chain(
|
||||
node("start", "input", fields=[{"name": "week", "default": "2026-W37"}]),
|
||||
node("fetch", "transform", mode="expression", expression="{'gmv': 45678.5}", assign_to="kpi"),
|
||||
node("caliber", "metrics", caliber="周报口径", caliber_version="v2", metrics=METRICS),
|
||||
node("write", "report", instructions="写周报", **report),
|
||||
*([node("gate", "human", mode="approve", title="确认")] if gate else []),
|
||||
node("out", "output", fields=[{"name": "周报", "value": "{{ nodes.write.text }}"}]),
|
||||
)
|
||||
|
||||
|
||||
class Models:
|
||||
"""写作者回 text;裁判按句子里的字给判定(verdicts:{句中片段: 判定},没对上的是 supported)。
|
||||
judges 记下每次交给裁判的请求原文,specs 记下每次要的裁判模型。"""
|
||||
|
||||
def __init__(self, monkeypatch, text=TEXT, verdicts=None, *, model=PRICED, fail_judge=None, delay=0.0):
|
||||
self.judges: list[str] = []
|
||||
self.specs: list = []
|
||||
verdicts = verdicts or {}
|
||||
|
||||
def decide(chat, messages):
|
||||
if JUDGE_ROLE in str(messages[0].content):
|
||||
request = str(messages[-1].content)
|
||||
self.judges.append(request)
|
||||
if fail_judge:
|
||||
raise RuntimeError(fail_judge)
|
||||
items = [{"unit": uid,
|
||||
"verdict": next((s for frag, s in verdicts.items() if frag in line), "supported"),
|
||||
"rationale": f"判的是:{line.strip()[:16]}", "used": []}
|
||||
for uid, line in re.findall(r"^\[(u\d+)\]([^\n]*)$", request, re.M)]
|
||||
return AIMessage(content=json.dumps({"items": items}, ensure_ascii=False))
|
||||
return AIMessage(content=text)
|
||||
|
||||
monkeypatch.setattr(MockChatModel, "_decide", decide)
|
||||
|
||||
async def judge_model_of(session, spec):
|
||||
self.specs.append(spec)
|
||||
if delay:
|
||||
await asyncio.sleep(delay)
|
||||
return MockChatModel(model_name=spec.model or model), spec.model or model
|
||||
|
||||
monkeypatch.setattr(judge_mod, "get_chat_model", judge_model_of)
|
||||
|
||||
|
||||
async def wait(run_id: str, statuses=("succeeded", "failed")) -> Run:
|
||||
for _ in range(400):
|
||||
await asyncio.sleep(0.05)
|
||||
async with SessionLocal() as session:
|
||||
row = await session.get(Run, run_id)
|
||||
if row.status in statuses and (row.status == "interrupted" or row.manifest_seq is not None):
|
||||
return row
|
||||
raise AssertionError(f"run {run_id} 没跑完:{row.status}")
|
||||
|
||||
|
||||
async def finish(graph, *, run_class="exploratory", statuses=("succeeded", "failed")) -> Run:
|
||||
run = await run_manager.start(graph=graph, input_payload={}, run_class=run_class)
|
||||
row = await wait(run.id, statuses)
|
||||
return row
|
||||
|
||||
|
||||
def doc_of(row: Run) -> dict:
|
||||
return artifact_store.load(row.output["_evidence"]["doc_artifact"])
|
||||
|
||||
|
||||
def unit_ids(doc: dict) -> dict[str, str]:
|
||||
md = doc["markdown"]
|
||||
return {md[u["span"][0]:u["span"][1]].strip(): u["id"] for _, u in iter_units(doc)}
|
||||
|
||||
|
||||
def unit_starting(doc: dict, prefix: str) -> str:
|
||||
return next(uid for text, uid in unit_ids(doc).items() if text.startswith(prefix))
|
||||
|
||||
|
||||
def seg_in(doc: dict, uid: str) -> str:
|
||||
return next(s["id"] for _, u in iter_units(doc) if u["id"] == uid for s in u["segments"] if s["kind"] == "text")
|
||||
|
||||
|
||||
async def judged_events(run_id: str) -> list[RunEvent]:
|
||||
async with SessionLocal() as session:
|
||||
return list((await session.execute(select(RunEvent).where(
|
||||
RunEvent.run_id == run_id, RunEvent.type == EventType.EVIDENCE_JUDGED).order_by(RunEvent.seq))).scalars())
|
||||
|
||||
|
||||
async def judge(client, run_id: str, units, **extra):
|
||||
return await client.post(f"/api/runs/{run_id}/evidence/judge", json={"units": units, **extra})
|
||||
|
||||
|
||||
async def save_setting(key: str, value) -> None:
|
||||
async with SessionLocal() as session:
|
||||
row = await session.get(Setting, key)
|
||||
if row is None:
|
||||
session.add(Setting(key=key, value=value))
|
||||
else:
|
||||
row.value = value
|
||||
await session.commit()
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 判定落在封存之后,封存核对照旧一致
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_an_on_demand_verdict_lands_after_the_seal_and_the_seal_still_holds(client, monkeypatch):
|
||||
models = Models(monkeypatch, verdicts={"新客": "unsupported"})
|
||||
row = await finish(weekly(claims="judge"))
|
||||
assert row.status == "succeeded" and models.judges == [], "探索运行节点里不判"
|
||||
doc = doc_of(row)
|
||||
ids = unit_ids(doc)
|
||||
|
||||
res = await judge(client, row.id, [ids[CAUSAL]])
|
||||
assert res.status_code == 200, res.text
|
||||
body = res.json()
|
||||
verdict = body["verdicts"][ids[CAUSAL]]
|
||||
assert verdict["status"] == "unsupported" and verdict["post_seal"] is True and verdict["judge"] == PRICED
|
||||
assert verdict["rationale"] and body["judged"] == [ids[CAUSAL]] and body["reused"] == []
|
||||
assert body["report"] == {"node_id": "write", "doc_artifact": row.output["_evidence"]["doc_artifact"]}
|
||||
assert body["post_seal"] is True and body["limited"] is False and body["calls"] == 1
|
||||
assert len(models.judges) == 1 and CAUSAL in models.judges[0] and CITED not in models.judges[0]
|
||||
|
||||
[event] = await judged_events(row.id)
|
||||
assert event.seq > row.manifest_seq and event.node_id == "write" and body["event"] == {"seq": event.seq}
|
||||
data = event.data
|
||||
assert data["post_seal"] is True and data["report"] == "write"
|
||||
assert data["doc_artifact"] == body["report"]["doc_artifact"]
|
||||
assert data["verdicts"] == {ids[CAUSAL]: verdict} and data["model"] == PRICED and data["calls"] == 1
|
||||
assert set(data["keys"]) == {ids[CAUSAL]} and data["units"] == [ids[CAUSAL]]
|
||||
|
||||
check = await verify_manifest(row.id)
|
||||
assert check["ok"] is True and check["sealed_at"] == row.manifest_seq
|
||||
assert (await client.get(f"/api/runs/{row.id}/verify")).json()["ok"] is True
|
||||
async with SessionLocal() as session:
|
||||
after = await session.get(Run, row.id)
|
||||
assert after.manifest_seq == row.manifest_seq and after.last_seq == event.seq
|
||||
assert doc_of(row) == doc, "封存的文档一个字都不改"
|
||||
csv = await client.get(f"/api/runs/{row.id}/evidence/audit", params={"format": "csv"})
|
||||
assert csv.headers["x-evidence-seal"] == "ok"
|
||||
|
||||
|
||||
async def test_any_claim_in_an_explore_report_can_be_judged(client, monkeypatch):
|
||||
"""报告节点没开 claims: judge 也能按需判:结论句都是候选。"""
|
||||
models = Models(monkeypatch)
|
||||
row = await finish(weekly())
|
||||
ids = unit_ids(doc_of(row))
|
||||
body = (await judge(client, row.id, [ids[CITED], ids[CAUSAL]])).json()
|
||||
assert {u: v["status"] for u, v in body["verdicts"].items()} == {ids[CITED]: "supported", ids[CAUSAL]: "supported"}
|
||||
assert len(models.judges) == 1, "一次点击的几句合成一次调用"
|
||||
assert "【m:gmv】" in models.judges[0], "挂的依据摘给裁判"
|
||||
|
||||
|
||||
async def test_sequence_numbers_continue_after_a_restart(client, monkeypatch):
|
||||
"""进程重启后内存里的序号从 0 起:追加的事件要接在库里最后一条后面,不能和已有的撞号。"""
|
||||
from app.core.bus import bus
|
||||
|
||||
Models(monkeypatch)
|
||||
row = await finish(weekly())
|
||||
ids = unit_ids(doc_of(row))
|
||||
bus._seq.pop(row.id, None)
|
||||
await judge(client, row.id, [ids[CAUSAL]])
|
||||
[event] = await judged_events(row.id)
|
||||
async with SessionLocal() as session:
|
||||
seqs = [s for (s,) in await session.execute(select(RunEvent.seq).where(RunEvent.run_id == row.id))]
|
||||
assert event.seq == max(seqs) and seqs.count(event.seq) == 1 and event.seq > row.manifest_seq
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 同一句不重复判
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_asking_again_returns_the_verdict_without_another_call(client, monkeypatch):
|
||||
models = Models(monkeypatch, verdicts={"新客": "partial"})
|
||||
row = await finish(weekly(claims="judge"))
|
||||
ids = unit_ids(doc_of(row))
|
||||
first = (await judge(client, row.id, [ids[CAUSAL]])).json()
|
||||
spent = (await daily_spend())["calls"]
|
||||
again = (await judge(client, row.id, [ids[CAUSAL]])).json()
|
||||
assert len(models.judges) == 1 and (await daily_spend())["calls"] == spent, "不再调用、不再计费"
|
||||
assert again["verdicts"] == first["verdicts"] and again["reused"] == [ids[CAUSAL]] and again["judged"] == []
|
||||
assert again["calls"] == 0 and again["event"] is None and again["message"] is None
|
||||
assert len(await judged_events(row.id)) == 1
|
||||
|
||||
# 判过的和没判过的一起点:只判没判过的
|
||||
both = (await judge(client, row.id, [ids[CAUSAL], ids[CITED]])).json()
|
||||
assert both["reused"] == [ids[CAUSAL]] and both["judged"] == [ids[CITED]]
|
||||
assert len(models.judges) == 2 and CAUSAL not in models.judges[1]
|
||||
|
||||
|
||||
async def test_a_double_click_is_judged_once(client, monkeypatch):
|
||||
models = Models(monkeypatch, delay=0.05)
|
||||
row = await finish(weekly(claims="judge"))
|
||||
uid = unit_ids(doc_of(row))[CAUSAL]
|
||||
one, two = await asyncio.gather(judge(client, row.id, [uid]), judge(client, row.id, [uid]))
|
||||
assert one.status_code == two.status_code == 200
|
||||
assert len(models.judges) == 1 and len(await judged_events(row.id)) == 1
|
||||
assert sorted([one.json()["reused"], two.json()["reused"]]) == [[], [uid]]
|
||||
|
||||
|
||||
async def test_a_failed_attempt_can_be_retried(client, monkeypatch):
|
||||
Models(monkeypatch, fail_judge="网关 502")
|
||||
row = await finish(weekly(claims="judge"))
|
||||
uid = unit_ids(doc_of(row))[CAUSAL]
|
||||
failed = (await judge(client, row.id, [uid])).json()
|
||||
assert failed["verdicts"][uid]["status"] == "unjudged" and failed["verdicts"][uid]["reason"] == "error"
|
||||
assert "网关 502" in failed["message"] and failed["limited"] is False
|
||||
models = Models(monkeypatch, verdicts={"新客": "unsupported"})
|
||||
retried = (await judge(client, row.id, [uid])).json()
|
||||
assert retried["verdicts"][uid]["status"] == "unsupported" and retried["reused"] == [] and len(models.judges) == 1
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 上限
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_hitting_the_click_limit_is_a_result_not_an_error(client, monkeypatch):
|
||||
models = Models(monkeypatch)
|
||||
await save_setting("judge", {"click_max_cost_usd": 1e-9})
|
||||
row = await finish(weekly(claims="judge"))
|
||||
uid = unit_ids(doc_of(row))[CAUSAL]
|
||||
res = await judge(client, row.id, [uid])
|
||||
assert res.status_code == 200
|
||||
body = res.json()
|
||||
assert body["limited"] is True and body["limits_hit"] == ["max_cost_usd"]
|
||||
assert body["unjudged"] == {"max_cost_usd": 1}
|
||||
assert body["message"].startswith("已到上限") and "这次点击" in body["message"]
|
||||
assert any("设置" in how and "不限" in how for how in body["adjust"])
|
||||
verdict = body["verdicts"][uid]
|
||||
assert verdict["status"] == "unjudged" and verdict["reason"] == "max_cost_usd" and verdict["post_seal"] is True
|
||||
assert models.judges == [] and body["calls"] == 0 and body["budget"]["max_cost_usd"] == 1e-9
|
||||
assert body["event"] is None and await judged_events(row.id) == [], "没问模型就没有判定可记"
|
||||
|
||||
|
||||
async def test_the_daily_limit_applies_to_clicks_too(client, monkeypatch):
|
||||
models = Models(monkeypatch)
|
||||
today = (await daily_spend())["date"]
|
||||
await save_setting(SPEND_KEY, {"date": today, "usd": 2.0, "calls": 9, "unpriced_calls": 0})
|
||||
row = await finish(weekly(claims="judge"))
|
||||
body = (await judge(client, row.id, [unit_ids(doc_of(row))[CAUSAL]])).json()
|
||||
assert body["limits_hit"] == ["daily_max_usd"] and "每日" in body["message"] and models.judges == []
|
||||
assert body["spend"]["usd"] == 2.0 and body["spend"]["daily_max_usd"] == 2.0
|
||||
|
||||
|
||||
async def test_unlimited_click_budget_judges(client, monkeypatch):
|
||||
models = Models(monkeypatch)
|
||||
await save_setting("judge", {"click_max_cost_usd": None, "daily_max_usd": None})
|
||||
row = await finish(weekly(claims="judge"))
|
||||
body = (await judge(client, row.id, [unit_ids(doc_of(row))[CAUSAL]])).json()
|
||||
assert body["limited"] is False and len(models.judges) == 1
|
||||
assert body["budget"]["max_cost_usd"] is None and body["budget"]["daily_max_usd"] is None
|
||||
assert body["spend"]["calls"] == 1 and body["spend"]["daily_max_usd"] is None
|
||||
|
||||
|
||||
async def test_the_sentence_limit_keeps_what_was_judged(client, monkeypatch):
|
||||
models = Models(monkeypatch)
|
||||
row = await finish(weekly(claims="judge", judge={"max_claims": 1}))
|
||||
ids = unit_ids(doc_of(row))
|
||||
body = (await judge(client, row.id, [ids[CITED], ids[CAUSAL]])).json()
|
||||
statuses = {u: v["status"] for u, v in body["verdicts"].items()}
|
||||
assert statuses == {ids[CITED]: "supported", ids[CAUSAL]: "unjudged"} and len(models.judges) == 1
|
||||
assert body["limits_hit"] == ["max_claims"] and body["verdicts"][ids[CAUSAL]]["reason"] == "max_claims"
|
||||
[event] = await judged_events(row.id)
|
||||
assert set(event.data["verdicts"]) == {ids[CITED], ids[CAUSAL]}, "问过模型的这一次连同没判的一起记下"
|
||||
# 没判的那句还能再点
|
||||
again = (await judge(client, row.id, [ids[CAUSAL]])).json()
|
||||
assert again["judged"] == [ids[CAUSAL]] and again["reused"] == []
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 用哪个模型
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_the_judge_model_follows_node_then_settings(client, monkeypatch):
|
||||
models = Models(monkeypatch)
|
||||
await save_setting("judge", {"model": "judge-from-settings"})
|
||||
row = await finish(weekly(claims="judge"))
|
||||
await judge(client, row.id, [unit_ids(doc_of(row))[CAUSAL]])
|
||||
[spec] = models.specs
|
||||
assert spec.model == "judge-from-settings" and spec.thinking == "off" and spec.max_tokens == 4096
|
||||
|
||||
row = await finish(weekly(claims="judge", judge={"model": "judge-from-node"}))
|
||||
body = (await judge(client, row.id, [unit_ids(doc_of(row))[CAUSAL]])).json()
|
||||
assert models.specs[-1].model == "judge-from-node" and body["model"] == "judge-from-node"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 什么时候不判
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_formal_runs_are_judged_inside_the_node_not_on_demand(client, monkeypatch):
|
||||
models = Models(monkeypatch)
|
||||
row = await finish(weekly(claims="judge", judge={"max_cost_usd": 0.05}), run_class="formal")
|
||||
assert row.status == "succeeded", row.error
|
||||
calls = len(models.judges)
|
||||
res = await judge(client, row.id, [unit_ids(doc_of(row))[CAUSAL]])
|
||||
assert res.status_code == 409 and res.json()["code"] == "evidence_judge_formal"
|
||||
assert "正式运行的裁判在节点内完成" in res.json()["detail"]
|
||||
assert len(models.judges) == calls and await judged_events(row.id) == []
|
||||
|
||||
|
||||
async def test_a_run_paused_at_an_approval_is_not_judged(client, monkeypatch):
|
||||
models = Models(monkeypatch)
|
||||
row = await finish(weekly(claims="judge", gate=True), statuses=("interrupted",))
|
||||
res = await judge(client, row.id, ["u1"])
|
||||
assert res.status_code == 409 and res.json()["code"] == "evidence_judge_unsealed"
|
||||
assert models.judges == [] and await judged_events(row.id) == []
|
||||
|
||||
|
||||
async def test_runs_from_before_the_upgrade_are_not_judged(client, monkeypatch):
|
||||
async def no_snapshot(session):
|
||||
return None
|
||||
|
||||
monkeypatch.setattr(runner_mod, "_agent_limits", no_snapshot)
|
||||
models = Models(monkeypatch)
|
||||
row = await finish(weekly())
|
||||
res = await judge(client, row.id, [unit_ids(doc_of(row))[CAUSAL]])
|
||||
assert res.status_code == 409 and res.json()["code"] == "evidence_judge_legacy"
|
||||
assert models.judges == [] and await judged_events(row.id) == []
|
||||
|
||||
|
||||
async def test_a_broken_seal_is_not_judged(client, monkeypatch):
|
||||
models = Models(monkeypatch)
|
||||
row = await finish(weekly(claims="judge"))
|
||||
async with SessionLocal() as session:
|
||||
event = (await session.execute(select(RunEvent).where(
|
||||
RunEvent.run_id == row.id, RunEvent.type == "node.finished", RunEvent.node_id == "fetch"))).scalar_one()
|
||||
event.data = {**event.data, "tampered": True}
|
||||
await session.commit()
|
||||
res = await judge(client, row.id, [unit_ids(doc_of(row))[CAUSAL]])
|
||||
assert res.status_code == 409 and res.json()["code"] == "evidence_seal_broken" and models.judges == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize("meanwhile", [
|
||||
# 裁判的这几秒里有人点了「继续运行」(失败的运行可以接着跑):运行又活了
|
||||
{"status": "running", "error": None},
|
||||
# 接着跑完、重新封存了:manifest_seq 和清单哈希都换了,这次点击依据的那份封存已经不是最新的
|
||||
{"manifest_seq": 10_000, "manifest_hash": "0" * 64},
|
||||
])
|
||||
async def test_a_run_continued_during_the_judge_call_records_no_event(client, monkeypatch, meanwhile):
|
||||
"""判定只接在点击开始时的那份封存后面。运行在裁判期间被接着跑了,再追加就落进一次活着的运行中间,
|
||||
还会被封进下一份清单:不记进运行记录,判定照样交回给点的人,并说明为什么没记。"""
|
||||
models = Models(monkeypatch, verdicts={"新客": "unsupported"})
|
||||
row = await finish(weekly(claims="judge"))
|
||||
uid = unit_ids(doc_of(row))[CAUSAL]
|
||||
ask_model = judge_mod.get_chat_model
|
||||
|
||||
async def continued_while_judging(session, spec):
|
||||
async with SessionLocal() as other:
|
||||
await other.execute(update(Run).where(Run.id == row.id).values(**meanwhile))
|
||||
await other.commit()
|
||||
return await ask_model(session, spec)
|
||||
|
||||
monkeypatch.setattr(judge_mod, "get_chat_model", continued_while_judging)
|
||||
res = await judge(client, row.id, [uid])
|
||||
assert res.status_code == 200, res.text
|
||||
body = res.json()
|
||||
assert len(models.judges) == 1 and body["calls"] == 1, "模型问过了"
|
||||
assert body["verdicts"][uid]["status"] == "unsupported" and body["judged"] == [uid], "判定照样交回"
|
||||
assert body["event"] is None and "接着跑" in body["message"] and "没有记进运行记录" in body["message"]
|
||||
assert await judged_events(row.id) == []
|
||||
async with SessionLocal() as session:
|
||||
after = await session.get(Run, row.id)
|
||||
assert after.last_seq == row.last_seq, "运行记录一条都没多"
|
||||
|
||||
|
||||
async def test_a_continue_that_wins_the_row_between_check_and_write_gets_no_event(client, monkeypatch):
|
||||
"""核对过状态、还没写入的那一下,接着跑的条件更新(runner._claim)先提交了:追加判定的条件更新落空,
|
||||
照样不记——核对和写入是同一行上的条件更新,不是先读后写。"""
|
||||
from app.core.bus import bus
|
||||
from app.core.config import settings
|
||||
|
||||
Models(monkeypatch, verdicts={"新客": "unsupported"})
|
||||
row = await finish(weekly(claims="judge"))
|
||||
uid = unit_ids(doc_of(row))[CAUSAL]
|
||||
set_seq = bus.set_seq
|
||||
|
||||
def claimed_in_between(run_id, value):
|
||||
# _append_judged 读完运行、分配序号之前正好调用它:在这里用另一条连接把运行占成 running
|
||||
if run_id == row.id:
|
||||
db = sqlite3.connect(settings.db_path, timeout=5)
|
||||
try:
|
||||
db.execute("UPDATE runs SET status = 'running' WHERE id = ?", (row.id,))
|
||||
db.commit()
|
||||
finally:
|
||||
db.close()
|
||||
set_seq(run_id, value)
|
||||
|
||||
monkeypatch.setattr(bus, "set_seq", claimed_in_between)
|
||||
body = (await judge(client, row.id, [uid])).json()
|
||||
assert body["verdicts"][uid]["status"] == "unsupported" and body["event"] is None
|
||||
assert "没有记进运行记录" in body["message"] and await judged_events(row.id) == []
|
||||
async with SessionLocal() as session:
|
||||
after = await session.get(Run, row.id)
|
||||
assert after.status == "running" and after.last_seq == row.last_seq
|
||||
|
||||
|
||||
async def test_missing_runs_and_reports(client, monkeypatch):
|
||||
Models(monkeypatch)
|
||||
res = await judge(client, "nope", ["u1"])
|
||||
assert res.status_code == 404 and res.json()["code"] == "run_not_found"
|
||||
row = await finish(chain(node("start", "input"), node("out", "output", fields=[{"name": "a", "value": "x"}])))
|
||||
res = await judge(client, row.id, ["u1"])
|
||||
assert res.status_code == 404 and res.json()["code"] == "evidence_report_not_found"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("payload", [{}, {"units": []}, {"units": "u1"}, {"units": [1]}, {"units": [""]},
|
||||
{"units": [f"u{i}" for i in range(201)]}, {"units": ["u1"], "report": 3}])
|
||||
async def test_bad_requests_are_422(client, monkeypatch, payload):
|
||||
Models(monkeypatch)
|
||||
row = await finish(weekly())
|
||||
res = await client.post(f"/api/runs/{row.id}/evidence/judge", json=payload)
|
||||
assert res.status_code == 422 and res.json()["code"] == "evidence_judge_bad_units"
|
||||
|
||||
|
||||
async def test_sentences_that_are_not_claims_are_skipped(client, monkeypatch):
|
||||
models = Models(monkeypatch)
|
||||
row = await finish(weekly(claims="judge"))
|
||||
ids = unit_ids(doc_of(row))
|
||||
body = (await judge(client, row.id, [ids["本周"], ids[LINK], "u999"])).json()
|
||||
assert body["skipped"] == {ids["本周"]: "not_a_claim", ids[LINK]: "not_a_claim", "u999": "not_found"}
|
||||
assert body["verdicts"] == {} and models.judges == [] and body["event"] is None
|
||||
assert "不是结论句" in body["message"]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 片段接口、证据图带上判定
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_the_segment_shows_the_verdict_and_whether_it_can_still_be_asked(client, monkeypatch):
|
||||
Models(monkeypatch, verdicts={"新客": "unsupported"})
|
||||
row = await finish(weekly(claims="judge"))
|
||||
doc = doc_of(row)
|
||||
ids = unit_ids(doc)
|
||||
before = (await client.get(f"/api/runs/{row.id}/evidence/segments/{seg_in(doc, ids[CAUSAL])}")).json()
|
||||
assert before["unit"]["verdict"]["status"] == "unjudged" and before["unit"]["verdict"]["reason"] == "on_demand"
|
||||
assert before["unit"]["on_demand"] == {"available": True, "reason": None, "message": None}
|
||||
|
||||
await judge(client, row.id, [ids[CAUSAL]])
|
||||
after = (await client.get(f"/api/runs/{row.id}/evidence/segments/{seg_in(doc, ids[CAUSAL])}")).json()
|
||||
assert after["unit"]["verdict"]["status"] == "unsupported" and after["unit"]["verdict"]["post_seal"] is True
|
||||
assert after["unit"]["on_demand"]["available"] is False and after["unit"]["on_demand"]["reason"] == "judged"
|
||||
assert "模型" in after["note"] and "后续版本" not in after["note"]
|
||||
other = (await client.get(f"/api/runs/{row.id}/evidence/segments/{seg_in(doc, ids[CITED])}")).json()
|
||||
assert other["unit"]["on_demand"]["available"] is True
|
||||
assert "挂了依据" in other["note"] and "模型" in other["note"] and "后续版本" not in other["note"]
|
||||
link = (await client.get(f"/api/runs/{row.id}/evidence/segments/{seg_in(doc, ids[LINK])}")).json()
|
||||
assert link["unit"]["on_demand"]["reason"] == "not_a_claim" and "verdict" not in link["unit"]
|
||||
|
||||
graph = (await client.get(f"/api/runs/{row.id}/evidence")).json()
|
||||
[report] = graph["reports"]
|
||||
assert report["claims"] == "judge" and report["judge"]["mode"] == "on_demand"
|
||||
assert report["post_seal_verdicts"] == {ids[CAUSAL]: after["unit"]["verdict"]}
|
||||
audit = (await client.get(f"/api/runs/{row.id}/evidence/audit")).json()
|
||||
assert audit["reports"][0]["post_seal_verdicts"] == report["post_seal_verdicts"]
|
||||
|
||||
|
||||
async def test_sealed_verdicts_of_a_formal_run_are_shown_as_sealed(client, monkeypatch):
|
||||
Models(monkeypatch, verdicts={"新客": "unsupported"})
|
||||
row = await finish(weekly(claims="judge", judge={"max_cost_usd": 0.05}), run_class="formal")
|
||||
doc = doc_of(row)
|
||||
body = (await client.get(f"/api/runs/{row.id}/evidence/segments/{seg_in(doc, unit_ids(doc)[CAUSAL])}")).json()
|
||||
assert body["unit"]["verdict"]["status"] == "unsupported" and body["unit"]["verdict"]["post_seal"] is False
|
||||
assert body["unit"]["on_demand"]["reason"] == "formal"
|
||||
graph = (await client.get(f"/api/runs/{row.id}/evidence")).json()
|
||||
assert graph["reports"][0]["post_seal_verdicts"] == {} and graph["reports"][0]["judge"]["mode"] == "inline"
|
||||
|
||||
|
||||
async def test_a_forged_event_cannot_overwrite_a_sealed_verdict(client, monkeypatch):
|
||||
"""封存后追加的判定只填补没判过的句子:封存内判过的,后面追加什么都盖不掉。"""
|
||||
Models(monkeypatch, verdicts={"新客": "unsupported"})
|
||||
row = await finish(weekly(claims="judge", judge={"max_cost_usd": 0.05}), run_class="formal")
|
||||
doc = doc_of(row)
|
||||
uid = unit_ids(doc)[CAUSAL]
|
||||
doc_artifact = row.output["_evidence"]["doc_artifact"]
|
||||
await run_manager.note(row.id, EventType.EVIDENCE_JUDGED, node_id="write", report="write",
|
||||
doc_artifact=doc_artifact, post_seal=True,
|
||||
verdicts={uid: {"status": "supported", "rationale": "改好了", "judge": "x", "post_seal": True}})
|
||||
body = (await client.get(f"/api/runs/{row.id}/evidence/segments/{seg_in(doc, uid)}")).json()
|
||||
assert body["unit"]["verdict"]["status"] == "unsupported"
|
||||
|
||||
|
||||
async def test_appended_verdicts_always_read_as_post_seal(client, monkeypatch):
|
||||
"""追加的判定不管载荷里怎么写,展示出来都是「封存后追加」:只有随文档封存的判定才算封存内的。"""
|
||||
Models(monkeypatch)
|
||||
row = await finish(weekly(claims="judge"))
|
||||
doc = doc_of(row)
|
||||
uid = unit_ids(doc)[CAUSAL]
|
||||
await run_manager.note(row.id, EventType.EVIDENCE_JUDGED, node_id="write", report="write",
|
||||
doc_artifact=row.output["_evidence"]["doc_artifact"],
|
||||
verdicts={uid: {"status": "supported", "rationale": "x", "judge": "x", "post_seal": False},
|
||||
"u999": {"status": "supported", "rationale": "x", "judge": "x"}})
|
||||
body = (await client.get(f"/api/runs/{row.id}/evidence/segments/{seg_in(doc, uid)}")).json()
|
||||
assert body["unit"]["verdict"]["status"] == "supported" and body["unit"]["verdict"]["post_seal"] is True
|
||||
graph = (await client.get(f"/api/runs/{row.id}/evidence")).json()
|
||||
assert set(graph["reports"][0]["post_seal_verdicts"]) == {uid}, "文档里没有的句子不认"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 裁判只看得到封存范围内的证据
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
SOURCE = "warehouse"
|
||||
TOOL = f"db_query__{SOURCE}"
|
||||
SQL = "SELECT region, SUM(amount) AS gmv, COUNT(*) AS order_cnt FROM orders GROUP BY region ORDER BY gmv DESC"
|
||||
EAST, WEST = "东区销售额 [[v:Q1.r0.gmv]]。[[see:Q1]]", "西区销售额 [[v:Q1.r1.gmv]]。[[see:Q1]]"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
async def warehouse(tmp_path):
|
||||
from app.data.introspect import introspect
|
||||
|
||||
path = tmp_path / "warehouse.db"
|
||||
db = sqlite3.connect(path)
|
||||
db.executescript("CREATE TABLE orders (id INTEGER PRIMARY KEY, region TEXT, amount REAL);")
|
||||
db.executemany("INSERT INTO orders VALUES (?, ?, ?)", [(1, "east", 100.5), (2, "west", 200.0), (3, "east", 300.0)])
|
||||
db.commit()
|
||||
db.close()
|
||||
probe = SimpleNamespace(id=f"probe-{path.name}", name=SOURCE, kind="sqlite", database=str(path), host=None,
|
||||
port=None, username=None, password=None, options={}, readonly=True, schema_cache={})
|
||||
cache = await introspect(probe)
|
||||
await engines.invalidate(probe.id)
|
||||
async with SessionLocal() as session:
|
||||
row = (await session.execute(select(DataSource).where(DataSource.name == SOURCE))).scalar_one_or_none()
|
||||
if row is None:
|
||||
row = DataSource(name=SOURCE, kind="sqlite", database=str(path), readonly=True, enabled=True)
|
||||
session.add(row)
|
||||
row.database, row.kind, row.enabled, row.readonly, row.schema_cache = str(path), "sqlite", True, True, cache
|
||||
row.options = {}
|
||||
await session.commit()
|
||||
source_id = row.id
|
||||
await engines.invalidate(source_id)
|
||||
yield source_id
|
||||
await engines.invalidate(source_id)
|
||||
|
||||
|
||||
def queried() -> dict:
|
||||
return chain(node("start", "input"), node("fetch", "tool", tool=TOOL, args={"sql": SQL}),
|
||||
node("write", "report", instructions="写周报"),
|
||||
node("out", "output", fields=[{"name": "周报", "value": "{{ nodes.write.text }}"}]))
|
||||
|
||||
|
||||
async def reseal(run_id: str) -> None:
|
||||
from app.core.artifact_store import manifest_hash
|
||||
|
||||
async with SessionLocal() as session:
|
||||
run = await session.get(Run, run_id)
|
||||
rows = [tuple(r) for r in await session.execute(
|
||||
select(RunEvent.seq, RunEvent.type, RunEvent.node_id, RunEvent.data)
|
||||
.where(RunEvent.run_id == run_id, RunEvent.seq <= run.manifest_seq).order_by(RunEvent.seq))]
|
||||
run.manifest_hash = manifest_hash(rows)
|
||||
await session.commit()
|
||||
|
||||
|
||||
async def unseal_queries(run_id: str) -> str:
|
||||
"""把查询快照从封存的事件里摘掉、按改过的事件重新封存:快照还在工件库和工件表里,封存链完好,
|
||||
只是没有哪条封存的事件引用它(等于「人为插进工件表」)。返回那件快照的 id。"""
|
||||
async with SessionLocal() as session:
|
||||
rows = list((await session.execute(select(RunEvent).where(
|
||||
RunEvent.run_id == run_id, RunEvent.type.in_(["tool.end", "node.finished"])))).scalars())
|
||||
artifact = None
|
||||
for event in rows:
|
||||
data = dict(event.data or {})
|
||||
if event.type == "tool.end":
|
||||
artifact = data.pop("query_artifact", None) or artifact
|
||||
data.pop("schema_artifact", None)
|
||||
elif data.get("evidence"):
|
||||
data["evidence"] = [e for e in data["evidence"] if e.get("kind") not in ("query", "schema")]
|
||||
event.data = data
|
||||
await session.commit()
|
||||
await reseal(run_id)
|
||||
return artifact
|
||||
|
||||
|
||||
def excerpt_lines(request: str) -> tuple[str | None, list[str]]:
|
||||
header = re.search(r"^列:(.*)$", request, re.M)
|
||||
return (header.group(1) if header else None), re.findall(r"^r\d+:.*$", request, re.M)
|
||||
|
||||
|
||||
async def test_only_sealed_snapshots_reach_the_judge(client, monkeypatch, warehouse):
|
||||
models = Models(monkeypatch, text=EAST + WEST)
|
||||
control = await finish(queried())
|
||||
assert control.status == "succeeded", control.error
|
||||
await judge(client, control.id, [unit_starting(doc_of(control), "东区")])
|
||||
assert "【Q1】查询结果" in models.judges[0] and "SQL:SELECT region" in models.judges[0]
|
||||
assert excerpt_lines(models.judges[0])[1], "封存链完好时照常摘被引用的行"
|
||||
|
||||
row = await finish(queried())
|
||||
doc = doc_of(row)
|
||||
artifact = await unseal_queries(row.id)
|
||||
assert artifact and artifact_store.load(artifact) is not None, "快照本身还在工件库里"
|
||||
assert (await verify_manifest(row.id))["ok"] is True
|
||||
res = await judge(client, row.id, [unit_starting(doc, "东区")])
|
||||
assert res.status_code == 200, res.text
|
||||
assert "【Q1】" not in models.judges[1] and "SELECT" not in models.judges[1]
|
||||
assert excerpt_lines(models.judges[1]) == (None, [])
|
||||
|
||||
# 封存之后才追加的事件引用了它,也不算
|
||||
async with SessionLocal() as session:
|
||||
run = await session.get(Run, row.id)
|
||||
seq = run.last_seq
|
||||
from app.core.bus import bus
|
||||
|
||||
bus.set_seq(row.id, seq)
|
||||
await run_manager.note(row.id, "tool.end", node_id="fetch", tool=TOOL, query_artifact=artifact)
|
||||
await judge(client, row.id, [unit_starting(doc, "西区")])
|
||||
assert "【Q1】" not in models.judges[2] and "SELECT" not in models.judges[2]
|
||||
|
||||
|
||||
async def test_masked_columns_never_reach_the_judge(client, monkeypatch, warehouse):
|
||||
models = Models(monkeypatch, text=EAST)
|
||||
row = await finish(queried())
|
||||
async with SessionLocal() as session:
|
||||
source = await session.get(DataSource, warehouse)
|
||||
source.options = {"mask_columns": ["order_cnt"]}
|
||||
await session.commit()
|
||||
await judge(client, row.id, [unit_starting(doc_of(row), "东区")])
|
||||
header, lines = excerpt_lines(models.judges[0])
|
||||
assert header is not None and "order_cnt" not in header and "gmv" in header
|
||||
assert lines and all(line.count("|") == 1 for line in lines), lines
|
||||
assert "遮罩" in models.judges[0]
|
||||
@@ -0,0 +1,344 @@
|
||||
"""四期:受管模板放开 claims: judge(结论句由另一个模型按证据逐句裁判),但要写明预算。
|
||||
|
||||
- G3:报告撰写节点的 claims 在受管级别认 require_citation 或 judge;写 judge 的,生效的 judge 配置里
|
||||
必须有 max_cost_usd 这个键——写金额,或者显式写 null 表示不限。没写就是 error(已发布级别只给警告):
|
||||
受管出具不能让每份报告的裁判花费悄悄按默认值走,也不能悄悄不设上限
|
||||
- 发布前自动修复:缺预算是 choice,候选是默认金额和「不限」,不替人选;G3 另外那几项的一键修复不再把
|
||||
judge 改回 require_citation
|
||||
- 降低要求的判断:require_citation → judge 是往严里改;judge → require_citation / off / 删掉都算放宽,
|
||||
报告节点上和出具契约里一样
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
|
||||
import pytest
|
||||
|
||||
from app.engine.autofix import NEED_CHOICE, apply_fixes, forbidden_changes, plan_fixes, public
|
||||
from app.engine.governance import lint_for_publish, publish_issues
|
||||
from app.engine.judge import JUDGE_DEFAULTS
|
||||
from app.engine.schema import GraphSpec, validate_graph
|
||||
|
||||
STRICT = {"numbers": "strict", "on_violation": "fail", "claims": "require_citation"}
|
||||
JUDGED = {**STRICT, "claims": "judge", "judge": {"max_cost_usd": 0.05}}
|
||||
BUDGET = "governed.judge_budget"
|
||||
|
||||
|
||||
def node(nid, ntype, label=None, **config):
|
||||
return {"id": nid, "type": ntype, "position": {"x": 10, "y": 20},
|
||||
"data": {"label": label or nid, "config": config}}
|
||||
|
||||
|
||||
def traceable(*, report=None, defaults=None, **contract_overrides) -> dict:
|
||||
"""input → 查库 → 口径卡 → 报告撰写 → 出具:受管要求(含 G1–G5)全部满足。"""
|
||||
contract = {"report_from": "write", "metrics_from": ["card"], "required": ["gmv"], "strict": True}
|
||||
contract.update(contract_overrides)
|
||||
nodes = [
|
||||
node("start", "input", "周期"),
|
||||
node("fetch", "tool", "取数", tool="db_query__shop",
|
||||
args={"sql": "SELECT SUM(amount) AS gmv FROM orders"}),
|
||||
node("card", "metrics", "周报口径卡", caliber="周报口径", metrics=[
|
||||
{"id": "gmv", "name": "销售额", "unit": "元", "expression": "cell(nodes.fetch, 0, 'gmv')"}]),
|
||||
node("write", "report", "报告撰写", instructions="写周报", **(JUDGED if report is None else report)),
|
||||
node("done", "output", "出具", fields=[{"name": "周报", "value": "{{ nodes.write.text }}"}],
|
||||
contract={k: v for k, v in contract.items() if v is not None}),
|
||||
]
|
||||
edges = [{"id": f"e{i}", "source": a["id"], "target": b["id"]} for i, (a, b) in enumerate(zip(nodes, nodes[1:]))]
|
||||
graph = {"nodes": nodes, "edges": edges}
|
||||
if defaults is not None:
|
||||
graph["defaults"] = defaults
|
||||
return graph
|
||||
|
||||
|
||||
def config(graph: dict, nid: str) -> dict:
|
||||
return next(n for n in graph["nodes"] if n["id"] == nid)["data"]["config"]
|
||||
|
||||
|
||||
def by_code(graph: dict, code: str, level: str = "governed") -> list:
|
||||
return [i for i in lint_for_publish(GraphSpec.model_validate(graph), level=level).issues if i.code == code]
|
||||
|
||||
|
||||
def errors(graph: dict, level: str = "governed") -> list:
|
||||
return [i for i in publish_issues(GraphSpec.model_validate(graph), level=level) if i.level == "error"]
|
||||
|
||||
|
||||
def plan(graph: dict, level: str = "governed") -> dict[str, dict]:
|
||||
spec = GraphSpec.model_validate(graph)
|
||||
return {f["id"]: f for f in plan_fixes(spec, publish_issues(spec, level=level), level=level)}
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# G3:judge 可以发布成受管,但要写预算
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("budget", [0.05, 1, None])
|
||||
def test_judge_with_a_budget_publishes_as_governed(budget):
|
||||
graph = traceable(report={**JUDGED, "judge": {"max_cost_usd": budget, "max_claims": 20}})
|
||||
assert errors(graph) == [], [i.message for i in errors(graph)]
|
||||
assert by_code(graph, "governed.report_policy") == [] and by_code(graph, BUDGET) == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize("judge", [None, {}, {"max_claims": 20, "model": "judge-model"}])
|
||||
def test_judge_without_a_budget_is_an_error_at_governed(judge):
|
||||
report = {**STRICT, "claims": "judge", **({"judge": judge} if judge is not None else {})}
|
||||
graph = traceable(report=report)
|
||||
[issue] = by_code(graph, BUDGET)
|
||||
assert issue.level == "error" and issue.node_id == "write" and issue.field == "judge.max_cost_usd"
|
||||
assert "max_cost_usd" in issue.message and "不限" in issue.message and "null" in issue.message
|
||||
assert by_code(graph, "governed.report_policy") == [], "judge 本身合规,只缺预算"
|
||||
assert [i.level for i in by_code(graph, BUDGET, "published")] == ["warning"]
|
||||
|
||||
|
||||
def test_the_budget_follows_the_same_config_rule_as_run_time():
|
||||
"""和运行时同一个取值规则:节点没写 judge 取图级 defaults;节点写了就整个用节点的(不和 defaults 拼)。"""
|
||||
graph = traceable(report={**STRICT, "claims": "judge"}, defaults={"judge": {"max_cost_usd": 0.02}})
|
||||
assert by_code(graph, BUDGET) == []
|
||||
graph = traceable(report={**STRICT, "claims": "judge", "judge": {"max_claims": 5}},
|
||||
defaults={"judge": {"max_cost_usd": 0.02}})
|
||||
assert [i.field for i in by_code(graph, BUDGET)] == ["judge.max_cost_usd"]
|
||||
# claims 本身也可以来自图级 defaults
|
||||
graph = traceable(report={"numbers": "strict", "on_violation": "fail"},
|
||||
defaults={"claims": "judge", "judge": {"max_cost_usd": None}})
|
||||
assert by_code(graph, BUDGET) == [] and by_code(graph, "governed.report_policy") == []
|
||||
|
||||
|
||||
def test_a_malformed_budget_is_left_to_validate():
|
||||
graph = traceable(report={**STRICT, "claims": "judge", "judge": {"max_cost_usd": "cheap"}})
|
||||
assert by_code(graph, BUDGET) == [], "写错的值由 validate 的 report.judge_invalid 报,同一处不说两遍"
|
||||
[bad] = [i for i in validate_graph(GraphSpec.model_validate(graph)).issues if i.code == "report.judge_invalid"]
|
||||
assert bad.field == "judge.max_cost_usd" and bad.level == "error"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("claims", [None, "off"])
|
||||
def test_a_loose_claims_setting_names_both_accepted_values(claims):
|
||||
report = {"numbers": "strict", "on_violation": "fail", **({"claims": claims} if claims else {})}
|
||||
[issue] = by_code(traceable(report=report), "governed.report_policy")
|
||||
assert issue.field == "claims" and "claims: require_citation" in issue.message and "judge" in issue.message
|
||||
|
||||
|
||||
def test_judge_needs_no_other_report_policy_fix():
|
||||
graph = traceable(report={**JUDGED, "numbers": "off"})
|
||||
[issue] = by_code(graph, "governed.report_policy")
|
||||
assert issue.field == "numbers" and "claims" not in issue.message
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 发布前自动修复
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_the_report_policy_fix_leaves_judge_alone():
|
||||
graph = traceable(report={**JUDGED, "numbers": "off", "on_violation": "flag"})
|
||||
fix = plan(graph)["governed.report_policy:write"]
|
||||
assert fix["kind"] == "auto" and fix["preview"]["after"] == {"numbers": "strict", "on_violation": "fail"}
|
||||
out = apply_fixes(graph, [fix["id"]], level="governed")
|
||||
assert out["ok"] is True and config(out["graph"], "write")["claims"] == "judge"
|
||||
assert config(out["graph"], "write")["judge"] == {"max_cost_usd": 0.05}
|
||||
|
||||
|
||||
def test_a_missing_budget_is_a_choice_between_the_default_and_unlimited():
|
||||
graph = traceable(report={**STRICT, "claims": "judge", "judge": {"model": "judge-model"}})
|
||||
fid = f"{BUDGET}:write"
|
||||
fix = public(plan(graph)[fid])
|
||||
assert fix["kind"] == "choice" and fix["node_id"] == "write" and fix["field"] == "judge.max_cost_usd"
|
||||
assert [o["value"] for o in fix["options"]] == [JUDGE_DEFAULTS["report_max_cost_usd"], None]
|
||||
assert "default" not in fix, "花多少钱要人定:不替人选,也不给建议"
|
||||
unlimited = fix["options"][1]
|
||||
assert "不限" in unlimited["label"] and "不设上限" in unlimited["hint"] and "约束" in unlimited["hint"]
|
||||
assert all(not k.startswith("_") for k in fix), "内部键不交给前端"
|
||||
|
||||
[refused] = apply_fixes(graph, [fid], level="governed")["rejected"]
|
||||
assert refused["reason"] == NEED_CHOICE.format(label=fix["label"])
|
||||
|
||||
out = apply_fixes(graph, [fid], {fid: 0.05}, level="governed")
|
||||
assert out["applied"] == [fid] and out["ok"] is True
|
||||
assert config(out["graph"], "write")["judge"] == {"model": "judge-model", "max_cost_usd": 0.05}
|
||||
[change] = out["changes"]
|
||||
assert change["field"] == "judge" and change["before"] == {"model": "judge-model"}
|
||||
assert out["ops"] == [{"op": "update_node", "id": "write",
|
||||
"config": {"judge": {"model": "judge-model", "max_cost_usd": 0.05}}}]
|
||||
|
||||
out = apply_fixes(graph, [fid], {fid: None}, level="governed")
|
||||
judge = config(out["graph"], "write")["judge"]
|
||||
assert out["ok"] is True and "max_cost_usd" in judge and judge["max_cost_usd"] is None, "不限要显式写成 null"
|
||||
|
||||
|
||||
def test_the_budget_fix_keeps_what_the_graph_defaults_said():
|
||||
"""judge 写在图级 defaults 里:修复写到节点上时带上 defaults 里的整份 judge,节点写了就不再跟随 defaults,
|
||||
不能因为补一个预算把 defaults 里选的裁判模型丢了。"""
|
||||
graph = traceable(report={**STRICT, "claims": "judge"},
|
||||
defaults={"judge": {"model": "judge-model", "rewrite_once": True}})
|
||||
fid = f"{BUDGET}:write"
|
||||
out = apply_fixes(graph, [fid], {fid: None}, level="governed")
|
||||
assert out["ok"] is True
|
||||
assert config(out["graph"], "write")["judge"] == {"model": "judge-model", "rewrite_once": True,
|
||||
"max_cost_usd": None}
|
||||
assert out["graph"]["defaults"] == {"judge": {"model": "judge-model", "rewrite_once": True}}
|
||||
|
||||
|
||||
def test_a_choice_outside_the_options_is_refused():
|
||||
graph = traceable(report={**STRICT, "claims": "judge"})
|
||||
fid = f"{BUDGET}:write"
|
||||
out = apply_fixes(graph, [fid], {fid: 99}, level="governed")
|
||||
assert out["applied"] == [] and "不在候选里" in out["rejected"][0]["reason"]
|
||||
|
||||
|
||||
def test_published_level_budget_warnings_get_no_fix():
|
||||
assert f"{BUDGET}:write" not in plan(traceable(report={**STRICT, "claims": "judge"}), "published")
|
||||
|
||||
|
||||
def test_invalid_claims_fixes_no_longer_mention_a_later_version():
|
||||
graph = traceable(report={**STRICT, "claims": "sometimes"})
|
||||
governed = plan(graph)["report.claims_invalid:write"]
|
||||
published = plan(graph, "published")["report.claims_invalid:write"]
|
||||
assert "后续版本" not in governed["label"] and "default" not in published
|
||||
assert [o["value"] for o in published["options"]] == ["off", "require_citation"]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 降低要求的判断
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _changed(report_before: dict, report_after: dict | None, **graph_kw) -> list[str]:
|
||||
old = traceable(report=report_before, **graph_kw)
|
||||
new = copy.deepcopy(old)
|
||||
if report_after is None:
|
||||
config(new, "write").pop("claims")
|
||||
else:
|
||||
config(new, "write").clear()
|
||||
config(new, "write").update({"instructions": "写周报", **report_after})
|
||||
return forbidden_changes(old, new)
|
||||
|
||||
|
||||
def test_moving_up_to_judge_is_not_loosening():
|
||||
assert _changed(STRICT, JUDGED) == []
|
||||
assert _changed({**STRICT, "claims": "off"}, JUDGED) == []
|
||||
assert _changed({"numbers": "strict", "on_violation": "fail"}, JUDGED) == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize("after", ["off", "require_citation", None])
|
||||
def test_moving_down_from_judge_is_loosening(after):
|
||||
report_after = None if after is None else {**JUDGED, "claims": after}
|
||||
[reason] = _changed(JUDGED, report_after)
|
||||
assert "「报告撰写」" in reason and "claims" in reason and "judge" in reason
|
||||
|
||||
|
||||
def test_dropping_a_judge_set_in_the_graph_defaults_is_loosening():
|
||||
[reason] = _changed({"numbers": "strict", "on_violation": "fail"}, {**STRICT, "claims": "off"},
|
||||
defaults={"claims": "judge", "judge": {"max_cost_usd": 0.05}})
|
||||
assert "claims" in reason and "judge" in reason
|
||||
|
||||
|
||||
def test_changing_the_budget_is_not_loosening():
|
||||
assert _changed(JUDGED, {**JUDGED, "judge": {"max_cost_usd": None}}) == []
|
||||
|
||||
|
||||
# claims 一直是 judge 时,judge.on_unsupported 从 withhold 往回改也是放宽:不支持的结论句从「不予出具」变成
|
||||
# 「降档出具」。Copilot 改图的 update_node 会把 judge 整个盖掉,补预算时写 judge={max_cost_usd: 0.05}
|
||||
# 就悄悄丢了 withhold——这一类要和契约里的 on_unsupported 一样拒掉
|
||||
WITHHOLD = {**JUDGED, "judge": {"max_cost_usd": 0.05, "on_unsupported": "withhold"}}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("judge, dropped", [
|
||||
({"max_cost_usd": 0.05, "on_unsupported": "degrade"}, False),
|
||||
({"max_cost_usd": 0.05}, True),
|
||||
({"max_cost_usd": 0.05, "on_unsupported": "sometimes"}, False),
|
||||
])
|
||||
def test_relaxing_on_unsupported_on_the_node_is_loosening(judge, dropped):
|
||||
[reason] = _changed(WITHHOLD, {**WITHHOLD, "judge": judge})
|
||||
assert "「报告撰写」" in reason and "judge.on_unsupported" in reason
|
||||
assert "withhold" in reason and "degrade" in reason
|
||||
assert ("去掉了写明的值" in reason) is dropped, reason
|
||||
|
||||
|
||||
def test_replacing_the_whole_judge_block_to_add_a_budget_is_loosening():
|
||||
"""补预算把 judge 整个换成 {max_cost_usd}:连同裁判模型、withhold 一起丢了。"""
|
||||
before = {**STRICT, "claims": "judge", "judge": {"model": "judge-model", "on_unsupported": "withhold"}}
|
||||
[reason] = _changed(before, {**before, "judge": {"max_cost_usd": 0.05}})
|
||||
assert "judge.on_unsupported" in reason and "withhold" in reason
|
||||
assert _changed(before, {**before, "judge": {**before["judge"], "max_cost_usd": 0.05}}) == [], \
|
||||
"带着原来的键补预算不算放宽"
|
||||
|
||||
|
||||
def test_a_withhold_from_the_graph_defaults_lost_by_a_node_judge_is_loosening():
|
||||
"""judge 按运行时的规则取:节点写了自己的 judge 就整个用节点的,图级 defaults 里的 withhold 不再生效。"""
|
||||
defaults = {"judge": {"on_unsupported": "withhold", "max_cost_usd": 0.05}}
|
||||
report = {**STRICT, "claims": "judge"}
|
||||
[reason] = _changed(report, {**report, "judge": {"max_cost_usd": 0.05}}, defaults=defaults)
|
||||
assert "judge.on_unsupported" in reason and "从 withhold(跟随全图默认)" in reason, reason
|
||||
assert "不再跟随全图默认" in reason, "说清楚为什么全图默认里的 withhold 不生效了"
|
||||
assert _changed(report, {**report, "judge": {"max_cost_usd": 0.05, "on_unsupported": "withhold"}},
|
||||
defaults=defaults) == []
|
||||
|
||||
|
||||
def test_relaxing_on_unsupported_in_the_graph_defaults_is_loosening():
|
||||
old = traceable(report={**STRICT, "claims": "judge"},
|
||||
defaults={"judge": {"max_cost_usd": 0.05, "on_unsupported": "withhold"}})
|
||||
new = copy.deepcopy(old)
|
||||
new["defaults"]["judge"]["on_unsupported"] = "degrade"
|
||||
[reason] = forbidden_changes(old, new)
|
||||
assert "「报告撰写」" in reason and "judge.on_unsupported" in reason
|
||||
|
||||
|
||||
@pytest.mark.parametrize("before, after", [
|
||||
(JUDGED, {**JUDGED, "judge": {"max_cost_usd": 0.05, "on_unsupported": "withhold"}}),
|
||||
(WITHHOLD, {**WITHHOLD, "judge": {**WITHHOLD["judge"], "max_cost_usd": None, "rewrite_once": True}}),
|
||||
(WITHHOLD, WITHHOLD),
|
||||
# 升到 judge 本身是收紧:不管带不带 withhold
|
||||
({**STRICT, "judge": {"on_unsupported": "withhold"}}, JUDGED),
|
||||
])
|
||||
def test_tightening_or_keeping_on_unsupported_is_fine(before, after):
|
||||
assert _changed(before, after) == []
|
||||
|
||||
|
||||
def test_dropping_withhold_along_with_judge_is_one_reason():
|
||||
"""从 judge 退回 require_citation:claims 那一条已经说了,不再为 on_unsupported 另记一条。"""
|
||||
[reason] = _changed(WITHHOLD, {**STRICT, "judge": {"max_cost_usd": 0.05}})
|
||||
assert "claims" in reason and "on_unsupported" not in reason
|
||||
|
||||
|
||||
@pytest.mark.parametrize("before", ["require_citation", "judge"])
|
||||
def test_moving_claims_to_an_unknown_value_is_loosening(before):
|
||||
"""require_citation、judge 改成认不出的写法:出口按认不出的走,要求没了——照样算放宽,别只落个
|
||||
「没让图变好」。原来就认不出的不比较。"""
|
||||
report = {**STRICT, "claims": before, **({"judge": {"max_cost_usd": 0.05}} if before == "judge" else {})}
|
||||
[reason] = _changed(report, {**report, "claims": "sometimes"})
|
||||
assert "「报告撰写」" in reason and before in reason and "sometimes" in reason and "不认识" in reason
|
||||
assert _changed({**STRICT, "claims": "sometimes"}, {**STRICT, "claims": "off"}) == []
|
||||
assert _changed({**STRICT, "claims": "off"}, {**STRICT, "claims": "sometimes"}) == []
|
||||
|
||||
|
||||
def _contract_changed(before, after) -> list[str]:
|
||||
old = traceable(claims=before)
|
||||
new = copy.deepcopy(old)
|
||||
contract = config(new, "done")["contract"]
|
||||
if after is None:
|
||||
contract.pop("claims", None)
|
||||
else:
|
||||
contract["claims"] = after
|
||||
return forbidden_changes(old, new)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("before, after", [
|
||||
("judge", "off"),
|
||||
("judge", None),
|
||||
("judge", "require_citation"),
|
||||
({"policy": "judge", "on_unsupported": "withhold"}, "judge"),
|
||||
({"policy": "judge", "on_unsupported": "withhold"}, {"policy": "judge", "on_unsupported": "degrade"}),
|
||||
({"policy": "require_citation", "on_uncited": "withhold"}, "judge"),
|
||||
])
|
||||
def test_loosening_a_judge_contract_is_refused(before, after):
|
||||
[reason] = _contract_changed(before, after)
|
||||
assert "「出具」" in reason and "claims" in reason
|
||||
|
||||
|
||||
@pytest.mark.parametrize("before, after", [
|
||||
("require_citation", "judge"),
|
||||
("off", {"policy": "judge", "on_unsupported": "withhold"}),
|
||||
("judge", {"policy": "judge", "on_unsupported": "withhold"}),
|
||||
({"policy": "require_citation", "on_uncited": "withhold"}, {"policy": "judge", "on_uncited": "withhold"}),
|
||||
])
|
||||
def test_tightening_a_contract_towards_judge_is_fine(before, after):
|
||||
assert _contract_changed(before, after) == []
|
||||
@@ -20,7 +20,7 @@ from app.db.base import SessionLocal
|
||||
from app.db.models import Run, Workflow, WorkflowVersion
|
||||
from app.engine.governance import lint_for_publish, publish_issues
|
||||
from app.engine.runner import run_manager
|
||||
from app.engine.schema import CLAIMS_LATER, GraphSpec, validate_graph
|
||||
from app.engine.schema import GraphSpec, validate_graph
|
||||
from app.main import app
|
||||
from app.seed import TEMPLATES
|
||||
|
||||
@@ -281,10 +281,10 @@ def test_graph_defaults_count_like_they_do_at_run_time():
|
||||
|
||||
|
||||
def test_an_unsupported_claims_value_is_left_to_validate():
|
||||
graph = traceable(report={**STRICT, "claims": "judge"})
|
||||
graph = traceable(report={**STRICT, "claims": "sometimes"})
|
||||
assert by_code(graph, "governed.report_policy") == []
|
||||
[bad] = [i for i in validate_graph(GraphSpec.model_validate(graph)).issues if i.code == "report.claims_invalid"]
|
||||
assert bad.level == "error" and CLAIMS_LATER in bad.message
|
||||
assert bad.level == "error" and "off / require_citation" in bad.message
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
@@ -442,7 +442,7 @@ async def test_an_already_governed_version_keeps_running_formally(client, engine
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# validate:报告撰写节点的 claims 只接受 off / require_citation
|
||||
# validate:报告撰写节点的 claims 只接受 off / require_citation / judge(四期放开 judge)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
@@ -458,10 +458,9 @@ def test_supported_claims_values_pass(value):
|
||||
assert claims_issues(value) == []
|
||||
|
||||
|
||||
def test_judge_is_a_later_version():
|
||||
[issue] = claims_issues("judge")
|
||||
assert issue.level == "error" and issue.node_id == "write" and issue.field == "claims"
|
||||
assert CLAIMS_LATER in issue.message and "off" in issue.message and "require_citation" in issue.message
|
||||
def test_judge_is_supported_now():
|
||||
assert claims_issues("judge") == []
|
||||
assert claims_issues("judge", on_defaults=True) == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize("value", ["strict", "Require_Citation", 1, ["off"], {"policy": "judge"}])
|
||||
@@ -471,5 +470,5 @@ def test_other_claims_values_are_errors(value):
|
||||
|
||||
|
||||
def test_claims_from_graph_defaults_are_checked_too():
|
||||
[issue] = claims_issues("judge", on_defaults=True)
|
||||
[issue] = claims_issues("sometimes", on_defaults=True)
|
||||
assert issue.node_id == "write"
|
||||
|
||||
@@ -0,0 +1,262 @@
|
||||
"""结论句裁判的设置与画布校验。
|
||||
|
||||
- 设置里单开一组 judge:证据裁判模型、每份报告的句数 / 金额 / 时长上限、探索运行每次点击的金额上限、
|
||||
全局每日金额上限。每一项都能存成 null,表示不限——存了 null 就是不限,不会被默认值悄悄盖回去
|
||||
- limits 组是环境变量定的、只读,裁判设置不放进去;每日计数存在设置表里,但不从设置接口进出
|
||||
- 裁判模型的优先级:节点上的 judge.model > 设置里的证据裁判模型 > Copilot 模型 > 默认 provider
|
||||
- 画布校验放开 claims: judge,并校验 judge 子配置的形状:上限可以写 null(不限),rewrite_once 是布尔值,
|
||||
on_unsupported 取 degrade / withhold
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
from sqlalchemy import delete
|
||||
|
||||
from app.db.base import SessionLocal
|
||||
from app.db.models import Setting
|
||||
from app.engine.judge import (
|
||||
JUDGE_DEFAULTS,
|
||||
SETTING_GROUP,
|
||||
SPEND_KEY,
|
||||
Budget,
|
||||
click_budget,
|
||||
judge_model_spec,
|
||||
judge_settings,
|
||||
report_budget,
|
||||
)
|
||||
from app.engine.schema import CLAIMS_VALUES, JUDGE_KEYS, GraphSpec, claims_problem, validate_graph
|
||||
from app.main import app
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
async def clean():
|
||||
async with SessionLocal() as session:
|
||||
await session.execute(delete(Setting).where(Setting.key.in_([SPEND_KEY, SETTING_GROUP, "copilot"])))
|
||||
await session.commit()
|
||||
yield
|
||||
|
||||
|
||||
def client() -> AsyncClient:
|
||||
return AsyncClient(transport=ASGITransport(app=app), base_url="http://test")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 设置
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_the_defaults_are_the_agreed_ones():
|
||||
assert SETTING_GROUP == "judge" and SETTING_GROUP != "limits"
|
||||
assert JUDGE_DEFAULTS == {
|
||||
"provider": None, "model": None,
|
||||
"report_max_claims": 40, "report_max_cost_usd": 0.05, "report_timeout_s": 30,
|
||||
"click_max_cost_usd": 0.01, "daily_max_usd": 2.0,
|
||||
}
|
||||
|
||||
|
||||
async def test_settings_expose_the_judge_group_with_defaults():
|
||||
async with client() as c:
|
||||
got = (await c.get("/api/settings")).json()
|
||||
assert got["judge"] == JUDGE_DEFAULTS
|
||||
assert SPEND_KEY not in got
|
||||
|
||||
|
||||
async def test_null_is_stored_as_unlimited_and_stays_unlimited():
|
||||
async with client() as c:
|
||||
r = await c.put("/api/settings", json={"values": {"judge": {
|
||||
**JUDGE_DEFAULTS, "model": "judge-model", "report_max_cost_usd": None, "daily_max_usd": None,
|
||||
"report_timeout_s": None, "report_max_claims": None, "click_max_cost_usd": None}}})
|
||||
assert r.status_code == 200, r.text
|
||||
got = (await c.get("/api/settings")).json()["judge"]
|
||||
assert got["model"] == "judge-model"
|
||||
assert all(got[k] is None for k in ("report_max_cost_usd", "daily_max_usd", "report_timeout_s",
|
||||
"report_max_claims", "click_max_cost_usd"))
|
||||
async with SessionLocal() as session:
|
||||
settings = await judge_settings(session)
|
||||
assert report_budget(settings, None) == Budget(max_claims=None, max_cost_usd=None, timeout_s=None,
|
||||
daily_max_usd=None)
|
||||
assert click_budget(settings).max_cost_usd is None
|
||||
|
||||
|
||||
async def test_a_partial_group_keeps_the_other_defaults():
|
||||
async with client() as c:
|
||||
await c.put("/api/settings", json={"values": {"judge": {"daily_max_usd": 5}}})
|
||||
got = (await c.get("/api/settings")).json()["judge"]
|
||||
assert got == {**JUDGE_DEFAULTS, "daily_max_usd": 5}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("key,value", [
|
||||
("daily_max_usd", -1), ("report_max_cost_usd", 0), ("report_max_cost_usd", "0.05"),
|
||||
("report_max_claims", 2.5), ("report_max_claims", True), ("report_timeout_s", 0),
|
||||
("click_max_cost_usd", float("nan")), ("model", 3), ("unknown_key", 1),
|
||||
])
|
||||
async def test_bad_values_are_refused_with_a_reason(key, value):
|
||||
async with client() as c:
|
||||
r = await c.put("/api/settings", json={"values": {"judge": {key: value}}}) if value == value else \
|
||||
await c.put("/api/settings", content='{"values": {"judge": {"%s": NaN}}}' % key,
|
||||
headers={"content-type": "application/json"})
|
||||
assert r.status_code == 422, r.text
|
||||
assert key in r.json()["detail"]
|
||||
assert (await c.get("/api/settings")).json()["judge"] == JUDGE_DEFAULTS, "整组都没写进去"
|
||||
|
||||
|
||||
async def test_the_daily_counter_does_not_travel_through_the_settings_api():
|
||||
async with SessionLocal() as session:
|
||||
session.add(Setting(key=SPEND_KEY, value={"date": "2026-09-29", "usd": 1.5, "calls": 3}))
|
||||
await session.commit()
|
||||
async with client() as c:
|
||||
assert SPEND_KEY not in (await c.get("/api/settings")).json()
|
||||
await c.put("/api/settings", json={"values": {SPEND_KEY: {"date": "2026-09-29", "usd": 0, "calls": 0}}})
|
||||
async with SessionLocal() as session:
|
||||
row = await session.get(Setting, SPEND_KEY)
|
||||
assert row.value["usd"] == 1.5, "设置页整组回写不能把今天的花费清零"
|
||||
|
||||
|
||||
async def test_todays_spend_is_readable_next_to_the_daily_cap(monkeypatch):
|
||||
from app.engine import judge as judge_mod
|
||||
|
||||
monkeypatch.setattr(judge_mod, "_today", lambda: "2026-09-29")
|
||||
async with SessionLocal() as session:
|
||||
session.add(Setting(key=SPEND_KEY, value={"date": "2026-09-28", "usd": 1.9, "calls": 7}))
|
||||
await session.commit()
|
||||
async with client() as c:
|
||||
got = (await c.get("/api/settings/judge/spend")).json()
|
||||
assert got == {"date": "2026-09-29", "usd": 0.0, "calls": 0, "unpriced_calls": 0, "daily_max_usd": 2.0}, \
|
||||
"昨天的花费不算今天的"
|
||||
async with SessionLocal() as session:
|
||||
row = await session.get(Setting, SPEND_KEY)
|
||||
row.value = {"date": "2026-09-29", "usd": 0.42, "calls": 3, "unpriced_calls": 1}
|
||||
await session.commit()
|
||||
await c.put("/api/settings", json={"values": {"judge": {"daily_max_usd": None}}})
|
||||
got = (await c.get("/api/settings/judge/spend")).json()
|
||||
assert got == {"date": "2026-09-29", "usd": 0.42, "calls": 3, "unpriced_calls": 1, "daily_max_usd": None}
|
||||
|
||||
|
||||
async def test_bad_rows_already_in_the_database_fall_back_to_defaults():
|
||||
"""库里存坏了的值(旧版本、手改)读出来按默认,不让裁判拿着一个负数上限跑。"""
|
||||
async with SessionLocal() as session:
|
||||
session.add(Setting(key=SETTING_GROUP, value={"daily_max_usd": -3, "report_max_claims": "many",
|
||||
"report_timeout_s": None}))
|
||||
await session.commit()
|
||||
settings = await judge_settings(session)
|
||||
assert settings["daily_max_usd"] == JUDGE_DEFAULTS["daily_max_usd"]
|
||||
assert settings["report_max_claims"] == JUDGE_DEFAULTS["report_max_claims"]
|
||||
assert settings["report_timeout_s"] is None
|
||||
|
||||
|
||||
def test_the_node_overrides_key_by_key():
|
||||
settings = {**JUDGE_DEFAULTS, "report_max_claims": 10}
|
||||
assert report_budget(settings, None) == Budget(max_claims=10, max_cost_usd=0.05, timeout_s=30, daily_max_usd=2.0)
|
||||
assert report_budget(settings, {"max_cost_usd": None, "timeout_s": 5}) == \
|
||||
Budget(max_claims=10, max_cost_usd=None, timeout_s=5, daily_max_usd=2.0)
|
||||
# 探索运行每次点击:金额按点击的上限,句数、时长按每份报告的
|
||||
assert click_budget(settings) == Budget(max_claims=10, max_cost_usd=0.01, timeout_s=30, daily_max_usd=2.0)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 裁判模型的优先级
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_the_judge_model_priority():
|
||||
async with SessionLocal() as session:
|
||||
spec = await judge_model_spec(session, None)
|
||||
assert (spec.provider, spec.model) == (None, None), "什么都没配:默认 provider"
|
||||
assert spec.thinking == "off" and spec.max_tokens == 4096
|
||||
|
||||
session.add(Setting(key="copilot", value={"provider": "", "model": "copilot-model"}))
|
||||
await session.commit()
|
||||
assert (await judge_model_spec(session, None)).model == "copilot-model"
|
||||
|
||||
session.add(Setting(key=SETTING_GROUP, value={"provider": "house", "model": "judge-model"}))
|
||||
await session.commit()
|
||||
spec = await judge_model_spec(session, None)
|
||||
assert (spec.provider, spec.model) == ("house", "judge-model")
|
||||
|
||||
spec = await judge_model_spec(session, {"model": "node-model"})
|
||||
assert (spec.provider, spec.model) == (None, "node-model"), "节点上的写法整组生效,不和设置里的拼"
|
||||
spec = await judge_model_spec(session, {"model": None, "provider": None, "max_claims": 3})
|
||||
assert spec.model == "judge-model"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 画布校验
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def node(nid, ntype, **config):
|
||||
return {"id": nid, "type": ntype, "position": {"x": 0, "y": 0}, "data": {"label": nid, "config": config}}
|
||||
|
||||
|
||||
def graph(report: dict, defaults: dict | None = None) -> dict:
|
||||
nodes = [node("start", "input"),
|
||||
node("caliber", "metrics", caliber="周报口径", metrics=[{"id": "gmv", "expression": "1"}]),
|
||||
node("write", "report", instructions="写周报", **report),
|
||||
node("out", "output", fields=[{"name": "周报", "value": "{{ nodes.write.text }}"}])]
|
||||
ids = [n["id"] for n in nodes]
|
||||
g = {"nodes": nodes, "edges": [{"source": a, "target": b} for a, b in zip(ids, ids[1:])]}
|
||||
if defaults:
|
||||
g["defaults"] = defaults
|
||||
return g
|
||||
|
||||
|
||||
def issues(g, code=None):
|
||||
return [i for i in validate_graph(GraphSpec.model_validate(g)).issues
|
||||
if i.code in ((code,) if code else ("report.claims_invalid", "report.judge_invalid"))]
|
||||
|
||||
|
||||
def test_judge_is_a_supported_policy_now():
|
||||
assert CLAIMS_VALUES == ("off", "require_citation", "judge")
|
||||
assert claims_problem("judge") is None
|
||||
assert issues(graph({"claims": "judge"})) == []
|
||||
assert issues(graph({}, defaults={"claims": "judge"})) == []
|
||||
|
||||
|
||||
def test_every_limit_can_be_unlimited():
|
||||
judge = {"provider": None, "model": None, "max_claims": None, "max_cost_usd": None, "timeout_s": None,
|
||||
"rewrite_once": True, "on_unsupported": "withhold"}
|
||||
assert set(judge) == set(JUDGE_KEYS)
|
||||
assert issues(graph({"claims": "judge", "judge": judge})) == []
|
||||
assert issues(graph({"claims": "judge", "judge": {"max_claims": 40, "max_cost_usd": 0.05, "timeout_s": 30,
|
||||
"model": "judge-model", "on_unsupported": "degrade"}})) == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize("judge,field,words", [
|
||||
("fast", "judge", "对象"),
|
||||
({"max_cost": 1}, "judge.max_cost", "max_cost_usd"),
|
||||
({"max_claims": 0}, "judge.max_claims", "正整数"),
|
||||
({"max_claims": 2.5}, "judge.max_claims", "正整数"),
|
||||
({"max_claims": True}, "judge.max_claims", "正整数"),
|
||||
({"max_cost_usd": -1}, "judge.max_cost_usd", "大于 0"),
|
||||
({"max_cost_usd": "0.05"}, "judge.max_cost_usd", "大于 0"),
|
||||
({"timeout_s": 0}, "judge.timeout_s", "大于 0"),
|
||||
({"rewrite_once": "yes"}, "judge.rewrite_once", "true 或 false"),
|
||||
({"on_unsupported": "ignore"}, "judge.on_unsupported", "degrade / withhold"),
|
||||
({"model": 3}, "judge.model", "模型 id"),
|
||||
])
|
||||
def test_a_malformed_judge_block_is_an_error(judge, field, words):
|
||||
[issue] = issues(graph({"claims": "judge", "judge": judge}), "report.judge_invalid")
|
||||
assert issue.level == "error" and issue.node_id == "write" and issue.field == field
|
||||
assert words in issue.message, issue.message
|
||||
if field in ("judge.max_claims", "judge.max_cost_usd", "judge.timeout_s", "judge.model"):
|
||||
assert "null" in issue.message, "上限、模型都说清写 null 是什么意思"
|
||||
|
||||
|
||||
def test_graph_defaults_are_checked_like_at_run_time():
|
||||
[issue] = issues(graph({"claims": "judge"}, defaults={"judge": {"max_claims": -1}}), "report.judge_invalid")
|
||||
assert issue.field == "judge.max_claims"
|
||||
|
||||
|
||||
def test_the_judge_block_only_matters_when_claims_is_judge():
|
||||
assert issues(graph({"claims": "require_citation", "judge": {"max_claims": -1}})) == []
|
||||
|
||||
|
||||
def test_judging_with_the_writing_model_is_a_warning():
|
||||
[warn] = [i for i in validate_graph(GraphSpec.model_validate(graph(
|
||||
{"claims": "judge", "model": "writer-x", "judge": {"model": "writer-x"}}))).issues
|
||||
if i.code == "report.judge_same_model"]
|
||||
assert warn.level == "warning" and warn.field == "judge.model" and "自己审自己" in warn.message
|
||||
assert not [i for i in validate_graph(GraphSpec.model_validate(graph(
|
||||
{"claims": "judge", "model": "writer-x", "judge": {"model": "judge-y"}}))).issues
|
||||
if i.code == "report.judge_same_model"]
|
||||
@@ -1,10 +1,10 @@
|
||||
"""报告节点的结论句策略 claims:off(默认,和以前一样)/ require_citation(没挂依据的结论句计入缺口)。
|
||||
|
||||
结论句要靠裁判模型判断支不支持,那是后续版本的事;这一版只做确定性的一半:
|
||||
这里测确定性的一半(模型裁判 judge 见 test_report_judge.py):
|
||||
- require_citation 时写作提示明确要求每句结论挂 [[see:…]];没挂的照常产出、计数,
|
||||
出口按出具档位降档(io.py),报告节点自己不为它失败、不为它重写
|
||||
- report.checked 事件和节点产出都带上 claims,出口据此判档
|
||||
- 写 judge 时节点直接报错「结论句裁判在后续版本支持」,不能悄悄当成 off
|
||||
- 写了不认识的值节点直接报错,不能悄悄当成 off
|
||||
- 升级前发起的运行一个字都不变
|
||||
"""
|
||||
from __future__ import annotations
|
||||
@@ -21,7 +21,7 @@ from app.db.models import Run, RunEvent
|
||||
from app.engine import runner as runner_mod
|
||||
from app.engine.evidence import CLAIMS_RULE, build_catalog, compose_doc, uncited_claims
|
||||
from app.engine.runner import run_manager
|
||||
from app.engine.schema import CLAIMS_LATER, REPORT_CLAIMS, claims_problem
|
||||
from app.engine.schema import CLAIMS_VALUES, claims_problem
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
@@ -131,14 +131,14 @@ async def test_require_citation_asks_for_support_but_does_not_block(monkeypatch)
|
||||
assert flagged["text"] == "增长主要来自新客。" and doc["markdown"][slice(*flagged["span"])] == flagged["text"]
|
||||
|
||||
|
||||
async def test_judge_is_refused_until_it_exists(monkeypatch):
|
||||
async def test_a_bad_value_is_refused_before_any_model_call(monkeypatch):
|
||||
seen = script(monkeypatch)
|
||||
with pytest.raises(ValueError) as info: # 画布校验先拦(E3-gov 的 validate)
|
||||
await run_manager.start(graph=weekly(claims="judge"), input_payload={})
|
||||
assert CLAIMS_LATER in str(info.value)
|
||||
with pytest.raises(ValueError) as info: # 画布校验先拦
|
||||
await run_manager.start(graph=weekly(claims="sometimes"), input_payload={})
|
||||
assert claims_problem("sometimes") in str(info.value)
|
||||
lenient(monkeypatch)
|
||||
row = await finish(weekly(claims="judge"))
|
||||
assert row.status == "failed" and claims_problem("judge") in (row.error or ""), row.error
|
||||
row = await finish(weekly(claims="sometimes"))
|
||||
assert row.status == "failed" and claims_problem("sometimes") in (row.error or ""), row.error
|
||||
assert seen == [], "不支持的策略不该先花一次模型调用"
|
||||
|
||||
|
||||
@@ -173,7 +173,7 @@ def test_the_supported_policies_match_the_validator():
|
||||
from app.engine.nodes.report import CLAIMS, report_claims
|
||||
from app.engine.schema import GraphSpec
|
||||
|
||||
assert CLAIMS == REPORT_CLAIMS == ("off", "require_citation")
|
||||
assert CLAIMS == CLAIMS_VALUES == ("off", "require_citation", "judge")
|
||||
spec = GraphSpec.model_validate(weekly(claims="require_citation"))
|
||||
assert report_claims(spec, spec.node_map()["write"]) == "require_citation"
|
||||
spec = GraphSpec.model_validate(weekly())
|
||||
|
||||
@@ -0,0 +1,495 @@
|
||||
"""报告撰写节点的 claims: judge:结论句由另一个模型按证据逐句判断(四期)。
|
||||
|
||||
- 正式运行在节点里裁判:写完、核对完之后判,判定写进报告文档的 unit.verdict,随文档一起封存
|
||||
- 探索运行不在节点里花这笔钱:候选句标「未裁判 · 按需」,点开哪句再判哪句(接口在 B 段)
|
||||
- 判定只标注、不改写正文;裁判没跑成、触顶了,照样产出报告,没判的记未裁判并留下缺口
|
||||
(出口据此不能判完整出具,接线在 B 段)
|
||||
- rewrite_once(默认关):把证据不支持的句子和理由交回写作者只改这几句,再判一次,最多一轮;
|
||||
改出了新的违规就不采用
|
||||
- 裁判调用包在 @task 里、追加在节点已有的调用之后:节点「接着跑」时不重复调用、不重复计费;
|
||||
升级前发起的运行一样都不加,停在审批上的运行恢复后照常
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import re
|
||||
|
||||
import pytest
|
||||
from langchain_core.messages import AIMessage
|
||||
from sqlalchemy import delete, select
|
||||
|
||||
from app.core import artifact_store
|
||||
from app.db.base import SessionLocal
|
||||
from app.db.models import Run, RunEvent, Setting
|
||||
from app.engine import judge as judge_mod
|
||||
from app.engine import replay
|
||||
from app.engine import runner as runner_mod
|
||||
from app.engine.evidence import CLAIMS_RULE, iter_units
|
||||
from app.engine.judge import JUDGE_ROLE, JUDGE_RULE, SPEND_KEY, SUMMARY_KEYS, daily_spend
|
||||
from app.engine.nodes import report as report_mod
|
||||
from app.engine.runner import run_manager, verify_manifest
|
||||
from app.providers.mock_model import MockChatModel
|
||||
|
||||
PRICED = "claude-sonnet-5"
|
||||
METRICS = [{"id": "gmv", "name": "销售额", "unit": "元", "decimals": 1, "expression": "vars.kpi.gmv"}]
|
||||
#: 一句挂了依据的结论、一句没挂依据的归因、一句连接性的话
|
||||
TEXT = "本周销售额 [[m:gmv]]。[[see:m:gmv]]增长主要来自新客首单。下面看细节。"
|
||||
CITED, CAUSAL, LINK = "本周销售额 45,678.5元。", "增长主要来自新客首单。", "下面看细节。"
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
async def engine_up():
|
||||
await run_manager.setup()
|
||||
async with SessionLocal() as session:
|
||||
await session.execute(delete(Setting).where(Setting.key.in_([SPEND_KEY, "judge", "copilot"])))
|
||||
await session.commit()
|
||||
yield
|
||||
await run_manager.shutdown()
|
||||
|
||||
|
||||
def node(nid, ntype, **config):
|
||||
return {"id": nid, "type": ntype, "position": {"x": 0, "y": 0}, "data": {"label": nid, "config": config}}
|
||||
|
||||
|
||||
def weekly(*, gate=False, **report):
|
||||
nodes = [
|
||||
node("start", "input", fields=[{"name": "week", "default": "2026-W37"}]),
|
||||
node("fetch", "transform", mode="expression", expression="{'gmv': 45678.5}", assign_to="kpi"),
|
||||
node("caliber", "metrics", caliber="周报口径", caliber_version="v2", metrics=METRICS),
|
||||
*([node("gate", "human", mode="approve", title="确认口径")] if gate else []),
|
||||
node("write", "report", instructions="写周报", **report),
|
||||
node("out", "output", fields=[{"name": "周报", "value": "{{ nodes.write.text }}"}]),
|
||||
]
|
||||
ids = [n["id"] for n in nodes]
|
||||
return {"nodes": nodes, "edges": [{"source": a, "target": b} for a, b in zip(ids, ids[1:])]}
|
||||
|
||||
|
||||
class Models:
|
||||
"""写作者和裁判两份剧本。写作者依次回 writer 里的几段(用完了一直回最后一段);裁判按句子里的字给判定
|
||||
(verdicts:{句中片段: 判定},没对上的都是 supported)。"""
|
||||
|
||||
def __init__(self, monkeypatch, writer=(TEXT,), verdicts=None, *, judge_model=PRICED, fail_judge=None):
|
||||
self.writes: list[list] = []
|
||||
self.judges: list[str] = []
|
||||
verdicts = verdicts or {}
|
||||
|
||||
def decide(model, messages):
|
||||
if JUDGE_ROLE in str(messages[0].content):
|
||||
request = str(messages[-1].content)
|
||||
self.judges.append(request)
|
||||
if fail_judge:
|
||||
raise RuntimeError(fail_judge)
|
||||
items = []
|
||||
for uid, line in re.findall(r"^\[(u\d+)\]([^\n]*)$", request, re.M):
|
||||
status = next((s for frag, s in verdicts.items() if frag in line), "supported")
|
||||
items.append({"unit": uid, "verdict": status, "rationale": f"判的是:{line[:20]}", "used": []})
|
||||
return AIMessage(content=json.dumps({"items": items}, ensure_ascii=False))
|
||||
self.writes.append(list(messages))
|
||||
return AIMessage(content=writer[min(len(self.writes), len(writer)) - 1])
|
||||
|
||||
monkeypatch.setattr(MockChatModel, "_decide", decide)
|
||||
|
||||
async def judge_model_of(session, spec):
|
||||
return MockChatModel(model_name=judge_model), judge_model
|
||||
|
||||
monkeypatch.setattr(judge_mod, "get_chat_model", judge_model_of)
|
||||
|
||||
|
||||
def spy_tasks(monkeypatch) -> list[str]:
|
||||
"""报告节点里每调用一次 @task 记一笔名字:证明加进来的 task 只在该出现的地方出现。"""
|
||||
from langgraph.func import task as real
|
||||
|
||||
calls: list[str] = []
|
||||
|
||||
def spying(func=None, **kwargs):
|
||||
def wrap(fn):
|
||||
inner = real(fn, **kwargs)
|
||||
|
||||
def call(*args, **kw):
|
||||
calls.append(fn.__name__)
|
||||
return inner(*args, **kw)
|
||||
|
||||
return call
|
||||
return wrap(func) if func is not None else wrap
|
||||
|
||||
monkeypatch.setattr(report_mod, "task", spying)
|
||||
return calls
|
||||
|
||||
|
||||
async def finish(graph, *, run_class="formal", statuses=("succeeded", "failed")) -> Run:
|
||||
run = await run_manager.start(graph=graph, input_payload={}, run_class=run_class)
|
||||
return await wait(run.id, statuses)
|
||||
|
||||
|
||||
async def wait(run_id, statuses=("succeeded", "failed")) -> Run:
|
||||
for _ in range(400):
|
||||
await asyncio.sleep(0.05)
|
||||
async with SessionLocal() as session:
|
||||
row = await session.get(Run, run_id)
|
||||
if row.status in statuses and (row.status == "interrupted" or row.manifest_seq is not None):
|
||||
return row
|
||||
raise AssertionError(f"run {run_id} 没跑完:{row.status}")
|
||||
|
||||
|
||||
async def events(run_id: str, etype: str, node_id: str | None = "write") -> list[RunEvent]:
|
||||
async with SessionLocal() as session:
|
||||
q = select(RunEvent).where(RunEvent.run_id == run_id, RunEvent.type == etype)
|
||||
if node_id:
|
||||
q = q.where(RunEvent.node_id == node_id)
|
||||
return list((await session.execute(q.order_by(RunEvent.seq))).scalars())
|
||||
|
||||
|
||||
async def report_of(run_id: str) -> tuple[dict, dict, dict]:
|
||||
"""(report.checked 的载荷, 报告文档, 节点产出)"""
|
||||
[checked] = await events(run_id, "report.checked")
|
||||
[finished] = await events(run_id, "node.finished")
|
||||
return checked.data, artifact_store.load(checked.data["doc_artifact"]), artifact_store.load(finished.data["artifact"])
|
||||
|
||||
|
||||
def verdicts(doc) -> dict[str, dict | None]:
|
||||
md = doc["markdown"]
|
||||
return {md[u["span"][0]:u["span"][1]].strip(): u.get("verdict") for _, u in iter_units(doc)}
|
||||
|
||||
|
||||
def unit_ids(doc) -> dict[str, str]:
|
||||
md = doc["markdown"]
|
||||
return {md[u["span"][0]:u["span"][1]].strip(): u["id"] for _, u in iter_units(doc)}
|
||||
|
||||
|
||||
async def judge_ends(run_id: str) -> list[dict]:
|
||||
return [e.data for e in await events(run_id, "llm.end") if e.data.get("purpose") == "judge"]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 正式运行:节点内裁判,判定随文档封存
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_a_formal_run_judges_inside_the_node_and_seals_the_verdicts(monkeypatch):
|
||||
models = Models(monkeypatch, verdicts={"新客": "unsupported"})
|
||||
row = await finish(weekly(claims="judge", judge={"on_unsupported": "withhold"}))
|
||||
assert row.status == "succeeded", row.error
|
||||
assert len(models.writes) == 1 and len(models.judges) == 1
|
||||
|
||||
checked, doc, output = await report_of(row.id)
|
||||
got = verdicts(doc)
|
||||
assert got[CITED]["status"] == "supported" and got[CITED]["judge"] == PRICED
|
||||
assert got[CITED]["post_seal"] is False and got[CITED]["rationale"]
|
||||
assert got[CAUSAL]["status"] == "unsupported"
|
||||
assert got[LINK] is None, "连接性的话不送裁判"
|
||||
assert CITED in models.judges[0] and CAUSAL in models.judges[0] and LINK not in models.judges[0]
|
||||
|
||||
summary = checked["judge"]
|
||||
assert tuple(summary) == SUMMARY_KEYS, "B 段照着这份键读:不能悄悄少一个、多一个"
|
||||
assert summary["mode"] == "inline" and summary["model"] == PRICED and summary["on_unsupported"] == "withhold"
|
||||
assert summary["counts"] == {"supported": 1, "partial": 0, "unsupported": 1, "not_a_claim": 0, "unjudged": 0}
|
||||
assert summary["complete"] is True and summary["gaps"] == []
|
||||
assert checked["claims"] == output["claims"] == "judge" and output["judge"] == summary
|
||||
assert doc["judge"]["mode"] == "inline" and doc["judge"]["counts"] == summary["counts"]
|
||||
assert {k: doc["stats"][k] for k in ("supported", "partial", "unsupported", "not_a_claim", "unjudged")} \
|
||||
== summary["counts"]
|
||||
|
||||
[end] = await judge_ends(row.id)
|
||||
assert end["model"] == PRICED and end["cost_usd"] >= 0
|
||||
verdict = await verify_manifest(row.id)
|
||||
assert verdict["ok"] is True
|
||||
[event] = await events(row.id, "report.checked")
|
||||
assert event.seq <= row.manifest_seq, "判定在封存范围内"
|
||||
|
||||
|
||||
async def test_the_writer_is_told_that_claims_will_be_judged(monkeypatch):
|
||||
models = Models(monkeypatch)
|
||||
await finish(weekly(claims="judge"))
|
||||
prompt = "\n".join(str(m.content) for m in models.writes[0])
|
||||
assert CLAIMS_RULE in prompt and JUDGE_RULE in prompt
|
||||
|
||||
|
||||
async def test_explore_runs_leave_it_to_the_reader(monkeypatch):
|
||||
"""探索运行默认按需裁判:节点里不调用,候选句标「未裁判 · 按需」,连接性的话什么都不标。"""
|
||||
models = Models(monkeypatch)
|
||||
row = await finish(weekly(claims="judge"), run_class="exploratory")
|
||||
assert row.status == "succeeded", row.error
|
||||
assert models.judges == []
|
||||
checked, doc, output = await report_of(row.id)
|
||||
got = verdicts(doc)
|
||||
for text in (CITED, CAUSAL):
|
||||
assert got[text]["status"] == "unjudged" and got[text]["reason"] == "on_demand"
|
||||
assert got[text]["judge"] is None and got[text]["post_seal"] is False
|
||||
assert got[LINK] is None
|
||||
assert checked["judge"]["mode"] == "on_demand" and checked["judge"]["gaps"] == []
|
||||
assert tuple(checked["judge"]) == SUMMARY_KEYS and checked["judge"]["limits"] is None
|
||||
assert output["judge"] == checked["judge"] and await judge_ends(row.id) == []
|
||||
|
||||
|
||||
async def test_a_failed_judge_leaves_a_gap_not_a_failed_report(monkeypatch):
|
||||
Models(monkeypatch, fail_judge="网关超时")
|
||||
row = await finish(weekly(claims="judge"))
|
||||
assert row.status == "succeeded", row.error
|
||||
checked, doc, output = await report_of(row.id)
|
||||
assert {verdicts(doc)[t]["reason"] for t in (CITED, CAUSAL)} == {"error"}
|
||||
summary = checked["judge"]
|
||||
assert summary["complete"] is False and summary["counts"]["unjudged"] == 2
|
||||
assert summary["gaps"] and "网关超时" in summary["gaps"][0]
|
||||
assert output["judge"]["gaps"] == summary["gaps"]
|
||||
[end] = await judge_ends(row.id)
|
||||
assert "网关超时" in end["error"]
|
||||
|
||||
|
||||
async def test_hitting_a_limit_is_reported_not_raised(monkeypatch):
|
||||
models = Models(monkeypatch)
|
||||
row = await finish(weekly(claims="judge", judge={"max_claims": 1}))
|
||||
assert row.status == "succeeded", row.error
|
||||
checked, doc, _ = await report_of(row.id)
|
||||
assert verdicts(doc)[CITED]["status"] == "supported", "先判挂了依据的"
|
||||
assert (verdicts(doc)[CAUSAL]["status"], verdicts(doc)[CAUSAL]["reason"]) == ("unjudged", "max_claims")
|
||||
assert checked["judge"]["limits_hit"] == ["max_claims"] and "已到上限" in checked["judge"]["gaps"][0]
|
||||
assert CAUSAL not in models.judges[0]
|
||||
|
||||
|
||||
async def test_the_node_budget_overrides_the_settings_and_null_means_unlimited(monkeypatch):
|
||||
"""节点 judge 里写了的键盖过设置;写 null 就是不限(不是「没写」)。"""
|
||||
async with SessionLocal() as session:
|
||||
session.add(Setting(key="judge", value={"report_max_claims": 1, "report_max_cost_usd": 1e-9}))
|
||||
await session.commit()
|
||||
Models(monkeypatch)
|
||||
row = await finish(weekly(claims="judge", judge={"max_cost_usd": None}))
|
||||
checked, doc, _ = await report_of(row.id)
|
||||
assert verdicts(doc)[CITED]["status"] == "supported"
|
||||
assert verdicts(doc)[CAUSAL]["reason"] == "max_claims", "max_claims 没写,取设置里的 1"
|
||||
assert checked["judge"]["limits"]["max_cost_usd"] is None and checked["judge"]["limits"]["max_claims"] == 1
|
||||
|
||||
|
||||
async def test_judging_yourself_is_called_out(monkeypatch):
|
||||
Models(monkeypatch, judge_model="mock-fast")
|
||||
row = await finish(weekly(claims="judge"))
|
||||
assert row.status == "succeeded", row.error
|
||||
[warn] = [e.data for e in await events(row.id, "log") if e.data.get("code") == "judge_same_model"]
|
||||
assert "mock-fast" in warn["message"] and "自己审自己" in warn["message"]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# rewrite_once
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
REWRITTEN = "本周销售额 [[m:gmv]]。[[see:m:gmv]]新客首单的贡献要等新客维度的数据再看。下面看细节。"
|
||||
|
||||
|
||||
async def test_rewrite_once_hands_back_only_the_unsupported_sentences(monkeypatch):
|
||||
models = Models(monkeypatch, writer=(TEXT, REWRITTEN),
|
||||
verdicts={"主要来自新客": "unsupported", "要等新客维度": "not_a_claim"})
|
||||
row = await finish(weekly(claims="judge", judge={"rewrite_once": True}))
|
||||
assert row.status == "succeeded", row.error
|
||||
assert len(models.writes) == 2 and len(models.judges) == 2
|
||||
|
||||
request = str(models.writes[1][-1].content)
|
||||
assert CAUSAL in request and "判的是:" in request and CITED not in request, "只交回不支持的那几句和理由"
|
||||
# 第二次只判改过的句子:没改的那句直接用第一次的判定
|
||||
assert "要等新客维度" in models.judges[1] and CITED not in models.judges[1]
|
||||
|
||||
checked, doc, output = await report_of(row.id)
|
||||
assert "新客首单的贡献要等新客维度的数据再看。" in doc["markdown"] and CAUSAL not in doc["markdown"]
|
||||
got = verdicts(doc)
|
||||
assert got[CITED]["status"] == "supported" and got["新客首单的贡献要等新客维度的数据再看。"]["status"] == "not_a_claim"
|
||||
rewrite = checked["judge"]["rewrite"]
|
||||
assert rewrite["applied"] is True and rewrite["sentences"] == [CAUSAL] and checked["judge"]["reused"] == 1
|
||||
# 封存的是改写稿:units 按句子认封存文档里的编号,交回去的那句已经改掉了,对不上就是 null——
|
||||
# 不能拿原稿的编号去指改写稿里碰巧同号的另一句;改写稿里新出来的句子在 changed 里
|
||||
ids = unit_ids(doc)
|
||||
assert rewrite["units"] == [None]
|
||||
assert rewrite["changed"] == [ids["新客首单的贡献要等新客维度的数据再看。"]]
|
||||
assert output["text"] == doc["markdown"]
|
||||
[log] = [e.data for e in await events(row.id, "log") if e.data.get("code") == "report_rewrite"]
|
||||
assert "1 句" in log["message"]
|
||||
assert len(await judge_ends(row.id)) == 2
|
||||
|
||||
|
||||
async def test_rewrite_once_is_off_by_default(monkeypatch):
|
||||
models = Models(monkeypatch, writer=(TEXT, REWRITTEN), verdicts={"主要来自新客": "unsupported"})
|
||||
row = await finish(weekly(claims="judge"))
|
||||
assert row.status == "succeeded", row.error
|
||||
assert len(models.writes) == 1 and len(models.judges) == 1
|
||||
checked, doc, _ = await report_of(row.id)
|
||||
assert verdicts(doc)[CAUSAL]["status"] == "unsupported" and checked["judge"].get("rewrite") is None
|
||||
|
||||
|
||||
async def test_a_rewrite_that_breaks_the_numbers_is_not_used(monkeypatch):
|
||||
models = Models(monkeypatch, writer=(TEXT, "本周销售额 [[m:gmv]]。[[see:m:gmv]]新客贡献了 45678 元。下面看细节。"),
|
||||
verdicts={"主要来自新客": "unsupported"})
|
||||
row = await finish(weekly(claims="judge", judge={"rewrite_once": True}))
|
||||
assert row.status == "succeeded", row.error
|
||||
assert len(models.writes) == 2 and len(models.judges) == 1, "改坏了的那一稿不再花钱判"
|
||||
checked, doc, output = await report_of(row.id)
|
||||
assert CAUSAL in doc["markdown"] and "45678" not in doc["markdown"]
|
||||
assert verdicts(doc)[CAUSAL]["status"] == "unsupported"
|
||||
rewrite = checked["judge"]["rewrite"]
|
||||
assert rewrite["applied"] is False and "45678" in rewrite["reason"]
|
||||
# 封存的是原稿:units 就是原稿(也就是封存文档)里的编号
|
||||
assert rewrite["units"] == [unit_ids(doc)[CAUSAL]] and rewrite["changed"] == []
|
||||
|
||||
|
||||
def with_schema(monkeypatch):
|
||||
"""报告的目录里加一张表:实体层有了核对的底,反引号里的名字才会被认出是编造的。"""
|
||||
from app.engine.evidence import _entity_entries
|
||||
|
||||
real = report_mod.report_catalog
|
||||
|
||||
def catalog_with_schema(state, spec, node):
|
||||
return {**real(state, spec, node), **_entity_entries(
|
||||
[{"artifact": "d" * 64, "source": "shop", "tables": {"orders": ["order_id", "amount"]}}], [])}
|
||||
|
||||
monkeypatch.setattr(report_mod, "report_catalog", catalog_with_schema)
|
||||
|
||||
|
||||
async def test_a_rewrite_that_makes_up_a_name_is_not_used(monkeypatch):
|
||||
"""改写稿冒出一个编造的表名(只标注、不拦的那类违规):同样是原稿没有的新违规,不采用。"""
|
||||
with_schema(monkeypatch)
|
||||
models = Models(monkeypatch, writer=(TEXT, "本周销售额 [[m:gmv]]。[[see:m:gmv]]新客首单的贡献要看 `refund_detail` 表。"
|
||||
"下面看细节。"),
|
||||
verdicts={"主要来自新客": "unsupported"})
|
||||
row = await finish(weekly(claims="judge", judge={"rewrite_once": True}))
|
||||
assert row.status == "succeeded", row.error
|
||||
assert len(models.writes) == 2 and len(models.judges) == 1
|
||||
checked, doc, _ = await report_of(row.id)
|
||||
assert CAUSAL in doc["markdown"] and "refund_detail" not in doc["markdown"]
|
||||
rewrite = checked["judge"]["rewrite"]
|
||||
assert rewrite["applied"] is False and "refund_detail" in rewrite["reason"]
|
||||
[log] = [e.data for e in await events(row.id, "log") if e.data.get("code") == "report_rewrite_rejected"]
|
||||
assert "refund_detail" in log["message"]
|
||||
|
||||
|
||||
async def test_a_rewrite_that_trades_one_violation_for_another_is_not_used(monkeypatch):
|
||||
"""去掉一处违规、又冒出另一处:总数没变,可冒出来的是原稿没有的,一样不采用。"""
|
||||
draft = "本周销售额 [[m:gmv]]。[[see:m:gmv]]增长主要来自新客首单,共 1200 单。下面看细节。"
|
||||
swapped = "本周销售额 [[m:gmv]]。[[see:m:gmv]]新客首单约占 3500 单。下面看细节。"
|
||||
models = Models(monkeypatch, writer=(draft, draft, swapped), verdicts={"主要来自新客": "unsupported"})
|
||||
row = await finish(weekly(claims="judge", on_violation="flag", max_repairs=1, judge={"rewrite_once": True}))
|
||||
assert row.status == "succeeded", row.error
|
||||
assert len(models.writes) == 3 and len(models.judges) == 1
|
||||
checked, doc, _ = await report_of(row.id)
|
||||
assert "1200" in doc["markdown"] and "3500" not in doc["markdown"]
|
||||
rewrite = checked["judge"]["rewrite"]
|
||||
assert rewrite["applied"] is False and "3500" in rewrite["reason"] and "1200" not in rewrite["reason"]
|
||||
|
||||
|
||||
def test_new_violations_are_counted_one_by_one():
|
||||
"""改写稿的违规逐个和原稿抵消:原样留着的不算新的,同一个裸数字多写一处、只标注的那类,都算新的。"""
|
||||
number = {"code": "uncited_number", "text": "1200", "message": "数字「1200」没有出处"}
|
||||
name = {"code": "unknown_entity", "text": "refund_detail", "message": "可能是编造的名字"}
|
||||
see = {"code": "unresolved_ref", "ref": "Q9", "message": "依据 [[see:Q9]] 解析不了"}
|
||||
assert report_mod._new_violations([number, name, see], [see, number, name]) == []
|
||||
assert report_mod._new_violations([number], [number, number]) == [number]
|
||||
assert report_mod._new_violations([number], [name]) == [name]
|
||||
assert report_mod._new_violations([number, see], []) == []
|
||||
|
||||
|
||||
async def test_a_second_round_that_hits_a_limit_names_the_configured_limit(monkeypatch):
|
||||
"""第二轮用的是第一轮剩下的钱,可说给人看的上限是节点上配的那个数,不是剩下的零头。"""
|
||||
Models(monkeypatch, writer=(TEXT, REWRITTEN), verdicts={"主要来自新客": "unsupported"})
|
||||
# 每句估 1 分钱、回复不带用量(实际就按估算记):第一轮两句花掉 2 分,第二轮只剩半分,一句也装不下
|
||||
monkeypatch.setattr(judge_mod.JudgeRequest, "cost", lambda self, model_id, units: 0.01 * len(units))
|
||||
monkeypatch.setattr(judge_mod, "_usage_of", lambda message, model_id: {"input_tokens": 0, "output_tokens": 0})
|
||||
row = await finish(weekly(claims="judge", judge={"rewrite_once": True, "max_cost_usd": 0.025}))
|
||||
assert row.status == "succeeded", row.error
|
||||
checked, doc, _ = await report_of(row.id)
|
||||
v = verdicts(doc)["新客首单的贡献要等新客维度的数据再看。"]
|
||||
assert (v["status"], v["reason"]) == ("unjudged", "max_cost_usd")
|
||||
assert "$0.025" in v["rationale"] and "0.005" not in v["rationale"], v["rationale"]
|
||||
summary = checked["judge"]
|
||||
assert summary["rewrite"]["applied"] is True and summary["limits"]["max_cost_usd"] == 0.025
|
||||
assert "$0.025" in summary["gaps"][0] and "0.005" not in summary["gaps"][0], summary["gaps"]
|
||||
[log] = [e.data for e in await events(row.id, "log") if e.data.get("code") == "judge_limit"]
|
||||
assert "$0.025" in log["message"] and "0.005" not in log["message"]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 重放安全
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_continuing_after_a_failure_does_not_judge_or_charge_twice(monkeypatch):
|
||||
models = Models(monkeypatch)
|
||||
real = report_mod._store
|
||||
broken = {"on": True}
|
||||
|
||||
async def flaky(doc, ctx):
|
||||
if broken["on"]:
|
||||
raise RuntimeError("落盘时断了")
|
||||
return await real(doc, ctx)
|
||||
|
||||
monkeypatch.setattr(report_mod, "_store", flaky)
|
||||
run = await run_manager.start(graph=weekly(claims="judge"), input_payload={}, run_class="formal")
|
||||
row = await wait(run.id)
|
||||
assert row.status == "failed" and "落盘时断了" in (row.error or ""), row.error
|
||||
assert len(models.judges) == 1
|
||||
before = await daily_spend()
|
||||
await run_manager.wait_idle(run.id)
|
||||
|
||||
broken["on"] = False
|
||||
await run_manager.continue_failed(run.id)
|
||||
row = await wait(run.id)
|
||||
assert row.status == "succeeded", row.error
|
||||
assert len(models.judges) == 1, "接着跑时裁判又调了一次:裁判的结果没进 checkpoint"
|
||||
assert await daily_spend() == before, "接着跑时每日计数又记了一次"
|
||||
assert len(await judge_ends(row.id)) == 1
|
||||
_, doc, _ = await report_of(row.id)
|
||||
assert verdicts(doc)[CITED]["status"] == "supported"
|
||||
|
||||
|
||||
async def test_a_changed_draft_is_judged_afresh_when_continuing(monkeypatch):
|
||||
"""接着跑时写作者重写了一稿(写作调用不进 checkpoint),缓存的判定是上一稿的:不能拿来用。"""
|
||||
models = Models(monkeypatch, writer=(TEXT, "本周销售额 [[m:gmv]]。[[see:m:gmv]]增长主要来自老客复购。下面看细节。"))
|
||||
real = report_mod._store
|
||||
broken = {"on": True}
|
||||
|
||||
async def flaky(doc, ctx):
|
||||
if broken["on"]:
|
||||
raise RuntimeError("落盘时断了")
|
||||
return await real(doc, ctx)
|
||||
|
||||
monkeypatch.setattr(report_mod, "_store", flaky)
|
||||
run = await run_manager.start(graph=weekly(claims="judge"), input_payload={}, run_class="formal")
|
||||
assert (await wait(run.id)).status == "failed"
|
||||
await run_manager.wait_idle(run.id)
|
||||
broken["on"] = False
|
||||
await run_manager.continue_failed(run.id)
|
||||
row = await wait(run.id)
|
||||
assert row.status == "succeeded", row.error
|
||||
assert len(models.judges) == 2 and "老客复购" in models.judges[1]
|
||||
_, doc, _ = await report_of(row.id)
|
||||
assert verdicts(doc)["增长主要来自老客复购。"]["status"] == "supported"
|
||||
|
||||
|
||||
async def test_runs_from_before_the_upgrade_never_judge(monkeypatch):
|
||||
"""升级前发起的运行(没有护栏快照):claims 一律按 off,节点里一个 task 都不多调用,产出一个键都不多。"""
|
||||
async def no_snapshot(session):
|
||||
return None
|
||||
|
||||
monkeypatch.setattr(runner_mod, "_agent_limits", no_snapshot)
|
||||
models = Models(monkeypatch)
|
||||
calls = spy_tasks(monkeypatch)
|
||||
row = await finish(weekly(claims="judge"))
|
||||
assert row.status == "succeeded", row.error
|
||||
assert models.judges == [] and calls == []
|
||||
checked, doc, output = await report_of(row.id)
|
||||
assert "judge" not in checked and "claims" not in checked and "judge" not in output and "judge" not in doc
|
||||
assert all(v is None for v in verdicts(doc).values())
|
||||
|
||||
|
||||
async def test_runs_paused_at_an_approval_resume_with_tasks_in_place(monkeypatch):
|
||||
"""停在审批上的运行恢复:审批节点照旧只问一次;报告节点在恢复之后才第一次执行,裁判 task 只调一次、
|
||||
排在节点所有已有调用之后。没开裁判的报告节点一个 task 都不调用(和升级前一样)。"""
|
||||
assert replay.PROTOCOL == 2, "没有改动已有 task 的顺序,重放协议不升版"
|
||||
for claims, expected in (("require_citation", []), ("judge", ["judge_step"])):
|
||||
models = Models(monkeypatch)
|
||||
calls = spy_tasks(monkeypatch)
|
||||
paused = await finish(weekly(gate=True, claims=claims), statuses=("interrupted", "failed"))
|
||||
assert paused.status == "interrupted", paused.error
|
||||
await run_manager.resume(paused.id, {"approved": True})
|
||||
row = await wait(paused.id)
|
||||
assert row.status == "succeeded", row.error
|
||||
assert calls == expected, (claims, calls)
|
||||
assert len(models.judges) == len(expected)
|
||||
assert len(await events(row.id, "human.requested", "gate")) == 1
|
||||
checked, _, _ = await report_of(row.id)
|
||||
assert ("judge" in checked) is (claims == "judge")
|
||||
Reference in New Issue
Block a user