可点击证据第四期:结论句由模型裁判;只引单元格的问数据图能发成受管

一、结论句裁判

- 报告节点新增 claims: judge:写完、核对完之后,把结论句和它引用的证据摘录交给裁判模型,
  判为支持 / 部分支持 / 不支持 / 不是结论,附一句理由;判定只标注,不改写正文
- 送裁判的句子按固定规则预筛:带引用的、含方向词或因果词的送去判,短句、过渡句不送
- 证据摘录:指标写名称、值和原式;查询给 SQL 前 300 字和被引用的行,被引用的列一定给到;遮罩列一律不给;
  规则里写明「数字已经由系统核对过,不要复算」
- 一份报告合并成一次调用,超出句数就分批;结构化输出失败时退回 JSON 解析,再失败记为未裁判
- 裁判模型的优先级:节点上写的 > 设置里的「证据裁判模型」> Copilot 的模型;思考关闭;
  和写作模型相同时给警告(等于自己审自己)
- rewrite_once(默认关):把不支持的句子和理由交回写作者只改这几句,再裁一次;
  改写稿冒出原稿没有的问题就不采用,保留原稿和原来的判定

二、成本上限

- 每份报告的句数、金额、时长,探索运行每次点击的金额,全局每日金额,每一项都可以自定义,也都可以设成不限;
  默认 40 句、$0.05、30 秒,每次点击 $0.01,每天 $2
- 调用前先估算,超出上限按优先级截断:先判带引用的,再判含方向词、因果词的,其余记为未裁判并写明「已到上限」;
  不报错,也不让报告失败
- 每日花费按本地日期计数,持久化在设置表里;重跑同一次运行不重复计费;价格目录里没有的模型只计次数,并写明估不出金额

三、正式运行与探索运行

- 正式运行在报告节点内裁判,判定写进报告文档、随文档封存;不支持的按 on_unsupported(degrade / withhold)降档或不予出具,
  裁判没跑成记缺口,不能判完整出具
- 探索运行按需裁判:POST /api/runs/{id}/evidence/judge,点开哪句判哪句;判定作为 evidence.judged 事件追加在封存之后,
  标「封存后追加」,封存核对仍然一致;只读封存范围内的工件;同一句不重复判;运行在裁判期间被接着跑时不记录
- 正式运行调这个接口返回 409,写明「正式运行的裁判在节点内完成」

四、治理与自动修复

- G3 放开 judge:受管级别写 judge 必须写预算(judge.max_cost_usd,或显式写 null 表示不限)
- 发布前自动修复:缺预算时给选项(默认金额 / 不限),不替人选;
  从 require_citation 改成 judge 不算降低要求,judge 改回 off、withhold 放宽成 degrade 算降低要求
- Copilot 手册:受管模板可以写 judge 并带预算;问数据按需裁判,不替人打开结论句策略

五、界面

- 结论句句末放徽标:支持 ◆、部分支持 ◇、不支持 ! 整句浅底、未裁判 ?;不用表示「已核对」的绿色;n / N 也跳到不支持的句子
- 证据面板里「模型的解释」:带「模型判断 · 模型名 · 非确定」,封存后追加的另外标出;
  「请模型判断这句」只在探索运行、还没判过时出现,触顶时写明已到上限和怎么调
- 出具横幅多一段结论句计数;设置页新增「证据裁判」一组(裁判模型、各项上限、今日花费),设成不限时写明费用受什么约束;
  报告节点可以配 judge;运行轨迹里裁判相关的事件都有中文标题

六、门禁修正

- 引用模式下,图里没有口径卡(只引查询单元格)的,出具契约不再要求 metrics_from,受管门禁也不再要求 required:
  strict 加 cells 已经保证每个数都点得开、对不上就不予出具;有口径卡的照旧要求。
  以前问数据的图怎么都发不成受管,Copilot 补上契约也会被判「冒出新的问题」作废
This commit is contained in:
ThearchyHelios
2026-09-29 11:10:53 +08:00
parent d071bf119f
commit 9727c21d8d
47 changed files with 12367 additions and 263 deletions
+72 -17
View File
@@ -237,6 +237,7 @@ def _apply_contract(
strict=bool(contract.get("strict")),
gaps=gaps,
unresolved=cited.get("unresolved"),
unsupported=cited.get("unsupported"),
uncited_claims=cited.get("uncited"),
claims_policy=cited.get("claims_policy"),
)
@@ -298,7 +299,7 @@ def _check_citations(
reports: list[dict[str, Any]], *, cells: bool = True, governed: bool = False,
) -> dict[str, Any]:
"""独立复核报告文档:{matched, unmatched, unresolved, gaps, doc_artifact, stats, claims_policy, uncited,
claims, entities}。
unsupported, claims, entities}。
不信报告节点自己的统计:文档按 id 从工件库取回(取回时复验哈希),目录按报告节点
同一套参数从状态里重建(不用文档里存的那份),每个带引用的片段重新解析、重新渲染、
@@ -313,7 +314,7 @@ def _check_citations(
out: dict[str, Any] = {"matched": [], "unmatched": [], "unresolved": [], "gaps": [],
"doc_artifact": None, "stats": None, "claims_policy": None, "uncited": None,
"claims": None, "entities": None}
"unsupported": None, "claims": None, "entities": None}
gaps = out["gaps"]
spec = ctx.run.spec
node = spec.node_map().get(report_from)
@@ -403,7 +404,7 @@ def _check_citations(
"span": [start, end], "eid": cite["eid"]})
_claims(out, contract, payload, evidence_on=ledger_enabled(ctx.run), governed=governed,
uncited=checked.get("uncited") or [])
uncited=checked.get("uncited") or [], doc=doc, title=node.title)
mine = next((r for r in reports if r["node"].id == report_from), None)
edited = mine["edited"] if mine else []
@@ -416,20 +417,29 @@ def _check_citations(
#: 没挂依据的结论句怎么处置,从松到严。认不出的写法按 degrade 算:写错一个词不能让缺口悄悄消失
_ON_UNCITED = ("ignore", "degrade", "withhold")
#: 证据不支持的结论句怎么处置(claims: judge),从松到严。认不出的写法同样按 degrade 算
_ON_UNSUPPORTED = ("degrade", "withhold")
#: 要求结论句挂依据的策略:judge 在这一点上和 require_citation 一样严,另外再按裁判的判定判档
_CITING = ("require_citation", "judge")
def _claims(out: dict[str, Any], contract: dict[str, Any], payload: dict[str, Any], *, evidence_on: bool,
governed: bool = False, uncited: list[dict[str, Any]]) -> None:
"""结论句策略:没挂依据的结论句按 claims_policy 交给 decide_tier 判档。
governed: bool = False, uncited: list[dict[str, Any]], doc: dict[str, Any] | None = None,
title: str = "") -> None:
"""结论句策略:没挂依据的、裁判判为证据不支持的结论句按 claims_policy 交给 decide_tier 判档。
两处可以要求挂依据:报告节点写作时记下的 claims(节点产出里的,写作提示按它要求过),和契约自己写的
require_citation(对象形式可以另写 on_uncited)。两处都写了取更严的那个——契约只能收紧、不能放松:
require_citation / judge(对象形式可以另写 on_uncited)。两处都写了取更严的那个——契约只能收紧、不能放松:
报告节点要求了,契约写 ignore 或 off 也照旧计入缺口,写 withhold 就收紧成不予出具。受管级别的正式运行
里 ignore 一律当 degrade:没有哪道门禁查契约里的 claims,不能让它把用户要的底线拉低。
报告节点写的是 judge 时,判定在文档里(按哈希取回的那份:判定是模型给的,出口复算不了,只能信封存的
文档)。正式运行在节点里判过:证据不支持的按 on_unsupported 判档(报告节点和契约取更严的),没判完的
(裁判没跑成、触顶)把裁判摘要里的缺口照抄进来——不能判完整出具,也不当成「不支持」。探索运行按需裁判,
这里只标注。契约要 judge、报告节点却没开的,照实记缺口:出口替不了报告节点去裁判。
数的是这里重新核对出来的 uncited,不信文档里记的 cites。升级前写的报告(节点产出里没有 claims 这个键,
包括跨着升级还没跑完的运行)一律不管,契约写了也照旧记「这一版还不支持」。judge(裁判模型)是后续版本
的事,契约写了照实记缺口。
包括跨着升级还没跑完的运行)一律不管,契约写了也照旧记「这一版还不支持」。
"""
written = payload.get("claims") if isinstance(payload, dict) else None
# 报告节点升级后才在产出里记 claims(off 也记):没有这个键就是升级前写的报告
@@ -437,23 +447,66 @@ def _claims(out: dict[str, Any], contract: dict[str, Any], payload: dict[str, An
declared = contract.get("claims")
name = declared.get("policy") if isinstance(declared, dict) else declared
asks: list[str] = []
if upgraded and written == "require_citation":
if upgraded and written in _CITING:
asks.append("degrade")
if upgraded and name == "require_citation":
if upgraded and name in _CITING:
wanted = declared.get("on_uncited") if isinstance(declared, dict) else None
asks.append(wanted if wanted in _ON_UNCITED else "degrade")
elif name not in (None, "", "off"):
out["gaps"].append("契约声明了结论句检查(claims),这一版还不支持,结论句没有核对")
judged = upgraded and written == "judge"
if upgraded and name == "judge" and not judged:
out["gaps"].append(f"契约要求结论句由模型裁判(claims: judge),报告「{title}」没有开 claims: judge,"
"结论句没有裁判")
if not asks:
return
on_uncited = max(asks, key=_ON_UNCITED.index)
if governed and on_uncited == "ignore":
on_uncited = "degrade"
policy: Any = "require_citation" if on_uncited == "degrade" else \
{"policy": "require_citation", "on_uncited": on_uncited}
out["claims_policy"], out["uncited"] = policy, uncited
out["claims"] = {"policy": "require_citation", "uncited_claims": len(uncited), "uncited": uncited[:_LISTED],
**({"on_uncited": on_uncited} if on_uncited != "degrade" else {})}
extra = {"on_uncited": on_uncited} if on_uncited != "degrade" else {}
if not judged:
policy: Any = "require_citation" if on_uncited == "degrade" else \
{"policy": "require_citation", "on_uncited": on_uncited}
out["claims_policy"], out["uncited"] = policy, uncited
out["claims"] = {"policy": "require_citation", "uncited_claims": len(uncited), "uncited": uncited[:_LISTED],
**extra}
return
from app.engine.evidence import iter_units
summary = (doc or {}).get("judge")
if not isinstance(summary, dict):
# 报告节点开了 judge、文档里却没有裁判摘要(不该发生):当作没判,不能判完整出具
summary = {"mode": "inline", "counts": {}, "unjudged": {}, "limits_hit": [], "complete": False,
"gaps": ["开了 claims: judge,文档里却没有裁判结果,结论句没有裁判"]}
asked = [summary.get("on_unsupported")]
if name == "judge" and isinstance(declared, dict):
asked.append(declared.get("on_unsupported"))
on_unsupported = max((a if a in _ON_UNSUPPORTED else "degrade" for a in asked), key=_ON_UNSUPPORTED.index)
inline = summary.get("mode") == "inline"
flagged: dict[str, list[dict[str, Any]]] = {"unsupported": [], "partial": []}
if inline:
# 探索运行按需裁判的判定不在文档里(在封存之后追加的 evidence.judged 事件里),这里只数正式运行的
markdown = str((doc or {}).get("markdown") or "")
for _, unit in iter_units(doc or {}):
verdict = unit.get("verdict")
status = verdict.get("status") if isinstance(verdict, dict) else None
if status in flagged:
span = unit.get("span") or [0, 0]
flagged[status].append({"unit": unit.get("id"), "span": span, "text": markdown[span[0]:span[1]],
"rationale": verdict.get("rationale") or ""})
out["gaps"].extend(f"报告「{title}」{gap}" for gap in summary.get("gaps") or [])
counts = {k: int((summary.get("counts") or {}).get(k) or 0)
for k in ("supported", "partial", "unsupported", "not_a_claim", "unjudged")}
out["claims_policy"] = {"policy": "judge", "on_unsupported": on_unsupported, "on_uncited": on_uncited}
out["uncited"], out["unsupported"] = uncited, flagged["unsupported"]
out["claims"] = {
"policy": "judge", "uncited_claims": len(uncited), "uncited": uncited[:_LISTED], **extra,
"on_unsupported": on_unsupported, "mode": "inline" if inline else "on_demand", "counts": counts,
"unsupported": flagged["unsupported"][:_LISTED], "partial": flagged["partial"][:_LISTED],
"unjudged": dict(summary.get("unjudged") or {}), "limits_hit": list(summary.get("limits_hit") or []),
"complete": bool(summary.get("complete")), "model": summary.get("model"),
}
def _declare_citations(
@@ -497,8 +550,10 @@ def _declare_citations(
metrics_checked=len(metrics),
matched_numbers=len(cited["matched"]),
matched=cited["matched"][:50],
# on_uncited 只在不是默认的 degrade 时才有(收紧成 withhold、探索运行的契约写了 ignore)
**({"claims": {k: cited["claims"][k] for k in ("policy", "uncited_claims", "on_uncited")
# on_uncited 只在不是默认的 degrade 时才有(收紧成 withhold、探索运行的契约写了 ignore);
# judge 另带不支持时怎么判档、在哪判的和各判定的句数(名单只在出具声明里)
**({"claims": {k: cited["claims"][k] for k in ("policy", "uncited_claims", "on_uncited", "on_unsupported",
"mode", "counts")
if k in cited["claims"]}} if cited.get("claims") else {}),
# 计数取复核的统计(声明里的名单最多列 _LISTED 个)
**({"entities": {"unknown": (cited["stats"] or {}).get("unknown_entities", len(entities["unknown"])),
+224 -11
View File
@@ -15,20 +15,32 @@
写了个哪里都找不到的名字)只标注:不重写、不失败,受管级别的正式出具按缺口降档;表结构快照
不全时核对不了的名字(unverified_entity)只标注。entities: off 整层不管表名、字段名。
结论句策略 claims:off(默认)不管;require_citation 在写作提示里要求结论句挂依据(自动链接的
名字不算依据),没挂的计入缺口、出口降档;judge(模型裁判)是后续版本的事,现在写了直接报错。
名字不算依据),没挂的计入缺口、出口降档;judge 另外请一个模型按证据逐句判断(engine/judge.py):
- 正式运行在节点里判:写完、核对完之后判,判定写进文档的 unit.verdict,随文档一起封存。判定只标注、
不改写正文;裁判没跑成、触顶了照样产出,没判的记未裁判并留下缺口,出口按 on_unsupported 判档
- 探索运行不在节点里花这笔钱:候选句标「未裁判 · 按需」,点开哪句再判哪句(证据接口)
- rewrite_once(默认关):证据不支持的句子连同理由交回写作者只改这几句,再判一次,最多一轮;
改写稿冒出原稿没有的违规(只标注的可疑实体也算)就不采用
- 裁判调用包在 @task 里,排在节点所有已有调用之后:「接着跑」时直接取 checkpoint 里的判定,
不重复调用、不重复计费。写作调用不进 checkpoint,接着跑时会重写一稿——所以判定带着输入指纹,
对不上这一稿的不用
出口契约复核(io.py)用 report_catalog 按同一套参数重建目录、独立重算,不信这里的自查。
"""
from __future__ import annotations
import time
from collections import Counter
from typing import Any
from langchain_core.messages import AIMessage, BaseMessage, HumanMessage, SystemMessage
from langgraph.func import task
from app.core.errors import describe_exception
from app.core.events import EventType
from app.db.base import SessionLocal
from app.engine import judge as judging
from app.engine.context import NodeContext, NodeError
from app.engine.evidence import (
CELLS_REASON,
@@ -43,7 +55,17 @@ from app.engine.evidence import (
ledger_enabled,
)
from app.engine.nodes.llm import _invoke_streaming, _model_spec, _report_call
from app.engine.schema import REPORT_CLAIMS, GraphNode, GraphSpec, NodeType, _ancestors, claims_problem
from app.engine.schema import (
CLAIMS_JUDGE,
CLAIMS_VALUES,
JUDGE_ON_UNSUPPORTED,
GraphNode,
GraphSpec,
NodeType,
_ancestors,
claims_problem,
judge_problems,
)
from app.engine.state import GraphState, message_text, thinking_text
from app.engine.toolcalls import leaked_markup, markup_warning
from app.providers.factory import ProviderNotConfigured, get_chat_model
@@ -52,8 +74,8 @@ ON_VIOLATION = ("fail", "flag")
NUMBERS = ("strict", "off")
#: 实体层:link(默认)核对表名、字段名并自动链接;off 整层不管(目录里不收表和字段)
ENTITIES = ("link", "off")
#: 结论句策略,和画布校验同一份(schema.REPORT_CLAIMS);judge 是后续版本的事
CLAIMS = REPORT_CLAIMS
#: 结论句策略,和画布校验同一份(schema.CLAIMS_VALUES):off / require_citation / judge
CLAIMS = CLAIMS_VALUES
#: 只标注、不让写作者重写也不判失败的违规:可疑实体(受管的正式出具按缺口降档)、表结构快照不全时
#: 核对不了的名字(出口只标注)
SOFT = frozenset({"unknown_entity", "unverified_entity"})
@@ -67,6 +89,8 @@ MARKUP_NUDGE = ("你上一条回复把工具调用写成了文字。报告撰写
"证据目录里,直接用引用标记写报告正文。")
MARKUP_ERROR = ("模型输出了工具调用的原始标记,而报告撰写节点不调用工具——数据已经在证据目录里。"
"换一个模型再试")
#: 接着跑时 checkpoint 里缓存的判定最多翻几条去找这一稿的(每条都是先前某一稿的,对不上就往后翻)
_STALE_LIMIT = 16
# --------------------------------------------------------------------------
@@ -120,14 +144,26 @@ def report_entities(spec: GraphSpec, node: GraphNode) -> str:
def report_claims(spec: GraphSpec, node: GraphNode) -> str:
"""报告节点的结论句策略:off / require_citation。出口判档按它(和节点执行时同一个取值规则)。
"""报告节点的结论句策略:off / require_citation / judge。出口判档按它(和节点执行时同一个取值规则)。
写错的值(包括 judge)画布校验就拦了,执行时节点也会报错,这里只把没写当成 off。
写错的值画布校验就拦了,执行时节点也会报错,这里只把没写当成 off。
"""
value = _setting(spec, node, "claims", "off")
return value if isinstance(value, str) and value in CLAIMS else "off"
def report_judge(spec: GraphSpec, node: GraphNode) -> dict[str, Any]:
"""claims: judge 的子配置:节点上的 judge(没写取图级 defaults),写了的键原样保留——上限写 null 就是
不限,没写的键运行时取设置里的默认(judge.report_budget)。rewrite_once、on_unsupported 补上缺省
(false、degrade):出口判档读 on_unsupported。写错的值画布校验就拦了,执行时节点也会报错。
"""
raw = _setting(spec, node, "judge")
raw = raw if isinstance(raw, dict) else {}
return {**raw, "rewrite_once": raw.get("rewrite_once") is True,
"on_unsupported": raw.get("on_unsupported") if raw.get("on_unsupported") in JUDGE_ON_UNSUPPORTED
else "degrade"}
def report_allowance(spec: GraphSpec, node: GraphNode) -> list[Any]:
"""报告自查放行的字面量:出口契约 report_from 指向它时,契约的 allow_numbers。
@@ -209,13 +245,18 @@ class _RenderedStream:
self._ctx.emit(EventType.LLM_TOKEN, delta=tail)
async def _default_on_violation(run_id: str) -> str:
"""探索运行默认标注、照常产出;正式运行默认失败——封存的发布版不能带着裸数字出具。"""
async def _run_class(run_id: str) -> str | None:
"""按运行记录上的 run_class 取:子图、审批之后续跑都是另起的执行上下文,只有运行记录靠得住。"""
from app.db.models import Run
async with SessionLocal() as session:
run = await session.get(Run, run_id)
return "fail" if run is not None and run.run_class == "formal" else "flag"
return run.run_class if run is not None else None
async def _default_on_violation(run_id: str) -> str:
"""探索运行默认标注、照常产出;正式运行默认失败——封存的发布版不能带着裸数字出具。"""
return "fail" if await _run_class(run_id) == "formal" else "flag"
def _options(ctx: NodeContext) -> tuple[str, int]:
@@ -225,6 +266,8 @@ def _options(ctx: NodeContext) -> tuple[str, int]:
# 画布校验也拦这两项;执行时再守一道,报的是同一句话,免得写错的值被悄悄当成默认
if problem := claims_problem(ctx.cfg("claims")):
raise NodeError(ctx.node.id, problem)
if ctx.cfg("claims") == CLAIMS_JUDGE and (problems := judge_problems(ctx.cfg("judge"))):
raise NodeError(ctx.node.id, ";".join(message for _, message in problems))
entities = ctx.cfg("entities", "link") or "link"
if entities not in ENTITIES:
raise NodeError(ctx.node.id, f"entities 只能是 {' / '.join(ENTITIES)},写的是 {entities!r}")
@@ -248,6 +291,29 @@ def _blocking(violations: list[dict[str, Any]], numbers: str) -> list[dict[str,
and not (numbers == "off" and v.get("code") == "uncited_number")]
def _violation_key(violation: dict[str, Any]) -> tuple[str, str]:
"""认一处违规:(code, 点名的字)。点名的字按 text、ref、message 的顺序取第一个有的。"""
named = violation.get("text") or violation.get("ref") or violation.get("message") or ""
return str(violation.get("code") or ""), str(named)
def _new_violations(before: list[dict[str, Any]], after: list[dict[str, Any]]) -> list[dict[str, Any]]:
"""after 里 before 没有的违规,按 (code, 点名的字) 逐个抵消(同一个裸数字多写一处也算新的)。
全部违规都比,包括只标注、不让重写的那几类(SOFT):改写稿冒出一个编造的表名,受管的正式出具就会
因此降档;去掉一处、又冒出另一处,总数没变,冒出来的也还是新的。
"""
left = Counter(_violation_key(v) for v in before)
fresh: list[dict[str, Any]] = []
for violation in after:
key = _violation_key(violation)
if left[key] > 0:
left[key] -= 1
else:
fresh.append(violation)
return fresh
def _named(violations: list[dict[str, Any]], limit: int = 3) -> str:
"""违规里点名的那几个字:「45678」「m:nope」。"""
names = [str(v.get("text") or v.get("ref") or "") for v in violations]
@@ -269,7 +335,10 @@ def _messages(ctx: NodeContext, state: GraphState, catalog: dict[str, Any], cell
system = ctx.render_str(ctx.cfg("system", ""), state)
instructions = ctx.render_str(ctx.cfg("instructions", ""), state).strip() \
or "根据下面的证据写一份简洁的报告,先总后分。"
rules = f"{MARKER_RULES}\n{CLAIMS_RULE}" if claims == "require_citation" else MARKER_RULES
rules = MARKER_RULES
if claims in ("require_citation", CLAIMS_JUDGE):
# 裁判只看写作者挂的依据:要判,就更得每句都挂上
rules = f"{rules}\n{CLAIMS_RULE}" + (f"\n{judging.JUDGE_RULE}" if claims == CLAIMS_JUDGE else "")
return [
SystemMessage(content="\n\n".join(p for p in (system, ROLE) if p)),
HumanMessage(content=(
@@ -362,11 +431,26 @@ async def run_report(state: GraphState, ctx: NodeContext) -> dict[str, Any]:
response, raw = await draft(messages)
doc = compose(raw)
judgement: dict[str, Any] | None = None
if claims == CLAIMS_JUDGE:
judge_cfg = report_judge(spec, ctx.node)
if await _run_class(ctx.run.run_id) != "formal":
# 探索运行默认按需裁判:点开哪句才判哪句,节点里不花这笔钱
judgement = judging.on_demand(doc, on_unsupported=judge_cfg["on_unsupported"],
rewrite_once=judge_cfg["rewrite_once"])
elif not (blocking and on_violation == "fail"):
# 反正要判失败的报告不值得再花钱判(节点紧接着就报错)
doc, response, blocking, judgement, usages = await _judge_inline(
ctx, doc, catalog, judge_cfg, model_id=model_id, messages=[*messages, response],
draft=draft, compose=compose, numbers=numbers, blocking=blocking)
spent.extend(usages)
doc_artifact = await _store(doc, ctx)
ctx.emit(EventType.REPORT_CHECKED, doc_artifact=doc_artifact, ok=not doc["violations"],
stats=doc["stats"], violations=doc["violations"][:20], repairs=repairs,
on_violation=on_violation, failed=bool(blocking) and on_violation == "fail",
**({"claims": claims} if evidence_on else {}))
**({"claims": claims} if evidence_on else {}),
**({"judge": judgement} if judgement is not None else {}))
if blocking and on_violation == "fail":
head = ";".join(v["message"] for v in blocking[:5])
more = f";…另有 {len(blocking) - 5} 处" if len(blocking) > 5 else ""
@@ -388,6 +472,7 @@ async def run_report(state: GraphState, ctx: NodeContext) -> dict[str, Any]:
"model": model_id,
"duration_ms": int((time.perf_counter() - started) * 1000),
**({"claims": claims} if evidence_on else {}),
**({"judge": judgement} if judgement is not None else {}),
}
updates: dict[str, Any] = {"nodes": {ctx.node.id: result}, "usage": usage}
if ctx.cfg("emit_message", True):
@@ -396,3 +481,131 @@ async def run_report(state: GraphState, ctx: NodeContext) -> dict[str, Any]:
if var_name:
updates["vars"] = {var_name: text}
return updates
# --------------------------------------------------------------------------
# claims: judge——正式运行在节点里裁判
# --------------------------------------------------------------------------
def _rewrite_request(sentences: list[tuple[str, str]]) -> str:
listed = "\n".join(f"{i}.「{text}」——{why or '证据不支持这句话'}" for i, (text, why) in enumerate(sentences, 1))
return (
"另一个模型按证据逐句核对了你的报告,下面这几句它认为证据不支持:\n"
f"{listed}\n\n"
"请只改这几句:补上证据目录里真有的依据 [[see:…]],或者把说法收窄到证据能支持的程度,或者删掉;"
"其余部分一字不改。数字仍然只能用引用标记写。不要解释改了什么,只输出改好的整篇报告正文。"
)
def _merge(first: dict[str, Any], second: dict[str, Any], budget: judging.Budget) -> dict[str, Any]:
"""改写前后两轮裁判合成一份:判定、句数、缺口按改写后这一稿,花费、调用、用时两轮相加。"""
merged = dict(second)
for key in ("cost_usd", "estimated_usd"):
merged[key] = round(first[key] + second[key], 6)
for key in ("calls", "duration_ms"):
merged[key] = first[key] + second[key]
merged["notes"] = list(dict.fromkeys([*first["notes"], *second["notes"]]))
merged["budget"] = budget.as_dict()
merged["model"] = second.get("model") or first.get("model")
merged["priced"] = first.get("priced")
return merged
async def _judge_inline(ctx: NodeContext, doc: dict[str, Any], catalog: dict[str, Any], judge_cfg: dict[str, Any], *,
model_id: str, messages: list[BaseMessage], draft: Any, compose: Any, numbers: str,
blocking: list[dict[str, Any]]
) -> tuple[dict[str, Any], BaseMessage, list[dict[str, Any]], dict[str, Any], list[dict[str, Any]]]:
"""写完、核对完之后请裁判判一遍,判定写进文档。返回 (文档, 写作者最后一稿, 要拦的违规, 裁判摘要, 用量)。
messages 是写作的对话、最后一条是写作者这一稿。rewrite_once 时把证据不支持的句子交回去只改这几句,
再判一次(没改的句子直接用第一次的判定),最多一轮;改写稿冒出原稿没有的违规就不采用,保留原稿。
摘要里的 rewrite = {units, sentences, applied, reason, changed}:
- sentences:交回去的句子(原稿里的原文)
- units:这几句在封存文档里的编号。没采用时封存的是原稿,就是原稿的编号;采用了封存的是改写稿,
编号按改写稿重排,按原文认:原句原样留在改写稿里的给新编号,改掉了的是 null
- changed:采用时改写稿里新出来的句子(原稿里没有这段原文)在封存文档里的编号;没采用是 []
"""
from app.engine.replay import _in_graph
async with SessionLocal() as session:
settings = await judging.judge_settings(session)
judge_spec = await judging.judge_model_spec(session, judge_cfg, settings=settings)
budget = judging.report_budget(settings, judge_cfg)
masks = await judging.source_masks(catalog)
@task
async def judge_step(request: judging.JudgeRequest, key: str, limits: judging.Budget, known: dict[str, Any],
first: bool) -> dict[str, Any]:
# 裁判进 checkpoint:「接着跑」时直接取这里的判定,不再调用、不再计费(每日计数也记在这里面)。
# 事件都在这里面发:取缓存时不会再发一遍。第二轮 limits 是剩下的钱和时间,说给人看的仍是配置的上限
outcome = await judging.run_request(request, spec=judge_spec, budget=limits, known=known, emit=ctx.emit,
shown=budget)
if first and outcome.get("model") and outcome["model"] == model_id:
outcome["same_model"] = True
ctx.emit(EventType.LOG, level="warn", code="judge_same_model",
message=f"裁判模型和写作模型都是「{model_id}」:等于自己审自己,模型写错的地方它多半也看不出来。"
"到设置里选一个不同的「证据裁判模型」,或者在节点的 judge.model 里指定")
return {**outcome, "key": key}
async def judge_round(round_doc: dict[str, Any], limits: judging.Budget, known: dict[str, Any],
first: bool) -> dict[str, Any]:
request = judging.prepare(round_doc, catalog, masked=masks)
key = judging.request_key(request, judge_spec, limits, known)
if _in_graph():
# 写作不进 checkpoint,接着跑时写作者会重写一稿:缓存里先前那一稿的判定对不上指纹,往后翻
for _ in range(_STALE_LIMIT):
outcome = await judge_step(request, key, limits, known, first)
if outcome.get("key") == key:
return outcome
return await judging.run_request(request, spec=judge_spec, budget=limits, known=known, emit=ctx.emit,
shown=budget)
outcome = await judge_round(doc, budget, {}, True)
same_model = bool(outcome.get("same_model"))
usages = [outcome["usage"]]
response = messages[-1]
rewrite: dict[str, Any] | None = None
bad = [uid for uid, v in outcome["verdicts"].items() if v["status"] == "unsupported"]
if judge_cfg["rewrite_once"] and bad:
texts = judging.unit_texts(doc)
sentences = [(texts.get(uid, ""), outcome["verdicts"][uid].get("rationale") or "") for uid in bad]
ctx.emit(EventType.LOG, level="warn", code="report_rewrite",
message=f"裁判认为 {len(bad)} 句结论证据不支持({_named([{'text': t} for t, _ in sentences])}),"
"已交回写作者只改这几句")
again, raw = await draft([*messages, HumanMessage(content=_rewrite_request(sentences))])
rewritten = compose(raw)
worse = _blocking(rewritten["violations"], numbers)
fresh = _new_violations(doc["violations"], rewritten["violations"])
rewrite = {"units": bad, "sentences": [t for t, _ in sentences], "applied": False, "reason": None,
"changed": []}
if fresh:
rewrite["reason"] = (f"改写稿冒出 {len(fresh)} 处原稿没有的问题({_named(fresh)}),没有采用,"
"保留原稿和原来的判定")
ctx.emit(EventType.LOG, level="warn", code="report_rewrite_rejected", message=rewrite["reason"])
else:
known = {outcome["keys"][u]: v for u, v in outcome["verdicts"].items() if v["status"] in judging.JUDGED}
# 第二轮用第一轮剩下的钱和时间:每份报告的上限管的是整份报告
left = judging.Budget(
max_claims=budget.max_claims,
max_cost_usd=None if budget.max_cost_usd is None else max(0.0, budget.max_cost_usd - outcome["cost_usd"]),
timeout_s=None if budget.timeout_s is None else max(0.0, budget.timeout_s - outcome["duration_ms"] / 1000),
daily_max_usd=budget.daily_max_usd)
second = await judge_round(rewritten, left, known, False)
usages.append(second["usage"])
outcome = _merge(outcome, second, budget)
doc, response, blocking = rewritten, again, worse
# 封存的是改写稿,编号按它重排:按原文认回去,改掉了的句子在改写稿里没有编号
now = judging.unit_texts(rewritten)
at: dict[str, str] = {}
for uid, text in now.items():
at.setdefault(text, uid)
before = set(texts.values())
rewrite.update(applied=True, units=[at.get(texts.get(uid, "")) for uid in bad],
changed=[uid for uid, text in now.items() if text not in before])
summary = judging.summarize(outcome, mode="inline", on_unsupported=judge_cfg["on_unsupported"],
rewrite_once=judge_cfg["rewrite_once"], same_model=same_model, rewrite=rewrite)
judging.apply_judgement(doc, outcome, summary)
return doc, response, blocking, summary, usages