mirror of
https://github.com/ThearchyHelios/AgentLab.git
synced 2026-11-21 22:12:00 +00:00
可点击证据第四期:结论句由模型裁判;只引单元格的问数据图能发成受管
一、结论句裁判
- 报告节点新增 claims: judge:写完、核对完之后,把结论句和它引用的证据摘录交给裁判模型,
判为支持 / 部分支持 / 不支持 / 不是结论,附一句理由;判定只标注,不改写正文
- 送裁判的句子按固定规则预筛:带引用的、含方向词或因果词的送去判,短句、过渡句不送
- 证据摘录:指标写名称、值和原式;查询给 SQL 前 300 字和被引用的行,被引用的列一定给到;遮罩列一律不给;
规则里写明「数字已经由系统核对过,不要复算」
- 一份报告合并成一次调用,超出句数就分批;结构化输出失败时退回 JSON 解析,再失败记为未裁判
- 裁判模型的优先级:节点上写的 > 设置里的「证据裁判模型」> Copilot 的模型;思考关闭;
和写作模型相同时给警告(等于自己审自己)
- rewrite_once(默认关):把不支持的句子和理由交回写作者只改这几句,再裁一次;
改写稿冒出原稿没有的问题就不采用,保留原稿和原来的判定
二、成本上限
- 每份报告的句数、金额、时长,探索运行每次点击的金额,全局每日金额,每一项都可以自定义,也都可以设成不限;
默认 40 句、$0.05、30 秒,每次点击 $0.01,每天 $2
- 调用前先估算,超出上限按优先级截断:先判带引用的,再判含方向词、因果词的,其余记为未裁判并写明「已到上限」;
不报错,也不让报告失败
- 每日花费按本地日期计数,持久化在设置表里;重跑同一次运行不重复计费;价格目录里没有的模型只计次数,并写明估不出金额
三、正式运行与探索运行
- 正式运行在报告节点内裁判,判定写进报告文档、随文档封存;不支持的按 on_unsupported(degrade / withhold)降档或不予出具,
裁判没跑成记缺口,不能判完整出具
- 探索运行按需裁判:POST /api/runs/{id}/evidence/judge,点开哪句判哪句;判定作为 evidence.judged 事件追加在封存之后,
标「封存后追加」,封存核对仍然一致;只读封存范围内的工件;同一句不重复判;运行在裁判期间被接着跑时不记录
- 正式运行调这个接口返回 409,写明「正式运行的裁判在节点内完成」
四、治理与自动修复
- G3 放开 judge:受管级别写 judge 必须写预算(judge.max_cost_usd,或显式写 null 表示不限)
- 发布前自动修复:缺预算时给选项(默认金额 / 不限),不替人选;
从 require_citation 改成 judge 不算降低要求,judge 改回 off、withhold 放宽成 degrade 算降低要求
- Copilot 手册:受管模板可以写 judge 并带预算;问数据按需裁判,不替人打开结论句策略
五、界面
- 结论句句末放徽标:支持 ◆、部分支持 ◇、不支持 ! 整句浅底、未裁判 ?;不用表示「已核对」的绿色;n / N 也跳到不支持的句子
- 证据面板里「模型的解释」:带「模型判断 · 模型名 · 非确定」,封存后追加的另外标出;
「请模型判断这句」只在探索运行、还没判过时出现,触顶时写明已到上限和怎么调
- 出具横幅多一段结论句计数;设置页新增「证据裁判」一组(裁判模型、各项上限、今日花费),设成不限时写明费用受什么约束;
报告节点可以配 judge;运行轨迹里裁判相关的事件都有中文标题
六、门禁修正
- 引用模式下,图里没有口径卡(只引查询单元格)的,出具契约不再要求 metrics_from,受管门禁也不再要求 required:
strict 加 cells 已经保证每个数都点得开、对不上就不予出具;有口径卡的照旧要求。
以前问数据的图怎么都发不成受管,Copilot 补上契约也会被判「冒出新的问题」作废
This commit is contained in:
@@ -237,6 +237,7 @@ def _apply_contract(
|
||||
strict=bool(contract.get("strict")),
|
||||
gaps=gaps,
|
||||
unresolved=cited.get("unresolved"),
|
||||
unsupported=cited.get("unsupported"),
|
||||
uncited_claims=cited.get("uncited"),
|
||||
claims_policy=cited.get("claims_policy"),
|
||||
)
|
||||
@@ -298,7 +299,7 @@ def _check_citations(
|
||||
reports: list[dict[str, Any]], *, cells: bool = True, governed: bool = False,
|
||||
) -> dict[str, Any]:
|
||||
"""独立复核报告文档:{matched, unmatched, unresolved, gaps, doc_artifact, stats, claims_policy, uncited,
|
||||
claims, entities}。
|
||||
unsupported, claims, entities}。
|
||||
|
||||
不信报告节点自己的统计:文档按 id 从工件库取回(取回时复验哈希),目录按报告节点
|
||||
同一套参数从状态里重建(不用文档里存的那份),每个带引用的片段重新解析、重新渲染、
|
||||
@@ -313,7 +314,7 @@ def _check_citations(
|
||||
|
||||
out: dict[str, Any] = {"matched": [], "unmatched": [], "unresolved": [], "gaps": [],
|
||||
"doc_artifact": None, "stats": None, "claims_policy": None, "uncited": None,
|
||||
"claims": None, "entities": None}
|
||||
"unsupported": None, "claims": None, "entities": None}
|
||||
gaps = out["gaps"]
|
||||
spec = ctx.run.spec
|
||||
node = spec.node_map().get(report_from)
|
||||
@@ -403,7 +404,7 @@ def _check_citations(
|
||||
"span": [start, end], "eid": cite["eid"]})
|
||||
|
||||
_claims(out, contract, payload, evidence_on=ledger_enabled(ctx.run), governed=governed,
|
||||
uncited=checked.get("uncited") or [])
|
||||
uncited=checked.get("uncited") or [], doc=doc, title=node.title)
|
||||
|
||||
mine = next((r for r in reports if r["node"].id == report_from), None)
|
||||
edited = mine["edited"] if mine else []
|
||||
@@ -416,20 +417,29 @@ def _check_citations(
|
||||
|
||||
#: 没挂依据的结论句怎么处置,从松到严。认不出的写法按 degrade 算:写错一个词不能让缺口悄悄消失
|
||||
_ON_UNCITED = ("ignore", "degrade", "withhold")
|
||||
#: 证据不支持的结论句怎么处置(claims: judge),从松到严。认不出的写法同样按 degrade 算
|
||||
_ON_UNSUPPORTED = ("degrade", "withhold")
|
||||
#: 要求结论句挂依据的策略:judge 在这一点上和 require_citation 一样严,另外再按裁判的判定判档
|
||||
_CITING = ("require_citation", "judge")
|
||||
|
||||
|
||||
def _claims(out: dict[str, Any], contract: dict[str, Any], payload: dict[str, Any], *, evidence_on: bool,
|
||||
governed: bool = False, uncited: list[dict[str, Any]]) -> None:
|
||||
"""结论句策略:没挂依据的结论句按 claims_policy 交给 decide_tier 判档。
|
||||
governed: bool = False, uncited: list[dict[str, Any]], doc: dict[str, Any] | None = None,
|
||||
title: str = "") -> None:
|
||||
"""结论句策略:没挂依据的、裁判判为证据不支持的结论句按 claims_policy 交给 decide_tier 判档。
|
||||
|
||||
两处可以要求挂依据:报告节点写作时记下的 claims(节点产出里的,写作提示按它要求过),和契约自己写的
|
||||
require_citation(对象形式可以另写 on_uncited)。两处都写了取更严的那个——契约只能收紧、不能放松:
|
||||
require_citation / judge(对象形式可以另写 on_uncited)。两处都写了取更严的那个——契约只能收紧、不能放松:
|
||||
报告节点要求了,契约写 ignore 或 off 也照旧计入缺口,写 withhold 就收紧成不予出具。受管级别的正式运行
|
||||
里 ignore 一律当 degrade:没有哪道门禁查契约里的 claims,不能让它把用户要的底线拉低。
|
||||
|
||||
报告节点写的是 judge 时,判定在文档里(按哈希取回的那份:判定是模型给的,出口复算不了,只能信封存的
|
||||
文档)。正式运行在节点里判过:证据不支持的按 on_unsupported 判档(报告节点和契约取更严的),没判完的
|
||||
(裁判没跑成、触顶)把裁判摘要里的缺口照抄进来——不能判完整出具,也不当成「不支持」。探索运行按需裁判,
|
||||
这里只标注。契约要 judge、报告节点却没开的,照实记缺口:出口替不了报告节点去裁判。
|
||||
|
||||
数的是这里重新核对出来的 uncited,不信文档里记的 cites。升级前写的报告(节点产出里没有 claims 这个键,
|
||||
包括跨着升级还没跑完的运行)一律不管,契约写了也照旧记「这一版还不支持」。judge(裁判模型)是后续版本
|
||||
的事,契约写了照实记缺口。
|
||||
包括跨着升级还没跑完的运行)一律不管,契约写了也照旧记「这一版还不支持」。
|
||||
"""
|
||||
written = payload.get("claims") if isinstance(payload, dict) else None
|
||||
# 报告节点升级后才在产出里记 claims(off 也记):没有这个键就是升级前写的报告
|
||||
@@ -437,23 +447,66 @@ def _claims(out: dict[str, Any], contract: dict[str, Any], payload: dict[str, An
|
||||
declared = contract.get("claims")
|
||||
name = declared.get("policy") if isinstance(declared, dict) else declared
|
||||
asks: list[str] = []
|
||||
if upgraded and written == "require_citation":
|
||||
if upgraded and written in _CITING:
|
||||
asks.append("degrade")
|
||||
if upgraded and name == "require_citation":
|
||||
if upgraded and name in _CITING:
|
||||
wanted = declared.get("on_uncited") if isinstance(declared, dict) else None
|
||||
asks.append(wanted if wanted in _ON_UNCITED else "degrade")
|
||||
elif name not in (None, "", "off"):
|
||||
out["gaps"].append("契约声明了结论句检查(claims),这一版还不支持,结论句没有核对")
|
||||
judged = upgraded and written == "judge"
|
||||
if upgraded and name == "judge" and not judged:
|
||||
out["gaps"].append(f"契约要求结论句由模型裁判(claims: judge),报告「{title}」没有开 claims: judge,"
|
||||
"结论句没有裁判")
|
||||
if not asks:
|
||||
return
|
||||
on_uncited = max(asks, key=_ON_UNCITED.index)
|
||||
if governed and on_uncited == "ignore":
|
||||
on_uncited = "degrade"
|
||||
policy: Any = "require_citation" if on_uncited == "degrade" else \
|
||||
{"policy": "require_citation", "on_uncited": on_uncited}
|
||||
out["claims_policy"], out["uncited"] = policy, uncited
|
||||
out["claims"] = {"policy": "require_citation", "uncited_claims": len(uncited), "uncited": uncited[:_LISTED],
|
||||
**({"on_uncited": on_uncited} if on_uncited != "degrade" else {})}
|
||||
extra = {"on_uncited": on_uncited} if on_uncited != "degrade" else {}
|
||||
if not judged:
|
||||
policy: Any = "require_citation" if on_uncited == "degrade" else \
|
||||
{"policy": "require_citation", "on_uncited": on_uncited}
|
||||
out["claims_policy"], out["uncited"] = policy, uncited
|
||||
out["claims"] = {"policy": "require_citation", "uncited_claims": len(uncited), "uncited": uncited[:_LISTED],
|
||||
**extra}
|
||||
return
|
||||
|
||||
from app.engine.evidence import iter_units
|
||||
|
||||
summary = (doc or {}).get("judge")
|
||||
if not isinstance(summary, dict):
|
||||
# 报告节点开了 judge、文档里却没有裁判摘要(不该发生):当作没判,不能判完整出具
|
||||
summary = {"mode": "inline", "counts": {}, "unjudged": {}, "limits_hit": [], "complete": False,
|
||||
"gaps": ["开了 claims: judge,文档里却没有裁判结果,结论句没有裁判"]}
|
||||
asked = [summary.get("on_unsupported")]
|
||||
if name == "judge" and isinstance(declared, dict):
|
||||
asked.append(declared.get("on_unsupported"))
|
||||
on_unsupported = max((a if a in _ON_UNSUPPORTED else "degrade" for a in asked), key=_ON_UNSUPPORTED.index)
|
||||
inline = summary.get("mode") == "inline"
|
||||
flagged: dict[str, list[dict[str, Any]]] = {"unsupported": [], "partial": []}
|
||||
if inline:
|
||||
# 探索运行按需裁判的判定不在文档里(在封存之后追加的 evidence.judged 事件里),这里只数正式运行的
|
||||
markdown = str((doc or {}).get("markdown") or "")
|
||||
for _, unit in iter_units(doc or {}):
|
||||
verdict = unit.get("verdict")
|
||||
status = verdict.get("status") if isinstance(verdict, dict) else None
|
||||
if status in flagged:
|
||||
span = unit.get("span") or [0, 0]
|
||||
flagged[status].append({"unit": unit.get("id"), "span": span, "text": markdown[span[0]:span[1]],
|
||||
"rationale": verdict.get("rationale") or ""})
|
||||
out["gaps"].extend(f"报告「{title}」{gap}" for gap in summary.get("gaps") or [])
|
||||
counts = {k: int((summary.get("counts") or {}).get(k) or 0)
|
||||
for k in ("supported", "partial", "unsupported", "not_a_claim", "unjudged")}
|
||||
out["claims_policy"] = {"policy": "judge", "on_unsupported": on_unsupported, "on_uncited": on_uncited}
|
||||
out["uncited"], out["unsupported"] = uncited, flagged["unsupported"]
|
||||
out["claims"] = {
|
||||
"policy": "judge", "uncited_claims": len(uncited), "uncited": uncited[:_LISTED], **extra,
|
||||
"on_unsupported": on_unsupported, "mode": "inline" if inline else "on_demand", "counts": counts,
|
||||
"unsupported": flagged["unsupported"][:_LISTED], "partial": flagged["partial"][:_LISTED],
|
||||
"unjudged": dict(summary.get("unjudged") or {}), "limits_hit": list(summary.get("limits_hit") or []),
|
||||
"complete": bool(summary.get("complete")), "model": summary.get("model"),
|
||||
}
|
||||
|
||||
|
||||
def _declare_citations(
|
||||
@@ -497,8 +550,10 @@ def _declare_citations(
|
||||
metrics_checked=len(metrics),
|
||||
matched_numbers=len(cited["matched"]),
|
||||
matched=cited["matched"][:50],
|
||||
# on_uncited 只在不是默认的 degrade 时才有(收紧成 withhold、探索运行的契约写了 ignore)
|
||||
**({"claims": {k: cited["claims"][k] for k in ("policy", "uncited_claims", "on_uncited")
|
||||
# on_uncited 只在不是默认的 degrade 时才有(收紧成 withhold、探索运行的契约写了 ignore);
|
||||
# judge 另带不支持时怎么判档、在哪判的和各判定的句数(名单只在出具声明里)
|
||||
**({"claims": {k: cited["claims"][k] for k in ("policy", "uncited_claims", "on_uncited", "on_unsupported",
|
||||
"mode", "counts")
|
||||
if k in cited["claims"]}} if cited.get("claims") else {}),
|
||||
# 计数取复核的统计(声明里的名单最多列 _LISTED 个)
|
||||
**({"entities": {"unknown": (cited["stats"] or {}).get("unknown_entities", len(entities["unknown"])),
|
||||
|
||||
@@ -15,20 +15,32 @@
|
||||
写了个哪里都找不到的名字)只标注:不重写、不失败,受管级别的正式出具按缺口降档;表结构快照
|
||||
不全时核对不了的名字(unverified_entity)只标注。entities: off 整层不管表名、字段名。
|
||||
结论句策略 claims:off(默认)不管;require_citation 在写作提示里要求结论句挂依据(自动链接的
|
||||
名字不算依据),没挂的计入缺口、出口降档;judge(模型裁判)是后续版本的事,现在写了直接报错。
|
||||
名字不算依据),没挂的计入缺口、出口降档;judge 另外请一个模型按证据逐句判断(engine/judge.py):
|
||||
|
||||
- 正式运行在节点里判:写完、核对完之后判,判定写进文档的 unit.verdict,随文档一起封存。判定只标注、
|
||||
不改写正文;裁判没跑成、触顶了照样产出,没判的记未裁判并留下缺口,出口按 on_unsupported 判档
|
||||
- 探索运行不在节点里花这笔钱:候选句标「未裁判 · 按需」,点开哪句再判哪句(证据接口)
|
||||
- rewrite_once(默认关):证据不支持的句子连同理由交回写作者只改这几句,再判一次,最多一轮;
|
||||
改写稿冒出原稿没有的违规(只标注的可疑实体也算)就不采用
|
||||
- 裁判调用包在 @task 里,排在节点所有已有调用之后:「接着跑」时直接取 checkpoint 里的判定,
|
||||
不重复调用、不重复计费。写作调用不进 checkpoint,接着跑时会重写一稿——所以判定带着输入指纹,
|
||||
对不上这一稿的不用
|
||||
|
||||
出口契约复核(io.py)用 report_catalog 按同一套参数重建目录、独立重算,不信这里的自查。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from collections import Counter
|
||||
from typing import Any
|
||||
|
||||
from langchain_core.messages import AIMessage, BaseMessage, HumanMessage, SystemMessage
|
||||
from langgraph.func import task
|
||||
|
||||
from app.core.errors import describe_exception
|
||||
from app.core.events import EventType
|
||||
from app.db.base import SessionLocal
|
||||
from app.engine import judge as judging
|
||||
from app.engine.context import NodeContext, NodeError
|
||||
from app.engine.evidence import (
|
||||
CELLS_REASON,
|
||||
@@ -43,7 +55,17 @@ from app.engine.evidence import (
|
||||
ledger_enabled,
|
||||
)
|
||||
from app.engine.nodes.llm import _invoke_streaming, _model_spec, _report_call
|
||||
from app.engine.schema import REPORT_CLAIMS, GraphNode, GraphSpec, NodeType, _ancestors, claims_problem
|
||||
from app.engine.schema import (
|
||||
CLAIMS_JUDGE,
|
||||
CLAIMS_VALUES,
|
||||
JUDGE_ON_UNSUPPORTED,
|
||||
GraphNode,
|
||||
GraphSpec,
|
||||
NodeType,
|
||||
_ancestors,
|
||||
claims_problem,
|
||||
judge_problems,
|
||||
)
|
||||
from app.engine.state import GraphState, message_text, thinking_text
|
||||
from app.engine.toolcalls import leaked_markup, markup_warning
|
||||
from app.providers.factory import ProviderNotConfigured, get_chat_model
|
||||
@@ -52,8 +74,8 @@ ON_VIOLATION = ("fail", "flag")
|
||||
NUMBERS = ("strict", "off")
|
||||
#: 实体层:link(默认)核对表名、字段名并自动链接;off 整层不管(目录里不收表和字段)
|
||||
ENTITIES = ("link", "off")
|
||||
#: 结论句策略,和画布校验同一份(schema.REPORT_CLAIMS);judge 是后续版本的事
|
||||
CLAIMS = REPORT_CLAIMS
|
||||
#: 结论句策略,和画布校验同一份(schema.CLAIMS_VALUES):off / require_citation / judge
|
||||
CLAIMS = CLAIMS_VALUES
|
||||
#: 只标注、不让写作者重写也不判失败的违规:可疑实体(受管的正式出具按缺口降档)、表结构快照不全时
|
||||
#: 核对不了的名字(出口只标注)
|
||||
SOFT = frozenset({"unknown_entity", "unverified_entity"})
|
||||
@@ -67,6 +89,8 @@ MARKUP_NUDGE = ("你上一条回复把工具调用写成了文字。报告撰写
|
||||
"证据目录里,直接用引用标记写报告正文。")
|
||||
MARKUP_ERROR = ("模型输出了工具调用的原始标记,而报告撰写节点不调用工具——数据已经在证据目录里。"
|
||||
"换一个模型再试")
|
||||
#: 接着跑时 checkpoint 里缓存的判定最多翻几条去找这一稿的(每条都是先前某一稿的,对不上就往后翻)
|
||||
_STALE_LIMIT = 16
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
@@ -120,14 +144,26 @@ def report_entities(spec: GraphSpec, node: GraphNode) -> str:
|
||||
|
||||
|
||||
def report_claims(spec: GraphSpec, node: GraphNode) -> str:
|
||||
"""报告节点的结论句策略:off / require_citation。出口判档按它(和节点执行时同一个取值规则)。
|
||||
"""报告节点的结论句策略:off / require_citation / judge。出口判档按它(和节点执行时同一个取值规则)。
|
||||
|
||||
写错的值(包括 judge)画布校验就拦了,执行时节点也会报错,这里只把没写当成 off。
|
||||
写错的值画布校验就拦了,执行时节点也会报错,这里只把没写当成 off。
|
||||
"""
|
||||
value = _setting(spec, node, "claims", "off")
|
||||
return value if isinstance(value, str) and value in CLAIMS else "off"
|
||||
|
||||
|
||||
def report_judge(spec: GraphSpec, node: GraphNode) -> dict[str, Any]:
|
||||
"""claims: judge 的子配置:节点上的 judge(没写取图级 defaults),写了的键原样保留——上限写 null 就是
|
||||
不限,没写的键运行时取设置里的默认(judge.report_budget)。rewrite_once、on_unsupported 补上缺省
|
||||
(false、degrade):出口判档读 on_unsupported。写错的值画布校验就拦了,执行时节点也会报错。
|
||||
"""
|
||||
raw = _setting(spec, node, "judge")
|
||||
raw = raw if isinstance(raw, dict) else {}
|
||||
return {**raw, "rewrite_once": raw.get("rewrite_once") is True,
|
||||
"on_unsupported": raw.get("on_unsupported") if raw.get("on_unsupported") in JUDGE_ON_UNSUPPORTED
|
||||
else "degrade"}
|
||||
|
||||
|
||||
def report_allowance(spec: GraphSpec, node: GraphNode) -> list[Any]:
|
||||
"""报告自查放行的字面量:出口契约 report_from 指向它时,契约的 allow_numbers。
|
||||
|
||||
@@ -209,13 +245,18 @@ class _RenderedStream:
|
||||
self._ctx.emit(EventType.LLM_TOKEN, delta=tail)
|
||||
|
||||
|
||||
async def _default_on_violation(run_id: str) -> str:
|
||||
"""探索运行默认标注、照常产出;正式运行默认失败——封存的发布版不能带着裸数字出具。"""
|
||||
async def _run_class(run_id: str) -> str | None:
|
||||
"""按运行记录上的 run_class 取:子图、审批之后续跑都是另起的执行上下文,只有运行记录靠得住。"""
|
||||
from app.db.models import Run
|
||||
|
||||
async with SessionLocal() as session:
|
||||
run = await session.get(Run, run_id)
|
||||
return "fail" if run is not None and run.run_class == "formal" else "flag"
|
||||
return run.run_class if run is not None else None
|
||||
|
||||
|
||||
async def _default_on_violation(run_id: str) -> str:
|
||||
"""探索运行默认标注、照常产出;正式运行默认失败——封存的发布版不能带着裸数字出具。"""
|
||||
return "fail" if await _run_class(run_id) == "formal" else "flag"
|
||||
|
||||
|
||||
def _options(ctx: NodeContext) -> tuple[str, int]:
|
||||
@@ -225,6 +266,8 @@ def _options(ctx: NodeContext) -> tuple[str, int]:
|
||||
# 画布校验也拦这两项;执行时再守一道,报的是同一句话,免得写错的值被悄悄当成默认
|
||||
if problem := claims_problem(ctx.cfg("claims")):
|
||||
raise NodeError(ctx.node.id, problem)
|
||||
if ctx.cfg("claims") == CLAIMS_JUDGE and (problems := judge_problems(ctx.cfg("judge"))):
|
||||
raise NodeError(ctx.node.id, ";".join(message for _, message in problems))
|
||||
entities = ctx.cfg("entities", "link") or "link"
|
||||
if entities not in ENTITIES:
|
||||
raise NodeError(ctx.node.id, f"entities 只能是 {' / '.join(ENTITIES)},写的是 {entities!r}")
|
||||
@@ -248,6 +291,29 @@ def _blocking(violations: list[dict[str, Any]], numbers: str) -> list[dict[str,
|
||||
and not (numbers == "off" and v.get("code") == "uncited_number")]
|
||||
|
||||
|
||||
def _violation_key(violation: dict[str, Any]) -> tuple[str, str]:
|
||||
"""认一处违规:(code, 点名的字)。点名的字按 text、ref、message 的顺序取第一个有的。"""
|
||||
named = violation.get("text") or violation.get("ref") or violation.get("message") or ""
|
||||
return str(violation.get("code") or ""), str(named)
|
||||
|
||||
|
||||
def _new_violations(before: list[dict[str, Any]], after: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
"""after 里 before 没有的违规,按 (code, 点名的字) 逐个抵消(同一个裸数字多写一处也算新的)。
|
||||
|
||||
全部违规都比,包括只标注、不让重写的那几类(SOFT):改写稿冒出一个编造的表名,受管的正式出具就会
|
||||
因此降档;去掉一处、又冒出另一处,总数没变,冒出来的也还是新的。
|
||||
"""
|
||||
left = Counter(_violation_key(v) for v in before)
|
||||
fresh: list[dict[str, Any]] = []
|
||||
for violation in after:
|
||||
key = _violation_key(violation)
|
||||
if left[key] > 0:
|
||||
left[key] -= 1
|
||||
else:
|
||||
fresh.append(violation)
|
||||
return fresh
|
||||
|
||||
|
||||
def _named(violations: list[dict[str, Any]], limit: int = 3) -> str:
|
||||
"""违规里点名的那几个字:「45678」「m:nope」。"""
|
||||
names = [str(v.get("text") or v.get("ref") or "") for v in violations]
|
||||
@@ -269,7 +335,10 @@ def _messages(ctx: NodeContext, state: GraphState, catalog: dict[str, Any], cell
|
||||
system = ctx.render_str(ctx.cfg("system", ""), state)
|
||||
instructions = ctx.render_str(ctx.cfg("instructions", ""), state).strip() \
|
||||
or "根据下面的证据写一份简洁的报告,先总后分。"
|
||||
rules = f"{MARKER_RULES}\n{CLAIMS_RULE}" if claims == "require_citation" else MARKER_RULES
|
||||
rules = MARKER_RULES
|
||||
if claims in ("require_citation", CLAIMS_JUDGE):
|
||||
# 裁判只看写作者挂的依据:要判,就更得每句都挂上
|
||||
rules = f"{rules}\n{CLAIMS_RULE}" + (f"\n{judging.JUDGE_RULE}" if claims == CLAIMS_JUDGE else "")
|
||||
return [
|
||||
SystemMessage(content="\n\n".join(p for p in (system, ROLE) if p)),
|
||||
HumanMessage(content=(
|
||||
@@ -362,11 +431,26 @@ async def run_report(state: GraphState, ctx: NodeContext) -> dict[str, Any]:
|
||||
response, raw = await draft(messages)
|
||||
doc = compose(raw)
|
||||
|
||||
judgement: dict[str, Any] | None = None
|
||||
if claims == CLAIMS_JUDGE:
|
||||
judge_cfg = report_judge(spec, ctx.node)
|
||||
if await _run_class(ctx.run.run_id) != "formal":
|
||||
# 探索运行默认按需裁判:点开哪句才判哪句,节点里不花这笔钱
|
||||
judgement = judging.on_demand(doc, on_unsupported=judge_cfg["on_unsupported"],
|
||||
rewrite_once=judge_cfg["rewrite_once"])
|
||||
elif not (blocking and on_violation == "fail"):
|
||||
# 反正要判失败的报告不值得再花钱判(节点紧接着就报错)
|
||||
doc, response, blocking, judgement, usages = await _judge_inline(
|
||||
ctx, doc, catalog, judge_cfg, model_id=model_id, messages=[*messages, response],
|
||||
draft=draft, compose=compose, numbers=numbers, blocking=blocking)
|
||||
spent.extend(usages)
|
||||
|
||||
doc_artifact = await _store(doc, ctx)
|
||||
ctx.emit(EventType.REPORT_CHECKED, doc_artifact=doc_artifact, ok=not doc["violations"],
|
||||
stats=doc["stats"], violations=doc["violations"][:20], repairs=repairs,
|
||||
on_violation=on_violation, failed=bool(blocking) and on_violation == "fail",
|
||||
**({"claims": claims} if evidence_on else {}))
|
||||
**({"claims": claims} if evidence_on else {}),
|
||||
**({"judge": judgement} if judgement is not None else {}))
|
||||
if blocking and on_violation == "fail":
|
||||
head = ";".join(v["message"] for v in blocking[:5])
|
||||
more = f";…另有 {len(blocking) - 5} 处" if len(blocking) > 5 else ""
|
||||
@@ -388,6 +472,7 @@ async def run_report(state: GraphState, ctx: NodeContext) -> dict[str, Any]:
|
||||
"model": model_id,
|
||||
"duration_ms": int((time.perf_counter() - started) * 1000),
|
||||
**({"claims": claims} if evidence_on else {}),
|
||||
**({"judge": judgement} if judgement is not None else {}),
|
||||
}
|
||||
updates: dict[str, Any] = {"nodes": {ctx.node.id: result}, "usage": usage}
|
||||
if ctx.cfg("emit_message", True):
|
||||
@@ -396,3 +481,131 @@ async def run_report(state: GraphState, ctx: NodeContext) -> dict[str, Any]:
|
||||
if var_name:
|
||||
updates["vars"] = {var_name: text}
|
||||
return updates
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# claims: judge——正式运行在节点里裁判
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _rewrite_request(sentences: list[tuple[str, str]]) -> str:
|
||||
listed = "\n".join(f"{i}.「{text}」——{why or '证据不支持这句话'}" for i, (text, why) in enumerate(sentences, 1))
|
||||
return (
|
||||
"另一个模型按证据逐句核对了你的报告,下面这几句它认为证据不支持:\n"
|
||||
f"{listed}\n\n"
|
||||
"请只改这几句:补上证据目录里真有的依据 [[see:…]],或者把说法收窄到证据能支持的程度,或者删掉;"
|
||||
"其余部分一字不改。数字仍然只能用引用标记写。不要解释改了什么,只输出改好的整篇报告正文。"
|
||||
)
|
||||
|
||||
|
||||
def _merge(first: dict[str, Any], second: dict[str, Any], budget: judging.Budget) -> dict[str, Any]:
|
||||
"""改写前后两轮裁判合成一份:判定、句数、缺口按改写后这一稿,花费、调用、用时两轮相加。"""
|
||||
merged = dict(second)
|
||||
for key in ("cost_usd", "estimated_usd"):
|
||||
merged[key] = round(first[key] + second[key], 6)
|
||||
for key in ("calls", "duration_ms"):
|
||||
merged[key] = first[key] + second[key]
|
||||
merged["notes"] = list(dict.fromkeys([*first["notes"], *second["notes"]]))
|
||||
merged["budget"] = budget.as_dict()
|
||||
merged["model"] = second.get("model") or first.get("model")
|
||||
merged["priced"] = first.get("priced")
|
||||
return merged
|
||||
|
||||
|
||||
async def _judge_inline(ctx: NodeContext, doc: dict[str, Any], catalog: dict[str, Any], judge_cfg: dict[str, Any], *,
|
||||
model_id: str, messages: list[BaseMessage], draft: Any, compose: Any, numbers: str,
|
||||
blocking: list[dict[str, Any]]
|
||||
) -> tuple[dict[str, Any], BaseMessage, list[dict[str, Any]], dict[str, Any], list[dict[str, Any]]]:
|
||||
"""写完、核对完之后请裁判判一遍,判定写进文档。返回 (文档, 写作者最后一稿, 要拦的违规, 裁判摘要, 用量)。
|
||||
|
||||
messages 是写作的对话、最后一条是写作者这一稿。rewrite_once 时把证据不支持的句子交回去只改这几句,
|
||||
再判一次(没改的句子直接用第一次的判定),最多一轮;改写稿冒出原稿没有的违规就不采用,保留原稿。
|
||||
|
||||
摘要里的 rewrite = {units, sentences, applied, reason, changed}:
|
||||
- sentences:交回去的句子(原稿里的原文)
|
||||
- units:这几句在封存文档里的编号。没采用时封存的是原稿,就是原稿的编号;采用了封存的是改写稿,
|
||||
编号按改写稿重排,按原文认:原句原样留在改写稿里的给新编号,改掉了的是 null
|
||||
- changed:采用时改写稿里新出来的句子(原稿里没有这段原文)在封存文档里的编号;没采用是 []
|
||||
"""
|
||||
from app.engine.replay import _in_graph
|
||||
|
||||
async with SessionLocal() as session:
|
||||
settings = await judging.judge_settings(session)
|
||||
judge_spec = await judging.judge_model_spec(session, judge_cfg, settings=settings)
|
||||
budget = judging.report_budget(settings, judge_cfg)
|
||||
masks = await judging.source_masks(catalog)
|
||||
|
||||
@task
|
||||
async def judge_step(request: judging.JudgeRequest, key: str, limits: judging.Budget, known: dict[str, Any],
|
||||
first: bool) -> dict[str, Any]:
|
||||
# 裁判进 checkpoint:「接着跑」时直接取这里的判定,不再调用、不再计费(每日计数也记在这里面)。
|
||||
# 事件都在这里面发:取缓存时不会再发一遍。第二轮 limits 是剩下的钱和时间,说给人看的仍是配置的上限
|
||||
outcome = await judging.run_request(request, spec=judge_spec, budget=limits, known=known, emit=ctx.emit,
|
||||
shown=budget)
|
||||
if first and outcome.get("model") and outcome["model"] == model_id:
|
||||
outcome["same_model"] = True
|
||||
ctx.emit(EventType.LOG, level="warn", code="judge_same_model",
|
||||
message=f"裁判模型和写作模型都是「{model_id}」:等于自己审自己,模型写错的地方它多半也看不出来。"
|
||||
"到设置里选一个不同的「证据裁判模型」,或者在节点的 judge.model 里指定")
|
||||
return {**outcome, "key": key}
|
||||
|
||||
async def judge_round(round_doc: dict[str, Any], limits: judging.Budget, known: dict[str, Any],
|
||||
first: bool) -> dict[str, Any]:
|
||||
request = judging.prepare(round_doc, catalog, masked=masks)
|
||||
key = judging.request_key(request, judge_spec, limits, known)
|
||||
if _in_graph():
|
||||
# 写作不进 checkpoint,接着跑时写作者会重写一稿:缓存里先前那一稿的判定对不上指纹,往后翻
|
||||
for _ in range(_STALE_LIMIT):
|
||||
outcome = await judge_step(request, key, limits, known, first)
|
||||
if outcome.get("key") == key:
|
||||
return outcome
|
||||
return await judging.run_request(request, spec=judge_spec, budget=limits, known=known, emit=ctx.emit,
|
||||
shown=budget)
|
||||
|
||||
outcome = await judge_round(doc, budget, {}, True)
|
||||
same_model = bool(outcome.get("same_model"))
|
||||
usages = [outcome["usage"]]
|
||||
response = messages[-1]
|
||||
rewrite: dict[str, Any] | None = None
|
||||
bad = [uid for uid, v in outcome["verdicts"].items() if v["status"] == "unsupported"]
|
||||
if judge_cfg["rewrite_once"] and bad:
|
||||
texts = judging.unit_texts(doc)
|
||||
sentences = [(texts.get(uid, ""), outcome["verdicts"][uid].get("rationale") or "") for uid in bad]
|
||||
ctx.emit(EventType.LOG, level="warn", code="report_rewrite",
|
||||
message=f"裁判认为 {len(bad)} 句结论证据不支持({_named([{'text': t} for t, _ in sentences])}),"
|
||||
"已交回写作者只改这几句")
|
||||
again, raw = await draft([*messages, HumanMessage(content=_rewrite_request(sentences))])
|
||||
rewritten = compose(raw)
|
||||
worse = _blocking(rewritten["violations"], numbers)
|
||||
fresh = _new_violations(doc["violations"], rewritten["violations"])
|
||||
rewrite = {"units": bad, "sentences": [t for t, _ in sentences], "applied": False, "reason": None,
|
||||
"changed": []}
|
||||
if fresh:
|
||||
rewrite["reason"] = (f"改写稿冒出 {len(fresh)} 处原稿没有的问题({_named(fresh)}),没有采用,"
|
||||
"保留原稿和原来的判定")
|
||||
ctx.emit(EventType.LOG, level="warn", code="report_rewrite_rejected", message=rewrite["reason"])
|
||||
else:
|
||||
known = {outcome["keys"][u]: v for u, v in outcome["verdicts"].items() if v["status"] in judging.JUDGED}
|
||||
# 第二轮用第一轮剩下的钱和时间:每份报告的上限管的是整份报告
|
||||
left = judging.Budget(
|
||||
max_claims=budget.max_claims,
|
||||
max_cost_usd=None if budget.max_cost_usd is None else max(0.0, budget.max_cost_usd - outcome["cost_usd"]),
|
||||
timeout_s=None if budget.timeout_s is None else max(0.0, budget.timeout_s - outcome["duration_ms"] / 1000),
|
||||
daily_max_usd=budget.daily_max_usd)
|
||||
second = await judge_round(rewritten, left, known, False)
|
||||
usages.append(second["usage"])
|
||||
outcome = _merge(outcome, second, budget)
|
||||
doc, response, blocking = rewritten, again, worse
|
||||
# 封存的是改写稿,编号按它重排:按原文认回去,改掉了的句子在改写稿里没有编号
|
||||
now = judging.unit_texts(rewritten)
|
||||
at: dict[str, str] = {}
|
||||
for uid, text in now.items():
|
||||
at.setdefault(text, uid)
|
||||
before = set(texts.values())
|
||||
rewrite.update(applied=True, units=[at.get(texts.get(uid, "")) for uid in bad],
|
||||
changed=[uid for uid, text in now.items() if text not in before])
|
||||
summary = judging.summarize(outcome, mode="inline", on_unsupported=judge_cfg["on_unsupported"],
|
||||
rewrite_once=judge_cfg["rewrite_once"], same_model=same_model, rewrite=rewrite)
|
||||
judging.apply_judgement(doc, outcome, summary)
|
||||
return doc, response, blocking, summary, usages
|
||||
|
||||
|
||||
Reference in New Issue
Block a user