可点击证据第四期:结论句由模型裁判;只引单元格的问数据图能发成受管

一、结论句裁判

- 报告节点新增 claims: judge:写完、核对完之后,把结论句和它引用的证据摘录交给裁判模型,
  判为支持 / 部分支持 / 不支持 / 不是结论,附一句理由;判定只标注,不改写正文
- 送裁判的句子按固定规则预筛:带引用的、含方向词或因果词的送去判,短句、过渡句不送
- 证据摘录:指标写名称、值和原式;查询给 SQL 前 300 字和被引用的行,被引用的列一定给到;遮罩列一律不给;
  规则里写明「数字已经由系统核对过,不要复算」
- 一份报告合并成一次调用,超出句数就分批;结构化输出失败时退回 JSON 解析,再失败记为未裁判
- 裁判模型的优先级:节点上写的 > 设置里的「证据裁判模型」> Copilot 的模型;思考关闭;
  和写作模型相同时给警告(等于自己审自己)
- rewrite_once(默认关):把不支持的句子和理由交回写作者只改这几句,再裁一次;
  改写稿冒出原稿没有的问题就不采用,保留原稿和原来的判定

二、成本上限

- 每份报告的句数、金额、时长,探索运行每次点击的金额,全局每日金额,每一项都可以自定义,也都可以设成不限;
  默认 40 句、$0.05、30 秒,每次点击 $0.01,每天 $2
- 调用前先估算,超出上限按优先级截断:先判带引用的,再判含方向词、因果词的,其余记为未裁判并写明「已到上限」;
  不报错,也不让报告失败
- 每日花费按本地日期计数,持久化在设置表里;重跑同一次运行不重复计费;价格目录里没有的模型只计次数,并写明估不出金额

三、正式运行与探索运行

- 正式运行在报告节点内裁判,判定写进报告文档、随文档封存;不支持的按 on_unsupported(degrade / withhold)降档或不予出具,
  裁判没跑成记缺口,不能判完整出具
- 探索运行按需裁判:POST /api/runs/{id}/evidence/judge,点开哪句判哪句;判定作为 evidence.judged 事件追加在封存之后,
  标「封存后追加」,封存核对仍然一致;只读封存范围内的工件;同一句不重复判;运行在裁判期间被接着跑时不记录
- 正式运行调这个接口返回 409,写明「正式运行的裁判在节点内完成」

四、治理与自动修复

- G3 放开 judge:受管级别写 judge 必须写预算(judge.max_cost_usd,或显式写 null 表示不限)
- 发布前自动修复:缺预算时给选项(默认金额 / 不限),不替人选;
  从 require_citation 改成 judge 不算降低要求,judge 改回 off、withhold 放宽成 degrade 算降低要求
- Copilot 手册:受管模板可以写 judge 并带预算;问数据按需裁判,不替人打开结论句策略

五、界面

- 结论句句末放徽标:支持 ◆、部分支持 ◇、不支持 ! 整句浅底、未裁判 ?;不用表示「已核对」的绿色;n / N 也跳到不支持的句子
- 证据面板里「模型的解释」:带「模型判断 · 模型名 · 非确定」,封存后追加的另外标出;
  「请模型判断这句」只在探索运行、还没判过时出现,触顶时写明已到上限和怎么调
- 出具横幅多一段结论句计数;设置页新增「证据裁判」一组(裁判模型、各项上限、今日花费),设成不限时写明费用受什么约束;
  报告节点可以配 judge;运行轨迹里裁判相关的事件都有中文标题

六、门禁修正

- 引用模式下,图里没有口径卡(只引查询单元格)的,出具契约不再要求 metrics_from,受管门禁也不再要求 required:
  strict 加 cells 已经保证每个数都点得开、对不上就不予出具;有口径卡的照旧要求。
  以前问数据的图怎么都发不成受管,Copilot 补上契约也会被判「冒出新的问题」作废
This commit is contained in:
ThearchyHelios
2026-09-29 11:10:53 +08:00
parent d071bf119f
commit 9727c21d8d
47 changed files with 12367 additions and 263 deletions
+4 -5
View File
@@ -118,22 +118,21 @@ def test_published_level_warnings_get_no_fix():
# --------------------------------------------------------------------------
def test_judge_becomes_require_citation_under_governed():
graph = traceable(report={**STRICT, "claims": "judge"})
def test_an_invalid_claims_value_becomes_require_citation_under_governed():
graph = traceable(report={**STRICT, "claims": "sometimes"})
fid = "report.claims_invalid:write"
fix = plan(graph)[fid]
assert fix["kind"] == "auto" and fix["preview"] == {"field": "claims", "before": "judge",
assert fix["kind"] == "auto" and fix["preview"] == {"field": "claims", "before": "sometimes",
"after": "require_citation"}
out = apply_fixes(graph, [fid], level="governed")
assert out["ok"] is True and config(out["graph"], "write")["claims"] == "require_citation"
def test_an_invalid_claims_value_is_a_choice_under_published():
graph = traceable(report={"claims": "judge"})
graph = traceable(report={"claims": "sometimes"})
fid = "report.claims_invalid:write"
fix = plan(graph, "published")[fid]
assert fix["kind"] == "choice" and [o["value"] for o in fix["options"]] == ["off", "require_citation"]
assert fix["default"] == "require_citation" # 离裁判最近的是「每句结论挂引用」,仍然要人确认
assert apply_fixes(graph, [fid], level="published")["applied"] == []
out = apply_fixes(graph, [fid], {fid: "off"}, level="published")
assert out["applied"] == [fid] and config(out["graph"], "write")["claims"] == "off" and out["ok"] is True
+111
View File
@@ -0,0 +1,111 @@
"""结论句预筛:哪些句子送裁判模型、按什么优先级,哪些直接判为连接性。
裁判按调用计费,一份报告里的过渡话、标题、表格里的一格、代码都不值得花钱判。预筛是确定性的
规则(同一份文档永远得到同一个结果),而且只看文档本身:
- 优先级 1:写作者给这句挂了依据(行内引用或 [[see:…]])
- 优先级 2:没挂依据,但有方向词、因果词(增长、主要来自、导致…)
- 优先级 3:其余有数字、实体这类实质内容、又不是短句的
- 不送:连接性的话、标题、代码、表格单元格;只有过渡词的(「原因如下:」);短句
预算不够时按优先级截断:先判 1,再判 2,剩下的记未裁判。
"""
from __future__ import annotations
from app.engine.evidence import build_catalog, compose_doc, iter_units
from app.engine.judge import SHORT_CHARS, candidates, claim_priority
def catalog():
return build_catalog(nodes={"caliber": {
"kind": "metric_set", "caliber": "周报口径", "caliber_version": "v2", "artifact": "f" * 64,
"metrics": [{"id": "gmv", "name": "销售额", "value": 45678.5, "unit": "元", "decimals": 1},
{"id": "orders", "name": "订单数", "value": 1234, "unit": "单"}]}})
RAW = """## 本周概览
本周销售额 [[m:gmv]]。增长主要来自新客首单。[[see:m:gmv]]原因如下:
下面看细节。订单一共是 1234 单,整体平稳。共 3 单。增长明显。
- 列表里订单 [[m:orders]]
- 过渡说明
| 区域 | 销售额 |
|---|---|
| 华东 | [[m:gmv]] |
```sql
SELECT 1
```
"""
def by_text(doc):
"""{句子原文: unit},测试按字找句子,不依赖编号。"""
md = doc["markdown"]
return {md[u["span"][0]:u["span"][1]].strip(): (b, u) for b, u in iter_units(doc)}
def priorities(doc):
return {text: claim_priority(u, b, doc["markdown"]) for text, (b, u) in by_text(doc).items()}
def test_the_rules():
got = priorities(compose_doc(RAW, catalog()))
assert got["本周销售额 45,678.5元。"] == 1 # 行内引用
assert got["增长主要来自新客首单。"] == 1 # 句末 [[see:]] 挂的依据
assert got["列表里订单 1,234单"] == 1 # 列表项也是句子
assert got["增长明显。"] == 2 # 没挂依据,有方向词:短也要判
assert got["订单一共是 1234 单,整体平稳。"] == 3 # 有数字、不是短句
assert got["原因如下:"] is None # 有「原因」但只是过渡
assert got["共 3 单。"] is None # 短句
assert got["下面看细节。"] is None # 连接性
assert got["过渡说明"] is None
assert got["本周概览"] is None # 标题
assert got["45,678.5元"] is None # 表格单元格:一格值不是一句结论
assert got["SELECT 1"] is None # 代码
def test_it_is_deterministic():
one, two = compose_doc(RAW, catalog()), compose_doc(RAW, catalog())
assert priorities(one) == priorities(two)
assert [c.unit for c in candidates(one)] == [c.unit for c in candidates(two)]
def test_candidates_come_in_priority_order_then_document_order():
doc = compose_doc(RAW, catalog())
texts = {u["id"]: t for t, (_, u) in by_text(doc).items()}
got = [(c.priority, texts[c.unit]) for c in candidates(doc)]
assert got == [
(1, "本周销售额 45,678.5元。"), (1, "增长主要来自新客首单。"), (1, "列表里订单 1,234单"),
(2, "增长明显。"),
(3, "订单一共是 1234 单,整体平稳。"),
]
def test_short_means_visible_characters_not_bytes():
"""短句按看得见的字算(标点、空白不算),数字的字也算进去。"""
assert SHORT_CHARS == 6
doc = compose_doc("订单 1234 单。订单总数 1234 单。", catalog())
got = priorities(doc)
assert got["订单 1234 单。"] == 3 and got["订单总数 1234 单。"] == 3
doc = compose_doc("共 12 单。", catalog())
assert priorities(doc)["共 12 单。"] is None
def test_an_explicit_request_overrides_the_screen_but_not_the_kind():
"""有人点开一句、明确要求判它:预筛放掉的结论句照判(按最低优先级);标题、连接性的话不是结论,不判。"""
doc = compose_doc(RAW, catalog())
units = {t: u["id"] for t, (_, u) in by_text(doc).items()}
picked = candidates(doc, units=[units["共 3 单。"], units["下面看细节。"], units["本周概览"], "u999"])
assert [(c.unit, c.priority) for c in picked] == [(units["共 3 单。"], 3)]
def test_cites_are_the_authors_own():
"""自动链接的名字不算依据(三期 D2):只提到表名的句子不会因此排进优先级 1。"""
ledger = [{"kind": "schema", "node_id": "fetch", "exec": 1, "artifact": "a" * 64, "source": "shop",
"schema": None, "synced_at": None, "tables": {"orders": ["amount", "region"]}}]
cat = build_catalog(nodes={}, ledger=ledger)
doc = compose_doc("明细在 `orders` 表里,按区域汇总之后东区最高。", cat)
[(text, prio)] = [(t, p) for t, p in priorities(doc).items()]
assert prio == 2, (text, prio)
+71
View File
@@ -326,3 +326,74 @@ async def test_the_report_node_and_the_exit_check_agree_when_the_status_changes_
assert seen == [True, True], seen
issuance = row.output["_issuance"]
assert issuance["tier"] == "withheld" and CELLS_REASON in issuance["unresolved"][0]["message"], issuance
# --------------------------------------------------------------------------
# 只引单元格、没有口径卡:契约不用写 metrics_from
#
# 问数据的典型图是 input → 取数 → 报告 → 出口,报告直接引用查过的单元格。受管级别在契约里
# 写 cells: true 就够了(用户拍板的第 5 条),以前校验和门禁无条件要 metrics_from,
# 这张图怎么都发不成受管,Copilot 补上契约也会被判「冒出新的问题」作废
# --------------------------------------------------------------------------
def cells_only(**contract):
contract = {"report_from": "write", "strict": True, "cells": True, **contract}
nodes = [
node("start", "input"),
node("fetch", "tool", tool="db_query__shop",
args={"sql": "SELECT SUM(amount) AS gmv, COUNT(*) AS orders FROM orders"}),
node("write", "report", instructions="写周报", numbers="strict", on_violation="fail",
claims="require_citation"),
node("out", "output", fields=[{"name": "周报", "value": "{{ nodes.write.text }}"}], contract=contract),
]
return {"nodes": nodes, "edges": [{"source": a["id"], "target": b["id"]} for a, b in zip(nodes, nodes[1:])]}
def contract_codes(graph: dict, level: str = "governed") -> tuple[list[str], list[str]]:
from app.engine.governance import lint_for_publish
from app.engine.schema import GraphSpec, validate_graph
spec = GraphSpec.model_validate(graph)
return ([i.code for i in validate_graph(spec).issues if i.level == "error"],
[i.code for i in lint_for_publish(spec, level=level).issues if i.level == "error"])
def test_a_cells_only_contract_needs_no_metrics_from():
validate, gate = contract_codes(cells_only())
assert validate == [], validate
assert gate == [], gate # 受管门禁整张图一条 error 都没有
def test_a_cited_contract_that_lists_required_metrics_still_needs_metrics_from():
validate, gate = contract_codes(cells_only(required=["gmv"]))
assert "contract.metrics_from_missing" in validate and "contract.metrics_from_missing" in gate
validate, gate = contract_codes(cells_only(expected=["gmv"]))
assert "contract.metrics_from_missing" in validate and "contract.metrics_from_missing" in gate
def test_a_narrative_contract_still_needs_metrics_from():
graph = cells_only(report_from=None, narrative="{{ nodes.write.text }}")
graph["nodes"][-1]["data"]["config"]["contract"].pop("report_from")
validate, _ = contract_codes(graph, level="published") # 受管级别不收 narrative,只看校验
assert "contract.metrics_from_missing" in validate
async def test_a_governed_formal_run_over_cells_only_issues_formally(client, monkeypatch):
Writer(monkeypatch, text="本周销售额 [[v:Q1.r0.gmv]]。")
row = await formal(client, cells_only(), status="governed")
assert row.status == "succeeded", row.error
issuance = row.output["_issuance"]
assert issuance["mode"] == "citations" and issuance["tier"] == "formal", issuance
assert issuance["unresolved"] == [] and issuance["gaps"] == [], issuance
[cell] = [m for m in issuance["matched"] if m.get("cell")]
assert cell["cell"] == "Q1.r0.gmv" and cell["token"] == "300.5"
def test_a_graph_with_a_caliber_card_still_needs_required_even_without_metrics_from():
graph = weekly(cells=True)
contract = graph["nodes"][-1]["data"]["config"]["contract"]
contract.pop("metrics_from"), contract.pop("required")
validate, gate = contract_codes(graph)
assert "contract.required_missing" in gate, gate # 不写 metrics_from 绕不过必需指标
assert validate == [], validate
+323
View File
@@ -0,0 +1,323 @@
"""出口契约(引用模式)按结论句裁判的判定判档(四期)。
- 报告节点 claims: judge 在正式运行里当场裁判,判定随文档封存。出口按文档里的判定判档:证据不支持的结论句
按 on_unsupported 降档(degrade,默认)或不予出具(withhold);裁判没跑成、触顶了没判完的,裁判摘要里的缺口
照抄进出具声明——不能判完整出具,也不因此不予出具
- 判定是模型给的,出口复算不了:信按哈希取回的文档
- judge 在挂引用这件事上和 require_citation 一样严:没挂依据的结论句照样计入缺口
- 探索运行按需裁判:出口只标注(候选句都是「未裁判 · 按需」),不因判定改档位
- 契约也能写 claims: judge:只能收紧(on_unsupported 取更严的);报告节点没开 judge 的,照实记缺口
- 升级前的运行照旧:契约写了也记「这一版还不支持」
"""
from __future__ import annotations
import asyncio
import json
import re
import uuid
import pytest
from httpx import ASGITransport, AsyncClient
from langchain_core.messages import AIMessage
from sqlalchemy import delete, select
from app.db.base import SessionLocal
from app.db.models import Run, RunEvent, Setting, Workflow, WorkflowVersion
from app.engine import judge as judge_mod
from app.engine import runner as runner_mod
from app.engine.judge import JUDGE_ROLE, SPEND_KEY
from app.engine.runner import run_manager
from app.main import app
from app.providers.mock_model import MockChatModel
JUDGE_MODEL = "claude-sonnet-5"
METRICS = [{"id": "gmv", "name": "销售额", "unit": "元", "decimals": 1, "expression": "vars.kpi.gmv"}]
#: 两句结论都挂了依据、一句连接性的话
CITED = "本周销售额 [[m:gmv]]。[[see:m:gmv]]增长主要来自新客首单。[[see:m:gmv]]下面看细节。"
#: 第二句结论没挂依据
UNCITED = "本周销售额 [[m:gmv]]。[[see:m:gmv]]增长主要来自新客首单。下面看细节。"
@pytest.fixture(autouse=True)
async def engine_up():
await run_manager.setup()
async with SessionLocal() as session:
await session.execute(delete(Setting).where(Setting.key.in_([SPEND_KEY, "judge", "copilot"])))
await session.commit()
yield
await run_manager.shutdown()
@pytest.fixture
async def client():
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as c:
yield c
def node(nid, ntype, **config):
return {"id": nid, "type": ntype, "position": {"x": 0, "y": 0}, "data": {"label": nid, "config": config}}
def weekly(*, report=None, contract=None) -> dict:
nodes = [
node("start", "input", fields=[{"name": "week", "default": "2026-W37"}]),
node("fetch", "transform", mode="expression", expression="{'gmv': 45678.5}", assign_to="kpi"),
node("caliber", "metrics", caliber="周报口径", caliber_version="v2", metrics=METRICS),
node("write", "report", instructions="写周报", **{"claims": "judge", "judge": {"max_cost_usd": 0.05},
**(report or {})}),
node("out", "output", fields=[{"name": "周报", "value": "{{ nodes.write.text }}"}],
contract={"report_from": "write", "metrics_from": ["caliber"], "required": ["gmv"], "strict": True,
**(contract or {})}),
]
ids = [n["id"] for n in nodes]
return {"nodes": nodes, "edges": [{"source": a, "target": b} for a, b in zip(ids, ids[1:])]}
class Models:
"""写作者回 text;裁判按句子里的字给判定(verdicts:{句中片段: 判定},没对上的是 supported)。"""
def __init__(self, monkeypatch, text=CITED, verdicts=None, *, fail_judge=None):
self.judges: list[str] = []
verdicts = verdicts or {}
def decide(model, messages):
if JUDGE_ROLE in str(messages[0].content):
request = str(messages[-1].content)
self.judges.append(request)
if fail_judge:
raise RuntimeError(fail_judge)
items = [{"unit": uid,
"verdict": next((s for frag, s in verdicts.items() if frag in line), "supported"),
"rationale": "摘录里有对应的指标" if "销售额" in line else "证据里没有新客维度", "used": []}
for uid, line in re.findall(r"^\[(u\d+)\]([^\n]*)$", request, re.M)]
return AIMessage(content=json.dumps({"items": items}, ensure_ascii=False))
return AIMessage(content=text)
monkeypatch.setattr(MockChatModel, "_decide", decide)
async def judge_model_of(session, spec):
return MockChatModel(model_name=JUDGE_MODEL), JUDGE_MODEL
monkeypatch.setattr(judge_mod, "get_chat_model", judge_model_of)
async def wait(run_id: str) -> Run:
for _ in range(400):
await asyncio.sleep(0.05)
async with SessionLocal() as session:
row = await session.get(Run, run_id)
if row.status in ("succeeded", "failed") and row.manifest_seq is not None:
return row
raise AssertionError(f"run {run_id} 没跑完:{row.status}")
async def issued(graph, *, run_class="formal") -> dict:
run = await run_manager.start(graph=graph, input_payload={}, run_class=run_class)
row = await wait(run.id)
assert row.status == "succeeded", row.error
return row.output["_issuance"]
async def governed(client, graph) -> Run:
"""把这张图发布成受管级别,从发布版本发起正式运行。"""
async with SessionLocal() as session:
wf = Workflow(name=f"judge-{uuid.uuid4().hex[:6]}", graph=graph, status="governed", published_version=1)
session.add(wf)
await session.flush()
session.add(WorkflowVersion(workflow_id=wf.id, version=1, graph=graph))
await session.commit()
wf_id = wf.id
res = await client.post("/api/runs", json={"workflow_id": wf_id, "run_class": "formal"})
assert res.status_code == 201, res.text
row = await wait(res.json()["id"])
assert row.status == "succeeded", row.error
return row
async def issuance_event(run_id: str) -> dict:
async with SessionLocal() as session:
[event] = (await session.execute(select(RunEvent).where(
RunEvent.run_id == run_id, RunEvent.type == "issuance"))).scalars()
return event.data
# --------------------------------------------------------------------------
# 正式运行:按封存的判定判档
# --------------------------------------------------------------------------
async def test_every_claim_supported_is_a_full_issuance(monkeypatch):
Models(monkeypatch)
issuance = await issued(weekly())
assert issuance["tier"] == "formal" and issuance["gaps"] == [], issuance
claims = issuance["claims"]
assert claims["policy"] == "judge" and claims["mode"] == "inline" and claims["complete"] is True
assert claims["counts"] == {"supported": 2, "partial": 0, "unsupported": 0, "not_a_claim": 0, "unjudged": 0}
assert claims["on_unsupported"] == "degrade" and claims["uncited_claims"] == 0
assert claims["unsupported"] == [] and claims["model"] == JUDGE_MODEL
async def test_an_unsupported_claim_degrades_by_default(monkeypatch):
Models(monkeypatch, verdicts={"新客": "unsupported"})
issuance = await issued(weekly())
assert issuance["tier"] == "degraded", "不支持按缺口降档,不是不予出具(契约是 strict 的)"
assert issuance["gaps"] == [], "判了就不是「没核」:档位来自判定,不来自缺口"
[flagged] = issuance["claims"]["unsupported"]
assert flagged["text"] == "增长主要来自新客首单。" and flagged["rationale"] == "证据里没有新客维度"
assert flagged["unit"] and len(flagged["span"]) == 2
assert issuance["claims"]["counts"]["unsupported"] == 1
async def test_withhold_withholds(monkeypatch):
Models(monkeypatch, verdicts={"新客": "unsupported"})
issuance = await issued(weekly(report={"judge": {"max_cost_usd": 0.05, "on_unsupported": "withhold"}}))
assert issuance["tier"] == "withheld" and issuance["claims"]["on_unsupported"] == "withhold"
async def test_partial_support_is_listed_but_does_not_cost_a_grade(monkeypatch):
Models(monkeypatch, verdicts={"新客": "partial"})
issuance = await issued(weekly(report={"judge": {"max_cost_usd": 0.05, "on_unsupported": "withhold"}}))
assert issuance["tier"] == "formal"
[item] = issuance["claims"]["partial"]
assert item["text"] == "增长主要来自新客首单。" and issuance["claims"]["counts"]["partial"] == 1
async def test_a_failed_judge_is_a_gap_not_a_withheld_report(monkeypatch):
Models(monkeypatch, fail_judge="网关 502")
issuance = await issued(weekly(report={"judge": {"max_cost_usd": 0.05, "on_unsupported": "withhold"}}))
assert issuance["tier"] == "degraded", "没判成不能盖完整出具,也不能当成「不支持」不予出具"
[gap] = [g for g in issuance["gaps"] if "裁判" in g]
assert gap.startswith("报告「write」") and "没裁判" in gap and "网关 502" in gap
assert issuance["claims"]["complete"] is False and issuance["claims"]["counts"]["unjudged"] == 2
async def test_hitting_a_limit_is_a_gap(monkeypatch):
Models(monkeypatch)
issuance = await issued(weekly(report={"judge": {"max_cost_usd": 0.05, "max_claims": 1}}))
assert issuance["tier"] == "degraded"
[gap] = [g for g in issuance["gaps"] if "裁判" in g]
assert "已到上限" in gap and "1 句" in gap
assert issuance["claims"]["limits_hit"] == ["max_claims"] and issuance["claims"]["counts"]["unjudged"] == 1
async def test_an_uncited_claim_still_counts_under_judge(monkeypatch):
"""judge 在挂引用这件事上和 require_citation 一样严:从 require_citation 换成 judge 不能让没挂依据的句子过关。"""
Models(monkeypatch, text=UNCITED)
issuance = await issued(weekly())
assert issuance["tier"] == "degraded" and issuance["claims"]["uncited_claims"] == 1
assert issuance["claims"]["uncited"][0]["text"] == "增长主要来自新客首单。"
async def test_the_issuance_event_carries_the_counts(monkeypatch):
Models(monkeypatch, verdicts={"新客": "unsupported"})
run = await run_manager.start(graph=weekly(), input_payload={}, run_class="formal")
row = await wait(run.id)
event = await issuance_event(row.id)
assert event["tier"] == "degraded"
assert event["claims"] == {"policy": "judge", "uncited_claims": 0, "on_unsupported": "degrade",
"mode": "inline", "counts": {"supported": 1, "partial": 0, "unsupported": 1,
"not_a_claim": 0, "unjudged": 0}}
async def test_a_governed_formal_run_grades_by_the_verdicts(client, monkeypatch):
Models(monkeypatch, verdicts={"新客": "unsupported"})
graph = weekly(report={"numbers": "strict", "on_violation": "fail",
"judge": {"max_cost_usd": None, "on_unsupported": "withhold"}})
row = await governed(client, graph)
issuance = row.output["_issuance"]
assert issuance["tier"] == "withheld" and issuance["claims"]["unsupported"][0]["text"] == "增长主要来自新客首单。"
# --------------------------------------------------------------------------
# 探索运行:只标注
# --------------------------------------------------------------------------
async def test_explore_runs_only_annotate(monkeypatch):
models = Models(monkeypatch, verdicts={"新客": "unsupported"})
issuance = await issued(weekly(report={"judge": {"max_cost_usd": 0.05, "on_unsupported": "withhold"}}),
run_class="exploratory")
assert models.judges == [], "探索运行节点里不花这笔钱"
assert issuance["tier"] == "formal" and issuance["gaps"] == []
claims = issuance["claims"]
assert claims["mode"] == "on_demand" and claims["complete"] is False
assert claims["counts"]["unjudged"] == 2 and claims["unjudged"] == {"on_demand": 2}
assert claims["unsupported"] == []
async def test_explore_runs_still_count_uncited_claims(monkeypatch):
Models(monkeypatch, text=UNCITED)
issuance = await issued(weekly(), run_class="exploratory")
assert issuance["tier"] == "degraded" and issuance["claims"]["uncited_claims"] == 1
# --------------------------------------------------------------------------
# 契约里的 claims: judge
# --------------------------------------------------------------------------
async def test_the_contract_can_tighten_on_unsupported(monkeypatch):
Models(monkeypatch, verdicts={"新客": "unsupported"})
issuance = await issued(weekly(contract={"claims": {"policy": "judge", "on_unsupported": "withhold"}}))
assert issuance["tier"] == "withheld" and issuance["claims"]["on_unsupported"] == "withhold"
assert not any("不支持" in g and "这一版" in g for g in issuance["gaps"])
async def test_the_contract_cannot_loosen_on_unsupported(monkeypatch):
Models(monkeypatch, verdicts={"新客": "unsupported"})
issuance = await issued(weekly(report={"judge": {"max_cost_usd": 0.05, "on_unsupported": "withhold"}},
contract={"claims": {"policy": "judge", "on_unsupported": "degrade"}}))
assert issuance["tier"] == "withheld"
async def test_a_contract_asking_for_judge_on_a_report_without_it_is_a_gap(monkeypatch):
Models(monkeypatch)
issuance = await issued(weekly(report={"claims": "require_citation"}, contract={"claims": "judge"}))
assert issuance["tier"] == "degraded"
[gap] = [g for g in issuance["gaps"] if "judge" in g]
assert "「write」" in gap and "没有开" in gap and "这一版" not in gap
assert issuance["claims"]["policy"] == "require_citation"
def test_a_run_in_flight_across_the_upgrade_keeps_the_old_gap():
"""升级前写的报告(产出里没有 claims)碰上写了 judge 的契约:照旧记「这一版还不支持」,不判档。"""
from app.engine.nodes.io import _claims
out = {"gaps": [], "claims_policy": None, "uncited": None, "claims": None, "unsupported": None}
_claims(out, {"claims": {"policy": "judge", "on_unsupported": "withhold"}}, {"text": "x"}, evidence_on=True,
uncited=[{"unit": "u1", "span": [0, 4], "text": "增长很快"}])
assert out["claims_policy"] is None and out["claims"] is None and out["unsupported"] is None
assert len(out["gaps"]) == 1 and "这一版还不支持" in out["gaps"][0]
async def test_runs_from_before_the_upgrade_ignore_judge(monkeypatch):
async def no_snapshot(session):
return None
monkeypatch.setattr(runner_mod, "_agent_limits", no_snapshot)
models = Models(monkeypatch, text=UNCITED, verdicts={"新客": "unsupported"})
issuance = await issued(weekly())
assert models.judges == [] and issuance["tier"] == "formal" and "claims" not in issuance
# --------------------------------------------------------------------------
# 答案复核(review.scan)认得裁判的几条警告
# --------------------------------------------------------------------------
@pytest.mark.parametrize(("code", "shown"), [
("judge_limit", True), ("judge_failed", True), ("report_rewrite", True), ("report_rewrite_rejected", True),
("judge_unpriced", False), ("judge_same_model", False),
])
def test_review_knows_the_judge_warnings(code, shown):
"""没判完、没跑成、改写过的是缺口(要说明);估不出金额、自己审自己只是配置上的提醒,轨迹里有就够了,
不值得为它再叫一次复核模型。"""
from app.engine.review import DEGRADED, scan
signals = scan([{"type": "log", "data": {"level": "warn", "code": code, "message": f"{code} 的原话"}}],
{"answer": "本周销售额 45,678.5元。"})
if shown:
[signal] = signals
assert signal.kind == code and signal.severity == DEGRADED and signal.detail == f"{code} 的原话"
else:
assert signals == []
+31 -9
View File
@@ -1,4 +1,5 @@
"""Copilot 的节点手册要教三期的写法:表名字段名、知识库原话、结论句策略,以及受管模板的报告节点怎么配。
"""Copilot 的节点手册要教三期、四期的写法:表名字段名、知识库原话、结论句策略(含模型裁判 judge),以及受管模板的
报告节点怎么配。
手册里没写的配置,Copilot 就不会用;写错的写法,报告节点会一直被打回。所以这里不只查关键词,
还把手册里举的标记例子交给真正的解析器,确认写法和系统认的是同一套。
@@ -7,9 +8,9 @@ from __future__ import annotations
import re
from app.api.copilot import NODE_REFERENCE, GenerateIn, _user_message
from app.api.copilot import _ASSIST_RULES, NODE_REFERENCE, GenerateIn, _user_message
from app.engine.evidence import QUOTE_MIN, SUPPORTED_KINDS, parse_markers
from app.engine.schema import REPORT_CLAIMS
from app.engine.schema import CLAIMS_VALUES, JUDGE_KEYS, JUDGE_ON_UNSUPPORTED
def _report_section() -> str:
@@ -22,12 +23,22 @@ def test_the_report_entry_teaches_entities_quotes_and_claims():
for words in (
["[[t:", "[[c:", "反引号", "可能是编造的名字"], # 表名、字段名
["[[q:K1|", "逐字", f"至少 {QUOTE_MIN} 个字", "[[see:K1]]"], # 知识库原话
["claims", "require_citation", "off", "judge", "后续版本"], # 结论句策略
["claims", "require_citation", "off", "judge"], # 结论句策略
["judge.max_cost_usd", "null", "不限", "rewrite_once", "on_unsupported", "按需"], # 模型裁判怎么配
["entities", "link"], # 实体层开关
):
assert all(w in report for w in words), [w for w in words if w not in report]
# 结论句策略的取值和校验认的是同一份
assert all(value in report for value in REPORT_CLAIMS)
# 结论句策略的取值和校验认的是同一份;judge 已经能用,不能再说「后续版本」「跑不起来」
assert all(value in report for value in CLAIMS_VALUES)
assert all(value in report for value in JUDGE_ON_UNSUPPORTED)
assert "后续版本" not in report and "跑不起来" not in report
# 裁判模型建议和写作模型不同:否则等于自己审自己
assert "不同" in report and "自己审自己" in report
def test_the_judge_keys_in_the_guide_are_the_ones_the_validator_knows():
named = set(re.findall(r"judge\.(\w+)", NODE_REFERENCE))
assert named and named <= set(JUDGE_KEYS), named - set(JUDGE_KEYS)
def test_the_markers_in_the_guide_parse_as_the_system_reads_them():
@@ -40,15 +51,26 @@ def test_the_markers_in_the_guide_parse_as_the_system_reads_them():
def test_governed_templates_spell_out_the_report_settings():
ref = NODE_REFERENCE
governed = ref[ref.index("受管"):]
governed = ref[ref.index("要按受管级别发布的模板"):]
for words in (['numbers: "strict"', 'on_violation: "fail"', 'claims: "require_citation"'],
['claims: "judge"', "judge.max_cost_usd", "null"],
["report_from", "narrative"], ["cite_fields: true"], ["evidence_role", "source"]):
assert all(w in governed for w in words), [w for w in words if w not in governed]
def test_the_publish_assist_knows_judge_is_the_stricter_one():
"""发布前的 Copilot 兜底只许往严里改:它要知道 judge 比 require_citation 严,不能为了过门禁把 judge 改回去。"""
assert "judge" in _ASSIST_RULES and "max_cost_usd" in _ASSIST_RULES
# update_node 把 judge 整个换掉:补预算只写 {max_cost_usd} 会把 withhold 连同裁判模型一起丢了
rules = " ".join(_ASSIST_RULES.split())
assert re.search(r"on_unsupported.{0,20}withhold.{0,20}不改回\s*degrade", rules), "withhold 不许改回 degrade"
assert re.search(r"补预算.{0,40}整份", rules), "补预算时 judge 要整份带上原来的键"
def test_the_answer_branch_mentions_names_and_quotes():
text = _user_message(GenerateIn(instruction="上周哪张表的退款最多", intent="answer"), patch=False)
assert "[[t:" in text and "反引号" in text and "编造" in text
assert "[[q:K1|" in text and "逐字" in text
# 问数是探索运行:不要替人打开结论句策略
assert not re.search(r"claims\s*[:=]\s*['\"]?require_citation", text)
# 问数是探索运行:不要替人打开结论句策略(结论句由读的人点开时按需请模型判断)
assert not re.search(r"claims\s*[:=]\s*['\"]?(require_citation|judge)", text)
assert "按需" in text
+554
View File
@@ -0,0 +1,554 @@
"""结论句裁判的内核(engine/judge.py):证据摘录、裁判调用、结构化输出兜底、预算与每日计数。
数字、实体是确定性的,系统自己能核对;「增长主要来自华东」这种话说得对不对,只能交给另一个模型按
证据判断。这是概率性的判断,所以每一步都要留余地、说实话:
- 一份报告合并成一次调用(超过每批的句数就分批),每句给 supported / partial / unsupported / not_a_claim
- 模型不支持结构化输出、或者给的不是合法结构时退回 JSON 解析,再失败就记「未裁判」,不编判定
- 裁判模型关掉 thinking,max_tokens 4096
- 预算(每份报告的句数、金额、时长,全局每日金额)每一项都能写 null 表示不限;触顶就停、已判的保留、
没判的记未裁判并说明是哪个上限——不报错、不让报告失败。金额超了按优先级截断:先判挂了依据的,
再判有方向词、因果词的
- 每日计数落库,按本地日期滚动
- 给裁判的证据摘录里没有数据源遮罩的列
所有模型调用都是 mock 剧本,不碰真接口。
"""
from __future__ import annotations
import json
import re
import pytest
from langchain_core.messages import AIMessage
from sqlalchemy import delete
from app.core import artifact_store
from app.core.events import EventType
from app.db.base import SessionLocal
from app.db.models import Setting
from app.engine import judge as judge_mod
from app.engine.evidence import build_catalog, compose_doc, iter_units
from app.engine.judge import (
BATCH_SIZE,
CHUNK_CHARS,
EXCERPT_COLS,
EXCERPT_ROWS,
JUDGE_ROLE,
JUDGE_RULES,
MAX_TOKENS,
RATIONALE_MAX,
SPEND_KEY,
SQL_CHARS,
Budget,
daily_spend,
judge_doc,
prepare,
source_masks,
)
from app.providers.factory import ModelSpec
from app.providers.mock_model import MockChatModel
#: 目录里有价格的模型:金额上限对它起作用
PRICED = "claude-sonnet-5"
UNLIMITED = Budget(max_claims=None, max_cost_usd=None, timeout_s=None, daily_max_usd=None)
SPEC = ModelSpec(model=PRICED)
# --------------------------------------------------------------------------
# 夹具:一张口径卡、一次查询(带手机号列)、一次知识库检索、一个运行输入
# --------------------------------------------------------------------------
def _store(content) -> str:
"""按 put_json 的落盘方式写进测试用的临时工件库:取回走真实的 load,复验哈希。"""
text = artifact_store.canonical_json(content)
artifact = artifact_store.content_hash(text)
path = artifact_store._path_of(artifact)
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(text, encoding="utf-8")
return artifact
CARD = {"kind": "metric_set", "caliber": "周报口径", "caliber_version": "v2", "metrics": [
{"id": "gmv", "name": "销售额", "unit": "元", "value": 45678.5, "decimals": 1, "format": "thousands",
"status": "ok", "expression": "vars.kpi.gmv", "rendered": "45,678.5元"},
{"id": "wow", "name": "环比增幅", "unit": "%", "value": 8.7, "decimals": 1, "format": "plain", "status": "ok",
"expression": "round((vars.kpi.gmv - vars.kpi.gmv_prev) / vars.kpi.gmv_prev * 100, 1)", "rendered": "8.7%"},
]}
LONG_SQL = "SELECT region, SUM(amount) AS amount, MAX(phone) AS phone FROM orders WHERE " + " AND ".join(
f"flag_{i} = 1" for i in range(60)) + " GROUP BY region ORDER BY amount DESC"
ROWS = [[f"区域{i}", 1000.0 - i, f"PHONE-{i}"] for i in range(30)]
SNAP = {"columns": ["region", "amount", "phone"], "rows": ROWS, "row_count": len(ROWS), "truncated": False,
"elapsed_ms": 3, "sql": LONG_SQL, "source": "shop"}
HITS = {"query": "退款", "collection": "docs", "hits": [
{"content": "退款主要集中在物流延误的订单。" * 60, "title": "售后周报", "document": "doc-1", "chunk": 0},
{"content": "第二段原文。", "title": "售后周报", "document": "doc-1", "chunk": 1}]}
@pytest.fixture
def stored():
card = {**CARD}
art = _store(card)
return {"card": art, "snap": _store(SNAP), "masked": _store({**SNAP, "mask_columns": ["phone"]}),
"hits": _store(HITS)}
def make_catalog(stored, snap="snap"):
ledger = [
{"kind": "metric_set", "node_id": "caliber", "exec": 1, "artifact": stored["card"], "caliber": "周报口径",
"version": "v2", "metrics": ["gmv", "wow"]},
{"kind": "query", "node_id": "fetch", "exec": 1, "artifact": stored[snap], "via": "b" * 64,
"call_id": "call_1", "tool": "db_query__shop", "source": "shop", "columns": SNAP["columns"],
"rows": len(ROWS), "truncated": False},
{"kind": "retrieval", "node_id": "kb", "exec": 1, "artifact": stored["hits"], "source": "docs", "rows": 2},
]
return build_catalog(nodes={"caliber": {**CARD, "artifact": stored["card"]}}, ledger=ledger,
inputs={"week": "2026-W37"})
RAW = ("## 本周概览\n\n"
"本周销售额 [[m:gmv]]。[[see:m:gmv]]"
"环比 [[m:wow]],增长主要来自华东。[[see:m:wow,Q1]]"
"增长主要来自新客首单。"
"订单一共是 1234 单,整体平稳。"
"下面看细节。")
def units_of(doc) -> dict[str, str]:
"""{句子原文: unit id}"""
md = doc["markdown"]
return {md[u["span"][0]:u["span"][1]].strip(): u["id"] for _, u in iter_units(doc)}
CITED, CITED2, CAUSAL, NUMBER = ("本周销售额 45,678.5元。", "环比 8.7%,增长主要来自华东。", "增长主要来自新客首单。",
"订单一共是 1234 单,整体平稳。")
@pytest.fixture(autouse=True)
async def clean_settings():
"""每日计数和裁判设置都落在设置表里:每个测试从干净的一天开始。"""
async with SessionLocal() as session:
await session.execute(delete(Setting).where(Setting.key.in_([SPEND_KEY, "judge", "copilot"])))
await session.commit()
yield
class Judge:
"""裁判模型的剧本。结构化调用和退回的纯文本调用分开记;verdicts 按 unit id 给判定,没写的都是 supported。"""
def __init__(self, monkeypatch, verdicts=None, *, model_id=PRICED, structured=None, plain=None,
latency=0.0, fail=None):
self.calls: list[dict] = []
self.specs: list[ModelSpec] = []
verdicts = verdicts or {}
def answer(units):
return json.dumps({"items": [{"unit": u, "verdict": verdicts.get(u, "supported"),
"rationale": f"{u} 的依据", "used": ["m:gmv"]} for u in units]},
ensure_ascii=False)
def decide(model, messages):
assert JUDGE_ROLE in str(messages[0].content), "这里只该有裁判调用"
units = re.findall(r"^\[(u\d+)\]", str(messages[-1].content), re.M)
kind = "structured" if model.response_format else "plain"
self.calls.append({"kind": kind, "units": units, "messages": list(messages)})
if fail:
raise RuntimeError(fail)
custom = structured if kind == "structured" else plain
if custom is not None:
return AIMessage(content=custom(units) if callable(custom) else custom)
return AIMessage(content=answer(units))
monkeypatch.setattr(MockChatModel, "_decide", decide)
async def fake_model(session, spec):
self.specs.append(spec)
return MockChatModel(model_name=model_id, latency=latency), model_id
monkeypatch.setattr(judge_mod, "get_chat_model", fake_model)
@property
def judged(self) -> list[list[str]]:
return [c["units"] for c in self.calls if c["kind"] == "structured"]
def status_of(outcome, doc, text):
return outcome["verdicts"][units_of(doc)[text]]
# --------------------------------------------------------------------------
# 调用与输出
# --------------------------------------------------------------------------
async def test_one_call_for_the_whole_report_and_the_verdict_shape(monkeypatch, stored):
judge = Judge(monkeypatch, {"u3": "unsupported"})
catalog = make_catalog(stored)
doc = compose_doc(RAW, catalog)
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
assert len(judge.calls) == 1, "一份报告合并成一次调用"
assert sorted(judge.judged[0]) == sorted(units_of(doc)[t] for t in (CITED, CITED2, CAUSAL, NUMBER))
v = status_of(outcome, doc, CITED)
assert v["status"] == "supported" and v["judge"] == PRICED and v["post_seal"] is False
assert v["rationale"] and v["used"] == ["m:gmv"]
assert outcome["counts"] == {"supported": 3, "partial": 0, "unsupported": 1, "not_a_claim": 0, "unjudged": 0}
assert outcome["model"] == PRICED and outcome["calls"] == 1 and outcome["gaps"] == []
assert units_of(doc)["下面看细节。"] not in outcome["verdicts"], "连接性的话不送裁判"
async def test_the_judge_model_runs_without_thinking(monkeypatch, stored):
judge = Judge(monkeypatch)
catalog = make_catalog(stored)
await judge_doc(compose_doc(RAW, catalog), catalog, spec=ModelSpec(model=PRICED, thinking="adaptive",
max_tokens=64000),
budget=Budget(max_claims=None, max_cost_usd=None, timeout_s=30, daily_max_usd=None))
[spec] = judge.specs
assert spec.thinking == "off" and spec.max_tokens == MAX_TOKENS == 4096
assert spec.model == PRICED and spec.timeout and spec.timeout <= 30
async def test_structured_output_failure_falls_back_to_json(monkeypatch, stored):
"""结构化输出失败(模型不支持、给的不是 JSON):退回纯文本调用,从回复里把 JSON 抠出来。"""
def fenced(units):
body = {"items": [{"unit": u, "verdict": "partial", "rationale": "只对了一半", "used": []} for u in units]}
return "判定如下:\n```json\n" + json.dumps(body, ensure_ascii=False) + "\n```"
judge = Judge(monkeypatch, structured="这不是 JSON", plain=fenced)
catalog = make_catalog(stored)
doc = compose_doc(RAW, catalog)
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
assert [c["kind"] for c in judge.calls] == ["structured", "plain"]
assert {v["status"] for v in outcome["verdicts"].values()} == {"partial"}
assert outcome["gaps"] == []
async def test_an_array_written_as_json_text_is_still_read(monkeypatch, stored):
"""有的模型走工具调用时把嵌套的数组写成一段 JSON 文本({"items": "[…]"}):照样认,不为它再花一次调用。"""
def stringified(units):
items = [{"unit": u, "verdict": "unsupported", "rationale": "对不上", "used": []} for u in units]
return json.dumps({"items": json.dumps(items, ensure_ascii=False)}, ensure_ascii=False)
judge = Judge(monkeypatch, structured=stringified)
catalog = make_catalog(stored)
doc = compose_doc(RAW, catalog)
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
assert [c["kind"] for c in judge.calls] == ["structured"]
assert {v["status"] for v in outcome["verdicts"].values()} == {"unsupported"}
async def test_a_broken_provider_is_not_called_batch_after_batch(monkeypatch, stored):
"""第一批就调不通(鉴权、网关挂了):后面几批不再一次次去撞,统一记未裁判。"""
judge = Judge(monkeypatch, fail="401 鉴权失败")
catalog = make_catalog(stored)
raw = "".join(f"第 {i} 个区域的销售额增长来自新客。[[see:m:gmv]]" for i in range(BATCH_SIZE + 5))
doc = compose_doc(raw, catalog)
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
assert len(judge.calls) == 2, "结构化一次、退回纯文本一次,第二批不再调用"
assert outcome["unjudged"] == {"error": BATCH_SIZE + 5} and "401 鉴权失败" in outcome["gaps"][0]
async def test_both_paths_failing_leaves_the_claims_unjudged(monkeypatch, stored):
Judge(monkeypatch, structured="不是 JSON", plain="还是不是")
catalog = make_catalog(stored)
doc = compose_doc(RAW, catalog)
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
assert {(v["status"], v["reason"]) for v in outcome["verdicts"].values()} == {("unjudged", "format")}
assert outcome["counts"]["unjudged"] == 4 and outcome["unjudged"] == {"format": 4}
assert outcome["gaps"] and "4 句" in outcome["gaps"][0]
async def test_bad_items_are_not_trusted(monkeypatch, stored):
"""判定不认识、理由太长、编出目录里没有的依据、漏判的句子:各按各的处理,不编判定。"""
catalog = make_catalog(stored)
doc = compose_doc(RAW, catalog)
ids = units_of(doc)
def messy(units):
return json.dumps({"items": [
{"unit": ids[CITED], "verdict": "supported", "rationale": "长" * 100, "used": ["m:gmv", "m:nope", "Q9"]},
{"unit": ids[CITED2], "verdict": "maybe", "rationale": "不认识的判定", "used": []},
{"unit": ids[CAUSAL], "verdict": "unsupported", "rationale": "没有新客维度", "used": []},
{"unit": "u999", "verdict": "supported", "rationale": "不存在的句子", "used": []},
]}, ensure_ascii=False)
Judge(monkeypatch, structured=messy)
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
v = outcome["verdicts"]
assert v[ids[CITED]]["rationale"] == "长" * RATIONALE_MAX and v[ids[CITED]]["used"] == ["m:gmv"]
assert (v[ids[CITED2]]["status"], v[ids[CITED2]]["reason"]) == ("unjudged", "format")
assert v[ids[CAUSAL]]["status"] == "unsupported"
assert (v[ids[NUMBER]]["status"], v[ids[NUMBER]]["reason"]) == ("unjudged", "missing")
assert "u999" not in v
async def test_a_failed_call_is_unjudged_not_an_error(monkeypatch, stored):
Judge(monkeypatch, fail="网关超时")
catalog = make_catalog(stored)
doc = compose_doc(RAW, catalog)
events: list = []
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED,
emit=lambda kind, **data: events.append((str(kind), data)))
assert {v["reason"] for v in outcome["verdicts"].values()} == {"error"}
assert "网关超时" in outcome["gaps"][0]
ends = [d for k, d in events if k == EventType.LLM_END]
assert ends and ends[-1]["purpose"] == "judge" and "网关超时" in ends[-1]["error"]
assert any(k == EventType.LOG and d["code"] == "judge_failed" for k, d in events)
# --------------------------------------------------------------------------
# 证据摘录
# --------------------------------------------------------------------------
def request_text(judge) -> str:
return "\n".join(str(m.content) for m in judge.calls[0]["messages"])
async def test_excerpts_follow_the_rules(monkeypatch, stored):
judge = Judge(monkeypatch)
catalog = make_catalog(stored)
raw = ("本周销售额 [[m:gmv]]。[[see:m:gmv]]"
"第 25 行的区域排在后面。[[see:Q1.r25]]"
"整体看华东领先。[[see:Q1]]"
"退款集中在物流延误的订单。[[see:K1]]"
"本周是 [[i:week]],增长来自新客。")
doc = compose_doc(raw, catalog)
await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
text = request_text(judge)
assert "数字已经由系统核对过,不要复算" in JUDGE_RULES and JUDGE_RULES in text
# 指标:名称 = 值,加原式
assert "销售额 = 45,678.5元" in text and "vars.kpi.gmv" in text
# 查询:SQL 前 300 字
assert LONG_SQL[:SQL_CHARS] in text and LONG_SQL[:SQL_CHARS + 20] not in text
# 被引用的行最多 20 行:点名的第 25 行先给,整份引用的再从头补满
rows = re.findall(r"^r(\d+):", text, re.M)
assert len(rows) == EXCERPT_ROWS and "25" in rows and "0" in rows and "22" not in rows
# 片段:最多 600 字
first = HITS["hits"][0]["content"]
assert first[:CHUNK_CHARS] in text and first[:CHUNK_CHARS + 1] not in text
assert "week = 2026-W37" in text
async def test_masked_columns_never_reach_the_judge(monkeypatch, stored):
judge = Judge(monkeypatch)
catalog = make_catalog(stored)
doc = compose_doc("整体看华东领先。[[see:Q1]]", catalog)
await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
assert "PHONE-0" in request_text(judge), "对照:没设遮罩时原值在摘录里"
# 数据源现在设的遮罩(列名不分大小写)
judge = Judge(monkeypatch)
await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED, masked={"shop": ["Phone"]})
text = request_text(judge)
assert "PHONE-" not in text and "区域0" in text
[header] = re.findall(r"^列:(.*)$", text, re.M)
assert "phone" not in header.lower() and "region" in header, "表头里也不给遮罩的列"
# 查询当时记下的遮罩(快照里的 mask_columns)
judge = Judge(monkeypatch)
catalog = make_catalog(stored, snap="masked")
doc = compose_doc("整体看华东领先。[[see:Q1]]", catalog)
await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
assert "PHONE-" not in request_text(judge)
#: 宽表:前 13 列是填充,被引用的列排在第 14、15 列,最后一列是手机号
WIDE_COLS = [f"c{i}" for i in range(13)] + ["refund_rate", "region", "phone"]
WIDE = {"columns": WIDE_COLS, "row_count": 5, "truncated": False, "elapsed_ms": 3, "source": "shop",
"sql": "SELECT * FROM wide_orders",
"rows": [[i * 100 + j for j in range(13)] + [f"RATE-{i}", f"区域{i}", f"PHONE-{i}"] for i in range(5)]}
def query_catalog(artifact: str, columns: list[str], *, source: str | None = "shop"):
ledger = [{"kind": "query", "node_id": "fetch", "exec": 1, "artifact": artifact, "tool": "db_query__shop",
**({"source": source} if source else {}), "columns": columns, "rows": 5, "truncated": False}]
return build_catalog(nodes={}, ledger=ledger)
def header_of(excerpt: str) -> tuple[list[str], str]:
"""(摘录给出的列, 表头那一整行)"""
[header] = re.findall(r"^列:(.*)$", excerpt, re.M)
return [c for c in header.split(" | ") if not c.startswith("…")], header
async def test_cited_columns_of_a_wide_table_always_reach_the_judge():
"""宽表只给前几列时,句子引用的列排在后面也一定给:裁判看不到被引用的格子,就只能判不支持。"""
catalog = query_catalog(_store(WIDE), WIDE_COLS)
doc = compose_doc("[[v:Q1.r2.region]] 的退款率最高,达到 [[v:Q1.r2.refund_rate]]。"
"手机尾号 [[v:Q1.r2.phone]] 的客户下单最多。", catalog)
excerpt = prepare(doc, catalog, masked={"shop": ["phone"]}).excerpts["Q1"]
shown, header = header_of(excerpt)
assert "region" in shown and "refund_rate" in shown, header
[row] = re.findall(r"^r2:(.*)$", excerpt, re.M)
cells = dict(zip(shown, row.split(" | "), strict=True))
assert cells["region"] == "区域2" and cells["refund_rate"] == "RATE-2"
# 遮罩的列被引用了也不给
assert "phone" not in header.lower() and "PHONE-" not in excerpt
# 没遮罩的 15 列给了 EXCERPT_COLS 列,「另有」按实际没给出的列数计
assert len(shown) == EXCERPT_COLS and header.endswith(f"…另有 {15 - EXCERPT_COLS} 列")
# 被引用的列比上限还多:一律保留
cited = "".join(f"[[v:Q1.r1.{c}]]、" for c in WIDE_COLS[:-1])
doc = compose_doc(f"这一行各列依次是 {cited}都比上周高。", catalog)
shown, header = header_of(prepare(doc, catalog).excerpts["Q1"])
assert set(shown) == set(WIDE_COLS[:-1]) and "另有 1 列" in header
async def test_current_masks_apply_when_the_entry_does_not_name_its_source(stored):
"""旧形状的台账条目没记数据源:按快照自己记的数据源找遮罩(和证据面板同一个口径),摘录里照样不给。"""
catalog = query_catalog(stored["snap"], SNAP["columns"], source=None)
doc = compose_doc("整体看华东领先。[[see:Q1]]", catalog)
excerpt = prepare(doc, catalog, masked={"shop": ["phone"]}).excerpts["Q1"]
assert "PHONE-" not in excerpt and "区域0" in excerpt and "数据源 shop" in excerpt
# 数据源现在设的遮罩也按快照里的数据源去查
from app.db.models import DataSource
catalog = query_catalog(_store({**SNAP, "source": "judge_mask_src"}), SNAP["columns"], source=None)
async with SessionLocal() as session:
session.add(DataSource(name="judge_mask_src", kind="sqlite", database="unused.db",
options={"mask_columns": "Phone"}))
await session.commit()
try:
assert await source_masks(catalog) == {"judge_mask_src": ["Phone"]}
finally:
async with SessionLocal() as session:
await session.execute(delete(DataSource).where(DataSource.name == "judge_mask_src"))
await session.commit()
# --------------------------------------------------------------------------
# 预算
# --------------------------------------------------------------------------
async def test_claims_over_max_claims_are_unjudged(monkeypatch, stored):
judge = Judge(monkeypatch)
catalog = make_catalog(stored)
doc = compose_doc(RAW, catalog)
outcome = await judge_doc(doc, catalog, spec=SPEC,
budget=Budget(max_claims=2, max_cost_usd=None, timeout_s=None, daily_max_usd=None))
ids = units_of(doc)
assert sorted(judge.judged[0]) == sorted([ids[CITED], ids[CITED2]]), "先判挂了依据的"
for text in (CAUSAL, NUMBER):
v = status_of(outcome, doc, text)
assert (v["status"], v["reason"]) == ("unjudged", "max_claims") and "已到上限" in v["rationale"]
assert outcome["limits_hit"] == ["max_claims"] and outcome["unjudged"] == {"max_claims": 2}
assert outcome["gaps"] and "已到上限" in outcome["gaps"][0]
async def test_cost_truncation_follows_priority(monkeypatch, stored):
catalog = make_catalog(stored)
doc = compose_doc(RAW, catalog)
request = prepare(doc, catalog)
by_text = {t: c for c in request.cands for t, u in units_of(doc).items() if u == c.unit}
without_p3 = request.cost(PRICED, [by_text[CITED], by_text[CITED2], by_text[CAUSAL]])
everything = request.cost(PRICED, request.cands)
assert 0 < without_p3 < everything
judge = Judge(monkeypatch)
budget = Budget(max_claims=None, max_cost_usd=(without_p3 + everything) / 2, timeout_s=None, daily_max_usd=None)
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=budget)
assert status_of(outcome, doc, NUMBER)["reason"] == "max_cost_usd"
assert all(status_of(outcome, doc, t)["status"] == "supported" for t in (CITED, CITED2, CAUSAL))
assert outcome["limits_hit"] == ["max_cost_usd"] and len(judge.judged[0]) == 3
cited_only = request.cost(PRICED, [by_text[CITED], by_text[CITED2]])
judge = Judge(monkeypatch)
budget = Budget(max_claims=None, max_cost_usd=cited_only * 1.001, timeout_s=None, daily_max_usd=None)
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=budget)
assert {status_of(outcome, doc, t)["reason"] for t in (CAUSAL, NUMBER)} == {"max_cost_usd"}
assert {status_of(outcome, doc, t)["status"] for t in (CITED, CITED2)} == {"supported"}
async def test_unlimited_never_truncates(monkeypatch, stored):
"""每一项都写 null:句数再多也全判(按每批上限分批),金额、时长、每日都不拦。"""
judge = Judge(monkeypatch)
catalog = make_catalog(stored)
raw = "".join(f"第 {i} 个区域的销售额增长来自新客。[[see:m:gmv]]" for i in range(BATCH_SIZE + 5))
doc = compose_doc(raw, catalog)
async with SessionLocal() as session: # 今天已经花掉很多:不限就是不限
session.add(Setting(key=SPEND_KEY, value={"date": judge_mod._today(), "usd": 999.0, "calls": 9}))
await session.commit()
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
assert [len(units) for units in judge.judged] == [BATCH_SIZE, 5]
assert outcome["counts"]["supported"] == BATCH_SIZE + 5 and outcome["counts"]["unjudged"] == 0
assert outcome["limits_hit"] == [] and outcome["gaps"] == []
async def test_the_daily_cap_resets_the_next_day(monkeypatch, stored):
catalog = make_catalog(stored)
doc = compose_doc(RAW, catalog)
cost = prepare(doc, catalog).cost(PRICED, prepare(doc, catalog).cands)
budget = Budget(max_claims=None, max_cost_usd=None, timeout_s=None, daily_max_usd=cost * 1.05)
day = {"today": "2026-09-28"}
monkeypatch.setattr(judge_mod, "_today", lambda: day["today"])
judge = Judge(monkeypatch)
first = await judge_doc(doc, catalog, spec=SPEC, budget=budget)
assert first["counts"]["unjudged"] == 0
spent = await daily_spend()
assert spent["date"] == "2026-09-28" and spent["usd"] == pytest.approx(cost) and spent["calls"] == 1
second = await judge_doc(doc, catalog, spec=SPEC, budget=budget)
assert {v["reason"] for v in second["verdicts"].values()} == {"daily_max_usd"}
assert second["limits_hit"] == ["daily_max_usd"] and len(judge.calls) == 1, "触顶就不再调用"
day["today"] = "2026-09-29"
third = await judge_doc(doc, catalog, spec=SPEC, budget=budget)
assert third["counts"]["unjudged"] == 0 and len(judge.calls) == 2
assert (await daily_spend())["date"] == "2026-09-29"
async def test_the_time_limit_stops_a_slow_judge(monkeypatch, stored):
Judge(monkeypatch, latency=2.0)
catalog = make_catalog(stored)
doc = compose_doc(RAW, catalog)
outcome = await judge_doc(doc, catalog, spec=SPEC,
budget=Budget(max_claims=None, max_cost_usd=None, timeout_s=0.2, daily_max_usd=None))
assert {v["reason"] for v in outcome["verdicts"].values()} == {"timeout_s"}
assert outcome["limits_hit"] == ["timeout_s"] and outcome["gaps"]
async def test_a_model_without_a_price_ignores_money_limits(monkeypatch, stored):
"""目录里没有价格的模型估不出金额:金额上限对它不起作用,并照实说「按令牌估不出金额」。"""
judge = Judge(monkeypatch, model_id="house-judge-model")
catalog = make_catalog(stored)
doc = compose_doc(RAW, catalog)
events: list = []
outcome = await judge_doc(doc, catalog, spec=ModelSpec(model="house-judge-model"),
budget=Budget(max_claims=None, max_cost_usd=1e-9, timeout_s=None, daily_max_usd=1e-9),
emit=lambda kind, **data: events.append((str(kind), data)))
assert len(judge.calls) == 1 and outcome["counts"]["unjudged"] == 0
assert outcome["priced"] is False and any("按令牌估不出金额" in n for n in outcome["notes"])
[log] = [d for k, d in events if k == EventType.LOG and d.get("code") == "judge_unpriced"]
assert "按令牌估不出金额" in log["message"]
# --------------------------------------------------------------------------
# 复用与按需
# --------------------------------------------------------------------------
async def test_known_sentences_are_not_judged_again(monkeypatch, stored):
judge = Judge(monkeypatch, {"u2": "unsupported"})
catalog = make_catalog(stored)
doc = compose_doc(RAW, catalog)
first = await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
known = {first["keys"][u]: v for u, v in first["verdicts"].items()}
again = await judge_doc(compose_doc(RAW, catalog), catalog, spec=SPEC, budget=UNLIMITED, known=known)
assert len(judge.calls) == 1 and again["reused"] == 4 and again["calls"] == 0
assert again["verdicts"] == first["verdicts"]
async def test_on_demand_judges_just_the_requested_sentence_after_the_seal(monkeypatch, stored):
judge = Judge(monkeypatch)
catalog = make_catalog(stored)
doc = compose_doc(RAW + "共 3 单。", catalog)
ids = units_of(doc)
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED, post_seal=True,
units=[ids[CAUSAL], ids["共 3 单。"], ids["本周概览"]])
assert sorted(judge.judged[0]) == sorted([ids[CAUSAL], ids["共 3 单。"]]), "点名要判的短句照判"
assert all(v["post_seal"] is True for v in outcome["verdicts"].values())
assert outcome["skipped"] == {ids["本周概览"]: "not_a_claim"}
+696
View File
@@ -0,0 +1,696 @@
"""按需裁判接口 POST /api/runs/{id}/evidence/judge(四期)。
探索运行不在节点里花裁判的钱:点开哪句才判哪句。
- 判定作为批注追加在封存之后(evidence.judged 事件,post_seal: true):封存核对只覆盖 manifest_seq 之前的
事件,所以追加之后 verify 照旧一致;封存的报告文档一个字都不改
- 只在探索运行里可用:正式运行的裁判在节点内完成,按需裁判返回 409;还没封存的、升级前发起的、封存链断了的
也不判
- 每次点击的上限、全局每日上限按设置控制;触顶不是报错,照实说「已到上限」和怎么调
- 同一次运行里同一句判过就直接给已有的判定,不再调用、不再花钱(连点两下也一样)
- 裁判只看得到封存范围内的证据:人为插进工件表的、封存之后才追加事件引用的快照,一行都不进摘录;
数据源遮罩的列同样不给
- 片段接口和证据图带上判定,封存内的和封存后追加的分得开
所有裁判都用 mock 模型给剧本,不调真接口。
"""
from __future__ import annotations
import asyncio
import json
import re
import sqlite3
from types import SimpleNamespace
import pytest
from httpx import ASGITransport, AsyncClient
from langchain_core.messages import AIMessage
from sqlalchemy import delete, select, update
from app.core import artifact_store
from app.core.events import EventType
from app.data.engine import engines
from app.db.base import SessionLocal
from app.db.models import DataSource, Run, RunEvent, Setting
from app.engine import judge as judge_mod
from app.engine import runner as runner_mod
from app.engine.evidence import iter_segments, iter_units
from app.engine.judge import JUDGE_ROLE, SPEND_KEY, daily_spend
from app.engine.runner import run_manager, verify_manifest
from app.main import app
from app.providers.mock_model import MockChatModel
PRICED = "claude-sonnet-5"
METRICS = [{"id": "gmv", "name": "销售额", "unit": "元", "decimals": 1, "expression": "vars.kpi.gmv"}]
#: 一句挂了依据的结论、一句没挂依据的归因、一句连接性的话
TEXT = "## 本周\n\n本周销售额 [[m:gmv]]。[[see:m:gmv]]增长主要来自新客首单。下面看细节。"
CITED, CAUSAL, LINK = "本周销售额 45,678.5元。", "增长主要来自新客首单。", "下面看细节。"
@pytest.fixture(autouse=True)
async def engine_up():
await run_manager.setup()
async with SessionLocal() as session:
await session.execute(delete(Setting).where(Setting.key.in_([SPEND_KEY, "judge", "copilot"])))
await session.commit()
yield
await run_manager.shutdown()
@pytest.fixture
async def client():
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as c:
yield c
def node(nid, ntype, **config):
return {"id": nid, "type": ntype, "position": {"x": 0, "y": 0}, "data": {"label": nid, "config": config}}
def chain(*nodes) -> dict:
ids = [n["id"] for n in nodes]
return {"nodes": list(nodes), "edges": [{"source": a, "target": b} for a, b in zip(ids, ids[1:])]}
def weekly(*, gate=False, **report) -> dict:
return chain(
node("start", "input", fields=[{"name": "week", "default": "2026-W37"}]),
node("fetch", "transform", mode="expression", expression="{'gmv': 45678.5}", assign_to="kpi"),
node("caliber", "metrics", caliber="周报口径", caliber_version="v2", metrics=METRICS),
node("write", "report", instructions="写周报", **report),
*([node("gate", "human", mode="approve", title="确认")] if gate else []),
node("out", "output", fields=[{"name": "周报", "value": "{{ nodes.write.text }}"}]),
)
class Models:
"""写作者回 text;裁判按句子里的字给判定(verdicts:{句中片段: 判定},没对上的是 supported)。
judges 记下每次交给裁判的请求原文,specs 记下每次要的裁判模型。"""
def __init__(self, monkeypatch, text=TEXT, verdicts=None, *, model=PRICED, fail_judge=None, delay=0.0):
self.judges: list[str] = []
self.specs: list = []
verdicts = verdicts or {}
def decide(chat, messages):
if JUDGE_ROLE in str(messages[0].content):
request = str(messages[-1].content)
self.judges.append(request)
if fail_judge:
raise RuntimeError(fail_judge)
items = [{"unit": uid,
"verdict": next((s for frag, s in verdicts.items() if frag in line), "supported"),
"rationale": f"判的是:{line.strip()[:16]}", "used": []}
for uid, line in re.findall(r"^\[(u\d+)\]([^\n]*)$", request, re.M)]
return AIMessage(content=json.dumps({"items": items}, ensure_ascii=False))
return AIMessage(content=text)
monkeypatch.setattr(MockChatModel, "_decide", decide)
async def judge_model_of(session, spec):
self.specs.append(spec)
if delay:
await asyncio.sleep(delay)
return MockChatModel(model_name=spec.model or model), spec.model or model
monkeypatch.setattr(judge_mod, "get_chat_model", judge_model_of)
async def wait(run_id: str, statuses=("succeeded", "failed")) -> Run:
for _ in range(400):
await asyncio.sleep(0.05)
async with SessionLocal() as session:
row = await session.get(Run, run_id)
if row.status in statuses and (row.status == "interrupted" or row.manifest_seq is not None):
return row
raise AssertionError(f"run {run_id} 没跑完:{row.status}")
async def finish(graph, *, run_class="exploratory", statuses=("succeeded", "failed")) -> Run:
run = await run_manager.start(graph=graph, input_payload={}, run_class=run_class)
row = await wait(run.id, statuses)
return row
def doc_of(row: Run) -> dict:
return artifact_store.load(row.output["_evidence"]["doc_artifact"])
def unit_ids(doc: dict) -> dict[str, str]:
md = doc["markdown"]
return {md[u["span"][0]:u["span"][1]].strip(): u["id"] for _, u in iter_units(doc)}
def unit_starting(doc: dict, prefix: str) -> str:
return next(uid for text, uid in unit_ids(doc).items() if text.startswith(prefix))
def seg_in(doc: dict, uid: str) -> str:
return next(s["id"] for _, u in iter_units(doc) if u["id"] == uid for s in u["segments"] if s["kind"] == "text")
async def judged_events(run_id: str) -> list[RunEvent]:
async with SessionLocal() as session:
return list((await session.execute(select(RunEvent).where(
RunEvent.run_id == run_id, RunEvent.type == EventType.EVIDENCE_JUDGED).order_by(RunEvent.seq))).scalars())
async def judge(client, run_id: str, units, **extra):
return await client.post(f"/api/runs/{run_id}/evidence/judge", json={"units": units, **extra})
async def save_setting(key: str, value) -> None:
async with SessionLocal() as session:
row = await session.get(Setting, key)
if row is None:
session.add(Setting(key=key, value=value))
else:
row.value = value
await session.commit()
# --------------------------------------------------------------------------
# 判定落在封存之后,封存核对照旧一致
# --------------------------------------------------------------------------
async def test_an_on_demand_verdict_lands_after_the_seal_and_the_seal_still_holds(client, monkeypatch):
models = Models(monkeypatch, verdicts={"新客": "unsupported"})
row = await finish(weekly(claims="judge"))
assert row.status == "succeeded" and models.judges == [], "探索运行节点里不判"
doc = doc_of(row)
ids = unit_ids(doc)
res = await judge(client, row.id, [ids[CAUSAL]])
assert res.status_code == 200, res.text
body = res.json()
verdict = body["verdicts"][ids[CAUSAL]]
assert verdict["status"] == "unsupported" and verdict["post_seal"] is True and verdict["judge"] == PRICED
assert verdict["rationale"] and body["judged"] == [ids[CAUSAL]] and body["reused"] == []
assert body["report"] == {"node_id": "write", "doc_artifact": row.output["_evidence"]["doc_artifact"]}
assert body["post_seal"] is True and body["limited"] is False and body["calls"] == 1
assert len(models.judges) == 1 and CAUSAL in models.judges[0] and CITED not in models.judges[0]
[event] = await judged_events(row.id)
assert event.seq > row.manifest_seq and event.node_id == "write" and body["event"] == {"seq": event.seq}
data = event.data
assert data["post_seal"] is True and data["report"] == "write"
assert data["doc_artifact"] == body["report"]["doc_artifact"]
assert data["verdicts"] == {ids[CAUSAL]: verdict} and data["model"] == PRICED and data["calls"] == 1
assert set(data["keys"]) == {ids[CAUSAL]} and data["units"] == [ids[CAUSAL]]
check = await verify_manifest(row.id)
assert check["ok"] is True and check["sealed_at"] == row.manifest_seq
assert (await client.get(f"/api/runs/{row.id}/verify")).json()["ok"] is True
async with SessionLocal() as session:
after = await session.get(Run, row.id)
assert after.manifest_seq == row.manifest_seq and after.last_seq == event.seq
assert doc_of(row) == doc, "封存的文档一个字都不改"
csv = await client.get(f"/api/runs/{row.id}/evidence/audit", params={"format": "csv"})
assert csv.headers["x-evidence-seal"] == "ok"
async def test_any_claim_in_an_explore_report_can_be_judged(client, monkeypatch):
"""报告节点没开 claims: judge 也能按需判:结论句都是候选。"""
models = Models(monkeypatch)
row = await finish(weekly())
ids = unit_ids(doc_of(row))
body = (await judge(client, row.id, [ids[CITED], ids[CAUSAL]])).json()
assert {u: v["status"] for u, v in body["verdicts"].items()} == {ids[CITED]: "supported", ids[CAUSAL]: "supported"}
assert len(models.judges) == 1, "一次点击的几句合成一次调用"
assert "【m:gmv】" in models.judges[0], "挂的依据摘给裁判"
async def test_sequence_numbers_continue_after_a_restart(client, monkeypatch):
"""进程重启后内存里的序号从 0 起:追加的事件要接在库里最后一条后面,不能和已有的撞号。"""
from app.core.bus import bus
Models(monkeypatch)
row = await finish(weekly())
ids = unit_ids(doc_of(row))
bus._seq.pop(row.id, None)
await judge(client, row.id, [ids[CAUSAL]])
[event] = await judged_events(row.id)
async with SessionLocal() as session:
seqs = [s for (s,) in await session.execute(select(RunEvent.seq).where(RunEvent.run_id == row.id))]
assert event.seq == max(seqs) and seqs.count(event.seq) == 1 and event.seq > row.manifest_seq
# --------------------------------------------------------------------------
# 同一句不重复判
# --------------------------------------------------------------------------
async def test_asking_again_returns_the_verdict_without_another_call(client, monkeypatch):
models = Models(monkeypatch, verdicts={"新客": "partial"})
row = await finish(weekly(claims="judge"))
ids = unit_ids(doc_of(row))
first = (await judge(client, row.id, [ids[CAUSAL]])).json()
spent = (await daily_spend())["calls"]
again = (await judge(client, row.id, [ids[CAUSAL]])).json()
assert len(models.judges) == 1 and (await daily_spend())["calls"] == spent, "不再调用、不再计费"
assert again["verdicts"] == first["verdicts"] and again["reused"] == [ids[CAUSAL]] and again["judged"] == []
assert again["calls"] == 0 and again["event"] is None and again["message"] is None
assert len(await judged_events(row.id)) == 1
# 判过的和没判过的一起点:只判没判过的
both = (await judge(client, row.id, [ids[CAUSAL], ids[CITED]])).json()
assert both["reused"] == [ids[CAUSAL]] and both["judged"] == [ids[CITED]]
assert len(models.judges) == 2 and CAUSAL not in models.judges[1]
async def test_a_double_click_is_judged_once(client, monkeypatch):
models = Models(monkeypatch, delay=0.05)
row = await finish(weekly(claims="judge"))
uid = unit_ids(doc_of(row))[CAUSAL]
one, two = await asyncio.gather(judge(client, row.id, [uid]), judge(client, row.id, [uid]))
assert one.status_code == two.status_code == 200
assert len(models.judges) == 1 and len(await judged_events(row.id)) == 1
assert sorted([one.json()["reused"], two.json()["reused"]]) == [[], [uid]]
async def test_a_failed_attempt_can_be_retried(client, monkeypatch):
Models(monkeypatch, fail_judge="网关 502")
row = await finish(weekly(claims="judge"))
uid = unit_ids(doc_of(row))[CAUSAL]
failed = (await judge(client, row.id, [uid])).json()
assert failed["verdicts"][uid]["status"] == "unjudged" and failed["verdicts"][uid]["reason"] == "error"
assert "网关 502" in failed["message"] and failed["limited"] is False
models = Models(monkeypatch, verdicts={"新客": "unsupported"})
retried = (await judge(client, row.id, [uid])).json()
assert retried["verdicts"][uid]["status"] == "unsupported" and retried["reused"] == [] and len(models.judges) == 1
# --------------------------------------------------------------------------
# 上限
# --------------------------------------------------------------------------
async def test_hitting_the_click_limit_is_a_result_not_an_error(client, monkeypatch):
models = Models(monkeypatch)
await save_setting("judge", {"click_max_cost_usd": 1e-9})
row = await finish(weekly(claims="judge"))
uid = unit_ids(doc_of(row))[CAUSAL]
res = await judge(client, row.id, [uid])
assert res.status_code == 200
body = res.json()
assert body["limited"] is True and body["limits_hit"] == ["max_cost_usd"]
assert body["unjudged"] == {"max_cost_usd": 1}
assert body["message"].startswith("已到上限") and "这次点击" in body["message"]
assert any("设置" in how and "不限" in how for how in body["adjust"])
verdict = body["verdicts"][uid]
assert verdict["status"] == "unjudged" and verdict["reason"] == "max_cost_usd" and verdict["post_seal"] is True
assert models.judges == [] and body["calls"] == 0 and body["budget"]["max_cost_usd"] == 1e-9
assert body["event"] is None and await judged_events(row.id) == [], "没问模型就没有判定可记"
async def test_the_daily_limit_applies_to_clicks_too(client, monkeypatch):
models = Models(monkeypatch)
today = (await daily_spend())["date"]
await save_setting(SPEND_KEY, {"date": today, "usd": 2.0, "calls": 9, "unpriced_calls": 0})
row = await finish(weekly(claims="judge"))
body = (await judge(client, row.id, [unit_ids(doc_of(row))[CAUSAL]])).json()
assert body["limits_hit"] == ["daily_max_usd"] and "每日" in body["message"] and models.judges == []
assert body["spend"]["usd"] == 2.0 and body["spend"]["daily_max_usd"] == 2.0
async def test_unlimited_click_budget_judges(client, monkeypatch):
models = Models(monkeypatch)
await save_setting("judge", {"click_max_cost_usd": None, "daily_max_usd": None})
row = await finish(weekly(claims="judge"))
body = (await judge(client, row.id, [unit_ids(doc_of(row))[CAUSAL]])).json()
assert body["limited"] is False and len(models.judges) == 1
assert body["budget"]["max_cost_usd"] is None and body["budget"]["daily_max_usd"] is None
assert body["spend"]["calls"] == 1 and body["spend"]["daily_max_usd"] is None
async def test_the_sentence_limit_keeps_what_was_judged(client, monkeypatch):
models = Models(monkeypatch)
row = await finish(weekly(claims="judge", judge={"max_claims": 1}))
ids = unit_ids(doc_of(row))
body = (await judge(client, row.id, [ids[CITED], ids[CAUSAL]])).json()
statuses = {u: v["status"] for u, v in body["verdicts"].items()}
assert statuses == {ids[CITED]: "supported", ids[CAUSAL]: "unjudged"} and len(models.judges) == 1
assert body["limits_hit"] == ["max_claims"] and body["verdicts"][ids[CAUSAL]]["reason"] == "max_claims"
[event] = await judged_events(row.id)
assert set(event.data["verdicts"]) == {ids[CITED], ids[CAUSAL]}, "问过模型的这一次连同没判的一起记下"
# 没判的那句还能再点
again = (await judge(client, row.id, [ids[CAUSAL]])).json()
assert again["judged"] == [ids[CAUSAL]] and again["reused"] == []
# --------------------------------------------------------------------------
# 用哪个模型
# --------------------------------------------------------------------------
async def test_the_judge_model_follows_node_then_settings(client, monkeypatch):
models = Models(monkeypatch)
await save_setting("judge", {"model": "judge-from-settings"})
row = await finish(weekly(claims="judge"))
await judge(client, row.id, [unit_ids(doc_of(row))[CAUSAL]])
[spec] = models.specs
assert spec.model == "judge-from-settings" and spec.thinking == "off" and spec.max_tokens == 4096
row = await finish(weekly(claims="judge", judge={"model": "judge-from-node"}))
body = (await judge(client, row.id, [unit_ids(doc_of(row))[CAUSAL]])).json()
assert models.specs[-1].model == "judge-from-node" and body["model"] == "judge-from-node"
# --------------------------------------------------------------------------
# 什么时候不判
# --------------------------------------------------------------------------
async def test_formal_runs_are_judged_inside_the_node_not_on_demand(client, monkeypatch):
models = Models(monkeypatch)
row = await finish(weekly(claims="judge", judge={"max_cost_usd": 0.05}), run_class="formal")
assert row.status == "succeeded", row.error
calls = len(models.judges)
res = await judge(client, row.id, [unit_ids(doc_of(row))[CAUSAL]])
assert res.status_code == 409 and res.json()["code"] == "evidence_judge_formal"
assert "正式运行的裁判在节点内完成" in res.json()["detail"]
assert len(models.judges) == calls and await judged_events(row.id) == []
async def test_a_run_paused_at_an_approval_is_not_judged(client, monkeypatch):
models = Models(monkeypatch)
row = await finish(weekly(claims="judge", gate=True), statuses=("interrupted",))
res = await judge(client, row.id, ["u1"])
assert res.status_code == 409 and res.json()["code"] == "evidence_judge_unsealed"
assert models.judges == [] and await judged_events(row.id) == []
async def test_runs_from_before_the_upgrade_are_not_judged(client, monkeypatch):
async def no_snapshot(session):
return None
monkeypatch.setattr(runner_mod, "_agent_limits", no_snapshot)
models = Models(monkeypatch)
row = await finish(weekly())
res = await judge(client, row.id, [unit_ids(doc_of(row))[CAUSAL]])
assert res.status_code == 409 and res.json()["code"] == "evidence_judge_legacy"
assert models.judges == [] and await judged_events(row.id) == []
async def test_a_broken_seal_is_not_judged(client, monkeypatch):
models = Models(monkeypatch)
row = await finish(weekly(claims="judge"))
async with SessionLocal() as session:
event = (await session.execute(select(RunEvent).where(
RunEvent.run_id == row.id, RunEvent.type == "node.finished", RunEvent.node_id == "fetch"))).scalar_one()
event.data = {**event.data, "tampered": True}
await session.commit()
res = await judge(client, row.id, [unit_ids(doc_of(row))[CAUSAL]])
assert res.status_code == 409 and res.json()["code"] == "evidence_seal_broken" and models.judges == []
@pytest.mark.parametrize("meanwhile", [
# 裁判的这几秒里有人点了「继续运行」(失败的运行可以接着跑):运行又活了
{"status": "running", "error": None},
# 接着跑完、重新封存了:manifest_seq 和清单哈希都换了,这次点击依据的那份封存已经不是最新的
{"manifest_seq": 10_000, "manifest_hash": "0" * 64},
])
async def test_a_run_continued_during_the_judge_call_records_no_event(client, monkeypatch, meanwhile):
"""判定只接在点击开始时的那份封存后面。运行在裁判期间被接着跑了,再追加就落进一次活着的运行中间,
还会被封进下一份清单:不记进运行记录,判定照样交回给点的人,并说明为什么没记。"""
models = Models(monkeypatch, verdicts={"新客": "unsupported"})
row = await finish(weekly(claims="judge"))
uid = unit_ids(doc_of(row))[CAUSAL]
ask_model = judge_mod.get_chat_model
async def continued_while_judging(session, spec):
async with SessionLocal() as other:
await other.execute(update(Run).where(Run.id == row.id).values(**meanwhile))
await other.commit()
return await ask_model(session, spec)
monkeypatch.setattr(judge_mod, "get_chat_model", continued_while_judging)
res = await judge(client, row.id, [uid])
assert res.status_code == 200, res.text
body = res.json()
assert len(models.judges) == 1 and body["calls"] == 1, "模型问过了"
assert body["verdicts"][uid]["status"] == "unsupported" and body["judged"] == [uid], "判定照样交回"
assert body["event"] is None and "接着跑" in body["message"] and "没有记进运行记录" in body["message"]
assert await judged_events(row.id) == []
async with SessionLocal() as session:
after = await session.get(Run, row.id)
assert after.last_seq == row.last_seq, "运行记录一条都没多"
async def test_a_continue_that_wins_the_row_between_check_and_write_gets_no_event(client, monkeypatch):
"""核对过状态、还没写入的那一下,接着跑的条件更新(runner._claim)先提交了:追加判定的条件更新落空,
照样不记——核对和写入是同一行上的条件更新,不是先读后写。"""
from app.core.bus import bus
from app.core.config import settings
Models(monkeypatch, verdicts={"新客": "unsupported"})
row = await finish(weekly(claims="judge"))
uid = unit_ids(doc_of(row))[CAUSAL]
set_seq = bus.set_seq
def claimed_in_between(run_id, value):
# _append_judged 读完运行、分配序号之前正好调用它:在这里用另一条连接把运行占成 running
if run_id == row.id:
db = sqlite3.connect(settings.db_path, timeout=5)
try:
db.execute("UPDATE runs SET status = 'running' WHERE id = ?", (row.id,))
db.commit()
finally:
db.close()
set_seq(run_id, value)
monkeypatch.setattr(bus, "set_seq", claimed_in_between)
body = (await judge(client, row.id, [uid])).json()
assert body["verdicts"][uid]["status"] == "unsupported" and body["event"] is None
assert "没有记进运行记录" in body["message"] and await judged_events(row.id) == []
async with SessionLocal() as session:
after = await session.get(Run, row.id)
assert after.status == "running" and after.last_seq == row.last_seq
async def test_missing_runs_and_reports(client, monkeypatch):
Models(monkeypatch)
res = await judge(client, "nope", ["u1"])
assert res.status_code == 404 and res.json()["code"] == "run_not_found"
row = await finish(chain(node("start", "input"), node("out", "output", fields=[{"name": "a", "value": "x"}])))
res = await judge(client, row.id, ["u1"])
assert res.status_code == 404 and res.json()["code"] == "evidence_report_not_found"
@pytest.mark.parametrize("payload", [{}, {"units": []}, {"units": "u1"}, {"units": [1]}, {"units": [""]},
{"units": [f"u{i}" for i in range(201)]}, {"units": ["u1"], "report": 3}])
async def test_bad_requests_are_422(client, monkeypatch, payload):
Models(monkeypatch)
row = await finish(weekly())
res = await client.post(f"/api/runs/{row.id}/evidence/judge", json=payload)
assert res.status_code == 422 and res.json()["code"] == "evidence_judge_bad_units"
async def test_sentences_that_are_not_claims_are_skipped(client, monkeypatch):
models = Models(monkeypatch)
row = await finish(weekly(claims="judge"))
ids = unit_ids(doc_of(row))
body = (await judge(client, row.id, [ids["本周"], ids[LINK], "u999"])).json()
assert body["skipped"] == {ids["本周"]: "not_a_claim", ids[LINK]: "not_a_claim", "u999": "not_found"}
assert body["verdicts"] == {} and models.judges == [] and body["event"] is None
assert "不是结论句" in body["message"]
# --------------------------------------------------------------------------
# 片段接口、证据图带上判定
# --------------------------------------------------------------------------
async def test_the_segment_shows_the_verdict_and_whether_it_can_still_be_asked(client, monkeypatch):
Models(monkeypatch, verdicts={"新客": "unsupported"})
row = await finish(weekly(claims="judge"))
doc = doc_of(row)
ids = unit_ids(doc)
before = (await client.get(f"/api/runs/{row.id}/evidence/segments/{seg_in(doc, ids[CAUSAL])}")).json()
assert before["unit"]["verdict"]["status"] == "unjudged" and before["unit"]["verdict"]["reason"] == "on_demand"
assert before["unit"]["on_demand"] == {"available": True, "reason": None, "message": None}
await judge(client, row.id, [ids[CAUSAL]])
after = (await client.get(f"/api/runs/{row.id}/evidence/segments/{seg_in(doc, ids[CAUSAL])}")).json()
assert after["unit"]["verdict"]["status"] == "unsupported" and after["unit"]["verdict"]["post_seal"] is True
assert after["unit"]["on_demand"]["available"] is False and after["unit"]["on_demand"]["reason"] == "judged"
assert "模型" in after["note"] and "后续版本" not in after["note"]
other = (await client.get(f"/api/runs/{row.id}/evidence/segments/{seg_in(doc, ids[CITED])}")).json()
assert other["unit"]["on_demand"]["available"] is True
assert "挂了依据" in other["note"] and "模型" in other["note"] and "后续版本" not in other["note"]
link = (await client.get(f"/api/runs/{row.id}/evidence/segments/{seg_in(doc, ids[LINK])}")).json()
assert link["unit"]["on_demand"]["reason"] == "not_a_claim" and "verdict" not in link["unit"]
graph = (await client.get(f"/api/runs/{row.id}/evidence")).json()
[report] = graph["reports"]
assert report["claims"] == "judge" and report["judge"]["mode"] == "on_demand"
assert report["post_seal_verdicts"] == {ids[CAUSAL]: after["unit"]["verdict"]}
audit = (await client.get(f"/api/runs/{row.id}/evidence/audit")).json()
assert audit["reports"][0]["post_seal_verdicts"] == report["post_seal_verdicts"]
async def test_sealed_verdicts_of_a_formal_run_are_shown_as_sealed(client, monkeypatch):
Models(monkeypatch, verdicts={"新客": "unsupported"})
row = await finish(weekly(claims="judge", judge={"max_cost_usd": 0.05}), run_class="formal")
doc = doc_of(row)
body = (await client.get(f"/api/runs/{row.id}/evidence/segments/{seg_in(doc, unit_ids(doc)[CAUSAL])}")).json()
assert body["unit"]["verdict"]["status"] == "unsupported" and body["unit"]["verdict"]["post_seal"] is False
assert body["unit"]["on_demand"]["reason"] == "formal"
graph = (await client.get(f"/api/runs/{row.id}/evidence")).json()
assert graph["reports"][0]["post_seal_verdicts"] == {} and graph["reports"][0]["judge"]["mode"] == "inline"
async def test_a_forged_event_cannot_overwrite_a_sealed_verdict(client, monkeypatch):
"""封存后追加的判定只填补没判过的句子:封存内判过的,后面追加什么都盖不掉。"""
Models(monkeypatch, verdicts={"新客": "unsupported"})
row = await finish(weekly(claims="judge", judge={"max_cost_usd": 0.05}), run_class="formal")
doc = doc_of(row)
uid = unit_ids(doc)[CAUSAL]
doc_artifact = row.output["_evidence"]["doc_artifact"]
await run_manager.note(row.id, EventType.EVIDENCE_JUDGED, node_id="write", report="write",
doc_artifact=doc_artifact, post_seal=True,
verdicts={uid: {"status": "supported", "rationale": "改好了", "judge": "x", "post_seal": True}})
body = (await client.get(f"/api/runs/{row.id}/evidence/segments/{seg_in(doc, uid)}")).json()
assert body["unit"]["verdict"]["status"] == "unsupported"
async def test_appended_verdicts_always_read_as_post_seal(client, monkeypatch):
"""追加的判定不管载荷里怎么写,展示出来都是「封存后追加」:只有随文档封存的判定才算封存内的。"""
Models(monkeypatch)
row = await finish(weekly(claims="judge"))
doc = doc_of(row)
uid = unit_ids(doc)[CAUSAL]
await run_manager.note(row.id, EventType.EVIDENCE_JUDGED, node_id="write", report="write",
doc_artifact=row.output["_evidence"]["doc_artifact"],
verdicts={uid: {"status": "supported", "rationale": "x", "judge": "x", "post_seal": False},
"u999": {"status": "supported", "rationale": "x", "judge": "x"}})
body = (await client.get(f"/api/runs/{row.id}/evidence/segments/{seg_in(doc, uid)}")).json()
assert body["unit"]["verdict"]["status"] == "supported" and body["unit"]["verdict"]["post_seal"] is True
graph = (await client.get(f"/api/runs/{row.id}/evidence")).json()
assert set(graph["reports"][0]["post_seal_verdicts"]) == {uid}, "文档里没有的句子不认"
# --------------------------------------------------------------------------
# 裁判只看得到封存范围内的证据
# --------------------------------------------------------------------------
SOURCE = "warehouse"
TOOL = f"db_query__{SOURCE}"
SQL = "SELECT region, SUM(amount) AS gmv, COUNT(*) AS order_cnt FROM orders GROUP BY region ORDER BY gmv DESC"
EAST, WEST = "东区销售额 [[v:Q1.r0.gmv]]。[[see:Q1]]", "西区销售额 [[v:Q1.r1.gmv]]。[[see:Q1]]"
@pytest.fixture
async def warehouse(tmp_path):
from app.data.introspect import introspect
path = tmp_path / "warehouse.db"
db = sqlite3.connect(path)
db.executescript("CREATE TABLE orders (id INTEGER PRIMARY KEY, region TEXT, amount REAL);")
db.executemany("INSERT INTO orders VALUES (?, ?, ?)", [(1, "east", 100.5), (2, "west", 200.0), (3, "east", 300.0)])
db.commit()
db.close()
probe = SimpleNamespace(id=f"probe-{path.name}", name=SOURCE, kind="sqlite", database=str(path), host=None,
port=None, username=None, password=None, options={}, readonly=True, schema_cache={})
cache = await introspect(probe)
await engines.invalidate(probe.id)
async with SessionLocal() as session:
row = (await session.execute(select(DataSource).where(DataSource.name == SOURCE))).scalar_one_or_none()
if row is None:
row = DataSource(name=SOURCE, kind="sqlite", database=str(path), readonly=True, enabled=True)
session.add(row)
row.database, row.kind, row.enabled, row.readonly, row.schema_cache = str(path), "sqlite", True, True, cache
row.options = {}
await session.commit()
source_id = row.id
await engines.invalidate(source_id)
yield source_id
await engines.invalidate(source_id)
def queried() -> dict:
return chain(node("start", "input"), node("fetch", "tool", tool=TOOL, args={"sql": SQL}),
node("write", "report", instructions="写周报"),
node("out", "output", fields=[{"name": "周报", "value": "{{ nodes.write.text }}"}]))
async def reseal(run_id: str) -> None:
from app.core.artifact_store import manifest_hash
async with SessionLocal() as session:
run = await session.get(Run, run_id)
rows = [tuple(r) for r in await session.execute(
select(RunEvent.seq, RunEvent.type, RunEvent.node_id, RunEvent.data)
.where(RunEvent.run_id == run_id, RunEvent.seq <= run.manifest_seq).order_by(RunEvent.seq))]
run.manifest_hash = manifest_hash(rows)
await session.commit()
async def unseal_queries(run_id: str) -> str:
"""把查询快照从封存的事件里摘掉、按改过的事件重新封存:快照还在工件库和工件表里,封存链完好,
只是没有哪条封存的事件引用它(等于「人为插进工件表」)。返回那件快照的 id。"""
async with SessionLocal() as session:
rows = list((await session.execute(select(RunEvent).where(
RunEvent.run_id == run_id, RunEvent.type.in_(["tool.end", "node.finished"])))).scalars())
artifact = None
for event in rows:
data = dict(event.data or {})
if event.type == "tool.end":
artifact = data.pop("query_artifact", None) or artifact
data.pop("schema_artifact", None)
elif data.get("evidence"):
data["evidence"] = [e for e in data["evidence"] if e.get("kind") not in ("query", "schema")]
event.data = data
await session.commit()
await reseal(run_id)
return artifact
def excerpt_lines(request: str) -> tuple[str | None, list[str]]:
header = re.search(r"^列:(.*)$", request, re.M)
return (header.group(1) if header else None), re.findall(r"^r\d+:.*$", request, re.M)
async def test_only_sealed_snapshots_reach_the_judge(client, monkeypatch, warehouse):
models = Models(monkeypatch, text=EAST + WEST)
control = await finish(queried())
assert control.status == "succeeded", control.error
await judge(client, control.id, [unit_starting(doc_of(control), "东区")])
assert "【Q1】查询结果" in models.judges[0] and "SQL:SELECT region" in models.judges[0]
assert excerpt_lines(models.judges[0])[1], "封存链完好时照常摘被引用的行"
row = await finish(queried())
doc = doc_of(row)
artifact = await unseal_queries(row.id)
assert artifact and artifact_store.load(artifact) is not None, "快照本身还在工件库里"
assert (await verify_manifest(row.id))["ok"] is True
res = await judge(client, row.id, [unit_starting(doc, "东区")])
assert res.status_code == 200, res.text
assert "【Q1】" not in models.judges[1] and "SELECT" not in models.judges[1]
assert excerpt_lines(models.judges[1]) == (None, [])
# 封存之后才追加的事件引用了它,也不算
async with SessionLocal() as session:
run = await session.get(Run, row.id)
seq = run.last_seq
from app.core.bus import bus
bus.set_seq(row.id, seq)
await run_manager.note(row.id, "tool.end", node_id="fetch", tool=TOOL, query_artifact=artifact)
await judge(client, row.id, [unit_starting(doc, "西区")])
assert "【Q1】" not in models.judges[2] and "SELECT" not in models.judges[2]
async def test_masked_columns_never_reach_the_judge(client, monkeypatch, warehouse):
models = Models(monkeypatch, text=EAST)
row = await finish(queried())
async with SessionLocal() as session:
source = await session.get(DataSource, warehouse)
source.options = {"mask_columns": ["order_cnt"]}
await session.commit()
await judge(client, row.id, [unit_starting(doc_of(row), "东区")])
header, lines = excerpt_lines(models.judges[0])
assert header is not None and "order_cnt" not in header and "gmv" in header
assert lines and all(line.count("|") == 1 for line in lines), lines
assert "遮罩" in models.judges[0]
+344
View File
@@ -0,0 +1,344 @@
"""四期:受管模板放开 claims: judge(结论句由另一个模型按证据逐句裁判),但要写明预算。
- G3:报告撰写节点的 claims 在受管级别认 require_citation 或 judge;写 judge 的,生效的 judge 配置里
必须有 max_cost_usd 这个键——写金额,或者显式写 null 表示不限。没写就是 error(已发布级别只给警告):
受管出具不能让每份报告的裁判花费悄悄按默认值走,也不能悄悄不设上限
- 发布前自动修复:缺预算是 choice,候选是默认金额和「不限」,不替人选;G3 另外那几项的一键修复不再把
judge 改回 require_citation
- 降低要求的判断:require_citation → judge 是往严里改;judge → require_citation / off / 删掉都算放宽,
报告节点上和出具契约里一样
"""
from __future__ import annotations
import copy
import pytest
from app.engine.autofix import NEED_CHOICE, apply_fixes, forbidden_changes, plan_fixes, public
from app.engine.governance import lint_for_publish, publish_issues
from app.engine.judge import JUDGE_DEFAULTS
from app.engine.schema import GraphSpec, validate_graph
STRICT = {"numbers": "strict", "on_violation": "fail", "claims": "require_citation"}
JUDGED = {**STRICT, "claims": "judge", "judge": {"max_cost_usd": 0.05}}
BUDGET = "governed.judge_budget"
def node(nid, ntype, label=None, **config):
return {"id": nid, "type": ntype, "position": {"x": 10, "y": 20},
"data": {"label": label or nid, "config": config}}
def traceable(*, report=None, defaults=None, **contract_overrides) -> dict:
"""input → 查库 → 口径卡 → 报告撰写 → 出具:受管要求(含 G1–G5)全部满足。"""
contract = {"report_from": "write", "metrics_from": ["card"], "required": ["gmv"], "strict": True}
contract.update(contract_overrides)
nodes = [
node("start", "input", "周期"),
node("fetch", "tool", "取数", tool="db_query__shop",
args={"sql": "SELECT SUM(amount) AS gmv FROM orders"}),
node("card", "metrics", "周报口径卡", caliber="周报口径", metrics=[
{"id": "gmv", "name": "销售额", "unit": "元", "expression": "cell(nodes.fetch, 0, 'gmv')"}]),
node("write", "report", "报告撰写", instructions="写周报", **(JUDGED if report is None else report)),
node("done", "output", "出具", fields=[{"name": "周报", "value": "{{ nodes.write.text }}"}],
contract={k: v for k, v in contract.items() if v is not None}),
]
edges = [{"id": f"e{i}", "source": a["id"], "target": b["id"]} for i, (a, b) in enumerate(zip(nodes, nodes[1:]))]
graph = {"nodes": nodes, "edges": edges}
if defaults is not None:
graph["defaults"] = defaults
return graph
def config(graph: dict, nid: str) -> dict:
return next(n for n in graph["nodes"] if n["id"] == nid)["data"]["config"]
def by_code(graph: dict, code: str, level: str = "governed") -> list:
return [i for i in lint_for_publish(GraphSpec.model_validate(graph), level=level).issues if i.code == code]
def errors(graph: dict, level: str = "governed") -> list:
return [i for i in publish_issues(GraphSpec.model_validate(graph), level=level) if i.level == "error"]
def plan(graph: dict, level: str = "governed") -> dict[str, dict]:
spec = GraphSpec.model_validate(graph)
return {f["id"]: f for f in plan_fixes(spec, publish_issues(spec, level=level), level=level)}
# --------------------------------------------------------------------------
# G3:judge 可以发布成受管,但要写预算
# --------------------------------------------------------------------------
@pytest.mark.parametrize("budget", [0.05, 1, None])
def test_judge_with_a_budget_publishes_as_governed(budget):
graph = traceable(report={**JUDGED, "judge": {"max_cost_usd": budget, "max_claims": 20}})
assert errors(graph) == [], [i.message for i in errors(graph)]
assert by_code(graph, "governed.report_policy") == [] and by_code(graph, BUDGET) == []
@pytest.mark.parametrize("judge", [None, {}, {"max_claims": 20, "model": "judge-model"}])
def test_judge_without_a_budget_is_an_error_at_governed(judge):
report = {**STRICT, "claims": "judge", **({"judge": judge} if judge is not None else {})}
graph = traceable(report=report)
[issue] = by_code(graph, BUDGET)
assert issue.level == "error" and issue.node_id == "write" and issue.field == "judge.max_cost_usd"
assert "max_cost_usd" in issue.message and "不限" in issue.message and "null" in issue.message
assert by_code(graph, "governed.report_policy") == [], "judge 本身合规,只缺预算"
assert [i.level for i in by_code(graph, BUDGET, "published")] == ["warning"]
def test_the_budget_follows_the_same_config_rule_as_run_time():
"""和运行时同一个取值规则:节点没写 judge 取图级 defaults;节点写了就整个用节点的(不和 defaults 拼)。"""
graph = traceable(report={**STRICT, "claims": "judge"}, defaults={"judge": {"max_cost_usd": 0.02}})
assert by_code(graph, BUDGET) == []
graph = traceable(report={**STRICT, "claims": "judge", "judge": {"max_claims": 5}},
defaults={"judge": {"max_cost_usd": 0.02}})
assert [i.field for i in by_code(graph, BUDGET)] == ["judge.max_cost_usd"]
# claims 本身也可以来自图级 defaults
graph = traceable(report={"numbers": "strict", "on_violation": "fail"},
defaults={"claims": "judge", "judge": {"max_cost_usd": None}})
assert by_code(graph, BUDGET) == [] and by_code(graph, "governed.report_policy") == []
def test_a_malformed_budget_is_left_to_validate():
graph = traceable(report={**STRICT, "claims": "judge", "judge": {"max_cost_usd": "cheap"}})
assert by_code(graph, BUDGET) == [], "写错的值由 validate 的 report.judge_invalid 报,同一处不说两遍"
[bad] = [i for i in validate_graph(GraphSpec.model_validate(graph)).issues if i.code == "report.judge_invalid"]
assert bad.field == "judge.max_cost_usd" and bad.level == "error"
@pytest.mark.parametrize("claims", [None, "off"])
def test_a_loose_claims_setting_names_both_accepted_values(claims):
report = {"numbers": "strict", "on_violation": "fail", **({"claims": claims} if claims else {})}
[issue] = by_code(traceable(report=report), "governed.report_policy")
assert issue.field == "claims" and "claims: require_citation" in issue.message and "judge" in issue.message
def test_judge_needs_no_other_report_policy_fix():
graph = traceable(report={**JUDGED, "numbers": "off"})
[issue] = by_code(graph, "governed.report_policy")
assert issue.field == "numbers" and "claims" not in issue.message
# --------------------------------------------------------------------------
# 发布前自动修复
# --------------------------------------------------------------------------
def test_the_report_policy_fix_leaves_judge_alone():
graph = traceable(report={**JUDGED, "numbers": "off", "on_violation": "flag"})
fix = plan(graph)["governed.report_policy:write"]
assert fix["kind"] == "auto" and fix["preview"]["after"] == {"numbers": "strict", "on_violation": "fail"}
out = apply_fixes(graph, [fix["id"]], level="governed")
assert out["ok"] is True and config(out["graph"], "write")["claims"] == "judge"
assert config(out["graph"], "write")["judge"] == {"max_cost_usd": 0.05}
def test_a_missing_budget_is_a_choice_between_the_default_and_unlimited():
graph = traceable(report={**STRICT, "claims": "judge", "judge": {"model": "judge-model"}})
fid = f"{BUDGET}:write"
fix = public(plan(graph)[fid])
assert fix["kind"] == "choice" and fix["node_id"] == "write" and fix["field"] == "judge.max_cost_usd"
assert [o["value"] for o in fix["options"]] == [JUDGE_DEFAULTS["report_max_cost_usd"], None]
assert "default" not in fix, "花多少钱要人定:不替人选,也不给建议"
unlimited = fix["options"][1]
assert "不限" in unlimited["label"] and "不设上限" in unlimited["hint"] and "约束" in unlimited["hint"]
assert all(not k.startswith("_") for k in fix), "内部键不交给前端"
[refused] = apply_fixes(graph, [fid], level="governed")["rejected"]
assert refused["reason"] == NEED_CHOICE.format(label=fix["label"])
out = apply_fixes(graph, [fid], {fid: 0.05}, level="governed")
assert out["applied"] == [fid] and out["ok"] is True
assert config(out["graph"], "write")["judge"] == {"model": "judge-model", "max_cost_usd": 0.05}
[change] = out["changes"]
assert change["field"] == "judge" and change["before"] == {"model": "judge-model"}
assert out["ops"] == [{"op": "update_node", "id": "write",
"config": {"judge": {"model": "judge-model", "max_cost_usd": 0.05}}}]
out = apply_fixes(graph, [fid], {fid: None}, level="governed")
judge = config(out["graph"], "write")["judge"]
assert out["ok"] is True and "max_cost_usd" in judge and judge["max_cost_usd"] is None, "不限要显式写成 null"
def test_the_budget_fix_keeps_what_the_graph_defaults_said():
"""judge 写在图级 defaults 里:修复写到节点上时带上 defaults 里的整份 judge,节点写了就不再跟随 defaults,
不能因为补一个预算把 defaults 里选的裁判模型丢了。"""
graph = traceable(report={**STRICT, "claims": "judge"},
defaults={"judge": {"model": "judge-model", "rewrite_once": True}})
fid = f"{BUDGET}:write"
out = apply_fixes(graph, [fid], {fid: None}, level="governed")
assert out["ok"] is True
assert config(out["graph"], "write")["judge"] == {"model": "judge-model", "rewrite_once": True,
"max_cost_usd": None}
assert out["graph"]["defaults"] == {"judge": {"model": "judge-model", "rewrite_once": True}}
def test_a_choice_outside_the_options_is_refused():
graph = traceable(report={**STRICT, "claims": "judge"})
fid = f"{BUDGET}:write"
out = apply_fixes(graph, [fid], {fid: 99}, level="governed")
assert out["applied"] == [] and "不在候选里" in out["rejected"][0]["reason"]
def test_published_level_budget_warnings_get_no_fix():
assert f"{BUDGET}:write" not in plan(traceable(report={**STRICT, "claims": "judge"}), "published")
def test_invalid_claims_fixes_no_longer_mention_a_later_version():
graph = traceable(report={**STRICT, "claims": "sometimes"})
governed = plan(graph)["report.claims_invalid:write"]
published = plan(graph, "published")["report.claims_invalid:write"]
assert "后续版本" not in governed["label"] and "default" not in published
assert [o["value"] for o in published["options"]] == ["off", "require_citation"]
# --------------------------------------------------------------------------
# 降低要求的判断
# --------------------------------------------------------------------------
def _changed(report_before: dict, report_after: dict | None, **graph_kw) -> list[str]:
old = traceable(report=report_before, **graph_kw)
new = copy.deepcopy(old)
if report_after is None:
config(new, "write").pop("claims")
else:
config(new, "write").clear()
config(new, "write").update({"instructions": "写周报", **report_after})
return forbidden_changes(old, new)
def test_moving_up_to_judge_is_not_loosening():
assert _changed(STRICT, JUDGED) == []
assert _changed({**STRICT, "claims": "off"}, JUDGED) == []
assert _changed({"numbers": "strict", "on_violation": "fail"}, JUDGED) == []
@pytest.mark.parametrize("after", ["off", "require_citation", None])
def test_moving_down_from_judge_is_loosening(after):
report_after = None if after is None else {**JUDGED, "claims": after}
[reason] = _changed(JUDGED, report_after)
assert "「报告撰写」" in reason and "claims" in reason and "judge" in reason
def test_dropping_a_judge_set_in_the_graph_defaults_is_loosening():
[reason] = _changed({"numbers": "strict", "on_violation": "fail"}, {**STRICT, "claims": "off"},
defaults={"claims": "judge", "judge": {"max_cost_usd": 0.05}})
assert "claims" in reason and "judge" in reason
def test_changing_the_budget_is_not_loosening():
assert _changed(JUDGED, {**JUDGED, "judge": {"max_cost_usd": None}}) == []
# claims 一直是 judge 时,judge.on_unsupported 从 withhold 往回改也是放宽:不支持的结论句从「不予出具」变成
# 「降档出具」。Copilot 改图的 update_node 会把 judge 整个盖掉,补预算时写 judge={max_cost_usd: 0.05}
# 就悄悄丢了 withhold——这一类要和契约里的 on_unsupported 一样拒掉
WITHHOLD = {**JUDGED, "judge": {"max_cost_usd": 0.05, "on_unsupported": "withhold"}}
@pytest.mark.parametrize("judge, dropped", [
({"max_cost_usd": 0.05, "on_unsupported": "degrade"}, False),
({"max_cost_usd": 0.05}, True),
({"max_cost_usd": 0.05, "on_unsupported": "sometimes"}, False),
])
def test_relaxing_on_unsupported_on_the_node_is_loosening(judge, dropped):
[reason] = _changed(WITHHOLD, {**WITHHOLD, "judge": judge})
assert "「报告撰写」" in reason and "judge.on_unsupported" in reason
assert "withhold" in reason and "degrade" in reason
assert ("去掉了写明的值" in reason) is dropped, reason
def test_replacing_the_whole_judge_block_to_add_a_budget_is_loosening():
"""补预算把 judge 整个换成 {max_cost_usd}:连同裁判模型、withhold 一起丢了。"""
before = {**STRICT, "claims": "judge", "judge": {"model": "judge-model", "on_unsupported": "withhold"}}
[reason] = _changed(before, {**before, "judge": {"max_cost_usd": 0.05}})
assert "judge.on_unsupported" in reason and "withhold" in reason
assert _changed(before, {**before, "judge": {**before["judge"], "max_cost_usd": 0.05}}) == [], \
"带着原来的键补预算不算放宽"
def test_a_withhold_from_the_graph_defaults_lost_by_a_node_judge_is_loosening():
"""judge 按运行时的规则取:节点写了自己的 judge 就整个用节点的,图级 defaults 里的 withhold 不再生效。"""
defaults = {"judge": {"on_unsupported": "withhold", "max_cost_usd": 0.05}}
report = {**STRICT, "claims": "judge"}
[reason] = _changed(report, {**report, "judge": {"max_cost_usd": 0.05}}, defaults=defaults)
assert "judge.on_unsupported" in reason and "从 withhold(跟随全图默认)" in reason, reason
assert "不再跟随全图默认" in reason, "说清楚为什么全图默认里的 withhold 不生效了"
assert _changed(report, {**report, "judge": {"max_cost_usd": 0.05, "on_unsupported": "withhold"}},
defaults=defaults) == []
def test_relaxing_on_unsupported_in_the_graph_defaults_is_loosening():
old = traceable(report={**STRICT, "claims": "judge"},
defaults={"judge": {"max_cost_usd": 0.05, "on_unsupported": "withhold"}})
new = copy.deepcopy(old)
new["defaults"]["judge"]["on_unsupported"] = "degrade"
[reason] = forbidden_changes(old, new)
assert "「报告撰写」" in reason and "judge.on_unsupported" in reason
@pytest.mark.parametrize("before, after", [
(JUDGED, {**JUDGED, "judge": {"max_cost_usd": 0.05, "on_unsupported": "withhold"}}),
(WITHHOLD, {**WITHHOLD, "judge": {**WITHHOLD["judge"], "max_cost_usd": None, "rewrite_once": True}}),
(WITHHOLD, WITHHOLD),
# 升到 judge 本身是收紧:不管带不带 withhold
({**STRICT, "judge": {"on_unsupported": "withhold"}}, JUDGED),
])
def test_tightening_or_keeping_on_unsupported_is_fine(before, after):
assert _changed(before, after) == []
def test_dropping_withhold_along_with_judge_is_one_reason():
"""从 judge 退回 require_citation:claims 那一条已经说了,不再为 on_unsupported 另记一条。"""
[reason] = _changed(WITHHOLD, {**STRICT, "judge": {"max_cost_usd": 0.05}})
assert "claims" in reason and "on_unsupported" not in reason
@pytest.mark.parametrize("before", ["require_citation", "judge"])
def test_moving_claims_to_an_unknown_value_is_loosening(before):
"""require_citation、judge 改成认不出的写法:出口按认不出的走,要求没了——照样算放宽,别只落个
「没让图变好」。原来就认不出的不比较。"""
report = {**STRICT, "claims": before, **({"judge": {"max_cost_usd": 0.05}} if before == "judge" else {})}
[reason] = _changed(report, {**report, "claims": "sometimes"})
assert "「报告撰写」" in reason and before in reason and "sometimes" in reason and "不认识" in reason
assert _changed({**STRICT, "claims": "sometimes"}, {**STRICT, "claims": "off"}) == []
assert _changed({**STRICT, "claims": "off"}, {**STRICT, "claims": "sometimes"}) == []
def _contract_changed(before, after) -> list[str]:
old = traceable(claims=before)
new = copy.deepcopy(old)
contract = config(new, "done")["contract"]
if after is None:
contract.pop("claims", None)
else:
contract["claims"] = after
return forbidden_changes(old, new)
@pytest.mark.parametrize("before, after", [
("judge", "off"),
("judge", None),
("judge", "require_citation"),
({"policy": "judge", "on_unsupported": "withhold"}, "judge"),
({"policy": "judge", "on_unsupported": "withhold"}, {"policy": "judge", "on_unsupported": "degrade"}),
({"policy": "require_citation", "on_uncited": "withhold"}, "judge"),
])
def test_loosening_a_judge_contract_is_refused(before, after):
[reason] = _contract_changed(before, after)
assert "「出具」" in reason and "claims" in reason
@pytest.mark.parametrize("before, after", [
("require_citation", "judge"),
("off", {"policy": "judge", "on_unsupported": "withhold"}),
("judge", {"policy": "judge", "on_unsupported": "withhold"}),
({"policy": "require_citation", "on_uncited": "withhold"}, {"policy": "judge", "on_uncited": "withhold"}),
])
def test_tightening_a_contract_towards_judge_is_fine(before, after):
assert _contract_changed(before, after) == []
+8 -9
View File
@@ -20,7 +20,7 @@ from app.db.base import SessionLocal
from app.db.models import Run, Workflow, WorkflowVersion
from app.engine.governance import lint_for_publish, publish_issues
from app.engine.runner import run_manager
from app.engine.schema import CLAIMS_LATER, GraphSpec, validate_graph
from app.engine.schema import GraphSpec, validate_graph
from app.main import app
from app.seed import TEMPLATES
@@ -281,10 +281,10 @@ def test_graph_defaults_count_like_they_do_at_run_time():
def test_an_unsupported_claims_value_is_left_to_validate():
graph = traceable(report={**STRICT, "claims": "judge"})
graph = traceable(report={**STRICT, "claims": "sometimes"})
assert by_code(graph, "governed.report_policy") == []
[bad] = [i for i in validate_graph(GraphSpec.model_validate(graph)).issues if i.code == "report.claims_invalid"]
assert bad.level == "error" and CLAIMS_LATER in bad.message
assert bad.level == "error" and "off / require_citation" in bad.message
# --------------------------------------------------------------------------
@@ -442,7 +442,7 @@ async def test_an_already_governed_version_keeps_running_formally(client, engine
# --------------------------------------------------------------------------
# validate:报告撰写节点的 claims 只接受 off / require_citation
# validate:报告撰写节点的 claims 只接受 off / require_citation / judge(四期放开 judge)
# --------------------------------------------------------------------------
@@ -458,10 +458,9 @@ def test_supported_claims_values_pass(value):
assert claims_issues(value) == []
def test_judge_is_a_later_version():
[issue] = claims_issues("judge")
assert issue.level == "error" and issue.node_id == "write" and issue.field == "claims"
assert CLAIMS_LATER in issue.message and "off" in issue.message and "require_citation" in issue.message
def test_judge_is_supported_now():
assert claims_issues("judge") == []
assert claims_issues("judge", on_defaults=True) == []
@pytest.mark.parametrize("value", ["strict", "Require_Citation", 1, ["off"], {"policy": "judge"}])
@@ -471,5 +470,5 @@ def test_other_claims_values_are_errors(value):
def test_claims_from_graph_defaults_are_checked_too():
[issue] = claims_issues("judge", on_defaults=True)
[issue] = claims_issues("sometimes", on_defaults=True)
assert issue.node_id == "write"
+262
View File
@@ -0,0 +1,262 @@
"""结论句裁判的设置与画布校验。
- 设置里单开一组 judge:证据裁判模型、每份报告的句数 / 金额 / 时长上限、探索运行每次点击的金额上限、
全局每日金额上限。每一项都能存成 null,表示不限——存了 null 就是不限,不会被默认值悄悄盖回去
- limits 组是环境变量定的、只读,裁判设置不放进去;每日计数存在设置表里,但不从设置接口进出
- 裁判模型的优先级:节点上的 judge.model > 设置里的证据裁判模型 > Copilot 模型 > 默认 provider
- 画布校验放开 claims: judge,并校验 judge 子配置的形状:上限可以写 null(不限),rewrite_once 是布尔值,
on_unsupported 取 degrade / withhold
"""
from __future__ import annotations
import pytest
from httpx import ASGITransport, AsyncClient
from sqlalchemy import delete
from app.db.base import SessionLocal
from app.db.models import Setting
from app.engine.judge import (
JUDGE_DEFAULTS,
SETTING_GROUP,
SPEND_KEY,
Budget,
click_budget,
judge_model_spec,
judge_settings,
report_budget,
)
from app.engine.schema import CLAIMS_VALUES, JUDGE_KEYS, GraphSpec, claims_problem, validate_graph
from app.main import app
@pytest.fixture(autouse=True)
async def clean():
async with SessionLocal() as session:
await session.execute(delete(Setting).where(Setting.key.in_([SPEND_KEY, SETTING_GROUP, "copilot"])))
await session.commit()
yield
def client() -> AsyncClient:
return AsyncClient(transport=ASGITransport(app=app), base_url="http://test")
# --------------------------------------------------------------------------
# 设置
# --------------------------------------------------------------------------
def test_the_defaults_are_the_agreed_ones():
assert SETTING_GROUP == "judge" and SETTING_GROUP != "limits"
assert JUDGE_DEFAULTS == {
"provider": None, "model": None,
"report_max_claims": 40, "report_max_cost_usd": 0.05, "report_timeout_s": 30,
"click_max_cost_usd": 0.01, "daily_max_usd": 2.0,
}
async def test_settings_expose_the_judge_group_with_defaults():
async with client() as c:
got = (await c.get("/api/settings")).json()
assert got["judge"] == JUDGE_DEFAULTS
assert SPEND_KEY not in got
async def test_null_is_stored_as_unlimited_and_stays_unlimited():
async with client() as c:
r = await c.put("/api/settings", json={"values": {"judge": {
**JUDGE_DEFAULTS, "model": "judge-model", "report_max_cost_usd": None, "daily_max_usd": None,
"report_timeout_s": None, "report_max_claims": None, "click_max_cost_usd": None}}})
assert r.status_code == 200, r.text
got = (await c.get("/api/settings")).json()["judge"]
assert got["model"] == "judge-model"
assert all(got[k] is None for k in ("report_max_cost_usd", "daily_max_usd", "report_timeout_s",
"report_max_claims", "click_max_cost_usd"))
async with SessionLocal() as session:
settings = await judge_settings(session)
assert report_budget(settings, None) == Budget(max_claims=None, max_cost_usd=None, timeout_s=None,
daily_max_usd=None)
assert click_budget(settings).max_cost_usd is None
async def test_a_partial_group_keeps_the_other_defaults():
async with client() as c:
await c.put("/api/settings", json={"values": {"judge": {"daily_max_usd": 5}}})
got = (await c.get("/api/settings")).json()["judge"]
assert got == {**JUDGE_DEFAULTS, "daily_max_usd": 5}
@pytest.mark.parametrize("key,value", [
("daily_max_usd", -1), ("report_max_cost_usd", 0), ("report_max_cost_usd", "0.05"),
("report_max_claims", 2.5), ("report_max_claims", True), ("report_timeout_s", 0),
("click_max_cost_usd", float("nan")), ("model", 3), ("unknown_key", 1),
])
async def test_bad_values_are_refused_with_a_reason(key, value):
async with client() as c:
r = await c.put("/api/settings", json={"values": {"judge": {key: value}}}) if value == value else \
await c.put("/api/settings", content='{"values": {"judge": {"%s": NaN}}}' % key,
headers={"content-type": "application/json"})
assert r.status_code == 422, r.text
assert key in r.json()["detail"]
assert (await c.get("/api/settings")).json()["judge"] == JUDGE_DEFAULTS, "整组都没写进去"
async def test_the_daily_counter_does_not_travel_through_the_settings_api():
async with SessionLocal() as session:
session.add(Setting(key=SPEND_KEY, value={"date": "2026-09-29", "usd": 1.5, "calls": 3}))
await session.commit()
async with client() as c:
assert SPEND_KEY not in (await c.get("/api/settings")).json()
await c.put("/api/settings", json={"values": {SPEND_KEY: {"date": "2026-09-29", "usd": 0, "calls": 0}}})
async with SessionLocal() as session:
row = await session.get(Setting, SPEND_KEY)
assert row.value["usd"] == 1.5, "设置页整组回写不能把今天的花费清零"
async def test_todays_spend_is_readable_next_to_the_daily_cap(monkeypatch):
from app.engine import judge as judge_mod
monkeypatch.setattr(judge_mod, "_today", lambda: "2026-09-29")
async with SessionLocal() as session:
session.add(Setting(key=SPEND_KEY, value={"date": "2026-09-28", "usd": 1.9, "calls": 7}))
await session.commit()
async with client() as c:
got = (await c.get("/api/settings/judge/spend")).json()
assert got == {"date": "2026-09-29", "usd": 0.0, "calls": 0, "unpriced_calls": 0, "daily_max_usd": 2.0}, \
"昨天的花费不算今天的"
async with SessionLocal() as session:
row = await session.get(Setting, SPEND_KEY)
row.value = {"date": "2026-09-29", "usd": 0.42, "calls": 3, "unpriced_calls": 1}
await session.commit()
await c.put("/api/settings", json={"values": {"judge": {"daily_max_usd": None}}})
got = (await c.get("/api/settings/judge/spend")).json()
assert got == {"date": "2026-09-29", "usd": 0.42, "calls": 3, "unpriced_calls": 1, "daily_max_usd": None}
async def test_bad_rows_already_in_the_database_fall_back_to_defaults():
"""库里存坏了的值(旧版本、手改)读出来按默认,不让裁判拿着一个负数上限跑。"""
async with SessionLocal() as session:
session.add(Setting(key=SETTING_GROUP, value={"daily_max_usd": -3, "report_max_claims": "many",
"report_timeout_s": None}))
await session.commit()
settings = await judge_settings(session)
assert settings["daily_max_usd"] == JUDGE_DEFAULTS["daily_max_usd"]
assert settings["report_max_claims"] == JUDGE_DEFAULTS["report_max_claims"]
assert settings["report_timeout_s"] is None
def test_the_node_overrides_key_by_key():
settings = {**JUDGE_DEFAULTS, "report_max_claims": 10}
assert report_budget(settings, None) == Budget(max_claims=10, max_cost_usd=0.05, timeout_s=30, daily_max_usd=2.0)
assert report_budget(settings, {"max_cost_usd": None, "timeout_s": 5}) == \
Budget(max_claims=10, max_cost_usd=None, timeout_s=5, daily_max_usd=2.0)
# 探索运行每次点击:金额按点击的上限,句数、时长按每份报告的
assert click_budget(settings) == Budget(max_claims=10, max_cost_usd=0.01, timeout_s=30, daily_max_usd=2.0)
# --------------------------------------------------------------------------
# 裁判模型的优先级
# --------------------------------------------------------------------------
async def test_the_judge_model_priority():
async with SessionLocal() as session:
spec = await judge_model_spec(session, None)
assert (spec.provider, spec.model) == (None, None), "什么都没配:默认 provider"
assert spec.thinking == "off" and spec.max_tokens == 4096
session.add(Setting(key="copilot", value={"provider": "", "model": "copilot-model"}))
await session.commit()
assert (await judge_model_spec(session, None)).model == "copilot-model"
session.add(Setting(key=SETTING_GROUP, value={"provider": "house", "model": "judge-model"}))
await session.commit()
spec = await judge_model_spec(session, None)
assert (spec.provider, spec.model) == ("house", "judge-model")
spec = await judge_model_spec(session, {"model": "node-model"})
assert (spec.provider, spec.model) == (None, "node-model"), "节点上的写法整组生效,不和设置里的拼"
spec = await judge_model_spec(session, {"model": None, "provider": None, "max_claims": 3})
assert spec.model == "judge-model"
# --------------------------------------------------------------------------
# 画布校验
# --------------------------------------------------------------------------
def node(nid, ntype, **config):
return {"id": nid, "type": ntype, "position": {"x": 0, "y": 0}, "data": {"label": nid, "config": config}}
def graph(report: dict, defaults: dict | None = None) -> dict:
nodes = [node("start", "input"),
node("caliber", "metrics", caliber="周报口径", metrics=[{"id": "gmv", "expression": "1"}]),
node("write", "report", instructions="写周报", **report),
node("out", "output", fields=[{"name": "周报", "value": "{{ nodes.write.text }}"}])]
ids = [n["id"] for n in nodes]
g = {"nodes": nodes, "edges": [{"source": a, "target": b} for a, b in zip(ids, ids[1:])]}
if defaults:
g["defaults"] = defaults
return g
def issues(g, code=None):
return [i for i in validate_graph(GraphSpec.model_validate(g)).issues
if i.code in ((code,) if code else ("report.claims_invalid", "report.judge_invalid"))]
def test_judge_is_a_supported_policy_now():
assert CLAIMS_VALUES == ("off", "require_citation", "judge")
assert claims_problem("judge") is None
assert issues(graph({"claims": "judge"})) == []
assert issues(graph({}, defaults={"claims": "judge"})) == []
def test_every_limit_can_be_unlimited():
judge = {"provider": None, "model": None, "max_claims": None, "max_cost_usd": None, "timeout_s": None,
"rewrite_once": True, "on_unsupported": "withhold"}
assert set(judge) == set(JUDGE_KEYS)
assert issues(graph({"claims": "judge", "judge": judge})) == []
assert issues(graph({"claims": "judge", "judge": {"max_claims": 40, "max_cost_usd": 0.05, "timeout_s": 30,
"model": "judge-model", "on_unsupported": "degrade"}})) == []
@pytest.mark.parametrize("judge,field,words", [
("fast", "judge", "对象"),
({"max_cost": 1}, "judge.max_cost", "max_cost_usd"),
({"max_claims": 0}, "judge.max_claims", "正整数"),
({"max_claims": 2.5}, "judge.max_claims", "正整数"),
({"max_claims": True}, "judge.max_claims", "正整数"),
({"max_cost_usd": -1}, "judge.max_cost_usd", "大于 0"),
({"max_cost_usd": "0.05"}, "judge.max_cost_usd", "大于 0"),
({"timeout_s": 0}, "judge.timeout_s", "大于 0"),
({"rewrite_once": "yes"}, "judge.rewrite_once", "true 或 false"),
({"on_unsupported": "ignore"}, "judge.on_unsupported", "degrade / withhold"),
({"model": 3}, "judge.model", "模型 id"),
])
def test_a_malformed_judge_block_is_an_error(judge, field, words):
[issue] = issues(graph({"claims": "judge", "judge": judge}), "report.judge_invalid")
assert issue.level == "error" and issue.node_id == "write" and issue.field == field
assert words in issue.message, issue.message
if field in ("judge.max_claims", "judge.max_cost_usd", "judge.timeout_s", "judge.model"):
assert "null" in issue.message, "上限、模型都说清写 null 是什么意思"
def test_graph_defaults_are_checked_like_at_run_time():
[issue] = issues(graph({"claims": "judge"}, defaults={"judge": {"max_claims": -1}}), "report.judge_invalid")
assert issue.field == "judge.max_claims"
def test_the_judge_block_only_matters_when_claims_is_judge():
assert issues(graph({"claims": "require_citation", "judge": {"max_claims": -1}})) == []
def test_judging_with_the_writing_model_is_a_warning():
[warn] = [i for i in validate_graph(GraphSpec.model_validate(graph(
{"claims": "judge", "model": "writer-x", "judge": {"model": "writer-x"}}))).issues
if i.code == "report.judge_same_model"]
assert warn.level == "warning" and warn.field == "judge.model" and "自己审自己" in warn.message
assert not [i for i in validate_graph(GraphSpec.model_validate(graph(
{"claims": "judge", "model": "writer-x", "judge": {"model": "judge-y"}}))).issues
if i.code == "report.judge_same_model"]
+10 -10
View File
@@ -1,10 +1,10 @@
"""报告节点的结论句策略 claims:off(默认,和以前一样)/ require_citation(没挂依据的结论句计入缺口)。
结论句要靠裁判模型判断支不支持,那是后续版本的事;这一版只做确定性的一半:
这里测确定性的一半(模型裁判 judge 见 test_report_judge.py):
- require_citation 时写作提示明确要求每句结论挂 [[see:…]];没挂的照常产出、计数,
出口按出具档位降档(io.py),报告节点自己不为它失败、不为它重写
- report.checked 事件和节点产出都带上 claims,出口据此判档
- 写 judge 时节点直接报错「结论句裁判在后续版本支持」,不能悄悄当成 off
- 写了不认识的值节点直接报错,不能悄悄当成 off
- 升级前发起的运行一个字都不变
"""
from __future__ import annotations
@@ -21,7 +21,7 @@ from app.db.models import Run, RunEvent
from app.engine import runner as runner_mod
from app.engine.evidence import CLAIMS_RULE, build_catalog, compose_doc, uncited_claims
from app.engine.runner import run_manager
from app.engine.schema import CLAIMS_LATER, REPORT_CLAIMS, claims_problem
from app.engine.schema import CLAIMS_VALUES, claims_problem
@pytest.fixture(autouse=True)
@@ -131,14 +131,14 @@ async def test_require_citation_asks_for_support_but_does_not_block(monkeypatch)
assert flagged["text"] == "增长主要来自新客。" and doc["markdown"][slice(*flagged["span"])] == flagged["text"]
async def test_judge_is_refused_until_it_exists(monkeypatch):
async def test_a_bad_value_is_refused_before_any_model_call(monkeypatch):
seen = script(monkeypatch)
with pytest.raises(ValueError) as info: # 画布校验先拦(E3-gov 的 validate)
await run_manager.start(graph=weekly(claims="judge"), input_payload={})
assert CLAIMS_LATER in str(info.value)
with pytest.raises(ValueError) as info: # 画布校验先拦
await run_manager.start(graph=weekly(claims="sometimes"), input_payload={})
assert claims_problem("sometimes") in str(info.value)
lenient(monkeypatch)
row = await finish(weekly(claims="judge"))
assert row.status == "failed" and claims_problem("judge") in (row.error or ""), row.error
row = await finish(weekly(claims="sometimes"))
assert row.status == "failed" and claims_problem("sometimes") in (row.error or ""), row.error
assert seen == [], "不支持的策略不该先花一次模型调用"
@@ -173,7 +173,7 @@ def test_the_supported_policies_match_the_validator():
from app.engine.nodes.report import CLAIMS, report_claims
from app.engine.schema import GraphSpec
assert CLAIMS == REPORT_CLAIMS == ("off", "require_citation")
assert CLAIMS == CLAIMS_VALUES == ("off", "require_citation", "judge")
spec = GraphSpec.model_validate(weekly(claims="require_citation"))
assert report_claims(spec, spec.node_map()["write"]) == "require_citation"
spec = GraphSpec.model_validate(weekly())
+495
View File
@@ -0,0 +1,495 @@
"""报告撰写节点的 claims: judge:结论句由另一个模型按证据逐句判断(四期)。
- 正式运行在节点里裁判:写完、核对完之后判,判定写进报告文档的 unit.verdict,随文档一起封存
- 探索运行不在节点里花这笔钱:候选句标「未裁判 · 按需」,点开哪句再判哪句(接口在 B 段)
- 判定只标注、不改写正文;裁判没跑成、触顶了,照样产出报告,没判的记未裁判并留下缺口
(出口据此不能判完整出具,接线在 B 段)
- rewrite_once(默认关):把证据不支持的句子和理由交回写作者只改这几句,再判一次,最多一轮;
改出了新的违规就不采用
- 裁判调用包在 @task 里、追加在节点已有的调用之后:节点「接着跑」时不重复调用、不重复计费;
升级前发起的运行一样都不加,停在审批上的运行恢复后照常
"""
from __future__ import annotations
import asyncio
import json
import re
import pytest
from langchain_core.messages import AIMessage
from sqlalchemy import delete, select
from app.core import artifact_store
from app.db.base import SessionLocal
from app.db.models import Run, RunEvent, Setting
from app.engine import judge as judge_mod
from app.engine import replay
from app.engine import runner as runner_mod
from app.engine.evidence import CLAIMS_RULE, iter_units
from app.engine.judge import JUDGE_ROLE, JUDGE_RULE, SPEND_KEY, SUMMARY_KEYS, daily_spend
from app.engine.nodes import report as report_mod
from app.engine.runner import run_manager, verify_manifest
from app.providers.mock_model import MockChatModel
PRICED = "claude-sonnet-5"
METRICS = [{"id": "gmv", "name": "销售额", "unit": "元", "decimals": 1, "expression": "vars.kpi.gmv"}]
#: 一句挂了依据的结论、一句没挂依据的归因、一句连接性的话
TEXT = "本周销售额 [[m:gmv]]。[[see:m:gmv]]增长主要来自新客首单。下面看细节。"
CITED, CAUSAL, LINK = "本周销售额 45,678.5元。", "增长主要来自新客首单。", "下面看细节。"
@pytest.fixture(autouse=True)
async def engine_up():
await run_manager.setup()
async with SessionLocal() as session:
await session.execute(delete(Setting).where(Setting.key.in_([SPEND_KEY, "judge", "copilot"])))
await session.commit()
yield
await run_manager.shutdown()
def node(nid, ntype, **config):
return {"id": nid, "type": ntype, "position": {"x": 0, "y": 0}, "data": {"label": nid, "config": config}}
def weekly(*, gate=False, **report):
nodes = [
node("start", "input", fields=[{"name": "week", "default": "2026-W37"}]),
node("fetch", "transform", mode="expression", expression="{'gmv': 45678.5}", assign_to="kpi"),
node("caliber", "metrics", caliber="周报口径", caliber_version="v2", metrics=METRICS),
*([node("gate", "human", mode="approve", title="确认口径")] if gate else []),
node("write", "report", instructions="写周报", **report),
node("out", "output", fields=[{"name": "周报", "value": "{{ nodes.write.text }}"}]),
]
ids = [n["id"] for n in nodes]
return {"nodes": nodes, "edges": [{"source": a, "target": b} for a, b in zip(ids, ids[1:])]}
class Models:
"""写作者和裁判两份剧本。写作者依次回 writer 里的几段(用完了一直回最后一段);裁判按句子里的字给判定
(verdicts:{句中片段: 判定},没对上的都是 supported)。"""
def __init__(self, monkeypatch, writer=(TEXT,), verdicts=None, *, judge_model=PRICED, fail_judge=None):
self.writes: list[list] = []
self.judges: list[str] = []
verdicts = verdicts or {}
def decide(model, messages):
if JUDGE_ROLE in str(messages[0].content):
request = str(messages[-1].content)
self.judges.append(request)
if fail_judge:
raise RuntimeError(fail_judge)
items = []
for uid, line in re.findall(r"^\[(u\d+)\]([^\n]*)$", request, re.M):
status = next((s for frag, s in verdicts.items() if frag in line), "supported")
items.append({"unit": uid, "verdict": status, "rationale": f"判的是:{line[:20]}", "used": []})
return AIMessage(content=json.dumps({"items": items}, ensure_ascii=False))
self.writes.append(list(messages))
return AIMessage(content=writer[min(len(self.writes), len(writer)) - 1])
monkeypatch.setattr(MockChatModel, "_decide", decide)
async def judge_model_of(session, spec):
return MockChatModel(model_name=judge_model), judge_model
monkeypatch.setattr(judge_mod, "get_chat_model", judge_model_of)
def spy_tasks(monkeypatch) -> list[str]:
"""报告节点里每调用一次 @task 记一笔名字:证明加进来的 task 只在该出现的地方出现。"""
from langgraph.func import task as real
calls: list[str] = []
def spying(func=None, **kwargs):
def wrap(fn):
inner = real(fn, **kwargs)
def call(*args, **kw):
calls.append(fn.__name__)
return inner(*args, **kw)
return call
return wrap(func) if func is not None else wrap
monkeypatch.setattr(report_mod, "task", spying)
return calls
async def finish(graph, *, run_class="formal", statuses=("succeeded", "failed")) -> Run:
run = await run_manager.start(graph=graph, input_payload={}, run_class=run_class)
return await wait(run.id, statuses)
async def wait(run_id, statuses=("succeeded", "failed")) -> Run:
for _ in range(400):
await asyncio.sleep(0.05)
async with SessionLocal() as session:
row = await session.get(Run, run_id)
if row.status in statuses and (row.status == "interrupted" or row.manifest_seq is not None):
return row
raise AssertionError(f"run {run_id} 没跑完:{row.status}")
async def events(run_id: str, etype: str, node_id: str | None = "write") -> list[RunEvent]:
async with SessionLocal() as session:
q = select(RunEvent).where(RunEvent.run_id == run_id, RunEvent.type == etype)
if node_id:
q = q.where(RunEvent.node_id == node_id)
return list((await session.execute(q.order_by(RunEvent.seq))).scalars())
async def report_of(run_id: str) -> tuple[dict, dict, dict]:
"""(report.checked 的载荷, 报告文档, 节点产出)"""
[checked] = await events(run_id, "report.checked")
[finished] = await events(run_id, "node.finished")
return checked.data, artifact_store.load(checked.data["doc_artifact"]), artifact_store.load(finished.data["artifact"])
def verdicts(doc) -> dict[str, dict | None]:
md = doc["markdown"]
return {md[u["span"][0]:u["span"][1]].strip(): u.get("verdict") for _, u in iter_units(doc)}
def unit_ids(doc) -> dict[str, str]:
md = doc["markdown"]
return {md[u["span"][0]:u["span"][1]].strip(): u["id"] for _, u in iter_units(doc)}
async def judge_ends(run_id: str) -> list[dict]:
return [e.data for e in await events(run_id, "llm.end") if e.data.get("purpose") == "judge"]
# --------------------------------------------------------------------------
# 正式运行:节点内裁判,判定随文档封存
# --------------------------------------------------------------------------
async def test_a_formal_run_judges_inside_the_node_and_seals_the_verdicts(monkeypatch):
models = Models(monkeypatch, verdicts={"新客": "unsupported"})
row = await finish(weekly(claims="judge", judge={"on_unsupported": "withhold"}))
assert row.status == "succeeded", row.error
assert len(models.writes) == 1 and len(models.judges) == 1
checked, doc, output = await report_of(row.id)
got = verdicts(doc)
assert got[CITED]["status"] == "supported" and got[CITED]["judge"] == PRICED
assert got[CITED]["post_seal"] is False and got[CITED]["rationale"]
assert got[CAUSAL]["status"] == "unsupported"
assert got[LINK] is None, "连接性的话不送裁判"
assert CITED in models.judges[0] and CAUSAL in models.judges[0] and LINK not in models.judges[0]
summary = checked["judge"]
assert tuple(summary) == SUMMARY_KEYS, "B 段照着这份键读:不能悄悄少一个、多一个"
assert summary["mode"] == "inline" and summary["model"] == PRICED and summary["on_unsupported"] == "withhold"
assert summary["counts"] == {"supported": 1, "partial": 0, "unsupported": 1, "not_a_claim": 0, "unjudged": 0}
assert summary["complete"] is True and summary["gaps"] == []
assert checked["claims"] == output["claims"] == "judge" and output["judge"] == summary
assert doc["judge"]["mode"] == "inline" and doc["judge"]["counts"] == summary["counts"]
assert {k: doc["stats"][k] for k in ("supported", "partial", "unsupported", "not_a_claim", "unjudged")} \
== summary["counts"]
[end] = await judge_ends(row.id)
assert end["model"] == PRICED and end["cost_usd"] >= 0
verdict = await verify_manifest(row.id)
assert verdict["ok"] is True
[event] = await events(row.id, "report.checked")
assert event.seq <= row.manifest_seq, "判定在封存范围内"
async def test_the_writer_is_told_that_claims_will_be_judged(monkeypatch):
models = Models(monkeypatch)
await finish(weekly(claims="judge"))
prompt = "\n".join(str(m.content) for m in models.writes[0])
assert CLAIMS_RULE in prompt and JUDGE_RULE in prompt
async def test_explore_runs_leave_it_to_the_reader(monkeypatch):
"""探索运行默认按需裁判:节点里不调用,候选句标「未裁判 · 按需」,连接性的话什么都不标。"""
models = Models(monkeypatch)
row = await finish(weekly(claims="judge"), run_class="exploratory")
assert row.status == "succeeded", row.error
assert models.judges == []
checked, doc, output = await report_of(row.id)
got = verdicts(doc)
for text in (CITED, CAUSAL):
assert got[text]["status"] == "unjudged" and got[text]["reason"] == "on_demand"
assert got[text]["judge"] is None and got[text]["post_seal"] is False
assert got[LINK] is None
assert checked["judge"]["mode"] == "on_demand" and checked["judge"]["gaps"] == []
assert tuple(checked["judge"]) == SUMMARY_KEYS and checked["judge"]["limits"] is None
assert output["judge"] == checked["judge"] and await judge_ends(row.id) == []
async def test_a_failed_judge_leaves_a_gap_not_a_failed_report(monkeypatch):
Models(monkeypatch, fail_judge="网关超时")
row = await finish(weekly(claims="judge"))
assert row.status == "succeeded", row.error
checked, doc, output = await report_of(row.id)
assert {verdicts(doc)[t]["reason"] for t in (CITED, CAUSAL)} == {"error"}
summary = checked["judge"]
assert summary["complete"] is False and summary["counts"]["unjudged"] == 2
assert summary["gaps"] and "网关超时" in summary["gaps"][0]
assert output["judge"]["gaps"] == summary["gaps"]
[end] = await judge_ends(row.id)
assert "网关超时" in end["error"]
async def test_hitting_a_limit_is_reported_not_raised(monkeypatch):
models = Models(monkeypatch)
row = await finish(weekly(claims="judge", judge={"max_claims": 1}))
assert row.status == "succeeded", row.error
checked, doc, _ = await report_of(row.id)
assert verdicts(doc)[CITED]["status"] == "supported", "先判挂了依据的"
assert (verdicts(doc)[CAUSAL]["status"], verdicts(doc)[CAUSAL]["reason"]) == ("unjudged", "max_claims")
assert checked["judge"]["limits_hit"] == ["max_claims"] and "已到上限" in checked["judge"]["gaps"][0]
assert CAUSAL not in models.judges[0]
async def test_the_node_budget_overrides_the_settings_and_null_means_unlimited(monkeypatch):
"""节点 judge 里写了的键盖过设置;写 null 就是不限(不是「没写」)。"""
async with SessionLocal() as session:
session.add(Setting(key="judge", value={"report_max_claims": 1, "report_max_cost_usd": 1e-9}))
await session.commit()
Models(monkeypatch)
row = await finish(weekly(claims="judge", judge={"max_cost_usd": None}))
checked, doc, _ = await report_of(row.id)
assert verdicts(doc)[CITED]["status"] == "supported"
assert verdicts(doc)[CAUSAL]["reason"] == "max_claims", "max_claims 没写,取设置里的 1"
assert checked["judge"]["limits"]["max_cost_usd"] is None and checked["judge"]["limits"]["max_claims"] == 1
async def test_judging_yourself_is_called_out(monkeypatch):
Models(monkeypatch, judge_model="mock-fast")
row = await finish(weekly(claims="judge"))
assert row.status == "succeeded", row.error
[warn] = [e.data for e in await events(row.id, "log") if e.data.get("code") == "judge_same_model"]
assert "mock-fast" in warn["message"] and "自己审自己" in warn["message"]
# --------------------------------------------------------------------------
# rewrite_once
# --------------------------------------------------------------------------
REWRITTEN = "本周销售额 [[m:gmv]]。[[see:m:gmv]]新客首单的贡献要等新客维度的数据再看。下面看细节。"
async def test_rewrite_once_hands_back_only_the_unsupported_sentences(monkeypatch):
models = Models(monkeypatch, writer=(TEXT, REWRITTEN),
verdicts={"主要来自新客": "unsupported", "要等新客维度": "not_a_claim"})
row = await finish(weekly(claims="judge", judge={"rewrite_once": True}))
assert row.status == "succeeded", row.error
assert len(models.writes) == 2 and len(models.judges) == 2
request = str(models.writes[1][-1].content)
assert CAUSAL in request and "判的是:" in request and CITED not in request, "只交回不支持的那几句和理由"
# 第二次只判改过的句子:没改的那句直接用第一次的判定
assert "要等新客维度" in models.judges[1] and CITED not in models.judges[1]
checked, doc, output = await report_of(row.id)
assert "新客首单的贡献要等新客维度的数据再看。" in doc["markdown"] and CAUSAL not in doc["markdown"]
got = verdicts(doc)
assert got[CITED]["status"] == "supported" and got["新客首单的贡献要等新客维度的数据再看。"]["status"] == "not_a_claim"
rewrite = checked["judge"]["rewrite"]
assert rewrite["applied"] is True and rewrite["sentences"] == [CAUSAL] and checked["judge"]["reused"] == 1
# 封存的是改写稿:units 按句子认封存文档里的编号,交回去的那句已经改掉了,对不上就是 null——
# 不能拿原稿的编号去指改写稿里碰巧同号的另一句;改写稿里新出来的句子在 changed 里
ids = unit_ids(doc)
assert rewrite["units"] == [None]
assert rewrite["changed"] == [ids["新客首单的贡献要等新客维度的数据再看。"]]
assert output["text"] == doc["markdown"]
[log] = [e.data for e in await events(row.id, "log") if e.data.get("code") == "report_rewrite"]
assert "1 句" in log["message"]
assert len(await judge_ends(row.id)) == 2
async def test_rewrite_once_is_off_by_default(monkeypatch):
models = Models(monkeypatch, writer=(TEXT, REWRITTEN), verdicts={"主要来自新客": "unsupported"})
row = await finish(weekly(claims="judge"))
assert row.status == "succeeded", row.error
assert len(models.writes) == 1 and len(models.judges) == 1
checked, doc, _ = await report_of(row.id)
assert verdicts(doc)[CAUSAL]["status"] == "unsupported" and checked["judge"].get("rewrite") is None
async def test_a_rewrite_that_breaks_the_numbers_is_not_used(monkeypatch):
models = Models(monkeypatch, writer=(TEXT, "本周销售额 [[m:gmv]]。[[see:m:gmv]]新客贡献了 45678 元。下面看细节。"),
verdicts={"主要来自新客": "unsupported"})
row = await finish(weekly(claims="judge", judge={"rewrite_once": True}))
assert row.status == "succeeded", row.error
assert len(models.writes) == 2 and len(models.judges) == 1, "改坏了的那一稿不再花钱判"
checked, doc, output = await report_of(row.id)
assert CAUSAL in doc["markdown"] and "45678" not in doc["markdown"]
assert verdicts(doc)[CAUSAL]["status"] == "unsupported"
rewrite = checked["judge"]["rewrite"]
assert rewrite["applied"] is False and "45678" in rewrite["reason"]
# 封存的是原稿:units 就是原稿(也就是封存文档)里的编号
assert rewrite["units"] == [unit_ids(doc)[CAUSAL]] and rewrite["changed"] == []
def with_schema(monkeypatch):
"""报告的目录里加一张表:实体层有了核对的底,反引号里的名字才会被认出是编造的。"""
from app.engine.evidence import _entity_entries
real = report_mod.report_catalog
def catalog_with_schema(state, spec, node):
return {**real(state, spec, node), **_entity_entries(
[{"artifact": "d" * 64, "source": "shop", "tables": {"orders": ["order_id", "amount"]}}], [])}
monkeypatch.setattr(report_mod, "report_catalog", catalog_with_schema)
async def test_a_rewrite_that_makes_up_a_name_is_not_used(monkeypatch):
"""改写稿冒出一个编造的表名(只标注、不拦的那类违规):同样是原稿没有的新违规,不采用。"""
with_schema(monkeypatch)
models = Models(monkeypatch, writer=(TEXT, "本周销售额 [[m:gmv]]。[[see:m:gmv]]新客首单的贡献要看 `refund_detail` 表。"
"下面看细节。"),
verdicts={"主要来自新客": "unsupported"})
row = await finish(weekly(claims="judge", judge={"rewrite_once": True}))
assert row.status == "succeeded", row.error
assert len(models.writes) == 2 and len(models.judges) == 1
checked, doc, _ = await report_of(row.id)
assert CAUSAL in doc["markdown"] and "refund_detail" not in doc["markdown"]
rewrite = checked["judge"]["rewrite"]
assert rewrite["applied"] is False and "refund_detail" in rewrite["reason"]
[log] = [e.data for e in await events(row.id, "log") if e.data.get("code") == "report_rewrite_rejected"]
assert "refund_detail" in log["message"]
async def test_a_rewrite_that_trades_one_violation_for_another_is_not_used(monkeypatch):
"""去掉一处违规、又冒出另一处:总数没变,可冒出来的是原稿没有的,一样不采用。"""
draft = "本周销售额 [[m:gmv]]。[[see:m:gmv]]增长主要来自新客首单,共 1200 单。下面看细节。"
swapped = "本周销售额 [[m:gmv]]。[[see:m:gmv]]新客首单约占 3500 单。下面看细节。"
models = Models(monkeypatch, writer=(draft, draft, swapped), verdicts={"主要来自新客": "unsupported"})
row = await finish(weekly(claims="judge", on_violation="flag", max_repairs=1, judge={"rewrite_once": True}))
assert row.status == "succeeded", row.error
assert len(models.writes) == 3 and len(models.judges) == 1
checked, doc, _ = await report_of(row.id)
assert "1200" in doc["markdown"] and "3500" not in doc["markdown"]
rewrite = checked["judge"]["rewrite"]
assert rewrite["applied"] is False and "3500" in rewrite["reason"] and "1200" not in rewrite["reason"]
def test_new_violations_are_counted_one_by_one():
"""改写稿的违规逐个和原稿抵消:原样留着的不算新的,同一个裸数字多写一处、只标注的那类,都算新的。"""
number = {"code": "uncited_number", "text": "1200", "message": "数字「1200」没有出处"}
name = {"code": "unknown_entity", "text": "refund_detail", "message": "可能是编造的名字"}
see = {"code": "unresolved_ref", "ref": "Q9", "message": "依据 [[see:Q9]] 解析不了"}
assert report_mod._new_violations([number, name, see], [see, number, name]) == []
assert report_mod._new_violations([number], [number, number]) == [number]
assert report_mod._new_violations([number], [name]) == [name]
assert report_mod._new_violations([number, see], []) == []
async def test_a_second_round_that_hits_a_limit_names_the_configured_limit(monkeypatch):
"""第二轮用的是第一轮剩下的钱,可说给人看的上限是节点上配的那个数,不是剩下的零头。"""
Models(monkeypatch, writer=(TEXT, REWRITTEN), verdicts={"主要来自新客": "unsupported"})
# 每句估 1 分钱、回复不带用量(实际就按估算记):第一轮两句花掉 2 分,第二轮只剩半分,一句也装不下
monkeypatch.setattr(judge_mod.JudgeRequest, "cost", lambda self, model_id, units: 0.01 * len(units))
monkeypatch.setattr(judge_mod, "_usage_of", lambda message, model_id: {"input_tokens": 0, "output_tokens": 0})
row = await finish(weekly(claims="judge", judge={"rewrite_once": True, "max_cost_usd": 0.025}))
assert row.status == "succeeded", row.error
checked, doc, _ = await report_of(row.id)
v = verdicts(doc)["新客首单的贡献要等新客维度的数据再看。"]
assert (v["status"], v["reason"]) == ("unjudged", "max_cost_usd")
assert "$0.025" in v["rationale"] and "0.005" not in v["rationale"], v["rationale"]
summary = checked["judge"]
assert summary["rewrite"]["applied"] is True and summary["limits"]["max_cost_usd"] == 0.025
assert "$0.025" in summary["gaps"][0] and "0.005" not in summary["gaps"][0], summary["gaps"]
[log] = [e.data for e in await events(row.id, "log") if e.data.get("code") == "judge_limit"]
assert "$0.025" in log["message"] and "0.005" not in log["message"]
# --------------------------------------------------------------------------
# 重放安全
# --------------------------------------------------------------------------
async def test_continuing_after_a_failure_does_not_judge_or_charge_twice(monkeypatch):
models = Models(monkeypatch)
real = report_mod._store
broken = {"on": True}
async def flaky(doc, ctx):
if broken["on"]:
raise RuntimeError("落盘时断了")
return await real(doc, ctx)
monkeypatch.setattr(report_mod, "_store", flaky)
run = await run_manager.start(graph=weekly(claims="judge"), input_payload={}, run_class="formal")
row = await wait(run.id)
assert row.status == "failed" and "落盘时断了" in (row.error or ""), row.error
assert len(models.judges) == 1
before = await daily_spend()
await run_manager.wait_idle(run.id)
broken["on"] = False
await run_manager.continue_failed(run.id)
row = await wait(run.id)
assert row.status == "succeeded", row.error
assert len(models.judges) == 1, "接着跑时裁判又调了一次:裁判的结果没进 checkpoint"
assert await daily_spend() == before, "接着跑时每日计数又记了一次"
assert len(await judge_ends(row.id)) == 1
_, doc, _ = await report_of(row.id)
assert verdicts(doc)[CITED]["status"] == "supported"
async def test_a_changed_draft_is_judged_afresh_when_continuing(monkeypatch):
"""接着跑时写作者重写了一稿(写作调用不进 checkpoint),缓存的判定是上一稿的:不能拿来用。"""
models = Models(monkeypatch, writer=(TEXT, "本周销售额 [[m:gmv]]。[[see:m:gmv]]增长主要来自老客复购。下面看细节。"))
real = report_mod._store
broken = {"on": True}
async def flaky(doc, ctx):
if broken["on"]:
raise RuntimeError("落盘时断了")
return await real(doc, ctx)
monkeypatch.setattr(report_mod, "_store", flaky)
run = await run_manager.start(graph=weekly(claims="judge"), input_payload={}, run_class="formal")
assert (await wait(run.id)).status == "failed"
await run_manager.wait_idle(run.id)
broken["on"] = False
await run_manager.continue_failed(run.id)
row = await wait(run.id)
assert row.status == "succeeded", row.error
assert len(models.judges) == 2 and "老客复购" in models.judges[1]
_, doc, _ = await report_of(row.id)
assert verdicts(doc)["增长主要来自老客复购。"]["status"] == "supported"
async def test_runs_from_before_the_upgrade_never_judge(monkeypatch):
"""升级前发起的运行(没有护栏快照):claims 一律按 off,节点里一个 task 都不多调用,产出一个键都不多。"""
async def no_snapshot(session):
return None
monkeypatch.setattr(runner_mod, "_agent_limits", no_snapshot)
models = Models(monkeypatch)
calls = spy_tasks(monkeypatch)
row = await finish(weekly(claims="judge"))
assert row.status == "succeeded", row.error
assert models.judges == [] and calls == []
checked, doc, output = await report_of(row.id)
assert "judge" not in checked and "claims" not in checked and "judge" not in output and "judge" not in doc
assert all(v is None for v in verdicts(doc).values())
async def test_runs_paused_at_an_approval_resume_with_tasks_in_place(monkeypatch):
"""停在审批上的运行恢复:审批节点照旧只问一次;报告节点在恢复之后才第一次执行,裁判 task 只调一次、
排在节点所有已有调用之后。没开裁判的报告节点一个 task 都不调用(和升级前一样)。"""
assert replay.PROTOCOL == 2, "没有改动已有 task 的顺序,重放协议不升版"
for claims, expected in (("require_citation", []), ("judge", ["judge_step"])):
models = Models(monkeypatch)
calls = spy_tasks(monkeypatch)
paused = await finish(weekly(gate=True, claims=claims), statuses=("interrupted", "failed"))
assert paused.status == "interrupted", paused.error
await run_manager.resume(paused.id, {"approved": True})
row = await wait(paused.id)
assert row.status == "succeeded", row.error
assert calls == expected, (claims, calls)
assert len(models.judges) == len(expected)
assert len(await events(row.id, "human.requested", "gate")) == 1
checked, _, _ = await report_of(row.id)
assert ("judge" in checked) is (claims == "judge")