mirror of
https://github.com/ThearchyHelios/AgentLab.git
synced 2026-11-19 19:12:01 +00:00
可点击证据第四期:结论句由模型裁判;只引单元格的问数据图能发成受管
一、结论句裁判
- 报告节点新增 claims: judge:写完、核对完之后,把结论句和它引用的证据摘录交给裁判模型,
判为支持 / 部分支持 / 不支持 / 不是结论,附一句理由;判定只标注,不改写正文
- 送裁判的句子按固定规则预筛:带引用的、含方向词或因果词的送去判,短句、过渡句不送
- 证据摘录:指标写名称、值和原式;查询给 SQL 前 300 字和被引用的行,被引用的列一定给到;遮罩列一律不给;
规则里写明「数字已经由系统核对过,不要复算」
- 一份报告合并成一次调用,超出句数就分批;结构化输出失败时退回 JSON 解析,再失败记为未裁判
- 裁判模型的优先级:节点上写的 > 设置里的「证据裁判模型」> Copilot 的模型;思考关闭;
和写作模型相同时给警告(等于自己审自己)
- rewrite_once(默认关):把不支持的句子和理由交回写作者只改这几句,再裁一次;
改写稿冒出原稿没有的问题就不采用,保留原稿和原来的判定
二、成本上限
- 每份报告的句数、金额、时长,探索运行每次点击的金额,全局每日金额,每一项都可以自定义,也都可以设成不限;
默认 40 句、$0.05、30 秒,每次点击 $0.01,每天 $2
- 调用前先估算,超出上限按优先级截断:先判带引用的,再判含方向词、因果词的,其余记为未裁判并写明「已到上限」;
不报错,也不让报告失败
- 每日花费按本地日期计数,持久化在设置表里;重跑同一次运行不重复计费;价格目录里没有的模型只计次数,并写明估不出金额
三、正式运行与探索运行
- 正式运行在报告节点内裁判,判定写进报告文档、随文档封存;不支持的按 on_unsupported(degrade / withhold)降档或不予出具,
裁判没跑成记缺口,不能判完整出具
- 探索运行按需裁判:POST /api/runs/{id}/evidence/judge,点开哪句判哪句;判定作为 evidence.judged 事件追加在封存之后,
标「封存后追加」,封存核对仍然一致;只读封存范围内的工件;同一句不重复判;运行在裁判期间被接着跑时不记录
- 正式运行调这个接口返回 409,写明「正式运行的裁判在节点内完成」
四、治理与自动修复
- G3 放开 judge:受管级别写 judge 必须写预算(judge.max_cost_usd,或显式写 null 表示不限)
- 发布前自动修复:缺预算时给选项(默认金额 / 不限),不替人选;
从 require_citation 改成 judge 不算降低要求,judge 改回 off、withhold 放宽成 degrade 算降低要求
- Copilot 手册:受管模板可以写 judge 并带预算;问数据按需裁判,不替人打开结论句策略
五、界面
- 结论句句末放徽标:支持 ◆、部分支持 ◇、不支持 ! 整句浅底、未裁判 ?;不用表示「已核对」的绿色;n / N 也跳到不支持的句子
- 证据面板里「模型的解释」:带「模型判断 · 模型名 · 非确定」,封存后追加的另外标出;
「请模型判断这句」只在探索运行、还没判过时出现,触顶时写明已到上限和怎么调
- 出具横幅多一段结论句计数;设置页新增「证据裁判」一组(裁判模型、各项上限、今日花费),设成不限时写明费用受什么约束;
报告节点可以配 judge;运行轨迹里裁判相关的事件都有中文标题
六、门禁修正
- 引用模式下,图里没有口径卡(只引查询单元格)的,出具契约不再要求 metrics_from,受管门禁也不再要求 required:
strict 加 cells 已经保证每个数都点得开、对不上就不予出具;有口径卡的照旧要求。
以前问数据的图怎么都发不成受管,Copilot 补上契约也会被判「冒出新的问题」作废
This commit is contained in:
@@ -118,22 +118,21 @@ def test_published_level_warnings_get_no_fix():
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_judge_becomes_require_citation_under_governed():
|
||||
graph = traceable(report={**STRICT, "claims": "judge"})
|
||||
def test_an_invalid_claims_value_becomes_require_citation_under_governed():
|
||||
graph = traceable(report={**STRICT, "claims": "sometimes"})
|
||||
fid = "report.claims_invalid:write"
|
||||
fix = plan(graph)[fid]
|
||||
assert fix["kind"] == "auto" and fix["preview"] == {"field": "claims", "before": "judge",
|
||||
assert fix["kind"] == "auto" and fix["preview"] == {"field": "claims", "before": "sometimes",
|
||||
"after": "require_citation"}
|
||||
out = apply_fixes(graph, [fid], level="governed")
|
||||
assert out["ok"] is True and config(out["graph"], "write")["claims"] == "require_citation"
|
||||
|
||||
|
||||
def test_an_invalid_claims_value_is_a_choice_under_published():
|
||||
graph = traceable(report={"claims": "judge"})
|
||||
graph = traceable(report={"claims": "sometimes"})
|
||||
fid = "report.claims_invalid:write"
|
||||
fix = plan(graph, "published")[fid]
|
||||
assert fix["kind"] == "choice" and [o["value"] for o in fix["options"]] == ["off", "require_citation"]
|
||||
assert fix["default"] == "require_citation" # 离裁判最近的是「每句结论挂引用」,仍然要人确认
|
||||
assert apply_fixes(graph, [fid], level="published")["applied"] == []
|
||||
out = apply_fixes(graph, [fid], {fid: "off"}, level="published")
|
||||
assert out["applied"] == [fid] and config(out["graph"], "write")["claims"] == "off" and out["ok"] is True
|
||||
|
||||
@@ -0,0 +1,111 @@
|
||||
"""结论句预筛:哪些句子送裁判模型、按什么优先级,哪些直接判为连接性。
|
||||
|
||||
裁判按调用计费,一份报告里的过渡话、标题、表格里的一格、代码都不值得花钱判。预筛是确定性的
|
||||
规则(同一份文档永远得到同一个结果),而且只看文档本身:
|
||||
|
||||
- 优先级 1:写作者给这句挂了依据(行内引用或 [[see:…]])
|
||||
- 优先级 2:没挂依据,但有方向词、因果词(增长、主要来自、导致…)
|
||||
- 优先级 3:其余有数字、实体这类实质内容、又不是短句的
|
||||
- 不送:连接性的话、标题、代码、表格单元格;只有过渡词的(「原因如下:」);短句
|
||||
预算不够时按优先级截断:先判 1,再判 2,剩下的记未裁判。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from app.engine.evidence import build_catalog, compose_doc, iter_units
|
||||
from app.engine.judge import SHORT_CHARS, candidates, claim_priority
|
||||
|
||||
|
||||
def catalog():
|
||||
return build_catalog(nodes={"caliber": {
|
||||
"kind": "metric_set", "caliber": "周报口径", "caliber_version": "v2", "artifact": "f" * 64,
|
||||
"metrics": [{"id": "gmv", "name": "销售额", "value": 45678.5, "unit": "元", "decimals": 1},
|
||||
{"id": "orders", "name": "订单数", "value": 1234, "unit": "单"}]}})
|
||||
|
||||
|
||||
RAW = """## 本周概览
|
||||
|
||||
本周销售额 [[m:gmv]]。增长主要来自新客首单。[[see:m:gmv]]原因如下:
|
||||
下面看细节。订单一共是 1234 单,整体平稳。共 3 单。增长明显。
|
||||
|
||||
- 列表里订单 [[m:orders]]
|
||||
- 过渡说明
|
||||
|
||||
| 区域 | 销售额 |
|
||||
|---|---|
|
||||
| 华东 | [[m:gmv]] |
|
||||
|
||||
```sql
|
||||
SELECT 1
|
||||
```
|
||||
"""
|
||||
|
||||
|
||||
def by_text(doc):
|
||||
"""{句子原文: unit},测试按字找句子,不依赖编号。"""
|
||||
md = doc["markdown"]
|
||||
return {md[u["span"][0]:u["span"][1]].strip(): (b, u) for b, u in iter_units(doc)}
|
||||
|
||||
|
||||
def priorities(doc):
|
||||
return {text: claim_priority(u, b, doc["markdown"]) for text, (b, u) in by_text(doc).items()}
|
||||
|
||||
|
||||
def test_the_rules():
|
||||
got = priorities(compose_doc(RAW, catalog()))
|
||||
assert got["本周销售额 45,678.5元。"] == 1 # 行内引用
|
||||
assert got["增长主要来自新客首单。"] == 1 # 句末 [[see:]] 挂的依据
|
||||
assert got["列表里订单 1,234单"] == 1 # 列表项也是句子
|
||||
assert got["增长明显。"] == 2 # 没挂依据,有方向词:短也要判
|
||||
assert got["订单一共是 1234 单,整体平稳。"] == 3 # 有数字、不是短句
|
||||
assert got["原因如下:"] is None # 有「原因」但只是过渡
|
||||
assert got["共 3 单。"] is None # 短句
|
||||
assert got["下面看细节。"] is None # 连接性
|
||||
assert got["过渡说明"] is None
|
||||
assert got["本周概览"] is None # 标题
|
||||
assert got["45,678.5元"] is None # 表格单元格:一格值不是一句结论
|
||||
assert got["SELECT 1"] is None # 代码
|
||||
|
||||
|
||||
def test_it_is_deterministic():
|
||||
one, two = compose_doc(RAW, catalog()), compose_doc(RAW, catalog())
|
||||
assert priorities(one) == priorities(two)
|
||||
assert [c.unit for c in candidates(one)] == [c.unit for c in candidates(two)]
|
||||
|
||||
|
||||
def test_candidates_come_in_priority_order_then_document_order():
|
||||
doc = compose_doc(RAW, catalog())
|
||||
texts = {u["id"]: t for t, (_, u) in by_text(doc).items()}
|
||||
got = [(c.priority, texts[c.unit]) for c in candidates(doc)]
|
||||
assert got == [
|
||||
(1, "本周销售额 45,678.5元。"), (1, "增长主要来自新客首单。"), (1, "列表里订单 1,234单"),
|
||||
(2, "增长明显。"),
|
||||
(3, "订单一共是 1234 单,整体平稳。"),
|
||||
]
|
||||
|
||||
|
||||
def test_short_means_visible_characters_not_bytes():
|
||||
"""短句按看得见的字算(标点、空白不算),数字的字也算进去。"""
|
||||
assert SHORT_CHARS == 6
|
||||
doc = compose_doc("订单 1234 单。订单总数 1234 单。", catalog())
|
||||
got = priorities(doc)
|
||||
assert got["订单 1234 单。"] == 3 and got["订单总数 1234 单。"] == 3
|
||||
doc = compose_doc("共 12 单。", catalog())
|
||||
assert priorities(doc)["共 12 单。"] is None
|
||||
|
||||
|
||||
def test_an_explicit_request_overrides_the_screen_but_not_the_kind():
|
||||
"""有人点开一句、明确要求判它:预筛放掉的结论句照判(按最低优先级);标题、连接性的话不是结论,不判。"""
|
||||
doc = compose_doc(RAW, catalog())
|
||||
units = {t: u["id"] for t, (_, u) in by_text(doc).items()}
|
||||
picked = candidates(doc, units=[units["共 3 单。"], units["下面看细节。"], units["本周概览"], "u999"])
|
||||
assert [(c.unit, c.priority) for c in picked] == [(units["共 3 单。"], 3)]
|
||||
|
||||
|
||||
def test_cites_are_the_authors_own():
|
||||
"""自动链接的名字不算依据(三期 D2):只提到表名的句子不会因此排进优先级 1。"""
|
||||
ledger = [{"kind": "schema", "node_id": "fetch", "exec": 1, "artifact": "a" * 64, "source": "shop",
|
||||
"schema": None, "synced_at": None, "tables": {"orders": ["amount", "region"]}}]
|
||||
cat = build_catalog(nodes={}, ledger=ledger)
|
||||
doc = compose_doc("明细在 `orders` 表里,按区域汇总之后东区最高。", cat)
|
||||
[(text, prio)] = [(t, p) for t, p in priorities(doc).items()]
|
||||
assert prio == 2, (text, prio)
|
||||
@@ -326,3 +326,74 @@ async def test_the_report_node_and_the_exit_check_agree_when_the_status_changes_
|
||||
assert seen == [True, True], seen
|
||||
issuance = row.output["_issuance"]
|
||||
assert issuance["tier"] == "withheld" and CELLS_REASON in issuance["unresolved"][0]["message"], issuance
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 只引单元格、没有口径卡:契约不用写 metrics_from
|
||||
#
|
||||
# 问数据的典型图是 input → 取数 → 报告 → 出口,报告直接引用查过的单元格。受管级别在契约里
|
||||
# 写 cells: true 就够了(用户拍板的第 5 条),以前校验和门禁无条件要 metrics_from,
|
||||
# 这张图怎么都发不成受管,Copilot 补上契约也会被判「冒出新的问题」作废
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def cells_only(**contract):
|
||||
contract = {"report_from": "write", "strict": True, "cells": True, **contract}
|
||||
nodes = [
|
||||
node("start", "input"),
|
||||
node("fetch", "tool", tool="db_query__shop",
|
||||
args={"sql": "SELECT SUM(amount) AS gmv, COUNT(*) AS orders FROM orders"}),
|
||||
node("write", "report", instructions="写周报", numbers="strict", on_violation="fail",
|
||||
claims="require_citation"),
|
||||
node("out", "output", fields=[{"name": "周报", "value": "{{ nodes.write.text }}"}], contract=contract),
|
||||
]
|
||||
return {"nodes": nodes, "edges": [{"source": a["id"], "target": b["id"]} for a, b in zip(nodes, nodes[1:])]}
|
||||
|
||||
|
||||
def contract_codes(graph: dict, level: str = "governed") -> tuple[list[str], list[str]]:
|
||||
from app.engine.governance import lint_for_publish
|
||||
from app.engine.schema import GraphSpec, validate_graph
|
||||
|
||||
spec = GraphSpec.model_validate(graph)
|
||||
return ([i.code for i in validate_graph(spec).issues if i.level == "error"],
|
||||
[i.code for i in lint_for_publish(spec, level=level).issues if i.level == "error"])
|
||||
|
||||
|
||||
def test_a_cells_only_contract_needs_no_metrics_from():
|
||||
validate, gate = contract_codes(cells_only())
|
||||
assert validate == [], validate
|
||||
assert gate == [], gate # 受管门禁整张图一条 error 都没有
|
||||
|
||||
|
||||
def test_a_cited_contract_that_lists_required_metrics_still_needs_metrics_from():
|
||||
validate, gate = contract_codes(cells_only(required=["gmv"]))
|
||||
assert "contract.metrics_from_missing" in validate and "contract.metrics_from_missing" in gate
|
||||
validate, gate = contract_codes(cells_only(expected=["gmv"]))
|
||||
assert "contract.metrics_from_missing" in validate and "contract.metrics_from_missing" in gate
|
||||
|
||||
|
||||
def test_a_narrative_contract_still_needs_metrics_from():
|
||||
graph = cells_only(report_from=None, narrative="{{ nodes.write.text }}")
|
||||
graph["nodes"][-1]["data"]["config"]["contract"].pop("report_from")
|
||||
validate, _ = contract_codes(graph, level="published") # 受管级别不收 narrative,只看校验
|
||||
assert "contract.metrics_from_missing" in validate
|
||||
|
||||
|
||||
async def test_a_governed_formal_run_over_cells_only_issues_formally(client, monkeypatch):
|
||||
Writer(monkeypatch, text="本周销售额 [[v:Q1.r0.gmv]]。")
|
||||
row = await formal(client, cells_only(), status="governed")
|
||||
assert row.status == "succeeded", row.error
|
||||
issuance = row.output["_issuance"]
|
||||
assert issuance["mode"] == "citations" and issuance["tier"] == "formal", issuance
|
||||
assert issuance["unresolved"] == [] and issuance["gaps"] == [], issuance
|
||||
[cell] = [m for m in issuance["matched"] if m.get("cell")]
|
||||
assert cell["cell"] == "Q1.r0.gmv" and cell["token"] == "300.5"
|
||||
|
||||
|
||||
def test_a_graph_with_a_caliber_card_still_needs_required_even_without_metrics_from():
|
||||
graph = weekly(cells=True)
|
||||
contract = graph["nodes"][-1]["data"]["config"]["contract"]
|
||||
contract.pop("metrics_from"), contract.pop("required")
|
||||
validate, gate = contract_codes(graph)
|
||||
assert "contract.required_missing" in gate, gate # 不写 metrics_from 绕不过必需指标
|
||||
assert validate == [], validate
|
||||
|
||||
@@ -0,0 +1,323 @@
|
||||
"""出口契约(引用模式)按结论句裁判的判定判档(四期)。
|
||||
|
||||
- 报告节点 claims: judge 在正式运行里当场裁判,判定随文档封存。出口按文档里的判定判档:证据不支持的结论句
|
||||
按 on_unsupported 降档(degrade,默认)或不予出具(withhold);裁判没跑成、触顶了没判完的,裁判摘要里的缺口
|
||||
照抄进出具声明——不能判完整出具,也不因此不予出具
|
||||
- 判定是模型给的,出口复算不了:信按哈希取回的文档
|
||||
- judge 在挂引用这件事上和 require_citation 一样严:没挂依据的结论句照样计入缺口
|
||||
- 探索运行按需裁判:出口只标注(候选句都是「未裁判 · 按需」),不因判定改档位
|
||||
- 契约也能写 claims: judge:只能收紧(on_unsupported 取更严的);报告节点没开 judge 的,照实记缺口
|
||||
- 升级前的运行照旧:契约写了也记「这一版还不支持」
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import re
|
||||
import uuid
|
||||
|
||||
import pytest
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
from langchain_core.messages import AIMessage
|
||||
from sqlalchemy import delete, select
|
||||
|
||||
from app.db.base import SessionLocal
|
||||
from app.db.models import Run, RunEvent, Setting, Workflow, WorkflowVersion
|
||||
from app.engine import judge as judge_mod
|
||||
from app.engine import runner as runner_mod
|
||||
from app.engine.judge import JUDGE_ROLE, SPEND_KEY
|
||||
from app.engine.runner import run_manager
|
||||
from app.main import app
|
||||
from app.providers.mock_model import MockChatModel
|
||||
|
||||
JUDGE_MODEL = "claude-sonnet-5"
|
||||
METRICS = [{"id": "gmv", "name": "销售额", "unit": "元", "decimals": 1, "expression": "vars.kpi.gmv"}]
|
||||
#: 两句结论都挂了依据、一句连接性的话
|
||||
CITED = "本周销售额 [[m:gmv]]。[[see:m:gmv]]增长主要来自新客首单。[[see:m:gmv]]下面看细节。"
|
||||
#: 第二句结论没挂依据
|
||||
UNCITED = "本周销售额 [[m:gmv]]。[[see:m:gmv]]增长主要来自新客首单。下面看细节。"
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
async def engine_up():
|
||||
await run_manager.setup()
|
||||
async with SessionLocal() as session:
|
||||
await session.execute(delete(Setting).where(Setting.key.in_([SPEND_KEY, "judge", "copilot"])))
|
||||
await session.commit()
|
||||
yield
|
||||
await run_manager.shutdown()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
async def client():
|
||||
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as c:
|
||||
yield c
|
||||
|
||||
|
||||
def node(nid, ntype, **config):
|
||||
return {"id": nid, "type": ntype, "position": {"x": 0, "y": 0}, "data": {"label": nid, "config": config}}
|
||||
|
||||
|
||||
def weekly(*, report=None, contract=None) -> dict:
|
||||
nodes = [
|
||||
node("start", "input", fields=[{"name": "week", "default": "2026-W37"}]),
|
||||
node("fetch", "transform", mode="expression", expression="{'gmv': 45678.5}", assign_to="kpi"),
|
||||
node("caliber", "metrics", caliber="周报口径", caliber_version="v2", metrics=METRICS),
|
||||
node("write", "report", instructions="写周报", **{"claims": "judge", "judge": {"max_cost_usd": 0.05},
|
||||
**(report or {})}),
|
||||
node("out", "output", fields=[{"name": "周报", "value": "{{ nodes.write.text }}"}],
|
||||
contract={"report_from": "write", "metrics_from": ["caliber"], "required": ["gmv"], "strict": True,
|
||||
**(contract or {})}),
|
||||
]
|
||||
ids = [n["id"] for n in nodes]
|
||||
return {"nodes": nodes, "edges": [{"source": a, "target": b} for a, b in zip(ids, ids[1:])]}
|
||||
|
||||
|
||||
class Models:
|
||||
"""写作者回 text;裁判按句子里的字给判定(verdicts:{句中片段: 判定},没对上的是 supported)。"""
|
||||
|
||||
def __init__(self, monkeypatch, text=CITED, verdicts=None, *, fail_judge=None):
|
||||
self.judges: list[str] = []
|
||||
verdicts = verdicts or {}
|
||||
|
||||
def decide(model, messages):
|
||||
if JUDGE_ROLE in str(messages[0].content):
|
||||
request = str(messages[-1].content)
|
||||
self.judges.append(request)
|
||||
if fail_judge:
|
||||
raise RuntimeError(fail_judge)
|
||||
items = [{"unit": uid,
|
||||
"verdict": next((s for frag, s in verdicts.items() if frag in line), "supported"),
|
||||
"rationale": "摘录里有对应的指标" if "销售额" in line else "证据里没有新客维度", "used": []}
|
||||
for uid, line in re.findall(r"^\[(u\d+)\]([^\n]*)$", request, re.M)]
|
||||
return AIMessage(content=json.dumps({"items": items}, ensure_ascii=False))
|
||||
return AIMessage(content=text)
|
||||
|
||||
monkeypatch.setattr(MockChatModel, "_decide", decide)
|
||||
|
||||
async def judge_model_of(session, spec):
|
||||
return MockChatModel(model_name=JUDGE_MODEL), JUDGE_MODEL
|
||||
|
||||
monkeypatch.setattr(judge_mod, "get_chat_model", judge_model_of)
|
||||
|
||||
|
||||
async def wait(run_id: str) -> Run:
|
||||
for _ in range(400):
|
||||
await asyncio.sleep(0.05)
|
||||
async with SessionLocal() as session:
|
||||
row = await session.get(Run, run_id)
|
||||
if row.status in ("succeeded", "failed") and row.manifest_seq is not None:
|
||||
return row
|
||||
raise AssertionError(f"run {run_id} 没跑完:{row.status}")
|
||||
|
||||
|
||||
async def issued(graph, *, run_class="formal") -> dict:
|
||||
run = await run_manager.start(graph=graph, input_payload={}, run_class=run_class)
|
||||
row = await wait(run.id)
|
||||
assert row.status == "succeeded", row.error
|
||||
return row.output["_issuance"]
|
||||
|
||||
|
||||
async def governed(client, graph) -> Run:
|
||||
"""把这张图发布成受管级别,从发布版本发起正式运行。"""
|
||||
async with SessionLocal() as session:
|
||||
wf = Workflow(name=f"judge-{uuid.uuid4().hex[:6]}", graph=graph, status="governed", published_version=1)
|
||||
session.add(wf)
|
||||
await session.flush()
|
||||
session.add(WorkflowVersion(workflow_id=wf.id, version=1, graph=graph))
|
||||
await session.commit()
|
||||
wf_id = wf.id
|
||||
res = await client.post("/api/runs", json={"workflow_id": wf_id, "run_class": "formal"})
|
||||
assert res.status_code == 201, res.text
|
||||
row = await wait(res.json()["id"])
|
||||
assert row.status == "succeeded", row.error
|
||||
return row
|
||||
|
||||
|
||||
async def issuance_event(run_id: str) -> dict:
|
||||
async with SessionLocal() as session:
|
||||
[event] = (await session.execute(select(RunEvent).where(
|
||||
RunEvent.run_id == run_id, RunEvent.type == "issuance"))).scalars()
|
||||
return event.data
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 正式运行:按封存的判定判档
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_every_claim_supported_is_a_full_issuance(monkeypatch):
|
||||
Models(monkeypatch)
|
||||
issuance = await issued(weekly())
|
||||
assert issuance["tier"] == "formal" and issuance["gaps"] == [], issuance
|
||||
claims = issuance["claims"]
|
||||
assert claims["policy"] == "judge" and claims["mode"] == "inline" and claims["complete"] is True
|
||||
assert claims["counts"] == {"supported": 2, "partial": 0, "unsupported": 0, "not_a_claim": 0, "unjudged": 0}
|
||||
assert claims["on_unsupported"] == "degrade" and claims["uncited_claims"] == 0
|
||||
assert claims["unsupported"] == [] and claims["model"] == JUDGE_MODEL
|
||||
|
||||
|
||||
async def test_an_unsupported_claim_degrades_by_default(monkeypatch):
|
||||
Models(monkeypatch, verdicts={"新客": "unsupported"})
|
||||
issuance = await issued(weekly())
|
||||
assert issuance["tier"] == "degraded", "不支持按缺口降档,不是不予出具(契约是 strict 的)"
|
||||
assert issuance["gaps"] == [], "判了就不是「没核」:档位来自判定,不来自缺口"
|
||||
[flagged] = issuance["claims"]["unsupported"]
|
||||
assert flagged["text"] == "增长主要来自新客首单。" and flagged["rationale"] == "证据里没有新客维度"
|
||||
assert flagged["unit"] and len(flagged["span"]) == 2
|
||||
assert issuance["claims"]["counts"]["unsupported"] == 1
|
||||
|
||||
|
||||
async def test_withhold_withholds(monkeypatch):
|
||||
Models(monkeypatch, verdicts={"新客": "unsupported"})
|
||||
issuance = await issued(weekly(report={"judge": {"max_cost_usd": 0.05, "on_unsupported": "withhold"}}))
|
||||
assert issuance["tier"] == "withheld" and issuance["claims"]["on_unsupported"] == "withhold"
|
||||
|
||||
|
||||
async def test_partial_support_is_listed_but_does_not_cost_a_grade(monkeypatch):
|
||||
Models(monkeypatch, verdicts={"新客": "partial"})
|
||||
issuance = await issued(weekly(report={"judge": {"max_cost_usd": 0.05, "on_unsupported": "withhold"}}))
|
||||
assert issuance["tier"] == "formal"
|
||||
[item] = issuance["claims"]["partial"]
|
||||
assert item["text"] == "增长主要来自新客首单。" and issuance["claims"]["counts"]["partial"] == 1
|
||||
|
||||
|
||||
async def test_a_failed_judge_is_a_gap_not_a_withheld_report(monkeypatch):
|
||||
Models(monkeypatch, fail_judge="网关 502")
|
||||
issuance = await issued(weekly(report={"judge": {"max_cost_usd": 0.05, "on_unsupported": "withhold"}}))
|
||||
assert issuance["tier"] == "degraded", "没判成不能盖完整出具,也不能当成「不支持」不予出具"
|
||||
[gap] = [g for g in issuance["gaps"] if "裁判" in g]
|
||||
assert gap.startswith("报告「write」") and "没裁判" in gap and "网关 502" in gap
|
||||
assert issuance["claims"]["complete"] is False and issuance["claims"]["counts"]["unjudged"] == 2
|
||||
|
||||
|
||||
async def test_hitting_a_limit_is_a_gap(monkeypatch):
|
||||
Models(monkeypatch)
|
||||
issuance = await issued(weekly(report={"judge": {"max_cost_usd": 0.05, "max_claims": 1}}))
|
||||
assert issuance["tier"] == "degraded"
|
||||
[gap] = [g for g in issuance["gaps"] if "裁判" in g]
|
||||
assert "已到上限" in gap and "1 句" in gap
|
||||
assert issuance["claims"]["limits_hit"] == ["max_claims"] and issuance["claims"]["counts"]["unjudged"] == 1
|
||||
|
||||
|
||||
async def test_an_uncited_claim_still_counts_under_judge(monkeypatch):
|
||||
"""judge 在挂引用这件事上和 require_citation 一样严:从 require_citation 换成 judge 不能让没挂依据的句子过关。"""
|
||||
Models(monkeypatch, text=UNCITED)
|
||||
issuance = await issued(weekly())
|
||||
assert issuance["tier"] == "degraded" and issuance["claims"]["uncited_claims"] == 1
|
||||
assert issuance["claims"]["uncited"][0]["text"] == "增长主要来自新客首单。"
|
||||
|
||||
|
||||
async def test_the_issuance_event_carries_the_counts(monkeypatch):
|
||||
Models(monkeypatch, verdicts={"新客": "unsupported"})
|
||||
run = await run_manager.start(graph=weekly(), input_payload={}, run_class="formal")
|
||||
row = await wait(run.id)
|
||||
event = await issuance_event(row.id)
|
||||
assert event["tier"] == "degraded"
|
||||
assert event["claims"] == {"policy": "judge", "uncited_claims": 0, "on_unsupported": "degrade",
|
||||
"mode": "inline", "counts": {"supported": 1, "partial": 0, "unsupported": 1,
|
||||
"not_a_claim": 0, "unjudged": 0}}
|
||||
|
||||
|
||||
async def test_a_governed_formal_run_grades_by_the_verdicts(client, monkeypatch):
|
||||
Models(monkeypatch, verdicts={"新客": "unsupported"})
|
||||
graph = weekly(report={"numbers": "strict", "on_violation": "fail",
|
||||
"judge": {"max_cost_usd": None, "on_unsupported": "withhold"}})
|
||||
row = await governed(client, graph)
|
||||
issuance = row.output["_issuance"]
|
||||
assert issuance["tier"] == "withheld" and issuance["claims"]["unsupported"][0]["text"] == "增长主要来自新客首单。"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 探索运行:只标注
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_explore_runs_only_annotate(monkeypatch):
|
||||
models = Models(monkeypatch, verdicts={"新客": "unsupported"})
|
||||
issuance = await issued(weekly(report={"judge": {"max_cost_usd": 0.05, "on_unsupported": "withhold"}}),
|
||||
run_class="exploratory")
|
||||
assert models.judges == [], "探索运行节点里不花这笔钱"
|
||||
assert issuance["tier"] == "formal" and issuance["gaps"] == []
|
||||
claims = issuance["claims"]
|
||||
assert claims["mode"] == "on_demand" and claims["complete"] is False
|
||||
assert claims["counts"]["unjudged"] == 2 and claims["unjudged"] == {"on_demand": 2}
|
||||
assert claims["unsupported"] == []
|
||||
|
||||
|
||||
async def test_explore_runs_still_count_uncited_claims(monkeypatch):
|
||||
Models(monkeypatch, text=UNCITED)
|
||||
issuance = await issued(weekly(), run_class="exploratory")
|
||||
assert issuance["tier"] == "degraded" and issuance["claims"]["uncited_claims"] == 1
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 契约里的 claims: judge
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_the_contract_can_tighten_on_unsupported(monkeypatch):
|
||||
Models(monkeypatch, verdicts={"新客": "unsupported"})
|
||||
issuance = await issued(weekly(contract={"claims": {"policy": "judge", "on_unsupported": "withhold"}}))
|
||||
assert issuance["tier"] == "withheld" and issuance["claims"]["on_unsupported"] == "withhold"
|
||||
assert not any("不支持" in g and "这一版" in g for g in issuance["gaps"])
|
||||
|
||||
|
||||
async def test_the_contract_cannot_loosen_on_unsupported(monkeypatch):
|
||||
Models(monkeypatch, verdicts={"新客": "unsupported"})
|
||||
issuance = await issued(weekly(report={"judge": {"max_cost_usd": 0.05, "on_unsupported": "withhold"}},
|
||||
contract={"claims": {"policy": "judge", "on_unsupported": "degrade"}}))
|
||||
assert issuance["tier"] == "withheld"
|
||||
|
||||
|
||||
async def test_a_contract_asking_for_judge_on_a_report_without_it_is_a_gap(monkeypatch):
|
||||
Models(monkeypatch)
|
||||
issuance = await issued(weekly(report={"claims": "require_citation"}, contract={"claims": "judge"}))
|
||||
assert issuance["tier"] == "degraded"
|
||||
[gap] = [g for g in issuance["gaps"] if "judge" in g]
|
||||
assert "「write」" in gap and "没有开" in gap and "这一版" not in gap
|
||||
assert issuance["claims"]["policy"] == "require_citation"
|
||||
|
||||
|
||||
def test_a_run_in_flight_across_the_upgrade_keeps_the_old_gap():
|
||||
"""升级前写的报告(产出里没有 claims)碰上写了 judge 的契约:照旧记「这一版还不支持」,不判档。"""
|
||||
from app.engine.nodes.io import _claims
|
||||
|
||||
out = {"gaps": [], "claims_policy": None, "uncited": None, "claims": None, "unsupported": None}
|
||||
_claims(out, {"claims": {"policy": "judge", "on_unsupported": "withhold"}}, {"text": "x"}, evidence_on=True,
|
||||
uncited=[{"unit": "u1", "span": [0, 4], "text": "增长很快"}])
|
||||
assert out["claims_policy"] is None and out["claims"] is None and out["unsupported"] is None
|
||||
assert len(out["gaps"]) == 1 and "这一版还不支持" in out["gaps"][0]
|
||||
|
||||
|
||||
async def test_runs_from_before_the_upgrade_ignore_judge(monkeypatch):
|
||||
async def no_snapshot(session):
|
||||
return None
|
||||
|
||||
monkeypatch.setattr(runner_mod, "_agent_limits", no_snapshot)
|
||||
models = Models(monkeypatch, text=UNCITED, verdicts={"新客": "unsupported"})
|
||||
issuance = await issued(weekly())
|
||||
assert models.judges == [] and issuance["tier"] == "formal" and "claims" not in issuance
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 答案复核(review.scan)认得裁判的几条警告
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize(("code", "shown"), [
|
||||
("judge_limit", True), ("judge_failed", True), ("report_rewrite", True), ("report_rewrite_rejected", True),
|
||||
("judge_unpriced", False), ("judge_same_model", False),
|
||||
])
|
||||
def test_review_knows_the_judge_warnings(code, shown):
|
||||
"""没判完、没跑成、改写过的是缺口(要说明);估不出金额、自己审自己只是配置上的提醒,轨迹里有就够了,
|
||||
不值得为它再叫一次复核模型。"""
|
||||
from app.engine.review import DEGRADED, scan
|
||||
|
||||
signals = scan([{"type": "log", "data": {"level": "warn", "code": code, "message": f"{code} 的原话"}}],
|
||||
{"answer": "本周销售额 45,678.5元。"})
|
||||
if shown:
|
||||
[signal] = signals
|
||||
assert signal.kind == code and signal.severity == DEGRADED and signal.detail == f"{code} 的原话"
|
||||
else:
|
||||
assert signals == []
|
||||
@@ -1,4 +1,5 @@
|
||||
"""Copilot 的节点手册要教三期的写法:表名字段名、知识库原话、结论句策略,以及受管模板的报告节点怎么配。
|
||||
"""Copilot 的节点手册要教三期、四期的写法:表名字段名、知识库原话、结论句策略(含模型裁判 judge),以及受管模板的
|
||||
报告节点怎么配。
|
||||
|
||||
手册里没写的配置,Copilot 就不会用;写错的写法,报告节点会一直被打回。所以这里不只查关键词,
|
||||
还把手册里举的标记例子交给真正的解析器,确认写法和系统认的是同一套。
|
||||
@@ -7,9 +8,9 @@ from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
from app.api.copilot import NODE_REFERENCE, GenerateIn, _user_message
|
||||
from app.api.copilot import _ASSIST_RULES, NODE_REFERENCE, GenerateIn, _user_message
|
||||
from app.engine.evidence import QUOTE_MIN, SUPPORTED_KINDS, parse_markers
|
||||
from app.engine.schema import REPORT_CLAIMS
|
||||
from app.engine.schema import CLAIMS_VALUES, JUDGE_KEYS, JUDGE_ON_UNSUPPORTED
|
||||
|
||||
|
||||
def _report_section() -> str:
|
||||
@@ -22,12 +23,22 @@ def test_the_report_entry_teaches_entities_quotes_and_claims():
|
||||
for words in (
|
||||
["[[t:", "[[c:", "反引号", "可能是编造的名字"], # 表名、字段名
|
||||
["[[q:K1|", "逐字", f"至少 {QUOTE_MIN} 个字", "[[see:K1]]"], # 知识库原话
|
||||
["claims", "require_citation", "off", "judge", "后续版本"], # 结论句策略
|
||||
["claims", "require_citation", "off", "judge"], # 结论句策略
|
||||
["judge.max_cost_usd", "null", "不限", "rewrite_once", "on_unsupported", "按需"], # 模型裁判怎么配
|
||||
["entities", "link"], # 实体层开关
|
||||
):
|
||||
assert all(w in report for w in words), [w for w in words if w not in report]
|
||||
# 结论句策略的取值和校验认的是同一份
|
||||
assert all(value in report for value in REPORT_CLAIMS)
|
||||
# 结论句策略的取值和校验认的是同一份;judge 已经能用,不能再说「后续版本」「跑不起来」
|
||||
assert all(value in report for value in CLAIMS_VALUES)
|
||||
assert all(value in report for value in JUDGE_ON_UNSUPPORTED)
|
||||
assert "后续版本" not in report and "跑不起来" not in report
|
||||
# 裁判模型建议和写作模型不同:否则等于自己审自己
|
||||
assert "不同" in report and "自己审自己" in report
|
||||
|
||||
|
||||
def test_the_judge_keys_in_the_guide_are_the_ones_the_validator_knows():
|
||||
named = set(re.findall(r"judge\.(\w+)", NODE_REFERENCE))
|
||||
assert named and named <= set(JUDGE_KEYS), named - set(JUDGE_KEYS)
|
||||
|
||||
|
||||
def test_the_markers_in_the_guide_parse_as_the_system_reads_them():
|
||||
@@ -40,15 +51,26 @@ def test_the_markers_in_the_guide_parse_as_the_system_reads_them():
|
||||
|
||||
def test_governed_templates_spell_out_the_report_settings():
|
||||
ref = NODE_REFERENCE
|
||||
governed = ref[ref.index("受管"):]
|
||||
governed = ref[ref.index("要按受管级别发布的模板"):]
|
||||
for words in (['numbers: "strict"', 'on_violation: "fail"', 'claims: "require_citation"'],
|
||||
['claims: "judge"', "judge.max_cost_usd", "null"],
|
||||
["report_from", "narrative"], ["cite_fields: true"], ["evidence_role", "source"]):
|
||||
assert all(w in governed for w in words), [w for w in words if w not in governed]
|
||||
|
||||
|
||||
def test_the_publish_assist_knows_judge_is_the_stricter_one():
|
||||
"""发布前的 Copilot 兜底只许往严里改:它要知道 judge 比 require_citation 严,不能为了过门禁把 judge 改回去。"""
|
||||
assert "judge" in _ASSIST_RULES and "max_cost_usd" in _ASSIST_RULES
|
||||
# update_node 把 judge 整个换掉:补预算只写 {max_cost_usd} 会把 withhold 连同裁判模型一起丢了
|
||||
rules = " ".join(_ASSIST_RULES.split())
|
||||
assert re.search(r"on_unsupported.{0,20}withhold.{0,20}不改回\s*degrade", rules), "withhold 不许改回 degrade"
|
||||
assert re.search(r"补预算.{0,40}整份", rules), "补预算时 judge 要整份带上原来的键"
|
||||
|
||||
|
||||
def test_the_answer_branch_mentions_names_and_quotes():
|
||||
text = _user_message(GenerateIn(instruction="上周哪张表的退款最多", intent="answer"), patch=False)
|
||||
assert "[[t:" in text and "反引号" in text and "编造" in text
|
||||
assert "[[q:K1|" in text and "逐字" in text
|
||||
# 问数是探索运行:不要替人打开结论句策略
|
||||
assert not re.search(r"claims\s*[:=]\s*['\"]?require_citation", text)
|
||||
# 问数是探索运行:不要替人打开结论句策略(结论句由读的人点开时按需请模型判断)
|
||||
assert not re.search(r"claims\s*[:=]\s*['\"]?(require_citation|judge)", text)
|
||||
assert "按需" in text
|
||||
|
||||
@@ -0,0 +1,554 @@
|
||||
"""结论句裁判的内核(engine/judge.py):证据摘录、裁判调用、结构化输出兜底、预算与每日计数。
|
||||
|
||||
数字、实体是确定性的,系统自己能核对;「增长主要来自华东」这种话说得对不对,只能交给另一个模型按
|
||||
证据判断。这是概率性的判断,所以每一步都要留余地、说实话:
|
||||
|
||||
- 一份报告合并成一次调用(超过每批的句数就分批),每句给 supported / partial / unsupported / not_a_claim
|
||||
- 模型不支持结构化输出、或者给的不是合法结构时退回 JSON 解析,再失败就记「未裁判」,不编判定
|
||||
- 裁判模型关掉 thinking,max_tokens 4096
|
||||
- 预算(每份报告的句数、金额、时长,全局每日金额)每一项都能写 null 表示不限;触顶就停、已判的保留、
|
||||
没判的记未裁判并说明是哪个上限——不报错、不让报告失败。金额超了按优先级截断:先判挂了依据的,
|
||||
再判有方向词、因果词的
|
||||
- 每日计数落库,按本地日期滚动
|
||||
- 给裁判的证据摘录里没有数据源遮罩的列
|
||||
|
||||
所有模型调用都是 mock 剧本,不碰真接口。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
|
||||
import pytest
|
||||
from langchain_core.messages import AIMessage
|
||||
from sqlalchemy import delete
|
||||
|
||||
from app.core import artifact_store
|
||||
from app.core.events import EventType
|
||||
from app.db.base import SessionLocal
|
||||
from app.db.models import Setting
|
||||
from app.engine import judge as judge_mod
|
||||
from app.engine.evidence import build_catalog, compose_doc, iter_units
|
||||
from app.engine.judge import (
|
||||
BATCH_SIZE,
|
||||
CHUNK_CHARS,
|
||||
EXCERPT_COLS,
|
||||
EXCERPT_ROWS,
|
||||
JUDGE_ROLE,
|
||||
JUDGE_RULES,
|
||||
MAX_TOKENS,
|
||||
RATIONALE_MAX,
|
||||
SPEND_KEY,
|
||||
SQL_CHARS,
|
||||
Budget,
|
||||
daily_spend,
|
||||
judge_doc,
|
||||
prepare,
|
||||
source_masks,
|
||||
)
|
||||
from app.providers.factory import ModelSpec
|
||||
from app.providers.mock_model import MockChatModel
|
||||
|
||||
#: 目录里有价格的模型:金额上限对它起作用
|
||||
PRICED = "claude-sonnet-5"
|
||||
UNLIMITED = Budget(max_claims=None, max_cost_usd=None, timeout_s=None, daily_max_usd=None)
|
||||
SPEC = ModelSpec(model=PRICED)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 夹具:一张口径卡、一次查询(带手机号列)、一次知识库检索、一个运行输入
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _store(content) -> str:
|
||||
"""按 put_json 的落盘方式写进测试用的临时工件库:取回走真实的 load,复验哈希。"""
|
||||
text = artifact_store.canonical_json(content)
|
||||
artifact = artifact_store.content_hash(text)
|
||||
path = artifact_store._path_of(artifact)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(text, encoding="utf-8")
|
||||
return artifact
|
||||
|
||||
|
||||
CARD = {"kind": "metric_set", "caliber": "周报口径", "caliber_version": "v2", "metrics": [
|
||||
{"id": "gmv", "name": "销售额", "unit": "元", "value": 45678.5, "decimals": 1, "format": "thousands",
|
||||
"status": "ok", "expression": "vars.kpi.gmv", "rendered": "45,678.5元"},
|
||||
{"id": "wow", "name": "环比增幅", "unit": "%", "value": 8.7, "decimals": 1, "format": "plain", "status": "ok",
|
||||
"expression": "round((vars.kpi.gmv - vars.kpi.gmv_prev) / vars.kpi.gmv_prev * 100, 1)", "rendered": "8.7%"},
|
||||
]}
|
||||
LONG_SQL = "SELECT region, SUM(amount) AS amount, MAX(phone) AS phone FROM orders WHERE " + " AND ".join(
|
||||
f"flag_{i} = 1" for i in range(60)) + " GROUP BY region ORDER BY amount DESC"
|
||||
ROWS = [[f"区域{i}", 1000.0 - i, f"PHONE-{i}"] for i in range(30)]
|
||||
SNAP = {"columns": ["region", "amount", "phone"], "rows": ROWS, "row_count": len(ROWS), "truncated": False,
|
||||
"elapsed_ms": 3, "sql": LONG_SQL, "source": "shop"}
|
||||
HITS = {"query": "退款", "collection": "docs", "hits": [
|
||||
{"content": "退款主要集中在物流延误的订单。" * 60, "title": "售后周报", "document": "doc-1", "chunk": 0},
|
||||
{"content": "第二段原文。", "title": "售后周报", "document": "doc-1", "chunk": 1}]}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def stored():
|
||||
card = {**CARD}
|
||||
art = _store(card)
|
||||
return {"card": art, "snap": _store(SNAP), "masked": _store({**SNAP, "mask_columns": ["phone"]}),
|
||||
"hits": _store(HITS)}
|
||||
|
||||
|
||||
def make_catalog(stored, snap="snap"):
|
||||
ledger = [
|
||||
{"kind": "metric_set", "node_id": "caliber", "exec": 1, "artifact": stored["card"], "caliber": "周报口径",
|
||||
"version": "v2", "metrics": ["gmv", "wow"]},
|
||||
{"kind": "query", "node_id": "fetch", "exec": 1, "artifact": stored[snap], "via": "b" * 64,
|
||||
"call_id": "call_1", "tool": "db_query__shop", "source": "shop", "columns": SNAP["columns"],
|
||||
"rows": len(ROWS), "truncated": False},
|
||||
{"kind": "retrieval", "node_id": "kb", "exec": 1, "artifact": stored["hits"], "source": "docs", "rows": 2},
|
||||
]
|
||||
return build_catalog(nodes={"caliber": {**CARD, "artifact": stored["card"]}}, ledger=ledger,
|
||||
inputs={"week": "2026-W37"})
|
||||
|
||||
|
||||
RAW = ("## 本周概览\n\n"
|
||||
"本周销售额 [[m:gmv]]。[[see:m:gmv]]"
|
||||
"环比 [[m:wow]],增长主要来自华东。[[see:m:wow,Q1]]"
|
||||
"增长主要来自新客首单。"
|
||||
"订单一共是 1234 单,整体平稳。"
|
||||
"下面看细节。")
|
||||
|
||||
|
||||
def units_of(doc) -> dict[str, str]:
|
||||
"""{句子原文: unit id}"""
|
||||
md = doc["markdown"]
|
||||
return {md[u["span"][0]:u["span"][1]].strip(): u["id"] for _, u in iter_units(doc)}
|
||||
|
||||
|
||||
CITED, CITED2, CAUSAL, NUMBER = ("本周销售额 45,678.5元。", "环比 8.7%,增长主要来自华东。", "增长主要来自新客首单。",
|
||||
"订单一共是 1234 单,整体平稳。")
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
async def clean_settings():
|
||||
"""每日计数和裁判设置都落在设置表里:每个测试从干净的一天开始。"""
|
||||
async with SessionLocal() as session:
|
||||
await session.execute(delete(Setting).where(Setting.key.in_([SPEND_KEY, "judge", "copilot"])))
|
||||
await session.commit()
|
||||
yield
|
||||
|
||||
|
||||
class Judge:
|
||||
"""裁判模型的剧本。结构化调用和退回的纯文本调用分开记;verdicts 按 unit id 给判定,没写的都是 supported。"""
|
||||
|
||||
def __init__(self, monkeypatch, verdicts=None, *, model_id=PRICED, structured=None, plain=None,
|
||||
latency=0.0, fail=None):
|
||||
self.calls: list[dict] = []
|
||||
self.specs: list[ModelSpec] = []
|
||||
verdicts = verdicts or {}
|
||||
|
||||
def answer(units):
|
||||
return json.dumps({"items": [{"unit": u, "verdict": verdicts.get(u, "supported"),
|
||||
"rationale": f"{u} 的依据", "used": ["m:gmv"]} for u in units]},
|
||||
ensure_ascii=False)
|
||||
|
||||
def decide(model, messages):
|
||||
assert JUDGE_ROLE in str(messages[0].content), "这里只该有裁判调用"
|
||||
units = re.findall(r"^\[(u\d+)\]", str(messages[-1].content), re.M)
|
||||
kind = "structured" if model.response_format else "plain"
|
||||
self.calls.append({"kind": kind, "units": units, "messages": list(messages)})
|
||||
if fail:
|
||||
raise RuntimeError(fail)
|
||||
custom = structured if kind == "structured" else plain
|
||||
if custom is not None:
|
||||
return AIMessage(content=custom(units) if callable(custom) else custom)
|
||||
return AIMessage(content=answer(units))
|
||||
|
||||
monkeypatch.setattr(MockChatModel, "_decide", decide)
|
||||
|
||||
async def fake_model(session, spec):
|
||||
self.specs.append(spec)
|
||||
return MockChatModel(model_name=model_id, latency=latency), model_id
|
||||
|
||||
monkeypatch.setattr(judge_mod, "get_chat_model", fake_model)
|
||||
|
||||
@property
|
||||
def judged(self) -> list[list[str]]:
|
||||
return [c["units"] for c in self.calls if c["kind"] == "structured"]
|
||||
|
||||
|
||||
def status_of(outcome, doc, text):
|
||||
return outcome["verdicts"][units_of(doc)[text]]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 调用与输出
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_one_call_for_the_whole_report_and_the_verdict_shape(monkeypatch, stored):
|
||||
judge = Judge(monkeypatch, {"u3": "unsupported"})
|
||||
catalog = make_catalog(stored)
|
||||
doc = compose_doc(RAW, catalog)
|
||||
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
|
||||
assert len(judge.calls) == 1, "一份报告合并成一次调用"
|
||||
assert sorted(judge.judged[0]) == sorted(units_of(doc)[t] for t in (CITED, CITED2, CAUSAL, NUMBER))
|
||||
v = status_of(outcome, doc, CITED)
|
||||
assert v["status"] == "supported" and v["judge"] == PRICED and v["post_seal"] is False
|
||||
assert v["rationale"] and v["used"] == ["m:gmv"]
|
||||
assert outcome["counts"] == {"supported": 3, "partial": 0, "unsupported": 1, "not_a_claim": 0, "unjudged": 0}
|
||||
assert outcome["model"] == PRICED and outcome["calls"] == 1 and outcome["gaps"] == []
|
||||
assert units_of(doc)["下面看细节。"] not in outcome["verdicts"], "连接性的话不送裁判"
|
||||
|
||||
|
||||
async def test_the_judge_model_runs_without_thinking(monkeypatch, stored):
|
||||
judge = Judge(monkeypatch)
|
||||
catalog = make_catalog(stored)
|
||||
await judge_doc(compose_doc(RAW, catalog), catalog, spec=ModelSpec(model=PRICED, thinking="adaptive",
|
||||
max_tokens=64000),
|
||||
budget=Budget(max_claims=None, max_cost_usd=None, timeout_s=30, daily_max_usd=None))
|
||||
[spec] = judge.specs
|
||||
assert spec.thinking == "off" and spec.max_tokens == MAX_TOKENS == 4096
|
||||
assert spec.model == PRICED and spec.timeout and spec.timeout <= 30
|
||||
|
||||
|
||||
async def test_structured_output_failure_falls_back_to_json(monkeypatch, stored):
|
||||
"""结构化输出失败(模型不支持、给的不是 JSON):退回纯文本调用,从回复里把 JSON 抠出来。"""
|
||||
def fenced(units):
|
||||
body = {"items": [{"unit": u, "verdict": "partial", "rationale": "只对了一半", "used": []} for u in units]}
|
||||
return "判定如下:\n```json\n" + json.dumps(body, ensure_ascii=False) + "\n```"
|
||||
|
||||
judge = Judge(monkeypatch, structured="这不是 JSON", plain=fenced)
|
||||
catalog = make_catalog(stored)
|
||||
doc = compose_doc(RAW, catalog)
|
||||
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
|
||||
assert [c["kind"] for c in judge.calls] == ["structured", "plain"]
|
||||
assert {v["status"] for v in outcome["verdicts"].values()} == {"partial"}
|
||||
assert outcome["gaps"] == []
|
||||
|
||||
|
||||
async def test_an_array_written_as_json_text_is_still_read(monkeypatch, stored):
|
||||
"""有的模型走工具调用时把嵌套的数组写成一段 JSON 文本({"items": "[…]"}):照样认,不为它再花一次调用。"""
|
||||
def stringified(units):
|
||||
items = [{"unit": u, "verdict": "unsupported", "rationale": "对不上", "used": []} for u in units]
|
||||
return json.dumps({"items": json.dumps(items, ensure_ascii=False)}, ensure_ascii=False)
|
||||
|
||||
judge = Judge(monkeypatch, structured=stringified)
|
||||
catalog = make_catalog(stored)
|
||||
doc = compose_doc(RAW, catalog)
|
||||
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
|
||||
assert [c["kind"] for c in judge.calls] == ["structured"]
|
||||
assert {v["status"] for v in outcome["verdicts"].values()} == {"unsupported"}
|
||||
|
||||
|
||||
async def test_a_broken_provider_is_not_called_batch_after_batch(monkeypatch, stored):
|
||||
"""第一批就调不通(鉴权、网关挂了):后面几批不再一次次去撞,统一记未裁判。"""
|
||||
judge = Judge(monkeypatch, fail="401 鉴权失败")
|
||||
catalog = make_catalog(stored)
|
||||
raw = "".join(f"第 {i} 个区域的销售额增长来自新客。[[see:m:gmv]]" for i in range(BATCH_SIZE + 5))
|
||||
doc = compose_doc(raw, catalog)
|
||||
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
|
||||
assert len(judge.calls) == 2, "结构化一次、退回纯文本一次,第二批不再调用"
|
||||
assert outcome["unjudged"] == {"error": BATCH_SIZE + 5} and "401 鉴权失败" in outcome["gaps"][0]
|
||||
|
||||
|
||||
async def test_both_paths_failing_leaves_the_claims_unjudged(monkeypatch, stored):
|
||||
Judge(monkeypatch, structured="不是 JSON", plain="还是不是")
|
||||
catalog = make_catalog(stored)
|
||||
doc = compose_doc(RAW, catalog)
|
||||
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
|
||||
assert {(v["status"], v["reason"]) for v in outcome["verdicts"].values()} == {("unjudged", "format")}
|
||||
assert outcome["counts"]["unjudged"] == 4 and outcome["unjudged"] == {"format": 4}
|
||||
assert outcome["gaps"] and "4 句" in outcome["gaps"][0]
|
||||
|
||||
|
||||
async def test_bad_items_are_not_trusted(monkeypatch, stored):
|
||||
"""判定不认识、理由太长、编出目录里没有的依据、漏判的句子:各按各的处理,不编判定。"""
|
||||
catalog = make_catalog(stored)
|
||||
doc = compose_doc(RAW, catalog)
|
||||
ids = units_of(doc)
|
||||
|
||||
def messy(units):
|
||||
return json.dumps({"items": [
|
||||
{"unit": ids[CITED], "verdict": "supported", "rationale": "长" * 100, "used": ["m:gmv", "m:nope", "Q9"]},
|
||||
{"unit": ids[CITED2], "verdict": "maybe", "rationale": "不认识的判定", "used": []},
|
||||
{"unit": ids[CAUSAL], "verdict": "unsupported", "rationale": "没有新客维度", "used": []},
|
||||
{"unit": "u999", "verdict": "supported", "rationale": "不存在的句子", "used": []},
|
||||
]}, ensure_ascii=False)
|
||||
|
||||
Judge(monkeypatch, structured=messy)
|
||||
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
|
||||
v = outcome["verdicts"]
|
||||
assert v[ids[CITED]]["rationale"] == "长" * RATIONALE_MAX and v[ids[CITED]]["used"] == ["m:gmv"]
|
||||
assert (v[ids[CITED2]]["status"], v[ids[CITED2]]["reason"]) == ("unjudged", "format")
|
||||
assert v[ids[CAUSAL]]["status"] == "unsupported"
|
||||
assert (v[ids[NUMBER]]["status"], v[ids[NUMBER]]["reason"]) == ("unjudged", "missing")
|
||||
assert "u999" not in v
|
||||
|
||||
|
||||
async def test_a_failed_call_is_unjudged_not_an_error(monkeypatch, stored):
|
||||
Judge(monkeypatch, fail="网关超时")
|
||||
catalog = make_catalog(stored)
|
||||
doc = compose_doc(RAW, catalog)
|
||||
events: list = []
|
||||
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED,
|
||||
emit=lambda kind, **data: events.append((str(kind), data)))
|
||||
assert {v["reason"] for v in outcome["verdicts"].values()} == {"error"}
|
||||
assert "网关超时" in outcome["gaps"][0]
|
||||
ends = [d for k, d in events if k == EventType.LLM_END]
|
||||
assert ends and ends[-1]["purpose"] == "judge" and "网关超时" in ends[-1]["error"]
|
||||
assert any(k == EventType.LOG and d["code"] == "judge_failed" for k, d in events)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 证据摘录
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def request_text(judge) -> str:
|
||||
return "\n".join(str(m.content) for m in judge.calls[0]["messages"])
|
||||
|
||||
|
||||
async def test_excerpts_follow_the_rules(monkeypatch, stored):
|
||||
judge = Judge(monkeypatch)
|
||||
catalog = make_catalog(stored)
|
||||
raw = ("本周销售额 [[m:gmv]]。[[see:m:gmv]]"
|
||||
"第 25 行的区域排在后面。[[see:Q1.r25]]"
|
||||
"整体看华东领先。[[see:Q1]]"
|
||||
"退款集中在物流延误的订单。[[see:K1]]"
|
||||
"本周是 [[i:week]],增长来自新客。")
|
||||
doc = compose_doc(raw, catalog)
|
||||
await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
|
||||
text = request_text(judge)
|
||||
assert "数字已经由系统核对过,不要复算" in JUDGE_RULES and JUDGE_RULES in text
|
||||
# 指标:名称 = 值,加原式
|
||||
assert "销售额 = 45,678.5元" in text and "vars.kpi.gmv" in text
|
||||
# 查询:SQL 前 300 字
|
||||
assert LONG_SQL[:SQL_CHARS] in text and LONG_SQL[:SQL_CHARS + 20] not in text
|
||||
# 被引用的行最多 20 行:点名的第 25 行先给,整份引用的再从头补满
|
||||
rows = re.findall(r"^r(\d+):", text, re.M)
|
||||
assert len(rows) == EXCERPT_ROWS and "25" in rows and "0" in rows and "22" not in rows
|
||||
# 片段:最多 600 字
|
||||
first = HITS["hits"][0]["content"]
|
||||
assert first[:CHUNK_CHARS] in text and first[:CHUNK_CHARS + 1] not in text
|
||||
assert "week = 2026-W37" in text
|
||||
|
||||
|
||||
async def test_masked_columns_never_reach_the_judge(monkeypatch, stored):
|
||||
judge = Judge(monkeypatch)
|
||||
catalog = make_catalog(stored)
|
||||
doc = compose_doc("整体看华东领先。[[see:Q1]]", catalog)
|
||||
await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
|
||||
assert "PHONE-0" in request_text(judge), "对照:没设遮罩时原值在摘录里"
|
||||
|
||||
# 数据源现在设的遮罩(列名不分大小写)
|
||||
judge = Judge(monkeypatch)
|
||||
await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED, masked={"shop": ["Phone"]})
|
||||
text = request_text(judge)
|
||||
assert "PHONE-" not in text and "区域0" in text
|
||||
[header] = re.findall(r"^列:(.*)$", text, re.M)
|
||||
assert "phone" not in header.lower() and "region" in header, "表头里也不给遮罩的列"
|
||||
|
||||
# 查询当时记下的遮罩(快照里的 mask_columns)
|
||||
judge = Judge(monkeypatch)
|
||||
catalog = make_catalog(stored, snap="masked")
|
||||
doc = compose_doc("整体看华东领先。[[see:Q1]]", catalog)
|
||||
await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
|
||||
assert "PHONE-" not in request_text(judge)
|
||||
|
||||
|
||||
#: 宽表:前 13 列是填充,被引用的列排在第 14、15 列,最后一列是手机号
|
||||
WIDE_COLS = [f"c{i}" for i in range(13)] + ["refund_rate", "region", "phone"]
|
||||
WIDE = {"columns": WIDE_COLS, "row_count": 5, "truncated": False, "elapsed_ms": 3, "source": "shop",
|
||||
"sql": "SELECT * FROM wide_orders",
|
||||
"rows": [[i * 100 + j for j in range(13)] + [f"RATE-{i}", f"区域{i}", f"PHONE-{i}"] for i in range(5)]}
|
||||
|
||||
|
||||
def query_catalog(artifact: str, columns: list[str], *, source: str | None = "shop"):
|
||||
ledger = [{"kind": "query", "node_id": "fetch", "exec": 1, "artifact": artifact, "tool": "db_query__shop",
|
||||
**({"source": source} if source else {}), "columns": columns, "rows": 5, "truncated": False}]
|
||||
return build_catalog(nodes={}, ledger=ledger)
|
||||
|
||||
|
||||
def header_of(excerpt: str) -> tuple[list[str], str]:
|
||||
"""(摘录给出的列, 表头那一整行)"""
|
||||
[header] = re.findall(r"^列:(.*)$", excerpt, re.M)
|
||||
return [c for c in header.split(" | ") if not c.startswith("…")], header
|
||||
|
||||
|
||||
async def test_cited_columns_of_a_wide_table_always_reach_the_judge():
|
||||
"""宽表只给前几列时,句子引用的列排在后面也一定给:裁判看不到被引用的格子,就只能判不支持。"""
|
||||
catalog = query_catalog(_store(WIDE), WIDE_COLS)
|
||||
doc = compose_doc("[[v:Q1.r2.region]] 的退款率最高,达到 [[v:Q1.r2.refund_rate]]。"
|
||||
"手机尾号 [[v:Q1.r2.phone]] 的客户下单最多。", catalog)
|
||||
excerpt = prepare(doc, catalog, masked={"shop": ["phone"]}).excerpts["Q1"]
|
||||
shown, header = header_of(excerpt)
|
||||
assert "region" in shown and "refund_rate" in shown, header
|
||||
[row] = re.findall(r"^r2:(.*)$", excerpt, re.M)
|
||||
cells = dict(zip(shown, row.split(" | "), strict=True))
|
||||
assert cells["region"] == "区域2" and cells["refund_rate"] == "RATE-2"
|
||||
# 遮罩的列被引用了也不给
|
||||
assert "phone" not in header.lower() and "PHONE-" not in excerpt
|
||||
# 没遮罩的 15 列给了 EXCERPT_COLS 列,「另有」按实际没给出的列数计
|
||||
assert len(shown) == EXCERPT_COLS and header.endswith(f"…另有 {15 - EXCERPT_COLS} 列")
|
||||
|
||||
# 被引用的列比上限还多:一律保留
|
||||
cited = "".join(f"[[v:Q1.r1.{c}]]、" for c in WIDE_COLS[:-1])
|
||||
doc = compose_doc(f"这一行各列依次是 {cited}都比上周高。", catalog)
|
||||
shown, header = header_of(prepare(doc, catalog).excerpts["Q1"])
|
||||
assert set(shown) == set(WIDE_COLS[:-1]) and "另有 1 列" in header
|
||||
|
||||
|
||||
async def test_current_masks_apply_when_the_entry_does_not_name_its_source(stored):
|
||||
"""旧形状的台账条目没记数据源:按快照自己记的数据源找遮罩(和证据面板同一个口径),摘录里照样不给。"""
|
||||
catalog = query_catalog(stored["snap"], SNAP["columns"], source=None)
|
||||
doc = compose_doc("整体看华东领先。[[see:Q1]]", catalog)
|
||||
excerpt = prepare(doc, catalog, masked={"shop": ["phone"]}).excerpts["Q1"]
|
||||
assert "PHONE-" not in excerpt and "区域0" in excerpt and "数据源 shop" in excerpt
|
||||
|
||||
# 数据源现在设的遮罩也按快照里的数据源去查
|
||||
from app.db.models import DataSource
|
||||
|
||||
catalog = query_catalog(_store({**SNAP, "source": "judge_mask_src"}), SNAP["columns"], source=None)
|
||||
async with SessionLocal() as session:
|
||||
session.add(DataSource(name="judge_mask_src", kind="sqlite", database="unused.db",
|
||||
options={"mask_columns": "Phone"}))
|
||||
await session.commit()
|
||||
try:
|
||||
assert await source_masks(catalog) == {"judge_mask_src": ["Phone"]}
|
||||
finally:
|
||||
async with SessionLocal() as session:
|
||||
await session.execute(delete(DataSource).where(DataSource.name == "judge_mask_src"))
|
||||
await session.commit()
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 预算
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_claims_over_max_claims_are_unjudged(monkeypatch, stored):
|
||||
judge = Judge(monkeypatch)
|
||||
catalog = make_catalog(stored)
|
||||
doc = compose_doc(RAW, catalog)
|
||||
outcome = await judge_doc(doc, catalog, spec=SPEC,
|
||||
budget=Budget(max_claims=2, max_cost_usd=None, timeout_s=None, daily_max_usd=None))
|
||||
ids = units_of(doc)
|
||||
assert sorted(judge.judged[0]) == sorted([ids[CITED], ids[CITED2]]), "先判挂了依据的"
|
||||
for text in (CAUSAL, NUMBER):
|
||||
v = status_of(outcome, doc, text)
|
||||
assert (v["status"], v["reason"]) == ("unjudged", "max_claims") and "已到上限" in v["rationale"]
|
||||
assert outcome["limits_hit"] == ["max_claims"] and outcome["unjudged"] == {"max_claims": 2}
|
||||
assert outcome["gaps"] and "已到上限" in outcome["gaps"][0]
|
||||
|
||||
|
||||
async def test_cost_truncation_follows_priority(monkeypatch, stored):
|
||||
catalog = make_catalog(stored)
|
||||
doc = compose_doc(RAW, catalog)
|
||||
request = prepare(doc, catalog)
|
||||
by_text = {t: c for c in request.cands for t, u in units_of(doc).items() if u == c.unit}
|
||||
without_p3 = request.cost(PRICED, [by_text[CITED], by_text[CITED2], by_text[CAUSAL]])
|
||||
everything = request.cost(PRICED, request.cands)
|
||||
assert 0 < without_p3 < everything
|
||||
|
||||
judge = Judge(monkeypatch)
|
||||
budget = Budget(max_claims=None, max_cost_usd=(without_p3 + everything) / 2, timeout_s=None, daily_max_usd=None)
|
||||
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=budget)
|
||||
assert status_of(outcome, doc, NUMBER)["reason"] == "max_cost_usd"
|
||||
assert all(status_of(outcome, doc, t)["status"] == "supported" for t in (CITED, CITED2, CAUSAL))
|
||||
assert outcome["limits_hit"] == ["max_cost_usd"] and len(judge.judged[0]) == 3
|
||||
|
||||
cited_only = request.cost(PRICED, [by_text[CITED], by_text[CITED2]])
|
||||
judge = Judge(monkeypatch)
|
||||
budget = Budget(max_claims=None, max_cost_usd=cited_only * 1.001, timeout_s=None, daily_max_usd=None)
|
||||
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=budget)
|
||||
assert {status_of(outcome, doc, t)["reason"] for t in (CAUSAL, NUMBER)} == {"max_cost_usd"}
|
||||
assert {status_of(outcome, doc, t)["status"] for t in (CITED, CITED2)} == {"supported"}
|
||||
|
||||
|
||||
async def test_unlimited_never_truncates(monkeypatch, stored):
|
||||
"""每一项都写 null:句数再多也全判(按每批上限分批),金额、时长、每日都不拦。"""
|
||||
judge = Judge(monkeypatch)
|
||||
catalog = make_catalog(stored)
|
||||
raw = "".join(f"第 {i} 个区域的销售额增长来自新客。[[see:m:gmv]]" for i in range(BATCH_SIZE + 5))
|
||||
doc = compose_doc(raw, catalog)
|
||||
async with SessionLocal() as session: # 今天已经花掉很多:不限就是不限
|
||||
session.add(Setting(key=SPEND_KEY, value={"date": judge_mod._today(), "usd": 999.0, "calls": 9}))
|
||||
await session.commit()
|
||||
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
|
||||
assert [len(units) for units in judge.judged] == [BATCH_SIZE, 5]
|
||||
assert outcome["counts"]["supported"] == BATCH_SIZE + 5 and outcome["counts"]["unjudged"] == 0
|
||||
assert outcome["limits_hit"] == [] and outcome["gaps"] == []
|
||||
|
||||
|
||||
async def test_the_daily_cap_resets_the_next_day(monkeypatch, stored):
|
||||
catalog = make_catalog(stored)
|
||||
doc = compose_doc(RAW, catalog)
|
||||
cost = prepare(doc, catalog).cost(PRICED, prepare(doc, catalog).cands)
|
||||
budget = Budget(max_claims=None, max_cost_usd=None, timeout_s=None, daily_max_usd=cost * 1.05)
|
||||
day = {"today": "2026-09-28"}
|
||||
monkeypatch.setattr(judge_mod, "_today", lambda: day["today"])
|
||||
|
||||
judge = Judge(monkeypatch)
|
||||
first = await judge_doc(doc, catalog, spec=SPEC, budget=budget)
|
||||
assert first["counts"]["unjudged"] == 0
|
||||
spent = await daily_spend()
|
||||
assert spent["date"] == "2026-09-28" and spent["usd"] == pytest.approx(cost) and spent["calls"] == 1
|
||||
|
||||
second = await judge_doc(doc, catalog, spec=SPEC, budget=budget)
|
||||
assert {v["reason"] for v in second["verdicts"].values()} == {"daily_max_usd"}
|
||||
assert second["limits_hit"] == ["daily_max_usd"] and len(judge.calls) == 1, "触顶就不再调用"
|
||||
|
||||
day["today"] = "2026-09-29"
|
||||
third = await judge_doc(doc, catalog, spec=SPEC, budget=budget)
|
||||
assert third["counts"]["unjudged"] == 0 and len(judge.calls) == 2
|
||||
assert (await daily_spend())["date"] == "2026-09-29"
|
||||
|
||||
|
||||
async def test_the_time_limit_stops_a_slow_judge(monkeypatch, stored):
|
||||
Judge(monkeypatch, latency=2.0)
|
||||
catalog = make_catalog(stored)
|
||||
doc = compose_doc(RAW, catalog)
|
||||
outcome = await judge_doc(doc, catalog, spec=SPEC,
|
||||
budget=Budget(max_claims=None, max_cost_usd=None, timeout_s=0.2, daily_max_usd=None))
|
||||
assert {v["reason"] for v in outcome["verdicts"].values()} == {"timeout_s"}
|
||||
assert outcome["limits_hit"] == ["timeout_s"] and outcome["gaps"]
|
||||
|
||||
|
||||
async def test_a_model_without_a_price_ignores_money_limits(monkeypatch, stored):
|
||||
"""目录里没有价格的模型估不出金额:金额上限对它不起作用,并照实说「按令牌估不出金额」。"""
|
||||
judge = Judge(monkeypatch, model_id="house-judge-model")
|
||||
catalog = make_catalog(stored)
|
||||
doc = compose_doc(RAW, catalog)
|
||||
events: list = []
|
||||
outcome = await judge_doc(doc, catalog, spec=ModelSpec(model="house-judge-model"),
|
||||
budget=Budget(max_claims=None, max_cost_usd=1e-9, timeout_s=None, daily_max_usd=1e-9),
|
||||
emit=lambda kind, **data: events.append((str(kind), data)))
|
||||
assert len(judge.calls) == 1 and outcome["counts"]["unjudged"] == 0
|
||||
assert outcome["priced"] is False and any("按令牌估不出金额" in n for n in outcome["notes"])
|
||||
[log] = [d for k, d in events if k == EventType.LOG and d.get("code") == "judge_unpriced"]
|
||||
assert "按令牌估不出金额" in log["message"]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 复用与按需
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_known_sentences_are_not_judged_again(monkeypatch, stored):
|
||||
judge = Judge(monkeypatch, {"u2": "unsupported"})
|
||||
catalog = make_catalog(stored)
|
||||
doc = compose_doc(RAW, catalog)
|
||||
first = await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED)
|
||||
known = {first["keys"][u]: v for u, v in first["verdicts"].items()}
|
||||
again = await judge_doc(compose_doc(RAW, catalog), catalog, spec=SPEC, budget=UNLIMITED, known=known)
|
||||
assert len(judge.calls) == 1 and again["reused"] == 4 and again["calls"] == 0
|
||||
assert again["verdicts"] == first["verdicts"]
|
||||
|
||||
|
||||
async def test_on_demand_judges_just_the_requested_sentence_after_the_seal(monkeypatch, stored):
|
||||
judge = Judge(monkeypatch)
|
||||
catalog = make_catalog(stored)
|
||||
doc = compose_doc(RAW + "共 3 单。", catalog)
|
||||
ids = units_of(doc)
|
||||
outcome = await judge_doc(doc, catalog, spec=SPEC, budget=UNLIMITED, post_seal=True,
|
||||
units=[ids[CAUSAL], ids["共 3 单。"], ids["本周概览"]])
|
||||
assert sorted(judge.judged[0]) == sorted([ids[CAUSAL], ids["共 3 单。"]]), "点名要判的短句照判"
|
||||
assert all(v["post_seal"] is True for v in outcome["verdicts"].values())
|
||||
assert outcome["skipped"] == {ids["本周概览"]: "not_a_claim"}
|
||||
@@ -0,0 +1,696 @@
|
||||
"""按需裁判接口 POST /api/runs/{id}/evidence/judge(四期)。
|
||||
|
||||
探索运行不在节点里花裁判的钱:点开哪句才判哪句。
|
||||
|
||||
- 判定作为批注追加在封存之后(evidence.judged 事件,post_seal: true):封存核对只覆盖 manifest_seq 之前的
|
||||
事件,所以追加之后 verify 照旧一致;封存的报告文档一个字都不改
|
||||
- 只在探索运行里可用:正式运行的裁判在节点内完成,按需裁判返回 409;还没封存的、升级前发起的、封存链断了的
|
||||
也不判
|
||||
- 每次点击的上限、全局每日上限按设置控制;触顶不是报错,照实说「已到上限」和怎么调
|
||||
- 同一次运行里同一句判过就直接给已有的判定,不再调用、不再花钱(连点两下也一样)
|
||||
- 裁判只看得到封存范围内的证据:人为插进工件表的、封存之后才追加事件引用的快照,一行都不进摘录;
|
||||
数据源遮罩的列同样不给
|
||||
- 片段接口和证据图带上判定,封存内的和封存后追加的分得开
|
||||
|
||||
所有裁判都用 mock 模型给剧本,不调真接口。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import re
|
||||
import sqlite3
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
from langchain_core.messages import AIMessage
|
||||
from sqlalchemy import delete, select, update
|
||||
|
||||
from app.core import artifact_store
|
||||
from app.core.events import EventType
|
||||
from app.data.engine import engines
|
||||
from app.db.base import SessionLocal
|
||||
from app.db.models import DataSource, Run, RunEvent, Setting
|
||||
from app.engine import judge as judge_mod
|
||||
from app.engine import runner as runner_mod
|
||||
from app.engine.evidence import iter_segments, iter_units
|
||||
from app.engine.judge import JUDGE_ROLE, SPEND_KEY, daily_spend
|
||||
from app.engine.runner import run_manager, verify_manifest
|
||||
from app.main import app
|
||||
from app.providers.mock_model import MockChatModel
|
||||
|
||||
PRICED = "claude-sonnet-5"
|
||||
METRICS = [{"id": "gmv", "name": "销售额", "unit": "元", "decimals": 1, "expression": "vars.kpi.gmv"}]
|
||||
#: 一句挂了依据的结论、一句没挂依据的归因、一句连接性的话
|
||||
TEXT = "## 本周\n\n本周销售额 [[m:gmv]]。[[see:m:gmv]]增长主要来自新客首单。下面看细节。"
|
||||
CITED, CAUSAL, LINK = "本周销售额 45,678.5元。", "增长主要来自新客首单。", "下面看细节。"
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
async def engine_up():
|
||||
await run_manager.setup()
|
||||
async with SessionLocal() as session:
|
||||
await session.execute(delete(Setting).where(Setting.key.in_([SPEND_KEY, "judge", "copilot"])))
|
||||
await session.commit()
|
||||
yield
|
||||
await run_manager.shutdown()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
async def client():
|
||||
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as c:
|
||||
yield c
|
||||
|
||||
|
||||
def node(nid, ntype, **config):
|
||||
return {"id": nid, "type": ntype, "position": {"x": 0, "y": 0}, "data": {"label": nid, "config": config}}
|
||||
|
||||
|
||||
def chain(*nodes) -> dict:
|
||||
ids = [n["id"] for n in nodes]
|
||||
return {"nodes": list(nodes), "edges": [{"source": a, "target": b} for a, b in zip(ids, ids[1:])]}
|
||||
|
||||
|
||||
def weekly(*, gate=False, **report) -> dict:
|
||||
return chain(
|
||||
node("start", "input", fields=[{"name": "week", "default": "2026-W37"}]),
|
||||
node("fetch", "transform", mode="expression", expression="{'gmv': 45678.5}", assign_to="kpi"),
|
||||
node("caliber", "metrics", caliber="周报口径", caliber_version="v2", metrics=METRICS),
|
||||
node("write", "report", instructions="写周报", **report),
|
||||
*([node("gate", "human", mode="approve", title="确认")] if gate else []),
|
||||
node("out", "output", fields=[{"name": "周报", "value": "{{ nodes.write.text }}"}]),
|
||||
)
|
||||
|
||||
|
||||
class Models:
|
||||
"""写作者回 text;裁判按句子里的字给判定(verdicts:{句中片段: 判定},没对上的是 supported)。
|
||||
judges 记下每次交给裁判的请求原文,specs 记下每次要的裁判模型。"""
|
||||
|
||||
def __init__(self, monkeypatch, text=TEXT, verdicts=None, *, model=PRICED, fail_judge=None, delay=0.0):
|
||||
self.judges: list[str] = []
|
||||
self.specs: list = []
|
||||
verdicts = verdicts or {}
|
||||
|
||||
def decide(chat, messages):
|
||||
if JUDGE_ROLE in str(messages[0].content):
|
||||
request = str(messages[-1].content)
|
||||
self.judges.append(request)
|
||||
if fail_judge:
|
||||
raise RuntimeError(fail_judge)
|
||||
items = [{"unit": uid,
|
||||
"verdict": next((s for frag, s in verdicts.items() if frag in line), "supported"),
|
||||
"rationale": f"判的是:{line.strip()[:16]}", "used": []}
|
||||
for uid, line in re.findall(r"^\[(u\d+)\]([^\n]*)$", request, re.M)]
|
||||
return AIMessage(content=json.dumps({"items": items}, ensure_ascii=False))
|
||||
return AIMessage(content=text)
|
||||
|
||||
monkeypatch.setattr(MockChatModel, "_decide", decide)
|
||||
|
||||
async def judge_model_of(session, spec):
|
||||
self.specs.append(spec)
|
||||
if delay:
|
||||
await asyncio.sleep(delay)
|
||||
return MockChatModel(model_name=spec.model or model), spec.model or model
|
||||
|
||||
monkeypatch.setattr(judge_mod, "get_chat_model", judge_model_of)
|
||||
|
||||
|
||||
async def wait(run_id: str, statuses=("succeeded", "failed")) -> Run:
|
||||
for _ in range(400):
|
||||
await asyncio.sleep(0.05)
|
||||
async with SessionLocal() as session:
|
||||
row = await session.get(Run, run_id)
|
||||
if row.status in statuses and (row.status == "interrupted" or row.manifest_seq is not None):
|
||||
return row
|
||||
raise AssertionError(f"run {run_id} 没跑完:{row.status}")
|
||||
|
||||
|
||||
async def finish(graph, *, run_class="exploratory", statuses=("succeeded", "failed")) -> Run:
|
||||
run = await run_manager.start(graph=graph, input_payload={}, run_class=run_class)
|
||||
row = await wait(run.id, statuses)
|
||||
return row
|
||||
|
||||
|
||||
def doc_of(row: Run) -> dict:
|
||||
return artifact_store.load(row.output["_evidence"]["doc_artifact"])
|
||||
|
||||
|
||||
def unit_ids(doc: dict) -> dict[str, str]:
|
||||
md = doc["markdown"]
|
||||
return {md[u["span"][0]:u["span"][1]].strip(): u["id"] for _, u in iter_units(doc)}
|
||||
|
||||
|
||||
def unit_starting(doc: dict, prefix: str) -> str:
|
||||
return next(uid for text, uid in unit_ids(doc).items() if text.startswith(prefix))
|
||||
|
||||
|
||||
def seg_in(doc: dict, uid: str) -> str:
|
||||
return next(s["id"] for _, u in iter_units(doc) if u["id"] == uid for s in u["segments"] if s["kind"] == "text")
|
||||
|
||||
|
||||
async def judged_events(run_id: str) -> list[RunEvent]:
|
||||
async with SessionLocal() as session:
|
||||
return list((await session.execute(select(RunEvent).where(
|
||||
RunEvent.run_id == run_id, RunEvent.type == EventType.EVIDENCE_JUDGED).order_by(RunEvent.seq))).scalars())
|
||||
|
||||
|
||||
async def judge(client, run_id: str, units, **extra):
|
||||
return await client.post(f"/api/runs/{run_id}/evidence/judge", json={"units": units, **extra})
|
||||
|
||||
|
||||
async def save_setting(key: str, value) -> None:
|
||||
async with SessionLocal() as session:
|
||||
row = await session.get(Setting, key)
|
||||
if row is None:
|
||||
session.add(Setting(key=key, value=value))
|
||||
else:
|
||||
row.value = value
|
||||
await session.commit()
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 判定落在封存之后,封存核对照旧一致
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_an_on_demand_verdict_lands_after_the_seal_and_the_seal_still_holds(client, monkeypatch):
|
||||
models = Models(monkeypatch, verdicts={"新客": "unsupported"})
|
||||
row = await finish(weekly(claims="judge"))
|
||||
assert row.status == "succeeded" and models.judges == [], "探索运行节点里不判"
|
||||
doc = doc_of(row)
|
||||
ids = unit_ids(doc)
|
||||
|
||||
res = await judge(client, row.id, [ids[CAUSAL]])
|
||||
assert res.status_code == 200, res.text
|
||||
body = res.json()
|
||||
verdict = body["verdicts"][ids[CAUSAL]]
|
||||
assert verdict["status"] == "unsupported" and verdict["post_seal"] is True and verdict["judge"] == PRICED
|
||||
assert verdict["rationale"] and body["judged"] == [ids[CAUSAL]] and body["reused"] == []
|
||||
assert body["report"] == {"node_id": "write", "doc_artifact": row.output["_evidence"]["doc_artifact"]}
|
||||
assert body["post_seal"] is True and body["limited"] is False and body["calls"] == 1
|
||||
assert len(models.judges) == 1 and CAUSAL in models.judges[0] and CITED not in models.judges[0]
|
||||
|
||||
[event] = await judged_events(row.id)
|
||||
assert event.seq > row.manifest_seq and event.node_id == "write" and body["event"] == {"seq": event.seq}
|
||||
data = event.data
|
||||
assert data["post_seal"] is True and data["report"] == "write"
|
||||
assert data["doc_artifact"] == body["report"]["doc_artifact"]
|
||||
assert data["verdicts"] == {ids[CAUSAL]: verdict} and data["model"] == PRICED and data["calls"] == 1
|
||||
assert set(data["keys"]) == {ids[CAUSAL]} and data["units"] == [ids[CAUSAL]]
|
||||
|
||||
check = await verify_manifest(row.id)
|
||||
assert check["ok"] is True and check["sealed_at"] == row.manifest_seq
|
||||
assert (await client.get(f"/api/runs/{row.id}/verify")).json()["ok"] is True
|
||||
async with SessionLocal() as session:
|
||||
after = await session.get(Run, row.id)
|
||||
assert after.manifest_seq == row.manifest_seq and after.last_seq == event.seq
|
||||
assert doc_of(row) == doc, "封存的文档一个字都不改"
|
||||
csv = await client.get(f"/api/runs/{row.id}/evidence/audit", params={"format": "csv"})
|
||||
assert csv.headers["x-evidence-seal"] == "ok"
|
||||
|
||||
|
||||
async def test_any_claim_in_an_explore_report_can_be_judged(client, monkeypatch):
|
||||
"""报告节点没开 claims: judge 也能按需判:结论句都是候选。"""
|
||||
models = Models(monkeypatch)
|
||||
row = await finish(weekly())
|
||||
ids = unit_ids(doc_of(row))
|
||||
body = (await judge(client, row.id, [ids[CITED], ids[CAUSAL]])).json()
|
||||
assert {u: v["status"] for u, v in body["verdicts"].items()} == {ids[CITED]: "supported", ids[CAUSAL]: "supported"}
|
||||
assert len(models.judges) == 1, "一次点击的几句合成一次调用"
|
||||
assert "【m:gmv】" in models.judges[0], "挂的依据摘给裁判"
|
||||
|
||||
|
||||
async def test_sequence_numbers_continue_after_a_restart(client, monkeypatch):
|
||||
"""进程重启后内存里的序号从 0 起:追加的事件要接在库里最后一条后面,不能和已有的撞号。"""
|
||||
from app.core.bus import bus
|
||||
|
||||
Models(monkeypatch)
|
||||
row = await finish(weekly())
|
||||
ids = unit_ids(doc_of(row))
|
||||
bus._seq.pop(row.id, None)
|
||||
await judge(client, row.id, [ids[CAUSAL]])
|
||||
[event] = await judged_events(row.id)
|
||||
async with SessionLocal() as session:
|
||||
seqs = [s for (s,) in await session.execute(select(RunEvent.seq).where(RunEvent.run_id == row.id))]
|
||||
assert event.seq == max(seqs) and seqs.count(event.seq) == 1 and event.seq > row.manifest_seq
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 同一句不重复判
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_asking_again_returns_the_verdict_without_another_call(client, monkeypatch):
|
||||
models = Models(monkeypatch, verdicts={"新客": "partial"})
|
||||
row = await finish(weekly(claims="judge"))
|
||||
ids = unit_ids(doc_of(row))
|
||||
first = (await judge(client, row.id, [ids[CAUSAL]])).json()
|
||||
spent = (await daily_spend())["calls"]
|
||||
again = (await judge(client, row.id, [ids[CAUSAL]])).json()
|
||||
assert len(models.judges) == 1 and (await daily_spend())["calls"] == spent, "不再调用、不再计费"
|
||||
assert again["verdicts"] == first["verdicts"] and again["reused"] == [ids[CAUSAL]] and again["judged"] == []
|
||||
assert again["calls"] == 0 and again["event"] is None and again["message"] is None
|
||||
assert len(await judged_events(row.id)) == 1
|
||||
|
||||
# 判过的和没判过的一起点:只判没判过的
|
||||
both = (await judge(client, row.id, [ids[CAUSAL], ids[CITED]])).json()
|
||||
assert both["reused"] == [ids[CAUSAL]] and both["judged"] == [ids[CITED]]
|
||||
assert len(models.judges) == 2 and CAUSAL not in models.judges[1]
|
||||
|
||||
|
||||
async def test_a_double_click_is_judged_once(client, monkeypatch):
|
||||
models = Models(monkeypatch, delay=0.05)
|
||||
row = await finish(weekly(claims="judge"))
|
||||
uid = unit_ids(doc_of(row))[CAUSAL]
|
||||
one, two = await asyncio.gather(judge(client, row.id, [uid]), judge(client, row.id, [uid]))
|
||||
assert one.status_code == two.status_code == 200
|
||||
assert len(models.judges) == 1 and len(await judged_events(row.id)) == 1
|
||||
assert sorted([one.json()["reused"], two.json()["reused"]]) == [[], [uid]]
|
||||
|
||||
|
||||
async def test_a_failed_attempt_can_be_retried(client, monkeypatch):
|
||||
Models(monkeypatch, fail_judge="网关 502")
|
||||
row = await finish(weekly(claims="judge"))
|
||||
uid = unit_ids(doc_of(row))[CAUSAL]
|
||||
failed = (await judge(client, row.id, [uid])).json()
|
||||
assert failed["verdicts"][uid]["status"] == "unjudged" and failed["verdicts"][uid]["reason"] == "error"
|
||||
assert "网关 502" in failed["message"] and failed["limited"] is False
|
||||
models = Models(monkeypatch, verdicts={"新客": "unsupported"})
|
||||
retried = (await judge(client, row.id, [uid])).json()
|
||||
assert retried["verdicts"][uid]["status"] == "unsupported" and retried["reused"] == [] and len(models.judges) == 1
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 上限
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_hitting_the_click_limit_is_a_result_not_an_error(client, monkeypatch):
|
||||
models = Models(monkeypatch)
|
||||
await save_setting("judge", {"click_max_cost_usd": 1e-9})
|
||||
row = await finish(weekly(claims="judge"))
|
||||
uid = unit_ids(doc_of(row))[CAUSAL]
|
||||
res = await judge(client, row.id, [uid])
|
||||
assert res.status_code == 200
|
||||
body = res.json()
|
||||
assert body["limited"] is True and body["limits_hit"] == ["max_cost_usd"]
|
||||
assert body["unjudged"] == {"max_cost_usd": 1}
|
||||
assert body["message"].startswith("已到上限") and "这次点击" in body["message"]
|
||||
assert any("设置" in how and "不限" in how for how in body["adjust"])
|
||||
verdict = body["verdicts"][uid]
|
||||
assert verdict["status"] == "unjudged" and verdict["reason"] == "max_cost_usd" and verdict["post_seal"] is True
|
||||
assert models.judges == [] and body["calls"] == 0 and body["budget"]["max_cost_usd"] == 1e-9
|
||||
assert body["event"] is None and await judged_events(row.id) == [], "没问模型就没有判定可记"
|
||||
|
||||
|
||||
async def test_the_daily_limit_applies_to_clicks_too(client, monkeypatch):
|
||||
models = Models(monkeypatch)
|
||||
today = (await daily_spend())["date"]
|
||||
await save_setting(SPEND_KEY, {"date": today, "usd": 2.0, "calls": 9, "unpriced_calls": 0})
|
||||
row = await finish(weekly(claims="judge"))
|
||||
body = (await judge(client, row.id, [unit_ids(doc_of(row))[CAUSAL]])).json()
|
||||
assert body["limits_hit"] == ["daily_max_usd"] and "每日" in body["message"] and models.judges == []
|
||||
assert body["spend"]["usd"] == 2.0 and body["spend"]["daily_max_usd"] == 2.0
|
||||
|
||||
|
||||
async def test_unlimited_click_budget_judges(client, monkeypatch):
|
||||
models = Models(monkeypatch)
|
||||
await save_setting("judge", {"click_max_cost_usd": None, "daily_max_usd": None})
|
||||
row = await finish(weekly(claims="judge"))
|
||||
body = (await judge(client, row.id, [unit_ids(doc_of(row))[CAUSAL]])).json()
|
||||
assert body["limited"] is False and len(models.judges) == 1
|
||||
assert body["budget"]["max_cost_usd"] is None and body["budget"]["daily_max_usd"] is None
|
||||
assert body["spend"]["calls"] == 1 and body["spend"]["daily_max_usd"] is None
|
||||
|
||||
|
||||
async def test_the_sentence_limit_keeps_what_was_judged(client, monkeypatch):
|
||||
models = Models(monkeypatch)
|
||||
row = await finish(weekly(claims="judge", judge={"max_claims": 1}))
|
||||
ids = unit_ids(doc_of(row))
|
||||
body = (await judge(client, row.id, [ids[CITED], ids[CAUSAL]])).json()
|
||||
statuses = {u: v["status"] for u, v in body["verdicts"].items()}
|
||||
assert statuses == {ids[CITED]: "supported", ids[CAUSAL]: "unjudged"} and len(models.judges) == 1
|
||||
assert body["limits_hit"] == ["max_claims"] and body["verdicts"][ids[CAUSAL]]["reason"] == "max_claims"
|
||||
[event] = await judged_events(row.id)
|
||||
assert set(event.data["verdicts"]) == {ids[CITED], ids[CAUSAL]}, "问过模型的这一次连同没判的一起记下"
|
||||
# 没判的那句还能再点
|
||||
again = (await judge(client, row.id, [ids[CAUSAL]])).json()
|
||||
assert again["judged"] == [ids[CAUSAL]] and again["reused"] == []
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 用哪个模型
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_the_judge_model_follows_node_then_settings(client, monkeypatch):
|
||||
models = Models(monkeypatch)
|
||||
await save_setting("judge", {"model": "judge-from-settings"})
|
||||
row = await finish(weekly(claims="judge"))
|
||||
await judge(client, row.id, [unit_ids(doc_of(row))[CAUSAL]])
|
||||
[spec] = models.specs
|
||||
assert spec.model == "judge-from-settings" and spec.thinking == "off" and spec.max_tokens == 4096
|
||||
|
||||
row = await finish(weekly(claims="judge", judge={"model": "judge-from-node"}))
|
||||
body = (await judge(client, row.id, [unit_ids(doc_of(row))[CAUSAL]])).json()
|
||||
assert models.specs[-1].model == "judge-from-node" and body["model"] == "judge-from-node"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 什么时候不判
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_formal_runs_are_judged_inside_the_node_not_on_demand(client, monkeypatch):
|
||||
models = Models(monkeypatch)
|
||||
row = await finish(weekly(claims="judge", judge={"max_cost_usd": 0.05}), run_class="formal")
|
||||
assert row.status == "succeeded", row.error
|
||||
calls = len(models.judges)
|
||||
res = await judge(client, row.id, [unit_ids(doc_of(row))[CAUSAL]])
|
||||
assert res.status_code == 409 and res.json()["code"] == "evidence_judge_formal"
|
||||
assert "正式运行的裁判在节点内完成" in res.json()["detail"]
|
||||
assert len(models.judges) == calls and await judged_events(row.id) == []
|
||||
|
||||
|
||||
async def test_a_run_paused_at_an_approval_is_not_judged(client, monkeypatch):
|
||||
models = Models(monkeypatch)
|
||||
row = await finish(weekly(claims="judge", gate=True), statuses=("interrupted",))
|
||||
res = await judge(client, row.id, ["u1"])
|
||||
assert res.status_code == 409 and res.json()["code"] == "evidence_judge_unsealed"
|
||||
assert models.judges == [] and await judged_events(row.id) == []
|
||||
|
||||
|
||||
async def test_runs_from_before_the_upgrade_are_not_judged(client, monkeypatch):
|
||||
async def no_snapshot(session):
|
||||
return None
|
||||
|
||||
monkeypatch.setattr(runner_mod, "_agent_limits", no_snapshot)
|
||||
models = Models(monkeypatch)
|
||||
row = await finish(weekly())
|
||||
res = await judge(client, row.id, [unit_ids(doc_of(row))[CAUSAL]])
|
||||
assert res.status_code == 409 and res.json()["code"] == "evidence_judge_legacy"
|
||||
assert models.judges == [] and await judged_events(row.id) == []
|
||||
|
||||
|
||||
async def test_a_broken_seal_is_not_judged(client, monkeypatch):
|
||||
models = Models(monkeypatch)
|
||||
row = await finish(weekly(claims="judge"))
|
||||
async with SessionLocal() as session:
|
||||
event = (await session.execute(select(RunEvent).where(
|
||||
RunEvent.run_id == row.id, RunEvent.type == "node.finished", RunEvent.node_id == "fetch"))).scalar_one()
|
||||
event.data = {**event.data, "tampered": True}
|
||||
await session.commit()
|
||||
res = await judge(client, row.id, [unit_ids(doc_of(row))[CAUSAL]])
|
||||
assert res.status_code == 409 and res.json()["code"] == "evidence_seal_broken" and models.judges == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize("meanwhile", [
|
||||
# 裁判的这几秒里有人点了「继续运行」(失败的运行可以接着跑):运行又活了
|
||||
{"status": "running", "error": None},
|
||||
# 接着跑完、重新封存了:manifest_seq 和清单哈希都换了,这次点击依据的那份封存已经不是最新的
|
||||
{"manifest_seq": 10_000, "manifest_hash": "0" * 64},
|
||||
])
|
||||
async def test_a_run_continued_during_the_judge_call_records_no_event(client, monkeypatch, meanwhile):
|
||||
"""判定只接在点击开始时的那份封存后面。运行在裁判期间被接着跑了,再追加就落进一次活着的运行中间,
|
||||
还会被封进下一份清单:不记进运行记录,判定照样交回给点的人,并说明为什么没记。"""
|
||||
models = Models(monkeypatch, verdicts={"新客": "unsupported"})
|
||||
row = await finish(weekly(claims="judge"))
|
||||
uid = unit_ids(doc_of(row))[CAUSAL]
|
||||
ask_model = judge_mod.get_chat_model
|
||||
|
||||
async def continued_while_judging(session, spec):
|
||||
async with SessionLocal() as other:
|
||||
await other.execute(update(Run).where(Run.id == row.id).values(**meanwhile))
|
||||
await other.commit()
|
||||
return await ask_model(session, spec)
|
||||
|
||||
monkeypatch.setattr(judge_mod, "get_chat_model", continued_while_judging)
|
||||
res = await judge(client, row.id, [uid])
|
||||
assert res.status_code == 200, res.text
|
||||
body = res.json()
|
||||
assert len(models.judges) == 1 and body["calls"] == 1, "模型问过了"
|
||||
assert body["verdicts"][uid]["status"] == "unsupported" and body["judged"] == [uid], "判定照样交回"
|
||||
assert body["event"] is None and "接着跑" in body["message"] and "没有记进运行记录" in body["message"]
|
||||
assert await judged_events(row.id) == []
|
||||
async with SessionLocal() as session:
|
||||
after = await session.get(Run, row.id)
|
||||
assert after.last_seq == row.last_seq, "运行记录一条都没多"
|
||||
|
||||
|
||||
async def test_a_continue_that_wins_the_row_between_check_and_write_gets_no_event(client, monkeypatch):
|
||||
"""核对过状态、还没写入的那一下,接着跑的条件更新(runner._claim)先提交了:追加判定的条件更新落空,
|
||||
照样不记——核对和写入是同一行上的条件更新,不是先读后写。"""
|
||||
from app.core.bus import bus
|
||||
from app.core.config import settings
|
||||
|
||||
Models(monkeypatch, verdicts={"新客": "unsupported"})
|
||||
row = await finish(weekly(claims="judge"))
|
||||
uid = unit_ids(doc_of(row))[CAUSAL]
|
||||
set_seq = bus.set_seq
|
||||
|
||||
def claimed_in_between(run_id, value):
|
||||
# _append_judged 读完运行、分配序号之前正好调用它:在这里用另一条连接把运行占成 running
|
||||
if run_id == row.id:
|
||||
db = sqlite3.connect(settings.db_path, timeout=5)
|
||||
try:
|
||||
db.execute("UPDATE runs SET status = 'running' WHERE id = ?", (row.id,))
|
||||
db.commit()
|
||||
finally:
|
||||
db.close()
|
||||
set_seq(run_id, value)
|
||||
|
||||
monkeypatch.setattr(bus, "set_seq", claimed_in_between)
|
||||
body = (await judge(client, row.id, [uid])).json()
|
||||
assert body["verdicts"][uid]["status"] == "unsupported" and body["event"] is None
|
||||
assert "没有记进运行记录" in body["message"] and await judged_events(row.id) == []
|
||||
async with SessionLocal() as session:
|
||||
after = await session.get(Run, row.id)
|
||||
assert after.status == "running" and after.last_seq == row.last_seq
|
||||
|
||||
|
||||
async def test_missing_runs_and_reports(client, monkeypatch):
|
||||
Models(monkeypatch)
|
||||
res = await judge(client, "nope", ["u1"])
|
||||
assert res.status_code == 404 and res.json()["code"] == "run_not_found"
|
||||
row = await finish(chain(node("start", "input"), node("out", "output", fields=[{"name": "a", "value": "x"}])))
|
||||
res = await judge(client, row.id, ["u1"])
|
||||
assert res.status_code == 404 and res.json()["code"] == "evidence_report_not_found"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("payload", [{}, {"units": []}, {"units": "u1"}, {"units": [1]}, {"units": [""]},
|
||||
{"units": [f"u{i}" for i in range(201)]}, {"units": ["u1"], "report": 3}])
|
||||
async def test_bad_requests_are_422(client, monkeypatch, payload):
|
||||
Models(monkeypatch)
|
||||
row = await finish(weekly())
|
||||
res = await client.post(f"/api/runs/{row.id}/evidence/judge", json=payload)
|
||||
assert res.status_code == 422 and res.json()["code"] == "evidence_judge_bad_units"
|
||||
|
||||
|
||||
async def test_sentences_that_are_not_claims_are_skipped(client, monkeypatch):
|
||||
models = Models(monkeypatch)
|
||||
row = await finish(weekly(claims="judge"))
|
||||
ids = unit_ids(doc_of(row))
|
||||
body = (await judge(client, row.id, [ids["本周"], ids[LINK], "u999"])).json()
|
||||
assert body["skipped"] == {ids["本周"]: "not_a_claim", ids[LINK]: "not_a_claim", "u999": "not_found"}
|
||||
assert body["verdicts"] == {} and models.judges == [] and body["event"] is None
|
||||
assert "不是结论句" in body["message"]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 片段接口、证据图带上判定
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_the_segment_shows_the_verdict_and_whether_it_can_still_be_asked(client, monkeypatch):
|
||||
Models(monkeypatch, verdicts={"新客": "unsupported"})
|
||||
row = await finish(weekly(claims="judge"))
|
||||
doc = doc_of(row)
|
||||
ids = unit_ids(doc)
|
||||
before = (await client.get(f"/api/runs/{row.id}/evidence/segments/{seg_in(doc, ids[CAUSAL])}")).json()
|
||||
assert before["unit"]["verdict"]["status"] == "unjudged" and before["unit"]["verdict"]["reason"] == "on_demand"
|
||||
assert before["unit"]["on_demand"] == {"available": True, "reason": None, "message": None}
|
||||
|
||||
await judge(client, row.id, [ids[CAUSAL]])
|
||||
after = (await client.get(f"/api/runs/{row.id}/evidence/segments/{seg_in(doc, ids[CAUSAL])}")).json()
|
||||
assert after["unit"]["verdict"]["status"] == "unsupported" and after["unit"]["verdict"]["post_seal"] is True
|
||||
assert after["unit"]["on_demand"]["available"] is False and after["unit"]["on_demand"]["reason"] == "judged"
|
||||
assert "模型" in after["note"] and "后续版本" not in after["note"]
|
||||
other = (await client.get(f"/api/runs/{row.id}/evidence/segments/{seg_in(doc, ids[CITED])}")).json()
|
||||
assert other["unit"]["on_demand"]["available"] is True
|
||||
assert "挂了依据" in other["note"] and "模型" in other["note"] and "后续版本" not in other["note"]
|
||||
link = (await client.get(f"/api/runs/{row.id}/evidence/segments/{seg_in(doc, ids[LINK])}")).json()
|
||||
assert link["unit"]["on_demand"]["reason"] == "not_a_claim" and "verdict" not in link["unit"]
|
||||
|
||||
graph = (await client.get(f"/api/runs/{row.id}/evidence")).json()
|
||||
[report] = graph["reports"]
|
||||
assert report["claims"] == "judge" and report["judge"]["mode"] == "on_demand"
|
||||
assert report["post_seal_verdicts"] == {ids[CAUSAL]: after["unit"]["verdict"]}
|
||||
audit = (await client.get(f"/api/runs/{row.id}/evidence/audit")).json()
|
||||
assert audit["reports"][0]["post_seal_verdicts"] == report["post_seal_verdicts"]
|
||||
|
||||
|
||||
async def test_sealed_verdicts_of_a_formal_run_are_shown_as_sealed(client, monkeypatch):
|
||||
Models(monkeypatch, verdicts={"新客": "unsupported"})
|
||||
row = await finish(weekly(claims="judge", judge={"max_cost_usd": 0.05}), run_class="formal")
|
||||
doc = doc_of(row)
|
||||
body = (await client.get(f"/api/runs/{row.id}/evidence/segments/{seg_in(doc, unit_ids(doc)[CAUSAL])}")).json()
|
||||
assert body["unit"]["verdict"]["status"] == "unsupported" and body["unit"]["verdict"]["post_seal"] is False
|
||||
assert body["unit"]["on_demand"]["reason"] == "formal"
|
||||
graph = (await client.get(f"/api/runs/{row.id}/evidence")).json()
|
||||
assert graph["reports"][0]["post_seal_verdicts"] == {} and graph["reports"][0]["judge"]["mode"] == "inline"
|
||||
|
||||
|
||||
async def test_a_forged_event_cannot_overwrite_a_sealed_verdict(client, monkeypatch):
|
||||
"""封存后追加的判定只填补没判过的句子:封存内判过的,后面追加什么都盖不掉。"""
|
||||
Models(monkeypatch, verdicts={"新客": "unsupported"})
|
||||
row = await finish(weekly(claims="judge", judge={"max_cost_usd": 0.05}), run_class="formal")
|
||||
doc = doc_of(row)
|
||||
uid = unit_ids(doc)[CAUSAL]
|
||||
doc_artifact = row.output["_evidence"]["doc_artifact"]
|
||||
await run_manager.note(row.id, EventType.EVIDENCE_JUDGED, node_id="write", report="write",
|
||||
doc_artifact=doc_artifact, post_seal=True,
|
||||
verdicts={uid: {"status": "supported", "rationale": "改好了", "judge": "x", "post_seal": True}})
|
||||
body = (await client.get(f"/api/runs/{row.id}/evidence/segments/{seg_in(doc, uid)}")).json()
|
||||
assert body["unit"]["verdict"]["status"] == "unsupported"
|
||||
|
||||
|
||||
async def test_appended_verdicts_always_read_as_post_seal(client, monkeypatch):
|
||||
"""追加的判定不管载荷里怎么写,展示出来都是「封存后追加」:只有随文档封存的判定才算封存内的。"""
|
||||
Models(monkeypatch)
|
||||
row = await finish(weekly(claims="judge"))
|
||||
doc = doc_of(row)
|
||||
uid = unit_ids(doc)[CAUSAL]
|
||||
await run_manager.note(row.id, EventType.EVIDENCE_JUDGED, node_id="write", report="write",
|
||||
doc_artifact=row.output["_evidence"]["doc_artifact"],
|
||||
verdicts={uid: {"status": "supported", "rationale": "x", "judge": "x", "post_seal": False},
|
||||
"u999": {"status": "supported", "rationale": "x", "judge": "x"}})
|
||||
body = (await client.get(f"/api/runs/{row.id}/evidence/segments/{seg_in(doc, uid)}")).json()
|
||||
assert body["unit"]["verdict"]["status"] == "supported" and body["unit"]["verdict"]["post_seal"] is True
|
||||
graph = (await client.get(f"/api/runs/{row.id}/evidence")).json()
|
||||
assert set(graph["reports"][0]["post_seal_verdicts"]) == {uid}, "文档里没有的句子不认"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 裁判只看得到封存范围内的证据
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
SOURCE = "warehouse"
|
||||
TOOL = f"db_query__{SOURCE}"
|
||||
SQL = "SELECT region, SUM(amount) AS gmv, COUNT(*) AS order_cnt FROM orders GROUP BY region ORDER BY gmv DESC"
|
||||
EAST, WEST = "东区销售额 [[v:Q1.r0.gmv]]。[[see:Q1]]", "西区销售额 [[v:Q1.r1.gmv]]。[[see:Q1]]"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
async def warehouse(tmp_path):
|
||||
from app.data.introspect import introspect
|
||||
|
||||
path = tmp_path / "warehouse.db"
|
||||
db = sqlite3.connect(path)
|
||||
db.executescript("CREATE TABLE orders (id INTEGER PRIMARY KEY, region TEXT, amount REAL);")
|
||||
db.executemany("INSERT INTO orders VALUES (?, ?, ?)", [(1, "east", 100.5), (2, "west", 200.0), (3, "east", 300.0)])
|
||||
db.commit()
|
||||
db.close()
|
||||
probe = SimpleNamespace(id=f"probe-{path.name}", name=SOURCE, kind="sqlite", database=str(path), host=None,
|
||||
port=None, username=None, password=None, options={}, readonly=True, schema_cache={})
|
||||
cache = await introspect(probe)
|
||||
await engines.invalidate(probe.id)
|
||||
async with SessionLocal() as session:
|
||||
row = (await session.execute(select(DataSource).where(DataSource.name == SOURCE))).scalar_one_or_none()
|
||||
if row is None:
|
||||
row = DataSource(name=SOURCE, kind="sqlite", database=str(path), readonly=True, enabled=True)
|
||||
session.add(row)
|
||||
row.database, row.kind, row.enabled, row.readonly, row.schema_cache = str(path), "sqlite", True, True, cache
|
||||
row.options = {}
|
||||
await session.commit()
|
||||
source_id = row.id
|
||||
await engines.invalidate(source_id)
|
||||
yield source_id
|
||||
await engines.invalidate(source_id)
|
||||
|
||||
|
||||
def queried() -> dict:
|
||||
return chain(node("start", "input"), node("fetch", "tool", tool=TOOL, args={"sql": SQL}),
|
||||
node("write", "report", instructions="写周报"),
|
||||
node("out", "output", fields=[{"name": "周报", "value": "{{ nodes.write.text }}"}]))
|
||||
|
||||
|
||||
async def reseal(run_id: str) -> None:
|
||||
from app.core.artifact_store import manifest_hash
|
||||
|
||||
async with SessionLocal() as session:
|
||||
run = await session.get(Run, run_id)
|
||||
rows = [tuple(r) for r in await session.execute(
|
||||
select(RunEvent.seq, RunEvent.type, RunEvent.node_id, RunEvent.data)
|
||||
.where(RunEvent.run_id == run_id, RunEvent.seq <= run.manifest_seq).order_by(RunEvent.seq))]
|
||||
run.manifest_hash = manifest_hash(rows)
|
||||
await session.commit()
|
||||
|
||||
|
||||
async def unseal_queries(run_id: str) -> str:
|
||||
"""把查询快照从封存的事件里摘掉、按改过的事件重新封存:快照还在工件库和工件表里,封存链完好,
|
||||
只是没有哪条封存的事件引用它(等于「人为插进工件表」)。返回那件快照的 id。"""
|
||||
async with SessionLocal() as session:
|
||||
rows = list((await session.execute(select(RunEvent).where(
|
||||
RunEvent.run_id == run_id, RunEvent.type.in_(["tool.end", "node.finished"])))).scalars())
|
||||
artifact = None
|
||||
for event in rows:
|
||||
data = dict(event.data or {})
|
||||
if event.type == "tool.end":
|
||||
artifact = data.pop("query_artifact", None) or artifact
|
||||
data.pop("schema_artifact", None)
|
||||
elif data.get("evidence"):
|
||||
data["evidence"] = [e for e in data["evidence"] if e.get("kind") not in ("query", "schema")]
|
||||
event.data = data
|
||||
await session.commit()
|
||||
await reseal(run_id)
|
||||
return artifact
|
||||
|
||||
|
||||
def excerpt_lines(request: str) -> tuple[str | None, list[str]]:
|
||||
header = re.search(r"^列:(.*)$", request, re.M)
|
||||
return (header.group(1) if header else None), re.findall(r"^r\d+:.*$", request, re.M)
|
||||
|
||||
|
||||
async def test_only_sealed_snapshots_reach_the_judge(client, monkeypatch, warehouse):
|
||||
models = Models(monkeypatch, text=EAST + WEST)
|
||||
control = await finish(queried())
|
||||
assert control.status == "succeeded", control.error
|
||||
await judge(client, control.id, [unit_starting(doc_of(control), "东区")])
|
||||
assert "【Q1】查询结果" in models.judges[0] and "SQL:SELECT region" in models.judges[0]
|
||||
assert excerpt_lines(models.judges[0])[1], "封存链完好时照常摘被引用的行"
|
||||
|
||||
row = await finish(queried())
|
||||
doc = doc_of(row)
|
||||
artifact = await unseal_queries(row.id)
|
||||
assert artifact and artifact_store.load(artifact) is not None, "快照本身还在工件库里"
|
||||
assert (await verify_manifest(row.id))["ok"] is True
|
||||
res = await judge(client, row.id, [unit_starting(doc, "东区")])
|
||||
assert res.status_code == 200, res.text
|
||||
assert "【Q1】" not in models.judges[1] and "SELECT" not in models.judges[1]
|
||||
assert excerpt_lines(models.judges[1]) == (None, [])
|
||||
|
||||
# 封存之后才追加的事件引用了它,也不算
|
||||
async with SessionLocal() as session:
|
||||
run = await session.get(Run, row.id)
|
||||
seq = run.last_seq
|
||||
from app.core.bus import bus
|
||||
|
||||
bus.set_seq(row.id, seq)
|
||||
await run_manager.note(row.id, "tool.end", node_id="fetch", tool=TOOL, query_artifact=artifact)
|
||||
await judge(client, row.id, [unit_starting(doc, "西区")])
|
||||
assert "【Q1】" not in models.judges[2] and "SELECT" not in models.judges[2]
|
||||
|
||||
|
||||
async def test_masked_columns_never_reach_the_judge(client, monkeypatch, warehouse):
|
||||
models = Models(monkeypatch, text=EAST)
|
||||
row = await finish(queried())
|
||||
async with SessionLocal() as session:
|
||||
source = await session.get(DataSource, warehouse)
|
||||
source.options = {"mask_columns": ["order_cnt"]}
|
||||
await session.commit()
|
||||
await judge(client, row.id, [unit_starting(doc_of(row), "东区")])
|
||||
header, lines = excerpt_lines(models.judges[0])
|
||||
assert header is not None and "order_cnt" not in header and "gmv" in header
|
||||
assert lines and all(line.count("|") == 1 for line in lines), lines
|
||||
assert "遮罩" in models.judges[0]
|
||||
@@ -0,0 +1,344 @@
|
||||
"""四期:受管模板放开 claims: judge(结论句由另一个模型按证据逐句裁判),但要写明预算。
|
||||
|
||||
- G3:报告撰写节点的 claims 在受管级别认 require_citation 或 judge;写 judge 的,生效的 judge 配置里
|
||||
必须有 max_cost_usd 这个键——写金额,或者显式写 null 表示不限。没写就是 error(已发布级别只给警告):
|
||||
受管出具不能让每份报告的裁判花费悄悄按默认值走,也不能悄悄不设上限
|
||||
- 发布前自动修复:缺预算是 choice,候选是默认金额和「不限」,不替人选;G3 另外那几项的一键修复不再把
|
||||
judge 改回 require_citation
|
||||
- 降低要求的判断:require_citation → judge 是往严里改;judge → require_citation / off / 删掉都算放宽,
|
||||
报告节点上和出具契约里一样
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
|
||||
import pytest
|
||||
|
||||
from app.engine.autofix import NEED_CHOICE, apply_fixes, forbidden_changes, plan_fixes, public
|
||||
from app.engine.governance import lint_for_publish, publish_issues
|
||||
from app.engine.judge import JUDGE_DEFAULTS
|
||||
from app.engine.schema import GraphSpec, validate_graph
|
||||
|
||||
STRICT = {"numbers": "strict", "on_violation": "fail", "claims": "require_citation"}
|
||||
JUDGED = {**STRICT, "claims": "judge", "judge": {"max_cost_usd": 0.05}}
|
||||
BUDGET = "governed.judge_budget"
|
||||
|
||||
|
||||
def node(nid, ntype, label=None, **config):
|
||||
return {"id": nid, "type": ntype, "position": {"x": 10, "y": 20},
|
||||
"data": {"label": label or nid, "config": config}}
|
||||
|
||||
|
||||
def traceable(*, report=None, defaults=None, **contract_overrides) -> dict:
|
||||
"""input → 查库 → 口径卡 → 报告撰写 → 出具:受管要求(含 G1–G5)全部满足。"""
|
||||
contract = {"report_from": "write", "metrics_from": ["card"], "required": ["gmv"], "strict": True}
|
||||
contract.update(contract_overrides)
|
||||
nodes = [
|
||||
node("start", "input", "周期"),
|
||||
node("fetch", "tool", "取数", tool="db_query__shop",
|
||||
args={"sql": "SELECT SUM(amount) AS gmv FROM orders"}),
|
||||
node("card", "metrics", "周报口径卡", caliber="周报口径", metrics=[
|
||||
{"id": "gmv", "name": "销售额", "unit": "元", "expression": "cell(nodes.fetch, 0, 'gmv')"}]),
|
||||
node("write", "report", "报告撰写", instructions="写周报", **(JUDGED if report is None else report)),
|
||||
node("done", "output", "出具", fields=[{"name": "周报", "value": "{{ nodes.write.text }}"}],
|
||||
contract={k: v for k, v in contract.items() if v is not None}),
|
||||
]
|
||||
edges = [{"id": f"e{i}", "source": a["id"], "target": b["id"]} for i, (a, b) in enumerate(zip(nodes, nodes[1:]))]
|
||||
graph = {"nodes": nodes, "edges": edges}
|
||||
if defaults is not None:
|
||||
graph["defaults"] = defaults
|
||||
return graph
|
||||
|
||||
|
||||
def config(graph: dict, nid: str) -> dict:
|
||||
return next(n for n in graph["nodes"] if n["id"] == nid)["data"]["config"]
|
||||
|
||||
|
||||
def by_code(graph: dict, code: str, level: str = "governed") -> list:
|
||||
return [i for i in lint_for_publish(GraphSpec.model_validate(graph), level=level).issues if i.code == code]
|
||||
|
||||
|
||||
def errors(graph: dict, level: str = "governed") -> list:
|
||||
return [i for i in publish_issues(GraphSpec.model_validate(graph), level=level) if i.level == "error"]
|
||||
|
||||
|
||||
def plan(graph: dict, level: str = "governed") -> dict[str, dict]:
|
||||
spec = GraphSpec.model_validate(graph)
|
||||
return {f["id"]: f for f in plan_fixes(spec, publish_issues(spec, level=level), level=level)}
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# G3:judge 可以发布成受管,但要写预算
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("budget", [0.05, 1, None])
|
||||
def test_judge_with_a_budget_publishes_as_governed(budget):
|
||||
graph = traceable(report={**JUDGED, "judge": {"max_cost_usd": budget, "max_claims": 20}})
|
||||
assert errors(graph) == [], [i.message for i in errors(graph)]
|
||||
assert by_code(graph, "governed.report_policy") == [] and by_code(graph, BUDGET) == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize("judge", [None, {}, {"max_claims": 20, "model": "judge-model"}])
|
||||
def test_judge_without_a_budget_is_an_error_at_governed(judge):
|
||||
report = {**STRICT, "claims": "judge", **({"judge": judge} if judge is not None else {})}
|
||||
graph = traceable(report=report)
|
||||
[issue] = by_code(graph, BUDGET)
|
||||
assert issue.level == "error" and issue.node_id == "write" and issue.field == "judge.max_cost_usd"
|
||||
assert "max_cost_usd" in issue.message and "不限" in issue.message and "null" in issue.message
|
||||
assert by_code(graph, "governed.report_policy") == [], "judge 本身合规,只缺预算"
|
||||
assert [i.level for i in by_code(graph, BUDGET, "published")] == ["warning"]
|
||||
|
||||
|
||||
def test_the_budget_follows_the_same_config_rule_as_run_time():
|
||||
"""和运行时同一个取值规则:节点没写 judge 取图级 defaults;节点写了就整个用节点的(不和 defaults 拼)。"""
|
||||
graph = traceable(report={**STRICT, "claims": "judge"}, defaults={"judge": {"max_cost_usd": 0.02}})
|
||||
assert by_code(graph, BUDGET) == []
|
||||
graph = traceable(report={**STRICT, "claims": "judge", "judge": {"max_claims": 5}},
|
||||
defaults={"judge": {"max_cost_usd": 0.02}})
|
||||
assert [i.field for i in by_code(graph, BUDGET)] == ["judge.max_cost_usd"]
|
||||
# claims 本身也可以来自图级 defaults
|
||||
graph = traceable(report={"numbers": "strict", "on_violation": "fail"},
|
||||
defaults={"claims": "judge", "judge": {"max_cost_usd": None}})
|
||||
assert by_code(graph, BUDGET) == [] and by_code(graph, "governed.report_policy") == []
|
||||
|
||||
|
||||
def test_a_malformed_budget_is_left_to_validate():
|
||||
graph = traceable(report={**STRICT, "claims": "judge", "judge": {"max_cost_usd": "cheap"}})
|
||||
assert by_code(graph, BUDGET) == [], "写错的值由 validate 的 report.judge_invalid 报,同一处不说两遍"
|
||||
[bad] = [i for i in validate_graph(GraphSpec.model_validate(graph)).issues if i.code == "report.judge_invalid"]
|
||||
assert bad.field == "judge.max_cost_usd" and bad.level == "error"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("claims", [None, "off"])
|
||||
def test_a_loose_claims_setting_names_both_accepted_values(claims):
|
||||
report = {"numbers": "strict", "on_violation": "fail", **({"claims": claims} if claims else {})}
|
||||
[issue] = by_code(traceable(report=report), "governed.report_policy")
|
||||
assert issue.field == "claims" and "claims: require_citation" in issue.message and "judge" in issue.message
|
||||
|
||||
|
||||
def test_judge_needs_no_other_report_policy_fix():
|
||||
graph = traceable(report={**JUDGED, "numbers": "off"})
|
||||
[issue] = by_code(graph, "governed.report_policy")
|
||||
assert issue.field == "numbers" and "claims" not in issue.message
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 发布前自动修复
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_the_report_policy_fix_leaves_judge_alone():
|
||||
graph = traceable(report={**JUDGED, "numbers": "off", "on_violation": "flag"})
|
||||
fix = plan(graph)["governed.report_policy:write"]
|
||||
assert fix["kind"] == "auto" and fix["preview"]["after"] == {"numbers": "strict", "on_violation": "fail"}
|
||||
out = apply_fixes(graph, [fix["id"]], level="governed")
|
||||
assert out["ok"] is True and config(out["graph"], "write")["claims"] == "judge"
|
||||
assert config(out["graph"], "write")["judge"] == {"max_cost_usd": 0.05}
|
||||
|
||||
|
||||
def test_a_missing_budget_is_a_choice_between_the_default_and_unlimited():
|
||||
graph = traceable(report={**STRICT, "claims": "judge", "judge": {"model": "judge-model"}})
|
||||
fid = f"{BUDGET}:write"
|
||||
fix = public(plan(graph)[fid])
|
||||
assert fix["kind"] == "choice" and fix["node_id"] == "write" and fix["field"] == "judge.max_cost_usd"
|
||||
assert [o["value"] for o in fix["options"]] == [JUDGE_DEFAULTS["report_max_cost_usd"], None]
|
||||
assert "default" not in fix, "花多少钱要人定:不替人选,也不给建议"
|
||||
unlimited = fix["options"][1]
|
||||
assert "不限" in unlimited["label"] and "不设上限" in unlimited["hint"] and "约束" in unlimited["hint"]
|
||||
assert all(not k.startswith("_") for k in fix), "内部键不交给前端"
|
||||
|
||||
[refused] = apply_fixes(graph, [fid], level="governed")["rejected"]
|
||||
assert refused["reason"] == NEED_CHOICE.format(label=fix["label"])
|
||||
|
||||
out = apply_fixes(graph, [fid], {fid: 0.05}, level="governed")
|
||||
assert out["applied"] == [fid] and out["ok"] is True
|
||||
assert config(out["graph"], "write")["judge"] == {"model": "judge-model", "max_cost_usd": 0.05}
|
||||
[change] = out["changes"]
|
||||
assert change["field"] == "judge" and change["before"] == {"model": "judge-model"}
|
||||
assert out["ops"] == [{"op": "update_node", "id": "write",
|
||||
"config": {"judge": {"model": "judge-model", "max_cost_usd": 0.05}}}]
|
||||
|
||||
out = apply_fixes(graph, [fid], {fid: None}, level="governed")
|
||||
judge = config(out["graph"], "write")["judge"]
|
||||
assert out["ok"] is True and "max_cost_usd" in judge and judge["max_cost_usd"] is None, "不限要显式写成 null"
|
||||
|
||||
|
||||
def test_the_budget_fix_keeps_what_the_graph_defaults_said():
|
||||
"""judge 写在图级 defaults 里:修复写到节点上时带上 defaults 里的整份 judge,节点写了就不再跟随 defaults,
|
||||
不能因为补一个预算把 defaults 里选的裁判模型丢了。"""
|
||||
graph = traceable(report={**STRICT, "claims": "judge"},
|
||||
defaults={"judge": {"model": "judge-model", "rewrite_once": True}})
|
||||
fid = f"{BUDGET}:write"
|
||||
out = apply_fixes(graph, [fid], {fid: None}, level="governed")
|
||||
assert out["ok"] is True
|
||||
assert config(out["graph"], "write")["judge"] == {"model": "judge-model", "rewrite_once": True,
|
||||
"max_cost_usd": None}
|
||||
assert out["graph"]["defaults"] == {"judge": {"model": "judge-model", "rewrite_once": True}}
|
||||
|
||||
|
||||
def test_a_choice_outside_the_options_is_refused():
|
||||
graph = traceable(report={**STRICT, "claims": "judge"})
|
||||
fid = f"{BUDGET}:write"
|
||||
out = apply_fixes(graph, [fid], {fid: 99}, level="governed")
|
||||
assert out["applied"] == [] and "不在候选里" in out["rejected"][0]["reason"]
|
||||
|
||||
|
||||
def test_published_level_budget_warnings_get_no_fix():
|
||||
assert f"{BUDGET}:write" not in plan(traceable(report={**STRICT, "claims": "judge"}), "published")
|
||||
|
||||
|
||||
def test_invalid_claims_fixes_no_longer_mention_a_later_version():
|
||||
graph = traceable(report={**STRICT, "claims": "sometimes"})
|
||||
governed = plan(graph)["report.claims_invalid:write"]
|
||||
published = plan(graph, "published")["report.claims_invalid:write"]
|
||||
assert "后续版本" not in governed["label"] and "default" not in published
|
||||
assert [o["value"] for o in published["options"]] == ["off", "require_citation"]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 降低要求的判断
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _changed(report_before: dict, report_after: dict | None, **graph_kw) -> list[str]:
|
||||
old = traceable(report=report_before, **graph_kw)
|
||||
new = copy.deepcopy(old)
|
||||
if report_after is None:
|
||||
config(new, "write").pop("claims")
|
||||
else:
|
||||
config(new, "write").clear()
|
||||
config(new, "write").update({"instructions": "写周报", **report_after})
|
||||
return forbidden_changes(old, new)
|
||||
|
||||
|
||||
def test_moving_up_to_judge_is_not_loosening():
|
||||
assert _changed(STRICT, JUDGED) == []
|
||||
assert _changed({**STRICT, "claims": "off"}, JUDGED) == []
|
||||
assert _changed({"numbers": "strict", "on_violation": "fail"}, JUDGED) == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize("after", ["off", "require_citation", None])
|
||||
def test_moving_down_from_judge_is_loosening(after):
|
||||
report_after = None if after is None else {**JUDGED, "claims": after}
|
||||
[reason] = _changed(JUDGED, report_after)
|
||||
assert "「报告撰写」" in reason and "claims" in reason and "judge" in reason
|
||||
|
||||
|
||||
def test_dropping_a_judge_set_in_the_graph_defaults_is_loosening():
|
||||
[reason] = _changed({"numbers": "strict", "on_violation": "fail"}, {**STRICT, "claims": "off"},
|
||||
defaults={"claims": "judge", "judge": {"max_cost_usd": 0.05}})
|
||||
assert "claims" in reason and "judge" in reason
|
||||
|
||||
|
||||
def test_changing_the_budget_is_not_loosening():
|
||||
assert _changed(JUDGED, {**JUDGED, "judge": {"max_cost_usd": None}}) == []
|
||||
|
||||
|
||||
# claims 一直是 judge 时,judge.on_unsupported 从 withhold 往回改也是放宽:不支持的结论句从「不予出具」变成
|
||||
# 「降档出具」。Copilot 改图的 update_node 会把 judge 整个盖掉,补预算时写 judge={max_cost_usd: 0.05}
|
||||
# 就悄悄丢了 withhold——这一类要和契约里的 on_unsupported 一样拒掉
|
||||
WITHHOLD = {**JUDGED, "judge": {"max_cost_usd": 0.05, "on_unsupported": "withhold"}}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("judge, dropped", [
|
||||
({"max_cost_usd": 0.05, "on_unsupported": "degrade"}, False),
|
||||
({"max_cost_usd": 0.05}, True),
|
||||
({"max_cost_usd": 0.05, "on_unsupported": "sometimes"}, False),
|
||||
])
|
||||
def test_relaxing_on_unsupported_on_the_node_is_loosening(judge, dropped):
|
||||
[reason] = _changed(WITHHOLD, {**WITHHOLD, "judge": judge})
|
||||
assert "「报告撰写」" in reason and "judge.on_unsupported" in reason
|
||||
assert "withhold" in reason and "degrade" in reason
|
||||
assert ("去掉了写明的值" in reason) is dropped, reason
|
||||
|
||||
|
||||
def test_replacing_the_whole_judge_block_to_add_a_budget_is_loosening():
|
||||
"""补预算把 judge 整个换成 {max_cost_usd}:连同裁判模型、withhold 一起丢了。"""
|
||||
before = {**STRICT, "claims": "judge", "judge": {"model": "judge-model", "on_unsupported": "withhold"}}
|
||||
[reason] = _changed(before, {**before, "judge": {"max_cost_usd": 0.05}})
|
||||
assert "judge.on_unsupported" in reason and "withhold" in reason
|
||||
assert _changed(before, {**before, "judge": {**before["judge"], "max_cost_usd": 0.05}}) == [], \
|
||||
"带着原来的键补预算不算放宽"
|
||||
|
||||
|
||||
def test_a_withhold_from_the_graph_defaults_lost_by_a_node_judge_is_loosening():
|
||||
"""judge 按运行时的规则取:节点写了自己的 judge 就整个用节点的,图级 defaults 里的 withhold 不再生效。"""
|
||||
defaults = {"judge": {"on_unsupported": "withhold", "max_cost_usd": 0.05}}
|
||||
report = {**STRICT, "claims": "judge"}
|
||||
[reason] = _changed(report, {**report, "judge": {"max_cost_usd": 0.05}}, defaults=defaults)
|
||||
assert "judge.on_unsupported" in reason and "从 withhold(跟随全图默认)" in reason, reason
|
||||
assert "不再跟随全图默认" in reason, "说清楚为什么全图默认里的 withhold 不生效了"
|
||||
assert _changed(report, {**report, "judge": {"max_cost_usd": 0.05, "on_unsupported": "withhold"}},
|
||||
defaults=defaults) == []
|
||||
|
||||
|
||||
def test_relaxing_on_unsupported_in_the_graph_defaults_is_loosening():
|
||||
old = traceable(report={**STRICT, "claims": "judge"},
|
||||
defaults={"judge": {"max_cost_usd": 0.05, "on_unsupported": "withhold"}})
|
||||
new = copy.deepcopy(old)
|
||||
new["defaults"]["judge"]["on_unsupported"] = "degrade"
|
||||
[reason] = forbidden_changes(old, new)
|
||||
assert "「报告撰写」" in reason and "judge.on_unsupported" in reason
|
||||
|
||||
|
||||
@pytest.mark.parametrize("before, after", [
|
||||
(JUDGED, {**JUDGED, "judge": {"max_cost_usd": 0.05, "on_unsupported": "withhold"}}),
|
||||
(WITHHOLD, {**WITHHOLD, "judge": {**WITHHOLD["judge"], "max_cost_usd": None, "rewrite_once": True}}),
|
||||
(WITHHOLD, WITHHOLD),
|
||||
# 升到 judge 本身是收紧:不管带不带 withhold
|
||||
({**STRICT, "judge": {"on_unsupported": "withhold"}}, JUDGED),
|
||||
])
|
||||
def test_tightening_or_keeping_on_unsupported_is_fine(before, after):
|
||||
assert _changed(before, after) == []
|
||||
|
||||
|
||||
def test_dropping_withhold_along_with_judge_is_one_reason():
|
||||
"""从 judge 退回 require_citation:claims 那一条已经说了,不再为 on_unsupported 另记一条。"""
|
||||
[reason] = _changed(WITHHOLD, {**STRICT, "judge": {"max_cost_usd": 0.05}})
|
||||
assert "claims" in reason and "on_unsupported" not in reason
|
||||
|
||||
|
||||
@pytest.mark.parametrize("before", ["require_citation", "judge"])
|
||||
def test_moving_claims_to_an_unknown_value_is_loosening(before):
|
||||
"""require_citation、judge 改成认不出的写法:出口按认不出的走,要求没了——照样算放宽,别只落个
|
||||
「没让图变好」。原来就认不出的不比较。"""
|
||||
report = {**STRICT, "claims": before, **({"judge": {"max_cost_usd": 0.05}} if before == "judge" else {})}
|
||||
[reason] = _changed(report, {**report, "claims": "sometimes"})
|
||||
assert "「报告撰写」" in reason and before in reason and "sometimes" in reason and "不认识" in reason
|
||||
assert _changed({**STRICT, "claims": "sometimes"}, {**STRICT, "claims": "off"}) == []
|
||||
assert _changed({**STRICT, "claims": "off"}, {**STRICT, "claims": "sometimes"}) == []
|
||||
|
||||
|
||||
def _contract_changed(before, after) -> list[str]:
|
||||
old = traceable(claims=before)
|
||||
new = copy.deepcopy(old)
|
||||
contract = config(new, "done")["contract"]
|
||||
if after is None:
|
||||
contract.pop("claims", None)
|
||||
else:
|
||||
contract["claims"] = after
|
||||
return forbidden_changes(old, new)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("before, after", [
|
||||
("judge", "off"),
|
||||
("judge", None),
|
||||
("judge", "require_citation"),
|
||||
({"policy": "judge", "on_unsupported": "withhold"}, "judge"),
|
||||
({"policy": "judge", "on_unsupported": "withhold"}, {"policy": "judge", "on_unsupported": "degrade"}),
|
||||
({"policy": "require_citation", "on_uncited": "withhold"}, "judge"),
|
||||
])
|
||||
def test_loosening_a_judge_contract_is_refused(before, after):
|
||||
[reason] = _contract_changed(before, after)
|
||||
assert "「出具」" in reason and "claims" in reason
|
||||
|
||||
|
||||
@pytest.mark.parametrize("before, after", [
|
||||
("require_citation", "judge"),
|
||||
("off", {"policy": "judge", "on_unsupported": "withhold"}),
|
||||
("judge", {"policy": "judge", "on_unsupported": "withhold"}),
|
||||
({"policy": "require_citation", "on_uncited": "withhold"}, {"policy": "judge", "on_uncited": "withhold"}),
|
||||
])
|
||||
def test_tightening_a_contract_towards_judge_is_fine(before, after):
|
||||
assert _contract_changed(before, after) == []
|
||||
@@ -20,7 +20,7 @@ from app.db.base import SessionLocal
|
||||
from app.db.models import Run, Workflow, WorkflowVersion
|
||||
from app.engine.governance import lint_for_publish, publish_issues
|
||||
from app.engine.runner import run_manager
|
||||
from app.engine.schema import CLAIMS_LATER, GraphSpec, validate_graph
|
||||
from app.engine.schema import GraphSpec, validate_graph
|
||||
from app.main import app
|
||||
from app.seed import TEMPLATES
|
||||
|
||||
@@ -281,10 +281,10 @@ def test_graph_defaults_count_like_they_do_at_run_time():
|
||||
|
||||
|
||||
def test_an_unsupported_claims_value_is_left_to_validate():
|
||||
graph = traceable(report={**STRICT, "claims": "judge"})
|
||||
graph = traceable(report={**STRICT, "claims": "sometimes"})
|
||||
assert by_code(graph, "governed.report_policy") == []
|
||||
[bad] = [i for i in validate_graph(GraphSpec.model_validate(graph)).issues if i.code == "report.claims_invalid"]
|
||||
assert bad.level == "error" and CLAIMS_LATER in bad.message
|
||||
assert bad.level == "error" and "off / require_citation" in bad.message
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
@@ -442,7 +442,7 @@ async def test_an_already_governed_version_keeps_running_formally(client, engine
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# validate:报告撰写节点的 claims 只接受 off / require_citation
|
||||
# validate:报告撰写节点的 claims 只接受 off / require_citation / judge(四期放开 judge)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
@@ -458,10 +458,9 @@ def test_supported_claims_values_pass(value):
|
||||
assert claims_issues(value) == []
|
||||
|
||||
|
||||
def test_judge_is_a_later_version():
|
||||
[issue] = claims_issues("judge")
|
||||
assert issue.level == "error" and issue.node_id == "write" and issue.field == "claims"
|
||||
assert CLAIMS_LATER in issue.message and "off" in issue.message and "require_citation" in issue.message
|
||||
def test_judge_is_supported_now():
|
||||
assert claims_issues("judge") == []
|
||||
assert claims_issues("judge", on_defaults=True) == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize("value", ["strict", "Require_Citation", 1, ["off"], {"policy": "judge"}])
|
||||
@@ -471,5 +470,5 @@ def test_other_claims_values_are_errors(value):
|
||||
|
||||
|
||||
def test_claims_from_graph_defaults_are_checked_too():
|
||||
[issue] = claims_issues("judge", on_defaults=True)
|
||||
[issue] = claims_issues("sometimes", on_defaults=True)
|
||||
assert issue.node_id == "write"
|
||||
|
||||
@@ -0,0 +1,262 @@
|
||||
"""结论句裁判的设置与画布校验。
|
||||
|
||||
- 设置里单开一组 judge:证据裁判模型、每份报告的句数 / 金额 / 时长上限、探索运行每次点击的金额上限、
|
||||
全局每日金额上限。每一项都能存成 null,表示不限——存了 null 就是不限,不会被默认值悄悄盖回去
|
||||
- limits 组是环境变量定的、只读,裁判设置不放进去;每日计数存在设置表里,但不从设置接口进出
|
||||
- 裁判模型的优先级:节点上的 judge.model > 设置里的证据裁判模型 > Copilot 模型 > 默认 provider
|
||||
- 画布校验放开 claims: judge,并校验 judge 子配置的形状:上限可以写 null(不限),rewrite_once 是布尔值,
|
||||
on_unsupported 取 degrade / withhold
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
from sqlalchemy import delete
|
||||
|
||||
from app.db.base import SessionLocal
|
||||
from app.db.models import Setting
|
||||
from app.engine.judge import (
|
||||
JUDGE_DEFAULTS,
|
||||
SETTING_GROUP,
|
||||
SPEND_KEY,
|
||||
Budget,
|
||||
click_budget,
|
||||
judge_model_spec,
|
||||
judge_settings,
|
||||
report_budget,
|
||||
)
|
||||
from app.engine.schema import CLAIMS_VALUES, JUDGE_KEYS, GraphSpec, claims_problem, validate_graph
|
||||
from app.main import app
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
async def clean():
|
||||
async with SessionLocal() as session:
|
||||
await session.execute(delete(Setting).where(Setting.key.in_([SPEND_KEY, SETTING_GROUP, "copilot"])))
|
||||
await session.commit()
|
||||
yield
|
||||
|
||||
|
||||
def client() -> AsyncClient:
|
||||
return AsyncClient(transport=ASGITransport(app=app), base_url="http://test")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 设置
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_the_defaults_are_the_agreed_ones():
|
||||
assert SETTING_GROUP == "judge" and SETTING_GROUP != "limits"
|
||||
assert JUDGE_DEFAULTS == {
|
||||
"provider": None, "model": None,
|
||||
"report_max_claims": 40, "report_max_cost_usd": 0.05, "report_timeout_s": 30,
|
||||
"click_max_cost_usd": 0.01, "daily_max_usd": 2.0,
|
||||
}
|
||||
|
||||
|
||||
async def test_settings_expose_the_judge_group_with_defaults():
|
||||
async with client() as c:
|
||||
got = (await c.get("/api/settings")).json()
|
||||
assert got["judge"] == JUDGE_DEFAULTS
|
||||
assert SPEND_KEY not in got
|
||||
|
||||
|
||||
async def test_null_is_stored_as_unlimited_and_stays_unlimited():
|
||||
async with client() as c:
|
||||
r = await c.put("/api/settings", json={"values": {"judge": {
|
||||
**JUDGE_DEFAULTS, "model": "judge-model", "report_max_cost_usd": None, "daily_max_usd": None,
|
||||
"report_timeout_s": None, "report_max_claims": None, "click_max_cost_usd": None}}})
|
||||
assert r.status_code == 200, r.text
|
||||
got = (await c.get("/api/settings")).json()["judge"]
|
||||
assert got["model"] == "judge-model"
|
||||
assert all(got[k] is None for k in ("report_max_cost_usd", "daily_max_usd", "report_timeout_s",
|
||||
"report_max_claims", "click_max_cost_usd"))
|
||||
async with SessionLocal() as session:
|
||||
settings = await judge_settings(session)
|
||||
assert report_budget(settings, None) == Budget(max_claims=None, max_cost_usd=None, timeout_s=None,
|
||||
daily_max_usd=None)
|
||||
assert click_budget(settings).max_cost_usd is None
|
||||
|
||||
|
||||
async def test_a_partial_group_keeps_the_other_defaults():
|
||||
async with client() as c:
|
||||
await c.put("/api/settings", json={"values": {"judge": {"daily_max_usd": 5}}})
|
||||
got = (await c.get("/api/settings")).json()["judge"]
|
||||
assert got == {**JUDGE_DEFAULTS, "daily_max_usd": 5}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("key,value", [
|
||||
("daily_max_usd", -1), ("report_max_cost_usd", 0), ("report_max_cost_usd", "0.05"),
|
||||
("report_max_claims", 2.5), ("report_max_claims", True), ("report_timeout_s", 0),
|
||||
("click_max_cost_usd", float("nan")), ("model", 3), ("unknown_key", 1),
|
||||
])
|
||||
async def test_bad_values_are_refused_with_a_reason(key, value):
|
||||
async with client() as c:
|
||||
r = await c.put("/api/settings", json={"values": {"judge": {key: value}}}) if value == value else \
|
||||
await c.put("/api/settings", content='{"values": {"judge": {"%s": NaN}}}' % key,
|
||||
headers={"content-type": "application/json"})
|
||||
assert r.status_code == 422, r.text
|
||||
assert key in r.json()["detail"]
|
||||
assert (await c.get("/api/settings")).json()["judge"] == JUDGE_DEFAULTS, "整组都没写进去"
|
||||
|
||||
|
||||
async def test_the_daily_counter_does_not_travel_through_the_settings_api():
|
||||
async with SessionLocal() as session:
|
||||
session.add(Setting(key=SPEND_KEY, value={"date": "2026-09-29", "usd": 1.5, "calls": 3}))
|
||||
await session.commit()
|
||||
async with client() as c:
|
||||
assert SPEND_KEY not in (await c.get("/api/settings")).json()
|
||||
await c.put("/api/settings", json={"values": {SPEND_KEY: {"date": "2026-09-29", "usd": 0, "calls": 0}}})
|
||||
async with SessionLocal() as session:
|
||||
row = await session.get(Setting, SPEND_KEY)
|
||||
assert row.value["usd"] == 1.5, "设置页整组回写不能把今天的花费清零"
|
||||
|
||||
|
||||
async def test_todays_spend_is_readable_next_to_the_daily_cap(monkeypatch):
|
||||
from app.engine import judge as judge_mod
|
||||
|
||||
monkeypatch.setattr(judge_mod, "_today", lambda: "2026-09-29")
|
||||
async with SessionLocal() as session:
|
||||
session.add(Setting(key=SPEND_KEY, value={"date": "2026-09-28", "usd": 1.9, "calls": 7}))
|
||||
await session.commit()
|
||||
async with client() as c:
|
||||
got = (await c.get("/api/settings/judge/spend")).json()
|
||||
assert got == {"date": "2026-09-29", "usd": 0.0, "calls": 0, "unpriced_calls": 0, "daily_max_usd": 2.0}, \
|
||||
"昨天的花费不算今天的"
|
||||
async with SessionLocal() as session:
|
||||
row = await session.get(Setting, SPEND_KEY)
|
||||
row.value = {"date": "2026-09-29", "usd": 0.42, "calls": 3, "unpriced_calls": 1}
|
||||
await session.commit()
|
||||
await c.put("/api/settings", json={"values": {"judge": {"daily_max_usd": None}}})
|
||||
got = (await c.get("/api/settings/judge/spend")).json()
|
||||
assert got == {"date": "2026-09-29", "usd": 0.42, "calls": 3, "unpriced_calls": 1, "daily_max_usd": None}
|
||||
|
||||
|
||||
async def test_bad_rows_already_in_the_database_fall_back_to_defaults():
|
||||
"""库里存坏了的值(旧版本、手改)读出来按默认,不让裁判拿着一个负数上限跑。"""
|
||||
async with SessionLocal() as session:
|
||||
session.add(Setting(key=SETTING_GROUP, value={"daily_max_usd": -3, "report_max_claims": "many",
|
||||
"report_timeout_s": None}))
|
||||
await session.commit()
|
||||
settings = await judge_settings(session)
|
||||
assert settings["daily_max_usd"] == JUDGE_DEFAULTS["daily_max_usd"]
|
||||
assert settings["report_max_claims"] == JUDGE_DEFAULTS["report_max_claims"]
|
||||
assert settings["report_timeout_s"] is None
|
||||
|
||||
|
||||
def test_the_node_overrides_key_by_key():
|
||||
settings = {**JUDGE_DEFAULTS, "report_max_claims": 10}
|
||||
assert report_budget(settings, None) == Budget(max_claims=10, max_cost_usd=0.05, timeout_s=30, daily_max_usd=2.0)
|
||||
assert report_budget(settings, {"max_cost_usd": None, "timeout_s": 5}) == \
|
||||
Budget(max_claims=10, max_cost_usd=None, timeout_s=5, daily_max_usd=2.0)
|
||||
# 探索运行每次点击:金额按点击的上限,句数、时长按每份报告的
|
||||
assert click_budget(settings) == Budget(max_claims=10, max_cost_usd=0.01, timeout_s=30, daily_max_usd=2.0)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 裁判模型的优先级
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_the_judge_model_priority():
|
||||
async with SessionLocal() as session:
|
||||
spec = await judge_model_spec(session, None)
|
||||
assert (spec.provider, spec.model) == (None, None), "什么都没配:默认 provider"
|
||||
assert spec.thinking == "off" and spec.max_tokens == 4096
|
||||
|
||||
session.add(Setting(key="copilot", value={"provider": "", "model": "copilot-model"}))
|
||||
await session.commit()
|
||||
assert (await judge_model_spec(session, None)).model == "copilot-model"
|
||||
|
||||
session.add(Setting(key=SETTING_GROUP, value={"provider": "house", "model": "judge-model"}))
|
||||
await session.commit()
|
||||
spec = await judge_model_spec(session, None)
|
||||
assert (spec.provider, spec.model) == ("house", "judge-model")
|
||||
|
||||
spec = await judge_model_spec(session, {"model": "node-model"})
|
||||
assert (spec.provider, spec.model) == (None, "node-model"), "节点上的写法整组生效,不和设置里的拼"
|
||||
spec = await judge_model_spec(session, {"model": None, "provider": None, "max_claims": 3})
|
||||
assert spec.model == "judge-model"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 画布校验
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def node(nid, ntype, **config):
|
||||
return {"id": nid, "type": ntype, "position": {"x": 0, "y": 0}, "data": {"label": nid, "config": config}}
|
||||
|
||||
|
||||
def graph(report: dict, defaults: dict | None = None) -> dict:
|
||||
nodes = [node("start", "input"),
|
||||
node("caliber", "metrics", caliber="周报口径", metrics=[{"id": "gmv", "expression": "1"}]),
|
||||
node("write", "report", instructions="写周报", **report),
|
||||
node("out", "output", fields=[{"name": "周报", "value": "{{ nodes.write.text }}"}])]
|
||||
ids = [n["id"] for n in nodes]
|
||||
g = {"nodes": nodes, "edges": [{"source": a, "target": b} for a, b in zip(ids, ids[1:])]}
|
||||
if defaults:
|
||||
g["defaults"] = defaults
|
||||
return g
|
||||
|
||||
|
||||
def issues(g, code=None):
|
||||
return [i for i in validate_graph(GraphSpec.model_validate(g)).issues
|
||||
if i.code in ((code,) if code else ("report.claims_invalid", "report.judge_invalid"))]
|
||||
|
||||
|
||||
def test_judge_is_a_supported_policy_now():
|
||||
assert CLAIMS_VALUES == ("off", "require_citation", "judge")
|
||||
assert claims_problem("judge") is None
|
||||
assert issues(graph({"claims": "judge"})) == []
|
||||
assert issues(graph({}, defaults={"claims": "judge"})) == []
|
||||
|
||||
|
||||
def test_every_limit_can_be_unlimited():
|
||||
judge = {"provider": None, "model": None, "max_claims": None, "max_cost_usd": None, "timeout_s": None,
|
||||
"rewrite_once": True, "on_unsupported": "withhold"}
|
||||
assert set(judge) == set(JUDGE_KEYS)
|
||||
assert issues(graph({"claims": "judge", "judge": judge})) == []
|
||||
assert issues(graph({"claims": "judge", "judge": {"max_claims": 40, "max_cost_usd": 0.05, "timeout_s": 30,
|
||||
"model": "judge-model", "on_unsupported": "degrade"}})) == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize("judge,field,words", [
|
||||
("fast", "judge", "对象"),
|
||||
({"max_cost": 1}, "judge.max_cost", "max_cost_usd"),
|
||||
({"max_claims": 0}, "judge.max_claims", "正整数"),
|
||||
({"max_claims": 2.5}, "judge.max_claims", "正整数"),
|
||||
({"max_claims": True}, "judge.max_claims", "正整数"),
|
||||
({"max_cost_usd": -1}, "judge.max_cost_usd", "大于 0"),
|
||||
({"max_cost_usd": "0.05"}, "judge.max_cost_usd", "大于 0"),
|
||||
({"timeout_s": 0}, "judge.timeout_s", "大于 0"),
|
||||
({"rewrite_once": "yes"}, "judge.rewrite_once", "true 或 false"),
|
||||
({"on_unsupported": "ignore"}, "judge.on_unsupported", "degrade / withhold"),
|
||||
({"model": 3}, "judge.model", "模型 id"),
|
||||
])
|
||||
def test_a_malformed_judge_block_is_an_error(judge, field, words):
|
||||
[issue] = issues(graph({"claims": "judge", "judge": judge}), "report.judge_invalid")
|
||||
assert issue.level == "error" and issue.node_id == "write" and issue.field == field
|
||||
assert words in issue.message, issue.message
|
||||
if field in ("judge.max_claims", "judge.max_cost_usd", "judge.timeout_s", "judge.model"):
|
||||
assert "null" in issue.message, "上限、模型都说清写 null 是什么意思"
|
||||
|
||||
|
||||
def test_graph_defaults_are_checked_like_at_run_time():
|
||||
[issue] = issues(graph({"claims": "judge"}, defaults={"judge": {"max_claims": -1}}), "report.judge_invalid")
|
||||
assert issue.field == "judge.max_claims"
|
||||
|
||||
|
||||
def test_the_judge_block_only_matters_when_claims_is_judge():
|
||||
assert issues(graph({"claims": "require_citation", "judge": {"max_claims": -1}})) == []
|
||||
|
||||
|
||||
def test_judging_with_the_writing_model_is_a_warning():
|
||||
[warn] = [i for i in validate_graph(GraphSpec.model_validate(graph(
|
||||
{"claims": "judge", "model": "writer-x", "judge": {"model": "writer-x"}}))).issues
|
||||
if i.code == "report.judge_same_model"]
|
||||
assert warn.level == "warning" and warn.field == "judge.model" and "自己审自己" in warn.message
|
||||
assert not [i for i in validate_graph(GraphSpec.model_validate(graph(
|
||||
{"claims": "judge", "model": "writer-x", "judge": {"model": "judge-y"}}))).issues
|
||||
if i.code == "report.judge_same_model"]
|
||||
@@ -1,10 +1,10 @@
|
||||
"""报告节点的结论句策略 claims:off(默认,和以前一样)/ require_citation(没挂依据的结论句计入缺口)。
|
||||
|
||||
结论句要靠裁判模型判断支不支持,那是后续版本的事;这一版只做确定性的一半:
|
||||
这里测确定性的一半(模型裁判 judge 见 test_report_judge.py):
|
||||
- require_citation 时写作提示明确要求每句结论挂 [[see:…]];没挂的照常产出、计数,
|
||||
出口按出具档位降档(io.py),报告节点自己不为它失败、不为它重写
|
||||
- report.checked 事件和节点产出都带上 claims,出口据此判档
|
||||
- 写 judge 时节点直接报错「结论句裁判在后续版本支持」,不能悄悄当成 off
|
||||
- 写了不认识的值节点直接报错,不能悄悄当成 off
|
||||
- 升级前发起的运行一个字都不变
|
||||
"""
|
||||
from __future__ import annotations
|
||||
@@ -21,7 +21,7 @@ from app.db.models import Run, RunEvent
|
||||
from app.engine import runner as runner_mod
|
||||
from app.engine.evidence import CLAIMS_RULE, build_catalog, compose_doc, uncited_claims
|
||||
from app.engine.runner import run_manager
|
||||
from app.engine.schema import CLAIMS_LATER, REPORT_CLAIMS, claims_problem
|
||||
from app.engine.schema import CLAIMS_VALUES, claims_problem
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
@@ -131,14 +131,14 @@ async def test_require_citation_asks_for_support_but_does_not_block(monkeypatch)
|
||||
assert flagged["text"] == "增长主要来自新客。" and doc["markdown"][slice(*flagged["span"])] == flagged["text"]
|
||||
|
||||
|
||||
async def test_judge_is_refused_until_it_exists(monkeypatch):
|
||||
async def test_a_bad_value_is_refused_before_any_model_call(monkeypatch):
|
||||
seen = script(monkeypatch)
|
||||
with pytest.raises(ValueError) as info: # 画布校验先拦(E3-gov 的 validate)
|
||||
await run_manager.start(graph=weekly(claims="judge"), input_payload={})
|
||||
assert CLAIMS_LATER in str(info.value)
|
||||
with pytest.raises(ValueError) as info: # 画布校验先拦
|
||||
await run_manager.start(graph=weekly(claims="sometimes"), input_payload={})
|
||||
assert claims_problem("sometimes") in str(info.value)
|
||||
lenient(monkeypatch)
|
||||
row = await finish(weekly(claims="judge"))
|
||||
assert row.status == "failed" and claims_problem("judge") in (row.error or ""), row.error
|
||||
row = await finish(weekly(claims="sometimes"))
|
||||
assert row.status == "failed" and claims_problem("sometimes") in (row.error or ""), row.error
|
||||
assert seen == [], "不支持的策略不该先花一次模型调用"
|
||||
|
||||
|
||||
@@ -173,7 +173,7 @@ def test_the_supported_policies_match_the_validator():
|
||||
from app.engine.nodes.report import CLAIMS, report_claims
|
||||
from app.engine.schema import GraphSpec
|
||||
|
||||
assert CLAIMS == REPORT_CLAIMS == ("off", "require_citation")
|
||||
assert CLAIMS == CLAIMS_VALUES == ("off", "require_citation", "judge")
|
||||
spec = GraphSpec.model_validate(weekly(claims="require_citation"))
|
||||
assert report_claims(spec, spec.node_map()["write"]) == "require_citation"
|
||||
spec = GraphSpec.model_validate(weekly())
|
||||
|
||||
@@ -0,0 +1,495 @@
|
||||
"""报告撰写节点的 claims: judge:结论句由另一个模型按证据逐句判断(四期)。
|
||||
|
||||
- 正式运行在节点里裁判:写完、核对完之后判,判定写进报告文档的 unit.verdict,随文档一起封存
|
||||
- 探索运行不在节点里花这笔钱:候选句标「未裁判 · 按需」,点开哪句再判哪句(接口在 B 段)
|
||||
- 判定只标注、不改写正文;裁判没跑成、触顶了,照样产出报告,没判的记未裁判并留下缺口
|
||||
(出口据此不能判完整出具,接线在 B 段)
|
||||
- rewrite_once(默认关):把证据不支持的句子和理由交回写作者只改这几句,再判一次,最多一轮;
|
||||
改出了新的违规就不采用
|
||||
- 裁判调用包在 @task 里、追加在节点已有的调用之后:节点「接着跑」时不重复调用、不重复计费;
|
||||
升级前发起的运行一样都不加,停在审批上的运行恢复后照常
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import re
|
||||
|
||||
import pytest
|
||||
from langchain_core.messages import AIMessage
|
||||
from sqlalchemy import delete, select
|
||||
|
||||
from app.core import artifact_store
|
||||
from app.db.base import SessionLocal
|
||||
from app.db.models import Run, RunEvent, Setting
|
||||
from app.engine import judge as judge_mod
|
||||
from app.engine import replay
|
||||
from app.engine import runner as runner_mod
|
||||
from app.engine.evidence import CLAIMS_RULE, iter_units
|
||||
from app.engine.judge import JUDGE_ROLE, JUDGE_RULE, SPEND_KEY, SUMMARY_KEYS, daily_spend
|
||||
from app.engine.nodes import report as report_mod
|
||||
from app.engine.runner import run_manager, verify_manifest
|
||||
from app.providers.mock_model import MockChatModel
|
||||
|
||||
PRICED = "claude-sonnet-5"
|
||||
METRICS = [{"id": "gmv", "name": "销售额", "unit": "元", "decimals": 1, "expression": "vars.kpi.gmv"}]
|
||||
#: 一句挂了依据的结论、一句没挂依据的归因、一句连接性的话
|
||||
TEXT = "本周销售额 [[m:gmv]]。[[see:m:gmv]]增长主要来自新客首单。下面看细节。"
|
||||
CITED, CAUSAL, LINK = "本周销售额 45,678.5元。", "增长主要来自新客首单。", "下面看细节。"
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
async def engine_up():
|
||||
await run_manager.setup()
|
||||
async with SessionLocal() as session:
|
||||
await session.execute(delete(Setting).where(Setting.key.in_([SPEND_KEY, "judge", "copilot"])))
|
||||
await session.commit()
|
||||
yield
|
||||
await run_manager.shutdown()
|
||||
|
||||
|
||||
def node(nid, ntype, **config):
|
||||
return {"id": nid, "type": ntype, "position": {"x": 0, "y": 0}, "data": {"label": nid, "config": config}}
|
||||
|
||||
|
||||
def weekly(*, gate=False, **report):
|
||||
nodes = [
|
||||
node("start", "input", fields=[{"name": "week", "default": "2026-W37"}]),
|
||||
node("fetch", "transform", mode="expression", expression="{'gmv': 45678.5}", assign_to="kpi"),
|
||||
node("caliber", "metrics", caliber="周报口径", caliber_version="v2", metrics=METRICS),
|
||||
*([node("gate", "human", mode="approve", title="确认口径")] if gate else []),
|
||||
node("write", "report", instructions="写周报", **report),
|
||||
node("out", "output", fields=[{"name": "周报", "value": "{{ nodes.write.text }}"}]),
|
||||
]
|
||||
ids = [n["id"] for n in nodes]
|
||||
return {"nodes": nodes, "edges": [{"source": a, "target": b} for a, b in zip(ids, ids[1:])]}
|
||||
|
||||
|
||||
class Models:
|
||||
"""写作者和裁判两份剧本。写作者依次回 writer 里的几段(用完了一直回最后一段);裁判按句子里的字给判定
|
||||
(verdicts:{句中片段: 判定},没对上的都是 supported)。"""
|
||||
|
||||
def __init__(self, monkeypatch, writer=(TEXT,), verdicts=None, *, judge_model=PRICED, fail_judge=None):
|
||||
self.writes: list[list] = []
|
||||
self.judges: list[str] = []
|
||||
verdicts = verdicts or {}
|
||||
|
||||
def decide(model, messages):
|
||||
if JUDGE_ROLE in str(messages[0].content):
|
||||
request = str(messages[-1].content)
|
||||
self.judges.append(request)
|
||||
if fail_judge:
|
||||
raise RuntimeError(fail_judge)
|
||||
items = []
|
||||
for uid, line in re.findall(r"^\[(u\d+)\]([^\n]*)$", request, re.M):
|
||||
status = next((s for frag, s in verdicts.items() if frag in line), "supported")
|
||||
items.append({"unit": uid, "verdict": status, "rationale": f"判的是:{line[:20]}", "used": []})
|
||||
return AIMessage(content=json.dumps({"items": items}, ensure_ascii=False))
|
||||
self.writes.append(list(messages))
|
||||
return AIMessage(content=writer[min(len(self.writes), len(writer)) - 1])
|
||||
|
||||
monkeypatch.setattr(MockChatModel, "_decide", decide)
|
||||
|
||||
async def judge_model_of(session, spec):
|
||||
return MockChatModel(model_name=judge_model), judge_model
|
||||
|
||||
monkeypatch.setattr(judge_mod, "get_chat_model", judge_model_of)
|
||||
|
||||
|
||||
def spy_tasks(monkeypatch) -> list[str]:
|
||||
"""报告节点里每调用一次 @task 记一笔名字:证明加进来的 task 只在该出现的地方出现。"""
|
||||
from langgraph.func import task as real
|
||||
|
||||
calls: list[str] = []
|
||||
|
||||
def spying(func=None, **kwargs):
|
||||
def wrap(fn):
|
||||
inner = real(fn, **kwargs)
|
||||
|
||||
def call(*args, **kw):
|
||||
calls.append(fn.__name__)
|
||||
return inner(*args, **kw)
|
||||
|
||||
return call
|
||||
return wrap(func) if func is not None else wrap
|
||||
|
||||
monkeypatch.setattr(report_mod, "task", spying)
|
||||
return calls
|
||||
|
||||
|
||||
async def finish(graph, *, run_class="formal", statuses=("succeeded", "failed")) -> Run:
|
||||
run = await run_manager.start(graph=graph, input_payload={}, run_class=run_class)
|
||||
return await wait(run.id, statuses)
|
||||
|
||||
|
||||
async def wait(run_id, statuses=("succeeded", "failed")) -> Run:
|
||||
for _ in range(400):
|
||||
await asyncio.sleep(0.05)
|
||||
async with SessionLocal() as session:
|
||||
row = await session.get(Run, run_id)
|
||||
if row.status in statuses and (row.status == "interrupted" or row.manifest_seq is not None):
|
||||
return row
|
||||
raise AssertionError(f"run {run_id} 没跑完:{row.status}")
|
||||
|
||||
|
||||
async def events(run_id: str, etype: str, node_id: str | None = "write") -> list[RunEvent]:
|
||||
async with SessionLocal() as session:
|
||||
q = select(RunEvent).where(RunEvent.run_id == run_id, RunEvent.type == etype)
|
||||
if node_id:
|
||||
q = q.where(RunEvent.node_id == node_id)
|
||||
return list((await session.execute(q.order_by(RunEvent.seq))).scalars())
|
||||
|
||||
|
||||
async def report_of(run_id: str) -> tuple[dict, dict, dict]:
|
||||
"""(report.checked 的载荷, 报告文档, 节点产出)"""
|
||||
[checked] = await events(run_id, "report.checked")
|
||||
[finished] = await events(run_id, "node.finished")
|
||||
return checked.data, artifact_store.load(checked.data["doc_artifact"]), artifact_store.load(finished.data["artifact"])
|
||||
|
||||
|
||||
def verdicts(doc) -> dict[str, dict | None]:
|
||||
md = doc["markdown"]
|
||||
return {md[u["span"][0]:u["span"][1]].strip(): u.get("verdict") for _, u in iter_units(doc)}
|
||||
|
||||
|
||||
def unit_ids(doc) -> dict[str, str]:
|
||||
md = doc["markdown"]
|
||||
return {md[u["span"][0]:u["span"][1]].strip(): u["id"] for _, u in iter_units(doc)}
|
||||
|
||||
|
||||
async def judge_ends(run_id: str) -> list[dict]:
|
||||
return [e.data for e in await events(run_id, "llm.end") if e.data.get("purpose") == "judge"]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 正式运行:节点内裁判,判定随文档封存
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_a_formal_run_judges_inside_the_node_and_seals_the_verdicts(monkeypatch):
|
||||
models = Models(monkeypatch, verdicts={"新客": "unsupported"})
|
||||
row = await finish(weekly(claims="judge", judge={"on_unsupported": "withhold"}))
|
||||
assert row.status == "succeeded", row.error
|
||||
assert len(models.writes) == 1 and len(models.judges) == 1
|
||||
|
||||
checked, doc, output = await report_of(row.id)
|
||||
got = verdicts(doc)
|
||||
assert got[CITED]["status"] == "supported" and got[CITED]["judge"] == PRICED
|
||||
assert got[CITED]["post_seal"] is False and got[CITED]["rationale"]
|
||||
assert got[CAUSAL]["status"] == "unsupported"
|
||||
assert got[LINK] is None, "连接性的话不送裁判"
|
||||
assert CITED in models.judges[0] and CAUSAL in models.judges[0] and LINK not in models.judges[0]
|
||||
|
||||
summary = checked["judge"]
|
||||
assert tuple(summary) == SUMMARY_KEYS, "B 段照着这份键读:不能悄悄少一个、多一个"
|
||||
assert summary["mode"] == "inline" and summary["model"] == PRICED and summary["on_unsupported"] == "withhold"
|
||||
assert summary["counts"] == {"supported": 1, "partial": 0, "unsupported": 1, "not_a_claim": 0, "unjudged": 0}
|
||||
assert summary["complete"] is True and summary["gaps"] == []
|
||||
assert checked["claims"] == output["claims"] == "judge" and output["judge"] == summary
|
||||
assert doc["judge"]["mode"] == "inline" and doc["judge"]["counts"] == summary["counts"]
|
||||
assert {k: doc["stats"][k] for k in ("supported", "partial", "unsupported", "not_a_claim", "unjudged")} \
|
||||
== summary["counts"]
|
||||
|
||||
[end] = await judge_ends(row.id)
|
||||
assert end["model"] == PRICED and end["cost_usd"] >= 0
|
||||
verdict = await verify_manifest(row.id)
|
||||
assert verdict["ok"] is True
|
||||
[event] = await events(row.id, "report.checked")
|
||||
assert event.seq <= row.manifest_seq, "判定在封存范围内"
|
||||
|
||||
|
||||
async def test_the_writer_is_told_that_claims_will_be_judged(monkeypatch):
|
||||
models = Models(monkeypatch)
|
||||
await finish(weekly(claims="judge"))
|
||||
prompt = "\n".join(str(m.content) for m in models.writes[0])
|
||||
assert CLAIMS_RULE in prompt and JUDGE_RULE in prompt
|
||||
|
||||
|
||||
async def test_explore_runs_leave_it_to_the_reader(monkeypatch):
|
||||
"""探索运行默认按需裁判:节点里不调用,候选句标「未裁判 · 按需」,连接性的话什么都不标。"""
|
||||
models = Models(monkeypatch)
|
||||
row = await finish(weekly(claims="judge"), run_class="exploratory")
|
||||
assert row.status == "succeeded", row.error
|
||||
assert models.judges == []
|
||||
checked, doc, output = await report_of(row.id)
|
||||
got = verdicts(doc)
|
||||
for text in (CITED, CAUSAL):
|
||||
assert got[text]["status"] == "unjudged" and got[text]["reason"] == "on_demand"
|
||||
assert got[text]["judge"] is None and got[text]["post_seal"] is False
|
||||
assert got[LINK] is None
|
||||
assert checked["judge"]["mode"] == "on_demand" and checked["judge"]["gaps"] == []
|
||||
assert tuple(checked["judge"]) == SUMMARY_KEYS and checked["judge"]["limits"] is None
|
||||
assert output["judge"] == checked["judge"] and await judge_ends(row.id) == []
|
||||
|
||||
|
||||
async def test_a_failed_judge_leaves_a_gap_not_a_failed_report(monkeypatch):
|
||||
Models(monkeypatch, fail_judge="网关超时")
|
||||
row = await finish(weekly(claims="judge"))
|
||||
assert row.status == "succeeded", row.error
|
||||
checked, doc, output = await report_of(row.id)
|
||||
assert {verdicts(doc)[t]["reason"] for t in (CITED, CAUSAL)} == {"error"}
|
||||
summary = checked["judge"]
|
||||
assert summary["complete"] is False and summary["counts"]["unjudged"] == 2
|
||||
assert summary["gaps"] and "网关超时" in summary["gaps"][0]
|
||||
assert output["judge"]["gaps"] == summary["gaps"]
|
||||
[end] = await judge_ends(row.id)
|
||||
assert "网关超时" in end["error"]
|
||||
|
||||
|
||||
async def test_hitting_a_limit_is_reported_not_raised(monkeypatch):
|
||||
models = Models(monkeypatch)
|
||||
row = await finish(weekly(claims="judge", judge={"max_claims": 1}))
|
||||
assert row.status == "succeeded", row.error
|
||||
checked, doc, _ = await report_of(row.id)
|
||||
assert verdicts(doc)[CITED]["status"] == "supported", "先判挂了依据的"
|
||||
assert (verdicts(doc)[CAUSAL]["status"], verdicts(doc)[CAUSAL]["reason"]) == ("unjudged", "max_claims")
|
||||
assert checked["judge"]["limits_hit"] == ["max_claims"] and "已到上限" in checked["judge"]["gaps"][0]
|
||||
assert CAUSAL not in models.judges[0]
|
||||
|
||||
|
||||
async def test_the_node_budget_overrides_the_settings_and_null_means_unlimited(monkeypatch):
|
||||
"""节点 judge 里写了的键盖过设置;写 null 就是不限(不是「没写」)。"""
|
||||
async with SessionLocal() as session:
|
||||
session.add(Setting(key="judge", value={"report_max_claims": 1, "report_max_cost_usd": 1e-9}))
|
||||
await session.commit()
|
||||
Models(monkeypatch)
|
||||
row = await finish(weekly(claims="judge", judge={"max_cost_usd": None}))
|
||||
checked, doc, _ = await report_of(row.id)
|
||||
assert verdicts(doc)[CITED]["status"] == "supported"
|
||||
assert verdicts(doc)[CAUSAL]["reason"] == "max_claims", "max_claims 没写,取设置里的 1"
|
||||
assert checked["judge"]["limits"]["max_cost_usd"] is None and checked["judge"]["limits"]["max_claims"] == 1
|
||||
|
||||
|
||||
async def test_judging_yourself_is_called_out(monkeypatch):
|
||||
Models(monkeypatch, judge_model="mock-fast")
|
||||
row = await finish(weekly(claims="judge"))
|
||||
assert row.status == "succeeded", row.error
|
||||
[warn] = [e.data for e in await events(row.id, "log") if e.data.get("code") == "judge_same_model"]
|
||||
assert "mock-fast" in warn["message"] and "自己审自己" in warn["message"]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# rewrite_once
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
REWRITTEN = "本周销售额 [[m:gmv]]。[[see:m:gmv]]新客首单的贡献要等新客维度的数据再看。下面看细节。"
|
||||
|
||||
|
||||
async def test_rewrite_once_hands_back_only_the_unsupported_sentences(monkeypatch):
|
||||
models = Models(monkeypatch, writer=(TEXT, REWRITTEN),
|
||||
verdicts={"主要来自新客": "unsupported", "要等新客维度": "not_a_claim"})
|
||||
row = await finish(weekly(claims="judge", judge={"rewrite_once": True}))
|
||||
assert row.status == "succeeded", row.error
|
||||
assert len(models.writes) == 2 and len(models.judges) == 2
|
||||
|
||||
request = str(models.writes[1][-1].content)
|
||||
assert CAUSAL in request and "判的是:" in request and CITED not in request, "只交回不支持的那几句和理由"
|
||||
# 第二次只判改过的句子:没改的那句直接用第一次的判定
|
||||
assert "要等新客维度" in models.judges[1] and CITED not in models.judges[1]
|
||||
|
||||
checked, doc, output = await report_of(row.id)
|
||||
assert "新客首单的贡献要等新客维度的数据再看。" in doc["markdown"] and CAUSAL not in doc["markdown"]
|
||||
got = verdicts(doc)
|
||||
assert got[CITED]["status"] == "supported" and got["新客首单的贡献要等新客维度的数据再看。"]["status"] == "not_a_claim"
|
||||
rewrite = checked["judge"]["rewrite"]
|
||||
assert rewrite["applied"] is True and rewrite["sentences"] == [CAUSAL] and checked["judge"]["reused"] == 1
|
||||
# 封存的是改写稿:units 按句子认封存文档里的编号,交回去的那句已经改掉了,对不上就是 null——
|
||||
# 不能拿原稿的编号去指改写稿里碰巧同号的另一句;改写稿里新出来的句子在 changed 里
|
||||
ids = unit_ids(doc)
|
||||
assert rewrite["units"] == [None]
|
||||
assert rewrite["changed"] == [ids["新客首单的贡献要等新客维度的数据再看。"]]
|
||||
assert output["text"] == doc["markdown"]
|
||||
[log] = [e.data for e in await events(row.id, "log") if e.data.get("code") == "report_rewrite"]
|
||||
assert "1 句" in log["message"]
|
||||
assert len(await judge_ends(row.id)) == 2
|
||||
|
||||
|
||||
async def test_rewrite_once_is_off_by_default(monkeypatch):
|
||||
models = Models(monkeypatch, writer=(TEXT, REWRITTEN), verdicts={"主要来自新客": "unsupported"})
|
||||
row = await finish(weekly(claims="judge"))
|
||||
assert row.status == "succeeded", row.error
|
||||
assert len(models.writes) == 1 and len(models.judges) == 1
|
||||
checked, doc, _ = await report_of(row.id)
|
||||
assert verdicts(doc)[CAUSAL]["status"] == "unsupported" and checked["judge"].get("rewrite") is None
|
||||
|
||||
|
||||
async def test_a_rewrite_that_breaks_the_numbers_is_not_used(monkeypatch):
|
||||
models = Models(monkeypatch, writer=(TEXT, "本周销售额 [[m:gmv]]。[[see:m:gmv]]新客贡献了 45678 元。下面看细节。"),
|
||||
verdicts={"主要来自新客": "unsupported"})
|
||||
row = await finish(weekly(claims="judge", judge={"rewrite_once": True}))
|
||||
assert row.status == "succeeded", row.error
|
||||
assert len(models.writes) == 2 and len(models.judges) == 1, "改坏了的那一稿不再花钱判"
|
||||
checked, doc, output = await report_of(row.id)
|
||||
assert CAUSAL in doc["markdown"] and "45678" not in doc["markdown"]
|
||||
assert verdicts(doc)[CAUSAL]["status"] == "unsupported"
|
||||
rewrite = checked["judge"]["rewrite"]
|
||||
assert rewrite["applied"] is False and "45678" in rewrite["reason"]
|
||||
# 封存的是原稿:units 就是原稿(也就是封存文档)里的编号
|
||||
assert rewrite["units"] == [unit_ids(doc)[CAUSAL]] and rewrite["changed"] == []
|
||||
|
||||
|
||||
def with_schema(monkeypatch):
|
||||
"""报告的目录里加一张表:实体层有了核对的底,反引号里的名字才会被认出是编造的。"""
|
||||
from app.engine.evidence import _entity_entries
|
||||
|
||||
real = report_mod.report_catalog
|
||||
|
||||
def catalog_with_schema(state, spec, node):
|
||||
return {**real(state, spec, node), **_entity_entries(
|
||||
[{"artifact": "d" * 64, "source": "shop", "tables": {"orders": ["order_id", "amount"]}}], [])}
|
||||
|
||||
monkeypatch.setattr(report_mod, "report_catalog", catalog_with_schema)
|
||||
|
||||
|
||||
async def test_a_rewrite_that_makes_up_a_name_is_not_used(monkeypatch):
|
||||
"""改写稿冒出一个编造的表名(只标注、不拦的那类违规):同样是原稿没有的新违规,不采用。"""
|
||||
with_schema(monkeypatch)
|
||||
models = Models(monkeypatch, writer=(TEXT, "本周销售额 [[m:gmv]]。[[see:m:gmv]]新客首单的贡献要看 `refund_detail` 表。"
|
||||
"下面看细节。"),
|
||||
verdicts={"主要来自新客": "unsupported"})
|
||||
row = await finish(weekly(claims="judge", judge={"rewrite_once": True}))
|
||||
assert row.status == "succeeded", row.error
|
||||
assert len(models.writes) == 2 and len(models.judges) == 1
|
||||
checked, doc, _ = await report_of(row.id)
|
||||
assert CAUSAL in doc["markdown"] and "refund_detail" not in doc["markdown"]
|
||||
rewrite = checked["judge"]["rewrite"]
|
||||
assert rewrite["applied"] is False and "refund_detail" in rewrite["reason"]
|
||||
[log] = [e.data for e in await events(row.id, "log") if e.data.get("code") == "report_rewrite_rejected"]
|
||||
assert "refund_detail" in log["message"]
|
||||
|
||||
|
||||
async def test_a_rewrite_that_trades_one_violation_for_another_is_not_used(monkeypatch):
|
||||
"""去掉一处违规、又冒出另一处:总数没变,可冒出来的是原稿没有的,一样不采用。"""
|
||||
draft = "本周销售额 [[m:gmv]]。[[see:m:gmv]]增长主要来自新客首单,共 1200 单。下面看细节。"
|
||||
swapped = "本周销售额 [[m:gmv]]。[[see:m:gmv]]新客首单约占 3500 单。下面看细节。"
|
||||
models = Models(monkeypatch, writer=(draft, draft, swapped), verdicts={"主要来自新客": "unsupported"})
|
||||
row = await finish(weekly(claims="judge", on_violation="flag", max_repairs=1, judge={"rewrite_once": True}))
|
||||
assert row.status == "succeeded", row.error
|
||||
assert len(models.writes) == 3 and len(models.judges) == 1
|
||||
checked, doc, _ = await report_of(row.id)
|
||||
assert "1200" in doc["markdown"] and "3500" not in doc["markdown"]
|
||||
rewrite = checked["judge"]["rewrite"]
|
||||
assert rewrite["applied"] is False and "3500" in rewrite["reason"] and "1200" not in rewrite["reason"]
|
||||
|
||||
|
||||
def test_new_violations_are_counted_one_by_one():
|
||||
"""改写稿的违规逐个和原稿抵消:原样留着的不算新的,同一个裸数字多写一处、只标注的那类,都算新的。"""
|
||||
number = {"code": "uncited_number", "text": "1200", "message": "数字「1200」没有出处"}
|
||||
name = {"code": "unknown_entity", "text": "refund_detail", "message": "可能是编造的名字"}
|
||||
see = {"code": "unresolved_ref", "ref": "Q9", "message": "依据 [[see:Q9]] 解析不了"}
|
||||
assert report_mod._new_violations([number, name, see], [see, number, name]) == []
|
||||
assert report_mod._new_violations([number], [number, number]) == [number]
|
||||
assert report_mod._new_violations([number], [name]) == [name]
|
||||
assert report_mod._new_violations([number, see], []) == []
|
||||
|
||||
|
||||
async def test_a_second_round_that_hits_a_limit_names_the_configured_limit(monkeypatch):
|
||||
"""第二轮用的是第一轮剩下的钱,可说给人看的上限是节点上配的那个数,不是剩下的零头。"""
|
||||
Models(monkeypatch, writer=(TEXT, REWRITTEN), verdicts={"主要来自新客": "unsupported"})
|
||||
# 每句估 1 分钱、回复不带用量(实际就按估算记):第一轮两句花掉 2 分,第二轮只剩半分,一句也装不下
|
||||
monkeypatch.setattr(judge_mod.JudgeRequest, "cost", lambda self, model_id, units: 0.01 * len(units))
|
||||
monkeypatch.setattr(judge_mod, "_usage_of", lambda message, model_id: {"input_tokens": 0, "output_tokens": 0})
|
||||
row = await finish(weekly(claims="judge", judge={"rewrite_once": True, "max_cost_usd": 0.025}))
|
||||
assert row.status == "succeeded", row.error
|
||||
checked, doc, _ = await report_of(row.id)
|
||||
v = verdicts(doc)["新客首单的贡献要等新客维度的数据再看。"]
|
||||
assert (v["status"], v["reason"]) == ("unjudged", "max_cost_usd")
|
||||
assert "$0.025" in v["rationale"] and "0.005" not in v["rationale"], v["rationale"]
|
||||
summary = checked["judge"]
|
||||
assert summary["rewrite"]["applied"] is True and summary["limits"]["max_cost_usd"] == 0.025
|
||||
assert "$0.025" in summary["gaps"][0] and "0.005" not in summary["gaps"][0], summary["gaps"]
|
||||
[log] = [e.data for e in await events(row.id, "log") if e.data.get("code") == "judge_limit"]
|
||||
assert "$0.025" in log["message"] and "0.005" not in log["message"]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 重放安全
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_continuing_after_a_failure_does_not_judge_or_charge_twice(monkeypatch):
|
||||
models = Models(monkeypatch)
|
||||
real = report_mod._store
|
||||
broken = {"on": True}
|
||||
|
||||
async def flaky(doc, ctx):
|
||||
if broken["on"]:
|
||||
raise RuntimeError("落盘时断了")
|
||||
return await real(doc, ctx)
|
||||
|
||||
monkeypatch.setattr(report_mod, "_store", flaky)
|
||||
run = await run_manager.start(graph=weekly(claims="judge"), input_payload={}, run_class="formal")
|
||||
row = await wait(run.id)
|
||||
assert row.status == "failed" and "落盘时断了" in (row.error or ""), row.error
|
||||
assert len(models.judges) == 1
|
||||
before = await daily_spend()
|
||||
await run_manager.wait_idle(run.id)
|
||||
|
||||
broken["on"] = False
|
||||
await run_manager.continue_failed(run.id)
|
||||
row = await wait(run.id)
|
||||
assert row.status == "succeeded", row.error
|
||||
assert len(models.judges) == 1, "接着跑时裁判又调了一次:裁判的结果没进 checkpoint"
|
||||
assert await daily_spend() == before, "接着跑时每日计数又记了一次"
|
||||
assert len(await judge_ends(row.id)) == 1
|
||||
_, doc, _ = await report_of(row.id)
|
||||
assert verdicts(doc)[CITED]["status"] == "supported"
|
||||
|
||||
|
||||
async def test_a_changed_draft_is_judged_afresh_when_continuing(monkeypatch):
|
||||
"""接着跑时写作者重写了一稿(写作调用不进 checkpoint),缓存的判定是上一稿的:不能拿来用。"""
|
||||
models = Models(monkeypatch, writer=(TEXT, "本周销售额 [[m:gmv]]。[[see:m:gmv]]增长主要来自老客复购。下面看细节。"))
|
||||
real = report_mod._store
|
||||
broken = {"on": True}
|
||||
|
||||
async def flaky(doc, ctx):
|
||||
if broken["on"]:
|
||||
raise RuntimeError("落盘时断了")
|
||||
return await real(doc, ctx)
|
||||
|
||||
monkeypatch.setattr(report_mod, "_store", flaky)
|
||||
run = await run_manager.start(graph=weekly(claims="judge"), input_payload={}, run_class="formal")
|
||||
assert (await wait(run.id)).status == "failed"
|
||||
await run_manager.wait_idle(run.id)
|
||||
broken["on"] = False
|
||||
await run_manager.continue_failed(run.id)
|
||||
row = await wait(run.id)
|
||||
assert row.status == "succeeded", row.error
|
||||
assert len(models.judges) == 2 and "老客复购" in models.judges[1]
|
||||
_, doc, _ = await report_of(row.id)
|
||||
assert verdicts(doc)["增长主要来自老客复购。"]["status"] == "supported"
|
||||
|
||||
|
||||
async def test_runs_from_before_the_upgrade_never_judge(monkeypatch):
|
||||
"""升级前发起的运行(没有护栏快照):claims 一律按 off,节点里一个 task 都不多调用,产出一个键都不多。"""
|
||||
async def no_snapshot(session):
|
||||
return None
|
||||
|
||||
monkeypatch.setattr(runner_mod, "_agent_limits", no_snapshot)
|
||||
models = Models(monkeypatch)
|
||||
calls = spy_tasks(monkeypatch)
|
||||
row = await finish(weekly(claims="judge"))
|
||||
assert row.status == "succeeded", row.error
|
||||
assert models.judges == [] and calls == []
|
||||
checked, doc, output = await report_of(row.id)
|
||||
assert "judge" not in checked and "claims" not in checked and "judge" not in output and "judge" not in doc
|
||||
assert all(v is None for v in verdicts(doc).values())
|
||||
|
||||
|
||||
async def test_runs_paused_at_an_approval_resume_with_tasks_in_place(monkeypatch):
|
||||
"""停在审批上的运行恢复:审批节点照旧只问一次;报告节点在恢复之后才第一次执行,裁判 task 只调一次、
|
||||
排在节点所有已有调用之后。没开裁判的报告节点一个 task 都不调用(和升级前一样)。"""
|
||||
assert replay.PROTOCOL == 2, "没有改动已有 task 的顺序,重放协议不升版"
|
||||
for claims, expected in (("require_citation", []), ("judge", ["judge_step"])):
|
||||
models = Models(monkeypatch)
|
||||
calls = spy_tasks(monkeypatch)
|
||||
paused = await finish(weekly(gate=True, claims=claims), statuses=("interrupted", "failed"))
|
||||
assert paused.status == "interrupted", paused.error
|
||||
await run_manager.resume(paused.id, {"approved": True})
|
||||
row = await wait(paused.id)
|
||||
assert row.status == "succeeded", row.error
|
||||
assert calls == expected, (claims, calls)
|
||||
assert len(models.judges) == len(expected)
|
||||
assert len(await events(row.id, "human.requested", "gate")) == 1
|
||||
checked, _, _ = await report_of(row.id)
|
||||
assert ("judge" in checked) is (claims == "judge")
|
||||
Reference in New Issue
Block a user