mirror of
https://github.com/ThearchyHelios/AgentLab.git
synced 2026-11-20 17:41:57 +00:00
c97390511e
渲染(evidence._see_cuts): - 括号是两边唯一的分隔、去掉之后数字字母会粘成一串的(「[[v:Q1.r0.销售额]]([[see:Q1]])2 月」渲染成「4,919.52 月」,片段 本身没错、月份不算裸数字,哪项核对都查不出来),括号换成一个空格;半角逗号夹着依据、去掉后一个之后同理(「212,5」→ 「212, 5」)。切口可以带替换串,连成一片的按并集换 - 只装着依据的括号去掉之后,整对括号当一个依据看外面:括号前悬着的停顿标点(「华南居首,([[see:Q1]])。」)、到句末只隔着 的空白、夹在两个分隔标点中间的后一个,和不带括号的写法收拾得一样 - 依据后面到句末标点只隔着空白的,空白一起去掉(「华南居首 [[see:Q1]] 。」不再剩「华南居首 。」) - 换行不再算句末:列表项末尾的「;」、软换行处连着两个分句的「,」是写作者自己的,依据挂在它后面正合规矩,照旧留着(上一轮 把它们删了) - 没有依据的文档渲染、内容哈希不变(随机两万份和基线逐份比过);流式分块输出仍等于整篇渲染 - 依据写在「依据」「见」「取自」「来源」这类引出语后面时引出语悬空,渲染时去不掉(引出语在依据前面,流式渲染得攒住每个可能 是引出语的字):测试里改成严格的 xfail 写明已知限制,不再把验收里的缺陷钉成预期;核查 agent 遇到这种写法记一条日志 核查 agent(verify_agent): - 有没有给理由按渲染出来的正文判:只写了依据([[see:Q1]]、「([[see:Q1]])」)显示出来是空的,占位词后面挂着依据的显示出来 只剩占位词,都算没给。以前按原文判,标记里的字也算,理由文档照样交回:审批卡画出一片空白,判「被推翻」的因为引用到了 Q1 被当成有出处、出具降档;封存之后的质询回答只剩系统说明却记成回答成功 - 「无」「暂无」「none」只在理由是另一个结果的说明时(双盲复算、反方审稿)算没给;结构里只有理由一项(审批质询)时理由就是 回答,「无。」是一个回答,不再报「模型没有给出回答内容」让提问人重问、再花一次钱 - 收尾超时的原话收成常量 SETTLE_TIMEOUT,调用方据此换成自己的说法 反方审稿:交结论那次超过时长上限的句子,理由写「反查未完成:已达上限(每句时长上限 N 秒)」(收紧过的说运行时长上限将至), 和时间线上的 challenge_limit 一致,不再把核查 agent 的内部原话摆上审批卡。 测试:补节点级用例钉住收尾超时之后两个节点对外的说法(challenge_limit 的 limit、claims_skipped、cause,双盲复算的无法复算 原因和 dual_check_limit,收紧过的两种);理由只有依据、占位词挂依据、「无。」作为回答的各层用例。每处改动都做过去掉实现、 测试变红的验证。
86 lines
4.9 KiB
Python
86 lines
4.9 KiB
Python
"""反方审稿的理由:依据写进括号不再断句,占位词不当理由(期 5 真实模型验收 M1)。
|
||
|
||
真实模型验收里,反方的理由「先按订单状态做全量概览核对口径([[see:Q1]]):」「复算写作者的原查询(同 SQL 同过滤条件,
|
||
[[see:Q2]])」渲染成「():」「(同 SQL 同过滤条件,)」:审稿记录(rationale 和 rationale_doc)随正式运行封存,审批卡上
|
||
照片段原文显示。
|
||
|
||
模型一律是 MockChatModel 剧本,区域、金额都是虚构的。
|
||
"""
|
||
from __future__ import annotations
|
||
|
||
from app.core import artifact_store
|
||
from tests.test_challenge_node import ( # noqa: F401 - 夹具按名字引入
|
||
NET,
|
||
REFUNDS,
|
||
Models,
|
||
Play,
|
||
engine_up,
|
||
finish,
|
||
graph,
|
||
output_of,
|
||
shop,
|
||
)
|
||
|
||
BROKEN = ("()", "(,", ",)", "()", "(,", ",)")
|
||
|
||
|
||
async def _record(monkeypatch, shop, plays) -> tuple[dict, dict]:
|
||
Models(monkeypatch, shop, plays=plays)
|
||
row = await finish(graph(shop))
|
||
assert row.status == "succeeded", row.error
|
||
out = await output_of(row.id, "chal")
|
||
return artifact_store.load(out["challenge_artifact"]), row.output["_issuance"]
|
||
|
||
|
||
async def test_see_in_brackets_does_not_break_the_rationale(monkeypatch, shop):
|
||
plays = {
|
||
"华南销售额超过华东": Play([NET], "refuted", "扣除退款后华南为 [[v:Q1.r1.净销售额]]([[see:Q1]]),"
|
||
"仍低于华东的 [[v:Q1.r0.净销售额]](同口径,[[see:Q1]])。"),
|
||
"居各区域之首": Play([REFUNDS], "weakened", "先核对退款口径([[see:Q1]]):华东含 [[v:Q1.r0.退款笔数]] 笔退款"
|
||
"([[see:Q1]],扣除后仍居首)。"),
|
||
}
|
||
record, issuance = await _record(monkeypatch, shop, plays)
|
||
first, second = record["claims"]
|
||
assert first["rationale"] == "先核对退款口径:华东含 1 笔退款(扣除后仍居首)。"
|
||
assert second["rationale"] == "扣除退款后华南为 900,仍低于华东的 1,200(同口径)。"
|
||
for claim in (first, second):
|
||
doc = claim["rationale_doc"]
|
||
assert doc["markdown"] == claim["rationale"] and doc["violations"] == []
|
||
assert not any(mark in claim["rationale"] for mark in BROKEN)
|
||
# 括号里的依据仍挂在句子上:判「被推翻」的引用到了自己的查询,照常采信、出具降档
|
||
assert second["verdict"] == "refuted" and issuance["tier"] == "degraded"
|
||
|
||
|
||
async def test_a_placeholder_rationale_is_recorded_as_no_rationale(monkeypatch, shop):
|
||
"""反方交回的理由只是占位词:记录里理由为空(审批卡据此写「反方没有给出理由」),判「被推翻」的照常不予采信。"""
|
||
plays = {"华南销售额超过华东": Play([NET], "refuted", "placeholder"),
|
||
"居各区域之首": Play([REFUNDS], "stands", "placeholder")}
|
||
record, issuance = await _record(monkeypatch, shop, plays)
|
||
first, second = record["claims"]
|
||
assert (first["verdict"], first["rationale"], first["rationale_doc"]) == ("stands", "", None)
|
||
assert second["verdict"] == "inconclusive"
|
||
assert second["rationale"] == "理由没有引用到任何查询结果,反方给出的「被推翻」不予采信,记为无法判断。反方没有给出理由"
|
||
assert "placeholder" not in str(record)
|
||
assert issuance["tier"] == "formal"
|
||
|
||
|
||
async def test_a_rationale_that_shows_nothing_is_recorded_as_no_rationale(monkeypatch, shop):
|
||
"""反方的理由只写了依据:显示出来是空的。以前按原文判,理由文档照样记下——审批卡按文档画出一片空白,不说「反方没有
|
||
给出理由」;判「被推翻」的因为引用到了 Q1 被当成有出处,照常采信、出具降档。"""
|
||
plays = {"华南销售额超过华东": Play([NET], "refuted", "[[see:Q1]]"),
|
||
"居各区域之首": Play([REFUNDS], "stands", "([[see:Q1]])")}
|
||
record, issuance = await _record(monkeypatch, shop, plays)
|
||
first, second = record["claims"]
|
||
assert (first["verdict"], first["rationale"], first["rationale_doc"]) == ("stands", "", None)
|
||
assert second["verdict"] == "inconclusive" and second["rationale_doc"] is None
|
||
assert second["rationale"].endswith("反方给出的「被推翻」不予采信,记为无法判断。反方没有给出理由")
|
||
assert issuance["tier"] == "formal", "没有理由的「被推翻」不能让出具降档"
|
||
|
||
|
||
async def test_a_placeholder_with_a_see_is_still_a_placeholder(monkeypatch, shop):
|
||
plays = {"华南销售额超过华东": Play([NET], "stands", "placeholder。[[see:Q1]]"),
|
||
"居各区域之首": Play([REFUNDS], "stands", "无。")}
|
||
record, _ = await _record(monkeypatch, shop, plays)
|
||
assert [(c["rationale"], c["rationale_doc"]) for c in record["claims"]] == [("", None), ("", None)]
|
||
assert "placeholder" not in str(record)
|