mirror of
https://github.com/ThearchyHelios/AgentLab.git
synced 2026-11-15 00:21:58 +00:00
M1 复核修正(第 1 轮):去掉只装着依据的括号不再把两边的数字粘成另一个数;括号去掉之后外面照一个依据再收拾;换行不算句末;核查 agent 的理由按渲染后的正文判有没有给;反方审稿收尾超时按时长上限说
渲染(evidence._see_cuts): - 括号是两边唯一的分隔、去掉之后数字字母会粘成一串的(「[[v:Q1.r0.销售额]]([[see:Q1]])2 月」渲染成「4,919.52 月」,片段 本身没错、月份不算裸数字,哪项核对都查不出来),括号换成一个空格;半角逗号夹着依据、去掉后一个之后同理(「212,5」→ 「212, 5」)。切口可以带替换串,连成一片的按并集换 - 只装着依据的括号去掉之后,整对括号当一个依据看外面:括号前悬着的停顿标点(「华南居首,([[see:Q1]])。」)、到句末只隔着 的空白、夹在两个分隔标点中间的后一个,和不带括号的写法收拾得一样 - 依据后面到句末标点只隔着空白的,空白一起去掉(「华南居首 [[see:Q1]] 。」不再剩「华南居首 。」) - 换行不再算句末:列表项末尾的「;」、软换行处连着两个分句的「,」是写作者自己的,依据挂在它后面正合规矩,照旧留着(上一轮 把它们删了) - 没有依据的文档渲染、内容哈希不变(随机两万份和基线逐份比过);流式分块输出仍等于整篇渲染 - 依据写在「依据」「见」「取自」「来源」这类引出语后面时引出语悬空,渲染时去不掉(引出语在依据前面,流式渲染得攒住每个可能 是引出语的字):测试里改成严格的 xfail 写明已知限制,不再把验收里的缺陷钉成预期;核查 agent 遇到这种写法记一条日志 核查 agent(verify_agent): - 有没有给理由按渲染出来的正文判:只写了依据([[see:Q1]]、「([[see:Q1]])」)显示出来是空的,占位词后面挂着依据的显示出来 只剩占位词,都算没给。以前按原文判,标记里的字也算,理由文档照样交回:审批卡画出一片空白,判「被推翻」的因为引用到了 Q1 被当成有出处、出具降档;封存之后的质询回答只剩系统说明却记成回答成功 - 「无」「暂无」「none」只在理由是另一个结果的说明时(双盲复算、反方审稿)算没给;结构里只有理由一项(审批质询)时理由就是 回答,「无。」是一个回答,不再报「模型没有给出回答内容」让提问人重问、再花一次钱 - 收尾超时的原话收成常量 SETTLE_TIMEOUT,调用方据此换成自己的说法 反方审稿:交结论那次超过时长上限的句子,理由写「反查未完成:已达上限(每句时长上限 N 秒)」(收紧过的说运行时长上限将至), 和时间线上的 challenge_limit 一致,不再把核查 agent 的内部原话摆上审批卡。 测试:补节点级用例钉住收尾超时之后两个节点对外的说法(challenge_limit 的 limit、claims_skipped、cause,双盲复算的无法复算 原因和 dual_check_limit,收紧过的两种);理由只有依据、占位词挂依据、「无。」作为回答的各层用例。每处改动都做过去掉实现、 测试变红的验证。
This commit is contained in:
@@ -1818,11 +1818,17 @@ _OPEN, _CLOSE = frozenset("(("), frozenset("))")
|
||||
#: 依据和括号、依据和依据之间的分隔标点(「([[see:Q3]],原报告…)」「[[see:Q1]]、[[see:Q2]]」)
|
||||
_SEPS = ",、;,;::"
|
||||
_SEP = frozenset(_SEPS)
|
||||
#: 其中的半角标点:按排版习惯后面隔一个空格。两个夹着依据的去掉后一个时,前一个不能直接贴着下文(「212,5」读成一个数)
|
||||
_HALF = frozenset(",;:")
|
||||
#: 其中的停顿标点(不含冒号):依据挂在它后面、紧接着就到句末时,它悬在句末(「华南居首,。」)。冒号后面另起一行
|
||||
#: 常常是它引出的表格、列表,不动
|
||||
_PAUSE = _SEP - frozenset("::")
|
||||
#: 句末标点:停顿标点悬在它前面时去掉(还有换行、全文结尾)。半角句点不算:后面跟着数字时它是小数点
|
||||
#: 句末标点:停顿标点悬在它前面时去掉(还有全文结尾)。半角句点不算:后面跟着数字时它是小数点。换行不算:列表项末尾
|
||||
#: 的「;」、软换行处连着两个分句的「,」是写作者自己的,依据挂在它后面正合「挂在句末」的规矩
|
||||
_ENDING = frozenset("。!?!?")
|
||||
#: 去掉一对括号之后会粘成一串的字:前面是数字、字母(或者标记的结尾、百分号、半角的逗号和小数点),后面是数字、字母
|
||||
#: (或者一个渲染出字的标记)。中文和数字之间不加空格也读得对,不算
|
||||
_NUMBERISH = frozenset("]%‰.,")
|
||||
#: 依据周围、渲染后会挤在一起的字:空白和分隔标点
|
||||
_GAP = frozenset(_BLANK) | _SEP
|
||||
#: 连着写的几个依据:中间只隔着空白、分隔标点
|
||||
@@ -1847,11 +1853,12 @@ def _tidy(text: str, catalog: dict[str, Any], snaps: _Snapshots, cells_allowed:
|
||||
还跟着写一遍([[m:visits]] 人次),读者看到的是「12,345人次 人次」。单字的单位后面紧跟着字时
|
||||
不动:写作者写的可能是另一个单位(口径卡是「人」,正文写「人次」),硬删会把不一致藏起来;
|
||||
- [[see:…]] 渲染成空,它周围留下的空白、括号、标点去掉(_see_cuts):「最高月份 [[see:Q2]]。」会剩下
|
||||
「最高月份 。」,「结果一致([[see:Q3]],原报告…)」会剩下「结果一致(,原报告…)」。
|
||||
「最高月份 。」,「结果一致([[see:Q3]],原报告…)」会剩下「结果一致(,原报告…)」。去掉之后两边的数字、字母
|
||||
会粘成一串的,换成一个空格(_glued)。
|
||||
|
||||
prev 是这段文字前面那个字(流式渲染分块时用),判断空白是不是在行首。
|
||||
"""
|
||||
cuts: list[tuple[int, int]] = []
|
||||
cuts: list[tuple[int, ...]] = []
|
||||
for marker in parse_markers(text):
|
||||
if marker["kind"] not in _UNIT_KINDS:
|
||||
continue
|
||||
@@ -1867,34 +1874,43 @@ def _tidy(text: str, catalog: dict[str, Any], snaps: _Snapshots, cells_allowed:
|
||||
cuts.extend(_see_cuts(text, group.start(), group.end(), prev))
|
||||
if not cuts:
|
||||
return text
|
||||
# 几处要去掉的字可能重叠(依据前面的空白、它和括号之间的标点):按并集去掉
|
||||
out, last = [], 0
|
||||
for a, b in sorted(cuts):
|
||||
if a >= last:
|
||||
out.append(text[last:a])
|
||||
last = b
|
||||
# 几处要去掉的字可能重叠(依据前面的空白、它和括号之间的标点):按并集去掉。带着替换的(第三项,去掉之后两边会
|
||||
# 粘在一起,换成一个空格)连成一片时整片换成它
|
||||
merged: list[list[Any]] = []
|
||||
for a, b, *rep in sorted(cuts):
|
||||
if merged and a < merged[-1][1]:
|
||||
merged[-1][1] = max(merged[-1][1], b)
|
||||
merged[-1][2] = merged[-1][2] or "".join(rep)
|
||||
else:
|
||||
last = max(last, b)
|
||||
merged.append([a, b, "".join(rep)])
|
||||
out, last = [], 0
|
||||
for a, b, rep in merged:
|
||||
out += [text[last:a], rep]
|
||||
last = b
|
||||
out.append(text[last:])
|
||||
return "".join(out)
|
||||
|
||||
|
||||
def _see_cuts(text: str, start: int, end: int, prev: str) -> list[tuple[int, int]]:
|
||||
def _see_cuts(text: str, start: int, end: int, prev: str, *, whole: bool = False) -> list[tuple[int, ...]]:
|
||||
"""一组依据(_SEE_GROUP,text[start:end])渲染成空之后,周围要一起去掉的字。标记本身留着:组装文档时句子靠它挂
|
||||
依据(unit.see),渲染时它本来就不显示。
|
||||
|
||||
- 括号里只有依据的(「合计 6,635.68([[see:Q1]])。」),连括号一起去掉,括号前的空白照依据前的空白处理
|
||||
- 括号里只有依据的(「合计 6,635.68([[see:Q1]])。」),连括号一起去掉;去掉之后整对括号当一个依据,外面照下面
|
||||
几条再收拾一遍(「华南居首,([[see:Q1]])。」的逗号)。括号两边的数字、字母会粘成一串的,括号换成一个空格
|
||||
(「4,919.5([[see:Q1]])2 月」不剩「4,919.52 月」,_glued)
|
||||
- 依据在括号开头(「([[see:Q3]],原报告…)」):它和括号之间、它后面的标点和空白;在括号末尾(「(同 SQL,
|
||||
[[see:Q2]])」)同理
|
||||
- 后面就到句末(句末标点、换行、全文结尾),依据前面、几个依据之间有停顿标点的(「华南居首,[[see:Q1]]。」):
|
||||
这些停顿标点(前面是冒号的留着:它引出的可能是下一行的表格、列表)
|
||||
- 后面就到句末(句末标点、全文结尾),依据前面、几个依据之间有停顿标点的(「华南居首,[[see:Q1]]。」):
|
||||
这些停顿标点(前面是冒号的留着:它引出的可能是下一行的表格、列表);没有停顿标点、到句末只隔着空白的,空白去掉
|
||||
(「华南居首 [[see:Q1]] 。」)
|
||||
- 以上几种里,几个依据之间的空白、分隔标点一并去掉(「([[see:Q1]]、[[see:Q2]])」)
|
||||
- 两边都是字的,依据之间的分隔标点可能就是分句的标点:按它断开,每段照 _run_cuts
|
||||
|
||||
不含依据的括号、标点一个字不动。要看的只有依据前后连着的空白、括号、分隔标点,外加再往外的一个字:流式渲染把
|
||||
这些字攒到后面的字来了才放(_TRAILING_SEE),分块渲染和整篇渲染判得一样。prev 是这段文字前面那个字。
|
||||
whole:text[start:end] 是一对只装着依据的括号(括号本身已经记了去掉),当一个依据看它外面。
|
||||
"""
|
||||
marks = list(MARKER_RE.finditer(text, start, end))
|
||||
marks = [] if whole else list(MARKER_RE.finditer(text, start, end))
|
||||
gaps = [(m.end(), n.start()) for m, n in zip(marks, marks[1:]) if m.end() < n.start()]
|
||||
lo, hi = start, end
|
||||
while lo > 0 and text[lo - 1] in _GAP:
|
||||
@@ -1904,39 +1920,70 @@ def _see_cuts(text: str, start: int, end: int, prev: str) -> list[tuple[int, int
|
||||
opener = text[lo - 1] if lo > 0 else prev
|
||||
closer = text[hi] if hi < len(text) else ""
|
||||
left, right = text[lo:start], text[end:hi]
|
||||
if opener in _OPEN and closer in _CLOSE:
|
||||
if lo > 0 and opener in _OPEN and closer in _CLOSE:
|
||||
# 括号里只有依据:连括号一起去掉
|
||||
return [(lo - 1, start), *gaps, (end, hi + 1), *_blank_cut(text, lo - 1, hi + 1, prev)]
|
||||
glue = " " if _glued(text, lo - 1, hi + 1, prev) else ""
|
||||
return [(lo - 1, start), *gaps, (end, hi + 1, glue), *_see_cuts(text, lo - 1, hi + 1, prev, whole=True)]
|
||||
if opener in _OPEN:
|
||||
return [(lo, start), *gaps, (end, hi)]
|
||||
if closer in _CLOSE:
|
||||
# 和括号之间只有空白的,照依据前的空白处理(行首的缩进不动)
|
||||
return [*gaps, (end, hi), *([(lo, start)] if _SEP & set(left) else _blank_cut(text, start, end, prev))]
|
||||
if (closer in _ENDING or closer in ("", "\n")) and _PAUSE & set(left + "".join(text[a:b] for a, b in gaps)):
|
||||
# 依据后面、句末前面的停顿标点是写作者自己的(「增长 [[see:Q1]],」),不是依据留下的,不动;只剩空白的去掉
|
||||
return [*gaps, *([] if _SEP & set(right) else [(end, hi)]),
|
||||
*([(lo, start)] if _PAUSE & set(left) else _blank_cut(text, start, end, prev))]
|
||||
return [cut for run in _SEE_RUN.finditer(text, start, end) for cut in _run_cuts(text, run.start(), run.end(), prev)]
|
||||
tail: list[tuple[int, ...]] = []
|
||||
if closer in _ENDING or closer == "":
|
||||
if _PAUSE & set(left + "".join(text[a:b] for a, b in gaps)):
|
||||
# 依据后面、句末前面的停顿标点是写作者自己的(「增长 [[see:Q1]],」),不是依据留下的,不动;只剩空白的去掉
|
||||
return [*gaps, *([] if _SEP & set(right) else [(end, hi)]),
|
||||
*([(lo, start)] if _PAUSE & set(left) else _blank_cut(text, start, end, prev))]
|
||||
if not _SEP & set(right):
|
||||
tail = [(end, hi)]
|
||||
runs = [(start, end)] if whole else [(run.start(), run.end()) for run in _SEE_RUN.finditer(text, start, end)]
|
||||
return [*tail, *(cut for a, b in runs for cut in _run_cuts(text, a, b, prev))]
|
||||
|
||||
|
||||
def _run_cuts(text: str, start: int, end: int, prev: str) -> list[tuple[int, int]]:
|
||||
"""两边都是字的一段依据(_SEE_RUN,中间只隔着空白):
|
||||
def _glued(text: str, start: int, end: int, prev: str) -> bool:
|
||||
"""去掉 text[start:end](一对只装着依据的括号)之后,两边的字会不会粘成一串:前面是数字、字母(或者标记的结尾、
|
||||
百分号、半角的逗号和小数点,_NUMBERISH),后面紧跟着数字、字母,或者一个渲染出字的标记。「[[v:Q1.r0.销售额]]
|
||||
([[see:Q1]])2 月」去掉括号读成「4,919.52 月」,是另一个数,而且哪项核对都查不出来。括号前有空白的不会粘:
|
||||
空白照 _blank_cut 留着。紧跟在括号后面的依据也不显示,越过它看下一个字(流式渲染把它和括号一起攒着)。"""
|
||||
before = text[start - 1] if start > 0 else prev
|
||||
if not (_wordish(before) or before in _NUMBERISH):
|
||||
return False
|
||||
while (marker := MARKER_RE.match(text, end)) is not None and marker.group("kind") == "see":
|
||||
end = marker.end()
|
||||
after = text[end:end + 1]
|
||||
if after == "[":
|
||||
return MARKER_RE.match(text, end) is not None
|
||||
return _wordish(after)
|
||||
|
||||
|
||||
def _wordish(ch: str) -> bool:
|
||||
"""数字(全角的也算)或者半角字母:两个挨在一起读成一个数、一个词。"""
|
||||
return ch.isdecimal() or (ch.isascii() and ch.isalnum())
|
||||
|
||||
|
||||
def _run_cuts(text: str, start: int, end: int, prev: str) -> list[tuple[int, ...]]:
|
||||
"""两边都是字的一段依据(_SEE_RUN,中间只隔着空白;或者一对只装着依据的括号):
|
||||
|
||||
- 几个依据之间的空白,和依据前面的空白(_blank_cut)
|
||||
- 夹在两个分隔标点中间(「结果一致,[[see:Q1]],原报告…」):去掉后面那个(连同它前面的空白,后面的空白留着
|
||||
隔开下文:「a, [[see:Q1]], b」→「a, b」)
|
||||
隔开下文:「a, [[see:Q1]], b」→「a, b」)。前面那个是半角标点、后面紧跟着字的,换成一个空格:「212,[[see:Q1]],5」
|
||||
不剩「212,5」
|
||||
"""
|
||||
marks = list(MARKER_RE.finditer(text, start, end))
|
||||
cuts = [(m.end(), n.start()) for m, n in zip(marks, marks[1:]) if m.end() < n.start()]
|
||||
cuts: list[tuple[int, ...]] = [(m.end(), n.start()) for m, n in zip(marks, marks[1:]) if m.end() < n.start()]
|
||||
cuts += _blank_cut(text, start, end, prev)
|
||||
lo, hi = start, end
|
||||
while lo > 0 and text[lo - 1] in _GAP:
|
||||
lo -= 1
|
||||
while hi < len(text) and text[hi] in _GAP:
|
||||
hi += 1
|
||||
right = text[end:hi]
|
||||
if _SEP & set(text[lo:start]) and _SEP & set(right):
|
||||
cuts.append((end, end + max(i for i, ch in enumerate(right) if ch in _SEP) + 1))
|
||||
left, right = text[lo:start], text[end:hi]
|
||||
if _SEP & set(left) and _SEP & set(right):
|
||||
cut = end + max(i for i, ch in enumerate(right) if ch in _SEP) + 1
|
||||
after = text[cut:cut + 1]
|
||||
spaced = left.rstrip(_BLANK)[-1] in _HALF and after not in ("", "\n") and after not in _BLANK
|
||||
cuts.append((end, cut, " " if spaced else ""))
|
||||
return cuts
|
||||
|
||||
|
||||
|
||||
@@ -703,9 +703,17 @@ class _Review:
|
||||
message=f"反方模型与报告的写作模型相同(「{outcome.model}」),难以发现写作模型自身的错误。"
|
||||
f"请在设置的「核查」一节选择另一个模型,或在节点的「{CHALLENGE_LABEL['model']}」中指定")
|
||||
claim, why = _claim_of(cand, outcome)
|
||||
# 触顶的计在哪一组:时长按剩余时间收紧过的另算一组(_SQUEEZED),说运行时长上限将至
|
||||
group = _SQUEEZED if outcome.limit == "timeout_s" and allot.tightened else outcome.limit
|
||||
if outcome.limit == "daily_max_usd" and outcome.result is None and not outcome.usage.get("calls"):
|
||||
# 派出去就撞上每日金额上限、一次模型都没调用:和后面不再派出的句子是同一回事,同一种说法
|
||||
claim = _claim(cand, "inconclusive", self._capped(), queries=outcome.queries, usage=outcome.usage)
|
||||
elif outcome.result is None and outcome.error == verify_agent.SETTLE_TIMEOUT:
|
||||
# 查完了、交结论那次超过时长上限:照上限说,和时间线上的 challenge_limit 一致。核查 agent 的原话(「收尾时
|
||||
# 超过时长上限,没能交结果」)是内部的说法,不摆上审批卡
|
||||
words = self._limit_words(_SQUEEZED if allot.tightened else "timeout_s")
|
||||
claim = _claim(cand, "inconclusive", f"反查未完成:已达上限({words})", queries=outcome.queries,
|
||||
usage=outcome.usage)
|
||||
elif lost and why is None:
|
||||
# 反查了,可该查的源有停用了的:判断照常采信,理由末尾写明哪几个没查(只写在任务里的话,审稿记录和
|
||||
# 审批卡上的「经受住」看着就像每个源都查过了)
|
||||
@@ -714,8 +722,7 @@ class _Review:
|
||||
claims.append(claim)
|
||||
usage = _add_usage(usage, outcome.usage)
|
||||
if outcome.limit in _CLAIM_LIMITS:
|
||||
squeezed = outcome.limit == "timeout_s" and allot.tightened
|
||||
counted = hit.setdefault(_SQUEEZED if squeezed else outcome.limit, [0, 0])
|
||||
counted = hit.setdefault(group, [0, 0])
|
||||
counted[0] += 1
|
||||
counted[1] += why is not None
|
||||
broke = outcome.limit == "daily_max_usd"
|
||||
|
||||
@@ -53,6 +53,7 @@ import asyncio
|
||||
import inspect
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
import time
|
||||
from collections.abc import Callable, Iterable
|
||||
from dataclasses import asdict, dataclass, field, replace
|
||||
@@ -100,6 +101,9 @@ _REF_DEPTH = 8
|
||||
RATIONALE = "rationale"
|
||||
#: 触顶原因
|
||||
LIMITS = ("max_queries", "timeout_s", "max_cost_usd", "daily_max_usd", "max_steps")
|
||||
#: 收尾那次调用超过时长上限、没交成结果时的原话(limit 记 timeout_s)。是内部的说法,给人看时调用方换成自己的话:质询的
|
||||
#: _error_words、反方审稿的「反查未完成:已达上限(…)」、双盲复算按 limit 写的无法复算原因
|
||||
SETTLE_TIMEOUT = "收尾时超过时长上限,没能交结果"
|
||||
#: 核查模型的输出到了上限被截断、没交成结果时给人看的话:双盲复算的无法复算、反方审稿的「反查未完成」、质询的报错都照
|
||||
#: 它说,也是 verify_truncated 日志的原话。cut 是 budget.cut_words 的「核查模型的输出达到上限(N token)被截断」。开头只说
|
||||
#: 「未能给出结果」:质询里提问人看到的就是这句,它问的是问题,不是在核查
|
||||
@@ -666,17 +670,23 @@ def _parsed(value: Any) -> dict[str, Any] | None:
|
||||
|
||||
|
||||
#: 只是占位词的理由(只留字母、数字和汉字,不分大小写):模型照着结构填了个字面的 placeholder、N/A,等于没给理由
|
||||
_PLACEHOLDERS = frozenset({"placeholder", "todo", "tbd", "na", "none", "null", "nil", "rationale", "reason",
|
||||
"无", "暂无", "略", "理由"})
|
||||
_PLACEHOLDERS = frozenset({"placeholder", "todo", "tbd", "na", "null", "nil", "rationale", "reason", "略", "理由"})
|
||||
#: 说「没有」的词:理由是另一个结果的说明时(双盲复算的数、反方审稿的结论),它等于没给理由;理由就是结果本身时(审批
|
||||
#: 质询,结构里只有理由一项)它可能就是回答——问「有没有测试账号」,答「无。」
|
||||
_NOTHING = frozenset({"none", "无", "暂无"})
|
||||
#: 依据写在引出语后面(「依据 [[see:Q1]]。」「结果见:[[see:Q1]]」「(来源:[[see:Q1]])」):依据渲染后不显示,引出语悬在
|
||||
#: 那里(「依据。」),渲染时去不掉(evidence._see_cuts 只收拾依据周围的空白、括号和标点)。交结果的提示要求写在句末,
|
||||
#: 模型不照做时记一条日志,验收时查得到
|
||||
_LEAD_IN_SEE = re.compile(r"(?:依据|见|取自|来源|出处)[ \t\u3000]*[::]?[ \t\u3000]*\[\[[ \t]*see[ \t]*:[^\[\]\n]*\]\]")
|
||||
|
||||
|
||||
def _no_rationale(value: Any) -> bool:
|
||||
"""交回的理由等于没给:不是文字、空的、只有标点,或者只是一个占位词。当没给理由处理(理由为空、没有理由文档),
|
||||
审批卡照「没有给出理由」说,不把占位词当理由展示。"""
|
||||
if not isinstance(value, str):
|
||||
return True
|
||||
word = "".join(ch for ch in value if ch.isalnum()).casefold()
|
||||
return not word or word in _PLACEHOLDERS
|
||||
def _no_rationale(shown: str, *, answer: bool = False) -> bool:
|
||||
"""渲染出来的理由等于没给:空的、只有标点(只写了依据:[[see:Q1]]、「([[see:Q1]])」渲染出来是空的),或者只是一个
|
||||
占位词(「placeholder。[[see:Q1]]」渲染出来只剩占位词)。按渲染后的正文判,不按原文:标记里的字(see、Q1)不显示。
|
||||
answer:理由就是交回的结果本身(审批质询的回答),说「没有」的词不算。当没给理由处理(理由为空、没有理由文档),
|
||||
审批卡照「没有给出理由」说,不把占位词当理由展示,判「被推翻」的也不当有出处。"""
|
||||
word = "".join(ch for ch in shown if ch.isalnum()).casefold()
|
||||
return not word or word in _PLACEHOLDERS or (not answer and word in _NOTHING)
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -1025,7 +1035,7 @@ async def _verify(state: _State, *, spec: ModelSpec, system: str, task: str, sou
|
||||
if settle_cut:
|
||||
return await report_cut("verify_result", settler.max_tokens)
|
||||
timed_out = isinstance(failure, asyncio.TimeoutError)
|
||||
why = "收尾时超过时长上限,没能交结果" if timed_out else \
|
||||
why = SETTLE_TIMEOUT if timed_out else \
|
||||
f"核查模型没有按结构交结果({explain_model_error(failure, model_id)})" if failure is not None else \
|
||||
"核查模型没有按结构交结果"
|
||||
# 收尾超时:没交成结果是因为时长上限,limit 照实记 timeout_s,和原因对得上——模型自己停了查询(stopped 为 None)
|
||||
@@ -1044,8 +1054,15 @@ async def _verify(state: _State, *, spec: ModelSpec, system: str, task: str, sou
|
||||
result = cells.walk(resolved, parsed) if _is_object(resolved) else {}
|
||||
rationale = parsed.get(RATIONALE)
|
||||
doc = None
|
||||
if not _no_rationale(rationale):
|
||||
if isinstance(rationale, str) and rationale.strip():
|
||||
doc = compose_doc(rationale, build_catalog(nodes={}, ledger=ledger), node_id=owner, run_id=run_id or None)
|
||||
if hits := _LEAD_IN_SEE.findall(rationale):
|
||||
logger.warning("核查 agent 的理由里依据写在了引出语后面,渲染后引出语会悬空(运行 %s,节点 %s):%s",
|
||||
run_id or "-", owner, "、".join(hits[:5]))
|
||||
# 按渲染出来的判:只写了依据的、占位词后面挂着依据的,原文里有字,显示出来没有理由。结构里只有理由一项(审批
|
||||
# 质询)时理由就是回答,「无」是一个回答
|
||||
if _no_rationale(doc["markdown"], answer=declared == [RATIONALE]):
|
||||
doc = None
|
||||
result[RATIONALE] = doc["markdown"] if doc else ""
|
||||
return outcome(result=result, cells=cells.cells, rationale=doc, limit=stopped, answered=answered)
|
||||
|
||||
@@ -1126,4 +1143,5 @@ async def _run_tool(tool: Any, name: str, args: dict[str, Any], call_id: str, re
|
||||
return content
|
||||
|
||||
|
||||
__all__ = ["LIMITS", "RATIONALE", "SETTLE_GRACE_S", "Default", "VerifyOutcome", "result_schema", "run_verify_agent"]
|
||||
__all__ = ["LIMITS", "RATIONALE", "SETTLE_GRACE_S", "SETTLE_TIMEOUT", "Default", "VerifyOutcome", "result_schema",
|
||||
"run_verify_agent"]
|
||||
|
||||
@@ -62,3 +62,24 @@ async def test_a_placeholder_rationale_is_recorded_as_no_rationale(monkeypatch,
|
||||
assert second["rationale"] == "理由没有引用到任何查询结果,反方给出的「被推翻」不予采信,记为无法判断。反方没有给出理由"
|
||||
assert "placeholder" not in str(record)
|
||||
assert issuance["tier"] == "formal"
|
||||
|
||||
|
||||
async def test_a_rationale_that_shows_nothing_is_recorded_as_no_rationale(monkeypatch, shop):
|
||||
"""反方的理由只写了依据:显示出来是空的。以前按原文判,理由文档照样记下——审批卡按文档画出一片空白,不说「反方没有
|
||||
给出理由」;判「被推翻」的因为引用到了 Q1 被当成有出处,照常采信、出具降档。"""
|
||||
plays = {"华南销售额超过华东": Play([NET], "refuted", "[[see:Q1]]"),
|
||||
"居各区域之首": Play([REFUNDS], "stands", "([[see:Q1]])")}
|
||||
record, issuance = await _record(monkeypatch, shop, plays)
|
||||
first, second = record["claims"]
|
||||
assert (first["verdict"], first["rationale"], first["rationale_doc"]) == ("stands", "", None)
|
||||
assert second["verdict"] == "inconclusive" and second["rationale_doc"] is None
|
||||
assert second["rationale"].endswith("反方给出的「被推翻」不予采信,记为无法判断。反方没有给出理由")
|
||||
assert issuance["tier"] == "formal", "没有理由的「被推翻」不能让出具降档"
|
||||
|
||||
|
||||
async def test_a_placeholder_with_a_see_is_still_a_placeholder(monkeypatch, shop):
|
||||
plays = {"华南销售额超过华东": Play([NET], "stands", "placeholder。[[see:Q1]]"),
|
||||
"居各区域之首": Play([REFUNDS], "stands", "无。")}
|
||||
record, _ = await _record(monkeypatch, shop, plays)
|
||||
assert [(c["rationale"], c["rationale_doc"]) for c in record["claims"]] == [("", None), ("", None)]
|
||||
assert "placeholder" not in str(record)
|
||||
|
||||
@@ -40,3 +40,12 @@ async def test_a_placeholder_rationale_is_not_shown_as_the_explanation(monkeypat
|
||||
mine = await _blind_input(monkeypatch, shop, "placeholder")
|
||||
assert mine["value"] == 330.0, "数照样采用"
|
||||
assert (mine["rationale"], mine["rationale_doc"]) == ("", None)
|
||||
|
||||
|
||||
async def test_a_rationale_that_shows_nothing_is_not_kept(monkeypatch, shop):
|
||||
"""只写了依据(显示出来是空的),或者占位词后面挂着依据:说明为空、不留文档。以前文档照样记下,审批卡按文档画出
|
||||
「理由」两个字,后面什么都没有。"""
|
||||
for said in ("([[see:Q1]])", "placeholder。[[see:Q1]]"):
|
||||
mine = await _blind_input(monkeypatch, shop, said)
|
||||
assert mine["value"] == 330.0, "数照样采用"
|
||||
assert (mine["rationale"], mine["rationale_doc"]) == ("", None), said
|
||||
|
||||
@@ -4,20 +4,27 @@
|
||||
[[see:Q1]]」,模型就把它写进括号、句中:「结果一致([[see:Q3]],原报告 SQL 未按 status 过滤)」「合计
|
||||
[[v:Q1.r1.金额合计]]([[see:Q1]])」,渲染后剩下「(,原报告…)」「()」,正式运行里随封存永久留着。
|
||||
|
||||
- 括号(全角、半角,一对里混用也算)里只有依据的,连括号一起去掉,括号前的空白照 see 前的空白处理
|
||||
- 括号(全角、半角,一对里混用也算)里只有依据的,连括号一起去掉;去掉之后整对括号当一个依据,外面照下面几条再收拾
|
||||
一遍(括号前悬着的停顿标点、括号前后的空白)
|
||||
- 去掉括号之后两边的数字、字母会粘成一串的(「4,919.5([[see:Q1]])2 月」),括号换成一个空格:粘起来读成另一个数,
|
||||
哪项核对都查不出来;半角逗号夹着依据、去掉后一个之后同理(「212,[[see:Q1]],5」→「212, 5」)
|
||||
- 依据在括号开头、后面跟着「,」「、」「;」「:」的,标点一起去掉;在括号末尾、前面紧挨着标点的同理
|
||||
- 依据夹在两个分隔标点中间的,去掉后一个;后面就到句末(句号、换行、全文结尾)、前面或几个依据之间有「,」「、」
|
||||
「;」的,去掉这些标点;依据后面、句末前面的标点是写作者自己的,不动
|
||||
- 依据夹在两个分隔标点中间的,去掉后一个;后面就到句末(句末标点、全文结尾)、前面或几个依据之间有「,」「、」
|
||||
「;」的,去掉这些标点,到句末只隔着空白的去掉空白;依据后面、句末前面的标点是写作者自己的,不动
|
||||
- 换行不算句末:列表项末尾的「;」、软换行处的「,」是写作者自己的,依据挂在它后面正合「挂在句末」的规矩,照旧留着
|
||||
- 括号里、句末前几个依据之间的分隔标点一并去掉;两边都是字时,依据之间的标点可能是分句的标点,照旧留着
|
||||
- 依据标记本身留在原文里:句子照旧挂着这条依据(see、cites),只是不显示
|
||||
- 不含依据的括号、没有依据的文档一个字不动:内容哈希和改动之前一样
|
||||
- 流式渲染分块输出拼起来仍然等于整篇渲染
|
||||
- 已知限制:依据写在「依据」「见」「取自」「来源」这类引出语后面时,引出语悬空(「依据。」「(来源)」)。渲染时去不掉,
|
||||
靠核查 agent 的提示词避免,模型不照做时核查 agent 记一条日志(test_verify_rationale_rules)
|
||||
|
||||
数字、表名、地名都是虚构的。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import random
|
||||
import re
|
||||
|
||||
import pytest
|
||||
|
||||
@@ -116,32 +123,107 @@ def test_punctuation_next_to_see_inside_a_bracket_goes_with_it(catalog, raw, sho
|
||||
("口径相同(按支付日,[[see:Q1]],不含退款)。", "口径相同(按支付日,不含退款)。"),
|
||||
("结果一致,[[see:Q1]],原报告未过滤。", "结果一致,原报告未过滤。"),
|
||||
("a, [[see:Q1]], b", "a, b"),
|
||||
("a,[[see:Q1]],b", "a, b"), # 半角逗号后面照排版习惯隔一个空格
|
||||
# 前面是分隔标点、后面就到句末:去掉前面那个
|
||||
("华南居首,[[see:Q1]]。", "华南居首。"),
|
||||
("华南居首、华东第二;[[see:Q1]]\n下一行", "华南居首、华东第二\n下一行"),
|
||||
("华南居首,[[see:Q1]]", "华南居首"),
|
||||
("华南居首,[[see:Q1]] 。", "华南居首。"),
|
||||
# 几个依据之间的分隔标点
|
||||
("华南居首 [[see:Q1]],[[see:m:gmv]]。", "华南居首。"),
|
||||
# 后面就到句末、中间只隔着空白:空白一起去掉,不剩「华南居首 。」
|
||||
("华南居首 [[see:Q1]] 。", "华南居首。"),
|
||||
("华南居首 [[see:Q1]] [[see:m:gmv]] 。", "华南居首。"),
|
||||
])
|
||||
def test_punctuation_left_dangling_by_see_is_dropped(catalog, raw, shown):
|
||||
assert render(raw, catalog) == shown
|
||||
assert compose(raw, catalog)["markdown"] == shown
|
||||
|
||||
|
||||
@pytest.mark.parametrize(("raw", "shown"), [
|
||||
# 括号前悬着的停顿标点:去掉括号之后和没写括号的「华南居首,[[see:Q1]]。」一样收拾
|
||||
("华南居首,([[see:Q1]])。", "华南居首。"),
|
||||
("华南居首、([[see:Q1]])", "华南居首"),
|
||||
("华南居首 ([[see:Q1]]) 。", "华南居首。"),
|
||||
("华南居首,([[see:Q1]]),华东第二。", "华南居首,华东第二。"), # 夹在两个逗号中间:去掉后一个
|
||||
("华南居首(([[see:Q1]]))。", "华南居首。"), # 套着的两层只装着依据,一起去掉
|
||||
# 换行不算句末:括号里外一样留着写作者的标点
|
||||
("华南居首、([[see:Q1]])\n下一行", "华南居首、\n下一行"),
|
||||
])
|
||||
def test_a_dropped_bracket_is_tidied_like_a_bare_see(catalog, raw, shown):
|
||||
assert render(raw, catalog) == shown
|
||||
assert compose(raw, catalog)["markdown"] == shown
|
||||
|
||||
|
||||
@pytest.mark.parametrize(("raw", "shown"), [
|
||||
# 括号是两边唯一的分隔:去掉之后数字会粘成另一个数,换成一个空格
|
||||
("华东 [[v:Q1.r0.销售额]]([[see:Q1]])2 月起持续下滑。", "华东 4,919.5 2 月起持续下滑。"),
|
||||
("合计 [[v:Q1.r0.销售额]]([[see:Q1]])2026 年同比持平。", "合计 4,919.5 2026 年同比持平。"),
|
||||
("订单 [[v:Q1.r0.订单数]]([[see:Q1]])2 个区域。", "订单 212 2 个区域。"),
|
||||
("增幅 [[m:yoy]]([[see:m:yoy]])3 个区域。", "增幅 12.5% 3 个区域。"),
|
||||
("合计 [[v:Q1.r0.销售额]]([[see:Q1]])[[v:Q1.r0.订单数]] 笔。", "合计 4,919.5 212 笔。"), # 后面的标记渲染出数字
|
||||
("合计 [[v:Q1.r0.销售额]]([[see:Q1]])[[see:m:gmv]]2 月。", "合计 4,919.5 2 月。"), # 越过不显示的依据看
|
||||
("编号 Q1([[see:Q1]])x", "编号 Q1 x"),
|
||||
("212,([[see:Q1]])5 个区域", "212, 5 个区域"),
|
||||
# 半角逗号夹着依据:去掉后一个之后,前一个不能直接贴着数字
|
||||
("212,[[see:Q1]],5 个区域", "212, 5 个区域"),
|
||||
# 不会粘的照旧去掉:中文和数字之间、括号前已有空白、后面是中文、隔着全角逗号
|
||||
("居首([[see:Q1]])2 月起", "居首2 月起"),
|
||||
("合计 [[v:Q1.r0.销售额]] ([[see:Q1]])2 月起", "合计 4,919.5 2 月起"),
|
||||
("合计 [[v:Q1.r0.销售额]]([[see:Q1]])元", "合计 4,919.5元"),
|
||||
("华南([[see:Q1]])居首", "华南居首"),
|
||||
("212,[[see:Q1]],5 个区域", "212,5 个区域"),
|
||||
])
|
||||
def test_dropping_a_bracket_does_not_glue_numbers_together(catalog, raw, shown):
|
||||
assert render(raw, catalog) == shown
|
||||
assert compose(raw, catalog)["markdown"] == shown
|
||||
|
||||
|
||||
def test_a_glued_number_would_have_slipped_through_the_checks(catalog):
|
||||
"""粘起来的数没有哪项核对查得出:片段本身对(4,919.5 后面是另一段文字),月份不算裸数字。所以只能在渲染时不粘。"""
|
||||
doc = compose("华东 [[v:Q1.r0.销售额]]([[see:Q1]])2 月起持续下滑。", catalog)
|
||||
assert doc["markdown"] == "华东 4,919.5 2 月起持续下滑。" and doc["violations"] == []
|
||||
assert verify_doc(doc, catalog, loader=LOADER)["violations"] == []
|
||||
[unit] = [unit for _, unit in iter_units(doc)]
|
||||
assert [s["alias"] for s in unit["see"]] == ["Q1"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(("raw", "shown"), [
|
||||
("华南居首。[[see:Q1]]", "华南居首。"),
|
||||
("华南居首 [[see:Q1]],华东第二。", "华南居首,华东第二。"),
|
||||
# 两个依据中间的逗号后面接着字:它是分句的逗号,不是依据之间的分隔,照旧留着
|
||||
("华南居首 [[see:Q1]],[[see:m:gmv]]华东第二。", "华南居首,华东第二。"),
|
||||
("华南居首 [[see:Q1]] [[see:m:gmv]] 。", "华南居首 。"), # 没有悬空的标点:照旧只去掉依据前的空白
|
||||
("华南居首 [[see:Q1]],", "华南居首,"), # 依据后面的逗号是写作者自己的,不动
|
||||
("数据如下:[[see:Q1]]\n| 区域 | 销售额 |", "数据如下:\n| 区域 | 销售额 |"), # 冒号引出的是下一行
|
||||
("结果见:[[see:Q1]]。", "结果见:。"), # 冒号前的引出语去不掉:靠提示词让模型写在句末
|
||||
("依据 [[see:Q1]]。", "依据。"),
|
||||
# 换行不算句末:列表项末尾的「;」、软换行处连着两个分句的「,」是写作者自己的
|
||||
("1. 华东居首;[[see:Q1]]\n2. 华南第二;[[see:Q1]]\n3. 华北第三。[[see:Q1]]",
|
||||
"1. 华东居首;\n2. 华南第二;\n3. 华北第三。"),
|
||||
("华东居首,[[see:Q1]]\n华南第二。", "华东居首,\n华南第二。"),
|
||||
("华南居首、华东第二;[[see:Q1]]\n下一行", "华南居首、华东第二;\n下一行"),
|
||||
("- 华东居首;([[see:Q1]])\n- 华南第二。", "- 华东居首;\n- 华南第二。"),
|
||||
])
|
||||
def test_see_outside_brackets_without_dangling_pauses_renders_as_before(catalog, raw, shown):
|
||||
def test_punctuation_the_writer_put_there_is_kept(catalog, raw, shown):
|
||||
assert render(raw, catalog) == shown
|
||||
assert compose(raw, catalog)["markdown"] == shown
|
||||
|
||||
|
||||
#: 引出语悬空:「依据」「见」「取自」「来源」「出处」后面(可能隔着冒号、空白)紧跟着句末、停顿标点或括号
|
||||
_DANGLING = re.compile(r"(?:依据|见|取自|来源|出处)[::]?[ \t]*(?:[。,、;))!?,;]|$)")
|
||||
|
||||
|
||||
@pytest.mark.xfail(strict=True, reason="已知限制:依据写在引出语后面时,渲染后引出语悬空。引出语在依据前面,渲染时要去掉"
|
||||
"它,流式渲染得先攒住每个可能是引出语的字;目前靠核查 agent 的提示词避免这种写法,模型"
|
||||
"不照做时记一条日志(verify_agent._LEAD_IN_SEE)")
|
||||
@pytest.mark.parametrize("raw", [
|
||||
"结果见:[[see:Q1]]。",
|
||||
"依据 [[see:Q1]]。",
|
||||
"数据取自 [[see:Q1]]。",
|
||||
"结果取自 [[see:Q1]],完整未截断。",
|
||||
"按支付日统计(见 [[see:Q1]])。",
|
||||
"按支付日统计(来源:[[see:Q1]])。",
|
||||
"按状态拆开(status 取值见 [[see:Q1]])。",
|
||||
])
|
||||
def test_see_after_a_lead_in_word_leaves_it_dangling(catalog, raw):
|
||||
assert not _DANGLING.search(render(raw, catalog)), render(raw, catalog)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("raw", [
|
||||
@@ -214,6 +296,9 @@ def _stream(catalog, raw: str, sizes: list[int]) -> list[str]:
|
||||
"结果一致([[see:Q1]],原报告未按状态过滤),合计 [[v:Q1.r0.销售额]]([[see:Q1]])。下一句",
|
||||
"前一句 ([[see:Q1]])后一句,复算原查询(同 SQL,[[see:Q1]]):华南居首,[[see:Q1]]。",
|
||||
"口径相同(按支付日,[[see:Q1]],不含退款)。a, [[see:Q1]], b\n华南、[[see:Q1]]\n末尾([[see:Q1]])",
|
||||
"华东 [[v:Q1.r0.销售额]]([[see:Q1]])2 月,合计 [[v:Q1.r0.销售额]]([[see:Q1]])[[see:m:gmv]][[v:Q1.r0.订单数]] 笔"
|
||||
",212,[[see:Q1]],5 个",
|
||||
"华南居首,([[see:Q1]])。华南居首 [[see:Q1]] 。1. 华东居首;[[see:Q1]]\n2. 华南第二,([[see:Q1]]),华东第二",
|
||||
])
|
||||
@pytest.mark.parametrize("size", range(1, 9))
|
||||
def test_stream_tidies_see_in_brackets_like_the_full_render(catalog, raw, size):
|
||||
@@ -225,7 +310,7 @@ def test_stream_tidies_see_in_brackets_like_the_full_render(catalog, raw, size):
|
||||
def _corpus(n: int) -> list[str]:
|
||||
rng = random.Random(20261006)
|
||||
pieces = ["[[see:Q1]]", "[[see:m:gmv]]", "[[v:Q1.r0.销售额]]", "[[m:gmv]]", "(", ")", "(", ")", ",", "、",
|
||||
";", ":", ",", ";", " ", " ", "。", "\n", "文字", "元", "[[", "]]", "["]
|
||||
";", ":", ",", ";", " ", " ", "。", "\n", "文字", "元", "[[", "]]", "[", "2", "x", "%", "!"]
|
||||
return ["".join(rng.choice(pieces) for _ in range(rng.randint(1, 18))) for _ in range(n)]
|
||||
|
||||
|
||||
|
||||
@@ -4,6 +4,9 @@
|
||||
([[see:Q1]])」渲染成「(,原报告…)」「()」,随正式运行封存;另一次回答查完了,收尾那次调用超过时长上限,报错写着
|
||||
「回答超过时长上限(180 秒)」,answer.limit 却是 null,界面上缺了「已达时长上限」。
|
||||
|
||||
回答只写了依据(「([[see:Q1]])」)时显示出来是空的:以前照样交回,封存之后的追问末尾补上「注:以上回答依据的是数据源
|
||||
当前的数据…」,回答正文只剩这句系统说明、却记成回答成功。现在按没给回答记。回答就是一个「无。」的,照常是回答。
|
||||
|
||||
模型一律是 MockChatModel 剧本,区域、金额都是虚构的。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
@@ -12,6 +15,7 @@ from app.core import artifact_store
|
||||
from app.engine import verify_agent
|
||||
from app.engine.inquiry import NO_ANSWER
|
||||
from tests.test_inquiry_api import ( # noqa: F401 - 夹具按名字引入
|
||||
EAST,
|
||||
NO_TEST,
|
||||
QUESTION,
|
||||
Models,
|
||||
@@ -20,10 +24,13 @@ from tests.test_inquiry_api import ( # noqa: F401 - 夹具按名字引入
|
||||
client,
|
||||
engine_up,
|
||||
events,
|
||||
finish,
|
||||
pause,
|
||||
pending,
|
||||
report_doc,
|
||||
set_verify,
|
||||
shop,
|
||||
unit_of,
|
||||
)
|
||||
from tests.test_verify_rationale_rules import slow_final
|
||||
|
||||
@@ -74,3 +81,25 @@ async def test_a_placeholder_answer_is_no_answer(monkeypatch, shop, client):
|
||||
answer = item["answer"]
|
||||
assert (answer["ok"], answer["text"], answer["limit"]) == (False, "", None)
|
||||
assert answer["error"] == NO_ANSWER
|
||||
|
||||
|
||||
async def test_an_answer_that_shows_nothing_is_no_answer_even_after_the_seal(monkeypatch, shop, client):
|
||||
"""封存之后的追问:回答只写了依据,显示出来是空的。以前回答文档照样组装,末尾补上「查的是现在的数据」那句说明,
|
||||
回答正文只剩系统说明、却记成回答成功。"""
|
||||
Models(monkeypatch, shop, plays={"排除测试账号": Play([NO_TEST], "([[see:Q1]])")})
|
||||
done = await finish(shop)
|
||||
assert done.status == "succeeded", done.error
|
||||
r = await ask(client, done.id, report="write", unit=unit_of(await report_doc(done.id), EAST), question=QUESTION)
|
||||
assert r.status_code == 200, r.text
|
||||
answer = r.json()["inquiry"]["answer"]
|
||||
assert (answer["ok"], answer["text"], answer["doc_artifact"]) == (False, "", None)
|
||||
assert answer["error"] == NO_ANSWER
|
||||
|
||||
|
||||
async def test_a_one_word_answer_is_an_answer(monkeypatch, shop, client):
|
||||
"""问的是「有没有」,答「无。」:这是回答,不是占位词,不能报「模型没有给出回答内容」让提问人重问、再花一次钱。"""
|
||||
Models(monkeypatch, shop, plays={"排除测试账号": Play([NO_TEST], "无。[[see:Q1]]")})
|
||||
item, [event] = await _asked(client, shop)
|
||||
answer = item["answer"]
|
||||
assert (answer["ok"], answer["text"], answer["error"]) == (True, "无。", None)
|
||||
assert event.data["answer_text"] == "无。"
|
||||
|
||||
@@ -6,13 +6,17 @@
|
||||
- 写进了括号的依据,理由文档里照样渲染干净(evidence._tidy),这句仍挂着那条依据
|
||||
- 模型自己停了工具调用、收尾那次调用超过了时长上限:limit 照实记 timeout_s。以前是 None——质询的报错写着「回答超过
|
||||
时长上限(180 秒)」,limit 却是空的,界面上缺了「已达时长上限」。不走结构化输出的模型(普通调用收尾)也一样
|
||||
- 交回的理由是空的、只有标点,或者只是一个占位词(字面的 placeholder、N/A):当作没给理由,不把占位词当理由展示
|
||||
- 交回的理由是空的、只有标点,或者只是一个占位词(字面的 placeholder、N/A):当作没给理由,不把占位词当理由展示。
|
||||
按渲染出来的正文判:只写了依据的([[see:Q1]]、「([[see:Q1]])」)显示出来是空的,占位词后面挂着依据的显示出来只剩
|
||||
占位词,都算没给。理由就是结果本身时(审批质询,结构里只有理由一项),「无」「none」是回答,不算占位
|
||||
- 依据写在「依据」「见」「取自」这类引出语后面(渲染时去不掉,引出语悬空):记一条日志,验收时查得到
|
||||
|
||||
模型一律是 MockChatModel 剧本,数据源是现造的 SQLite 文件,数字都是虚构的。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import re
|
||||
import time
|
||||
|
||||
@@ -20,7 +24,7 @@ import pytest
|
||||
|
||||
from app.engine import verify_agent
|
||||
from app.engine.evidence import iter_units
|
||||
from app.engine.inquiry import INQUIRY_SYSTEM, INQUIRY_TASK_PROMPT
|
||||
from app.engine.inquiry import INQUIRY_SCHEMA, INQUIRY_SYSTEM, INQUIRY_TASK_PROMPT
|
||||
from app.engine.nodes.challenge import CHALLENGE_SYSTEM, CHALLENGE_TASK_PROMPT
|
||||
from app.engine.nodes.dual_check import BLIND_SYSTEM, BLIND_TASK_PROMPT
|
||||
from app.engine.verify_agent import VERIFY_FINAL_PROMPT, VERIFY_SYSTEM
|
||||
@@ -130,7 +134,8 @@ async def test_a_final_call_that_runs_out_of_time_after_the_query_limit_records_
|
||||
# 占位词当理由
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
@pytest.mark.parametrize("said", ["placeholder", "PLACEHOLDER", " ", "N/A", "(略)", "…", "<rationale>", "TODO"])
|
||||
@pytest.mark.parametrize("said", ["placeholder", "PLACEHOLDER", " ", "N/A", "(略)", "…", "<rationale>", "TODO", "无",
|
||||
"暂无。", "None"])
|
||||
async def test_a_placeholder_rationale_counts_as_none(monkeypatch, shop, said):
|
||||
name, _ = shop
|
||||
Script(monkeypatch, [q(name, VALID_SUM, "c1")], {**FOUND, "rationale": said})
|
||||
@@ -139,9 +144,67 @@ async def test_a_placeholder_rationale_counts_as_none(monkeypatch, shop, said):
|
||||
assert out.result["rationale"] == "" and out.rationale is None
|
||||
|
||||
|
||||
@pytest.mark.parametrize("said", ["无异常。", "见 [[v:Q1.r0.total]]。", "略高于上周。"])
|
||||
@pytest.mark.parametrize("said", ["[[see:Q1]]", "([[see:Q1]])", " [[see:Q1]]。", "placeholder。[[see:Q1]]",
|
||||
"N/A([[see:Q1]])", "TODO [[see:Q1]]"])
|
||||
async def test_a_rationale_that_shows_nothing_counts_as_none(monkeypatch, shop, said):
|
||||
"""只写了依据、或者占位词后面挂着依据:原文里看着有字,显示出来没有理由。以前按原文判,标记里的字(see、Q1)也算,
|
||||
理由文档照样交回——审批卡按文档画出一片空白,判「被推翻」的还因为引用到了 Q1 被当成有出处。"""
|
||||
name, _ = shop
|
||||
Script(monkeypatch, [q(name, VALID_SUM, "c1")], {**FOUND, "rationale": said})
|
||||
out = await run(name)
|
||||
assert out.result["value"] == 230, "数照样取"
|
||||
assert out.result["rationale"] == "" and out.rationale is None
|
||||
|
||||
|
||||
@pytest.mark.parametrize("said", ["无异常。", "见 [[v:Q1.r0.total]]。", "略高于上周。", "[[v:Q1.r0.total]]",
|
||||
"一致。[[see:Q1]]"])
|
||||
async def test_a_short_real_rationale_is_kept(monkeypatch, shop, said):
|
||||
name, _ = shop
|
||||
Script(monkeypatch, [q(name, VALID_SUM, "c1")], {**FOUND, "rationale": said})
|
||||
out = await run(name)
|
||||
assert out.result["rationale"] and out.rationale is not None
|
||||
|
||||
|
||||
@pytest.mark.parametrize(("said", "shown"), [("无。", "无。"), ("None.", "None."), ("暂无。[[see:Q1]]", "暂无。")])
|
||||
async def test_nothing_is_an_answer_when_the_rationale_is_the_answer(monkeypatch, shop, said, shown):
|
||||
"""结构里只有理由一项(审批质询):理由就是回答。问「有没有测试账号」,答「无。」是一个回答,不能当成没给、让提问人
|
||||
重新提问再花一次钱。"""
|
||||
name, _ = shop
|
||||
Script(monkeypatch, [q(name, VALID_SUM, "c1")], {"rationale": said})
|
||||
out = await run(name, output_schema=INQUIRY_SCHEMA)
|
||||
assert out.result["rationale"] == shown and out.rationale["markdown"] == shown
|
||||
|
||||
|
||||
@pytest.mark.parametrize("said", ["placeholder", "null", "[[see:Q1]]", "N/A。[[see:Q1]]"])
|
||||
async def test_a_placeholder_answer_still_counts_as_none(monkeypatch, shop, said):
|
||||
name, _ = shop
|
||||
Script(monkeypatch, [q(name, VALID_SUM, "c1")], {"rationale": said})
|
||||
out = await run(name, output_schema=INQUIRY_SCHEMA)
|
||||
assert out.result["rationale"] == "" and out.rationale is None
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 依据写在引出语后面
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
async def test_see_after_a_lead_in_word_is_logged(monkeypatch, shop, caplog):
|
||||
"""「依据 [[see:Q1]]。」「(来源:[[see:Q1]])」渲染后引出语悬空,渲染时去不掉(见 test_evidence_see_brackets 的已知
|
||||
限制):记一条日志,写明是哪几处,验收时查得到。只记标记和引出语,不记数。"""
|
||||
name, _ = shop
|
||||
said = "合计 [[v:Q1.r0.total]],依据 [[see:Q1]]。口径一致(来源:[[see:Q1]])。"
|
||||
Script(monkeypatch, [q(name, VALID_SUM, "c1")], {**FOUND, "rationale": said})
|
||||
with caplog.at_level(logging.WARNING, logger="app.engine.verify_agent"):
|
||||
out = await run(name, node_id="chal", run_id="run-1")
|
||||
assert out.rationale is not None, "照常交回"
|
||||
[record] = [r for r in caplog.records if "引出语" in r.getMessage()]
|
||||
message = record.getMessage()
|
||||
assert "依据 [[see:Q1]]" in message and "来源:[[see:Q1]]" in message and "run-1" in message
|
||||
assert "230" not in message
|
||||
|
||||
|
||||
async def test_see_at_the_end_of_the_sentence_is_not_logged(monkeypatch, shop, caplog):
|
||||
name, _ = shop
|
||||
Script(monkeypatch, [q(name, VALID_SUM, "c1")], {**FOUND, "rationale": "合计 [[v:Q1.r0.total]],结果一致。[[see:Q1]]"})
|
||||
with caplog.at_level(logging.WARNING, logger="app.engine.verify_agent"):
|
||||
await run(name)
|
||||
assert not [r for r in caplog.records if "引出语" in r.getMessage()]
|
||||
|
||||
@@ -0,0 +1,139 @@
|
||||
"""核查 agent 收尾超时之后,反方审稿、双盲复算对外怎么说(期 5 真实模型验收 M1 的复核)。
|
||||
|
||||
核查 agent 查完了、自己停下,交结果那次调用超过了时长上限:verify_agent 照实记 limit=timeout_s(以前模型自己停下时是
|
||||
None,test_verify_rationale_rules 钉住了这一层)。两个节点据此按时长上限说,不再把核查 agent 的原话(「收尾时超过时长
|
||||
上限,没能交结果」)摆上审批卡、时间线:
|
||||
|
||||
- 反方审稿:这句计入 challenge_limit(limit=timeout_s,claims_skipped 算上它),不再记成 challenge_unreviewed;审稿记录
|
||||
里这句的理由是「反查未完成:已达上限(每句时长上限 N 秒)」。时长按剩余时间收紧过的那组带 cause=run_deadline,理由
|
||||
说运行时长上限将至
|
||||
- 双盲复算:无法复算的原因是「盲算已达每项时长上限(N 秒)」,发 dual_check_limit;收紧过的说运行时长上限将至,不发
|
||||
dual_check_limit(它叫人调高每项的上限,调高了只会更快撞线),时间线另起一条带 cause 的 dual_check_unchecked
|
||||
|
||||
交结果那次用 slow_final 拖住,收尾宽限调到 0.3 秒;收紧过的那组把 allot() 看到的钟拨到「给下游留出余量、扣掉收尾宽限
|
||||
之后只剩 0.8 秒」(真的钟照走,核查 agent 按真的 0.8 秒到点)。模型一律是 MockChatModel 剧本,数据源是现造的 SQLite
|
||||
文件,区域、金额都是虚构的。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from app.core import artifact_store
|
||||
from app.engine import deadline, verify_agent
|
||||
from tests.fixtures.dual_check_shop import RIGHT, Blind
|
||||
from tests.test_challenge_node import ( # noqa: F401 - 夹具按名字引入
|
||||
Models,
|
||||
engine_up,
|
||||
finish,
|
||||
graph,
|
||||
logs,
|
||||
output_of,
|
||||
shop,
|
||||
)
|
||||
from tests.test_dual_check_node import clean_settings, dual_output, events, start # noqa: F401
|
||||
from tests.test_dual_check_node import graph as dual_graph
|
||||
from tests.test_dual_check_node import logs as dual_logs
|
||||
from tests.test_dual_check_node import shop as dual_shop # noqa: F401
|
||||
from tests.test_verify_rationale_rules import slow_final
|
||||
|
||||
#: 收紧过的那组:allot() 算出的查库时长
|
||||
ROOM = 0.8
|
||||
#: 收紧过的时长到点时,反方审稿每句的理由、双盲复算的无法复算原因(默认 600 秒的一段)
|
||||
SQUEEZED_CLAIM = "反查未完成:已达上限(本次运行即将达到 600 秒的时长上限,反查时长已按剩余时间收紧)"
|
||||
SQUEEZED_BLIND = "本次运行即将达到 600 秒的时长上限,盲算时长已按剩余时间收紧,到时未能完成"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def settles_slowly(monkeypatch):
|
||||
"""交结果那次调用拖 5 秒,收尾宽限只给 0.3 秒:查完了的核查一律在收尾时超时。"""
|
||||
monkeypatch.setattr(verify_agent, "SETTLE_GRACE_S", 0.3)
|
||||
slow_final(monkeypatch)
|
||||
|
||||
|
||||
def squeeze(monkeypatch) -> None:
|
||||
"""allot() 看到的「现在」:这一段的截止时刻扣掉给下游的余量(下游只有成果节点,一份)、收尾宽限,再往前 ROOM 秒。
|
||||
每开始一项都一样,所以每一项的时长都收紧到 ROOM。"""
|
||||
def now() -> float:
|
||||
found = deadline.current()
|
||||
return found.deadline - deadline.reserve(found.seconds) - verify_agent.SETTLE_GRACE_S - ROOM - 0.05
|
||||
|
||||
monkeypatch.setattr(deadline, "_now", now)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 反方审稿
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
async def test_a_claim_whose_verdict_runs_out_of_time_counts_as_hitting_the_limit(monkeypatch, shop, settles_slowly):
|
||||
models = Models(monkeypatch, shop)
|
||||
row = await finish(graph(shop, timeout_s=0.5), run_class="exploratory")
|
||||
assert row.status == "succeeded", row.error
|
||||
assert [kind for kind, _, _ in models.calls].count("final") == 0, "交结论那次都没等到回复"
|
||||
|
||||
[limit] = await logs(row.id, "challenge_limit")
|
||||
assert (limit["limit"], limit["claims_skipped"]) == ("timeout_s", 2)
|
||||
assert "cause" not in limit and "每句时长上限 0.5 秒" in limit["message"] and "2 句未能给出结论" in limit["message"]
|
||||
assert await logs(row.id, "challenge_unreviewed") == [], "触了上限的由 challenge_limit 说,不另记成未能反查"
|
||||
|
||||
out = await output_of(row.id, "chal")
|
||||
assert out["counts"]["inconclusive"] == 2 and out["unreviewed"] == 2
|
||||
record = artifact_store.load(out["challenge_artifact"])
|
||||
for claim in record["claims"]:
|
||||
assert claim["verdict"] == "inconclusive" and claim["rationale_doc"] is None
|
||||
assert claim["rationale"] == "反查未完成:已达上限(每句时长上限 0.5 秒)"
|
||||
assert len(claim["queries"]) == 1, "查到的照记"
|
||||
assert "没能交结果" not in str(record)
|
||||
assert row.output["_issuance"]["tier"] == "formal", "没给出结论的句子不让出具降档"
|
||||
|
||||
|
||||
async def test_a_squeezed_claim_whose_verdict_runs_out_of_time_says_the_run_deadline(monkeypatch, shop,
|
||||
settles_slowly):
|
||||
Models(monkeypatch, shop)
|
||||
squeeze(monkeypatch)
|
||||
row = await finish(graph(shop), run_class="exploratory")
|
||||
assert row.status == "succeeded", row.error
|
||||
|
||||
[limit] = await logs(row.id, "challenge_limit")
|
||||
assert (limit["limit"], limit["claims_skipped"], limit["claims"]) == ("timeout_s", 2, 2)
|
||||
assert limit["cause"] == deadline.RUN_DEADLINE and limit["run_seconds"] == 600
|
||||
assert "每句时长上限" not in limit["message"], "收紧过的不叫人调高每句的上限"
|
||||
assert await logs(row.id, "challenge_unreviewed") == []
|
||||
record = artifact_store.load((await output_of(row.id, "chal"))["challenge_artifact"])
|
||||
assert [c["rationale"] for c in record["claims"]] == [SQUEEZED_CLAIM, SQUEEZED_CLAIM]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 双盲复算
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
async def test_a_blind_check_whose_result_runs_out_of_time_names_the_time_limit(monkeypatch, dual_shop,
|
||||
settles_slowly):
|
||||
name, _ = dual_shop
|
||||
blind = Blind(monkeypatch, name, {"销售额": (RIGHT, "gmv")})
|
||||
run = await start(dual_graph(name, RIGHT), options={"verify": {"dual_check": {"timeout_s": 0.5}}})
|
||||
assert run.status == "succeeded", run.error
|
||||
assert blind.runs == 0, "交结果那次没等到回复"
|
||||
evs = await events(run.id)
|
||||
assert [(d["limit"], d["metric"]) for d in dual_logs(evs, "dual_check_limit")] == [("timeout_s", "gmv")]
|
||||
out = dual_output(run, evs)
|
||||
assert out["status"] == "unchecked" and out["metrics"][0]["status"] == "unchecked"
|
||||
assert out["metrics"][0]["reason"] == "盲算已达每项时长上限(0.5 秒)"
|
||||
assert out["issuance_gaps"] == ["指标「销售额」无法复算:盲算已达每项时长上限(0.5 秒)"]
|
||||
[said] = dual_logs(evs, "dual_check_unchecked")
|
||||
assert "cause" not in said and said["metrics"][0]["reason"] == "盲算已达每项时长上限(0.5 秒)"
|
||||
assert "没能交结果" not in str(out)
|
||||
|
||||
|
||||
async def test_a_squeezed_blind_check_whose_result_runs_out_of_time_says_the_run_deadline(monkeypatch, dual_shop,
|
||||
settles_slowly):
|
||||
name, _ = dual_shop
|
||||
Blind(monkeypatch, name, {"销售额": (RIGHT, "gmv")})
|
||||
squeeze(monkeypatch)
|
||||
run = await start(dual_graph(name, RIGHT))
|
||||
assert run.status == "succeeded", run.error
|
||||
evs = await events(run.id)
|
||||
assert dual_logs(evs, "dual_check_limit") == [], "收紧过的不叫人调高每项的上限"
|
||||
out = dual_output(run, evs)
|
||||
assert out["metrics"][0]["status"] == "unchecked" and out["metrics"][0]["reason"] == SQUEEZED_BLIND
|
||||
[said] = dual_logs(evs, "dual_check_unchecked")
|
||||
assert said["cause"] == deadline.RUN_DEADLINE and said["run_seconds"] == 600
|
||||
Reference in New Issue
Block a user