核查模型的输出额度改为 16384、服务方嫌大时退回 4096,截断照实说:工具循环、收尾的每次调用都看有没有到了上限被截断

额度:judge.MAX_TOKENS 从 4096 改为 16384(核查的 verify_model_spec 沿用它)。经 OpenAI 兼容接入的推理模型收不到 thinking=off,思考算在额度里,4096 常常想完就没额度发查询、交结果;16384 是常见网关接受的输出上限里偏大的一档,cite_fields 的抽取也是这个数。服务方嫌大(400/422 且报错提到 max_tokens 一类)时退回 judge.FALLBACK_MAX_TOKENS(4096)再试一次,工具循环和收尾共用一个开关,之后都用小的;小额度的模型要退回时才建。

截断:工具循环里没有工具调用、finish_reason=length(或 stop_reason=max_tokens)不再当成「模型说完了」,一条查询都没成功时不再收尾(收尾只能交「查不到」,双盲复算就会记成「盲算没有查到这个数」);查到过东西的照常收尾,取不到要的数时同样交回截断的原因,取得到的照常交、notes 里注明。收尾被截断(Anthropic 强制工具调用写到一半、openai 结构化输出抛 LengthFinishReasonError、普通调用只有思考)时交回截断的原因,不再退回普通调用多花一次额度;截断那次的用量从异常里取,照记账。给人看的原因写明「核查模型的输出达到上限(N token)被截断,核查未能完成。通常是推理占满了输出额度,可在「设置 → 偏好设置 → 核查」中换用不带推理的模型」,经 error 交给双盲复算的无法复算原因、反方审稿的「反查未完成」、质询的报错;节点里调用时另记一条 verify_truncated,llm.end 标出 truncated 和额度。没有截断时原有说法不变。

测试:新增 test_verify_truncation(假模型返回 finish_reason=length、只有思考、LengthFinishReasonError,覆盖工具循环、收尾结构化和普通调用、退回 4096、别的 400 不重试、三条渠道);三处钉着 4096 的断言改成 16384。
This commit is contained in:
ThearchyHelios
2026-10-06 12:58:37 +08:00
parent 4dd40ed201
commit a552f4feab
7 changed files with 447 additions and 24 deletions
+3 -1
View File
@@ -549,7 +549,9 @@ async def verify_model_spec(session: AsyncSession, node_config: dict[str, Any] |
"""核查用哪个模型:节点配置里的 provider / model > 设置里的核查模型 > 证据裁判模型的取值链(裁判设置 >
助手模型 > 默认接入,见 judge.judge_model_spec)。哪一级写了就整组用哪一级,不跨级拼接入和模型。
和裁判一样关掉 thinking、max_tokens 4096:核查最后要按结构交结果(强制工具调用),和思考模式不兼容。
和裁判一样关掉 thinking:核查最后要按结构交结果(强制工具调用),和思考模式不兼容。额度也和裁判一样(judge.MAX_TOKENS,
16384):经 OpenAI 兼容接入的推理模型收不到 thinking=off,思考算在额度里,4096 常常想完就没额度查库、交结果了;
服务方嫌大时核查 agent 退回 judge.FALLBACK_MAX_TOKENS(engine/verify_agent.py)。
建议和被核查的模型不同,否则等于自己核自己;比较、提醒由调用方做(它知道被核查的是哪个模型)。
"""
node = node_config if isinstance(node_config, dict) else {}
+11 -3
View File
@@ -12,7 +12,8 @@
(按需裁判传只认封存工件的那个)。规则里写明数字已经由系统核对过。带期 4 文档标记的报告,上传表格的
查询在 SQL 之后另有来历行(来源、口径、已接受、区域外文字,_provenance_lines)
3. 调用(judge_doc):一份报告合并成一次调用,超过每批的句数就分批;结构化输出失败退回 JSON 解析,
再失败就记未裁判。thinking 关掉,max_tokens 4096
再失败就记未裁判。thinking 关掉,输出额度 MAX_TOKENS(推理模型的思考也算在里面),服务方嫌大时退回
FALLBACK_MAX_TOKENS;输出到了上限被截断的记未裁判(truncated),照实说是额度不够、不再退回纯文本多花一次
4. 预算:每份报告的句数、金额、时长和全局每日金额,每一项都能是 None(不限)。调用前按目录价估算,
超了按优先级截断;触顶就停,已判的保留,没判的记未裁判并说明是哪个上限——不报错
5. 每日计数落在设置表(SPEND_KEY),按本地日期滚动。目录里没有价格的模型估不出金额,金额上限对它
@@ -104,7 +105,13 @@ BATCH_SIZE = 8
#: 一波最多几批同时问:几批共用剩下的时长,句数上限内的句子一两波就判完。第一批单独问:接入调不通(鉴权、
#: 网关挂了)时只撞这一次,不让一整波一起失败
PARALLEL_BATCHES = 4
MAX_TOKENS = 4096
#: 裁判(和核查:api/settings.py 的 verify_model_spec、engine/verify_agent.py 沿用这一档)的输出额度。thinking 关掉只对
#: Anthropic 有用:经 OpenAI 兼容接入的推理模型(DeepSeek、GLM、Kimi 这一类)收不到 thinking=off,思考算在额度里,原来的
#: 4096 常常想完就没额度写判定了(cite_fields 的抽取在 2026-10 撞过同一堵墙,那边也改成了 16384)。16384 是常见网关、
#: 模型接受的输出上限里偏大的一档(GPT-4o 的上限就是它);更小的服务方会拒收,那时退回 FALLBACK_MAX_TOKENS
MAX_TOKENS = 16384
#: 服务方嫌 MAX_TOKENS 太大(400 / 422,报错里有 max_tokens 一类字样)时退回这个:原来的额度,各家都收。之后的调用都用它
FALLBACK_MAX_TOKENS = 4096
#: 证据摘录:SQL 前几个字、每次查询最多几行几列、每段知识库片段最多几个字、整份引用的检索给几段。
#: 列数上限只管没被引用的列:句子引用的格子所在的列一律给(遮罩的除外),再用别的列补满到这么多列
SQL_CHARS = 300
@@ -897,7 +904,8 @@ def _text(value: Any) -> str | None:
async def judge_model_spec(session: Any, node_judge: dict[str, Any] | None = None, *,
settings: dict[str, Any] | None = None) -> ModelSpec:
"""裁判用哪个模型:节点上的 judge.model(连同 provider)> 设置里的证据裁判模型 > Copilot 模型 >
默认 provider。哪一级写了就整组用哪一级,不跨级拼 provider 和模型。thinking 关、max_tokens 4096。
默认 provider。哪一级写了就整组用哪一级,不跨级拼 provider 和模型。thinking 关、额度 MAX_TOKENS(推理模型关不掉
思考,见 MAX_TOKENS 的说明;服务方嫌大时 run_request 退回 FALLBACK_MAX_TOKENS)。
建议和写作模型不同,否则等于自己审自己(报告节点发现相同会发 judge_same_model 警告)。
"""
+94 -16
View File
@@ -25,6 +25,12 @@ query_artifact 和 role: verify——随运行封存、证据面板能下钻。
每日上限不传就按设置里的「核查每日金额上限」,明确传 None 才是不限。模型不在价格目录里时估不出金额,金额上限
对它无效,照裁判的说法记一条说明。每次调用都记进核查的每日账(engine/spend.py)。
**输出额度。** 经 OpenAI 兼容接入的推理模型收不到 thinking=off,思考也算在输出额度里:额度按 spec(缺省 judge.MAX_TOKENS),
服务方嫌大时退回 judge.FALLBACK_MAX_TOKENS 再试一次,之后工具循环、收尾都用小的(providers/budget.py)。每次调用都看
有没有到了上限被截断:工具循环里被截断(话没说完、也没发查询)不算模型自己停下,一条查询都没成功时不再收尾;收尾被截断
时不再退回普通调用多花一次额度。因为截断没交成结果的,error 照实说是输出额度(TRUNCATED_ERROR),不说成「没查到」
「没按结构交」;节点里调用时另记一条 verify_truncated。
永远不抛:模型没配好、调用失败、格式不对、触顶、库写不进去,都落在返回值里(error / limit)。模型调用都经
get_chat_model,测试换成剧本。
"""
@@ -49,11 +55,12 @@ from app.db.base import SessionLocal
from app.engine import spend
from app.engine.evidence import build_catalog, cell_eid, compose_doc, query_entry_fields, schema_ledger_entry
from app.engine.expressions import CellError, column_kind, locate_cell, numeric_text
from app.engine.judge import _json_in, _tokens, _usd
from app.engine.judge import FALLBACK_MAX_TOKENS, MAX_TOKENS, _json_in, _tokens, _usd
from app.engine.nodes.llm import _DIGEST_BUDGET, _digest, _split_structured, _typed, explain_model_error
from app.engine.state import message_text
from app.engine.toolcalls import ToolTimeout, limit_fields, limit_of, run_bounded
from app.providers import catalog as pricing
from app.providers.budget import BudgetFallback, cut_off, cut_words, error_usage, truncated_error
from app.providers.factory import ModelSpec, ProviderNotConfigured, bind_tools_safely, get_chat_model
from app.tools.datasource import QUERY_PREFIX, SCHEMA_PREFIX, build_datasource_tools
from app.tools.registry import ToolArgsError, ToolContext, Versions, args_model_of, prepare_args
@@ -79,6 +86,12 @@ _REF_DEPTH = 8
RATIONALE = "rationale"
#: 触顶原因
LIMITS = ("max_queries", "timeout_s", "max_cost_usd", "daily_max_usd", "max_steps")
#: 核查模型的输出到了上限被截断时给人看的话。TRUNCATED_ERROR 是没交成结果的原因:双盲复算的无法复算、反方审稿的「反查
#: 未完成」、质询的报错都照它说,也是 verify_truncated 日志的原话。TRUNCATED_SETTLED 是截断前查到过东西、照常收尾了的,
#: 记进 notes 和日志。cut 是 budget.cut_words 的「核查模型的输出达到上限(N token)被截断」
TRUNCATED_ERROR = "{cut},核查未能完成。{advice}"
TRUNCATED_SETTLED = "{cut},已按截断前查到的结果收尾,结果可能不完整。{advice}"
TRUNCATED_ADVICE = "通常是推理占满了输出额度,可在「设置 → 偏好设置 → 核查」中换用不带推理的模型"
VERIFY_SYSTEM = (
"你是数据核查员:只读地查库,用查询结果完成交给你的任务。\n"
@@ -129,7 +142,8 @@ class VerifyOutcome:
source, tool, call_id}。
rationale:理由组装成的报告文档(evidence.compose_doc 的形状,violations 里有裸数字)。usage:{input_tokens,
output_tokens, total_tokens, cost_usd, calls, unpriced_calls}。limit:触顶原因(LIMITS 之一),没触顶为 None。
answered:模型是不是自己停下来的(没被上限掐断)。
answered:模型是不是自己停下来的(没被上限掐断,也没因为输出到了上限被截断)。因为截断没交成结果的,error 是
TRUNCATED_ERROR;截断前查到过东西、照常交了结果的,error 为 None,notes 里记一句 TRUNCATED_SETTLED。
"""
result: dict[str, Any] | None
@@ -686,13 +700,16 @@ async def _verify(state: _State, *, spec: ModelSpec, system: str, task: str, sou
def left() -> float | None:
return None if deadline is None else deadline - time.monotonic()
# ---- 模型
# ---- 模型。额度没写的按 MAX_TOKENS:推理模型的思考算在输出额度里,4096 常常想完就没额度发查询、交结果了
spec = spec.model_copy(update={"max_tokens": spec.max_tokens or MAX_TOKENS})
small = FALLBACK_MAX_TOKENS if spec.max_tokens > FALLBACK_MAX_TOKENS else None
try:
async with SessionLocal() as session:
base_model, model_id = await get_chat_model(session, spec)
first, model_id = await get_chat_model(session, spec)
except Exception as e: # noqa: BLE001 - 核查模型没配好:这回核不了,照实交回
why = not_configured(e) if isinstance(e, ProviderNotConfigured) else describe_exception(e)
return outcome(error=f"核查模型不可用({why})")
base_model = _with_fallback(first, spec, small)
state.model = model_id
if daily_max_usd is Default.SETTINGS:
from app.api.settings import verify_settings # 在这里导入:api.settings 导入了 engine 里的模块
@@ -743,7 +760,8 @@ async def _verify(state: _State, *, spec: ModelSpec, system: str, task: str, sou
raw = got if isinstance(got, BaseMessage) else _split_structured(got)[0]
except (asyncio.TimeoutError, Exception) as e: # noqa: BLE001 - 调用失败落在返回值里
failure = e
inp, out = _usage(raw)
# openai 的结构化输出被截断时抛异常,用量在异常里:额度是真花出去了
inp, out = _usage(raw) if raw is not None else error_usage(failure)
cost = await budget.settle(day, estimate, inp, out)
usage["input_tokens"] += inp
usage["output_tokens"] += out
@@ -756,6 +774,10 @@ async def _verify(state: _State, *, spec: ModelSpec, system: str, task: str, sou
if failure is not None:
end["error"] = "超过核查的时长上限,已放弃这次调用" if isinstance(failure, asyncio.TimeoutError) \
else explain_model_error(failure, model_id)
if truncated_error(failure) or cut_off(raw):
# 到了上限被截断:不是「调用失败,请稍后重试」,重试解决不了额度问题
limit = base_model.max_tokens
end.update(truncated=True, max_tokens=limit, error=cut_words("核查模型", limit))
await _announce(emit, EventType.LLM_END, **end)
return got, raw, failure
@@ -766,6 +788,7 @@ async def _verify(state: _State, *, spec: ModelSpec, system: str, task: str, sou
said = ""
attempts = 0
done: dict[str, str] = {} # 成功过的查询(工具名 + 参数)→ 编号:一模一样的不再执行
cut: tuple[str, int | None] | None = None # 最近一次因为输出到了上限被截断的调用:(哪一步, 那次的额度)
for _ in range(steps):
remaining = left()
if remaining is not None and remaining <= 0:
@@ -781,11 +804,18 @@ async def _verify(state: _State, *, spec: ModelSpec, system: str, task: str, sou
if isinstance(failure, asyncio.TimeoutError):
stopped = "timeout_s"
break
if truncated_error(failure):
cut = ("verify", base_model.max_tokens)
break
if failure is not None or not isinstance(response, BaseMessage):
return outcome(error=explain_model_error(failure, model_id) if failure else "核查模型没有给出回复")
messages.append(response)
calls = getattr(response, "tool_calls", None) or []
if not calls:
if cut_off(response):
# 话没说完、也没发查询就到了上限(推理模型想完了额度也用完了):不是模型自己停下来的,不能当成说完了去收尾
cut, said = ("verify", base_model.max_tokens), message_text(response) or said
break
answered, said = True, message_text(response)
break
said = message_text(response) or said
@@ -850,12 +880,35 @@ async def _verify(state: _State, *, spec: ModelSpec, system: str, task: str, sou
await _announce(emit, EventType.LOG, level="warn", code="verify_limit", limit=stopped,
message=f"核查已达上限({words}),已停止查询,按已有结果收尾")
# ---- 收尾:按结构交结果。关掉 thinking:结构化输出靠强制工具调用,和思考模式不兼容
async def report_cut(delivered: bool) -> str:
"""被截断过:记一条 verify_truncated,返回给人看的话(没交成结果的原因,或者交了结果时记进 notes 的说明)。"""
purpose, limit = cut or ("verify", None)
text = (TRUNCATED_SETTLED if delivered else TRUNCATED_ERROR).format(cut=cut_words("核查模型", limit),
advice=TRUNCATED_ADVICE)
await _announce(emit, EventType.LOG, level="warn", code="verify_truncated", purpose=purpose, max_tokens=limit,
delivered=delivered, message=text)
return text
async def give_up(error: str, limit: str | None = None) -> VerifyOutcome:
"""没交成结果,原因照 error 说。工具循环里被截断过的,时间线上照记一条:收尾失败可能就是它带出来的。"""
if cut is not None:
await report_cut(False)
return outcome(limit=limit or stopped, answered=answered, error=error)
if cut is not None and not queries:
# 一条查询都没成功就被截断:收尾只能交「查不到」,双盲复算会记成「盲算没有查到这个数」,原因却是输出额度。
# 收尾那次多半同样被推理占满,不再为它多花一次额度,直接交回截断的原因
return outcome(limit=stopped, answered=answered, error=await report_cut(False))
# ---- 收尾:按结构交结果。关掉 thinking:结构化输出靠强制工具调用,和思考模式不兼容(经 OpenAI 兼容接入的推理模型
# 关不掉,所以额度和工具循环一样,退回小额度也和工具循环共用一个开关)
settle_spec = spec.model_copy(update={"thinking": "off"})
try:
async with SessionLocal() as session:
settler, _ = await get_chat_model(session, spec.model_copy(update={"thinking": "off"}))
settle_model, _ = await get_chat_model(session, settle_spec)
except Exception as e: # noqa: BLE001
return outcome(limit=stopped, answered=answered, error=f"核查模型不可用({describe_exception(e)})")
return await give_up(f"核查模型不可用({describe_exception(e)})")
settler = _with_fallback(settle_model, settle_spec, small, shared_with=base_model)
final = _final_messages(head, task, queries, said, words)
try:
structured = settler.with_structured_output(schema, include_raw=True)
@@ -870,11 +923,12 @@ async def _verify(state: _State, *, spec: ModelSpec, system: str, task: str, sou
parsed: dict[str, Any] | None = None
failure: BaseException | None = None
settle_cut = False # 收尾那次到了上限被截断
if structured is not None:
estimate = budget.estimate(_text_of(final) + schema_text, _OUT_FINAL)
day, refused = await budget.reserve(estimate, kept=kept)
if refused:
return outcome(limit=stopped or refused, answered=answered, error=f"已达{words_of(refused)},没能交结果")
return await give_up(f"已达{words_of(refused)},没能交结果", refused)
got, raw, failure = await ask(final, structured.ainvoke, estimate, day or "", "verify_result", until_finish())
parsed = None if failure is not None else _parsed(_split_structured(got)[1])
if parsed is None and raw is not None:
@@ -882,43 +936,67 @@ async def _verify(state: _State, *, spec: ModelSpec, system: str, task: str, sou
for candidate in [message_text(raw), *(c.get("args") for c in getattr(raw, "tool_calls", None) or [])]:
if (parsed := _parsed(candidate)) is not None:
break
if parsed is None and not isinstance(failure, asyncio.TimeoutError):
settle_cut = parsed is None and (truncated_error(failure) or bool(cut_off(raw)))
if parsed is None and not isinstance(failure, asyncio.TimeoutError) and not settle_cut:
# 不支持结构化输出的模型:普通调用一次,结构写进消息,从回复里把 JSON 抠出来。结构化那次已经做过的,
# 这一次是补救,照常按上限拦;没做过的,这一次就是收尾,循环里留过钱
# 这一次是补救,照常按上限拦;没做过的,这一次就是收尾,循环里留过钱。结构化那次是被截断的不补救:
# 推理占满额度时普通调用多半一样被截断,白花一次额度
plain = _plain_messages(final, schema)
remaining = until_finish()
if remaining is None or remaining > 0:
estimate = budget.estimate(_text_of(plain), _OUT_FINAL)
day, refused = await budget.reserve(estimate, kept=kept and structured is None)
if refused and structured is None:
return outcome(limit=stopped or refused, answered=answered,
error=f"已达{words_of(refused)},没能交结果")
return await give_up(f"已达{words_of(refused)},没能交结果", refused)
if not refused:
got, _, failure = await ask(plain, settler.ainvoke, estimate, day or "", "verify_result", remaining)
parsed = None if failure is not None else _parsed(message_text(got) if got is not None else None)
settle_cut = parsed is None and (truncated_error(failure) or bool(cut_off(got)))
if parsed is None:
if settle_cut:
cut = ("verify_result", settler.max_tokens)
return outcome(limit=stopped, answered=answered, error=await report_cut(False))
why = "收尾时超过时长上限,没能交结果" if isinstance(failure, asyncio.TimeoutError) else \
f"核查模型没有按结构交结果({explain_model_error(failure, model_id)})" if failure is not None else \
"核查模型没有按结构交结果"
return outcome(limit=stopped, answered=answered, error=why)
return await give_up(why)
declared = list((resolved.get("properties") or {}) if _is_object(resolved) else {})
if declared and not any(k in parsed for k in declared):
# 一个要求的字段都没交:不是「查不到」,是没按结构交。不这么分,字段名对不上的回复会被当成什么都没查到
return outcome(limit=stopped, answered=answered,
error="核查模型没有按结构交结果(交回的内容里没有要求的字段:"
return await give_up("核查模型没有按结构交结果(交回的内容里没有要求的字段:"
+ "、".join(declared[:8]) + ("等" if len(declared) > 8 else "") + ")")
# ---- 取值、组装理由
cells = _Cells(queries)
result = cells.walk(resolved, parsed) if _is_object(resolved) else {}
if cut is not None and any(c["status"] != "resolved" for c in cells.cells):
# 工具循环被截断,截断前查到的里面没有要的数(或者引用指不到):话没说完就停了,说「没查到」会让人以为数据里
# 没有。不交这份残缺的结果,原因照截断说;查过的照记
return outcome(cells=cells.cells, limit=stopped, answered=answered, error=await report_cut(False))
rationale = parsed.get(RATIONALE)
doc = None
if isinstance(rationale, str) and rationale.strip():
doc = compose_doc(rationale, build_catalog(nodes={}, ledger=ledger), node_id=owner, run_id=run_id or None)
result[RATIONALE] = doc["markdown"] if doc else ""
if cut is not None:
# 截断前查到的够用:照常交结果,说明记进 notes,时间线上照记一条(结果可能不完整)
notes.append(await report_cut(True))
return outcome(result=result, cells=cells.cells, rationale=doc, limit=stopped, answered=answered)
def _with_fallback(model: Any, spec: ModelSpec, small: int | None, *, shared_with: BudgetFallback | None = None
) -> BudgetFallback:
"""按 spec 建好的模型包一层:服务方嫌额度大时按 small 重建一个再调,要退回时才建,平常一个都不多建。small 为 None
是额度本来就不比它大,没有可退的。经本模块的 get_chat_model 建,测试换成剧本时小额度的那个也是剧本。"""
async def build() -> Any:
async with SessionLocal() as session:
return (await get_chat_model(session, spec.model_copy(update={"max_tokens": small})))[0]
return BudgetFallback(model, build=build if small else None, limits=(spec.max_tokens, small or spec.max_tokens),
shared_with=shared_with)
async def _run_tool(tool: Any, name: str, args: dict[str, Any], call_id: str, remaining: float | None,
emit: Emit | None, queries: list[dict[str, Any]], ledger: list[dict[str, Any]],
owner: str) -> str | None:
+2 -2
View File
@@ -6,7 +6,7 @@
- 一份报告合并成一次调用(超过每批的句数就分批),每句给 supported / partial / contradicted / insufficient /
not_a_claim
- 模型不支持结构化输出、或者给的不是合法结构时退回 JSON 解析,再失败就记「未裁判」,不编判定
- 裁判模型关掉 thinking,max_tokens 4096
- 裁判模型关掉 thinking,输出额度 16384(推理模型的思考也算在里面);服务方嫌大时退回 4096,截断照实说(见 test_judge_truncation)
- 预算(每份报告的句数、金额、时长,全局每日金额)每一项都能写 null 表示不限;触顶就停、已判的保留、
没判的记未裁判并说明是哪个上限——不报错、不让报告失败。金额超了按优先级截断:先判挂了依据的,
再判有方向词、因果词的
@@ -207,7 +207,7 @@ async def test_the_judge_model_runs_without_thinking(monkeypatch, stored):
max_tokens=64000),
budget=Budget(max_claims=None, max_cost_usd=None, timeout_s=30, daily_max_usd=None))
[spec] = judge.specs
assert spec.thinking == "off" and spec.max_tokens == MAX_TOKENS == 4096
assert spec.thinking == "off" and spec.max_tokens == MAX_TOKENS == 16384
assert spec.model == PRICED and spec.timeout and spec.timeout <= 30
+1 -1
View File
@@ -350,7 +350,7 @@ async def test_the_judge_model_follows_node_then_settings(client, monkeypatch):
row = await finish(weekly(claims="judge"))
await judge(client, row.id, [unit_ids(doc_of(row))[CAUSAL]])
[spec] = models.specs
assert spec.model == "judge-from-settings" and spec.thinking == "off" and spec.max_tokens == 4096
assert spec.model == "judge-from-settings" and spec.thinking == "off" and spec.max_tokens == 16384
row = await finish(weekly(claims="judge", judge={"model": "judge-from-node"}))
body = (await judge(client, row.id, [unit_ids(doc_of(row))[CAUSAL]])).json()
+2 -1
View File
@@ -17,6 +17,7 @@ from app.db.base import SessionLocal
from app.db.models import Setting
from app.engine.judge import (
JUDGE_DEFAULTS,
MAX_TOKENS,
SETTING_GROUP,
SPEND_KEY,
Budget,
@@ -166,7 +167,7 @@ async def test_the_judge_model_priority():
async with SessionLocal() as session:
spec = await judge_model_spec(session, None)
assert (spec.provider, spec.model) == (None, None), "什么都没配:默认 provider"
assert spec.thinking == "off" and spec.max_tokens == 4096
assert spec.thinking == "off" and spec.max_tokens == MAX_TOKENS == 16384
session.add(Setting(key="copilot", value={"provider": "", "model": "copilot-model"}))
await session.commit()
+334
View File
@@ -0,0 +1,334 @@
"""核查 agent(engine/verify_agent.py)的输出额度:截断照实说,服务方嫌额度大时退回小额度。
核查(双盲复算、反方审稿、审批质询共用)原来固定 max_tokens 4096、thinking off。经 OpenAI 兼容接入的推理模型收不到
thinking=off,思考就能把 4096 用光:这次调用 finish_reason=length,没有工具调用也没有 JSON。于是工具循环把它当成
「模型说完了」去收尾,双盲复算记成「盲算没有查到这个数」;收尾被截断时记成「核查模型没有按结构交结果」;质询回答失败
——都没说出真正的原因。这里守几件事:
- 额度:默认 16384(judge.MAX_TOKENS,核查和裁判共用);服务方嫌大时退回 4096 再试一次,之后工具循环、收尾都用小的,
只在还没有任何输出时重来;别的 400 不重试
- 工具循环里被截断(没有工具调用、话没说完)不算模型自己停下:一条查询都没成功时不再收尾,直接交回截断的原因;
查到过东西的照常按查到的收尾,取不到要的数时同样交回截断的原因
- 收尾被截断(Anthropic 的 stop_reason=max_tokens、openai 结构化输出抛的 LengthFinishReasonError、普通调用的
finish_reason=length):交回截断的原因,不再退回普通调用多花一次额度;截断那次的用量照记
- 时间线上记一条 verify_truncated(节点里调用时),llm.end 标出这次被截断
- 原因经现有的渠道交出去:双盲复算的无法复算原因、反方审稿的「反查未完成」、质询的报错;没有截断时原有说法不变
模型是剧本,数据源是现造的 SQLite 文件(借 test_verify_agent 的夹具),表名、数字都是虚构的。
"""
from __future__ import annotations
import json
from langchain_core.messages import AIMessage, BaseMessage
from openai import LengthFinishReasonError
from openai.types import CompletionUsage
from openai.types.chat import ChatCompletion
from app.core.events import EventType
from app.engine import judge as judging
from app.engine import verify_agent
from app.engine.inquiry import _error_words
from app.engine.nodes.challenge import _claim_of
from app.engine.nodes.dual_check import _slim
from tests.test_verify_agent import VALID_SUM, clean_spend, q, run, shop # noqa: F401 - 夹具按名字引入
CUT_16K = "核查模型的输出达到上限(16384 token)被截断"
ADVICE = "通常是推理占满了输出额度,可在「设置 → 偏好设置 → 核查」中换用不带推理的模型"
FOUND = {"value": {"alias": "Q1", "row": 0, "column": "total"}, "verdict": "一致",
"rationale": "未作废订单的金额合计是 [[v:Q1.r0.total]]。"}
def cut_reply(output: int = 16384) -> AIMessage:
"""推理模型把额度全用在思考上:正文为空、没有工具调用,finish_reason=length(DeepSeek 经 LiteLLM 的样子)。"""
return AIMessage(content="", additional_kwargs={"reasoning_content": "先想想该查哪张表……"},
response_metadata={"finish_reason": "length"},
usage_metadata={"input_tokens": 900, "output_tokens": output, "total_tokens": 900 + output})
def anthropic_cut() -> dict:
"""结构化输出走强制工具调用(Anthropic)时被截断:原始回复 stop_reason=max_tokens,解析不出结果。"""
raw = AIMessage(content=[], response_metadata={"stop_reason": "max_tokens"},
usage_metadata={"input_tokens": 700, "output_tokens": 16384, "total_tokens": 17084})
return {"raw": raw, "parsed": None, "parsing_error": None}
def length_error(prompt: int = 800, completion: int = 16384) -> LengthFinishReasonError:
"""openai 的结构化输出(json_schema)在 finish_reason=length 时抛的那个错。"""
usage = CompletionUsage(completion_tokens=completion, prompt_tokens=prompt, total_tokens=prompt + completion)
return LengthFinishReasonError(completion=ChatCompletion.model_construct(
id="x", choices=[], created=0, model="m", object="chat.completion", usage=usage))
def query_step(name: str, sql: str = VALID_SUM, call_id: str = "c1") -> AIMessage:
return AIMessage(content="查一下。", tool_calls=[{"name": n, "args": a, "id": i} for n, a, i in q(name, sql, call_id)])
class BudgetRejected(Exception):
"""服务方嫌输出额度太大时的 400(DeepSeek chat 系只收 8K)。"""
status_code = 400
class Reasoner:
"""核查模型的剧本。steps:工具循环里每一轮的回复,用完了回「查完了。」;structured / plain:收尾时结构化调用、
普通调用的回复(AIMessage、结构化结果,或者一个异常就抛它)。calls 按顺序记下每次调用是哪一种。"""
def __init__(self, steps=(), *, structured=None, plain=None, unstructured: bool = False) -> None:
self.steps = list(steps)
self.structured = structured
self.plain = plain
self.unstructured = unstructured
self.calls: list[str] = []
def bind_tools(self, tools, **kwargs):
return _View(self, "step")
def with_structured_output(self, schema, **kwargs):
if self.unstructured:
raise NotImplementedError("这个模型不支持结构化输出")
return _View(self, "final")
async def ainvoke(self, messages, *args, **kwargs):
return self.answer("plain")
def answer(self, kind: str):
self.calls.append(kind)
if kind == "step":
reply = self.steps.pop(0) if self.steps else AIMessage(content="查完了。")
else:
reply = self.structured if kind == "final" else self.plain
if isinstance(reply, BaseException):
raise reply
return reply
class _View:
def __init__(self, model: Reasoner, kind: str) -> None:
self.model, self.kind = model, kind
async def ainvoke(self, messages, *args, **kwargs):
return self.model.answer(self.kind)
class Rejecting:
"""一开口就被拒的模型:工具循环、收尾都一样。"""
def __init__(self, message: str = "max_tokens: 16384 > 4096, which is the maximum allowed number of output tokens"):
self.message = message
self.calls = 0
def bind_tools(self, tools, **kwargs):
return self
def with_structured_output(self, schema, **kwargs):
return self
async def ainvoke(self, messages, *args, **kwargs):
self.calls += 1
raise BudgetRejected(self.message)
class Models:
"""按额度给模型:rejects_above 给了,额度比它大的一律交 Rejecting。specs 记下每次建模型的配置。"""
def __init__(self, monkeypatch, model, *, rejects_above: int | None = None, rejecting: Rejecting | None = None):
self.specs = []
self.rejecting = rejecting or Rejecting()
async def fake(session, spec):
self.specs.append(spec)
if rejects_above is not None and (spec.max_tokens or 0) > rejects_above:
return self.rejecting, "reasoner-x"
return model, "reasoner-x"
monkeypatch.setattr(verify_agent, "get_chat_model", fake)
def collector():
events: list[tuple] = []
return events, (lambda kind, **data: events.append((kind, data)))
def truncated_logs(events) -> list[dict]:
return [d for k, d in events if k == EventType.LOG and d.get("code") == "verify_truncated"]
# --------------------------------------------------------------------------
# 工具循环里被截断
# --------------------------------------------------------------------------
async def test_thinking_that_eats_the_budget_in_the_loop_is_reported_as_truncation(monkeypatch, shop):
name, _ = shop
model = Reasoner([cut_reply()], structured={"value": None, "verdict": "无法判断", "rationale": "没有查询结果"})
Models(monkeypatch, model)
events, emit = collector()
out = await run(name, emit=emit, node_id="dc")
assert out.result is None and out.queries == []
assert out.error == f"{CUT_16K},核查未能完成。{ADVICE}"
assert model.calls == ["step"], "一条查询都没有:收尾只能交「查不到」,不再为它多花一次额度"
assert out.answered is False, "被截断不是模型自己停下来的"
assert out.usage["output_tokens"] == 16384, "截断那次的额度是真花出去了"
[log] = truncated_logs(events)
assert log["level"] == "warn" and log["max_tokens"] == 16384 and log["delivered"] is False
assert log["purpose"] == "verify" and log["message"] == out.error
[end] = [d for k, d in events if k == EventType.LLM_END]
assert end["truncated"] is True and end["max_tokens"] == 16384 and CUT_16K in end["error"]
async def test_a_cut_after_a_query_still_settles_on_what_was_found(monkeypatch, shop):
name, _ = shop
model = Reasoner([query_step(name), cut_reply()], structured=FOUND)
Models(monkeypatch, model)
events, emit = collector()
out = await run(name, emit=emit, node_id="dc")
assert model.calls == ["step", "step", "final"]
assert out.error is None and out.result["value"] == 230, "查到的数照常交:截断前的查询是真查过的"
assert out.answered is False
assert any(CUT_16K in n and "已按截断前查到的结果收尾" in n for n in out.notes)
[log] = truncated_logs(events)
assert log["delivered"] is True and log["purpose"] == "verify" and log["max_tokens"] == 16384
async def test_a_cut_that_leaves_the_number_unfound_is_reported_as_truncation(monkeypatch, shop):
"""查过、可要的数没查到就被截断了:说「没查到」会让人以为数据里没有,原因要写截断。"""
name, _ = shop
model = Reasoner([query_step(name, "SELECT COUNT(*) AS n FROM orders"), cut_reply()],
structured={"value": None, "verdict": "无法判断", "rationale": "还没查到合计"})
Models(monkeypatch, model)
events, emit = collector()
out = await run(name, emit=emit, node_id="dc")
assert out.result is None and out.error == f"{CUT_16K},核查未能完成。{ADVICE}"
assert [c["status"] for c in out.cells] == ["missing"] and len(out.queries) == 1, "查过的照记"
[log] = truncated_logs(events)
assert log["delivered"] is False and log["purpose"] == "verify"
# --------------------------------------------------------------------------
# 收尾被截断
# --------------------------------------------------------------------------
async def test_a_cut_off_settle_is_reported_and_not_retried_as_plain(monkeypatch, shop):
"""Anthropic 系:强制工具调用写到一半到了上限。退回普通调用也多半被推理占满,不再白花一次额度。"""
name, _ = shop
model = Reasoner([query_step(name)], structured=anthropic_cut(), plain=AIMessage(content=json.dumps(FOUND)))
Models(monkeypatch, model)
events, emit = collector()
out = await run(name, emit=emit, node_id="dc")
assert model.calls == ["step", "step", "final"], "没有再退回普通调用"
assert out.result is None and out.error == f"{CUT_16K},核查未能完成。{ADVICE}"
[log] = truncated_logs(events)
assert log["purpose"] == "verify_result" and log["delivered"] is False
end = [d for k, d in events if k == EventType.LLM_END][-1]
assert end["purpose"] == "verify_result" and end["truncated"] is True and CUT_16K in end["error"]
async def test_an_openai_length_error_in_the_settle_is_truncation_not_a_failed_call(monkeypatch, shop):
"""openai 的结构化输出解析时抛 LengthFinishReasonError:不是「调用失败,请稍后重试」,是额度被推理占满。"""
name, _ = shop
model = Reasoner([query_step(name)], structured=length_error(completion=16384))
Models(monkeypatch, model)
events, emit = collector()
out = await run(name, emit=emit, node_id="dc")
assert model.calls == ["step", "step", "final"]
assert out.error == f"{CUT_16K},核查未能完成。{ADVICE}"
assert out.usage["output_tokens"] == 16384 and out.usage["input_tokens"] >= 800, "异常里带着的用量照记"
end = [d for k, d in events if k == EventType.LLM_END][-1]
assert "请稍后重试" not in end["error"] and CUT_16K in end["error"] and end["output_tokens"] == 16384
async def test_a_plain_settle_cut_off_is_reported(monkeypatch, shop):
"""不支持结构化输出的模型:收尾直接走普通调用,回复只有思考、finish_reason=length。"""
name, _ = shop
model = Reasoner([query_step(name)], plain=cut_reply(), unstructured=True)
Models(monkeypatch, model)
out = await run(name)
assert model.calls == ["step", "step", "plain"]
assert out.result is None and out.error == f"{CUT_16K},核查未能完成。{ADVICE}"
async def test_without_truncation_the_old_wording_stays(monkeypatch, shop):
name, _ = shop
model = Reasoner([query_step(name)], structured="不是 JSON", plain=AIMessage(content="还是不是 JSON"))
Models(monkeypatch, model)
events, emit = collector()
out = await run(name, emit=emit, node_id="dc")
assert model.calls == ["step", "step", "final", "plain"], "格式不对照旧退回普通调用补救一次"
assert out.error == "核查模型没有按结构交结果"
assert truncated_logs(events) == []
assert not any(d.get("truncated") for k, d in events if k == EventType.LLM_END)
# --------------------------------------------------------------------------
# 额度与退回
# --------------------------------------------------------------------------
async def test_the_default_budget_leaves_room_for_reasoning(monkeypatch, shop):
name, _ = shop
models = Models(monkeypatch, Reasoner([query_step(name)], structured=FOUND))
out = await run(name)
assert out.error is None and out.result["value"] == 230
assert [s.max_tokens for s in models.specs] == [16384, 16384], "工具循环、收尾都按 16384,不多建模型"
assert models.specs[1].thinking == "off"
async def test_a_rejected_budget_falls_back_to_4096_for_the_loop_and_the_settle(monkeypatch, shop):
name, _ = shop
model = Reasoner([query_step(name)], structured=FOUND)
models = Models(monkeypatch, model, rejects_above=4096)
out = await run(name)
assert out.error is None and out.result["value"] == 230
assert [s.max_tokens for s in models.specs] == [16384, 4096, 16384, 4096]
assert [s.thinking for s in models.specs][2:] == ["off", "off"], "收尾的小额度模型同样关掉思考"
assert models.rejecting.calls == 1, "退回过一次之后收尾直接用小的,不再先撞一次"
assert model.calls == ["step", "step", "final"]
async def test_a_cut_after_the_fallback_names_the_small_budget(monkeypatch, shop):
name, _ = shop
Models(monkeypatch, Reasoner([cut_reply(output=4096)]), rejects_above=4096)
out = await run(name)
assert out.error.startswith("核查模型的输出达到上限(4096 token)被截断")
async def test_other_rejections_are_not_retried_with_the_small_budget(monkeypatch, shop):
name, _ = shop
models = Models(monkeypatch, Reasoner(), rejects_above=4096,
rejecting=Rejecting("model 'reasoner-x' does not exist"))
out = await run(name)
assert out.result is None and "reasoner-x" in out.error
assert [s.max_tokens for s in models.specs] == [16384], "不是额度的问题:不拿小额度再撞一次"
async def test_the_verify_spec_asks_for_the_larger_budget():
"""核查模型的配置(api/settings.py 的 verify_model_spec 沿用证据裁判的额度):16384,思考照旧关掉。"""
assert judging.MAX_TOKENS == 16384 and judging.FALLBACK_MAX_TOKENS == 4096
# --------------------------------------------------------------------------
# 原因经现有的渠道交出去
# --------------------------------------------------------------------------
async def test_the_reason_reaches_dual_check_challenge_and_inquiry(monkeypatch, shop):
name, _ = shop
Models(monkeypatch, Reasoner([cut_reply()]))
out = await run(name)
expected = f"{CUT_16K},核查未能完成。{ADVICE}"
# 双盲复算:无法复算的原因
slim = _slim(out)
assert slim["value"] is None and slim["reason"] == expected
# 反方审稿:这句记为无法判断,理由写「反查未完成」,没能反查的原因就是截断
cand = judging.Candidate(unit="u1", priority=1, order=0, text="华南每周的销售额都高于华东", section="",
aliases=("Q1",))
claim, why = _claim_of(cand, out)
assert claim["verdict"] == "inconclusive" and claim["rationale"] == f"反查未完成:{expected}" and why == expected
# 审批质询:提问人看到的报错照原话
assert _error_words(out, timeout_s=60, daily_max_usd=1.0) == expected
def test_the_reply_shapes_used_here_are_messages():
assert isinstance(cut_reply(), BaseMessage) and isinstance(anthropic_cut()["raw"], BaseMessage)