核查因运行时长上限将至没核成的,时间线另起一条并带上原因(cause: run_deadline);收紧过的核查 agent 触顶时也改说时长上限将至;给下游留的余量看下游还有几个要调模型的节点,时长上限调短时余量跟着缩小

为什么(复核第 1 轮):没开始的句子、指标和收紧后到点的,以前和别的原因记在同一条日志里,没有可机读的字段,界面只能照旧叫人调高每句、每项的上限(调高了只会更快撞线),或去查核查模型和数据源(它们本来就没问题);前面有一句因别的原因没审时,运行时长上限这个原因还被藏进「等」里。双盲复算只给下游留 60 秒,惯常接法「双盲复算 → 写报告 → 反方审稿 → 成果」里写报告一慢,运行照样在写报告时被中止;时长上限调到 180 秒左右时,固定的 60 秒余量又让核查一项都开始不了。

- 反方审稿:没开始的句子另发一条 challenge_unreviewed,带 cause、run_seconds,不混进别的原因;收紧后到点的那条 challenge_limit 照记 timeout_s(老界面认得),另带 cause、run_seconds 和触顶的句数 claims,同一节点里有句子触了自己的上限时两条分得开。产出的 unreviewed 照算
- 双盲复算:没开始、收紧后到点的盲算结果带 cause、run_seconds,_compare 记无法复算时一并带上(只这一行),dual_check_unchecked 按有无 cause 分两条发,时限将至的那条另带 cause、run_seconds,原话列指标名(最多 5 个)
- deadline.relabel 包住交给核查 agent 的 emit:收紧过的那一项到点时,verify_limit 改说「本次运行即将达到 600 秒的时长上限,时长已按剩余时间收紧到 52.3 秒」,带 cause;别的事件、别的上限原样转发,没收紧时原样交回
- 余量按份算:一份取 60 秒和这一段的 10% 中大的,这一段不到 240 秒时按 25%;下游这一段里(到人工审批为止)每个还要调模型的节点(模型、Agent、协作、子工作流、写报告、双盲复算、反方审稿)各再加一份,合计不超过这一段的一半。默认 600 秒的一段、下游只有成果节点时和以前一样
- 没有截止时刻时一切照旧;决定都还在原来的 @task 里,接着运行取 checkpoint

测试:反方审稿、双盲复算各补一条「另一个原因 + 时限将至」两条分开说、带不带 cause 的用例,反方审稿补一条触了自己上限和收紧后到点分得开的用例;原有用例补上 cause、run_seconds、原话的断言。假的钟钉住下游有写报告时第 3 项只查 40~50 秒(没有时 100~110 秒);真的钟用 10 秒一段、盲算查到时长用完、写报告要 3 秒,以前在写报告时被中止,现在运行成功。单元测试钉住短时长的余量、下游每个节点加一份和一半的上限、数下游节点(人工审批之后不算、汇合只算一次、环回自己不算)、relabel 改什么不改什么。去掉下游余量、去掉 relabel 后对应用例都变红。
This commit is contained in:
ThearchyHelios
2026-10-06 14:25:38 +08:00
parent d9acfbfc0b
commit e3f5d566d0
5 changed files with 365 additions and 39 deletions
+78 -11
View File
@@ -12,21 +12,44 @@ runner 给每一段执行(两次人工介入之间的那一段)套一道 asy
截止时刻用事件循环的钟(loop.time()),和 asyncio.timeout 同一个钟。没有上限时(不经 runner 执行的节点:单元测试、
审批质询接口)为 None,allot() 原样放行,节点的行为和没有这道检查时一样。
因此没核成的,节点发日志时带上 cause: RUN_DEADLINE 和这一段的总时长(run_seconds),另起一条,不和别的原因混在一起:
界面据此说「运行时长上限将至」、给对的下一步——调高核查自己的上限只会更快撞线,检查模型、数据源也无济于事。
"""
from __future__ import annotations
import asyncio
import contextlib
import math
from collections.abc import AsyncIterator
from collections.abc import AsyncIterator, Callable
from contextvars import ContextVar
from dataclasses import dataclass
from typing import Any
#: 给下游节点留的余量:核查节点后面通常还有成果节点(出具契约复核报告),也可能还有写报告、发通知这类节点。核查把
#: 时间用到最后一刻,后面的节点一开始就撞上时长上限,整次运行照样被中止。取 RESERVE_S 和这一段总时长的 RESERVE_SHARE
#: 中大的:60 秒够成果节点复核、再做一两次普通的模型调用;管理员调大了时长上限的,多半是因为节点本来就慢,余量跟着放大
from app.core.events import EventType
from app.engine.schema import GraphSpec, NodeType
#: 给下游节点留的余量,按份算。一份给收尾的节点:核查节点后面通常还有成果节点(出具契约复核报告),核查把时间用到
#: 最后一刻,后面的节点一开始就撞上时长上限,整次运行照样被中止。一份取 RESERVE_S 和这一段总时长的 RESERVE_SHARE 中
#: 大的:60 秒够成果节点复核、再做一两次普通的模型调用;管理员调大了时长上限的,多半是因为节点本来就慢,余量跟着放大
RESERVE_S = 60.0
RESERVE_SHARE = 0.1
#: 这一段不到 RESERVE_S / SHORT_SHARE(240 秒)时,一份按总时长的 SHORT_SHARE 算:时长上限调得很短(比如 180 秒)时,
#: 固定的 60 秒余量加上收尾宽限,前面的节点用掉一分钟核查就一项都开始不了,比没有这道检查时还差
SHORT_SHARE = 0.25
#: 下游这一段里每个还要调模型的节点再各加一份:Copilot 指南里的惯常接法是「口径卡 → 双盲复算 → 写报告 → 反方审稿 →
#: 成果」,双盲复算只留一份的话,写报告那次模型调用一慢,运行就在写报告时被中止。反方审稿也记一份:默认 600 秒的一段
#: 一份是 60 秒,正好够它开始一句(它自己那份余量之外,还要收尾宽限和最短的查库时长各 30 秒)。估得粗,宁可多留:
#: 核查少做几项记成没核成,好过整次运行被中止
MODEL_NODES = frozenset({NodeType.LLM, NodeType.AGENT, NodeType.SUPERVISOR, NodeType.SUBGRAPH, NodeType.REPORT,
NodeType.DUAL_CHECK, NodeType.CHALLENGE})
#: 余量合计最多占这一段总时长的这么多:下游节点再多,核查节点也总还能开始几项
RESERVE_CAP = 0.5
#: 日志里的 cause:这几项没核成是因为运行时长上限将至(不再开始、收紧后到了点)
RUN_DEADLINE = "run_deadline"
#: 收紧过的核查到了点:核查 agent 自己发的 verify_limit 改成这一句(relabel)
SQUEEZED_LIMIT = ("核查已达上限(本次运行即将达到 {seconds:g} 秒的时长上限,时长已按剩余时间收紧到 {timeout:g} 秒),"
"已停止查询,按已有结果收尾")
@dataclass(frozen=True)
@@ -56,9 +79,35 @@ def current() -> Segment | None:
return _SEGMENT.get()
def reserve(seconds: float) -> float:
"""给下游节点留的余量(秒),理由见 RESERVE_S。"""
return max(RESERVE_S, RESERVE_SHARE * seconds)
def reserve(seconds: float, after: int = 0) -> float:
"""给下游节点留的余量(秒):一份给收尾的节点,after 是下游这一段里还要调模型的节点数(models_after),各再加一份;
合计不超过总时长的 RESERVE_CAP。一份多大、为什么,见 RESERVE_S、SHORT_SHARE、MODEL_NODES。"""
share = max(min(RESERVE_S, SHORT_SHARE * seconds), RESERVE_SHARE * seconds)
return min(share * (1 + max(after, 0)), RESERVE_CAP * seconds)
def models_after(spec: GraphSpec, node_id: str) -> int:
"""这个节点下游、这一段执行里还要调模型的节点(MODEL_NODES)有几个。沿连线往下走,人工审批节点不往后走:审批之后是
新的一段,重新计时。分支汇合、环路里的节点只算一次,自己不算;分支上不一定执行的也算上,宁可多留。"""
nodes = spec.node_map()
children: dict[str, list[str]] = {}
for edge in spec.edges:
children.setdefault(edge.source, []).append(edge.target)
seen: set[str] = {node_id}
stack = list(children.get(node_id, []))
count = 0
while stack:
cur = stack.pop()
node = nodes.get(cur)
if cur in seen or node is None:
continue
seen.add(cur)
if node.type == NodeType.HUMAN:
continue
if node.type in MODEL_NODES:
count += 1
stack.extend(children.get(cur, []))
return count
def _now() -> float:
@@ -77,9 +126,10 @@ class Allot:
seconds: float | None = None
def allot(timeout_s: float | None, *, grace: float) -> Allot:
def allot(timeout_s: float | None, *, grace: float, after: int = 0) -> Allot:
"""开始一项核查之前问一句:够不够,这一项的时长上限收紧到多少。timeout_s 是这一项配置的时长上限(None 不限),
grace 是到点之后收尾那次调用的宽限(verify_agent.SETTLE_GRACE_S)。
grace 是到点之后收尾那次调用的宽限(verify_agent.SETTLE_GRACE_S),after 是下游这一段里还要调模型的节点数
(models_after,按它多留余量)。
一项最坏要用「时长上限 + 收尾宽限」。剩下的时间先扣掉给下游的余量、再扣掉收尾宽限,就是这一项查库最多能用的时长:
比配置的短就收紧到它(取到 0.1 秒,往下取:时间线上写「时长上限 69.7 秒」,不写一串小数)。收紧之后不到
@@ -89,7 +139,7 @@ def allot(timeout_s: float | None, *, grace: float) -> Allot:
found = _SEGMENT.get()
if found is None:
return Allot(True, timeout_s)
room = math.floor((found.deadline - _now() - reserve(found.seconds) - grace) * 10) / 10
room = math.floor((found.deadline - _now() - reserve(found.seconds, after) - grace) * 10) / 10
limit = room if timeout_s is None else min(timeout_s, room)
floor = grace if timeout_s is None else min(timeout_s, grace)
if limit < floor:
@@ -97,4 +147,21 @@ def allot(timeout_s: float | None, *, grace: float) -> Allot:
return Allot(True, limit, tightened=timeout_s is None or room < timeout_s, seconds=found.seconds)
__all__ = ["RESERVE_S", "RESERVE_SHARE", "Allot", "Segment", "allot", "current", "reserve", "segment"]
def relabel(emit: Callable[..., Any], allot: Allot) -> Callable[..., Any]:
"""交给核查 agent 的 emit。这一项的时长收紧过时,它到点自己发的 verify_limit 写的是收紧后的秒数(「时长上限 52.3
秒」),和任何设置都对不上,读的人会去找这个数是在哪儿配的、甚至去调大它:改说运行时长上限将至,带上 cause 和这一段
的总时长。别的事件、别的上限原样转发;没收紧时原样交回 emit。"""
if not allot.tightened:
return emit
def wrapped(kind: Any, **data: Any) -> Any:
if str(kind) == EventType.LOG and data.get("code") == "verify_limit" and data.get("limit") == "timeout_s":
data = {**data, "cause": RUN_DEADLINE, "run_seconds": allot.seconds,
"message": SQUEEZED_LIMIT.format(seconds=allot.seconds, timeout=allot.timeout_s)}
return emit(kind, **data)
return wrapped
__all__ = ["MODEL_NODES", "RESERVE_CAP", "RESERVE_S", "RESERVE_SHARE", "RUN_DEADLINE", "SHORT_SHARE", "Allot",
"Segment", "allot", "current", "models_after", "relabel", "reserve", "segment"]
+32 -13
View File
@@ -36,8 +36,9 @@
逐句串行反查:并行的话几句的工具调用在时间线上交错,看不出哪条查询在审哪一句;每日余额按先后扣,结果也才确定。
串行就可能撞上运行的时长上限(默认 5 句、每句 90 秒再加 30 秒收尾宽限,最坏 600 秒,正好是一段执行的上限),整次运行
被中止。所以每句开始之前看这一段执行还剩多少时间(engine/deadline.py):给下游留出余量后不够再审一句,这句和后面的
都不再开始,记为未能反查(challenge_unreviewed 写明运行时长上限将至),节点照常收尾;够的话这句的时长上限按剩余时间
收紧,到点照样按查到的结果收尾。
都不再开始,记为未能反查,节点照常收尾;够的话这句的时长上限按剩余时间收紧,到点照样按查到的结果收尾。这两种在
时间线上都带 cause: run_deadline(没开始的另起一条 challenge_unreviewed,收紧后到点的是 challenge_limit 里单独的
一条),界面据此说运行时长上限将至,不叫人调大每句的上限,也不叫人去查模型和数据源。
整个反查连同存审稿记录、发日志都在一个 @task 里:接着跑时取 checkpoint 里的结果,不重复查库、不重复记账,时间线
上也不再发一遍。结果带着输入指纹,接着跑时改了审稿本身的配置就重审;同一次执行里的重试沿用已经审完的(见 _reviewed)。
"""
@@ -98,7 +99,7 @@ _REVIEWS_KEY = "challenge_reviews"
_LATE_CLAIM = "本次运行即将达到 {seconds:g} 秒的时长上限,为保证运行按时完成,这句未反查"
_LATE_REST = "本次运行即将达到 {seconds:g} 秒的时长上限,为保证运行按时完成,剩下 {count} 句未审"
#: 时长按剩余时间收紧过的句子触了时长上限:单记一类(_limit_words 说运行时长上限将至,不说「每句时长上限 90 秒」——
#: 那会叫人调大每句的上限,调大了只会更快撞线),challenge_limit 照记 timeout_s,界面认得
#: 那会叫人调大每句的上限,调大了只会更快撞线),challenge_limit 照记 timeout_s(老界面认得),另带 cause
_SQUEEZED = "squeezed"
CHALLENGE_SYSTEM = (
@@ -518,7 +519,7 @@ class _Review:
max_claims, max_queries, timeout_s, daily_max_usd
self.spec, self.masks, self.writer_model = spec, masks, writer_model
self.fetch = _fetcher()
#: 这一段执行的总时长(engine/deadline.py):有句子的时长按剩余时间收紧过时记下,那几句触顶时说明里写
#: 这一段执行的总时长(engine/deadline.py):有句子的时长按剩余时间收紧过、或者不再开始时记下,说明和日志里写
self.run_seconds: float | None = None
#: 这份报告目录里的查询编号:摘录里指向它们的写法标上「原报告」(mark_original)
self.queries = {a for a, e in catalog.items() if isinstance(e, dict) and e.get("kind") == "query"}
@@ -575,6 +576,9 @@ class _Review:
unreviewed: list[str] = [] # 没能反查的句子各自的原因(触顶的由 challenge_limit 说,不在这里)
partial: list[list[str]] = [] # 反查了、但该查的源有停用了的句子:各自没查成的源
late: tuple[str, str] | None = None # 运行时长上限将至:(每句的理由, 时间线上的原因),后面的句子都照这个记
overdue = 0 # 因此没开始的句数:时间线另起一条(带 cause),不混进 unreviewed
# 下游这一段里还要调模型的节点:各多留一份余量(engine/deadline.py)。图在运行里不变,算一次
after = deadline.models_after(ctx.run.spec, ctx.node.id)
def unreached(cand: judging.Candidate, rationale: str, why: str) -> None:
claims.append(_claim(cand, "inconclusive", rationale))
@@ -589,7 +593,8 @@ class _Review:
unreached(cand, f"反查未完成:{broken}", broken)
continue
if late is not None:
unreached(cand, *late)
claims.append(_claim(cand, "inconclusive", late[0]))
overdue += 1
continue
try:
own = claim_sources(cand.aliases, self.catalog, self.fetch)
@@ -613,19 +618,22 @@ class _Review:
continue
# 派出去之前看这一段执行还剩多少时间:给下游留出余量后不够再审一句,这句和后面的都不再开始、记为未能反查,
# 节点照常收尾,运行不被中止;够的话这句的时长上限按剩余时间收紧。在 @task 里定,接着运行时不再重新定
allot = deadline.allot(self.timeout_s, grace=verify_agent.SETTLE_GRACE_S)
allot = deadline.allot(self.timeout_s, grace=verify_agent.SETTLE_GRACE_S, after=after)
if not allot.start:
self.run_seconds = allot.seconds
late = (_LATE_CLAIM.format(seconds=allot.seconds),
_LATE_REST.format(seconds=allot.seconds, count=len(self.picked) - at))
unreached(cand, *late)
claims.append(_claim(cand, "inconclusive", late[0]))
overdue += 1
continue
if allot.tightened:
self.run_seconds = allot.seconds
# 收紧过的:核查 agent 到点自己发的 verify_limit 改说运行时长上限将至(deadline.relabel),不写收紧后的秒数
outcome = await run_verify_agent(
spec=self.spec, system=CHALLENGE_SYSTEM, task=task_text, sources=live,
output_schema=CHALLENGE_SCHEMA, max_queries=self.max_queries, timeout_s=allot.timeout_s,
daily_max_usd=self.daily_max_usd, emit=ctx.emit, run_id=ctx.run.run_id, node_id=ctx.node.id,
data_versions=run_versions(ctx.run))
daily_max_usd=self.daily_max_usd, emit=deadline.relabel(ctx.emit, allot), run_id=ctx.run.run_id,
node_id=ctx.node.id, data_versions=run_versions(ctx.run))
if outcome.error and not outcome.model:
broken = outcome.error # 连模型都没建成(没配好、接入出错):换一句也一样
model = model or outcome.model
@@ -655,9 +663,9 @@ class _Review:
unreviewed.append(why)
counts = {v: sum(1 for c in claims if c["verdict"] == v) for v in VERDICTS}
# 没能反查的句数:没派出去、派出去却没查成的(unreviewed 里各有原因),加上触顶时没能给出结论的。都记为无法判断,
# 可「无法判断 2 句」既可能是查了、数据不够,也可能是根本没查:审批卡要分开说
missed = len(unreviewed) + sum(skipped for _, skipped in hit.values())
# 没能反查的句数:没派出去、派出去却没查成的(unreviewed 里各有原因),运行时长上限将至没开始的,加上触顶时没能给出
# 结论的。都记为无法判断,可「无法判断 2 句」既可能是查了、数据不够,也可能是根本没查:审批卡要分开说
missed = len(unreviewed) + overdue + sum(skipped for _, skipped in hit.values())
artifact = await self._store({"report": self.report_id, "doc_artifact": self.doc_artifact, "claims": claims,
"model": model})
for limit, (touched, skipped) in hit.items():
@@ -665,13 +673,24 @@ class _Review:
message = f"已达上限({words}),{skipped} 句未反查" if limit == "daily_max_usd" else \
f"有 {touched} 句反查已达上限({words}),已按查到的结果收尾" + \
(f",其中 {skipped} 句未能给出结论" if skipped else "")
# 收紧过的那一组照记 timeout_s(老界面认得),另带 cause、这一段的总时长和触顶的句数:界面据此说运行时长
# 上限将至,不叫人调高每句的上限;同一节点里还有句子触了自己的上限时,两条也分得开
squeezed = {"cause": deadline.RUN_DEADLINE, "run_seconds": self.run_seconds, "claims": touched} \
if limit == _SQUEEZED else {}
ctx.emit(EventType.LOG, level="warn", code="challenge_limit",
limit="timeout_s" if limit == _SQUEEZED else limit, claims_skipped=skipped, message=message)
limit="timeout_s" if limit == _SQUEEZED else limit, claims_skipped=skipped, message=message,
**squeezed)
if unreviewed:
more = "等" if len(set(unreviewed)) > 1 else ""
ctx.emit(EventType.LOG, level="warn", code="challenge_unreviewed", claims_skipped=len(unreviewed),
reason=unreviewed[0],
message=f"反方审稿有 {len(unreviewed)} 句结论未能反查,已记为无法判断:{unreviewed[0]}{more}")
if overdue and late is not None:
# 运行时长上限将至没开始的另起一条,带 cause:不被前面句子的原因盖掉、藏进「等」里,界面也不会为它叫人去查
# 模型和数据源
ctx.emit(EventType.LOG, level="warn", code="challenge_unreviewed", cause=deadline.RUN_DEADLINE,
run_seconds=self.run_seconds, claims_skipped=overdue, reason=late[1],
message=f"反方审稿有 {overdue} 句结论未能反查,已记为无法判断:{late[1]}")
if partial:
names = list(dict.fromkeys(s for gone in partial for s in gone))
ctx.emit(EventType.LOG, level="warn", code="challenge_sources_lost", claims=len(partial), sources=names,
+31 -8
View File
@@ -24,7 +24,8 @@ SQL,就跟着差异说明进了审批卡、对照工件和目录日志。完
按发起审批时的目录各会往目录里写什么、不写的为什么(dual_check_facts.preview,和真写同一套判断);几个指标落到目录
同一项上时每种选择组合合起来会怎样写(preview_joint),放在对照工件的 catalog_joint 里。
盲算触顶、出错、交不出数,或者输入本身没法单独复算的指标按「无法复算」记缺口,发 dual_check_unchecked。运行这一段的
时长上限将至时不再开始新的盲算、开始的按剩余时间收紧时长(engine/deadline.py),没来得及的照此记,运行不被中止。
时长上限将至时不再开始新的盲算、开始的按剩余时间收紧时长(engine/deadline.py),没来得及的照此记,运行不被中止;
这几个另发一条 dual_check_unchecked,带 cause: run_deadline,界面据此说原因、给对的下一步。
**产出的状态。** 全部一致是 agreed;没有不一致、但有无法复算的:一个都没核成是 unchecked,核成了一部分是 partial;
有不一致、人裁决过的是 resolved。产出和台账都带 counts(一致、不一致、无法复算各几个)。
@@ -121,6 +122,9 @@ GAP_UNCHECKED = "指标「{name}」无法复算:{reason}"
#: 运行时长上限将至(engine/deadline.py):没来得及开始的盲算、时长按剩余时间收紧后到了点的盲算,都按无法复算记
LATE = "本次运行即将达到 {seconds:g} 秒的时长上限,为保证运行按时完成,未进行盲算"
SQUEEZED = "本次运行即将达到 {seconds:g} 秒的时长上限,盲算时长已按剩余时间收紧,到时未能完成"
#: 这几个指标另起的那条 dual_check_unchecked(带 cause)的原话
LATE_UNCHECKED = ("本次运行即将达到 {seconds:g} 秒的时长上限,为保证运行按时完成,双盲复算有 {count} 个指标未能复算:"
"{names},按无法复算计入缺口")
BLIND_SYSTEM = (
"你在做双盲复算:另一条流程已经算出了口径卡上的一个指标,你要在看不到它的 SQL 和结果的情况下,按业务定义独立算出"
@@ -418,7 +422,7 @@ async def _check(state: GraphState, ctx: NodeContext, targets: list[dict[str, An
if agreed and not any(r["status"] == DIFFERENT for r in rows):
ctx.emit(EventType.LOG, level="info", code="dual_check_agreed", metrics=agreed,
message=f"双盲复算一致:{agreed} 个指标两边的结果相同,已放行")
if unchecked := [r for r in rows if r["status"] == UNCHECKED]:
if unchecked := [r for r in rows if r["status"] == UNCHECKED and not r.get("cause")]:
# 结构上没法单独复算的(合并查询、遮罩列)以前一条 dual_check_* 日志都没有,时间线上看不出为什么没核成
parts = ";".join(f"「{r['name']}」({r.get('reason') or '原因不明'})" for r in unchecked[:5])
ctx.emit(EventType.LOG, level="warn", code="dual_check_unchecked",
@@ -426,6 +430,15 @@ async def _check(state: GraphState, ctx: NodeContext, targets: list[dict[str, An
for r in unchecked],
message=f"双盲复算未能核对 {len(unchecked)} 个指标:{parts}"
+ ("等" if len(unchecked) > 5 else "") + ",按无法复算计入缺口")
if late := [r for r in rows if r["status"] == UNCHECKED and r.get("cause")]:
# 运行时长上限将至没核成的(_blind 记下 cause)另起一条:界面据此说原因、给下一步,调高每项的上限只会更快撞线
seconds = late[0].get("run_seconds")
ctx.emit(EventType.LOG, level="warn", code="dual_check_unchecked", cause=late[0]["cause"], run_seconds=seconds,
metrics=[{"id": r["metric_id"], "name": r["name"], "reason": r.get("reason") or "原因不明"}
for r in late],
message=LATE_UNCHECKED.format(seconds=seconds or 0, count=len(late),
names="".join(f"「{r['name']}」" for r in late[:5])
+ ("等" if len(late) > 5 else "")))
return {"rows": rows, "results": results, "model": model, "same_model": same, "usage": usage}
@@ -730,22 +743,27 @@ async def _blind(ctx: NodeContext, jobs: dict[str, dict[str, Any]], options: _Op
catalogs[job["source"]] = await _catalog_text(sources, job["source"])
outcomes: dict[str, Any] = {}
squeezed: dict[str, float] = {} # 时长按剩余时间收紧过的盲算 → 这一段执行的总时长(到了点时说明里写)
late: dict[str, float] = {} # 运行时长上限将至、没开始的盲算 → 同上
# 下游这一段里还要调模型的节点(写报告、反方审稿……):各多留一份余量(engine/deadline.py)
after = deadline.models_after(ctx.run.spec, ctx.node.id)
async def one(key: str, job: dict[str, Any]) -> None:
# 给下游留出余量后不够再做一项:不再开始,交回一个没有模型的结果,按无法复算记——第一项就不够时,和核查模型
# 用不了一样,后面的不再一个个去试。在 @task 里定,重放时不再重新定
allot = deadline.allot(options.timeout_s, grace=verify_agent.SETTLE_GRACE_S)
allot = deadline.allot(options.timeout_s, grace=verify_agent.SETTLE_GRACE_S, after=after)
if not allot.start:
late[key] = allot.seconds
outcomes[key] = VerifyOutcome(result=None, cells=[], queries=[], rationale=None, usage={},
error=LATE.format(seconds=allot.seconds))
return
if allot.tightened:
squeezed[key] = allot.seconds
# 收紧过的:核查 agent 到点自己发的 verify_limit 改说运行时长上限将至(deadline.relabel),不写收紧后的秒数
outcomes[key] = await run_verify_agent(
spec=spec, system=BLIND_SYSTEM, task=_task_text(job, catalogs[job["source"]]), sources=[job["source"]],
output_schema=BLIND_SCHEMA, max_queries=options.max_queries, timeout_s=allot.timeout_s,
daily_max_usd=options.daily_max_usd, emit=ctx.emit, run_id=ctx.run.run_id, node_id=ctx.node.id,
data_versions=run_versions(ctx.run))
daily_max_usd=options.daily_max_usd, emit=deadline.relabel(ctx.emit, allot), run_id=ctx.run.run_id,
node_id=ctx.node.id, data_versions=run_versions(ctx.run))
keys = list(jobs)
await one(keys[0], jobs[keys[0]])
@@ -767,10 +785,14 @@ async def _blind(ctx: NodeContext, jobs: dict[str, dict[str, Any]], options: _Op
results[key] = dict(results[keys[0]])
continue
results[key] = _slim(outcome)
if key in squeezed and results[key]["limit"] == "timeout_s":
if key in late:
# 没开始的:带上 cause 和这一段的总时长,指标按它记无法复算时时间线另起一条(_check)
results[key].update(cause=deadline.RUN_DEADLINE, run_seconds=late[key])
elif key in squeezed and results[key]["limit"] == "timeout_s":
# 收紧过的时长到了点:照样按无法复算记(同触了每项时长上限),原因说运行时长上限将至。不留 limit,免得发
# dual_check_limit——它叫人调高每项的上限,调高了只会更快撞线
results[key].update(value=None, limit=None, reason=SQUEEZED.format(seconds=squeezed[key]))
results[key].update(value=None, limit=None, reason=SQUEEZED.format(seconds=squeezed[key]),
cause=deadline.RUN_DEADLINE, run_seconds=squeezed[key])
for k in usage:
usage[k] += outcome.usage.get(k) or 0
if outcome.model and model is None:
@@ -900,7 +922,8 @@ def _compare(row: dict[str, Any], metric: dict[str, Any], results: dict[str, Any
row.update(status=UNCHECKED, reason="盲算" + _limit_text(found["limit"], options))
return
if found.get("value") is None:
row.update(status=UNCHECKED, reason=found.get("reason") or "盲算没有交回可用的数值")
row.update(status=UNCHECKED, reason=found.get("reason") or "盲算没有交回可用的数值",
**{k: found[k] for k in ("cause", "run_seconds") if k in found}) # 运行时长上限将至(_blind)
return
values[inp["path"]] = found["value"]
outcome = recompute.recompute(metric, recompute.Changes(inputs=values), recompute.Sources(
+93 -3
View File
@@ -2,8 +2,10 @@
问 allot() 够不够、时长收紧到多少。
- 没有截止时刻(不经 runner 执行):原样放行,时长上限还是传进来的那个值
- 有截止时刻:先给下游留出余量(60 秒或这一段总时长的 10%,取大的)、再扣收尾宽限;比配置的时长短就收紧,收紧之后
不到 min(配置的时长, 收尾宽限) 就不再开始;配置的是不限时也按剩余时间收紧
- 有截止时刻:先给下游留出余量(一份取 60 秒或这一段总时长的 10% 中大的,这一段不到 240 秒时按 25%;下游这一段里每个
还要调模型的节点再各加一份,合计不超过一半)、再扣收尾宽限;比配置的时长短就收紧,收紧之后不到 min(配置的时长,
收尾宽限) 就不再开始;配置的是不限时也按剩余时间收紧
- 收紧过的核查到了点,核查 agent 自己发的 verify_limit 改说运行时长上限将至,带上 cause(relabel)
- runner 每一段执行都放一份:节点、节点里的 @task 都看得到,截止时刻就是 asyncio.timeout 的那一个;审批恢复后的那一段
是新的截止时刻;运行结束后清掉
"""
@@ -16,11 +18,12 @@ from langgraph.func import task
from sqlalchemy import select
from app.core.config import settings
from app.core.events import EventType
from app.db.base import SessionLocal
from app.db.models import Approval, Run
from app.engine import compiler, deadline
from app.engine.runner import run_manager
from app.engine.schema import NodeType
from app.engine.schema import GraphSpec, NodeType
class Clock:
@@ -100,12 +103,99 @@ def test_the_reserve_is_sixty_seconds_or_a_tenth(seconds, kept):
assert deadline.reserve(seconds) == kept
@pytest.mark.parametrize("seconds, kept", [(240, 60.0), (200, 50.0), (180, 45.0), (120, 30.0), (60, 15.0)])
def test_a_short_segment_keeps_a_quarter(seconds, kept):
"""这一段不到 240 秒:一份余量按总时长的 25% 算,不再固定 60 秒——不然 180 秒的一段,前面的节点用掉 60 秒之后核查
一项都开始不了,比没有这道检查时还差。"""
assert deadline.reserve(seconds) == kept
@pytest.mark.parametrize("seconds, after, kept", [
(600, 1, 120.0), (600, 2, 180.0), (600, 4, 300.0), (600, 9, 300.0), # 最多留一半
(1800, 2, 540.0), (180, 1, 90.0), (180, 3, 90.0),
])
def test_each_model_node_downstream_adds_a_share(seconds, after, kept):
"""下游这一段里每个还要调模型的节点再各加一份:双盲复算后面接着写报告、反方审稿时,只留 60 秒的话写报告的那次模型
调用一慢,运行照样被中止。合计不超过这一段总时长的一半,核查节点总还能开始几项。"""
assert deadline.reserve(seconds, after=after) == kept
async def test_the_reserve_follows_the_segment_length(clock):
"""时长上限调到 1800 秒:给下游留 180 秒,剩 300 秒时这一项最多 90 秒(300 − 180 − 30)。"""
async with deadline.segment(clock.now + 300, 1800):
assert deadline.allot(120, grace=30.0).timeout_s == pytest.approx(90.0)
async def test_downstream_model_nodes_take_their_share(clock):
"""下游还有写报告、反方审稿:剩 300 秒时给下游留 180 秒,这一项最多 90 秒(300 − 180 − 30);只剩 230 秒时不再开始。"""
async with deadline.segment(clock.now + 300, 600):
assert deadline.allot(120, grace=30.0, after=2) == deadline.Allot(True, 90.0, tightened=True, seconds=600)
assert deadline.allot(120, grace=30.0).timeout_s == 120, "下游没有调模型的节点:照旧只留一份"
async with deadline.segment(clock.now + 239, 600):
assert not deadline.allot(120, grace=30.0, after=2).start
async def test_a_short_segment_still_starts(clock):
"""单次运行时限只有 150 秒、前面用掉 50 秒:留 37.5 秒余量、30 秒收尾宽限,这一项还能查 32.5 秒,照样开始。"""
async with deadline.segment(clock.now + 100, 150):
answer = deadline.allot(90, grace=30.0)
assert answer.start and answer.timeout_s == pytest.approx(32.5) and answer.tightened
def spec_of(*nodes, edges) -> GraphSpec:
return GraphSpec.model_validate({"nodes": [node(nid, ntype) for nid, ntype in nodes],
"edges": [{"source": a, "target": b} for a, b in edges]})
def test_model_nodes_after_count_this_segment_only():
"""数下游这一段里还要调模型的节点:写报告、反方审稿、模型、Agent……各算一个;取数、整形、成果这类不算;人工审批之后
是新的一段(重新计时),不算;分支汇合到同一个节点只算一次。"""
spec = spec_of(("start", "input"), ("fetch", "tool"), ("card", "metrics"), ("dual", "dual_check"),
("write", "report"), ("chal", "challenge"), ("shape", "transform"), ("out", "output"),
edges=[("start", "fetch"), ("fetch", "card"), ("card", "dual"), ("dual", "write"),
("write", "chal"), ("chal", "shape"), ("shape", "out")])
assert deadline.models_after(spec, "dual") == 2
assert deadline.models_after(spec, "write") == 1
assert deadline.models_after(spec, "chal") == 0
gated = spec_of(("dual", "dual_check"), ("ask", "llm"), ("gate", "human"), ("write", "report"), ("out", "output"),
edges=[("dual", "ask"), ("dual", "gate"), ("gate", "write"), ("write", "out"), ("ask", "out")])
assert deadline.models_after(gated, "dual") == 1, "人工审批之后的写报告在下一段,不占这一段的时间"
forked = spec_of(("dual", "dual_check"), ("a", "agent"), ("b", "agent"), ("join", "report"),
edges=[("dual", "a"), ("dual", "b"), ("a", "join"), ("b", "join")])
assert deadline.models_after(forked, "dual") == 3
looped = spec_of(("chal", "challenge"), ("ask", "llm"), edges=[("chal", "ask"), ("ask", "chal")])
assert deadline.models_after(looped, "chal") == 1, "环回到自己:自己不算"
assert deadline.models_after(looped, "nowhere") == 0
async def test_a_squeezed_check_says_the_run_deadline_when_it_stops():
"""收紧过的核查到了点:核查 agent 发的 verify_limit 写的是收紧后的秒数(「时长上限 52.3 秒」),和任何设置都对不上。
relabel 包住 emit,把这一条改说运行时长上限将至、带上 cause 和这一段的总时长;别的事件、别的上限原样转发;没收紧
时原样交回 emit。"""
sent: list[tuple[str, dict]] = []
def emit(kind, **data):
sent.append((str(kind), data))
assert deadline.relabel(emit, deadline.Allot(True, 90)) is emit
assert deadline.relabel(emit, deadline.Allot(True, 90, seconds=600)) is emit, "有截止时刻但没收紧:照旧"
wrapped = deadline.relabel(emit, deadline.Allot(True, 52.3, tightened=True, seconds=600))
wrapped(EventType.LOG, level="warn", code="verify_limit", limit="timeout_s",
message="核查已达上限(时长上限 52.3 秒),已停止查询,按已有结果收尾")
wrapped(EventType.LOG, level="warn", code="verify_limit", limit="max_queries",
message="核查已达上限(最多查询 6 次),已停止查询,按已有结果收尾")
wrapped(EventType.TOOL_END, tool="db_query__shop", limit="timeout_s")
squeezed, queries, tool = sent
assert squeezed == (EventType.LOG, {
"level": "warn", "code": "verify_limit", "limit": "timeout_s", "cause": deadline.RUN_DEADLINE,
"run_seconds": 600,
"message": "核查已达上限(本次运行即将达到 600 秒的时长上限,时长已按剩余时间收紧到 52.3 秒),已停止查询,"
"按已有结果收尾"})
assert deadline.RUN_DEADLINE == "run_deadline"
assert queries[1]["message"] == "核查已达上限(最多查询 6 次),已停止查询,按已有结果收尾" and "cause" not in queries[1]
assert tool == (EventType.TOOL_END, {"tool": "db_query__shop", "limit": "timeout_s"})
async def test_the_segment_is_cleared_on_the_way_out(clock):
async with deadline.segment(clock.now + 600, 600):
assert deadline.current() == deadline.Segment(clock.now + 600, 600.0)
+131 -4
View File
@@ -5,6 +5,10 @@
运行时长上限将至,不说「每句时长上限 90 秒」;接着运行时取 checkpoint 里的审稿,不重审
- 双盲复算:同理,没来得及开始的盲算按无法复算记(照现有的路径记缺口);收紧过的时长到点也按无法复算记,不发提醒
调高上限的 dual_check_limit;审批恢复后重放,盲算不再执行、哪几项没来得及照旧
- 因运行时长上限将至没核成的,时间线另起一条、带 cause: run_deadline 和这一段的总时长(run_seconds):界面据此说清
原因、给对的下一步(调高核查自己的上限只会更快撞线,检查模型、数据源也无济于事),不和别的原因混在一条里、被「等」
盖掉;收紧过的核查 agent 自己发的 verify_limit 也改说运行时长上限将至
- 下游还有写报告这类要调模型的节点时,核查给它们多留一份余量:双盲复算后面接着写报告,运行不在写报告时被中止
- 时间充裕时(默认 600 秒的一段)时长上限照传原值:和没有这道检查时一样
两种钟:精确的计数用假的钟(每派出一项就把 allot() 看到的钟往前拨若干秒,真实只过去几毫秒,runner 的 asyncio.timeout
@@ -24,10 +28,11 @@ from app.core import artifact_store
from app.core.config import settings
from app.db.base import SessionLocal
from app.db.models import Approval, Run
from app.engine import deadline, verify_agent
from app.engine import compiler, deadline, verify_agent
from app.engine.nodes import challenge as challenge_mod
from app.engine.nodes import dual_check as dual_check_mod
from app.engine.runner import run_manager, verify_manifest
from app.engine.schema import NodeType
from app.main import app
from app.providers.mock_model import MockChatModel
from tests.fixtures.dual_check_shop import WEEK, Blind, add_source, build
@@ -116,6 +121,7 @@ async def test_five_claims_with_time_for_two_review_two(monkeypatch, shop):
[warn] = await logs(row.id, "challenge_unreviewed")
assert warn["claims_skipped"] == 3 and warn["reason"] == LATE_REST
assert warn["cause"] == deadline.RUN_DEADLINE and warn["run_seconds"] == 600, "界面按 cause 说原因、给下一步"
assert warn["message"] == f"反方审稿有 3 句结论未能反查,已记为无法判断:{LATE_REST}"
assert await logs(row.id, "challenge_limit") == [], "没有哪一句触了自己的上限"
[done] = await logs(row.id, "challenge_done")
@@ -137,13 +143,52 @@ async def test_a_claim_squeezed_by_the_deadline_says_so(monkeypatch, shop):
[limit] = await logs(row.id, "challenge_limit")
assert limit["limit"] == "timeout_s" and limit["claims_skipped"] == 0
assert limit["cause"] == deadline.RUN_DEADLINE and limit["run_seconds"] == 600 and limit["claims"] == 1, \
"界面认 cause:不叫人调高每句的上限"
assert "本次运行即将达到 600 秒的时长上限" in limit["message"] and "收紧" in limit["message"]
assert "每句时长上限" not in limit["message"]
out = await output_of(row.id, "chal")
assert out["counts"]["refuted"] == 1, "收紧后按查到的结果收尾:反方给的结论照常采信"
assert out["unreviewed"] == 3
[warn] = await logs(row.id, "challenge_unreviewed")
assert warn["claims_skipped"] == 3
assert warn["claims_skipped"] == 3 and warn["cause"] == deadline.RUN_DEADLINE
async def test_the_run_deadline_gets_its_own_line(monkeypatch, shop):
"""第 1 句反查一条查询都没执行成功(另一个原因),第 3 句起来不及审:时间线两条 challenge_unreviewed,各说各的原因。
运行时长上限将至的那条带 cause,不被前一句的原因盖掉、藏进「等」里——界面也就不会为了它叫人去查模型和数据源。"""
plays = dict(FIVE_PLAYS)
plays["居各区域之首"] = Play([], "weakened", "没有查询。")
models = Models(monkeypatch, shop, writer=(FIVE,), plays=plays)
Ticker(monkeypatch, challenge_mod, step=250)
row = await finish(graph(shop))
assert row.status == "succeeded", row.error
assert models.reviewed() == ORDER[:2]
out = await output_of(row.id, "chal")
assert out["unreviewed"] == 4 and out["counts"]["inconclusive"] == 4
other, late = await logs(row.id, "challenge_unreviewed")
assert other["claims_skipped"] == 1 and other["reason"] == "反查没有执行成功任何查询" and "cause" not in other
assert other["message"] == "反方审稿有 1 句结论未能反查,已记为无法判断:反查没有执行成功任何查询"
assert late["claims_skipped"] == 3 and late["reason"] == LATE_REST
assert late["cause"] == deadline.RUN_DEADLINE and late["run_seconds"] == 600
assert late["message"] == f"反方审稿有 3 句结论未能反查,已记为无法判断:{LATE_REST}"
async def test_its_own_limit_and_the_run_deadline_are_told_apart(monkeypatch, shop):
"""第 1 句触了自己的每句时长上限(90 秒,没收紧),第 2 句的时长按剩余时间收紧后也到了点:两条 challenge_limit 都是
timeout_s,只有收紧的那条带 cause——时间线上不是两行一样的「反方审稿已达时长上限」,叫人调高每句上限的只有第 1 条。"""
Models(monkeypatch, shop, writer=(FIVE,), plays=FIVE_PLAYS)
ticker = Ticker(monkeypatch, challenge_mod, step=440, hit=lambda timeout: True)
row = await finish(graph(shop))
assert row.status == "succeeded", row.error
first, second = ticker.timeouts
assert first == 90 and second < 90, ticker.timeouts
own, squeezed = await logs(row.id, "challenge_limit")
assert own["limit"] == squeezed["limit"] == "timeout_s"
assert "cause" not in own and "每句时长上限 90 秒" in own["message"]
assert squeezed["cause"] == deadline.RUN_DEADLINE and squeezed["run_seconds"] == 600 and squeezed["claims"] == 1
assert "每句时长上限" not in squeezed["message"]
async def test_no_time_at_all_reviews_nothing_but_finishes(monkeypatch, shop):
@@ -156,7 +201,7 @@ async def test_no_time_at_all_reviews_nothing_but_finishes(monkeypatch, shop):
out = await output_of(row.id, "chal")
assert out["counts"] == {"stands": 0, "weakened": 0, "refuted": 0, "inconclusive": 5} and out["unreviewed"] == 5
[warn] = await logs(row.id, "challenge_unreviewed")
assert warn["claims_skipped"] == 5 and "剩下 5 句未审" in warn["reason"]
assert warn["claims_skipped"] == 5 and "剩下 5 句未审" in warn["reason"] and warn["cause"] == deadline.RUN_DEADLINE
assert row.output["_issuance"]["tier"] == "formal"
@@ -215,7 +260,13 @@ async def test_a_slow_review_no_longer_aborts_the_run(monkeypatch, shop):
assert out["unreviewed"] == len(late)
[warn] = await logs(row.id, "challenge_unreviewed")
assert warn["claims_skipped"] == len(late) and f"剩下 {len(late)} 句未审" in warn["reason"]
assert "本次运行即将达到 7 秒的时长上限" in warn["reason"]
assert "本次运行即将达到 7 秒的时长上限" in warn["reason"] and warn["cause"] == deadline.RUN_DEADLINE
# 核查 agent 到点自己也发 verify_limit:时长收紧过的那几句说运行时长上限将至,写的秒数不会是一个和设置对不上的数
for said in await logs(row.id, "verify_limit"):
if said.get("cause") == deadline.RUN_DEADLINE:
assert "本次运行即将达到 7 秒的时长上限,时长已按剩余时间收紧到" in said["message"] and said["run_seconds"] == 7
else:
assert "时长上限 1.5 秒" in said["message"] or said["limit"] != "timeout_s", said
# --------------------------------------------------------------------------
@@ -289,10 +340,31 @@ async def test_five_metrics_with_time_for_two_check_two(monkeypatch, orders):
[unchecked] = await logs(row.id, "dual_check_unchecked")
assert unchecked["metrics"] == [{"id": m, "name": NAMES[m], "reason": LATE_BLIND} for m in ("buyers", "top", "low")]
assert unchecked["cause"] == deadline.RUN_DEADLINE and unchecked["run_seconds"] == 600, "界面按 cause 说原因、给下一步"
assert unchecked["message"] == ("本次运行即将达到 600 秒的时长上限,为保证运行按时完成,双盲复算有 3 个指标未能复算:"
"「付款客户数」「最大单笔」「最小单笔」,按无法复算计入缺口")
assert await logs(row.id, "dual_check_limit") == []
assert (await verify_manifest(row.id))["ok"] is True
async def test_blind_checks_cut_by_the_deadline_get_their_own_line(monkeypatch, orders):
"""「订单数」的盲算没查到这个数(另一个原因),后 3 项来不及开始:时间线两条 dual_check_unchecked。运行时长上限将至的
那条带 cause、只列那 3 项,界面据此说原因、给下一步;另一条照旧,不带 cause。"""
Blind(monkeypatch, orders, {name: plan for name, plan in PLANS.items() if name != "订单数"})
Ticker(monkeypatch, verify_agent, step=260)
row = await dual_run(dual_graph(orders))
assert row.status == "succeeded", row.error
out = await output_of(row.id, "dual")
assert unchecked_reasons(out) == {"orders": "盲算没有查到这个数", "buyers": LATE_BLIND, "top": LATE_BLIND,
"low": LATE_BLIND}
other, late = await logs(row.id, "dual_check_unchecked")
assert other["metrics"] == [{"id": "orders", "name": "订单数", "reason": "盲算没有查到这个数"}] and "cause" not in other
assert other["message"] == "双盲复算未能核对 1 个指标:「订单数」(盲算没有查到这个数),按无法复算计入缺口"
assert late["cause"] == deadline.RUN_DEADLINE and late["run_seconds"] == 600
assert [m["id"] for m in late["metrics"]] == ["buyers", "top", "low"]
assert all(m["reason"] == LATE_BLIND for m in late["metrics"])
async def test_no_time_for_even_the_first_blind_check(monkeypatch, orders):
"""一开始就只剩 40 秒:第一项都不开始(后面的也就不再一个个去试),五项都按无法复算记,节点状态是一个都没核成。"""
blind = Blind(monkeypatch, orders, PLANS)
@@ -323,6 +395,28 @@ async def test_a_blind_check_squeezed_by_the_deadline_says_so(monkeypatch, order
assert "每项时长上限" not in reasons["orders"]
assert {reasons[m] for m in ("buyers", "top", "low")} == {LATE_BLIND}
assert await logs(row.id, "dual_check_limit") == []
[unchecked] = await logs(row.id, "dual_check_unchecked")
assert unchecked["cause"] == deadline.RUN_DEADLINE and [m["id"] for m in unchecked["metrics"]] == \
["orders", "buyers", "top", "low"], "收紧后到点的和没来得及开始的是同一个原因:一条说完"
def dual_report_graph(source: str) -> dict:
"""取数 → 口径卡 → 双盲复算 → 写报告 → 成果:Copilot 指南里的惯常接法,双盲复算下游还有一次写报告的模型调用。"""
nodes = dual_graph(source)["nodes"]
nodes.insert(4, node("write", "report", "写周报", instructions="写周报"))
return {"nodes": nodes, "edges": [{"source": a["id"], "target": b["id"]} for a, b in zip(nodes, nodes[1:])]}
@pytest.mark.parametrize("build, low, high", [(dual_graph, 100, 110), (dual_report_graph, 40, 50)])
async def test_a_report_downstream_gets_its_share(monkeypatch, orders, build, low, high):
"""第 3 项开始时只剩约 200 秒:下游只有成果节点时留 60 秒余量,这一项还能查 110 秒左右;下游还要写报告时再给它留
60 秒,这一项只查 50 秒左右——双盲复算用到最后一刻的话,写报告那次模型调用一慢,整次运行照样被中止。"""
Blind(monkeypatch, orders, PLANS)
ticker = Ticker(monkeypatch, verify_agent, step=200)
row = await dual_run(build(orders))
assert row.status == "succeeded", row.error
first, second, third = ticker.timeouts
assert first == second == 120 and low < third < high, ticker.timeouts
async def test_resuming_after_the_review_replays_the_blind_checks(monkeypatch, orders):
@@ -383,3 +477,36 @@ async def test_slow_blind_checks_no_longer_abort_the_run(monkeypatch, orders):
"一项接一项:来得及的在前"
[unchecked] = await logs(row.id, "dual_check_unchecked")
assert {m["id"] for m in unchecked["metrics"]} == set(reasons)
async def test_a_slow_report_after_the_blind_checks_still_finishes(monkeypatch, orders):
"""真的钟:双盲复算 → 写报告 → 成果,这一段只有 10 秒。盲算一项接一项,每项都查到时长用完(查不完的查询);写报告要
3 秒。双盲复算给下游的写报告多留了一份余量(4 秒),收尾之后报告写得完,运行照常完成——只留一份(2 秒)的话,盲算
用到最后,报告写到一半整次运行就被中止。几项来得及随机器快慢变,只断言不变的。"""
monkeypatch.setattr(settings, "max_run_seconds", 10)
monkeypatch.setattr(deadline, "RESERVE_S", 2.0)
monkeypatch.setattr(verify_agent, "SETTLE_GRACE_S", 0.4)
monkeypatch.setattr(dual_check_mod, "BLIND_CONCURRENCY", 1)
async def slow_model(session, spec):
return MockChatModel(model_name="slow-verify", latency=0.1), "slow-verify"
monkeypatch.setattr(verify_agent, "get_chat_model", slow_model)
endless = [f"SELECT COUNT(*) AS blind_v FROM orders WHERE id > {i}" for i in range(300)]
Blind(monkeypatch, orders, {name: (endless, "blind_v") for name in NAMES.values()})
wrote: list[float] = []
async def slow_report(state, ctx):
await asyncio.sleep(3.0)
wrote.append(asyncio.get_running_loop().time())
return {}
monkeypatch.setitem(compiler.RUNNERS, NodeType.REPORT, slow_report)
row = await dual_run(dual_report_graph(orders),
options={"verify": {"dual_check": {"timeout_s": 3, "max_queries": None}}})
assert row.status == "succeeded", row.error
assert "时长上限,已中止" not in (row.error or "")
assert len(wrote) == 1, "报告写完了"
out = await output_of(row.id, "dual")
reasons = unchecked_reasons(out)
assert any("本次运行即将达到 10 秒的时长上限" in why for why in reasons.values()), reasons