期 4 复核修正(第 1 轮,后端):预告组合先算个数再列、上限按每项算,核查 agent 的两个日志代码归类,未能反查的句数守住不多于无法判断

- preview_joint 以前先把每一项的全部组合列成列表才和 JOINT_LIMIT 比,二十个指标落到同一项上就是几十亿种,在服务进程里(asyncio.to_thread)能把整个服务拖死;现在先按 ∏(1+k)−1−Σk 算出个数(超过上限即停),超了整项跳过、不列。上限也改成每一项各算各的:以前几项共用一份额度,前一项用掉以后,后面没超上限的项也被悄悄跳过,和文档说的「超过的那一项不预告」对不上。补两条测试:二十个指标落到同一项上时一个组合都不多列,两项各 72 种都预告
- review.py 认得核查 agent(双盲复算、反方审稿共用)发的两条:verify_unpriced 同 judge_unpriced 只是配置提醒,不再叫复核模型、给答案加降级说明;verify_limit 记成它自己的 kind(步数上限只有它说),不再落成笼统的 warn
- 反方审稿测试补上「未能反查的句数不多于无法判断」的断言(审批卡上的两个数靠这个约定对得上),模块说明改写准:触顶没给出结论的可能查过几次,原因里也有报告未引用数据库查询
- 双盲复算夹具加一种组合 time_rejected:业务日期被驳回过的那一列由退款率采纳(被挡下,带着写到哪),销售额采纳另一列;服务端先归拢、说法不同一项都不写(夹具测试按 record_choices 真写核对),审批卡没有合并预告时的推断按它检查
This commit is contained in:
ThearchyHelios
2026-10-06 09:31:00 +08:00
parent 11d44ebc8a
commit 62b9361d88
7 changed files with 259 additions and 25 deletions
+21 -9
View File
@@ -74,8 +74,9 @@ ASPECT_LABEL = {
_VERB = {"count": "计数", "sum": "求和", "avg": "求平均"}
#: 写入时版本对不上(别人刚改过这张表)最多重试几次
_RETRIES = 2
#: 同一次裁决里几个指标落到同一项上时,最多预告多少种组合(preview_joint)。每个指标两边都落在这一项上时是
#: 3 的指标数次方:四个指标 72 种还在里面,再多的那一项不预告,审批卡回到按其余指标当前的选择有条件地说
#: 同一次裁决里几个指标落到同一项上时,这一项最多预告多少种组合(preview_joint,每一项各算各的,不共用额度)。
#: 每个指标两边都落在这一项上时近于 3 的指标数次方:四个指标 72 种还在里面,再多的那一项不预告,审批卡回到按其余指标
#: 当前的选择有条件地说。组合数先算出来再比(_combos),不把组合列出来比:二十个指标就是几十亿种
JOINT_LIMIT = 128
#: 无效记录的取值:被排除的取值(或码值表里它的意思)里有这些词,才算在排除无效记录
@@ -820,6 +821,18 @@ def _joint_outcome(notes: Mapping[str, Any], members: list[Fact], tables: Mappin
return result
def _combos(sides: Sequence[int], limit: int) -> int:
"""几个指标落到同一项上时有几种组合(至少两个指标落上来的):每个指标要么没落上来、要么落上来的那几边之一,
∏(1 + k) − 1 − Σk(k 是这个指标落到这一项上的边数,减去一个都没有、只有一个的)。连乘到超过 limit 能确定的地方
就停,返回 limit + 1:只用来和上限比,不需要真算出几十亿。"""
total, cap = 1, limit + 1 + sum(sides)
for k in sides:
total *= 1 + k
if total > cap:
return limit + 1
return total - 1 - sum(sides)
def preview_joint(candidates: Sequence[Mapping[str, Any]],
catalogs: Mapping[str, tuple[Mapping[str, Any] | None, Mapping[str, Mapping[str, Any]]]]
) -> list[dict[str, Any]]:
@@ -829,8 +842,11 @@ def preview_joint(candidates: Sequence[Mapping[str, Any]],
candidates 照节点交给真写的先后(不一致的指标的先后),带 metric_id,不带 choice;catalogs 是 {数据源名: (表结构,
{表名: notes})},同 preview 的两个参数。每个指标的两边先各按 derive 认出落到哪一项(同 preview),落到同一项上的
几个指标,每种「谁选了哪一边」的组合按 _joint_outcome 算一遍:真写时这几个指标就是这样被归拢、写或不写的。组合超过
JOINT_LIMIT 的那一项整项不预告。永远不抛:算不出来的那一项不预告。"""
几个指标,每种「谁选了哪一边」的组合按 _joint_outcome 算一遍:真写时这几个指标就是这样被归拢、写或不写的。
上限按每一项算:一项的组合数(_combos,先算个数,不先列出来)超过 JOINT_LIMIT 的,这一项整项不预告,别的项照常
预告,几项之间不共用额度。一项不超过 JOINT_LIMIT 种、每个指标最多落到两项上,总的计算量和指标数成正比。
永远不抛:算不出来的那一项不预告。"""
from itertools import product
landed: dict[tuple[str, str, str], dict[int, list[tuple[str, Fact]]]] = {}
@@ -844,16 +860,12 @@ def preview_joint(candidates: Sequence[Mapping[str, Any]],
if fact is not None:
landed.setdefault((fact.source, fact.table, fact.item), {}).setdefault(index, []).append((side, fact))
out: list[dict[str, Any]] = []
budget = JOINT_LIMIT
for (source, table, item), by_metric in landed.items():
if len(by_metric) < 2:
if len(by_metric) < 2 or _combos([len(sides) for sides in by_metric.values()], JOINT_LIMIT) > JOINT_LIMIT:
continue
options = [[None, *sides] for _, sides in sorted(by_metric.items())]
combos = [[c for c in combo if c is not None] for combo in product(*options)]
combos = [c for c in combos if len(c) >= 2]
if len(combos) > budget:
continue
budget -= len(combos)
schema, notes = catalogs.get(source, (None, {}))
tables = schema.get("tables") if isinstance(schema, Mapping) else None
try:
+4 -3
View File
@@ -23,9 +23,10 @@
5. 处置:被推翻的按 on_refuted 处置。degrade(默认)在产出里写 issuance_gaps,成果节点的出具契约照单收进缺口、
出具降档(nodes/io.py 的 _upstream_issuance_gaps,所以反方审稿要接在成果节点上游,画布校验会提醒);fail 判节点
失败;flag 只记录。产出和审批卡里的 on_refuted 是实际生效的那一种(只有 degrade / flag,见 _finish);degrades
说被推翻的到底会不会让出具降档(按降档处置、又接在带出具契约的成果节点上游),unreviewed 是没能反查的句数(核查
模型不可用、触顶、数据源不在了……这几句记为无法判断,但根本没查成)。审批卡(human.py 的 _reviewed_reports)附上
审稿概况,据这两项说「出具会降档」「N 句没能反查」
说被推翻的到底会不会让出具降档(按降档处置、又接在带出具契约的成果节点上游),unreviewed 是未能反查的句数(核查
模型不可用、触顶没给出结论、数据源不在了、报告没有引用数据库查询……这几句记为无法判断,但反查没有完成,触顶的
可能已经查过几次)。每一句都记为无法判断,所以 unreviewed 不会大于 counts 里的 inconclusive。审批卡(human.py 的
_reviewed_reports)附上审稿概况,据这两项说「出具会降档」「N 句未能反查」
上限:句数、每句查询次数、每句时长、全局每日金额(核查和裁判各记各的账,engine/spend.py)。都按发起时快照的核查
设置(run.options["verify"]),节点上写了的盖过它:审批停下期间改了设置,恢复、接着跑仍按发起时的那份。没有快照
+5
View File
@@ -114,6 +114,8 @@ WARN_CODES: dict[str, tuple[str, str]] = {
# 双盲复算、反方审稿的核查模型和被核查的模型相同:同 judge_same_model,只关乎独立性,结论本身没受影响
"dual_check_same_model": ("dual_check_same_model", BENIGN),
"challenge_same_model": ("challenge_same_model", BENIGN),
# 核查 agent(engine/verify_agent.py,双盲复算、反方审稿共用)估不出金额:同 judge_unpriced,配置上的提醒
"verify_unpriced": ("verify_unpriced", BENIGN),
# 核查没做完、没通过:盲算触顶、两路不一致、指标无法复算;反方审稿触顶、推翻了结论、有句子没能反查、只查了
# 部分数据源。答案照常交付(出具契约另按缺口降档),是缺口不是断裂,按各自的 kind 记,界面说得清是哪一种
"dual_check_limit": ("dual_check_limit", DEGRADED),
@@ -123,6 +125,9 @@ WARN_CODES: dict[str, tuple[str, str]] = {
"challenge_refuted": ("challenge_refuted", DEGRADED),
"challenge_unreviewed": ("challenge_unreviewed", DEGRADED),
"challenge_sources_lost": ("challenge_sources_lost", DEGRADED),
# 核查 agent 自己触顶、按已有结果收尾:多半和节点自己的 *_limit 说的是同一次触顶,照样是缺口,记成它自己的 kind,
# 不落成笼统的 warn——步数上限(max_steps)只有它说
"verify_limit": ("verify_limit", DEGRADED),
}
# 这些 kind 重跑一次大概率能好:原因说得清,且不是「问题本身没问明白」。
+10 -7
View File
@@ -411,7 +411,8 @@ async def test_a_verdict_without_any_query_is_not_taken(monkeypatch, shop):
assert east["verdict"] == "inconclusive" and "「经受住」" in east["rationale"]
[warn] = await logs(row.id, "challenge_unreviewed")
assert warn["claims_skipped"] == 2 and "没有执行成功任何查询" in warn["reason"]
assert out["unreviewed"] == 2, "一条查询都没执行成功:两句都是没能反查,不是查过了判断不了"
assert out["unreviewed"] == 2, "一条查询都没执行成功:两句都是未能反查,不是查过了判断不了"
assert out["unreviewed"] <= out["counts"]["inconclusive"], "未能反查的每一句都记为无法判断,审批卡上两个数对得上"
async def test_a_refutation_that_cites_none_of_its_queries_is_not_taken(monkeypatch, shop):
@@ -428,7 +429,7 @@ async def test_a_refutation_that_cites_none_of_its_queries_is_not_taken(monkeypa
assert south["verdict"] == "inconclusive" and len(south["queries"]) == 1 and south["rationale_doc"] is None
assert "没有引用" in south["rationale"] and "扣除退款后华南仍低于华东。" in south["rationale"]
assert east["verdict"] == "stands", "「经受住」不要求引用:它不改变出具"
assert await logs(row.id, "challenge_unreviewed") == [], "查过了,不算没能反查"
assert await logs(row.id, "challenge_unreviewed") == [], "查过了,不算未能反查"
assert out["unreviewed"] == 0
@@ -537,7 +538,7 @@ async def test_a_claim_with_part_of_its_sources_gone_says_so(monkeypatch, tmp_pa
[lost] = await logs(row.id, "challenge_sources_lost")
assert lost["level"] == "warn" and lost["claims"] == 2 and lost["sources"] == [other]
assert f"「{other}」" in lost["message"] and "2 句" in lost["message"]
assert await logs(row.id, "challenge_unreviewed") == [], "两句都查过了,不算没能反查"
assert await logs(row.id, "challenge_unreviewed") == [], "两句都查过了,不算未能反查"
finally:
await engines.invalidate(other_id)
@@ -671,8 +672,9 @@ async def test_the_time_limit_applies_to_each_claim_and_still_settles(monkeypatc
assert south["verdict"] == "inconclusive" and 1 <= len(south["queries"]) < 20
limits = await logs(row.id, "challenge_limit")
assert [(lm["limit"], lm["claims_skipped"]) for lm in limits] == [("timeout_s", 0)]
# 触顶但按查到的结果收了尾的那句,判断是反方自己给的,不算没能反查;另一句一条查询都没执行,算
assert (await output_of(row.id, "chal"))["unreviewed"] == 1
# 触顶但按查到的结果收了尾的那句,判断是反方自己给的,不算未能反查;另一句一条查询都没执行,算
out = await output_of(row.id, "chal")
assert out["unreviewed"] == 1 and out["counts"]["inconclusive"] == 2
async def test_the_daily_cap_stops_the_remaining_claims(monkeypatch, shop):
@@ -692,7 +694,8 @@ async def test_the_daily_cap_stops_the_remaining_claims(monkeypatch, shop):
"第一句派出去就撞上上限、一次模型都没调用,和后面不再派出的句子是同一回事,同一种说法"
[limit] = await logs(row.id, "challenge_limit")
assert limit["limit"] == "daily_max_usd" and limit["claims_skipped"] == 2
assert out["unreviewed"] == 2, "撞上每日金额上限没反查的,也是没能反查"
assert out["unreviewed"] == 2, "撞上每日金额上限没反查的,也是未能反查"
assert out["unreviewed"] <= out["counts"]["inconclusive"]
assert len(await logs(row.id, "verify_limit")) == 1, "第一句撞上每日上限之后,后面的句子不再派 agent 去试"
assert row.output["_issuance"]["tier"] == "formal"
@@ -1014,7 +1017,7 @@ async def test_an_unavailable_model_leaves_the_claims_inconclusive(monkeypatch,
# 时间线上看得出根本没审成:不是只有一条「审了 2 句」
[warn] = await logs(row.id, "challenge_unreviewed")
assert warn["level"] == "warn" and warn["claims_skipped"] == 2 and "核查模型不可用" in warn["reason"]
assert out["unreviewed"] == 2
assert out["unreviewed"] == 2 and out["unreviewed"] <= out["counts"]["inconclusive"]
[done] = await logs(row.id, "challenge_done")
assert "2 句未能反查" in done["message"]
+40
View File
@@ -654,6 +654,46 @@ def test_too_many_combinations_are_left_to_the_timeline():
assert len(facts.preview_joint(picked[:3], {"shop": (SCHEMA, {})})) == 3 ** 3 - 1 - 3 * 2
def test_combinations_are_counted_before_they_are_listed(monkeypatch):
"""二十个指标都落到同一项上:组合数是 3 的二十次方量级,得先算出个数、超了就跳过,不能先列出来再比上限——
列一遍就要几十 GB 内存,在服务进程里算(asyncio.to_thread)会把整个服务拖死。预告时从 itertools.product 取了
多少个组合记下来:这一项不预告,取的不该超过上限"""
import itertools
from app.engine import dual_check_facts as facts
drawn = 0
real = itertools.product
def guarded(*iterables, **kw):
nonlocal drawn
for combo in real(*iterables, **kw):
drawn += 1
# 红的时候不至于真把内存吃光:取得比上限多出一截就停
assert drawn <= facts.JOINT_LIMIT + 64, "列出的组合远多于上限:没有先算个数"
yield combo
stem = candidate(BY_ORDER_DATE, RIGHT, "a")
picked = [{**stem, "metric": f"指标{i}", "metric_id": f"m{i}"} for i in range(20)]
monkeypatch.setattr(itertools, "product", guarded)
assert facts.preview_joint(picked, {"shop": (SCHEMA, {})}) == []
assert drawn <= facts.JOINT_LIMIT
def test_each_item_has_its_own_limit():
"""上限按每一项算,不是几项共用一份额度:两个数据源的业务日期各有四个指标落上来(各 72 种组合),两项都预告。
以前共用 JOINT_LIMIT 的额度,前一项用掉 72 种,后一项明明没超上限也被悄悄跳过"""
from app.engine import dual_check_facts as facts
picked = [{**candidate(BY_ORDER_DATE, RIGHT, "a", metric=f"指标{i}", source=source), "metric_id": f"{source}-{i}"}
for source in ("shop", "outlet") for i in range(4)]
joint = facts.preview_joint(picked, {"shop": (SCHEMA, {}), "outlet": (SCHEMA, {})})
per = 3 ** 4 - 1 - 4 * 2
assert per <= facts.JOINT_LIMIT < 2 * per
assert sorted({e["source"] for e in joint}) == ["outlet", "shop"]
assert len(joint) == 2 * per
@pytest.mark.parametrize("choices", [("a", "b"), ("b", "b"), ("b", "a"), ("a", None)])
async def test_the_joint_preview_is_what_gets_written(shop, choices):
"""预告和真写逐字一致:按这组选择,各指标的预告(单独落一项的看 preview,几个落到同一项的看 preview_joint)说写的,
+9 -6
View File
@@ -2,8 +2,10 @@
没登记的代码一律按 degraded、kind 记成笼统的 warn:时间线复核和问数据页说不清是哪一种缺口,模型相同这种只关乎独立性、
不影响结论的提醒也会被当成缺口去叫一次复核模型。这里守住两类:
- 模型相同(dual_check_same_model、challenge_same_model)同 judge_same_model,是配置上的提醒,不进信号;
- 触顶、不一致、无法复算、被推翻、没能反查、只查了部分数据源是缺口,按各自的 kind 记成 degraded。
- 模型相同(dual_check_same_model、challenge_same_model)同 judge_same_model,核查模型不在价格目录中(verify_unpriced)
同 judge_unpriced,是配置上的提醒,不进信号;
- 触顶、不一致、无法复算、被推翻、未能反查、只查了部分数据源是缺口,按各自的 kind 记成 degraded。核查 agent 自己触顶
(verify_limit,双盲复算、反方审稿都会发)也是:步数上限只有它说。
"""
from __future__ import annotations
@@ -19,16 +21,17 @@ def warn(message: str, code: str) -> dict:
return {"type": str(EventType.LOG), "node_id": "gate", "data": {"level": "warn", "message": message, "code": code}}
@pytest.mark.parametrize("code", ["dual_check_same_model", "challenge_same_model"])
def test_same_model_reminders_do_not_trigger_review(code):
"""和 judge_same_model 一样:配置上的提醒,答案本身没受影响,不值得为它叫一次复核模型"""
@pytest.mark.parametrize("code", ["dual_check_same_model", "challenge_same_model", "verify_unpriced"])
def test_configuration_reminders_do_not_trigger_review(code):
"""和 judge_same_model、judge_unpriced 一样:配置上的提醒,答案本身没受影响,不值得为它叫一次复核模型、
在答案上多一句降级说明"""
assert rv.WARN_CODES[code] == (code, rv.BENIGN)
assert rv.scan([warn("核查模型与原流程相同(「mx-1」)", code)], OK_OUTPUT) == []
@pytest.mark.parametrize("code", [
"dual_check_limit", "dual_check_disagreed", "dual_check_unchecked",
"challenge_limit", "challenge_refuted", "challenge_unreviewed", "challenge_sources_lost",
"challenge_limit", "challenge_refuted", "challenge_unreviewed", "challenge_sources_lost", "verify_limit",
])
def test_verify_gaps_are_degraded_signals_of_their_own_kind(code):
"""核查没做完、没通过:答案能用,但有缺口要说清是哪一种(kind 是代码本身,不是笼统的 warn)"""
+170
View File
@@ -1471,6 +1471,176 @@
}
}
]
},
"time_rejected": {
"gmv": null,
"refund_rate": "time",
"choices": {
"gmv": "a",
"refund_rate": "b"
},
"notes": {
"orders": {
"business_date": {
"value": {
"column": "created_at"
},
"source": "llm",
"status": "rejected"
}
}
},
"written": [],
"catalog": {
"gmv": {
"a": {
"write": true,
"source": "shop",
"table": "orders",
"item": "business_date",
"value": {
"column": "paid_at"
}
},
"b": {
"write": false,
"source": "shop",
"table": "orders",
"item": "business_date",
"value": {
"column": "created_at"
},
"reason": "「orders」的业务日期「created_at」曾被人工驳回,与这次指标「销售额」的选择相反,未写入。如需更改,请到数据目录中修改"
}
},
"refund_rate": {
"a": {
"write": true,
"source": "shop",
"table": "orders",
"item": "business_date",
"value": {
"column": "paid_at"
}
},
"b": {
"write": false,
"source": "shop",
"table": "orders",
"item": "business_date",
"value": {
"column": "created_at"
},
"reason": "「orders」的业务日期「created_at」曾被人工驳回,与这次指标「退款率」的选择相反,未写入。如需更改,请到数据目录中修改"
}
}
},
"joint": [
{
"source": "shop",
"table": "orders",
"item": "business_date",
"members": [
{
"metric_id": "gmv",
"choice": "a"
},
{
"metric_id": "refund_rate",
"choice": "a"
}
],
"outcomes": {
"gmv": {
"write": true,
"value": {
"column": "paid_at"
}
},
"refund_rate": {
"write": true,
"value": {
"column": "paid_at"
}
}
}
},
{
"source": "shop",
"table": "orders",
"item": "business_date",
"members": [
{
"metric_id": "gmv",
"choice": "a"
},
{
"metric_id": "refund_rate",
"choice": "b"
}
],
"outcomes": {
"gmv": {
"write": false,
"reason": "同一次裁决里对「orders」的业务日期给出了不同的说法(指标「销售额」采纳的是「paid_at」;指标「退款率」采纳的是「created_at」),无法确定哪一个是数据事实,都未写入数据目录"
},
"refund_rate": {
"write": false,
"reason": "同一次裁决里对「orders」的业务日期给出了不同的说法(指标「销售额」采纳的是「paid_at」;指标「退款率」采纳的是「created_at」),无法确定哪一个是数据事实,都未写入数据目录"
}
}
},
{
"source": "shop",
"table": "orders",
"item": "business_date",
"members": [
{
"metric_id": "gmv",
"choice": "b"
},
{
"metric_id": "refund_rate",
"choice": "a"
}
],
"outcomes": {
"gmv": {
"write": false,
"reason": "同一次裁决里对「orders」的业务日期给出了不同的说法(指标「销售额」采纳的是「created_at」;指标「退款率」采纳的是「paid_at」),无法确定哪一个是数据事实,都未写入数据目录"
},
"refund_rate": {
"write": false,
"reason": "同一次裁决里对「orders」的业务日期给出了不同的说法(指标「销售额」采纳的是「created_at」;指标「退款率」采纳的是「paid_at」),无法确定哪一个是数据事实,都未写入数据目录"
}
}
},
{
"source": "shop",
"table": "orders",
"item": "business_date",
"members": [
{
"metric_id": "gmv",
"choice": "b"
},
{
"metric_id": "refund_rate",
"choice": "b"
}
],
"outcomes": {
"gmv": {
"write": false,
"reason": "「orders」的业务日期「created_at」曾被人工驳回,与这次指标「销售额」、「退款率」的选择相反,未写入。如需更改,请到数据目录中修改"
},
"refund_rate": {
"write": false,
"reason": "「orders」的业务日期「created_at」曾被人工驳回,与这次指标「销售额」、「退款率」的选择相反,未写入。如需更改,请到数据目录中修改"
}
}
}
]
}
},
"catalog_blocked": {