双盲复算复核整改(第 2 轮):遮罩的分组列不收窄,收窄时去掉 HAVING,审批卡带上处置和写目录的预告,同一次裁决里的有效记录条件合起来写

节点(nodes/dual_check.py)
- 差异说明收窄到这一格时,分组键的结果列或它用到的列设了遮罩,就不收窄、照整条比。原流程、盲算各按自己那份快照
  的遮罩(查询当时记下的加数据源现在设的)判断。以前按这一行的分组值补等值条件,「region = '华东'」这样的遮罩列
  取值会跟着差异说明进审批 payload、human.requested 事件、对照工件和 dual_check_catalog 日志
- 按行收窄时连 HAVING 一起去掉:这一行既然在结果里,它所在的组就满足 HAVING。留着会多出一条「分组后的筛选」,
  还会占掉「只差一个方面」的名额,真正的差异(比如时间列)就写不进目录
- 审批 payload 在约定的几项之外带上实际生效的 on_reject。摘要里每个指标另带 catalog:选 a、选 b 时,按发起审批
  时的目录各会往数据目录里写什么({write: true, source, table, item, value}),不写的给出原因({write: false,
  reason})。审核人要看得出不采用原流程会不会让运行失败、某个选择会不会被记成人工确认的目录项
  (on_reject 以前只能从图快照里读,子工作流里的节点读不到)

写数据目录(dual_check_facts.py)
- 写入的判断拆成不读库的 _plan:真写(_write)和审批卡上的预告(preview)用同一套判断,不会说一套写一套
- 同一次裁决里落到同一项上的几个说法合成一项(_merged):
  - 说法一样的只写一次,备注列出每个指标
  - 有效记录条件用 AND 合成一句。derive 只收排除无效记录的条件,「排除退款」和「排除测试单」并不矛盾,以前两条
    都被丢掉,原因还写成「互相矛盾」
  - 业务日期、去重规则说法不同的仍然都不写,原因改为「给出了不同的说法」
- 合并后再和目录里已有的推断条件合起来写时,要求每个被采纳的一边都按已有条件筛过;有一个没按它筛,就不动并说明
  是哪个指标

测试
- 遮罩分组列(原流程侧、盲算侧)的取值不出现在 payload、对照工件、节点产出和节点事件里
- HAVING 收窄后只剩时间列一处差异,并写进目录
- payload 的 on_reject、摘要里的预告(一致、已确认不覆盖、指标自己的范围几种情形)
- 有效记录条件合并(含和已有推断条件合写、有一个被采纳的一边没用已有条件时不动),同一说法只写一次
- 补上复核指出的三处接线缺口,各有一条图级用例:
  - 带取舍的列名(gmv_by_order_date、paid_orders)不交给盲算
  - 取值在运行输入里的排除条件不写目录
  - 分组列上的 is_test = 0 不写目录
- 13 处变异(遮罩判断两侧、HAVING、上面三处接线、合并、预告、on_reject 等)去掉后,相关测试都会变红
This commit is contained in:
ThearchyHelios
2026-10-05 16:43:14 +08:00
parent 1491b5c7f3
commit f7d1a2ecba
4 changed files with 456 additions and 115 deletions
+149 -57
View File
@@ -23,10 +23,17 @@ is_test = 0、is_valid = 1、deleted_at IS NULL);只涉及一张表。拿不
**怎么写。** 照对话提案的规矩核对(plan_patch:列在不在表结构里、取值格式),记为人工填写、已确认,备注写明来自哪次 **怎么写。** 照对话提案的规矩核对(plan_patch:列在不在表结构里、取值格式),记为人工填写、已确认,备注写明来自哪次
运行、哪个节点、哪个指标。目录里这一项已经是人工确认过的另一种说法时不覆盖,照实交回原因;已经是同样的说法就什么都 运行、哪个节点、哪个指标。目录里这一项已经是人工确认过的另一种说法时不覆盖,照实交回原因;已经是同样的说法就什么都
不写(再写一遍也不升版本)。有效记录条件是一整句条件:目录里已有的条件被采纳的一边也按它筛了的,和新条件合起来写 不写(再写一遍也不升版本)。有效记录条件是一整句条件:目录里已有的条件被采纳的一边(几个指标并成一项时是每一个)
(已确认的不改,照实说);被采纳的一边没按它筛的,两者的关系拿不准,不动。推断的去重规则说的未必是同一件事,也不 也按它筛了的,和新条件合起来写(已确认的不改,照实说);有一个没按它筛的,两者的关系拿不准,不动。推断的去重规则
替换。同一次裁决里几个指标对同一项给出不同说法的,一个都不写。整个写入由节点放在 ask() 之后的一个 @task 里:审批 说的未必是同一件事,也不替换。整个写入由节点放在 ask() 之后的一个 @task 里:审批恢复重放、接着运行时不会再写一遍。
恢复重放、接着运行时不会再写一遍。
**同一次裁决里几个指标落到同一项上。** 说法一样的写一次,备注里列出每个指标。有效记录条件是能用 AND 叠加的排除条件
(这里只收排除无效记录的),几个指标各自采纳的条件都有人的选择撑着,合成一句写——「排除退款」和「排除测试单」并不
矛盾。业务日期一张表只有一个,去重规则说的未必是同一件事,说法不同就一个都不写。
**预告。** 审批卡上要让审核人看得出选了之后会往目录里写什么(preview):按发起审批时的目录,选原流程、选盲算各会
写什么、不写的为什么。和真写是同一套判断(derive 认事实、_plan 对目录),所以只在两处可能和最后写的不一样:审批停下
期间目录被人改了,或者同一次裁决里几个指标落到同一项上(上一段)。
catalog.save_patch 不收备注,这里用它的同一套原料(read_entry、apply_patch、write_entry 的乐观锁)写,只多把备注放进项里。 catalog.save_patch 不收备注,这里用它的同一套原料(read_entry、apply_patch、write_entry 的乐观锁)写,只多把备注放进项里。
""" """
@@ -35,7 +42,7 @@ from __future__ import annotations
import logging import logging
import re import re
from collections.abc import Mapping from collections.abc import Mapping
from dataclasses import dataclass from dataclasses import dataclass, replace
from typing import Any from typing import Any
from sqlalchemy import select from sqlalchemy import select
@@ -79,7 +86,8 @@ _SCOPE_DIGITS = 2
@dataclass(frozen=True) @dataclass(frozen=True)
class Fact: class Fact:
"""要写进目录的一项:哪个数据源的哪张表、哪一项、写什么。metric、choice 只用来写备注和说明;sql、dialect 是被采纳 """要写进目录的一项:哪个数据源的哪张表、哪一项、写什么。metric、choice 只用来写备注和说明;sql、dialect 是被采纳
那一边的写法和方言,有效记录条件和目录里已有的条件合并时要看它还按哪些条件筛了。""" 那一边的写法和方言,有效记录条件和目录里已有的条件合并时要看它还按哪些条件筛了。also 是同一次裁决里落到同一项上、
并进这一项的其余几个指标(各自的指标、选择、写法),value 是合起来之后的说法。"""
source: str source: str
table: str table: str
@@ -89,6 +97,18 @@ class Fact:
choice: str choice: str
sql: str = "" sql: str = ""
dialect: str | None = None dialect: str | None = None
also: tuple[Fact, ...] = ()
def _members(fact: Fact) -> tuple[Fact, ...]:
"""这一项背后的每个指标的选择:并过的连同并进来的。"""
return (fact, *fact.also)
def _names(fact: Fact) -> str:
"""说明里的指标叫法:「销售额」,几个指标并成一项时「销售额」、「订单金额」。"""
names = list(dict.fromkeys(m.metric for m in _members(fact)))
return "、".join(f"「{n}」" for n in names[:4]) + (f"等 {len(names)} 个" if len(names) > 4 else "")
def derive(candidate: Mapping[str, Any], schema_cache: Mapping[str, Any] | None, def derive(candidate: Mapping[str, Any], schema_cache: Mapping[str, Any] | None,
@@ -402,28 +422,45 @@ def _conditions_of(sql: str, read: str | None) -> set[str] | None:
return {_normalized(part, read) for where in tree.find_all(exp.Where) for part in _and_parts(where.this)} return {_normalized(part, read) for where in tree.find_all(exp.Where) for part in _and_parts(where.this)}
def _and_join(base: str | None, values: list[str], dialect: str | None) -> str | None:
"""几句条件用 AND 接成一句:已经有的(base 里的、前面接过的,按 _normalized 比)不再接;最外层是 OR 的加括号,
不然后面接的条件只并到 OR 的后一半上。每句保留原来的写法。有一句解析不了返回 None。"""
from sqlglot import exp
read = _read(dialect)
texts: list[str] = []
seen: set[str] = set()
for text in [*([base] if base else []), *values]:
tree = _parse(str(text), read)
if tree is None:
return None
keys = {_normalized(p, read) for p in _and_parts(tree)}
if keys <= seen:
continue
seen |= keys
texts.append(f"({text})" if isinstance(tree, exp.Or) else str(text))
return " AND ".join(texts)
def _combined_filter(existing: str, fact: Fact) -> tuple[str | None, str | None]: def _combined_filter(existing: str, fact: Fact) -> tuple[str | None, str | None]:
"""目录里已有的有效记录条件和这次的新条件合起来:(合起来的整句条件, None) 或 (None, 为什么不动)。 """目录里已有的有效记录条件和这次的新条件合起来:(合起来的整句条件, None) 或 (None, 为什么不动)。
有效记录条件是一整句(copilot 提案也要求「原有条件也要带上」)。只写新条件会把两边都按着筛的原有条件静默替换掉; 有效记录条件是一整句(copilot 提案也要求「原有条件也要带上」)。只写新条件会把两边都按着筛的原有条件静默替换掉;
原有条件被采纳的一边没按它筛,审核人的选择说明不了它对不对,也不能替人删掉。""" 原有条件有一个被采纳的一边没按它筛,审核人的选择说明不了它对不对(合起来写就等于替人确认了它),也不能替人删掉。"""
label = catalog.TABLE_FIELD_LABEL["valid_filter"] label = catalog.TABLE_FIELD_LABEL["valid_filter"]
read = _read(fact.dialect) read = _read(fact.dialect)
have_tree, new_tree = _parse(existing, read), _parse(str(fact.value), read) have_tree = _parse(existing, read)
if have_tree is None or new_tree is None: combined = _and_join(existing, [str(m.value) for m in _members(fact)], fact.dialect) \
if have_tree is not None else None
if combined is None:
return None, f"「{fact.table}」已有的{label}「{existing}」无法解析,与这次的条件合不起来,未改动" return None, f"「{fact.table}」已有的{label}「{existing}」无法解析,与这次的条件合不起来,未改动"
have = [_normalized(p, read) for p in _and_parts(have_tree)] have = {_normalized(p, read) for p in _and_parts(have_tree)}
used = _conditions_of(fact.sql, read) or set() for member in _members(fact):
if not set(have) <= used: if not have <= (_conditions_of(member.sql, read) or set()):
return None, (f"「{fact.table}」已有的{label}「{existing}」,被采纳的{SIDE_LABEL.get(fact.choice, fact.choice)}" return None, (f"「{fact.table}」已有的{label}「{existing}」,指标「{member.metric}」被采纳的"
"没有按它筛选,两者的关系无法确定,未改动。如需更改,请到数据目录中修改") f"{SIDE_LABEL.get(member.choice, member.choice)}没有按它筛选,两者的关系无法确定,未改动。"
if {_normalized(p, read) for p in _and_parts(new_tree)} <= set(have): "如需更改,请到数据目录中修改")
return existing, None return combined, None
from sqlglot import exp
# 原有条件最外层是 OR 的,接 AND 之前要加括号,不然新条件只并到 OR 的后一半上
head = f"({existing})" if isinstance(have_tree, exp.Or) else existing
return f"{head} AND {fact.value}", None
def _dedup(item: Mapping[str, Any], side: str, candidate: Mapping[str, Any], def _dedup(item: Mapping[str, Any], side: str, candidate: Mapping[str, Any],
@@ -491,9 +528,13 @@ def _aggregate_pair(a: Any, b: Any, dialect: Any) -> tuple[str, tuple[bool, bool
def note_of(fact: Fact, *, run_id: str, node_title: str) -> str: def note_of(fact: Fact, *, run_id: str, node_title: str) -> str:
"""目录项上的备注:来自哪次运行、哪个节点、哪个指标、采纳了哪一边。""" """目录项上的备注:来自哪次运行、哪个节点、哪些指标、各采纳了哪一边。"""
return (f"双盲复算(运行 {run_id},节点「{node_title}」):审核人在指标「{fact.metric}」上" parts = []
f"采用了{SIDE_LABEL.get(fact.choice, fact.choice)}的写法") for side, label in SIDE_LABEL.items():
names = list(dict.fromkeys(m.metric for m in _members(fact) if m.choice == side))
if names:
parts.append(f"在指标{'、'.join(f'「{n}」' for n in names)}上采用了{label}的写法")
return f"双盲复算(运行 {run_id},节点「{node_title}」):审核人" + ",".join(parts)
def _value_text(item: str, value: Any) -> str: def _value_text(item: str, value: Any) -> str:
@@ -509,21 +550,38 @@ def _same_fact(item: str, a: Any, b: Any) -> bool:
return isinstance(a, str) and isinstance(b, str) and " ".join(a.split()) == " ".join(b.split()) return isinstance(a, str) and isinstance(b, str) and " ".join(a.split()) == " ".join(b.split())
def _contradiction(facts: list[Fact]) -> str: def _differing(facts: list[Fact]) -> str:
"""同一次裁决里对同一项给出的不同说法,说清是谁、各说了什么。""" """同一次裁决里对同一项给出的不同说法,说清是谁、各说了什么。"""
first = facts[0] first = facts[0]
label = catalog.TABLE_FIELD_LABEL.get(first.item, first.item) label = catalog.TABLE_FIELD_LABEL.get(first.item, first.item)
said = ";".join(f"指标「{f.metric}」采纳的是「{_value_text(f.item, f.value)}」" for f in facts[:4]) said = ";".join(f"指标「{f.metric}」采纳的是「{_value_text(f.item, f.value)}」" for f in facts[:4])
return (f"同一次裁决里对「{first.table}」的{label}给出的选择互相矛盾({said}),无法确定哪一个是数据事实," return (f"同一次裁决里对「{first.table}」的{label}给出了不同的说法({said}),无法确定哪一个是数据事实,"
"都未写入数据目录") "都未写入数据目录")
def _merged(members: list[Fact]) -> tuple[Fact | None, str | None]:
"""同一次裁决里落到同一项上的几个说法合成一项:(合起来的那一项, None) 或 (None, 为什么都不写)。
说法都一样的写一次。有效记录条件用 AND 合成一句(derive 只收排除无效记录的条件,几句叠起来不会互相打架)。
业务日期一张表只有一个、去重规则说的未必是同一件事,说法不同就都不写。"""
first, rest = members[0], tuple(members[1:])
if not rest:
return first, None
if first.item == "valid_filter":
value = _and_join(None, [str(m.value) for m in members], first.dialect)
return (replace(first, value=value, also=rest), None) if value is not None else (None, _differing(members))
if all(_same_fact(first.item, m.value, first.value) for m in rest):
return replace(first, also=rest), None
return None, _differing(members)
async def record_choices(candidates: list[Mapping[str, Any]], *, run_id: str, node_title: str, async def record_choices(candidates: list[Mapping[str, Any]], *, run_id: str, node_title: str,
actor: str | None) -> dict[str, list[dict[str, Any]]]: actor: str | None) -> dict[str, list[dict[str, Any]]]:
"""把审核人的选择写进数据目录:{written: [{source, table, item}], skipped: [{reason}]}。永远不抛:数据源没了、 """把审核人的选择写进数据目录:{written: [{source, table, item}], skipped: [{reason}]}。永远不抛:数据源没了、
目录写不进去,都落在 skipped 里照实说。 目录写不进去,都落在 skipped 里照实说。
先把每个候选认成目录项,再按(数据源、表、项)归拢:同一项上说法互相矛盾的整组不写,剩下的逐个写。""" 先把每个候选认成目录项,再按(数据源、表、项)归拢成一项(_merged:说法一样的、有效记录条件合起来写,说法
不同的整组不写),逐项写。"""
skipped: list[dict[str, Any]] = [] skipped: list[dict[str, Any]] = []
facts: list[Fact] = [] facts: list[Fact] = []
for candidate in candidates: for candidate in candidates:
@@ -549,10 +607,10 @@ async def record_choices(candidates: list[Mapping[str, Any]], *, run_id: str, no
groups.setdefault((fact.source, fact.table, fact.item), []).append(fact) groups.setdefault((fact.source, fact.table, fact.item), []).append(fact)
written: list[dict[str, Any]] = [] written: list[dict[str, Any]] = []
for members in groups.values(): for members in groups.values():
if any(not _same_fact(f.item, f.value, members[0].value) for f in members[1:]): fact, why = _merged(members)
skipped.append({"reason": _contradiction(members)}) if fact is None:
skipped.append({"reason": why})
continue continue
for fact in members:
done, why = await _record(fact, run_id=run_id, node_title=node_title, actor=actor) done, why = await _record(fact, run_id=run_id, node_title=node_title, actor=actor)
if done: if done:
written.append({"source": fact.source, "table": fact.table, "item": fact.item}) written.append({"source": fact.source, "table": fact.table, "item": fact.item})
@@ -566,29 +624,25 @@ async def _record(fact: Fact, *, run_id: str, node_title: str, actor: str | None
async with SessionLocal() as session: async with SessionLocal() as session:
row = (await session.execute(select(DataSource).where(DataSource.name == fact.source))).scalar_one_or_none() row = (await session.execute(select(DataSource).where(DataSource.name == fact.source))).scalar_one_or_none()
if row is None: if row is None:
return False, f"数据源「{fact.source}」已不存在,指标「{fact.metric}」的选择未写入数据目录" return False, f"数据源「{fact.source}」已不存在,指标{_names(fact)}的选择未写入数据目录"
return await _write(session, row, fact, note=note_of(fact, run_id=run_id, node_title=node_title), return await _write(session, row, fact, note=note_of(fact, run_id=run_id, node_title=node_title),
actor=actor) actor=actor)
except Exception: # noqa: BLE001 - 目录写不进去不能毁掉审批的结果:缺口照样按选择记 except Exception: # noqa: BLE001 - 目录写不进去不能毁掉审批的结果:缺口照样按选择记
logger.exception("双盲复算的选择写入数据目录失败(数据源 %s)", fact.source) logger.exception("双盲复算的选择写入数据目录失败(数据源 %s)", fact.source)
return False, f"指标「{fact.metric}」的选择写入数据目录时出错,未写入" return False, f"指标{_names(fact)}的选择写入数据目录时出错,未写入"
async def _write(session: Any, row: DataSource, fact: Fact, *, note: str, actor: str | None) -> tuple[bool, str | None]: def _plan(notes: Mapping[str, Any], fact: Fact, tables: Mapping[str, Any] | None) -> tuple[Any, str | None]:
"""写一项。(写了没有, 没写的原因)。""" """按目录里这张表现在的说法,算出这一项要写成什么:(要写的取值, None) 或 (None, 为什么不写)。不读库、不写库:
真写(_write)和审批卡上的预告(preview)用的是这同一套判断。"""
label = catalog.TABLE_FIELD_LABEL.get(fact.item, fact.item) label = catalog.TABLE_FIELD_LABEL.get(fact.item, fact.item)
side = SIDE_LABEL.get(fact.choice, fact.choice) names = _names(fact)
tables = (row.schema_cache or {}).get("tables") if isinstance(row.schema_cache, dict) else None
for _ in range(_RETRIES):
entry = await catalog.read_entry(session, row.id, fact.table)
notes = entry.notes if entry else {}
version = entry.version if entry else 0
stored = notes.get(fact.item) if isinstance(notes.get(fact.item), dict) else None stored = notes.get(fact.item) if isinstance(notes.get(fact.item), dict) else None
if stored is not None and stored.get("status") == "rejected" and _same_fact(fact.item, stored.get("value"), if stored is not None and stored.get("status") == "rejected" and _same_fact(fact.item, stored.get("value"),
fact.value): fact.value):
# 同一个说法以前被人明确驳回过:两次人工判断对不上,不替人拿主意 # 同一个说法以前被人明确驳回过:两次人工判断对不上,不替人拿主意
return False, (f"「{fact.table}」的{label}「{_value_text(fact.item, fact.value)}」曾被人工驳回," return None, (f"「{fact.table}」的{label}「{_value_text(fact.item, fact.value)}」曾被人工驳回,"
f"与这次指标「{fact.metric}」的选择相反,未写入。如需更改,请到数据目录中修改") f"与这次指标{names}的选择相反,未写入。如需更改,请到数据目录中修改")
live = stored if stored is not None and stored.get("status") != "rejected" else None live = stored if stored is not None and stored.get("status") != "rejected" else None
value = fact.value value = fact.value
if fact.item == "business_date" and live is not None and isinstance(live.get("value"), dict) \ if fact.item == "business_date" and live is not None and isinstance(live.get("value"), dict) \
@@ -599,36 +653,74 @@ async def _write(session: Any, row: DataSource, fact: Fact, *, note: str, actor:
and live["value"].strip() and not _same_fact(fact.item, live["value"], value): and live["value"].strip() and not _same_fact(fact.item, live["value"], value):
combined, why = _combined_filter(live["value"].strip(), fact) combined, why = _combined_filter(live["value"].strip(), fact)
if combined is None: if combined is None:
return False, why return None, why
if live.get("status") == "confirmed" and not _same_fact(fact.item, combined, live["value"]): if live.get("status") == "confirmed" and not _same_fact(fact.item, combined, live["value"]):
return False, (f"「{fact.table}」的{label}已有人工确认过的说法「{live['value'].strip()}」,被采纳的" adopted = f"被采纳的{SIDE_LABEL.get(fact.choice, fact.choice)}" if not fact.also else "这次几个指标被采纳的写法"
f"{side}另外多一个条件「{fact.value}」,未改动。如需补上,请到数据目录中修改") return None, (f"「{fact.table}」的{label}已有人工确认过的说法「{live['value'].strip()}」,{adopted}"
f"另外多出条件「{fact.value}」,未改动。如需补上,请到数据目录中修改")
value = combined value = combined
elif fact.item == "dedup" and isinstance(live, dict) and live.get("status") != "confirmed" \ elif fact.item == "dedup" and isinstance(live, dict) and live.get("status") != "confirmed" \
and not _same_fact(fact.item, live.get("value"), value): and not _same_fact(fact.item, live.get("value"), value):
# 推断的去重规则说的未必是同一件事(按哪一列去重、哪种汇总要去重),替换掉就把它丢了 # 推断的去重规则说的未必是同一件事(按哪一列去重、哪种汇总要去重),替换掉就把它丢了
return False, (f"「{fact.table}」的{label}已有推断的说法「{_value_text(fact.item, live.get('value'))}」," return None, (f"「{fact.table}」的{label}已有推断的说法「{_value_text(fact.item, live.get('value'))}」,"
f"与这次指标「{fact.metric}」采纳的「{value}」不同,无法确定两者的关系,未覆盖。" f"与这次指标{names}采纳的「{value}」不同,无法确定两者的关系,未覆盖。"
"如需更改,请到数据目录中修改") "如需更改,请到数据目录中修改")
changes = [{"path": fact.item, "value": value, "reason": note}] plan = catalog.plan_patch(notes, [{"path": fact.item, "value": value}], table=fact.table, tables=tables)
plan = catalog.plan_patch(notes, changes, table=fact.table, tables=tables)
if plan.problems: if plan.problems:
return False, f"「{fact.table}」的{label}未写入数据目录:{plan.problems[0]}" return None, f"「{fact.table}」的{label}未写入数据目录:{plan.problems[0]}"
change = plan.changes[0] change = plan.changes[0]
if change.state == "same": if change.state == "same":
return False, (f"「{fact.table}」的{label}已是人工确认过的「{_value_text(fact.item, value)}」," return None, f"「{fact.table}」的{label}已是人工确认过的「{_value_text(fact.item, value)}」,无需重复写入"
"无需重复写入")
if change.before_status == "confirmed": if change.before_status == "confirmed":
return False, (f"「{fact.table}」的{label}已有人工确认过的说法「{_value_text(fact.item, change.before)}」," return None, (f"「{fact.table}」的{label}已有人工确认过的说法「{_value_text(fact.item, change.before)}」,"
f"与这次指标「{fact.metric}」采纳的「{_value_text(fact.item, value)}」不同,未覆盖。" f"与这次指标{names}采纳的「{_value_text(fact.item, value)}」不同,未覆盖。"
"如需更改,请到数据目录中修改") "如需更改,请到数据目录中修改")
return change.after, None
def preview(candidate: Mapping[str, Any], schema_cache: Mapping[str, Any] | None,
notes: Mapping[str, Mapping[str, Any]] | None = None) -> dict[str, dict[str, Any]]:
"""审批卡上的预告:审核人在这个指标上选原流程(a)、选盲算(b),按目前的数据目录各会往目录里写什么。
candidate 同 derive(choice 不用给);notes 是这个数据源的目录({表名: notes})。返回 {a: …, b: …},每边是
{write: true, source, table, item, value}(value 是真要写进去的说法:和目录里已有的有效记录条件合起来的,是合起来
之后的整句)或 {write: false, reason}。和恢复后真写是同一套判断,只在模块说明里说的两处可能不一样。永远不抛。"""
notes = notes or {}
tables = schema_cache.get("tables") if isinstance(schema_cache, Mapping) else None
out: dict[str, dict[str, Any]] = {}
for side in SIDE_LABEL:
try:
fact, why = derive({**candidate, "choice": side}, schema_cache, notes)
value = None
if fact is not None:
value, why = _plan(notes.get(fact.table) or {}, fact, tables)
except Exception: # noqa: BLE001 - 预告算不出来不能拦住审批:照实说,以写入后的结果为准
logger.exception("双盲复算预告数据目录的写入失败(指标 %s)", candidate.get("metric"))
fact, why = None, f"指标「{candidate.get('metric')}」的选择核对数据目录时出错,无法预告是否写入"
out[side] = {"write": True, "source": fact.source, "table": fact.table, "item": fact.item, "value": value} \
if fact is not None and why is None else {"write": False, "reason": why}
return out
async def _write(session: Any, row: DataSource, fact: Fact, *, note: str, actor: str | None) -> tuple[bool, str | None]:
"""写一项。(写了没有, 没写的原因)。写什么、写不写由 _plan 按读到的目录定;版本对不上(别人刚改过)重读再来。"""
label = catalog.TABLE_FIELD_LABEL.get(fact.item, fact.item)
tables = (row.schema_cache or {}).get("tables") if isinstance(row.schema_cache, dict) else None
for _ in range(_RETRIES):
entry = await catalog.read_entry(session, row.id, fact.table)
notes = entry.notes if entry else {}
version = entry.version if entry else 0
value, why = _plan(notes, fact, tables)
if why is not None:
return False, why
at = catalog.now_iso() at = catalog.now_iso()
try: try:
merged = catalog.apply_patch(notes, changes, table=fact.table, tables=tables, at=at) merged = catalog.apply_patch(notes, [{"path": fact.item, "value": value, "reason": note}],
table=fact.table, tables=tables, at=at)
except catalog.CatalogInvalid as e: except catalog.CatalogInvalid as e:
return False, f"「{fact.table}」的{label}未写入数据目录:{e}" return False, f"「{fact.table}」的{label}未写入数据目录:{e}"
# 不论原来是谁提的:这一项现在出自人的选择,记为人工填写、已确认,备注写明出处 # 不论原来是谁提的:这一项现在出自人的选择,记为人工填写、已确认,备注写明出处
merged[fact.item] = catalog.make_item(change.after, "human", "confirmed", note=note, at=at) merged[fact.item] = catalog.make_item(value, "human", "confirmed", note=note, at=at)
try: try:
await catalog.write_entry(session, row.id, fact.table, merged, if_version=version, actor=actor) await catalog.write_entry(session, row.id, fact.table, merged, if_version=version, actor=actor)
except catalog.CatalogConflict: except catalog.CatalogConflict:
@@ -640,4 +732,4 @@ async def _write(session: Any, row: DataSource, fact: Fact, *, note: str, actor:
return False, f"「{fact.table}」的数据目录正被修改,{label}未能写入,请稍后在数据目录中确认" return False, f"「{fact.table}」的数据目录正被修改,{label}未能写入,请稍后在数据目录中确认"
__all__ = ["FACT_ITEMS", "Fact", "derive", "note_of", "record_choices"] __all__ = ["FACT_ITEMS", "Fact", "derive", "note_of", "preview", "record_choices"]
+41 -23
View File
@@ -14,11 +14,14 @@
里看不出日期、口径的取舍时才给,好让它分得清哪一项是哪一项。 里看不出日期、口径的取舍时才给,好让它分得清哪一项是哪一项。
**一致就放行,不一致交给人。** 不一致时逐个对不上的输入说清两条 SQL 差在哪(data/sql_semantics.explain_sql_diff): **一致就放行,不一致交给人。** 不一致时逐个对不上的输入说清两条 SQL 差在哪(data/sql_semantics.explain_sql_diff):
两边先收窄到这一格(只留算出这一列的那一项;分组的按这一行的分组值改成等值条件),一条查询算好几列、分组后取一行时 两边先收窄到这一格(只留算出这一列的那一项;分组的按这一行的分组值改成等值条件,HAVING 一并去掉),一条查询算好几列、
才不会混进「选出的列」「分组」这类和这一格无关的差别。完整对照落成一件工件(审批 payload 经 runner._safe 会截断, 分组后取一行时才不会混进「选出的列」「分组」这类和这一格无关的差别。分组的列设了遮罩时不收窄:这一行的分组值写进
SQL,就跟着差异说明进了审批卡、对照工件和目录日志。完整对照落成一件工件(审批 payload 经 runner._safe 会截断,
长 SQL 放不下),用 replay.ask 挂起。审核人逐个指标选采用原流程(a)、采用盲算(b)或两边都不对(neither):a 没有 长 SQL 放不下),用 replay.ask 挂起。审核人逐个指标选采用原流程(a)、采用盲算(b)或两边都不对(neither):a 没有
缺口;b、neither、没选的写进产出的 issuance_gaps,成果节点的出具契约据此降档(on_reject 为 fail 时节点直接失败)。 缺口;b、neither、没选的写进产出的 issuance_gaps,成果节点的出具契约据此降档(on_reject 为 fail 时节点直接失败)。
两边只差一个方面、又对得上数据目录事实的,选择写进数据目录(engine/dual_check_facts.py,指标自己的筛选范围不写)。 两边只差一个方面、又对得上数据目录事实的,选择写进数据目录(engine/dual_check_facts.py,指标自己的筛选范围不写)。
选了会怎样要在卡上看得出:payload 在约定的几项之外带上实际生效的 on_reject,摘要里每个指标带 catalog——选 a、选 b
按发起审批时的目录各会往目录里写什么、不写的为什么(dual_check_facts.preview,和真写同一套判断)。
盲算触顶、出错、交不出数,或者输入本身没法单独复算的指标按「无法复算」记缺口,发 dual_check_unchecked。 盲算触顶、出错、交不出数,或者输入本身没法单独复算的指标按「无法复算」记缺口,发 dual_check_unchecked。
**产出的状态。** 全部一致是 agreed;没有不一致、但有无法复算的:一个都没核成是 unchecked,核成了一部分是 partial; **产出的状态。** 全部一致是 agreed;没有不一致、但有无法复算的:一个都没核成是 unchecked,核成了一部分是 partial;
@@ -54,6 +57,7 @@ from app.db.models import DataSource
from app.engine import recompute from app.engine import recompute
from app.engine.approval import read_decision from app.engine.approval import read_decision
from app.engine.context import NodeContext, NodeError from app.engine.context import NodeContext, NodeError
from app.engine.dual_check_facts import preview, record_choices
from app.engine.evidence import build_catalog, is_number, next_exec from app.engine.evidence import build_catalog, is_number, next_exec
from app.engine.expressions import ExpressionError, _cell_call, _chain_root, cell_value, column_kind, numeric_text, \ from app.engine.expressions import ExpressionError, _cell_call, _chain_root, cell_value, column_kind, numeric_text, \
parse_expression parse_expression
@@ -907,10 +911,14 @@ async def _review(state: GraphState, ctx: NodeContext, checked: dict[str, Any],
"""不一致的指标逐个说清差在哪,完整对照落成工件,组好审批的 payload。返回 {payload, facts, card_artifact}。 """不一致的指标逐个说清差在哪,完整对照落成工件,组好审批的 payload。返回 {payload, facts, card_artifact}。
两条 SQL 先收窄到这一格再比(_narrowed):原流程一条查询算好几列、分组后取一行时,整条拿去比会混进「选出的列」 两条 SQL 先收窄到这一格再比(_narrowed):原流程一条查询算好几列、分组后取一行时,整条拿去比会混进「选出的列」
「分组」这类和这一格无关的差别,审批卡上只列前几条,审核人看到的多半是噪声,只差一个方面的也就写不进目录。 「分组」这类和这一格无关的差别,审批卡上只列前几条,审核人看到的多半是噪声,只差一个方面的也就写不进目录。两边
各按自己那份快照要遮的列收窄(原流程的、盲算的快照各记着查询当时的遮罩,再加数据源现在设的)。
facts 是每个不一致指标写数据目录要用的东西(dual_check_facts.derive 的候选,少一个 choice):只有一个输入对不上时是 facts 是每个不一致指标写数据目录要用的东西(dual_check_facts.derive 的候选,少一个 choice):只有一个输入对不上时是
那一处完整的差异条目(带两边比较的写法和涉及的表、列),几个输入都对不上时只说「不止一处」;scope 是这个指标自己的 那一处完整的差异条目(带两边比较的写法和涉及的表、列),几个输入都对不上时只说「不止一处」;scope 是这个指标自己的
范围(口径说明、所在分组、运行输入、分组的列),有效记录条件据此排除指标自己的筛选范围。 范围(口径说明、所在分组、运行输入、分组的列),有效记录条件据此排除指标自己的筛选范围。scope 只在节点里用、不进
payload 和工件(所在分组的取值里可能有遮罩列的)。
payload 在约定的几项之外带 on_reject(实际生效的处置),摘要里每个指标带 catalog(选 a、选 b 各会往数据目录里写
什么,dual_check_facts.preview):审核人在卡上要看得出没采用原流程会不会让运行失败、选了会不会记成人工确认的目录项。
""" """
rows = [r for r in checked["rows"] if r["status"] == DIFFERENT] rows = [r for r in checked["rows"] if r["status"] == DIFFERENT]
results = checked["results"] results = checked["results"]
@@ -938,9 +946,11 @@ async def _review(state: GraphState, ctx: NodeContext, checked: dict[str, Any],
if inp["key"] not in row["differing"]: if inp["key"] not in row["differing"]:
continue continue
blind_snap, _ = sources.snapshot(str(blind["artifact"])) if blind.get("artifact") else (None, None) blind_snap, _ = sources.snapshot(str(blind["artifact"])) if blind.get("artifact") else (None, None)
a_sql = _narrowed(inp["sql"], inp["column"], _record(original_snap, inp["row"]), kind) or inp["sql"] blind_hidden = await sources.hidden(str(blind["artifact"])) if blind.get("artifact") else frozenset()
b_sql = _narrowed(blind.get("sql"), blind.get("column"), _record(blind_snap, blind.get("row")), kind) \ a_sql = _narrowed(inp["sql"], inp["column"], _record(original_snap, inp["row"]), kind,
or blind.get("sql") await sources.hidden(inp["artifact"])) or inp["sql"]
b_sql = _narrowed(blind.get("sql"), blind.get("column"), _record(blind_snap, blind.get("row")), kind,
blind_hidden) or blind.get("sql")
diff = await _explain(a_sql, b_sql, infos[inp["source"]]) diff = await _explain(a_sql, b_sql, infos[inp["source"]])
per_input.append((inp, blind, diff)) per_input.append((inp, blind, diff))
compared.append((a_sql, str(b_sql or ""))) compared.append((a_sql, str(b_sql or "")))
@@ -960,22 +970,24 @@ async def _review(state: GraphState, ctx: NodeContext, checked: dict[str, Any],
"items": [{"aspect": i.get("aspect"), "text": i.get("text")} for i in merged["items"]], "items": [{"aspect": i.get("aspect"), "text": i.get("text")} for i in merged["items"]],
"fallback": merged["fallback"], "text_diff": merged["text_diff"]}, "fallback": merged["fallback"], "text_diff": merged["text_diff"]},
}) })
summary.append({
"metric_id": row["metric_id"], "name": row["name"], "unit": row["unit"],
"original_rendered": row["original"]["rendered"], "blind_rendered": row["blind"]["rendered"],
"delta_pct": row["delta_pct"], "diff": [str(i.get("text")) for i in merged["items"][:SUMMARY_DIFFS]],
"fallback": not any(i.get("aspect") not in ("other", "parse") for i in merged["items"]),
})
first = per_input[0][0] if per_input else (row["inputs"][0] if row["inputs"] else {}) first = per_input[0][0] if per_input else (row["inputs"][0] if row["inputs"] else {})
single = compared[0] if len(per_input) == 1 else None single = compared[0] if len(per_input) == 1 else None
info = infos.get(str(first.get("source"))) or {}
facts[row["metric_id"]] = { facts[row["metric_id"]] = {
"metric_id": row["metric_id"], "metric": row["name"], "source": first.get("source"), "metric_id": row["metric_id"], "metric": row["name"], "source": first.get("source"),
"dialect": (infos.get(str(first.get("source"))) or {}).get("kind"), "dialect": info.get("kind"),
"sql": {"a": single[0], "b": single[1]} if single else {}, "sql": {"a": single[0], "b": single[1]} if single else {},
"diff": {"equivalent": merged["equivalent"], "fallback": merged["fallback"] or single is None, "diff": {"equivalent": merged["equivalent"], "fallback": merged["fallback"] or single is None,
"items": merged["items"]}, "items": merged["items"]},
"scope": {"texts": [t for t in dict.fromkeys(scope_texts) if t], "columns": sorted(scope_columns)}, "scope": {"texts": [t for t in dict.fromkeys(scope_texts) if t], "columns": sorted(scope_columns)},
} }
summary.append({
"metric_id": row["metric_id"], "name": row["name"], "unit": row["unit"],
"original_rendered": row["original"]["rendered"], "blind_rendered": row["blind"]["rendered"],
"delta_pct": row["delta_pct"], "diff": [str(i.get("text")) for i in merged["items"][:SUMMARY_DIFFS]],
"fallback": not any(i.get("aspect") not in ("other", "parse") for i in merged["items"]),
"catalog": preview(facts[row["metric_id"]], info.get("schema"), info.get("notes")),
})
content = {"metrics": metrics, "model": checked.get("model"), "same_model": bool(checked.get("same_model"))} content = {"metrics": metrics, "model": checked.get("model"), "same_model": bool(checked.get("same_model"))}
try: try:
artifact = await artifact_store.put_json(content, kind="dual_check", run_id=ctx.run.run_id, artifact = await artifact_store.put_json(content, kind="dual_check", run_id=ctx.run.run_id,
@@ -985,7 +997,7 @@ async def _review(state: GraphState, ctx: NodeContext, checked: dict[str, Any],
payload = { payload = {
"kind": "dual_check", "node_id": ctx.node.id, "mode": "input", "kind": "dual_check", "node_id": ctx.node.id, "mode": "input",
"title": f"双盲复算:{len(rows)} 个指标两边结果不一致,请选择采用哪一边的结果", "title": f"双盲复算:{len(rows)} 个指标两边结果不一致,请选择采用哪一边的结果",
"card_artifact": artifact, "summary": summary, "card_artifact": artifact, "summary": summary, "on_reject": options.on_reject,
} }
names = "、".join(f"「{r['name']}」" for r in rows[:5]) + (f"等 {len(rows)} 个" if len(rows) > 5 else "") names = "、".join(f"「{r['name']}」" for r in rows[:5]) + (f"等 {len(rows)} 个" if len(rows) > 5 else "")
ctx.emit(EventType.LOG, level="warn", code="dual_check_disagreed", ctx.emit(EventType.LOG, level="warn", code="dual_check_disagreed",
@@ -995,17 +1007,18 @@ async def _review(state: GraphState, ctx: NodeContext, checked: dict[str, Any],
async def _source_info(sources: _Sources, name: str) -> dict[str, Any]: async def _source_info(sources: _Sources, name: str) -> dict[str, Any]:
"""比较两条 SQL 要用的:方言、表结构、列的中文名(数据目录里的)。数据源没了就都不给,照样比原文。""" """比较两条 SQL、预告目录写入要用的:方言、表结构、列的中文名和整份数据目录。数据源没了就都不给,照样比原文。"""
from app.data import catalog from app.data import catalog
from app.data.sql_semantics import catalog_labels from app.data.sql_semantics import catalog_labels
row = await sources.source(name) row = await sources.source(name)
if row is None: if row is None:
return {"kind": None, "schema": None, "names": {}} return {"kind": None, "schema": None, "names": {}, "notes": {}}
async with SessionLocal() as session: async with SessionLocal() as session:
entries = await catalog.read_catalog(session, row.id) entries = await catalog.read_catalog(session, row.id)
notes = {table: entry.notes for table, entry in entries.items()}
return {"kind": row.kind, "schema": row.schema_cache if isinstance(row.schema_cache, dict) else None, return {"kind": row.kind, "schema": row.schema_cache if isinstance(row.schema_cache, dict) else None,
"names": catalog_labels({table: entry.notes for table, entry in entries.items()})} "names": catalog_labels(notes), "notes": notes}
async def _explain(original: Any, blind: Any, info: dict[str, Any]) -> dict[str, Any]: async def _explain(original: Any, blind: Any, info: dict[str, Any]) -> dict[str, Any]:
@@ -1088,12 +1101,16 @@ def _sql_literal(value: Any) -> Any:
return None return None
def _narrowed(sql: Any, column: Any, record: Mapping[str, Any] | None, kind: str | None) -> str | None: def _narrowed(sql: Any, column: Any, record: Mapping[str, Any] | None, kind: str | None,
hidden: frozenset[str] = frozenset()) -> str | None:
"""把一条查询收窄到这一格:最外层 SELECT 只留算出这一列的那一项;分组的去掉分组,按这一行的分组值补上等值条件 """把一条查询收窄到这一格:最外层 SELECT 只留算出这一列的那一项;分组的去掉分组,按这一行的分组值补上等值条件
(华东那一行 → WHERE … AND region = '华东'),排序、取行数跟着去掉。 (华东那一行 → WHERE … AND region = '华东'),HAVING、排序、取行数跟着去掉——这一行既然在结果里,它所在的组就
满足 HAVING,留着只会多出一条和这一格无关的差异。
差异说明拿收窄后的两条比,才只说这一格是怎么算出来的。认不准的(SELECT *、DISTINCT、窗口函数、集合运算、分组键 差异说明拿收窄后的两条比,才只说这一格是怎么算出来的。认不准的(SELECT *、DISTINCT、窗口函数、集合运算、分组键
对不上结果列、这一行没有分组值)返回 None,调用方照整条比。""" 对不上结果列、这一行没有分组值)返回 None,调用方照整条比。hidden 是这份快照要遮的列(小写):分组键的结果列、
或者它用到的列在里面时也返回 None——分组值写进 SQL 就跟着差异说明进了审批卡、对照工件和目录日志,遮罩列的值
不能出现在那些地方。"""
import sqlglot import sqlglot
from sqlglot import exp from sqlglot import exp
@@ -1128,6 +1145,8 @@ def _narrowed(sql: Any, column: Any, record: Mapping[str, Any] | None, kind: str
if name is None or name not in values: if name is None or name not in values:
return None return None
expr = projections[index].this if isinstance(projections[index], exp.Alias) else projections[index] expr = projections[index].this if isinstance(projections[index], exp.Alias) else projections[index]
if name in hidden or any(col.name.lower() in hidden for col in expr.find_all(exp.Column)):
return None
if values[name] is None: if values[name] is None:
conditions.append(exp.Is(this=expr.copy(), expression=exp.Null())) conditions.append(exp.Is(this=expr.copy(), expression=exp.Null()))
continue continue
@@ -1135,7 +1154,7 @@ def _narrowed(sql: Any, column: Any, record: Mapping[str, Any] | None, kind: str
if literal is None: if literal is None:
return None return None
conditions.append(exp.EQ(this=expr.copy(), expression=literal)) conditions.append(exp.EQ(this=expr.copy(), expression=literal))
for arg in ("group", "order", "limit", "offset"): for arg in ("group", "having", "order", "limit", "offset"):
tree.set(arg, None) tree.set(arg, None)
tree = tree.where(*conditions, copy=False) tree = tree.where(*conditions, copy=False)
elif tree.args.get("limit") is None and tree.args.get("offset") is None: elif tree.args.get("limit") is None and tree.args.get("offset") is None:
@@ -1233,7 +1252,6 @@ def _resolved_text(rows: list[dict[str, Any]], choices: dict[str, str]) -> str:
async def _catalog(ctx: NodeContext, candidates: list[dict[str, Any]]) -> dict[str, Any]: async def _catalog(ctx: NodeContext, candidates: list[dict[str, Any]]) -> dict[str, Any]:
from app.data.catalog import TABLE_FIELD_LABEL from app.data.catalog import TABLE_FIELD_LABEL
from app.engine.dual_check_facts import record_choices
result = await record_choices(candidates, run_id=ctx.run.run_id, node_title=ctx.node.title, actor=ctx.actor()) result = await record_choices(candidates, run_id=ctx.run.run_id, node_title=ctx.node.title, actor=ctx.actor())
written, skipped = result["written"], result["skipped"] written, skipped = result["written"], result["skipped"]
+99 -4
View File
@@ -7,7 +7,10 @@
- 指标自己的筛选范围(只看华东、只看线上)不是有效记录条件:取值出现在指标名称、口径说明、所在分组、运行输入里的, - 指标自己的筛选范围(只看华东、只看线上)不是有效记录条件:取值出现在指标名称、口径说明、所在分组、运行输入里的,
条件用到分组的列的,不是在排除退款、作废、测试这类无效记录的,都不写 条件用到分组的列的,不是在排除退款、作废、测试这类无效记录的,都不写
- 写的是人工填写、已确认,备注写明运行、节点、指标;同样的说法已确认过就不再写(不升版本),确认过的不同说法不覆盖; - 写的是人工填写、已确认,备注写明运行、节点、指标;同样的说法已确认过就不再写(不升版本),确认过的不同说法不覆盖;
目录里已有的有效记录条件被采纳的一边也用着的,合起来写,没用着的不动;同一次裁决里互相矛盾的选择都不写 目录里已有的有效记录条件被采纳的一边也用着的,合起来写,没用着的不动
- 同一次裁决里几个指标落到同一项上:说法一样的写一次;有效记录条件(都在排除无效记录)用 AND 合成一句写;业务日期
说法不同的都不写
- 审批卡上的预告(preview):选 a、选 b 各会写什么、不写的为什么,按目录里已有的说法算,和真写同一套判断
差异条目一律由 sql_semantics.explain_sql_diff 真算出来,和节点交给审核人看的是同一份。表名、数字都是虚构的。 差异条目一律由 sql_semantics.explain_sql_diff 真算出来,和节点交给审核人看的是同一份。表名、数字都是虚构的。
""" """
@@ -18,7 +21,7 @@ import pytest
from app.data import catalog from app.data import catalog
from app.data.sql_semantics import explain_sql_diff from app.data.sql_semantics import explain_sql_diff
from app.db.base import SessionLocal from app.db.base import SessionLocal
from app.engine.dual_check_facts import derive, record_choices from app.engine.dual_check_facts import derive, preview, record_choices
from tests.fixtures.dual_check_shop import ( from tests.fixtures.dual_check_shop import (
BOTH_WRONG, BOTH_WRONG,
BUYERS, BUYERS,
@@ -237,6 +240,13 @@ async def seed(source_id: str, notes: dict) -> None:
await catalog.write_entry(session, source_id, "orders", notes, if_version=0, actor="数据组") await catalog.write_entry(session, source_id, "orders", notes, if_version=0, actor="数据组")
async def seed_again(source_id: str, notes: dict) -> None:
"""把目录这张表整份改回 notes(在现有版本上写)。"""
async with SessionLocal() as session:
entry = await catalog.read_entry(session, source_id, "orders")
await catalog.write_entry(session, source_id, "orders", notes, if_version=entry.version, actor="数据组")
async def test_a_choice_is_written_as_a_confirmed_human_item_with_its_origin(shop): async def test_a_choice_is_written_as_a_confirmed_human_item_with_its_origin(shop):
name, source_id = shop name, source_id = shop
out = await record_choices([candidate(BY_ORDER_DATE, RIGHT, "b", source=name)], run_id="run-42", out = await record_choices([candidate(BY_ORDER_DATE, RIGHT, "b", source=name)], run_id="run-42",
@@ -359,7 +369,8 @@ async def test_a_rejected_item_is_not_brought_back_but_a_rejected_other_value_do
async def test_contradictory_choices_in_one_decision_write_nothing(shop): async def test_contradictory_choices_in_one_decision_write_nothing(shop):
"""同一次裁决里,一个指标采纳了按下单日、另一个采纳了按支付日:写哪个都是替人拿主意,都不写。""" """同一次裁决里,一个指标采纳了按下单日、另一个采纳了按支付日:一张表只有一个业务日期,写哪个都是替人拿主意,
都不写。"""
name, source_id = shop name, source_id = shop
out = await record_choices([candidate(BY_ORDER_DATE, RIGHT, "a", metric="销售额", source=name), out = await record_choices([candidate(BY_ORDER_DATE, RIGHT, "a", metric="销售额", source=name),
candidate(BY_ORDER_DATE, RIGHT, "b", metric="复核销售额", source=name)], candidate(BY_ORDER_DATE, RIGHT, "b", metric="复核销售额", source=name)],
@@ -367,10 +378,94 @@ async def test_contradictory_choices_in_one_decision_write_nothing(shop):
assert out["written"] == [] assert out["written"] == []
[skipped] = out["skipped"] [skipped] = out["skipped"]
assert "「销售额」" in skipped["reason"] and "「复核销售额」" in skipped["reason"] assert "「销售额」" in skipped["reason"] and "「复核销售额」" in skipped["reason"]
assert "order_date" in skipped["reason"] and "paid_date" in skipped["reason"] and "互相矛盾" in skipped["reason"] assert "order_date" in skipped["reason"] and "paid_date" in skipped["reason"]
assert "给出了不同的说法" in skipped["reason"] and "无法确定哪一个是数据事实" in skipped["reason"]
assert await entry_of(source_id) is None assert await entry_of(source_id) is None
async def test_complementary_valid_filters_in_one_decision_are_combined(shop):
"""同一次裁决里,一个指标采纳了排除退款、另一个采纳了排除测试单:两条都在排除无效记录,各有人的选择撑着,
不互相矛盾,用 AND 合成一句写。"""
name, source_id = shop
out = await record_choices([
candidate(SUM_WEEK, extra("status <> 'refunded'"), "b", metric="销售额", source=name),
candidate(SUM_WEEK, extra("is_test = 0"), "b", metric="订单金额", source=name),
# 第三个指标采纳的是和第一个一样的条件:合并时不重复
candidate(COUNT_WEEK, extra("status <> 'refunded'", COUNT_WEEK), "b", metric="订单数", source=name),
], run_id="run-1", node_title="复核", actor=None)
assert out == {"written": [{"source": name, "table": "orders", "item": "valid_filter"}], "skipped": []}
entry = await entry_of(source_id)
item = entry.notes["valid_filter"]
assert item["value"] == "status <> 'refunded' AND is_test = 0" and entry.version == 1
assert (item["source"], item["status"]) == ("human", "confirmed")
assert all(f"「{m}」" in item["note"] for m in ("销售额", "订单金额", "订单数")) and "run-1" in item["note"]
async def test_combined_filters_extend_an_inferred_condition_every_adopted_side_uses(shop):
"""目录里推断的「channel <> 'test'」:几个被采纳的一边都按它筛了,合起来写;有一个没按它筛的,关系拿不准,不动。"""
name, source_id = shop
await seed(source_id, {"valid_filter": catalog.make_item("channel <> 'test'", "llm")})
base = extra("channel <> 'test'")
both = [candidate(base, extra("status <> 'refunded'", base), "b", metric="销售额", source=name),
candidate(base, extra("is_test = 0", base), "b", metric="订单金额", source=name)]
out = await record_choices(both, run_id="run-1", node_title="复核", actor=None)
assert out["written"] == [{"source": name, "table": "orders", "item": "valid_filter"}], out
assert (await entry_of(source_id)).notes["valid_filter"]["value"] == \
"channel <> 'test' AND status <> 'refunded' AND is_test = 0"
await seed_again(source_id, {"valid_filter": catalog.make_item("channel <> 'test'", "llm")})
one_without = [both[0], candidate(SUM_WEEK, extra("is_test = 0"), "b", metric="订单金额", source=name)]
out = await record_choices(one_without, run_id="run-2", node_title="复核", actor=None)
assert out["written"] == []
reason = out["skipped"][0]["reason"]
assert "「channel <> 'test'」" in reason and "指标「订单金额」被采纳的盲算没有按它筛选" in reason and "未改动" in reason
assert (await entry_of(source_id)).notes["valid_filter"]["value"] == "channel <> 'test'"
async def test_the_same_choice_from_two_metrics_is_written_once(shop):
name, source_id = shop
out = await record_choices([candidate(BY_ORDER_DATE, RIGHT, "b", metric="销售额", source=name),
candidate(BY_ORDER_DATE, RIGHT, "b", metric="订单金额", source=name)],
run_id="run-1", node_title="复核", actor=None)
assert out == {"written": [{"source": name, "table": "orders", "item": "business_date"}], "skipped": []}
entry = await entry_of(source_id)
assert entry.version == 1 and "「销售额」" in entry.notes["business_date"]["note"]
assert "「订单金额」" in entry.notes["business_date"]["note"]
# --------------------------------------------------------------------------
# 审批卡上的预告:选哪一边各会写什么(和真写同一套判断)
# --------------------------------------------------------------------------
def test_the_preview_says_what_each_choice_would_write():
picked = candidate(BY_ORDER_DATE, RIGHT, "a")
assert preview(picked, SCHEMA, {}) == {
"a": {"write": True, "source": "shop", "table": "orders", "item": "business_date",
"value": {"column": "order_date"}},
"b": {"write": True, "source": "shop", "table": "orders", "item": "business_date",
"value": {"column": "paid_date"}},
}
# 只有被采纳的一边有条件时才写:选另一边照实说为什么不写
said = preview(candidate(WITH_REFUNDS, RIGHT, "a"), SCHEMA, {})
assert said["a"]["write"] is False and "被采纳的原流程没有这个条件" in said["a"]["reason"]
assert said["b"] == {"write": True, "source": "shop", "table": "orders", "item": "valid_filter",
"value": "status <> 'refunded'"}
def test_the_preview_reads_the_catalog_as_it_is():
"""目录里已有的说法照写入时的规矩算:确认过的不覆盖、同样的说法不重写、推断的有效记录条件合起来写。"""
confirmed = {"orders": {"business_date": catalog.make_item({"column": "order_date"}, "human", "confirmed")}}
said = preview(candidate(BY_ORDER_DATE, RIGHT, "a"), SCHEMA, confirmed)
assert said["a"]["write"] is False and "无需重复写入" in said["a"]["reason"]
assert said["b"]["write"] is False and "已有人工确认过的说法「order_date」" in said["b"]["reason"]
inferred = {"orders": {"valid_filter": catalog.make_item("is_test = 0", "llm")}}
base = f"SELECT SUM(amount) AS gmv FROM orders WHERE is_test = 0 AND paid_date {WEEK}"
said = preview(candidate(base, extra("status <> 'refunded'", base), "a"), SCHEMA, inferred)
assert said["b"]["write"] is True and said["b"]["value"] == "is_test = 0 AND status <> 'refunded'"
async def test_a_missing_source_or_an_unrecordable_choice_is_reported(shop): async def test_a_missing_source_or_an_unrecordable_choice_is_reported(shop):
name, _ = shop name, _ = shop
out = await record_choices([candidate(BY_ORDER_DATE, RIGHT, "b", source="no_such_source"), out = await record_choices([candidate(BY_ORDER_DATE, RIGHT, "b", source="no_such_source"),
+143 -7
View File
@@ -4,13 +4,15 @@
- 一致:不挂起,产出 agreed,日志 dual_check_agreed;盲算的查询发 tool.end(role: verify),台账只有一条 dual_check, - 一致:不挂起,产出 agreed,日志 dual_check_agreed;盲算的查询发 tool.end(role: verify),台账只有一条 dual_check,
用量计入运行;盲算 agent 的任务里没有原 SQL,也没有原流程结果的列名 用量计入运行;盲算 agent 的任务里没有原 SQL,也没有原流程结果的列名
- 一个都没核成是 unchecked、只核成一部分是 partial,日志 dual_check_unchecked 说清哪几个、为什么 - 一个都没核成是 unchecked、只核成一部分是 partial,日志 dual_check_unchecked 说清哪几个、为什么
- 不一致:挂起,审批 payload 和对照工件按约定;a / b / neither 三种选择的产出和缺口,没选的照实记,驳回、on_reject - 不一致:挂起,审批 payload 和对照工件按约定,payload 另带实际生效的 on_reject、摘要里每个指标带选 a、选 b 各会往
为 fail 时失败 数据目录里写什么的预告;a / b / neither 三种选择的产出和缺口,没选的照实记,驳回、on_reject 为 fail 时失败
- 差异说明只比这一格所在的那一列(一条查询算好几列、分组后取一行的,收窄了再比) - 差异说明只比这一格所在的那一列(一条查询算好几列、分组后取一行的,收窄了再比,HAVING 一并去掉);分组的列设了
遮罩时不收窄,遮罩列的取值不进审批卡、对照工件、产出和事件
- 只差一个方面时写数据目录(业务日期、有效记录条件),差了不止一处不写,人工确认过的不同说法不覆盖,指标自己的 - 只差一个方面时写数据目录(业务日期、有效记录条件),差了不止一处不写,人工确认过的不同说法不覆盖,指标自己的
筛选范围不写,两条 SQL 规范化后一致的不写 筛选范围(取值在运行输入里、用到分组的列)不写,两条 SQL 规范化后一致的不写
- 全局关闭直通、节点覆盖;设置按发起时的快照;重放时盲算不再执行、事件不重复;报告的写作目录不含盲算的查询 - 全局关闭直通、节点覆盖;设置按发起时的快照;重放时盲算不再执行、事件不重复;报告的写作目录不含盲算的查询
- 触顶、分组结果的维度(只给取值、遮罩列不给)、两个指标用到同一格只盲算一次、遮罩列上的盲算结果不采用 - 触顶、分组结果的维度(只给取值、遮罩列不给)、带着取舍的列名不交给盲算、两个指标用到同一格只盲算一次、遮罩列上的
盲算结果不采用
""" """
from __future__ import annotations from __future__ import annotations
@@ -234,6 +236,23 @@ async def test_the_blind_agent_is_not_told_which_date_column_the_original_groupe
assert "order_date" not in task and "gmv" not in task assert "order_date" not in task and "gmv" not in task
async def test_column_names_carrying_a_choice_are_not_given_as_hints(monkeypatch, shop):
"""几项输入都来自查询、又没写口径说明时,列名才交给盲算区分计算式里的几项;列名里带着原流程的取舍的(按下单日、
已支付)不给:「gmv_by_order_date」等于告诉它按下单日归属。看不出取舍的(total)照给。"""
name, _ = shop
wide = (f"SELECT SUM(amount) AS gmv_by_order_date, COUNT(*) AS paid_orders, SUM(amount) AS total FROM orders "
f"WHERE status <> 'refunded' AND order_date {WEEK}")
aov = {"id": "aov", "name": "客单价", "unit": "元", "decimals": 2, "format": "plain",
"expression": "(cell(nodes.fetch, 0, 'gmv_by_order_date') + cell(nodes.fetch, 0, 'total')) "
"/ cell(nodes.fetch, 0, 'paid_orders')"}
blind = Blind(monkeypatch, name, {}) # 盲算都交不出数:这里只看任务原文
await start(graph(name, wide, metrics=[aov], dual={"definitions": {}}))
assert len(blind.tasks) == 3
said = "\n".join(blind.tasks)
assert "gmv_by_order_date" not in said and "paid_orders" not in said
assert "「total」" in said
async def test_two_metrics_using_one_cell_are_checked_once(monkeypatch, shop): async def test_two_metrics_using_one_cell_are_checked_once(monkeypatch, shop):
name, _ = shop name, _ = shop
share = {"id": "gmv_k", "name": "销售额(千元)", "unit": "千元", "decimals": 2, "format": "thousands", share = {"id": "gmv_k", "name": "销售额(千元)", "unit": "千元", "decimals": 2, "format": "thousands",
@@ -352,16 +371,25 @@ async def test_disagreement_pauses_with_the_agreed_payload_and_card(monkeypatch,
approval = await pending(run.id) approval = await pending(run.id)
assert approval.mode == "input" and approval.node_id == "dual" assert approval.mode == "input" and approval.node_id == "dual"
payload = approval.payload payload = approval.payload
assert set(payload) == {"kind", "node_id", "mode", "title", "card_artifact", "summary"} # 约定的六个键,另加 on_reject(实际生效的处置):审核人在卡上要看得出没采用原流程会不会让运行失败
assert set(payload) == {"kind", "node_id", "mode", "title", "card_artifact", "summary", "on_reject"}
assert payload["kind"] == "dual_check" and payload["node_id"] == "dual" and payload["mode"] == "input" assert payload["kind"] == "dual_check" and payload["node_id"] == "dual" and payload["mode"] == "input"
assert payload["on_reject"] == "degrade"
[item] = payload["summary"] [item] = payload["summary"]
assert set(item) == {"metric_id", "name", "unit", "original_rendered", "blind_rendered", "delta_pct", "diff", assert set(item) == {"metric_id", "name", "unit", "original_rendered", "blind_rendered", "delta_pct", "diff",
"fallback"} "fallback", "catalog"}
assert (item["metric_id"], item["name"], item["unit"]) == ("gmv", "销售额", "元") assert (item["metric_id"], item["name"], item["unit"]) == ("gmv", "销售额", "元")
assert item["original_rendered"].startswith("310") and item["blind_rendered"].startswith("330") assert item["original_rendered"].startswith("310") and item["blind_rendered"].startswith("330")
assert item["delta_pct"] == round((330 - 310) / 310 * 100, 2) assert item["delta_pct"] == round((330 - 310) / 310 * 100, 2)
assert item["fallback"] is False and len(item["diff"]) == 1 assert item["fallback"] is False and len(item["diff"]) == 1
assert "order_date" in item["diff"][0] and "paid_date" in item["diff"][0] and "原流程" in item["diff"][0] assert "order_date" in item["diff"][0] and "paid_date" in item["diff"][0] and "原流程" in item["diff"][0]
# 选哪一边会往数据目录里写什么,按发起审批时的目录预告:只差时间列一处,选哪边就记哪一列
assert item["catalog"] == {
"a": {"write": True, "source": name, "table": "orders", "item": "business_date",
"value": {"column": "order_date"}},
"b": {"write": True, "source": name, "table": "orders", "item": "business_date",
"value": {"column": "paid_date"}},
}
card = artifact_store.load(payload["card_artifact"]) card = artifact_store.load(payload["card_artifact"])
assert set(card) == {"metrics", "model", "same_model"} and card["same_model"] is False assert set(card) == {"metrics", "model", "same_model"} and card["same_model"] is False
@@ -470,6 +498,8 @@ async def test_on_reject_fail_stops_the_run(monkeypatch, shop, client):
name, source_id = shop name, source_id = shop
Blind(monkeypatch, name, {"销售额": (RIGHT, "gmv")}) Blind(monkeypatch, name, {"销售额": (RIGHT, "gmv")})
run = await start(graph(name, BY_ORDER_DATE, dual={"on_reject": "fail"})) run = await start(graph(name, BY_ORDER_DATE, dual={"on_reject": "fail"}))
# 审批卡上看得出:这一步没采用原流程,运行就会停下
assert (await pending(run.id)).payload["on_reject"] == "fail"
done = await decide(client, run.id, {"gmv": "b"}) done = await decide(client, run.id, {"gmv": "b"})
assert done.status == "failed" and "审核人没有采用原流程的结果" in (done.error or "") and "销售额" in done.error assert done.status == "failed" and "审核人没有采用原流程的结果" in (done.error or "") and "销售额" in done.error
evs = await events(run.id) evs = await events(run.id)
@@ -546,6 +576,10 @@ async def test_a_confirmed_catalog_entry_is_not_overwritten(monkeypatch, shop, c
actor="数据组") actor="数据组")
Blind(monkeypatch, name, {"销售额": (RIGHT, "gmv")}) Blind(monkeypatch, name, {"销售额": (RIGHT, "gmv")})
run = await start(graph(name, BY_ORDER_DATE)) run = await start(graph(name, BY_ORDER_DATE))
# 预告按目录里已有的说法算:选原流程是同一个说法、不用再写,选盲算不会覆盖已确认的
preview = (await pending(run.id)).payload["summary"][0]["catalog"]
assert preview["a"]["write"] is False and "无需重复写入" in preview["a"]["reason"]
assert preview["b"]["write"] is False and "已有人工确认过的说法「order_date」" in preview["b"]["reason"]
done = await decide(client, run.id, {"gmv": "b"}) done = await decide(client, run.id, {"gmv": "b"})
[wrote] = logs(await events(done.id), "dual_check_catalog") [wrote] = logs(await events(done.id), "dual_check_catalog")
assert wrote["written"] == [] assert wrote["written"] == []
@@ -564,6 +598,9 @@ async def test_a_metric_scope_filter_is_not_written_as_a_valid_filter(monkeypatc
run = await start(graph(name, original, metrics=[east], dual={"definitions": {"east": "华东地区已支付订单的金额"}})) run = await start(graph(name, original, metrics=[east], dual={"definitions": {"east": "华东地区已支付订单的金额"}}))
[item] = (await pending(run.id)).payload["summary"] [item] = (await pending(run.id)).payload["summary"]
assert len(item["diff"]) == 1 and "region = '华东'" in item["diff"][0] assert len(item["diff"]) == 1 and "region = '华东'" in item["diff"][0]
# 预告和真写的是同一套判断:两边选哪个都不写,原因照实
assert item["catalog"]["a"]["write"] is False and "可能是指标自己的筛选范围" in item["catalog"]["a"]["reason"]
assert item["catalog"]["b"]["write"] is False and "被采纳的盲算没有这个条件" in item["catalog"]["b"]["reason"]
done = await decide(client, run.id, {"east": "a"}) done = await decide(client, run.id, {"east": "a"})
assert done.status == "succeeded", done.error assert done.status == "succeeded", done.error
[wrote] = logs(await events(run.id), "dual_check_catalog") [wrote] = logs(await events(run.id), "dual_check_catalog")
@@ -631,6 +668,61 @@ async def test_a_grouped_query_is_compared_on_the_row_it_was_taken_from(monkeypa
assert (await notes_of(source_id)).notes["business_date"]["value"] == {"column": "paid_date"} assert (await notes_of(source_id)).notes["business_date"]["value"] == {"column": "paid_date"}
async def test_a_having_clause_is_dropped_when_narrowing_to_one_row(monkeypatch, shop, client):
"""原流程分组后另有 HAVING:取的这一行既然在结果里,它所在的组就满足 HAVING,对这一格是噪声。收窄时连 HAVING
一起去掉,差异只剩真正的那一处(时间列),写得进目录。"""
name, source_id = shop
by_region = (f"SELECT region, SUM(amount) AS gmv FROM orders WHERE status <> 'refunded' AND order_date {WEEK} "
"GROUP BY region HAVING SUM(amount) > 10 ORDER BY region")
east_paid = RIGHT.replace("WHERE ", "WHERE region = '华东' AND ")
east = {**GMV, "id": "east", "name": "华东销售额"}
Blind(monkeypatch, name, {"华东销售额": (east_paid, "gmv")})
run = await start(graph(name, by_region, metrics=[east], dual={"definitions": {}}))
[item] = (await pending(run.id)).payload["summary"]
assert item["original_rendered"].startswith("200") and item["blind_rendered"].startswith("300")
assert len(item["diff"]) == 1 and "order_date" in item["diff"][0] and "paid_date" in item["diff"][0]
await decide(client, run.id, {"east": "b"})
assert (await notes_of(source_id)).notes["business_date"]["value"] == {"column": "paid_date"}
async def test_a_condition_on_a_grouped_column_is_not_written(monkeypatch, shop, client, tmp_path):
"""原流程按 is_test 分组、口径卡取 is_test = 0 那一行:收窄后原流程多出「is_test = 0」,看着像排除测试单的有效
记录条件,可它用的是原流程分组的列——是这个指标按组取的范围,不写目录。取值 0 太短、不在任何文字里,只有分组的列
拦得下它。"""
name, source_id = shop
db = sqlite3.connect(tmp_path / "shop.db")
db.execute("INSERT INTO orders VALUES (6, '华东', 105, 40.0, 'paid', '2026-09-03', '2026-09-03', 'online', 1)")
db.commit()
db.close()
by_flag = (f"SELECT is_test, SUM(amount) AS gmv FROM orders WHERE status <> 'refunded' AND paid_date {WEEK} "
"GROUP BY is_test ORDER BY is_test")
Blind(monkeypatch, name, {"销售额": (RIGHT, "gmv")})
run = await start(graph(name, by_flag, dual={"definitions": {}}))
[item] = (await pending(run.id)).payload["summary"]
assert item["original_rendered"].startswith("330") and item["blind_rendered"].startswith("370")
assert len(item["diff"]) == 1 and "is_test = 0" in item["diff"][0]
done = await decide(client, run.id, {"gmv": "a"})
[wrote] = logs(await events(done.id), "dual_check_catalog")
assert wrote["written"] == [] and "分组的列" in wrote["skipped"][0]["reason"]
assert await notes_of(source_id) is None
async def test_a_condition_matching_a_run_input_is_not_written(monkeypatch, shop, client):
"""排除退款的条件看着像有效记录条件,可取值正是本次运行的输入(这一期要排除的状态):是这个指标这一期的筛选范围,
不是整张表的事实,不写目录。没有这个输入时同样的选择会写(test_a_refund_filter_becomes_the_valid_row_filter)。"""
name, source_id = shop
spec = graph(name, RIGHT, dual={"definitions": {}})
spec["nodes"][0]["data"]["config"]["fields"] = [{"name": "exclude_status"}]
Blind(monkeypatch, name, {"销售额": (WITH_REFUNDS, "gmv")})
run = await start(spec, {"exclude_status": "refunded"})
[item] = (await pending(run.id)).payload["summary"]
assert len(item["diff"]) == 1 and "refunded" in item["diff"][0]
done = await decide(client, run.id, {"gmv": "a"})
[wrote] = logs(await events(done.id), "dual_check_catalog")
assert wrote["written"] == [] and "本次运行的输入" in wrote["skipped"][0]["reason"]
assert await notes_of(source_id) is None
async def test_two_differing_inputs_do_not_write_the_catalog(monkeypatch, shop, client): async def test_two_differing_inputs_do_not_write_the_catalog(monkeypatch, shop, client):
"""一个指标的两个输入都对不上:审核人选的是整个指标哪边对,拆不出是哪一处,不写目录。""" """一个指标的两个输入都对不上:审核人选的是整个指标哪边对,拆不出是哪一处,不写目录。"""
name, source_id = shop name, source_id = shop
@@ -733,6 +825,50 @@ async def test_masked_group_columns_are_not_shown_to_the_blind_agent(monkeypatch
assert "华东" not in task and "region" not in task assert "华东" not in task and "region" not in task
async def test_masked_group_values_stay_out_of_the_card_and_the_catalog_log(monkeypatch, masked_shop, client):
"""原流程按遮罩列(region)分组、口径卡取华东那一行:差异说明收窄到这一行时,不能拿这一行的分组值补等值条件——
那样「region = '华东'」就进了审批卡、对照工件、事件和目录日志。遮罩的分组列上不收窄,照整条比。"""
name, _ = masked_shop
first = {**GMV, "id": "first", "name": "首组销售额", "expression": "cell(nodes.fetch, 0, 'gmv')"}
Blind(monkeypatch, name, {"首组销售额": (RIGHT, "gmv")})
run = await start(graph(name, BY_REGION, metrics=[first], dual={"definitions": {}}))
assert run.status == "interrupted", run.error
payload = (await pending(run.id)).payload
card = artifact_store.load(payload["card_artifact"])
done = await decide(client, run.id, {"first": "a"})
assert done.status == "succeeded", done.error
evs = await events(run.id)
_no_masked_value(payload, card, done, evs)
# 照整条比:原流程分组、盲算没分组都列出来,不止差一个方面,选了也不写目录
assert len(payload["summary"][0]["diff"]) > 1
[wrote] = logs(evs, "dual_check_catalog")
assert wrote["written"] == [] and "不止差一个方面" in wrote["skipped"][0]["reason"]
async def test_a_masked_group_column_of_the_blind_query_is_not_narrowed_either(monkeypatch, masked_shop, client):
"""盲算自己按遮罩列分组、交了华东那一行:同样不拿这一行的分组值改写盲算的 SQL。"""
name, _ = masked_shop
Blind(monkeypatch, name, {"销售额": (BY_REGION, "gmv")})
run = await start(graph(name, BY_ORDER_DATE))
assert run.status == "interrupted", run.error
payload = (await pending(run.id)).payload
assert payload["summary"][0]["blind_rendered"].startswith("300")
card = artifact_store.load(payload["card_artifact"])
done = await decide(client, run.id, {"gmv": "b"})
assert done.status == "succeeded", done.error
_no_masked_value(payload, card, done, await events(run.id))
def _no_masked_value(payload: dict, card: dict, done: Run, evs: list[RunEvent], value: str = "华东") -> None:
"""遮罩列上的取值不出现在审批 payload、对照工件、节点产出和节点发的事件里。查询本身的 tool.end、模型的往来按
设计带着原始结果(证据面板显示时再遮),不在此列。"""
seen = {"payload": payload, "card": card, "output": dual_output(done, evs),
"events": [e.data for e in evs if (e.node_id == "dual" or e.type.startswith("human.") or e.type == "log")
and not e.type.startswith(("tool.", "llm."))]}
for where, content in seen.items():
assert value not in json.dumps(content, ensure_ascii=False), where
# -------------------------------------------------------------------------- # --------------------------------------------------------------------------
# 设置:全局开关、节点覆盖、发起时的快照 # 设置:全局开关、节点覆盖、发起时的快照
# -------------------------------------------------------------------------- # --------------------------------------------------------------------------