运行对比复核修正(第 2 轮):行段相同、一边是整份结果的引用不再算同一处,默认设置的变化只落到会读它的节点上,上游 Agent 交出的结果不同时不再说变化来自上游数据

- Agent 两次引用同一段行,一边是整份结果、一边不是(这次多了北区,没引用的西区还在快照里):判为无法比较,写明哪边引用了整份结果、另一边引用了哪几行、结果共几行;快照读不出来、核对不了是不是整份结果的同样不算同一处。以前按同一处算,西区被算成「数据没了」,占比超过 100%
- 图级默认设置的变化按「哪类节点运行时会读这一项」落到节点上(_CFG_KEYS,照各执行器经 NodeContext.cfg 读的配置项登记):结论句裁判、结论句的处理只有报告节点读,不再把 Agent 记成配置变了;条件分支让模型分类、结构校验让模型修复、知识检索用模型重排时也读默认模型,同样落到它们身上。新测试按执行器源码核对这张表,执行器新读了一项没登记就失败
- 代码、数据整形这类节点的上游有 Agent、模型调用、人工审批等节点,两次交出的产出不同(工件按内容寻址,按工件 id 比;产出没落成工件的核对不了):它交的数变了不再说变化来自上游数据,判为无法比较并点名是哪个节点
- 上游节点改了配置时说明里点名是哪个节点;默认设置那句拆短,写出跟着变的是哪几项(模型接入不写 provider)
- 补测试守住复核列出的未覆盖规则:容器运行输入还原不了、合并查询的输入追不下去、上游只有运行输入变了、审批默认设置、图里 SQL 只改格式(代码节点在下游)、跨节点先配字面量相同的、少了一次查询、只有一边落在快照上、查过库的写入节点改了配置、代码节点上游有节点改了配置、旧口径叶子的出处说不清时不拆、单格引用换了行;变异 99 处全部打红
This commit is contained in:
ThearchyHelios
2026-10-05 11:28:41 +08:00
parent d176ff77ff
commit 3254744f6d
3 changed files with 486 additions and 41 deletions
+155 -28
View File
@@ -22,12 +22,13 @@
- cell() 那一格、Agent 按出处交来的字段落在哪份快照上就是哪条查询,两边各自的快照直接比 SQL;快照是合并查询的
结果时,再按别名一路追到喂它的每一份上游快照。只差字面量时还要看图里的 SQL:图里的 SQL 改过(改了写死的筛选值、
系数)是模板变化,图里一字未改的才是运行时代入的参数
- Agent 按出处交来的字段还要引用同一列、同一段行(两边都引用整份结果的不算换了):取哪一列、哪几行是 Agent 每次
运行时定的,换了就不是同一个数
- Agent 按出处交来的字段还要引用同一列、同一段行(两边都引用整份结果的不算换了;行段相同、一边是整份结果一边
不是的算换了):取哪一列、哪几行是 Agent 每次运行时定的,换了就不是同一个数
- 不落在快照上的:模型、人交出的值(Agent 没按出处交、模型调用、人工审批……)没有出处,值变了说不清是数据还是它
本身的差异;代码、数据整形这类节点看它自己查过的库,不查库的看它的配置和画布上的全部上游,上游的查询结果两次
一模一样、值却变了的同样说不清
- 图级的变化落到节点上:连线有增删的,终点节点算改了;默认设置改了的,没有自行设置这几项的模型类节点算改了
本身的差异;代码、数据整形这类节点看它自己查过的库,不查库的看它的配置和画布上的全部上游。上游的查询结果两次
一模一样、值却变了的同样说不清;上游有模型、人交出值的节点、两次交出的不同的也说不清
- 图级的变化落到节点上:连线有增删的,终点节点算改了;默认设置改了的,运行时会读这几项(_CFG_KEYS)、又没有自行
设置的节点算改了
- 严格拆分(只对口径变了的):旧表达式的每个叶子路径都能在这次的口径卡输入里取得值时,
口径效应 = f_new(x_new) − f_old(x_new),数据效应 = f_old(x_new) − f_old(x_old)。f_old(x_new) 用推演重算
(engine/recompute.py)的还原和求值算,取整按旧口径卡的小数位;三个数按十进制精确相减,两效应之和逐位等于
@@ -88,12 +89,49 @@ _RATIO = "percent_of_ratio"
#: 图里写 SQL 的配置项:调用工具节点的参数、合并查询的合并 SQL
_SQL_KEY = {"tool": "args.sql", "merge": "sql"}
#: 交出的值由模型或人给出的节点(类型、口径卡记的来历):同样的输入、同样的配置,交出的值也可能不同
_GENERATED = frozenset({"agent", "llm", "supervisor", "human", "memory", "retrieve", "subgraph"})
_GENERATED_VIA = frozenset({"agent", "agent_field", "llm", "team", "human", "memory", "retrieval", "subgraph"})
#: 图级默认设置只有模型类节点在读(节点没配 provider、model 时回退到它,GraphSpec.defaults)
_MODEL_TYPES = frozenset({"llm", "agent", "supervisor"})
_GENERATED = frozenset({"agent", "llm", "supervisor", "human", "memory", "retrieve", "subgraph", "report"})
_GENERATED_VIA = frozenset({"agent", "agent_field", "llm", "team", "human", "memory", "retrieval", "subgraph",
"report"})
#: 调模型时读的配置项(nodes/llm._model_spec)
_MODEL_KEYS = frozenset({"provider", "model", "temperature", "max_tokens", "thinking", "effort"})
#: 各类节点运行时按 NodeContext.cfg 读的配置项:节点上没写这一项就回退到图级默认设置(GraphSpec.defaults),默认
#: 设置改了这一项,没自行设置它的这类节点交出的值可能跟着变;不读它的不受影响(结论句裁判只有报告节点读)。读了也
#: 不影响交出的值的见 _QUIET_DEFAULTS。合并查询、双盲复算、反方审稿不经 cfg 读配置。执行器新读了一项要在这里登记,
#: tests/test_compare_engine.py 按源码核对
_CFG_KEYS: dict[str, frozenset[str]] = {
"input": frozenset({"fields"}),
"output": frozenset({"fields", "contract"}),
"transform": frozenset({"mode", "expression", "template", "assign_to"}),
"llm": _MODEL_KEYS | {"system", "skills", "use_history", "prompt", "output_schema", "emit_message", "assign_to"},
"agent": _MODEL_KEYS | {"system", "skills", "use_history", "prompt", "output_schema", "cite_fields", "tools",
"memory_scope", "collection", "parallel_tools", "max_steps", "budget_tokens",
"budget_usd", "approval", "emit_message", "assign_to"},
"supervisor": frozenset({"agents", "max_rounds", "on_exhausted", "goal", "provider", "model", "max_tokens",
"max_parallel", "approval", "emit_message", "assign_to"}),
"subgraph": frozenset({"workflow_id", "workflow_version", "input", "assign_to"}),
"tool": frozenset({"tool", "args", "fail_fast", "memory_scope", "collection", "approval", "assign_to"}),
"code": frozenset({"code", "evidence_role", "language", "timeout", "memory_mb", "network", "isolation", "files",
"fail_fast", "approval", "assign_to"}),
"branch": frozenset({"cases", "mode"}),
"loop": frozenset({"mode", "max_iterations", "item_var", "items", "condition"}),
"human": frozenset({"mode", "title", "message", "schema", "context", "draft", "stop_on_reject", "assign_to"}),
"validate": frozenset({"schema", "source", "max_retries", "repair_with_llm", "fail_fast", "assign_to"}),
"memory": frozenset({"action", "scope", "query", "limit", "kind", "content", "importance", "assign_to"}),
"retrieve": frozenset({"query", "collection", "limit", "alpha", "rerank", "min_score", "assign_to"}),
"metrics": frozenset({"metrics", "caliber", "caliber_version", "on_missing", "assign_to"}),
"report": _MODEL_KEYS | {"numbers", "claims", "judge", "entities", "max_repairs", "system", "instructions",
"on_violation", "evidence_from", "metrics_from", "emit_message", "assign_to"},
}
#: 只在调模型时才读的(_calls_model):条件分支让模型分类、结构校验让模型修复(默认开着)、知识检索用模型重排
_CFG_KEYS_IF: dict[str, frozenset[str]] = {
"branch": frozenset({"input", "instruction", "provider", "model"}),
"validate": frozenset({"provider", "model"}),
"retrieve": _MODEL_KEYS,
}
#: 改了也不影响交出的值的默认设置:审批策略只管调用工具前问不问人
_QUIET_DEFAULTS = frozenset({"approval"})
#: 配置键的叫法,engine/labels 里没有的这两个:节点类型;模型服务叫「模型接入」(lib/terms.ts),不写 provider
_KEY_LABEL = {"type": "节点类型", "provider": "模型接入"}
#: 浮点能原样写回的十进制有效数字位数(DBL_DIG)
_FLOAT_DIGITS = 15
@@ -267,6 +305,16 @@ def _queries(side: Side) -> list[_Query]:
return out
def _outputs(side: Side) -> dict[str, tuple[Any, ...]]:
"""每个节点在封存范围内交出的产出工件(node.finished 的 artifact),按执行的先后。工件按内容寻址:两次运行的同一个
节点交出的一模一样,工件 id 就一样。产出没落成工件的记 None,核对不了。"""
out: dict[str, list[Any]] = {}
for _, etype, node_id, data in side.events:
if etype == EventType.NODE_FINISHED and node_id and isinstance(data, dict):
out.setdefault(node_id, []).append(data.get("artifact"))
return {node_id: tuple(found) for node_id, found in out.items()}
def _pair(base: list[_Query], target: list[_Query],
norm: Callable[[_Query], NormalizedSql | None]) -> list[tuple[_Query | None, _Query | None]]:
"""配对,一轮比一轮宽:
@@ -563,7 +611,7 @@ def _config_keys(was: dict[str, Any], now: dict[str, Any]) -> tuple[list[str], b
def _key_label(key: str, node_type: Any) -> str:
return "节点类型" if key == "type" else field_label(key, node_type)
return _KEY_LABEL.get(key) or field_label(key, node_type)
def _unset(value: Any) -> bool:
@@ -571,6 +619,29 @@ def _unset(value: Any) -> bool:
return value in (None, "")
def _reads(node: dict[str, Any], defaults: dict[str, Any]) -> frozenset[str]:
"""这个节点运行时按 NodeContext.cfg 读的配置项(_CFG_KEYS)。条件分支、结构校验、知识检索调不调模型看它那次的
设置,调模型的才读模型那几项。"""
kind = str(node.get("type"))
keys = _CFG_KEYS.get(kind, frozenset())
if kind in _CFG_KEYS_IF and _calls_model(kind, node, defaults):
keys = keys | _CFG_KEYS_IF[kind]
return keys
def _calls_model(kind: str, node: dict[str, Any], defaults: dict[str, Any]) -> bool:
"""条件分支是让模型分类、结构校验让模型修复(默认开着)、知识检索用模型重排:和执行器同一个判断、同一个缺省。"""
def cfg(key: str, default: Any) -> Any:
value = _config(node).get(key)
return defaults.get(key, default) if _unset(value) else value
if kind == "branch":
return cfg("mode", "expression") == "llm"
if kind == "validate":
return bool(cfg("repair_with_llm", True))
return (cfg("rerank", "off") or "off") == "model"
def _graph_sql(node: dict[str, Any] | None) -> tuple[str, str | None] | None:
"""节点在图里写的 SQL 和它的数据源:调用工具节点是参数里的 SQL(数据源从工具名来),合并查询是合并 SQL。
别的节点(Agent 自己写 SQL)返回 None。"""
@@ -641,9 +712,12 @@ class _Compare:
self.base_nodes, self.target_nodes = _nodes(base.graph), _nodes(target.graph)
self.base_queries, self.target_queries = _queries(base), _queries(target)
self.pairs = _pair(self.base_queries, self.target_queries, self._norm_of)
self.template, self.changed = self._template()
self.template, self.changed, self.inherited = self._template()
self.titles = {**{k: self._title_of(n) for k, n in self.base_nodes.items()},
**{k: self._title_of(n) for k, n in self.target_nodes.items()}}
#: 画布上的先后(这次运行的图在前):说明里点名上游节点时按它挑第一个
self.order = {n: i for i, n in enumerate(dict.fromkeys([*self.target_nodes, *self.base_nodes]))}
self.base_outputs, self.target_outputs = _outputs(base), _outputs(target)
self.parents: dict[str, set[str]] = {}
for source, target_id, _ in _edges(base.graph) | _edges(target.graph):
self.parents.setdefault(target_id, set()).add(source)
@@ -865,9 +939,25 @@ class _Compare:
def _moved(self, node_id: str) -> str:
"""一个影响取值的节点变化(self.changed)写成一句话。"""
name = f"上游节点「{self.title(node_id)}」"
return {"added": f"{name}是这次新增的", "removed": f"{name}这次被删除了", "edges": f"{name}的连线有变化",
"defaults": f"工作流默认设置有变化,{name}没有单独设置这些项,随之改变"
}.get(self.changed.get(node_id, ""), f"{name}的配置有变化")
change = self.changed.get(node_id, "")
if change == "defaults":
keys = self.inherited.get(node_id, [])
shown = "、".join(_key_label(k, self._type_of(node_id)) for k in keys[:3])
more = f"等 {len(keys)} 项" if len(keys) > 3 else ""
return f"{name}跟随的工作流默认设置({shown}{more})有变化"
return {"added": f"{name}是这次新增的", "removed": f"{name}这次被删除了",
"edges": f"{name}的连线有变化"}.get(change, f"{name}的配置有变化")
def _ordered(self, nodes: Iterable[str]) -> list[str]:
"""按画布上的先后排(这次运行的图在前)。"""
return sorted(nodes, key=lambda n: self.order.get(n, len(self.order)))
def _regenerated(self, node_id: str) -> bool:
"""模型、人交出值的节点(_GENERATED)两次交出的产出不同,或者核对不了。一模一样的没带进它本身的差异。"""
if self._type_of(node_id) not in _GENERATED:
return False
before, after = self.base_outputs.get(node_id, ()), self.target_outputs.get(node_id, ())
return before != after or None in before or None in after
def provenance(self, path: str, b: Any, t: Any, *, moved: bool = True) -> _Change:
"""一个输入的出处在两次运行之间变没变(b、t 是推演重算还原出来的输入)。moved 是这个指标的值变没变:输入是
@@ -904,11 +994,19 @@ class _Compare:
return _Change("template", self._moved(writer))
# 写它的节点不查库(代码、数据整形……):看画布上它的全部上游;写它的节点不知道就看整张图
scope = self._upstream(writer) if writer else set(self.titles)
moved_nodes = [n for n in scope if n in self.changed]
moved_nodes = self._ordered(n for n in scope if n in self.changed)
worst = _worst(self._query_change(b2, t2) for b2, t2 in self.pairs if (t2 or b2).node_id in scope)
where = f"来自「{self.title(writer)}」" if writer else "不知道由哪个节点写入"
if moved_nodes or worst.kind in ("sql", "template", "unknown"):
if moved_nodes:
return _Change("unknown", f"输入「{path}」{where},{self._moved(moved_nodes[0])},无法判断变化来源")
if worst.kind in ("sql", "template", "unknown"):
return _Change("unknown", f"输入「{path}」{where},对不上具体的查询;它的上游有变化,无法判断变化来源")
if not same and (spoken := self._ordered(n for n in scope if self._regenerated(n))):
# 上游有模型、人交出的值,两次不同:它交的数变了,可能是数据变了,也可能是模型本身的差异(它可能用的
# 正是 Agent 没按出处交的数)。上游的查询结果变没变都一样说不清
kind = TYPE_LABEL.get(self._type_of(spoken[0]) or "", "节点")
return _Change("unknown", f"输入「{path}」{where},它的上游节点「{self.title(spoken[0])}」({kind})两次"
"交出的结果不同,无法判断变化是否来自数据")
if worst.kind == "same" and not same and not self._data_moved(scope):
# 上游查到的数据两次一模一样,它交的值却变了:不是数据变的。上游只有运行输入变了的,算参数
if (self.base.inputs or {}) != (self.target.inputs or {}) \
@@ -939,16 +1037,35 @@ class _Compare:
return change
def _citation(self, agent: Any, b: Any, t: Any, qb: _Query, qt: _Query) -> _Change | None:
"""Agent 按出处交来的字段两次引用的是不是同一处:同一列(对象数组是同一组列)、同一行或同一段行。行段不同、但
两边引用的都是整份结果(这次多了几行)的算同一处。不是同一处返回说明。"""
"""Agent 按出处交来的字段两次引用的是不是同一处:同一列(对象数组是同一组列)、同一行或同一段行。两边引用的都是
整份结果的算同一处(这次多了几行也是);行段相同、一边是整份结果一边不是的不算——没引用的那几行还在这次的快照
里,按维度配对会把它们算成「数据没了」。不是同一处、核对不了是不是同一处的,返回说明。"""
name = f"Agent「{self.title(agent)}」" if agent else "Agent"
lb, lt = _locator(b), _locator(t)
cb, ct = lb.get("column") or lb.get("columns"), lt.get("column") or lt.get("columns")
if cb != ct:
return _Change("unknown", f"{name}引用的列由{_columns_text(cb)}改为{_columns_text(ct)},取的不是同一个数")
if (b.row, b.rows) == (t.row, t.rows) or (b.row is None and t.row is None and _whole(b, qb) and _whole(t, qt)):
what = "同一个数" if b.row is not None and t.row is not None else "同一组数"
changed = _Change("unknown", f"{name}引用的行由{_rows_text(b)}改为{_rows_text(t)},取的不是{what}")
if b.row is not None or t.row is not None:
# 单格(数组字段里的一个元素、按出处交的一格):取哪一行由口径卡的下标和 Agent 交的行段一起定,同一行才是
# 同一处
return None if (b.row, b.rows) == (t.row, t.rows) else changed
wb, wt = _whole(b, qb), _whole(t, qt)
same_span = b.rows == t.rows
if (wb and wt) or (same_span and wb is False and wt is False):
return None
return _Change("unknown", f"{name}引用的行由{_rows_text(b)}改为{_rows_text(t)},取的不是同一组数")
if (wb is None or wt is None) and (same_span or (wb is not False and wt is not False)):
# 快照读不出来,核对不了是不是整份结果:行段相同的可能一边整份一边不是,行段不同的也可能两边都是整份
side = "基准运行" if wb is None else "这次运行"
return _Change("unknown", f"{side}的查询快照无法读取,无法核对 {name}引用的是不是整份结果")
if not same_span:
return changed
if wb:
return _Change("unknown", f"{name}在基准运行中引用了整份结果,这次只引用了{_rows_text(t)}"
f"(结果共 {len(qt.snap['rows'])} 行),取的不是同一组数")
return _Change("unknown", f"{name}在基准运行中只引用了{_rows_text(b)}(结果共 {len(qb.snap['rows'])} 行),"
"这次引用了整份结果,取的不是同一组数")
def _generated(self, path: str, writer: str, same: bool) -> _Change:
"""模型、人交出的值(Agent 没按出处交、模型调用、人工审批……):没有出处,值变了说不清是数据变了还是它本身的
@@ -982,15 +1099,17 @@ class _Compare:
# ---------------------------------------------------------------- 模板
def _template(self) -> tuple[dict[str, Any], dict[str, str]]:
def _template(self) -> tuple[dict[str, Any], dict[str, str], dict[str, list[str]]]:
"""两次运行的图快照比较:新增、删除的节点,配置有变化的节点和配置项名(不展示值)。
changed 是影响取值的变化,判断输入的出处时用:节点 → added / removed / config / edges / defaults。只改名称的
不算;图里的 SQL 只改了空白、大小写的不算(规范化以后一模一样)。图级的变化也落到节点上:连线有增删的,终点
节点交出的值可能变了(接进来的东西、执行的条件变了);默认设置改了的,没有自行设置这几项的模型类节点跟着变。"""
节点交出的值可能变了(接进来的东西、执行的条件变了);默认设置改了的,运行时会读这几项(_reads)、又没有自行
设置的节点跟着变,inherited 记着各自跟着变的是哪几项。"""
before, after = self.base_nodes, self.target_nodes
nodes: list[dict[str, Any]] = []
changed: dict[str, str] = {}
inherited: dict[str, list[str]] = {}
def entry(node_id: str, node: dict[str, Any], change: str, keys: list[str]) -> dict[str, Any]:
return {"node_id": node_id, "title": self._title_of(node), "change": change, "keys": keys,
@@ -1023,9 +1142,14 @@ class _Compare:
if defaults_before.get(k) != defaults_after.get(k) and k not in _QUIET_DEFAULTS]
for node_id, node in after.items():
was = before.get(node_id)
if differing and was is not None and str(node.get("type")) in _MODEL_TYPES and any(
_unset(_config(node).get(k)) and _unset(_config(was).get(k)) for k in differing):
if not differing or was is None:
continue
reads = _reads(was, defaults_before) | _reads(node, defaults_after)
hit = [k for k in differing
if k in reads and _unset(_config(node).get(k)) and _unset(_config(was).get(k))]
if hit:
changed.setdefault(node_id, "defaults")
inherited[node_id] = hit
same_edges = edges_before == edges_after
same_defaults = defaults_before == defaults_after
if not same_edges:
@@ -1033,7 +1157,7 @@ class _Compare:
if not same_defaults:
self.notes.append(DEFAULTS_CHANGED)
return {"base_version": self.base.run.get("version"), "target_version": self.target.run.get("version"),
"same_graph": not nodes and same_edges and same_defaults, "nodes": nodes}, changed
"same_graph": not nodes and same_edges and same_defaults, "nodes": nodes}, changed, inherited
# ---------------------------------------------------------------- 指标
@@ -1422,11 +1546,14 @@ def _same_input(b: Any, t: Any) -> bool | None:
return recompute._same(b.item.get("value"), t.item.get("value"))
def _whole(inp: Any, q: _Query) -> bool:
"""这个输入引用的是整份查询结果:数组字段的行段从第一行到最后一行,或者容器本身就是整份结果。"""
def _whole(inp: Any, q: _Query) -> bool | None:
"""这个输入引用的是不是整份查询结果:容器本身就是整份结果,或者数组字段的行段从第一行到最后一行。快照读不出来、
不知道结果共几行的返回 None。"""
if inp.table:
return True
return inp.rows is not None and q.snap is not None and inp.rows == (0, len(q.snap["rows"]) - 1)
if q.snap is None:
return None
return inp.rows is not None and inp.rows == (0, len(q.snap["rows"]) - 1)
def _rows_text(inp: Any) -> str:
+4 -1
View File
@@ -38,10 +38,13 @@ def chain(*nodes):
return {"nodes": list(nodes), "edges": [{"source": a["id"], "target": b["id"]} for a, b in zip(nodes, nodes[1:])]}
def weekly_graph(sql=None, *, between=(), card_label="周报口径卡"):
def weekly_graph(sql=None, *, between=(), after_fetch=(), card_label="周报口径卡"):
"""一条链:开始 → 取数 → [after_fetch] → Agent → [between] → 口径卡 → 写周报 → 成果。after_fetch 里的节点不在
Agent 的下游(它的上游只有取数节点),between 里的在。"""
return chain(
node("start", "input", "开始"),
node("fetch", "tool", "区域取数", tool=TOOL, args={"sql": sql or region_sql()}),
*after_fetch,
node("bot", "agent", "区域分析", prompt="按区域列出金额", tools=[TOOL], cite_fields=True),
*between,
node("card", "metrics", card_label, caliber="周报口径", caliber_version="v1"),
+327 -12
View File
@@ -9,16 +9,20 @@ SQL,Agent 节点 bot 自己再查一次、把各区域的金额作为数组字
为 1;对不上时占比为空并说明;截断的快照不归因,照实写
- 遮罩列的值绝不出现在结果里;用到遮罩列的指标不拆分
- 模板对比:新增、删除的节点,配置有变化的节点和配置项名
- 输入的出处:合并查询追到它的上游查询;Agent 按出处交的字段还要引用同一列、同一段行;没按出处交的数、上游数据
没变数却变了的,说不清变化从哪来(incomparable);图里写死的字面量改了、默认设置和连线变了都算模板变化
- 输入的出处:合并查询追到它的上游查询;Agent 按出处交的字段还要引用同一列、同一段行(行段相同、一边是整份结果
一边不是的不算);没按出处交的数、上游数据没变数却变了的、上游的 Agent 两次交出的结果不同的,说不清变化从哪来
(incomparable);图里写死的字面量改了、连线变了、读到的默认设置变了都算模板变化
"""
from __future__ import annotations
import copy
import inspect
import json
import re
from decimal import Decimal
from app.engine import compare
from app.engine.compiler import RUNNERS
from app.engine.merge_query import MERGE_SOURCE
from tests.fixtures.compare_runs import (
BASE_ROWS,
@@ -70,7 +74,9 @@ def test_same_sql_new_rows_is_data_only():
assert (east["base_rendered"], east["target_rendered"], east["delta_rendered"]) == ("1,200.5元", "1,500.5元", "+300.0元")
assert east["delta_pct"] == round(300 / 1200.5 * 100, 2)
assert east["card_id"] == "card" and east["card_title"] == "周报口径卡" and east["name"] == "东区金额"
assert east["decomposition"] is None and east["decomposition_note"] is None and east["reasons"]
assert east["decomposition"] is None and east["decomposition_note"] is None
# 两个输入都落在查询快照上:直接说取数的 SQL 没变、查询结果变了
assert east["reasons"] == total["reasons"] == [compare.DATA_FROM_QUERIES]
assert total["delta"] == 200.0 and total["delta_rendered"] == "+200.0元"
assert result["summary"] == {"unchanged": 0, "data_only": 2, "caliber_and_data": 0, "sql_changed": 0,
"params_only": 0, "added": 0, "removed": 0, "incomparable": 0}
@@ -558,10 +564,14 @@ def code_metrics(value):
return build
def calc_node(code="print(1)"):
return node("calc", "code", "调整", code=code, assign_to="adj")
def test_an_input_from_a_code_node_follows_its_upstream():
"""代码节点接在取数节点后面(不在 Agent 下游):看它画布上的全部上游。"""
def make(run_id, rows, value, sql=None):
run = Fake(run_id, graph=weekly_graph(sql, between=(node("calc", "code", "调整", code="print(1)",
assign_to="adj"),)))
run = Fake(run_id, graph=weekly_graph(sql, after_fetch=(calc_node(),)))
sql = sql or region_sql()
run.tool("fetch", sql, rows=rows)
run.agent("bot", sql, rows=rows)
@@ -583,11 +593,73 @@ def test_an_input_from_a_code_node_follows_its_upstream():
assert compare.DATA_FROM_UPSTREAM not in calc["reasons"]
def test_a_code_node_after_an_agent_that_answered_differently_is_incomparable():
"""代码节点在 Agent 下游,算的是 Agent 没按出处交的占比:Agent 两次交出的结果不同,变化可能来自数据,也可能来自
模型本身的差异,不能说变化来自上游数据。Agent 两次交出的一模一样时照常(见
test_swapped_queries_with_the_same_literals_pair_up_exactly)。"""
def make(run_id, rows, share, value=None):
value = share * 2 if value is None else value
run = Fake(run_id, graph=weekly_graph(between=(calc_node("result = nodes.bot.data.kpi.share * 2"),)))
run.tool("fetch", region_sql(), rows=rows)
run.plain_agent("bot", [region_sql()], {"kpi": {"share": share}}, rows=rows)
run.code("calc", value, "adj")
run.card(code_metrics(value)(None, None, rows, rows))
return run
calc = by_id(run_compare(make("r1", BASE_ROWS, 0.46), make("r2", NEXT_ROWS, 0.52)))["calc"]
assert calc["classification"] == "incomparable" and calc["attribution"] is None
assert calc["reasons"] == ["输入「vars.adj」来自「调整」,它的上游节点「区域分析」(Agent)两次交出的结果不同,"
"无法判断变化是否来自数据"]
# Agent 的产出两次都没落成工件(写工件失败):一样不一样核对不了,同样不报成数据变化
def unstored(run):
run.events = [(seq, etype, nid, {k: v for k, v in data.items() if k != "artifact"}
if (etype, nid) == ("node.finished", "bot") else data) for seq, etype, nid, data in run.events]
return run
calc = by_id(run_compare(unstored(make("r1", BASE_ROWS, 0.46)),
unstored(make("r2", NEXT_ROWS, 0.46, value=1.0))))["calc"]
assert calc["classification"] == "incomparable" and "区域分析" in calc["reasons"][0]
def test_a_reconfigured_node_upstream_of_a_code_node_is_named():
"""代码节点上游的数据整形节点改了表达式(SQL 一字没改):代码节点交的数变了说不清从哪来,说明里点出改的是哪个节点。"""
def make(run_id, expression, rows, value):
shape = node("shape", "transform", "整理", mode="expression", expression=expression, assign_to="kept")
run = Fake(run_id, graph=weekly_graph(after_fetch=(shape, calc_node())))
run.tool("fetch", region_sql(), rows=rows)
run.agent("bot", region_sql(), rows=rows)
run.code("calc", value, "adj")
run.card(code_metrics(value)(None, None, rows, rows))
return run
calc = by_id(run_compare(make("r1", "nodes.fetch.rows", BASE_ROWS, 10),
make("r2", "nodes.fetch.rows[:2]", NEXT_ROWS, 12)))["calc"]
assert calc["classification"] == "incomparable"
assert calc["reasons"] == ["输入「vars.adj」来自「调整」,上游节点「整理」的配置有变化,无法判断变化来源"]
def test_a_run_input_upstream_of_a_code_node_is_a_parameter_change():
"""代码节点上游查到的数据两次一模一样,只有运行输入变了:它交的数变了算参数变化。"""
def make(run_id, goal, value):
run = Fake(run_id, graph=weekly_graph(after_fetch=(calc_node(),)), inputs={"goal": goal})
run.add("node.started", "start", {"node_type": "input"})
run.add("node.finished", "start", {"artifact": run.put({"goal": goal})})
run.tool("fetch", region_sql())
run.agent("bot", region_sql())
run.code("calc", value, "adj")
run.card(code_metrics(value)(None, None, BASE_ROWS, BASE_ROWS))
return run
calc = by_id(run_compare(make("r1", 1000, 10), make("r2", 1500, 15)))["calc"]
assert calc["classification"] == "params_only"
assert calc["reasons"] == ["输入「vars.adj」来自「调整」,它的上游只有运行输入有变化"]
def test_a_value_from_a_querying_node_that_moved_without_its_data_is_incomparable():
"""调用工具节点整份交出的结果(不是 cell() 取的格):它查到的行两次一样,交出的值(这里是查询耗时)却变了,
变化不来自数据;查到的行也变了的,照常是数据变化。"""
def make(run_id, elapsed, rows=BASE_ROWS):
run = Fake(run_id)
def make(run_id, elapsed, rows=BASE_ROWS, graph=None):
run = Fake(run_id, graph=graph)
run.tool("fetch", region_sql(), rows=rows)
run.agent("bot", region_sql(), rows=rows)
run.card([metric("cost", "vars.res.elapsed_ms", elapsed,
@@ -597,6 +669,12 @@ def test_a_value_from_a_querying_node_that_moved_without_its_data_is_incomparabl
assert cost["classification"] == "incomparable" and any("区域取数" in r for r in cost["reasons"])
cost = by_id(run_compare(make("r1", 3), make("r2", 5, NEXT_ROWS)))["cost"]
assert cost["classification"] == "data_only"
# 取数节点自己改了配置(SQL 没动):它整份交出的结果可能跟着变,算模板变化
graph = weekly_graph()
graph["nodes"][1]["data"]["config"]["fail_fast"] = False
cost = by_id(run_compare(make("r1", 3), make("r2", 5, NEXT_ROWS, graph)))["cost"]
assert cost["classification"] == "sql_changed", cost["reasons"]
assert cost["reasons"] == ["上游节点「区域取数」的配置有变化"]
def test_data_from_upstream_is_only_claimed_when_the_graph_is_the_same():
@@ -609,7 +687,7 @@ def test_data_from_upstream_is_only_claimed_when_the_graph_is_the_same():
run.card(code_metrics(value)(None, None, rows, rows))
return run
base_graph = weekly_graph(between=(node("calc", "code", "调整", code="print(1)", assign_to="adj"),))
base_graph = weekly_graph(after_fetch=(calc_node(),))
target_graph = copy.deepcopy(base_graph)
target_graph["nodes"][-1]["data"]["config"] = {"fields": [{"name": "周报", "value": "{{ nodes.write.text }}"}]}
calc = by_id(run_compare(make("r1", BASE_ROWS, 10, base_graph), make("r2", NEXT_ROWS, 12, target_graph)))["calc"]
@@ -676,6 +754,27 @@ def test_an_extra_query_by_the_writer_is_a_sql_change():
extra = "SELECT COUNT(*) AS n FROM refunds"
share = by_id(run_compare(uncited("r1", 0.46), uncited("r2", 0.52, sqls=[region_sql(), extra])))["share"]
assert share["classification"] == "sql_changed" and any("多了一次查询" in r for r in share["reasons"])
# 反过来这次少查了一次:同样是取数变了
share = by_id(run_compare(uncited("r1", 0.46, sqls=[region_sql(), extra]), uncited("r2", 0.52)))["share"]
assert share["classification"] == "sql_changed" and any("少了一次查询" in r for r in share["reasons"])
def test_an_input_on_a_snapshot_on_one_side_only_is_incomparable():
"""基准运行没找到东区金额那一格落在哪份快照上(卡上只记了值),这次找到了:只有一边对得上查询快照,比不了取数。"""
def make(run_id, rows, *, located):
run = Fake(run_id)
fetch = run.tool("fetch", region_sql(), rows=rows)
run.agent("bot", region_sql(), rows=rows)
item = cell_in(0, "gmv", rows[0][1], fetch)
if not located:
kept = {k: v for k, v in item.items() if k not in ("artifact", "locator", "eid")}
item = {**kept, "status": "unresolved"}
run.card([metric("east", EAST, rows[0][1], [item], name="东区金额")])
return run
east = by_id(run_compare(make("r1", BASE_ROWS, located=False), make("r2", NEXT_ROWS, located=True)))["east"]
assert east["classification"] == "incomparable" and east["attribution"] is None
assert east["reasons"] == [f"输入「{EAST}」只有一边对得上查询快照,无法比较取数"]
def test_a_changed_default_model_is_a_template_change_for_agents_that_follow_it():
@@ -685,10 +784,20 @@ def test_a_changed_default_model_is_a_template_change_for_agents_that_follow_it(
target_graph["defaults"]["model"] = "model-b"
result = run_compare(uncited("r1", 0.46, graph=base_graph), uncited("r2", 0.52, graph=target_graph))
share = by_id(result)["share"]
assert share["classification"] == "sql_changed" and any("默认设置" in r for r in share["reasons"]), share["reasons"]
assert share["classification"] == "sql_changed"
assert share["reasons"] == ["上游节点「区域分析」跟随的工作流默认设置(模型)有变化"]
assert result["template"]["same_graph"] is False and compare.DEFAULTS_CHANGED in result["notes"]
# 取数节点不用模型:东区金额不受影响
assert by_id(result)["east"]["classification"] == "unchanged"
# 按出处交的字段:取法可能跟着模型变了
total = by_id(run_compare(week("r1", graph=base_graph), week("r2", NEXT_ROWS, graph=target_graph)))["total"]
assert total["classification"] == "sql_changed"
assert total["reasons"] == ["上游节点「区域分析」跟随的工作流默认设置(模型)有变化,它按出处交来的字段可能换了取法"]
# 一次改了好几项:写界面上的叫法(模型接入不写 provider),多的只写项数
many = copy.deepcopy(base_graph)
many["defaults"] = {"provider": "openai", "model": "model-b", "temperature": 0.5, "max_tokens": 2000}
share = by_id(run_compare(uncited("r1", 0.46, graph=base_graph), uncited("r2", 0.52, graph=many)))["share"]
assert share["reasons"] == ["上游节点「区域分析」跟随的工作流默认设置(模型接入、模型、温度等 4 项)有变化"]
# Agent 自己指定了模型:默认设置变了它不跟着变
for graph in (base_graph, target_graph):
graph["nodes"][2]["data"]["config"]["model"] = "model-x"
@@ -696,6 +805,68 @@ def test_a_changed_default_model_is_a_template_change_for_agents_that_follow_it(
assert share["classification"] == "incomparable"
def test_defaults_that_agents_do_not_read_leave_them_alone():
"""只改了 Agent 不读的默认设置:结论句裁判、结论句的处理只有报告节点读,审批策略只管调用工具前问不问人。按出处
交的字段、取数节点的格子照常是数据变化。"""
for before, after in (({"judge": {"max_cost_usd": 0.5}}, {"judge": {"max_cost_usd": 1.0}}),
({"claims": "off"}, {"claims": "judge"}),
({"approval": "dangerous"}, {"approval": "never"})):
base_graph, target_graph = weekly_graph(), weekly_graph()
base_graph["defaults"], target_graph["defaults"] = before, after
result = run_compare(week("r1", graph=base_graph), week("r2", NEXT_ROWS, graph=target_graph))
found = {m["metric_id"]: (m["classification"], m["reasons"]) for m in result["metrics"]}
assert found == {"east": ("data_only", [compare.DATA_FROM_QUERIES]),
"total": ("data_only", [compare.DATA_FROM_QUERIES])}, (before, found)
assert compare.DEFAULTS_CHANGED in result["notes"]
def test_a_model_default_reaches_every_node_that_calls_the_model():
"""默认模型换了,跟着换的不只 Agent:让模型分类的条件分支、让模型修复的结构校验(默认开着)、用模型重排的知识检索
也是。它们在代码节点上游时,代码节点交的数变了不能说来自上游数据;不调模型的照常。"""
def calc(extra):
def make(run_id, model, rows, value):
graph = weekly_graph(after_fetch=(extra, calc_node()))
graph["defaults"] = {"provider": "anthropic", "model": model}
run = Fake(run_id, graph=graph)
run.tool("fetch", region_sql(), rows=rows)
run.agent("bot", region_sql(), rows=rows)
run.code("calc", value, "adj")
run.card(code_metrics(value)(None, None, rows, rows))
return run
return by_id(run_compare(make("r1", "model-a", BASE_ROWS, 10), make("r2", "model-b", NEXT_ROWS, 12)))["calc"]
cases = [{"key": "peak", "condition": "true"}]
for extra in (node("route", "branch", "分流", mode="llm", instruction="判断是否旺季", cases=cases),
node("route", "validate", "校验", schema={"type": "object"}),
node("route", "retrieve", "检索", query="旺季", rerank="model")):
found = calc(extra)
assert found["classification"] == "incomparable", (extra["type"], found["reasons"])
assert found["reasons"] == [f"输入「vars.adj」来自「调整」,上游节点「{extra['data']['label']}」跟随的工作流默认"
"设置(模型)有变化,无法判断变化来源"]
for extra in (node("route", "branch", "分流", mode="expression", cases=cases),
node("route", "validate", "校验", schema={"type": "object"}, repair_with_llm=False),
node("route", "retrieve", "检索", query="旺季")):
found = calc(extra)
assert found["classification"] == "data_only", (extra["type"], found["reasons"])
def test_every_setting_a_node_reads_is_known_to_the_comparison():
"""节点运行时按 NodeContext.cfg 读配置,节点上没写就回退到图级默认设置。对比按「哪类节点读哪几项」把默认设置的
变化落到节点上(compare._CFG_KEYS):执行器新读了一项、这里没登记,默认设置改了这一项就会被漏掉。"""
by_module: dict = {}
for kind, runner in RUNNERS.items():
by_module.setdefault(inspect.getmodule(runner), set()).add(str(kind))
for module, kinds in by_module.items():
source = inspect.getsource(module)
read = set(re.findall(r'ctx\.cfg\(\s*"(\w+)"', source))
read |= set(re.findall(r'_setting\(spec, node, "(\w+)"', source))
if "ctx.approval_mode()" in source:
read.add("approval") # NodeContext.approval_mode 同样先读节点、再回退到默认设置
known = set().union(
*(compare._CFG_KEYS.get(k, frozenset()) | compare._CFG_KEYS_IF.get(k, frozenset()) for k in kinds))
assert read <= known, (module.__name__, sorted(read - known))
def cited(run_id, rows=BASE_ROWS, *, column="gmv", span=None):
"""Agent 按出处交回数组字段:引用快照的哪一列、哪一段行由调用方定。"""
run = Fake(run_id)
@@ -725,6 +896,78 @@ def test_an_agent_citing_other_rows_is_not_a_data_change():
assert total["classification"] == "data_only" and total["attribution"]["rows"]
#: 按区域排序的结果这次多了北区,排在南区前面,西区挪到第 4 行
WITH_NORTH = [["东区", 1200.5, 100.0], ["北区", 90.0, 0.0], ["南区", 800.0, 50.0], ["西区", 600.0, 0.0]]
def test_the_same_rows_are_not_the_same_citation_when_only_one_side_is_the_whole_result():
"""Agent 两次都引用第 1 至 3 行:基准运行里那是整份结果,这次西区还在第 4 行、没被引用。行段一样也不是同一处,
不能把西区算成「数据没了」(占比也会超过 100%);反过来(基准只引用了一部分、这次是整份)也一样。"""
total = by_id(run_compare(cited("r1"), cited("r2", WITH_NORTH, span=(0, 2))))["total"]
assert total["classification"] == "incomparable" and total["attribution"] is None
assert total["reasons"] == ["Agent「区域分析」在基准运行中引用了整份结果,这次只引用了第 1 至 3 行(结果共 4 行),"
"取的不是同一组数"]
total = by_id(run_compare(cited("r1", WITH_NORTH, span=(0, 2)), cited("r2")))["total"]
assert total["classification"] == "incomparable" and total["attribution"] is None
assert total["reasons"] == ["Agent「区域分析」在基准运行中只引用了第 1 至 3 行(结果共 4 行),这次引用了整份结果,"
"取的不是同一组数"]
# 两边都只引用了前 3 行、都不是整份结果:同一处,照常是数据变化
later = [["东区", 1500.5, 120.0], ["北区", 95.0, 0.0], ["南区", 700.0, 40.0], ["西区", 600.0, 10.0]]
total = by_id(run_compare(cited("r1", WITH_NORTH, span=(0, 2)), cited("r2", later, span=(0, 2))))["total"]
assert total["classification"] == "data_only" and total["attribution"]["rows"]
def test_cited_rows_on_an_unreadable_snapshot_are_not_taken_as_the_same():
"""基准运行 Agent 那次查询的快照文件不在了:两边都引用第 1 至 3 行,可基准那次是不是整份结果核对不了,不报成
数据变化。"""
base = cited("r1")
del base.store[base.queries[1]]
total = by_id(run_compare(base, cited("r2", WITH_NORTH, span=(0, 2))))["total"]
assert total["classification"] == "incomparable"
assert total["reasons"] == ["基准运行的查询快照无法读取,无法核对 Agent「区域分析」引用的是不是整份结果"]
# 这次引用的是整份结果、基准那次的行段不同:同样核对不了是不是两边都是整份
total = by_id(run_compare(base, cited("r2", WITH_NORTH)))["total"]
assert total["classification"] == "incomparable" and "无法核对" in total["reasons"][0]
def test_an_element_of_a_cited_array_follows_its_row():
"""口径卡取 Agent 数组字段里的第一个元素:Agent 交的行段换了起点,第一个元素落到了别的行,不是同一个数;起点没变
的照常是数据变化。"""
def make(run_id, rows, span):
run = Fake(run_id)
run.tool("fetch", region_sql(), rows=rows)
bot = run.agent("bot", region_sql(), rows=rows, span=span)
row = span[0]
item = {"path": "vars.kpi.amounts[0]", "value": rows[row][1], "node_id": "bot", "via": "agent_field",
"field": "amounts[0]", "ref": f"Q1.r{row}.gmv", "artifact": bot,
"locator": {"row": row, "column": "gmv"}, "eid": f"ev:cell:{bot[:8]}:{row}:gmv", "status": "verified"}
run.card([metric("first", "vars.kpi.amounts[0]", rows[row][1], [item])])
return run
first = by_id(run_compare(make("r1", BASE_ROWS, (0, 2)), make("r2", NEXT_ROWS, (1, 2))))["first"]
assert first["classification"] == "incomparable"
assert first["reasons"] == ["Agent「区域分析」引用的行由第 1 行改为第 2 行,取的不是同一个数"]
first = by_id(run_compare(make("r1", BASE_ROWS, (0, 2)), make("r2", NEXT_ROWS, (0, 2))))["first"]
assert (first["classification"], first["reasons"]) == ("data_only", [compare.DATA_FROM_QUERIES])
def test_no_split_when_an_old_leaf_now_cites_another_column():
"""口径变了(v1 → v2),旧口径用到的 Agent 字段这次引用的是退款列:说不清它只是数据变了,不拆,说明为什么。"""
def make(run_id, column, expression, factor, version):
run = Fake(run_id)
run.tool("fetch", region_sql())
bot = run.agent("bot", region_sql(), column=column)
value = sum(r[COLUMNS.index(column)] for r in BASE_ROWS) * factor
run.card([metric("total", expression, value, [amounts_in(bot, BASE_ROWS, column=column)])], version=version)
return run
total = by_id(run_compare(make("r1", "gmv", "sum(vars.kpi.amounts)", 1, "v1"),
make("r2", "refund", "sum(vars.kpi.amounts) * 2", 2, "v2")))["total"]
assert total["classification"] == "caliber_and_data" and total["decomposition"] is None
assert total["decomposition_note"].startswith("无法确认旧口径用到的输入只是数据有变化")
assert "refund" in total["decomposition_note"]
def test_an_unknown_writer_on_one_side_is_named_plainly():
"""一边记不出 vars.adj 是哪个节点写的:说清是哪一边没记录,不写「None」。"""
def make(run_id, writer, value):
@@ -751,11 +994,32 @@ def test_a_list_run_input_is_compared_by_its_content():
run.card([metric("goal", "sum(input.targets)", float(sum(targets)), [item])])
return run
goal = by_id(run_compare(make("r1", [100.0, 200.0]), make("r2", [100.0, 500.0])))["goal"]
assert goal["classification"] == "params_only" and any("targets" in r for r in goal["reasons"]), goal["reasons"]
assert goal["classification"] == "params_only"
assert goal["reasons"] == ["运行输入「input.targets」的内容有变化"]
goal = by_id(run_compare(make("r1", [100.0, 200.0]), make("r2", [100.0, 200.0])))["goal"]
assert goal["classification"] == "unchanged"
def test_a_list_run_input_without_its_full_value_is_not_taken_as_unchanged():
"""运行输入是列表,入口节点当时的产出不在(还原不出完整值),卡上只记了「[2 项]」:大概一样、指标的值却变了,
核对不了它变没变,不报成数据变化;项数变了的照常是参数变化;指标的值没变的照常没变。"""
def make(run_id, targets):
run = Fake(run_id, inputs={"targets": targets})
item = {"path": "input.targets", "value": f"[{len(targets)} 项]", "node_id": "start", "via": "input",
"field": "targets", "status": "ok"}
run.card([metric("goal", "sum(input.targets)", float(sum(targets)), [item])])
return run
goal = by_id(run_compare(make("r1", [100.0, 200.0]), make("r2", [100.0, 500.0])))["goal"]
assert goal["classification"] == "incomparable"
assert goal["reasons"] == ["运行输入「input.targets」是列表或对象,当时没有保存完整值,无法核对是否有变化"]
goal = by_id(run_compare(make("r1", [100.0, 200.0]), make("r2", [100.0, 200.0, 50.0])))["goal"]
assert goal["classification"] == "params_only"
assert goal["reasons"] == ["运行输入「input.targets」的内容有变化"]
goal = by_id(run_compare(make("r1", [100.0, 200.0]), make("r2", [100.0, 200.0])))["goal"]
assert (goal["classification"], goal["reasons"]) == ("unchanged", [])
def test_a_run_input_change_is_a_parameter_change():
def make(run_id, goal):
run = Fake(run_id, inputs={"goal": goal})
@@ -874,9 +1138,25 @@ def test_swapped_queries_with_the_same_literals_pair_up_exactly():
result = run_compare(make("r1", ["东区", "南区"], 10, 1.0), make("r2", ["南区", "东区"], 12, 2.0))
assert [(q["structure"], q["literals"]) for q in result["queries"]] == [("same", []), ("same", [])]
# Agent 两次交出的一模一样(只是查询的先后换了):它没带进模型本身的差异
assert by_id(result)["calc"]["classification"] == "data_only"
def test_renamed_nodes_pair_their_queries_by_literals_first():
"""节点换了 ID,同一个模板查了东区和南区、这次先后互换:跨节点按 SQL 配时同样先配字面量也相同的,两条都没变,
不会配成两条「参数不同」。"""
def make(run_id, node_id, regions):
run = Fake(run_id)
for region in regions:
run.tool(node_id, f"SELECT SUM(amount) AS gmv FROM orders WHERE region = '{region}'", ["gmv"], [[1.0]],
types={"gmv": "number"})
return run
result = run_compare(make("r1", "fetch", ["东区", "南区"]), make("r2", "pull", ["南区", "东区"]))
assert [(q["node_id"], q["structure"], q["literals"]) for q in result["queries"]] == [
("pull", "same", []), ("pull", "same", [])]
def test_a_switched_data_source_is_a_structure_change():
"""SQL 一字不差、换了数据源:查的不是同一份数据。"""
result = run_compare(week("r1"), week("r2", NEXT_ROWS, source="shop_backup"))
@@ -903,6 +1183,17 @@ def test_reformatting_the_workflow_sql_is_not_a_change():
result = run_compare(week("r1"), week("r2", NEXT_ROWS, sql=loose, graph=weekly_graph(loose)))
assert {m["classification"] for m in result["metrics"]} == {"data_only"}
# 代码节点在它下游:取数节点不算改了配置,变化照常来自上游数据
def make(run_id, sql, rows, value):
run = Fake(run_id, graph=weekly_graph(sql, after_fetch=(calc_node(),)))
run.tool("fetch", sql, rows=rows)
run.agent("bot", sql, rows=rows)
run.code("calc", value, "adj")
run.card(code_metrics(value)(None, None, rows, rows))
return run
calc = by_id(run_compare(make("r1", region_sql(), BASE_ROWS, 10), make("r2", loose, NEXT_ROWS, 12)))["calc"]
assert (calc["classification"], calc["reasons"]) == ("data_only", [compare.DATA_FROM_UPSTREAM_SCOPED])
def test_renaming_a_node_is_not_a_change():
graph = weekly_graph()
@@ -918,9 +1209,10 @@ OUTER_SQL = "SELECT region, gmv FROM m ORDER BY region"
MERGED_TYPES = {"region": "text", "gmv": "number"}
def merged(run_id, sql, rows, *, template=None, nested=False, twice=False):
def merged(run_id, sql, rows, *, template=None, nested=False, twice=False, lost=False, bare=False, unsealed=False):
"""取数节点查一次,合并节点把它当输入 a 再执行一条合并 SQL(nested:再套一层合并;twice:这次多接一个输入 b),
口径卡取合并结果的一格。"""
口径卡取合并结果的一格。追不下去的几种(只用于一层合并):lost 合并的快照文件不在了;bare 合并没记录输入(快照、
merge.end 里都没有);unsealed 上游那次查询不在封存范围内(事件里没有它)。"""
nodes = [node("start", "input", "开始"), node("fetch", "tool", "区域取数", tool=TOOL, args={"sql": template or sql}),
node("join", "merge", "合并", sql=MERGE_SQL, inputs={"a": "fetch", **({"b": "fetch"} if twice else {})})]
if nested:
@@ -936,6 +1228,14 @@ def merged(run_id, sql, rows, *, template=None, nested=False, twice=False):
types=MERGED_TYPES), "outer"
run.card([metric("east", f"cell(nodes.{owner}, 0, 'gmv')", table[0][1],
[cell_in(0, "gmv", table[0][1], top, node_id=owner)], name="东区金额")])
if lost:
del run.store[top]
if bare:
run.store[top] = {k: v for k, v in run.store[top].items() if k != "inputs"}
run.events = [(seq, etype, nid, {k: v for k, v in data.items() if k != "inputs"}
if etype == "merge.end" else data) for seq, etype, nid, data in run.events]
if unsealed:
run.events = [e for e in run.events if e[2] != "fetch"]
return run
@@ -961,6 +1261,21 @@ def test_a_merged_cell_follows_the_queries_feeding_the_merge():
assert east["east"]["classification"] == "sql_changed" and any("「b」" in r for r in east["east"]["reasons"])
def test_a_merged_cell_whose_inputs_cannot_be_followed():
"""合并的快照文件不在了:按 merge.end 记的输入接着追,上游加了筛选照样认得出。合并没记录输入、上游那次查询不在
封存范围内:说不清上游的取数变没变,不报成数据变化。"""
paid = region_sql(extra=" AND status = 'paid'")
east = by_id(run_compare(merged("r1", region_sql(), BASE_ROWS, lost=True),
merged("r2", paid, NEXT_ROWS, lost=True)))["east"]
assert east["classification"] == "sql_changed", east["reasons"]
assert any("「a」" in r and "区域取数" in r for r in east["reasons"]), east["reasons"]
for broken, said in (({"bare": True}, "没有记录它的输入"), ({"unsealed": True}, "不在封存范围内")):
east = by_id(run_compare(merged("r1", region_sql(), BASE_ROWS, **broken),
merged("r2", region_sql(), NEXT_ROWS, **broken)))["east"]
assert east["classification"] == "incomparable", (broken, east["reasons"])
assert any(said in r for r in east["reasons"]), (broken, east["reasons"])
def test_an_agent_reconfigured_with_the_same_sql_is_a_template_change():
"""Agent 改了提示词、SQL 一字不差:它按出处交来的字段可能换了取法,不能报成数据变化。"""
graph = weekly_graph()