Files
AgentLab/backend/tests/test_report_masked_metric_sure.py
T
ThearchyHelios ce71f7d5c1 期 5 验收修复 N1 第 2 轮:取自遮罩格的指标只按确定取自遮罩列的认、转抄上游口径卡的也认;显示文字核对不误判正常写法,总览有一项无法核对就不给绿色的通过
复核第 1 轮(abb2707)找到四处:写作目录按证据面板宁可多遮的那一套认指标出处,SQL 认不准时没碰号码列的销售额也成了
「已遮罩」,正式运行引用它就判失败(相对基线回退);「紧挨着负号」连破折号「——」「—」和汉字「一」都算,「运费统一10元」
「合计——7,036.50元」原样导出变红(相对基线回退);紧挨着改字只堵住了数字、负号、小数点,接「万」、画删除线、从右往左
书写的字母、l / I / O 照样全绿,根子在总览把无法核对的项也给绿色;第二张口径卡转抄遮罩指标(nodes.<卡>.metrics[0].value、
vars.<assign_to>.metrics[0].value、nodes.<卡>.text)时号码照样进写作目录、文档证据目录和可验证报告。

取自遮罩格的指标只按确定的那一套认(写作目录、报告自查、可验证报告同一个判断)
- 写作目录(api/evidence.py 的 withheld_metrics)认指标出处时遮罩列只认确定取自遮罩列的(_InputMasks 的 sure:读得出来的
  快照按单元格引用自查的 _cited_masks,读不出来的按数据源现在设的列名),和 [[v:…]] 引用同一格的自查同一个口径:WITH 写了
  列名表、VALUES 当表、子查询里有没起别名的号码表达式、只装着分档标签的列,指标照常显示、照常引用;改了名字、套了函数的
  号码列照旧「已遮罩」。证据面板、推演照旧宁可多遮。报告自查按目录里记的 masked 认,跟着同一个判断
- 可验证报告认老文档(证据目录里还写着原值,整份只给哈希)按同一个判断:api/bundle.py 先按证据面板的那一套认好每份查询
  快照里确定取自遮罩列的列和这次运行查到过的遮罩值(certain_masks),交给 build_bundle(sure_columns、sure_secrets),
  _Store.metric_word 的 sure 按它们认。认口径卡能不能随文件附带照旧宁可多遮(卡上记着输入的值),SQL 认不准的卡同基线只给 id

转抄上游口径卡的指标按上游那个指标认
- recompute.card_input 认出输入转抄了哪张卡的哪几个指标:nodes.<卡>.metrics[i].value、vars.<卡的 assign_to>.metrics[i]、
  整个指标、输入、渲染算转抄那个指标;整张卡、清单文字、整份指标列表、下标不是字面量的同整份查询结果(值里有遮罩值才不给,
  数指标个数的照给);只取名称、单位、口径名的不算
- 证据面板 / 写作目录(_InputMasks._copied)、可验证报告(_Store._copied_kind)都用同一个 _metric_masks / metric_word 往上追,
  上游取自遮罩格,转抄的也不给;追了 CARD_DEPTH(8)张还没追到头(循环里抄自己上一轮的产出)的宁可多遮
- 写作目录读快照前把运行状态里每张口径卡的输入一并读好,不在事件循环里读盘

显示文字核对(verifier.js)
- 判不通过只剩紧挨着的数字、小数点,和真当负号用的三个(半角连字符减号、数学减号、全角减号)紧挨着数字开头、前面又不是
  字母数字时(「SKU-12」「A-3」的连字符是编号的一部分)。破折号(——、—、–、―,小号的、竖排的)、波浪线、汉字「一」、
  假名长音、制表符横线这些易混字当普通的字。连字符、不断行连字符、上下标负号、粗减号这类平时不当负号用的短横紧挨着数字
  开头判无法核对(以前判不通过;基线也是无法核对)
- 判无法核对(带原因):后面接着量级单位(万、亿、千、百、兆、萬、億、%、‰、‱、k、K、M、B 及全角,中间隔着空格也算;
  核对过的字以「万」「亿」结尾再接一个也算),核对过的字画着删除线或紧挨着画了删除线的字,紧挨着长得像数字的字母和数字
  符号(l、I、O、上下标数字、圈码、罗马数字、汉字〇),从右往左书写的字母和方向控制字符(照旧)
- 总览(overallOf):一项无法核对都没有、每份报告都核对上才是绿色的「全部通过」;有无法核对的写中性的「可核对的项全部通过,
  另有 N 项无法核对」,下面醒目地写明这个文件判断不了真伪,要在系统中核对封存(运行 ID、GET /api/runs/<id>/verify、定时任务
  通知里的清单哈希前 12 位)。以前无法核对的项照样给绿色,伪造者只要让读者看到的那个数落进无法核对就行
- 交出 layoutOf 给 node 里逐条核排法;开发数据 39 份报告 1591 处、验收数据 58 份报告 1695 处引用按新规则扫过,没有一处判
  不通过或无法核对

测试
- 新增 tests/test_report_masked_metric_sure.py(四种 SQL 认不准的销售额、分档标签照给照引、确定取自号码列的照旧遮、老文档
  照常附带、写作目录和可验证报告逐个指标同一个判断)、tests/test_report_masked_card_copies.py(三种转抄写法在写作目录、文档、
  自查、证据面板、可验证报告里都不给号码,没有报告的运行、老文档,转抄普通指标照给,认转抄的各种写法,追不到头的宁可多遮)、
  tests/test_bundle_prose_export.py(「统一」「——」「—」「SKU-」「共…家」「–」真跑报告撰写节点、从接口导出,全部通过、总览绿)
- test_bundle_shown:负号只认三个、前面是汉字的照判不通过;像负号的短横、量级单位、删除线、长得像数字的字母各自判无法核对、
  总览中性;正常写法原样导出全绿;总览只在一项无法核对都没有时是绿的
- 样本页新增 prose(写作者自己写在引用旁边的普通文字,原样导出全绿);check-bundle 补数学减号、紧挨着汉字加负号判不通过,
  接万(隔空格)、接亿元、接 %、接 k、订单号接万、画删除线另写一个数、前面加小写 L、后面接大写 O、长得像负号的连字符判无法
  核对,各自点名原因、总览中性并写明去系统里核对;有无法核对项的样本页(base、legacy、continued、missing-card、masked-card、
  masked-source、nan)总览改断言中性,从右往左书写的字母那条不再断言总览是绿的
- 每处改动都做了「去掉实现、测试变红」:写作目录不用 sure、老文档不用 sure、sure 列不用、转抄认不出(三处各自)、追不到头
  按核对不了、负号的字母数字例外、负号恢复成 \p{Pd} 全部、数学减号全角减号不算、量级单位、隔空格、以万结尾、删除线、像数字
  的字母、像负号的短横、总览有无法核对照样绿
2026-10-12 19:09:50 +08:00

208 lines
12 KiB
Python
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
"""值取自遮罩格的口径卡指标只按「确定取自遮罩列」认:SQL 认不准、整份遮的快照里,没确定取自遮罩列的普通金额、计数、标签列,
指标照常显示、照常引用。
复核第 1 轮发现(相对基线的回退):写作目录按证据面板宁可多遮的那一套(_mask_sets 的 hidden)认指标的出处。SQL 认不准时
(WITH 写了列名表、VALUES 当表、子查询里有没起别名的表达式)整份结果的列都算遮罩列,完全没碰号码列的销售额也被写成
「已遮罩」,正式运行引用它就判失败,写作者也改不掉;同一格按 [[v:…]] 引用却不报违规(单元格引用的自查只认确定的那一套,
_cited_masks)。可验证报告认老文档时同样多算,证据目录里写着这类普通金额的老文档整份只给哈希。
现在写作目录、报告自查、可验证报告(含老文档只给哈希的判断)认指标取不取自遮罩格,都按单元格引用自查的那一套
(_cited_masks:_mask_sets 的 sure,再按血缘排除值确实不取自遮罩列的)。证据面板、推演照旧宁可多遮。
真跑的部分用虚构的 SQLite 收银库(号码是编的),写作者是 MockChatModel 照剧本写。
"""
from __future__ import annotations
import base64
import json
import sqlite3
import pytest
from httpx import ASGITransport, AsyncClient
from sqlalchemy import select
from app.core.artifact_store import read_bytes
from app.data.engine import engines
from app.db.base import SessionLocal
from app.db.models import DataSource
from app.engine.runner import run_manager
from app.main import app
from tests.test_bundle_shown import verdicts
from tests.test_report_masked_metrics import doc_of, events, finish, metric_lines, node, script
ROWS = [(1, "顾客甲", "19900001101", 1234.5), (2, "顾客乙", "19900001102", 2345.25), (3, "顾客丙", "19900001103", 3456.75)]
PHONES = [r[2] for r in ROWS]
GMV = {"id": "gmv", "name": "销售额", "unit": "元", "decimals": 2, "format": "thousands",
"expression": "cell(nodes.q, 0, 'gmv')"}
#: 没碰号码列的查询,SQL 却认不准结果里哪一列取自哪一列(证据面板、可验证报告的快照按整份遮):(SQL, 销售额的渲染)
PLAIN = [
# WITH 写了列名表
("WITH s(cid, amt) AS (SELECT id, amount FROM customers) SELECT SUM(amt) AS gmv FROM s", "7,036.50元"),
# VALUES 当参数表
("WITH params(lo) AS (VALUES (1000)) SELECT SUM(amount) AS gmv FROM customers, params WHERE amount >= lo",
"7,036.50元"),
("WITH v(id) AS (VALUES (1), (2)) SELECT SUM(c.amount) AS gmv FROM customers c JOIN v ON v.id = c.id",
"3,579.75元"),
# 子查询里有没起别名的号码表达式,外层只取金额
("SELECT id, amount AS gmv FROM (SELECT id, UPPER(phone), amount FROM customers) ORDER BY id", "1,234.50元"),
]
def leaks(value) -> list[str]:
raw = value if isinstance(value, str) else json.dumps(value, ensure_ascii=False, default=str)
return [p for p in PHONES if p in raw or p in raw.replace(",", "")]
@pytest.fixture
async def engine_up():
await run_manager.setup()
yield
await run_manager.shutdown()
@pytest.fixture
async def client(engine_up):
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as c:
yield c
@pytest.fixture
async def pos(tmp_path, engine_up):
"""收银库:顾客的号码设了遮罩。"""
path = tmp_path / "pos.db"
db = sqlite3.connect(path)
db.executescript("CREATE TABLE customers (id INTEGER PRIMARY KEY, name TEXT, phone TEXT, amount REAL);")
db.executemany("INSERT INTO customers VALUES (?, ?, ?, ?)", ROWS)
db.commit()
db.close()
async with SessionLocal() as session:
row = (await session.execute(select(DataSource).where(DataSource.name == "n1pos"))).scalar_one_or_none()
if row is None:
row = DataSource(name="n1pos", kind="sqlite", database=str(path), readonly=True, enabled=True)
session.add(row)
row.database, row.kind, row.enabled, row.readonly = str(path), "sqlite", True, True
row.options = {"mask_columns": ["phone"]}
await session.commit()
source_id = row.id
await engines.invalidate(source_id)
yield path
await engines.invalidate(source_id)
def flow(sql: str, metrics: list[dict], **write) -> dict:
"""收银查询 → 口径卡 → 报告(违规处置默认判失败)→ 成果。"""
nodes = [
node("start", "input"),
node("q", "tool", tool="db_query__n1pos", args={"sql": sql}),
node("card", "metrics", caliber="收银口径", caliber_version="v1", metrics=metrics),
node("write", "report", instructions="写一句", **{"on_violation": "fail", **write}),
node("out", "output", fields=[{"name": "报告", "value": "{{ nodes.write.text }}"}]),
]
return {"nodes": nodes, "edges": [{"source": a["id"], "target": b["id"]} for a, b in zip(nodes, nodes[1:])]}
# --------------------------------------------------------------------------
# 写作目录、报告自查
# --------------------------------------------------------------------------
@pytest.mark.parametrize("sql, shown", PLAIN)
async def test_an_amount_from_a_query_whose_sql_cannot_be_traced_is_given_and_citable(monkeypatch, pos, sql, shown):
"""没碰号码列的销售额:写作目录照写它的值(以前写「已遮罩」),正式运行引用它照常通过(以前判失败),同一格按
[[v:…]] 引用同样不报违规——按指标引用和按格引用是同一个判断。"""
seen = script(monkeypatch, "本期销售额 [[m:gmv]],对照 [[v:Q1.r0.gmv]]。")
row = await finish(flow(sql, [GMV]))
assert row.status == "succeeded", row.error
lines = metric_lines(seen[0])
assert f"销售额 = {shown}" in lines and "已遮罩" not in lines, lines
[checked] = [e.data for e in await events(row.id) if e.type == "report.checked"]
assert checked["ok"] is True and checked["repairs"] == 0, checked
doc = await doc_of(row.id)
assert "masked" not in doc["catalog"]["m:gmv"] and f"本期销售额 {shown}" in doc["markdown"], doc["markdown"]
async def test_a_label_column_that_only_bands_the_masked_column_is_given_and_citable(monkeypatch, pos):
"""只装着分档标签的列(CASE 里写死的「已留」「未留」):值里没有号码,指标照给、照常引用。"""
seen = script(monkeypatch, "首位顾客 [[m:has]] 号码,对照 [[v:Q1.r0.has_phone]]。")
sql = "SELECT id, CASE WHEN phone IS NULL THEN '未留' ELSE '已留' END AS has_phone FROM customers ORDER BY id"
row = await finish(flow(sql, [{"id": "has", "name": "首位顾客留号", "format": "plain",
"expression": "cell(nodes.q, 0, 'has_phone')"}]))
assert row.status == "succeeded", row.error
assert "首位顾客留号 = 已留" in metric_lines(seen[0])
doc = await doc_of(row.id)
assert "masked" not in doc["catalog"]["m:has"] and "首位顾客 已留 号码" in doc["markdown"], doc["markdown"]
async def test_a_metric_whose_column_is_surely_derived_from_the_masked_one_is_still_withheld(monkeypatch, pos):
"""对照:SQL 里认得出取自号码列的结果列(套了函数、改了名字)照旧写「已遮罩」,同一张卡上的金额照给;引用它的正式运行
照旧判失败。"""
seen = script(monkeypatch, "首位顾客号码 [[m:p0]],消费 [[m:spend0]]。")
row = await finish(flow("SELECT id, UPPER(phone) AS p, amount FROM customers ORDER BY id", [
{"id": "p0", "name": "首位顾客号码", "format": "plain", "expression": "cell(nodes.q, 0, 'p')"},
{"id": "spend0", "name": "首位顾客消费", "unit": "元", "decimals": 2, "format": "thousands",
"expression": "cell(nodes.q, 0, 'amount')"}]))
lines = metric_lines(seen[0])
assert "首位顾客号码 = 已遮罩(取自设置了遮罩的列,不要引用)" in lines and "首位顾客消费 = 1,234.50元" in lines, lines
assert leaks(lines) == []
assert row.status == "failed" and "m:p0" in (row.error or "") and leaks(row.error) == [], row.error
# --------------------------------------------------------------------------
# 可验证报告:老文档只给哈希的判断
# --------------------------------------------------------------------------
async def test_an_old_doc_carrying_an_ordinary_amount_from_an_untraceable_query_is_embedded(client, monkeypatch, pos):
"""升级前写的文档(写作前不认遮罩,这里把那一步换成什么都不认):证据目录里写着没碰号码列的销售额。以前可验证报告按
宁可多遮的那一套认,整份文档只给哈希、页面说「有 1 份报告未能核对」;现在和写作目录同一个判断,文档照常附带、核对得上。
口径卡照旧只给 id(卡上记着输入的值,SQL 认不准的快照按整份遮,同基线),引用它的数字无法核对。"""
import app.api.evidence as evidence_api
async def nothing(*_args, **_kwargs):
return []
monkeypatch.setattr(evidence_api, "withheld_metrics", nothing)
script(monkeypatch, "本期销售额 [[m:gmv]]。")
row = await finish(flow(PLAIN[1][0], [GMV]))
assert row.status == "succeeded", row.error
resp = await client.get(f"/api/runs/{row.id}/bundle.html")
assert resp.status_code == 200, resp.text
from tests.test_bundle_masked_metrics import data_of
bundle = data_of(resp.text)
[doc] = [a for a in bundle["artifacts"] if a["kind"] == "report_doc"]
assert "b64" in doc and "masked" not in doc, doc
assert json.loads(base64.b64decode(doc["b64"]))["catalog"]["m:gmv"]["value"] == 7036.5
got = verdicts(bundle)
assert got["docs"] == [{"node": "write", "trusted": True}], got["docs"]
assert leaks(resp.text) == []
@pytest.mark.parametrize("sql, expected", [
*[(sql, None) for sql, _ in PLAIN],
("SELECT id, UPPER(phone) AS gmv FROM customers ORDER BY id", "masked"),
("SELECT id, phone AS gmv FROM customers ORDER BY id", "masked"),
("SELECT id, CASE WHEN phone IS NULL THEN '未留' ELSE '已留' END AS gmv FROM customers ORDER BY id", None),
])
async def test_the_bundle_judges_old_docs_like_the_report_writer(monkeypatch, pos, sql, expected):
"""可验证报告认老文档里的指标给不给(_Store.metric_word 的 sure),和报告撰写节点的写作目录(_metric_masks 的
sure、by_value=False)一样;证据面板照旧宁可多遮。"""
from app.api.evidence import MASKED, _mask_fingerprint, _Masks, _metric_in, _metric_masks, _sealed, certain_masks
from app.bundle.build import _Store
script(monkeypatch, "收银完成。")
row = await finish(flow(sql, [{**GMV, "format": "plain", "unit": "", "decimals": None}], on_violation="flag"))
assert row.status == "succeeded", row.error
sealed = await _sealed(row.id)
entry = next(e for e in sealed.entries if e.get("kind") == "metric_set")
card = json.loads(read_bytes(entry["artifact"], verify=False))
sure, secrets = await certain_masks(sealed, _Masks(sealed))
store = _Store(lambda a: read_bytes(a, verify=False), await _mask_fingerprint(), sealed.events,
sure_columns=sure, sure_secrets=secrets)
metric = _metric_in(card, "gmv")
_, word = await _metric_masks(metric, "card", entry["artifact"], sealed, _Masks(sealed), by_value=False, sure=True)
writer = {MASKED: "masked"}.get(word or "", word)
bundle = store.metric_word(entry["artifact"], metric, sealed.events, sure=True)
assert writer == bundle == expected, (writer, bundle)