W3-a 关联覆盖率下降、清理目录跨表、扩容器集成测试到对象索引和取值观察

- 剖析核对关联关系时,覆盖率从够有确证的线(95%)掉到线以下的记一条 coverage_drop(取值变化,warning):剖析本身照旧
  按规则降回推断(人工确认的只补上新的覆盖率),这里只把「覆盖率下降」留进变化记录。人点的剖析也记(变化记录全站一份)。
  关系的发现里多带 previous_coverage。
- 处理动作「清理目录」逐表清:这张表上说到被删的列的项,加上别的表指向它的关联关系(表删了时尤其如此);记这张表的
  目录版本,别的表也动了的另记 catalog_versions。
- 剖析任务的逐表进度多一类 observe(取值观察的四类合成一组),前端「取值观察 N/M」用它。
- 定时维护补一条测试:AGENTLAB_SCHEDULER 关着时调度循环不起,到点的维护不触发、不建任务。
- scripts/dialect-it.sh / dialect_it.py 扩到第 2 波的对象索引字典查询(走字典查询不退回 Inspector、类型、表注释、行数
  估算、Oracle 小写名的 quote)和这一波的取值观察(经剖析引擎只观察:行数、小表码值整表、大表 PG / Oracle 块采样、MySQL
  前 N 行、最新业务日期排除无效日期,max_date_sql 直接跑)。夹具的订单表加了表注释。本机跑了一次四种库的容器
  (postgres:16、mysql:8.4、mariadb:11、gvenzl/oracle-free:23-slim):121 项全过、0 项不通过,跑完容器已删、镜像留着。

验证:后端全量 14046 通过(11 跳过、8 预期失败)。
This commit is contained in:
ThearchyHelios
2026-10-16 17:28:07 +08:00
parent f5c94cf12d
commit ed72ee68d6
8 changed files with 191 additions and 36 deletions
+2 -2
View File
@@ -1,8 +1,8 @@
#!/usr/bin/env bash
# 方言的容器集成测试(下一轮 03 细案 C2):本机 docker 起 PostgreSQL 16、MySQL 8.4、MariaDB 11、Oracle Free 23 四个容器,
# 建虚构的夹具(保留字列、Oracle 小写带引号的表、CHAR 码值、MySQL ENUM、零日期 / 无穷大 / 占位日期、做旧的统计信息、
# 超过整表统计上限的大表、带索引的日期列),用 AgentLab 自己的探查和剖析代码跑全部剖析查询和取值观察的新方法,
# 断言零报错、结论和手算的一致。
# 超过整表统计上限的大表、带索引的日期列),用 AgentLab 自己的探查和剖析代码跑全部剖析查询、对象索引的字典查询
# (第 2 波)和取值观察的四类检查(第 3 波),断言零报错、结论和手算的一致。
#
# 为什么不进 CI:要拉几 GB 的镜像、Oracle 起一次要一两分钟。要用户同意才在本机跑。跑完删容器、留镜像。
#
+60 -7
View File
@@ -5,8 +5,12 @@
2. 用 AgentLab 自己的探查(app.data.introspect)读结构,核对类型写法、ENUM 取值、Oracle 小写带引号的名字、索引首列;
3. 用 AgentLab 自己的剖析(catalog_profile.profile_table,目录读写换成内存里的)逐表剖析,经只读连接、守卫、单条时限,
和线上走的是同一条路;
4. 直接跑取值观察要用的新查询(按日行数、时间窗口里的取值分布、块采样);
5. 断言:所有查询零报错(留痕里没有超时、失败、守卫拒),每个结论和下面按夹具定义手算的一致。
4. 直接跑取值观察要用的新查询(按日行数、时间窗口里的取值分布、块采样、最新业务日期);
5. 对象索引的字典查询(第 2 波,data/source_objects.list_objects:MySQL information_schema.TABLES、PostgreSQL pg_class 加
obj_description、Oracle ALL_OBJECTS 加 ALL_TAB_COMMENTS):走字典查询(不退回 Inspector)、类型、注释、行数估算对得上;
6. 取值观察的四类检查(第 3 波,data/catalog_profile_observe)经剖析引擎跑一遍(只观察,目录读写换成内存里的):行数、
码值观察(小表整表、大表 PG / Oracle 块采样、MySQL 按存储顺序的前 N 行)、最新业务日期(大表业务日期列打头有索引);
7. 断言:所有查询零报错(留痕里没有超时、失败、守卫拒),每个结论和下面按夹具定义手算的一致。
夹具的数(手算的依据):orders 40 行,id 1–40。
- store_code = id % 5 + 1(整数子列,指向文字主键 stores.code,'1'–'5' 都在);
@@ -36,7 +40,7 @@ from sqlalchemy import text # noqa: E402
from sqlalchemy.ext.asyncio import create_async_engine # noqa: E402
from app.core.crypto import encrypt # noqa: E402
from app.data import catalog, catalog_profile, introspect # noqa: E402
from app.data import catalog, catalog_profile, introspect, source_objects # noqa: E402
from app.data import engine as data_engine # noqa: E402
from app.data.catalog_profile import ProfileSettings, SqlDialect # noqa: E402
@@ -122,9 +126,13 @@ def _ddl(t: Target) -> list[str]:
f"{_q(t, 'level')} {intg}", f"paid_at {ts}", f"{_q(t, 'date')} DATE"]
if t.mysql:
cols.append("pay_way ENUM('cash','card','coupon')")
out = [f"CREATE TABLE stores (code {vc}(10) PRIMARY KEY, name {vc}(20))",
f"CREATE TABLE orders ({', '.join(cols)})",
orders = f"CREATE TABLE orders ({', '.join(cols)})"
if t.mysql:
orders += " COMMENT='订单'"
out = [f"CREATE TABLE stores (code {vc}(10) PRIMARY KEY, name {vc}(20))", orders,
"CREATE INDEX ix_orders_paid_at ON orders (paid_at)"]
if not t.mysql:
out.append("COMMENT ON TABLE orders IS '订单'")
events = f"CREATE TABLE events (id {intg} PRIMARY KEY, kind {vc}(10))"
if t.pg:
events += " WITH (autovacuum_enabled = false)"
@@ -218,7 +226,8 @@ async def _setup(t: Target) -> None:
# ---------------------------------------------------------------- 剖析:目录换成内存里的
async def _profile(source: Any, tables: list[str], notes: dict[str, dict[str, Any]]) -> dict[str, Any]:
async def _profile(source: Any, tables: list[str], notes: dict[str, dict[str, Any]], *,
observe: bool = False) -> dict[str, Any]:
cx = catalog_profile.make_context(source, SETTINGS, options={})
captured: dict[str, Any] = {}
@@ -233,7 +242,7 @@ async def _profile(source: Any, tables: list[str], notes: dict[str, dict[str, An
raise AssertionError(f"剖析要了没预料的库操作:{fn}")
for table in tables:
await catalog_profile.profile_table(cx, table, db=db)
await catalog_profile.profile_table(cx, table, db=db, regular=not observe, observe=observe)
return captured
@@ -359,6 +368,50 @@ async def run_target(t: Target) -> None:
check(total <= N_EVENTS_OLD + N_EVENTS_NEW, f"块采样跑得通(取到 {total} 行)")
else:
check(sampled is None, "MySQL / MariaDB 没有块采样,照实返回 None")
# ---- 对象索引的字典查询(第 2 波)
index = await source_objects.list_objects(source, None)
found = {o.name: o for o in index.objects}
check(index.method == "dictionary" and index.note is None,
f"对象索引走字典查询,没有退回 Inspector:{index.method} {index.note or ''}")
wanted = {"orders", "events", "stores"} | ({"visits"} if t.oracle else set())
check(wanted <= set(found), f"对象索引列出了夹具表:{sorted(found)}")
check(found["orders"].kind == "table" and found["orders"].comment == "订单",
f"对象索引的类型、表注释:{found.get('orders')}")
if t.oracle:
check(found["visits"].quote is True, "Oracle 小写带引号的表在对象索引里记了 quote")
est = found["events"].est_rows
check(est is None or est > 0, f"对象索引的行数估算(做过统计的表):{est}")
# ---- 取值观察(第 3 波):只观察,经剖析引擎、只读守卫、单条时限
paid = _key(orders, "paid_at")
obs_notes = {
"orders": {"business_date": {"value": {"column": paid}, "source": "human", "status": "confirmed"},
"columns": {_key(orders, "status"): {"codes": {"value": {"A": "", "B": ""}, "source": "human",
"status": "confirmed", "complete": True}}}},
"events": {"columns": {_key(events, "kind"): {"codes": {"value": {"open": "", "close": ""},
"source": "profile", "status": "proposed"}}}},
}
seen = await _profile(source, ["orders", "events"], obs_notes, observe=True)
logs = [q for _, r in seen.values() for q in r.query_log]
bad = [f"{q['kind']} {q['target']}:{q['outcome']} {q['detail']}" for q in logs if q["outcome"] != "ok"]
check(not bad, f"取值观察的 {len(logs)} 条查询零报错" + (f":{bad[:3]}" if bad else ""))
got_o = {(o["kind"], o["column"].lower()): o for o in seen["orders"][1].observations}
got_e = {(o["kind"], o["column"].lower()): o for o in seen["events"][1].observations}
check(got_o.get(("rows", ""), {}).get("data", {}).get("rows") == N_ORDERS, f"orders 的行数观察:{got_o.get(('rows', ''))}")
codes = got_o.get(("codes_observe", "status"))
check(codes is not None and codes["basis_key"] == "full"
and Counter(dict(map(tuple, codes["data"]["values"]))) == EXPECT["status"],
f"小表的码值观察看整表、和手算一致:{codes and codes['data']}")
fresh = got_o.get(("freshness", "paid_at"))
check(fresh is not None and fresh["data"]["max"][:19].replace("T", " ") == str(EXPECT["paid_at"][1]),
f"最新业务日期排除了无效日期、和手算一致:{fresh and fresh['data']}")
ev = got_e.get(("codes_observe", "kind"))
method = "sample" if (t.pg or t.oracle) else "first_rows"
check(ev is not None and ev["basis"]["method"] == method,
f"大表、没有业务日期:{'块采样' if method == 'sample' else '按存储顺序的前 N 行'}({ev and ev['basis_key']})")
res = await data_engine.run_query(source, d.max_date_sql(tbl, paid_col))
check(bool(res.rows) and str(res.rows[0][0])[:19].replace("T", " ") == str(EXPECT["paid_at"][1]),
f"max_date_sql 直接跑:{res.rows}")
finally:
await data_engine.engines.invalidate(source.id)