剖析与探查的方言修正(C1)和没有真库时的方言验证(C2)

为什么:剖析是全站唯一主动查业务库的地方,以前只在守卫放行和字符串片段两层上测过,从没在真库上跑过。
本机四个容器(PG 16、MySQL 8.4、MariaDB 11、Oracle Free 23)实测,旧的 IN 列表写法在 PostgreSQL 上整数子列
对文字主键直接报 operator does not exist: character varying = integer,关系一条都核对不了。

C1 方言修正:
- IN 列表的字面量按被指向列的类型族写(key_literals):父列文字写 '12',父列数字只收整数写法的值,父列是时间、
  对不上的不拼并在备注里写「两端类型不同」;子表是 CHAR(n) 时去掉补齐的尾随空格。
- 列类型改存按方言编译的完整写法(VARCHAR2(20 CHAR)、NUMBER(10, 0)、ENUM('a','b')、TIMESTAMP WITH TIME ZONE),
  编译不了退回 str()。SQLAlchemy 把 Oracle 的 VARCHAR2 反射成通用 VARCHAR(容器实测),照数据字典写成 VARCHAR2。
  ENUM / SET 另记 enum_values;Oracle 小写带引号的表和列记 quote_name;各索引(含主键)的首列记进
  index_first_columns(下一波取值观察判断能不能对大表发窗口查询)。新探查的快照哈希会变,旧运行封存的不受影响。
- catalog._type_family 认 ENUM / SET 是文字、INTERVAL 不是数字,去掉引号里的取值再认;新增 column_family
  (有 enum_values 的是文字,PostgreSQL 枚举编译出来只有类型名);is_date_column 不再把 ENUM('date') 认成日期列。
- 码值候选:ENUM / SET 不看列名直接算候选,ENUM 列定义里这次没出现的取值也写进码值(含义待填);VARCHAR2(20 CHAR)
  的长度按 20 算;CHAR 码值去尾随空格;MySQL 上写明按排序规则分组、比较(可能不分大小写)。
- 统计信息估出的行数不超过整表统计上限时再跑一次有上限的计数确认,统计过期的大表不再被当成小表做整表统计。
- 日期列的最小值、最大值只看 0001-01-02 到 9999-12-31 之间的值,零日期、占位日期、PG 的 ±infinity 不计入,
  照实写「另有 N 个无效日期未计入」,日期范围里带 invalid 个数(界面显示)。
- 新增取值观察用的方言方法 date_literal / day_trunc / sample_from,以及按日行数、时间窗口里的取值分布、块采样三类
  查询的拼法(窗口上端截在 9999-12-31 之前,不让 infinity 多出一天)。
- 说明文字不再说「抽样」:大表看的是「按存储顺序取前 N 行」;设置项「抽样键值数」改叫「核对键值数」。

C2 验证:
- tests/test_profile_dialect_golden.py:五种方言 × 全部剖析和观察查询的完整 SQL 钉在 tests/fixtures/dialect/,
  sqlglot 逐条解析、过只读守卫,再用 qualify + annotate_types 核对 IN 两边类型族一致(附旧写法的对照)。
- scripts/dialect-it.sh(不进 CI):本机起四个容器,建虚构夹具(保留字列、Oracle 小写带引号的表、CHAR 码值、
  MySQL ENUM、零日期 / infinity / 占位日期、做旧的统计信息、超上限的大表、带索引的日期列),用 AgentLab 自己的
  探查和剖析代码跑全部查询,断言零报错、结论和手算一致。四种库都通过;MariaDB 插入时自己更新行数估算,统计信息
  做不旧,照实打印。把 IN 列表改回旧写法回跑,PG 上照样报出上面那条错误。跑完删容器、留镜像。
This commit is contained in:
ThearchyHelios
2026-10-16 10:25:28 +08:00
parent 4819b21269
commit ee7b8b2f97
21 changed files with 1499 additions and 82 deletions
+27
View File
@@ -0,0 +1,27 @@
-- 剖析与取值观察的 SQL 金样:mariadb(tests/test_profile_dialect_golden.py 生成,改动要人审)
-- stats
SELECT TABLE_ROWS FROM information_schema.TABLES WHERE TABLE_SCHEMA = 'sales' AND TABLE_NAME = 'orders';
-- stats_quoted
SELECT TABLE_ROWS FROM information_schema.TABLES WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = 'visits';
-- bounded
SELECT COUNT(*) AS n FROM (SELECT 1 AS one FROM sales.orders LIMIT 100001) s;
-- distinct
SELECT DISTINCT store_code FROM sales.orders WHERE store_code IS NOT NULL LIMIT 2000;
-- distinct_capped
SELECT DISTINCT store_code FROM (SELECT store_code FROM sales.orders WHERE store_code IS NOT NULL LIMIT 100000) s LIMIT 2000;
-- match
SELECT COUNT(DISTINCT code) AS n FROM stores WHERE code IN ('12', '7');
-- unique
SELECT COUNT(`level`) AS n, COUNT(DISTINCT `level`) AS d FROM `visits`;
-- value_counts
SELECT status, COUNT(*) AS cnt FROM sales.orders WHERE status IS NOT NULL GROUP BY status ORDER BY cnt DESC, status LIMIT 21;
-- value_counts_capped
SELECT status, COUNT(*) AS cnt FROM (SELECT status FROM sales.orders WHERE status IS NOT NULL LIMIT 100000) s WHERE status IS NOT NULL GROUP BY status ORDER BY cnt DESC, status LIMIT 21;
-- min_max
SELECT MIN(CASE WHEN paid_at >= DATE '0001-01-02' AND paid_at < DATE '9999-12-31' THEN paid_at END), MAX(CASE WHEN paid_at >= DATE '0001-01-02' AND paid_at < DATE '9999-12-31' THEN paid_at END), SUM(CASE WHEN paid_at < DATE '0001-01-02' OR paid_at >= DATE '9999-12-31' THEN 1 ELSE 0 END), MIN(CASE WHEN biz_day >= '0001-01-02' AND biz_day < '9999-12-31' THEN biz_day END), MAX(CASE WHEN biz_day >= '0001-01-02' AND biz_day < '9999-12-31' THEN biz_day END), SUM(CASE WHEN biz_day < '0001-01-02' OR biz_day >= '9999-12-31' THEN 1 ELSE 0 END) FROM sales.orders;
-- date_window
SELECT status, COUNT(*) AS cnt FROM sales.orders WHERE paid_at >= DATE '2026-08-27' AND paid_at < DATE '9999-12-31' AND status IS NOT NULL GROUP BY status ORDER BY cnt DESC, status LIMIT 21;
-- day_volume
SELECT DATE(paid_at) AS d, COUNT(*) AS n FROM sales.orders WHERE paid_at >= DATE '2026-08-27' AND paid_at < DATE '9999-12-31' GROUP BY DATE(paid_at) ORDER BY d;
-- sample
-- (这种方言没有这类查询)
+27
View File
@@ -0,0 +1,27 @@
-- 剖析与取值观察的 SQL 金样:mysql(tests/test_profile_dialect_golden.py 生成,改动要人审)
-- stats
SELECT TABLE_ROWS FROM information_schema.TABLES WHERE TABLE_SCHEMA = 'sales' AND TABLE_NAME = 'orders';
-- stats_quoted
SELECT TABLE_ROWS FROM information_schema.TABLES WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = 'visits';
-- bounded
SELECT COUNT(*) AS n FROM (SELECT 1 AS one FROM sales.orders LIMIT 100001) s;
-- distinct
SELECT DISTINCT store_code FROM sales.orders WHERE store_code IS NOT NULL LIMIT 2000;
-- distinct_capped
SELECT DISTINCT store_code FROM (SELECT store_code FROM sales.orders WHERE store_code IS NOT NULL LIMIT 100000) s LIMIT 2000;
-- match
SELECT COUNT(DISTINCT code) AS n FROM stores WHERE code IN ('12', '7');
-- unique
SELECT COUNT(`level`) AS n, COUNT(DISTINCT `level`) AS d FROM `visits`;
-- value_counts
SELECT status, COUNT(*) AS cnt FROM sales.orders WHERE status IS NOT NULL GROUP BY status ORDER BY cnt DESC, status LIMIT 21;
-- value_counts_capped
SELECT status, COUNT(*) AS cnt FROM (SELECT status FROM sales.orders WHERE status IS NOT NULL LIMIT 100000) s WHERE status IS NOT NULL GROUP BY status ORDER BY cnt DESC, status LIMIT 21;
-- min_max
SELECT MIN(CASE WHEN paid_at >= DATE '0001-01-02' AND paid_at < DATE '9999-12-31' THEN paid_at END), MAX(CASE WHEN paid_at >= DATE '0001-01-02' AND paid_at < DATE '9999-12-31' THEN paid_at END), SUM(CASE WHEN paid_at < DATE '0001-01-02' OR paid_at >= DATE '9999-12-31' THEN 1 ELSE 0 END), MIN(CASE WHEN biz_day >= '0001-01-02' AND biz_day < '9999-12-31' THEN biz_day END), MAX(CASE WHEN biz_day >= '0001-01-02' AND biz_day < '9999-12-31' THEN biz_day END), SUM(CASE WHEN biz_day < '0001-01-02' OR biz_day >= '9999-12-31' THEN 1 ELSE 0 END) FROM sales.orders;
-- date_window
SELECT status, COUNT(*) AS cnt FROM sales.orders WHERE paid_at >= DATE '2026-08-27' AND paid_at < DATE '9999-12-31' AND status IS NOT NULL GROUP BY status ORDER BY cnt DESC, status LIMIT 21;
-- day_volume
SELECT DATE(paid_at) AS d, COUNT(*) AS n FROM sales.orders WHERE paid_at >= DATE '2026-08-27' AND paid_at < DATE '9999-12-31' GROUP BY DATE(paid_at) ORDER BY d;
-- sample
-- (这种方言没有这类查询)
+27
View File
@@ -0,0 +1,27 @@
-- 剖析与取值观察的 SQL 金样:oracle(tests/test_profile_dialect_golden.py 生成,改动要人审)
-- stats
SELECT NUM_ROWS FROM ALL_TABLES WHERE OWNER = 'SALES' AND TABLE_NAME = 'ORDERS';
-- stats_quoted
SELECT NUM_ROWS FROM ALL_TABLES WHERE OWNER = SYS_CONTEXT('USERENV', 'CURRENT_SCHEMA') AND TABLE_NAME = 'visits';
-- bounded
SELECT COUNT(*) AS n FROM (SELECT * FROM (SELECT 1 AS one FROM sales.orders) WHERE ROWNUM <= 100001) s;
-- distinct
SELECT * FROM (SELECT DISTINCT store_code FROM sales.orders WHERE store_code IS NOT NULL) WHERE ROWNUM <= 2000;
-- distinct_capped
SELECT * FROM (SELECT DISTINCT store_code FROM (SELECT * FROM (SELECT store_code FROM sales.orders WHERE store_code IS NOT NULL) WHERE ROWNUM <= 100000) s) WHERE ROWNUM <= 2000;
-- match
SELECT COUNT(DISTINCT code) AS n FROM stores WHERE code IN ('12', '7');
-- unique
SELECT COUNT("level") AS n, COUNT(DISTINCT "level") AS d FROM "visits";
-- value_counts
SELECT * FROM (SELECT status, COUNT(*) AS cnt FROM sales.orders WHERE status IS NOT NULL GROUP BY status ORDER BY cnt DESC, status) WHERE ROWNUM <= 21;
-- value_counts_capped
SELECT * FROM (SELECT status, COUNT(*) AS cnt FROM (SELECT * FROM (SELECT status FROM sales.orders WHERE status IS NOT NULL) WHERE ROWNUM <= 100000) s WHERE status IS NOT NULL GROUP BY status ORDER BY cnt DESC, status) WHERE ROWNUM <= 21;
-- min_max
SELECT MIN(CASE WHEN paid_at >= DATE '0001-01-02' AND paid_at < DATE '9999-12-31' THEN paid_at END), MAX(CASE WHEN paid_at >= DATE '0001-01-02' AND paid_at < DATE '9999-12-31' THEN paid_at END), SUM(CASE WHEN paid_at < DATE '0001-01-02' OR paid_at >= DATE '9999-12-31' THEN 1 ELSE 0 END), MIN(CASE WHEN biz_day >= '0001-01-02' AND biz_day < '9999-12-31' THEN biz_day END), MAX(CASE WHEN biz_day >= '0001-01-02' AND biz_day < '9999-12-31' THEN biz_day END), SUM(CASE WHEN biz_day < '0001-01-02' OR biz_day >= '9999-12-31' THEN 1 ELSE 0 END) FROM sales.orders;
-- date_window
SELECT * FROM (SELECT status, COUNT(*) AS cnt FROM sales.orders WHERE paid_at >= DATE '2026-08-27' AND paid_at < DATE '9999-12-31' AND status IS NOT NULL GROUP BY status ORDER BY cnt DESC, status) WHERE ROWNUM <= 21;
-- day_volume
SELECT TRUNC(paid_at) AS d, COUNT(*) AS n FROM sales.orders WHERE paid_at >= DATE '2026-08-27' AND paid_at < DATE '9999-12-31' GROUP BY TRUNC(paid_at) ORDER BY d;
-- sample
SELECT * FROM (SELECT status, COUNT(*) AS cnt FROM sales.orders SAMPLE BLOCK (1) WHERE status IS NOT NULL GROUP BY status ORDER BY cnt DESC, status) WHERE ROWNUM <= 21;
+27
View File
@@ -0,0 +1,27 @@
-- 剖析与取值观察的 SQL 金样:postgresql(tests/test_profile_dialect_golden.py 生成,改动要人审)
-- stats
SELECT c.reltuples FROM pg_catalog.pg_class c JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace WHERE n.nspname = 'sales' AND c.relname = 'orders';
-- stats_quoted
SELECT c.reltuples FROM pg_catalog.pg_class c JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace WHERE n.nspname = current_schema() AND c.relname = 'visits';
-- bounded
SELECT COUNT(*) AS n FROM (SELECT 1 AS one FROM sales.orders LIMIT 100001) s;
-- distinct
SELECT DISTINCT store_code FROM sales.orders WHERE store_code IS NOT NULL LIMIT 2000;
-- distinct_capped
SELECT DISTINCT store_code FROM (SELECT store_code FROM sales.orders WHERE store_code IS NOT NULL LIMIT 100000) s LIMIT 2000;
-- match
SELECT COUNT(DISTINCT code) AS n FROM stores WHERE code IN ('12', '7');
-- unique
SELECT COUNT("level") AS n, COUNT(DISTINCT "level") AS d FROM "visits";
-- value_counts
SELECT status, COUNT(*) AS cnt FROM sales.orders WHERE status IS NOT NULL GROUP BY status ORDER BY cnt DESC, status LIMIT 21;
-- value_counts_capped
SELECT status, COUNT(*) AS cnt FROM (SELECT status FROM sales.orders WHERE status IS NOT NULL LIMIT 100000) s WHERE status IS NOT NULL GROUP BY status ORDER BY cnt DESC, status LIMIT 21;
-- min_max
SELECT MIN(CASE WHEN paid_at >= DATE '0001-01-02' AND paid_at < DATE '9999-12-31' THEN paid_at END), MAX(CASE WHEN paid_at >= DATE '0001-01-02' AND paid_at < DATE '9999-12-31' THEN paid_at END), SUM(CASE WHEN paid_at < DATE '0001-01-02' OR paid_at >= DATE '9999-12-31' THEN 1 ELSE 0 END), MIN(CASE WHEN biz_day >= '0001-01-02' AND biz_day < '9999-12-31' THEN biz_day END), MAX(CASE WHEN biz_day >= '0001-01-02' AND biz_day < '9999-12-31' THEN biz_day END), SUM(CASE WHEN biz_day < '0001-01-02' OR biz_day >= '9999-12-31' THEN 1 ELSE 0 END) FROM sales.orders;
-- date_window
SELECT status, COUNT(*) AS cnt FROM sales.orders WHERE paid_at >= DATE '2026-08-27' AND paid_at < DATE '9999-12-31' AND status IS NOT NULL GROUP BY status ORDER BY cnt DESC, status LIMIT 21;
-- day_volume
SELECT CAST(paid_at AS DATE) AS d, COUNT(*) AS n FROM sales.orders WHERE paid_at >= DATE '2026-08-27' AND paid_at < DATE '9999-12-31' GROUP BY CAST(paid_at AS DATE) ORDER BY d;
-- sample
SELECT status, COUNT(*) AS cnt FROM sales.orders TABLESAMPLE SYSTEM (1) WHERE status IS NOT NULL GROUP BY status ORDER BY cnt DESC, status LIMIT 21;
+27
View File
@@ -0,0 +1,27 @@
-- 剖析与取值观察的 SQL 金样:sqlite(tests/test_profile_dialect_golden.py 生成,改动要人审)
-- stats
-- (这种方言没有这类查询)
-- stats_quoted
-- (这种方言没有这类查询)
-- bounded
SELECT COUNT(*) AS n FROM (SELECT 1 AS one FROM sales.orders LIMIT 100001) s;
-- distinct
SELECT DISTINCT store_code FROM sales.orders WHERE store_code IS NOT NULL LIMIT 2000;
-- distinct_capped
SELECT DISTINCT store_code FROM (SELECT store_code FROM sales.orders WHERE store_code IS NOT NULL LIMIT 100000) s LIMIT 2000;
-- match
SELECT COUNT(DISTINCT code) AS n FROM stores WHERE code IN ('12', '7');
-- unique
SELECT COUNT("level") AS n, COUNT(DISTINCT "level") AS d FROM "visits";
-- value_counts
SELECT status, COUNT(*) AS cnt FROM sales.orders WHERE status IS NOT NULL GROUP BY status ORDER BY cnt DESC, status LIMIT 21;
-- value_counts_capped
SELECT status, COUNT(*) AS cnt FROM (SELECT status FROM sales.orders WHERE status IS NOT NULL LIMIT 100000) s WHERE status IS NOT NULL GROUP BY status ORDER BY cnt DESC, status LIMIT 21;
-- min_max
SELECT MIN(CASE WHEN paid_at >= '0001-01-02' AND paid_at < '9999-12-31' THEN paid_at END), MAX(CASE WHEN paid_at >= '0001-01-02' AND paid_at < '9999-12-31' THEN paid_at END), SUM(CASE WHEN paid_at < '0001-01-02' OR paid_at >= '9999-12-31' THEN 1 ELSE 0 END), MIN(CASE WHEN biz_day >= '0001-01-02' AND biz_day < '9999-12-31' THEN biz_day END), MAX(CASE WHEN biz_day >= '0001-01-02' AND biz_day < '9999-12-31' THEN biz_day END), SUM(CASE WHEN biz_day < '0001-01-02' OR biz_day >= '9999-12-31' THEN 1 ELSE 0 END) FROM sales.orders;
-- date_window
SELECT status, COUNT(*) AS cnt FROM sales.orders WHERE paid_at >= '2026-08-27' AND paid_at < '9999-12-31' AND status IS NOT NULL GROUP BY status ORDER BY cnt DESC, status LIMIT 21;
-- day_volume
SELECT substr(paid_at, 1, 10) AS d, COUNT(*) AS n FROM sales.orders WHERE paid_at >= '2026-08-27' AND paid_at < '9999-12-31' GROUP BY substr(paid_at, 1, 10) ORDER BY d;
-- sample
-- (这种方言没有这类查询)
+1 -1
View File
@@ -402,7 +402,7 @@ async def test_a_big_table_is_checked_on_its_first_rows(scenic):
found = await catalog_profile.check_valid_filter(source, "visits", source.schema_cache["tables"]["visits"],
"status = 1")
assert found.sampled and found.rows == 100
assert found.note.startswith("剖析核对:按前 100 行统计,")
assert found.note.startswith("剖析核对:按存储顺序取前 100 行统计,")
async def test_unchecked_filters_are_kept_with_the_reason(scenic, tmp_path):
+2 -2
View File
@@ -240,7 +240,7 @@ async def test_profile_verifies_name_relations_and_proposes_codes_and_business_d
assert rel["to_table"] == parent
assert (rel["source"], rel["status"]) == ("profile", "verified")
assert rel["coverage"] == 1.0 and rel["cardinality"] == "many_to_one"
assert "数据剖析" in rel["note"] and "抽样" in rel["note"]
assert "数据剖析" in rel["note"] and "子表取了" in rel["note"] and "抽样" not in rel["note"]
# 子表一侧数过 COUNT 和 COUNT(DISTINCT)、确有重复:基数是用数据核实的(A6,一对多关联的检查据此给 error)
assert rel["cardinality_checked"] is True
# A7:说明里不写日期,时间看这一项的 updated_at(界面按本地时间显示)
@@ -318,7 +318,7 @@ async def test_wrong_name_relation_stays_proposed_with_coverage_in_note(client,
rel = _relation((await _notes(client, sid, "parking_records"))["notes"], "lot_id")
assert rel["status"] == "proposed" and rel["source"] == "profile"
assert rel["coverage"] == 0.15
assert "抽样覆盖率 15%" in rel["note"] and "可能不是这条关系" in rel["note"] and "20 个" in rel["note"]
assert "覆盖率 15%" in rel["note"] and "可能不是这条关系" in rel["note"] and "20 个" in rel["note"]
finding = _table(body, "parking_records")["findings"][0]
assert finding["status"] == "proposed" and finding["coverage"] == 0.15
# 两个日期类列:不提议业务日期,取值范围只在剖析结果里
+10 -2
View File
@@ -40,7 +40,7 @@ def test_settings_default_off_and_parse_leniently():
({"table_max_s": 9}, "每张表时长上限"),
({"table_max_s": 601}, "每张表时长上限"),
({"query_timeout_s": 61}, "单条查询时限"),
({"sample_size": 5}, "抽样键值数"),
({"sample_size": 5}, "核对键值数"),
({"max_scan_rows": -1}, "整表统计行数上限"),
({"enabled": "true"}, "开启数据剖析"),
({"table_max_queries": True}, "每张表最多查询数"),
@@ -143,8 +143,16 @@ def test_profile_sql_passes_readonly_guard_on_every_dialect(kind):
d.unique_check_sql(tbl, col),
d.value_counts_sql(tbl, col, 21, scan_cap=None),
d.value_counts_sql(tbl, col, 21, scan_cap=100_000),
d.min_max_sql(tbl, [col]),
d.min_max_sql(tbl, [(col, False)]),
d.min_max_sql(tbl, [(col, True)]),
d.day_volume_sql(tbl, col, "2026-08-01"),
d.window_value_counts_sql(tbl, col, col, "2026-08-01", 21),
]
sampled = d.sample_value_counts_sql(tbl, col, 1, 21)
if kind in ("postgresql", "oracle"):
sqls.append(sampled)
else:
assert sampled is None # MySQL、SQLite 没有块采样
for sql in sqls:
assert guard.check(sql, readonly=True, dialect=kind) == sql, sql
+1 -1
View File
@@ -365,7 +365,7 @@ async def test_profile_marks_codes_complete_only_when_the_whole_table_was_read(c
capped = await make_source(scenic_db, options=_on(max_scan_rows=0, sample_size=10), draft=["visits"])
await _profile(client, capped, ["visits"])
codes = (await _notes(client, capped, "visits"))["notes"]["columns"]["status"]["codes"]
assert "按前 10 行统计" in codes["note"] and "complete" not in codes
assert "按存储顺序取前 10 行统计" in codes["note"] and "complete" not in codes
# ==========================================================================
@@ -0,0 +1,178 @@
"""剖析 SQL 的方言金样(下一轮 03 细案 C2):没有真库时挡住方言写法的回退。
三层,各挡各的:
1. **金样**:每种方言(mysql、mariadb、postgresql、oracle、sqlite)× 每类查询(统计信息、有上限的计数、取不同键值、
对键值、唯一性、取值分布、日期最小最大、时间窗口里的取值分布、按日行数、块采样),完整 SQL 钉在
tests/fixtures/dialect/<方言>.sql,人审过一次;以后拼法有任何改动都会出 diff。要更新金样:
``UPDATE_DIALECT_GOLDEN=1 pytest tests/test_profile_dialect_golden.py``,再逐行看 git diff。
2. **sqlglot 逐条解析**:按方言读一遍(mariadb 按 mysql 读),挡住语法写坏了(少括号、关键字拼错)。
3. **IN 两边类型族一致**:用 sqlglot 的 qualify + annotate_types,配上表结构里的类型,核对 IN 列表的字面量和被指向
列是同一类型族(PostgreSQL 上 varchar 父列配整数子列时,字面量必须带引号)。
挡不住的照实写:sqlglot 不知道真库的统计信息是不是过期、零日期驱动怎么返回、Oracle 小写对象是否存在、排序规则
怎么比——这些要靠 scripts/dialect-it.sh 在本机容器里跑真库(不进 CI)。
"""
from __future__ import annotations
import os
from pathlib import Path
import pytest
import sqlglot
from sqlglot import exp
from sqlglot.optimizer.annotate_types import annotate_types
from sqlglot.optimizer.qualify import qualify
from app.data import catalog, guard
from app.data.catalog_profile import IN_CHUNK, SqlDialect, key_literals
GOLDEN = Path(__file__).parent / "fixtures" / "dialect"
DIALECTS = ("mysql", "mariadb", "postgresql", "oracle", "sqlite")
#: sqlglot 的方言名
READ = {"mysql": "mysql", "mariadb": "mysql", "postgresql": "postgres", "oracle": "oracle", "sqlite": "sqlite"}
#: 夹具表结构:一张带 schema 的订单表、一张门店表(编码是文字),Oracle 的一张小写带引号的表和一个小写带引号的列
META = {
"orders": {"schema": "sales", "columns": [
{"name": "id", "type": "INTEGER"}, {"name": "store_code", "type": "INTEGER"},
{"name": "status", "type": "CHAR(2)"}, {"name": "paid_at", "type": "TIMESTAMP"},
{"name": "biz_day", "type": "VARCHAR(10)"}, {"name": "level", "type": "INTEGER"}],
"primary_key": ["id"]},
"stores": {"schema": None, "columns": [{"name": "code", "type": "VARCHAR(10)"}], "primary_key": ["code"]},
"visits": {"schema": None, "quote_name": True, "columns": [
{"name": "id", "type": "INTEGER"}, {"name": "level", "type": "INTEGER", "quote_name": True}],
"primary_key": ["id"]},
}
def _queries(kind: str) -> list[tuple[str, str | None]]:
"""一种方言的全部剖析、观察查询:(名字, SQL)。没有这种查询(SQLite 没有统计信息、MySQL 没有块采样)的 SQL 为 None。"""
d = SqlDialect(kind)
orders, stores, visits = d.table(META["orders"], "orders"), d.table(META["stores"], "stores"), \
d.table(META["visits"], "visits")
status, code = d.column(META["orders"], "status"), d.column(META["stores"], "code")
level = d.column(META["visits"], "level")
keys, _ = key_literals([12, 7], "number", parent="text", child_type="INTEGER")
out: list[tuple[str, str | None]] = [
("stats", d.stats_sql(META["orders"], "orders")),
("stats_quoted", d.stats_sql(META["visits"], "visits")),
("bounded", d.bounded_count_sql(orders, 100_001)),
("distinct", d.distinct_sample_sql(orders, d.column(META["orders"], "store_code"), 2000, scan_cap=None)),
("distinct_capped", d.distinct_sample_sql(orders, d.column(META["orders"], "store_code"), 2000,
scan_cap=100_000)),
("match", d.match_count_sql(stores, code, keys)),
("unique", d.unique_check_sql(visits, level)),
("value_counts", d.value_counts_sql(orders, status, 21, scan_cap=None)),
("value_counts_capped", d.value_counts_sql(orders, status, 21, scan_cap=100_000)),
("min_max", d.min_max_sql(orders, [(d.column(META["orders"], "paid_at"), False),
(d.column(META["orders"], "biz_day"), True)])),
("date_window", d.window_value_counts_sql(orders, status, d.column(META["orders"], "paid_at"),
"2026-08-27", 21)),
("day_volume", d.day_volume_sql(orders, d.column(META["orders"], "paid_at"), "2026-08-27")),
("sample", d.sample_value_counts_sql(orders, status, 1, 21)),
]
return out
def _render(kind: str) -> str:
lines = [f"-- 剖析与取值观察的 SQL 金样:{kind}(tests/test_profile_dialect_golden.py 生成,改动要人审)"]
for name, sql in _queries(kind):
lines.append(f"-- {name}")
lines.append(f"{sql};" if sql is not None else "-- (这种方言没有这类查询)")
return "\n".join(lines) + "\n"
@pytest.mark.parametrize("kind", DIALECTS)
def test_profile_sql_matches_the_golden_file(kind):
path = GOLDEN / f"{kind}.sql"
text = _render(kind)
if os.environ.get("UPDATE_DIALECT_GOLDEN") == "1":
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(text, encoding="utf-8")
assert path.read_text(encoding="utf-8") == text, f"{kind} 的剖析 SQL 和金样不一致:看 diff,确认无误后重新生成金样"
@pytest.mark.parametrize("kind", DIALECTS)
def test_every_profile_query_parses_and_passes_the_guard(kind):
for name, sql in _queries(kind):
if sql is None:
continue
parsed = sqlglot.parse(sql, read=READ[kind], error_level=sqlglot.ErrorLevel.RAISE)
assert len(parsed) == 1 and parsed[0] is not None, (kind, name)
assert guard.check(sql, readonly=True, dialect=kind) == sql, (kind, name)
@pytest.mark.parametrize("kind", ["postgresql", "oracle", "mysql"])
def test_observation_queries_use_the_right_day_and_sample_functions(kind):
"""按天截断和块采样各家写法不同,解析成语法树再看用的是哪个函数,不只比字符串。"""
sql = dict(_queries(kind))
day = sqlglot.parse_one(sql["day_volume"], read=READ[kind])
if kind == "oracle":
assert any(isinstance(n, (exp.DateTrunc, exp.TimestampTrunc, exp.Anonymous)) or n.key == "trunc"
for n in day.walk()), sql["day_volume"]
assert "TRUNC(paid_at)" in sql["day_volume"]
elif kind == "postgresql":
assert any(isinstance(n, exp.Cast) for n in day.walk())
else:
assert "DATE(paid_at)" in sql["day_volume"]
if kind == "mysql":
assert sql["sample"] is None
else:
sampled = sqlglot.parse_one(sql["sample"], read=READ[kind])
assert any(isinstance(n, exp.TableSample) for n in sampled.walk()), sql["sample"]
# ---------------------------------------------------------------- IN 两边的类型族
_FAMILY_OF = [(exp.DataType.TEXT_TYPES, "text"), (exp.DataType.NUMERIC_TYPES, "number"),
(exp.DataType.TEMPORAL_TYPES, "time")]
def _sg_family(t: exp.DataType | None) -> str | None:
if t is None:
return None
for group, name in _FAMILY_OF:
if t.this in group:
return name
return None
def _in_families(sql: str, kind: str, schema: dict[str, dict[str, str]]) -> list[tuple[str | None, set[str | None]]]:
"""sql 里每个 IN:(被比较的列的类型族, 字面量的类型族集合),按 annotate_types 配上表结构的类型推出来。"""
tree = qualify(sqlglot.parse_one(sql, read=READ[kind]), schema=schema, dialect=READ[kind],
validate_qualify_columns=False)
tree = annotate_types(tree, schema=schema, dialect=READ[kind])
return [(_sg_family(node.this.type), {_sg_family(x.type) for x in node.expressions})
for node in tree.find_all(exp.In)]
@pytest.mark.parametrize("kind", DIALECTS)
@pytest.mark.parametrize("parent_type, child_values, child_type", [
("VARCHAR(10)", [12, 7, 30], "INTEGER"), # 父列文字、子列数:字面量要带引号
("INTEGER", ["12", "0007", "A1"], "VARCHAR(10)"), # 父列数、子列文字:只收整数写法,写成数字
("VARCHAR(10)", ["A ", "BC"], "CHAR(2)"), # 定长子列:去掉补齐的空格
("INTEGER", [1, 2], "INTEGER"),
])
def test_in_list_literals_have_the_parent_columns_family(kind, parent_type, child_values, child_type):
d = SqlDialect(kind)
parent = {"name": "code", "type": parent_type}
literals, _ = key_literals(child_values, None, parent=catalog.column_family(parent), child_type=child_type)
assert literals
sql = d.match_count_sql("stores", d.quote("code"), literals[:IN_CHUNK])
found = _in_families(sql, kind, {"stores": {"code": parent_type}})
assert found, sql
for column_family, literal_families in found:
assert literal_families == {column_family}, (kind, sql, column_family, literal_families)
def test_the_old_way_would_mismatch_on_postgres():
"""对照:以前按子表取回的值写字面量,varchar 父列配整数子列在 sqlglot 推出的类型上就两边不同——PostgreSQL 真库上
这是「operator does not exist: character varying = integer」。"""
from app.data.catalog_profile import sql_literal
old = [lit for v in (12, 7) if (lit := sql_literal(v, "number"))]
sql = SqlDialect("postgresql").match_count_sql("stores", "code", old)
[(column_family, literal_families)] = _in_families(sql, "postgresql", {"stores": {"code": "VARCHAR(10)"}})
assert column_family == "text" and literal_families == {"number"}
+296
View File
@@ -0,0 +1,296 @@
"""剖析与探查的方言修正(下一轮 03 细案 C1):不连库的部分。
- 列类型按方言编译的完整写法:类型族、日期列、码值候选都认得 ENUM / SET、VARCHAR2(20 CHAR)、INTERVAL 这些写法;
- IN 列表的字面量按被指向列的类型族生成(PG 上文字列配整数子列会直接报错,Oracle 会把父列转成数报 ORA-01722);
- Oracle 小写带引号的对象(quote_name)强制加引号、不转大写;
- 统计信息估出的行数不超过整表统计上限时再数一次确认;
- 日期列的最小值、最大值排除零日期和哨兵日期,照实写排除了几个;
- 取值观察要用的三个方言方法(date_literal / day_trunc / sample_from);
- 说明文字不再说「抽样」:大表看的是按存储顺序的前 N 行。
"""
from __future__ import annotations
import asyncio
import datetime as _dt
from typing import Any
import pytest
from sqlalchemy import VARCHAR
from sqlalchemy.dialects import mysql, oracle, postgresql, sqlite
from sqlalchemy.sql.elements import quoted_name
from app.data import catalog, catalog_profile, introspect
from app.data.catalog_profile import ProfileSettings, SqlDialect, TableSize
from app.data.engine import QueryResult
DIALECTS = ("mysql", "mariadb", "postgresql", "oracle", "sqlite")
# ---------------------------------------------------------------- 类型的写法
@pytest.mark.parametrize("type_name, family", [
("ENUM('paid','refund')", "text"),
("ENUM('int','date')", "text"), # 取值里的词不能让它变成数字或时间
("SET('a','b')", "text"),
("VARCHAR2(20 CHAR)", "text"),
("NVARCHAR2(10)", "text"),
("VARCHAR(20) COLLATE utf8mb4_bin", "text"),
("NUMBER(10, 0)", "number"),
("INTEGER(11)", "number"),
("BIGINT UNSIGNED", "number"),
("DOUBLE PRECISION", "number"),
("TIMESTAMP WITH TIME ZONE", "time"),
("TIMESTAMP(3)", "time"),
("INTERVAL DAY(2) TO SECOND(6)", None), # 时长不是数也不是时刻;以前按「INT」认成了数字
("order_status", None),
("", None),
])
def test_type_family_reads_dialect_full_types(type_name, family):
assert catalog._type_family(type_name) == family
def test_type_family_of_a_column_uses_its_enum_values():
"""PostgreSQL 的枚举类型编译出来是类型名(order_status),只有取值列表能说明它是文字。"""
assert catalog.column_family({"name": "status", "type": "order_status", "enum_values": ["a", "b"]}) == "text"
assert catalog.column_family({"name": "n", "type": "INTEGER"}) == "number"
assert catalog.column_family({"name": "x"}) is None
def test_enum_values_never_make_a_date_column():
assert not catalog.is_date_column("status", "ENUM('date','time')")
assert catalog.is_date_column("paid_at", "TIMESTAMP WITH TIME ZONE")
assert catalog.is_date_column("biz_dt", "VARCHAR2(10 CHAR)") # 文字存日期、名字像日期
assert not catalog.is_date_column("gap", "INTERVAL DAY(2) TO SECOND(6)")
@pytest.mark.parametrize("col, ok", [
({"type": "VARCHAR2(20 CHAR)"}, True),
({"type": "VARCHAR2(200 CHAR)"}, False),
({"type": "CHAR(2)"}, True),
({"type": "ENUM('paid','refund')"}, True),
({"type": "SET('a','b')"}, True),
({"type": "order_status", "enum_values": ["a", "b"]}, True),
({"type": "NUMBER(10, 0)"}, True),
({"type": "NUMBER(10, 2)"}, False),
({"type": "TIMESTAMP(3)"}, False),
({"type": "INTERVAL DAY(2) TO SECOND(6)"}, False),
])
def test_code_type_accepts_enum_and_char_semantics(col, ok):
assert catalog_profile._code_type_ok(col.get("type"), col.get("enum_values")) is ok
def test_enum_columns_are_code_candidates_whatever_their_name():
"""ENUM 的取值列表有限,直接算码值候选:pay_way 这种名字不像码值的列以前被漏掉。"""
meta = {"columns": [{"name": "id", "type": "INTEGER"}, {"name": "pay_way", "type": "ENUM('cash','card')",
"enum_values": ["cash", "card"]},
{"name": "pay_note", "type": "VARCHAR(20)"}],
"primary_key": ["id"]}
assert catalog_profile._code_columns(meta, {}) == ["pay_way"]
# ---------------------------------------------------------------- 探查:类型写法、取值列表、引号、索引首列
@pytest.mark.parametrize("dialect, col_type, text", [
(mysql.dialect(), mysql.ENUM("paid", "refund"), "ENUM('paid','refund')"),
(mysql.dialect(), mysql.TINYINT(display_width=1), "TINYINT(1)"),
(oracle.dialect(), oracle.VARCHAR2(20), "VARCHAR2(20 CHAR)"),
# Oracle 的 VARCHAR2 反射回来是通用的 VARCHAR(容器集成测试实测):照数据字典写成 VARCHAR2
(oracle.dialect(), VARCHAR(10), "VARCHAR2(10 CHAR)"),
(oracle.dialect(), oracle.NUMBER(10, 0), "NUMBER(10, 0)"),
(postgresql.dialect(), postgresql.ENUM("a", "b", name="order_status"), "order_status"),
(sqlite.dialect(), sqlite.DATETIME(), "DATETIME"),
])
def test_reflected_type_is_compiled_for_the_dialect(dialect, col_type, text):
assert introspect.type_text(col_type, dialect) == text
def test_type_that_cannot_compile_falls_back_to_str():
from sqlalchemy.types import NullType
assert introspect.type_text(NullType(), sqlite.dialect()) == "NULL"
assert introspect.type_text(None, sqlite.dialect()) == ""
def test_column_entry_records_enum_values_and_oracle_quoted_names():
ora = oracle.dialect()
entry = introspect.column_entry({"name": quoted_name("level", quote=True), "type": oracle.VARCHAR2(10),
"nullable": True, "comment": None}, ora)
assert entry == {"name": "level", "type": "VARCHAR2(10 CHAR)", "nullable": True, "comment": None,
"quote_name": True}
plain = introspect.column_entry({"name": "status", "type": mysql.ENUM("a", "b"), "nullable": False}, mysql.dialect())
assert plain["enum_values"] == ["a", "b"] and "quote_name" not in plain
members = introspect.column_entry({"name": "tags", "type": mysql.SET("x", "y")}, mysql.dialect())
assert members["enum_values"] == ["x", "y"] and members["type"] == "SET('x','y')"
def test_index_first_columns_are_recorded():
indexes = [{"column_names": ["paid_at", "id"]}, {"column_names": [None]}, {"column_names": ["store_id"]},
{"column_names": ["paid_at"]}]
assert introspect.index_first_columns(indexes, ["id"]) == ["id", "paid_at", "store_id"]
# ---------------------------------------------------------------- IN 列表的字面量按被指向列的类型族
def test_key_literals_follow_the_parent_family():
lits, mismatched = catalog_profile.key_literals([12, 7], "number", parent="text", child_type="INTEGER")
assert lits == ["'12'", "'7'"] and mismatched == 0
lits, mismatched = catalog_profile.key_literals(["12", "A1", "-3"], "text", parent="number",
child_type="VARCHAR(10)")
assert lits == ["12", "-3"] and mismatched == 1
lits, mismatched = catalog_profile.key_literals([1, 2], "number", parent="time", child_type="INTEGER")
assert lits == [] and mismatched == 2
# 父列类型不知道:照子表取回的值的类型写(以前的做法)
assert catalog_profile.key_literals([3, "x"], None, parent=None, child_type=None) == (["3", "'x'"], 0)
def test_char_keys_lose_their_padding():
"""PG、Oracle 的 CHAR(n) 取回来带尾随空格:拼成 'A ' 去和 VARCHAR 父列比就对不上。"""
lits, _ = catalog_profile.key_literals(["A ", "BC"], "text", parent="text", child_type="CHAR(2)")
assert lits == ["'A'", "'BC'"]
lits, _ = catalog_profile.key_literals(["A "], "text", parent="text", child_type="VARCHAR(2)")
assert lits == ["'A '"]
# ---------------------------------------------------------------- Oracle 小写带引号的对象
def test_oracle_quote_name_forces_quotes_and_keeps_case():
ora = SqlDialect("oracle")
meta = {"schema": "sales", "quote_name": True,
"columns": [{"name": "level", "type": "VARCHAR2(10 CHAR)", "quote_name": True},
{"name": "amount", "type": "NUMBER(10, 2)"}]}
assert ora.table(meta, "visits") == 'sales."visits"'
assert ora.column(meta, "level") == '"level"' and ora.column(meta, "amount") == "amount"
stats = ora.stats_sql(meta, "visits")
assert "TABLE_NAME = 'visits'" in stats and "OWNER = 'SALES'" in stats
# 没有 quote_name 的照旧:不加引号,Oracle 自己转大写
assert ora.table({"schema": None}, "visits") == "visits"
assert "TABLE_NAME = 'VISITS'" in ora.stats_sql({}, "visits")
# ---------------------------------------------------------------- 统计信息过期:再数一次确认
def _result(value: Any) -> QueryResult:
return QueryResult(columns=["n"], rows=[[value]], row_count=1, truncated=False, elapsed_ms=1, sql="")
def _runner(answers: list[Any], seen: list[str]):
async def run(sql: str, **_kw: Any) -> QueryResult:
seen.append(sql)
return _result(answers.pop(0))
return run
@pytest.mark.parametrize("stats, counted, expected", [
(500, 500, TableSize(500, "count")), # 统计信息说小、数过也小:准数
(500, 100_001, TableSize(None, "count", at_least=100_001)), # 统计信息过期:其实是大表
(5_000_000, None, TableSize(5_000_000, "stats")), # 统计信息说大:不必再数
])
def test_small_stats_are_confirmed_with_a_bounded_count(stats, counted, expected):
seen: list[str] = []
answers = [stats] + ([counted] if counted is not None else [])
size, skips = asyncio.run(catalog_profile.estimate_size(_runner(answers, seen), SqlDialect("postgresql"),
{"schema": None}, "visits", max_scan_rows=100_000))
assert size == expected and skips == []
assert len(seen) == (2 if counted is not None else 1)
if counted is not None:
assert "LIMIT 100001" in seen[1]
# ---------------------------------------------------------------- 日期列:排除零日期和哨兵日期
@pytest.mark.parametrize("kind", DIALECTS)
def test_min_max_excludes_invalid_dates_and_counts_them(kind):
d = SqlDialect(kind)
sql = d.min_max_sql("t", [("paid_at", False), ("biz_day", True)])
# 日期类型的列按日期字面量比,文字存的日期按字符串比(PG 上 varchar 和 date 比较直接报错)
lo, hi = d.date_literal("0001-01-02"), d.date_literal("9999-12-31")
assert f"paid_at >= {lo}" in sql and f"paid_at < {hi}" in sql
assert "biz_day >= '0001-01-02'" in sql and "biz_day < '9999-12-31'" in sql
assert sql.count("SUM(CASE WHEN") == 2
def _dates_cx(row: list[Any], kind: str = "mysql"):
settings = ProfileSettings(enabled=True)
source = type("S", (), {"id": "s1", "name": "shop", "kind": kind, "options": {}})()
job = catalog_profile.ProfileContext(source=source, settings=settings, dialect=SqlDialect(kind),
runner=None, tables={"orders": {"columns": [
{"name": "paid_at", "type": "DATETIME"}]}}, masked=set())
job.sizes["orders"] = TableSize(10, "count")
result = catalog_profile.TableResult(table="orders")
cx = catalog_profile._TableCx(job, "orders", result, catalog_profile.TableHooks())
async def run(sql: str, **_kw: Any) -> QueryResult:
return QueryResult(columns=["a", "b", "c"], rows=[row], row_count=1, truncated=False, elapsed_ms=1, sql=sql)
cx.run = run # type: ignore[method-assign]
return cx, result
def test_dates_note_says_how_many_invalid_values_were_left_out():
cx, result = _dates_cx(["2026-01-03 08:00:00", "2026-09-30 21:00:00", 4])
out = catalog_profile._TableOutcome()
asyncio.run(catalog_profile._check_dates(cx, "orders", ["paid_at"], 1, out, result))
assert result.date_ranges == [{"column": "paid_at", "min": "2026-01-03 08:00:00", "max": "2026-09-30 21:00:00",
"invalid": 4}]
assert "另有 4 个无效日期" in out.business_date.note and "0000-00-00" in out.business_date.note
def test_only_invalid_dates_is_no_data():
cx, result = _dates_cx([None, None, 9])
with pytest.raises(catalog_profile._Skip) as e:
asyncio.run(catalog_profile._check_dates(cx, "orders", ["paid_at"], 1, catalog_profile._TableOutcome(),
result))
assert e.value.reason == "no_data" and "9 个无效日期" in e.value.detail
# ---------------------------------------------------------------- 取值观察要用的方言方法
@pytest.mark.parametrize("kind, literal, day, sample", [
("mysql", "DATE '2026-09-01'", "DATE(paid_at)", None),
("mariadb", "DATE '2026-09-01'", "DATE(paid_at)", None),
("postgresql", "DATE '2026-09-01'", "CAST(paid_at AS DATE)", "orders TABLESAMPLE SYSTEM (1.5)"),
("oracle", "DATE '2026-09-01'", "TRUNC(paid_at)", "orders SAMPLE BLOCK (1.5)"),
("sqlite", "'2026-09-01'", "substr(paid_at, 1, 10)", None),
])
def test_observation_dialect_methods(kind, literal, day, sample):
d = SqlDialect(kind)
assert d.date_literal(_dt.date(2026, 9, 1)) == literal == d.date_literal("2026-09-01")
assert d.date_literal("2026-09-01", text=True) == "'2026-09-01'"
assert d.day_trunc("paid_at") == day
assert d.sample_from("orders", 1.5) == sample
@pytest.mark.parametrize("bad", ["2026-9-1", "2026-09-01'; --", "0000-00-00", 20260901, None])
def test_date_literal_refuses_anything_but_a_real_date(bad):
with pytest.raises(ValueError):
SqlDialect("postgresql").date_literal(bad)
@pytest.mark.parametrize("bad", [0, -1, 100, 101, float("nan")])
def test_sample_percent_must_be_between_0_and_100(bad):
with pytest.raises(ValueError):
SqlDialect("postgresql").sample_from("orders", bad)
# ---------------------------------------------------------------- 说明文字
def test_notes_say_storage_order_not_sampling():
note = catalog_profile._relation_note(sample=2000, matched=2000, coverage=1.0, scan_cap=100_000, skipped_values=0,
mismatched=0, parent=(True, "主键"), child=(False, "count"), holds=True,
collation=None)
assert "按存储顺序只看前 100000 行" in note and "抽样" not in note
assert catalog_profile.FILTER_SAMPLED.format(rows="100", share="1%").startswith("剖析核对:按存储顺序取前 100 行统计")
def test_mysql_notes_mention_the_collation():
note = catalog_profile._relation_note(sample=10, matched=10, coverage=1.0, scan_cap=None, skipped_values=0,
mismatched=0, parent=(True, "主键"), child=(False, "count"), holds=True,
collation=catalog_profile.MYSQL_COLLATION_NOTE)
assert "排序规则" in note and "不分大小写" in note