mirror of
https://github.com/ThearchyHelios/AgentLab.git
synced 2026-11-18 18:11:58 +00:00
SQL 差异说明收尾(第 2 轮):PostgreSQL 符号运算符和 || 混用不判等价;条件写在聚合里、时间窗口、反连接、经主键定死的列不再说错
- 保真守卫(_misread):PostgreSQL 写成符号、sqlglot 却当成 LIKE 那一级来读的运算符(~、~*、~~、~~*、@@、-|-、@> 等) 和别的二元运算混用时拦下(PostgreSQL 里它们和 ||、& 同级、从左往右,status ~ 'p' || 'x' 是 (status ~ 'p') || 'x'); 另拦 PostgreSQL 的前缀 @、|/、||/、COLLATE 后面的 ::;不知道是哪种库时拦块注释里套着的 /*、紧跟着字的 --、#、双引号 括起来的名字(MySQL 默认的 sql_mode 里是字符串)。都只给兜底,附原文的逐行对照。 - 条件写在聚合的参数里(SUM(status = 'paid')、COUNT(… OR NULL)、COUNT(NULLIF(…))、SUM(amount * (1 - is_deleted))) 算作这一边对这一列有条件(_Usage 的 case),另一边多出同一列上的 WHERE 条件时不再说「这一边没有这个条件」,只给兜底。 - 时间:timed 认 BETWEEN 和时间列上 LIKE 像日期的模式(paid_at LIKE '2026-09%'),时间窗口不再被说成有效行过滤; 「另一边没有按时间筛选」之前看另一边可能是时间窗口的条件(聚合参数里的时间比较、date_key、stat_month 这类列、 20260901 / 202609 这类整数、上下界一对),有就不说;认不准是不是时间列的窗口(_window_like)不说成有效行过滤。 两边的时间列在不同的表上时写成「表.列」。 - 反连接:用到 LEFT JOIN 可空一侧的列的条件(WHERE r.id IS NULL、r.order_id IS NULL)不说成有效行过滤,时间列在 可空一侧时也不说按它筛时间。 - 给了表结构时,聚合参数所在的表的键在组里定死了(按 u.id 分组时的 u.city),MAX 对 MIN、AVG(DISTINCT) 对 AVG 不说。 - 另一边原文也 LEFT JOIN 了、规范化因为结果不怕重复去掉了的表,不说「另一边没有关联」。 - 回归语料:收尾第 1 轮之后复验报的全部 blocker / major 收进 sql_pairs(_DIFF_R5、_DIFF_TIME_R5、_SAME_HARD_R5、 _SAME_SCHEMA_R5 执行;REGRESSIONS_R5、EXPLAIN_TRAPS_R5 只查结论),新增对应的测试。 - 模块说明照实补上:新增的守卫、时间条件的认法、只给兜底的情况、收窄换来的代价、MySQL 按默认 sql_mode 读的假设。 认出等价的比例:语料(SQLite,给表结构)169 对里 157 对,和改动前一样;PostgreSQL、MySQL 也不变;不知道是哪种库时 少认 1 对(same:quoted-names,双引号括起来的名字)。
This commit is contained in:
Vendored
+211
-3
@@ -16,8 +16,9 @@ OR 里的标量子查询改成 CROSS JOIN,都是复验时真踩到的),所
|
||||
DIALECT_SAME_R2、KNOWN_UNRECOGNIZED(各方言,只查结论)里;第 3 轮的在 _DIFF_R3、_SAME_R3、_SAME_TYPED_R3(要给表结构才认得出,Pair.typed)、_SAME_SCHEMA_R3(要靠
|
||||
表结构里的主键才看得出等价,Pair.needs_schema)、_SAME_HARD_R3(执行)和 REGRESSIONS_R3、DIALECT_SAME_R3(只查
|
||||
结论)里;第 4 轮的在 _DIFF_R4、_SAME_HARD_R4、_SAME_SCHEMA_R4(执行)和 REGRESSIONS_R4、DIALECT_SAME_R4、
|
||||
CONSTRAINT_TRAPS_R4(只查结论,靠约束改写的陷阱另造一个小库执行)里。规范化不再靠约束改写,needs_schema 的对一律
|
||||
不判等价,给了表结构时只许给兜底。LIMIT 都带能定出唯一顺序的排序,没有 ORDER BY 的 LIMIT 不进语料(EXISTS 里的 LIMIT 1 OFFSET 1 只看有没有
|
||||
CONSTRAINT_TRAPS_R4(只查结论,靠约束改写的陷阱另造一个小库执行)里;第 5 轮的在 _DIFF_R5、_DIFF_TIME_R5、
|
||||
_SAME_HARD_R5、_SAME_SCHEMA_R5(执行)和 REGRESSIONS_R5、EXPLAIN_TRAPS_R5(只查结论)里。规范化不再靠约束改写,
|
||||
needs_schema 的对一律不判等价,给了表结构时只许给兜底。LIMIT 都带能定出唯一顺序的排序,没有 ORDER BY 的 LIMIT 不进语料(EXISTS 里的 LIMIT 1 OFFSET 1 只看有没有
|
||||
第二行,不看先后)。过去复验报过的反例都收进来了:第 5 轮回退的「只在 IN 子查询里用到的 CTE」在 PAIRS 里;复验者留在 /private/tmp 的脚本里、在这几组数据上
|
||||
结果不同的对是 VALIDATOR_DIFF(也进 PAIRS 执行);旧测试(五轮复核修正按复验报告补的用例,各方言,用到的列这个库里
|
||||
不全有)里报过不同的对是 REGRESSIONS、意思一样的对是 DIALECT_SAME,这两份不执行,只查判没判等价。
|
||||
@@ -3002,6 +3003,205 @@ def coupon_schema(dialect: str) -> dict:
|
||||
return _coupon_schema(mysql=dialect == "mysql")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------- 第 5 轮复验的反例
|
||||
# 收尾第 1 轮之后的复验(回归、漏报、错报三位复验者)报的全部 blocker / major,外加自己在真实的 PostgreSQL 18.6 上
|
||||
# 差分压测(不加括号的运算符混用对 sqlglot 读出来的树每个操作数都加括号)找到的同一类写法。复验者的 channel 列改写成
|
||||
# 这里的 region,customers / invoices 改写成 users / orders;数仓的 sales_fact 这个库里没有,只查结论(见
|
||||
# EXPLAIN_TRAPS_R5)。
|
||||
|
||||
#: 不许判等价、也不许给具体说法的(SQLite 上执行):两条确实不同,以前却说错了不同在哪——
|
||||
#: - 条件写在聚合的参数里(SUM(status = 'paid')、COUNT(… OR NULL)、SUM(… IN (…))、COUNT(NULLIF(…))、
|
||||
#: SUM(amount * (status = 'paid'))、SUM(amount * (1 - is_deleted))):这一边也只数、只加满足条件的行,说「这一边没有
|
||||
#: 这个条件」等于说它把所有行都算进去了(真正的不同是另一边没有计数为 0 的组、空集时是 NULL 还是 0);
|
||||
#: - 时间条件写在聚合的参数里(SUM(paid_at >= … AND paid_at < …)):这一边是按 paid_at 归月的,不能说「没有按时间
|
||||
#: 筛选」;
|
||||
#: - 反连接(LEFT JOIN … WHERE 可空一侧的列 IS NULL):说的是「没关联上」,不是有效行过滤;
|
||||
#: - 规范化因为结果不怕重复去掉了的 LEFT JOIN:原文明明写了 LEFT JOIN refunds,不能说「这一边没有关联 refunds 表」。
|
||||
_DIFF_R5 = [
|
||||
("r5-sum-of-condition", "SELECT region, SUM(status = 'paid') FROM orders GROUP BY region",
|
||||
"SELECT region, COUNT(*) FROM orders WHERE status = 'paid' GROUP BY region"),
|
||||
("r5-count-condition-or-null", "SELECT region, COUNT(status = 'paid' OR NULL) FROM orders GROUP BY region",
|
||||
"SELECT region, COUNT(*) FROM orders WHERE status = 'paid' GROUP BY region"),
|
||||
("r5-sum-of-in-list", "SELECT region, SUM(status IN ('paid', 'shipped')) FROM orders GROUP BY region",
|
||||
"SELECT region, COUNT(*) FROM orders WHERE status IN ('paid', 'shipped') GROUP BY region"),
|
||||
("r5-count-nullif", "SELECT region, COUNT(NULLIF(status, 'void')) FROM orders GROUP BY region",
|
||||
"SELECT region, COUNT(*) FROM orders WHERE status <> 'void' GROUP BY region"),
|
||||
("r5-sum-times-condition", "SELECT SUM(amount * (status = 'paid')) FROM orders",
|
||||
"SELECT SUM(amount) FROM orders WHERE status = 'paid'"),
|
||||
("r5-sum-times-flag", "SELECT region, SUM(amount * (1 - is_deleted)) FROM orders GROUP BY region",
|
||||
"SELECT region, SUM(amount) FROM orders WHERE is_deleted = 0 GROUP BY region"),
|
||||
("r5-sum-times-flag-ungrouped", "SELECT SUM(amount * (1 - is_deleted)) FROM orders",
|
||||
"SELECT SUM(amount) FROM orders WHERE is_deleted = 0"),
|
||||
("r5-time-window-inside-sum", "SELECT region, SUM(paid_at >= '2026-09-01' AND paid_at < '2026-10-01') FROM orders "
|
||||
"GROUP BY region",
|
||||
"SELECT region, COUNT(*) FROM orders WHERE created_at >= '2026-09-01' AND created_at < '2026-10-01' "
|
||||
"GROUP BY region"),
|
||||
("r5-time-lower-bound-inside-sum", "SELECT SUM(paid_at >= '2026-09-01') FROM orders",
|
||||
"SELECT COUNT(*) FROM orders WHERE paid_at >= '2026-09-01'"),
|
||||
("r5-anti-join-vs-inner-join", "SELECT COUNT(*) FROM orders o LEFT JOIN payments p ON p.order_id = o.id "
|
||||
"WHERE p.id IS NULL",
|
||||
"SELECT COUNT(*) FROM orders o JOIN payments p ON p.order_id = o.id"),
|
||||
("r5-anti-join-key-vs-nullable-column", "SELECT u.id, u.name FROM users u LEFT JOIN orders o ON o.user_id = u.id "
|
||||
"WHERE o.id IS NULL",
|
||||
"SELECT u.id, u.name FROM users u LEFT JOIN orders o ON o.user_id = u.id WHERE o.created_at IS NULL"),
|
||||
("r5-dropped-left-join-vs-inner-join", "SELECT COUNT(DISTINCT o.user_id) FROM orders o LEFT JOIN refunds r "
|
||||
"ON r.order_id = o.id",
|
||||
"SELECT COUNT(DISTINCT o.user_id) FROM orders o JOIN refunds r ON r.order_id = o.id"),
|
||||
]
|
||||
|
||||
#: 时间列上的日期范围(BETWEEN、LIKE '2026-09%')是统计窗口,不是有效行过滤(SQLite 上执行,不许判等价):以前在结果
|
||||
#: 看行先后的一段里(按日期表达式分组、取前 N 行)BETWEEN 不拆,被说成两条「只保留 … 的行」,时间列这一事实丢了;
|
||||
#: LIKE 取月也被说成有效行过滤。现在说成 time,或者只给兜底,不许说成 filter。每一项 (编号, A, B, A 的时间列, B 的
|
||||
#: 时间列):SQLite 里说成 time 时 detail 里就是这两列(没有按时间筛选的一边是空列表)
|
||||
_DIFF_TIME_R5 = [
|
||||
("r5-between-on-two-time-columns-grouped-by-date",
|
||||
"SELECT date(paid_at) AS d, SUM(amount) FROM orders WHERE paid_at BETWEEN '2026-09-01' AND '2026-09-30 23:59:59' "
|
||||
"GROUP BY 1",
|
||||
"SELECT date(paid_at) AS d, SUM(amount) FROM orders "
|
||||
"WHERE created_at BETWEEN '2026-09-01' AND '2026-09-30 23:59:59' GROUP BY 1", ["paid_at"], ["created_at"]),
|
||||
("r5-between-on-two-time-columns-top-n",
|
||||
"SELECT region, SUM(amount) AS revenue FROM orders WHERE paid_at BETWEEN '2026-09-01' AND '2026-09-30 23:59:59' "
|
||||
"GROUP BY region ORDER BY revenue DESC, region LIMIT 3",
|
||||
"SELECT region, SUM(amount) AS revenue FROM orders "
|
||||
"WHERE created_at BETWEEN '2026-09-01' AND '2026-09-30 23:59:59' "
|
||||
"GROUP BY region ORDER BY revenue DESC, region LIMIT 3", ["paid_at"], ["created_at"]),
|
||||
("r5-between-window-on-one-side-grouped-by-date",
|
||||
"SELECT date(paid_at) AS d, SUM(amount) FROM orders WHERE status = 'paid' GROUP BY 1",
|
||||
"SELECT date(paid_at) AS d, SUM(amount) FROM orders WHERE status = 'paid' "
|
||||
"AND paid_at BETWEEN '2026-09-01' AND '2026-09-30 23:59:59' GROUP BY 1", [], ["paid_at"]),
|
||||
("r5-like-month-vs-no-window", "SELECT COUNT(*) FROM orders",
|
||||
"SELECT COUNT(*) FROM orders WHERE paid_at LIKE '2026-09%'", [], ["paid_at"]),
|
||||
("r5-range-vs-like-month-on-another-column",
|
||||
"SELECT COUNT(*) FROM orders WHERE paid_at >= '2026-09-01' AND paid_at < '2026-10-01'",
|
||||
"SELECT COUNT(*) FROM orders WHERE created_at LIKE '2026-09%'", ["paid_at"], ["created_at"]),
|
||||
]
|
||||
|
||||
#: 意思一样、要按语义推理才看得出的(SQLite 上执行):可空一侧的关联键 u.id 为空就是没关联上,整行都是空值,多写的
|
||||
#: u.name IS NULL 被 u.id IS NULL 蕴含——不靠约束。以前说「B 只保留 users.name IS NULL 的行,A 没有这个条件」
|
||||
_SAME_HARD_R5 = [
|
||||
("r5-anti-join-extra-null-test",
|
||||
"SELECT o.id FROM orders o LEFT JOIN users u ON u.id = o.user_id WHERE u.id IS NULL",
|
||||
"SELECT o.id FROM orders o LEFT JOIN users u ON u.id = o.user_id WHERE u.id IS NULL AND u.name IS NULL"),
|
||||
]
|
||||
|
||||
#: 意思一样、要靠表结构里的主键才看得出的(SQLite 上执行,needs_schema):给了表结构时只许给兜底。
|
||||
#: - 两种反连接的写法:关联键 r.order_id 为空就是没关联上,蕴含 r.id 为空;r.id 是主键(不为空)时反过来也成立。没给
|
||||
#: 表结构时也不许说具体的:r.id IS NULL 那一边留下的行更多(关联上了、r.id 却为空的行),说它「只保留 … 的行、另一边
|
||||
#: 没有这个条件」把留得多的一边说成了严的一边(见 test_anti_joins_are_never_called_a_filter);
|
||||
#: - 按主键分组(或者经关联条件等于主键):同一组里这张表的列只有一个值,LEFT JOIN、内连接把行复制了也一样,MAX 和
|
||||
#: MIN 一样、AVG(DISTINCT) 和 AVG 一样。没给表结构时说「A 取最大值,B 取最小值」按已知信息成立
|
||||
_SAME_SCHEMA_R5 = [
|
||||
("r5-anti-join-key-vs-join-column", "SELECT COUNT(*) FROM orders o LEFT JOIN refunds r ON r.order_id = o.id "
|
||||
"WHERE r.id IS NULL",
|
||||
"SELECT COUNT(*) FROM orders o LEFT JOIN refunds r ON r.order_id = o.id WHERE r.order_id IS NULL"),
|
||||
("r5-max-vs-min-of-a-column-the-group-key-determines", "SELECT u.id, MAX(u.city), COUNT(o.id) FROM users u "
|
||||
"LEFT JOIN orders o ON o.user_id = u.id GROUP BY u.id",
|
||||
"SELECT u.id, MIN(u.city), COUNT(o.id) FROM users u LEFT JOIN orders o ON o.user_id = u.id GROUP BY u.id"),
|
||||
("r5-avg-distinct-of-a-column-the-group-key-determines", "SELECT o.id, AVG(DISTINCT o.amount), COUNT(r.id) "
|
||||
"FROM orders o LEFT JOIN refunds r ON r.order_id = o.id GROUP BY o.id",
|
||||
"SELECT o.id, AVG(o.amount), COUNT(r.id) FROM orders o LEFT JOIN refunds r ON r.order_id = o.id GROUP BY o.id"),
|
||||
("r5-max-vs-min-through-the-join-key-top-n", "SELECT o.user_id, MAX(u.name) AS name, SUM(o.amount) AS total "
|
||||
"FROM orders o JOIN users u ON u.id = o.user_id WHERE o.status = 'paid' GROUP BY o.user_id "
|
||||
"ORDER BY total DESC, o.user_id LIMIT 5",
|
||||
"SELECT o.user_id, MIN(u.name) AS name, SUM(o.amount) AS total FROM orders o JOIN users u ON u.id = o.user_id "
|
||||
"WHERE o.status = 'paid' GROUP BY o.user_id ORDER BY total DESC, o.user_id LIMIT 5"),
|
||||
]
|
||||
|
||||
#: 只在 PostgreSQL 才有的写法(不执行,只查不判等价、只给兜底、附原文的逐行对照):复验者和自己在真实的 PostgreSQL
|
||||
#: 18.6 上跑出结果不同。PostgreSQL 里 ~、~*、~~、~~*、@@ 这些写成符号的匹配运算符和 ||、&、# 同级(「别的运算符」),
|
||||
#: 从左往右结合:status ~ 'p' || 'x' 是 (status ~ 'p') || 'x',得到 'truex' 这样的文字;sqlglot 把它们放在比 || 松的
|
||||
#: 一级,读成 status ~ ('p' || 'x')。前缀的 @ 是取绝对值、和 - 20 一起是 @ (amount - 20),sqlglot 读成参数 $amount;
|
||||
#: |/ 是双精度的平方根,sqlglot 写成 SQRT(numeric 的 SQRT 多出几位小数);x COLLATE "C"::text 的 :: 套在整个 COLLATE
|
||||
#: 上,sqlglot 读成套在排序规则的名字上。不知道是哪种库时:块注释里再写 /*(PostgreSQL 嵌套、SQLite / MySQL 不嵌套)、
|
||||
#: 后面紧跟着字的 --(MySQL 里是两个减号)各库读法不同。每一项 (编号, 方言, A, B),方言为空串是不知道
|
||||
REGRESSIONS_R5: list[tuple[str, str, str, str]] = [
|
||||
("pg_regex_match_then_concat", "postgres", "SELECT id, status ~ 'p' || 'x' FROM orders",
|
||||
"SELECT id, status ~ ('p' || 'x') FROM orders"),
|
||||
("pg_regex_imatch_then_concat_grouped", "postgres",
|
||||
"SELECT status ~* '^PA' || '' AS f, COUNT(*) FROM orders GROUP BY 1",
|
||||
"SELECT status ~* ('^PA' || '') AS f, COUNT(*) FROM orders GROUP BY 1"),
|
||||
("pg_regex_match_then_concat_in_cte", "postgres",
|
||||
"WITH t AS (SELECT id, status ~ '^p' || '' AS f FROM orders) SELECT f, COUNT(*) FROM t GROUP BY f",
|
||||
"WITH t AS (SELECT id, status ~ ('^p' || '') AS f FROM orders) SELECT f, COUNT(*) FROM t GROUP BY f"),
|
||||
("pg_regex_match_then_concat_with_a_time_filter", "postgres",
|
||||
"SELECT id, status ~ 'p' || 'x' FROM orders WHERE created_at >= '2026-09-01'",
|
||||
"SELECT id, status ~ ('p' || 'x') FROM orders"),
|
||||
("pg_like_operator_then_concat", "postgres", "SELECT id, status ~~ 'p%' || 'x' FROM orders",
|
||||
"SELECT id, status ~~ ('p%' || 'x') FROM orders"),
|
||||
("pg_ilike_operator_then_concat", "postgres", "SELECT id, status ~~* 'P%' || 'x' FROM orders",
|
||||
"SELECT id, status ~~* ('P%' || 'x') FROM orders"),
|
||||
("pg_text_search_then_concat", "postgres", "SELECT id, to_tsvector(status) @@ 'paid' || '' FROM orders",
|
||||
"SELECT id, to_tsvector(status) @@ ('paid' || '') FROM orders"),
|
||||
("pg_absolute_value_prefix", "postgres", "SELECT id, @ amount - 20 FROM orders",
|
||||
"SELECT id, (@ amount) - 20 FROM orders"),
|
||||
("pg_square_root_prefix", "postgres", "SELECT id, |/ amount FROM orders", "SELECT id, SQRT(amount) FROM orders"),
|
||||
("pg_collate_then_cast", "postgres", "SELECT id, status COLLATE \"C\"::text FROM orders",
|
||||
"SELECT id, status COLLATE (\"C\"::text) FROM orders"),
|
||||
# 嵌套着读是一整段注释(SELECT id);不嵌套着读,/* a /* b */ 到第一个 */ 就完了,+ 1 是代码,-- */ 是行注释
|
||||
("unknown_dialect_nested_block_comment", "", "SELECT id /* a /* b */ + 1 -- */\nFROM orders",
|
||||
"SELECT id\nFROM orders"),
|
||||
# sqlglot 当成注释吃掉 --1;MySQL 里 amount--1 是 amount - (-1)
|
||||
("unknown_dialect_dashes_without_space", "", "SELECT id, amount--1\nFROM orders", "SELECT id, amount\nFROM orders"),
|
||||
# sqlglot 的通用读法(和 PostgreSQL、SQLite 一样)把 "status" 读成列名;MySQL 默认的 sql_mode 里双引号括起来的是
|
||||
# 字符串,"status" = 'paid' 恒为假、SELECT "status" 每行都是 'status'(自查时发现,规格外,按最保守的做法拦下)
|
||||
("unknown_dialect_double_quoted_name", "", "SELECT COUNT(*) FROM orders WHERE \"status\" = 'paid'",
|
||||
"SELECT COUNT(*) FROM orders WHERE status = 'paid'"),
|
||||
("unknown_dialect_double_quoted_select", "", "SELECT \"status\" FROM orders", "SELECT status FROM orders"),
|
||||
]
|
||||
|
||||
#: 第 5 轮:要专门的表结构才看得出的(不执行,只查给的都是兜底)。每一项 (编号, 方言, A, B, 表结构, 没给表结构时也查):
|
||||
#: - 数仓事实表的业务日期键(date_key BETWEEN 20260901 AND 20260930、stat_month = 202609):认不出是不是时间列时,
|
||||
#: 不许说另一边「没有按时间筛选」,也不许把它说成有效行过滤;
|
||||
#: - 反连接的两种写法:关联上的行里主键、NOT NULL 的列不为空,两种写法都是「没关联上」;
|
||||
#: - 经关联条件等于分组键、又是那张表的主键:同一组里那张表的列只有一个值,MAX 和 MIN 一样(没给表结构时这么说按已知
|
||||
#: 信息成立,不查)。
|
||||
_SALES_FACT = {"tables": {"sales_fact": {"columns": [
|
||||
{"name": "id", "type": "INTEGER", "nullable": False}, {"name": "date_key", "type": "INTEGER", "nullable": True},
|
||||
{"name": "stat_month", "type": "INTEGER", "nullable": True}, {"name": "amount", "type": "REAL", "nullable": True},
|
||||
{"name": "created_at", "type": "TEXT", "nullable": True}], "primary_key": ["id"]}}}
|
||||
_INVOICES = {"tables": {
|
||||
"customers": {"columns": [{"name": "cid", "type": "INTEGER", "nullable": False},
|
||||
{"name": "cname", "type": "TEXT", "nullable": True},
|
||||
{"name": "region", "type": "TEXT", "nullable": True}], "primary_key": ["cid"]},
|
||||
"invoices": {"columns": [{"name": "iid", "type": "INTEGER", "nullable": False},
|
||||
{"name": "cid", "type": "INTEGER", "nullable": True},
|
||||
{"name": "issued_at", "type": "TEXT", "nullable": False},
|
||||
{"name": "amount", "type": "REAL", "nullable": True},
|
||||
{"name": "state", "type": "TEXT", "nullable": True}], "primary_key": ["iid"]}}}
|
||||
_SALES_FACT_SEPTEMBER = ("SELECT SUM(amount) FROM sales_fact "
|
||||
"WHERE created_at >= '2026-09-01' AND created_at < '2026-10-01'")
|
||||
EXPLAIN_TRAPS_R5: list[tuple[str, str, str, str, dict, bool]] = [
|
||||
("date_key_between_vs_created_at_range", "sqlite",
|
||||
"SELECT SUM(amount) FROM sales_fact WHERE date_key BETWEEN 20260901 AND 20260930",
|
||||
_SALES_FACT_SEPTEMBER, _SALES_FACT, True),
|
||||
("date_key_between_vs_created_at_range", "postgres",
|
||||
"SELECT SUM(amount) FROM sales_fact WHERE date_key BETWEEN 20260901 AND 20260930",
|
||||
_SALES_FACT_SEPTEMBER, _SALES_FACT, True),
|
||||
("date_key_between_vs_created_at_range", "",
|
||||
"SELECT SUM(amount) FROM sales_fact WHERE date_key BETWEEN 20260901 AND 20260930",
|
||||
_SALES_FACT_SEPTEMBER, _SALES_FACT, True),
|
||||
("stat_month_vs_created_at_range", "sqlite", "SELECT SUM(amount) FROM sales_fact WHERE stat_month = 202609",
|
||||
_SALES_FACT_SEPTEMBER, _SALES_FACT, True),
|
||||
("anti_join_primary_key_vs_not_null_column", "sqlite",
|
||||
"SELECT c.cid, c.cname FROM customers c LEFT JOIN invoices i ON i.cid = c.cid WHERE i.iid IS NULL",
|
||||
"SELECT c.cid, c.cname FROM customers c LEFT JOIN invoices i ON i.cid = c.cid WHERE i.issued_at IS NULL",
|
||||
_INVOICES, True),
|
||||
("anti_join_join_column_vs_primary_key", "postgres",
|
||||
"SELECT c.cid FROM customers c LEFT JOIN invoices i ON i.cid = c.cid WHERE i.cid IS NULL",
|
||||
"SELECT c.cid FROM customers c LEFT JOIN invoices i ON i.cid = c.cid WHERE i.iid IS NULL", _INVOICES, True),
|
||||
("max_vs_min_through_the_join_key_top_n", "mysql",
|
||||
"SELECT i.cid, MAX(c.cname) AS name, SUM(i.amount) AS total FROM invoices i JOIN customers c ON c.cid = i.cid "
|
||||
"WHERE i.state = 'paid' GROUP BY i.cid ORDER BY total DESC, i.cid LIMIT 5",
|
||||
"SELECT i.cid, MIN(c.cname) AS name, SUM(i.amount) AS total FROM invoices i JOIN customers c ON c.cid = i.cid "
|
||||
"WHERE i.state = 'paid' GROUP BY i.cid ORDER BY total DESC, i.cid LIMIT 5", _INVOICES, False),
|
||||
("max_vs_min_of_a_column_the_group_key_determines", "sqlite",
|
||||
"SELECT c.cid, MAX(c.region) AS region, COUNT(i.iid) AS n FROM customers c LEFT JOIN invoices i ON i.cid = c.cid "
|
||||
"GROUP BY c.cid",
|
||||
"SELECT c.cid, MIN(c.region) AS region, COUNT(i.iid) AS n FROM customers c LEFT JOIN invoices i ON i.cid = c.cid "
|
||||
"GROUP BY c.cid", _INVOICES, False),
|
||||
]
|
||||
|
||||
|
||||
def build_coupon_database() -> sqlite3.Connection:
|
||||
"""CONSTRAINT_TRAPS_R4 的小库(SQLite,内存):TEXT 主键里同时有 '1' 和 '01'、'paid' 和 'Paid',部分唯一索引下同一个
|
||||
code 有一行已删、一行未删。"""
|
||||
@@ -3028,11 +3228,19 @@ PAIRS: list[Pair] = [Pair(f"same:{i}", a, b, "same") for i, a, b in _SAME] + \
|
||||
[Pair(f"hard:{i}", a, b, "same") for i, a, b in _SAME_HARD_R3] + \
|
||||
[Pair(f"diff:{i}", a, b, "diff") for i, a, b in _DIFF_R4] + \
|
||||
[Pair(f"hard:{i}", a, b, "same") for i, a, b in _SAME_HARD_R4] + \
|
||||
[Pair(f"hard:{i}", a, b, "same", needs_schema=True) for i, a, b in _SAME_SCHEMA_R4]
|
||||
[Pair(f"hard:{i}", a, b, "same", needs_schema=True) for i, a, b in _SAME_SCHEMA_R4] + \
|
||||
[Pair(f"diff:{i}", a, b, "diff") for i, a, b in _DIFF_R5] + \
|
||||
[Pair(f"diff:{i}", a, b, "diff") for i, a, b, _, _ in _DIFF_TIME_R5] + \
|
||||
[Pair(f"hard:{i}", a, b, "same") for i, a, b in _SAME_HARD_R5] + \
|
||||
[Pair(f"hard:{i}", a, b, "same", needs_schema=True) for i, a, b in _SAME_SCHEMA_R5]
|
||||
|
||||
#: 第 4 轮复验报的、在 SQLite 上执行的那几对的编号(测试里按类别引用)
|
||||
ROUND_FOUR_DIFF = tuple(f"diff:{i}" for i, _, _ in _DIFF_R4)
|
||||
ROUND_FOUR_SAME = tuple(f"hard:{i}" for i, _, _ in _SAME_HARD_R4 + _SAME_SCHEMA_R4)
|
||||
#: 第 5 轮:真不同、只许给兜底的;时间窗口(只许说成 time 或者只给兜底)和各自的时间列;意思一样的
|
||||
ROUND_FIVE_DIFF = tuple(f"diff:{i}" for i, _, _ in _DIFF_R5)
|
||||
ROUND_FIVE_TIME = {f"diff:{i}": (ta, tb) for i, _, _, ta, tb in _DIFF_TIME_R5}
|
||||
ROUND_FIVE_SAME = tuple(f"hard:{i}" for i, _, _ in _SAME_HARD_R5 + _SAME_SCHEMA_R5)
|
||||
|
||||
#: 按编号找语料里的一对(测试里引用第 2 轮的反例用)
|
||||
PAIRS_BY_ID: dict[str, Pair] = {p.id: p for p in PAIRS}
|
||||
|
||||
@@ -20,6 +20,11 @@ schema_cache)两种情况下各判一次,两种都要守住上面的规矩
|
||||
MySQL 字符串里的反斜杠转义、数字写法)一律不判等价、只给兜底;靠约束改写的陷阱(TEXT 主键对数值列、COLLATE、部分唯一
|
||||
索引、MySQL 的隐式转换)在现造的小库上执行、一律不判等价;在 SQLite 上执行的那几对再拿 PostgreSQL / MySQL 的读法
|
||||
各判一遍,意思一样的只许给兜底。
|
||||
|
||||
第 5 轮的反例同样分两种:PostgreSQL 写成符号的 ~ / ~* / ~~ / ~~* / @@ 和 || 混用、前缀的 @、|/、COLLATE 后面跟 ::,
|
||||
不知道是哪种库时套着的块注释、紧跟着字的 --、双引号括起来的名字,一律不判等价、只给兜底;具体说法说错了的(条件写在聚合里被说成「另一边
|
||||
没有这个条件」、时间窗口被说成有效行过滤或者「没有按时间筛选」、反连接被说成有效行过滤、按主键定下来的列说 MAX 对
|
||||
MIN、被规范化去掉的 LEFT JOIN 说成「另一边没有」)在 SQLite 上执行,再拿三种方言各判一遍。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -228,3 +233,133 @@ def test_sqlite_sql_the_parser_reads_wrongly_gets_only_the_fallback(databases, p
|
||||
_check_shape(diff, pair.a, pair.b)
|
||||
assert not diff.equivalent and _only_fallback(diff), (pid, diff.items)
|
||||
assert diff.text_diff == (f"- {pair.a}", f"+ {pair.b}")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 第 5 轮复验的反例(收尾第 1 轮之后)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _raw_line_diff(diff, x: str, y: str) -> bool:
|
||||
"""text_diff 是原文的逐行对照:「- 」开头的是 x 的原文行,「+ 」开头的是 y 的原文行。"""
|
||||
lines_x = {line.rstrip() for line in x.strip().splitlines()}
|
||||
lines_y = {line.rstrip() for line in y.strip().splitlines()}
|
||||
return bool(diff.text_diff) and all(
|
||||
(line[2:] in lines_x) if line.startswith("- ") else (line.startswith("+ ") and line[2:] in lines_y)
|
||||
for line in diff.text_diff)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("name,dialect,a,b", sql_pairs.REGRESSIONS_R5,
|
||||
ids=[f"{n}-{d or 'unknown'}" for n, d, _, _ in sql_pairs.REGRESSIONS_R5])
|
||||
def test_round_five_parser_misreads_are_never_equivalent_and_get_only_the_fallback(name, dialect, a, b):
|
||||
"""第 5 轮 blocker(PostgreSQL):~、~*、~~、~~*、@@ 这些写成符号的匹配运算符和 || 同级、从左往右结合,sqlglot 让 ||
|
||||
先结合,status ~ 'p' || 'x' 和 status ~ ('p' || 'x') 读成了同一棵树(真实的 PostgreSQL 上一个是 'truex'、一个是
|
||||
false);自己差分压测又找到前缀的 @(取绝对值,sqlglot 读成参数)、|/(双精度的平方根,sqlglot 写成 SQRT)、
|
||||
COLLATE 后面跟 ::。不知道是哪种库时,块注释里再写 /*、紧跟着字的 --、双引号括起来的名字(MySQL 默认的 sql_mode 里
|
||||
是字符串)各库读法不同。原文里有这些写法就不在语法树上判等价、也不给具体说法,只给兜底,附原文的逐行对照。"""
|
||||
for schema in (None, SCHEMA):
|
||||
for x, y in ((a, b), (b, a)):
|
||||
diff = explain_sql_diff(x, y, dialect or None, schema=schema)
|
||||
_check_shape(diff, x, y)
|
||||
assert not diff.equivalent, f"{name} 不该判等价(schema={bool(schema)})"
|
||||
assert _only_fallback(diff), (name, diff.items)
|
||||
assert _raw_line_diff(diff, x, y), (name, diff.text_diff)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("pid", [p for p in sql_pairs.ROUND_FIVE_DIFF
|
||||
if p != "diff:r5-dropped-left-join-vs-inner-join"])
|
||||
@pytest.mark.parametrize("dialect", ("sqlite", "postgres", "mysql"))
|
||||
def test_round_five_differences_are_not_explained_with_the_wrong_difference(pid, dialect):
|
||||
"""第 5 轮 major:两条确实不同,以前却说错了不同在哪——条件、时间窗口写在聚合的参数里(SUM(status = 'paid')、
|
||||
SUM(amount * (1 - is_deleted))、SUM(paid_at >= …))被说成「这一边没有这个条件」「没有按时间筛选」;反连接(LEFT
|
||||
JOIN … WHERE 可空一侧 IS NULL)被说成有效行过滤。这几处现在只给兜底。"""
|
||||
pair = sql_pairs.PAIRS_BY_ID[pid]
|
||||
for schema in (None, SCHEMA):
|
||||
for x, y in ((pair.a, pair.b), (pair.b, pair.a)):
|
||||
diff = explain_sql_diff(x, y, dialect, schema=schema)
|
||||
_check_shape(diff, x, y)
|
||||
assert not diff.equivalent and _only_fallback(diff), (pid, dialect, bool(schema), diff.items)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("dialect", ("sqlite", "postgres", "mysql"))
|
||||
def test_a_left_join_the_normalization_dropped_is_not_called_missing(dialect):
|
||||
"""第 5 轮 minor:COUNT(DISTINCT …) 不怕重复,规范化去掉了没用到的 LEFT JOIN refunds;另一边内连接 refunds 时,以前
|
||||
说「B 关联了 refunds 表,A 没有」,可 A 的原文明明写了 LEFT JOIN refunds。不许说「没有」(PostgreSQL、MySQL 没给表
|
||||
结构时不去掉这次 LEFT JOIN,「A 用 LEFT JOIN 关联 refunds 表,B 用 INNER JOIN」和原文对得上,可以说)。"""
|
||||
pair = sql_pairs.PAIRS_BY_ID["diff:r5-dropped-left-join-vs-inner-join"]
|
||||
for schema in (None, SCHEMA):
|
||||
for x, y in ((pair.a, pair.b), (pair.b, pair.a)):
|
||||
diff = explain_sql_diff(x, y, dialect, schema=schema)
|
||||
_check_shape(diff, x, y)
|
||||
assert not diff.equivalent
|
||||
assert not any(d.aspect in ("join", "tables") and "没有" in d.text for d in diff.items), diff.items
|
||||
|
||||
|
||||
@pytest.mark.parametrize("pid", sorted(sql_pairs.ROUND_FIVE_TIME))
|
||||
@pytest.mark.parametrize("dialect", ("sqlite", "postgres", "mysql"))
|
||||
def test_round_five_time_windows_are_never_called_a_filter(pid, dialect):
|
||||
"""第 5 轮 major:时间列上的日期范围是统计窗口,不是有效行过滤。以前在结果看行先后的一段里(按日期表达式分组、取前
|
||||
N 行)BETWEEN 不拆,被说成两条「只保留 … 的行,另一边没有这个条件」,fallback 为假、正好够写进数据目录;LIKE
|
||||
'2026-09%' 取月也被说成有效行过滤。现在不许说成 filter;说成 time 时两边的时间列得对(没有按时间筛选的一边是空的)。
|
||||
SQLite 上一定说成 time。"""
|
||||
pair = sql_pairs.PAIRS_BY_ID[pid]
|
||||
cols_a, cols_b = sql_pairs.ROUND_FIVE_TIME[pid]
|
||||
for schema in (None, SCHEMA):
|
||||
for (x, y), (ca, cb) in (((pair.a, pair.b), (cols_a, cols_b)), ((pair.b, pair.a), (cols_b, cols_a))):
|
||||
diff = explain_sql_diff(x, y, dialect, schema=schema)
|
||||
_check_shape(diff, x, y)
|
||||
assert not diff.equivalent
|
||||
assert not any(d.aspect == "filter" for d in diff.items), (pid, dialect, diff.items)
|
||||
times = [d for d in diff.items if d.aspect == "time"]
|
||||
for item in times:
|
||||
assert [c["column"] for c in item.detail["a"]] == ca and \
|
||||
[c["column"] for c in item.detail["b"]] == cb, (pid, dialect, item)
|
||||
if dialect == "sqlite":
|
||||
assert len(times) == 1 and not diff.fallback, (pid, diff.items)
|
||||
|
||||
|
||||
def _round_five_same() -> list[tuple[str, str, str, bool]]:
|
||||
return [(p.id, p.a, p.b, p.needs_schema) for p in (sql_pairs.PAIRS_BY_ID[i] for i in sql_pairs.ROUND_FIVE_SAME)]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("pid,a,b,needs_schema", _round_five_same(), ids=[p[0] for p in _round_five_same()])
|
||||
@pytest.mark.parametrize("dialect", ("sqlite", "postgres", "mysql"))
|
||||
def test_round_five_equivalent_rewrites_get_at_most_the_fallback_in_every_dialect(pid, a, b, needs_schema, dialect):
|
||||
"""第 5 轮 major:意思一样,以前却说出了具体的不同——反连接里被 u.id IS NULL 蕴含的 u.name IS NULL、两种反连接的
|
||||
写法(查主键为空、查关联键为空)、按主键分组时 MAX 对 MIN、AVG(DISTINCT) 对 AVG(同一组里那张表的列只有一个值,
|
||||
LEFT JOIN 把行复制了也一样)。认得出就判等价,认不出只许给兜底;要靠主键的,没给表结构时说得出具体的不同不算错。"""
|
||||
for schema in (None, SCHEMA):
|
||||
if needs_schema and schema is None:
|
||||
continue
|
||||
for x, y in ((a, b), (b, a)):
|
||||
diff = explain_sql_diff(x, y, dialect, schema=schema)
|
||||
_check_shape(diff, x, y)
|
||||
assert diff.equivalent or _only_fallback(diff), (pid, dialect, bool(schema), diff.items)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("dialect", ("sqlite", "postgres", "mysql"))
|
||||
def test_anti_joins_are_never_called_a_filter(dialect):
|
||||
"""第 5 轮 major:LEFT JOIN … WHERE r.id IS NULL 和 … WHERE r.order_id IS NULL 是反连接最常见的两种写法。关联键为空
|
||||
就是没关联上,蕴含 r.id 为空;r.id 是主键时两者一样,没有主键时查 r.id 的一边留下的行更多(关联上了、r.id 却为空的)。
|
||||
以前说「A 只保留 refunds.id IS NULL 的行,B 没有这个条件」:把留得多的一边说成了严的一边,还当成有效行过滤。给不给
|
||||
表结构都不许判等价(要靠主键),也只许给兜底。"""
|
||||
pair = sql_pairs.PAIRS_BY_ID["hard:r5-anti-join-key-vs-join-column"]
|
||||
for schema in (None, SCHEMA):
|
||||
for x, y in ((pair.a, pair.b), (pair.b, pair.a)):
|
||||
diff = explain_sql_diff(x, y, dialect, schema=schema)
|
||||
_check_shape(diff, x, y)
|
||||
assert not diff.equivalent and _only_fallback(diff), (dialect, bool(schema), diff.items)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("name,dialect,a,b,schema,bare", sql_pairs.EXPLAIN_TRAPS_R5,
|
||||
ids=[f"{n}-{d or 'unknown'}" for n, d, *_ in sql_pairs.EXPLAIN_TRAPS_R5])
|
||||
def test_round_five_traps_that_need_their_own_schema_get_only_the_fallback(name, dialect, a, b, schema, bare):
|
||||
"""第 5 轮 major(要专门的表结构,不执行):数仓事实表按业务日期键(date_key BETWEEN 20260901 AND 20260930、
|
||||
stat_month = 202609)筛的一边,以前被说成「没有按时间筛选」、日期键的范围被说成有效行过滤;反连接查主键、查 NOT NULL
|
||||
的列、查关联键,以前被说成两条有效行过滤;经关联条件等于分组键的主键定下来的列,以前说「A 取最大值,B 取最小值」。
|
||||
这些都只给兜底(经主键推出来的那几对,没给表结构时这么说按已知信息成立,不查)。"""
|
||||
for given in ((schema, None) if bare else (schema,)):
|
||||
for x, y in ((a, b), (b, a)):
|
||||
diff = explain_sql_diff(x, y, dialect or None, schema=given)
|
||||
_check_shape(diff, x, y)
|
||||
assert not diff.equivalent and _only_fallback(diff), (name, dialect, bool(given), diff.items)
|
||||
|
||||
@@ -1439,6 +1439,14 @@ def test_differences_that_cannot_change_the_result_are_not_described(a, b):
|
||||
("SELECT COUNT(DISTINCT status) FROM orders WHERE id = 5", "SELECT COUNT(status) FROM orders WHERE id = 5"),
|
||||
("SELECT o.id FROM orders o LEFT JOIN users u ON u.id = o.user_id", "SELECT o.id FROM orders o"),
|
||||
("SELECT COUNT(*) FROM orders o JOIN users u ON u.id = o.user_id", "SELECT COUNT(*) FROM orders o"),
|
||||
# 第 5 轮:按主键分组(或者经关联条件等于主键),同一组里这张表的列只有一个值——LEFT JOIN 把行复制了也一样
|
||||
("SELECT u.id, MAX(u.city), COUNT(o.id) FROM users u LEFT JOIN orders o ON o.user_id = u.id GROUP BY u.id",
|
||||
"SELECT u.id, MIN(u.city), COUNT(o.id) FROM users u LEFT JOIN orders o ON o.user_id = u.id GROUP BY u.id"),
|
||||
("SELECT o.id, AVG(DISTINCT o.amount), COUNT(r.id) FROM orders o LEFT JOIN refunds r ON r.order_id = o.id "
|
||||
"GROUP BY o.id",
|
||||
"SELECT o.id, AVG(o.amount), COUNT(r.id) FROM orders o LEFT JOIN refunds r ON r.order_id = o.id GROUP BY o.id"),
|
||||
("SELECT o.user_id, MAX(u.name), SUM(o.amount) FROM orders o JOIN users u ON u.id = o.user_id GROUP BY o.user_id",
|
||||
"SELECT o.user_id, MIN(u.name), SUM(o.amount) FROM orders o JOIN users u ON u.id = o.user_id GROUP BY o.user_id"),
|
||||
], ids=lambda v: None)
|
||||
def test_constraints_only_make_the_explanation_say_less(a, b):
|
||||
"""表结构里的约束不拿来判等价,只拿来少说:主键不为空(id IS NOT NULL 恒为真)、主键定下来每组只有一行(SUM 和
|
||||
@@ -1453,3 +1461,161 @@ def test_constraints_only_make_the_explanation_say_less(a, b):
|
||||
diff = explain_sql_diff(x, y, "sqlite")
|
||||
check_shape(diff)
|
||||
assert any(d.aspect != "other" for d in diff.items), (x, y, diff.items)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 第 5 轮:PostgreSQL 写成符号的运算符、各库读法不同的注释;条件写在聚合里、时间窗口、反连接、经主键定下来的列
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("dialect,sql", [
|
||||
("postgres", "SELECT id, status ~ 'p' || 'x' FROM orders"),
|
||||
("postgres", "SELECT id, status ~* 'p' || 'x' FROM orders"),
|
||||
("postgres", "SELECT id, status ~~ 'p%' || 'x' FROM orders"),
|
||||
("postgres", "SELECT id, status ~~* 'P%' || 'x' FROM orders"),
|
||||
("postgres", "SELECT id, to_tsvector(status) @@ 'paid' || '' FROM orders"),
|
||||
("postgres", "SELECT id, tags @> 'a' || 'b' FROM orders"),
|
||||
("postgres", "SELECT id, @ amount - 20 FROM orders"),
|
||||
("postgres", "SELECT id, |/ amount FROM orders"),
|
||||
("postgres", "SELECT id, status COLLATE \"C\"::text FROM orders"),
|
||||
(None, "SELECT id /* a /* b */ + 1 -- */\nFROM orders"),
|
||||
(None, "SELECT id, amount--1\nFROM orders"),
|
||||
(None, "SELECT COUNT(*) FROM orders WHERE \"status\" = 'paid'"),
|
||||
], ids=["regex-concat", "iregex-concat", "like-op-concat", "ilike-op-concat", "text-search-concat", "contains-concat",
|
||||
"absolute-value", "square-root", "collate-cast", "unknown-nested-comment", "unknown-dashes",
|
||||
"unknown-double-quotes"])
|
||||
def test_operators_postgres_reads_differently_from_sqlglot_are_held_back(dialect, sql):
|
||||
"""PostgreSQL 里 ~、~*、~~、~~*、@@、@> 这些写成符号的运算符和 ||、&、# 同级(「别的运算符」)、从左往右结合,sqlglot
|
||||
把它们当成 LIKE、IN 那一级、比 || 松:status ~ 'p' || 'x' 在 PostgreSQL 里是 (status ~ 'p') || 'x'。前缀的 @ 是取
|
||||
绝对值(sqlglot 读成参数),|/ 是双精度的平方根(sqlglot 写成 SQRT),x COLLATE "C"::text 的 :: 套在整个 COLLATE 上。
|
||||
不知道是哪种库时,块注释里的 /*、紧跟着字的 --、双引号括起来的名字(MySQL 默认的 sql_mode 里是字符串)各库读法不同。
|
||||
这些都不在语法树上判等价。"""
|
||||
from app.data.sql_semantics import _lost_in_parsing, _parse, _read
|
||||
|
||||
read = _read(dialect)
|
||||
tree, error = _parse(sql, read)
|
||||
assert error is None and _lost_in_parsing(sql, read, tree)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("dialect,sql", [
|
||||
("postgres", "SELECT id FROM orders WHERE status LIKE 'p' || '%'"),
|
||||
("postgres", "SELECT id FROM orders WHERE status ILIKE 'p' || '%' AND amount > 1"),
|
||||
("postgres", "SELECT id FROM orders WHERE status ~ ('p' || 'x')"),
|
||||
("postgres", "SELECT id FROM orders WHERE status ~ '^p' AND amount + 1 > 2"),
|
||||
("postgres", "SELECT id, status::text || 'x' FROM orders WHERE status COLLATE \"C\" > 'a'"),
|
||||
("sqlite", "SELECT id /* a /* b */ + 1 -- */\nFROM orders"),
|
||||
("postgres", "SELECT id /* a /* b */ + 1 -- */\nFROM orders"),
|
||||
("mysql", "SELECT id /* a /* b */ + 1 -- */\nFROM orders"),
|
||||
("sqlite", "SELECT id, amount--1\nFROM orders"),
|
||||
("postgres", "SELECT id, amount--1\nFROM orders"),
|
||||
("postgres", "SELECT COUNT(*) FROM orders WHERE \"status\" = 'paid'"),
|
||||
("mysql", "SELECT COUNT(*) FROM orders WHERE status = \"paid\""),
|
||||
], ids=["pg-like-concat", "pg-ilike-concat", "pg-regex-parenthesized", "pg-regex-own-segment", "pg-cast-collate-apart",
|
||||
"sqlite-comment", "pg-comment", "mysql-comment", "sqlite-dashes", "pg-dashes", "pg-double-quotes",
|
||||
"mysql-double-quotes"])
|
||||
def test_writings_each_engine_reads_like_sqlglot_are_not_held_back(dialect, sql):
|
||||
"""写成关键字的 LIKE / ILIKE 本来就比 || 松(PostgreSQL 和 sqlglot 读得一样);加了括号、AND 隔开的不算混用;知道是
|
||||
SQLite、PostgreSQL、MySQL 时,注释怎么嵌套、-- 后面要不要空白、双引号括起来的是名字还是字符串,sqlglot 和库读得一样:
|
||||
照常规范化,加个空格照样认得出等价。"""
|
||||
from app.data.sql_semantics import _lost_in_parsing, _parse, _read
|
||||
|
||||
read = _read(dialect)
|
||||
tree, error = _parse(sql, read)
|
||||
assert error is None and not _lost_in_parsing(sql, read, tree)
|
||||
assert explain_sql_diff(sql, sql.replace(" FROM", " FROM"), dialect).equivalent
|
||||
|
||||
|
||||
@pytest.mark.parametrize("pid,expected", [
|
||||
("diff:r5-between-on-two-time-columns-grouped-by-date", "A 按 paid_at 筛时间,B 按 created_at"),
|
||||
("diff:r5-between-on-two-time-columns-top-n", "A 按 paid_at 筛时间,B 按 created_at"),
|
||||
("diff:r5-between-window-on-one-side-grouped-by-date", "A 没有按时间筛选,B 按 paid_at 筛时间"),
|
||||
("diff:r5-like-month-vs-no-window", "A 没有按时间筛选,B 按 paid_at 筛时间"),
|
||||
("diff:r5-range-vs-like-month-on-another-column",
|
||||
"A 按 paid_at 筛时间,B 按 created_at(A 是 paid_at >= '2026-09-01' AND paid_at < '2026-10-01',"
|
||||
"B 是 created_at LIKE '2026-09%')"),
|
||||
])
|
||||
def test_time_windows_written_as_between_or_like_are_told_as_time_columns(pid, expected):
|
||||
"""时间列上的 BETWEEN、LIKE '2026-09%'(SQLite 里日期存成文字,取月常这么写)是统计窗口:说成按哪一列筛时间,不说成
|
||||
「只保留 … 的行」(那是有效行过滤,会写进数据目录)。结果看行先后的一段(按日期表达式分组、取前 N 行)里 BETWEEN
|
||||
不拆成两个比较,照样认得出。"""
|
||||
pair = sql_pairs.PAIRS_BY_ID[pid]
|
||||
for schema in (None, sql_pairs.schema_cache()):
|
||||
diff = explain_sql_diff(pair.a, pair.b, "sqlite", schema=schema)
|
||||
check_shape(diff)
|
||||
assert [(d.aspect, d.text) for d in diff.items] == [("time", expected)], diff.items
|
||||
|
||||
|
||||
@pytest.mark.parametrize("a,b,expected", [
|
||||
# 聚合的参数就是那一列(不是写在参数里的条件):另一边确实没有按 status 筛
|
||||
("SELECT region, SUM(amount) FROM orders GROUP BY region",
|
||||
"SELECT region, SUM(amount) FROM orders WHERE status = 'paid' GROUP BY region",
|
||||
[("filter", "B 只保留 status = 'paid' 的行,A 没有这个条件")]),
|
||||
("SELECT SUM(amount) FROM orders", "SELECT SUM(amount) FROM orders WHERE amount > 0",
|
||||
[("filter", "B 只保留 amount > 0 的行,A 没有这个条件")]),
|
||||
# 只是选出来给人看的标记(不聚合,行都在结果里):另一边确实把别的行也列出来了
|
||||
("SELECT id, status = 'paid' AS is_paid FROM orders",
|
||||
"SELECT id, status = 'paid' AS is_paid FROM orders WHERE status = 'paid'",
|
||||
[("filter", "B 只保留 status = 'paid' 的行,A 没有这个条件")]),
|
||||
# 判空、按状态筛这类和日期无关的条件照说;时间列上的判空是有效行过滤,不是时间窗口
|
||||
("SELECT COUNT(*) FROM orders", "SELECT COUNT(*) FROM orders WHERE paid_at IS NOT NULL",
|
||||
[("filter", "B 排除了 paid_at 为空的行,A 没有")]),
|
||||
("SELECT COUNT(*) FROM orders", "SELECT COUNT(*) FROM orders WHERE grade = 3",
|
||||
[("filter", "B 只保留 grade = 3 的行,A 没有这个条件")]),
|
||||
], ids=lambda v: None)
|
||||
def test_filters_the_other_side_really_does_not_have_are_still_told(a, b, expected):
|
||||
"""收窄之后的对照:另一边确实没有这个条件时照说。"""
|
||||
diff = explain_sql_diff(a, b, "sqlite")
|
||||
check_shape(diff)
|
||||
assert [(d.aspect, d.text) for d in diff.items] == expected, diff.items
|
||||
|
||||
|
||||
@pytest.mark.parametrize("a,b", [
|
||||
# 条件、时间窗口写在聚合的参数里:这一边也只算满足条件的行
|
||||
("SELECT region, SUM(CASE WHEN status = 'paid' THEN 1 ELSE 0 END) FROM orders GROUP BY region",
|
||||
"SELECT region, COUNT(*) FROM orders WHERE status = 'paid' GROUP BY region"),
|
||||
("SELECT region, SUM(IIF(status = 'paid', amount, 0)) FROM orders GROUP BY region",
|
||||
"SELECT region, SUM(amount) FROM orders WHERE status = 'paid' GROUP BY region"),
|
||||
("SELECT region, AVG(status = 'paid') FROM orders GROUP BY region",
|
||||
"SELECT region, AVG(status = 'paid') FROM orders WHERE status = 'paid' GROUP BY region"),
|
||||
("SELECT region, SUM(amount * is_paid) FROM orders GROUP BY region",
|
||||
"SELECT region, SUM(amount) FROM orders WHERE is_paid = 1 GROUP BY region"),
|
||||
("SELECT SUM(amount) FILTER (WHERE created_at >= '2026-09-01') FROM orders",
|
||||
"SELECT SUM(amount) FROM orders WHERE paid_at >= '2026-09-01'"),
|
||||
# 分不清是不是日期的列上的窗口(业务日期键、统计月份、分区):既不说成有效行过滤,也不说另一边「没有按时间筛选」
|
||||
("SELECT SUM(amount) FROM sales WHERE stat_month = 202609", "SELECT SUM(amount) FROM sales"),
|
||||
("SELECT SUM(amount) FROM sales WHERE pt = '20260901'", "SELECT SUM(amount) FROM sales"),
|
||||
("SELECT SUM(amount) FROM sales WHERE biz_key BETWEEN 20260901 AND 20260930",
|
||||
"SELECT SUM(amount) FROM sales WHERE created_at >= '2026-09-01' AND created_at < '2026-10-01'"),
|
||||
("SELECT SUM(amount) FROM sales WHERE fiscal_year = 2026",
|
||||
"SELECT SUM(amount) FROM sales WHERE created_at >= '2026-01-01'"),
|
||||
# 另一边对别的列有范围(BETWEEN、上下界一对):可能是认不出的时间窗口,不说「没有按时间筛选」
|
||||
("SELECT SUM(amount) FROM sales WHERE day_no BETWEEN 20697 AND 20726",
|
||||
"SELECT SUM(amount) FROM sales WHERE created_at >= '2026-09-01' AND created_at < '2026-10-01'"),
|
||||
# 反连接:可空一侧的列为空说的是「没关联上」
|
||||
("SELECT COUNT(*) FROM orders o LEFT JOIN refunds r ON r.order_id = o.id WHERE r.state IS NULL",
|
||||
"SELECT COUNT(*) FROM orders o LEFT JOIN refunds r ON r.order_id = o.id"),
|
||||
("SELECT COUNT(*) FROM orders o LEFT JOIN refunds r ON r.order_id = o.id WHERE r.refunded_at IS NULL",
|
||||
"SELECT COUNT(*) FROM orders o LEFT JOIN refunds r ON r.order_id = o.id WHERE r.id IS NULL"),
|
||||
], ids=lambda v: None)
|
||||
@pytest.mark.parametrize("dialect", DIALECTS)
|
||||
def test_conditions_the_other_side_has_in_another_form_are_not_called_missing(a, b, dialect):
|
||||
"""第 5 轮:另一边其实有这个条件(写在聚合的参数里、写在 CASE / IIF / FILTER 里、乘上 0 / 1 的标记),或者这个条件
|
||||
可能是认不出的时间窗口,或者是反连接——说「另一边没有这个条件」「另一边没有按时间筛选」就说错了,只给兜底。"""
|
||||
for x, y in ((a, b), (b, a)):
|
||||
diff = explain_sql_diff(x, y, dialect)
|
||||
check_shape(diff)
|
||||
assert not diff.equivalent and diff.items == [FacetDiff("other", None, None, FALLBACK)], (x, y, diff.items)
|
||||
|
||||
|
||||
def test_time_columns_on_different_tables_are_written_with_their_tables():
|
||||
"""两边的时间列在不同的表上:光写「B 按 created_at」读起来像订单的下单时间,其实是用户的注册时间。写成「表.列」;
|
||||
说「另一边没有按时间筛选」的一边关联了好几张表时也写表名。detail 照旧给各自的表。"""
|
||||
a = "SELECT COUNT(*) FROM orders o JOIN users u ON u.id = o.user_id WHERE o.paid_at >= '2026-09-01'"
|
||||
b = "SELECT COUNT(*) FROM orders o JOIN users u ON u.id = o.user_id WHERE u.created_at >= '2026-09-01'"
|
||||
plain = "SELECT COUNT(*) FROM orders o JOIN users u ON u.id = o.user_id"
|
||||
[item] = explain_sql_diff(a, b, "sqlite", names={"users.created_at": "注册时间"}).items
|
||||
assert item.text == "A 按 orders.paid_at 筛时间,B 按注册时间 users.created_at"
|
||||
assert item.detail == {"a": [{"table": "orders", "column": "paid_at"}],
|
||||
"b": [{"table": "users", "column": "created_at"}]}
|
||||
[item] = explain_sql_diff(b, plain, "sqlite").items
|
||||
assert (item.aspect, item.text) == ("time", "A 按 users.created_at 筛时间,B 没有按时间筛选")
|
||||
|
||||
Reference in New Issue
Block a user