mirror of
https://github.com/ThearchyHelios/AgentLab.git
synced 2026-11-15 06:51:59 +00:00
6d4f452f36
复核发现上一版把 ON 单独排成一行后,「规范化后不同的行」丢了关联条件的上下文:只差一个关联条件时那一栏只剩 「AND refunds.state = 'done'」,有两个 LEFT JOIN 时看不出是哪个 JOIN 的,也看不出是关联条件还是筛选条件;两边 JOIN 的表不同时,两边都有的那一行「ON」被 difflib 当成相同的行吃掉,表名下面直接跟条件;关联子查询换成两张表时也一样。 这几种都只给兜底说明,审核人全靠这一栏逐行对照。 - sql_semantics:ON 挂在 JOIN 那一行的末尾(表名、子查询的右括号后面),条件各占一行缩在下面,仍比 WHERE 的条件 深一级(ON 里和 WHERE 里的同一个条件不当成同一行)。text_diff 里不同的行前面带上它的各级上级行(排在前面、缩进比 它浅的最近一行,一层层数到顶格),以「 」开头、两边相同,只为标出不同的行在哪个 WHERE、哪个 JOIN 的 ON、哪层 子查询里;截到 40 行时末尾落单的上级行不留。只动排版:拿测试里的 SQL 语料(990 对,4 种方言、有无表结构、两个 方向)逐对比过,equivalent、items、fallback、level 和改之前完全一致,有没有不同的行也一致,不再有单独一行的 ON。 - SqlDiff 的约定多一条:「 」开头的是上下文行,只跟着不同的行出现。test_sql_semantics 的 check_shape 原来只认 「- 」「+ 」,改成也认「 」、并要求最后一行是不同的行(374 个用例只因这一行断言变红,判定和说明都没变)。 - 审批卡(DualCheckApproval):两栏各自只放用得着的上级行(这一边不同的行往上数,缩进更浅的最近一行,一层层往上), 只有另一边用得着的不放;上级行不高亮、用淡一级的字色(同原文栏里没标的行),标题后面说明「淡色的行两边相同,标出 所在的子句」,悬停也是这句;这一边没有不同的行时照旧说没有,老工件没有上级行,照旧。lineDiff 注释里「条件并成 一行」的旧说法改正。 - 测试:test_sql_text_diff 里 ON 那条的期望改成带 JOIN 的上级行,补两个 LEFT JOIN 只差一个关联条件(两个方向)、 两边 JOIN 的表不同、关联子查询换成两张表、同一个条件在 ON 和 WHERE 里、顶格的行不带上下文、40 行截断几条;去掉 上级行、去掉「ON 挂在行末」、去掉截断后的清理,各自让对应的用例变红。 - 夹具 dual_check.json 按 test_dual_check_fixture 的办法重新生成(只变 text_diff),另加 join_condition 变体(只差 一个关联条件、只给兜底);check-verify 查两栏的上级行(WHERE)列在不同的行前面、不高亮、标题有说明,join_condition 下原流程一栏是带 ON 的 JOIN 加缩进的条件、盲算一栏不画只有原流程用得着的上级行,深浅色对比度、390px 不溢出。
8038 lines
453 KiB
Python
8038 lines
453 KiB
Python
"""SQL 语义:两条 SQL 算的是不是一回事,不是的话差在哪。
|
||
|
||
双盲复算要回答「盲算和原流程的 SQL 差在哪」(A 按支付时间筛、B 按下单时间筛;A 排除了作废的行、B 没有),人据此选
|
||
A / B,只差一个方面、又对得上数据目录里的事实(时间归属、有效行过滤、去重)时选择会写进数据目录;运行对比要回答
|
||
「这一期和上一期的 SQL 是不是同一个模板」(定时运行每次的日期不同,不能算模板变了);反方审稿要看自己的查询和原
|
||
查询的口径是否一致。几处共用这里:
|
||
|
||
- normalize_sql:字面量换成占位符之后的规范文本,加上字面量本身(按规范文本里的顺序)。两条 SQL 只有字面量不同,
|
||
规范文本就相同;关键字大小写、空白、普通注释、不加引号的名字的大小写都不算不同;结构变了的一定不同——解析时会
|
||
丢掉写法的(MySQL 会执行的 /*! … */ 注释、写成 ! 的 NOT、SQLite 的 MOD() 这类重新生成时换了名字的函数)改按原文
|
||
逐词规范化,会执行的注释里的代码也算结构。
|
||
- explain_sql_diff:两条 SQL 是不是等价(SqlDiff.equivalent),不是的话差在哪(items、text_diff)。
|
||
- diff_facets:照旧收两份 facets,交给 explain_sql_diff 比两份记下的原文,返回它的 items(含兜底条目):完全一致
|
||
(规范形式一字不差)才返回空列表。
|
||
- facets:把一条查询拆成表、连接、WHERE 合取式、时间条件、分组、聚合、算出来的列、HAVING、取哪几行(给调用方取
|
||
结构用;CTE、子查询按位置编号,见 _Statement)。facets 不受下面等价判定的改动影响。
|
||
|
||
**等价怎么判。** 过去几轮按方面归纳任意 SQL,每轮修好一批又冒出一批(还回退过),收不拢;现在先判等价、再说差在哪。
|
||
两边各自规范化,规范化的每一步都只做保持语义的变换,同一级的两个规范形式一字不差(比语法树逐个节点的类型和参数,
|
||
见 _fingerprint,括号不算;不比生成的 SQL:sqlglot 按方言生成时会把不同的写法写成一样,SQLite 的 CAST(x AS DATE)
|
||
生成 DATE(x))才算等价,否则一律不等价、条目一定不为空。原文一字不差的两条直接算等价。规范形式分三级(见
|
||
_canonical_forms),哪一级对上都算:
|
||
|
||
- optimizer:先按方言统一名字(_normalize_names),CTE、子查询的列名表写成里面那段的列别名,USING 写成 ON,FETCH
|
||
FIRST 写成 LIMIT,IS NOT NULL、NOT LIKE 这类带 negate 的写法统一成 NOT (…);然后 sqlglot 的 qualify 认列(给了
|
||
数据源的 schema_cache 就按它认、展开 SELECT *);再把整条语句的来源别名改成互不相同的,引用了好几次、只筛行选列的
|
||
CTE 每处写回一份,只引用一次的 CTE 写回用到它的地方,只从几张表筛行选列的子查询并进外层(_merge_derived:只并
|
||
内连接一侧的),外层 WHERE 里只涉及分组子查询选出来的列的条件挪进它的 HAVING(同一个值),去掉没用的 CTE。
|
||
- light:不认列,只做上面不需要认列的几步。
|
||
- parsed:解析后只统一名字的大小写、引号。
|
||
|
||
两级共用的几步(见 _finish,各函数的说明写了为什么保持语义):原样透出子查询全部列的一段换成那个子查询;两个来源
|
||
的 RIGHT JOIN 写成 LEFT JOIN;WHERE 把外连接可空一侧没关联上的行筛掉了的写成内连接(_rejects 分「一定是空值」和
|
||
「一定不成立」:NOT 只能套前者,NOT IN 子查询、NOT IN ()、x op ALL (…)、NOT (x = ANY …)、PostgreSQL 的 || 都留得下
|
||
没关联上的行);不分组、结果只是跳过空值的聚合、参数在可空一侧为空时一定为空的,LEFT JOIN 写成内连接
|
||
(_null_blind_left_joins:COALESCE(SUM(r.amount), 0) 和 INNER JOIN 一样);ON 之外没用到、结果又不怕重复的行(去重、
|
||
只有 MIN / MAX / 去重的聚合、IN / EXISTS 子查询)的 LEFT JOIN 去掉;开头那组内连接的条件挪进 WHERE;WHERE、HAVING、
|
||
ON 里的 TRUE、1 = 1、分组后的 COUNT(*) >= 1 去掉(没有 GROUP BY、别处也没有聚合的 HAVING TRUE 留着:PostgreSQL
|
||
里它让语句只出一行);同一组 AND 里另一项已经保证不为空的 x IS NOT NULL 去掉;给了表结构、两列声明的类型一样又「相等
|
||
就是一样」时,WHERE 里两列相等的,SELECT / GROUP BY / HAVING / ORDER BY 里都换成同一列(_substitute_equal_columns);
|
||
EXISTS 子查询写成 SELECT 1(去重、分组、排序、取前 n 行、UNION 还是 UNION ALL 都不影响);IN 子查询里不影响取值集合
|
||
的 DISTINCT、GROUP BY、ORDER BY 去掉、UNION 写成 UNION ALL(只在那一列「相等就是一样」时);SELECT DISTINCT 写成按
|
||
选出来的列 GROUP BY、已经按选出来的列分组的 DISTINCT 去掉;COUNT(1) 写成 COUNT(*)、MIN / MAX 里的 DISTINCT 去掉;
|
||
SQLite 的 IIF、MySQL 的 IF 写成 CASE,CASE 的 ELSE NULL 去掉;IN 列表去重排序、只有一个常量的写成 =;BETWEEN 写成
|
||
两个比较(SYMMETRIC、MySQL 里上下界类型不同或不是常量的不改);NOT 往里推:套比较写成反过来的比较(ANY / ALL 不动)、
|
||
NOT NOT 条件去掉、德摩根律(三值逻辑下照样成立);LIMIT / OFFSET 的整数算式算成一个数(SQLite、PostgreSQL),SQLite
|
||
的负数 LIMIT、PostgreSQL 的 LIMIT NULL 写成不限行数;ORDER BY 的 ASC、OFFSET 0、和前面一模一样的排序键、分组键去掉;
|
||
最外层 ORDER BY 引用的别名换成表达式,最外层和条件里的子查询选出来的列名去掉;最后起规范的名字、交换律成立的地方
|
||
排好序。
|
||
|
||
**不靠表结构里的约束改写。** 主键、唯一约束、外键、NOT NULL 一律不拿来改写、判等价:凭键去掉没用到的 LEFT JOIN、
|
||
去掉 DISTINCT,把 COUNT(x) 当成 COUNT(*),凭主键认分组的函数依赖,凭 NOT NULL 去掉 IS NOT NULL,都不做。第 4 轮复验
|
||
在这些地方抓到判错的等价:SQLite 的类型亲和(TEXT 主键对 INTEGER 列,'1' 和 '01' 都等于 1,一行关联上两行)、比较
|
||
写了 COLLATE NOCASE、部分唯一索引(只管没删的行,探查照样记成 unique)、MySQL 拿字符串和数值比时的隐式转换,都会让
|
||
「唯一」不成立,表结构里看不全,宁可认不出。表结构只拿来认列、展开 SELECT *、看列声明的类型(「相等就是一样」、是不是
|
||
精确数),约束只拿来让差异说明少说(见下面的具体说法)。
|
||
|
||
**保真守卫:原文里有 sqlglot 会读错、会丢掉的写法时,不在语法树上判等价(连只统一名字的那一级都不比),也不给具体
|
||
说法,只给兜底说明,附原文的逐行对照**(见 _lost_in_parsing、_misread)。按方言切出来的词逐个看、按括号深度分层,
|
||
不拿正则硬匹配:
|
||
|
||
- 读错的:MySQL / MariaDB(以及不知道是哪种库时)字符串里的反斜杠转义(sqlglot 不认 \\Z、\\0,原样留着反斜杠,和
|
||
写成 \\\\Z 的撞成同一个值);数字后面紧跟着字母、下划线(1_000 是一千、0o17 是 15,sqlglot 切成数字加别名);两个字符串
|
||
常量挨着写(MySQL 里是拼接,SQLite 里是别名);同一层括号、同一段条件里比较运算和 IS / IN / LIKE / BETWEEN / 另一个
|
||
比较运算没加括号地混用(a = b IS NULL、a < 100 IN (0, 1)、a = 100 < 200:各方言的结合顺序和 sqlglot 的不一样);
|
||
按位运算(& | ^ << >> ~,PostgreSQL 的 # 这类)和算术、拼接、另一种按位运算没加括号地混用;写成符号、sqlglot 却当成
|
||
LIKE 那一级来读的运算符(PostgreSQL 的 ~、~*、~~、~~*、@@、-|-……)和别的二元运算混用(PostgreSQL 里它们和 ||、&
|
||
同级、从左往右:status ~ 'p' || 'x' 是 (status ~ 'p') || 'x',sqlglot 读成 status ~ ('p' || 'x');写成关键字的
|
||
LIKE / ILIKE 本来就比 || 松,不算);PostgreSQL 的前缀 @(取绝对值,sqlglot 读成参数)、|/ 和 ||/(双精度的开方,
|
||
sqlglot 写成 numeric 的 SQRT)、COLLATE 后面同一段里的 ::;XOR;不知道是哪种库时的 ||、#、块注释里套着的 /*(PostgreSQL
|
||
嵌套、SQLite / MySQL 不嵌套)、后面紧跟着字的 --(MySQL 里是两个减号)、双引号括起来的名字(MySQL 默认的 sql_mode 里
|
||
是字符串),MySQL 以外的 &&。第 5 轮在真实的
|
||
PostgreSQL 18.6、MySQL 8.0、SQLite 上做过差分压测(不加括号的随机运算符混用对 sqlglot 读出来的树每个操作数都加
|
||
括号,各几千条):读错了的都在这里拦下。
|
||
- 丢掉的:一元加号(SQLite 里 +a 去掉了类型亲和)、0x10 和 x'10' 这类只留下值的字面量、换了名字的类型名、MySQL 会执行
|
||
的注释(/*! … */)、写成 ! 的 NOT;解析时换了写法又没留位置的函数名(见 _mark_surfaces:SQLite 的 MOD() 和 % 解析成
|
||
同一个节点,记下原文写的名字就不会判成等价)。MySQL 里后面不跟空白的 -- 是两个减号,先拆开再解析。
|
||
|
||
**不走规范化两级的(只比解析后的原样)**:不确定的函数(RANDOM():两次求值不是同一个值,去重 AND / OR / IN 列表里
|
||
的项、并子查询都会改变求值次数);递归 CTE(SQLite 不写 RECURSIVE 的也是)、LATERAL、表函数、VALUES、实表别名带
|
||
列名、ROLLUP 这些结构;子查询里写了表名的列、而这个别名在里外层各指一个不同来源、认不准是哪一个的(SQLite 里层找
|
||
不到这一列就往外层找)。optimizer 一级另外不出的:认不全列;WHERE / GROUP BY / HAVING 里、或者套着的子查询里有和
|
||
列别名同名、又可能是表的列的名字(SQLite 的 WHERE、ON、相关子查询都认列别名);FROM 里的子查询、CTE 选出来的列有
|
||
同名的。
|
||
|
||
**故意不做的改写**(复验在这些地方抓到过不保持语义的例子):sqlglot 的 simplify(数字常量按 Decimal 折叠,SQLite
|
||
按浮点数算 0.1 + 0.2 ≠ 0.3;-(-x) 化成 x;BETWEEN SYMMETRIC 丢了 SYMMETRIC)、unnest_subqueries、canonicalize、
|
||
pushdown_projections;HAVING 里的条件挪进 WHERE;只有一项、又不是常量的 IN 列表写成 =(SQLite 里 a IN (b) 是
|
||
a = +b);改列别名、把 EXISTS 写成 SELECT 1、去掉列名之前,本段(含 WHERE、ON、相关子查询)还按名字引用这些别名的
|
||
(见 _alias_refs);USING 的列和某个列别名同名的(ORDER BY 里单独一个名字先认列别名);靠约束的改写(见上)。
|
||
|
||
**结果看行出来的先后的一段(见 _Sensitivity)不调换来源的先后、不排 GROUP BY 的键、不改 DISTINCT、不并子查询、不把
|
||
引用了好几次的 CTE 拆开,也不按 WHERE 把外连接改成内连接、不去掉条件(连一模一样写了两遍的也不去)、不去重排序键和
|
||
分组键、不换列、不把 BETWEEN 拆成两个比较、不去掉 MIN / MAX 里的 DISTINCT**:取行(LIMIT / OFFSET / FETCH,有并列时
|
||
取哪几行也看先后)、DISTINCT ON、窗口函数、拼串拼数组的聚合(GROUP_CONCAT)、聚合里既不分组也不聚合的列、当成标量用
|
||
的子查询,去重、分组、MIN / MAX、去重的 UNION 留下的值「相等却不一样」的时候(MySQL 默认不分大小写的字符串、
|
||
PostgreSQL numeric 的 2.0 和 2.00、算出来的表达式),以及 PostgreSQL、MySQL 里对不知道是不是精确数的值求和;给这样
|
||
一段供行的子查询、CTE 也算。第 3 轮压测在真实的库上看到:SQLite 里 WHERE 让 LEFT JOIN 降成内连接之后,扫描的先后和
|
||
直接写内连接的不一样(EXPLAIN QUERY PLAN 一个先扫 refunds、一个先扫 orders);MySQL 里同一个条件写两遍,按条件估算
|
||
的行数变了、换了关联的先后——GROUP_CONCAT 拼出来的先后都跟着变。
|
||
|
||
**假设(做不到的)**:SQLite 没给表结构时按列有声明的类型、默认的 BINARY 排序规则算——声明了 COLLATE NOCASE 的列、
|
||
没有类型亲和的列(能同时存 1 和 1.0)表结构里看不出来;比较两边都是列时对调两边(SQLite 比较两列用左边那列声明的
|
||
排序规则)。SQLite 的求和当成和先后无关。PostgreSQL、MySQL 没给表结构时一律不当成「相等就是一样」、也不当成精确数。
|
||
MySQL 的表名按 Linux 上分大小写算;MySQL 按默认的 sql_mode 读:双引号括起来的是字符串、|| 是 OR(开了 ANSI_QUOTES、
|
||
PIPES_AS_CONCAT 的库这两处的意思不同,sqlglot 的 MySQL 读法和这里都按默认的算)。
|
||
|
||
等价的结论另有执行比对守门(tests/test_sql_equivalence.py):虚构库上五组随机数据、一组极端数据、一组专挑写法差异的
|
||
数据,判等价的两条每组结果都得一样,结果不一样的绝不能判等价;复验报过的反例都收在语料里。
|
||
|
||
**具体说法(见 _Explainer)。** 只在两边的规范形式都是「简单形状」时说(_simple_block:整条语句就是一段 SELECT,没有
|
||
CTE、子查询、UNION 这类合并、窗口函数、DISTINCT ON;FROM 里只有实表,连接只有内连接、CROSS JOIN、逗号和 LEFT JOIN;
|
||
LIMIT 是至少为 1 的常数;没有认不准的名字、SELECT 列表里没有 sqlglot 不认识的函数;不认列的那一级里原文没有在
|
||
WHERE / GROUP BY 里用列别名的),而且只说四类:
|
||
|
||
- 时间列(time):两边各在一张底层表的一列上按时间筛、列不同(另一边哪儿都没用这一列筛选,两边的时间条件都不是恒为
|
||
真的,时间列不在 LEFT JOIN 的可空一侧);一边按时间筛了、另一边哪儿都没有可能是时间窗口的条件的(见下),也说成
|
||
time(「A 按 paid_at 筛时间,B 没有按时间筛选」)。时间条件是一列(可以套着日期函数、CAST)和常量比较、IN、BETWEEN,
|
||
列名像时间或者常量像日期;或者时间列 LIKE 一个像日期的模式(SQLite 里日期存成文字,paid_at LIKE '2026-09%' 是取月)。
|
||
两边的时间列在不同的表上时写成「表.列」。detail 给两边的 [{table, column}]。
|
||
「另一边可能按时间筛了」(_Usage.temporal)认得很宽:条件里(WHERE、ON、HAVING,聚合的参数、CASE、FILTER 里也算)
|
||
用到可能是时间的列(date_key、stat_month、pt、fiscal_year 这类,见 _timeish_name)、日期函数、像日期的常量
|
||
(20260901、202609 这类整数,'2026'、'2026Q3'、'2026-09%'),或者对同一个对象有上下界(BETWEEN、>= 和 < 一对)。
|
||
- 多一个筛选条件(filter):一边多出来、只涉及一张底层表的列和常量比的条件——另一边整条语句(WHERE、ON、HAVING、CASE /
|
||
FILTER 里,聚合的参数里套在别的运算、函数里的:SUM(status = 'paid')、COUNT(… OR NULL)、COUNT(NULLIF(…))、
|
||
SUM(amount * (1 - is_deleted));判空、或者条件只筛掉空值时连光一列的聚合参数)对这些列没有任何条件;这一边对这些列
|
||
的条件全是另一边没有的(不是被另一边也有的条件蕴含的);比的对象就是那一列(不是 LOWER(x) 这类认不出取值范围的);
|
||
按取值逐段算过不是恒为真的(_Range.always:表结构里确定不为空、又不在真的可能整行为空的外连接一侧的列不取空值);
|
||
不用到 LEFT JOIN 可空一侧的列(WHERE r.id IS NULL 是反连接,说的是「没关联上」;r.order_id IS NULL 和 ON 里的
|
||
r.order_id = o.id 一起已经蕴含 r.id IS NULL);不是可能的时间窗口(_window_like:date_key BETWEEN 20260901 AND
|
||
20260930、stat_month = 202609——认不准是不是时间列,说成有效行过滤会被写进数据目录)。detail 给条件涉及的实表列。
|
||
- 聚合与去重(aggregate / group_by):选出来的列只差在顶层的聚合函数上(同一列做参数,SUM 对 AVG;COUNT(*) 对 SUM(x))
|
||
或者去重上(COUNT / SUM / AVG 去不去重、COUNT(*) 对 COUNT(DISTINCT x)、SELECT DISTINCT 对不去重),分组一样。参数
|
||
得是光一列、在每组里不是定死的(不被 WHERE 定成常量、不等于分组键、表结构里它那张表的键没有在组里定死——按 u.id 分组
|
||
时 MAX(u.city) 和 MIN(u.city) 一样,LEFT JOIN 把行复制了也一样);去重不起作用的不说(MIN / MAX;不分组的聚合只出
|
||
一行;已经按选出来的列分组;只取一行;表结构里的键说明每行本来就不一样);每组只有一行时 SUM、AVG、MIN、MAX 不说。
|
||
- 关联的表、关联类型(tables / join):换了一张表;一边多关联了一张表(表结构里有指向它的外键、关联条件又用到它的键的
|
||
不说);一边多 LEFT JOIN 了一张表(在 ON 之外没用到、结果看得见重复的行、表结构里这张表的键不在 ON 里);同一张表、
|
||
同样的条件,LEFT JOIN 对内连接(可空一侧在 WHERE、HAVING、后面的连接里都没用到——后面的内连接引用可空一侧时没关联上
|
||
的行反正留不下来;结果看得见没关联上的那些行,不是只取可空一侧跳过空值的聚合、FILTER;原文写的是 RIGHT JOIN、或者
|
||
被 WHERE / 聚合改成内连接的,说法和原文对不上,不说)。另一边原文也 LEFT JOIN 了这张表、只是规范化因为结果不怕重复
|
||
把它去掉了的,不说「另一边没有」。
|
||
|
||
每一句都先确认这处不同真会影响结果:两边的 WHERE、HAVING、ON 里有可能恒为假的条件(常量条件,同一个对象上取不到值的
|
||
几个条件)时什么都不说;结果看不见行的(SELECT COUNT(*) > -1 这类只出一行、值又和行无关的)不说筛选、关联上的不同;
|
||
表结构里的约束(主键、唯一约束、外键、NOT NULL)说明不起作用的不说——约束只拿来少说,不拿来判等价。被 WHERE 用常量定死
|
||
的列出现在排序、分组里的差异,恒为真的 HAVING,都不在四类里,不说。两边给人看的写法一样、和已经说过的一句正好反过来的
|
||
不说。
|
||
|
||
**兜底。** 规范形式不一致、一条具体说法都没有时,给一条 aspect="other" 的「两条 SQL 的写法不同,未能归纳成具体差异,
|
||
请对照原文」;具体说法只覆盖了一部分时追加「另有未能归纳的差异,请对照原文」;fallback 为真。text_diff 给出规范化后
|
||
不同的行(difflib 比两边 sqlglot pretty 的输出,WHERE、ON、HAVING 的合取式每个条件一行,不同的行前面带上标出所在子句的
|
||
上级行,最多 40 行,见 _text_diff);有一边没有规范形式(保真守卫拦下的)时给原文的逐行对照。
|
||
|
||
仍做不到的(只给兜底说明,或者认不出等价):
|
||
- 认不出的等价改写:要靠约束才成立的(主键不为空时的 COUNT(主键) 和 COUNT(*)、按主键关联又没用到的 LEFT JOIN、带
|
||
主键的 DISTINCT、按主键分组时的函数依赖);IN 子查询和 EXISTS、NOT IN 和 NOT EXISTS 互换(空值时本来就不等价)、
|
||
IN / EXISTS 子查询和关联同一张表再去重、条件挪进相关子查询、IN 和 OR、冗余的条件、COALESCE 和 CASE 这类写法不同的
|
||
等价表达式、窗口函数的 PARTITION BY 换先后、HAVING 里分组键上的条件和 WHERE、去重的 UNION / INTERSECT 调换各支、
|
||
SQLite 里套在子查询里的 UNION、三项以上的加乘换结合、和方言有关的同义写法(DATE(x) 和 CAST(x AS DATE)、不在
|
||
_SAFE_RESPELLINGS 里的函数别名)、各支互不相交的 UNION ALL 和 UNION、只出一组时的排序、被 WHERE 定成常量的排序键
|
||
和分组键、恒为真的 HAVING、外层不在乎的子查询里的不同、LATERAL、递归 CTE、实表别名带列名、USING 用在第一个连接以外的
|
||
地方;保真守卫拦下的写法;没给表结构时 WHERE、GROUP BY、HAVING、相关子查询里用列别名的;结果看行出来的先后的一段调换
|
||
关联先后、按 WHERE 把外连接改成内连接、换成相等的列;没给表结构(或者类型不一样)时按关联条件相等的两列换着分组、
|
||
排序;PostgreSQL、MySQL 没给表结构时去重、分组、求和的一段调换关联先后、并子查询、去掉没用到的 LEFT JOIN、把只取
|
||
可空一侧聚合的 LEFT JOIN 写成内连接。
|
||
- 只给兜底的:不是简单形状的(CTE、子查询、UNION、窗口函数,规范化没把它们并掉的);四类以外的不同(同一列上的时间
|
||
范围、同一列上换了筛选的值、关联条件、分组、HAVING、排序、取几行、选出来的列、聚合的参数);几张表、几列交错的条件;
|
||
自己关联自己、同名的表关联了几次的;换了表又有关联条件不同的;条件写在聚合里的一边对条件写在 WHERE 里的一边(SUM(status
|
||
= 'paid') 对 COUNT(*) … WHERE status = 'paid':真正的不同是计数为 0 的组、空集时是 0 还是 NULL,说不清);反连接;
|
||
可能是时间窗口、又认不准是不是时间列的条件(业务日期键、统计月份、分区,第几天这类范围)。
|
||
- 已知的漏洞(只会说出「真会影响结果」以外的话,不会判错等价):几列交错的矛盾条件(a > b AND b > a)认不出恒为假,
|
||
两边都带着它时别处的不同其实看不出来;结果看不看得见行只按「SELECT 列表里有没有一项本身就是聚合」粗看,CASE WHEN
|
||
COUNT(*) > 0 THEN 1 END 这类恒为一个值的表达式不算聚合,认作看不见(少说),可 MAX(x) 旁边并排的那一项恒为一个值时
|
||
仍按看得见说;表结构里没写外键、没写键的,按「可能对不上、可能重复」说(库里实际有约束时这句话可能不起作用);没给
|
||
表结构时按主键分组的 MAX(x) 对 MIN(x)、AVG(DISTINCT x) 对 AVG(x) 照说(那时不知道键);「可能是时间」只认列名里的
|
||
词和像日期的常量,一边按 id 的单边范围、按批次号这类筛出来的其实是一段时间时,另一边仍会被说成「没有按时间筛选」;
|
||
存成 Unix 秒的时间列(created >= 1788220800)认不出是时间,会说成有效行条件;列名带 time 的时长列(wait_time <
|
||
3600)会被说成「按它筛时间」;被 CURRENT_DATE、DATE('now', …)、CAST(…)、上下界相同的 BETWEEN 定死的列不算定死,
|
||
MAX(stat_date) 对 MIN(stat_date) 这类仍会照说。这几类只影响说明的准确度,不会判错等价;双盲复算写数据目录时另有一道
|
||
门槛(业务日期只在两边的列都确实是日期 / 时间类型时才写,有效行条件只收排除无效记录的写法),不会因此写进错的事实。
|
||
- 收窄换来的代价:一边的聚合参数里用到某一列(SUM(amount - discount)),另一边多一个这一列上的条件(WHERE discount
|
||
> 0)时,这个条件不再说成「另一边没有这个条件」(认不出参数里的运算是不是在筛);列名带 day / month / period 这类
|
||
词、常量像日期的条件(delivery_days < 3、period_type = 'monthly')不再说成有效行过滤;另一边有任何上下界(amount
|
||
BETWEEN 10 AND 100)时不再说它「没有按时间筛选」。都只给兜底。
|
||
|
||
**和 sqlcheck 一样宁可漏报。** 解析不了(语法错、方言特性、多条语句、不是查询)就只给一条 aspect="parse";从不抛异常,
|
||
规范化或说明本身出了意外就当作这一级没有、说不清,记日志。只读语法树,不执行 SQL。
|
||
"""
|
||
from __future__ import annotations
|
||
|
||
import hashlib
|
||
import itertools
|
||
import logging
|
||
import math
|
||
import re
|
||
from collections import deque
|
||
from collections.abc import Iterable, Iterator, Mapping
|
||
from dataclasses import dataclass, field, fields, is_dataclass, replace
|
||
from typing import Any
|
||
|
||
from app.data import catalog
|
||
from app.data.names import name_key
|
||
from app.data.sqlcheck import _bare_column, _conjuncts, _parents, _unparen, dialect_of
|
||
|
||
logger = logging.getLogger(__name__)
|
||
|
||
#: 规范文本里字面量的占位符
|
||
PLACEHOLDER = "?"
|
||
#: 太长的 SQL 不做语法分析(normalize_sql 退回逐词规范化,facets 带 parse_error):解析耗时随长度增长
|
||
MAX_SQL_CHARS = 100_000
|
||
#: 每段 SELECT 在语法树上记段号的 meta 键。条件、表达式要复制一份再改写,复制出来的子查询靠段号找回原来那一段
|
||
#: 算好的来源(别名、外层):复制品没有父节点,自己找不到外层
|
||
_SEGMENT = "sql_semantics_segment"
|
||
#: 派生来源(引用 CTE 的表和 CTE 的定义、FROM / JOIN 里的子查询)在语法树上记比较用编号的 meta 键(见
|
||
#: _Statement._number_derived)。复制出来的条件里套着子查询时,靠它把 CTE 名、子查询的别名换成编号
|
||
_DERIVED = "sql_semantics_derived"
|
||
#: 不写编号的写法(render 的 anon 模式)里派生来源一律写成这个:排比较的两边谁在前时先看实表的列还是派生来源的列
|
||
_ANON = "@"
|
||
#: render 里比较用的写法(见 render):名字按 name_key,别名、各自起的名字都去掉
|
||
_COMPARING = frozenset({"key", "loose", "anon", "shape", "pierce"})
|
||
#: 外连接写在 WHERE 里会把可空那一侧没关联上的行(那一侧的列都是空)筛掉的条件:拿空值比都不成立
|
||
_NULL_REJECTING = ("EQ", "NEQ", "GT", "GTE", "LT", "LTE", "Like", "ILike", "Between", "In")
|
||
|
||
|
||
# ==========================================================================
|
||
# 解析
|
||
# ==========================================================================
|
||
|
||
|
||
def _read(dialect: str | None) -> str | None:
|
||
"""数据源类型(sqlite / postgresql / mysql …)或 sqlglot 的方言名 → 解析用的方言。认不出的按 sqlglot 的通用读法。"""
|
||
found = dialect_of(dialect)
|
||
if found or not dialect:
|
||
return found
|
||
from sqlglot.dialects.dialect import Dialect
|
||
|
||
name = dialect.strip().lower()
|
||
try:
|
||
Dialect.get_or_raise(name)
|
||
except Exception: # noqa: BLE001 - 不认识的方言名:按通用读法
|
||
return None
|
||
return name
|
||
|
||
|
||
def _parse_problem(error: Exception) -> str:
|
||
"""sqlglot 的报错 → 一句给人看的话。原文是英文、还带终端的高亮控制符,只取出错的位置。"""
|
||
errors = getattr(error, "errors", None) or []
|
||
first = errors[0] if errors and isinstance(errors[0], dict) else {}
|
||
line, col = first.get("line"), first.get("col")
|
||
if isinstance(line, int) and isinstance(col, int):
|
||
return f"SQL 无法解析(第 {line} 行第 {col} 列附近)"
|
||
return "SQL 无法解析"
|
||
|
||
|
||
def _parse(sql: Any, read: str | None) -> tuple[Any, str | None]:
|
||
"""(语法树, None) 或 (None, 为什么分析不了)。只认单条查询。"""
|
||
import sqlglot
|
||
from sqlglot import exp
|
||
|
||
if not isinstance(sql, str) or not sql.strip():
|
||
return None, "SQL 为空"
|
||
text = sql.strip()
|
||
if len(text) > MAX_SQL_CHARS:
|
||
return None, f"SQL 超过 {MAX_SQL_CHARS} 个字符,没有做语法分析"
|
||
try:
|
||
statements = [s for s in sqlglot.parse(text, read=read) if s is not None]
|
||
except Exception as e: # noqa: BLE001 - 语法错、方言特性、sqlglot 不支持的写法
|
||
return None, _parse_problem(e)
|
||
if not statements:
|
||
return None, "SQL 为空"
|
||
if len(statements) > 1:
|
||
return None, f"一次只能分析一条 SQL,这里有 {len(statements)} 条"
|
||
if not isinstance(statements[0], exp.Query):
|
||
return None, "不是查询语句(只分析 SELECT、WITH 开头的查询)"
|
||
return statements[0], None
|
||
|
||
|
||
# ==========================================================================
|
||
# 规范化:字面量换成占位符
|
||
# ==========================================================================
|
||
|
||
|
||
@dataclass(frozen=True)
|
||
class NormalizedSql:
|
||
"""规范文本和字面量。text 相同就是同一个模板,literals 是这一次填进去的值(SQL 写法,字符串带引号)。
|
||
|
||
parse_error 不为空时 text 是逐词规范化的结果(关键字转大写、不加引号的名字转小写、字面量换成占位符):
|
||
同一个模板只改了字面量的两条 SQL 照样得到同一个 text,只是空白、括号之外的写法差异认不出来。
|
||
"""
|
||
|
||
text: str
|
||
literals: tuple[str, ...] = ()
|
||
parse_error: str | None = None
|
||
|
||
def same_shape(self, other: "NormalizedSql") -> bool:
|
||
"""两条 SQL 是不是同一个模板(只有字面量不同也算)。"""
|
||
return self.text == other.text
|
||
|
||
def as_dict(self) -> dict[str, Any]:
|
||
out: dict[str, Any] = {"text": self.text, "literals": list(self.literals)}
|
||
if self.parse_error:
|
||
out["parse_error"] = self.parse_error
|
||
return out
|
||
|
||
|
||
def _is_literal(token: Any) -> bool:
|
||
"""字符串(含 N'…'、x'…'、$$…$$ 这些变体)和数字。"""
|
||
name = token.token_type.name
|
||
return name == "NUMBER" or name.endswith("STRING")
|
||
|
||
|
||
def _tokens(text: str, read: str | None) -> list[Any]:
|
||
from sqlglot.dialects.dialect import Dialect
|
||
|
||
return list(Dialect.get_or_raise(read).tokenize(text))
|
||
|
||
|
||
def normalize_sql(sql: str, dialect: str | None = None) -> NormalizedSql:
|
||
"""字面量换成占位符后的规范文本 + 字面量。
|
||
|
||
先按方言解析、重新生成一遍(去掉注释、统一关键字和空白、按方言的规则转换不加引号的名字的大小写),再把生成的
|
||
文本切成词,字符串和数字换成「?」:字面量按规范文本里出现的顺序排,不受语法树里子句存放顺序的影响。解析
|
||
不了就直接对原文逐词规范化,带上 parse_error。解析时会丢掉写法的(MySQL 会执行的 /*! … */ 注释、写成 ! 的 NOT、
|
||
SQLite 的 MOD() 这类重新生成时换了名字的函数,见 _lost_in_parsing)也对原文逐词规范化(不带 parse_error):
|
||
重新生成会把不同的结构写成一样。MySQL 里后面不跟空白的 -- 是两个减号,先拆开(见 _mysql_dashes)。从不抛异常。
|
||
"""
|
||
read = _read(dialect)
|
||
if read == "mysql" and isinstance(sql, str):
|
||
sql = _mysql_dashes(sql)
|
||
root, problem = _parse(sql, read)
|
||
if root is not None and (_lost_in_parsing(sql.strip(), read, root) or _respelled(sql.strip(), read, root)):
|
||
# 解析时会丢掉写法的(MySQL 会执行的注释、!、SQLite 的 MOD()……):重新生成的文本会把不同的结构写成一样,
|
||
# 改按原文逐词规范化(会执行的注释里的代码也算进去)
|
||
return _by_tokens(sql, read, None)
|
||
if root is not None:
|
||
try:
|
||
from sqlglot.optimizer.normalize_identifiers import normalize_identifiers
|
||
|
||
canonical = normalize_identifiers(root.copy(), dialect=read).sql(dialect=read, comments=False)
|
||
pieces: list[str] = []
|
||
literals: list[str] = []
|
||
pos = 0
|
||
for token in _tokens(canonical, read):
|
||
if _is_literal(token):
|
||
pieces += [canonical[pos:token.start], PLACEHOLDER]
|
||
literals.append(canonical[token.start:token.end + 1])
|
||
pos = token.end + 1
|
||
pieces.append(canonical[pos:])
|
||
return NormalizedSql("".join(pieces), tuple(literals))
|
||
except Exception: # noqa: BLE001 - 生成、切词出错:退回逐词规范化,照实标出
|
||
logger.warning("SQL 规范化失败,退回逐词规范化", exc_info=True)
|
||
problem = "SQL 的写法无法规范化"
|
||
return _by_tokens(sql if isinstance(sql, str) else "", read, problem)
|
||
|
||
|
||
def _by_tokens(text: str, read: str | None, problem: str | None) -> NormalizedSql:
|
||
"""逐词规范化(解析不了的 SQL、解析时会丢掉写法的 SQL):关键字转大写,不加引号的名字按 name_key,带引号的名字
|
||
原样,字面量换成占位符;! 和 NOT 各写各的。MySQL 会执行的注释(/*!版本号 … */)照样逐词规范化后写进去:里面是
|
||
要执行的代码,版本号算结构(决定执不执行),里面的字面量照样换成占位符。普通注释不算。"""
|
||
try:
|
||
tokens = _tokens(text, read)
|
||
except Exception: # noqa: BLE001 - 引号、括号没配对:只能把空白压成一个
|
||
return NormalizedSql(" ".join(text.split()), (), problem or "SQL 无法解析")
|
||
words: list[str] = []
|
||
literals: list[str] = []
|
||
glued = False
|
||
for i, token in enumerate(tokens):
|
||
raw = text[token.start:token.end + 1]
|
||
if glued:
|
||
# 数字后面紧跟着的名字(1_000 切成 1 和 _000、0o17 切成 0 和 o17):和前面的数字是一个字面量
|
||
literals[-1] += raw
|
||
glued = False
|
||
continue
|
||
nxt = tokens[i + 1] if i + 1 < len(tokens) else None
|
||
if token.token_type.name == "NUMBER" and nxt is not None and nxt.start == token.end + 1 and \
|
||
nxt.token_type.name in ("VAR", "NUMBER"):
|
||
glued = True
|
||
if _is_literal(token):
|
||
words.append(PLACEHOLDER)
|
||
literals.append(raw)
|
||
elif token.token_type.name == "VAR":
|
||
words.append(name_key(token.text))
|
||
elif token.token_type.name == "IDENTIFIER":
|
||
words.append(raw)
|
||
else:
|
||
words.append(raw if token.token_type.name == "NOT" and token.text == "!" else token.text.upper())
|
||
for comment in token.comments or []:
|
||
found = re.match(r"\s*(M?!)(\d*)(.*)", comment, re.S)
|
||
if found is None:
|
||
continue
|
||
inner = _by_tokens(found.group(3), read, None)
|
||
words += [f"/*{found.group(1)}{found.group(2)}", inner.text, "*/"]
|
||
literals += inner.literals
|
||
return NormalizedSql(" ".join(words), tuple(literals), problem)
|
||
|
||
|
||
# ==========================================================================
|
||
# 各个方面
|
||
# ==========================================================================
|
||
|
||
#: 一列:(表, 列)。表是给人看的真实表名(CTE、子查询写它的名字),认不出是哪张表时为 None
|
||
ColumnRef = tuple[str | None, str]
|
||
|
||
|
||
@dataclass(frozen=True)
|
||
class Part:
|
||
"""一个条件、一个分组键、一个聚合参数……
|
||
|
||
key 用来比较:别名换成真实表名,CTE、子查询写位置编号(见 _Statement._number_derived),表名、列名按
|
||
name_key,等式和比较的两边排好顺序;loose 同 key 但不写实表的表名(派生来源的编号照写:它不是表名,换了表也还是
|
||
那一个),两边查的表换了一张(orders 换成 orders_archive)时拿它
|
||
比其余各方面,免得同一个条件只因表名不同被报成不同。text 给人看:好几张表都有的列写「表.列」(真实表名,不是
|
||
别名),其余只写列名;full 一律写表名,两边的 text 一样、key 却不同时改用它说明。excludes 只在 WHERE 的「排除」
|
||
写法上有:<>、NOT、NOT IN、IS NOT NULL 去掉否定之后排除的是什么(IS NOT NULL 写成「列 为空」)。column_keys 和
|
||
columns 一一对应,是各列比较用的写法(实表.列;CTE、子查询的列写成「编号.列」):两边对同一列的不同条件靠它配成
|
||
一对说。相关子查询里引用外层的列,比较用的写法(key、loose、column_keys)在列名后面带「^」(往外几层带几个,见
|
||
_Statement._depth):内外读同一张表时,c.parent_id = o.id 和 c.id = o.parent_id 才分得开。
|
||
|
||
scope 是这一项在哪一层(条件、分组键、HAVING 上有,见 _Statement._layers):空串是最外层,「|2」是最外层 UNION
|
||
的第 2 支,「@3」是派生来源 @3 里(「@3|2」是它的 UNION 第 2 支),「?」是条件、SELECT 列表里的子查询。facets 按
|
||
(scope, key)去重:同一个条件在外层和子查询里各写了一次是两项。
|
||
|
||
后面四项不进比较(两个 Part 相等只看前面几项),只用来把不同说清楚:
|
||
origins 是这一项用到的派生来源的列各自的来历 [(「t1.total」, 「SUM(orders.amount)」)](见 _Statement._origin):
|
||
两边写法一模一样(都是 t1.total > 100),各自的 t1 却不是同一个时,带上它才分得清。pierce 是一路透过派生来源写到
|
||
实表的 key,只在中途被挡住时才有(这一项按派生来源的列写,见 _Statement.through);stage 说是被什么挡住的(「取 t
|
||
的前 10 行」「算 t 的窗口函数」):两边 pierce 一样、只有一边被挡住,是同一个条件筛的先后不同。roles 是用到的自己
|
||
关联自己的别名各是哪一个 [(别名, 在关联条件里出的列, 来源的名字)](见 _Statement._roles):关联的方向反了,两边都
|
||
写 a.amount,各自的 a 却不是同一个时,带上它才分得清。
|
||
"""
|
||
|
||
key: str
|
||
text: str
|
||
columns: tuple[ColumnRef, ...] = ()
|
||
excludes: str | None = None
|
||
loose: str = ""
|
||
full: str = ""
|
||
column_keys: tuple[str, ...] = ()
|
||
scope: str = ""
|
||
origins: tuple[tuple[str, str], ...] = field(default=(), compare=False)
|
||
pierce: str = field(default="", compare=False)
|
||
stage: str = field(default="", compare=False)
|
||
roles: tuple[tuple[str, str, str], ...] = field(default=(), compare=False)
|
||
|
||
@staticmethod
|
||
def plain(key: str, text: str) -> "Part":
|
||
"""没有列的一项(表名、COUNT(*) 的「*」):比较用的写法都是 key,给人看的都是 text。"""
|
||
return Part(key=key, text=text, loose=key, full=text)
|
||
|
||
|
||
@dataclass(frozen=True)
|
||
class JoinFacet:
|
||
"""一次连接。kind:inner / left / right / full / cross / semi / anti / natural。table 是关联进来的那一侧,other
|
||
是另一侧:关联条件里出现的别的表(给人看的真实表名);没有条件的(CROSS JOIN)是在它之前出现的各个来源。on 是
|
||
关联条件的合取式排好序后用 AND 连起来的写法,没有条件是 None。
|
||
|
||
sides 是这次连接涉及的来源(比较用的写法,排好序):关联进来的那一侧,加上关联条件里直接引用的这一段的来源(不
|
||
透过 CTE 找实表);没有条件的加上它之前出现的各个来源。内连接、CROSS JOIN 两边换个先后意思不变(FROM paid JOIN
|
||
refunded 和 FROM refunded JOIN paid),比它们时按 sides 和条件比,不按哪一侧是关联进来的。
|
||
|
||
keep、nullable 只在 LEFT / RIGHT JOIN 上有:保留全部行的主表那一侧、关联不上时记空的那一侧(比较用的写法,排好
|
||
序)。LEFT JOIN 关联进来的那一侧可空,关联条件里的其余来源(没有就是它之前的各个来源)是主表;RIGHT JOIN 反过来。
|
||
orders LEFT JOIN refunds 和 refunds RIGHT JOIN orders 是同一回事,和 refunds LEFT JOIN orders 不是:按主表比,
|
||
不按写的是 LEFT 还是 RIGHT。rejects 是同一段 WHERE 里把可空那一侧没关联上的行筛掉的条件(那一侧的列
|
||
拿来比较、IN、LIKE……,空值都不成立):有它这次外连接实际等于内连接。scope 是这次连接在哪一层(见 Part.scope):
|
||
外层关联了 refunds,和 EXISTS 子查询里同样关联了 refunds,是两次连接。"""
|
||
|
||
kind: str
|
||
table: Part
|
||
on: Part | None = None
|
||
other: tuple[str, ...] = ()
|
||
sides: tuple[str, ...] = ()
|
||
keep: tuple[str, ...] = ()
|
||
nullable: tuple[str, ...] = ()
|
||
rejects: Part | None = None
|
||
scope: str = ""
|
||
|
||
|
||
@dataclass(frozen=True)
|
||
class TimePredicate:
|
||
"""WHERE 里的一个时间条件。op:= / <> / > / >= / < / <= / between / in。value 是另一侧给人看的写法:
|
||
BETWEEN 是「低 AND 高」,IN 是「(…)」。
|
||
|
||
column 是按哪一列筛时间:条件写的是 CTE、子查询算出来的列(WITH t AS (SELECT DATE(paid_at) AS d …) … WHERE
|
||
d >= …)时,它在里面是同一列套着日期函数、CAST、MIN / MAX 的,换成里面那一列,一直换到实表(paid_at):按支付
|
||
时间还是下单时间筛,日期表达式写在 CTE 里也认得出。via 是路过的派生来源的列(「编号.列」,不进比较):时间那一面
|
||
已经报了按哪一列筛不同,派生来源那一列的定义不同就不再另报一遍。table 是这一列一直换到实表的 (表, 列)(不进
|
||
比较):读同一张表的几个派生来源同在一段时,column 照写派生来源的列(已付那份的 created_at、未付那份的 created_at
|
||
是两列,见 _Statement._seal),给调用方对照数据目录的还是 orders.created_at。"""
|
||
|
||
column: Part
|
||
op: str
|
||
value: str
|
||
condition: Part
|
||
via: tuple[str, ...] = field(default=(), compare=False)
|
||
table: ColumnRef | None = field(default=None, compare=False)
|
||
|
||
|
||
@dataclass(frozen=True)
|
||
class Aggregate:
|
||
"""一个聚合。func 是函数名的小写(sum / count / avg / min / max …);arg 是参数,COUNT(*)、COUNT(1) 的 arg
|
||
是「*」,去重聚合是 DISTINCT 里面的表达式;text 是整个聚合给人看的写法。窗口函数不算(它们按所在的那一列比,见
|
||
Computed)。"""
|
||
|
||
func: str
|
||
arg: Part
|
||
distinct: bool
|
||
text: str
|
||
scope: str = "" # 在哪一层(见 Part.scope):外层算的和标量子查询里算的同一个聚合是两项
|
||
|
||
|
||
@dataclass(frozen=True)
|
||
class Computed:
|
||
"""选出来的一列是怎么算的。两种:
|
||
|
||
- 最外层(UNION 的各支)选出来的列,不是光一列(分组那一面比)、也不是字面量的,比如 SUM(p.total) - SUM(r.total)、
|
||
SUM(amount) OVER (PARTITION BY user_id)、CASE …;ROUND、CAST、COALESCE 给默认值这类外壳先剥掉(见
|
||
_unwrap_value)。光一个聚合也收:另一边可能拿同一个合计再算了一下(SUM(amount) 对 SUM(amount) / 100)。scope、
|
||
scope_key 为 None,name 为空串。
|
||
- 派生来源(CTE、FROM 里的子查询)里被外面用到、又不是原样透出一列的列:它在里面的定义(amount * 0.9、
|
||
DATE(paid_at)、ROW_NUMBER() OVER (…)、p.total - r.total)。外面按「t.net」引用它,两边写的都是 t.net,算法却可能
|
||
不同,只比外面的写法看不出来。scope 是派生来源给人看的名字,scope_key 是它的位置编号,
|
||
name 是列名的 name_key,shown 是列名的原样写法。
|
||
|
||
expr 是定义(按它那一段的来源写的 Part;UNION 的各支用「|」连起来)。template 是把里面的聚合换成占位符之后的写法、
|
||
aggregates 是里面的聚合(按出现的先后):两边只差在聚合上(SUM(amount) * 100 对 SUM(paid_amount) * 100)、聚合
|
||
那一面又已经报了,这里不再报一遍;聚合的集合一样、只是位置换了(SUM(a) - SUM(b) 对 SUM(b) - SUM(a))才在这里报。
|
||
"""
|
||
|
||
scope: str | None
|
||
scope_key: str | None
|
||
name: str
|
||
shown: str
|
||
expr: Part
|
||
template: Part
|
||
aggregates: tuple[Aggregate, ...] = ()
|
||
|
||
|
||
@dataclass(frozen=True)
|
||
class LimitFacet:
|
||
"""一段查询取哪几行。count、offset 是行数和跳过行数的写法(没写为 None,OFFSET 0 也算没写);order 是这一段的
|
||
排序键 (Part, 是否降序):只取前几行时,按什么排决定了取到的是哪几行。scope 是哪一段:None 是最外层,CTE 写它的
|
||
名字,FROM 里的子查询写它的别名(没有别名写「子查询」)。条件里套着的子查询不单算:它整个写在那个条件里比。
|
||
scope_key 是这一段所属派生来源的编号(和 Part.key 里的同一套编号;最外层、UNION 的某一支这类不是派生来源整段的
|
||
为 None)。"""
|
||
|
||
count: str | None
|
||
offset: str | None = None
|
||
order: tuple[tuple[Part, bool], ...] = ()
|
||
scope: str | None = None
|
||
scope_key: str | None = None
|
||
|
||
|
||
@dataclass(frozen=True)
|
||
class SetOpFacet:
|
||
"""一段查询把几支 SELECT 合起来(UNION、UNION ALL、INTERSECT、EXCEPT)。ops 是各处合并的写法,按从左到右的先后
|
||
(「UNION」「UNION ALL」…);branches 是一共几支。scope 是哪一段:None 是最外层,CTE、FROM 里的子查询写它的名字;
|
||
scope_key 是那一段所在的层(见 Part.scope:最外层是空串,派生来源是它的位置编号)。条件里的子查询不单算:
|
||
它整个写在那个条件里比。"""
|
||
|
||
ops: tuple[str, ...]
|
||
branches: int
|
||
scope: str | None = None
|
||
scope_key: str = ""
|
||
|
||
|
||
@dataclass(frozen=True)
|
||
class Facets:
|
||
"""一条查询的各个方面。每一项都按在 SQL 里出现的先后排、按 key 去重;parse_error 不为空时其余都是空的。
|
||
|
||
limit 是最外层 LIMIT 的行数(写法);limits 是各段取哪几行(最外层在前,见 LimitFacet),只有写了 LIMIT 或
|
||
OFFSET 的段才有。
|
||
|
||
sources 是各段 FROM / JOIN 里的来源(实表、CTE、子查询;key 是比较用的写法,派生来源是编号,text 是给人看的
|
||
名字),按出现的先后、按 key 去重;条件里套着的子查询(IN (SELECT … FROM users))的来源不算,它们只筛行,不是
|
||
连接的一头。外连接的主表、可空那一侧按它写名字。
|
||
|
||
expressions 是算出来的列(见 Computed)。derived 是用到了的派生来源(key 是编号,text 是名字,含条件里套着的子
|
||
查询用到的 CTE):两边写法一样、同名的派生来源却不是同一个时,拿它说清「盲算的 t1 对应原流程的 t2」。
|
||
|
||
where、group_by、having、joins、aggregates、time 的每一项都记着在哪一层(Part.scope、JoinFacet.scope、
|
||
Aggregate.scope),按(层, key)去重;scopes 是这条语句里有的各层(key 是层的标记,text 是给人看的说法:「外层」
|
||
「t」「t 的 UNION 第 2 支」「条件的子查询」),给人说是哪一层时用。group_by 里也有当分组用的
|
||
SELECT DISTINCT 选出来的各项(这一段没有 GROUP BY、没有聚合时,和按它们分组出的行一样),distinct 是这样收进来的
|
||
那几层:说「去重」不说「分组」。set_ops 是各段的 UNION、INTERSECT、EXCEPT(见 SetOpFacet)。grain 是最后出来
|
||
的行按什么分(key 为「group:分组键…」「one」「rows」,见 _Statement._grain;最外层是 UNION 的为 None)。
|
||
|
||
只收用到了的段:没用到的 CTE(定义了、外面没引用)里的条件、聚合、表不影响结果,不收(见 _Statement._enliven)。
|
||
|
||
_origin 是 facets 记下的 (原文, 表结构),不进比较、不进 as_dict:diff_facets 拿它交给 explain_sql_diff 比原文。
|
||
"""
|
||
|
||
dialect: str | None = None
|
||
tables: tuple[Part, ...] = ()
|
||
joins: tuple[JoinFacet, ...] = ()
|
||
where: tuple[Part, ...] = ()
|
||
time: tuple[TimePredicate, ...] = ()
|
||
group_by: tuple[Part, ...] = ()
|
||
aggregates: tuple[Aggregate, ...] = ()
|
||
having: tuple[Part, ...] = ()
|
||
limit: str | None = None
|
||
limits: tuple[LimitFacet, ...] = ()
|
||
sources: tuple[Part, ...] = ()
|
||
expressions: tuple[Computed, ...] = ()
|
||
derived: tuple[Part, ...] = ()
|
||
set_ops: tuple[SetOpFacet, ...] = ()
|
||
distinct: tuple[str, ...] = ()
|
||
grain: Part | None = None
|
||
scopes: tuple[Part, ...] = ()
|
||
parse_error: str | None = None
|
||
_origin: tuple[str, Mapping[str, Any] | None] | None = field(default=None, compare=False, repr=False)
|
||
|
||
def as_dict(self) -> dict[str, Any]:
|
||
return _jsonable(self)
|
||
|
||
|
||
def _jsonable(value: Any) -> Any:
|
||
if is_dataclass(value) and not isinstance(value, type):
|
||
return {f.name: _jsonable(getattr(value, f.name)) for f in fields(value) if not f.name.startswith("_")}
|
||
if isinstance(value, (list, tuple)):
|
||
return [_jsonable(v) for v in value]
|
||
return value
|
||
|
||
|
||
def facets(sql: str, dialect: str | None = None, *, schema: Mapping[str, Any] | None = None) -> Facets:
|
||
"""把一条查询拆成各个方面。从不抛异常:解析不了、或者分析时出了意外,返回带 parse_error 的 Facets。
|
||
|
||
schema 是数据源的 schema_cache 形状({"tables": {表名: {"columns": [{"name": …}]}}}),可以不给。给了的话,
|
||
几张表连在一起、列又没加表前缀时按它认列是哪张表的;不给就照写列名,比较时可能把同一列当成两列。
|
||
|
||
返回的 Facets 记着原文和表结构(_origin):diff_facets 拿它交给 explain_sql_diff 比原文。
|
||
"""
|
||
read = _read(dialect)
|
||
root, problem = _parse(sql, read)
|
||
if root is None:
|
||
return Facets(dialect=read, parse_error=problem)
|
||
try:
|
||
return replace(_Statement(root, read, schema).facets(), _origin=(sql, schema))
|
||
except Exception: # noqa: BLE001 - 分析本身的缺陷不能连累调用方:当作分析不了,记下来修
|
||
logger.warning("SQL 语义分析失败", exc_info=True)
|
||
return Facets(dialect=read, parse_error="SQL 的结构无法分析")
|
||
|
||
|
||
# ---- 语法树上的小工具
|
||
|
||
#: 比较:(运算符的写法, 两边对调之后的类型)
|
||
_COMPARE: dict[str, tuple[str, str]] = {"EQ": ("=", "EQ"), "NEQ": ("<>", "NEQ"), "GT": (">", "LT"),
|
||
"GTE": (">=", "LTE"), "LT": ("<", "GT"), "LTE": ("<=", "GTE")}
|
||
#: 日期、时间函数(sqlglot 把各方言的写法归成这些类型:strftime、DATE_FORMAT 都是 TimeToStr)
|
||
_DATE_FUNCS = ("CurrentDate", "CurrentDatetime", "CurrentTimestamp", "CurrentTime", "Date", "Datetime", "DateAdd",
|
||
"DateSub", "DateDiff", "DateTrunc", "TimestampTrunc", "DatetimeTrunc", "TimeToStr", "StrToDate",
|
||
"StrToTime", "TsOrDsToDate", "TsOrDsToTimestamp", "TsOrDsAdd", "DateStrToDate", "TimeStrToTime",
|
||
"UnixToTime", "Interval", "Extract", "DateFromParts", "LastDay", "Year", "Month", "Day", "Week",
|
||
"Quarter", "DayOfWeek")
|
||
#: sqlglot 不认识、原样当普通函数的日期函数名
|
||
_DATE_NAMES = frozenset({"date", "datetime", "strftime", "julianday", "now", "today", "to_date", "to_char",
|
||
"date_format", "unixepoch", "curdate", "sysdate", "getdate"})
|
||
#: 像日期、时间的字符串:2026-09-01、2026-09、2026/9/1、2026-09-01 08:00:00
|
||
_DATE_TEXT = re.compile(r"^\s*\d{4}[-/]\d{1,2}(?:[-/]\d{1,2})?(?:[ T]\d{1,2}:\d{2}(?::\d{2}(?:\.\d+)?)?)?\s*$")
|
||
#: 中文列名里像日期、时间的写法(英文列名按 catalog.is_date_column)
|
||
_CN_TIME = re.compile(r"(?:日期|时间|时刻|年月|月份)$|^(?:年|月|日|周|日期|时间)$")
|
||
_PLAIN_NAME = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
|
||
|
||
|
||
def _nested(node: Any, stop: Any) -> bool:
|
||
"""node 在 stop 底下的某个子查询里(不属于 stop 这一段)。"""
|
||
from sqlglot import exp
|
||
|
||
return any(isinstance(p, (exp.Select, exp.Subquery)) for p in _parents(node, stop=stop))
|
||
|
||
|
||
def _own_columns(node: Any) -> list[Any]:
|
||
"""node 里属于这一段的列(不进嵌套的子查询),不含 t.*。"""
|
||
from sqlglot import exp
|
||
|
||
if node is None:
|
||
return []
|
||
found = [node] if isinstance(node, exp.Column) else \
|
||
[c for c in node.find_all(exp.Column) if not _nested(c, node)]
|
||
return [c for c in found if c.name and not c.is_star]
|
||
|
||
|
||
def _is_star(item: Any) -> bool:
|
||
"""SELECT 列表里的一项是 * 或 t.*。"""
|
||
from sqlglot import exp
|
||
|
||
return isinstance(item, exp.Star) or (isinstance(item, exp.Column) and item.is_star)
|
||
|
||
|
||
def _inner_queries(query: Any, skip: Iterable[Any] = ()) -> list[Any]:
|
||
"""query 里直接套着的查询(不再往里找),按出现的先后:条件、SELECT 列表里的子查询,UNION 的各支,子查询里面
|
||
那段。WITH 里的 CTE 定义不算(在用到它的地方才算),skip 里的节点整个跳过。"""
|
||
from sqlglot import exp
|
||
|
||
skipped = {id(n) for n in skip}
|
||
found = []
|
||
stack = list(reversed(list(query.iter_expressions())))
|
||
while stack:
|
||
node = stack.pop()
|
||
if id(node) in skipped or isinstance(node, exp.With):
|
||
continue
|
||
if isinstance(node, exp.Query):
|
||
found.append(node)
|
||
continue
|
||
stack.extend(reversed(list(node.iter_expressions())))
|
||
return found
|
||
|
||
|
||
def _named_output(select: Any) -> bool:
|
||
"""这段 SELECT 选出来的列会被别处按名字引用:它(或者它所在的 UNION)是 FROM / JOIN 里的子查询、CTE 的定义。
|
||
IN、EXISTS、标量子查询只看取值,选出来的列叫什么不影响口径。"""
|
||
from sqlglot import exp
|
||
|
||
node = select
|
||
while isinstance(node.parent, (exp.Subquery, exp.SetOperation)):
|
||
node = node.parent
|
||
return isinstance(node.parent, (exp.From, exp.Join, exp.CTE, exp.Lateral))
|
||
|
||
|
||
def _alias(item: Any) -> str:
|
||
"""FROM / JOIN 项的别名(没有别名的实表、CTE 是它的名字);表函数这类认不出的来源是空串。LATERAL 的别名
|
||
挂在 Lateral 节点上,不在里面的子查询上。"""
|
||
from sqlglot import exp
|
||
|
||
return item.alias_or_name if isinstance(item, (exp.Table, exp.Subquery, exp.Lateral)) else ""
|
||
|
||
|
||
def _derived_body(item: Any) -> Any:
|
||
"""FROM / JOIN 项是子查询(含 LATERAL 子查询)时,里面那段查询;别的(实表、CTE、表函数)返回 None。"""
|
||
from sqlglot import exp
|
||
|
||
if isinstance(item, exp.Lateral):
|
||
item = item.this
|
||
return item.this if isinstance(item, exp.Subquery) else None
|
||
|
||
|
||
def _branches(query: Any) -> list[Any]:
|
||
"""一段查询的各支 SELECT,按出现的先后:UNION 这类拆开,外面套的括号去掉;认不出的返回空列表。"""
|
||
from sqlglot import exp
|
||
|
||
while isinstance(query, exp.Subquery):
|
||
query = query.this
|
||
if isinstance(query, exp.SetOperation):
|
||
return _branches(query.this) + _branches(query.expression)
|
||
return [query] if isinstance(query, exp.Select) else []
|
||
|
||
|
||
def _output_names(query: Any) -> list[str | None]:
|
||
"""派生来源选出来的各列叫什么(外面按这个名字引用它们):按第一支 SELECT 的位置,* 为 None。CTE、子查询自己
|
||
写了列名的(WITH t(n) AS …、(…) AS t(a, b),递归 CTE 常这么写)按写的列名。"""
|
||
from sqlglot import exp
|
||
|
||
branches = _branches(query)
|
||
if not branches:
|
||
return []
|
||
names = [None if _is_star(item) else item.alias_or_name for item in branches[0].selects]
|
||
holder = query.parent
|
||
if isinstance(holder, exp.Subquery) and isinstance(holder.parent, exp.Lateral) and not holder.alias:
|
||
holder = holder.parent # LATERAL 的别名(连同列名)挂在 LATERAL 上
|
||
alias = holder.args.get("alias") if isinstance(holder, (exp.CTE, exp.Subquery, exp.Lateral)) else None
|
||
columns = [c.name for c in alias.columns] if isinstance(alias, exp.TableAlias) else []
|
||
for i, column in enumerate(columns[:len(names)]):
|
||
names[i] = column
|
||
return names
|
||
|
||
|
||
def _own_nodes(select: Any) -> Iterator[Any]:
|
||
"""一段 SELECT 自己的节点:不进 CTE、子查询(它们是别的段)。"""
|
||
from sqlglot import exp
|
||
|
||
stack = [select]
|
||
while stack:
|
||
node = stack.pop()
|
||
yield node
|
||
children = [c for c in node.iter_expressions()
|
||
if not isinstance(c, (exp.Select, exp.Subquery, exp.CTE, exp.With))]
|
||
stack.extend(reversed(children))
|
||
|
||
|
||
def _datey(node: Any) -> bool:
|
||
"""node 里有日期函数、转成日期的 CAST,或者像日期的字符串。"""
|
||
from sqlglot import exp
|
||
|
||
kinds = tuple(k for k in (getattr(exp, n, None) for n in _DATE_FUNCS) if isinstance(k, type))
|
||
for n in node.walk():
|
||
if isinstance(n, kinds):
|
||
return True
|
||
if isinstance(n, exp.Anonymous) and str(n.name).lower() in _DATE_NAMES:
|
||
return True
|
||
if isinstance(n, exp.Cast) and n.to is not None and any(w in n.to.this.name for w in ("DATE", "TIME")):
|
||
return True
|
||
if isinstance(n, exp.Literal) and n.is_string and _DATE_TEXT.match(str(n.this)):
|
||
return True
|
||
return False
|
||
|
||
|
||
def _value_column(node: Any) -> Any:
|
||
"""派生来源一列的定义是不是「还是那一列」:光一列,或者同一列套着日期函数、CAST、MIN / MAX(按支付时间算的日期、
|
||
每人最后一次付款的时间);是的话返回那一列,不是返回 None。窗口函数(ROW_NUMBER() OVER (ORDER BY paid_at) 是排名,
|
||
不是时间)、别的聚合、好几列算出来的、套着子查询的都不算。"""
|
||
from sqlglot import exp
|
||
|
||
n = _unparen(node)
|
||
if isinstance(n, exp.Column):
|
||
return n if n.name and not n.is_star else None
|
||
if any(isinstance(x, (exp.Window, exp.Select, exp.Subquery)) or
|
||
(isinstance(x, exp.AggFunc) and not isinstance(x, (exp.Min, exp.Max))) for x in n.walk()):
|
||
return None
|
||
cols = _own_columns(n)
|
||
if len(cols) != 1:
|
||
return None
|
||
if _datey(n) or _bare_column(n) is not None or any(isinstance(x, (exp.Min, exp.Max)) for x in n.walk()):
|
||
return cols[0]
|
||
return None
|
||
|
||
|
||
def _unwrap_value(node: Any) -> Any:
|
||
"""剥掉只改写法、不改是哪个数的外壳(括号、类型转换、ROUND、COALESCE / IFNULL 给默认值):最外层的
|
||
ROUND(SUM(amount), 2)、COALESCE(SUM(amount), 0) 还是那个合计,只是显示上的取舍。"""
|
||
from sqlglot import exp
|
||
|
||
while True:
|
||
node = _unparen(node)
|
||
if isinstance(node, (exp.Cast, exp.TryCast, exp.Round)) and node.this is not None:
|
||
node = node.this
|
||
elif isinstance(node, exp.Coalesce) and node.expressions and \
|
||
all(isinstance(_unparen(e), exp.Literal) for e in node.expressions):
|
||
node = node.this
|
||
else:
|
||
return node
|
||
|
||
|
||
def _as_date(node: Any) -> Any:
|
||
"""「取日期」的几种写法统一成 CAST(… AS DATE)(比较用,给人看的照原样):DATE(paid_at)、paid_at::date、
|
||
TRY_CAST(paid_at AS DATE),MySQL 的 DATE() 读成 TsOrDsToDate。各方言、各人写法不同,取的都是同一天;带时区偏移、
|
||
格式的(DATE(paid_at, '+8 hours'))不是同一天,不动。"""
|
||
from sqlglot import exp
|
||
|
||
def to_date(n: Any) -> bool:
|
||
return isinstance(n, exp.DataType) and n.this == exp.DataType.Type.DATE
|
||
|
||
if isinstance(node, exp.Date) and node.this is not None and not node.args.get("zone") \
|
||
and not node.args.get("expressions"):
|
||
return exp.Cast(this=node.this, to=exp.DataType.build("DATE"))
|
||
if isinstance(node, exp.TsOrDsToDate) and node.this is not None and not node.args.get("format"):
|
||
return exp.Cast(this=node.this, to=exp.DataType.build("DATE"))
|
||
if isinstance(node, exp.TryCast) and to_date(node.args.get("to")):
|
||
return exp.Cast(this=node.this, to=exp.DataType.build("DATE"))
|
||
return node
|
||
|
||
|
||
def _atomic(node: Any) -> bool:
|
||
"""算式里的一项不是别的加减乘除:两边对调不会改变怎么结合(a * b / c 换成 b / c * a 就不一样了)。"""
|
||
from sqlglot import exp
|
||
|
||
return not isinstance(_unparen(node), (exp.Add, exp.Sub, exp.Mul, exp.Div, exp.Mod, exp.IntDiv))
|
||
|
||
|
||
def _temporal_name(name: str) -> bool:
|
||
"""列名像日期、时间:英文按 catalog.is_date_column 的命名规则(paid_at、order_date),中文看结尾(支付日期)。"""
|
||
return catalog.is_date_column(name, None) or bool(_CN_TIME.search(name))
|
||
|
||
|
||
#: 切词后有一个词是这些的列名「可能是时间」:数仓的业务日期键 date_key、统计月份 stat_month、分区 pt / ds、财年
|
||
#: fiscal_year、第几天 day_no……认不准是不是时间列(_temporal_name 认的是一定像日期的),只拿来少说:这样的列上的
|
||
#: 窗口不说成有效行过滤,另一边有它时也不说「没有按时间筛选」
|
||
_TIMEISH_WORDS = frozenset({"date", "dates", "time", "times", "day", "days", "dt", "ds", "ts", "timestamp", "datetime",
|
||
"month", "months", "mon", "year", "years", "yr", "fy", "fiscal", "week", "weeks", "wk",
|
||
"quarter", "qtr", "period", "hour", "hr", "minute", "ym", "ymd", "yyyymm", "yyyymmdd",
|
||
"yyyy", "pt", "datekey", "datekeys"})
|
||
_CN_TIMEISH = re.compile(r"日期|时间|时刻|年月|月份|年度|季度|账期|期间|统计期|周期|[年月日周]$")
|
||
#: 可能是日期的数字写法:yyyymm、yyyymmdd(20260901、202609,数仓常这么存日期键、月份)
|
||
_YMD_DIGITS = re.compile(r"^(?:19|20|21)\d{2}(?:0[1-9]|1[0-2])(?:0[1-9]|[12]\d|3[01])?$")
|
||
#: 可能是日期的文字:年('2026')、季度('2026Q3'、'2026-Q3')、周('2026-W36')
|
||
_PARTIAL_DATE_TEXT = re.compile(r"^\s*(?:19|20|21)\d{2}(?:[-/]?[Qq][1-4]|-?[Ww]\d{1,2})?\s*$")
|
||
|
||
|
||
def _timeish_name(name: str) -> bool:
|
||
"""列名可能是时间(见 _TIMEISH_WORDS):比 _temporal_name 宽。"""
|
||
return _temporal_name(name) or any(w in _TIMEISH_WORDS for w in catalog.name_words(name)) or \
|
||
bool(_CN_TIMEISH.search(name))
|
||
|
||
|
||
def _date_like_literal(node: Any) -> bool:
|
||
"""常量可能是日期:像日期的字符串(2026-09-01、2026-09、2026、20260901、2026Q3),yyyymm / yyyymmdd 的整数,
|
||
LIKE 的模式去掉两头的 % / _ 之后像日期('2026-09%')。"""
|
||
from sqlglot import exp
|
||
|
||
node = _unparen(node)
|
||
if not isinstance(node, exp.Literal):
|
||
return False
|
||
text = str(node.this).strip()
|
||
if node.is_string:
|
||
text = text.strip("%_")
|
||
return bool(_DATE_TEXT.match(text) or _PARTIAL_DATE_TEXT.match(text) or _YMD_DIGITS.match(text))
|
||
return bool(_YMD_DIGITS.match(text))
|
||
|
||
|
||
#: 拿列和值比的谓词(判空不算:x IS NOT NULL 是有效行过滤,不是时间窗口)
|
||
_WINDOW_PREDICATES = ("EQ", "NEQ", "GT", "GTE", "LT", "LTE", "In", "Between", "Like", "ILike", "RegexpLike",
|
||
"RegexpILike", "Glob", "SimilarTo")
|
||
|
||
|
||
def _window_like(node: Any) -> bool:
|
||
"""node 里有可能是时间窗口的比较:拿可能是时间的列(见 _timeish_name)、日期函数、像日期的常量(见
|
||
_date_like_literal、_datey)去比。判空不算。只拿来少说:这样的条件不说成有效行过滤,另一边有它时也不说「没有按
|
||
时间筛选」。"""
|
||
from sqlglot import exp
|
||
|
||
for n in node.walk():
|
||
if type(n).__name__ not in _WINDOW_PREDICATES:
|
||
continue
|
||
if _datey(n) or any(_date_like_literal(x) for x in n.find_all(exp.Literal)) or \
|
||
any(c.name and _timeish_name(c.name) for c in n.find_all(exp.Column)):
|
||
return True
|
||
return False
|
||
|
||
|
||
def _bounded_both_ways(tree: Any) -> bool:
|
||
"""语句里有对同一个对象的上下界(BETWEEN,或者 x >= 常量 和 x < 常量 都有):可能是认不出的时间窗口(第几天
|
||
day_no BETWEEN 20697 AND 20726)。只拿来少说。"""
|
||
from sqlglot import exp
|
||
|
||
if tree.find(exp.Between) is not None:
|
||
return True
|
||
low: set[str] = set()
|
||
high: set[str] = set()
|
||
for n in tree.find_all(exp.GT, exp.GTE, exp.LT, exp.LTE):
|
||
left, right = _literal_value(n.this) is not None, _literal_value(n.expression) is not None
|
||
if left == right:
|
||
continue
|
||
subject = n.expression if left else n.this
|
||
upward = isinstance(n, (exp.GT, exp.GTE)) != left
|
||
(low if upward else high).add(_fingerprint(subject))
|
||
return bool(low & high)
|
||
|
||
|
||
def _join_kind(join: Any) -> str:
|
||
side = str(join.args.get("side") or "").lower()
|
||
kind = str(join.args.get("kind") or "").lower()
|
||
if side in ("left", "right", "full"):
|
||
return side
|
||
if join.args.get("method"):
|
||
return "natural"
|
||
if kind in ("semi", "anti"):
|
||
return kind
|
||
if kind == "cross" or not (join.args.get("on") is not None or join.args.get("using")):
|
||
return "cross"
|
||
return "inner"
|
||
|
||
|
||
def _dedupe(items: Iterable[Any], key: Any) -> tuple[Any, ...]:
|
||
seen: dict[Any, Any] = {}
|
||
for item in items:
|
||
seen.setdefault(key(item), item)
|
||
return tuple(seen.values())
|
||
|
||
|
||
class _Sources(dict):
|
||
"""一段 SELECT 的来源 {别名的 name_key: _Source}。outer 是外层那段的来源:相关子查询里引用外层的别名(o.id)
|
||
从这里找。"""
|
||
|
||
outer: "_Sources | None" = None
|
||
|
||
|
||
@dataclass
|
||
class _Source:
|
||
"""一段 SELECT 的一个来源。kind:table(实表)/ derived(CTE、子查询、LATERAL 子查询)/ other(表函数、认不出的
|
||
别名)。"""
|
||
|
||
kind: str
|
||
name: str # 给人看:表名(有库名就带上)、CTE 名、子查询(含 LATERAL)的别名
|
||
key: str # 比较用:实表是 name_key,派生来源是编号(见 _Statement._number_derived)
|
||
idents: tuple[Any, ...] = () # 表名各段的 Identifier:给人看的写法照原样写回去
|
||
query: Any = None # derived:里面那段查询
|
||
role: str = "" # 同一个来源在这一段里出现了不止一次时,这一次是哪一个(见 _Statement._roles)
|
||
alias: str = "" # 这一段里引用它的名字(别名,没有别名是表名、CTE 名):给人看时分内外层用
|
||
opaque: bool = False # derived:同一段里别的来源透过去读的也是同一个来源,不透过去(见 _Statement._seal)
|
||
|
||
@property
|
||
def ref(self) -> str:
|
||
"""比较用的「哪一个来源」:key,自己关联自己时带上 role(orders#parent_id)。"""
|
||
return f"{self.key}#{self.role}" if self.role else self.key
|
||
|
||
|
||
class _Statement:
|
||
"""一条语句的分析:CTE、实表、每段 SELECT 的来源,以及把表达式写成 key / text 的规则。"""
|
||
|
||
def __init__(self, root: Any, read: str | None, schema: Mapping[str, Any] | None) -> None:
|
||
from sqlglot import exp
|
||
|
||
self.root = root
|
||
self.read = read
|
||
#: 各段 SELECT(原树上的节点),下标就是记在 meta 里的段号
|
||
self.segments: list[Any] = []
|
||
for select in root.find_all(exp.Select, bfs=False):
|
||
select.meta[_SEGMENT] = len(self.segments)
|
||
self.segments.append(select)
|
||
self.ctes: dict[str, Any] = {}
|
||
#: CTE 的定义(exp.CTE 节点),和 ctes 同一个键:编号也记在定义上,定义写进条件时靠它换掉名字
|
||
self._cte_defs: dict[str, Any] = {}
|
||
for cte in root.find_all(exp.CTE):
|
||
if cte.alias:
|
||
self.ctes.setdefault(name_key(cte.alias), cte.this)
|
||
self._cte_defs.setdefault(name_key(cte.alias), cte)
|
||
self.columns: dict[str, set[str]] = {}
|
||
for name, meta in ((schema or {}).get("tables") or {}).items():
|
||
if isinstance(meta, Mapping):
|
||
self.columns[name_key(str(name))] = {name_key(str(c.get("name"))) for c in meta.get("columns") or []
|
||
if isinstance(c, Mapping) and c.get("name")}
|
||
self._memo: dict[int, _Sources] = {}
|
||
self._shared: set[str] | None = None
|
||
#: 各派生来源的结构标识 {位置编号: 摘要}(见 _signature)、各派生来源一列的来历 {(里面那段的 id, 列名): 来历}
|
||
#: (见 _origin):都和编号无关,重新编号不用清
|
||
self._signatures: dict[int, str] = {}
|
||
self._origins: dict[tuple[int, str], str | None] = {}
|
||
#: 派生来源各列的名字 {里面那段的 id: [列名]}、每段外面筛和里面筛一不一样 {(那段的 id, 列名): 说法}:只看
|
||
#: 语法树,算一次记下来(透视、认列时反复要用)
|
||
self._names_memo: dict[int, list[str | None]] = {}
|
||
self._blocks_memo: dict[tuple[int, str], str | None] = {}
|
||
self._number_derived()
|
||
self._enliven(())
|
||
|
||
# ---- 来源
|
||
|
||
def _is_cte(self, table: Any) -> bool:
|
||
return not table.text("db") and not table.text("catalog") and name_key(table.name) in self.ctes
|
||
|
||
def _reach(self, extra: Iterable[str] = ()) -> set[str]:
|
||
"""从最外层那段查询出发(再加上 extra 里的 CTE)一路引用到的 CTE 名:FROM / JOIN 里的、条件和 SELECT 列表里的
|
||
子查询里的都算,用到的 CTE 里面再引用的也算。CTE 的定义本身不算引用:没人用的 CTE 里引用的别的 CTE 也没用到。"""
|
||
from sqlglot import exp
|
||
|
||
live = {name for name in extra if name in self.ctes}
|
||
stack = [self.root, *(self.ctes[name] for name in live)]
|
||
while stack:
|
||
for node in stack.pop().walk(prune=lambda n: isinstance(n, exp.CTE)):
|
||
if isinstance(node, exp.Table) and isinstance(node.this, exp.Identifier) and node.name \
|
||
and self._is_cte(node) and (name := name_key(node.name)) not in live:
|
||
live.add(name)
|
||
stack.append(self.ctes[name])
|
||
return live
|
||
|
||
def _enliven(self, extra: Iterable[str]) -> None:
|
||
"""定下哪些段算数:最外层那段查询,加上从它一路引用到的 CTE(extra 里的 CTE 也当作用到了)。没
|
||
用到的 CTE 里的条件、聚合、表不影响结果:收进来的话,盲算定义了只筛已付的 CTE 却直接查 orders,会被当成也筛了
|
||
已付;定义了没用的 CTE 里多一个条件,又会被报成多筛了一遍。用到的实表(给人看时要不要写表名)、共有的列名跟着
|
||
重算。"""
|
||
from sqlglot import exp
|
||
|
||
live = self._reach(extra)
|
||
#: 没用到的 CTE 里的节点(id):facets 只收别的段
|
||
self._dead = {id(n) for name, cte in self._cte_defs.items() if name not in live for n in cte.this.walk()}
|
||
tables: dict[str, _Source] = {}
|
||
for table in self._alive(exp.Table):
|
||
if isinstance(table.this, exp.Identifier) and table.name and not self._is_cte(table):
|
||
source = self._table(table)
|
||
tables.setdefault(source.key, source)
|
||
self.tables = list(tables.values())
|
||
#: 整条语句只用到一张实表:给人看的写法一律不写表名
|
||
self.single = len(self.tables) <= 1
|
||
self._shared = None
|
||
|
||
def _alive(self, *types: Any) -> Iterator[Any]:
|
||
"""用到了的段里这些类型的节点(深度优先、按出现的先后),没用到的 CTE 里的不算(见 _enliven)。"""
|
||
return (node for node in self.root.find_all(*types, bfs=False) if id(node) not in self._dead)
|
||
|
||
@staticmethod
|
||
def _items(select: Any) -> list[Any]:
|
||
"""一段 SELECT 的 FROM、JOIN 项,按出现的先后。"""
|
||
first = select.args.get("from_")
|
||
return ([first.this] if first is not None else []) + [j.this for j in select.args.get("joins") or []]
|
||
|
||
def _number_derived(self) -> None:
|
||
"""给派生来源(引用 CTE 的表、FROM / JOIN 里的子查询和 LATERAL 子查询)编比较用的 key:@1、@2…,记在节点的
|
||
meta 上。
|
||
|
||
CTE 名、子查询的别名是各自起的(双盲复算的盲算更是),拿它作 key,同一个口径只因起的名字不同就被报成不同;
|
||
拿里面那段查询的写法作 key 也不行:里面差一个条件,外层按它的列写的条件就全跟着变,同一处不同报好几遍,还会
|
||
出现两边各说一句「只保留 total > 100 的行」的自相矛盾。所以单看一条语句时按位置编:从最外层往里一层一层地
|
||
编,同一层按出现的先后;CTE 在第一次用到的地方编号,在哪儿用到都是这一个编号。写成 CTE 还是 FROM 里的子查询、
|
||
CTE 定义的先后都不影响编号,里层多一段、少一段也不挪动外层的编号。这套编号只给 facets 用;比两条语句不靠
|
||
它(见 explain_sql_diff:两边各自规范化,CTE 按定义起名)。
|
||
"""
|
||
from sqlglot import exp
|
||
|
||
ctes: dict[str, int] = {}
|
||
count = 0
|
||
#: 编了号的派生来源 [(位置编号, 起的名字, 里面那段查询, CTE 的 name_key 或 None)],按编号排:facets 按它们
|
||
#: 找各列的定义(见 _expressions)
|
||
self._derived: list[tuple[int, str, Any, str | None]] = []
|
||
|
||
def mark(node: Any, number: int) -> None:
|
||
node.meta[_DERIVED] = f"@{number}"
|
||
|
||
def visit(select: Any) -> list[Any]:
|
||
"""给这一段还没编号的 FROM / JOIN 项编号,返回要接着往里看的查询(子查询、第一次用到的 CTE 的定义)。"""
|
||
nonlocal count
|
||
inner = []
|
||
for item in self._items(select):
|
||
body = _derived_body(item)
|
||
if body is not None:
|
||
if _DERIVED not in item.meta:
|
||
count += 1
|
||
mark(item, count)
|
||
self._derived.append((count, _alias(item), body, None))
|
||
inner.append(body)
|
||
elif isinstance(item, exp.Table) and isinstance(item.this, exp.Identifier) and item.name \
|
||
and self._is_cte(item):
|
||
name = name_key(item.name)
|
||
if name not in ctes:
|
||
count += 1
|
||
ctes[name] = count
|
||
mark(self._cte_defs[name], count)
|
||
self._derived.append((count, self._cte_defs[name].alias, self.ctes[name], name))
|
||
inner.append(self.ctes[name])
|
||
mark(item, ctes[name])
|
||
return inner
|
||
|
||
queue: deque[Any] = deque([self.root])
|
||
while queue:
|
||
query = queue.popleft()
|
||
items: list[Any] = []
|
||
if isinstance(query, exp.Select):
|
||
queue.extend(visit(query))
|
||
items = self._items(query)
|
||
queue.extend(_inner_queries(query, skip=items))
|
||
# 没走到的(没用到的 CTE 里的、表函数参数里的)按出现的先后补上:每个派生来源都得有编号
|
||
for select in self.root.find_all(exp.Select, bfs=False):
|
||
visit(select)
|
||
# 没用到的 CTE 也编上号(排在最后):每个派生来源都得有编号
|
||
for name, cte in self._cte_defs.items():
|
||
if name not in ctes:
|
||
count += 1
|
||
ctes[name] = count
|
||
mark(cte, count)
|
||
self._derived.append((count, cte.alias, self.ctes[name], name))
|
||
#: {位置编号: 里面那段查询}、{位置编号: 现在比较用的编号}、反过来 {比较用的编号: 位置编号}
|
||
self._bodies = {number: body for number, _, body, _ in self._derived}
|
||
self._labels = {number: f"@{number}" for number in self._bodies}
|
||
self._numbers = {label: number for number, label in self._labels.items()}
|
||
|
||
def column_key(self, source: _Source, column: str) -> str:
|
||
"""比较用的列名:按 name_key。"""
|
||
return name_key(column)
|
||
|
||
def _shape(self, node: Any, sources: dict[str, _Source]) -> str:
|
||
"""按结构写(render 的 shape 模式,比较的两边也按结构排):配对用,和名字、编号无关。"""
|
||
return self.render(self.canonical(node, sources, by="shape"), sources, "shape")
|
||
|
||
def _signature(self, number: int) -> str:
|
||
"""派生来源的结构标识(shape 写法里引用它的地方写这个):里面那段查询按 shape 写法写出来再取摘要。里面再套着
|
||
的派生来源也写它们各自的结构标识,所以付款合计、退款合计两个 CTE 的同名列 total 在外面一层也分得开,和起的
|
||
名字、编号无关;取摘要是免得一层套一层越写越长。递归 CTE 引用自己时写「@」。"""
|
||
if number not in self._signatures:
|
||
self._signatures[number] = _ANON
|
||
text = self._shape(self._bodies[number], {})
|
||
self._signatures[number] = "@" + hashlib.sha1(text.encode("utf-8")).hexdigest()[:12]
|
||
return self._signatures[number]
|
||
|
||
def _signature_of(self, label: str) -> str:
|
||
"""比较用的编号 → 那个派生来源的结构标识;认不出的编号写「@」。"""
|
||
number = self._numbers.get(label)
|
||
return self._signature(number) if number is not None else _ANON
|
||
|
||
@staticmethod
|
||
def _table(table: Any) -> _Source:
|
||
from sqlglot import exp
|
||
|
||
idents = tuple(i for i in (table.args.get("catalog"), table.args.get("db"), table.this)
|
||
if isinstance(i, exp.Identifier))
|
||
return _Source("table", ".".join(i.name for i in idents), ".".join(name_key(i.name) for i in idents),
|
||
tuple(i.copy() for i in idents))
|
||
|
||
def sources(self, select: Any) -> _Sources:
|
||
"""{别名的 name_key: 来源}。没有别名的实表按表名;认不出的来源按位置编号。"""
|
||
from sqlglot import exp
|
||
|
||
if id(select) in self._memo:
|
||
return self._memo[id(select)]
|
||
out = _Sources()
|
||
aliases: dict[str, str] = {}
|
||
for i, node in enumerate(self._items(select)):
|
||
alias = _alias(node)
|
||
slot = name_key(alias) if alias else f"#{i}"
|
||
if isinstance(node, exp.Table) and isinstance(node.this, exp.Identifier) and node.name:
|
||
if self._is_cte(node):
|
||
out[slot] = _Source("derived", node.name, node.meta.get(_DERIVED, slot), (node.this.copy(),),
|
||
self.ctes[name_key(node.name)])
|
||
else:
|
||
out[slot] = self._table(node)
|
||
elif (body := _derived_body(node)) is not None:
|
||
out[slot] = _Source("derived", alias or "子查询", node.meta.get(_DERIVED, slot),
|
||
(exp.to_identifier(alias),) if alias else (), body)
|
||
else:
|
||
out[slot] = _Source("other", alias, slot)
|
||
out[slot].alias = alias
|
||
aliases[slot] = alias
|
||
self._roles(select, out, aliases)
|
||
self._memo[id(select)] = out
|
||
parent = select.parent
|
||
while parent is not None and not isinstance(parent, exp.Select):
|
||
parent = parent.parent
|
||
out.outer = self.sources(parent) if parent is not None else None
|
||
self._seal(out)
|
||
return out
|
||
|
||
def _base(self, source: _Source) -> str:
|
||
"""一路透过「只读一个来源」的派生来源(through 能透过去的那一路),最后读的是哪个来源:「种类:key」。"""
|
||
from sqlglot import exp
|
||
|
||
for _ in range(8):
|
||
if source.kind != "derived" or not isinstance(source.query, exp.Select):
|
||
break
|
||
inner = self.sources(source.query)
|
||
if len(inner) != 1:
|
||
break
|
||
source = next(iter(inner.values()))
|
||
return f"{source.kind}:{source.key}"
|
||
|
||
def _seal(self, sources: dict[str, _Source]) -> None:
|
||
"""同一段里有几个来源透过去读的是同一个来源时,其中的派生来源都不透过去(就地标 opaque,见 through)。
|
||
|
||
两个原样透出的 CTE 都读 orders(已付的一份、未付的一份):透过去 p.amount、o.amount 都成了 orders.amount,外层
|
||
用的是哪一份就分不出来了(SUM(p.amount) 对 SUM(o.amount)、p.amount > 100 对 o.amount > 100 都成了一样);CTE 和
|
||
它自己读的那张表关联(父单、子单)、同一个原样透出的 CTE 关联两次也一样,自己关联自己时 _roles 设的 role 透过去
|
||
也就丢了。这时这几个派生来源的列照写它们自己的编号:用的是已付那份
|
||
还是未付那份分得开。只有一个来源读那张表时照旧透过去:先在 CTE 里筛一遍再查和直接查
|
||
是同一列。"""
|
||
groups: dict[str, list[_Source]] = {}
|
||
for source in sources.values():
|
||
groups.setdefault(self._base(source), []).append(source)
|
||
for found in groups.values():
|
||
if len(found) > 1:
|
||
for source in found:
|
||
if source.kind == "derived":
|
||
source.opaque = True
|
||
|
||
@staticmethod
|
||
def _roles(select: Any, sources: dict[str, _Source], aliases: Mapping[str, str]) -> None:
|
||
"""同一个来源在这一段里出现了不止一次(自己关联自己:orders o1 JOIN orders o2、base b1 JOIN base b2)时,各自
|
||
按在关联条件(ON、WHERE 里两列相等)里出的是哪几列区分,记在 role 上(就地改):比较用的写法写「来源#列」,
|
||
和别名、先后都无关。b2.user_id = b1.referrer_id 里 b1 出 referrer_id、b2 出 user_id,b1.total > b2.total 写反了
|
||
就认得出来;不然两边都写成 base.total,写反了也一样。出的列一样(b1.user_id = b2.user_id)分不出,也不用分:
|
||
两个对调过来意思一样。给人看的写法这时写别名:都写 orders.amount 读不出是哪一个。"""
|
||
from sqlglot import exp
|
||
|
||
groups: dict[str, list[str]] = {}
|
||
for slot, source in sources.items():
|
||
if source.kind in ("table", "derived"):
|
||
groups.setdefault(source.key, []).append(slot)
|
||
repeated = [slots for slots in groups.values() if len(slots) > 1]
|
||
if not repeated:
|
||
return
|
||
conds = [j.args["on"] for j in select.args.get("joins") or [] if j.args.get("on") is not None]
|
||
if (where := select.args.get("where")) is not None:
|
||
conds.append(where.this)
|
||
joined: dict[str, set[str]] = {}
|
||
for cond in conds:
|
||
for eq in _conjuncts(cond):
|
||
n = _unparen(eq)
|
||
if not isinstance(n, exp.EQ):
|
||
continue
|
||
cols = [_unparen(n.this), _unparen(n.expression)]
|
||
slots = [name_key(c.table) if isinstance(c, exp.Column) and c.table else None for c in cols]
|
||
if None in slots or slots[0] == slots[1] or not all(s in sources for s in slots):
|
||
continue
|
||
for col, slot in zip(cols, slots):
|
||
joined.setdefault(slot, set()).add(name_key(col.name))
|
||
for slots in repeated:
|
||
roles = {slot: "+".join(sorted(joined.get(slot, ()))) for slot in slots}
|
||
for slot in slots:
|
||
if roles[slot] and list(roles.values()).count(roles[slot]) == 1:
|
||
sources[slot].role = roles[slot]
|
||
if aliases.get(slot):
|
||
sources[slot].idents = (exp.to_identifier(aliases[slot]),)
|
||
|
||
def _exposes(self, source: _Source, column: str) -> bool:
|
||
"""这个来源可能有这一列:实表看表结构(没给表结构就当可能有),CTE、子查询看它选出来的列。"""
|
||
from sqlglot import exp
|
||
|
||
if source.kind == "table":
|
||
return source.key not in self.columns or name_key(column) in self.columns[source.key]
|
||
if source.kind == "derived" and isinstance(source.query, exp.Query):
|
||
return any(called is None or name_key(called) == name_key(column)
|
||
for called in self.names(source.query))
|
||
return True
|
||
|
||
def locate(self, column: Any, sources: dict[str, _Source]) -> tuple[str | None, _Source | None]:
|
||
"""列 → (来源的别名键, 来源)。别名在这一段找不到的,到外层找(相关子查询引用外层),再找不到按「别的来源」
|
||
照写(别名写错了);没加前缀、又有不止一个来源可能有它的,返回 (None, None)。"""
|
||
qualifier = column.table
|
||
if qualifier:
|
||
slot = name_key(qualifier)
|
||
if slot in sources:
|
||
return slot, sources[slot]
|
||
outer = getattr(sources, "outer", None)
|
||
while outer is not None:
|
||
if slot in outer:
|
||
return None, outer[slot] # 外层的来源:不算这一段的关联
|
||
outer = outer.outer
|
||
return None, _Source("other", qualifier, slot, (column.args["table"].copy(),))
|
||
hits = [(slot, s) for slot, s in sources.items() if self._exposes(s, column.name)]
|
||
return hits[0] if len(hits) == 1 else (None, None)
|
||
|
||
@staticmethod
|
||
def _depth(column: Any, sources: dict[str, _Source]) -> int:
|
||
"""列是往外第几层那段的来源的(相关子查询引用外层的别名 o.id 是 1);这一段的、认不出的是 0。
|
||
|
||
比较用的写法里外层的列另带「^」(orders."id^"):相关子查询读的表和外层是同一张时(orders 里找子单、父单),
|
||
c.parent_id = o.id 和 c.id = o.parent_id 不带标记都成了 orders.id = orders.parent_id,分不出来。标记按列所在
|
||
那一段算,从哪一段开始写都一样。"""
|
||
qualifier = column.table
|
||
if not qualifier:
|
||
return 0
|
||
slot = name_key(qualifier)
|
||
if slot in sources:
|
||
return 0
|
||
outer, depth = getattr(sources, "outer", None), 1
|
||
while outer is not None:
|
||
if slot in outer:
|
||
return depth
|
||
outer, depth = outer.outer, depth + 1
|
||
return 0
|
||
|
||
def _pass(self, source: _Source | None, column: str) -> tuple[_Source, str] | None:
|
||
"""派生来源只读一个来源、把它的这一列原样透出来(直接选出这一列,或者 SELECT *)时,里面一层的 (来源, 列名);
|
||
不是原样透出(算出来的列、读好几个来源)返回 None。先认写了名字的列:SELECT *, amount * 0.9 AS net 里的 net
|
||
是算出来的,不是里面那张表的 net。"""
|
||
from sqlglot import exp
|
||
|
||
if source is None or source.kind != "derived" or not isinstance(source.query, exp.Select):
|
||
return None
|
||
inner = self.sources(source.query)
|
||
if len(inner) != 1:
|
||
return None
|
||
only = next(iter(inner.values()))
|
||
names = self.names(source.query)
|
||
named = [item for item, called in zip(source.query.selects, names)
|
||
if called is not None and name_key(called) == name_key(column)]
|
||
if named:
|
||
target = _unparen(named[0].this if isinstance(named[0], exp.Alias) else named[0])
|
||
return (only, target.name) if isinstance(target, exp.Column) else None
|
||
return (only, column) if None in names else None
|
||
|
||
def names(self, query: Any) -> list[str | None]:
|
||
"""派生来源选出来的各列叫什么(见 _output_names),算一次记下来。"""
|
||
if id(query) not in self._names_memo:
|
||
self._names_memo[id(query)] = _output_names(query)
|
||
return self._names_memo[id(query)]
|
||
|
||
def _blocks(self, select: Any, column: str) -> str | None:
|
||
"""见 _blocking,算一次记下来。"""
|
||
memo = (id(select), name_key(column))
|
||
if memo not in self._blocks_memo:
|
||
self._blocks_memo[memo] = self._blocking(select, column)
|
||
return self._blocks_memo[memo]
|
||
|
||
@staticmethod
|
||
def _blocking(select: Any, column: str) -> str | None:
|
||
"""在这一段外面按 column 筛和在它里面筛结果不同时,说它做了什么(「取前几行」「窗口函数」);一样返回 None。
|
||
|
||
取前几行(LIMIT、OFFSET):先取前 10 行再筛已付,和先筛已付再取前 10 行,取到的不是同一批。窗口函数(含
|
||
DISTINCT ON):先算每人的第一单再筛已付(第一单就是已付的人),和先筛已付再算(有已付订单的人)不是一回事;
|
||
只有 column 是每个窗口的分区键时一样(按分区键筛掉的是整个分区,分区里的排名、合计不受影响)。"""
|
||
from sqlglot import exp
|
||
|
||
if select.args.get("limit") is not None or select.args.get("offset") is not None:
|
||
return "limit"
|
||
distinct = select.args.get("distinct")
|
||
if isinstance(distinct, exp.Distinct) and distinct.args.get("on") is not None:
|
||
keys = {name_key(_unparen(e).name) for e in distinct.args["on"].expressions
|
||
if isinstance(_unparen(e), exp.Column)}
|
||
if name_key(column) not in keys:
|
||
return "window"
|
||
for node in _own_nodes(select):
|
||
if isinstance(node, exp.Window):
|
||
keys = {name_key(_unparen(e).name) for e in node.args.get("partition_by") or []
|
||
if isinstance(_unparen(e), exp.Column)}
|
||
if name_key(column) not in keys:
|
||
return "window"
|
||
return None
|
||
|
||
def through(self, source: _Source | None, column: str, *, pierce: bool = False,
|
||
unseal: bool = False) -> tuple[_Source | None, str]:
|
||
"""CTE、子查询只是把一张表的列原样透出来(SELECT *,或者直接选出这一列)时,换成那张表的列:
|
||
原流程直接查 orders、盲算先在 CTE 里筛一遍再查,算的是同一列。
|
||
|
||
这一段取前几行、或者算了窗口函数时不透过去(见 _blocks),照写这个派生来源的列:外面的 t.status = 'paid'
|
||
和写在 t 里面的 status = 'paid' 不是同一个条件,透过去就都成了 orders.status = 'paid',分不出来。pierce 时
|
||
不管这些,一直透到底:拿来认出两边是不是同一个条件、只是筛的先后不同(见 Part.pierce)。
|
||
|
||
同一段里别的来源透过去读的也是同一个来源时(opaque,见 _seal)不透过去,pierce 也不透:透过去就分不出是
|
||
哪一份了。unseal 时连这个也不管,只拿来给人指出最后读的是哪张表的哪一列(时间条件的 detail)。"""
|
||
for _ in range(4):
|
||
if source is not None and source.opaque and not unseal:
|
||
break
|
||
found = self._pass(source, column)
|
||
if found is None or (not pierce and self._blocks(source.query, found[1])):
|
||
break
|
||
source, column = found
|
||
return source, column
|
||
|
||
def resolve(self, column: Any, sources: dict[str, _Source], *, pierce: bool = False,
|
||
unseal: bool = False) -> tuple[_Source | None, str]:
|
||
return self.through(self.locate(column, sources)[1], column.name, pierce=pierce, unseal=unseal)
|
||
|
||
def _definition(self, source: _Source | None, column: str) -> list[tuple[Any, _Sources]] | None:
|
||
"""派生来源一列在里面的定义:[(定义的表达式, 它那一支的来源)],UNION 每支一项(按列的位置)。只有 SELECT *
|
||
透出来的列,定义就是里面那一支的同名列。认不出(不是派生来源、没有这一列、某一支在那个位置是 *)返回 None。"""
|
||
from sqlglot import exp
|
||
|
||
if source is None or source.kind != "derived" or source.query is None:
|
||
return None
|
||
branches = _branches(source.query)
|
||
if not branches:
|
||
return None
|
||
names = [None if called is None else name_key(called) for called in self.names(source.query)]
|
||
if name_key(column) not in names:
|
||
if None not in names:
|
||
return None
|
||
return [(exp.column(column), self.sources(branch)) for branch in branches]
|
||
position = names.index(name_key(column))
|
||
out = []
|
||
for branch in branches:
|
||
if position >= len(branch.selects) or _is_star(branch.selects[position]):
|
||
return None
|
||
item = branch.selects[position]
|
||
out.append((item.this if isinstance(item, exp.Alias) else item, self.sources(branch)))
|
||
return out
|
||
|
||
def shared(self) -> set[str]:
|
||
"""好几张表都有的列名(name_key):这条语句里出现时挂在不止一张表上(orders.order_id、refunds.order_id),
|
||
或者表结构里不止一张用到的表有它。给人看时只有这些列写表名,其余只写列名,读起来和原 SQL 一样短。"""
|
||
from sqlglot import exp
|
||
|
||
if self._shared is None:
|
||
owners: dict[str, set[str]] = {}
|
||
for select in self._alive(exp.Select):
|
||
sources = self.sources(select)
|
||
for node in _own_nodes(select):
|
||
if isinstance(node, exp.Column) and node.name and not node.is_star:
|
||
source, name = self.resolve(node, sources)
|
||
if source is not None:
|
||
owners.setdefault(name_key(name), set()).add(source.ref)
|
||
used = [t.key for t in self.tables]
|
||
self._shared = {name for name, keys in owners.items()
|
||
if len(keys) > 1 or sum(1 for t in used if name in self.columns.get(t, ())) > 1}
|
||
return self._shared
|
||
|
||
# ---- 写成 key / text
|
||
|
||
@staticmethod
|
||
def _owner(node: Any) -> Any:
|
||
"""node 往上最近的一段 SELECT(带段号的,node 本身是也算);没有就是 None:node 在调用方那一段里。"""
|
||
from sqlglot import exp
|
||
|
||
while node is not None:
|
||
if isinstance(node, exp.Select) and node.meta.get(_SEGMENT) is not None:
|
||
return node
|
||
node = node.parent
|
||
return None
|
||
|
||
def scope(self, node: Any, default: dict[str, _Source]) -> dict[str, _Source]:
|
||
"""node 所在那一段的来源。条件里套着的子查询(EXISTS、IN (SELECT …))是另一段:别名在那一段里定义,相关
|
||
子查询还要到外层找。复制出来的子查询带着段号,按段号找回原树上那一段算好的来源;不在子查询里就是 default。"""
|
||
owner = self._owner(node)
|
||
return default if owner is None else self.sources(self.segments[owner.meta[_SEGMENT]])
|
||
|
||
def render(self, node: Any, sources: dict[str, _Source], mode: str = "text") -> str:
|
||
"""七种写法。比较用的五种(_COMPARING),名字按 name_key:key、loose 见 Part,key 写实表的表名,loose 不写,
|
||
CTE、子查询都写编号;pierce 同 key,只是一路透过取前几行、算窗口函数的派生来源(见 through),拿来认出同一个
|
||
条件筛的先后不同;anon 写实表的表名、派生来源一律写「@」(排比较的两边谁在前时先分实表的列和派生来源的列);
|
||
shape 写实表的表名、派生来源写它的结构标识(见 _signature:配对用,和名字、编号都无关)。text、full 给人看
|
||
(见 Part),text 只在好几张表都有这一列时写真实表名,full 一律写(关联条件说的就是两张表各出哪一列,用 full)。
|
||
|
||
node 里套着的子查询逐段改写:每一列按它所在那一段的来源认,子查询里的表去掉别名(列已经写成了真实表名)。
|
||
子查询里拿两张表的列互相比的(相关子查询),给人看时和关联条件一样写表名;内外读的是同一个来源(orders 里找
|
||
子单)时写表名分不出谁是谁,照写别名、留着表别名。比较用的写法里外层的列在列名后面带「^」(见 _depth),各自
|
||
起的名字去掉(见 _drop_names)。
|
||
"""
|
||
from sqlglot import exp
|
||
|
||
copy = node.copy()
|
||
if mode in _COMPARING:
|
||
copy = copy.transform(_as_date, copy=False)
|
||
groups: dict[int, tuple[Any, list[Any]]] = {}
|
||
for col in copy.find_all(exp.Column):
|
||
if col.name and not col.is_star:
|
||
owner = self._owner(col)
|
||
groups.setdefault(id(owner), (owner, []))[1].append(col)
|
||
#: 给人看时照写别名的段(id):里面的列和外层的列读的是同一个来源(见下)
|
||
keep: set[int] = set()
|
||
for owner, cols in groups.values():
|
||
scope = sources if owner is None else self.sources(self.segments[owner.meta[_SEGMENT]])
|
||
found = [(col, *self.resolve(col, scope, pierce=mode == "pierce"), self._depth(col, scope))
|
||
for col in cols]
|
||
here = mode
|
||
seen = {(s.ref, depth) for _, s, _, depth in found if s is not None}
|
||
# 同一个来源既是这一段的、又是外层的(相关子查询读的表和外层是同一张):写表名分不出谁是谁,写别名
|
||
clash = {ref for ref, _ in seen if sum(1 for other, _ in seen if other == ref) > 1}
|
||
if owner is not None and mode == "text" and len(seen) > 1:
|
||
here = "full"
|
||
for col, source, name, depth in found:
|
||
alias = None
|
||
if mode not in _COMPARING and source is not None and source.ref in clash:
|
||
located = self.locate(col, scope)[1]
|
||
alias = col.args["table"].copy() if col.table else \
|
||
exp.to_identifier(located.alias) if located is not None and located.alias else None
|
||
keep.add(id(owner))
|
||
self._write(col, source, name, here, depth=depth, alias=alias)
|
||
for table in copy.find_all(exp.Table):
|
||
if isinstance(table.this, exp.Identifier) and (owner := self._owner(table)) is not None:
|
||
if id(owner) not in keep:
|
||
table.set("alias", None)
|
||
if mode in _COMPARING:
|
||
for arg in ("catalog", "db", "this"):
|
||
ident = table.args.get(arg)
|
||
if isinstance(ident, exp.Identifier):
|
||
table.set(arg, exp.to_identifier(name_key(ident.name)))
|
||
if mode in _COMPARING:
|
||
self._drop_names(copy, mode)
|
||
return copy.sql(dialect=self.read, comments=False)
|
||
|
||
def _drop_names(self, copy: Any, mode: str) -> None:
|
||
"""比较用的写法里抹掉套着的子查询里各自起的名字(就地改):引用 CTE 的表、FROM 里的子查询(含 LATERAL)、
|
||
子查询里自带的 CTE 定义写编号(anon 写「@」,shape 里引用 CTE 的表写它的结构标识、别的写「@」;引用它们的列
|
||
已经照这样写了);只看取值的子查询(IN、EXISTS、标量子查询)去掉选出来的列名。FROM 里的子查询、CTE 定义选出来
|
||
的列名留着:外层按这个名字引用它们,去掉了就分不出外层用的是哪一列。复制出来的那段本身没有父节点,认不出是
|
||
FROM 里的子查询,它选出来的列名也会去掉。"""
|
||
from sqlglot import exp
|
||
|
||
for node in copy.find_all(exp.Table, exp.Subquery, exp.Lateral, exp.CTE):
|
||
derived = node.meta.get(_DERIVED)
|
||
if not derived:
|
||
continue
|
||
if mode == "shape":
|
||
# 引用 CTE 的表写它是哪个(结构标识);FROM 里的子查询、CTE 定义本身整段写在这里,不用再标
|
||
mark = exp.to_identifier(self._signature_of(derived) if isinstance(node, exp.Table) else _ANON)
|
||
else:
|
||
mark = exp.to_identifier(_ANON if mode == "anon" else derived)
|
||
if isinstance(node, exp.Table):
|
||
for arg in ("catalog", "db", "alias"):
|
||
node.set(arg, None)
|
||
node.set("this", mark)
|
||
elif isinstance(node.args.get("alias"), exp.TableAlias):
|
||
node.args["alias"].set("this", mark)
|
||
else:
|
||
node.set("alias", exp.TableAlias(this=mark))
|
||
for select in copy.find_all(exp.Select):
|
||
if not _named_output(select):
|
||
select.set("expressions", [e.this if isinstance(e, exp.Alias) else e for e in select.expressions])
|
||
|
||
def _write(self, col: Any, source: _Source | None, name: str, mode: str, *, depth: int = 0,
|
||
alias: Any = None) -> None:
|
||
"""按 render 的写法改写一列(就地改)。depth 是它属于往外第几层的来源(见 _depth):比较用的写法里另带
|
||
「^」。alias 是给人看时照写的别名(内外层读同一个来源时,见 render):这时列名也照原样写,不写透过去的。"""
|
||
from sqlglot import exp
|
||
|
||
quoted = isinstance(col.this, exp.Identifier) and bool(col.this.args.get("quoted"))
|
||
if mode in _COMPARING:
|
||
if source is None:
|
||
owner = None
|
||
elif source.kind == "derived":
|
||
owner = _ANON if mode == "anon" else source.ref if mode != "shape" else \
|
||
self._signature_of(source.key) + (f"#{source.role}" if source.role else "")
|
||
else:
|
||
owner = None if mode == "loose" else source.ref
|
||
# 外层的列在列名后面带标记(不带在表名上:表名带了标记要加引号,排比较的两边时外层的列就都排到前面去了,
|
||
# 只该在两边表名一样时才影响先后)
|
||
column = self.column_key(source, name) if source is not None else name_key(name)
|
||
col.set("this", exp.to_identifier(column + "^" * depth))
|
||
col.set("table", exp.to_identifier(owner) if owner else None)
|
||
col.set("db", None)
|
||
col.set("catalog", None)
|
||
return
|
||
if alias is not None:
|
||
col.set("table", alias)
|
||
col.set("db", None)
|
||
col.set("catalog", None)
|
||
return
|
||
col.set("this", exp.Identifier(this=name, quoted=quoted or not _PLAIN_NAME.match(name)))
|
||
# 自己关联自己的(有 role)一律写别名:只写列名读不出是哪一个。不透过去的派生来源(opaque)不算「整条语句只用到
|
||
# 一张实表」:两个 CTE 都读 orders,好几个都有的列(shared)得写出是哪一个的
|
||
if source is None or (source.kind != "other" and mode != "full" and not source.role
|
||
and ((self.single and not source.opaque) or name_key(name) not in self.shared())):
|
||
col.set("table", None)
|
||
col.set("db", None)
|
||
col.set("catalog", None)
|
||
else:
|
||
*heads, last = source.idents or (exp.to_identifier(source.name),)
|
||
col.set("table", last.copy())
|
||
col.set("db", heads[-1].copy() if heads else None)
|
||
col.set("catalog", heads[-2].copy() if len(heads) > 1 else None)
|
||
|
||
def canonical(self, node: Any, sources: dict[str, _Source], *, by: str = "key") -> Any:
|
||
"""比较的两边排好顺序(列在左、字面量在右;两边都有列的按 _rank 排),IN 的取值排好序;加法、乘法的两项也按
|
||
这个规矩排(amount * 0.9 和 0.9 * amount 是同一个算法),只排两项都不是别的加减乘除的(见 _atomic)。返回新的
|
||
树。套着的子查询里的比较按它那一段的来源排:拿外层的来源去认子查询的别名,排出来的先后就取决于别名怎么起。"""
|
||
from sqlglot import exp
|
||
|
||
def fix(n: Any) -> Any:
|
||
found = _COMPARE.get(type(n).__name__)
|
||
swap = (type(n).__name__, type(n).__name__) if isinstance(n, (exp.Add, exp.Mul)) and \
|
||
_atomic(n.this) and _atomic(n.expression) else found
|
||
if swap is not None and isinstance(n, exp.Binary):
|
||
left, right = n.this, n.expression
|
||
lc, rc = bool(_own_columns(left)), bool(_own_columns(right))
|
||
here = self.scope(n, sources) if lc and rc else sources
|
||
if (rc and not lc) or (lc and rc and self._rank(left, here, by) > self._rank(right, here, by)):
|
||
return getattr(exp, swap[1])(this=right, expression=left)
|
||
elif isinstance(n, exp.In) and n.expressions and not n.args.get("query") \
|
||
and not any(_own_columns(e) for e in n.expressions):
|
||
n.set("expressions", sorted(n.expressions, key=lambda e: e.sql(dialect=self.read)))
|
||
return n
|
||
|
||
return node.transform(fix)
|
||
|
||
def _rank(self, node: Any, sources: dict[str, _Source], by: str = "key") -> tuple[str, str]:
|
||
"""比较的两边谁排前面:先按不写编号的写法(实表的列、派生来源的列分得开),分不出(两个派生来源的同名列)
|
||
再按 by 的写法分。
|
||
|
||
by 是 key 时按位置编号排;算结构标识时 by 是 shape,按结构标识排:位置编号决定不了先后。"""
|
||
return self.render(node, sources, "anon"), self.render(node, sources, by)
|
||
|
||
def part(self, node: Any, sources: dict[str, _Source], *, excludes: bool = False) -> Part:
|
||
#: 用到的各列 {(哪一个来源, 列): ((给人看的表名, 列), 比较用的写法)}:按「哪一个来源」分,自己关联自己的两个别名
|
||
#: 是两列(o1.amount、o2.amount)
|
||
refs: dict[tuple[str, str], tuple[ColumnRef, str]] = {}
|
||
origins: list[tuple[str, str]] = []
|
||
roles: list[tuple[str, str, str]] = []
|
||
stage = ""
|
||
for col in _own_columns(node):
|
||
located = self.locate(col, sources)[1]
|
||
# 外层的列另带标记(见 _depth):相关子查询里 c.amount 和外层的 o.amount 是两列
|
||
mark = "^" * self._depth(col, sources)
|
||
source, name = self.through(located, col.name)
|
||
refs.setdefault((source.ref if source is not None else "", name_key(name) + mark),
|
||
((source.name if source is not None else None, name),
|
||
f"{source.ref}.{self.column_key(source, name)}{mark}" if source is not None
|
||
else f".{name_key(name)}{mark}"))
|
||
if source is not None and source.role and source.idents:
|
||
if (note := (source.idents[-1].name, source.role, source.name)) not in roles:
|
||
roles.append(note)
|
||
if source is None or source.kind != "derived":
|
||
continue
|
||
# 透视在这个派生来源上停住了,一路透到底却还能往里走:是它取前几行、算窗口函数挡住的
|
||
if not stage and self.through(located, col.name, pierce=True)[0] is not source:
|
||
stage = self._stage(source, name)
|
||
if (origin := self._origin(source, name)) is not None:
|
||
if (pair := (self.render(col, sources, "full"), origin)) not in origins:
|
||
origins.append(pair)
|
||
full = self.render(node, sources, "full")
|
||
key = self.render(node, sources, "key")
|
||
pierce = self.render(node, sources, "pierce") if stage else ""
|
||
if pierce == key:
|
||
pierce = stage = ""
|
||
columns = tuple(ref for ref, _ in refs.values())
|
||
# 拿两张表的列互相比的(相关子查询里的 o.id = r.order_id)和关联条件一样,给人看也写表名
|
||
across = len({table for table, _ in columns}) > 1
|
||
return Part(key=key, text=full if across else self.render(node, sources),
|
||
columns=columns, excludes=self._excludes(node, sources) if excludes else None,
|
||
loose=self.render(node, sources, "loose"), full=full,
|
||
column_keys=tuple(written for _, written in refs.values()),
|
||
origins=tuple(origins), pierce=pierce, stage=stage, roles=tuple(roles))
|
||
|
||
def _stage(self, source: _Source, column: str) -> str:
|
||
"""挡住了透视的派生来源做了什么(给人看,见 _blocks):「取 t 的前 10 行」「算 t 的窗口函数」。"""
|
||
from sqlglot import exp
|
||
|
||
query = source.query
|
||
if self._blocks(query, column) == "limit":
|
||
limit = query.args.get("limit")
|
||
count = limit.expression if isinstance(limit, exp.Limit) else limit.args.get("count") \
|
||
if limit is not None else None
|
||
written = count.sql(dialect=self.read, comments=False) if count is not None else ""
|
||
return _say("取", source.name, "的前", written, "行") if written.isdigit() else \
|
||
_say("按", "LIMIT / OFFSET", "取", source.name, "的行")
|
||
return _say("算", source.name, "的窗口函数")
|
||
|
||
def _origin(self, source: _Source, column: str, depth: int = 0) -> str | None:
|
||
"""派生来源一列的来历(给人看):它在里面怎么算的,里面再用到的派生来源的列也换成各自的来历,一直换到实表的
|
||
列,一律写表名(SUM(orders.amount))。两边同名的派生来源其实不是同一个时(盲算的 t1 是原流程的 t2),光写
|
||
「t1.total」两边一样,带上来历才说得清各自是付款合计还是退款合计。UNION 只写第一支;认不出定义、套得太深
|
||
返回 None。"""
|
||
if source.kind != "derived" or source.query is None or depth > 4:
|
||
return None
|
||
memo = (id(source.query), name_key(column))
|
||
if memo not in self._origins:
|
||
self._origins[memo] = None # 递归 CTE 绕回自己时到这里为止
|
||
found = self._definition(source, column)
|
||
self._origins[memo] = self._expand(*found[0], depth + 1) if found else None
|
||
return self._origins[memo]
|
||
|
||
def _expand(self, node: Any, sources: dict[str, _Source], depth: int) -> str:
|
||
"""把 node 写成给人看的 full 写法,里面用到的派生来源的列换成各自的来历(见 _origin)。换进去的是一段
|
||
写好的文字:先用占位符占住位置,整段写好之后再换回来;放在运算符里的加括号,免得优先级变了。"""
|
||
from sqlglot import exp
|
||
|
||
bare = _unparen(node)
|
||
if isinstance(bare, exp.Column):
|
||
source, name = self.resolve(bare, sources)
|
||
inner = self._origin(source, name, depth) if source is not None else None
|
||
return inner if inner is not None else self.render(bare, sources, "full")
|
||
copy = node.copy()
|
||
subs: dict[str, str] = {}
|
||
for col in _own_columns(copy):
|
||
source, name = self.resolve(col, sources)
|
||
inner = self._origin(source, name, depth) if source is not None else None
|
||
if inner is None:
|
||
continue
|
||
mark = exp.Placeholder(this=f"sqlsem{len(subs)}")
|
||
wrap = " " in inner and isinstance(col.parent, (exp.Binary, exp.Unary)) and \
|
||
not isinstance(col.parent, exp.Paren)
|
||
subs[mark.sql(dialect=self.read)] = f"({inner})" if wrap else inner
|
||
col.replace(mark)
|
||
text = self.render(copy, sources, "full")
|
||
for mark, inner in subs.items():
|
||
text = text.replace(mark, inner)
|
||
return text
|
||
|
||
def _excludes(self, node: Any, sources: dict[str, _Source]) -> str | None:
|
||
"""排除写法去掉否定之后排除的是什么:a <> 1 → a = 1;NOT IN、NOT LIKE、NOT BETWEEN 同理;IS NOT NULL → 「a 为空」。
|
||
各方言把否定放的位置不一样(NOT 包一层,或者节点上带 negate),两种都认。"""
|
||
from sqlglot import exp
|
||
|
||
n = _unparen(node)
|
||
if isinstance(n, exp.NEQ):
|
||
return self.render(exp.EQ(this=n.this.copy(), expression=n.expression.copy()), sources)
|
||
inner = None
|
||
if isinstance(n, exp.Not):
|
||
inner = _unparen(n.this).copy()
|
||
elif n.args.get("negate"):
|
||
inner = n.copy()
|
||
inner.set("negate", None)
|
||
if inner is None:
|
||
return None
|
||
if isinstance(inner, exp.Is) and isinstance(_unparen(inner.expression), exp.Null):
|
||
return f"{self.render(inner.this, sources)} 为空"
|
||
return self.render(inner, sources)
|
||
|
||
# ---- 各个方面
|
||
|
||
def facets(self, extra: Iterable[str] = ()) -> Facets:
|
||
"""各个方面。extra 是没用到、也照样收进来的 CTE(name_key),别的没用到的 CTE 不收(见 _enliven)。
|
||
条件、分组键、连接、聚合、HAVING 都记下在哪一层(见 _layers),按(层, key)去重。"""
|
||
from sqlglot import exp
|
||
|
||
self._enliven(extra)
|
||
tags = self._layers(extra)
|
||
joins: list[JoinFacet] = []
|
||
where: list[Part] = []
|
||
time: list[TimePredicate] = []
|
||
group: list[Part] = []
|
||
distinct: list[str] = []
|
||
aggregates: list[Aggregate] = []
|
||
having: list[Part] = []
|
||
used: list[Part] = []
|
||
for select in self._alive(exp.Select):
|
||
sources = self.sources(select)
|
||
# 没走到的段(条件、SELECT 列表里的子查询,以及只在它们里面用到的派生来源)在「?」那一层
|
||
layer = tags.get(id(select), "?")
|
||
if select is self.root or not self._in_condition(select):
|
||
used += [Part.plain(s.ref, s.name) for s in sources.values()]
|
||
drafts = self._join_drafts(select, sources)
|
||
clause = select.args.get("where")
|
||
for cond in _conjuncts(clause.this) if clause is not None else []:
|
||
canon = self.canonical(cond, sources)
|
||
if self._joins_on_where(canon, sources, drafts):
|
||
continue
|
||
part = replace(self.part(canon, sources, excludes=True), scope=layer)
|
||
where.append(part)
|
||
self._rejects(canon, part, sources, drafts)
|
||
if found := self._time(canon, sources):
|
||
time.append(replace(found, condition=replace(found.condition, scope=layer)))
|
||
joins += [replace(self._join(d, sources), scope=layer) for d in drafts]
|
||
keys = self._group(select, sources)
|
||
if not keys and (keys := self._distinct_keys(select, sources)):
|
||
distinct.append(layer)
|
||
group += [replace(p, scope=layer) for p in keys]
|
||
aggregates += [replace(g, scope=layer) for g in self._aggregates(select, sources)]
|
||
having += [replace(p, scope=layer) for p in self._having(select, sources)]
|
||
derived = [Part.plain(self._labels[number], name or "子查询")
|
||
for number, name, body, _ in self._derived if id(body) not in self._dead]
|
||
return Facets(
|
||
dialect=self.read,
|
||
tables=tuple(Part.plain(s.key, s.name) for s in self.tables),
|
||
joins=_dedupe(joins, lambda j: (j.scope, j.kind, j.table.key, j.on.key if j.on else None)),
|
||
where=_dedupe(where, lambda p: (p.scope, p.key)),
|
||
time=_dedupe(time, lambda t: (t.condition.scope, t.condition.key)),
|
||
group_by=_dedupe(group, lambda p: (p.scope, p.key)),
|
||
aggregates=_dedupe(aggregates, lambda a: (a.scope, a.func, a.arg.key, a.distinct)),
|
||
having=_dedupe(having, lambda p: (p.scope, p.key)),
|
||
limit=self._limit(),
|
||
limits=tuple(self._limits()),
|
||
sources=_dedupe(used, lambda p: p.key),
|
||
expressions=_dedupe(self._expressions(), lambda c: (c.scope_key, c.name, c.expr.key)),
|
||
derived=_dedupe(derived, lambda p: p.key),
|
||
set_ops=tuple(self._set_ops(tags)),
|
||
distinct=tuple(dict.fromkeys(distinct)),
|
||
grain=self._grain(self.root),
|
||
scopes=self._scope_parts(tags),
|
||
)
|
||
|
||
# ---- 层
|
||
|
||
def _layers(self, extra: Iterable[str] = ()) -> dict[int, str]:
|
||
"""每段 SELECT 在哪一层:{段的 id: 层的标记}(见 Part.scope)。
|
||
|
||
最外层是空串,UNION 的第 k 支(k ≥ 2)在后面加「|k」。FROM / JOIN 里的派生来源(CTE、子查询、LATERAL 子查询)
|
||
是它自己的位置编号;只从一个来源里筛出一部分行的(见 _transparent)
|
||
和用到它的那一段算同一层:写在这种 CTE 里的条件和写在外层是一回事。读同一个来源的几个派生来源同在一段时
|
||
(见 _seal)各算各的层:两个 CTE 里各有一个 is_deleted = 0 是两个条件。派生来源在哪一层按在 FROM / JOIN 里
|
||
第一次用到它的地方定,从外往里一层一层地走。没走到的段(条件、SELECT 列表里的子查询,以及只在它们里面用到
|
||
的派生来源)由 facets 记成「?」。extra 是没用到、也照样收进来的 CTE,在它自己的编号那一层。"""
|
||
from sqlglot import exp
|
||
|
||
tags: dict[int, str] = {}
|
||
queue: deque[tuple[Any, str]] = deque()
|
||
|
||
def push(query: Any, tag: str) -> None:
|
||
for i, branch in enumerate(_branches(query)):
|
||
queue.append((branch, tag if i == 0 else f"{tag}|{i + 1}"))
|
||
|
||
def drain() -> None:
|
||
while queue:
|
||
select, tag = queue.popleft()
|
||
if id(select) in tags:
|
||
continue
|
||
tags[id(select)] = tag
|
||
sources = self.sources(select)
|
||
for i, item in enumerate(self._items(select)):
|
||
body, label = _derived_body(item), item.meta.get(_DERIVED)
|
||
if body is None and isinstance(item, exp.Table) and isinstance(item.this, exp.Identifier) \
|
||
and item.name and self._is_cte(item):
|
||
body = self.ctes[name_key(item.name)]
|
||
if body is None or not label:
|
||
continue
|
||
alias = _alias(item)
|
||
source = sources.get(name_key(alias) if alias else f"#{i}")
|
||
lifted = source is not None and not source.opaque and self._transparent(item, body)
|
||
push(body, tag if lifted else label)
|
||
|
||
push(self.root, "")
|
||
drain()
|
||
for name in extra:
|
||
if name in self._cte_defs and (label := self._cte_defs[name].meta.get(_DERIVED)):
|
||
push(self.ctes[name], label)
|
||
drain()
|
||
return tags
|
||
|
||
def _transparent(self, item: Any, body: Any) -> bool:
|
||
"""派生来源只是从一个来源里筛出一部分行(不分组、不聚合、不去重、不是 UNION,也不是 LATERAL):写在它里面的
|
||
条件和写在用到它的那一段是一回事,算同一层(见 _layers)。各列怎么算不影响有哪些行;取前几行、算窗口函数的
|
||
也算同一层,筛的先后另由 Part.pierce / stage 说。"""
|
||
from sqlglot import exp
|
||
|
||
if isinstance(item, exp.Lateral) or not isinstance(body, exp.Select) or len(self._items(body)) != 1:
|
||
return False
|
||
if any(body.args.get(k) is not None for k in ("group", "having", "distinct")):
|
||
return False
|
||
return not self._aggregating(body)
|
||
|
||
@staticmethod
|
||
def _aggregating(select: Any) -> bool:
|
||
"""这一段自己算了聚合(窗口函数、套着的子查询里的不算)。"""
|
||
from sqlglot import exp
|
||
|
||
return any(isinstance(n, exp.AggFunc) and not isinstance(n.parent, exp.Window) for n in _own_nodes(select))
|
||
|
||
def _scope_parts(self, tags: Mapping[int, str]) -> tuple[Part, ...]:
|
||
"""各层给人看的说法(Part.plain(层的标记, 说法)):最外层「外层」,派生来源写它的名字,UNION 的各支写「t 的
|
||
UNION 第 2 支」,条件里的子查询「条件的子查询」。"""
|
||
names = {self._labels[number]: name or "子查询" for number, name, _, _ in self._derived}
|
||
unions = {self._labels[number] for number, _, body, _ in self._derived if len(_branches(body)) > 1}
|
||
root_union = len(_branches(self.root)) > 1
|
||
out: dict[str, Part] = {}
|
||
for tag in dict.fromkeys([*tags.values(), "?"]):
|
||
head, _, branch = tag.partition("|")
|
||
if head == "?":
|
||
text = "条件的子查询"
|
||
elif head == "":
|
||
text = f"UNION 第 {branch or 1} 支" if root_union else "外层"
|
||
else:
|
||
name = names.get(head, "子查询")
|
||
text = _say(name, "的 UNION 第", branch or "1", "支") if head in unions else name
|
||
out[tag] = Part.plain(tag, text)
|
||
return tuple(out.values())
|
||
|
||
def _set_ops(self, tags: Mapping[int, str]) -> list[SetOpFacet]:
|
||
"""各段的 UNION、INTERSECT、EXCEPT(见 SetOpFacet),条件里的子查询里的不算。"""
|
||
from sqlglot import exp
|
||
|
||
out = []
|
||
for node in self._alive(exp.SetOperation):
|
||
parent = node.parent
|
||
while isinstance(parent, exp.Subquery):
|
||
parent = parent.parent
|
||
branches = _branches(node)
|
||
if isinstance(parent, exp.SetOperation) or not branches:
|
||
continue # 只看一串合并最上面的那一个
|
||
tag = tags.get(id(branches[0]), "?")
|
||
if tag != "?":
|
||
out.append(SetOpFacet(ops=tuple(self._ops(node)), branches=len(branches),
|
||
scope=self._scope_name(node), scope_key=tag))
|
||
return out
|
||
|
||
@classmethod
|
||
def _ops(cls, node: Any) -> list[str]:
|
||
"""一串合并从左到右各处的写法:UNION、UNION ALL、INTERSECT、EXCEPT ALL……"""
|
||
from sqlglot import exp
|
||
|
||
while isinstance(node, exp.Subquery):
|
||
node = node.this
|
||
if not isinstance(node, exp.SetOperation):
|
||
return []
|
||
word = {"Union": "UNION", "Intersect": "INTERSECT", "Except": "EXCEPT"}.get(type(node).__name__,
|
||
type(node).__name__.upper())
|
||
if not node.args.get("distinct"):
|
||
word += " ALL"
|
||
return cls._ops(node.this) + [word] + cls._ops(node.expression)
|
||
|
||
def _distinct_keys(self, select: Any, sources: dict[str, _Source]) -> list[Part]:
|
||
"""SELECT DISTINCT(不带 ON)选出来的各项。没有 GROUP BY、没有聚合时,对这几项去重和按它们分组出的行一模一样,
|
||
当这一段的分组键收(见 Facets.distinct):SELECT DISTINCT user_id 和 GROUP BY user_id 不算不同,一边在子查询里
|
||
先去重、另一边没有照报。DISTINCT ON 是每组取一行,按窗口函数算(见 _blocking)。"""
|
||
from sqlglot import exp
|
||
|
||
distinct = select.args.get("distinct")
|
||
if not isinstance(distinct, exp.Distinct) or distinct.args.get("on") is not None \
|
||
or select.args.get("group") is not None or self._aggregating(select):
|
||
return []
|
||
out = []
|
||
for item in select.selects:
|
||
if _is_star(item):
|
||
out.append(Part.plain("*", "*"))
|
||
continue
|
||
target = item.this if isinstance(item, exp.Alias) else item
|
||
out.append(self.part(self.canonical(target, sources), sources))
|
||
return out
|
||
|
||
def _grain(self, query: Any, depth: int = 0) -> Part | None:
|
||
"""最后出来的行按什么分:分组(GROUP BY、当分组用的 DISTINCT)是「group:各分组键」,没分组却算了聚合是
|
||
「one」(只出一行),都没有、只从一个派生来源取数的看那个派生来源,别的是「rows」(逐行)。最外层是 UNION
|
||
的不算(返回 None)。分组键挪了一层(先在子查询里按 user_id 分组、外层数用户数,对外层按 user_id 分组)按
|
||
整条语句比分组键看不出来:外层出一行还是每个用户一行,看它。"""
|
||
from sqlglot import exp
|
||
|
||
while isinstance(query, exp.Subquery):
|
||
query = query.this
|
||
if not isinstance(query, exp.Select) or depth > 6:
|
||
return None
|
||
sources = self.sources(query)
|
||
keys = self._group(query, sources)
|
||
if not keys and self._aggregating(query):
|
||
return Part.plain("one", "")
|
||
keys = keys or self._distinct_keys(query, sources)
|
||
if keys:
|
||
return Part(key="group:" + ", ".join(sorted({p.key for p in keys})),
|
||
text="、".join(dict.fromkeys(p.text for p in keys)),
|
||
loose="group:" + ", ".join(sorted({p.loose for p in keys})),
|
||
full="、".join(dict.fromkeys(p.full for p in keys)))
|
||
items = self._items(query)
|
||
if len(items) == 1 and not isinstance(items[0], exp.Lateral):
|
||
item, body = items[0], _derived_body(items[0])
|
||
if body is None and isinstance(item, exp.Table) and isinstance(item.this, exp.Identifier) \
|
||
and item.name and self._is_cte(item):
|
||
body = self.ctes[name_key(item.name)]
|
||
if body is not None:
|
||
return self._grain(body, depth + 1)
|
||
return Part.plain("rows", "")
|
||
|
||
def _rejects(self, cond: Any, part: Part, sources: dict[str, _Source], drafts: list[dict[str, Any]]) -> None:
|
||
"""同一段 WHERE 里的条件把外连接可空那一侧没关联上的行筛掉了的,记在那次连接上(见 JoinFacet.rejects):LEFT
|
||
JOIN 关联进来的那一侧、RIGHT JOIN 之前的各个来源可空,那一侧的列直接拿来比较、IN、LIKE、BETWEEN,或者写 IS NOT
|
||
NULL,空值都不成立。IS NULL、OR、套在 COALESCE / IFNULL / CASE 里的不认(宁可漏报)。"""
|
||
from sqlglot import exp
|
||
|
||
n = _unparen(cond)
|
||
if isinstance(n, exp.Not):
|
||
n = _unparen(n.this)
|
||
if isinstance(n, exp.Is):
|
||
if not isinstance(_unparen(n.expression), exp.Null):
|
||
return
|
||
n = n.this # IS NOT NULL
|
||
elif type(n).__name__ not in _NULL_REJECTING:
|
||
return
|
||
elif type(n).__name__ not in _NULL_REJECTING:
|
||
return
|
||
hit = set()
|
||
for col in _own_columns(n):
|
||
if col is not n and any(isinstance(p, (exp.Coalesce, exp.Case, exp.If)) for p in _parents(col, stop=n)):
|
||
continue
|
||
if (slot := self.locate(col, sources)[0]) is not None:
|
||
hit.add(slot)
|
||
for draft in drafts:
|
||
nullable = {draft["slot"]} if draft["kind"] == "left" else \
|
||
set(draft["before_slots"]) if draft["kind"] == "right" else set()
|
||
if draft.get("rejects") is None and hit & nullable:
|
||
draft["rejects"] = part
|
||
|
||
def _join_drafts(self, select: Any, sources: dict[str, _Source]) -> list[dict[str, Any]]:
|
||
from sqlglot import exp
|
||
|
||
first = select.args.get("from_")
|
||
first_alias = _alias(first.this) if first is not None else ""
|
||
drafts: list[dict[str, Any]] = []
|
||
for i, join in enumerate(select.args.get("joins") or []):
|
||
alias = _alias(join.this)
|
||
slot = name_key(alias) if alias else f"#{i + 1}"
|
||
conds: list[Any] = []
|
||
if join.args.get("on") is not None:
|
||
conds = [self.canonical(c, sources) for c in _conjuncts(join.args["on"])]
|
||
elif join.args.get("using") and first_alias and alias:
|
||
# USING (id):按「FROM 的那张表.id = 这张表.id」写,和 ON 的写法放在一起比
|
||
conds = [self.canonical(exp.EQ(this=exp.column(ident.name, table=first_alias),
|
||
expression=exp.column(ident.name, table=alias)), sources)
|
||
for ident in join.args["using"]]
|
||
before = [(key, s) for key, s in list(sources.items())[:i + 1] if key != slot]
|
||
drafts.append({"kind": _join_kind(join), "slot": slot, "source": sources.get(slot), "conds": conds,
|
||
"before": [s.name for _, s in before], "before_keys": [s.ref for _, s in before],
|
||
"before_slots": [key for key, _ in before]})
|
||
return drafts
|
||
|
||
def _joins_on_where(self, cond: Any, sources: dict[str, _Source], drafts: list[dict[str, Any]]) -> bool:
|
||
"""WHERE 里两张表的列等式(逗号连接、或者 INNER JOIN 之外又写的等值关联)并进那次连接的条件,不算筛选。
|
||
LEFT JOIN 这类外连接不并:写在 WHERE 里会把外连接变成内连接,和写在 ON 里不是一回事。"""
|
||
from sqlglot import exp
|
||
|
||
n = _unparen(cond)
|
||
if not isinstance(n, exp.EQ):
|
||
return False
|
||
left, right = _unparen(n.this), _unparen(n.expression)
|
||
if not (isinstance(left, exp.Column) and isinstance(right, exp.Column)):
|
||
return False
|
||
a, b = self.locate(left, sources)[0], self.locate(right, sources)[0]
|
||
if a is None or b is None or a == b:
|
||
return False
|
||
for draft in reversed(drafts):
|
||
if draft["slot"] in (a, b) and draft["kind"] in ("inner", "cross"):
|
||
draft["conds"].append(cond)
|
||
draft["kind"] = "inner"
|
||
return True
|
||
return False
|
||
|
||
def _join(self, draft: dict[str, Any], sources: dict[str, _Source]) -> JoinFacet:
|
||
source = draft["source"]
|
||
table = Part.plain("?", "?") if source is None else \
|
||
Part(key=source.ref, text=source.name, loose=source.ref, full=source.name)
|
||
forms = sorted((self.render(c, sources, "key"), self.render(c, sources, "loose"),
|
||
self.render(c, sources, "full")) for c in draft["conds"])
|
||
on = Part(key=" AND ".join(f[0] for f in forms), text=" AND ".join(f[2] for f in forms),
|
||
loose=" AND ".join(sorted(f[1] for f in forms)), full=" AND ".join(f[2] for f in forms)) \
|
||
if forms else None
|
||
mentioned = {table for c in draft["conds"] for table, _ in self.part(c, sources).columns}
|
||
other = [name for name in dict.fromkeys(t for t in mentioned if t) if name != table.text] if draft["conds"] \
|
||
else draft["before"]
|
||
# 涉及的来源按条件里直接引用的算(slot 不为 None 的是这一段的来源):透过 CTE 找到的实表两边可能一样,
|
||
# 关联的却是两个不同的 CTE
|
||
sides = {source.ref} if source is not None else set()
|
||
if draft["conds"]:
|
||
sides |= {found.ref for c in draft["conds"] for col in _own_columns(c)
|
||
for slot, found in [self.locate(col, sources)] if slot is not None and found is not None}
|
||
else:
|
||
sides |= set(draft["before_keys"])
|
||
# 外连接的主表、可空那一侧:LEFT JOIN 关联进来的那一侧可空,RIGHT JOIN 反过来;另一侧按关联条件里出现的来源
|
||
# 算(没有就是它之前的各个来源),和内连接的 sides 一样不受前面别的连接先后的影响
|
||
keep: set[str] = set()
|
||
nullable: set[str] = set()
|
||
if draft["kind"] in ("left", "right") and source is not None:
|
||
near, far = {source.ref}, (sides - {source.ref}) or set(draft["before_keys"])
|
||
keep, nullable = (far, near) if draft["kind"] == "left" else (near, far)
|
||
return JoinFacet(kind=draft["kind"], table=table, on=on, other=tuple(sorted(other)),
|
||
sides=tuple(sorted(sides)), keep=tuple(sorted(keep)), nullable=tuple(sorted(nullable)),
|
||
rejects=draft.get("rejects"))
|
||
|
||
def _time(self, cond: Any, sources: dict[str, _Source]) -> TimePredicate | None:
|
||
"""时间条件:一边恰好一列(可以套着日期函数、CAST),另一边没有列;列名像日期,或者用了日期函数、像日期的字符串。"""
|
||
from sqlglot import exp
|
||
|
||
n = _unparen(cond)
|
||
if n.args.get("negate"):
|
||
return None
|
||
if isinstance(n, exp.Between):
|
||
subject, values, op = n.this, [n.args.get("low"), n.args.get("high")], "between"
|
||
elif isinstance(n, exp.In) and n.expressions and not n.args.get("query"):
|
||
subject, values, op = n.this, list(n.expressions), "in"
|
||
elif type(n).__name__ in _COMPARE and isinstance(n, exp.Binary):
|
||
subject, values, op = n.this, [n.expression], _COMPARE[type(n).__name__][0]
|
||
else:
|
||
return None
|
||
if subject is None or any(v is None or _own_columns(v) for v in values):
|
||
return None
|
||
column = _bare_column(subject)
|
||
cols = [column] if column is not None else _own_columns(subject)
|
||
if len(cols) != 1 or any(isinstance(x, (exp.AggFunc, exp.Select)) for x in subject.walk()):
|
||
return None
|
||
base, scope, via, datey = self._time_column(cols[0], sources)
|
||
if not (_temporal_name(base.name) or _temporal_name(cols[0].name) or datey or _datey(subject)
|
||
or any(_datey(v) for v in values)):
|
||
return None
|
||
shown = [self.render(v, sources) for v in values]
|
||
value = f"{shown[0]} AND {shown[1]}" if op == "between" else f"({', '.join(shown)})" if op == "in" \
|
||
else shown[0]
|
||
# 给调用方对照数据目录的那一列一直换到实表:不透过去的派生来源(见 _seal)比较时照写它自己,可它的这一列
|
||
# 读的还是那张表的那一列
|
||
real, real_scope, _, _ = self._time_column(cols[0], sources, unseal=True)
|
||
found = self.resolve(real, real_scope, pierce=True, unseal=True)
|
||
table = found[0].name if found[0] is not None and found[0].kind == "table" else None
|
||
return TimePredicate(column=self.part(base, scope), op=op, value=value,
|
||
condition=self.part(cond, sources, excludes=True), via=via,
|
||
table=(table, found[1]) if table else None)
|
||
|
||
def _time_column(self, column: Any, sources: dict[str, _Source], *,
|
||
unseal: bool = False) -> tuple[Any, _Sources, tuple[str, ...], bool]:
|
||
"""时间条件里的那一列是 CTE、子查询算出来的、在里面是同一列套着日期函数、CAST、MIN / MAX 时(见 _value_column),
|
||
换成里面那一列,一直换到实表:(那一列, 它那一段的来源, 路过的派生来源的列「编号.列」, 路上有没有日期函数)。
|
||
原样透出的列 resolve 已经透过去了;这里补上算了一下的。不透过去的派生来源(见 _seal)到它为止:两个 CTE 都读
|
||
orders 时,按已付那份的 created_at 筛还是按未付那份的筛,正是要分清的;unseal 时不管这个(见 through)。"""
|
||
via: list[str] = []
|
||
datey = False
|
||
for _ in range(4):
|
||
source, name = self.resolve(column, sources, unseal=unseal)
|
||
if source is not None and source.opaque and not unseal:
|
||
break
|
||
found = self._definition(source, name) if source is not None else None
|
||
inner = _value_column(found[0][0]) if found and len(found) == 1 else None
|
||
if inner is None:
|
||
break
|
||
via.append(f"{source.key}.{self.column_key(source, name)}")
|
||
datey = datey or _datey(found[0][0])
|
||
column, sources = inner, found[0][1]
|
||
return column, sources, tuple(via), datey
|
||
|
||
@staticmethod
|
||
def _aliases(query: Any) -> dict[str, Any]:
|
||
"""SELECT 里的别名 {name_key: 别名指的表达式}。"""
|
||
from sqlglot import exp
|
||
|
||
return {name_key(s.alias): s.this for s in query.selects if isinstance(s, exp.Alias) and s.alias}
|
||
|
||
@staticmethod
|
||
def _target(item: Any, selects: list[Any], aliases: dict[str, Any]) -> Any:
|
||
"""GROUP BY、ORDER BY 里的一项:写序号(1)的换成 SELECT 里对应的表达式,写别名的换成别名指的表达式
|
||
(MySQL、SQLite 的读法),其余原样。"""
|
||
from sqlglot import exp
|
||
|
||
target = _unparen(item)
|
||
if isinstance(target, exp.Literal) and not target.is_string and str(target.this).isdigit():
|
||
index = int(target.this) - 1
|
||
if 0 <= index < len(selects):
|
||
return selects[index].this if isinstance(selects[index], exp.Alias) else selects[index]
|
||
elif isinstance(target, exp.Column) and not target.table and name_key(target.name) in aliases:
|
||
return aliases[name_key(target.name)]
|
||
return target
|
||
|
||
def _group(self, select: Any, sources: dict[str, _Source]) -> list[Part]:
|
||
"""分组键。GROUP BY 1 换成 SELECT 里对应的表达式,GROUP BY 别名换成别名指的表达式(MySQL、SQLite 的读法)。"""
|
||
group = select.args.get("group")
|
||
if group is None:
|
||
return []
|
||
selects, aliases = select.selects, self._aliases(select)
|
||
return [self.part(self.canonical(self._target(item, selects, aliases), sources), sources)
|
||
for item in group.expressions]
|
||
|
||
def _having(self, select: Any, sources: dict[str, _Source]) -> list[Part]:
|
||
"""HAVING 的各个合取式。引用 SELECT 别名的列(HAVING s > 100,MySQL、SQLite 的写法)换成别名指的表达式,
|
||
和 GROUP BY 别名一个读法:不换的话,写别名和写表达式的同一个条件会被报成两个不同的条件。"""
|
||
from sqlglot import exp
|
||
|
||
clause = select.args.get("having")
|
||
if clause is None:
|
||
return []
|
||
aliases = self._aliases(select)
|
||
out = []
|
||
for cond in _conjuncts(clause.this):
|
||
copy = cond.copy()
|
||
for col in _own_columns(copy):
|
||
target = aliases.get(name_key(col.name)) if not col.table else None
|
||
# 聚合函数里的名字指的是表里的列,不是别名:SUM(amount) AS amount 之后的 SUM(amount) 不能再换
|
||
if target is None or any(isinstance(p, exp.AggFunc) for p in _parents(col, stop=None)):
|
||
continue
|
||
if col is copy:
|
||
copy = target.copy()
|
||
break
|
||
col.replace(target.copy())
|
||
out.append(self.part(self.canonical(copy, sources), sources))
|
||
return out
|
||
|
||
def _aggregates(self, select: Any, sources: dict[str, _Source]) -> list[Aggregate]:
|
||
from sqlglot import exp
|
||
|
||
return [self._aggregate(node, sources) for node in _own_nodes(select)
|
||
if isinstance(node, exp.AggFunc) and not isinstance(node.parent, exp.Window)]
|
||
|
||
def _aggregate(self, node: Any, sources: dict[str, _Source]) -> Aggregate:
|
||
from sqlglot import exp
|
||
|
||
try:
|
||
func = str(node.sql_name()).lower()
|
||
except Exception: # noqa: BLE001 - 没有固定函数名的聚合:按类型名
|
||
func = node.key
|
||
arg = node.this
|
||
distinct = isinstance(arg, exp.Distinct)
|
||
if distinct:
|
||
items = list(arg.expressions)
|
||
arg = items[0] if len(items) == 1 else exp.Tuple(expressions=[i.copy() for i in items])
|
||
if arg is None or isinstance(arg, exp.Star) or \
|
||
(func == "count" and not distinct and isinstance(_unparen(arg), exp.Literal)):
|
||
part = Part.plain("*", "*") # COUNT(*)、COUNT(1):数的是行数
|
||
else:
|
||
part = self.part(self.canonical(arg, sources), sources)
|
||
return Aggregate(func=func, arg=part, distinct=distinct, text=self.render(node, sources))
|
||
|
||
# ---- 算出来的列
|
||
|
||
def _expressions(self) -> list[Computed]:
|
||
"""算出来的列(见 Computed):最外层(UNION 的各支)选出来、不是光一列也不是字面量的;用到了的派生来源里
|
||
被外面用到、又不是原样透出一列的列(原样透出的,外面的写法 resolve 已经透成了里面那一列,不用再比定义)。"""
|
||
from sqlglot import exp
|
||
|
||
out: list[Computed] = []
|
||
for select in _branches(self.root):
|
||
sources = self.sources(select)
|
||
for item in select.selects:
|
||
target = _unwrap_value(item.this if isinstance(item, exp.Alias) else item)
|
||
if not self._plain_output(target):
|
||
out.append(self._computed([(target, sources)], None, None, "", ""))
|
||
used = self._used()
|
||
for number, name, body, _ in self._derived:
|
||
label = self._labels[number]
|
||
if id(body) in self._dead or label not in used:
|
||
continue
|
||
source = _Source("derived", name or "子查询", label, (), body)
|
||
columns = [called for called in self.names(body) if called]
|
||
columns += sorted(c for c in used[label] if c not in {name_key(x) for x in columns})
|
||
for column in dict.fromkeys(columns):
|
||
if name_key(column) not in used[label] or self.through(source, column)[0] is not source:
|
||
continue
|
||
if found := self._definition(source, column):
|
||
out.append(self._computed(found, source.name, label, self.column_key(source, column), column))
|
||
return out
|
||
|
||
@staticmethod
|
||
def _plain_output(target: Any) -> bool:
|
||
"""最外层选出来的一项不用按算法比:光一列(分组那一面比)、*、字面量。光一个聚合照收:另一边可能是拿同一个
|
||
聚合再算了一下(SUM(amount) 对 SUM(amount) / 100),得配成一对说;两边只差聚合本身的,聚合那一面已经报了。"""
|
||
from sqlglot import exp
|
||
|
||
return isinstance(_unparen(target), (exp.Column, exp.Star, exp.Literal, exp.Null, exp.Boolean))
|
||
|
||
def _computed(self, branches: list[tuple[Any, _Sources]], scope: str | None, scope_key: str | None, name: str,
|
||
shown: str) -> Computed:
|
||
"""一列的定义(UNION 每支一个)→ Computed:定义本身、聚合换成占位符之后的写法、里面的聚合。"""
|
||
from sqlglot import exp
|
||
|
||
exprs: list[Part] = []
|
||
templates: list[Part] = []
|
||
aggregates: list[Aggregate] = []
|
||
for node, sources in branches:
|
||
canon = self.canonical(node, sources)
|
||
exprs.append(self.part(canon, sources))
|
||
# 这一段自己的聚合(窗口函数、套着的子查询里的不算),按出现的先后
|
||
own = [n for n in canon.walk(prune=lambda x: isinstance(x, (exp.Select, exp.Subquery)))
|
||
if isinstance(n, exp.AggFunc) and not isinstance(n.parent, exp.Window)
|
||
and not any(isinstance(p, exp.AggFunc) for p in _parents(n, stop=None))]
|
||
aggregates += [self._aggregate(n, sources) for n in own]
|
||
if own and own[0] is canon:
|
||
templates.append(Part.plain(PLACEHOLDER, PLACEHOLDER)) # 光一个聚合
|
||
continue
|
||
for n in own:
|
||
n.replace(exp.Placeholder())
|
||
templates.append(Part(key=self.render(canon, sources, "key"), text="",
|
||
loose=self.render(canon, sources, "loose")))
|
||
|
||
def joined(parts: list[Part]) -> Part:
|
||
if len(parts) == 1:
|
||
return parts[0]
|
||
return Part(key=" | ".join(p.key for p in parts), text=" | ".join(p.text for p in parts),
|
||
loose=" | ".join(p.loose for p in parts), full=" | ".join(p.full for p in parts),
|
||
columns=tuple(c for p in parts for c in p.columns),
|
||
column_keys=tuple(k for p in parts for k in p.column_keys),
|
||
origins=tuple(o for p in parts for o in p.origins))
|
||
|
||
return Computed(scope=scope, scope_key=scope_key, name=name, shown=shown, expr=joined(exprs),
|
||
template=joined(templates), aggregates=tuple(aggregates))
|
||
|
||
def _used(self) -> dict[str, set[str]]:
|
||
"""用到了的段里引用了派生来源的哪些列:{派生来源的编号: {列名的 name_key}}。原样透出的列连同里面那一层也算
|
||
用到了;最外层的 SELECT *(t.*)用到了那个来源的每一列;USING 的列两边都算。"""
|
||
from sqlglot import exp
|
||
|
||
used: dict[str, set[str]] = {}
|
||
|
||
def mark(source: _Source | None, column: str) -> None:
|
||
for _ in range(8):
|
||
if source is None or source.kind != "derived":
|
||
return
|
||
used.setdefault(source.key, set()).add(name_key(column))
|
||
if (found := self._pass(source, column)) is None:
|
||
return
|
||
source, column = found
|
||
|
||
def mark_all(source: _Source | None, depth: int = 0) -> None:
|
||
if source is None or source.kind != "derived" or depth > 4:
|
||
return
|
||
for branch in _branches(source.query)[:1]:
|
||
inner = self.sources(branch)
|
||
for item, called in zip(branch.selects, self.names(source.query)):
|
||
if called is None:
|
||
for found in self._starred(item, inner):
|
||
mark_all(found, depth + 1)
|
||
else:
|
||
mark(source, called)
|
||
|
||
outermost = {id(s) for s in _branches(self.root)}
|
||
for select in self._alive(exp.Select):
|
||
sources = self.sources(select)
|
||
for node in _own_nodes(select):
|
||
if isinstance(node, exp.Column) and node.name and not node.is_star:
|
||
mark(self.locate(node, sources)[1], node.name)
|
||
if id(select) in outermost:
|
||
for item in select.selects:
|
||
if _is_star(item):
|
||
for found in self._starred(item, sources):
|
||
mark_all(found)
|
||
for join in select.args.get("joins") or []:
|
||
for ident in join.args.get("using") or []:
|
||
for found in sources.values():
|
||
if self._exposes(found, ident.name):
|
||
mark(found, ident.name)
|
||
return used
|
||
|
||
@staticmethod
|
||
def _starred(item: Any, sources: dict[str, _Source]) -> list[_Source]:
|
||
"""SELECT 列表里的 *、t.* 选的是哪几个来源。"""
|
||
from sqlglot import exp
|
||
|
||
if isinstance(item, exp.Column) and item.table:
|
||
found = sources.get(name_key(item.table))
|
||
return [found] if found is not None else []
|
||
return list(sources.values())
|
||
|
||
def _limit(self) -> str | None:
|
||
from sqlglot import exp
|
||
|
||
node = self.root.args.get("limit")
|
||
if node is None:
|
||
return None
|
||
value = node.expression if isinstance(node, exp.Limit) else node.args.get("count")
|
||
return value.sql(dialect=self.read, comments=False) if value is not None else None
|
||
|
||
def _limits(self) -> list[LimitFacet]:
|
||
"""各段取哪几行:最外层在前,其余按出现的先后。条件里套着的子查询(WHERE、HAVING、ON 里的)不单算:
|
||
它整个写在那个条件的 key 里,单算的话同一处不同会报两遍。"""
|
||
from sqlglot import exp
|
||
|
||
out = []
|
||
for query in self._alive(exp.Select, exp.SetOperation):
|
||
if query is not self.root and self._in_condition(query):
|
||
continue
|
||
if (found := self._limit_of(query)) is not None:
|
||
out.append(found)
|
||
return out
|
||
|
||
def _in_condition(self, node: Any) -> bool:
|
||
from sqlglot import exp
|
||
|
||
while node is not None and node is not self.root:
|
||
if isinstance(node, (exp.Where, exp.Having)) or \
|
||
(isinstance(node.parent, exp.Join) and node.arg_key == "on"):
|
||
return True
|
||
node = node.parent
|
||
return False
|
||
|
||
def _limit_of(self, query: Any) -> LimitFacet | None:
|
||
"""一段的 LIMIT / OFFSET,连同这一段的排序;两样都没写返回 None。"""
|
||
from sqlglot import exp
|
||
|
||
def written(node: Any) -> str | None:
|
||
return node.sql(dialect=self.read, comments=False) if node is not None else None
|
||
|
||
limit, offset = query.args.get("limit"), query.args.get("offset")
|
||
count = written(limit.expression if isinstance(limit, exp.Limit) else limit.args.get("count")) \
|
||
if limit is not None else None
|
||
if offset is None and isinstance(limit, exp.Limit):
|
||
offset = limit.args.get("offset") # 有的方言把 OFFSET 挂在 LIMIT 底下
|
||
skip = written(offset.expression if isinstance(offset, exp.Offset) else offset)
|
||
if skip is not None and skip.strip() == "0":
|
||
skip = None # OFFSET 0 等于没写
|
||
if limit is None and skip is None:
|
||
return None
|
||
return LimitFacet(count=count, offset=skip, order=self._order(query), scope=self._scope_name(query),
|
||
scope_key=self._scope_key(query))
|
||
|
||
@staticmethod
|
||
def _scope_key(query: Any) -> str | None:
|
||
"""query 整段是哪个派生来源(CTE 的定义、FROM 里的子查询、LATERAL 子查询)的编号;最外层、UNION 的某一支、
|
||
SELECT 列表里的子查询这类不是派生来源整段的返回 None。编号记在 CTE 定义、FROM / JOIN 项上,往外找一两层。"""
|
||
from sqlglot import exp
|
||
|
||
node = query
|
||
while isinstance(node.parent, (exp.Subquery, exp.Lateral, exp.CTE)):
|
||
node = node.parent
|
||
if found := node.meta.get(_DERIVED):
|
||
return found
|
||
return None
|
||
|
||
def _order(self, query: Any) -> tuple[tuple[Part, bool], ...]:
|
||
"""这一段的排序键 (Part, 是否降序)。ORDER BY 1、ORDER BY 别名换成 SELECT 里的表达式(UNION 按第一支的)。"""
|
||
from sqlglot import exp
|
||
|
||
order = query.args.get("order")
|
||
if order is None:
|
||
return ()
|
||
select = query if isinstance(query, exp.Select) else None
|
||
sources = self.sources(select) if select is not None else _Sources()
|
||
selects = list(query.selects)
|
||
aliases = self._aliases(query)
|
||
out = []
|
||
for item in order.expressions:
|
||
target = self._target(item.this if isinstance(item, exp.Ordered) else item, selects, aliases)
|
||
desc = bool(item.args.get("desc")) if isinstance(item, exp.Ordered) else False
|
||
out.append((self.part(self.canonical(target, sources), sources), desc))
|
||
return tuple(out)
|
||
|
||
def _scope_name(self, query: Any) -> str | None:
|
||
from sqlglot import exp
|
||
|
||
if query is self.root:
|
||
return None
|
||
parent = query.parent
|
||
if isinstance(parent, exp.Subquery) and not parent.alias and isinstance(parent.parent, exp.Lateral):
|
||
parent = parent.parent # LATERAL 的别名挂在 LATERAL 上,不在里面的子查询上
|
||
if isinstance(parent, (exp.CTE, exp.Subquery, exp.Lateral)) and parent.alias:
|
||
return parent.alias
|
||
return "子查询"
|
||
|
||
|
||
# ==========================================================================
|
||
# 等价判定:只做保持语义的变换,比规范形式
|
||
# ==========================================================================
|
||
|
||
#: 规范形式的几级(见 _canonical_forms)。等价的结论只在同一级的两个规范形式一字不差时才下
|
||
_OPTIMIZER, _LIGHT, _PARSED = "optimizer", "light", "parsed"
|
||
#: 同一段里同名的几个来源(自己关联自己)谁先谁后逐个排列着试,取写出来最小的那个;排列数超过这个就只按原来的
|
||
#: 先后(认得出的等价改写少一些,不会错判)
|
||
_MAX_ORDERINGS = 64
|
||
#: 一条语句里一共试多少种排列(里层的子查询在外层每试一种时都要重排一遍):用完了就都只按原来的先后
|
||
_ORDERING_BUDGET = 2000
|
||
#: 原来的名字(CTE 名、FROM 里子查询的别名)记在里面那段查询的 meta 里:起了规范的名字之后,差异说明还要照原样说
|
||
_ORIGINAL = "sql_semantics_name"
|
||
#: 写的是 LEFT / RIGHT JOIN、WHERE 把没关联上的行筛掉了、按内连接比的来源(记在那张表上,值是原来写的连接):差异
|
||
#: 说明里照实说「写的是 LEFT JOIN,实际是 INNER JOIN」
|
||
_REJECTED = "sql_semantics_rejected"
|
||
#: 原文写的是 LEFT JOIN、因为结果只是可空一侧跳过空值的聚合而按内连接比的来源(记在那张表上,见
|
||
#: _null_blind_left_joins):差异说明不拿它说「A 用 INNER JOIN」(原文写的是 LEFT JOIN,两种写法结果又一样)
|
||
_NULL_BLIND = "sql_semantics_null_blind"
|
||
#: 去掉了的 LEFT JOIN 关联的表(记在那一段 SELECT 上,值是表名的列表,见 _drop_unused_left_joins):差异说明不拿
|
||
#: 「另一边没有关联这张表」说事(原文是关联了的,只是有没有这次关联结果一样)
|
||
_DROPPED_JOINS = "sql_semantics_dropped_joins"
|
||
#: 从 RIGHT JOIN 改写成的 LEFT JOIN(记在 JOIN 上):差异说明里不说「用 LEFT JOIN」,说以哪一侧为主表
|
||
_FLIPPED = "sql_semantics_flipped"
|
||
#: GROUP BY 是从 SELECT DISTINCT 改写来的:差异说明里说「去重」
|
||
_FROM_DISTINCT = "sql_semantics_distinct"
|
||
#: 认不准是哪一列的列(记在列名的标识符上):SQLite 里不带表名、带双引号、表结构里又没有的名字(找不到这一列时被当成
|
||
#: 字符串);条件、标量子查询里不带表名的列(没给表结构时 qualify 归给子查询里的表,可它也可能是外层的列)。差异说明
|
||
#: 不拿它说具体的
|
||
_UNSURE = "sql_semantics_unsure"
|
||
#: 不确定的函数:每次求值结果都可能不同(两次 RANDOM() 不是同一个值),去重 AND / OR / IN 列表里一模一样的项、并子查询、
|
||
#: 合并定义一样的 CTE、改写 BETWEEN 都会改变求值的次数。有就不走规范化的两级(只比解析后的原样)
|
||
_VOLATILE = frozenset({"random", "rand", "randn", "newid", "uuid", "gen_random_uuid", "uuid_generate_v4", "random_uuid",
|
||
"randomblob", "setseed", "nextval", "sys_guid", "changes", "last_insert_rowid", "total_changes",
|
||
"sysdate", "clock_timestamp", "timeofday", "uuid_short", "random_bytes", "gen_random_bytes",
|
||
"pg_sleep", "sleep", "currval", "lastval", "last_insert_id", "found_rows", "row_count",
|
||
"get_lock", "release_lock", "benchmark", "uuid_v4", "uuidv4", "uuidv7", "gen_random_uuid_v7"})
|
||
#: 结果看行出来的先后的聚合:拼成串、拼成数组、取任意一个、取第一个 / 最后一个(类名和函数名都按小写去下划线比)
|
||
_ORDERED_AGGS = frozenset({"groupconcat", "stringagg", "arrayagg", "arrayconcatagg", "arrayuniqueagg", "jsonarrayagg",
|
||
"jsonobjectagg", "jsonagg", "jsonbagg", "jsongrouparray", "jsongroupobject", "listagg",
|
||
"xmlagg", "anyvalue", "first", "last", "firstvalue", "lastvalue", "nthvalue",
|
||
"jsonbobjectagg", "jsonobjectaggstrict", "arrayaggdistinct", "bitstringagg"})
|
||
#: 浮点数求和类的聚合:浮点数加法不满足结合律,加的先后不同结果的最后几位就不同(1902.2 和 1902.2000000000003)
|
||
_FLOAT_AGGS = frozenset({"sum", "avg", "stddev", "stddevpop", "stddevsamp", "variance", "variancepop", "varpop",
|
||
"varsamp", "covarpop", "covarsamp", "corr", "total"})
|
||
#: 拿空值做比较、算术都得空值(外连接没关联上的那一侧整行是空值)
|
||
_STRICT_UNARY = ("Neg", "Cast", "Lower", "Upper", "Length", "Abs")
|
||
_STRICT_BINARY = ("Add", "Sub", "Mul", "Div", "Mod")
|
||
#: 解析时只留下值、丢了写法的字面量:SQLite 里 0x10 是整数 16、x'10' 是 BLOB,解析出来是同一个节点;N'…'、E'…'、
|
||
#: $$…$$ 这些也一样,带了就不在语法树上判等价
|
||
_LOSSY_LITERALS = frozenset({"HEX_STRING", "BIT_STRING", "BYTE_STRING", "NATIONAL_STRING", "UNICODE_STRING",
|
||
"HEREDOC_STRING", "RAW_STRING"})
|
||
#: 类型名的同义写法:解析后重新生成换了个名字、在这个方言里却是同一个类型(SQLite 按名字里的字样定类型亲和:INT 和
|
||
#: INTEGER 都是整数亲和,VARCHAR 和 TEXT 都是文本亲和)。不在这里的换了名字就算解析丢了写法(SQLite 的 STRING 是数值
|
||
#: 亲和、TEXT 是文本亲和,sqlglot 解析成同一个类型)
|
||
_TYPE_SYNONYMS = {
|
||
"sqlite": {"INT": "INTEGER", "VARCHAR": "TEXT", "CHAR": "TEXT", "NVARCHAR": "TEXT", "CLOB": "TEXT",
|
||
"DOUBLE": "REAL", "FLOAT": "REAL"},
|
||
"postgres": {"INTEGER": "INT", "INT4": "INT", "INT8": "BIGINT", "INT2": "SMALLINT", "FLOAT8": "DOUBLE",
|
||
"FLOAT4": "REAL", "BOOL": "BOOLEAN", "NUMERIC": "DECIMAL", "CHARACTER": "CHAR"},
|
||
"mysql": {"INTEGER": "INT", "DEC": "DECIMAL", "NUMERIC": "DECIMAL", "BOOL": "BOOLEAN"},
|
||
}
|
||
#: FROM 列表到这些词就结束了(之后的逗号不再是连接)
|
||
_CLAUSE_ENDS = frozenset({"WHERE", "GROUP_BY", "HAVING", "ORDER_BY", "LIMIT", "OFFSET", "FETCH", "UNION", "INTERSECT",
|
||
"EXCEPT", "WINDOW", "QUALIFY", "SEMICOLON", "RETURNING", "FOR", "CONNECT_BY", "START_WITH",
|
||
"CLUSTER_BY", "DISTRIBUTE_BY", "SORT_BY", "PREWHERE", "LOCK"})
|
||
_ASCII_LOWER = str.maketrans("ABCDEFGHIJKLMNOPQRSTUVWXYZ", "abcdefghijklmnopqrstuvwxyz")
|
||
_FLIP = {"GT": "LT", "LT": "GT", "GTE": "LTE", "LTE": "GTE", "EQ": "EQ", "NEQ": "NEQ"}
|
||
_NEGATE = {"EQ": "NEQ", "NEQ": "EQ", "GT": "LTE", "LTE": "GT", "GTE": "LT", "LT": "GTE"}
|
||
|
||
|
||
class _Unsupported(Exception):
|
||
"""这一级规范化处理不了这条语句的结构(递归 CTE、LATERAL、表函数……),这一级不出规范形式。"""
|
||
|
||
|
||
#: 一元加号前面可能是这些词(不是一个值的结尾)
|
||
_OPERAND_KEYWORDS = frozenset({"SELECT", "WHERE", "AND", "OR", "NOT", "ON", "WHEN", "THEN", "ELSE", "HAVING", "BY", "IN",
|
||
"IS", "LIKE", "GLOB", "BETWEEN", "CASE", "AS", "DISTINCT", "ALL", "SET", "VALUES",
|
||
"LIMIT", "OFFSET", "RETURN", "ESCAPE", "FILTER", "OVER", "PARTITION", "USING", "IF",
|
||
"EXISTS", "ANY", "SOME", "FROM", "JOIN", "WITH", "ORDER", "GROUP"})
|
||
|
||
|
||
#: 函数名的同义写法(原文写的名字 → 规范的名字,空串是规范形式里不是函数调用):在这个方言里意思确实一样,sqlglot
|
||
#: 也解析成同一个节点。别的写法解析成同一个节点、意思却不同的(SQLite 的 MOD(12.5, 2) 是 0.5,12.5 % 2 先转整数得 0;
|
||
#: PostgreSQL 的 LOG10(x) 是浮点数、LOG(10, x) 是 numeric;SQLite 没有 CHAR_LENGTH,sqlglot 读成 LENGTH),两边写的
|
||
#: 函数名不一样就不判等价(见 _call_signature)。每一项都对着方言的文档核过:
|
||
#: - SQLite:IFNULL(a, b) 就是两个参数的 COALESCE;SUBSTR 和 SUBSTRING、POW 和 POWER、CEILING 和 CEIL 是同一个函数;
|
||
#: IIF 和 CASE 生成同样的字节码;like(x, y) 就是 y LIKE x、glob(x, y) 就是 y GLOB x(SQLite 文档原话)。
|
||
#: - PostgreSQL:substr 和 substring、char_length 和 length(按字符)、now() 和 current_timestamp、pow 和 power、
|
||
#: ceiling 和 ceil、mod(a, b) 和 a % b、variance 和 var_samp、btrim(x) 和 trim(x) 都是同一个。
|
||
#: - MySQL:IFNULL 和两个参数的 COALESCE、SUBSTR 和 SUBSTRING、INSTR(s, t) 和 LOCATE(t, s)、POSITION(t IN s) 和
|
||
#: LOCATE(t, s)、POW 和 POWER、CEILING 和 CEIL、MOD(a, b) 和 a % b、LCASE / UCASE 和 LOWER / UPPER、一个参数的 LOG
|
||
#: 和 LN、ISNULL(x) 和 x IS NULL、IF 和 CASE 都是同一个。
|
||
_SAFE_RESPELLINGS: dict[str, dict[str, str]] = {
|
||
"sqlite": {"IFNULL": "COALESCE", "SUBSTR": "SUBSTRING", "POW": "POWER", "CEILING": "CEIL", "IIF": "", "LIKE": "",
|
||
"GLOB": ""},
|
||
"postgres": {"SUBSTR": "SUBSTRING", "CHAR_LENGTH": "LENGTH", "CHARACTER_LENGTH": "LENGTH", "NOW": "",
|
||
"POW": "POWER", "CEILING": "CEIL", "MOD": "", "VARIANCE": "VAR_SAMP", "BTRIM": "TRIM"},
|
||
"mysql": {"IFNULL": "COALESCE", "SUBSTR": "SUBSTRING", "INSTR": "LOCATE", "POSITION": "LOCATE", "POW": "POWER",
|
||
"CEILING": "CEIL", "MOD": "", "LCASE": "LOWER", "UCASE": "UPPER", "LOG": "LN", "ISNULL": "", "IF": ""},
|
||
}
|
||
#: 后面跟着括号却不是函数名的词
|
||
_NOT_CALLS = frozenset({"IN", "EXISTS", "OVER", "VALUES", "AS", "ON", "USING", "FILTER", "WITHIN", "ANY", "ALL", "SOME",
|
||
"ROW", "AND", "OR", "NOT", "WHERE", "SELECT", "FROM", "JOIN", "HAVING", "BY", "WHEN", "THEN",
|
||
"ELSE", "CASE", "UNION", "INTERSECT", "EXCEPT", "WITH", "RECURSIVE", "LATERAL", "TABLE",
|
||
"INTO", "SET", "DISTINCT", "LIMIT", "OFFSET", "FETCH", "ORDER", "GROUP", "PARTITION", "RANGE",
|
||
"ROWS", "BETWEEN", "IS", "ESCAPE", "INTERVAL", "RETURN", "NULLS", "OUTER", "INNER", "CROSS",
|
||
"NATURAL", "FULL", "MATERIALIZED", "QUALIFY", "WINDOW", "TUPLE"})
|
||
|
||
|
||
def _mysql_dashes(text: str) -> str:
|
||
"""MySQL 里 -- 后面要跟空白(或控制字符)才是注释,不然就是两个减号:credit--1 是 credit - (-1)(MySQL 文档的
|
||
原例)。sqlglot 一律当成注释、把后面整行吃掉,这里先把这样的 -- 拆成 「- -」 再交给它解析,意思不变。字符串
|
||
('…'、"…",反斜杠和成对的引号转义)、反引号里的名字、/* … */ 和 # 注释、真正的 -- 注释里面的不动。"""
|
||
out: list[str] = []
|
||
i, n = 0, len(text)
|
||
while i < n:
|
||
ch = text[i]
|
||
if ch in "'\"`":
|
||
j = i + 1
|
||
while j < n:
|
||
if text[j] == "\\" and ch != "`":
|
||
j += 2
|
||
continue
|
||
if text[j] == ch:
|
||
if j + 1 < n and text[j + 1] == ch:
|
||
j += 2
|
||
continue
|
||
break
|
||
j += 1
|
||
out.append(text[i:j + 1])
|
||
i = j + 1
|
||
continue
|
||
if text.startswith("/*", i):
|
||
j = text.find("*/", i + 2)
|
||
j = n if j < 0 else j + 2
|
||
out.append(text[i:j])
|
||
i = j
|
||
continue
|
||
if ch == "#" or (text.startswith("--", i) and (i + 2 >= n or text[i + 2].isspace() or ord(text[i + 2]) < 32)):
|
||
j = text.find("\n", i)
|
||
j = n if j < 0 else j
|
||
out.append(text[i:j])
|
||
i = j
|
||
continue
|
||
if text.startswith("--", i):
|
||
out.append("- ")
|
||
i += 1
|
||
continue
|
||
out.append(ch)
|
||
i += 1
|
||
return "".join(out)
|
||
|
||
|
||
def _executable_comment(tokens: list[Any]) -> bool:
|
||
"""MySQL / MariaDB 会执行 /*! … */、/*M! … */ 里的代码(版本号写在 ! 后面的,服务器版本够了才执行),sqlglot 当成
|
||
普通注释扔掉。别的库里这只是注释,一样不在语法树上判等价(只会少认几对等价)。"""
|
||
return any(c.lstrip().startswith(("!", "M!")) for t in tokens for c in (t.comments or []))
|
||
|
||
|
||
def _call_names(tokens: list[Any]) -> list[str]:
|
||
"""原文里写成「名字(」的函数名(大写)。IN (、EXISTS (、OVER ( 这类后面跟括号的关键字不算。"""
|
||
out = []
|
||
for token, nxt in zip(tokens, tokens[1:]):
|
||
if nxt.token_type.name == "L_PAREN" and token.token_type.name not in ("STRING", "NUMBER", "IDENTIFIER") \
|
||
and _PLAIN_NAME.match(token.text or "") and token.text.upper() not in _NOT_CALLS:
|
||
out.append(token.text.upper())
|
||
return out
|
||
|
||
|
||
#: 原文写的函数名(按同义写法换算之后)和 sqlglot 重新生成的不一样时,记在函数节点的 meta 里,比等价时算进去(见
|
||
#: _mark_surfaces、_fingerprint)
|
||
_SURFACE = "sql_semantics_surface"
|
||
|
||
|
||
def _mark_surfaces(tree: Any, text: str, read: str | None) -> bool:
|
||
"""函数节点上记下原文写的名字(换算成规范的同义写法,见 _SAFE_RESPELLINGS)——只在它和 sqlglot 重新生成的名字
|
||
不一样时记:sqlglot 把方言里意思不同的写法解析成同一个节点时(SQLite 的 MOD(a, 2) 和 a % 2 都是 Mod;PostgreSQL 的
|
||
LOG10(x) 和 LOG(10, x) 都是 Log),两边的语法树就因为这一项而不同,不会判等价。解析器在函数节点的 meta 里留着
|
||
函数名在原文里的位置(start / end),按位置取原文。
|
||
|
||
返回能不能记全:原文写的某个函数名在重新生成的写法里不见了(又不是同义写法),却没有哪个节点记下了它(解析器
|
||
换了写法、没留位置,PostgreSQL 的 DATE_PART(…) 解析成 EXTRACT 就是这样)时返回 False,调用方就不在语法树上判等价。"""
|
||
from sqlglot import exp
|
||
|
||
mapping = _SAFE_RESPELLINGS.get(read or "", {})
|
||
attached: set[str] = set()
|
||
leaves = (exp.Column, exp.Identifier, exp.Literal, exp.Star, exp.Null, exp.Boolean, exp.Var, exp.Placeholder,
|
||
exp.Parameter, exp.DataType)
|
||
for node in tree.walk():
|
||
if isinstance(node, leaves):
|
||
continue
|
||
start, end = node.meta.get("start"), node.meta.get("end")
|
||
if not isinstance(start, int) or not isinstance(end, int) or not 0 <= start <= end < len(text):
|
||
continue
|
||
written = text[start:end + 1].upper()
|
||
# 只认写成「名字(」的函数调用(Mod 这类运算节点从 MOD(…) 解析出来时也带着名字的位置)
|
||
if not _PLAIN_NAME.match(written) or not text[end + 1:].lstrip().startswith("("):
|
||
continue
|
||
written = mapping.get(written, written)
|
||
try:
|
||
generated = node.sql(dialect=read, comments=False)
|
||
except Exception: # noqa: BLE001 - 生成不了:照写的记
|
||
generated = ""
|
||
found = re.match(r"\s*([A-Za-z_][A-Za-z0-9_]*)\s*\(", generated)
|
||
if written and (found is None or found.group(1).upper() != written):
|
||
node.meta[_SURFACE] = written
|
||
attached.add(text[start:end + 1].upper())
|
||
try:
|
||
tokens = _tokens(text, read)
|
||
present = {t.text.upper() for t in _tokens(tree.sql(dialect=read, comments=False), read)}
|
||
except Exception: # noqa: BLE001 - 切词、重新生成不了:当成记不全
|
||
return False
|
||
# CAST 一类(解析成 Cast 节点、不留位置)换写法也是同一个意思(CAST(x AS INT) 和 x::INT、MySQL 的 CONVERT(x, CHAR)),
|
||
# 也不会和别的函数撞成同一个节点(SQLite 的 CAST(x AS DATE) 重新生成成 DATE(x),可 DATE(x) 解析出来是 Date 节点)
|
||
return all(name in present or name in mapping or name in attached or name in ("CAST", "TRY_CAST", "CONVERT")
|
||
for name in _call_names(tokens))
|
||
|
||
|
||
def _respelled(text: str, read: str | None, tree: Any) -> bool:
|
||
"""解析后重新生成时,原文写的函数名不见了、又不是这个方言里的同义写法(normalize_sql 用:重新生成的文本会把
|
||
MOD() 和 % 写成一样,这样的语句改按原文逐词规范化)。"""
|
||
try:
|
||
tokens = _tokens(text, read)
|
||
again = _tokens(tree.sql(dialect=read, comments=False), read)
|
||
except Exception: # noqa: BLE001 - 切词、重新生成不了:当成丢了写法
|
||
return True
|
||
present = {t.text.upper() for t in again}
|
||
safe = _SAFE_RESPELLINGS.get(read or "", {})
|
||
return any(name not in present and name not in safe for name in _call_names(tokens))
|
||
|
||
|
||
def _type_words(tokens: list[Any], read: str | None) -> list[str]:
|
||
from sqlglot.parser import Parser
|
||
|
||
synonyms = _TYPE_SYNONYMS.get(read or "", {})
|
||
words = [t.text.upper() for t in tokens if t.token_type in Parser.TYPE_TOKENS]
|
||
return sorted(synonyms.get(w, w) for w in words)
|
||
|
||
|
||
#: 比较运算:同一层括号、同一段条件里有两个(或者和下面的谓词各一个)没加括号地连着写,各方言的结合顺序和 sqlglot 的
|
||
#: 不一样(sqlglot 让 IS、IN、LIKE、BETWEEN 都比 = 结合得紧,还让 < > 比 = 紧)
|
||
_MISREAD_COMPARISONS = frozenset({"EQ", "NEQ", "LT", "GT", "LTE", "GTE", "NULLSAFE_EQ"})
|
||
#: 谓词:IS(含 IS NOT、IS [NOT] DISTINCT FROM)、IN、LIKE 一类、BETWEEN、PostgreSQL / SQLite 的后缀 ISNULL、NOTNULL
|
||
_MISREAD_PREDICATES = frozenset({"IS", "IN", "LIKE", "ILIKE", "GLOB", "RLIKE", "IRLIKE", "SIMILAR_TO", "BETWEEN",
|
||
"ISNULL", "NOTNULL", "MATCH", "MEMBER_OF", "SOUNDS_LIKE"})
|
||
#: 按位运算和 sqlglot 当成同一级的「别的运算符」(PostgreSQL 的 #、JSON 的 -> / ->>、数组的 @> / <@ / &&……):和别的
|
||
#: 二元运算混用时,各方言的结合顺序和 sqlglot 的不一样(MySQL 里 & 比 | 紧、<< 比 & 紧、^ 比 * 紧;PostgreSQL 里前缀 ~
|
||
#: 比 + 松;sqlglot 把它们都放在同一级、从左往右)
|
||
_MISREAD_BITWISE = frozenset({"AMP", "PIPE", "CARET", "TILDE", "HASH", "ARROW", "DARROW", "HASH_ARROW", "DHASH_ARROW",
|
||
"LR_ARROW", "AT_GT", "LT_AT", "DQMARK", "CARET_AT", "QMARK_AMP", "QMARK_PIPE", "HASH_DASH",
|
||
"PIPE_SLASH", "DPIPE_SLASH", "OPERATOR"})
|
||
#: 算术和拼接
|
||
_MISREAD_ARITHMETIC = frozenset({"PLUS", "DASH", "STAR", "SLASH", "MOD", "DIV", "DPIPE"})
|
||
#: 到这些词一段条件(表达式)就结束了:之后是另一个子句、另一项,结合顺序和前面无关。只列确定是子句、分隔的词:漏列
|
||
#: 一个只会让一段拉长、多拦下几条(宁可认不出),不会漏拦。ALL / ANY、DISTINCT、OVER、FILTER、WITH 不在里面(x > ALL
|
||
#: (…)、IS DISTINCT FROM、窗口函数、WITH TIME ZONE 都在一个表达式中间);LEFT / RIGHT 后面紧跟括号的是函数
|
||
_MISREAD_SEGMENT_ENDS = frozenset({
|
||
"AND", "OR", "XOR", "COMMA", "SEMICOLON", "SELECT", "FROM", "WHERE", "HAVING", "ON", "USING", "WHEN", "THEN", "ELSE",
|
||
"GROUP_BY", "ORDER_BY", "PARTITION_BY", "LIMIT", "OFFSET", "FETCH", "UNION", "INTERSECT", "EXCEPT", "JOIN", "INNER",
|
||
"OUTER", "LEFT", "RIGHT", "FULL", "CROSS", "NATURAL", "STRAIGHT_JOIN", "ALIAS", "ASC", "DESC", "WINDOW", "QUALIFY",
|
||
"RETURNING", "LATERAL", "VALUES", "SET", "INTO", "FOR", "LOCK", "PREWHERE", "CLUSTER_BY", "DISTRIBUTE_BY", "SORT_BY",
|
||
"CONNECT_BY", "START_WITH", "ROWS", "RANGE"})
|
||
#: 一个值的最后一个词(之后跟着的是中缀运算符):用来分清 ~ 是前缀的按位取反还是中缀的正则匹配、LIKE( / ISNULL( 是
|
||
#: 函数还是谓词
|
||
_OPERAND_ENDS = frozenset({"VAR", "IDENTIFIER", "NUMBER", "R_PAREN", "R_BRACKET", "NULL", "TRUE", "FALSE", "STAR",
|
||
"PLACEHOLDER", "PARAMETER", "END"})
|
||
#: 字符串里的反斜杠是转义的方言(MySQL / MariaDB,以及不知道是哪种库时):sqlglot 不认 \Z(Ctrl+Z)、\0(NUL),
|
||
#: 原样留着反斜杠,和写成 \\Z 的撞成同一个值。SQLite、PostgreSQL 的普通字符串里反斜杠就是反斜杠
|
||
_PLAIN_BACKSLASH = frozenset({"sqlite", "postgres"})
|
||
#: sqlglot 读注释的方式和库一样的方言:块注释 SQLite、MySQL 不嵌套(sqlglot 的这两种读法也不嵌套),PostgreSQL 嵌套
|
||
#: (sqlglot 也嵌套)。别的方言、不知道是哪种库时,块注释里再写 /* 就说不准注释到哪儿为止
|
||
_COMMENTS_AS_SQLGLOT = frozenset({"sqlite", "mysql", "postgres"})
|
||
|
||
|
||
def _symbolic(word: str) -> bool:
|
||
"""运算符写成符号(~、~*、@@、-|-),不是写成关键字(LIKE、ILIKE、REGEXP)。"""
|
||
return bool(word) and not any(ch.isalnum() or ch == "_" for ch in word)
|
||
|
||
|
||
def _range_tokens(read: str | None) -> frozenset[str]:
|
||
"""这个方言里 sqlglot 当成 LIKE、IN 那一级(比 ||、+ 松)来读的词的类型名(RANGE_PARSERS)。"""
|
||
from sqlglot.dialects.dialect import Dialect
|
||
|
||
try:
|
||
return frozenset(t.name for t in Dialect.get_or_raise(read).parser_class.RANGE_PARSERS)
|
||
except Exception: # noqa: BLE001 - 取不到:只靠下面列出来的几类
|
||
return frozenset()
|
||
|
||
|
||
def _misread(tokens: list[Any], text: str, read: str | None) -> str | None:
|
||
"""原文里有已知会被 sqlglot 读错的写法时返回是哪一种(不在语法树上判等价,也不说具体的不同);没有返回 None。
|
||
按方言切出来的词逐个看、按括号深度分层(CASE … END 也算一层),不拿正则在原文上硬匹配:
|
||
|
||
- 字符串里有反斜杠(方言是 MySQL / MariaDB,或者不知道是哪种库时,见 _PLAIN_BACKSLASH);
|
||
- 两个字符串常量挨着写:MySQL 里 'a' 'b' 是 'ab',SQLite 里却是 'a' AS b,sqlglot 一律读成拼接;
|
||
- 数字后面紧跟着字母、下划线、数字(1_000、0o17):SQLite 3.46 起、PostgreSQL 16 起 1_000 是一千、0o17 是 15,
|
||
sqlglot 切成数字 1 加别名 _000;
|
||
- XOR(MySQL 里 AND 比 XOR 结合得紧,sqlglot 反过来);方言不知道时的 ||(MySQL 里是 OR)、MySQL 以外的 &&;
|
||
- 同一层括号、同一段条件里,比较运算和 IS / IN / LIKE / BETWEEN / 另一个比较运算没加括号地混用(a = b IS NULL、
|
||
a < 100 IN (0, 1)、a = 100 < 200):SQLite 里 IS、IN 和 = 同级从左往右,PostgreSQL 里 IS 比 = 松,MySQL 里
|
||
比较运算都同级从左往右,sqlglot 让 IS / IN 先结合;
|
||
- 同一层括号、同一段里,按位运算(& | ^ << >> ~ 和 PostgreSQL 的 # 这类)、COLLATE 和别的二元运算(算术、拼接、
|
||
另一种按位运算)混用;
|
||
- 写成符号、sqlglot 却当成 LIKE / IN 那一级来读的运算符(PostgreSQL 的 ~、~*、~~、~~*、@@、-|-、&<……)和别的二元
|
||
运算混用:PostgreSQL 里它们是「别的运算符」,和 ||、&、# 同级、从左往右结合,status ~ 'p' || 'x' 是
|
||
(status ~ 'p') || 'x'(得到 'truex' 这样的文字),sqlglot 读成 status ~ ('p' || 'x')。写成关键字的 LIKE、ILIKE
|
||
本来就比 || 松,PostgreSQL 和 sqlglot 读得一样,不算;
|
||
- PostgreSQL(以及不知道是哪种库时)的前缀 @(取绝对值,@ amount - 20 是 @ (amount - 20),sqlglot 读成参数
|
||
$amount)、|/ 和 ||/(双精度的开方,sqlglot 写成 SQRT / CBRT,numeric 的 SQRT 结果多出几位小数);COLLATE 后面
|
||
同一段里又有 ::(PostgreSQL 里 x COLLATE "C"::text 的 :: 套在整个 COLLATE 上,sqlglot 读成套在排序规则的名字上);
|
||
- 不知道是哪种库时:块注释里再写 /*(PostgreSQL 嵌套、SQLite / MySQL 不嵌套,见 _COMMENTS_AS_SQLGLOT)、后面紧跟着
|
||
字的 --(MySQL 里 amount--1 是 amount - (-1),别的库是注释)、#(MySQL 里是注释,PostgreSQL 里是按位异或)、双引号
|
||
括起来的名字(MySQL 默认的 sql_mode 里 "status" 是字符串,别的库和 sqlglot 的通用读法里是列名);
|
||
- 前缀的 NOT 后面同一段里既有谓词、又有算术 / 拼接 / 按位运算(NOT status IN (…) * amount、NOT paid_at IS NULL * amount):
|
||
SQLite 里 NOT 最松,sqlglot 的 NOT 只取等值那一级。宁可多拦(NOT a = 1 + 2 其实读得对),只给兜底不会说错。
|
||
|
||
一段条件到 AND、OR、逗号、子句关键字为止(BETWEEN 自己的 AND 不算;IS [NOT] DISTINCT FROM 的 FROM 不算)。"""
|
||
if read not in _PLAIN_BACKSLASH and any(
|
||
t.token_type.name.endswith("STRING") and "\\" in text[t.start:t.end + 1] for t in tokens):
|
||
return "字符串里的反斜杠转义"
|
||
if read not in _COMMENTS_AS_SQLGLOT and any("/*" in c for t in tokens for c in (t.comments or [])):
|
||
return "块注释里套着 /*"
|
||
if read is None and (_mysql_dashes(text) != text or any(t.token_type.name == "HASH" for t in tokens)):
|
||
return "-- 和 # 在这个方言里是不是注释说不准"
|
||
if read is None and any(t.token_type.name == "IDENTIFIER" and text[t.start:t.start + 1] == '"' for t in tokens):
|
||
# sqlglot 的通用读法(和 PostgreSQL、SQLite 一样)把 "status" 读成名字,MySQL 默认的 sql_mode 里它是字符串:
|
||
# WHERE "status" = 'paid' 恒为假,和 WHERE status = 'paid' 读成了同一棵树
|
||
return "双引号括起来的是名字还是字符串说不准"
|
||
ranged = _range_tokens(read)
|
||
for prev, token in zip(tokens, tokens[1:]):
|
||
if prev.token_type.name.endswith("STRING") and token.token_type.name.endswith("STRING"):
|
||
return "两个字符串常量挨着写"
|
||
if prev.token_type.name == "NUMBER" and token.start == prev.end + 1 and \
|
||
(text[token.start:token.start + 1].isalnum() or text[token.start:token.start + 1] == "_"):
|
||
return "数字的写法(1_000、0o17)"
|
||
frames: list[dict[str, Any]] = [{}]
|
||
|
||
def fresh(frame: dict[str, Any]) -> None:
|
||
frame.update(compare=0, between=0, bitwise=set(), arithmetic=False, collated=False, negated=False)
|
||
|
||
def done(frame: dict[str, Any]) -> str | None:
|
||
if frame["compare"] >= 2:
|
||
return "比较运算和 IS / IN / LIKE / BETWEEN 没加括号地混用"
|
||
if frame["negated"] and frame["compare"] and (frame["arithmetic"] or frame["bitwise"]):
|
||
# 前缀的 NOT 后面同一段里既有谓词、又有算术 / 拼接 / 按位运算(NOT status IN (…) * amount):SQLite 里 NOT 最松、
|
||
# * 比 IN 紧,是 NOT ((status IN (…)) * amount);sqlglot 的 NOT 只取等值那一级,读成 (NOT status IN (…)) * amount
|
||
return "前缀的 NOT 和谓词、算术运算没加括号地混用"
|
||
if frame["bitwise"] and (len(frame["bitwise"]) >= 2 or frame["arithmetic"]):
|
||
return "按位运算、COLLATE 和别的运算没加括号地混用"
|
||
return None
|
||
|
||
fresh(frames[0])
|
||
i, n = 0, len(tokens)
|
||
while i < n:
|
||
token = tokens[i]
|
||
kind, word = token.token_type.name, token.text.upper()
|
||
frame = frames[-1]
|
||
prev = tokens[i - 1] if i else None
|
||
nxt = tokens[i + 1] if i + 1 < n else None
|
||
after_operand = prev is not None and (prev.token_type.name in _OPERAND_ENDS or
|
||
prev.token_type.name.endswith("STRING"))
|
||
called = nxt is not None and nxt.token_type.name == "L_PAREN" and not after_operand
|
||
if kind == "XOR" or (kind == "VAR" and word == "XOR"):
|
||
return "XOR"
|
||
if (kind == "DPIPE" and read is None) or (kind == "DAMP" and read != "mysql"):
|
||
return f"{token.text} 在这个方言里的意思说不准"
|
||
if kind == "PARAMETER" and token.text == "@" and read in ("postgres", None):
|
||
# PostgreSQL 里前缀的 @ 是取绝对值(@ amount - 20 是 @ (amount - 20)),sqlglot 读成参数 $amount
|
||
return "PostgreSQL 的前缀 @"
|
||
if kind in ("PIPE_SLASH", "DPIPE_SLASH"):
|
||
# PostgreSQL 的 |/、||/ 是双精度的开方,sqlglot 写成 SQRT / CBRT:numeric 的 SQRT 多出几位小数
|
||
return f"PostgreSQL 的 {token.text}"
|
||
if kind == "DCOLON" and frame["collated"]:
|
||
# x COLLATE "C"::text:PostgreSQL 里 :: 套在整个 COLLATE 上,sqlglot 读成套在排序规则的名字上
|
||
return "COLLATE 后面跟着 ::"
|
||
if kind in ("L_PAREN", "L_BRACKET", "CASE"):
|
||
frames.append({})
|
||
fresh(frames[-1])
|
||
elif kind in ("R_PAREN", "R_BRACKET", "END"):
|
||
why = done(frame)
|
||
if why:
|
||
return why
|
||
if len(frames) > 1:
|
||
frames.pop()
|
||
elif kind == "AND" and frame["between"]:
|
||
frame["between"] -= 1
|
||
elif (kind in _MISREAD_SEGMENT_ENDS and not (kind in ("LEFT", "RIGHT") and called)) or \
|
||
(kind in ("DAMP", "DPIPE") and read == "mysql"):
|
||
why = done(frame)
|
||
if why:
|
||
return why
|
||
fresh(frame)
|
||
elif kind in _MISREAD_COMPARISONS:
|
||
# << 和 >> 切出来是两个挨着的 < / >:是移位,不是两个比较
|
||
if kind in ("LT", "GT") and nxt is not None and nxt.start == token.end + 1 and nxt.token_type.name == kind:
|
||
frame["bitwise"].add(token.text * 2)
|
||
i += 2
|
||
continue
|
||
frame["compare"] += 1
|
||
elif token.text == "~" and not after_operand:
|
||
# 前缀的 ~ 是按位取反(PostgreSQL 切成 RLIKE,别的方言切成 TILDE)
|
||
frame["bitwise"].add("~")
|
||
elif kind in _MISREAD_PREDICATES and not (called and kind not in ("IN", "BETWEEN", "IS")):
|
||
# MySQL 的 ISNULL(x)、SQLite 的 like(x, y)、MATCH (…) AGAINST 这类写在值的位置、后面跟括号的是函数
|
||
frame["compare"] += 1
|
||
if _symbolic(token.text):
|
||
# 写成符号的 ~、~*、~~、~~*:PostgreSQL 里和 ||、& 同级,sqlglot 当成 LIKE 那一级(比 || 松)
|
||
frame["bitwise"].add(kind)
|
||
if kind == "BETWEEN":
|
||
frame["between"] += 1
|
||
elif kind == "IS":
|
||
# IS NOT DISTINCT FROM:后面的 NOT、DISTINCT、FROM 都是这一个谓词的一部分
|
||
j = i + 1
|
||
if j < n and tokens[j].token_type.name == "NOT":
|
||
j += 1
|
||
if j + 1 < n and tokens[j].token_type.name == "DISTINCT" and tokens[j + 1].token_type.name == "FROM":
|
||
i = j + 2
|
||
continue
|
||
i = j
|
||
continue
|
||
elif kind == "NOT" and not after_operand and not (nxt is not None and nxt.token_type.name == "NULL"):
|
||
# 前缀的 NOT(前面不是操作数):x NOT IN / NOT LIKE / NOT BETWEEN 跟在操作数后面,不算;IS NOT 在 IS 那一支整个算过
|
||
frame["negated"] = True
|
||
elif kind == "NOT" and nxt is not None and nxt.token_type.name == "NULL":
|
||
# SQLite 的后缀 x NOT NULL(前面是 IS 的已经整个算过了)
|
||
frame["compare"] += 1
|
||
elif kind in _MISREAD_BITWISE or (kind in ranged and _symbolic(token.text)):
|
||
# 后一种:@@、-|-、&<、@? 这些写成符号的运算符,sqlglot 当成 LIKE 那一级,PostgreSQL 里和 ||、& 同级
|
||
frame["bitwise"].add(kind)
|
||
elif kind == "COLLATE":
|
||
# MySQL 的 COLLATE 比 + 结合得紧,sqlglot 的 MySQL 读法让 a + b COLLATE x 整个套上 COLLATE:和算术、拼接、
|
||
# 按位运算混用时当成按位运算一样拦下
|
||
frame["bitwise"].add(kind)
|
||
frame["collated"] = True
|
||
elif kind in _MISREAD_ARITHMETIC:
|
||
frame["arithmetic"] = True
|
||
i += 1
|
||
for frame in frames:
|
||
why = done(frame)
|
||
if why:
|
||
return why
|
||
return None
|
||
|
||
|
||
def _lost_in_parsing(text: str, read: str | None, tree: Any) -> bool:
|
||
"""原文里有 sqlglot 解析时丢掉、或者读错了、语义却不一样的写法,有就不在语法树上判等价:
|
||
|
||
- 一元加号:SQLite 里 +a 去掉了列的类型亲和,+a = '10' 和 a = '10' 结果不同,解析出来却是同一棵树;
|
||
- 只留下值、丢了写法的字面量(0x10 和 x'10',见 _LOSSY_LITERALS);
|
||
- 会执行的注释(MySQL 的 /*! … */,见 _executable_comment);
|
||
- 写成 ! 的 NOT:MySQL 里 ! 比比较运算结合得紧(! status = 'paid' 是 (!status) = 'paid'),NOT 比比较结合得松,
|
||
sqlglot 都按 NOT 读;
|
||
- 类型名:解析后重新生成换了名字、又不是这个方言里的同义写法(SQLite 的 CAST(x AS STRING) 是数值亲和,解析成
|
||
和 CAST(x AS TEXT) 一样的节点);
|
||
- 已知会被读错的写法(MySQL 字符串里的反斜杠转义、1_000 这类数字、没加括号地混用的比较 / 按位运算、PostgreSQL 写成
|
||
符号的匹配运算符和 || 混用、前缀的 @、|/、XOR,不知道是哪种库时套着的块注释、紧跟着字的 --、双引号括起来的名字,
|
||
见 _misread)。"""
|
||
try:
|
||
tokens = _tokens(text, read)
|
||
except Exception: # noqa: BLE001 - 切词都不行的,解析也过不了,那边会说
|
||
return False
|
||
if _executable_comment(tokens):
|
||
return True
|
||
why = _misread(tokens, text, read)
|
||
if why:
|
||
logger.debug("SQL 有 sqlglot 会读错的写法(%s),不在语法树上判等价", why)
|
||
return True
|
||
prev = None
|
||
for token in tokens:
|
||
kind = token.token_type.name
|
||
if kind in _LOSSY_LITERALS or (kind == "NOT" and token.text == "!"):
|
||
return True
|
||
if kind == "PLUS":
|
||
if prev is None:
|
||
return True
|
||
pk, word = prev.token_type.name, prev.text
|
||
operand = pk in ("VAR", "IDENTIFIER", "NUMBER", "R_PAREN", "R_BRACKET", "NULL", "TRUE", "FALSE", "END",
|
||
"STAR") or pk.endswith("STRING") or (
|
||
_PLAIN_NAME.match(word) is not None and word.upper() not in _OPERAND_KEYWORDS)
|
||
if not operand:
|
||
return True
|
||
prev = token
|
||
try:
|
||
again = _tokens(tree.sql(dialect=read, comments=False), read)
|
||
except Exception: # noqa: BLE001 - 重新生成不了:当成丢了写法
|
||
return True
|
||
return _type_words(tokens, read) != _type_words(again, read)
|
||
|
||
|
||
def _mark_comma_joins(tree: Any, text: str, read: str | None) -> None:
|
||
"""SQLite 里逗号连接和 CROSS JOIN 解析成同一个节点(都是 kind=CROSS),可 SQLite 不调换 CROSS JOIN 两边的先后、
|
||
逗号连接会调换:结果看行出来的先后时(GROUP_CONCAT、有并列的 LIMIT)两种写法不一样。按原文认出哪些是逗号连接,
|
||
去掉它们的 CROSS(和别的方言解析出来的一样)。认不出时整条语句的 CROSS 连接标成认不出(见 _fingerprint)。"""
|
||
from sqlglot import exp
|
||
|
||
joins = [j for j in tree.find_all(exp.Join) if str(j.args.get("kind") or "").upper() == "CROSS"]
|
||
if not joins:
|
||
return
|
||
try:
|
||
tokens = _tokens(text, read)
|
||
except Exception: # noqa: BLE001
|
||
tokens = []
|
||
marks: list[tuple[int, bool]] = []
|
||
depth = 0
|
||
in_from: set[int] = set()
|
||
for tok in tokens:
|
||
name = tok.token_type.name
|
||
if name == "L_PAREN":
|
||
depth += 1
|
||
elif name == "R_PAREN":
|
||
in_from.discard(depth)
|
||
depth -= 1
|
||
elif name == "FROM":
|
||
in_from.add(depth)
|
||
elif name in _CLAUSE_ENDS:
|
||
in_from.discard(depth)
|
||
elif name == "COMMA" and depth in in_from:
|
||
marks.append((tok.start, True))
|
||
elif name in ("JOIN", "STRAIGHT_JOIN"):
|
||
marks.append((tok.start, False))
|
||
for join in joins:
|
||
starts = [i.meta["start"] for i in join.this.find_all(exp.Identifier) if isinstance(i.meta.get("start"), int)]
|
||
before = [m for m in marks if starts and m[0] < min(starts)]
|
||
if not before:
|
||
join.meta["sql_semantics_join"] = f"unknown:{id(tree)}"
|
||
continue
|
||
comma = max(before)[1]
|
||
if comma:
|
||
join.set("kind", None)
|
||
|
||
|
||
@dataclass
|
||
class _Form:
|
||
"""一条语句在某一级的规范形式。key 一样才算等价,text 是生成的 SQL(给人看);pre 是起规范名字之前的树。"""
|
||
|
||
level: str
|
||
text: str
|
||
tree: Any
|
||
pre: Any = None
|
||
qualified: bool = False
|
||
|
||
@property
|
||
def key(self) -> str:
|
||
"""比等价用的写法:语法树逐个节点的类型和参数(见 _fingerprint)。不拿生成的 SQL 比:sqlglot 按方言生成时
|
||
会把不同的写法写成同一个样子(SQLite 的 CAST(x AS DATE) 生成 DATE(x),两者在 SQLite 里结果不同)。"""
|
||
return _fingerprint(self.tree)
|
||
|
||
|
||
def _fingerprint(node: Any) -> str:
|
||
"""语法树的确定写法:每个节点写类名和全部参数(按参数名排序,字符串原样),不同的树一定写得不一样。括号不算
|
||
(语法树的结构已经定了怎么结合:SQLite 里 (a) 和 a 的类型亲和也一样);类型标注、注释、meta 不算,只有认不出
|
||
是逗号还是 CROSS JOIN 的连接带上标记(见 _mark_comma_joins)。"""
|
||
from sqlglot import exp
|
||
|
||
if isinstance(node, exp.Paren):
|
||
return _fingerprint(node.this)
|
||
if isinstance(node, exp.Expr):
|
||
parts = [f"{k}={_fingerprint(v)}" for k, v in sorted(node.args.items())
|
||
if v is not None and v is not False and v != []]
|
||
mark = node.meta.get("sql_semantics_join") if isinstance(node, exp.Join) else None
|
||
if mark:
|
||
parts.append(f"@join={mark}")
|
||
surface = node.meta.get(_SURFACE)
|
||
if surface:
|
||
parts.append(f"@surface={surface}")
|
||
return f"{type(node).__name__}({','.join(parts)})"
|
||
if isinstance(node, list):
|
||
return "[" + ",".join(_fingerprint(v) for v in node) + "]"
|
||
return repr(node)
|
||
|
||
|
||
def _key(node: Any, read: str | None = None) -> tuple[str, str]:
|
||
"""排序、去重用的键:只看语法树(见 _fingerprint),括号写没写不影响先后。"""
|
||
return _fingerprint(node), ""
|
||
|
||
|
||
@dataclass
|
||
class _Schema:
|
||
"""数据源的表结构:mapping 给 qualify(只要列名),types 记各列声明的类型(判断「相等的值是不是写法也一样」,见
|
||
_Sensitivity),columns 是所有表的列名(小写);not_null 是确定不会为空的列(声明了 NOT NULL,或者是主键:
|
||
PostgreSQL / MySQL 的主键列都不为空,SQLite 只有单独一列、声明成 INTEGER 的主键是 rowid、不会为空——别的主键
|
||
在 SQLite 里照样能存空值),pk 是各表的主键列,unique 是唯一约束 / 唯一索引的列组,fks 是外键(列, 指向的表,
|
||
指向的列)。
|
||
|
||
约束(not_null、pk、unique、fks)只拿来让差异说明少说(表结构说某处不同不起作用就不说),不拿来改写、判等价:类型
|
||
亲和、COLLATE、部分唯一索引、MySQL 的隐式转换都会让约束说的「唯一」「对得上」不成立(见 _finish)。"""
|
||
|
||
mapping: dict[str, dict[str, str]] | None = None
|
||
types: dict[str, dict[str, str]] = field(default_factory=dict)
|
||
columns: set[str] = field(default_factory=set)
|
||
not_null: dict[str, set[str]] = field(default_factory=dict)
|
||
pk: dict[str, list[str]] = field(default_factory=dict)
|
||
unique: dict[str, list[list[str]]] = field(default_factory=dict)
|
||
fks: dict[str, list[tuple[list[str], str, list[str]]]] = field(default_factory=dict)
|
||
|
||
|
||
def _schema_info(schema: Mapping[str, Any] | None) -> _Schema:
|
||
"""schema_cache → _Schema。qualify 用的类型一律记 UNKNOWN:只要列名(认列属于哪张表、展开 SELECT *);按类型做
|
||
的改写和方言有关,不做。nullable 为 False 的列、主键列另记下来(哪些主键列确定不为空和方言有关,见
|
||
_Schema.not_null 和 _not_null_column)。"""
|
||
tables = schema.get("tables") if isinstance(schema, Mapping) else None
|
||
if not isinstance(tables, Mapping):
|
||
return _Schema()
|
||
mapping: dict[str, dict[str, str]] = {}
|
||
types: dict[str, dict[str, str]] = {}
|
||
not_null: dict[str, set[str]] = {}
|
||
pk: dict[str, list[str]] = {}
|
||
unique: dict[str, list[list[str]]] = {}
|
||
fks: dict[str, list[tuple[list[str], str, list[str]]]] = {}
|
||
for name, meta in tables.items():
|
||
columns = meta.get("columns") if isinstance(meta, Mapping) else None
|
||
cols = [c for c in columns or [] if isinstance(c, Mapping) and isinstance(c.get("name"), str) and c["name"]]
|
||
if cols:
|
||
key = str(name).translate(_ASCII_LOWER)
|
||
mapping[str(name)] = {c["name"]: "UNKNOWN" for c in cols}
|
||
types[key] = {c["name"].translate(_ASCII_LOWER): str(c.get("type") or "") for c in cols}
|
||
not_null[key] = {c["name"].translate(_ASCII_LOWER) for c in cols if c.get("nullable") is False}
|
||
keys = meta.get("primary_key") if isinstance(meta, Mapping) else None
|
||
if isinstance(keys, list) and all(isinstance(k, str) for k in keys):
|
||
pk[key] = [k.translate(_ASCII_LOWER) for k in keys]
|
||
groups = meta.get("unique") if isinstance(meta, Mapping) else None
|
||
if isinstance(groups, list):
|
||
unique[key] = [[c.translate(_ASCII_LOWER) for c in g] for g in groups
|
||
if isinstance(g, list) and g and all(isinstance(c, str) for c in g)]
|
||
refs = meta.get("foreign_keys") if isinstance(meta, Mapping) else None
|
||
for fk in refs if isinstance(refs, list) else []:
|
||
if not isinstance(fk, Mapping) or not isinstance(fk.get("to_table"), str):
|
||
continue
|
||
cols = [c.translate(_ASCII_LOWER) for c in fk.get("columns") or [] if isinstance(c, str)]
|
||
to = [c.translate(_ASCII_LOWER) for c in fk.get("to_columns") or [] if isinstance(c, str)]
|
||
if cols:
|
||
fks.setdefault(key, []).append((cols, fk["to_table"].translate(_ASCII_LOWER), to))
|
||
return _Schema(mapping or None, types, {c for cols in types.values() for c in cols}, not_null, pk, unique, fks)
|
||
|
||
|
||
def _not_null_column(info: _Schema, table: str, column: str, read: str | None) -> bool:
|
||
"""表结构里确定不会为空的列:声明了 NOT NULL;或者是主键——PostgreSQL / MySQL 的主键列都不为空,SQLite 只有单独
|
||
一列、声明的类型正好是 INTEGER 的主键(rowid 的别名)不为空,别的主键因为历史原因能存空值。"""
|
||
t, c = table.translate(_ASCII_LOWER), column.translate(_ASCII_LOWER)
|
||
if c in info.not_null.get(t, set()):
|
||
return True
|
||
keys = info.pk.get(t) or []
|
||
if c not in keys:
|
||
return False
|
||
if read == "sqlite":
|
||
return len(keys) == 1 and info.types.get(t, {}).get(c, "").strip().upper() == "INTEGER"
|
||
return read in ("postgres", "mysql")
|
||
|
||
|
||
def _canonical_forms(root: Any, read: str | None, info: _Schema) -> dict[str, _Form]:
|
||
"""一条语句的几级规范形式:
|
||
|
||
- optimizer:认列(sqlglot 的 qualify:认清每一列属于哪个来源,给了表结构就展开 SELECT *)之后的全套改写(见
|
||
_optimized、_finish)。认不全列、有和列别名同名又可能是表的列的名字、不确定的函数、递归 CTE、LATERAL 之类的
|
||
不出这一级。
|
||
- light:不认列,只做不需要认列的改写(见 _light)。
|
||
- parsed:解析后只统一名字的大小写、引号(见 _normalize_names),总是有。
|
||
|
||
每一级的每一步都只做保持语义的变换,某一级出不来就没有这一级,从不抛异常。"""
|
||
out: dict[str, _Form] = {}
|
||
for level, build in ((_OPTIMIZER, _optimized), (_LIGHT, _light)):
|
||
try:
|
||
out[level] = build(root, read, info)
|
||
except _Unsupported as e:
|
||
logger.debug("SQL 规范化(%s)跳过:%s", level, e)
|
||
except Exception as e: # noqa: BLE001 - sqlglot 认不了列、写法不支持:这一级不出,别的级照样比
|
||
from sqlglot.errors import SqlglotError
|
||
|
||
if isinstance(e, SqlglotError):
|
||
logger.debug("SQL 规范化(%s)跳过:%s", level, e)
|
||
else:
|
||
logger.warning("SQL 规范化(%s)失败", level, exc_info=True)
|
||
try:
|
||
tree = root.copy()
|
||
_normalize_names(tree, read, info)
|
||
out[_PARSED] = _Form(_PARSED, tree.sql(dialect=read, comments=False), tree)
|
||
except Exception: # noqa: BLE001
|
||
logger.warning("SQL 重新生成失败", exc_info=True)
|
||
return out
|
||
|
||
|
||
def _optimized(root: Any, read: str | None, info: _Schema) -> _Form:
|
||
"""认列之后的规范形式:统一名字 → 列名表写进定义 → USING 写成 ON → FETCH 写成 LIMIT → qualify 认列(给了表结构
|
||
先按表结构,表结构里没有的列再不给表结构认一遍)→ 来源别名改成整条语句里互不相同的 → 只引用一次的 CTE 写回用到
|
||
它的地方 → 只筛行、透列的子查询并进外层 → 去掉没用的 CTE → 两级共用的几步(_finish)。"""
|
||
from sqlglot.errors import OptimizeError
|
||
from sqlglot.optimizer.qualify import qualify
|
||
|
||
tree = root.copy()
|
||
why = _unsupported(tree, read) or ("不确定的函数" if _volatile(tree) else None) or \
|
||
("派生来源的列同名" if _duplicate_outputs(tree) else None) or \
|
||
("别名被子查询遮住" if _shadowed_reference(tree, info) else None)
|
||
if why:
|
||
raise _Unsupported(why)
|
||
_normalize_names(tree, read, info)
|
||
_remember_names(tree)
|
||
_mark_unsure(tree)
|
||
_push_column_lists(tree)
|
||
_expand_using(tree)
|
||
_qualify_stars(tree)
|
||
_fetch_to_limit(tree)
|
||
_normalize_negation(tree)
|
||
qualified = None
|
||
for mapping in ([info.mapping, None] if info.mapping else [None]):
|
||
if _alias_or_column(tree, read, mapping):
|
||
continue
|
||
try:
|
||
qualified = qualify(tree.copy(), dialect=read, schema=mapping, validate_qualify_columns=True,
|
||
quote_identifiers=False, identify=False, expand_stars=True, expand_alias_refs=True)
|
||
break
|
||
except OptimizeError as e:
|
||
logger.debug("SQL 认列失败:%s", e)
|
||
if qualified is None:
|
||
raise _Unsupported("认不全列")
|
||
tree = qualified
|
||
if _duplicate_outputs(tree):
|
||
raise _Unsupported("派生来源的列同名(展开 * 之后)")
|
||
_split_simple_ctes(tree, read, info)
|
||
_inline_ctes(tree)
|
||
# 写回的 CTE 定义各是一份副本,里面的别名一样:同一个 CTE 写回外层和相关子查询两处,并进去之后子查询里本来指外层
|
||
# 的列就被里层同名的别名遮住了。写回之后再把整条语句的别名改成互不相同的
|
||
_unique_aliases(tree)
|
||
_merge_derived(tree, read, info)
|
||
_push_into_groups(tree, read, info)
|
||
_drop_unused_ctes(tree)
|
||
return _finish(tree, read, info, level=_OPTIMIZER, qualified=True)
|
||
|
||
|
||
def _light(root: Any, read: str | None, info: _Schema) -> _Form:
|
||
"""不认列的规范形式:统一名字、列名表写进定义、USING 写成 ON、FETCH 写成 LIMIT、只引用一次的 CTE 写回用到它的
|
||
地方,再走 _finish 里不需要认列的几步。"""
|
||
tree = root.copy()
|
||
why = _unsupported(tree, read) or ("不确定的函数" if _volatile(tree) else None) or \
|
||
("别名被子查询遮住" if _shadowed_reference(tree, info) else None)
|
||
if why:
|
||
raise _Unsupported(why)
|
||
_normalize_names(tree, read, info)
|
||
_remember_names(tree)
|
||
_mark_unsure(tree)
|
||
_push_column_lists(tree)
|
||
_expand_using(tree)
|
||
_qualify_stars(tree)
|
||
_fetch_to_limit(tree)
|
||
_normalize_negation(tree)
|
||
_inline_ctes(tree)
|
||
_drop_unused_ctes(tree)
|
||
return _finish(tree, read, info, level=_LIGHT, qualified=False)
|
||
|
||
|
||
def _finish(tree: Any, read: str | None, info: _Schema, *, level: str, qualified: bool) -> _Form:
|
||
"""两级共用的几步,都只做保持语义的变换(为什么保持语义见各函数的说明),按这个先后:
|
||
|
||
0. 原样选出 FROM 里一个子查询全部列、什么别的都不做的一段换成那个子查询(_unwrap_passthrough)。
|
||
1. 动来源的:两个来源的 RIGHT JOIN 写成 LEFT JOIN;WHERE 把外连接可空一侧没关联上的行筛掉了的写成内连接;只取可空
|
||
一侧跳过空值的聚合时 LEFT JOIN 写成内连接;开头那组内连接的条件挪进 WHERE、写成逗号连接;恒为真的条件、同一组
|
||
AND 里另一项已经保证不为空的 IS NOT NULL 去掉;没用到、又不怕重复的 LEFT JOIN 去掉;(认列之后)按关联条件
|
||
相等、类型一样的两列换成同一列。结果看行出来的先后的一段(见 _Sensitivity)这些都不做——只有挪内连接的 ON
|
||
条件、去掉 1 = 1 这类常量条件照做(库里一视同仁,扫描的先后不变)。表结构里的约束(主键、唯一约束、NOT NULL)
|
||
一律不拿来改写:SQLite 的类型亲和(TEXT 主键对数值列,'1' 和 '01' 都等于 1)、比较写了 COLLATE、部分唯一索引
|
||
(只管没删的行,探查照样记成 unique)、MySQL 拿字符串和数值比时的隐式转换都会让「唯一」不成立,表结构里看不全,
|
||
宁可认不出。
|
||
2. 只看有没有、在不在的子查询:EXISTS 写成 SELECT 1,IN 子查询里不影响取值集合的去重、分组、排序、UNION 去掉。
|
||
3. 去重的:SELECT DISTINCT 写成按选出来的列 GROUP BY、已经按选出来的列分组的 DISTINCT 去掉(只在「相等」就是
|
||
「一样」、不看行出来的先后时做)。
|
||
4. 和来源无关的:COUNT(1) 写成 COUNT(*)、MIN / MAX 里的 DISTINCT 去掉、IIF / IF 写成 CASE;IN 列表去重排序、只有一个
|
||
常量的写成 =;BETWEEN 写成两个比较;NOT 套比较写成反过来的比较、NOT NOT 去掉;LIMIT 的同义写法;不影响结合的
|
||
括号、ORDER BY 的 ASC、OFFSET 0、重复的排序键去掉;最外层 ORDER BY 引用的别名换成表达式,最外层和条件里的子查询
|
||
选出来的列名去掉。
|
||
5. 起规范的名字、排好交换律成立的地方(见 _named)。
|
||
"""
|
||
tree = _unwrap_passthrough(tree, read)
|
||
sens = _Sensitivity(tree, read, info)
|
||
_right_to_left(tree, sens)
|
||
sens = _Sensitivity(tree, read, info)
|
||
_fold_not(tree, sens)
|
||
_reject_nulls(tree, read, sens)
|
||
_null_blind_left_joins(tree, read, sens)
|
||
_flatten_prefix(tree, sens, read)
|
||
_drop_true_conjuncts(tree)
|
||
_drop_implied_not_null(tree, read, sens)
|
||
_drop_unused_left_joins(tree, sens)
|
||
if qualified:
|
||
_substitute_equal_columns(tree, read, info, sens)
|
||
_exists_subqueries(tree)
|
||
sens = _Sensitivity(tree, read, info)
|
||
_membership_subqueries(tree, sens)
|
||
_distinct_to_group(tree, sens)
|
||
_count_star(tree)
|
||
_minmax_distinct(tree, sens)
|
||
_case_forms(tree, read)
|
||
_in_lists(tree)
|
||
_expand_between(tree, read, sens)
|
||
_fold_not(tree, sens)
|
||
_normalize_limits(tree, read)
|
||
_strip_parens(tree)
|
||
_strip_asc(tree)
|
||
_zero_offset(tree)
|
||
_order_by_aliases(tree)
|
||
_dedupe_order(tree, sens)
|
||
_drop_output_aliases(tree)
|
||
pre = tree.copy()
|
||
sens = _Sensitivity(tree, read, info)
|
||
tree = _named(tree, read, qualified, sens)
|
||
return _Form(level, tree.sql(dialect=read, comments=False), tree, pre, qualified)
|
||
|
||
|
||
def _named(tree: Any, read: str | None, qualified: bool, sens: "_Sensitivity", hint: "_Hint | None" = None) -> Any:
|
||
"""起规范的名字、排好交换律成立的地方,最后统一引号(只给需要的标识符加)。"""
|
||
from sqlglot.optimizer.qualify_columns import quote_identifiers
|
||
|
||
_Namer(tree, read, qualified, sens, hint).canon(tree, 0, {})
|
||
return quote_identifiers(tree, dialect=read, identify=False)
|
||
|
||
|
||
# ---- 守门:哪些结构不走规范化的两级
|
||
|
||
def _volatile(tree: Any) -> bool:
|
||
from sqlglot import exp
|
||
|
||
for node in tree.walk():
|
||
if isinstance(node, (exp.Rand, exp.Randn)) or type(node).__name__.lower() in _VOLATILE:
|
||
return True
|
||
if isinstance(node, exp.Anonymous) and str(node.name).lower() in _VOLATILE:
|
||
return True
|
||
return False
|
||
|
||
|
||
def _unsupported(tree: Any, read: str | None) -> str | None:
|
||
"""规范化的两级不处理的结构:递归 CTE(写了 RECURSIVE,或者 CTE 引用自己、引用后面的 CTE——SQLite 里不写
|
||
RECURSIVE 也是递归)、LATERAL、UNNEST、PIVOT、表函数、VALUES、SEMI / ANTI / ASOF / NATURAL 这些连接、实表别名
|
||
带列名(PostgreSQL 的 orders AS o(a, b) 给列改了名)、BETWEEN SYMMETRIC、FETCH … WITH TIES / PERCENT、ROLLUP /
|
||
CUBE / GROUPING SETS、SELECT INTO、加锁。"""
|
||
from sqlglot import exp
|
||
|
||
for node in tree.walk():
|
||
if isinstance(node, exp.With):
|
||
if node.args.get("recursive"):
|
||
return "递归 CTE"
|
||
names = [c.alias_or_name for c in node.expressions]
|
||
for i, cte in enumerate(node.expressions):
|
||
later = set(names[i:])
|
||
if any(t.name in later and not t.args.get("db") for t in cte.this.find_all(exp.Table)):
|
||
return "CTE 引用自己或后面的 CTE"
|
||
if isinstance(node, (exp.Lateral, exp.Unnest, exp.Pivot, exp.TableSample, exp.Values, exp.Into, exp.Lock)):
|
||
return type(node).__name__
|
||
if isinstance(node, exp.Table):
|
||
if not isinstance(node.this, exp.Identifier):
|
||
return "表函数"
|
||
alias = node.args.get("alias")
|
||
if isinstance(alias, exp.TableAlias) and alias.columns:
|
||
return "实表别名带列名"
|
||
if isinstance(node, exp.Join) and (node.args.get("method") or str(node.kind or "").upper() in
|
||
("SEMI", "ANTI", "ASOF", "POSITIONAL", "STRAIGHT_JOIN", "ARRAY")):
|
||
return "连接方式"
|
||
if isinstance(node, exp.Between) and node.args.get("symmetric"):
|
||
return "BETWEEN SYMMETRIC"
|
||
if isinstance(node, exp.Fetch):
|
||
options = node.args.get("limit_options")
|
||
if options is not None and (options.args.get("percent") or options.args.get("with_ties")):
|
||
return "FETCH WITH TIES / PERCENT"
|
||
if isinstance(node, exp.Group) and any(node.args.get(k) for k in ("rollup", "cube", "grouping_sets", "totals")):
|
||
return "ROLLUP / CUBE / GROUPING SETS"
|
||
return None
|
||
|
||
|
||
def _duplicate_outputs(tree: Any) -> bool:
|
||
"""FROM 里的子查询、CTE 选出来的列有同名的(SELECT user_id AS k, SUM(amount) AS k …):外面按名字引用 k 时 SQLite
|
||
认第一个、PostgreSQL / MySQL 报错,按名字找定义的改写(外层条件挪进 HAVING、并子查询、展开 *)都可能找错。有就
|
||
不走认列那一级。"""
|
||
from sqlglot import exp
|
||
|
||
for holder in tree.find_all(exp.CTE, exp.Subquery):
|
||
if isinstance(holder, exp.Subquery) and not isinstance(holder.parent, (exp.From, exp.Join)):
|
||
continue
|
||
leaves = _leaves(holder.this)
|
||
if leaves and isinstance(leaves[0], exp.Select):
|
||
names = [p.alias_or_name for p in leaves[0].expressions if p.alias_or_name and not _is_star(p)]
|
||
if len(names) != len(set(names)):
|
||
return True
|
||
return False
|
||
|
||
|
||
def _shadowed_reference(tree: Any, info: _Schema) -> bool:
|
||
"""子查询里写了表名(别名)的列,而这个别名在子查询和外层各指一个不同的来源(FROM orders o … EXISTS (SELECT 1 FROM
|
||
refunds AS o WHERE o.status = 'paid')):SQLite 先在里层的 o 里找 status,找不到就往外层找(真实的 SQLite 上
|
||
试过),sqlglot 一律认里层的。里层是实表、表结构里有这一列(或者是派生来源、选出来的列里有它)时才确定是里层的;
|
||
认不准就不走规范化的两级。同一张表在里外层用同一个别名的(FROM orders WHERE … (SELECT … FROM orders))不算:
|
||
里外层的列一样。"""
|
||
from sqlglot import exp
|
||
|
||
def signature(item: Any, ctes: dict[str, Any]) -> Any:
|
||
if isinstance(item, exp.Table) and isinstance(item.this, exp.Identifier):
|
||
key = item.name.translate(_ASCII_LOWER)
|
||
if not item.args.get("db") and key in ctes:
|
||
return "cte", id(ctes[key])
|
||
return "table", ".".join(p for p in (item.catalog, item.db, item.name) if p).translate(_ASCII_LOWER)
|
||
return "derived", id(item)
|
||
|
||
def certain(item: Any, column: str, ctes: dict[str, Any]) -> bool:
|
||
kind, _ = signature(item, ctes)
|
||
if kind == "table":
|
||
columns = info.types.get(item.name.translate(_ASCII_LOWER))
|
||
return columns is not None and column.translate(_ASCII_LOWER) in columns
|
||
body = item.this if isinstance(item, exp.Subquery) else ctes[item.name.translate(_ASCII_LOWER)].this
|
||
return any(n is not None and n == column for n in _output_names(body))
|
||
|
||
def walk(query: Any, frames: list[dict[str, Any]], ctes: dict[str, Any]) -> bool:
|
||
if isinstance(query, exp.Subquery):
|
||
return walk(query.this, frames, ctes)
|
||
with_ = query.args.get("with_")
|
||
if with_ is not None:
|
||
ctes = dict(ctes)
|
||
for cte in with_.expressions:
|
||
if walk(cte.this, frames, ctes):
|
||
return True
|
||
ctes[cte.alias_or_name.translate(_ASCII_LOWER)] = cte
|
||
if isinstance(query, exp.SetOperation):
|
||
return walk(query.left, frames, ctes) or walk(query.right, frames, ctes)
|
||
if not isinstance(query, exp.Select):
|
||
return False
|
||
frame = {item.alias_or_name: item for item in _items_of(query) if item.alias_or_name}
|
||
for item in _items_of(query):
|
||
if isinstance(item, exp.Subquery) and walk(item.this, frames, ctes):
|
||
return True
|
||
stack = frames + [frame]
|
||
for node in _block_nodes(query):
|
||
if isinstance(node, exp.Column) and node.table:
|
||
defs = [f[node.table] for f in stack if node.table in f]
|
||
if len({signature(d, ctes) for d in defs}) > 1 and not certain(defs[-1], node.name, ctes):
|
||
return True
|
||
return any(walk(sub, stack, ctes) for sub in _nested_queries(query))
|
||
|
||
return walk(tree, [], {})
|
||
|
||
|
||
# ---- 名字
|
||
|
||
def _identifier_role(ident: Any) -> str:
|
||
"""一个标识符在语法树里是什么:column(列名)/ column_alias(列的别名、CTE 和子查询的列名表)/ table(表名、
|
||
列前面的表名或别名)/ table_alias(表、子查询、CTE 的别名)/ other。"""
|
||
from sqlglot import exp
|
||
|
||
parent, key = ident.parent, ident.arg_key
|
||
if isinstance(parent, exp.Column):
|
||
return "column" if key == "this" else "table"
|
||
if isinstance(parent, exp.Table):
|
||
return "table"
|
||
if isinstance(parent, exp.TableAlias):
|
||
return "column_alias" if key == "columns" else "table_alias"
|
||
if isinstance(parent, exp.Alias):
|
||
return "column_alias"
|
||
return "other"
|
||
|
||
|
||
def _visible_ctes(node: Any) -> dict[str, Any]:
|
||
"""node 所在位置看得见的 CTE(名字 → CTE),里层的遮住外层的。"""
|
||
from sqlglot import exp
|
||
|
||
chain = []
|
||
cur = node
|
||
while cur is not None:
|
||
with_ = cur.args.get("with_") if isinstance(cur, exp.Expr) else None
|
||
if with_ is not None:
|
||
chain.append(with_)
|
||
cur = cur.parent
|
||
out: dict[str, Any] = {}
|
||
for with_ in reversed(chain):
|
||
for cte in with_.expressions:
|
||
out[cte.alias_or_name.translate(_ASCII_LOWER)] = cte
|
||
return out
|
||
|
||
|
||
def _column_in_scope(col: Any, name: str, info: _Schema) -> bool:
|
||
"""SQLite 里不带表名的名字 name(已转小写)在 col 所在的这一段、外层各段的来源里确实是一列:实表在表结构里、有这一
|
||
列;CTE、FROM 里的子查询选出来的列(或者写的列名表)里有这个名字。认不出来源的列(没给表结构、选的是 *)一律当成
|
||
不知道(返回 False):调用方就原样留着,不会把可能是字符串的 "State" 和 "state" 认成同一列。"""
|
||
from sqlglot import exp
|
||
|
||
ctes = _visible_ctes(col)
|
||
node = col.parent
|
||
while node is not None:
|
||
if isinstance(node, exp.Select):
|
||
for item in _items_of(node):
|
||
body = None
|
||
if isinstance(item, exp.Subquery):
|
||
body = item.this
|
||
elif isinstance(item, exp.Table) and not item.args.get("db") and \
|
||
item.name.translate(_ASCII_LOWER) in ctes:
|
||
body = ctes[item.name.translate(_ASCII_LOWER)].this
|
||
if body is not None:
|
||
names = _output_names(body)
|
||
if any(n is not None and n.translate(_ASCII_LOWER) == name for n in names):
|
||
return True
|
||
continue
|
||
if isinstance(item, exp.Table):
|
||
columns = info.types.get(item.name.translate(_ASCII_LOWER))
|
||
if columns is not None and name in columns:
|
||
return True
|
||
node = node.parent
|
||
return False
|
||
|
||
|
||
def _normalize_names(tree: Any, read: str | None, info: _Schema) -> None:
|
||
"""按方言的规则统一名字的大小写和引号,只做库里认成同一个名字的改写;处理过的标识符标成 case_sensitive,后面
|
||
qualify 不再按 sqlglot 的规则改(sqlglot 认为 SQLite 带引号的名字也不分大小写,可 SQLite 找不到这一列时把双引号
|
||
里的当成字符串,'Paid' 和 'paid' 就不同了)。只认 ASCII 字母的大小写(库里也只折 ASCII)。
|
||
|
||
- SQLite:不带引号的名字、带引号的表名和别名不分大小写,统一成小写;条件、SELECT 列表里不带表名、带引号的列名
|
||
("Paid")原样留着,除非它所在的这一段或外层各段的来源里确实有这一列(见 _column_in_scope:找得到这一列就不会被
|
||
当成字符串)。只看这条语句用到的表:表结构里别的表有同名的列不算(orders 没有 state 列时 "State" 是字符串,
|
||
refunds 有 state 列不相干)。
|
||
- PostgreSQL(和认不出的方言):不带引号的名字折成小写;带引号的里面没有大写字母就和不带引号的是同一个名字。
|
||
- MySQL:列名、列的别名不分大小写,统一成小写;表名、表的别名在 Linux 上分大小写,原样。
|
||
"""
|
||
from sqlglot import exp
|
||
|
||
for ident in list(tree.find_all(exp.Identifier)):
|
||
name = ident.this
|
||
if not isinstance(name, str):
|
||
continue
|
||
role = _identifier_role(ident)
|
||
quoted = bool(ident.args.get("quoted"))
|
||
lower = name.translate(_ASCII_LOWER)
|
||
if read == "sqlite":
|
||
keep = quoted and role == "column" and not ident.parent.args.get("table") and \
|
||
not _column_in_scope(ident.parent, lower, info)
|
||
if keep:
|
||
ident.meta[_UNSURE] = True
|
||
else:
|
||
name, quoted = lower, False
|
||
elif read == "mysql":
|
||
if role in ("column", "column_alias"):
|
||
name = lower
|
||
quoted = False
|
||
elif not quoted or lower == name:
|
||
name, quoted = lower, False
|
||
if quoted or not _PLAIN_NAME.match(name):
|
||
quoted = True
|
||
ident.set("this", name)
|
||
ident.set("quoted", quoted)
|
||
ident.meta["case_sensitive"] = True
|
||
|
||
|
||
def _mark_unsure(tree: Any) -> None:
|
||
"""条件里的子查询(IN、EXISTS、标量)里不带表名的列标成认不准(见 _UNSURE):它可能是子查询里的表的列,也可能是
|
||
外层的列,表结构里没有就分不清。只影响差异说明说不说具体的,不影响规范形式。"""
|
||
from sqlglot import exp
|
||
|
||
for col in tree.find_all(exp.Column):
|
||
if col.table or not isinstance(col.this, exp.Identifier):
|
||
continue
|
||
node = col.parent
|
||
while node is not None:
|
||
if isinstance(node, exp.Select) and (_scalar_context(node) or _condition_query(node)):
|
||
col.this.meta[_UNSURE] = True
|
||
break
|
||
node = node.parent
|
||
|
||
|
||
def _remember_names(tree: Any) -> None:
|
||
"""CTE、FROM 里的子查询原来的名字记在里面那段查询上(见 _ORIGINAL)。"""
|
||
from sqlglot import exp
|
||
|
||
for cte in tree.find_all(exp.CTE):
|
||
cte.this.meta.setdefault(_ORIGINAL, cte.alias_or_name)
|
||
for sub in tree.find_all(exp.Subquery):
|
||
if isinstance(sub.parent, (exp.From, exp.Join)) and sub.alias:
|
||
sub.this.meta.setdefault(_ORIGINAL, sub.alias)
|
||
|
||
|
||
def _push_column_lists(tree: Any) -> None:
|
||
"""CTE、FROM 里的子查询自己写了列名表的(WITH p(uid, total) AS (…)、(…) AS t(x, y)),列名写成里面那段(UNION 是
|
||
第一支)选出来的列的别名,列名表去掉:外面按这些名字引用,两种写法是一回事。里面那段选的是 *(不知道有几列)、
|
||
列数对不上、本段 ORDER BY / GROUP BY / HAVING 还按原来的别名引用的,没法改写,不走规范化的两级。"""
|
||
from sqlglot import exp
|
||
|
||
for holder in list(tree.find_all(exp.CTE, exp.Subquery)):
|
||
alias = holder.args.get("alias")
|
||
if not isinstance(alias, exp.TableAlias) or not alias.columns:
|
||
continue
|
||
body = holder.this
|
||
target = _leaves(body)[0] if isinstance(body, exp.SetOperation) else body
|
||
names = [c.name for c in alias.columns]
|
||
if not isinstance(target, exp.Select) or _starry(target) or len(target.expressions) != len(names):
|
||
raise _Unsupported("列名表对不上选出来的列")
|
||
own = {p.alias for p in target.expressions if isinstance(p, exp.Alias) and p.alias}
|
||
if isinstance(body, exp.SetOperation):
|
||
if body.args.get("order") is not None:
|
||
raise _Unsupported("合并的结果按名字排序")
|
||
# 本段的 WHERE、ON、GROUP BY、HAVING、ORDER BY、子查询里按名字引用原来的列别名、或者正好叫新名字的(SQLite 的
|
||
# WHERE 认列别名:WITH t(a, b) AS (SELECT amount AS b, id AS a FROM orders WHERE b > 50) 里的 b 是 amount),
|
||
# 改名之后就指到别的列上了
|
||
if _alias_refs(target) & (own | set(names)):
|
||
raise _Unsupported("列名表改名之后本段的引用对不上")
|
||
target.set("expressions", [exp.alias_(p.unalias().copy() if isinstance(p, exp.Alias) else p.copy(), n,
|
||
copy=False) for p, n in zip(target.expressions, names)])
|
||
alias.set("columns", None)
|
||
|
||
|
||
def _expand_using(tree: Any) -> None:
|
||
"""JOIN … USING (c) 写成 ON 左表.c = 右表.c,这一段里不带表名的 c 写成左表.c(内连接、LEFT JOIN 时合并出来的 c
|
||
就是左表的 c)。只改得了「FROM 只有一个来源、第一个连接就是 USING、SELECT 里没有 *」的写法:别的(左边好几个
|
||
来源时 c 是哪一张的、SELECT * 列的先后、RIGHT / FULL JOIN 合并出来的值、子查询里引用 c)没法照原样改写,不走
|
||
规范化的两级。"""
|
||
from sqlglot import exp
|
||
|
||
for select in list(tree.find_all(exp.Select)):
|
||
joins = select.args.get("joins") or []
|
||
if not any(j.args.get("using") for j in joins):
|
||
continue
|
||
if len(joins) < 1 or not joins[0].args.get("using") or any(j.args.get("using") for j in joins[1:]):
|
||
raise _Unsupported("USING")
|
||
join = joins[0]
|
||
if str(join.side or "").upper() not in ("", "LEFT") or join.args.get("on") is not None or \
|
||
any(_is_star(p) for p in select.expressions):
|
||
raise _Unsupported("USING")
|
||
left, right = select.args["from_"].this, join.this
|
||
la, ra = left.alias_or_name, right.alias_or_name
|
||
if not la or not ra:
|
||
raise _Unsupported("USING")
|
||
names = [u.name for u in join.args["using"]]
|
||
for sub in _nested_queries(select):
|
||
if any(not c.table and c.name in names for c in sub.find_all(exp.Column)):
|
||
raise _Unsupported("子查询里引用 USING 的列")
|
||
# 有列别名和 USING 的列同名:ORDER BY(SQLite 的 WHERE、GROUP BY、HAVING 也可能)里单独一个名字先认列别名,
|
||
# SELECT -o.id AS id … USING (id) ORDER BY id 是按 -o.id 排,不能写成左表的 id
|
||
if any(isinstance(p, exp.Alias) and p.alias in names for p in select.expressions):
|
||
raise _Unsupported("列别名和 USING 的列同名")
|
||
for node in list(_block_nodes(select)):
|
||
if isinstance(node, exp.Column) and not node.table and node.name in names:
|
||
node.set("table", exp.to_identifier(la))
|
||
conds = [exp.EQ(this=exp.column(n, table=la), expression=exp.column(n, table=ra)) for n in names]
|
||
join.set("using", None)
|
||
join.set("on", _and_all(conds))
|
||
|
||
|
||
def _qualify_stars(tree: Any) -> None:
|
||
"""几个来源的 SELECT * 写成按 FROM 的先后逐个来源的 t.*:两种写法是一回事(没有 USING / NATURAL 时 * 就是各来源的
|
||
全部列依次排开)。sqlglot 的 qualify 展开 * 时把实表的列排在 FROM 里的子查询前面,和库里的先后不一样,先写成 t.*
|
||
它就按写的先后展开。有来源没有名字的不动(这时也不走规范化的两级)。"""
|
||
from sqlglot import exp
|
||
|
||
for select in list(tree.find_all(exp.Select)):
|
||
if not any(isinstance(p, exp.Star) for p in select.expressions):
|
||
continue
|
||
items = _items_of(select)
|
||
if len(items) < 2 or any(not i.alias_or_name for i in items):
|
||
continue
|
||
stars = [exp.Column(this=exp.Star(), table=exp.to_identifier(i.alias_or_name)) for i in items]
|
||
new: list[Any] = []
|
||
for p in select.expressions:
|
||
new += [c.copy() for c in stars] if isinstance(p, exp.Star) else [p]
|
||
select.set("expressions", new)
|
||
|
||
|
||
def _fetch_to_limit(tree: Any) -> None:
|
||
"""FETCH FIRST n ROWS ONLY(不带 WITH TIES / PERCENT)就是 LIMIT n(sqlglot 把 FETCH 放在 limit 这个参数上)。"""
|
||
from sqlglot import exp
|
||
|
||
for query in list(tree.find_all(exp.Select, exp.SetOperation)):
|
||
for key in ("limit", "fetch"):
|
||
fetch = query.args.get(key)
|
||
if not isinstance(fetch, exp.Fetch):
|
||
continue
|
||
count = fetch.args.get("count") or exp.Literal.number(1)
|
||
query.set(key, None)
|
||
query.set("limit", exp.Limit(expression=count.copy()))
|
||
|
||
|
||
# ---- 认列属于哪个来源
|
||
|
||
class _Resolver:
|
||
"""按别名认清每一列属于哪个来源(相关子查询里的列到外层找):refs[id(来源)] 是引用它的列,cte_of[id(表)] 是它
|
||
引用的 CTE,owner[id(来源)] 是来源所在的那段 SELECT。不带表名的列不认(轻量一级)。"""
|
||
|
||
def __init__(self, tree: Any) -> None:
|
||
self.refs: dict[int, list[Any]] = {}
|
||
self.cte_of: dict[int, Any] = {}
|
||
self.owner: dict[int, Any] = {}
|
||
self.source_of: dict[int, Any] = {}
|
||
self._walk(tree, [], {})
|
||
|
||
def _walk(self, query: Any, frames: list[dict[str, Any]], ctes: dict[str, Any]) -> None:
|
||
from sqlglot import exp
|
||
|
||
if isinstance(query, exp.Subquery):
|
||
self._walk(query.this, frames, ctes)
|
||
return
|
||
with_ = query.args.get("with_")
|
||
if with_ is not None:
|
||
ctes = dict(ctes)
|
||
for cte in with_.expressions:
|
||
self._walk(cte.this, frames, ctes)
|
||
ctes[cte.alias_or_name] = cte
|
||
if isinstance(query, exp.SetOperation):
|
||
self._walk(query.left, frames, ctes)
|
||
self._walk(query.right, frames, ctes)
|
||
return
|
||
if not isinstance(query, exp.Select):
|
||
raise _Unsupported(type(query).__name__)
|
||
frame: dict[str, Any] = {}
|
||
for item in _items_of(query):
|
||
if isinstance(item, exp.Subquery):
|
||
self._walk(item.this, frames, ctes)
|
||
elif isinstance(item, exp.Table) and isinstance(item.this, exp.Identifier):
|
||
if not item.args.get("db") and item.name in ctes:
|
||
self.cte_of[id(item)] = ctes[item.name]
|
||
else:
|
||
raise _Unsupported(type(item).__name__)
|
||
name = item.alias_or_name
|
||
if not name or name in frame:
|
||
raise _Unsupported("来源没有名字或重名")
|
||
frame[name] = item
|
||
self.owner[id(item)] = query
|
||
self.refs.setdefault(id(item), [])
|
||
stack = frames + [frame]
|
||
for node in _block_nodes(query):
|
||
if isinstance(node, exp.Column) and node.table:
|
||
for f in reversed(stack):
|
||
if node.table in f:
|
||
self.refs[id(f[node.table])].append(node)
|
||
self.source_of[id(node)] = f[node.table]
|
||
break
|
||
for sub in _nested_queries(query):
|
||
self._walk(sub, stack, ctes)
|
||
|
||
def body(self, item: Any) -> Any:
|
||
"""派生来源里面那段查询(FROM 里的子查询、引用的 CTE);实表是 None。"""
|
||
from sqlglot import exp
|
||
|
||
if isinstance(item, exp.Subquery):
|
||
return item.this
|
||
cte = self.cte_of.get(id(item))
|
||
return cte.this if cte is not None else None
|
||
|
||
|
||
def _unique_aliases(tree: Any) -> None:
|
||
"""(认列之后)整条语句里每个来源的别名改成互不相同的(_s1、_s2…),引用它的列跟着改:按原来的别名认清每一列
|
||
属于哪个来源之后再改,相关子查询里引用外层的列不会认错;之后并子查询、写回 CTE 时名字不会撞上。"""
|
||
from sqlglot import exp
|
||
|
||
resolver = _Resolver(tree)
|
||
taken = {t.name for t in tree.find_all(exp.Table)} | {c.alias_or_name for c in tree.find_all(exp.CTE)}
|
||
n = 0
|
||
for item in [i for s in tree.find_all(exp.Select) for i in _items_of(s)]:
|
||
n += 1
|
||
name = f"_s{n}"
|
||
while name in taken:
|
||
name += "_"
|
||
alias = item.args.get("alias")
|
||
if isinstance(alias, exp.TableAlias):
|
||
alias.set("this", exp.to_identifier(name))
|
||
else:
|
||
item.set("alias", exp.TableAlias(this=exp.to_identifier(name)))
|
||
for col in resolver.refs.get(id(item), []):
|
||
col.set("table", exp.to_identifier(name))
|
||
|
||
|
||
def _nested_with(tree: Any) -> bool:
|
||
from sqlglot import exp
|
||
|
||
top = tree.args.get("with_")
|
||
return any(isinstance(n, exp.With) and n is not top for n in tree.walk())
|
||
|
||
|
||
def _inline_ctes(tree: Any) -> None:
|
||
"""只引用一次的非递归顶层 CTE 写回引用它的地方,成为 FROM 里的子查询(别名照旧):写成 CTE 还是子查询是同一个
|
||
意思。语句里还有别的 WITH 的不动(名字可能被里层的 CTE 遮住)。"""
|
||
from sqlglot import exp
|
||
|
||
with_ = tree.args.get("with_")
|
||
if with_ is None or with_.args.get("recursive") or _nested_with(tree):
|
||
return
|
||
changed = True
|
||
while changed and with_.expressions:
|
||
changed = False
|
||
resolver = _Resolver(tree)
|
||
for cte in list(with_.expressions):
|
||
refs = [t for t in tree.find_all(exp.Table) if resolver.cte_of.get(id(t)) is cte]
|
||
if len(refs) != 1:
|
||
continue
|
||
ref = refs[0]
|
||
sub = exp.Subquery(this=cte.this.copy(), alias=exp.TableAlias(this=exp.to_identifier(ref.alias_or_name)))
|
||
ref.replace(sub)
|
||
cte.pop()
|
||
changed = True
|
||
break
|
||
if not with_.expressions:
|
||
tree.set("with_", None)
|
||
|
||
|
||
def _drop_unused_ctes(tree: Any) -> None:
|
||
"""没有被引用的非递归 CTE 去掉:查询语句里没用到的 CTE 不影响结果。"""
|
||
from sqlglot import exp
|
||
|
||
changed = True
|
||
while changed:
|
||
changed = False
|
||
resolver = _Resolver(tree)
|
||
used = {id(c) for c in resolver.cte_of.values()}
|
||
for with_ in list(tree.find_all(exp.With)):
|
||
if with_.args.get("recursive"):
|
||
continue
|
||
for cte in list(with_.expressions):
|
||
if id(cte) not in used:
|
||
cte.pop()
|
||
changed = True
|
||
if not with_.expressions and isinstance(with_.parent, exp.Expr):
|
||
with_.parent.set("with_", None)
|
||
|
||
|
||
def _mergeable(body: Any) -> bool:
|
||
"""FROM 里的子查询能不能并进外层:只从几张实表(内连接)里筛行、选列(没有分组、聚合、去重、排序、取行、窗口函数、
|
||
UNION、WITH,选出来的列里没有子查询),并进去之后每一行、每个值都一样。"""
|
||
from sqlglot import exp
|
||
|
||
if not isinstance(body, exp.Select):
|
||
return False
|
||
if {k for k, v in body.args.items() if v} - {"expressions", "from_", "where", "joins"}:
|
||
return False
|
||
sources = _items_of(body)
|
||
if not sources or any(not isinstance(i, exp.Table) for i in sources):
|
||
return False
|
||
if any(not _inner(j) for j in body.args.get("joins") or []):
|
||
return False
|
||
stars = [p for p in body.expressions if _is_star(p)]
|
||
if stars and (len(body.expressions) != 1 or not isinstance(body.expressions[0], exp.Star) or len(sources) != 1):
|
||
return False
|
||
for p in body.expressions:
|
||
if any(isinstance(n, (exp.AggFunc, exp.Window, exp.Query, exp.Subquery)) for n in p.walk()):
|
||
return False
|
||
names = [p.alias_or_name for p in body.expressions]
|
||
if not stars and (not all(names) or len(set(names)) != len(names)):
|
||
return False
|
||
where = body.args.get("where")
|
||
return where is None or not any(isinstance(n, (exp.AggFunc, exp.Window)) for n in where.walk())
|
||
|
||
|
||
def _merge_derived(tree: Any, read: str | None, info: _Schema) -> None:
|
||
"""(认列之后)只筛行、选列的子查询(见 _mergeable)并进外层:外层引用它的列换成里面的定义,里面的 WHERE 和外层的
|
||
WHERE(和它里面内连接的条件)用 AND 连起来,来源换成里面那几张表。只并内连接一侧(FROM 或内连接进来的)的:外连接
|
||
可空一侧没关联上的行该是空值,并进来就成了算出来的值。结果看行出来的先后的外层(见 _Sensitivity)不并。
|
||
外层有 SELECT * 的:子查询选的就是 *(换成那张表,列一样)或者它是唯一的来源(* 换成它选出来的列)才并。"""
|
||
from sqlglot import exp
|
||
|
||
def substitute(col: Any, expr: Any) -> None:
|
||
new = expr.copy()
|
||
if not isinstance(new, (exp.Column, exp.Literal, exp.Null, exp.Boolean)):
|
||
new = exp.Paren(this=new)
|
||
col.replace(new)
|
||
|
||
for _ in range(64):
|
||
resolver = _Resolver(tree)
|
||
# 浮点数求和这一条不挡并子查询:只筛行选列的子查询三种库自己也会并进外层(PostgreSQL 的 subquery pull-up、
|
||
# MySQL 的 derived_merge、SQLite 的 flattening),并不并执行起来先后一样
|
||
sens = _Sensitivity(tree, read, info, resolver, floats=False)
|
||
done = False
|
||
for select in reversed(list(tree.find_all(exp.Select))):
|
||
if sens.of(select):
|
||
continue
|
||
joins = select.args.get("joins") or []
|
||
if any(str(j.side or "").upper() in ("RIGHT", "FULL") for j in joins):
|
||
continue
|
||
items = _items_of(select)
|
||
for index, item in enumerate(items):
|
||
if not isinstance(item, exp.Subquery) or not _mergeable(item.this):
|
||
continue
|
||
if index and not _inner(joins[index - 1]):
|
||
continue
|
||
body = item.this
|
||
sources = _items_of(body)
|
||
src = sources[0]
|
||
star_body = isinstance(body.expressions[0], exp.Star)
|
||
defs = {} if star_body else {p.alias_or_name: p.unalias() for p in body.expressions}
|
||
refs = resolver.refs.get(id(item), [])
|
||
if not star_body and any(c.name not in defs for c in refs):
|
||
continue
|
||
stars = [p for p in select.expressions if _is_star(p)]
|
||
bare = [p for p in stars if isinstance(p, exp.Star)]
|
||
if bare and not star_body and len(items) != 1:
|
||
continue
|
||
alias = item.alias_or_name
|
||
for col in refs:
|
||
if star_body:
|
||
col.set("table", exp.to_identifier(src.alias_or_name))
|
||
else:
|
||
substitute(col, defs[col.name])
|
||
if not star_body:
|
||
expanded = [exp.alias_(e.copy(), n, copy=False) for n, e in defs.items()]
|
||
new_list = []
|
||
for p in select.expressions:
|
||
if isinstance(p, exp.Star) or (isinstance(p, exp.Column) and p.is_star and p.table == alias):
|
||
new_list += [e.copy() for e in expanded]
|
||
else:
|
||
new_list.append(p)
|
||
select.set("expressions", new_list)
|
||
else:
|
||
for p in stars:
|
||
if isinstance(p, exp.Column) and p.table == alias:
|
||
p.set("table", exp.to_identifier(src.alias_or_name))
|
||
conds = []
|
||
inner_where = body.args.get("where")
|
||
if inner_where is not None:
|
||
conds += _conjuncts(inner_where.this)
|
||
for join in body.args.get("joins") or []:
|
||
if join.args.get("on") is not None:
|
||
conds += _conjuncts(join.args["on"])
|
||
if index:
|
||
own = joins[index - 1]
|
||
if own.args.get("on") is not None:
|
||
conds += _conjuncts(own.args["on"])
|
||
position = index - 1
|
||
new_joins = joins[:position] + [exp.Join(this=s.copy()) for s in sources] + joins[position + 1:]
|
||
select.set("joins", new_joins)
|
||
else:
|
||
select.args["from_"].set("this", src.copy())
|
||
select.set("joins", [exp.Join(this=s.copy()) for s in sources[1:]] + list(joins) or None)
|
||
outer = select.args.get("where")
|
||
conds += _conjuncts(outer.this) if outer is not None else []
|
||
if conds:
|
||
select.set("where", exp.Where(this=_and_all([c.copy() for c in conds])))
|
||
done = True
|
||
break
|
||
if done:
|
||
break
|
||
if not done:
|
||
return
|
||
|
||
|
||
def _split_simple_ctes(tree: Any, read: str | None, info: "_Schema") -> None:
|
||
"""引用了好几次、只从几张表里筛行选列的 CTE(见 _mergeable),每处引用各写回一份 FROM 里的子查询(别名照旧):
|
||
这样的 CTE 是确定的(有不确定函数的语句不走这一级),算几遍结果一样,写回之后每处都能并进外层。语句里还有别的
|
||
WITH 的不动(名字可能被里层的 CTE 遮住);有一处引用在结果看行出来先后的一段里的不动(PostgreSQL 把引用了好几次
|
||
的 CTE 先算出来存着,写回之后扫描的先后可能就变了)。"""
|
||
from sqlglot import exp
|
||
|
||
with_ = tree.args.get("with_")
|
||
if with_ is None or with_.args.get("recursive") or _nested_with(tree):
|
||
return
|
||
resolver = _Resolver(tree)
|
||
sens = _Sensitivity(tree, read, info, resolver)
|
||
for cte in list(with_.expressions):
|
||
if not _mergeable(cte.this):
|
||
continue
|
||
refs = [t for t in tree.find_all(exp.Table) if resolver.cte_of.get(id(t)) is cte]
|
||
if len(refs) < 2 or any(r.find_ancestor(exp.CTE) is not None for r in refs) or \
|
||
any(sens.of(resolver.owner[id(r)]) for r in refs):
|
||
continue
|
||
for ref in refs:
|
||
ref.replace(exp.Subquery(this=cte.this.copy(), alias=exp.TableAlias(this=exp.to_identifier(ref.alias_or_name))))
|
||
cte.pop()
|
||
if not with_.expressions:
|
||
tree.set("with_", None)
|
||
|
||
|
||
def _grouped(expression: Any, keys: set[str]) -> bool:
|
||
"""分组的子查询选出来的一列在每组里只有一个值:整个是分组键,或者每一列都在聚合函数里、或者本身是分组键。"""
|
||
from sqlglot import exp
|
||
|
||
if _fingerprint(expression) in keys:
|
||
return True
|
||
if any(isinstance(n, (exp.Window, exp.Query, exp.Subquery)) for n in expression.walk()):
|
||
return False
|
||
for col in expression.find_all(exp.Column):
|
||
if _fingerprint(col) not in keys and col.find_ancestor(exp.AggFunc) is None:
|
||
return False
|
||
return True
|
||
|
||
|
||
def _push_into_groups(tree: Any, read: str | None, info: "_Schema") -> None:
|
||
"""(认列之后)外层 WHERE 里只涉及一个分组子查询(FROM 或内连接进来的)选出来的列的条件,换成子查询里对应的表达式、
|
||
挪进它的 HAVING:子查询的每一行就是一组,外层按这一行的值筛和 HAVING 按这一组的值筛,用的是同一个值(分组键取的
|
||
是同一个代表值,聚合是同一个结果)。条件涉及分组键时,分组键得是「相等就是一样」的(见 _Sensitivity._identity_safe):
|
||
MySQL 会把外层只涉及分组键的条件推进子查询的 WHERE 里先筛行,不分大小写时 BINARY city = 'SH' 推进去和放在 HAVING
|
||
里结果不同(在真实的 MySQL 8.0 上跑出来过)。子查询有窗口函数、取行、去重、ROLLUP 这些的不挪;HAVING 里的条件
|
||
不往 WHERE 挪(相等不等于一样时,组里的行各是各的值)。"""
|
||
from sqlglot import exp
|
||
|
||
sens = _Sensitivity(tree, read, info)
|
||
for select in list(tree.find_all(exp.Select)):
|
||
where = select.args.get("where")
|
||
joins = select.args.get("joins") or []
|
||
if where is None or any(str(j.side or "").upper() in ("RIGHT", "FULL") for j in joins):
|
||
continue
|
||
for index, item in enumerate(_items_of(select)):
|
||
if not isinstance(item, exp.Subquery) or (index and not _inner(joins[index - 1])):
|
||
continue
|
||
body = item.this
|
||
if not isinstance(body, exp.Select) or body.args.get("group") is None or sens.of(body) or \
|
||
any(body.args.get(k) for k in ("limit", "offset", "distinct", "qualify", "windows", "with_")) or \
|
||
any(n.find(exp.Window) for n in body.expressions):
|
||
continue
|
||
outputs = {p.alias_or_name: p.unalias() for p in body.expressions if not _is_star(p)}
|
||
keys = {_fingerprint(k) for k in body.args["group"].expressions}
|
||
alias = item.alias_or_name
|
||
moved, kept = [], []
|
||
for cond in _conjuncts(where.this):
|
||
cols = list(cond.find_all(exp.Column))
|
||
if not cols or cond.find(exp.Query, exp.Subquery) is not None or \
|
||
any(c.table != alias or c.name not in outputs or not _grouped(outputs[c.name], keys)
|
||
for c in cols) or \
|
||
any(k is not None and not sens._identity_safe(k, body)
|
||
for c in cols for k in [outputs[c.name] if _fingerprint(outputs[c.name]) in keys else None]):
|
||
kept.append(cond)
|
||
continue
|
||
new = cond.copy()
|
||
for col in list(new.find_all(exp.Column)):
|
||
col.replace(exp.Paren(this=outputs[col.name].copy()))
|
||
moved.append(new)
|
||
if moved:
|
||
having = body.args.get("having")
|
||
body.set("having", exp.Having(this=_and_all(([having.this] if having else []) + moved)))
|
||
select.set("where", exp.Where(this=_and_all(kept)) if kept else None)
|
||
where = select.args.get("where")
|
||
if where is None:
|
||
break
|
||
|
||
|
||
def _distinct_to_group(tree: Any, sens: "_Sensitivity") -> None:
|
||
"""SELECT DISTINCT a, b(没有分组、聚合、窗口函数)和 GROUP BY a, b 出的行一样(空值也算一组)——只在选出来的都是
|
||
「相等就是一样」的列(见 _Sensitivity._identity_safe)、这一段不看行出来的先后(DISTINCT 按扫到的先后出行,GROUP BY
|
||
按分组键排好了出)时改:不然两种写法留下的代表值、取到的行可能不同。ORDER BY 用到没选出来的列时不改(两种写法对
|
||
这种排序的规矩不一样)。已经按选出来的列分组的 DISTINCT 去掉(每组一行、各行的分组键都不同)。"""
|
||
from sqlglot import exp
|
||
|
||
for select in list(tree.find_all(exp.Select)):
|
||
distinct = select.args.get("distinct")
|
||
if distinct is None or distinct.args.get("on") or sens.of(select) or _starry(select):
|
||
continue
|
||
group = select.args.get("group")
|
||
if group is not None:
|
||
selected = {_fingerprint(p.unalias()) for p in select.expressions}
|
||
if group.expressions and all(_fingerprint(k) in selected for k in group.expressions) and \
|
||
not any(group.args.get(k) for k in ("rollup", "cube", "grouping_sets")) and \
|
||
not any(p.find(exp.Window) for p in select.expressions):
|
||
select.set("distinct", None)
|
||
continue
|
||
if select.args.get("having") is not None or _aggregating(select) or \
|
||
any(p.find(exp.Window, exp.Query, exp.Subquery) for p in select.expressions):
|
||
continue
|
||
keys = [p.unalias() for p in select.expressions]
|
||
if not all(isinstance(k, exp.Column) and sens._identity_safe(k, select) for k in keys):
|
||
continue
|
||
order = select.args.get("order")
|
||
if order is not None:
|
||
allowed = {_fingerprint(k) for k in keys}
|
||
aliases = {p.alias for p in select.expressions if p.alias}
|
||
if any(_fingerprint(o.this) not in allowed and not (
|
||
isinstance(o.this, exp.Column) and not o.this.table and o.this.name in aliases)
|
||
for o in order.expressions):
|
||
continue
|
||
new = exp.Group(expressions=[k.copy() for k in keys])
|
||
new.meta[_FROM_DISTINCT] = True
|
||
select.set("distinct", None)
|
||
select.set("group", new)
|
||
|
||
|
||
def _membership_subqueries(tree: Any, sens: "_Sensitivity") -> None:
|
||
"""IN / NOT IN (子查询) 只看取值的集合(含不含空值):里面不取行时,
|
||
|
||
- 没有聚合的 ORDER BY 不影响(有聚合的 ORDER BY 让它成了聚合查询,只出一行);
|
||
- 选出来的只是一列、「相等就是一样」(见 _Sensitivity._identity_safe)时,DISTINCT 不影响;按这一列(和别的列)
|
||
分组、没有 HAVING 也没有聚合时 GROUP BY 不影响(每一行都在某一组里,各组选出来的就是这一列在各行的取值);
|
||
合并的各支都是这样的一列时,UNION 和 UNION ALL 一样。「相等却不一样」时不改:MySQL 默认不分大小写,去重留下
|
||
'SH' 之后 BINARY 'sh' 就找不到了。"""
|
||
from sqlglot import exp
|
||
|
||
def single_column(sub: Any) -> Any:
|
||
if not isinstance(sub, exp.Select) or len(sub.expressions) != 1:
|
||
return None
|
||
column = sub.expressions[0].unalias()
|
||
return column if isinstance(column, exp.Column) and not column.is_star and \
|
||
sens._identity_safe(column, sub) else None
|
||
|
||
for node in list(tree.find_all(exp.In)):
|
||
query = node.args.get("query")
|
||
body = query.this if isinstance(query, exp.Subquery) else query
|
||
if body is None:
|
||
continue
|
||
if isinstance(body, exp.SetOperation):
|
||
leaves = _union_leaves(body)
|
||
if leaves is None or any(single_column(leaf) is None for leaf in leaves):
|
||
continue
|
||
_set_union_all(body)
|
||
else:
|
||
leaves = [body]
|
||
for sub in leaves:
|
||
if not isinstance(sub, exp.Select) or any(sub.args.get(k) for k in ("limit", "offset", "fetch", "qualify")):
|
||
continue
|
||
if any(isinstance(n, exp.AggFunc) and not isinstance(n.parent, exp.Window) for n in _block_nodes(sub)) or \
|
||
sub.args.get("having") is not None:
|
||
continue
|
||
if not any(n.find(exp.Window) for n in sub.expressions):
|
||
sub.set("order", None)
|
||
column = single_column(sub)
|
||
group = sub.args.get("group")
|
||
if group is not None:
|
||
if column is None or any(group.args.get(k) for k in ("rollup", "cube", "grouping_sets", "totals")) or \
|
||
_fingerprint(column) not in {_fingerprint(k) for k in group.expressions}:
|
||
continue
|
||
sub.set("group", None)
|
||
distinct = sub.args.get("distinct")
|
||
if distinct is not None and not distinct.args.get("on") and column is not None:
|
||
sub.set("distinct", None)
|
||
|
||
|
||
# ---- 结果看不看行出来的先后
|
||
|
||
class _Sensitivity:
|
||
"""哪几段 SELECT 的结果看行出来的先后:这样的一段不调换来源的先后、不把 CROSS JOIN 写成逗号连接、不排 GROUP BY
|
||
的键、不把 RIGHT JOIN 写成 LEFT JOIN,不往里并子查询——这些改写都会改变库里扫描的先后,实际取到的行、值就不同了。
|
||
|
||
本段自己看先后的:取行(LIMIT / OFFSET / FETCH,有 ORDER BY 也可能有并列);DISTINCT ON;窗口函数;拼串、拼数组、
|
||
取任意一个这类聚合(GROUP_CONCAT);聚合里既不分组也不聚合的列(SQLite、MySQL 取组里任意一行的值);当成标量
|
||
用的子查询(SQLite 取第一行),除非是不分组的聚合(只出一行);去重、分组、MIN / MAX、去重的 UNION 留下的是先
|
||
出现的那个值,而「相等」不等于「写法一样」的时候(MySQL 默认不分大小写的排序规则下的 'SH' 和 'sh'、PostgreSQL
|
||
numeric 的 2.0 和 2.00、SQLite 没有类型亲和的列里的 1 和 1.0、算出来的表达式)看先后,见 _identity_safe;
|
||
PostgreSQL、MySQL 里对不知道是不是精确数的值求和、求平均(浮点数加的先后不同,结果的最后几位就不同,见 _exact)。
|
||
给看先后的一段供行的(它 FROM 里的子查询、引用的 CTE,层层往里)也看先后。"""
|
||
|
||
def __init__(self, tree: Any, read: str | None, info: _Schema, resolver: _Resolver | None = None,
|
||
floats: bool = True) -> None:
|
||
from sqlglot import exp
|
||
|
||
self.read = read
|
||
self.info = info
|
||
self.floats = floats
|
||
try:
|
||
self.resolver = resolver or _Resolver(tree)
|
||
except _Unsupported:
|
||
self.resolver = None
|
||
selects = list(tree.find_all(exp.Select))
|
||
marked = {id(s) for s in selects if self._own(s)}
|
||
for op in tree.find_all(exp.SetOperation):
|
||
if op.args.get("distinct") or isinstance(op, (exp.Intersect, exp.Except)):
|
||
for leaf in _leaves(op):
|
||
if isinstance(leaf, exp.Select) and \
|
||
not all(self._identity_safe(p.unalias(), leaf) for p in leaf.expressions):
|
||
marked.add(id(leaf))
|
||
by_id = {id(s): s for s in selects}
|
||
stack = [by_id[i] for i in marked if i in by_id]
|
||
while stack:
|
||
select = stack.pop()
|
||
for item in _items_of(select):
|
||
body = self.resolver.body(item) if self.resolver is not None else (
|
||
item.this if isinstance(item, exp.Subquery) else None)
|
||
for leaf in _leaves(body) if body is not None else []:
|
||
if isinstance(leaf, exp.Select) and id(leaf) not in marked:
|
||
marked.add(id(leaf))
|
||
stack.append(leaf)
|
||
self.marked = marked
|
||
|
||
def of(self, select: Any) -> bool:
|
||
return id(select) in self.marked
|
||
|
||
def _own(self, select: Any) -> bool:
|
||
from sqlglot import exp
|
||
|
||
if any(select.args.get(k) for k in ("limit", "offset", "fetch")):
|
||
return True
|
||
distinct = select.args.get("distinct")
|
||
if distinct is not None and distinct.args.get("on"):
|
||
return True
|
||
own = list(_block_nodes(select))
|
||
if any(isinstance(n, exp.Window) or _order_dependent(n) for n in own):
|
||
return True
|
||
if _scalar_context(select) and not (_aggregating(select) and select.args.get("group") is None):
|
||
return True
|
||
if _loose_columns(select):
|
||
return True
|
||
if distinct is not None and not all(self._identity_safe(p.unalias(), select) for p in select.expressions):
|
||
return True
|
||
group = select.args.get("group")
|
||
if group is not None and not all(self._identity_safe(k, select) for k in group.expressions):
|
||
return True
|
||
for n in own:
|
||
if isinstance(n, (exp.Min, exp.Max)) and not n.expressions and \
|
||
not self._identity_safe(n.this.expressions[0] if isinstance(n.this, exp.Distinct) and
|
||
n.this.expressions else n.this, select):
|
||
return True
|
||
if self.floats and self.read != "sqlite" and _float_aggregate(n) and not self._exact(n.this, select):
|
||
return True
|
||
return False
|
||
|
||
def _exact(self, node: Any, select: Any, depth: int = 0) -> bool:
|
||
"""node 的取值是不是精确的数(整数、定点小数):浮点数求和的先后不同结果就不同(PostgreSQL、MySQL 按扫到的先后
|
||
一个个加;SQLite 用 Kahan-Babuska-Neumaier 求和、关联先后也不跟着 FROM 的先后变,当成精确的)。不知道类型的不算。"""
|
||
from sqlglot import exp
|
||
|
||
node = _unparen(node)
|
||
if isinstance(node, exp.Distinct) and len(node.expressions) == 1:
|
||
node = node.expressions[0]
|
||
if isinstance(node, exp.Count) or (isinstance(node, exp.Literal) and node.is_int):
|
||
return True
|
||
if isinstance(node, (exp.Add, exp.Sub, exp.Mul)):
|
||
return self._exact(node.left, select, depth) and self._exact(node.right, select, depth)
|
||
if not isinstance(node, exp.Column) or node.is_star or depth > 16 or self.resolver is None:
|
||
return False
|
||
item = self.resolver.source_of.get(id(node))
|
||
if item is None:
|
||
items = _items_of(select)
|
||
item = items[0] if len(items) == 1 and not node.table else None
|
||
if item is None:
|
||
return False
|
||
body = self.resolver.body(item)
|
||
if body is None:
|
||
columns = self.info.types.get(item.name.translate(_ASCII_LOWER)) if isinstance(item, exp.Table) else None
|
||
declared = (columns or {}).get(node.name.translate(_ASCII_LOWER), "").lower()
|
||
return any(declared.startswith(w) for w in ("int", "integer", "bigint", "smallint", "tinyint", "mediumint",
|
||
"serial", "bigserial", "decimal", "numeric", "dec"))
|
||
if isinstance(body, exp.Select):
|
||
for p in body.expressions:
|
||
if not _is_star(p) and p.alias_or_name == node.name:
|
||
return self._exact(p.unalias(), body, depth + 1)
|
||
return False
|
||
|
||
def _identity_safe(self, node: Any, select: Any, depth: int = 0) -> bool:
|
||
"""node 的取值里「相等」是不是就是「一样」:COUNT 是整数;光一列的话看它是哪张表的哪一列、声明的什么类型。"""
|
||
from sqlglot import exp
|
||
|
||
node = _unparen(node)
|
||
if isinstance(node, exp.Count):
|
||
return True
|
||
if not isinstance(node, exp.Column) or node.is_star or depth > 16:
|
||
return False
|
||
item = self.resolver.source_of.get(id(node)) if self.resolver is not None else None
|
||
if item is None:
|
||
items = _items_of(select)
|
||
item = items[0] if len(items) == 1 and not node.table else None
|
||
if item is None:
|
||
return self._safe_type(None)
|
||
body = self.resolver.body(item) if self.resolver is not None else (
|
||
item.this if isinstance(item, exp.Subquery) else None)
|
||
if body is None:
|
||
if not isinstance(item, exp.Table):
|
||
return False
|
||
columns = self.info.types.get(item.name.translate(_ASCII_LOWER))
|
||
declared = columns.get(node.name.translate(_ASCII_LOWER)) if columns is not None else None
|
||
return self._safe_type(declared)
|
||
if not isinstance(body, exp.Select):
|
||
return False
|
||
for p in body.expressions:
|
||
if not _is_star(p) and p.alias_or_name == node.name:
|
||
return self._identity_safe(p.unalias(), body, depth + 1)
|
||
sources = _items_of(body)
|
||
if any(isinstance(p, exp.Star) for p in body.expressions) and len(sources) == 1:
|
||
return self._through_star(sources[0], node.name, depth + 1)
|
||
return False
|
||
|
||
def _through_star(self, item: Any, name: str, depth: int) -> bool:
|
||
"""SELECT * 原样透出来的一列:到它唯一的来源里找同名的列。"""
|
||
from sqlglot import exp
|
||
|
||
body = self.resolver.body(item) if self.resolver is not None else None
|
||
if body is None:
|
||
if not isinstance(item, exp.Table):
|
||
return False
|
||
columns = self.info.types.get(item.name.translate(_ASCII_LOWER))
|
||
return self._safe_type(columns.get(name.translate(_ASCII_LOWER)) if columns is not None else None)
|
||
if not isinstance(body, exp.Select) or depth > 16:
|
||
return False
|
||
for p in body.expressions:
|
||
if not _is_star(p) and p.alias_or_name == name:
|
||
return self._identity_safe(p.unalias(), body, depth + 1)
|
||
sources = _items_of(body)
|
||
if any(isinstance(p, exp.Star) for p in body.expressions) and len(sources) == 1:
|
||
return self._through_star(sources[0], name, depth + 1)
|
||
return False
|
||
|
||
def _safe_type(self, declared: str | None) -> bool:
|
||
"""声明的类型下「相等」是不是就是「一样」。SQLite 没给表结构时按列有声明的类型、默认的排序规则算(假设);
|
||
有类型的话 BLOB、没写类型(没有类型亲和)的列可能混着 1 和 1.0。PostgreSQL、MySQL 不知道类型就不算:numeric
|
||
的 2.0 和 2.00、MySQL 默认不分大小写的字符串都是相等写法却不一样。"""
|
||
if self.read == "sqlite":
|
||
if declared is None:
|
||
return True
|
||
upper = declared.upper()
|
||
if "INT" in upper or any(w in upper for w in ("CHAR", "CLOB", "TEXT", "REAL", "FLOA", "DOUB")):
|
||
return True
|
||
return bool(upper) and "BLOB" not in upper
|
||
if declared is None:
|
||
return False
|
||
lower = declared.lower().strip()
|
||
if self.read == "mysql":
|
||
return any(lower.startswith(w) for w in ("int", "integer", "bigint", "smallint", "tinyint", "mediumint",
|
||
"date", "datetime", "timestamp", "time", "year", "decimal",
|
||
"numeric", "bit", "bool"))
|
||
return any(lower.startswith(w) for w in ("int", "integer", "bigint", "smallint", "serial", "bigserial",
|
||
"text", "varchar", "character", "char", "bpchar", "date",
|
||
"timestamp", "time", "bool", "uuid"))
|
||
|
||
|
||
def _float_aggregate(node: Any) -> bool:
|
||
"""浮点数求和类的聚合(不在窗口函数里的)。"""
|
||
from sqlglot import exp
|
||
|
||
if isinstance(node.parent, exp.Window):
|
||
return False
|
||
name = type(node).__name__.lower()
|
||
if isinstance(node, exp.Anonymous):
|
||
name = str(node.name).lower().replace("_", "")
|
||
return name in _FLOAT_AGGS
|
||
|
||
|
||
def _order_dependent(node: Any) -> bool:
|
||
from sqlglot import exp
|
||
|
||
name = type(node).__name__.lower()
|
||
if isinstance(node, exp.Anonymous):
|
||
name = str(node.name).lower().replace("_", "")
|
||
return name in _ORDERED_AGGS
|
||
|
||
|
||
def _aggregating(select: Any) -> bool:
|
||
"""这一段是聚合查询:有 GROUP BY,或者自己的 SELECT 列表、HAVING、ORDER BY 里有聚合函数(不算窗口函数)。"""
|
||
from sqlglot import exp
|
||
|
||
if select.args.get("group") is not None or select.args.get("having") is not None:
|
||
return True
|
||
return any(isinstance(n, exp.AggFunc) and not isinstance(n.parent, exp.Window) for n in _block_nodes(select)
|
||
if not isinstance(n.parent, exp.Window))
|
||
|
||
|
||
def _scalar_context(query: Any) -> bool:
|
||
"""这段查询当成一个值用(标量子查询):不在 FROM / JOIN、IN、EXISTS、ANY / ALL、CTE 里,也不是整条语句。"""
|
||
from sqlglot import exp
|
||
|
||
node = query
|
||
while True:
|
||
parent = node.parent
|
||
if parent is None:
|
||
return False
|
||
if isinstance(parent, exp.Subquery):
|
||
node = parent
|
||
continue
|
||
if isinstance(parent, exp.SetOperation):
|
||
node = parent
|
||
continue
|
||
if isinstance(parent, (exp.From, exp.Join, exp.Exists, exp.CTE, exp.Any, exp.All)):
|
||
return False
|
||
if isinstance(parent, exp.In) and node.arg_key == "query":
|
||
return False
|
||
return True
|
||
|
||
|
||
def _loose_columns(select: Any) -> bool:
|
||
"""聚合的一段里既不在聚合函数里、也不是分组键(的一部分)的列:SQLite、MySQL 取组里任意一行的值。"""
|
||
from sqlglot import exp
|
||
|
||
if not _aggregating(select):
|
||
return False
|
||
group = select.args.get("group")
|
||
keys = {_fingerprint(k) for k in (group.expressions if group is not None else [])}
|
||
|
||
def loose(node: Any) -> bool:
|
||
if isinstance(node, (exp.Query, exp.Subquery)) or _fingerprint(node) in keys:
|
||
return False
|
||
if isinstance(node, exp.AggFunc) and not isinstance(node.parent, exp.Window):
|
||
return False
|
||
if isinstance(node, exp.Window):
|
||
return False
|
||
if isinstance(node, exp.Column):
|
||
return not node.is_star
|
||
return any(loose(c) for c in node.iter_expressions())
|
||
|
||
clauses = [c for c in (*select.expressions, select.args.get("having"), select.args.get("order")) if c is not None]
|
||
return any(loose(c) for c in clauses)
|
||
|
||
|
||
# ---- 动来源的改写
|
||
|
||
def _unwrap_passthrough(tree: Any, read: str | None) -> Any:
|
||
"""原样选出 FROM 里唯一一个子查询的全部列、什么别的都不做的一段(SELECT * FROM (…) t、SELECT t.a, t.b FROM (…) t),
|
||
换成那个子查询:出的行一样,列的先后也一样。最外层和条件里的子查询列名不影响取值,列名可以不同;别的地方(FROM 里
|
||
的子查询、CTE)外面按名字引用,名字也得一样。不换的:子查询自己带 ORDER BY / LIMIT / OFFSET(套一层之后顺序、取行
|
||
不一定照旧);子查询是 UNION 这类合并、库是 SQLite(和不知道是哪种库时)——SQLite 从子查询里读合并的结果时,按第一支
|
||
那一列的类型亲和转换别的支的值(REAL 列在前,另一支的整数 2 读出来成了 2.0),套不套这一层结果不一样。"""
|
||
from sqlglot import exp
|
||
|
||
def outputs(query: Any) -> list[str] | None:
|
||
if any(query.args.get(k) is not None for k in ("order", "limit", "offset", "fetch")):
|
||
return None
|
||
if isinstance(query, exp.SetOperation):
|
||
if read in (None, "sqlite"):
|
||
return None
|
||
leaves = _leaves(query)
|
||
return outputs(leaves[0]) if leaves and isinstance(leaves[0], exp.Select) else None
|
||
if not isinstance(query, exp.Select) or _starry(query) or query.args.get("with_") is not None:
|
||
return None
|
||
names = [p.alias_or_name for p in query.expressions]
|
||
return names if all(names) and len(set(names)) == len(names) else None
|
||
|
||
changed = True
|
||
while changed:
|
||
changed = False
|
||
for select in list(tree.find_all(exp.Select)):
|
||
if select.args.get("joins") or select.args.get("from_") is None or \
|
||
{k for k, v in select.args.items() if v} - {"expressions", "from_", "with_"} or \
|
||
(select.args.get("with_") is not None and select is not tree):
|
||
continue
|
||
item = select.args["from_"].this
|
||
if not isinstance(item, exp.Subquery):
|
||
continue
|
||
body = item.this
|
||
names = outputs(body)
|
||
if names is None:
|
||
continue
|
||
alias = item.alias_or_name
|
||
star = len(select.expressions) == 1 and isinstance(select.expressions[0], exp.Star)
|
||
cols = [p.unalias() for p in select.expressions]
|
||
if not star and (len(cols) != len(names) or any(
|
||
not isinstance(c, exp.Column) or c.is_star or c.name != n or (c.table and c.table != alias)
|
||
for c, n in zip(cols, names))):
|
||
continue
|
||
renamed = not star and any(p.alias_or_name != n for p, n in zip(select.expressions, names))
|
||
parent = select.parent
|
||
loose = select is tree or isinstance(parent, (exp.In, exp.Exists)) or (
|
||
isinstance(parent, exp.Subquery) and not isinstance(parent.parent, (exp.From, exp.Join, exp.CTE)))
|
||
if renamed and not loose:
|
||
continue
|
||
new = body.copy()
|
||
if select is tree:
|
||
if select.args.get("with_") is not None:
|
||
if new.args.get("with_") is not None:
|
||
continue
|
||
new.set("with_", select.args["with_"])
|
||
tree = new
|
||
else:
|
||
select.replace(new)
|
||
changed = True
|
||
break
|
||
return tree
|
||
|
||
|
||
def _bare_star(select: Any) -> bool:
|
||
from sqlglot import exp
|
||
|
||
return any(isinstance(p, exp.Star) for p in select.expressions)
|
||
|
||
|
||
def _right_to_left(tree: Any, sens: _Sensitivity) -> None:
|
||
"""只有两个来源的 a RIGHT JOIN b ON p 就是 b LEFT JOIN a ON p。选的是 * 时列的先后跟着变、结果看行出来的先后的
|
||
一段扫描的先后跟着变,不改。"""
|
||
from sqlglot import exp
|
||
|
||
for select in list(tree.find_all(exp.Select)):
|
||
joins = select.args.get("joins") or []
|
||
if len(joins) != 1 or str(joins[0].side or "").upper() != "RIGHT" or not _plain_join(joins[0]) or \
|
||
joins[0].args.get("on") is None or _bare_star(select) or sens.of(select):
|
||
continue
|
||
first = select.args["from_"]
|
||
left, right = first.this, joins[0].this
|
||
first.set("this", right.copy())
|
||
joins[0].set("this", left.copy())
|
||
joins[0].set("side", "LEFT")
|
||
joins[0].set("kind", None)
|
||
joins[0].meta[_FLIPPED] = True
|
||
for join in tree.find_all(exp.Join):
|
||
if join.side and str(join.kind or "").upper() == "OUTER":
|
||
join.set("kind", None)
|
||
|
||
|
||
def _quantified(node: Any) -> bool:
|
||
"""ANY / SOME / ALL (子查询或数组):sqlglot 有时解析成 Any / All 节点,有时(ALL (ARRAY[]::int[]))解析成名字叫
|
||
ALL 的函数调用。拿它比较的结果不是普通比较的规矩:x > ALL (空集) 是真,NOT (x = ANY …) 也不是 x <> ANY …。"""
|
||
from sqlglot import exp
|
||
|
||
node = _unparen(node)
|
||
return isinstance(node, (exp.Any, exp.All)) or (
|
||
isinstance(node, exp.Anonymous) and str(node.name).upper() in ("ALL", "ANY", "SOME"))
|
||
|
||
|
||
def _null_propagating(node: Any, alias: Any, read: str | None) -> bool:
|
||
"""alias 那一侧整行是空值时,node 一定是空值:那一侧的列,或者对它做算术、取负、取大小写、取长度、绝对值、转类型;
|
||
SQLite 里的 || 拼接(拿空值拼得空值)。PostgreSQL 的 || 不算(空的数组拼上一个数组还是那个数组:NULL || ARRAY['x']
|
||
是 ARRAY['x']),MySQL 的 || 默认是 OR,CONCAT 在 PostgreSQL 里跳过空值:都不算。alias 也可以是判断「这一列是不是
|
||
那个为空的东西」的函数(见 _drop_implied_not_null:按一列为空算)。"""
|
||
from sqlglot import exp
|
||
|
||
node = _unparen(node)
|
||
if isinstance(node, exp.Column):
|
||
if callable(alias):
|
||
return bool(alias(node))
|
||
return bool(node.table) and node.table == alias
|
||
if type(node).__name__ in _STRICT_UNARY:
|
||
return _null_propagating(node.this, alias, read)
|
||
if type(node).__name__ in _STRICT_BINARY or (isinstance(node, exp.DPipe) and read == "sqlite"):
|
||
return _null_propagating(node.left, alias, read) or _null_propagating(node.right, alias, read)
|
||
return False
|
||
|
||
|
||
#: 拿空值比较得空值的比较(ANY / ALL 另算,见 _quantified)
|
||
_COMPARISON_NODES = ("EQ", "NEQ", "GT", "GTE", "LT", "LTE", "Like", "ILike")
|
||
|
||
|
||
def _strict(cond: Any, alias: Any, read: str | None) -> bool:
|
||
"""alias 那一侧整行是空值时,cond 一定是空值(不是真、也不是假):一边一定是空值的比较、LIKE、BETWEEN,非空常量
|
||
列表的 IN,以及这些用 NOT、AND、OR 连起来(NOT 空值还是空值,空值 AND / OR 空值还是空值)。
|
||
不算的:IN 子查询(子查询没有行时是假,不是空值)、x IN ()(SQLite 允许空列表,恒为假)、ANY / ALL(子查询没有行
|
||
时是假或真)、IS [NOT] NULL、COALESCE、CASE。"""
|
||
from sqlglot import exp
|
||
|
||
cond = _unparen(cond)
|
||
if type(cond).__name__ in _COMPARISON_NODES:
|
||
if _quantified(cond.left) or _quantified(cond.right):
|
||
return False
|
||
return _null_propagating(cond.left, alias, read) or _null_propagating(cond.right, alias, read)
|
||
if isinstance(cond, exp.In):
|
||
if cond.args.get("query") or cond.args.get("unnest") or cond.args.get("field") or not cond.expressions:
|
||
return False
|
||
return _null_propagating(cond.this, alias, read)
|
||
if isinstance(cond, exp.Between):
|
||
return _null_propagating(cond.this, alias, read)
|
||
if isinstance(cond, exp.Not):
|
||
return _strict(cond.this, alias, read)
|
||
if isinstance(cond, (exp.And, exp.Or)):
|
||
return _strict(cond.left, alias, read) and _strict(cond.right, alias, read)
|
||
return False
|
||
|
||
|
||
def _rejects(cond: Any, alias: Any, read: str | None) -> bool:
|
||
"""alias 那一侧整行是空值时 cond 一定不成立(是空值或假),WHERE 里有它就留不下没关联上的行:
|
||
|
||
- 一定是空值的(见 _strict);
|
||
- x IS NOT NULL;x IN (子查询)、x op ANY (子查询)(子查询没有行时是假,有行时 x 为空是空值);x IN ()(恒为假);
|
||
BETWEEN 的上下界一定是空值的(x >= 空值 AND x <= b 不会是真);
|
||
- NOT 只能套一定是空值的(NOT 假是真:NOT (x IN (空集)) 在 x 为空时是真,NOT IN (空集) 留得下没关联上的行);
|
||
- AND 里有一项不成立、OR 里每一项都不成立。
|
||
|
||
x op ALL (…) 不算(子查询没有行时是真);IS NULL、COALESCE、CASE、<=>、IS DISTINCT FROM 之类不算。"""
|
||
from sqlglot import exp
|
||
|
||
cond = _unparen(cond)
|
||
if _strict(cond, alias, read):
|
||
return True
|
||
if type(cond).__name__ in _COMPARISON_NODES:
|
||
for side, other in ((cond.left, cond.right), (cond.right, cond.left)):
|
||
if isinstance(_unparen(other), exp.Any) and _null_propagating(side, alias, read):
|
||
return True
|
||
return False
|
||
if isinstance(cond, exp.In):
|
||
if cond.args.get("unnest") or cond.args.get("field"):
|
||
return False
|
||
if not cond.expressions and not cond.args.get("query"):
|
||
return True
|
||
return bool(cond.args.get("query")) and _null_propagating(cond.this, alias, read)
|
||
if isinstance(cond, exp.Between):
|
||
return any(_null_propagating(cond.args.get(k), alias, read) for k in ("low", "high") if cond.args.get(k))
|
||
if isinstance(cond, exp.Not):
|
||
inner = _unparen(cond.this)
|
||
if isinstance(inner, exp.Is) and isinstance(_unparen(inner.expression), exp.Null):
|
||
return _null_propagating(inner.this, alias, read)
|
||
return _strict(inner, alias, read)
|
||
if isinstance(cond, exp.And):
|
||
return _rejects(cond.left, alias, read) or _rejects(cond.right, alias, read)
|
||
if isinstance(cond, exp.Or):
|
||
return _rejects(cond.left, alias, read) and _rejects(cond.right, alias, read)
|
||
return False
|
||
|
||
|
||
def _reject_nulls(tree: Any, read: str | None, sens: "_Sensitivity") -> None:
|
||
"""LEFT JOIN 关联进来的那一侧,同一段 WHERE 里有把它整行为空的行筛掉的条件(见 _rejects):没关联上的行反正留
|
||
不下来,这次外连接就是内连接。有 RIGHT / FULL JOIN 的段不动;结果看行出来先后的一段(GROUP_CONCAT、有并列的取行,
|
||
见 _Sensitivity)也不动:SQLite 自己把这样的 LEFT JOIN 降成内连接之后,扫描的先后和直接写内连接的不一样(真实的
|
||
SQLite 上 EXPLAIN QUERY PLAN 一个先扫 refunds、一个先扫 orders,GROUP_CONCAT 拼出来的先后就不同,第 3 轮自己压测
|
||
时找到的)。"""
|
||
from sqlglot import exp
|
||
|
||
for select in list(tree.find_all(exp.Select)):
|
||
joins = select.args.get("joins") or []
|
||
where = select.args.get("where")
|
||
if where is None or any(str(j.side or "").upper() in ("RIGHT", "FULL") for j in joins) or sens.of(select):
|
||
continue
|
||
conds = _conjuncts(where.this)
|
||
for join in joins:
|
||
if str(join.side or "").upper() != "LEFT" or not _plain_join(join):
|
||
continue
|
||
alias = join.this.alias_or_name
|
||
if alias and any(_rejects(c, alias, read) for c in conds):
|
||
join.set("side", None)
|
||
join.set("kind", None)
|
||
join.this.meta[_REJECTED] = "RIGHT JOIN" if join.meta.get(_FLIPPED) else "LEFT JOIN"
|
||
|
||
|
||
#: 跳过空值的聚合:参数在一行里是空值时这一行不算数(COUNT(*) 数的是行数,不算)
|
||
_NULL_SKIPPING = ("Sum", "Avg", "Min", "Max", "Count")
|
||
|
||
|
||
def _aggregate_args(select: Any) -> list[Any] | None:
|
||
"""这一段 SELECT 列表、HAVING、ORDER BY 里的聚合(不算窗口函数)都是跳过空值的那几种时,返回它们的参数(DISTINCT
|
||
拆开);有别的聚合、COUNT(*)、多个参数的、套着子查询或窗口函数的返回 None。"""
|
||
from sqlglot import exp
|
||
|
||
clauses = [c for c in (*select.expressions, select.args.get("having"), select.args.get("order")) if c is not None]
|
||
if any(c.find(exp.Query, exp.Subquery, exp.Window) is not None for c in clauses):
|
||
return None
|
||
args = []
|
||
for clause in clauses:
|
||
for agg in clause.find_all(exp.AggFunc):
|
||
if type(agg).__name__ not in _NULL_SKIPPING or agg.args.get("expressions"):
|
||
return None
|
||
arg = agg.this
|
||
if isinstance(arg, exp.Distinct):
|
||
if len(arg.expressions) != 1:
|
||
return None
|
||
arg = arg.expressions[0]
|
||
if arg is None or isinstance(arg, exp.Star):
|
||
return None
|
||
args.append(arg)
|
||
return args
|
||
|
||
|
||
def _null_blind_left_joins(tree: Any, read: str | None, sens: _Sensitivity) -> None:
|
||
"""不分组、结果只是聚合的一段(只出一行),所有聚合都跳过空值(SUM、AVG、MIN、MAX、COUNT(x),可以去重)、参数在
|
||
LEFT JOIN 可空一侧整行为空时一定是空值(见 _null_propagating):这次 LEFT JOIN 写成内连接结果一样。LEFT JOIN 的
|
||
结果是内连接的结果加上没关联上的行(那一侧整行是空值),之后的内连接、LEFT JOIN、WHERE 都是一行一行处理的,没
|
||
关联上的行带来的那些行里这一侧也都是空值,在这些聚合里都被跳过;COALESCE(SUM(r.amount), 0)、SUM(x) / COUNT(y)、
|
||
旁边多一个常量列都只是拿这几个聚合再算。不改的:分组(没关联上的行自己成一组)、COUNT(*)、参数可能不为空的
|
||
(COALESCE(r.amount, 0))、聚合以外还有列、有 RIGHT / FULL JOIN、结果看行出来先后的一段(PostgreSQL / MySQL 里
|
||
浮点数求和的先后会变,见 _Sensitivity)。"""
|
||
from sqlglot import exp
|
||
|
||
for select in list(tree.find_all(exp.Select)):
|
||
joins = select.args.get("joins") or []
|
||
if not joins or any(str(j.side or "").upper() in ("RIGHT", "FULL") for j in joins):
|
||
continue
|
||
group = select.args.get("group")
|
||
if (group is not None and group.expressions) or sens.of(select) or _loose_columns(select):
|
||
continue
|
||
args = _aggregate_args(select)
|
||
if not args:
|
||
continue
|
||
for join in joins:
|
||
if str(join.side or "").upper() != "LEFT" or not _plain_join(join) or join.args.get("on") is None:
|
||
continue
|
||
alias = join.this.alias_or_name
|
||
if alias and all(_null_propagating(arg, alias, read) for arg in args):
|
||
join.set("side", None)
|
||
join.set("kind", None)
|
||
join.this.meta[_NULL_BLIND] = True
|
||
|
||
|
||
def _membership_context(select: Any) -> bool:
|
||
"""这一段(或者它所在的 UNION 的一支)是 IN 子查询、EXISTS 子查询的全部:重复的行不影响结果。"""
|
||
from sqlglot import exp
|
||
|
||
node = select
|
||
while isinstance(node.parent, (exp.Subquery, exp.Union)):
|
||
if isinstance(node.parent, exp.Subquery) and node.parent.alias:
|
||
return False
|
||
node = node.parent
|
||
parent = node.parent
|
||
return isinstance(parent, exp.Exists) or (isinstance(parent, exp.In) and node.arg_key == "query")
|
||
|
||
|
||
def _dup_insensitive(select: Any) -> bool:
|
||
"""这一段的结果不怕来源的行重复:聚合查询里的聚合都是 MIN、MAX、COUNT / SUM / AVG(DISTINCT …) 这类去重的;不聚合
|
||
时 SELECT DISTINCT(不是 DISTINCT ON),或者整段是 IN / EXISTS 子查询。有窗口函数的不算。"""
|
||
from sqlglot import exp
|
||
|
||
own = list(_block_nodes(select))
|
||
if any(isinstance(n, exp.Window) for n in own):
|
||
return False
|
||
distinct = select.args.get("distinct")
|
||
if distinct is not None and distinct.args.get("on"):
|
||
return False
|
||
if _aggregating(select):
|
||
aggs = [n for n in own if isinstance(n, exp.AggFunc) and not isinstance(n.parent, exp.Window)]
|
||
return all((isinstance(a, (exp.Min, exp.Max)) and not a.args.get("expressions")) or
|
||
(type(a).__name__ in ("Count", "Sum", "Avg") and isinstance(a.this, exp.Distinct) and
|
||
len(a.this.expressions) == 1) for a in aggs)
|
||
return distinct is not None or _membership_context(select)
|
||
|
||
|
||
def _drop_unused_left_joins(tree: Any, sens: _Sensitivity) -> None:
|
||
"""LEFT JOIN 进来的表除了它自己的 ON 之外哪儿都没用到(本段、相关子查询、后面的连接都不引用它),而且结果不怕重复
|
||
的行(SELECT DISTINCT;聚合都是 MIN、MAX、COUNT(DISTINCT …) 这类去重的;整段是 IN / EXISTS 子查询):这次 LEFT
|
||
JOIN 去掉结果一样——LEFT JOIN 不会让左边的行变少,只会让关联上好几行的左边的行重复出现,别的列的取值不变。不去的:
|
||
有 COUNT(*)、SUM 这类怕重复的聚合(表结构里说 ON 是按唯一键关联的也不去:类型亲和、COLLATE、部分唯一索引、MySQL
|
||
的隐式转换都能让一行关联上好几行,见 _finish),有 RIGHT / FULL JOIN、窗口函数、SELECT *,有不带表名的列(分不清
|
||
是不是这张表的),结果看行出来先后的一段(DISTINCT 留下先扫到的那个值、取行,见 _Sensitivity)。"""
|
||
from sqlglot import exp
|
||
|
||
changed = True
|
||
while changed:
|
||
changed = False
|
||
for select in list(tree.find_all(exp.Select)):
|
||
joins = select.args.get("joins") or []
|
||
if not joins or any(str(j.side or "").upper() in ("RIGHT", "FULL") for j in joins) or sens.of(select) or \
|
||
any(isinstance(p, exp.Star) for p in select.expressions):
|
||
continue
|
||
insensitive = _dup_insensitive(select)
|
||
own = list(_block_nodes(select))
|
||
nested = [c for sub in _nested_queries(select) for c in sub.find_all(exp.Column)]
|
||
columns = [n for n in own if isinstance(n, exp.Column)] + nested
|
||
if any(not c.table for c in columns):
|
||
continue
|
||
for join in joins:
|
||
if str(join.side or "").upper() != "LEFT" or not _plain_join(join) or join.args.get("on") is None:
|
||
continue
|
||
alias = join.this.alias_or_name
|
||
on = {id(n) for n in join.args["on"].walk()}
|
||
if alias and insensitive and not any(c.table == alias and id(c) not in on for c in columns):
|
||
select.set("joins", [j for j in joins if j is not join] or None)
|
||
dropped = select.meta.setdefault(_DROPPED_JOINS, [])
|
||
dropped += [t.name for t in join.this.find_all(exp.Table)] or [alias]
|
||
changed = True
|
||
break
|
||
if changed:
|
||
break
|
||
|
||
|
||
def _drop_implied_not_null(tree: Any, read: str | None, sens: _Sensitivity) -> None:
|
||
"""WHERE、HAVING、ON 里 AND 连着的 x IS NOT NULL,同一组 AND 里另有一项在 x 为空时一定不成立(x = y、x > 3、
|
||
x IN (…),见 _rejects)时去掉:那一项成立时 x 已经不为空(这几处只认「是真」,假和空值一样筛掉)。表结构里说 x
|
||
不为空(NOT NULL、主键)的不去:约束不拿来改写(见 _finish)。
|
||
|
||
只认带表名的一列;外连接改内连接(_reject_nulls)在这之前做完,靠 IS NOT NULL 改成的内连接不受影响。"""
|
||
from sqlglot import exp
|
||
|
||
def target(part: Any) -> Any:
|
||
part = _unparen(part)
|
||
if isinstance(part, exp.Not):
|
||
inner = _unparen(part.this)
|
||
if isinstance(inner, exp.Is) and isinstance(_unparen(inner.expression), exp.Null) and \
|
||
isinstance(_unparen(inner.this), exp.Column) and _unparen(inner.this).table:
|
||
return _unparen(inner.this)
|
||
return None
|
||
|
||
for select in list(tree.find_all(exp.Select)):
|
||
if sens.of(select):
|
||
# 少一个条件可能换一种扫描的先后(走不走索引),结果看行出来先后的一段不动
|
||
continue
|
||
joins = select.args.get("joins") or []
|
||
holders = [(select.args.get("where"), "this"), (select.args.get("having"), "this")] + \
|
||
[(j, "on") for j in joins if j.args.get("on") is not None]
|
||
for holder, key in holders:
|
||
if holder is None:
|
||
continue
|
||
parts = _conjuncts(holder.args.get(key))
|
||
kept: list[Any] = []
|
||
for i, part in enumerate(parts):
|
||
column = target(part)
|
||
if column is not None:
|
||
fp = _fingerprint(column)
|
||
others = kept + parts[i + 1:]
|
||
if any(_rejects(o, lambda c, fp=fp: _fingerprint(c) == fp, read) for o in others):
|
||
continue
|
||
kept.append(part)
|
||
if len(kept) == len(parts):
|
||
continue
|
||
if kept:
|
||
holder.set(key, _and_all([p.copy() for p in kept]))
|
||
elif isinstance(holder, exp.Join):
|
||
holder.set(key, exp.true())
|
||
elif isinstance(holder, exp.Having) and not _grouped_select(select):
|
||
holder.set(key, exp.true())
|
||
else:
|
||
select.set(holder.arg_key, None)
|
||
|
||
|
||
#: 换成同一组相等的列之后,原文写的那一列(实表名、列名)记在新列的 meta 里:比等价时不算(见 _fingerprint),差异
|
||
#: 说明给人看时照原文写(见 _Explainer.show)
|
||
_WRITTEN_COLUMN = "sql_semantics_written_column"
|
||
|
||
|
||
def _substitute_equal_columns(tree: Any, read: str | None, info: _Schema, sens: _Sensitivity) -> None:
|
||
"""(认列之后)WHERE 里 AND 连着的「两列相等」(内连接的条件挪进 WHERE 之后也在这里)对 WHERE 之后的每一行都成立:
|
||
两列都不为空、而且相等。两列声明的类型完全一样、这个类型下「相等就是一样」(整数、文本这类,见 _identity_type)时,
|
||
SELECT 列表、GROUP BY、HAVING、ORDER BY 里用哪一列都是同一个值:都换成同一组里排在最前的那一列(按表名、列名排;
|
||
同一张表关联了两次、分不清该选哪个别名的不换)。这样「按 users.id 分组」和「按 orders.user_id 分组」(ON
|
||
o.user_id = u.id)就认得出是一回事。不换的:没给表结构、类型不一样(SQLite 里 1 和 '1' 也相等)、LEFT JOIN 的
|
||
ON 里的相等(没关联上的行一边是空值)、本段有既不分组也不聚合的列(PostgreSQL 里那是错的写法,换了反而成了对的)。"""
|
||
from sqlglot import exp
|
||
|
||
for select in list(tree.find_all(exp.Select)):
|
||
where = select.args.get("where")
|
||
if where is None or _loose_columns(select) or sens.of(select):
|
||
continue
|
||
items = {i.alias_or_name: i for i in _items_of(select)}
|
||
ctes = _visible_ctes(select)
|
||
|
||
def base(col: Any) -> tuple[str, str, str] | None:
|
||
item = items.get(col.table)
|
||
if not isinstance(item, exp.Table) or item.args.get("db") or \
|
||
item.name.translate(_ASCII_LOWER) in ctes:
|
||
return None
|
||
table, column = item.name.translate(_ASCII_LOWER), col.name.translate(_ASCII_LOWER)
|
||
declared = info.types.get(table, {}).get(column)
|
||
if not declared or not _identity_type(declared, read, sens):
|
||
return None
|
||
return table, column, declared.strip().upper()
|
||
|
||
parent: dict[tuple[str, str], tuple[str, str]] = {}
|
||
|
||
def find(x: tuple[str, str]) -> tuple[str, str]:
|
||
parent.setdefault(x, x)
|
||
while parent[x] != x:
|
||
x = parent[x]
|
||
return x
|
||
|
||
bases: dict[tuple[str, str], tuple[str, str, str]] = {}
|
||
for cond in _conjuncts(where.this):
|
||
cond = _unparen(cond)
|
||
if not isinstance(cond, exp.EQ):
|
||
continue
|
||
left, right = _unparen(cond.this), _unparen(cond.expression)
|
||
if not (isinstance(left, exp.Column) and isinstance(right, exp.Column) and left.table and right.table):
|
||
continue
|
||
bl, br = base(left), base(right)
|
||
if bl is None or br is None or bl[2] != br[2]:
|
||
continue
|
||
kl, kr = (left.table, left.name), (right.table, right.name)
|
||
bases[kl], bases[kr] = bl, br
|
||
parent[find(kl)] = find(kr)
|
||
classes: dict[tuple[str, str], list[tuple[str, str]]] = {}
|
||
for member in list(parent):
|
||
classes.setdefault(find(member), []).append(member)
|
||
replace: dict[tuple[str, str], tuple[str, str]] = {}
|
||
for members in classes.values():
|
||
if len(members) < 2:
|
||
continue
|
||
ranked = sorted(members, key=lambda m: bases[m][:2])
|
||
if len({bases[m][:2] for m in members}) != len(members):
|
||
continue
|
||
for m in members:
|
||
if m != ranked[0]:
|
||
replace[m] = ranked[0]
|
||
if not replace:
|
||
continue
|
||
clauses = [*select.expressions, select.args.get("group"), select.args.get("having"), select.args.get("order")]
|
||
for clause in [c for c in clauses if c is not None]:
|
||
for col in list(clause.find_all(exp.Column)):
|
||
if col.find_ancestor(exp.Select) is not select or (col.table, col.name) not in replace:
|
||
continue
|
||
table, name = replace[(col.table, col.name)]
|
||
new = exp.column(name, table=table)
|
||
new.meta[_WRITTEN_COLUMN] = bases[(col.table, col.name)][:2]
|
||
if col.parent is select and col.arg_key == "expressions":
|
||
new = exp.alias_(new, col.alias_or_name, copy=False)
|
||
col.replace(new)
|
||
|
||
|
||
def _identity_type(declared: str, read: str | None, sens: _Sensitivity | None) -> bool:
|
||
"""声明的类型下「相等就是一样」、可以拿同一组相等的列互换:SQLite 里整数、文本亲和的列(实数亲和的 0.0 和 -0.0
|
||
相等却不一样,数值亲和、没有亲和的列能同时存 1 和 1.0);PostgreSQL、MySQL 按 _Sensitivity._safe_type。"""
|
||
upper = declared.upper()
|
||
if read == "sqlite":
|
||
return "INT" in upper or any(w in upper for w in ("CHAR", "CLOB", "TEXT"))
|
||
return sens is not None and sens._safe_type(declared)
|
||
|
||
|
||
def _int_value(node: Any) -> int | None:
|
||
"""只由整数常量和加、减、乘、取负组成的式子的值(LIMIT 1 + 2 是 3);别的返回 None。"""
|
||
from sqlglot import exp
|
||
|
||
node = _unparen(node)
|
||
if isinstance(node, exp.Literal) and node.is_int:
|
||
return int(node.this)
|
||
if isinstance(node, exp.Neg):
|
||
value = _int_value(node.this)
|
||
return None if value is None else -value
|
||
if isinstance(node, (exp.Add, exp.Sub, exp.Mul)):
|
||
left, right = _int_value(node.left), _int_value(node.right)
|
||
if left is None or right is None:
|
||
return None
|
||
if isinstance(node, exp.Add):
|
||
return left + right
|
||
return left - right if isinstance(node, exp.Sub) else left * right
|
||
return None
|
||
|
||
|
||
def _normalize_limits(tree: Any, read: str | None) -> None:
|
||
"""取行的几种同义写法:
|
||
|
||
- SQLite、PostgreSQL 的 LIMIT / OFFSET 可以写算式:只由整数常量加减乘出来的算成一个数(LIMIT 1 + 2 就是 LIMIT 3)。
|
||
MySQL 的 LIMIT 只能写常量,算式是语法错,不算。
|
||
- SQLite 的 LIMIT 是负数就是不限行数(SQLite 文档原话):没有 OFFSET 时去掉 LIMIT,有 OFFSET 时统一写成 LIMIT -1。
|
||
PostgreSQL 的负数 LIMIT 报错,不动。
|
||
- PostgreSQL 的 LIMIT NULL 和 LIMIT ALL(sqlglot 解析时已经去掉)就是不限行数(PostgreSQL 文档原话)。SQLite 的
|
||
LIMIT NULL 报错,不动。"""
|
||
from sqlglot import exp
|
||
|
||
def number(value: int) -> Any:
|
||
return exp.Literal.number(value) if value >= 0 else exp.Neg(this=exp.Literal.number(-value))
|
||
|
||
if read not in ("sqlite", "postgres"):
|
||
return
|
||
for query in list(tree.find_all(exp.Select, exp.SetOperation)):
|
||
for key in ("limit", "offset"):
|
||
node = query.args.get(key)
|
||
if not isinstance(node, (exp.Limit, exp.Offset)) or node.expression is None:
|
||
continue
|
||
value = _int_value(node.expression)
|
||
if value is not None:
|
||
node.set("expression", number(value))
|
||
limit = query.args.get("limit")
|
||
if not isinstance(limit, exp.Limit) or limit.expression is None:
|
||
continue
|
||
if isinstance(query, exp.Select) and (_loose_columns(query) or any(
|
||
isinstance(n, exp.Window) or _order_dependent(n) for n in _block_nodes(query))):
|
||
# 有 GROUP_CONCAT、窗口函数、既不分组也不聚合的列:去不去掉 LIMIT 子句可能换一种扫描的先后,结果跟着变
|
||
continue
|
||
value = _int_value(limit.expression)
|
||
if read == "sqlite" and value is not None and value < 0:
|
||
if query.args.get("offset") is None:
|
||
query.set("limit", None)
|
||
else:
|
||
limit.set("expression", number(-1))
|
||
elif read == "postgres" and isinstance(_unparen(limit.expression), exp.Null):
|
||
query.set("limit", None)
|
||
|
||
|
||
def _dedupe_order(tree: Any, sens: _Sensitivity) -> None:
|
||
"""ORDER BY 里和前面某一项一模一样的项去掉(不管升序降序):前面那项相等的行,这一项也相等,排不出先后。结果看行
|
||
出来先后的一段(取行、窗口函数、GROUP_CONCAT)不去:并列的行谁在前要看排序的实现,多一个排序键可能就换了。"""
|
||
from sqlglot import exp
|
||
|
||
for order in list(tree.find_all(exp.Order)):
|
||
owner = order.find_ancestor(exp.Select)
|
||
if owner is None or sens.of(owner):
|
||
continue
|
||
seen: set[str] = set()
|
||
kept = []
|
||
for item in order.expressions:
|
||
key = _fingerprint(item.this if isinstance(item, exp.Ordered) else item)
|
||
if key not in seen:
|
||
seen.add(key)
|
||
kept.append(item)
|
||
if len(kept) != len(order.expressions):
|
||
order.set("expressions", kept)
|
||
|
||
|
||
def _flatten_prefix(tree: Any, sens: _Sensitivity, read: str | None) -> None:
|
||
"""开头那组内连接(FROM 和紧跟着的内连接、CROSS JOIN、逗号连接,到第一个外连接为止)的 ON 条件挪进 WHERE、写成
|
||
逗号连接:内连接的条件写在 ON 还是 WHERE 一样;后面的 LEFT JOIN 对左边每一行各自关联,左边先筛后筛一样。后面有
|
||
RIGHT / FULL JOIN 的不动:没关联上的那一侧整行空值,挪到 WHERE 的条件会把它们筛掉(PostgreSQL、MySQL 里逗号还比
|
||
JOIN 结合得松,a, b RIGHT JOIN c 是 a 乘上 b RIGHT JOIN c)。内连接的条件写在 ON 还是 WHERE,优化器一视同仁,结果
|
||
看行出来先后的一段也照做;只有 SQLite 里结果看先后、又写了 CROSS JOIN 的一段不动(SQLite 不调换 CROSS JOIN 两边、
|
||
会调换逗号连接)。"""
|
||
from sqlglot import exp
|
||
|
||
for select in list(tree.find_all(exp.Select)):
|
||
joins = select.args.get("joins") or []
|
||
if not joins or select.args.get("from_") is None or \
|
||
any(str(j.side or "").upper() in ("RIGHT", "FULL") for j in joins):
|
||
continue
|
||
if read in (None, "sqlite") and sens.of(select) and \
|
||
any(str(j.kind or "").upper() == "CROSS" or j.meta.get("sql_semantics_join") for j in joins):
|
||
continue
|
||
moved: list[Any] = []
|
||
for join in joins:
|
||
if not _inner(join):
|
||
break
|
||
on = join.args.get("on")
|
||
if on is not None:
|
||
moved += _conjuncts(on)
|
||
join.set("on", None)
|
||
join.set("kind", None)
|
||
if not sens.of(select):
|
||
join.meta.pop("sql_semantics_join", None)
|
||
if moved:
|
||
where = select.args.get("where")
|
||
select.set("where", exp.Where(this=_and_all((_conjuncts(where.this) if where else []) + moved)))
|
||
|
||
|
||
# ---- 和来源无关的改写
|
||
|
||
def _items_of(select: Any) -> list[Any]:
|
||
first = select.args.get("from_")
|
||
return ([first.this] if first is not None else []) + [j.this for j in select.args.get("joins") or []]
|
||
|
||
|
||
def _and_all(items: list[Any]) -> Any:
|
||
from sqlglot import exp
|
||
|
||
if not items:
|
||
return None
|
||
return exp.and_(*items, copy=False) if len(items) > 1 else items[0]
|
||
|
||
|
||
def _plain_join(join: Any) -> bool:
|
||
return not (join.args.get("using") or join.args.get("method") or join.args.get("natural"))
|
||
|
||
|
||
def _comma(join: Any) -> bool:
|
||
"""开头那组内连接里的一项:写成逗号连接、条件都在 WHERE 里(见 _flatten_prefix)。"""
|
||
return _plain_join(join) and not join.side and not join.args.get("kind") and join.args.get("on") is None
|
||
|
||
|
||
def _inner(join: Any) -> bool:
|
||
return _plain_join(join) and not join.side and str(join.kind or "").upper() in ("", "INNER", "CROSS")
|
||
|
||
|
||
def _constant(node: Any) -> bool:
|
||
from sqlglot import exp
|
||
|
||
return isinstance(node, (exp.Literal, exp.Null, exp.Boolean)) or (
|
||
isinstance(node, exp.Neg) and isinstance(node.this, exp.Literal))
|
||
|
||
|
||
def _strip_asc(tree: Any) -> None:
|
||
from sqlglot import exp
|
||
|
||
for o in tree.find_all(exp.Ordered):
|
||
if o.args.get("desc") is False:
|
||
o.set("desc", None)
|
||
|
||
|
||
def _normalize_negation(tree: Any) -> None:
|
||
"""x IS NOT NULL、x NOT LIKE y 这类带 negate 的写法统一写成 NOT (x IS NULL)、NOT (x LIKE y):各方言解析出来的形状
|
||
不一样(PostgreSQL 的 IS NOT NULL 是 Is(negate=True),SQLite 的是 Not(Is)),意思一样。"""
|
||
from sqlglot import exp
|
||
|
||
for node in list(tree.find_all(exp.Predicate)):
|
||
if node.args.get("negate") is True:
|
||
node.set("negate", None)
|
||
copy = node.copy()
|
||
node.replace(exp.Not(this=exp.Paren(this=copy)))
|
||
|
||
|
||
def _case_forms(tree: Any, read: str | None) -> None:
|
||
"""CASE … ELSE NULL END 的 ELSE NULL 去掉(不写 ELSE 就是空值);SQLite 的 IIF(c, a, b)、MySQL 的 IF(c, a, b) 写成
|
||
CASE WHEN c THEN a ELSE b END(SQLite 文档:两者生成同样的字节码;MySQL 的 IF 在 c 为真时取 a,否则取 b)。"""
|
||
from sqlglot import exp
|
||
|
||
if read in ("sqlite", "mysql"):
|
||
for node in list(tree.find_all(exp.If)):
|
||
if isinstance(node.parent, exp.Case) or node.args.get("true") is None:
|
||
continue
|
||
case = exp.Case(ifs=[exp.If(this=node.this.copy(), true=node.args["true"].copy())])
|
||
if node.args.get("false") is not None:
|
||
case.set("default", node.args["false"].copy())
|
||
node.replace(case)
|
||
for case in tree.find_all(exp.Case):
|
||
if isinstance(case.args.get("default"), exp.Null):
|
||
case.set("default", None)
|
||
|
||
|
||
def _drop_true_conjuncts(tree: Any) -> None:
|
||
"""WHERE、HAVING、ON 里恒为真的项(TRUE、1 = 1 这种同一个常量和自己相等;分组之后 HAVING COUNT(*) >= 1 这种对
|
||
每一组都成立的)去掉:x AND TRUE 就是 x。全都去掉了的 HAVING 只在已经是聚合查询时去掉(见 _grouped_select)。"""
|
||
from sqlglot import exp
|
||
|
||
def true(node: Any) -> bool:
|
||
node = _unparen(node)
|
||
if isinstance(node, exp.Boolean):
|
||
return bool(node.this)
|
||
if isinstance(node, (exp.EQ, exp.GTE, exp.LTE)):
|
||
left, right = _unparen(node.this), _unparen(node.expression)
|
||
return isinstance(left, exp.Literal) and isinstance(right, exp.Literal) and \
|
||
_fingerprint(left) == _fingerprint(right)
|
||
return False
|
||
|
||
def always(node: Any, holder: Any) -> bool:
|
||
# 分组之后每组至少一行:HAVING COUNT(*) >= 1、COUNT(*) > 0 对每一组都成立(不分组时没有行也出一组,COUNT(*)
|
||
# 是 0,不算)。按取值逐段算(见 _Range),COUNT(x) 可能是 0,不算
|
||
if not isinstance(holder, exp.Having) or not isinstance(holder.parent, exp.Select):
|
||
return False
|
||
group = holder.parent.args.get("group")
|
||
if group is None or not group.expressions:
|
||
return False
|
||
found = _Range(None).subject(node)
|
||
subject = _unparen(found[1]) if found is not None else None
|
||
if not (isinstance(subject, exp.Count) and isinstance(subject.this, exp.Star)):
|
||
return False
|
||
return _Range(None).compare([node], [], grouped=True) is True
|
||
|
||
holders = [(w, "this") for w in tree.find_all(exp.Where, exp.Having)] + \
|
||
[(j, "on") for j in tree.find_all(exp.Join) if j.args.get("on") is not None]
|
||
for holder, key in holders:
|
||
cond = holder.args.get(key)
|
||
parts = _conjuncts(cond)
|
||
kept = [p for p in parts if not true(p) and not always(p, holder)]
|
||
if len(kept) == len(parts):
|
||
continue
|
||
if kept:
|
||
holder.set(key, _and_all([p.copy() for p in kept]))
|
||
elif isinstance(holder, exp.Join):
|
||
holder.set(key, exp.true())
|
||
elif isinstance(holder, exp.Having) and not _grouped_select(holder.parent):
|
||
# 没有 GROUP BY、别处也没有聚合的 HAVING 让整条语句成了只出一行的聚合(PostgreSQL:SELECT 1 FROM orders
|
||
# HAVING 1 = 1 只出一行),MySQL 里又是按行筛:整个去掉就变了,统一写成 HAVING TRUE
|
||
holder.set(key, exp.true())
|
||
else:
|
||
holder.parent.set(holder.arg_key, None)
|
||
|
||
|
||
def _grouped_select(select: Any) -> bool:
|
||
"""这一段不看 HAVING 也已经是聚合查询:有 GROUP BY,或者 SELECT 列表、ORDER BY 里有聚合函数(不算窗口函数)。这时
|
||
HAVING TRUE 去不去掉都是那些组。"""
|
||
from sqlglot import exp
|
||
|
||
if not isinstance(select, exp.Select):
|
||
return False
|
||
group = select.args.get("group")
|
||
if group is not None and group.expressions:
|
||
return True
|
||
clauses = [*select.expressions, select.args.get("order")]
|
||
return any(not isinstance(n.parent, exp.Window) and n.find_ancestor(exp.Select) is select
|
||
for c in clauses if c is not None for n in c.find_all(exp.AggFunc))
|
||
|
||
|
||
def _count_star(tree: Any) -> None:
|
||
"""COUNT(1)、COUNT('x'):数的是不为空的常量,就是行数。COUNT(DISTINCT 1)(有行就是 1)、COUNT(NULL) 不是。"""
|
||
from sqlglot import exp
|
||
|
||
for c in list(tree.find_all(exp.Count)):
|
||
if isinstance(c.this, exp.Literal) and not c.args.get("distinct"):
|
||
c.set("this", exp.Star())
|
||
|
||
|
||
def _minmax_distinct(tree: Any, sens: "_Sensitivity | None" = None) -> None:
|
||
"""MIN(DISTINCT x)、MAX(DISTINCT x) 就是 MIN(x)、MAX(x):去不去重最小最大的都是那个值。结果看行出来先后的一段不改
|
||
(带 DISTINCT 的聚合可能换一种执行方式)。"""
|
||
from sqlglot import exp
|
||
|
||
for node in list(tree.find_all(exp.Min, exp.Max)):
|
||
owner = node.find_ancestor(exp.Select)
|
||
if sens is not None and owner is not None and sens.of(owner):
|
||
continue
|
||
arg = node.this
|
||
if not node.expressions and isinstance(arg, exp.Distinct) and len(arg.expressions) == 1 and \
|
||
not arg.args.get("on"):
|
||
node.set("this", arg.expressions[0])
|
||
|
||
|
||
def _simple_value(node: Any) -> bool:
|
||
"""只是列、常量、* 和它们的加减乘除(没有函数调用:PostgreSQL 里 SELECT 列表的集合返回函数会改变行数)。"""
|
||
from sqlglot import exp
|
||
|
||
return all(isinstance(n, (exp.Column, exp.Literal, exp.Null, exp.Boolean, exp.Star, exp.Identifier, exp.Paren,
|
||
exp.Neg, exp.Add, exp.Sub, exp.Mul, exp.Div, exp.Alias)) for n in node.walk())
|
||
|
||
|
||
def _union_leaves(query: Any) -> list[Any] | None:
|
||
"""一串只用 UNION / UNION ALL 连起来、哪一层都不排序不取行的合并的各支;有 INTERSECT / EXCEPT(看的是取值)、
|
||
ORDER BY / LIMIT / OFFSET(取哪几行看去没去重)、WITH 的返回 None。"""
|
||
from sqlglot import exp
|
||
|
||
leaves: list[Any] = []
|
||
stack = [query]
|
||
while stack:
|
||
node = stack.pop()
|
||
if isinstance(node, exp.Subquery):
|
||
if node.alias:
|
||
return None
|
||
stack.append(node.this)
|
||
elif isinstance(node, exp.SetOperation):
|
||
if not isinstance(node, exp.Union) or \
|
||
any(node.args.get(k) is not None for k in ("order", "limit", "offset", "fetch", "with_")):
|
||
return None
|
||
stack += [node.expression, node.this]
|
||
else:
|
||
leaves.append(node)
|
||
return leaves
|
||
|
||
|
||
def _set_union_all(query: Any) -> None:
|
||
"""一串 UNION 都改成 UNION ALL(调用方确认了重复的行不影响结果)。"""
|
||
from sqlglot import exp
|
||
|
||
for node in [query, *query.find_all(exp.Union)]:
|
||
if isinstance(node, exp.Union):
|
||
node.set("distinct", False)
|
||
|
||
|
||
def _exists_rewritable(sub: Any) -> bool:
|
||
"""EXISTS 里的这一段能不能写成 SELECT 1、去掉去重 / 分组 / 排序 / 取前 n 行(见 _exists_subqueries)。"""
|
||
from sqlglot import exp
|
||
|
||
if not isinstance(sub, exp.Select):
|
||
return False
|
||
if any(sub.args.get(k) is not None for k in ("having", "offset", "fetch", "qualify")):
|
||
return False
|
||
if any(isinstance(n, exp.AggFunc) and not isinstance(n.parent, exp.Window) for n in _block_nodes(sub)):
|
||
return False
|
||
distinct = sub.args.get("distinct")
|
||
if distinct is not None and distinct.args.get("on"):
|
||
return False
|
||
group = sub.args.get("group")
|
||
if group is not None and any(group.args.get(k) for k in ("rollup", "cube", "grouping_sets", "totals")):
|
||
return False
|
||
limit = sub.args.get("limit")
|
||
if limit is not None:
|
||
value = limit.expression
|
||
if not (isinstance(value, exp.Literal) and value.is_int and int(value.this) >= 1):
|
||
return False
|
||
if not all(_simple_value(p) for p in sub.expressions):
|
||
return False
|
||
# WHERE 里按名字引用了选出来的列别名(SQLite 认:WHERE … AND a2 > 100 里的 a2 是 r.amount * 2 AS a2):写成
|
||
# SELECT 1 之后这个名字就指不到原来的列了
|
||
return not (_alias_refs(sub) & {p.alias for p in sub.expressions if isinstance(p, exp.Alias) and p.alias})
|
||
|
||
|
||
def _exists_subqueries(tree: Any) -> None:
|
||
"""EXISTS 只看有没有行:选什么、去不去重、怎么分组、怎么排都不影响,统一写成 SELECT 1;只取前 n 行(n ≥ 1、不跳过
|
||
行)也不影响有没有行;分组(没有 HAVING、没有聚合)时有行就至少有一组;合并的各支是 UNION 还是 UNION ALL 也不
|
||
影响(有没有行和去不去重无关),各支都写得成 SELECT 1 时一起写。动之前先确认这一段不是聚合查询(SELECT、HAVING、
|
||
ORDER BY 里有聚合函数时总出一行,ORDER BY COUNT(*) 去掉就成了普通查询)、不跳过行(DISTINCT、GROUP BY 在 OFFSET
|
||
之前算,去掉它们行数就变了)、选的只是列和常量(PostgreSQL 的集合返回函数会改变行数)、WHERE 里没按名字引用选出来
|
||
的列别名。"""
|
||
from sqlglot import exp
|
||
|
||
for ex in list(tree.find_all(exp.Exists)):
|
||
body = ex.this.this if isinstance(ex.this, exp.Subquery) else ex.this
|
||
if isinstance(body, exp.SetOperation):
|
||
leaves = _union_leaves(body)
|
||
if leaves is None:
|
||
continue
|
||
_set_union_all(body)
|
||
else:
|
||
leaves = [body]
|
||
if not all(_exists_rewritable(leaf) for leaf in leaves):
|
||
continue
|
||
for sub in leaves:
|
||
for key in ("limit", "distinct", "order", "group"):
|
||
sub.set(key, None)
|
||
sub.set("expressions", [exp.Literal.number(1)])
|
||
|
||
|
||
def _in_lists(tree: Any) -> None:
|
||
"""IN 列表是个集合:一模一样的项去掉、排序;只有一个常量的写成 =(NOT IN 跟着成了 <>,空值的结果一样)。只有
|
||
一项、又不是常量的不改:SQLite 里 a IN (b) 是 a = +b,右边那一列不带类型亲和,和 a = b 结果不同。"""
|
||
from sqlglot import exp
|
||
|
||
for node in list(tree.find_all(exp.In)):
|
||
if not node.expressions or node.args.get("query") or node.args.get("unnest") or node.args.get("field"):
|
||
continue
|
||
uniq = {_fingerprint(i): i for i in node.expressions}
|
||
ordered = [uniq[k] for k in sorted(uniq)]
|
||
if len(ordered) == 1 and _constant(ordered[0]):
|
||
node.replace(exp.EQ(this=node.this, expression=ordered[0]))
|
||
else:
|
||
node.set("expressions", ordered)
|
||
|
||
|
||
def _expand_between(tree: Any, read: str | None, sens: "_Sensitivity | None" = None) -> None:
|
||
"""x BETWEEN a AND b 就是 x >= a AND x <= b(SQLite 文档原话,PostgreSQL、MySQL 也是)。x 里有子查询、不确定的
|
||
函数的不改(BETWEEN 只算一次 x)。"""
|
||
from sqlglot import exp
|
||
|
||
for node in list(tree.find_all(exp.Between)):
|
||
x = node.this
|
||
if node.args.get("symmetric") or x is None or x.find(exp.Query, exp.Subquery) is not None or _volatile(x):
|
||
continue
|
||
owner = node.find_ancestor(exp.Select)
|
||
if sens is not None and owner is not None and sens.of(owner):
|
||
# 结果看行出来先后的一段不改:一个 BETWEEN 和两个比较估算出的行数可能不同(MySQL 的 condition filtering),
|
||
# 换一种扫描的先后,GROUP_CONCAT 拼出来的先后就跟着变
|
||
continue
|
||
if read == "mysql":
|
||
# MySQL 的 BETWEEN 按三个参数一起定比较的类型,两个比较各定各的:上下界一个数字一个字符串(name BETWEEN 2
|
||
# AND '3',name = '10' 时 BETWEEN 是 0、两个比较是 1)、或者上下界不是常量(类型不知道)时不改
|
||
low, high = _literal_value(node.args.get("low")), _literal_value(node.args.get("high"))
|
||
if low is None or high is None or low[0] != high[0]:
|
||
continue
|
||
pair = exp.And(this=exp.GTE(this=x.copy(), expression=node.args["low"].copy()),
|
||
expression=exp.LTE(this=x.copy(), expression=node.args["high"].copy()))
|
||
node.replace(exp.Paren(this=pair))
|
||
|
||
|
||
def _predicate(node: Any) -> bool:
|
||
"""取值只会是真、假、空值的条件(比较、IS、IN、LIKE、EXISTS、BETWEEN,和它们的 AND / OR / NOT)。"""
|
||
from sqlglot import exp
|
||
|
||
node = _unparen(node)
|
||
if isinstance(node, (exp.Predicate, exp.EQ, exp.NEQ, exp.GT, exp.GTE, exp.LT, exp.LTE, exp.Is, exp.In,
|
||
exp.Like, exp.ILike, exp.Exists, exp.Between)):
|
||
return True
|
||
if isinstance(node, (exp.And, exp.Or)):
|
||
return _predicate(node.left) and _predicate(node.right)
|
||
if isinstance(node, exp.Not):
|
||
return _predicate(node.this)
|
||
return False
|
||
|
||
|
||
def _fold_not(tree: Any, sens: "_Sensitivity | None" = None) -> None:
|
||
"""NOT 往里推(三值逻辑下都成立,两边有空值时结果一样):NOT 套比较写成反过来的比较(NOT a = b 就是 a <> b,NOT a > b
|
||
就是 a <= b);NOT NOT p 写成 p(p 是条件时;NOT NOT 5 是 1,不是 5);NOT (p AND q) 写成 NOT p OR NOT q、NOT (p OR q)
|
||
写成 NOT p AND NOT q(德摩根律,p、q 是条件时;空值 AND / OR 的真值表左右对称,三值逻辑下照样成立)。ANY / ALL 不动;
|
||
结果看行出来先后的一段只做前两样(换了条件的形状可能换一种扫描的先后)。"""
|
||
from sqlglot import exp
|
||
|
||
def sensitive(node: Any) -> bool:
|
||
owner = node.find_ancestor(exp.Select)
|
||
return sens is not None and owner is not None and sens.of(owner)
|
||
|
||
changed = True
|
||
while changed:
|
||
changed = False
|
||
for node in list(tree.find_all(exp.Not)):
|
||
if node.parent is None and node is not tree:
|
||
continue
|
||
inner = _unparen(node.this)
|
||
name = type(inner).__name__
|
||
if name in _NEGATE and not _quantified(inner.this) and not _quantified(inner.expression):
|
||
node.replace(getattr(exp, _NEGATE[name])(this=inner.this, expression=inner.expression))
|
||
changed = True
|
||
elif isinstance(inner, exp.Not) and _predicate(inner.this):
|
||
node.replace(inner.this)
|
||
changed = True
|
||
elif isinstance(inner, (exp.And, exp.Or)) and _predicate(inner.left) and _predicate(inner.right) and \
|
||
not sensitive(node):
|
||
left = exp.Not(this=exp.Paren(this=inner.left.copy()))
|
||
right = exp.Not(this=exp.Paren(this=inner.right.copy()))
|
||
flipped = exp.Or(this=left, expression=right) if isinstance(inner, exp.And) else \
|
||
exp.And(this=left, expression=right)
|
||
node.replace(exp.Paren(this=flipped))
|
||
changed = True
|
||
if changed:
|
||
break
|
||
|
||
|
||
#: 括号里是这些(运算符的一部分)时,外面是比较运算的话括号可以去掉:算术比比较结合得紧
|
||
_ARITHMETIC = ("Add", "Sub", "Mul", "Div", "Mod")
|
||
_COMPARISON = ("EQ", "NEQ", "GT", "GTE", "LT", "LTE")
|
||
|
||
|
||
def _strip_parens(tree: Any) -> None:
|
||
"""不影响结合的括号去掉(比较时本来就不看括号,见 _fingerprint;这里只为生成的写法好读):括号外面不是运算符
|
||
(函数参数、别名、SELECT 列表、ORDER BY、GROUP BY、WHERE 整个条件、IN 列表的一项……)、括号里是一列一个常量一次
|
||
函数调用,或者括号里是算术、外面是比较。"""
|
||
from sqlglot import exp
|
||
|
||
for paren in list(tree.find_all(exp.Paren)):
|
||
inner, parent = paren.this, paren.parent
|
||
if inner is None or parent is None or isinstance(inner, (exp.Query, exp.Subquery, exp.Tuple)):
|
||
continue
|
||
atomic = isinstance(inner, (exp.Column, exp.Literal, exp.Null, exp.Boolean, exp.Paren)) or (
|
||
isinstance(inner, exp.Func) and not isinstance(inner, (exp.Binary, exp.Unary, exp.Predicate)))
|
||
outside = isinstance(parent, (exp.Alias, exp.Distinct, exp.Select, exp.Ordered, exp.Where, exp.Having,
|
||
exp.Group, exp.Window)) or (
|
||
isinstance(parent, exp.Func) and not isinstance(parent, (exp.Binary, exp.Unary, exp.Predicate)))
|
||
loose = type(inner).__name__ in _ARITHMETIC and type(parent).__name__ in _COMPARISON
|
||
if isinstance(parent, exp.In) and paren.arg_key == "expressions":
|
||
outside = True
|
||
if atomic or outside or loose:
|
||
paren.replace(inner)
|
||
|
||
|
||
def _zero_offset(tree: Any) -> None:
|
||
from sqlglot import exp
|
||
|
||
for offset in list(tree.find_all(exp.Offset)):
|
||
value = offset.expression
|
||
if isinstance(value, exp.Literal) and value.is_int and int(value.this) == 0:
|
||
offset.pop()
|
||
|
||
|
||
def _order_by_aliases(tree: Any) -> None:
|
||
"""最外层 ORDER BY 引用 SELECT 的别名的,换成那个表达式:别名接下来要去掉。不换的:别名指的是数字常量(ORDER BY 1
|
||
是按第一列排);别名指的是一个不带表名、又和某个别名同名的列(ORDER BY 里单独一个名字先认别名:SELECT user_id AS
|
||
amount, amount AS user_id … ORDER BY amount 换成 ORDER BY user_id 就成了按第二列排)。"""
|
||
from sqlglot import exp
|
||
|
||
if not isinstance(tree, exp.Select) or tree.args.get("order") is None:
|
||
return
|
||
aliases = {p.alias: p for p in tree.expressions if p.alias}
|
||
names = [p.alias_or_name for p in tree.expressions]
|
||
for o in tree.args["order"].expressions:
|
||
key = o.this
|
||
if not (isinstance(key, exp.Column) and not key.table and key.name in aliases) or names.count(key.name) > 1:
|
||
continue
|
||
target = aliases[key.name].unalias()
|
||
if target.is_number or (isinstance(target, exp.Column) and not target.table and target.name in aliases):
|
||
continue
|
||
o.set("this", target.copy())
|
||
|
||
|
||
def _drop_aliases(query: Any) -> None:
|
||
from sqlglot import exp
|
||
|
||
if not isinstance(query, exp.Select):
|
||
return
|
||
used = _alias_refs(query)
|
||
names = [p.alias_or_name for p in query.expressions]
|
||
query.set("expressions", [p.unalias() if isinstance(p, exp.Alias) and p.alias not in used and
|
||
names.count(p.alias) == 1 else p for p in query.expressions])
|
||
|
||
|
||
def _drop_output_aliases(tree: Any) -> None:
|
||
"""最外层(UNION 不按名字排序时各支都是)和条件里的子查询选出来的列名不影响任何取值,去掉;本段里还按名字引用
|
||
的留着。"""
|
||
from sqlglot import exp
|
||
|
||
if isinstance(tree, exp.SetOperation):
|
||
if tree.args.get("order") is None:
|
||
for leaf in _leaves(tree):
|
||
_drop_aliases(leaf)
|
||
else:
|
||
_drop_aliases(tree)
|
||
for node in tree.find_all(exp.Exists, exp.In):
|
||
query = node.this if isinstance(node, exp.Exists) else node.args.get("query")
|
||
_drop_aliases(query.this if isinstance(query, exp.Subquery) else query)
|
||
for sub in tree.find_all(exp.Subquery):
|
||
if not isinstance(sub.parent, (exp.From, exp.Join, exp.Exists, exp.In, exp.CTE)) and not sub.alias:
|
||
_drop_aliases(sub.this)
|
||
|
||
|
||
def _leaves(query: Any) -> list[Any]:
|
||
"""一棵 UNION / INTERSECT / EXCEPT 树的各支(套着括号的也拆开)。"""
|
||
from sqlglot import exp
|
||
|
||
if isinstance(query, exp.Subquery):
|
||
return _leaves(query.this)
|
||
if isinstance(query, exp.SetOperation):
|
||
return _leaves(query.left) + _leaves(query.right)
|
||
return [query]
|
||
|
||
|
||
def _block_nodes(select: Any) -> Iterator[Any]:
|
||
"""这一段自己的节点:不进套着的查询;FROM / JOIN 的来源不算(ON 条件算)。"""
|
||
from sqlglot import exp
|
||
|
||
stack = [v for k, value in select.args.items() if k not in ("with_", "from_")
|
||
for v in (value if isinstance(value, list) else [value]) if isinstance(v, exp.Expr)]
|
||
while stack:
|
||
node = stack.pop()
|
||
if isinstance(node, (exp.Query, exp.Subquery)):
|
||
continue
|
||
if isinstance(node, exp.Join):
|
||
if node.args.get("on") is not None:
|
||
stack.append(node.args["on"])
|
||
continue
|
||
yield node
|
||
stack += [v for value in node.args.values() for v in (value if isinstance(value, list) else [value])
|
||
if isinstance(v, exp.Expr)]
|
||
|
||
|
||
def _alias_refs(select: Any) -> set[str]:
|
||
"""这一段里可能按名字引用本段列别名的不带表名的名字:WHERE、ON、GROUP BY、HAVING、ORDER BY、窗口里的,以及套在
|
||
本段里的子查询里的全部(SQLite 的 WHERE、ON、相关子查询都认外层的列别名:EXISTS (… WHERE r.amount = b) 里的 b
|
||
是外层 SELECT amount AS b 的 b,在真实的 SQLite 上试过)。SELECT 列表自己的表达式不算(库里不认同一个列表里前面
|
||
的别名),FROM 里的来源不算。改名、去掉列别名之前(列名表写进定义、EXISTS 写成 SELECT 1、去掉没用的列名、CTE 的
|
||
列改叫 c1…)都先看这里,有同名的就不动:不然这些名字就指到别处去了。"""
|
||
from sqlglot import exp
|
||
|
||
out: set[str] = set()
|
||
stack: list[tuple[Any, bool]] = []
|
||
for key, value in select.args.items():
|
||
if key in ("with_", "from_"):
|
||
continue
|
||
for v in (value if isinstance(value, list) else [value]):
|
||
if isinstance(v, exp.Expr):
|
||
stack.append((v, key == "expressions"))
|
||
while stack:
|
||
node, projection = stack.pop()
|
||
if isinstance(node, exp.Join):
|
||
if node.args.get("on") is not None:
|
||
stack.append((node.args["on"], False))
|
||
continue
|
||
if isinstance(node, (exp.Query, exp.Subquery)):
|
||
out |= {c.name for c in node.find_all(exp.Column) if not c.table}
|
||
continue
|
||
if isinstance(node, exp.Window):
|
||
out |= {c.name for c in node.find_all(exp.Column) if not c.table}
|
||
continue
|
||
if isinstance(node, exp.Column) and not node.table and not projection:
|
||
out.add(node.name)
|
||
stack += [(child, projection) for child in node.iter_expressions()]
|
||
return out
|
||
|
||
|
||
def _nested_queries(select: Any) -> list[Any]:
|
||
"""这一段自己的表达式里直接套着的查询(条件、SELECT 列表、ON 里的子查询),不含 FROM 里的子查询。"""
|
||
from sqlglot import exp
|
||
|
||
out: list[Any] = []
|
||
stack = [v for k, value in select.args.items() if k not in ("with_", "from_")
|
||
for v in (value if isinstance(value, list) else [value]) if isinstance(v, exp.Expr)]
|
||
while stack:
|
||
node = stack.pop()
|
||
if isinstance(node, exp.Subquery):
|
||
out.append(node.this)
|
||
elif isinstance(node, exp.Query):
|
||
out.append(node)
|
||
elif isinstance(node, exp.Join):
|
||
if node.args.get("on") is not None:
|
||
stack.append(node.args["on"])
|
||
else:
|
||
stack += [v for value in node.args.values() for v in (value if isinstance(value, list) else [value])
|
||
if isinstance(v, exp.Expr)]
|
||
return out
|
||
|
||
|
||
def _starry(select: Any) -> bool:
|
||
from sqlglot import exp
|
||
|
||
return any(isinstance(p, exp.Star) or (isinstance(p, exp.Column) and p.is_star) for p in select.expressions)
|
||
|
||
|
||
def _alias_or_column(tree: Any, read: str | None, schema: dict[str, dict[str, str]] | None) -> bool:
|
||
"""有不带表名、又和这一段的列别名同名的名字,qualify 会把它当成别名展开,库里认的却可能是表的列,这一遍不认列:
|
||
|
||
- WHERE、GROUP BY、HAVING 里:库里认的是列还是别名各方言不一样(SQLite 的 GROUP BY、HAVING 都先认列:HAVING
|
||
amount > 10 里的 amount 是组里任意一行的 amount,不是 SUM(amount) AS amount),qualify 却一律当成别名(GROUP BY
|
||
created_at 成了按 DATE(created_at) 分组)。表结构里有这一段的每张表、这个名字又不是其中哪张表的列时,它只能是
|
||
别名,照常认。
|
||
- SELECT 列表里引用前面列的别名:SQLite、PostgreSQL、MySQL 都不认,名字指的是表的列(SELECT user_id AS amount,
|
||
amount * 2 里的 amount 是表的 amount 列),不是表的列就报错;qualify 没有表结构时却当成别名展开成 user_id * 2,
|
||
有表结构时不是表的列的也展开。表结构里有这一段的每张表、这个名字又是其中的列时,qualify 先认列,照常认。"""
|
||
from sqlglot import exp
|
||
|
||
columns = {name.lower(): {c.lower() for c in cols} for name, cols in (schema or {}).items()}
|
||
for select in tree.find_all(exp.Select):
|
||
aliases = {p.alias.lower() for p in select.expressions if isinstance(p, exp.Alias) and p.alias}
|
||
if not aliases:
|
||
continue
|
||
items = _items_of(select)
|
||
known = all(isinstance(i, exp.Table) and i.name.lower() in columns for i in items)
|
||
real = set().union(*(columns[i.name.lower()] for i in items)) if known and items else set()
|
||
for key in ("where", "group", "having"):
|
||
node = select.args.get(key)
|
||
if node is None:
|
||
continue
|
||
names = {c.name.lower() for c in node.find_all(exp.Column) if not c.table and not _nested(c, node)}
|
||
if (names & aliases) - (set() if not known else (aliases - real)):
|
||
return True
|
||
earlier: set[str] = set()
|
||
for projection in select.expressions:
|
||
names = {c.name.lower() for c in projection.find_all(exp.Column)
|
||
if not c.table and (c is projection or not _nested(c, projection))}
|
||
if (names & earlier) - real:
|
||
return True
|
||
if isinstance(projection, exp.Alias) and projection.alias:
|
||
earlier.add(projection.alias.lower())
|
||
# 套在这一段里的子查询用了和这一段列别名同名、不带表名的名字:SQLite 的相关子查询认外层的列别名(EXISTS (…
|
||
# WHERE r.amount = b) 里的 b 是外层 SELECT id AS b 的 id),qualify 却把它归给子查询里的表(r.b),两条把 b 指向
|
||
# 不同列的语句会认成一样
|
||
nested = {c.name.lower() for sub in _nested_queries(select) for c in sub.find_all(exp.Column) if not c.table}
|
||
if nested & aliases:
|
||
return True
|
||
return False
|
||
|
||
|
||
# ---- 起规范的名字
|
||
|
||
@dataclass
|
||
class _Hint:
|
||
"""按另一边起名(差异说明用):定义一样的 CTE 叫另一边的名字;names 是这一边原来的 CTE 名 → 另一边的名字;taken
|
||
是另一边用过的名字,配不上的另起名字时避开。"""
|
||
|
||
bodies: dict[str, str] = field(default_factory=dict)
|
||
names: dict[str, str] = field(default_factory=dict)
|
||
taken: set[str] = field(default_factory=set)
|
||
|
||
|
||
class _Namer:
|
||
"""给一条语句起规范的名字(见 _finish 第 3 步)。
|
||
|
||
先按原来的别名认清每一列属于哪个来源(见 _Resolver);之后改来源的别名时把这些列一起改掉,不再按名字找,换名不会
|
||
把列认错。别名带层数:内外层的别名不会同名,相关子查询里引用外层的列不会被里层遮住;同一层的兄弟子查询各自看不见
|
||
对方,可以重名。"""
|
||
|
||
def __init__(self, tree: Any, read: str | None, qualified: bool, sens: _Sensitivity,
|
||
hint: _Hint | None = None) -> None:
|
||
from sqlglot import exp
|
||
|
||
self.tree = tree
|
||
self.read = read
|
||
self.qualified = qualified
|
||
self.sens = sens
|
||
self.hint = hint
|
||
resolver = _Resolver(tree)
|
||
self.refs = resolver.refs
|
||
self.cte_of = resolver.cte_of
|
||
self.base_names = {t.name.lower() for t in tree.find_all(exp.Table) if t.name}
|
||
self.budget = _ORDERING_BUDGET
|
||
self.starred = self._starred_sources(tree)
|
||
|
||
def _starred_sources(self, tree: Any) -> set[int]:
|
||
"""被 SELECT * 读着的来源(* 没展开):它选出来的列名、先后会原样传到外面,不能改名、重排。"""
|
||
from sqlglot import exp
|
||
|
||
out: set[int] = set()
|
||
for select in tree.find_all(exp.Select):
|
||
for p in select.expressions:
|
||
if isinstance(p, exp.Star):
|
||
out |= {id(i) for i in _items_of(select)}
|
||
elif isinstance(p, exp.Column) and p.is_star:
|
||
out |= {id(i) for i in _items_of(select) if i.alias_or_name == p.table}
|
||
return out
|
||
|
||
def _text(self, node: Any) -> tuple[str, str]:
|
||
return _key(node, self.read)
|
||
|
||
def _alias(self, base: str, depth: int, k: int) -> str:
|
||
name = base if depth == 0 else f"{base}@{depth}"
|
||
if k > 1:
|
||
name = f"{name}#{k}"
|
||
while (depth or k > 1) and name.lower() in self.base_names:
|
||
name += "#"
|
||
return name
|
||
|
||
def canon(self, query: Any, depth: int, cte_names: dict[int, str]) -> tuple[str, str]:
|
||
from sqlglot import exp
|
||
|
||
if isinstance(query, exp.Subquery):
|
||
return self.canon(query.this, depth, cte_names)
|
||
with_ = query.args.get("with_")
|
||
if with_ is not None:
|
||
cte_names = dict(cte_names)
|
||
self._name_ctes(with_, depth, cte_names)
|
||
if isinstance(query, exp.SetOperation):
|
||
self._canon_setop(query, depth, cte_names)
|
||
return self._text(query)
|
||
return self._canon_select(query, depth, cte_names)
|
||
|
||
def _name_ctes(self, with_: Any, depth: int, cte_names: dict[int, str]) -> None:
|
||
"""CTE 按依赖分层,同一层按规范化后的定义排序起名;定义一模一样的合成一个(引用它们的都改叫同一个名字)。"""
|
||
from sqlglot import exp
|
||
|
||
hint = self.hint if depth == 0 else None
|
||
pending = list(with_.expressions)
|
||
kept: list[Any] = []
|
||
by_text: dict[tuple[str, str], str] = {}
|
||
prefix = "cte" if depth == 0 else f"cte@{depth}_"
|
||
n = 0
|
||
while pending:
|
||
names = {c.alias_or_name for c in pending}
|
||
ready = [c for c in pending if not any(t.name in names and t.name != c.alias_or_name and not t.args.get("db")
|
||
for t in c.this.find_all(exp.Table))]
|
||
if not ready:
|
||
raise _Unsupported("CTE 互相引用")
|
||
texts = []
|
||
for cte in ready:
|
||
original = cte.alias_or_name
|
||
body = self.canon(cte.this, depth + 1 if depth else 0, cte_names)
|
||
refs = [t for t in self.tree.find_all(exp.Table) if self.cte_of.get(id(t)) is cte]
|
||
if self._rename_outputs(cte.this, refs):
|
||
body = self._text(cte.this)
|
||
texts.append((body, original, cte))
|
||
texts.sort(key=lambda x: x[0])
|
||
for body, original, cte in texts:
|
||
pending.remove(cte)
|
||
if body in by_text:
|
||
cte_names[id(cte)] = by_text[body]
|
||
continue
|
||
name = (hint.names.get(original) or hint.bodies.get(body[0])) if hint else None
|
||
while name is None or name.lower() in self.base_names or name in by_text.values():
|
||
n += 1
|
||
name = f"{prefix}{n}"
|
||
if hint and name in hint.taken:
|
||
name = None
|
||
cte_names[id(cte)] = name
|
||
by_text[body] = name
|
||
cte.set("alias", exp.TableAlias(this=exp.to_identifier(name)))
|
||
kept.append(cte)
|
||
with_.set("expressions", kept)
|
||
|
||
def _rename_outputs(self, body: Any, holders: list[Any]) -> bool:
|
||
"""(认列之后)CTE、FROM 里的子查询选出来的列改叫 c1、c2…,引用它们的列跟着改:外面按名字引用,名字是各自起的
|
||
(total 还是 sum_amt)。没有被 SELECT * 原样读着(* 没展开时列名、先后会传到外面)的,先按定义排序:外面按名字
|
||
引用,先后不影响。UNION、选了 *、列名重复、本段里还按名字引用自己的列、GROUP BY / ORDER BY 按位置引用的不改。"""
|
||
from sqlglot import exp
|
||
|
||
if not self.qualified or not isinstance(body, exp.Select) or any(id(h) in self.starred for h in holders):
|
||
return False
|
||
projections = body.expressions
|
||
names = [p.alias_or_name for p in projections]
|
||
if not projections or any(not n or _is_star(p) for n, p in zip(names, projections)) or \
|
||
len(set(names)) != len(names):
|
||
return False
|
||
# 本段(含相关子查询)按名字引用了自己的列别名、或者正好用了要起的名字 c1…:改名之后就指到别处了。没写别名
|
||
# 的光一列(SELECT user_id … WHERE user_id > 1)改不改名,WHERE 里的 user_id 都还是那一列
|
||
aliases = {p.alias for p in projections if isinstance(p, exp.Alias) and p.alias}
|
||
if _alias_refs(body) & (aliases | {f"c{i}" for i in range(1, len(projections) + 1)}):
|
||
return False
|
||
for clause in ("group", "order"):
|
||
node = body.args.get(clause)
|
||
keys = [k.this if isinstance(k, exp.Ordered) else k for k in (node.expressions if node is not None else [])]
|
||
if any(k.is_number for k in keys):
|
||
return False
|
||
ordered = sorted(projections, key=lambda p: self._text(p.unalias()))
|
||
mapping = {p.alias_or_name: f"c{i}" for i, p in enumerate(ordered, 1)}
|
||
body.set("expressions", [exp.alias_(p.unalias().copy(), mapping[p.alias_or_name], copy=False)
|
||
for p in ordered])
|
||
for holder in holders:
|
||
for col in self.refs.get(id(holder), []):
|
||
if col.name in mapping:
|
||
col.set("this", exp.to_identifier(mapping[col.name]))
|
||
return True
|
||
|
||
def _canon_setop(self, query: Any, depth: int, cte_names: dict[int, str]) -> None:
|
||
"""连着同一种合并写的几支各自规范化;调换先后不改变结果的(见 _reorderable)再按规范化后的写法排序。"""
|
||
kind = (type(query), bool(query.args.get("distinct")))
|
||
branches: list[Any] = []
|
||
|
||
def collect(node: Any) -> None:
|
||
from sqlglot import exp
|
||
|
||
if isinstance(node, exp.SetOperation) and (type(node), bool(node.args.get("distinct"))) == kind and \
|
||
not any(node.args.get(k) for k in ("order", "limit", "offset", "with_")):
|
||
collect(node.left)
|
||
collect(node.right)
|
||
else:
|
||
branches.append(node)
|
||
|
||
collect(query.left)
|
||
collect(query.right)
|
||
texts = [(self.canon(b, depth, cte_names), i, b) for i, b in enumerate(branches)]
|
||
if len(branches) < 2 or not self._reorderable(query):
|
||
return
|
||
ordered = [b for _, _, b in sorted(texts, key=lambda x: (x[0], x[1]))]
|
||
node = ordered[0]
|
||
for b in ordered[1:-1]:
|
||
node = type(query)(this=node, expression=b, distinct=query.args.get("distinct"))
|
||
query.set("this", node)
|
||
query.set("expression", ordered[-1])
|
||
|
||
def _reorderable(self, query: Any) -> bool:
|
||
"""这一处合并的各支调换先后,结果(行、值)一样不一样。只认整条语句最外层、不排序不取行的 UNION ALL:
|
||
|
||
- UNION(去重)、INTERSECT 留先出现的那个值,相等不等于一样(SQLite 里 REAL 列的 2.0 和 INTEGER 列的 2、MySQL
|
||
默认不分大小写的 'SH' 和 'sh');EXCEPT 本来就分左右。
|
||
- 带 LIMIT / OFFSET 的取的是先出来的几行;ORDER BY 按名字排的看第一支的列名;SQLite 给合并的结果排序时用最左
|
||
一支那列声明的排序规则。PostgreSQL、MySQL 按位置排序(ORDER BY 1)的可以排。
|
||
- 套在 CTE、子查询里的:SQLite 从子查询读合并的结果时按第一支的类型亲和、排序规则转换别的支,不排;别的方言
|
||
只在各支选出来的列名、先后都一样(外面按列名引用,列名取第一支的)、外面不看行出来的先后(不去重、不取行,
|
||
见 _Sensitivity)时排。"""
|
||
from sqlglot import exp
|
||
|
||
if not isinstance(query, exp.Union) or query.args.get("distinct"):
|
||
return False
|
||
if any(query.args.get(k) is not None for k in ("limit", "offset", "fetch")):
|
||
return False
|
||
sqlite = self.read in (None, "sqlite")
|
||
order = query.args.get("order")
|
||
if order is not None and (sqlite or any(not (isinstance(o.this, exp.Literal) and o.this.is_int)
|
||
for o in order.expressions)):
|
||
return False
|
||
top = query
|
||
while isinstance(top.parent, exp.Subquery):
|
||
top = top.parent
|
||
if top is self.tree:
|
||
return True
|
||
if sqlite or order is not None:
|
||
return False
|
||
leaves = _leaves(query)
|
||
if any(not isinstance(leaf, exp.Select) or self.sens.of(leaf) or _starry(leaf) for leaf in leaves):
|
||
return False
|
||
names = [[p.alias_or_name for p in leaf.expressions] for leaf in leaves]
|
||
return all(n == names[0] for n in names) and all(names[0])
|
||
|
||
def _signature(self, item: Any, cte_names: dict[int, str], subs: dict[int, tuple[str, str]]) -> tuple[str, str]:
|
||
"""来源的 (排序用的标识, 起别名用的名字):实表是表名,CTE 是它的规范名,FROM 里的子查询是规范化后的写法。"""
|
||
from sqlglot import exp
|
||
|
||
if isinstance(item, exp.Table):
|
||
cte = self.cte_of.get(id(item))
|
||
if cte is not None:
|
||
name = cte_names.get(id(cte), cte.alias_or_name)
|
||
return f"cte:{name}", name
|
||
return "table:" + ".".join(p for p in (item.catalog, item.db, item.name) if p), item.name
|
||
return f"sub:{subs[id(item)][0]}", "sub"
|
||
|
||
def _canon_select(self, select: Any, depth: int, cte_names: dict[int, str]) -> tuple[str, str]:
|
||
from sqlglot import exp
|
||
|
||
items = _items_of(select)
|
||
subs: dict[int, tuple[str, str]] = {}
|
||
for item in items:
|
||
if isinstance(item, exp.Subquery):
|
||
subs[id(item)] = self.canon(item.this, depth + 1, cte_names)
|
||
if self._rename_outputs(item.this, [item]):
|
||
subs[id(item)] = self._text(item.this)
|
||
elif id(item) in self.cte_of:
|
||
cte = self.cte_of[id(item)]
|
||
item.set("this", exp.to_identifier(cte_names.get(id(cte), cte.alias_or_name)))
|
||
keys = {id(i): self._signature(i, cte_names, subs) for i in items}
|
||
joins = select.args.get("joins") or []
|
||
n_prefix = 0
|
||
if items:
|
||
n_prefix = 1
|
||
for join in joins:
|
||
if _comma(join):
|
||
n_prefix += 1
|
||
else:
|
||
break
|
||
prefix, rest = items[:n_prefix], joins[max(n_prefix - 1, 0):] if items else joins
|
||
# 后面有 RIGHT / FULL JOIN 时开头那组来源不调换:PostgreSQL、MySQL 里 a, b RIGHT JOIN c 是 a 乘上 (b RIGHT JOIN c),
|
||
# 换成 b, a 就成了 b 乘上 (a RIGHT JOIN c)
|
||
movable = not _bare_star(select) and not self.sens.of(select) and \
|
||
not any(str(j.side or "").upper() in ("RIGHT", "FULL") for j in joins)
|
||
free_lefts = movable and bool(rest) and self._independent_lefts(prefix, rest)
|
||
|
||
def groups(xs: list[Any], key: Any) -> list[list[Any]]:
|
||
xs = sorted(xs, key=key)
|
||
return [list(g) for _, g in itertools.groupby(xs, key=key)]
|
||
|
||
prefix_groups = groups(prefix, lambda s: keys[id(s)][0]) if movable else [[s] for s in prefix]
|
||
rest_groups = groups(rest, lambda j: keys[id(j.this)][0]) if free_lefts else [[j] for j in rest]
|
||
total = 1
|
||
for g in prefix_groups + rest_groups:
|
||
total *= math.factorial(len(g))
|
||
if total > _MAX_ORDERINGS:
|
||
break
|
||
if total > _MAX_ORDERINGS or total > self.budget:
|
||
# 排列太多(十几个同名来源):先算个数再展开,超了就只按排好的先后
|
||
options = [[g] for g in prefix_groups + rest_groups]
|
||
total = 1
|
||
else:
|
||
options = [[list(p) for p in itertools.permutations(g)] if len(g) > 1 else [g]
|
||
for g in prefix_groups + rest_groups]
|
||
self.budget -= total
|
||
combos = list(itertools.product(*options))
|
||
best: tuple[tuple[str, str], int] | None = None
|
||
for index, combo in enumerate(combos):
|
||
text = self._apply(select, depth, cte_names, keys, combo, len(prefix_groups))
|
||
if best is None or text < best[0]:
|
||
best = (text, index)
|
||
if best is None:
|
||
return self._text(select)
|
||
if len(combos) > 1:
|
||
self._apply(select, depth, cte_names, keys, combos[best[1]], len(prefix_groups))
|
||
return best[0]
|
||
|
||
@staticmethod
|
||
def _independent_lefts(prefix: list[Any], rest: list[Any]) -> bool:
|
||
"""跟在一组内连接后面的几个 LEFT JOIN,各自的 ON 只引用这组来源和自己(不引用别的 LEFT JOIN 进来的表、没有
|
||
子查询):它们之间谁先谁后结果一样。"""
|
||
from sqlglot import exp
|
||
|
||
names = {s.alias_or_name for s in prefix}
|
||
for join in rest:
|
||
if str(join.side or "").upper() != "LEFT" or not _plain_join(join):
|
||
return False
|
||
on = join.args.get("on")
|
||
if on is None or on.find(exp.Select, exp.Subquery) is not None:
|
||
return False
|
||
if not {c.table for c in on.find_all(exp.Column)} <= names | {join.this.alias_or_name}:
|
||
return False
|
||
return True
|
||
|
||
def _apply(self, select: Any, depth: int, cte_names: dict[int, str], keys: dict[int, tuple[str, str]],
|
||
combo: tuple[list[Any], ...], n_prefix_groups: int) -> tuple[str, str]:
|
||
from sqlglot import exp
|
||
|
||
prefix = [s for g in combo[:n_prefix_groups] for s in g]
|
||
rest = [j for g in combo[n_prefix_groups:] for j in g]
|
||
if prefix:
|
||
select.args["from_"].set("this", prefix[0])
|
||
select.set("joins", [exp.Join(this=s) for s in prefix[1:]] + rest or None)
|
||
counts: dict[str, int] = {}
|
||
for item in _items_of(select):
|
||
_, base = keys[id(item)]
|
||
counts[base] = counts.get(base, 0) + 1
|
||
name = self._alias(base, depth, counts[base])
|
||
item.set("alias", exp.TableAlias(this=exp.to_identifier(name)))
|
||
for col in self.refs.get(id(item), []):
|
||
col.set("table", exp.to_identifier(name))
|
||
for sub in _nested_queries(select):
|
||
self.canon(sub, depth + 1, cte_names)
|
||
_sort_own(select, self.read, self.sens.of(select))
|
||
return self._text(select)
|
||
|
||
|
||
def _fixed(node: Any) -> bool:
|
||
"""不涉及任何列、子查询的表达式(常量、常量算出来的日期):排比较、加乘的两边时放在右边,好读。"""
|
||
from sqlglot import exp
|
||
|
||
return node.find(exp.Column, exp.Select, exp.Subquery) is None
|
||
|
||
|
||
def _sort_own(select: Any, read: str | None, sensitive: bool = False) -> None:
|
||
"""这一段自己的表达式里交换律成立的地方排好(里层先排):AND、OR(各项都是条件时去掉一模一样的项:x AND x 在
|
||
SELECT 列表里不是 x)、比较(不涉及列的一边在右,两边都涉及列时排好、大于小于跟着换方向;两边都写了 COLLATE 的
|
||
不换,SQLite 用左边那个)、两项的加和乘(只换这一个节点的两边,不跨项挪动:浮点数加法不满足结合律)、IN 列表、
|
||
GROUP BY(结果看行出来的先后时不排:SQLite 按分组键排好了出行)。"""
|
||
from sqlglot import exp
|
||
|
||
def key(node: Any) -> tuple[str, str]:
|
||
return _key(node, read)
|
||
|
||
def depth(node: Any) -> int:
|
||
d = 0
|
||
while node is not select and node.parent is not None:
|
||
node = node.parent
|
||
d += 1
|
||
return d
|
||
|
||
for node in sorted(_block_nodes(select), key=depth, reverse=True):
|
||
if node is not select and node.parent is None:
|
||
continue
|
||
if isinstance(node, (exp.And, exp.Or)):
|
||
if isinstance(_unparen_parent(node), type(node)):
|
||
continue
|
||
parts = _flat(node, type(node))
|
||
if all(_predicate(p) for p in parts) and not sensitive:
|
||
# 一模一样的条件去掉一个。结果看行出来先后的一段不去:MySQL 按条件估算行数,同一个条件写两遍就换了一种
|
||
# 关联的先后,GROUP_CONCAT 拼出来的先后跟着变(第 3 轮在真实的 MySQL 8.0 上压测时看到的)
|
||
uniq: dict[tuple[str, str], Any] = {}
|
||
for part in parts:
|
||
uniq.setdefault(key(part), part)
|
||
ordered = [uniq[k] for k in sorted(uniq)]
|
||
else:
|
||
ordered = sorted(parts, key=key)
|
||
new = ordered[0] if len(ordered) == 1 else (
|
||
exp.and_(*ordered, copy=False) if isinstance(node, exp.And) else exp.or_(*ordered, copy=False))
|
||
node.replace(new)
|
||
elif type(node).__name__ in _FLIP:
|
||
left, right = node.this, node.expression
|
||
if _quantified(left) or _quantified(right):
|
||
continue
|
||
if left.find(exp.Collate) is not None and right.find(exp.Collate) is not None:
|
||
continue
|
||
if (_fixed(left) and not _fixed(right)) or (_fixed(left) == _fixed(right) and key(left) > key(right)):
|
||
node.replace(getattr(exp, _FLIP[type(node).__name__])(this=right, expression=left))
|
||
elif isinstance(node, (exp.Add, exp.Mul)):
|
||
left, right = node.this, node.expression
|
||
if (_fixed(left) and not _fixed(right)) or (_fixed(left) == _fixed(right) and key(left) > key(right)):
|
||
node.set("this", right)
|
||
node.set("expression", left)
|
||
elif isinstance(node, exp.Group):
|
||
# 一模一样的分组键写两遍和写一遍分出的组一样;结果看行出来先后的一段不动(分组键变了可能换一种扫描的先后)
|
||
if node.expressions and not sensitive:
|
||
uniq_keys: dict[tuple[str, str], Any] = {}
|
||
for k in node.expressions:
|
||
uniq_keys.setdefault(key(k), k)
|
||
node.set("expressions", sorted(uniq_keys.values(), key=key))
|
||
elif isinstance(node, exp.In) and node.expressions:
|
||
uniq = {key(i): i for i in node.expressions}
|
||
node.set("expressions", [uniq[k] for k in sorted(uniq)])
|
||
|
||
|
||
def _flat(node: Any, cls: type) -> list[Any]:
|
||
"""AND / OR 连着的各项(穿过括号:BETWEEN 改写出来的 (x >= a AND x <= b) 也拆开)。"""
|
||
out: list[Any] = []
|
||
stack = [node]
|
||
while stack:
|
||
n = stack.pop()
|
||
inner = _unparen(n)
|
||
if isinstance(inner, cls):
|
||
stack += [inner.expression, inner.this]
|
||
else:
|
||
out.append(n)
|
||
return out
|
||
|
||
|
||
def _unparen_parent(node: Any) -> Any:
|
||
from sqlglot import exp
|
||
|
||
parent = node.parent
|
||
while isinstance(parent, exp.Paren):
|
||
parent = parent.parent
|
||
return parent
|
||
|
||
|
||
# ==========================================================================
|
||
# 差异说明
|
||
# ==========================================================================
|
||
|
||
|
||
@dataclass(frozen=True)
|
||
class FacetDiff:
|
||
"""一处不同。aspect:time(时间列)/ time_range(同一时间列、范围不同)/ filter(筛选条件)/ aggregate(聚合函数、
|
||
去重计数)/ expression(选出来的列)/ join(关联的表、关联类型、关联条件、外连接的主表)/ tables(查的是哪张表)/
|
||
set_op(几支查询怎么合并)/ group_by(分组键、去重)/ having / limit(取哪几行,含取行前的排序)/ order(只是最终
|
||
结果的排序不同,不影响取到哪些行)/ parse(解析不了)/ other(兜底:未能归纳成具体差异,见 explain_sql_diff)。a、b
|
||
是两边各自的写法(给人看,没有就是 None),text 是一句中文说明。detail 只在 time 上有:两边的时间列
|
||
[{table, column}](各是各的表),调用方据此对照数据目录(业务日期);在 filter 上有 {"columns": [{table, column}]}:
|
||
条件只涉及的实表列,对照有效行过滤用。"""
|
||
|
||
aspect: str
|
||
a: Any
|
||
b: Any
|
||
text: str
|
||
detail: dict[str, Any] | None = None
|
||
|
||
def as_dict(self) -> dict[str, Any]:
|
||
out = {"aspect": self.aspect, "a": self.a, "b": self.b, "text": self.text}
|
||
if self.detail is not None:
|
||
out["detail"] = self.detail
|
||
return out
|
||
|
||
|
||
@dataclass(frozen=True)
|
||
class SqlDiff:
|
||
"""两条 SQL 差在哪(explain_sql_diff 的结果)。
|
||
|
||
equivalent:两边经过只保持语义的变换之后,规范形式一字不差(level 是哪一级的规范形式,见 _canonical_forms)。
|
||
items:具体差异,等价时为空;不等价时一定不为空——归纳不出具体差异时是一条 aspect="other" 的兜底说明。
|
||
fallback:用了兜底说明(一条具体说法都没有,或者具体说法只覆盖了一部分)。
|
||
text_diff:规范化之后两条 SQL 不同的行(difflib 比两边 sqlglot pretty 的输出,WHERE、ON、HAVING 的合取式每个条件
|
||
一行,「- 」开头是 A 的、「+ 」开头是 B 的;不同的行前面带上它的各级上级行,「 」开头、两边相同,只为标出不同的行
|
||
在哪个子句里;最多 40 行,见 _text_diff),给界面高亮;等价时为空。"""
|
||
|
||
equivalent: bool
|
||
items: list[FacetDiff] = field(default_factory=list)
|
||
fallback: bool = False
|
||
text_diff: tuple[str, ...] = ()
|
||
level: str | None = None
|
||
|
||
def as_dict(self) -> dict[str, Any]:
|
||
return {"equivalent": self.equivalent, "items": [d.as_dict() for d in self.items], "fallback": self.fallback,
|
||
"text_diff": list(self.text_diff), "level": self.level}
|
||
|
||
|
||
#: 兜底说明
|
||
_FALLBACK = "两条 SQL 的写法不同,未能归纳成具体差异,请对照原文"
|
||
_FALLBACK_REST = "另有未能归纳的差异,请对照原文"
|
||
#: text_diff 最多列几行
|
||
_MAX_DIFF_LINES = 40
|
||
#: 逐行比较时不看的、条件前面的 AND(见 _text_diff)
|
||
_LEADING_AND = re.compile(r"^(\s*)AND ")
|
||
#: 各方言「每个条件一行」的生成器类,按 sqlglot 的生成器类缓存(见 _line_generator)
|
||
_LINE_GENERATORS: dict[type, type] = {}
|
||
|
||
_CJK_PUNCT = frozenset(",。;:、!?「」『』()《》【】…—")
|
||
#: 聚合函数的说法
|
||
_VERBS = {"sum": "求和", "count": "计数", "avg": "求平均", "min": "取最小值", "max": "取最大值"}
|
||
#: 只取日期的字符串(2026-09-01):PostgreSQL、MySQL 里拿它和时间列比,按字符串比的先后和按日期比的一样
|
||
_DATE_ONLY = re.compile(r"^\d{4}-\d{2}-\d{2}$")
|
||
_COMPARE_OPS = ("EQ", "NEQ", "GT", "GTE", "LT", "LTE")
|
||
#: LIKE 的对象的两种取值(见 _Range.always):匹配上模式的串、匹配不上的串
|
||
_LIKE_MATCH, _LIKE_MISS = "\x00like:match", "\x00like:miss"
|
||
#: 判断恒为真时最多试多少种取值的组合:再多就当认不出(只会少说)
|
||
_MAX_COMBINATIONS = 20000
|
||
|
||
|
||
def _wide(ch: str) -> bool:
|
||
return "⺀" <= ch <= "鿿" or "豈" <= ch <= "" or "" <= ch <= ""
|
||
|
||
|
||
def _say(*parts: str) -> str:
|
||
"""拼一句话:汉字和英文、数字、SQL 之间加一个空格,挨着全角标点的不加,汉字和汉字之间不加。
|
||
|
||
两边的叫法由调用方定(A / B,或者「原流程」「盲算」),所以空格只能拼的时候按字判断。
|
||
"""
|
||
out = ""
|
||
for part in parts:
|
||
if not part:
|
||
continue
|
||
if out and not out[-1].isspace() and not part[0].isspace():
|
||
a, b = out[-1], part[0]
|
||
if not (a in _CJK_PUNCT or b in _CJK_PUNCT or (_wide(a) and _wide(b))):
|
||
out += " "
|
||
out += part
|
||
return out
|
||
|
||
|
||
def catalog_labels(notes: Mapping[str, Mapping[str, Any]] | None) -> dict[str, str]:
|
||
"""数据目录({表名: notes})里列的中文名 → diff_facets / explain_sql_diff 的 names:{「表.列」: 中文名};只在一张
|
||
表里有、或者各表叫法一致的列另记 {列: 中文名}。被驳回的项不算。"""
|
||
out: dict[str, str] = {}
|
||
bare: dict[str, tuple[str, set[str]]] = {}
|
||
for table, raw in (notes or {}).items():
|
||
visible = catalog.visible_notes(raw) if isinstance(raw, Mapping) else {}
|
||
columns = visible.get("columns") if isinstance(visible.get("columns"), Mapping) else {}
|
||
for column, items in columns.items():
|
||
item = items.get("label") if isinstance(items, Mapping) else None
|
||
label = item.get("value") if isinstance(item, Mapping) else None
|
||
if isinstance(label, str) and label.strip():
|
||
out[f"{table}.{column}"] = label.strip()
|
||
bare.setdefault(name_key(str(column)), (str(column), set()))[1].add(label.strip())
|
||
for column, labels in bare.values():
|
||
if len(labels) == 1:
|
||
out.setdefault(column, next(iter(labels)))
|
||
return out
|
||
|
||
|
||
def _labeler(names: Mapping[str, str] | None) -> Any:
|
||
index = {name_key(str(k)): str(v) for k, v in (names or {}).items() if isinstance(v, str) and v.strip()}
|
||
|
||
def label(ref: ColumnRef) -> str | None:
|
||
table, column = ref
|
||
return (index.get(name_key(f"{table}.{column}")) if table else None) or index.get(name_key(column))
|
||
|
||
return label
|
||
|
||
|
||
# ---- 同一个对象上的条件:留下的取值一样不一样
|
||
|
||
def _literal_value(node: Any) -> tuple[str, Any] | None:
|
||
"""常量的值:("n", Decimal) 或 ("s", 字符串);不是数字、字符串常量返回 None。"""
|
||
from decimal import Decimal, InvalidOperation
|
||
|
||
from sqlglot import exp
|
||
|
||
node = _unparen(node)
|
||
if isinstance(node, exp.Neg):
|
||
inner = _literal_value(node.this)
|
||
return ("n", -inner[1]) if inner is not None and inner[0] == "n" else None
|
||
if isinstance(node, exp.Literal):
|
||
if node.is_string:
|
||
return "s", str(node.this)
|
||
try:
|
||
return "n", Decimal(str(node.this))
|
||
except (InvalidOperation, ValueError):
|
||
return None
|
||
return None
|
||
|
||
|
||
class _Range:
|
||
"""一个对象(一列、一个聚合、一个函数)上只和常量比的条件(=、<>、<、<=、>、>=、IN 常量列表、IS [NOT] NULL、
|
||
BETWEEN,用 AND / OR / NOT 连起来),在对象的各段取值上成立不成立:用到的常量把取值切成一段段(小于最小的、
|
||
每个常量本身、相邻两个之间、大于最大的),每段里条件的真假都一样,各取一个代表算一遍,再算对象是空值时。两组
|
||
条件在每一段、空值上都同真同不真,留下的行就一样。
|
||
|
||
字符串的大小先后:SQLite 按默认的 BINARY 排序规则;PostgreSQL(语言排序规则下 'a' < 'B')、MySQL(默认不分
|
||
大小写)只认等不等,除非常量都是只取日期的字符串(2026-09-01,按字符串比和按日期比先后一样)。MySQL 的字符串
|
||
等不等不分大小写。COUNT 的取值是整数(分组时至少是 1)。"""
|
||
|
||
def __init__(self, read: str | None) -> None:
|
||
self.read = read
|
||
|
||
def subject(self, cond: Any) -> tuple[str, Any, list[tuple[str, Any]], bool] | None:
|
||
"""cond 只是同一个对象和常量比时,返回 (对象的指纹, 对象, 常量, 有没有比大小);不然 None。"""
|
||
from sqlglot import exp
|
||
|
||
subjects: dict[str, Any] = {}
|
||
literals: list[tuple[str, Any]] = []
|
||
ranged = [False]
|
||
|
||
def side(subj: Any, lits: list[Any]) -> bool:
|
||
if subj is None or _literal_value(subj) is not None or isinstance(_unparen(subj), exp.Null) or \
|
||
subj.find(exp.Query, exp.Subquery) is not None or _volatile(subj):
|
||
return False
|
||
subjects[_fingerprint(subj)] = subj
|
||
literals.extend(lits)
|
||
return True
|
||
|
||
def walk(n: Any) -> bool:
|
||
n = _unparen(n)
|
||
if isinstance(n, (exp.And, exp.Or)):
|
||
return walk(n.left) and walk(n.right)
|
||
if isinstance(n, exp.Not):
|
||
return walk(n.this)
|
||
name = type(n).__name__
|
||
if name in _COMPARE_OPS:
|
||
lv, rv = _literal_value(n.expression), _literal_value(n.this)
|
||
if name not in ("EQ", "NEQ"):
|
||
ranged[0] = True
|
||
if lv is not None and rv is None:
|
||
return side(n.this, [lv])
|
||
if rv is not None and lv is None:
|
||
return side(n.expression, [rv])
|
||
return False
|
||
if isinstance(n, exp.In):
|
||
if n.args.get("query") or n.args.get("unnest") or n.args.get("field") or not n.expressions:
|
||
return False
|
||
lits = []
|
||
for item in n.expressions:
|
||
if isinstance(_unparen(item), exp.Null):
|
||
continue
|
||
value = _literal_value(item)
|
||
if value is None:
|
||
return False
|
||
lits.append(value)
|
||
return side(n.this, lits)
|
||
if isinstance(n, exp.Is):
|
||
return isinstance(_unparen(n.expression), exp.Null) and side(n.this, [])
|
||
if n.args.get("negate"):
|
||
return False
|
||
if isinstance(n, exp.Between):
|
||
low, high = _literal_value(n.args.get("low")), _literal_value(n.args.get("high"))
|
||
ranged[0] = True
|
||
return not n.args.get("symmetric") and low is not None and high is not None and \
|
||
side(n.this, [low, high])
|
||
return False
|
||
|
||
if not walk(cond) or len(subjects) != 1:
|
||
return None
|
||
kinds = {k for k, _ in literals}
|
||
if len(kinds) > 1:
|
||
return None
|
||
if ranged[0] and kinds == {"s"} and self.read != "sqlite" and \
|
||
not all(_DATE_ONLY.match(v) for _, v in literals):
|
||
return None
|
||
fp, subj = next(iter(subjects.items()))
|
||
return fp, subj, literals, ranged[0]
|
||
|
||
def _norm(self, value: tuple[str, Any]) -> Any:
|
||
"""MySQL 默认的排序规则不分大小写、比较时不看末尾的空格(PAD SPACE:'paid' = 'paid '),按这个规则比的值。"""
|
||
kind, v = value
|
||
if kind == "s" and self.read == "mysql":
|
||
return v.casefold().rstrip(" ")
|
||
return v
|
||
|
||
def _pieces(self, subject: Any, literals: list[tuple[str, Any]], ranged: bool, grouped: bool) -> list[Any]:
|
||
"""代表各段取值的值(不含空值)。"""
|
||
from decimal import Decimal
|
||
|
||
from sqlglot import exp
|
||
|
||
values = sorted({self._norm(v) for v in literals})
|
||
kind = literals[0][0] if literals else "n"
|
||
if not values:
|
||
return [Decimal(0)] if kind == "n" else ["x"]
|
||
if not ranged:
|
||
other = values[-1] + 1 if kind == "n" else values[-1] + "\x00~"
|
||
return values + [other]
|
||
out: list[Any] = []
|
||
if kind == "n":
|
||
integral = isinstance(_unparen(subject), exp.Count)
|
||
low = Decimal(1) if integral and grouped and isinstance(_unparen(subject).this, exp.Star) else (
|
||
Decimal(0) if integral else None)
|
||
candidates = [values[0] - 1] + [x for i, v in enumerate(values)
|
||
for x in ([v] + ([(v + values[i + 1]) / 2] if i + 1 < len(values) else []))]
|
||
candidates.append(values[-1] + 1)
|
||
if integral:
|
||
ints: set[Decimal] = set()
|
||
for i, v in enumerate(values):
|
||
ints |= {v.to_integral_value(rounding="ROUND_FLOOR") + d for d in (-1, 0, 1)}
|
||
ints |= {v.to_integral_value(rounding="ROUND_CEILING") + d for d in (-1, 0, 1)}
|
||
candidates = sorted(ints)
|
||
for c in candidates:
|
||
if low is not None and c < low:
|
||
continue
|
||
out.append(c)
|
||
return out or [low or Decimal(0)]
|
||
if values[0] != "":
|
||
out.append("")
|
||
for i, v in enumerate(values):
|
||
out.append(v)
|
||
between = v + "\x00"
|
||
if i + 1 >= len(values) or between < values[i + 1]:
|
||
out.append(between)
|
||
return out
|
||
|
||
def _truth(self, n: Any, v: Any) -> str:
|
||
"""n 在对象取 v(None 是空值)时是真(T)、假(F)还是空值(N)。"""
|
||
from sqlglot import exp
|
||
|
||
n = _unparen(n)
|
||
if isinstance(n, exp.And):
|
||
a, b = self._truth(n.left, v), self._truth(n.right, v)
|
||
return "F" if "F" in (a, b) else ("N" if "N" in (a, b) else "T")
|
||
if isinstance(n, exp.Or):
|
||
a, b = self._truth(n.left, v), self._truth(n.right, v)
|
||
return "T" if "T" in (a, b) else ("N" if "N" in (a, b) else "F")
|
||
if isinstance(n, exp.Not):
|
||
t = self._truth(n.this, v)
|
||
return {"T": "F", "F": "T"}.get(t, "N")
|
||
if isinstance(n, exp.Is):
|
||
hit = v is None
|
||
return "T" if hit != bool(n.args.get("negate")) else "F"
|
||
if v is None:
|
||
return "N"
|
||
name = type(n).__name__
|
||
if name in _COMPARE_OPS:
|
||
right = _literal_value(n.expression)
|
||
if right is not None:
|
||
lit, op = self._norm(right), name
|
||
else:
|
||
lit, op = self._norm(_literal_value(n.this)), _FLIP[name]
|
||
if type(v) is not type(lit):
|
||
return "N"
|
||
result = {"EQ": v == lit, "NEQ": v != lit, "GT": v > lit, "GTE": v >= lit, "LT": v < lit,
|
||
"LTE": v <= lit}[op]
|
||
return "T" if result else "F"
|
||
if isinstance(n, exp.In):
|
||
hit = any(not isinstance(_unparen(i), exp.Null) and self._norm(_literal_value(i)) == v
|
||
for i in n.expressions)
|
||
if hit:
|
||
return "T"
|
||
return "N" if any(isinstance(_unparen(i), exp.Null) for i in n.expressions) else "F"
|
||
if isinstance(n, exp.Between):
|
||
low, high = self._norm(_literal_value(n.args["low"])), self._norm(_literal_value(n.args["high"]))
|
||
return "T" if low <= v <= high else "F"
|
||
return "N"
|
||
|
||
def satisfiable(self, conds: list[Any], grouped: bool = False, never_null: bool = False) -> bool | None:
|
||
"""同一个对象只和常量比的几个条件(AND 连)有没有一个取值(含空值,never_null 时不含)让它们都成立;认不出
|
||
返回 None。"""
|
||
from sqlglot import exp
|
||
|
||
parsed = [self.subject(c) for c in conds]
|
||
if not parsed or any(p is None for p in parsed) or len({p[0] for p in parsed}) != 1:
|
||
return None
|
||
literals = [lit for p in parsed for lit in p[2]]
|
||
if len({k for k, _ in literals}) > 1:
|
||
return None
|
||
subject = parsed[0][1]
|
||
values: list[Any] = self._pieces(subject, literals, any(p[3] for p in parsed), grouped)
|
||
if not isinstance(_unparen(subject), exp.Count) and not never_null:
|
||
values.append(None)
|
||
return any(all(self._truth(c, v) == "T" for c in conds) for v in values)
|
||
|
||
def _atoms(self, cond: Any) -> list[tuple[str, Any, list[tuple[str, Any]], bool, str | None]] | None:
|
||
"""条件拆成「一个对象和常量比」的几项(AND / OR / NOT 连起来):[(对象的指纹, 对象, 常量, 有没有比大小, LIKE 的
|
||
模式)];有别的写法(两个对象比、子查询、函数套着条件……)返回 None。"""
|
||
from sqlglot import exp
|
||
|
||
out: list[tuple[str, Any, list[tuple[str, Any]], bool, str | None]] = []
|
||
|
||
def side(subj: Any, lits: list[Any], ranged: bool, like: str | None = None) -> bool:
|
||
if subj is None or _literal_value(subj) is not None or isinstance(_unparen(subj), exp.Null) or \
|
||
subj.find(exp.Query, exp.Subquery) is not None or _volatile(subj):
|
||
return False
|
||
out.append((_fingerprint(subj), subj, lits, ranged, like))
|
||
return True
|
||
|
||
def walk(n: Any) -> bool:
|
||
n = _unparen(n)
|
||
if isinstance(n, (exp.And, exp.Or)):
|
||
return walk(n.left) and walk(n.right)
|
||
if isinstance(n, exp.Not):
|
||
return walk(n.this)
|
||
name = type(n).__name__
|
||
if name in _COMPARE_OPS:
|
||
lv, rv = _literal_value(n.expression), _literal_value(n.this)
|
||
ranged = name not in ("EQ", "NEQ")
|
||
if lv is not None and rv is None:
|
||
return side(n.this, [lv], ranged)
|
||
if rv is not None and lv is None:
|
||
return side(n.expression, [rv], ranged)
|
||
return False
|
||
if isinstance(n, exp.In):
|
||
if n.args.get("query") or n.args.get("unnest") or n.args.get("field") or not n.expressions:
|
||
return False
|
||
lits = []
|
||
for item in n.expressions:
|
||
if isinstance(_unparen(item), exp.Null):
|
||
continue
|
||
value = _literal_value(item)
|
||
if value is None:
|
||
return False
|
||
lits.append(value)
|
||
return side(n.this, lits, False)
|
||
if isinstance(n, exp.Is):
|
||
return isinstance(_unparen(n.expression), exp.Null) and side(n.this, [], False)
|
||
if n.args.get("negate"):
|
||
return False
|
||
if isinstance(n, exp.Between):
|
||
low, high = _literal_value(n.args.get("low")), _literal_value(n.args.get("high"))
|
||
return not n.args.get("symmetric") and low is not None and high is not None and \
|
||
side(n.this, [low, high], True)
|
||
if isinstance(n, (exp.Like, exp.ILike)):
|
||
pattern = _unparen(n.expression)
|
||
return isinstance(pattern, exp.Literal) and pattern.is_string and \
|
||
side(n.this, [], False, str(pattern.this))
|
||
return False
|
||
|
||
return out if walk(cond) else None
|
||
|
||
def _analyze(self, conds: list[Any]) -> dict[str, dict[str, Any]] | None:
|
||
per: dict[str, dict[str, Any]] = {}
|
||
for c in conds:
|
||
atoms = self._atoms(c)
|
||
if atoms is None:
|
||
return None
|
||
for fp, subj, lits, ranged, like in atoms:
|
||
entry = per.setdefault(fp, {"node": subj, "lits": [], "ranged": False, "likes": set(), "plain": 0})
|
||
entry["lits"] += lits
|
||
entry["ranged"] |= ranged
|
||
if like is None:
|
||
entry["plain"] += 1
|
||
else:
|
||
entry["likes"].add(like)
|
||
for entry in per.values():
|
||
kinds = {k for k, _ in entry["lits"]}
|
||
# LIKE 只能单独出现在一个对象上(x LIKE 'a%' OR x NOT LIKE 'a%' 这类和别的条件交错的算不清)
|
||
if (entry["likes"] and (entry["plain"] or len(entry["likes"]) > 1)) or len(kinds) > 1:
|
||
return None
|
||
if entry["ranged"] and kinds == {"s"} and self.read != "sqlite" and \
|
||
not all(_DATE_ONLY.match(v) for _, v in entry["lits"]):
|
||
return None
|
||
return per
|
||
|
||
def subjects(self, conds: list[Any]) -> dict[str, Any] | None:
|
||
"""几个条件(每个都由「对象和常量比」用 AND / OR / NOT 连起来)里拿来比的对象 {指纹: 对象};认不出返回 None。"""
|
||
per = self._analyze(conds)
|
||
return None if per is None else {fp: entry["node"] for fp, entry in per.items()}
|
||
|
||
def always(self, conds: list[Any], never_null: Iterable[str] = ()) -> bool | None:
|
||
"""几个条件(AND 连)是不是恒为真:每个对象按自己用到的常量切成几段取值(见 _pieces,LIKE 的对象分「匹配上」
|
||
「匹配不上」两种,只由 % 组成的模式不为空就一定匹配上),加上空值(never_null 里的对象不取空值),各个对象的
|
||
取值逐个组合算一遍,都是真才是恒为真。组合太多、认不出时返回 None。"""
|
||
import itertools
|
||
|
||
from sqlglot import exp
|
||
|
||
per = self._analyze(conds)
|
||
if per is None:
|
||
return None
|
||
skip = set(never_null)
|
||
domains: list[list[Any]] = []
|
||
for fp, entry in per.items():
|
||
if entry["likes"]:
|
||
(pattern,) = entry["likes"]
|
||
values: list[Any] = [_LIKE_MATCH] if pattern and set(pattern) <= {"%"} else [_LIKE_MATCH, _LIKE_MISS]
|
||
else:
|
||
values = self._pieces(entry["node"], entry["lits"], entry["ranged"], False)
|
||
if fp not in skip and not isinstance(_unparen(entry["node"]), exp.Count):
|
||
values = values + [None]
|
||
domains.append(values)
|
||
if math.prod(len(d) for d in domains) > _MAX_COMBINATIONS:
|
||
return None
|
||
names = list(per)
|
||
for combo in itertools.product(*domains):
|
||
assign = dict(zip(names, combo))
|
||
if any(self._truth_at(c, assign) != "T" for c in conds):
|
||
return False
|
||
return True
|
||
|
||
def _truth_at(self, n: Any, assign: Mapping[str, Any]) -> str:
|
||
"""n 在各对象取 assign 里的值时是真(T)、假(F)还是空值(N)。"""
|
||
from sqlglot import exp
|
||
|
||
n = _unparen(n)
|
||
if isinstance(n, exp.And):
|
||
a, b = self._truth_at(n.left, assign), self._truth_at(n.right, assign)
|
||
return "F" if "F" in (a, b) else ("N" if "N" in (a, b) else "T")
|
||
if isinstance(n, exp.Or):
|
||
a, b = self._truth_at(n.left, assign), self._truth_at(n.right, assign)
|
||
return "T" if "T" in (a, b) else ("N" if "N" in (a, b) else "F")
|
||
if isinstance(n, exp.Not):
|
||
return {"T": "F", "F": "T"}.get(self._truth_at(n.this, assign), "N")
|
||
if isinstance(n, (exp.Like, exp.ILike)):
|
||
v = assign[_fingerprint(n.this)]
|
||
return "N" if v is None else ("T" if v == _LIKE_MATCH else "F")
|
||
if type(n).__name__ in _COMPARE_OPS:
|
||
subject = n.this if _literal_value(n.expression) is not None else n.expression
|
||
else:
|
||
subject = n.this
|
||
return self._truth(n, assign[_fingerprint(subject)])
|
||
|
||
def compare(self, conds_a: list[Any], conds_b: list[Any], grouped: bool = False) -> bool | None:
|
||
"""两组条件(各自 AND 连,可以一边为空 = 不筛)留下的取值是不是一样:一样 True,不一样 False,认不出 None。"""
|
||
parsed = [self.subject(c) for c in conds_a + conds_b]
|
||
if any(p is None for p in parsed) or len({p[0] for p in parsed}) != 1:
|
||
return None
|
||
subject = parsed[0][1]
|
||
literals = [lit for p in parsed for lit in p[2]]
|
||
if len({k for k, _ in literals}) > 1:
|
||
return None
|
||
ranged = any(p[3] for p in parsed)
|
||
from sqlglot import exp
|
||
|
||
values = self._pieces(subject, literals, ranged, grouped)
|
||
if not isinstance(_unparen(subject), exp.Count):
|
||
values.append(None)
|
||
for v in values:
|
||
ta = all(self._truth(c, v) == "T" for c in conds_a)
|
||
tb = all(self._truth(c, v) == "T" for c in conds_b)
|
||
if ta != tb:
|
||
return False
|
||
return True
|
||
|
||
|
||
# ---- 一边用到了哪些列(判断「另一边没有」能不能说)
|
||
|
||
class _Usage:
|
||
"""一条语句里每一列(按定义找到实表列,认不出来源的只记列名)用在了哪儿:
|
||
|
||
- cond:WHERE、ON、HAVING 里,条件里的子查询(IN、EXISTS、标量)选出来的列;
|
||
- case:CASE / IF / FILTER 里(SUM(CASE WHEN status = 'paid' THEN amount END) 等于先按 status 筛再求和),以及聚合
|
||
的参数里套在别的运算、函数里的列(SUM(status = 'paid')、COUNT(status = 'paid' OR NULL)、COUNT(NULLIF(status,
|
||
'void'))、SUM(amount * (status = 'paid'))、SUM(amount * (1 - is_deleted)):乘上 0、变成空值的行就是被筛掉了,
|
||
和写 CASE 是一回事。认不出是不是在筛,一律当成在筛,只会少说);
|
||
- agg:光一列做聚合函数、窗口函数的参数(COUNT(x)、SUM(x) 跳过空值,等于先筛掉 x 为空的行)。
|
||
|
||
光选出来、分组、排序的列不记。说「另一边没有这个条件」「另一边没有按时间筛选」之前,先看另一边在这一段能影响到的
|
||
地方(整条语句,除了合并里别的支)用没用到这些列。
|
||
|
||
temporal:这条语句哪儿可能按时间筛了——在条件里(cond / case)用到可能是时间的列(_timeish_name:paid_at,也包括
|
||
date_key、stat_month 这类认不准的);WHERE、HAVING、ON 里有日期函数、像日期的常量;任何地方(聚合的参数、CASE、
|
||
FILTER 里也算)有可能是时间窗口的比较(_window_like:20260901 这类整数、'2026-09%' 这类模式);对同一个对象有
|
||
上下界(BETWEEN、x >= … 和 x < … 都有,见 _bounded_both_ways)。这时不说它「没有按时间筛选」:第 5 轮复验抓到
|
||
SUM(paid_at >= '2026-09-01' AND paid_at < '2026-10-01')、date_key BETWEEN 20260901 AND 20260930 都被说成了没有按时间
|
||
筛选。"""
|
||
|
||
def __init__(self, tree: Any) -> None:
|
||
from sqlglot import exp
|
||
|
||
self.nodes: dict[tuple[str | None, str], dict[int, str]] = {}
|
||
self.temporal = False
|
||
try:
|
||
self.resolver: _Resolver | None = _Resolver(tree)
|
||
except _Unsupported:
|
||
self.resolver = None
|
||
for col in tree.find_all(exp.Column):
|
||
if col.is_star or not col.name:
|
||
continue
|
||
kind = self._kind(col)
|
||
if kind is None:
|
||
continue
|
||
bases = self._bases(col, 0)
|
||
if isinstance(col.this, exp.Identifier) and col.this.meta.get(_UNSURE):
|
||
bases.append((None, col.name))
|
||
for table, column in bases:
|
||
self.nodes.setdefault(_usage_key(table, column), {})[id(col)] = kind
|
||
if kind != "agg" and (_timeish_name(column) or _timeish_name(col.name)):
|
||
self.temporal = True
|
||
for node in tree.find_all(exp.Where, exp.Having, exp.Join):
|
||
target = node.args.get("on") if isinstance(node, exp.Join) else node.this
|
||
if target is not None and _datey(target):
|
||
self.temporal = True
|
||
if _window_like(tree) or _bounded_both_ways(tree):
|
||
self.temporal = True
|
||
|
||
@staticmethod
|
||
def _owner(node: Any) -> tuple[Any, str | None, bool, bool, bool]:
|
||
"""node 属于哪一段 SELECT、在它的哪个子句里、路上有没有 CASE / IF / FILTER、有没有聚合 / 窗口函数,以及在聚合的
|
||
参数里是不是套在别的运算、函数里(不是光一列:括号、DISTINCT、CAST 不算)。"""
|
||
from sqlglot import exp
|
||
|
||
case = agg = computed = False
|
||
child, parent = node, node.parent
|
||
while parent is not None:
|
||
if isinstance(parent, (exp.Case, exp.If, exp.Filter)):
|
||
case = True
|
||
if isinstance(parent, (exp.AggFunc, exp.Window)):
|
||
agg = True
|
||
elif not agg and not isinstance(parent, (exp.Paren, exp.Distinct, exp.Cast, exp.TryCast)):
|
||
# 只换类型(CAST)还是那一列;别的运算、函数(比较、NULLIF、乘上标记……)可能就是在筛
|
||
computed = True
|
||
if isinstance(parent, exp.Select):
|
||
return parent, child.arg_key, case, agg, computed
|
||
child, parent = parent, parent.parent
|
||
return None, None, case, agg, computed
|
||
|
||
def _kind(self, col: Any) -> str | None:
|
||
select, key, case, agg, computed = self._owner(col)
|
||
if select is None:
|
||
return None
|
||
if key == "expressions":
|
||
if _scalar_context(select) or _condition_query(select):
|
||
return "cond"
|
||
if case or (agg and computed):
|
||
return "case"
|
||
return "agg" if agg else None
|
||
if key in ("group", "order", "distinct"):
|
||
return "case" if case else None
|
||
return "cond"
|
||
|
||
def _bases(self, col: Any, depth: int) -> list[tuple[str | None, str]]:
|
||
if depth > 16 or self.resolver is None:
|
||
return [(None, col.name)]
|
||
item = self.resolver.source_of.get(id(col))
|
||
if item is None:
|
||
select = self._owner(col)[0]
|
||
items = _items_of(select) if select is not None else []
|
||
if col.table or len(items) != 1:
|
||
return [(None, col.name)]
|
||
item = items[0]
|
||
return self._from_item(item, col.name, depth)
|
||
|
||
def _from_item(self, item: Any, name: str, depth: int) -> list[tuple[str | None, str]]:
|
||
from sqlglot import exp
|
||
|
||
body = self.resolver.body(item) if self.resolver is not None else None
|
||
if body is None:
|
||
return [(item.name, name)] if isinstance(item, exp.Table) else [(None, name)]
|
||
leaves = _leaves(body)
|
||
if not leaves or not isinstance(leaves[0], exp.Select) or depth > 16:
|
||
return [(None, name)]
|
||
names = [p.alias_or_name for p in leaves[0].expressions]
|
||
if name not in names:
|
||
sources = _items_of(leaves[0])
|
||
if len(leaves) == 1 and any(isinstance(p, exp.Star) for p in leaves[0].expressions) and len(sources) == 1:
|
||
return self._from_item(sources[0], name, depth + 1)
|
||
return [(None, name)]
|
||
index = names.index(name)
|
||
out: list[tuple[str | None, str]] = []
|
||
for leaf in leaves:
|
||
if not isinstance(leaf, exp.Select) or index >= len(leaf.expressions):
|
||
return [(None, name)]
|
||
expr = leaf.expressions[index].unalias()
|
||
for c in ([expr] if isinstance(expr, exp.Column) else list(expr.find_all(exp.Column))):
|
||
if not c.is_star:
|
||
out += self._bases(c, depth + 1)
|
||
return out or [(None, name)]
|
||
|
||
def touches(self, refs: Iterable[tuple[str, str]], skip: Iterable[Any] = (),
|
||
kinds: tuple[str, ...] = ("cond", "case"), scope: Any = None) -> bool:
|
||
"""这些实表列用在了 kinds 里(认不出来源的同名列也算);skip 里的条件中的那几处不算,scope 所在合并的别的支
|
||
(不碰 scope 的行)不算。"""
|
||
from sqlglot import exp
|
||
|
||
skipped = {id(c) for cond in skip for c in cond.find_all(exp.Column)}
|
||
node = scope
|
||
while node is not None and node.parent is not None:
|
||
parent = node.parent
|
||
if isinstance(parent, exp.SetOperation):
|
||
for branch in (parent.this, parent.expression):
|
||
if branch is not node:
|
||
skipped |= {id(c) for c in branch.find_all(exp.Column)}
|
||
node = parent
|
||
for table, column in refs:
|
||
for key in (_usage_key(table, column), _usage_key(None, column)):
|
||
if any(i not in skipped and k in kinds for i, k in self.nodes.get(key, {}).items()):
|
||
return True
|
||
return False
|
||
|
||
def touches_name(self, columns: Iterable[str], kinds: tuple[str, ...] = ("cond", "case")) -> bool:
|
||
"""不管是哪张表的,叫这几个名字的列用在了 kinds 里(两边换了一张表时用:同一个意思的列在两边各属各的表)。"""
|
||
names = {_usage_key(None, c)[1] for c in columns}
|
||
return any(column in names and any(k in kinds for k in found.values())
|
||
for (_, column), found in self.nodes.items())
|
||
|
||
|
||
def _usage_key(table: str | None, column: str) -> tuple[str | None, str]:
|
||
"""_Usage 记一列用的键:表名、列名都不分(ASCII)大小写。SQLite、MySQL 的列名本来就不分;PostgreSQL 带引号的分,
|
||
这里一律当成同一列,只会多算一处「另一边用到了」,少说一句,不会说错。"""
|
||
return (table.translate(_ASCII_LOWER) if table else None), column.translate(_ASCII_LOWER)
|
||
|
||
|
||
def _kinds_for(conds: list[Any]) -> tuple[str, ...]:
|
||
"""说某个条件只在一边有之前,要看另一边哪些地方用到了这些列:条件里有判空(IS NULL / IS NOT NULL)的连聚合的
|
||
参数也看(COUNT(x) 本来就跳过 x 为空的行)。"""
|
||
from sqlglot import exp
|
||
|
||
if any(isinstance(n, exp.Is) for c in conds for n in c.walk()):
|
||
return "cond", "case", "agg"
|
||
return "cond", "case"
|
||
|
||
|
||
def _condition_query(select: Any) -> bool:
|
||
"""这段查询是条件里的子查询(IN、EXISTS、ANY / ALL):选出来的列参与比较。"""
|
||
from sqlglot import exp
|
||
|
||
node = select
|
||
while isinstance(node.parent, (exp.Subquery, exp.SetOperation)):
|
||
node = node.parent
|
||
parent = node.parent
|
||
return isinstance(parent, (exp.Exists, exp.Any, exp.All)) or (isinstance(parent, exp.In) and
|
||
node.arg_key == "query")
|
||
|
||
|
||
# ---- 一段 SELECT
|
||
|
||
@dataclass
|
||
class _Rel:
|
||
"""一段里的一个来源:alias 是规范别名,name 是实表名(CTE 是规范名,子查询是 sub),kind 是 table / cte / derived,
|
||
shown 是给人看的名字(实表是表名,CTE、子查询是原来起的名字),rejected 表示写的是 LEFT JOIN、被 WHERE 变成了
|
||
内连接。"""
|
||
|
||
alias: str
|
||
name: str
|
||
kind: str
|
||
shown: str
|
||
node: Any
|
||
rejected: str | None = None
|
||
|
||
@property
|
||
def table(self) -> bool:
|
||
return self.kind == "table"
|
||
|
||
|
||
def _rel_of(item: Any, ctes: Mapping[str, str]) -> _Rel:
|
||
from sqlglot import exp
|
||
|
||
alias = item.alias_or_name
|
||
if isinstance(item, exp.Subquery):
|
||
return _Rel(alias, "sub", "derived", item.this.meta.get(_ORIGINAL) or alias, item, item.meta.get(_REJECTED))
|
||
if item.name in ctes:
|
||
return _Rel(alias, item.name, "cte", ctes[item.name], item, item.meta.get(_REJECTED))
|
||
return _Rel(alias, item.name, "table", item.name, item, item.meta.get(_REJECTED))
|
||
|
||
|
||
@dataclass
|
||
class _Block:
|
||
"""一段 SELECT 拆开来比:开头那组逗号连接的来源(rels)、后面的连接(joins:LEFT JOIN、没写成逗号连接的内连接,
|
||
按先后)、WHERE、GROUP BY、HAVING 的各项,选出来的列,排序和取行。"""
|
||
|
||
select: Any
|
||
rels: list[_Rel]
|
||
joins: list[Any]
|
||
where: list[Any]
|
||
group: list[Any]
|
||
having: list[Any]
|
||
exprs: list[Any]
|
||
order: list[Any]
|
||
limit: str | None
|
||
offset: str | None
|
||
distinct: bool
|
||
distinct_on: bool
|
||
windowed: bool
|
||
ctes: dict[str, str]
|
||
deduped: bool = False
|
||
|
||
@classmethod
|
||
def of(cls, select: Any, read: str | None, ctes: Mapping[str, str], root: bool = False) -> "_Block | None":
|
||
from sqlglot import exp
|
||
|
||
if not isinstance(select, exp.Select) or (select.args.get("with_") is not None and not root):
|
||
return None
|
||
items = _items_of(select)
|
||
if not items or any(not isinstance(i, (exp.Table, exp.Subquery)) for i in items):
|
||
return None
|
||
joins = select.args.get("joins") or []
|
||
n = 1
|
||
for join in joins:
|
||
if _comma(join):
|
||
n += 1
|
||
else:
|
||
break
|
||
where, group, having = select.args.get("where"), select.args.get("group"), select.args.get("having")
|
||
order, limit, offset = select.args.get("order"), select.args.get("limit"), select.args.get("offset")
|
||
distinct = select.args.get("distinct")
|
||
|
||
def number(node: Any) -> str | None:
|
||
return node.expression.sql(dialect=read) if node is not None and node.expression is not None else None
|
||
|
||
return cls(select, [_rel_of(i, ctes) for i in items[:n]], joins[n - 1:],
|
||
_conjuncts(where.this) if where is not None else [],
|
||
list(group.expressions) if group is not None else [],
|
||
_conjuncts(having.this) if having is not None else [],
|
||
list(select.expressions), list(order.expressions) if order is not None else [],
|
||
number(limit), number(offset), distinct is not None,
|
||
bool(distinct is not None and distinct.args.get("on")),
|
||
any(isinstance(n, exp.Window) for n in _block_nodes(select)), dict(ctes),
|
||
bool(group is not None and group.meta.get(_FROM_DISTINCT)))
|
||
|
||
def all_rels(self) -> list[_Rel]:
|
||
return self.rels + [_rel_of(j.this, self.ctes) for j in self.joins]
|
||
|
||
def rel(self, alias: str) -> _Rel | None:
|
||
for r in self.all_rels():
|
||
if r.alias == alias:
|
||
return r
|
||
return None
|
||
|
||
def names(self) -> list[str]:
|
||
return [r.name for r in self.all_rels()]
|
||
|
||
|
||
# ---- 具体说法只在简单形状上说
|
||
|
||
#: 简单形状的一段 SELECT 能有的子句:别的(QUALIFY、WINDOW、优化器提示、加锁……)一律不算简单
|
||
_SIMPLE_ARGS = frozenset({"expressions", "from_", "joins", "where", "group", "having", "order", "limit", "offset",
|
||
"distinct"})
|
||
#: 不分组、去重之后结果只看取值的集合、不看有几行的聚合(DISTINCT 写不写一样)
|
||
_DISTINCT_BLIND = frozenset({"min", "max"})
|
||
|
||
|
||
def _simple_block(tree: Any, read: str | None) -> "_Block | None":
|
||
"""规范形式是「简单形状」时拆成 _Block,不是返回 None。简单形状:整条语句就是一段 SELECT——没有 CTE、子查询(FROM
|
||
里的、条件里的、SELECT 列表里的都不行)、UNION 这类合并、窗口函数、DISTINCT ON;FROM 里只有实表,连接只有内连接、
|
||
CROSS JOIN、逗号和 LEFT JOIN(RIGHT / FULL JOIN、NATURAL 都不算);LIMIT / OFFSET 是写死的数,LIMIT 至少是 1
|
||
(LIMIT 0 两边都没有行,什么不同都看不出来);没有认不准是哪一列的名字(见 _UNSURE)。
|
||
|
||
为什么只在简单形状上说:复验在派生表、合并、窗口函数、子查询上一再抓到说出不存在的差异——外层是 COUNT(DISTINCT)、
|
||
MAX、IN 时子查询里的 UNION 对 UNION ALL、去重、取几行都不起作用,CTE 只在 IN 里用到时里面去不去重也不起作用;
|
||
逐个情况补规则补不完。这些一律只给兜底说明。"""
|
||
from sqlglot import exp
|
||
|
||
if not isinstance(tree, exp.Select):
|
||
return None
|
||
if any(value not in (None, False, []) and key not in _SIMPLE_ARGS for key, value in tree.args.items()):
|
||
return None
|
||
for node in tree.walk():
|
||
if node is not tree and isinstance(node, (exp.Query, exp.Subquery, exp.Window)):
|
||
return None
|
||
if isinstance(node, exp.Identifier) and node.meta.get(_UNSURE):
|
||
return None
|
||
# SELECT 列表、HAVING、ORDER BY 里 sqlglot 不认识的函数可能是聚合(SQLite 的 TOTAL):认不清这一段聚不聚合,去重、
|
||
# LEFT JOIN 看不看得见没关联上的行都说不准
|
||
for clause in (*tree.expressions, tree.args.get("having"), tree.args.get("order")):
|
||
if clause is not None and clause.find(exp.Anonymous) is not None:
|
||
return None
|
||
distinct = tree.args.get("distinct")
|
||
if distinct is not None and distinct.args.get("on"):
|
||
return None
|
||
first = tree.args.get("from_")
|
||
if first is None or not _base_table(first.this):
|
||
return None
|
||
for join in tree.args.get("joins") or []:
|
||
side, kind = str(join.side or "").upper(), str(join.kind or "").upper()
|
||
if not _base_table(join.this) or not _plain_join(join) or side not in ("", "LEFT") or \
|
||
kind not in ("", "INNER", "CROSS", "OUTER") or (side == "LEFT" and join.args.get("on") is None):
|
||
return None
|
||
for key, least in (("limit", 1), ("offset", 0)):
|
||
node = tree.args.get(key)
|
||
if node is not None:
|
||
value = _int_value(node.expression) if node.expression is not None else None
|
||
if value is None or value < least:
|
||
return None
|
||
return _Block.of(tree, read, {}, root=True)
|
||
|
||
|
||
def _base_table(item: Any) -> bool:
|
||
from sqlglot import exp
|
||
|
||
return isinstance(item, exp.Table) and isinstance(item.this, exp.Identifier)
|
||
|
||
|
||
def _alias_named(root: Any) -> bool:
|
||
"""原文里有 WHERE、ON、GROUP BY、HAVING 用了和同一段的列别名同名、不带表名的名字:SQLite 的 WHERE、GROUP BY 都认
|
||
列别名(WHERE d >= '2026-09-01' 里的 d 是 DATE(paid_at) AS d),不认列的那一级规范化把它当成表的一列,说出来的
|
||
「另一边没有这个条件」「按 d 筛时间」就是错的。"""
|
||
from sqlglot import exp
|
||
|
||
for select in root.find_all(exp.Select):
|
||
aliases = {p.alias.translate(_ASCII_LOWER) for p in select.expressions if isinstance(p, exp.Alias) and p.alias}
|
||
if not aliases:
|
||
continue
|
||
clauses = [select.args.get(k) for k in ("where", "group", "having")] + \
|
||
[j.args.get("on") for j in select.args.get("joins") or []]
|
||
for clause in clauses:
|
||
if clause is None:
|
||
continue
|
||
for col in clause.find_all(exp.Column):
|
||
if not col.table and not _nested(col, clause) and col.name.translate(_ASCII_LOWER) in aliases:
|
||
return True
|
||
return False
|
||
|
||
|
||
class _Explainer:
|
||
"""在两边的规范形式上找有把握、又确实影响结果的不同,写成具体的中文句子,同时记下是不是所有不同都说到了。
|
||
|
||
只在两边都是简单形状时说(见 _simple_block),也只说下面四类;别的不同(同一列上的时间范围、同一列上换了筛选的值、
|
||
关联条件、分组、HAVING、排序、取几行、选出来的列、聚合的参数……)一律算没说到,交给兜底说明:
|
||
|
||
- time:底层表上按哪一列筛时间不同(两边各在一列上按时间筛,列不同,另一边哪儿都没用这一列筛选,时间列不在 LEFT
|
||
JOIN 的可空一侧);一边按时间筛了、另一边哪儿都没有可能是时间窗口的条件的(见 _Usage.temporal),算多出来的
|
||
筛选,也说成 time。时间条件见 timed(比较、IN、BETWEEN,时间列 LIKE '2026-09%')。
|
||
- filter:一边多一个只涉及一张底层表的列、和常量比的筛选条件:另一边整条语句(WHERE、ON、HAVING、CASE / FILTER 里,
|
||
聚合的参数里套在别的运算、函数里的,判空、或者条件只筛掉空值时连光一列的聚合参数)对这些列没有任何条件;这一边
|
||
对这些列的条件全是另一边没有的(不是被另一边也有的条件蕴含的,x > 5 之外多写的 x IS NOT NULL 不说);比的对象就是
|
||
那一列(LOWER(x) <> 'PAID' 这类认不出取值范围、可能恒为真);按取值逐段算过不是恒为真的(见 _Range.always,一定
|
||
不为空的列不算空值,见 _never_null);不用到 LEFT JOIN 可空一侧的列(反连接,见 _on_nullable_side);不是认不准
|
||
的时间窗口(见 _window_like)。
|
||
- aggregate / group_by:选出来的列只差在顶层的聚合函数(同一列做参数,SUM 对 AVG;COUNT(*) 对 SUM(x))或者去重上
|
||
(COUNT / SUM / AVG 去不去重,COUNT(*) 对 COUNT(DISTINCT x),SELECT DISTINCT 对不去重),分组一样。参数在组里
|
||
定死的不说(被 WHERE / ON 定成常量、等于分组键、本身是常量、它那张表的键在组里定死了,见 _fixed_in_group);去重
|
||
不起作用的不说:MIN / MAX;不分组的聚合只出一行;已经按选出来的列分组;只取一行;表结构里的键说明每行本来就不一样
|
||
(见 _maybe_unique)。每组只有一行时 SUM、AVG、MIN、MAX 都是那一行的值,也不说。
|
||
- join / tables:关联的表不同(换了一张表、一边多关联了一张表)或者关联类型不同(同一个条件,LEFT JOIN 对内连接)。
|
||
LEFT JOIN 对内连接要可空一侧在别处(WHERE、HAVING、后面的连接)都没用到、结果看得见没关联上的那些行(不是只取可空
|
||
一侧跳过空值的聚合、FILTER);一边多出来的 LEFT JOIN 要那张表在 ON 之外没用到、结果怕重复的行;表结构说关联起来
|
||
每行正好一行(外键、键)的不说。原文写的是 RIGHT JOIN、被 WHERE 或者聚合改成内连接的,说法和原文对不上,不说;
|
||
另一边原文也 LEFT JOIN 了这张表、被规范化去掉了的(见 _drop_unused_left_joins),不说「另一边没有」。
|
||
|
||
两边的 WHERE、HAVING、ON 里有可能恒为假的条件(常量条件、同一个对象上取不到值的几个条件)时什么都不说:结果都是空的,
|
||
别处的不同看不出来。结果看不见行的(只出一行、SELECT 列表里又没有一项本身就是聚合,见 _rows_visible)不说筛选、
|
||
关联上的不同。说的都是「这一段的这一项两边各写的是什么」;两边给人看的写法一样、和已经说过的一句正好反过来的不说。"""
|
||
|
||
def __init__(self, read: str | None, labels: tuple[str, str], names: Mapping[str, str] | None,
|
||
info: _Schema) -> None:
|
||
self.read = read
|
||
self.la, self.lb = labels
|
||
self.label = _labeler(names)
|
||
self.info = info
|
||
self.items: list[FacetDiff] = []
|
||
self.range = _Range(read)
|
||
self.usage_a: _Usage | None = None
|
||
self.usage_b: _Usage | None = None
|
||
#: 换了一张表(见 _swapped)
|
||
self.swapped = False
|
||
#: 两边的结果看得见各自有哪些行(见 _rows_visible):看不见时筛选、关联上的不同都说不准
|
||
self.visible = True
|
||
#: 有一句话没说出口(两边给人看的写法一样,或者和前面的话矛盾):这一处不同就没说到
|
||
self.withheld = False
|
||
|
||
def sql(self, node: Any) -> str:
|
||
"""给人看的写法:按方言生成;生成的写法读回来不是同一棵树时(SQLite 把 CAST(x AS DATE) 生成 DATE(x),两者
|
||
在 SQLite 里结果不同)改用 sqlglot 的通用写法,免得给人看的和实际的不是一回事。"""
|
||
import sqlglot
|
||
|
||
text = node.sql(dialect=self.read, comments=False)
|
||
try:
|
||
if _fingerprint(sqlglot.parse_one(text, read=self.read)) == _fingerprint(node):
|
||
return text
|
||
except Exception: # noqa: BLE001 - 片段单独读不回来:用通用写法
|
||
pass
|
||
return node.sql(comments=False)
|
||
|
||
@staticmethod
|
||
def fp(node: Any) -> str:
|
||
return _fingerprint(node)
|
||
|
||
# ---- 入口:整条语句
|
||
|
||
def statement(self, form_a: _Form, form_b: _Form, roots: tuple[Any, Any] = (None, None)) -> bool:
|
||
"""比两条语句的规范形式,返回是不是全都说到了。roots 是两边解析出来的原样的树(不认列的那一级要拿它看原文里
|
||
有没有用列别名的条件)。"""
|
||
A, B = _simple_block(form_a.tree, self.read), _simple_block(form_b.tree, self.read)
|
||
if A is None or B is None:
|
||
return False
|
||
if not (form_a.qualified and form_b.qualified) and any(r is None or _alias_named(r) for r in roots):
|
||
return False
|
||
if self._dead(A) or self._dead(B):
|
||
return False
|
||
self.visible = self._rows_visible(A) and self._rows_visible(B)
|
||
self.usage_a, self.usage_b = _Usage(form_a.tree), _Usage(form_b.tree)
|
||
matched = self.relations(A, B)
|
||
if matched is None:
|
||
return False
|
||
covered, mapping, skip_a, skip_b = matched
|
||
covered &= self.conditions(A, B, mapping, skip_a, skip_b)
|
||
covered &= self.outputs(A, B, mapping)
|
||
covered &= self.rest(A, B, mapping)
|
||
return covered and not self.withheld
|
||
|
||
# ---- 说法的拼法
|
||
|
||
def add(self, aspect: str, a: Any, b: Any, text: str, detail: dict[str, Any] | None = None) -> None:
|
||
"""记一句话。两边给人看的写法一样(「A 算的是 1,B 算的是 1」)、和已经说过的一句正好反过来(A 说 X、B 说 Y
|
||
对上 A 说 Y、B 说 X)的不说:读起来自相矛盾,这一处算没说到。"""
|
||
if isinstance(a, str) and isinstance(b, str) and a == b:
|
||
self.withheld = True
|
||
return
|
||
for old in self.items:
|
||
if old.aspect == aspect and old.a == b and old.b == a and (a is not None or b is not None):
|
||
self.items.remove(old)
|
||
self.withheld = True
|
||
return
|
||
if old.text == text:
|
||
return
|
||
self.items.append(FacetDiff(aspect, a, b, text, detail))
|
||
|
||
def base_columns(self, node: Any, block: _Block) -> list[tuple[_Rel, Any]] | None:
|
||
"""node 只涉及这一段的实表列(同名的表在这一段只出现一次)时,返回各列和它的来源;否则 None。"""
|
||
from sqlglot import exp
|
||
|
||
names = block.names()
|
||
out = []
|
||
for col in node.find_all(exp.Column):
|
||
if col.is_star:
|
||
return None
|
||
rel = block.rel(col.table) if col.table else (block.rels[0] if len(block.all_rels()) == 1 else None)
|
||
if rel is None or not rel.table or names.count(rel.name) != 1:
|
||
return None
|
||
out.append((rel, col))
|
||
return out
|
||
|
||
def refs(self, node: Any, block: _Block) -> set[tuple[str, str]] | None:
|
||
cols = self.base_columns(node, block)
|
||
return None if cols is None else {(r.name, c.name) for r, c in cols}
|
||
|
||
def show(self, node: Any, block: _Block, *, full: bool = False) -> str:
|
||
"""给人看的写法:只有一个来源时列不写表名,否则写实表名(规范别名换回表名);选出来的列去掉规范化起的别名。
|
||
full 时一律写表名。"""
|
||
from sqlglot import exp
|
||
|
||
copy = node.unalias().copy() if isinstance(node, exp.Alias) else node.copy()
|
||
single = len(block.all_rels()) == 1 and not full
|
||
for col in copy.find_all(exp.Column):
|
||
written = col.meta.get(_WRITTEN_COLUMN)
|
||
if written is not None:
|
||
# 规范化按关联条件换成了同一组里的另一列(见 _substitute_equal_columns):给人看原文写的那一列
|
||
col.set("this", exp.to_identifier(written[1]))
|
||
col.set("table", None if single else exp.to_identifier(written[0]))
|
||
continue
|
||
rel = block.rel(col.table) if col.table else None
|
||
if rel is not None:
|
||
col.set("table", None if single else exp.to_identifier(rel.shown))
|
||
return self.sql(copy)
|
||
|
||
def show_all(self, conds: list[Any], block: _Block) -> str:
|
||
"""几个条件用 AND 连起来给人看:OR 套上括号(按 sqlglot 的结合规则生成)。"""
|
||
from sqlglot import exp
|
||
|
||
if len(conds) == 1:
|
||
return self.show(conds[0], block)
|
||
return self.show(exp.and_(*[c.copy() for c in conds], copy=False), block)
|
||
|
||
def named(self, table: str, column: str, *, qualified: bool = False) -> str:
|
||
"""列带中文名(有的话):「支付时间 paid_at」。qualified 时写成「表.列」:两边的时间列在不同的表上(A 按
|
||
orders.paid_at、B 按 users.created_at),或者说「另一边没有按时间筛选」的这一边关联了好几张表时,光写 created_at
|
||
分不清是订单的下单时间还是用户的注册时间。"""
|
||
label = self.label((table, column))
|
||
shown = f"{table}.{column}" if qualified else column
|
||
return _say(label, shown) if label else shown
|
||
|
||
@staticmethod
|
||
def what(rel: _Rel) -> str:
|
||
return _say(rel.shown, "表")
|
||
|
||
@staticmethod
|
||
def mapped(node: Any, mapping: Mapping[str, str]) -> Any:
|
||
"""B 的一个节点按来源的对应关系(B 的别名 → A 的别名)改名后的副本:只用来和 A 比是不是同一个写法。"""
|
||
from sqlglot import exp
|
||
|
||
if not mapping:
|
||
return node
|
||
copy = node.copy()
|
||
for col in copy.find_all(exp.Column):
|
||
if col.table in mapping:
|
||
col.set("table", exp.to_identifier(mapping[col.table]))
|
||
return copy
|
||
|
||
def mfp(self, node: Any, mapping: Mapping[str, str]) -> str:
|
||
return self.fp(self.mapped(node, mapping))
|
||
|
||
# ---- 先确认看得出不同:条件不会恒为假
|
||
|
||
def _dead(self, block: _Block) -> bool:
|
||
"""这一段的条件可能恒为假(结果一定是空的,或者只剩一行全是空值的聚合),两边就算别处不同也看不出来:常量条件
|
||
(没有列、没有聚合的,1 = 0);同一个对象只和常量比的几个条件按取值逐段算,哪个值都不成立的(x > 5 AND x < 3、
|
||
分组后 COUNT(*) < 0)。几列交错的矛盾(x > y AND y > x)认不出来,见模块说明。"""
|
||
from sqlglot import exp
|
||
|
||
groups = [(block.where, False), (block.having, bool(block.group))] + \
|
||
[(_conjuncts(j.args["on"]), False) for j in block.joins if j.args.get("on") is not None]
|
||
never_null = self._never_null(block)
|
||
for conds, grouped in groups:
|
||
by_subject: dict[str, list[Any]] = {}
|
||
for c in conds:
|
||
if c.find(exp.Column) is None and c.find(exp.AggFunc) is None:
|
||
return True
|
||
found = self.range.subject(c)
|
||
if found is not None:
|
||
by_subject.setdefault(found[0], []).append(c)
|
||
if any(self.range.satisfiable(group, grouped, fp in never_null) is False
|
||
for fp, group in by_subject.items()):
|
||
return True
|
||
return False
|
||
|
||
def _nullable(self, block: _Block) -> set[str]:
|
||
"""外连接可空一侧、而且真的可能整行为空的来源:LEFT JOIN 进来的,后面没有引用它的内连接、WHERE 里也没有把它
|
||
整行为空的行筛掉的条件(见 _rejects)——不然没关联上的行反正留不下来,它的列就和内连接时一样。"""
|
||
from sqlglot import exp
|
||
|
||
out = set()
|
||
for i, join in enumerate(block.joins):
|
||
if str(join.side or "").upper() != "LEFT":
|
||
continue
|
||
alias = join.this.alias_or_name
|
||
later = any(not str(k.side or "") and k.args.get("on") is not None and
|
||
alias in {c.table for c in k.args["on"].find_all(exp.Column)} for k in block.joins[i + 1:])
|
||
if not later and not any(_rejects(c, alias, self.read) for c in block.where):
|
||
out.add(alias)
|
||
return out
|
||
|
||
def _never_null(self, block: _Block) -> set[str]:
|
||
"""这一段里一定不为空的列(的指纹):表结构里确定不为空(NOT NULL、主键,见 _not_null_column),又不在真的可能
|
||
整行为空的外连接一侧(见 _nullable)。只拿来少说:恒为真的条件、取不到值的条件多认出几个。"""
|
||
from sqlglot import exp
|
||
|
||
nullable = self._nullable(block)
|
||
out = set()
|
||
for col in block.select.find_all(exp.Column):
|
||
cols = self.base_columns(col, block) if col.name and not col.is_star else None
|
||
if cols and cols[0][0].alias not in nullable and \
|
||
_not_null_column(self.info, cols[0][0].name, col.name, self.read):
|
||
out.add(_fingerprint(col))
|
||
return out
|
||
|
||
@staticmethod
|
||
def _top_aggregates(block: _Block) -> list[Any]:
|
||
"""SELECT 列表里本身就是一个聚合的项(外面最多套着 ROUND、CAST、给默认值的 COALESCE,见 _unwrap_value),不带
|
||
FILTER、不是窗口函数。COUNT(*) > -1、SUM(x) / COUNT(*) 这类拿聚合再算的不算:前者恒为真,看不出有几行。"""
|
||
from sqlglot import exp
|
||
|
||
out = []
|
||
for e in block.exprs:
|
||
node = _unwrap_value(e.unalias())
|
||
if isinstance(node, exp.AggFunc) and not isinstance(node.parent, (exp.Filter, exp.Window)):
|
||
out.append(node)
|
||
return out
|
||
|
||
def _rows_visible(self, block: _Block) -> bool:
|
||
"""结果看得见这一段筛下来、关联出来的是哪些行:不聚合(行就在结果里)、分组(组在不在看得见),或者不分组的
|
||
聚合里有一项本身就是聚合(见 _top_aggregates)。SELECT COUNT(*) > -1 FROM t 这类只出一行、值又和行无关的,
|
||
多一个条件、少一张表都看不出来。"""
|
||
if not _aggregating(block.select) or block.group:
|
||
return True
|
||
return bool(self._top_aggregates(block))
|
||
|
||
# ---- 来源
|
||
|
||
@staticmethod
|
||
def _shape(block: _Block) -> tuple[dict[str, tuple[str, str]], list[tuple[str, str, str, str]]]:
|
||
"""这一段的来源:开头那组内连接 {别名: (表名, 种类)} 和后面的连接 [(别名, 表名, 种类, ON 的写法)],种类是 inner /
|
||
left。"""
|
||
prefix = {r.alias: (r.name.translate(_ASCII_LOWER), "inner") for r in block.rels}
|
||
joins = [(j.this.alias_or_name, j.this.name.translate(_ASCII_LOWER),
|
||
"left" if str(j.side or "").upper() == "LEFT" else "inner",
|
||
_fingerprint(j.args.get("on")) if j.args.get("on") is not None else "") for j in block.joins]
|
||
return prefix, joins
|
||
|
||
def _touched(self, usage: _Usage | None, cols: Iterable[tuple[str, str]],
|
||
kinds: tuple[str, ...] = ("cond", "case")) -> bool:
|
||
"""另一边用没用到这几列(见 _Usage.touches)。换了表的(_swapped)只按列名看:A 的 orders.is_deleted 在 B 那边
|
||
是 orders_archive.is_deleted,按表名看会以为 B 没用到。"""
|
||
if usage is None:
|
||
return True
|
||
if self.swapped:
|
||
return usage.touches_name({c for _, c in cols}, kinds)
|
||
return usage.touches(cols, kinds=kinds)
|
||
|
||
def relations(self, A: _Block, B: _Block) -> tuple[bool, dict[str, str], set[str], set[str]] | None:
|
||
"""两段读的来源对不对得上。对得上返回 (说到了没有, B 的别名 → A 的别名, A 已经说过的 WHERE 项, B 已经说过的
|
||
WHERE 项);对不上返回 None(整条只给兜底)。"""
|
||
pa, ja = self._shape(A)
|
||
pb, jb = self._shape(B)
|
||
if pa == pb and ja == jb:
|
||
return True, {}, set(), set()
|
||
if not ja and not jb:
|
||
only_a = [r for r in A.rels if r.alias not in pb]
|
||
only_b = [r for r in B.rels if r.alias not in pa]
|
||
if len(only_a) == 1 and len(only_b) == 1:
|
||
return self._swapped(A, B, only_a[0], only_b[0])
|
||
if len(only_a) == 1 and not only_b:
|
||
return self._extra_inner(A, B, only_a[0], 0)
|
||
if len(only_b) == 1 and not only_a:
|
||
return self._extra_inner(B, A, only_b[0], 1)
|
||
return None
|
||
kinds_a = {**{a: k for a, (_, k) in pa.items()}, **{a: k for a, _, k, _ in ja}}
|
||
kinds_b = {**{a: k for a, (_, k) in pb.items()}, **{a: k for a, _, k, _ in jb}}
|
||
names_a = {**{a: n for a, (n, _) in pa.items()}, **{a: n for a, n, _, _ in ja}}
|
||
names_b = {**{a: n for a, (n, _) in pb.items()}, **{a: n for a, n, _, _ in jb}}
|
||
only_a, only_b = set(kinds_a) - set(kinds_b), set(kinds_b) - set(kinds_a)
|
||
if any(names_a[x] != names_b[x] for x in set(names_a) & set(names_b)):
|
||
return None
|
||
if len(only_a) == 1 and not only_b and kinds_a[next(iter(only_a))] == "left":
|
||
return self._extra_left(A, B, next(iter(only_a)), 0)
|
||
if len(only_b) == 1 and not only_a and kinds_b[next(iter(only_b))] == "left":
|
||
return self._extra_left(B, A, next(iter(only_b)), 1)
|
||
if only_a or only_b:
|
||
return None
|
||
flipped = [x for x in kinds_a if kinds_a[x] != kinds_b[x]]
|
||
if len(flipped) != 1:
|
||
return None
|
||
x = flipped[0]
|
||
return self._left_inner(A, B, x, 0) if kinds_a[x] == "left" else self._left_inner(B, A, x, 1)
|
||
|
||
def _swapped(self, A: _Block, B: _Block, x: _Rel, y: _Rel) -> tuple[bool, dict[str, str], set[str], set[str]] | None:
|
||
"""换了一张表:开头那组来源只差一张实表(各自只出现一次、名字不只差在大小写上,都没带库名)。换了表的那一边,
|
||
列照它自己的表说。"""
|
||
if not (x.table and y.table) or A.names().count(x.name) != 1 or B.names().count(y.name) != 1 or \
|
||
x.name.translate(_ASCII_LOWER) == y.name.translate(_ASCII_LOWER) or \
|
||
any(r.node.args.get(k) for r in (x, y) for k in ("db", "catalog")):
|
||
return None
|
||
if not self.visible:
|
||
self.swapped = True
|
||
return False, {y.alias: x.alias}, set(), set()
|
||
self.add("tables", x.name, y.name, _say(self.la, "查的是", x.name, "表,", self.lb, "查的是", y.name, "表"))
|
||
self.swapped = True
|
||
return True, {y.alias: x.alias}, set(), set()
|
||
|
||
def _extra_inner(self, big: _Block, small: _Block, rel: _Rel, side: int
|
||
) -> tuple[bool, dict[str, str], set[str], set[str]] | None:
|
||
"""一边多(内)关联了一张表:WHERE 里涉及这张表的条件就是关联的条件(两边都没有别的连接)。内连接既会筛掉关联
|
||
不上的行、又会把关联上好几行的复制,结果一般不同;表结构说每行正好关联上一行(见 _lossless)时不说。"""
|
||
from sqlglot import exp
|
||
|
||
if not rel.table or big.names().count(rel.name) != 1:
|
||
return None
|
||
lbig, lsmall = (self.la, self.lb) if side == 0 else (self.lb, self.la)
|
||
conds = [c for c in big.where if rel.alias in {col.table for col in c.find_all(exp.Column)}]
|
||
skip = {self.fp(c) for c in conds}
|
||
skips = (skip, set()) if side == 0 else (set(), skip)
|
||
# 另一边原文也 LEFT JOIN 了这张表、规范化因为结果不怕重复把它去掉了(见 _drop_unused_left_joins):不能说「另一边
|
||
# 没有」(第 5 轮复验:COUNT(DISTINCT …) … LEFT JOIN refunds 对内连接 refunds,以前说「A 没有」关联 refunds)
|
||
dropped = small.select.meta.get(_DROPPED_JOINS) or []
|
||
if not self.visible or rel.name in dropped or self._lossless(big, rel, conds):
|
||
return False, {}, *skips
|
||
shown = _say("(关联的条件是", self.show_all(conds, big), ")") if conds and \
|
||
all(self.base_columns(c, big) is not None for c in conds) else ""
|
||
what = self.what(rel)
|
||
self.add("join", what if side == 0 else None, what if side == 1 else None,
|
||
_say(lbig, "关联了", what, shown, ",", lsmall, "没有"))
|
||
return True, {}, *skips
|
||
|
||
def _extra_left(self, big: _Block, small: _Block, alias: str, side: int
|
||
) -> tuple[bool, dict[str, str], set[str], set[str]] | None:
|
||
"""一边多 LEFT JOIN 了一张表,别的来源两边一样:这张表在它自己的 ON 之外哪儿都没用到时,有没有这次关联只差在
|
||
左边的行会不会因为关联上好几行而重复。结果不怕重复(去重、MIN / MAX、去重的聚合)、表结构里这张表的键出现在 ON
|
||
里(可能每行最多关联上一行)时说不准,不说。"""
|
||
pb, jb = self._shape(big)
|
||
ps, js = self._shape(small)
|
||
if pb != ps or [j for j in jb if j[0] != alias] != js:
|
||
return None
|
||
join = next(j for j in big.joins if j.this.alias_or_name == alias)
|
||
rel = big.rel(alias)
|
||
# 另一边原文也 LEFT JOIN 了这张表、规范化因为结果不怕重复把它去掉了(见 _drop_unused_left_joins):不能说「另一边
|
||
# 没有」
|
||
dropped = small.select.meta.get(_DROPPED_JOINS) or []
|
||
if rel is None or not rel.table or big.names().count(rel.name) != 1 or join.meta.get(_FLIPPED) or \
|
||
rel.name in dropped or self._used_outside(big, alias, join) or not self._dup_visible(big) or \
|
||
self._keyed(rel, _conjuncts(join.args["on"])):
|
||
return False, {}, set(), set()
|
||
lbig, lsmall = (self.la, self.lb) if side == 0 else (self.lb, self.la)
|
||
what = self.what(rel)
|
||
self.add("join", what if side == 0 else None, what if side == 1 else None,
|
||
_say(lbig, "用 LEFT JOIN 关联", what, ",", lsmall, "没有"))
|
||
return True, {}, set(), set()
|
||
|
||
def _left_inner(self, left: _Block, inner: _Block, alias: str, side: int
|
||
) -> tuple[bool, dict[str, str], set[str], set[str]] | None:
|
||
"""同一张表,left 那边 LEFT JOIN、inner 那边内连接,关联条件一样(left 的 ON 各项就是 inner 那边 ON 和 WHERE 里
|
||
涉及这张表的各项),别的来源两边一样。说「A 用 LEFT JOIN 关联 X,B 用 INNER JOIN」之前确认两种写法结果真的可能
|
||
不同(见 _observes_unmatched);可空一侧在 WHERE、HAVING、后面的连接里用到了的(后面的内连接引用可空一侧时,没关联
|
||
上的行反正留不下来)、表结构说每行都关联得上的(见 _lossless)、原文写的是 RIGHT JOIN 或者被 WHERE / 聚合改成
|
||
内连接的(说法和原文对不上)都不说。说不了时照样返回对得上(关联条件那几项不再比),这一处算没说到。"""
|
||
from sqlglot import exp
|
||
|
||
pl, jl = self._shape(left)
|
||
pi, ji = self._shape(inner)
|
||
if {a: v for a, v in pl.items() if a != alias} != {a: v for a, v in pi.items() if a != alias} or \
|
||
[j for j in jl if j[0] != alias] != [j for j in ji if j[0] != alias]:
|
||
return None
|
||
join = next(j for j in left.joins if j.this.alias_or_name == alias)
|
||
on = {self.fp(c) for c in _conjuncts(join.args["on"])}
|
||
inner_join = next((j for j in inner.joins if j.this.alias_or_name == alias), None)
|
||
in_where = [c for c in inner.where if alias in {col.table for col in c.find_all(exp.Column)}]
|
||
inner_on = [self.fp(c) for c in _conjuncts(inner_join.args["on"])] if inner_join is not None and \
|
||
inner_join.args.get("on") is not None else []
|
||
# inner 那边的 ON 只能是 left 的 ON 里的项;WHERE 里涉及这张表、又不是 left 的 ON 里的项,是另外的筛选条件,
|
||
# 留给 conditions 比
|
||
if not set(inner_on) <= on or not on <= set(inner_on) | {self.fp(c) for c in in_where}:
|
||
return None
|
||
skip = {self.fp(c) for c in in_where if self.fp(c) in on}
|
||
skips = (set(), skip) if side == 0 else (skip, set())
|
||
rel = left.rel(alias)
|
||
marked = any(r.node.meta.get(_REJECTED) or r.node.meta.get(_NULL_BLIND) for b in (left, inner)
|
||
for r in b.all_rels()) or any(j.meta.get(_FLIPPED) for b in (left, inner) for j in b.joins)
|
||
if rel is None or not rel.table or left.names().count(rel.name) != 1 or marked or \
|
||
self._used_outside(left, alias, join, outputs=False) or not self._observes_unmatched(left, alias) or \
|
||
self._lossless(left, rel, _conjuncts(join.args["on"])):
|
||
return False, {}, *skips
|
||
lleft, linner = (self.la, self.lb) if side == 0 else (self.lb, self.la)
|
||
what = self.what(rel)
|
||
text = _say(lleft, "用 LEFT JOIN 关联", what, ",", linner, "用 INNER JOIN") if side == 0 else \
|
||
_say(linner, "用 INNER JOIN 关联", what, ",", lleft, "用 LEFT JOIN")
|
||
self.add("join", "OUTER JOIN" if side == 0 else "INNER JOIN", "INNER JOIN" if side == 0 else "OUTER JOIN", text)
|
||
return True, {}, *skips
|
||
|
||
def _dup_visible(self, block: _Block) -> bool:
|
||
"""结果看得见重复的行:不聚合、不去重(重复的行就在结果里;只取第一行的看不出——重复出来的行和原来那行一样);
|
||
聚合时 SELECT 列表里有一项本身就是不去重的 COUNT / SUM / AVG(见 _top_aggregates:MIN / MAX、去重的聚合、拿聚合
|
||
再算的看不出)。"""
|
||
from sqlglot import exp
|
||
|
||
if _dup_insensitive(block.select) or block.distinct or block.deduped:
|
||
return False
|
||
if not _aggregating(block.select):
|
||
return not (block.limit is not None and block.offset is None and
|
||
(not block.limit.isdigit() or int(block.limit) < 2))
|
||
return any(isinstance(a, (exp.Count, exp.Sum, exp.Avg)) and not isinstance(a.this, exp.Distinct) and
|
||
not a.args.get("expressions") for a in self._top_aggregates(block))
|
||
|
||
@staticmethod
|
||
def _used_outside(block: _Block, alias: str, join: Any, *, outputs: bool = True) -> bool:
|
||
"""alias 那张表的列在它自己的 ON 之外用到了(outputs 为假时不看 SELECT 列表、GROUP BY、ORDER BY:可空一侧的列
|
||
选出来、分组、排序都不会把没关联上的行筛掉);有不带表名的列(分不清是不是这张表的)也算用到了。"""
|
||
from sqlglot import exp
|
||
|
||
own = {id(n) for n in join.args["on"].walk()}
|
||
holders = [block.select.args.get(k) for k in ("where", "having")] + \
|
||
[j.args.get("on") for j in block.joins if j is not join]
|
||
if outputs:
|
||
holders += [*block.select.expressions, block.select.args.get("group"), block.select.args.get("order")]
|
||
for holder in holders:
|
||
for col in holder.find_all(exp.Column) if holder is not None else []:
|
||
if id(col) not in own and (col.table == alias or not col.table):
|
||
return True
|
||
return False
|
||
|
||
def _observes_unmatched(self, block: _Block, alias: str) -> bool:
|
||
"""LEFT JOIN 比内连接多出来的只是没关联上的那些行(可空一侧整行是空值):结果看得见它们才可能不同。看得见的:
|
||
分组(只有没关联上的行的组只在 LEFT JOIN 里有;HAVING 用到可空一侧时说不准);不聚合(多出来的行就在结果里);
|
||
不分组的聚合里有 COUNT(*)、或者对保留一侧的列做的聚合(不带 FILTER)。只取可空一侧跳过空值的聚合(SUM(r.x)、
|
||
TOTAL、COUNT(r.x)、MAX、FILTER (WHERE r.x > 0) 里的 COUNT(*))看不见,两种写法结果一样。"""
|
||
from sqlglot import exp
|
||
|
||
if _loose_columns(block.select):
|
||
return False
|
||
if block.group:
|
||
return not any(c.table == alias or not c.table for cond in block.having for c in cond.find_all(exp.Column))
|
||
if not _aggregating(block.select):
|
||
return True
|
||
if block.having:
|
||
return False
|
||
for agg in self._top_aggregates(block):
|
||
if agg.args.get("expressions"):
|
||
continue
|
||
arg = agg.this
|
||
if isinstance(arg, exp.Distinct):
|
||
arg = arg.expressions[0] if len(arg.expressions) == 1 else None
|
||
if isinstance(agg, exp.Count) and isinstance(arg, exp.Star):
|
||
return True
|
||
if type(agg).__name__ in _NULL_SKIPPING and isinstance(arg, exp.Column) and arg.table and arg.table != alias:
|
||
return True
|
||
return False
|
||
|
||
def _keys(self, rel: _Rel) -> list[list[str]]:
|
||
table = rel.name.translate(_ASCII_LOWER)
|
||
return ([self.info.pk[table]] if self.info.pk.get(table) else []) + self.info.unique.get(table, [])
|
||
|
||
def _keyed(self, rel: _Rel, conds: list[Any]) -> bool:
|
||
"""表结构里 rel 那张表有键(主键、唯一约束),键的某一列出现在关联条件里:可能每行最多关联上它一行(不管条件写成
|
||
= 还是 IN),说不准会不会重复。只拿来少说:约束说的「唯一」不一定成立,不拿它判等价。"""
|
||
from sqlglot import exp
|
||
|
||
keys = {k for key in self._keys(rel) for k in key}
|
||
return any(col.table == rel.alias and col.name.translate(_ASCII_LOWER) in keys
|
||
for c in conds for col in c.find_all(exp.Column))
|
||
|
||
def _lossless(self, block: _Block, rel: _Rel, conds: list[Any]) -> bool:
|
||
"""表结构说内连接 rel 每行都关联得上、又最多一行:这一段别的某张表有指向 rel 的外键,关联条件里又用到了 rel
|
||
的键。这时有没有这次内连接、LEFT JOIN 还是内连接结果都一样,不说。只拿来少说:外键在 SQLite 里默认不检查、
|
||
外键列可能为空,不一定真的每行都对得上。"""
|
||
table = rel.name.translate(_ASCII_LOWER)
|
||
if not self._keyed(rel, conds):
|
||
return False
|
||
return any(to == table for r in block.all_rels() if r.alias != rel.alias and r.table
|
||
for _, to, _ in self.info.fks.get(r.name.translate(_ASCII_LOWER), []))
|
||
|
||
# ---- WHERE
|
||
|
||
def timed(self, cond: Any, block: _Block) -> tuple[str, str] | None:
|
||
"""时间条件:一列(可以套着日期函数、CAST)和常量比较(=、<>、<、<=、>、>=、IN、BETWEEN),列名像时间或者常量
|
||
像日期;或者时间列(列名像时间、套着日期函数)LIKE 一个像日期的模式(SQLite 里日期存成文字,paid_at LIKE
|
||
'2026-09%' 是取月)。返回 (实表, 列)。只认一定是时间的:date_key BETWEEN 20260901 AND 20260930 这类认不准的
|
||
(见 _window_like)不算,调用方只给兜底。"""
|
||
from sqlglot import exp
|
||
|
||
like = isinstance(cond, (exp.Like, exp.ILike)) and not cond.args.get("escape")
|
||
if isinstance(cond, exp.In):
|
||
column_side, others = cond.this, list(cond.expressions)
|
||
elif isinstance(cond, exp.Between):
|
||
if cond.args.get("symmetric"):
|
||
return None
|
||
column_side, others = cond.this, [cond.args.get("low"), cond.args.get("high")]
|
||
elif like:
|
||
column_side, others = cond.this, [cond.expression]
|
||
elif not isinstance(cond, (exp.EQ, exp.NEQ, exp.GT, exp.GTE, exp.LT, exp.LTE)):
|
||
return None
|
||
elif _fixed(cond.expression):
|
||
column_side, others = cond.this, [cond.expression]
|
||
elif _fixed(cond.this):
|
||
column_side, others = cond.expression, [cond.this]
|
||
else:
|
||
return None
|
||
if not others or not all(o is not None and _fixed(o) for o in others):
|
||
return None
|
||
cols = self.base_columns(column_side, block)
|
||
if not cols or len({(r.alias, c.name) for r, c in cols}) != 1 or _value_column(column_side) is None:
|
||
return None
|
||
rel, col = cols[0]
|
||
if like:
|
||
# 模式得像日期(去掉两头的 %、_ 之后),对象得是时间列:status LIKE '2026%' 不是时间窗口
|
||
if not _date_like_literal(others[0]) or not (_temporal_name(col.name) or _datey(column_side)):
|
||
return None
|
||
return rel.name, col.name
|
||
if _temporal_name(col.name) or any(_datey(o) for o in others) or _datey(column_side):
|
||
return rel.name, col.name
|
||
return None
|
||
|
||
@staticmethod
|
||
def _left_joined(block: _Block) -> set[str]:
|
||
"""这一段里 LEFT JOIN 进来的来源(可空一侧)的别名。"""
|
||
return {j.this.alias_or_name for j in block.joins if str(j.side or "").upper() == "LEFT"}
|
||
|
||
def _on_nullable_side(self, conds: list[Any], block: _Block) -> bool:
|
||
"""几个条件用到了 LEFT JOIN 可空一侧的列:可空一侧的列为空说的是「没关联上」(反连接),不为空的条件又让 LEFT
|
||
JOIN 变得像内连接——都是关联上的事,不是有效行过滤、时间窗口。认不出列的来源时也算用到了(少说)。"""
|
||
from sqlglot import exp
|
||
|
||
left = self._left_joined(block)
|
||
if not left:
|
||
return False
|
||
for cond in conds:
|
||
for col in cond.find_all(exp.Column):
|
||
if col.is_star:
|
||
continue
|
||
cols = self.base_columns(col, block)
|
||
if not cols or cols[0][0].alias in left:
|
||
return True
|
||
return False
|
||
|
||
def _unmatched(self, xs: list[Any], ys: list[Any], mapping: Mapping[str, str]) -> tuple[list[Any], list[Any]]:
|
||
rest = list(ys)
|
||
only_x = []
|
||
for x in xs:
|
||
found = next((y for y in rest if self.mfp(y, mapping) == self.fp(x)), None)
|
||
if found is None:
|
||
only_x.append(x)
|
||
else:
|
||
rest.remove(found)
|
||
return only_x, rest
|
||
|
||
def conditions(self, A: _Block, B: _Block, mapping: Mapping[str, str], skip_a: set[str],
|
||
skip_b: set[str]) -> bool:
|
||
"""WHERE:两边没配上的条件里说得准的两类——按哪一列筛时间不同(_times)、一边多出来的筛选条件(_extra_filters);
|
||
剩下的算没说到。"""
|
||
xa = [c for c in A.where if self.fp(c) not in skip_a]
|
||
xb = [c for c in B.where if self.fp(c) not in skip_b]
|
||
only_a, only_b = self._unmatched(xa, xb, mapping)
|
||
if not only_a and not only_b:
|
||
return True
|
||
if not self.visible:
|
||
return False
|
||
only_a, only_b, covered = self._times(A, B, only_a, only_b)
|
||
covered &= self._extra_filters(A, B, xa, only_a, 0)
|
||
covered &= self._extra_filters(B, A, xb, only_b, 1)
|
||
return covered
|
||
|
||
def _times(self, A: _Block, B: _Block, only_a: list[Any], only_b: list[Any]) -> tuple[list[Any], list[Any], bool]:
|
||
"""两边各在一列上按时间筛、列不同:说「A 按 X 筛时间,B 按 Y」。要求:两边整个 WHERE 里的时间条件各只在这一列
|
||
上,这一列上的条件全是另一边没有的(配上了的说明另一边也用这一列筛),另一边哪儿都没用这一列筛选,两边的时间
|
||
条件都不是恒为真的。说到了的从两边拿掉;不是这种情况的原样还回去(一边按时间筛的算多出来的筛选,见
|
||
_extra_filters;同一列上范围不同的说不了,算没说到)。"""
|
||
ta = [(c, t) for c in only_a if (t := self.timed(c, A)) is not None]
|
||
tb = [(c, t) for c in only_b if (t := self.timed(c, B)) is not None]
|
||
cols_a, cols_b = {t for _, t in ta}, {t for _, t in tb}
|
||
if not ta or not tb or cols_a == cols_b:
|
||
return only_a, only_b, True
|
||
if len(cols_a) != 1 or len(cols_b) != 1 or self.usage_a is None or self.usage_b is None:
|
||
return only_a, only_b, False
|
||
(ka,), (kb,) = cols_a, cols_b
|
||
conds_a, conds_b = [c for c, _ in ta], [c for c, _ in tb]
|
||
all_a = {t for c in A.where if (t := self.timed(c, A)) is not None}
|
||
all_b = {t for c in B.where if (t := self.timed(c, B)) is not None}
|
||
on_a = [c for c in A.where if (self.refs(c, A) or set()) & {ka}]
|
||
on_b = [c for c in B.where if (self.refs(c, B) or set()) & {kb}]
|
||
# 时间列在 LEFT JOIN 的可空一侧:条件管的是关联上的那些行,说成「按它筛时间」说不准
|
||
if all_a != cols_a or all_b != cols_b or len(on_a) != len(conds_a) or len(on_b) != len(conds_b) or \
|
||
self._touched(self.usage_b, {ka}) or self._touched(self.usage_a, {kb}) or \
|
||
self._on_nullable_side(conds_a, A) or self._on_nullable_side(conds_b, B) or \
|
||
self._always(conds_a, A) is True or self._always(conds_b, B) is True:
|
||
return only_a, only_b, False
|
||
qualified = not self.swapped and ka[0] != kb[0]
|
||
text = _say(self.la, "按", self.named(*ka, qualified=qualified), "筛时间,", self.lb, "按",
|
||
self.named(*kb, qualified=qualified))
|
||
if self._time_shape(conds_a) != self._time_shape(conds_b):
|
||
text += _say("(", self.la, "是", self.show_all(_by_bound(conds_a), A), ",", self.lb, "是",
|
||
self.show_all(_by_bound(conds_b), B), ")")
|
||
detail = {"a": [{"table": ka[0], "column": ka[1]}], "b": [{"table": kb[0], "column": kb[1]}]}
|
||
self.add("time", ka[1], kb[1], text, detail)
|
||
return [c for c in only_a if all(c is not x for x in conds_a)], \
|
||
[c for c in only_b if all(c is not x for x in conds_b)], True
|
||
|
||
def _extra_filters(self, block: _Block, other: _Block, where: list[Any], rest: list[Any], side: int) -> bool:
|
||
"""block 这一边多出来的筛选条件(rest 是它没配上的 WHERE 项)。按用到的列连成一组(x > 5 和 x IS NOT NULL
|
||
是一组)一组一组看,说得准的说「A 只保留 … 的行,B 没有这个条件」「A 排除了 … 的行,B 没有」「A 按 X 筛时间,
|
||
B 没有按时间筛选」,说不准的算没说到。说得准要:这一组全是另一边没有的(有一项配上了,说明另一边对这些列也有
|
||
条件,多出来的可能被它蕴含);只涉及一张实表的列;另一边整条语句对这些列没有任何条件(见 _Usage:写在聚合的参数
|
||
里的 SUM(status = 'paid')、SUM(amount * (1 - is_deleted)) 也算有);不是时间条件的,比的对象就是那几列本身
|
||
(LOWER(x)、LENGTH(x) 的取值范围认不出,x <> 'PAID' 可能恒为真);按取值逐段算过不是恒为真的(见 _always)。
|
||
|
||
这几种不说(第 5 轮复验抓到的):用到 LEFT JOIN 可空一侧的列的(WHERE r.id IS NULL 是反连接,说的是「没关联
|
||
上」;r.order_id IS NULL 和 ON 里的 r.order_id = o.id 一起已经蕴含 r.id IS NULL,说「另一边没有这个条件」把留得
|
||
多的一边说成了严的一边);可能是时间窗口、又认不准是哪一列的时间的(date_key BETWEEN 20260901 AND 20260930、
|
||
stat_month = 202609,见 _window_like:说成有效行过滤会被写进数据目录);说「另一边没有按时间筛选」时另一边可能
|
||
按时间筛了(见 _Usage.temporal)。"""
|
||
usage = self.usage_b if side == 0 else self.usage_a
|
||
me, them = (self.la, self.lb) if side == 0 else (self.lb, self.la)
|
||
covered = True
|
||
pending = list(rest)
|
||
while pending:
|
||
group = self._linked(pending[0], where, block)
|
||
pending = [c for c in pending if all(c is not g for g in group)]
|
||
refs = [self.refs(g, block) for g in group]
|
||
if any(all(g is not r for r in rest) for g in group) or any(r is None for r in refs) or usage is None or \
|
||
self._on_nullable_side(group, block):
|
||
covered = False
|
||
continue
|
||
cols = set().union(*refs)
|
||
times = {t for g in group if (t := self.timed(g, block)) is not None}
|
||
if not times and any(_window_like(g) for g in group):
|
||
# 可能是时间窗口,又认不准是不是时间列:既不说成有效行过滤,也不说成按它筛时间
|
||
covered = False
|
||
continue
|
||
always = self._always(group, block)
|
||
kinds = _kinds_for(group)
|
||
if "agg" not in kinds and self._always(group, block, assume_values=True) is True:
|
||
# 只筛掉了这几列为空的行(x > 5 OR x <= 5):SUM(x)、MIN(x) 这类本来就跳过空值,另一边聚合的参数用到
|
||
# 这几列时,有没有这个条件结果可能一样
|
||
kinds = ("cond", "case", "agg")
|
||
if len({t for t, _ in cols}) != 1 or self._touched(usage, cols, kinds) or (
|
||
times and (len(times) != 1 or len(cols) != 1 or usage.temporal or always is True)) or (
|
||
not times and (not self._plain_subjects(group) or always is not False)):
|
||
covered = False
|
||
continue
|
||
columns = {"columns": [{"table": t, "column": c} for t, c in sorted(cols)]}
|
||
if times:
|
||
(key,) = times
|
||
shown_key = self.named(*key, qualified=not self.swapped and len(block.all_rels()) > 1)
|
||
text = _say(me, "按", shown_key, "筛时间,", them, "没有按时间筛选") if side == 0 else \
|
||
_say(them, "没有按时间筛选,", me, "按", shown_key, "筛时间")
|
||
self.add("time", key[1] if side == 0 else None, key[1] if side == 1 else None, text,
|
||
{"a" if side == 0 else "b": [{"table": key[0], "column": key[1]}],
|
||
"b" if side == 0 else "a": []})
|
||
continue
|
||
shown = self.show_all(group, block)
|
||
excluded = self._excludes(group[0], block) if len(group) == 1 else None
|
||
text = _say(me, "排除了", excluded, "的行,", them, "没有") if excluded else \
|
||
_say(me, "只保留", shown, "的行,", them, "没有这个条件")
|
||
self.add("filter", shown if side == 0 else None, shown if side == 1 else None, text, columns)
|
||
return covered
|
||
|
||
def _linked(self, cond: Any, where: list[Any], block: _Block) -> list[Any]:
|
||
"""WHERE 里和 cond 用到同一列(连着传过去)的各项,按原来的先后。认不出用到哪些列的只算它自己。"""
|
||
group = [cond]
|
||
cols = self.refs(cond, block)
|
||
if not cols:
|
||
return group
|
||
changed = True
|
||
while changed:
|
||
changed = False
|
||
for c in where:
|
||
if any(c is g for g in group):
|
||
continue
|
||
theirs = self.refs(c, block)
|
||
if theirs is not None and theirs & cols:
|
||
group.append(c)
|
||
cols |= theirs
|
||
changed = True
|
||
return [c for c in where if any(c is g for g in group)]
|
||
|
||
def _plain_subjects(self, conds: list[Any]) -> bool:
|
||
"""几个条件里拿来和常量比的对象都是光一列(不是 LOWER(x)、x + 1 这类算出来的值)。"""
|
||
from sqlglot import exp
|
||
|
||
subjects = self.range.subjects(conds)
|
||
return subjects is not None and all(isinstance(_unparen(node), exp.Column) for node in subjects.values())
|
||
|
||
def _always(self, conds: list[Any], block: _Block, *, assume_values: bool = False) -> bool | None:
|
||
"""几个条件(AND 连起来)是不是恒为真:认得出(每个比的都是对象和常量)时按每个对象的各段取值逐个组合算一遍
|
||
(见 _Range.always);一定不为空的列(见 _never_null)不取空值,assume_values 时每个对象都不取空值(看这几个
|
||
条件是不是只筛掉了空值)。认不出返回 None。"""
|
||
subjects = self.range.subjects(conds)
|
||
if subjects is None:
|
||
return None
|
||
never_null = set(subjects) if assume_values else self._never_null(block) & set(subjects)
|
||
return self.range.always(conds, never_null)
|
||
|
||
@staticmethod
|
||
def _time_shape(conds: list[Any]) -> list[str]:
|
||
"""时间条件把那一列换成占位符之后的写法:两边只差时间列时一样。"""
|
||
from sqlglot import exp
|
||
|
||
out = []
|
||
for c in conds:
|
||
copy = c.copy()
|
||
for col in list(copy.find_all(exp.Column)):
|
||
col.replace(exp.column("_time_"))
|
||
out.append(_fingerprint(copy))
|
||
return sorted(out)
|
||
|
||
def _excludes(self, cond: Any, block: _Block) -> str | None:
|
||
"""「排除」的说法:排除的是哪些行。<> 常量、NOT IN 常量列表、NOT LIKE 除了等于 / 落在列表里 / 匹配上的行,
|
||
这一列为空的行也筛掉了(拿空值比是空值,WHERE 只留下真的),照实说出来(表结构里确定不为空的列不说);IS NOT
|
||
NULL 排除的就是为空的行。别的 NOT(NOT (a AND b) 还把 a、b 有一个为空、另一个成立的行也筛掉了)说不清排除了
|
||
哪些,返回 None,调用方改说「只保留 … 的行」。"""
|
||
from sqlglot import exp
|
||
|
||
def and_null(subject: Any, shown: str) -> str:
|
||
cols = self.base_columns(subject, block) if isinstance(_unparen(subject), exp.Column) else None
|
||
if cols and all(_not_null_column(self.info, r.name, c.name, self.read) for r, c in cols):
|
||
return shown
|
||
return _say(shown, "的行和", self.show(subject, block), "为空")
|
||
|
||
cond = _unparen(cond)
|
||
if isinstance(cond, exp.NEQ):
|
||
left, right = cond.this, cond.expression
|
||
subject = left if _fixed(right) and not _fixed(left) else (
|
||
right if _fixed(left) and not _fixed(right) else None)
|
||
if subject is None or _quantified(left) or _quantified(right):
|
||
return None
|
||
return and_null(subject, self.show(exp.EQ(this=left.copy(), expression=right.copy()), block))
|
||
if isinstance(cond, exp.Not):
|
||
inner = _unparen(cond.this)
|
||
if isinstance(inner, exp.Is) and isinstance(_unparen(inner.expression), exp.Null):
|
||
return _say(self.show(inner.this, block), "为空")
|
||
if isinstance(inner, exp.In) and inner.expressions and not inner.args.get("query") and \
|
||
all(_fixed(i) and not isinstance(_unparen(i), exp.Null) for i in inner.expressions) and \
|
||
not _fixed(inner.this):
|
||
return and_null(inner.this, self.show(inner, block))
|
||
if isinstance(inner, (exp.Like, exp.ILike)) and _fixed(inner.expression) and not _fixed(inner.this):
|
||
return and_null(inner.this, self.show(inner, block))
|
||
return None
|
||
|
||
# ---- 选出来的列:聚合函数、去重
|
||
|
||
def outputs(self, A: _Block, B: _Block, mapping: Mapping[str, str]) -> bool:
|
||
"""选出来的列和分组。两边选出来的列一样、只差去重的,说去重(_dedup);分组一样、只差在几列顶层的聚合上的,
|
||
逐列说(_aggregates)。别的不同算没说到。"""
|
||
ta = [self.fp(e.unalias()) for e in A.exprs]
|
||
tb = [self.mfp(e.unalias(), mapping) for e in B.exprs]
|
||
ga = sorted(self.fp(g) for g in A.group)
|
||
gb = sorted(self.mfp(g, mapping) for g in B.group)
|
||
dedup_a, dedup_b = A.distinct or A.deduped, B.distinct or B.deduped
|
||
if ta == tb and ga == gb and A.distinct == B.distinct:
|
||
return True
|
||
if ta == tb and dedup_a != dedup_b:
|
||
return self._dedup(A, B, ga, gb)
|
||
if ga != gb or A.distinct != B.distinct or dedup_a != dedup_b or len(ta) != len(tb):
|
||
return False
|
||
covered = True
|
||
for x, y, fx, fy in zip(A.exprs, B.exprs, ta, tb):
|
||
if fx != fy and not self._aggregates(x, y, A, B, mapping):
|
||
covered = False
|
||
return covered
|
||
|
||
def _dedup(self, A: _Block, B: _Block, ga: list[str], gb: list[str]) -> bool:
|
||
"""一边去重(SELECT DISTINCT,规范化写成了按选出来的列 GROUP BY)、另一边不去重,选出来的列一样。去重不起作用的
|
||
不说:不分组的聚合只出一行;已经按选出来的列分组;只取一行;表结构里的键说明每行本来就不一样。"""
|
||
from sqlglot import exp
|
||
|
||
side = 0 if (A.distinct or A.deduped) else 1
|
||
D, N = (A, B) if side == 0 else (B, A)
|
||
if D.deduped:
|
||
if N.group or D.distinct:
|
||
return False
|
||
elif ga != gb:
|
||
return False
|
||
if _aggregating(D.select) and not D.deduped:
|
||
if not D.group:
|
||
return False
|
||
outs = {self.fp(e.unalias()) for e in D.exprs}
|
||
if all(self.fp(g) in outs for g in D.group):
|
||
return False
|
||
if D.limit is not None and D.offset is None and (not D.limit.isdigit() or int(D.limit) < 2):
|
||
return False
|
||
chosen = {self._col_key(c, D) for e in D.exprs if isinstance(c := _unparen(e.unalias()), exp.Column)}
|
||
if self._maybe_unique(D, {k for k in chosen if k is not None}):
|
||
return False
|
||
me, them = (self.la, self.lb) if side == 0 else (self.lb, self.la)
|
||
if D.deduped:
|
||
keys = "、".join(self.show(k, D) for k in D.group)
|
||
text = _say(me, "对", keys, "去重,", them, "没有去重")
|
||
else:
|
||
keys = "DISTINCT"
|
||
text = _say(me, "用 SELECT DISTINCT 去掉了重复的行,", them, "没有")
|
||
self.add("group_by", keys if side == 0 else None, keys if side == 1 else None, text)
|
||
return True
|
||
|
||
def _aggregates(self, x: Any, y: Any, A: _Block, B: _Block, mapping: Mapping[str, str]) -> bool:
|
||
"""同一位置上两边各是一个顶层的聚合(不套在别的表达式里、不带 FILTER):只说函数不同(同一个参数)和去重不同
|
||
(COUNT / SUM / AVG 去不去重、COUNT(*) 对 COUNT(DISTINCT x))。COUNT(x) 对 COUNT(*)(只差在 x 为空的行)、参数
|
||
不同的都不说。"""
|
||
from sqlglot import exp
|
||
|
||
xu, yu = x.unalias(), y.unalias()
|
||
ga, gb = _aggregate(xu), _aggregate(yu)
|
||
if ga is None or gb is None:
|
||
return False
|
||
(fa, arg_a, da), (fb, arg_b, db) = ga, gb
|
||
# 参数要是光一列,而且在每一组里不是定死的:MIN(5) 和 MAX(5)、WHERE x = 5 之后的 MIN(x) 和 MAX(x)、按 g 分组后的
|
||
# AVG(DISTINCT g) 和 AVG(g)、x 定成 1 时的 SUM(x) 和 COUNT(*) 都一样
|
||
for arg, block in ((arg_a, A), (arg_b, B)):
|
||
if arg is not None and (not isinstance(_unparen(arg), exp.Column) or self.base_columns(arg, block) is None
|
||
or self._fixed_in_group(block, _unparen(arg))):
|
||
return False
|
||
same_arg = (arg_a is None and arg_b is None) or (
|
||
arg_a is not None and arg_b is not None and self.fp(arg_a) == self.mfp(arg_b, mapping))
|
||
wa = self.show(arg_a, A) if arg_a is not None else "*"
|
||
wb = self.show(arg_b, B) if arg_b is not None else "*"
|
||
what_a = ("数的是行数",) if fa == "count" and arg_a is None else ("对", wa, _verb(fa, da))
|
||
what_b = ("数的是行数",) if fb == "count" and arg_b is None else ("对", wb, _verb(fb, db))
|
||
if same_arg and arg_a is not None and fa != fb and da == db:
|
||
# 每组只有一行时 SUM、AVG、MIN、MAX 都是那一行的值
|
||
if {fa, fb} <= {"sum", "avg", "min", "max"} and (self._single_rows(A) or self._single_rows(B)):
|
||
return False
|
||
text = _say(self.la, "对", wa, _verb(fa, da) + ",", self.lb, _verb(fb, db))
|
||
elif same_arg and arg_a is not None and fa == fb and da != db:
|
||
block, arg = (A, arg_a) if da else (B, arg_b)
|
||
if fa in _DISTINCT_BLIND or self._unique_value(block, arg):
|
||
return False
|
||
text = _say(self.la, "对", wa, _verb(fa, da) + ",", self.lb, "没去重") if da else \
|
||
_say(self.la, "对", wa, _verb(fa, da) + "时没去重,", self.lb, _verb(fb, db))
|
||
elif fa == fb == "count" and (arg_a is None) != (arg_b is None):
|
||
block, arg, distinct = (B, arg_b, db) if arg_a is None else (A, arg_a, da)
|
||
if not distinct or self._unique_value(block, arg):
|
||
return False
|
||
text = _say(self.la, *what_a, ",", self.lb, *what_b)
|
||
elif fa != fb and (arg_a is None) != (arg_b is None):
|
||
text = _say(self.la, *what_a, ",", self.lb, *what_b)
|
||
else:
|
||
return False
|
||
self.add("aggregate", self.show(x, A), self.show(y, B), text)
|
||
return True
|
||
|
||
def _fixed_in_group(self, block: _Block, col: Any) -> bool:
|
||
"""col 在每一组里是不是只有一个值:被 WHERE、ON 里的「列 = 常量」定死,或者等于某个分组键(顺着 WHERE、ON 里
|
||
两列相等传过去),或者它那张表的某个键(主键、唯一约束)在组里定死了(同一组里这张表就只有一行的值:按 u.id 分组
|
||
时 u.city 只有一个值,LEFT JOIN、内连接把行复制了也一样,MAX(u.city) 和 MIN(u.city)、AVG(DISTINCT o.amount) 和
|
||
AVG(o.amount) 都一样——第 5 轮复验抓到以前照说不误)。认不出是哪一列的也当成定死了(少说)。
|
||
|
||
表结构里的键只拿来少说:类型亲和、COLLATE 让「唯一」不成立时,这里多当成定死了,只会少说一句,不会说错。"""
|
||
from sqlglot import exp
|
||
|
||
key = self._col_key(col, block)
|
||
fixed = self._pinned(block) | {self._col_key(_unparen(g), block) for g in block.group
|
||
if isinstance(_unparen(g), exp.Column)}
|
||
keys = {r.alias: self._keys(r) for r in block.all_rels() if r.table}
|
||
whole: set[str] = set()
|
||
changed = True
|
||
while changed:
|
||
changed = False
|
||
for c in self._all_conds(block):
|
||
c = _unparen(c)
|
||
if isinstance(c, exp.EQ) and isinstance(_unparen(c.this), exp.Column) and \
|
||
isinstance(_unparen(c.expression), exp.Column):
|
||
left, right = self._col_key(_unparen(c.this), block), self._col_key(_unparen(c.expression), block)
|
||
for x, y in ((left, right), (right, left)):
|
||
if x is not None and (x in fixed or x[0] in whole) and y is not None and y not in fixed:
|
||
fixed.add(y)
|
||
changed = True
|
||
for alias, groups in keys.items():
|
||
if alias not in whole and any(key_cols and all((alias, k) in fixed for k in key_cols)
|
||
for key_cols in groups):
|
||
whole.add(alias)
|
||
changed = True
|
||
return key is None or key in fixed or key[0] in whole
|
||
|
||
# ---- 表结构里的键:只拿来少说
|
||
|
||
@staticmethod
|
||
def _all_conds(block: _Block) -> list[Any]:
|
||
"""WHERE 和各个连接的 ON 里 AND 连着的各项。LEFT JOIN 的 ON 也算:「列 = 常量」「两列相等」在关联上的行里成立,
|
||
没关联上的行那一侧整行是空值——拿来判断「定死了」「可能唯一」只会多认(少说),不会少认。"""
|
||
return block.where + [p for j in block.joins if j.args.get("on") is not None for p in _conjuncts(j.args["on"])]
|
||
|
||
def _pinned(self, block: _Block) -> set[tuple[str, str]]:
|
||
"""WHERE、ON 里被「列 = 常量」定死的列。"""
|
||
from sqlglot import exp
|
||
|
||
out = set()
|
||
for c in self._all_conds(block):
|
||
c = _unparen(c)
|
||
if isinstance(c, exp.EQ):
|
||
for col, other in ((c.this, c.expression), (c.expression, c.this)):
|
||
col = _unparen(col)
|
||
if isinstance(col, exp.Column) and _literal_value(other) is not None:
|
||
out.add(self._col_key(col, block))
|
||
return {k for k in out if k is not None}
|
||
|
||
@staticmethod
|
||
def _col_key(col: Any, block: _Block) -> tuple[str, str] | None:
|
||
rels = block.all_rels()
|
||
alias = col.table or (rels[0].alias if len(rels) == 1 else None)
|
||
return (alias, col.name.translate(_ASCII_LOWER)) if alias else None
|
||
|
||
def _maybe_unique(self, block: _Block, chosen: set[tuple[str, str]]) -> bool:
|
||
"""表结构里的键说明:chosen 这几列(加上 WHERE 里被常量定死的列)定下来之后,这一段最多只剩一行——每张表都能由
|
||
定下来的列推出它的某个键(主键、唯一约束),等式两边一边定下来另一边也定下来,一张表的键定下来它的列都定下来。
|
||
这时去重、每组只取一个值不起作用,不说。只拿来少说:认不全列(不带表名)时当成可能唯一。没给键时 False。"""
|
||
from sqlglot import exp
|
||
|
||
rels = block.all_rels()
|
||
keys = {r.alias: self._keys(r) for r in rels if r.table}
|
||
if not any(keys.values()):
|
||
return False
|
||
if any(c.name and not c.table for c in block.select.find_all(exp.Column)) and len(rels) > 1:
|
||
return True
|
||
known = {k for k in chosen if k is not None} | self._pinned(block)
|
||
pairs = []
|
||
for c in self._all_conds(block):
|
||
c = _unparen(c)
|
||
if isinstance(c, exp.EQ) and isinstance(_unparen(c.this), exp.Column) and \
|
||
isinstance(_unparen(c.expression), exp.Column):
|
||
left, right = self._col_key(_unparen(c.this), block), self._col_key(_unparen(c.expression), block)
|
||
if left and right:
|
||
pairs.append((left, right))
|
||
whole: set[str] = set()
|
||
changed = True
|
||
while changed:
|
||
changed = False
|
||
for left, right in pairs:
|
||
for x, y in ((left, right), (right, left)):
|
||
if (x in known or x[0] in whole) and y not in known:
|
||
known.add(y)
|
||
changed = True
|
||
for alias, groups in keys.items():
|
||
if alias not in whole and any(all((alias, k) in known for k in key) for key in groups if key):
|
||
whole.add(alias)
|
||
changed = True
|
||
return all(r.alias in whole for r in rels)
|
||
|
||
def _unique_value(self, block: _Block, arg: Any) -> bool:
|
||
"""每组里 arg 的取值是不是可能各不相同(DISTINCT 不起作用):arg 是一列、和分组键一起定下来就只剩一行时是;
|
||
不是光一列的、表结构又给了键的,当成可能是(少说)。"""
|
||
from sqlglot import exp
|
||
|
||
node = _unparen(arg)
|
||
groups = {self._col_key(_unparen(g), block) for g in block.group if isinstance(_unparen(g), exp.Column)}
|
||
if not isinstance(node, exp.Column):
|
||
return any(self._keys(r) for r in block.all_rels() if r.table)
|
||
return self._maybe_unique(block, groups | {self._col_key(node, block)})
|
||
|
||
def _single_rows(self, block: _Block) -> bool:
|
||
"""每组最多一行:分组键(和 WHERE 里定死的列)定下来就只剩一行。"""
|
||
from sqlglot import exp
|
||
|
||
groups = {self._col_key(_unparen(g), block) for g in block.group if isinstance(_unparen(g), exp.Column)}
|
||
if block.group and len(groups) != len(block.group):
|
||
return any(self._keys(r) for r in block.all_rels() if r.table)
|
||
return self._maybe_unique(block, groups)
|
||
|
||
# ---- 别的子句
|
||
|
||
def rest(self, A: _Block, B: _Block, mapping: Mapping[str, str]) -> bool:
|
||
"""HAVING、排序、取几行:不同就算没说到(这几处的不同不说具体的:恒为真的 HAVING、被 WHERE 定成常量的排序键、
|
||
只出一组时的排序都不影响结果,逐个认不全)。"""
|
||
if sorted(self.fp(c) for c in A.having) != sorted(self.mfp(c, mapping) for c in B.having):
|
||
return False
|
||
if [self.fp(o) for o in A.order] != [self.mfp(o, mapping) for o in B.order]:
|
||
return False
|
||
return A.limit == B.limit and A.offset == B.offset
|
||
|
||
|
||
def _aggregate(node: Any) -> tuple[str, Any, bool] | None:
|
||
"""光一个聚合(SUM / AVG / COUNT / MIN / MAX,MIN / MAX 只有一个参数:两个参数的 max(a, b) 是取较大的那个,不是
|
||
聚合):(函数, 参数(COUNT(*) 是 None), 是否去重)。"""
|
||
from sqlglot import exp
|
||
|
||
kinds = {exp.Sum: "sum", exp.Avg: "avg", exp.Count: "count", exp.Min: "min", exp.Max: "max"}
|
||
func = kinds.get(type(node))
|
||
if func is None or node.args.get("expressions"):
|
||
return None
|
||
arg = node.this
|
||
distinct = isinstance(arg, exp.Distinct)
|
||
if distinct:
|
||
if len(arg.expressions) != 1:
|
||
return None
|
||
arg = arg.expressions[0]
|
||
if isinstance(arg, exp.Star):
|
||
arg = None
|
||
return func, arg, distinct
|
||
|
||
|
||
def _verb(func: str, distinct: bool) -> str:
|
||
return ("去重" if distinct else "") + _VERBS.get(func, f"用 {func.upper()} 汇总")
|
||
|
||
|
||
def _by_bound(conds: list[Any]) -> list[Any]:
|
||
"""时间条件给人看时下界在前:>=、> 在前,<、<= 在后,其余照原来的先后。"""
|
||
from sqlglot import exp
|
||
|
||
def rank(c: Any) -> int:
|
||
return 0 if isinstance(c, (exp.GTE, exp.GT)) else 1 if isinstance(c, (exp.LT, exp.LTE)) else 2
|
||
|
||
return sorted(conds, key=rank)
|
||
|
||
|
||
def _text_diff(a: _Form | None, b: _Form | None, read: str | None) -> tuple[str, ...]:
|
||
"""规范化之后两条 SQL 不同的行:difflib 比两边 sqlglot pretty 的输出,「- 」是 A 的、「+ 」是 B 的。只有一边有这一级
|
||
的规范形式时不比(只列一边的行读起来像另一边是空的),返回空的,调用方改给原文的逐行对照(见 _raw_diff)。
|
||
|
||
两边按同一套排版写(_diff_lines):WHERE、ON、HAVING 的合取式每个条件一行。sqlglot 的 pretty 按宽度决定换不换行,
|
||
两个条件放得下就排在一行、三个就拆开,同一个条件在两边落在不同的行上,就被当成「不同的行」(2026-10 验收:两边完全
|
||
相同的日期区间被标成了差异,审核人会以为统计期也不一样)。比的时候不看条件前面的 AND:一边多出来的条件排在最前面
|
||
时,另一边同一个条件只差在前面有没有 AND。列出来的仍是各自原样的行。只动排版,判不判等价、差异说明都不看这里。
|
||
|
||
不同的行前面带上它的各级上级行(_ancestors:排在前面、缩进比它浅的最近一行,一层层数到顶格),两边相同的那几行以
|
||
「 」开头列出,只为标出不同的行在哪个子句里——哪个 WHERE、哪个 JOIN 的 ON、哪层子查询。只列不同的行时,只差一个
|
||
关联条件的那一边只剩一行「AND 条件」,看不出是哪个 JOIN 的关联条件、还是筛选条件(2026-10 复核)。上级行在两边只差
|
||
前面有没有 AND 时,列不带 AND 的写法(两边共同的部分)。上级行本身不同的,照样是「- 」「+ 」行。"""
|
||
import difflib
|
||
|
||
if a is None or b is None:
|
||
return ()
|
||
la, lb = _diff_lines(a, read), _diff_lines(b, read)
|
||
ka, kb = [_LEADING_AND.sub(r"\1", x) for x in la], [_LEADING_AND.sub(r"\1", x) for x in lb]
|
||
# (标记, A 的行号, B 的行号):ndiff 按顺序走完两边,「 」两边各进一行,「- 」「+ 」各进自己那边一行,「? 」是行内
|
||
# 提示、不占行
|
||
rows: list[tuple[str, int, int]] = []
|
||
i = j = 0
|
||
for line in difflib.ndiff(ka, kb):
|
||
tag = line[:2]
|
||
if tag == " ":
|
||
rows.append((tag, i, j))
|
||
i, j = i + 1, j + 1
|
||
elif tag == "- ":
|
||
rows.append((tag, i, -1))
|
||
i += 1
|
||
elif tag == "+ ":
|
||
rows.append((tag, -1, j))
|
||
j += 1
|
||
need_a = _ancestors(la, [i for tag, i, _ in rows if tag == "- "])
|
||
need_b = _ancestors(lb, [j for tag, _, j in rows if tag == "+ "])
|
||
out: list[str] = []
|
||
for tag, i, j in rows:
|
||
if tag == "- ":
|
||
out.append(f"- {la[i]}")
|
||
elif tag == "+ ":
|
||
out.append(f"+ {lb[j]}")
|
||
elif i in need_a or j in need_b:
|
||
out.append(f" {la[i] if la[i] == lb[j] else ka[i]}")
|
||
out = out[:_MAX_DIFF_LINES]
|
||
# 截到 40 行时末尾剩下的上级行后面已经没有不同的行,不留:上级行只跟着不同的行出现
|
||
while out and out[-1].startswith(" "):
|
||
out.pop()
|
||
return tuple(out)
|
||
|
||
|
||
def _ancestors(lines: list[str], changed: list[int]) -> set[int]:
|
||
"""changed 这些行的各级上级行的行号:排在前面、缩进比它浅的最近一行是它的上级,一层层往上数到顶格。sqlglot 的
|
||
pretty 里子句的内容缩在子句名下面(WHERE 下的条件、JOIN … ON 下的关联条件、EXISTS( 下的子查询),所以数出来的就是
|
||
它所在的各层子句。空白行不当上级。"""
|
||
def depth(line: str) -> int:
|
||
return len(line) - len(line.lstrip(" "))
|
||
|
||
parent = [-1] * len(lines)
|
||
# 往上数时还可能当上级的行:从上往下缩进一行比一行深,比当前行深或一样深的都不是它的上级
|
||
stack: list[int] = []
|
||
for n, line in enumerate(lines):
|
||
if not line.strip():
|
||
continue
|
||
while stack and depth(lines[stack[-1]]) >= depth(line):
|
||
stack.pop()
|
||
parent[n] = stack[-1] if stack else -1
|
||
stack.append(n)
|
||
out: set[int] = set()
|
||
for n in changed:
|
||
k = parent[n]
|
||
# 已经在里面的,它往上的各级也已经在里面了
|
||
while k >= 0 and k not in out:
|
||
out.add(k)
|
||
k = parent[k]
|
||
return out
|
||
|
||
|
||
def _diff_lines(form: _Form, read: str | None) -> list[str]:
|
||
"""逐行对照用的写法:sqlglot pretty,WHERE、HAVING、ON 的合取式每个条件一行,ON 挂在 JOIN 那一行的末尾(见
|
||
_line_generator)。生成不了就退回普通的 pretty,再不行用规范形式的原文。"""
|
||
try:
|
||
text = _line_generator(read).generate(form.tree)
|
||
except Exception: # noqa: BLE001 - 只是给人看的排版,出错不能连累差异说明
|
||
try:
|
||
text = form.tree.sql(dialect=read, pretty=True, comments=False)
|
||
except Exception: # noqa: BLE001
|
||
text = form.text
|
||
return [line.rstrip() for line in text.splitlines()]
|
||
|
||
|
||
def _condition_root(node: Any) -> str | None:
|
||
"""node 是 WHERE、HAVING 的整个条件("clause")、JOIN 的 ON 条件("on"),还是别处的(None)。"""
|
||
from sqlglot import exp
|
||
|
||
parent = node.parent
|
||
if isinstance(parent, (exp.Where, exp.Having)) and node.arg_key == "this":
|
||
return "clause"
|
||
if isinstance(parent, exp.Join) and node.arg_key == "on":
|
||
return "on"
|
||
return None
|
||
|
||
|
||
def _line_generator(read: str | None) -> Any:
|
||
"""这种方言的 sqlglot 生成器(pretty、不带注释),只改条件的排法:WHERE、HAVING、ON 的整个条件按 AND 拆开,每个条件
|
||
一行,第二个起前面写 AND(各层子查询里的也一样),ON 挂在 JOIN 那一行的末尾。别的照 sqlglot 原样:条件本身太长时
|
||
照样折行,两边同一个条件折得一样。
|
||
|
||
- 条件本身是 OR(XOR)连起来的一律加括号:和别的条件并列时本来就要加,单独一个时不加的话,同一个条件两边写法不同。
|
||
条件外面多余的括号、AND 套 AND 的括号去掉(_conjuncts),不改意思。
|
||
- ON 挂在 JOIN 那一行的末尾(表名、子查询的右括号后面),条件各占一行缩在下面:写在 ON 后面的话,第一个条件和后面
|
||
的差在「ON」「AND」上,换个先后就对不上;缩进比 WHERE 的条件深一级,ON 里和 WHERE 里的同一个条件不会被当成同一
|
||
行。ON 不单独占一行:那样两边 JOIN 的表不同时,两边都有的「ON」被当成相同的行,条件前面就没了 ON(2026-10 复核)。
|
||
"""
|
||
from sqlglot import exp
|
||
from sqlglot.dialects.dialect import Dialect
|
||
|
||
dialect = Dialect.get_or_raise(read)
|
||
base = dialect.generator_class
|
||
cls = _LINE_GENERATORS.get(base)
|
||
if cls is None:
|
||
class _ConditionLines(base): # type: ignore[misc, valid-type]
|
||
def sql(self, expression: Any, key: str | None = None, comment: bool = True) -> str:
|
||
root = _condition_root(expression) if key is None and isinstance(expression, exp.Expr) else None
|
||
conds = _conjuncts(expression) if root else []
|
||
if not conds:
|
||
return super().sql(expression, key, comment)
|
||
lines = [self._one_condition(c) for c in conds]
|
||
text = "\n".join([lines[0], *(f"AND {line}" for line in lines[1:])])
|
||
# ON:join_sql 拼成「ON <条件>」并给条件的各行加一级缩进,开头换行就是条件都在 ON 下面;单独一行的 ON
|
||
# 再由下面的 join_sql 挂到上一行末尾
|
||
return "\n" + self.indent(text) if root == "on" else text
|
||
|
||
def join_sql(self, expression: Any) -> str:
|
||
# sqlglot 拼出来是换行、缩进一级、「ON 」再换行(ON 单独一行,条件在下面):把这个 ON 接到上一行(表名、
|
||
# 子查询的右括号)末尾。只认这一层的:子查询里、条件里套着的 JOIN 都缩得更深,对不上
|
||
pad = " " * self.pad
|
||
return super().join_sql(expression).replace(f"\n{pad}ON \n", " ON\n", 1)
|
||
|
||
def _one_condition(self, node: Any) -> str:
|
||
# 只有一个条件时它就是整个条件,再走 self.sql 又会被认成整个条件、绕不出来:这一层直接用 sqlglot 原来的
|
||
# 写法(里面各层子查询的条件照样经过上面的 sql)
|
||
text = super().sql(node)
|
||
if not isinstance(node, exp.Connector):
|
||
return text
|
||
if "\n" not in text:
|
||
return f"({text})"
|
||
return f"({self.seg(self.indent(text), sep='')}{self.seg(')', sep='')}"
|
||
|
||
cls = _LINE_GENERATORS[base] = _ConditionLines
|
||
return cls(dialect=dialect, pretty=True, comments=False)
|
||
|
||
|
||
def _raw_diff(a: str, b: str) -> tuple[str, ...]:
|
||
"""原文的逐行对照(没有规范形式可比时用:解析时会读错、丢掉写法的,见 _lost_in_parsing):difflib 比两边原文的行
|
||
(去掉行尾空白),「- 」是 A 的、「+ 」是 B 的;比不出不同的行(只差在行内空白)就整条各列一行。"""
|
||
import difflib
|
||
|
||
def lines(text: str) -> list[str]:
|
||
return [line.rstrip() for line in text.strip().splitlines() if line.strip()]
|
||
|
||
out = [line for line in difflib.ndiff(lines(a), lines(b)) if line[:2] in ("- ", "+ ")]
|
||
return tuple(out[:_MAX_DIFF_LINES]) or (f"- {a.strip()}", f"+ {b.strip()}")
|
||
|
||
|
||
def explain_sql_diff(a: str, b: str, dialect: str | None = None, *, schema: Mapping[str, Any] | None = None,
|
||
names: Mapping[str, str] | None = None, labels: tuple[str, str] = ("A", "B")) -> SqlDiff:
|
||
"""两条 SQL 是不是等价、不是的话差在哪。从不抛异常。
|
||
|
||
等价怎么判:两边各自规范化(见 _canonical_forms:认列之后的那一级、不认列的轻量一级、只统一名字的一级),每一步都
|
||
只做保持语义的变换;同一级的两个规范形式完全一样(比语法树,见 _Form.key)才算等价,否则一律不等价,items 一定
|
||
不为空。schema 是数据源的 schema_cache,给了就按它认列、展开 SELECT *(表结构要和库对得上);里面的约束(主键、
|
||
唯一约束、外键、NOT NULL)不拿来判等价,只拿来让说明少说。原文里有 sqlglot 会读错、会丢掉的写法的(MySQL 字符串
|
||
里的反斜杠转义、1_000、没加括号地混用的比较和按位运算、PostgreSQL 写成符号的 ~ / ~~ / @@ 和 || 混用、前缀的 @、XOR、
|
||
一元加号、0x10、换了名字的类型名,不知道是哪种库时套着的块注释、紧跟着字的 --、双引号括起来的名字……,见
|
||
_lost_in_parsing、_misread)不在语法树上判等价,只给兜底和原文的逐行对照。
|
||
|
||
不等价时怎么说:两边的规范形式都是简单形状时(一段 SELECT、实表、普通的连接,见 _simple_block),只把四类有把握、
|
||
真会影响结果的不同写成具体的中文句子(按哪一列筛时间、一边多出来的筛选条件、顶层的聚合函数和去重、关联的表和关联
|
||
类型,见 _Explainer);具体说法一条都没有时给一条 aspect="other" 的兜底说明(「两条 SQL 的写法不同,未能归纳成具体
|
||
差异,请对照原文」),只覆盖了一部分时追加一条「另有未能归纳的差异」,fallback 为真;text_diff 给出规范化之后不同
|
||
的行(没有规范形式可比时给原文的逐行对照)。
|
||
|
||
labels 是两边的叫法(缺省 A / B,双盲复算传「原流程」「盲算」);names 是列的中文名({「表.列」或「列」: 中文名},
|
||
可以用 catalog_labels 从数据目录取),时间列写成「支付时间 paid_at」(两边的时间列在不同的表上时写「表.列」)。任一边
|
||
解析不了:只给一条 aspect="parse"。
|
||
"""
|
||
la, lb = labels
|
||
read = _read(dialect)
|
||
if read == "mysql":
|
||
a, b = (_mysql_dashes(x) if isinstance(x, str) else x for x in (a, b))
|
||
ra, ea = _parse(a, read)
|
||
rb, eb = _parse(b, read)
|
||
if ea or eb:
|
||
if ea and eb:
|
||
text = _say(la, ":", ea, ";", lb, ":", eb, ";没有逐项比较")
|
||
else:
|
||
side, error = (la, ea) if ea else (lb, eb)
|
||
text = _say(side, ":", error or "", ",没有逐项比较")
|
||
return SqlDiff(False, [FacetDiff("parse", ea, eb, text)])
|
||
if a.strip() == b.strip():
|
||
# 一字不差就是同一条语句(带会执行的注释、RANDOM() 这些不在语法树上判等价的写法也一样)
|
||
return SqlDiff(True, level=_PARSED)
|
||
info = _schema_info(schema)
|
||
forms_a: dict[str, _Form] = {}
|
||
forms_b: dict[str, _Form] = {}
|
||
try:
|
||
marked_a = _mark_surfaces(ra, a.strip(), read)
|
||
marked_b = _mark_surfaces(rb, b.strip(), read)
|
||
if read == "sqlite":
|
||
_mark_comma_joins(ra, a.strip(), read)
|
||
_mark_comma_joins(rb, b.strip(), read)
|
||
forms_a = {} if not marked_a or _lost_in_parsing(a.strip(), read, ra) else _canonical_forms(ra, read, info)
|
||
forms_b = {} if not marked_b or _lost_in_parsing(b.strip(), read, rb) else _canonical_forms(rb, read, info)
|
||
except Exception: # noqa: BLE001 - 规范化本身的缺陷不能连累调用方:当作说不清,记下来修
|
||
logger.warning("SQL 规范化失败", exc_info=True)
|
||
forms_a, forms_b = {}, {}
|
||
for level in (_OPTIMIZER, _LIGHT, _PARSED):
|
||
fa, fb = forms_a.get(level), forms_b.get(level)
|
||
if fa is not None and fb is not None and fa.key == fb.key:
|
||
return SqlDiff(True, level=level)
|
||
level = next((lv for lv in (_OPTIMIZER, _LIGHT) if lv in forms_a and lv in forms_b), None)
|
||
items: list[FacetDiff] = []
|
||
covered = False
|
||
if level is not None:
|
||
explainer = _Explainer(read, (la, lb), names, info)
|
||
try:
|
||
covered = explainer.statement(forms_a[level], forms_b[level], (ra, rb))
|
||
items = explainer.items
|
||
except Exception: # noqa: BLE001 - 说明本身的缺陷不能连累调用方:当作说不清,记下来修
|
||
logger.warning("SQL 差异说明失败", exc_info=True)
|
||
items, covered = [], False
|
||
shown = level or _PARSED
|
||
if not items:
|
||
items = [FacetDiff("other", None, None, _FALLBACK)]
|
||
fallback = True
|
||
elif not covered:
|
||
items = items + [FacetDiff("other", None, None, _FALLBACK_REST)]
|
||
fallback = True
|
||
else:
|
||
fallback = False
|
||
text_diff = _text_diff(forms_a.get(shown), forms_b.get(shown), read) or _raw_diff(a, b)
|
||
return SqlDiff(False, items, fallback, text_diff, level)
|
||
|
||
|
||
def diff_facets(a: Facets, b: Facets, labels: tuple[str, str] = ("A", "B"), *,
|
||
names: Mapping[str, str] | None = None) -> list[FacetDiff]:
|
||
"""两份 facets 记下的原文交给 explain_sql_diff 比,返回它的 items(含兜底条目):完全一致(规范形式一字不差)
|
||
才返回空列表,不等价时一定不为空。
|
||
|
||
labels、names 见 explain_sql_diff。任一边解析不了:只返回一条 parse。不是 facets 算出来的 Facets(没记原文)
|
||
比不了原文:同一个对象才返回空列表,否则返回一条兜底说明(facets 记的结构不全,两份一样不说明 SQL 一样)。
|
||
"""
|
||
la, lb = labels
|
||
if a.parse_error or b.parse_error:
|
||
if a.parse_error and b.parse_error:
|
||
text = _say(la, ":", a.parse_error, ";", lb, ":", b.parse_error, ";没有逐项比较")
|
||
else:
|
||
side, error = (la, a.parse_error) if a.parse_error else (lb, b.parse_error)
|
||
text = _say(side, ":", error or "", ",没有逐项比较")
|
||
return [FacetDiff("parse", a.parse_error, b.parse_error, text)]
|
||
if a._origin is None or b._origin is None:
|
||
return [] if a is b else [FacetDiff("other", None, None, _FALLBACK)]
|
||
(sql_a, schema_a), (sql_b, schema_b) = a._origin, b._origin
|
||
return explain_sql_diff(sql_a, sql_b, a.dialect or b.dialect, schema=schema_a or schema_b, names=names,
|
||
labels=labels).items
|
||
|
||
|
||
__all__ = ["MAX_SQL_CHARS", "PLACEHOLDER", "Aggregate", "ColumnRef", "Computed", "FacetDiff", "Facets", "JoinFacet",
|
||
"LimitFacet", "NormalizedSql", "Part", "SqlDiff", "TimePredicate", "catalog_labels", "diff_facets",
|
||
"explain_sql_diff", "facets", "normalize_sql"]
|