ThearchyHelios
1a6082f033
大库的对象索引、探查范围和后台探查(next-round 03 A1),schema_view 预筛与按需加入、剖析事实的后端(A2、遗留 e 的底座)
为什么:几千张表的库以前用不了——探查每个源最多取 200 个对象、按数据库列出的顺序截断,同步请求、整份替换、不排队;
落在窗口外的表在目录、look、剖析里被说成「表已不在」。
做了什么:
- db/models_sources.py:source_objects(对象索引一行一个对象:范围、原因、结构指纹与 struct_version、first_seen / last_seen /
removed_at、reflected_at、按需加入的留痕)、introspect_jobs / introspect_tasks(后台探查任务与逐表状态)。
- data/source_objects.py:每种方言一条字典查询建对象索引(失败退回 Inspector,上限 5 万,截断时不判删除);探查范围规则
(options.introspect_scope:通配、按 name_key、排除优先、显式列出 / 移出;有目录的表、已发布模板写死 SQL 用到的表 pinned;
命中超上限拒绝并说出命中数);唯一的 rebuild_schema_cache 从范围内已反射的行物化 schema_cache(形状不变,另加 scope 摘要);
三种「不在」分开说(removed / missing / out_of_scope / reflect_failed);按需加入(当场只读反射、落盘、留痕、进程内按源加锁)。
- data/introspect_jobs.py:探查改成后台任务(core/job_runtime,lane kind introspect,和同源起草、剖析串行;按批反射、逐表记状态、
可停止、读完的照样物化;重启标中断不续跑)。
- api/datasources.py:POST /introspect 返回 202 和任务,dry_run 同步只刷新索引、返回命中数;/introspect-jobs 读任务、停止;
/introspect-scope 读、预览、保存规则;/schema?table= 带剖析事实、说清三种「不在」;DataSourceOut 加 schema_scope、introspect_job。
- schema_view:search_tables(BM25 倒排,遮罩列只用列名,范围外只按名字和注释、权重低一档)、profile_lines(剖析事实一节,
附「只供写查询参考」,遮罩按渲染那一刻重滤,上传表格快照不一致时不给)、absent_detail。
- look:候选超过 PREFILTER_LIMIT 时只给预筛结果 + 已用到的表 + 一跳关联;点名或挑中范围外的表按需加入(每次查看最多 4 张),
look 记录加 added_to_scope、prefiltered;表详情带剖析事实。
- tools/datasource:db_schema__ 加 keyword、不传 table 时只列前 50 张;点名范围外的表按需加入(每次运行最多 8 张);查询结果
照实列「未按目录检查的表」(unchecked_tables,不新增检查规则);建工具时读剖析事实。
- 剖析:_relation_ends 分清「范围外,可以加入」和「表结构里没有」;任务记上传表格的 snapshot_id;last_facts。表助手、/schema
用同一段剖析事实。
- 目录接口:清单只列范围内,absent_reason 分开说;GET /catalog/search 查对象索引,POST /catalog/scope/add 加入范围。
- 第 1 波留下的一处:check_valid_filter 和 stored_valid_filter 拼有效记录条件时按 quote_name 强制加引号。
- 工具参数纠错:空着的可选参数不止一个时,只认拼写明显接近的那一个(db_schema 多了 keyword 以后 tabel 仍能纠成 table)。
- 测试:3,000 张表的合成库(对象索引、缺省范围超上限拒绝、规则圈 400 张后台读完、停止、重启标中断、和剖析排队、同步起草
409)、删除与结构版本、Inspector 回退、截断不判删除、范围规则、pinned、方言金样;改了同步探查的老测试改为等任务做完。
2026-10-16 14:17:06 +08:00
..
2026-10-15 07:15:37 +08:00
2026-10-16 14:17:06 +08:00
2026-10-15 12:09:29 +08:00
2026-10-16 12:46:48 +08:00
2026-10-04 23:40:20 +08:00
2026-10-06 11:44:58 +08:00
2026-10-14 21:55:51 +08:00
2026-09-30 08:05:15 +08:00
2026-09-30 08:05:15 +08:00
2026-09-30 08:05:15 +08:00
2026-09-25 15:20:46 +08:00
2026-09-30 08:05:15 +08:00
2026-10-14 21:55:51 +08:00
2026-10-14 21:55:51 +08:00
2026-10-15 12:09:29 +08:00
2026-10-14 21:55:51 +08:00
2026-10-15 07:15:37 +08:00
2026-10-16 10:14:27 +08:00
2026-10-15 07:15:37 +08:00
2026-10-14 21:55:51 +08:00
2026-10-15 08:35:28 +08:00
2026-10-15 12:09:29 +08:00
2026-10-15 13:55:19 +08:00
2026-10-15 08:35:28 +08:00
2026-09-29 05:26:34 +08:00
2026-10-04 12:03:14 +08:00
2026-10-16 10:14:27 +08:00
2026-10-15 08:35:28 +08:00
2026-10-15 07:15:37 +08:00
2026-10-14 20:05:23 +08:00
2026-09-30 08:05:15 +08:00
2026-09-30 08:05:15 +08:00
2026-09-30 08:05:15 +08:00
2026-10-15 14:32:25 +08:00
2026-10-15 13:55:19 +08:00
2026-10-15 17:31:20 +08:00
2026-10-15 13:55:19 +08:00
2026-10-15 13:55:19 +08:00
2026-10-16 10:34:18 +08:00
2026-10-16 10:14:27 +08:00
2026-09-30 09:48:16 +08:00
2026-10-16 12:46:48 +08:00
2026-10-16 12:46:48 +08:00
2026-10-12 17:23:27 +08:00
2026-10-12 19:09:50 +08:00
2026-10-06 19:21:23 +08:00
2026-10-12 19:09:50 +08:00
2026-10-16 12:46:48 +08:00
2026-10-12 23:01:09 +08:00
2026-09-17 18:29:16 +08:00
2026-10-14 22:08:33 +08:00
2026-10-15 14:44:26 +08:00
2026-10-14 21:55:51 +08:00
2026-10-16 14:17:06 +08:00
2026-10-16 14:17:06 +08:00
2026-10-14 09:59:19 +08:00
2026-10-16 10:25:28 +08:00
2026-10-14 14:07:14 +08:00
2026-10-14 22:17:00 +08:00
2026-10-13 20:30:02 +08:00
2026-10-14 20:05:23 +08:00
2026-10-13 20:30:02 +08:00
2026-10-16 10:45:45 +08:00
2026-10-15 19:43:29 +08:00
2026-10-16 10:25:28 +08:00
2026-10-16 14:17:06 +08:00
2026-10-04 01:07:15 +08:00
2026-10-14 16:11:04 +08:00
2026-10-04 02:00:17 +08:00
2026-10-14 09:51:49 +08:00
2026-10-03 11:09:09 +08:00
2026-10-16 14:17:06 +08:00
2026-10-16 12:02:41 +08:00
2026-10-04 02:19:26 +08:00
2026-10-03 12:14:17 +08:00
2026-10-14 16:11:04 +08:00
2026-10-16 10:28:01 +08:00
2026-10-07 01:09:36 +08:00
2026-10-14 20:05:23 +08:00
2026-10-12 15:55:19 +08:00
2026-10-16 10:59:15 +08:00
2026-09-29 11:10:53 +08:00
2026-09-30 08:05:15 +08:00
2026-09-24 23:15:19 +08:00
2026-10-06 16:42:29 +08:00
2026-10-05 16:41:59 +08:00
2026-10-05 16:41:59 +08:00
2026-09-30 08:05:15 +08:00
2026-09-30 08:05:15 +08:00
2026-09-30 08:05:15 +08:00
2026-09-30 08:05:15 +08:00
2026-10-06 19:51:29 +08:00
2026-09-30 10:04:52 +08:00
2026-10-15 13:55:19 +08:00
2026-10-16 12:46:36 +08:00
2026-10-14 21:55:51 +08:00
2026-10-16 10:45:45 +08:00
2026-10-16 10:59:15 +08:00
2026-10-04 11:07:40 +08:00
2026-10-15 13:55:19 +08:00
2026-10-05 15:08:53 +08:00
2026-10-04 11:10:28 +08:00
2026-10-14 20:05:23 +08:00
2026-09-29 11:10:53 +08:00
2026-10-16 10:45:45 +08:00
2026-10-16 10:45:45 +08:00
2026-10-16 10:45:45 +08:00
2026-10-16 10:45:45 +08:00
2026-10-16 10:45:45 +08:00
2026-10-16 10:45:45 +08:00
2026-10-16 10:45:45 +08:00
2026-10-14 20:05:23 +08:00
2026-10-16 10:45:45 +08:00
2026-09-25 15:41:30 +08:00
2026-10-15 14:59:40 +08:00
2026-10-16 10:45:45 +08:00
2026-10-16 12:23:19 +08:00
2026-09-27 10:16:32 +08:00
2026-09-30 08:05:15 +08:00
2026-10-05 00:26:04 +08:00
2026-10-16 14:17:06 +08:00
2026-09-30 08:05:15 +08:00
2026-10-02 07:38:44 +08:00
2026-10-02 14:38:10 +08:00
2026-10-16 10:24:22 +08:00
2026-10-05 15:54:03 +08:00
2026-10-05 15:54:03 +08:00
2026-10-14 15:38:15 +08:00
2026-10-07 01:34:59 +08:00
2026-10-16 10:14:27 +08:00
2026-10-12 12:24:14 +08:00
2026-10-06 18:03:12 +08:00
2026-09-18 09:39:24 +08:00
2026-10-15 07:15:37 +08:00
2026-10-01 13:42:04 +08:00
2026-10-16 10:18:12 +08:00
2026-09-28 11:47:23 +08:00
2026-09-30 08:05:15 +08:00
2026-10-06 12:02:03 +08:00
2026-09-30 08:05:15 +08:00
2026-09-30 08:05:15 +08:00
2026-10-06 12:58:37 +08:00
2026-10-16 11:04:41 +08:00
2026-09-29 05:26:34 +08:00
2026-09-29 05:26:34 +08:00
2026-09-30 08:05:15 +08:00
2026-10-12 13:50:31 +08:00
2026-10-12 17:23:27 +08:00
2026-10-12 13:50:31 +08:00
2026-10-12 17:23:27 +08:00
2026-10-12 10:48:19 +08:00
2026-10-04 02:35:33 +08:00
2026-09-30 08:05:15 +08:00
2026-09-30 08:05:15 +08:00
2026-10-12 12:24:14 +08:00
2026-09-30 15:03:31 +08:00
2026-10-01 13:42:04 +08:00
2026-09-30 08:05:15 +08:00
2026-10-06 11:37:22 +08:00
2026-10-01 13:42:04 +08:00
2026-09-30 08:05:15 +08:00
2026-10-04 11:02:06 +08:00
2026-09-25 14:33:56 +08:00
2026-10-15 13:55:19 +08:00
2026-10-02 07:38:44 +08:00
2026-10-15 13:55:19 +08:00
2026-09-30 08:05:15 +08:00
2026-09-30 10:04:52 +08:00
2026-09-30 08:05:15 +08:00
2026-09-27 10:16:32 +08:00
2026-09-30 08:05:15 +08:00
2026-09-27 10:16:32 +08:00
2026-10-12 15:55:19 +08:00
2026-10-07 01:09:36 +08:00
2026-10-06 22:52:28 +08:00
2026-10-12 15:55:19 +08:00
2026-10-05 01:18:03 +08:00
2026-10-16 10:33:41 +08:00
2026-10-01 13:42:04 +08:00
2026-10-03 09:30:50 +08:00
2026-10-16 14:17:06 +08:00
2026-09-28 11:47:23 +08:00
2026-09-28 11:47:23 +08:00
2026-10-05 18:45:59 +08:00
2026-09-25 15:30:32 +08:00
2026-10-16 09:59:44 +08:00
2026-09-29 16:07:19 +08:00
2026-10-12 13:50:31 +08:00
2026-10-02 14:38:10 +08:00
2026-09-30 10:04:52 +08:00
2026-09-30 10:04:52 +08:00
2026-09-30 10:04:52 +08:00
2026-10-06 12:58:37 +08:00
2026-10-06 14:30:15 +08:00
2026-09-27 10:16:32 +08:00
2026-09-28 11:47:45 +08:00
2026-09-25 12:47:07 +08:00
2026-09-30 08:05:15 +08:00
2026-10-14 20:05:23 +08:00
2026-09-30 08:05:15 +08:00
2026-09-30 08:05:15 +08:00
2026-10-12 10:48:19 +08:00
2026-10-12 10:48:19 +08:00
2026-10-07 01:09:36 +08:00
2026-10-12 13:14:36 +08:00
2026-10-06 19:11:14 +08:00
2026-10-06 21:50:48 +08:00
2026-10-12 10:50:40 +08:00
2026-10-13 16:59:58 +08:00
2026-10-12 12:18:34 +08:00
2026-10-07 01:09:36 +08:00
2026-10-14 20:05:23 +08:00
2026-10-04 01:08:42 +08:00
2026-10-04 02:17:32 +08:00
2026-10-03 09:36:20 +08:00
2026-10-03 10:40:47 +08:00
2026-09-28 11:47:23 +08:00
2026-10-15 07:15:37 +08:00
2026-10-05 03:26:43 +08:00
2026-10-15 07:15:37 +08:00
2026-09-30 11:19:25 +08:00
2026-10-06 14:29:48 +08:00
2026-09-30 08:05:15 +08:00
2026-10-01 13:42:04 +08:00
2026-09-25 23:14:50 +08:00
2026-09-30 08:05:15 +08:00
2026-10-14 20:05:23 +08:00
2026-10-16 12:12:14 +08:00
2026-10-16 10:14:27 +08:00
2026-10-04 12:27:38 +08:00
2026-10-02 07:38:44 +08:00
2026-10-16 10:18:12 +08:00
2026-10-15 12:09:29 +08:00
2026-10-15 07:15:37 +08:00
2026-10-16 10:14:27 +08:00
2026-10-14 22:08:33 +08:00
2026-10-16 10:14:27 +08:00
2026-10-16 12:02:41 +08:00
2026-10-16 12:06:24 +08:00
2026-10-02 14:38:10 +08:00
2026-10-02 14:38:10 +08:00
2026-10-03 09:36:20 +08:00
2026-10-02 14:38:10 +08:00
2026-10-12 10:17:40 +08:00
2026-10-02 14:38:10 +08:00
2026-10-06 19:43:44 +08:00
2026-10-06 19:43:44 +08:00
2026-10-06 17:29:18 +08:00
2026-10-06 19:43:44 +08:00
2026-10-06 19:43:44 +08:00
2026-10-16 10:18:12 +08:00
2026-09-30 08:05:15 +08:00
2026-10-16 10:34:18 +08:00
2026-10-04 02:52:43 +08:00
2026-09-28 21:20:04 +08:00
2026-09-30 08:05:15 +08:00
2026-09-25 14:44:56 +08:00
2026-10-02 07:38:44 +08:00
2026-10-02 07:38:44 +08:00
2026-10-02 07:38:44 +08:00
2026-10-01 22:00:07 +08:00
2026-10-01 22:00:07 +08:00
2026-10-02 07:38:44 +08:00
2026-10-02 07:38:44 +08:00
2026-10-02 07:38:44 +08:00
2026-10-16 11:16:33 +08:00
2026-10-02 07:38:44 +08:00
2026-10-01 22:00:07 +08:00
2026-10-02 07:38:44 +08:00
2026-10-02 14:38:10 +08:00
2026-10-02 07:38:44 +08:00
2026-10-16 11:16:33 +08:00
2026-10-16 11:16:33 +08:00
2026-10-02 07:38:44 +08:00
2026-10-02 07:38:44 +08:00
2026-10-16 12:23:19 +08:00
2026-10-01 22:00:07 +08:00
2026-10-16 11:09:39 +08:00
2026-10-02 07:38:44 +08:00
2026-10-16 11:09:39 +08:00
2026-10-16 12:23:19 +08:00
2026-10-02 07:38:44 +08:00
2026-10-16 11:09:39 +08:00
2026-10-16 11:09:39 +08:00
2026-10-16 11:09:39 +08:00
2026-10-16 11:09:39 +08:00
2026-10-05 18:50:34 +08:00
2026-10-06 09:20:09 +08:00
2026-10-16 10:14:27 +08:00
2026-10-14 20:05:23 +08:00
2026-10-04 11:28:23 +08:00
2026-10-04 12:28:07 +08:00
2026-09-30 08:05:15 +08:00
2026-09-30 08:05:15 +08:00
2026-10-04 11:33:09 +08:00
2026-10-12 14:26:43 +08:00
2026-10-13 10:04:30 +08:00
2026-10-10 12:59:21 +08:00
2026-10-06 13:32:38 +08:00
2026-09-30 10:04:52 +08:00
2026-10-12 13:50:31 +08:00
2026-10-12 23:01:09 +08:00
2026-10-12 12:33:32 +08:00
2026-10-12 23:01:09 +08:00
2026-10-12 23:01:09 +08:00
2026-10-12 19:09:50 +08:00
2026-10-12 17:23:27 +08:00
2026-10-12 13:50:31 +08:00
2026-10-12 17:23:27 +08:00
2026-09-30 15:03:31 +08:00
2026-09-30 08:05:15 +08:00
2026-10-04 02:24:49 +08:00
2026-10-04 02:33:15 +08:00
2026-09-30 08:05:15 +08:00
2026-10-06 09:31:00 +08:00
2026-09-30 08:05:15 +08:00
2026-10-05 01:18:03 +08:00
2026-10-05 18:45:59 +08:00
2026-10-06 14:32:30 +08:00
2026-09-26 23:11:31 +08:00
2026-10-14 22:08:33 +08:00
2026-10-05 00:07:07 +08:00
2026-09-30 08:05:15 +08:00
2026-10-16 12:46:35 +08:00
2026-09-30 08:05:15 +08:00
2026-09-30 08:05:15 +08:00
2026-10-14 20:05:23 +08:00
2026-09-27 10:16:32 +08:00
2026-10-14 20:05:23 +08:00
2026-10-14 21:55:51 +08:00
2026-10-04 01:52:24 +08:00
2026-09-27 23:37:32 +08:00
2026-10-07 00:02:15 +08:00
2026-10-16 10:07:15 +08:00
2026-10-06 08:48:56 +08:00
2026-10-06 16:42:29 +08:00
2026-10-05 08:50:57 +08:00
2026-10-05 08:37:57 +08:00
2026-10-15 07:15:37 +08:00
2026-10-05 18:39:10 +08:00
2026-10-06 08:48:56 +08:00
2026-10-16 10:14:27 +08:00
2026-10-15 07:15:37 +08:00
2026-09-30 08:05:15 +08:00
2026-10-15 12:09:29 +08:00
2026-10-05 08:37:57 +08:00
2026-09-25 16:05:14 +08:00
2026-10-16 10:54:30 +08:00
2026-10-02 07:38:44 +08:00
2026-10-16 11:04:41 +08:00
2026-10-06 08:10:41 +08:00
2026-09-29 05:26:34 +08:00
2026-09-25 14:44:56 +08:00
2026-10-07 01:34:59 +08:00
2026-10-07 01:34:59 +08:00
2026-10-16 12:53:07 +08:00
2026-10-01 13:42:04 +08:00
2026-09-30 08:05:15 +08:00
2026-09-30 08:05:15 +08:00
2026-10-02 07:38:44 +08:00
2026-10-02 07:38:44 +08:00
2026-10-02 07:38:44 +08:00
2026-10-02 07:38:44 +08:00
2026-10-02 07:38:44 +08:00
2026-10-01 22:00:07 +08:00
2026-10-14 21:55:51 +08:00
2026-09-30 08:05:15 +08:00
2026-09-26 23:11:31 +08:00
2026-10-15 07:15:37 +08:00
2026-09-28 21:20:04 +08:00
2026-09-18 17:57:20 +08:00
2026-10-07 00:02:15 +08:00
2026-10-14 20:05:23 +08:00
2026-10-15 07:15:37 +08:00
2026-10-14 21:55:51 +08:00
2026-09-30 08:05:15 +08:00
2026-09-30 08:05:15 +08:00
2026-10-03 11:09:09 +08:00
2026-09-30 08:05:15 +08:00
2026-09-30 08:05:15 +08:00
2026-10-16 10:45:45 +08:00
2026-10-15 13:55:19 +08:00
2026-09-27 10:16:32 +08:00
2026-09-30 08:05:15 +08:00
2026-09-30 08:05:15 +08:00
2026-10-06 21:50:48 +08:00
2026-10-05 01:46:55 +08:00
2026-10-06 14:32:30 +08:00
2026-10-05 18:45:44 +08:00
2026-10-12 12:24:14 +08:00
2026-10-07 00:02:15 +08:00
2026-10-14 22:17:00 +08:00
2026-10-07 01:19:26 +08:00
2026-10-05 00:06:03 +08:00
2026-10-06 14:30:03 +08:00
2026-09-29 05:26:34 +08:00
2026-09-30 09:48:16 +08:00
2026-10-16 10:17:37 +08:00
2026-10-15 07:15:37 +08:00
2026-10-12 17:23:27 +08:00
2026-10-05 06:49:33 +08:00
2026-09-26 23:11:31 +08:00
2026-10-14 21:55:51 +08:00
2026-09-28 02:42:40 +08:00
2026-10-06 23:56:31 +08:00
2026-10-01 22:00:07 +08:00
2026-10-01 13:42:04 +08:00