合入集成分支(阶段 2A 数据剖析、4B 剖析与 SQL 检查界面及修正)到 4A:两边功能都保留

冲突的解法:
- 后端 api/catalog.py:同时导入 catalog_impact 和 catalog_profile、SnapshotTampered;data/catalog.py:__all__ 取两边的并集重排。
- 2A 放宽了码值校验(含义可以为空,剖析写进来的是「含义待填写」):目录修改提案的码值补充改为含义为空的码不覆盖已有含义、
  只在原来没有这个码时加进去,补了测试;卡片和过程里的码值把空含义写成「含义待填写」,和目录页同一个说法。
- 前端 types.ts、terms.ts:两边各在同一处追加了新段落,按顺序都留下(提案、影响面、目录版本在前,剖析、码值、SQL 检查在后),
  没有重复的键;client.ts、decode.ts、CatalogPage.tsx、TableDetail.tsx 的导入取并集。
- check-catalog.mjs:影响面和剖析、保存数据源三条假路由都留下,各节和最后核对真库数量的逻辑不变。
- 4A 的测试数据源改用 tests/fixtures/sources.py 的 drop_source 删除,顺带作废连接池。
This commit is contained in:
ThearchyHelios
2026-10-03 12:35:47 +08:00
51 changed files with 4803 additions and 147 deletions
+65 -2
View File
@@ -8,9 +8,10 @@
- POST /catalog/{table}/patch 保存目录修改提案(乐观锁),记为人工确认
- GET /catalog/{table}/impact 影响面:引用这张表的已发布、受管模板(data/catalog_impact.py)
- POST /catalog/draft 同步起草(注释、外键、命名推断,可选模型)
- POST /catalog/profile 同步剖析:对业务库发少量只读查询,核对关系、取码值候选(data/catalog_profile.py)
规则都在 data/catalog.py,这里只做取源、转换形状和把异常翻成状态码:版本不符 409,结构不合规、审阅路径
不对 422,源或表不存在 404。所有写操作记录请求头 X-Actor 的署名(自报、未认证)。
不对 422,源或表不存在 404。所有写操作记录请求头 X-Actor 的署名(自报、未认证)。剖析没开启、正在剖析 409。
上传表格的源按当前快照的表结构回答(table_versions.resolve_source),和查询实际用的库是同一个版本。
"""
@@ -23,7 +24,8 @@ from pydantic import BaseModel, Field
from sqlalchemy.ext.asyncio import AsyncSession
from app.api.runs import actor_of
from app.data import catalog, catalog_impact, introspect, table_versions
from app.data import catalog, catalog_impact, catalog_profile, introspect, table_versions
from app.data.engine import SnapshotTampered
from app.db.base import get_session
from app.db.models import DataSource
@@ -65,6 +67,11 @@ class CatalogDraftIn(BaseModel):
use_model: bool = False
class CatalogProfileIn(BaseModel):
#: 要剖析的表;不给或为空时按使用次数取前 10 张目录里有待核实关系的表
tables: list[str] | None = Field(default=None, max_length=200)
async def _resolved(session: AsyncSession, source_id: str) -> tuple[DataSource, Any]:
"""(数据源记录, 用来读表结构的源):上传源解析成绑定当前快照的视图。"""
row = await session.get(DataSource, source_id)
@@ -227,6 +234,62 @@ async def draft_catalog(source_id: str, payload: CatalogDraftIn,
}
def _profile_table_out(t: catalog_profile.TableProfile) -> dict[str, Any]:
return {
"table_name": t.table,
"queries": t.queries,
"row_estimate": t.row_estimate.to_dict() if t.row_estimate else None,
"findings": t.findings,
"skipped": [s.to_dict() for s in t.skipped],
"date_ranges": t.date_ranges,
"added": t.added,
"updated": t.updated,
"removed": t.removed,
"version": t.version,
"error": t.error,
}
@router.post("/{source_id}/catalog/profile")
async def profile_catalog_tables(source_id: str, payload: CatalogProfileIn,
x_actor: str | None = Header(default=None),
session: AsyncSession = Depends(get_session)) -> dict[str, Any]:
"""同步剖析。返回每张表用了几条查询、写进目录的发现、跳过了什么及原因。
默认关闭:数据源设置里没开启数据剖析、数据源停用,409;同一个数据源已经在剖析,409。
"""
row, source = await _resolved(session, source_id)
try:
catalog_profile.ensure_enabled(row)
except catalog_profile.ProfileDisabled as e:
raise HTTPException(409, str(e)) from e
cache = source.schema_cache or {}
if not cache.get("tables"):
raise HTTPException(409, f"{introspect.why_empty(cache)},无法进行数据剖析。请先在数据源卡片上点「探查结构」")
actor = actor_of(x_actor)
try:
report = await catalog_profile.profile_catalog(session, row, source, tables=payload.tables or None,
actor=actor)
except (catalog_profile.ProfileDisabled, catalog_profile.ProfileBusy) as e:
raise HTTPException(409, str(e)) from e
except SnapshotTampered as e:
raise HTTPException(409, str(e)) from e
rows = [_profile_table_out(t) for t in report.tables]
note = None
if not rows:
note = "目录里还没有待核实的关联关系,请先起草数据目录,或指定要剖析的表"
return {
"profiled_at": report.profiled_at,
"actor": actor,
"settings": report.settings.to_dict(),
"queries_used": report.queries_used,
"stopped": report.stopped,
"tables": rows,
"total": {k: sum(r[k] for r in rows) for k in ("added", "updated", "removed")},
"note": note,
}
@router.post("/{source_id}/catalog/{table}/review")
async def review_catalog_item(source_id: str, table: str, payload: CatalogReviewIn,
x_actor: str | None = Header(default=None),
+3 -1
View File
@@ -23,6 +23,7 @@ from app.core.errors import AUTH, NETWORK, TIMEOUT, classify, explain, raw
from app.core.config import settings
from app.core.crypto import encrypt, mask
from app.data import catalog as data_catalog
from app.data.catalog_profile import profile_settings_problem
from app.data import introspect as introspect_mod
from app.data import table_versions
from app.data.engine import (
@@ -336,7 +337,8 @@ _MASK_FIELD = {
def _refuse_bad_options(options: dict[str, Any] | None) -> None:
problem = query_timeout_problem(options) or mask_columns_problem(options)
problem = (query_timeout_problem(options) or mask_columns_problem(options)
or profile_settings_problem(options))
if problem:
raise HTTPException(422, problem)
+80 -18
View File
@@ -15,13 +15,14 @@ notes 结构(每个「项」都是 {value, source, status},可选 note、upd
"valid_filter": 项(str,SQL 条件片段), "dedup": 项(str),
"columns": {列名: {"label": 项, "meaning": 项, "unit": 项,
"measure": 项("flow"|"stock"|"ratio"|"identifier"|"status"|"attribute"),
"codes": 项(dict[str, str])}},
"codes": 项(dict[码值, 含义];含义可为空串,表示待填写)}},
"relations": [{"id", "columns", "to_table", "to_columns", "cardinality", "coverage",
"source", "status", "note"?}]}
来源(source):comment 数据库注释、fk 外键约束、name 命名推断、profile 数据剖析(阶段 2)、llm 模型起草、
human 人工填写。状态(status):proposed 推断(命名推断、模型起草、数据库注释)、verified 有确证(外键
约束;阶段 2 的数据剖析也写这个状态)、confirmed 人工确认、rejected 人工驳回。只有 confirmed 和 verified
来源(source):comment 数据库注释、fk 外键约束、name 命名推断、profile 数据剖析(data/catalog_profile.py)、
llm 模型起草、human 人工填写。状态(status):proposed 推断(命名推断、模型起草、数据库注释)、verified 有确证
(外键约束;数据剖析核实了覆盖率和父键唯一的关系也是,见 profile_relation_holds)、confirmed 人工确认、
rejected 人工驳回。只有 confirmed 和 verified
将来会触发错误级检查;给模型看时 proposed 的项标「推断,未确认」;rejected 的项留着(防止下次起草又提出
来),但模型和检查都看不到。
@@ -98,6 +99,33 @@ def initial_status(source: str) -> str:
return _INITIAL_STATUS.get(source, "proposed")
#: 数据剖析把关系升为有确证的抽样覆盖率下限(data/catalog_profile.py 判定,撤销审阅时按它判回剖析的结论)
PROFILE_VERIFY_COVERAGE = 0.95
def profile_relation_holds(rel: Mapping[str, Any]) -> bool:
"""一条数据剖析得出的关系站不站得住:抽样覆盖率不低于 PROFILE_VERIFY_COVERAGE,而且被指向列核实唯一。
剖析只在被指向列核实唯一时才写基数(多对一或一对一),所以「基数有值」就是「父键唯一」。结论全在关系
自己的字段里,撤销审阅时照原样判得回来,不必另存一份「审阅前的状态」。
"""
coverage = rel.get("coverage")
return (rel.get("source") == "profile" and isinstance(coverage, (int, float)) and not isinstance(coverage, bool)
and coverage >= PROFILE_VERIFY_COVERAGE and rel.get("cardinality") in ("many_to_one", "one_to_one"))
def unreviewed_status(item: Mapping[str, Any]) -> str:
"""撤销审阅后回到的状态:来源的初始状态。
数据剖析是例外:它的状态随数据而定(覆盖率够、父键唯一的关系是有确证,其余是推断),不能一律回到
initial_status("profile") 的推断——那样人工确认过再撤销,有确证就无故丢了,要等下一次剖析才回来。
剖析提的码值候选、业务日期永远是推断。
"""
if item.get("source") == "profile" and "to_table" in item:
return "verified" if profile_relation_holds(item) else "proposed"
return initial_status(item.get("source"))
def make_item(value: Any, source: str, status: str | None = None, *, note: str | None = None,
at: str | None = None) -> dict[str, Any]:
"""构造一个项。status 不给时按来源取初始状态(initial_status)。"""
@@ -153,8 +181,10 @@ def _business_date_ok(value: Any) -> bool:
def _codes_ok(value: Any) -> bool:
"""码值对照:{码值: 含义},都是文字。含义可以是空串——数据剖析只知道列里出现过哪些取值,含义要等人填;
以前要求含义非空,剖析得到的码值候选就写不进来。"""
return (isinstance(value, dict) and bool(value)
and all(isinstance(k, str) and isinstance(v, str) and v.strip() for k, v in value.items()))
and all(isinstance(k, str) and isinstance(v, str) for k, v in value.items()))
#: 表类型、度量类型在界面上的叫法(和 frontend/src/lib/terms.ts 的 CATALOG_KIND_LABEL / CATALOG_MEASURE_LABEL 一致)。
@@ -587,7 +617,7 @@ def review_item(notes: Mapping[str, Any] | None, path: str, action: str, *, at:
# 人工填写的项没有「推断时的样子」可回:撤销就是删掉,下次起草可以重新提出
del slots[key]
return _assemble(slots)
item["status"] = {"confirm": "confirmed", "reject": "rejected"}.get(action) or initial_status(item.get("source"))
item["status"] = {"confirm": "confirmed", "reject": "rejected"}.get(action) or unreviewed_status(item)
item["updated_at"] = at or now_iso()
return _assemble(slots)
@@ -652,7 +682,7 @@ def apply_human_edit(existing: Mapping[str, Any] | None, submitted: Mapping[str,
continue
kept = dict(prev)
status = item.get("status")
if status != prev.get("status") and status in ("confirmed", "rejected", initial_status(prev.get("source"))):
if status != prev.get("status") and status in ("confirmed", "rejected", unreviewed_status(prev)):
kept["status"] = status
kept["updated_at"] = at
if item.get("note") != prev.get("note"):
@@ -779,6 +809,17 @@ def _patch_relation(table: str, raw: Any, path_id: str | None, tables: Mapping[s
return value, None
def _merge_codes(before: Mapping[str, str], value: Mapping[str, str]) -> dict[str, str]:
"""码值的补充:给出的码加进去或改掉原来的含义。含义为空表示「只知道有这个码、含义待填写」(数据剖析的码值
候选就是这样),所以提案里含义为空的码不覆盖已有的含义,只在原来没有这个码时加进去。"""
out = dict(before)
for code, meaning in value.items():
if not str(meaning).strip() and str(out.get(code) or "").strip():
continue
out[code] = meaning
return out
def plan_patch(notes: Mapping[str, Any] | None, changes: Any, *, table: str,
tables: Mapping[str, Any] | None = None) -> PatchPlan:
"""逐项核对一条目录修改提案,算出改前、改后。不改入参、不抛异常:不合法的项写进 problems。
@@ -790,7 +831,8 @@ def plan_patch(notes: Mapping[str, Any] | None, changes: Any, *, table: str,
规则:
- 取值格式和 validate_notes 同一套(_TABLE_VALUE_RULES / _COLUMN_VALUE_RULES / _relation_problems)。
- 码值是补充:在现有(没被驳回的)码值上加或改给出的码,不整份替换——「status=9 表示作废」不该把已确认的
其余码值删掉。
其余码值删掉。含义为空的码(数据剖析写进来的「含义待填写」)可以被补上含义;提案里含义为空的码只在原来
没有这个码时加进去,不会把已有的含义抹掉(_merge_codes)。
- 看起来像计算公式的文字不收(_FORMULA):公式进口径卡。
- 同一个路径出现两次只认第一次。
"""
@@ -864,7 +906,7 @@ def plan_patch(notes: Mapping[str, Any] | None, changes: Any, *, table: str,
before = _value_of(key, live) if live is not None else None
after = value
if key[0] == "c" and key[2] == "codes" and isinstance(before, dict):
after = {**before, **value}
after = _merge_codes(before, value)
if before is not None and before == after:
state = "same" if live.get("status") == "confirmed" else "confirm"
else:
@@ -1094,6 +1136,15 @@ def infer_name_relations(schema_cache: Mapping[str, Any] | None, table: str, *,
candidates: set[str] = set()
for form in _singulars(words[-1]):
candidates |= index.get((prefix, form), set())
if not candidates:
# 直接对不上:库里的表常带业务前缀(store_tags),列名却只写词根(tagId)。
# 只认和引用它的表共用前缀的那张,从最长的共用前缀试起;仍然不止一张就不推
own = _words(table)
for k in range(len(own) - 1, 0, -1):
for form in _singulars(words[-1]):
candidates |= index.get(("".join(own[:k]) + prefix, form), set())
if candidates:
break
if len(candidates) != 1:
continue
target = next(iter(candidates))
@@ -1625,6 +1676,17 @@ def _date_text(value: Mapping[str, Any]) -> str:
return ";".join(parts)
def _codes_text(codes: Mapping[str, Any]) -> str:
"""码值对照的文字:有含义的写「1=有效」,含义待填写的单独说明——不能写成「1=」,模型会当成含义就是空。"""
filled = [f"{k}={v}" for k, v in codes.items() if str(v).strip()]
pending = [str(k) for k, v in codes.items() if not str(v).strip()]
if not pending:
return "、".join(filled)
if not filled:
return f"{'、'.join(pending)}(含义待填写)"
return f"{'、'.join(filled)},{'、'.join(pending)} 的含义待填写"
def _relation_text(rel: Mapping[str, Any]) -> str:
left = ", ".join(rel.get("columns") or [])
right = ", ".join(f"{rel.get('to_table')}.{c}" for c in rel.get("to_columns") or [])
@@ -1675,7 +1737,7 @@ def _notes_lines(notes: Mapping[str, Any] | None, *, meta: Mapping[str, Any] | N
if name == "measure":
text = MEASURE_LABEL.get(value, str(value))
elif name == "codes":
text = "、".join(f"{k}={v}" for k, v in value.items())
text = _codes_text(value)
else:
text = str(value)
# 第一段是中文名(没有中文名时是含义),不加前缀;其余写明是什么
@@ -1932,13 +1994,13 @@ __all__ = [
"CATALOG_DRAFT_SYSTEM", "COLUMN_FIELDS", "COLUMN_FIELD_LABEL", "COLUMN_MEASURES", "CatalogConflict",
"CatalogEntry", "CatalogInvalid", "CatalogModelUnavailable", "CatalogPathError", "DRAFT_DEFAULT_TABLES",
"DraftReport", "INFERRED_MARK", "ITEM_SOURCES", "ITEM_STATUSES", "JoinEdge", "KIND_LABEL", "MEASURE_LABEL",
"MergeStats", "NEW_RELATION_PATH", "PATCH_MAX_CHANGES", "PatchChange", "PatchPlan", "QUERY_SNAPSHOT_KIND",
"REVIEW_ACTIONS", "TABLE_FIELDS", "TABLE_FIELD_LABEL", "TABLE_KINDS", "TableDraft", "TableDraftResult",
"UI_KIND_LABEL", "UI_MEASURE_LABEL", "apply_human_edit", "apply_patch", "describe_slot", "draft_catalog",
"draft_structure", "draft_with_model", "fk_relations", "frozen_catalog", "infer_name_relations",
"MergeStats", "NEW_RELATION_PATH", "PATCH_MAX_CHANGES", "PROFILE_VERIFY_COVERAGE", "PatchChange", "PatchPlan",
"QUERY_SNAPSHOT_KIND", "REVIEW_ACTIONS", "TABLE_FIELDS", "TABLE_FIELD_LABEL", "TABLE_KINDS", "TableDraft",
"TableDraftResult", "UI_KIND_LABEL", "UI_MEASURE_LABEL", "apply_human_edit", "apply_patch", "describe_slot",
"draft_catalog", "draft_structure", "draft_with_model", "fk_relations", "frozen_catalog", "infer_name_relations",
"initial_status", "join_paths", "make_item", "merge_notes", "now_iso", "parse_path", "plan_patch",
"query_snapshot_meta", "read_catalog", "read_entry", "relation_graph", "relation_id", "render_table_index",
"render_table_notes", "resolve_draft_model", "resolve_table_name", "review_entry", "review_item", "same_notes",
"save_human_edit", "save_patch", "status_counts", "system_notes_source", "table_usage", "validate_notes",
"visible_notes", "write_entry",
"profile_relation_holds", "query_snapshot_meta", "read_catalog", "read_entry", "relation_graph", "relation_id",
"render_table_index", "render_table_notes", "resolve_draft_model", "resolve_table_name", "review_entry",
"review_item", "same_notes", "save_human_edit", "save_patch", "status_counts", "system_notes_source",
"table_usage", "unreviewed_status", "validate_notes", "visible_notes", "write_entry",
]
File diff suppressed because it is too large Load Diff
+6 -1
View File
@@ -58,11 +58,16 @@ MAX_QUERY_TIMEOUT_S = 600
#: 完整快照仍能按工件 id 取到,SQL 里给列起个别名也能绕开
MASK_COLUMNS_OPTION = "mask_columns"
#: 数据源 options 里的数据剖析设置(一个对象:开关、查询次数上限、单条时限……)。取值和校验见
#: data/catalog_profile.py;放在这里只为让拼连接串时摘掉它
CATALOG_PROFILE_OPTION = "catalog_profile"
# options 里这些 key 是 AgentLab 自己的配置,不是驱动参数,拼 URL 时要摘掉。
# schema:探查哪个 schema(企业库里只读账号名下常常什么都没有,数据在别处)
# query_timeout_s:查询时限,由数据层按语句下发给数据库,见 _server_deadline
# mask_columns:证据面板遮罩的列,驱动不认识它,拼进连接串会被当成未知参数拒掉
_NON_DRIVER_OPTIONS = frozenset({"schema", QUERY_TIMEOUT_OPTION, MASK_COLUMNS_OPTION})
# catalog_profile:数据剖析设置,同上;它还是个对象,拼进去连 URL 都不成形
_NON_DRIVER_OPTIONS = frozenset({"schema", QUERY_TIMEOUT_OPTION, MASK_COLUMNS_OPTION, CATALOG_PROFILE_OPTION})
_MASK_SPLIT = re.compile(r"[,,、;;\n]+")
+42 -2
View File
@@ -169,13 +169,17 @@ class SqlChecker:
和助手挑表看到的是同一张图(catalog.relation_graph):目录里的关系,加上没起草过的表按外键、命名现推的;
被驳回的关系不进图,也不会被现推回来。
"""
return [e for e in self.edges_from(a) if e.to_table.lower() == b.lower()]
def edges_from(self, a: str) -> list[catalog.JoinEdge]:
"""从 a 出发的全部边。关系图第一次用到时才建,之后复用。"""
if self._edges is None:
graph = catalog.relation_graph(self._raw, schema_cache=self.schema_cache)
index: dict[str, list[catalog.JoinEdge]] = {}
for table, edges in graph.items():
index.setdefault(table.lower(), []).extend(edges)
self._edges = index
return [e for e in self._edges.get(a.lower(), []) if e.to_table.lower() == b.lower()]
return list(self._edges.get(a.lower(), []))
# ---- 检查
@@ -557,10 +561,46 @@ class _ScopeCheck:
continue # 两张表都没写目录:没有可对照的,不判
condition = " AND ".join(filter(None, (self.excerpt(c) for c in group["conds"])))
if best is None:
self._join_checks.append(self._join_unknown(a, b, candidates, condition))
parent = self._shared_parent(a, b, pairs)
self._join_checks.append(self._join_sibling(a, b, parent, condition) if parent
else self._join_unknown(a, b, candidates, condition))
elif best.status not in _STRONG:
self._join_checks.append(self._join_proposed(a, b, best, condition))
def _shared_parent(self, a: _Table, b: _Table, pairs: set[tuple[str, str]]) -> str | None:
"""兄弟关联:每一对连接列都是两边各自指向同一张父表的同一组列(a.member_id → members.id ← b.member_id)。
返回父表名;不是这种连法返回 None。
目录里不会记 a 和 b 之间的关系,但这不是「连错了列」:比如评价表和标签关联表各自按门店编号指向门店、
彼此按门店编号相连。照「对不上任何关系」报警告是误报。
"""
def parents(table: _Table, column: str) -> set[tuple[str, tuple[str, ...]]]:
return {(e.to_table.lower(), tuple(c.lower() for c in e.to_columns))
for e in self.checker.edges_from(table.name)
if tuple(c.lower() for c in e.from_columns) == (column,)
and e.cardinality in (None, "many_to_one", "one_to_one")}
shared: set[tuple[str, tuple[str, ...]]] | None = None
for left, right in pairs:
common = parents(a, left) & parents(b, right)
shared = common if shared is None else shared & common
if not shared:
return None
names = {table for table, _ in shared or ()}
return next(iter(names)) if len(names) == 1 else None
def _join_sibling(self, a: _Table, b: _Table, parent: str, condition: str) -> SqlCheck:
item = (self.checker.notes_of(parent) or {}).get("label")
label = str(item["value"]) if isinstance(item, dict) and isinstance(item.get("value"), str) \
and item["value"].strip() else parent
return SqlCheck(
code="join_unconfirmed", level="info", table=b.name, sql_excerpt=condition or None,
message=(f"「{a.label()}」与「{b.label()}」之间没有直接的关系,这里经共同的「{label}」相连。"
"关联后行数可能成倍增加:确认其中一侧每个键只有一行,或先各自汇总再关联"),
for_model=(f"{a.name} 和 {b.name} 按 {condition or '这组条件'} 关联:两边都指向 {parent},彼此是一对多对多。"
"如果其中一侧每个键不止一行,关联后另一侧的行会被重复,求和、计数会放大;"
"先把一侧筛到每个键一行(例如只取主记录),或分别汇总到这个键再关联"))
def _join_unknown(self, a: _Table, b: _Table, candidates: list[catalog.JoinEdge], condition: str) -> SqlCheck:
known = ";".join(dict.fromkeys(e.condition() for e in candidates))
hint = f"目录里这两张表的关系是:{known}。照它改关联条件" if known else \