全栈安全加固、性能提速与架构演进:连接池与沙箱止血、双路混合检索、多版本归档及前端分包治理

- 第一批(安全加固与生命周期止血):隔离测试宿主机密钥与 .env 泄漏;修复 SQLite 取消查询挂死、:memory: 连接池报错与 EngineCache 世代竞态;补齐运行与评测删除时的 Checkpointer 线程、Artifact 磁盘文件与无主附件回收;加固三套沙箱路径越界、进程组清理与 Seatbelt 数据目录隔离,封堵 SSRF IPv4-mapped IPv6 元数据绕过与响应体 OOM;修复推理模型多轮 reasoning_content、结构化输出 Token 漏计与缓存折扣计费
- 第二批(数据库/集群/问数提速与首屏分包):补齐核心表索引与 SQLite WAL 调优,列表接口按需 defer 大列,收敛启动扫描并打通跨进程 slot 槽位唤醒;优化跨源全局选表、哈希桶数字绑定、流式 CSV 导入及多方言分析算子;剥离首屏静态引入的 @xyflow/react 与全局样式(首屏主包降至 461.59 kB),补充流式 rAF 攒批、Store 精细订阅与表单无障碍绑定
- 第三批(架构演进与工程自动化闭环):记录库超挂接阈值历史版本自动增量归档至单库多表(解除 SQLite 10 库 ATTACH 硬上限);重构知识库为词面 BM25 与全集合归一化向量矩阵真·双路独立召回并支持游标分批重建;升级多方言感知 SQL 模板词法分析(E'...' / $$...$$ / q'[...]' / # / 嵌套注释)与 Decimal / bool / NaN 参数校验,消除 ledger 层对 api 路由层的反向依赖;拆分 3101 行 CaliberDrafts 页面为 7 个领域子模块,消除检查脚本本机路径硬编码并新增 GitHub Actions CI 门禁与三批回归测试套件
This commit is contained in:
ThearchyHelios
2026-11-13 14:51:36 +08:00
parent 09949fc8ac
commit eddeded2a4
106 changed files with 7178 additions and 3913 deletions
+11 -4
View File
@@ -45,14 +45,18 @@ def _shanghai_time_banner() -> str:
)
_METRIC_RULE_LINE = (
" - 若下方「已命中的标准语义指标卡片」直接对应用户所问指标且口径完整,"
"第 1 轮直接调用 `metric_query`(支持 `date_from`、`date_to`、`group_by_date`、`group_by`、`filters`)拿到数据后立即作答。\n"
)
DIRECT_SYSTEM_PROMPT = """你是 AgentLab 的「快速直查数据分析助手」。你的任务是基于已预命中的指标卡片、标准业务实体与数据目录,以最少的工具轮次、最快速度给出准确、清晰、结构化的 Markdown 数据回答。
【核心取数与防错纪律】
1. **{time_banner}**
2. **严控轮次预算,杜绝多轮逆向摸底(最多 1~2 轮工具调用)**:
- 快速直查的目标是秒级直取答案,**严禁**展开多轮跨表试探与字段考古。
- 若下方「已命中的标准语义指标卡片」直接对应用户所问指标且口径完整,第 1 轮直接调用 `metric_query`(支持 `date_from`、`date_to`、`group_by_date`、`group_by`、`filters`)拿到数据后立即作答。
- 若第 1 轮返回空集、全空分组或口径明显异常(如底表混存多种记录类型、汇总层级或分类口径而缺少筛选条件),**最多再做 1 轮并行 `db_query__` 补查**(在同一轮内同时核对关联维表或配置表映射并查询过滤后的目标汇总 SQL),拿到结果立即收口作答;若仍存在跨表复杂口径分歧,直接呈现已核实的事实并建议用户点击右上角「切换深度分析」。
{metric_rule} - 若第 1 轮返回空集、全空分组或口径明显异常(如底表混存多种记录类型、汇总层级或分类口径而缺少筛选条件),**最多再做 1 轮并行 `db_query__` 补查**(在同一轮内同时核对关联维表或配置表映射并查询过滤后的目标汇总 SQL),拿到结果立即收口作答;若仍存在跨表复杂口径分歧,直接呈现已核实的事实并建议用户点击右上角「切换深度分析」。
3. **合并查询,杜绝碎片化往返**:
- 手写 `db_query__<源名>` 时,尽量在 **1 条(最多不超过 2 条)SQL** 内一次性查全所需指标与近期对比窗口(例如查「今天怎么样」时,可直接查近 7 天按日汇总 `WHERE stat_date >= '<7日前>' ORDER BY stat_date DESC`,一条 SQL 同时看清今天是否有数、最新到哪一天、以及相比前几天的走势!)。
- 已开启并行工具调用(`parallel_tools=True`):确实需要同时看汇总与下钻明细、或查两张互不依赖的表时,必须在**同一轮**一次性并行发出工具调用,切勿拆成多轮串行调用。
@@ -83,7 +87,10 @@ def build_direct_graph(
tools.append(METRIC_QUERY)
tools.extend(["calculator", "kb_search"])
base_sys = DIRECT_SYSTEM_PROMPT.format(time_banner=_shanghai_time_banner())
base_sys = DIRECT_SYSTEM_PROMPT.format(
time_banner=_shanghai_time_banner(),
metric_rule=_METRIC_RULE_LINE if has_metrics else "",
)
if bundle.system_context:
full_sys = f"{base_sys}\n\n---\n\n{_escape_template_braces(bundle.system_context)}"
else:
@@ -252,7 +259,7 @@ class DirectChatDriver(ChatDriver):
self.grounding_summary = bundle.ui_summary
# 2. 构建确定性直查图(含零延迟证据核验)并落库、推送预命中事件
direct_graph = build_direct_graph(bundle=bundle, has_metrics=True)
direct_graph = build_direct_graph(bundle=bundle, has_metrics=bundle.has_enabled_metrics)
if instruction and instruction.strip() and instruction.strip() != self.question.strip():
# 用户点「重试本轮」且仍保持快速直查时,把重试提示附进用户 prompt
agent_node = next((n for n in direct_graph["nodes"] if n.get("type") == "agent"), None)
+56 -24
View File
@@ -473,6 +473,38 @@ def _auto_bind_bare_numbers(
if not cells:
return text
cell_order = {id(c): idx for idx, c in enumerate(cells)}
text_code_by_raw: dict[str, list[_CellCandidate]] = {}
numeric_cells: list[_CellCandidate] = []
numeric_by_exact: dict[Decimal, list[_CellCandidate]] = {}
for c in cells:
if c.is_text_code:
text_code_by_raw.setdefault(c.raw_text, []).append(c)
else:
numeric_cells.append(c)
numeric_by_exact.setdefault(c.dec_value, []).append(c)
quant_cache: dict[tuple[str, int], dict[Decimal, list[_CellCandidate]]] = {}
def _quant_buckets(scale_kind: str, places: int) -> dict[Decimal, list[_CellCandidate]]:
key = (scale_kind, places)
cached = quant_cache.get(key)
if cached is not None:
return cached
buckets: dict[Decimal, list[_CellCandidate]] = {}
quant = Decimal(1).scaleb(-places)
if scale_kind == "pct":
for c in numeric_cells:
q_val = (c.dec_value * Decimal(100)).quantize(quant, rounding=ROUND_HALF_UP)
buckets.setdefault(q_val, []).append(c)
else:
divisor = Decimal(10000) if scale_kind == "万" else Decimal(100_000_000)
for c in numeric_cells:
q_val = (c.dec_value / divisor).quantize(quant, rounding=ROUND_HALF_UP)
buckets.setdefault(q_val, []).append(c)
quant_cache[key] = buckets
return buckets
fences = _fenced_line_ranges(text)
lines = text.splitlines(keepends=True)
table_headers = _table_headers_by_line(lines)
@@ -546,31 +578,31 @@ def _auto_bind_bare_numbers(
continue
has_leading_zero = bool(_LEADING_ZERO_INT_RE.match(raw_clean))
places = max(int(tok.decimals or 0), 0)
matched: list[tuple[_CellCandidate, str | None]] = []
for cand in cells:
if has_leading_zero:
if cand.is_text_code and cand.raw_text == raw_clean:
matched.append((cand, None))
continue
if cand.is_text_code:
if not tok.is_percent and not unit_conv and cand.raw_text == raw_clean:
matched.append((cand, None))
continue
if unit_conv and unit_scale is not None:
scaled_u = cand.dec_value / unit_scale
places = max(int(tok.decimals or 0), 0)
quant = Decimal(1).scaleb(-places)
if scaled_u.quantize(quant, rounding=ROUND_HALF_UP) == target_dec:
matched.append((cand, unit_conv))
elif cand.dec_value == target_dec:
matched.append((cand, None))
elif tok.is_percent:
# 单元格存 0.1258,正文写 12.6% 或 12.58%
scaled = cand.dec_value * Decimal(100)
places = max(int(tok.decimals or 0), 0)
quant = Decimal(1).scaleb(-places)
if scaled.quantize(quant, rounding=ROUND_HALF_UP) == target_dec:
matched.append((cand, "pct"))
if has_leading_zero:
matched = [(c, None) for c in text_code_by_raw.get(raw_clean, ())]
elif unit_conv and unit_scale is not None:
matched = [(c, unit_conv) for c in _quant_buckets(unit_conv, places).get(target_dec, ())]
elif not tok.is_percent:
tc_hits = [(c, None) for c in text_code_by_raw.get(raw_clean, ())]
num_hits = [(c, None) for c in numeric_by_exact.get(target_dec, ())]
if tc_hits and num_hits:
matched = sorted(tc_hits + num_hits, key=lambda pair: cell_order[id(pair[0])])
else:
matched = tc_hits or num_hits
else:
# 单元格存 12.6(直接相等)或存 0.1258(正文写 12.6% 或 12.58%)
exact_hits = [(c, None) for c in numeric_by_exact.get(target_dec, ())]
pct_hits = [
(c, "pct")
for c in _quant_buckets("pct", places).get(target_dec, ())
if c.dec_value != target_dec
]
if exact_hits and pct_hits:
matched = sorted(exact_hits + pct_hits, key=lambda pair: cell_order[id(pair[0])])
else:
matched = exact_hits or pct_hits
if not matched:
continue
+101 -60
View File
@@ -51,6 +51,7 @@ class GroundingBundle:
guardrails: list[dict[str, str]] = field(default_factory=list)
system_context: str = ""
ui_summary: dict[str, Any] = field(default_factory=dict)
has_enabled_metrics: bool = False
def _term_in_text(term: str, text_lower: str, ascii_words: set[str]) -> bool:
@@ -395,13 +396,14 @@ async def ground_question(
else:
narrowed_sources = list(candidate_sources)
# 5. 数据目录 BM25 直选 + 指标底表 + 实体绑定表 -> 组装高信噪比表结构与防错护栏
# 5. 数据目录 BM25 直选 + 指标底表 + 实体绑定表 -> 跨源全局打分选表,避免首源独占全部槽位
matched_tables: list[dict[str, Any]] = []
guardrails: list[dict[str, str]] = []
table_blocks: list[str] = []
total_detail_chars = 0
for ds in narrowed_sources:
ds_ctx_list: list[dict[str, Any]] = []
for ds_idx, ds in enumerate(narrowed_sources):
cache = ds.schema_cache if isinstance(ds.schema_cache, Mapping) else {}
tables_map = (cache.get("tables") or {}) if isinstance(cache, Mapping) else {}
if not isinstance(tables_map, Mapping) or not tables_map:
@@ -414,74 +416,112 @@ async def ground_question(
facts = await catalog_profile_jobs.last_facts(session, ds.id)
except Exception: # noqa: BLE001
facts = {}
picked_for_ds: list[str] = []
reasons_for_table: dict[str, str] = {}
# (a) 命中指标的底表优先入选
for mm in matched_metrics:
if mm["source_id"] == ds.id:
resolved = catalog.resolve_table_name(cache, mm["table_name"])
if resolved and resolved not in picked_for_ds:
picked_for_ds.append(resolved)
reasons_for_table[resolved] = f"指标「{mm['name']}」底表"
# (b) 命中实体的绑定表入选
for me in matched_entities:
for b in me["bindings"]:
if name_key(b["source_name"]) == name_key(ds.name):
resolved = catalog.resolve_table_name(cache, b["table_name"])
if resolved and resolved not in picked_for_ds and len(picked_for_ds) < MAX_GROUNDED_TABLES:
picked_for_ds.append(resolved)
reasons_for_table[resolved] = f"实体「{me['label']}」绑定表"
masked = schema_view.masked_set(ds)
try:
rel_graph = schema_view.relation_graph(ds, raw_notes)
except Exception: # noqa: BLE001
rel_graph = {}
def _include_companion_lookup(limit_cap: int) -> list[dict[str, Any]]:
if not picked_for_ds:
return []
comps = find_companion_lookup_tables(
tables_map,
picked_for_ds,
visible_by_table=visible_by_table,
rel_graph=rel_graph,
masked_set=masked,
ds_ctx_list.append(
{
"idx": ds_idx,
"ds": ds,
"cache": cache,
"tables_map": tables_map,
"raw_notes": raw_notes,
"visible_by_table": visible_by_table,
"facts": facts,
"masked": masked,
"rel_graph": rel_graph,
"picked": [],
"reasons": {},
}
)
total_picked = 0
def _pick_table(ctx: dict[str, Any], tname: str, reason: str, *, cap: int = MAX_GROUNDED_TABLES) -> bool:
nonlocal total_picked
if total_picked >= cap or tname in ctx["picked"]:
return False
ctx["picked"].append(tname)
ctx["reasons"][tname] = reason
total_picked += 1
return True
# (a) 命中指标的底表跨源最优先入选
for ctx in ds_ctx_list:
ds = ctx["ds"]
for mm in matched_metrics:
if mm["source_id"] == ds.id:
resolved = catalog.resolve_table_name(ctx["cache"], mm["table_name"])
if resolved:
_pick_table(ctx, resolved, f"指标「{mm['name']}」底表", cap=MAX_GROUNDED_TABLES + 2)
# (b) 命中实体的绑定表跨源次优先入选
for ctx in ds_ctx_list:
ds = ctx["ds"]
for me in matched_entities:
for b in me["bindings"]:
if name_key(b["source_name"]) == name_key(ds.name):
resolved = catalog.resolve_table_name(ctx["cache"], b["table_name"])
if resolved:
_pick_table(ctx, resolved, f"实体「{me['label']}」绑定表")
# (c) 各候选源 BM25 检索统一收集并按得分全局排序选表(0 LLM 调用)
if total_picked < MAX_GROUNDED_TABLES:
global_bm25_hits: list[tuple[float, int, int, dict[str, Any], str]] = []
for ctx in ds_ctx_list:
hits = schema_view.search_tables(
ctx["ds"],
ctx["visible_by_table"],
combined_q,
limit=MAX_GROUNDED_TABLES,
)
for comp in comps:
if comp["col_count"] > 12:
continue
ct_name = comp["table_name"]
if ct_name not in picked_for_ds and len(picked_for_ds) + len(matched_tables) < limit_cap:
picked_for_ds.append(ct_name)
reasons_for_table[ct_name] = "关联维表与字典"
for rank, h in enumerate(hits):
if h.in_scope and h.name not in ctx["picked"]:
global_bm25_hits.append((float(h.score), -ctx["idx"], -rank, ctx, h.name))
global_bm25_hits.sort(key=lambda item: (item[0], item[1], item[2]), reverse=True)
for _score, _neg_ds_idx, _neg_rank, ctx, tname in global_bm25_hits:
if total_picked >= MAX_GROUNDED_TABLES:
break
_pick_table(ctx, tname, "目录语义检索命中")
# (d) 若用户明确指定了该小库(<= 5 张表)且尚未命中任何表,直接全给
for ctx in ds_ctx_list:
ds = ctx["ds"]
if not ctx["picked"] and ds in explicit_sources and len(ctx["tables_map"]) <= 5:
for tname in list(ctx["tables_map"].keys()):
if not _pick_table(ctx, tname, "指定数据源核心表"):
break
return comps
_include_companion_lookup(MAX_GROUNDED_TABLES)
# (c) BM25 数据目录检索直选(0 LLM 调用)
remaining_slots = MAX_GROUNDED_TABLES - len(matched_tables) - len(picked_for_ds)
if remaining_slots > 0:
hits = schema_view.search_tables(ds, visible_by_table, combined_q, limit=remaining_slots + 2)
for h in hits:
if h.in_scope and h.name not in picked_for_ds and len(picked_for_ds) + len(matched_tables) < MAX_GROUNDED_TABLES:
picked_for_ds.append(h.name)
reasons_for_table[h.name] = "目录语义检索命中"
# (d) 若用户明确指定了该小库(<= 5 张表)且尚未命中任何表,直接全给
if not picked_for_ds and ds in explicit_sources and len(tables_map) <= 5:
for tname in list(tables_map.keys())[:MAX_GROUNDED_TABLES - len(matched_tables)]:
picked_for_ds.append(tname)
reasons_for_table[tname] = "指定数据源核心表"
companion_links = _include_companion_lookup(MAX_GROUNDED_TABLES + 1)
# 按数据源顺序补齐关联维表与一跳连接表,渲染表结构与护栏
for ctx in ds_ctx_list:
ds = ctx["ds"]
picked_for_ds: list[str] = ctx["picked"]
if not picked_for_ds:
continue
reasons_for_table: dict[str, str] = ctx["reasons"]
tables_map = ctx["tables_map"]
raw_notes = ctx["raw_notes"]
visible_by_table = ctx["visible_by_table"]
facts = ctx["facts"]
masked = ctx["masked"]
rel_graph = ctx["rel_graph"]
companion_links = find_companion_lookup_tables(
tables_map,
picked_for_ds,
visible_by_table=visible_by_table,
rel_graph=rel_graph,
masked_set=masked,
)
for comp in companion_links:
if comp["col_count"] > 12:
continue
ct_name = comp["table_name"]
if _pick_table(ctx, ct_name, "关联维表与字典", cap=MAX_GROUNDED_TABLES + 1):
break
# 补一跳连接中间表(若有)
try:
@@ -546,7 +586,7 @@ async def ground_question(
ds_sections.append("表间标准连接条件:\n" + "\n".join(join_lines[:12]))
table_blocks.append("\n".join(ds_sections))
if len(matched_tables) >= MAX_GROUNDED_TABLES:
if len(matched_tables) >= MAX_GROUNDED_TABLES + 1:
break
elapsed_ms = max(1, int((time.perf_counter() - t0) * 1000))
@@ -719,4 +759,5 @@ async def ground_question(
guardrails=guardrails,
system_context="\n\n".join(ctx_parts),
ui_summary=ui_summary,
has_enabled_metrics=bool(scoped_metrics),
)