全栈安全加固、性能提速与架构演进:连接池与沙箱止血、双路混合检索、多版本归档及前端分包治理

- 第一批(安全加固与生命周期止血):隔离测试宿主机密钥与 .env 泄漏;修复 SQLite 取消查询挂死、:memory: 连接池报错与 EngineCache 世代竞态;补齐运行与评测删除时的 Checkpointer 线程、Artifact 磁盘文件与无主附件回收;加固三套沙箱路径越界、进程组清理与 Seatbelt 数据目录隔离,封堵 SSRF IPv4-mapped IPv6 元数据绕过与响应体 OOM;修复推理模型多轮 reasoning_content、结构化输出 Token 漏计与缓存折扣计费
- 第二批(数据库/集群/问数提速与首屏分包):补齐核心表索引与 SQLite WAL 调优,列表接口按需 defer 大列,收敛启动扫描并打通跨进程 slot 槽位唤醒;优化跨源全局选表、哈希桶数字绑定、流式 CSV 导入及多方言分析算子;剥离首屏静态引入的 @xyflow/react 与全局样式(首屏主包降至 461.59 kB),补充流式 rAF 攒批、Store 精细订阅与表单无障碍绑定
- 第三批(架构演进与工程自动化闭环):记录库超挂接阈值历史版本自动增量归档至单库多表(解除 SQLite 10 库 ATTACH 硬上限);重构知识库为词面 BM25 与全集合归一化向量矩阵真·双路独立召回并支持游标分批重建;升级多方言感知 SQL 模板词法分析(E'...' / $$...$$ / q'[...]' / # / 嵌套注释)与 Decimal / bool / NaN 参数校验,消除 ledger 层对 api 路由层的反向依赖;拆分 3101 行 CaliberDrafts 页面为 7 个领域子模块,消除检查脚本本机路径硬编码并新增 GitHub Actions CI 门禁与三批回归测试套件
This commit is contained in:
ThearchyHelios
2026-11-13 14:51:36 +08:00
parent 09949fc8ac
commit eddeded2a4
106 changed files with 7178 additions and 3913 deletions
+31 -13
View File
@@ -40,12 +40,16 @@ def keep_set(runs: list[EvalRun], keep: int) -> set[str]:
return kept
async def _clean_one(session: AsyncSession, run: EvalRun) -> tuple[int, int, set[str]]:
"""清一次评测的明细:(删了几项结果, 删了几次运行, 要回收的工件 id)。不提交。"""
async def _clean_one(
session: AsyncSession,
run: EvalRun,
origin_runs: dict[str, set[str]],
) -> tuple[int, set[str], set[str]]:
"""清一次评测的明细:(删了几项结果, 删了哪几次运行, 要回收的工件 id)。不提交。"""
from app.db.models import Artifact, Run
results = list((await session.execute(select(EvalResult).where(EvalResult.eval_run_id == run.id))).scalars())
run_ids: set[str] = set()
run_ids: set[str] = set(origin_runs.get(run.id, ()))
artifacts: set[str] = set()
for r in results:
if r.run_id:
@@ -55,11 +59,7 @@ async def _clean_one(session: AsyncSession, run: EvalRun) -> tuple[int, int, set
artifacts.add(r.details_artifact)
await session.execute(delete(Artifact).where(Artifact.id == r.details_artifact, Artifact.run_id == "")
.execution_options(synchronize_session=False))
# 这次评测发起的、结果里没记下的运行(重来的前一次、停止时没写进结果的):按来源找
for rid, origin in (await session.execute(select(Run.id, Run.origin).where(Run.origin_kind == "eval"))).all():
if (origin or {}).get("eval_run_id") == run.id:
run_ids.add(rid)
removed = 0
deleted_runs: set[str] = set()
for rid in sorted(run_ids):
row = await session.get(Run, rid)
if row is None or row.origin_kind != "eval" or row.status in _LIVE:
@@ -67,17 +67,20 @@ async def _clean_one(session: AsyncSession, run: EvalRun) -> tuple[int, int, set
artifacts |= set((await session.execute(select(Artifact.id).where(Artifact.run_id == rid))).scalars())
await session.execute(delete(Artifact).where(Artifact.run_id == rid).execution_options(synchronize_session=False))
await session.delete(row)
removed += 1
deleted_runs.add(rid)
for r in results:
await session.delete(r)
run.cleaned_at = utcnow()
return len(results), removed, artifacts
return len(results), deleted_runs, artifacts
async def clean_suite(suite_id: str, *, keep: int | None = None) -> dict[str, Any]:
"""按保留规则清一个题集:{runs: 清了几次评测, results, deleted_runs, files: 回收了几个工件文件}。"""
from app.api.settings import eval_settings
from app.core.artifact_store import forget
from app.db.models import Run
from app.engine import checkpoints
from app.files import store as files_store
async with SessionLocal() as session:
if keep is None:
@@ -86,15 +89,30 @@ async def clean_suite(suite_id: str, *, keep: int | None = None) -> dict[str, An
.order_by(EvalRun.created_at.desc(), EvalRun.id.desc()))).scalars())
kept = keep_set(runs, keep)
targets = [r for r in runs if r.id not in kept and r.cleaned_at is None]
results = removed = 0
origin_runs: dict[str, set[str]] = {}
if targets:
target_ids = {r.id for r in targets}
# 评测发起的、结果里没记下的运行(重来的前一次、停止时没写进结果的):整批只扫一次按来源归组
for rid, origin in (await session.execute(select(Run.id, Run.origin).where(Run.origin_kind == "eval"))).all():
erid = (origin or {}).get("eval_run_id")
if isinstance(erid, str) and erid in target_ids:
origin_runs.setdefault(erid, set()).add(rid)
results = 0
deleted_run_ids: set[str] = set()
artifacts: set[str] = set()
for run in targets:
n, m, found = await _clean_one(session, run)
n, gone_runs, found = await _clean_one(session, run, origin_runs)
results += n
removed += m
deleted_run_ids |= gone_runs
artifacts |= found
released_shas = await files_store.release_runs(session, deleted_run_ids)
await session.commit()
removed = len(deleted_run_ids)
files = await forget(artifacts) if artifacts else []
if released_shas:
await files_store.drop_unreferenced(released_shas)
if deleted_run_ids:
await checkpoints.delete_threads(deleted_run_ids)
if targets:
logger.info("评测清理:题集 %s 清了 %d 次评测的明细(%d 项结果、%d 次运行、%d 个工件文件)", suite_id, len(targets),
results, removed, len(files))