yilunjiang/AgentLab
1
0
Fork 0
mirror of https://github.com/ThearchyHelios/AgentLab.git synced 2026-11-15 04:11:58 +00:00
Code Issues Packages Projects Releases Wiki Activity

合入数据目录起草改造：用模型起草改成后台任务，逐表记状态和原因，写完一张就并入目录，可以停止、只重试失败的表，服务重启时没跑完的标中断、不自动续跑；保留模型思考，输出额度 32768（服务方不收时退回 8192）；列分段均分、宽表先起草表级再分段起草列，额度用完或超时就拆小再问、段大小随之减半、每段调用次数封顶，接连没写出来时给提醒；重新起草不丢已有的模型起草项；模型拿不准而留空的列逐表记下并提示；数据源上的起草要求和本次补充说明交给模型（不能推翻只记数据事实的规矩）；设置里单独的起草模型（数据目录和按配方导入的 AI 起草共用）；库注释就是名字本身或标识符写法的不当中文名；旧的同步起草遇到进行中的任务回 409

Browse Source
This commit is contained in:
ThearchyHelios
2026-10-14 07:27:04 +08:00
parent 49583b273f 1130f22201
commit a87b935ebf
35 changed files with 8066 additions and 658 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+2 -1
View File
@@ -276,7 +276,8 @@ flowchart LR
| 环节 | 做法 |
|---|---|
| 起草 | 探查时读外键和唯一约束；按 `xxx_id`、`xxxId` 推断关联，表名带业务前缀的也认；可选让模型起草中文名、粒度、度量类型。导入表格的源已有系统按核对结果生成的说明，目录不重复、不改写它 |
| 起草 | 探查时读外键和唯一约束；按 `xxx_id`、`xxxId` 推断关联，表名带业务前缀的也认；可选让模型起草中文名、粒度、表类型、业务主键、业务日期和每列的含义、单位、度量类型。导入表格的源已有系统按核对结果生成的说明，目录不重复、不改写它 |
| 模型起草 | 作为后台任务逐表进行，每张表记下状态和原因，写完一张就并入目录；关掉对话框不停，可以停止、只重试失败的表，清单上标出上次起草失败或只完成一部分的表。服务重启时没跑完的任务标成中断，不会自己续跑，点重试接上。保留模型的思考，输出额度 32768（服务方不收时退回 8192）；每次调用最多 40 列，宽表先起草表级的项、再把列均分成几段起草，按表的先后排队，不插到窄表前面；额度用完或超时就拆小再问（超时的两半同时问，拆到每段 10 列仍接连超时就不再拆），累计两次后新开始的段减半（最低 10 列），每段连同它分开、拆小的各块最多调用 8 次，缺的表和列只补问缺的，被截断的回复先留下已经写完整的部分；最近 6 次调用有 4 次没写出来时提醒换模型。没写出来的原因写清是第几段、哪几列。重新起草时模型这次留空或没通过校验的项不删以前起草的；模型拿不准、留空的列（目录里还没有中文名和含义的）逐表记下，可以补充起草要求后重起草。库注释就是表名、列名本身或者是标识符写法的，不记进目录（不当中文名，也不当说明、含义）。数据源上存的「起草要求」和这次的补充说明一起交给模型，帮它理解业务、统一用词，不能推翻「只记数据事实」的规矩，交回的内容照样逐项校验。用哪个模型在「设置 → 偏好设置」的「起草模型」里选，空着跟随助手的模型；按配方导入表格的 AI 起草也用它 |
| 数据剖析 | 默认关闭，按数据源开启。只读、经过守卫，单条查询和整次剖析都有上限，大表用统计信息估算行数，遮罩的列不取样。抽样核实推断出的关联，补上覆盖率和基数，站得住的升为已验证；状态列给出码值候选，含义等人填写 |
| 对话里纠正 | 用户说「status=9 表示作废」，助手给出一条目录修改提案，列出改前、改后和理由，人确认后才保存；公式不收，留给口径卡 |
| 版本与提醒 | 目录随表结构快照冻结进证据，每次运行用的是哪一版查得到。发布时记下模板引用的表的目录版本，正式运行时目录有变化就提醒；agent 运行时才写 SQL 的，按「可能涉及」单独提醒。只提醒，不拦运行 |
backend/app/api/catalog.py
+208 -14
View File
@@ -7,11 +7,16 @@
- POST /catalog/{table}/patch/preview 目录修改提案的预览（只读）：对着当前目录算改前、改后
- POST /catalog/{table}/patch 保存目录修改提案（乐观锁），记为人工确认
- GET /catalog/{table}/impact 影响面：引用这张表的已发布、受管模板（data/catalog_impact.py）
- POST /catalog/draft 同步起草（注释、外键、命名推断，可选模型）
- POST /catalog/draft 同步起草（注释、外键、命名推断，可选模型）。界面改用下面的后台起草任务，这个留着；
同一个源有排队或进行中的后台任务时 409 catalog_draft_busy
- PUT /catalog/guidance 数据源级的起草要求（使用者补充的背景和要求，起草时交给模型）
- POST /catalog/draft-jobs 建后台起草任务（202）；GET /draft-jobs/active、/latest、/{job_id} 读任务；
POST /draft-jobs/{job_id}/cancel 停止，/retry 只重试没写好的表（data/catalog_jobs.py）
- POST /catalog/profile 同步剖析：对业务库发少量只读查询，核对关系、取码值候选（data/catalog_profile.py）
规则都在 data/catalog.py，这里只做取源、转换形状和把异常翻成状态码：版本不符 409，结构不合规、审阅路径
不对 422，源或表不存在 404。所有写操作记录请求头 X-Actor 的署名（自报、未认证）。剖析没开启、正在剖析 409。
起草任务的 409 带机读码（api/coded.py）：同一个源已有排队或进行中的任务、起草模型用不了、要停止的任务已经结束。
上传表格的源按当前快照的表结构回答（table_versions.resolve_source），和查询实际用的库是同一个版本。
"""
@@ -19,13 +24,17 @@ from __future__ import annotations
from typing import Any, Literal
from fastapi import APIRouter, Depends, Header, HTTPException
from pydantic import BaseModel, Field
from fastapi import APIRouter, Body, Depends, Header, HTTPException
from fastapi.exceptions import RequestValidationError
from pydantic import BaseModel, Field, ValidationError
from sqlalchemy.ext.asyncio import AsyncSession
from app.api.coded import PROFILE_BUSY, SCHEMA_MISSING, SNAPSHOT_TAMPERED, CodedHTTPException
from app.api.coded import (
CATALOG_DRAFT_BUSY, CATALOG_DRAFT_FINISHED, CATALOG_DRAFT_MODEL_UNAVAILABLE, PROFILE_BUSY, SCHEMA_MISSING,
SNAPSHOT_TAMPERED, CodedHTTPException,
)
from app.api.runs import actor_of
from app.data import catalog, catalog_impact, catalog_profile, introspect, table_versions
from app.data import catalog, catalog_impact, catalog_jobs, catalog_profile, introspect, table_versions
from app.data.engine import SnapshotTampered
from app.db.base import get_session
from app.db.models import DataSource
@@ -64,10 +73,26 @@ class CatalogPatchIn(CatalogPatchPreviewIn):
class CatalogDraftIn(BaseModel):
#: 要起草的表；不给或为空时按使用次数取前 20 张
tables: list[str] | None = Field(default=None, max_length=200)
#: 是否请助手的模型起草中文名、粒度、列的含义等
#: 是否请起草模型（设置里的起草模型，没设沿用助手的模型）写中文名、粒度、列的含义等
use_model: bool = False
class DraftJobIn(BaseModel):
#: 要起草的表；不给或为空时按使用次数取前 20 张。最多 500 张（超了 422，说明是中文）
tables: list[str] | None = None
#: 是否请起草模型写中文名、粒度、列的含义等
use_model: bool = False
#: 这一次的补充说明，和数据源级的起草要求一起交给模型（最多 1000 字）
note: str | None = None
#: PUT /catalog/guidance 的请求 {guidance: string}：数据源级的起草要求，空串（或 null）清除，最多 2000 字。必须写
#: 这一项：表名恰好叫 guidance 时，整份提交那张表目录的请求也落到这个路径上，不能把它当成「清除起草要求」。
#: 没写、不是文字都回 422，说明是中文的一句话（同超长）
GUIDANCE_MISSING = "请求里要写起草要求（guidance 一项，一段文字；要清除起草要求请传空串）"
GUIDANCE_NOT_TEXT = "起草要求要是一段文字（要清除起草要求请传空串）"
class CatalogProfileIn(BaseModel):
#: 要剖析的表；不给或为空时按使用次数取前 10 张目录里有待核实关系的表
tables: list[str] | None = Field(default=None, max_length=200)
@@ -100,7 +125,7 @@ def _item_value(notes: dict[str, Any], name: str) -> tuple[Any, str | None]:
def _summary_row(name: str, meta: dict[str, Any] | None, entry: catalog.CatalogEntry | None,
usage: int) -> dict[str, Any]:
usage: int, last_draft: dict[str, Any] | None = None) -> dict[str, Any]:
notes = entry.notes if entry else {}
label, label_status = _item_value(notes, "label")
kind, _ = _item_value(notes, "kind")
@@ -120,6 +145,9 @@ def _summary_row(name: str, meta: dict[str, Any] | None, entry: catalog.CatalogE
"version": entry.version if entry else 0,
"updated_at": _iso(entry.updated_at) if entry else None,
"updated_by": entry.updated_by if entry else None,
# 这张表最近一次起草任务里的那一项：{status, job_id, finished_at, error, model_error, blank_columns}，没起草过
# 是 null
"last_draft": last_draft,
}
@@ -180,15 +208,18 @@ async def list_catalog(source_id: str, session: AsyncSession = Depends(get_sessi
说明有没有截断，schema_total 是数据库里一共几张表（含视图），界面据此说明「只探查了前 200 张」。老缓存没记
总数时按探查到的张数算。
"""
_, source = await _resolved(session, source_id)
row, source = await _resolved(session, source_id)
cache = source.schema_cache or {}
tables = cache.get("tables") or {}
entries = await catalog.read_catalog(session, source.id)
usage = await catalog.table_usage(session, source)
drafts = await catalog_jobs.last_drafts(session, source.id)
order = {name: i for i, name in enumerate(tables)}
rows = [_summary_row(name, meta, entries.get(name), usage.get(name, 0)) for name, meta in tables.items()]
rows = [_summary_row(name, meta, entries.get(name), usage.get(name, 0), drafts.get(name))
for name, meta in tables.items()]
rows.sort(key=lambda r: (-r["usage"], order[r["table_name"]]))
rows += [_summary_row(name, None, entry, 0) for name, entry in sorted(entries.items()) if name not in tables]
rows += [_summary_row(name, None, entry, 0, drafts.get(name))
for name, entry in sorted(entries.items()) if name not in tables]
return {
"tables": rows,
"system_notes": catalog.system_notes_source(source),
@@ -196,6 +227,10 @@ async def list_catalog(source_id: str, session: AsyncSession = Depends(get_sessi
"schema_note": None if tables else introspect.why_empty(cache),
"schema_truncated": bool(cache.get("truncated")),
"schema_total": max(_int(cache.get("total")), len(tables)),
# 数据源级的起草要求（没有是 null）
"guidance": catalog_jobs.guidance_of(row),
# 排队或进行中的起草任务 {id, status, total, done}，界面据此显示进度；没有是 null
"active_job": await catalog_jobs.job_brief(session, await catalog_jobs.active_job(session, source.id)),
}
@@ -207,6 +242,156 @@ async def get_catalog_table(source_id: str, table: str,
return await _detail(session, source, key, entry)
async def _source_row(session: AsyncSession, source_id: str) -> DataSource:
row = await session.get(DataSource, source_id)
if row is None:
raise HTTPException(404, "数据源不存在，可能已被删除")
return row
@router.put("/{source_id}/catalog/guidance")
async def put_catalog_guidance(source_id: str, body: dict[str, Any] = Body(...),
x_actor: str | None = Header(default=None),
session: AsyncSession = Depends(get_session)) -> Any:
"""存数据源级的起草要求：去首尾空白，空串清除，最多 2000 字（超了 422）。上传表格的源存在它自己的数据源记录上。
要登记在 PUT /catalog/{table} 前面：不然「guidance」会被当成表名。反过来，表名恰好叫 guidance 时，整份提交那张表
目录的请求（{notes, if_version}，没有 guidance 这一项）也落到这里：照旧当作那张表的整份提交。"""
if "guidance" not in body and "notes" in body:
try:
table_payload = CatalogPutIn.model_validate(body)
except ValidationError as e:
raise RequestValidationError(e.errors()) from e
return await put_catalog_table(source_id, "guidance", table_payload, x_actor=x_actor, session=session)
if "guidance" not in body:
raise HTTPException(422, GUIDANCE_MISSING)
guidance = body["guidance"]
if guidance is not None and not isinstance(guidance, str):
raise HTTPException(422, GUIDANCE_NOT_TEXT)
row = await _source_row(session, source_id)
try:
value = await catalog_jobs.save_guidance(session, row, guidance)
except catalog_jobs.TextTooLong as e:
raise HTTPException(422, str(e)) from e
return {"guidance": value}
async def _ensure_model(session: AsyncSession) -> None:
"""起草模型用不了（没有接入、停用、只有演示接入……）：409，不建任务。说法同同步起草时的 model_error。"""
try:
await catalog_jobs.check_model(session)
except catalog.CatalogModelUnavailable as e:
raise CodedHTTPException(409, str(e), CATALOG_DRAFT_MODEL_UNAVAILABLE) from e
def _no_schema(cache: dict[str, Any]) -> HTTPException:
return HTTPException(409, f"{introspect.why_empty(cache)}，无法起草数据目录。请先在数据源卡片上点「探查结构」")
@router.post("/{source_id}/catalog/draft-jobs", status_code=202)
async def create_draft_job(source_id: str, payload: DraftJobIn, x_actor: str | None = Header(default=None),
session: AsyncSession = Depends(get_session)) -> dict[str, Any]:
"""建后台起草任务，马上返回（202，任务是排队中）。同一个源已有排队或进行中的任务 409 catalog_draft_busy（界面改去
GET /draft-jobs/active 显示那一个）；用模型而起草模型用不了 409 catalog_draft_model_unavailable；没有表结构 409。
表结构里没有的表照样建进任务、状态 skipped。"""
if payload.tables and len(payload.tables) > catalog_jobs.MAX_TABLES:
raise HTTPException(422, f"一次最多起草 {catalog_jobs.MAX_TABLES} 张表，当前 {len(payload.tables)} 张，"
"请分几次起草")
try:
note = catalog_jobs.clean_text(payload.note, catalog_jobs.NOTE_MAX, "本次说明")
except catalog_jobs.TextTooLong as e:
raise HTTPException(422, str(e)) from e
row, source = await _resolved(session, source_id)
try:
await catalog_jobs.ensure_idle(session, row)
except catalog_jobs.DraftBusy as e:
raise CodedHTTPException(409, str(e), CATALOG_DRAFT_BUSY) from e
cache = source.schema_cache or {}
if not cache.get("tables"):
raise _no_schema(cache)
if payload.use_model:
await _ensure_model(session)
try:
job, tasks = await catalog_jobs.create_job(session, row, source, tables=payload.tables or None,
use_model=payload.use_model, note=note, actor=actor_of(x_actor))
except catalog_jobs.DraftBusy as e:
raise CodedHTTPException(409, str(e), CATALOG_DRAFT_BUSY) from e
catalog_jobs.manager.start(job.id)
return {"job": catalog_jobs.job_dict(job, tasks)}
@router.get("/{source_id}/catalog/draft-jobs/active")
async def active_draft_job(source_id: str, session: AsyncSession = Depends(get_session)) -> dict[str, Any]:
"""排队或进行中的那个起草任务，没有是 null。"""
row = await _source_row(session, source_id)
return {"job": await catalog_jobs.job_out(session, await catalog_jobs.active_job(session, row.id))}
@router.get("/{source_id}/catalog/draft-jobs/latest")
async def latest_draft_job(source_id: str, session: AsyncSession = Depends(get_session)) -> dict[str, Any]:
"""最近建的一个起草任务（任何状态），没有是 null。"""
row = await _source_row(session, source_id)
return {"job": await catalog_jobs.job_out(session, await catalog_jobs.latest_job(session, row.id))}
async def _job_or_404(session: AsyncSession, source_id: str, job_id: str) -> tuple[DataSource, Any]:
row = await _source_row(session, source_id)
job = await catalog_jobs.load(session, row.id, job_id)
if job is None:
raise HTTPException(404, "起草任务不存在，可能属于别的数据源，或者已随数据源删除")
return row, job
@router.get("/{source_id}/catalog/draft-jobs/{job_id}")
async def get_draft_job(source_id: str, job_id: str, session: AsyncSession = Depends(get_session)) -> dict[str, Any]:
_, job = await _job_or_404(session, source_id, job_id)
return {"job": await catalog_jobs.job_out(session, job)}
@router.post("/{source_id}/catalog/draft-jobs/{job_id}/cancel")
async def cancel_draft_job(source_id: str, job_id: str,
session: AsyncSession = Depends(get_session)) -> dict[str, Any]:
"""停止：排队的直接停止；进行中的停掉正在进行的模型调用，没开始和没写完的表记停止，已经写进目录的不回滚。
已经结束的 409 catalog_draft_finished。返回停止之后的任务。"""
row, job = await _job_or_404(session, source_id, job_id)
owner = row.id
try:
await catalog_jobs.cancel(session, job)
except catalog_jobs.DraftFinished as e:
raise CodedHTTPException(409, str(e), CATALOG_DRAFT_FINISHED) from e
session.expire_all() # 后台在别的会话里收的尾：重新读
return {"job": await catalog_jobs.job_out(session, await catalog_jobs.load(session, owner, job_id))}
@router.post("/{source_id}/catalog/draft-jobs/{job_id}/retry", status_code=202)
async def retry_draft_job(source_id: str, job_id: str, x_actor: str | None = Header(default=None),
session: AsyncSession = Depends(get_session)) -> dict[str, Any]:
"""重试：新任务，只含原任务里没写好（failed、partial）和停止了（cancelled）的表；用不用模型、本次说明沿用，起草要求
取数据源当前的。有排队或进行中的任务 409 catalog_draft_busy；没有可重试的表 422；起草模型用不了 409。"""
_, old = await _job_or_404(session, source_id, job_id)
row, source = await _resolved(session, source_id)
try:
await catalog_jobs.ensure_idle(session, row)
except catalog_jobs.DraftBusy as e:
raise CodedHTTPException(409, str(e), CATALOG_DRAFT_BUSY) from e
try:
tables = await catalog_jobs.retry_tables(session, old)
except catalog_jobs.NothingToRetry as e:
raise HTTPException(422, str(e)) from e
cache = source.schema_cache or {}
if not cache.get("tables"):
raise _no_schema(cache)
if old.use_model:
await _ensure_model(session)
try:
job, tasks = await catalog_jobs.create_job(session, row, source, tables=tables, use_model=old.use_model,
note=old.note, actor=actor_of(x_actor), retry_of=old.id)
except catalog_jobs.DraftBusy as e:
raise CodedHTTPException(409, str(e), CATALOG_DRAFT_BUSY) from e
catalog_jobs.manager.start(job.id)
return {"job": catalog_jobs.job_dict(job, tasks)}
@router.put("/{source_id}/catalog/{table}")
async def put_catalog_table(source_id: str, table: str, payload: CatalogPutIn,
x_actor: str | None = Header(default=None),
@@ -228,17 +413,26 @@ async def put_catalog_table(source_id: str, table: str, payload: CatalogPutIn,
async def draft_catalog(source_id: str, payload: CatalogDraftIn,
x_actor: str | None = Header(default=None),
session: AsyncSession = Depends(get_session)) -> dict[str, Any]:
"""同步起草。返回每张表新增、更新、删除了几项；模型用不了或某张表的模型起草失败，照实写在结果里。
"""同步起草（界面改用后台起草任务，这个接口留着，返回形状不变）。引擎同后台任务（分段、按失败原因处理），起草要求
取数据源级的那份。返回每张表新增、更新、删除了几项；模型用不了或某张表的模型起草失败，照实写在结果里。
model_items：模型给了这张表几项可用内容（没用模型、模型这部分失败时为 null）。界面据此区分「模型给了内容、
只是和现有目录一致」（model_items > 0、三个数都是 0）和「模型没给内容」（model_error 有值）。
同一个源有排队或进行中的后台起草任务时 409 catalog_draft_busy（和建任务同一句话）：两边同时写同一个源的目录、
抢模型名额。
"""
_, source = await _resolved(session, source_id)
row, source = await _resolved(session, source_id)
try:
await catalog_jobs.ensure_idle(session, row)
except catalog_jobs.DraftBusy as e:
raise CodedHTTPException(409, str(e), CATALOG_DRAFT_BUSY) from e
cache = source.schema_cache or {}
if not cache.get("tables"):
raise HTTPException(409, f"{introspect.why_empty(cache)}，无法起草数据目录。请先在数据源卡片上点「探查结构」")
raise _no_schema(cache)
report = await catalog.draft_catalog(session, source, tables=payload.tables or None,
use_model=payload.use_model, actor=actor_of(x_actor))
use_model=payload.use_model, actor=actor_of(x_actor),
guidance=catalog_jobs.guidance_of(row))
rows = [{"table_name": r.table, "added": r.added, "updated": r.updated, "removed": r.removed,
"version": r.version, "error": r.error, "model_error": r.model_error, "model_items": r.model_items}
for r in report.tables]
backend/app/api/coded.py
+5
View File
@@ -19,6 +19,11 @@ DATASOURCE_INACTIVE = "datasource_inactive"
SCHEMA_MISSING = "schema_missing"
PROFILE_BUSY = "profile_busy"
SNAPSHOT_TAMPERED = "snapshot_tampered"
#: 数据目录的后台起草任务（api/catalog.py 的 /catalog/draft-jobs）：同一个源已有排队或进行中的任务（409，界面改去
#: GET /draft-jobs/active 显示那一个）、起草模型用不了（409，不建任务）、要停止的任务已经结束了（409）
CATALOG_DRAFT_BUSY = "catalog_draft_busy"
CATALOG_DRAFT_MODEL_UNAVAILABLE = "catalog_draft_model_unavailable"
CATALOG_DRAFT_FINISHED = "catalog_draft_finished"
class CodedHTTPException(HTTPException):
backend/app/api/datasources.py
+22 -4
View File
@@ -27,8 +27,8 @@ from app.data.catalog_profile import profile_settings_rejection
from app.data import introspect as introspect_mod
from app.data import table_versions
from app.data.engine import (
MASK_COLUMNS_OPTION, MAX_QUERY_TIMEOUT_S, QUERY_TIMEOUT_OPTION, SUPPORTED_KINDS, SnapshotTampered, build_url,
engine_args, engines, mask_columns_problem, query_timeout_problem,
CATALOG_DRAFT_GUIDANCE_OPTION, MASK_COLUMNS_OPTION, MAX_QUERY_TIMEOUT_S, QUERY_TIMEOUT_OPTION, SUPPORTED_KINDS,
SnapshotTampered, build_url, engine_args, engines, mask_columns_problem, query_timeout_problem,
)
from app.db.base import get_session, new_id
from app.db.models import DataSource, ImportStaging, SourceSnapshot, TableImport, TableRecipe
@@ -124,6 +124,22 @@ def _import_mode(row: DataSource) -> str | None:
return "simple" if _is_upload(row) else None
#: options 里由别的接口维护的项：数据源表单不显示、不提交它们，整份回写 options 时原样留着（同模型接入 extra 里的
#: last_check）。数据目录的起草要求只经 PUT /catalog/guidance 改（api/catalog.py）：表单把认不得的项放进「高级参数」
#: 的单行输入框，多行的要求会被压成一行
_MANAGED_OPTIONS = (CATALOG_DRAFT_GUIDANCE_OPTION,)
def _shown_options(options: dict[str, Any] | None) -> dict[str, Any]:
return {k: v for k, v in (options or {}).items() if k not in _MANAGED_OPTIONS}
def _with_managed(options: dict[str, Any], saved: dict[str, Any] | None) -> dict[str, Any]:
"""表单提交的 options + 存着的、由别的接口维护的项（提交里带了也不认）。"""
kept = {k: v for k, v in (saved or {}).items() if k in _MANAGED_OPTIONS}
return {**_shown_options(options), **kept}
def _to_out(row: DataSource, snapshot: dict[str, Any] | None = None,
recipe_info: dict[str, Any] | None = None) -> DataSourceOut:
cache = row.schema_cache or {}
@@ -137,7 +153,7 @@ def _to_out(row: DataSource, snapshot: dict[str, Any] | None = None,
schema_error=str(cache.get("error") or "") if cache.get("failed") else "",
available_schemas=list(cache.get("available_schemas") or []),
id=row.id, name=row.name, kind=row.kind, host=row.host, port=row.port,
database=row.database, username=row.username, options=row.options or {},
database=row.database, username=row.username, options=_shown_options(row.options),
readonly=row.readonly, description=row.description, enabled=row.enabled,
password_masked=mask(row.password),
has_password=bool(row.password),
@@ -429,7 +445,8 @@ async def create_source(
_refuse_reserved(payload.kind, payload.database)
row = DataSource(
**payload.model_dump(exclude={"password"}),
**payload.model_dump(exclude={"password", "options"}),
options=_with_managed(payload.options, None),
password=encrypt(payload.password),
)
_settle_oracle(row, None, None)
@@ -472,6 +489,7 @@ async def update_source(
raise HTTPException(422, _UPLOAD_DIR_READONLY)
if "options" in data:
_refuse_bad_options(data["options"])
data["options"] = _with_managed(data["options"] or {}, row.options)
if "password" in data:
# 空串表示清空，非空表示换新的
data["password"] = encrypt(data["password"]) if data["password"] else None
backend/app/api/settings.py
+43 -2
View File
@@ -14,6 +14,7 @@ from app.api.coded import CodedHTTPException
from app.core.errors import explain, not_configured, raw
from app.core.config import settings as app_settings
from app.core.crypto import encrypt, mask
from app.data.draft_model import CATALOG_DRAFT_SETTING_KEY, setting_pair
from app.db.base import get_session
from app.db.models import Provider, Setting
from app.engine.judge import JUDGE_DEFAULTS, MAX_TOKENS as JUDGE_MAX_TOKENS
@@ -569,6 +570,39 @@ async def verify_model_spec(session: AsyncSession, node_config: dict[str, Any] |
return ModelSpec(provider=provider, model=model, max_tokens=JUDGE_MAX_TOKENS, thinking="off")
# --------------------------------------------------------------------------
# 起草模型：数据目录的「用模型起草」和按配方导入的 AI 起草用哪个模型
# --------------------------------------------------------------------------
#: 设置里 catalog_draft 这一组：{provider, model}。都留空时沿用助手的模型，再没有就是默认接入（取值顺序见
#: data/draft_model.py，两处起草共用）。只放接入和模型：额度、思考由两处起草各自定
CATALOG_DRAFT_GROUP = CATALOG_DRAFT_SETTING_KEY
CATALOG_DRAFT_DEFAULTS: dict[str, Any] = {"provider": None, "model": None}
#: 设置页「起草模型」一节各项的叫法：报错时写它，不写键名
CATALOG_DRAFT_LABEL = {"provider": "起草模型 · 接入", "model": "起草模型 · 模型"}
def catalog_draft_problem(value: Any) -> str | None:
"""设置页写进来的 catalog_draft 这一组有什么问题，没问题返回 None。点名的接入在不在由 _provider_problem 查。"""
if not isinstance(value, dict):
return f"「起草模型」设置格式有误，需要是一个对象，当前为「{value}」"
for key, item in value.items():
if key not in CATALOG_DRAFT_DEFAULTS:
return (f"「起草模型」设置中不支持「{key}」，可设置的项："
+ "、".join(f"「{CATALOG_DRAFT_LABEL[k]}」" for k in CATALOG_DRAFT_DEFAULTS))
if item is not None and not isinstance(item, str):
# 「跟随助手的模型」是设置页接入下拉框空选项的原话
what = "模型接入的名称" if key == "provider" else "模型 ID"
return f"「{CATALOG_DRAFT_LABEL[key]}」需要填写{what}，留空表示跟随助手的模型，当前为「{item}」"
return None
def catalog_draft_values(stored: Any) -> dict[str, Any]:
"""库里存的 catalog_draft 一组 → 补齐默认、规范化（去掉首尾空白，空串和存坏了的当没填）。"""
provider, model = setting_pair(stored)
return {"provider": provider, "model": model}
# --------------------------------------------------------------------------
# 通知：定时任务推送 Webhook 时拼运行链接用的对外访问地址
# --------------------------------------------------------------------------
@@ -661,6 +695,8 @@ DEFAULT_SETTINGS: dict[str, Any] = {
VERIFY_GROUP: verify_values(None),
# 通知：定时任务的 Webhook 里拼运行链接用的对外访问地址
NOTIFY_GROUP: notify_values(None),
# 起草模型：数据目录起草、按配方导入的 AI 起草用的模型，留空沿用助手的模型
CATALOG_DRAFT_GROUP: catalog_draft_values(None),
}
@@ -717,6 +753,7 @@ async def get_settings(session: AsyncSession = Depends(get_session)) -> dict[str
merged[JUDGE_GROUP] = judge_values(stored.get(JUDGE_GROUP))
merged[VERIFY_GROUP] = verify_values(stored.get(VERIFY_GROUP))
merged[NOTIFY_GROUP] = notify_values(stored.get(NOTIFY_GROUP))
merged[CATALOG_DRAFT_GROUP] = catalog_draft_values(stored.get(CATALOG_DRAFT_GROUP))
for key, value in stored.items():
merged.setdefault(key, value)
return merged
@@ -731,8 +768,8 @@ PROVIDER_INVALID = "settings_provider_invalid"
def _provider_fields() -> dict[tuple[str, str], tuple[str, str | None]]:
"""设置里点名模型接入的项：(组, 键) → (这一项在界面上的叫法, 同一组模型那一项的叫法)。门控、裁判、核查的叫法和
设置页一致；助手的模型由「编排」页助手栏的「助手使用的模型」单独设定（PUT /api/copilot/model），只有这一个下拉框
"""设置里点名模型接入的项：(组, 键) → (这一项在界面上的叫法, 同一组模型那一项的叫法)。门控、裁判、核查、起草模型的
叫法和设置页一致；助手的模型由「编排」页助手栏的「助手使用的模型」单独设定（PUT /api/copilot/model），只有这一个下拉框
（模型那一项为 None），这里只防经设置接口写进去的那一份。"""
from app.api.copilot import COPILOT_SETTING_KEY # 在这里导入：copilot 模块很大，设置模块被引擎各处导入
from app.providers.factory import ASSISTANT_MODEL_FIELD
@@ -741,6 +778,7 @@ def _provider_fields() -> dict[tuple[str, str], tuple[str, str | None]]:
("run", "tool_gate_provider"): ("门控模型 · 接入", "门控模型 · 模型"),
(JUDGE_GROUP, "provider"): (JUDGE_LABEL["provider"], JUDGE_LABEL["model"]),
(VERIFY_GROUP, "provider"): (VERIFY_SETTING_LABEL["provider"], VERIFY_SETTING_LABEL["model"]),
(CATALOG_DRAFT_GROUP, "provider"): (CATALOG_DRAFT_LABEL["provider"], CATALOG_DRAFT_LABEL["model"]),
(COPILOT_SETTING_KEY, "provider"): (ASSISTANT_MODEL_FIELD, None),
}
@@ -798,6 +836,9 @@ async def put_settings(
raise HTTPException(422, problem)
if VERIFY_GROUP in payload.values and (problem := verify_problem(payload.values[VERIFY_GROUP])):
raise HTTPException(422, problem)
if CATALOG_DRAFT_GROUP in payload.values and (
problem := catalog_draft_problem(payload.values[CATALOG_DRAFT_GROUP])):
raise HTTPException(422, problem)
if NOTIFY_GROUP in payload.values:
if problem := notify_problem(payload.values[NOTIFY_GROUP]):
raise HTTPException(422, problem)
backend/app/data/catalog.py
+332 -246
View File
@@ -591,21 +591,41 @@ class MergeStats:
return MergeStats(self.added + other.added, self.updated + other.updated, self.removed + other.removed)
def _about_gone_columns(key: Slot, item: Mapping[str, Any], columns: set[str]) -> bool:
"""这一项说的列已经不在表结构里了：列级的项看列名，业务主键、业务日期看它们指着的列。"""
if key[0] == "c":
return key[1] not in columns
value = item.get("value")
if key == ("t", "keys") and isinstance(value, list):
return any(str(col) not in columns for col in value)
if key == ("t", "business_date") and isinstance(value, Mapping) and value.get("column") is not None:
return str(value["column"]) not in columns
return False
def merge_notes(existing: Mapping[str, Any] | None, draft: Mapping[str, Any] | None, *,
covered: Iterable[str] = (), at: str | None = None) -> tuple[dict[str, Any], MergeStats]:
covered: Iterable[str] = (), at: str | None = None,
columns: Iterable[str] | None = None) -> tuple[dict[str, Any], MergeStats]:
"""把起草结果并入已有目录，返回 (新目录, 计数)。不改入参。
规则（逐个槽位）：
- 已有项是 confirmed 或 rejected：不动。驳回的留着，同一项下次起草就不会再冒出来。
- 起草结果里没有这一项：已有项的来源在 covered 里（这一轮完整算过这个来源）就删掉，否则留着。
外键约束被删掉以后，原来那条「有确证」的关系不能还挂着；而这一轮没调模型，就不能因此删掉模型起草的项。
- 起草结果里没有这一项：
- 模型起草的项（llm）留着。模型这次留空、给的值没通过校验（_from_model 里就丢掉了），都不能把以前起草好的
删掉——实测整库重跑一次，模型这次没给的几百项全被删掉，有的表覆盖率从八成多掉到五成。只有这一轮完整起草过
（covered 含 llm）、而它说的列已经不在表结构里（columns：表结构里的列名；业务主键、业务日期指着的列也算）
才删；不知道表结构（columns 为 None）时不删。
- 别的来源：来源在 covered 里（这一轮完整算过这个来源）就删掉，否则留着。外键约束被删掉以后，原来那条「有确证」
的关系不能还挂着；库注释改了，旧的那条注释也不能还挂着。
- 同一来源：值或状态有变化就更新。数据剖析写的备注只是日期变了不算变化（same_profile_note），保留旧的项。
- 不同来源：来源更可信的（_SOURCE_RANK）顶掉不如它的，反过来不行。
新增和更新的项盖上 at（缺省为当前时间）作为 updated_at；内容没变的项保持原样，修改时间也不动。
removed 只数真删掉的项。
"""
at = at or now_iso()
covered = set(covered)
known = {str(c) for c in columns} if columns is not None else None
old, new = _slots(existing), _slots(draft)
out: dict[Slot, dict[str, Any]] = {}
added = updated = removed = 0
@@ -614,7 +634,11 @@ def merge_notes(existing: Mapping[str, Any] | None, draft: Mapping[str, Any] | N
if item.get("status") in _LOCKED:
out[key] = item
elif cand is None:
if item.get("source") in covered:
if item.get("source") == "llm":
gone = "llm" in covered and known is not None and _about_gone_columns(key, item, known)
else:
gone = item.get("source") in covered
if gone:
removed += 1
else:
out[key] = item
@@ -1130,10 +1154,15 @@ async def save_patch(session: AsyncSession, source_id: str, table: str, changes:
@dataclass(frozen=True)
class TableDraft:
"""一张表的起草结果。covered：这一轮完整算过的来源（merge_notes 据此删掉这些来源里已经没有的旧项）。"""
"""一张表的起草结果。covered：这一轮完整算过的来源（merge_notes 据此删掉这些来源里已经没有的旧项；模型起草的项
只在列已经不在表结构里时才删）。
error：模型起草只写出了一部分（有的段没写出来）时，没写出来的是哪部分、为什么。这时 covered 不含 llm：没写全
就不算完整起草过。"""
notes: dict[str, Any]
covered: frozenset[str] = frozenset()
error: str | None = None
def system_notes_source(source: Any) -> bool:
@@ -1176,6 +1205,34 @@ def _comment_is_label(text: str) -> bool:
return len(text) <= _LABEL_MAX and not _SENTENCE_MARKS.search(text)
#: 标识符写法的注释：只有字母、数字、下划线、点，不带空格（「member_card」「ord.status」「GMV」）
_IDENTIFIER = re.compile(r"[A-Za-z0-9_.]+")
#: 比较注释和名字时忽略的字符：下划线、连字符、空白
_NAME_FILLER = re.compile(r"[\s_\-]+")
def _bare_name(text: str) -> str:
return _NAME_FILLER.sub("", text).lower()
def comment_role(text: str, name: str) -> str | None:
"""数据库注释在目录里记成什么：label 中文名；note 一段解释（表记进说明，列记进含义）；None 不记。
- 注释就是名字本身（和表名、列名相同，忽略大小写、下划线、连字符和空白；表名带 schema 前缀的也比不带前缀的），
或者是标识符写法（只有字母、数字、下划线、点，不带空格）：说明不了什么，不记。以前这种注释短、不带句读，就当
中文名记下了；库注释的级别又比模型起草高，模型起草的中文名永远进不了目录。交给模型的表结构里照旧有这条注释；
- 短、不带句读的是名字（「实收金额」）；
- 否则是一段解释（「状态：1 表示有效…」）。
只影响以后的起草：已有的目录不另做迁移，起草到那张表时按这一轮的注释重新算（旧的那条随之删掉）。
"""
bare = _bare_name(text)
names = {name, name.rsplit(".", 1)[-1]}
if not bare or any(bare == _bare_name(n) for n in names) or _IDENTIFIER.fullmatch(text):
return None
return "label" if _comment_is_label(text) else "note"
def _is_key(meta: Mapping[str, Any], columns: list[str]) -> bool:
"""这几列恰好是这张表的主键或某个唯一约束（不分大小写、不计顺序）。"""
wanted = {str(c).lower() for c in columns}
@@ -1367,7 +1424,8 @@ def draft_structure(source: Any, table: str) -> TableDraft:
"""只看表结构就能起草的部分：数据库注释、外键约束、命名推断。table 必须是 schema_cache["tables"] 的键。
- 数据库注释 → 中文名或含义（comment / proposed）：短而不带句读的是名字，记进表或列的中文名；否则是
一段解释，记进表的说明或列的含义。导入表格的源例外（system_notes_source）：说明是系统生成的，不拷。
一段解释，记进表的说明或列的含义；就是名字本身、标识符写法的不记（comment_role）。导入表格的源例外
（system_notes_source）：说明是系统生成的，不拷。
- 外键约束 → 关系（fk / verified）。
- 命名推断 → 关系（name / proposed）。
"""
@@ -1378,13 +1436,15 @@ def draft_structure(source: Any, table: str) -> TableDraft:
notes: dict[str, Any] = {}
if not system_notes_source(source):
comment = (meta.get("comment") or "").strip()
if comment:
notes["label" if _comment_is_label(comment) else "description"] = make_item(comment, "comment")
role = comment_role(comment, table) if comment else None
if role is not None:
notes["label" if role == "label" else "description"] = make_item(comment, "comment")
columns: dict[str, dict[str, Any]] = {}
for col in meta.get("columns") or []:
text = (col.get("comment") or "").strip()
if text:
columns[col["name"]] = {("label" if _comment_is_label(text) else "meaning"): make_item(text, "comment")}
role = comment_role(text, str(col["name"])) if text else None
if role is not None:
columns[col["name"]] = {("label" if role == "label" else "meaning"): make_item(text, "comment")}
if columns:
notes["columns"] = columns
fks = fk_relations(cache, table)
@@ -1401,15 +1461,39 @@ def draft_structure(source: Any, table: str) -> TableDraft:
# 起草：模型（可选，按表分批，结构化输出）
# ==========================================================================
#: 每批最多几张表、合计多少列。一批太大模型容易漏表、输出被截断；太小则调用次数多
#: 每次调用最多几张表、合计多少列（data/catalog_draft.py 按它分段）。实测每列约 50 个输出 token（不算思考），
#: 会思考的模型思考也算在输出额度里：一段太大，想完就没额度写正文。列数超过 MODEL_BATCH_COLUMNS 的宽表拆成
#: 「表级一段 + 列分段」，列分段数是列数除以它向上取整，各段列数平均（相差不超过 1），不留很小的尾段
MODEL_BATCH_TABLES = 4
MODEL_BATCH_COLUMNS = 120
#: 同时发出的批数
MODEL_BATCH_COLUMNS = 40
#: 一次起草里额度用完或超时累计这么多次，之后新开始的段列数上限减半（MODEL_BATCH_COLUMNS → 一半 → …），
#: 最低 MODEL_MIN_PART_COLUMNS 列
MODEL_SHRINK_AFTER = 2
MODEL_MIN_PART_COLUMNS = 10
#: 一段（连同它拆出来的、补问的、重试的）最多调用几次模型，到了次数还没写完的记没写出来
MODEL_PART_MAX_CALLS = 8
#: 提醒：最近 MODEL_NOTICE_WINDOW 次调用里有 MODEL_NOTICE_ON 次以上没写出来时提醒换模型；之后少于 MODEL_NOTICE_OFF
#: 次时清掉
MODEL_NOTICE_WINDOW = 6
MODEL_NOTICE_ON = 4
MODEL_NOTICE_OFF = 2
#: 全进程同时进行的模型调用最多几个（所有起草任务、同步起草共用）
MODEL_CONCURRENCY = 3
#: 一批最多等多久
MODEL_TIMEOUT_S = 120
#: 输出额度。一批四张表的目录远用不完；给太大的额度有的服务方会直接拒收
MODEL_MAX_TOKENS = 8192
#: 单次调用最多等多久（秒）。会思考的模型想得久，一段 40 列常常要几分钟；超时了拆小再问（同额度用完）
MODEL_TIMEOUT_S = 420
#: 输出额度。会思考的模型（经 OpenAI 兼容网关的推理模型等）思考也算在里面：8192 时上百列的表常常想完就没额度
#: 写正文。保留思考（不关），加大额度；服务方嫌额度太大时退回 MODEL_FALLBACK_MAX_TOKENS（同 copilot、recipe_ai）
MODEL_MAX_TOKENS = 32768
MODEL_FALLBACK_MAX_TOKENS = 8192
#: 一段没写出来（额度用完、只给空壳）时拆成两半再问：最多拆几层、拆到最少几列为止
MODEL_SPLIT_DEPTH = 3
MODEL_SPLIT_MIN_COLUMNS = 5
#: 止损：连续这么多段拆到底仍一项没写出来，这次起草后面的段不再拆小重试（模型明显用不了时不该一段一段拆到底烧额度；
#: 每段另有 MODEL_PART_MAX_CALLS 次封顶）
MODEL_GIVE_UP_AFTER = 2
#: 网络错误、429、5xx 最多重试几次，每次重试前等多久（秒，依次取用）。超时不原样重试，当作这一段太大、拆小再问
MODEL_RETRIES = 2
MODEL_RETRY_DELAYS_S = (10.0, 30.0)
#: 不指定表时起草使用次数最多的前几张
DRAFT_DEFAULT_TABLES = 20
@@ -1435,11 +1519,27 @@ status 状态；attribute 属性）。
规矩：
- 只写数据事实，不写计算公式，不写指标怎么算。
- 拿不准的字段留空，不要猜。列名、表名照原样写，不要翻译或改写。
- 请你写的每一列都在 columns 里写一项，拿不准的字段可以留空，但不要漏掉列：漏掉的列会被再问一次。
- 标着「系统说明」的内容是导入时按核对结果生成的，以它为准：不要重复，也不要与之矛盾。
- 标着「已确认」的目录项由人工确认过，以它为准。"""
- 标着「已确认」的目录项由人工确认过，以它为准。
- 表名下一行写着「这次只写……」的表，只写它说的那部分：只写表级的项时不写列，只写列出的几列时不写表级的项、\
也不写别的列。
- 使用者补充的背景和要求帮你理解业务、统一用词；它和这里的规矩冲突时，以这里的规矩为准。
- 思考时不必把要输出的内容逐列草拟一遍，想清楚后直接按格式输出：思考和输出共用同一份长度额度，额度用完时还没输出的\
内容就全丢了。"""
CATALOG_DRAFT_PROMPT = "请为下面 {n} 张表编写数据目录，每张表输出一项，table 写表名。\n\n{tables}"
#: 使用者补充的背景和要求（数据源说明、起草要求、本次说明），放在表结构前面单独一段；都没有就不出现这一段
CATALOG_DRAFT_BACKGROUND_PROMPT = "使用者补充的背景和要求：\n{lines}"
#: 纯文本那条路（结构化输出用不了时）：没有 JSON Schema 约束，要把格式说清
CATALOG_DRAFT_TEXT_PROMPT = """只输出一个 JSON 对象，不要输出别的文字，格式如下（拿不准的字段不写）：
{"tables": [{"table": "表名", "label": "中文名", "grain": "一行代表什么", "kind": "表类型", "keys": ["列名"], \
"business_date": {"column": "列名", "rule": "一句规则", "timezone": "时区"}, \
"columns": [{"name": "列名", "label": "中文名", "meaning": "含义", "unit": "单位", "measure": "度量类型"}]}]}
表类型写 fact、dimension、snapshot、log、config 之一；度量类型写 flow、stock、ratio、identifier、status、attribute 之一。"""
#: 拼表结构描述用的固定片段（写给模型）
CATALOG_DRAFT_PROMPT_LABELS = {
"table": "表 {name}",
@@ -1455,6 +1555,16 @@ CATALOG_DRAFT_PROMPT_LABELS = {
"col_system": "系统说明：{text}",
"unique": "唯一约束：{cols}",
"confirmed": "已确认的目录项：",
# 分段起草：表名下一行说这次只写哪部分（宽表的表级一段、列分段，拆小重问时的半段）
"only_table": "这次只写表级的项（中文名、粒度、表类型、业务主键、业务日期），不写列",
"only_columns": "这次只写下面列出的 {n} 列（这张表共 {total} 列），不写表级的项",
"table_and_columns": "这次只写表级的项和这 {n} 列：{cols}",
"brief_columns": "列（共 {n} 列，这里只列名字和类型）：",
"drafted": "已起草的表级内容：",
# 使用者补充的背景和要求
"description": "数据源说明：{text}",
"guidance": "起草要求：{text}",
"note": "本次说明：{text}",
}
CATALOG_DRAFT_PROMPT_SCHEMA: dict[str, Any] = {
@@ -1499,99 +1609,69 @@ CATALOG_DRAFT_PROMPT_SCHEMA: dict[str, Any] = {
"required": ["tables"],
}
_P = CATALOG_DRAFT_PROMPT_LABELS
class DraftModel:
"""起草用的模型：先按 MODEL_MAX_TOKENS 的额度调；服务方嫌额度太大（budget_rejected）时换成 MODEL_FALLBACK_MAX_TOKENS
的那一个，之后的调用都用它（shrink）。一次起草（一个起草任务、一次同步起草）共用一个，退回之后不再先失败一次。
只转发起草用到的两个方法：with_structured_output 每次都取当前那个模型的，ainvoke 同理。"""
def __init__(self, big: Any, small: Any | None, model_name: str = "") -> None:
self.model, self._small = big, small
self.model_name = model_name
self.budget = MODEL_MAX_TOKENS
def shrink(self) -> bool:
"""换成小额度的模型。已经换过、没有小的，返回 False。"""
if self._small is None or self.model is self._small:
return False
self.model, self.budget = self._small, MODEL_FALLBACK_MAX_TOKENS
return True
def with_structured_output(self, schema: Any, **kwargs: Any) -> Any:
return self.model.with_structured_output(schema, **kwargs)
async def ainvoke(self, messages: list[Any]) -> Any:
return await self.model.ainvoke(messages)
async def resolve_draft_model(session: AsyncSession) -> tuple[Any, str]:
"""(模型, 模型 id)：用设置里助手的模型，关掉思考、温度 0。没有可用的模型抛 CatalogModelUnavailable。
"""(模型, 模型 id)：用哪个接入、哪个模型按 data/draft_model.py 的顺序（设置里的起草模型 > 助手的模型 > 默认接入），
和按配方导入的 AI 起草共用。没有可用的模型抛 CatalogModelUnavailable。
不关思考（spec 里不传 thinking，按接入默认）、温度 0、额度 MODEL_MAX_TOKENS，服务方嫌大时退回
MODEL_FALLBACK_MAX_TOKENS（DraftModel）。单次调用的时限 MODEL_TIMEOUT_S 也交给 SDK，免得它先按全局的 5 分钟断开。
SDK 自己不重试（max_retries=0）：429、5xx、网络错误由起草引擎退避重试（最多 MODEL_RETRIES 次）；超时当作这一段
太大，拆小再问（data/catalog_draft.py），不原样重试。不然两层叠起来一段最多发 9 次请求，报错却写「已重试 2 次」，
网关正在限流时还会放大流量。
只建模型对象，不发请求。
和 recipe_ai.resolve_draft_model 一样不用 get_chat_model：它在没有接入时静默退回演示模型，拿演示模型
起草出来的目录会被当成真的写进库里。数据层不导入接口层，设置键取 recipe_ai 里那份（与助手的同值，
那边有测试核对）。
起草出来的目录会被当成真的写进库里。
"""
from app.data.recipe_ai import COPILOT_SETTING_KEY
from app.db.models import Setting
from app.providers.factory import (
ModelSpec, ProviderNotConfigured, assistant_unavailable, build_chat_model, resolve_provider,
)
from app.data.draft_model import DraftModelUnavailable, draft_unavailable, resolve_draft_provider
from app.providers.factory import ModelSpec, ProviderNotConfigured, build_chat_model
row = await session.get(Setting, COPILOT_SETTING_KEY)
saved = (row.value if row else None) or {}
spec = ModelSpec(provider=saved.get("provider") or None, model=saved.get("model") or None,
max_tokens=MODEL_MAX_TOKENS, temperature=0, thinking="off")
try:
# 存的接入名不存在、已停用，只存了模型名、提供它的接入已停用：resolve_provider 都会报出来
provider = await resolve_provider(session, spec.provider, spec.model)
except ProviderNotConfigured as e:
raise CatalogModelUnavailable(assistant_unavailable(e)) from e
if provider is None:
raise CatalogModelUnavailable("未配置模型接入，无法由助手起草数据目录。请到「设置 → 模型接入」添加")
if provider.kind == "mock":
raise CatalogModelUnavailable("当前只有演示用的模型接入，无法用于起草数据目录。"
"请到「设置 → 模型接入」添加真实的模型接入")
provider, choice = await resolve_draft_provider(
session, none_reason="未配置模型接入，无法由模型起草数据目录。请到「设置 → 模型接入」添加",
mock_reason="当前只有演示用的模型接入，无法用于起草数据目录。请到「设置 → 模型接入」添加真实的模型接入")
except DraftModelUnavailable as e:
raise CatalogModelUnavailable(str(e)) from e
spec = ModelSpec(provider=choice.provider, model=choice.model, max_tokens=MODEL_MAX_TOKENS, temperature=0,
timeout=MODEL_TIMEOUT_S, max_retries=0)
try:
model = build_chat_model(provider, spec)
big = build_chat_model(provider, spec)
small = build_chat_model(provider, spec.model_copy(update={"max_tokens": MODEL_FALLBACK_MAX_TOKENS}))
except ProviderNotConfigured as e:
raise CatalogModelUnavailable(assistant_unavailable(e)) from e
return model, spec.model or provider.default_model or ""
raise CatalogModelUnavailable(draft_unavailable(e, choice.origin)) from e
model_id = spec.model or provider.default_model or ""
return DraftModel(big, small, model_id), model_id
def _confirmed_only(notes: Mapping[str, Any] | None) -> dict[str, Any]:
return _assemble({k: v for k, v in _slots(notes).items() if v.get("status") in ("confirmed", "verified")})
def _describe_for_model(name: str, meta: Mapping[str, Any], *, system: bool, existing: Mapping[str, Any] | None) -> str:
"""一张表交给模型的结构描述：列、类型、主键、外键、唯一约束、注释；不含任何数据值。"""
lines = [(_P["view"] if meta.get("is_view") else _P["table"]).format(name=name)]
comment = (meta.get("comment") or "").strip()
if comment:
lines.append((_P["system"] if system else _P["comment"]).format(text=comment))
pk = set(meta.get("primary_key") or [])
fk_of: dict[str, str] = {}
for fk in meta.get("foreign_keys") or []:
if isinstance(fk, dict) and len(fk.get("columns") or []) == 1:
to_cols = fk.get("to_columns") or []
fk_of[fk["columns"][0]] = f"{fk.get('to_table')}.{to_cols[0]}" if to_cols else str(fk.get("to_table"))
lines.append(_P["columns"])
for col in meta.get("columns") or []:
marks = []
if col["name"] in pk:
marks.append(_P["pk"])
if not col.get("nullable", True):
marks.append(_P["not_null"])
if col["name"] in fk_of:
marks.append(_P["fk"].format(target=fk_of[col["name"]]))
if text := (col.get("comment") or "").strip():
marks.append((_P["col_system"] if system else _P["col_comment"]).format(text=text))
line = _P["column"].format(name=col["name"], type=col.get("type") or "")
lines.append(line + ("，" + "，".join(marks) if marks else ""))
for group in meta.get("unique") or []:
if isinstance(group, list) and group:
lines.append(_P["unique"].format(cols="、".join(group)))
confirmed = _notes_lines(_confirmed_only(existing), meta=meta, system_notes=False)
if confirmed:
lines.append(_P["confirmed"])
lines += confirmed
return "\n".join(lines)
def _batches(names: list[str], tables: Mapping[str, Any]) -> list[list[str]]:
"""按表分批：每批不超过 MODEL_BATCH_TABLES 张、合计不超过 MODEL_BATCH_COLUMNS 列（一张表超了就自成一批）。"""
out: list[list[str]] = []
batch: list[str] = []
width = 0
for name in names:
n = len((tables.get(name) or {}).get("columns") or [])
if batch and (len(batch) >= MODEL_BATCH_TABLES or width + n > MODEL_BATCH_COLUMNS):
out.append(batch)
batch, width = [], 0
batch.append(name)
width += n
if batch:
out.append(batch)
return out
#: 模型给了这张表、但一项可用内容都没有（只有表名的空壳，或者给的值全不合规）：这张表记模型失败，不算起草成功。
#: 不然界面只看到「新增 0 项」，会说成「没有变化」
MODEL_EMPTY_REASON = "模型没有给出这张表的可用内容"
@@ -1621,39 +1701,6 @@ def _model_tables(raw: Any) -> list[dict[str, Any]] | None:
return [t for t in raw if isinstance(t, dict)]
async def _ask_model(model: Any, messages: list[Any],
usable: Callable[[list[dict[str, Any]]], bool] | None = None) -> list[dict[str, Any]]:
"""问一批。先走结构化输出；模型不支持、调用报错、给回来的不是约定的结构，或者 usable 说它不够用（实测见过
结构化输出只给 {"tables":[{"table":"x"}]} 这样的空壳），再退回纯文本、从正文里取 JSON。
结构化输出报错也退回（同 judge._ask）：有的网关不支持工具调用，一调就报错，纯文本却能用。代价是真的
网络故障会多试一次，两次加起来仍受 MODEL_TIMEOUT_S 约束。
两条路都拿到了东西时，结构化的在前、纯文本的在后一起交回（同一张表各有一份，调用方挑能用的那份）；纯文本那次
失败了就只交回结构化的那份（调用方据此给空壳的表记失败）。
"""
try:
parsed = _model_tables(await model.with_structured_output(CATALOG_DRAFT_PROMPT_SCHEMA).ainvoke(messages))
except Exception: # noqa: BLE001 - 退回纯文本
parsed = None
if parsed is not None and (usable is None or usable(parsed)):
return parsed
from app.engine.state import message_text
try:
reply = await model.ainvoke(messages)
except Exception: # noqa: BLE001 - 结构化那份还在：交回它，空壳的表由调用方记失败
if parsed is not None:
return parsed
raise
text = _model_tables(message_text(reply))
if text is None:
if parsed is not None:
return parsed
raise ValueError("模型的回复不是约定的格式")
return [*(parsed or []), *text]
def _clean_text(value: Any, limit: int = _TEXT_MAX) -> str | None:
if not isinstance(value, str):
return None
@@ -1664,7 +1711,8 @@ def _clean_text(value: Any, limit: int = _TEXT_MAX) -> str | None:
def _from_model(entry: Mapping[str, Any], meta: Mapping[str, Any], *, system: bool) -> dict[str, Any]:
"""模型给的一张表 → 目录项（llm / proposed）。不合规的值整项丢掉，不硬塞：列名对不上表结构的、
表类型和度量类型不在取值里的、业务主键里有不存在的列的、看起来是计算公式的（_formula_like，公式进口径卡）。
导入表格的源，说明已经覆盖的字段也丢掉。"""
该是文字的地方写成了数组、对象（列名、表类型、业务日期的列、度量类型……）同样丢掉，不抛异常：实测见过把列名
写成数组的，以前在这里抛 TypeError，整张表记出错。导入表格的源，说明已经覆盖的字段也丢掉。"""
columns = {c["name"]: c for c in meta.get("columns") or []}
notes: dict[str, Any] = {}
table_has_system_note = system and bool((meta.get("comment") or "").strip())
@@ -1673,21 +1721,23 @@ def _from_model(entry: Mapping[str, Any], meta: Mapping[str, Any], *, system: bo
continue
if (text := _clean_text(entry.get(name))) is not None and not _formula_like(("t", name), text):
notes[name] = make_item(text, "llm")
if entry.get("kind") in TABLE_KINDS:
notes["kind"] = make_item(entry["kind"], "llm")
kind = entry.get("kind")
if isinstance(kind, str) and kind in TABLE_KINDS:
notes["kind"] = make_item(kind, "llm")
keys = entry.get("keys")
if _str_list_ok(keys) and all(k in columns for k in keys):
notes["keys"] = make_item(list(dict.fromkeys(keys)), "llm")
date = entry.get("business_date")
if isinstance(date, dict) and date.get("column") in columns:
if isinstance(date, dict) and isinstance(date.get("column"), str) and date["column"] in columns:
value = {"column": date["column"]}
for k in ("rule", "timezone"):
if (text := _clean_text(date.get(k))) is not None:
value[k] = text
notes["business_date"] = make_item(value, "llm")
out_cols: dict[str, dict[str, Any]] = {}
for col in entry.get("columns") or []:
if not isinstance(col, dict) or col.get("name") not in columns:
listed = entry.get("columns")
for col in listed if isinstance(listed, list) else []:
if not isinstance(col, dict) or not isinstance(col.get("name"), str) or col["name"] not in columns:
continue
name = col["name"]
has_system_note = system and bool((columns[name].get("comment") or "").strip())
@@ -1698,8 +1748,9 @@ def _from_model(entry: Mapping[str, Any], meta: Mapping[str, Any], *, system: bo
text = _clean_text(col.get(field_name), limit)
if text is not None and not _formula_like(("c", name, field_name), text):
items[field_name] = make_item(text, "llm")
if col.get("measure") in COLUMN_MEASURES:
items["measure"] = make_item(col["measure"], "llm")
measure = col.get("measure")
if isinstance(measure, str) and measure in COLUMN_MEASURES:
items["measure"] = make_item(measure, "llm")
if items:
out_cols[name] = items
if out_cols:
@@ -1708,65 +1759,30 @@ def _from_model(entry: Mapping[str, Any], meta: Mapping[str, Any], *, system: bo
async def draft_with_model(model: Any, source: Any, tables: list[str], *,
existing: Mapping[str, Mapping[str, Any]] | None = None) -> dict[str, TableDraft | str]:
existing: Mapping[str, Mapping[str, Any]] | None = None,
guidance: str | None = None, note: str | None = None) -> dict[str, TableDraft | str]:
"""用模型起草这些表的中文名、粒度、表类型、业务主键、业务日期和每列的中文名、含义、单位、度量类型。
model 要有 with_structured_output(schema) 或 ainvoke(messages)（测试注入假模型）。按表分批（_batches），
最多 MODEL_CONCURRENCY 批同时发。返回 {表名: TableDraft 或失败原因}：一批失败只影响这一批的表，
其余照常；失败的表不算 covered，已有的模型起草项不会因此被删。
实现在 data/catalog_draft.py（分段、按失败原因处理、捡回被截断的部分）；这里是给老调用方的形状：返回
{表名: TableDraft 或失败原因}。全部写出来的 covered 是 {"llm"}；只写出一部分的仍是 TableDraft，covered 为空、
error 写明哪部分没写出来；一项都没写出来的是失败原因。表结构里没有的表不在结果里。
交给模型的只有表结构（列、类型、主键、外键、唯一约束、注释），不发任何数据值；existing 里已确认、
有确证的项作为「以此为准」的背景一起交过去。
model 要有 with_structured_output(schema) 或 ainvoke(messages)（测试注入假模型）。交给模型的只有表结构和使用者
补充的背景（数据源说明、guidance 起草要求、note 本次说明），不发任何数据值；existing 里已确认、有确证的项作为
「以此为准」的背景一起交过去。
"""
from langchain_core.messages import HumanMessage, SystemMessage
from app.data import catalog_draft
all_tables = (getattr(source, "schema_cache", None) or {}).get("tables") or {}
system = system_notes_source(source)
names = [t for t in tables if isinstance(all_tables.get(t), dict)]
results: dict[str, TableDraft | str] = {}
gate = asyncio.Semaphore(MODEL_CONCURRENCY)
async def run(batch: list[str]) -> None:
described = "\n\n".join(_describe_for_model(n, all_tables[n], system=system,
existing=(existing or {}).get(n)) for n in batch)
messages = [SystemMessage(content=CATALOG_DRAFT_SYSTEM),
HumanMessage(content=CATALOG_DRAFT_PROMPT.format(n=len(batch), tables=described))]
def drafted(entries: list[dict[str, Any]]) -> dict[str, dict[str, Any]]:
"""这批里每张表能用的那份：同一张表给了几份（结构化、纯文本各一份）时取第一份有可用内容的。"""
out: dict[str, dict[str, Any]] = {}
for entry in entries:
name = str(entry.get("table"))
if name in batch and name not in out:
notes = _from_model(entry, all_tables[name], system=system)
if notes:
out[name] = notes
return out
try:
async with gate:
got = await asyncio.wait_for(
_ask_model(model, messages, usable=lambda entries: len(drafted(entries)) == len(batch)),
timeout=MODEL_TIMEOUT_S)
except asyncio.TimeoutError:
for n in batch:
results[n] = f"模型起草超时：{MODEL_TIMEOUT_S} 秒内没有收到回复"
return
except Exception as e: # noqa: BLE001 - 一批失败不影响其他批和其他来源
from app.core.errors import describe_exception, first_line
for n in batch:
results[n] = f"模型起草失败：{first_line(e) or describe_exception(e)}"
return
named = {str(t.get("table")) for t in got}
usable = drafted(got)
for n in batch:
if n in usable:
results[n] = TableDraft(notes=usable[n], covered=frozenset({"llm"}))
else:
results[n] = MODEL_EMPTY_REASON if n in named else "模型的回复里没有这张表"
await asyncio.gather(*(run(b) for b in _batches(names, all_tables)))
return {n: results[n] for n in names if n in results}
outcomes = await catalog_draft.run_model_draft(
model, source, tables, existing=existing, background=catalog_draft.Background(guidance=guidance, note=note))
out: dict[str, TableDraft | str] = {}
for name, outcome in outcomes.items():
if outcome.status == "failed":
out[name] = outcome.error or MODEL_EMPTY_REASON
else:
out[name] = TableDraft(notes=outcome.notes, covered=frozenset({"llm"}) if outcome.status == "ok"
else frozenset(), error=outcome.error)
return out
# ==========================================================================
@@ -1776,7 +1792,8 @@ async def draft_with_model(model: Any, source: Any, tables: list[str], *,
@dataclass
class TableDraftResult:
"""一张表这次起草的结果。error：这张表没起草（表结构里没有、写入一直冲突）；model_error：只是模型那部分失败。"""
"""一张表这次起草的结果。error：这张表没起草（表结构里没有、写入一直冲突）；model_error：只是模型那部分失败，
或者只写出了一部分（哪部分没写出来、为什么）。"""
table: str
added: int = 0
@@ -1785,9 +1802,17 @@ class TableDraftResult:
version: int = 0
error: str | None = None
model_error: str | None = None
#: 模型给了这张表几项可用内容（并入目录之前，按项数）。没用模型、模型这部分失败时为 None。
#: 模型给了这张表几项可用内容（并入目录之前，按项数）。没用模型、模型这部分一项都没写出来时为 None。
#: 大于 0 而新增、更新、删除都是 0，才是「模型给了内容、只是和现有目录一致」
model_items: int | None = None
#: ok 写入了（用模型时各段都写出来了）、partial 模型只写出一部分、failed 没写入或模型一段都没写出来、
#: skipped 表结构里没有这张表
status: str = "ok"
#: 模型分几段起草（不用模型时为 None）
parts_total: int | None = None
#: 模型拿不准、留空了的列（catalog_draft.ModelOutcome.blank，按表结构的顺序），只留并入目录之后仍然没有中文名和
#: 含义的；表的状态不因此变
blank: list[str] = field(default_factory=list)
@dataclass
@@ -1802,35 +1827,113 @@ class DraftReport:
model_error: str | None = None
async def draft_catalog(session: AsyncSession, source: Any, *, tables: list[str] | None = None,
use_model: bool = False, model: Any = None, actor: str | None = None) -> DraftReport:
"""起草这些表的目录并写入，同步完成。
#: 表结构里没有要起草的表（被删了、改名了，或者名字写错了）
def missing_table_error(name: str) -> str:
return f"表结构里没有 {name}，请先重新探查结构"
#: 写入一直撞上别人刚改过（乐观锁重读重并三次仍冲突）
WRITE_CONFLICT_ERROR = "这张表的数据目录正被其他人修改，请稍后再起草"
async def write_draft(session: AsyncSession, source: Any, table: str, modeled: Any = None, *, at: str,
actor: str | None) -> TableDraftResult:
"""一张表的起草并入目录并写入：数据库注释、外键约束、命名推断照做，modeled（catalog_draft.ModelOutcome，
没用模型时为 None）有可用内容的再并上。同步起草和后台起草任务（每张表一结束就调）共用这一处。
- source：手工源的 DataSource，或上传源绑定当前快照的视图（table_versions.resolve_source）；按 source.id 存。
- tables 为空时按使用次数（table_usage）取前 DRAFT_DEFAULT_TABLES 张，次数相同按表结构里的顺序。
- 每张表：数据库注释、外键约束、命名推断照做；use_model 时再加模型起草。model 可注入（测试用假模型），
不给就按设置解析（resolve_draft_model）；模型用不了、某一批失败，都不影响其余来源照常写入。
- 并入已有目录按 merge_notes 的规则：人工确认、驳回过的项不动。内容没变不写、不升版本。
- 模型这次没给的项（留空、没通过校验）不删以前起草的；列已经不在表结构里的，模型各段都写出来时（covered 含 llm）
照旧删掉。模型只写出一部分的不算完整起草过（covered 不含 llm）。
- 模型留空的列（blank）按并入之后的目录再筛一遍：已经有中文名或含义的（以前起草的、库注释、起草进行中有人填的）
不算留空。
- 写入撞上别人刚改过（乐观锁）就重读重并，最多三次。
"""
cache = getattr(source, "schema_cache", None) or {}
all_tables = cache.get("tables") or {}
report = DraftReport()
if tables:
wanted = list(dict.fromkeys(str(t) for t in tables))
result = TableDraftResult(table=table)
structure = draft_structure(source, table)
meta = ((getattr(source, "schema_cache", None) or {}).get("tables") or {}).get(table) or {}
columns = [str(c["name"]) for c in meta.get("columns") or [] if isinstance(c, dict) and c.get("name") is not None]
model_draft: TableDraft | None = None
if modeled is not None:
result.parts_total = modeled.parts_total
result.model_error = modeled.error
if modeled.items:
result.model_items = modeled.items
result.blank = list(getattr(modeled, "blank", None) or [])
model_draft = TableDraft(notes=modeled.notes, covered=frozenset({"llm"}) if modeled.status == "ok"
else frozenset())
for _attempt in range(3):
entry = await read_entry(session, source.id, table)
version = entry.version if entry else 0
merged, stats = merge_notes(entry.notes if entry else {}, structure.notes, covered=structure.covered, at=at,
columns=columns)
if model_draft is not None:
merged, more = merge_notes(merged, model_draft.notes, covered=model_draft.covered, at=at, columns=columns)
stats += more
result.added, result.updated, result.removed = stats.added, stats.updated, stats.removed
result.version = version
if entry is None and not merged:
break # 什么也没起草出来，不建空目录
try:
written = await write_entry(session, source.id, table, merged, if_version=version, actor=actor)
except CatalogConflict:
continue
result.version = written.version
break
else:
usage = await table_usage(session, source)
order = {name: i for i, name in enumerate(all_tables)}
wanted = sorted(all_tables, key=lambda n: (-usage.get(n, 0), order[n]))[:DRAFT_DEFAULT_TABLES]
result.error = WRITE_CONFLICT_ERROR
result.added = result.updated = result.removed = 0
result.blank = []
if result.blank:
named = visible_notes(merged).get("columns") or {}
result.blank = [c for c in result.blank if not any(f in (named.get(c) or {}) for f in ("label", "meaning"))]
if result.error or (modeled is not None and modeled.status == "failed"):
result.status = "failed"
elif modeled is not None and modeled.status == "partial":
result.status = "partial"
return result
def pick_tables(cache: Mapping[str, Any], wanted: Iterable[str]) -> tuple[list[str], list[str]]:
"""要起草的表 → (表结构里的键，去重、按给的顺序；表结构里没有的原样名字)。"""
picked: list[str] = []
for name in wanted:
unknown: list[str] = []
for name in dict.fromkeys(str(t) for t in wanted):
key = resolve_table_name(cache, name)
if key is None:
report.tables.append(TableDraftResult(table=name, error=f"表结构里没有 {name}，请先重新探查结构"))
unknown.append(name)
elif key not in picked:
picked.append(key)
return picked, unknown
llm: dict[str, TableDraft | str] = {}
async def default_tables(session: AsyncSession, source: Any) -> list[str]:
"""不指定表时起草哪些：按使用次数（table_usage）取前 DRAFT_DEFAULT_TABLES 张，次数相同按表结构里的顺序。"""
all_tables = (getattr(source, "schema_cache", None) or {}).get("tables") or {}
usage = await table_usage(session, source)
order = {name: i for i, name in enumerate(all_tables)}
return sorted(all_tables, key=lambda n: (-usage.get(n, 0), order[n]))[:DRAFT_DEFAULT_TABLES]
async def draft_catalog(session: AsyncSession, source: Any, *, tables: list[str] | None = None,
use_model: bool = False, model: Any = None, actor: str | None = None,
guidance: str | None = None, note: str | None = None) -> DraftReport:
"""起草这些表的目录并写入，同步完成（POST /catalog/draft；界面改用后台起草任务，见 data/catalog_jobs.py）。
- source：手工源的 DataSource，或上传源绑定当前快照的视图（table_versions.resolve_source）；按 source.id 存。
- tables 为空时按使用次数取前 DRAFT_DEFAULT_TABLES 张（default_tables）。
- 每张表：数据库注释、外键约束、命名推断照做；use_model 时再加模型起草（data/catalog_draft.py：分段、按失败
原因处理）。model 可注入（测试用假模型），不给就按设置解析（resolve_draft_model）；模型用不了、某一段失败，
都不影响其余来源照常写入。guidance 是数据源级的起草要求，note 是本次说明，和数据源说明一起交给模型。
- 并入、写入见 write_draft。
"""
cache = getattr(source, "schema_cache", None) or {}
report = DraftReport()
wanted = list(tables) if tables else await default_tables(session, source)
picked, unknown = pick_tables(cache, wanted)
for name in unknown:
report.tables.append(TableDraftResult(table=name, error=missing_table_error(name), status="skipped"))
outcomes: dict[str, Any] = {}
if use_model and picked:
try:
if model is None:
@@ -1840,41 +1943,17 @@ async def draft_catalog(session: AsyncSession, source: Any, *, tables: list[str]
except CatalogModelUnavailable as e:
report.model_error = str(e)
else:
from app.data import catalog_draft
report.model_used = True
existing = await read_catalog(session, source.id)
llm = await draft_with_model(model, source, picked, existing={t: e.notes for t, e in existing.items()})
outcomes = await catalog_draft.run_model_draft(
model, source, picked, existing={t: e.notes for t, e in existing.items()},
background=catalog_draft.Background(guidance=guidance, note=note))
at = now_iso()
for table in picked:
result = TableDraftResult(table=table)
structure = draft_structure(source, table)
modeled = llm.get(table)
if isinstance(modeled, str):
result.model_error = modeled
elif isinstance(modeled, TableDraft):
result.model_items = len(_slots(modeled.notes))
for _attempt in range(3):
entry = await read_entry(session, source.id, table)
version = entry.version if entry else 0
merged, stats = merge_notes(entry.notes if entry else {}, structure.notes, covered=structure.covered,
at=at)
if isinstance(modeled, TableDraft):
merged, more = merge_notes(merged, modeled.notes, covered=modeled.covered, at=at)
stats += more
result.added, result.updated, result.removed = stats.added, stats.updated, stats.removed
result.version = version
if entry is None and not merged:
break # 什么也没起草出来，不建空目录
try:
written = await write_entry(session, source.id, table, merged, if_version=version, actor=actor)
except CatalogConflict:
continue
result.version = written.version
break
else:
result.error = "这张表的数据目录正被其他人修改，请稍后再起草"
result.added = result.updated = result.removed = 0
report.tables.append(result)
report.tables.append(await write_draft(session, source, table, outcomes.get(table), at=at, actor=actor))
return report
@@ -2239,15 +2318,22 @@ def frozen_catalog(entries: Mapping[str, CatalogEntry], tables: Iterable[str]) -
__all__ = [
"CARDINALITIES", "CARDINALITY_LABEL", "CATALOG_DRAFT_PROMPT", "CATALOG_DRAFT_PROMPT_SCHEMA",
"CATALOG_DRAFT_SYSTEM", "COLUMN_FIELDS", "COLUMN_FIELD_LABEL", "COLUMN_MEASURES", "CatalogConflict",
"CARDINALITIES", "CARDINALITY_LABEL", "CATALOG_DRAFT_BACKGROUND_PROMPT", "CATALOG_DRAFT_PROMPT",
"CATALOG_DRAFT_PROMPT_LABELS", "CATALOG_DRAFT_PROMPT_SCHEMA", "CATALOG_DRAFT_SYSTEM", "CATALOG_DRAFT_TEXT_PROMPT",
"COLUMN_FIELDS", "COLUMN_FIELD_LABEL", "COLUMN_MEASURES", "CatalogConflict",
"CatalogEntry", "CatalogInvalid", "CatalogModelUnavailable", "CatalogPathError", "DRAFT_DEFAULT_TABLES",
"DraftReport", "INFERRED_MARK", "ITEM_SOURCES", "ITEM_STATUSES", "JoinEdge", "KIND_LABEL", "MEASURE_LABEL",
"MODEL_EMPTY_REASON", "MergeStats", "NEW_RELATION_PATH", "PATCH_MAX_CHANGES", "PROFILE_NOTE_PREFIX",
"PROFILE_VERIFY_COVERAGE",
"DraftModel", "DraftReport", "INFERRED_MARK", "ITEM_SOURCES", "ITEM_STATUSES", "JoinEdge", "KIND_LABEL",
"MEASURE_LABEL", "MODEL_BATCH_COLUMNS", "MODEL_BATCH_TABLES", "MODEL_CONCURRENCY", "MODEL_EMPTY_REASON",
"MODEL_FALLBACK_MAX_TOKENS", "MODEL_MAX_TOKENS", "MODEL_MIN_PART_COLUMNS", "MODEL_NOTICE_OFF", "MODEL_NOTICE_ON",
"MODEL_NOTICE_WINDOW", "MODEL_PART_MAX_CALLS", "MODEL_RETRIES", "MODEL_RETRY_DELAYS_S", "MODEL_SHRINK_AFTER",
"MODEL_SPLIT_DEPTH", "MODEL_SPLIT_MIN_COLUMNS", "MODEL_TIMEOUT_S", "MergeStats", "NEW_RELATION_PATH",
"PATCH_MAX_CHANGES",
"PROFILE_NOTE_PREFIX", "PROFILE_VERIFY_COVERAGE", "WRITE_CONFLICT_ERROR", "default_tables",
"missing_table_error", "pick_tables", "write_draft",
"PatchChange", "PatchPlan", "QUERY_SNAPSHOT_KIND", "REVIEW_ACTIONS", "TABLE_FIELDS", "TABLE_FIELD_LABEL",
"TABLE_KINDS", "TableDraft", "TableDraftResult", "UI_KIND_LABEL", "UI_MEASURE_LABEL", "apply_human_edit",
"apply_patch", "codes_complete", "describe_slot", "draft_catalog", "draft_structure", "draft_with_model",
"apply_patch", "codes_complete", "comment_role", "describe_slot", "draft_catalog", "draft_structure",
"draft_with_model",
"fk_relations", "frozen_catalog", "infer_name_relations", "initial_status", "is_date_column", "is_profile_note",
"join_paths",
"make_item", "merge_notes", "name_words", "now_iso", "parse_path", "plan_patch", "profile_relation_holds",
backend/app/data/catalog_draft.py
+1540
View File
File diff suppressed because it is too large Load Diff
backend/app/data/catalog_jobs.py
+667
View File
@@ -0,0 +1,667 @@
"""数据目录的后台起草任务，以及数据源级的起草要求。
以前起草是一次同步长请求（POST /catalog/draft，界面每 12 张表一个请求）：一个请求出错整个对话框就停，关掉对话框
之后没有任何记录哪些表失败、为什么。现在：
- **建任务**（create_job）：同一个数据源同时只有一个排队或进行中的任务（DraftBusy；部分唯一索引兜底）。不指定表时
按使用次数取前 20 张；表结构里没有的表照样建进任务、状态 skipped。起草要求（数据源级）和本次说明随任务记下快照。
- **在后台起草**（manager.start）：进程内的 asyncio 任务，引擎是 data/catalog_draft.py（分段、按失败原因处理）。
每张表一结束就并入目录（catalog.write_draft，规则和同步起草一样：人工确认、驳回过的不动，模型这次没给的项不删
以前起草的，乐观锁冲突重读重并）；逐表的状态、到第几段、计数、模型留空的列，任务的用量（含超时、没写出来的调用
次数）、当前的列数上限、提醒随时落库，接口读的是库里的状态。
- **停止**（cancel）：排队的直接 cancelled；进行中的停掉正在进行的模型调用，没开始和没写完的表记 cancelled，已经写进
目录的不回滚，任务最后是 cancelled。
- **重试**（retry_tables + create_job）：新任务，只含原任务里 failed、partial、cancelled 的表；用不用模型、本次说明沿用，
起草要求取数据源当前的。
- **服务重启**：启动时（startup）把排队、进行中的任务标成 interrupted，没结束的表记 cancelled、写明「服务重启，起草
中断」。不自动续跑：复制出来的数据目录（检查、验收）启动时不能自己去调模型；界面让人点「重试」续上。关停时
（manager.shutdown）停掉进行中的任务，同样标成 interrupted。
任务的库写入一律经 _JobRun._write：一条一条来，shield 起来——停止任务时不会打断写到一半的库操作，收尾前等它们写完。
**起草要求**存在数据源 options 的 catalog_draft_guidance 里（上传表格的源存在它自己的 DataSource 行上），一段文字，
去首尾空白、最多 GUIDANCE_MAX 字，空串等于没有。只经 PUT /catalog/guidance 改：数据源接口不显示它、整份回写 options
时留着（api/datasources.py），拼连接串时摘掉（data/engine.py）。
"""
from __future__ import annotations
import asyncio
import logging
from collections.abc import Awaitable, Callable
from typing import Any
from sqlalchemy import func, select, update
from sqlalchemy.exc import IntegrityError
from sqlalchemy.ext.asyncio import AsyncSession
from app.data import catalog, catalog_draft
from app.data.engine import CATALOG_DRAFT_GUIDANCE_OPTION
from app.db.base import SessionLocal, new_id, utcnow
from app.db.models import CatalogDraftJob, CatalogDraftTask, DataSource
logger = logging.getLogger(__name__)
#: 任务还没结束的状态（同一个源只能有一个）、结束了的状态
ACTIVE = ("queued", "running")
FINISHED = ("succeeded", "partial", "failed", "cancelled", "interrupted")
#: 逐表的状态，以及其中算「已结束」的、重试时要带上的
TASK_STATUSES = ("pending", "running", "ok", "partial", "failed", "skipped", "cancelled")
TASK_DONE = ("ok", "partial", "failed", "skipped", "cancelled")
RETRYABLE = ("failed", "partial", "cancelled")
_OPEN = ("pending", "running")
#: 一个任务最多几张表；起草要求、本次说明最多几个字
MAX_TABLES = 500
GUIDANCE_MAX = 2000
NOTE_MAX = 1000
#: 目录清单上「这张表最近一次起草」只看最近这么多个任务
LAST_DRAFT_JOBS = 100
#: 逐表最多记几个模型留空的列名
BLANK_SAMPLE = 5
#: 停止进行中的任务时最多等它收尾多久（秒）；关停服务时另给（shutdown 的 grace）
STOP_GRACE_S = 10.0
INTERRUPTED_ERROR = "服务重启，起草中断"
UNFINISHED_ERROR = "起草结束时这张表还没有写完，请重试这张表"
WRITE_ERROR = "写入数据目录时出错：{line}"
JOB_ERROR = "起草任务出错：{line}"
SOURCE_GONE = "数据源不存在，可能已被删除"
class DraftBusy(Exception):
"""这个数据源已经有排队或进行中的起草任务。message 给人看。"""
class DraftFinished(Exception):
"""要停止的任务已经结束了。"""
class NothingToRetry(Exception):
"""原任务里没有需要重试的表。"""
class TextTooLong(ValueError):
"""起草要求、本次说明超长。message 给人看。"""
class _JobFailed(Exception):
"""任务整体出错（数据源没了、快照没了、模型用不了）：message 就是任务的 error。"""
# ==========================================================================
# 起草要求、本次说明
# ==========================================================================
def clean_text(value: Any, limit: int, label: str) -> str | None:
"""去首尾空白；空串（和不是字符串的）等于没有；超过 limit 个字抛 TextTooLong。"""
text = value.strip() if isinstance(value, str) else ""
if len(text) > limit:
raise TextTooLong(f"{label}最多 {limit} 字，当前 {len(text)} 字，请删减后再保存")
return text or None
def guidance_of(row: Any) -> str | None:
"""数据源级的起草要求（没有是 None）。"""
raw = (getattr(row, "options", None) or {}).get(CATALOG_DRAFT_GUIDANCE_OPTION)
return raw.strip() or None if isinstance(raw, str) else None
async def save_guidance(session: AsyncSession, row: DataSource, value: Any) -> str | None:
"""存起草要求，返回存下的（清除了是 None）。超长抛 TextTooLong，什么都不改。"""
text = clean_text(value, GUIDANCE_MAX, "起草要求")
options = {k: v for k, v in (row.options or {}).items() if k != CATALOG_DRAFT_GUIDANCE_OPTION}
if text:
options[CATALOG_DRAFT_GUIDANCE_OPTION] = text
row.options = options
await session.commit()
return text
# ==========================================================================
# 读任务
# ==========================================================================
def _iso(value: Any) -> str | None:
return value.isoformat() if value is not None else None
async def active_job(session: AsyncSession, source_id: str) -> CatalogDraftJob | None:
"""排队或进行中的那个（同一个源最多一个）。"""
return (await session.execute(
select(CatalogDraftJob).where(CatalogDraftJob.source_id == source_id, CatalogDraftJob.status.in_(ACTIVE))
.order_by(CatalogDraftJob.created_at.desc()).limit(1)
)).scalar_one_or_none()
async def latest_job(session: AsyncSession, source_id: str) -> CatalogDraftJob | None:
"""最近建的一个（任何状态）。"""
return (await session.execute(
select(CatalogDraftJob).where(CatalogDraftJob.source_id == source_id)
.order_by(CatalogDraftJob.created_at.desc()).limit(1)
)).scalar_one_or_none()
async def load(session: AsyncSession, source_id: str, job_id: str) -> CatalogDraftJob | None:
"""这个源的这个任务；不存在、不属于这个源都是 None。"""
job = await session.get(CatalogDraftJob, job_id)
return job if job is not None and job.source_id == source_id else None
async def tasks_of(session: AsyncSession, job_id: str) -> list[CatalogDraftTask]:
return list((await session.execute(
select(CatalogDraftTask).where(CatalogDraftTask.job_id == job_id).order_by(CatalogDraftTask.seq)
)).scalars())
def task_dict(task: CatalogDraftTask) -> dict[str, Any]:
return {
"table_name": task.table_name,
"status": task.status,
"columns": task.column_count,
"parts_total": task.parts_total,
"parts_done": task.parts_done,
"added": task.added,
"updated": task.updated,
"removed": task.removed,
"version": task.version,
"model_items": task.model_items,
# 模型拿不准、留空了的列（中文名和含义都没给）：几列、最多 5 个列名（按表结构的顺序）。表的状态不因此变
"blank_columns": task.blank_columns or 0,
"blank_sample": list(task.blank_sample or []),
"error": task.error,
"model_error": task.model_error,
"started_at": _iso(task.started_at),
"finished_at": _iso(task.finished_at),
}
def _counts(statuses: list[str]) -> dict[str, int]:
counts = dict.fromkeys(TASK_STATUSES, 0)
for status in statuses:
counts[status] = counts.get(status, 0) + 1
return counts
def job_dict(job: CatalogDraftJob, tasks: list[CatalogDraftTask]) -> dict[str, Any]:
"""接口返回的任务（CatalogDraftJob）：逐表的状态按起草顺序；计数、合计都按逐表的算。"""
counts = _counts([t.status for t in tasks])
return {
"id": job.id,
"source_id": job.source_id,
"status": job.status,
"use_model": bool(job.use_model),
"model": job.model,
"guidance": job.guidance,
"note": job.note,
"created_at": _iso(job.created_at),
"started_at": _iso(job.started_at),
"finished_at": _iso(job.finished_at),
"created_by": job.created_by,
"retry_of": job.retry_of,
"total": len(tasks),
"done": sum(counts.get(s, 0) for s in TASK_DONE),
"counts": counts,
"added": sum(t.added or 0 for t in tasks),
"updated": sum(t.updated or 0 for t in tasks),
"removed": sum(t.removed or 0 for t in tasks),
"blank_columns": sum(t.blank_columns or 0 for t in tasks),
# 超时的调用拿不到用量，token 按 0 算（网关那边可能已经生成了）；failed_calls 含额度用完、只给空壳、超时、出错
"usage": {"calls": job.calls or 0, "input_tokens": job.input_tokens or 0,
"output_tokens": job.output_tokens or 0, "timeouts": job.timeouts or 0,
"failed_calls": job.failed_calls or 0},
# 新开始的段每段最多几列（不用模型、还没开始时为 null）；提醒：最近几次调用多数没写出来时的一句话，没有为 null
"part_columns": job.part_columns,
"notice": job.notice,
"error": job.error,
"tasks": [task_dict(t) for t in tasks],
}
async def job_out(session: AsyncSession, job: CatalogDraftJob | None) -> dict[str, Any] | None:
return None if job is None else job_dict(job, await tasks_of(session, job.id))
async def job_brief(session: AsyncSession, job: CatalogDraftJob | None) -> dict[str, Any] | None:
"""目录清单顶上的进度：{id, status, total, done}。"""
if job is None:
return None
rows = (await session.execute(
select(CatalogDraftTask.status, func.count()).where(CatalogDraftTask.job_id == job.id)
.group_by(CatalogDraftTask.status)
)).all()
counts = {status: n for status, n in rows}
return {"id": job.id, "status": job.status, "total": sum(counts.values()),
"done": sum(counts.get(s, 0) for s in TASK_DONE)}
async def last_drafts(session: AsyncSession, source_id: str) -> dict[str, dict[str, Any]]:
"""每张表最近一次起草任务里的那一项：{表名: {status, job_id, finished_at, error, model_error, blank_columns}}。
任务还在进行，就是进行中的状态。只看最近 LAST_DRAFT_JOBS 个任务；每张表取哪一项在库里挑好（清单页起草时会反复取）。"""
recent = (select(CatalogDraftJob.id).where(CatalogDraftJob.source_id == source_id)
.order_by(CatalogDraftJob.created_at.desc()).limit(LAST_DRAFT_JOBS)).scalar_subquery()
rank = func.row_number().over(partition_by=CatalogDraftTask.table_name,
order_by=(CatalogDraftJob.created_at.desc(), CatalogDraftTask.seq)).label("rank")
inner = (select(CatalogDraftTask.table_name, CatalogDraftTask.status, CatalogDraftTask.job_id,
CatalogDraftTask.finished_at, CatalogDraftTask.error, CatalogDraftTask.model_error,
CatalogDraftTask.blank_columns, rank)
.join(CatalogDraftJob, CatalogDraftJob.id == CatalogDraftTask.job_id)
.where(CatalogDraftJob.id.in_(recent))).subquery()
rows = (await session.execute(select(inner).where(inner.c.rank == 1))).mappings()
return {row["table_name"]: {"status": row["status"], "job_id": row["job_id"],
"finished_at": _iso(row["finished_at"]), "error": row["error"],
"model_error": row["model_error"], "blank_columns": row["blank_columns"] or 0}
for row in rows}
# ==========================================================================
# 建任务、重试、停止
# ==========================================================================
def busy_message(row: Any) -> str:
return f"数据源「{getattr(row, 'name', '')}」正在起草数据目录，请等这次起草结束（或先停止它）再开始新的起草"
async def ensure_idle(session: AsyncSession, row: Any) -> None:
"""这个源没有排队或进行中的任务；有就抛 DraftBusy。"""
if await active_job(session, row.id) is not None:
raise DraftBusy(busy_message(row))
async def check_model(session: AsyncSession) -> None:
"""起草模型能不能用：不能用抛 catalog.CatalogModelUnavailable（只建模型对象，不发请求）。"""
await catalog.resolve_draft_model(session)
async def create_job(session: AsyncSession, row: Any, source: Any, *, tables: list[str] | None, use_model: bool,
note: str | None, actor: str | None,
retry_of: str | None = None) -> tuple[CatalogDraftJob, list[CatalogDraftTask]]:
"""建一个排队的任务（还没开始，调用方接着 manager.start）。row 是数据源记录（起草要求取它的），source 是读表结构
用的源（上传源是绑定当前快照的视图）。有排队或进行中的任务抛 DraftBusy。"""
await ensure_idle(session, row)
cache = getattr(source, "schema_cache", None) or {}
all_tables = cache.get("tables") or {}
wanted = list(tables) if tables else await catalog.default_tables(session, source)
job = CatalogDraftJob(id=new_id(), source_id=row.id, status="queued", use_model=bool(use_model),
guidance=guidance_of(row), note=note, created_by=actor, retry_of=retry_of)
tasks: list[CatalogDraftTask] = []
seen: set[str] = set()
now = utcnow()
for name in dict.fromkeys(str(t) for t in wanted):
key = catalog.resolve_table_name(cache, name)
if key is None:
tasks.append(CatalogDraftTask(job_id=job.id, seq=len(tasks), table_name=name, status="skipped",
error=catalog.missing_table_error(name), finished_at=now))
continue
if key in seen:
continue
seen.add(key)
columns = len((all_tables.get(key) or {}).get("columns") or [])
tasks.append(CatalogDraftTask(job_id=job.id, seq=len(tasks), table_name=key, status="pending",
column_count=columns,
parts_total=catalog_draft.parts_for(columns) if use_model else None))
session.add(job)
try:
await session.flush()
session.add_all(tasks)
await session.commit()
except IntegrityError as e:
# 部分唯一索引：另一个请求同时给这个源建了任务
await session.rollback()
raise DraftBusy(busy_message(row)) from e
return job, tasks
async def retry_tables(session: AsyncSession, job: CatalogDraftJob) -> list[str]:
"""重试要带上的表：原任务里 failed、partial、cancelled 的，按原来的顺序。没有抛 NothingToRetry。"""
names = [t.table_name for t in await tasks_of(session, job.id) if t.status in RETRYABLE]
if not names:
raise NothingToRetry("这次起草里没有失败、只写出一部分或停止了的表，不需要重试")
return names
async def cancel(session: AsyncSession, job: CatalogDraftJob) -> None:
"""停止一个任务。排队的直接 cancelled；进行中的交给 manager 停（停掉模型调用、没写完的表记 cancelled）。
已经结束的抛 DraftFinished。"""
if job.status in FINISHED:
raise DraftFinished("这次起草已经结束了，不需要停止")
if job.status == "queued":
now = utcnow()
claimed = await session.execute(
update(CatalogDraftJob).where(CatalogDraftJob.id == job.id, CatalogDraftJob.status == "queued")
.values(status="cancelled", finished_at=now).execution_options(synchronize_session=False))
if claimed.rowcount == 1:
await session.execute(
update(CatalogDraftTask).where(CatalogDraftTask.job_id == job.id, CatalogDraftTask.status.in_(_OPEN))
.values(status="cancelled", finished_at=now).execution_options(synchronize_session=False))
await session.commit()
return
await session.rollback() # 刚被后台认领、开始了：按进行中的停
if await manager.stop(job.id, "user", grace=STOP_GRACE_S):
return
# 库里说在进行，这个进程里却没有在跑它（记录对不上）：直接记停止
await _close(session, job.id, "cancelled", None, "cancelled", None)
async def _close(session: AsyncSession, job_id: str, job_status: str, job_error: str | None, task_status: str,
task_error: str | None) -> None:
"""收尾：还没结束的表记 task_status，任务（还没结束的话）记 job_status。"""
now = utcnow()
await session.execute(
update(CatalogDraftTask).where(CatalogDraftTask.job_id == job_id, CatalogDraftTask.status.in_(_OPEN))
.values(status=task_status, error=task_error, finished_at=now).execution_options(synchronize_session=False))
await session.execute(
update(CatalogDraftJob).where(CatalogDraftJob.id == job_id, CatalogDraftJob.status.in_(ACTIVE))
.values(status=job_status, error=job_error, finished_at=now).execution_options(synchronize_session=False))
await session.commit()
async def startup() -> int:
"""服务启动时：排队、进行中的任务标成 interrupted，没结束的表记 cancelled、写明服务重启。不自动续跑。返回标了几个。"""
async with SessionLocal() as session:
ids = [job_id for job_id in (await session.execute(
select(CatalogDraftJob.id).where(CatalogDraftJob.status.in_(ACTIVE)))).scalars()
if not manager.running(job_id)]
for job_id in ids:
await _close(session, job_id, "interrupted", INTERRUPTED_ERROR, "cancelled", INTERRUPTED_ERROR)
if ids:
logger.warning("服务重启：%d 个数据目录起草任务没有做完，已标为中断，可在数据目录页重试", len(ids))
return len(ids)
# ==========================================================================
# 在后台起草
# ==========================================================================
class _JobRun:
"""一个任务在后台的一次执行。"""
def __init__(self, job_id: str) -> None:
self.job_id = job_id
self.task: asyncio.Task[None] | None = None
#: 为什么停：user 使用者点了停止，shutdown 服务关停（标中断）
self.stop_reason: str | None = None
self._writes: set[asyncio.Task[Any]] = set()
self._lock = asyncio.Lock()
self._started: set[str] = set()
self.source: Any = None
self.actor: str | None = None
self.at = ""
# ---------------------------------------------------------------- 库写入
async def _write(self, fn: Callable[..., Awaitable[Any]], *args: Any) -> Any:
"""一条库写入：同一个任务的写入一条一条来；shield 起来，停止任务时不会打断写到一半的库操作。"""
async def go() -> Any:
async with self._lock:
async with SessionLocal() as session:
return await fn(session, *args)
task = asyncio.ensure_future(go())
self._writes.add(task)
task.add_done_callback(self._writes.discard)
return await asyncio.shield(task)
async def _settle(self) -> None:
"""等手上的库写入都写完（停止、出错收尾之前）。"""
while self._writes:
await asyncio.gather(*list(self._writes), return_exceptions=True)
# ---------------------------------------------------------------- 主流程
async def run(self) -> None:
try:
await self._run()
except asyncio.CancelledError:
current = asyncio.current_task()
if current is not None:
current.uncancel()
if self.stop_reason == "user":
await self._end("cancelled", None, "cancelled", None)
else:
await self._end("interrupted", INTERRUPTED_ERROR, "cancelled", INTERRUPTED_ERROR)
except _JobFailed as e:
await self._end("failed", str(e), "failed", str(e))
except Exception as e: # noqa: BLE001 - 任务整体出错：照实记下，不留在「起草中」
from app.core.errors import describe_exception, first_line
logger.exception("数据目录起草任务 %s 出错", self.job_id)
reason = JOB_ERROR.format(line=first_line(e) or describe_exception(e))
await self._end("failed", reason, "failed", reason)
async def _end(self, job_status: str, job_error: str | None, task_status: str, task_error: str | None) -> None:
"""停止、出错时收尾：等手上的库写入写完，再把没结束的表和任务记下来。收尾本身写不进去只记日志（下次启动时
startup 会把它标成中断）。"""
try:
await self._settle()
await self._write(_close, self.job_id, job_status, job_error, task_status, task_error)
except asyncio.CancelledError:
raise
except Exception: # noqa: BLE001
logger.exception("数据目录起草任务 %s 收尾时写库出错", self.job_id)
async def _run(self) -> None:
if not await self._write(self._claim):
return # 开始之前就被停止了
async with SessionLocal() as session:
job = await session.get(CatalogDraftJob, self.job_id)
row = await session.get(DataSource, job.source_id) if job is not None else None
if job is None or row is None:
raise _JobFailed(SOURCE_GONE)
from app.data import table_versions
try:
source = await table_versions.resolve_source(session, row)
except table_versions.SnapshotMissing as e:
raise _JobFailed(str(e)) from e
pending = [t.table_name for t in await tasks_of(session, job.id) if t.status == "pending"]
use_model, background = bool(job.use_model), catalog_draft.Background(guidance=job.guidance, note=job.note)
self.actor = job.created_by
model: Any = None
existing: dict[str, Any] = {}
if use_model and pending:
try:
model, model_id = await catalog.resolve_draft_model(session)
except catalog.CatalogModelUnavailable as e:
raise _JobFailed(str(e)) from e
existing = {t: e.notes for t, e in (await catalog.read_catalog(session, source.id)).items()}
self.source, self.at = source, catalog.now_iso()
all_tables = (getattr(source, "schema_cache", None) or {}).get("tables") or {}
names = [n for n in pending if isinstance(all_tables.get(n), dict)]
gone = [n for n in pending if n not in names]
if gone:
# 建任务之后表结构变了（重新探查过）：这几张表跳过
await self._write(self._skip, gone)
if use_model and names:
await self._write(self._set_model, model_id)
hooks = catalog_draft.Hooks(call_started=self._call_started, call_done=self._call_done,
part_done=self._part_done, table_done=self._table_done)
await catalog_draft.run_model_draft(model, source, names, existing=existing, background=background,
hooks=hooks)
else:
for name in names:
await self._call_started([name])
await self._table_done(name, None)
await self._settle()
await self._write(self._finish)
# ---------------------------------------------------------------- 进度回调（引擎调）
async def _call_started(self, tables: list[str]) -> None:
fresh = [t for t in dict.fromkeys(tables) if t not in self._started]
if fresh:
self._started.update(fresh)
await self._write(self._mark_running, fresh)
async def _call_done(self, report: catalog_draft.CallReport) -> None:
await self._write(self._add_call, report)
async def _part_done(self, table: str) -> None:
await self._write(self._bump_part, table)
async def _table_done(self, table: str, outcome: catalog_draft.ModelOutcome | None) -> None:
await self._write(self._write_table, table, outcome)
# ---------------------------------------------------------------- 库操作
def _task(self, *where: Any) -> Any:
return update(CatalogDraftTask).where(CatalogDraftTask.job_id == self.job_id, *where) \
.execution_options(synchronize_session=False)
async def _claim(self, session: AsyncSession) -> bool:
result = await session.execute(
update(CatalogDraftJob).where(CatalogDraftJob.id == self.job_id, CatalogDraftJob.status == "queued")
.values(status="running", started_at=utcnow()).execution_options(synchronize_session=False))
await session.commit()
return result.rowcount == 1
async def _set_model(self, session: AsyncSession, model_id: str) -> None:
await session.execute(update(CatalogDraftJob).where(CatalogDraftJob.id == self.job_id)
.values(model=model_id or None, part_columns=max(1, int(catalog.MODEL_BATCH_COLUMNS)))
.execution_options(synchronize_session=False))
await session.commit()
async def _skip(self, session: AsyncSession, names: list[str]) -> None:
now = utcnow()
for name in names:
await session.execute(self._task(CatalogDraftTask.table_name == name, CatalogDraftTask.status == "pending")
.values(status="skipped", error=catalog.missing_table_error(name), finished_at=now))
await session.commit()
async def _mark_running(self, session: AsyncSession, names: list[str]) -> None:
await session.execute(self._task(CatalogDraftTask.table_name.in_(names), CatalogDraftTask.status == "pending")
.values(status="running", started_at=utcnow()))
await session.commit()
async def _add_call(self, session: AsyncSession, report: catalog_draft.CallReport) -> None:
"""一次调用：用量、超时、没写出来的次数累加，当前的列数上限和提醒照这次调用之后的写（按调用的先后写进来）。"""
await session.execute(
update(CatalogDraftJob).where(CatalogDraftJob.id == self.job_id)
.values(calls=CatalogDraftJob.calls + 1,
input_tokens=CatalogDraftJob.input_tokens + int(report.input_tokens),
output_tokens=CatalogDraftJob.output_tokens + int(report.output_tokens),
timeouts=func.coalesce(CatalogDraftJob.timeouts, 0) + int(report.timeout),
failed_calls=func.coalesce(CatalogDraftJob.failed_calls, 0) + int(report.failure is not None),
part_columns=report.part_columns, notice=report.notice)
.execution_options(synchronize_session=False))
await session.commit()
async def _bump_part(self, session: AsyncSession, table: str) -> None:
await session.execute(self._task(CatalogDraftTask.table_name == table)
.values(parts_done=CatalogDraftTask.parts_done + 1))
await session.commit()
async def _write_table(self, session: AsyncSession, table: str,
outcome: catalog_draft.ModelOutcome | None) -> None:
"""一张表结束：并入目录、写入，记下这张表的结果。"""
try:
result = await catalog.write_draft(session, self.source, table, outcome, at=self.at, actor=self.actor)
except Exception as e: # noqa: BLE001 - 这一张写不进去不影响别的表
from app.core.errors import describe_exception, first_line
logger.exception("起草任务 %s 写入表 %s 的数据目录时出错", self.job_id, table)
await session.rollback()
result = catalog.TableDraftResult(table=table, status="failed",
error=WRITE_ERROR.format(line=first_line(e) or describe_exception(e)),
model_error=outcome.error if outcome is not None else None)
values: dict[str, Any] = {
"status": result.status, "added": result.added, "updated": result.updated, "removed": result.removed,
"version": result.version, "model_items": result.model_items, "error": result.error,
"model_error": result.model_error, "finished_at": utcnow(),
"blank_columns": len(result.blank), "blank_sample": result.blank[:BLANK_SAMPLE] or None,
}
if outcome is not None:
values["parts_done"] = outcome.parts_done
await session.execute(self._task(CatalogDraftTask.table_name == table, CatalogDraftTask.status.in_(_OPEN))
.values(**values))
await session.commit()
async def _finish(self, session: AsyncSession) -> None:
"""起草完：万一还有没收尾的表（进度回调写库失败之类）记失败；有 failed、partial、cancelled 的表是 partial，
否则 succeeded。"""
now = utcnow()
await session.execute(self._task(CatalogDraftTask.status.in_(_OPEN))
.values(status="failed", error=UNFINISHED_ERROR, finished_at=now))
statuses = list((await session.execute(
select(CatalogDraftTask.status).where(CatalogDraftTask.job_id == self.job_id))).scalars())
status = "partial" if any(s in RETRYABLE for s in statuses) else "succeeded"
await session.execute(
update(CatalogDraftJob).where(CatalogDraftJob.id == self.job_id, CatalogDraftJob.status == "running")
.values(status=status, finished_at=now).execution_options(synchronize_session=False))
await session.commit()
class DraftJobs:
"""进程内的起草任务（照 app/scheduler 的写法：一个 asyncio 任务一件事，停的时候给宽限）。"""
def __init__(self) -> None:
self._runs: dict[str, _JobRun] = {}
def _alive(self, job_id: str) -> _JobRun | None:
run = self._runs.get(job_id)
if run is None or run.task is None or run.task.done():
return None
try:
if run.task.get_loop() is not asyncio.get_running_loop():
self._runs.pop(job_id, None) # 别的事件循环留下的（测试每个用例一个循环）
return None
except RuntimeError:
return None
return run
def start(self, job_id: str) -> None:
"""开始在后台起草这个（排队中的）任务。"""
run = _JobRun(job_id)
run.task = asyncio.get_running_loop().create_task(run.run(), name=f"catalog-draft-{job_id}")
self._runs[job_id] = run
def forget(_: asyncio.Task[None]) -> None:
if self._runs.get(job_id) is run:
self._runs.pop(job_id, None)
run.task.add_done_callback(forget)
def running(self, job_id: str) -> bool:
return self._alive(job_id) is not None
async def stop(self, job_id: str, reason: str, *, grace: float = STOP_GRACE_S) -> bool:
"""停掉这个进程里在跑的任务，等它收尾（最多 grace 秒）。不在这个进程里跑返回 False。"""
run = self._alive(job_id)
if run is None:
return False
if run.stop_reason is None:
run.stop_reason = reason
run.task.cancel() # type: ignore[union-attr]
await asyncio.wait([run.task], timeout=grace) # type: ignore[list-item]
return True
async def wait(self, job_id: str, timeout: float | None = None) -> None:
"""等这个任务做完（测试、调试用）。不在跑直接返回。"""
run = self._alive(job_id)
if run is not None:
await asyncio.wait([run.task], timeout=timeout) # type: ignore[list-item]
async def shutdown(self, *, grace: float = 3.0) -> None:
"""关停：停掉进行中的任务（标中断），最多等 grace 秒收尾。"""
runs = [run for job_id in list(self._runs) if (run := self._alive(job_id)) is not None]
for run in runs:
if run.stop_reason is None:
run.stop_reason = "shutdown"
run.task.cancel() # type: ignore[union-attr]
if runs:
await asyncio.wait([run.task for run in runs], timeout=grace) # type: ignore[misc]
logger.info("已停止 %d 个数据目录起草任务", len(runs))
manager = DraftJobs()
__all__ = ["ACTIVE", "BLANK_SAMPLE", "DraftBusy", "DraftFinished", "DraftJobs", "FINISHED", "GUIDANCE_MAX",
"INTERRUPTED_ERROR", "MAX_TABLES", "NOTE_MAX", "NothingToRetry", "RETRYABLE", "TASK_DONE", "TASK_STATUSES",
"TextTooLong",
"active_job", "cancel", "check_model", "clean_text", "create_job", "ensure_idle", "guidance_of", "job_brief",
"job_dict", "job_out", "last_drafts", "latest_job", "load", "manager", "retry_tables", "save_guidance",
"startup", "tasks_of"]
backend/app/data/draft_model.py
+114
View File
@@ -0,0 +1,114 @@
"""起草用哪个模型：数据目录的「用模型起草」和按配方导入的 AI 起草共用这一处取值。
顺序（哪一级写了就整组用哪一级，不跨级拼接入和模型）：
1. 设置里的起草模型（catalog_draft 一组：{provider, model}），填了接入或模型就用它；
2. 否则沿用助手的模型设置（copilot 一组，「编排」页助手栏的「助手使用的模型」）；
3. 再没有就是默认接入（providers/factory.resolve_provider 的回退）。
这里只决定「用哪个接入、哪个模型」。额度、思考、温度由两边各自定：数据目录起草保留思考、额度 32768
（data/catalog.resolve_draft_model），配方 AI 起草照旧关思考（data/recipe_ai.resolve_draft_model）。
**为什么不用 get_chat_model。** 它在没有接入时静默退回演示模型，拿演示模型起草出来的东西会被当成真的。
两边都自己解析接入：没有、停用、只有演示接入、缺 key，都在调用模型之前判定为不可用，说清是哪一级的设置出了问题。
数据层不导入接口层：设置键在这里定义，api/settings.py 从这里取（助手那一项和 api/copilot.py 同值，有测试核对）。
"""
from __future__ import annotations
from dataclasses import dataclass
from typing import Any
from sqlalchemy.ext.asyncio import AsyncSession
#: 设置里起草模型那一组的键
CATALOG_DRAFT_SETTING_KEY = "catalog_draft"
#: 助手的模型设置键（与 api/copilot.py 的 COPILOT_SETTING_KEY 同值）
COPILOT_SETTING_KEY = "copilot"
#: 起草模型在设置页上的位置。报错里让人去这里改；和前端设置页放这一组的地方一致
DRAFT_MODEL_PLACE = "「设置 → 偏好设置」的「起草模型」"
#: 取自哪一级：draft 设置里的起草模型、assistant 助手的模型、default 默认接入
ORIGINS = ("draft", "assistant", "default")
@dataclass(frozen=True)
class DraftModelChoice:
"""起草用的接入和模型（都可以为 None：接入为 None 时按模型名找、再退回默认接入；模型为 None 时用接入的默认模型）。"""
provider: str | None
model: str | None
origin: str
def _text(value: Any) -> str | None:
"""设置里的一项：字符串去掉首尾空白，空串和别的类型（存坏了的）都当没填。"""
return value.strip() or None if isinstance(value, str) else None
def setting_pair(stored: Any) -> tuple[str | None, str | None]:
"""库里存的一组 → (接入, 模型)，规范化同 _text。"""
stored = stored if isinstance(stored, dict) else {}
return _text(stored.get("provider")), _text(stored.get("model"))
async def draft_model_choice(session: AsyncSession) -> DraftModelChoice:
"""按上面的顺序取。只读设置，不查接入在不在（resolve_provider 去查）。"""
from app.db.models import Setting
for key, origin in ((CATALOG_DRAFT_SETTING_KEY, "draft"), (COPILOT_SETTING_KEY, "assistant")):
row = await session.get(Setting, key)
provider, model = setting_pair(row.value if row is not None else None)
if provider or model:
return DraftModelChoice(provider, model, origin)
return DraftModelChoice(None, None, "default")
def draft_unavailable(e: BaseException, origin: str) -> str:
"""取到的接入用不了（点名的不存在、已停用，缺 key、没有模型……）时给人看的那句话。
沿用助手的模型（或默认接入）时，和助手取不到模型时同一句（factory.assistant_unavailable）；用的是设置里的起草
模型时，照实说是起草模型那一项、去设置页哪里改。"""
from app.core.errors import not_configured
from app.providers.factory import NAMED_FIX, assistant_unavailable, model_fact
if origin != "draft":
return assistant_unavailable(e)
code = getattr(e, "code", "")
if code in NAMED_FIX:
said = f"起草模型使用的{getattr(e, 'reason', '') or e}。"
if owner := getattr(e, "model_owner", ""):
return (f"{said}{NAMED_FIX[code]}；{model_fact(getattr(e, 'name', ''), owner)}，"
f"也可在{DRAFT_MODEL_PLACE}中把它填进模型一栏")
return f"{said}{NAMED_FIX[code]}，或在{DRAFT_MODEL_PLACE}中重新选择"
return (f"起草模型尚未配置完成：{not_configured(e, with_hint=False)}。请到「设置 → 模型接入」检查，"
f"或在{DRAFT_MODEL_PLACE}中重新选择")
class DraftModelUnavailable(RuntimeError):
"""没有可用于起草的模型。message 给人看；在调用模型之前判定。"""
async def resolve_draft_provider(session: AsyncSession, *, none_reason: str, mock_reason: str) -> tuple[Any, DraftModelChoice]:
"""(接入, 取值)。用不了时抛 DraftModelUnavailable：没有任何启用的接入说 none_reason，只有演示接入说 mock_reason
（两边各有各的说法），点名的接入不存在、已停用按取自哪一级说（draft_unavailable）。
只解析接入，不建模型对象：建的时候缺 key 之类的问题由调用方经 draft_unavailable 说。"""
from app.providers.factory import ProviderNotConfigured, resolve_provider
choice = await draft_model_choice(session)
try:
# 存的接入名不存在、已停用，只存了模型名、提供它的接入已停用：resolve_provider 都会报出来
provider = await resolve_provider(session, choice.provider, choice.model)
except ProviderNotConfigured as e:
raise DraftModelUnavailable(draft_unavailable(e, choice.origin)) from e
if provider is None:
raise DraftModelUnavailable(none_reason)
if provider.kind == "mock":
raise DraftModelUnavailable(mock_reason)
return provider, choice
__all__ = ["CATALOG_DRAFT_SETTING_KEY", "COPILOT_SETTING_KEY", "DRAFT_MODEL_PLACE", "DraftModelChoice",
"DraftModelUnavailable", "draft_model_choice", "draft_unavailable", "resolve_draft_provider",
"setting_pair"]
backend/app/data/engine.py
+7 -1
View File
@@ -62,12 +62,18 @@ MASK_COLUMNS_OPTION = "mask_columns"
#: data/catalog_profile.py；放在这里只为让拼连接串时摘掉它
CATALOG_PROFILE_OPTION = "catalog_profile"
#: 数据源 options 里的数据目录起草要求：使用者补充的背景和要求，起草时交给模型（一段文字，最多 2000 字）。
#: 只经 PUT /api/datasources/{id}/catalog/guidance 维护，见 data/catalog_jobs.py；放在这里只为让拼连接串时摘掉它
CATALOG_DRAFT_GUIDANCE_OPTION = "catalog_draft_guidance"
# options 里这些 key 是 AgentLab 自己的配置，不是驱动参数，拼 URL 时要摘掉。
# schema：探查哪个 schema（企业库里只读账号名下常常什么都没有，数据在别处）
# query_timeout_s：查询时限，由数据层按语句下发给数据库，见 _server_deadline
# mask_columns：证据面板遮罩的列，驱动不认识它，拼进连接串会被当成未知参数拒掉
# catalog_profile：数据剖析设置，同上；它还是个对象，拼进去连 URL 都不成形
_NON_DRIVER_OPTIONS = frozenset({"schema", QUERY_TIMEOUT_OPTION, MASK_COLUMNS_OPTION, CATALOG_PROFILE_OPTION})
# catalog_draft_guidance：数据目录的起草要求，一段中文，拼进去连接直接失败
_NON_DRIVER_OPTIONS = frozenset({"schema", QUERY_TIMEOUT_OPTION, MASK_COLUMNS_OPTION, CATALOG_PROFILE_OPTION,
CATALOG_DRAFT_GUIDANCE_OPTION})
_MASK_SPLIT = re.compile(r"[,，、;；\n]+")
backend/app/data/recipe_ai.py
+20 -27
View File
@@ -6,7 +6,8 @@
必须人工确认之后才能启用。每月重传按已确认的配方重放，不再调用模型。
**为什么不用 get_chat_model。** 它在没有接入时会静默退回演示模型；拿演示模型起草等于假装起草了。这里
自己按助手的模型设置解析接入，没有、停用、只有演示接入、缺 key 都在调用模型之前判定为不可用。
自己解析接入（取值顺序和数据目录起草共用 data/draft_model.py：设置里的起草模型 > 助手的模型 > 默认接入），
没有、停用、只有演示接入、缺 key 都在调用模型之前判定为不可用。
**调用参数照 c837cdb 修过的问题来**：关思考（thinking="off"）、额度 32768、服务方嫌额度大时退回 8192；
正文一律经 message_text 抽取（回复可能是块列表）；输出被截断单独记一种失败，不当成「不是 JSON」去修订。
@@ -42,6 +43,7 @@ from app.data.recipe_types import (
GridCell,
Recipe,
)
from app.data.draft_model import COPILOT_SETTING_KEY as _DRAFT_COPILOT_KEY
from app.data.tabular import EXCEL_ERRORS
from app.data.xlsx_scan import WorkbookScan, cell_ref, col_letter
@@ -58,8 +60,9 @@ class AiTooLarge(ValueError):
# 没有 AiFailed：调用之后的任何失败都体现在返回值里（draft=None、error、usage 照记）
#: 助手的模型设置键（与 app/api/copilot.py 的 COPILOT_SETTING_KEY 同值；数据层不导入接口层，测试断言两者相等）
COPILOT_SETTING_KEY = "copilot"
#: 助手的模型设置键（与 app/api/copilot.py 的 COPILOT_SETTING_KEY 同值；数据层不导入接口层，测试断言两者相等）。
#: 取模型时由 data/draft_model.py 读，这里留着这个名字给测试和老代码用
COPILOT_SETTING_KEY = _DRAFT_COPILOT_KEY
#: 1 次起草 + 最多 2 次修订
AI_MAX_ATTEMPTS = 3
#: 单次调用的超时（秒）；测试 monkeypatch 调小
@@ -252,37 +255,27 @@ class _BudgetFallback:
return await self.model.ainvoke(messages)
def _unconfigured(e: BaseException) -> str:
"""和 copilot._unconfigured 同一句（providers/factory.assistant_unavailable）。"""
from app.providers.factory import assistant_unavailable
return assistant_unavailable(e)
async def resolve_draft_model(session: AsyncSession) -> tuple[Any | None, str, AiAvailability]:
"""(可调用的模型, 模型 id, 可用性)；不可用时模型为 None。只建模型对象，不发请求。"""
from app.db.models import Setting
from app.providers.factory import ModelSpec, ProviderNotConfigured, build_chat_model, resolve_provider
"""(可调用的模型, 模型 id, 可用性)；不可用时模型为 None。只建模型对象，不发请求。
用哪个接入、哪个模型和数据目录起草共用一处取值（data/draft_model.py：起草模型 > 助手的模型 > 默认接入）；
额度、关思考照旧是这里自己的。"""
from app.data.draft_model import DraftModelUnavailable, draft_unavailable, resolve_draft_provider
from app.providers.factory import ModelSpec, ProviderNotConfigured, build_chat_model
row = await session.get(Setting, COPILOT_SETTING_KEY)
saved = (row.value if row else None) or {}
spec = ModelSpec(provider=saved.get("provider") or None, model=saved.get("model") or None,
max_tokens=AI_MAX_TOKENS, temperature=0, thinking="off")
try:
provider = await resolve_provider(session, spec.provider, spec.model)
except ProviderNotConfigured as e:
# 存的接入名不存在、已停用，只存了模型名、提供它的接入已停用：resolve_provider 都会报出来
return None, "", AiAvailability(False, reason=_unconfigured(e))
if provider is None:
return None, "", AiAvailability(False, reason="未配置模型接入，无法使用 AI 起草。请到「设置 → 模型接入」添加")
if provider.kind == "mock":
return None, "", AiAvailability(False, reason="当前只有演示用的模型接入，无法用于 AI 起草。"
"请到「设置 → 模型接入」添加真实的模型接入")
provider, choice = await resolve_draft_provider(
session, none_reason="未配置模型接入，无法使用 AI 起草。请到「设置 → 模型接入」添加",
mock_reason="当前只有演示用的模型接入，无法用于 AI 起草。请到「设置 → 模型接入」添加真实的模型接入")
except DraftModelUnavailable as e:
return None, "", AiAvailability(False, reason=str(e))
spec = ModelSpec(provider=choice.provider, model=choice.model, max_tokens=AI_MAX_TOKENS, temperature=0,
thinking="off")
try:
big = build_chat_model(provider, spec)
small = build_chat_model(provider, spec.model_copy(update={"max_tokens": AI_FALLBACK_MAX_TOKENS}))
except ProviderNotConfigured as e:
return None, "", AiAvailability(False, reason=_unconfigured(e))
return None, "", AiAvailability(False, reason=draft_unavailable(e, choice.origin))
model_id = spec.model or provider.default_model or ""
return (_BudgetFallback(big, small, model_id), model_id,
AiAvailability(True, model=model_id, provider=provider.name))
backend/app/db/base.py
+8
View File
@@ -204,6 +204,14 @@ _COLUMN_MIGRATIONS: list[tuple[str, str, str]] = [
# 没有选项，也不是定时任务发起的
("runs", "options", "JSON DEFAULT '{}'"),
("runs", "schedule_id", "VARCHAR(32)"),
# 数据目录起草任务（验收后修正）：超时、没写出来的调用次数，当前的列数上限，提醒；逐表的模型留空的列。
# 存量任务回填 0 和 NULL：升级前没记这些
("catalog_draft_jobs", "timeouts", "INTEGER DEFAULT 0"),
("catalog_draft_jobs", "failed_calls", "INTEGER DEFAULT 0"),
("catalog_draft_jobs", "part_columns", "INTEGER"),
("catalog_draft_jobs", "notice", "TEXT"),
("catalog_draft_tasks", "blank_columns", "INTEGER DEFAULT 0"),
("catalog_draft_tasks", "blank_sample", "JSON"),
]
#: 迁移补上的列要建的索引：(索引名, 表, 列)。create_all 不碰已有的表，表上新列的索引也就不会建。
backend/app/db/models.py
+81 -1
View File
@@ -4,7 +4,7 @@ from datetime import datetime
from typing import Any
from sqlalchemy import (
Float, ForeignKey, Index, Integer, LargeBinary, String, Text, UniqueConstraint,
Float, ForeignKey, Index, Integer, LargeBinary, String, Text, UniqueConstraint, text,
)
from sqlalchemy.orm import Mapped, mapped_column, relationship
@@ -348,6 +348,86 @@ class CatalogNote(Base, TimestampMixin):
updated_by: Mapped[str | None] = mapped_column(String(100), default=None)
class CatalogDraftJob(Base, TimestampMixin):
"""数据目录的一次后台起草任务（data/catalog_jobs.py）。逐表的状态在 catalog_draft_tasks。
以前起草是一次同步长请求，出错整个对话框就停、关掉之后没有任何记录哪些表失败、为什么；现在建任务后在后台起草，
进度（每张表到第几段、计数、用量）随时落库，接口读的是库里的状态。
同一个数据源同时只有一个排队或进行中的任务：接口先查一遍，部分唯一索引兜底（两个请求同时建，后提交的撞上）。
服务重启时还没结束的标成 interrupted，不自动续跑。挂外键、级联删除：任务是数据源的附属记录，源删掉了留着没有意义。
"""
__tablename__ = "catalog_draft_jobs"
__table_args__ = (
Index("ix_catalog_draft_jobs_source_created", "source_id", "created_at"),
Index("ux_catalog_draft_jobs_active", "source_id", unique=True,
sqlite_where=text("status IN ('queued', 'running')")),
)
id: Mapped[str] = mapped_column(String(32), primary_key=True, default=new_id)
source_id: Mapped[str] = mapped_column(ForeignKey("data_sources.id", ondelete="CASCADE"))
# queued | running | succeeded | partial | failed | cancelled | interrupted
status: Mapped[str] = mapped_column(String(20), default="queued")
use_model: Mapped[bool] = mapped_column(default=False)
# 实际用的模型 id（用模型时开始起草后写上）
model: Mapped[str | None] = mapped_column(String(200), default=None)
# 建任务时的快照：数据源级的起草要求、这一次的补充说明
guidance: Mapped[str | None] = mapped_column(Text, default=None)
note: Mapped[str | None] = mapped_column(Text, default=None)
# 请求头 X-Actor 的署名（自报、未认证）
created_by: Mapped[str | None] = mapped_column(String(100), default=None)
started_at: Mapped[datetime | None] = mapped_column(UTCDateTime, default=None)
finished_at: Mapped[datetime | None] = mapped_column(UTCDateTime, default=None)
# 用量：调用次数、输入和输出 token（拿不到用量的调用按 0 算，超时的调用也是 0）、超时几次、没写出来几次（额度用完、
# 只给空壳、超时、出错）
calls: Mapped[int] = mapped_column(Integer, default=0)
input_tokens: Mapped[int] = mapped_column(Integer, default=0)
output_tokens: Mapped[int] = mapped_column(Integer, default=0)
timeouts: Mapped[int] = mapped_column(Integer, default=0)
failed_calls: Mapped[int] = mapped_column(Integer, default=0)
# 新开始的段每段最多几列（额度用完、超时多了会减半；用模型时开始起草后写上）
part_columns: Mapped[int | None] = mapped_column(Integer, default=None)
# 提醒：最近几次调用多数没写出来时的一句话（建议换起草模型、减少表或停止），少了清掉
notice: Mapped[str | None] = mapped_column(Text, default=None)
# 任务整体出错的原因（逐表的原因在 catalog_draft_tasks）
error: Mapped[str | None] = mapped_column(Text, default=None)
# 重试建的任务记下是重试哪一个
retry_of: Mapped[str | None] = mapped_column(String(32), default=None)
class CatalogDraftTask(Base):
"""起草任务里的一张表。按 seq（起草顺序）排。状态：pending 排队、running 起草中、ok、partial 模型只写出一部分、
failed、skipped 表结构里没有、cancelled（停止或服务重启）。"""
__tablename__ = "catalog_draft_tasks"
__table_args__ = (UniqueConstraint("job_id", "seq"),)
id: Mapped[str] = mapped_column(String(32), primary_key=True, default=new_id)
job_id: Mapped[str] = mapped_column(ForeignKey("catalog_draft_jobs.id", ondelete="CASCADE"), index=True)
seq: Mapped[int] = mapped_column(Integer)
# 表结构里的键；表结构里没有的（skipped）是请求里的原样名字
table_name: Mapped[str] = mapped_column(String(255))
status: Mapped[str] = mapped_column(String(20), default="pending")
column_count: Mapped[int] = mapped_column(Integer, default=0)
# 模型分几段起草（不用模型时为 NULL）、已经结束了几段
parts_total: Mapped[int | None] = mapped_column(Integer, default=None)
parts_done: Mapped[int] = mapped_column(Integer, default=0)
added: Mapped[int] = mapped_column(Integer, default=0)
updated: Mapped[int] = mapped_column(Integer, default=0)
removed: Mapped[int] = mapped_column(Integer, default=0)
version: Mapped[int] = mapped_column(Integer, default=0)
model_items: Mapped[int | None] = mapped_column(Integer, default=None)
# 模型拿不准、留空了的列（中文名和含义都没给）：几列，最多记 5 个列名。表的状态不因此变
blank_columns: Mapped[int] = mapped_column(Integer, default=0)
blank_sample: Mapped[list[Any] | None] = mapped_column(default=None)
# error：整张表没起草（表结构里没有、写入一直冲突、服务重启中断）；model_error：模型那部分没写出来的原因和办法
error: Mapped[str | None] = mapped_column(Text, default=None)
model_error: Mapped[str | None] = mapped_column(Text, default=None)
started_at: Mapped[datetime | None] = mapped_column(UTCDateTime, default=None)
finished_at: Mapped[datetime | None] = mapped_column(UTCDateTime, default=None)
class Setting(Base, TimestampMixin):
"""键值形式的用户设置，前端 Settings 页直接读写。"""
backend/app/main.py
+11 -1
View File
@@ -70,6 +70,15 @@ async def lifespan(_: FastAPI) -> AsyncIterator[None]:
except Exception: # noqa: BLE001
logger.exception("上传表格的版本整理失败，服务照常启动")
# 上次没做完的数据目录起草任务：标成中断（没结束的表记停止），不自动续跑——拷来的数据目录（检查脚本、验收副本）
# 启动时不能替人去调模型。界面上点「重试」接着起草没写好的表
from app.data import catalog_jobs
try:
await catalog_jobs.startup()
except Exception: # noqa: BLE001
logger.exception("数据目录起草任务的启动整理失败，服务照常启动")
await run_manager.setup()
from app.sandbox.manager import sandbox_manager as _sbm
@@ -84,8 +93,9 @@ async def lifespan(_: FastAPI) -> AsyncIterator[None]:
try:
yield
finally:
# 先停调度，再停运行：停运行的途中不该再有新运行被发起
# 先停调度，再停运行：停运行的途中不该再有新运行被发起。起草任务停下来标成中断，下次启动不会再续
await scheduler.stop()
await catalog_jobs.manager.shutdown()
await run_manager.shutdown()
from app.sandbox.manager import sandbox_manager
backend/app/providers/factory.py
+7 -1
View File
@@ -28,6 +28,9 @@ class ModelSpec(BaseModel):
thinking: str | None = None # off | adaptive
effort: str | None = None # low | medium | high | xhigh | max
timeout: int | None = None
# SDK 自己遇到 429、5xx、连接错误、超时时重试几次；None 照 SDK 默认（openai、anthropic 都是 2 次）。
# 自己按失败原因重试的调用方（数据目录起草）设 0，免得两层重试叠在一起、实际次数和报错写的对不上
max_retries: int | None = None
extra: dict[str, Any] = Field(default_factory=dict)
@@ -227,6 +230,8 @@ def build_chat_model(provider: Provider, spec: ModelSpec) -> BaseChatModel:
# 以前只有节点显式配了才传，而没有任何地方给 spec.timeout 赋过值——所有调用
# 都走 SDK 默认的 10 分钟
kwargs["timeout"] = spec.timeout or settings.model_timeout_seconds
if spec.max_retries is not None:
kwargs["max_retries"] = spec.max_retries
# 采样参数：能力允许才传
if spec.temperature is not None and catalog.supports_sampling(model):
@@ -314,7 +319,8 @@ def build_chat_model(provider: Provider, spec: ModelSpec) -> BaseChatModel:
# 且 api_key=None 时不会发出 x-api-key。
import anthropic
client_args = {"base_url": base_url, "api_key": None, "auth_token": api_key}
client_args = {"base_url": base_url, "api_key": None, "auth_token": api_key,
"max_retries": chat.max_retries}
chat._client = anthropic.Anthropic(**client_args)
chat._async_client = anthropic.AsyncAnthropic(**client_args)
return chat
backend/tests/fixtures/catalog/fake_model.py
Vendored
+106 -10
View File
@@ -3,10 +3,14 @@
model = FakeModel({"visits": {"label": "入园记录", "kind": "fact"}}, fail={"orders"})
await catalog.draft_with_model(model, source, ["visits"])
- 结构化输出（with_structured_output(...).ainvoke）从提示词里认出这一批有哪些表（「表 <表名>」），
返回 {"tables": [{"table": 表名, **replies[表名]}, …]}；
- 结构化输出（with_structured_output(...).ainvoke）从提示词里认出这一段请它写哪些表、每张表写哪部分（requested：
整张表、只写表级的项、只写列出的几列、表级的项加几列），返回 {"tables": [{"table": 表名, …}, …]}，只给请它写的
那部分（整张表给 replies[表名] 的全部；只写几列时只给 replies 里这几列）；
- 照提示词的要求，请它写的每一列都列出来：replies 里没写的列只给列名（{"name": 列名}，字段留空，是拿不准），
不整列漏掉。list_all=False 时只给 replies 里有的列（模拟漏列的模型：漏掉的列会被补问一次，补问后仍没给的算没写出来）；
- 直接 ainvoke 是纯文本那条路：同样的内容写成一段正文（AIMessage），前面带一句话；
- fail 里的表所在的那一批抛异常，用来验证「一批失败只影响这一批」；
- fail 里的表所在的那一段抛异常，用来验证「一段失败只影响这一段」；
- hang=N：头 N 次调用一直不回（等起草引擎按时限掐掉，当作超时）；
- calls 记下每次收到的提示词全文，用来核对交给模型的内容（只有结构，没有数据）；modes 记下每次走的是
structured 还是 text。
@@ -19,15 +23,95 @@ structured、text 模拟实测见过的几种回复（两条路各自设）：
"""
from __future__ import annotations
import asyncio
import json
from typing import Any
from app.data.catalog import CATALOG_DRAFT_PROMPT_LABELS as _P
#: 表名下一行写着「这次只写…」时，请模型写的是哪一部分（data/catalog_draft.py 按这几句写提示词）
_ONLY_TABLE = _P["only_table"]
_ONLY_COLUMNS = _P["only_columns"].split("{")[0]
_TABLE_AND = _P["table_and_columns"].split("{")[0]
def requested(text: str) -> dict[str, tuple[str, list[str] | None]]:
"""提示词里请模型写哪些表、每张表写哪部分：{表名: (哪部分, 列)}。哪部分是 full（整张表）、table（只写表级的项）、
columns（只写列出的列，列是表结构描述里列出的那几列）、table+columns（表级的项加「这次只写」那一行点名的列）。"""
out: dict[str, tuple[str, list[str] | None]] = {}
for chunk in text.split("\n\n"):
lines = chunk.split("\n")
head = lines[0]
if not head.startswith("表 "):
continue
name = head[2:]
if name.endswith("（视图）"):
name = name[:-len("（视图）")]
second = lines[1] if len(lines) > 1 else ""
listed = [line[2:].split(" ")[0] for line in lines if line.startswith("- ")]
if second.startswith(_ONLY_TABLE):
out[name] = ("table", None)
elif second.startswith(_ONLY_COLUMNS):
out[name] = ("columns", listed)
elif second.startswith(_TABLE_AND):
out[name] = ("table+columns", second.split("：", 1)[1].split("、"))
else:
out[name] = ("full", None)
return out
def answer(reply: dict, name: str, part: str, columns: list[str] | None) -> dict:
"""按请它写的那部分，从 replies[表名] 里取。"""
if part == "full":
return {"table": name, **reply}
out: dict[str, Any] = {"table": name}
if part in ("table", "table+columns"):
out.update({k: v for k, v in reply.items() if k != "columns"})
if part in ("columns", "table+columns"):
wanted = set(columns or [])
out["columns"] = [c for c in reply.get("columns") or [] if isinstance(c, dict) and c.get("name") in wanted]
return out
def listed(text: str) -> dict[str, list[str]]:
"""提示词里每张表的结构描述列出了哪些列（「- 列名 类型」那几行）。"""
out: dict[str, list[str]] = {}
for chunk in text.split("\n\n"):
lines = chunk.split("\n")
if not lines[0].startswith("表 "):
continue
name = lines[0][2:]
if name.endswith("（视图）"):
name = name[:-len("（视图）")]
out[name] = [line[2:].split(" ")[0] for line in lines if line.startswith("- ")]
return out
def wanted_columns(text: str, name: str, part: str, columns: list[str] | None) -> list[str]:
"""这一段请它写这张表的哪几列：整张表是结构描述里的全部列；只写表级的项时没有。"""
if part == "full":
return listed(text).get(name, [])
if part in ("columns", "table+columns"):
return list(columns or [])
return []
def with_every_column(entry: dict, wanted: list[str]) -> dict:
"""请写的列都列出来：没写的只给列名（字段留空）。"""
have = {c.get("name") for c in entry.get("columns") or [] if isinstance(c, dict)}
missing = [{"name": c} for c in wanted if c not in have]
if not missing:
return entry
return {**entry, "columns": list(entry.get("columns") or []) + missing}
class FakeModel:
def __init__(self, replies: dict[str, dict], fail: set[str] = frozenset(), *,
structured: str = "tables", text: str = "tables") -> None:
structured: str = "tables", text: str = "tables", list_all: bool = True, hang: int = 0) -> None:
self.replies, self.fail, self.calls = replies, set(fail), []
self.structured, self.text = structured, text
self.list_all = list_all
self.hang = hang
self.modes: list[str] = []
def with_structured_output(self, schema: Any, **_: Any) -> "_Structured":
@@ -37,24 +121,36 @@ class FakeModel:
async def ainvoke(self, messages: list[Any]) -> Any:
from langchain_core.messages import AIMessage
reply = self._reply(messages, "text")
reply = await self._answer(messages, "text")
return AIMessage(content="草稿如下：" + json.dumps(reply, ensure_ascii=False))
async def _answer(self, messages: list[Any], mode: str) -> dict:
if self.hang > 0:
self.hang -= 1
self.calls.append("\n".join(str(getattr(m, "content", m)) for m in messages))
self.modes.append(mode)
await asyncio.sleep(3600)
return self._reply(messages, mode)
def _reply(self, messages: list[Any], mode: str) -> dict:
text = "\n".join(str(getattr(m, "content", m)) for m in messages)
self.calls.append(text)
self.modes.append(mode)
names = [t for t in self.replies if f"表 {t}\n" in text or f"表 {t}（" in text]
if any(t in self.fail for t in names):
asked = requested(text)
names = [t for t in asked if t in self.replies]
if any(t in self.fail for t in asked):
raise RuntimeError("网关超时")
shape = self.structured if mode == "structured" else self.text
if shape == "error":
raise NotImplementedError("这条路用不了")
if shape == "hollow":
return {"tables": [{"table": t} for t in names]}
answers = [answer(self.replies[t], t, *asked[t]) for t in names]
if self.list_all:
answers = [with_every_column(a, wanted_columns(text, t, *asked[t])) for a, t in zip(answers, names)]
if shape == "single":
return {"table": names[0], **self.replies[names[0]]} if names else {}
return {"tables": [{"table": t, **self.replies[t]} for t in names]}
return answers[0] if answers else {}
return {"tables": answers}
class _Structured:
@@ -62,4 +158,4 @@ class _Structured:
self.owner = owner
async def ainvoke(self, messages: list[Any]) -> dict:
return self.owner._reply(messages, "structured")
return await self.owner._answer(messages, "structured")
backend/tests/test_catalog_draft.py
+49 -3
View File
@@ -67,6 +67,48 @@ def test_long_table_comment_goes_to_description():
assert draft.notes["description"]["value"].startswith("游客每次检票入园")
@pytest.mark.parametrize("comment, role", [
("实收金额", "label"),
("Paid amount", "label"), # 带空格的英文短语照旧当名字
("状态：1 表示有效，0 表示作废", "note"),
("paid_amt", None), ("PAID-AMT", None), ("Paid Amt", None), # 和列名相同（忽略大小写、下划线、连字符、空白）
("member_level_code", None), ("ord.status", None), ("GMV", None), ("v2", None), # 标识符写法
("member-level", "label"), # 带连字符、和名字不同：不是标识符写法
])
def test_comment_role_by_text_and_name(comment, role):
assert catalog.comment_role(comment, "paid_amt") == role
def test_a_comment_that_is_just_the_name_or_an_identifier_is_not_a_label():
"""注释就是表名本身（列注释就是列名）、或者是标识符写法：不当中文名，也不当说明（不然它比模型起草可信，模型起草的
中文名、含义永远进不了目录）；交给模型的表结构里照旧有这条注释。"""
src = _source({"member_card": _table(
_col("id"), _col("card_no", "TEXT", comment="CARD-NO"), _col("open_dt", "TEXT", comment="open_date"),
_col("bal", "REAL", comment="卡内余额"), comment="MEMBER CARD")})
draft = catalog.draft_structure(src, "member_card")
assert "label" not in draft.notes and "description" not in draft.notes
assert draft.notes["columns"] == {"bal": {"label": make_item("卡内余额", "comment")}}
assert "comment" in draft.covered
# schema 前缀不算：注释写的是不带前缀的表名，同样不当中文名
src = _source({"core.member_card": _table(_col("id"), comment="MEMBER_CARD")})
assert "label" not in catalog.draft_structure(src, "core.member_card").notes
async def test_a_redraft_drops_an_old_label_that_was_just_the_name_and_takes_the_model_one():
"""以前按老规矩把「注释就是表名」记成了中文名（注释的级别高于模型起草）：以后起草时这条注释不再当中文名，旧的那一项
随这一轮的注释一起删掉，模型起草的中文名就能进目录。已有目录不另做迁移，起草到这张表时才变。"""
tables = {"member_card": _table(_col("id"), _col("bal", "REAL"), comment="MEMBER_CARD", fks=[])}
row = await _db_source(tables)
async with SessionLocal() as session:
await catalog.write_entry(session, row.id, "member_card", {"label": make_item("MEMBER_CARD", "comment")},
if_version=0, actor=None)
report = await catalog.draft_catalog(session, row, tables=["member_card"], use_model=True,
model=FakeModel({"member_card": {"label": "会员卡"}}))
entry = await catalog.read_entry(session, row.id, "member_card")
assert entry.notes["label"]["value"] == "会员卡" and entry.notes["label"]["source"] == "llm"
assert (report.tables[0].added, report.tables[0].removed) == (1, 1)
@pytest.mark.parametrize("kw", [{"origin": "upload"}, {"import_mode": "recipe"}])
def test_imported_table_comments_are_not_copied(kw):
"""导入表格的说明是按核对结果生成的（recipe_notes），数据目录不重复它们。"""
@@ -274,11 +316,15 @@ async def test_model_single_table_object_without_wrapper_is_recognized():
async def test_hollow_structured_output_retries_with_text():
"""结构化输出只给了 {"tables":[{"table":"visits"}]} 这样的空壳：改走纯文本再问一次，用纯文本给的内容。"""
model = FakeModel({"visits": {"label": "入园记录", "kind": "fact"}}, structured="hollow", text="tables")
"""结构化输出只给了 {"tables":[{"table":"visits"}]} 这样的空壳、这一段又拆不开（不到 10 列）：改走纯文本再问一次，
用纯文本给的内容。纯文本只给了表级的项、一列都没列出来：再补问一次没给的列（只问缺的，data/catalog_draft.py）；
补问后仍没给，这几列算没写出来，表是只写出一部分（旧的模型起草项不删）。"""
model = FakeModel({"visits": {"label": "入园记录", "kind": "fact"}}, structured="hollow", text="tables",
list_all=False)
got = await catalog.draft_with_model(model, _source(_scenic_tables()), ["visits"])
assert model.modes == ["structured", "text"]
assert model.modes == ["structured", "text", "text"]
assert got["visits"].notes == {"label": make_item("入园记录", "llm"), "kind": make_item("fact", "llm")}
assert got["visits"].covered == frozenset() and "有 5 列（id 到 status）没写出来" in got["visits"].error
async def test_hollow_structured_then_single_text_object():
backend/tests/test_catalog_draft_engine.py
+1301
View File
File diff suppressed because it is too large Load Diff
backend/tests/test_catalog_draft_jobs.py
+781
View File
@@ -0,0 +1,781 @@
"""数据目录的后台起草任务（data/catalog_jobs.py、/api/datasources/{id}/catalog/draft-jobs）与起草要求。
- 建任务 202；同一个源已有排队或进行中的任务 409 catalog_draft_busy；起草模型用不了 409
catalog_draft_model_unavailable（不建任务）；没有表结构 409；表结构里没有的表照样建进任务、状态 skipped
- 在后台起草，每张表一结束就并入目录；进度、计数、用量落库，接口读的是库里的状态
- 停止：排队的直接 cancelled；进行中的停掉正在进行的模型调用，没写完的表记 cancelled，已经写进目录的不回滚；已结束的 409
- 服务启动时把没结束的任务标成 interrupted（没结束的表记 cancelled、写明服务重启），不自动续跑；关停时同样
- 重试：只含原任务里 failed、partial、cancelled 的表，本次说明沿用、起草要求取数据源当前的；没有可重试的 422
- 目录清单：每行 last_draft（带模型留空的列数），顶层 guidance、active_job
- 重新起草时模型这次留空、给的值没通过校验：以前起草好的模型项留着，删除只在列已经不在表结构里时才有
- 用量记超时次数、没写出来的调用次数；任务记当前的列数上限、提醒（最近 6 次调用 4 次没写出来）；逐表记模型留空的列
- 旧的同步起草：同一个源有排队或进行中的任务时 409 catalog_draft_busy
- 起草要求存在数据源 options 里（上传表格的源存在它自己那一行），数据源接口不显示、整份回写 options 时留着，拼连接串时摘掉
表名、列名、说明文字都是虚构的。
"""
from __future__ import annotations
import asyncio
import uuid
from types import SimpleNamespace
from urllib.parse import quote
import pytest
from httpx import ASGITransport, AsyncClient
from sqlalchemy import select
from app.data import catalog, catalog_jobs
from app.data.engine import CATALOG_DRAFT_GUIDANCE_OPTION, build_url
from app.db.base import SessionLocal
from app.db.models import CatalogDraftJob, CatalogDraftTask, DataSource
from app.main import app as _fastapi_app
from tests.fixtures.catalog.fake_model import FakeModel, requested
from tests.fixtures.sources import drop_source
ACTOR = {"X-Actor": quote("王敏")}
# --------------------------------------------------------------------------
# 夹具
# --------------------------------------------------------------------------
def _col(name: str, type_: str = "INTEGER", comment: str | None = None) -> dict:
return {"name": name, "type": type_, "nullable": True, "comment": comment}
def _tables() -> dict:
return {
"visits": {"qualified": "visits", "columns": [_col("id"), _col("ticket_no", "TEXT"), _col("visit_time", "TEXT"),
_col("channel_id")],
"primary_key": ["id"], "comment": None, "is_view": False,
"foreign_keys": [{"columns": ["channel_id"], "to_table": "channels", "to_columns": ["id"]}]},
"channels": {"qualified": "channels", "columns": [_col("id"), _col("name", "TEXT")], "primary_key": ["id"],
"comment": "售票渠道", "is_view": False, "foreign_keys": []},
"gates": {"qualified": "gates", "columns": [_col("id"), _col("gate_code", "TEXT")], "primary_key": ["id"],
"comment": None, "is_view": False, "foreign_keys": []},
}
REPLIES = {
"visits": {"label": "入园记录", "kind": "fact", "columns": [{"name": "ticket_no", "label": "票号"}]},
"channels": {"label": "渠道", "kind": "dimension"},
"gates": {"label": "闸口", "kind": "dimension"},
}
_CREATED: list[str] = []
@pytest.fixture(autouse=True)
async def _cleanup():
yield
# 后台任务先收完（停不下来的停掉），再删源：删源会级联删掉任务
await catalog_jobs.manager.shutdown(grace=5)
while _CREATED:
await drop_source(_CREATED.pop())
async def _source(tables: dict | None = None, **kw) -> str:
async with SessionLocal() as session:
row = DataSource(name=f"cdj_{uuid.uuid4().hex[:10]}", kind="sqlite", database=":memory:",
schema_cache={"tables": tables if tables is not None else _tables()}, **kw)
session.add(row)
await session.commit()
_CREATED.append(row.id)
return row.id
@pytest.fixture
async def client():
async with AsyncClient(transport=ASGITransport(app=_fastapi_app), base_url="http://test") as c:
yield c
def _base(sid: str) -> str:
return f"/api/datasources/{sid}/catalog"
def use(monkeypatch, model) -> None:
"""起草模型换成假模型（建任务时查一次能不能用，开始起草时再取一次）。"""
async def resolve(session):
return model, "fake-model"
monkeypatch.setattr(catalog, "resolve_draft_model", resolve)
async def finished(job_id: str) -> None:
await catalog_jobs.manager.wait(job_id, timeout=10)
async def get_job(client, sid: str, job_id: str) -> dict:
r = await client.get(f"{_base(sid)}/draft-jobs/{job_id}")
assert r.status_code == 200, r.text
return r.json()["job"]
class Held:
"""假模型：请求里有 hold 里的表时，等 release 才回；被停止时记下这次调用被取消了。"""
def __init__(self, replies: dict, hold: set[str]) -> None:
self.inner, self.hold = FakeModel(replies), set(hold)
self.release = asyncio.Event()
self.cancelled = 0
self.model_name = "held-model"
def with_structured_output(self, schema, **_):
outer = self
class _S:
async def ainvoke(self, messages):
text = "\n".join(str(m.content) for m in messages)
if outer.hold & set(requested(text)):
try:
await outer.release.wait()
except asyncio.CancelledError:
outer.cancelled += 1
raise
return outer.inner._reply(messages, "structured")
return _S()
async def ainvoke(self, messages):
return await self.inner.ainvoke(messages)
async def until(predicate, timeout: float = 5.0) -> None:
for _ in range(int(timeout / 0.01)):
if await predicate():
return
await asyncio.sleep(0.01)
raise AssertionError("等不到")
# --------------------------------------------------------------------------
# 建任务、在后台起草
# --------------------------------------------------------------------------
async def test_a_job_drafts_in_the_background_and_writes_each_table(client, monkeypatch):
sid = await _source()
model = FakeModel(REPLIES)
use(monkeypatch, model)
await client.put(f"{_base(sid)}/guidance", json={"guidance": "金额一律按元写单位"})
r = await client.post(f"{_base(sid)}/draft-jobs", headers=ACTOR,
json={"tables": ["visits", "nope", "channels"], "use_model": True, "note": " 看渠道 "})
assert r.status_code == 202, r.text
job = r.json()["job"]
assert job["status"] in ("queued", "running") and job["source_id"] == sid
assert job["use_model"] is True and job["guidance"] == "金额一律按元写单位" and job["note"] == "看渠道"
assert job["created_by"] == "王敏" and job["total"] == 3
assert [t["table_name"] for t in job["tasks"]] == ["visits", "nope", "channels"], "按起草顺序"
await finished(job["id"])
job = await get_job(client, sid, job["id"])
assert job["status"] == "succeeded" and job["error"] is None and job["model"] == "fake-model"
assert job["done"] == 3 and job["counts"] == {"pending": 0, "running": 0, "ok": 2, "partial": 0, "failed": 0,
"skipped": 1, "cancelled": 0}
assert job["started_at"] and job["finished_at"] and job["usage"]["calls"] >= 1
by = {t["table_name"]: t for t in job["tasks"]}
assert by["nope"]["status"] == "skipped" and by["nope"]["error"] == "表结构里没有 nope，请先重新探查结构"
visits = by["visits"]
assert visits["status"] == "ok" and visits["parts_total"] == 1 and visits["parts_done"] == 1
assert visits["model_items"] == 3 and visits["added"] == 4 and visits["version"] == 1 and visits["columns"] == 4
assert visits["started_at"] and visits["finished_at"] and visits["error"] is None
assert job["added"] == sum(t["added"] for t in job["tasks"])
# 起草要求和本次说明都交给了模型
assert "起草要求：金额一律按元写单位" in model.calls[0] and "本次说明：看渠道" in model.calls[0]
detail = (await client.get(f"{_base(sid)}/visits")).json()
assert detail["notes"]["label"]["value"] == "入园记录" and detail["updated_by"] == "王敏"
async def test_a_job_without_the_model_writes_structure_only(client):
sid = await _source()
r = await client.post(f"{_base(sid)}/draft-jobs", json={"tables": [], "use_model": False})
assert r.status_code == 202, r.text
job = r.json()["job"]
assert job["total"] == 3, "不给表时按使用次数取前 20 张"
await finished(job["id"])
job = await get_job(client, sid, job["id"])
assert job["status"] == "succeeded" and job["model"] is None and job["usage"]["calls"] == 0
assert all(t["parts_total"] is None and t["model_items"] is None for t in job["tasks"])
assert {t["table_name"]: t["added"] for t in job["tasks"]}["visits"] == 1 # 外键一条
async def test_a_table_whose_model_part_failed_is_failed_but_structure_is_written(client, monkeypatch):
sid = await _source()
use(monkeypatch, FakeModel(REPLIES, fail={"visits"}))
monkeypatch.setattr(catalog, "MODEL_BATCH_TABLES", 1)
job = (await client.post(f"{_base(sid)}/draft-jobs", json={"tables": ["visits", "channels"],
"use_model": True})).json()["job"]
await finished(job["id"])
job = await get_job(client, sid, job["id"])
by = {t["table_name"]: t for t in job["tasks"]}
assert job["status"] == "partial"
assert by["visits"]["status"] == "failed" and "网关超时" in by["visits"]["model_error"]
assert by["visits"]["error"] is None and by["visits"]["added"] == 1, "外键照常写入"
assert by["channels"]["status"] == "ok"
class HalfThenHollow:
"""假模型：第一次只给前一半列，补问时只给一个空壳（表名对上、一项都没有）。"""
model_name = "half-model"
def __init__(self, table: str, reply: dict) -> None:
self.table, self.reply, self.calls = table, reply, 0
def with_structured_output(self, schema, **_):
outer = self
class _S:
async def ainvoke(self, messages):
outer.calls += 1
if outer.calls == 1:
cols = outer.reply["columns"]
return {"tables": [{"table": outer.table, **outer.reply, "columns": cols[:len(cols) // 2]}]}
return {"tables": [{"table": outer.table}]}
return _S()
async def ainvoke(self, messages):
raise AssertionError("这里不该走纯文本")
async def test_a_followup_that_comes_back_empty_keeps_the_earlier_model_items(client, monkeypatch):
"""补问只给空壳：表是 partial、任务是 partial，以前起草过的列（待确认的模型起草项）不因这次没写全被删掉。"""
cols = [_col(f"c{i:02d}") for i in range(1, 9)]
sid = await _source({"tickets": {"qualified": "tickets", "columns": cols, "primary_key": ["c01"], "comment": None,
"is_view": False, "foreign_keys": []}})
full = {"label": "门票", "kind": "fact",
"columns": [{"name": f"c{i:02d}", "label": f"第{i}列"} for i in range(1, 9)]}
use(monkeypatch, FakeModel({"tickets": full}))
first = (await client.post(f"{_base(sid)}/draft-jobs", json={"tables": ["tickets"], "use_model": True})).json()
await finished(first["job"]["id"])
before = (await client.get(f"{_base(sid)}/tickets")).json()["notes"]["columns"]
assert {c for c, items in before.items() if "label" in items} == {f"c{i:02d}" for i in range(1, 9)}
model = HalfThenHollow("tickets", full)
use(monkeypatch, model)
second = (await client.post(f"{_base(sid)}/draft-jobs", json={"tables": ["tickets"], "use_model": True})).json()
await finished(second["job"]["id"])
job = await get_job(client, sid, second["job"]["id"])
task = job["tasks"][0]
assert model.calls == 2
assert job["status"] == "partial" and task["status"] == "partial" and task["removed"] == 0
assert "有 4 列（c05 到 c08）没写出来" in task["model_error"] and "可以只重试这张表" in task["model_error"]
after = (await client.get(f"{_base(sid)}/tickets")).json()["notes"]["columns"]
assert {c for c, items in after.items() if "label" in items} == {f"c{i:02d}" for i in range(1, 9)}
def _tickets(n: int = 6) -> dict:
return {"tickets": {"qualified": "tickets", "columns": [_col(f"c{i:02d}") for i in range(1, n + 1)],
"primary_key": ["c01"], "comment": None, "is_view": False, "foreign_keys": []}}
async def _draft(client, sid: str, monkeypatch, model, tables: list[str]) -> dict:
use(monkeypatch, model)
r = await client.post(f"{_base(sid)}/draft-jobs", json={"tables": tables, "use_model": True})
assert r.status_code == 202, r.text
await finished(r.json()["job"]["id"])
return await get_job(client, sid, r.json()["job"]["id"])
async def test_a_redraft_keeps_what_the_model_left_blank_this_time(client, monkeypatch):
"""重新起草（比如补了起草要求再起草一次）时模型这次留空、或者给的值没通过校验：以前起草好的模型项留着，不算删除；
给了合格新值的才替换。以前整库重跑一次删掉几百项，表仍记「已完成」。"""
sid = await _source(_tickets())
full = {"label": "门票", "kind": "fact",
"columns": [{"name": f"c{i:02d}", "label": f"第{i}列", "meaning": f"第{i}列的含义"} for i in range(1, 6)]}
first = await _draft(client, sid, monkeypatch, FakeModel({"tickets": full}), ["tickets"])
assert first["tasks"][0]["status"] == "ok" and first["blank_columns"] == 1, "c06 第一次就留空了"
before = (await client.get(f"{_base(sid)}/tickets")).json()["notes"]
# 这一次：表的中文名换了；c03 的中文名像公式（校验丢掉）、含义换了；c04 只给了中文名；其余列列出了但都留空
again = {"label": "门票明细", "columns": [{"name": "c03", "label": "=c01+c02", "meaning": "新的含义"},
{"name": "c04", "label": "第4列"}]}
second = await _draft(client, sid, monkeypatch, FakeModel({"tickets": again}), ["tickets"])
task = second["tasks"][0]
assert task["status"] == "ok" and task["removed"] == 0 and second["removed"] == 0
assert (task["added"], task["updated"]) == (0, 2)
after = (await client.get(f"{_base(sid)}/tickets")).json()["notes"]
assert after["label"]["value"] == "门票明细" and after["kind"] == before["kind"]
assert after["columns"]["c03"]["label"] == before["columns"]["c03"]["label"]
assert after["columns"]["c03"]["meaning"]["value"] == "新的含义"
for col in ("c01", "c02", "c05"):
assert after["columns"][col] == before["columns"][col], col
# 留空的列只记目录里还是没有中文名、含义的（c06）：c01、c02、c05 这次留空，以前起草好的还在，不算留空——不然照
# 提示补了起草要求再起草，提示指着的却是目录里有中文名的列，一直消不掉
assert task["blank_columns"] == 1 and task["blank_sample"] == ["c06"]
async def test_model_items_of_a_column_gone_from_the_schema_are_removed(client, monkeypatch):
sid = await _source(_tickets())
full = {"label": "门票", "columns": [{"name": f"c{i:02d}", "label": f"第{i}列"} for i in range(1, 7)]}
await _draft(client, sid, monkeypatch, FakeModel({"tickets": full}), ["tickets"])
async with SessionLocal() as session:
row = await session.get(DataSource, sid)
row.schema_cache = {"tables": _tickets(5)} # 重新探查过：c06 已经没有了
await session.commit()
job = await _draft(client, sid, monkeypatch, FakeModel({"tickets": {"label": "门票"}}), ["tickets"])
assert job["tasks"][0]["removed"] == 1
after = (await client.get(f"{_base(sid)}/tickets")).json()["notes"]
assert set(after["columns"]) == {"c01", "c02", "c03", "c04", "c05"}
async def test_usage_counts_timeouts_and_failed_calls(client, monkeypatch):
"""超时的调用拆小再问，用量里记超时几次、没写出来几次（超时的调用拿不到用量，token 记 0）；额度用完或超时累计 2 次，
新开始的段列数上限减半，任务上记着当前的上限。"""
monkeypatch.setattr(catalog, "MODEL_TIMEOUT_S", 0.05)
sid = await _source(_tickets(24))
replies = {"tickets": {"label": "门票", "columns": [{"name": f"c{i:02d}", "label": f"第{i}列"}
for i in range(1, 25)]}}
model = FakeModel(replies, hang=2)
job = await _draft(client, sid, monkeypatch, model, ["tickets"])
assert job["status"] == "succeeded" and job["tasks"][0]["status"] == "ok"
assert job["usage"] == {"calls": 5, "input_tokens": 0, "output_tokens": 0, "timeouts": 2, "failed_calls": 2}
assert job["part_columns"] == 20 and job["notice"] is None
# 24 列整张问超时 → 两半同时问：前 12 列（带表级的项）超时、后 12 列写出 → 前 12 列拆成 6、6
assert [len(requested(c)["tickets"][1] or []) for c in model.calls] == [0, 12, 12, 6, 6]
plain = await _draft(client, sid, monkeypatch, FakeModel(replies), ["tickets"])
assert plain["part_columns"] == 40 and plain["usage"]["timeouts"] == plain["usage"]["failed_calls"] == 0
async def test_the_notice_is_kept_on_the_job(client, monkeypatch):
"""最近 6 次调用有 4 次以上没写出来：任务上写一句提醒（次数、原因、建议换起草模型）；不用模型的任务没有。"""
monkeypatch.setattr(catalog, "MODEL_BATCH_TABLES", 1)
tables = {f"t{i}": {"qualified": f"t{i}", "columns": [_col("id"), _col("v")], "primary_key": ["id"],
"comment": None, "is_view": False, "foreign_keys": []} for i in range(1, 6)}
sid = await _source(tables)
replies = {name: {"label": f"表{name}"} for name in tables}
job = await _draft(client, sid, monkeypatch, FakeModel(replies, fail={"t1", "t2", "t3", "t4"}), list(tables))
assert job["usage"]["calls"] == 5 and job["usage"]["failed_calls"] == 4
assert job["notice"] == ("最近 5 次调用有 4 次没写出来（出错 4 次）。建议在「设置 → 偏好设置」的「起草模型」换一个模型、"
"缩小起草范围，或者停止这次起草")
plain = (await client.post(f"{_base(sid)}/draft-jobs", json={"tables": ["t1"]})).json()["job"]
await finished(plain["id"])
plain = await get_job(client, sid, plain["id"])
assert plain["notice"] is None and plain["part_columns"] is None and plain["blank_columns"] == 0
async def test_blank_columns_reach_the_task_the_job_and_the_list(client, monkeypatch):
"""模型列出了、中文名和含义都没给的列：逐表记列数和最多 5 个列名，任务上是合计，目录清单的 last_draft 也带列数。"""
tables = {**_tickets(8), "gates": _tables()["gates"]}
sid = await _source(tables)
replies = {"tickets": {"label": "门票", "columns": [{"name": "c01", "label": "编号"}]},
"gates": {"label": "闸口", "columns": [{"name": "id", "label": "编号"}]}}
job = await _draft(client, sid, monkeypatch, FakeModel(replies), ["tickets", "gates"])
by = {t["table_name"]: t for t in job["tasks"]}
assert by["tickets"]["status"] == "ok" and by["tickets"]["blank_columns"] == 7
assert by["tickets"]["blank_sample"] == ["c02", "c03", "c04", "c05", "c06"]
assert by["gates"]["blank_columns"] == 1 and by["gates"]["blank_sample"] == ["gate_code"]
assert job["blank_columns"] == 8
rows = {r["table_name"]: r for r in (await client.get(_base(sid))).json()["tables"]}
assert rows["tickets"]["last_draft"]["blank_columns"] == 7 and rows["gates"]["last_draft"]["blank_columns"] == 1
async def test_blank_columns_are_counted_against_the_catalog_as_written(client, monkeypatch):
"""留空的列按并入目录之后的结果算：起草进行中有人给某一列填了中文名，模型这次留空，这一列不算留空。"""
sid = await _source(_tickets(4))
model = Held({"tickets": {"label": "门票", "columns": [{"name": "c01", "label": "编号"}]}}, hold={"tickets"})
use(monkeypatch, model)
r = await client.post(f"{_base(sid)}/draft-jobs", json={"tables": ["tickets"], "use_model": True})
job_id = r.json()["job"]["id"]
async def asking() -> bool:
return (await get_job(client, sid, job_id))["tasks"][0]["status"] == "running"
await until(asking)
saved = await client.put(f"{_base(sid)}/tickets", json={"notes": {"columns": {"c02": {"label": {"value": "闸机号"}}}},
"if_version": 0})
assert saved.status_code == 200, saved.text
model.release.set()
await finished(job_id)
task = (await get_job(client, sid, job_id))["tasks"][0]
assert task["status"] == "ok" and task["blank_columns"] == 2 and task["blank_sample"] == ["c03", "c04"]
after = (await client.get(f"{_base(sid)}/tickets")).json()["notes"]["columns"]
assert after["c02"]["label"]["value"] == "闸机号" and after["c01"]["label"]["value"] == "编号"
async def test_the_sync_draft_is_409_while_a_job_is_queued_or_running(client, monkeypatch):
"""旧的同步起草不和后台任务同时写同一个源的目录、抢模型名额：有排队或进行中的任务时 409（和建任务同一句话）。"""
sid = await _source()
start = catalog_jobs.manager.start
monkeypatch.setattr(catalog_jobs.manager, "start", lambda job_id: None) # 排着、不开始
queued = (await client.post(f"{_base(sid)}/draft-jobs", json={"tables": ["gates"]})).json()["job"]
r = await client.post(f"{_base(sid)}/draft", json={"tables": ["channels"]})
assert r.status_code == 409 and r.json()["code"] == "catalog_draft_busy", r.text
busy = await client.post(f"{_base(sid)}/draft-jobs", json={"tables": ["gates"]})
assert busy.status_code == 409 and r.json()["detail"] == busy.json()["detail"]
assert (await client.get(f"{_base(sid)}/channels")).json()["version"] == 0, "什么都没写"
await client.post(f"{_base(sid)}/draft-jobs/{queued['id']}/cancel")
monkeypatch.setattr(catalog_jobs.manager, "start", start)
model = Held(REPLIES, hold={"visits"})
use(monkeypatch, model)
running = (await client.post(f"{_base(sid)}/draft-jobs", json={"tables": ["visits"], "use_model": True})).json()
r = await client.post(f"{_base(sid)}/draft", json={"tables": ["channels"]})
assert r.status_code == 409 and r.json()["code"] == "catalog_draft_busy", r.text
model.release.set()
await finished(running["job"]["id"])
r = await client.post(f"{_base(sid)}/draft", json={"tables": ["channels"]})
assert r.status_code == 200, r.text
def test_old_job_tables_get_the_new_columns():
"""升级前建的起草任务表：新列补上，存量行的计数是 0、提醒和列数上限为空。"""
from sqlalchemy import create_engine, text
from app.db.base import _migrate
db = create_engine("sqlite://")
with db.begin() as conn:
conn.execute(text("CREATE TABLE catalog_draft_jobs (id VARCHAR(32) PRIMARY KEY, calls INTEGER)"))
conn.execute(text("CREATE TABLE catalog_draft_tasks (id VARCHAR(32) PRIMARY KEY, removed INTEGER)"))
conn.execute(text("INSERT INTO catalog_draft_jobs (id, calls) VALUES ('j1', 3)"))
conn.execute(text("INSERT INTO catalog_draft_tasks (id, removed) VALUES ('t1', 0)"))
_migrate(conn)
_migrate(conn)
assert conn.execute(text("SELECT timeouts, failed_calls, part_columns, notice FROM catalog_draft_jobs")) \
.one() == (0, 0, None, None)
assert conn.execute(text("SELECT blank_columns, blank_sample FROM catalog_draft_tasks")).one() == (0, None)
async def test_a_job_that_cannot_start_is_failed_with_the_reason(client, monkeypatch):
"""建任务时起草模型还能用，开始起草时已经用不了（接入刚被停用）：任务整体 failed，error 写原因，没开始的表也记 failed。"""
sid = await _source()
calls: list[int] = []
async def flaky(session):
calls.append(1)
if len(calls) > 1:
raise catalog.CatalogModelUnavailable("起草模型使用的模型接入「虚构接入」已停用")
return FakeModel(REPLIES), "fake-model"
monkeypatch.setattr(catalog, "resolve_draft_model", flaky)
job = (await client.post(f"{_base(sid)}/draft-jobs", json={"tables": ["visits", "gates"],
"use_model": True})).json()["job"]
await finished(job["id"])
got = await get_job(client, sid, job["id"])
assert got["status"] == "failed" and got["error"] == "起草模型使用的模型接入「虚构接入」已停用"
assert {t["status"] for t in got["tasks"]} == {"failed"} and got["finished_at"]
assert all(t["error"] == got["error"] for t in got["tasks"])
retry = await client.post(f"{_base(sid)}/draft-jobs/{job['id']}/retry")
assert retry.status_code == 409 and retry.json()["code"] == "catalog_draft_model_unavailable"
async def test_busy_source_is_409_and_the_active_job_is_reachable(client, monkeypatch):
sid = await _source()
model = Held(REPLIES, hold={"visits"})
use(monkeypatch, model)
first = (await client.post(f"{_base(sid)}/draft-jobs", json={"tables": ["visits"], "use_model": True})).json()
r = await client.post(f"{_base(sid)}/draft-jobs", json={"tables": ["gates"], "use_model": False})
assert r.status_code == 409 and r.json()["code"] == "catalog_draft_busy", r.text
active = (await client.get(f"{_base(sid)}/draft-jobs/active")).json()["job"]
assert active["id"] == first["job"]["id"]
listed = (await client.get(_base(sid))).json()
assert listed["active_job"]["id"] == first["job"]["id"] and listed["active_job"]["total"] == 1
assert listed["active_job"]["status"] in ("queued", "running") and listed["active_job"]["done"] == 0
model.release.set()
await finished(first["job"]["id"])
assert (await client.get(f"{_base(sid)}/draft-jobs/active")).json()["job"] is None
latest = (await client.get(f"{_base(sid)}/draft-jobs/latest")).json()["job"]
assert latest["id"] == first["job"]["id"] and latest["status"] == "succeeded"
assert (await client.get(_base(sid))).json()["active_job"] is None
async def test_unavailable_model_is_409_and_no_job_is_created(client, monkeypatch):
sid = await _source()
async def unavailable(session):
raise catalog.CatalogModelUnavailable("未配置模型接入，无法由模型起草数据目录。请到「设置 → 模型接入」添加")
monkeypatch.setattr(catalog, "resolve_draft_model", unavailable)
r = await client.post(f"{_base(sid)}/draft-jobs", json={"use_model": True})
assert r.status_code == 409, r.text
assert r.json()["code"] == "catalog_draft_model_unavailable" and "模型接入" in r.json()["detail"]
assert (await client.get(f"{_base(sid)}/draft-jobs/latest")).json()["job"] is None
async def test_without_schema_or_with_bad_input_nothing_is_created(client):
sid = await _source(tables={})
r = await client.post(f"{_base(sid)}/draft-jobs", json={"use_model": False})
assert r.status_code == 409 and "探查结构" in r.json()["detail"]
sid = await _source()
r = await client.post(f"{_base(sid)}/draft-jobs", json={"use_model": False, "note": "说" * 1001})
assert r.status_code == 422 and "1000" in r.json()["detail"]
r = await client.post(f"{_base(sid)}/draft-jobs", json={"tables": [f"t{i}" for i in range(501)]})
assert r.status_code == 422 and "500" in r.json()["detail"]
assert (await client.get(f"{_base(sid)}/draft-jobs/latest")).json()["job"] is None
assert (await client.get(f"{_base('missing-source')}/draft-jobs/latest")).status_code == 404
assert (await client.get(f"{_base(sid)}/draft-jobs/not-a-job")).status_code == 404
async def test_a_job_of_another_source_is_404(client):
a, b = await _source(), await _source()
job = (await client.post(f"{_base(a)}/draft-jobs", json={"tables": ["gates"]})).json()["job"]
await finished(job["id"])
assert (await client.get(f"{_base(b)}/draft-jobs/{job['id']}")).status_code == 404
assert (await client.post(f"{_base(b)}/draft-jobs/{job['id']}/cancel")).status_code == 404
assert (await client.post(f"{_base(b)}/draft-jobs/{job['id']}/retry")).status_code == 404
# --------------------------------------------------------------------------
# 停止、服务重启
# --------------------------------------------------------------------------
async def test_cancelling_a_queued_job(client, monkeypatch):
sid = await _source()
monkeypatch.setattr(catalog_jobs.manager, "start", lambda job_id: None) # 不让它开始
job = (await client.post(f"{_base(sid)}/draft-jobs", json={"tables": ["gates", "channels"]})).json()["job"]
assert job["status"] == "queued"
r = await client.post(f"{_base(sid)}/draft-jobs/{job['id']}/cancel")
assert r.status_code == 200, r.text
job = r.json()["job"]
assert job["status"] == "cancelled" and job["finished_at"]
assert {t["status"] for t in job["tasks"]} == {"cancelled"}
again = await client.post(f"{_base(sid)}/draft-jobs/{job['id']}/cancel")
assert again.status_code == 409 and again.json()["code"] == "catalog_draft_finished"
async def test_cancelling_a_running_job_stops_the_call_and_keeps_what_is_written(client, monkeypatch):
sid = await _source()
model = Held(REPLIES, hold={"visits"})
use(monkeypatch, model)
monkeypatch.setattr(catalog, "MODEL_BATCH_TABLES", 1)
job = (await client.post(f"{_base(sid)}/draft-jobs",
json={"tables": ["channels", "visits"], "use_model": True})).json()["job"]
async def channels_written() -> bool:
tasks = {t["table_name"]: t for t in (await get_job(client, sid, job["id"]))["tasks"]}
return tasks["channels"]["status"] == "ok" and tasks["visits"]["status"] == "running"
await until(channels_written)
r = await client.post(f"{_base(sid)}/draft-jobs/{job['id']}/cancel")
assert r.status_code == 200, r.text
after = r.json()["job"]
by = {t["table_name"]: t for t in after["tasks"]}
assert after["status"] == "cancelled" and after["finished_at"]
assert by["channels"]["status"] == "ok" and by["visits"]["status"] == "cancelled"
assert model.cancelled == 1, "正在进行的模型调用被停掉"
detail = (await client.get(f"{_base(sid)}/channels")).json()
assert detail["notes"]["kind"]["value"] == "dimension", "已经写进目录的不回滚"
assert (await client.get(f"{_base(sid)}/draft-jobs/active")).json()["job"] is None
async def test_startup_marks_unfinished_jobs_interrupted_and_does_not_resume(client):
sid = await _source()
async with SessionLocal() as session:
job = CatalogDraftJob(source_id=sid, status="running", use_model=True)
session.add(job)
await session.flush()
for seq, (name, status) in enumerate([("visits", "ok"), ("channels", "running"), ("gates", "pending")]):
session.add(CatalogDraftTask(job_id=job.id, seq=seq, table_name=name, status=status))
await session.commit()
job_id = job.id
assert await catalog_jobs.startup() >= 1
got = await get_job(client, sid, job_id)
by = {t["table_name"]: t for t in got["tasks"]}
assert got["status"] == "interrupted" and got["error"] == "服务重启，起草中断" and got["finished_at"]
assert by["visits"]["status"] == "ok" and by["visits"]["error"] is None
assert by["channels"]["status"] == by["gates"]["status"] == "cancelled"
assert by["channels"]["error"] == by["gates"]["error"] == "服务重启，起草中断"
assert not catalog_jobs.manager.running(job_id), "不自动续跑"
async def test_shutdown_stops_running_jobs_as_interrupted(client, monkeypatch):
sid = await _source()
model = Held(REPLIES, hold={"visits"})
use(monkeypatch, model)
job = (await client.post(f"{_base(sid)}/draft-jobs", json={"tables": ["visits"], "use_model": True})).json()["job"]
async def running() -> bool:
return (await get_job(client, sid, job["id"]))["tasks"][0]["status"] == "running"
await until(running)
await catalog_jobs.manager.shutdown(grace=5)
got = await get_job(client, sid, job["id"])
assert got["status"] == "interrupted" and got["tasks"][0]["status"] == "cancelled"
assert got["tasks"][0]["error"] == "服务重启，起草中断"
# --------------------------------------------------------------------------
# 重试
# --------------------------------------------------------------------------
async def test_retry_takes_only_failed_partial_and_cancelled_tables(client, monkeypatch):
sid = await _source()
use(monkeypatch, FakeModel(REPLIES, fail={"visits"}))
monkeypatch.setattr(catalog, "MODEL_BATCH_TABLES", 1)
await client.put(f"{_base(sid)}/guidance", json={"guidance": "旧的要求"})
first = (await client.post(f"{_base(sid)}/draft-jobs", json={
"tables": ["visits", "channels", "nope"], "use_model": True, "note": "第一次"})).json()["job"]
await finished(first["id"])
await client.put(f"{_base(sid)}/guidance", json={"guidance": "新的要求"})
assert (await get_job(client, sid, first["id"]))["guidance"] == "旧的要求", "任务记下的是当时的快照"
use(monkeypatch, FakeModel(REPLIES))
r = await client.post(f"{_base(sid)}/draft-jobs/{first['id']}/retry", headers=ACTOR)
assert r.status_code == 202, r.text
retry = r.json()["job"]
assert [t["table_name"] for t in retry["tasks"]] == ["visits"]
assert retry["use_model"] is True and retry["note"] == "第一次" and retry["guidance"] == "新的要求"
assert retry["retry_of"] == first["id"] and retry["created_by"] == "王敏"
await finished(retry["id"])
assert (await get_job(client, sid, retry["id"]))["status"] == "succeeded"
nothing = await client.post(f"{_base(sid)}/draft-jobs/{retry['id']}/retry")
assert nothing.status_code == 422 and "没有" in nothing.json()["detail"]
async def test_retry_while_busy_is_409(client, monkeypatch):
sid = await _source()
monkeypatch.setattr(catalog_jobs.manager, "start", lambda job_id: None)
job = (await client.post(f"{_base(sid)}/draft-jobs", json={"tables": ["gates"]})).json()["job"]
await client.post(f"{_base(sid)}/draft-jobs/{job['id']}/cancel")
other = (await client.post(f"{_base(sid)}/draft-jobs", json={"tables": ["channels"]})).json()["job"]
assert other["status"] == "queued"
r = await client.post(f"{_base(sid)}/draft-jobs/{job['id']}/retry")
assert r.status_code == 409 and r.json()["code"] == "catalog_draft_busy"
async def test_retry_with_an_unavailable_model_is_409(client, monkeypatch):
sid = await _source()
monkeypatch.setattr(catalog_jobs.manager, "start", lambda job_id: None)
use(monkeypatch, FakeModel(REPLIES))
job = (await client.post(f"{_base(sid)}/draft-jobs", json={"tables": ["gates"], "use_model": True})).json()["job"]
await client.post(f"{_base(sid)}/draft-jobs/{job['id']}/cancel")
async def unavailable(session):
raise catalog.CatalogModelUnavailable("当前只有演示用的模型接入，无法用于起草数据目录")
monkeypatch.setattr(catalog, "resolve_draft_model", unavailable)
r = await client.post(f"{_base(sid)}/draft-jobs/{job['id']}/retry")
assert r.status_code == 409 and r.json()["code"] == "catalog_draft_model_unavailable"
# --------------------------------------------------------------------------
# 目录清单上的起草状态
# --------------------------------------------------------------------------
async def test_the_list_carries_last_draft_per_table(client, monkeypatch):
sid = await _source()
use(monkeypatch, FakeModel(REPLIES, fail={"visits"}))
monkeypatch.setattr(catalog, "MODEL_BATCH_TABLES", 1)
first = (await client.post(f"{_base(sid)}/draft-jobs", json={"tables": ["visits", "channels"],
"use_model": True})).json()["job"]
await finished(first["id"])
second = (await client.post(f"{_base(sid)}/draft-jobs", json={"tables": ["channels"]})).json()["job"]
await finished(second["id"])
body = (await client.get(_base(sid))).json()
rows = {r["table_name"]: r for r in body["tables"]}
assert rows["gates"]["last_draft"] is None
visits = rows["visits"]["last_draft"]
assert visits["status"] == "failed" and visits["job_id"] == first["id"] and visits["finished_at"]
assert "网关超时" in visits["model_error"] and visits["error"] is None
assert rows["channels"]["last_draft"]["job_id"] == second["id"], "取最近一次任务里的那一项"
assert body["guidance"] is None and body["active_job"] is None
# --------------------------------------------------------------------------
# 起草要求
# --------------------------------------------------------------------------
async def test_guidance_is_saved_trimmed_cleared_and_limited(client):
sid = await _source()
r = await client.put(f"{_base(sid)}/guidance", json={"guidance": " 票务库，金额按元。\n状态 9 是作废 "})
assert r.status_code == 200, r.text
assert r.json() == {"guidance": "票务库，金额按元。\n状态 9 是作废"}
assert (await client.get(_base(sid))).json()["guidance"] == "票务库，金额按元。\n状态 9 是作废"
too_long = await client.put(f"{_base(sid)}/guidance", json={"guidance": "字" * 2001})
assert too_long.status_code == 422 and "2000" in too_long.json()["detail"]
assert (await client.get(_base(sid))).json()["guidance"] == "票务库，金额按元。\n状态 9 是作废"
assert (await client.put(f"{_base(sid)}/guidance", json={"guidance": "字" * 2000})).status_code == 200
cleared = await client.put(f"{_base(sid)}/guidance", json={"guidance": " "})
assert cleared.json() == {"guidance": None}
async with SessionLocal() as session:
row = await session.get(DataSource, sid)
assert CATALOG_DRAFT_GUIDANCE_OPTION not in (row.options or {})
assert (await client.put(f"{_base('missing-source')}/guidance", json={"guidance": "x"})).status_code == 404
async def test_a_table_named_guidance_can_still_be_saved(client):
"""起草要求的路径和「整份提交一张表」撞名：表名恰好叫 guidance 时，{notes, if_version} 照旧是那张表的提交，
不会被当成清除起草要求；两样都没写的 422。"""
tables = _tables()
tables["guidance"] = {"qualified": "guidance", "columns": [_col("id"), _col("body", "TEXT")],
"primary_key": ["id"], "comment": None, "is_view": False, "foreign_keys": []}
sid = await _source(tables)
await client.put(f"{_base(sid)}/guidance", json={"guidance": "金额按元"})
r = await client.put(f"{_base(sid)}/guidance", json={"notes": {"label": {"value": "操作指引"}}, "if_version": 0},
headers=ACTOR)
assert r.status_code == 200, r.text
assert r.json()["table_name"] == "guidance" and r.json()["notes"]["label"]["value"] == "操作指引"
assert (await client.get(_base(sid))).json()["guidance"] == "金额按元"
for body in ({}, {"guidance": 3}, {"guidance": ["金额按元"]}):
r = await client.put(f"{_base(sid)}/guidance", json=body)
assert r.status_code == 422, body
assert isinstance(r.json()["detail"], str) and "起草要求" in r.json()["detail"], "说明是中文的一句话"
assert (await client.get(_base(sid))).json()["guidance"] == "金额按元"
async def test_guidance_of_an_uploaded_source_lives_on_its_own_row(client):
sid = await _source(origin="upload")
r = await client.put(f"{_base(sid)}/guidance", json={"guidance": "按期上传的客流日报"})
assert r.status_code == 200, r.text
async with SessionLocal() as session:
row = await session.get(DataSource, sid)
assert row.options[CATALOG_DRAFT_GUIDANCE_OPTION] == "按期上传的客流日报"
async def test_the_datasource_form_neither_shows_nor_drops_the_guidance(client):
sid = await _source(options={"schema": "main"})
await client.put(f"{_base(sid)}/guidance", json={"guidance": "金额按元"})
shown = (await client.get("/api/datasources")).json()
mine = next(s for s in shown if s["id"] == sid)
assert CATALOG_DRAFT_GUIDANCE_OPTION not in mine["options"] and mine["options"]["schema"] == "main"
# 表单整份回写 options（不带起草要求，或者带着一份别的）：起草要求留着，只认 PUT /catalog/guidance
r = await client.patch(f"/api/datasources/{sid}", json={"options": {"schema": "other",
CATALOG_DRAFT_GUIDANCE_OPTION: "篡改"}})
assert r.status_code == 200, r.text
assert CATALOG_DRAFT_GUIDANCE_OPTION not in r.json()["options"]
assert (await client.get(_base(sid))).json()["guidance"] == "金额按元"
async with SessionLocal() as session:
assert (await session.get(DataSource, sid)).options == {"schema": "other",
CATALOG_DRAFT_GUIDANCE_OPTION: "金额按元"}
def test_the_guidance_never_reaches_the_driver():
src = SimpleNamespace(kind="mysql", host="db.example", port=3306, database="shop", username="reader",
password=None, options={"charset": "utf8mb4", CATALOG_DRAFT_GUIDANCE_OPTION: "金额按元"})
url = build_url(src)
assert "charset=utf8mb4" in url and CATALOG_DRAFT_GUIDANCE_OPTION not in url
async def test_the_sync_draft_uses_the_source_guidance(client, monkeypatch):
sid = await _source(description="虚构的景区票务库")
model = FakeModel(REPLIES)
use(monkeypatch, model)
await client.put(f"{_base(sid)}/guidance", json={"guidance": "渠道表的名称是对外展示名"})
r = await client.post(f"{_base(sid)}/draft", json={"tables": ["channels"], "use_model": True})
assert r.status_code == 200, r.text
assert "数据源说明：虚构的景区票务库\n起草要求：渠道表的名称是对外展示名" in model.calls[0]
assert set(r.json()) == {"tables", "total", "model_used", "model", "model_error"}, "旧的同步接口返回形状不变"
async def test_jobs_go_away_with_their_source(client):
sid = await _source()
job = (await client.post(f"{_base(sid)}/draft-jobs", json={"tables": ["gates"]})).json()["job"]
await finished(job["id"])
await drop_source(sid)
_CREATED.remove(sid)
async with SessionLocal() as session:
assert (await session.execute(select(CatalogDraftJob).where(CatalogDraftJob.id == job["id"]))).first() is None
assert (await session.execute(select(CatalogDraftTask).where(CatalogDraftTask.job_id == job["id"]))).first() \
is None
backend/tests/test_catalog_draft_settings.py
+270
View File
@@ -0,0 +1,270 @@
"""起草模型（设置里的 catalog_draft 一组）与起草取模型的顺序。
- 设置组 catalog_draft：{provider, model}，经 PUT /api/settings 写、GET 读，没存过补 {provider: null, model: null}；
provider 不是字符串、点名的接入不存在或已停用 → 422，说法写设置页上的叫法「起草模型 · 接入」「起草模型 · 模型」
- 取模型的顺序（数据目录起草和按配方导入的 AI 起草共用 draft_model.draft_model_choice）：起草模型填了接入或模型就整组
用它；否则沿用助手的模型设置；再没有就是默认接入。不跨级拼接入和模型
- 数据目录起草不关思考（按接入默认）、额度 32768，服务方嫌大退回 8192；配方 AI 起草只改取模型的顺序，额度、思考不动
接入名、模型名都是虚构的。
"""
from __future__ import annotations
from typing import Any
import pytest
from httpx import ASGITransport, AsyncClient
from sqlalchemy import delete
from app.data import catalog, draft_model
from app.data import recipe_ai as A
from app.db.base import SessionLocal
from app.db.models import Provider, Setting
from app.main import app
LIVE, OFF, GONE = "起草用接入", "起草用停用接入", "起草用不存在的接入"
async def _clean() -> None:
async with SessionLocal() as session:
await session.execute(delete(Setting).where(Setting.key.in_([draft_model.CATALOG_DRAFT_SETTING_KEY])))
await session.execute(delete(Provider).where(Provider.name.in_([LIVE, OFF])))
await session.commit()
@pytest.fixture
async def providers():
"""一个启用、一个停用的演示类接入（不联网）；用完删掉。"""
await _clean()
async with SessionLocal() as session:
session.add_all([
Provider(name=LIVE, kind="mock", models=[{"id": "起草模型甲"}], default_model="起草模型甲", enabled=True,
extra={}),
Provider(name=OFF, kind="mock", models=[{"id": "起草模型乙"}], default_model="起草模型乙", enabled=False,
extra={}),
])
await session.commit()
yield
await _clean()
@pytest.fixture
async def client():
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as c:
yield c
# --------------------------------------------------------------------------
# 设置组的读写与校验
# --------------------------------------------------------------------------
async def test_the_group_is_exposed_with_empty_defaults(client, providers):
got = (await client.get("/api/settings")).json()
assert got["catalog_draft"] == {"provider": None, "model": None}
async def test_the_group_round_trips(client, providers):
r = await client.put("/api/settings", json={"values": {"catalog_draft": {"provider": LIVE, "model": "起草模型甲"}}})
assert r.status_code == 200, r.text
assert r.json()["catalog_draft"] == {"provider": LIVE, "model": "起草模型甲"}
got = (await client.get("/api/settings")).json()["catalog_draft"]
assert got == {"provider": LIVE, "model": "起草模型甲"}
# 只写模型、接入留空：照存，接入是 null
r = await client.put("/api/settings", json={"values": {"catalog_draft": {"provider": " ", "model": "x"}}})
assert r.status_code == 200, r.text
assert r.json()["catalog_draft"] == {"provider": None, "model": "x"}
@pytest.mark.parametrize("value,label", [
({"provider": 3}, "起草模型 · 接入"),
({"model": ["a"]}, "起草模型 · 模型"),
({"temperature": 0}, "temperature"),
("起草模型甲", "起草模型"),
])
async def test_bad_values_are_refused_with_the_settings_page_wording(client, providers, value, label):
r = await client.put("/api/settings", json={"values": {"catalog_draft": value, "ui": {"theme": "dark"}}})
assert r.status_code == 422, r.text
assert label in r.json()["detail"]
got = (await client.get("/api/settings")).json()
assert got["catalog_draft"] == {"provider": None, "model": None} and got["ui"]["theme"] != "dark"
@pytest.mark.parametrize("name,reason,fix", [
(GONE, f"模型接入「{GONE}」不存在", "请在「设置 → 模型接入」中核对名称"),
(OFF, f"模型接入「{OFF}」已停用", "请在「设置 → 模型接入」中启用它"),
])
async def test_a_missing_or_disabled_provider_is_refused_with_the_field(client, providers, name, reason, fix):
r = await client.put("/api/settings", json={"values": {"catalog_draft": {"provider": name}}})
assert r.status_code == 422, r.text
body = r.json()
assert body["code"] == "settings_provider_invalid" and body["field"] == "catalog_draft.provider"
assert body["detail"] == f"「起草模型 · 接入」指定的{reason}。{fix}，或换一个接入"
async with SessionLocal() as session:
assert await session.get(Setting, "catalog_draft") is None
# --------------------------------------------------------------------------
# 取模型的顺序
# --------------------------------------------------------------------------
class _MemDB:
"""独立的内存库：取模型只看这里的接入和设置，不受别的测试建的接入影响。"""
async def __aenter__(self):
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
from sqlalchemy.pool import StaticPool
import app.db.models # noqa: F401
from app.db.base import Base
self.engine = create_async_engine("sqlite+aiosqlite:///:memory:", poolclass=StaticPool)
async with self.engine.begin() as conn:
await conn.run_sync(Base.metadata.create_all)
self.session = async_sessionmaker(self.engine, expire_on_commit=False, class_=AsyncSession)()
return self.session
async def __aexit__(self, *exc):
await self.session.close()
await self.engine.dispose()
@pytest.fixture
async def mem():
async with _MemDB() as session:
yield session
@pytest.fixture
def no_env_keys(monkeypatch):
for k in ("ANTHROPIC_API_KEY", "ANTHROPIC_AUTH_TOKEN", "OPENAI_API_KEY", "ANTHROPIC_BASE_URL"):
monkeypatch.delenv(k, raising=False)
async def _setup(session, *, draft: dict | None = None, copilot: dict | None = None) -> None:
from app.core.crypto import encrypt
for name, model in (("甲接入", "模型甲"), ("乙接入", "模型乙"), ("丙接入", "模型丙")):
session.add(Provider(name=name, kind="anthropic", api_key=encrypt("sk-test"), models=[{"id": model}],
default_model=model, enabled=True, extra={}))
if draft is not None:
session.add(Setting(key=draft_model.CATALOG_DRAFT_SETTING_KEY, value=draft))
if copilot is not None:
session.add(Setting(key=draft_model.COPILOT_SETTING_KEY, value=copilot))
await session.commit()
@pytest.mark.parametrize("draft,copilot,expected", [
(None, None, (None, None, "default")),
(None, {"provider": "乙接入", "model": ""}, ("乙接入", None, "assistant")),
({"provider": None, "model": None}, {"provider": "", "model": "模型乙"}, (None, "模型乙", "assistant")),
({"provider": "", "model": "模型丙"}, {"provider": "乙接入", "model": "模型乙"}, (None, "模型丙", "draft")),
({"provider": "甲接入", "model": None}, {"provider": "乙接入", "model": "模型乙"}, ("甲接入", None, "draft")),
({"provider": 7}, {"provider": "乙接入"}, ("乙接入", None, "assistant")), # 存坏了的按没填
])
async def test_choice_order(mem, draft, copilot, expected):
await _setup(mem, draft=draft, copilot=copilot)
got = await draft_model.draft_model_choice(mem)
assert (got.provider, got.model, got.origin) == expected
def _spy(monkeypatch) -> list[Any]:
from app.providers import factory
specs: list[Any] = []
class Built:
def __init__(self, provider, spec):
self.provider, self.spec = provider.name, spec
def fake_build(provider, spec):
specs.append((provider.name, spec))
return Built(provider, spec)
monkeypatch.setattr(factory, "build_chat_model", fake_build)
return specs
async def test_catalog_draft_uses_the_draft_setting_keeps_thinking_and_a_big_budget(mem, no_env_keys, monkeypatch):
await _setup(mem, draft={"provider": "甲接入", "model": None}, copilot={"provider": "乙接入", "model": "模型乙"})
specs = _spy(monkeypatch)
model, model_id = await catalog.resolve_draft_model(mem)
assert model_id == "模型甲"
assert [p for p, _ in specs] == ["甲接入", "甲接入"]
assert [s.max_tokens for _, s in specs] == [catalog.MODEL_MAX_TOKENS, catalog.MODEL_FALLBACK_MAX_TOKENS]
assert catalog.MODEL_MAX_TOKENS == 32768 and catalog.MODEL_FALLBACK_MAX_TOKENS == 8192
assert all(s.thinking is None and s.temperature == 0 for _, s in specs), "不关思考，按接入默认"
assert all(s.timeout == catalog.MODEL_TIMEOUT_S == 420 for _, s in specs)
assert all(s.max_retries == 0 for _, s in specs), "重试只由起草引擎按失败原因来做，SDK 自己不再重试"
assert model.budget == 32768 and model.model_name == "模型甲"
@pytest.mark.parametrize("kind,extra", [
("openai_compatible", {}),
("anthropic", {}),
("anthropic", {"auth_style": "bearer"}),
], ids=["openai-compatible", "anthropic", "anthropic-bearer"])
def test_the_built_model_does_not_retry_on_its_own_when_asked(kind, extra):
"""ModelSpec.max_retries 一直传到 SDK 客户端（Bearer 鉴权换上的客户端也是）；不写就照 SDK 默认。"""
from app.core.crypto import encrypt
from app.providers.factory import ModelSpec, build_chat_model
provider = Provider(name="虚构接入", kind=kind, api_key=encrypt("sk-test"), base_url="https://gateway.example/v1",
models=[], default_model="model-x", extra=extra)
def client_retries(chat: Any) -> int:
client = chat.root_async_client if kind == "openai_compatible" else chat._async_client
return client.max_retries
assert client_retries(build_chat_model(provider, ModelSpec(model="model-x", max_retries=0))) == 0
assert client_retries(build_chat_model(provider, ModelSpec(model="model-x"))) == 2
async def test_catalog_draft_falls_back_to_the_assistant_model(mem, no_env_keys, monkeypatch):
await _setup(mem, copilot={"provider": "", "model": "模型乙"})
specs = _spy(monkeypatch)
_, model_id = await catalog.resolve_draft_model(mem)
assert model_id == "模型乙" and specs[0][0] == "乙接入"
async def test_recipe_ai_draft_follows_the_same_order_but_keeps_its_own_spec(mem, no_env_keys, monkeypatch):
await _setup(mem, draft={"provider": "", "model": "模型丙"}, copilot={"provider": "乙接入", "model": "模型乙"})
specs = _spy(monkeypatch)
model, model_id, avail = await A.resolve_draft_model(mem)
assert avail.available and model_id == "模型丙" and avail.provider == "丙接入"
assert [s.max_tokens for _, s in specs] == [A.AI_MAX_TOKENS, A.AI_FALLBACK_MAX_TOKENS]
assert all(s.thinking == "off" and s.temperature == 0 for _, s in specs), "配方 AI 起草的思考设置不动"
assert all(s.max_retries is None for _, s in specs), "配方 AI 起草的重试照旧（SDK 默认）"
async def test_a_broken_draft_setting_says_where_to_fix_it(mem, no_env_keys):
"""起草模型点名的接入不存在：报错说的是起草模型，不是助手的模型。沿用助手模型时照旧说助手。"""
from app.core.crypto import encrypt
mem.add(Provider(name="停用的接入", kind="anthropic", api_key=encrypt("k"), models=[{"id": "模型丁"}],
default_model="模型丁", enabled=False, extra={}))
mem.add(Setting(key=draft_model.CATALOG_DRAFT_SETTING_KEY, value={"provider": "停用的接入"}))
await mem.commit()
with pytest.raises(catalog.CatalogModelUnavailable) as e:
await catalog.resolve_draft_model(mem)
assert "起草模型" in str(e.value) and "停用的接入" in str(e.value) and "已停用" in str(e.value)
assert "助手" not in str(e.value)
_, _, avail = await A.resolve_draft_model(mem)
assert not avail.available and "起草模型" in avail.reason and "助手" not in avail.reason
await mem.execute(delete(Setting))
mem.add(Setting(key=draft_model.COPILOT_SETTING_KEY, value={"provider": "停用的接入"}))
await mem.commit()
with pytest.raises(catalog.CatalogModelUnavailable) as e:
await catalog.resolve_draft_model(mem)
assert "助手使用的" in str(e.value)
async def test_only_a_demo_provider_is_refused(mem, no_env_keys):
mem.add(Provider(name="演示", kind="mock", models=[{"id": "mock-fast"}], default_model="mock-fast", enabled=True,
extra={}))
await mem.commit()
with pytest.raises(catalog.CatalogModelUnavailable) as e:
await catalog.resolve_draft_model(mem)
assert "演示" in str(e.value)
backend/tests/test_catalog_merge.py
+49 -2
View File
@@ -87,11 +87,58 @@ def test_merge_removes_stale_items_only_for_covered_sources():
def test_merge_keeps_column_entries_tidy():
existing = {"columns": {"amount": {"unit": make_item("元", "llm")}}}
merged, stats = catalog.merge_notes(existing, {}, covered={"llm"}, at=AT)
existing = {"columns": {"amount": {"unit": make_item("元", "comment")}}}
merged, stats = catalog.merge_notes(existing, {}, covered={"comment"}, at=AT)
assert merged == {} and stats.removed == 1
COLUMNS = ["id", "amount", "visit_date"]
def test_a_redraft_keeps_model_items_the_model_left_blank_this_time():
"""模型这次对某一项没给值（留空，或者给的值没通过校验、在 _from_model 里就丢掉了）：以前模型起草的项留着，不算删除。
只有给了合格的新值才替换。"""
existing = {"label": make_item("入园记录", "llm"), "kind": make_item("fact", "llm"),
"keys": make_item(["id"], "llm"), "business_date": make_item({"column": "visit_date"}, "llm"),
"columns": {"amount": {"label": make_item("实收金额", "llm"), "unit": make_item("元", "llm")},
"visit_date": {"meaning": make_item("游客检票入园的日期", "llm")}}}
draft = {"label": make_item("入园明细", "llm"), "columns": {"amount": {"unit": make_item("元", "llm")}}}
merged, stats = catalog.merge_notes(existing, draft, covered={"llm"}, at=AT, columns=COLUMNS)
assert stats == catalog.MergeStats(added=0, updated=1, removed=0)
assert merged["label"]["value"] == "入园明细" and merged["label"]["updated_at"] == AT
for key in ("kind", "keys", "business_date"):
assert merged[key] == existing[key], key
assert merged["columns"] == existing["columns"]
# 不知道表结构里有哪些列时更不删
merged, stats = catalog.merge_notes(existing, {}, covered={"llm"}, at=AT)
assert merged == existing and stats.removed == 0
def test_model_items_of_columns_gone_from_the_schema_are_still_removed():
"""表结构里已经没有的列：这一轮完整起草过（covered 含 llm）时，模型起草的这一列的项照旧删掉；业务主键、业务日期
指着这一列的也删。没完整起草过（covered 不含 llm）的不删。"""
existing = {"keys": make_item(["id", "old_no"], "llm"), "business_date": make_item({"column": "old_day"}, "llm"),
"grain": make_item("每次入园一行", "llm"),
"columns": {"old_no": {"label": make_item("旧编号", "llm"), "meaning": make_item("已经删掉的列", "llm")},
"amount": {"label": make_item("实收金额", "llm")}}}
merged, stats = catalog.merge_notes(existing, {}, covered={"llm"}, at=AT, columns=COLUMNS)
assert stats.removed == 4
assert set(merged) == {"grain", "columns"} and set(merged["columns"]) == {"amount"}
merged, stats = catalog.merge_notes(existing, {}, covered=set(), at=AT, columns=COLUMNS)
assert merged == existing and stats.removed == 0
def test_other_sources_keep_their_covered_rule():
"""只改模型起草的「覆盖即删」：数据库注释、外键、命名推断、剖析照旧——这一轮完整算过、却没有了的删掉。"""
existing = {"label": make_item("入园记录", "comment"),
"columns": {"amount": {"label": make_item("实收金额", "comment"),
"meaning": make_item("模型写的含义", "llm")}},
"relations": [_rel(["gate_id"], "gates", ["id"], source="name")]}
merged, stats = catalog.merge_notes(existing, {}, covered={"comment", "name", "llm"}, at=AT, columns=COLUMNS)
assert stats.removed == 3
assert merged == {"columns": {"amount": {"meaning": make_item("模型写的含义", "llm")}}}
# ---------------------------------------------------------------- 单项审阅
backend/tests/test_masking_tool_boundary.py
+3 -1
View File
@@ -12,6 +12,7 @@
from __future__ import annotations
import json
import re
import pytest
from httpx import ASGITransport, AsyncClient
@@ -86,7 +87,8 @@ async def test_the_model_and_the_snapshot_see_only_masks_when_the_analysis_fails
assert _clean(json.dumps(snapshot, ensure_ascii=False))
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
got = await client.get(f"/api/artifacts/{out.queries[0]['artifact']}")
assert got.status_code == 200 and _clean(got.text) and "1200" not in got.text
# 金额原值不出现。先去掉哈希再查：工件 id、快照里记的哈希是随机的十六进制，偶尔碰巧含「1200」
assert got.status_code == 200 and _clean(got.text) and "1200" not in re.sub(r"[0-9a-f]{32,}", "", got.text)
assert out.result["value"] is None
assert out.cells[0]["reason"] == "Q1 的写法认不全，整份结果查询时已遮罩，不采信"
frontend/src/api/client.ts
+35 -6
View File
@@ -1,5 +1,5 @@
import type {
ActivateSnapshotBody, ActivateSnapshotOut, AiPreview, CatalogDetail, CatalogDraftOut, CatalogImpact, CatalogList, CatalogNotes,
ActivateSnapshotBody, ActivateSnapshotOut, AiPreview, CatalogDetail, CatalogDraftJob, CatalogImpact, CatalogList, CatalogNotes,
CatalogPatchPreview, CatalogPatchSubmit, CatalogProfileOut, CatalogReviewAction, CommitOut, CurrentRecipe, EditPreview,
EditRequest, ImportRecord,
ManifestOut, PurgeRawBody, PurgeRawOut, QuestionAnswer, Recipe, RedraftRulesOut, RemovePeriodBody, RemovePeriodOut,
@@ -31,6 +31,9 @@ export type { ApiErrorKind, RequestOptions, UploadOptions, UploadProgress } from
const isObj = (v: unknown): v is Record<string, unknown> => !!v && typeof v === 'object' && !Array.isArray(v)
/** 数据目录的接口前缀：/datasources/{id}/catalog */
const catalogBase = (sourceId: string) => `/datasources/${encodeURIComponent(sourceId)}/catalog`
/**
* 上传表格被退回、要用户先选怎么处理（422 带 decision）时，取出那份决定；别的错误返回 null。
*
@@ -409,12 +412,38 @@ export const api = {
post<CatalogDetail>(`/datasources/${encodeURIComponent(sourceId)}/catalog/${encodeURIComponent(table)}/patch`,
{ changes, if_version: ifVersion }),
/**
* 同步起草（注释、外键、命名推断；use_model 时再请助手的模型起草）。不给 tables 时按使用次数取前 20 张。
* 人工确认、驳回过的项不动。没有表结构时回 409
* 建一个起草任务（后台进行：注释、外键、命名推断；use_model 时再请起草模型起草，带上数据源的起草要求和这次的 note）。
* 不给 tables 时按使用次数取前 20 张；表结构里没有的表照样建进任务、记 skipped。人工确认、驳回过的项不动。
* 409：同一个数据源已有排队、进行中的任务（code 为 catalog_draft_busy，改去取 activeDraftJob）；起草模型用不了
* （catalog_draft_model_unavailable，不建任务）；没有表结构
*/
draft: (sourceId: string, body: { tables?: string[]; use_model: boolean }, opts?: RequestOptions) =>
request<CatalogDraftOut>(`/datasources/${encodeURIComponent(sourceId)}/catalog/draft`,
{ method: 'POST', body: JSON.stringify(body), ...opts }),
createDraftJob: (sourceId: string, body: { tables?: string[]; use_model: boolean; note?: string }) =>
post<{ job: CatalogDraftJob }>(`${catalogBase(sourceId)}/draft-jobs`, body),
/** 排队或进行中的那个起草任务；没有为 null */
activeDraftJob: (sourceId: string, opts?: RequestOptions) =>
get<{ job: CatalogDraftJob | null }>(`${catalogBase(sourceId)}/draft-jobs/active`, opts),
/** 最近建的一个起草任务（任何状态）；没有为 null */
latestDraftJob: (sourceId: string, opts?: RequestOptions) =>
get<{ job: CatalogDraftJob | null }>(`${catalogBase(sourceId)}/draft-jobs/latest`, opts),
/** 一个起草任务的当前状态；不存在或不属于这个数据源回 404 */
draftJob: (sourceId: string, jobId: string, opts?: RequestOptions) =>
get<{ job: CatalogDraftJob }>(`${catalogBase(sourceId)}/draft-jobs/${encodeURIComponent(jobId)}`, opts),
/**
* 停止起草：排队的直接停；进行中的停掉正在进行的模型调用，没开始和没写完的表记已停止，已经写进目录的不回滚。
* 已经结束的回 409（catalog_draft_finished）
*/
cancelDraftJob: (sourceId: string, jobId: string) =>
post<{ job: CatalogDraftJob }>(`${catalogBase(sourceId)}/draft-jobs/${encodeURIComponent(jobId)}/cancel`),
/**
* 只重试失败的表：新任务只含原任务里失败、部分完成、已停止的表，用模型和补充说明沿用，起草要求取数据源现在的。
* 没有可重试的表 422；有进行中的任务 409（catalog_draft_busy）；和建任务一样，起草模型用不了（原任务用模型时）
* 409 catalog_draft_model_unavailable、没有表结构 409
*/
retryDraftJob: (sourceId: string, jobId: string) =>
post<{ job: CatalogDraftJob }>(`${catalogBase(sourceId)}/draft-jobs/${encodeURIComponent(jobId)}/retry`),
/** 存数据源的起草要求（去首尾空白，最多 2000 字，空串清除）。上传表格建的源存在它自己的数据源上 */
putGuidance: (sourceId: string, guidance: string) =>
put<{ guidance: string | null }>(`${catalogBase(sourceId)}/guidance`, { guidance }),
/**
* 同步剖析：对业务库发少量只读查询，核对关系、取码值候选，结论写进目录。不给 tables 时按使用次数取前 10 张
* 有待核实关系的表。没开启、数据源停用、没有表结构、同一个数据源正在剖析、快照被改动时回 409（detail 是一句中文）
frontend/src/lib/terms.ts
+205 -19
View File
@@ -43,10 +43,11 @@
*/
import type {
CatalogCardinality, CatalogMeasure, CatalogProfileNumberKey, CatalogSource, CatalogStatus, CatalogTableKind, NodeType, SnapshotReasonCode,
CatalogCardinality, CatalogDraftJobStatus, CatalogDraftTaskStatus, CatalogMeasure, CatalogProfileNumberKey, CatalogSource, CatalogStatus,
CatalogTableKind, NodeType, SnapshotReasonCode,
SqlCheckCode, SqlCheckLevel, ToolTrust,
} from '../types'
import { formatCost, formatNumber } from './format'
import { formatCost, formatNumber, formatTokens } from './format'
export const TERMS = {
workflow: '工作流',
@@ -963,6 +964,8 @@ export const PROVIDER_REF_TEXT = {
judgeBroken: '裁判调用将失败，所有结论句记为未裁判',
// 和核查一组「不存在」那句（VERIFY_SETTING_TEXT.section.providerGone）同一个后果
verifyBroken: '核查调用将失败',
// 起草模型（数据目录的用模型起草、按配方导入的 AI 起草）：开始起草时就报错，不建任务
draftBroken: '用模型起草时将报错，无法起草',
/**
* 存着的接入名恰好是某个启用接入的模型（多半是把模型名填进了接入那一栏）：照「启用它」去做，启用的是那条旧接入。
* 和服务端保存时 422 的那一句（factory.model_note）一字不差；两栏写这一组的叫法（「门控模型 · 模型」「门控模型 · 接入」）
@@ -2198,7 +2201,7 @@ export const CATALOG_TEXT = {
noSchemaBody: (note: string) => `${note.replace(/[。.]$/, '')}。数据目录按表结构逐表记录，请先回到数据源卡片点击「探查结构」。`,
noSchemaUploadBody: '这个表格还没有可用的表结构，请重新上传后再编写数据目录。',
emptyTitle: '还没有数据目录',
emptyBody: '先起草，再逐表确认。起草按数据库注释、外键约束和列名推断，也可以请助手的模型起草中文名和含义。',
emptyBody: '先起草，再逐表确认。起草按数据库注释、外键约束和列名推断，也可以用模型起草中文名和含义。',
// ---- 导入表格的说明
systemTitle: '导入时生成的说明',
systemBody: '这个数据源由导入表格生成，表和列的说明由系统按核对结果生成，只读，不能在数据目录中改写。',
@@ -2260,29 +2263,17 @@ export const CATALOG_TEXT = {
scopeAll: (n: number) => `全部 ${formatNumber(n)} 张表`,
draftBase: '按数据库注释、外键约束和列名推断起草。已确认、已驳回的项不会被改动；起草出的项都是推断，需要逐项确认。',
useModel: '用模型起草',
useModelHint: '请助手的模型按表结构起草中文名、说明、粒度和列的含义。只发送表结构（表名、列名、类型、约束、注释），不发送数据；耗时较长，并产生模型费用。',
useModelHint: '请起草模型按表结构起草中文名、说明、粒度和列的含义。只发送表结构（表名、列名、类型、约束、注释）和起草要求，不发送数据；在后台进行，耗时较长，并产生模型费用。',
draftStart: (n: number) => `起草 ${formatNumber(n)} 张表`,
drafting: '正在起草…',
draftProgress: (done: number, total: number) => `已完成 ${formatNumber(done)} / ${formatNumber(total)} 张表`,
draftStop: '停止',
draftStopping: '当前这一批完成后停止…',
draftDone: '起草完成',
draftStopped: (done: number, total: number) => `已停止：完成 ${formatNumber(done)} / ${formatNumber(total)} 张表，其余未起草`,
draftFailed: '起草中断',
draftSummary: (added: number, updated: number, removed: number) =>
`新增 ${formatNumber(added)} 项、更新 ${formatNumber(updated)} 项${removed ? `、删除 ${formatNumber(removed)} 项` : ''}`,
// 只在起草确实给出了内容、只是和现有目录一致时用：有表的模型起草失败或没给出可用内容（model_error）时
// 不能说「没有变化」，改用 draftNothingWritten
// 只在起草确实给出了内容、只是和现有目录一致时用：模型没给出可用内容时说 draftNothingWritten，有表没写全时说
// draftNothingWrittenFailed，不能说「没有变化」
draftNoChange: '没有变化：起草出的内容与现有目录一致',
draftNothingWritten: '目录没有更新：模型没有给出可用的内容，原因见下方',
draftModelSkipped: (reason: string) => `模型未参与起草：${reason.replace(/[。.]$/, '')}。已按注释、外键和命名起草。`,
draftModelUsed: (model: string) => `模型：${model}`,
draftTableErrors: (n: number) => `${formatNumber(n)} 张表未能起草`,
draftModelErrors: (n: number) => `${formatNumber(n)} 张表的模型起草失败，已按注释、外键和命名起草`,
draftNothingWrittenFailed: '目录没有更新：部分表未写全，原因见下方',
draftClose: '完成',
moreTables: (n: number) => `…另有 ${formatNumber(n)} 张`,
cancel: '取消',
draftBackground: '起草仍在进行，当前这一批完成后停止',
// ---- 表详情
backToList: '返回表清单',
prev: '上一张',
@@ -2389,6 +2380,201 @@ export const CATALOG_TEXT = {
reloadDiscardAction: '放弃修改并重新载入',
}
// ===========================================================================
// 数据目录的起草任务（/api/datasources/{id}/catalog/draft-jobs）：起草在服务端后台进行，关掉对话框不停；逐表起草，
// 每张表一结束就写进目录。
//
// 术语：
// - 「起草任务」的状态：排队中、正在起草、起草完成、部分表未写全、起草失败、已停止、服务重启中断。
// 一张表：等待、起草中、已完成、部分完成、失败、已跳过（表结构里没有）、已停止。
// - 「起草要求」：使用者补充给模型的业务背景和用词约定，存在数据源上，以后每次用模型起草都带上；
// 「本次补充说明」只用于这一次（重试时沿用）。两者都只交给模型，不改起草规则：冲突时以规则为准。
// - 「起草模型」：设置里单独的一项（偏好设置 → 起草模型），留空跟随助手的模型。
// - 宽表的「段」：超过 40 列的表拆成表级一段和若干列分段，逐段起草；写的是「已完成 3 / 9 段」。模型多次用完额度或
// 超时后，之后的段缩小（每段最多 20 列、10 列）。
// - 「只重试失败的表」：重新起草失败、部分完成和已停止的表；已经写进目录的内容不回滚、不受影响。
// - 「没写出来」：一次调用没有写出内容（额度用完、只给空壳、超时、出错）。接连没写出来时服务端给一句「提醒」，
// 好转后收起；提醒是提醒的样式，不是错误。超时的调用不计用量。
// - 「留空」：模型列出了一列（或这一段请它写了），中文名和含义却都没给，写「N 列模型拿不准、留空了」。表的状态不因此
// 变，不算没写全；办法是在起草要求里补充缩写和术语的含义，再重新起草，已有的内容不会丢。
// ===========================================================================
/** 起草要求、本次补充说明的字数上限（服务端同一个数：超长 422） */
export const DRAFT_GUIDANCE_MAX = 2000
export const DRAFT_NOTE_MAX = 1000
export const CATALOG_DRAFT_TEXT = {
// ---- 设置阶段：起草要求、本次补充说明、起草模型
guidance: '起草要求',
guidanceHint: '写给模型的业务背景和用词约定，保存在这个数据源上，以后每次用模型起草都会带上。例如：「核销」指检票入园；金额单位是元',
guidanceRule: '起草要求只帮助模型理解业务、统一用词；与起草规则冲突时以规则为准，起草出的项仍需逐项确认',
guidancePlaceholder: '业务背景、常用说法、单位约定……',
note: '本次补充说明',
noteHint: '只用于这一次起草，例如「这一批是会员相关的表」；重试时沿用',
notePlaceholder: '可不填',
count: (n: number, max: number) => `${formatNumber(n)} / ${formatNumber(max)} 字`,
tooLong: (max: number) => `不超过 ${formatNumber(max)} 字`,
sourceDescription: (text: string) => `数据源说明也会一并交给模型：${text}`,
model: '起草模型',
/** 设置里没配起草模型：跟随助手的模型，写出当前是哪个（取不到就不写） */
modelFollow: (current: string | null) => (current ? `跟随助手的模型（当前：${current}）` : '跟随助手的模型'),
/** 设置里配了：模型（没填模型就是那个接入的默认模型）和接入 */
modelNamed: (model: string | null, provider: string | null) => (model
? (provider ? `${model}（模型接入「${provider}」）` : model)
: `模型接入「${provider}」的默认模型`),
modelWhere: '可在「设置 → 偏好设置 → 起草模型」中更换',
tooMany: (max: number) => `一次最多起草 ${formatNumber(max)} 张表，请缩小范围`,
saveGuidanceFailed: (reason: string) => `起草要求未能保存，未开始起草：${reason}`,
startFailed: (reason: string) => `未能开始起草：${reason}`,
// ---- 上一次起草（设置阶段，没写全、有留空列的才提）
last: (state: string, at: string) => `上一次起草（${at}）：${state}`,
/** 上一次的结果：状态，有留空的列时接着写几列 */
lastState: (title: string, blank: number) => (blank ? `${title}，${formatNumber(blank)} 列模型拿不准、留空了` : title),
lastOpen: '查看',
// ---- 从结果、表详情回到设置：范围就是那几张表
scopeBlank: (n: number) => `有留空列的 ${formatNumber(n)} 张表`,
scopeTable: (name: string) => `这张表：${name}`,
/**
* 预先选好的范围里有表已经不在表结构里（打开表详情以后别处重新探查了结构）：写在那一项下面。一张都不剩时那一项
* 照旧选着、不能开始，不退回别的范围
*/
presetMissing: (gone: number, total: number) => (gone < total
? `其中 ${formatNumber(gone)} 张在表结构中已没有，不会起草`
: total === 1 ? '表结构中已没有这张表，不能起草' : `表结构中已没有这 ${formatNumber(total)} 张表，不能起草`),
// ---- 进行中
queued: '排队中，等待开始…',
running: '正在起草…',
// 服务端点了停止就取消进行中的模型调用（不等它答完，一次调用可能要几分钟），只等手上正在写入目录的那张表写完
stopping: '正在停止：取消进行中的模型调用，正在写入目录的表写完后停止…',
progress: (done: number, total: number) => `已完成 ${formatNumber(done)} / ${formatNumber(total)} 张表`,
/** 结束以后：全部写出来的表有几张（停止、失败、部分完成的不算） */
progressDone: (ok: number, total: number) => `写完 ${formatNumber(ok)} / ${formatNumber(total)} 张表`,
parts: (done: number, total: number) => `${formatNumber(done)} / ${formatNumber(total)} 段`,
partsHint: '超过 40 列的表拆成表级一段和若干列分段，逐段起草',
hide: '在后台继续',
hideHint: '关闭对话框不会停止起草，进度显示在目录页上方；每张表起草完就写进目录',
stop: '停止',
busy: '这个数据源已有进行中的起草任务，下面显示的是它的进度',
modelUsed: (model: string) => `模型：${model}`,
usage: (calls: number, input: number, output: number) =>
`调用模型 ${formatNumber(calls)} 次，输入 ${formatTokens(input, { compact: true })}、输出 ${formatTokens(output, { compact: true })}`,
/**
* 用量那一行接着写的：没写出来、超时的调用各几次（没有的不写）。没写出来的次数里含超时的，写成「其中」，免得
* 读成两个数相加
*/
usageFailures: (timeouts: number, failed: number) => {
if (failed && timeouts && timeouts <= failed) return `没写出来 ${formatNumber(failed)} 次（其中超时 ${formatNumber(timeouts)} 次）`
return [failed ? `没写出来 ${formatNumber(failed)} 次` : '', timeouts ? `超时 ${formatNumber(timeouts)} 次` : ''].filter(Boolean).join('、')
},
usageFailuresHint: '没写出来：调用没有写出内容，包括额度用完、只给空壳、超时和出错',
// 超时的调用服务端照实记 0 个 token，服务方却可能已经生成了内容
timeoutNote: '超时的调用不计用量，实际用量可能更多',
/** 额度用完、超时多了，服务端把之后的段缩小了 */
partColumns: (n: number) => `模型多次用完额度或超时，之后的段每段最多 ${formatNumber(n)} 列`,
// ---- 留空的列：模型列出了、中文名和含义却都没给。表的状态不因此变
blank: (n: number) => `${formatNumber(n)} 列模型拿不准、留空了`,
/** 留空的列名（服务端最多给 5 个）；没列全时写一共几列 */
blankSample: (sample: string[], n: number) =>
`留空的列：${sample.join('、')}${n > sample.length ? ` 等 ${formatNumber(n)} 列` : ''}`,
blankTotal: (columns: number, tables: number) => `共 ${formatNumber(columns)} 列模型拿不准、留空了（${formatNumber(tables)} 张表）`,
blankAdvice: '可以在起草要求里补充缩写和术语的含义，再重新起草这几张表，已有的内容不会丢',
blankRedraft: (n: number) => `重新起草这 ${formatNumber(n)} 张表`,
blankRedraftHint: '回到起草设置，范围是这几张表：先在起草要求里补充缩写和术语的含义，再开始起草',
// ---- 结束
title: {
queued: '排队中，等待开始…',
running: '正在起草…',
succeeded: '起草完成',
partial: '起草完成，部分表未写全',
failed: '起草失败',
cancelled: '已停止',
interrupted: '服务重启，起草中断',
} as Record<CatalogDraftJobStatus, string>,
interrupted: '服务重启时这次起草还没有完成，没写完的表未起草。已写进目录的内容保留；可以只重试这些表，接着起草',
cancelled: '已写进目录的内容保留，没写完的表未起草',
// 任务整体出错的原因（job.error）照服务端的原话写在标题「起草失败」下面，不再加前缀：服务端的原因有的已经是整句
// （「数据源不存在……」），有的自带「起草任务出错：」开头。没结束的表记失败、原因同任务的，逐表不再重复
/** 没写全的表：失败、部分完成、已停止各几张；跳过的（表结构里没有）另说 */
problems: (c: { failed: number; partial: number; cancelled: number; skipped: number }) => {
const parts = [
c.failed ? `失败 ${formatNumber(c.failed)} 张` : '',
c.partial ? `部分完成 ${formatNumber(c.partial)} 张` : '',
c.cancelled ? `已停止 ${formatNumber(c.cancelled)} 张` : '',
].filter(Boolean)
const bad = parts.length ? `${formatNumber(c.failed + c.partial + c.cancelled)} 张表未写全：${parts.join('、')}` : ''
const skipped = c.skipped ? `${formatNumber(c.skipped)} 张表已跳过（表结构里没有）` : ''
return [bad, skipped].filter(Boolean).join('；')
},
retry: (n: number) => `只重试失败的表（${formatNumber(n)} 张）`,
retryHint: '重新起草失败、部分完成和已停止的表；已写进目录的内容不受影响',
showAll: (n: number) => `显示全部 ${formatNumber(n)} 张表`,
showProblems: '只看未写全的表',
showProblemsBlank: '只看未写全、有留空列的表',
tasks: '逐表进度',
taskStatus: {
pending: '等待', running: '起草中', ok: '已完成', partial: '部分完成', failed: '失败', skipped: '已跳过', cancelled: '已停止',
} as Record<CatalogDraftTaskStatus, string>,
taskAdded: (added: number, updated: number) => (added || updated
? `+${formatNumber(added)}${updated ? ` · 改 ${formatNumber(updated)}` : ''}` : ''),
taskAddedHint: (added: number, updated: number, removed: number) =>
`新增 ${formatNumber(added)} 项、更新 ${formatNumber(updated)} 项${removed ? `、删除 ${formatNumber(removed)} 项` : ''}`,
// ---- 目录页上方的进度、结束提示
strip: (done: number, total: number) => `正在起草：已完成 ${formatNumber(done)} / ${formatNumber(total)} 张表`,
stripQueued: (total: number) => `起草任务排队中，共 ${formatNumber(total)} 张表`,
stripOpen: '查看进度',
toast: {
succeeded: (summary: string) => `数据目录起草完成：${summary}`,
partial: (n: number) => `数据目录起草完成，${formatNumber(n)} 张表未写全`,
/** 完成、部分完成的提示后面接着写留空的列（没有就不写） */
withBlank: (text: string, blank: number) => (blank ? `${text}；${formatNumber(blank)} 列模型拿不准、留空了` : text),
failed: (reason: string) => (reason ? `数据目录起草失败：${reason}` : '数据目录起草失败'),
cancelled: '数据目录起草已停止',
interrupted: '服务重启，数据目录起草中断',
},
viewResult: '查看结果',
// ---- 清单上的标记、表详情里的说明
lastFailed: '上次起草失败',
lastPartial: '上次起草部分完成',
lastWith: (label: string, reason: string) => `${label}：${reason}`,
/** 上次起草有留空的列：清单上是轻提示（不是失败标记），表详情顶部同一句 */
lastBlank: (n: number) => `上次起草有 ${formatNumber(n)} 列模型拿不准、留空了`,
lastBlankAdvice: '可以在起草要求里补充缩写和术语的含义，再重新起草这张表，已有的内容不会丢',
draftGuidance: '补充起草要求',
draftGuidanceHint: '打开起草设置，范围是这张表：先在起草要求里补充缩写和术语的含义，再开始起草',
redraft: '重新起草这张表',
redraftHint: '用模型重新起草这张表（不带补充说明，带上数据源的起草要求）；人工确认、驳回过的项不会被改动',
redraftBusy: '已有起草任务在进行，结束后再重新起草这张表',
redraftEditing: '正在编辑这张表，保存或取消后再重新起草',
/** 表结构里已经没有的表（只剩目录）：上次起草的结果照写，不给重新起草（服务端只会把它记成已跳过） */
redraftMissing: '表结构中已没有这张表，不能重新起草',
}
/** 设置页「偏好设置」里「起草模型」一节（服务端 api/settings.py 的 catalog_draft 一组） */
export const DRAFT_MODEL_SETTING_TEXT = {
title: '起草模型',
hint: '数据目录的「用模型起草」和按配方导入表格时的 AI 起草使用这个模型，留空则跟随助手的模型。'
+ '起草一次要写出几十到上百列的说明，建议选择输出额度大、能稳定按格式输出的模型；会思考的模型，思考也计入输出额度。',
/** 和服务端 _provider_fields 里的叫法一字不差：保存被拒时的报错写的是它 */
provider: '起草模型 · 接入',
model: '起草模型 · 模型',
notSet: '未单独配置，跟随助手的模型',
providerDefault: '跟随助手的模型',
providerFromModel: '按模型名找接入',
providerOff: '（已停用）',
providerGoneOption: (name: string) => `${name}（不存在）`,
providerGone: (name: string) => `模型接入「${name}」不存在：${PROVIDER_REF_TEXT.draftBroken}`,
modelDefault: '留空：用接入的默认模型',
modelFollow: '留空：跟随助手的模型',
modelOf: (model: string) => `留空则使用 ${model}`,
/** 只填了起草模型、接入留空时实际用哪个接入（同核查一组的说法，只是上一级是助手的模型） */
byModel: (model: string, found: { name: string; enabled: boolean } | null | undefined, fallback: string | null | undefined) => {
if (found && !found.enabled) return `只填了模型：「${model}」属于已停用的接入「${found.name}」，${PROVIDER_REF_TEXT.draftBroken}。请启用该接入或选择其他接入`
if (found) return `只填了模型：将按模型名使用接入「${found.name}」。「跟随助手的模型」仅在接入和模型都留空时生效`
return `只填了模型：所有接入的模型列表中都没有「${model}」，将使用默认接入${fallback ? `「${fallback}」` : ''}调用。`
+ '如该模型不属于默认接入，请选择对应的接入。「跟随助手的模型」仅在接入和模型都留空时生效'
},
}
// ===========================================================================
// 目录修改提案：用户在助手（画布）或问数据页说出一条数据事实时，助手提出的数据目录修改。
//
frontend/src/pages/DraftModelSettings.tsx
+84
View File
@@ -0,0 +1,84 @@
import { modelOptions, providerOfModel, useCatalog } from '../store/catalog'
import { Field, SectionBar } from '../components/ui'
import { DRAFT_MODEL_SETTING_TEXT as T, PROVIDER_REF_TEXT } from '../lib/terms'
import { ProviderOptions } from './VerifySettings'
/*
* 设置页「偏好设置」里「起草模型」一节：数据目录的「用模型起草」和按配方导入表格时的 AI 起草用哪个模型
* （服务端 api/settings.py 的 catalog_draft 一组：{provider, model}）。
*
* 取模型的顺序（服务端）：这一组填了接入或模型就用它（只填模型时按模型名找接入，不跨级拼）；两项都留空跟随助手的
* 模型；再没有就是默认接入。草稿由 PrefsTab 持有、和其余设置共用一条保存条，这里只画；两栏的叫法和服务端
* _provider_fields 一字不差，保存被拒（422 带 field：catalog_draft.provider）时报错写的就是它
*/
export function DraftModelSection({ provider, model, error, onEdit }: {
provider: string
model: string
/** 服务端按这一项拒收时的整句（设置页从 422 的 field 落过来） */
error?: string
onEdit: (patch: { draftProvider?: string; draftModel?: string }) => void
}) {
const providers = useCatalog((s) => s.providers)
const enabledProviders = providers.filter((p) => p.enabled)
// 只填了模型、按模型名找不到接入时服务端用默认接入（第一个启用的真实接入，没有就第一个启用的）
const defaultProvider = enabledProviders.find((p) => p.kind !== 'mock') ?? enabledProviders[0]
const found = provider ? providers.find((p) => p.name === provider) : undefined
const missing = !!provider && providers.length > 0 && !found
// 存着的接入不存在、已停用：开始起草时就会报错（不建任务），在这一格下面说。名字恰好是某个启用接入的模型时
// 补一句怎么用它（同设置页别的几组，和服务端保存时那句一致）
const trap = found?.enabled === false || missing ? providerOfModel(providers, provider) : undefined
const trapNote = trap?.enabled ? `。${PROVIDER_REF_TEXT.modelNote(provider, trap.provider.name, T.model, T.provider)}` : ''
const providerError = error ?? (missing ? `${T.providerGone(provider)}${trapNote}`
: found && !found.enabled ? `${PROVIDER_REF_TEXT.disabled(provider)}：${PROVIDER_REF_TEXT.draftBroken}${trapNote}` : undefined)
const modelName = model.trim()
const byModel = !provider && !!modelName
const notSet = !provider && !modelName
const owner = byModel ? providerOfModel(providers, modelName) : undefined
const byModelHint = byModel && providers.length > 0
? T.byModel(modelName, owner && { name: owner.provider.name, enabled: owner.enabled }, defaultProvider?.name) : ''
// 接入留空时按模型名找接入：所有启用接入的模型都能选；选了接入只列这一家的
const models = found ? (found.models ?? []).map((m) => m.id) : [...new Set(modelOptions(providers).map((o) => o.value))]
const pickProvider = (name: string) => {
// 换了接入，原来填的模型不在新接入的清单里就清掉：拿 A 家的模型名去调 B 家必然失败
const next = name ? providers.find((p) => p.name === name) : undefined
const keep = !name || !model || (next?.models ?? []).some((m) => m.id === model) || next?.default_model === model
onEdit({ draftProvider: name, ...(keep ? {} : { draftModel: '' }) })
}
return (
<section data-draft-model-settings="">
<SectionBar title={T.title} hint={T.hint} />
<div role="group" aria-labelledby="pref-draft-model-label" aria-describedby={notSet ? 'pref-draft-model-hint' : undefined}
data-draft-model-group={notSet ? 'follow' : 'set'}>
<div id="pref-draft-model-label" className="sr-only">{T.title}</div>
{notSet && <p id="pref-draft-model-hint" className="mb-2 text-2xs leading-relaxed text-faint">{T.notSet}</p>}
<div className="grid grid-cols-2 gap-3 max-sm:grid-cols-1">
<Field label={T.provider} htmlFor="pref-draft-provider" error={providerError}
hint={byModelHint ? (
<span data-draft-by-model={owner ? (owner.enabled ? 'found' : 'disabled') : 'default'}
style={owner?.enabled ? undefined : { color: 'var(--st-waiting)' }}>{byModelHint}</span>
) : undefined}>
{(p) => (
<select {...p} className="field" value={provider} onChange={(e) => pickProvider(e.target.value)}>
<option value="">{byModel ? T.providerFromModel : T.providerDefault}</option>
{missing && <option value={provider}>{T.providerGoneOption(provider)}</option>}
<ProviderOptions providers={providers} current={provider} />
</select>
)}
</Field>
<Field label={T.model} htmlFor="pref-draft-model"
hint={found?.enabled && found.default_model ? T.modelOf(found.default_model) : undefined}>
{(p) => (
<>
<input {...p} className="field mono text-xs" value={model} list="pref-draft-models" spellCheck={false}
placeholder={provider ? T.modelDefault : T.modelFollow} onChange={(e) => onEdit({ draftModel: e.target.value })} />
<datalist id="pref-draft-models">{models.map((m) => <option key={m} value={m} />)}</datalist>
</>
)}
</Field>
</div>
</div>
</section>
)
}
frontend/src/pages/SettingsPage.tsx
+27 -4
View File
@@ -23,12 +23,14 @@ import { normalizeTheme } from '../lib/theme'
import { useLeaveGuard } from '../lib/leave'
import { localActor, setLocalActor } from '../lib/actor'
import {
AGENT_GUARD_TEXT, JUDGE_SETTING_TEXT, PROVIDER_REF_TEXT, TOOL_GATE_TEXT, VERIFY_SETTING_TEXT, judgeByModelText, judgeUnlimitedText,
AGENT_GUARD_TEXT, DRAFT_MODEL_SETTING_TEXT, JUDGE_SETTING_TEXT, PROVIDER_REF_TEXT, TOOL_GATE_TEXT, VERIFY_SETTING_TEXT, judgeByModelText,
judgeUnlimitedText,
} from '../lib/terms'
import type { ThemePref } from '../lib/theme'
import {
ProviderOptions, VerifySection, verifyChanged, verifyDraftOf, verifyErrors, verifyLabels, verifyPayload, type VerifyDraft,
} from './VerifySettings'
import { DraftModelSection } from './DraftModelSettings'
// 提到模块级：tab 名同时是 URL 的最后一段，两处各写一份迟早对不上。
// 数据源搬去了顶级的「数据」页（/data），/settings/datasources 跳过去
@@ -626,6 +628,8 @@ function PrefsTab() {
judgeProvider: string; judgeModel: string
// 证据裁判的五项上限：null 是不限（存 null），字符串是正在填的数
jClaims: string | null; jCost: string | null; jTimeout: string | null; jClick: string | null; jDaily: string | null
// 起草模型（数据目录的用模型起草、按配方导入的 AI 起草）：空串 = 跟随助手的模型（存 null）
draftProvider: string; draftModel: string
} | null>(null)
const [spend, setSpend] = useState<{ usd?: number; calls?: number; unpriced_calls?: number } | null>(null)
// 核查一组（双盲复算、反方审稿、审批质询）的草稿另放一份，和上面的共用保存条；verifySaves 是存过几次，今日核查花费跟着再取
@@ -673,6 +677,9 @@ function PrefsTab() {
jTimeout: limitText(values?.judge, 'report_timeout_s'),
jClick: limitText(values?.judge, 'click_max_cost_usd'),
jDaily: limitText(values?.judge, 'daily_max_usd'),
// 起草模型（服务端 catalog_draft 一组；老服务端没有这一组时两项都是空，这一节也不显示）
draftProvider: typeof values?.catalog_draft?.provider === 'string' ? values.catalog_draft.provider : '',
draftModel: typeof values?.catalog_draft?.model === 'string' ? values.catalog_draft.model : '',
}
const cur = draft ?? saved
const stepCap = Number(values?.limits?.max_agent_steps) || 100
@@ -748,7 +755,8 @@ function PrefsTab() {
savedActor.current = actor
}
const judgeChanged = changed.some((k) => JUDGE_DRAFT.has(k))
const runChanged = changed.some((k) => k !== 'actor' && !JUDGE_DRAFT.has(k))
const draftChanged = changed.some((k) => DRAFT_MODEL_DRAFT.has(k))
const runChanged = changed.some((k) => k !== 'actor' && !JUDGE_DRAFT.has(k) && !DRAFT_MODEL_DRAFT.has(k))
try {
// 裁判一组整组写：后端对没给的键回落到默认值，只发改过的那一项会把别的项冲回默认
const judge = judgeChanged ? {
@@ -762,7 +770,9 @@ function PrefsTab() {
} : null
// 核查一组同样整组写（VerifySettings 的 verifyPayload）
const verify = verifyKeys.length ? verifyPayload(verifyCur) : null
if (runChanged || judge || verify) {
// 起草模型一组：两项一起写，空着的存 null（跟随助手的模型）
const catalogDraft = draftChanged ? { provider: cur.draftProvider || null, model: cur.draftModel.trim() || null } : null
if (runChanged || judge || verify || catalogDraft) {
const run = {
...(values.run ?? {}),
default_memory_scope: cur.scope,
@@ -774,12 +784,16 @@ function PrefsTab() {
agent_budget_tokens: cur.budgetTokens.trim() === '' ? null : Number(cur.budgetTokens),
agent_budget_usd: cur.budgetUsd.trim() === '' ? null : Number(cur.budgetUsd),
}
const out = await api.settings.put({ ...(runChanged ? { run } : {}), ...(judge ? { judge } : {}), ...(verify ? { verify } : {}) })
const out = await api.settings.put({
...(runChanged ? { run } : {}), ...(judge ? { judge } : {}), ...(verify ? { verify } : {}),
...(catalogDraft ? { catalog_draft: catalogDraft } : {}),
})
setValues((v) => ({
...(v ?? {}),
...(runChanged ? { run: out?.run ?? run } : {}),
...(judge ? { judge: out?.judge ?? judge } : {}),
...(verify ? { verify: out?.verify ?? verify } : {}),
...(catalogDraft ? { catalog_draft: out?.catalog_draft ?? catalogDraft } : {}),
}))
// 每日上限改了：旁边那句「今天已花」的分母跟着变，顺手再取一次
if (judge) void loadSpend()
@@ -1090,6 +1104,11 @@ function PrefsTab() {
{values.verify && <VerifySection cur={verifyCur} onEdit={editVerify} refresh={verifySaves}
errors={fieldError?.field === 'verify.provider' ? { ...verifyErrs, provider: fieldError.message } : verifyErrs} />}
{values.catalog_draft && (
<DraftModelSection provider={cur.draftProvider} model={cur.draftModel} onEdit={edit}
error={fieldError?.field === 'catalog_draft.provider' ? fieldError.message : undefined} />
)}
</>
)}
@@ -1113,6 +1132,7 @@ function PrefsTab() {
judgeProvider: '裁判模型的接入', judgeModel: '裁判模型',
jClaims: '每份报告最多裁判句数', jCost: '每份报告的金额上限', jTimeout: '每份报告的时长上限',
jClick: '每次点击的金额上限', jDaily: '每日金额上限',
draftProvider: DRAFT_MODEL_SETTING_TEXT.provider, draftModel: DRAFT_MODEL_SETTING_TEXT.model,
})[k]).concat(verifyLabels(verifyKeys)).join('、')}
</span>
</span>
@@ -1147,10 +1167,13 @@ function PrefsTab() {
/** 服务端按项拒收时（422 带 field）那一项在页面上是哪个控件：点名的模型接入的三格 */
const PROVIDER_FIELD_IDS: Record<string, string> = {
'run.tool_gate_provider': 'pref-gate-provider', 'judge.provider': 'pref-judge-provider', 'verify.provider': 'pref-verify-provider',
'catalog_draft.provider': 'pref-draft-provider',
}
/** 裁判那一组在草稿里的键：它们改了才写 judge 一组 */
const JUDGE_DRAFT = new Set<string>(['judgeProvider', 'judgeModel', 'jClaims', 'jCost', 'jTimeout', 'jClick', 'jDaily'])
/** 起草模型那一组在草稿里的键：它们改了才写 catalog_draft 一组 */
const DRAFT_MODEL_DRAFT = new Set<string>(['draftProvider', 'draftModel'])
/** 勾掉「不限」时填回的数：同后端 JUDGE_DEFAULTS */
const JUDGE_FALLBACK = {
frontend/src/pages/catalog/CatalogPage.tsx
+127 -15
View File
@@ -1,15 +1,19 @@
import { useCallback, useEffect, useMemo, useRef, useState } from 'react'
import type { ReactNode } from 'react'
import { useNavigate, useParams, useSearchParams } from 'react-router-dom'
import { ArrowLeft, BookMarked, Database, ListFilter as ListFilterIcon, ScanSearch, Sparkles } from 'lucide-react'
import { ArrowLeft, BookMarked, Database, ListFilter as ListFilterIcon, ScanSearch, Sparkles, TriangleAlert } from 'lucide-react'
import clsx from 'clsx'
import { ApiError, api } from '../../api/client'
import type { CatalogDetail, CatalogList, CatalogTableRow } from '../../types'
import { EmptyState, ErrorState, PageHeader, Skeleton, Spinner, toast, useTicker } from '../../components/ui'
import { useCatalog, useDatasources, useOnReconnect } from '../../store/catalog'
import { CATALOG_TEXT as CT, PROFILE_STOP_LABEL, PROFILE_TEXT as PT, SCHEMA_PARTIAL_TEXT as SPT } from '../../lib/terms'
import {
CATALOG_DRAFT_TEXT as DT, CATALOG_TEXT as CT, PROFILE_STOP_LABEL, PROFILE_TEXT as PT, SCHEMA_PARTIAL_TEXT as SPT,
} from '../../lib/terms'
import { StatusLegend, SystemNotesNotice } from './parts'
import { DraftDialog } from './DraftDialog'
import { isJobActive, jobBlank, retryableCount, useDraftJob } from './draftJob'
import type { DraftPreset, DraftStartBody } from './draftJob'
import { ProfileDialog } from './ProfileDialog'
import type { ProfileJob } from './ProfileDialog'
import { ProfileSettingsDialog } from './ProfileSettings'
@@ -99,7 +103,11 @@ export function CatalogPage() {
const search = writeListParams(params, list).toString()
searchRef.current = search ? `?${search}` : ''
const [selected, setSelected] = useState<Set<string>>(() => new Set())
const [drafting, setDrafting] = useState(false)
/**
* 起草对话框开着没有、打开时显示什么：setup 选范围开始；job 显示页面跟着的任务（进行中的进度、刚结束的结果）。
* busy：是撞上已有任务（409）转过去显示的；preset：设置阶段预先选好的范围（表详情里「补充起草要求」）
*/
const [draftOpen, setDraftOpen] = useState<{ view: 'setup' | 'job'; busy?: boolean; preset?: DraftPreset } | null>(null)
/** 起草写进了这些表：详情据此重新载入（编辑中的不动，保存时服务端会用版本号拦下） */
const [drafted, setDrafted] = useState<{ seq: number; tables: Set<string> }>({ seq: 0, tables: new Set() })
const [profiling, setProfiling] = useState(false)
@@ -142,11 +150,67 @@ export function CatalogPage() {
}))
}, [])
const onDrafted = useCallback((tables: string[]) => {
setDrafted((cur) => ({ seq: cur.seq + 1, tables: new Set(tables) }))
setSelected(new Set())
void load()
}, [load])
/**
* 起草任务（后台进行，状态在页面上）：每张表一结束就重取清单、正在看的表在里面就重新载入（编辑中的不动，保存时服务端
* 会用版本号拦下）；结束时重取一次清单。对话框关着时结束的，弹提示并给「查看结果」
*/
const draftOpenRef = useRef(draftOpen)
draftOpenRef.current = draftOpen
const draft = useDraftJob(sourceId, data?.active_job, {
onTablesEnded: (tables) => {
setDrafted((cur) => ({ seq: cur.seq + 1, tables: new Set(tables) }))
void load()
},
onFinished: (job) => {
void load()
if (draftOpenRef.current) return
const src = job.source_id
const opts = {
key: `catalog-draft:${src}`, duration: 10000,
action: { label: DT.viewResult, onClick: () => { if (sourceRef.current === src) setDraftOpen({ view: 'job' }) } },
}
// 留空的列不算没写全，提示里接着写一句，点「查看结果」看是哪几张表、怎么办
const blank = jobBlank(job)
if (job.status === 'succeeded') toast.ok(DT.toast.withBlank(DT.toast.succeeded(CT.draftSummary(job.added, job.updated, job.removed)), blank), opts)
else if (job.status === 'partial') toast.warn(DT.toast.withBlank(DT.toast.partial(retryableCount(job)), blank), opts)
else if (job.status === 'failed') toast.error(DT.toast.failed(job.error ?? ''), opts)
else if (job.status === 'interrupted') toast.warn(DT.toast.interrupted, opts)
else toast.warn(DT.toast.cancelled, opts)
},
})
const draftJob = draft.job
const draftActive = isJobActive(draftJob)
/** 打开起草对话框：有进行中的任务就直接显示它；上一次的结果看过了，再点就是重新开始 */
const openDraft = useCallback(() => setDraftOpen({ view: draftActive ? 'job' : 'setup' }), [draftActive])
const startDraft = useCallback(async (body: DraftStartBody) => {
const r = await draft.start(body)
// 选中的表已经交给起草任务了：清空，免得接着点「起草所选」又起草一遍
if (!r.busy) setSelected(new Set())
return r
}, [draft.start])
/** 详情里「重新起草这张表」：只含这张表、用模型、不带补充说明。模型用不了等开始不了的，直接提示原因 */
const redraftTable = useCallback(async (t: string) => {
try {
const r = await draft.start({ tables: [t], use_model: true })
setDraftOpen({ view: 'job', busy: r.busy })
} catch (e) {
toast.error(e)
}
}, [draft.start])
/** 详情里「补充起草要求」（上次起草有留空的列）：打开起草设置，范围只有这张表，先补起草要求再开始 */
const draftWithGuidance = useCallback((t: string) => {
setDraftOpen({ view: 'setup', preset: { tables: [t], label: DT.scopeTable(t) } })
}, [])
const onGuidanceSaved = useCallback((g: string | null) => {
setData((cur) => cur && ({ ...cur, guidance: g }))
}, [])
// 上方的进度：跟着的任务在进行就按它；跟着的就是清单说的那一个、已经结束了，就不信清单（清单还没重取回来）
const listActive = isJobActive(data?.active_job) ? data?.active_job ?? null : null
const stripJob: { done: number; total: number; status: string } | null = draftJob && (draftActive || draftJob.id === listActive?.id)
? (draftActive ? draftJob : null)
: listActive
// 调用接连没写出来的提醒：关着对话框时也要看得到（上方的进度带着它）
const stripNotice = draftActive ? draftJob?.notice?.trim() || null : null
const options = savedOptions?.sourceId === sourceId ? savedOptions.options : source?.options
const profileSettings = useMemo(() => profileSettingsOf(options), [options])
@@ -262,7 +326,7 @@ export function CatalogPage() {
const empty = rows.length > 0 && rows.every((r) => progressOf(r.counts) === 'none')
const hasTables = rows.length > 0
const draftButton = (primary = false) => (
<button type="button" className={clsx('btn btn-sm', primary && 'btn-primary')} onClick={() => setDrafting(true)} data-catalog-draft="">
<button type="button" className={clsx('btn btn-sm', primary && 'btn-primary')} onClick={openDraft} data-catalog-draft="">
<Sparkles size={11} aria-hidden /> {CT.draft}
</button>
)
@@ -281,7 +345,7 @@ export function CatalogPage() {
)
// 页头的「起草」窄屏只留图标，标题才放得下
const headerDraft = (
<button type="button" className="btn btn-sm" onClick={() => setDrafting(true)} aria-label={CT.draft} title={CT.draft} data-catalog-draft="">
<button type="button" className="btn btn-sm" onClick={openDraft} aria-label={CT.draft} title={CT.draft} data-catalog-draft="">
<Sparkles size={11} aria-hidden /> <span className="hidden sm:inline">{CT.draft}</span>
</button>
)
@@ -335,7 +399,7 @@ export function CatalogPage() {
onSelected={setSelected}
active={table}
onOpen={openTable}
onDraft={() => setDrafting(true)}
onDraft={openDraft}
onProfile={openProfile}
/>
</aside>
@@ -343,7 +407,8 @@ export function CatalogPage() {
{table
? (
<TableDetail sourceId={sourceId} table={table} rows={rows} drafted={drafted} fillCodes={fillCodes} prev={prev} next={next}
onFillDone={onFillDone} onOpen={openTable} onBack={backToList} onDetail={onDetail} />
onFillDone={onFillDone} onOpen={openTable} onBack={backToList} onDetail={onDetail}
draftBusy={!!stripJob} onRedraft={redraftTable} onDraftGuidance={draftWithGuidance} />
)
: <Overview rows={rows} empty={empty} systemNotes={data.system_notes} draft={draftButton(true)} onOpen={openTable} />}
</section>
@@ -373,16 +438,28 @@ export function CatalogPage() {
onGoSource={() => navigate(back)} />
)}
{hasTables && usage.used > 0 && !empty && <UsageSummary {...usage} active={usedOnly ? filter : null} onFocus={focusUsed} />}
{hasTables && stripJob && !draftOpen && <DraftStrip job={stripJob} notice={stripNotice} onOpen={openDraft} />}
{body}
{drafting && data && (
{draftOpen && data && (
<DraftDialog
sourceId={sourceId}
sourceDescription={source?.description ?? ''}
guidance={data.guidance ?? ''}
rows={rows}
visible={visible}
selected={[...selected]}
filtered={filtered}
onClose={() => setDrafting(false)}
onDrafted={onDrafted}
job={draftJob}
initial={draftOpen.view}
initialBusy={!!draftOpen.busy}
preset={draftOpen.preset ?? null}
stopping={draft.stopping}
onStart={startDraft}
onCancel={draft.cancel}
onRetry={draft.retry}
onTrack={draft.track}
onGuidanceSaved={onGuidanceSaved}
onClose={() => setDraftOpen(null)}
/>
)}
{profiling && data && (
@@ -412,6 +489,41 @@ export function CatalogPage() {
)
}
/**
* 起草任务在进行：目录页上方一条进度（关掉对话框任务照常进行，每张表起草完就写进目录）。点「查看进度」回到对话框。
* notice：调用接连没写出来时服务端给的提醒，跟着写一行（全文在悬停里，对话框里也有）
*/
function DraftStrip({ job, notice, onOpen }: {
job: { done: number; total: number; status: string }
notice: string | null
onOpen: () => void
}) {
const text = job.status === 'queued' ? DT.stripQueued(job.total) : DT.strip(job.done, job.total)
return (
<div className="flex shrink-0 items-center gap-2 border-b px-4 py-2 text-xs"
style={{ background: 'color-mix(in srgb, var(--accent) 6%, var(--bg-panel))' }}
data-catalog-draft-progress={`${job.done}/${job.total}`}>
<Spinner size={12} />
<div className="min-w-0 flex-1">
<p className="truncate font-medium" title={text}>{text}</p>
{notice && (
<p className="mt-0.5 flex min-w-0 items-center gap-1 text-2xs text-[var(--warn)]" role="status" title={notice}
data-catalog-draft-notice="">
<TriangleAlert size={10} className="shrink-0" aria-hidden />
<span className="min-w-0 truncate">{notice}</span>
</p>
)}
<div className="mt-1 h-1 max-w-xs overflow-hidden rounded-full bg-hover" role="progressbar" aria-label={text}
aria-valuemin={0} aria-valuemax={job.total} aria-valuenow={job.done}>
<div className="h-full rounded-full bg-[var(--accent)] transition-[width] duration-300"
style={{ width: `${job.total ? (job.done / job.total) * 100 : 0}%` }} />
</div>
</div>
<button type="button" className="btn btn-sm shrink-0" onClick={onOpen} data-catalog-draft-open="">{DT.stripOpen}</button>
</div>
)
}
/** 媒体查询的当前结果，跟着窗口大小变 */
function useMedia(query: string): boolean {
const [on, setOn] = useState(() => typeof matchMedia !== 'function' || matchMedia(query).matches)
frontend/src/pages/catalog/DraftDialog.tsx
+487 -164
View File
@@ -1,66 +1,58 @@
import { useMemo, useRef, useState } from 'react'
import { Sparkles } from 'lucide-react'
import { useEffect, useMemo, useRef, useState } from 'react'
import { CircleCheck, CircleDashed, CircleMinus, CircleSlash, CircleX, Info, RotateCcw, Sparkles, TriangleAlert } from 'lucide-react'
import type { LucideIcon } from 'lucide-react'
import clsx from 'clsx'
import { api } from '../../api/client'
import type { CatalogDraftRow, CatalogTableRow } from '../../types'
import { Modal, Notice, Spinner } from '../../components/ui'
import { ApiError, api } from '../../api/client'
import type { CatalogDraftJob, CatalogDraftTask, CatalogDraftTaskStatus, CatalogTableRow } from '../../types'
import { Field, Modal, Notice, Spinner } from '../../components/ui'
import { errorMessage } from '../../lib/errors'
import { CATALOG_TEXT as CT } from '../../lib/terms'
import { formatTime } from '../../lib/format'
import { CATALOG_DRAFT_TEXT as DT, CATALOG_TEXT as CT, DRAFT_GUIDANCE_MAX, DRAFT_NOTE_MAX } from '../../lib/terms'
import {
DRAFT_MAX_TABLES, DRAFT_MODEL_UNAVAILABLE, DRAFT_PART_COLUMNS, DRAFT_TOP_N, RETRYABLE, blankOf, isJobActive, jobBlank, needsLook,
nothingWritten, retryableCount, taskReason,
} from './draftJob'
import type { DraftPreset, DraftStartBody } from './draftJob'
// ===========================================================================
// 起草数据目录。接口是同步的，一次请求起草完才返回；表多时分批调用，每批回来就更新进度和累计的新增、更新、删除，
// 中途可以停止（当前这一批做完后不再发下一批）。模型整体用不了时第一批就会知道：之后的批次不再请模型，结果里
// 写明原因。已经写进去的批次不回滚——起草只增改推断项，人工确认、驳回过的项服务端不会动。
// 起草数据目录。起草是服务端的后台任务：这里选范围、写起草要求和本次补充说明，建任务；之后显示页面跟着的那个任务
// （进行中的进度、结束后的结果），任务本身的状态在页面上（draftJob.ts），关掉对话框照常进行。
//
// 三段：设置（范围、用不用模型；用模型时写起草要求、补充说明，写明用哪个起草模型）→ 进行中（总进度、合计、调用接连
// 没写出来时的提醒、逐表的状态和宽表的段进度、没写全的原因、留空的列；可以停止、在后台继续）→ 结束（汇总、没写全的表
// 和原因、留空的列合计、只重试失败的表）。
// 已经写进目录的内容不回滚——起草只增改推断项，人工确认、驳回过的项服务端不会动。
//
// 留空的列（模型列出了、中文名和含义都没给）不算没写全，表照样是已完成；结果里写合计和办法——在起草要求里补充缩写和
// 术语的含义，再重新起草这几张表。「重新起草这几张表」回到设置：范围就是那几张、勾着用模型、焦点在起草要求，
// 补完再开始。表详情里的「补充起草要求」也是这样打开（范围是那一张，见 preset）。预先选好的范围里的表已经不在表结构
// 里时不退回别的范围：那一项照旧选着、写明原因，不能开始。
// ===========================================================================
/** 每批几张表。只按注释、外键和命名起草时只看探查缓存，很快；请模型时服务端每 4 张一批、最多 3 批并发 */
const BATCH_PLAIN = 40
const BATCH_MODEL = 12
/** 服务端不给 tables 时取前 20 张，这里给出同样的范围，进度才算得出来 */
const TOP_N = 20
/** 结果里逐张列出的表最多几张，其余合成一句 */
const LIST_MAX = 8
type Scope = 'preset' | 'selected' | 'filtered' | 'top' | 'all'
type Scope = 'selected' | 'filtered' | 'top' | 'all'
type Phase = 'setup' | 'running' | 'done'
/** 字数按服务端的算法：去掉首尾空白后的字符数（按码位，不按 UTF-16 的长度） */
const charCount = (s: string) => [...s.trim()].length
interface Outcome {
done: number
total: number
added: number
updated: number
removed: number
tableErrors: CatalogDraftRow[]
modelErrors: CatalogDraftRow[]
modelSkipped: string | null
model: string | null
failure: string | null
stopped: boolean
/** 这一次勾了「用模型起草」 */
modelAsked: boolean
/** 模型给了可用内容（model_items > 0）的表有几张；服务端没给 model_items（老服务端）时 modelItemsKnown 为假 */
modelItems: number
modelItemsKnown: boolean
}
const fresh = (total: number, modelAsked = false): Outcome => ({
done: 0, total, added: 0, updated: 0, removed: 0, tableErrors: [], modelErrors: [], modelSkipped: null, model: null,
failure: null, stopped: false, modelAsked, modelItems: 0, modelItemsKnown: false,
})
/** 起草要求输入框的 id：预先选好范围回到设置时焦点放在这里 */
const GUIDANCE_ID = 'catalog-draft-guidance'
/**
* 一项都没写进去时怎么说。「没有变化」只在确实是「起草出了内容、和现有目录一致」时说：有表没起草、模型在某些表上
* 没给出内容（model_error），或者勾了用模型、却没有一张表拿到模型的可用内容（模型整体用不了、model_items 都是 0），
* 都说「目录没有更新」，原因写在下面
* 建任务（开始、只重试失败的表）没建成怎么说。起草模型用不了（409，建任务和重试都会回）：服务端那句话已经写了原因和
* 去处，照原样，长句也不拆成「标题：原因」；别的错误前面写明没有开始
*/
function nothingWrittenText(out: Outcome): string {
const failed = out.modelErrors.length > 0 || out.tableErrors.length > 0
const modelEmpty = out.modelAsked && (!!out.modelSkipped || (out.modelItemsKnown && out.modelItems === 0))
return failed || modelEmpty ? CT.draftNothingWritten : CT.draftNoChange
}
const startErrorText = (e: unknown) =>
(e instanceof ApiError && e.code === DRAFT_MODEL_UNAVAILABLE ? e.message : DT.startFailed(errorMessage(e)))
export function DraftDialog({ sourceId, rows, visible, selected, filtered, onClose, onDrafted }: {
export function DraftDialog({
sourceId, sourceDescription, guidance, rows, visible, selected, filtered, job, initial, initialBusy = false, preset: initialPreset = null,
stopping, onStart, onCancel, onRetry, onTrack, onGuidanceSaved, onClose,
}: {
sourceId: string
/** 数据源说明：用模型起草时连同起草要求一起交给模型 */
sourceDescription: string
/** 数据源存着的起草要求（没有为空串） */
guidance: string
/** 全部表（服务端的顺序：使用次数多的在前） */
rows: CatalogTableRow[]
/** 当前筛选结果 */
@@ -68,93 +60,233 @@ export function DraftDialog({ sourceId, rows, visible, selected, filtered, onClo
selected: string[]
/** 清单上有没有生效的搜索或筛选：有时才给「当前筛选结果」这一项 */
filtered: boolean
/** 页面跟着的起草任务（进行中的、刚结束的） */
job: CatalogDraftJob | null
/** 打开时显示什么：setup 选范围、开始；job 显示页面跟着的那个任务 */
initial: 'setup' | 'job'
/** 打开时显示的任务是撞上已有任务（409）转过来的：写明 */
initialBusy?: boolean
/**
* 打开设置时预先选好的范围（表详情里「补充起草要求」：只有那张表）。给了就选它；里面有可起草的表时勾上用模型、
* 焦点放在起草要求上（打开它就是为了先补起草要求再起草）。一张可起草的都没有（已不在表结构里）时照旧选着它、
* 写明原因、不能开始，也不勾用模型——不退回别的范围
*/
preset?: DraftPreset | null
/** 点了停止、服务端还没停下来 */
stopping: boolean
/** 建任务；409 已有任务时页面改取进行中的那一个（busy 为真） */
onStart: (body: DraftStartBody) => Promise<{ job: CatalogDraftJob; busy: boolean }>
onCancel: () => Promise<void>
onRetry: (from: CatalogDraftJob) => Promise<{ job: CatalogDraftJob; busy: boolean }>
/** 设置阶段点开了上一次起草的结果（或者发现别处建的进行中的任务）：页面改跟这个任务 */
onTrack: (job: CatalogDraftJob) => void
/** 起草要求存好了：页面记下新的这一份 */
onGuidanceSaved: (guidance: string | null) => void
onClose: () => void
/** 有批次写进去了：重取清单、刷新正在看的表 */
onDrafted: (tables: string[]) => void
}) {
// 表结构里已经没有的表起草不了（服务端会回「表结构里没有」），不算进范围
// 表结构里已经没有的表起草不了（服务端会记「已跳过」），不算进范围
const live = useMemo(() => rows.filter((r) => r.in_schema).map((r) => r.table_name), [rows])
const liveSet = useMemo(() => new Set(live), [live])
/** 预先选好的范围：打开时给的，或者结果里点了「重新起草这几张表」 */
const [preset, setPreset] = useState<DraftPreset | null>(initialPreset)
const scopes = useMemo(() => {
const out: { key: Scope; label: string; tables: string[] }[] = []
const out: { key: Scope; label: string; tables: string[]; gone?: number }[] = []
// 预先选好的范围总在第一项：里面的表已经不在表结构里的（打开表详情以后别处重新探查了结构）不起草、写明有几张；
// 一张都不剩也留着这一项，不退回别的范围——以为只起草那一张，实际按别的范围用模型起草了
if (preset) {
const pre = preset.tables.filter((t) => liveSet.has(t))
out.push({ key: 'preset', label: preset.label, tables: pre, gone: preset.tables.length - pre.length })
}
const sel = selected.filter((t) => liveSet.has(t))
if (sel.length) out.push({ key: 'selected', label: CT.scopeSelected(sel.length), tables: sel })
const vis = visible.filter((r) => r.in_schema).map((r) => r.table_name)
if (filtered && vis.length && vis.length !== live.length) out.push({ key: 'filtered', label: CT.scopeFiltered(vis.length), tables: vis })
if (live.length > TOP_N) out.push({ key: 'top', label: CT.scopeTop(TOP_N), tables: live.slice(0, TOP_N) })
if (live.length > DRAFT_TOP_N) out.push({ key: 'top', label: CT.scopeTop(DRAFT_TOP_N), tables: live.slice(0, DRAFT_TOP_N) })
out.push({ key: 'all', label: CT.scopeAll(live.length), tables: live })
return out
}, [selected, visible, filtered, live, liveSet])
}, [preset, selected, visible, filtered, live, liveSet])
const [scope, setScope] = useState<Scope>(() => scopes[0].key)
const [useModel, setUseModel] = useState(false)
const [phase, setPhase] = useState<Phase>('setup')
const [out, setOut] = useState<Outcome>(() => fresh(0))
const [stopping, setStopping] = useState(false)
const stopRef = useRef(false)
const tables = scopes.find((s) => s.key === scope)?.tables ?? scopes[0].tables
/** 起草的就是这一项：选的那一项不在了（所选、筛选变了）就是第一项，单选框也照它勾，看得见起草的是哪些 */
const current = scopes.find((s) => s.key === scope) ?? scopes[0]
const tables = current.tables
const tooMany = tables.length > DRAFT_MAX_TABLES
/** 打开时给的范围里有可起草的表：勾上用模型、焦点放进起草要求（打开它就是为了先补起草要求）；一张都没有就都不做 */
const presetLive = !!initialPreset && initialPreset.tables.some((t) => liveSet.has(t))
const [useModel, setUseModel] = useState(presetLive)
/** 回到设置以后把焦点放进起草要求（预先选好范围的两种入口都是为了先补起草要求） */
const [focusGuidance, setFocusGuidance] = useState(presetLive)
const [guidanceText, setGuidanceText] = useState(guidance)
const [note, setNote] = useState('')
const guidanceLen = charCount(guidanceText)
const noteLen = charCount(note)
const guidanceOver = guidanceLen > DRAFT_GUIDANCE_MAX
const noteOver = noteLen > DRAFT_NOTE_MAX
const guidanceChanged = guidanceText.trim() !== guidance.trim()
const run = async () => {
stopRef.current = false
setStopping(false)
setPhase('running')
const acc = fresh(tables.length, useModel)
setOut({ ...acc })
const touched: string[] = []
let withModel = useModel
let i = 0
while (i < tables.length) {
if (stopRef.current) { acc.stopped = true; break }
const batch = tables.slice(i, i + (withModel ? BATCH_MODEL : BATCH_PLAIN))
try {
const r = await api.dataCatalog.draft(sourceId, { tables: batch, use_model: withModel })
acc.added += r.total.added
acc.updated += r.total.updated
acc.removed += r.total.removed
acc.tableErrors.push(...r.tables.filter((t) => t.error))
acc.modelErrors.push(...r.tables.filter((t) => !t.error && t.model_error))
acc.modelItems += r.tables.filter((t) => !t.error && (t.model_items ?? 0) > 0).length
if (r.tables.some((t) => t.model_items !== undefined)) acc.modelItemsKnown = true
if (r.model) acc.model = r.model
// 模型整体用不了（没配置、已停用……）：后面的批次不再请模型，免得每批都撞一次同样的错
if (withModel && r.model_error && !r.model_used) { acc.modelSkipped = r.model_error; withModel = false }
touched.push(...r.tables.filter((t) => !t.error).map((t) => t.table_name))
} catch (e) {
acc.failure = errorMessage(e)
break
const [mode, setMode] = useState<'setup' | 'job'>(initial === 'job' && job ? 'job' : 'setup')
const [starting, setStarting] = useState(false)
const [startError, setStartError] = useState<string | null>(null)
/** 是 409（已有任务）转过来显示的那一个 */
const [busy, setBusy] = useState(initialBusy)
const [acting, setActing] = useState(false)
const [actionError, setActionError] = useState<string | null>(null)
/** 上一次起草（没写全、有留空列的才提），设置阶段写一行、可以点开 */
const [last, setLast] = useState<CatalogDraftJob | null>(null)
/** 起草模型的说法：设置里的起草模型，没配就是跟随助手的模型 */
const [modelLine, setModelLine] = useState<{ text: string; warn: string | null } | null>(null)
const onTrackRef = useRef(onTrack)
onTrackRef.current = onTrack
const inSetup = mode === 'setup' || !job
// 设置阶段：取上一次起草。别处（另一个标签页）建的任务还在进行：直接显示它
useEffect(() => {
if (!inSetup) return
let alive = true
api.dataCatalog.latestDraftJob(sourceId).then(({ job: latest }) => {
if (!alive || !latest) return
if (isJobActive(latest)) {
onTrackRef.current(latest)
setBusy(true)
setMode('job')
} else if (latest.status !== 'succeeded' || jobBlank(latest) > 0) {
setLast(latest)
}
i += batch.length
acc.done = i
setOut({ ...acc })
}, () => {})
return () => { alive = false }
}, [inSetup, sourceId])
// 预先选好范围回到设置（打开时给的、结果里点的）：焦点放进起草要求
useEffect(() => {
if (!focusGuidance || mode !== 'setup' || !useModel) return
const el = document.getElementById(GUIDANCE_ID)
if (!el) return
el.focus()
setFocusGuidance(false)
}, [focusGuidance, mode, useModel])
// 起草模型：设置里的 catalog_draft 填了就是它；没填跟随助手的模型（写出当前是哪个，用不了时写原因）
useEffect(() => {
let alive = true
void Promise.allSettled([api.settings.get(), api.copilot.getModel()]).then(([s, c]) => {
if (!alive) return
const cfg = s.status === 'fulfilled' ? s.value?.catalog_draft : null
const provider = typeof cfg?.provider === 'string' && cfg.provider.trim() ? cfg.provider.trim() : null
const model = typeof cfg?.model === 'string' && cfg.model.trim() ? cfg.model.trim() : null
if (provider || model) {
setModelLine({ text: DT.modelNamed(model, provider), warn: null })
return
}
const assistant = c.status === 'fulfilled' ? c.value : null
setModelLine({ text: DT.modelFollow(assistant?.effective_model ?? null), warn: assistant?.error ?? null })
})
return () => { alive = false }
}, [])
const begin = async () => {
if (starting || !tables.length || tooMany || (useModel && (guidanceOver || noteOver))) return
setStarting(true)
setStartError(null)
try {
// 起草要求改过：先存（存不上就不开始，免得模型拿着旧的那份起草）
if (useModel && guidanceChanged) {
try {
const saved = await api.dataCatalog.putGuidance(sourceId, guidanceText.trim())
onGuidanceSaved(saved.guidance)
} catch (e) {
setStartError(DT.saveGuidanceFailed(errorMessage(e)))
return
}
}
const extra = useModel && note.trim() ? { note: note.trim() } : {}
const r = await onStart({ tables, use_model: useModel, ...extra })
setBusy(r.busy)
setMode('job')
} catch (e) {
// 起草模型用不了（没有接入、已停用、只有演示接入）时留在这一步，写明原因
setStartError(startErrorText(e))
} finally {
setStarting(false)
}
setOut({ ...acc })
setPhase('done')
if (touched.length) onDrafted(touched)
}
const stop = () => {
stopRef.current = true
setStopping(true)
const act = async (fn: () => Promise<void>, describe: (e: unknown) => string = errorMessage) => {
setActing(true)
setActionError(null)
try {
await fn()
} catch (e) {
setActionError(describe(e))
} finally {
setActing(false)
}
}
const stop = () => act(onCancel)
// 重试是新建一个任务：没建成（起草模型用不了、没有表结构……）和开始时同一个说法，留在结果里
const retry = (from: CatalogDraftJob) => act(async () => {
const r = await onRetry(from)
setBusy(r.busy)
}, startErrorText)
const openLast = (j: CatalogDraftJob) => {
onTrack(j)
setBusy(false)
setMode('job')
}
/**
* 结果里「重新起草这几张表」（有留空列的表）：回到设置，范围就是这几张，勾上用模型，焦点放进起草要求——先补上
* 缩写和术语的含义再开始。任务本身不动（页面照旧跟着它，开始新的起草才换）
*/
const redraftBlank = (names: string[]) => {
setPreset({ tables: names, label: DT.scopeBlank(names.length) })
setScope('preset')
setUseModel(true)
setFocusGuidance(true)
setStartError(null)
setActionError(null)
setBusy(false)
setLast(null)
setMode('setup')
}
/** 结果里有留空列、还在表结构里的表（「重新起草这几张表」的范围） */
const blankTables = useMemo(
() => (job ? job.tasks.filter((t) => blankOf(t) > 0 && liveSet.has(t.table_name)).map((t) => t.table_name) : []),
[job, liveSet],
)
// 进行中关窗 = 停止：当前这一批做完再关，结果照常显示
const requestClose = () => (phase === 'running' ? stop() : onClose())
const phase: 'setup' | 'running' | 'done' = inSetup ? 'setup' : isJobActive(job) ? 'running' : 'done'
const canStart = !!tables.length && !tooMany && !(useModel && (guidanceOver || noteOver)) && !starting
const retryable = job && phase === 'done' ? retryableCount(job) : 0
const footer = phase === 'setup'
? (
<>
<button className="btn" onClick={onClose}>{CT.cancel}</button>
<button className="btn btn-primary" onClick={() => void run()} disabled={!tables.length} data-draft-start="">
<Sparkles size={12} aria-hidden /> {CT.draftStart(tables.length)}
<button className="btn btn-primary" onClick={() => void begin()} disabled={!canStart} data-draft-start="">
{starting ? <Spinner size={11} /> : <Sparkles size={12} aria-hidden />} {CT.draftStart(tables.length)}
</button>
</>
)
: phase === 'running'
? <button className="btn" onClick={stop} disabled={stopping} data-draft-stop="">{CT.draftStop}</button>
: <button className="btn btn-primary" onClick={onClose} data-autofocus data-draft-close="">{CT.draftClose}</button>
? (
<>
<button className="btn" onClick={onClose} title={DT.hideHint} data-draft-hide="">{DT.hide}</button>
<button className="btn" onClick={() => void stop()} disabled={stopping || acting} data-draft-stop="">{DT.stop}</button>
</>
)
: (
<>
{retryable > 0 && job && (
<button className="btn" onClick={() => void retry(job)} disabled={acting} title={DT.retryHint} data-draft-retry="">
{acting ? <Spinner size={11} /> : <RotateCcw size={11} aria-hidden />} {DT.retry(retryable)}
</button>
)}
<button className="btn btn-primary" onClick={onClose} data-autofocus data-draft-close="">{CT.draftClose}</button>
</>
)
return (
<Modal open onClose={requestClose} title={CT.draftTitle} width={560} footer={footer}>
<div className="space-y-4" data-draft-dialog={phase}>
<Modal open onClose={onClose} title={CT.draftTitle} width={600} footer={footer}
dirty={phase === 'setup' && useModel && (guidanceChanged || !!note.trim())}>
<div className="min-w-0 space-y-4" data-draft-dialog={phase}>
{phase === 'setup' && (
<>
<fieldset>
@@ -162,77 +294,268 @@ export function DraftDialog({ sourceId, rows, visible, selected, filtered, onClo
<div className="space-y-1.5" role="radiogroup" aria-label={CT.draftScope}>
{scopes.map((s) => (
<label key={s.key} className={clsx('flex cursor-pointer items-center gap-2 rounded-md border px-3 py-2 text-xs',
scope === s.key ? 'border-[var(--accent)] bg-accent-soft' : 'hover:bg-hover')} data-draft-scope={s.key}>
<input type="radio" name="catalog-draft-scope" checked={scope === s.key} onChange={() => setScope(s.key)} />
<span>{s.label}</span>
current.key === s.key ? 'border-[var(--accent)] bg-accent-soft' : 'hover:bg-hover')} data-draft-scope={s.key}>
<input type="radio" name="catalog-draft-scope" checked={current.key === s.key} onChange={() => setScope(s.key)} />
<span className="min-w-0 break-words">
{s.label}
{!!s.gone && (
<span className={clsx('block text-2xs', s.tables.length ? 'text-faint' : 'text-[var(--warn)]')} data-draft-preset-missing="">
{DT.presetMissing(s.gone, s.gone + s.tables.length)}
</span>
)}
</span>
</label>
))}
</div>
{tooMany && <p className="mt-1 text-2xs text-[var(--err)]" data-draft-too-many="">{DT.tooMany(DRAFT_MAX_TABLES)}</p>}
</fieldset>
<p className="text-2xs leading-relaxed text-faint">{CT.draftBase}</p>
<label className="flex cursor-pointer items-start gap-2 rounded-md border px-3 py-2.5 text-xs hover:bg-hover" data-draft-model="">
<input type="checkbox" className="mt-0.5" checked={useModel} onChange={(e) => setUseModel(e.target.checked)} />
<span className="min-w-0">
<span className="font-medium">{CT.useModel}</span>
<span className="mt-0.5 block text-2xs leading-relaxed text-faint">{CT.useModelHint}</span>
</span>
</label>
<div className="rounded-md border">
<label className="flex cursor-pointer items-start gap-2 rounded-md px-3 py-2.5 text-xs hover:bg-hover" data-draft-model="">
<input type="checkbox" className="mt-0.5" checked={useModel} onChange={(e) => setUseModel(e.target.checked)} />
<span className="min-w-0">
<span className="font-medium">{CT.useModel}</span>
<span className="mt-0.5 block text-2xs leading-relaxed text-faint">{CT.useModelHint}</span>
</span>
</label>
{useModel && (
<div className="space-y-3 border-t px-3 py-3">
<div className="text-2xs leading-relaxed" data-draft-model-name="">
<span className="text-dim">{DT.model}：</span>
<span className="break-words font-medium">{modelLine ? modelLine.text : '…'}</span>
<span className="block text-faint">{DT.modelWhere}</span>
{modelLine?.warn && <span className="block text-[var(--warn)]" data-draft-model-warn="">{modelLine.warn}</span>}
</div>
<div data-draft-guidance="">
<Field label={DT.guidance} htmlFor={GUIDANCE_ID} hint={DT.guidanceHint}
error={guidanceOver ? DT.tooLong(DRAFT_GUIDANCE_MAX) : undefined}>
{(p) => (
<textarea {...p} className="field min-h-[4.5rem] resize-y text-xs leading-relaxed" rows={3} value={guidanceText}
placeholder={DT.guidancePlaceholder} onChange={(e) => setGuidanceText(e.target.value)} />
)}
</Field>
<Counter n={guidanceLen} max={DRAFT_GUIDANCE_MAX} />
</div>
<div data-draft-note="">
<Field label={DT.note} htmlFor="catalog-draft-note" hint={DT.noteHint}
error={noteOver ? DT.tooLong(DRAFT_NOTE_MAX) : undefined}>
{(p) => (
<textarea {...p} className="field min-h-[3rem] resize-y text-xs leading-relaxed" rows={2} value={note}
placeholder={DT.notePlaceholder} onChange={(e) => setNote(e.target.value)} />
)}
</Field>
<Counter n={noteLen} max={DRAFT_NOTE_MAX} />
</div>
{sourceDescription.trim() && (
<p className="break-words text-2xs leading-relaxed text-dim" data-draft-source-description="">
{DT.sourceDescription(sourceDescription.trim())}
</p>
)}
<p className="text-2xs leading-relaxed text-faint">{DT.guidanceRule}</p>
</div>
)}
</div>
{last && (
// 没写全的用警告色；都写完了、只是有留空列的是提示
<div className="flex flex-wrap items-center gap-x-2 gap-y-1 rounded-md border px-3 py-2 text-2xs text-dim" data-draft-last={last.status}>
{last.status === 'succeeded'
? <Info size={11} className="shrink-0" style={{ color: 'var(--accent)' }} aria-hidden />
: <TriangleAlert size={11} className="shrink-0" style={{ color: 'var(--warn)' }} aria-hidden />}
<span className="min-w-0 flex-1">
{DT.last(DT.lastState(DT.title[last.status], jobBlank(last)), formatTime(last.finished_at ?? last.created_at))}
</span>
<button type="button" className="btn btn-sm" onClick={() => openLast(last)} data-draft-last-open="">{DT.lastOpen}</button>
</div>
)}
{startError && <Notice tone="err" attr={{ 'data-draft-start-error': '' }}>{startError}</Notice>}
</>
)}
{phase !== 'setup' && (
<div className="space-y-3" aria-live="polite">
<div className="flex items-center gap-2 text-xs">
{phase === 'running' && <Spinner size={13} />}
<span className="font-medium" data-draft-title="">
{phase === 'running'
? (stopping ? CT.draftStopping : CT.drafting)
: out.failure ? CT.draftFailed : out.stopped ? CT.draftStopped(out.done, out.total) : CT.draftDone}
</span>
<span className="flex-1" />
<span className="tnum text-2xs text-faint" data-draft-progress={`${out.done}/${out.total}`}>{CT.draftProgress(out.done, out.total)}</span>
</div>
<div className="h-1.5 overflow-hidden rounded-full bg-hover" role="progressbar" aria-label={CT.draftProgress(out.done, out.total)}
aria-valuemin={0} aria-valuemax={out.total} aria-valuenow={out.done}>
<div className="h-full rounded-full transition-[width] duration-300"
style={{ width: `${out.total ? (out.done / out.total) * 100 : 0}%`, background: out.failure ? 'var(--err)' : 'var(--accent)' }} />
</div>
<p className="tnum text-sm" data-draft-summary="">
{out.added + out.updated + out.removed === 0 && phase === 'done' && !out.failure
? nothingWrittenText(out)
: CT.draftSummary(out.added, out.updated, out.removed)}
</p>
{out.model && <p className="text-2xs text-faint">{CT.draftModelUsed(out.model)}</p>}
{out.modelSkipped && (
<Notice tone="warn" attr={{ 'data-draft-model-error': '' }}>{CT.draftModelSkipped(out.modelSkipped)}</Notice>
)}
{out.failure && <Notice tone="err" attr={{ 'data-draft-failure': '' }}>{out.failure}</Notice>}
{out.tableErrors.length > 0 && (
<Notice tone="err" attr={{ 'data-draft-table-errors': String(out.tableErrors.length) }}>
<p>{CT.draftTableErrors(out.tableErrors.length)}</p>
<ErrorList rows={out.tableErrors} pick={(r) => r.error ?? ''} />
</Notice>
)}
{out.modelErrors.length > 0 && (
<Notice tone="warn" attr={{ 'data-draft-table-model-errors': String(out.modelErrors.length) }}>
<p>{CT.draftModelErrors(out.modelErrors.length)}</p>
<ErrorList rows={out.modelErrors} pick={(r) => r.model_error ?? ''} />
</Notice>
)}
</div>
{phase !== 'setup' && job && (
<JobView key={job.id} job={job} running={phase === 'running'} stopping={stopping} busy={busy}
blankTables={blankTables} onRedraftBlank={redraftBlank} />
)}
{phase !== 'setup' && actionError && <Notice tone="err" attr={{ 'data-draft-action-error': '' }}>{actionError}</Notice>}
</div>
</Modal>
)
}
function ErrorList({ rows, pick }: { rows: CatalogDraftRow[]; pick: (r: CatalogDraftRow) => string }) {
function Counter({ n, max }: { n: number; max: number }) {
return (
<ul className="mt-1 space-y-0.5 text-dim">
{rows.slice(0, LIST_MAX).map((r) => (
<li key={r.table_name} className="break-words"><span className="mono">{r.table_name}</span>：{pick(r)}</li>
))}
{rows.length > LIST_MAX && <li>{CT.moreTables(rows.length - LIST_MAX)}</li>}
</ul>
<div className={clsx('mt-0.5 text-right text-2xs tnum', n > max ? 'text-[var(--err)]' : 'text-faint')} aria-hidden>
{DT.count(n, max)}
</div>
)
}
/**
* 任务的进度或结果：标题、总进度、提醒（进行中）、合计、模型和用量（超时、没写出来的次数）、没写全的表、留空的列合计
* （结束后）、逐表列表
*/
function JobView({ job, running, stopping, busy, blankTables, onRedraftBlank }: {
job: CatalogDraftJob
running: boolean
stopping: boolean
busy: boolean
/** 有留空列、还在表结构里的表：结束后给「重新起草这几张表」 */
blankTables: string[]
onRedraftBlank: (tables: string[]) => void
}) {
const problems = job.tasks.filter((t) => RETRYABLE.has(t.status))
// 结束后默认只列要看的表：没写全的、有留空列的（几百张表里找它们太费劲），可以展开全部
const attention = job.tasks.filter(needsLook)
const blankCount = job.tasks.filter((t) => blankOf(t) > 0).length
const [showAll, setShowAll] = useState(false)
const onlyProblems = !running && attention.length > 0 && !showAll
const tasks = onlyProblems ? attention : job.tasks
const title = running ? (stopping ? DT.stopping : DT.title[job.status]) : DT.title[job.status]
const empty = nothingWritten(job)
const summary = empty === 'model' ? CT.draftNothingWritten
: empty === 'failed' ? CT.draftNothingWrittenFailed
: empty === 'same' ? CT.draftNoChange
: CT.draftSummary(job.added, job.updated, job.removed)
const problemText = !running ? DT.problems({
failed: job.tasks.filter((t) => t.status === 'failed').length,
partial: job.tasks.filter((t) => t.status === 'partial').length,
cancelled: job.tasks.filter((t) => t.status === 'cancelled').length,
skipped: job.tasks.filter((t) => t.status === 'skipped').length,
}) : ''
// 进行中按已结束的表数走；结束以后写全部写出来的有几张（停止、失败的不算「完成」）
const ok = job.tasks.filter((t) => t.status === 'ok').length
const value = running ? job.done : ok
const progress = running ? DT.progress(job.done, job.total) : DT.progressDone(ok, job.total)
const tone = job.status === 'failed' ? 'var(--err)' : running ? 'var(--accent)' : problems.length ? 'var(--warn)' : 'var(--st-done)'
const pct = job.total ? (value / job.total) * 100 : 0
const { usage } = job
const timeouts = usage.timeouts ?? 0
const failures = DT.usageFailures(timeouts, usage.failed_calls ?? 0)
// 调用接连没写出来时服务端给的提醒：进行中写在进度条下面（提醒，不是错误）；好转后服务端清掉
const notice = running ? job.notice?.trim() ?? '' : ''
// 额度用完、超时多了，之后的段缩小了：进行中写一句，段数为什么变多看得明白
const partColumns = running && job.use_model && job.part_columns != null && job.part_columns < DRAFT_PART_COLUMNS ? job.part_columns : null
const blankTotal = jobBlank(job)
return (
<div className="min-w-0 space-y-3">
{busy && <Notice tone="info" attr={{ 'data-draft-busy': '' }}>{DT.busy}</Notice>}
<div className="flex flex-wrap items-center gap-x-2 gap-y-0.5 text-xs" role="status" aria-live="polite">
{running && <Spinner size={13} />}
<span className="font-medium" data-draft-title="">{title}</span>
<span className="flex-1" />
<span className="tnum text-2xs text-faint" data-draft-progress={`${job.done}/${job.total}`}>{progress}</span>
</div>
<div className="h-1.5 overflow-hidden rounded-full bg-hover" role="progressbar" aria-label={progress}
aria-valuemin={0} aria-valuemax={job.total} aria-valuenow={value}>
<div className="h-full rounded-full transition-[width] duration-300" style={{ width: `${pct}%`, background: tone }} />
</div>
{notice && <Notice tone="warn" className="break-words" attr={{ 'data-draft-notice': '' }}>{notice}</Notice>}
<p className="tnum text-sm" data-draft-summary="">{summary}</p>
{(job.model || usage.calls > 0 || failures) && (
<div className="space-y-0.5 text-2xs leading-relaxed text-faint">
<p className="flex flex-wrap gap-x-3 gap-y-0.5" data-draft-usage-line="">
{job.model && <span className="break-all" data-draft-model-used="">{DT.modelUsed(job.model)}</span>}
{usage.calls > 0 && (
<span className="tnum" data-draft-usage="">{DT.usage(usage.calls, usage.input_tokens, usage.output_tokens)}</span>
)}
{failures && (
<span className="tnum text-[var(--warn)]" title={DT.usageFailuresHint} data-draft-usage-failures="">{failures}</span>
)}
</p>
{timeouts > 0 && <p data-draft-timeout-note="">{DT.timeoutNote}</p>}
{partColumns != null && <p data-draft-part-columns={partColumns}>{DT.partColumns(partColumns)}</p>}
</div>
)}
{running && <p className="text-2xs leading-relaxed text-faint">{DT.hideHint}</p>}
{job.status === 'failed' && job.error && <Notice tone="err" attr={{ 'data-draft-job-error': '' }}>{job.error}</Notice>}
{job.status === 'interrupted' && <Notice tone="warn" attr={{ 'data-draft-interrupted': '' }}>{DT.interrupted}</Notice>}
{job.status === 'cancelled' && <p className="text-2xs text-dim" data-draft-cancelled="">{DT.cancelled}</p>}
{problemText && (
<p className="text-xs leading-relaxed" data-draft-problems="">{problemText}</p>
)}
{!running && blankTotal > 0 && (
// 留空的列不算没写全（表照样是已完成）：写合计和办法，给「重新起草这几张表」回到设置先补起草要求
<Notice tone="info" attr={{ 'data-draft-blank-summary': '' }}>
<p className="font-medium">{DT.blankTotal(blankTotal, blankCount)}</p>
<p className="mt-0.5 text-dim">{DT.blankAdvice}</p>
{blankTables.length > 0 && (
<button type="button" className="btn btn-sm mt-2" title={DT.blankRedraftHint} onClick={() => onRedraftBlank(blankTables)}
data-draft-blank-redraft="">
<Sparkles size={11} aria-hidden /> {DT.blankRedraft(blankTables.length)}
</button>
)}
</Notice>
)}
<div>
<div className="mb-1 flex items-center gap-2">
<span className="text-2xs font-medium text-dim" id="catalog-draft-tasks-label">{DT.tasks}</span>
<span className="flex-1" />
{!running && attention.length > 0 && attention.length < job.tasks.length && (
<button type="button" className="rounded px-1 text-2xs text-[var(--accent)] outline-none hover:underline focus-visible:ring-2 focus-visible:ring-[var(--accent)]"
aria-pressed={showAll} onClick={() => setShowAll((v) => !v)} data-draft-show-all="">
{showAll ? (blankCount ? DT.showProblemsBlank : DT.showProblems) : DT.showAll(job.tasks.length)}
</button>
)}
</div>
<ul className="max-h-64 divide-y divide-[var(--hairline)] overflow-y-auto overflow-x-hidden rounded-md border"
aria-labelledby="catalog-draft-tasks-label" data-draft-tasks="">
{tasks.map((t) => <TaskRow key={t.table_name} task={t} jobError={job.error} />)}
</ul>
</div>
</div>
)
}
const TASK_ICON: Record<CatalogDraftTaskStatus, { Icon: LucideIcon | null; color: string }> = {
pending: { Icon: CircleDashed, color: 'var(--text-faint)' },
running: { Icon: null, color: 'var(--accent)' },
ok: { Icon: CircleCheck, color: 'var(--st-done)' },
partial: { Icon: TriangleAlert, color: 'var(--warn)' },
failed: { Icon: CircleX, color: 'var(--err)' },
skipped: { Icon: CircleMinus, color: 'var(--text-faint)' },
cancelled: { Icon: CircleSlash, color: 'var(--text-faint)' },
}
/**
* 逐表的一行。jobError：任务整体的原因（整体出错、服务重启中断时服务端把没结束的表也记上同一句）。和它一样的不再逐表
* 重复——几百张表各写一遍同一句，真正各有原因的那几张反而找不到；任务的原因已经写在上面（标题、提示）。
* 有留空的列时再写一行「N 列模型拿不准、留空了」：列名悬停看得到，也可以展开（服务端最多给 5 个）
*/
function TaskRow({ task: t, jobError }: { task: CatalogDraftTask; jobError: string | null }) {
const { Icon, color } = TASK_ICON[t.status]
const reason = jobError && t.error === jobError ? (t.model_error ?? '') : taskReason(t)
const parts = t.parts_total != null && t.parts_total > 1 && t.status !== 'pending' && t.status !== 'skipped'
const changes = DT.taskAdded(t.added, t.updated)
const blank = blankOf(t)
const sample = t.blank_sample?.filter(Boolean) ?? []
return (
<li className="min-w-0 px-3 py-1.5" data-draft-task={t.table_name} data-status={t.status}>
<div className="flex min-w-0 items-center gap-2 text-xs">
{Icon ? <Icon size={12} className="shrink-0" style={{ color }} aria-hidden /> : <Spinner size={12} />}
<span className="mono min-w-0 flex-1 truncate" title={t.table_name}>{t.table_name}</span>
{parts && (
<span className="tnum shrink-0 text-2xs text-faint" title={DT.partsHint} data-draft-task-parts="">
{DT.parts(t.parts_done, t.parts_total!)}
</span>
)}
{changes && (
<span className="tnum shrink-0 text-2xs text-faint" title={DT.taskAddedHint(t.added, t.updated, t.removed)}>{changes}</span>
)}
<span className="shrink-0 text-2xs" style={{ color: t.status === 'pending' || t.status === 'skipped' || t.status === 'cancelled' ? undefined : color }}
data-draft-task-status="">
{DT.taskStatus[t.status]}
</span>
</div>
{reason && (
<p className="mt-0.5 break-words pl-5 text-2xs leading-relaxed text-dim" data-draft-task-reason="">{reason}</p>
)}
{blank > 0 && (sample.length
? (
<details className="mt-0.5 pl-5 text-2xs leading-relaxed text-dim" data-draft-task-blank={blank}>
<summary className="cursor-pointer select-none hover:text-fg" title={DT.blankSample(sample, blank)}>{DT.blank(blank)}</summary>
<p className="break-words text-faint" data-draft-task-blank-sample="">{DT.blankSample(sample, blank)}</p>
</details>
)
: <p className="mt-0.5 pl-5 text-2xs leading-relaxed text-dim" data-draft-task-blank={blank}>{DT.blank(blank)}</p>)}
</li>
)
}
frontend/src/pages/catalog/TableDetail.tsx
+61 -3
View File
@@ -1,6 +1,6 @@
import { useCallback, useEffect, useMemo, useRef, useState } from 'react'
import type { ReactNode } from 'react'
import { AlertTriangle, ArrowLeft, ChevronLeft, ChevronRight, GitBranch, ListChecks, Lock, Pencil, Plus, RotateCw, Search, X } from 'lucide-react'
import { AlertTriangle, ArrowLeft, ChevronLeft, ChevronRight, GitBranch, ListChecks, Lock, Pencil, Plus, RotateCw, Search, Sparkles, X } from 'lucide-react'
import clsx from 'clsx'
import { ApiError, api } from '../../api/client'
import type {
@@ -11,7 +11,7 @@ import { DeleteButton, EmptyState, ErrorState, Notice, Skeleton, Spinner, confir
import { useLeaveGuard } from '../../lib/leave'
import { formatDateTime, formatNumber, formatTime } from '../../lib/format'
import {
CATALOG_CARDINALITY_LABEL, CATALOG_IMPACT_TEXT, CATALOG_KIND_HINT, CATALOG_KIND_LABEL, CATALOG_TABLE_FIELD_HINT,
CATALOG_CARDINALITY_LABEL, CATALOG_DRAFT_TEXT as DT, CATALOG_IMPACT_TEXT, CATALOG_KIND_HINT, CATALOG_KIND_LABEL, CATALOG_TABLE_FIELD_HINT,
CATALOG_TABLE_FIELD_LABEL, CATALOG_TEXT as CT, CATALOG_UI_TEXT as UT, CODES_TEXT as KT,
} from '../../lib/terms'
import { CatalogImpactList } from '../../components/CatalogImpact'
@@ -21,6 +21,7 @@ import { ColumnTable, NARROW, Value, pendingCodes } from './ColumnTable'
import type { ColumnModel } from './ColumnTable'
import { CodesDialog } from './CodesDialog'
import { coverageText } from './profile'
import { lastBlankOf, taskReason } from './draftJob'
import {
COLUMN_FIELDS, DATE_KEYS, TABLE_FIELDS, buildNotes, changeCount, columnPath, confirmProposed, countsOf, formFromNotes,
relationDraft, relationPath, relationRemoval, tablePath, unknownKeys, validateForm,
@@ -51,7 +52,9 @@ const PROBLEM_TEXT = {
type Editing = { initial: EditForm; form: EditForm }
type Conflict = 'edit' | 'review'
export function TableDetail({ sourceId, table, rows, drafted, fillCodes, onFillDone, prev, next, onOpen, onBack, onDetail }: {
export function TableDetail({
sourceId, table, rows, drafted, fillCodes, onFillDone, prev, next, onOpen, onBack, onDetail, draftBusy = false, onRedraft, onDraftGuidance,
}: {
sourceId: string
table: string
/** 全部表：关联关系的目标表能点就跳过去，编辑时给目标表做候选 */
@@ -72,6 +75,15 @@ export function TableDetail({ sourceId, table, rows, drafted, fillCodes, onFillD
onBack: () => void
/** 载入或写入之后的单表目录：清单据此更新这一行 */
onDetail: (d: CatalogDetail) => void
/** 这个数据源有排队、进行中的起草任务：「重新起草这张表」等它结束 */
draftBusy?: boolean
/**
* 上次起草失败、部分完成或有留空列时的「重新起草这张表」：建只含这张表的起草任务（用模型、不带补充说明）。
* 表结构里已经没有的表不给
*/
onRedraft?: (table: string) => Promise<void>
/** 上次起草有留空列时的「补充起草要求」：打开起草设置，范围只有这张表，先补起草要求再开始。表结构里已经没有的表不给 */
onDraftGuidance?: (table: string) => void
}) {
const [detail, setDetail] = useState<CatalogDetail | null>(null)
const [loadError, setLoadError] = useState<unknown>(null)
@@ -101,6 +113,7 @@ export function TableDetail({ sourceId, table, rows, drafted, fillCodes, onFillD
*/
const [savedAt, setSavedAt] = useState<number | null>(null)
const [impact, setImpact] = useState<{ version: number; n: number } | null>(null)
const [redrafting, setRedrafting] = useState(false)
const loadSeq = useRef(0)
const onDetailRef = useRef(onDetail)
onDetailRef.current = onDetail
@@ -308,6 +321,11 @@ export function TableDetail({ sourceId, table, rows, drafted, fillCodes, onFillD
const notes = detail.notes
const counts = countsOf(notes)
/** 这张表最近一次起草的结果（清单上那一行带的） */
const lastDraft = rows.find((r) => r.table_name === table)?.last_draft ?? null
/** 没写全（失败、部分完成）；有留空的列（模型拿不准，表照样写完了） */
const lastBad = lastDraft && (lastDraft.status === 'failed' || lastDraft.status === 'partial') ? lastDraft : null
const lastBlank = lastBlankOf(lastDraft)
const label = notes.label && notes.label.status !== 'rejected' ? notes.label.value : null
const kind = notes.kind && notes.kind.status !== 'rejected' ? notes.kind.value : null
const st = detail.structure
@@ -446,6 +464,46 @@ export function TableDetail({ sourceId, table, rows, drafted, fillCodes, onFillD
</Notice>
)}
{writeError && <Notice tone="err" attr={{ 'data-catalog-write-error': '' }}>{writeError}</Notice>}
{lastDraft && (lastBad || lastBlank > 0) && (
// 上次起草没写全：写明原因；有留空的列：写几列、怎么办（在起草要求里补充缩写和术语的含义）。只有留空的列时是
// 提示（表照样写完了），不是失败。给「重新起草这张表」（只含这张表、用模型）；有留空的列再给「补充起草要求」
// （打开起草设置，范围是这张表）。有起草任务在进行、正在编辑这张表时先不给点：起草会写进这张表，编辑中的保存
// 就会撞上版本冲突。表结构里已经没有这张表（只剩目录）时上次的结果照写，不给这两个按钮、不建议重新起草，
// 写明为什么：服务端只会把它记成「已跳过」
<Notice tone={lastBad ? (lastBad.status === 'failed' ? 'err' : 'warn') : 'info'} attr={{ 'data-catalog-last-draft': lastDraft.status }}>
{lastBad && <p className="font-medium">{lastBad.status === 'failed' ? DT.lastFailed : DT.lastPartial}</p>}
{lastBad && taskReason(lastBad) && (
<p className="mt-0.5 break-words text-dim" data-catalog-last-draft-reason="">{taskReason(lastBad)}</p>
)}
{lastBlank > 0 && (
<>
<p className={lastBad ? 'mt-0.5 text-dim' : 'font-medium'} data-catalog-last-draft-blank={lastBlank}>{DT.lastBlank(lastBlank)}</p>
{detail.in_schema && <p className="mt-0.5 text-dim">{DT.lastBlankAdvice}</p>}
</>
)}
{!detail.in_schema && <p className="mt-1 text-2xs text-faint" data-catalog-redraft-missing="">{DT.redraftMissing}</p>}
{detail.in_schema && onRedraft && (
<>
<div className="mt-2 flex flex-wrap items-center gap-1.5">
<button type="button" className="btn btn-sm" disabled={draftBusy || redrafting || !!edit} title={DT.redraftHint}
onClick={async () => {
setRedrafting(true)
try { await onRedraft(table) } finally { setRedrafting(false) }
}} data-catalog-redraft="">
{redrafting ? <Spinner size={11} /> : <Sparkles size={11} aria-hidden />} {DT.redraft}
</button>
{lastBlank > 0 && onDraftGuidance && (
<button type="button" className="btn btn-sm btn-ghost" disabled={draftBusy || redrafting || !!edit} title={DT.draftGuidanceHint}
onClick={() => onDraftGuidance(table)} data-catalog-draft-guidance="">
<Pencil size={11} aria-hidden /> {DT.draftGuidance}
</button>
)}
</div>
<p className="mt-1 text-2xs text-faint">{edit ? DT.redraftEditing : draftBusy ? DT.redraftBusy : DT.redraftHint}</p>
</>
)}
</Notice>
)}
{!detail.in_schema && (
<Notice tone="warn" attr={{ 'data-catalog-missing': '' }}>{CT.missingBody}</Notice>
)}
frontend/src/pages/catalog/TableIndex.tsx
+30 -4
View File
@@ -1,20 +1,24 @@
import { memo, useCallback, useEffect, useRef } from 'react'
import type { KeyboardEvent as ReactKeyboardEvent, MouseEvent as ReactMouseEvent } from 'react'
import { AlertTriangle, History, ScanSearch, Search, Sparkles, X } from 'lucide-react'
import { AlertTriangle, History, Info, ScanSearch, Search, Sparkles, X } from 'lucide-react'
import clsx from 'clsx'
import type { CatalogTableRow } from '../../types'
import { useRadioGroup } from '../../components/ui'
import { formatNumber } from '../../lib/format'
import { CATALOG_KIND_LABEL, CATALOG_TEXT as CT, CATALOG_UI_TEXT as UT, PROFILE_TEXT as PT } from '../../lib/terms'
import {
CATALOG_DRAFT_TEXT as DT, CATALOG_KIND_LABEL, CATALOG_TEXT as CT, CATALOG_UI_TEXT as UT, PROFILE_TEXT as PT,
} from '../../lib/terms'
import { CountBadges } from './parts'
import { LIST_FILTERS, LIST_SORTS, progressOf } from './model'
import { lastBlankOf, taskReason } from './draftJob'
import type { ListFilter, ListSort } from './model'
// ===========================================================================
// 表清单（数据目录页的左栏）：搜索、按审阅进度筛选、排序、多选后批量起草。
// 一行两层：上面是中文名和各状态的项数，下面是表名、类型、关联数和使用次数；还没填中文名的表上面直接写表名，
// 下面淡淡地写「未填写中文名」——一屏几十行「未填写中文名」认不出是哪张表。键盘：↑↓ 在行之间移动，
// Home / End 到两头，Enter 打开；复选框按住 Shift 点选可以连选一段。
// 下面淡淡地写「未填写中文名」——一屏几十行「未填写中文名」认不出是哪张表。上次起草失败、部分完成的表再多一行，
// 写明原因（全文在悬停里，点开表还有「重新起草这张表」）；上次起草有留空列（模型拿不准）的表另有一行淡色的轻提示，
// 不是失败标记。键盘：↑↓ 在行之间移动，Home / End 到两头，Enter 打开；复选框按住 Shift 点选可以连选一段。
// ===========================================================================
export function TableIndex({
@@ -205,6 +209,14 @@ const Row = memo(function Row({ row: r, index, active, checked, onToggle, onOpen
onKey: (i: number, e: ReactKeyboardEvent<HTMLButtonElement>) => void
}) {
const progress = progressOf(r.counts)
// 上次起草没写全：失败、部分完成各一种说法，后面接原因
const last = r.last_draft && (r.last_draft.status === 'failed' || r.last_draft.status === 'partial') ? r.last_draft : null
const lastLabel = last ? (last.status === 'failed' ? DT.lastFailed : DT.lastPartial) : ''
const lastText = last ? (taskReason(last) ? DT.lastWith(lastLabel, taskReason(last)) : lastLabel) : ''
// 上次起草有留空的列：表照样写完了，只是模型对这几列拿不准。淡色的轻提示，和失败标记分开。表结构里已经没有的表
// 起草不了：悬停里不建议重新起草
const blank = lastBlankOf(r.last_draft)
const blankTitle = r.in_schema ? `${DT.lastBlank(blank)}。${DT.lastBlankAdvice}` : DT.lastBlank(blank)
return (
<li
className={clsx('relative flex items-stretch border-b border-hairline', active ? 'bg-accent-soft' : 'hover:bg-hover')}
@@ -247,6 +259,20 @@ const Row = memo(function Row({ row: r, index, active, checked, onToggle, onOpen
<AlertTriangle size={10} aria-hidden /> {CT.missing}
</span>
)}
{last && (
<span className="mt-0.5 flex min-w-0 items-center gap-1 text-2xs" title={lastText} data-row-last-draft={last.status}
style={{ color: last.status === 'failed' ? 'var(--err)' : 'var(--warn)' }}>
<AlertTriangle size={10} className="shrink-0" aria-hidden />
<span className="min-w-0 truncate">{lastText}</span>
</span>
)}
{blank > 0 && (
<span className="mt-0.5 flex min-w-0 items-center gap-1 text-2xs text-faint" title={blankTitle}
data-row-blank={blank}>
<Info size={10} className="shrink-0" aria-hidden />
<span className="min-w-0 truncate">{DT.lastBlank(blank)}</span>
</span>
)}
</button>
</li>
)
frontend/src/pages/catalog/draftJob.ts
+221
View File
@@ -0,0 +1,221 @@
import { useCallback, useEffect, useRef, useState } from 'react'
import { ApiError, api } from '../../api/client'
import type { CatalogActiveDraftJob, CatalogDraftJob, CatalogDraftTask, CatalogDraftTaskStatus } from '../../types'
// ===========================================================================
// 起草任务（/api/datasources/{id}/catalog/draft-jobs）在页面上的状态：建、停、重试，进行中每 1.5 秒取一次进度，
// 结束就停。放在页面上而不是对话框里：关掉对话框任务照常进行，目录页上方的进度、结束时的提示都靠它；再打开对话框
// 看到的还是同一个任务。
//
// 取回来的状态只往前走：已经结束的任务不会被一个晚到的「进行中」翻回去，已结束的表数不减少（停止接口的回答和
// 正在路上的那次取进度谁先到都一样）。
// ===========================================================================
/** 进行中多久取一次进度 */
export const DRAFT_POLL_MS = 1500
/** 一次最多起草几张表（服务端同一个上限） */
export const DRAFT_MAX_TABLES = 500
/** 不选表时起草使用次数最多的前几张（服务端不给 tables 时也是这个数） */
export const DRAFT_TOP_N = 20
/** 服务端的机读码：同一个数据源已有排队、进行中的任务；起草模型用不了 */
export const DRAFT_BUSY = 'catalog_draft_busy'
export const DRAFT_MODEL_UNAVAILABLE = 'catalog_draft_model_unavailable'
const DRAFT_FINISHED = 'catalog_draft_finished'
export const isJobActive = (j: { status: string } | null | undefined): boolean =>
!!j && (j.status === 'queued' || j.status === 'running')
const TASK_ENDED = new Set<CatalogDraftTaskStatus>(['ok', 'partial', 'failed', 'skipped', 'cancelled'])
/** 「只重试失败的表」带上的：失败、部分完成、已停止 */
export const RETRYABLE = new Set<CatalogDraftTaskStatus>(['failed', 'partial', 'cancelled'])
export const isTaskEnded = (t: CatalogDraftTask): boolean => TASK_ENDED.has(t.status)
export const retryableCount = (j: CatalogDraftJob): number => j.tasks.filter((t) => RETRYABLE.has(t.status)).length
/** 一张表这次为什么没写全：整张表没起草的原因在前，模型那部分的原因在后 */
export const taskReason = (t: { error: string | null; model_error: string | null }): string => t.error ?? t.model_error ?? ''
/** 每段最多几列（服务端起草的起始上限）。任务里的当前上限比它小：额度用完、超时多了，之后的段缩小了 */
export const DRAFT_PART_COLUMNS = 40
/** 模型拿不准、留空的列数（任务里的一张表、清单上的上次起草）；老服务端没有这个键时为 0 */
export const blankOf = (t: { blank_columns?: number } | null | undefined): number => Math.max(0, t?.blank_columns ?? 0)
/**
* 清单、表详情上「上次起草有 N 列模型拿不准、留空了」的 N：这张表的那次起草结束了才算（还在起草的那一次写到一半，
* 不说「上次」）
*/
export const lastBlankOf = (d: { status: CatalogDraftTaskStatus; blank_columns?: number } | null | undefined): number =>
(d && TASK_ENDED.has(d.status) ? blankOf(d) : 0)
/** 一次起草里留空的列合计：服务端给了合计就用它，老服务端没有就按逐表加 */
export const jobBlank = (j: CatalogDraftJob): number => (j.blank_columns != null ? blankOf(j) : j.tasks.reduce((n, t) => n + blankOf(t), 0))
/** 结束后默认列出的表：没写全的（失败、部分完成、已停止）和有留空列的 */
export const needsLook = (t: CatalogDraftTask): boolean => RETRYABLE.has(t.status) || blankOf(t) > 0
/** 回到设置时预先选好的范围（结果里「重新起草这几张表」、表详情里「补充起草要求」） */
export interface DraftPreset {
tables: string[]
/** 范围那一项的写法 */
label: string
}
/**
* 一项都没写进去的任务该怎么说（只对写完了的任务：完成、部分完成）。null：照常写合计。
* 用模型、却没有一张表拿到模型的可用内容：目录没有更新、模型没给内容；有表没写全：目录没有更新、原因见下方；
* 都写完了只是和现有目录一致：没有变化
*/
export function nothingWritten(j: CatalogDraftJob): 'model' | 'failed' | 'same' | null {
if (j.added + j.updated + j.removed > 0 || (j.status !== 'succeeded' && j.status !== 'partial')) return null
const live = j.tasks.filter((t) => t.status !== 'skipped')
if (j.use_model && live.length && !live.some((t) => (t.model_items ?? 0) > 0)) return 'model'
if (j.tasks.some((t) => RETRYABLE.has(t.status))) return 'failed'
return 'same'
}
export interface DraftStartBody { tables?: string[]; use_model: boolean; note?: string }
export function useDraftJob(sourceId: string, active: CatalogActiveDraftJob | null | undefined, on: {
/** 这一次取回来新结束的表（写进目录了，或者失败、停止了）：重取清单，正在看的表在里面就重新载入 */
onTablesEnded: (tables: string[]) => void
/** 看着它从排队、进行中变成结束（这一页上亲眼看到的，进页面时已经结束的不算） */
onFinished: (job: CatalogDraftJob) => void
}) {
// 任务挂在数据源上：换了数据源，上一个源的任务不再显示（那边照常进行）
const [held, setHeld] = useState<{ src: string; job: CatalogDraftJob } | null>(null)
const job = held?.src === sourceId ? held.job : null
const jobRef = useRef(job)
jobRef.current = job
const sourceRef = useRef(sourceId)
sourceRef.current = sourceId
const onRef = useRef(on)
onRef.current = on
/** 点了停止、服务端还没停下来的任务 */
const [stopping, setStopping] = useState<string | null>(null)
/** 取进度失败（网络抖了一下）、或取回来的是旧状态时，原样再排一拍 */
const [tick, setTick] = useState(0)
/** 换上取回来的状态。不往回走；返回是否换上了 */
const accept = useCallback((next: CatalogDraftJob, src: string): boolean => {
if (src !== sourceRef.current) return false
const cur = jobRef.current
const same = cur?.id === next.id
if (same && cur) {
if (!isJobActive(cur) && isJobActive(next)) return false
if (isJobActive(next) && next.done < cur.done) return false
}
// 新结束的表：同一个任务比上一次多结束的；头一回见到的进行中任务（别处建的、刷新页面前建的）已经结束的那些也算，
// 清单可能还没跟上。头一回见到的已结束任务（点开上一次的结果）不算，那些早就写进去了
const before = new Map((same && cur ? cur.tasks : []).map((t) => [t.table_name, t.status]))
const ended = same || isJobActive(next)
? next.tasks
.filter((t) => isTaskEnded(t) && t.status !== 'skipped' && !TASK_ENDED.has(before.get(t.table_name) ?? 'pending'))
.map((t) => t.table_name)
: []
jobRef.current = next
setHeld({ src, job: next })
if (ended.length) onRef.current.onTablesEnded(ended)
if (same && cur && isJobActive(cur) && !isJobActive(next)) onRef.current.onFinished(next)
return true
}, [])
/** 换成另一个任务（刚建的、重试建的、409 时进行中的那个、对话框里点开的上一次）。同一个任务照常只往前走 */
const track = useCallback((next: CatalogDraftJob) => {
accept(next, sourceRef.current)
}, [accept])
// 进行中：隔 1.5 秒取一次，结束就停
const jobId = job?.id
const jobActive = isJobActive(job)
useEffect(() => {
if (!jobId || !jobActive) return
const src = sourceId
let alive = true
const t = setTimeout(async () => {
try {
const { job: next } = await api.dataCatalog.draftJob(src, jobId)
if (alive && !accept(next, src)) setTick((n) => n + 1)
} catch (e) {
if (!alive) return
// 任务没了（不属于这个数据源、库被换掉了）：不再跟着它；别的失败（网络抖动、服务重启中）下一拍再取
if (e instanceof ApiError && e.status === 404) setHeld((h) => (h?.job.id === jobId ? null : h))
else setTick((n) => n + 1)
}
}, DRAFT_POLL_MS)
return () => {
alive = false
clearTimeout(t)
}
}, [sourceId, jobId, jobActive, job, tick, accept])
// 清单说有进行中的任务、这一页还不知道（别的标签页建的、刷新了页面）：取来跟着它
const activeId = isJobActive(active) ? active?.id : undefined
useEffect(() => {
if (!activeId) return
const cur = jobRef.current
if (cur?.id === activeId) return
const src = sourceId
let alive = true
api.dataCatalog.draftJob(src, activeId).then(({ job: next }) => { if (alive) accept(next, src) }, () => {})
return () => { alive = false }
}, [sourceId, activeId, accept])
/** 建任务。409 已有任务时改去取进行中的那一个（busy 为真）；其余的错误照原样抛给调用方 */
const start = useCallback(async (body: DraftStartBody): Promise<{ job: CatalogDraftJob; busy: boolean }> => {
const src = sourceRef.current
try {
const { job: next } = await api.dataCatalog.createDraftJob(src, body)
accept(next, src)
return { job: next, busy: false }
} catch (e) {
if (!(e instanceof ApiError && e.status === 409 && e.code === DRAFT_BUSY)) throw e
const { job: current } = await api.dataCatalog.activeDraftJob(src)
if (!current) throw e
accept(current, src)
return { job: current, busy: true }
}
}, [accept])
/**
* 只重试失败的表：同 start，409 已有任务时改去显示进行中的那一个；起草模型用不了（409）、没有可重试的表（422）等
* 照原样抛给调用方
*/
const retry = useCallback(async (from: CatalogDraftJob): Promise<{ job: CatalogDraftJob; busy: boolean }> => {
const src = sourceRef.current
try {
const { job: next } = await api.dataCatalog.retryDraftJob(src, from.id)
accept(next, src)
return { job: next, busy: false }
} catch (e) {
if (!(e instanceof ApiError && e.status === 409 && e.code === DRAFT_BUSY)) throw e
const { job: current } = await api.dataCatalog.activeDraftJob(src)
if (!current) throw e
accept(current, src)
return { job: current, busy: true }
}
}, [accept])
/** 停止。服务端停下正在进行的调用之前任务还是进行中：这期间 stopping 为这个任务。已经结束了（409）就取一次最新的 */
const cancel = useCallback(async () => {
const cur = jobRef.current
if (!cur || !isJobActive(cur)) return
const src = sourceRef.current
setStopping(cur.id)
try {
const { job: next } = await api.dataCatalog.cancelDraftJob(src, cur.id)
accept(next, src)
} catch (e) {
if (e instanceof ApiError && e.status === 409 && e.code === DRAFT_FINISHED) {
const { job: next } = await api.dataCatalog.draftJob(src, cur.id)
accept(next, src)
return
}
setStopping(null)
throw e
}
}, [accept])
return { job, track, start, retry, cancel, stopping: !!job && stopping === job.id && isJobActive(job) }
}
frontend/src/types.ts
+121 -14
View File
@@ -2790,6 +2790,8 @@ export interface CatalogTableRow {
version: number
updated_at: string | null
updated_by: string | null
/** 这张表最近一次起草任务里的那一项（任务还在进行就是进行中的状态）；没起草过为 null。老服务端没有这个键 */
last_draft?: CatalogLastDraft | null
}
/** GET /datasources/{id}/catalog */
@@ -2804,6 +2806,10 @@ export interface CatalogList {
schema_truncated?: boolean
/** 数据库里一共几张表（含视图）；没截断时等于探查到的张数 */
schema_total?: number
/** 这个数据源的起草要求（使用者补充给模型的背景和要求）；没填为 null。老服务端没有这一项 */
guidance?: string | null
/** 排队或进行中的起草任务（页面据此显示进度）；没有为 null。老服务端没有这一项 */
active_job?: CatalogActiveDraftJob | null
}
/** 表结构里的一列 */
@@ -2851,32 +2857,133 @@ export interface CatalogDetail {
/** 单项审阅：确认、驳回、撤销审阅（回到来源的初始状态；人工填写的项直接删掉） */
export type CatalogReviewAction = 'confirm' | 'reject' | 'reset'
/** 起草结果里的一张表 */
export interface CatalogDraftRow {
// ---------------------------------------------------------------------------
// 起草任务（/api/datasources/{id}/catalog/draft-jobs）：起草在服务端后台进行，逐表起草、每张表一结束就并入目录。
// 关掉对话框不停任务；服务重启时还没结束的任务记 interrupted，不自动续跑，由人点「重试」接着起草。
// ---------------------------------------------------------------------------
/**
* 起草任务的状态：排队、进行中、完成、部分完成（有失败、部分完成或停止的表）、失败（任务整体出错）、已停止（用户停止）、
* 服务重启中断
*/
export type CatalogDraftJobStatus = 'queued' | 'running' | 'succeeded' | 'partial' | 'failed' | 'cancelled' | 'interrupted'
/**
* 任务里一张表的状态。ok：写入成功（或没有东西要写），用模型时各段都写出来了；partial：用模型，有的段没写出来；
* failed：整张表没起草（error），或用模型时一段都没写出来（model_error），按注释、外键、命名起草的部分照常写入；
* skipped：表结构里没有这张表；cancelled：停止或服务重启时还没开始、没写完
*/
export type CatalogDraftTaskStatus = 'pending' | 'running' | 'ok' | 'partial' | 'failed' | 'skipped' | 'cancelled'
/** 起草任务里的一张表（按起草顺序） */
export interface CatalogDraftTask {
table_name: string
status: CatalogDraftTaskStatus
/** 列数 */
columns: number
/** 模型分几段起草（超过 40 列的宽表拆成表级一段加列分段）；不用模型时为 null */
parts_total: number | null
parts_done: number
added: number
updated: number
removed: number
version: number
/** 这张表没有起草（表结构里没有、写入一直冲突） */
error: string | null
/** 只是模型那部分失败，其余来源照常写入。模型给了这张表、却一项可用内容都没有，也记在这里 */
model_error: string | null
/**
* 模型给了这张表几项可用内容（并入目录之前）。没用模型、模型这部分失败时为 null（老后端没有这个键）。
* 模型给了这张表几项可用内容（并入目录之前）。没用模型、模型这部分失败时为 null。
* 大于 0 而新增、更新、删除都是 0，才是「模型给了内容、只是和现有目录一致」
*/
model_items?: number | null
model_items: number | null
/**
* 模型拿不准、留空的列数：模型列出了（或这一段请它写了），中文名和含义却都没给（给了却都没通过校验的也算）。
* 表的状态不因此变（照旧可以是已完成）。老服务端没有这个键
*/
blank_columns?: number
/** 留空的列里的前几个（最多 5 个，列名照表结构原样）。老服务端没有这个键 */
blank_sample?: string[]
/** 整张表没起草的原因：表结构里没有、写入一直冲突、服务重启中断 */
error: string | null
/** 模型那部分没写出来的原因和办法（第几段、哪些列、为什么） */
model_error: string | null
started_at: string | null
finished_at: string | null
}
/** POST /datasources/{id}/catalog/draft */
export interface CatalogDraftOut {
tables: CatalogDraftRow[]
total: { added: number; updated: number; removed: number }
model_used: boolean
/** 起草任务的模型用量 */
export interface CatalogDraftUsage {
/** 调用次数（没写出来的、超时的也算） */
calls: number
/** 超时的调用不计 token（照实记 0：服务方可能已经生成了内容） */
input_tokens: number
output_tokens: number
/** 超时的调用次数。老服务端没有这个键 */
timeouts?: number
/** 没写出来的调用次数：额度用完、只给空壳、超时、出错。老服务端没有这个键 */
failed_calls?: number
}
/** 一次起草任务 */
export interface CatalogDraftJob {
id: string
source_id: string
status: CatalogDraftJobStatus
use_model: boolean
/** 实际用的模型 id；不用模型、还没开始时为 null */
model: string | null
/** 模型整体用不了（没有配置、已停用……）：只按注释、外键和命名起草 */
/** 这次用到的起草要求（数据源级那份的快照） */
guidance: string | null
/** 这次的补充说明 */
note: string | null
created_at: string
started_at: string | null
finished_at: string | null
/** 谁建的（X-Actor） */
created_by: string | null
/** 重试建的任务记着原任务的 id，别的为 null。接口约定里没有这一项，服务端多给的；界面不用，老服务端没有这个键 */
retry_of?: string | null
/** 表数 */
total: number
/** 已结束的表数 */
done: number
counts: Record<CatalogDraftTaskStatus, number>
added: number
updated: number
removed: number
/** 各表模型拿不准、留空的列数合计。老服务端没有这个键 */
blank_columns?: number
/** 模型调用的用量；拿不到用量时照实是 0 */
usage: CatalogDraftUsage
/**
* 当前每段最多几列：一次起草里额度用完或超时累计 2 次后，之后新开始的段减半（40 → 20 → 10，最低 10）。
* 不用模型时可能为 null；老服务端没有这个键
*/
part_columns?: number | null
/**
* 调用接连没写出来时的提醒（一句中文：最近几次里几次没写出来、原因，建议换起草模型、缩小范围或停止）；
* 好转后服务端清掉，为 null。老服务端没有这个键
*/
notice?: string | null
/** 任务整体出错的原因 */
error: string | null
tasks: CatalogDraftTask[]
}
/** 目录清单顶上的进行中任务（GET /datasources/{id}/catalog 的 active_job） */
export interface CatalogActiveDraftJob {
id: string
status: CatalogDraftJobStatus
total: number
done: number
}
/** 清单一行上这张表最近一次起草的结果 */
export interface CatalogLastDraft {
status: CatalogDraftTaskStatus
job_id: string
finished_at: string | null
error: string | null
model_error: string | null
/** 那次起草模型拿不准、留空的列数（同 CatalogDraftTask.blank_columns）。老服务端没有这个键 */
blank_columns?: number
}
/**
scripts/check-all.mjs
+1 -1
View File
@@ -97,7 +97,7 @@ const ORDER = [
const ESTIMATE = {
challenge: 30,
stream: 173, chat: 160, studio: 149, manage: 145, runs: 112, publish: 92, shell: 87,
evidence: 91, versions: 60, catalog: 45, 'ui-kit': 70, ui: 61, guards: 8, 'canvas-layout': 2, copy: 2, tokens: 1, trace: 1, decode: 1,
evidence: 91, versions: 60, catalog: 140, 'ui-kit': 70, ui: 61, guards: 8, 'canvas-layout': 2, copy: 2, tokens: 1, trace: 1, decode: 1,
// 占位，换成真检查时按实测改
schedules: 60, compare: 45, verify: 71, inquiry: 60, bundle: 16, whatif: 33, 'report-export': 30,
}
scripts/check-catalog.mjs
+951 -114
View File
File diff suppressed because it is too large Load Diff
scripts/check-manage.mjs
+90
View File
@@ -589,6 +589,96 @@ await section('设置 · 门控模型', async () => {
await old.close()
})
await section('设置 · 起草模型', async () => {
// 数据目录「用模型起草」和按配方导入表格时的 AI 起草用的模型（catalog_draft 一组）：两项都留空跟随助手的模型；只填模型时
// 按模型名找接入；存着的接入停用了在那一格下面说；服务端按项拒收（422 带 field）时整句落到那一格下面。设置读写、接入表都是假的
const PROVIDERS = [
{ id: 'check-draft-p1', name: 'demo-main', kind: 'openai', base_url: 'https://example.com/v1', default_model: 'big-model',
models: [{ id: 'big-model' }], enabled: true, extra: {}, api_key_masked: '', has_key: true },
{ id: 'check-draft-p2', name: 'demo-wide', kind: 'openai', base_url: 'https://example.com/v1', default_model: 'wide-1',
models: [{ id: 'wide-1' }, { id: 'wide-2' }], enabled: true, extra: {}, api_key_masked: '', has_key: true },
{ id: 'check-draft-p3', name: 'demo-off', kind: 'openai', base_url: 'https://example.com/v1', default_model: 'off-model',
models: [{ id: 'off-model' }], enabled: false, extra: {}, api_key_masked: '', has_key: true },
]
let stored = await get('/settings')
stored = { ...stored, catalog_draft: { provider: null, model: null } }
// 服务端的说法（api/settings._provider_problem）：主语是这一项在设置页上的叫法
const REFUSED = '「起草模型 · 接入」指定的模型接入「demo-wide」已停用。请在「设置 → 模型接入」中启用它，或换一个接入'
let refuse = false
const { page, sent, errors, close } = await open('/settings/prefs', {
handlers: [
[/^GET \/settings$/, (route) => json(stored)(route)],
[/^PUT \/settings$/, (route, { body }) => {
if (refuse) {
refuse = false
return route.fulfill({ status: 422, contentType: 'application/json',
body: JSON.stringify({ detail: REFUSED, code: 'settings_provider_invalid', field: 'catalog_draft.provider' }) })
}
stored = { ...stored, ...body.values }
return json(stored)(route)
}],
[/^GET \/providers$/, json(PROVIDERS)],
],
})
const box = page.locator('[data-draft-model-settings]')
await box.waitFor()
const text = (await box.innerText()).replace(/\s+/g, ' ')
check('偏好设置里有「起草模型」一节：说明数据目录和导入表格的起草都用它，留空跟随助手的模型',
text.includes('起草模型') && text.includes('数据目录') && text.includes('导入表格') && text.includes('跟随助手的模型'), text.slice(0, 160))
const provider = page.getByLabel('起草模型 · 接入')
const model = page.getByLabel('起草模型 · 模型')
check('……接入留空写「跟随助手的模型」，模型的占位也写明', (await provider.locator('option').first().innerText()) === '跟随助手的模型'
&& (await model.getAttribute('placeholder')) === '留空：跟随助手的模型', await provider.locator('option').first().innerText())
check('……停用的接入列着但选不了', await provider.locator('option[value="demo-off"]').evaluate((o) => o.disabled))
// 只填模型：按模型名找接入，写明用的是哪家
await model.fill('wide-2')
check('只填模型：写明按模型名使用哪个接入', (await box.innerText()).includes('将按模型名使用接入「demo-wide」'), (await box.innerText()).replace(/\s+/g, ' ').slice(0, 200))
check('……接入那一格的空选项改叫「按模型名找接入」', (await provider.locator('option').first().innerText()) === '按模型名找接入')
await provider.selectOption('demo-wide')
const bar = (await page.locator('[data-prefs-bar]').innerText().catch(() => '')).replace(/\s+/g, ' ')
check('……改了两项：保存条写「起草模型 · 接入、起草模型 · 模型」', bar.includes('有 2 项未保存') && bar.includes('起草模型 · 接入')
&& bar.includes('起草模型 · 模型'), bar)
await page.getByRole('button', { name: /保存设置/ }).click()
await page.waitForTimeout(500)
const put = sent.filter((s) => s.key === 'PUT /settings').at(-1)
check('保存只 PUT catalog_draft 一组：接入、模型', Object.keys(put?.body?.values ?? {}).join() === 'catalog_draft'
&& JSON.stringify(put.body.values.catalog_draft) === JSON.stringify({ provider: 'demo-wide', model: 'wide-2' }), JSON.stringify(put?.body?.values))
await shot(page, 'prefs-draft-model')
// 换接入：原来填的模型不在新接入里就清掉；都清空存成 null（跟随助手的模型）
await provider.selectOption('demo-main')
check('换了接入，旧接入的模型清掉', await model.inputValue() === '', await model.inputValue())
await provider.selectOption('')
await page.getByRole('button', { name: /保存设置/ }).click()
await page.waitForTimeout(500)
const put2 = sent.filter((s) => s.key === 'PUT /settings').at(-1)?.body?.values?.catalog_draft
check('……两项都留空：存成 null', JSON.stringify(put2) === JSON.stringify({ provider: null, model: null }), JSON.stringify(put2))
// 服务端按项拒收：整句落到接入那一格下面，焦点移过去
refuse = true
await provider.selectOption('demo-wide')
await page.getByRole('button', { name: /保存设置/ }).click()
await page.waitForTimeout(500)
const said = await page.locator('#pref-draft-provider-error').innerText().catch(() => '')
check('服务端按项拒收（422 带 field）：整句落到「起草模型 · 接入」下面', said === REFUSED, said)
check('……焦点移到那一格', await page.locator('#pref-draft-provider').evaluate((el) => el === document.activeElement))
check('没有运行时报错', errors.length === 0, errors[0] ?? '')
await close()
// 存着的接入后来停用了：那一格下面说清起草会怎样
const disabled = { ...stored, catalog_draft: { provider: 'demo-off', model: null } }
const off = await open('/settings/prefs', { handlers: [[/^GET \/settings$/, (route) => json(disabled)(route)], [/^GET \/providers$/, json(PROVIDERS)]] })
await off.page.locator('[data-draft-model-settings]').waitFor().catch(() => {})
const offSaid = await off.page.locator('#pref-draft-provider-error').innerText().catch(() => '')
check('存着的接入已停用：那一格下面说清用模型起草会失败', offSaid.includes('模型接入「demo-off」已停用') && offSaid.includes('起草'), offSaid)
check('没有运行时报错', off.errors.length === 0, off.errors[0] ?? '')
await off.close()
// 老服务端：设置里没有这一组，不显示这一节
const legacy = Object.fromEntries(Object.entries(stored).filter(([k]) => k !== 'catalog_draft'))
const old = await open('/settings/prefs', { handlers: [[/^GET \/settings$/, (route) => json(legacy)(route)], [/^GET \/providers$/, json(PROVIDERS)]] })
check('老服务端没有这一组：不显示「起草模型」，也没有保存条', await old.page.locator('[data-draft-model-settings]').count() === 0
&& await old.page.locator('[data-prefs-bar]').count() === 0)
await old.close()
})
await section('设置 · 点名的接入已停用或不存在', async () => {
// 点名停用的接入以前照常调用，现在运行时照实报错（K1）：存着的值后来被停用了，那一格下面要说；别的停用接入列着但选不了；
// 服务端按项拒收（422 带 field）时整句落到那一格下面、焦点移过去，保存条上原因不截断。接入表、设置读写都是假的