合入下一轮第 1 波(docs/design/next-round):评测底座(后台任务运行时上移、系统运行不进使用统计、全站唯一的问题聚类、目录视图接缝、问数据生成过程 answer_ops 与用量计量、问数据的决定下沉后端、每日账泛化、定时目标底座、通知事件登记表);已知遗留(人工驳回记为被驳回不算失败、个人渠道「我发起的实例」生效、导入清单如实记认证、使用者选人按候选池核对、自由形蓝图「谁来做」、tick 按 id 断言);目录方言与 Excel 期 5(剖析方言修正与金样、本机四库容器集成测试、列表按期累积、穿插小计、表格对象起草、CSV 走配方);集成底座(安全响应头、对外访问地址只留一个、开登录时收起 /docs、集成接口错误码)。验收:后端全量 13463 通过;tsc、文案检查;本波相关的 15 项浏览器检查 4939 项全过;四种数据库容器集成测试全过;泄露扫描干净

This commit is contained in:
ThearchyHelios
2026-10-16 13:01:47 +08:00
229 changed files with 13097 additions and 1415 deletions
+24 -3
View File
@@ -90,6 +90,21 @@ http://127.0.0.1:8000 。前端产物不存在或比源码旧时才重新构建
MCP 要管理员先在「工具 → 端点」登记。
- `/api/copilot/generate` 改图时只出改动操作,不再是整张新图;每次运行多一条 `run.origin` 事件(在封存范围内)。
**从旧版本升级(2026-10 下一轮第 1 波)**:有几处行为变化会影响现有部署:
- 人工驳回(审批、动作确认、填写、代码和工具审批、双盲复算)的运行记为「已取消 · 被驳回」,不再算失败:通知走「实例
结束」,定时任务推「未出具(被驳回)」,不能继续运行。配了失败重试或「出错时继续」的关卡,驳回不再重问、也不接着往下走。
已封存的老运行不改写。
- 开了登录时,应用入口和待办表单里的选人控件按候选池核对(缺省是能用这个应用的人),池外的名字回 422(`principal`)。
- 页面默认只能被本站嵌入(`frame-ancestors 'self'`,另加 `X-Frame-Options`)。已经用 iframe 嵌 AgentLab 的门户,
在 `AGENTLAB_FRAME_ANCESTORS` 里写上门户的地址。
- 「对外访问地址」只剩一个设置(「设置 → 通用」;`AGENTLAB_PUBLIC_URL` 优先,写错了启动就报错)。设置里填了地址的部署,
OIDC 回调改按这个地址拼,和身份提供方登记的不一致时要去改登记。开了登录时 `/docs`、`/openapi.json` 要搭建者登录。
- 探查结构时列类型改存各数据库的完整写法(`VARCHAR2(20 CHAR)`、`ENUM('a','b')`):升级后每个数据源点一次「探查结构」。
新运行的表结构快照哈希会变,旧运行不受影响。MySQL / MariaDB 上文字列的剖析说明多一句排序规则,ENUM 列补上数据里没出现的
取值,日期列写明排除了几个无效日期,所以升级后第一次重新剖析,这些表的目录各升一次版本。剖析设置的「抽样键值数」改叫
「核对键值数」(`DISTINCT … LIMIT` 取的不是随机样本)。
---
## 核心能力
@@ -103,10 +118,10 @@ http://127.0.0.1:8000 。前端产物不存在或比源码旧时才重新构建
| **能力登记与端点** | 所有可调用的工具登记在一处,逐项治理读写效果、审批、字段语义、遮罩;外部系统按测试 / 生产两个环境登记成端点,只有登记的端点能连内网,认证头加密;端点读回的表格和查询快照一样能作证([详见](#能力登记与端点)) |
| **Agent 护栏** | 同样的调用不重复执行、连续拿不到新信息就收尾、令牌与金额预算、上下文接近窗口先压缩再收尾;步数只留兜底(默认 100) |
| **代码沙箱** | 双档隔离:系统沙箱(Seatbelt / bubblewrap,冷启动约 25ms)或 microVM(独立 Linux 内核,内存限额真实生效) |
| **人工介入** | 执行暂停并落盘,人工批准 / 驳回 / 改稿后从断点继续;进程重启不丢状态;可派给岗位、发起人或按表达式算出的人,会签(任一人 / 全体同意,一票否决 / 至少 m 人),设时限,超时提醒、升级或驳回,永远不会自动批准 |
| **人工介入** | 执行暂停并落盘,人工批准 / 驳回 / 改稿后从断点继续;进程重启不丢状态;可派给岗位、发起人或按表达式算出的人,会签(任一人 / 全体同意,一票否决 / 至少 m 人),设时限,超时提醒、升级或驳回,永远不会自动批准;驳回是人的决定,运行记为「被驳回」,不算失败、不重试 |
| **断点续跑** | 失败的运行可从失败节点继续,已完成的节点不重复执行;探索运行允许修改节点配置(拓扑必须一致),正式运行固定在发布的版本上、不能改配置后继续;继续运行不改发起人 |
| **知识库** | 向量 + BM25 混合检索、倒排索引、可选模型重排;PDF / Word / PowerPoint / HTML / Markdown 解析;向量模型可接本地端点 |
| **表格入库** | Excel / CSV 上传后转为可用 SQL 查询的表,自动生成 `db_query__<name>` 工具——数字由 SQL 计算得出,而非模型读取 |
| **表格入库** | Excel / CSV 上传后转为可用 SQL 查询的表,自动生成 `db_query__<name>` 工具——数字由 SQL 计算得出,而非模型读取;复杂报表按取数配方导入(CSV 同样可以),每月一份的明细能按期累积成一张表,夹在明细中间的小计核对后不进数据 |
| **数据目录** | 给数据源的每张表记下粒度、业务主键、业务日期、有效记录条件,每个字段的含义、单位、度量类型和码值,以及表与表的关联;由外键、命名推断、数据剖析和模型起草,逐项确认后生效;表详情里可以和助手对话完善一张表,审阅队列按影响排好先审哪些,目录能导出、导入到别的实例;助手建图、agent 查结构、SQL 检查都读它 |
| **SQL 检查** | 按数据目录检查一对多关联后重复计算、存量跨时间相加、比率被相加等七类问题;助手自查时交回修正,受管模板发布时挡住,运行时标在查询结果上、契约指标降档 |
| **跨库合并** | 合并查询节点:各库先聚合到同一个键,再在库外按键合并;截断的输入拒收,证据一路追到两边的原始查询 |
@@ -115,7 +130,7 @@ http://127.0.0.1:8000 。前端产物不存在或比源码旧时才重新构建
| **自然语言编排** | 助手:一句话生成或改写整张图,自动排版;system 里每个数据源只给一行概览,要看表结构、能力参数、工作流版本时由助手自己「查看」,一次给齐字段、目录和连接条件;交付前按运行时同一套规则自查,查出会运行失败的问题先交回去改,改不好就不自动运行([详见](#助手按需查看)) |
| **结构化成果** | JSON Schema 校验,不合格时将错误回传模型自动返工 |
| **产出复核** | 运行结束后规则扫描事件流,发现异常才调用模型复核;复核可重组答案但**不得引入原答案中不存在的数字** |
| **追踪与成本** | 全量事件落库可回放;逐节点耗时、token 用量、按目录价折算的成本 |
| **追踪与成本** | 全量事件落库可回放;逐节点耗时、token 用量、按目录价折算的成本;问数据每一轮的花费含助手建图、挑表和复核 |
| **三档出具** | 口径卡(受控指标集)+ 出具契约:叙述中每个数字回指指标集,或由报告撰写节点按引用逐段核对,formal / degraded / withheld 三档判定 |
| **可点击证据** | 报告撰写节点只写引用标记,数字和表格由系统照快照填值;数字、表名字段名、知识库原话都点得开,一路指到查询结果里的那一格;结论句可要求挂依据或交给另一个模型裁判;证据清单可导出,报告导出为自包含 HTML(数字点得开口径与出处),另有封存哈希核对件「可验证报告」 |
| **发布前检查** | 受管级别加证据门禁 G1–G5;问题给出一键修复 / 选项 / 交给 Copilot 三类修法,一律先预览、不许降低要求;老工作流一键升级为可追溯结构 |
@@ -738,6 +753,12 @@ Excel / CSV 若传入知识库,只能被切块检索,数字即成为模型
- 内网只对登记的端点开放(管理员开「允许访问内网」),元数据地址和高危端口照样拦;模型给出的地址照旧做完整的 SSRF 检查
- 开了登录:会话 cookie 在库里只存 token 的哈希;非 GET 请求核对来源标记头和 Origin;使用者只能读有授权的运行,看到的查询结果一律遮罩
- 写外部系统必须经「调用工具」节点:先留生效记录再发送,确认卡上的内容哈希在发送前重算比对;结果未知永不自动重发
- 安全响应头:`nosniff`、`Referrer-Policy`,以及 `frame-ancestors`(缺省 `'self'`,`AGENTLAB_FRAME_ANCESTORS` 列出允许
嵌整站的同站门户;只有 `'self'` 时另加 `X-Frame-Options: SAMEORIGIN`)。严格的 `script-src` 目前只发
`Content-Security-Policy-Report-Only`,观察一个版本没有误报再强制。HSTS 交给反向代理
- 集成方要用的接口(应用、待办、工件、角色拒绝)的错误一律带机读码(`code`),`detail` 是给人看的原话,可以改
- 评测、口径对账、口径召回是系统代为发起的运行:证据照常可查,但不进使用统计(探索问题聚类、工具白名单、目录使用
次数),记录页默认不列并写明有几次没列出
---
+26 -19
View File
@@ -12,14 +12,18 @@ import dataclasses
import json
from typing import Any
from fastapi import APIRouter, Depends, Header, HTTPException, Query
from fastapi import APIRouter, Depends, Header, Query
from fastapi.responses import JSONResponse
from pydantic import BaseModel, Field
from sqlalchemy import func, select
from sqlalchemy.exc import IntegrityError
from sqlalchemy.ext.asyncio import AsyncSession
from app.api.coded import CodedHTTPException
from app.api.coded import (
APP_BINDINGS_INVALID, APP_DRAFT_INVALID, APP_GRANTS_INVALID, APP_HOOK_NOT_DECLARED, APP_HOOK_NOT_FOUND,
APP_INSTANCE_NOT_FOUND, APP_NOT_FOUND, APP_PARAMS_INVALID, APP_VERSION_NOT_FOUND, FORM_INVALID, RUN_LAUNCH_REFUSED,
CodedHTTPException,
)
from app.api.deps import Principal, actor_of, current_principal
from app.apps.access import ROLE_TEXT, app_role, at_least, grants_of
from app.db.base import get_session
@@ -60,11 +64,11 @@ def _principal(x_actor: str | None) -> Principal:
async def _app(session: AsyncSession, app_id: str, principal: Principal, need: str) -> tuple[App, str]:
app = await session.get(App, app_id)
if app is None:
raise HTTPException(404, APP_GONE)
raise CodedHTTPException(404, APP_GONE, APP_NOT_FOUND)
role = await app_role(session, principal, app)
if role is None:
# 没有任何授权的人不该知道这个应用在不在
raise HTTPException(404, APP_GONE)
# 没有任何授权的人不该知道这个应用在不在:码和不存在的一样
raise CodedHTTPException(404, APP_GONE, APP_NOT_FOUND)
if not at_least(role, need):
raise CodedHTTPException(403, f"需要这个应用的「{ROLE_TEXT[need]}」权限", NO_ACCESS)
return app, role
@@ -314,7 +318,7 @@ async def derive(payload: DeriveIn, session: AsyncSession = Depends(get_session)
try:
return {"spec": await derive_app(session, payload.workflow_id, payload.version, name=payload.name)}
except DeriveRefused as e:
raise HTTPException(e.status_code, e.message) from e
raise CodedHTTPException(e.status_code, e.message, e.code) from e
@router.get("/{app_id}")
@@ -359,13 +363,13 @@ async def update_app(app_id: str, payload: AppPatch, session: AsyncSession = Dep
app, role = await _app(session, app_id, principal, "manage")
_refuse_archived(app, "修改")
if payload.draft_bindings is not None and (problem := _bindings_problem(payload.draft_bindings)):
raise HTTPException(422, problem)
raise CodedHTTPException(422, problem, APP_BINDINGS_INVALID)
if app.scope == "personal" and payload.draft is not None:
# 个人应用的草稿就是在用的那一份:只许改呈现,固定的蓝图、视图来源只由「存成我的应用」改
from app.blueprints.personal import draft_problem
if problem := draft_problem(app.draft or {}, payload.draft):
raise HTTPException(422, problem)
raise CodedHTTPException(422, problem, APP_DRAFT_INVALID)
for key in ("name", "description", "icon"):
if (value := getattr(payload, key)) is not None:
setattr(app, key, value)
@@ -568,7 +572,7 @@ async def get_version(app_id: str, version: int, session: AsyncSession = Depends
row = (await session.execute(select(AppVersion).where(
AppVersion.app_id == app.id, AppVersion.version == version))).scalar_one_or_none()
if row is None:
raise HTTPException(404, f"「{app.name}」没有 v{version}")
raise CodedHTTPException(404, f"「{app.name}」没有 v{version}", APP_VERSION_NOT_FOUND)
# 岗位绑定是这一个部署里的人和组:管理者才看得到
return {**_version_row(row), "spec": row.spec, "bindings": row.bindings if role == "manage" else {"slots": {}}}
@@ -611,7 +615,7 @@ async def put_grants(app_id: str, payload: GrantsIn, session: AsyncSession = Dep
_refuse_personal(app, "授权给别人")
wanted, problem = await grants_problem(session, [(i.principal_kind, i.principal, i.role) for i in payload.items])
if problem:
raise HTTPException(422, problem)
raise CodedHTTPException(422, problem, APP_GRANTS_INVALID)
return {"items": await replace_grants(session, app, wanted, actor=actor_of(x_actor))}
@@ -713,7 +717,8 @@ async def rotate_hook(app_id: str, name: str, session: AsyncSession = Depends(ge
draft, published, rows = await _hook_state(session, app)
declared = draft.get(name) or published.get(name)
if declared is None:
raise HTTPException(404, f"草稿和当前发布版里都没有声明钩子「{name}」,请先在「钩子」一节里声明它")
raise CodedHTTPException(404, f"草稿和当前发布版里都没有声明钩子「{name}」,请先在「钩子」一节里声明它",
APP_HOOK_NOT_DECLARED)
# 从读行锁到提交:两位管理者同时点「生成密钥」,后到的那次读得到前一次插入的行,只轮换它(见 process/hooks.py)
async with hooks.hook_lock(app.id, name):
row, secret, created = await hooks.rotate_hook(session, app.id, name, kind=declared["kind"],
@@ -731,7 +736,7 @@ async def _hook_named(session: AsyncSession, app: App, name: str) -> AppHook:
row = await hook_row(session, app.id, name)
if row is None:
raise HTTPException(404, f"钩子「{name}」还没有生成密钥")
raise CodedHTTPException(404, f"钩子「{name}」还没有生成密钥", APP_HOOK_NOT_FOUND)
return row
@@ -778,9 +783,9 @@ def _params(raw: str | None) -> dict[str, Any]:
try:
value = json.loads(raw)
except ValueError as e:
raise HTTPException(422, "页面参数需要是 JSON 对象") from e
raise CodedHTTPException(422, "页面参数需要是 JSON 对象", APP_PARAMS_INVALID) from e
if not isinstance(value, dict):
raise HTTPException(422, "页面参数需要是 JSON 对象")
raise CodedHTTPException(422, "页面参数需要是 JSON 对象", APP_PARAMS_INVALID)
return value
@@ -788,14 +793,16 @@ async def _instance(session: AsyncSession, app: App, run_id: str, principal: Pri
"""实例要属于这个应用;只有 use 的人只能看自己发起的(关登录时都能看)。"""
run = await session.get(Run, run_id)
if run is None or run.app_id != app.id or run.origin_kind not in ("app_entry", "hook"):
raise HTTPException(404, "实例不存在")
raise CodedHTTPException(404, "实例不存在", APP_INSTANCE_NOT_FOUND)
if principal.mode != "off" and not at_least(role, "observe") and run.started_by != principal.username:
raise CodedHTTPException(403, "只能查看自己发起的实例", NO_ACCESS)
return run
def _view_refused(e: Any) -> HTTPException:
return CodedHTTPException(e.status_code, e.message, e.code) if e.code else HTTPException(e.status_code, e.message)
def _view_refused(e: Any) -> CodedHTTPException:
"""视图、入口、补算、发起运行的拒绝 → 带 code 的 HTTP 错误。只有引擎不发起运行(LaunchRefused)有不带码的几种(工作流
没发布、版本不存在、图没通过校验……),补上同一类的 run_launch_refused,detail 照旧说清是哪一种。"""
return CodedHTTPException(e.status_code, e.message, e.code or RUN_LAUNCH_REFUSED)
@router.get("/{app_id}/pages/{page_id}")
@@ -834,7 +841,7 @@ async def start(app_id: str, entry_id: str, payload: StartIn, session: AsyncSess
try:
run = await start_entry(app.id, entry_id, payload.values, principal)
except FormRejected as e:
return JSONResponse(status_code=422, content={"detail": "表单没有通过校验", "errors": e.errors})
return JSONResponse(status_code=422, content={"detail": "表单没有通过校验", "code": FORM_INVALID, "errors": e.errors})
except (EntryRefused, LaunchRefused) as e:
raise _view_refused(e) from e
return {"run_id": run.id, "title": instance_title(run)}
@@ -905,7 +912,7 @@ async def refresh_view(app_id: str, view_id: str, payload: RefreshIn, session: A
run, cached = await run_view(session, app=app, version=version, view_id=view_id, params=payload.params,
principal=principal, actor=actor_of(x_actor))
except FormRejected as e:
return JSONResponse(status_code=422, content={"detail": "参数没有通过校验", "errors": e.errors})
return JSONResponse(status_code=422, content={"detail": "参数没有通过校验", "code": FORM_INVALID, "errors": e.errors})
except (ViewRefused, LaunchRefused) as e:
raise _view_refused(e) from e
return {"run_id": run.id, "cached": cached}
+9 -6
View File
@@ -2,9 +2,12 @@ from __future__ import annotations
from typing import Any
from fastapi import APIRouter, HTTPException
from fastapi import APIRouter
from sqlalchemy import select
from app.api.coded import (
ARTIFACT_FORBIDDEN, ARTIFACT_KIND_FORBIDDEN, ARTIFACT_NOT_FOUND, ARTIFACT_TAMPERED, CodedHTTPException,
)
from app.api.deps import can_read_artifact, current_principal
from app.core import artifact_store
@@ -23,14 +26,14 @@ async def get_artifact(artifact_id: str) -> dict[str, Any]:
principal = current_principal()
restricted = principal.mode != "off" and not principal.at_least("builder")
if principal.mode != "off" and not await can_read_artifact(principal, artifact_id):
raise HTTPException(403, "没有权限查看这件工件")
raise CodedHTTPException(403, "没有权限查看这件工件", ARTIFACT_FORBIDDEN)
try:
content = artifact_store.load(artifact_id)
except ValueError as e:
# 哈希对不上:明确报给调用方,这正是工件库存在的意义
raise HTTPException(409, str(e)) from e
raise CodedHTTPException(409, str(e), ARTIFACT_TAMPERED) from e
if content is None:
raise HTTPException(404, "工件不存在,可能已被删除")
raise CodedHTTPException(404, "工件不存在,可能已被删除", ARTIFACT_NOT_FOUND)
if restricted:
content = await _masked_for_user(principal, artifact_id, content)
return {"id": artifact_id, "content": content}
@@ -68,7 +71,7 @@ async def _masked_for_user(principal: Any, artifact_id: str, content: Any) -> An
return await _masked_metrics(principal, artifact_id, content, rows)
if _RECEIPT in kinds and isinstance(content, dict):
return await _masked_receipt(content, rows)
raise HTTPException(403, "这类工件的原文只对搭建者及以上的角色开放")
raise CodedHTTPException(403, "这类工件的原文只对搭建者及以上的角色开放", ARTIFACT_KIND_FORBIDDEN)
async def _masked_snapshot(content: dict[str, Any], *, endpoint: bool) -> dict[str, Any]:
@@ -106,7 +109,7 @@ async def _masked_metrics(principal: Any, artifact_id: str, card: dict[str, Any]
owner = (run_id, node_id or "")
break
if owner is None:
raise HTTPException(403, "没有权限查看这件工件")
raise CodedHTTPException(403, "没有权限查看这件工件", ARTIFACT_FORBIDDEN)
sealed = await _sealed(owner[0])
masks = _Masks(sealed)
metrics: list[Any] = []
+6 -3
View File
@@ -17,6 +17,7 @@ from pydantic import BaseModel, Field
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.api.coded import AUTH_REQUIRED
from app.api.deps import Principal, current_principal, decode_actor, ensure_role
from app.auth import (
MODES, SETTING_KEY, current_mode, env_mode, load_settings, remember_settings, stored_settings, user_can_chat,
@@ -25,7 +26,6 @@ from app.auth import oidc, sessions
from app.auth.audit import record
from app.auth.passwords import hash_password, password_problem, verify_password
from app.core import crypto
from app.core.config import settings
from app.db.base import get_session
from app.db.models import Setting, User
@@ -60,7 +60,7 @@ async def me(x_actor: str | None = Header(default=None), session: AsyncSession =
return await principal_full(session, Principal(username=actor, display_name=actor, role="admin", mode="off"))
if not principal.username:
can_bootstrap = principal.mode == "local" and not await _local_admin_exists(session)
return JSONResponse(status_code=401, content={"detail": "请先登录", "code": "auth_required",
return JSONResponse(status_code=401, content={"detail": "请先登录", "code": AUTH_REQUIRED,
"mode": principal.mode, "can_bootstrap": can_bootstrap})
return await principal_full(session, principal)
@@ -113,7 +113,10 @@ async def logout(request: Request, session: AsyncSession = Depends(get_session))
def _redirect_uri(request: Request) -> str:
base = settings.public_url.rstrip("/") or str(request.base_url).rstrip("/")
"""回调地址:对外访问地址(环境变量优先,其次设置,core/public_url.py),都没有按请求自己的地址拼。"""
from app.core.public_url import public_base_url
base = public_base_url() or str(request.base_url).rstrip("/")
return f"{base}/api/auth/oidc/callback"
+2 -1
View File
@@ -53,7 +53,8 @@ async def export_bundle(run_id: str) -> Response:
raise CodedHTTPException(404, "运行记录不存在", RUN_NOT_FOUND)
info = {"id": run.id, "workflow_id": run.workflow_id, "workflow_name": run.workflow_name,
"version": run.version, "version_hash": run.version_hash, "run_class": run.run_class,
"status": run.status, "started_at": run.started_at, "finished_at": run.finished_at}
"status": run.status, "end_reason": run.end_reason, "started_at": run.started_at,
"finished_at": run.finished_at}
# 封存之后追加的事件(按需裁判、审批质询):_sealed 只留了封存范围内的,这几条另取,单列在页面上
later = [tuple(r) for r in await session.execute(
select(RunEvent.seq, RunEvent.type, RunEvent.node_id, RunEvent.data)
+49
View File
@@ -29,6 +29,55 @@ CATALOG_DRAFT_BUSY = "catalog_draft_busy"
CATALOG_DRAFT_MODEL_UNAVAILABLE = "catalog_draft_model_unavailable"
CATALOG_DRAFT_FINISHED = "catalog_draft_finished"
# ---------------------------------------------------------------- 集成接口(04 细案 A7)
# 外部系统用访问密钥调的应用、待办、工件接口,以及角色、运行读权限的拒绝:按 code 认是哪种错,detail 照旧是给人看的那句
#: 身份:没登录(401,中间件、auth.require 都回它)、角色不够(403,路由级依赖、接口内部的 ensure_role、开登录时的 /docs)
AUTH_REQUIRED = "auth_required"
ROLE_REQUIRED = "role_required"
#: 运行:这次运行读不了(403,api/deps 的证据、运行接口);别人的个人应用刷新、个人蓝图试运行只给拥有者(403)
RUN_FORBIDDEN = "run_forbidden"
RUN_PRIVATE = "run_private"
#: 问数据和助手:使用者没开问数据,或者这个接口不在问数据的范围里(403)
CHAT_FORBIDDEN = "chat_forbidden"
#: 应用(api/apps.py)。app_not_found 同入口发起时的 apps/entries.APP_GONE;没有任何授权的人也回它(不该知道应用在不在)
APP_NOT_FOUND = "app_not_found"
APP_VERSION_NOT_FOUND = "app_version_not_found"
APP_INSTANCE_NOT_FOUND = "app_instance_not_found"
#: 草稿里、当前发布版里都没有声明这个钩子(404);声明了、还没生成密钥(404)
APP_HOOK_NOT_DECLARED = "app_hook_not_declared"
APP_HOOK_NOT_FOUND = "app_hook_not_found"
#: 改草稿、授权时写进来的不合规(422):岗位绑定、个人应用的草稿(只许改呈现)、授权的人和组
APP_BINDINGS_INVALID = "app_bindings_invalid"
APP_DRAFT_INVALID = "app_draft_invalid"
APP_GRANTS_INVALID = "app_grants_invalid"
#: 页面参数不是 JSON 对象(422)
APP_PARAMS_INVALID = "app_params_invalid"
#: 入口表单、按需视图的参数没通过校验(422,另带 errors:[{path, code, message}])。同待办回复、入站钩子的表单错误
FORM_INVALID = "form_invalid"
#: 发起实例、刷新视图、补算时引擎不发起运行,又没有更细的码(engine/launch.LaunchRefused 没带 code 的那几种)
RUN_LAUNCH_REFUSED = "run_launch_refused"
# 派生应用草稿时工作流、版本不存在(404)的码在 apps/derive.py(WORKFLOW_GONE、VERSION_GONE);入口、视图、补算的码
# 在 apps/entries.py、apps/resolve.py、apps/backfill.py,同这里一样都是 app_ 开头
#: 待办(api/tasks.py)。服务层(process/tasks.py 的 TaskRefused)没带码的拒绝按状态补上同一类的码:404 不存在;403
#: 不能这样做(不是管理员也不是应用的管理者,不能改派、催办);409 待办已经处理过、关闭了、刚被别人回复,或者所属的
#: 运行不再等这条回复——刷新看最新状态;422 回复的内容不对。带了码的照用自己的(task_not_assigned、form_invalid……)
TASK_NOT_FOUND = "task_not_found"
TASK_FORBIDDEN = "task_forbidden"
TASK_CLOSED = "task_closed"
TASK_INVALID = "task_invalid"
#: 工件(api/artifacts.py):没有权限(403)、这类工件的原文只给搭建者(403)、不存在(404)、内容和哈希对不上(409)
ARTIFACT_FORBIDDEN = "artifact_forbidden"
ARTIFACT_KIND_FORBIDDEN = "artifact_kind_forbidden"
ARTIFACT_NOT_FOUND = "artifact_not_found"
ARTIFACT_TAMPERED = "artifact_tampered"
#: 设置(api/settings.py):对外访问地址由环境变量 AGENTLAB_PUBLIC_URL 指定时,设置里那一项不能改成别的值(409)
PUBLIC_URL_ENV_LOCKED = "public_url_env_locked"
class CodedHTTPException(HTTPException):
"""field:出错的是哪一项(表单据此把报错落到那一格)。可选,没有就不出现在返回里。"""
+21 -9
View File
@@ -118,7 +118,7 @@ class ConversationOut(BaseModel):
turn_count: int = 0
#: 列表里给个副标题,让人一眼认出是哪次聊天
last_question: str = ""
#: 最后一轮停在哪:running | waiting | error | cancelled | suspended | done,
#: 最后一轮停在哪:running | waiting | error | cancelled | rejected | suspended | done,
#: 没有轮次时为 None。见 last_status()
last_status: str | None = None
#: 最后一轮的运行。建图阶段就断了的没有
@@ -173,18 +173,24 @@ BUILD_STALE = timedelta(minutes=15)
def last_status(
turn: Any, run_status: str | None, pending: bool, now: datetime | None = None,
turn: Any, run_status: str | None, pending: bool, now: datetime | None = None, end_reason: str | None = None,
) -> str:
"""一轮停在哪。轮次状态只有 running / done / error 三种,细分要看运行和元数据。
轮次状态是前端写的:页面关掉、服务重启时它停在 running,这时要以运行的状态为准;
运行也没有的,是断在了建图阶段。已经收尾的轮次以轮次自己为准——它写下的就是
用户看到的结局(取消、服务重启挂起在 error 里分开记)。
被审批人驳回结束的(运行 cancelled + end_reason=rejected,05-已知遗留 (a))单独记 rejected,不和手动停止混成
cancelled:前端落库时 meta 里记着 rejected(outcome 照旧是 cancelled),页面关掉没落库的看运行的 end_reason。
"""
outcome = (turn_meta(turn) or {}).get("outcome")
meta = turn_meta(turn) or {}
outcome = meta.get("outcome")
if turn.status == "done":
return "done"
if turn.status == "error":
if outcome == "cancelled" and meta.get("rejected"):
return "rejected"
if outcome == "cancelled" or (turn.error or "").strip() == "已取消":
return "cancelled"
if outcome == "suspended":
@@ -192,7 +198,7 @@ def last_status(
live = _from_run(run_status, pending)
return live if live in ("running", "waiting") else "suspended"
return "error"
live = _from_run(run_status, pending)
live = _from_run(run_status, pending, end_reason)
if live:
return live
updated = turn.updated_at
@@ -202,7 +208,9 @@ def last_status(
return "error" if stale else "running"
def _from_run(run_status: str | None, pending: bool) -> str | None:
def _from_run(run_status: str | None, pending: bool, end_reason: str | None = None) -> str | None:
if run_status == "cancelled" and end_reason == "rejected":
return "rejected"
return {
"queued": "running", "running": "running",
"interrupted": "waiting" if pending else "suspended",
@@ -236,11 +244,14 @@ async def _statuses(
run_of = {cid: turn_run_id(t) for cid, t in turns.items()}
run_ids = {r for r in run_of.values() if r}
runs: dict[str, str] = {}
ended: dict[str, str | None] = {}
waiting: set[str] = set()
if run_ids:
runs = dict((await session.execute(
select(Run.id, Run.status).where(Run.id.in_(run_ids))
)).all())
rows = (await session.execute(
select(Run.id, Run.status, Run.end_reason).where(Run.id.in_(run_ids))
)).all()
runs = {r: st for r, st, _ in rows}
ended = {r: reason for r, _, reason in rows}
stopped = [r for r, st in runs.items() if st == "interrupted"]
if stopped:
waiting = set((await session.execute(
@@ -251,7 +262,8 @@ async def _statuses(
out: dict[str, tuple[str, str | None]] = {}
for cid, turn in turns.items():
run_id = run_of[cid] if run_of[cid] in runs else None
out[cid] = (last_status(turn, runs.get(run_id or ""), run_id in waiting, now),
out[cid] = (last_status(turn, runs.get(run_id or ""), run_id in waiting, now,
end_reason=ended.get(run_id or "")),
run_id)
return out
+140 -76
View File
@@ -110,11 +110,12 @@ def _tool_catalog(rows: list[Any], *, builtins: frozenset[str] | None = None) ->
async def _look_service(session: AsyncSession, payload: GenerateIn, sources: list[Any], *,
graph: dict[str, Any] | None, kinds: tuple[str, ...] = ("table", "capability"),
) -> Any:
"""这一轮的查看服务(copilot_context.look_service)。挑选模型用同一个助手模型,第一次要挑时才拿。"""
catalog_view: Any = None) -> Any:
"""这一轮的查看服务(copilot_context.look_service)。挑选模型用同一个助手模型,第一次要挑时才拿。catalog_view:表详情
里给哪一份数据目录(data/catalog_view.py),不给是当前目录。"""
spec = await copilot_model_spec(session, payload, max_tokens=copilot_context.PICK_MAX_TOKENS)
return await copilot_context.look_service(sources, spec=spec, graph=graph, kinds=kinds,
answer=payload.intent == "answer")
answer=payload.intent == "answer", catalog_view=catalog_view)
NODE_REFERENCE = """\
@@ -1756,45 +1757,81 @@ def _repair_request(
@router.post("/generate-stream")
async def generate_stream(payload: GenerateIn, session: AsyncSession = Depends(get_session)):
async def generate_stream(payload: GenerateIn):
"""流式版生成:SSE 逐操作推送,前端边收边把节点摆上画布。
生成过程(copilot_gen)读到 look 就停下、查看、接着写,查看记录作为 look 事件发给前端(前端存进这一轮的
meta.looks)。结束时(done 或流断)发一条 final:服务端把累积的图排版、校验后整体给出——过程可见性来自操作流,
最终质量仍由和手工编排相同的 layout + validate 保证。模型问了用户(ask)的,这一轮到此为止,不自查修正、不自动运行。
生成过程整个在 answer_ops 里(评测直接迭代它,和问数据页走同一条路),这里只包一层 SSE。开流前的拒绝(模型没配好、
对话读不了、限定的数据源都不在)发生在 answer_ops 交出第一条操作之前:先取第一条,拒绝照旧是 HTTP 错误,不进流。
"""
from fastapi.responses import StreamingResponse
from app.data.sqlcheck import load_checkers
ops = answer_ops(payload)
first = await ops.__anext__()
async def event_stream():
yield _sse(first)
async for op in ops:
yield _sse(op)
return StreamingResponse(
event_stream(),
media_type="text/event-stream",
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
)
async def answer_ops(payload: GenerateIn, *, catalog_view: Any = None):
"""生成过程:按需求起草、按需查看、自查修正、排版校验,逐条交出操作(op dict)。问数据页(经 SSE)和评测(直接迭代)
走的是这同一条路(细案 01 做法 2)。
读到 look 就停下、查看、接着写,查看记录作为 look 操作交出(前端存进这一轮的 meta.looks)。结束时(done 或流断)交一条
final:累积的图排版、校验后整体给出——过程可见性来自操作流,最终质量仍由和手工编排相同的 layout + validate 保证。模型
问了用户(ask)的,这一轮到此为止,不自查修正、不自动运行。
开流前的准备(取数据源、读目录检查器、取助手模型、拼历史、组查看服务)用自己的短会话,在交出第一条操作之前做完:
拒绝在这里抛 HTTPException(调用方没迭代到第一条就知道)。catalog_view 是这一轮读哪一份数据目录(data/catalog_view.py:
查看的表详情、自查的 SQL 检查器都按它),不给是当前目录。
final 带 next(chat/turn.next_step:run 运行、ready 只建不跑、fail 判失败),问数据页照它走。最后多交一条 usage
(providers/usage.py):{op: usage, by_model, input_tokens, output_tokens, calls, cost_usd,
unpriced_calls}——这一轮助手建图(含挑表、接着写、自查修正)花了多少。前面每一条和抽出来之前逐条一样。运行不在这里,
不计。
"""
from app.api.deps import restricted
from app.chat import turn
from app.data.catalog_view import CatalogView
from app.data.sqlcheck import load_checkers
from app.db.base import SessionLocal
from app.providers import usage
# 带了对话的,先核对这个人能不能用它(D-d):历史、上一轮的图都从它来
await _ensure_conversation_access(session, payload.conversation_id)
# 开登录后的 user 角色(D-c):图只能用 chat_policy 放行的节点和工具,提示词先说、自查按同一个判断挡
chat_user = restricted()
sources = await _sources(session, payload.datasource_ids)
scope = {r.name for r in sources} if payload.datasource_ids else None
# 基于数据目录的 SQL 检查器:在开流之前读好目录(流里不再碰这个会话),自查和收尾共用
checkers = await load_checkers(session, sources)
nodes, edges = _start_graph(payload)
# 用户摆好的坐标,收尾排版时原样留着
pinned = _pinned_positions(payload.base_graph)
view = CatalogView.parse(catalog_view)
async with SessionLocal() as session:
# 带了对话的,先核对这个人能不能用它(D-d):历史、上一轮的图都从它来
await _ensure_conversation_access(session, payload.conversation_id)
# 开登录后的 user 角色(D-c):图只能用 chat_policy 放行的节点和工具,提示词先说、自查按同一个判断挡
chat_user = restricted()
sources = await _sources(session, payload.datasource_ids)
scope = {r.name for r in sources} if payload.datasource_ids else None
# 基于数据目录的 SQL 检查器:在开流之前读好目录(流里不再碰这个会话),自查和收尾共用
checkers = await load_checkers(session, sources, catalog_view=view)
nodes, edges = _start_graph(payload)
# 用户摆好的坐标,收尾排版时原样留着
pinned = _pinned_positions(payload.base_graph)
try:
model, model_id = await _op_model(session, payload)
except ProviderNotConfigured as e:
raise HTTPException(400, _unconfigured(e)) from e
try:
model, model_id = await _op_model(session, payload)
except ProviderNotConfigured as e:
raise HTTPException(400, _unconfigured(e)) from e
human = _with_history(_graph_request(payload), await _history_section(session, payload.conversation_id))
# 兜底用:模型只发了改动操作时,把它们重放到这张图上
prev_graph = await _previous_graph(session, payload.conversation_id)
# 比对工具绑定的基准:画布上是改之前的图;问数据页是上一轮的图(沿用它的节点 id)
baseline = copy.deepcopy(
(payload.base_graph or {}).get("nodes") or (prev_graph or {}).get("nodes") or [])
# 查看服务:读目录、读能力登记在开流之前做完。现有工作流(画布上的原图、问数据追问时上一轮的图)查过的表列在概览里
looks = await _look_service(session, payload, sources, graph=payload.base_graph if (
payload.base_graph or {}).get("nodes") else prev_graph)
human = _with_history(_graph_request(payload), await _history_section(session, payload.conversation_id))
# 兜底用:模型只发了改动操作时,把它们重放到这张图上
prev_graph = await _previous_graph(session, payload.conversation_id)
# 比对工具绑定的基准:画布上是改之前的图;问数据页是上一轮的图(沿用它的节点 id)
baseline = copy.deepcopy(
(payload.base_graph or {}).get("nodes") or (prev_graph or {}).get("nodes") or [])
# 查看服务:读目录、读能力登记在开流之前做完。现有工作流(画布上的原图、问数据追问时上一轮的图)查过的表列在概览里
looks = await _look_service(session, payload, sources, graph=payload.base_graph if (
payload.base_graph or {}).get("nodes") else prev_graph, catalog_view=view)
# 这一份 system 一直用到最后:接着写、自查修正都沿用,不换
system = _graph_system(sources, scope, await copilot_context.section(looks), looks, chat_user=chat_user)
applier = GraphApplier(nodes, edges)
@@ -1803,8 +1840,8 @@ async def generate_stream(payload: GenerateIn, session: AsyncSession = Depends(g
checker=GraphChecker(applier, scope=scope, sources=sources, baseline=baseline, checkers=checkers,
chat_user=chat_user))
async def event_stream():
yield _sse({"op": "model", "model": model_id})
async def steps():
yield {"op": "model", "model": model_id}
patches = 0
try:
async for op in _with_heartbeat(gen.draft(human), started=started):
@@ -1813,23 +1850,24 @@ async def generate_stream(payload: GenerateIn, session: AsyncSession = Depends(g
# reply 先压着(applier.replied):前端收到 reply 就收尾,提案得排在它前面转出
if (checked := await catalog_patch_op(op, sources)) is not None:
patches += 1
yield _sse(checked)
yield checked
continue
yield _sse(op)
yield op
if applier.replied is not None:
# 这句话不需要工作流:后面那套排版校验对着一张空图跑,只会得到"图是空的,先拖一个节点进来"
yield _sse(applier.replied)
yield applier.replied
return
if gen.asked is not None and not applier.effective:
# 先问清楚才能动手,什么都还没改:这一轮到此为止。照样发一条 final 收尾,带上问题(asked)和没动过的图:
# 前端据此把这一轮记成「助手反问」、问题进历史(以前流就这么断了,问数据页一直停在「进行中」),
# 不运行、不重排。图给一份对象(画布上原来的图、问数据上一轮的图,都没有就是空图),老前端照样认得
yield _sse(_asked_final(gen.asked, payload.base_graph, prev_graph))
asked_final = _asked_final(gen.asked, payload.base_graph, prev_graph)
yield {**asked_final, "next": turn.next_step(asked_final)}
return
# 只发了目录修改提案,既没回答也没改图(用户只是在说一条数据事实):补一句回答收尾,
# 不能按下面的「模型没有给出任何修改」报错——提案已经摆在用户面前了
if patches and all(o.get("op") == "plan" for o in applier.seen):
yield _sse({"op": "reply", "text": PATCH_ONLY_REPLY})
yield {"op": "reply", "text": PATCH_ONLY_REPLY}
return
# 流正常结束,却没有 done、也没有一个操作能认(格式跑偏,每一行都被跳过):
# 往下走就是拿原图去自查、交付,界面上写「无需修改」——而模型什么都没说
@@ -1837,12 +1875,11 @@ async def generate_stream(payload: GenerateIn, session: AsyncSession = Depends(g
raise CopilotIncomplete("模型没有给出任何修改,也没有说明原因",
hint="画布未修改。可以重试,或把需求描述得更具体")
except CopilotIncomplete as e:
yield _sse({"op": "error", "message": f"助手本轮未完成:{e}", "hint": e.hint, "detail": e.detail})
yield {"op": "error", "message": f"助手本轮未完成:{e}", "hint": e.hint, "detail": e.detail}
return
except Exception as e: # noqa: BLE001
reason, hint = explain_error(e)
yield _sse({"op": "error", "message": f"助手本轮未完成:{reason}",
"hint": hint, "detail": raw_error(e)})
yield {"op": "error", "message": f"助手本轮未完成:{reason}", "hint": hint, "detail": raw_error(e)}
return
# 兜底:一个操作都没落地,而上一轮有图。
@@ -1858,7 +1895,7 @@ async def generate_stream(payload: GenerateIn, session: AsyncSession = Depends(g
# 以前这些问题照样交付、照样自动开跑——跑到那一步才炸(开发库里四次运行死在
# 循环 / 分支条件里套了 {{ }}),前面的步骤白跑,报错还停在半路
async for op in _with_heartbeat(gen.repair(), phase="repairing"):
yield _sse(op)
yield op
remaining = _blocking_issues(nodes, edges, scope, sources=sources, baseline=baseline, checkers=checkers,
chat_user=chat_user)
# 工具绑定变化不挡运行,但要在自查这一步就说出来:工具被改没了的图照样
@@ -1871,11 +1908,9 @@ async def generate_stream(payload: GenerateIn, session: AsyncSession = Depends(g
# 改不好也照实交付、列出问题,但不自动运行:明知跑不起来的图,开跑只会
# 在半路报一个用户看不懂的错
autorun = False
yield _sse({"op": "check", "status": "failed", "issues": remaining,
"warnings": warnings})
yield {"op": "check", "status": "failed", "issues": remaining, "warnings": warnings}
elif remaining is not None:
yield _sse({"op": "check", "status": "passed", "repaired": gen.repaired,
"warnings": warnings})
yield {"op": "check", "status": "passed", "repaired": gen.repaired, "warnings": warnings}
# 收尾:排版 + 校验,把最终图整体交付。改图时旧节点留在原处,只排新节点
try:
@@ -1913,16 +1948,17 @@ async def generate_stream(payload: GenerateIn, session: AsyncSession = Depends(g
# 改了图又问了用户(这一轮照样结束、不自动运行):问题跟着 final 交付,前端记进这一轮
**({"asked": _asked_of(gen.asked)} if gen.asked is not None else {}),
}
# 建完之后怎么走(chat/turn.next_step):run 运行、ready 只建不跑、fail 判失败。问数据页照它走,评测用同一个函数
final["next"] = turn.next_step(final)
except Exception as e: # noqa: BLE001
final = {"op": "error", "message": f"模型返回的工作流结构有误:{graph_error(e)}",
"hint": "请重试,或把需求描述得更具体", "detail": raw_error(e)}
yield _sse(final)
yield final
return StreamingResponse(
event_stream(),
media_type="text/event-stream",
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
)
spent = usage.meter()
async for op in usage.metered(steps(), spent):
yield op
yield {"op": "usage", **spent.summary()}
# --------------------------------------------------------------------------
@@ -2774,13 +2810,31 @@ class ReviewIn(BaseModel):
#: 默认只在扫出异常信号时才叫模型。打开它就每次都复核——排查时有用,
#: 平时不开:干净的运行没什么可复核的,代价却是每次问数都多一次调用
always: bool = False
#: 这一轮已经自动试了几次(从 0 数,同 chat/turn.retry_after_review):结果里的 retry_instruction 据此给不给。
#: 不传按第一次算
attempt: int = Field(default=0, ge=0)
@router.post("/review")
async def review_run(
payload: ReviewIn, session: AsyncSession = Depends(get_session)
) -> dict[str, Any]:
"""接住一次编排的产出,判断它能不能就这么交给用户。
"""接住一次编排的产出,判断它能不能就这么交给用户(规则见 review_of_run)。这里只核对能不能读这次运行。"""
from app.api.deps import PRIVATE_RUN, can_read_run, current_principal
from app.db.models import Run
run = await session.get(Run, payload.run_id)
if not run:
raise HTTPException(404, "运行记录不存在,可能已被删除")
if not await can_read_run(current_principal(), run):
raise HTTPException(403, PRIVATE_RUN)
return await review_of_run(session, run, question=payload.question, provider=payload.provider,
model=payload.model, always=payload.always, attempt=payload.attempt)
async def review_of_run(session: AsyncSession, run: Any, *, question: str, provider: str | None = None,
model: str | None = None, always: bool = False, attempt: int = 0) -> dict[str, Any]:
"""复核一次运行(问数据页经 /copilot/review、评测直接调,同一个函数)。
「跑完了」和「答得对」是两回事。运行状态只说明流程走完了,而过程中检索
降级、工具报错、agent 步数用满这些事,原先一件都不会进到答案里——用户拿到
@@ -2788,47 +2842,57 @@ async def review_run(
规则层先扫一遍(engine/review.py:scan),没扫出东西就到此为止、直接返回,
不花一次模型调用;扫出来了才叫模型重组答案并把异常说破。
"""
from app.api.deps import PRIVATE_RUN, can_read_run, current_principal
from app.db.models import Run, RunEvent
from app.engine import review as rv
run = await session.get(Run, payload.run_id)
if not run:
raise HTTPException(404, "运行记录不存在,可能已被删除")
if not await can_read_run(current_principal(), run):
raise HTTPException(403, PRIVATE_RUN)
结果是 ReviewResult.as_dict(),另带两项:
- retry_instruction:要不要带着原图重来一次(chat/turn.retry_after_review,按 attempt 算这一轮还能不能再试)——要的话
是交给助手的那句话(chat/turn.retry_instruction),不要是 null。问数据页照它走;
- usage:复核这一次模型调用花了多少(providers/usage.py 的汇总,没调模型时是全 0),这一轮的花费 = 助手 + 运行 + 复核。
"""
from app.chat import turn
from app.db.models import RunEvent
from app.engine import review as rv
from app.providers import usage
rows = list((await session.execute(
select(RunEvent).where(RunEvent.run_id == payload.run_id).order_by(RunEvent.seq)
select(RunEvent).where(RunEvent.run_id == run.id).order_by(RunEvent.seq)
)).scalars())
output = run.output if isinstance(run.output, dict) else None
signals = rv.scan(rows, output)
answer = rv.answer_text(output)
spent = usage.meter()
if not signals and not payload.always:
return rv.ReviewResult(verdict="ok", signals=[]).as_dict()
def done(result: Any) -> dict[str, Any]:
out = result.as_dict()
out["retry_instruction"] = (
turn.retry_instruction(question, turn.review_reason(out), turn.has_nodes(run.graph))
if turn.retry_after_review(out, attempt) else None)
out["usage"] = spent.summary()
return out
if not signals and not always:
return done(rv.ReviewResult(verdict="ok", signals=[]))
try:
spec = await copilot_model_spec(session, GenerateIn(
instruction="review", provider=payload.provider, model=payload.model,
instruction="review", provider=provider, model=model,
), max_tokens=4096)
model, _ = await get_chat_model(session, spec)
chat_model, _ = await get_chat_model(session, spec)
except (ProviderNotConfigured, HTTPException) as e:
# 没配模型不该让这一轮挂掉:规则层已经知道出了什么事,照样说得出话
return rv.ReviewResult(
return done(rv.ReviewResult(
verdict="annotated",
note=";".join(s.detail for s in signals) or (
_unconfigured(e) if isinstance(e, ProviderNotConfigured) else str(e.detail)
),
retry=rv.should_retry(signals),
signals=signals, severity=rv.worst(signals),
).as_dict()
))
result = await rv.review(
model, question=payload.question, answer=answer, signals=signals,
# 报告撰写节点产出的答案逐段对应着证据文档:只能加说明,不能改写
locked=bool(output and output.get("_evidence")),
)
return result.as_dict()
with usage.metering(spent):
result = await rv.review(
chat_model, question=question, answer=answer, signals=signals,
# 报告撰写节点产出的答案逐段对应着证据文档:只能加说明,不能改写
locked=bool(output and output.get("_evidence")),
)
return done(result)
+1 -1
View File
@@ -293,7 +293,7 @@ async def blueprint_stream(payload: BlueprintStreamIn, session: AsyncSession = D
system = "\n\n".join(part for part in (
BLUEPRINT_SYSTEM_PROMPT, _templates_text(personal=personal), await looks.overview(), BLUEPRINT_PROTOCOL_PROMPT,
looks.guide(), BLUEPRINT_LOOK_WHEN_PROMPT) if part)
missing = [i for i in blueprint_issues(spec, for_="trial") if i["level"] == "error"]
missing = [i for i in blueprint_issues(spec, for_="trial", graph=row.graph) if i["level"] == "error"]
human = copilot._with_history(BLUEPRINT_REQUEST_PROMPT.format(
spec=json.dumps(spec, ensure_ascii=False), instruction=instruction,
issues=BLUEPRINT_ISSUES_PROMPT.format(lines="\n".join(f"- {i['path']} {i['message']}" for i in missing))
+4 -3
View File
@@ -65,16 +65,17 @@ SYSTEM_LOOK_UPGRADE_NEED = "要升级的工作流里,查库节点的 SQL 用
async def look_service(sources: list[Any], *, spec: ModelSpec | None, graph: Mapping[str, Any] | None = None,
kinds: tuple[str, ...] = ("table", "capability"), answer: bool = False) -> LookService:
kinds: tuple[str, ...] = ("table", "capability"), answer: bool = False,
catalog_view: Any = None) -> LookService:
"""按入口组装这一轮的查看服务。读目录、读能力登记都用各自的短会话,开流前后都能调。
spec 是助手的模型规格:挑选模型第一次要用时才按它拿(关思考、额度按 look/pick.PICK_MAX_TOKENS)。graph 是现有工作流,概览里列出它查过的表。
answer:问数据(意图为回答),能力只列只读的。
answer:问数据(意图为回答),能力只列只读的。catalog_view:表详情里给哪一份数据目录(data/catalog_view.py),不给是当前目录。
"""
picker = Picker(spec)
made: list[Any] = []
if "table" in kinds:
made.append(await TableKind.load(sources, graph=graph, picker=picker))
made.append(await TableKind.load(sources, graph=graph, picker=picker, catalog_view=catalog_view))
if "capability" in kinds:
made.append(await CapabilityKind.load(picker=picker, read_only=answer))
return LookService(made)
+3 -2
View File
@@ -18,7 +18,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
from app.api import health
from app.api.coded import PROFILE_SETTINGS_INVALID, CodedHTTPException
from app.api.deps import signer_of
from app.api.deps import signer_of, signer_verified
from app.core.errors import AUTH, NETWORK, TIMEOUT, classify, explain, raw
from app.core.config import settings
from app.core.crypto import encrypt, mask
@@ -1104,7 +1104,8 @@ async def purge_raw(
if imp.raw_state == "purged":
raise CodedHTTPException(409, "这次导入的原件已经清除", "raw_already_purged")
try:
purged = await table_versions.purge_import_raw(session, imp, reason=reason, signed_by=signed_by)
purged = await table_versions.purge_import_raw(session, imp, reason=reason, signed_by=signed_by,
signer_verified=signer_verified())
except table_versions.StoreUnavailable as e:
raise CodedHTTPException(503, str(e), "store_unavailable") from e
deleted, others = purged.deleted, purged.others
+22 -10
View File
@@ -26,9 +26,11 @@ from dataclasses import dataclass, field
from typing import Any
from urllib.parse import unquote
from fastapi import Depends, HTTPException
from fastapi import Depends
from starlette.requests import HTTPConnection
from app.api.coded import CHAT_FORBIDDEN, ROLE_REQUIRED, RUN_FORBIDDEN, RUN_PRIVATE, CodedHTTPException
#: 登录方式
AUTH_MODES = ("off", "local", "oidc")
#: 角色由高到低
@@ -105,6 +107,15 @@ def signer_of(given: Any, header: str | None, limit: int = 100) -> str | None:
return decode_actor(header)
def signer_verified() -> bool:
"""signer_of 记下的署名是不是认证过的身份:开了登录、这次请求的身份认证过时为真;关登录时署名是自报的,为假。
导入清单、累积快照清单、作废记录、清除原件记录的「是否认证」记这个(docs/design/next-round/05-已知遗留.md (c)),
和审计表、待办派单、run.origin 的 verified 同一个口径。证据要在生成那一刻定下:调用方和 signed_by 一起往下传。"""
principal = _principal.get()
return principal is not None and principal.mode != "off" and bool(principal.verified)
def current_principal() -> Principal:
"""这次请求的完整身份。关登录时:等同 admin、未认证,username 是中间件按 X-Actor 解出的自报署名;中间件没写
(不在请求里,比如测试直接调函数、启动时的整理、调度循环)时同样等同 admin、未认证,username 为空。"""
@@ -140,7 +151,7 @@ async def can_read_artifact(principal: Principal, artifact_id: str) -> bool:
def ensure_role(role: str) -> None:
"""接口内部用:至少是 role,否则 401(没登录)/ 403(角色不够)。关登录时放行。
"""接口内部用:至少是 role,否则 401(没登录,auth_required)/ 403(角色不够,role_required)。关登录时放行。
用在要看请求内容才知道要什么角色的地方(比如 MCP 的 stdio 只有管理员能建)。"""
from app.auth import require
@@ -169,8 +180,8 @@ async def ensure_conversation_access(session: Any, conversation_id: str | None)
return await check(session, conversation_id)
def _forbidden() -> HTTPException:
return HTTPException(403, "没有权限查看这次运行")
def _forbidden() -> CodedHTTPException:
return CodedHTTPException(403, "没有权限查看这次运行", RUN_FORBIDDEN)
#: user 角色在自己的问数据运行上能读的(GET /api/runs/{id} 之后的部分):运行记录、事件、运行时的图,加上实时流
@@ -235,13 +246,13 @@ async def _run_scope(conn: HTTPConnection) -> None:
return
ensure_role("user")
if not user_can_chat():
raise HTTPException(403, _RUNS_FOR_BUILDERS)
raise CodedHTTPException(403, _RUNS_FOR_BUILDERS, ROLE_REQUIRED)
method = conn.scope.get("method", "GET")
run_id = conn.path_params.get("run_id")
if run_id is None:
if method == "POST" and conn.url.path.rstrip("/") == "/api/runs":
return
raise HTTPException(403, _RUNS_FOR_BUILDERS)
raise CodedHTTPException(403, _RUNS_FOR_BUILDERS, ROLE_REQUIRED)
run = await _load_run(run_id)
rest = conn.url.path.rstrip("/").removeprefix(f"/api/runs/{run_id}")
if method == "GET" and rest in _CHAT_RUN_READS:
@@ -252,7 +263,7 @@ async def _run_scope(conn: HTTPConnection) -> None:
raise _forbidden()
if method == "POST" and rest == "/cancel":
return
raise HTTPException(403, "没有权限:这项操作需要搭建者及以上的角色")
raise CodedHTTPException(403, "没有权限:这项操作需要搭建者及以上的角色", ROLE_REQUIRED)
async def _load_run(run_id: str | None) -> Any:
@@ -282,7 +293,7 @@ async def _refuse_private(conn: HTTPConnection, principal: Principal) -> None:
from fastapi import WebSocketException, status
raise WebSocketException(code=status.WS_1008_POLICY_VIOLATION)
raise HTTPException(403, PRIVATE_RUN)
raise CodedHTTPException(403, PRIVATE_RUN, RUN_PRIVATE)
async def _private_run_scope(conn: HTTPConnection) -> None:
@@ -310,7 +321,7 @@ async def _evidence_scope(conn: HTTPConnection) -> None:
return
ensure_role("user")
if conn.scope.get("method", "GET") != "GET":
raise HTTPException(403, "没有权限:这项操作需要搭建者及以上的角色")
raise CodedHTTPException(403, "没有权限:这项操作需要搭建者及以上的角色", ROLE_REQUIRED)
run_id = conn.path_params.get("run_id")
if not run_id or not await can_read_run(principal, run_id):
raise _forbidden()
@@ -327,7 +338,8 @@ async def _chat_scope(conn: HTTPConnection) -> None:
method = conn.scope.get("method", "GET")
if user_can_chat() and (path.startswith("/api/conversations") or (method, path) in _CHAT_ROUTES):
return
raise HTTPException(403, "没有权限:问数据和助手需要搭建者及以上的角色,或由管理员为使用者开启问数据")
raise CodedHTTPException(403, "没有权限:问数据和助手需要搭建者及以上的角色,或由管理员为使用者开启问数据",
CHAT_FORBIDDEN)
#: user 角色开了问数据时能用的其余接口:问数据的生成流、读当前助手模型、问数据页选数据源用的数据源名单
+1 -1
View File
@@ -2851,7 +2851,7 @@ async def export_report_html(run_id: str, report: str | None = None) -> Response
if run is None: # 两次读之间被删掉了
raise CodedHTTPException(404, "运行记录不存在", RUN_NOT_FOUND)
info = {"id": run.id, "workflow_id": run.workflow_id, "workflow_name": run.workflow_name,
"run_class": run.run_class, "status": run.status,
"run_class": run.run_class, "status": run.status, "end_reason": run.end_reason,
"started_at": run.started_at, "finished_at": run.finished_at}
data = build_report(info, report_node=chosen.node_id, doc_artifact=chosen.doc_artifact,
doc_hash_ok=chosen.hash_ok, doc=doc, seal=sealed.seal, chains=chains, refs=refs,
+25 -46
View File
@@ -1,6 +1,5 @@
from __future__ import annotations
import json
from collections import defaultdict
from typing import Any
@@ -45,11 +44,14 @@ async def tool_usage(
merged.extend(agent.get("tools") or [])
granted[node.id] = sorted(set(merged))
from app.engine.origin import counted_runs
# 只算计入统计的运行(engine/origin.counted_runs):评测、口径对账、召回替人跑的运行用到了哪些工具不说明白名单该不该收
run_ids = [
rid
for (rid,) in await session.execute(
select(Run.id)
.where(Run.workflow_id == workflow_id)
.where(Run.workflow_id == workflow_id, counted_runs())
.order_by(Run.created_at.desc())
.limit(limit_runs)
)
@@ -95,62 +97,39 @@ async def exploratory_clusters(
) -> dict[str, Any]:
"""探索性问题聚类:同一个问题被反复问,就该长成模板里的固定一节。
这就是模板生长的触发器。用本地 embedding 做贪心聚类,
出现 ≥3 次的簇标记为晋升候选。
这就是模板生长的触发器。取问题、聚类全站只有一份(app/chat/questions.py):只取问题原文(不含 history),问数据一轮
一条(自动重来不多算),评测这类系统派生的运行、别人的个人应用刷新和个人蓝图试运行都不进。出现 ≥3 次的簇标记为晋升
候选。embedder 是聚类用的向量模型:本地哈希的语义能力弱,结果只是下限。
"""
from sqlalchemy import not_
from app.api.deps import current_principal
from app.auth.access import hidden_runs
from app.memory.embeddings import cosine, embed_texts
from app.chat.questions import cluster_questions, embedder, recent_questions
rows = list(
await session.execute(
select(Run.id, Run.input, Run.workflow_name, Run.created_at)
# 别人的个人应用刷新、个人蓝图试运行只给拥有者(同运行列表),不进聚类
.where(Run.run_class != "formal", not_(hidden_runs(current_principal())))
.order_by(Run.created_at.desc())
.limit(limit)
)
)
items = []
for rid, input_payload, wf_name, created in rows:
text = json.dumps(input_payload or {}, ensure_ascii=False, sort_keys=True)
if len(text) <= 4: # 空输入没有聚类意义
continue
items.append({"run_id": rid, "text": text, "workflow": wf_name,
"at": created.isoformat() if created else None})
if not items:
return {"clusters": [], "considered": 0}
questions = await recent_questions(session, current_principal(), limit=limit)
if not questions:
return {"clusters": [], "considered": 0, "embedder": embedder()}
clusters = await cluster_questions(questions, threshold=threshold)
vectors = await embed_texts([i["text"] for i in items])
clusters: list[dict[str, Any]] = []
for i, item in enumerate(items):
placed = False
for cluster in clusters:
if cosine(vectors[i], vectors[cluster["seed"]]) >= threshold:
cluster["members"].append(item)
placed = True
break
if not placed:
clusters.append({"seed": i, "members": [item]})
def _at(value: Any) -> str | None:
return value.isoformat() if value else None
out = [
{
"count": len(c["members"]),
"representative": c["members"][0]["text"][:200],
"workflow": c["members"][0]["workflow"],
"run_ids": [m["run_id"] for m in c["members"]][:20],
"first_at": c["members"][-1]["at"],
"last_at": c["members"][0]["at"],
"promote_candidate": len(c["members"]) >= 3,
"count": c.count,
"representative": c.members[0].text[:200],
"workflow": c.members[0].workflow,
"run_ids": [m.run_id for m in c.members if m.run_id][:20],
"turn_ids": [m.turn_id for m in c.members if m.turn_id][:20],
"first_at": _at(c.members[-1].at),
"last_at": _at(c.members[0].at),
"promote_candidate": c.count >= 3,
}
for c in clusters
if len(c["members"]) >= 2
if c.count >= 2
]
out.sort(key=lambda c: c["count"], reverse=True)
return {
"clusters": out,
"considered": len(items),
"considered": len(questions),
"embedder": embedder(),
"hint": "被标记为可固化的问题反复出现,建议用「提取模板」把它固定为工作流中的一步",
}
+53
View File
@@ -27,6 +27,9 @@ router = APIRouter(prefix="/api", tags=["identity"])
#: 人和组的名录(GET /api/directory):只读、给搭建者及以上(main.py 挂 require_role("builder")),岗位绑定、应用授权的
#: 选人控件据此给候选、核对填的名字在不在。只有用户名、显示名、是否停用和组名,不含邮箱、角色、登录时间
directory_router = APIRouter(prefix="/api", tags=["identity"])
#: 使用者选人的候选(GET /api/directory/candidates,05-已知遗留 (b)):任何登录的人都能调(main.py 挂 require_role("user")),
#: 按用途给候选池,只有用户名和显示名
candidates_router = APIRouter(prefix="/api", tags=["identity"])
def _iso(value: Any) -> str | None:
@@ -224,6 +227,56 @@ async def directory(session: AsyncSession = Depends(get_session)) -> dict[str, A
"groups": [{"name": g.name, "label": g.label} for g in known.groups.values()]}
@candidates_router.get("/directory/candidates")
async def candidates(purpose: str, app_id: str | None = None, entry: str | None = None, task_id: str | None = None,
field: str | None = None, q: str | None = None, limit: int = 20,
session: AsyncSession = Depends(get_session)) -> dict[str, Any]:
"""使用者选人时的候选:{users: [{username, display_name}], groups: [{name, label}], truncated, needs_query?}。
不给是否停用(停用的直接不列)、角色、邮箱、组成员。purpose=form:表单里的选人控件,带 app_id 和 entry(入口表单)
或 task_id(待办表单)、field(哪个控件),池按字段的 x-principal 算、缺省是能用这个应用的人,要求调用的人能用这个
应用(否则 403);应用授权给了所有人时不带前缀 q 不列(needs_query)。purpose=review:个人应用提交审核,启用的搭建者
和管理员。q 按用户名或显示名的开头筛,limit 最多 50。开了登录时提交的值按同一个池核对(auth/directory.principal_errors)。"""
from app.api.coded import CodedHTTPException
from app.api.deps import current_principal
from app.apps.access import app_role, at_least
from app.apps.tree import entry_of, resolve_form
from app.auth.directory import PURPOSES, field_pool, principal_fields, review_pool
from app.db.models_apps import App, AppVersion
from app.db.models_process import Task
if purpose not in PURPOSES:
raise CodedHTTPException(422, f"候选的用途只能是 {' / '.join(PURPOSES)}", "candidates_purpose")
if purpose == "review":
return (await review_pool(session)).pick(q, limit) | {"groups": []}
principal = current_principal()
bindings: dict[str, Any] | None = None
form: Any = None
task = await session.get(Task, task_id) if task_id else None
app = await session.get(App, task.app_id if task is not None else app_id) if (task or app_id) else None
if app is None or (task_id and task is None):
raise CodedHTTPException(404, "应用或待办不存在", "candidates_not_found")
if not at_least(await app_role(session, principal, app), "use"):
raise CodedHTTPException(403, "你没有这个应用的使用权限,看不到它的候选人", "candidates_forbidden")
if task is not None:
form = task.form
if task.app_version is not None:
bindings = await session.scalar(select(AppVersion.bindings).where(
AppVersion.app_id == app.id, AppVersion.version == task.app_version))
else:
version = (await session.execute(select(AppVersion).where(
AppVersion.app_id == app.id, AppVersion.version == app.published_version))).scalar_one_or_none() \
if app.published_version else None
spec = version.spec if version is not None else {}
found = entry_of(spec or {}, entry or "") if entry else None
form = resolve_form(spec or {}, found.get("form")) if found else None
bindings = version.bindings if version is not None else None
target = principal_fields(form).get(field or "")
if target is None:
raise CodedHTTPException(404, "表单里没有这个选人控件", "candidates_not_found")
return (await field_pool(session, app, bindings, target)).pick(q, limit)
# ---------------------------------------------------------------- 审计
+13 -9
View File
@@ -21,7 +21,7 @@ from app.core.crypto import encrypt
from app.db.base import SessionLocal
from app.db.models import NotifyChannel, NotifyDelivery
from app.notify import core
from app.notify.service import EVENTS, EVENT_LABEL, channel_out
from app.notify.service import EVENTS, channel_out, subscribe_problem
router = APIRouter(prefix="/api/notify", tags=["notify"])
@@ -61,12 +61,14 @@ async def _safe_personal(scope: str, url: str) -> None:
raise HTTPException(422, problem)
def _validated(payload: ChannelIn, *, url: str | None) -> None:
def _validated(payload: ChannelIn, *, url: str | None, saved: list[str] | None = None) -> None:
"""saved 是渠道已经订阅着的事件:只核对这次新勾的(notify/service.subscribe_problem)。以前存下的照旧——当时界面对
哪种渠道都列全部事件,现在才按事件推给哪几条路核对,不能让这些渠道连开关、改名都 422。"""
if payload.format not in core.FORMATS:
raise HTTPException(422, f"消息格式只能是{'、'.join(core.FORMAT_LABEL.values())}之一")
unknown = [e for e in payload.events if e not in EVENTS]
if unknown:
raise HTTPException(422, f"不认识的通知事件:{'、'.join(unknown)}。可选:{'、'.join(EVENT_LABEL.values())}")
for event in dict.fromkeys(payload.events):
if event not in (saved or ()) and (problem := subscribe_problem(event, payload.scope)):
raise HTTPException(422, problem)
if url is not None and (problem := core.format_problem(payload.format, url)):
raise HTTPException(422, problem)
@@ -84,8 +86,10 @@ async def list_channels() -> dict[str, Any]:
@router.get("/events")
async def list_events() -> list[dict[str, str]]:
return [{"event": e, "label": EVENT_LABEL[e]} for e in EVENTS]
async def list_events() -> list[dict[str, Any]]:
"""登记的通知事件和推给哪几条路(typesApp.ts 的 NotifyEventRow):设置页、应用的通知按 scopes 列可订阅的事件。
scopes 是空的,是登记了、还没有推送的,照实给出来,由界面不列。"""
return [{"event": name, "label": event.label, "scopes": list(event.scopes)} for name, event in EVENTS.items()]
@router.post("/channels", status_code=201)
@@ -150,9 +154,9 @@ async def update_channel(channel_id: str, payload: ChannelIn) -> dict[str, Any]:
from app.core.crypto import decrypt
current = decrypt(row.url) or ""
_validated(payload, url=current or None)
_validated(payload, url=current or None, saved=list(row.events or []))
else:
_validated(payload, url=url)
_validated(payload, url=url, saved=list(row.events or []))
await _safe_personal(row.scope, url)
row.url = encrypt(url)
row.name, row.format = payload.name.strip(), payload.format
+54 -7
View File
@@ -56,6 +56,9 @@ class RunOut(BaseModel):
input: dict[str, Any]
output: dict[str, Any]
error: str | None
#: 结束在 cancelled 的原因(docs/design/next-round/00-总方案.md X9):rejected 人工驳回、stopped 手动停止。
#: 其余终态、加这一列之前的运行为空。界面、通知、定时推送都读这一个字段,不再解析报错原文
end_reason: str | None = None
usage: dict[str, Any]
run_class: str | None = "exploratory"
version: int | None = None
@@ -171,13 +174,54 @@ async def list_runs(
以前只有 limit:待审批的运行排在第 145 位,列表只取 100 条,点了徽标也找不到它。
筛选都走服务端,翻页用 created_at 做游标(before 传上一页最后一条的 created_at)。
别人的个人应用刷新、个人蓝图试运行不列(只给拥有者,auth/access.hidden_runs),管理员也一样。
没按来源筛时,系统派生的运行(评测、口径对账、口径召回:engine/origin.SYSTEM_ORIGINS)默认不列——几十道评测题的运行
会把真实用户的运行挤出第一页;按来源点名就列。没列出几次由 GET /api/runs/unlisted 照实说,界面写在列表底下。
"""
stmt = _filtered(select(Run), workflow_id=workflow_id, status=status, run_class=run_class, q=q,
schedule_id=schedule_id, origin_kind=origin_kind)
stmt = stmt.order_by(Run.created_at.desc()).limit(limit)
if before is not None:
stmt = stmt.where(Run.created_at < before)
return list((await session.execute(stmt)).scalars())
@router.get("/unlisted")
async def unlisted_runs(
workflow_id: str | None = Query(default=None),
status: str | None = Query(default=None),
run_class: str | None = None,
q: str | None = Query(default=None),
schedule_id: str | None = Query(default=None),
origin_kind: str | None = Query(default=None),
session: AsyncSession = Depends(get_session),
) -> dict[str, Any]:
"""同样的筛选下,运行列表因为「系统派生的运行默认不列」而没列出的有几次:{total, by_origin: {来源: 次数}}。
按来源筛了的(origin_kind 给了)什么都不藏,total 为 0。别人的个人应用刷新、个人蓝图试运行不算在内:它们不是「没列出」,
是读不了。"""
from sqlalchemy import func
from app.engine.origin import SYSTEM_ORIGINS
if _csv(origin_kind):
return {"total": 0, "by_origin": {}}
stmt = _filtered(select(Run.origin_kind, func.count()), workflow_id=workflow_id, status=status, run_class=run_class,
q=q, schedule_id=schedule_id, origin_kind=",".join(SYSTEM_ORIGINS))
rows = (await session.execute(stmt.group_by(Run.origin_kind))).all()
by_origin = {kind: int(n) for kind, n in rows if kind and n}
return {"total": sum(by_origin.values()), "by_origin": by_origin}
def _filtered(stmt: Any, *, workflow_id: str | None, status: str | None, run_class: str | None, q: str | None,
schedule_id: str | None, origin_kind: str | None) -> Any:
"""运行列表的筛选条件(列表和「没列出几次」共用一份)。"""
from sqlalchemy import not_
from app.api.deps import current_principal
from app.auth.access import hidden_runs
from app.engine.origin import counted_runs
stmt = select(Run).where(not_(hidden_runs(current_principal()))).order_by(Run.created_at.desc()).limit(limit)
stmt = stmt.where(not_(hidden_runs(current_principal())))
if workflow_id == UNSAVED:
stmt = stmt.where(Run.workflow_id.is_(None))
elif workflow_id:
@@ -193,9 +237,10 @@ async def list_runs(
if origins := _csv(origin_kind):
kinds = [o for o in origins if o != NO_ORIGIN]
stmt = stmt.where(or_(Run.origin_kind.in_(kinds), *([Run.origin_kind.is_(None)] if NO_ORIGIN in origins else [])))
if before is not None:
stmt = stmt.where(Run.created_at < before)
return list((await session.execute(stmt)).scalars())
else:
# 没点名来源:只列计入统计的运行,系统派生的(评测、口径对账、召回)不列;来源为空的老运行照常列出
stmt = stmt.where(counted_runs())
return stmt
def _csv(value: str | None) -> list[str]:
@@ -289,11 +334,13 @@ async def cancel_run(run_id: str, x_actor: str | None = Header(default=None)) ->
run = await session.get(Run, run_id)
if run is None:
raise HTTPException(404, "运行记录不存在")
raise HTTPException(409, _cannot_cancel(run.status))
raise HTTPException(409, _cannot_cancel(run.status, run.end_reason))
return {"ok": True, "status": outcome}
def _cannot_cancel(status: str) -> str:
def _cannot_cancel(status: str, end_reason: str | None = None) -> str:
if status == "cancelled" and end_reason == "rejected":
return "这次运行已被驳回结束,没有要停止的。"
return {
"succeeded": "这次运行已完成,没有要停止的。",
"failed": "这次运行已失败结束,没有要停止的;如需继续,请点「继续运行」。",
@@ -396,7 +443,7 @@ async def continue_run(
except KeyError as e:
raise _missing(e) from e
except ContinueRefused as e:
# 正式运行带图(formal_graph_locked)、写入结果未知还没核实(action_unresolved):带机器码
# 正式运行带图(formal_graph_locked)、写入结果未知还没核实(action_unresolved)、被驳回了的(run_rejected):带机器码
raise CodedHTTPException(409, str(e), e.code) from e
except ValueError as e:
raise HTTPException(409, str(e)) from e
+4 -1
View File
@@ -491,6 +491,8 @@ async def create_schedule_row(session: AsyncSession, body: dict[str, Any], *, ac
values = schedule_values(body)
hook = values.pop("webhook")
workflow = await _check_workflow(session, values["workflow_id"], values["run_class"], values["input_template"])
# 目标的几列只经 scheduler/service 的一处封装写(工作流目标:target_id 同 workflow_id)
values |= service.target_columns("workflow", values["workflow_id"])
row = Schedule(**values, workflow_name=workflow.name,
next_run_at=service.next_run(values["cron"], values["timezone"], values["enabled"]),
created_by=actor, **_webhook_columns(hook))
@@ -528,7 +530,8 @@ async def update_schedule(schedule_id: str, request: Request,
if "name" in body:
values["name"] = _name(body["name"])
if "workflow_id" in body:
values["workflow_id"] = _workflow_id(body["workflow_id"])
# 目标的几列一起改(scheduler/service.target_columns):只改 workflow_id 的话 target_id 还指着原来的工作流
values |= service.target_columns("workflow", _workflow_id(body["workflow_id"]))
if "cron" in body:
values["cron"] = _cron(body["cron"])
if "timezone" in body:
+40 -27
View File
@@ -10,7 +10,8 @@ from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.api import health
from app.api.coded import CodedHTTPException
from app.api.coded import PUBLIC_URL_ENV_LOCKED, CodedHTTPException
from app.core import public_url
from app.core.errors import explain, not_configured, raw
from app.core.config import settings as app_settings
from app.core.crypto import encrypt, mask
@@ -604,35 +605,32 @@ def catalog_draft_values(stored: Any) -> dict[str, Any]:
# --------------------------------------------------------------------------
# 通知:定时任务推送 Webhook 时拼运行链接用的对外访问地址
# 对外访问地址:通知、Webhook 里的链接,单点登录的回调,来源核对(core/public_url.py)
# --------------------------------------------------------------------------
#: 设置里 notify 这一组。public_base_url 是别人打开 AgentLab 用的地址(https://agentlab.example.com),通知里的
#: 运行链接 = 它 + /runs/{运行 id}。服务端不知道自己在反向代理后面叫什么,只能由人填;留空的通知里不附链接
NOTIFY_GROUP = "notify"
NOTIFY_DEFAULTS: dict[str, Any] = {"public_base_url": None}
NOTIFY_SETTING_LABEL = {"public_base_url": "对外访问地址"}
#: 运行链接 = 它 + /runs/{运行 id}。服务端不知道自己在反向代理后面叫什么,只能由人填;留空的通知里不附链接。
#: 全站只认一个对外地址(core/public_url.py):环境变量 AGENTLAB_PUBLIC_URL 设了的话这一项不生效、不能改(PUT 回 409)。
#: 键名沿用 notify 一组,界面上在「设置 → 通用」
NOTIFY_GROUP = public_url.SETTING_GROUP
NOTIFY_DEFAULTS: dict[str, Any] = {public_url.SETTING_KEY: None}
NOTIFY_SETTING_LABEL = {public_url.SETTING_KEY: "对外访问地址"}
#: GET /api/settings 另给的只读一组:{value: 生效的对外访问地址, env_locked: 是不是环境变量指定的}。设置页据此显示
#: 「由环境变量指定」;PUT 带上它也不落库(同 limits)
PUBLIC_URL_GROUP = "public_url"
def _base_url_problem(value: Any) -> str | None:
if value is None or (isinstance(value, str) and not value.strip()):
return None
label = NOTIFY_SETTING_LABEL["public_base_url"]
label = NOTIFY_SETTING_LABEL[public_url.SETTING_KEY]
hint = (f"「{label}」需要是 http:// 或 https:// 开头的完整地址(例如 https://agentlab.example.com),"
f"留空表示通知里不附运行链接,当前为「{value}」")
f"留空表示通知里不附链接,当前为「{value}」")
if not isinstance(value, str):
return hint
from urllib.parse import urlsplit
try:
parts = urlsplit(value.strip())
except ValueError:
return hint
if parts.scheme not in ("http", "https") or not parts.hostname or " " in value.strip():
return hint
if parts.query or parts.fragment:
return f"「{label}」不能带 ? 或 # 之后的部分,运行链接会接在地址后面,当前为「{value}」"
return None
if (problem := public_url.url_problem(value)) == "suffix":
return f"「{label}」不能带 ? 或 # 之后的部分,链接会接在地址后面,当前为「{value}」"
return hint if problem else None
def notify_problem(value: Any) -> str | None:
@@ -651,18 +649,23 @@ def notify_problem(value: Any) -> str | None:
def notify_values(stored: Any) -> dict[str, Any]:
"""库里存的(或刚写进来的)notify 一组 → 补齐默认、规范化:去掉首尾空白和末尾的 /,空串当没填;存坏了的按没填。"""
stored = stored if isinstance(stored, dict) else {}
url = stored.get("public_base_url")
if not isinstance(url, str) or not url.strip() or _base_url_problem(url):
return {"public_base_url": None}
return {"public_base_url": url.strip().rstrip("/")}
return {public_url.SETTING_KEY: public_url.normalize(stored.get(public_url.SETTING_KEY))}
async def notify_settings(session: AsyncSession) -> dict[str, Any]:
"""设置里的 notify 一组,没存过的补默认。定时任务推送 Webhook 时从这里读。"""
"""设置里存的 notify 一组,没存过的补默认。要拼链接的地方不读它:一律经 core/public_url(环境变量优先)。"""
row = await session.get(Setting, NOTIFY_GROUP)
return notify_values(row.value if row else None)
def public_url_values(stored_notify: Any) -> dict[str, Any]:
"""GET /api/settings 里只读的 public_url 一组:生效的那个地址(环境变量优先,其次库里存的这一项),是不是环境变量
指定的。按库里存的算,不读进程缓存:设置页看到的就是落库的样子。"""
locked = public_url.env_locked()
value = public_url.env_url() if locked else notify_values(stored_notify)[public_url.SETTING_KEY]
return {"value": value, "env_locked": locked}
# --------------------------------------------------------------------------
# 应用:按需视图的每日上限(D-j)、使用者能不能建个人应用(docs/design/normal-user-apps.md N7)
# --------------------------------------------------------------------------
@@ -754,7 +757,7 @@ DEFAULT_SETTINGS: dict[str, Any] = {
JUDGE_GROUP: dict(JUDGE_DEFAULTS),
# 核查(双盲复算、反方审稿、审批质询):核查模型、三项的开关和上限、每日金额上限。和裁判一组一样由用户定
VERIFY_GROUP: verify_values(None),
# 通知:定时任务的 Webhook 里拼运行链接用的对外访问地址
# 对外访问地址(存在 notify 一组):通知、Webhook 里的链接,单点登录的回调,来源核对。环境变量优先(core/public_url.py)
NOTIFY_GROUP: notify_values(None),
# 起草模型:数据目录起草、按配方导入的 AI 起草用的模型,留空沿用助手的模型
CATALOG_DRAFT_GROUP: catalog_draft_values(None),
@@ -808,6 +811,7 @@ async def get_settings(session: AsyncSession = Depends(get_session)) -> dict[str
# limits 是服务端配置(环境变量),只读:库里可能存着旧版设置页写进去的一份
#(见过 max_agent_steps: 25),它从来不生效,却会盖住真实的上限,界面照着错的数许诺
stored.pop("limits", None)
stored.pop(PUBLIC_URL_GROUP, None)
# 裁判、核查的每日计数是引擎记的账,不是用户设置:设置页整组回写时带着一份旧的,会把今天的花费清零
stored.pop(JUDGE_SPEND, None)
stored.pop(VERIFY_SPEND, None)
@@ -818,6 +822,7 @@ async def get_settings(session: AsyncSession = Depends(get_session)) -> dict[str
merged[JUDGE_GROUP] = judge_values(stored.get(JUDGE_GROUP))
merged[VERIFY_GROUP] = verify_values(stored.get(VERIFY_GROUP))
merged[NOTIFY_GROUP] = notify_values(stored.get(NOTIFY_GROUP))
merged[PUBLIC_URL_GROUP] = public_url_values(stored.get(NOTIFY_GROUP))
merged[CATALOG_DRAFT_GROUP] = catalog_draft_values(stored.get(CATALOG_DRAFT_GROUP))
merged[APPS_GROUP] = apps_values(stored.get(APPS_GROUP))
for key, value in stored.items():
@@ -910,6 +915,11 @@ async def put_settings(
raise HTTPException(422, problem)
# 存规范化之后的:末尾带 / 的地址拼链接会多出一个斜杠
payload.values[NOTIFY_GROUP] = notify_values(payload.values[NOTIFY_GROUP])
if public_url.env_locked() and payload.values[NOTIFY_GROUP] != await notify_settings(session):
# 环境变量指定了对外访问地址:改这一项不会生效,照实拒绝,不悄悄存一个用不上的值。原样写回(整组回写)不算改
raise CodedHTTPException(
409, f"对外访问地址由环境变量 {public_url.ENV_NAME} 指定({public_url.env_url()}),不能在设置里修改。"
"要改请改部署配置后重启服务", PUBLIC_URL_ENV_LOCKED, field=f"{NOTIFY_GROUP}.{public_url.SETTING_KEY}")
if APPS_GROUP in payload.values:
if problem := apps_problem(payload.values[APPS_GROUP]):
raise HTTPException(422, problem)
@@ -917,9 +927,9 @@ async def put_settings(
if found := await _provider_problem(session, payload.values):
raise CodedHTTPException(422, found[1], PROVIDER_INVALID, field=found[0])
for key, value in payload.values.items():
if key in (TRUST_KEY, "limits", JUDGE_SPEND, VERIFY_SPEND, *_hidden_groups()):
if key in (TRUST_KEY, "limits", PUBLIC_URL_GROUP, JUDGE_SPEND, VERIFY_SPEND, *_hidden_groups()):
# 信任三档只走 PUT /api/tools/trust:设置页整组回写时带着一份旧的,
# 会把刚在工具页改的档位冲掉。limits 是环境变量定的,写进库也不生效。
# 会把刚在工具页改的档位冲掉。limits 是环境变量定的、public_url 是算出来的,写进库也不生效。
# 裁判、核查的每日计数是引擎记的账,只有引擎自己写。登录设置只经 /api/auth/settings 改(_hidden_groups)
continue
row = await session.get(Setting, key)
@@ -928,6 +938,9 @@ async def put_settings(
else:
session.add(Setting(key=key, value=value))
await session.commit()
if NOTIFY_GROUP in payload.values:
# 中间件、单点登录的回调按进程缓存取对外访问地址:写完库就刷新,不用重启
public_url.remember(payload.values[NOTIFY_GROUP][public_url.SETTING_KEY])
return await get_settings(session)
+4 -3
View File
@@ -21,7 +21,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
from app.api.coded import CodedHTTPException
from app.api.datasources import _out
from app.api.deps import signer_of
from app.api.deps import signer_of, signer_verified
from app.data import source_versions, table_versions
from app.data.recipe_imports import ImportRefused
from app.db.base import get_session
@@ -170,7 +170,8 @@ async def remove_period(source_id: str, import_id: str, request: Request, x_acto
source = await _source(session, source_id)
try:
out = await source_versions.remove_period(session, source, import_id, expected_current=expected,
reason=reason or "", signed_by=_signed(body, x_actor))
reason=reason or "", signed_by=_signed(body, x_actor),
signer_verified=signer_verified())
except _REFUSALS as e:
raise _refused(e) from e
return {"source": await _source_out(session, source_id), **out}
@@ -193,7 +194,7 @@ async def revoke_acceptance(source_id: str, import_id: str, request: Request,
try:
out = await source_versions.revoke_acceptance(
session, source, import_id, expected_current=expected, expected_target=target, ack_mask_lost=ack,
reason=reason or "", signed_by=_signed(body, x_actor))
reason=reason or "", signed_by=_signed(body, x_actor), signer_verified=signer_verified())
except _REFUSALS as e:
raise _refused(e) from e
return {"source": await _source_out(session, source_id), **out}
+3 -2
View File
@@ -28,7 +28,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
from app.api.coded import CodedHTTPException
from app.api.datasources import _NAME_PATTERN, _out, read_upload
from app.api.deps import current_principal, signer_of
from app.api.deps import current_principal, signer_of, signer_verified
from app.data import recipe_imports, table_versions
from app.data.recipe_imports import EditRequest, ImportRefused
from app.data.recipe_types import Acceptance, AiAvailability, PeriodInput, Recipe, Selection
@@ -422,7 +422,8 @@ async def commit(staging_id: str, request: Request, x_actor: str | None = Header
try:
result = await recipe_imports.commit_staging(
session, staging, trial_id=trial_id if isinstance(trial_id, str) else "",
confirmations=confirmations, acceptances=_acceptances(body.get("acceptances"), who), signed_by=who)
confirmations=confirmations, acceptances=_acceptances(body.get("acceptances"), who), signed_by=who,
signer_verified=signer_verified())
except _REFUSALS as e:
raise _refused(e) from e
source = await session.get(DataSource, result.source_id, populate_existing=True)
+14 -9
View File
@@ -1,17 +1,20 @@
"""待办(BE-2):列表、计数、详情、回复、改派、催办。设计文档 §3.7、§3.10。
身份经 api/deps.current_principal 取;关登录时它没有名字,用自报的署名(X-Actor)补上,回复人一栏才写得出人。
回复被拒回 403 / 404 / 409 / 422,{detail, code};表单不通过时另带 errors(同应用入口的 422)。
回复被拒回 403 / 404 / 409 / 422,{detail, code},一律带 code(外部系统按它认);表单不通过时另带 errors(同应用入口的
422)。服务层没带码的拒绝按状态补上同一类的码(api/coded.py 的 TASK_*)。
"""
from __future__ import annotations
from typing import Any, Literal
from fastapi import APIRouter, Header, HTTPException, Query
from fastapi import APIRouter, Header, Query
from fastapi.responses import JSONResponse
from pydantic import BaseModel, Field
from app.api.coded import CodedHTTPException
from app.api.coded import (
ROLE_REQUIRED, TASK_CLOSED, TASK_FORBIDDEN, TASK_INVALID, TASK_NOT_FOUND, CodedHTTPException,
)
from app.api.deps import current_principal, decode_actor
from app.process import tasks as task_service
from app.process.tasks import FormInvalid, TaskRefused
@@ -23,12 +26,14 @@ def principal_of(x_actor: str | None) -> Any:
return task_service.principal_with_actor(current_principal(), decode_actor(x_actor))
def refused(e: TaskRefused) -> HTTPException | JSONResponse:
#: 服务层(process/tasks.py、process/actions.py、blueprints/personal.py)没带码的拒绝按状态归到哪一类
_CODE_BY_STATUS = {404: TASK_NOT_FOUND, 403: TASK_FORBIDDEN, 409: TASK_CLOSED, 422: TASK_INVALID}
def refused(e: TaskRefused) -> CodedHTTPException | JSONResponse:
if isinstance(e, FormInvalid):
return JSONResponse({"detail": str(e), "code": e.code, "errors": e.errors}, status_code=422)
if e.code:
return CodedHTTPException(e.status_code, str(e), e.code)
return HTTPException(e.status_code, str(e))
return CodedHTTPException(e.status_code, str(e), e.code or _CODE_BY_STATUS.get(e.status_code, TASK_CLOSED))
def _raise(e: TaskRefused) -> Any:
@@ -53,7 +58,7 @@ async def list_tasks(
) -> dict[str, Any]:
principal = principal_of(x_actor)
if assigned == "all" and principal.mode != "off" and not principal.at_least("builder"):
raise HTTPException(403, "只能查看派给自己的待办")
raise CodedHTTPException(403, "只能查看派给自己的待办", ROLE_REQUIRED)
statuses = None if status == "all" else _csv(status) or ["pending"]
items = await task_service.list_tasks(principal, assigned=assigned, statuses=statuses, app_id=app_id,
run_id=run_id, limit=limit)
@@ -71,7 +76,7 @@ async def get_task(task_id: str, x_actor: str | None = Header(default=None)) ->
principal = principal_of(x_actor)
out = await task_service.detail(task_id, principal)
if out is None or not task_service.visible(out, principal):
raise HTTPException(404, "待办不存在,所属的运行可能已被删除")
raise CodedHTTPException(404, "待办不存在,所属的运行可能已被删除", TASK_NOT_FOUND)
return out
+9 -3
View File
@@ -24,13 +24,19 @@ VIEW = "latest"
INSTANCE_VIEW = "inst"
ENTRY = "start"
RECONCILER = "reconciler"
#: 派生不了时的机器码:工作流、版本不存在(404)
WORKFLOW_GONE = "workflow_not_found"
VERSION_GONE = "workflow_version_not_found"
class DeriveRefused(Exception):
def __init__(self, status_code: int, message: str) -> None:
"""派生不了。code 是给程序认的机器码,接口照它回。"""
def __init__(self, status_code: int, message: str, code: str) -> None:
super().__init__(message)
self.status_code = status_code
self.message = message
self.code = code
def _safe(text: str) -> str:
@@ -45,10 +51,10 @@ async def derive_app(session: AsyncSession, workflow_id: str, version: int, *, n
workflow = await session.get(Workflow, workflow_id)
if workflow is None:
raise DeriveRefused(404, "工作流不存在,可能已被删除")
raise DeriveRefused(404, "工作流不存在,可能已被删除", WORKFLOW_GONE)
row = await version_row(session, workflow_id, version)
if row is None:
raise DeriveRefused(404, f"「{workflow.name}」不存在 v{version} 版本")
raise DeriveRefused(404, f"「{workflow.name}」不存在 v{version} 版本", VERSION_GONE)
state = PinState(alias=PIN, pin={"workflow_id": workflow_id, "version": version}, workflow=workflow, row=row)
schedule = (await session.execute(select(Schedule).where(
Schedule.workflow_id == workflow_id, Schedule.run_class == "formal", Schedule.enabled.is_(True))
+7
View File
@@ -113,6 +113,13 @@ async def start_entry(app_id: str, entry_id: str, values: dict[str, Any], princi
form = resolve_form(spec, entry.get("form")) or {"type": "object", "properties": {}}
if errors := validate_form(form, values if isinstance(values, dict) else None):
raise FormRejected(errors)
# 选人控件:开了登录时按候选池核对(auth/directory.py,05-已知遗留 (b))。以前原样传下去,填错的名字会把待办
# 派给一个不存在的人——没人能处理,也不报错
from app.auth.directory import principal_errors
if errors := await principal_errors(session, app, version.bindings, form, values,
getattr(principal, "mode", "off")):
raise FormRejected(errors)
# 表单里的文件(docs/design/normal-user-apps.md N9):核对是提交人在这个入口传的、还没用过,先占住;表单值换成
# 登记的那一份引用(连同 sha256 进运行输入、随封存固定)。发起成功绑到这次运行,发起不成放回去
values, claimed = await _claim_files(form, values, principal, app.id, entry_id)
+10 -1
View File
@@ -28,6 +28,8 @@ _FIELD_KEYS = frozenset({
"minLength", "maxLength", "maxItems", "format", "items", "x-widget",
# 文件控件的限制:accept(类型或扩展名的列表)、max_mb(docs/design/normal-user-apps.md §3.8)
"x-file",
# 选人控件的候选池:{pool: app | group:<组> | slot:<岗位>}(auth/directory.py,05-已知遗留 (b))
"x-principal",
})
#: 文件控件只认这几项写法
_FILE_FIELD_KEYS = frozenset({"type", "title", "description", "readOnly", "x-widget", "x-file"})
@@ -88,6 +90,13 @@ def _field_problems(field: Any) -> list[str]:
return out + _file_field_problems(field)
if "x-file" in field:
out.append("不是文件控件,不能写文件的限制")
if "x-principal" in field:
from app.auth.directory import pool_problem
if field.get("x-widget") != "principal":
out.append("不是选人控件,不能写候选范围")
elif problem := pool_problem(field["x-principal"]):
out.append(problem)
if kind not in FORM_TYPES:
out.append(f"的类型只能是 {' / '.join(FORM_TYPES)}")
if "format" in field and field["format"] not in FORM_FORMATS:
@@ -153,7 +162,7 @@ def _clean_schema(form: dict[str, Any]) -> dict[str, Any]:
continue
props[name] = {k: v for k, v in field.items()
if k in _FIELD_KEYS and k not in ("format", "x-widget", "x-enum-labels", "title",
"description", "readOnly", "default")}
"description", "readOnly", "default", "x-principal")}
return {"type": "object", "properties": props, "required": [r for r in form.get("required") or [] if r in props]}
+38 -10
View File
@@ -51,6 +51,8 @@ NEEDS_UPGRADE = "app_needs_upgrade"
STATUS_TEXT = {"queued": "排队中", "running": "运行中", "interrupted": "等待处理", "succeeded": "已完成",
"failed": "失败", "cancelled": "已取消"}
#: 被驳回、又说不出是哪一步时(没有节点可指)的说法
REJECTED_TEXT = "被驳回"
TIER_TEXT = {"formal": "完整出具", "degraded": "降档出具", "withheld": "不予出具"}
NO_TIER = "未声明出具契约"
#: 蓝图试运行(docs/design/normal-user-apps.md N4):印章的来源、说明,趋势只有一个点时的说明,预览里入口按钮为什么点不了
@@ -1277,8 +1279,9 @@ async def _trails(session: AsyncSession, runs: list[Run]) -> dict[str, _Trail]:
for row in (await session.execute(select(ActionRecord).where(ActionRecord.run_id.in_(ids))
.order_by(ActionRecord.created_at, ActionRecord.attempt))).scalars():
out[row.run_id].actions.append(row)
# 节点事件只有结束了的运行要看(走到了哪个成果节点、停在哪个节点上)
wanted = [r.id for r in runs if r.status in ("succeeded", "failed")]
# 节点事件只有结束了的运行要看(走到了哪个成果节点、停在哪个节点上)。取消的只有被驳回的要看(停在哪一步)
wanted = [r.id for r in runs if r.status in ("succeeded", "failed")
or (r.status == "cancelled" and r.end_reason == "rejected")]
if wanted:
for run_id, etype, node_id in (await session.execute(
select(RunEvent.run_id, RunEvent.type, RunEvent.node_id).where(
@@ -1325,6 +1328,26 @@ def rejected_outputs(graph: Any) -> dict[str, set[str]]:
return out
def _rejection_text(trail: _Trail, node_id: str, title: str) -> str | None:
"""停在被人驳回的待办上时怎么说:动作确认写「「付款」未确认(被驳回)」,别的写「「x」被驳回」;不是驳回的为 None。"""
kind, decision = _decision(trail, node_id)
if kind == "action_confirm" and decision == "reject":
return f"「{title}」未确认(被驳回)"
if decision in _REJECTIONS:
return f"「{title}」被驳回"
return None
async def legacy_rejected(session: AsyncSession, run: Run) -> bool:
"""加 end_reason 之前被人驳回的运行记的是 failed(封存不改写):失败的那一步上最近一条处理完的待办是驳回。和实例页
认存量驳回(instance_state 的 failed 分支)是同一份判断;runner 据此不许它接着跑。"""
if run.status != "failed":
return False
trail = (await _trails(session, [run]))[run.id]
failed = [nid for etype, nid in trail.events if etype == "node.failed"]
return bool(failed) and _decision(trail, failed[-1])[1] in _REJECTIONS
def instance_state(run: Run, trail: _Trail) -> dict[str, str]:
"""实例给人看的状态 {code, text}:code 是界面画状态标记用的状态(运行的那几种,加 held),text 说这件事办到哪了、
结论是什么,不只是运行状态(P1)。
@@ -1334,10 +1357,12 @@ def instance_state(run: Run, trail: _Trail) -> dict[str, str]:
否定的那种结局——这次运行里人工节点驳回了、从它的「驳回」出口一路走到的成果节点(rejected_outputs)——code 记
cancelled(同「未确认(被驳回)」,是人的决定,不画成成功),文字照旧写成果节点的标题;
- 停着:有已发出、结果未知的动作,写「等待核实写入结果」;否则按在等的待办说(等待审批、等待确认发送……);
- 失败:停在一个被人驳回的待办上的——动作确认被驳回(「付款」未确认(被驳回))、审批设了驳回即终止、补充输入被
驳回——是人的决定,不是故障,code 记 cancelled;核实为未生效、选了放弃的同理。外部系统拒绝之类照写失败。
- 被驳回(cancelled + end_reason=rejected,engine/context.NodeRejected):动作确认被驳回写「「付款」未确认(被驳回)」,
审批设了驳回即终止、补充输入被驳回等写「「x」被驳回」,code 记 cancelled;手动停止的写「已取消」;
- 失败:加 end_reason 之前被人驳回的运行记的是 failed(封存不改写),停在被驳回的待办上的照上一条说,和新形状说法
一样;核实为未生效、选了放弃的同理,code 记 cancelled。外部系统拒绝之类照写失败。
运行本身的状态不改(驳回动作确认在引擎里就是节点失败、运行结束):这里只是把它说清楚。"""
通知(notify/service.notify_run)的文字也取自这里:实例页和推送说同一句话。"""
from app.apps.tree import node_map
nodes = node_map(run.graph)
@@ -1360,15 +1385,18 @@ def instance_state(run: Run, trail: _Trail) -> dict[str, str]:
negative = any(_decision(trail, gate)[1] in _REJECTIONS for gate in gates)
return {"code": "cancelled" if negative else "succeeded", "text": title(reached[0])}
return {"code": "succeeded", "text": STATUS_TEXT["succeeded"]}
if run.status == "cancelled" and run.end_reason == "rejected":
failed = [nid for etype, nid in trail.events if etype == "node.failed"]
node_id = run.error_node_id or (failed[-1] if failed else None)
said = _rejection_text(trail, node_id, title(node_id)) if node_id else None
# 待办没留下(老的审批、没开出待办)时按驳回的那一步说,不退回「已取消」:这是人的决定,不是有人停了它
return {"code": "cancelled", "text": said or (f"「{title(node_id)}」被驳回" if node_id else REJECTED_TEXT)}
if run.status == "failed":
failed = [nid for etype, nid in trail.events if etype == "node.failed"]
if failed:
node_id = failed[-1]
kind, decision = _decision(trail, node_id)
if kind == "action_confirm" and decision == "reject":
return {"code": "cancelled", "text": f"「{title(node_id)}」未确认(被驳回)"}
if decision in _REJECTIONS:
return {"code": "cancelled", "text": f"「{title(node_id)}」被驳回"}
if said := _rejection_text(trail, node_id, title(node_id)):
return {"code": "cancelled", "text": said}
record = next((a for a in reversed(trail.actions) if a.node_id == node_id), None)
if record is not None and record.state == "failed":
if (record.resolution or {}).get("outcome") == "abandon":
+3 -2
View File
@@ -47,9 +47,10 @@ FILE_FIELD_TYPE = "object"
FILE_REF_KEYS = ("file_id", "name", "size", "sha256", "type")
#: 表单字段能用的 format。不支持 pattern:JS 和 Python 的正则语义不同,会出现前端放行、后端拒收
FORM_FORMATS = ("date", "date-time", "email", "uri")
#: 表单校验错误的 code。file:文件控件的值不对(没上传、不是本人传的、已经用过、类型或大小不符,app/files/store.py)
#: 表单校验错误的 code。file:文件控件的值不对(没上传、不是本人传的、已经用过、类型或大小不符,app/files/store.py);
#: principal:开了登录时选人控件的值不在候选池里(auth/directory.principal_errors)
FORM_ERROR_CODES = ("required", "type", "enum", "minimum", "maximum", "min_length", "max_length", "max_items", "format",
"file")
"file", "principal")
class _Strict(BaseModel):
+18 -9
View File
@@ -15,7 +15,9 @@ import logging
import os
from typing import Any
from fastapi import Depends, HTTPException
from fastapi import Depends
from app.api.coded import AUTH_REQUIRED, ROLE_REQUIRED, CodedHTTPException
logger = logging.getLogger(__name__)
@@ -77,17 +79,24 @@ def remember_settings(value: dict[str, Any]) -> None:
_loaded = True
def refusal(principal: Any, role: str) -> CodedHTTPException | None:
"""principal 够不够 role:够(或关登录)是 None;不够交回该回的错——没登录 401(auth_required),角色不够 403
(role_required)。路由级依赖、接口内部的 ensure_role、中间件拦 /docs 都用它,同一个判断、同一句话。"""
if principal.mode == "off":
return None
if not principal.username:
return CodedHTTPException(401, "请先登录", AUTH_REQUIRED)
if not principal.at_least(role):
return CodedHTTPException(403, f"没有权限:这项操作需要{ROLE_LABEL.get(role, role)}及以上的角色", ROLE_REQUIRED)
return None
def require(role: str) -> None:
"""当前请求至少是 role,否则 401(没登录)/ 403(角色不够)。关登录时放行。"""
"""当前请求至少是 role,否则 401(没登录)/ 403(角色不够),都带 code。关登录时放行。"""
from app.api.deps import current_principal
principal = current_principal()
if principal.mode == "off":
return
if not principal.username:
raise HTTPException(401, "请先登录")
if not principal.at_least(role):
raise HTTPException(403, f"没有权限:这项操作需要{ROLE_LABEL.get(role, role)}及以上的角色")
if (refused := refusal(current_principal(), role)) is not None:
raise refused
def role_dependency(role: str) -> Any:
+177 -1
View File
@@ -1,6 +1,10 @@
"""人和组的名录:选人控件的候选(GET /api/directory)、应用检查核对岗位绑定和授权、改授权时核对填的名字,同一处取、
同一个口径(停用的人也在名录里,标 disabled:已经绑着他的岗位、授权要能认出「这个人停用了」,不是「没有这个人」)。
使用者选人(表单里的选人控件、个人应用提交审核)不拿整份名录,拿按用途算的候选池(candidates,GET
/api/directory/candidates):只有这件事能选的人,只给用户名和显示名,停用的不列;开了登录时提交的值按同一个池核对
(docs/design/next-round/05-已知遗留.md (b))。名录整份仍只给搭建者。
名录里对不上怎么算看登录方式(同前端 PrincipalPicker):
- 账号密码登录:人和组都在本系统里建,名录里没有就是填错了(或人被删了),停用的人处理不了待办——都算错;
- 单点登录:人和组第一次登录后才进名录,名录里没有的照实提醒、不拒收;停用的人名录里认得出,照样算错;
@@ -10,7 +14,7 @@ from __future__ import annotations
from collections.abc import Iterable
from dataclasses import dataclass, field
from typing import Literal
from typing import Any, Literal
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
@@ -95,3 +99,175 @@ def mismatch(directory: Directory, mode: str, kind: str, name: str) -> Mismatch
return Mismatch(kind, name, problem, False,
f"{what}名录里还没有:单点登录的人和组第一次登录后才进名录,名字要和身份提供方里的一致才生效")
return Mismatch(kind, name, problem, True, f"{what}名录里没有")
# --------------------------------------------------------------------------
# 候选池:使用者选人时能选谁(05-已知遗留 (b))
# --------------------------------------------------------------------------
#: 候选池的用途:form 表单里的选人控件(入口表单、待办表单);review 个人应用提交审核选审核的人
PURPOSES = ("form", "review")
#: 一次最多给几位候选
CANDIDATES_MAX = 50
#: 表单字段声明候选池(x-principal.pool):app 能用这个应用的人(缺省);group:<组> 这个组的成员;slot:<岗位> 这个岗位
#: 绑定的人和组(按应用版本的 bindings)
POOL_APP = "app"
POOL_GROUP = "group:"
POOL_SLOT = "slot:"
def pool_problem(value: Any) -> str | None:
"""x-principal 的写法对不对:只能是 {pool: app | group:<组> | slot:<岗位>}。没问题为 None。"""
if not isinstance(value, dict) or set(value) != {"pool"} or not isinstance(value.get("pool"), str):
return "的候选范围只能写成 {\"pool\": \"app\"}、{\"pool\": \"group:<组名>\"} 或 {\"pool\": \"slot:<岗位>\"}"
pool = value["pool"].strip()
if pool == POOL_APP:
return None
for prefix in (POOL_GROUP, POOL_SLOT):
if pool.startswith(prefix) and pool[len(prefix):].strip():
return None
return "的候选范围只能是 app(能用这个应用的人)、group:<组名> 或 slot:<岗位>"
def pool_of(field: Any) -> str:
"""表单字段声明的候选池;没声明、写法不对的按缺省(能用这个应用的人)。"""
setting = field.get("x-principal") if isinstance(field, dict) else None
if isinstance(setting, dict) and pool_problem(setting) is None:
return setting["pool"].strip()
return POOL_APP
@dataclass
class Pool:
"""一个候选池。open:池就是全部启用的人(应用授权给了所有人)——不带前缀不列,免得等于发通讯录。"""
users: dict[str, DirectoryUser] = field(default_factory=dict)
groups: dict[str, DirectoryGroup] = field(default_factory=dict)
open: bool = False
def has(self, name: str) -> bool:
"""提交的这个名字在不在池里(用户名或组名)。"""
return name in self.users or name in self.groups
def pick(self, q: str | None, limit: int = CANDIDATES_MAX) -> dict[str, Any]:
"""接口给的那一份:{users: [{username, display_name}], groups: [{name, label}], truncated};按前缀筛
(用户名或显示名,不分大小写)。池是 open 的、又没带前缀:什么都不给,needs_query 为真。"""
prefix = (q or "").strip().lower()
if self.open and not prefix:
return {"users": [], "groups": [], "truncated": False, "needs_query": True}
limit = max(1, min(int(limit or CANDIDATES_MAX), CANDIDATES_MAX))
def hit(*names: str) -> bool:
return not prefix or any(n.lower().startswith(prefix) for n in names if n)
users = [u for u in sorted(self.users.values(), key=lambda u: u.username) if hit(u.username, u.display_name)]
groups = [g for g in sorted(self.groups.values(), key=lambda g: g.name) if hit(g.name, g.label)]
return {"users": [{"username": u.username, "display_name": u.display_name} for u in users[:limit]],
"groups": [{"name": g.name, "label": g.label} for g in groups[:limit]],
"truncated": len(users) > limit or len(groups) > limit}
async def _members(session: AsyncSession, group: str) -> dict[str, DirectoryUser]:
from app.db.models import Group, GroupMember, User
rows = (await session.execute(
select(User).join(GroupMember, GroupMember.user_id == User.id).join(Group, Group.id == GroupMember.group_id)
.where(Group.name == group, User.disabled.is_(False)))).scalars()
return {u.username: DirectoryUser(u.username, u.display_name or u.username, False) for u in rows}
async def _people(session: AsyncSession, items: Iterable[tuple[str, str]]) -> Pool:
"""一组(种类, 名字)展开成池:人按名录认(停用的、名录里没有的不列),组列出组本身、再展开成员。"""
from app.db.models import Group, User
pool = Pool()
items = list(items)
names = sorted({n for k, n in items if k == "user" and n})
if names:
for u in (await session.execute(select(User).where(User.username.in_(names), User.disabled.is_(False))))\
.scalars():
pool.users[u.username] = DirectoryUser(u.username, u.display_name or u.username, False)
groups = sorted({n for k, n in items if k == "group" and n})
if groups:
for g in (await session.execute(select(Group).where(Group.name.in_(groups)))).scalars():
pool.groups[g.name] = DirectoryGroup(g.name, g.label or g.name)
pool.users.update(await _members(session, g.name))
return pool
async def _everyone(session: AsyncSession) -> Pool:
from app.db.models import Group, User
pool = Pool(open=True)
for u in (await session.execute(select(User).where(User.disabled.is_(False)))).scalars():
pool.users[u.username] = DirectoryUser(u.username, u.display_name or u.username, False)
for g in (await session.execute(select(Group))).scalars():
pool.groups[g.name] = DirectoryGroup(g.name, g.label or g.name)
return pool
async def app_pool(session: AsyncSession, app: Any) -> Pool:
"""能用这个应用的人:使用及以上的授权(人、组展开成员),加上管理它的人(管理员、建它的人)。授权给了所有人的,池就是
全部启用的人(open)。按 apps/access 的口径:搭建者没有授权时只能查看全部实例,不算「能用」的候选。"""
from app.apps.access import RANK, grants_of
from app.db.models import User
grants = [g for g in await grants_of(session, app.id) if RANK.get(g.role, 0) >= RANK["use"]]
if any(g.principal_kind == "everyone" for g in grants):
return await _everyone(session)
admins = (await session.execute(select(User.username).where(User.role == "admin"))).scalars()
items = [(g.principal_kind, g.principal) for g in grants] + [("user", a) for a in admins]
if getattr(app, "created_by", None):
items.append(("user", app.created_by))
return await _people(session, items)
async def field_pool(session: AsyncSession, app: Any, bindings: dict[str, Any] | None, field: Any) -> Pool:
"""表单里一个选人控件的候选池(按字段的 x-principal,缺省能用这个应用的人)。bindings 是这次用到的应用版本的绑定。"""
pool = pool_of(field)
if pool.startswith(POOL_GROUP):
name = pool[len(POOL_GROUP):].strip()
return Pool(users=await _members(session, name))
if pool.startswith(POOL_SLOT):
bound = ((bindings or {}).get("slots") or {}).get(pool[len(POOL_SLOT):].strip())
items = [(str(b.get("kind") or "user"), str(b.get("name") or "")) for b in bound or [] if isinstance(b, dict)]
if any(k == "everyone" for k, _ in items):
return await _everyone(session)
return await _people(session, items)
return await app_pool(session, app)
async def review_pool(session: AsyncSession) -> Pool:
"""个人应用提交审核能交给谁:启用的搭建者和管理员(同 blueprints/personal._review_targets 的规则)。"""
from app.db.models import User
rows = (await session.execute(select(User).where(User.role.in_(("builder", "admin")),
User.disabled.is_(False)))).scalars()
return Pool(users={u.username: DirectoryUser(u.username, u.display_name or u.username, False) for u in rows})
def principal_fields(form: Any) -> dict[str, dict[str, Any]]:
"""表单里的选人控件:{字段名: 字段}。"""
props = form.get("properties") if isinstance(form, dict) else None
return {name: f for name, f in (props or {}).items() if isinstance(f, dict) and f.get("x-widget") == "principal"}
#: 选人控件的值不在候选池里时表单错误的 code(apps/spec.FORM_ERROR_CODES 的 principal)
PRINCIPAL_ERROR_CODE = "principal"
async def principal_errors(session: AsyncSession, app: Any, bindings: dict[str, Any] | None, form: Any,
values: Any, mode: str) -> list[dict[str, str]]:
"""开了登录时按候选池核对表单里选人控件的值:不在池里的按表单错误交回(code=principal),话写成「「x」不在可选的人里」。
关登录不核对:没有用户表,签批记的是各自填的署名。没填的不在这里报(必填由表单校验报)。"""
if mode == "off" or not isinstance(values, dict):
return []
out: list[dict[str, str]] = []
for name, f in principal_fields(form).items():
value = values.get(name)
if value in (None, "") or not isinstance(value, str):
continue
pool = await field_pool(session, app, bindings, f)
if not pool.has(value.strip()):
hint = ":单点登录的人要先登录一次才进名录" if mode == "oidc" else ",请从候选中选择"
out.append({"path": name, "code": PRINCIPAL_ERROR_CODE, "message": f"「{value.strip()}」不在可选的人里{hint}"})
return out
+20 -6
View File
@@ -9,6 +9,8 @@
- 会话 cookie → Principal(认证过的用户名、角色、组);没有会话是匿名(username=None),X-Actor 一律不看;
- /api 下除了登录本身、健康检查、入站钩子,匿名一律 401。各路由的角色要求另由 main.py 的路由级依赖管,这里兜底:
哪个路由忘了挂依赖,也不会对没登录的人敞开;
- 接口说明(FastAPI 自带的 /docs、/redoc、/openapi.json)和 /api 一样拦:匿名 401,不到搭建者 403。它们是 FastAPI 自己
挂的路由,挂不上路由级依赖,所以角色也在这里判。以前只拦 /api/ 开头的,开了登录接口全貌也对匿名的人敞开;
- 非 GET 的请求必须带 X-AgentLab-Request 头,带了 Origin 的必须是本站或登记的前端地址(CSRF)。自定义请求头跨站发不出来
(要过 CORS 预检,只放行配置里的前端地址),SameSite=Lax 的 cookie 也不随跨站的表单提交带过去;
- WebSocket:没登录的握手前就关掉;登录了的交给路由,按运行判(api/deps.py 的 run_scope:user 角色只能连自己能读、
@@ -19,8 +21,8 @@ from __future__ import annotations
import json
from http.cookies import SimpleCookie
from typing import Any
from urllib.parse import urlsplit
from app.api.coded import AUTH_REQUIRED
from app.core.config import settings
#: 开登录时匿名也能访问的接口
@@ -28,6 +30,8 @@ _PUBLIC = frozenset({"/api/health", "/api/auth/me", "/api/auth/login", "/api/aut
"/api/auth/oidc/callback", "/api/auth/jwks", "/api/auth/bootstrap"})
#: 只认签名、不走会话的入站钩子(也不查 CSRF:调用方是外部系统,没有浏览器的 Origin)
_HOOKS = "/api/hooks/"
#: FastAPI 自带的接口说明(main.py 里 FastAPI() 的缺省地址):开登录时要搭建者
_DOCS = ("/docs", "/redoc", "/openapi.json")
_SAFE_METHODS = frozenset({"GET", "HEAD", "OPTIONS"})
CSRF_HEADER = "x-agentlab-request"
@@ -49,11 +53,17 @@ def _cookie(headers: dict[str, str], name: str) -> str | None:
return morsel.value if morsel else None
def _is_docs(path: str) -> bool:
return any(path == p or path.startswith(p + "/") for p in _DOCS)
def _origin_ok(origin: str, headers: dict[str, str], scope: dict[str, Any]) -> bool:
from app.core.public_url import origin_of, public_base_url
allowed = {o.rstrip("/") for o in settings.cors_origins}
if settings.public_url:
parts = urlsplit(settings.public_url)
allowed.add(f"{parts.scheme}://{parts.netloc}")
# 对外访问地址(环境变量或设置里的那一项,core/public_url.py):反向代理后面请求的 Host 不一定是它
if address := public_base_url():
allowed.add(origin_of(address))
host = headers.get("x-forwarded-host") or headers.get("host")
if host:
scheme = headers.get("x-forwarded-proto") or scope.get("scheme") or "http"
@@ -92,7 +102,7 @@ class AuthMiddleware:
await self.app(scope, receive, send)
return
from app.api.deps import reset_principal, set_principal
from app.auth import current_mode
from app.auth import current_mode, refusal
# 登录设置在启动时读进缓存(main.py 的 _platform_startup),改设置的接口改完就刷新;这里不查库
mode = current_mode()
@@ -130,7 +140,11 @@ class AuthMiddleware:
return
if (scope["type"] == "http" and is_api and not principal.username and path not in _PUBLIC
and not path.startswith(_HOOKS)):
await _reply(send, 401, "请先登录", "auth_required")
await _reply(send, 401, "请先登录", AUTH_REQUIRED)
return
if scope["type"] == "http" and _is_docs(path) and (refused := refusal(principal, "builder")) is not None:
# 和路由级依赖(auth.require)同一个判断、同一句话:没登录 401,角色不够 403
await _reply(send, refused.status_code, refused.detail, refused.code)
return
token = set_principal(principal)
+1 -1
View File
@@ -198,7 +198,7 @@ def instantiate(blueprint: dict[str, Any], *, filled: dict[str, Any] | None = No
spec = _spec_of(blueprint)
ref = spec.get("template") or {}
template = get_template(str(ref.get("id") or ""))
problems = [i for i in blueprint_issues(spec, for_="trial") if i["level"] == "error"]
problems = [i for i in blueprint_issues(spec, for_="trial", graph=graph) if i["level"] == "error"]
if template is None:
from app.apps.spec import empty_spec
+85 -9
View File
@@ -520,8 +520,47 @@ def _container(spec: dict[str, Any], tokens: list[str]) -> tuple[dict[str, Any],
raise BpRefused(f"不能改这个位置:/{'/'.join(tokens)}")
def _set(spec: dict[str, Any], tokens: list[str], value: Any, source: str) -> None:
def is_freeform(spec: dict[str, Any]) -> bool:
from app.blueprints.templates._common import is_freeform as imports_graph
return imports_graph(spec)
#: 自由形的「谁来做」只定由谁做(05-已知遗留 (d)、遗留-Q5)
FREEFORM_PEOPLE_ONLY_WHO = "自由形蓝图的「谁来做」只定每一步由谁做;会签、时限、超时、加减步骤在画布上改"
FREEFORM_PEOPLE_USER = "「谁来做」要由你来定:派活给谁是人的决定,助手不替你选"
def _freeform_who(spec: dict[str, Any], tokens: list[str], value: Any, source: str, graph: Any) -> None:
"""自由形的 /people/<i>/who:i 是按导入的图列出的第几个岗位(freeform.people_rows 的岗位行,界面照那一份的 path 写),
绑定存进 spec.people[](id 就是岗位名、只有 who)。只收人和组:岗位、发起人在图上已经写定了。"""
from app.blueprints.templates import freeform
from app.blueprints.templates._common import with_graph
if len(tokens) != 3 or tokens[2] != "who":
raise BpRefused(FREEFORM_PEOPLE_ONLY_WHO, code="bp.template_mismatch")
if source != "user":
raise BpRefused(FREEFORM_PEOPLE_USER)
if graph is None:
raise BpRefused("这张蓝图还没有导入的图,没法按图定谁来做")
rows = [r for r in freeform.people_rows(with_graph(spec, graph)) if r["kind"] == "slot"]
slot = rows[_index(tokens[1], len(rows))]["slot"]
checked = _checked("people", "who", value, spec)
if checked and any(p["kind"] not in ("user", "group") for p in checked):
raise BpRefused("这里只选人或组:岗位、发起人在图上已经定了")
entries = spec.setdefault("people", [])
entry = next((e for e in entries if e.get("id") == slot), None)
if entry is None:
entry = {"id": slot}
entries.append(entry)
entry["who"] = _item(checked, source)
def _set(spec: dict[str, Any], tokens: list[str], value: Any, source: str, graph: Any = None) -> None:
card = tokens[0]
if card == "people" and is_freeform(spec):
_freeform_who(spec, tokens, value, source, graph)
return
if card == "access":
if len(tokens) != 2 or tokens[1] not in ACCESS_KINDS:
raise BpRefused("「谁能看」只有使用、查看全部、管理三档")
@@ -577,6 +616,8 @@ def _set(spec: dict[str, Any], tokens: list[str], value: Any, source: str) -> No
def _add(spec: dict[str, Any], tokens: list[str], value: Any, source: str) -> None:
card = tokens[0]
if card == "people" and is_freeform(spec):
raise BpRefused(FREEFORM_PEOPLE_ONLY_WHO, code="bp.template_mismatch")
if card == "access":
if len(tokens) not in (2, 3) or tokens[1] not in ACCESS_KINDS:
raise BpRefused("「谁能看」只有使用、查看全部、管理三档")
@@ -598,8 +639,13 @@ def _add(spec: dict[str, Any], tokens: list[str], value: Any, source: str) -> No
entries.insert(at, _new_entry(card, value, source, spec))
def _remove(spec: dict[str, Any], tokens: list[str], source: str) -> None:
def _remove(spec: dict[str, Any], tokens: list[str], source: str, graph: Any = None) -> None:
card = tokens[0]
if card == "people" and is_freeform(spec):
if len(tokens) != 3:
raise BpRefused(FREEFORM_PEOPLE_ONLY_WHO, code="bp.template_mismatch")
_freeform_who(spec, tokens, None, source, graph)
return
if card == "access":
if len(tokens) != 3 or tokens[1] not in ACCESS_KINDS:
raise BpRefused("「谁能看」要写明删哪一档的第几个")
@@ -655,8 +701,9 @@ def _template(spec: dict[str, Any], op: BpOp, source: str) -> None:
prefill(template.id, spec)
def apply_bp_op(spec: dict[str, Any], op: dict[str, Any], *, source: str) -> dict[str, Any]:
"""把一个 bp_* 操作落到蓝图上,交回新的蓝图(不改入参)。source 是 user / assistant。不合规时抛 BpRefused。"""
def apply_bp_op(spec: dict[str, Any], op: dict[str, Any], *, source: str, graph: Any = None) -> dict[str, Any]:
"""把一个 bp_* 操作落到蓝图上,交回新的蓝图(不改入参)。source 是 user / assistant。不合规时抛 BpRefused。
graph 是蓝图存的图:自由形的「谁来做」按它列岗位(没给时那张卡片改不了)。"""
if source not in BP_SOURCES:
raise BpRefused(f"操作的来源只能是用户或助手,当前为「{source}」")
if not isinstance(op, dict) or op.get("op") not in BP_OPS:
@@ -677,11 +724,11 @@ def apply_bp_op(spec: dict[str, Any], op: dict[str, Any], *, source: str) -> dic
if source != "user" and (only := next((k for k in USER_ONLY if tuple(tokens[:len(k)]) == k), None)):
raise BpRefused(_USER_ONLY_TEXT[only[-1]])
if parsed.op == "bp_set":
_set(out, tokens, parsed.value, source)
_set(out, tokens, parsed.value, source, graph)
elif parsed.op == "bp_add":
_add(out, tokens, parsed.value, source)
else:
_remove(out, tokens, source)
_remove(out, tokens, source, graph)
try:
return parse_blueprint(out).model_dump(mode="json", by_alias=True)
except ValidationError as e:
@@ -708,9 +755,9 @@ def _value(item: Any) -> Any:
return item.get("value") if isinstance(item, dict) else None
def blueprint_issues(spec: dict[str, Any], *, for_: str) -> list[dict[str, Any]]:
def blueprint_issues(spec: dict[str, Any], *, for_: str, graph: Any = None) -> list[dict[str, Any]]:
"""蓝图检查(§3.13 bp.*):for_ = trial(缺项挡试运行)/ publish(待确认项挡发布)/ personal。
每条 {code, level, path, message}。"""
每条 {code, level, path, message}。graph 是蓝图存的图:自由形的「谁来做」按它查(没给时不查这一张)。"""
from app.blueprints.templates import get_template
if for_ not in ISSUE_PURPOSES:
@@ -743,7 +790,10 @@ def blueprint_issues(spec: dict[str, Any], *, for_: str) -> list[dict[str, Any]]
_data_issues(spec, "data" in required, add)
if "cadence" in required and _value(spec.get("cadence")) is None:
add("bp.missing", "/cadence", "「多久更新」还没定:每天、每周还是每月,几点,统计哪一期")
if "people" in used:
if "people" in used and template.imports_graph:
if graph is not None and for_ != "personal":
out += _freeform_people_issues(spec, graph, publish)
elif "people" in used:
_people_issues(spec, "people" in required, publish, add)
if "fields" in used:
_field_issues(spec, "fields" in required, add)
@@ -836,6 +886,32 @@ def _people_issues(spec: dict[str, Any], required: bool, publish: bool, add: Cal
f"「{label}」的条件拿「{threshold['field']}」比大小,它要是数字字段")
def _freeform_people_issues(spec: dict[str, Any], graph: Any, publish: bool) -> list[dict[str, Any]]:
"""自由形的「谁来做」(freeform.people_rows):岗位、核实人没绑的报 bp.people_unbound(试运行 warning:待办派给试运行的
人自己;发布 error),路径就是那一行的 path,带上 slot(发布单据此去掉同一件事的 app.slot_unbound / app.reconciler_unbound)。
图里已经没有的岗位、卡片上还留着绑定的,照实提醒(warning),不静默删掉。"""
from app.blueprints.templates import freeform
from app.blueprints.templates._common import with_graph
out: list[dict[str, Any]] = []
level = "error" if publish else "warning"
rows = freeform.people_rows(with_graph(spec, graph))
for row in rows:
if row["kind"] not in ("slot", "reconciler") or row["who"]:
continue
message = (f"{row['label']}还没定:{row['detail']},写入外部系统的结果不明时要有人去目标系统里核实"
if row["kind"] == "reconciler" else f"{row['label']}还没定谁来处理({row['detail']})")
out.append({"code": "bp.people_unbound", "level": level, "path": row["path"], "message": message,
"slot": row["slot"]})
known = {r["slot"] for r in rows if r["kind"] == "slot"}
for step in spec.get("people") or []:
slot = str(step.get("id") or "")
if slot and slot not in known and _value(step.get("who")):
out.append({"code": "bp.template_mismatch", "level": "warning", "path": "/people",
"message": f"岗位「{slot}」在导入的图里已经没有了:卡片上留着的这条绑定不会生效"})
return out
def _field_issues(spec: dict[str, Any], required: bool, add: Callable[..., None]) -> None:
fields = spec.get("fields") or []
if required and not fields:
+33 -1
View File
@@ -70,9 +70,13 @@ NOT_INSTANTIATED = "plan.not_instantiated"
GRAPH_CHANGED = "plan.graph_changed"
UNTRIALED = "plan.untrialed"
TRIAL_FAILED = "plan.trial_failed"
#: 最近一次试运行在某一步被人驳回、没有走完(cancelled + end_reason=rejected):人的决定,不是故障,不挡发布、照实说
TRIAL_REJECTED = "plan.trial_rejected"
TRIAL_DIFFERS = "plan.trial_differs"
#: 蓝图检查:写回的结果不明时没人核实(blueprints/ops.blueprint_issues)
RECONCILER_UNBOUND = "bp.reconciler_unbound"
#: 蓝图检查:「谁来做」有一步还没定由谁处理;自由形的带着 slot(哪个岗位,核实人是 apps/derive.RECONCILER)
PEOPLE_UNBOUND = "bp.people_unbound"
ADMIN_REQUIRED = "plan.admin_required"
DOWNGRADE_ADMIN = "plan.downgrade_admin"
PREFLIGHT = "plan.preflight"
@@ -237,8 +241,10 @@ async def _simulated_pins(session: AsyncSession, spec: dict[str, Any], target: _
def bindings_of(bp: Blueprint) -> dict[str, Any]:
"""岗位绑定:「谁来做」卡片上写了具体的人或组的,按模板定的岗位名绑上(templates/_common.bindings_of)。"""
from app.blueprints.templates._common import bindings_of as from_cards
from app.blueprints.templates._common import with_graph
return from_cards(bp.spec or {})
# 带上蓝图存的图:自由形的核实人要不要绑按导入的图算(05-已知遗留 (d))
return from_cards(with_graph(bp.spec or {}, bp.graph))
def grant_items(bp: Blueprint) -> list[tuple[str, str, str]]:
@@ -352,6 +358,14 @@ async def _preflight(session: AsyncSession, bp: Blueprint, target: _Target, *, l
issues = [i for i in issues if i.get("code") != "app.reconciler_unbound"
and not (i.get("code") == "app.slot_unbound" and i.get("path") == f"slots.{RECONCILER}")]
if unbound := {str(i["slot"]) for i in out if i.get("code") == PEOPLE_UNBOUND and i.get("slot")}:
# 自由形的「谁来做」:岗位、核实人没绑,蓝图检查已经指到卡片上那一行(bp.people_unbound 带着 slot),应用检查说的是
# 同一件事,不再重复列(同上面核实岗位的办法)
from app.apps.derive import RECONCILER
issues = [i for i in issues if not (i.get("code") == "app.slot_unbound"
and str(i.get("path") or "").removeprefix("slots.") in unbound)
and not (i.get("code") == "app.reconciler_unbound" and RECONCILER in unbound)]
out += [{**i, "source": "app"} for i in issues]
# 授权:给的人和组要在名录里(同授权接口)
_, problem = await grants_problem(session, grant_items(bp))
@@ -363,6 +377,12 @@ async def _preflight(session: AsyncSession, bp: Blueprint, target: _Target, *, l
if is_trial_of(trial, bp) and trial.status == "failed":
why = (trial.error or "").strip().splitlines()[0][:200] if (trial.error or "").strip() else ""
out.append(_problem(TRIAL_FAILED, f"最近一次试运行失败了{f':{why}' if why else ''}", source="plan"))
elif is_trial_of(trial, bp) and trial.status == "cancelled" and trial.end_reason == "rejected":
# 搭建者在试运行里驳回了写入、审批:那是他的决定,不是图出了错,不说成「试运行失败了」、也不挡发布;只照实说
# 没走完——驳回之后的步骤没用真数看过
where = _node_title(trial.graph, trial.error_node_id)
out.append(_problem(TRIAL_REJECTED, f"最近一次试运行{f'在「{where}」' if where else ''}被驳回、没有走完",
"warning", source="plan"))
elif not is_trial_of(trial, bp) or trial.status != "succeeded":
out.append(_problem(UNTRIALED, "还没有成功的试运行:发布之前没用真数看过结果", "warning", source="plan"))
elif graph_hash(trial.graph or {}) != target.digest:
@@ -370,6 +390,17 @@ async def _preflight(session: AsyncSession, bp: Blueprint, target: _Target, *, l
return [_with_who(p) for p in out]
def _node_title(graph: Any, node_id: str | None) -> str | None:
"""节点在图上的名字(没有名字的给 id);没有节点时为 None。"""
if not node_id:
return None
for node in (graph or {}).get("nodes") or []:
if isinstance(node, dict) and node.get("id") == node_id:
data = node.get("data") if isinstance(node.get("data"), dict) else {}
return str(data.get("label") or node_id)
return node_id
def _with_who(issue: dict[str, Any]) -> dict[str, Any]:
"""问题带上白话(没登记白话的照旧只有技术说法)和归谁处理(you / assistant / admin,摘要按它分三栏)。"""
from app.plain import who_of, with_plain
@@ -428,6 +459,7 @@ def _summary(bp: Blueprint, target: _Target, *, level: str, first_run: bool, opt
if current and current > target.version else ""
codes = {p.get("code") for p in problems}
tried = ";发布的内容和最近一次试运行不同(试运行之后又改过),建议再试一次" if TRIAL_DIFFERS in codes else \
";最近一次试运行被驳回、没有走完" if TRIAL_REJECTED in codes else \
";还没有用真数试运行过" if UNTRIALED in codes else ""
if target.published_as == level:
say("publish_workflow", f"「{name}」v{target.version} 已经按{LEVEL_TEXT[level]}级别发布过,不重复发布{tried}")
+15 -5
View File
@@ -183,7 +183,7 @@ async def apply_ops_and_save(session: AsyncSession, blueprint: Any, ops: list[di
try:
op = await _verified_reuse(session, raw)
before = _at(spec, op.get("path"))
spec = apply_bp_op(spec, op, source=source)
spec = apply_bp_op(spec, op, source=source, graph=blueprint.graph)
except BpRefused as e:
if not partial:
raise
@@ -313,7 +313,7 @@ def blueprint_issues_of(blueprint: Any) -> list[dict[str, Any]]:
from app.plain import with_plain
spec = blueprint.spec or {}
issues = blueprint_issues(spec, for_="personal" if blueprint.scope == "personal" else "publish")
issues = blueprint_issues(spec, for_="personal" if blueprint.scope == "personal" else "publish", graph=blueprint.graph)
for joint_id, state in (blueprint.joints or {}).items():
if isinstance(state, dict) and state.get("status") in ("pending", "refused"):
issues.append({"code": "tpl.joint_unfilled" if state["status"] == "pending" else "tpl.joint_refused",
@@ -375,11 +375,19 @@ async def blueprint_view(session: AsyncSession, blueprint: Any) -> dict[str, Any
return blueprint_out(blueprint, extra=await data_source_issues(session, blueprint))
def _derived_people(blueprint: Any) -> list[dict[str, Any]]:
from app.blueprints.templates import freeform
from app.blueprints.templates._common import with_graph
return freeform.people_rows(with_graph(blueprint.spec or {}, getattr(blueprint, "graph", None)))
def blueprint_out(blueprint: Any, *, issues: bool = True, extra: list[dict[str, Any]] | None = None) -> dict[str, Any]:
"""一张蓝图交给前端的样子(frontend/src/typesBlueprint.ts 的 Blueprint)。关节只给状态和模板声明的说法、类型、分组
(JointState:status、by、note、label、kind、group),不给填的值(值在图上看)。reconciler_needed:写回卡片上要不要
摆「结果不明时谁去核实」(目标能力没登记查询,templates/_common.needs_reconciler),界面不自己判。"""
from app.blueprints.templates._common import needs_reconciler
摆「结果不明时谁去核实」(目标能力没登记查询,templates/_common.needs_reconciler),界面不自己判。derived_people:自由形
按导入的图列出的「谁来做」(templates/freeform.people_rows,05-已知遗留 (d)),别的模板为空。"""
from app.blueprints.templates._common import is_freeform, needs_reconciler, with_graph
joints = {jid: {k: v for k, v in state.items() if k in ("status", "by", "note", "label", "kind", "group")}
for jid, state in (blueprint.joints or {}).items() if isinstance(state, dict)}
@@ -389,7 +397,9 @@ def blueprint_out(blueprint: Any, *, issues: bool = True, extra: list[dict[str,
"template_version": blueprint.template_version, "source": blueprint.source or {},
"conversation_id": blueprint.conversation_id, "workflow_id": blueprint.workflow_id, "app_id": blueprint.app_id,
"last_trial_run_id": blueprint.last_trial_run_id, "graph_hash": blueprint.graph_hash, "joints": joints,
"copied_from": blueprint.copied_from, "reconciler_needed": needs_reconciler(blueprint.spec or {}),
"copied_from": blueprint.copied_from,
"reconciler_needed": needs_reconciler(with_graph(blueprint.spec or {}, getattr(blueprint, "graph", None))),
"derived_people": _derived_people(blueprint) if is_freeform(blueprint.spec or {}) else None,
"created_at": blueprint.created_at.isoformat() if blueprint.created_at else None,
"updated_at": blueprint.updated_at.isoformat() if blueprint.updated_at else None,
}
@@ -78,6 +78,19 @@ def context(bp: dict[str, Any]) -> dict[str, Any]:
return ctx if isinstance(ctx, dict) else {}
def with_graph(spec: dict[str, Any], graph: Any) -> dict[str, Any]:
"""只有卡片(Blueprint.spec)、没有实例化上下文的地方,带上蓝图存的图再交给模板的函数:自由形的「谁来做」、要不要
核实人都按导入的图算(freeform.people_rows / needs_reconciler)。"""
return {**spec, CONTEXT_KEY: {**context(spec), "graph": graph}}
def is_freeform(spec: dict[str, Any]) -> bool:
"""卡片上选的是导入图的模板(自由形)。"""
ref = spec.get("template")
template = get_template(str(ref.get("id") or "")) if isinstance(ref, dict) else None
return template is not None and template.imports_graph
def app_name(bp: dict[str, Any], fallback: str) -> str:
"""应用和工作流的名字:实例化时给的名字,没给就用目的的前一段。"""
name = str(context(bp).get("name") or "").strip()
@@ -434,6 +447,12 @@ def needs_reconciler(bp: dict[str, Any]) -> bool:
蓝图(reconciler_needed)共用这一份。"""
ref = bp.get("template")
template = get_template(str(ref.get("id") or "")) if isinstance(ref, dict) else None
if template is not None and template.imports_graph:
# 自由形没有「写回哪里」:按导入的图里的写入算(bp 要带着图,with_graph),核实人存在 /writeback/reconciler
# (05-已知遗留 (d))
from app.blueprints.templates import freeform
return freeform.needs_reconciler(bp)
if template is None or "writeback" not in template.cards:
return False
capability = value((bp.get("writeback") or {}).get("capability"))
+67 -1
View File
@@ -1,5 +1,11 @@
"""自由形 freeform(§3.3):从已有的运行(问数据的一轮回答,经 from-run 提取)或已发布的工作流版本导入,图原样用,
没有关节。蓝图只有目的、谁能看、多久更新三张卡片;要改图只能回画布(搭建者)。
没有关节。蓝图有目的、谁能看、多久更新、谁来做四张卡片;要改图只能回画布(搭建者)。
「谁来做」(docs/design/next-round/05-已知遗留.md (d)):骨架就是导入的图,卡片只填「谁」这个关节——按图列出用到的岗位
(人工节点派给谁、超时升级给谁、写入的发送确认派给谁,people_rows),写入能力没登记查询时再加一行「写入结果不明时谁去核实」。
只定由谁做:会签、时限、超时在画布上改(改了会改图,图哈希就和发布的那一版对不上)。绑定存在 spec.people[](Step.id 就是
岗位名、只写 who,_common.slot_of / bindings_of 原样生效),核实人存在 /writeback/reconciler。以前这张卡片不存在,第一次
发布挡在 app.slot_unbound / app.reconciler_unbound 上,白话却叫人去「谁来做」里选人。
图由实例化的上下文给(graph,导入时存在蓝图上)。应用骨架按 apps/derive.py 的派生规则:有人工节点或写入的按流程,
否则按看板;写入节点按能力登记的进程内快照认(和运行时同一个判断,capabilities.registry.writes_externally)。
@@ -38,6 +44,65 @@ def _writes(graph: dict[str, Any]) -> dict[str, str]:
return out
#: 「谁来做」一行里每处用途怎么说(apps/tree.SLOT_ROLES,加上核实)
USE_TEXT = {"assignee": "处理", "escalation": "超时升级", "confirm": "发送前确认", "reconcile": "写入结果不明时去核实"}
#: 不按岗位派的步骤(只读行):发起人自己、按表达式算出来的人、等外部系统回调
FIXED_TEXT = {"initiator": "由发起人自己处理", "expression": "按图里的表达式算出由谁处理(在画布上改)",
"hook": "等外部系统回调,不派给人"}
#: 核实人那一行的名字
RECONCILER_LABEL = "写入结果不明时谁去核实"
def people_rows(bp: dict[str, Any]) -> list[dict[str, Any]]:
"""按导入的图列出「谁来做」:[{kind, slot, label, detail, path, who, uses}]。
kind=slot:图里引用的一个岗位(同一个岗位用在几处合成一行,按节点出现的顺序),path 是 /people/<i>/who(i 是岗位行的
序号),who 是卡片上绑的人或组;kind=reconciler:写入能力没登记查询时的核实人,path 是 /writeback/reconciler;
kind=initiator / expression / hook:不按岗位派的步骤,只读(path 为空)。label 用节点在图上的名字拼(岗位名是画布上
随手写的 id),detail 写全每一处用途。界面照这一份画,不自己从图里推。"""
from app.apps.derive import RECONCILER
from app.apps.tree import node_map, slot_refs
graph = imported(bp)
writes = _writes(graph)
nodes = node_map(graph)
bound = {str(step.get("id")): c.value(step.get("who")) for step in bp.get("people") or [] if isinstance(step, dict)}
rows: list[dict[str, Any]] = []
by_slot: dict[str, dict[str, Any]] = {}
for ref in slot_refs(graph, writes):
row = by_slot.get(ref["slot"])
if row is None:
row = by_slot[ref["slot"]] = {"kind": "slot", "slot": ref["slot"], "path": f"/people/{len(by_slot)}/who",
"who": bound.get(ref["slot"]), "uses": []}
rows.append(row)
row["uses"].append({"node_id": ref["node_id"], "title": ref["title"], "role": ref["role"]})
unchecked = [(nid, cap) for nid, cap in writes.items() if not c.write_lookup(cap)]
if unchecked:
rows.append({"kind": "reconciler", "slot": RECONCILER, "path": "/writeback/reconciler",
"who": c.value((bp.get("writeback") or {}).get("reconciler")),
"uses": [{"node_id": nid, "title": nodes[nid]["title"], "role": "reconcile"} for nid, _ in unchecked]})
for nid, n in nodes.items():
if n["type"] != "human" and nid not in writes:
continue
assignee = n["config"].get("assignee")
kind = assignee.get("kind") if isinstance(assignee, dict) else None
if kind in FIXED_TEXT:
rows.append({"kind": kind, "slot": None, "path": None, "who": None,
"uses": [{"node_id": nid, "title": n["title"],
"role": "assignee" if n["type"] == "human" else "confirm"}]})
for row in rows:
titles = list(dict.fromkeys(u["title"] for u in row["uses"]))
row["label"] = RECONCILER_LABEL if row["kind"] == "reconciler" else "、".join(f"「{t}」" for t in titles)
row["detail"] = FIXED_TEXT[row["kind"]] if row["kind"] in FIXED_TEXT else ";".join(
f"「{u['title']}」{USE_TEXT[u['role']]}" for u in row["uses"])
return rows
def needs_reconciler(bp: dict[str, Any]) -> bool:
"""导入的图里有写入能力没登记查询(同 app_skeleton 加核实岗位的判断)。"""
return any(not c.write_lookup(cap) for cap in _writes(imported(bp)).values())
def app_skeleton(bp: dict[str, Any], graph: dict[str, Any]) -> dict[str, Any]:
from app.apps.derive import ENTRY, INSTANCE_VIEW, PIN, VIEW, _entries, _results
from app.apps.forms import input_schema
@@ -103,6 +168,7 @@ TEMPLATE = register(TemplateDef(
applies_when="用户已经有一次满意的问数据回答,或者一个已发布的工作流,想直接做成应用;需求在五个场景之外时也用它。",
cards={"purpose": "必填:一句话说清这个应用做什么",
"cadence": "可选:要不要定期自动出数",
"people": "可选:只定每一步由谁做;会签、时限、超时在画布上改",
"access": "可选:谁能用、谁能看全部、谁能管理"},
personal="导入的图满足问数据的规则(只读、没有人工节点和写入)时可以",
skeleton=skeleton, app_skeleton=app_skeleton, schedule=schedule, tags=["导入"],
+1 -1
View File
@@ -147,7 +147,7 @@ async def materialize(session: AsyncSession, bp: Blueprint, *, principal: Princi
digest = graph_hash(graph or {})
app_spec = _draft_pins(result.app_spec, bp, digest)
# 实例化交回的是挡路的 error;蓝图检查里的提醒(模板版本旧了之类)另外带上
warnings = [i for i in blueprint_issues(bp.spec or {}, for_="trial") if i.get("level") != "error"]
warnings = [i for i in blueprint_issues(bp.spec or {}, for_="trial", graph=bp.graph) if i.get("level") != "error"]
problems = [with_plain(p) for p in [*result.problems, *warnings]]
bp.graph, bp.graph_hash, bp.joints = graph, digest, result.joints
bp.updated_by = principal.username if principal else bp.updated_by
+3 -3
View File
@@ -1237,8 +1237,8 @@ def build_bundle(
) -> dict[str, Any]:
"""一次已封存运行 → 页面里的数据(bundle-data)。
- run:运行记录的几样(id、workflow_id、workflow_name、version、version_hash、run_class、status、
started_at、finished_at)
- run:运行记录的几样(id、workflow_id、workflow_name、version、version_hash、run_class、status、end_reason、
started_at、finished_at)。end_reason 分开同是 cancelled 的被驳回和手动停止(05-已知遗留 (a)),不在封存范围内
- events:这次运行的全部事件 (seq, type, node_id, data),按数据库里 seq 的顺序。和封存核对
(runner.verify_manifest)读的是同一个顺序:seq 相同的几条,浏览器按这里的先后排,清单才算得一样
- manifest_seq / manifest_hash:封存范围(老运行是按结束事件推算的那个,legacy 为真)和封存哈希
@@ -1306,7 +1306,7 @@ def build_bundle(
"meta": {
"run_id": run.get("id"), "workflow_id": run.get("workflow_id"), "workflow_name": display_name(run),
"version": run.get("version"), "version_hash": run.get("version_hash"),
"run_class": run.get("run_class"), "status": run.get("status"),
"run_class": run.get("run_class"), "status": run.get("status"), "end_reason": run.get("end_reason"),
"started_at": _iso(run.get("started_at")), "finished_at": _iso(run.get("finished_at")),
"manifest_seq": manifest_seq, "manifest_hash": manifest_hash, "legacy": bool(legacy),
"tier": _tier(sealed), "exported_at": exported_at,
+19 -6
View File
@@ -49,7 +49,9 @@
const KIND_LABEL = { report_doc: '报告文档', metric_set: '口径卡', query: '查询快照' }
const TIER_LABEL = { formal: '完整出具', degraded: '降档出具', withheld: '不予出具' }
const CLASS_LABEL = { formal: '正式运行', exploratory: '探索运行' }
const STATUS_LABEL = { succeeded: '已完成', failed: '已失败', cancelled: '已取消' }
// rejected:被审批人驳回结束(运行记录 cancelled + end_reason=rejected)。封存里它和手动停止是同一种结束事件(运行取消),
// 分开二者的原因在那条事件的内容里,内容不随文件附带:见 checkMeta
const STATUS_LABEL = { succeeded: '已完成', failed: '已失败', cancelled: '已取消', rejected: '被驳回' }
/** 结束类事件 → 它说明的运行结果(runner:成功、失败、取消各发一条,落库之后才封存) */
const TERMINAL = { 'run.finished': 'succeeded', 'run.failed': 'failed', 'run.cancelled': 'cancelled' }
const EVENT_LABEL = {
@@ -84,6 +86,7 @@
range: (seq, count) => `第 1–${seq} 条事件(范围内 ${count} 条)`,
rangeLegacy: '(早期运行,范围按结束事件推算)',
markPass: '已核对',
markPassRejected: '已核对运行取消,被驳回取自运行记录',
markFail: (claim) => `文件里写的是「${claim}」,见核对明细 ⑥`,
markNa: '无法核对',
noReports: '这次运行没有报告文档,下方只核对运行记录。',
@@ -208,6 +211,8 @@
inputFrom: (query, row, column) => (query ? `查询 ${query}` : '') + (Number.isInteger(row) ? ` · 第 ${row + 1} 行` : '') + (column ? ` · ${column}` : ''),
inputNotAttached: '快照未随文件附带(见核对明细 ③),输入的值按口径卡的记录展示',
statusOk: (seq, what) => `与第 ${seq} 条事件(${what})一致`,
statusOkRejected: (seq, what) => `与第 ${seq} 条事件(${what})一致。被驳回和手动停止在封存里是同一种结束事件,`
+ '「被驳回」取自运行记录(这条事件的内容不随文件附带),需要在系统中对照',
statusMismatch: (claim, seq, what) => `文件里写的是「${claim}」,与封存范围内最后一条结束类事件(第 ${seq} 条,${what})不符`,
statusLegacy: '早期运行在结束事件落库之前封存,结束事件不在封存范围内,运行结果无法核对',
statusNoEvent: '封存范围内没有运行结束、失败或取消的事件,运行结果无法核对',
@@ -1509,7 +1514,8 @@
*/
function checkMeta(meta, out) {
const items = []
const add = (field, shown, claim, result, reason) => items.push({ field, shown, claim, result, reason })
// mark:显示值旁边那一句的特例(默认按结果写已核对 / 不通过 / 无法核对)
const add = (field, shown, claim, result, reason, mark) => items.push({ field, shown, claim, result, reason, mark })
const order = out.order
const sealOk = out.manifest[0].result === PASS
const label = (map, value) => map[value] || (value ? String(value) : '—')
@@ -1519,7 +1525,10 @@
// 它就是范围内的最后一条;早期运行在 run.finished 落库之前封存,成功结束的那条在范围外,失败、取消的那条在范围的最后。
// 先失败、经继续运行接着跑的,范围里那条运行失败之后运行又继续了,它不是运行的结束:拿它去对,元信息改成「已失败」
// 也会被标成已核对。真正截止的那条不在范围里时,只有早期运行成功结束说得通(无法核对),别的说法都没有事件印证
const statusClaim = label(STATUS_LABEL, meta.status)
// 被驳回(cancelled + end_reason=rejected)照运行记录写「被驳回」,不写成手动停止的「已取消」;能核对的只有结束事件
// 是运行取消这一半,另一半写明取自运行记录
const rejected = meta.status === 'cancelled' && meta.end_reason === 'rejected'
const statusClaim = rejected ? STATUS_LABEL.rejected : label(STATUS_LABEL, meta.status)
const lastSeq = order.length ? order[order.length - 1].seq : null
const final = [...order].reverse().find((ev) => TERMINAL[ev.type])
// 同一序号的几条(老数据里有)都算最后:结束事件和别的事件撞了序号,不能因此说它之后运行又继续了
@@ -1527,8 +1536,12 @@
if (!sealOk) add('status', statusClaim, statusClaim, NA, T.metaNeedsManifest)
else if (end) {
const ok = TERMINAL[end.type] === meta.status
add('status', label(STATUS_LABEL, TERMINAL[end.type]), statusClaim, ok ? PASS : FAIL,
ok ? T.statusOk(end.seq, what(end)) : T.statusMismatch(statusClaim, end.seq, what(end)))
if (ok && rejected) {
add('status', statusClaim, statusClaim, PASS, T.statusOkRejected(end.seq, what(end)), T.markPassRejected)
} else {
add('status', label(STATUS_LABEL, TERMINAL[end.type]), statusClaim, ok ? PASS : FAIL,
ok ? T.statusOk(end.seq, what(end)) : T.statusMismatch(statusClaim, end.seq, what(end)))
}
} else if (meta.legacy && meta.status === 'succeeded') add('status', statusClaim, statusClaim, NA, T.statusLegacy)
// 显示的值写「—」:范围里没有说明运行怎样结束的事件,不能把文件里写的当成核对出来的
else if (final) add('status', '—', statusClaim, FAIL, T.statusContinued(statusClaim, final.seq, what(final)))
@@ -1899,7 +1912,7 @@
const checked = (field, fallback) => {
const item = out.meta.find((m) => m.field === field)
if (!item) return fallback
const note = item.result === PASS ? T.markPass : item.result === FAIL ? T.markFail(item.claim) : T.markNa
const note = item.mark ?? (item.result === PASS ? T.markPass : item.result === FAIL ? T.markFail(item.claim) : T.markNa)
return h('span', { 'data-meta': field, 'data-result': item.result }, item.shown,
h('span', { class: item.result === FAIL ? 'bad' : 'muted' }, `(${note})`))
}
+5
View File
@@ -0,0 +1,5 @@
"""问数据(一轮:提问 → 建图 → 运行 → 复核)在后端的部分。
- turn.py:一轮里的几个「决定」(要不要运行、输入怎么填、要不要带原图重来、重来时说什么),问数据页和评测走同一套;
- questions.py:问过的问题原文和它们的聚类,全站只有这一份(治理接口、以后的口径账本、评测都调它)。
"""
+115
View File
@@ -0,0 +1,115 @@
"""问过的问题和它们的聚类:全站只有这一份(docs/design/next-round/00-总方案.md X2)。
同一个问题被反复问,就该长成模板里的固定一节、沉淀成口径——探索问题聚类(/governance/exploratory-clusters)、以后口径
账本的「可沉淀的问题」、评测的问题重复度(E3)量的都是这件事,各算一份的话几个数对不上,所以取问题和聚类只写在这里。
**取什么。** 只取问题原文,不取整份输入:以前把运行的整份 input(含 history)拿去聚,同一段对话里前几轮的问答会把不同的
问题聚成一类,重复度偏高。
- 问数据:一轮一条(conversation_turns.question,用户原本问的那句话)。同一轮里的自动重来、用户点的「重试」只是换了这一轮
挂着的运行,不多出一条——按运行数的话,答坏了自动重来一次就被算成问了两遍。没挂运行的轮次(只回了一句、助手反问)
也算:问题照样被问过。
- 别的探索运行(画布试跑、蓝图试运行、接口……):输入里有 question 这个字段的取它;没有的不收(输入里别的字段不是问题)。
问数据的运行由上面一轮一条代表,这里不再数。
**不收什么。** 系统派生的运行(评测、口径对账、口径召回:engine/origin.counted_runs)不收,评测题不能把聚类刷高;
这个人读不了的只给拥有者的运行(auth/access.hidden_runs)不收;正式运行不收(探索才谈得上「沉淀」)。
聚类用 memory/embeddings 的向量贪心聚:按时间从新到旧,和某一类的种子足够像就并进去,否则自立一类。本地向量是词频
哈希、语义能力弱,结果是下限——调用方据 embedder 照实写明用的是哪个向量模型。
"""
from __future__ import annotations
from collections.abc import Sequence
from dataclasses import dataclass, field
from datetime import datetime
from typing import Any
from sqlalchemy import func, not_, select
from sqlalchemy.ext.asyncio import AsyncSession
#: 缺省的相似度阈值(同治理接口一直以来的 0.62)
DEFAULT_THRESHOLD = 0.62
@dataclass(frozen=True)
class Question:
"""问过的一个问题。turn_id 是问数据的那一轮(别的探索运行没有);run_id 是它挂着的运行(只回了一句的轮次没有)。"""
text: str
at: datetime | None
run_id: str | None = None
turn_id: str | None = None
workflow: str | None = None
origin: str | None = None
@dataclass
class QuestionCluster:
"""一类问题:members 按时间从新到旧,第一个是种子。"""
members: list[Question] = field(default_factory=list)
@property
def count(self) -> int:
return len(self.members)
async def recent_questions(session: AsyncSession, principal: Any, *, limit: int) -> list[Question]:
"""最近问过的问题,从新到旧,最多 limit 条(取法见模块开头)。principal 用来去掉这个人读不了的只给拥有者的运行。"""
from app.auth.access import hidden_runs
from app.db.models import Conversation, ConversationTurn, Run
from app.engine.origin import counted_runs
hidden = hidden_runs(principal)
turns = (await session.execute(
select(ConversationTurn.id, ConversationTurn.question, ConversationTurn.created_at, ConversationTurn.run_id,
Run.workflow_name)
.join(Conversation, Conversation.id == ConversationTurn.conversation_id)
.outerjoin(Run, Run.id == ConversationTurn.run_id)
.where(Conversation.kind == "chat", func.trim(ConversationTurn.question) != "", counted_runs(), not_(hidden))
.order_by(ConversationTurn.created_at.desc()).limit(limit)
)).all()
runs = (await session.execute(
select(Run.id, Run.input, Run.workflow_name, Run.created_at, Run.origin_kind)
.where(Run.run_class != "formal", func.coalesce(Run.origin_kind, "") != "chat", counted_runs(), not_(hidden),
func.json_type(Run.input, "$.question") == "text")
.order_by(Run.created_at.desc()).limit(limit)
)).all()
out = [Question(text=question.strip(), at=at, run_id=run_id, turn_id=turn_id, workflow=workflow, origin="chat")
for turn_id, question, at, run_id, workflow in turns]
for run_id, payload, workflow, at, origin in runs:
text = (payload or {}).get("question") if isinstance(payload, dict) else None
if isinstance(text, str) and text.strip():
out.append(Question(text=text.strip(), at=at, run_id=run_id, workflow=workflow, origin=origin))
out.sort(key=lambda q: q.at.timestamp() if q.at is not None else 0.0, reverse=True)
return out[:limit]
async def cluster_questions(questions: Sequence[Question], *, threshold: float = DEFAULT_THRESHOLD,
) -> list[QuestionCluster]:
"""向量贪心聚类(顺序同 questions:从新到旧)。"""
from app.memory import embeddings
if not questions:
return []
vectors = await embeddings.embed_texts([q.text for q in questions])
clusters: list[tuple[int, QuestionCluster]] = []
for i, question in enumerate(questions):
for seed, cluster in clusters:
if embeddings.cosine(vectors[i], vectors[seed]) >= threshold:
cluster.members.append(question)
break
else:
clusters.append((i, QuestionCluster(members=[question])))
return [cluster for _, cluster in clusters]
def embedder() -> str:
"""聚类用的是哪个向量模型(memory/embeddings.embedder_id):本地哈希的结果只是下限,调用方照实写出来。"""
from app.memory import embeddings
return embeddings.embedder_id()
__all__ = ["DEFAULT_THRESHOLD", "Question", "QuestionCluster", "cluster_questions", "embedder", "recent_questions"]
+129
View File
@@ -0,0 +1,129 @@
"""问数据一轮里的几个「决定」(细案 01 做法 1,总方案 2.1)。
一轮问数据是:建图(助手的生成过程,api/copilot.answer_ops)→ 要不要运行 → 运行(输入怎么填)→ 复核 → 要不要带着原图
重来一次、重来时说什么。以前这几个决定都写在问数据页里(frontend/src/store/chat.ts 的 plan / launch / settle / retry),
后端跑不了一轮问数据:评测要么照着抄一份(迟早走样,量出来的不是用户拿到的那一轮),要么做不了;这也违反「界面层不放
逻辑」。现在决定写在这里,是纯函数:
- next_step:生成过程交出结局之后做什么;
- fill_input:用户的问题填进这张图的入口字段;
- retry_after_review:复核之后要不要带着原图重来一次;
- retry_instruction:重来时交给助手的那句话(两种措辞)。
接口把决定交给问数据页:/copilot/generate-stream 的 final 带 next(run | ready | fail),/copilot/review 的结果带
retry_instruction(要重来时是那句话,不重来是 null)。问数据页照后端给的走;老后端没有这两个字段时退回它自己那份同样的
规则(tests/test_chat_turn_policy.py 逐条对照两边的常量和措辞)。评测(第 3 波)用同一组函数把一轮串起来。
只搬「决定」:重连、断点续看、取消、复核等多久这些仍在问数据页(把整轮编排搬到后端要重做这些,单独立项)。
"""
from __future__ import annotations
import re
from collections.abc import Mapping
from typing import Any
#: 一轮最多自动试几次(含第一次):第二次还不行多半是问题本身没说清,该让用户介入(同 chat.ts 的 MAX_ATTEMPTS)
MAX_ATTEMPTS = 2
#: final 之后的三种走法(final.next):运行、只建不跑(摆着等用户点运行)、判失败(没过检查的图不拿去跑)
NEXT_RUN, NEXT_READY, NEXT_FAIL = "run", "ready", "fail"
NEXT_STEPS = (NEXT_RUN, NEXT_READY, NEXT_FAIL)
#: 别的结局:直接回答了(reply)、助手反问了等用户答(asked)、没自查过的老服务端交来没过检查的图时重来一次(retry)
NEXT_REPLY, NEXT_ASKED, NEXT_RETRY = "reply", "asked", "retry"
#: 问题往哪个入口字段放:名字像「问题」的优先(同 chat.ts fillInput 的正则,大小写不敏感)
QUESTION_FIELD = re.compile(r"question|query|topic|input|问题|需求|主题", re.IGNORECASE)
#: 带着原图重来时交给助手的话:原图交回去改(调大步数、改参数这种事就是改一个字段,不是推倒重来)
RETRY_WITH_BASE = ("这个工作流上一次运行没有成功:{reason}。\n\n"
"请针对这个原因改它(比如调大最大步数、改正工具参数、换个查询方式),"
"改完继续回答原来的问题:{question}")
#: 没有图可交回(上一次连图都没建出来)时:原问题后面缀上原因,从零再建
RETRY_FROM_SCRATCH = "{question}\n\n(上一次没有成功:{reason}。请针对这个原因调整,重新给出完整的工作流。)"
def _blockers(final: Mapping[str, Any]) -> list[str]:
return [str(i.get("message") or "") for i in final.get("issues") or []
if isinstance(i, Mapping) and i.get("level") == "error"]
def next_step(terminal: Mapping[str, Any] | None, *, self_checked: bool = True) -> str:
"""生成过程交出结局之后做什么。terminal 是最后那条结局操作(final / reply / error);流结束时只有反问(ask)、没有
别的结局的给 None。
- reply:直接回答,这一轮到此为止;error:判失败;只有反问:等用户回答;
- final:有 error 级问题时,服务端已经自查过(收到过 check 的结论)就判失败——没过的问题已经交回模型在同一轮里改过
了,再整轮重来是再烧一次完整生成、改的还是同一批问题;没自查过(老服务端)才重来一次(retry);模型说不运行
(autorun=false:用户要的是流程本身、或者问了用户)只建不跑;其余运行。
"""
if terminal is None:
return NEXT_ASKED
kind = terminal.get("op")
if kind == "reply":
return NEXT_REPLY
if kind != "final":
return NEXT_FAIL
if _blockers(terminal):
return NEXT_FAIL if self_checked else NEXT_RETRY
if terminal.get("autorun") is False:
return NEXT_READY
return NEXT_RUN
def blocked_reason(final: Mapping[str, Any]) -> str:
"""没过检查的图重来时的原因(只有老服务端那条路用得到)。"""
return "生成的工作流没有通过校验:" + ";".join(_blockers(final))
def fill_input(graph: Mapping[str, Any] | None, question: str, history: str = "") -> dict[str, Any]:
"""用户的问题填进这张图的入口字段(同 chat.ts 的 fillInput)。
入口没声明字段:{question, history}(下游可能用 input.question)。声明了:每个字段先填默认值(没有就空串),问题放进
名字像「问题」的那个,其次第一个必填的,再不然第一个;history(前几轮压成的文字)照样带上——图跑起来时模型得知道
「它们」指的是上一轮那批数据。
"""
nodes = (graph or {}).get("nodes") or []
entry = next((n for n in nodes if isinstance(n, Mapping) and n.get("type") == "input"), None)
config = ((entry or {}).get("data") or {}).get("config") or {}
fields = [f for f in (config.get("fields") or []) if isinstance(f, Mapping)]
if not fields:
return {"question": question, "history": history}
payload: dict[str, Any] = {"history": history}
for f in fields:
if f.get("name"):
payload[str(f["name"])] = f["default"] if f.get("default") is not None else ""
named = next((f for f in fields if QUESTION_FIELD.search(str(f.get("name") or ""))), None)
target = named or next((f for f in fields if f.get("required")), None) or fields[0]
if target.get("name"):
payload[str(target["name"])] = question
return payload
def review_reason(review: Mapping[str, Any]) -> str:
"""复核判不可用时重来的原因:各条信号的说明,没有就用给用户看的那句说明。"""
details = [str(s.get("detail") or "") for s in review.get("signals") or [] if isinstance(s, Mapping)]
return ";".join(d for d in details if d) or str(review.get("note") or "")
def retry_after_review(review: Mapping[str, Any] | None, attempt: int) -> bool:
"""复核之后要不要带着原图重来一次:复核说换个配置重跑大概率就好(retry)、答案不可信(severity=broken)、这一轮还没
试满(attempt 从 0 数,attempt + 1 < MAX_ATTEMPTS)。"""
if not review:
return False
return bool(review.get("retry")) and review.get("severity") == "broken" and attempt + 1 < MAX_ATTEMPTS
def retry_instruction(question: str, reason: str, has_base: bool) -> str:
"""重来时交给助手的话。has_base:有上一次建出的图可以交回去改。"""
template = RETRY_WITH_BASE if has_base else RETRY_FROM_SCRATCH
return template.format(question=question, reason=reason)
def has_nodes(graph: Any) -> bool:
"""一张图有没有节点(有才交回去改)。"""
return isinstance(graph, Mapping) and bool(graph.get("nodes"))
__all__ = ["MAX_ATTEMPTS", "NEXT_ASKED", "NEXT_FAIL", "NEXT_READY", "NEXT_REPLY", "NEXT_RETRY", "NEXT_RUN", "NEXT_STEPS",
"QUESTION_FIELD", "RETRY_FROM_SCRATCH", "RETRY_WITH_BASE", "blocked_reason", "fill_input", "has_nodes",
"next_step", "retry_after_review", "retry_instruction", "review_reason"]
+48 -4
View File
@@ -1,9 +1,10 @@
from __future__ import annotations
import ipaddress
import re
from functools import lru_cache
from pathlib import Path
from typing import Annotated
from typing import Annotated, Any
from pydantic import field_validator
from pydantic_settings import BaseSettings, NoDecode, SettingsConfigDict
@@ -11,6 +12,9 @@ from pydantic_settings import BaseSettings, NoDecode, SettingsConfigDict
BACKEND_DIR = Path(__file__).resolve().parents[2]
PROJECT_DIR = BACKEND_DIR.parent
#: 嵌入白名单里一项的主机部分:域名(可以 *. 开头)或方括号里的 IPv6,可以带端口;不许路径、查询、用户名
_ORIGIN_HOST = re.compile(r"(?:\*\.)?(?:[A-Za-z0-9-]+\.)*[A-Za-z0-9-]+(?::\d{1,5})?|\[[0-9A-Fa-f:.]+\](?::\d{1,5})?")
# session_id 拼进路径前必须过这一关。允许的字符刻意收得很窄:字母数字加
# 连字符下划线——点号也不放行,免得出现 ".." 这种能往上走的名字。
_SESSION_MAX = 64
@@ -60,10 +64,18 @@ class Settings(BaseSettings):
auth_session_max_days: int = 7
# 会话 cookie 带不带 Secure。不设时按请求本身是不是 https 定:本机 http 开发照常能登录
auth_cookie_secure: bool | None = None
# 对外的访问地址(反向代理后面时填,如 https://agentlab.example.com)。OIDC 的回调地址按它拼;
# 不设时按请求的 Host 拼
# 对外的访问地址(反向代理后面时填,如 https://agentlab.example.com)。全站只认一个(core/public_url.py):
# 设了它,设置里的「对外访问地址」就不生效、不能改;通知和 Webhook 的链接、OIDC 的回调、来源核对都按它拼。
# 不设时用设置里的那一项,再没有就按请求的 Host 拼(通知里不附链接)
public_url: str = ""
# --- 安全响应头(app/web_security.py)---
# 允许哪些站点用 iframe 嵌整站(同站的门户,如 https://portal.example.com),逗号分隔。本站自己总是允许的。
# 不设时只有本站能嵌(frame-ancestors 'self',另加 X-Frame-Options: SAMEORIGIN)
frame_ancestors: Annotated[list[str], NoDecode] = []
# 分享页(/s)允许被哪些站点嵌,逗号分隔,写法同上。分享页只认这一项,不认上面那一项
embed_origins: Annotated[list[str], NoDecode] = []
# --- 沙箱 ---
sandbox_backend: str = "auto" # auto | microvm | seatbelt | bubblewrap | local | off
sandbox_timeout: int = 30
@@ -121,7 +133,8 @@ class Settings(BaseSettings):
"https://223.5.5.5/resolve",
]
@field_validator("http_tool_allowlist", "http_tool_fake_ip_ranges", "http_tool_doh_urls", mode="before")
@field_validator("http_tool_allowlist", "http_tool_fake_ip_ranges", "http_tool_doh_urls", "frame_ancestors",
"embed_origins", mode="before")
@classmethod
def _comma_list(cls, value: object) -> object:
# 环境变量里写逗号分隔;也兼容 JSON 数组的写法
@@ -134,6 +147,37 @@ class Settings(BaseSettings):
return [part.strip() for part in text.split(",") if part.strip()]
return value
@field_validator("public_url")
@classmethod
def _valid_public_url(cls, value: str) -> str:
# 写错了启动时就报出来:以前拼出 agentlab.example.com/api/auth/oidc/callback 这样的回调,到登录时才发现
from app.core.public_url import ENV_NAME, normalize, url_problem
if not value.strip():
return ""
if problem := url_problem(value):
what = "不能带 ? 或 # 之后的部分" if problem == "suffix" else "需要是 http:// 或 https:// 开头的完整地址"
raise ValueError(f"{ENV_NAME}={value!r} {what},例如 https://agentlab.example.com")
return normalize(value) or ""
@field_validator("frame_ancestors", "embed_origins")
@classmethod
def _valid_origins(cls, value: list[str], info: Any) -> list[str]:
# 只收「协议://主机[:端口]」,主机可以写 *.example.com。路径、裸 *、没写协议的写错了启动时就报出来,别到嵌入
# 失败(或者整站被谁都能嵌)才发现。'self' 本来就有,写了也不重复
name = f"AGENTLAB_{(info.field_name or '').upper()}"
out: list[str] = []
for raw in value:
item = raw.strip().rstrip("/")
if item == "'self'":
continue
scheme, _, host = item.partition("://")
if scheme.lower() not in ("http", "https") or not _ORIGIN_HOST.fullmatch(host):
raise ValueError(f"{name} 里的「{raw}」需要写成「协议://主机」(可以带端口),例如 https://portal.example.com")
if (origin := f"{scheme.lower()}://{host.lower()}") not in out:
out.append(origin)
return out
@field_validator("auth_cookie_secure", mode="before")
@classmethod
def _blank_is_auto(cls, value: object) -> object:
+439
View File
@@ -0,0 +1,439 @@
"""后台任务的共用运行时:数据目录的起草(data/catalog_jobs.py)和剖析(data/catalog_profile_jobs.py)各自建表、各自的步骤,
生命周期是同一套;之后的评测、后台探查也用它(docs/design/next-round/00-总方案.md X5)。
**为什么在 core。** 最早只服务数据目录,放在 data/catalog_job_runtime.py;评测(一次评测一个任务、每题一项)和后台探查
(同源排队)要的是同一样东西,从 data 包里导入一个「数据目录的」运行时说不通。原路径留着再导出(同一个模块对象里的
名字,lanes 也是同一条队),已有的 import 不用改,正在改起草、剖析的工作也不会因此冲突。
**为什么抽出来。** 剖析以前是一次同步长请求,和起草改成后台任务之前是同一类问题:关掉页面结果就没了、不能停、不留痕。
两边要的是同一样东西——进程内跑、能停、重启标中断、写库不被停止打断、同源排队——各写一份,迟早一边修了另一边没修。
共用的是生命周期代码,不是表:两种任务的进度完全不同(起草按段、块、模型用量,剖析按检查项、查询),合成一张
「kind + JSON」的通用任务表要丢掉起草那些强类型列,还要迁移刚稳定的起草数据。
- **状态词**(ACTIVE、FINISHED、TASK_STATUSES、TASK_DONE、RETRYABLE、OPEN):任务 queued → running → succeeded / partial
/ failed / cancelled / interrupted;逐表 pending → running → ok / partial / failed / skipped / cancelled。两种任务、
接口、界面都用这一套词。
- **SerialWriter**:同一个任务的库写入一条一条来,shield 起来——停止任务时不会打断写到一半的库操作;收尾前 settle 等它们
写完。
- **JobRun / JobManager**:进程内一个 asyncio 任务一件事(照 app/scheduler 的写法):start / stop(grace) / wait / running
/ shutdown。JobRun.run 把「进队、等轮到自己、执行、按停止的原因收尾、出队」写一次,各种任务只写执行和收尾的那几步。
- **SourceLanes**:每个数据源一条先来先到的队。起草、剖析都打业务库或模型、都写同一个源的目录:同源串行(先建的先跑),
不同源并行。任务阻塞地占:建出来是 queued,轮到了才 running。同步起草、导入目录这类同步写操作不阻塞地试占:占不到
说明源上正有别的事,接口回 409;占到了,执行期间新建的任务排在它后面。**队里的事是开放的**:运行时不认具体哪几种,
每种在自己的模块里 register_lane_kind 登记一次(kind → 叫法),进队时只核对登记过——第 2 波的后台探查(introspect)
登记一行就排进同一条队,不用改这里;拼错的 kind 当场报错,不会悄悄排成一种谁都不认识的事。
- **close_job / interrupt_on_startup**:收尾(还没结束的表记成某个状态,任务记成某个状态),和服务启动时的整理:排队、
进行中的标 interrupted,没结束的表记 cancelled。**不自动续跑**:复制出来的数据目录(检查脚本、验收副本)启动时不能
自己去调模型、查业务库,界面让人点「重试」接着做。
进程内登记、进程内的队:服务端是单进程部署,多进程部署时要换成库里的锁。
"""
from __future__ import annotations
import asyncio
import contextlib
import logging
from collections.abc import AsyncIterator, Awaitable, Callable, Iterator
from typing import Any
from sqlalchemy import select, update
from sqlalchemy.ext.asyncio import AsyncSession
from app.db.base import SessionLocal, utcnow
logger = logging.getLogger(__name__)
#: 任务还没结束的状态(同一个源同一种任务只能有一个)、结束了的状态
ACTIVE = ("queued", "running")
FINISHED = ("succeeded", "partial", "failed", "cancelled", "interrupted")
#: 逐表的状态,其中算「已结束」的、重试时要带上的、还没结束的
TASK_STATUSES = ("pending", "running", "ok", "partial", "failed", "skipped", "cancelled")
TASK_DONE = ("ok", "partial", "failed", "skipped", "cancelled")
RETRYABLE = ("failed", "partial", "cancelled")
OPEN = ("pending", "running")
#: 停止进行中的任务时最多等它收尾多久(秒);关停服务时另给(shutdown 的 grace)
STOP_GRACE_S = 10.0
SHUTDOWN_GRACE_S = 3.0
class JobFailed(Exception):
"""任务整体出错(数据源没了、快照没了、模型用不了……):message 就是任务的 error。"""
# ==========================================================================
# 库写入
# ==========================================================================
class SerialWriter:
"""一个任务的库写入:一条一条来,每条开一个会话;shield 起来,停止任务时不会打断写到一半的库操作。"""
def __init__(self) -> None:
self._writes: set[asyncio.Task[Any]] = set()
self._lock = asyncio.Lock()
async def __call__(self, fn: Callable[..., Awaitable[Any]], *args: Any) -> Any:
async def go() -> Any:
async with self._lock:
async with SessionLocal() as session:
return await fn(session, *args)
task = asyncio.ensure_future(go())
self._writes.add(task)
task.add_done_callback(self._writes.discard)
return await asyncio.shield(task)
async def settle(self) -> None:
"""等手上的库写入都写完(停止、出错收尾之前)。"""
while self._writes:
await asyncio.gather(*list(self._writes), return_exceptions=True)
# ==========================================================================
# 每个源一条队
# ==========================================================================
#: 同源队里能排的事:{kind: 叫法}。开放的,各种事在自己的模块里登记(register_lane_kind):数据目录登记了 draft 起草任务、
#: profile 剖析任务、sync_draft 同步起草、import 导入目录(data/catalog_jobs.py)
_LANE_KINDS: dict[str, str] = {}
def register_lane_kind(kind: str, label: str) -> str:
"""登记一种同源队里的事(重复登记同一个叫法无害,换叫法报错:同一个 kind 只能有一种说法)。返回 kind。"""
if not kind or not label:
raise ValueError("同源队里的事要有 kind 和叫法")
known = _LANE_KINDS.get(kind)
if known is not None and known != label:
raise ValueError(f"同源队里的「{kind}」已经登记成「{known}」,不能再登记成「{label}」")
_LANE_KINDS[kind] = label
return kind
def lane_kinds() -> dict[str, str]:
"""登记过的各种事:{kind: 叫法}。"""
return dict(_LANE_KINDS)
def lane_label(kind: str) -> str:
"""一种事的叫法(「起草」「剖析」……);没登记的照原样给 kind。"""
return _LANE_KINDS.get(kind, kind)
class LaneTicket:
"""在一个源的队里的一个位置。kind 是占着它的是什么(登记过的一种,见 register_lane_kind);job_id 是任务的(同步写
操作没有)。轮到它时 granted 置位。"""
def __init__(self, lanes: SourceLanes, source_id: str, kind: str, job_id: str | None,
loop: asyncio.AbstractEventLoop) -> None:
self._lanes = lanes
self.source_id = source_id
self.kind = kind
self.job_id = job_id
self.loop = loop
self.granted = asyncio.Event()
self.released = False
async def wait(self) -> None:
"""等轮到自己。"""
await self.granted.wait()
def release(self) -> None:
"""出队(轮到了、没轮到都行,重复调用无害):排在后面的第一个接着开始。"""
self._lanes._release(self)
class LaneBusy(Exception):
"""不阻塞地试占时这个源的队里已经有人:holder 是排在最前面(正在做)的那一个。"""
def __init__(self, holder: LaneTicket) -> None:
super().__init__(holder.kind)
self.holder = holder
class SourceLanes:
"""每个数据源一条先来先到的队。"""
def __init__(self) -> None:
self._queues: dict[str, list[LaneTicket]] = {}
def _queue(self, source_id: str) -> list[LaneTicket]:
"""这个源的队。别的事件循环留下的(测试每个用例一个循环)去掉:那个循环里没人会再出队了。"""
queue = self._queues.get(source_id) or []
loop = asyncio.get_running_loop()
alive = [t for t in queue if t.loop is loop and not t.released]
if alive:
self._queues[source_id] = alive
alive[0].granted.set()
else:
self._queues.pop(source_id, None)
return alive
def enter(self, source_id: str, kind: str, job_id: str | None = None) -> LaneTicket:
"""排进队尾,马上返回(不等)。排在最前面的立刻 granted;调用方 await ticket.wait() 等轮到自己。kind 要登记过。"""
if kind not in _LANE_KINDS:
raise ValueError(f"同源队里没有登记「{kind}」这种事,先 register_lane_kind")
queue = self._queue(source_id)
ticket = LaneTicket(self, source_id, kind, job_id, asyncio.get_running_loop())
queue.append(ticket)
self._queues[source_id] = queue
if len(queue) == 1:
ticket.granted.set()
return ticket
def try_enter(self, source_id: str, kind: str) -> LaneTicket:
"""队空着才占(马上就轮到);有人在队里抛 LaneBusy。"""
queue = self._queue(source_id)
if queue:
raise LaneBusy(queue[0])
return self.enter(source_id, kind)
def holders(self, source_id: str) -> list[LaneTicket]:
"""队里的各位,排在最前面(正在做)的在前。"""
return list(self._queue(source_id))
def ahead(self, source_id: str, job_id: str) -> list[str] | None:
"""排在这个任务前面的各位是什么(kind,正在做的在前);这个任务不在队里(刚建、后台还没排进来)返回 None。"""
queue = self._queue(source_id)
for i, ticket in enumerate(queue):
if ticket.job_id == job_id:
return [t.kind for t in queue[:i]]
return None
def _release(self, ticket: LaneTicket) -> None:
if ticket.released:
return
ticket.released = True
queue = self._queues.get(ticket.source_id)
if not queue or ticket not in queue:
return
head = queue[0] is ticket
queue.remove(ticket)
if not queue:
self._queues.pop(ticket.source_id, None)
elif head:
queue[0].granted.set()
@contextlib.asynccontextmanager
async def hold(self, source_id: str, kind: str, job_id: str | None = None) -> AsyncIterator[LaneTicket]:
"""阻塞地占:排队,轮到了再进去,出来时出队。"""
ticket = self.enter(source_id, kind, job_id)
try:
await ticket.wait()
yield ticket
finally:
ticket.release()
@contextlib.contextmanager
def try_hold(self, source_id: str, kind: str) -> Iterator[LaneTicket]:
"""不阻塞地试占(同步起草、导入目录):占不到抛 LaneBusy;占到了,执行期间新建的任务排在后面。"""
ticket = self.try_enter(source_id, kind)
try:
yield ticket
finally:
ticket.release()
lanes = SourceLanes()
# ==========================================================================
# 收尾、启动时的整理
# ==========================================================================
async def close_job(session: AsyncSession, job_model: Any, task_model: Any, job_id: str, job_status: str,
job_error: str | None, task_status: str, task_error: str | None) -> None:
"""收尾:还没结束的表记 task_status(原因 task_error),任务(还没结束的话)记 job_status。提交。"""
now = utcnow()
await session.execute(
update(task_model).where(task_model.job_id == job_id, task_model.status.in_(OPEN))
.values(status=task_status, error=task_error, finished_at=now).execution_options(synchronize_session=False))
await session.execute(
update(job_model).where(job_model.id == job_id, job_model.status.in_(ACTIVE))
.values(status=job_status, error=job_error, finished_at=now).execution_options(synchronize_session=False))
await session.commit()
async def cancel_queued(session: AsyncSession, job_model: Any, task_model: Any, job_id: str) -> bool:
"""停止一个排队中的任务:任务和没结束的表直接记 cancelled。已经被后台认领(开始了)返回 False,交给调用方按进行中
的停。"""
now = utcnow()
claimed = await session.execute(
update(job_model).where(job_model.id == job_id, job_model.status == "queued")
.values(status="cancelled", finished_at=now).execution_options(synchronize_session=False))
if claimed.rowcount != 1:
await session.rollback()
return False
await session.execute(
update(task_model).where(task_model.job_id == job_id, task_model.status.in_(OPEN))
.values(status="cancelled", finished_at=now).execution_options(synchronize_session=False))
await session.commit()
return True
async def claim(session: AsyncSession, job_model: Any, job_id: str) -> bool:
"""认领:排队中 → 进行中。开始之前就被停止了返回 False。"""
result = await session.execute(
update(job_model).where(job_model.id == job_id, job_model.status == "queued")
.values(status="running", started_at=utcnow()).execution_options(synchronize_session=False))
await session.commit()
return result.rowcount == 1
async def interrupt_on_startup(job_model: Any, close: Callable[[AsyncSession, str], Awaitable[None]], *,
running: Callable[[str], bool]) -> int:
"""服务启动时:排队、进行中、而且不在这个进程里跑的任务逐个 close(各自记成 interrupted、没结束的表记 cancelled)。
不自动续跑。返回整理了几个。"""
async with SessionLocal() as session:
ids = [job_id for job_id in (await session.execute(
select(job_model.id).where(job_model.status.in_(ACTIVE)))).scalars() if not running(job_id)]
for job_id in ids:
await close(session, job_id)
return len(ids)
# ==========================================================================
# 在后台执行
# ==========================================================================
class JobRun:
"""一个任务在后台的一次执行。子类给 kind(进队时的身份)、model(任务表,进队前读源和状态),写 _run(执行)和
_stopped / _failed / _crashed(停止、整体出错、意外出错时的收尾)。"""
kind = ""
model: Any = None
def __init__(self, job_id: str) -> None:
self.job_id = job_id
self.task: asyncio.Task[None] | None = None
#: 为什么停:user 使用者点了停止,shutdown 服务关停(标中断)
self.stop_reason: str | None = None
self.writer = SerialWriter()
self.ticket: LaneTicket | None = None
async def _write(self, fn: Callable[..., Awaitable[Any]], *args: Any) -> Any:
"""一条库写入(SerialWriter:一条一条来,停止时不被打断)。"""
return await self.writer(fn, *args)
async def _settle(self) -> None:
await self.writer.settle()
async def run(self) -> None:
try:
try:
await self._queue()
await self._run()
except asyncio.CancelledError:
current = asyncio.current_task()
if current is not None:
current.uncancel()
await self._stopped()
except JobFailed as e:
await self._failed(str(e))
except Exception as e: # noqa: BLE001 - 任务整体出错:照实记下,不留在「进行中」
await self._crashed(e)
finally:
# 收尾写完才出队:排在后面的任务开始时,这个任务已经记成结束了
if self.ticket is not None:
self.ticket.release()
def lane_of(self, job: Any) -> str | None:
"""这个任务排在哪条队里:缺省是任务的数据源(job.source_id)。不占源的任务(以后的评测:开始时拍目录快照,不挡
起草、剖析)覆盖成返回 None,不进队、直接执行。"""
return job.source_id
async def _queue(self) -> None:
"""进这个源的队,等轮到自己。任务不在了、已经不是排队中(开始之前就被停止了)的不进队:_run 认领不到,直接结束。"""
async with SessionLocal() as session:
job = await session.get(self.model, self.job_id)
source_id = self.lane_of(job) if job is not None and job.status == "queued" else None
if source_id is None:
return
self.ticket = lanes.enter(source_id, self.kind, self.job_id)
await self.ticket.wait()
async def _run(self) -> None:
raise NotImplementedError
async def _stopped(self) -> None:
raise NotImplementedError
async def _failed(self, reason: str) -> None:
raise NotImplementedError
async def _crashed(self, e: Exception) -> None:
raise NotImplementedError
class JobManager:
"""进程内的一种后台任务:一个 asyncio 任务一件事,停的时候给宽限。factory 按任务编号造一次执行(JobRun);label 是
日志里的叫法(「数据目录起草任务」),name 是 asyncio 任务名的前缀。"""
def __init__(self, factory: Callable[[str], JobRun], *, label: str, name: str) -> None:
self._runs: dict[str, JobRun] = {}
self._factory = factory
self.label = label
self.name = name
def _alive(self, job_id: str) -> JobRun | None:
run = self._runs.get(job_id)
if run is None or run.task is None or run.task.done():
return None
try:
if run.task.get_loop() is not asyncio.get_running_loop():
self._runs.pop(job_id, None) # 别的事件循环留下的(测试每个用例一个循环)
return None
except RuntimeError:
return None
return run
def start(self, job_id: str) -> None:
"""开始在后台执行这个(排队中的)任务:先进这个源的队,轮到了才开始。"""
run = self._factory(job_id)
run.task = asyncio.get_running_loop().create_task(run.run(), name=f"{self.name}-{job_id}")
self._runs[job_id] = run
def forget(_: asyncio.Task[None]) -> None:
if self._runs.get(job_id) is run:
self._runs.pop(job_id, None)
run.task.add_done_callback(forget)
def running(self, job_id: str) -> bool:
return self._alive(job_id) is not None
async def stop(self, job_id: str, reason: str, *, grace: float = STOP_GRACE_S) -> bool:
"""停掉这个进程里在跑(或在排队等)的任务,等它收尾(最多 grace 秒)。不在这个进程里返回 False。"""
run = self._alive(job_id)
if run is None:
return False
if run.stop_reason is None:
run.stop_reason = reason
run.task.cancel() # type: ignore[union-attr]
await asyncio.wait([run.task], timeout=grace) # type: ignore[list-item]
return True
async def wait(self, job_id: str, timeout: float | None = None) -> None:
"""等这个任务做完(测试、调试用)。不在跑直接返回。"""
run = self._alive(job_id)
if run is not None:
await asyncio.wait([run.task], timeout=timeout) # type: ignore[list-item]
async def shutdown(self, *, grace: float = SHUTDOWN_GRACE_S) -> None:
"""关停:停掉进行中、排队等着的任务(标中断),最多等 grace 秒收尾。"""
runs = [run for job_id in list(self._runs) if (run := self._alive(job_id)) is not None]
for run in runs:
if run.stop_reason is None:
run.stop_reason = "shutdown"
run.task.cancel() # type: ignore[union-attr]
if runs:
await asyncio.wait([run.task for run in runs], timeout=grace) # type: ignore[misc]
logger.info("已停止 %d 个%s", len(runs), self.label)
__all__ = ["ACTIVE", "FINISHED", "JobFailed", "JobManager", "JobRun", "LaneBusy", "LaneTicket", "OPEN", "RETRYABLE",
"SHUTDOWN_GRACE_S", "STOP_GRACE_S", "SerialWriter", "SourceLanes", "TASK_DONE", "TASK_STATUSES", "cancel_queued",
"claim", "close_job", "interrupt_on_startup", "lane_kinds", "lane_label", "lanes", "register_lane_kind"]
+97
View File
@@ -0,0 +1,97 @@
"""对外访问地址:别人打开 AgentLab 用的那个地址(https://agentlab.example.com),全站只有这一个说法(04 细案 B4)。
用到它的地方:通知和定时任务 Webhook 里的链接、单点登录的回调地址、非 GET 请求的来源核对(auth/middleware.py);
之后的分享链接、接口说明也从这里取。服务端不知道自己在反向代理后面叫什么,只能由人告诉它。
取值(public_base_url):
- 环境变量 AGENTLAB_PUBLIC_URL 优先:部署时定死,设置页上显示「由环境变量指定」、不能改(同登录方式的 env_locked);
- 没设时取设置里 notify 一组的 public_base_url:沿用这个键,界面上叫「对外访问地址」,在「设置 → 通用」里改;
- 都没有是 None:通知里不附链接,单点登录的回调和来源核对按请求自己的 Host 认。
以前两处各管各的:环境变量只管回调和来源核对、设置只管通知里的链接——只配了环境变量的部署,通知里不带链接;只在设置里
填了的,回调又按请求的 Host 拼。违背「一种说法」。
设置的值缓存在进程里:中间件按请求核对来源,不能每次查库。启动时读一次(main._platform_startup),改设置的接口写完库就
remember;手里本来就有会话的异步路径(通知、定时任务 Webhook)用 refresh 取,顺手按库里的值对齐缓存。和登录设置的缓存一样
以单进程部署为前提。
这个模块不在顶层导入 core/config:config 的校验器要用这里的 url_problem,顶层互相导入会在启动时绕成环。
"""
from __future__ import annotations
from typing import Any
from urllib.parse import urlsplit
#: 设置里存它的那一组、那一项(api/settings.py 的 NOTIFY_GROUP 就是这一组)
SETTING_GROUP = "notify"
SETTING_KEY = "public_base_url"
#: 环境变量的名字(core/config.Settings.public_url 按 AGENTLAB_ 前缀读它)
ENV_NAME = "AGENTLAB_PUBLIC_URL"
_stored: str | None = None
def url_problem(value: str) -> str | None:
"""地址的形状有什么问题:None 是没问题;"form" 不是 http:// 或 https:// 开头的完整地址;"suffix" 带了 ? 或 #
(链接要接在地址后面)。设置接口按它出中文的说法,环境变量按它在启动时报错。可以带路径(反向代理挂在子路径下)。"""
text = value.strip()
try:
parts = urlsplit(text)
except ValueError:
return "form"
if parts.scheme not in ("http", "https") or not parts.hostname or " " in text:
return "form"
if parts.query or parts.fragment:
return "suffix"
return None
def normalize(value: Any) -> str | None:
"""存着的(或刚填的)值 → 去掉首尾空白和末尾的 /;空的、形状不对的当没填(None)。末尾带 / 的地址拼链接会多出一个斜杠。"""
if not isinstance(value, str) or not value.strip() or url_problem(value):
return None
return value.strip().rstrip("/")
def env_url() -> str | None:
"""环境变量给的地址;没设为 None。形状在启动时由 config 的校验器查过、末尾的 / 去掉了。"""
from app.core.config import settings
return settings.public_url or None
def env_locked() -> bool:
"""环境变量设了:设置里的那一项不生效、不能改。"""
return env_url() is not None
def public_base_url() -> str | None:
"""对外访问地址:环境变量优先,其次设置(进程内缓存),都没有是 None。不查库,中间件里也能调。"""
return env_url() or _stored
def remember(value: Any) -> None:
"""改设置的接口写完库之后调:缓存换成新存的值(形状不对、空的当没填)。"""
global _stored
_stored = normalize(value)
async def refresh(session: Any = None) -> str | None:
"""按库里存的设置对齐缓存,交回对外访问地址。启动时、手里有会话的异步路径用。"""
from app.db.base import SessionLocal
from app.db.models import Setting
if session is None:
async with SessionLocal() as own:
row = await own.get(Setting, SETTING_GROUP)
else:
row = await session.get(Setting, SETTING_GROUP)
value = row.value if row is not None else None
remember(value.get(SETTING_KEY) if isinstance(value, dict) else None)
return public_base_url()
def origin_of(url: str) -> str:
"""地址里的「协议://主机[:端口]」:来源核对按它比。"""
parts = urlsplit(url)
return f"{parts.scheme}://{parts.netloc}"
+112 -15
View File
@@ -420,6 +420,18 @@ async def read_catalog(session: AsyncSession, source_id: str) -> dict[str, Catal
return {row.table_name: _entry(row) for row in rows}
async def read_catalogs(session: AsyncSession, source_ids: Iterable[str]) -> dict[str, dict[str, CatalogEntry]]:
"""几个数据源的目录一次查询读出:{源 id: {表名: CatalogEntry}},没有目录的源不出现。给模型、给自查读目录的地方经
data/catalog_view.py 调它,不直接查 catalog_notes。"""
ids = sorted({i for i in source_ids if i})
if not ids:
return {}
out: dict[str, dict[str, CatalogEntry]] = {}
for row in (await session.execute(select(CatalogNote).where(CatalogNote.source_id.in_(ids)))).scalars():
out.setdefault(row.source_id, {})[row.table_name] = _entry(row)
return out
async def write_entry(session: AsyncSession, source_id: str, table: str, notes: dict[str, Any], *,
if_version: int, actor: str | None, commit: bool = True) -> CatalogEntry:
"""整份写入一张表的目录(乐观锁),提交事务,返回写入后的目录。
@@ -567,18 +579,48 @@ def _undated(note: str) -> str:
return _PROFILE_NOTE_DAY.sub("数据剖析", note, count=1)
#: 剖析备注改过的说法:(升级前的写法, 现在的写法)。大表看的是按存储顺序的前 N 行(多半是最老的数据)、关系取的是
#: 数据库先给出的那几个键值,都不是随机抽样,说法改成照实的(data/catalog_profile.py 的 _relation_note、码值的
#: 「统计范围」、FILTER_SAMPLED)。只换这几处固定的句式,换完和现在的写法一字不差才算同一句
_LEGACY_PROFILE_WORDING = (
(re.compile(r"抽样覆盖率 (?=\S+?,可能不是这条关系)"), "覆盖率 "),
(re.compile(r"子表抽样 (?=\d+ 个不同键值)"), "子表取了 "),
(re.compile(r"(只看前 (?=\d+ 行))"), "(按存储顺序只看前 "),
(re.compile(r"按前 (?=[\d,]+ 行统计)"), "按存储顺序取前 "),
)
#: 起草时核对有效行过滤写进备注的那句(catalog_profile.FILTER_SAMPLED / FILTER_SAMPLED_EDGE):接在模型给的依据后面,
#: 整段不以「数据剖析:」开头,只换这一句
_LEGACY_FILTER_WORDING = re.compile(r"剖析核对:按前 (?=[\d,]+ 行统计)")
def _current_wording(note: str) -> str:
"""剖析备注里升级前的说法换成现在的写法(_LEGACY_PROFILE_WORDING)。只用来比较,不改写目录里存着的备注。"""
for pattern, now in _LEGACY_PROFILE_WORDING:
note = pattern.sub(now, note)
return note
def same_profile_note(a: Any, b: Any) -> bool:
"""两段备注是否相同;数据剖析写的备注只差开头的日期(升级前每次都写当天的日期),或者一段带日期、一段是不带
日期的新写法,也算相同。
日期的新写法,或者只差改过的几处说法(_LEGACY_PROFILE_WORDING:「抽样覆盖率」→「覆盖率」之类),也算相同。
隔天重跑、数据没变,结论一字不差、只有日期变了:要是这也算「变了」,目录就升一个版本,发布时记下的目录版本
对不上,之后每次正式运行都提醒「目录有变化」。升级后第一次重新剖析也是一样:结论没变,只是新写法不带日期。
对不上,之后每次正式运行都提醒「目录有变化」。升级后第一次重新剖析也是一样:结论没变,只是新写法不带日期、
措辞改了。算相同时保留旧的那段(目录不动),旧说法要等结论真的变了、重写备注时才换掉。
起草写的备注(有效行过滤)里接着的剖析核对那句同理,只认那一句的旧说法(_LEGACY_FILTER_WORDING)。
"""
if a == b:
return True
if not (is_profile_note(a) and is_profile_note(b)):
if not (isinstance(a, str) and isinstance(b, str)):
return False
return _undated(a) == _undated(b)
profile_a, profile_b = is_profile_note(a), is_profile_note(b)
if profile_a and profile_b:
return _current_wording(_undated(a)) == _current_wording(_undated(b))
if profile_a or profile_b:
return False
return (_LEGACY_FILTER_WORDING.sub("剖析核对:按存储顺序取前 ", a)
== _LEGACY_FILTER_WORDING.sub("剖析核对:按存储顺序取前 ", b))
def _same_core(a: Mapping[str, Any], b: Mapping[str, Any]) -> bool:
@@ -592,6 +634,14 @@ def visible_notes(notes: Mapping[str, Any] | None) -> dict[str, Any]:
return _assemble({k: v for k, v in _slots(notes).items() if v.get("status") != "rejected"})
def restrict_notes(notes: Mapping[str, Any] | None, statuses: Iterable[str]) -> dict[str, Any]:
"""只留状态在 statuses 里的项(表级项、列级项、关系都按项算);驳回的项原样留着——给人、给模型看的照旧经
visible_notes 去掉,冻结快照和 SQL 检查要靠驳回的关系编号认出不该按外键、命名现推回来的关系(frozen_catalog)。
目录视图「只用已核实、已确认的项」用它(data/catalog_view.py)。"""
keep = set(statuses) | {"rejected"}
return _assemble({k: v for k, v in _slots(notes).items() if v.get("status") in keep})
def status_counts(notes: Mapping[str, Any] | None) -> dict[str, int]:
"""各状态的项数(表级项、列级项、关系都算),四种状态都有键。"""
counts = dict.fromkeys(ITEM_STATUSES, 0)
@@ -1358,14 +1408,17 @@ _DATE_WORDS = frozenset({"date", "time", "day", "dt", "ts", "timestamp", "dateti
_TEXT_TYPES = ("CHAR", "TEXT", "STRING", "CLOB")
def is_date_column(name: str, type_name: str | None) -> bool:
def is_date_column(name: str, type_name: str | None, *, enum_values: Any = None) -> bool:
"""日期类列:类型是日期、时间戳(DATE、DATETIME、TIMESTAMP……);或者类型是文字(或者不知道类型)、名字像日期
(SQLite 的日期常存成文字)。只有时刻的 TIME、整数存的时间戳不算:取值不是日期。
(SQLite 的日期常存成文字)。只有时刻的 TIME、整数存的时间戳不算:取值不是日期。有取值列表的(MySQL 的 ENUM /
SET、探查记下 enum_values 的 PostgreSQL 枚举)不算:PG 的枚举编译出来是类型名本身,update_reason 里也有「DATE」。
数据剖析(表里只有一个日期类列时提议它作业务日期)和 SQL 检查(按业务日期以外的时间列统计)共用这一个判断。
以前两边各写一套:剖析认的「唯一日期列」,SQL 检查可能认为还有别的时间列,反过来也一样,两边对不上。
"""
t = (type_name or "").upper()
t = _type_words(type_name)
if enum_values or _MYSQL_ENUM.match(t) or "INTERVAL" in t:
return False
if "DATE" in t or "TIMESTAMP" in t:
return True
if t and not any(k in t for k in _TEXT_TYPES):
@@ -1400,11 +1453,41 @@ def _name_index(tables: Mapping[str, Any]) -> dict[tuple[str, str], set[str]]:
return index
#: 类型写法里带引号的部分(ENUM('int','date') 的取值):认类型时去掉,取值里的词不能让一列变成数字或时间
_QUOTED_IN_TYPE = re.compile(r"'(?:[^']|'')*'")
def _type_words(type_name: str | None) -> str:
"""认类型用的写法:大写、去掉引号里的取值。探查存的是按方言编译的完整写法(ENUM('paid','refund')、
VARCHAR2(20 CHAR)、TIMESTAMP WITH TIME ZONE),不再是 SQLAlchemy 的通用写法。"""
return _QUOTED_IN_TYPE.sub("''", (type_name or "").upper()).strip()
#: MySQL 的 ENUM / SET 编译出来的写法:ENUM('a','b')、SET('x','y'),只认带括号的。PostgreSQL 的枚举编译出来是类型名本身,
#: setup_status、enum_kind 这种以 set、enum 开头的名字不能按前缀认成 MySQL 的 SET / ENUM——PG 的枚举一律靠探查记下的
#: enum_values 认(column_family)
_MYSQL_ENUM = re.compile(r"^(ENUM|SET)\s*\(")
def _mysql_enum_kind(type_name: str | None) -> str | None:
"""类型写法是 MySQL 的 ENUM(…) / SET(…) 时返回 "ENUM" / "SET",否则 None。"""
m = _MYSQL_ENUM.match(_type_words(type_name))
return m.group(1) if m else None
def _type_family(type_name: str | None) -> str | None:
"""列类型的大类:数字、文字、时间;认不出来返回 None(不据此排除)。"""
t = (type_name or "").upper()
"""列类型的大类:数字、文字、时间;认不出来返回 None(不据此排除)。
认的是按方言编译的完整写法:MySQL 的 ENUM / SET 是文字(取值列表里的词不算),Oracle 的 INTERVAL DAY TO SECOND
是时长、不算数字(以前按子串「INT」认成了数字)。PostgreSQL 的枚举编译出来是类型名本身,只看类型认不出来,
要看列上的取值列表(column_family)。"""
t = _type_words(type_name)
if not t:
return None
if _MYSQL_ENUM.match(t):
return "text"
if "INTERVAL" in t:
return None
if any(k in t for k in ("CHAR", "TEXT", "CLOB", "STRING", "UUID")):
return "text"
if any(k in t for k in ("INT", "NUMBER", "NUMERIC", "DECIMAL", "SERIAL", "REAL", "FLOAT", "DOUBLE")):
@@ -1414,6 +1497,16 @@ def _type_family(type_name: str | None) -> str | None:
return None
def column_family(col: Mapping[str, Any] | None) -> str | None:
"""表结构里一列的类型大类:有取值列表(探查记下的 enum_values:MySQL 的 ENUM / SET、PostgreSQL 的枚举)的是
文字,其余按类型写法认(_type_family)。"""
if not isinstance(col, Mapping):
return None
if col.get("enum_values"):
return "text"
return _type_family(col.get("type"))
def _id_stem(column: str) -> str | None:
m = _SNAKE_ID.match(column) or _CAMEL_ID.match(column)
return m.group("stem") if m else None
@@ -1469,8 +1562,8 @@ def infer_name_relations(schema_cache: Mapping[str, Any] | None, table: str, *,
pk = list(target_meta.get("primary_key") or [])
if len(pk) != 1:
continue
pk_type = next((c.get("type") for c in target_meta.get("columns") or [] if c.get("name") == pk[0]), None)
mine, theirs = _type_family(col.get("type")), _type_family(pk_type)
pk_col = next((c for c in target_meta.get("columns") or [] if c.get("name") == pk[0]), None)
mine, theirs = column_family(col), column_family(pk_col)
if mine and theirs and mine != theirs:
continue
out.append(_relation(table, [name], target, pk, "name", cardinality=_cardinality(meta, [name])))
@@ -2408,15 +2501,18 @@ async def query_snapshot_tables(session: AsyncSession, source: Any) -> list[tupl
使用次数(table_usage)和「这张表被哪些运行用过」(data/catalog_review.py)共用这一份取法:数据源查询工具每次成功
查询都存一份查询快照工件(tools/datasource.py),工件引用表 artifacts 里按 (内容哈希, 运行) 各记一行;只收属于某次运行
(runs 里有这次运行)、属于这个源的行,工具库里的试用(不属于任何运行)不算。
(runs 里有这次运行)、属于这个源的行,工具库里的试用(不属于任何运行)不算;系统派生的运行(评测、口径对账、召回,
engine/origin.counted_runs)也不算——一跑评测,常用表的排序、起草的默认范围、审阅队列的权重就跟着被评测题刷高了。
表名在查询当时就记进了引用的 meta(query_snapshot_meta),这里只读 artifacts 一张表;升级前的老引用没有 meta,退回读
工件内容里的源名和 SQL,读过的按工件 id 缓存(内容永不改变)。SQL 里写的表名去掉 schema 前缀后按 name_key(只把
ASCII 字母转小写,和 SQLite 比较标识符的口径一致)对到表结构上,对不上的不计。
"""
from app.engine.origin import counted_runs
rows = (await session.execute(
select(Artifact.id, Artifact.meta, Artifact.run_id).join(Run, Run.id == Artifact.run_id)
.where(Artifact.kind == QUERY_SNAPSHOT_KIND)
.where(Artifact.kind == QUERY_SNAPSHOT_KIND, counted_runs())
)).all()
tables = (getattr(source, "schema_cache", None) or {}).get("tables") or {}
lookup: dict[str, str] = {}
@@ -2505,8 +2601,9 @@ __all__ = [
"fk_relations", "frozen_catalog", "infer_name_relations", "initial_status", "is_date_column", "is_profile_note",
"join_paths",
"make_item", "merge_notes", "name_words", "now_iso", "parse_path", "plan_patch", "profile_relation_holds",
"query_snapshot_meta", "read_catalog", "read_entry", "relation_graph", "relation_id", "render_table_index",
"render_table_notes", "resolve_draft_model", "resolve_table_name", "review_entry", "review_item", "same_notes",
"query_snapshot_meta", "read_catalog", "read_catalogs", "read_entry", "relation_graph", "relation_id", "render_table_index",
"render_table_notes", "resolve_draft_model", "resolve_table_name", "restrict_notes", "review_entry", "review_item",
"same_notes",
"same_profile_note", "save_human_edit", "save_patch", "status_counts", "structure_relation_ids",
"system_notes_source", "table_usage", "unreviewed_status", "validate_notes", "visible_notes", "write_entry",
]
+7 -394
View File
@@ -1,400 +1,13 @@
"""数据目录后台任务的共用运行时:起草(data/catalog_jobs.py)和剖析(data/catalog_profile_jobs.py)各自建表、各自的步骤,
生命周期是同一套。
"""后台任务运行时的老路径:已上移到 app/core/job_runtime.py(docs/design/next-round/00-总方案.md X5)。
**为什么抽出来。** 剖析以前是一次同步长请求,和起草改成后台任务之前是同一类问题:关掉页面结果就没了、不能停、不留痕。
两边要的是同一样东西——进程内跑、能停、重启标中断、写库不被停止打断、同源排队——各写一份,迟早一边修了另一边没修。
共用的是生命周期代码,不是表:两种任务的进度完全不同(起草按段、块、模型用量,剖析按检查项、查询),合成一张
「kind + JSON」的通用任务表要丢掉起草那些强类型列,还要迁移刚稳定的起草数据。
- **状态词**(ACTIVE、FINISHED、TASK_STATUSES、TASK_DONE、RETRYABLE、OPEN):任务 queued → running → succeeded / partial
/ failed / cancelled / interrupted;逐表 pending → running → ok / partial / failed / skipped / cancelled。两种任务、
接口、界面都用这一套词。
- **SerialWriter**:同一个任务的库写入一条一条来,shield 起来——停止任务时不会打断写到一半的库操作;收尾前 settle 等它们
写完。
- **JobRun / JobManager**:进程内一个 asyncio 任务一件事(照 app/scheduler 的写法):start / stop(grace) / wait / running
/ shutdown。JobRun.run 把「进队、等轮到自己、执行、按停止的原因收尾、出队」写一次,各种任务只写执行和收尾的那几步。
- **SourceLanes**:每个数据源一条先来先到的队。起草、剖析都打业务库或模型、都写同一个源的目录:同源串行(先建的先跑),
不同源并行。任务阻塞地占:建出来是 queued,轮到了才 running。同步起草、导入目录这类同步写操作不阻塞地试占:占不到
说明源上正有别的事,接口回 409;占到了,执行期间新建的任务排在它后面。
- **close_job / interrupt_on_startup**:收尾(还没结束的表记成某个状态,任务记成某个状态),和服务启动时的整理:排队、
进行中的标 interrupted,没结束的表记 cancelled。**不自动续跑**:复制出来的数据目录(检查脚本、验收副本)启动时不能
自己去调模型、查业务库,界面让人点「重试」接着做。
进程内登记、进程内的队:服务端是单进程部署,多进程部署时要换成库里的锁。
这里把那个模块原样放在老名字下(sys.modules 指向同一个模块对象):起草、剖析和别处的 `from app.data import
catalog_job_runtime as runtime`、`from app.data.catalog_job_runtime import …` 拿到的是同一批名字、同一条队(lanes),
测试里改它的属性也就是改那个模块。新代码从 app.core.job_runtime 导入。
"""
from __future__ import annotations
import asyncio
import contextlib
import logging
from collections.abc import AsyncIterator, Awaitable, Callable, Iterator
from typing import Any
import sys
from sqlalchemy import select, update
from sqlalchemy.ext.asyncio import AsyncSession
from app.core import job_runtime
from app.db.base import SessionLocal, utcnow
logger = logging.getLogger(__name__)
#: 任务还没结束的状态(同一个源同一种任务只能有一个)、结束了的状态
ACTIVE = ("queued", "running")
FINISHED = ("succeeded", "partial", "failed", "cancelled", "interrupted")
#: 逐表的状态,其中算「已结束」的、重试时要带上的、还没结束的
TASK_STATUSES = ("pending", "running", "ok", "partial", "failed", "skipped", "cancelled")
TASK_DONE = ("ok", "partial", "failed", "skipped", "cancelled")
RETRYABLE = ("failed", "partial", "cancelled")
OPEN = ("pending", "running")
#: 停止进行中的任务时最多等它收尾多久(秒);关停服务时另给(shutdown 的 grace)
STOP_GRACE_S = 10.0
SHUTDOWN_GRACE_S = 3.0
class JobFailed(Exception):
"""任务整体出错(数据源没了、快照没了、模型用不了……):message 就是任务的 error。"""
# ==========================================================================
# 库写入
# ==========================================================================
class SerialWriter:
"""一个任务的库写入:一条一条来,每条开一个会话;shield 起来,停止任务时不会打断写到一半的库操作。"""
def __init__(self) -> None:
self._writes: set[asyncio.Task[Any]] = set()
self._lock = asyncio.Lock()
async def __call__(self, fn: Callable[..., Awaitable[Any]], *args: Any) -> Any:
async def go() -> Any:
async with self._lock:
async with SessionLocal() as session:
return await fn(session, *args)
task = asyncio.ensure_future(go())
self._writes.add(task)
task.add_done_callback(self._writes.discard)
return await asyncio.shield(task)
async def settle(self) -> None:
"""等手上的库写入都写完(停止、出错收尾之前)。"""
while self._writes:
await asyncio.gather(*list(self._writes), return_exceptions=True)
# ==========================================================================
# 每个源一条队
# ==========================================================================
class LaneTicket:
"""在一个源的队里的一个位置。kind 是占着它的是什么:draft 起草任务、profile 剖析任务、sync_draft 同步起草、
import 导入目录;job_id 是任务的(同步写操作没有)。轮到它时 granted 置位。"""
def __init__(self, lanes: SourceLanes, source_id: str, kind: str, job_id: str | None,
loop: asyncio.AbstractEventLoop) -> None:
self._lanes = lanes
self.source_id = source_id
self.kind = kind
self.job_id = job_id
self.loop = loop
self.granted = asyncio.Event()
self.released = False
async def wait(self) -> None:
"""等轮到自己。"""
await self.granted.wait()
def release(self) -> None:
"""出队(轮到了、没轮到都行,重复调用无害):排在后面的第一个接着开始。"""
self._lanes._release(self)
class LaneBusy(Exception):
"""不阻塞地试占时这个源的队里已经有人:holder 是排在最前面(正在做)的那一个。"""
def __init__(self, holder: LaneTicket) -> None:
super().__init__(holder.kind)
self.holder = holder
class SourceLanes:
"""每个数据源一条先来先到的队。"""
def __init__(self) -> None:
self._queues: dict[str, list[LaneTicket]] = {}
def _queue(self, source_id: str) -> list[LaneTicket]:
"""这个源的队。别的事件循环留下的(测试每个用例一个循环)去掉:那个循环里没人会再出队了。"""
queue = self._queues.get(source_id) or []
loop = asyncio.get_running_loop()
alive = [t for t in queue if t.loop is loop and not t.released]
if alive:
self._queues[source_id] = alive
alive[0].granted.set()
else:
self._queues.pop(source_id, None)
return alive
def enter(self, source_id: str, kind: str, job_id: str | None = None) -> LaneTicket:
"""排进队尾,马上返回(不等)。排在最前面的立刻 granted;调用方 await ticket.wait() 等轮到自己。"""
queue = self._queue(source_id)
ticket = LaneTicket(self, source_id, kind, job_id, asyncio.get_running_loop())
queue.append(ticket)
self._queues[source_id] = queue
if len(queue) == 1:
ticket.granted.set()
return ticket
def try_enter(self, source_id: str, kind: str) -> LaneTicket:
"""队空着才占(马上就轮到);有人在队里抛 LaneBusy。"""
queue = self._queue(source_id)
if queue:
raise LaneBusy(queue[0])
return self.enter(source_id, kind)
def holders(self, source_id: str) -> list[LaneTicket]:
"""队里的各位,排在最前面(正在做)的在前。"""
return list(self._queue(source_id))
def ahead(self, source_id: str, job_id: str) -> list[str] | None:
"""排在这个任务前面的各位是什么(kind,正在做的在前);这个任务不在队里(刚建、后台还没排进来)返回 None。"""
queue = self._queue(source_id)
for i, ticket in enumerate(queue):
if ticket.job_id == job_id:
return [t.kind for t in queue[:i]]
return None
def _release(self, ticket: LaneTicket) -> None:
if ticket.released:
return
ticket.released = True
queue = self._queues.get(ticket.source_id)
if not queue or ticket not in queue:
return
head = queue[0] is ticket
queue.remove(ticket)
if not queue:
self._queues.pop(ticket.source_id, None)
elif head:
queue[0].granted.set()
@contextlib.asynccontextmanager
async def hold(self, source_id: str, kind: str, job_id: str | None = None) -> AsyncIterator[LaneTicket]:
"""阻塞地占:排队,轮到了再进去,出来时出队。"""
ticket = self.enter(source_id, kind, job_id)
try:
await ticket.wait()
yield ticket
finally:
ticket.release()
@contextlib.contextmanager
def try_hold(self, source_id: str, kind: str) -> Iterator[LaneTicket]:
"""不阻塞地试占(同步起草、导入目录):占不到抛 LaneBusy;占到了,执行期间新建的任务排在后面。"""
ticket = self.try_enter(source_id, kind)
try:
yield ticket
finally:
ticket.release()
lanes = SourceLanes()
# ==========================================================================
# 收尾、启动时的整理
# ==========================================================================
async def close_job(session: AsyncSession, job_model: Any, task_model: Any, job_id: str, job_status: str,
job_error: str | None, task_status: str, task_error: str | None) -> None:
"""收尾:还没结束的表记 task_status(原因 task_error),任务(还没结束的话)记 job_status。提交。"""
now = utcnow()
await session.execute(
update(task_model).where(task_model.job_id == job_id, task_model.status.in_(OPEN))
.values(status=task_status, error=task_error, finished_at=now).execution_options(synchronize_session=False))
await session.execute(
update(job_model).where(job_model.id == job_id, job_model.status.in_(ACTIVE))
.values(status=job_status, error=job_error, finished_at=now).execution_options(synchronize_session=False))
await session.commit()
async def cancel_queued(session: AsyncSession, job_model: Any, task_model: Any, job_id: str) -> bool:
"""停止一个排队中的任务:任务和没结束的表直接记 cancelled。已经被后台认领(开始了)返回 False,交给调用方按进行中
的停。"""
now = utcnow()
claimed = await session.execute(
update(job_model).where(job_model.id == job_id, job_model.status == "queued")
.values(status="cancelled", finished_at=now).execution_options(synchronize_session=False))
if claimed.rowcount != 1:
await session.rollback()
return False
await session.execute(
update(task_model).where(task_model.job_id == job_id, task_model.status.in_(OPEN))
.values(status="cancelled", finished_at=now).execution_options(synchronize_session=False))
await session.commit()
return True
async def claim(session: AsyncSession, job_model: Any, job_id: str) -> bool:
"""认领:排队中 → 进行中。开始之前就被停止了返回 False。"""
result = await session.execute(
update(job_model).where(job_model.id == job_id, job_model.status == "queued")
.values(status="running", started_at=utcnow()).execution_options(synchronize_session=False))
await session.commit()
return result.rowcount == 1
async def interrupt_on_startup(job_model: Any, close: Callable[[AsyncSession, str], Awaitable[None]], *,
running: Callable[[str], bool]) -> int:
"""服务启动时:排队、进行中、而且不在这个进程里跑的任务逐个 close(各自记成 interrupted、没结束的表记 cancelled)。
不自动续跑。返回整理了几个。"""
async with SessionLocal() as session:
ids = [job_id for job_id in (await session.execute(
select(job_model.id).where(job_model.status.in_(ACTIVE)))).scalars() if not running(job_id)]
for job_id in ids:
await close(session, job_id)
return len(ids)
# ==========================================================================
# 在后台执行
# ==========================================================================
class JobRun:
"""一个任务在后台的一次执行。子类给 kind(进队时的身份)、model(任务表,进队前读源和状态),写 _run(执行)和
_stopped / _failed / _crashed(停止、整体出错、意外出错时的收尾)。"""
kind = ""
model: Any = None
def __init__(self, job_id: str) -> None:
self.job_id = job_id
self.task: asyncio.Task[None] | None = None
#: 为什么停:user 使用者点了停止,shutdown 服务关停(标中断)
self.stop_reason: str | None = None
self.writer = SerialWriter()
self.ticket: LaneTicket | None = None
async def _write(self, fn: Callable[..., Awaitable[Any]], *args: Any) -> Any:
"""一条库写入(SerialWriter:一条一条来,停止时不被打断)。"""
return await self.writer(fn, *args)
async def _settle(self) -> None:
await self.writer.settle()
async def run(self) -> None:
try:
try:
await self._queue()
await self._run()
except asyncio.CancelledError:
current = asyncio.current_task()
if current is not None:
current.uncancel()
await self._stopped()
except JobFailed as e:
await self._failed(str(e))
except Exception as e: # noqa: BLE001 - 任务整体出错:照实记下,不留在「进行中」
await self._crashed(e)
finally:
# 收尾写完才出队:排在后面的任务开始时,这个任务已经记成结束了
if self.ticket is not None:
self.ticket.release()
async def _queue(self) -> None:
"""进这个源的队,等轮到自己。任务不在了、已经不是排队中(开始之前就被停止了)的不进队:_run 认领不到,直接结束。"""
async with SessionLocal() as session:
job = await session.get(self.model, self.job_id)
source_id = job.source_id if job is not None and job.status == "queued" else None
if source_id is None:
return
self.ticket = lanes.enter(source_id, self.kind, self.job_id)
await self.ticket.wait()
async def _run(self) -> None:
raise NotImplementedError
async def _stopped(self) -> None:
raise NotImplementedError
async def _failed(self, reason: str) -> None:
raise NotImplementedError
async def _crashed(self, e: Exception) -> None:
raise NotImplementedError
class JobManager:
"""进程内的一种后台任务:一个 asyncio 任务一件事,停的时候给宽限。factory 按任务编号造一次执行(JobRun);label 是
日志里的叫法(「数据目录起草任务」),name 是 asyncio 任务名的前缀。"""
def __init__(self, factory: Callable[[str], JobRun], *, label: str, name: str) -> None:
self._runs: dict[str, JobRun] = {}
self._factory = factory
self.label = label
self.name = name
def _alive(self, job_id: str) -> JobRun | None:
run = self._runs.get(job_id)
if run is None or run.task is None or run.task.done():
return None
try:
if run.task.get_loop() is not asyncio.get_running_loop():
self._runs.pop(job_id, None) # 别的事件循环留下的(测试每个用例一个循环)
return None
except RuntimeError:
return None
return run
def start(self, job_id: str) -> None:
"""开始在后台执行这个(排队中的)任务:先进这个源的队,轮到了才开始。"""
run = self._factory(job_id)
run.task = asyncio.get_running_loop().create_task(run.run(), name=f"{self.name}-{job_id}")
self._runs[job_id] = run
def forget(_: asyncio.Task[None]) -> None:
if self._runs.get(job_id) is run:
self._runs.pop(job_id, None)
run.task.add_done_callback(forget)
def running(self, job_id: str) -> bool:
return self._alive(job_id) is not None
async def stop(self, job_id: str, reason: str, *, grace: float = STOP_GRACE_S) -> bool:
"""停掉这个进程里在跑(或在排队等)的任务,等它收尾(最多 grace 秒)。不在这个进程里返回 False。"""
run = self._alive(job_id)
if run is None:
return False
if run.stop_reason is None:
run.stop_reason = reason
run.task.cancel() # type: ignore[union-attr]
await asyncio.wait([run.task], timeout=grace) # type: ignore[list-item]
return True
async def wait(self, job_id: str, timeout: float | None = None) -> None:
"""等这个任务做完(测试、调试用)。不在跑直接返回。"""
run = self._alive(job_id)
if run is not None:
await asyncio.wait([run.task], timeout=timeout) # type: ignore[list-item]
async def shutdown(self, *, grace: float = SHUTDOWN_GRACE_S) -> None:
"""关停:停掉进行中、排队等着的任务(标中断),最多等 grace 秒收尾。"""
runs = [run for job_id in list(self._runs) if (run := self._alive(job_id)) is not None]
for run in runs:
if run.stop_reason is None:
run.stop_reason = "shutdown"
run.task.cancel() # type: ignore[union-attr]
if runs:
await asyncio.wait([run.task for run in runs], timeout=grace) # type: ignore[misc]
logger.info("已停止 %d 个%s", len(runs), self.label)
__all__ = ["ACTIVE", "FINISHED", "JobFailed", "JobManager", "JobRun", "LaneBusy", "LaneTicket", "OPEN", "RETRYABLE",
"SHUTDOWN_GRACE_S", "STOP_GRACE_S", "SerialWriter", "SourceLanes", "TASK_DONE", "TASK_STATUSES", "cancel_queued",
"claim", "close_job", "interrupt_on_startup", "lanes"]
sys.modules[__name__] = job_runtime
+5
View File
@@ -55,6 +55,11 @@ logger = logging.getLogger(__name__)
_OPEN = runtime.OPEN
#: 起草这边占同源队的几种事(叫法登记在运行时里,见 core/job_runtime.register_lane_kind):后台起草任务、同步起草、导入目录
runtime.register_lane_kind("draft", "起草")
runtime.register_lane_kind("sync_draft", "同步起草")
runtime.register_lane_kind("import", "导入目录")
#: 一个任务最多几张表;起草要求、本次说明最多几个字
MAX_TABLES = 500
GUIDANCE_MAX = 2000
+318 -58
View File
@@ -41,9 +41,9 @@ import re
import time
from collections.abc import Awaitable, Callable
from dataclasses import asdict, dataclass, field
from datetime import datetime, timezone
from datetime import date as _date, datetime, timezone
from decimal import Decimal
from typing import Any
from typing import Any, NamedTuple
from sqlalchemy import select, update
from sqlalchemy.dialects import mysql, oracle, postgresql, sqlite
@@ -99,7 +99,7 @@ _NUMBER_FIELDS: dict[str, tuple[str, float, float, bool]] = {
"table_max_queries": ("每张表最多查询数", 1, 200, True),
"table_max_s": ("每张表时长上限(秒)", 10, 600, False),
"query_timeout_s": ("单条查询时限(秒)", 1, 60, False),
"sample_size": ("抽样键值数", 10, 10_000, True),
"sample_size": ("核对键值数", 10, 10_000, True),
"max_scan_rows": ("整表统计行数上限", 0, 10_000_000, True),
}
_ENABLED_LABEL = "开启数据剖析"
@@ -241,16 +241,96 @@ def sql_literal(value: Any, kind: str | None) -> str | None:
return "'" + value.replace("'", "''") + "'"
#: 定长文字类型(CHAR(2)、NCHAR、bpchar、CHARACTER(2)):PostgreSQL、Oracle 取回的值带尾随空格补齐
_FIXED_CHAR = re.compile(r"^\s*(?:N?CHAR|BPCHAR|CHARACTER)\b(?!\s+VARYING)", re.IGNORECASE)
def is_fixed_char(type_name: str | None) -> bool:
"""定长文字列(CHAR(n)):值的尾随空格是补齐出来的,不是取值的一部分(SQL 比较时也不算)。"""
return bool(type_name) and bool(_FIXED_CHAR.match(str(type_name)))
class KeyLiterals(NamedTuple):
"""key_literals 的结果。literals:去父表里对的 IN 列表字面量(去重、保持顺序);unmatchable:按类型就对不上父键、
算作没对上的键值个数(去重后);unsafe:值本身不能安全拼进 SQL、不计入核对的个数。"""
literals: list[str]
unmatchable: int
unsafe: int
def key_literals(values: list[Any], kind: str | None, *, parent: str | None,
child_type: str | None) -> KeyLiterals:
"""从子表取回的键值 → 去父表里对的 IN 列表字面量(KeyLiterals)。
字面量按**被指向列**的类型族写(catalog.column_family),不是按子表取回的值:
- 父列是文字、子列是数:写成 '123'。PostgreSQL 上 varchar 列 IN (123) 直接报「operator does not exist」;
- 父列是数、子列是文字:只拼整数写法的值('0012' 也照 12 写)。别的('A1'、'WALKIN')不拼——Oracle 上数字列
IN ('A1') 会把字面量转成数报 ORA-01722,MySQL 会把 'A1' 转成 0 去比、对上不该对上的行——但它们**照样是子表
的键值**:整数主键里永远没有 'WALKIN',所以算作没对上,计入 unmatchable,由调用方留在覆盖率的分母里。要是
从分母里拿掉,散客代号占九成的一列只剩 1 个整数写法的键对上,就成了「覆盖率 100%」;
- 父列是时间:不在这里处理,_check_relation 先整条跳过(各家时间字面量写法不同、精度不同,键不核对);
- 父列类型认不出:照子表取回的值的类型写(sql_literal,以前的做法)。
子表是定长文字(CHAR(n))时先去掉尾随空格:PG、Oracle 取回的 'A ' 拿去和 VARCHAR 父列比就对不上,而按 CHAR 的
语义尾随空格本来就不算。不安全的值(反斜杠、「:名字」、控制字符、太长)照 sql_literal 的规矩跳过,记进 unsafe
(调用方另记「类型不支持核对,未计入」)。两个不同的取值写成同一个字面量('0012' 和 '12')只算一个,不算 unsafe。
"""
if parent == "time":
raise ValueError("被指向列是日期时间的关系不核对键值(见 _check_relation)")
fixed = is_fixed_char(child_type)
out: list[str] = []
unmatchable: set[str] = set()
unsafe = 0
for value in values:
if fixed and isinstance(value, str):
value = value.rstrip(" ")
if parent == "text":
if isinstance(value, bool):
unsafe += 1
continue
if isinstance(value, (int, float, Decimal)):
lit = sql_literal(value, "number")
lit = None if lit is None else "'" + lit + "'"
else:
lit = sql_literal(value, "text")
elif parent == "number":
if isinstance(value, str):
text = value.strip()
if not _INT_TEXT.fullmatch(text):
unmatchable.add(text)
continue
lit = str(int(text))
else:
lit = sql_literal(value, "number")
else:
lit = sql_literal(value, kind)
if lit is None:
unsafe += 1
else:
out.append(lit)
return KeyLiterals(list(dict.fromkeys(out)), len(unmatchable), unsafe)
_DIALECT_MODULES = {"sqlite": sqlite, "postgres": postgresql, "postgresql": postgresql, "mysql": mysql,
"mariadb": mysql, "oracle": oracle}
#: 日期列的最小值、最大值只看这个区间里的值(含下限、不含上限):0000-00-00(MySQL 关了严格模式写进去的零日期)、
#: 0001-01-01、9999-12-31(「没有」「永久」的占位日期)、PostgreSQL 的 -infinity / infinity 都落在区间外。区间外的
#: 值照实数出来(「有 N 个无效日期未计入」),不让一个占位日期把业务日期的范围撑成两千年。两端整天排除:带时刻的
#: 9999-12-31 23:59:59、文字存的「0001-01-01 00:00:00」也算占位
VALID_DATE_FROM = "0001-01-02"
VALID_DATE_BEFORE = "9999-12-31"
_ISO_DAY = re.compile(r"\d{4}-\d{2}-\d{2}")
class SqlDialect:
"""剖析 SQL 的方言差异:标识符怎么加引号、怎么只取前 n 行、统计信息在哪个系统视图里。只拼字符串,不连库。
"""剖析 SQL 的方言差异:标识符怎么加引号、怎么只取前 n 行、统计信息在哪个系统视图里、日期字面量和按天截断
怎么写、有没有块采样。只拼字符串,不连库。
标识符交给 SQLAlchemy 各方言的 identifier_preparer:需要时才加引号(memberTags 加、visits 不加),
MySQL 用反引号。Oracle 尤其不能一律加双引号:探查拿到的表名是 SQLAlchemy 规整过的小写,库里其实是
大写,"visits" 加了引号就成了另一张不存在的表;不加引号 Oracle 自己转大写才对得上。
大写,"visits" 加了引号就成了另一张不存在的表;不加引号 Oracle 自己转大写才对得上。反过来,库里真是小写的
对象(建表时写了 "visits")探查时记了 quote_name,这种要强制加引号、不转大写(table、column、stats_sql)。
"""
def __init__(self, kind: str | None) -> None:
@@ -262,13 +342,64 @@ class SqlDialect:
self._prep = self._dialect.identifier_preparer
self.oracle = self.kind == "oracle"
def quote(self, name: str) -> str:
return self._prep.quote(name)
def quote(self, name: str, *, force: bool = False) -> str:
"""标识符。force:探查记了 quote_name 的名字,原样加引号(Oracle 小写带引号的对象)。"""
return self._prep.quote_identifier(name) if force else self._prep.quote(name)
def table(self, meta: dict[str, Any] | None, name: str) -> str:
"""写进 FROM 的表名:探查时指定了 schema 的带上它(Oracle 只读账号名下没有对象,全靠 schema 前缀)。"""
schema = (meta or {}).get("schema")
return f"{self._prep.quote_schema(schema)}.{self.quote(name)}" if schema else self.quote(name)
quoted = self.quote(name, force=(meta or {}).get("quote_name") is True)
return f"{self._prep.quote_schema(schema)}.{quoted}" if schema else quoted
def column(self, meta: dict[str, Any] | None, name: str) -> str:
"""写进 SQL 的列名:表结构里这一列记了 quote_name 的强制加引号。"""
cols = (meta or {}).get("columns") or []
entry = next((c for c in cols if isinstance(c, dict) and c.get("name") == name), None)
return self.quote(name, force=bool(entry and entry.get("quote_name") is True))
def date_literal(self, value: Any, *, text: bool = False) -> str:
"""日期字面量(取值观察的时间窗口、日期列的有效区间用)。PostgreSQL、MySQL、Oracle 都认 DATE 'YYYY-MM-DD';
SQLite 没有日期类型,日期存成文字,写字符串。text:列是文字存的日期(VARCHAR),按字符串比——PostgreSQL 上
varchar 和 date 比较会直接报错。只收真实存在的日期(date 或 YYYY-MM-DD),别的抛 ValueError:拼进 SQL 的
只能是我们自己算出来的日子。"""
if isinstance(value, datetime):
value = value.date()
if isinstance(value, _date):
day = value.isoformat()
elif isinstance(value, str) and _ISO_DAY.fullmatch(value):
try:
day = _date.fromisoformat(value).isoformat()
except ValueError:
raise ValueError(f"不是有效的日期:{value}") from None
else:
raise ValueError(f"不是有效的日期:{value!r}")
if text or self.kind == "sqlite":
return f"'{day}'"
return f"DATE '{day}'"
def day_trunc(self, column: str) -> str:
"""时间列截到天(按日行数用)。PostgreSQL CAST(x AS DATE);MySQL DATE(x);Oracle 必须用 TRUNC(x):Oracle 的
DATE 带时分秒,CAST(ts AS DATE) 截不掉;SQLite 的日期是文字,取前 10 个字符。"""
if self.kind in ("postgres", "postgresql"):
return f"CAST({column} AS DATE)"
if self.kind in ("mysql", "mariadb"):
return f"DATE({column})"
if self.oracle:
return f"TRUNC({column})"
return f"substr({column}, 1, 10)"
def sample_from(self, table: str, percent: float) -> str | None:
"""块采样的 FROM:PostgreSQL TABLESAMPLE SYSTEM (p)、Oracle SAMPLE BLOCK (p),按数据块随机取约 p% 的行;
MySQL、SQLite 没有块采样,返回 None(调用方退回「按存储顺序的前 N 行」,并照实写口径)。"""
p = float(percent)
if not (0 < p < 100):
raise ValueError(f"采样比例要在 0 到 100 之间:{percent}")
if self.kind in ("postgres", "postgresql"):
return f"{table} TABLESAMPLE SYSTEM ({p:g})"
if self.oracle:
return f"{table} SAMPLE BLOCK ({p:g})"
return None
def limit(self, sql: str, n: int) -> str:
"""只取前 n 行。Oracle 11g 没有 LIMIT 也没有 FETCH FIRST,一律套一层 ROWNUM。"""
@@ -304,7 +435,8 @@ class SqlDialect:
if self.oracle:
denorm = self._dialect.denormalize_name
owner = sql_literal(denorm(schema), "text") if schema else "SYS_CONTEXT('USERENV', 'CURRENT_SCHEMA')"
table = sql_literal(denorm(name), "text")
# 小写带引号的对象(quote_name)在数据字典里就是小写,不能转成大写去找
table = sql_literal(name if (meta or {}).get("quote_name") is True else denorm(name), "text")
if not (owner and table):
return None
return f"SELECT NUM_ROWS FROM ALL_TABLES WHERE OWNER = {owner} AND TABLE_NAME = {table}"
@@ -335,8 +467,49 @@ class SqlDialect:
f"WHERE {column} IS NOT NULL GROUP BY {column} ORDER BY cnt DESC, {column}")
return self.limit(sql, n)
def min_max_sql(self, table: str, columns: list[str]) -> str:
return f"SELECT {', '.join(f'MIN({c}), MAX({c})' for c in columns)} FROM {table}"
def min_max_sql(self, table: str, columns: list[tuple[str, bool]]) -> str:
"""日期列的最小值、最大值和无效日期的个数:每列三个值(MIN、MAX、区间外的个数)。columns 是 (写进 SQL 的列,
是不是文字存的日期)。只看有效区间(VALID_DATE_FROM ≤ x < VALID_DATE_BEFORE)里的值,见 VALID_DATE_FROM。
整表统计,只对小表用。"""
parts = []
for c, text in columns:
lo = self.date_literal(VALID_DATE_FROM, text=text)
hi = self.date_literal(VALID_DATE_BEFORE, text=text)
valid = f"{c} >= {lo} AND {c} < {hi}"
parts.append(f"MIN(CASE WHEN {valid} THEN {c} END), MAX(CASE WHEN {valid} THEN {c} END), "
f"SUM(CASE WHEN {c} < {lo} OR {c} >= {hi} THEN 1 ELSE 0 END)")
return f"SELECT {', '.join(parts)} FROM {table}"
# ---- 取值观察(下一轮第 3 波用):窗口、按日行数、块采样。这里先把各方言的写法定下来,金样和容器集成测试都覆盖
def _window(self, column: str, since: Any, text: bool) -> str:
"""时间窗口的条件:since 起(含),并且早于占位日期 9999-12-31——PostgreSQL 的 infinity、Oracle 的 9999-12-31
都大于任何一天,不截掉上端就会在「最近 35 天」里多出一天。"""
return (f"{column} >= {self.date_literal(since, text=text)} "
f"AND {column} < {self.date_literal(VALID_DATE_BEFORE, text=text)}")
def day_volume_sql(self, table: str, column: str, since: Any, *, text: bool = False) -> str:
"""最近若干天按天计数:since 起(含)每天几行。只在日期列打头有索引时对大表发(走索引的范围扫描)。"""
day = self.day_trunc(column)
return (f"SELECT {day} AS d, COUNT(*) AS n FROM {table} "
f"WHERE {self._window(column, since, text)} GROUP BY {day} ORDER BY d")
def window_value_counts_sql(self, table: str, column: str, date_column: str, since: Any, n: int, *,
text: bool = False) -> str:
"""时间窗口里的取值分布:业务日期 since 起(含)的行里,出现最多的 n 个取值和各自的行数。"""
sql = (f"SELECT {column}, COUNT(*) AS cnt FROM {table} "
f"WHERE {self._window(date_column, since, text)} AND {column} IS NOT NULL "
f"GROUP BY {column} ORDER BY cnt DESC, {column}")
return self.limit(sql, n)
def sample_value_counts_sql(self, table: str, column: str, percent: float, n: int) -> str | None:
"""块采样里的取值分布;没有块采样的方言返回 None。"""
source = self.sample_from(table, percent)
if source is None:
return None
sql = (f"SELECT {column}, COUNT(*) AS cnt FROM {source} WHERE {column} IS NOT NULL "
f"GROUP BY {column} ORDER BY cnt DESC, {column}")
return self.limit(sql, n)
# ==========================================================================
@@ -581,8 +754,10 @@ async def estimate_size(run: Callable[..., Awaitable[QueryResult]], dialect: Sql
"""先读统计信息;读不到(SQLite、没分析过的表)再数到 max_scan_rows + 1 行为止。
统计信息为 0 或负数(PostgreSQL 没分析过的表是 -1,MySQL 的估算值可能是 0)当作不知道:一张其实很大的表
要是被当成空表,后面就会对它做整表统计。返回 (行数, 途中没查成的几条)。这张表的查询次数、时长用完(或者任务止损)
时抛 _Stop。
要是被当成空表,后面就会对它做整表统计。统计信息说的行数不超过 max_scan_rows 时也再数一次(多一条有上限的
计数):统计信息可能是很久以前的(表后来涨到了几千万行,ANALYZE 没再跑过),照它把大表当小表,就会对它做
整表的 COUNT(DISTINCT)、MIN/MAX。数过的结果是准数(count),数到上限还没数完就按大表算(at_least)。
返回 (行数, 途中没查成的几条)。这张表的查询次数、时长用完(或者任务止损)时抛 _Stop。
"""
skips: list[_Skip] = []
sql = dialect.stats_sql(meta, name)
@@ -592,7 +767,9 @@ async def estimate_size(run: Callable[..., Awaitable[QueryResult]], dialect: Sql
except _Skip as e:
skips.append(e)
else:
if rows is not None and rows > 0:
if rows is not None and rows > max(max_scan_rows, 0):
return TableSize(rows, "stats"), skips
if rows is not None and rows > 0 and max_scan_rows <= 0:
return TableSize(rows, "stats"), skips
if max_scan_rows <= 0:
return TableSize(None, "unknown"), skips
@@ -752,6 +929,8 @@ class _CodesOutcome:
rows: int
note: str
complete: bool = False
#: 列定义(ENUM)列出的取值:这次没出现的也写进码值(含义待填),码值是「可能的取值」
declared: list[str] = field(default_factory=list)
@dataclass
@@ -901,6 +1080,11 @@ def _column(meta: dict[str, Any], name: str) -> str | None:
if isinstance(c, dict) and str(c.get("name", "")).lower() == lowered), None)
def _column_meta(meta: dict[str, Any], name: str) -> dict[str, Any] | None:
"""表结构里这一列(类型、取值列表、quote_name);name 是表结构里的写法(_column 的返回值)。"""
return next((c for c in meta.get("columns") or [] if isinstance(c, dict) and c.get("name") == name), None)
def _constraint(meta: dict[str, Any], column: str) -> str | None:
"""这一列单独是主键或某个唯一约束时返回「主键」/「唯一约束」,否则 None。"""
wanted = [column.lower()]
@@ -950,7 +1134,7 @@ async def _unique(cx: _TableCx, table: str, meta: dict[str, Any], column: str,
if not size.within(cx.settings.max_scan_rows):
return None, "too_large"
d = cx.dialect
res = await cx.run(d.unique_check_sql(d.table(meta, table), d.quote(column)))
res = await cx.run(d.unique_check_sql(d.table(meta, table), d.column(meta, column)))
row = res.rows[0] if res.rows else [None, None]
total, distinct = _as_int(row[0]), _as_int(row[1])
if total is None or distinct is None:
@@ -969,20 +1153,39 @@ def _fk_note(child: tuple[bool | None, str]) -> str:
return f"{catalog.PROFILE_NOTE_PREFIX}外键约束已保证引用完整,未抽样核对覆盖率;子表一侧{basis}:{shape}"
#: MySQL 上比较文字按列的排序规则:常见的排序规则不分大小写(_ci),PAD SPACE 的还忽略尾随空格。覆盖率、唯一性、
#: 码值的分组都按它算,写进说明,人才知道「A01」和「a01」在这里算同一个值
MYSQL_COLLATION_NOTE = "MySQL 按列的排序规则比较文字(常见的排序规则不分大小写),覆盖率和唯一性按它计算"
MYSQL_CODES_COLLATION_NOTE = "MySQL 按列的排序规则分组(常见的排序规则不分大小写),大小写不同的写法可能并成一个取值"
def _collation_note(dialect: SqlDialect, family: str | None, note: str) -> str | None:
"""MySQL、MariaDB 上文字列(或类型认不出的列)的排序规则说明;别的方言、数字列没有。"""
return note if dialect.kind in ("mysql", "mariadb") and family in ("text", None) else None
def _relation_note(*, sample: int, matched: int, coverage: float, scan_cap: int | None,
skipped_values: int, parent: tuple[bool | None, str], child: tuple[bool | None, str] | None,
holds: bool) -> str:
"""关系的剖析备注:抽样规模、覆盖率、父键和子键唯一的依据。人据此判断这条结论有多可靠。
不写日期:什么时候剖析的看这一项的 updated_at(界面按本地时间显示)。"""
holds: bool, unmatchable: int = 0, collation: str | None = None) -> str:
"""关系的剖析备注:取了多少个键值、看了多少行、覆盖率、父键和子键唯一的依据。人据此判断这条结论有多可靠。
不写日期:什么时候剖析的看这一项的 updated_at(界面按本地时间显示)。
不说「抽样」:DISTINCT … LIMIT 取的是数据库先给出的那几个值,大表上是按存储顺序的前若干行里的值,不是随机抽的
(多半是最老的数据)。unmatchable:被指向列是数字、子表取到的却不是整数写法的键值个数(key_literals)——已经算在
sample 里、算作没对上,这里说明为什么没去父表里对。"""
parts: list[str] = []
low = coverage < catalog.PROFILE_VERIFY_COVERAGE
if low:
parts.append(f"抽样覆盖率 {_percent(coverage)},可能不是这条关系")
scope = f"(只看前 {scan_cap} 行)" if scan_cap else ""
sampled = f"子表抽样 {sample} 个不同键值{scope},父表对上 {matched} 个"
parts.append(f"覆盖率 {_percent(coverage)},可能不是这条关系")
scope = f"(按存储顺序只看前 {scan_cap} 行)" if scan_cap else ""
sampled = f"子表取了 {sample} 个不同键值{scope},父表对上 {matched} 个"
parts.append(sampled if low else f"{sampled},覆盖率 {_percent(coverage)}")
if skipped_values:
parts.append(f"另有 {skipped_values} 个键值的类型不支持核对,未计入")
if unmatchable:
parts.append(f"其中 {unmatchable} 个不是整数写法,被指向列是数字,算作没对上")
if collation:
parts.append(collation)
unique, how = parent
if unique is True:
parts.append(f"被指向列是{how}" if how in ("主键", "唯一约束") else "被指向列经计数核对唯一")
@@ -1026,24 +1229,32 @@ async def _check_relation(cx: _TableCx, table: str, rel: dict[str, Any], out: _T
return
d, s = cx.dialect, cx.settings
child_col, parent_col = _column_meta(meta, col), _column_meta(pmeta, pcol)
parent_family = catalog.column_family(parent_col)
if parent_family == "time":
# 时间键不核对:各家时间字面量写法、精度不同,拼出来的 IN 列表对不对得上说不准。不发子表的取值查询
raise _Skip("unsupported", f"{parent}.{pcol} 是日期时间,各库的时间写法和精度不同,不核对这种关系的覆盖率")
size = await cx.size_with_data(table, result)
scan_cap = cx.scan_cap(size)
child_table, parent_table = d.table(meta, table), d.table(pmeta, parent)
res = await cx.run(d.distinct_sample_sql(child_table, d.quote(col), s.sample_size, scan_cap=scan_cap),
res = await cx.run(d.distinct_sample_sql(child_table, d.column(meta, col), s.sample_size, scan_cap=scan_cap),
max_rows=s.sample_size)
kind = (res.column_types or {}).get(res.columns[0]) if res.columns else None
values = [row[0] for row in res.rows if row]
literals = list(dict.fromkeys(lit for v in values if (lit := sql_literal(v, kind)) is not None))
literals, unmatchable, unsafe = key_literals(values, kind, parent=parent_family,
child_type=(child_col or {}).get("type"))
if not values:
raise _Skip("no_data", f"{table}.{col} 没有非空值,无法核对")
if not literals:
if not literals and not unmatchable:
raise _Skip("unsupported", f"{table}.{col} 的取值类型不支持核对(只核对整数和文字)")
# 分母是子表取到的全部键值:不是整数写法、按类型就对不上整数父键的(unmatchable)也在里面,算作没对上
sample = len(literals) + unmatchable
matched = 0
for start in range(0, len(literals), IN_CHUNK):
sql = d.match_count_sql(parent_table, d.quote(pcol), literals[start:start + IN_CHUNK])
sql = d.match_count_sql(parent_table, d.column(pmeta, pcol), literals[start:start + IN_CHUNK])
matched += _as_int(_first(await cx.run(sql))) or 0
matched = min(matched, len(literals))
coverage = round(matched / len(literals), 4)
coverage = round(matched / sample, 4)
parent_unique = await _unique(cx, parent, pmeta, pcol, result)
child_unique = await _unique(cx, table, meta, col, result) if parent_unique[0] else None
@@ -1051,12 +1262,13 @@ async def _check_relation(cx: _TableCx, table: str, rel: dict[str, Any], out: _T
if child_unique is not None:
cardinality = "one_to_one" if child_unique[0] is True else "many_to_one"
holds = coverage >= catalog.PROFILE_VERIFY_COVERAGE and cardinality is not None
note = _relation_note(sample=len(literals), matched=matched, coverage=coverage, scan_cap=scan_cap,
skipped_values=len(values) - len(literals), parent=parent_unique, child=child_unique,
holds=holds)
note = _relation_note(sample=sample, matched=matched, coverage=coverage, scan_cap=scan_cap,
skipped_values=unsafe, parent=parent_unique, child=child_unique, holds=holds,
unmatchable=unmatchable,
collation=_collation_note(d, parent_family, MYSQL_COLLATION_NOTE))
checked = child_unique is not None and child_unique[0] is not None
out.relations.append(_RelationOutcome(rid=str(rel["id"]), target=_relation_target(rel), coverage=coverage,
cardinality=cardinality, holds=holds, sample=len(literals),
cardinality=cardinality, holds=holds, sample=sample,
matched=matched, note=note, cardinality_checked=checked))
@@ -1076,17 +1288,28 @@ _NOT_CODE_TAILS = frozenset({"id", "no", "num", "number", "name", "desc", "descr
"at", "date", "time", "on", "amount", "price", "count", "qty", "rate", "url"})
#: 像日期的取值:2026-07-01、2026/07/01,后面可以跟时间
_DATE_TEXT = re.compile(r"^\d{4}[-/]\d{2}[-/]\d{2}")
#: 日期列最小值、最大值不计入的值(见 VALID_DATE_FROM)在说明里的说法
INVALID_DATES = "零日期 0000-00-00、0001-01-01、9999-12-31 这类占位日期或无穷大"
def _code_type_ok(type_name: str | None) -> bool:
"""整数,或者短文字(定长不超过 MAX_CODE_LEN;SQLite 的 TEXT 不带长度,也算)。"""
t = (type_name or "").upper()
def _is_enum(type_name: str | None, enum_values: Any = None) -> bool:
"""取值列表有限的列:MySQL 的 ENUM(…) / SET(…),或者探查记了取值列表(PostgreSQL 的枚举:类型写法只是类型名,
setup_status 这种以 set 开头的名字不算 SET,见 catalog._MYSQL_ENUM)。"""
return bool(enum_values) or catalog._mysql_enum_kind(type_name) is not None
def _code_type_ok(type_name: str | None, enum_values: Any = None) -> bool:
"""ENUM / SET(取值列表有限,直接算码值候选)、整数,或者短文字(定长不超过 MAX_CODE_LEN;SQLite 的 TEXT 不带
长度,也算)。类型是按方言编译的完整写法:VARCHAR2(20 CHAR) 的长度按 20 算。"""
if _is_enum(type_name, enum_values):
return True
t = catalog._type_words(type_name)
if not t or any(k in t for k in ("CLOB", "BLOB", "BINARY", "JSON", "DATE", "TIME", "INTERVAL", "REAL", "FLOAT",
"DOUBLE", "BOOL")):
return False
if "INT" in t:
return True
size = re.search(r"\(\s*(\d+)\s*(?:,\s*(\d+)\s*)?\)", t)
size = re.search(r"\(\s*(\d+)(?:\s+(?:CHAR|BYTE))?\s*(?:,\s*(\d+)\s*)?\)", t)
if any(k in t for k in ("NUMBER", "NUMERIC", "DECIMAL")):
return size is None or not size.group(2) or int(size.group(2)) == 0
if any(k in t for k in ("CHAR", "TEXT", "STRING")):
@@ -1100,9 +1323,9 @@ def _looks_like_code(name: str) -> bool:
def _code_columns(meta: dict[str, Any], notes: dict[str, Any]) -> list[str]:
"""要取值分布的列:目录里度量类型记为状态的、剖析以前提过码值候选的(再核一次)、或者类型和命名像状态、
类型、渠道的整数或短文字列。主键、唯一约束列、关系里的键列不算;码值人工确认或驳回过的不再取。
被遮罩的列也列出来,由检查记为「被遮罩」跳过——要让人看得见为什么它没有码值。"""
"""要取值分布的列:目录里度量类型记为状态的、剖析以前提过码值候选的(再核一次)、ENUM / SET 列(取值列表有限,
不看名字)、或者类型和命名像状态、类型、渠道的整数或短文字列。主键、唯一约束列、关系里的键列不算;码值人工确认
或驳回过的不再取。被遮罩的列也列出来,由检查记为「被遮罩」跳过——要让人看得见为什么它没有码值。"""
pk = [c.lower() for c in meta.get("primary_key") or []]
unique = {str(g[0]).lower() for g in meta.get("unique") or [] if isinstance(g, list) and len(g) == 1}
keys = {str(c).lower() for r in notes.get("relations") or [] if isinstance(r, dict) for c in r.get("columns") or []}
@@ -1117,23 +1340,28 @@ def _code_columns(meta: dict[str, Any], notes: dict[str, Any]) -> list[str]:
status_measure = (isinstance(measure, dict) and measure.get("value") == "status"
and measure.get("status") != "rejected")
profiled = isinstance(codes, dict) and codes.get("source") == "profile"
if not (status_measure or profiled or _looks_like_code(name)):
enum = _is_enum(col.get("type"), col.get("enum_values"))
if not (status_measure or profiled or enum or _looks_like_code(name)):
continue
lowered = name.lower()
if (pk == [lowered]) or lowered in unique or lowered in keys or not _code_type_ok(col.get("type")):
if (pk == [lowered]) or lowered in unique or lowered in keys or not _code_type_ok(col.get("type"),
col.get("enum_values")):
continue
out.append(name)
return out
def _code_key(value: Any) -> str | None:
"""取值分布里的一个取值 → 码值(文字)。只认整数和不太长的文字。"""
def _code_key(value: Any, *, fixed: bool = False) -> str | None:
"""取值分布里的一个取值 → 码值(文字)。只认整数和不太长的文字。fixed:定长文字列(CHAR(n)),去掉补齐的尾随
空格(PG、Oracle 取回的是 'A ',WHERE status = 'A' 照样对得上,码值写 'A')。"""
if value is None or isinstance(value, bool):
return None
if isinstance(value, int):
return str(value)
if isinstance(value, float):
return str(int(value)) if value.is_integer() else None
if isinstance(value, str) and fixed:
value = value.rstrip(" ")
if isinstance(value, str) and value.strip() and len(value) <= MAX_CODE_LEN and not _CONTROL.search(value):
return value
return None
@@ -1151,34 +1379,53 @@ async def _check_codes(cx: _TableCx, table: str, column: str, out: _TableOutcome
if cx.is_masked(column):
raise _Skip("masked", f"{table}.{column} 在数据源设置中被遮罩,不取值", carry=False)
d, meta = cx.dialect, cx.tables[table]
col_meta = _column_meta(meta, column) or {}
size = await cx.size_with_data(table, result)
scan_cap = cx.scan_cap(size)
res = await cx.run(d.value_counts_sql(d.table(meta, table), d.quote(column), CODES_MAX + 1, scan_cap=scan_cap),
res = await cx.run(d.value_counts_sql(d.table(meta, table), d.column(meta, column), CODES_MAX + 1,
scan_cap=scan_cap),
max_rows=CODES_MAX + 1)
if not res.rows:
raise _Skip("no_data", f"{table}.{column} 没有非空值,无法取值分布")
if len(res.rows) > CODES_MAX:
raise _Skip("high_cardinality", f"{table}.{column} 的取值超过 {CODES_MAX} 个,不作为码值候选", carry=False)
values: list[tuple[str, int]] = []
fixed = is_fixed_char(col_meta.get("type"))
enum = _is_enum(col_meta.get("type"), col_meta.get("enum_values"))
counts: dict[str, int] = {}
for row in res.rows:
key = _code_key(row[0])
key = _code_key(row[0], fixed=fixed)
if key is None:
raise _Skip("unsupported", f"{table}.{column} 的取值不是整数或短文字,不作为码值候选", carry=False)
values.append((key, _as_int(row[1]) or 0))
if len(values) >= 3 and all(n == 1 for _, n in values):
# 每个取值只出现一次(票种编码 T01…T09):这是一列标识,不是状态码
counts[key] = counts.get(key, 0) + (_as_int(row[1]) or 0)
values = list(counts.items())
if len(values) >= 3 and all(n == 1 for _, n in values) and not enum:
# 每个取值只出现一次(票种编码 T01…T09):这是一列标识,不是状态码。ENUM 的取值列表是列定义给的,不按这条判
raise _Skip("high_cardinality", f"{table}.{column} 的取值各不相同,像标识而不是码值,不作为码值候选",
carry=False)
rows = sum(n for _, n in values)
scope = f"按前 {scan_cap} 行统计" if scan_cap else "统计全表"
# 不说「抽样」:大表看的是按存储顺序的前若干行(多半是最老的数据),不是随机抽的
scope = f"按存储顺序取前 {scan_cap} 行统计" if scan_cap else "统计全表"
spread = "、".join(f"{k}({_share(n, rows)})" for k, n in values)
note = f"{NOTE_PREFIX}{scope},{rows} 行非空值共 {len(values)} 个取值:{spread}。"
out.codes.append(_CodesOutcome(column=column, values=values, rows=rows, note=note, complete=scan_cap is None))
extra: list[str] = []
# ENUM 的取值列表(SET 的是成员,取值是成员的组合,不算):这次没出现的也是可能的取值
is_set = catalog._mysql_enum_kind(col_meta.get("type")) == "SET"
declared = [] if is_set else [str(v) for v in col_meta.get("enum_values") or []]
unseen = [v for v in declared if v not in counts]
if unseen:
extra.append(f"列定义里另有 {len(unseen)} 个取值这次没有出现:{'、'.join(unseen[:CODES_MAX])}")
family = "text" if enum else catalog.column_family(col_meta)
collation = _collation_note(d, family, MYSQL_CODES_COLLATION_NOTE)
if collation:
extra.append(collation)
tail = "".join(f";{x}" for x in extra)
note = f"{NOTE_PREFIX}{scope},{rows} 行非空值共 {len(values)} 个取值:{spread}{tail}。"
out.codes.append(_CodesOutcome(column=column, values=values, rows=rows, note=note, complete=scan_cap is None,
declared=unseen))
def _is_date_column(col: dict[str, Any]) -> bool:
"""日期类列(catalog.is_date_column:和 SQL 检查认时间列是同一个判断)。"""
return catalog.is_date_column(str(col.get("name") or ""), col.get("type"))
return catalog.is_date_column(str(col.get("name") or ""), col.get("type"), enum_values=col.get("enum_values"))
async def _check_dates(cx: _TableCx, table: str, columns: list[str], total: int, out: _TableOutcome,
@@ -1190,20 +1437,31 @@ async def _check_dates(cx: _TableCx, table: str, columns: list[str], total: int,
if not size.within(cx.settings.max_scan_rows):
raise _Skip("too_large", f"{table} 的行数超过整表统计的行数上限({cx.settings.max_scan_rows} 行),"
"不取日期列的最小值和最大值")
res = await cx.run(d.min_max_sql(d.table(meta, table), [d.quote(c) for c in columns]))
# 文字存的日期(SQLite 的 TEXT、业务库里的 VARCHAR 日期)按字符串和有效区间比;日期类型的按日期字面量比
texts = [catalog.column_family(_column_meta(meta, c)) != "time" for c in columns]
res = await cx.run(d.min_max_sql(d.table(meta, table), [(d.column(meta, c), t) for c, t in zip(columns, texts)]))
row = res.rows[0] if res.rows else []
invalid: dict[str, int] = {}
for i, column in enumerate(columns):
low, high = (row[2 * i], row[2 * i + 1]) if len(row) >= 2 * i + 2 else (None, None)
low, high, bad = (row[3 * i], row[3 * i + 1], row[3 * i + 2]) if len(row) >= 3 * i + 3 else (None, None, 0)
invalid[column] = _as_int(bad) or 0
if low is None or high is None or not (_DATE_TEXT.match(str(low)) and _DATE_TEXT.match(str(high))):
continue
result.date_ranges.append({"column": column, "min": str(low), "max": str(high)})
found: dict[str, Any] = {"column": column, "min": str(low), "max": str(high)}
if invalid[column]:
found["invalid"] = invalid[column]
result.date_ranges.append(found)
if total != 1:
return
if not result.date_ranges:
if invalid.get(columns[0]):
raise _Skip("no_data", f"{table}.{columns[0]} 只有 {invalid[columns[0]]} 个无效日期({INVALID_DATES}),"
"没有可识别的日期值")
raise _Skip("no_data", f"{table}.{columns[0]} 没有可识别的日期值")
found = result.date_ranges[0]
dropped = (f"另有 {found['invalid']} 个无效日期({INVALID_DATES})未计入," if found.get("invalid") else "")
note = (f"{NOTE_PREFIX}表里只有这一个日期类列,取值从 {str(found['min'])[:19]} 到 "
f"{str(found['max'])[:19]},可作为业务日期的参考。")
f"{str(found['max'])[:19]},{dropped}可作为业务日期的参考。")
out.business_date = _DateOutcome(column=found["column"], low=found["min"], high=found["max"], note=note)
@@ -1226,7 +1484,7 @@ def _relation_finding(o: _RelationOutcome, rel: dict[str, Any], *, status: str,
elif status == "verified":
summary = f"覆盖率 {_percent(o.coverage)}," + ("仍为有确证" if previous == "verified" else "升为有确证")
elif o.coverage < catalog.PROFILE_VERIFY_COVERAGE:
summary = f"抽样覆盖率 {_percent(o.coverage)},保持推断"
summary = f"覆盖率 {_percent(o.coverage)},保持推断"
else:
summary = "被指向列未核实唯一,保持推断"
return {"kind": "relation", "path": f"relations.{o.rid}", "target": o.target,
@@ -1241,6 +1499,8 @@ def _codes_value(c: _CodesOutcome, item: dict[str, Any] | None) -> dict[str, str
沿用它的含义;它写了、数据里没出现的取值也留着——码值是「可能的取值」,没出现不等于不存在。"""
old = item.get("value") if isinstance(item, dict) and isinstance(item.get("value"), dict) else {}
value = {key: str(old.get(key) or "") for key, _ in c.values}
for key in c.declared:
value.setdefault(key, str(old.get(key) or ""))
for key, meaning in old.items():
if key not in value and isinstance(meaning, str) and meaning.strip():
value[key] = meaning
@@ -1744,8 +2004,8 @@ def ensure_enabled(row: Any, settings: ProfileSettings | None = None) -> Profile
#: 有效行过滤核对的说法(写进这一项的 note;前面是模型给的依据)
FILTER_CHECKED = "剖析核对:排除 {share} 的行(共 {rows} 行)"
FILTER_SAMPLED = "剖析核对:按前 {rows} 行统计,排除 {share} 的行"
FILTER_SAMPLED_EDGE = "剖析核对:按前 {rows} 行统计,{what},请人工核对"
FILTER_SAMPLED = "剖析核对:按存储顺序取前 {rows} 行统计,排除 {share} 的行"
FILTER_SAMPLED_EDGE = "剖析核对:按存储顺序取前 {rows} 行统计,{what},请人工核对"
FILTER_UNCHECKED = "没有核对排除比例({why})"
FILTER_NOT_ENABLED = "数据源没有开数据剖析"
+3
View File
@@ -58,6 +58,9 @@ from app.db.models import CatalogDraftJob, CatalogProfileJob, CatalogProfileTask
logger = logging.getLogger(__name__)
#: 剖析任务占同源队(叫法登记在运行时里,见 core/job_runtime.register_lane_kind)
runtime.register_lane_kind("profile", "剖析")
#: 一个任务最多几张表
MAX_TABLES = 500
#: 每个源保留几个结束了的剖析任务(某张表最近一次剖析所在的任务另外保留)
+181
View File
@@ -0,0 +1,181 @@
"""目录视图:运行和助手读数据目录统一经这一个接缝(docs/design/next-round/01-效果度量与回归评测.md 做法 5,总方案 X7)。
**为什么要接缝。** 目录在四处被读:运行时的 db_schema / db_query 工具(目录同时冻结进表结构快照)、助手查看(look 表详情)、
自查用的 SQL 检查器、双盲复算(交给盲算的说明、比较两条 SQL 用的中文名)。以前各读各的、没有统一的开关,「开目录和关
目录」「只用已核实的项」「起草模型 A 的目录和 B 的」这些对照都做不了,评测期间目录被起草或剖析改了也不知道。现在四处都
经这里取,一处定读哪一份;lint 测试(tests/test_catalog_view.py)不许这几个模块绕过它直接 read_catalog、直接查 CatalogNote。
**视图**(CatalogView):
- mode:live 当前目录(缺省,行为和以前一样);snapshot 一份冻结的目录快照(工件 id 在 snapshot,内容见 SNAPSHOT_KIND);
off 不给目录。
- statuses:all 全部(驳回的照旧不给模型看);reviewed 只要已核实(verified)和已确认(confirmed)的项,推断的
(proposed)去掉——量的是模型起草、还没审过的那些项是帮忙还是添乱。
**视图从哪来。** 运行从 Run.options['catalog_view'] 读(launch_run 收 options,发起时定下、随 run.started 封存,审批恢复、
重启续跑都沿用——用上下文变量传的话恢复之后就丢了);生成过程(问数据的 answer_ops、评测的作答)从参数读。都不给就是 live。
冻结进表结构快照时,不是缺省视图的在工件引用的 meta 里记下视图(describe),证据面板据此写得出「本次运行目录:关」。
**快照的内容**(第 2 波的评测题集包写,这里只定读法):kind=eval_catalog_snapshot 的内容寻址工件,
{"sources": {源 id: {"tables": {表名: {"version": 版本, "notes": 目录}}}}}。快照里没有这个源就是这个源没有目录;
快照工件不在了照实报错(CatalogSnapshotMissing),不悄悄退回当前目录——那样对照就白做了。
"""
from __future__ import annotations
import logging
from collections.abc import Iterable, Mapping
from dataclasses import dataclass
from typing import Any
from app.data import catalog
from app.data.catalog import CatalogEntry
logger = logging.getLogger(__name__)
#: Run.options 里放视图的键
OPTION_KEY = "catalog_view"
MODES = ("live", "snapshot", "off")
#: 各状态集合留下哪些项的状态(all 不筛:驳回的由各处照旧经 visible_notes 去掉)
STATUS_SETS: dict[str, tuple[str, ...] | None] = {"all": None, "reviewed": ("verified", "confirmed")}
#: 目录快照工件的 kind(细案 01 做法 6)
SNAPSHOT_KIND = "eval_catalog_snapshot"
class CatalogViewInvalid(ValueError):
"""视图写得不对(服务端代码写错了才会有:客户端写不了 Run.options)。"""
class CatalogSnapshotMissing(RuntimeError):
"""视图指定的目录快照工件不在了、或者内容认不出。"""
@dataclass(frozen=True)
class CatalogView:
mode: str = "live"
statuses: str = "all"
#: mode=snapshot 时快照工件的 id
snapshot: str | None = None
def __post_init__(self) -> None:
if self.mode not in MODES:
raise CatalogViewInvalid(f"目录视图的 mode 只能是 {'、'.join(MODES)},当前为「{self.mode}」")
if self.statuses not in STATUS_SETS:
raise CatalogViewInvalid(f"目录视图的 statuses 只能是 {'、'.join(STATUS_SETS)},当前为「{self.statuses}」")
if (self.mode == "snapshot") != bool(self.snapshot):
raise CatalogViewInvalid("目录视图只有 mode=snapshot 时带快照工件 id,而且必须带")
@property
def is_default(self) -> bool:
return self.mode == "live" and self.statuses == "all"
def as_option(self) -> dict[str, Any]:
"""写进 Run.options['catalog_view'] 的形状。"""
return {"mode": self.mode, "statuses": self.statuses, **({"snapshot": self.snapshot} if self.snapshot else {})}
@classmethod
def parse(cls, raw: Any) -> CatalogView:
"""Run.options['catalog_view']、生成参数 → 视图。None 是缺省(live);写得不对抛 CatalogViewInvalid。"""
if raw is None:
return LIVE
if isinstance(raw, CatalogView):
return raw
if not isinstance(raw, Mapping):
raise CatalogViewInvalid(f"目录视图需要是一个对象,当前为「{raw}」")
unknown = set(raw) - {"mode", "statuses", "snapshot"}
if unknown:
raise CatalogViewInvalid(f"目录视图里不认识的键:{'、'.join(sorted(unknown))}")
return cls(mode=str(raw.get("mode") or "live"), statuses=str(raw.get("statuses") or "all"),
snapshot=str(raw["snapshot"]) if raw.get("snapshot") else None)
LIVE = CatalogView()
def of_options(options: Mapping[str, Any] | None) -> CatalogView:
"""一次运行的视图(Run.options 里的 catalog_view)。没有就是 live。"""
return CatalogView.parse((options or {}).get(OPTION_KEY))
def of_run(run_ctx: Any) -> CatalogView:
"""运行里的节点构造 ToolContext 时用:RunContext.extra['run_options'] 里的视图(runner 每段执行都放这份)。"""
from app.engine.runner import RUN_OPTIONS_KEY
return of_options((getattr(run_ctx, "extra", None) or {}).get(RUN_OPTIONS_KEY))
def describe(view: CatalogView | None) -> dict[str, Any] | None:
"""记进冻结快照 meta 的说法:缺省视图(live、全部)不记,别的照 as_option 记下。"""
view = view or LIVE
return None if view.is_default else view.as_option()
def restrict(view: CatalogView | None, notes: Mapping[str, Any] | None) -> dict[str, Any]:
"""一张表的目录在这个视图下留下什么(状态集合;驳回的项原样留着,见 catalog.restrict_notes)。"""
allowed = STATUS_SETS[(view or LIVE).statuses]
if allowed is None:
return dict(notes or {})
return catalog.restrict_notes(notes, allowed)
def _restricted(view: CatalogView, entries: Mapping[str, CatalogEntry]) -> dict[str, CatalogEntry]:
if STATUS_SETS[view.statuses] is None:
return dict(entries)
out: dict[str, CatalogEntry] = {}
for table, entry in entries.items():
if notes := restrict(view, entry.notes):
out[table] = CatalogEntry(table=entry.table, notes=notes, version=entry.version,
updated_by=entry.updated_by, updated_at=entry.updated_at)
return out
def _snapshot_tables(view: CatalogView, source_id: str) -> dict[str, CatalogEntry]:
from app.core.artifact_store import load
try:
content = load(view.snapshot or "")
except ValueError as e: # 内容和哈希对不上
raise CatalogSnapshotMissing(f"目录快照 {str(view.snapshot)[:12]}… 读不出:{e}") from e
if not isinstance(content, dict) or not isinstance(content.get("sources"), dict):
raise CatalogSnapshotMissing(f"目录快照 {str(view.snapshot)[:12]}… 不在了或内容认不出,这次不能按它读目录")
tables = ((content["sources"].get(source_id) or {}).get("tables")) or {}
out: dict[str, CatalogEntry] = {}
for table, body in tables.items():
if isinstance(body, Mapping) and isinstance(body.get("notes"), Mapping):
out[str(table)] = CatalogEntry(table=str(table), notes=dict(body["notes"]),
version=int(body.get("version") or 0))
return out
async def read_many(session: Any, view: CatalogView | None, source_ids: Iterable[str]) -> dict[str, dict[str, CatalogEntry]]:
"""几个数据源在这个视图下的目录:{源 id: {表名: CatalogEntry}},没有目录的源不出现。live 一次查询读出。"""
view = view or LIVE
ids = [i for i in dict.fromkeys(source_ids) if i]
if view.mode == "off" or not ids:
return {}
if view.mode == "snapshot":
raw = {sid: found for sid in ids if (found := _snapshot_tables(view, sid))}
else:
raw = await catalog.read_catalogs(session, ids)
return {sid: kept for sid, entries in raw.items() if (kept := _restricted(view, entries))}
async def read(view: CatalogView | None, source: Any) -> dict[str, CatalogEntry]:
"""一个数据源在这个视图下的目录:{表名: CatalogEntry}。用自己的短会话读(调用方的会话里已经加载的对象不会因为回滚
过期)。live 读不到(库出错)当没有目录、记日志:目录是附加说明,不能挡住查询;快照不在了照实抛 CatalogSnapshotMissing。"""
from app.db.base import SessionLocal
view = view or LIVE
source_id = getattr(source, "id", None)
if view.mode == "off" or not source_id:
return {}
if view.mode == "snapshot":
return _restricted(view, _snapshot_tables(view, source_id))
try:
async with SessionLocal() as own:
return (await read_many(own, view, [source_id])).get(source_id, {})
except Exception: # noqa: BLE001
logger.exception("读取数据源 %s 的数据目录失败,本次不附目录", getattr(source, "name", source_id))
return {}
__all__ = ["CatalogSnapshotMissing", "CatalogView", "CatalogViewInvalid", "LIVE", "MODES", "OPTION_KEY", "SNAPSHOT_KIND",
"STATUS_SETS", "describe", "of_options", "of_run", "read", "read_many", "restrict"]
+206
View File
@@ -0,0 +1,206 @@
"""CSV 走配方(Excel 期 5 第 4 项):把 CSV 适配成配方读取层认的样子——一张可见工作表的扫描结果(WorkbookScan)
和网格(Grid / 逐行流),之后的执行器、核对、说明、溯源、按期累积全部复用,不另写一套 CSV 的解析(否则会和配方
语言分叉,违背「一种说法」)。
**怎么认出是 CSV。** 和简单导入(tabular._CSV)一样按扩展名定:只有 .csv / .tsv 按 CSV 读,并且内容得是文字
(不是 zip、不是旧版 .xls 的 OLE 文件头、前 64 KB 里没有 NUL 字节)。以前只看内容,企业系统常见的「HTML 表格冒充
.xls」、误传的文本文件都会被拆成一张叫「CSV」的工作表去起草、核对,不再像以前那样明确拒收;现在别的扩展名照旧交给
xlsx_scan 拒收,内容是网页或 XML 的另给一句能照着做的话。编码识别复用 tabular 读 CSV 的那一段(UTF-8 / GB18030 /
带 BOM 的 UTF-16);.tsv 一律按制表符分隔(同 tabular),.csv 嗅探逗号、分号、制表符、竖线。扫描时定下的分隔符
记进工作表的 path(CSV_PATH 后面跟分隔符),之后读格子用同一个,不再各嗅各的。
**坐标沿用「工作表!A1」。** 细案原写「格子坐标记成 R{行}C{列}」,这里没这么做(见交付说明的偏离项):坐标在执行器、
核对、修复按钮、框选、溯源、裁判摘录、前端网格里一共有十来处解析和拼写,引入第二种写法就要处处认两种,漏一处就是
着色错位、下钻找不到格子;期 1 读 CSV 时(tabular._shape_reasons)用的也是 A1。CSV 没有工作表,名字固定为
CSV_SHEET:每月文件名不同(orders_2026-08.csv、orders_2026-09.csv),配方按工作表名认,名字必须稳定。行号是第几条
记录(引号里的换行不另起一行),列号是第几个字段,和用 Excel 打开这个 CSV 看到的 A1 坐标一致。
**格子的值。** CSV 里只有文字。只有写法毫无歧义的才转成数(-12、80.5,不超过 15 位有效数字),其余一律原样留作文字:
「1,234」(千分位,交给配方的 text_number 决定)、「007」(前导零是编码的一部分)、「+5」「1e5」「12%」、日期写法
(执行器按 DATE 列解析文字日期,起草按文字认日期)、「=SUM(…)」(CSV 里不是公式,Excel 打开时才会当公式算)。
空字段、只有空白的字段是空格(和 xlsx_scan 的口径一致)。没有公式、合并单元格、隐藏行列。
"""
from __future__ import annotations
import csv
import io
import re
from dataclasses import dataclass
from typing import Any, Iterator
from app.data.recipe_types import GridCell
from app.data.xlsx_scan import Bounds, SheetScan, UnsupportedTable, WorkbookScan
__all__ = ["CSV_PATH", "CSV_SHEET", "CSV_SUFFIXES", "MAX_COLS", "MAX_ROWS", "CsvTable", "cell_value", "is_csv_sheet",
"looks_like_csv", "looks_like_markup", "parse", "rows", "scan", "scan_any", "sheet_delimiter"]
#: CSV 的「工作表名」:配方按它认(SheetMatch.name),每月的文件名不同也认得出
CSV_SHEET = "CSV"
#: SheetScan.path 的开头:读取层据此知道这张「工作表」要从 CSV 里读,不去压缩包里找 XML。后面跟着扫描时定下的分隔符
#: ("csv:,"、"csv:\t");只有「csv:」的(这一版之前的暂存区)读的时候再嗅一次
CSV_PATH = "csv:"
#: 按 CSV 读的扩展名,和简单导入(tabular._CSV)一致
CSV_SUFFIXES = (".csv", ".tsv")
#: 嗅探认的分隔符
_DELIMITERS = ",;\t|"
#: 内容是网页或 XML(「HTML 表格冒充 .xls」、Excel 2003 的 XML 表格)时的说法
MARKUP_MESSAGE = ("无法打开该文件:它其实是网页或 XML 格式(有的系统导出的「Excel」文件是这样),不是 Excel 工作簿。"
"请用 Excel 打开后另存为 .xlsx 或 .csv 再上传")
NOT_TEXT_MESSAGE = ("无法按 CSV 读取这个文件:扩展名是 CSV,内容却不是文字(可能是图片、PDF 之类的文件改了扩展名)。"
"请确认上传的文件,或在 Excel 中另存为 .csv 后重新上传")
#: 和 Excel 的上限一致:超过的 CSV 用 Excel 打开也装不下,交给模型的格子坐标也写不出来
MAX_ROWS = 1_048_576
MAX_COLS = 16_384
#: 嗅探看多少字节
_SNIFF = 64 * 1024
#: 写法毫无歧义的整数、小数(不带正号、不带前导零、不用科学计数法)
_INT = re.compile(r"-?(?:0|[1-9][0-9]*)")
_DEC = re.compile(r"-?(?:0|[1-9][0-9]*)\.[0-9]+")
#: 超过这么多位有效数字的不转成数:转成浮点会丢位(19 位卡号撞值),留作文字
_DIGITS_MAX = 15
def looks_like_csv(raw: bytes) -> bool:
"""这份原件按 CSV 读:不是 zip、不是 OLE(旧版 .xls)、开头没有 NUL 字节、不是空文件。带 BOM 的 UTF-16 例外
(每个字符都带 NUL,Excel「Unicode 文本」另存出来的就是它)。"""
if not raw or raw[:4] == b"PK\x03\x04" or raw[:8] == b"\xd0\xcf\x11\xe0\xa1\xb1\x1a\xe1":
return False
if raw[:2] in (b"\xff\xfe", b"\xfe\xff"):
return True
return b"\x00" not in raw[:_SNIFF]
def looks_like_markup(raw: bytes) -> bool:
"""内容是网页或 XML:去掉 BOM 和开头的空白后以「<」开头。真的工作簿以 zip(PK)或 OLE 文件头开头,不会是这样。"""
head = raw[:4096]
if head[:2] in (b"\xff\xfe", b"\xfe\xff"):
try:
text = head.decode("utf-16", errors="ignore")
except UnicodeDecodeError:
return False
else:
text = head.decode("latin-1")
return text.lstrip("\ufeff\xef\xbb\xbf \t\r\n").startswith("<")
def _suffix(filename: str | None) -> str:
name = str(filename or "").rsplit("/", 1)[-1]
return "." + name.rsplit(".", 1)[-1].lower() if "." in name else ""
def cell_value(text: str) -> Any:
"""CSV 的一个字段 → 格子的值:空 / 只有空白是 None(空格),写法毫无歧义的整数、小数转成数,其余原样(文字)。"""
t = text.strip()
if not t:
return None
digits = len(t.lstrip("-").replace(".", "").lstrip("0")) or 1
if digits <= _DIGITS_MAX:
if _INT.fullmatch(t):
return int(t)
if _DEC.fullmatch(t):
return float(t)
return text
@dataclass
class CsvTable:
"""解析好的 CSV:records[i] 是第 i + 1 条记录的字段(原文)。"""
records: list[list[str]]
delimiter: str
def parse(raw: bytes, delimiter: str | None = None) -> CsvTable:
"""原件 → 记录。编码照 tabular 读 CSV 的做法;delimiter 不给时嗅探(同 tabular)。字段太长、引号不配对读不了时抛
UnsupportedTable。"""
from app.data import tabular
text = tabular._decode_csv(raw)
if delimiter is None:
try:
delimiter = csv.Sniffer().sniff(text[:8192], delimiters=_DELIMITERS).delimiter
except csv.Error:
delimiter = "," # 嗅不出来就按最常见的来
records: list[list[str]] = []
try:
for record in csv.reader(io.StringIO(text, newline=""), delimiter=delimiter):
if len(records) >= MAX_ROWS:
raise UnsupportedTable(f"CSV 文件超过 {MAX_ROWS:,} 行,超出可以导入的上限")
if len(record) > MAX_COLS:
raise UnsupportedTable(f"CSV 文件第 {len(records) + 1} 行有 {len(record):,} 列,超过 {MAX_COLS:,} 列的上限")
records.append(record)
except csv.Error as e:
raise UnsupportedTable(f"无法按 CSV 读取这个文件:第 {len(records) + 1} 行的写法不正确({e})。"
"请在 Excel 中重新另存为 CSV 后上传") from e
return CsvTable(records=records, delimiter=delimiter)
def _cells(table: CsvTable) -> Iterator[tuple[int, list[tuple[int, GridCell]]]]:
for r, record in enumerate(table.records, 1):
out = [(c, GridCell(v)) for c, field in enumerate(record, 1) if (v := cell_value(field)) is not None]
if out:
yield r, out
def is_csv_sheet(sheet: Any) -> bool:
"""这张「工作表」是 CSV 适配出来的(path 以 CSV_PATH 开头)。"""
return str(getattr(sheet, "path", "") or "").startswith(CSV_PATH)
def sheet_delimiter(sheet: Any) -> str | None:
"""扫描时定下的分隔符(记在 path 里);这一版之前的暂存区没记,返回 None(读的时候再嗅)。"""
tail = str(getattr(sheet, "path", "") or "")[len(CSV_PATH):]
return tail if len(tail) == 1 and tail in _DELIMITERS else None
def scan(raw: bytes, *, delimiter: str | None = None) -> WorkbookScan:
"""CSV 原件 → 扫描结果:一张可见工作表(名字 CSV_SHEET、路径 CSV_PATH 加分隔符),边界、非空格数、逐行非空格数的
游程。没有合并区、隐藏行列、公式、错误值、离群格(CSV 里每一格都是写出来的,没有残留格式)。"""
table = parse(raw, delimiter)
sheet = SheetScan(name=CSV_SHEET, state="visible", path=CSV_PATH + table.delimiter)
lo_r = lo_c = hi_r = hi_c = 0
runs: list[tuple[int, int, int]] = []
for r, cells in _cells(table):
n = len(cells)
sheet.nonempty += n
first, last = cells[0][0], cells[-1][0]
if not lo_r:
lo_r, lo_c, hi_c = r, first, last
lo_c, hi_c, hi_r = min(lo_c, first), max(hi_c, last), r
if runs and runs[-1][1] == r - 1 and runs[-1][2] == n:
runs[-1] = (runs[-1][0], r, n)
else:
runs.append((r, r, n))
if lo_r:
sheet.bounds = Bounds(lo_r, lo_c, hi_r, hi_c)
sheet.dimension = sheet.bounds.a1()
sheet.row_runs = runs
return WorkbookScan(sheets=[sheet], format="csv")
def scan_any(raw: bytes, filename: str | None) -> WorkbookScan:
"""配方导入的扫描入口:.csv / .tsv 走这里的适配(.tsv 按制表符分隔),其余照旧交给 xlsx_scan(读不了照旧拒收,
文案不变);内容是网页或 XML 的另给一句能照着做的话(MARKUP_MESSAGE)。扩展名是 CSV、内容却是工作簿(xlsx 改了
扩展名)的照工作簿读。"""
from app.data import xlsx_scan
suffix = _suffix(filename)
if suffix in CSV_SUFFIXES:
if looks_like_csv(raw):
return scan(raw, delimiter="\t" if suffix == ".tsv" else None)
if raw[:4] != b"PK\x03\x04":
raise UnsupportedTable(NOT_TEXT_MESSAGE)
elif looks_like_markup(raw):
raise UnsupportedTable(MARKUP_MESSAGE)
return xlsx_scan.scan(raw)
def rows(table: CsvTable, *, min_row: int, max_row: int, min_col: int,
max_col: int) -> Iterator[tuple[int, list[tuple[int, GridCell]]]]:
"""按行流式给一个矩形里的非空格:(行号, [(列号, 格)…]),口径同 xlsx_cells.sheet_rows(整行为空的不给)。"""
for r in range(max(min_row, 1), min(max_row, len(table.records)) + 1):
record = table.records[r - 1]
out = [(c, GridCell(v)) for c in range(max(min_col, 1), min(max_col, len(record)) + 1)
if (v := cell_value(record[c - 1])) is not None]
if out:
yield r, out
+92 -9
View File
@@ -165,6 +165,89 @@ def _unique_groups(constraints: list[dict[str, Any]], indexes: list[dict[str, An
return groups
def _storage_type(col_type: Any) -> Any:
"""PostgreSQL 的域(DOMAIN)换成它底下的类型(域套域的一路往下找);别的类型原样返回。"""
from sqlalchemy.dialects.postgresql import DOMAIN
seen = 0
while isinstance(col_type, DOMAIN) and getattr(col_type, "data_type", None) is not None and seen < 8:
col_type, seen = col_type.data_type, seen + 1
return col_type
def type_text(col_type: Any, dialect: Any) -> str:
"""反射到的列类型 → 存进表结构的写法:按这个数据源的方言编译的完整写法(VARCHAR2(20 CHAR)、NUMBER(10, 0)、
ENUM('paid','refund')、TIMESTAMP WITH TIME ZONE),编译不了的退回 str()。PostgreSQL 的域(DOMAIN)记它底下的类型:
域编译出来只是域名(settlement_amount),看不出是数字还是文字,名字里的词(set、UPDATE 里的 DATE)还会让类型大类、
日期列、码值候选认错;模型照着写 SQL 也要知道它存的是什么。
以前存的是 str(类型),即 SQLAlchemy 的通用写法:Oracle 的 VARCHAR2、NUMBER 丢了长度和精度,MySQL 的 ENUM 丢了
取值列表,PostgreSQL 的 TIMESTAMP 丢了时区。类型写不全,表结构前后比对就看不出类型变了,ENUM 状态列也进不了
码值候选。这一改会让新探查的表结构快照哈希变化(提示词里的类型写法变长),旧运行封存的快照不受影响。"""
if col_type is None:
return ""
col_type = _storage_type(col_type)
try:
compiled = str(col_type.compile(dialect=dialect))
except Exception: # noqa: BLE001 - 编译不了(NullType 等)就用通用写法,不让一列拖垮整次探查
return str(col_type)
if getattr(dialect, "name", "") == "oracle" and compiled.startswith("VARCHAR("):
# SQLAlchemy 把 Oracle 的 VARCHAR2 反射成通用的 VARCHAR,按 Oracle 方言编译出来是 VARCHAR(10 CHAR)(容器集成
# 测试实测)。Oracle 的数据字典里只有 VARCHAR2(建表写 VARCHAR 也存成 VARCHAR2),照数据字典写
compiled = "VARCHAR2" + compiled[len("VARCHAR"):]
return compiled
def _enum_values(col_type: Any) -> list[str] | None:
"""ENUM(MySQL、PostgreSQL 的枚举,都是 sqltypes.Enum 的子类)的取值列表,MySQL SET 的成员列表;别的类型 None。
套在域(DOMAIN)里的枚举也算(同 type_text)。"""
from sqlalchemy.dialects.mysql import SET
from sqlalchemy.types import Enum
col_type = _storage_type(col_type)
if isinstance(col_type, Enum):
values = list(col_type.enums or [])
elif isinstance(col_type, SET):
values = list(col_type.values or [])
else:
return None
return [str(v) for v in values]
def _forced_quote(name: Any) -> bool:
"""名字必须原样加引号:Oracle 反射回来的全小写对象(建表时写了 "visits")是 quoted_name(quote=True)。
它存进 JSON 就成了普通字符串,剖析拼 SQL 时不加引号,Oracle 把它转成大写 VISITS,查的是另一个不存在的对象。"""
return getattr(name, "quote", None) is True
def column_entry(col: Mapping[str, Any], dialect: Any) -> dict[str, Any]:
"""反射到的一列 → 表结构里的一列:名字、按方言编译的类型、可空、注释;ENUM / SET 另记取值列表(enum_values),
要强制加引号的名字另记 quote_name。只在有的时候写这两个键:其余列和以前一字不差。"""
entry: dict[str, Any] = {
"name": str(col["name"]),
"type": type_text(col.get("type"), dialect),
"nullable": bool(col.get("nullable", True)),
"comment": col.get("comment") or None,
}
values = _enum_values(col.get("type"))
if values:
entry["enum_values"] = values
if _forced_quote(col.get("name")):
entry["quote_name"] = True
return entry
def index_first_columns(indexes: list[dict[str, Any]], pk: list[str]) -> list[str]:
"""哪些列打头有索引:各索引(含主键)的第一列,去重、按名字排序。取值观察看大表的最新业务日期、按日行数时,
只在业务日期列打头有索引时才发窗口查询(按它走索引,不扫全表)。表达式索引(列名是 None)不算。"""
first = {str(pk[0])} if pk else set()
for ix in indexes:
cols = ix.get("column_names") or []
if cols and cols[0] is not None:
first.add(str(cols[0]))
return sorted(first)
async def introspect(source: Any, *, schema: str | None = None) -> dict[str, Any]:
"""探查整个库的结构,结果直接存进 DataSource.schema_cache。
@@ -266,24 +349,20 @@ async def introspect(source: Any, *, schema: str | None = None) -> dict[str, Any
# 只读账号名下没有这些对象,全靠跨 schema 授权
"qualified": f"{target_schema}.{name}" if target_schema else name,
"schema": target_schema,
"columns": [
{
"name": c["name"],
"type": str(c.get("type") or ""),
"nullable": bool(c.get("nullable", True)),
"comment": c.get("comment") or None,
}
for c in columns
],
"columns": [column_entry(c, sync_conn.dialect) for c in columns],
"primary_key": pk,
"comment": comment,
"is_view": name in views,
}
if _forced_quote(name):
tables[name]["quote_name"] = True
if name in fk_ok:
tables[name]["foreign_keys"] = _foreign_keys(fk_by.get(key) or [], target_schema, default_schema)
if name in uq_ok:
tables[name]["unique"] = _unique_groups(uq_by.get(key) or [],
(ix_by.get(key) or []) if name in ix_ok else [], pk)
if name in ix_ok:
tables[name]["index_first_columns"] = index_first_columns(ix_by.get(key) or [], pk)
payload: dict[str, Any] = {"tables": tables, "truncated": truncated,
"total": len(names) + len(views)}
if failures and not tables:
@@ -471,6 +550,10 @@ def describe_table(source: Any, table: str, *, catalog: Mapping[str, Mapping[str
marks.append("非空")
if col.get("comment"):
marks.append(col["comment"])
values = col.get("enum_values")
if values and "(" not in str(col.get("type") or ""):
# PostgreSQL 的枚举类型写法只有类型名(order_status):取值列表另外写出来,模型才知道能写哪些值
marks.append("取值:" + "、".join(str(v) for v in values[:30]) + ("…" if len(values) > 30 else ""))
suffix = f" {'、'.join(marks)}" if marks else ""
lines.append(f" {col['name']} {col['type']}{suffix}")
if catalog:
+79 -4
View File
@@ -43,6 +43,9 @@ from app.data.recipe_parsers import (
text_label,
)
from app.data.recipe_types import (
LIST_PERIOD_END,
LIST_PERIOD_START,
LIST_SUBTOTAL_DIM,
LIST_TOTAL_DIM,
ColumnOut,
CrosstabBlock,
@@ -271,6 +274,8 @@ _FIELD_LABEL = {
"parts": "组成列", "claims": "认领", "a": "一方", "b": "另一方", "reason": "理由",
"ignore_rows": "按行标签忽略的行", "ignore_columns": "按表头忽略的列", "ignore_outside": "按同一行文字忽略的数字",
"label": "标签", "anchor": "同一行的文字",
"subtotals": "小计行", "no_subtotals": "没有小计行", "scope": "小计管哪些明细", "group_column": "分组的列", "period": "每行属于哪一期",
"column": "日期列", "check": "核对方式", "synthesize": "加统计期列",
}
@@ -608,15 +613,25 @@ class _Checker:
for si, bi, block in self.lists:
base = ("sheets", si, "blocks", bi)
cols = [(c.name, base + ("columns", n, "name")) for n, c in enumerate(block.columns)]
if block.period is not None and block.period.synthesize:
# 期 5:加上的统计期起止两列也是这张表的列,和表头推出的列名不能撞
pp = base + ("period", "synthesize")
cols += [(LIST_PERIOD_START, pp), (LIST_PERIOD_END, pp)]
out.append(_Writer(block.table, base + ("table",), "data", cols, origin=_pointer(base)))
numeric = [(c.name, base + ("columns", n, "name")) for n, c in enumerate(block.columns) if c.type in _NUMERIC]
total = block.rows.total_row
if total is not None and total.keep_as:
tp = base + ("rows", "total_row", "keep_as")
tcols = [(LIST_TOTAL_DIM, tp)]
tcols += [(c.name, base + ("columns", n, "name"))
for n, c in enumerate(block.columns) if c.type in _NUMERIC]
out.append(_Writer(total.keep_as, tp, "reported_total", tcols,
out.append(_Writer(total.keep_as, tp, "reported_total", [(LIST_TOTAL_DIM, tp)] + numeric,
origin=_pointer(base + ("rows", "total_row"))))
sub = block.rows.subtotals
if sub is not None and sub.keep_as:
sp = base + ("rows", "subtotals", "keep_as")
scols = [(LIST_SUBTOTAL_DIM, sp)]
if sub.scope == "group_column" and sub.group_column:
scols.append((sub.group_column, base + ("rows", "subtotals", "group_column")))
out.append(_Writer(sub.keep_as, sp, "reported_total", scols + numeric,
origin=_pointer(base + ("rows", "subtotals"))))
return out
# ---- 8. 名字 ----
@@ -680,6 +695,10 @@ class _Checker:
if sheet.context:
continue
for bi, block in enumerate(sheet.blocks):
if isinstance(block, ListBlock) and block.period is not None:
# 期 5:每行属于哪一期要拿统计期来核对(按日期列)或填进去(加起止两列)
self.add(("sheets", si, "blocks", bi, "period"), "covers_without_context",
"要按统计期核对或填写每一行的统计期,但这个工作表没有配置统计期")
if not isinstance(block, CrosstabBlock):
continue
ap = ("sheets", si, "blocks", bi, "axis")
@@ -873,6 +892,10 @@ class _Checker:
if total is not None and any(ch.isdigit() for ch in total.pick):
self.add(("sheets", si, "blocks", bi, "rows", "total_row", "pick"), "period_literal",
f"合计行的开头词「{total.pick}」不能含数字(配方里不能写死年份或期次)")
sub = block.rows.subtotals
if sub is not None and any(ch.isdigit() for ch in sub.pick):
self.add(("sheets", si, "blocks", bi, "rows", "subtotals", "pick"), "period_literal",
f"小计行的开头词「{sub.pick}」不能含数字(配方里不能写死年份或期次)")
# ---- 4、9、13. 单位:键是本表列名,值在词表里,与原表标签、表头里的单位一致 ----
@@ -902,10 +925,13 @@ class _Checker:
self.unit_against(seg.keep_as.table, seg.keep_as.value, target.locate.title, "分段标题")
for _si, _bi, block in self.lists:
total = block.rows.total_row
sub = block.rows.subtotals
for c in block.columns:
self.unit_against(block.table, c.name, c.header, "表头")
if total is not None and total.keep_as and c.type in _NUMERIC:
self.unit_against(total.keep_as, c.name, c.header, "表头")
if sub is not None and sub.keep_as and c.type in _NUMERIC:
self.unit_against(sub.keep_as, c.name, c.header, "表头")
def after_target(self, si: int, bi: int, seg: DerivedSegment) -> DimensionSegment | None:
"""合计分段紧跟的那个 dimension 分段(同一块里按 id 找);找不到或不是 dimension 段时已报 derived_after_invalid。"""
@@ -998,6 +1024,8 @@ class _Checker:
if total is not None and total.label_column not in names:
self.add(base + ("rows", "total_row", "label_column"), "column_unknown",
f"合计标签所在的列「{total.label_column}」不在这个列表的列里{self._near(total.label_column, names)}")
self.check_subtotals(base, block, names)
self.check_list_period(base, block, names)
seen: dict[str, str] = {}
for n, c in enumerate(block.columns):
key = match_key(c.header)
@@ -1006,6 +1034,53 @@ class _Checker:
f"表头「{c.header}」与「{seen[key]}」是同一个表头")
seen.setdefault(key, c.header)
def check_subtotals(self, base: tuple[Any, ...], block: ListBlock, names: list[str]) -> None:
"""期 5:小计行的标签列、分组列要是这个列表的列;按组小计要给分组列;小计和合计的开头词不能互为前缀
(「合计」「合计小计」:一行就分不清是小计还是合计);配了小计又声明没有小计(no_subtotals)是矛盾的。"""
sub = block.rows.subtotals
if sub is None:
return
sp = base + ("rows", "subtotals")
if block.rows.no_subtotals:
self.add(base + ("rows", "no_subtotals"), "subtotals_conflict",
"配了小计行,又声明了这张列表没有小计行:二选一")
if sub.label_column not in names:
self.add(sp + ("label_column",), "column_unknown",
f"小计标签所在的列「{sub.label_column}」不在这个列表的列里{self._near(sub.label_column, names)}")
if sub.scope == "group_column":
if not sub.group_column:
self.add(sp + ("group_column",), "subtotal_group_missing", "按组小计时要指明按哪一列分组")
elif sub.group_column not in names:
self.add(sp + ("group_column",), "column_unknown",
f"小计分组的列「{sub.group_column}」不在这个列表的列里{self._near(sub.group_column, names)}")
elif sub.group_column:
self.add(sp + ("group_column",), "subtotal_group_missing",
"小计管「上一行小计之后的全部明细」时不按列分组,请去掉分组的列,或改为按组小计")
total = block.rows.total_row
if total is not None:
a, b = match_key(sub.pick), match_key(total.pick)
if a.startswith(b) or b.startswith(a):
self.add(sp + ("pick",), "subtotal_pick_conflict",
f"小计行的开头词「{sub.pick}」和合计行的「{total.pick}」一个是另一个的开头,分不清是小计还是合计")
def check_list_period(self, base: tuple[Any, ...], block: ListBlock, names: list[str]) -> None:
"""期 5:每行属于哪一期,按日期列核对和加起止两列只能二选一;按日期列时那一列要是这个列表的日期列。"""
period = block.period
if period is None:
return
pp = base + ("period",)
if (period.column is None) == (period.synthesize is None):
self.add(pp, "list_period_invalid", "每一行属于哪一期只能二选一:按一个日期列核对,或加统计期起止两列")
return
if period.column is not None:
col = next((c for c in block.columns if c.name == period.column), None)
if col is None:
self.add(pp + ("column",), "column_unknown",
f"核对统计期的列「{period.column}」不在这个列表的列里{self._near(period.column, names)}")
elif col.type != "DATE":
self.add(pp + ("column",), "list_period_invalid",
f"核对统计期的列「{period.column}」的类型要是日期")
# ---- 期 3:忽略规则、表头上方的标题(P3-SPEC 9.2) ----
def check_ignores(self) -> None:
+31 -8
View File
@@ -30,6 +30,7 @@ from app.data.recipe_parsers import hour_range, hour_range_total, match_key, tex
from app.data.recipe_types import (
UNION_VER, AccumulatePlan, ChangeClass, CheckResult, ColumnOut, CrosstabBlock, DerivedSegment,
DimensionSegment, PartInfo, Recipe, UnionPart, UnionReport, accumulate_blockers, derive_tables,
list_period_columns,
)
from app.data.table_versions import settle_journal, sha_json
@@ -508,14 +509,19 @@ def union_options(parts: list[tuple[str, str, str]], target_recipe_sha: str, *,
class _Table:
"""并集里的一张表:列(目标配方的在前、退役的按给定顺序追加在后)、主键、日期轴列。"""
"""并集里的一张表:列(目标配方的在前、退役的按给定顺序追加在后)、主键、日期轴列。期 5:列表写入的表按 period
分期(list_period_columns):按日期列的,U3 核对那一列落在该期统计期内;加统计期起止两列的,核对两列就是该期。"""
def __init__(self, name: str, cols: list[ColumnOut], grain: list[str]) -> None:
def __init__(self, name: str, cols: list[ColumnOut], grain: list[str],
period: tuple[str, ...] | None = None) -> None:
self.name = name
self.cols = cols
self.grain = grain
self.axis = next((c.name for c in cols if c.role == "axis"), None)
self.names = [c.name for c in cols]
self.period = period if period is None or all(c in self.names for c in period[1:]) else None
if self.axis is None and self.period is not None and self.period[0] == "column":
self.axis = self.period[1]
def _copy_col(col: ColumnOut) -> ColumnOut:
@@ -551,6 +557,7 @@ def _layout(target: Recipe, parts: list[UnionPart],
out: dict[str, _Table] = {}
retired = retired or {}
periods = list_period_columns(target)
for name, cols in tables.items():
mine = [_copy_col(c) for c in cols]
known = {c.name for c in mine}
@@ -561,7 +568,7 @@ def _layout(target: Recipe, parts: list[UnionPart],
mine.append(_copy_col(found) if found is not None else _copy_col(col))
known.add(col.name)
spec = specs.get(name)
out[name] = _Table(name, mine, list(spec.grain) if spec is not None else [])
out[name] = _Table(name, mine, list(spec.grain) if spec is not None else [], periods.get(name))
for name, cols in retired.items():
if name in out:
continue
@@ -831,20 +838,36 @@ def _finish(conn: sqlite3.Connection, layout: dict[str, _Table], parts: list[Uni
u2.append(f"表「{table.name}」有 {dup} 组主键重复")
checks.append(_check("U2", "union_pk", "累积后主键不重复", checked=len(keyed), failed=u2, sql=sql2))
# U3 统计期:第 k 期的 rowid 区间里,日期都在 [start_k, end_k] 之内
# U3 统计期:第 k 期的 rowid 区间里,日期都在 [start_k, end_k] 之内。期 5:按日期列分期的列表,日期为空也算不在;
# 加统计期起止两列的列表,两列必须正好是这一期
u3: list[str] = []
sql3, params3, checked3 = None, [], 0
for k, (part, ranges) in enumerate(zip(parts, part_rows), 1):
for name, span in ranges.items():
axis = layout[name].axis
if axis is None or axis not in part_cols[k - 1].get(name, []):
continue
table = layout[name]
have = part_cols[k - 1].get(name, [])
a, b = span["union"]
if table.period is not None and table.period[0] == "synthesize":
start_col, end_col = table.period[1], table.period[2]
if start_col not in have or end_col not in have or b < a:
continue
checked3 += 1
sql3 = (f"SELECT COUNT(*) FROM {_q(name)} WHERE rowid BETWEEN ? AND ? "
f"AND ({_q(start_col)} IS NOT ? OR {_q(end_col)} IS NOT ?)")
params3 = [a, b, part.start, part.end]
bad = conn.execute(sql3, params3).fetchone()[0]
if bad:
u3.append(f"第 {k} 期({_period_text(part.start, part.end)})的「{name}」有 {bad} 行的统计期不是这一期")
continue
axis = table.axis
if axis is None or axis not in have:
continue
if b < a:
continue
checked3 += 1
nulls = f"{_q(axis)} IS NULL OR " if table.period is not None else ""
sql3 = (f"SELECT COUNT(*) FROM {_q(name)} WHERE rowid BETWEEN ? AND ? "
f"AND ({_q(axis)} < ? OR {_q(axis)} > ?)")
f"AND ({nulls}{_q(axis)} < ? OR {_q(axis)} > ?)")
params3 = [a, b, part.start, part.end]
bad = conn.execute(sql3, params3).fetchone()[0]
if bad:
+42 -4
View File
@@ -23,6 +23,7 @@ from typing import Any, Callable, Iterable
from app.data.engine import open_checked_sqlite
from app.data.recipe_types import (
SUBTOTAL_KEY,
CheckResult,
ColumnOut,
CrosstabBlock,
@@ -34,6 +35,7 @@ from app.data.recipe_types import (
Recipe,
SumEq,
derive_tables,
list_period_columns,
)
#: 无法核对的原因(契约 CheckResult.reasons 的键)的固定顺序:细节、说明都按这个顺序列
@@ -71,24 +73,33 @@ class CheckPlan:
k: dict[str, str] = field(default_factory=dict)
#: derived 分段键 → G 编号(只给有公式格的 derived 段)
g: dict[str, str] = field(default_factory=dict)
#: 列表块键 → T 编号(每个有合计行的列表块一条)
#: 列表块键 → T 编号(每个有合计行的列表块一条);期 5:块键 + SUBTOTAL_KEY → 这个块的小计核对(一条)
t: dict[str, str] = field(default_factory=dict)
items: dict[str, list[DerivedItem]] = field(default_factory=dict)
def _recipe_keys(recipe: Recipe) -> tuple[list[str], list[str]]:
"""配方里 derived 分段键、带合计行的列表块键,按配方里的先后。"""
"""配方里 derived 分段键、带合计行的列表块键,按配方里的先后。期 5:带小计的列表块另有一个键(块键 +
SUBTOTAL_KEY),排在这个块的合计行后面:同一块的小计和合计各编一个 T,结论分开写。"""
derived: list[str] = []
totals: list[str] = []
for sheet in recipe.sheets:
for block in sheet.blocks:
if isinstance(block, CrosstabBlock):
derived += [s.id for s in block.segments if isinstance(s, DerivedSegment)]
elif block.rows.total_row is not None:
continue
if block.rows.total_row is not None:
totals.append(block.id)
if block.rows.subtotals is not None:
totals.append(f"{block.id}{SUBTOTAL_KEY}")
return derived, totals
def is_subtotal_key(key: str) -> bool:
"""T 编号的键是不是一个块的小计核对(期 5)。"""
return key.endswith(SUBTOTAL_KEY)
def _ordered(recipe_keys: list[str], items: list[DerivedItem]) -> list[str]:
"""先按配方里的先后;配方里找不到的分段(不该出现)按执行结果里第一次出现的先后排在后面,不丢。"""
present: list[str] = []
@@ -570,6 +581,25 @@ def _check_pk(sql: _Sql, cid: str, title: str, table: str, grain: list[str]) ->
# ---------------------------------------------------------------- 入口
def _check_period_rows(sql: _Sql, cid: str, title: str, table: str, column: str, period: Any,
lineage: dict[str, dict[str, list[list[Any]]]]) -> CheckResult:
"""期 5:列表按日期列分期(ListPeriod.column、check=within_context)时,每行的这一列都落在本期统计期内。
日期为空的行也算不在:看不出它属于哪一期。不在的行按期累积会混进别的期(统计期互不重叠的前提就破了),
所以是结构类、不能写理由接受。统计期没定下来(要人录入)时不会走到这里:那时不建库。"""
if period is None or not period.start or not period.end:
raise _Broken("本期没有统计期,无法核对每行的日期")
q = (f"SELECT rowid FROM {_q(table)} WHERE {_q(column)} IS NULL OR {_q(column)} < ? OR {_q(column)} > ? "
"ORDER BY rowid")
params = [period.start, period.end]
bad = [r[0] for r in sql.all(q, params)]
total = sql.one(f"SELECT COUNT(*) FROM {_q(table)}")[0]
cells = [c for rowid in bad[:_DETAIL_MAX] if (c := lineage_cell(lineage, table, column, rowid))]
details = [f"有 {len(bad)} 行的「{column}」为空或不在 {period.start} 至 {period.end} 之内"] if bad else []
return CheckResult(id=cid, kind="period_rows", title=title, status="mismatch" if bad else "passed",
category="structure", checked=int(total), failed=len(bad), sql=q, params=params,
details=details, cells=cells)
def _axis_names(recipe: Recipe) -> dict[str, str]:
"""derived 分段键 → 所在交叉表块的轴列名(K 的铺满判定不按日期分)。"""
out: dict[str, str] = {}
@@ -629,7 +659,9 @@ def run_checks(db_path: str, extraction: Extraction, recipe: Recipe) -> list[Che
jobs.append((cid, "formula_refs", f"合计公式引用(分段「{seg}」)",
lambda t, seg=seg, cid=cid: _check_g(sql, cid, t, plan.items[seg])))
for block, cid in plan.t.items():
jobs.append((cid, "column_sum", f"合计行按列求和(块「{block}」)",
title = (f"小计行按各组明细求和(块「{block.removesuffix(SUBTOTAL_KEY)}」)" if is_subtotal_key(block)
else f"合计行按列求和(块「{block}」)")
jobs.append((cid, "column_sum", title,
lambda t, block=block, cid=cid: _check_t(sql, cid, t, plan.items[block], full_calc)))
for rel in recipe.relations:
if isinstance(rel, SumEq):
@@ -646,6 +678,12 @@ def run_checks(db_path: str, extraction: Extraction, recipe: Recipe) -> list[Che
for i, table in enumerate([t for t in tables if grains.get(t)], 1):
jobs.append((f"P{i}", "pk_unique", f"主键唯一:「{table}」",
lambda t, table=table, i=i: _check_pk(sql, f"P{i}", t, table, grains[table])))
period = extraction.period
dated = [(t, spec[1]) for t, spec in list_period_columns(recipe).items() if spec[0] == "column" and t in tables]
for i, (table, column) in enumerate(dated, 1):
jobs.append((f"D{i}", "period_rows", f"「{table}」每行的「{column}」都在本期统计期内",
lambda t, table=table, column=column, i=i: _check_period_rows(
sql, f"D{i}", t, table, column, period, extraction.lineage)))
try:
return list(extraction.context_checks) + [_guarded(sql, *job) for job in jobs]
finally:
+39 -1
View File
@@ -43,7 +43,7 @@ from app.data.recipe_diff import (
)
from app.data.recipe_parsers import UNITS, match_key, period_residue, split_unit_suffix
from app.data.recipe_types import (
ConfirmItem, CrosstabBlock, DerivedSegment, Dismissed, ListBlock, MeasuresSegment, NotComparable, Recipe,
LIST_PERIOD_END, LIST_PERIOD_START, SUBTOTAL_KEY, ConfirmItem, CrosstabBlock, DerivedSegment, Dismissed, ListBlock, MeasuresSegment, NotComparable, Recipe,
SheetRecipe, SumEq, derive_tables,
)
@@ -523,6 +523,13 @@ def _signatures(recipe: Recipe, sheet_name: Callable[[SheetRecipe], str]) -> dic
total = block.rows.total_row
out[f"total_row:{block.id}"] = _sig(
None if total is None else (total.label_column, total.pick, total.keep_as))
# 期 5:穿插的小计、每行属于哪一期
subt = block.rows.subtotals
out[f"subtotals:{block.id}"] = _sig(
None if subt is None else (subt.label_column, subt.pick, subt.scope, subt.group_column, subt.keep_as))
out[f"no_subtotals:{block.id}"] = _sig(block.rows.no_subtotals)
period = block.period
out[f"list_period:{block.id}"] = _sig(None if period is None else (period.column, period.synthesize))
for r in recipe.relations:
if isinstance(r, SumEq):
out[f"relation:{r.id}"] = _sig("sum_eq", r.table, r.total, sorted(r.parts))
@@ -1058,6 +1065,29 @@ def _recipe_items(bag: _Bag, recipe: Recipe, ex: dict[str, Any], facts: Any, *,
detail = (f"按明细核对的列:{'、'.join(nums)};其余列合计行里的内容不核对" if nums
else "这个块没有数字列,合计行里的内容不核对")
bag.add(f"total_row:{b.id}", f"{prefix}改作核对{keep}", detail)
for b in lists:
# 期 5:穿插在明细中间的小计行:不进数据,按各组明细核对。要人确认:这几行从此不在本表里
subt = b.rows.subtotals
if subt is not None:
rows = rows_text(derived_rows.get(f"{b.id}{SUBTOTAL_KEY}"))
prefix = f"「{b.id}」中间的小计行" + (f"({rows})" if rows else "")
keep = f",原值另存表「{subt.keep_as}」" if subt.keep_as else ",原值不另存"
scope = ("每行小计核对上一行小计之后的全部明细" if subt.scope == "since_previous"
else f"每行小计核对紧挨在它上面、「{subt.group_column}」相同的那几行明细")
nums = [c.name for c in b.columns if c.type in ("INTEGER", "REAL")]
detail = f"{scope};按明细核对的列:{'、'.join(nums)}" if nums else f"{scope};这个块没有数字列,不核对"
bag.add(f"subtotals:{b.id}", f"{prefix}改作核对、不进数据{keep}", detail)
elif b.rows.no_subtotals:
# 声明没有小计行是放宽:以合计字样开头的行从此照常进数据,不再报「像小计」。配上(或改过)的那一次确认
bag.add(f"no_subtotals:{b.id}", f"「{b.id}」没有小计行:以合计字样开头的行都按数据导入,不再提醒像小计的行",
"如果其实有穿插的小计,对这些列求和会把小计重复算进去")
period = b.period
if period is not None and period.column:
bag.add(f"list_period:{b.id}", f"「{b.id}」每一行的「{period.column}」都要在本期统计期内",
"按期累积时各期的行按这一列区分;有一行不在本期统计期内(或为空),这一期整份拒收")
elif period is not None and period.synthesize:
bag.add(f"list_period:{b.id}", f"「{b.id}」每一行加上本期统计期的起、止两列",
f"列名「{LIST_PERIOD_START}」「{LIST_PERIOD_END}」,取自表内的统计期或人工录入的统计期")
# 年份取自统计期:表头是日期格时年份来自格子本身,不算
forms: dict[str, set[str]] = {}
@@ -1235,6 +1265,14 @@ def _period_items(bag: _Bag, recipe: Recipe, ex: dict[str, Any], ctx: ConfirmCon
bag.add(f"rows_after_stop:{blk}", ";".join(str(p.get("message") or "") for p in probs),
"、".join(cells[:20]), source="outside")
# 期 5:没配小计的列表里像小计的数据行:照常导入了,要人确认它们确实是数据(不是的话用修复按钮改作小计核对)
for p in ex.get("problems") or []:
p = plain(p)
if isinstance(p, dict) and p.get("code") == "subtotal_suspect":
blk = str((p.get("fix_args") or {}).get("block") or "、".join(str(c) for c in p.get("cells") or []))
bag.add(f"subtotal_suspect:{blk}", str(p.get("message") or ""),
"、".join(str(c) for c in (p.get("cells") or [])[:20]), source="outside")
# 列表块只有表头、没有数据行(AU-1):问题对象里没有块键,按消息里的「列表「块」」认
for p in ex.get("problems") or []:
p = plain(p)
+236 -15
View File
@@ -39,6 +39,14 @@ pk_duplicate 同时出现)。块级的定位失败(找不到日期表头、
个矩形。
- 回执里的「排除的行」rows_excluded(H2:排除的行要写进回执):隐藏行、列表跳过的空行、按规则忽略的行、
停止之后没导入的文字行、只核对不另存的合计行。
**期 5(下一轮 03 细案 E)又加了两件事,同样没用新字段的配方写出的库逐字节不变:**
- 列表中间穿插的小计(ListRows.subtotals):小计行不进数据,每个数字格一条合计格,按它所管的那组明细(上一行小计
之后的全部明细,或紧挨在上面、分组列相同的那几行)交给核对模块用 SQL 求和比对;整行记进排除的行(subtotal)。
- 列表按期累积时由系统加的统计期起止两列(ListPeriod.synthesize):列表在统计期定下来之前就抽完了,这两格先放
占位,统计期定下来以后写库时换成真值(_Sink.fill_period);按日期列分期的(ListPeriod.column)由核对模块的 D
核对查每行都在本期统计期内。
CSV 原件经 csv_grid 适配成一张「工作表」,这里照常按网格或逐行流读(xlsx_cells 按 SheetScan.path 分派)。
"""
from __future__ import annotations
@@ -59,15 +67,16 @@ from app.data import engine as _data_engine
from app.data import tabular, xlsx_cells
from app.data.names import canon
from app.data.recipe_parsers import (
AxisDate, HourRange, HourTotal, Period, classify_outside, cn_date_range, cn_year_month, filename_period,
TOTAL_WORDS, AxisDate, HourRange, HourTotal, Period, classify_outside, cn_date_range, cn_year_month,
filename_period,
hour_range, hour_range_total, is_pure_period_cell, looks_numeric_text, match_key, month_day_or_date,
period_residue, text_label, thousands_number,
)
from app.data.recipe_types import (
AxisOut, CanonEntry, CheckResult, ColumnOut, ContextSpec, CrosstabBlock, DerivedItem, DimensionSegment,
ExcludedRows, Extraction, Grid, GridCell, LedgerSheet, ListBlock, MeasuresSegment, OutsideText,
SUBTOTAL_KEY, AxisOut, CanonEntry, CheckResult, ColumnOut, ContextSpec, CrosstabBlock, DerivedItem,
DimensionSegment, ExcludedRows, Extraction, Grid, GridCell, LedgerSheet, ListBlock, MeasuresSegment, OutsideText,
PeriodInput, PeriodOut, Problem, Recipe, RegionMark, SegmentLabels, SheetRecipe, TableOut, TableSpec,
derive_tables,
derive_tables, list_period_columns,
)
from app.data.xlsx_scan import MAX_MERGED, SheetScan, WorkbookScan, cell_ref, col_letter
@@ -101,7 +110,8 @@ PROBLEM_CODES: dict[str, str] = {
# 列表定位
"header_not_found": "structure", "header_ambiguous": "structure", "column_missing": "structure",
"column_extra": "structure", "rows_after_stop": "confirm", "total_row_missing": "structure",
"total_row_blank": "structure", "list_empty": "confirm",
"total_row_blank": "structure", "list_empty": "confirm", "subtotal_blank": "structure",
"subtotal_suspect": "confirm",
# 值
"value_not_integer": "structure", "value_text_number": "structure", "value_blank": "structure",
"value_error": "structure", "value_not_number": "structure", "value_not_date": "structure",
@@ -125,6 +135,7 @@ PROBLEM_FIX: dict[str, str] = {
"axis_extra_cells": "ignore_cells", "column_extra": "ignore_cells", "row_unclaimed": "ignore_cells",
"outside_number": "ignore_cells",
"hidden_rows": "declare_hidden", "hidden_cols": "declare_hidden", "sheet_missing": "rename_sheet",
"subtotal_suspect": "declare_subtotals",
}
#: 列表找表头的窗口:从 after_title 的下一行(或已用区域首行)起往下这么多行
@@ -143,7 +154,7 @@ PLACEHOLDER_TEXTS_MAX = 8
PLACEHOLDER_TEXT_LEN = 8
#: 排除的行在回执里的顺序(同一工作表内按原因、块、锚点、首行排)
_EXCLUDED_ORDER = ("hidden_excluded", "blank_skipped", "ignored_rows", "ignored_outside", "after_stop",
"total_not_kept")
"total_not_kept", "subtotal")
#: 公式区域展开的上限:更大的区域不可能全落在基表里,直接判「引用了基表之外的格」
REF_CELLS_MAX = 200_000
@@ -613,11 +624,18 @@ class _Problems:
class _Sink:
"""一张表的行:主键去重、计数、溯源、批量写库(有库时)。rowid 就是插入序号(新库、1 起)。"""
def __init__(self, eng: _Engine, name: str, cols: list[ColumnOut], spec: TableSpec | None) -> None:
def __init__(self, eng: _Engine, name: str, cols: list[ColumnOut], spec: TableSpec | None,
synth: tuple[str, str] | None = None) -> None:
self.eng = eng
self.name = name
self.cols = cols
names = [c.name for c in cols]
#: 期 5:由系统加的统计期起止两列(ListPeriod.synthesize)在一行里的位置。列表是在统计期定下来之前抽完的,
#: 这两格先放占位(_PERIOD_START / _PERIOD_END),统计期定下来以后(fill_period)写库时换成真值;在那之前
#: 不分批写库(flush 等着),整张表留在内存里
self.synth_idx = (names.index(synth[0]), names.index(synth[1])) \
if synth and synth[0] in names and synth[1] in names else None
self.period: tuple[str | None, str | None] | None = None
self.grain = [g for g in (spec.grain if spec else []) if g in names]
self.grain_idx = [names.index(g) for g in self.grain]
self.check_pk = bool(self.grain_idx) and not eng.partial
@@ -682,9 +700,22 @@ class _Sink:
self.eng.problems.issue("value_blank", sheet, f"工作表「{sheet}」的表「{self.name}」的主键列{_quoted(names)}",
coords, text=_PK_NULL, group="pk_null")
def flush(self) -> None:
def fill_period(self, start: str | None, end: str | None) -> None:
"""统计期定下来了:加的统计期起止两列从此写真值(start、end 是 YYYY-MM-DD)。"""
self.period = (start, end)
def flush(self, *, final: bool = False) -> None:
if self.synth_idx is not None and self.period is None and not final:
return # 统计期还没定:加的两列还是占位,先不写
if self.buffer and self.eng.conn is not None:
self.eng.conn.executemany(self.insert_sql, self.buffer)
rows = self.buffer
if self.synth_idx is not None:
a, b = self.synth_idx
start, end = self.period or (None, None)
fill = {a: start, b: end}
rows = [tuple(fill.get(i, v) if v is _PERIOD_START or v is _PERIOD_END else v
for i, v in enumerate(row)) for row in rows]
self.eng.conn.executemany(self.insert_sql, rows)
self.buffer.clear()
@@ -695,6 +726,12 @@ class _Sink:
_FAIL = object()
#: 列表数据区里不能填充的合并单元格覆盖的空格(已经报了 merged_in_values)
_MERGED = object()
#: 期 5:加的统计期起止两列在统计期定下来之前的占位(_Sink.fill_period)。主键判重时它们和自己相等:
#: 同一期的行统计期都一样,判重只看其余的主键列,结论不变
_PERIOD_START = object()
_PERIOD_END = object()
#: 小计按组时「还没有组」
_NO_GROUP = object()
# --------------------------------------------------------------------------
@@ -1625,6 +1662,31 @@ class _ListRun:
self.ignore_keys = {k for rule in block.ignore_columns if (k := match_key(rule.header))}
total = block.rows.total_row
self.total_idx = next((i for i, c in enumerate(self.cols) if total and c.name == total.label_column), None)
# 期 5:穿插的小计行(ListRows.subtotals)。sub_first:上一行小计(或表头)之后第一行明细的 rowid;按组小计
# 时 run_first / run_value 是紧挨在上面、分组列取值相同的那一段明细的首个 rowid 和取值。sub_rows:小计所在的
# 行 → 它管的明细的 rowid(合计行的公式引用了小计格时,按它换算回明细)
self.sub = block.rows.subtotals
sub = self.sub
self.sub_idx = next((i for i, c in enumerate(self.cols) if sub and c.name == sub.label_column), None)
self.sub_key = match_key(sub.pick) if sub else ""
self.group_idx = next((i for i, c in enumerate(self.cols)
if sub and sub.scope == "group_column" and c.name == sub.group_column), None)
self.sub_first: int | None = None
self.run_first: int | None = None
self.run_value: Any = _NO_GROUP
self.sub_hidden = False
self.sub_rows: dict[int, list[int]] = {}
#: 期 5:加统计期起止两列(ListPeriod.synthesize):每行末尾两格占位,统计期定下来以后由 _Sink 换成真值
self.synth = block.period is not None and block.period.synthesize is not None
#: 期 5:没配小计时,像夹在明细中间的小计的数据行(行号)和命中的合计字样:照常按数据导入,收尾时报
#: subtotal_suspect 要人确认(带修复按钮:改作小计核对)。人声明过没有小计(no_subtotals)的不看
self.suspects: list[int] = []
self.suspect_word: str | None = None
self.watch_suspects = block.rows.subtotals is None and not block.rows.no_subtotals
#: 上一行普通明细在哪些文字、日期列(列序号,不含标签所在的首列)上有值;还没有明细时是空集
self.prev_filled: frozenset[int] = frozenset()
#: 等着下一行明细确认的候选:(行号, 合计字样, 上一行明细有值、这一行空着的列)
self.pending: list[tuple[int, str, frozenset[int]]] = []
# ---------------- 喂入 ----------------
@@ -1692,6 +1754,7 @@ class _ListRun:
if self.located:
for i, text in self.headers_read.items():
self.eng.headers[(self.block.table, self.cols[i].name)] = text
self._report_suspects()
# ---------------- 表头 ----------------
@@ -2001,6 +2064,7 @@ class _ListRun:
fills = self._merges_at(r, inb)
if self.sh.hidden_row(r):
self.hidden_seen = True
self.sub_hidden = True
if self.sh.sr.hidden.rows == "exclude":
self.excluded_rows.add(r)
self.sh.layout(r, self.layout_hidden)
@@ -2010,10 +2074,85 @@ class _ListRun:
label = inb.get(self.colmap.get(self.total_idx, -1))
pick = match_key(block.rows.total_row.pick) # type: ignore[union-attr]
if label is not None and match_key(_text(label.value)).startswith(pick):
if self.watch_suspects:
self._settle_suspects(None)
self._total_row(r, inb, cells, label)
return
if self.sub_idx is not None:
label = inb.get(self.colmap.get(self.sub_idx, -1))
if label is not None and match_key(_text(label.value)).startswith(self.sub_key):
self._subtotal_row(r, inb, cells, label)
return
elif self.watch_suspects:
self._watch_suspect(r, inb)
self._record(r, inb, cells, fills)
def _watch_suspect(self, r: int, inb: dict[int, GridCell]) -> None:
"""期 5:没配小计、也没声明没有小计时,认夹在明细中间的小计(照常按数据导入,收尾时报 subtotal_suspect)。
要同时满足才算:首列以合计字样开头、其余文字和日期列都空着、至少一格有数(_suspect_word);上一行明细在某个
文字或日期列上有值、这一行那一列空着;下一行明细在同一列上也有值,或者下面紧跟着合计行。所以候选先挂着,
等下一行明细来了再定(_settle_suspects)。
为什么这么严(复核反例):只有「标签列 + 数字列」的键值清单(「累计收入 300」「汇总调整 5」),「其余文字列
为空」自然成立,什么也说明不了;放在最后、后面没有明细的行也不是「夹在中间」。报错了就是每期都要勾的
确认项,按配方导入一期的无人值守运行会被它挡住。"""
filled = frozenset(i for i, col in enumerate(self.cols)
if col.type in ("TEXT", "DATE") and (c := self.colmap.get(i)) is not None
and c != self.c1 and c in inb)
word = self._suspect_word(inb) if self.prev_filled else None
if word is not None:
self.pending.append((r, word, self.prev_filled))
return
self._settle_suspects(filled)
self.prev_filled = filled
def _settle_suspects(self, filled: frozenset[int] | None) -> None:
"""挂着的候选有了下文:filled 是紧跟着的那行明细有值的文字、日期列(None 是紧跟着合计行)。上一行明细有值、
候选空着的列,下一行明细也有值,才算夹在明细中间。"""
for row, word, gap in self.pending:
if filled is None or gap & filled:
self.suspects.append(row)
self.suspect_word = self.suspect_word or word
self.pending = []
def _suspect_word(self, inb: dict[int, GridCell]) -> str | None:
"""期 5:这一行本身像不像小计:首列是以合计字样(TOTAL_WORDS)开头的文字,其余文字、日期列都空着,至少一格
有数。像的话返回那个字样。以前这种行静默当数据导入,求和就把小计重复算了进去(H2);现在照常导入,但收尾时
报出来要人确认,并给「改作小计核对」的修复按钮。合计行已经在前面认走了。上下文的条件见 _watch_suspect。"""
cell = inb.get(self.c1)
if cell is None or cell.formula is not None or not isinstance(cell.value, str):
return None
key = match_key(cell.value)
word = next((w for w in TOTAL_WORDS if key.startswith(w)), None)
if word is None or not any(self.colmap.get(i) == self.c1 for i in range(len(self.cols))):
return None
numeric = False
for i, col in enumerate(self.cols):
c = self.colmap.get(i)
if c is None or c == self.c1 or c not in inb:
continue
if col.type in ("TEXT", "DATE"):
return None
numeric = True
return word if numeric else None
def _report_suspects(self) -> None:
"""收尾:像小计的数据行合成一条 subtotal_suspect(需确认),修复参数带块 id、标签列、字样、行号和标签格。"""
if not self.suspects or not self.located:
return
label = next(col.name for i, col in enumerate(self.cols) if self.colmap.get(i) == self.c1)
cells = [self.sh.coord(r, self.c1) for r in self.suspects]
rows = _rows_text(self.suspects)
word = self.suspect_word or ""
where = (f"{self.where}:第 {rows} 行像夹在明细中间的小计(「{label}」列以「{word}」开头,上下明细都有值的"
"文字列在这一行空着),按数据导入了")
msg = (f"{where}。如果它们是小计,对这一列求和会把小计重复算进去:请用修复按钮改作小计核对;"
"确实是数据时,确认后照常导入;以后各期不想再确认,在配方的「小计行」里勾选「没有小计行」")
self.eng.problems.add("subtotal_suspect", msg, where, cells[:CELLS_MAX], sheet=self.sh.name,
fix_args={"block": self.block.id, "label_column": label, "pick": word,
"rows": list(self.suspects), "cells": cells[:CELLS_MAX]})
def _plan(self) -> list[tuple[Any, int, bool, str]]:
"""每列的(列定义、所在列号、是不是主键、位置说明):定下表头之后算一次,逐行复用。"""
plan = self.__dict__.get("_plan_cache")
@@ -2073,6 +2212,8 @@ class _ListRun:
v = None
values.append(v)
lineage.append((col.name, r, c))
if self.synth:
values += [_PERIOD_START, _PERIOD_END]
ignored = self.ignored
claims = {c: ("ignored_column" if c in ignored else "value") for c in inb}
self.sh.layout(r, self.layout_data)
@@ -2084,6 +2225,12 @@ class _ListRun:
if self.rowid_first is None:
self.rowid_first = rowid
self.rowid_last = rowid
if self.sub_first is None:
self.sub_first = rowid
if self.group_idx is not None:
group = values[self.group_idx]
if self.run_value is _NO_GROUP or group != self.run_value:
self.run_first, self.run_value = rowid, group
self.emit(r, cells, claims, True)
def _total_row(self, r: int, inb: dict[int, GridCell], cells: list[tuple[int, GridCell]],
@@ -2142,7 +2289,67 @@ class _ListRun:
self.stop = ("total", r)
self.emit(r, cells, claims, True)
def _ref_lookup(self, rc: tuple[int, int]) -> str | int:
def _subtotal_row(self, r: int, inb: dict[int, GridCell], cells: list[tuple[int, GridCell]],
label: GridCell) -> None:
"""期 5:穿插在明细中间的一行小计。不进数据:标签格和数字格的去向是 total_label / total_value(同合计行),
每个数字格一条合计格(DerivedItem,kind=column_sum,分段键是块 id + SUBTOTAL_KEY),核对模块按它所管的那组
明细(rowid 闭区间)用 SQL 逐列求和比对,执行器不做加法(H1);整行记进排除的行(reason=subtotal)。配了
keep_as 的另存一行(小计项、按组时的分组取值、各数字列)。
它管哪几行:since_previous 是上一行小计(或表头)之后的全部明细;group_column 是其中紧挨在它上面、分组列
取值相同的那一段。一行明细都没有时区间为空,核对判「无法核对」(合计行之上没有明细行)。数字格全空的小计
没有一列能核对,报 subtotal_blank(结构类):不报的话这一行就悄悄没了核对。"""
eng, sh, block, sub = self.eng, self.sh, self.block, self.sub
assert sub is not None
claims = {c: ("ignored_column" if c in self.ignored else "total_label") for c in inb}
label_raw = _text(label.value)
first = self.sub_first
if sub.scope == "group_column" and first is not None and self.run_first is not None:
first = max(first, self.run_first)
last = self.rowid_last if first is not None else None
numeric = [(i, col) for i, col in enumerate(self.cols) if col.type in ("INTEGER", "REAL")]
if numeric and not any(self.colmap[i] in inb for i, _ in numeric):
coord = sh.coord(r, self.colmap[self.sub_idx]) # type: ignore[index]
where = f"{self.where}:小计行({coord.split('!')[-1]})的数字格全部为空"
eng.problems.add(
"subtotal_blank",
f"{where},无法按这一组明细核对。请在文件中补上小计后重新上传;原表不再有小计时,在配方面板的"
"「小计行」中取消「有小计行」",
f"{where},无法核对", [coord], sheet=sh.name)
key = f"{block.id}{SUBTOTAL_KEY}"
keep_values: list[Any] = []
for i, col in numeric:
c = self.colmap[i]
cell = inb.get(c)
if cell is None:
keep_values.append(None)
continue
claims[c] = "total_value"
got = eng.convert_number(cell, col.type, block.values, sh.name, f"{self.where}的小计行", r, c,
data_area=False)
keep_values.append(got)
item = DerivedItem(
kind="column_sum", segment=key, sheet=sh.name, cell=sh.coord(r, c), label_raw=label_raw,
label=canon(label_raw), value=None if got is _FAIL else got, is_formula=cell.formula is not None,
formula=cell.formula, base_table=block.table, base_value=col.name, rowid_first=first,
rowid_last=last, hidden_in_range=self.sub_hidden, keep_table=sub.keep_as)
if item.is_formula:
eng.resolve_refs(item, cell.formula or "", self._ref_lookup)
eng.ex.derived.append(item)
if sub.keep_as and sub.keep_as in eng.sinks:
ks = eng.sinks[sub.keep_as]
ks.sources.append(f"{sh.sr.id}/{block.id}")
group = [None if self.run_value is _NO_GROUP else self.run_value] if self.group_idx is not None else []
lin = [(col.name, r, self.colmap[i]) for i, col in numeric]
ks.add([canon(label_raw), *group, *keep_values], sheet=sh.name,
at=(r, self.colmap[self.sub_idx]), cells=lin, fails=True) # type: ignore[index]
n = sum(1 for role in claims.values() if role in ("total_label", "total_value"))
sh.exclude("subtotal", r, r, n, block=block.id, anchor=sub.pick)
self.sub_rows[r] = list(range(first, last + 1)) if first is not None and last is not None else []
self.sub_first, self.run_first, self.run_value, self.sub_hidden = None, None, _NO_GROUP, False
self.emit(r, cells, claims, True)
def _ref_lookup(self, rc: tuple[int, int]) -> str | int | list[int]:
r, c = rc
if self.c1 <= c <= self.c2:
rowid = self.row_rowid.get(r)
@@ -2150,6 +2357,9 @@ class _ListRun:
return rowid
if r in self.excluded_rows:
return "hidden"
if r in self.sub_rows:
# 合计行的公式引用了小计格(=C5+C9):小计格代表它管的那组明细
return self.sub_rows[r]
return "outside"
@@ -2343,7 +2553,8 @@ class _Engine:
self.canon_log[key] = CanonEntry(table, col.name, text, stored, 1, _coord(sheet, r, c))
return stored
def resolve_refs(self, item: DerivedItem, formula: str, lookup: Callable[[tuple[int, int]], str | int]) -> None:
def resolve_refs(self, item: DerivedItem, formula: str,
lookup: Callable[[tuple[int, int]], str | int | list[int]]) -> None:
"""合计公式的引用经溯源:全在基表 → ref_rowids;有隐藏行或分类汇总函数 → hidden;
有不在基表的 → outside;形状认不出 → unrecognized。执行器不做加法(H1)。"""
shape = formula_refs(formula)
@@ -2357,7 +2568,9 @@ class _Engine:
outside = not cells
for rc in cells:
hit = lookup(rc)
if isinstance(hit, int):
if isinstance(hit, list):
rowids.extend(hit) # 期 5:引用了小计格,换算成它管的明细
elif isinstance(hit, int):
rowids.append(hit)
elif hit == "hidden":
hidden = True
@@ -2607,8 +2820,9 @@ class _Engine:
self.ex.problems = self.problems.finish(set())
return self.ex
self.cols = cols
synth = {t: (spec[1], spec[2]) for t, spec in list_period_columns(self.recipe).items() if spec[0] == "synthesize"}
for name, cs in cols.items():
self.sinks[name] = _Sink(self, name, cs, self.specs.get(name))
self.sinks[name] = _Sink(self, name, cs, self.specs.get(name), synth.get(name))
if self.db_path is not None:
_remove_db(self.db_path)
self.conn = _open_sqlite(self.db_path, readonly=False)
@@ -2622,6 +2836,11 @@ class _Engine:
crosstabs.extend(self._phase1(sh))
t2 = time.perf_counter()
period, period_reported, _ = self._period()
if self.ex.period is not None:
# 期 5:列表加的统计期起止两列现在才有值(列表在统计期定下来之前就抽完了)
for sink in self.sinks.values():
if sink.synth_idx is not None:
sink.fill_period(self.ex.period.start, self.ex.period.end)
for ct in crosstabs:
ct.complete_axis(period, period_reported)
ct.extract()
@@ -2650,7 +2869,7 @@ class _Engine:
_remove_db(self.db_path) # type: ignore[arg-type]
else:
for sink in self.sinks.values():
sink.flush()
sink.flush(final=True)
self.conn.execute("COMMIT")
self.conn.close()
self.conn = None
@@ -2748,7 +2967,9 @@ class _Engine:
sh.failed = True
return []
b = s.bounds
if b is not None and not self.partial and b.area > tabular.SPARSE_FACTOR * max(s.nonempty, tabular.SPARSE_FLOOR):
# 稀疏判定挡的是 Excel 里误输入、残留格式撑大的已用区域;CSV 的每一格都是写出来的,空字段多只是数据稀疏
if b is not None and not self.partial and self.scan.format != "csv" \
and b.area > tabular.SPARSE_FACTOR * max(s.nonempty, tabular.SPARSE_FLOOR):
far = "、".join(list(dict.fromkeys(s.far_cells))[:3])
msg = (f"工作表「{sh.name}」的已用区域 {b.a1()} 共 {b.area:,} 个单元格,但只有 {s.nonempty:,} 个单元格有内容;"
f"远处的单元格(如 {far})可能是误输入或残留格式。请在 Excel 中删除这些单元格后重新上传")
+62 -1
View File
@@ -253,6 +253,8 @@ def _key(kind: str, target: dict[str, Any]) -> str:
return f"declare_hidden:{t['sheet']}:{t['axis']}"
if kind == "rename_sheet":
return f"rename_sheet:{t['sheet_id']}"
if kind == "declare_subtotals":
return f"declare_subtotals:{t['block']}"
raise KeyError(kind)
@@ -750,6 +752,62 @@ def _o_declare_total(recipe: dict, p: FixProposal, opt: FixOption, reason: str |
return ops, summary
# ==========================================================================
# 期 5:列表中间像小计的行改作小计核对
# ==========================================================================
def _subtotal_keep_name(recipe: dict, block: dict) -> str:
used = {collide_key(t.get("name") or "") for t in recipe.get("tables") or []}
return unique_name(f"{block.get('table')}_小计", used)
def _p_declare_subtotals(recipe: dict, prob: Any, args: dict, anchor: FixAnchor) -> FixProposal | None:
"""subtotal_suspect → 列表块加 rows.subtotals(标签列、开头词取自执行器给的 fix_args:文件里的合计字样)。
块已经配了小计、标签列不在块里、开头词含数字时不提议。"""
found = find_block(recipe, args.get("block"))
if found is None:
return None
_si, _bi, block = found
label, pick = args.get("label_column"), args.get("pick")
if block.get("layout") != "list" or (block.get("rows") or {}).get("subtotals"):
return None
if label not in [c.get("name") for c in block.get("columns") or []] or not isinstance(pick, str) \
or not pick or has_digit(pick):
return None
target = {"block": block["id"], "label_column": label, "pick": pick}
keep = _subtotal_keep_name(recipe, block)
return _proposal("declare_subtotals", _key("declare_subtotals", target), code=_get(prob, "code"),
title=f"列表「{block['id']}」中间「{pick}」开头的行是小计",
cells=list(args.get("cells") or _get(prob, "cells") or []), target=target,
options=[FixOption("check_only", "改作小计核对,不另存",
detail="每行小计按上一行小计之后的明细逐列求和核对,小计行不进数据,记进回执的"
"「排除的行」"),
FixOption("keep", f"改作小计核对,原值另存表「{keep}」",
detail="同上,原表的小计另存成一张表;另存的小计表不能按期累积")],
anchor=anchor)
def _o_declare_subtotals(recipe: dict, p: FixProposal, opt: FixOption, reason: str | None) -> tuple[list, list[str]]:
found = find_block(recipe, p.target.get("block"))
if found is None:
raise _Stale
si, bi, block = found
if (block.get("rows") or {}).get("subtotals"):
raise _Stale
keep = _subtotal_keep_name(recipe, block) if opt.value == "keep" else None
value = {"label_column": p.target["label_column"], "pick": p.target["pick"], "scope": "since_previous",
"group_column": None, "keep_as": keep}
ops: list[dict[str, Any]] = [{"op": "add", "path": ptr("sheets", si, "blocks", bi, "rows", "subtotals"),
"value": value}]
if keep:
ops.append({"op": "add", "path": "/tables/-", "value": {
"name": keep, "grain": [], "kind": "reported_total", "units": {}, "note": ""}})
summary = [f"列表「{block['id']}」中间「{p.target['pick']}」开头的行改作小计核对,不进数据"
+ (f",原值另存表「{keep}」" if keep else ",不另存")]
return ops, summary
# ==========================================================================
# ⑥ 忽略(按文字记)
# ==========================================================================
@@ -998,12 +1056,13 @@ _Builder = Callable[[dict, Any, dict, FixAnchor], "FixProposal | None"]
_BUILDERS: dict[str, _Builder] = {
"remove_label": _p_remove_label, "add_label": _p_add_label, "rename_title": _p_rename_title,
"declare_total": _p_declare_total, "ignore_cells": _p_ignore_cells, "declare_placeholder": _p_declare_placeholder,
"declare_hidden": _p_declare_hidden, "rename_sheet": _p_rename_sheet,
"declare_hidden": _p_declare_hidden, "rename_sheet": _p_rename_sheet, "declare_subtotals": _p_declare_subtotals,
}
_OPS: dict[str, Callable[[dict, FixProposal, FixOption, str | None], tuple[list, list[str]]]] = {
"remove_label": _o_remove_label, "add_label": _o_add_label, "edit_members": _o_edit_members,
"rename_title": _o_rename_title, "declare_total": _o_declare_total, "ignore_cells": _o_ignore_cells,
"declare_placeholder": _o_declare_placeholder, "declare_hidden": _o_declare_hidden, "rename_sheet": _o_rename_sheet,
"declare_subtotals": _o_declare_subtotals,
}
@@ -1064,6 +1123,8 @@ def _edit_title(p: FixProposal, opt: FixOption) -> str:
return f"工作表「{t.get('sheet')}」:{opt.label}"
if p.kind == "rename_sheet":
return opt.label
if p.kind == "declare_subtotals":
return f"列表「{t.get('block')}」中间「{t.get('pick')}」开头的行{opt.label}"
return p.title
+19 -12
View File
@@ -177,7 +177,8 @@ class Pipeline:
recipe_sha256: Callable[[Recipe], str]
canonical: Callable[[Recipe], dict[str, Any]]
apply_patch: Callable[[dict[str, Any], list[dict[str, Any]]], dict[str, Any]]
scan: Callable[[bytes], WorkbookScan]
#: (原件, 文件名) → 扫描结果。文件名定按 CSV 读还是按工作簿读(csv_grid.scan_any)
scan: Callable[[bytes, str], WorkbookScan]
scan_to_json: Callable[[WorkbookScan], dict[str, Any]]
scan_from_json: Callable[[dict[str, Any]], WorkbookScan]
read_grid: Callable[..., Grid]
@@ -225,7 +226,7 @@ class Pipeline:
def default_pipeline() -> Pipeline:
"""接上真模块。函数体里惰性导入:这些模块(openpyxl、执行器、起草器)导入不便宜,接口模块加载时不必付。"""
from app.data import (
recipe, recipe_accumulate, recipe_ai, recipe_checks, recipe_compare, recipe_confirm, recipe_diff,
csv_grid, recipe, recipe_accumulate, recipe_ai, recipe_checks, recipe_compare, recipe_confirm, recipe_diff,
recipe_engine, recipe_fixes, recipe_notes, recipe_select, recipe_suggest, xlsx_cells, xlsx_scan,
)
@@ -234,7 +235,8 @@ def default_pipeline() -> Pipeline:
recipe_sha256=recipe.recipe_sha256,
canonical=recipe.canonical_recipe,
apply_patch=recipe.apply_patch,
scan=xlsx_scan.scan,
# CSV / TSV 也走配方:csv_grid 适配成一张「工作表」,其余照旧交给 xlsx_scan(Excel 期 5)
scan=csv_grid.scan_any,
scan_to_json=xlsx_scan.scan_to_json,
scan_from_json=xlsx_scan.scan_from_json,
read_grid=xlsx_cells.read_grid,
@@ -473,7 +475,7 @@ def _open_book(p: Pipeline, raw: bytes, scan: WorkbookScan) -> _Book:
def _book_of(p: Pipeline, snap: _Snap) -> _Book:
raw = _read_raw(snap.raw_sha256)
scan = p.scan_from_json(snap.scan) if snap.scan else p.scan(raw)
scan = p.scan_from_json(snap.scan) if snap.scan else p.scan(raw, snap.file_name)
return _open_book(p, raw, scan)
@@ -771,7 +773,7 @@ class _Staged:
def _stage_work(p: Pipeline, raw: bytes, filename: str) -> _Staged:
"""首次导入(或切换)的暂存:扫描、网格预览、规则起草(带干跑)、对草稿做一次静态校验。"""
scan = p.scan(raw)
scan = p.scan(raw, filename)
book = _open_book(p, raw, scan)
dry = _Capture(_dry_fn(p, book, filename, {}))
d = p.draft(scan, book.grids, filename, validate=lambda data, f, o: p.validate(data, facts=f, origin=o),
@@ -859,7 +861,7 @@ async def stage_upload(session: AsyncSession, *, raw: bytes, filename: str, name
def _replay_work(p: Pipeline, raw: bytes, current: dict[str, Any], filename: str, snap: _Snap,
*, evaluate: bool) -> tuple[_Staged, _Eval | None]:
"""上传新一期 / 修改配方:扫描(或取存下的扫描)、按现行配方的写法取系统发现,修改配方时再评估一遍。"""
scan = p.scan_from_json(snap.scan) if snap.scan else p.scan(raw)
scan = p.scan_from_json(snap.scan) if snap.scan else p.scan(raw, filename)
book = _open_book(p, raw, scan)
facts = p.detect_facts(book.grids, scan, recipe=current)
staged = _Staged(scan_json=p.scan_to_json(scan), previews=[p.preview(g) for g in book.grids.values()],
@@ -1859,7 +1861,7 @@ def _trial_work(p: Pipeline, snap: _Snap, work: dict[str, Any], facts: DraftFact
if parsed is None or problems:
return _TrialRun(parsed=parsed, problems=list(problems))
raw = _read_raw(snap.raw_sha256)
scan = p.scan_from_json(snap.scan) if snap.scan else p.scan(raw)
scan = p.scan_from_json(snap.scan) if snap.scan else p.scan(raw, snap.file_name)
path.parent.mkdir(parents=True, exist_ok=True)
# 先写 .tmp- 再改名:半成品归启动清理(_sweep_tmp)管,发布、回收不会把半个库当成试运行库
tmp = path.parent / f"{path.name}{raw_store.TMP_MARK}{secrets.token_hex(6)}"
@@ -2361,7 +2363,7 @@ def _manifest(*, staging_id: str, kind: str, source_id: str, info: table_version
ex_dict: dict[str, Any], file: dict[str, Any], recipe_meta: dict[str, Any], checks: list[dict[str, Any]],
overrides: list[dict[str, Any]], waivers: list[dict[str, Any]], confirmed: list[dict[str, Any]],
notes: SchemaNotes, ai: dict[str, Any], signed_by: str | None,
edits: list[dict[str, Any]] | None = None) -> dict[str, Any]:
edits: list[dict[str, Any]] | None = None, signer_verified: bool = False) -> dict[str, Any]:
receipt = copy.deepcopy(ex_dict)
receipt.pop("problems", None)
return {
@@ -2383,7 +2385,8 @@ def _manifest(*, staging_id: str, kind: str, source_id: str, info: table_version
"ai": ai,
# 期 3:修复、框选的修改记录(含已被整份替换覆盖的,作为历史;不带撤销用的修改前起点)
"edits": copy.deepcopy(list(edits or [])),
"signed_by": {"name": signed_by, "verified": False},
# 是否认证照实记(05-已知遗留 (c)):开了登录、署名是登录的人时为真。以前写死 False,封存的证据记错了事实
"signed_by": {"name": signed_by, "verified": bool(signer_verified and signed_by)},
"staging_id": staging_id, "kind": kind,
"created_at": _now_iso(),
}
@@ -2411,7 +2414,8 @@ def _file_sha(path: Path) -> str | None:
async def commit_staging(session: AsyncSession, staging: ImportStaging, *, trial_id: str, confirmations: list[str],
acceptances: list[Acceptance], signed_by: str | None) -> CommitResult:
acceptances: list[Acceptance], signed_by: str | None,
signer_verified: bool = False) -> CommitResult:
"""POST imports/{id}/commit(7.8):预检 → (与现行导入相同时)不新建记录 → 发布构建、写配方、导入记录、
快照、清单,切指针。失败时指针不动,暂存区退回 drafting(试运行库已被发布流程消耗)。
@@ -2532,6 +2536,8 @@ async def commit_staging(session: AsyncSession, staging: ImportStaging, *, trial
ex = extraction_from_dict(ex_dict)
who = signed_by or staging.signed_by
# 署名是这次请求的(signed_by)才谈得上认证;退回暂存区上记的那个,是否认证这里不知道,照实记未认证
who_verified = bool(signer_verified and signed_by)
acc = [Acceptance(check_id=a.check_id, reason=a.reason.strip(), signed_by=who) for a in given.values()]
# 这一期自己的说明(带实际的接受)进它的导入清单:版本页逐期显示「改写后的那一句」从这里取(第 5 节)。
# 快照冻结的说明在多期时另按各期最弱的结论合并(2.8),只剩本期时就是它
@@ -2603,7 +2609,8 @@ async def commit_staging(session: AsyncSession, staging: ImportStaging, *, trial
file={"name": file_name, "size": file_size, "raw_sha256": raw_sha},
recipe_meta={"id": recipe_id, "sha256": recipe_sha, "origin": origin, "canonical": canonical},
checks=checks_full, overrides=overrides, waivers=waivers, confirmed=confirmed,
notes=notes if union_info is None else period_notes, ai=ai, signed_by=who, edits=edits)
notes=notes if union_info is None else period_notes, ai=ai, signed_by=who, edits=edits,
signer_verified=who_verified)
async def schema_cache_for(view: table_versions.SourceView, info: table_versions.PublishInfo) -> dict[str, Any]:
from app.core import artifact_store
@@ -2663,7 +2670,7 @@ async def commit_staging(session: AsyncSession, staging: ImportStaging, *, trial
"table_hashes": dict(union_report.get("table_hashes") or {}), "union_ver": recipe_types.UNION_VER},
parts=ordered, report=union_report, label_sets=list((plan or {}).get("label_sets") or []),
gaps=list((plan or {}).get("gaps") or []), replaced=(plan or {}).get("replaces"), removed=None,
notes=notes, signed_by=who)
notes=notes, signed_by=who, signer_verified=who_verified)
smid = await artifact_store.put_json(doc, kind="snapshot_manifest",
meta={"source_id": source_id, "snapshot_id": info.snapshot_id})
cache["import_manifests"] = [mid if iid is None else by_imp[iid].info.manifest_artifact
+66 -9
View File
@@ -27,6 +27,9 @@ from app.data.names import canon
from app.data.recipe_checks import date_columns, plan_checks, split_coord
from app.data.recipe_parsers import UNITS
from app.data.recipe_types import (
LIST_PERIOD_END,
LIST_PERIOD_START,
SUBTOTAL_KEY,
Acceptance,
CheckResult,
ColumnOut,
@@ -46,6 +49,7 @@ from app.data.recipe_types import (
TableNote,
UnionNotePart,
derive_tables,
list_period_columns,
prose_problems,
render_prose,
)
@@ -134,6 +138,13 @@ NOTE_TEXT = {
"ignored_rows_union": "部分期的原表另有按配方忽略的行,内容见证据面板",
"ignored_cells_union": "部分期的原表另有按配方忽略的单元格,内容见证据面板",
"dim_values_vary": "各期的 {dim} 取值不完全相同:跨期比较前请先确认两期都有的取值",
# ---- 期 5:列表按期累积、穿插的小计。列表可以累积以后,空行之后的文字行也要有并集说法
"rows_after_stop_union": "部分期的原表在空行之后另有未导入的文字行,内容见证据面板",
"subtotal_kept": "原表穿插的小计在 {table},不要与本表相加",
"subtotal_not_kept": "原表穿插的小计行未导入本表",
"list_subtotal_passed": "原表穿插的小计行的值,已按各组明细求和核对一致:不要与 {base} 相加,也不要彼此相加",
"list_subtotal_unverified": "原表穿插的小计行的值,按原表保存的值存储。本期部分小计未能核对:不要与 {base} 相加,"
"也不要彼此相加",
}
#: 列说明的片段模板
@@ -148,6 +159,9 @@ COLUMN_TEXT = {
"col_partial_periods": "部分期没有这一列的数据,为空值",
"col_absent": "当前版本的数据中没有这一列,为空值",
"col_retired": "这一列自某一期起不再导入,此后各期为空值",
# ---- 期 5:列表按期累积时由系统加的统计期起止两列(ListPeriod.synthesize)。不含数字(说明不许报数)
"period_start": "本期统计期的第一天,取自表内的统计期或人工录入的统计期,格式 YYYY-MM-DD",
"period_end": "本期统计期的最后一天,取自表内的统计期或人工录入的统计期,格式 YYYY-MM-DD",
}
#: 「逐日 / 逐行」「个别日期 / 个别行」:粒度只有一列日期时按日说,否则按行说
@@ -161,7 +175,7 @@ _K_REASON = {r: f"total_k_{r}" for r in ("uncached", "blank", "null_detail", "ti
_T_REASON = {r: f"list_total_{r}" for r in ("uncached", "blank", "null_detail", "tiling", "hidden")}
#: 允许出现「一致」的模板:只在对应的 K / T 通过时才会选用
CONSISTENT_KEYS = frozenset({"total_k_passed", "total_k_passed_g_unverifiable", "list_total_passed",
"list_total_passed_partial"})
"list_total_passed_partial", "list_subtotal_passed"})
def _tok(kind: str, name: str) -> str:
@@ -202,6 +216,8 @@ class _Writers:
lists: list[ListBlock] = field(default_factory=list)
#: 把合计行另存进本表的列表块
list_totals: list[ListBlock] = field(default_factory=list)
#: 期 5:把穿插的小计另存进本表的列表块
list_subtotals: list[ListBlock] = field(default_factory=list)
def add_sheet(self, sheet: SheetRecipe) -> None:
if all(s is not sheet for s in self.sheets):
@@ -242,6 +258,9 @@ def _writers(recipe: Recipe) -> dict[str, _Writers]:
total = block.rows.total_row
if total is not None and total.keep_as:
w(total.keep_as, sheet).list_totals.append(block)
sub = block.rows.subtotals
if sub is not None and sub.keep_as:
w(sub.keep_as, sheet).list_subtotals.append(block)
return out
@@ -395,6 +414,15 @@ def _base_total_fragments(w: _Writers) -> list[NoteFragment]:
b.rows.total_row is not None and not b.rows.total_row.keep_as for b in w.lists)
if dropped:
out.append(NoteFragment("total_not_kept", None, NOTE_TEXT["total_not_kept"]))
# 期 5:穿插的小计行不在本表里:另存在哪,或者没导入
for b in w.lists:
sub = b.rows.subtotals
if sub is None:
continue
if sub.keep_as:
_add_unique(out, _frag("subtotal_kept", sub.keep_as, NOTE_TEXT["subtotal_kept"].format(table=_tab(sub.keep_as))))
elif all(f.key != "subtotal_not_kept" for f in out):
out.append(NoteFragment("subtotal_not_kept", None, NOTE_TEXT["subtotal_not_kept"]))
return out
@@ -450,6 +478,13 @@ def _kept_total_fragments(ctx: _Ctx, table: str, w: _Writers) -> list[NoteFragme
else:
frag = _frag(key, table, NOTE_TEXT[key].format(base=_tab(block.table)))
_add_unique(out, frag)
for block in w.list_subtotals:
# 期 5:另存的小计:T 通过才说「核对一致」,没核对上的(无法核对)照实说部分未能核对;不一致的整份拒收
t = ctx.check(table, ctx.plan.t.get(f"{block.id}{SUBTOTAL_KEY}"), f"块「{block.id}」的小计行")
if t is None or _blocking(t):
continue
key = "list_subtotal_passed" if t.status == "passed" else "list_subtotal_unverified"
_add_unique(out, _frag(key, table, NOTE_TEXT[key].format(base=_tab(block.table))))
return out
@@ -653,10 +688,17 @@ def _date_key(ctx: _Ctx, w: _Writers, table: str, col: ColumnOut) -> str | None:
return None
def _period_column_key(col: ColumnOut) -> str | None:
"""期 5:系统加的统计期起止两列(role=period)用哪条列说明。"""
if col.role != "period":
return None
return "period_start" if col.name == LIST_PERIOD_START else "period_end" if col.name == LIST_PERIOD_END else None
def _null_eligible(ctx: _Ctx, w: _Writers, table: str, col: ColumnOut) -> bool:
"""这一列要不要考虑空值说明:值列;主键不会是空值;表内合计表的空值已由表说明(未能核对)交代。"""
value_col = col.role in ("measure", "value") and col.type in ("INTEGER", "REAL")
is_total = bool(w.kept_from or w.list_totals)
is_total = bool(w.kept_from or w.list_totals or w.list_subtotals)
return value_col and not is_total and col.name not in set(ctx.grains.get(table, []))
@@ -668,6 +710,8 @@ def _column_templates(ctx: _Ctx, table: str, null_counts: Mapping[str, Mapping[s
frags: list[str] = []
if (date := _date_key(ctx, w, table, col)) is not None:
frags.append(COLUMN_TEXT[date])
if (period := _period_column_key(col)) is not None:
frags.append(COLUMN_TEXT[period])
if col.unit:
frags.append(COLUMN_TEXT["unit"].format(unit=_tok("单位", col.unit)))
has_nulls = counts is None or counts.get(col.name, 0) > 0
@@ -793,22 +837,27 @@ _FAMILY: dict[str, str] = {
"hidden_excluded": "hidden_excluded", "hidden_included": "hidden_included",
"ignored_columns": "ignored_columns", "ignored_rows": "ignored_rows", "ignored_cells": "ignored_cells",
"rows_after_stop": "rows_after_stop",
"subtotal_kept": "subtotal_kept", "subtotal_not_kept": "subtotal_not_kept",
"list_subtotal_passed": "list_total", "list_subtotal_unverified": "list_total",
"period_conflict_accepted": "period_conflict", "period_conflict_pending": "period_conflict",
"period_human": "period_human", "outside_digits": "outside_digits",
"note": "note",
}
#: 合并后的拼接顺序(与 _table_fragments 一致;累积特有的几句排在用户写的表说明之前)
_FAMILY_ORDER = ("grain", "sum_eq", "not_comparable", "hour_range", "const", "total_kept", "total_not_kept", "total_k",
"list_total", "total_overlap", "hidden_excluded", "hidden_included", "ignored_columns",
_FAMILY_ORDER = ("grain", "sum_eq", "not_comparable", "hour_range", "const", "total_kept", "total_not_kept",
"subtotal_kept", "subtotal_not_kept", "total_k", "list_total", "total_overlap", "hidden_excluded", "hidden_included", "ignored_columns",
"ignored_rows", "ignored_cells", "rows_after_stop", "period_conflict", "period_human",
"outside_digits", "union", "note")
#: 与核对结论无关、只描述表结构的片段族:用目标配方那一期的;只在早期出现的 subject(退役的表、另存表)用
#: 最后一个含它的那一期的
_STRUCTURAL = frozenset({"grain", "hour_range", "const", "total_kept", "total_not_kept", "total_overlap", "note"})
_STRUCTURAL = frozenset({"grain", "hour_range", "const", "total_kept", "total_not_kept", "total_overlap", "note",
"subtotal_kept", "subtotal_not_kept"})
#: 「任一期出现就写」的片段族 → 并集里的说法
_ANY_UNION = {"hidden_excluded": "hidden_excluded_union", "hidden_included": "hidden_included_union",
"ignored_columns": "ignored_columns_union", "ignored_rows": "ignored_rows_union",
"ignored_cells": "ignored_cells_union", "outside_digits": "outside_digits_union"}
"ignored_cells": "ignored_cells_union", "outside_digits": "outside_digits_union",
# 期 5:列表可以按期累积了,空行之后的文字行任一期有就写
"rows_after_stop": "rows_after_stop_union"}
_K_PASSED = frozenset({"total_k_passed", "total_k_passed_g_unverifiable"})
@@ -943,20 +992,26 @@ def _merge_group(table: str, family: str, subject: str | None, group: dict[int,
if family == "period_human":
key = "period_human_all" if len(present) == len(having) else "period_human_union"
return NoteFragment(key, subject, NOTE_TEXT[key])
# list_total、rows_after_stop 只有列表才有,而列表不能按期累积(P3-SPEC 2.2);不认识的片段族同样没有并集说法。
# list_total 只在列表的合计行、小计行另存的表上,这种表不能按期累积(accumulate_blockers);不认识的片段族同样
# 没有并集说法。
# 记 problems(不能保存),绝不沿用某一期的单期说法。消息给人看,不露片段族的键名
problems.append(f"表「{table}」:{_UNMERGEABLE.get(family, '有一段说明')}不能按期合并,无法生成说明")
return None
#: 没有并集说法的片段族 → 给人看的叫法(problems 里不露键名)
_UNMERGEABLE = {"list_total": "列表合计行的核对结论", "rows_after_stop": "列表空行之后的文字行"}
_UNMERGEABLE = {"list_total": "列表合计行、小计行的核对结论"}
def _axis_column(recipe: Recipe, table: str) -> str | None:
"""各期按哪一列区分(说明里「请按 {col} 筛选」):交叉表的日期轴;期 5 起列表按 period 绑定的日期列,或者
系统加的统计期起那一列。"""
tables, _ = derive_tables(recipe)
axes = [c.name for c in tables.get(table, []) if c.role == "axis"]
return axes[0] if len(axes) == 1 else None
if len(axes) == 1:
return axes[0]
spec = list_period_columns(recipe).get(table)
return spec[1] if spec else None
def _union_table_fragments(table: str, having: list[int], parts: list[_UPart], ref: int, target: Recipe,
@@ -1037,6 +1092,8 @@ def _union_column_templates(table: str, columns: list[ColumnOut], having: list[i
frags.append(COLUMN_TEXT["date_year_from_period"])
elif "date" in dates or (not with_col and col.role == "axis"):
frags.append(COLUMN_TEXT["date"])
if (period := _period_column_key(col)) is not None:
frags.append(COLUMN_TEXT[period])
if target_cols is not None and col.name in target_cols:
unit = target_cols[col.name].unit
elif with_col:
+15
View File
@@ -427,6 +427,21 @@ def _bottom_box(ctx: _Ctx, plan: Any, first: int) -> bool:
ctx.fail("selection_bottom_not_anchorable",
f"框里只有表头和合计行(第 {last} 行),没有数据行:请把框扩大到数据所在的行")
return False
# 期 5:框里夹在中间、第一列以合计字样开头、下面紧接着还有同一张表的明细的行(起草器 _scan_block 的同一条
# 规则),照起草器提议成小计(_settle_subtotals);提议不成的照旧报错
mid = [r for r in data[1:] if S._total_word(S._label_of(grid.get(r, c1))) is not None]
rest = [r for r in data if r not in mid]
if mid and rest and not any(grid.get(r, c) is not None and S._total_word(grid.get(r, c).value) is not None
for r in data for c in range(c1 + 1, c2 + 1)):
trial = S._ListPlan(header_rows=plan.header_rows, cols=plan.cols, headers=plan.headers, rows=rest,
subtotal_rows=list(mid),
subtotal_words=[S._total_word(S._label_of(grid.get(r, c1))) or "" for r in mid])
types = S._types_of(grid, rest, plan.cols)
if all(S._more_rows(grid, r, trial, S._bottom(grid), types) for r in mid):
S._settle_subtotals(grid, trial)
if trial.subtotal_word is not None:
plan.subtotal_rows, plan.subtotal_word = trial.subtotal_rows, trial.subtotal_word
data = [r for r in data if r not in mid]
if not _no_stray_totals(ctx, data, c1, c2):
return False
bottom = S._bottom(grid)
+189 -14
View File
@@ -12,6 +12,11 @@ validate)和执行器干跑(注入的 dry_run),再由人逐条确认之
哪些是区域外文字、哪些格没有去处)直接用执行器的结论:同一件事只有执行器一套判法,这里自己定位只是没有
干跑时的近似。
**期 5(下一轮 03 细案 E)。** 有 Excel 表格对象(插入 → 表格)的工作表优先按表格对象出列表块(_lists_from_tables:
表头、列范围、汇总行都是表格对象给的,锚点仍按表头文字);夹在明细中间、第一列以同一个合计字样开头的行提议成
穿插的小计(_settle_subtotals),不再一律记起草失败;每个列表都问一道「每期怎么更新」(q_list_period),选按期
累积时同时绑定每一行属于哪一期(按日期列核对,或加统计期起止两列)。
**问题的 effects 怎么写。** 每个选项是一组 JSON Patch,路径按传进来的配方写;接口层从 answers_base 起、按
问题顺序重放全部回答(P2-SPEC 7.2)。因此:
- 关系的两个选项都用整条 replace(「登记」写回完整的 sum_eq / not_comparable,「不登记」写 dismissed),
@@ -63,7 +68,7 @@ from app.data.recipe_types import (
derive_tables,
)
from app.data.tabular import DATE_HEADER_MIN, EXCEL_ERRORS
from app.data.xlsx_scan import WorkbookScan, cell_ref, col_letter, parse_ref
from app.data.xlsx_scan import TableScan, WorkbookScan, cell_ref, col_letter, parse_ref
logger = logging.getLogger(__name__)
@@ -612,8 +617,15 @@ class _ListPlan:
#: 按数据行收进来并记起草失败
adjacent: list[tuple[int, list[int]]] = field(default_factory=list)
#: 不在末尾第一列的合计字样:(行, 列)。第一列在中间或最上面、或者落在别的列上,都无法按合计行核对,
#: 记起草失败;不记就会把合计当明细导入,求和翻倍(H2)
#: 记起草失败;不记就会把合计当明细导入,求和翻倍(H2)。期 5:第一列里同一个字样的穿插行先记进 subtotal_rows,
#: 收完整块再判能不能提议成小计(_settle_subtotals);判不了的才回到这里
stray_totals: list[tuple[int, int]] = field(default_factory=list)
#: 期 5:穿插在明细中间、第一列以合计字样开头的行(不进 rows)和它们的字样;能提议成小计时 subtotal_word 非空
subtotal_rows: list[int] = field(default_factory=list)
subtotal_words: list[str] = field(default_factory=list)
subtotal_word: str | None = None
#: 期 5:按 Excel 的表格对象起草的(xlsx_scan.TableScan 的名字):表名优先用它,卡片里写明
table_object: str | None = None
# 命名阶段填
id: str = ""
table: str = ""
@@ -808,13 +820,44 @@ def _scan_block(grid: Grid, plan: _ListPlan, bottom: int) -> None:
if word is not None and plan.rows and not _more_rows(grid, r, plan, bottom, types):
plan.total_row, plan.total_word = r, word
break
others = [(r, c) for c, cell in zip(cols[1:], cells[1:])
if cell is not None and _total_word(cell.value) is not None]
plan.stray_totals.extend(others)
if word is not None and plan.rows and not others:
# 期 5:夹在明细中间的合计字样,先当小计候选:不进数据行,收完整块再判(_settle_subtotals)
plan.subtotal_rows.append(r)
plan.subtotal_words.append(word)
r += 1
continue
if word is not None:
plan.stray_totals.append((r, cols[0]))
plan.stray_totals.extend((r, c) for c, cell in zip(cols[1:], cells[1:])
if cell is not None and _total_word(cell.value) is not None)
plan.rows.append(r)
acc.add(r)
r += 1
_settle_subtotals(grid, plan)
def _settle_subtotals(grid: Grid, plan: _ListPlan) -> None:
"""期 5:夹在明细中间的合计字样能不能提议成小计(ListRows.subtotals,规则起草不再把它们记成失败)。
能的条件:都在第一列、是同一个字样,每一行至少有一格数(没有数的小计核对不了)。末尾那一行和它们字样相同时
(表以最后一组的小计结束,没有总计),它也是小计,不是合计行。不能的(字样不同、有一行没有数)照旧记进
stray_totals(起草失败,交给人或 AI),这些行回到数据行里——和期 4 的结果一样。"""
if not plan.subtotal_rows:
return
words = set(plan.subtotal_words)
numeric = [c for c in plan.cols[1:] if any(_value_num(grid.get(r, c)) is not None for r in plan.rows)]
ok = len(words) == 1 and all(any(_value_num(grid.get(r, c)) is not None for c in numeric)
for r in plan.subtotal_rows)
if ok:
plan.subtotal_word = next(iter(words))
if plan.total_row is not None and plan.total_word == plan.subtotal_word:
plan.subtotal_rows.append(plan.total_row)
plan.total_row = plan.total_word = None
return
plan.stray_totals.extend((r, plan.cols[0]) for r in plan.subtotal_rows)
plan.rows = sorted(set(plan.rows) | set(plan.subtotal_rows))
plan.subtotal_rows, plan.subtotal_words = [], []
def _list_failures(sheet: str, grid: Grid, plan: _ListPlan) -> list[_Failure]:
@@ -898,7 +941,8 @@ def _analyze_lists(grid: Grid) -> _Lists:
_finish_block(grid, plan)
out.failures.extend(_list_failures(grid.sheet, grid, plan))
out.blocks.append(plan)
end = max((p.total_row or (p.rows[-1] if p.rows else p.header_rows[-1])) for p in made)
end = max(max([p.total_row or 0, *p.subtotal_rows, p.rows[-1] if p.rows else p.header_rows[-1]])
for p in made)
while r_idx < len(rows) and rows[r_idx] <= end:
r_idx += 1
return out
@@ -945,7 +989,7 @@ def _finish_block(grid: Grid, plan: _ListPlan) -> None:
break
if cells:
break
last = plan.total_row or rows[-1]
last = max([plan.total_row or 0, rows[-1], *plan.subtotal_rows])
for r in range(plan.header_rows[0], last + 1):
for c in cols:
if grid.get(r, c) is not None or r in plan.header_rows:
@@ -975,15 +1019,77 @@ class _Sheet:
hidden_cols: list[int] = field(default_factory=list)
def _analyze_sheet(grid: Grid, sid: str) -> _Sheet:
def _lists_from_tables(grid: Grid, objects: list[TableScan]) -> _Lists | None:
"""期 5:按 Excel 的表格对象(插入 → 表格)起草列表块。表格对象明确给出了表头行、列范围、数据行和汇总行,比按
「像不像表头」猜可靠:数字格空着的行不会被当成表头,紧挨着的两张表也分得开。
出来的块和按规则猜的一样只记文字:锚点仍按表头文字(不记坐标,下一期表格挪了位置照样认得出),汇总行按第一列
的文字映射成合计行(total_row),夹在中间的合计字样按小计提议(_settle_subtotals)。表格对象的 ref 写坏了、
没有表头行时返回 None,退回按规则猜。"""
out = _Lists(sheet=grid.sheet)
spans = sorted(((t.span(), t) for t in objects), key=lambda x: (x[0] or (0, 0, 0, 0))[:2])
for span, t in spans:
if span is None or t.header_row_count < 1:
return None
r1, c1, r2, c2 = span
hrows = list(range(r1, r1 + t.header_row_count))
cols = list(range(c1, c2 + 1))
texts = _header_texts(grid, hrows, cols)
if set(texts) != set(cols):
blank = [col_letter(c) for c in cols if c not in texts]
msg = (f"工作表「{grid.sheet}」:表格「{t.name}」的表头有空着的列({'、'.join(blank)} 列),"
"按列表导入要每一列都有表头文字")
out.failures.append(_Failure(msg, msg))
continue
plan = _ListPlan(header_rows=hrows, cols=cols, headers={c: texts[c] for c in cols}, table_object=t.name)
last_data = r2 - max(t.totals_row_count, 0)
rows = [r for r in grid.row_numbers() if hrows[-1] < r <= last_data and any(grid.get(r, c) is not None
for c in cols)]
prev = hrows[-1]
for r in rows:
if r > prev + 1:
plan.skip_blank = True
plan.skipped.extend(range(prev + 1, r))
prev = r
word = _total_word(_label_of(grid.get(r, c1)))
if word is not None and plan.rows:
plan.subtotal_rows.append(r)
plan.subtotal_words.append(word)
continue
plan.rows.append(r)
if not plan.rows:
msg = f"工作表「{grid.sheet}」:表格「{t.name}」的表头下面没有数据行"
out.failures.append(_Failure(msg, msg))
continue
if t.totals_row_count > 0:
tr = last_data + 1
label = _label_of(grid.get(tr, c1))
word = _total_word(label) or (label if label and len(label) <= 12 and not any(
ch.isdigit() for ch in label) else None)
if word is not None:
plan.total_row, plan.total_word = tr, word
else:
why = ("起草只读了表格前面的部分,没读到汇总行" if grid.truncated and tr > max(grid.row_numbers())
else "汇总行的第一列没有可用作合计标签的文字(不能含数字、不超过 12 个字)")
msg = f"工作表「{grid.sheet}」:表格「{t.name}」的汇总行(第 {tr} 行)无法按合计行核对:{why}"
out.failures.append(_Failure(msg, msg))
_settle_subtotals(grid, plan)
_finish_block(grid, plan)
out.failures.extend(_list_failures(grid.sheet, grid, plan))
out.blocks.append(plan)
return out
def _analyze_sheet(grid: Grid, sid: str, objects: list[TableScan] | None = None) -> _Sheet:
sh = _Sheet(sheet=grid.sheet, sid=sid)
axis = _axis_row(grid)
from_objects = _lists_from_tables(grid, objects) if objects and axis is None else None
if axis is not None:
sh.cross = _analyze_crosstab(grid, *axis)
sh.region = set(sh.cross.region)
sh.failures.extend(sh.cross.failures)
else:
sh.lists = _analyze_lists(grid)
sh.lists = from_objects if from_objects is not None else _analyze_lists(grid)
sh.failures.extend(sh.lists.failures)
for b in sh.lists.blocks:
sh.region |= b.region
@@ -1111,7 +1217,8 @@ def _measure_column(label: str, pos: int) -> tuple[str, str | None, str | None]:
def _analyze(scan: WorkbookScan, grids: dict[str, Grid]) -> _Analysis:
visible = _visible_grids(scan, grids)
hidden = [s.name for s in scan.sheets if s.state != "visible"]
sheets = [_analyze_sheet(g, f"s{i + 1}") for i, g in enumerate(visible)]
sheets = [_analyze_sheet(g, f"s{i + 1}", [t for t in scan.tables if t.sheet == g.sheet])
for i, g in enumerate(visible)]
failures = [f for sh in sheets for f in sh.failures]
tables_used: set[str] = set()
@@ -1266,7 +1373,9 @@ def list_block(sheet: str, b: _ListPlan, *, block_id: str, tables_used: set[str]
合计表名撞了已用的名字时照起草器的老行为:不另建表定义,keep_as 仍写这个名字(交给静态校验报)。"""
b.id = block_id
if table is None:
title = b.title if b.title and not any(ch.isdigit() for ch in b.title) else None
# 期 5:按表格对象起草的,表名优先用表格对象的名字(不含数字时;Excel 默认的「表1」不用)
named = b.table_object if b.table_object and not any(ch.isdigit() for ch in b.table_object) else None
title = named or (b.title if b.title and not any(ch.isdigit() for ch in b.title) else None)
want = _ENUM_PREFIX.sub("", P.split_unit_suffix(title)[0]) if title else sheet
n_fallback = len(tables_used) + 1
b.table = _unique(to_sql_name(want, fallback=f"表{n_fallback}") if want else f"表{n_fallback}", tables_used)
@@ -1293,6 +1402,11 @@ def list_block(sheet: str, b: _ListPlan, *, block_id: str, tables_used: set[str]
total_spec = {"name": keep, "grain": ["合计项"], "kind": "reported_total", "units": num_units, "note": ""}
b.keep_table = keep
total = {"label_column": b.names[b.cols[0]], "pick": b.total_word, "keep_as": keep}
subtotals = None
if b.subtotal_word is not None:
# 期 5:穿插的小计只核对、不另存(另存的小计表不能按期累积,要另存由人在配方面板里打开)
subtotals = {"label_column": b.names[b.cols[0]], "pick": b.subtotal_word, "scope": "since_previous",
"group_column": None, "keep_as": None}
grain = [b.names[b.grain_col]] if b.grain_col is not None else []
main = {"name": b.table, "grain": grain, "kind": "data", "units": dict(b.units), "note": ""}
fill = bool(b.merged_fill) and all(r2 > r1 and c1 == c2 and b.types.get(c1) == "TEXT"
@@ -1300,7 +1414,8 @@ def list_block(sheet: str, b: _ListPlan, *, block_id: str, tables_used: set[str]
block = {
"id": b.id, "layout": "list", "table": b.table, "header_rows": len(b.header_rows),
"after_title": None, "columns": cols_out, "extra_columns": "reject",
"rows": {"blank_rows": "skip" if b.skip_blank else "stop", "total_row": total},
"rows": {"blank_rows": "skip" if b.skip_blank else "stop", "total_row": total,
**({"subtotals": subtotals} if subtotals is not None else {})},
"values": {"placeholders": [{"text": e[0], "meaning": "无数据"} for e in b.placeholders.values()],
"blank": "null", "text_number": "reject", "formula": "accept_cached"},
"merged_data": "fill" if fill else "reject",
@@ -1797,6 +1912,8 @@ class _ListLoc:
rows: list[int]
total_row: int | None
region: set[tuple[int, int]] = field(default_factory=set)
#: 期 5:穿插的小计行
subtotal_rows: list[int] = field(default_factory=list)
def _find_grid(name: str, grids: dict[str, Grid]) -> Grid | None:
@@ -1943,19 +2060,26 @@ def _locate_list(block: ListBlock, grid: Grid, taken: set[int]) -> _ListLoc | No
bottom = _bottom(grid)
rr = r + 1
pick = P.match_key(block.rows.total_row.pick) if block.rows.total_row else None
sub_pick = P.match_key(block.rows.subtotals.pick) if block.rows.subtotals else None
subs: list[int] = []
while rr <= bottom:
if all(grid.get(rr, c) is None for c in cols):
if block.rows.blank_rows == "skip":
rr += 1
continue
break
if pick and P.match_key(_label_of(grid.get(rr, cols[0])) or "").startswith(pick):
label = P.match_key(_label_of(grid.get(rr, cols[0])) or "")
if pick and label.startswith(pick):
total = rr
break
if sub_pick and label.startswith(sub_pick):
subs.append(rr)
rr += 1
continue
rows.append(rr)
rr += 1
loc = _ListLoc(hrows, cols, rows, total)
for row in [*hrows, *rows, *([total] if total else [])]:
loc = _ListLoc(hrows, cols, rows, total, subtotal_rows=subs)
for row in [*hrows, *rows, *subs, *([total] if total else [])]:
loc.region.update((row, c) for c in cols)
return loc
@@ -2130,6 +2254,9 @@ def questions_for(recipe: dict[str, Any], facts: DraftFacts, grids: dict[str, Gr
if ext is not None and ext.sheets.skipped_hidden:
for nm in _dedupe([item.get("sheet", "") for item in ext.sheets.skipped_hidden]):
cards.append(_hidden_sheet_card(nm))
# 期 5:列表每期怎么更新的问题排在其余问题之后、q_mode 之前:它只和按期累积有关,不该顶掉期 2 起排在前面的问题
questions = ([q for q in questions if not q.id.startswith("q_list_period:")]
+ [q for q in questions if q.id.startswith("q_list_period:")])
blockers = accumulate_blockers(parsed)
cards.append(_mode_card(blockers))
if not blockers:
@@ -2543,9 +2670,57 @@ def _list_cards(si: int, bi: int, sheet: Any, block: ListBlock, loc: _ListLoc |
reason += f",原值另存表「{total.keep_as}」" if total.keep_as else ",只核对不另存"
cells = _coords(name, [(loc.total_row, loc.cols[0])]) if loc is not None and loc.total_row else []
cards.append(Card(f"total_row:{block.id}", f"{where}是合计行,改作核对", reason, cells))
subt = block.rows.subtotals
if subt is not None:
rows = loc.subtotal_rows if loc is not None else []
where = _rows_text(rows) if rows else f"「{subt.pick}」开头的行"
scope = ("上一行小计之后的全部明细" if subt.scope == "since_previous"
else f"紧挨在它上面、「{subt.group_column}」相同的那几行明细")
reason = f"每行小计按列求和与{scope}核对,小计行不进数据"
reason += f",原值另存表「{subt.keep_as}」" if subt.keep_as else ",只核对不另存"
cells = _coords(name, [(r, loc.cols[0]) for r in rows]) if loc is not None else []
cards.append(Card(f"subtotals:{block.id}", f"{where}是穿插的小计行,改作核对", reason, cells))
q = _list_period_question(si, bi, sheet, block, recipe)
questions.append(q)
cards.append(Card(f"list_period:{block.id}", "这份明细每期怎么更新",
"每期替换:当前版本只含最新一期。按期累积:每期一份明细加进当前版本、可以跨期查询,要说明每一行"
"属于哪一期——选一个日期列,核对每行都在本期统计期内;明细里没有能代表统计期的日期列时,"
"选「每行加上本期统计期的起止两列」", [], q.id))
return cards, questions
def _list_period_question(si: int, bi: int, sheet: Any, block: ListBlock, recipe: dict[str, Any]) -> Question:
"""期 5:列表每期怎么更新、按期累积时每一行属于哪一期(ListBlock.period),一道题定:每期替换(不绑定);按期
累积并按某个日期列核对(每个日期列一个选项);按期累积并给每行加统计期起止两列。后两种同时把导入模式改成按期
累积,所在工作表没配统计期的一并加上统计期(回执里统计期取自表内,没有就请人录入)。
为什么不分成「先绑定、再问 q_mode」两道题:问题集合按起草的草稿生成、回答时不变(recipe_imports.answer),
列表的草稿还没绑定统计期,不符合按期累积的资格,q_mode 根本不会出现。几个列表各答各的:有的选累积、有的选替换
时,静态校验照实报出没绑定的那个列表不能累积。"""
base = ["sheets", si, "blocks", bi, "period"]
context = [] if sheet.context else _add_ops(recipe, ["sheets", si, "context"], [
{"id": "统计期", "kind": "period", "parser": "cn_date_range", "prefer_prefix": None, "cross_check": "filename"}])
mode = _add_ops(recipe, ["mode"], "accumulate")
# 「每期替换」排第一个:它就是草稿本来的样子(期 4 起各处取第一个选项的地方都不会因此改了配方)
options: list[dict[str, Any]] = [{"value": "none", "label": "每期替换", "needs_reason": False}]
effects: dict[str, list[dict[str, Any]]] = {"none": _add_ops(recipe, base, None)}
for c in block.columns:
if c.type != "DATE":
continue
value = f"column:{c.name}"
options.append({"value": value, "label": f"按期累积,按「{c.name}」核对每行都在本期统计期内",
"needs_reason": False})
effects[value] = context + mode + _add_ops(recipe, base, {"column": c.name, "check": "within_context",
"synthesize": None})
options.append({"value": "synthesize", "label": "按期累积,每行加上本期统计期的起止两列", "needs_reason": False})
effects["synthesize"] = context + mode + _add_ops(recipe, base, {"column": None, "check": "within_context",
"synthesize": "统计期"})
current = block.period
default = ("synthesize" if current is not None and current.synthesize else
f"column:{current.column}" if current is not None and current.column else "none")
return Question(f"q_list_period:{block.id}", f"列表「{block.id}」每期怎么更新", options, default, effects)
# ==========================================================================
# 期 3:按规则重新起草,并把名字对齐到现行配方(P3-SPEC 6.3)
# ==========================================================================
+115 -18
View File
@@ -300,11 +300,32 @@ class TotalRow(_M):
keep_as: str | None = None
class Subtotals(_M):
"""穿插在明细中间的小计行:不作数据,按它所管的那一组明细逐列求和核对;keep_as 给了就另存一张表。"""
#: 小计标签所在的列(列名)
label_column: str
#: 小计行的标签以这个词开头(match_key 比较)。取自实际格子文字的候选词,不能含数字
pick: str = Field(min_length=1, max_length=12)
#: 一行小计管哪几行明细:since_previous 上一行小计(或表头)之后的全部明细;group_column 紧挨在它上面、
#: group_column 这一列取值相同的那几行明细
scope: Literal["since_previous", "group_column"] = "since_previous"
#: scope=group_column 时按哪一列分组(列名);since_previous 时为 None
group_column: str | None = None
#: 原表的小计另存成哪张表(kind=reported_total);None 只核对不另存
keep_as: str | None = None
class ListRows(_M):
#: 数据中间的空行:stop 到此为止(默认);skip 跳过继续(回执记个数)
blank_rows: Literal["stop", "skip"] = "stop"
#: 合计行:到合计行为止,合计行按列求和核对
total_row: TotalRow | None = None
#: 期 5:穿插在明细中间的小计行(Subtotals)。默认 None = 期 4 的行为(像小计的行照常是数据或起草失败)
subtotals: Subtotals | None = None
#: 期 5:人确认过这张列表没有小计行:以合计字样开头的行就是数据(「累计充值」这类键值清单),执行器不再报
#: subtotal_suspect,每期也就不用再勾那一项确认。和 subtotals 二选一(静态校验 subtotals_conflict)
no_subtotals: bool = False
class ListValues(_M):
@@ -317,6 +338,18 @@ class ListValues(_M):
formula: Literal["accept_cached", "reject"] = "accept_cached"
class ListPeriod(_M):
"""列表按期累积时,每一行属于哪一期,二选一:column 按表里的一个日期列,每行的日期都要落在本期的统计期内;
synthesize 每行加上本期统计期的起、止两列(明细里没有日期列,或者日期不代表它属于哪一期时)。"""
#: 按这个日期列(ListColumn.name,类型 DATE)核对;和 synthesize 二选一
column: str | None = None
#: 怎么核对:within_context 每行的日期都要落在本期统计期内(空着的也不行)
check: Literal["within_context"] = "within_context"
#: 加统计期起、止两列(LIST_PERIOD_START、LIST_PERIOD_END);和 column 二选一
synthesize: Literal["统计期"] | None = None
class ListBlock(_M):
id: str = Field(min_length=1, max_length=32)
layout: Literal["list"]
@@ -333,6 +366,8 @@ class ListBlock(_M):
merged_data: Literal["reject", "fill"] = "reject"
#: 期 3:按表头忽略的列(IgnoreColumn),只针对这些列;其余多出的列照 extra_columns。默认空 = 期 2 的行为
ignore_columns: list[IgnoreColumn] = Field(default_factory=list, max_length=50)
#: 期 5:按期累积时每一行属于哪一期(ListPeriod)。默认 None = 期 4 的行为(列表只能每期替换)
period: ListPeriod | None = None
Block = Annotated[Union[CrosstabBlock, ListBlock], Field(discriminator="layout")]
@@ -441,9 +476,10 @@ class Problem:
fix_args: dict[str, Any] | None = None
#: 修复按钮的九种(P3-SPEC 3.2 ①–⑨)。Problem.fix、FixProposal.kind、确认项 fix:<目标键> 的 kind 都取自这里
#: 修复按钮的种类(P3-SPEC 3.2 ①–⑨;期 5 加 declare_subtotals:列表中间像小计的行改作小计核对)。Problem.fix、
#: FixProposal.kind、确认项 fix:<目标键> 的 kind 都取自这里
FIX_KINDS = ("remove_label", "add_label", "edit_members", "rename_title", "declare_total", "ignore_cells",
"declare_placeholder", "declare_hidden", "rename_sheet")
"declare_placeholder", "declare_hidden", "rename_sheet", "declare_subtotals")
@dataclass
@@ -462,13 +498,14 @@ CheckStatus = Literal["passed", "mismatch", "unverifiable", "info"]
@dataclass
class CheckResult:
#: 稳定 id:K1… 合计核对、G1… 公式引用、T1… 列表合计、R1… 关系(沿用配方里的 id)、C1 统计期、C2 文件名、
#: N1… 行数、P1… 主键;期 3 的并集整体核对 U1 行数、U2 主键、U3 统计期(物化时跑,都是结构类,P3-SPEC 2.7)。
#: 稳定 id:K1… 合计核对、G1… 公式引用、T1… 列表合计(期 5 起也含列表中间的小计)、R1… 关系(沿用配方里的
#: id)、C1 统计期、C2 文件名、N1… 行数、P1… 主键、D1… 列表的日期落在统计期内(期 5,ListPeriod.column);
#: 期 3 的并集整体核对 U1 行数、U2 主键、U3 统计期(物化时跑,都是结构类,P3-SPEC 2.7)。
#: F 编号留给系统发现的事实(Fact.id),两者不混用
id: str
kind: Literal["derived_sum", "formula_refs", "column_sum", "relation_sum_eq", "relation_not_comparable",
"context_agree", "filename_period", "row_count", "pk_unique",
"union_rows", "union_pk", "union_period"]
"union_rows", "union_pk", "union_period", "period_rows"]
title: str
status: CheckStatus
#: 不通过时的类别:structure / data_quality / info
@@ -759,9 +796,10 @@ class SheetsOut:
#: hidden_excluded = hidden.rows=exclude 排除的隐藏行;blank_skipped = 列表 blank_rows=skip 跳过的空行;
#: ignored_rows = 交叉表 ignore_rows 忽略的行;ignored_outside = ignore_outside 忽略了区域外数字格的行;
#: after_stop = 列表停止之后没导入的文字行;total_not_kept = 参与核对、但不另存的合计行(交叉表 keep_as 为空的
#: derived 段、列表 total_row.keep_as 为空的合计行)
#: derived 段、列表 total_row.keep_as 为空的合计行);期 5:subtotal = 列表中间的小计行(参与核对、不进数据,
#: 另存与否都记在这里:它们都不在本表里)
ExcludedReason = Literal["hidden_excluded", "blank_skipped", "ignored_rows", "ignored_outside", "after_stop",
"total_not_kept"]
"total_not_kept", "subtotal"]
@dataclass
@@ -1031,6 +1069,15 @@ def prose_problems(template: str, *, known: dict[str, set[str]] | None = None) -
#: 列表合计行另存成表时,合计项列的固定列名
LIST_TOTAL_DIM = "合计项"
#: 期 5:列表的小计行另存成表时,小计项列的固定列名
LIST_SUBTOTAL_DIM = "小计项"
#: 期 5:列表按期累积、由系统给每行加统计期时(ListPeriod.synthesize)的两列:本期统计期的第一天、最后一天,
#: 写法 YYYY-MM-DD(同 DATE 列,存 TEXT)
LIST_PERIOD_START = "统计期起"
LIST_PERIOD_END = "统计期止"
#: 期 5:小计核对的键(核对编号 T、合计格 DerivedItem.segment)= 块 id 加这个后缀。块 id 在整份配方内唯一,
#: 同一块的合计行用块 id 本身,两者不相混
SUBTOTAL_KEY = "·小计"
_SQL_TYPE = {"INTEGER": "INTEGER", "REAL": "REAL", "TEXT": "TEXT", "DATE": "TEXT"}
@@ -1107,16 +1154,42 @@ def derive_tables(recipe: Recipe) -> tuple[dict[str, list[ColumnOut]], list[Reci
cols = [ColumnOut(c.name, _SQL_TYPE[c.type], header=c.header,
role="measure" if c.type in ("INTEGER", "REAL") else "text")
for c in block.columns]
if block.period is not None and block.period.synthesize:
# 期 5:每行加上本期统计期的起止(role=period:没有格子来历,取值来自统计期)
cols += [ColumnOut(LIST_PERIOD_START, "TEXT", role="period"),
ColumnOut(LIST_PERIOD_END, "TEXT", role="period")]
put(block.table, cols, base)
total = block.rows.total_row
numeric = [c for c in cols if c.type in ("INTEGER", "REAL")]
if total is not None and total.keep_as:
tcols = [ColumnOut(LIST_TOTAL_DIM, "TEXT", role="dim")]
tcols += [ColumnOut(c.name, c.type, header=c.header, role="value")
for c in cols if c.type in ("INTEGER", "REAL")]
tcols += [ColumnOut(c.name, c.type, header=c.header, role="value") for c in numeric]
put(total.keep_as, tcols, f"{base}/rows/total_row")
sub = block.rows.subtotals
if sub is not None and sub.keep_as:
# 期 5:小计另存:小计项(标签的规范写法)、按组小计时的分组列、各数字列
scols = [ColumnOut(LIST_SUBTOTAL_DIM, "TEXT", role="dim")]
if sub.scope == "group_column" and sub.group_column:
scols.append(ColumnOut(sub.group_column, "TEXT", role="dim"))
scols += [ColumnOut(c.name, c.type, header=c.header, role="value") for c in numeric]
put(sub.keep_as, scols, f"{base}/rows/subtotals")
return out, problems
def list_period_columns(recipe: Recipe) -> dict[str, tuple[str, ...]]:
"""期 5:列表写入的表 → 它按什么分期:("column", 日期列) 或 ("synthesize", 统计期起列, 统计期止列)。只列配了
ListBlock.period 的列表;累积计划的 U3、单期的 D 核对、说明里「按哪一列筛选」共用这一份。"""
out: dict[str, tuple[str, ...]] = {}
for sheet in recipe.sheets:
for block in sheet.blocks:
if isinstance(block, ListBlock) and block.period is not None:
if block.period.synthesize:
out.setdefault(block.table, ("synthesize", LIST_PERIOD_START, LIST_PERIOD_END))
elif block.period.column:
out.setdefault(block.table, ("column", block.period.column))
return out
# ==========================================================================
# 期 3:按期累积的资格(静态校验、起草器、累积计划共用同一份,P3-SPEC 2.2)
# ==========================================================================
@@ -1130,10 +1203,13 @@ def accumulate_blockers(recipe: Recipe) -> list[RecipeProblem]:
1. 至少一个工作表配置了统计期上下文(SheetRecipe.context 非空),否则报 accumulate_needs_period(path=/mode);
2. derive_tables 推出的每张表都「按统计期分得开」,否则逐表报 accumulate_unkeyed(path=/tables/{ti},表没在
tables 里声明时指向第一个写入它的块),消息写明哪一条不满足:
- 只由交叉表块写入(measures、dimension 段,或者 derived 段的 keep_as)——列表写入的表一律不满足;
- 写入它的每个交叉表块,axis.checks 都含 covers_context(日期恰好落在统计期内);
- 这张表的 grain 含那个块的 axis.name。
两条都成立时,各期的日期集合互不相交(统计期不重叠由累积计划保证),主键里不需要另加统计期列。
- 交叉表写入的(measures、dimension 段,或者 derived 段的 keep_as):写入它的每个交叉表块,axis.checks 都含
covers_context(日期恰好落在统计期内),并且这张表的 grain 含那个块的 axis.name;
- 期 5:列表写入的明细表:写入它的每个列表块都配了 period(按日期列核对,或加统计期起止两列),并且所在的
工作表配了统计期。列表的合计行、小计行另存的表不能累积:各期的合计项、小计项相同,主键分不开各期;
- 同一张表不能既由交叉表又由列表写入。
都成立时,各期的行按统计期分得开(统计期不重叠由累积计划保证;列表明细跨期重复的同一条记录由并集的主键核对
U2 拒收)。
"""
problems: list[RecipeProblem] = []
if not any(sheet.context for sheet in recipe.sheets):
@@ -1144,6 +1220,11 @@ def accumulate_blockers(recipe: Recipe) -> list[RecipeProblem]:
writers: dict[str, list[CrosstabBlock | ListBlock]] = {}
first_path: dict[str, str] = {}
#: 列表的合计行、小计行另存的表 → 写入它的块
totals: dict[str, list[ListBlock]] = {}
#: 列表块 → 所在工作表有没有配统计期
ctx_of: dict[int, bool] = {}
def _add(table: str, block: CrosstabBlock | ListBlock, path: str) -> None:
got = writers.setdefault(table, [])
if all(b is not block for b in got):
@@ -1161,16 +1242,32 @@ def accumulate_blockers(recipe: Recipe) -> list[RecipeProblem]:
_add(seg.keep_as.table, block, path)
else:
_add(block.table, block, path)
total = block.rows.total_row
if total is not None and total.keep_as:
_add(total.keep_as, block, path)
ctx_of[id(block)] = bool(sheet.context)
for kept in (block.rows.total_row.keep_as if block.rows.total_row else None,
block.rows.subtotals.keep_as if block.rows.subtotals else None):
if kept:
_add(kept, block, path)
totals.setdefault(kept, []).append(block)
index = {t.name: ti for ti, t in enumerate(recipe.tables)}
grains = {t.name: list(t.grain) for t in recipe.tables}
for name in tables:
blocks = writers.get(name, [])
if any(isinstance(b, ListBlock) for b in blocks):
reasons = ["列表形态的表目前只支持每期替换"]
lists = [b for b in blocks if isinstance(b, ListBlock)]
if lists and len(lists) != len(blocks):
reasons = ["同一张表既由交叉表又由列表写入,各期的行分不开"]
elif name in totals:
reasons = ["列表的合计行、小计行另存的表不能按期累积:各期的合计项、小计项相同,主键分不开各期"
"(可以改为只核对、不另存)"]
elif lists:
reasons = []
unbound = [b.id for b in lists if b.period is None]
if unbound:
reasons.append("、".join(f"列表「{x}」" for x in unbound)
+ "没有指定每一行属于哪一期(按日期列核对,或加统计期起止两列)")
no_ctx = [b.id for b in lists if b.period is not None and not ctx_of.get(id(b))]
if no_ctx:
reasons.append("、".join(f"列表「{x}」" for x in no_ctx) + "所在的工作表没有配置统计期")
else:
reasons = []
uncovered = [b.id for b in blocks if isinstance(b, CrosstabBlock) and "covers_context" not in b.axis.checks]
+30 -12
View File
@@ -110,6 +110,16 @@ def load_manifest(artifact_id: str | None) -> dict[str, Any] | None:
return doc if isinstance(doc, dict) else None
def _signer_verified(artifact_id: str | None) -> bool:
"""导入清单里记下的「署名是否认证」;没有清单(简单导入)、读不出来的照实算未认证。"""
try:
doc = load_manifest(artifact_id)
except ValueError:
return False
signer = (doc or {}).get("signed_by")
return bool(signer.get("verified")) if isinstance(signer, dict) else False
def _rows_of(report: dict[str, Any] | None) -> dict[str, int]:
"""构建回执里各表的行数(执行器回执、期 1 的解析回执都是 tables: [{name, rows}])。"""
out: dict[str, int] = {}
@@ -394,7 +404,7 @@ def columns_doc(report: dict[str, Any], periods: list[str]) -> dict[str, dict[st
def manifest_doc(*, source_id: str, snapshot_id: str, action: str, recipe: dict[str, Any],
union: dict[str, Any], parts: list[Part | dict[str, Any]], report: dict[str, Any],
label_sets: list[Any], gaps: list[dict[str, str]], replaced: Any, removed: Any,
notes: SchemaNotes, signed_by: str | None) -> dict[str, Any]:
notes: SchemaNotes, signed_by: str | None, signer_verified: bool = False) -> dict[str, Any]:
"""快照清单(2.9)。parts 按统计期排序,与 report.part_rows 同序;本期写成 dict(导入记录还没落库)。"""
part_rows = report.get("part_rows") or []
keys = []
@@ -422,7 +432,8 @@ def manifest_doc(*, source_id: str, snapshot_id: str, action: str, recipe: dict[
"checks": copy.deepcopy(report.get("checks") or []),
"notes": {"templates": dict(notes.templates),
"rendered": {t: {"comment": n.comment, "columns": dict(n.columns)} for t, n in notes.tables.items()}},
"signed_by": {"name": signed_by, "verified": False},
# 是否认证照实记(05-已知遗留 (c),同导入清单)
"signed_by": {"name": signed_by, "verified": bool(signer_verified and signed_by)},
"created_at": utcnow().isoformat(),
}
@@ -684,8 +695,10 @@ async def _current(session: AsyncSession, source: DataSource) -> _Current:
mode=snap.mode or "replace", recipe=recipe)
def _revoked_record(reason: str, signed_by: str | None) -> dict[str, Any]:
return {"at": utcnow().isoformat(), "reason": reason, "signed_by": signed_by or None, "signed_by_verified": False}
def _revoked_record(reason: str, signed_by: str | None, signer_verified: bool = False) -> dict[str, Any]:
"""作废接受的记录。是否认证照实记(05-已知遗留 (c)):开了登录、署名是登录的人时为真。"""
return {"at": utcnow().isoformat(), "reason": reason, "signed_by": signed_by or None,
"signed_by_verified": bool(signer_verified and signed_by)}
async def rollback_target(session: AsyncSession, source: DataSource, import_id: str) -> SourceSnapshot | None:
@@ -739,7 +752,8 @@ async def revoke_plan(session: AsyncSession, source: DataSource, imp: TableImpor
async def _rebuild_without(session: AsyncSession, cur: _Current, imp: TableImport, *, expected_current: str | None,
kind: str, reason: str, signed_by: str | None, revoke: bool) -> tuple[str, bool]:
kind: str, reason: str, signed_by: str | None, revoke: bool,
signer_verified: bool = False) -> tuple[str, bool]:
"""从当前版本去掉一期、生成(或复用、复活)新快照(7.4)。配方和导入模式不变。返回 (快照 id, 是否复用)。
revoke=True 时同一个事务里写 table_imports.revoked(7.5):复用路径在调 activate_snapshot 之前改好(随它的提交
@@ -758,7 +772,7 @@ async def _rebuild_without(session: AsyncSession, cur: _Current, imp: TableImpor
sid = table_versions.snapshot_id(source_id, keep, recipe_types.RECIPE_ENGINE_VER, mode="accumulate",
recipe_sha=target_sha)
imp_id = imp.id
revoked = _revoked_record(reason, signed_by) if revoke else None
revoked = _revoked_record(reason, signed_by, signer_verified) if revoke else None
activation = {"kind": kind, "import_id": imp_id, "reason": reason, "signed_by": signed_by}
existing = await session.get(SourceSnapshot, sid)
@@ -871,7 +885,7 @@ async def _rebuild_without(session: AsyncSession, cur: _Current, imp: TableImpor
"table_hashes": dict((info.union_report or report_json or {}).get("table_hashes") or {}),
"union_ver": recipe_types.UNION_VER},
parts=parts, report=info.union_report or report_json or {}, label_sets=label_sets, gaps=gaps,
replaced=None, removed=removed, notes=notes, signed_by=signed_by)
replaced=None, removed=removed, notes=notes, signed_by=signed_by, signer_verified=signer_verified)
cache["snapshot_manifest"] = await artifact_store.put_json(
doc, kind="snapshot_manifest", meta={"source_id": source_id, "snapshot_id": info.snapshot_id})
return cache
@@ -931,7 +945,7 @@ def _check_base(cur: _Current, expected_current: str | None) -> None:
async def remove_period(session: AsyncSession, source: DataSource, import_id: str, *, expected_current: str | None,
reason: str, signed_by: str | None) -> dict[str, Any]:
reason: str, signed_by: str | None, signer_verified: bool = False) -> dict[str, Any]:
"""POST …/imports/{id}/remove(7.4)。"""
imp = await session.get(TableImport, import_id)
if imp is None or imp.source_id != source.id:
@@ -947,13 +961,15 @@ async def remove_period(session: AsyncSession, source: DataSource, import_id: st
if len(cur.imports) < 2:
raise ImportRefused(409, "last_period", _LAST_PERIOD)
sid, reused = await _rebuild_without(session, cur, imp, expected_current=expected_current, kind="remove_period",
reason=reason, signed_by=signed_by, revoke=False)
reason=reason, signed_by=signed_by, revoke=False,
signer_verified=signer_verified)
return {"snapshot_id": sid, "removed_import_id": import_id, "reused": reused}
async def revoke_acceptance(session: AsyncSession, source: DataSource, import_id: str, *,
expected_current: str | None, expected_target: str | None,
ack_mask_lost: list[str] | None, reason: str, signed_by: str | None) -> dict[str, Any]:
ack_mask_lost: list[str] | None, reason: str, signed_by: str | None,
signer_verified: bool = False) -> dict[str, Any]:
"""POST …/imports/{id}/revoke-acceptance(7.5)。累积模式(至少两期)= 移除这一期;否则回滚到第 5 节定义的目标。"""
imp = await session.get(TableImport, import_id)
if imp is None or imp.source_id != source.id:
@@ -968,7 +984,7 @@ async def revoke_acceptance(session: AsyncSession, source: DataSource, import_id
if _accumulating(cur):
sid, _reused = await _rebuild_without(session, cur, imp, expected_current=expected_current,
kind="revoke_acceptance", reason=reason, signed_by=signed_by,
revoke=True)
revoke=True, signer_verified=signer_verified)
return {"snapshot_id": sid, "action": "remove_period"}
target = await rollback_target(session, source, imp.id)
if target is None:
@@ -976,7 +992,7 @@ async def revoke_acceptance(session: AsyncSession, source: DataSource, import_id
if target.id != expected_target:
raise ImportRefused(409, "revoke_target_changed", _TARGET_CHANGED)
target_id = target.id
imp.revoked = _revoked_record(reason, signed_by)
imp.revoked = _revoked_record(reason, signed_by, signer_verified)
try:
await table_versions.activate_snapshot(
session, source, target_id, expected_current=expected_current, kind="revoke_acceptance", reason=reason,
@@ -1049,6 +1065,8 @@ async def import_records(session: AsyncSession, source: DataSource, imps: list[T
# 按配方导入才有(简单导入为空):配方、统计期、署名,接受的条数
"recipe_id": imp.recipe_id, "period_start": imp.period_start, "period_end": imp.period_end,
"signed_by": imp.signed_by, "overrides": len(imp.overrides or []), "waivers": len(imp.waivers or []),
# 署名是不是认证过的身份:按这一期导入清单里记下的(05-已知遗留 (c);旧清单写的是 False,照记下的说,不改写)
"signed_by_verified": _signer_verified(imp.manifest_artifact),
# 期 3(7.6)
"acceptances": _acceptance_rows(imp), "revoked": imp.revoked, "manifest_artifact": imp.manifest_artifact,
"recipe_seq": seqs.get(imp.recipe_id) if imp.recipe_id else None, "in_current": in_current,
+14 -14
View File
@@ -326,7 +326,7 @@ class _Table:
「唯一日期列」和这里认的时间列对得上)。"""
meta = next((c for c in self.meta.get("columns") or []
if isinstance(c, Mapping) and str(c.get("name")).lower() == column.lower()), None)
return catalog.is_date_column(column, (meta or {}).get("type"))
return catalog.is_date_column(column, (meta or {}).get("type"), enum_values=(meta or {}).get("enum_values"))
#: 一列解析到的位置:(表, 列的真名)
@@ -1084,36 +1084,36 @@ def graph_checks(nodes: Iterable[Any], checkers: Mapping[str, SqlChecker], *,
return out
async def load_checkers(session: Any, sources: Iterable[Any]) -> dict[str, SqlChecker]:
async def load_checkers(session: Any, sources: Iterable[Any], *, catalog_view: Any = None) -> dict[str, SqlChecker]:
"""给一批数据源(DataSource 行)各建一个检查器:{源名: SqlChecker}。没有目录的源不建。
一次查询读出这些源的全部目录。读不到就当没有目录(返回空 dict):检查是附加的,不能挡住自查和发布本身。
一次查询读出这些源的全部目录,读哪一份由 catalog_view 定(data/catalog_view.py:当前目录、目录快照、关;只用已核实
的项),不给是当前目录。读不到当前目录就当没有目录(返回空 dict):检查是附加的,不能挡住自查和发布本身;视图指定的
目录快照不在了照实报错。
"""
from sqlalchemy import select
from app.db.models import CatalogNote
from app.data.catalog_view import CatalogSnapshotMissing, CatalogView, read_many
rows = [s for s in sources if getattr(s, "id", None) and dialect_of(getattr(s, "kind", None))]
if not rows:
return {}
try:
notes = (await session.execute(
select(CatalogNote.source_id, CatalogNote.table_name, CatalogNote.notes)
.where(CatalogNote.source_id.in_([r.id for r in rows]))
)).all()
entries = await read_many(session, CatalogView.parse(catalog_view), [r.id for r in rows])
except CatalogSnapshotMissing:
raise
except Exception: # noqa: BLE001
logger.exception("读取数据目录失败,本次不做基于目录的 SQL 检查")
return {}
by_source: dict[str, dict[str, Any]] = {}
for source_id, table, value in notes:
if isinstance(value, dict) and value:
by_source.setdefault(source_id, {})[table] = value
for source_id, tables in entries.items():
for table, entry in tables.items():
if isinstance(entry.notes, dict) and entry.notes:
by_source.setdefault(source_id, {})[table] = entry.notes
return {r.name: SqlChecker(kind=r.kind, schema_cache=r.schema_cache, notes=by_source[r.id], source_name=r.name)
for r in rows if r.id in by_source}
async def load_checkers_by_name(session: Any, names: Iterable[str]) -> dict[str, SqlChecker]:
"""按数据源名建检查器(只看启用着的源)。发布门禁用:图里写的是源名。"""
"""按数据源名建检查器(只看启用着的源)。发布门禁用:图里写的是源名,读的是当前目录。"""
from sqlalchemy import select
from app.db.models import DataSource
+4 -2
View File
@@ -2130,7 +2130,7 @@ class RawPurge:
async def purge_import_raw(
session: AsyncSession, imp: TableImport, *, reason: str, signed_by: str | None
session: AsyncSession, imp: TableImport, *, reason: str, signed_by: str | None, signer_verified: bool = False
) -> RawPurge:
"""清除一次导入的原件(提交)。按两元组解包得到 (是否删了文件, 一并标成已清除的其他导入记录),
discarded_stagings 是一并放弃的暂存区。
@@ -2160,7 +2160,9 @@ async def purge_import_raw(
now = utcnow()
record = {
"at": now.isoformat(), "reason": reason,
"signed_by": (signed_by or "").strip() or None, "signed_by_verified": False,
# 是否认证照实记(05-已知遗留 (c)):开了登录、署名是登录的人时为真
"signed_by": (signed_by or "").strip() or None,
"signed_by_verified": bool(signer_verified and (signed_by or "").strip()),
}
imp.raw_state, imp.purged = "purged", record
for other in others:
+14 -1
View File
@@ -24,7 +24,7 @@ from collections.abc import Iterator
from typing import Any
from xml.etree import ElementTree as ET
from app.data import tabular
from app.data import csv_grid, tabular
from app.data.recipe_types import Grid, GridCell
from app.data.xlsx_scan import SheetScan, UnsupportedTable, WorkbookScan, _broken, cell_ref, parse_ref
@@ -70,6 +70,14 @@ class Workbooks:
self.raw = raw
self._values: Any = None
self._formulas: Any = None
self._csv: Any = None
def csv(self, delimiter: str | None = None) -> Any:
"""CSV 原件解析好的记录(csv_grid.CsvTable),第一次要读时才解析,同一次执行里复用。delimiter 是扫描时定下的
分隔符(csv_grid.sheet_delimiter),不给时再嗅一次。"""
if self._csv is None:
self._csv = csv_grid.parse(self.raw, delimiter)
return self._csv
def values(self) -> Any:
if self._values is None:
@@ -126,6 +134,11 @@ def sheet_rows(books: Workbooks, sheet: SheetScan, *, min_row: int, max_row: int
行数和每行的宽度都一样。文件内容写坏了(见 _READ_ERRORS)抛 UnsupportedTable,日志只记异常类型。
"""
try:
if csv_grid.is_csv_sheet(sheet):
# CSV 适配出来的「工作表」:从解析好的记录里取,口径同下(只给非空格、整行为空的不给)。分隔符用扫描时定下的
yield from csv_grid.rows(books.csv(csv_grid.sheet_delimiter(sheet)), min_row=min_row, max_row=max_row,
min_col=min_col, max_col=max_col)
return
yield from _sheet_rows(books, sheet, min_row=min_row, max_row=max_row, min_col=min_col, max_col=max_col)
except UnsupportedTable:
raise
+5 -1
View File
@@ -216,6 +216,9 @@ class WorkbookScan:
tables: list[TableScan] = field(default_factory=list)
#: 定义名称 → 原文。工作表级的名称记作「工作表名!名称」,免得同名互相覆盖
defined_names: dict[str, str] = field(default_factory=dict)
#: 原件的格式:xlsx,或者 csv(csv_grid.scan 适配出来的一张「工作表」,读取层据 SheetScan.path 从 CSV 里读)。
#: 老暂存区的扫描结果里没有这个键,按 xlsx 读
format: str = "xlsx"
def sheet(self, name: str) -> SheetScan | None:
return next((s for s in self.sheets if s.name == name), None)
@@ -1090,4 +1093,5 @@ def scan_from_json(data: dict[str, Any]) -> WorkbookScan:
for t in tables:
t.columns = list(t.columns)
return WorkbookScan(sheets=sheets, full_calc_on_load=bool(data.get("full_calc_on_load")), tables=tables,
defined_names=dict(data.get("defined_names") or {}))
defined_names=dict(data.get("defined_names") or {}),
format=str(data.get("format") or "xlsx"))
+6
View File
@@ -239,6 +239,12 @@ _COLUMN_MIGRATIONS: list[tuple[str, str, str]] = [
("apps", "scope", "VARCHAR(20) DEFAULT 'shared'"),
("apps", "owner", "VARCHAR(100)"),
("apps", "blueprint_id", "VARCHAR(32)"),
# 下一轮(docs/design/next-round/00-总方案.md X4、X9):运行为什么结束在 cancelled;定时的目标种类和 id;
# 定时触发建的那次评测。存量行:end_reason 为空,target_kind 是 workflow(target_id 由 scheduler 按 workflow_id 读)
("runs", "end_reason", "VARCHAR(20)"),
("schedules", "target_kind", "VARCHAR(20) DEFAULT 'workflow'"),
("schedules", "target_id", "VARCHAR(32)"),
("schedule_fires", "eval_run_id", "VARCHAR(32)"),
]
#: 迁移补上的列要建的索引:(索引名, 表, 列或多列)。create_all 不碰已有的表,表上新列的索引也就不会建。
+11
View File
@@ -654,6 +654,9 @@ class Run(Base, TimestampMixin):
input: Mapped[dict[str, Any]] = mapped_column(default=dict)
output: Mapped[dict[str, Any]] = mapped_column(default=dict)
error: Mapped[str | None] = mapped_column(Text, default=None)
# 运行为什么结束在 cancelled(docs/design/next-round/00-总方案.md X9):rejected 人工驳回、stopped 手动停止。
# 其余终态、加这一列之前的运行为空
end_reason: Mapped[str | None] = mapped_column(String(20), default=None)
usage: Mapped[dict[str, Any]] = mapped_column(default=dict) # tokens / cost / 各节点耗时
started_at: Mapped[datetime | None] = mapped_column(default=None)
finished_at: Mapped[datetime | None] = mapped_column(default=None)
@@ -975,6 +978,12 @@ class Schedule(Base, TimestampMixin):
id: Mapped[str] = mapped_column(String(32), primary_key=True, default=new_id)
name: Mapped[str] = mapped_column(String(200), default="")
# 定时的目标(next-round X4,定时只有一套):workflow 发起工作流运行;eval_suite 跑一次评测题集;
# source_maintenance 维护一个数据源(探查、剖析、取值观察)。加这一列之前的任务都是 workflow
target_kind: Mapped[str] = mapped_column(String(20), default="workflow")
# 目标的 id:workflow 时同 workflow_id;其余两种是题集 id、数据源 id
target_id: Mapped[str | None] = mapped_column(String(32), default=None, index=True)
# 非空(老表的约束,只加不改的迁移放宽不了):目标不是工作流时存空串,读写都经 scheduler/service 里的一处封装
workflow_id: Mapped[str] = mapped_column(String(32), index=True)
# 保存时的工作流名。工作流还在时接口给它现在的名字,删掉之后才用这一份
workflow_name: Mapped[str] = mapped_column(String(200), default="")
@@ -1020,6 +1029,8 @@ class ScheduleFire(Base):
# started | skipped_overlap | skipped_misfire | failed_to_start
status: Mapped[str] = mapped_column(String(30), default="started")
run_id: Mapped[str | None] = mapped_column(String(32), default=None, index=True)
# 目标是评测题集时,这一次触发建的那次评测(target_kind=eval_suite,next-round X4)
eval_run_id: Mapped[str | None] = mapped_column(String(32), default=None)
reason: Mapped[str | None] = mapped_column(Text, default=None)
# 渲染后的输入,就是交给运行的那一份
inputs: Mapped[dict[str, Any]] = mapped_column(default=dict)
+4
View File
@@ -39,6 +39,9 @@ class Decision:
#: 审批卡上点的是「始终允许」:批准这次,并且这个工具以后改由门控把关(tools/trust.py)。
#: 只认明确的 True,而且只跟着批准走
always: bool = False
#: 这次驳回是待办超时、按「超时驳回」替人做的(process/tasks.py 的 tick),不是有人点了驳回。只跟着驳回走,
#: 节点据此在 NodeRejected 上记 decision=expired_reject
expired: bool = False
def _word(value: Any) -> str:
@@ -71,6 +74,7 @@ def read_decision(response: Any) -> Decision:
code=code if isinstance(code, str) and code else None,
value=response.get("value"),
always=verdict is True and response.get("always") is True,
expired=verdict is False and response.get("expired") is True,
)
verdict = _verdict(response)
# 不是一个是/否词的字符串("拒绝,这个表是生产库")既算不上批准,本身又
+10 -3
View File
@@ -10,7 +10,7 @@ from langgraph.graph import END, START, StateGraph
from app.core.errors import describe_exception, raw_detail
from app.core.events import EventType
from app.engine.context import NodeContext, NodeError, RunContext
from app.engine.context import NodeContext, NodeError, NodeRejected, RunContext
from app.engine.expressions import open_template_scope, template_scope
from app.engine.nodes import (
challenge, control, dual_check, human, io, knowledge, llm, merge, metrics, multi, report, tools,
@@ -144,6 +144,13 @@ def _wrap(
raise
except asyncio.CancelledError:
raise
except NodeRejected as e:
# 人的决定(驳回、超时驳回),不是故障(总方案 X9):「失败重试」「出错时继续」都是给故障准备的。重试只会把
# 同一件事再问一遍人;on_error=continue 会把驳回吞掉接着往下走,最后以 succeeded 结束,通知、定时推送、
# 发布单都看不到这次驳回。所以不重试、不被容错接住;节点这一步照旧记一条 node.failed(时间线、实例进度
# 按它收尾这一步),交给 runner 收成 cancelled + end_reason=rejected
last_error = e
break
except Exception as e: # noqa: BLE001
# NodeError 以前在这里直接 raise,于是节点自己抛的错既不重试、
# 也不发 node.failed、更不被 on_error=continue 接住。而 NodeError
@@ -168,8 +175,8 @@ def _wrap(
ctx.emit(EventType.NODE_FAILED, error=message, duration_ms=elapsed, node_id=node.id,
detail=_detail(last_error))
# 容错模式:记下错误继续往下走,而不是让整张图挂掉
if node.config.get("on_error") == "continue":
# 容错模式:记下错误继续往下走,而不是让整张图挂掉。驳回不算错误,不走这里(见上)
if node.config.get("on_error") == "continue" and not isinstance(last_error, NodeRejected):
return {
"nodes": {node.id: {"error": message, "failed": True}},
"trail": [{"node_id": node.id, "error": message, "ts": time.time()}],
+30
View File
@@ -106,3 +106,33 @@ class NodeError(RuntimeError):
super().__init__(message)
self.node_id = node_id
self.retryable = retryable
#: 驳回发生在哪种关卡上(NodeRejected.kind):人工节点的三种模式、写入前的动作确认、代码执行审批、工具调用审批、双盲裁决
REJECTION_KINDS = ("approve", "input", "edit", "action_confirm", "code", "tool", "dual_check")
class NodeRejected(NodeError):
"""人的决定让运行到这里结束:驳回、超时驳回(docs/design/next-round/05-已知遗留.md (a))。
这不是故障。以前六处驳回都抛普通的 NodeError,运行记成 failed:通知推「实例失败」,定时推「运行失败」,发布单把
驳回过的试运行说成「试运行失败了」,记录页还给「继续运行」——接着跑只会读到同一个驳回答复、再停一次。现在 runner
单独接住它,收成 cancelled + end_reason=rejected,run.cancelled 带上在哪一步、谁、结论和备注,消费方都读这一个字段。
是 NodeError 的子类:节点这一步照旧记 node.failed(时间线、实例进度按它收尾);但不重试、不被 on_error=continue
接住(engine/compiler._execute:那两项是给故障准备的,驳回被吞掉运行就以 succeeded 结束了)。原话(str)照旧写给人看,
Run.error 和以前一字不差。kind 见 REJECTION_KINDS;decision 是 reject 或 expired_reject(同待办的结论)。"""
def __init__(self, node_id: str, message: str, *, kind: str, decision: str = "reject",
actor: str | None = None, note: str = "") -> None:
super().__init__(node_id, message)
self.kind = kind
self.decision = decision
self.actor = actor
self.note = note
@classmethod
def of(cls, ctx: Any, message: str, *, kind: str, decision: Any) -> NodeRejected:
"""按节点上下文和读出来的审批结论(engine/approval.Decision)造一个:谁驳回的取 ctx.actor()。"""
return cls(ctx.node.id, message, kind=kind, decision="expired_reject" if decision.expired else "reject",
actor=ctx.actor(), note=decision.note or "")
+10 -51
View File
@@ -37,7 +37,6 @@ import re
import time
from collections.abc import Callable, Iterable
from dataclasses import asdict, dataclass
from datetime import datetime
from typing import Any
from langchain_core.messages import AIMessage, BaseMessage, HumanMessage, SystemMessage
@@ -68,6 +67,7 @@ from app.engine.evidence import (
)
from app.engine.expressions import CellError, column_kind, locate_cell
from app.engine.merge_query import MERGE_SOURCE
from app.engine import spend
from app.engine.nodes.llm import _split_structured, _usage_of
from app.engine.state import message_text
from app.providers import catalog as pricing
@@ -855,7 +855,7 @@ JUDGE_DEFAULTS: dict[str, Any] = {
"click_max_cost_usd": 0.01, # 探索运行里每点一次「请模型判断」的金额上限
"daily_max_usd": 2.0, # 全局每日裁判金额上限(按本地日期)
}
#: 每日计数存在设置表的这一行:{date: 本地日期, usd, calls, unpriced_calls}。不从设置接口进出
#: 每日计数存在设置表的这一行:{date: 本地日期, usd, calls, unpriced_calls}。不从设置接口进出(账本见 engine/spend.py)
SPEND_KEY = "judge_spend"
_COUNTS = frozenset({"report_max_claims"})
_SECONDS = frozenset({"report_timeout_s"})
@@ -1024,61 +1024,20 @@ async def source_masks(catalog: dict[str, Any], *, loader: Callable[[str], Any]
# 每日计数
# --------------------------------------------------------------------------
_spend_lock = asyncio.Lock()
def _today() -> str:
"""本地日期:每日上限按它滚动。"""
return datetime.now().astimezone().date().isoformat()
"""裁判这本账的日历:本地日期(同 engine/spend._today)。单独留一个名字:测试换掉它来模拟裁判这边跨天。"""
return spend._today()
def _spend_of(row: Any) -> dict[str, Any]:
value = row.value if row is not None and isinstance(row.value, dict) else {}
today = _today()
if value.get("date") != today:
return {"date": today, "usd": 0.0, "calls": 0, "unpriced_calls": 0}
return {"date": today, "usd": float(value.get("usd") or 0.0), "calls": int(value.get("calls") or 0),
"unpriced_calls": int(value.get("unpriced_calls") or 0)}
#: 裁判这本每日账(engine/spend.py 的 Ledger:预留、找补、按本地日期滚动和核查、评测是同一套代码,各记各的键)
judge_spend = spend.Ledger(SPEND_KEY, today=lambda: _today())
_LEDGER = judge_spend
async def daily_spend() -> dict[str, Any]:
"""今天(本地日期)裁判已经花了多少:{date, usd, calls, unpriced_calls}。估不出金额的调用只计次数。"""
async with SessionLocal() as session:
return _spend_of(await session.get(Setting, SPEND_KEY))
async def _charge(amount: float, *, limit: float | None, priced: bool) -> str | None:
"""记一次调用、先按估算预留金额。加上这笔会超过每日上限就不记,返回 None;记下了返回记在哪天。"""
async with _spend_lock:
async with SessionLocal() as session:
row = await session.get(Setting, SPEND_KEY)
spend = _spend_of(row)
if limit is not None and spend["usd"] + amount > limit + 1e-12:
return None
spend["usd"] = round(spend["usd"] + amount, 8)
spend["calls"] += 1
spend["unpriced_calls"] += 0 if priced else 1
if row is None:
session.add(Setting(key=SPEND_KEY, value=spend))
else:
row.value = spend
await session.commit()
return spend["date"]
async def _settle(date: str, delta: float) -> None:
"""调用结束后按实际用量找补预留的差额。跨了天就不找补:预留记在前一天,今天从零算。"""
if not delta:
return
async with _spend_lock:
async with SessionLocal() as session:
row = await session.get(Setting, SPEND_KEY)
spend = _spend_of(row)
if row is None or spend["date"] != date:
return
spend["usd"] = max(0.0, round(spend["usd"] + delta, 8))
row.value = spend
await session.commit()
return await _LEDGER.daily()
# --------------------------------------------------------------------------
@@ -1318,7 +1277,7 @@ async def run_request(request: JudgeRequest, *, spec: ModelSpec, budget: Budget,
pending = pending[:len(batch)]
pending = pending[len(batch):]
charged = cost or 0.0
day = await _charge(charged, limit=budget.daily_max_usd if priced else None, priced=priced)
day = await _LEDGER.charge(charged, limit=budget.daily_max_usd if priced else None, priced=priced)
if day is None: # 估算之后、记账之前别的裁判先花掉了今天剩下的钱
stop([*batch, *pending], "daily_max_usd")
pending = []
@@ -1353,7 +1312,7 @@ async def run_request(request: JudgeRequest, *, spec: ModelSpec, budget: Budget,
inp, out = sum(t["input_tokens"] for t in tokens), sum(t["output_tokens"] for t in tokens)
# 回复没带用量(有的网关不给、调用超时)就按估算记:宁可多记,不能让每日上限漏掉
actual = (pricing.estimate_cost(model_id, inp, out) if inp or out else charged) if priced else 0.0
await _settle(day, actual - charged)
await _LEDGER.settle(day, actual - charged)
spent += actual
usage["input_tokens"] += inp
usage["output_tokens"] += out
+3 -1
View File
@@ -67,6 +67,7 @@ from app.core.errors import describe_exception
from app.core.events import EventType
from app.db.base import SessionLocal
from app.db.models import DataSource
from app.data import catalog_view
from app.engine import deadline, verify_agent
from app.engine import judge as judging
from app.engine.context import NodeContext, NodeError
@@ -767,7 +768,8 @@ class _Review:
spec=self.spec, system=CHALLENGE_SYSTEM, task=task_text, sources=live,
output_schema=CHALLENGE_SCHEMA, max_queries=self.max_queries, timeout_s=allot.timeout_s,
daily_max_usd=self.daily_max_usd, emit=deadline.relabel(ctx.emit, allot), run_id=ctx.run.run_id,
node_id=ctx.node.id, data_versions=run_versions(ctx.run), masked_values=known)
node_id=ctx.node.id, data_versions=run_versions(ctx.run), masked_values=known,
catalog_view=catalog_view.of_run(ctx.run))
outcome = await asyncio.to_thread(_masked_rationale, outcome, known, given=[cand.text], kinds=self.kinds,
masks=self.masks)
if outcome.error and not outcome.model:
+16 -13
View File
@@ -68,9 +68,10 @@ from app.core.errors import describe_exception
from app.core.events import EventType
from app.db.base import SessionLocal
from app.db.models import DataSource
from app.data import catalog_view
from app.engine import deadline, recompute
from app.engine.approval import read_decision
from app.engine.context import NodeContext, NodeError
from app.engine.context import NodeContext, NodeError, NodeRejected
from app.engine.dual_check_facts import preview, preview_joint, record_choices, upload_dates
from app.engine.evidence import build_catalog, is_number, next_exec
from app.engine.expressions import ExpressionError, _cell_call, _chain_root, cell_value, column_kind, numeric_text, \
@@ -346,9 +347,11 @@ def _skipped(ctx: NodeContext, reason: str) -> dict[str, Any]:
class _Sources:
"""一次复算里要读的查询快照、数据源和快照要遮的列:同一份只读一次、只算一次。"""
"""一次复算里要读的查询快照、数据源和快照要遮的列:同一份只读一次、只算一次。view 是这次运行的目录视图
(data/catalog_view.py):交给盲算的目录、比较两条 SQL 用的中文名都按它读,和运行时工具给模型看的是同一份。"""
def __init__(self) -> None:
def __init__(self, view: Any = None) -> None:
self.view = catalog_view.CatalogView.parse(view)
self.snapshots: dict[str, tuple[Any, str | None]] = {}
self.rows: dict[str, DataSource | None] = {}
self.masks: dict[tuple[str, str | None], frozenset[str]] = {}
@@ -782,14 +785,14 @@ async def _constant_columns(snapshot: Mapping[str, Any], sources: _Sources) -> f
async def _catalog_text(sources: _Sources, name: str) -> str:
"""交给盲算 agent 的数据目录:表清单(表名、中文名、粒度)和有目录的表的业务说明(业务日期、有效记录条件、去重、
字段含义、码值、关联关系)。太长的截断,其余请它用 db_schema 查。"""
字段含义、码值、关联关系)。太长的截断,其余请它用 db_schema 查。读哪一份由这次运行的目录视图定(sources.view),
和运行时工具给模型看的是同一份。"""
from app.data import catalog
row = await sources.source(name)
if row is None:
return BLIND_NO_CATALOG_PROMPT
async with SessionLocal() as session:
entries = await catalog.read_catalog(session, row.id)
entries = await catalog_view.read(sources.view, row)
cache = row.schema_cache if isinstance(row.schema_cache, dict) else {}
tables = cache.get("tables") or {}
notes = {table: entry.notes for table, entry in entries.items()}
@@ -843,7 +846,7 @@ async def _blind(ctx: NodeContext, jobs: dict[str, dict[str, Any]], options: _Op
return {}, None, False, usage
async with SessionLocal() as session:
spec = await verify_model_spec(session, ctx.config, settings=dict(settings))
sources = _Sources()
sources = _Sources(catalog_view.of_run(ctx.run))
catalogs: dict[str, str] = {}
for job in jobs.values():
if job["source"] not in catalogs:
@@ -870,7 +873,7 @@ async def _blind(ctx: NodeContext, jobs: dict[str, dict[str, Any]], options: _Op
spec=spec, system=BLIND_SYSTEM, task=_task_text(job, catalogs[job["source"]]), sources=[job["source"]],
output_schema=BLIND_SCHEMA, max_queries=options.max_queries, timeout_s=allot.timeout_s,
daily_max_usd=options.daily_max_usd, emit=deadline.relabel(ctx.emit, allot), run_id=ctx.run.run_id,
node_id=ctx.node.id, data_versions=run_versions(ctx.run))
node_id=ctx.node.id, data_versions=run_versions(ctx.run), catalog_view=sources.view)
keys = list(jobs)
await one(keys[0], jobs[keys[0]])
@@ -1078,7 +1081,7 @@ async def _review(state: GraphState, ctx: NodeContext, checked: dict[str, Any],
"""
rows = [r for r in checked["rows"] if r["status"] == DIFFERENT]
results = checked["results"]
sources = _Sources()
sources = _Sources(catalog_view.of_run(ctx.run))
infos: dict[str, dict[str, Any]] = {}
metrics: list[dict[str, Any]] = []
summary: list[dict[str, Any]] = []
@@ -1179,15 +1182,15 @@ async def _source_info(sources: _Sources, name: str) -> dict[str, Any]:
(uploaded:导入时日期一律存成文字)和它导入时认作日期的列(upload_dates,按当前版本的配方读,见
dual_check_facts.upload_dates)——业务日期据此认上传表格的日期列,认不出的原因照实说(_dated_problem)。两项都进
候选:审批卡的预告按发起审批时读的这一份;裁决后的真写按写入时的版本重读 upload_dates,和现读的表结构一起以写入时
为准(dual_check_facts.record_choices)。数据源没了就都不给,照样比原文。"""
from app.data import catalog
为准(dual_check_facts.record_choices)。数据源没了就都不给,照样比原文。目录按这次运行的目录视图读(sources.view),
比较两条 SQL 用到的中文名和运行时给模型看的是同一份。"""
from app.data.sql_semantics import catalog_labels
row = await sources.source(name)
if row is None:
return {"kind": None, "schema": None, "names": {}, "notes": {}, "uploaded": False, "upload_dates": {}}
entries = await catalog_view.read(sources.view, row)
async with SessionLocal() as session:
entries = await catalog.read_catalog(session, row.id)
dates = await upload_dates(session, row)
notes = {table: entry.notes for table, entry in entries.items()}
return {"kind": row.kind, "schema": row.schema_cache if isinstance(row.schema_cache, dict) else None,
@@ -1396,7 +1399,7 @@ def _choices(ctx: NodeContext, response: Any, rows: list[dict[str, Any]]) -> tup
填写模式一样:这一步没有驳回的出口。"""
decision = read_decision(response)
if isinstance(response, dict) and decision.rejected:
raise NodeError(ctx.node.id, f"人工驳回:{decision.note or '未说明原因'}")
raise NodeRejected.of(ctx, f"人工驳回:{decision.note or '未说明原因'}", kind="dual_check", decision=decision)
value = decision.value if isinstance(decision.value, dict) else response if isinstance(response, dict) else {}
raw = value.get("choices") if isinstance(value.get("choices"), dict) else {}
choices = {r["metric_id"]: raw[r["metric_id"]] for r in rows if raw.get(r["metric_id"]) in CHOICES}
+3 -3
View File
@@ -8,7 +8,7 @@ from typing import Any
from app.core.events import EventType
from app.db.base import SessionLocal
from app.engine.approval import read_decision
from app.engine.context import NodeContext, NodeError
from app.engine.context import NodeContext, NodeError, NodeRejected
from app.engine.replay import ask, once
from app.engine.state import GraphState, merge_usage, message_text, template_context
from app.providers import catalog
@@ -52,7 +52,7 @@ async def run_human(state: GraphState, ctx: NodeContext) -> dict[str, Any]:
"__decision__": "approved" if approved else "rejected",
}
if not approved and ctx.cfg("stop_on_reject", False):
raise NodeError(ctx.node.id, f"人工驳回:{result['note'] or '未说明原因'}")
raise NodeRejected.of(ctx, f"人工驳回:{result['note'] or '未说明原因'}", kind="approve", decision=decision)
else: # edit / input
# 驳回在这两种模式下以前是个摆设:__decision__ 硬编码成 approved,
# approved 字段看都不看。界面上按钮在、toast 说"已驳回",运行却照常
@@ -65,7 +65,7 @@ async def run_human(state: GraphState, ctx: NodeContext) -> dict[str, Any]:
# 只有结构体里明说了"不"才算驳回:这两种模式下裸字符串就是内容本身,
# 用户在输入框里回一句"no"是在交稿,不是在驳回。
if isinstance(response, dict) and decision.rejected:
raise NodeError(ctx.node.id, f"人工驳回:{decision.note or '未说明原因'}")
raise NodeRejected.of(ctx, f"人工驳回:{decision.note or '未说明原因'}", kind=mode, decision=decision)
note = decision.note if isinstance(response, dict) else "" # 裸字符串是内容,不是备注
result = {"value": decision.value, "note": note, "__decision__": "approved"}
+3
View File
@@ -48,6 +48,7 @@ from app.providers.factory import (
get_chat_model,
)
from app.engine.sql_problems import event_checks
from app.data import catalog_view
from app.tools.datasource import QUERY_PREFIX, run_masks_results, run_reads_only, run_scope_fixed, run_versions
from app.tools.registry import (
ToolArgsError,
@@ -341,6 +342,8 @@ async def _resolve_tools(ctx: NodeContext, state: GraphState) -> list[BaseTool]:
mask_results=run_masks_results(ctx.run),
force_readonly=run_reads_only(ctx.run),
fixed_scope=fixed,
# 读哪一份数据目录:发起时定下的(data/catalog_view.py),审批恢复、续跑都沿用
catalog_view=catalog_view.of_run(ctx.run),
# 数据源工具运行中途补固定版本时经它留一条事件(tools/datasource.py 的 _pin_late)
emit=ctx.emit,
)
+3
View File
@@ -28,6 +28,7 @@ from app.engine.toolcalls import (
from app.providers import catalog
from app.providers.factory import ModelSpec, bind_tools_safely, get_chat_model
from app.engine.sql_problems import event_checks
from app.data import catalog_view
from app.tools.datasource import QUERY_PREFIX, run_masks_results, run_reads_only, run_scope_fixed, run_versions
from app.tools.registry import (
ToolArgsError,
@@ -317,6 +318,8 @@ async def run_supervisor(state: GraphState, ctx: NodeContext) -> dict[str, Any]:
mask_results=run_masks_results(ctx.run),
force_readonly=run_reads_only(ctx.run),
fixed_scope=run_scope_fixed(ctx.run),
# 读哪一份数据目录:发起时定下的(data/catalog_view.py),审批恢复、续跑都沿用
catalog_view=catalog_view.of_run(ctx.run),
# 数据源工具运行中途补固定版本时经它留一条事件(tools/datasource.py 的 _pin_late)
emit=ctx.emit,
)
+13 -9
View File
@@ -11,7 +11,7 @@ from langgraph.func import task
from app.core.artifact_store import canonical_json, content_hash
from app.engine.approval import read_decision
from app.engine.context import NodeContext, NodeError
from app.engine.context import NodeContext, NodeError, NodeRejected
from app.engine.evidence import ledger_enabled, next_exec, query_entry_fields, schema_ledger_entry
from app.engine.replay import ask, once
from app.engine.state import GraphState
@@ -20,6 +20,7 @@ from app.sandbox.base import SandboxLimits
from app.sandbox.manager import sandbox_manager
from app.engine.sql_problems import event_checks
from app.capabilities.registry import is_platform_query, snapshot_owned
from app.data import catalog_view
from app.tools.datasource import (DATA_UNAVAILABLE, SCHEMA_PREFIX, run_masks_results, run_reads_only,
run_scope_fixed, run_versions)
from app.tools.registry import (
@@ -54,6 +55,8 @@ def _tool_ctx(ctx: NodeContext) -> ToolContext:
mask_results=run_masks_results(ctx.run),
force_readonly=run_reads_only(ctx.run),
fixed_scope=fixed,
# 读哪一份数据目录:发起时定下的(data/catalog_view.py),审批恢复、续跑都沿用
catalog_view=catalog_view.of_run(ctx.run),
# 数据源工具运行中途补固定版本时经它留一条事件(tools/datasource.py 的 _pin_late)
emit=ctx.emit,
)
@@ -127,8 +130,8 @@ async def run_tool(state: GraphState, ctx: NodeContext) -> dict[str, Any]:
await once(ctx, EventType.HUMAN_RESOLVED, tool=name, approved=decision.approved,
note=decision.note, actor=ctx.actor(), **({"always": True} if always else {}))
if not decision.approved:
raise NodeError(ctx.node.id, f"工具 {name} 的调用已被驳回"
+ (f":{decision.note}" if decision.note else ""))
raise NodeRejected.of(ctx, f"工具 {name} 的调用已被驳回" + (f":{decision.note}" if decision.note else ""),
kind="tool", decision=decision)
limit = limit_of(tool, name, args)
ctx.emit(EventType.TOOL_START, tool=name, args=args, **limit_fields(limit))
@@ -299,10 +302,11 @@ async def _run_write(state: GraphState, ctx: NodeContext, name: str, args: dict[
await once(ctx, EventType.HUMAN_RESOLVED, tool=name, approved=decision.approved, note=decision.note,
actor=ctx.actor(), payload_hash=confirmed)
if not decision.approved:
# 驳回是人的决定,不是故障:运行到这里结束(状态记失败,没有可走的出口),说法写成「未确认(被驳回)」,
# 实例页按待办的结论另说(apps/resolve.instance_states)
raise NodeError(ctx.node.id, f"「{ctx.node.title}」未确认(被驳回),没有发送"
+ (f":{decision.note}" if decision.note else ""))
# 驳回是人的决定,不是故障:运行到这里结束(没有可走的出口),收成 cancelled + end_reason=rejected
# (NodeRejected),说法写成「未确认(被驳回)」,实例页、通知按同一个结论说(apps/resolve.instance_state)
raise NodeRejected.of(ctx, f"「{ctx.node.title}」未确认(被驳回),没有发送"
+ (f":{decision.note}" if decision.note else ""),
kind="action_confirm", decision=decision)
if confirmed != prep.payload_hash:
# 动作确认不接受改参数:要改在上游的表单里改。确认之后上游的数据变了,看过的就不是要发的,拒发
raise NodeError(ctx.node.id, f"「{name}」确认时看到的内容和现在要发送的不一致(内容指纹对不上),"
@@ -380,8 +384,8 @@ async def run_code(state: GraphState, ctx: NodeContext) -> dict[str, Any]:
await once(ctx, EventType.HUMAN_RESOLVED, approved=decision.approved, note=decision.note,
actor=ctx.actor())
if not decision.approved:
raise NodeError(ctx.node.id, "代码执行已被驳回"
+ (f":{decision.note}" if decision.note else ""))
raise NodeRejected.of(ctx, "代码执行已被驳回" + (f":{decision.note}" if decision.note else ""),
kind="code", decision=decision)
# 隔离档位:strict 要硬件级(microVM),fast 要低延迟(Seatbelt/bwrap),
# 留空跟随整机默认。要的那档不可用时会安静退回默认,不阻断运行。
+23
View File
@@ -24,11 +24,34 @@ ORIGIN_KINDS = (
"api", # 直接调接口,没说是哪
"trial", # 蓝图的试运行(探索运行,docs/design/normal-user-apps.md N4)
"personal_app", # 个人应用的刷新(探索运行,N7)
# 下一轮(docs/design/next-round/00-总方案.md X1):都只能由服务端写
"eval", # 评测:作答和算期望的运行(探索运行)
"ledger_issue", # 问数据的答案按已治理口径正式出具(正式运行)
"reconcile", # 口径对账:这次的取数和口径在同一份数据上各算一次(探索运行)
"recall", # 口径召回:新旧两版口径在同一份数据上各算一次(探索运行)
)
#: 客户端在 POST /api/runs 里能声明的来源
CLIENT_ORIGINS = ("chat", "canvas", "launcher", "api")
#: 系统派生的运行(docs/design/next-round/00-总方案.md X1):评测、口径对账、口径召回都是服务端为了量、为了比替人跑的,
#: 不是有人在用。它们是完整的运行(证据、封存都在、点得开),但不进使用统计——探索问题聚类、工具白名单衰减、数据目录的
#: 使用次数和「被哪些运行用过」、审阅队列的使用权重、以后的账本统计——否则一跑评测,治理信号和目录的使用热度就失真了。
#: 记录页默认也不列(筛选里能选,列表照实写有几次没列出)。ledger_issue 是替用户正式出具,照常计入
SYSTEM_ORIGINS = ("eval", "reconcile", "recall")
def counted_runs() -> Any:
"""SQL 条件:计入使用统计的运行(排除 SYSTEM_ORIGINS)。所有使用统计都只算它圈进来的运行,别处不另写一份过滤。
来源按非空比较(coalesce,同 auth/access.hidden_runs):加来源这一列之前的老运行 origin_kind 为空,NOT IN 碰上 NULL
会把它们一起筛掉;外连接没对上运行的行(问数据只回了一句、没有运行的那一轮)同理照常计入。"""
from sqlalchemy import func
from app.db.models import Run
return func.coalesce(Run.origin_kind, "").not_in(SYSTEM_ORIGINS)
def origin_note(kind: str, **data: Any) -> dict[str, Any]:
"""run.origin 事件的数据:kind 加上调用方给的字段(app_id、app_version、spec_hash、entry / view、hook、
+61 -10
View File
@@ -23,7 +23,7 @@ from app.db.base import SessionLocal
from app.db.models import Approval, Run, RunEvent, Workflow
from app.engine.approval import read_decision
from app.engine.compiler import compile_graph, initial_state
from app.engine.context import NodeError, RunContext
from app.engine.context import NodeError, NodeRejected, RunContext
from app.engine.deadline import segment
from app.engine.replay import PROTOCOL, PROTOCOL_KEY, protocol_of
from app.engine.schema import GraphSpec, NodeType, loop_steps, topology_of, validate_graph
@@ -42,9 +42,18 @@ _EPHEMERAL = {EventType.LLM_TOKEN, EventType.LLM_THINKING_DELTA}
RUN_OPTIONS_KEY = "run_options"
#: 接着跑被拒的机器码:正式运行带图(F2),有结果未知、还没核实的写入
#: 接着跑被拒的机器码:正式运行带图(F2),有结果未知、还没核实的写入,被人驳回了的
FORMAL_GRAPH_LOCKED = "formal_graph_locked"
ACTION_UNRESOLVED = "action_unresolved"
RUN_REJECTED = "run_rejected"
#: 运行为什么结束在 cancelled(Run.end_reason,docs/design/next-round/00-总方案.md X9)
END_REJECTED = "rejected"
END_STOPPED = "stopped"
#: 被驳回的运行不能继续运行:从断点重跑会读到同一个驳回答复、再停在同一处
REJECTED_CONTINUE = "被驳回的运行不能继续运行:改好内容后重新发起"
#: 服务重启时还在排队的运行改成失败时写的话(run.error 和补发的 run.failed 是同一句)
QUEUED_AT_RESTART = "服务重启时这次运行仍在排队,没有可恢复的进度,请重新发起运行"
class ContinueRefused(ValueError):
@@ -94,6 +103,9 @@ class RunManager:
stranded = list((await session.execute(
select(Run.id, Run.last_seq).where(Run.status == "running")
)).all())
queued = list((await session.execute(
select(Run.id, Run.last_seq).where(Run.status == "queued")
)).all())
await session.execute(
update(Run)
.where(Run.status == "running")
@@ -102,7 +114,7 @@ class RunManager:
await session.execute(
update(Run)
.where(Run.status == "queued")
.values(status="failed", error="服务重启时这次运行仍在排队,没有可恢复的进度,请重新发起运行")
.values(status="failed", error=QUEUED_AT_RESTART)
)
await session.commit()
# 进程被强杀时连 server_shutdown 日志都来不及发,事件流停在半路。补上这一条,
@@ -113,6 +125,11 @@ class RunManager:
"level": "warn", "code": "server_shutdown",
"message": "服务上次未正常关闭,这次运行已中断;可从断点继续运行",
})
# 排队中的改成了失败,也补一条 run.failed:以前只改状态,时间线里没有失败原因这一行(05-已知遗留 (h))。
# 不写结束时间、不封存:它从没开始执行,同以前(定时通知按「没有开始时间」认出它,scheduler/webhook.py)
for run_id, last_seq in queued:
bus.set_seq(run_id, last_seq or 0)
await self._emit(run_id, EventType.RUN_FAILED, data={"error": QUEUED_AT_RESTART})
async def shutdown(self) -> None:
self._closing = True
@@ -505,8 +522,13 @@ class RunManager:
Approval.run_id == run_id, Approval.status.in_(["pending", "answered"])
).limit(1)
)).first() is not None
if run.status == "cancelled" and run.end_reason == END_REJECTED:
raise ContinueRefused(REJECTED_CONTINUE, code=RUN_REJECTED)
if not (run.status == "failed" or (run.status == "interrupted" and not has_pending)):
raise ValueError(_cannot_continue(run.status, has_pending))
if run.status == "failed" and await _rejected_before(session, run):
# 加 end_reason 之前被驳回的运行记的是 failed(封存不改写):接着跑会读到同一个驳回答复、再停一次
raise ContinueRefused(REJECTED_CONTINUE, code=RUN_REJECTED)
if graph is not None and run.run_class == "formal":
raise ContinueRefused(
"正式运行固定在发布的版本上,不能修改配置后继续运行。如需修改,请在画布上修改并发布新版本后重新发起",
@@ -613,7 +635,7 @@ class RunManager:
# 或者另一个取消先到了,这里就什么都不做
claimed = await session.execute(
update(Run).where(Run.id == run_id, Run.status == "interrupted")
.values(status="cancelled", finished_at=now)
.values(status="cancelled", end_reason=END_STOPPED, finished_at=now)
)
if claimed.rowcount != 1:
await session.rollback()
@@ -631,7 +653,7 @@ class RunManager:
bus.set_seq(run_id, await _seq_floor(session, run_id, run.last_seq))
# 和状态同一个事务提交,理由同 _finalize
event = self._event(run_id, EventType.RUN_CANCELLED, data={
"timing": timing, "actor": actor,
"reason": END_STOPPED, "timing": timing, "actor": actor,
"message": f"放弃了这次运行,{closed} 条待审批一并关闭" if closed
else "放弃了这次挂起的运行",
})
@@ -684,6 +706,8 @@ class RunManager:
#: 终态事件(run.failed / run.cancelled)等 _finalize 算完耗时再发,timing 才带得上
terminal: tuple[EventType, dict[str, Any]] | None = None
error_node: str | None = None
#: 收成 cancelled 的原因(Run.end_reason):rejected 人的驳回,stopped 手动停止
end_reason: str | None = None
#: 结果未知、挂起等核实的动作(ActionPending),收尾之后发 action.unknown、安排核实
pending_action: str | None = None
@@ -796,7 +820,9 @@ class RunManager:
else:
status = "cancelled"
error = "用户取消"
terminal = (EventType.RUN_CANCELLED, {"actor": self._stopped_by.pop(run_id, None)})
end_reason = END_STOPPED
terminal = (EventType.RUN_CANCELLED, {"reason": END_STOPPED,
"actor": self._stopped_by.pop(run_id, None)})
raise
except ActionPending as e:
# 写外部系统的结果未知(app/process/actions.py):不在节点里等,运行收成挂起、不封存,等核实之后
@@ -829,6 +855,12 @@ class RunManager:
"(AGENTLAB_MAX_GRAPH_STEPS)"
)
terminal = (EventType.RUN_FAILED, {"error": error})
except NodeRejected as e:
# 人的决定(驳回、超时驳回),不是故障:收成 cancelled,同手动停止走一样的收尾(关掉并行支路上的审批、
# 回收沙箱、关流、封存、不许继续运行)。原话照旧记在 error 上,在哪一步、谁、结论、备注进 run.cancelled
status, end_reason = "cancelled", END_REJECTED
error, error_node = str(e), e.node_id
terminal = (EventType.RUN_CANCELLED, rejected_data(e, spec))
except Exception as e: # noqa: BLE001
status = "failed"
error, terminal = _failure(e, spec)
@@ -840,7 +872,7 @@ class RunManager:
self._finalizing.add(run_id)
try:
await self._finalize(run_id, status, error, final_state, elapsed,
terminal=terminal, error_node=error_node)
terminal=terminal, error_node=error_node, end_reason=end_reason)
finally:
# 收尾被打断(关停时的取消、库连接出错)也得把自己摘掉。以前写在
# _finalize 后面,一被打断就漏掉,留下的任务让下一次 shutdown 的
@@ -944,6 +976,7 @@ class RunManager:
*,
terminal: tuple[EventType, dict[str, Any]] | None = None,
error_node: str | None = None,
end_reason: str | None = None,
) -> None:
usage = dict(state.get("usage") or {})
output = dict(state.get("output") or {})
@@ -1000,6 +1033,7 @@ class RunManager:
run.output = output
run.usage = usage
run.error_node_id = error_node
run.end_reason = end_reason
if status != "interrupted":
run.finished_at = now
if status in ("failed", "cancelled"):
@@ -1024,11 +1058,12 @@ class RunManager:
# 关停时挂起的也要让在线的客户端收尾:进程马上就没了,不说一声,
# 界面会一直以为它还在跑
await bus.close(run_id)
if app_id and status in ("succeeded", "failed"):
# 应用实例结束、失败:按应用订阅推送(app/notify/service.py)。放到后台,接收端慢不拖着收尾
if app_id and (status in ("succeeded", "failed") or end_reason == END_REJECTED):
# 应用实例结束、失败:按应用订阅推送(app/notify/service.py)。放到后台,接收端慢不拖着收尾。被驳回的走
# 「实例结束」(next-round X6):发起人要知道结果,订阅「实例失败」的人盯的是故障;手动停止的不推,同以前
from app.notify.service import notify_run, spawn
spawn(notify_run(run_id, "run.finished" if status == "succeeded" else "run.failed"))
spawn(notify_run(run_id, "run.failed" if status == "failed" else "run.finished"))
async def _seal(self, run_id: str) -> None:
"""对到此为止的全部事件算清单哈希,记下封到了哪一条。"""
@@ -1269,6 +1304,22 @@ def _trust_event(tool_trust: dict[str, str] | None) -> Any:
return "formal" if tool_trust.get(FORMAL_MARK) else snapshot_levels(tool_trust)
def rejected_data(exc: NodeRejected, spec: GraphSpec) -> dict[str, Any]:
"""被驳回的 run.cancelled 带什么:原因(rejected)、在哪一步(节点 id 和图上的名字)、哪种关卡、结论(驳回 / 超时
驳回)、谁、备注。进封存:谁在哪一步驳回、为什么,事后查得到。"""
node = spec.node_map().get(exc.node_id)
return {"reason": END_REJECTED, "node_id": exc.node_id, "label": node.title if node else exc.node_id,
"kind": exc.kind, "decision": exc.decision, "actor": exc.actor, "note": exc.note}
async def _rejected_before(session: Any, run: Run) -> bool:
"""加 end_reason 之前被人驳回、记成 failed 的运行:失败的那一步上最近一条处理完的待办是驳回(和实例页认驳回的是
同一份判断,apps/resolve.legacy_rejected)。"""
from app.apps.resolve import legacy_rejected
return await legacy_rejected(session, run)
def _failure(exc: BaseException, spec: GraphSpec) -> tuple[str, tuple[EventType, dict[str, Any]]]:
"""失败的首行报错(给人看)和 run.failed 的负载(带上定位和原始异常)。"""
if isinstance(exc, NodeError):
+99 -60
View File
@@ -1,11 +1,18 @@
"""核查花费账:双盲复算、反方审稿、审批质询里的模型调用,按本地日期记一本每日账。
"""模型调用的每日花费账:一本账一个键,按本地日期记,先预留、后找补(docs/design/next-round/01-效果度量与回归评测.md 做法 17)。
照结论句裁判的每日计数做(judge.py 的 SPEND_KEY、_charge、_settle):调用前按目录价估算、先预留,加上这笔会超过
每日上限就不记;调用后按实际用量找补差额。账落在设置表里,按本地日期滚动。
调用前按目录价估算、先预留,加上这笔会超过每日上限就不记;调用后按实际用量找补差额。账落在设置表里,按本地日期
滚动,不从设置接口进出(api/settings.py 只给只读的「今天花了多少」)。
**为什么和裁判分开记。** 核查是一个 agent 查库、来回好几轮,一次花的钱比裁判判一份报告多得多。记在同一本账上,
核查一多,当天的裁判就全因为「每日上限」判不了——报告的出具会因此降档,而降档的原因和这份报告毫无关系。两本账
各管各的上限,互不挤占。
**一套代码,几本账。** 以前核查(这个模块)和结论句裁判(judge.py 的 _charge、_settle)各写一份一模一样的读改写,评测
要是再抄一份就是第三份。现在都是 Ledger(key) 的实例,各记各的键:
- verify_spend:核查——双盲复算、反方审稿、审批质询里的模型调用;
- judge_spend:结论句裁判(实例在 engine/judge.py,键 judge_spend:日历跟着 judge._today,裁判的测试换它模拟跨天);
- eval_spend:评测(预留,第 3 波的评测执行接上;设置和接口在那时加)。
**为什么各记各的。** 核查是一个 agent 查库、来回好几轮,一次花的钱比裁判判一份报告多得多;评测一次几十道题。记在同一本
账上,一边一多,另一边当天就全因为「每日上限」做不了——报告的出具会因此降档,而降档的原因和这份报告毫无关系。几本账
各管各的上限,互不挤占。三本按同一个本地日期滚动(_today)。
目录里没有价格的模型估不出金额:只计次数(unpriced_calls),金额上限对它不起作用,由调用方照实说明。
"""
@@ -13,27 +20,26 @@ from __future__ import annotations
import asyncio
import weakref
from collections.abc import Callable
from datetime import datetime
from typing import Any
from app.db.base import SessionLocal
from app.db.models import Setting
#: 每日计数存在设置表的这一行:{date: 本地日期, usd, calls, unpriced_calls}。不从设置接口进出(api/settings.py)
SPEND_KEY = "verify_spend"
_locks: "weakref.WeakKeyDictionary[asyncio.AbstractEventLoop, asyncio.Lock]" = weakref.WeakKeyDictionary()
#: 每本账的读改写锁:{事件循环: {键: 锁}}。asyncio.Lock 不能跨循环用(测试每个用例一个循环),按循环各一组;同一个键
#: 同一把——几个核查并行时各自预留,不串行的话两边读到同一个余额,会一起越过上限
_locks: "weakref.WeakKeyDictionary[asyncio.AbstractEventLoop, dict[str, asyncio.Lock]]" = weakref.WeakKeyDictionary()
def _lock() -> asyncio.Lock:
"""读改写这一行时持的锁,按事件循环各一把(asyncio.Lock 不能跨循环用)。
几个核查并行时各自预留:不串行的话两边读到同一个余额,会一起越过上限。
"""
def _lock(key: str) -> asyncio.Lock:
loop = asyncio.get_running_loop()
lock = _locks.get(loop)
group = _locks.get(loop)
if group is None:
group = _locks[loop] = {}
lock = group.get(key)
if lock is None:
lock = _locks[loop] = asyncio.Lock()
lock = group[key] = asyncio.Lock()
return lock
@@ -42,57 +48,90 @@ def _today() -> str:
return datetime.now().astimezone().date().isoformat()
def _spend_of(row: Any) -> dict[str, Any]:
value = row.value if row is not None and isinstance(row.value, dict) else {}
today = _today()
if value.get("date") != today:
return {"date": today, "usd": 0.0, "calls": 0, "unpriced_calls": 0}
return {"date": today, "usd": float(value.get("usd") or 0.0), "calls": int(value.get("calls") or 0),
"unpriced_calls": int(value.get("unpriced_calls") or 0)}
class Ledger:
"""一本每日账,存在设置表 key 这一行:{date: 本地日期, usd, calls, unpriced_calls}。
today 是这本账的日历(测试换掉它来模拟跨天);不给就按本模块的 _today,调用时才取,换掉 _today 对没给 today 的几本
账都生效。"""
def __init__(self, key: str, *, today: Callable[[], str] | None = None) -> None:
self.key = key
self._clock = today
def _date(self) -> str:
return self._clock() if self._clock is not None else _today()
def _spend_of(self, row: Any) -> dict[str, Any]:
value = row.value if row is not None and isinstance(row.value, dict) else {}
today = self._date()
if value.get("date") != today:
return {"date": today, "usd": 0.0, "calls": 0, "unpriced_calls": 0}
return {"date": today, "usd": float(value.get("usd") or 0.0), "calls": int(value.get("calls") or 0),
"unpriced_calls": int(value.get("unpriced_calls") or 0)}
async def daily(self) -> dict[str, Any]:
"""今天(本地日期)这本账已经花了多少:{date, usd, calls, unpriced_calls}。估不出金额的调用只计次数。"""
async with SessionLocal() as session:
return self._spend_of(await session.get(Setting, self.key))
async def charge(self, amount: float, *, limit: float | None, priced: bool) -> str | None:
"""记一次调用、先按估算预留金额。加上这笔会超过每日上限就不记,返回 None;记下了返回记在哪天(找补时要用)。
limit 为 None 表示不限。估不出金额的调用(priced=False)照记次数,金额记 0,调用方应传 limit=None:
拿 0 去比上限,等于说它不花钱。
"""
async with _lock(self.key):
async with SessionLocal() as session:
row = await session.get(Setting, self.key)
spend = self._spend_of(row)
if limit is not None and spend["usd"] + amount > limit + 1e-12:
return None
spend["usd"] = round(spend["usd"] + amount, 8)
spend["calls"] += 1
spend["unpriced_calls"] += 0 if priced else 1
if row is None:
session.add(Setting(key=self.key, value=spend))
else:
row.value = spend
await session.commit()
return spend["date"]
async def settle(self, date: str, delta: float) -> None:
"""调用结束后按实际用量找补预留的差额。跨了天就不找补:预留记在前一天,今天从零算。"""
if not delta:
return
async with _lock(self.key):
async with SessionLocal() as session:
row = await session.get(Setting, self.key)
spend = self._spend_of(row)
if row is None or spend["date"] != date:
return
spend["usd"] = max(0.0, round(spend["usd"] + delta, 8))
row.value = spend
await session.commit()
#: 核查、评测各一本(裁判那本在 engine/judge.py 的 judge_spend)
verify_spend = Ledger("verify_spend")
eval_spend = Ledger("eval_spend")
#: 核查那本账的老名字(verify_agent、设置接口、已有测试从这里取)
SPEND_KEY = verify_spend.key
async def daily_spend() -> dict[str, Any]:
"""今天(本地日期)核查已经花了多少:{date, usd, calls, unpriced_calls}。估不出金额的调用只计次数。"""
async with SessionLocal() as session:
return _spend_of(await session.get(Setting, SPEND_KEY))
"""今天核查已经花了多少(verify_spend.daily)。"""
return await verify_spend.daily()
async def charge(amount: float, *, limit: float | None, priced: bool) -> str | None:
"""记一次调用、先按估算预留金额。加上这笔会超过每日上限就不记,返回 None;记下了返回记在哪天(找补时要用)。
limit 为 None 表示不限。估不出金额的调用(priced=False)照记次数,金额记 0,调用方应传 limit=None:
拿 0 去比上限,等于说它不花钱。
"""
async with _lock():
async with SessionLocal() as session:
row = await session.get(Setting, SPEND_KEY)
spend = _spend_of(row)
if limit is not None and spend["usd"] + amount > limit + 1e-12:
return None
spend["usd"] = round(spend["usd"] + amount, 8)
spend["calls"] += 1
spend["unpriced_calls"] += 0 if priced else 1
if row is None:
session.add(Setting(key=SPEND_KEY, value=spend))
else:
row.value = spend
await session.commit()
return spend["date"]
"""核查记一次调用(verify_spend.charge)。"""
return await verify_spend.charge(amount, limit=limit, priced=priced)
async def settle(date: str, delta: float) -> None:
"""调用结束后按实际用量找补预留的差额。跨了天就不找补:预留记在前一天,今天从零算。"""
if not delta:
return
async with _lock():
async with SessionLocal() as session:
row = await session.get(Setting, SPEND_KEY)
spend = _spend_of(row)
if row is None or spend["date"] != date:
return
spend["usd"] = max(0.0, round(spend["usd"] + delta, 8))
row.value = spend
await session.commit()
"""核查找补(verify_spend.settle)。"""
await verify_spend.settle(date, delta)
__all__ = ["SPEND_KEY", "charge", "daily_spend", "settle"]
__all__ = ["Ledger", "SPEND_KEY", "charge", "daily_spend", "eval_spend", "settle", "verify_spend"]
+6 -2
View File
@@ -758,6 +758,7 @@ async def run_verify_agent(
node_id: str = "",
data_versions: dict[str, Any] | Versions | None = Versions.UNSET,
masked_values: Any = None,
catalog_view: Any = None,
) -> VerifyOutcome:
"""让一个只读 agent 查库,按 output_schema 交回有出处的结果。永远不抛。
@@ -775,6 +776,8 @@ async def run_verify_agent(
- masked_values:调用方已经认出的遮罩值(engine/masking 的 _Secrets,审批质询、反方审稿收的报告快照里的值)。
查询工具从第一条查询起就按它们按值遮:agent 查到的备注、别的表里写着同一个号码时,模型看不到。随查询时遮掉的值
一起交回(VerifyOutcome.masked_values)。不改它。
- catalog_view:db_schema 给 agent 看哪一份数据目录(data/catalog_view.py)。运行里的节点传这次运行的视图
(catalog_view.of_run);不给是当前目录(质询接口)。
"""
state = _State(started=time.monotonic(), seen=SeenValues(
given=masked_values if masked_values is not None else _Secrets()))
@@ -782,7 +785,7 @@ async def run_verify_agent(
out = await _verify(state, spec=spec, system=system, task=task, sources=sources,
output_schema=output_schema, max_queries=max_queries, timeout_s=timeout_s,
max_cost_usd=max_cost_usd, daily_max_usd=daily_max_usd, max_steps=max_steps, emit=emit,
run_id=run_id, node_id=node_id, data_versions=data_versions)
run_id=run_id, node_id=node_id, data_versions=data_versions, catalog_view=catalog_view)
except Exception as e: # noqa: BLE001 - 永远不抛:库写不进去、建工具出错这类意外也落在返回值里
logger.warning("核查出错", exc_info=True)
out = state.outcome(error=f"核查出错({describe_exception(e)})")
@@ -798,6 +801,7 @@ async def run_verify_agent(
async def _verify(state: _State, *, spec: ModelSpec, system: str, task: str, sources: Iterable[str],
output_schema: dict[str, Any], max_queries: int | None, timeout_s: float | None,
max_cost_usd: float | None, daily_max_usd: float | None | Default, max_steps: int | None,
catalog_view: Any = None,
emit: Emit | None, run_id: str, node_id: str,
data_versions: dict[str, Any] | Versions | None) -> VerifyOutcome:
deadline = state.started + timeout_s if timeout_s is not None else None
@@ -834,7 +838,7 @@ async def _verify(state: _State, *, spec: ModelSpec, system: str, task: str, sou
# ---- 工具:只读的 db_schema / db_query,查询结果先遮再交给模型(遮掉的值并进 state.seen)
wanted = list(dict.fromkeys(s for s in sources if isinstance(s, str) and s))
ctx = ToolContext(run_id=run_id, node_id=node_id, data_versions=data_versions, emit=emit,
ctx = ToolContext(run_id=run_id, node_id=node_id, data_versions=data_versions, emit=emit, catalog_view=catalog_view,
extra={MASKED_VALUES_KEY: state.seen})
async with SessionLocal() as session:
tools = await build_datasource_tools([f"{p}{s}" for s in wanted for p in (SCHEMA_PREFIX, QUERY_PREFIX)], ctx,
+8 -4
View File
@@ -159,12 +159,16 @@ class TableKind:
@classmethod
async def load(cls, sources: list[Any], *, graph: Mapping[str, Any] | None = None,
picker: Picker | None = None) -> TableKind:
"""读各源的数据目录。graph 是现有工作流(改图时的原图、追问时上一轮的图),概览里列出它查过的表。"""
from app.tools.datasource import _catalog_of
picker: Picker | None = None, catalog_view: Any = None) -> TableKind:
"""读各源的数据目录。graph 是现有工作流(改图时的原图、追问时上一轮的图),概览里列出它查过的表。
读哪一份目录由 catalog_view 定(data/catalog_view.py:当前目录、目录快照、关;只用已核实的项),不给是当前目录。
查看结果里的目录和运行时 db_schema 给模型看的、自查用的是同一份——评测的「目录对照」量的才是整份贡献。"""
from app.data.catalog_view import CatalogView, read
view = CatalogView.parse(catalog_view)
seeds = graph_tables(graph, sources)
plans = [SourcePlan(source=s, entries=await _catalog_of(s) if introspect.table_names(s) else {},
plans = [SourcePlan(source=s, entries=await read(view, s) if introspect.table_names(s) else {},
seeds=seeds.get(s.name, [])) for s in sources]
return cls(plans, picker=picker)
+11 -3
View File
@@ -30,6 +30,7 @@ from app.core.config import settings
from app.db.base import init_db
from app.engine.runner import run_manager
from app.seed import seed_defaults
from app.web_security import SecurityHeaders
logging.basicConfig(
level=logging.INFO,
@@ -145,16 +146,20 @@ async def lifespan(_: FastAPI) -> AsyncIterator[None]:
async def _platform_startup() -> None:
"""应用平台的启动整理(BE-1):登录设置读进缓存;自定义工具请求头、MCP env 和请求头里的存量明文加密一遍(幂等);
存量的内网 MCP 服务改为绑定登记端点(只在升级后第一次启动时做);能力登记同步一次。后三样出错只记日志:最坏是登记
暂时不全,不该让服务起不来。"""
"""应用平台的启动整理(BE-1):登录设置、对外访问地址读进缓存;自定义工具请求头、MCP env 和请求头里的存量明文加密
一遍(幂等);存量的内网 MCP 服务改为绑定登记端点(只在升级后第一次启动时做);能力登记同步一次。后三样出错只记
日志:最坏是登记暂时不全,不该让服务起不来。"""
from app.auth import current_mode, load_settings
from app.capabilities import registry as capability_registry
from app.capabilities.secrets import seal_existing
from app.core import public_url
from app.db.base import SessionLocal
await load_settings(force=True)
logger.info("登录方式:%s", current_mode())
# 对外访问地址(core/public_url.py)同样读进缓存:中间件核对来源时不查库
if address := await public_url.refresh():
logger.info("对外访问地址:%s(%s)", address, "环境变量" if public_url.env_locked() else "设置")
try:
if sealed := await seal_existing():
logger.info("已加密 %d 处存量的明文密钥(自定义工具请求头、MCP 环境变量和请求头)", sealed)
@@ -192,6 +197,8 @@ app.add_middleware(
allow_methods=["*"],
allow_headers=["*"],
)
# 安全响应头(app/web_security.py):最后加、在最外层,CORS 预检和登录检查自己回的 401 / 403 也带上
app.add_middleware(SecurityHeaders)
# 路由级权限(§3.12)。关登录时这些依赖全部放行;开了登录,中间件先挡掉没登录的,这里再按角色和运行挡
_BUILDER = [require_role("builder")]
@@ -238,6 +245,7 @@ for router, guard in (
(identity_api.router, _ADMIN),
# 选人用的名录(用户名、显示名、是否停用、组名):岗位绑定、应用授权由搭建者做,只读
(identity_api.directory_router, _BUILDER),
(identity_api.candidates_router, _USER),
# 能力登记、端点:搭建者只读,改动在接口里要管理员
(capabilities_api.router, _BUILDER),
# 普通用户建应用:个人应用的接口共用 /api/apps 前缀,要在 apps_api 之前登记;蓝图助手共用 /api/copilot 前缀,
+102 -21
View File
@@ -7,15 +7,29 @@
同一个渠道只推一次。待办改派、升级、催办都是「有一条待办要你处理」,订阅了 task.opened 的渠道一并收到。
消息里只放标题、状态、链接(P4,沿用定时任务 Webhook 的规矩):不放报告里的数字,待办的标题用节点在图上的名字
(搭图时起的),不用按运行数据渲染出来的那一句。链接要设了对外访问地址(设置里的 notify)才有。
(搭图时起的),不用按运行数据渲染出来的那一句。链接要有对外访问地址(环境变量或设置,core/public_url.py)才有。
推送放在后台(spawn):接收端挂住时超时加重试要几十秒,不能拖着开待办、运行收尾。测试用 drain 等它们推完。
事件只在这里登记(EVENTS,总方案 X6):每个事件一行,名字、界面上的叫法(前端 lib/terms.ts 的 NOTIFY_TEXT.eventLabel
逐字一致,test_notify_events 对照)、推给哪几条路(scopes:global 全局渠道自己订阅、user 相关的人自己的个人渠道、app 应用
版本 spec.notify 的订阅)。_channels 只走登记了的路,没登记的事件推不出去;接口按 scopes 收订阅,设置页和应用的通知按
GET /api/notify/events 列。scopes 是空的,是登记了、还没有推送的事件:不收订阅、界面上不列——勾了却收不到,就是「看着
生效、其实没生效」(个人渠道勾了实例结束、失败却收不到,04-对外集成与触达 B1,就是这么错的)。本轮第 1 波把六个新事件
一次登记好;实现推送的那一块写自己的 notify_*,只改自己那一行的 scopes,不再改这张表的别处、也不再改前端的叫法表:几块
并行改同一处必然冲突。以前存下的订阅不按 scopes 重新核对(api/notify._validated 只核对新勾的)。
消息的结构,新事件一律照这个写(P4、D9):标题、状态、条数、链接,外加编号(#前 6 位)和搭建时起的名字(应用、工作流、
节点在图上的名字,题集、数据源、访问密钥的名字)。条数说的是有几件事要处理(几个变化、通过几题、几条待办),可以放。
不放:报告和答案里的数(金额、比例、差值、期望值),表名、列名、取值(码值、样例行),SQL,按运行数据渲染出来的句子。
接收端是群机器人、外部系统,消息出了这个服务就不归遮罩管了。
"""
from __future__ import annotations
import asyncio
import logging
from collections.abc import Coroutine
from dataclasses import dataclass
from typing import Any
from sqlalchemy import select
@@ -26,12 +40,60 @@ from app.notify.core import Message, Outcome, host_of
logger = logging.getLogger(__name__)
#: 可以订阅的事件
EVENTS = ("task.opened", "run.finished", "run.failed", "action.unknown", "review.done")
EVENT_LABEL = {"task.opened": "有待办派给我", "run.finished": "实例完成", "run.failed": "实例失败",
"action.unknown": "写入结果未知",
# 个人应用提交审核之后,搭建者发布了或退回了(docs/design/normal-user-apps.md N7)
"review.done": "我提交审核的应用有结果"}
@dataclass(frozen=True)
class Event:
"""一个可订阅的通知事件(见模块说明「事件只在这里登记」)。"""
#: 界面上的叫法:和前端 NOTIFY_TEXT.eventLabel 同一句
label: str
#: 推给哪几条路(_channels):global 全局渠道自己订阅,user 相关的人自己的个人渠道,app 应用版本 spec.notify 的订阅。
#: 空的:登记了、还没有推送
scopes: tuple[str, ...]
_EVERY = ("global", "user", "app")
#: 渠道范围在说法里怎么叫(同前端 NOTIFY_TEXT.scopes)
SCOPE_TEXT = {"global": "全局", "user": "个人", "app": "应用"}
#: 可以订阅的事件,只在这里登记
EVENTS: dict[str, Event] = {
# 待办派出、改派、升级、催办(_AS):全局渠道、派给的人自己的个人渠道、应用的订阅
"task.opened": Event("待办打开", _EVERY),
# 应用实例结束、失败:个人渠道是发起人自己的(04-对外集成与触达 B1)
"run.finished": Event("运行结束", _EVERY),
"run.failed": Event("运行失败", _EVERY),
# 写入外部系统的结果未知:个人渠道是核实待办派给的人
"action.unknown": Event("动作结果未知", _EVERY),
# 个人应用提交审核之后,搭建者发布了或退回了(docs/design/normal-user-apps.md N7):发给提交人,不归哪个应用
"review.done": Event("我提交审核的应用有结果", ("global", "user")),
# ---- 本轮登记、还没有推送(总方案 X6):实现推送的那一块只改自己这一行的 scopes,消息照模块说明的规矩 ----
# 数据源有要处理的结构 / 取值变化(03 B3):只推 severity 为 action 的,按源汇总;源名、各级别条数、链接
"source.changed": Event("数据源有变化要处理", ()),
# 口径升版后召回重算,数真变了(02 第 7 项):答案的主人有 recall 待办;口径名、版本、链接,不放新旧值和差值
"ledger.recall": Event("口径升版,答案的数变了", ()),
# 定时评测有题从通过变成不通过 / 评测没跑完(01 第 15 项):题集名、通过数、退步题数、链接;不放期望值和答案里的数
"eval.regressed": Event("评测有退步", ()),
"eval.failed": Event("评测失败", ()),
# 每天一条待办汇总(04 B3):只推个人渠道;条数、最早的时限、最多 5 条(节点在图上的名字)、去待办箱的链接
"task.digest": Event("每天的待办汇总", ()),
# 访问密钥到期前 7 天、1 天各一次(04 B3):个人密钥推本人,服务账号的推全局渠道;密钥名、到期日、链接
"key.expiring": Event("访问密钥快到期", ()),
}
EVENT_LABEL = {name: event.label for name, event in EVENTS.items()}
def subscribe_problem(event: str, scope: str) -> str | None:
"""范围是 scope 的渠道能不能订阅 event:不能的回一句为什么(接口原样回 422),能的是 None。"""
spec = EVENTS.get(event)
if spec is None:
offered = "、".join(e.label for e in EVENTS.values() if scope in e.scopes)
return f"不认识的通知事件:{event}。可选:{offered}"
if not spec.scopes:
return f"「{spec.label}」的通知还没有推送,暂时不能订阅"
if scope not in spec.scopes:
return f"「{spec.label}」不推给范围是「{SCOPE_TEXT.get(scope, scope)}」的渠道"
return None
#: 这几种也是「有一条待办要你处理」,按 task.opened 的订阅推
_AS = {"task.reassigned": "task.opened", "task.escalated": "task.opened", "task.reminded": "task.opened"}
_TASK_HEAD = {"task.opened": "新的待办", "task.reassigned": "待办已改派给你", "task.escalated": "待办已超时,升级处理",
@@ -91,15 +153,19 @@ def generic_body(message: dict[str, Any], msg: Message) -> dict[str, Any]:
async def _base_url(session: Any) -> str | None:
from app.api.settings import notify_settings
"""拼链接用的对外访问地址:全站同一个(环境变量优先,其次设置),手里有会话就按库里的值对齐缓存。"""
from app.core import public_url
return (await notify_settings(session))["public_base_url"]
return await public_url.refresh(session)
async def _channels(session: Any, event: str, run: Run | None, users: list[str]) -> list[NotifyChannel]:
wanted = _AS.get(event, event)
if (registered := EVENTS.get(wanted)) is None:
raise ValueError(f"通知事件 {wanted} 没有登记:先在 notify/service.EVENTS 登记,再推送")
paths = registered.scopes
found: dict[str, NotifyChannel] = {}
if run is not None and run.app_id and run.app_version is not None:
if "app" in paths and run is not None and run.app_id and run.app_version is not None:
row = (await session.execute(select(AppVersion.spec, AppVersion.bindings).where(
AppVersion.app_id == run.app_id, AppVersion.version == run.app_version))).first()
spec, bindings = (row[0], row[1]) if row is not None else ({}, {})
@@ -115,11 +181,12 @@ async def _channels(session: Any, event: str, run: Run | None, users: list[str])
for row in (await session.execute(select(NotifyChannel).where(
NotifyChannel.id.in_(subscribed), NotifyChannel.enabled.is_(True)))).scalars():
found[row.id] = row
for row in (await session.execute(select(NotifyChannel).where(
NotifyChannel.scope == "global", NotifyChannel.enabled.is_(True)))).scalars():
if wanted in (row.events or []):
found.setdefault(row.id, row)
if users:
if "global" in paths:
for row in (await session.execute(select(NotifyChannel).where(
NotifyChannel.scope == "global", NotifyChannel.enabled.is_(True)))).scalars():
if wanted in (row.events or []):
found.setdefault(row.id, row)
if "user" in paths and users:
owners = list((await session.execute(select(User.id).where(User.username.in_(users)))).scalars())
if owners:
for row in (await session.execute(select(NotifyChannel).where(
@@ -201,22 +268,36 @@ async def notify_task(task_id: str, event: str, *, users: list[str]) -> int:
async def notify_run(run_id: str, event: str) -> int:
"""应用实例结束、失败(run.finished / run.failed)。只推应用发起的运行。"""
"""应用实例结束、失败(run.finished / run.failed)。只推应用发起的运行。
状态那一句取自实例页的说法(apps/resolve.instance_state),两边说同一句话:被驳回的(cancelled + end_reason=rejected)
走「实例结束」、写「「付款」未确认(被驳回)」;走到「已退回」成果的写「已退回」,不再一律写「已完成」。ok 只在真的
办成了(code 是 succeeded)时为真。
发起人是认证过的身份(run.origin.actor_verified)时,他自己订阅了这件事的个人渠道也收得到(「我发起的实例完成 / 失败」,
04-对外集成与触达 B1)。以前这里给 _channels 的 users 是空的,个人渠道勾了这两项永远收不到。自报的署名不算:关登录
时谁都能填别人的名字。"""
from app.apps.resolve import _trails, instance_state
async with SessionLocal() as session:
run = await session.get(Run, run_id)
if run is None or not run.app_id:
return 0
channels = await _channels(session, event, run, [])
verified = bool((run.origin or {}).get("actor_verified")) and bool(run.started_by)
channels = await _channels(session, event, run, [run.started_by] if verified else [])
if not channels:
return 0
base = await _base_url(session)
where = await _where(session, run)
state = instance_state(run, (await _trails(session, [run]))[run.id])
finished = event == "run.finished"
lines = [("状态", "已完成" if finished else "失败"), ("实例", f"#{run_id[:6]}")]
if not finished and run.error_node_id:
ok = state["code"] == "succeeded"
lines = [("状态", state["text"]), ("实例", f"#{run_id[:6]}")]
if not finished and run.error_node_id and state["code"] == "failed":
lines.append(("失败节点", f"「{_label(run.graph, run.error_node_id)}」"))
message = {"event": event, "status": run.status, "ok": finished,
"title": f"{where or '应用'}的实例{'已完成' if finished else '失败'}", "lines": lines,
head = ("已完成" if ok else "已结束") if finished else ("已结束" if state["code"] == "cancelled" else "失败")
message = {"event": event, "status": run.status, "ok": ok,
"title": f"{where or '应用'}的实例{head}", "lines": lines,
"links": [("查看实例", f"{base}/apps/{run.app_id}/i/{run_id}")] if base else [],
"subject": {"run_id": run_id, "app_id": run.app_id}}
return await _push(channels, message, subject_kind="run", subject_id=run_id, event=event, run_id=run_id,
+3
View File
@@ -181,6 +181,9 @@ PLAIN: dict[str, Plain] = {
"plan.trial_failed": Plain(
"最近一次试运行失败了", "用真数运行出了错,这样发布出去,每次出数都会是同一个错。", _Y,
"回到试运行看出错的原因(可以交给助手修),修好后再用真数试一次。"),
"plan.trial_rejected": Plain(
"最近一次试运行被驳回、没有走完", "那是你(或审批人)的决定,不是出错;驳回之后的步骤没用真数看过。", _Y,
"要看完整结果,再试运行一次并批准;确认不需要的话可以直接发布。"),
"plan.untrialed": Plain(
"还没用真数试运行过", "没看过真数就发布,可能不是你要的结果。", _Y, "先试运行一次,看过结果再发布。"),
"plan.trial_differs": Plain(
+5 -4
View File
@@ -572,17 +572,18 @@ async def _continue(run_id: str, actor: str | None) -> Any:
return None
async def tick(now: datetime | None = None) -> int:
"""调度循环每一拍:对登记了 lookup 的结果未知动作带退避地查询。返回查了几条。"""
async def tick(now: datetime | None = None) -> list[str]:
"""调度循环每一拍:对登记了 lookup 的结果未知动作带退避地查询。交回这一拍查过的动作记录 id(同 tasks.tick)。"""
now = now or utcnow()
async with SessionLocal() as session:
due = list((await session.execute(select(ActionRecord.id, ActionRecord.next_lookup_at).where(
ActionRecord.state == "unknown", ActionRecord.next_lookup_at.is_not(None),
ActionRecord.next_lookup_at <= now))).all())
done = 0
done: list[str] = []
for action_id, claimed_at in due:
try:
done += await _lookup(action_id, claimed_at, now)
if await _lookup(action_id, claimed_at, now):
done.append(action_id)
except Exception: # noqa: BLE001 - 一条出错不拖住这一拍的其他动作
logger.exception("动作 %s 的结果查询失败,下一拍再试", action_id)
return done
+27 -5
View File
@@ -406,6 +406,22 @@ def _check_hash(card: dict[str, Any], given: Any, *, approving: bool) -> None:
status_code=422, code="payload_hash_mismatch")
async def _check_principals(session: AsyncSession, task: Task, principal: Any, values: dict[str, Any]) -> None:
"""待办表单里的选人控件:开了登录时按候选池核对(同入口表单,auth/directory.principal_errors)。不属于应用的待办
(试运行、画布)没有池,不核对。"""
from app.auth.directory import principal_errors
if not task.app_id or getattr(principal, "mode", "off") == "off":
return
app = await session.get(App, task.app_id)
if app is None:
return
bindings = await session.scalar(select(AppVersion.bindings).where(
AppVersion.app_id == task.app_id, AppVersion.version == task.app_version)) if task.app_version else None
if errors := await principal_errors(session, app, bindings, task.form, values, principal.mode):
raise FormInvalid(errors)
def _validated(form: dict[str, Any], values: dict[str, Any]) -> dict[str, Any]:
from app.apps.forms import validate_form
@@ -491,6 +507,8 @@ async def respond(task_id: str, principal: Any, *, decision: Any, values: dict[s
rows = (await _assignments(session, [task.id])).get(task.id, [])
mine = _pick(task, rows, principal)
response, verdict = _engine_response(task, approval, decision, values, note, payload_hash)
if task.form and verdict in ("approve", "submit") and values:
await _check_principals(session, task, principal, values)
response = await _bind_files(session, task, principal, response)
now = utcnow()
marked = await session.execute(
@@ -761,8 +779,9 @@ async def _notify(task_id: str, event: str, users: list[str]) -> int:
# --------------------------------------------------------------------------
async def tick(now: datetime | None = None) -> int:
"""调度循环每一拍:提醒、升级、超时驳回(永不自动批准)。返回处理的条数。试运行的待办不处理(只显示时限)。"""
async def tick(now: datetime | None = None) -> list[str]:
"""调度循环每一拍:提醒、升级、超时驳回(永不自动批准)。交回这一拍处理过的待办 id(生效可查:办了哪几条,
不只是几条;测试按自己造的那条断言,不受同一个库里别的待办影响)。试运行的待办不处理(只显示时限)。"""
now = now or utcnow()
action = func.json_extract(Task.on_timeout, "$.action")
trial = select(Run.id).where(Run.origin_kind == TRIAL_ORIGIN)
@@ -773,10 +792,11 @@ async def tick(now: datetime | None = None) -> int:
and_(Task.due_at.is_not(None), Task.due_at <= now, action == "reject"),
and_(Task.due_at.is_not(None), Task.due_at <= now, action == "escalate", Task.escalation_level == 0),
)).order_by(Task.due_at))).scalars())
done = 0
done: list[str] = []
for task_id in due:
try:
done += await _on_timeout(task_id, now)
if await _on_timeout(task_id, now):
done.append(task_id)
except Exception: # noqa: BLE001 - 一条出错不拖住这一拍的其他待办
logger.exception("待办 %s 的超时处理失败,下一拍再试", task_id)
return done
@@ -838,7 +858,9 @@ async def _on_timeout(task_id: str, now: datetime) -> int:
from app.engine.runner import run_manager
try:
await run_manager.resume(run_id, {"approved": False, "note": EXPIRED_NOTE}, approval_id=approval_id)
# expired:节点据此把这次驳回记成超时驳回(engine/approval.Decision.expired → NodeRejected.decision)
await run_manager.resume(run_id, {"approved": False, "note": EXPIRED_NOTE, "expired": True},
approval_id=approval_id)
except (ValueError, KeyError):
logger.warning("待办 %s 超时驳回时运行已无法恢复", task_id)
async with SessionLocal() as session:
+11 -5
View File
@@ -18,6 +18,7 @@ engine/nodes/llm.py(cite_fields 的抽取)、data/recipe_ai.py 各有一份
from __future__ import annotations
from collections.abc import Awaitable, Callable, Hashable
from contextlib import aclosing
from typing import Any
#: 服务方嫌额度太大时报错里会出现的字样:超过模型的输出上限(DeepSeek chat 系只到 8K),
@@ -167,19 +168,24 @@ class BudgetFallback:
return await (await self._current()).ainvoke(messages, *args, **kwargs)
async def astream(self, messages: Any, *args: Any, **kwargs: Any):
# 里面那条流用 aclosing 包着:调用方读到一半不读了(助手读到 done、look 就关流),关这一层时里面那条也当场关掉、
# 模型那一次调用当场收尾(LangChain 的回调这时把已经收到的用量交出来,providers/usage.py)。不包的话里面那条要等
# 垃圾回收才关,收尾落在调用方算完账之后
used = await self._current()
started = False
try:
async for chunk in used.astream(messages, *args, **kwargs):
started = True
yield chunk
async with aclosing(used.astream(messages, *args, **kwargs)) as stream:
async for chunk in stream:
started = True
yield chunk
return
except Exception as e:
if started or not self._can_fall_back(used) or not self._switch.rejected(e):
raise
self._switch.fall_back()
async for chunk in (await self._current()).astream(messages, *args, **kwargs):
yield chunk
async with aclosing((await self._current()).astream(messages, *args, **kwargs)) as stream:
async for chunk in stream:
yield chunk
def _derived(self, make: Callable[[Any], Any]) -> BudgetFallback:
"""对大小两个模型做同一件事(绑工具、要结构化输出),包成共用开关的一层。小的还没建时,建好再做。"""
+204
View File
@@ -0,0 +1,204 @@
"""模型用量计量:助手建图(含挑表、接着写、自查修正)和复核这两段花了多少(细案 01 做法 3)。
**为什么要。** 运行按 llm.end 事件记用量,问数据一轮里另外两块——助手建图、复核——一直不记:这一轮的花费是低报的,
评测也算不出按角色的成本。在每个调用点手写累计(生成、挑表、退回小额度重来、复核各一处)迟早漏掉一处,所以用 LangChain
的回调在模型这一层统一截:和 langchain_core.callbacks.get_usage_metadata_callback 同一个机制(一个登记过的上下文变量,
模型每次调用时把它挂成回调),按模型累计 input / output tokens 和调用次数。
**为什么不直接用 get_usage_metadata_callback。** 它是一个 with 块,块里设上下文变量、出块时按 token 复原。生成过程是一个
async generator:with 块跨着 yield 的话,变量在两次 yield 之间留在调用方的上下文里——调用方这时去发起运行,运行的任务
会继承它,运行里的调用就被重复计一遍;调用方中途不迭代了、在别的任务里收尾,复原还会报「token 不属于这个上下文」。
这里只在每一步 __anext__ 期间设上变量、交出操作之前复原(metered):生成过程里起的任务(心跳那一层每一步一个任务)
在创建时拷走上下文,照样计到;调用方自己的代码一行都不在计量里。
**金额。** 按 providers/catalog.estimate_cost 算,和运行的 llm.end 一个口径。估不出的调用记 unpriced_calls:模型不在价格
目录里,或者用量没报全而目录价又不是 0。估不出的不当成 0 元,调用方据此写「≥」或「金额无法估算」。用量没报全有两种:
回复没带用量(有的网关流式不给);读到一半就不读了——生成过程读到 done、look、ask 那一行就关掉模型的流(后面的闲话不要、
也不再为它付钱),用量在流的末尾才报全(Anthropic 的输出 token 在最后一条,OpenAI 兼容系开了 stream_usage 也在最后),
关掉的那一次只拿得到已经报了的部分。这一次照样计次数、计已报的 token,记成估不出,不悄悄少算。
模型名按发起调用时的参数取(invocation_params 的 model / model_name,就是接入里配的模型 id,价格目录按它查),取不到再
看回复里的 model_name。
"""
from __future__ import annotations
import threading
from collections.abc import AsyncIterator, Iterator
from contextlib import contextmanager
from contextvars import ContextVar
from typing import Any
from uuid import UUID
from langchain_core.callbacks import BaseCallbackHandler
from langchain_core.tracers.context import register_configure_hook
#: 取不到模型名的调用记在这个名下
UNKNOWN_MODEL = "unknown"
class Meter(BaseCallbackHandler):
"""一次计量:按模型累计用量。线程安全(同步模型的回调可能在线程里来)。"""
#: 同步模型在线程里调用时也要回调
run_inline = True
def __init__(self) -> None:
super().__init__()
self._lock = threading.Lock()
self._pending: dict[UUID, str] = {}
self._models: dict[str, dict[str, int]] = {}
# ------------------------------------------------------------------ 回调
def _started(self, run_id: UUID, serialized: dict[str, Any] | None, kwargs: dict[str, Any]) -> None:
params = kwargs.get("invocation_params") or {}
model = params.get("model") or params.get("model_name") or params.get("model_id")
if not model and isinstance(serialized, dict):
model = (serialized.get("kwargs") or {}).get("model") or (serialized.get("kwargs") or {}).get("model_name")
with self._lock:
self._pending[run_id] = str(model) if model else ""
def on_chat_model_start(self, serialized: dict[str, Any], messages: Any, *, run_id: UUID,
**kwargs: Any) -> None:
self._started(run_id, serialized, kwargs)
def on_llm_start(self, serialized: dict[str, Any], prompts: Any, *, run_id: UUID, **kwargs: Any) -> None:
self._started(run_id, serialized, kwargs)
def on_llm_end(self, response: Any, *, run_id: UUID, **kwargs: Any) -> None:
message = None
try:
message = response.generations[0][0].message
except (AttributeError, IndexError, TypeError):
pass
usage = getattr(message, "usage_metadata", None) or None
meta = getattr(message, "response_metadata", None) or {}
with self._lock:
model = self._pending.pop(run_id, "") or str(meta.get("model_name") or meta.get("model") or "") \
or UNKNOWN_MODEL
row = self._models.setdefault(model, {"input_tokens": 0, "output_tokens": 0, "calls": 0, "unreported": 0})
row["calls"] += 1
if usage:
row["input_tokens"] += int(usage.get("input_tokens") or 0)
row["output_tokens"] += int(usage.get("output_tokens") or 0)
else:
row["unreported"] += 1
def on_llm_error(self, error: BaseException, *, run_id: UUID, **kwargs: Any) -> None:
"""流读到一半停了(生成过程读到 done / look / ask 就关流,或者连接断了):LangChain 照样交来已经收到的那几段。
有收到东西的照计一次、记已报的 token,算用量没报全;一段都没收到的(请求刚发出去就被拒了)不计。"""
from langchain_core.outputs import ChatGenerationChunk
response = kwargs.get("response")
partial = None
try:
first = response.generations[0][0] if response is not None and response.generations[0] else None
partial = first.message if isinstance(first, ChatGenerationChunk) else None
except (AttributeError, IndexError, TypeError):
partial = None
with self._lock:
model = self._pending.pop(run_id, "")
if partial is None:
return
meta = getattr(partial, "response_metadata", None) or {}
model = model or str(meta.get("model_name") or meta.get("model") or "") or UNKNOWN_MODEL
row = self._models.setdefault(model, {"input_tokens": 0, "output_tokens": 0, "calls": 0, "unreported": 0})
row["calls"] += 1
row["unreported"] += 1
reported = getattr(partial, "usage_metadata", None) or {}
row["input_tokens"] += int(reported.get("input_tokens") or 0)
row["output_tokens"] += int(reported.get("output_tokens") or 0)
# ------------------------------------------------------------------ 合并、汇总
def merge(self, other: Meter) -> None:
"""把另一次计量并进来(嵌套的计量出块时并进外层)。"""
with other._lock:
rows = {k: dict(v) for k, v in other._models.items()}
with self._lock:
for model, row in rows.items():
mine = self._models.setdefault(model, dict.fromkeys(row, 0))
for key, value in row.items():
mine[key] = mine.get(key, 0) + value
def summary(self) -> dict[str, Any]:
"""{by_model: {模型: {input_tokens, output_tokens, calls, cost_usd, unpriced_calls}}, input_tokens, output_tokens,
calls, cost_usd, unpriced_calls}。cost_usd 只是估得出的那部分。"""
from app.providers.catalog import estimate_cost, find_pricing
with self._lock:
rows = {k: dict(v) for k, v in self._models.items()}
by_model: dict[str, dict[str, Any]] = {}
for model, row in sorted(rows.items()):
pricing = find_pricing(model)
free = pricing is not None and not any(pricing.values())
if pricing is None:
unpriced = row["calls"]
else:
unpriced = 0 if free else row["unreported"]
by_model[model] = {"input_tokens": row["input_tokens"], "output_tokens": row["output_tokens"],
"calls": row["calls"],
"cost_usd": round(estimate_cost(model, row["input_tokens"], row["output_tokens"]), 6),
"unpriced_calls": unpriced}
total = {key: sum(r[key] for r in by_model.values())
for key in ("input_tokens", "output_tokens", "calls", "unpriced_calls")}
return {"by_model": by_model, **total, "cost_usd": round(sum(r["cost_usd"] for r in by_model.values()), 6)}
#: 计量挂在这个上下文变量上(inheritable:模型调用时 LangChain 按它把 Meter 挂成回调,子任务拷走上下文时一并带走)
_CURRENT: ContextVar[Meter | None] = ContextVar("agentlab_usage_meter", default=None)
register_configure_hook(_CURRENT, inheritable=True)
def meter() -> Meter:
"""新的一次计量。用 metering(一段代码)或 metered(一个生成过程)把它挂上。"""
return Meter()
def current() -> Meter | None:
return _CURRENT.get()
@contextmanager
def metering(m: Meter) -> Iterator[Meter]:
"""这一段代码里的模型调用计进 m。只给不跨 yield 的一段用(复核那一次调用);生成过程用 metered。
嵌套的:出块时并进外层的那一次,外层照样看得到。复原用 set 而不是按 token reset:在别的任务里收尾也不报错。"""
outer = _CURRENT.get()
_CURRENT.set(m)
try:
yield m
finally:
_CURRENT.set(outer)
if outer is not None and outer is not m:
outer.merge(m)
async def metered(source: AsyncIterator[Any], m: Meter) -> AsyncIterator[Any]:
"""迭代一个生成过程,它的模型调用计进 m:每一步 __anext__ 期间挂上,交出操作之前复原(见模块开头)。
开始时外层已经有一次计量的(评测把一整题的助手用量记成一份):结束时把 m 并进去,同 metering。"""
enclosing = _CURRENT.get()
iterator = source.__aiter__()
try:
while True:
outer = _CURRENT.get()
_CURRENT.set(m)
try:
item = await iterator.__anext__()
except StopAsyncIteration:
return
finally:
_CURRENT.set(outer)
yield item
finally:
if enclosing is not None and enclosing is not m:
enclosing.merge(m)
def empty() -> dict[str, Any]:
"""一次调用都没有时的汇总(同 Meter().summary())。"""
return Meter().summary()
__all__ = ["Meter", "UNKNOWN_MODEL", "current", "empty", "meter", "metered", "metering"]
+3 -1
View File
@@ -320,7 +320,8 @@ def build_report(
) -> dict[str, Any]:
"""页面里的数据(report-data)。
- run:运行记录的几样(id、workflow_id、workflow_name、run_class、status、started_at、finished_at)
- run:运行记录的几样(id、workflow_id、workflow_name、run_class、status、end_reason、started_at、finished_at);
end_reason 分开同是 cancelled 的被驳回和手动停止(05-已知遗留 (a))
- report_node / doc_artifact / doc_hash_ok:选中的报告节点、它的文档工件、取回时哈希核对的结果
- doc:报告文档(正文的服务端渲染另走 render_body,这里不给:数据块只装点开数字要用的东西)
- seal:{_sealed 的 seal 字段里的几样}(sealed、ok、manifest_hash、manifest_seq)
@@ -334,6 +335,7 @@ def build_report(
"meta": {
"run_id": run.get("id"), "workflow_id": run.get("workflow_id"),
"workflow_name": display_name(run), "run_class": run.get("run_class"), "status": run.get("status"),
"end_reason": run.get("end_reason"),
"started_at": _iso(run.get("started_at")), "finished_at": _iso(run.get("finished_at")),
"report": {"node_id": report_node, "label": (nodes.get(report_node) or {}).get("label") or report_node},
"doc_artifact": doc_artifact, "doc_hash_ok": doc_hash_ok,
+3 -2
View File
@@ -25,7 +25,7 @@
metaRun: '运行 ID', metaClass: '运行类别', metaStatus: '运行状态', metaReport: '报告节点',
metaStarted: '开始', metaFinished: '结束', metaExported: '导出时间',
classLabel: { formal: '正式运行', exploratory: '探索运行' },
statusLabel: { succeeded: '已完成', failed: '已失败', cancelled: '已取消', running: '进行中', waiting: '等待审批' },
statusLabel: { succeeded: '已完成', failed: '已失败', cancelled: '已取消', rejected: '被驳回', running: '进行中', waiting: '等待审批' },
docHashBad: '这份报告文档与哈希不一致(疑似被修改),正文仅供参考,其中的口径和出处未经核对。',
sealNone: '这次运行导出时尚未封存:下列出处来自导出那一刻的系统记录,未经封存核对。',
sealOk: (hash) => `运行已封存,清单哈希 ${hash}。要向系统外的人证明文件未被修改,请导出「可验证报告」(记录页 · 凭证详情),拿封存哈希去系统对照。`,
@@ -147,7 +147,8 @@
$('meta').replaceChildren(...[...[
[T.metaRun, h('code', {}, String(meta.run_id || ''))],
[T.metaClass, label(T.classLabel, meta.run_class)],
[T.metaStatus, label(T.statusLabel, meta.status)],
// 被驳回(cancelled + end_reason=rejected)照运行记录写「被驳回」,不写成手动停止的「已取消」
[T.metaStatus, label(T.statusLabel, meta.status === 'cancelled' && meta.end_reason === 'rejected' ? 'rejected' : meta.status)],
[T.metaReport, String(meta.report?.label || meta.report?.node_id || NONE)],
[T.metaStarted, when(meta.started_at)],
[T.metaFinished, when(meta.finished_at)],
+3 -1
View File
@@ -130,7 +130,9 @@ async def process_due(now: datetime | None = None) -> None:
for name, step in (("待办的时限处理", tasks.tick), ("写入结果的查询", actions.tick),
("没有提交的上传文件清理", files_store.cleanup_tick)):
try:
await step(now)
done = await step(now)
if isinstance(done, list) and done:
logger.info("%s:这一拍处理了 %d 条", name, len(done))
except Exception: # noqa: BLE001
logger.exception("%s这一拍出错,下一拍照常进行", name)
+78 -31
View File
@@ -3,6 +3,11 @@
调度循环(loop.py)到点调 fire_once,接口的「立即运行」也调它:重叠跳过、渲染输入、记触发、发起运行这一套
只有一份。发起本身走 engine/launch.py 的 launch_run,和 HTTP 发起是同一套规则——正式运行只能从当前发布版本
发起,这一条不会在定时任务这边被绕过。
**定时的目标**(docs/design/next-round/00-总方案.md X4:定时只有一套):一个定时任务指向一个目标——工作流、评测题集
(第 3 波)或数据源维护(第 3 波)。认领、补跑、重叠跳过、触发记录、通知都只有这一套,fire_once 按目标种类分派。
老表的 workflow_id 是非空约束,只加不改的迁移放宽不了:目标不是工作流时它存空串。这条规矩只在 target_of /
target_columns 这一处,别处读写目标都经它们,不直接碰这两列。
"""
from __future__ import annotations
@@ -64,6 +69,33 @@ INTERRUPTED_REASON = "服务在发起运行前停止,这次触发没有发起
GENERIC_REASON = "发起运行时出错,详细信息见服务日志"
#: 定时的目标种类(Schedule.target_kind),顺序即以后界面上的顺序。这一波 fire_once 只会发起 workflow
TARGET_KINDS = ("workflow", "eval_suite", "source_maintenance")
TARGET_LABEL = {"workflow": "工作流", "eval_suite": "评测题集", "source_maintenance": "数据源维护"}
#: 目标种类这个版本还发起不了(库里有别的版本建的任务、手改过库):照实记「未能发起」,不当成工作流去跑
TARGET_UNSUPPORTED = "这个定时任务的目标是「{kind}」,当前版本还不能按时发起这类目标,这次触发没有发起"
def target_of(schedule: Any) -> tuple[str, str]:
"""一个定时任务的目标:(种类, id)。加列之前建的任务 target_kind 缺省 workflow、target_id 为空,目标就是 workflow_id。
workflow 时 id 以 workflow_id 为准(两列由 target_columns 一起写,对得上;老任务只有这一列)。"""
kind = getattr(schedule, "target_kind", None) or "workflow"
if kind == "workflow":
return kind, schedule.workflow_id or getattr(schedule, "target_id", None) or ""
return kind, getattr(schedule, "target_id", None) or ""
def target_columns(kind: str, target_id: str) -> dict[str, str]:
"""写目标的几列:{target_kind, target_id, workflow_id}。workflow 时 target_id 同 workflow_id;别的种类 workflow_id 存空串
(老表的非空约束)。种类不认识、id 为空抛 ValueError。"""
if kind not in TARGET_KINDS:
raise ValueError(f"不认识的定时目标种类:{kind}")
if not target_id:
raise ValueError("定时任务的目标 id 不能为空")
return {"target_kind": kind, "target_id": target_id, "workflow_id": target_id if kind == "workflow" else ""}
class TemplateError(ValueError):
"""输入模板写错了。str() 是给人看的中文说明,指出是哪个输入。"""
@@ -282,46 +314,61 @@ async def fire_once(schedule_id: str, fire_at: datetime, kind: str, *, actor: st
"""处理一次触发,返回记下的触发记录。同一 (任务, fire_at, kind) 已经有记录(两拍撞在一起、立即运行连点两下)
时不再发起,返回已有的那一条;任务不存在返回 None。
上一次运行还没结束就记 skipped_overlap;输入渲染不了、launch_run 拒绝或出错记 failed_to_start(拒绝写原话,
其他异常写通用说明、详细信息进日志);发起成功记 started 和运行 id。
按目标种类分派(target_of):这一波只有工作流(_fire_workflow),别的种类记 failed_to_start、写明原因,不发起。
"""
async with launch_lock():
async with SessionLocal() as session:
schedule = await session.get(Schedule, schedule_id)
if schedule is None:
return None
name, workflow_id, run_class = schedule.name, schedule.workflow_id, schedule.run_class
target_kind, target_id = target_of(schedule)
name, run_class = schedule.name, schedule.run_class
tz, cron_expr, template = schedule.timezone, schedule.cron, schedule.input_template
try:
inputs = render_inputs(check_template(template), template_values(fire_at, tz, cron_expr))
except (TemplateError, cron.CronError, cron.TimezoneError) as e:
# 存进库之后才坏掉的(手改过库、时区库升级删了时区):照实说是哪一项,不发起
return await _record(schedule_id, fire_at, kind, "failed_to_start", f"无法生成这次运行的输入:{e}", {})
busy = await _active_run(schedule_id)
if busy is not None:
return await _record(schedule_id, fire_at, kind, "skipped_overlap", overlap_reason(*busy), inputs)
fire = await _record(schedule_id, fire_at, kind, "failed_to_start", INTERRUPTED_REASON, inputs, fresh=True)
if fire is None:
return await _existing(schedule_id, fire_at, kind)
from app.engine.launch import LaunchRefused, launch_run
if target_kind != "workflow":
reason = TARGET_UNSUPPORTED.format(kind=TARGET_LABEL.get(target_kind, target_kind))
return await _record(schedule_id, fire_at, kind, "failed_to_start", reason, {})
return await _fire_workflow(schedule_id, fire_at, kind, actor=actor, name=name, workflow_id=target_id,
run_class=run_class, tz=tz, cron_expr=cron_expr, template=template)
_LAUNCHING.add(fire.id)
async def _fire_workflow(schedule_id: str, fire_at: datetime, kind: str, *, actor: str | None, name: str,
workflow_id: str, run_class: str, tz: str, cron_expr: str, template: Any,
) -> ScheduleFire | None:
"""目标是工作流的一次触发(持着 launch_lock)。
上一次运行还没结束就记 skipped_overlap;输入渲染不了、launch_run 拒绝或出错记 failed_to_start(拒绝写原话,
其他异常写通用说明、详细信息进日志);发起成功记 started 和运行 id。
"""
try:
inputs = render_inputs(check_template(template), template_values(fire_at, tz, cron_expr))
except (TemplateError, cron.CronError, cron.TimezoneError) as e:
# 存进库之后才坏掉的(手改过库、时区库升级删了时区):照实说是哪一项,不发起
return await _record(schedule_id, fire_at, kind, "failed_to_start", f"无法生成这次运行的输入:{e}", {})
busy = await _active_run(schedule_id)
if busy is not None:
return await _record(schedule_id, fire_at, kind, "skipped_overlap", overlap_reason(*busy), inputs)
fire = await _record(schedule_id, fire_at, kind, "failed_to_start", INTERRUPTED_REASON, inputs, fresh=True)
if fire is None:
return await _existing(schedule_id, fire_at, kind)
from app.engine.launch import LaunchRefused, launch_run
_LAUNCHING.add(fire.id)
try:
try:
try:
async with SessionLocal() as session:
run = await launch_run(session, workflow_id=workflow_id, run_class=run_class,
input_payload=inputs, actor=actor or actor_label(name),
schedule_id=schedule_id, origin_kind="schedule",
origin={"schedule_id": schedule_id, "schedule": name,
"fire_id": fire.id, "fire_kind": kind})
except LaunchRefused as e:
return await _finish(fire.id, status="failed_to_start", reason=e.message)
except Exception: # noqa: BLE001 - 发起失败不能让调度循环停下;详细信息进日志,记录上写通用说明
logger.exception("定时任务「%s」(%s)发起运行失败", name, schedule_id)
return await _finish(fire.id, status="failed_to_start", reason=GENERIC_REASON)
return await _finish(fire.id, status="started", reason=None, run_id=run.id)
finally:
_LAUNCHING.discard(fire.id)
async with SessionLocal() as session:
run = await launch_run(session, workflow_id=workflow_id, run_class=run_class,
input_payload=inputs, actor=actor or actor_label(name),
schedule_id=schedule_id, origin_kind="schedule",
origin={"schedule_id": schedule_id, "schedule": name,
"fire_id": fire.id, "fire_kind": kind})
except LaunchRefused as e:
return await _finish(fire.id, status="failed_to_start", reason=e.message)
except Exception: # noqa: BLE001 - 发起失败不能让调度循环停下;详细信息进日志,记录上写通用说明
logger.exception("定时任务「%s」(%s)发起运行失败", name, schedule_id)
return await _finish(fire.id, status="failed_to_start", reason=GENERIC_REASON)
return await _finish(fire.id, status="started", reason=None, run_id=run.id)
finally:
_LAUNCHING.discard(fire.id)
async def record_skip(schedule_id: str, fire_at: datetime, reason: str) -> ScheduleFire | None:
+39 -7
View File
@@ -5,7 +5,8 @@
再导出,调用方和测试不用改;_transport、RETRY_DELAYS 仍是这个模块自己的(测试照旧替换它们),send 每次调用时读。
通知里只有任务名、状态、出具档位、缺口条数、运行链接、封存哈希(界面上叫清单哈希)前 12 位和起止时间,失败时
另写失败节点的名字和下一步(不许诺能继续运行);运行成功结束后和同一任务上一次成功的运行的对比(auto_compare)只写
另写失败节点的名字和下一步(不许诺能继续运行);被人驳回的那一期按「运行结束」推「未出具(被驳回)」、写驳回的那一步,
手动停止的不推;运行成功结束后和同一任务上一次成功的运行的对比(auto_compare)只写
各类指标的个数和对比页链接。不放报告里的具体数字,也不放失败时的原始报错——报错里可能带着查询到的数据,Webhook
推去的群不一定有权看。
"""
@@ -71,6 +72,11 @@ NEXT_AS_USUAL = "下一次触发会照常发起"
#: 从没开始执行、任务又不在排期上(停用了):原因刚写过,记录页上也没有更多可看的,下一步是重新发起。「立即运行」
#: 是定时任务页上那个按钮的名字,停用的任务也能点
RERUN_ACTION = "如需重新发起,请在 AgentLab 的「定时任务」中点击「立即运行」"
#: 被人驳回的那一期(cancelled + end_reason=rejected,next-round X9):是人的决定,不是故障。按「运行结束」推(订阅
#: 「运行失败」的人盯的是故障),标题、状态照实写被驳回;驳回的备注不写(可能带着数),驳回的是哪一步写节点名
REJECTED_HEADLINE = "未出具(被驳回)"
REJECTED_STATUS = "被驳回"
REJECTED_ACTION = "驳回的说明请在 AgentLab 的「记录」中查看"
# --------------------------------------------------------------------------
@@ -110,12 +116,18 @@ class Notice:
#: 挂起的原因是写外部系统的结果未知(等核实,核实后自动接着跑),不是服务重启。只在 suspended 时有意义
action_pending: bool = False
error_node: str | None = None
#: 运行结束在 cancelled 的原因(Run.end_reason):rejected 时按「未出具(被驳回)」说,error_node 是驳回的那一步
end_reason: str | None = None
next_fire: bool = False
compare: dict[str, Any] | None = None
compare_url: str | None = None
fmt: str | None = None
signed: bool = False
@property
def rejected(self) -> bool:
return self.status == "cancelled" and self.end_reason == "rejected"
@property
def never_started(self) -> bool:
"""失败的运行从没开始执行(没有开始时间):服务重启时还在排队、被改成失败的就是这种,没有断点可续。"""
@@ -142,6 +154,8 @@ def message(notice: Notice) -> tuple[str, list[tuple[str, str]], str | None]:
headline = "未能发起运行"
elif notice.suspended:
headline = "运行已挂起"
elif notice.rejected:
headline = REJECTED_HEADLINE
else:
headline = _HEADLINE.get(notice.event, notice.event)
title = f"定时任务「{notice.schedule_name}」{headline}"
@@ -163,6 +177,10 @@ def message(notice: Notice) -> tuple[str, list[tuple[str, str]], str | None]:
lines += [("原因", ACTION_PENDING_REASON), ("处理", ACTION_PENDING_ACTION)]
elif notice.suspended:
lines += [("原因", SUSPENDED_REASON), ("处理", SUSPENDED_ACTION)]
elif notice.rejected:
if notice.error_node:
lines.append(("驳回节点", f"「{notice.error_node}」"))
lines.append(("处理", f"{REJECTED_ACTION};{NEXT_AS_USUAL}" if notice.next_fire else REJECTED_ACTION))
elif notice.status == "failed":
# 不写原始报错(可能带着查询到的数据),写失败节点的名字和下一步
if notice.error_node:
@@ -219,6 +237,8 @@ def _status_label(notice: Notice) -> str:
if notice.pending_approvals:
return "等待审批"
return "已挂起,等待核实" if notice.action_pending else "已挂起,可继续运行"
if notice.rejected:
return REJECTED_STATUS
return _STATUS_LABEL.get(notice.status or "", notice.status or "")
@@ -239,7 +259,9 @@ def _links(notice: Notice) -> list[tuple[str, str]]:
def _message(notice: Notice) -> core.Message:
title, lines, _ = message(notice)
return core.Message(title=title, lines=lines, links=_links(notice), ok=notice.event in ("finished", "test"))
# 被驳回的那一期按 finished 推,但没出具:不画成成功的颜色
return core.Message(title=title, lines=lines, links=_links(notice),
ok=notice.event in ("finished", "test") and not notice.rejected)
def _text(notice: Notice) -> str:
@@ -332,10 +354,17 @@ async def _record(schedule_id: str, fire_id: str | None, run_id: str | None, eve
#: 运行到了这些状态才可能要推送
_WATCHED = ("succeeded", "failed", "cancelled", "interrupted")
#: 结束的运行 → 事件。cancelled 是有人手动停的,不推。interrupted 分停在审批和挂起两种,见 _state_of
#: 结束的运行 → 事件。cancelled 是有人手动停的,不推;被驳回的(end_reason=rejected)按 finished 推,见 _event_of。
#: interrupted 分停在审批和挂起两种,见 _state_of
_EVENT_OF = {"succeeded": "finished", "failed": "failed", "cancelled": None}
def _event_of(run: Run) -> str | None:
if run.status == "cancelled" and run.end_reason == "rejected":
return "finished"
return _EVENT_OF[run.status]
@dataclass
class _Job:
fmt: str
@@ -395,9 +424,10 @@ async def notify_due(now: datetime | None = None) -> int:
.order_by(ScheduleFire.created_at)
)).all()
if rows:
from app.api.settings import notify_settings
# 链接里的地址:全站同一个对外访问地址(环境变量优先,其次设置,core/public_url.py)
from app.core import public_url
base_url = (await notify_settings(session))["public_base_url"]
base_url = await public_url.refresh(session)
for fire, run in rows:
seen = await _state_of(session, fire, run, now)
if seen.state is None or seen.state == fire.notified_status:
@@ -469,7 +499,7 @@ async def _state_of(session: Any, fire: ScheduleFire, run: Run | None, now: date
since = run.finished_at or run.updated_at or fire.created_at
if not sealed and (now - service.as_utc(since)).total_seconds() < SEAL_WAIT_S:
return _Seen(None)
return _Seen(run.status, _EVENT_OF[run.status], sealed=sealed)
return _Seen(run.status, _event_of(run), sealed=sealed)
if run.status != "interrupted":
return _Seen(None)
counts = dict((await session.execute(
@@ -516,7 +546,9 @@ async def _notice(session: Any, schedule: Schedule, fire: ScheduleFire, run: Run
pending_approvals=pending or None,
suspended=run is not None and run.status == "interrupted" and not pending,
action_pending=seen.action_pending,
error_node=_failed_node(run) if run is not None and run.status == "failed" else None,
error_node=_failed_node(run) if run is not None and (run.status == "failed" or run.end_reason == "rejected")
else None,
end_reason=run.end_reason if run is not None else None,
next_fire=bool(schedule.enabled and schedule.next_run_at is not None),
)
+14 -11
View File
@@ -277,7 +277,7 @@ async def build_datasource_tools(
except table_versions.SnapshotMissing as e:
tools.extend(_unavailable_tool(row, prefix, str(e)) for prefix in prefixes)
continue
entries = await _catalog_of(row)
entries = await _catalog_of(row, ctx.catalog_view)
for prefix in prefixes:
if prefix == QUERY_PREFIX:
tools.append(_make_query_tool(source, ctx, fixed=fixed, catalog=entries, force_readonly=force_readonly,
@@ -287,20 +287,17 @@ async def build_datasource_tools(
return tools
async def _catalog_of(row: DataSource) -> dict[str, data_catalog.CatalogEntry]:
async def _catalog_of(row: Any, view: Any = None) -> dict[str, data_catalog.CatalogEntry]:
"""这个源的数据目录,建工具时读一次:db_schema 给模型看的和查询时冻结进表结构快照的是同一版。
读不到就当没有目录,工具照常可用——目录是附加说明,不能挡住查询。用单独的会话读:出错要回滚,
而回滚会让调用方会话里已经加载的对象(row)全部过期,之后再读属性就是异步环境里的懒加载。
读哪一份由目录视图定(view,data/catalog_view.py:当前目录、目录快照、关;只用已核实的项),不给就是当前目录。
读不到就当没有目录,工具照常可用——目录是附加说明,不能挡住查询;视图指定的目录快照不在了照实报错(不悄悄换成当前
目录)。用单独的会话读:出错要回滚,而回滚会让调用方会话里已经加载的对象(row)全部过期,之后再读属性就是异步环境里
的懒加载。
"""
from app.db.base import SessionLocal
from app.data import catalog_view
try:
async with SessionLocal() as own:
return await data_catalog.read_catalog(own, row.id)
except Exception: # noqa: BLE001
logger.exception("读取数据源 %s 的数据目录失败,本次不附目录", row.name)
return {}
return await catalog_view.read(catalog_view.CatalogView.parse(view), row)
# --------------------------------------------------------------------------
@@ -753,6 +750,12 @@ async def _store_schema(source: Any, ctx: ToolContext, *,
if frozen := data_catalog.frozen_catalog(catalog or {}, tables):
content[CATALOG_SNAPSHOT_KEY] = frozen
meta["catalog_tables"] = len(frozen)
# 这次运行读的不是缺省的当前目录(评测的对照:关、目录快照、只用已核实的项):记在引用的 meta 里,证据面板写得出
# 「本次运行目录:关」。不进内容哈希,缺省视图不记,快照和以前一字不差
from app.data.catalog_view import CatalogView, describe
if (view := describe(CatalogView.parse(getattr(ctx, "catalog_view", None)))) is not None:
meta["catalog_view"] = view
try:
return await put_json(content, kind=SCHEMA_SNAPSHOT, run_id=ctx.run_id or "", node_id=ctx.node_id or "",
meta=meta)
+3
View File
@@ -49,6 +49,9 @@ class ToolContext:
#: 个人应用的刷新、个人蓝图的试运行)。记忆、知识库的内容不经遮罩:kb_search 的 collection 参数(调用工具节点写的、
#: 参数名纠错改出来的、图级 defaults 补上的、模型挑的)一律拒,只能检索 collection 这一个
fixed_scope: bool = False
#: 读哪一份数据目录(data/catalog_view.CatalogView):运行里由节点从 Run.options 取来(catalog_view.of_run),db_schema
#: 给模型看的、查询冻结进表结构快照的、查询时的 SQL 检查都按它。None 是当前目录(缺省,行为同以前)
catalog_view: Any = None
extra: dict[str, Any] = field(default_factory=dict)

Some files were not shown because too many files have changed in this diff Show More