mirror of
https://github.com/ThearchyHelios/AgentLab.git
synced 2026-11-17 20:31:58 +00:00
合入下一轮第 1 波(docs/design/next-round):评测底座(后台任务运行时上移、系统运行不进使用统计、全站唯一的问题聚类、目录视图接缝、问数据生成过程 answer_ops 与用量计量、问数据的决定下沉后端、每日账泛化、定时目标底座、通知事件登记表);已知遗留(人工驳回记为被驳回不算失败、个人渠道「我发起的实例」生效、导入清单如实记认证、使用者选人按候选池核对、自由形蓝图「谁来做」、tick 按 id 断言);目录方言与 Excel 期 5(剖析方言修正与金样、本机四库容器集成测试、列表按期累积、穿插小计、表格对象起草、CSV 走配方);集成底座(安全响应头、对外访问地址只留一个、开登录时收起 /docs、集成接口错误码)。验收:后端全量 13463 通过;tsc、文案检查;本波相关的 15 项浏览器检查 4939 项全过;四种数据库容器集成测试全过;泄露扫描干净
This commit is contained in:
@@ -90,6 +90,21 @@ http://127.0.0.1:8000 。前端产物不存在或比源码旧时才重新构建
|
||||
MCP 要管理员先在「工具 → 端点」登记。
|
||||
- `/api/copilot/generate` 改图时只出改动操作,不再是整张新图;每次运行多一条 `run.origin` 事件(在封存范围内)。
|
||||
|
||||
**从旧版本升级(2026-10 下一轮第 1 波)**:有几处行为变化会影响现有部署:
|
||||
|
||||
- 人工驳回(审批、动作确认、填写、代码和工具审批、双盲复算)的运行记为「已取消 · 被驳回」,不再算失败:通知走「实例
|
||||
结束」,定时任务推「未出具(被驳回)」,不能继续运行。配了失败重试或「出错时继续」的关卡,驳回不再重问、也不接着往下走。
|
||||
已封存的老运行不改写。
|
||||
- 开了登录时,应用入口和待办表单里的选人控件按候选池核对(缺省是能用这个应用的人),池外的名字回 422(`principal`)。
|
||||
- 页面默认只能被本站嵌入(`frame-ancestors 'self'`,另加 `X-Frame-Options`)。已经用 iframe 嵌 AgentLab 的门户,
|
||||
在 `AGENTLAB_FRAME_ANCESTORS` 里写上门户的地址。
|
||||
- 「对外访问地址」只剩一个设置(「设置 → 通用」;`AGENTLAB_PUBLIC_URL` 优先,写错了启动就报错)。设置里填了地址的部署,
|
||||
OIDC 回调改按这个地址拼,和身份提供方登记的不一致时要去改登记。开了登录时 `/docs`、`/openapi.json` 要搭建者登录。
|
||||
- 探查结构时列类型改存各数据库的完整写法(`VARCHAR2(20 CHAR)`、`ENUM('a','b')`):升级后每个数据源点一次「探查结构」。
|
||||
新运行的表结构快照哈希会变,旧运行不受影响。MySQL / MariaDB 上文字列的剖析说明多一句排序规则,ENUM 列补上数据里没出现的
|
||||
取值,日期列写明排除了几个无效日期,所以升级后第一次重新剖析,这些表的目录各升一次版本。剖析设置的「抽样键值数」改叫
|
||||
「核对键值数」(`DISTINCT … LIMIT` 取的不是随机样本)。
|
||||
|
||||
---
|
||||
|
||||
## 核心能力
|
||||
@@ -103,10 +118,10 @@ http://127.0.0.1:8000 。前端产物不存在或比源码旧时才重新构建
|
||||
| **能力登记与端点** | 所有可调用的工具登记在一处,逐项治理读写效果、审批、字段语义、遮罩;外部系统按测试 / 生产两个环境登记成端点,只有登记的端点能连内网,认证头加密;端点读回的表格和查询快照一样能作证([详见](#能力登记与端点)) |
|
||||
| **Agent 护栏** | 同样的调用不重复执行、连续拿不到新信息就收尾、令牌与金额预算、上下文接近窗口先压缩再收尾;步数只留兜底(默认 100) |
|
||||
| **代码沙箱** | 双档隔离:系统沙箱(Seatbelt / bubblewrap,冷启动约 25ms)或 microVM(独立 Linux 内核,内存限额真实生效) |
|
||||
| **人工介入** | 执行暂停并落盘,人工批准 / 驳回 / 改稿后从断点继续;进程重启不丢状态;可派给岗位、发起人或按表达式算出的人,会签(任一人 / 全体同意,一票否决 / 至少 m 人),设时限,超时提醒、升级或驳回,永远不会自动批准 |
|
||||
| **人工介入** | 执行暂停并落盘,人工批准 / 驳回 / 改稿后从断点继续;进程重启不丢状态;可派给岗位、发起人或按表达式算出的人,会签(任一人 / 全体同意,一票否决 / 至少 m 人),设时限,超时提醒、升级或驳回,永远不会自动批准;驳回是人的决定,运行记为「被驳回」,不算失败、不重试 |
|
||||
| **断点续跑** | 失败的运行可从失败节点继续,已完成的节点不重复执行;探索运行允许修改节点配置(拓扑必须一致),正式运行固定在发布的版本上、不能改配置后继续;继续运行不改发起人 |
|
||||
| **知识库** | 向量 + BM25 混合检索、倒排索引、可选模型重排;PDF / Word / PowerPoint / HTML / Markdown 解析;向量模型可接本地端点 |
|
||||
| **表格入库** | Excel / CSV 上传后转为可用 SQL 查询的表,自动生成 `db_query__<name>` 工具——数字由 SQL 计算得出,而非模型读取 |
|
||||
| **表格入库** | Excel / CSV 上传后转为可用 SQL 查询的表,自动生成 `db_query__<name>` 工具——数字由 SQL 计算得出,而非模型读取;复杂报表按取数配方导入(CSV 同样可以),每月一份的明细能按期累积成一张表,夹在明细中间的小计核对后不进数据 |
|
||||
| **数据目录** | 给数据源的每张表记下粒度、业务主键、业务日期、有效记录条件,每个字段的含义、单位、度量类型和码值,以及表与表的关联;由外键、命名推断、数据剖析和模型起草,逐项确认后生效;表详情里可以和助手对话完善一张表,审阅队列按影响排好先审哪些,目录能导出、导入到别的实例;助手建图、agent 查结构、SQL 检查都读它 |
|
||||
| **SQL 检查** | 按数据目录检查一对多关联后重复计算、存量跨时间相加、比率被相加等七类问题;助手自查时交回修正,受管模板发布时挡住,运行时标在查询结果上、契约指标降档 |
|
||||
| **跨库合并** | 合并查询节点:各库先聚合到同一个键,再在库外按键合并;截断的输入拒收,证据一路追到两边的原始查询 |
|
||||
@@ -115,7 +130,7 @@ http://127.0.0.1:8000 。前端产物不存在或比源码旧时才重新构建
|
||||
| **自然语言编排** | 助手:一句话生成或改写整张图,自动排版;system 里每个数据源只给一行概览,要看表结构、能力参数、工作流版本时由助手自己「查看」,一次给齐字段、目录和连接条件;交付前按运行时同一套规则自查,查出会运行失败的问题先交回去改,改不好就不自动运行([详见](#助手按需查看)) |
|
||||
| **结构化成果** | JSON Schema 校验,不合格时将错误回传模型自动返工 |
|
||||
| **产出复核** | 运行结束后规则扫描事件流,发现异常才调用模型复核;复核可重组答案但**不得引入原答案中不存在的数字** |
|
||||
| **追踪与成本** | 全量事件落库可回放;逐节点耗时、token 用量、按目录价折算的成本 |
|
||||
| **追踪与成本** | 全量事件落库可回放;逐节点耗时、token 用量、按目录价折算的成本;问数据每一轮的花费含助手建图、挑表和复核 |
|
||||
| **三档出具** | 口径卡(受控指标集)+ 出具契约:叙述中每个数字回指指标集,或由报告撰写节点按引用逐段核对,formal / degraded / withheld 三档判定 |
|
||||
| **可点击证据** | 报告撰写节点只写引用标记,数字和表格由系统照快照填值;数字、表名字段名、知识库原话都点得开,一路指到查询结果里的那一格;结论句可要求挂依据或交给另一个模型裁判;证据清单可导出,报告导出为自包含 HTML(数字点得开口径与出处),另有封存哈希核对件「可验证报告」 |
|
||||
| **发布前检查** | 受管级别加证据门禁 G1–G5;问题给出一键修复 / 选项 / 交给 Copilot 三类修法,一律先预览、不许降低要求;老工作流一键升级为可追溯结构 |
|
||||
@@ -738,6 +753,12 @@ Excel / CSV 若传入知识库,只能被切块检索,数字即成为模型
|
||||
- 内网只对登记的端点开放(管理员开「允许访问内网」),元数据地址和高危端口照样拦;模型给出的地址照旧做完整的 SSRF 检查
|
||||
- 开了登录:会话 cookie 在库里只存 token 的哈希;非 GET 请求核对来源标记头和 Origin;使用者只能读有授权的运行,看到的查询结果一律遮罩
|
||||
- 写外部系统必须经「调用工具」节点:先留生效记录再发送,确认卡上的内容哈希在发送前重算比对;结果未知永不自动重发
|
||||
- 安全响应头:`nosniff`、`Referrer-Policy`,以及 `frame-ancestors`(缺省 `'self'`,`AGENTLAB_FRAME_ANCESTORS` 列出允许
|
||||
嵌整站的同站门户;只有 `'self'` 时另加 `X-Frame-Options: SAMEORIGIN`)。严格的 `script-src` 目前只发
|
||||
`Content-Security-Policy-Report-Only`,观察一个版本没有误报再强制。HSTS 交给反向代理
|
||||
- 集成方要用的接口(应用、待办、工件、角色拒绝)的错误一律带机读码(`code`),`detail` 是给人看的原话,可以改
|
||||
- 评测、口径对账、口径召回是系统代为发起的运行:证据照常可查,但不进使用统计(探索问题聚类、工具白名单、目录使用
|
||||
次数),记录页默认不列并写明有几次没列出
|
||||
|
||||
---
|
||||
|
||||
|
||||
+26
-19
@@ -12,14 +12,18 @@ import dataclasses
|
||||
import json
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, Depends, Header, HTTPException, Query
|
||||
from fastapi import APIRouter, Depends, Header, Query
|
||||
from fastapi.responses import JSONResponse
|
||||
from pydantic import BaseModel, Field
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.api.coded import CodedHTTPException
|
||||
from app.api.coded import (
|
||||
APP_BINDINGS_INVALID, APP_DRAFT_INVALID, APP_GRANTS_INVALID, APP_HOOK_NOT_DECLARED, APP_HOOK_NOT_FOUND,
|
||||
APP_INSTANCE_NOT_FOUND, APP_NOT_FOUND, APP_PARAMS_INVALID, APP_VERSION_NOT_FOUND, FORM_INVALID, RUN_LAUNCH_REFUSED,
|
||||
CodedHTTPException,
|
||||
)
|
||||
from app.api.deps import Principal, actor_of, current_principal
|
||||
from app.apps.access import ROLE_TEXT, app_role, at_least, grants_of
|
||||
from app.db.base import get_session
|
||||
@@ -60,11 +64,11 @@ def _principal(x_actor: str | None) -> Principal:
|
||||
async def _app(session: AsyncSession, app_id: str, principal: Principal, need: str) -> tuple[App, str]:
|
||||
app = await session.get(App, app_id)
|
||||
if app is None:
|
||||
raise HTTPException(404, APP_GONE)
|
||||
raise CodedHTTPException(404, APP_GONE, APP_NOT_FOUND)
|
||||
role = await app_role(session, principal, app)
|
||||
if role is None:
|
||||
# 没有任何授权的人不该知道这个应用在不在
|
||||
raise HTTPException(404, APP_GONE)
|
||||
# 没有任何授权的人不该知道这个应用在不在:码和不存在的一样
|
||||
raise CodedHTTPException(404, APP_GONE, APP_NOT_FOUND)
|
||||
if not at_least(role, need):
|
||||
raise CodedHTTPException(403, f"需要这个应用的「{ROLE_TEXT[need]}」权限", NO_ACCESS)
|
||||
return app, role
|
||||
@@ -314,7 +318,7 @@ async def derive(payload: DeriveIn, session: AsyncSession = Depends(get_session)
|
||||
try:
|
||||
return {"spec": await derive_app(session, payload.workflow_id, payload.version, name=payload.name)}
|
||||
except DeriveRefused as e:
|
||||
raise HTTPException(e.status_code, e.message) from e
|
||||
raise CodedHTTPException(e.status_code, e.message, e.code) from e
|
||||
|
||||
|
||||
@router.get("/{app_id}")
|
||||
@@ -359,13 +363,13 @@ async def update_app(app_id: str, payload: AppPatch, session: AsyncSession = Dep
|
||||
app, role = await _app(session, app_id, principal, "manage")
|
||||
_refuse_archived(app, "修改")
|
||||
if payload.draft_bindings is not None and (problem := _bindings_problem(payload.draft_bindings)):
|
||||
raise HTTPException(422, problem)
|
||||
raise CodedHTTPException(422, problem, APP_BINDINGS_INVALID)
|
||||
if app.scope == "personal" and payload.draft is not None:
|
||||
# 个人应用的草稿就是在用的那一份:只许改呈现,固定的蓝图、视图来源只由「存成我的应用」改
|
||||
from app.blueprints.personal import draft_problem
|
||||
|
||||
if problem := draft_problem(app.draft or {}, payload.draft):
|
||||
raise HTTPException(422, problem)
|
||||
raise CodedHTTPException(422, problem, APP_DRAFT_INVALID)
|
||||
for key in ("name", "description", "icon"):
|
||||
if (value := getattr(payload, key)) is not None:
|
||||
setattr(app, key, value)
|
||||
@@ -568,7 +572,7 @@ async def get_version(app_id: str, version: int, session: AsyncSession = Depends
|
||||
row = (await session.execute(select(AppVersion).where(
|
||||
AppVersion.app_id == app.id, AppVersion.version == version))).scalar_one_or_none()
|
||||
if row is None:
|
||||
raise HTTPException(404, f"「{app.name}」没有 v{version}")
|
||||
raise CodedHTTPException(404, f"「{app.name}」没有 v{version}", APP_VERSION_NOT_FOUND)
|
||||
# 岗位绑定是这一个部署里的人和组:管理者才看得到
|
||||
return {**_version_row(row), "spec": row.spec, "bindings": row.bindings if role == "manage" else {"slots": {}}}
|
||||
|
||||
@@ -611,7 +615,7 @@ async def put_grants(app_id: str, payload: GrantsIn, session: AsyncSession = Dep
|
||||
_refuse_personal(app, "授权给别人")
|
||||
wanted, problem = await grants_problem(session, [(i.principal_kind, i.principal, i.role) for i in payload.items])
|
||||
if problem:
|
||||
raise HTTPException(422, problem)
|
||||
raise CodedHTTPException(422, problem, APP_GRANTS_INVALID)
|
||||
return {"items": await replace_grants(session, app, wanted, actor=actor_of(x_actor))}
|
||||
|
||||
|
||||
@@ -713,7 +717,8 @@ async def rotate_hook(app_id: str, name: str, session: AsyncSession = Depends(ge
|
||||
draft, published, rows = await _hook_state(session, app)
|
||||
declared = draft.get(name) or published.get(name)
|
||||
if declared is None:
|
||||
raise HTTPException(404, f"草稿和当前发布版里都没有声明钩子「{name}」,请先在「钩子」一节里声明它")
|
||||
raise CodedHTTPException(404, f"草稿和当前发布版里都没有声明钩子「{name}」,请先在「钩子」一节里声明它",
|
||||
APP_HOOK_NOT_DECLARED)
|
||||
# 从读行锁到提交:两位管理者同时点「生成密钥」,后到的那次读得到前一次插入的行,只轮换它(见 process/hooks.py)
|
||||
async with hooks.hook_lock(app.id, name):
|
||||
row, secret, created = await hooks.rotate_hook(session, app.id, name, kind=declared["kind"],
|
||||
@@ -731,7 +736,7 @@ async def _hook_named(session: AsyncSession, app: App, name: str) -> AppHook:
|
||||
|
||||
row = await hook_row(session, app.id, name)
|
||||
if row is None:
|
||||
raise HTTPException(404, f"钩子「{name}」还没有生成密钥")
|
||||
raise CodedHTTPException(404, f"钩子「{name}」还没有生成密钥", APP_HOOK_NOT_FOUND)
|
||||
return row
|
||||
|
||||
|
||||
@@ -778,9 +783,9 @@ def _params(raw: str | None) -> dict[str, Any]:
|
||||
try:
|
||||
value = json.loads(raw)
|
||||
except ValueError as e:
|
||||
raise HTTPException(422, "页面参数需要是 JSON 对象") from e
|
||||
raise CodedHTTPException(422, "页面参数需要是 JSON 对象", APP_PARAMS_INVALID) from e
|
||||
if not isinstance(value, dict):
|
||||
raise HTTPException(422, "页面参数需要是 JSON 对象")
|
||||
raise CodedHTTPException(422, "页面参数需要是 JSON 对象", APP_PARAMS_INVALID)
|
||||
return value
|
||||
|
||||
|
||||
@@ -788,14 +793,16 @@ async def _instance(session: AsyncSession, app: App, run_id: str, principal: Pri
|
||||
"""实例要属于这个应用;只有 use 的人只能看自己发起的(关登录时都能看)。"""
|
||||
run = await session.get(Run, run_id)
|
||||
if run is None or run.app_id != app.id or run.origin_kind not in ("app_entry", "hook"):
|
||||
raise HTTPException(404, "实例不存在")
|
||||
raise CodedHTTPException(404, "实例不存在", APP_INSTANCE_NOT_FOUND)
|
||||
if principal.mode != "off" and not at_least(role, "observe") and run.started_by != principal.username:
|
||||
raise CodedHTTPException(403, "只能查看自己发起的实例", NO_ACCESS)
|
||||
return run
|
||||
|
||||
|
||||
def _view_refused(e: Any) -> HTTPException:
|
||||
return CodedHTTPException(e.status_code, e.message, e.code) if e.code else HTTPException(e.status_code, e.message)
|
||||
def _view_refused(e: Any) -> CodedHTTPException:
|
||||
"""视图、入口、补算、发起运行的拒绝 → 带 code 的 HTTP 错误。只有引擎不发起运行(LaunchRefused)有不带码的几种(工作流
|
||||
没发布、版本不存在、图没通过校验……),补上同一类的 run_launch_refused,detail 照旧说清是哪一种。"""
|
||||
return CodedHTTPException(e.status_code, e.message, e.code or RUN_LAUNCH_REFUSED)
|
||||
|
||||
|
||||
@router.get("/{app_id}/pages/{page_id}")
|
||||
@@ -834,7 +841,7 @@ async def start(app_id: str, entry_id: str, payload: StartIn, session: AsyncSess
|
||||
try:
|
||||
run = await start_entry(app.id, entry_id, payload.values, principal)
|
||||
except FormRejected as e:
|
||||
return JSONResponse(status_code=422, content={"detail": "表单没有通过校验", "errors": e.errors})
|
||||
return JSONResponse(status_code=422, content={"detail": "表单没有通过校验", "code": FORM_INVALID, "errors": e.errors})
|
||||
except (EntryRefused, LaunchRefused) as e:
|
||||
raise _view_refused(e) from e
|
||||
return {"run_id": run.id, "title": instance_title(run)}
|
||||
@@ -905,7 +912,7 @@ async def refresh_view(app_id: str, view_id: str, payload: RefreshIn, session: A
|
||||
run, cached = await run_view(session, app=app, version=version, view_id=view_id, params=payload.params,
|
||||
principal=principal, actor=actor_of(x_actor))
|
||||
except FormRejected as e:
|
||||
return JSONResponse(status_code=422, content={"detail": "参数没有通过校验", "errors": e.errors})
|
||||
return JSONResponse(status_code=422, content={"detail": "参数没有通过校验", "code": FORM_INVALID, "errors": e.errors})
|
||||
except (ViewRefused, LaunchRefused) as e:
|
||||
raise _view_refused(e) from e
|
||||
return {"run_id": run.id, "cached": cached}
|
||||
|
||||
@@ -2,9 +2,12 @@ from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from fastapi import APIRouter
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.api.coded import (
|
||||
ARTIFACT_FORBIDDEN, ARTIFACT_KIND_FORBIDDEN, ARTIFACT_NOT_FOUND, ARTIFACT_TAMPERED, CodedHTTPException,
|
||||
)
|
||||
from app.api.deps import can_read_artifact, current_principal
|
||||
from app.core import artifact_store
|
||||
|
||||
@@ -23,14 +26,14 @@ async def get_artifact(artifact_id: str) -> dict[str, Any]:
|
||||
principal = current_principal()
|
||||
restricted = principal.mode != "off" and not principal.at_least("builder")
|
||||
if principal.mode != "off" and not await can_read_artifact(principal, artifact_id):
|
||||
raise HTTPException(403, "没有权限查看这件工件")
|
||||
raise CodedHTTPException(403, "没有权限查看这件工件", ARTIFACT_FORBIDDEN)
|
||||
try:
|
||||
content = artifact_store.load(artifact_id)
|
||||
except ValueError as e:
|
||||
# 哈希对不上:明确报给调用方,这正是工件库存在的意义
|
||||
raise HTTPException(409, str(e)) from e
|
||||
raise CodedHTTPException(409, str(e), ARTIFACT_TAMPERED) from e
|
||||
if content is None:
|
||||
raise HTTPException(404, "工件不存在,可能已被删除")
|
||||
raise CodedHTTPException(404, "工件不存在,可能已被删除", ARTIFACT_NOT_FOUND)
|
||||
if restricted:
|
||||
content = await _masked_for_user(principal, artifact_id, content)
|
||||
return {"id": artifact_id, "content": content}
|
||||
@@ -68,7 +71,7 @@ async def _masked_for_user(principal: Any, artifact_id: str, content: Any) -> An
|
||||
return await _masked_metrics(principal, artifact_id, content, rows)
|
||||
if _RECEIPT in kinds and isinstance(content, dict):
|
||||
return await _masked_receipt(content, rows)
|
||||
raise HTTPException(403, "这类工件的原文只对搭建者及以上的角色开放")
|
||||
raise CodedHTTPException(403, "这类工件的原文只对搭建者及以上的角色开放", ARTIFACT_KIND_FORBIDDEN)
|
||||
|
||||
|
||||
async def _masked_snapshot(content: dict[str, Any], *, endpoint: bool) -> dict[str, Any]:
|
||||
@@ -106,7 +109,7 @@ async def _masked_metrics(principal: Any, artifact_id: str, card: dict[str, Any]
|
||||
owner = (run_id, node_id or "")
|
||||
break
|
||||
if owner is None:
|
||||
raise HTTPException(403, "没有权限查看这件工件")
|
||||
raise CodedHTTPException(403, "没有权限查看这件工件", ARTIFACT_FORBIDDEN)
|
||||
sealed = await _sealed(owner[0])
|
||||
masks = _Masks(sealed)
|
||||
metrics: list[Any] = []
|
||||
|
||||
@@ -17,6 +17,7 @@ from pydantic import BaseModel, Field
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.api.coded import AUTH_REQUIRED
|
||||
from app.api.deps import Principal, current_principal, decode_actor, ensure_role
|
||||
from app.auth import (
|
||||
MODES, SETTING_KEY, current_mode, env_mode, load_settings, remember_settings, stored_settings, user_can_chat,
|
||||
@@ -25,7 +26,6 @@ from app.auth import oidc, sessions
|
||||
from app.auth.audit import record
|
||||
from app.auth.passwords import hash_password, password_problem, verify_password
|
||||
from app.core import crypto
|
||||
from app.core.config import settings
|
||||
from app.db.base import get_session
|
||||
from app.db.models import Setting, User
|
||||
|
||||
@@ -60,7 +60,7 @@ async def me(x_actor: str | None = Header(default=None), session: AsyncSession =
|
||||
return await principal_full(session, Principal(username=actor, display_name=actor, role="admin", mode="off"))
|
||||
if not principal.username:
|
||||
can_bootstrap = principal.mode == "local" and not await _local_admin_exists(session)
|
||||
return JSONResponse(status_code=401, content={"detail": "请先登录", "code": "auth_required",
|
||||
return JSONResponse(status_code=401, content={"detail": "请先登录", "code": AUTH_REQUIRED,
|
||||
"mode": principal.mode, "can_bootstrap": can_bootstrap})
|
||||
return await principal_full(session, principal)
|
||||
|
||||
@@ -113,7 +113,10 @@ async def logout(request: Request, session: AsyncSession = Depends(get_session))
|
||||
|
||||
|
||||
def _redirect_uri(request: Request) -> str:
|
||||
base = settings.public_url.rstrip("/") or str(request.base_url).rstrip("/")
|
||||
"""回调地址:对外访问地址(环境变量优先,其次设置,core/public_url.py),都没有按请求自己的地址拼。"""
|
||||
from app.core.public_url import public_base_url
|
||||
|
||||
base = public_base_url() or str(request.base_url).rstrip("/")
|
||||
return f"{base}/api/auth/oidc/callback"
|
||||
|
||||
|
||||
|
||||
@@ -53,7 +53,8 @@ async def export_bundle(run_id: str) -> Response:
|
||||
raise CodedHTTPException(404, "运行记录不存在", RUN_NOT_FOUND)
|
||||
info = {"id": run.id, "workflow_id": run.workflow_id, "workflow_name": run.workflow_name,
|
||||
"version": run.version, "version_hash": run.version_hash, "run_class": run.run_class,
|
||||
"status": run.status, "started_at": run.started_at, "finished_at": run.finished_at}
|
||||
"status": run.status, "end_reason": run.end_reason, "started_at": run.started_at,
|
||||
"finished_at": run.finished_at}
|
||||
# 封存之后追加的事件(按需裁判、审批质询):_sealed 只留了封存范围内的,这几条另取,单列在页面上
|
||||
later = [tuple(r) for r in await session.execute(
|
||||
select(RunEvent.seq, RunEvent.type, RunEvent.node_id, RunEvent.data)
|
||||
|
||||
@@ -29,6 +29,55 @@ CATALOG_DRAFT_BUSY = "catalog_draft_busy"
|
||||
CATALOG_DRAFT_MODEL_UNAVAILABLE = "catalog_draft_model_unavailable"
|
||||
CATALOG_DRAFT_FINISHED = "catalog_draft_finished"
|
||||
|
||||
# ---------------------------------------------------------------- 集成接口(04 细案 A7)
|
||||
# 外部系统用访问密钥调的应用、待办、工件接口,以及角色、运行读权限的拒绝:按 code 认是哪种错,detail 照旧是给人看的那句
|
||||
|
||||
#: 身份:没登录(401,中间件、auth.require 都回它)、角色不够(403,路由级依赖、接口内部的 ensure_role、开登录时的 /docs)
|
||||
AUTH_REQUIRED = "auth_required"
|
||||
ROLE_REQUIRED = "role_required"
|
||||
#: 运行:这次运行读不了(403,api/deps 的证据、运行接口);别人的个人应用刷新、个人蓝图试运行只给拥有者(403)
|
||||
RUN_FORBIDDEN = "run_forbidden"
|
||||
RUN_PRIVATE = "run_private"
|
||||
#: 问数据和助手:使用者没开问数据,或者这个接口不在问数据的范围里(403)
|
||||
CHAT_FORBIDDEN = "chat_forbidden"
|
||||
|
||||
#: 应用(api/apps.py)。app_not_found 同入口发起时的 apps/entries.APP_GONE;没有任何授权的人也回它(不该知道应用在不在)
|
||||
APP_NOT_FOUND = "app_not_found"
|
||||
APP_VERSION_NOT_FOUND = "app_version_not_found"
|
||||
APP_INSTANCE_NOT_FOUND = "app_instance_not_found"
|
||||
#: 草稿里、当前发布版里都没有声明这个钩子(404);声明了、还没生成密钥(404)
|
||||
APP_HOOK_NOT_DECLARED = "app_hook_not_declared"
|
||||
APP_HOOK_NOT_FOUND = "app_hook_not_found"
|
||||
#: 改草稿、授权时写进来的不合规(422):岗位绑定、个人应用的草稿(只许改呈现)、授权的人和组
|
||||
APP_BINDINGS_INVALID = "app_bindings_invalid"
|
||||
APP_DRAFT_INVALID = "app_draft_invalid"
|
||||
APP_GRANTS_INVALID = "app_grants_invalid"
|
||||
#: 页面参数不是 JSON 对象(422)
|
||||
APP_PARAMS_INVALID = "app_params_invalid"
|
||||
#: 入口表单、按需视图的参数没通过校验(422,另带 errors:[{path, code, message}])。同待办回复、入站钩子的表单错误
|
||||
FORM_INVALID = "form_invalid"
|
||||
#: 发起实例、刷新视图、补算时引擎不发起运行,又没有更细的码(engine/launch.LaunchRefused 没带 code 的那几种)
|
||||
RUN_LAUNCH_REFUSED = "run_launch_refused"
|
||||
# 派生应用草稿时工作流、版本不存在(404)的码在 apps/derive.py(WORKFLOW_GONE、VERSION_GONE);入口、视图、补算的码
|
||||
# 在 apps/entries.py、apps/resolve.py、apps/backfill.py,同这里一样都是 app_ 开头
|
||||
|
||||
#: 待办(api/tasks.py)。服务层(process/tasks.py 的 TaskRefused)没带码的拒绝按状态补上同一类的码:404 不存在;403
|
||||
#: 不能这样做(不是管理员也不是应用的管理者,不能改派、催办);409 待办已经处理过、关闭了、刚被别人回复,或者所属的
|
||||
#: 运行不再等这条回复——刷新看最新状态;422 回复的内容不对。带了码的照用自己的(task_not_assigned、form_invalid……)
|
||||
TASK_NOT_FOUND = "task_not_found"
|
||||
TASK_FORBIDDEN = "task_forbidden"
|
||||
TASK_CLOSED = "task_closed"
|
||||
TASK_INVALID = "task_invalid"
|
||||
|
||||
#: 工件(api/artifacts.py):没有权限(403)、这类工件的原文只给搭建者(403)、不存在(404)、内容和哈希对不上(409)
|
||||
ARTIFACT_FORBIDDEN = "artifact_forbidden"
|
||||
ARTIFACT_KIND_FORBIDDEN = "artifact_kind_forbidden"
|
||||
ARTIFACT_NOT_FOUND = "artifact_not_found"
|
||||
ARTIFACT_TAMPERED = "artifact_tampered"
|
||||
|
||||
#: 设置(api/settings.py):对外访问地址由环境变量 AGENTLAB_PUBLIC_URL 指定时,设置里那一项不能改成别的值(409)
|
||||
PUBLIC_URL_ENV_LOCKED = "public_url_env_locked"
|
||||
|
||||
|
||||
class CodedHTTPException(HTTPException):
|
||||
"""field:出错的是哪一项(表单据此把报错落到那一格)。可选,没有就不出现在返回里。"""
|
||||
|
||||
@@ -118,7 +118,7 @@ class ConversationOut(BaseModel):
|
||||
turn_count: int = 0
|
||||
#: 列表里给个副标题,让人一眼认出是哪次聊天
|
||||
last_question: str = ""
|
||||
#: 最后一轮停在哪:running | waiting | error | cancelled | suspended | done,
|
||||
#: 最后一轮停在哪:running | waiting | error | cancelled | rejected | suspended | done,
|
||||
#: 没有轮次时为 None。见 last_status()
|
||||
last_status: str | None = None
|
||||
#: 最后一轮的运行。建图阶段就断了的没有
|
||||
@@ -173,18 +173,24 @@ BUILD_STALE = timedelta(minutes=15)
|
||||
|
||||
|
||||
def last_status(
|
||||
turn: Any, run_status: str | None, pending: bool, now: datetime | None = None,
|
||||
turn: Any, run_status: str | None, pending: bool, now: datetime | None = None, end_reason: str | None = None,
|
||||
) -> str:
|
||||
"""一轮停在哪。轮次状态只有 running / done / error 三种,细分要看运行和元数据。
|
||||
|
||||
轮次状态是前端写的:页面关掉、服务重启时它停在 running,这时要以运行的状态为准;
|
||||
运行也没有的,是断在了建图阶段。已经收尾的轮次以轮次自己为准——它写下的就是
|
||||
用户看到的结局(取消、服务重启挂起在 error 里分开记)。
|
||||
|
||||
被审批人驳回结束的(运行 cancelled + end_reason=rejected,05-已知遗留 (a))单独记 rejected,不和手动停止混成
|
||||
cancelled:前端落库时 meta 里记着 rejected(outcome 照旧是 cancelled),页面关掉没落库的看运行的 end_reason。
|
||||
"""
|
||||
outcome = (turn_meta(turn) or {}).get("outcome")
|
||||
meta = turn_meta(turn) or {}
|
||||
outcome = meta.get("outcome")
|
||||
if turn.status == "done":
|
||||
return "done"
|
||||
if turn.status == "error":
|
||||
if outcome == "cancelled" and meta.get("rejected"):
|
||||
return "rejected"
|
||||
if outcome == "cancelled" or (turn.error or "").strip() == "已取消":
|
||||
return "cancelled"
|
||||
if outcome == "suspended":
|
||||
@@ -192,7 +198,7 @@ def last_status(
|
||||
live = _from_run(run_status, pending)
|
||||
return live if live in ("running", "waiting") else "suspended"
|
||||
return "error"
|
||||
live = _from_run(run_status, pending)
|
||||
live = _from_run(run_status, pending, end_reason)
|
||||
if live:
|
||||
return live
|
||||
updated = turn.updated_at
|
||||
@@ -202,7 +208,9 @@ def last_status(
|
||||
return "error" if stale else "running"
|
||||
|
||||
|
||||
def _from_run(run_status: str | None, pending: bool) -> str | None:
|
||||
def _from_run(run_status: str | None, pending: bool, end_reason: str | None = None) -> str | None:
|
||||
if run_status == "cancelled" and end_reason == "rejected":
|
||||
return "rejected"
|
||||
return {
|
||||
"queued": "running", "running": "running",
|
||||
"interrupted": "waiting" if pending else "suspended",
|
||||
@@ -236,11 +244,14 @@ async def _statuses(
|
||||
run_of = {cid: turn_run_id(t) for cid, t in turns.items()}
|
||||
run_ids = {r for r in run_of.values() if r}
|
||||
runs: dict[str, str] = {}
|
||||
ended: dict[str, str | None] = {}
|
||||
waiting: set[str] = set()
|
||||
if run_ids:
|
||||
runs = dict((await session.execute(
|
||||
select(Run.id, Run.status).where(Run.id.in_(run_ids))
|
||||
)).all())
|
||||
rows = (await session.execute(
|
||||
select(Run.id, Run.status, Run.end_reason).where(Run.id.in_(run_ids))
|
||||
)).all()
|
||||
runs = {r: st for r, st, _ in rows}
|
||||
ended = {r: reason for r, _, reason in rows}
|
||||
stopped = [r for r, st in runs.items() if st == "interrupted"]
|
||||
if stopped:
|
||||
waiting = set((await session.execute(
|
||||
@@ -251,7 +262,8 @@ async def _statuses(
|
||||
out: dict[str, tuple[str, str | None]] = {}
|
||||
for cid, turn in turns.items():
|
||||
run_id = run_of[cid] if run_of[cid] in runs else None
|
||||
out[cid] = (last_status(turn, runs.get(run_id or ""), run_id in waiting, now),
|
||||
out[cid] = (last_status(turn, runs.get(run_id or ""), run_id in waiting, now,
|
||||
end_reason=ended.get(run_id or "")),
|
||||
run_id)
|
||||
return out
|
||||
|
||||
|
||||
+140
-76
@@ -110,11 +110,12 @@ def _tool_catalog(rows: list[Any], *, builtins: frozenset[str] | None = None) ->
|
||||
|
||||
async def _look_service(session: AsyncSession, payload: GenerateIn, sources: list[Any], *,
|
||||
graph: dict[str, Any] | None, kinds: tuple[str, ...] = ("table", "capability"),
|
||||
) -> Any:
|
||||
"""这一轮的查看服务(copilot_context.look_service)。挑选模型用同一个助手模型,第一次要挑时才拿。"""
|
||||
catalog_view: Any = None) -> Any:
|
||||
"""这一轮的查看服务(copilot_context.look_service)。挑选模型用同一个助手模型,第一次要挑时才拿。catalog_view:表详情
|
||||
里给哪一份数据目录(data/catalog_view.py),不给是当前目录。"""
|
||||
spec = await copilot_model_spec(session, payload, max_tokens=copilot_context.PICK_MAX_TOKENS)
|
||||
return await copilot_context.look_service(sources, spec=spec, graph=graph, kinds=kinds,
|
||||
answer=payload.intent == "answer")
|
||||
answer=payload.intent == "answer", catalog_view=catalog_view)
|
||||
|
||||
|
||||
NODE_REFERENCE = """\
|
||||
@@ -1756,45 +1757,81 @@ def _repair_request(
|
||||
|
||||
|
||||
@router.post("/generate-stream")
|
||||
async def generate_stream(payload: GenerateIn, session: AsyncSession = Depends(get_session)):
|
||||
async def generate_stream(payload: GenerateIn):
|
||||
"""流式版生成:SSE 逐操作推送,前端边收边把节点摆上画布。
|
||||
|
||||
生成过程(copilot_gen)读到 look 就停下、查看、接着写,查看记录作为 look 事件发给前端(前端存进这一轮的
|
||||
meta.looks)。结束时(done 或流断)发一条 final:服务端把累积的图排版、校验后整体给出——过程可见性来自操作流,
|
||||
最终质量仍由和手工编排相同的 layout + validate 保证。模型问了用户(ask)的,这一轮到此为止,不自查修正、不自动运行。
|
||||
生成过程整个在 answer_ops 里(评测直接迭代它,和问数据页走同一条路),这里只包一层 SSE。开流前的拒绝(模型没配好、
|
||||
对话读不了、限定的数据源都不在)发生在 answer_ops 交出第一条操作之前:先取第一条,拒绝照旧是 HTTP 错误,不进流。
|
||||
"""
|
||||
from fastapi.responses import StreamingResponse
|
||||
|
||||
from app.data.sqlcheck import load_checkers
|
||||
ops = answer_ops(payload)
|
||||
first = await ops.__anext__()
|
||||
|
||||
async def event_stream():
|
||||
yield _sse(first)
|
||||
async for op in ops:
|
||||
yield _sse(op)
|
||||
|
||||
return StreamingResponse(
|
||||
event_stream(),
|
||||
media_type="text/event-stream",
|
||||
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
|
||||
)
|
||||
|
||||
|
||||
async def answer_ops(payload: GenerateIn, *, catalog_view: Any = None):
|
||||
"""生成过程:按需求起草、按需查看、自查修正、排版校验,逐条交出操作(op dict)。问数据页(经 SSE)和评测(直接迭代)
|
||||
走的是这同一条路(细案 01 做法 2)。
|
||||
|
||||
读到 look 就停下、查看、接着写,查看记录作为 look 操作交出(前端存进这一轮的 meta.looks)。结束时(done 或流断)交一条
|
||||
final:累积的图排版、校验后整体给出——过程可见性来自操作流,最终质量仍由和手工编排相同的 layout + validate 保证。模型
|
||||
问了用户(ask)的,这一轮到此为止,不自查修正、不自动运行。
|
||||
|
||||
开流前的准备(取数据源、读目录检查器、取助手模型、拼历史、组查看服务)用自己的短会话,在交出第一条操作之前做完:
|
||||
拒绝在这里抛 HTTPException(调用方没迭代到第一条就知道)。catalog_view 是这一轮读哪一份数据目录(data/catalog_view.py:
|
||||
查看的表详情、自查的 SQL 检查器都按它),不给是当前目录。
|
||||
|
||||
final 带 next(chat/turn.next_step:run 运行、ready 只建不跑、fail 判失败),问数据页照它走。最后多交一条 usage
|
||||
(providers/usage.py):{op: usage, by_model, input_tokens, output_tokens, calls, cost_usd,
|
||||
unpriced_calls}——这一轮助手建图(含挑表、接着写、自查修正)花了多少。前面每一条和抽出来之前逐条一样。运行不在这里,
|
||||
不计。
|
||||
"""
|
||||
from app.api.deps import restricted
|
||||
from app.chat import turn
|
||||
from app.data.catalog_view import CatalogView
|
||||
from app.data.sqlcheck import load_checkers
|
||||
from app.db.base import SessionLocal
|
||||
from app.providers import usage
|
||||
|
||||
# 带了对话的,先核对这个人能不能用它(D-d):历史、上一轮的图都从它来
|
||||
await _ensure_conversation_access(session, payload.conversation_id)
|
||||
# 开登录后的 user 角色(D-c):图只能用 chat_policy 放行的节点和工具,提示词先说、自查按同一个判断挡
|
||||
chat_user = restricted()
|
||||
sources = await _sources(session, payload.datasource_ids)
|
||||
scope = {r.name for r in sources} if payload.datasource_ids else None
|
||||
# 基于数据目录的 SQL 检查器:在开流之前读好目录(流里不再碰这个会话),自查和收尾共用
|
||||
checkers = await load_checkers(session, sources)
|
||||
nodes, edges = _start_graph(payload)
|
||||
# 用户摆好的坐标,收尾排版时原样留着
|
||||
pinned = _pinned_positions(payload.base_graph)
|
||||
view = CatalogView.parse(catalog_view)
|
||||
async with SessionLocal() as session:
|
||||
# 带了对话的,先核对这个人能不能用它(D-d):历史、上一轮的图都从它来
|
||||
await _ensure_conversation_access(session, payload.conversation_id)
|
||||
# 开登录后的 user 角色(D-c):图只能用 chat_policy 放行的节点和工具,提示词先说、自查按同一个判断挡
|
||||
chat_user = restricted()
|
||||
sources = await _sources(session, payload.datasource_ids)
|
||||
scope = {r.name for r in sources} if payload.datasource_ids else None
|
||||
# 基于数据目录的 SQL 检查器:在开流之前读好目录(流里不再碰这个会话),自查和收尾共用
|
||||
checkers = await load_checkers(session, sources, catalog_view=view)
|
||||
nodes, edges = _start_graph(payload)
|
||||
# 用户摆好的坐标,收尾排版时原样留着
|
||||
pinned = _pinned_positions(payload.base_graph)
|
||||
|
||||
try:
|
||||
model, model_id = await _op_model(session, payload)
|
||||
except ProviderNotConfigured as e:
|
||||
raise HTTPException(400, _unconfigured(e)) from e
|
||||
try:
|
||||
model, model_id = await _op_model(session, payload)
|
||||
except ProviderNotConfigured as e:
|
||||
raise HTTPException(400, _unconfigured(e)) from e
|
||||
|
||||
human = _with_history(_graph_request(payload), await _history_section(session, payload.conversation_id))
|
||||
# 兜底用:模型只发了改动操作时,把它们重放到这张图上
|
||||
prev_graph = await _previous_graph(session, payload.conversation_id)
|
||||
# 比对工具绑定的基准:画布上是改之前的图;问数据页是上一轮的图(沿用它的节点 id)
|
||||
baseline = copy.deepcopy(
|
||||
(payload.base_graph or {}).get("nodes") or (prev_graph or {}).get("nodes") or [])
|
||||
# 查看服务:读目录、读能力登记在开流之前做完。现有工作流(画布上的原图、问数据追问时上一轮的图)查过的表列在概览里
|
||||
looks = await _look_service(session, payload, sources, graph=payload.base_graph if (
|
||||
payload.base_graph or {}).get("nodes") else prev_graph)
|
||||
human = _with_history(_graph_request(payload), await _history_section(session, payload.conversation_id))
|
||||
# 兜底用:模型只发了改动操作时,把它们重放到这张图上
|
||||
prev_graph = await _previous_graph(session, payload.conversation_id)
|
||||
# 比对工具绑定的基准:画布上是改之前的图;问数据页是上一轮的图(沿用它的节点 id)
|
||||
baseline = copy.deepcopy(
|
||||
(payload.base_graph or {}).get("nodes") or (prev_graph or {}).get("nodes") or [])
|
||||
# 查看服务:读目录、读能力登记在开流之前做完。现有工作流(画布上的原图、问数据追问时上一轮的图)查过的表列在概览里
|
||||
looks = await _look_service(session, payload, sources, graph=payload.base_graph if (
|
||||
payload.base_graph or {}).get("nodes") else prev_graph, catalog_view=view)
|
||||
# 这一份 system 一直用到最后:接着写、自查修正都沿用,不换
|
||||
system = _graph_system(sources, scope, await copilot_context.section(looks), looks, chat_user=chat_user)
|
||||
applier = GraphApplier(nodes, edges)
|
||||
@@ -1803,8 +1840,8 @@ async def generate_stream(payload: GenerateIn, session: AsyncSession = Depends(g
|
||||
checker=GraphChecker(applier, scope=scope, sources=sources, baseline=baseline, checkers=checkers,
|
||||
chat_user=chat_user))
|
||||
|
||||
async def event_stream():
|
||||
yield _sse({"op": "model", "model": model_id})
|
||||
async def steps():
|
||||
yield {"op": "model", "model": model_id}
|
||||
patches = 0
|
||||
try:
|
||||
async for op in _with_heartbeat(gen.draft(human), started=started):
|
||||
@@ -1813,23 +1850,24 @@ async def generate_stream(payload: GenerateIn, session: AsyncSession = Depends(g
|
||||
# reply 先压着(applier.replied):前端收到 reply 就收尾,提案得排在它前面转出
|
||||
if (checked := await catalog_patch_op(op, sources)) is not None:
|
||||
patches += 1
|
||||
yield _sse(checked)
|
||||
yield checked
|
||||
continue
|
||||
yield _sse(op)
|
||||
yield op
|
||||
if applier.replied is not None:
|
||||
# 这句话不需要工作流:后面那套排版校验对着一张空图跑,只会得到"图是空的,先拖一个节点进来"
|
||||
yield _sse(applier.replied)
|
||||
yield applier.replied
|
||||
return
|
||||
if gen.asked is not None and not applier.effective:
|
||||
# 先问清楚才能动手,什么都还没改:这一轮到此为止。照样发一条 final 收尾,带上问题(asked)和没动过的图:
|
||||
# 前端据此把这一轮记成「助手反问」、问题进历史(以前流就这么断了,问数据页一直停在「进行中」),
|
||||
# 不运行、不重排。图给一份对象(画布上原来的图、问数据上一轮的图,都没有就是空图),老前端照样认得
|
||||
yield _sse(_asked_final(gen.asked, payload.base_graph, prev_graph))
|
||||
asked_final = _asked_final(gen.asked, payload.base_graph, prev_graph)
|
||||
yield {**asked_final, "next": turn.next_step(asked_final)}
|
||||
return
|
||||
# 只发了目录修改提案,既没回答也没改图(用户只是在说一条数据事实):补一句回答收尾,
|
||||
# 不能按下面的「模型没有给出任何修改」报错——提案已经摆在用户面前了
|
||||
if patches and all(o.get("op") == "plan" for o in applier.seen):
|
||||
yield _sse({"op": "reply", "text": PATCH_ONLY_REPLY})
|
||||
yield {"op": "reply", "text": PATCH_ONLY_REPLY}
|
||||
return
|
||||
# 流正常结束,却没有 done、也没有一个操作能认(格式跑偏,每一行都被跳过):
|
||||
# 往下走就是拿原图去自查、交付,界面上写「无需修改」——而模型什么都没说
|
||||
@@ -1837,12 +1875,11 @@ async def generate_stream(payload: GenerateIn, session: AsyncSession = Depends(g
|
||||
raise CopilotIncomplete("模型没有给出任何修改,也没有说明原因",
|
||||
hint="画布未修改。可以重试,或把需求描述得更具体")
|
||||
except CopilotIncomplete as e:
|
||||
yield _sse({"op": "error", "message": f"助手本轮未完成:{e}", "hint": e.hint, "detail": e.detail})
|
||||
yield {"op": "error", "message": f"助手本轮未完成:{e}", "hint": e.hint, "detail": e.detail}
|
||||
return
|
||||
except Exception as e: # noqa: BLE001
|
||||
reason, hint = explain_error(e)
|
||||
yield _sse({"op": "error", "message": f"助手本轮未完成:{reason}",
|
||||
"hint": hint, "detail": raw_error(e)})
|
||||
yield {"op": "error", "message": f"助手本轮未完成:{reason}", "hint": hint, "detail": raw_error(e)}
|
||||
return
|
||||
|
||||
# 兜底:一个操作都没落地,而上一轮有图。
|
||||
@@ -1858,7 +1895,7 @@ async def generate_stream(payload: GenerateIn, session: AsyncSession = Depends(g
|
||||
# 以前这些问题照样交付、照样自动开跑——跑到那一步才炸(开发库里四次运行死在
|
||||
# 循环 / 分支条件里套了 {{ }}),前面的步骤白跑,报错还停在半路
|
||||
async for op in _with_heartbeat(gen.repair(), phase="repairing"):
|
||||
yield _sse(op)
|
||||
yield op
|
||||
remaining = _blocking_issues(nodes, edges, scope, sources=sources, baseline=baseline, checkers=checkers,
|
||||
chat_user=chat_user)
|
||||
# 工具绑定变化不挡运行,但要在自查这一步就说出来:工具被改没了的图照样
|
||||
@@ -1871,11 +1908,9 @@ async def generate_stream(payload: GenerateIn, session: AsyncSession = Depends(g
|
||||
# 改不好也照实交付、列出问题,但不自动运行:明知跑不起来的图,开跑只会
|
||||
# 在半路报一个用户看不懂的错
|
||||
autorun = False
|
||||
yield _sse({"op": "check", "status": "failed", "issues": remaining,
|
||||
"warnings": warnings})
|
||||
yield {"op": "check", "status": "failed", "issues": remaining, "warnings": warnings}
|
||||
elif remaining is not None:
|
||||
yield _sse({"op": "check", "status": "passed", "repaired": gen.repaired,
|
||||
"warnings": warnings})
|
||||
yield {"op": "check", "status": "passed", "repaired": gen.repaired, "warnings": warnings}
|
||||
|
||||
# 收尾:排版 + 校验,把最终图整体交付。改图时旧节点留在原处,只排新节点
|
||||
try:
|
||||
@@ -1913,16 +1948,17 @@ async def generate_stream(payload: GenerateIn, session: AsyncSession = Depends(g
|
||||
# 改了图又问了用户(这一轮照样结束、不自动运行):问题跟着 final 交付,前端记进这一轮
|
||||
**({"asked": _asked_of(gen.asked)} if gen.asked is not None else {}),
|
||||
}
|
||||
# 建完之后怎么走(chat/turn.next_step):run 运行、ready 只建不跑、fail 判失败。问数据页照它走,评测用同一个函数
|
||||
final["next"] = turn.next_step(final)
|
||||
except Exception as e: # noqa: BLE001
|
||||
final = {"op": "error", "message": f"模型返回的工作流结构有误:{graph_error(e)}",
|
||||
"hint": "请重试,或把需求描述得更具体", "detail": raw_error(e)}
|
||||
yield _sse(final)
|
||||
yield final
|
||||
|
||||
return StreamingResponse(
|
||||
event_stream(),
|
||||
media_type="text/event-stream",
|
||||
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
|
||||
)
|
||||
spent = usage.meter()
|
||||
async for op in usage.metered(steps(), spent):
|
||||
yield op
|
||||
yield {"op": "usage", **spent.summary()}
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
@@ -2774,13 +2810,31 @@ class ReviewIn(BaseModel):
|
||||
#: 默认只在扫出异常信号时才叫模型。打开它就每次都复核——排查时有用,
|
||||
#: 平时不开:干净的运行没什么可复核的,代价却是每次问数都多一次调用
|
||||
always: bool = False
|
||||
#: 这一轮已经自动试了几次(从 0 数,同 chat/turn.retry_after_review):结果里的 retry_instruction 据此给不给。
|
||||
#: 不传按第一次算
|
||||
attempt: int = Field(default=0, ge=0)
|
||||
|
||||
|
||||
@router.post("/review")
|
||||
async def review_run(
|
||||
payload: ReviewIn, session: AsyncSession = Depends(get_session)
|
||||
) -> dict[str, Any]:
|
||||
"""接住一次编排的产出,判断它能不能就这么交给用户。
|
||||
"""接住一次编排的产出,判断它能不能就这么交给用户(规则见 review_of_run)。这里只核对能不能读这次运行。"""
|
||||
from app.api.deps import PRIVATE_RUN, can_read_run, current_principal
|
||||
from app.db.models import Run
|
||||
|
||||
run = await session.get(Run, payload.run_id)
|
||||
if not run:
|
||||
raise HTTPException(404, "运行记录不存在,可能已被删除")
|
||||
if not await can_read_run(current_principal(), run):
|
||||
raise HTTPException(403, PRIVATE_RUN)
|
||||
return await review_of_run(session, run, question=payload.question, provider=payload.provider,
|
||||
model=payload.model, always=payload.always, attempt=payload.attempt)
|
||||
|
||||
|
||||
async def review_of_run(session: AsyncSession, run: Any, *, question: str, provider: str | None = None,
|
||||
model: str | None = None, always: bool = False, attempt: int = 0) -> dict[str, Any]:
|
||||
"""复核一次运行(问数据页经 /copilot/review、评测直接调,同一个函数)。
|
||||
|
||||
「跑完了」和「答得对」是两回事。运行状态只说明流程走完了,而过程中检索
|
||||
降级、工具报错、agent 步数用满这些事,原先一件都不会进到答案里——用户拿到
|
||||
@@ -2788,47 +2842,57 @@ async def review_run(
|
||||
|
||||
规则层先扫一遍(engine/review.py:scan),没扫出东西就到此为止、直接返回,
|
||||
不花一次模型调用;扫出来了才叫模型重组答案并把异常说破。
|
||||
"""
|
||||
from app.api.deps import PRIVATE_RUN, can_read_run, current_principal
|
||||
from app.db.models import Run, RunEvent
|
||||
from app.engine import review as rv
|
||||
|
||||
run = await session.get(Run, payload.run_id)
|
||||
if not run:
|
||||
raise HTTPException(404, "运行记录不存在,可能已被删除")
|
||||
if not await can_read_run(current_principal(), run):
|
||||
raise HTTPException(403, PRIVATE_RUN)
|
||||
结果是 ReviewResult.as_dict(),另带两项:
|
||||
- retry_instruction:要不要带着原图重来一次(chat/turn.retry_after_review,按 attempt 算这一轮还能不能再试)——要的话
|
||||
是交给助手的那句话(chat/turn.retry_instruction),不要是 null。问数据页照它走;
|
||||
- usage:复核这一次模型调用花了多少(providers/usage.py 的汇总,没调模型时是全 0),这一轮的花费 = 助手 + 运行 + 复核。
|
||||
"""
|
||||
from app.chat import turn
|
||||
from app.db.models import RunEvent
|
||||
from app.engine import review as rv
|
||||
from app.providers import usage
|
||||
|
||||
rows = list((await session.execute(
|
||||
select(RunEvent).where(RunEvent.run_id == payload.run_id).order_by(RunEvent.seq)
|
||||
select(RunEvent).where(RunEvent.run_id == run.id).order_by(RunEvent.seq)
|
||||
)).scalars())
|
||||
|
||||
output = run.output if isinstance(run.output, dict) else None
|
||||
signals = rv.scan(rows, output)
|
||||
answer = rv.answer_text(output)
|
||||
spent = usage.meter()
|
||||
|
||||
if not signals and not payload.always:
|
||||
return rv.ReviewResult(verdict="ok", signals=[]).as_dict()
|
||||
def done(result: Any) -> dict[str, Any]:
|
||||
out = result.as_dict()
|
||||
out["retry_instruction"] = (
|
||||
turn.retry_instruction(question, turn.review_reason(out), turn.has_nodes(run.graph))
|
||||
if turn.retry_after_review(out, attempt) else None)
|
||||
out["usage"] = spent.summary()
|
||||
return out
|
||||
|
||||
if not signals and not always:
|
||||
return done(rv.ReviewResult(verdict="ok", signals=[]))
|
||||
|
||||
try:
|
||||
spec = await copilot_model_spec(session, GenerateIn(
|
||||
instruction="review", provider=payload.provider, model=payload.model,
|
||||
instruction="review", provider=provider, model=model,
|
||||
), max_tokens=4096)
|
||||
model, _ = await get_chat_model(session, spec)
|
||||
chat_model, _ = await get_chat_model(session, spec)
|
||||
except (ProviderNotConfigured, HTTPException) as e:
|
||||
# 没配模型不该让这一轮挂掉:规则层已经知道出了什么事,照样说得出话
|
||||
return rv.ReviewResult(
|
||||
return done(rv.ReviewResult(
|
||||
verdict="annotated",
|
||||
note=";".join(s.detail for s in signals) or (
|
||||
_unconfigured(e) if isinstance(e, ProviderNotConfigured) else str(e.detail)
|
||||
),
|
||||
retry=rv.should_retry(signals),
|
||||
signals=signals, severity=rv.worst(signals),
|
||||
).as_dict()
|
||||
))
|
||||
|
||||
result = await rv.review(
|
||||
model, question=payload.question, answer=answer, signals=signals,
|
||||
# 报告撰写节点产出的答案逐段对应着证据文档:只能加说明,不能改写
|
||||
locked=bool(output and output.get("_evidence")),
|
||||
)
|
||||
return result.as_dict()
|
||||
with usage.metering(spent):
|
||||
result = await rv.review(
|
||||
chat_model, question=question, answer=answer, signals=signals,
|
||||
# 报告撰写节点产出的答案逐段对应着证据文档:只能加说明,不能改写
|
||||
locked=bool(output and output.get("_evidence")),
|
||||
)
|
||||
return done(result)
|
||||
|
||||
@@ -293,7 +293,7 @@ async def blueprint_stream(payload: BlueprintStreamIn, session: AsyncSession = D
|
||||
system = "\n\n".join(part for part in (
|
||||
BLUEPRINT_SYSTEM_PROMPT, _templates_text(personal=personal), await looks.overview(), BLUEPRINT_PROTOCOL_PROMPT,
|
||||
looks.guide(), BLUEPRINT_LOOK_WHEN_PROMPT) if part)
|
||||
missing = [i for i in blueprint_issues(spec, for_="trial") if i["level"] == "error"]
|
||||
missing = [i for i in blueprint_issues(spec, for_="trial", graph=row.graph) if i["level"] == "error"]
|
||||
human = copilot._with_history(BLUEPRINT_REQUEST_PROMPT.format(
|
||||
spec=json.dumps(spec, ensure_ascii=False), instruction=instruction,
|
||||
issues=BLUEPRINT_ISSUES_PROMPT.format(lines="\n".join(f"- {i['path']} {i['message']}" for i in missing))
|
||||
|
||||
@@ -65,16 +65,17 @@ SYSTEM_LOOK_UPGRADE_NEED = "要升级的工作流里,查库节点的 SQL 用
|
||||
|
||||
|
||||
async def look_service(sources: list[Any], *, spec: ModelSpec | None, graph: Mapping[str, Any] | None = None,
|
||||
kinds: tuple[str, ...] = ("table", "capability"), answer: bool = False) -> LookService:
|
||||
kinds: tuple[str, ...] = ("table", "capability"), answer: bool = False,
|
||||
catalog_view: Any = None) -> LookService:
|
||||
"""按入口组装这一轮的查看服务。读目录、读能力登记都用各自的短会话,开流前后都能调。
|
||||
|
||||
spec 是助手的模型规格:挑选模型第一次要用时才按它拿(关思考、额度按 look/pick.PICK_MAX_TOKENS)。graph 是现有工作流,概览里列出它查过的表。
|
||||
answer:问数据(意图为回答),能力只列只读的。
|
||||
answer:问数据(意图为回答),能力只列只读的。catalog_view:表详情里给哪一份数据目录(data/catalog_view.py),不给是当前目录。
|
||||
"""
|
||||
picker = Picker(spec)
|
||||
made: list[Any] = []
|
||||
if "table" in kinds:
|
||||
made.append(await TableKind.load(sources, graph=graph, picker=picker))
|
||||
made.append(await TableKind.load(sources, graph=graph, picker=picker, catalog_view=catalog_view))
|
||||
if "capability" in kinds:
|
||||
made.append(await CapabilityKind.load(picker=picker, read_only=answer))
|
||||
return LookService(made)
|
||||
|
||||
@@ -18,7 +18,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.api import health
|
||||
from app.api.coded import PROFILE_SETTINGS_INVALID, CodedHTTPException
|
||||
from app.api.deps import signer_of
|
||||
from app.api.deps import signer_of, signer_verified
|
||||
from app.core.errors import AUTH, NETWORK, TIMEOUT, classify, explain, raw
|
||||
from app.core.config import settings
|
||||
from app.core.crypto import encrypt, mask
|
||||
@@ -1104,7 +1104,8 @@ async def purge_raw(
|
||||
if imp.raw_state == "purged":
|
||||
raise CodedHTTPException(409, "这次导入的原件已经清除", "raw_already_purged")
|
||||
try:
|
||||
purged = await table_versions.purge_import_raw(session, imp, reason=reason, signed_by=signed_by)
|
||||
purged = await table_versions.purge_import_raw(session, imp, reason=reason, signed_by=signed_by,
|
||||
signer_verified=signer_verified())
|
||||
except table_versions.StoreUnavailable as e:
|
||||
raise CodedHTTPException(503, str(e), "store_unavailable") from e
|
||||
deleted, others = purged.deleted, purged.others
|
||||
|
||||
+22
-10
@@ -26,9 +26,11 @@ from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
from urllib.parse import unquote
|
||||
|
||||
from fastapi import Depends, HTTPException
|
||||
from fastapi import Depends
|
||||
from starlette.requests import HTTPConnection
|
||||
|
||||
from app.api.coded import CHAT_FORBIDDEN, ROLE_REQUIRED, RUN_FORBIDDEN, RUN_PRIVATE, CodedHTTPException
|
||||
|
||||
#: 登录方式
|
||||
AUTH_MODES = ("off", "local", "oidc")
|
||||
#: 角色由高到低
|
||||
@@ -105,6 +107,15 @@ def signer_of(given: Any, header: str | None, limit: int = 100) -> str | None:
|
||||
return decode_actor(header)
|
||||
|
||||
|
||||
def signer_verified() -> bool:
|
||||
"""signer_of 记下的署名是不是认证过的身份:开了登录、这次请求的身份认证过时为真;关登录时署名是自报的,为假。
|
||||
|
||||
导入清单、累积快照清单、作废记录、清除原件记录的「是否认证」记这个(docs/design/next-round/05-已知遗留.md (c)),
|
||||
和审计表、待办派单、run.origin 的 verified 同一个口径。证据要在生成那一刻定下:调用方和 signed_by 一起往下传。"""
|
||||
principal = _principal.get()
|
||||
return principal is not None and principal.mode != "off" and bool(principal.verified)
|
||||
|
||||
|
||||
def current_principal() -> Principal:
|
||||
"""这次请求的完整身份。关登录时:等同 admin、未认证,username 是中间件按 X-Actor 解出的自报署名;中间件没写
|
||||
(不在请求里,比如测试直接调函数、启动时的整理、调度循环)时同样等同 admin、未认证,username 为空。"""
|
||||
@@ -140,7 +151,7 @@ async def can_read_artifact(principal: Principal, artifact_id: str) -> bool:
|
||||
|
||||
|
||||
def ensure_role(role: str) -> None:
|
||||
"""接口内部用:至少是 role,否则 401(没登录)/ 403(角色不够)。关登录时放行。
|
||||
"""接口内部用:至少是 role,否则 401(没登录,auth_required)/ 403(角色不够,role_required)。关登录时放行。
|
||||
用在要看请求内容才知道要什么角色的地方(比如 MCP 的 stdio 只有管理员能建)。"""
|
||||
from app.auth import require
|
||||
|
||||
@@ -169,8 +180,8 @@ async def ensure_conversation_access(session: Any, conversation_id: str | None)
|
||||
return await check(session, conversation_id)
|
||||
|
||||
|
||||
def _forbidden() -> HTTPException:
|
||||
return HTTPException(403, "没有权限查看这次运行")
|
||||
def _forbidden() -> CodedHTTPException:
|
||||
return CodedHTTPException(403, "没有权限查看这次运行", RUN_FORBIDDEN)
|
||||
|
||||
|
||||
#: user 角色在自己的问数据运行上能读的(GET /api/runs/{id} 之后的部分):运行记录、事件、运行时的图,加上实时流
|
||||
@@ -235,13 +246,13 @@ async def _run_scope(conn: HTTPConnection) -> None:
|
||||
return
|
||||
ensure_role("user")
|
||||
if not user_can_chat():
|
||||
raise HTTPException(403, _RUNS_FOR_BUILDERS)
|
||||
raise CodedHTTPException(403, _RUNS_FOR_BUILDERS, ROLE_REQUIRED)
|
||||
method = conn.scope.get("method", "GET")
|
||||
run_id = conn.path_params.get("run_id")
|
||||
if run_id is None:
|
||||
if method == "POST" and conn.url.path.rstrip("/") == "/api/runs":
|
||||
return
|
||||
raise HTTPException(403, _RUNS_FOR_BUILDERS)
|
||||
raise CodedHTTPException(403, _RUNS_FOR_BUILDERS, ROLE_REQUIRED)
|
||||
run = await _load_run(run_id)
|
||||
rest = conn.url.path.rstrip("/").removeprefix(f"/api/runs/{run_id}")
|
||||
if method == "GET" and rest in _CHAT_RUN_READS:
|
||||
@@ -252,7 +263,7 @@ async def _run_scope(conn: HTTPConnection) -> None:
|
||||
raise _forbidden()
|
||||
if method == "POST" and rest == "/cancel":
|
||||
return
|
||||
raise HTTPException(403, "没有权限:这项操作需要搭建者及以上的角色")
|
||||
raise CodedHTTPException(403, "没有权限:这项操作需要搭建者及以上的角色", ROLE_REQUIRED)
|
||||
|
||||
|
||||
async def _load_run(run_id: str | None) -> Any:
|
||||
@@ -282,7 +293,7 @@ async def _refuse_private(conn: HTTPConnection, principal: Principal) -> None:
|
||||
from fastapi import WebSocketException, status
|
||||
|
||||
raise WebSocketException(code=status.WS_1008_POLICY_VIOLATION)
|
||||
raise HTTPException(403, PRIVATE_RUN)
|
||||
raise CodedHTTPException(403, PRIVATE_RUN, RUN_PRIVATE)
|
||||
|
||||
|
||||
async def _private_run_scope(conn: HTTPConnection) -> None:
|
||||
@@ -310,7 +321,7 @@ async def _evidence_scope(conn: HTTPConnection) -> None:
|
||||
return
|
||||
ensure_role("user")
|
||||
if conn.scope.get("method", "GET") != "GET":
|
||||
raise HTTPException(403, "没有权限:这项操作需要搭建者及以上的角色")
|
||||
raise CodedHTTPException(403, "没有权限:这项操作需要搭建者及以上的角色", ROLE_REQUIRED)
|
||||
run_id = conn.path_params.get("run_id")
|
||||
if not run_id or not await can_read_run(principal, run_id):
|
||||
raise _forbidden()
|
||||
@@ -327,7 +338,8 @@ async def _chat_scope(conn: HTTPConnection) -> None:
|
||||
method = conn.scope.get("method", "GET")
|
||||
if user_can_chat() and (path.startswith("/api/conversations") or (method, path) in _CHAT_ROUTES):
|
||||
return
|
||||
raise HTTPException(403, "没有权限:问数据和助手需要搭建者及以上的角色,或由管理员为使用者开启问数据")
|
||||
raise CodedHTTPException(403, "没有权限:问数据和助手需要搭建者及以上的角色,或由管理员为使用者开启问数据",
|
||||
CHAT_FORBIDDEN)
|
||||
|
||||
|
||||
#: user 角色开了问数据时能用的其余接口:问数据的生成流、读当前助手模型、问数据页选数据源用的数据源名单
|
||||
|
||||
@@ -2851,7 +2851,7 @@ async def export_report_html(run_id: str, report: str | None = None) -> Response
|
||||
if run is None: # 两次读之间被删掉了
|
||||
raise CodedHTTPException(404, "运行记录不存在", RUN_NOT_FOUND)
|
||||
info = {"id": run.id, "workflow_id": run.workflow_id, "workflow_name": run.workflow_name,
|
||||
"run_class": run.run_class, "status": run.status,
|
||||
"run_class": run.run_class, "status": run.status, "end_reason": run.end_reason,
|
||||
"started_at": run.started_at, "finished_at": run.finished_at}
|
||||
data = build_report(info, report_node=chosen.node_id, doc_artifact=chosen.doc_artifact,
|
||||
doc_hash_ok=chosen.hash_ok, doc=doc, seal=sealed.seal, chains=chains, refs=refs,
|
||||
|
||||
@@ -1,6 +1,5 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from collections import defaultdict
|
||||
from typing import Any
|
||||
|
||||
@@ -45,11 +44,14 @@ async def tool_usage(
|
||||
merged.extend(agent.get("tools") or [])
|
||||
granted[node.id] = sorted(set(merged))
|
||||
|
||||
from app.engine.origin import counted_runs
|
||||
|
||||
# 只算计入统计的运行(engine/origin.counted_runs):评测、口径对账、召回替人跑的运行用到了哪些工具不说明白名单该不该收
|
||||
run_ids = [
|
||||
rid
|
||||
for (rid,) in await session.execute(
|
||||
select(Run.id)
|
||||
.where(Run.workflow_id == workflow_id)
|
||||
.where(Run.workflow_id == workflow_id, counted_runs())
|
||||
.order_by(Run.created_at.desc())
|
||||
.limit(limit_runs)
|
||||
)
|
||||
@@ -95,62 +97,39 @@ async def exploratory_clusters(
|
||||
) -> dict[str, Any]:
|
||||
"""探索性问题聚类:同一个问题被反复问,就该长成模板里的固定一节。
|
||||
|
||||
这就是模板生长的触发器。用本地 embedding 做贪心聚类,
|
||||
出现 ≥3 次的簇标记为晋升候选。
|
||||
这就是模板生长的触发器。取问题、聚类全站只有一份(app/chat/questions.py):只取问题原文(不含 history),问数据一轮
|
||||
一条(自动重来不多算),评测这类系统派生的运行、别人的个人应用刷新和个人蓝图试运行都不进。出现 ≥3 次的簇标记为晋升
|
||||
候选。embedder 是聚类用的向量模型:本地哈希的语义能力弱,结果只是下限。
|
||||
"""
|
||||
from sqlalchemy import not_
|
||||
|
||||
from app.api.deps import current_principal
|
||||
from app.auth.access import hidden_runs
|
||||
from app.memory.embeddings import cosine, embed_texts
|
||||
from app.chat.questions import cluster_questions, embedder, recent_questions
|
||||
|
||||
rows = list(
|
||||
await session.execute(
|
||||
select(Run.id, Run.input, Run.workflow_name, Run.created_at)
|
||||
# 别人的个人应用刷新、个人蓝图试运行只给拥有者(同运行列表),不进聚类
|
||||
.where(Run.run_class != "formal", not_(hidden_runs(current_principal())))
|
||||
.order_by(Run.created_at.desc())
|
||||
.limit(limit)
|
||||
)
|
||||
)
|
||||
items = []
|
||||
for rid, input_payload, wf_name, created in rows:
|
||||
text = json.dumps(input_payload or {}, ensure_ascii=False, sort_keys=True)
|
||||
if len(text) <= 4: # 空输入没有聚类意义
|
||||
continue
|
||||
items.append({"run_id": rid, "text": text, "workflow": wf_name,
|
||||
"at": created.isoformat() if created else None})
|
||||
if not items:
|
||||
return {"clusters": [], "considered": 0}
|
||||
questions = await recent_questions(session, current_principal(), limit=limit)
|
||||
if not questions:
|
||||
return {"clusters": [], "considered": 0, "embedder": embedder()}
|
||||
clusters = await cluster_questions(questions, threshold=threshold)
|
||||
|
||||
vectors = await embed_texts([i["text"] for i in items])
|
||||
clusters: list[dict[str, Any]] = []
|
||||
for i, item in enumerate(items):
|
||||
placed = False
|
||||
for cluster in clusters:
|
||||
if cosine(vectors[i], vectors[cluster["seed"]]) >= threshold:
|
||||
cluster["members"].append(item)
|
||||
placed = True
|
||||
break
|
||||
if not placed:
|
||||
clusters.append({"seed": i, "members": [item]})
|
||||
def _at(value: Any) -> str | None:
|
||||
return value.isoformat() if value else None
|
||||
|
||||
out = [
|
||||
{
|
||||
"count": len(c["members"]),
|
||||
"representative": c["members"][0]["text"][:200],
|
||||
"workflow": c["members"][0]["workflow"],
|
||||
"run_ids": [m["run_id"] for m in c["members"]][:20],
|
||||
"first_at": c["members"][-1]["at"],
|
||||
"last_at": c["members"][0]["at"],
|
||||
"promote_candidate": len(c["members"]) >= 3,
|
||||
"count": c.count,
|
||||
"representative": c.members[0].text[:200],
|
||||
"workflow": c.members[0].workflow,
|
||||
"run_ids": [m.run_id for m in c.members if m.run_id][:20],
|
||||
"turn_ids": [m.turn_id for m in c.members if m.turn_id][:20],
|
||||
"first_at": _at(c.members[-1].at),
|
||||
"last_at": _at(c.members[0].at),
|
||||
"promote_candidate": c.count >= 3,
|
||||
}
|
||||
for c in clusters
|
||||
if len(c["members"]) >= 2
|
||||
if c.count >= 2
|
||||
]
|
||||
out.sort(key=lambda c: c["count"], reverse=True)
|
||||
return {
|
||||
"clusters": out,
|
||||
"considered": len(items),
|
||||
"considered": len(questions),
|
||||
"embedder": embedder(),
|
||||
"hint": "被标记为可固化的问题反复出现,建议用「提取模板」把它固定为工作流中的一步",
|
||||
}
|
||||
|
||||
@@ -27,6 +27,9 @@ router = APIRouter(prefix="/api", tags=["identity"])
|
||||
#: 人和组的名录(GET /api/directory):只读、给搭建者及以上(main.py 挂 require_role("builder")),岗位绑定、应用授权的
|
||||
#: 选人控件据此给候选、核对填的名字在不在。只有用户名、显示名、是否停用和组名,不含邮箱、角色、登录时间
|
||||
directory_router = APIRouter(prefix="/api", tags=["identity"])
|
||||
#: 使用者选人的候选(GET /api/directory/candidates,05-已知遗留 (b)):任何登录的人都能调(main.py 挂 require_role("user")),
|
||||
#: 按用途给候选池,只有用户名和显示名
|
||||
candidates_router = APIRouter(prefix="/api", tags=["identity"])
|
||||
|
||||
|
||||
def _iso(value: Any) -> str | None:
|
||||
@@ -224,6 +227,56 @@ async def directory(session: AsyncSession = Depends(get_session)) -> dict[str, A
|
||||
"groups": [{"name": g.name, "label": g.label} for g in known.groups.values()]}
|
||||
|
||||
|
||||
@candidates_router.get("/directory/candidates")
|
||||
async def candidates(purpose: str, app_id: str | None = None, entry: str | None = None, task_id: str | None = None,
|
||||
field: str | None = None, q: str | None = None, limit: int = 20,
|
||||
session: AsyncSession = Depends(get_session)) -> dict[str, Any]:
|
||||
"""使用者选人时的候选:{users: [{username, display_name}], groups: [{name, label}], truncated, needs_query?}。
|
||||
|
||||
不给是否停用(停用的直接不列)、角色、邮箱、组成员。purpose=form:表单里的选人控件,带 app_id 和 entry(入口表单)
|
||||
或 task_id(待办表单)、field(哪个控件),池按字段的 x-principal 算、缺省是能用这个应用的人,要求调用的人能用这个
|
||||
应用(否则 403);应用授权给了所有人时不带前缀 q 不列(needs_query)。purpose=review:个人应用提交审核,启用的搭建者
|
||||
和管理员。q 按用户名或显示名的开头筛,limit 最多 50。开了登录时提交的值按同一个池核对(auth/directory.principal_errors)。"""
|
||||
from app.api.coded import CodedHTTPException
|
||||
from app.api.deps import current_principal
|
||||
from app.apps.access import app_role, at_least
|
||||
from app.apps.tree import entry_of, resolve_form
|
||||
from app.auth.directory import PURPOSES, field_pool, principal_fields, review_pool
|
||||
from app.db.models_apps import App, AppVersion
|
||||
from app.db.models_process import Task
|
||||
|
||||
if purpose not in PURPOSES:
|
||||
raise CodedHTTPException(422, f"候选的用途只能是 {' / '.join(PURPOSES)}", "candidates_purpose")
|
||||
if purpose == "review":
|
||||
return (await review_pool(session)).pick(q, limit) | {"groups": []}
|
||||
principal = current_principal()
|
||||
bindings: dict[str, Any] | None = None
|
||||
form: Any = None
|
||||
task = await session.get(Task, task_id) if task_id else None
|
||||
app = await session.get(App, task.app_id if task is not None else app_id) if (task or app_id) else None
|
||||
if app is None or (task_id and task is None):
|
||||
raise CodedHTTPException(404, "应用或待办不存在", "candidates_not_found")
|
||||
if not at_least(await app_role(session, principal, app), "use"):
|
||||
raise CodedHTTPException(403, "你没有这个应用的使用权限,看不到它的候选人", "candidates_forbidden")
|
||||
if task is not None:
|
||||
form = task.form
|
||||
if task.app_version is not None:
|
||||
bindings = await session.scalar(select(AppVersion.bindings).where(
|
||||
AppVersion.app_id == app.id, AppVersion.version == task.app_version))
|
||||
else:
|
||||
version = (await session.execute(select(AppVersion).where(
|
||||
AppVersion.app_id == app.id, AppVersion.version == app.published_version))).scalar_one_or_none() \
|
||||
if app.published_version else None
|
||||
spec = version.spec if version is not None else {}
|
||||
found = entry_of(spec or {}, entry or "") if entry else None
|
||||
form = resolve_form(spec or {}, found.get("form")) if found else None
|
||||
bindings = version.bindings if version is not None else None
|
||||
target = principal_fields(form).get(field or "")
|
||||
if target is None:
|
||||
raise CodedHTTPException(404, "表单里没有这个选人控件", "candidates_not_found")
|
||||
return (await field_pool(session, app, bindings, target)).pick(q, limit)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 审计
|
||||
|
||||
|
||||
|
||||
@@ -21,7 +21,7 @@ from app.core.crypto import encrypt
|
||||
from app.db.base import SessionLocal
|
||||
from app.db.models import NotifyChannel, NotifyDelivery
|
||||
from app.notify import core
|
||||
from app.notify.service import EVENTS, EVENT_LABEL, channel_out
|
||||
from app.notify.service import EVENTS, channel_out, subscribe_problem
|
||||
|
||||
router = APIRouter(prefix="/api/notify", tags=["notify"])
|
||||
|
||||
@@ -61,12 +61,14 @@ async def _safe_personal(scope: str, url: str) -> None:
|
||||
raise HTTPException(422, problem)
|
||||
|
||||
|
||||
def _validated(payload: ChannelIn, *, url: str | None) -> None:
|
||||
def _validated(payload: ChannelIn, *, url: str | None, saved: list[str] | None = None) -> None:
|
||||
"""saved 是渠道已经订阅着的事件:只核对这次新勾的(notify/service.subscribe_problem)。以前存下的照旧——当时界面对
|
||||
哪种渠道都列全部事件,现在才按事件推给哪几条路核对,不能让这些渠道连开关、改名都 422。"""
|
||||
if payload.format not in core.FORMATS:
|
||||
raise HTTPException(422, f"消息格式只能是{'、'.join(core.FORMAT_LABEL.values())}之一")
|
||||
unknown = [e for e in payload.events if e not in EVENTS]
|
||||
if unknown:
|
||||
raise HTTPException(422, f"不认识的通知事件:{'、'.join(unknown)}。可选:{'、'.join(EVENT_LABEL.values())}")
|
||||
for event in dict.fromkeys(payload.events):
|
||||
if event not in (saved or ()) and (problem := subscribe_problem(event, payload.scope)):
|
||||
raise HTTPException(422, problem)
|
||||
if url is not None and (problem := core.format_problem(payload.format, url)):
|
||||
raise HTTPException(422, problem)
|
||||
|
||||
@@ -84,8 +86,10 @@ async def list_channels() -> dict[str, Any]:
|
||||
|
||||
|
||||
@router.get("/events")
|
||||
async def list_events() -> list[dict[str, str]]:
|
||||
return [{"event": e, "label": EVENT_LABEL[e]} for e in EVENTS]
|
||||
async def list_events() -> list[dict[str, Any]]:
|
||||
"""登记的通知事件和推给哪几条路(typesApp.ts 的 NotifyEventRow):设置页、应用的通知按 scopes 列可订阅的事件。
|
||||
scopes 是空的,是登记了、还没有推送的,照实给出来,由界面不列。"""
|
||||
return [{"event": name, "label": event.label, "scopes": list(event.scopes)} for name, event in EVENTS.items()]
|
||||
|
||||
|
||||
@router.post("/channels", status_code=201)
|
||||
@@ -150,9 +154,9 @@ async def update_channel(channel_id: str, payload: ChannelIn) -> dict[str, Any]:
|
||||
from app.core.crypto import decrypt
|
||||
|
||||
current = decrypt(row.url) or ""
|
||||
_validated(payload, url=current or None)
|
||||
_validated(payload, url=current or None, saved=list(row.events or []))
|
||||
else:
|
||||
_validated(payload, url=url)
|
||||
_validated(payload, url=url, saved=list(row.events or []))
|
||||
await _safe_personal(row.scope, url)
|
||||
row.url = encrypt(url)
|
||||
row.name, row.format = payload.name.strip(), payload.format
|
||||
|
||||
+54
-7
@@ -56,6 +56,9 @@ class RunOut(BaseModel):
|
||||
input: dict[str, Any]
|
||||
output: dict[str, Any]
|
||||
error: str | None
|
||||
#: 结束在 cancelled 的原因(docs/design/next-round/00-总方案.md X9):rejected 人工驳回、stopped 手动停止。
|
||||
#: 其余终态、加这一列之前的运行为空。界面、通知、定时推送都读这一个字段,不再解析报错原文
|
||||
end_reason: str | None = None
|
||||
usage: dict[str, Any]
|
||||
run_class: str | None = "exploratory"
|
||||
version: int | None = None
|
||||
@@ -171,13 +174,54 @@ async def list_runs(
|
||||
以前只有 limit:待审批的运行排在第 145 位,列表只取 100 条,点了徽标也找不到它。
|
||||
筛选都走服务端,翻页用 created_at 做游标(before 传上一页最后一条的 created_at)。
|
||||
别人的个人应用刷新、个人蓝图试运行不列(只给拥有者,auth/access.hidden_runs),管理员也一样。
|
||||
没按来源筛时,系统派生的运行(评测、口径对账、口径召回:engine/origin.SYSTEM_ORIGINS)默认不列——几十道评测题的运行
|
||||
会把真实用户的运行挤出第一页;按来源点名就列。没列出几次由 GET /api/runs/unlisted 照实说,界面写在列表底下。
|
||||
"""
|
||||
stmt = _filtered(select(Run), workflow_id=workflow_id, status=status, run_class=run_class, q=q,
|
||||
schedule_id=schedule_id, origin_kind=origin_kind)
|
||||
stmt = stmt.order_by(Run.created_at.desc()).limit(limit)
|
||||
if before is not None:
|
||||
stmt = stmt.where(Run.created_at < before)
|
||||
return list((await session.execute(stmt)).scalars())
|
||||
|
||||
|
||||
@router.get("/unlisted")
|
||||
async def unlisted_runs(
|
||||
workflow_id: str | None = Query(default=None),
|
||||
status: str | None = Query(default=None),
|
||||
run_class: str | None = None,
|
||||
q: str | None = Query(default=None),
|
||||
schedule_id: str | None = Query(default=None),
|
||||
origin_kind: str | None = Query(default=None),
|
||||
session: AsyncSession = Depends(get_session),
|
||||
) -> dict[str, Any]:
|
||||
"""同样的筛选下,运行列表因为「系统派生的运行默认不列」而没列出的有几次:{total, by_origin: {来源: 次数}}。
|
||||
|
||||
按来源筛了的(origin_kind 给了)什么都不藏,total 为 0。别人的个人应用刷新、个人蓝图试运行不算在内:它们不是「没列出」,
|
||||
是读不了。"""
|
||||
from sqlalchemy import func
|
||||
|
||||
from app.engine.origin import SYSTEM_ORIGINS
|
||||
|
||||
if _csv(origin_kind):
|
||||
return {"total": 0, "by_origin": {}}
|
||||
stmt = _filtered(select(Run.origin_kind, func.count()), workflow_id=workflow_id, status=status, run_class=run_class,
|
||||
q=q, schedule_id=schedule_id, origin_kind=",".join(SYSTEM_ORIGINS))
|
||||
rows = (await session.execute(stmt.group_by(Run.origin_kind))).all()
|
||||
by_origin = {kind: int(n) for kind, n in rows if kind and n}
|
||||
return {"total": sum(by_origin.values()), "by_origin": by_origin}
|
||||
|
||||
|
||||
def _filtered(stmt: Any, *, workflow_id: str | None, status: str | None, run_class: str | None, q: str | None,
|
||||
schedule_id: str | None, origin_kind: str | None) -> Any:
|
||||
"""运行列表的筛选条件(列表和「没列出几次」共用一份)。"""
|
||||
from sqlalchemy import not_
|
||||
|
||||
from app.api.deps import current_principal
|
||||
from app.auth.access import hidden_runs
|
||||
from app.engine.origin import counted_runs
|
||||
|
||||
stmt = select(Run).where(not_(hidden_runs(current_principal()))).order_by(Run.created_at.desc()).limit(limit)
|
||||
stmt = stmt.where(not_(hidden_runs(current_principal())))
|
||||
if workflow_id == UNSAVED:
|
||||
stmt = stmt.where(Run.workflow_id.is_(None))
|
||||
elif workflow_id:
|
||||
@@ -193,9 +237,10 @@ async def list_runs(
|
||||
if origins := _csv(origin_kind):
|
||||
kinds = [o for o in origins if o != NO_ORIGIN]
|
||||
stmt = stmt.where(or_(Run.origin_kind.in_(kinds), *([Run.origin_kind.is_(None)] if NO_ORIGIN in origins else [])))
|
||||
if before is not None:
|
||||
stmt = stmt.where(Run.created_at < before)
|
||||
return list((await session.execute(stmt)).scalars())
|
||||
else:
|
||||
# 没点名来源:只列计入统计的运行,系统派生的(评测、口径对账、召回)不列;来源为空的老运行照常列出
|
||||
stmt = stmt.where(counted_runs())
|
||||
return stmt
|
||||
|
||||
|
||||
def _csv(value: str | None) -> list[str]:
|
||||
@@ -289,11 +334,13 @@ async def cancel_run(run_id: str, x_actor: str | None = Header(default=None)) ->
|
||||
run = await session.get(Run, run_id)
|
||||
if run is None:
|
||||
raise HTTPException(404, "运行记录不存在")
|
||||
raise HTTPException(409, _cannot_cancel(run.status))
|
||||
raise HTTPException(409, _cannot_cancel(run.status, run.end_reason))
|
||||
return {"ok": True, "status": outcome}
|
||||
|
||||
|
||||
def _cannot_cancel(status: str) -> str:
|
||||
def _cannot_cancel(status: str, end_reason: str | None = None) -> str:
|
||||
if status == "cancelled" and end_reason == "rejected":
|
||||
return "这次运行已被驳回结束,没有要停止的。"
|
||||
return {
|
||||
"succeeded": "这次运行已完成,没有要停止的。",
|
||||
"failed": "这次运行已失败结束,没有要停止的;如需继续,请点「继续运行」。",
|
||||
@@ -396,7 +443,7 @@ async def continue_run(
|
||||
except KeyError as e:
|
||||
raise _missing(e) from e
|
||||
except ContinueRefused as e:
|
||||
# 正式运行带图(formal_graph_locked)、写入结果未知还没核实(action_unresolved):带机器码
|
||||
# 正式运行带图(formal_graph_locked)、写入结果未知还没核实(action_unresolved)、被驳回了的(run_rejected):带机器码
|
||||
raise CodedHTTPException(409, str(e), e.code) from e
|
||||
except ValueError as e:
|
||||
raise HTTPException(409, str(e)) from e
|
||||
|
||||
@@ -491,6 +491,8 @@ async def create_schedule_row(session: AsyncSession, body: dict[str, Any], *, ac
|
||||
values = schedule_values(body)
|
||||
hook = values.pop("webhook")
|
||||
workflow = await _check_workflow(session, values["workflow_id"], values["run_class"], values["input_template"])
|
||||
# 目标的几列只经 scheduler/service 的一处封装写(工作流目标:target_id 同 workflow_id)
|
||||
values |= service.target_columns("workflow", values["workflow_id"])
|
||||
row = Schedule(**values, workflow_name=workflow.name,
|
||||
next_run_at=service.next_run(values["cron"], values["timezone"], values["enabled"]),
|
||||
created_by=actor, **_webhook_columns(hook))
|
||||
@@ -528,7 +530,8 @@ async def update_schedule(schedule_id: str, request: Request,
|
||||
if "name" in body:
|
||||
values["name"] = _name(body["name"])
|
||||
if "workflow_id" in body:
|
||||
values["workflow_id"] = _workflow_id(body["workflow_id"])
|
||||
# 目标的几列一起改(scheduler/service.target_columns):只改 workflow_id 的话 target_id 还指着原来的工作流
|
||||
values |= service.target_columns("workflow", _workflow_id(body["workflow_id"]))
|
||||
if "cron" in body:
|
||||
values["cron"] = _cron(body["cron"])
|
||||
if "timezone" in body:
|
||||
|
||||
+40
-27
@@ -10,7 +10,8 @@ from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.api import health
|
||||
from app.api.coded import CodedHTTPException
|
||||
from app.api.coded import PUBLIC_URL_ENV_LOCKED, CodedHTTPException
|
||||
from app.core import public_url
|
||||
from app.core.errors import explain, not_configured, raw
|
||||
from app.core.config import settings as app_settings
|
||||
from app.core.crypto import encrypt, mask
|
||||
@@ -604,35 +605,32 @@ def catalog_draft_values(stored: Any) -> dict[str, Any]:
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 通知:定时任务推送 Webhook 时拼运行链接用的对外访问地址
|
||||
# 对外访问地址:通知、Webhook 里的链接,单点登录的回调,来源核对(core/public_url.py)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
#: 设置里 notify 这一组。public_base_url 是别人打开 AgentLab 用的地址(https://agentlab.example.com),通知里的
|
||||
#: 运行链接 = 它 + /runs/{运行 id}。服务端不知道自己在反向代理后面叫什么,只能由人填;留空的通知里不附链接
|
||||
NOTIFY_GROUP = "notify"
|
||||
NOTIFY_DEFAULTS: dict[str, Any] = {"public_base_url": None}
|
||||
NOTIFY_SETTING_LABEL = {"public_base_url": "对外访问地址"}
|
||||
#: 运行链接 = 它 + /runs/{运行 id}。服务端不知道自己在反向代理后面叫什么,只能由人填;留空的通知里不附链接。
|
||||
#: 全站只认一个对外地址(core/public_url.py):环境变量 AGENTLAB_PUBLIC_URL 设了的话这一项不生效、不能改(PUT 回 409)。
|
||||
#: 键名沿用 notify 一组,界面上在「设置 → 通用」
|
||||
NOTIFY_GROUP = public_url.SETTING_GROUP
|
||||
NOTIFY_DEFAULTS: dict[str, Any] = {public_url.SETTING_KEY: None}
|
||||
NOTIFY_SETTING_LABEL = {public_url.SETTING_KEY: "对外访问地址"}
|
||||
#: GET /api/settings 另给的只读一组:{value: 生效的对外访问地址, env_locked: 是不是环境变量指定的}。设置页据此显示
|
||||
#: 「由环境变量指定」;PUT 带上它也不落库(同 limits)
|
||||
PUBLIC_URL_GROUP = "public_url"
|
||||
|
||||
|
||||
def _base_url_problem(value: Any) -> str | None:
|
||||
if value is None or (isinstance(value, str) and not value.strip()):
|
||||
return None
|
||||
label = NOTIFY_SETTING_LABEL["public_base_url"]
|
||||
label = NOTIFY_SETTING_LABEL[public_url.SETTING_KEY]
|
||||
hint = (f"「{label}」需要是 http:// 或 https:// 开头的完整地址(例如 https://agentlab.example.com),"
|
||||
f"留空表示通知里不附运行链接,当前为「{value}」")
|
||||
f"留空表示通知里不附链接,当前为「{value}」")
|
||||
if not isinstance(value, str):
|
||||
return hint
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
try:
|
||||
parts = urlsplit(value.strip())
|
||||
except ValueError:
|
||||
return hint
|
||||
if parts.scheme not in ("http", "https") or not parts.hostname or " " in value.strip():
|
||||
return hint
|
||||
if parts.query or parts.fragment:
|
||||
return f"「{label}」不能带 ? 或 # 之后的部分,运行链接会接在地址后面,当前为「{value}」"
|
||||
return None
|
||||
if (problem := public_url.url_problem(value)) == "suffix":
|
||||
return f"「{label}」不能带 ? 或 # 之后的部分,链接会接在地址后面,当前为「{value}」"
|
||||
return hint if problem else None
|
||||
|
||||
|
||||
def notify_problem(value: Any) -> str | None:
|
||||
@@ -651,18 +649,23 @@ def notify_problem(value: Any) -> str | None:
|
||||
def notify_values(stored: Any) -> dict[str, Any]:
|
||||
"""库里存的(或刚写进来的)notify 一组 → 补齐默认、规范化:去掉首尾空白和末尾的 /,空串当没填;存坏了的按没填。"""
|
||||
stored = stored if isinstance(stored, dict) else {}
|
||||
url = stored.get("public_base_url")
|
||||
if not isinstance(url, str) or not url.strip() or _base_url_problem(url):
|
||||
return {"public_base_url": None}
|
||||
return {"public_base_url": url.strip().rstrip("/")}
|
||||
return {public_url.SETTING_KEY: public_url.normalize(stored.get(public_url.SETTING_KEY))}
|
||||
|
||||
|
||||
async def notify_settings(session: AsyncSession) -> dict[str, Any]:
|
||||
"""设置里的 notify 一组,没存过的补默认。定时任务推送 Webhook 时从这里读。"""
|
||||
"""设置里存的 notify 一组,没存过的补默认。要拼链接的地方不读它:一律经 core/public_url(环境变量优先)。"""
|
||||
row = await session.get(Setting, NOTIFY_GROUP)
|
||||
return notify_values(row.value if row else None)
|
||||
|
||||
|
||||
def public_url_values(stored_notify: Any) -> dict[str, Any]:
|
||||
"""GET /api/settings 里只读的 public_url 一组:生效的那个地址(环境变量优先,其次库里存的这一项),是不是环境变量
|
||||
指定的。按库里存的算,不读进程缓存:设置页看到的就是落库的样子。"""
|
||||
locked = public_url.env_locked()
|
||||
value = public_url.env_url() if locked else notify_values(stored_notify)[public_url.SETTING_KEY]
|
||||
return {"value": value, "env_locked": locked}
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 应用:按需视图的每日上限(D-j)、使用者能不能建个人应用(docs/design/normal-user-apps.md N7)
|
||||
# --------------------------------------------------------------------------
|
||||
@@ -754,7 +757,7 @@ DEFAULT_SETTINGS: dict[str, Any] = {
|
||||
JUDGE_GROUP: dict(JUDGE_DEFAULTS),
|
||||
# 核查(双盲复算、反方审稿、审批质询):核查模型、三项的开关和上限、每日金额上限。和裁判一组一样由用户定
|
||||
VERIFY_GROUP: verify_values(None),
|
||||
# 通知:定时任务的 Webhook 里拼运行链接用的对外访问地址
|
||||
# 对外访问地址(存在 notify 一组):通知、Webhook 里的链接,单点登录的回调,来源核对。环境变量优先(core/public_url.py)
|
||||
NOTIFY_GROUP: notify_values(None),
|
||||
# 起草模型:数据目录起草、按配方导入的 AI 起草用的模型,留空沿用助手的模型
|
||||
CATALOG_DRAFT_GROUP: catalog_draft_values(None),
|
||||
@@ -808,6 +811,7 @@ async def get_settings(session: AsyncSession = Depends(get_session)) -> dict[str
|
||||
# limits 是服务端配置(环境变量),只读:库里可能存着旧版设置页写进去的一份
|
||||
#(见过 max_agent_steps: 25),它从来不生效,却会盖住真实的上限,界面照着错的数许诺
|
||||
stored.pop("limits", None)
|
||||
stored.pop(PUBLIC_URL_GROUP, None)
|
||||
# 裁判、核查的每日计数是引擎记的账,不是用户设置:设置页整组回写时带着一份旧的,会把今天的花费清零
|
||||
stored.pop(JUDGE_SPEND, None)
|
||||
stored.pop(VERIFY_SPEND, None)
|
||||
@@ -818,6 +822,7 @@ async def get_settings(session: AsyncSession = Depends(get_session)) -> dict[str
|
||||
merged[JUDGE_GROUP] = judge_values(stored.get(JUDGE_GROUP))
|
||||
merged[VERIFY_GROUP] = verify_values(stored.get(VERIFY_GROUP))
|
||||
merged[NOTIFY_GROUP] = notify_values(stored.get(NOTIFY_GROUP))
|
||||
merged[PUBLIC_URL_GROUP] = public_url_values(stored.get(NOTIFY_GROUP))
|
||||
merged[CATALOG_DRAFT_GROUP] = catalog_draft_values(stored.get(CATALOG_DRAFT_GROUP))
|
||||
merged[APPS_GROUP] = apps_values(stored.get(APPS_GROUP))
|
||||
for key, value in stored.items():
|
||||
@@ -910,6 +915,11 @@ async def put_settings(
|
||||
raise HTTPException(422, problem)
|
||||
# 存规范化之后的:末尾带 / 的地址拼链接会多出一个斜杠
|
||||
payload.values[NOTIFY_GROUP] = notify_values(payload.values[NOTIFY_GROUP])
|
||||
if public_url.env_locked() and payload.values[NOTIFY_GROUP] != await notify_settings(session):
|
||||
# 环境变量指定了对外访问地址:改这一项不会生效,照实拒绝,不悄悄存一个用不上的值。原样写回(整组回写)不算改
|
||||
raise CodedHTTPException(
|
||||
409, f"对外访问地址由环境变量 {public_url.ENV_NAME} 指定({public_url.env_url()}),不能在设置里修改。"
|
||||
"要改请改部署配置后重启服务", PUBLIC_URL_ENV_LOCKED, field=f"{NOTIFY_GROUP}.{public_url.SETTING_KEY}")
|
||||
if APPS_GROUP in payload.values:
|
||||
if problem := apps_problem(payload.values[APPS_GROUP]):
|
||||
raise HTTPException(422, problem)
|
||||
@@ -917,9 +927,9 @@ async def put_settings(
|
||||
if found := await _provider_problem(session, payload.values):
|
||||
raise CodedHTTPException(422, found[1], PROVIDER_INVALID, field=found[0])
|
||||
for key, value in payload.values.items():
|
||||
if key in (TRUST_KEY, "limits", JUDGE_SPEND, VERIFY_SPEND, *_hidden_groups()):
|
||||
if key in (TRUST_KEY, "limits", PUBLIC_URL_GROUP, JUDGE_SPEND, VERIFY_SPEND, *_hidden_groups()):
|
||||
# 信任三档只走 PUT /api/tools/trust:设置页整组回写时带着一份旧的,
|
||||
# 会把刚在工具页改的档位冲掉。limits 是环境变量定的,写进库也不生效。
|
||||
# 会把刚在工具页改的档位冲掉。limits 是环境变量定的、public_url 是算出来的,写进库也不生效。
|
||||
# 裁判、核查的每日计数是引擎记的账,只有引擎自己写。登录设置只经 /api/auth/settings 改(_hidden_groups)
|
||||
continue
|
||||
row = await session.get(Setting, key)
|
||||
@@ -928,6 +938,9 @@ async def put_settings(
|
||||
else:
|
||||
session.add(Setting(key=key, value=value))
|
||||
await session.commit()
|
||||
if NOTIFY_GROUP in payload.values:
|
||||
# 中间件、单点登录的回调按进程缓存取对外访问地址:写完库就刷新,不用重启
|
||||
public_url.remember(payload.values[NOTIFY_GROUP][public_url.SETTING_KEY])
|
||||
return await get_settings(session)
|
||||
|
||||
|
||||
|
||||
@@ -21,7 +21,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.api.coded import CodedHTTPException
|
||||
from app.api.datasources import _out
|
||||
from app.api.deps import signer_of
|
||||
from app.api.deps import signer_of, signer_verified
|
||||
from app.data import source_versions, table_versions
|
||||
from app.data.recipe_imports import ImportRefused
|
||||
from app.db.base import get_session
|
||||
@@ -170,7 +170,8 @@ async def remove_period(source_id: str, import_id: str, request: Request, x_acto
|
||||
source = await _source(session, source_id)
|
||||
try:
|
||||
out = await source_versions.remove_period(session, source, import_id, expected_current=expected,
|
||||
reason=reason or "", signed_by=_signed(body, x_actor))
|
||||
reason=reason or "", signed_by=_signed(body, x_actor),
|
||||
signer_verified=signer_verified())
|
||||
except _REFUSALS as e:
|
||||
raise _refused(e) from e
|
||||
return {"source": await _source_out(session, source_id), **out}
|
||||
@@ -193,7 +194,7 @@ async def revoke_acceptance(source_id: str, import_id: str, request: Request,
|
||||
try:
|
||||
out = await source_versions.revoke_acceptance(
|
||||
session, source, import_id, expected_current=expected, expected_target=target, ack_mask_lost=ack,
|
||||
reason=reason or "", signed_by=_signed(body, x_actor))
|
||||
reason=reason or "", signed_by=_signed(body, x_actor), signer_verified=signer_verified())
|
||||
except _REFUSALS as e:
|
||||
raise _refused(e) from e
|
||||
return {"source": await _source_out(session, source_id), **out}
|
||||
|
||||
@@ -28,7 +28,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.api.coded import CodedHTTPException
|
||||
from app.api.datasources import _NAME_PATTERN, _out, read_upload
|
||||
from app.api.deps import current_principal, signer_of
|
||||
from app.api.deps import current_principal, signer_of, signer_verified
|
||||
from app.data import recipe_imports, table_versions
|
||||
from app.data.recipe_imports import EditRequest, ImportRefused
|
||||
from app.data.recipe_types import Acceptance, AiAvailability, PeriodInput, Recipe, Selection
|
||||
@@ -422,7 +422,8 @@ async def commit(staging_id: str, request: Request, x_actor: str | None = Header
|
||||
try:
|
||||
result = await recipe_imports.commit_staging(
|
||||
session, staging, trial_id=trial_id if isinstance(trial_id, str) else "",
|
||||
confirmations=confirmations, acceptances=_acceptances(body.get("acceptances"), who), signed_by=who)
|
||||
confirmations=confirmations, acceptances=_acceptances(body.get("acceptances"), who), signed_by=who,
|
||||
signer_verified=signer_verified())
|
||||
except _REFUSALS as e:
|
||||
raise _refused(e) from e
|
||||
source = await session.get(DataSource, result.source_id, populate_existing=True)
|
||||
|
||||
@@ -1,17 +1,20 @@
|
||||
"""待办(BE-2):列表、计数、详情、回复、改派、催办。设计文档 §3.7、§3.10。
|
||||
|
||||
身份经 api/deps.current_principal 取;关登录时它没有名字,用自报的署名(X-Actor)补上,回复人一栏才写得出人。
|
||||
回复被拒回 403 / 404 / 409 / 422,{detail, code};表单不通过时另带 errors(同应用入口的 422)。
|
||||
回复被拒回 403 / 404 / 409 / 422,{detail, code},一律带 code(外部系统按它认);表单不通过时另带 errors(同应用入口的
|
||||
422)。服务层没带码的拒绝按状态补上同一类的码(api/coded.py 的 TASK_*)。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Literal
|
||||
|
||||
from fastapi import APIRouter, Header, HTTPException, Query
|
||||
from fastapi import APIRouter, Header, Query
|
||||
from fastapi.responses import JSONResponse
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from app.api.coded import CodedHTTPException
|
||||
from app.api.coded import (
|
||||
ROLE_REQUIRED, TASK_CLOSED, TASK_FORBIDDEN, TASK_INVALID, TASK_NOT_FOUND, CodedHTTPException,
|
||||
)
|
||||
from app.api.deps import current_principal, decode_actor
|
||||
from app.process import tasks as task_service
|
||||
from app.process.tasks import FormInvalid, TaskRefused
|
||||
@@ -23,12 +26,14 @@ def principal_of(x_actor: str | None) -> Any:
|
||||
return task_service.principal_with_actor(current_principal(), decode_actor(x_actor))
|
||||
|
||||
|
||||
def refused(e: TaskRefused) -> HTTPException | JSONResponse:
|
||||
#: 服务层(process/tasks.py、process/actions.py、blueprints/personal.py)没带码的拒绝按状态归到哪一类
|
||||
_CODE_BY_STATUS = {404: TASK_NOT_FOUND, 403: TASK_FORBIDDEN, 409: TASK_CLOSED, 422: TASK_INVALID}
|
||||
|
||||
|
||||
def refused(e: TaskRefused) -> CodedHTTPException | JSONResponse:
|
||||
if isinstance(e, FormInvalid):
|
||||
return JSONResponse({"detail": str(e), "code": e.code, "errors": e.errors}, status_code=422)
|
||||
if e.code:
|
||||
return CodedHTTPException(e.status_code, str(e), e.code)
|
||||
return HTTPException(e.status_code, str(e))
|
||||
return CodedHTTPException(e.status_code, str(e), e.code or _CODE_BY_STATUS.get(e.status_code, TASK_CLOSED))
|
||||
|
||||
|
||||
def _raise(e: TaskRefused) -> Any:
|
||||
@@ -53,7 +58,7 @@ async def list_tasks(
|
||||
) -> dict[str, Any]:
|
||||
principal = principal_of(x_actor)
|
||||
if assigned == "all" and principal.mode != "off" and not principal.at_least("builder"):
|
||||
raise HTTPException(403, "只能查看派给自己的待办")
|
||||
raise CodedHTTPException(403, "只能查看派给自己的待办", ROLE_REQUIRED)
|
||||
statuses = None if status == "all" else _csv(status) or ["pending"]
|
||||
items = await task_service.list_tasks(principal, assigned=assigned, statuses=statuses, app_id=app_id,
|
||||
run_id=run_id, limit=limit)
|
||||
@@ -71,7 +76,7 @@ async def get_task(task_id: str, x_actor: str | None = Header(default=None)) ->
|
||||
principal = principal_of(x_actor)
|
||||
out = await task_service.detail(task_id, principal)
|
||||
if out is None or not task_service.visible(out, principal):
|
||||
raise HTTPException(404, "待办不存在,所属的运行可能已被删除")
|
||||
raise CodedHTTPException(404, "待办不存在,所属的运行可能已被删除", TASK_NOT_FOUND)
|
||||
return out
|
||||
|
||||
|
||||
|
||||
@@ -24,13 +24,19 @@ VIEW = "latest"
|
||||
INSTANCE_VIEW = "inst"
|
||||
ENTRY = "start"
|
||||
RECONCILER = "reconciler"
|
||||
#: 派生不了时的机器码:工作流、版本不存在(404)
|
||||
WORKFLOW_GONE = "workflow_not_found"
|
||||
VERSION_GONE = "workflow_version_not_found"
|
||||
|
||||
|
||||
class DeriveRefused(Exception):
|
||||
def __init__(self, status_code: int, message: str) -> None:
|
||||
"""派生不了。code 是给程序认的机器码,接口照它回。"""
|
||||
|
||||
def __init__(self, status_code: int, message: str, code: str) -> None:
|
||||
super().__init__(message)
|
||||
self.status_code = status_code
|
||||
self.message = message
|
||||
self.code = code
|
||||
|
||||
|
||||
def _safe(text: str) -> str:
|
||||
@@ -45,10 +51,10 @@ async def derive_app(session: AsyncSession, workflow_id: str, version: int, *, n
|
||||
|
||||
workflow = await session.get(Workflow, workflow_id)
|
||||
if workflow is None:
|
||||
raise DeriveRefused(404, "工作流不存在,可能已被删除")
|
||||
raise DeriveRefused(404, "工作流不存在,可能已被删除", WORKFLOW_GONE)
|
||||
row = await version_row(session, workflow_id, version)
|
||||
if row is None:
|
||||
raise DeriveRefused(404, f"「{workflow.name}」不存在 v{version} 版本")
|
||||
raise DeriveRefused(404, f"「{workflow.name}」不存在 v{version} 版本", VERSION_GONE)
|
||||
state = PinState(alias=PIN, pin={"workflow_id": workflow_id, "version": version}, workflow=workflow, row=row)
|
||||
schedule = (await session.execute(select(Schedule).where(
|
||||
Schedule.workflow_id == workflow_id, Schedule.run_class == "formal", Schedule.enabled.is_(True))
|
||||
|
||||
@@ -113,6 +113,13 @@ async def start_entry(app_id: str, entry_id: str, values: dict[str, Any], princi
|
||||
form = resolve_form(spec, entry.get("form")) or {"type": "object", "properties": {}}
|
||||
if errors := validate_form(form, values if isinstance(values, dict) else None):
|
||||
raise FormRejected(errors)
|
||||
# 选人控件:开了登录时按候选池核对(auth/directory.py,05-已知遗留 (b))。以前原样传下去,填错的名字会把待办
|
||||
# 派给一个不存在的人——没人能处理,也不报错
|
||||
from app.auth.directory import principal_errors
|
||||
|
||||
if errors := await principal_errors(session, app, version.bindings, form, values,
|
||||
getattr(principal, "mode", "off")):
|
||||
raise FormRejected(errors)
|
||||
# 表单里的文件(docs/design/normal-user-apps.md N9):核对是提交人在这个入口传的、还没用过,先占住;表单值换成
|
||||
# 登记的那一份引用(连同 sha256 进运行输入、随封存固定)。发起成功绑到这次运行,发起不成放回去
|
||||
values, claimed = await _claim_files(form, values, principal, app.id, entry_id)
|
||||
|
||||
@@ -28,6 +28,8 @@ _FIELD_KEYS = frozenset({
|
||||
"minLength", "maxLength", "maxItems", "format", "items", "x-widget",
|
||||
# 文件控件的限制:accept(类型或扩展名的列表)、max_mb(docs/design/normal-user-apps.md §3.8)
|
||||
"x-file",
|
||||
# 选人控件的候选池:{pool: app | group:<组> | slot:<岗位>}(auth/directory.py,05-已知遗留 (b))
|
||||
"x-principal",
|
||||
})
|
||||
#: 文件控件只认这几项写法
|
||||
_FILE_FIELD_KEYS = frozenset({"type", "title", "description", "readOnly", "x-widget", "x-file"})
|
||||
@@ -88,6 +90,13 @@ def _field_problems(field: Any) -> list[str]:
|
||||
return out + _file_field_problems(field)
|
||||
if "x-file" in field:
|
||||
out.append("不是文件控件,不能写文件的限制")
|
||||
if "x-principal" in field:
|
||||
from app.auth.directory import pool_problem
|
||||
|
||||
if field.get("x-widget") != "principal":
|
||||
out.append("不是选人控件,不能写候选范围")
|
||||
elif problem := pool_problem(field["x-principal"]):
|
||||
out.append(problem)
|
||||
if kind not in FORM_TYPES:
|
||||
out.append(f"的类型只能是 {' / '.join(FORM_TYPES)}")
|
||||
if "format" in field and field["format"] not in FORM_FORMATS:
|
||||
@@ -153,7 +162,7 @@ def _clean_schema(form: dict[str, Any]) -> dict[str, Any]:
|
||||
continue
|
||||
props[name] = {k: v for k, v in field.items()
|
||||
if k in _FIELD_KEYS and k not in ("format", "x-widget", "x-enum-labels", "title",
|
||||
"description", "readOnly", "default")}
|
||||
"description", "readOnly", "default", "x-principal")}
|
||||
return {"type": "object", "properties": props, "required": [r for r in form.get("required") or [] if r in props]}
|
||||
|
||||
|
||||
|
||||
+38
-10
@@ -51,6 +51,8 @@ NEEDS_UPGRADE = "app_needs_upgrade"
|
||||
|
||||
STATUS_TEXT = {"queued": "排队中", "running": "运行中", "interrupted": "等待处理", "succeeded": "已完成",
|
||||
"failed": "失败", "cancelled": "已取消"}
|
||||
#: 被驳回、又说不出是哪一步时(没有节点可指)的说法
|
||||
REJECTED_TEXT = "被驳回"
|
||||
TIER_TEXT = {"formal": "完整出具", "degraded": "降档出具", "withheld": "不予出具"}
|
||||
NO_TIER = "未声明出具契约"
|
||||
#: 蓝图试运行(docs/design/normal-user-apps.md N4):印章的来源、说明,趋势只有一个点时的说明,预览里入口按钮为什么点不了
|
||||
@@ -1277,8 +1279,9 @@ async def _trails(session: AsyncSession, runs: list[Run]) -> dict[str, _Trail]:
|
||||
for row in (await session.execute(select(ActionRecord).where(ActionRecord.run_id.in_(ids))
|
||||
.order_by(ActionRecord.created_at, ActionRecord.attempt))).scalars():
|
||||
out[row.run_id].actions.append(row)
|
||||
# 节点事件只有结束了的运行要看(走到了哪个成果节点、停在哪个节点上)
|
||||
wanted = [r.id for r in runs if r.status in ("succeeded", "failed")]
|
||||
# 节点事件只有结束了的运行要看(走到了哪个成果节点、停在哪个节点上)。取消的只有被驳回的要看(停在哪一步)
|
||||
wanted = [r.id for r in runs if r.status in ("succeeded", "failed")
|
||||
or (r.status == "cancelled" and r.end_reason == "rejected")]
|
||||
if wanted:
|
||||
for run_id, etype, node_id in (await session.execute(
|
||||
select(RunEvent.run_id, RunEvent.type, RunEvent.node_id).where(
|
||||
@@ -1325,6 +1328,26 @@ def rejected_outputs(graph: Any) -> dict[str, set[str]]:
|
||||
return out
|
||||
|
||||
|
||||
def _rejection_text(trail: _Trail, node_id: str, title: str) -> str | None:
|
||||
"""停在被人驳回的待办上时怎么说:动作确认写「「付款」未确认(被驳回)」,别的写「「x」被驳回」;不是驳回的为 None。"""
|
||||
kind, decision = _decision(trail, node_id)
|
||||
if kind == "action_confirm" and decision == "reject":
|
||||
return f"「{title}」未确认(被驳回)"
|
||||
if decision in _REJECTIONS:
|
||||
return f"「{title}」被驳回"
|
||||
return None
|
||||
|
||||
|
||||
async def legacy_rejected(session: AsyncSession, run: Run) -> bool:
|
||||
"""加 end_reason 之前被人驳回的运行记的是 failed(封存不改写):失败的那一步上最近一条处理完的待办是驳回。和实例页
|
||||
认存量驳回(instance_state 的 failed 分支)是同一份判断;runner 据此不许它接着跑。"""
|
||||
if run.status != "failed":
|
||||
return False
|
||||
trail = (await _trails(session, [run]))[run.id]
|
||||
failed = [nid for etype, nid in trail.events if etype == "node.failed"]
|
||||
return bool(failed) and _decision(trail, failed[-1])[1] in _REJECTIONS
|
||||
|
||||
|
||||
def instance_state(run: Run, trail: _Trail) -> dict[str, str]:
|
||||
"""实例给人看的状态 {code, text}:code 是界面画状态标记用的状态(运行的那几种,加 held),text 说这件事办到哪了、
|
||||
结论是什么,不只是运行状态(P1)。
|
||||
@@ -1334,10 +1357,12 @@ def instance_state(run: Run, trail: _Trail) -> dict[str, str]:
|
||||
否定的那种结局——这次运行里人工节点驳回了、从它的「驳回」出口一路走到的成果节点(rejected_outputs)——code 记
|
||||
cancelled(同「未确认(被驳回)」,是人的决定,不画成成功),文字照旧写成果节点的标题;
|
||||
- 停着:有已发出、结果未知的动作,写「等待核实写入结果」;否则按在等的待办说(等待审批、等待确认发送……);
|
||||
- 失败:停在一个被人驳回的待办上的——动作确认被驳回(「付款」未确认(被驳回))、审批设了驳回即终止、补充输入被
|
||||
驳回——是人的决定,不是故障,code 记 cancelled;核实为未生效、选了放弃的同理。外部系统拒绝之类照写失败。
|
||||
- 被驳回(cancelled + end_reason=rejected,engine/context.NodeRejected):动作确认被驳回写「「付款」未确认(被驳回)」,
|
||||
审批设了驳回即终止、补充输入被驳回等写「「x」被驳回」,code 记 cancelled;手动停止的写「已取消」;
|
||||
- 失败:加 end_reason 之前被人驳回的运行记的是 failed(封存不改写),停在被驳回的待办上的照上一条说,和新形状说法
|
||||
一样;核实为未生效、选了放弃的同理,code 记 cancelled。外部系统拒绝之类照写失败。
|
||||
|
||||
运行本身的状态不改(驳回动作确认在引擎里就是节点失败、运行结束):这里只是把它说清楚。"""
|
||||
通知(notify/service.notify_run)的文字也取自这里:实例页和推送说同一句话。"""
|
||||
from app.apps.tree import node_map
|
||||
|
||||
nodes = node_map(run.graph)
|
||||
@@ -1360,15 +1385,18 @@ def instance_state(run: Run, trail: _Trail) -> dict[str, str]:
|
||||
negative = any(_decision(trail, gate)[1] in _REJECTIONS for gate in gates)
|
||||
return {"code": "cancelled" if negative else "succeeded", "text": title(reached[0])}
|
||||
return {"code": "succeeded", "text": STATUS_TEXT["succeeded"]}
|
||||
if run.status == "cancelled" and run.end_reason == "rejected":
|
||||
failed = [nid for etype, nid in trail.events if etype == "node.failed"]
|
||||
node_id = run.error_node_id or (failed[-1] if failed else None)
|
||||
said = _rejection_text(trail, node_id, title(node_id)) if node_id else None
|
||||
# 待办没留下(老的审批、没开出待办)时按驳回的那一步说,不退回「已取消」:这是人的决定,不是有人停了它
|
||||
return {"code": "cancelled", "text": said or (f"「{title(node_id)}」被驳回" if node_id else REJECTED_TEXT)}
|
||||
if run.status == "failed":
|
||||
failed = [nid for etype, nid in trail.events if etype == "node.failed"]
|
||||
if failed:
|
||||
node_id = failed[-1]
|
||||
kind, decision = _decision(trail, node_id)
|
||||
if kind == "action_confirm" and decision == "reject":
|
||||
return {"code": "cancelled", "text": f"「{title(node_id)}」未确认(被驳回)"}
|
||||
if decision in _REJECTIONS:
|
||||
return {"code": "cancelled", "text": f"「{title(node_id)}」被驳回"}
|
||||
if said := _rejection_text(trail, node_id, title(node_id)):
|
||||
return {"code": "cancelled", "text": said}
|
||||
record = next((a for a in reversed(trail.actions) if a.node_id == node_id), None)
|
||||
if record is not None and record.state == "failed":
|
||||
if (record.resolution or {}).get("outcome") == "abandon":
|
||||
|
||||
@@ -47,9 +47,10 @@ FILE_FIELD_TYPE = "object"
|
||||
FILE_REF_KEYS = ("file_id", "name", "size", "sha256", "type")
|
||||
#: 表单字段能用的 format。不支持 pattern:JS 和 Python 的正则语义不同,会出现前端放行、后端拒收
|
||||
FORM_FORMATS = ("date", "date-time", "email", "uri")
|
||||
#: 表单校验错误的 code。file:文件控件的值不对(没上传、不是本人传的、已经用过、类型或大小不符,app/files/store.py)
|
||||
#: 表单校验错误的 code。file:文件控件的值不对(没上传、不是本人传的、已经用过、类型或大小不符,app/files/store.py);
|
||||
#: principal:开了登录时选人控件的值不在候选池里(auth/directory.principal_errors)
|
||||
FORM_ERROR_CODES = ("required", "type", "enum", "minimum", "maximum", "min_length", "max_length", "max_items", "format",
|
||||
"file")
|
||||
"file", "principal")
|
||||
|
||||
|
||||
class _Strict(BaseModel):
|
||||
|
||||
@@ -15,7 +15,9 @@ import logging
|
||||
import os
|
||||
from typing import Any
|
||||
|
||||
from fastapi import Depends, HTTPException
|
||||
from fastapi import Depends
|
||||
|
||||
from app.api.coded import AUTH_REQUIRED, ROLE_REQUIRED, CodedHTTPException
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -77,17 +79,24 @@ def remember_settings(value: dict[str, Any]) -> None:
|
||||
_loaded = True
|
||||
|
||||
|
||||
def refusal(principal: Any, role: str) -> CodedHTTPException | None:
|
||||
"""principal 够不够 role:够(或关登录)是 None;不够交回该回的错——没登录 401(auth_required),角色不够 403
|
||||
(role_required)。路由级依赖、接口内部的 ensure_role、中间件拦 /docs 都用它,同一个判断、同一句话。"""
|
||||
if principal.mode == "off":
|
||||
return None
|
||||
if not principal.username:
|
||||
return CodedHTTPException(401, "请先登录", AUTH_REQUIRED)
|
||||
if not principal.at_least(role):
|
||||
return CodedHTTPException(403, f"没有权限:这项操作需要{ROLE_LABEL.get(role, role)}及以上的角色", ROLE_REQUIRED)
|
||||
return None
|
||||
|
||||
|
||||
def require(role: str) -> None:
|
||||
"""当前请求至少是 role,否则 401(没登录)/ 403(角色不够)。关登录时放行。"""
|
||||
"""当前请求至少是 role,否则 401(没登录)/ 403(角色不够),都带 code。关登录时放行。"""
|
||||
from app.api.deps import current_principal
|
||||
|
||||
principal = current_principal()
|
||||
if principal.mode == "off":
|
||||
return
|
||||
if not principal.username:
|
||||
raise HTTPException(401, "请先登录")
|
||||
if not principal.at_least(role):
|
||||
raise HTTPException(403, f"没有权限:这项操作需要{ROLE_LABEL.get(role, role)}及以上的角色")
|
||||
if (refused := refusal(current_principal(), role)) is not None:
|
||||
raise refused
|
||||
|
||||
|
||||
def role_dependency(role: str) -> Any:
|
||||
|
||||
@@ -1,6 +1,10 @@
|
||||
"""人和组的名录:选人控件的候选(GET /api/directory)、应用检查核对岗位绑定和授权、改授权时核对填的名字,同一处取、
|
||||
同一个口径(停用的人也在名录里,标 disabled:已经绑着他的岗位、授权要能认出「这个人停用了」,不是「没有这个人」)。
|
||||
|
||||
使用者选人(表单里的选人控件、个人应用提交审核)不拿整份名录,拿按用途算的候选池(candidates,GET
|
||||
/api/directory/candidates):只有这件事能选的人,只给用户名和显示名,停用的不列;开了登录时提交的值按同一个池核对
|
||||
(docs/design/next-round/05-已知遗留.md (b))。名录整份仍只给搭建者。
|
||||
|
||||
名录里对不上怎么算看登录方式(同前端 PrincipalPicker):
|
||||
- 账号密码登录:人和组都在本系统里建,名录里没有就是填错了(或人被删了),停用的人处理不了待办——都算错;
|
||||
- 单点登录:人和组第一次登录后才进名录,名录里没有的照实提醒、不拒收;停用的人名录里认得出,照样算错;
|
||||
@@ -10,7 +14,7 @@ from __future__ import annotations
|
||||
|
||||
from collections.abc import Iterable
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Literal
|
||||
from typing import Any, Literal
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
@@ -95,3 +99,175 @@ def mismatch(directory: Directory, mode: str, kind: str, name: str) -> Mismatch
|
||||
return Mismatch(kind, name, problem, False,
|
||||
f"{what}名录里还没有:单点登录的人和组第一次登录后才进名录,名字要和身份提供方里的一致才生效")
|
||||
return Mismatch(kind, name, problem, True, f"{what}名录里没有")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 候选池:使用者选人时能选谁(05-已知遗留 (b))
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
#: 候选池的用途:form 表单里的选人控件(入口表单、待办表单);review 个人应用提交审核选审核的人
|
||||
PURPOSES = ("form", "review")
|
||||
#: 一次最多给几位候选
|
||||
CANDIDATES_MAX = 50
|
||||
#: 表单字段声明候选池(x-principal.pool):app 能用这个应用的人(缺省);group:<组> 这个组的成员;slot:<岗位> 这个岗位
|
||||
#: 绑定的人和组(按应用版本的 bindings)
|
||||
POOL_APP = "app"
|
||||
POOL_GROUP = "group:"
|
||||
POOL_SLOT = "slot:"
|
||||
|
||||
|
||||
def pool_problem(value: Any) -> str | None:
|
||||
"""x-principal 的写法对不对:只能是 {pool: app | group:<组> | slot:<岗位>}。没问题为 None。"""
|
||||
if not isinstance(value, dict) or set(value) != {"pool"} or not isinstance(value.get("pool"), str):
|
||||
return "的候选范围只能写成 {\"pool\": \"app\"}、{\"pool\": \"group:<组名>\"} 或 {\"pool\": \"slot:<岗位>\"}"
|
||||
pool = value["pool"].strip()
|
||||
if pool == POOL_APP:
|
||||
return None
|
||||
for prefix in (POOL_GROUP, POOL_SLOT):
|
||||
if pool.startswith(prefix) and pool[len(prefix):].strip():
|
||||
return None
|
||||
return "的候选范围只能是 app(能用这个应用的人)、group:<组名> 或 slot:<岗位>"
|
||||
|
||||
|
||||
def pool_of(field: Any) -> str:
|
||||
"""表单字段声明的候选池;没声明、写法不对的按缺省(能用这个应用的人)。"""
|
||||
setting = field.get("x-principal") if isinstance(field, dict) else None
|
||||
if isinstance(setting, dict) and pool_problem(setting) is None:
|
||||
return setting["pool"].strip()
|
||||
return POOL_APP
|
||||
|
||||
|
||||
@dataclass
|
||||
class Pool:
|
||||
"""一个候选池。open:池就是全部启用的人(应用授权给了所有人)——不带前缀不列,免得等于发通讯录。"""
|
||||
users: dict[str, DirectoryUser] = field(default_factory=dict)
|
||||
groups: dict[str, DirectoryGroup] = field(default_factory=dict)
|
||||
open: bool = False
|
||||
|
||||
def has(self, name: str) -> bool:
|
||||
"""提交的这个名字在不在池里(用户名或组名)。"""
|
||||
return name in self.users or name in self.groups
|
||||
|
||||
def pick(self, q: str | None, limit: int = CANDIDATES_MAX) -> dict[str, Any]:
|
||||
"""接口给的那一份:{users: [{username, display_name}], groups: [{name, label}], truncated};按前缀筛
|
||||
(用户名或显示名,不分大小写)。池是 open 的、又没带前缀:什么都不给,needs_query 为真。"""
|
||||
prefix = (q or "").strip().lower()
|
||||
if self.open and not prefix:
|
||||
return {"users": [], "groups": [], "truncated": False, "needs_query": True}
|
||||
limit = max(1, min(int(limit or CANDIDATES_MAX), CANDIDATES_MAX))
|
||||
|
||||
def hit(*names: str) -> bool:
|
||||
return not prefix or any(n.lower().startswith(prefix) for n in names if n)
|
||||
|
||||
users = [u for u in sorted(self.users.values(), key=lambda u: u.username) if hit(u.username, u.display_name)]
|
||||
groups = [g for g in sorted(self.groups.values(), key=lambda g: g.name) if hit(g.name, g.label)]
|
||||
return {"users": [{"username": u.username, "display_name": u.display_name} for u in users[:limit]],
|
||||
"groups": [{"name": g.name, "label": g.label} for g in groups[:limit]],
|
||||
"truncated": len(users) > limit or len(groups) > limit}
|
||||
|
||||
|
||||
async def _members(session: AsyncSession, group: str) -> dict[str, DirectoryUser]:
|
||||
from app.db.models import Group, GroupMember, User
|
||||
|
||||
rows = (await session.execute(
|
||||
select(User).join(GroupMember, GroupMember.user_id == User.id).join(Group, Group.id == GroupMember.group_id)
|
||||
.where(Group.name == group, User.disabled.is_(False)))).scalars()
|
||||
return {u.username: DirectoryUser(u.username, u.display_name or u.username, False) for u in rows}
|
||||
|
||||
|
||||
async def _people(session: AsyncSession, items: Iterable[tuple[str, str]]) -> Pool:
|
||||
"""一组(种类, 名字)展开成池:人按名录认(停用的、名录里没有的不列),组列出组本身、再展开成员。"""
|
||||
from app.db.models import Group, User
|
||||
|
||||
pool = Pool()
|
||||
items = list(items)
|
||||
names = sorted({n for k, n in items if k == "user" and n})
|
||||
if names:
|
||||
for u in (await session.execute(select(User).where(User.username.in_(names), User.disabled.is_(False))))\
|
||||
.scalars():
|
||||
pool.users[u.username] = DirectoryUser(u.username, u.display_name or u.username, False)
|
||||
groups = sorted({n for k, n in items if k == "group" and n})
|
||||
if groups:
|
||||
for g in (await session.execute(select(Group).where(Group.name.in_(groups)))).scalars():
|
||||
pool.groups[g.name] = DirectoryGroup(g.name, g.label or g.name)
|
||||
pool.users.update(await _members(session, g.name))
|
||||
return pool
|
||||
|
||||
|
||||
async def _everyone(session: AsyncSession) -> Pool:
|
||||
from app.db.models import Group, User
|
||||
|
||||
pool = Pool(open=True)
|
||||
for u in (await session.execute(select(User).where(User.disabled.is_(False)))).scalars():
|
||||
pool.users[u.username] = DirectoryUser(u.username, u.display_name or u.username, False)
|
||||
for g in (await session.execute(select(Group))).scalars():
|
||||
pool.groups[g.name] = DirectoryGroup(g.name, g.label or g.name)
|
||||
return pool
|
||||
|
||||
|
||||
async def app_pool(session: AsyncSession, app: Any) -> Pool:
|
||||
"""能用这个应用的人:使用及以上的授权(人、组展开成员),加上管理它的人(管理员、建它的人)。授权给了所有人的,池就是
|
||||
全部启用的人(open)。按 apps/access 的口径:搭建者没有授权时只能查看全部实例,不算「能用」的候选。"""
|
||||
from app.apps.access import RANK, grants_of
|
||||
from app.db.models import User
|
||||
|
||||
grants = [g for g in await grants_of(session, app.id) if RANK.get(g.role, 0) >= RANK["use"]]
|
||||
if any(g.principal_kind == "everyone" for g in grants):
|
||||
return await _everyone(session)
|
||||
admins = (await session.execute(select(User.username).where(User.role == "admin"))).scalars()
|
||||
items = [(g.principal_kind, g.principal) for g in grants] + [("user", a) for a in admins]
|
||||
if getattr(app, "created_by", None):
|
||||
items.append(("user", app.created_by))
|
||||
return await _people(session, items)
|
||||
|
||||
|
||||
async def field_pool(session: AsyncSession, app: Any, bindings: dict[str, Any] | None, field: Any) -> Pool:
|
||||
"""表单里一个选人控件的候选池(按字段的 x-principal,缺省能用这个应用的人)。bindings 是这次用到的应用版本的绑定。"""
|
||||
pool = pool_of(field)
|
||||
if pool.startswith(POOL_GROUP):
|
||||
name = pool[len(POOL_GROUP):].strip()
|
||||
return Pool(users=await _members(session, name))
|
||||
if pool.startswith(POOL_SLOT):
|
||||
bound = ((bindings or {}).get("slots") or {}).get(pool[len(POOL_SLOT):].strip())
|
||||
items = [(str(b.get("kind") or "user"), str(b.get("name") or "")) for b in bound or [] if isinstance(b, dict)]
|
||||
if any(k == "everyone" for k, _ in items):
|
||||
return await _everyone(session)
|
||||
return await _people(session, items)
|
||||
return await app_pool(session, app)
|
||||
|
||||
|
||||
async def review_pool(session: AsyncSession) -> Pool:
|
||||
"""个人应用提交审核能交给谁:启用的搭建者和管理员(同 blueprints/personal._review_targets 的规则)。"""
|
||||
from app.db.models import User
|
||||
|
||||
rows = (await session.execute(select(User).where(User.role.in_(("builder", "admin")),
|
||||
User.disabled.is_(False)))).scalars()
|
||||
return Pool(users={u.username: DirectoryUser(u.username, u.display_name or u.username, False) for u in rows})
|
||||
|
||||
|
||||
def principal_fields(form: Any) -> dict[str, dict[str, Any]]:
|
||||
"""表单里的选人控件:{字段名: 字段}。"""
|
||||
props = form.get("properties") if isinstance(form, dict) else None
|
||||
return {name: f for name, f in (props or {}).items() if isinstance(f, dict) and f.get("x-widget") == "principal"}
|
||||
|
||||
|
||||
#: 选人控件的值不在候选池里时表单错误的 code(apps/spec.FORM_ERROR_CODES 的 principal)
|
||||
PRINCIPAL_ERROR_CODE = "principal"
|
||||
|
||||
|
||||
async def principal_errors(session: AsyncSession, app: Any, bindings: dict[str, Any] | None, form: Any,
|
||||
values: Any, mode: str) -> list[dict[str, str]]:
|
||||
"""开了登录时按候选池核对表单里选人控件的值:不在池里的按表单错误交回(code=principal),话写成「「x」不在可选的人里」。
|
||||
关登录不核对:没有用户表,签批记的是各自填的署名。没填的不在这里报(必填由表单校验报)。"""
|
||||
if mode == "off" or not isinstance(values, dict):
|
||||
return []
|
||||
out: list[dict[str, str]] = []
|
||||
for name, f in principal_fields(form).items():
|
||||
value = values.get(name)
|
||||
if value in (None, "") or not isinstance(value, str):
|
||||
continue
|
||||
pool = await field_pool(session, app, bindings, f)
|
||||
if not pool.has(value.strip()):
|
||||
hint = ":单点登录的人要先登录一次才进名录" if mode == "oidc" else ",请从候选中选择"
|
||||
out.append({"path": name, "code": PRINCIPAL_ERROR_CODE, "message": f"「{value.strip()}」不在可选的人里{hint}"})
|
||||
return out
|
||||
|
||||
@@ -9,6 +9,8 @@
|
||||
- 会话 cookie → Principal(认证过的用户名、角色、组);没有会话是匿名(username=None),X-Actor 一律不看;
|
||||
- /api 下除了登录本身、健康检查、入站钩子,匿名一律 401。各路由的角色要求另由 main.py 的路由级依赖管,这里兜底:
|
||||
哪个路由忘了挂依赖,也不会对没登录的人敞开;
|
||||
- 接口说明(FastAPI 自带的 /docs、/redoc、/openapi.json)和 /api 一样拦:匿名 401,不到搭建者 403。它们是 FastAPI 自己
|
||||
挂的路由,挂不上路由级依赖,所以角色也在这里判。以前只拦 /api/ 开头的,开了登录接口全貌也对匿名的人敞开;
|
||||
- 非 GET 的请求必须带 X-AgentLab-Request 头,带了 Origin 的必须是本站或登记的前端地址(CSRF)。自定义请求头跨站发不出来
|
||||
(要过 CORS 预检,只放行配置里的前端地址),SameSite=Lax 的 cookie 也不随跨站的表单提交带过去;
|
||||
- WebSocket:没登录的握手前就关掉;登录了的交给路由,按运行判(api/deps.py 的 run_scope:user 角色只能连自己能读、
|
||||
@@ -19,8 +21,8 @@ from __future__ import annotations
|
||||
import json
|
||||
from http.cookies import SimpleCookie
|
||||
from typing import Any
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from app.api.coded import AUTH_REQUIRED
|
||||
from app.core.config import settings
|
||||
|
||||
#: 开登录时匿名也能访问的接口
|
||||
@@ -28,6 +30,8 @@ _PUBLIC = frozenset({"/api/health", "/api/auth/me", "/api/auth/login", "/api/aut
|
||||
"/api/auth/oidc/callback", "/api/auth/jwks", "/api/auth/bootstrap"})
|
||||
#: 只认签名、不走会话的入站钩子(也不查 CSRF:调用方是外部系统,没有浏览器的 Origin)
|
||||
_HOOKS = "/api/hooks/"
|
||||
#: FastAPI 自带的接口说明(main.py 里 FastAPI() 的缺省地址):开登录时要搭建者
|
||||
_DOCS = ("/docs", "/redoc", "/openapi.json")
|
||||
_SAFE_METHODS = frozenset({"GET", "HEAD", "OPTIONS"})
|
||||
CSRF_HEADER = "x-agentlab-request"
|
||||
|
||||
@@ -49,11 +53,17 @@ def _cookie(headers: dict[str, str], name: str) -> str | None:
|
||||
return morsel.value if morsel else None
|
||||
|
||||
|
||||
def _is_docs(path: str) -> bool:
|
||||
return any(path == p or path.startswith(p + "/") for p in _DOCS)
|
||||
|
||||
|
||||
def _origin_ok(origin: str, headers: dict[str, str], scope: dict[str, Any]) -> bool:
|
||||
from app.core.public_url import origin_of, public_base_url
|
||||
|
||||
allowed = {o.rstrip("/") for o in settings.cors_origins}
|
||||
if settings.public_url:
|
||||
parts = urlsplit(settings.public_url)
|
||||
allowed.add(f"{parts.scheme}://{parts.netloc}")
|
||||
# 对外访问地址(环境变量或设置里的那一项,core/public_url.py):反向代理后面请求的 Host 不一定是它
|
||||
if address := public_base_url():
|
||||
allowed.add(origin_of(address))
|
||||
host = headers.get("x-forwarded-host") or headers.get("host")
|
||||
if host:
|
||||
scheme = headers.get("x-forwarded-proto") or scope.get("scheme") or "http"
|
||||
@@ -92,7 +102,7 @@ class AuthMiddleware:
|
||||
await self.app(scope, receive, send)
|
||||
return
|
||||
from app.api.deps import reset_principal, set_principal
|
||||
from app.auth import current_mode
|
||||
from app.auth import current_mode, refusal
|
||||
|
||||
# 登录设置在启动时读进缓存(main.py 的 _platform_startup),改设置的接口改完就刷新;这里不查库
|
||||
mode = current_mode()
|
||||
@@ -130,7 +140,11 @@ class AuthMiddleware:
|
||||
return
|
||||
if (scope["type"] == "http" and is_api and not principal.username and path not in _PUBLIC
|
||||
and not path.startswith(_HOOKS)):
|
||||
await _reply(send, 401, "请先登录", "auth_required")
|
||||
await _reply(send, 401, "请先登录", AUTH_REQUIRED)
|
||||
return
|
||||
if scope["type"] == "http" and _is_docs(path) and (refused := refusal(principal, "builder")) is not None:
|
||||
# 和路由级依赖(auth.require)同一个判断、同一句话:没登录 401,角色不够 403
|
||||
await _reply(send, refused.status_code, refused.detail, refused.code)
|
||||
return
|
||||
|
||||
token = set_principal(principal)
|
||||
|
||||
@@ -198,7 +198,7 @@ def instantiate(blueprint: dict[str, Any], *, filled: dict[str, Any] | None = No
|
||||
spec = _spec_of(blueprint)
|
||||
ref = spec.get("template") or {}
|
||||
template = get_template(str(ref.get("id") or ""))
|
||||
problems = [i for i in blueprint_issues(spec, for_="trial") if i["level"] == "error"]
|
||||
problems = [i for i in blueprint_issues(spec, for_="trial", graph=graph) if i["level"] == "error"]
|
||||
if template is None:
|
||||
from app.apps.spec import empty_spec
|
||||
|
||||
|
||||
@@ -520,8 +520,47 @@ def _container(spec: dict[str, Any], tokens: list[str]) -> tuple[dict[str, Any],
|
||||
raise BpRefused(f"不能改这个位置:/{'/'.join(tokens)}")
|
||||
|
||||
|
||||
def _set(spec: dict[str, Any], tokens: list[str], value: Any, source: str) -> None:
|
||||
def is_freeform(spec: dict[str, Any]) -> bool:
|
||||
from app.blueprints.templates._common import is_freeform as imports_graph
|
||||
|
||||
return imports_graph(spec)
|
||||
|
||||
|
||||
#: 自由形的「谁来做」只定由谁做(05-已知遗留 (d)、遗留-Q5)
|
||||
FREEFORM_PEOPLE_ONLY_WHO = "自由形蓝图的「谁来做」只定每一步由谁做;会签、时限、超时、加减步骤在画布上改"
|
||||
FREEFORM_PEOPLE_USER = "「谁来做」要由你来定:派活给谁是人的决定,助手不替你选"
|
||||
|
||||
|
||||
def _freeform_who(spec: dict[str, Any], tokens: list[str], value: Any, source: str, graph: Any) -> None:
|
||||
"""自由形的 /people/<i>/who:i 是按导入的图列出的第几个岗位(freeform.people_rows 的岗位行,界面照那一份的 path 写),
|
||||
绑定存进 spec.people[](id 就是岗位名、只有 who)。只收人和组:岗位、发起人在图上已经写定了。"""
|
||||
from app.blueprints.templates import freeform
|
||||
from app.blueprints.templates._common import with_graph
|
||||
|
||||
if len(tokens) != 3 or tokens[2] != "who":
|
||||
raise BpRefused(FREEFORM_PEOPLE_ONLY_WHO, code="bp.template_mismatch")
|
||||
if source != "user":
|
||||
raise BpRefused(FREEFORM_PEOPLE_USER)
|
||||
if graph is None:
|
||||
raise BpRefused("这张蓝图还没有导入的图,没法按图定谁来做")
|
||||
rows = [r for r in freeform.people_rows(with_graph(spec, graph)) if r["kind"] == "slot"]
|
||||
slot = rows[_index(tokens[1], len(rows))]["slot"]
|
||||
checked = _checked("people", "who", value, spec)
|
||||
if checked and any(p["kind"] not in ("user", "group") for p in checked):
|
||||
raise BpRefused("这里只选人或组:岗位、发起人在图上已经定了")
|
||||
entries = spec.setdefault("people", [])
|
||||
entry = next((e for e in entries if e.get("id") == slot), None)
|
||||
if entry is None:
|
||||
entry = {"id": slot}
|
||||
entries.append(entry)
|
||||
entry["who"] = _item(checked, source)
|
||||
|
||||
|
||||
def _set(spec: dict[str, Any], tokens: list[str], value: Any, source: str, graph: Any = None) -> None:
|
||||
card = tokens[0]
|
||||
if card == "people" and is_freeform(spec):
|
||||
_freeform_who(spec, tokens, value, source, graph)
|
||||
return
|
||||
if card == "access":
|
||||
if len(tokens) != 2 or tokens[1] not in ACCESS_KINDS:
|
||||
raise BpRefused("「谁能看」只有使用、查看全部、管理三档")
|
||||
@@ -577,6 +616,8 @@ def _set(spec: dict[str, Any], tokens: list[str], value: Any, source: str) -> No
|
||||
|
||||
def _add(spec: dict[str, Any], tokens: list[str], value: Any, source: str) -> None:
|
||||
card = tokens[0]
|
||||
if card == "people" and is_freeform(spec):
|
||||
raise BpRefused(FREEFORM_PEOPLE_ONLY_WHO, code="bp.template_mismatch")
|
||||
if card == "access":
|
||||
if len(tokens) not in (2, 3) or tokens[1] not in ACCESS_KINDS:
|
||||
raise BpRefused("「谁能看」只有使用、查看全部、管理三档")
|
||||
@@ -598,8 +639,13 @@ def _add(spec: dict[str, Any], tokens: list[str], value: Any, source: str) -> No
|
||||
entries.insert(at, _new_entry(card, value, source, spec))
|
||||
|
||||
|
||||
def _remove(spec: dict[str, Any], tokens: list[str], source: str) -> None:
|
||||
def _remove(spec: dict[str, Any], tokens: list[str], source: str, graph: Any = None) -> None:
|
||||
card = tokens[0]
|
||||
if card == "people" and is_freeform(spec):
|
||||
if len(tokens) != 3:
|
||||
raise BpRefused(FREEFORM_PEOPLE_ONLY_WHO, code="bp.template_mismatch")
|
||||
_freeform_who(spec, tokens, None, source, graph)
|
||||
return
|
||||
if card == "access":
|
||||
if len(tokens) != 3 or tokens[1] not in ACCESS_KINDS:
|
||||
raise BpRefused("「谁能看」要写明删哪一档的第几个")
|
||||
@@ -655,8 +701,9 @@ def _template(spec: dict[str, Any], op: BpOp, source: str) -> None:
|
||||
prefill(template.id, spec)
|
||||
|
||||
|
||||
def apply_bp_op(spec: dict[str, Any], op: dict[str, Any], *, source: str) -> dict[str, Any]:
|
||||
"""把一个 bp_* 操作落到蓝图上,交回新的蓝图(不改入参)。source 是 user / assistant。不合规时抛 BpRefused。"""
|
||||
def apply_bp_op(spec: dict[str, Any], op: dict[str, Any], *, source: str, graph: Any = None) -> dict[str, Any]:
|
||||
"""把一个 bp_* 操作落到蓝图上,交回新的蓝图(不改入参)。source 是 user / assistant。不合规时抛 BpRefused。
|
||||
graph 是蓝图存的图:自由形的「谁来做」按它列岗位(没给时那张卡片改不了)。"""
|
||||
if source not in BP_SOURCES:
|
||||
raise BpRefused(f"操作的来源只能是用户或助手,当前为「{source}」")
|
||||
if not isinstance(op, dict) or op.get("op") not in BP_OPS:
|
||||
@@ -677,11 +724,11 @@ def apply_bp_op(spec: dict[str, Any], op: dict[str, Any], *, source: str) -> dic
|
||||
if source != "user" and (only := next((k for k in USER_ONLY if tuple(tokens[:len(k)]) == k), None)):
|
||||
raise BpRefused(_USER_ONLY_TEXT[only[-1]])
|
||||
if parsed.op == "bp_set":
|
||||
_set(out, tokens, parsed.value, source)
|
||||
_set(out, tokens, parsed.value, source, graph)
|
||||
elif parsed.op == "bp_add":
|
||||
_add(out, tokens, parsed.value, source)
|
||||
else:
|
||||
_remove(out, tokens, source)
|
||||
_remove(out, tokens, source, graph)
|
||||
try:
|
||||
return parse_blueprint(out).model_dump(mode="json", by_alias=True)
|
||||
except ValidationError as e:
|
||||
@@ -708,9 +755,9 @@ def _value(item: Any) -> Any:
|
||||
return item.get("value") if isinstance(item, dict) else None
|
||||
|
||||
|
||||
def blueprint_issues(spec: dict[str, Any], *, for_: str) -> list[dict[str, Any]]:
|
||||
def blueprint_issues(spec: dict[str, Any], *, for_: str, graph: Any = None) -> list[dict[str, Any]]:
|
||||
"""蓝图检查(§3.13 bp.*):for_ = trial(缺项挡试运行)/ publish(待确认项挡发布)/ personal。
|
||||
每条 {code, level, path, message}。"""
|
||||
每条 {code, level, path, message}。graph 是蓝图存的图:自由形的「谁来做」按它查(没给时不查这一张)。"""
|
||||
from app.blueprints.templates import get_template
|
||||
|
||||
if for_ not in ISSUE_PURPOSES:
|
||||
@@ -743,7 +790,10 @@ def blueprint_issues(spec: dict[str, Any], *, for_: str) -> list[dict[str, Any]]
|
||||
_data_issues(spec, "data" in required, add)
|
||||
if "cadence" in required and _value(spec.get("cadence")) is None:
|
||||
add("bp.missing", "/cadence", "「多久更新」还没定:每天、每周还是每月,几点,统计哪一期")
|
||||
if "people" in used:
|
||||
if "people" in used and template.imports_graph:
|
||||
if graph is not None and for_ != "personal":
|
||||
out += _freeform_people_issues(spec, graph, publish)
|
||||
elif "people" in used:
|
||||
_people_issues(spec, "people" in required, publish, add)
|
||||
if "fields" in used:
|
||||
_field_issues(spec, "fields" in required, add)
|
||||
@@ -836,6 +886,32 @@ def _people_issues(spec: dict[str, Any], required: bool, publish: bool, add: Cal
|
||||
f"「{label}」的条件拿「{threshold['field']}」比大小,它要是数字字段")
|
||||
|
||||
|
||||
def _freeform_people_issues(spec: dict[str, Any], graph: Any, publish: bool) -> list[dict[str, Any]]:
|
||||
"""自由形的「谁来做」(freeform.people_rows):岗位、核实人没绑的报 bp.people_unbound(试运行 warning:待办派给试运行的
|
||||
人自己;发布 error),路径就是那一行的 path,带上 slot(发布单据此去掉同一件事的 app.slot_unbound / app.reconciler_unbound)。
|
||||
图里已经没有的岗位、卡片上还留着绑定的,照实提醒(warning),不静默删掉。"""
|
||||
from app.blueprints.templates import freeform
|
||||
from app.blueprints.templates._common import with_graph
|
||||
|
||||
out: list[dict[str, Any]] = []
|
||||
level = "error" if publish else "warning"
|
||||
rows = freeform.people_rows(with_graph(spec, graph))
|
||||
for row in rows:
|
||||
if row["kind"] not in ("slot", "reconciler") or row["who"]:
|
||||
continue
|
||||
message = (f"{row['label']}还没定:{row['detail']},写入外部系统的结果不明时要有人去目标系统里核实"
|
||||
if row["kind"] == "reconciler" else f"{row['label']}还没定谁来处理({row['detail']})")
|
||||
out.append({"code": "bp.people_unbound", "level": level, "path": row["path"], "message": message,
|
||||
"slot": row["slot"]})
|
||||
known = {r["slot"] for r in rows if r["kind"] == "slot"}
|
||||
for step in spec.get("people") or []:
|
||||
slot = str(step.get("id") or "")
|
||||
if slot and slot not in known and _value(step.get("who")):
|
||||
out.append({"code": "bp.template_mismatch", "level": "warning", "path": "/people",
|
||||
"message": f"岗位「{slot}」在导入的图里已经没有了:卡片上留着的这条绑定不会生效"})
|
||||
return out
|
||||
|
||||
|
||||
def _field_issues(spec: dict[str, Any], required: bool, add: Callable[..., None]) -> None:
|
||||
fields = spec.get("fields") or []
|
||||
if required and not fields:
|
||||
|
||||
@@ -70,9 +70,13 @@ NOT_INSTANTIATED = "plan.not_instantiated"
|
||||
GRAPH_CHANGED = "plan.graph_changed"
|
||||
UNTRIALED = "plan.untrialed"
|
||||
TRIAL_FAILED = "plan.trial_failed"
|
||||
#: 最近一次试运行在某一步被人驳回、没有走完(cancelled + end_reason=rejected):人的决定,不是故障,不挡发布、照实说
|
||||
TRIAL_REJECTED = "plan.trial_rejected"
|
||||
TRIAL_DIFFERS = "plan.trial_differs"
|
||||
#: 蓝图检查:写回的结果不明时没人核实(blueprints/ops.blueprint_issues)
|
||||
RECONCILER_UNBOUND = "bp.reconciler_unbound"
|
||||
#: 蓝图检查:「谁来做」有一步还没定由谁处理;自由形的带着 slot(哪个岗位,核实人是 apps/derive.RECONCILER)
|
||||
PEOPLE_UNBOUND = "bp.people_unbound"
|
||||
ADMIN_REQUIRED = "plan.admin_required"
|
||||
DOWNGRADE_ADMIN = "plan.downgrade_admin"
|
||||
PREFLIGHT = "plan.preflight"
|
||||
@@ -237,8 +241,10 @@ async def _simulated_pins(session: AsyncSession, spec: dict[str, Any], target: _
|
||||
def bindings_of(bp: Blueprint) -> dict[str, Any]:
|
||||
"""岗位绑定:「谁来做」卡片上写了具体的人或组的,按模板定的岗位名绑上(templates/_common.bindings_of)。"""
|
||||
from app.blueprints.templates._common import bindings_of as from_cards
|
||||
from app.blueprints.templates._common import with_graph
|
||||
|
||||
return from_cards(bp.spec or {})
|
||||
# 带上蓝图存的图:自由形的核实人要不要绑按导入的图算(05-已知遗留 (d))
|
||||
return from_cards(with_graph(bp.spec or {}, bp.graph))
|
||||
|
||||
|
||||
def grant_items(bp: Blueprint) -> list[tuple[str, str, str]]:
|
||||
@@ -352,6 +358,14 @@ async def _preflight(session: AsyncSession, bp: Blueprint, target: _Target, *, l
|
||||
|
||||
issues = [i for i in issues if i.get("code") != "app.reconciler_unbound"
|
||||
and not (i.get("code") == "app.slot_unbound" and i.get("path") == f"slots.{RECONCILER}")]
|
||||
if unbound := {str(i["slot"]) for i in out if i.get("code") == PEOPLE_UNBOUND and i.get("slot")}:
|
||||
# 自由形的「谁来做」:岗位、核实人没绑,蓝图检查已经指到卡片上那一行(bp.people_unbound 带着 slot),应用检查说的是
|
||||
# 同一件事,不再重复列(同上面核实岗位的办法)
|
||||
from app.apps.derive import RECONCILER
|
||||
|
||||
issues = [i for i in issues if not (i.get("code") == "app.slot_unbound"
|
||||
and str(i.get("path") or "").removeprefix("slots.") in unbound)
|
||||
and not (i.get("code") == "app.reconciler_unbound" and RECONCILER in unbound)]
|
||||
out += [{**i, "source": "app"} for i in issues]
|
||||
# 授权:给的人和组要在名录里(同授权接口)
|
||||
_, problem = await grants_problem(session, grant_items(bp))
|
||||
@@ -363,6 +377,12 @@ async def _preflight(session: AsyncSession, bp: Blueprint, target: _Target, *, l
|
||||
if is_trial_of(trial, bp) and trial.status == "failed":
|
||||
why = (trial.error or "").strip().splitlines()[0][:200] if (trial.error or "").strip() else ""
|
||||
out.append(_problem(TRIAL_FAILED, f"最近一次试运行失败了{f':{why}' if why else ''}", source="plan"))
|
||||
elif is_trial_of(trial, bp) and trial.status == "cancelled" and trial.end_reason == "rejected":
|
||||
# 搭建者在试运行里驳回了写入、审批:那是他的决定,不是图出了错,不说成「试运行失败了」、也不挡发布;只照实说
|
||||
# 没走完——驳回之后的步骤没用真数看过
|
||||
where = _node_title(trial.graph, trial.error_node_id)
|
||||
out.append(_problem(TRIAL_REJECTED, f"最近一次试运行{f'在「{where}」' if where else ''}被驳回、没有走完",
|
||||
"warning", source="plan"))
|
||||
elif not is_trial_of(trial, bp) or trial.status != "succeeded":
|
||||
out.append(_problem(UNTRIALED, "还没有成功的试运行:发布之前没用真数看过结果", "warning", source="plan"))
|
||||
elif graph_hash(trial.graph or {}) != target.digest:
|
||||
@@ -370,6 +390,17 @@ async def _preflight(session: AsyncSession, bp: Blueprint, target: _Target, *, l
|
||||
return [_with_who(p) for p in out]
|
||||
|
||||
|
||||
def _node_title(graph: Any, node_id: str | None) -> str | None:
|
||||
"""节点在图上的名字(没有名字的给 id);没有节点时为 None。"""
|
||||
if not node_id:
|
||||
return None
|
||||
for node in (graph or {}).get("nodes") or []:
|
||||
if isinstance(node, dict) and node.get("id") == node_id:
|
||||
data = node.get("data") if isinstance(node.get("data"), dict) else {}
|
||||
return str(data.get("label") or node_id)
|
||||
return node_id
|
||||
|
||||
|
||||
def _with_who(issue: dict[str, Any]) -> dict[str, Any]:
|
||||
"""问题带上白话(没登记白话的照旧只有技术说法)和归谁处理(you / assistant / admin,摘要按它分三栏)。"""
|
||||
from app.plain import who_of, with_plain
|
||||
@@ -428,6 +459,7 @@ def _summary(bp: Blueprint, target: _Target, *, level: str, first_run: bool, opt
|
||||
if current and current > target.version else ""
|
||||
codes = {p.get("code") for p in problems}
|
||||
tried = ";发布的内容和最近一次试运行不同(试运行之后又改过),建议再试一次" if TRIAL_DIFFERS in codes else \
|
||||
";最近一次试运行被驳回、没有走完" if TRIAL_REJECTED in codes else \
|
||||
";还没有用真数试运行过" if UNTRIALED in codes else ""
|
||||
if target.published_as == level:
|
||||
say("publish_workflow", f"「{name}」v{target.version} 已经按{LEVEL_TEXT[level]}级别发布过,不重复发布{tried}")
|
||||
|
||||
@@ -183,7 +183,7 @@ async def apply_ops_and_save(session: AsyncSession, blueprint: Any, ops: list[di
|
||||
try:
|
||||
op = await _verified_reuse(session, raw)
|
||||
before = _at(spec, op.get("path"))
|
||||
spec = apply_bp_op(spec, op, source=source)
|
||||
spec = apply_bp_op(spec, op, source=source, graph=blueprint.graph)
|
||||
except BpRefused as e:
|
||||
if not partial:
|
||||
raise
|
||||
@@ -313,7 +313,7 @@ def blueprint_issues_of(blueprint: Any) -> list[dict[str, Any]]:
|
||||
from app.plain import with_plain
|
||||
|
||||
spec = blueprint.spec or {}
|
||||
issues = blueprint_issues(spec, for_="personal" if blueprint.scope == "personal" else "publish")
|
||||
issues = blueprint_issues(spec, for_="personal" if blueprint.scope == "personal" else "publish", graph=blueprint.graph)
|
||||
for joint_id, state in (blueprint.joints or {}).items():
|
||||
if isinstance(state, dict) and state.get("status") in ("pending", "refused"):
|
||||
issues.append({"code": "tpl.joint_unfilled" if state["status"] == "pending" else "tpl.joint_refused",
|
||||
@@ -375,11 +375,19 @@ async def blueprint_view(session: AsyncSession, blueprint: Any) -> dict[str, Any
|
||||
return blueprint_out(blueprint, extra=await data_source_issues(session, blueprint))
|
||||
|
||||
|
||||
def _derived_people(blueprint: Any) -> list[dict[str, Any]]:
|
||||
from app.blueprints.templates import freeform
|
||||
from app.blueprints.templates._common import with_graph
|
||||
|
||||
return freeform.people_rows(with_graph(blueprint.spec or {}, getattr(blueprint, "graph", None)))
|
||||
|
||||
|
||||
def blueprint_out(blueprint: Any, *, issues: bool = True, extra: list[dict[str, Any]] | None = None) -> dict[str, Any]:
|
||||
"""一张蓝图交给前端的样子(frontend/src/typesBlueprint.ts 的 Blueprint)。关节只给状态和模板声明的说法、类型、分组
|
||||
(JointState:status、by、note、label、kind、group),不给填的值(值在图上看)。reconciler_needed:写回卡片上要不要
|
||||
摆「结果不明时谁去核实」(目标能力没登记查询,templates/_common.needs_reconciler),界面不自己判。"""
|
||||
from app.blueprints.templates._common import needs_reconciler
|
||||
摆「结果不明时谁去核实」(目标能力没登记查询,templates/_common.needs_reconciler),界面不自己判。derived_people:自由形
|
||||
按导入的图列出的「谁来做」(templates/freeform.people_rows,05-已知遗留 (d)),别的模板为空。"""
|
||||
from app.blueprints.templates._common import is_freeform, needs_reconciler, with_graph
|
||||
|
||||
joints = {jid: {k: v for k, v in state.items() if k in ("status", "by", "note", "label", "kind", "group")}
|
||||
for jid, state in (blueprint.joints or {}).items() if isinstance(state, dict)}
|
||||
@@ -389,7 +397,9 @@ def blueprint_out(blueprint: Any, *, issues: bool = True, extra: list[dict[str,
|
||||
"template_version": blueprint.template_version, "source": blueprint.source or {},
|
||||
"conversation_id": blueprint.conversation_id, "workflow_id": blueprint.workflow_id, "app_id": blueprint.app_id,
|
||||
"last_trial_run_id": blueprint.last_trial_run_id, "graph_hash": blueprint.graph_hash, "joints": joints,
|
||||
"copied_from": blueprint.copied_from, "reconciler_needed": needs_reconciler(blueprint.spec or {}),
|
||||
"copied_from": blueprint.copied_from,
|
||||
"reconciler_needed": needs_reconciler(with_graph(blueprint.spec or {}, getattr(blueprint, "graph", None))),
|
||||
"derived_people": _derived_people(blueprint) if is_freeform(blueprint.spec or {}) else None,
|
||||
"created_at": blueprint.created_at.isoformat() if blueprint.created_at else None,
|
||||
"updated_at": blueprint.updated_at.isoformat() if blueprint.updated_at else None,
|
||||
}
|
||||
|
||||
@@ -78,6 +78,19 @@ def context(bp: dict[str, Any]) -> dict[str, Any]:
|
||||
return ctx if isinstance(ctx, dict) else {}
|
||||
|
||||
|
||||
def with_graph(spec: dict[str, Any], graph: Any) -> dict[str, Any]:
|
||||
"""只有卡片(Blueprint.spec)、没有实例化上下文的地方,带上蓝图存的图再交给模板的函数:自由形的「谁来做」、要不要
|
||||
核实人都按导入的图算(freeform.people_rows / needs_reconciler)。"""
|
||||
return {**spec, CONTEXT_KEY: {**context(spec), "graph": graph}}
|
||||
|
||||
|
||||
def is_freeform(spec: dict[str, Any]) -> bool:
|
||||
"""卡片上选的是导入图的模板(自由形)。"""
|
||||
ref = spec.get("template")
|
||||
template = get_template(str(ref.get("id") or "")) if isinstance(ref, dict) else None
|
||||
return template is not None and template.imports_graph
|
||||
|
||||
|
||||
def app_name(bp: dict[str, Any], fallback: str) -> str:
|
||||
"""应用和工作流的名字:实例化时给的名字,没给就用目的的前一段。"""
|
||||
name = str(context(bp).get("name") or "").strip()
|
||||
@@ -434,6 +447,12 @@ def needs_reconciler(bp: dict[str, Any]) -> bool:
|
||||
蓝图(reconciler_needed)共用这一份。"""
|
||||
ref = bp.get("template")
|
||||
template = get_template(str(ref.get("id") or "")) if isinstance(ref, dict) else None
|
||||
if template is not None and template.imports_graph:
|
||||
# 自由形没有「写回哪里」:按导入的图里的写入算(bp 要带着图,with_graph),核实人存在 /writeback/reconciler
|
||||
# (05-已知遗留 (d))
|
||||
from app.blueprints.templates import freeform
|
||||
|
||||
return freeform.needs_reconciler(bp)
|
||||
if template is None or "writeback" not in template.cards:
|
||||
return False
|
||||
capability = value((bp.get("writeback") or {}).get("capability"))
|
||||
|
||||
@@ -1,5 +1,11 @@
|
||||
"""自由形 freeform(§3.3):从已有的运行(问数据的一轮回答,经 from-run 提取)或已发布的工作流版本导入,图原样用,
|
||||
没有关节。蓝图只有目的、谁能看、多久更新三张卡片;要改图只能回画布(搭建者)。
|
||||
没有关节。蓝图有目的、谁能看、多久更新、谁来做四张卡片;要改图只能回画布(搭建者)。
|
||||
|
||||
「谁来做」(docs/design/next-round/05-已知遗留.md (d)):骨架就是导入的图,卡片只填「谁」这个关节——按图列出用到的岗位
|
||||
(人工节点派给谁、超时升级给谁、写入的发送确认派给谁,people_rows),写入能力没登记查询时再加一行「写入结果不明时谁去核实」。
|
||||
只定由谁做:会签、时限、超时在画布上改(改了会改图,图哈希就和发布的那一版对不上)。绑定存在 spec.people[](Step.id 就是
|
||||
岗位名、只写 who,_common.slot_of / bindings_of 原样生效),核实人存在 /writeback/reconciler。以前这张卡片不存在,第一次
|
||||
发布挡在 app.slot_unbound / app.reconciler_unbound 上,白话却叫人去「谁来做」里选人。
|
||||
|
||||
图由实例化的上下文给(graph,导入时存在蓝图上)。应用骨架按 apps/derive.py 的派生规则:有人工节点或写入的按流程,
|
||||
否则按看板;写入节点按能力登记的进程内快照认(和运行时同一个判断,capabilities.registry.writes_externally)。
|
||||
@@ -38,6 +44,65 @@ def _writes(graph: dict[str, Any]) -> dict[str, str]:
|
||||
return out
|
||||
|
||||
|
||||
#: 「谁来做」一行里每处用途怎么说(apps/tree.SLOT_ROLES,加上核实)
|
||||
USE_TEXT = {"assignee": "处理", "escalation": "超时升级", "confirm": "发送前确认", "reconcile": "写入结果不明时去核实"}
|
||||
#: 不按岗位派的步骤(只读行):发起人自己、按表达式算出来的人、等外部系统回调
|
||||
FIXED_TEXT = {"initiator": "由发起人自己处理", "expression": "按图里的表达式算出由谁处理(在画布上改)",
|
||||
"hook": "等外部系统回调,不派给人"}
|
||||
#: 核实人那一行的名字
|
||||
RECONCILER_LABEL = "写入结果不明时谁去核实"
|
||||
|
||||
|
||||
def people_rows(bp: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
"""按导入的图列出「谁来做」:[{kind, slot, label, detail, path, who, uses}]。
|
||||
|
||||
kind=slot:图里引用的一个岗位(同一个岗位用在几处合成一行,按节点出现的顺序),path 是 /people/<i>/who(i 是岗位行的
|
||||
序号),who 是卡片上绑的人或组;kind=reconciler:写入能力没登记查询时的核实人,path 是 /writeback/reconciler;
|
||||
kind=initiator / expression / hook:不按岗位派的步骤,只读(path 为空)。label 用节点在图上的名字拼(岗位名是画布上
|
||||
随手写的 id),detail 写全每一处用途。界面照这一份画,不自己从图里推。"""
|
||||
from app.apps.derive import RECONCILER
|
||||
from app.apps.tree import node_map, slot_refs
|
||||
|
||||
graph = imported(bp)
|
||||
writes = _writes(graph)
|
||||
nodes = node_map(graph)
|
||||
bound = {str(step.get("id")): c.value(step.get("who")) for step in bp.get("people") or [] if isinstance(step, dict)}
|
||||
rows: list[dict[str, Any]] = []
|
||||
by_slot: dict[str, dict[str, Any]] = {}
|
||||
for ref in slot_refs(graph, writes):
|
||||
row = by_slot.get(ref["slot"])
|
||||
if row is None:
|
||||
row = by_slot[ref["slot"]] = {"kind": "slot", "slot": ref["slot"], "path": f"/people/{len(by_slot)}/who",
|
||||
"who": bound.get(ref["slot"]), "uses": []}
|
||||
rows.append(row)
|
||||
row["uses"].append({"node_id": ref["node_id"], "title": ref["title"], "role": ref["role"]})
|
||||
unchecked = [(nid, cap) for nid, cap in writes.items() if not c.write_lookup(cap)]
|
||||
if unchecked:
|
||||
rows.append({"kind": "reconciler", "slot": RECONCILER, "path": "/writeback/reconciler",
|
||||
"who": c.value((bp.get("writeback") or {}).get("reconciler")),
|
||||
"uses": [{"node_id": nid, "title": nodes[nid]["title"], "role": "reconcile"} for nid, _ in unchecked]})
|
||||
for nid, n in nodes.items():
|
||||
if n["type"] != "human" and nid not in writes:
|
||||
continue
|
||||
assignee = n["config"].get("assignee")
|
||||
kind = assignee.get("kind") if isinstance(assignee, dict) else None
|
||||
if kind in FIXED_TEXT:
|
||||
rows.append({"kind": kind, "slot": None, "path": None, "who": None,
|
||||
"uses": [{"node_id": nid, "title": n["title"],
|
||||
"role": "assignee" if n["type"] == "human" else "confirm"}]})
|
||||
for row in rows:
|
||||
titles = list(dict.fromkeys(u["title"] for u in row["uses"]))
|
||||
row["label"] = RECONCILER_LABEL if row["kind"] == "reconciler" else "、".join(f"「{t}」" for t in titles)
|
||||
row["detail"] = FIXED_TEXT[row["kind"]] if row["kind"] in FIXED_TEXT else ";".join(
|
||||
f"「{u['title']}」{USE_TEXT[u['role']]}" for u in row["uses"])
|
||||
return rows
|
||||
|
||||
|
||||
def needs_reconciler(bp: dict[str, Any]) -> bool:
|
||||
"""导入的图里有写入能力没登记查询(同 app_skeleton 加核实岗位的判断)。"""
|
||||
return any(not c.write_lookup(cap) for cap in _writes(imported(bp)).values())
|
||||
|
||||
|
||||
def app_skeleton(bp: dict[str, Any], graph: dict[str, Any]) -> dict[str, Any]:
|
||||
from app.apps.derive import ENTRY, INSTANCE_VIEW, PIN, VIEW, _entries, _results
|
||||
from app.apps.forms import input_schema
|
||||
@@ -103,6 +168,7 @@ TEMPLATE = register(TemplateDef(
|
||||
applies_when="用户已经有一次满意的问数据回答,或者一个已发布的工作流,想直接做成应用;需求在五个场景之外时也用它。",
|
||||
cards={"purpose": "必填:一句话说清这个应用做什么",
|
||||
"cadence": "可选:要不要定期自动出数",
|
||||
"people": "可选:只定每一步由谁做;会签、时限、超时在画布上改",
|
||||
"access": "可选:谁能用、谁能看全部、谁能管理"},
|
||||
personal="导入的图满足问数据的规则(只读、没有人工节点和写入)时可以",
|
||||
skeleton=skeleton, app_skeleton=app_skeleton, schedule=schedule, tags=["导入"],
|
||||
|
||||
@@ -147,7 +147,7 @@ async def materialize(session: AsyncSession, bp: Blueprint, *, principal: Princi
|
||||
digest = graph_hash(graph or {})
|
||||
app_spec = _draft_pins(result.app_spec, bp, digest)
|
||||
# 实例化交回的是挡路的 error;蓝图检查里的提醒(模板版本旧了之类)另外带上
|
||||
warnings = [i for i in blueprint_issues(bp.spec or {}, for_="trial") if i.get("level") != "error"]
|
||||
warnings = [i for i in blueprint_issues(bp.spec or {}, for_="trial", graph=bp.graph) if i.get("level") != "error"]
|
||||
problems = [with_plain(p) for p in [*result.problems, *warnings]]
|
||||
bp.graph, bp.graph_hash, bp.joints = graph, digest, result.joints
|
||||
bp.updated_by = principal.username if principal else bp.updated_by
|
||||
|
||||
@@ -1237,8 +1237,8 @@ def build_bundle(
|
||||
) -> dict[str, Any]:
|
||||
"""一次已封存运行 → 页面里的数据(bundle-data)。
|
||||
|
||||
- run:运行记录的几样(id、workflow_id、workflow_name、version、version_hash、run_class、status、
|
||||
started_at、finished_at)
|
||||
- run:运行记录的几样(id、workflow_id、workflow_name、version、version_hash、run_class、status、end_reason、
|
||||
started_at、finished_at)。end_reason 分开同是 cancelled 的被驳回和手动停止(05-已知遗留 (a)),不在封存范围内
|
||||
- events:这次运行的全部事件 (seq, type, node_id, data),按数据库里 seq 的顺序。和封存核对
|
||||
(runner.verify_manifest)读的是同一个顺序:seq 相同的几条,浏览器按这里的先后排,清单才算得一样
|
||||
- manifest_seq / manifest_hash:封存范围(老运行是按结束事件推算的那个,legacy 为真)和封存哈希
|
||||
@@ -1306,7 +1306,7 @@ def build_bundle(
|
||||
"meta": {
|
||||
"run_id": run.get("id"), "workflow_id": run.get("workflow_id"), "workflow_name": display_name(run),
|
||||
"version": run.get("version"), "version_hash": run.get("version_hash"),
|
||||
"run_class": run.get("run_class"), "status": run.get("status"),
|
||||
"run_class": run.get("run_class"), "status": run.get("status"), "end_reason": run.get("end_reason"),
|
||||
"started_at": _iso(run.get("started_at")), "finished_at": _iso(run.get("finished_at")),
|
||||
"manifest_seq": manifest_seq, "manifest_hash": manifest_hash, "legacy": bool(legacy),
|
||||
"tier": _tier(sealed), "exported_at": exported_at,
|
||||
|
||||
@@ -49,7 +49,9 @@
|
||||
const KIND_LABEL = { report_doc: '报告文档', metric_set: '口径卡', query: '查询快照' }
|
||||
const TIER_LABEL = { formal: '完整出具', degraded: '降档出具', withheld: '不予出具' }
|
||||
const CLASS_LABEL = { formal: '正式运行', exploratory: '探索运行' }
|
||||
const STATUS_LABEL = { succeeded: '已完成', failed: '已失败', cancelled: '已取消' }
|
||||
// rejected:被审批人驳回结束(运行记录 cancelled + end_reason=rejected)。封存里它和手动停止是同一种结束事件(运行取消),
|
||||
// 分开二者的原因在那条事件的内容里,内容不随文件附带:见 checkMeta
|
||||
const STATUS_LABEL = { succeeded: '已完成', failed: '已失败', cancelled: '已取消', rejected: '被驳回' }
|
||||
/** 结束类事件 → 它说明的运行结果(runner:成功、失败、取消各发一条,落库之后才封存) */
|
||||
const TERMINAL = { 'run.finished': 'succeeded', 'run.failed': 'failed', 'run.cancelled': 'cancelled' }
|
||||
const EVENT_LABEL = {
|
||||
@@ -84,6 +86,7 @@
|
||||
range: (seq, count) => `第 1–${seq} 条事件(范围内 ${count} 条)`,
|
||||
rangeLegacy: '(早期运行,范围按结束事件推算)',
|
||||
markPass: '已核对',
|
||||
markPassRejected: '已核对运行取消,被驳回取自运行记录',
|
||||
markFail: (claim) => `文件里写的是「${claim}」,见核对明细 ⑥`,
|
||||
markNa: '无法核对',
|
||||
noReports: '这次运行没有报告文档,下方只核对运行记录。',
|
||||
@@ -208,6 +211,8 @@
|
||||
inputFrom: (query, row, column) => (query ? `查询 ${query}` : '') + (Number.isInteger(row) ? ` · 第 ${row + 1} 行` : '') + (column ? ` · ${column}` : ''),
|
||||
inputNotAttached: '快照未随文件附带(见核对明细 ③),输入的值按口径卡的记录展示',
|
||||
statusOk: (seq, what) => `与第 ${seq} 条事件(${what})一致`,
|
||||
statusOkRejected: (seq, what) => `与第 ${seq} 条事件(${what})一致。被驳回和手动停止在封存里是同一种结束事件,`
|
||||
+ '「被驳回」取自运行记录(这条事件的内容不随文件附带),需要在系统中对照',
|
||||
statusMismatch: (claim, seq, what) => `文件里写的是「${claim}」,与封存范围内最后一条结束类事件(第 ${seq} 条,${what})不符`,
|
||||
statusLegacy: '早期运行在结束事件落库之前封存,结束事件不在封存范围内,运行结果无法核对',
|
||||
statusNoEvent: '封存范围内没有运行结束、失败或取消的事件,运行结果无法核对',
|
||||
@@ -1509,7 +1514,8 @@
|
||||
*/
|
||||
function checkMeta(meta, out) {
|
||||
const items = []
|
||||
const add = (field, shown, claim, result, reason) => items.push({ field, shown, claim, result, reason })
|
||||
// mark:显示值旁边那一句的特例(默认按结果写已核对 / 不通过 / 无法核对)
|
||||
const add = (field, shown, claim, result, reason, mark) => items.push({ field, shown, claim, result, reason, mark })
|
||||
const order = out.order
|
||||
const sealOk = out.manifest[0].result === PASS
|
||||
const label = (map, value) => map[value] || (value ? String(value) : '—')
|
||||
@@ -1519,7 +1525,10 @@
|
||||
// 它就是范围内的最后一条;早期运行在 run.finished 落库之前封存,成功结束的那条在范围外,失败、取消的那条在范围的最后。
|
||||
// 先失败、经继续运行接着跑的,范围里那条运行失败之后运行又继续了,它不是运行的结束:拿它去对,元信息改成「已失败」
|
||||
// 也会被标成已核对。真正截止的那条不在范围里时,只有早期运行成功结束说得通(无法核对),别的说法都没有事件印证
|
||||
const statusClaim = label(STATUS_LABEL, meta.status)
|
||||
// 被驳回(cancelled + end_reason=rejected)照运行记录写「被驳回」,不写成手动停止的「已取消」;能核对的只有结束事件
|
||||
// 是运行取消这一半,另一半写明取自运行记录
|
||||
const rejected = meta.status === 'cancelled' && meta.end_reason === 'rejected'
|
||||
const statusClaim = rejected ? STATUS_LABEL.rejected : label(STATUS_LABEL, meta.status)
|
||||
const lastSeq = order.length ? order[order.length - 1].seq : null
|
||||
const final = [...order].reverse().find((ev) => TERMINAL[ev.type])
|
||||
// 同一序号的几条(老数据里有)都算最后:结束事件和别的事件撞了序号,不能因此说它之后运行又继续了
|
||||
@@ -1527,8 +1536,12 @@
|
||||
if (!sealOk) add('status', statusClaim, statusClaim, NA, T.metaNeedsManifest)
|
||||
else if (end) {
|
||||
const ok = TERMINAL[end.type] === meta.status
|
||||
add('status', label(STATUS_LABEL, TERMINAL[end.type]), statusClaim, ok ? PASS : FAIL,
|
||||
ok ? T.statusOk(end.seq, what(end)) : T.statusMismatch(statusClaim, end.seq, what(end)))
|
||||
if (ok && rejected) {
|
||||
add('status', statusClaim, statusClaim, PASS, T.statusOkRejected(end.seq, what(end)), T.markPassRejected)
|
||||
} else {
|
||||
add('status', label(STATUS_LABEL, TERMINAL[end.type]), statusClaim, ok ? PASS : FAIL,
|
||||
ok ? T.statusOk(end.seq, what(end)) : T.statusMismatch(statusClaim, end.seq, what(end)))
|
||||
}
|
||||
} else if (meta.legacy && meta.status === 'succeeded') add('status', statusClaim, statusClaim, NA, T.statusLegacy)
|
||||
// 显示的值写「—」:范围里没有说明运行怎样结束的事件,不能把文件里写的当成核对出来的
|
||||
else if (final) add('status', '—', statusClaim, FAIL, T.statusContinued(statusClaim, final.seq, what(final)))
|
||||
@@ -1899,7 +1912,7 @@
|
||||
const checked = (field, fallback) => {
|
||||
const item = out.meta.find((m) => m.field === field)
|
||||
if (!item) return fallback
|
||||
const note = item.result === PASS ? T.markPass : item.result === FAIL ? T.markFail(item.claim) : T.markNa
|
||||
const note = item.mark ?? (item.result === PASS ? T.markPass : item.result === FAIL ? T.markFail(item.claim) : T.markNa)
|
||||
return h('span', { 'data-meta': field, 'data-result': item.result }, item.shown,
|
||||
h('span', { class: item.result === FAIL ? 'bad' : 'muted' }, `(${note})`))
|
||||
}
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
"""问数据(一轮:提问 → 建图 → 运行 → 复核)在后端的部分。
|
||||
|
||||
- turn.py:一轮里的几个「决定」(要不要运行、输入怎么填、要不要带原图重来、重来时说什么),问数据页和评测走同一套;
|
||||
- questions.py:问过的问题原文和它们的聚类,全站只有这一份(治理接口、以后的口径账本、评测都调它)。
|
||||
"""
|
||||
@@ -0,0 +1,115 @@
|
||||
"""问过的问题和它们的聚类:全站只有这一份(docs/design/next-round/00-总方案.md X2)。
|
||||
|
||||
同一个问题被反复问,就该长成模板里的固定一节、沉淀成口径——探索问题聚类(/governance/exploratory-clusters)、以后口径
|
||||
账本的「可沉淀的问题」、评测的问题重复度(E3)量的都是这件事,各算一份的话几个数对不上,所以取问题和聚类只写在这里。
|
||||
|
||||
**取什么。** 只取问题原文,不取整份输入:以前把运行的整份 input(含 history)拿去聚,同一段对话里前几轮的问答会把不同的
|
||||
问题聚成一类,重复度偏高。
|
||||
|
||||
- 问数据:一轮一条(conversation_turns.question,用户原本问的那句话)。同一轮里的自动重来、用户点的「重试」只是换了这一轮
|
||||
挂着的运行,不多出一条——按运行数的话,答坏了自动重来一次就被算成问了两遍。没挂运行的轮次(只回了一句、助手反问)
|
||||
也算:问题照样被问过。
|
||||
- 别的探索运行(画布试跑、蓝图试运行、接口……):输入里有 question 这个字段的取它;没有的不收(输入里别的字段不是问题)。
|
||||
问数据的运行由上面一轮一条代表,这里不再数。
|
||||
|
||||
**不收什么。** 系统派生的运行(评测、口径对账、口径召回:engine/origin.counted_runs)不收,评测题不能把聚类刷高;
|
||||
这个人读不了的只给拥有者的运行(auth/access.hidden_runs)不收;正式运行不收(探索才谈得上「沉淀」)。
|
||||
|
||||
聚类用 memory/embeddings 的向量贪心聚:按时间从新到旧,和某一类的种子足够像就并进去,否则自立一类。本地向量是词频
|
||||
哈希、语义能力弱,结果是下限——调用方据 embedder 照实写明用的是哪个向量模型。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Sequence
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import func, not_, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
#: 缺省的相似度阈值(同治理接口一直以来的 0.62)
|
||||
DEFAULT_THRESHOLD = 0.62
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Question:
|
||||
"""问过的一个问题。turn_id 是问数据的那一轮(别的探索运行没有);run_id 是它挂着的运行(只回了一句的轮次没有)。"""
|
||||
|
||||
text: str
|
||||
at: datetime | None
|
||||
run_id: str | None = None
|
||||
turn_id: str | None = None
|
||||
workflow: str | None = None
|
||||
origin: str | None = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class QuestionCluster:
|
||||
"""一类问题:members 按时间从新到旧,第一个是种子。"""
|
||||
|
||||
members: list[Question] = field(default_factory=list)
|
||||
|
||||
@property
|
||||
def count(self) -> int:
|
||||
return len(self.members)
|
||||
|
||||
|
||||
async def recent_questions(session: AsyncSession, principal: Any, *, limit: int) -> list[Question]:
|
||||
"""最近问过的问题,从新到旧,最多 limit 条(取法见模块开头)。principal 用来去掉这个人读不了的只给拥有者的运行。"""
|
||||
from app.auth.access import hidden_runs
|
||||
from app.db.models import Conversation, ConversationTurn, Run
|
||||
from app.engine.origin import counted_runs
|
||||
|
||||
hidden = hidden_runs(principal)
|
||||
turns = (await session.execute(
|
||||
select(ConversationTurn.id, ConversationTurn.question, ConversationTurn.created_at, ConversationTurn.run_id,
|
||||
Run.workflow_name)
|
||||
.join(Conversation, Conversation.id == ConversationTurn.conversation_id)
|
||||
.outerjoin(Run, Run.id == ConversationTurn.run_id)
|
||||
.where(Conversation.kind == "chat", func.trim(ConversationTurn.question) != "", counted_runs(), not_(hidden))
|
||||
.order_by(ConversationTurn.created_at.desc()).limit(limit)
|
||||
)).all()
|
||||
runs = (await session.execute(
|
||||
select(Run.id, Run.input, Run.workflow_name, Run.created_at, Run.origin_kind)
|
||||
.where(Run.run_class != "formal", func.coalesce(Run.origin_kind, "") != "chat", counted_runs(), not_(hidden),
|
||||
func.json_type(Run.input, "$.question") == "text")
|
||||
.order_by(Run.created_at.desc()).limit(limit)
|
||||
)).all()
|
||||
out = [Question(text=question.strip(), at=at, run_id=run_id, turn_id=turn_id, workflow=workflow, origin="chat")
|
||||
for turn_id, question, at, run_id, workflow in turns]
|
||||
for run_id, payload, workflow, at, origin in runs:
|
||||
text = (payload or {}).get("question") if isinstance(payload, dict) else None
|
||||
if isinstance(text, str) and text.strip():
|
||||
out.append(Question(text=text.strip(), at=at, run_id=run_id, workflow=workflow, origin=origin))
|
||||
out.sort(key=lambda q: q.at.timestamp() if q.at is not None else 0.0, reverse=True)
|
||||
return out[:limit]
|
||||
|
||||
|
||||
async def cluster_questions(questions: Sequence[Question], *, threshold: float = DEFAULT_THRESHOLD,
|
||||
) -> list[QuestionCluster]:
|
||||
"""向量贪心聚类(顺序同 questions:从新到旧)。"""
|
||||
from app.memory import embeddings
|
||||
|
||||
if not questions:
|
||||
return []
|
||||
vectors = await embeddings.embed_texts([q.text for q in questions])
|
||||
clusters: list[tuple[int, QuestionCluster]] = []
|
||||
for i, question in enumerate(questions):
|
||||
for seed, cluster in clusters:
|
||||
if embeddings.cosine(vectors[i], vectors[seed]) >= threshold:
|
||||
cluster.members.append(question)
|
||||
break
|
||||
else:
|
||||
clusters.append((i, QuestionCluster(members=[question])))
|
||||
return [cluster for _, cluster in clusters]
|
||||
|
||||
|
||||
def embedder() -> str:
|
||||
"""聚类用的是哪个向量模型(memory/embeddings.embedder_id):本地哈希的结果只是下限,调用方照实写出来。"""
|
||||
from app.memory import embeddings
|
||||
|
||||
return embeddings.embedder_id()
|
||||
|
||||
|
||||
__all__ = ["DEFAULT_THRESHOLD", "Question", "QuestionCluster", "cluster_questions", "embedder", "recent_questions"]
|
||||
@@ -0,0 +1,129 @@
|
||||
"""问数据一轮里的几个「决定」(细案 01 做法 1,总方案 2.1)。
|
||||
|
||||
一轮问数据是:建图(助手的生成过程,api/copilot.answer_ops)→ 要不要运行 → 运行(输入怎么填)→ 复核 → 要不要带着原图
|
||||
重来一次、重来时说什么。以前这几个决定都写在问数据页里(frontend/src/store/chat.ts 的 plan / launch / settle / retry),
|
||||
后端跑不了一轮问数据:评测要么照着抄一份(迟早走样,量出来的不是用户拿到的那一轮),要么做不了;这也违反「界面层不放
|
||||
逻辑」。现在决定写在这里,是纯函数:
|
||||
|
||||
- next_step:生成过程交出结局之后做什么;
|
||||
- fill_input:用户的问题填进这张图的入口字段;
|
||||
- retry_after_review:复核之后要不要带着原图重来一次;
|
||||
- retry_instruction:重来时交给助手的那句话(两种措辞)。
|
||||
|
||||
接口把决定交给问数据页:/copilot/generate-stream 的 final 带 next(run | ready | fail),/copilot/review 的结果带
|
||||
retry_instruction(要重来时是那句话,不重来是 null)。问数据页照后端给的走;老后端没有这两个字段时退回它自己那份同样的
|
||||
规则(tests/test_chat_turn_policy.py 逐条对照两边的常量和措辞)。评测(第 3 波)用同一组函数把一轮串起来。
|
||||
|
||||
只搬「决定」:重连、断点续看、取消、复核等多久这些仍在问数据页(把整轮编排搬到后端要重做这些,单独立项)。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from collections.abc import Mapping
|
||||
from typing import Any
|
||||
|
||||
#: 一轮最多自动试几次(含第一次):第二次还不行多半是问题本身没说清,该让用户介入(同 chat.ts 的 MAX_ATTEMPTS)
|
||||
MAX_ATTEMPTS = 2
|
||||
|
||||
#: final 之后的三种走法(final.next):运行、只建不跑(摆着等用户点运行)、判失败(没过检查的图不拿去跑)
|
||||
NEXT_RUN, NEXT_READY, NEXT_FAIL = "run", "ready", "fail"
|
||||
NEXT_STEPS = (NEXT_RUN, NEXT_READY, NEXT_FAIL)
|
||||
#: 别的结局:直接回答了(reply)、助手反问了等用户答(asked)、没自查过的老服务端交来没过检查的图时重来一次(retry)
|
||||
NEXT_REPLY, NEXT_ASKED, NEXT_RETRY = "reply", "asked", "retry"
|
||||
|
||||
#: 问题往哪个入口字段放:名字像「问题」的优先(同 chat.ts fillInput 的正则,大小写不敏感)
|
||||
QUESTION_FIELD = re.compile(r"question|query|topic|input|问题|需求|主题", re.IGNORECASE)
|
||||
|
||||
#: 带着原图重来时交给助手的话:原图交回去改(调大步数、改参数这种事就是改一个字段,不是推倒重来)
|
||||
RETRY_WITH_BASE = ("这个工作流上一次运行没有成功:{reason}。\n\n"
|
||||
"请针对这个原因改它(比如调大最大步数、改正工具参数、换个查询方式),"
|
||||
"改完继续回答原来的问题:{question}")
|
||||
#: 没有图可交回(上一次连图都没建出来)时:原问题后面缀上原因,从零再建
|
||||
RETRY_FROM_SCRATCH = "{question}\n\n(上一次没有成功:{reason}。请针对这个原因调整,重新给出完整的工作流。)"
|
||||
|
||||
|
||||
def _blockers(final: Mapping[str, Any]) -> list[str]:
|
||||
return [str(i.get("message") or "") for i in final.get("issues") or []
|
||||
if isinstance(i, Mapping) and i.get("level") == "error"]
|
||||
|
||||
|
||||
def next_step(terminal: Mapping[str, Any] | None, *, self_checked: bool = True) -> str:
|
||||
"""生成过程交出结局之后做什么。terminal 是最后那条结局操作(final / reply / error);流结束时只有反问(ask)、没有
|
||||
别的结局的给 None。
|
||||
|
||||
- reply:直接回答,这一轮到此为止;error:判失败;只有反问:等用户回答;
|
||||
- final:有 error 级问题时,服务端已经自查过(收到过 check 的结论)就判失败——没过的问题已经交回模型在同一轮里改过
|
||||
了,再整轮重来是再烧一次完整生成、改的还是同一批问题;没自查过(老服务端)才重来一次(retry);模型说不运行
|
||||
(autorun=false:用户要的是流程本身、或者问了用户)只建不跑;其余运行。
|
||||
"""
|
||||
if terminal is None:
|
||||
return NEXT_ASKED
|
||||
kind = terminal.get("op")
|
||||
if kind == "reply":
|
||||
return NEXT_REPLY
|
||||
if kind != "final":
|
||||
return NEXT_FAIL
|
||||
if _blockers(terminal):
|
||||
return NEXT_FAIL if self_checked else NEXT_RETRY
|
||||
if terminal.get("autorun") is False:
|
||||
return NEXT_READY
|
||||
return NEXT_RUN
|
||||
|
||||
|
||||
def blocked_reason(final: Mapping[str, Any]) -> str:
|
||||
"""没过检查的图重来时的原因(只有老服务端那条路用得到)。"""
|
||||
return "生成的工作流没有通过校验:" + ";".join(_blockers(final))
|
||||
|
||||
|
||||
def fill_input(graph: Mapping[str, Any] | None, question: str, history: str = "") -> dict[str, Any]:
|
||||
"""用户的问题填进这张图的入口字段(同 chat.ts 的 fillInput)。
|
||||
|
||||
入口没声明字段:{question, history}(下游可能用 input.question)。声明了:每个字段先填默认值(没有就空串),问题放进
|
||||
名字像「问题」的那个,其次第一个必填的,再不然第一个;history(前几轮压成的文字)照样带上——图跑起来时模型得知道
|
||||
「它们」指的是上一轮那批数据。
|
||||
"""
|
||||
nodes = (graph or {}).get("nodes") or []
|
||||
entry = next((n for n in nodes if isinstance(n, Mapping) and n.get("type") == "input"), None)
|
||||
config = ((entry or {}).get("data") or {}).get("config") or {}
|
||||
fields = [f for f in (config.get("fields") or []) if isinstance(f, Mapping)]
|
||||
if not fields:
|
||||
return {"question": question, "history": history}
|
||||
payload: dict[str, Any] = {"history": history}
|
||||
for f in fields:
|
||||
if f.get("name"):
|
||||
payload[str(f["name"])] = f["default"] if f.get("default") is not None else ""
|
||||
named = next((f for f in fields if QUESTION_FIELD.search(str(f.get("name") or ""))), None)
|
||||
target = named or next((f for f in fields if f.get("required")), None) or fields[0]
|
||||
if target.get("name"):
|
||||
payload[str(target["name"])] = question
|
||||
return payload
|
||||
|
||||
|
||||
def review_reason(review: Mapping[str, Any]) -> str:
|
||||
"""复核判不可用时重来的原因:各条信号的说明,没有就用给用户看的那句说明。"""
|
||||
details = [str(s.get("detail") or "") for s in review.get("signals") or [] if isinstance(s, Mapping)]
|
||||
return ";".join(d for d in details if d) or str(review.get("note") or "")
|
||||
|
||||
|
||||
def retry_after_review(review: Mapping[str, Any] | None, attempt: int) -> bool:
|
||||
"""复核之后要不要带着原图重来一次:复核说换个配置重跑大概率就好(retry)、答案不可信(severity=broken)、这一轮还没
|
||||
试满(attempt 从 0 数,attempt + 1 < MAX_ATTEMPTS)。"""
|
||||
if not review:
|
||||
return False
|
||||
return bool(review.get("retry")) and review.get("severity") == "broken" and attempt + 1 < MAX_ATTEMPTS
|
||||
|
||||
|
||||
def retry_instruction(question: str, reason: str, has_base: bool) -> str:
|
||||
"""重来时交给助手的话。has_base:有上一次建出的图可以交回去改。"""
|
||||
template = RETRY_WITH_BASE if has_base else RETRY_FROM_SCRATCH
|
||||
return template.format(question=question, reason=reason)
|
||||
|
||||
|
||||
def has_nodes(graph: Any) -> bool:
|
||||
"""一张图有没有节点(有才交回去改)。"""
|
||||
return isinstance(graph, Mapping) and bool(graph.get("nodes"))
|
||||
|
||||
|
||||
__all__ = ["MAX_ATTEMPTS", "NEXT_ASKED", "NEXT_FAIL", "NEXT_READY", "NEXT_REPLY", "NEXT_RETRY", "NEXT_RUN", "NEXT_STEPS",
|
||||
"QUESTION_FIELD", "RETRY_FROM_SCRATCH", "RETRY_WITH_BASE", "blocked_reason", "fill_input", "has_nodes",
|
||||
"next_step", "retry_after_review", "retry_instruction", "review_reason"]
|
||||
@@ -1,9 +1,10 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import ipaddress
|
||||
import re
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
from typing import Annotated
|
||||
from typing import Annotated, Any
|
||||
|
||||
from pydantic import field_validator
|
||||
from pydantic_settings import BaseSettings, NoDecode, SettingsConfigDict
|
||||
@@ -11,6 +12,9 @@ from pydantic_settings import BaseSettings, NoDecode, SettingsConfigDict
|
||||
BACKEND_DIR = Path(__file__).resolve().parents[2]
|
||||
PROJECT_DIR = BACKEND_DIR.parent
|
||||
|
||||
#: 嵌入白名单里一项的主机部分:域名(可以 *. 开头)或方括号里的 IPv6,可以带端口;不许路径、查询、用户名
|
||||
_ORIGIN_HOST = re.compile(r"(?:\*\.)?(?:[A-Za-z0-9-]+\.)*[A-Za-z0-9-]+(?::\d{1,5})?|\[[0-9A-Fa-f:.]+\](?::\d{1,5})?")
|
||||
|
||||
# session_id 拼进路径前必须过这一关。允许的字符刻意收得很窄:字母数字加
|
||||
# 连字符下划线——点号也不放行,免得出现 ".." 这种能往上走的名字。
|
||||
_SESSION_MAX = 64
|
||||
@@ -60,10 +64,18 @@ class Settings(BaseSettings):
|
||||
auth_session_max_days: int = 7
|
||||
# 会话 cookie 带不带 Secure。不设时按请求本身是不是 https 定:本机 http 开发照常能登录
|
||||
auth_cookie_secure: bool | None = None
|
||||
# 对外的访问地址(反向代理后面时填,如 https://agentlab.example.com)。OIDC 的回调地址按它拼;
|
||||
# 不设时按请求的 Host 拼
|
||||
# 对外的访问地址(反向代理后面时填,如 https://agentlab.example.com)。全站只认一个(core/public_url.py):
|
||||
# 设了它,设置里的「对外访问地址」就不生效、不能改;通知和 Webhook 的链接、OIDC 的回调、来源核对都按它拼。
|
||||
# 不设时用设置里的那一项,再没有就按请求的 Host 拼(通知里不附链接)
|
||||
public_url: str = ""
|
||||
|
||||
# --- 安全响应头(app/web_security.py)---
|
||||
# 允许哪些站点用 iframe 嵌整站(同站的门户,如 https://portal.example.com),逗号分隔。本站自己总是允许的。
|
||||
# 不设时只有本站能嵌(frame-ancestors 'self',另加 X-Frame-Options: SAMEORIGIN)
|
||||
frame_ancestors: Annotated[list[str], NoDecode] = []
|
||||
# 分享页(/s)允许被哪些站点嵌,逗号分隔,写法同上。分享页只认这一项,不认上面那一项
|
||||
embed_origins: Annotated[list[str], NoDecode] = []
|
||||
|
||||
# --- 沙箱 ---
|
||||
sandbox_backend: str = "auto" # auto | microvm | seatbelt | bubblewrap | local | off
|
||||
sandbox_timeout: int = 30
|
||||
@@ -121,7 +133,8 @@ class Settings(BaseSettings):
|
||||
"https://223.5.5.5/resolve",
|
||||
]
|
||||
|
||||
@field_validator("http_tool_allowlist", "http_tool_fake_ip_ranges", "http_tool_doh_urls", mode="before")
|
||||
@field_validator("http_tool_allowlist", "http_tool_fake_ip_ranges", "http_tool_doh_urls", "frame_ancestors",
|
||||
"embed_origins", mode="before")
|
||||
@classmethod
|
||||
def _comma_list(cls, value: object) -> object:
|
||||
# 环境变量里写逗号分隔;也兼容 JSON 数组的写法
|
||||
@@ -134,6 +147,37 @@ class Settings(BaseSettings):
|
||||
return [part.strip() for part in text.split(",") if part.strip()]
|
||||
return value
|
||||
|
||||
@field_validator("public_url")
|
||||
@classmethod
|
||||
def _valid_public_url(cls, value: str) -> str:
|
||||
# 写错了启动时就报出来:以前拼出 agentlab.example.com/api/auth/oidc/callback 这样的回调,到登录时才发现
|
||||
from app.core.public_url import ENV_NAME, normalize, url_problem
|
||||
|
||||
if not value.strip():
|
||||
return ""
|
||||
if problem := url_problem(value):
|
||||
what = "不能带 ? 或 # 之后的部分" if problem == "suffix" else "需要是 http:// 或 https:// 开头的完整地址"
|
||||
raise ValueError(f"{ENV_NAME}={value!r} {what},例如 https://agentlab.example.com")
|
||||
return normalize(value) or ""
|
||||
|
||||
@field_validator("frame_ancestors", "embed_origins")
|
||||
@classmethod
|
||||
def _valid_origins(cls, value: list[str], info: Any) -> list[str]:
|
||||
# 只收「协议://主机[:端口]」,主机可以写 *.example.com。路径、裸 *、没写协议的写错了启动时就报出来,别到嵌入
|
||||
# 失败(或者整站被谁都能嵌)才发现。'self' 本来就有,写了也不重复
|
||||
name = f"AGENTLAB_{(info.field_name or '').upper()}"
|
||||
out: list[str] = []
|
||||
for raw in value:
|
||||
item = raw.strip().rstrip("/")
|
||||
if item == "'self'":
|
||||
continue
|
||||
scheme, _, host = item.partition("://")
|
||||
if scheme.lower() not in ("http", "https") or not _ORIGIN_HOST.fullmatch(host):
|
||||
raise ValueError(f"{name} 里的「{raw}」需要写成「协议://主机」(可以带端口),例如 https://portal.example.com")
|
||||
if (origin := f"{scheme.lower()}://{host.lower()}") not in out:
|
||||
out.append(origin)
|
||||
return out
|
||||
|
||||
@field_validator("auth_cookie_secure", mode="before")
|
||||
@classmethod
|
||||
def _blank_is_auto(cls, value: object) -> object:
|
||||
|
||||
@@ -0,0 +1,439 @@
|
||||
"""后台任务的共用运行时:数据目录的起草(data/catalog_jobs.py)和剖析(data/catalog_profile_jobs.py)各自建表、各自的步骤,
|
||||
生命周期是同一套;之后的评测、后台探查也用它(docs/design/next-round/00-总方案.md X5)。
|
||||
|
||||
**为什么在 core。** 最早只服务数据目录,放在 data/catalog_job_runtime.py;评测(一次评测一个任务、每题一项)和后台探查
|
||||
(同源排队)要的是同一样东西,从 data 包里导入一个「数据目录的」运行时说不通。原路径留着再导出(同一个模块对象里的
|
||||
名字,lanes 也是同一条队),已有的 import 不用改,正在改起草、剖析的工作也不会因此冲突。
|
||||
|
||||
**为什么抽出来。** 剖析以前是一次同步长请求,和起草改成后台任务之前是同一类问题:关掉页面结果就没了、不能停、不留痕。
|
||||
两边要的是同一样东西——进程内跑、能停、重启标中断、写库不被停止打断、同源排队——各写一份,迟早一边修了另一边没修。
|
||||
共用的是生命周期代码,不是表:两种任务的进度完全不同(起草按段、块、模型用量,剖析按检查项、查询),合成一张
|
||||
「kind + JSON」的通用任务表要丢掉起草那些强类型列,还要迁移刚稳定的起草数据。
|
||||
|
||||
- **状态词**(ACTIVE、FINISHED、TASK_STATUSES、TASK_DONE、RETRYABLE、OPEN):任务 queued → running → succeeded / partial
|
||||
/ failed / cancelled / interrupted;逐表 pending → running → ok / partial / failed / skipped / cancelled。两种任务、
|
||||
接口、界面都用这一套词。
|
||||
- **SerialWriter**:同一个任务的库写入一条一条来,shield 起来——停止任务时不会打断写到一半的库操作;收尾前 settle 等它们
|
||||
写完。
|
||||
- **JobRun / JobManager**:进程内一个 asyncio 任务一件事(照 app/scheduler 的写法):start / stop(grace) / wait / running
|
||||
/ shutdown。JobRun.run 把「进队、等轮到自己、执行、按停止的原因收尾、出队」写一次,各种任务只写执行和收尾的那几步。
|
||||
- **SourceLanes**:每个数据源一条先来先到的队。起草、剖析都打业务库或模型、都写同一个源的目录:同源串行(先建的先跑),
|
||||
不同源并行。任务阻塞地占:建出来是 queued,轮到了才 running。同步起草、导入目录这类同步写操作不阻塞地试占:占不到
|
||||
说明源上正有别的事,接口回 409;占到了,执行期间新建的任务排在它后面。**队里的事是开放的**:运行时不认具体哪几种,
|
||||
每种在自己的模块里 register_lane_kind 登记一次(kind → 叫法),进队时只核对登记过——第 2 波的后台探查(introspect)
|
||||
登记一行就排进同一条队,不用改这里;拼错的 kind 当场报错,不会悄悄排成一种谁都不认识的事。
|
||||
- **close_job / interrupt_on_startup**:收尾(还没结束的表记成某个状态,任务记成某个状态),和服务启动时的整理:排队、
|
||||
进行中的标 interrupted,没结束的表记 cancelled。**不自动续跑**:复制出来的数据目录(检查脚本、验收副本)启动时不能
|
||||
自己去调模型、查业务库,界面让人点「重试」接着做。
|
||||
|
||||
进程内登记、进程内的队:服务端是单进程部署,多进程部署时要换成库里的锁。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import contextlib
|
||||
import logging
|
||||
from collections.abc import AsyncIterator, Awaitable, Callable, Iterator
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import select, update
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.db.base import SessionLocal, utcnow
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
#: 任务还没结束的状态(同一个源同一种任务只能有一个)、结束了的状态
|
||||
ACTIVE = ("queued", "running")
|
||||
FINISHED = ("succeeded", "partial", "failed", "cancelled", "interrupted")
|
||||
#: 逐表的状态,其中算「已结束」的、重试时要带上的、还没结束的
|
||||
TASK_STATUSES = ("pending", "running", "ok", "partial", "failed", "skipped", "cancelled")
|
||||
TASK_DONE = ("ok", "partial", "failed", "skipped", "cancelled")
|
||||
RETRYABLE = ("failed", "partial", "cancelled")
|
||||
OPEN = ("pending", "running")
|
||||
|
||||
#: 停止进行中的任务时最多等它收尾多久(秒);关停服务时另给(shutdown 的 grace)
|
||||
STOP_GRACE_S = 10.0
|
||||
SHUTDOWN_GRACE_S = 3.0
|
||||
|
||||
|
||||
class JobFailed(Exception):
|
||||
"""任务整体出错(数据源没了、快照没了、模型用不了……):message 就是任务的 error。"""
|
||||
|
||||
|
||||
# ==========================================================================
|
||||
# 库写入
|
||||
# ==========================================================================
|
||||
|
||||
|
||||
class SerialWriter:
|
||||
"""一个任务的库写入:一条一条来,每条开一个会话;shield 起来,停止任务时不会打断写到一半的库操作。"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._writes: set[asyncio.Task[Any]] = set()
|
||||
self._lock = asyncio.Lock()
|
||||
|
||||
async def __call__(self, fn: Callable[..., Awaitable[Any]], *args: Any) -> Any:
|
||||
async def go() -> Any:
|
||||
async with self._lock:
|
||||
async with SessionLocal() as session:
|
||||
return await fn(session, *args)
|
||||
|
||||
task = asyncio.ensure_future(go())
|
||||
self._writes.add(task)
|
||||
task.add_done_callback(self._writes.discard)
|
||||
return await asyncio.shield(task)
|
||||
|
||||
async def settle(self) -> None:
|
||||
"""等手上的库写入都写完(停止、出错收尾之前)。"""
|
||||
while self._writes:
|
||||
await asyncio.gather(*list(self._writes), return_exceptions=True)
|
||||
|
||||
|
||||
# ==========================================================================
|
||||
# 每个源一条队
|
||||
# ==========================================================================
|
||||
|
||||
|
||||
#: 同源队里能排的事:{kind: 叫法}。开放的,各种事在自己的模块里登记(register_lane_kind):数据目录登记了 draft 起草任务、
|
||||
#: profile 剖析任务、sync_draft 同步起草、import 导入目录(data/catalog_jobs.py)
|
||||
_LANE_KINDS: dict[str, str] = {}
|
||||
|
||||
|
||||
def register_lane_kind(kind: str, label: str) -> str:
|
||||
"""登记一种同源队里的事(重复登记同一个叫法无害,换叫法报错:同一个 kind 只能有一种说法)。返回 kind。"""
|
||||
if not kind or not label:
|
||||
raise ValueError("同源队里的事要有 kind 和叫法")
|
||||
known = _LANE_KINDS.get(kind)
|
||||
if known is not None and known != label:
|
||||
raise ValueError(f"同源队里的「{kind}」已经登记成「{known}」,不能再登记成「{label}」")
|
||||
_LANE_KINDS[kind] = label
|
||||
return kind
|
||||
|
||||
|
||||
def lane_kinds() -> dict[str, str]:
|
||||
"""登记过的各种事:{kind: 叫法}。"""
|
||||
return dict(_LANE_KINDS)
|
||||
|
||||
|
||||
def lane_label(kind: str) -> str:
|
||||
"""一种事的叫法(「起草」「剖析」……);没登记的照原样给 kind。"""
|
||||
return _LANE_KINDS.get(kind, kind)
|
||||
|
||||
|
||||
class LaneTicket:
|
||||
"""在一个源的队里的一个位置。kind 是占着它的是什么(登记过的一种,见 register_lane_kind);job_id 是任务的(同步写
|
||||
操作没有)。轮到它时 granted 置位。"""
|
||||
|
||||
def __init__(self, lanes: SourceLanes, source_id: str, kind: str, job_id: str | None,
|
||||
loop: asyncio.AbstractEventLoop) -> None:
|
||||
self._lanes = lanes
|
||||
self.source_id = source_id
|
||||
self.kind = kind
|
||||
self.job_id = job_id
|
||||
self.loop = loop
|
||||
self.granted = asyncio.Event()
|
||||
self.released = False
|
||||
|
||||
async def wait(self) -> None:
|
||||
"""等轮到自己。"""
|
||||
await self.granted.wait()
|
||||
|
||||
def release(self) -> None:
|
||||
"""出队(轮到了、没轮到都行,重复调用无害):排在后面的第一个接着开始。"""
|
||||
self._lanes._release(self)
|
||||
|
||||
|
||||
class LaneBusy(Exception):
|
||||
"""不阻塞地试占时这个源的队里已经有人:holder 是排在最前面(正在做)的那一个。"""
|
||||
|
||||
def __init__(self, holder: LaneTicket) -> None:
|
||||
super().__init__(holder.kind)
|
||||
self.holder = holder
|
||||
|
||||
|
||||
class SourceLanes:
|
||||
"""每个数据源一条先来先到的队。"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._queues: dict[str, list[LaneTicket]] = {}
|
||||
|
||||
def _queue(self, source_id: str) -> list[LaneTicket]:
|
||||
"""这个源的队。别的事件循环留下的(测试每个用例一个循环)去掉:那个循环里没人会再出队了。"""
|
||||
queue = self._queues.get(source_id) or []
|
||||
loop = asyncio.get_running_loop()
|
||||
alive = [t for t in queue if t.loop is loop and not t.released]
|
||||
if alive:
|
||||
self._queues[source_id] = alive
|
||||
alive[0].granted.set()
|
||||
else:
|
||||
self._queues.pop(source_id, None)
|
||||
return alive
|
||||
|
||||
def enter(self, source_id: str, kind: str, job_id: str | None = None) -> LaneTicket:
|
||||
"""排进队尾,马上返回(不等)。排在最前面的立刻 granted;调用方 await ticket.wait() 等轮到自己。kind 要登记过。"""
|
||||
if kind not in _LANE_KINDS:
|
||||
raise ValueError(f"同源队里没有登记「{kind}」这种事,先 register_lane_kind")
|
||||
queue = self._queue(source_id)
|
||||
ticket = LaneTicket(self, source_id, kind, job_id, asyncio.get_running_loop())
|
||||
queue.append(ticket)
|
||||
self._queues[source_id] = queue
|
||||
if len(queue) == 1:
|
||||
ticket.granted.set()
|
||||
return ticket
|
||||
|
||||
def try_enter(self, source_id: str, kind: str) -> LaneTicket:
|
||||
"""队空着才占(马上就轮到);有人在队里抛 LaneBusy。"""
|
||||
queue = self._queue(source_id)
|
||||
if queue:
|
||||
raise LaneBusy(queue[0])
|
||||
return self.enter(source_id, kind)
|
||||
|
||||
def holders(self, source_id: str) -> list[LaneTicket]:
|
||||
"""队里的各位,排在最前面(正在做)的在前。"""
|
||||
return list(self._queue(source_id))
|
||||
|
||||
def ahead(self, source_id: str, job_id: str) -> list[str] | None:
|
||||
"""排在这个任务前面的各位是什么(kind,正在做的在前);这个任务不在队里(刚建、后台还没排进来)返回 None。"""
|
||||
queue = self._queue(source_id)
|
||||
for i, ticket in enumerate(queue):
|
||||
if ticket.job_id == job_id:
|
||||
return [t.kind for t in queue[:i]]
|
||||
return None
|
||||
|
||||
def _release(self, ticket: LaneTicket) -> None:
|
||||
if ticket.released:
|
||||
return
|
||||
ticket.released = True
|
||||
queue = self._queues.get(ticket.source_id)
|
||||
if not queue or ticket not in queue:
|
||||
return
|
||||
head = queue[0] is ticket
|
||||
queue.remove(ticket)
|
||||
if not queue:
|
||||
self._queues.pop(ticket.source_id, None)
|
||||
elif head:
|
||||
queue[0].granted.set()
|
||||
|
||||
@contextlib.asynccontextmanager
|
||||
async def hold(self, source_id: str, kind: str, job_id: str | None = None) -> AsyncIterator[LaneTicket]:
|
||||
"""阻塞地占:排队,轮到了再进去,出来时出队。"""
|
||||
ticket = self.enter(source_id, kind, job_id)
|
||||
try:
|
||||
await ticket.wait()
|
||||
yield ticket
|
||||
finally:
|
||||
ticket.release()
|
||||
|
||||
@contextlib.contextmanager
|
||||
def try_hold(self, source_id: str, kind: str) -> Iterator[LaneTicket]:
|
||||
"""不阻塞地试占(同步起草、导入目录):占不到抛 LaneBusy;占到了,执行期间新建的任务排在后面。"""
|
||||
ticket = self.try_enter(source_id, kind)
|
||||
try:
|
||||
yield ticket
|
||||
finally:
|
||||
ticket.release()
|
||||
|
||||
|
||||
lanes = SourceLanes()
|
||||
|
||||
|
||||
# ==========================================================================
|
||||
# 收尾、启动时的整理
|
||||
# ==========================================================================
|
||||
|
||||
|
||||
async def close_job(session: AsyncSession, job_model: Any, task_model: Any, job_id: str, job_status: str,
|
||||
job_error: str | None, task_status: str, task_error: str | None) -> None:
|
||||
"""收尾:还没结束的表记 task_status(原因 task_error),任务(还没结束的话)记 job_status。提交。"""
|
||||
now = utcnow()
|
||||
await session.execute(
|
||||
update(task_model).where(task_model.job_id == job_id, task_model.status.in_(OPEN))
|
||||
.values(status=task_status, error=task_error, finished_at=now).execution_options(synchronize_session=False))
|
||||
await session.execute(
|
||||
update(job_model).where(job_model.id == job_id, job_model.status.in_(ACTIVE))
|
||||
.values(status=job_status, error=job_error, finished_at=now).execution_options(synchronize_session=False))
|
||||
await session.commit()
|
||||
|
||||
|
||||
async def cancel_queued(session: AsyncSession, job_model: Any, task_model: Any, job_id: str) -> bool:
|
||||
"""停止一个排队中的任务:任务和没结束的表直接记 cancelled。已经被后台认领(开始了)返回 False,交给调用方按进行中
|
||||
的停。"""
|
||||
now = utcnow()
|
||||
claimed = await session.execute(
|
||||
update(job_model).where(job_model.id == job_id, job_model.status == "queued")
|
||||
.values(status="cancelled", finished_at=now).execution_options(synchronize_session=False))
|
||||
if claimed.rowcount != 1:
|
||||
await session.rollback()
|
||||
return False
|
||||
await session.execute(
|
||||
update(task_model).where(task_model.job_id == job_id, task_model.status.in_(OPEN))
|
||||
.values(status="cancelled", finished_at=now).execution_options(synchronize_session=False))
|
||||
await session.commit()
|
||||
return True
|
||||
|
||||
|
||||
async def claim(session: AsyncSession, job_model: Any, job_id: str) -> bool:
|
||||
"""认领:排队中 → 进行中。开始之前就被停止了返回 False。"""
|
||||
result = await session.execute(
|
||||
update(job_model).where(job_model.id == job_id, job_model.status == "queued")
|
||||
.values(status="running", started_at=utcnow()).execution_options(synchronize_session=False))
|
||||
await session.commit()
|
||||
return result.rowcount == 1
|
||||
|
||||
|
||||
async def interrupt_on_startup(job_model: Any, close: Callable[[AsyncSession, str], Awaitable[None]], *,
|
||||
running: Callable[[str], bool]) -> int:
|
||||
"""服务启动时:排队、进行中、而且不在这个进程里跑的任务逐个 close(各自记成 interrupted、没结束的表记 cancelled)。
|
||||
不自动续跑。返回整理了几个。"""
|
||||
async with SessionLocal() as session:
|
||||
ids = [job_id for job_id in (await session.execute(
|
||||
select(job_model.id).where(job_model.status.in_(ACTIVE)))).scalars() if not running(job_id)]
|
||||
for job_id in ids:
|
||||
await close(session, job_id)
|
||||
return len(ids)
|
||||
|
||||
|
||||
# ==========================================================================
|
||||
# 在后台执行
|
||||
# ==========================================================================
|
||||
|
||||
|
||||
class JobRun:
|
||||
"""一个任务在后台的一次执行。子类给 kind(进队时的身份)、model(任务表,进队前读源和状态),写 _run(执行)和
|
||||
_stopped / _failed / _crashed(停止、整体出错、意外出错时的收尾)。"""
|
||||
|
||||
kind = ""
|
||||
model: Any = None
|
||||
|
||||
def __init__(self, job_id: str) -> None:
|
||||
self.job_id = job_id
|
||||
self.task: asyncio.Task[None] | None = None
|
||||
#: 为什么停:user 使用者点了停止,shutdown 服务关停(标中断)
|
||||
self.stop_reason: str | None = None
|
||||
self.writer = SerialWriter()
|
||||
self.ticket: LaneTicket | None = None
|
||||
|
||||
async def _write(self, fn: Callable[..., Awaitable[Any]], *args: Any) -> Any:
|
||||
"""一条库写入(SerialWriter:一条一条来,停止时不被打断)。"""
|
||||
return await self.writer(fn, *args)
|
||||
|
||||
async def _settle(self) -> None:
|
||||
await self.writer.settle()
|
||||
|
||||
async def run(self) -> None:
|
||||
try:
|
||||
try:
|
||||
await self._queue()
|
||||
await self._run()
|
||||
except asyncio.CancelledError:
|
||||
current = asyncio.current_task()
|
||||
if current is not None:
|
||||
current.uncancel()
|
||||
await self._stopped()
|
||||
except JobFailed as e:
|
||||
await self._failed(str(e))
|
||||
except Exception as e: # noqa: BLE001 - 任务整体出错:照实记下,不留在「进行中」
|
||||
await self._crashed(e)
|
||||
finally:
|
||||
# 收尾写完才出队:排在后面的任务开始时,这个任务已经记成结束了
|
||||
if self.ticket is not None:
|
||||
self.ticket.release()
|
||||
|
||||
def lane_of(self, job: Any) -> str | None:
|
||||
"""这个任务排在哪条队里:缺省是任务的数据源(job.source_id)。不占源的任务(以后的评测:开始时拍目录快照,不挡
|
||||
起草、剖析)覆盖成返回 None,不进队、直接执行。"""
|
||||
return job.source_id
|
||||
|
||||
async def _queue(self) -> None:
|
||||
"""进这个源的队,等轮到自己。任务不在了、已经不是排队中(开始之前就被停止了)的不进队:_run 认领不到,直接结束。"""
|
||||
async with SessionLocal() as session:
|
||||
job = await session.get(self.model, self.job_id)
|
||||
source_id = self.lane_of(job) if job is not None and job.status == "queued" else None
|
||||
if source_id is None:
|
||||
return
|
||||
self.ticket = lanes.enter(source_id, self.kind, self.job_id)
|
||||
await self.ticket.wait()
|
||||
|
||||
async def _run(self) -> None:
|
||||
raise NotImplementedError
|
||||
|
||||
async def _stopped(self) -> None:
|
||||
raise NotImplementedError
|
||||
|
||||
async def _failed(self, reason: str) -> None:
|
||||
raise NotImplementedError
|
||||
|
||||
async def _crashed(self, e: Exception) -> None:
|
||||
raise NotImplementedError
|
||||
|
||||
|
||||
class JobManager:
|
||||
"""进程内的一种后台任务:一个 asyncio 任务一件事,停的时候给宽限。factory 按任务编号造一次执行(JobRun);label 是
|
||||
日志里的叫法(「数据目录起草任务」),name 是 asyncio 任务名的前缀。"""
|
||||
|
||||
def __init__(self, factory: Callable[[str], JobRun], *, label: str, name: str) -> None:
|
||||
self._runs: dict[str, JobRun] = {}
|
||||
self._factory = factory
|
||||
self.label = label
|
||||
self.name = name
|
||||
|
||||
def _alive(self, job_id: str) -> JobRun | None:
|
||||
run = self._runs.get(job_id)
|
||||
if run is None or run.task is None or run.task.done():
|
||||
return None
|
||||
try:
|
||||
if run.task.get_loop() is not asyncio.get_running_loop():
|
||||
self._runs.pop(job_id, None) # 别的事件循环留下的(测试每个用例一个循环)
|
||||
return None
|
||||
except RuntimeError:
|
||||
return None
|
||||
return run
|
||||
|
||||
def start(self, job_id: str) -> None:
|
||||
"""开始在后台执行这个(排队中的)任务:先进这个源的队,轮到了才开始。"""
|
||||
run = self._factory(job_id)
|
||||
run.task = asyncio.get_running_loop().create_task(run.run(), name=f"{self.name}-{job_id}")
|
||||
self._runs[job_id] = run
|
||||
|
||||
def forget(_: asyncio.Task[None]) -> None:
|
||||
if self._runs.get(job_id) is run:
|
||||
self._runs.pop(job_id, None)
|
||||
|
||||
run.task.add_done_callback(forget)
|
||||
|
||||
def running(self, job_id: str) -> bool:
|
||||
return self._alive(job_id) is not None
|
||||
|
||||
async def stop(self, job_id: str, reason: str, *, grace: float = STOP_GRACE_S) -> bool:
|
||||
"""停掉这个进程里在跑(或在排队等)的任务,等它收尾(最多 grace 秒)。不在这个进程里返回 False。"""
|
||||
run = self._alive(job_id)
|
||||
if run is None:
|
||||
return False
|
||||
if run.stop_reason is None:
|
||||
run.stop_reason = reason
|
||||
run.task.cancel() # type: ignore[union-attr]
|
||||
await asyncio.wait([run.task], timeout=grace) # type: ignore[list-item]
|
||||
return True
|
||||
|
||||
async def wait(self, job_id: str, timeout: float | None = None) -> None:
|
||||
"""等这个任务做完(测试、调试用)。不在跑直接返回。"""
|
||||
run = self._alive(job_id)
|
||||
if run is not None:
|
||||
await asyncio.wait([run.task], timeout=timeout) # type: ignore[list-item]
|
||||
|
||||
async def shutdown(self, *, grace: float = SHUTDOWN_GRACE_S) -> None:
|
||||
"""关停:停掉进行中、排队等着的任务(标中断),最多等 grace 秒收尾。"""
|
||||
runs = [run for job_id in list(self._runs) if (run := self._alive(job_id)) is not None]
|
||||
for run in runs:
|
||||
if run.stop_reason is None:
|
||||
run.stop_reason = "shutdown"
|
||||
run.task.cancel() # type: ignore[union-attr]
|
||||
if runs:
|
||||
await asyncio.wait([run.task for run in runs], timeout=grace) # type: ignore[misc]
|
||||
logger.info("已停止 %d 个%s", len(runs), self.label)
|
||||
|
||||
|
||||
__all__ = ["ACTIVE", "FINISHED", "JobFailed", "JobManager", "JobRun", "LaneBusy", "LaneTicket", "OPEN", "RETRYABLE",
|
||||
"SHUTDOWN_GRACE_S", "STOP_GRACE_S", "SerialWriter", "SourceLanes", "TASK_DONE", "TASK_STATUSES", "cancel_queued",
|
||||
"claim", "close_job", "interrupt_on_startup", "lane_kinds", "lane_label", "lanes", "register_lane_kind"]
|
||||
@@ -0,0 +1,97 @@
|
||||
"""对外访问地址:别人打开 AgentLab 用的那个地址(https://agentlab.example.com),全站只有这一个说法(04 细案 B4)。
|
||||
|
||||
用到它的地方:通知和定时任务 Webhook 里的链接、单点登录的回调地址、非 GET 请求的来源核对(auth/middleware.py);
|
||||
之后的分享链接、接口说明也从这里取。服务端不知道自己在反向代理后面叫什么,只能由人告诉它。
|
||||
|
||||
取值(public_base_url):
|
||||
- 环境变量 AGENTLAB_PUBLIC_URL 优先:部署时定死,设置页上显示「由环境变量指定」、不能改(同登录方式的 env_locked);
|
||||
- 没设时取设置里 notify 一组的 public_base_url:沿用这个键,界面上叫「对外访问地址」,在「设置 → 通用」里改;
|
||||
- 都没有是 None:通知里不附链接,单点登录的回调和来源核对按请求自己的 Host 认。
|
||||
|
||||
以前两处各管各的:环境变量只管回调和来源核对、设置只管通知里的链接——只配了环境变量的部署,通知里不带链接;只在设置里
|
||||
填了的,回调又按请求的 Host 拼。违背「一种说法」。
|
||||
|
||||
设置的值缓存在进程里:中间件按请求核对来源,不能每次查库。启动时读一次(main._platform_startup),改设置的接口写完库就
|
||||
remember;手里本来就有会话的异步路径(通知、定时任务 Webhook)用 refresh 取,顺手按库里的值对齐缓存。和登录设置的缓存一样
|
||||
以单进程部署为前提。
|
||||
|
||||
这个模块不在顶层导入 core/config:config 的校验器要用这里的 url_problem,顶层互相导入会在启动时绕成环。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
#: 设置里存它的那一组、那一项(api/settings.py 的 NOTIFY_GROUP 就是这一组)
|
||||
SETTING_GROUP = "notify"
|
||||
SETTING_KEY = "public_base_url"
|
||||
#: 环境变量的名字(core/config.Settings.public_url 按 AGENTLAB_ 前缀读它)
|
||||
ENV_NAME = "AGENTLAB_PUBLIC_URL"
|
||||
|
||||
_stored: str | None = None
|
||||
|
||||
|
||||
def url_problem(value: str) -> str | None:
|
||||
"""地址的形状有什么问题:None 是没问题;"form" 不是 http:// 或 https:// 开头的完整地址;"suffix" 带了 ? 或 #
|
||||
(链接要接在地址后面)。设置接口按它出中文的说法,环境变量按它在启动时报错。可以带路径(反向代理挂在子路径下)。"""
|
||||
text = value.strip()
|
||||
try:
|
||||
parts = urlsplit(text)
|
||||
except ValueError:
|
||||
return "form"
|
||||
if parts.scheme not in ("http", "https") or not parts.hostname or " " in text:
|
||||
return "form"
|
||||
if parts.query or parts.fragment:
|
||||
return "suffix"
|
||||
return None
|
||||
|
||||
|
||||
def normalize(value: Any) -> str | None:
|
||||
"""存着的(或刚填的)值 → 去掉首尾空白和末尾的 /;空的、形状不对的当没填(None)。末尾带 / 的地址拼链接会多出一个斜杠。"""
|
||||
if not isinstance(value, str) or not value.strip() or url_problem(value):
|
||||
return None
|
||||
return value.strip().rstrip("/")
|
||||
|
||||
|
||||
def env_url() -> str | None:
|
||||
"""环境变量给的地址;没设为 None。形状在启动时由 config 的校验器查过、末尾的 / 去掉了。"""
|
||||
from app.core.config import settings
|
||||
|
||||
return settings.public_url or None
|
||||
|
||||
|
||||
def env_locked() -> bool:
|
||||
"""环境变量设了:设置里的那一项不生效、不能改。"""
|
||||
return env_url() is not None
|
||||
|
||||
|
||||
def public_base_url() -> str | None:
|
||||
"""对外访问地址:环境变量优先,其次设置(进程内缓存),都没有是 None。不查库,中间件里也能调。"""
|
||||
return env_url() or _stored
|
||||
|
||||
|
||||
def remember(value: Any) -> None:
|
||||
"""改设置的接口写完库之后调:缓存换成新存的值(形状不对、空的当没填)。"""
|
||||
global _stored
|
||||
_stored = normalize(value)
|
||||
|
||||
|
||||
async def refresh(session: Any = None) -> str | None:
|
||||
"""按库里存的设置对齐缓存,交回对外访问地址。启动时、手里有会话的异步路径用。"""
|
||||
from app.db.base import SessionLocal
|
||||
from app.db.models import Setting
|
||||
|
||||
if session is None:
|
||||
async with SessionLocal() as own:
|
||||
row = await own.get(Setting, SETTING_GROUP)
|
||||
else:
|
||||
row = await session.get(Setting, SETTING_GROUP)
|
||||
value = row.value if row is not None else None
|
||||
remember(value.get(SETTING_KEY) if isinstance(value, dict) else None)
|
||||
return public_base_url()
|
||||
|
||||
|
||||
def origin_of(url: str) -> str:
|
||||
"""地址里的「协议://主机[:端口]」:来源核对按它比。"""
|
||||
parts = urlsplit(url)
|
||||
return f"{parts.scheme}://{parts.netloc}"
|
||||
+112
-15
@@ -420,6 +420,18 @@ async def read_catalog(session: AsyncSession, source_id: str) -> dict[str, Catal
|
||||
return {row.table_name: _entry(row) for row in rows}
|
||||
|
||||
|
||||
async def read_catalogs(session: AsyncSession, source_ids: Iterable[str]) -> dict[str, dict[str, CatalogEntry]]:
|
||||
"""几个数据源的目录一次查询读出:{源 id: {表名: CatalogEntry}},没有目录的源不出现。给模型、给自查读目录的地方经
|
||||
data/catalog_view.py 调它,不直接查 catalog_notes。"""
|
||||
ids = sorted({i for i in source_ids if i})
|
||||
if not ids:
|
||||
return {}
|
||||
out: dict[str, dict[str, CatalogEntry]] = {}
|
||||
for row in (await session.execute(select(CatalogNote).where(CatalogNote.source_id.in_(ids)))).scalars():
|
||||
out.setdefault(row.source_id, {})[row.table_name] = _entry(row)
|
||||
return out
|
||||
|
||||
|
||||
async def write_entry(session: AsyncSession, source_id: str, table: str, notes: dict[str, Any], *,
|
||||
if_version: int, actor: str | None, commit: bool = True) -> CatalogEntry:
|
||||
"""整份写入一张表的目录(乐观锁),提交事务,返回写入后的目录。
|
||||
@@ -567,18 +579,48 @@ def _undated(note: str) -> str:
|
||||
return _PROFILE_NOTE_DAY.sub("数据剖析", note, count=1)
|
||||
|
||||
|
||||
#: 剖析备注改过的说法:(升级前的写法, 现在的写法)。大表看的是按存储顺序的前 N 行(多半是最老的数据)、关系取的是
|
||||
#: 数据库先给出的那几个键值,都不是随机抽样,说法改成照实的(data/catalog_profile.py 的 _relation_note、码值的
|
||||
#: 「统计范围」、FILTER_SAMPLED)。只换这几处固定的句式,换完和现在的写法一字不差才算同一句
|
||||
_LEGACY_PROFILE_WORDING = (
|
||||
(re.compile(r"抽样覆盖率 (?=\S+?,可能不是这条关系)"), "覆盖率 "),
|
||||
(re.compile(r"子表抽样 (?=\d+ 个不同键值)"), "子表取了 "),
|
||||
(re.compile(r"(只看前 (?=\d+ 行))"), "(按存储顺序只看前 "),
|
||||
(re.compile(r"按前 (?=[\d,]+ 行统计)"), "按存储顺序取前 "),
|
||||
)
|
||||
#: 起草时核对有效行过滤写进备注的那句(catalog_profile.FILTER_SAMPLED / FILTER_SAMPLED_EDGE):接在模型给的依据后面,
|
||||
#: 整段不以「数据剖析:」开头,只换这一句
|
||||
_LEGACY_FILTER_WORDING = re.compile(r"剖析核对:按前 (?=[\d,]+ 行统计)")
|
||||
|
||||
|
||||
def _current_wording(note: str) -> str:
|
||||
"""剖析备注里升级前的说法换成现在的写法(_LEGACY_PROFILE_WORDING)。只用来比较,不改写目录里存着的备注。"""
|
||||
for pattern, now in _LEGACY_PROFILE_WORDING:
|
||||
note = pattern.sub(now, note)
|
||||
return note
|
||||
|
||||
|
||||
def same_profile_note(a: Any, b: Any) -> bool:
|
||||
"""两段备注是否相同;数据剖析写的备注只差开头的日期(升级前每次都写当天的日期),或者一段带日期、一段是不带
|
||||
日期的新写法,也算相同。
|
||||
日期的新写法,或者只差改过的几处说法(_LEGACY_PROFILE_WORDING:「抽样覆盖率」→「覆盖率」之类),也算相同。
|
||||
|
||||
隔天重跑、数据没变,结论一字不差、只有日期变了:要是这也算「变了」,目录就升一个版本,发布时记下的目录版本
|
||||
对不上,之后每次正式运行都提醒「目录有变化」。升级后第一次重新剖析也是一样:结论没变,只是新写法不带日期。
|
||||
对不上,之后每次正式运行都提醒「目录有变化」。升级后第一次重新剖析也是一样:结论没变,只是新写法不带日期、
|
||||
措辞改了。算相同时保留旧的那段(目录不动),旧说法要等结论真的变了、重写备注时才换掉。
|
||||
|
||||
起草写的备注(有效行过滤)里接着的剖析核对那句同理,只认那一句的旧说法(_LEGACY_FILTER_WORDING)。
|
||||
"""
|
||||
if a == b:
|
||||
return True
|
||||
if not (is_profile_note(a) and is_profile_note(b)):
|
||||
if not (isinstance(a, str) and isinstance(b, str)):
|
||||
return False
|
||||
return _undated(a) == _undated(b)
|
||||
profile_a, profile_b = is_profile_note(a), is_profile_note(b)
|
||||
if profile_a and profile_b:
|
||||
return _current_wording(_undated(a)) == _current_wording(_undated(b))
|
||||
if profile_a or profile_b:
|
||||
return False
|
||||
return (_LEGACY_FILTER_WORDING.sub("剖析核对:按存储顺序取前 ", a)
|
||||
== _LEGACY_FILTER_WORDING.sub("剖析核对:按存储顺序取前 ", b))
|
||||
|
||||
|
||||
def _same_core(a: Mapping[str, Any], b: Mapping[str, Any]) -> bool:
|
||||
@@ -592,6 +634,14 @@ def visible_notes(notes: Mapping[str, Any] | None) -> dict[str, Any]:
|
||||
return _assemble({k: v for k, v in _slots(notes).items() if v.get("status") != "rejected"})
|
||||
|
||||
|
||||
def restrict_notes(notes: Mapping[str, Any] | None, statuses: Iterable[str]) -> dict[str, Any]:
|
||||
"""只留状态在 statuses 里的项(表级项、列级项、关系都按项算);驳回的项原样留着——给人、给模型看的照旧经
|
||||
visible_notes 去掉,冻结快照和 SQL 检查要靠驳回的关系编号认出不该按外键、命名现推回来的关系(frozen_catalog)。
|
||||
目录视图「只用已核实、已确认的项」用它(data/catalog_view.py)。"""
|
||||
keep = set(statuses) | {"rejected"}
|
||||
return _assemble({k: v for k, v in _slots(notes).items() if v.get("status") in keep})
|
||||
|
||||
|
||||
def status_counts(notes: Mapping[str, Any] | None) -> dict[str, int]:
|
||||
"""各状态的项数(表级项、列级项、关系都算),四种状态都有键。"""
|
||||
counts = dict.fromkeys(ITEM_STATUSES, 0)
|
||||
@@ -1358,14 +1408,17 @@ _DATE_WORDS = frozenset({"date", "time", "day", "dt", "ts", "timestamp", "dateti
|
||||
_TEXT_TYPES = ("CHAR", "TEXT", "STRING", "CLOB")
|
||||
|
||||
|
||||
def is_date_column(name: str, type_name: str | None) -> bool:
|
||||
def is_date_column(name: str, type_name: str | None, *, enum_values: Any = None) -> bool:
|
||||
"""日期类列:类型是日期、时间戳(DATE、DATETIME、TIMESTAMP……);或者类型是文字(或者不知道类型)、名字像日期
|
||||
(SQLite 的日期常存成文字)。只有时刻的 TIME、整数存的时间戳不算:取值不是日期。
|
||||
(SQLite 的日期常存成文字)。只有时刻的 TIME、整数存的时间戳不算:取值不是日期。有取值列表的(MySQL 的 ENUM /
|
||||
SET、探查记下 enum_values 的 PostgreSQL 枚举)不算:PG 的枚举编译出来是类型名本身,update_reason 里也有「DATE」。
|
||||
|
||||
数据剖析(表里只有一个日期类列时提议它作业务日期)和 SQL 检查(按业务日期以外的时间列统计)共用这一个判断。
|
||||
以前两边各写一套:剖析认的「唯一日期列」,SQL 检查可能认为还有别的时间列,反过来也一样,两边对不上。
|
||||
"""
|
||||
t = (type_name or "").upper()
|
||||
t = _type_words(type_name)
|
||||
if enum_values or _MYSQL_ENUM.match(t) or "INTERVAL" in t:
|
||||
return False
|
||||
if "DATE" in t or "TIMESTAMP" in t:
|
||||
return True
|
||||
if t and not any(k in t for k in _TEXT_TYPES):
|
||||
@@ -1400,11 +1453,41 @@ def _name_index(tables: Mapping[str, Any]) -> dict[tuple[str, str], set[str]]:
|
||||
return index
|
||||
|
||||
|
||||
#: 类型写法里带引号的部分(ENUM('int','date') 的取值):认类型时去掉,取值里的词不能让一列变成数字或时间
|
||||
_QUOTED_IN_TYPE = re.compile(r"'(?:[^']|'')*'")
|
||||
|
||||
|
||||
def _type_words(type_name: str | None) -> str:
|
||||
"""认类型用的写法:大写、去掉引号里的取值。探查存的是按方言编译的完整写法(ENUM('paid','refund')、
|
||||
VARCHAR2(20 CHAR)、TIMESTAMP WITH TIME ZONE),不再是 SQLAlchemy 的通用写法。"""
|
||||
return _QUOTED_IN_TYPE.sub("''", (type_name or "").upper()).strip()
|
||||
|
||||
|
||||
#: MySQL 的 ENUM / SET 编译出来的写法:ENUM('a','b')、SET('x','y'),只认带括号的。PostgreSQL 的枚举编译出来是类型名本身,
|
||||
#: setup_status、enum_kind 这种以 set、enum 开头的名字不能按前缀认成 MySQL 的 SET / ENUM——PG 的枚举一律靠探查记下的
|
||||
#: enum_values 认(column_family)
|
||||
_MYSQL_ENUM = re.compile(r"^(ENUM|SET)\s*\(")
|
||||
|
||||
|
||||
def _mysql_enum_kind(type_name: str | None) -> str | None:
|
||||
"""类型写法是 MySQL 的 ENUM(…) / SET(…) 时返回 "ENUM" / "SET",否则 None。"""
|
||||
m = _MYSQL_ENUM.match(_type_words(type_name))
|
||||
return m.group(1) if m else None
|
||||
|
||||
|
||||
def _type_family(type_name: str | None) -> str | None:
|
||||
"""列类型的大类:数字、文字、时间;认不出来返回 None(不据此排除)。"""
|
||||
t = (type_name or "").upper()
|
||||
"""列类型的大类:数字、文字、时间;认不出来返回 None(不据此排除)。
|
||||
|
||||
认的是按方言编译的完整写法:MySQL 的 ENUM / SET 是文字(取值列表里的词不算),Oracle 的 INTERVAL DAY TO SECOND
|
||||
是时长、不算数字(以前按子串「INT」认成了数字)。PostgreSQL 的枚举编译出来是类型名本身,只看类型认不出来,
|
||||
要看列上的取值列表(column_family)。"""
|
||||
t = _type_words(type_name)
|
||||
if not t:
|
||||
return None
|
||||
if _MYSQL_ENUM.match(t):
|
||||
return "text"
|
||||
if "INTERVAL" in t:
|
||||
return None
|
||||
if any(k in t for k in ("CHAR", "TEXT", "CLOB", "STRING", "UUID")):
|
||||
return "text"
|
||||
if any(k in t for k in ("INT", "NUMBER", "NUMERIC", "DECIMAL", "SERIAL", "REAL", "FLOAT", "DOUBLE")):
|
||||
@@ -1414,6 +1497,16 @@ def _type_family(type_name: str | None) -> str | None:
|
||||
return None
|
||||
|
||||
|
||||
def column_family(col: Mapping[str, Any] | None) -> str | None:
|
||||
"""表结构里一列的类型大类:有取值列表(探查记下的 enum_values:MySQL 的 ENUM / SET、PostgreSQL 的枚举)的是
|
||||
文字,其余按类型写法认(_type_family)。"""
|
||||
if not isinstance(col, Mapping):
|
||||
return None
|
||||
if col.get("enum_values"):
|
||||
return "text"
|
||||
return _type_family(col.get("type"))
|
||||
|
||||
|
||||
def _id_stem(column: str) -> str | None:
|
||||
m = _SNAKE_ID.match(column) or _CAMEL_ID.match(column)
|
||||
return m.group("stem") if m else None
|
||||
@@ -1469,8 +1562,8 @@ def infer_name_relations(schema_cache: Mapping[str, Any] | None, table: str, *,
|
||||
pk = list(target_meta.get("primary_key") or [])
|
||||
if len(pk) != 1:
|
||||
continue
|
||||
pk_type = next((c.get("type") for c in target_meta.get("columns") or [] if c.get("name") == pk[0]), None)
|
||||
mine, theirs = _type_family(col.get("type")), _type_family(pk_type)
|
||||
pk_col = next((c for c in target_meta.get("columns") or [] if c.get("name") == pk[0]), None)
|
||||
mine, theirs = column_family(col), column_family(pk_col)
|
||||
if mine and theirs and mine != theirs:
|
||||
continue
|
||||
out.append(_relation(table, [name], target, pk, "name", cardinality=_cardinality(meta, [name])))
|
||||
@@ -2408,15 +2501,18 @@ async def query_snapshot_tables(session: AsyncSession, source: Any) -> list[tupl
|
||||
|
||||
使用次数(table_usage)和「这张表被哪些运行用过」(data/catalog_review.py)共用这一份取法:数据源查询工具每次成功
|
||||
查询都存一份查询快照工件(tools/datasource.py),工件引用表 artifacts 里按 (内容哈希, 运行) 各记一行;只收属于某次运行
|
||||
(runs 里有这次运行)、属于这个源的行,工具库里的试用(不属于任何运行)不算。
|
||||
(runs 里有这次运行)、属于这个源的行,工具库里的试用(不属于任何运行)不算;系统派生的运行(评测、口径对账、召回,
|
||||
engine/origin.counted_runs)也不算——一跑评测,常用表的排序、起草的默认范围、审阅队列的权重就跟着被评测题刷高了。
|
||||
|
||||
表名在查询当时就记进了引用的 meta(query_snapshot_meta),这里只读 artifacts 一张表;升级前的老引用没有 meta,退回读
|
||||
工件内容里的源名和 SQL,读过的按工件 id 缓存(内容永不改变)。SQL 里写的表名去掉 schema 前缀后按 name_key(只把
|
||||
ASCII 字母转小写,和 SQLite 比较标识符的口径一致)对到表结构上,对不上的不计。
|
||||
"""
|
||||
from app.engine.origin import counted_runs
|
||||
|
||||
rows = (await session.execute(
|
||||
select(Artifact.id, Artifact.meta, Artifact.run_id).join(Run, Run.id == Artifact.run_id)
|
||||
.where(Artifact.kind == QUERY_SNAPSHOT_KIND)
|
||||
.where(Artifact.kind == QUERY_SNAPSHOT_KIND, counted_runs())
|
||||
)).all()
|
||||
tables = (getattr(source, "schema_cache", None) or {}).get("tables") or {}
|
||||
lookup: dict[str, str] = {}
|
||||
@@ -2505,8 +2601,9 @@ __all__ = [
|
||||
"fk_relations", "frozen_catalog", "infer_name_relations", "initial_status", "is_date_column", "is_profile_note",
|
||||
"join_paths",
|
||||
"make_item", "merge_notes", "name_words", "now_iso", "parse_path", "plan_patch", "profile_relation_holds",
|
||||
"query_snapshot_meta", "read_catalog", "read_entry", "relation_graph", "relation_id", "render_table_index",
|
||||
"render_table_notes", "resolve_draft_model", "resolve_table_name", "review_entry", "review_item", "same_notes",
|
||||
"query_snapshot_meta", "read_catalog", "read_catalogs", "read_entry", "relation_graph", "relation_id", "render_table_index",
|
||||
"render_table_notes", "resolve_draft_model", "resolve_table_name", "restrict_notes", "review_entry", "review_item",
|
||||
"same_notes",
|
||||
"same_profile_note", "save_human_edit", "save_patch", "status_counts", "structure_relation_ids",
|
||||
"system_notes_source", "table_usage", "unreviewed_status", "validate_notes", "visible_notes", "write_entry",
|
||||
]
|
||||
|
||||
@@ -1,400 +1,13 @@
|
||||
"""数据目录后台任务的共用运行时:起草(data/catalog_jobs.py)和剖析(data/catalog_profile_jobs.py)各自建表、各自的步骤,
|
||||
生命周期是同一套。
|
||||
"""后台任务运行时的老路径:已上移到 app/core/job_runtime.py(docs/design/next-round/00-总方案.md X5)。
|
||||
|
||||
**为什么抽出来。** 剖析以前是一次同步长请求,和起草改成后台任务之前是同一类问题:关掉页面结果就没了、不能停、不留痕。
|
||||
两边要的是同一样东西——进程内跑、能停、重启标中断、写库不被停止打断、同源排队——各写一份,迟早一边修了另一边没修。
|
||||
共用的是生命周期代码,不是表:两种任务的进度完全不同(起草按段、块、模型用量,剖析按检查项、查询),合成一张
|
||||
「kind + JSON」的通用任务表要丢掉起草那些强类型列,还要迁移刚稳定的起草数据。
|
||||
|
||||
- **状态词**(ACTIVE、FINISHED、TASK_STATUSES、TASK_DONE、RETRYABLE、OPEN):任务 queued → running → succeeded / partial
|
||||
/ failed / cancelled / interrupted;逐表 pending → running → ok / partial / failed / skipped / cancelled。两种任务、
|
||||
接口、界面都用这一套词。
|
||||
- **SerialWriter**:同一个任务的库写入一条一条来,shield 起来——停止任务时不会打断写到一半的库操作;收尾前 settle 等它们
|
||||
写完。
|
||||
- **JobRun / JobManager**:进程内一个 asyncio 任务一件事(照 app/scheduler 的写法):start / stop(grace) / wait / running
|
||||
/ shutdown。JobRun.run 把「进队、等轮到自己、执行、按停止的原因收尾、出队」写一次,各种任务只写执行和收尾的那几步。
|
||||
- **SourceLanes**:每个数据源一条先来先到的队。起草、剖析都打业务库或模型、都写同一个源的目录:同源串行(先建的先跑),
|
||||
不同源并行。任务阻塞地占:建出来是 queued,轮到了才 running。同步起草、导入目录这类同步写操作不阻塞地试占:占不到
|
||||
说明源上正有别的事,接口回 409;占到了,执行期间新建的任务排在它后面。
|
||||
- **close_job / interrupt_on_startup**:收尾(还没结束的表记成某个状态,任务记成某个状态),和服务启动时的整理:排队、
|
||||
进行中的标 interrupted,没结束的表记 cancelled。**不自动续跑**:复制出来的数据目录(检查脚本、验收副本)启动时不能
|
||||
自己去调模型、查业务库,界面让人点「重试」接着做。
|
||||
|
||||
进程内登记、进程内的队:服务端是单进程部署,多进程部署时要换成库里的锁。
|
||||
这里把那个模块原样放在老名字下(sys.modules 指向同一个模块对象):起草、剖析和别处的 `from app.data import
|
||||
catalog_job_runtime as runtime`、`from app.data.catalog_job_runtime import …` 拿到的是同一批名字、同一条队(lanes),
|
||||
测试里改它的属性也就是改那个模块。新代码从 app.core.job_runtime 导入。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import contextlib
|
||||
import logging
|
||||
from collections.abc import AsyncIterator, Awaitable, Callable, Iterator
|
||||
from typing import Any
|
||||
import sys
|
||||
|
||||
from sqlalchemy import select, update
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from app.core import job_runtime
|
||||
|
||||
from app.db.base import SessionLocal, utcnow
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
#: 任务还没结束的状态(同一个源同一种任务只能有一个)、结束了的状态
|
||||
ACTIVE = ("queued", "running")
|
||||
FINISHED = ("succeeded", "partial", "failed", "cancelled", "interrupted")
|
||||
#: 逐表的状态,其中算「已结束」的、重试时要带上的、还没结束的
|
||||
TASK_STATUSES = ("pending", "running", "ok", "partial", "failed", "skipped", "cancelled")
|
||||
TASK_DONE = ("ok", "partial", "failed", "skipped", "cancelled")
|
||||
RETRYABLE = ("failed", "partial", "cancelled")
|
||||
OPEN = ("pending", "running")
|
||||
|
||||
#: 停止进行中的任务时最多等它收尾多久(秒);关停服务时另给(shutdown 的 grace)
|
||||
STOP_GRACE_S = 10.0
|
||||
SHUTDOWN_GRACE_S = 3.0
|
||||
|
||||
|
||||
class JobFailed(Exception):
|
||||
"""任务整体出错(数据源没了、快照没了、模型用不了……):message 就是任务的 error。"""
|
||||
|
||||
|
||||
# ==========================================================================
|
||||
# 库写入
|
||||
# ==========================================================================
|
||||
|
||||
|
||||
class SerialWriter:
|
||||
"""一个任务的库写入:一条一条来,每条开一个会话;shield 起来,停止任务时不会打断写到一半的库操作。"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._writes: set[asyncio.Task[Any]] = set()
|
||||
self._lock = asyncio.Lock()
|
||||
|
||||
async def __call__(self, fn: Callable[..., Awaitable[Any]], *args: Any) -> Any:
|
||||
async def go() -> Any:
|
||||
async with self._lock:
|
||||
async with SessionLocal() as session:
|
||||
return await fn(session, *args)
|
||||
|
||||
task = asyncio.ensure_future(go())
|
||||
self._writes.add(task)
|
||||
task.add_done_callback(self._writes.discard)
|
||||
return await asyncio.shield(task)
|
||||
|
||||
async def settle(self) -> None:
|
||||
"""等手上的库写入都写完(停止、出错收尾之前)。"""
|
||||
while self._writes:
|
||||
await asyncio.gather(*list(self._writes), return_exceptions=True)
|
||||
|
||||
|
||||
# ==========================================================================
|
||||
# 每个源一条队
|
||||
# ==========================================================================
|
||||
|
||||
|
||||
class LaneTicket:
|
||||
"""在一个源的队里的一个位置。kind 是占着它的是什么:draft 起草任务、profile 剖析任务、sync_draft 同步起草、
|
||||
import 导入目录;job_id 是任务的(同步写操作没有)。轮到它时 granted 置位。"""
|
||||
|
||||
def __init__(self, lanes: SourceLanes, source_id: str, kind: str, job_id: str | None,
|
||||
loop: asyncio.AbstractEventLoop) -> None:
|
||||
self._lanes = lanes
|
||||
self.source_id = source_id
|
||||
self.kind = kind
|
||||
self.job_id = job_id
|
||||
self.loop = loop
|
||||
self.granted = asyncio.Event()
|
||||
self.released = False
|
||||
|
||||
async def wait(self) -> None:
|
||||
"""等轮到自己。"""
|
||||
await self.granted.wait()
|
||||
|
||||
def release(self) -> None:
|
||||
"""出队(轮到了、没轮到都行,重复调用无害):排在后面的第一个接着开始。"""
|
||||
self._lanes._release(self)
|
||||
|
||||
|
||||
class LaneBusy(Exception):
|
||||
"""不阻塞地试占时这个源的队里已经有人:holder 是排在最前面(正在做)的那一个。"""
|
||||
|
||||
def __init__(self, holder: LaneTicket) -> None:
|
||||
super().__init__(holder.kind)
|
||||
self.holder = holder
|
||||
|
||||
|
||||
class SourceLanes:
|
||||
"""每个数据源一条先来先到的队。"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._queues: dict[str, list[LaneTicket]] = {}
|
||||
|
||||
def _queue(self, source_id: str) -> list[LaneTicket]:
|
||||
"""这个源的队。别的事件循环留下的(测试每个用例一个循环)去掉:那个循环里没人会再出队了。"""
|
||||
queue = self._queues.get(source_id) or []
|
||||
loop = asyncio.get_running_loop()
|
||||
alive = [t for t in queue if t.loop is loop and not t.released]
|
||||
if alive:
|
||||
self._queues[source_id] = alive
|
||||
alive[0].granted.set()
|
||||
else:
|
||||
self._queues.pop(source_id, None)
|
||||
return alive
|
||||
|
||||
def enter(self, source_id: str, kind: str, job_id: str | None = None) -> LaneTicket:
|
||||
"""排进队尾,马上返回(不等)。排在最前面的立刻 granted;调用方 await ticket.wait() 等轮到自己。"""
|
||||
queue = self._queue(source_id)
|
||||
ticket = LaneTicket(self, source_id, kind, job_id, asyncio.get_running_loop())
|
||||
queue.append(ticket)
|
||||
self._queues[source_id] = queue
|
||||
if len(queue) == 1:
|
||||
ticket.granted.set()
|
||||
return ticket
|
||||
|
||||
def try_enter(self, source_id: str, kind: str) -> LaneTicket:
|
||||
"""队空着才占(马上就轮到);有人在队里抛 LaneBusy。"""
|
||||
queue = self._queue(source_id)
|
||||
if queue:
|
||||
raise LaneBusy(queue[0])
|
||||
return self.enter(source_id, kind)
|
||||
|
||||
def holders(self, source_id: str) -> list[LaneTicket]:
|
||||
"""队里的各位,排在最前面(正在做)的在前。"""
|
||||
return list(self._queue(source_id))
|
||||
|
||||
def ahead(self, source_id: str, job_id: str) -> list[str] | None:
|
||||
"""排在这个任务前面的各位是什么(kind,正在做的在前);这个任务不在队里(刚建、后台还没排进来)返回 None。"""
|
||||
queue = self._queue(source_id)
|
||||
for i, ticket in enumerate(queue):
|
||||
if ticket.job_id == job_id:
|
||||
return [t.kind for t in queue[:i]]
|
||||
return None
|
||||
|
||||
def _release(self, ticket: LaneTicket) -> None:
|
||||
if ticket.released:
|
||||
return
|
||||
ticket.released = True
|
||||
queue = self._queues.get(ticket.source_id)
|
||||
if not queue or ticket not in queue:
|
||||
return
|
||||
head = queue[0] is ticket
|
||||
queue.remove(ticket)
|
||||
if not queue:
|
||||
self._queues.pop(ticket.source_id, None)
|
||||
elif head:
|
||||
queue[0].granted.set()
|
||||
|
||||
@contextlib.asynccontextmanager
|
||||
async def hold(self, source_id: str, kind: str, job_id: str | None = None) -> AsyncIterator[LaneTicket]:
|
||||
"""阻塞地占:排队,轮到了再进去,出来时出队。"""
|
||||
ticket = self.enter(source_id, kind, job_id)
|
||||
try:
|
||||
await ticket.wait()
|
||||
yield ticket
|
||||
finally:
|
||||
ticket.release()
|
||||
|
||||
@contextlib.contextmanager
|
||||
def try_hold(self, source_id: str, kind: str) -> Iterator[LaneTicket]:
|
||||
"""不阻塞地试占(同步起草、导入目录):占不到抛 LaneBusy;占到了,执行期间新建的任务排在后面。"""
|
||||
ticket = self.try_enter(source_id, kind)
|
||||
try:
|
||||
yield ticket
|
||||
finally:
|
||||
ticket.release()
|
||||
|
||||
|
||||
lanes = SourceLanes()
|
||||
|
||||
|
||||
# ==========================================================================
|
||||
# 收尾、启动时的整理
|
||||
# ==========================================================================
|
||||
|
||||
|
||||
async def close_job(session: AsyncSession, job_model: Any, task_model: Any, job_id: str, job_status: str,
|
||||
job_error: str | None, task_status: str, task_error: str | None) -> None:
|
||||
"""收尾:还没结束的表记 task_status(原因 task_error),任务(还没结束的话)记 job_status。提交。"""
|
||||
now = utcnow()
|
||||
await session.execute(
|
||||
update(task_model).where(task_model.job_id == job_id, task_model.status.in_(OPEN))
|
||||
.values(status=task_status, error=task_error, finished_at=now).execution_options(synchronize_session=False))
|
||||
await session.execute(
|
||||
update(job_model).where(job_model.id == job_id, job_model.status.in_(ACTIVE))
|
||||
.values(status=job_status, error=job_error, finished_at=now).execution_options(synchronize_session=False))
|
||||
await session.commit()
|
||||
|
||||
|
||||
async def cancel_queued(session: AsyncSession, job_model: Any, task_model: Any, job_id: str) -> bool:
|
||||
"""停止一个排队中的任务:任务和没结束的表直接记 cancelled。已经被后台认领(开始了)返回 False,交给调用方按进行中
|
||||
的停。"""
|
||||
now = utcnow()
|
||||
claimed = await session.execute(
|
||||
update(job_model).where(job_model.id == job_id, job_model.status == "queued")
|
||||
.values(status="cancelled", finished_at=now).execution_options(synchronize_session=False))
|
||||
if claimed.rowcount != 1:
|
||||
await session.rollback()
|
||||
return False
|
||||
await session.execute(
|
||||
update(task_model).where(task_model.job_id == job_id, task_model.status.in_(OPEN))
|
||||
.values(status="cancelled", finished_at=now).execution_options(synchronize_session=False))
|
||||
await session.commit()
|
||||
return True
|
||||
|
||||
|
||||
async def claim(session: AsyncSession, job_model: Any, job_id: str) -> bool:
|
||||
"""认领:排队中 → 进行中。开始之前就被停止了返回 False。"""
|
||||
result = await session.execute(
|
||||
update(job_model).where(job_model.id == job_id, job_model.status == "queued")
|
||||
.values(status="running", started_at=utcnow()).execution_options(synchronize_session=False))
|
||||
await session.commit()
|
||||
return result.rowcount == 1
|
||||
|
||||
|
||||
async def interrupt_on_startup(job_model: Any, close: Callable[[AsyncSession, str], Awaitable[None]], *,
|
||||
running: Callable[[str], bool]) -> int:
|
||||
"""服务启动时:排队、进行中、而且不在这个进程里跑的任务逐个 close(各自记成 interrupted、没结束的表记 cancelled)。
|
||||
不自动续跑。返回整理了几个。"""
|
||||
async with SessionLocal() as session:
|
||||
ids = [job_id for job_id in (await session.execute(
|
||||
select(job_model.id).where(job_model.status.in_(ACTIVE)))).scalars() if not running(job_id)]
|
||||
for job_id in ids:
|
||||
await close(session, job_id)
|
||||
return len(ids)
|
||||
|
||||
|
||||
# ==========================================================================
|
||||
# 在后台执行
|
||||
# ==========================================================================
|
||||
|
||||
|
||||
class JobRun:
|
||||
"""一个任务在后台的一次执行。子类给 kind(进队时的身份)、model(任务表,进队前读源和状态),写 _run(执行)和
|
||||
_stopped / _failed / _crashed(停止、整体出错、意外出错时的收尾)。"""
|
||||
|
||||
kind = ""
|
||||
model: Any = None
|
||||
|
||||
def __init__(self, job_id: str) -> None:
|
||||
self.job_id = job_id
|
||||
self.task: asyncio.Task[None] | None = None
|
||||
#: 为什么停:user 使用者点了停止,shutdown 服务关停(标中断)
|
||||
self.stop_reason: str | None = None
|
||||
self.writer = SerialWriter()
|
||||
self.ticket: LaneTicket | None = None
|
||||
|
||||
async def _write(self, fn: Callable[..., Awaitable[Any]], *args: Any) -> Any:
|
||||
"""一条库写入(SerialWriter:一条一条来,停止时不被打断)。"""
|
||||
return await self.writer(fn, *args)
|
||||
|
||||
async def _settle(self) -> None:
|
||||
await self.writer.settle()
|
||||
|
||||
async def run(self) -> None:
|
||||
try:
|
||||
try:
|
||||
await self._queue()
|
||||
await self._run()
|
||||
except asyncio.CancelledError:
|
||||
current = asyncio.current_task()
|
||||
if current is not None:
|
||||
current.uncancel()
|
||||
await self._stopped()
|
||||
except JobFailed as e:
|
||||
await self._failed(str(e))
|
||||
except Exception as e: # noqa: BLE001 - 任务整体出错:照实记下,不留在「进行中」
|
||||
await self._crashed(e)
|
||||
finally:
|
||||
# 收尾写完才出队:排在后面的任务开始时,这个任务已经记成结束了
|
||||
if self.ticket is not None:
|
||||
self.ticket.release()
|
||||
|
||||
async def _queue(self) -> None:
|
||||
"""进这个源的队,等轮到自己。任务不在了、已经不是排队中(开始之前就被停止了)的不进队:_run 认领不到,直接结束。"""
|
||||
async with SessionLocal() as session:
|
||||
job = await session.get(self.model, self.job_id)
|
||||
source_id = job.source_id if job is not None and job.status == "queued" else None
|
||||
if source_id is None:
|
||||
return
|
||||
self.ticket = lanes.enter(source_id, self.kind, self.job_id)
|
||||
await self.ticket.wait()
|
||||
|
||||
async def _run(self) -> None:
|
||||
raise NotImplementedError
|
||||
|
||||
async def _stopped(self) -> None:
|
||||
raise NotImplementedError
|
||||
|
||||
async def _failed(self, reason: str) -> None:
|
||||
raise NotImplementedError
|
||||
|
||||
async def _crashed(self, e: Exception) -> None:
|
||||
raise NotImplementedError
|
||||
|
||||
|
||||
class JobManager:
|
||||
"""进程内的一种后台任务:一个 asyncio 任务一件事,停的时候给宽限。factory 按任务编号造一次执行(JobRun);label 是
|
||||
日志里的叫法(「数据目录起草任务」),name 是 asyncio 任务名的前缀。"""
|
||||
|
||||
def __init__(self, factory: Callable[[str], JobRun], *, label: str, name: str) -> None:
|
||||
self._runs: dict[str, JobRun] = {}
|
||||
self._factory = factory
|
||||
self.label = label
|
||||
self.name = name
|
||||
|
||||
def _alive(self, job_id: str) -> JobRun | None:
|
||||
run = self._runs.get(job_id)
|
||||
if run is None or run.task is None or run.task.done():
|
||||
return None
|
||||
try:
|
||||
if run.task.get_loop() is not asyncio.get_running_loop():
|
||||
self._runs.pop(job_id, None) # 别的事件循环留下的(测试每个用例一个循环)
|
||||
return None
|
||||
except RuntimeError:
|
||||
return None
|
||||
return run
|
||||
|
||||
def start(self, job_id: str) -> None:
|
||||
"""开始在后台执行这个(排队中的)任务:先进这个源的队,轮到了才开始。"""
|
||||
run = self._factory(job_id)
|
||||
run.task = asyncio.get_running_loop().create_task(run.run(), name=f"{self.name}-{job_id}")
|
||||
self._runs[job_id] = run
|
||||
|
||||
def forget(_: asyncio.Task[None]) -> None:
|
||||
if self._runs.get(job_id) is run:
|
||||
self._runs.pop(job_id, None)
|
||||
|
||||
run.task.add_done_callback(forget)
|
||||
|
||||
def running(self, job_id: str) -> bool:
|
||||
return self._alive(job_id) is not None
|
||||
|
||||
async def stop(self, job_id: str, reason: str, *, grace: float = STOP_GRACE_S) -> bool:
|
||||
"""停掉这个进程里在跑(或在排队等)的任务,等它收尾(最多 grace 秒)。不在这个进程里返回 False。"""
|
||||
run = self._alive(job_id)
|
||||
if run is None:
|
||||
return False
|
||||
if run.stop_reason is None:
|
||||
run.stop_reason = reason
|
||||
run.task.cancel() # type: ignore[union-attr]
|
||||
await asyncio.wait([run.task], timeout=grace) # type: ignore[list-item]
|
||||
return True
|
||||
|
||||
async def wait(self, job_id: str, timeout: float | None = None) -> None:
|
||||
"""等这个任务做完(测试、调试用)。不在跑直接返回。"""
|
||||
run = self._alive(job_id)
|
||||
if run is not None:
|
||||
await asyncio.wait([run.task], timeout=timeout) # type: ignore[list-item]
|
||||
|
||||
async def shutdown(self, *, grace: float = SHUTDOWN_GRACE_S) -> None:
|
||||
"""关停:停掉进行中、排队等着的任务(标中断),最多等 grace 秒收尾。"""
|
||||
runs = [run for job_id in list(self._runs) if (run := self._alive(job_id)) is not None]
|
||||
for run in runs:
|
||||
if run.stop_reason is None:
|
||||
run.stop_reason = "shutdown"
|
||||
run.task.cancel() # type: ignore[union-attr]
|
||||
if runs:
|
||||
await asyncio.wait([run.task for run in runs], timeout=grace) # type: ignore[misc]
|
||||
logger.info("已停止 %d 个%s", len(runs), self.label)
|
||||
|
||||
|
||||
__all__ = ["ACTIVE", "FINISHED", "JobFailed", "JobManager", "JobRun", "LaneBusy", "LaneTicket", "OPEN", "RETRYABLE",
|
||||
"SHUTDOWN_GRACE_S", "STOP_GRACE_S", "SerialWriter", "SourceLanes", "TASK_DONE", "TASK_STATUSES", "cancel_queued",
|
||||
"claim", "close_job", "interrupt_on_startup", "lanes"]
|
||||
sys.modules[__name__] = job_runtime
|
||||
|
||||
@@ -55,6 +55,11 @@ logger = logging.getLogger(__name__)
|
||||
|
||||
_OPEN = runtime.OPEN
|
||||
|
||||
#: 起草这边占同源队的几种事(叫法登记在运行时里,见 core/job_runtime.register_lane_kind):后台起草任务、同步起草、导入目录
|
||||
runtime.register_lane_kind("draft", "起草")
|
||||
runtime.register_lane_kind("sync_draft", "同步起草")
|
||||
runtime.register_lane_kind("import", "导入目录")
|
||||
|
||||
#: 一个任务最多几张表;起草要求、本次说明最多几个字
|
||||
MAX_TABLES = 500
|
||||
GUIDANCE_MAX = 2000
|
||||
|
||||
@@ -41,9 +41,9 @@ import re
|
||||
import time
|
||||
from collections.abc import Awaitable, Callable
|
||||
from dataclasses import asdict, dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
from datetime import date as _date, datetime, timezone
|
||||
from decimal import Decimal
|
||||
from typing import Any
|
||||
from typing import Any, NamedTuple
|
||||
|
||||
from sqlalchemy import select, update
|
||||
from sqlalchemy.dialects import mysql, oracle, postgresql, sqlite
|
||||
@@ -99,7 +99,7 @@ _NUMBER_FIELDS: dict[str, tuple[str, float, float, bool]] = {
|
||||
"table_max_queries": ("每张表最多查询数", 1, 200, True),
|
||||
"table_max_s": ("每张表时长上限(秒)", 10, 600, False),
|
||||
"query_timeout_s": ("单条查询时限(秒)", 1, 60, False),
|
||||
"sample_size": ("抽样键值数", 10, 10_000, True),
|
||||
"sample_size": ("核对键值数", 10, 10_000, True),
|
||||
"max_scan_rows": ("整表统计行数上限", 0, 10_000_000, True),
|
||||
}
|
||||
_ENABLED_LABEL = "开启数据剖析"
|
||||
@@ -241,16 +241,96 @@ def sql_literal(value: Any, kind: str | None) -> str | None:
|
||||
return "'" + value.replace("'", "''") + "'"
|
||||
|
||||
|
||||
#: 定长文字类型(CHAR(2)、NCHAR、bpchar、CHARACTER(2)):PostgreSQL、Oracle 取回的值带尾随空格补齐
|
||||
_FIXED_CHAR = re.compile(r"^\s*(?:N?CHAR|BPCHAR|CHARACTER)\b(?!\s+VARYING)", re.IGNORECASE)
|
||||
|
||||
|
||||
def is_fixed_char(type_name: str | None) -> bool:
|
||||
"""定长文字列(CHAR(n)):值的尾随空格是补齐出来的,不是取值的一部分(SQL 比较时也不算)。"""
|
||||
return bool(type_name) and bool(_FIXED_CHAR.match(str(type_name)))
|
||||
|
||||
|
||||
class KeyLiterals(NamedTuple):
|
||||
"""key_literals 的结果。literals:去父表里对的 IN 列表字面量(去重、保持顺序);unmatchable:按类型就对不上父键、
|
||||
算作没对上的键值个数(去重后);unsafe:值本身不能安全拼进 SQL、不计入核对的个数。"""
|
||||
literals: list[str]
|
||||
unmatchable: int
|
||||
unsafe: int
|
||||
|
||||
|
||||
def key_literals(values: list[Any], kind: str | None, *, parent: str | None,
|
||||
child_type: str | None) -> KeyLiterals:
|
||||
"""从子表取回的键值 → 去父表里对的 IN 列表字面量(KeyLiterals)。
|
||||
|
||||
字面量按**被指向列**的类型族写(catalog.column_family),不是按子表取回的值:
|
||||
- 父列是文字、子列是数:写成 '123'。PostgreSQL 上 varchar 列 IN (123) 直接报「operator does not exist」;
|
||||
- 父列是数、子列是文字:只拼整数写法的值('0012' 也照 12 写)。别的('A1'、'WALKIN')不拼——Oracle 上数字列
|
||||
IN ('A1') 会把字面量转成数报 ORA-01722,MySQL 会把 'A1' 转成 0 去比、对上不该对上的行——但它们**照样是子表
|
||||
的键值**:整数主键里永远没有 'WALKIN',所以算作没对上,计入 unmatchable,由调用方留在覆盖率的分母里。要是
|
||||
从分母里拿掉,散客代号占九成的一列只剩 1 个整数写法的键对上,就成了「覆盖率 100%」;
|
||||
- 父列是时间:不在这里处理,_check_relation 先整条跳过(各家时间字面量写法不同、精度不同,键不核对);
|
||||
- 父列类型认不出:照子表取回的值的类型写(sql_literal,以前的做法)。
|
||||
子表是定长文字(CHAR(n))时先去掉尾随空格:PG、Oracle 取回的 'A ' 拿去和 VARCHAR 父列比就对不上,而按 CHAR 的
|
||||
语义尾随空格本来就不算。不安全的值(反斜杠、「:名字」、控制字符、太长)照 sql_literal 的规矩跳过,记进 unsafe
|
||||
(调用方另记「类型不支持核对,未计入」)。两个不同的取值写成同一个字面量('0012' 和 '12')只算一个,不算 unsafe。
|
||||
"""
|
||||
if parent == "time":
|
||||
raise ValueError("被指向列是日期时间的关系不核对键值(见 _check_relation)")
|
||||
fixed = is_fixed_char(child_type)
|
||||
out: list[str] = []
|
||||
unmatchable: set[str] = set()
|
||||
unsafe = 0
|
||||
for value in values:
|
||||
if fixed and isinstance(value, str):
|
||||
value = value.rstrip(" ")
|
||||
if parent == "text":
|
||||
if isinstance(value, bool):
|
||||
unsafe += 1
|
||||
continue
|
||||
if isinstance(value, (int, float, Decimal)):
|
||||
lit = sql_literal(value, "number")
|
||||
lit = None if lit is None else "'" + lit + "'"
|
||||
else:
|
||||
lit = sql_literal(value, "text")
|
||||
elif parent == "number":
|
||||
if isinstance(value, str):
|
||||
text = value.strip()
|
||||
if not _INT_TEXT.fullmatch(text):
|
||||
unmatchable.add(text)
|
||||
continue
|
||||
lit = str(int(text))
|
||||
else:
|
||||
lit = sql_literal(value, "number")
|
||||
else:
|
||||
lit = sql_literal(value, kind)
|
||||
if lit is None:
|
||||
unsafe += 1
|
||||
else:
|
||||
out.append(lit)
|
||||
return KeyLiterals(list(dict.fromkeys(out)), len(unmatchable), unsafe)
|
||||
|
||||
|
||||
_DIALECT_MODULES = {"sqlite": sqlite, "postgres": postgresql, "postgresql": postgresql, "mysql": mysql,
|
||||
"mariadb": mysql, "oracle": oracle}
|
||||
|
||||
|
||||
#: 日期列的最小值、最大值只看这个区间里的值(含下限、不含上限):0000-00-00(MySQL 关了严格模式写进去的零日期)、
|
||||
#: 0001-01-01、9999-12-31(「没有」「永久」的占位日期)、PostgreSQL 的 -infinity / infinity 都落在区间外。区间外的
|
||||
#: 值照实数出来(「有 N 个无效日期未计入」),不让一个占位日期把业务日期的范围撑成两千年。两端整天排除:带时刻的
|
||||
#: 9999-12-31 23:59:59、文字存的「0001-01-01 00:00:00」也算占位
|
||||
VALID_DATE_FROM = "0001-01-02"
|
||||
VALID_DATE_BEFORE = "9999-12-31"
|
||||
_ISO_DAY = re.compile(r"\d{4}-\d{2}-\d{2}")
|
||||
|
||||
|
||||
class SqlDialect:
|
||||
"""剖析 SQL 的方言差异:标识符怎么加引号、怎么只取前 n 行、统计信息在哪个系统视图里。只拼字符串,不连库。
|
||||
"""剖析 SQL 的方言差异:标识符怎么加引号、怎么只取前 n 行、统计信息在哪个系统视图里、日期字面量和按天截断
|
||||
怎么写、有没有块采样。只拼字符串,不连库。
|
||||
|
||||
标识符交给 SQLAlchemy 各方言的 identifier_preparer:需要时才加引号(memberTags 加、visits 不加),
|
||||
MySQL 用反引号。Oracle 尤其不能一律加双引号:探查拿到的表名是 SQLAlchemy 规整过的小写,库里其实是
|
||||
大写,"visits" 加了引号就成了另一张不存在的表;不加引号 Oracle 自己转大写才对得上。
|
||||
大写,"visits" 加了引号就成了另一张不存在的表;不加引号 Oracle 自己转大写才对得上。反过来,库里真是小写的
|
||||
对象(建表时写了 "visits")探查时记了 quote_name,这种要强制加引号、不转大写(table、column、stats_sql)。
|
||||
"""
|
||||
|
||||
def __init__(self, kind: str | None) -> None:
|
||||
@@ -262,13 +342,64 @@ class SqlDialect:
|
||||
self._prep = self._dialect.identifier_preparer
|
||||
self.oracle = self.kind == "oracle"
|
||||
|
||||
def quote(self, name: str) -> str:
|
||||
return self._prep.quote(name)
|
||||
def quote(self, name: str, *, force: bool = False) -> str:
|
||||
"""标识符。force:探查记了 quote_name 的名字,原样加引号(Oracle 小写带引号的对象)。"""
|
||||
return self._prep.quote_identifier(name) if force else self._prep.quote(name)
|
||||
|
||||
def table(self, meta: dict[str, Any] | None, name: str) -> str:
|
||||
"""写进 FROM 的表名:探查时指定了 schema 的带上它(Oracle 只读账号名下没有对象,全靠 schema 前缀)。"""
|
||||
schema = (meta or {}).get("schema")
|
||||
return f"{self._prep.quote_schema(schema)}.{self.quote(name)}" if schema else self.quote(name)
|
||||
quoted = self.quote(name, force=(meta or {}).get("quote_name") is True)
|
||||
return f"{self._prep.quote_schema(schema)}.{quoted}" if schema else quoted
|
||||
|
||||
def column(self, meta: dict[str, Any] | None, name: str) -> str:
|
||||
"""写进 SQL 的列名:表结构里这一列记了 quote_name 的强制加引号。"""
|
||||
cols = (meta or {}).get("columns") or []
|
||||
entry = next((c for c in cols if isinstance(c, dict) and c.get("name") == name), None)
|
||||
return self.quote(name, force=bool(entry and entry.get("quote_name") is True))
|
||||
|
||||
def date_literal(self, value: Any, *, text: bool = False) -> str:
|
||||
"""日期字面量(取值观察的时间窗口、日期列的有效区间用)。PostgreSQL、MySQL、Oracle 都认 DATE 'YYYY-MM-DD';
|
||||
SQLite 没有日期类型,日期存成文字,写字符串。text:列是文字存的日期(VARCHAR),按字符串比——PostgreSQL 上
|
||||
varchar 和 date 比较会直接报错。只收真实存在的日期(date 或 YYYY-MM-DD),别的抛 ValueError:拼进 SQL 的
|
||||
只能是我们自己算出来的日子。"""
|
||||
if isinstance(value, datetime):
|
||||
value = value.date()
|
||||
if isinstance(value, _date):
|
||||
day = value.isoformat()
|
||||
elif isinstance(value, str) and _ISO_DAY.fullmatch(value):
|
||||
try:
|
||||
day = _date.fromisoformat(value).isoformat()
|
||||
except ValueError:
|
||||
raise ValueError(f"不是有效的日期:{value}") from None
|
||||
else:
|
||||
raise ValueError(f"不是有效的日期:{value!r}")
|
||||
if text or self.kind == "sqlite":
|
||||
return f"'{day}'"
|
||||
return f"DATE '{day}'"
|
||||
|
||||
def day_trunc(self, column: str) -> str:
|
||||
"""时间列截到天(按日行数用)。PostgreSQL CAST(x AS DATE);MySQL DATE(x);Oracle 必须用 TRUNC(x):Oracle 的
|
||||
DATE 带时分秒,CAST(ts AS DATE) 截不掉;SQLite 的日期是文字,取前 10 个字符。"""
|
||||
if self.kind in ("postgres", "postgresql"):
|
||||
return f"CAST({column} AS DATE)"
|
||||
if self.kind in ("mysql", "mariadb"):
|
||||
return f"DATE({column})"
|
||||
if self.oracle:
|
||||
return f"TRUNC({column})"
|
||||
return f"substr({column}, 1, 10)"
|
||||
|
||||
def sample_from(self, table: str, percent: float) -> str | None:
|
||||
"""块采样的 FROM:PostgreSQL TABLESAMPLE SYSTEM (p)、Oracle SAMPLE BLOCK (p),按数据块随机取约 p% 的行;
|
||||
MySQL、SQLite 没有块采样,返回 None(调用方退回「按存储顺序的前 N 行」,并照实写口径)。"""
|
||||
p = float(percent)
|
||||
if not (0 < p < 100):
|
||||
raise ValueError(f"采样比例要在 0 到 100 之间:{percent}")
|
||||
if self.kind in ("postgres", "postgresql"):
|
||||
return f"{table} TABLESAMPLE SYSTEM ({p:g})"
|
||||
if self.oracle:
|
||||
return f"{table} SAMPLE BLOCK ({p:g})"
|
||||
return None
|
||||
|
||||
def limit(self, sql: str, n: int) -> str:
|
||||
"""只取前 n 行。Oracle 11g 没有 LIMIT 也没有 FETCH FIRST,一律套一层 ROWNUM。"""
|
||||
@@ -304,7 +435,8 @@ class SqlDialect:
|
||||
if self.oracle:
|
||||
denorm = self._dialect.denormalize_name
|
||||
owner = sql_literal(denorm(schema), "text") if schema else "SYS_CONTEXT('USERENV', 'CURRENT_SCHEMA')"
|
||||
table = sql_literal(denorm(name), "text")
|
||||
# 小写带引号的对象(quote_name)在数据字典里就是小写,不能转成大写去找
|
||||
table = sql_literal(name if (meta or {}).get("quote_name") is True else denorm(name), "text")
|
||||
if not (owner and table):
|
||||
return None
|
||||
return f"SELECT NUM_ROWS FROM ALL_TABLES WHERE OWNER = {owner} AND TABLE_NAME = {table}"
|
||||
@@ -335,8 +467,49 @@ class SqlDialect:
|
||||
f"WHERE {column} IS NOT NULL GROUP BY {column} ORDER BY cnt DESC, {column}")
|
||||
return self.limit(sql, n)
|
||||
|
||||
def min_max_sql(self, table: str, columns: list[str]) -> str:
|
||||
return f"SELECT {', '.join(f'MIN({c}), MAX({c})' for c in columns)} FROM {table}"
|
||||
def min_max_sql(self, table: str, columns: list[tuple[str, bool]]) -> str:
|
||||
"""日期列的最小值、最大值和无效日期的个数:每列三个值(MIN、MAX、区间外的个数)。columns 是 (写进 SQL 的列,
|
||||
是不是文字存的日期)。只看有效区间(VALID_DATE_FROM ≤ x < VALID_DATE_BEFORE)里的值,见 VALID_DATE_FROM。
|
||||
整表统计,只对小表用。"""
|
||||
parts = []
|
||||
for c, text in columns:
|
||||
lo = self.date_literal(VALID_DATE_FROM, text=text)
|
||||
hi = self.date_literal(VALID_DATE_BEFORE, text=text)
|
||||
valid = f"{c} >= {lo} AND {c} < {hi}"
|
||||
parts.append(f"MIN(CASE WHEN {valid} THEN {c} END), MAX(CASE WHEN {valid} THEN {c} END), "
|
||||
f"SUM(CASE WHEN {c} < {lo} OR {c} >= {hi} THEN 1 ELSE 0 END)")
|
||||
return f"SELECT {', '.join(parts)} FROM {table}"
|
||||
|
||||
# ---- 取值观察(下一轮第 3 波用):窗口、按日行数、块采样。这里先把各方言的写法定下来,金样和容器集成测试都覆盖
|
||||
|
||||
def _window(self, column: str, since: Any, text: bool) -> str:
|
||||
"""时间窗口的条件:since 起(含),并且早于占位日期 9999-12-31——PostgreSQL 的 infinity、Oracle 的 9999-12-31
|
||||
都大于任何一天,不截掉上端就会在「最近 35 天」里多出一天。"""
|
||||
return (f"{column} >= {self.date_literal(since, text=text)} "
|
||||
f"AND {column} < {self.date_literal(VALID_DATE_BEFORE, text=text)}")
|
||||
|
||||
def day_volume_sql(self, table: str, column: str, since: Any, *, text: bool = False) -> str:
|
||||
"""最近若干天按天计数:since 起(含)每天几行。只在日期列打头有索引时对大表发(走索引的范围扫描)。"""
|
||||
day = self.day_trunc(column)
|
||||
return (f"SELECT {day} AS d, COUNT(*) AS n FROM {table} "
|
||||
f"WHERE {self._window(column, since, text)} GROUP BY {day} ORDER BY d")
|
||||
|
||||
def window_value_counts_sql(self, table: str, column: str, date_column: str, since: Any, n: int, *,
|
||||
text: bool = False) -> str:
|
||||
"""时间窗口里的取值分布:业务日期 since 起(含)的行里,出现最多的 n 个取值和各自的行数。"""
|
||||
sql = (f"SELECT {column}, COUNT(*) AS cnt FROM {table} "
|
||||
f"WHERE {self._window(date_column, since, text)} AND {column} IS NOT NULL "
|
||||
f"GROUP BY {column} ORDER BY cnt DESC, {column}")
|
||||
return self.limit(sql, n)
|
||||
|
||||
def sample_value_counts_sql(self, table: str, column: str, percent: float, n: int) -> str | None:
|
||||
"""块采样里的取值分布;没有块采样的方言返回 None。"""
|
||||
source = self.sample_from(table, percent)
|
||||
if source is None:
|
||||
return None
|
||||
sql = (f"SELECT {column}, COUNT(*) AS cnt FROM {source} WHERE {column} IS NOT NULL "
|
||||
f"GROUP BY {column} ORDER BY cnt DESC, {column}")
|
||||
return self.limit(sql, n)
|
||||
|
||||
|
||||
# ==========================================================================
|
||||
@@ -581,8 +754,10 @@ async def estimate_size(run: Callable[..., Awaitable[QueryResult]], dialect: Sql
|
||||
"""先读统计信息;读不到(SQLite、没分析过的表)再数到 max_scan_rows + 1 行为止。
|
||||
|
||||
统计信息为 0 或负数(PostgreSQL 没分析过的表是 -1,MySQL 的估算值可能是 0)当作不知道:一张其实很大的表
|
||||
要是被当成空表,后面就会对它做整表统计。返回 (行数, 途中没查成的几条)。这张表的查询次数、时长用完(或者任务止损)
|
||||
时抛 _Stop。
|
||||
要是被当成空表,后面就会对它做整表统计。统计信息说的行数不超过 max_scan_rows 时也再数一次(多一条有上限的
|
||||
计数):统计信息可能是很久以前的(表后来涨到了几千万行,ANALYZE 没再跑过),照它把大表当小表,就会对它做
|
||||
整表的 COUNT(DISTINCT)、MIN/MAX。数过的结果是准数(count),数到上限还没数完就按大表算(at_least)。
|
||||
返回 (行数, 途中没查成的几条)。这张表的查询次数、时长用完(或者任务止损)时抛 _Stop。
|
||||
"""
|
||||
skips: list[_Skip] = []
|
||||
sql = dialect.stats_sql(meta, name)
|
||||
@@ -592,7 +767,9 @@ async def estimate_size(run: Callable[..., Awaitable[QueryResult]], dialect: Sql
|
||||
except _Skip as e:
|
||||
skips.append(e)
|
||||
else:
|
||||
if rows is not None and rows > 0:
|
||||
if rows is not None and rows > max(max_scan_rows, 0):
|
||||
return TableSize(rows, "stats"), skips
|
||||
if rows is not None and rows > 0 and max_scan_rows <= 0:
|
||||
return TableSize(rows, "stats"), skips
|
||||
if max_scan_rows <= 0:
|
||||
return TableSize(None, "unknown"), skips
|
||||
@@ -752,6 +929,8 @@ class _CodesOutcome:
|
||||
rows: int
|
||||
note: str
|
||||
complete: bool = False
|
||||
#: 列定义(ENUM)列出的取值:这次没出现的也写进码值(含义待填),码值是「可能的取值」
|
||||
declared: list[str] = field(default_factory=list)
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -901,6 +1080,11 @@ def _column(meta: dict[str, Any], name: str) -> str | None:
|
||||
if isinstance(c, dict) and str(c.get("name", "")).lower() == lowered), None)
|
||||
|
||||
|
||||
def _column_meta(meta: dict[str, Any], name: str) -> dict[str, Any] | None:
|
||||
"""表结构里这一列(类型、取值列表、quote_name);name 是表结构里的写法(_column 的返回值)。"""
|
||||
return next((c for c in meta.get("columns") or [] if isinstance(c, dict) and c.get("name") == name), None)
|
||||
|
||||
|
||||
def _constraint(meta: dict[str, Any], column: str) -> str | None:
|
||||
"""这一列单独是主键或某个唯一约束时返回「主键」/「唯一约束」,否则 None。"""
|
||||
wanted = [column.lower()]
|
||||
@@ -950,7 +1134,7 @@ async def _unique(cx: _TableCx, table: str, meta: dict[str, Any], column: str,
|
||||
if not size.within(cx.settings.max_scan_rows):
|
||||
return None, "too_large"
|
||||
d = cx.dialect
|
||||
res = await cx.run(d.unique_check_sql(d.table(meta, table), d.quote(column)))
|
||||
res = await cx.run(d.unique_check_sql(d.table(meta, table), d.column(meta, column)))
|
||||
row = res.rows[0] if res.rows else [None, None]
|
||||
total, distinct = _as_int(row[0]), _as_int(row[1])
|
||||
if total is None or distinct is None:
|
||||
@@ -969,20 +1153,39 @@ def _fk_note(child: tuple[bool | None, str]) -> str:
|
||||
return f"{catalog.PROFILE_NOTE_PREFIX}外键约束已保证引用完整,未抽样核对覆盖率;子表一侧{basis}:{shape}"
|
||||
|
||||
|
||||
#: MySQL 上比较文字按列的排序规则:常见的排序规则不分大小写(_ci),PAD SPACE 的还忽略尾随空格。覆盖率、唯一性、
|
||||
#: 码值的分组都按它算,写进说明,人才知道「A01」和「a01」在这里算同一个值
|
||||
MYSQL_COLLATION_NOTE = "MySQL 按列的排序规则比较文字(常见的排序规则不分大小写),覆盖率和唯一性按它计算"
|
||||
MYSQL_CODES_COLLATION_NOTE = "MySQL 按列的排序规则分组(常见的排序规则不分大小写),大小写不同的写法可能并成一个取值"
|
||||
|
||||
|
||||
def _collation_note(dialect: SqlDialect, family: str | None, note: str) -> str | None:
|
||||
"""MySQL、MariaDB 上文字列(或类型认不出的列)的排序规则说明;别的方言、数字列没有。"""
|
||||
return note if dialect.kind in ("mysql", "mariadb") and family in ("text", None) else None
|
||||
|
||||
|
||||
def _relation_note(*, sample: int, matched: int, coverage: float, scan_cap: int | None,
|
||||
skipped_values: int, parent: tuple[bool | None, str], child: tuple[bool | None, str] | None,
|
||||
holds: bool) -> str:
|
||||
"""关系的剖析备注:抽样规模、覆盖率、父键和子键唯一的依据。人据此判断这条结论有多可靠。
|
||||
不写日期:什么时候剖析的看这一项的 updated_at(界面按本地时间显示)。"""
|
||||
holds: bool, unmatchable: int = 0, collation: str | None = None) -> str:
|
||||
"""关系的剖析备注:取了多少个键值、看了多少行、覆盖率、父键和子键唯一的依据。人据此判断这条结论有多可靠。
|
||||
不写日期:什么时候剖析的看这一项的 updated_at(界面按本地时间显示)。
|
||||
|
||||
不说「抽样」:DISTINCT … LIMIT 取的是数据库先给出的那几个值,大表上是按存储顺序的前若干行里的值,不是随机抽的
|
||||
(多半是最老的数据)。unmatchable:被指向列是数字、子表取到的却不是整数写法的键值个数(key_literals)——已经算在
|
||||
sample 里、算作没对上,这里说明为什么没去父表里对。"""
|
||||
parts: list[str] = []
|
||||
low = coverage < catalog.PROFILE_VERIFY_COVERAGE
|
||||
if low:
|
||||
parts.append(f"抽样覆盖率 {_percent(coverage)},可能不是这条关系")
|
||||
scope = f"(只看前 {scan_cap} 行)" if scan_cap else ""
|
||||
sampled = f"子表抽样 {sample} 个不同键值{scope},父表对上 {matched} 个"
|
||||
parts.append(f"覆盖率 {_percent(coverage)},可能不是这条关系")
|
||||
scope = f"(按存储顺序只看前 {scan_cap} 行)" if scan_cap else ""
|
||||
sampled = f"子表取了 {sample} 个不同键值{scope},父表对上 {matched} 个"
|
||||
parts.append(sampled if low else f"{sampled},覆盖率 {_percent(coverage)}")
|
||||
if skipped_values:
|
||||
parts.append(f"另有 {skipped_values} 个键值的类型不支持核对,未计入")
|
||||
if unmatchable:
|
||||
parts.append(f"其中 {unmatchable} 个不是整数写法,被指向列是数字,算作没对上")
|
||||
if collation:
|
||||
parts.append(collation)
|
||||
unique, how = parent
|
||||
if unique is True:
|
||||
parts.append(f"被指向列是{how}" if how in ("主键", "唯一约束") else "被指向列经计数核对唯一")
|
||||
@@ -1026,24 +1229,32 @@ async def _check_relation(cx: _TableCx, table: str, rel: dict[str, Any], out: _T
|
||||
return
|
||||
|
||||
d, s = cx.dialect, cx.settings
|
||||
child_col, parent_col = _column_meta(meta, col), _column_meta(pmeta, pcol)
|
||||
parent_family = catalog.column_family(parent_col)
|
||||
if parent_family == "time":
|
||||
# 时间键不核对:各家时间字面量写法、精度不同,拼出来的 IN 列表对不对得上说不准。不发子表的取值查询
|
||||
raise _Skip("unsupported", f"{parent}.{pcol} 是日期时间,各库的时间写法和精度不同,不核对这种关系的覆盖率")
|
||||
size = await cx.size_with_data(table, result)
|
||||
scan_cap = cx.scan_cap(size)
|
||||
child_table, parent_table = d.table(meta, table), d.table(pmeta, parent)
|
||||
res = await cx.run(d.distinct_sample_sql(child_table, d.quote(col), s.sample_size, scan_cap=scan_cap),
|
||||
res = await cx.run(d.distinct_sample_sql(child_table, d.column(meta, col), s.sample_size, scan_cap=scan_cap),
|
||||
max_rows=s.sample_size)
|
||||
kind = (res.column_types or {}).get(res.columns[0]) if res.columns else None
|
||||
values = [row[0] for row in res.rows if row]
|
||||
literals = list(dict.fromkeys(lit for v in values if (lit := sql_literal(v, kind)) is not None))
|
||||
literals, unmatchable, unsafe = key_literals(values, kind, parent=parent_family,
|
||||
child_type=(child_col or {}).get("type"))
|
||||
if not values:
|
||||
raise _Skip("no_data", f"{table}.{col} 没有非空值,无法核对")
|
||||
if not literals:
|
||||
if not literals and not unmatchable:
|
||||
raise _Skip("unsupported", f"{table}.{col} 的取值类型不支持核对(只核对整数和文字)")
|
||||
# 分母是子表取到的全部键值:不是整数写法、按类型就对不上整数父键的(unmatchable)也在里面,算作没对上
|
||||
sample = len(literals) + unmatchable
|
||||
matched = 0
|
||||
for start in range(0, len(literals), IN_CHUNK):
|
||||
sql = d.match_count_sql(parent_table, d.quote(pcol), literals[start:start + IN_CHUNK])
|
||||
sql = d.match_count_sql(parent_table, d.column(pmeta, pcol), literals[start:start + IN_CHUNK])
|
||||
matched += _as_int(_first(await cx.run(sql))) or 0
|
||||
matched = min(matched, len(literals))
|
||||
coverage = round(matched / len(literals), 4)
|
||||
coverage = round(matched / sample, 4)
|
||||
|
||||
parent_unique = await _unique(cx, parent, pmeta, pcol, result)
|
||||
child_unique = await _unique(cx, table, meta, col, result) if parent_unique[0] else None
|
||||
@@ -1051,12 +1262,13 @@ async def _check_relation(cx: _TableCx, table: str, rel: dict[str, Any], out: _T
|
||||
if child_unique is not None:
|
||||
cardinality = "one_to_one" if child_unique[0] is True else "many_to_one"
|
||||
holds = coverage >= catalog.PROFILE_VERIFY_COVERAGE and cardinality is not None
|
||||
note = _relation_note(sample=len(literals), matched=matched, coverage=coverage, scan_cap=scan_cap,
|
||||
skipped_values=len(values) - len(literals), parent=parent_unique, child=child_unique,
|
||||
holds=holds)
|
||||
note = _relation_note(sample=sample, matched=matched, coverage=coverage, scan_cap=scan_cap,
|
||||
skipped_values=unsafe, parent=parent_unique, child=child_unique, holds=holds,
|
||||
unmatchable=unmatchable,
|
||||
collation=_collation_note(d, parent_family, MYSQL_COLLATION_NOTE))
|
||||
checked = child_unique is not None and child_unique[0] is not None
|
||||
out.relations.append(_RelationOutcome(rid=str(rel["id"]), target=_relation_target(rel), coverage=coverage,
|
||||
cardinality=cardinality, holds=holds, sample=len(literals),
|
||||
cardinality=cardinality, holds=holds, sample=sample,
|
||||
matched=matched, note=note, cardinality_checked=checked))
|
||||
|
||||
|
||||
@@ -1076,17 +1288,28 @@ _NOT_CODE_TAILS = frozenset({"id", "no", "num", "number", "name", "desc", "descr
|
||||
"at", "date", "time", "on", "amount", "price", "count", "qty", "rate", "url"})
|
||||
#: 像日期的取值:2026-07-01、2026/07/01,后面可以跟时间
|
||||
_DATE_TEXT = re.compile(r"^\d{4}[-/]\d{2}[-/]\d{2}")
|
||||
#: 日期列最小值、最大值不计入的值(见 VALID_DATE_FROM)在说明里的说法
|
||||
INVALID_DATES = "零日期 0000-00-00、0001-01-01、9999-12-31 这类占位日期或无穷大"
|
||||
|
||||
|
||||
def _code_type_ok(type_name: str | None) -> bool:
|
||||
"""整数,或者短文字(定长不超过 MAX_CODE_LEN;SQLite 的 TEXT 不带长度,也算)。"""
|
||||
t = (type_name or "").upper()
|
||||
def _is_enum(type_name: str | None, enum_values: Any = None) -> bool:
|
||||
"""取值列表有限的列:MySQL 的 ENUM(…) / SET(…),或者探查记了取值列表(PostgreSQL 的枚举:类型写法只是类型名,
|
||||
setup_status 这种以 set 开头的名字不算 SET,见 catalog._MYSQL_ENUM)。"""
|
||||
return bool(enum_values) or catalog._mysql_enum_kind(type_name) is not None
|
||||
|
||||
|
||||
def _code_type_ok(type_name: str | None, enum_values: Any = None) -> bool:
|
||||
"""ENUM / SET(取值列表有限,直接算码值候选)、整数,或者短文字(定长不超过 MAX_CODE_LEN;SQLite 的 TEXT 不带
|
||||
长度,也算)。类型是按方言编译的完整写法:VARCHAR2(20 CHAR) 的长度按 20 算。"""
|
||||
if _is_enum(type_name, enum_values):
|
||||
return True
|
||||
t = catalog._type_words(type_name)
|
||||
if not t or any(k in t for k in ("CLOB", "BLOB", "BINARY", "JSON", "DATE", "TIME", "INTERVAL", "REAL", "FLOAT",
|
||||
"DOUBLE", "BOOL")):
|
||||
return False
|
||||
if "INT" in t:
|
||||
return True
|
||||
size = re.search(r"\(\s*(\d+)\s*(?:,\s*(\d+)\s*)?\)", t)
|
||||
size = re.search(r"\(\s*(\d+)(?:\s+(?:CHAR|BYTE))?\s*(?:,\s*(\d+)\s*)?\)", t)
|
||||
if any(k in t for k in ("NUMBER", "NUMERIC", "DECIMAL")):
|
||||
return size is None or not size.group(2) or int(size.group(2)) == 0
|
||||
if any(k in t for k in ("CHAR", "TEXT", "STRING")):
|
||||
@@ -1100,9 +1323,9 @@ def _looks_like_code(name: str) -> bool:
|
||||
|
||||
|
||||
def _code_columns(meta: dict[str, Any], notes: dict[str, Any]) -> list[str]:
|
||||
"""要取值分布的列:目录里度量类型记为状态的、剖析以前提过码值候选的(再核一次)、或者类型和命名像状态、
|
||||
类型、渠道的整数或短文字列。主键、唯一约束列、关系里的键列不算;码值人工确认或驳回过的不再取。
|
||||
被遮罩的列也列出来,由检查记为「被遮罩」跳过——要让人看得见为什么它没有码值。"""
|
||||
"""要取值分布的列:目录里度量类型记为状态的、剖析以前提过码值候选的(再核一次)、ENUM / SET 列(取值列表有限,
|
||||
不看名字)、或者类型和命名像状态、类型、渠道的整数或短文字列。主键、唯一约束列、关系里的键列不算;码值人工确认
|
||||
或驳回过的不再取。被遮罩的列也列出来,由检查记为「被遮罩」跳过——要让人看得见为什么它没有码值。"""
|
||||
pk = [c.lower() for c in meta.get("primary_key") or []]
|
||||
unique = {str(g[0]).lower() for g in meta.get("unique") or [] if isinstance(g, list) and len(g) == 1}
|
||||
keys = {str(c).lower() for r in notes.get("relations") or [] if isinstance(r, dict) for c in r.get("columns") or []}
|
||||
@@ -1117,23 +1340,28 @@ def _code_columns(meta: dict[str, Any], notes: dict[str, Any]) -> list[str]:
|
||||
status_measure = (isinstance(measure, dict) and measure.get("value") == "status"
|
||||
and measure.get("status") != "rejected")
|
||||
profiled = isinstance(codes, dict) and codes.get("source") == "profile"
|
||||
if not (status_measure or profiled or _looks_like_code(name)):
|
||||
enum = _is_enum(col.get("type"), col.get("enum_values"))
|
||||
if not (status_measure or profiled or enum or _looks_like_code(name)):
|
||||
continue
|
||||
lowered = name.lower()
|
||||
if (pk == [lowered]) or lowered in unique or lowered in keys or not _code_type_ok(col.get("type")):
|
||||
if (pk == [lowered]) or lowered in unique or lowered in keys or not _code_type_ok(col.get("type"),
|
||||
col.get("enum_values")):
|
||||
continue
|
||||
out.append(name)
|
||||
return out
|
||||
|
||||
|
||||
def _code_key(value: Any) -> str | None:
|
||||
"""取值分布里的一个取值 → 码值(文字)。只认整数和不太长的文字。"""
|
||||
def _code_key(value: Any, *, fixed: bool = False) -> str | None:
|
||||
"""取值分布里的一个取值 → 码值(文字)。只认整数和不太长的文字。fixed:定长文字列(CHAR(n)),去掉补齐的尾随
|
||||
空格(PG、Oracle 取回的是 'A ',WHERE status = 'A' 照样对得上,码值写 'A')。"""
|
||||
if value is None or isinstance(value, bool):
|
||||
return None
|
||||
if isinstance(value, int):
|
||||
return str(value)
|
||||
if isinstance(value, float):
|
||||
return str(int(value)) if value.is_integer() else None
|
||||
if isinstance(value, str) and fixed:
|
||||
value = value.rstrip(" ")
|
||||
if isinstance(value, str) and value.strip() and len(value) <= MAX_CODE_LEN and not _CONTROL.search(value):
|
||||
return value
|
||||
return None
|
||||
@@ -1151,34 +1379,53 @@ async def _check_codes(cx: _TableCx, table: str, column: str, out: _TableOutcome
|
||||
if cx.is_masked(column):
|
||||
raise _Skip("masked", f"{table}.{column} 在数据源设置中被遮罩,不取值", carry=False)
|
||||
d, meta = cx.dialect, cx.tables[table]
|
||||
col_meta = _column_meta(meta, column) or {}
|
||||
size = await cx.size_with_data(table, result)
|
||||
scan_cap = cx.scan_cap(size)
|
||||
res = await cx.run(d.value_counts_sql(d.table(meta, table), d.quote(column), CODES_MAX + 1, scan_cap=scan_cap),
|
||||
res = await cx.run(d.value_counts_sql(d.table(meta, table), d.column(meta, column), CODES_MAX + 1,
|
||||
scan_cap=scan_cap),
|
||||
max_rows=CODES_MAX + 1)
|
||||
if not res.rows:
|
||||
raise _Skip("no_data", f"{table}.{column} 没有非空值,无法取值分布")
|
||||
if len(res.rows) > CODES_MAX:
|
||||
raise _Skip("high_cardinality", f"{table}.{column} 的取值超过 {CODES_MAX} 个,不作为码值候选", carry=False)
|
||||
values: list[tuple[str, int]] = []
|
||||
fixed = is_fixed_char(col_meta.get("type"))
|
||||
enum = _is_enum(col_meta.get("type"), col_meta.get("enum_values"))
|
||||
counts: dict[str, int] = {}
|
||||
for row in res.rows:
|
||||
key = _code_key(row[0])
|
||||
key = _code_key(row[0], fixed=fixed)
|
||||
if key is None:
|
||||
raise _Skip("unsupported", f"{table}.{column} 的取值不是整数或短文字,不作为码值候选", carry=False)
|
||||
values.append((key, _as_int(row[1]) or 0))
|
||||
if len(values) >= 3 and all(n == 1 for _, n in values):
|
||||
# 每个取值只出现一次(票种编码 T01…T09):这是一列标识,不是状态码
|
||||
counts[key] = counts.get(key, 0) + (_as_int(row[1]) or 0)
|
||||
values = list(counts.items())
|
||||
if len(values) >= 3 and all(n == 1 for _, n in values) and not enum:
|
||||
# 每个取值只出现一次(票种编码 T01…T09):这是一列标识,不是状态码。ENUM 的取值列表是列定义给的,不按这条判
|
||||
raise _Skip("high_cardinality", f"{table}.{column} 的取值各不相同,像标识而不是码值,不作为码值候选",
|
||||
carry=False)
|
||||
rows = sum(n for _, n in values)
|
||||
scope = f"按前 {scan_cap} 行统计" if scan_cap else "统计全表"
|
||||
# 不说「抽样」:大表看的是按存储顺序的前若干行(多半是最老的数据),不是随机抽的
|
||||
scope = f"按存储顺序取前 {scan_cap} 行统计" if scan_cap else "统计全表"
|
||||
spread = "、".join(f"{k}({_share(n, rows)})" for k, n in values)
|
||||
note = f"{NOTE_PREFIX}{scope},{rows} 行非空值共 {len(values)} 个取值:{spread}。"
|
||||
out.codes.append(_CodesOutcome(column=column, values=values, rows=rows, note=note, complete=scan_cap is None))
|
||||
extra: list[str] = []
|
||||
# ENUM 的取值列表(SET 的是成员,取值是成员的组合,不算):这次没出现的也是可能的取值
|
||||
is_set = catalog._mysql_enum_kind(col_meta.get("type")) == "SET"
|
||||
declared = [] if is_set else [str(v) for v in col_meta.get("enum_values") or []]
|
||||
unseen = [v for v in declared if v not in counts]
|
||||
if unseen:
|
||||
extra.append(f"列定义里另有 {len(unseen)} 个取值这次没有出现:{'、'.join(unseen[:CODES_MAX])}")
|
||||
family = "text" if enum else catalog.column_family(col_meta)
|
||||
collation = _collation_note(d, family, MYSQL_CODES_COLLATION_NOTE)
|
||||
if collation:
|
||||
extra.append(collation)
|
||||
tail = "".join(f";{x}" for x in extra)
|
||||
note = f"{NOTE_PREFIX}{scope},{rows} 行非空值共 {len(values)} 个取值:{spread}{tail}。"
|
||||
out.codes.append(_CodesOutcome(column=column, values=values, rows=rows, note=note, complete=scan_cap is None,
|
||||
declared=unseen))
|
||||
|
||||
|
||||
def _is_date_column(col: dict[str, Any]) -> bool:
|
||||
"""日期类列(catalog.is_date_column:和 SQL 检查认时间列是同一个判断)。"""
|
||||
return catalog.is_date_column(str(col.get("name") or ""), col.get("type"))
|
||||
return catalog.is_date_column(str(col.get("name") or ""), col.get("type"), enum_values=col.get("enum_values"))
|
||||
|
||||
|
||||
async def _check_dates(cx: _TableCx, table: str, columns: list[str], total: int, out: _TableOutcome,
|
||||
@@ -1190,20 +1437,31 @@ async def _check_dates(cx: _TableCx, table: str, columns: list[str], total: int,
|
||||
if not size.within(cx.settings.max_scan_rows):
|
||||
raise _Skip("too_large", f"{table} 的行数超过整表统计的行数上限({cx.settings.max_scan_rows} 行),"
|
||||
"不取日期列的最小值和最大值")
|
||||
res = await cx.run(d.min_max_sql(d.table(meta, table), [d.quote(c) for c in columns]))
|
||||
# 文字存的日期(SQLite 的 TEXT、业务库里的 VARCHAR 日期)按字符串和有效区间比;日期类型的按日期字面量比
|
||||
texts = [catalog.column_family(_column_meta(meta, c)) != "time" for c in columns]
|
||||
res = await cx.run(d.min_max_sql(d.table(meta, table), [(d.column(meta, c), t) for c, t in zip(columns, texts)]))
|
||||
row = res.rows[0] if res.rows else []
|
||||
invalid: dict[str, int] = {}
|
||||
for i, column in enumerate(columns):
|
||||
low, high = (row[2 * i], row[2 * i + 1]) if len(row) >= 2 * i + 2 else (None, None)
|
||||
low, high, bad = (row[3 * i], row[3 * i + 1], row[3 * i + 2]) if len(row) >= 3 * i + 3 else (None, None, 0)
|
||||
invalid[column] = _as_int(bad) or 0
|
||||
if low is None or high is None or not (_DATE_TEXT.match(str(low)) and _DATE_TEXT.match(str(high))):
|
||||
continue
|
||||
result.date_ranges.append({"column": column, "min": str(low), "max": str(high)})
|
||||
found: dict[str, Any] = {"column": column, "min": str(low), "max": str(high)}
|
||||
if invalid[column]:
|
||||
found["invalid"] = invalid[column]
|
||||
result.date_ranges.append(found)
|
||||
if total != 1:
|
||||
return
|
||||
if not result.date_ranges:
|
||||
if invalid.get(columns[0]):
|
||||
raise _Skip("no_data", f"{table}.{columns[0]} 只有 {invalid[columns[0]]} 个无效日期({INVALID_DATES}),"
|
||||
"没有可识别的日期值")
|
||||
raise _Skip("no_data", f"{table}.{columns[0]} 没有可识别的日期值")
|
||||
found = result.date_ranges[0]
|
||||
dropped = (f"另有 {found['invalid']} 个无效日期({INVALID_DATES})未计入," if found.get("invalid") else "")
|
||||
note = (f"{NOTE_PREFIX}表里只有这一个日期类列,取值从 {str(found['min'])[:19]} 到 "
|
||||
f"{str(found['max'])[:19]},可作为业务日期的参考。")
|
||||
f"{str(found['max'])[:19]},{dropped}可作为业务日期的参考。")
|
||||
out.business_date = _DateOutcome(column=found["column"], low=found["min"], high=found["max"], note=note)
|
||||
|
||||
|
||||
@@ -1226,7 +1484,7 @@ def _relation_finding(o: _RelationOutcome, rel: dict[str, Any], *, status: str,
|
||||
elif status == "verified":
|
||||
summary = f"覆盖率 {_percent(o.coverage)}," + ("仍为有确证" if previous == "verified" else "升为有确证")
|
||||
elif o.coverage < catalog.PROFILE_VERIFY_COVERAGE:
|
||||
summary = f"抽样覆盖率 {_percent(o.coverage)},保持推断"
|
||||
summary = f"覆盖率 {_percent(o.coverage)},保持推断"
|
||||
else:
|
||||
summary = "被指向列未核实唯一,保持推断"
|
||||
return {"kind": "relation", "path": f"relations.{o.rid}", "target": o.target,
|
||||
@@ -1241,6 +1499,8 @@ def _codes_value(c: _CodesOutcome, item: dict[str, Any] | None) -> dict[str, str
|
||||
沿用它的含义;它写了、数据里没出现的取值也留着——码值是「可能的取值」,没出现不等于不存在。"""
|
||||
old = item.get("value") if isinstance(item, dict) and isinstance(item.get("value"), dict) else {}
|
||||
value = {key: str(old.get(key) or "") for key, _ in c.values}
|
||||
for key in c.declared:
|
||||
value.setdefault(key, str(old.get(key) or ""))
|
||||
for key, meaning in old.items():
|
||||
if key not in value and isinstance(meaning, str) and meaning.strip():
|
||||
value[key] = meaning
|
||||
@@ -1744,8 +2004,8 @@ def ensure_enabled(row: Any, settings: ProfileSettings | None = None) -> Profile
|
||||
|
||||
#: 有效行过滤核对的说法(写进这一项的 note;前面是模型给的依据)
|
||||
FILTER_CHECKED = "剖析核对:排除 {share} 的行(共 {rows} 行)"
|
||||
FILTER_SAMPLED = "剖析核对:按前 {rows} 行统计,排除 {share} 的行"
|
||||
FILTER_SAMPLED_EDGE = "剖析核对:按前 {rows} 行统计,{what},请人工核对"
|
||||
FILTER_SAMPLED = "剖析核对:按存储顺序取前 {rows} 行统计,排除 {share} 的行"
|
||||
FILTER_SAMPLED_EDGE = "剖析核对:按存储顺序取前 {rows} 行统计,{what},请人工核对"
|
||||
FILTER_UNCHECKED = "没有核对排除比例({why})"
|
||||
FILTER_NOT_ENABLED = "数据源没有开数据剖析"
|
||||
|
||||
|
||||
@@ -58,6 +58,9 @@ from app.db.models import CatalogDraftJob, CatalogProfileJob, CatalogProfileTask
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
#: 剖析任务占同源队(叫法登记在运行时里,见 core/job_runtime.register_lane_kind)
|
||||
runtime.register_lane_kind("profile", "剖析")
|
||||
|
||||
#: 一个任务最多几张表
|
||||
MAX_TABLES = 500
|
||||
#: 每个源保留几个结束了的剖析任务(某张表最近一次剖析所在的任务另外保留)
|
||||
|
||||
@@ -0,0 +1,181 @@
|
||||
"""目录视图:运行和助手读数据目录统一经这一个接缝(docs/design/next-round/01-效果度量与回归评测.md 做法 5,总方案 X7)。
|
||||
|
||||
**为什么要接缝。** 目录在四处被读:运行时的 db_schema / db_query 工具(目录同时冻结进表结构快照)、助手查看(look 表详情)、
|
||||
自查用的 SQL 检查器、双盲复算(交给盲算的说明、比较两条 SQL 用的中文名)。以前各读各的、没有统一的开关,「开目录和关
|
||||
目录」「只用已核实的项」「起草模型 A 的目录和 B 的」这些对照都做不了,评测期间目录被起草或剖析改了也不知道。现在四处都
|
||||
经这里取,一处定读哪一份;lint 测试(tests/test_catalog_view.py)不许这几个模块绕过它直接 read_catalog、直接查 CatalogNote。
|
||||
|
||||
**视图**(CatalogView):
|
||||
- mode:live 当前目录(缺省,行为和以前一样);snapshot 一份冻结的目录快照(工件 id 在 snapshot,内容见 SNAPSHOT_KIND);
|
||||
off 不给目录。
|
||||
- statuses:all 全部(驳回的照旧不给模型看);reviewed 只要已核实(verified)和已确认(confirmed)的项,推断的
|
||||
(proposed)去掉——量的是模型起草、还没审过的那些项是帮忙还是添乱。
|
||||
|
||||
**视图从哪来。** 运行从 Run.options['catalog_view'] 读(launch_run 收 options,发起时定下、随 run.started 封存,审批恢复、
|
||||
重启续跑都沿用——用上下文变量传的话恢复之后就丢了);生成过程(问数据的 answer_ops、评测的作答)从参数读。都不给就是 live。
|
||||
冻结进表结构快照时,不是缺省视图的在工件引用的 meta 里记下视图(describe),证据面板据此写得出「本次运行目录:关」。
|
||||
|
||||
**快照的内容**(第 2 波的评测题集包写,这里只定读法):kind=eval_catalog_snapshot 的内容寻址工件,
|
||||
{"sources": {源 id: {"tables": {表名: {"version": 版本, "notes": 目录}}}}}。快照里没有这个源就是这个源没有目录;
|
||||
快照工件不在了照实报错(CatalogSnapshotMissing),不悄悄退回当前目录——那样对照就白做了。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from collections.abc import Iterable, Mapping
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
from app.data import catalog
|
||||
from app.data.catalog import CatalogEntry
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
#: Run.options 里放视图的键
|
||||
OPTION_KEY = "catalog_view"
|
||||
MODES = ("live", "snapshot", "off")
|
||||
#: 各状态集合留下哪些项的状态(all 不筛:驳回的由各处照旧经 visible_notes 去掉)
|
||||
STATUS_SETS: dict[str, tuple[str, ...] | None] = {"all": None, "reviewed": ("verified", "confirmed")}
|
||||
#: 目录快照工件的 kind(细案 01 做法 6)
|
||||
SNAPSHOT_KIND = "eval_catalog_snapshot"
|
||||
|
||||
|
||||
class CatalogViewInvalid(ValueError):
|
||||
"""视图写得不对(服务端代码写错了才会有:客户端写不了 Run.options)。"""
|
||||
|
||||
|
||||
class CatalogSnapshotMissing(RuntimeError):
|
||||
"""视图指定的目录快照工件不在了、或者内容认不出。"""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CatalogView:
|
||||
mode: str = "live"
|
||||
statuses: str = "all"
|
||||
#: mode=snapshot 时快照工件的 id
|
||||
snapshot: str | None = None
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if self.mode not in MODES:
|
||||
raise CatalogViewInvalid(f"目录视图的 mode 只能是 {'、'.join(MODES)},当前为「{self.mode}」")
|
||||
if self.statuses not in STATUS_SETS:
|
||||
raise CatalogViewInvalid(f"目录视图的 statuses 只能是 {'、'.join(STATUS_SETS)},当前为「{self.statuses}」")
|
||||
if (self.mode == "snapshot") != bool(self.snapshot):
|
||||
raise CatalogViewInvalid("目录视图只有 mode=snapshot 时带快照工件 id,而且必须带")
|
||||
|
||||
@property
|
||||
def is_default(self) -> bool:
|
||||
return self.mode == "live" and self.statuses == "all"
|
||||
|
||||
def as_option(self) -> dict[str, Any]:
|
||||
"""写进 Run.options['catalog_view'] 的形状。"""
|
||||
return {"mode": self.mode, "statuses": self.statuses, **({"snapshot": self.snapshot} if self.snapshot else {})}
|
||||
|
||||
@classmethod
|
||||
def parse(cls, raw: Any) -> CatalogView:
|
||||
"""Run.options['catalog_view']、生成参数 → 视图。None 是缺省(live);写得不对抛 CatalogViewInvalid。"""
|
||||
if raw is None:
|
||||
return LIVE
|
||||
if isinstance(raw, CatalogView):
|
||||
return raw
|
||||
if not isinstance(raw, Mapping):
|
||||
raise CatalogViewInvalid(f"目录视图需要是一个对象,当前为「{raw}」")
|
||||
unknown = set(raw) - {"mode", "statuses", "snapshot"}
|
||||
if unknown:
|
||||
raise CatalogViewInvalid(f"目录视图里不认识的键:{'、'.join(sorted(unknown))}")
|
||||
return cls(mode=str(raw.get("mode") or "live"), statuses=str(raw.get("statuses") or "all"),
|
||||
snapshot=str(raw["snapshot"]) if raw.get("snapshot") else None)
|
||||
|
||||
|
||||
LIVE = CatalogView()
|
||||
|
||||
|
||||
def of_options(options: Mapping[str, Any] | None) -> CatalogView:
|
||||
"""一次运行的视图(Run.options 里的 catalog_view)。没有就是 live。"""
|
||||
return CatalogView.parse((options or {}).get(OPTION_KEY))
|
||||
|
||||
|
||||
def of_run(run_ctx: Any) -> CatalogView:
|
||||
"""运行里的节点构造 ToolContext 时用:RunContext.extra['run_options'] 里的视图(runner 每段执行都放这份)。"""
|
||||
from app.engine.runner import RUN_OPTIONS_KEY
|
||||
|
||||
return of_options((getattr(run_ctx, "extra", None) or {}).get(RUN_OPTIONS_KEY))
|
||||
|
||||
|
||||
def describe(view: CatalogView | None) -> dict[str, Any] | None:
|
||||
"""记进冻结快照 meta 的说法:缺省视图(live、全部)不记,别的照 as_option 记下。"""
|
||||
view = view or LIVE
|
||||
return None if view.is_default else view.as_option()
|
||||
|
||||
|
||||
def restrict(view: CatalogView | None, notes: Mapping[str, Any] | None) -> dict[str, Any]:
|
||||
"""一张表的目录在这个视图下留下什么(状态集合;驳回的项原样留着,见 catalog.restrict_notes)。"""
|
||||
allowed = STATUS_SETS[(view or LIVE).statuses]
|
||||
if allowed is None:
|
||||
return dict(notes or {})
|
||||
return catalog.restrict_notes(notes, allowed)
|
||||
|
||||
|
||||
def _restricted(view: CatalogView, entries: Mapping[str, CatalogEntry]) -> dict[str, CatalogEntry]:
|
||||
if STATUS_SETS[view.statuses] is None:
|
||||
return dict(entries)
|
||||
out: dict[str, CatalogEntry] = {}
|
||||
for table, entry in entries.items():
|
||||
if notes := restrict(view, entry.notes):
|
||||
out[table] = CatalogEntry(table=entry.table, notes=notes, version=entry.version,
|
||||
updated_by=entry.updated_by, updated_at=entry.updated_at)
|
||||
return out
|
||||
|
||||
|
||||
def _snapshot_tables(view: CatalogView, source_id: str) -> dict[str, CatalogEntry]:
|
||||
from app.core.artifact_store import load
|
||||
|
||||
try:
|
||||
content = load(view.snapshot or "")
|
||||
except ValueError as e: # 内容和哈希对不上
|
||||
raise CatalogSnapshotMissing(f"目录快照 {str(view.snapshot)[:12]}… 读不出:{e}") from e
|
||||
if not isinstance(content, dict) or not isinstance(content.get("sources"), dict):
|
||||
raise CatalogSnapshotMissing(f"目录快照 {str(view.snapshot)[:12]}… 不在了或内容认不出,这次不能按它读目录")
|
||||
tables = ((content["sources"].get(source_id) or {}).get("tables")) or {}
|
||||
out: dict[str, CatalogEntry] = {}
|
||||
for table, body in tables.items():
|
||||
if isinstance(body, Mapping) and isinstance(body.get("notes"), Mapping):
|
||||
out[str(table)] = CatalogEntry(table=str(table), notes=dict(body["notes"]),
|
||||
version=int(body.get("version") or 0))
|
||||
return out
|
||||
|
||||
|
||||
async def read_many(session: Any, view: CatalogView | None, source_ids: Iterable[str]) -> dict[str, dict[str, CatalogEntry]]:
|
||||
"""几个数据源在这个视图下的目录:{源 id: {表名: CatalogEntry}},没有目录的源不出现。live 一次查询读出。"""
|
||||
view = view or LIVE
|
||||
ids = [i for i in dict.fromkeys(source_ids) if i]
|
||||
if view.mode == "off" or not ids:
|
||||
return {}
|
||||
if view.mode == "snapshot":
|
||||
raw = {sid: found for sid in ids if (found := _snapshot_tables(view, sid))}
|
||||
else:
|
||||
raw = await catalog.read_catalogs(session, ids)
|
||||
return {sid: kept for sid, entries in raw.items() if (kept := _restricted(view, entries))}
|
||||
|
||||
|
||||
async def read(view: CatalogView | None, source: Any) -> dict[str, CatalogEntry]:
|
||||
"""一个数据源在这个视图下的目录:{表名: CatalogEntry}。用自己的短会话读(调用方的会话里已经加载的对象不会因为回滚
|
||||
过期)。live 读不到(库出错)当没有目录、记日志:目录是附加说明,不能挡住查询;快照不在了照实抛 CatalogSnapshotMissing。"""
|
||||
from app.db.base import SessionLocal
|
||||
|
||||
view = view or LIVE
|
||||
source_id = getattr(source, "id", None)
|
||||
if view.mode == "off" or not source_id:
|
||||
return {}
|
||||
if view.mode == "snapshot":
|
||||
return _restricted(view, _snapshot_tables(view, source_id))
|
||||
try:
|
||||
async with SessionLocal() as own:
|
||||
return (await read_many(own, view, [source_id])).get(source_id, {})
|
||||
except Exception: # noqa: BLE001
|
||||
logger.exception("读取数据源 %s 的数据目录失败,本次不附目录", getattr(source, "name", source_id))
|
||||
return {}
|
||||
|
||||
|
||||
__all__ = ["CatalogSnapshotMissing", "CatalogView", "CatalogViewInvalid", "LIVE", "MODES", "OPTION_KEY", "SNAPSHOT_KIND",
|
||||
"STATUS_SETS", "describe", "of_options", "of_run", "read", "read_many", "restrict"]
|
||||
@@ -0,0 +1,206 @@
|
||||
"""CSV 走配方(Excel 期 5 第 4 项):把 CSV 适配成配方读取层认的样子——一张可见工作表的扫描结果(WorkbookScan)
|
||||
和网格(Grid / 逐行流),之后的执行器、核对、说明、溯源、按期累积全部复用,不另写一套 CSV 的解析(否则会和配方
|
||||
语言分叉,违背「一种说法」)。
|
||||
|
||||
**怎么认出是 CSV。** 和简单导入(tabular._CSV)一样按扩展名定:只有 .csv / .tsv 按 CSV 读,并且内容得是文字
|
||||
(不是 zip、不是旧版 .xls 的 OLE 文件头、前 64 KB 里没有 NUL 字节)。以前只看内容,企业系统常见的「HTML 表格冒充
|
||||
.xls」、误传的文本文件都会被拆成一张叫「CSV」的工作表去起草、核对,不再像以前那样明确拒收;现在别的扩展名照旧交给
|
||||
xlsx_scan 拒收,内容是网页或 XML 的另给一句能照着做的话。编码识别复用 tabular 读 CSV 的那一段(UTF-8 / GB18030 /
|
||||
带 BOM 的 UTF-16);.tsv 一律按制表符分隔(同 tabular),.csv 嗅探逗号、分号、制表符、竖线。扫描时定下的分隔符
|
||||
记进工作表的 path(CSV_PATH 后面跟分隔符),之后读格子用同一个,不再各嗅各的。
|
||||
|
||||
**坐标沿用「工作表!A1」。** 细案原写「格子坐标记成 R{行}C{列}」,这里没这么做(见交付说明的偏离项):坐标在执行器、
|
||||
核对、修复按钮、框选、溯源、裁判摘录、前端网格里一共有十来处解析和拼写,引入第二种写法就要处处认两种,漏一处就是
|
||||
着色错位、下钻找不到格子;期 1 读 CSV 时(tabular._shape_reasons)用的也是 A1。CSV 没有工作表,名字固定为
|
||||
CSV_SHEET:每月文件名不同(orders_2026-08.csv、orders_2026-09.csv),配方按工作表名认,名字必须稳定。行号是第几条
|
||||
记录(引号里的换行不另起一行),列号是第几个字段,和用 Excel 打开这个 CSV 看到的 A1 坐标一致。
|
||||
|
||||
**格子的值。** CSV 里只有文字。只有写法毫无歧义的才转成数(-12、80.5,不超过 15 位有效数字),其余一律原样留作文字:
|
||||
「1,234」(千分位,交给配方的 text_number 决定)、「007」(前导零是编码的一部分)、「+5」「1e5」「12%」、日期写法
|
||||
(执行器按 DATE 列解析文字日期,起草按文字认日期)、「=SUM(…)」(CSV 里不是公式,Excel 打开时才会当公式算)。
|
||||
空字段、只有空白的字段是空格(和 xlsx_scan 的口径一致)。没有公式、合并单元格、隐藏行列。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import io
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Iterator
|
||||
|
||||
from app.data.recipe_types import GridCell
|
||||
from app.data.xlsx_scan import Bounds, SheetScan, UnsupportedTable, WorkbookScan
|
||||
|
||||
__all__ = ["CSV_PATH", "CSV_SHEET", "CSV_SUFFIXES", "MAX_COLS", "MAX_ROWS", "CsvTable", "cell_value", "is_csv_sheet",
|
||||
"looks_like_csv", "looks_like_markup", "parse", "rows", "scan", "scan_any", "sheet_delimiter"]
|
||||
|
||||
#: CSV 的「工作表名」:配方按它认(SheetMatch.name),每月的文件名不同也认得出
|
||||
CSV_SHEET = "CSV"
|
||||
#: SheetScan.path 的开头:读取层据此知道这张「工作表」要从 CSV 里读,不去压缩包里找 XML。后面跟着扫描时定下的分隔符
|
||||
#: ("csv:,"、"csv:\t");只有「csv:」的(这一版之前的暂存区)读的时候再嗅一次
|
||||
CSV_PATH = "csv:"
|
||||
#: 按 CSV 读的扩展名,和简单导入(tabular._CSV)一致
|
||||
CSV_SUFFIXES = (".csv", ".tsv")
|
||||
#: 嗅探认的分隔符
|
||||
_DELIMITERS = ",;\t|"
|
||||
#: 内容是网页或 XML(「HTML 表格冒充 .xls」、Excel 2003 的 XML 表格)时的说法
|
||||
MARKUP_MESSAGE = ("无法打开该文件:它其实是网页或 XML 格式(有的系统导出的「Excel」文件是这样),不是 Excel 工作簿。"
|
||||
"请用 Excel 打开后另存为 .xlsx 或 .csv 再上传")
|
||||
NOT_TEXT_MESSAGE = ("无法按 CSV 读取这个文件:扩展名是 CSV,内容却不是文字(可能是图片、PDF 之类的文件改了扩展名)。"
|
||||
"请确认上传的文件,或在 Excel 中另存为 .csv 后重新上传")
|
||||
#: 和 Excel 的上限一致:超过的 CSV 用 Excel 打开也装不下,交给模型的格子坐标也写不出来
|
||||
MAX_ROWS = 1_048_576
|
||||
MAX_COLS = 16_384
|
||||
#: 嗅探看多少字节
|
||||
_SNIFF = 64 * 1024
|
||||
#: 写法毫无歧义的整数、小数(不带正号、不带前导零、不用科学计数法)
|
||||
_INT = re.compile(r"-?(?:0|[1-9][0-9]*)")
|
||||
_DEC = re.compile(r"-?(?:0|[1-9][0-9]*)\.[0-9]+")
|
||||
#: 超过这么多位有效数字的不转成数:转成浮点会丢位(19 位卡号撞值),留作文字
|
||||
_DIGITS_MAX = 15
|
||||
|
||||
|
||||
def looks_like_csv(raw: bytes) -> bool:
|
||||
"""这份原件按 CSV 读:不是 zip、不是 OLE(旧版 .xls)、开头没有 NUL 字节、不是空文件。带 BOM 的 UTF-16 例外
|
||||
(每个字符都带 NUL,Excel「Unicode 文本」另存出来的就是它)。"""
|
||||
if not raw or raw[:4] == b"PK\x03\x04" or raw[:8] == b"\xd0\xcf\x11\xe0\xa1\xb1\x1a\xe1":
|
||||
return False
|
||||
if raw[:2] in (b"\xff\xfe", b"\xfe\xff"):
|
||||
return True
|
||||
return b"\x00" not in raw[:_SNIFF]
|
||||
|
||||
|
||||
def looks_like_markup(raw: bytes) -> bool:
|
||||
"""内容是网页或 XML:去掉 BOM 和开头的空白后以「<」开头。真的工作簿以 zip(PK)或 OLE 文件头开头,不会是这样。"""
|
||||
head = raw[:4096]
|
||||
if head[:2] in (b"\xff\xfe", b"\xfe\xff"):
|
||||
try:
|
||||
text = head.decode("utf-16", errors="ignore")
|
||||
except UnicodeDecodeError:
|
||||
return False
|
||||
else:
|
||||
text = head.decode("latin-1")
|
||||
return text.lstrip("\ufeff\xef\xbb\xbf \t\r\n").startswith("<")
|
||||
|
||||
|
||||
def _suffix(filename: str | None) -> str:
|
||||
name = str(filename or "").rsplit("/", 1)[-1]
|
||||
return "." + name.rsplit(".", 1)[-1].lower() if "." in name else ""
|
||||
|
||||
|
||||
def cell_value(text: str) -> Any:
|
||||
"""CSV 的一个字段 → 格子的值:空 / 只有空白是 None(空格),写法毫无歧义的整数、小数转成数,其余原样(文字)。"""
|
||||
t = text.strip()
|
||||
if not t:
|
||||
return None
|
||||
digits = len(t.lstrip("-").replace(".", "").lstrip("0")) or 1
|
||||
if digits <= _DIGITS_MAX:
|
||||
if _INT.fullmatch(t):
|
||||
return int(t)
|
||||
if _DEC.fullmatch(t):
|
||||
return float(t)
|
||||
return text
|
||||
|
||||
|
||||
@dataclass
|
||||
class CsvTable:
|
||||
"""解析好的 CSV:records[i] 是第 i + 1 条记录的字段(原文)。"""
|
||||
|
||||
records: list[list[str]]
|
||||
delimiter: str
|
||||
|
||||
|
||||
def parse(raw: bytes, delimiter: str | None = None) -> CsvTable:
|
||||
"""原件 → 记录。编码照 tabular 读 CSV 的做法;delimiter 不给时嗅探(同 tabular)。字段太长、引号不配对读不了时抛
|
||||
UnsupportedTable。"""
|
||||
from app.data import tabular
|
||||
|
||||
text = tabular._decode_csv(raw)
|
||||
if delimiter is None:
|
||||
try:
|
||||
delimiter = csv.Sniffer().sniff(text[:8192], delimiters=_DELIMITERS).delimiter
|
||||
except csv.Error:
|
||||
delimiter = "," # 嗅不出来就按最常见的来
|
||||
records: list[list[str]] = []
|
||||
try:
|
||||
for record in csv.reader(io.StringIO(text, newline=""), delimiter=delimiter):
|
||||
if len(records) >= MAX_ROWS:
|
||||
raise UnsupportedTable(f"CSV 文件超过 {MAX_ROWS:,} 行,超出可以导入的上限")
|
||||
if len(record) > MAX_COLS:
|
||||
raise UnsupportedTable(f"CSV 文件第 {len(records) + 1} 行有 {len(record):,} 列,超过 {MAX_COLS:,} 列的上限")
|
||||
records.append(record)
|
||||
except csv.Error as e:
|
||||
raise UnsupportedTable(f"无法按 CSV 读取这个文件:第 {len(records) + 1} 行的写法不正确({e})。"
|
||||
"请在 Excel 中重新另存为 CSV 后上传") from e
|
||||
return CsvTable(records=records, delimiter=delimiter)
|
||||
|
||||
|
||||
def _cells(table: CsvTable) -> Iterator[tuple[int, list[tuple[int, GridCell]]]]:
|
||||
for r, record in enumerate(table.records, 1):
|
||||
out = [(c, GridCell(v)) for c, field in enumerate(record, 1) if (v := cell_value(field)) is not None]
|
||||
if out:
|
||||
yield r, out
|
||||
|
||||
|
||||
def is_csv_sheet(sheet: Any) -> bool:
|
||||
"""这张「工作表」是 CSV 适配出来的(path 以 CSV_PATH 开头)。"""
|
||||
return str(getattr(sheet, "path", "") or "").startswith(CSV_PATH)
|
||||
|
||||
|
||||
def sheet_delimiter(sheet: Any) -> str | None:
|
||||
"""扫描时定下的分隔符(记在 path 里);这一版之前的暂存区没记,返回 None(读的时候再嗅)。"""
|
||||
tail = str(getattr(sheet, "path", "") or "")[len(CSV_PATH):]
|
||||
return tail if len(tail) == 1 and tail in _DELIMITERS else None
|
||||
|
||||
|
||||
def scan(raw: bytes, *, delimiter: str | None = None) -> WorkbookScan:
|
||||
"""CSV 原件 → 扫描结果:一张可见工作表(名字 CSV_SHEET、路径 CSV_PATH 加分隔符),边界、非空格数、逐行非空格数的
|
||||
游程。没有合并区、隐藏行列、公式、错误值、离群格(CSV 里每一格都是写出来的,没有残留格式)。"""
|
||||
table = parse(raw, delimiter)
|
||||
sheet = SheetScan(name=CSV_SHEET, state="visible", path=CSV_PATH + table.delimiter)
|
||||
lo_r = lo_c = hi_r = hi_c = 0
|
||||
runs: list[tuple[int, int, int]] = []
|
||||
for r, cells in _cells(table):
|
||||
n = len(cells)
|
||||
sheet.nonempty += n
|
||||
first, last = cells[0][0], cells[-1][0]
|
||||
if not lo_r:
|
||||
lo_r, lo_c, hi_c = r, first, last
|
||||
lo_c, hi_c, hi_r = min(lo_c, first), max(hi_c, last), r
|
||||
if runs and runs[-1][1] == r - 1 and runs[-1][2] == n:
|
||||
runs[-1] = (runs[-1][0], r, n)
|
||||
else:
|
||||
runs.append((r, r, n))
|
||||
if lo_r:
|
||||
sheet.bounds = Bounds(lo_r, lo_c, hi_r, hi_c)
|
||||
sheet.dimension = sheet.bounds.a1()
|
||||
sheet.row_runs = runs
|
||||
return WorkbookScan(sheets=[sheet], format="csv")
|
||||
|
||||
|
||||
def scan_any(raw: bytes, filename: str | None) -> WorkbookScan:
|
||||
"""配方导入的扫描入口:.csv / .tsv 走这里的适配(.tsv 按制表符分隔),其余照旧交给 xlsx_scan(读不了照旧拒收,
|
||||
文案不变);内容是网页或 XML 的另给一句能照着做的话(MARKUP_MESSAGE)。扩展名是 CSV、内容却是工作簿(xlsx 改了
|
||||
扩展名)的照工作簿读。"""
|
||||
from app.data import xlsx_scan
|
||||
|
||||
suffix = _suffix(filename)
|
||||
if suffix in CSV_SUFFIXES:
|
||||
if looks_like_csv(raw):
|
||||
return scan(raw, delimiter="\t" if suffix == ".tsv" else None)
|
||||
if raw[:4] != b"PK\x03\x04":
|
||||
raise UnsupportedTable(NOT_TEXT_MESSAGE)
|
||||
elif looks_like_markup(raw):
|
||||
raise UnsupportedTable(MARKUP_MESSAGE)
|
||||
return xlsx_scan.scan(raw)
|
||||
|
||||
|
||||
def rows(table: CsvTable, *, min_row: int, max_row: int, min_col: int,
|
||||
max_col: int) -> Iterator[tuple[int, list[tuple[int, GridCell]]]]:
|
||||
"""按行流式给一个矩形里的非空格:(行号, [(列号, 格)…]),口径同 xlsx_cells.sheet_rows(整行为空的不给)。"""
|
||||
for r in range(max(min_row, 1), min(max_row, len(table.records)) + 1):
|
||||
record = table.records[r - 1]
|
||||
out = [(c, GridCell(v)) for c in range(max(min_col, 1), min(max_col, len(record)) + 1)
|
||||
if (v := cell_value(record[c - 1])) is not None]
|
||||
if out:
|
||||
yield r, out
|
||||
@@ -165,6 +165,89 @@ def _unique_groups(constraints: list[dict[str, Any]], indexes: list[dict[str, An
|
||||
return groups
|
||||
|
||||
|
||||
def _storage_type(col_type: Any) -> Any:
|
||||
"""PostgreSQL 的域(DOMAIN)换成它底下的类型(域套域的一路往下找);别的类型原样返回。"""
|
||||
from sqlalchemy.dialects.postgresql import DOMAIN
|
||||
|
||||
seen = 0
|
||||
while isinstance(col_type, DOMAIN) and getattr(col_type, "data_type", None) is not None and seen < 8:
|
||||
col_type, seen = col_type.data_type, seen + 1
|
||||
return col_type
|
||||
|
||||
|
||||
def type_text(col_type: Any, dialect: Any) -> str:
|
||||
"""反射到的列类型 → 存进表结构的写法:按这个数据源的方言编译的完整写法(VARCHAR2(20 CHAR)、NUMBER(10, 0)、
|
||||
ENUM('paid','refund')、TIMESTAMP WITH TIME ZONE),编译不了的退回 str()。PostgreSQL 的域(DOMAIN)记它底下的类型:
|
||||
域编译出来只是域名(settlement_amount),看不出是数字还是文字,名字里的词(set、UPDATE 里的 DATE)还会让类型大类、
|
||||
日期列、码值候选认错;模型照着写 SQL 也要知道它存的是什么。
|
||||
|
||||
以前存的是 str(类型),即 SQLAlchemy 的通用写法:Oracle 的 VARCHAR2、NUMBER 丢了长度和精度,MySQL 的 ENUM 丢了
|
||||
取值列表,PostgreSQL 的 TIMESTAMP 丢了时区。类型写不全,表结构前后比对就看不出类型变了,ENUM 状态列也进不了
|
||||
码值候选。这一改会让新探查的表结构快照哈希变化(提示词里的类型写法变长),旧运行封存的快照不受影响。"""
|
||||
if col_type is None:
|
||||
return ""
|
||||
col_type = _storage_type(col_type)
|
||||
try:
|
||||
compiled = str(col_type.compile(dialect=dialect))
|
||||
except Exception: # noqa: BLE001 - 编译不了(NullType 等)就用通用写法,不让一列拖垮整次探查
|
||||
return str(col_type)
|
||||
if getattr(dialect, "name", "") == "oracle" and compiled.startswith("VARCHAR("):
|
||||
# SQLAlchemy 把 Oracle 的 VARCHAR2 反射成通用的 VARCHAR,按 Oracle 方言编译出来是 VARCHAR(10 CHAR)(容器集成
|
||||
# 测试实测)。Oracle 的数据字典里只有 VARCHAR2(建表写 VARCHAR 也存成 VARCHAR2),照数据字典写
|
||||
compiled = "VARCHAR2" + compiled[len("VARCHAR"):]
|
||||
return compiled
|
||||
|
||||
|
||||
def _enum_values(col_type: Any) -> list[str] | None:
|
||||
"""ENUM(MySQL、PostgreSQL 的枚举,都是 sqltypes.Enum 的子类)的取值列表,MySQL SET 的成员列表;别的类型 None。
|
||||
套在域(DOMAIN)里的枚举也算(同 type_text)。"""
|
||||
from sqlalchemy.dialects.mysql import SET
|
||||
from sqlalchemy.types import Enum
|
||||
|
||||
col_type = _storage_type(col_type)
|
||||
if isinstance(col_type, Enum):
|
||||
values = list(col_type.enums or [])
|
||||
elif isinstance(col_type, SET):
|
||||
values = list(col_type.values or [])
|
||||
else:
|
||||
return None
|
||||
return [str(v) for v in values]
|
||||
|
||||
|
||||
def _forced_quote(name: Any) -> bool:
|
||||
"""名字必须原样加引号:Oracle 反射回来的全小写对象(建表时写了 "visits")是 quoted_name(quote=True)。
|
||||
它存进 JSON 就成了普通字符串,剖析拼 SQL 时不加引号,Oracle 把它转成大写 VISITS,查的是另一个不存在的对象。"""
|
||||
return getattr(name, "quote", None) is True
|
||||
|
||||
|
||||
def column_entry(col: Mapping[str, Any], dialect: Any) -> dict[str, Any]:
|
||||
"""反射到的一列 → 表结构里的一列:名字、按方言编译的类型、可空、注释;ENUM / SET 另记取值列表(enum_values),
|
||||
要强制加引号的名字另记 quote_name。只在有的时候写这两个键:其余列和以前一字不差。"""
|
||||
entry: dict[str, Any] = {
|
||||
"name": str(col["name"]),
|
||||
"type": type_text(col.get("type"), dialect),
|
||||
"nullable": bool(col.get("nullable", True)),
|
||||
"comment": col.get("comment") or None,
|
||||
}
|
||||
values = _enum_values(col.get("type"))
|
||||
if values:
|
||||
entry["enum_values"] = values
|
||||
if _forced_quote(col.get("name")):
|
||||
entry["quote_name"] = True
|
||||
return entry
|
||||
|
||||
|
||||
def index_first_columns(indexes: list[dict[str, Any]], pk: list[str]) -> list[str]:
|
||||
"""哪些列打头有索引:各索引(含主键)的第一列,去重、按名字排序。取值观察看大表的最新业务日期、按日行数时,
|
||||
只在业务日期列打头有索引时才发窗口查询(按它走索引,不扫全表)。表达式索引(列名是 None)不算。"""
|
||||
first = {str(pk[0])} if pk else set()
|
||||
for ix in indexes:
|
||||
cols = ix.get("column_names") or []
|
||||
if cols and cols[0] is not None:
|
||||
first.add(str(cols[0]))
|
||||
return sorted(first)
|
||||
|
||||
|
||||
async def introspect(source: Any, *, schema: str | None = None) -> dict[str, Any]:
|
||||
"""探查整个库的结构,结果直接存进 DataSource.schema_cache。
|
||||
|
||||
@@ -266,24 +349,20 @@ async def introspect(source: Any, *, schema: str | None = None) -> dict[str, Any
|
||||
# 只读账号名下没有这些对象,全靠跨 schema 授权
|
||||
"qualified": f"{target_schema}.{name}" if target_schema else name,
|
||||
"schema": target_schema,
|
||||
"columns": [
|
||||
{
|
||||
"name": c["name"],
|
||||
"type": str(c.get("type") or ""),
|
||||
"nullable": bool(c.get("nullable", True)),
|
||||
"comment": c.get("comment") or None,
|
||||
}
|
||||
for c in columns
|
||||
],
|
||||
"columns": [column_entry(c, sync_conn.dialect) for c in columns],
|
||||
"primary_key": pk,
|
||||
"comment": comment,
|
||||
"is_view": name in views,
|
||||
}
|
||||
if _forced_quote(name):
|
||||
tables[name]["quote_name"] = True
|
||||
if name in fk_ok:
|
||||
tables[name]["foreign_keys"] = _foreign_keys(fk_by.get(key) or [], target_schema, default_schema)
|
||||
if name in uq_ok:
|
||||
tables[name]["unique"] = _unique_groups(uq_by.get(key) or [],
|
||||
(ix_by.get(key) or []) if name in ix_ok else [], pk)
|
||||
if name in ix_ok:
|
||||
tables[name]["index_first_columns"] = index_first_columns(ix_by.get(key) or [], pk)
|
||||
payload: dict[str, Any] = {"tables": tables, "truncated": truncated,
|
||||
"total": len(names) + len(views)}
|
||||
if failures and not tables:
|
||||
@@ -471,6 +550,10 @@ def describe_table(source: Any, table: str, *, catalog: Mapping[str, Mapping[str
|
||||
marks.append("非空")
|
||||
if col.get("comment"):
|
||||
marks.append(col["comment"])
|
||||
values = col.get("enum_values")
|
||||
if values and "(" not in str(col.get("type") or ""):
|
||||
# PostgreSQL 的枚举类型写法只有类型名(order_status):取值列表另外写出来,模型才知道能写哪些值
|
||||
marks.append("取值:" + "、".join(str(v) for v in values[:30]) + ("…" if len(values) > 30 else ""))
|
||||
suffix = f" {'、'.join(marks)}" if marks else ""
|
||||
lines.append(f" {col['name']} {col['type']}{suffix}")
|
||||
if catalog:
|
||||
|
||||
@@ -43,6 +43,9 @@ from app.data.recipe_parsers import (
|
||||
text_label,
|
||||
)
|
||||
from app.data.recipe_types import (
|
||||
LIST_PERIOD_END,
|
||||
LIST_PERIOD_START,
|
||||
LIST_SUBTOTAL_DIM,
|
||||
LIST_TOTAL_DIM,
|
||||
ColumnOut,
|
||||
CrosstabBlock,
|
||||
@@ -271,6 +274,8 @@ _FIELD_LABEL = {
|
||||
"parts": "组成列", "claims": "认领", "a": "一方", "b": "另一方", "reason": "理由",
|
||||
"ignore_rows": "按行标签忽略的行", "ignore_columns": "按表头忽略的列", "ignore_outside": "按同一行文字忽略的数字",
|
||||
"label": "标签", "anchor": "同一行的文字",
|
||||
"subtotals": "小计行", "no_subtotals": "没有小计行", "scope": "小计管哪些明细", "group_column": "分组的列", "period": "每行属于哪一期",
|
||||
"column": "日期列", "check": "核对方式", "synthesize": "加统计期列",
|
||||
}
|
||||
|
||||
|
||||
@@ -608,15 +613,25 @@ class _Checker:
|
||||
for si, bi, block in self.lists:
|
||||
base = ("sheets", si, "blocks", bi)
|
||||
cols = [(c.name, base + ("columns", n, "name")) for n, c in enumerate(block.columns)]
|
||||
if block.period is not None and block.period.synthesize:
|
||||
# 期 5:加上的统计期起止两列也是这张表的列,和表头推出的列名不能撞
|
||||
pp = base + ("period", "synthesize")
|
||||
cols += [(LIST_PERIOD_START, pp), (LIST_PERIOD_END, pp)]
|
||||
out.append(_Writer(block.table, base + ("table",), "data", cols, origin=_pointer(base)))
|
||||
numeric = [(c.name, base + ("columns", n, "name")) for n, c in enumerate(block.columns) if c.type in _NUMERIC]
|
||||
total = block.rows.total_row
|
||||
if total is not None and total.keep_as:
|
||||
tp = base + ("rows", "total_row", "keep_as")
|
||||
tcols = [(LIST_TOTAL_DIM, tp)]
|
||||
tcols += [(c.name, base + ("columns", n, "name"))
|
||||
for n, c in enumerate(block.columns) if c.type in _NUMERIC]
|
||||
out.append(_Writer(total.keep_as, tp, "reported_total", tcols,
|
||||
out.append(_Writer(total.keep_as, tp, "reported_total", [(LIST_TOTAL_DIM, tp)] + numeric,
|
||||
origin=_pointer(base + ("rows", "total_row"))))
|
||||
sub = block.rows.subtotals
|
||||
if sub is not None and sub.keep_as:
|
||||
sp = base + ("rows", "subtotals", "keep_as")
|
||||
scols = [(LIST_SUBTOTAL_DIM, sp)]
|
||||
if sub.scope == "group_column" and sub.group_column:
|
||||
scols.append((sub.group_column, base + ("rows", "subtotals", "group_column")))
|
||||
out.append(_Writer(sub.keep_as, sp, "reported_total", scols + numeric,
|
||||
origin=_pointer(base + ("rows", "subtotals"))))
|
||||
return out
|
||||
|
||||
# ---- 8. 名字 ----
|
||||
@@ -680,6 +695,10 @@ class _Checker:
|
||||
if sheet.context:
|
||||
continue
|
||||
for bi, block in enumerate(sheet.blocks):
|
||||
if isinstance(block, ListBlock) and block.period is not None:
|
||||
# 期 5:每行属于哪一期要拿统计期来核对(按日期列)或填进去(加起止两列)
|
||||
self.add(("sheets", si, "blocks", bi, "period"), "covers_without_context",
|
||||
"要按统计期核对或填写每一行的统计期,但这个工作表没有配置统计期")
|
||||
if not isinstance(block, CrosstabBlock):
|
||||
continue
|
||||
ap = ("sheets", si, "blocks", bi, "axis")
|
||||
@@ -873,6 +892,10 @@ class _Checker:
|
||||
if total is not None and any(ch.isdigit() for ch in total.pick):
|
||||
self.add(("sheets", si, "blocks", bi, "rows", "total_row", "pick"), "period_literal",
|
||||
f"合计行的开头词「{total.pick}」不能含数字(配方里不能写死年份或期次)")
|
||||
sub = block.rows.subtotals
|
||||
if sub is not None and any(ch.isdigit() for ch in sub.pick):
|
||||
self.add(("sheets", si, "blocks", bi, "rows", "subtotals", "pick"), "period_literal",
|
||||
f"小计行的开头词「{sub.pick}」不能含数字(配方里不能写死年份或期次)")
|
||||
|
||||
# ---- 4、9、13. 单位:键是本表列名,值在词表里,与原表标签、表头里的单位一致 ----
|
||||
|
||||
@@ -902,10 +925,13 @@ class _Checker:
|
||||
self.unit_against(seg.keep_as.table, seg.keep_as.value, target.locate.title, "分段标题")
|
||||
for _si, _bi, block in self.lists:
|
||||
total = block.rows.total_row
|
||||
sub = block.rows.subtotals
|
||||
for c in block.columns:
|
||||
self.unit_against(block.table, c.name, c.header, "表头")
|
||||
if total is not None and total.keep_as and c.type in _NUMERIC:
|
||||
self.unit_against(total.keep_as, c.name, c.header, "表头")
|
||||
if sub is not None and sub.keep_as and c.type in _NUMERIC:
|
||||
self.unit_against(sub.keep_as, c.name, c.header, "表头")
|
||||
|
||||
def after_target(self, si: int, bi: int, seg: DerivedSegment) -> DimensionSegment | None:
|
||||
"""合计分段紧跟的那个 dimension 分段(同一块里按 id 找);找不到或不是 dimension 段时已报 derived_after_invalid。"""
|
||||
@@ -998,6 +1024,8 @@ class _Checker:
|
||||
if total is not None and total.label_column not in names:
|
||||
self.add(base + ("rows", "total_row", "label_column"), "column_unknown",
|
||||
f"合计标签所在的列「{total.label_column}」不在这个列表的列里{self._near(total.label_column, names)}")
|
||||
self.check_subtotals(base, block, names)
|
||||
self.check_list_period(base, block, names)
|
||||
seen: dict[str, str] = {}
|
||||
for n, c in enumerate(block.columns):
|
||||
key = match_key(c.header)
|
||||
@@ -1006,6 +1034,53 @@ class _Checker:
|
||||
f"表头「{c.header}」与「{seen[key]}」是同一个表头")
|
||||
seen.setdefault(key, c.header)
|
||||
|
||||
def check_subtotals(self, base: tuple[Any, ...], block: ListBlock, names: list[str]) -> None:
|
||||
"""期 5:小计行的标签列、分组列要是这个列表的列;按组小计要给分组列;小计和合计的开头词不能互为前缀
|
||||
(「合计」「合计小计」:一行就分不清是小计还是合计);配了小计又声明没有小计(no_subtotals)是矛盾的。"""
|
||||
sub = block.rows.subtotals
|
||||
if sub is None:
|
||||
return
|
||||
sp = base + ("rows", "subtotals")
|
||||
if block.rows.no_subtotals:
|
||||
self.add(base + ("rows", "no_subtotals"), "subtotals_conflict",
|
||||
"配了小计行,又声明了这张列表没有小计行:二选一")
|
||||
if sub.label_column not in names:
|
||||
self.add(sp + ("label_column",), "column_unknown",
|
||||
f"小计标签所在的列「{sub.label_column}」不在这个列表的列里{self._near(sub.label_column, names)}")
|
||||
if sub.scope == "group_column":
|
||||
if not sub.group_column:
|
||||
self.add(sp + ("group_column",), "subtotal_group_missing", "按组小计时要指明按哪一列分组")
|
||||
elif sub.group_column not in names:
|
||||
self.add(sp + ("group_column",), "column_unknown",
|
||||
f"小计分组的列「{sub.group_column}」不在这个列表的列里{self._near(sub.group_column, names)}")
|
||||
elif sub.group_column:
|
||||
self.add(sp + ("group_column",), "subtotal_group_missing",
|
||||
"小计管「上一行小计之后的全部明细」时不按列分组,请去掉分组的列,或改为按组小计")
|
||||
total = block.rows.total_row
|
||||
if total is not None:
|
||||
a, b = match_key(sub.pick), match_key(total.pick)
|
||||
if a.startswith(b) or b.startswith(a):
|
||||
self.add(sp + ("pick",), "subtotal_pick_conflict",
|
||||
f"小计行的开头词「{sub.pick}」和合计行的「{total.pick}」一个是另一个的开头,分不清是小计还是合计")
|
||||
|
||||
def check_list_period(self, base: tuple[Any, ...], block: ListBlock, names: list[str]) -> None:
|
||||
"""期 5:每行属于哪一期,按日期列核对和加起止两列只能二选一;按日期列时那一列要是这个列表的日期列。"""
|
||||
period = block.period
|
||||
if period is None:
|
||||
return
|
||||
pp = base + ("period",)
|
||||
if (period.column is None) == (period.synthesize is None):
|
||||
self.add(pp, "list_period_invalid", "每一行属于哪一期只能二选一:按一个日期列核对,或加统计期起止两列")
|
||||
return
|
||||
if period.column is not None:
|
||||
col = next((c for c in block.columns if c.name == period.column), None)
|
||||
if col is None:
|
||||
self.add(pp + ("column",), "column_unknown",
|
||||
f"核对统计期的列「{period.column}」不在这个列表的列里{self._near(period.column, names)}")
|
||||
elif col.type != "DATE":
|
||||
self.add(pp + ("column",), "list_period_invalid",
|
||||
f"核对统计期的列「{period.column}」的类型要是日期")
|
||||
|
||||
# ---- 期 3:忽略规则、表头上方的标题(P3-SPEC 9.2) ----
|
||||
|
||||
def check_ignores(self) -> None:
|
||||
|
||||
@@ -30,6 +30,7 @@ from app.data.recipe_parsers import hour_range, hour_range_total, match_key, tex
|
||||
from app.data.recipe_types import (
|
||||
UNION_VER, AccumulatePlan, ChangeClass, CheckResult, ColumnOut, CrosstabBlock, DerivedSegment,
|
||||
DimensionSegment, PartInfo, Recipe, UnionPart, UnionReport, accumulate_blockers, derive_tables,
|
||||
list_period_columns,
|
||||
)
|
||||
from app.data.table_versions import settle_journal, sha_json
|
||||
|
||||
@@ -508,14 +509,19 @@ def union_options(parts: list[tuple[str, str, str]], target_recipe_sha: str, *,
|
||||
|
||||
|
||||
class _Table:
|
||||
"""并集里的一张表:列(目标配方的在前、退役的按给定顺序追加在后)、主键、日期轴列。"""
|
||||
"""并集里的一张表:列(目标配方的在前、退役的按给定顺序追加在后)、主键、日期轴列。期 5:列表写入的表按 period
|
||||
分期(list_period_columns):按日期列的,U3 核对那一列落在该期统计期内;加统计期起止两列的,核对两列就是该期。"""
|
||||
|
||||
def __init__(self, name: str, cols: list[ColumnOut], grain: list[str]) -> None:
|
||||
def __init__(self, name: str, cols: list[ColumnOut], grain: list[str],
|
||||
period: tuple[str, ...] | None = None) -> None:
|
||||
self.name = name
|
||||
self.cols = cols
|
||||
self.grain = grain
|
||||
self.axis = next((c.name for c in cols if c.role == "axis"), None)
|
||||
self.names = [c.name for c in cols]
|
||||
self.period = period if period is None or all(c in self.names for c in period[1:]) else None
|
||||
if self.axis is None and self.period is not None and self.period[0] == "column":
|
||||
self.axis = self.period[1]
|
||||
|
||||
|
||||
def _copy_col(col: ColumnOut) -> ColumnOut:
|
||||
@@ -551,6 +557,7 @@ def _layout(target: Recipe, parts: list[UnionPart],
|
||||
|
||||
out: dict[str, _Table] = {}
|
||||
retired = retired or {}
|
||||
periods = list_period_columns(target)
|
||||
for name, cols in tables.items():
|
||||
mine = [_copy_col(c) for c in cols]
|
||||
known = {c.name for c in mine}
|
||||
@@ -561,7 +568,7 @@ def _layout(target: Recipe, parts: list[UnionPart],
|
||||
mine.append(_copy_col(found) if found is not None else _copy_col(col))
|
||||
known.add(col.name)
|
||||
spec = specs.get(name)
|
||||
out[name] = _Table(name, mine, list(spec.grain) if spec is not None else [])
|
||||
out[name] = _Table(name, mine, list(spec.grain) if spec is not None else [], periods.get(name))
|
||||
for name, cols in retired.items():
|
||||
if name in out:
|
||||
continue
|
||||
@@ -831,20 +838,36 @@ def _finish(conn: sqlite3.Connection, layout: dict[str, _Table], parts: list[Uni
|
||||
u2.append(f"表「{table.name}」有 {dup} 组主键重复")
|
||||
checks.append(_check("U2", "union_pk", "累积后主键不重复", checked=len(keyed), failed=u2, sql=sql2))
|
||||
|
||||
# U3 统计期:第 k 期的 rowid 区间里,日期都在 [start_k, end_k] 之内
|
||||
# U3 统计期:第 k 期的 rowid 区间里,日期都在 [start_k, end_k] 之内。期 5:按日期列分期的列表,日期为空也算不在;
|
||||
# 加统计期起止两列的列表,两列必须正好是这一期
|
||||
u3: list[str] = []
|
||||
sql3, params3, checked3 = None, [], 0
|
||||
for k, (part, ranges) in enumerate(zip(parts, part_rows), 1):
|
||||
for name, span in ranges.items():
|
||||
axis = layout[name].axis
|
||||
if axis is None or axis not in part_cols[k - 1].get(name, []):
|
||||
continue
|
||||
table = layout[name]
|
||||
have = part_cols[k - 1].get(name, [])
|
||||
a, b = span["union"]
|
||||
if table.period is not None and table.period[0] == "synthesize":
|
||||
start_col, end_col = table.period[1], table.period[2]
|
||||
if start_col not in have or end_col not in have or b < a:
|
||||
continue
|
||||
checked3 += 1
|
||||
sql3 = (f"SELECT COUNT(*) FROM {_q(name)} WHERE rowid BETWEEN ? AND ? "
|
||||
f"AND ({_q(start_col)} IS NOT ? OR {_q(end_col)} IS NOT ?)")
|
||||
params3 = [a, b, part.start, part.end]
|
||||
bad = conn.execute(sql3, params3).fetchone()[0]
|
||||
if bad:
|
||||
u3.append(f"第 {k} 期({_period_text(part.start, part.end)})的「{name}」有 {bad} 行的统计期不是这一期")
|
||||
continue
|
||||
axis = table.axis
|
||||
if axis is None or axis not in have:
|
||||
continue
|
||||
if b < a:
|
||||
continue
|
||||
checked3 += 1
|
||||
nulls = f"{_q(axis)} IS NULL OR " if table.period is not None else ""
|
||||
sql3 = (f"SELECT COUNT(*) FROM {_q(name)} WHERE rowid BETWEEN ? AND ? "
|
||||
f"AND ({_q(axis)} < ? OR {_q(axis)} > ?)")
|
||||
f"AND ({nulls}{_q(axis)} < ? OR {_q(axis)} > ?)")
|
||||
params3 = [a, b, part.start, part.end]
|
||||
bad = conn.execute(sql3, params3).fetchone()[0]
|
||||
if bad:
|
||||
|
||||
@@ -23,6 +23,7 @@ from typing import Any, Callable, Iterable
|
||||
|
||||
from app.data.engine import open_checked_sqlite
|
||||
from app.data.recipe_types import (
|
||||
SUBTOTAL_KEY,
|
||||
CheckResult,
|
||||
ColumnOut,
|
||||
CrosstabBlock,
|
||||
@@ -34,6 +35,7 @@ from app.data.recipe_types import (
|
||||
Recipe,
|
||||
SumEq,
|
||||
derive_tables,
|
||||
list_period_columns,
|
||||
)
|
||||
|
||||
#: 无法核对的原因(契约 CheckResult.reasons 的键)的固定顺序:细节、说明都按这个顺序列
|
||||
@@ -71,24 +73,33 @@ class CheckPlan:
|
||||
k: dict[str, str] = field(default_factory=dict)
|
||||
#: derived 分段键 → G 编号(只给有公式格的 derived 段)
|
||||
g: dict[str, str] = field(default_factory=dict)
|
||||
#: 列表块键 → T 编号(每个有合计行的列表块一条)
|
||||
#: 列表块键 → T 编号(每个有合计行的列表块一条);期 5:块键 + SUBTOTAL_KEY → 这个块的小计核对(一条)
|
||||
t: dict[str, str] = field(default_factory=dict)
|
||||
items: dict[str, list[DerivedItem]] = field(default_factory=dict)
|
||||
|
||||
|
||||
def _recipe_keys(recipe: Recipe) -> tuple[list[str], list[str]]:
|
||||
"""配方里 derived 分段键、带合计行的列表块键,按配方里的先后。"""
|
||||
"""配方里 derived 分段键、带合计行的列表块键,按配方里的先后。期 5:带小计的列表块另有一个键(块键 +
|
||||
SUBTOTAL_KEY),排在这个块的合计行后面:同一块的小计和合计各编一个 T,结论分开写。"""
|
||||
derived: list[str] = []
|
||||
totals: list[str] = []
|
||||
for sheet in recipe.sheets:
|
||||
for block in sheet.blocks:
|
||||
if isinstance(block, CrosstabBlock):
|
||||
derived += [s.id for s in block.segments if isinstance(s, DerivedSegment)]
|
||||
elif block.rows.total_row is not None:
|
||||
continue
|
||||
if block.rows.total_row is not None:
|
||||
totals.append(block.id)
|
||||
if block.rows.subtotals is not None:
|
||||
totals.append(f"{block.id}{SUBTOTAL_KEY}")
|
||||
return derived, totals
|
||||
|
||||
|
||||
def is_subtotal_key(key: str) -> bool:
|
||||
"""T 编号的键是不是一个块的小计核对(期 5)。"""
|
||||
return key.endswith(SUBTOTAL_KEY)
|
||||
|
||||
|
||||
def _ordered(recipe_keys: list[str], items: list[DerivedItem]) -> list[str]:
|
||||
"""先按配方里的先后;配方里找不到的分段(不该出现)按执行结果里第一次出现的先后排在后面,不丢。"""
|
||||
present: list[str] = []
|
||||
@@ -570,6 +581,25 @@ def _check_pk(sql: _Sql, cid: str, title: str, table: str, grain: list[str]) ->
|
||||
# ---------------------------------------------------------------- 入口
|
||||
|
||||
|
||||
def _check_period_rows(sql: _Sql, cid: str, title: str, table: str, column: str, period: Any,
|
||||
lineage: dict[str, dict[str, list[list[Any]]]]) -> CheckResult:
|
||||
"""期 5:列表按日期列分期(ListPeriod.column、check=within_context)时,每行的这一列都落在本期统计期内。
|
||||
日期为空的行也算不在:看不出它属于哪一期。不在的行按期累积会混进别的期(统计期互不重叠的前提就破了),
|
||||
所以是结构类、不能写理由接受。统计期没定下来(要人录入)时不会走到这里:那时不建库。"""
|
||||
if period is None or not period.start or not period.end:
|
||||
raise _Broken("本期没有统计期,无法核对每行的日期")
|
||||
q = (f"SELECT rowid FROM {_q(table)} WHERE {_q(column)} IS NULL OR {_q(column)} < ? OR {_q(column)} > ? "
|
||||
"ORDER BY rowid")
|
||||
params = [period.start, period.end]
|
||||
bad = [r[0] for r in sql.all(q, params)]
|
||||
total = sql.one(f"SELECT COUNT(*) FROM {_q(table)}")[0]
|
||||
cells = [c for rowid in bad[:_DETAIL_MAX] if (c := lineage_cell(lineage, table, column, rowid))]
|
||||
details = [f"有 {len(bad)} 行的「{column}」为空或不在 {period.start} 至 {period.end} 之内"] if bad else []
|
||||
return CheckResult(id=cid, kind="period_rows", title=title, status="mismatch" if bad else "passed",
|
||||
category="structure", checked=int(total), failed=len(bad), sql=q, params=params,
|
||||
details=details, cells=cells)
|
||||
|
||||
|
||||
def _axis_names(recipe: Recipe) -> dict[str, str]:
|
||||
"""derived 分段键 → 所在交叉表块的轴列名(K 的铺满判定不按日期分)。"""
|
||||
out: dict[str, str] = {}
|
||||
@@ -629,7 +659,9 @@ def run_checks(db_path: str, extraction: Extraction, recipe: Recipe) -> list[Che
|
||||
jobs.append((cid, "formula_refs", f"合计公式引用(分段「{seg}」)",
|
||||
lambda t, seg=seg, cid=cid: _check_g(sql, cid, t, plan.items[seg])))
|
||||
for block, cid in plan.t.items():
|
||||
jobs.append((cid, "column_sum", f"合计行按列求和(块「{block}」)",
|
||||
title = (f"小计行按各组明细求和(块「{block.removesuffix(SUBTOTAL_KEY)}」)" if is_subtotal_key(block)
|
||||
else f"合计行按列求和(块「{block}」)")
|
||||
jobs.append((cid, "column_sum", title,
|
||||
lambda t, block=block, cid=cid: _check_t(sql, cid, t, plan.items[block], full_calc)))
|
||||
for rel in recipe.relations:
|
||||
if isinstance(rel, SumEq):
|
||||
@@ -646,6 +678,12 @@ def run_checks(db_path: str, extraction: Extraction, recipe: Recipe) -> list[Che
|
||||
for i, table in enumerate([t for t in tables if grains.get(t)], 1):
|
||||
jobs.append((f"P{i}", "pk_unique", f"主键唯一:「{table}」",
|
||||
lambda t, table=table, i=i: _check_pk(sql, f"P{i}", t, table, grains[table])))
|
||||
period = extraction.period
|
||||
dated = [(t, spec[1]) for t, spec in list_period_columns(recipe).items() if spec[0] == "column" and t in tables]
|
||||
for i, (table, column) in enumerate(dated, 1):
|
||||
jobs.append((f"D{i}", "period_rows", f"「{table}」每行的「{column}」都在本期统计期内",
|
||||
lambda t, table=table, column=column, i=i: _check_period_rows(
|
||||
sql, f"D{i}", t, table, column, period, extraction.lineage)))
|
||||
try:
|
||||
return list(extraction.context_checks) + [_guarded(sql, *job) for job in jobs]
|
||||
finally:
|
||||
|
||||
@@ -43,7 +43,7 @@ from app.data.recipe_diff import (
|
||||
)
|
||||
from app.data.recipe_parsers import UNITS, match_key, period_residue, split_unit_suffix
|
||||
from app.data.recipe_types import (
|
||||
ConfirmItem, CrosstabBlock, DerivedSegment, Dismissed, ListBlock, MeasuresSegment, NotComparable, Recipe,
|
||||
LIST_PERIOD_END, LIST_PERIOD_START, SUBTOTAL_KEY, ConfirmItem, CrosstabBlock, DerivedSegment, Dismissed, ListBlock, MeasuresSegment, NotComparable, Recipe,
|
||||
SheetRecipe, SumEq, derive_tables,
|
||||
)
|
||||
|
||||
@@ -523,6 +523,13 @@ def _signatures(recipe: Recipe, sheet_name: Callable[[SheetRecipe], str]) -> dic
|
||||
total = block.rows.total_row
|
||||
out[f"total_row:{block.id}"] = _sig(
|
||||
None if total is None else (total.label_column, total.pick, total.keep_as))
|
||||
# 期 5:穿插的小计、每行属于哪一期
|
||||
subt = block.rows.subtotals
|
||||
out[f"subtotals:{block.id}"] = _sig(
|
||||
None if subt is None else (subt.label_column, subt.pick, subt.scope, subt.group_column, subt.keep_as))
|
||||
out[f"no_subtotals:{block.id}"] = _sig(block.rows.no_subtotals)
|
||||
period = block.period
|
||||
out[f"list_period:{block.id}"] = _sig(None if period is None else (period.column, period.synthesize))
|
||||
for r in recipe.relations:
|
||||
if isinstance(r, SumEq):
|
||||
out[f"relation:{r.id}"] = _sig("sum_eq", r.table, r.total, sorted(r.parts))
|
||||
@@ -1058,6 +1065,29 @@ def _recipe_items(bag: _Bag, recipe: Recipe, ex: dict[str, Any], facts: Any, *,
|
||||
detail = (f"按明细核对的列:{'、'.join(nums)};其余列合计行里的内容不核对" if nums
|
||||
else "这个块没有数字列,合计行里的内容不核对")
|
||||
bag.add(f"total_row:{b.id}", f"{prefix}改作核对{keep}", detail)
|
||||
for b in lists:
|
||||
# 期 5:穿插在明细中间的小计行:不进数据,按各组明细核对。要人确认:这几行从此不在本表里
|
||||
subt = b.rows.subtotals
|
||||
if subt is not None:
|
||||
rows = rows_text(derived_rows.get(f"{b.id}{SUBTOTAL_KEY}"))
|
||||
prefix = f"「{b.id}」中间的小计行" + (f"({rows})" if rows else "")
|
||||
keep = f",原值另存表「{subt.keep_as}」" if subt.keep_as else ",原值不另存"
|
||||
scope = ("每行小计核对上一行小计之后的全部明细" if subt.scope == "since_previous"
|
||||
else f"每行小计核对紧挨在它上面、「{subt.group_column}」相同的那几行明细")
|
||||
nums = [c.name for c in b.columns if c.type in ("INTEGER", "REAL")]
|
||||
detail = f"{scope};按明细核对的列:{'、'.join(nums)}" if nums else f"{scope};这个块没有数字列,不核对"
|
||||
bag.add(f"subtotals:{b.id}", f"{prefix}改作核对、不进数据{keep}", detail)
|
||||
elif b.rows.no_subtotals:
|
||||
# 声明没有小计行是放宽:以合计字样开头的行从此照常进数据,不再报「像小计」。配上(或改过)的那一次确认
|
||||
bag.add(f"no_subtotals:{b.id}", f"「{b.id}」没有小计行:以合计字样开头的行都按数据导入,不再提醒像小计的行",
|
||||
"如果其实有穿插的小计,对这些列求和会把小计重复算进去")
|
||||
period = b.period
|
||||
if period is not None and period.column:
|
||||
bag.add(f"list_period:{b.id}", f"「{b.id}」每一行的「{period.column}」都要在本期统计期内",
|
||||
"按期累积时各期的行按这一列区分;有一行不在本期统计期内(或为空),这一期整份拒收")
|
||||
elif period is not None and period.synthesize:
|
||||
bag.add(f"list_period:{b.id}", f"「{b.id}」每一行加上本期统计期的起、止两列",
|
||||
f"列名「{LIST_PERIOD_START}」「{LIST_PERIOD_END}」,取自表内的统计期或人工录入的统计期")
|
||||
|
||||
# 年份取自统计期:表头是日期格时年份来自格子本身,不算
|
||||
forms: dict[str, set[str]] = {}
|
||||
@@ -1235,6 +1265,14 @@ def _period_items(bag: _Bag, recipe: Recipe, ex: dict[str, Any], ctx: ConfirmCon
|
||||
bag.add(f"rows_after_stop:{blk}", ";".join(str(p.get("message") or "") for p in probs),
|
||||
"、".join(cells[:20]), source="outside")
|
||||
|
||||
# 期 5:没配小计的列表里像小计的数据行:照常导入了,要人确认它们确实是数据(不是的话用修复按钮改作小计核对)
|
||||
for p in ex.get("problems") or []:
|
||||
p = plain(p)
|
||||
if isinstance(p, dict) and p.get("code") == "subtotal_suspect":
|
||||
blk = str((p.get("fix_args") or {}).get("block") or "、".join(str(c) for c in p.get("cells") or []))
|
||||
bag.add(f"subtotal_suspect:{blk}", str(p.get("message") or ""),
|
||||
"、".join(str(c) for c in (p.get("cells") or [])[:20]), source="outside")
|
||||
|
||||
# 列表块只有表头、没有数据行(AU-1):问题对象里没有块键,按消息里的「列表「块」」认
|
||||
for p in ex.get("problems") or []:
|
||||
p = plain(p)
|
||||
|
||||
@@ -39,6 +39,14 @@ pk_duplicate 同时出现)。块级的定位失败(找不到日期表头、
|
||||
个矩形。
|
||||
- 回执里的「排除的行」rows_excluded(H2:排除的行要写进回执):隐藏行、列表跳过的空行、按规则忽略的行、
|
||||
停止之后没导入的文字行、只核对不另存的合计行。
|
||||
|
||||
**期 5(下一轮 03 细案 E)又加了两件事,同样没用新字段的配方写出的库逐字节不变:**
|
||||
- 列表中间穿插的小计(ListRows.subtotals):小计行不进数据,每个数字格一条合计格,按它所管的那组明细(上一行小计
|
||||
之后的全部明细,或紧挨在上面、分组列相同的那几行)交给核对模块用 SQL 求和比对;整行记进排除的行(subtotal)。
|
||||
- 列表按期累积时由系统加的统计期起止两列(ListPeriod.synthesize):列表在统计期定下来之前就抽完了,这两格先放
|
||||
占位,统计期定下来以后写库时换成真值(_Sink.fill_period);按日期列分期的(ListPeriod.column)由核对模块的 D
|
||||
核对查每行都在本期统计期内。
|
||||
CSV 原件经 csv_grid 适配成一张「工作表」,这里照常按网格或逐行流读(xlsx_cells 按 SheetScan.path 分派)。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -59,15 +67,16 @@ from app.data import engine as _data_engine
|
||||
from app.data import tabular, xlsx_cells
|
||||
from app.data.names import canon
|
||||
from app.data.recipe_parsers import (
|
||||
AxisDate, HourRange, HourTotal, Period, classify_outside, cn_date_range, cn_year_month, filename_period,
|
||||
TOTAL_WORDS, AxisDate, HourRange, HourTotal, Period, classify_outside, cn_date_range, cn_year_month,
|
||||
filename_period,
|
||||
hour_range, hour_range_total, is_pure_period_cell, looks_numeric_text, match_key, month_day_or_date,
|
||||
period_residue, text_label, thousands_number,
|
||||
)
|
||||
from app.data.recipe_types import (
|
||||
AxisOut, CanonEntry, CheckResult, ColumnOut, ContextSpec, CrosstabBlock, DerivedItem, DimensionSegment,
|
||||
ExcludedRows, Extraction, Grid, GridCell, LedgerSheet, ListBlock, MeasuresSegment, OutsideText,
|
||||
SUBTOTAL_KEY, AxisOut, CanonEntry, CheckResult, ColumnOut, ContextSpec, CrosstabBlock, DerivedItem,
|
||||
DimensionSegment, ExcludedRows, Extraction, Grid, GridCell, LedgerSheet, ListBlock, MeasuresSegment, OutsideText,
|
||||
PeriodInput, PeriodOut, Problem, Recipe, RegionMark, SegmentLabels, SheetRecipe, TableOut, TableSpec,
|
||||
derive_tables,
|
||||
derive_tables, list_period_columns,
|
||||
)
|
||||
from app.data.xlsx_scan import MAX_MERGED, SheetScan, WorkbookScan, cell_ref, col_letter
|
||||
|
||||
@@ -101,7 +110,8 @@ PROBLEM_CODES: dict[str, str] = {
|
||||
# 列表定位
|
||||
"header_not_found": "structure", "header_ambiguous": "structure", "column_missing": "structure",
|
||||
"column_extra": "structure", "rows_after_stop": "confirm", "total_row_missing": "structure",
|
||||
"total_row_blank": "structure", "list_empty": "confirm",
|
||||
"total_row_blank": "structure", "list_empty": "confirm", "subtotal_blank": "structure",
|
||||
"subtotal_suspect": "confirm",
|
||||
# 值
|
||||
"value_not_integer": "structure", "value_text_number": "structure", "value_blank": "structure",
|
||||
"value_error": "structure", "value_not_number": "structure", "value_not_date": "structure",
|
||||
@@ -125,6 +135,7 @@ PROBLEM_FIX: dict[str, str] = {
|
||||
"axis_extra_cells": "ignore_cells", "column_extra": "ignore_cells", "row_unclaimed": "ignore_cells",
|
||||
"outside_number": "ignore_cells",
|
||||
"hidden_rows": "declare_hidden", "hidden_cols": "declare_hidden", "sheet_missing": "rename_sheet",
|
||||
"subtotal_suspect": "declare_subtotals",
|
||||
}
|
||||
|
||||
#: 列表找表头的窗口:从 after_title 的下一行(或已用区域首行)起往下这么多行
|
||||
@@ -143,7 +154,7 @@ PLACEHOLDER_TEXTS_MAX = 8
|
||||
PLACEHOLDER_TEXT_LEN = 8
|
||||
#: 排除的行在回执里的顺序(同一工作表内按原因、块、锚点、首行排)
|
||||
_EXCLUDED_ORDER = ("hidden_excluded", "blank_skipped", "ignored_rows", "ignored_outside", "after_stop",
|
||||
"total_not_kept")
|
||||
"total_not_kept", "subtotal")
|
||||
#: 公式区域展开的上限:更大的区域不可能全落在基表里,直接判「引用了基表之外的格」
|
||||
REF_CELLS_MAX = 200_000
|
||||
|
||||
@@ -613,11 +624,18 @@ class _Problems:
|
||||
class _Sink:
|
||||
"""一张表的行:主键去重、计数、溯源、批量写库(有库时)。rowid 就是插入序号(新库、1 起)。"""
|
||||
|
||||
def __init__(self, eng: _Engine, name: str, cols: list[ColumnOut], spec: TableSpec | None) -> None:
|
||||
def __init__(self, eng: _Engine, name: str, cols: list[ColumnOut], spec: TableSpec | None,
|
||||
synth: tuple[str, str] | None = None) -> None:
|
||||
self.eng = eng
|
||||
self.name = name
|
||||
self.cols = cols
|
||||
names = [c.name for c in cols]
|
||||
#: 期 5:由系统加的统计期起止两列(ListPeriod.synthesize)在一行里的位置。列表是在统计期定下来之前抽完的,
|
||||
#: 这两格先放占位(_PERIOD_START / _PERIOD_END),统计期定下来以后(fill_period)写库时换成真值;在那之前
|
||||
#: 不分批写库(flush 等着),整张表留在内存里
|
||||
self.synth_idx = (names.index(synth[0]), names.index(synth[1])) \
|
||||
if synth and synth[0] in names and synth[1] in names else None
|
||||
self.period: tuple[str | None, str | None] | None = None
|
||||
self.grain = [g for g in (spec.grain if spec else []) if g in names]
|
||||
self.grain_idx = [names.index(g) for g in self.grain]
|
||||
self.check_pk = bool(self.grain_idx) and not eng.partial
|
||||
@@ -682,9 +700,22 @@ class _Sink:
|
||||
self.eng.problems.issue("value_blank", sheet, f"工作表「{sheet}」的表「{self.name}」的主键列{_quoted(names)}",
|
||||
coords, text=_PK_NULL, group="pk_null")
|
||||
|
||||
def flush(self) -> None:
|
||||
def fill_period(self, start: str | None, end: str | None) -> None:
|
||||
"""统计期定下来了:加的统计期起止两列从此写真值(start、end 是 YYYY-MM-DD)。"""
|
||||
self.period = (start, end)
|
||||
|
||||
def flush(self, *, final: bool = False) -> None:
|
||||
if self.synth_idx is not None and self.period is None and not final:
|
||||
return # 统计期还没定:加的两列还是占位,先不写
|
||||
if self.buffer and self.eng.conn is not None:
|
||||
self.eng.conn.executemany(self.insert_sql, self.buffer)
|
||||
rows = self.buffer
|
||||
if self.synth_idx is not None:
|
||||
a, b = self.synth_idx
|
||||
start, end = self.period or (None, None)
|
||||
fill = {a: start, b: end}
|
||||
rows = [tuple(fill.get(i, v) if v is _PERIOD_START or v is _PERIOD_END else v
|
||||
for i, v in enumerate(row)) for row in rows]
|
||||
self.eng.conn.executemany(self.insert_sql, rows)
|
||||
self.buffer.clear()
|
||||
|
||||
|
||||
@@ -695,6 +726,12 @@ class _Sink:
|
||||
_FAIL = object()
|
||||
#: 列表数据区里不能填充的合并单元格覆盖的空格(已经报了 merged_in_values)
|
||||
_MERGED = object()
|
||||
#: 期 5:加的统计期起止两列在统计期定下来之前的占位(_Sink.fill_period)。主键判重时它们和自己相等:
|
||||
#: 同一期的行统计期都一样,判重只看其余的主键列,结论不变
|
||||
_PERIOD_START = object()
|
||||
_PERIOD_END = object()
|
||||
#: 小计按组时「还没有组」
|
||||
_NO_GROUP = object()
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
@@ -1625,6 +1662,31 @@ class _ListRun:
|
||||
self.ignore_keys = {k for rule in block.ignore_columns if (k := match_key(rule.header))}
|
||||
total = block.rows.total_row
|
||||
self.total_idx = next((i for i, c in enumerate(self.cols) if total and c.name == total.label_column), None)
|
||||
# 期 5:穿插的小计行(ListRows.subtotals)。sub_first:上一行小计(或表头)之后第一行明细的 rowid;按组小计
|
||||
# 时 run_first / run_value 是紧挨在上面、分组列取值相同的那一段明细的首个 rowid 和取值。sub_rows:小计所在的
|
||||
# 行 → 它管的明细的 rowid(合计行的公式引用了小计格时,按它换算回明细)
|
||||
self.sub = block.rows.subtotals
|
||||
sub = self.sub
|
||||
self.sub_idx = next((i for i, c in enumerate(self.cols) if sub and c.name == sub.label_column), None)
|
||||
self.sub_key = match_key(sub.pick) if sub else ""
|
||||
self.group_idx = next((i for i, c in enumerate(self.cols)
|
||||
if sub and sub.scope == "group_column" and c.name == sub.group_column), None)
|
||||
self.sub_first: int | None = None
|
||||
self.run_first: int | None = None
|
||||
self.run_value: Any = _NO_GROUP
|
||||
self.sub_hidden = False
|
||||
self.sub_rows: dict[int, list[int]] = {}
|
||||
#: 期 5:加统计期起止两列(ListPeriod.synthesize):每行末尾两格占位,统计期定下来以后由 _Sink 换成真值
|
||||
self.synth = block.period is not None and block.period.synthesize is not None
|
||||
#: 期 5:没配小计时,像夹在明细中间的小计的数据行(行号)和命中的合计字样:照常按数据导入,收尾时报
|
||||
#: subtotal_suspect 要人确认(带修复按钮:改作小计核对)。人声明过没有小计(no_subtotals)的不看
|
||||
self.suspects: list[int] = []
|
||||
self.suspect_word: str | None = None
|
||||
self.watch_suspects = block.rows.subtotals is None and not block.rows.no_subtotals
|
||||
#: 上一行普通明细在哪些文字、日期列(列序号,不含标签所在的首列)上有值;还没有明细时是空集
|
||||
self.prev_filled: frozenset[int] = frozenset()
|
||||
#: 等着下一行明细确认的候选:(行号, 合计字样, 上一行明细有值、这一行空着的列)
|
||||
self.pending: list[tuple[int, str, frozenset[int]]] = []
|
||||
|
||||
# ---------------- 喂入 ----------------
|
||||
|
||||
@@ -1692,6 +1754,7 @@ class _ListRun:
|
||||
if self.located:
|
||||
for i, text in self.headers_read.items():
|
||||
self.eng.headers[(self.block.table, self.cols[i].name)] = text
|
||||
self._report_suspects()
|
||||
|
||||
# ---------------- 表头 ----------------
|
||||
|
||||
@@ -2001,6 +2064,7 @@ class _ListRun:
|
||||
fills = self._merges_at(r, inb)
|
||||
if self.sh.hidden_row(r):
|
||||
self.hidden_seen = True
|
||||
self.sub_hidden = True
|
||||
if self.sh.sr.hidden.rows == "exclude":
|
||||
self.excluded_rows.add(r)
|
||||
self.sh.layout(r, self.layout_hidden)
|
||||
@@ -2010,10 +2074,85 @@ class _ListRun:
|
||||
label = inb.get(self.colmap.get(self.total_idx, -1))
|
||||
pick = match_key(block.rows.total_row.pick) # type: ignore[union-attr]
|
||||
if label is not None and match_key(_text(label.value)).startswith(pick):
|
||||
if self.watch_suspects:
|
||||
self._settle_suspects(None)
|
||||
self._total_row(r, inb, cells, label)
|
||||
return
|
||||
if self.sub_idx is not None:
|
||||
label = inb.get(self.colmap.get(self.sub_idx, -1))
|
||||
if label is not None and match_key(_text(label.value)).startswith(self.sub_key):
|
||||
self._subtotal_row(r, inb, cells, label)
|
||||
return
|
||||
elif self.watch_suspects:
|
||||
self._watch_suspect(r, inb)
|
||||
self._record(r, inb, cells, fills)
|
||||
|
||||
def _watch_suspect(self, r: int, inb: dict[int, GridCell]) -> None:
|
||||
"""期 5:没配小计、也没声明没有小计时,认夹在明细中间的小计(照常按数据导入,收尾时报 subtotal_suspect)。
|
||||
|
||||
要同时满足才算:首列以合计字样开头、其余文字和日期列都空着、至少一格有数(_suspect_word);上一行明细在某个
|
||||
文字或日期列上有值、这一行那一列空着;下一行明细在同一列上也有值,或者下面紧跟着合计行。所以候选先挂着,
|
||||
等下一行明细来了再定(_settle_suspects)。
|
||||
|
||||
为什么这么严(复核反例):只有「标签列 + 数字列」的键值清单(「累计收入 300」「汇总调整 5」),「其余文字列
|
||||
为空」自然成立,什么也说明不了;放在最后、后面没有明细的行也不是「夹在中间」。报错了就是每期都要勾的
|
||||
确认项,按配方导入一期的无人值守运行会被它挡住。"""
|
||||
filled = frozenset(i for i, col in enumerate(self.cols)
|
||||
if col.type in ("TEXT", "DATE") and (c := self.colmap.get(i)) is not None
|
||||
and c != self.c1 and c in inb)
|
||||
word = self._suspect_word(inb) if self.prev_filled else None
|
||||
if word is not None:
|
||||
self.pending.append((r, word, self.prev_filled))
|
||||
return
|
||||
self._settle_suspects(filled)
|
||||
self.prev_filled = filled
|
||||
|
||||
def _settle_suspects(self, filled: frozenset[int] | None) -> None:
|
||||
"""挂着的候选有了下文:filled 是紧跟着的那行明细有值的文字、日期列(None 是紧跟着合计行)。上一行明细有值、
|
||||
候选空着的列,下一行明细也有值,才算夹在明细中间。"""
|
||||
for row, word, gap in self.pending:
|
||||
if filled is None or gap & filled:
|
||||
self.suspects.append(row)
|
||||
self.suspect_word = self.suspect_word or word
|
||||
self.pending = []
|
||||
|
||||
def _suspect_word(self, inb: dict[int, GridCell]) -> str | None:
|
||||
"""期 5:这一行本身像不像小计:首列是以合计字样(TOTAL_WORDS)开头的文字,其余文字、日期列都空着,至少一格
|
||||
有数。像的话返回那个字样。以前这种行静默当数据导入,求和就把小计重复算了进去(H2);现在照常导入,但收尾时
|
||||
报出来要人确认,并给「改作小计核对」的修复按钮。合计行已经在前面认走了。上下文的条件见 _watch_suspect。"""
|
||||
cell = inb.get(self.c1)
|
||||
if cell is None or cell.formula is not None or not isinstance(cell.value, str):
|
||||
return None
|
||||
key = match_key(cell.value)
|
||||
word = next((w for w in TOTAL_WORDS if key.startswith(w)), None)
|
||||
if word is None or not any(self.colmap.get(i) == self.c1 for i in range(len(self.cols))):
|
||||
return None
|
||||
numeric = False
|
||||
for i, col in enumerate(self.cols):
|
||||
c = self.colmap.get(i)
|
||||
if c is None or c == self.c1 or c not in inb:
|
||||
continue
|
||||
if col.type in ("TEXT", "DATE"):
|
||||
return None
|
||||
numeric = True
|
||||
return word if numeric else None
|
||||
|
||||
def _report_suspects(self) -> None:
|
||||
"""收尾:像小计的数据行合成一条 subtotal_suspect(需确认),修复参数带块 id、标签列、字样、行号和标签格。"""
|
||||
if not self.suspects or not self.located:
|
||||
return
|
||||
label = next(col.name for i, col in enumerate(self.cols) if self.colmap.get(i) == self.c1)
|
||||
cells = [self.sh.coord(r, self.c1) for r in self.suspects]
|
||||
rows = _rows_text(self.suspects)
|
||||
word = self.suspect_word or ""
|
||||
where = (f"{self.where}:第 {rows} 行像夹在明细中间的小计(「{label}」列以「{word}」开头,上下明细都有值的"
|
||||
"文字列在这一行空着),按数据导入了")
|
||||
msg = (f"{where}。如果它们是小计,对这一列求和会把小计重复算进去:请用修复按钮改作小计核对;"
|
||||
"确实是数据时,确认后照常导入;以后各期不想再确认,在配方的「小计行」里勾选「没有小计行」")
|
||||
self.eng.problems.add("subtotal_suspect", msg, where, cells[:CELLS_MAX], sheet=self.sh.name,
|
||||
fix_args={"block": self.block.id, "label_column": label, "pick": word,
|
||||
"rows": list(self.suspects), "cells": cells[:CELLS_MAX]})
|
||||
|
||||
def _plan(self) -> list[tuple[Any, int, bool, str]]:
|
||||
"""每列的(列定义、所在列号、是不是主键、位置说明):定下表头之后算一次,逐行复用。"""
|
||||
plan = self.__dict__.get("_plan_cache")
|
||||
@@ -2073,6 +2212,8 @@ class _ListRun:
|
||||
v = None
|
||||
values.append(v)
|
||||
lineage.append((col.name, r, c))
|
||||
if self.synth:
|
||||
values += [_PERIOD_START, _PERIOD_END]
|
||||
ignored = self.ignored
|
||||
claims = {c: ("ignored_column" if c in ignored else "value") for c in inb}
|
||||
self.sh.layout(r, self.layout_data)
|
||||
@@ -2084,6 +2225,12 @@ class _ListRun:
|
||||
if self.rowid_first is None:
|
||||
self.rowid_first = rowid
|
||||
self.rowid_last = rowid
|
||||
if self.sub_first is None:
|
||||
self.sub_first = rowid
|
||||
if self.group_idx is not None:
|
||||
group = values[self.group_idx]
|
||||
if self.run_value is _NO_GROUP or group != self.run_value:
|
||||
self.run_first, self.run_value = rowid, group
|
||||
self.emit(r, cells, claims, True)
|
||||
|
||||
def _total_row(self, r: int, inb: dict[int, GridCell], cells: list[tuple[int, GridCell]],
|
||||
@@ -2142,7 +2289,67 @@ class _ListRun:
|
||||
self.stop = ("total", r)
|
||||
self.emit(r, cells, claims, True)
|
||||
|
||||
def _ref_lookup(self, rc: tuple[int, int]) -> str | int:
|
||||
def _subtotal_row(self, r: int, inb: dict[int, GridCell], cells: list[tuple[int, GridCell]],
|
||||
label: GridCell) -> None:
|
||||
"""期 5:穿插在明细中间的一行小计。不进数据:标签格和数字格的去向是 total_label / total_value(同合计行),
|
||||
每个数字格一条合计格(DerivedItem,kind=column_sum,分段键是块 id + SUBTOTAL_KEY),核对模块按它所管的那组
|
||||
明细(rowid 闭区间)用 SQL 逐列求和比对,执行器不做加法(H1);整行记进排除的行(reason=subtotal)。配了
|
||||
keep_as 的另存一行(小计项、按组时的分组取值、各数字列)。
|
||||
|
||||
它管哪几行:since_previous 是上一行小计(或表头)之后的全部明细;group_column 是其中紧挨在它上面、分组列
|
||||
取值相同的那一段。一行明细都没有时区间为空,核对判「无法核对」(合计行之上没有明细行)。数字格全空的小计
|
||||
没有一列能核对,报 subtotal_blank(结构类):不报的话这一行就悄悄没了核对。"""
|
||||
eng, sh, block, sub = self.eng, self.sh, self.block, self.sub
|
||||
assert sub is not None
|
||||
claims = {c: ("ignored_column" if c in self.ignored else "total_label") for c in inb}
|
||||
label_raw = _text(label.value)
|
||||
first = self.sub_first
|
||||
if sub.scope == "group_column" and first is not None and self.run_first is not None:
|
||||
first = max(first, self.run_first)
|
||||
last = self.rowid_last if first is not None else None
|
||||
numeric = [(i, col) for i, col in enumerate(self.cols) if col.type in ("INTEGER", "REAL")]
|
||||
if numeric and not any(self.colmap[i] in inb for i, _ in numeric):
|
||||
coord = sh.coord(r, self.colmap[self.sub_idx]) # type: ignore[index]
|
||||
where = f"{self.where}:小计行({coord.split('!')[-1]})的数字格全部为空"
|
||||
eng.problems.add(
|
||||
"subtotal_blank",
|
||||
f"{where},无法按这一组明细核对。请在文件中补上小计后重新上传;原表不再有小计时,在配方面板的"
|
||||
"「小计行」中取消「有小计行」",
|
||||
f"{where},无法核对", [coord], sheet=sh.name)
|
||||
key = f"{block.id}{SUBTOTAL_KEY}"
|
||||
keep_values: list[Any] = []
|
||||
for i, col in numeric:
|
||||
c = self.colmap[i]
|
||||
cell = inb.get(c)
|
||||
if cell is None:
|
||||
keep_values.append(None)
|
||||
continue
|
||||
claims[c] = "total_value"
|
||||
got = eng.convert_number(cell, col.type, block.values, sh.name, f"{self.where}的小计行", r, c,
|
||||
data_area=False)
|
||||
keep_values.append(got)
|
||||
item = DerivedItem(
|
||||
kind="column_sum", segment=key, sheet=sh.name, cell=sh.coord(r, c), label_raw=label_raw,
|
||||
label=canon(label_raw), value=None if got is _FAIL else got, is_formula=cell.formula is not None,
|
||||
formula=cell.formula, base_table=block.table, base_value=col.name, rowid_first=first,
|
||||
rowid_last=last, hidden_in_range=self.sub_hidden, keep_table=sub.keep_as)
|
||||
if item.is_formula:
|
||||
eng.resolve_refs(item, cell.formula or "", self._ref_lookup)
|
||||
eng.ex.derived.append(item)
|
||||
if sub.keep_as and sub.keep_as in eng.sinks:
|
||||
ks = eng.sinks[sub.keep_as]
|
||||
ks.sources.append(f"{sh.sr.id}/{block.id}")
|
||||
group = [None if self.run_value is _NO_GROUP else self.run_value] if self.group_idx is not None else []
|
||||
lin = [(col.name, r, self.colmap[i]) for i, col in numeric]
|
||||
ks.add([canon(label_raw), *group, *keep_values], sheet=sh.name,
|
||||
at=(r, self.colmap[self.sub_idx]), cells=lin, fails=True) # type: ignore[index]
|
||||
n = sum(1 for role in claims.values() if role in ("total_label", "total_value"))
|
||||
sh.exclude("subtotal", r, r, n, block=block.id, anchor=sub.pick)
|
||||
self.sub_rows[r] = list(range(first, last + 1)) if first is not None and last is not None else []
|
||||
self.sub_first, self.run_first, self.run_value, self.sub_hidden = None, None, _NO_GROUP, False
|
||||
self.emit(r, cells, claims, True)
|
||||
|
||||
def _ref_lookup(self, rc: tuple[int, int]) -> str | int | list[int]:
|
||||
r, c = rc
|
||||
if self.c1 <= c <= self.c2:
|
||||
rowid = self.row_rowid.get(r)
|
||||
@@ -2150,6 +2357,9 @@ class _ListRun:
|
||||
return rowid
|
||||
if r in self.excluded_rows:
|
||||
return "hidden"
|
||||
if r in self.sub_rows:
|
||||
# 合计行的公式引用了小计格(=C5+C9):小计格代表它管的那组明细
|
||||
return self.sub_rows[r]
|
||||
return "outside"
|
||||
|
||||
|
||||
@@ -2343,7 +2553,8 @@ class _Engine:
|
||||
self.canon_log[key] = CanonEntry(table, col.name, text, stored, 1, _coord(sheet, r, c))
|
||||
return stored
|
||||
|
||||
def resolve_refs(self, item: DerivedItem, formula: str, lookup: Callable[[tuple[int, int]], str | int]) -> None:
|
||||
def resolve_refs(self, item: DerivedItem, formula: str,
|
||||
lookup: Callable[[tuple[int, int]], str | int | list[int]]) -> None:
|
||||
"""合计公式的引用经溯源:全在基表 → ref_rowids;有隐藏行或分类汇总函数 → hidden;
|
||||
有不在基表的 → outside;形状认不出 → unrecognized。执行器不做加法(H1)。"""
|
||||
shape = formula_refs(formula)
|
||||
@@ -2357,7 +2568,9 @@ class _Engine:
|
||||
outside = not cells
|
||||
for rc in cells:
|
||||
hit = lookup(rc)
|
||||
if isinstance(hit, int):
|
||||
if isinstance(hit, list):
|
||||
rowids.extend(hit) # 期 5:引用了小计格,换算成它管的明细
|
||||
elif isinstance(hit, int):
|
||||
rowids.append(hit)
|
||||
elif hit == "hidden":
|
||||
hidden = True
|
||||
@@ -2607,8 +2820,9 @@ class _Engine:
|
||||
self.ex.problems = self.problems.finish(set())
|
||||
return self.ex
|
||||
self.cols = cols
|
||||
synth = {t: (spec[1], spec[2]) for t, spec in list_period_columns(self.recipe).items() if spec[0] == "synthesize"}
|
||||
for name, cs in cols.items():
|
||||
self.sinks[name] = _Sink(self, name, cs, self.specs.get(name))
|
||||
self.sinks[name] = _Sink(self, name, cs, self.specs.get(name), synth.get(name))
|
||||
if self.db_path is not None:
|
||||
_remove_db(self.db_path)
|
||||
self.conn = _open_sqlite(self.db_path, readonly=False)
|
||||
@@ -2622,6 +2836,11 @@ class _Engine:
|
||||
crosstabs.extend(self._phase1(sh))
|
||||
t2 = time.perf_counter()
|
||||
period, period_reported, _ = self._period()
|
||||
if self.ex.period is not None:
|
||||
# 期 5:列表加的统计期起止两列现在才有值(列表在统计期定下来之前就抽完了)
|
||||
for sink in self.sinks.values():
|
||||
if sink.synth_idx is not None:
|
||||
sink.fill_period(self.ex.period.start, self.ex.period.end)
|
||||
for ct in crosstabs:
|
||||
ct.complete_axis(period, period_reported)
|
||||
ct.extract()
|
||||
@@ -2650,7 +2869,7 @@ class _Engine:
|
||||
_remove_db(self.db_path) # type: ignore[arg-type]
|
||||
else:
|
||||
for sink in self.sinks.values():
|
||||
sink.flush()
|
||||
sink.flush(final=True)
|
||||
self.conn.execute("COMMIT")
|
||||
self.conn.close()
|
||||
self.conn = None
|
||||
@@ -2748,7 +2967,9 @@ class _Engine:
|
||||
sh.failed = True
|
||||
return []
|
||||
b = s.bounds
|
||||
if b is not None and not self.partial and b.area > tabular.SPARSE_FACTOR * max(s.nonempty, tabular.SPARSE_FLOOR):
|
||||
# 稀疏判定挡的是 Excel 里误输入、残留格式撑大的已用区域;CSV 的每一格都是写出来的,空字段多只是数据稀疏
|
||||
if b is not None and not self.partial and self.scan.format != "csv" \
|
||||
and b.area > tabular.SPARSE_FACTOR * max(s.nonempty, tabular.SPARSE_FLOOR):
|
||||
far = "、".join(list(dict.fromkeys(s.far_cells))[:3])
|
||||
msg = (f"工作表「{sh.name}」的已用区域 {b.a1()} 共 {b.area:,} 个单元格,但只有 {s.nonempty:,} 个单元格有内容;"
|
||||
f"远处的单元格(如 {far})可能是误输入或残留格式。请在 Excel 中删除这些单元格后重新上传")
|
||||
|
||||
@@ -253,6 +253,8 @@ def _key(kind: str, target: dict[str, Any]) -> str:
|
||||
return f"declare_hidden:{t['sheet']}:{t['axis']}"
|
||||
if kind == "rename_sheet":
|
||||
return f"rename_sheet:{t['sheet_id']}"
|
||||
if kind == "declare_subtotals":
|
||||
return f"declare_subtotals:{t['block']}"
|
||||
raise KeyError(kind)
|
||||
|
||||
|
||||
@@ -750,6 +752,62 @@ def _o_declare_total(recipe: dict, p: FixProposal, opt: FixOption, reason: str |
|
||||
return ops, summary
|
||||
|
||||
|
||||
# ==========================================================================
|
||||
# 期 5:列表中间像小计的行改作小计核对
|
||||
# ==========================================================================
|
||||
|
||||
|
||||
def _subtotal_keep_name(recipe: dict, block: dict) -> str:
|
||||
used = {collide_key(t.get("name") or "") for t in recipe.get("tables") or []}
|
||||
return unique_name(f"{block.get('table')}_小计", used)
|
||||
|
||||
|
||||
def _p_declare_subtotals(recipe: dict, prob: Any, args: dict, anchor: FixAnchor) -> FixProposal | None:
|
||||
"""subtotal_suspect → 列表块加 rows.subtotals(标签列、开头词取自执行器给的 fix_args:文件里的合计字样)。
|
||||
块已经配了小计、标签列不在块里、开头词含数字时不提议。"""
|
||||
found = find_block(recipe, args.get("block"))
|
||||
if found is None:
|
||||
return None
|
||||
_si, _bi, block = found
|
||||
label, pick = args.get("label_column"), args.get("pick")
|
||||
if block.get("layout") != "list" or (block.get("rows") or {}).get("subtotals"):
|
||||
return None
|
||||
if label not in [c.get("name") for c in block.get("columns") or []] or not isinstance(pick, str) \
|
||||
or not pick or has_digit(pick):
|
||||
return None
|
||||
target = {"block": block["id"], "label_column": label, "pick": pick}
|
||||
keep = _subtotal_keep_name(recipe, block)
|
||||
return _proposal("declare_subtotals", _key("declare_subtotals", target), code=_get(prob, "code"),
|
||||
title=f"列表「{block['id']}」中间「{pick}」开头的行是小计",
|
||||
cells=list(args.get("cells") or _get(prob, "cells") or []), target=target,
|
||||
options=[FixOption("check_only", "改作小计核对,不另存",
|
||||
detail="每行小计按上一行小计之后的明细逐列求和核对,小计行不进数据,记进回执的"
|
||||
"「排除的行」"),
|
||||
FixOption("keep", f"改作小计核对,原值另存表「{keep}」",
|
||||
detail="同上,原表的小计另存成一张表;另存的小计表不能按期累积")],
|
||||
anchor=anchor)
|
||||
|
||||
|
||||
def _o_declare_subtotals(recipe: dict, p: FixProposal, opt: FixOption, reason: str | None) -> tuple[list, list[str]]:
|
||||
found = find_block(recipe, p.target.get("block"))
|
||||
if found is None:
|
||||
raise _Stale
|
||||
si, bi, block = found
|
||||
if (block.get("rows") or {}).get("subtotals"):
|
||||
raise _Stale
|
||||
keep = _subtotal_keep_name(recipe, block) if opt.value == "keep" else None
|
||||
value = {"label_column": p.target["label_column"], "pick": p.target["pick"], "scope": "since_previous",
|
||||
"group_column": None, "keep_as": keep}
|
||||
ops: list[dict[str, Any]] = [{"op": "add", "path": ptr("sheets", si, "blocks", bi, "rows", "subtotals"),
|
||||
"value": value}]
|
||||
if keep:
|
||||
ops.append({"op": "add", "path": "/tables/-", "value": {
|
||||
"name": keep, "grain": [], "kind": "reported_total", "units": {}, "note": ""}})
|
||||
summary = [f"列表「{block['id']}」中间「{p.target['pick']}」开头的行改作小计核对,不进数据"
|
||||
+ (f",原值另存表「{keep}」" if keep else ",不另存")]
|
||||
return ops, summary
|
||||
|
||||
|
||||
# ==========================================================================
|
||||
# ⑥ 忽略(按文字记)
|
||||
# ==========================================================================
|
||||
@@ -998,12 +1056,13 @@ _Builder = Callable[[dict, Any, dict, FixAnchor], "FixProposal | None"]
|
||||
_BUILDERS: dict[str, _Builder] = {
|
||||
"remove_label": _p_remove_label, "add_label": _p_add_label, "rename_title": _p_rename_title,
|
||||
"declare_total": _p_declare_total, "ignore_cells": _p_ignore_cells, "declare_placeholder": _p_declare_placeholder,
|
||||
"declare_hidden": _p_declare_hidden, "rename_sheet": _p_rename_sheet,
|
||||
"declare_hidden": _p_declare_hidden, "rename_sheet": _p_rename_sheet, "declare_subtotals": _p_declare_subtotals,
|
||||
}
|
||||
_OPS: dict[str, Callable[[dict, FixProposal, FixOption, str | None], tuple[list, list[str]]]] = {
|
||||
"remove_label": _o_remove_label, "add_label": _o_add_label, "edit_members": _o_edit_members,
|
||||
"rename_title": _o_rename_title, "declare_total": _o_declare_total, "ignore_cells": _o_ignore_cells,
|
||||
"declare_placeholder": _o_declare_placeholder, "declare_hidden": _o_declare_hidden, "rename_sheet": _o_rename_sheet,
|
||||
"declare_subtotals": _o_declare_subtotals,
|
||||
}
|
||||
|
||||
|
||||
@@ -1064,6 +1123,8 @@ def _edit_title(p: FixProposal, opt: FixOption) -> str:
|
||||
return f"工作表「{t.get('sheet')}」:{opt.label}"
|
||||
if p.kind == "rename_sheet":
|
||||
return opt.label
|
||||
if p.kind == "declare_subtotals":
|
||||
return f"列表「{t.get('block')}」中间「{t.get('pick')}」开头的行{opt.label}"
|
||||
return p.title
|
||||
|
||||
|
||||
|
||||
@@ -177,7 +177,8 @@ class Pipeline:
|
||||
recipe_sha256: Callable[[Recipe], str]
|
||||
canonical: Callable[[Recipe], dict[str, Any]]
|
||||
apply_patch: Callable[[dict[str, Any], list[dict[str, Any]]], dict[str, Any]]
|
||||
scan: Callable[[bytes], WorkbookScan]
|
||||
#: (原件, 文件名) → 扫描结果。文件名定按 CSV 读还是按工作簿读(csv_grid.scan_any)
|
||||
scan: Callable[[bytes, str], WorkbookScan]
|
||||
scan_to_json: Callable[[WorkbookScan], dict[str, Any]]
|
||||
scan_from_json: Callable[[dict[str, Any]], WorkbookScan]
|
||||
read_grid: Callable[..., Grid]
|
||||
@@ -225,7 +226,7 @@ class Pipeline:
|
||||
def default_pipeline() -> Pipeline:
|
||||
"""接上真模块。函数体里惰性导入:这些模块(openpyxl、执行器、起草器)导入不便宜,接口模块加载时不必付。"""
|
||||
from app.data import (
|
||||
recipe, recipe_accumulate, recipe_ai, recipe_checks, recipe_compare, recipe_confirm, recipe_diff,
|
||||
csv_grid, recipe, recipe_accumulate, recipe_ai, recipe_checks, recipe_compare, recipe_confirm, recipe_diff,
|
||||
recipe_engine, recipe_fixes, recipe_notes, recipe_select, recipe_suggest, xlsx_cells, xlsx_scan,
|
||||
)
|
||||
|
||||
@@ -234,7 +235,8 @@ def default_pipeline() -> Pipeline:
|
||||
recipe_sha256=recipe.recipe_sha256,
|
||||
canonical=recipe.canonical_recipe,
|
||||
apply_patch=recipe.apply_patch,
|
||||
scan=xlsx_scan.scan,
|
||||
# CSV / TSV 也走配方:csv_grid 适配成一张「工作表」,其余照旧交给 xlsx_scan(Excel 期 5)
|
||||
scan=csv_grid.scan_any,
|
||||
scan_to_json=xlsx_scan.scan_to_json,
|
||||
scan_from_json=xlsx_scan.scan_from_json,
|
||||
read_grid=xlsx_cells.read_grid,
|
||||
@@ -473,7 +475,7 @@ def _open_book(p: Pipeline, raw: bytes, scan: WorkbookScan) -> _Book:
|
||||
|
||||
def _book_of(p: Pipeline, snap: _Snap) -> _Book:
|
||||
raw = _read_raw(snap.raw_sha256)
|
||||
scan = p.scan_from_json(snap.scan) if snap.scan else p.scan(raw)
|
||||
scan = p.scan_from_json(snap.scan) if snap.scan else p.scan(raw, snap.file_name)
|
||||
return _open_book(p, raw, scan)
|
||||
|
||||
|
||||
@@ -771,7 +773,7 @@ class _Staged:
|
||||
|
||||
def _stage_work(p: Pipeline, raw: bytes, filename: str) -> _Staged:
|
||||
"""首次导入(或切换)的暂存:扫描、网格预览、规则起草(带干跑)、对草稿做一次静态校验。"""
|
||||
scan = p.scan(raw)
|
||||
scan = p.scan(raw, filename)
|
||||
book = _open_book(p, raw, scan)
|
||||
dry = _Capture(_dry_fn(p, book, filename, {}))
|
||||
d = p.draft(scan, book.grids, filename, validate=lambda data, f, o: p.validate(data, facts=f, origin=o),
|
||||
@@ -859,7 +861,7 @@ async def stage_upload(session: AsyncSession, *, raw: bytes, filename: str, name
|
||||
def _replay_work(p: Pipeline, raw: bytes, current: dict[str, Any], filename: str, snap: _Snap,
|
||||
*, evaluate: bool) -> tuple[_Staged, _Eval | None]:
|
||||
"""上传新一期 / 修改配方:扫描(或取存下的扫描)、按现行配方的写法取系统发现,修改配方时再评估一遍。"""
|
||||
scan = p.scan_from_json(snap.scan) if snap.scan else p.scan(raw)
|
||||
scan = p.scan_from_json(snap.scan) if snap.scan else p.scan(raw, filename)
|
||||
book = _open_book(p, raw, scan)
|
||||
facts = p.detect_facts(book.grids, scan, recipe=current)
|
||||
staged = _Staged(scan_json=p.scan_to_json(scan), previews=[p.preview(g) for g in book.grids.values()],
|
||||
@@ -1859,7 +1861,7 @@ def _trial_work(p: Pipeline, snap: _Snap, work: dict[str, Any], facts: DraftFact
|
||||
if parsed is None or problems:
|
||||
return _TrialRun(parsed=parsed, problems=list(problems))
|
||||
raw = _read_raw(snap.raw_sha256)
|
||||
scan = p.scan_from_json(snap.scan) if snap.scan else p.scan(raw)
|
||||
scan = p.scan_from_json(snap.scan) if snap.scan else p.scan(raw, snap.file_name)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
# 先写 .tmp- 再改名:半成品归启动清理(_sweep_tmp)管,发布、回收不会把半个库当成试运行库
|
||||
tmp = path.parent / f"{path.name}{raw_store.TMP_MARK}{secrets.token_hex(6)}"
|
||||
@@ -2361,7 +2363,7 @@ def _manifest(*, staging_id: str, kind: str, source_id: str, info: table_version
|
||||
ex_dict: dict[str, Any], file: dict[str, Any], recipe_meta: dict[str, Any], checks: list[dict[str, Any]],
|
||||
overrides: list[dict[str, Any]], waivers: list[dict[str, Any]], confirmed: list[dict[str, Any]],
|
||||
notes: SchemaNotes, ai: dict[str, Any], signed_by: str | None,
|
||||
edits: list[dict[str, Any]] | None = None) -> dict[str, Any]:
|
||||
edits: list[dict[str, Any]] | None = None, signer_verified: bool = False) -> dict[str, Any]:
|
||||
receipt = copy.deepcopy(ex_dict)
|
||||
receipt.pop("problems", None)
|
||||
return {
|
||||
@@ -2383,7 +2385,8 @@ def _manifest(*, staging_id: str, kind: str, source_id: str, info: table_version
|
||||
"ai": ai,
|
||||
# 期 3:修复、框选的修改记录(含已被整份替换覆盖的,作为历史;不带撤销用的修改前起点)
|
||||
"edits": copy.deepcopy(list(edits or [])),
|
||||
"signed_by": {"name": signed_by, "verified": False},
|
||||
# 是否认证照实记(05-已知遗留 (c)):开了登录、署名是登录的人时为真。以前写死 False,封存的证据记错了事实
|
||||
"signed_by": {"name": signed_by, "verified": bool(signer_verified and signed_by)},
|
||||
"staging_id": staging_id, "kind": kind,
|
||||
"created_at": _now_iso(),
|
||||
}
|
||||
@@ -2411,7 +2414,8 @@ def _file_sha(path: Path) -> str | None:
|
||||
|
||||
|
||||
async def commit_staging(session: AsyncSession, staging: ImportStaging, *, trial_id: str, confirmations: list[str],
|
||||
acceptances: list[Acceptance], signed_by: str | None) -> CommitResult:
|
||||
acceptances: list[Acceptance], signed_by: str | None,
|
||||
signer_verified: bool = False) -> CommitResult:
|
||||
"""POST imports/{id}/commit(7.8):预检 → (与现行导入相同时)不新建记录 → 发布构建、写配方、导入记录、
|
||||
快照、清单,切指针。失败时指针不动,暂存区退回 drafting(试运行库已被发布流程消耗)。
|
||||
|
||||
@@ -2532,6 +2536,8 @@ async def commit_staging(session: AsyncSession, staging: ImportStaging, *, trial
|
||||
|
||||
ex = extraction_from_dict(ex_dict)
|
||||
who = signed_by or staging.signed_by
|
||||
# 署名是这次请求的(signed_by)才谈得上认证;退回暂存区上记的那个,是否认证这里不知道,照实记未认证
|
||||
who_verified = bool(signer_verified and signed_by)
|
||||
acc = [Acceptance(check_id=a.check_id, reason=a.reason.strip(), signed_by=who) for a in given.values()]
|
||||
# 这一期自己的说明(带实际的接受)进它的导入清单:版本页逐期显示「改写后的那一句」从这里取(第 5 节)。
|
||||
# 快照冻结的说明在多期时另按各期最弱的结论合并(2.8),只剩本期时就是它
|
||||
@@ -2603,7 +2609,8 @@ async def commit_staging(session: AsyncSession, staging: ImportStaging, *, trial
|
||||
file={"name": file_name, "size": file_size, "raw_sha256": raw_sha},
|
||||
recipe_meta={"id": recipe_id, "sha256": recipe_sha, "origin": origin, "canonical": canonical},
|
||||
checks=checks_full, overrides=overrides, waivers=waivers, confirmed=confirmed,
|
||||
notes=notes if union_info is None else period_notes, ai=ai, signed_by=who, edits=edits)
|
||||
notes=notes if union_info is None else period_notes, ai=ai, signed_by=who, edits=edits,
|
||||
signer_verified=who_verified)
|
||||
|
||||
async def schema_cache_for(view: table_versions.SourceView, info: table_versions.PublishInfo) -> dict[str, Any]:
|
||||
from app.core import artifact_store
|
||||
@@ -2663,7 +2670,7 @@ async def commit_staging(session: AsyncSession, staging: ImportStaging, *, trial
|
||||
"table_hashes": dict(union_report.get("table_hashes") or {}), "union_ver": recipe_types.UNION_VER},
|
||||
parts=ordered, report=union_report, label_sets=list((plan or {}).get("label_sets") or []),
|
||||
gaps=list((plan or {}).get("gaps") or []), replaced=(plan or {}).get("replaces"), removed=None,
|
||||
notes=notes, signed_by=who)
|
||||
notes=notes, signed_by=who, signer_verified=who_verified)
|
||||
smid = await artifact_store.put_json(doc, kind="snapshot_manifest",
|
||||
meta={"source_id": source_id, "snapshot_id": info.snapshot_id})
|
||||
cache["import_manifests"] = [mid if iid is None else by_imp[iid].info.manifest_artifact
|
||||
|
||||
@@ -27,6 +27,9 @@ from app.data.names import canon
|
||||
from app.data.recipe_checks import date_columns, plan_checks, split_coord
|
||||
from app.data.recipe_parsers import UNITS
|
||||
from app.data.recipe_types import (
|
||||
LIST_PERIOD_END,
|
||||
LIST_PERIOD_START,
|
||||
SUBTOTAL_KEY,
|
||||
Acceptance,
|
||||
CheckResult,
|
||||
ColumnOut,
|
||||
@@ -46,6 +49,7 @@ from app.data.recipe_types import (
|
||||
TableNote,
|
||||
UnionNotePart,
|
||||
derive_tables,
|
||||
list_period_columns,
|
||||
prose_problems,
|
||||
render_prose,
|
||||
)
|
||||
@@ -134,6 +138,13 @@ NOTE_TEXT = {
|
||||
"ignored_rows_union": "部分期的原表另有按配方忽略的行,内容见证据面板",
|
||||
"ignored_cells_union": "部分期的原表另有按配方忽略的单元格,内容见证据面板",
|
||||
"dim_values_vary": "各期的 {dim} 取值不完全相同:跨期比较前请先确认两期都有的取值",
|
||||
# ---- 期 5:列表按期累积、穿插的小计。列表可以累积以后,空行之后的文字行也要有并集说法
|
||||
"rows_after_stop_union": "部分期的原表在空行之后另有未导入的文字行,内容见证据面板",
|
||||
"subtotal_kept": "原表穿插的小计在 {table},不要与本表相加",
|
||||
"subtotal_not_kept": "原表穿插的小计行未导入本表",
|
||||
"list_subtotal_passed": "原表穿插的小计行的值,已按各组明细求和核对一致:不要与 {base} 相加,也不要彼此相加",
|
||||
"list_subtotal_unverified": "原表穿插的小计行的值,按原表保存的值存储。本期部分小计未能核对:不要与 {base} 相加,"
|
||||
"也不要彼此相加",
|
||||
}
|
||||
|
||||
#: 列说明的片段模板
|
||||
@@ -148,6 +159,9 @@ COLUMN_TEXT = {
|
||||
"col_partial_periods": "部分期没有这一列的数据,为空值",
|
||||
"col_absent": "当前版本的数据中没有这一列,为空值",
|
||||
"col_retired": "这一列自某一期起不再导入,此后各期为空值",
|
||||
# ---- 期 5:列表按期累积时由系统加的统计期起止两列(ListPeriod.synthesize)。不含数字(说明不许报数)
|
||||
"period_start": "本期统计期的第一天,取自表内的统计期或人工录入的统计期,格式 YYYY-MM-DD",
|
||||
"period_end": "本期统计期的最后一天,取自表内的统计期或人工录入的统计期,格式 YYYY-MM-DD",
|
||||
}
|
||||
|
||||
#: 「逐日 / 逐行」「个别日期 / 个别行」:粒度只有一列日期时按日说,否则按行说
|
||||
@@ -161,7 +175,7 @@ _K_REASON = {r: f"total_k_{r}" for r in ("uncached", "blank", "null_detail", "ti
|
||||
_T_REASON = {r: f"list_total_{r}" for r in ("uncached", "blank", "null_detail", "tiling", "hidden")}
|
||||
#: 允许出现「一致」的模板:只在对应的 K / T 通过时才会选用
|
||||
CONSISTENT_KEYS = frozenset({"total_k_passed", "total_k_passed_g_unverifiable", "list_total_passed",
|
||||
"list_total_passed_partial"})
|
||||
"list_total_passed_partial", "list_subtotal_passed"})
|
||||
|
||||
|
||||
def _tok(kind: str, name: str) -> str:
|
||||
@@ -202,6 +216,8 @@ class _Writers:
|
||||
lists: list[ListBlock] = field(default_factory=list)
|
||||
#: 把合计行另存进本表的列表块
|
||||
list_totals: list[ListBlock] = field(default_factory=list)
|
||||
#: 期 5:把穿插的小计另存进本表的列表块
|
||||
list_subtotals: list[ListBlock] = field(default_factory=list)
|
||||
|
||||
def add_sheet(self, sheet: SheetRecipe) -> None:
|
||||
if all(s is not sheet for s in self.sheets):
|
||||
@@ -242,6 +258,9 @@ def _writers(recipe: Recipe) -> dict[str, _Writers]:
|
||||
total = block.rows.total_row
|
||||
if total is not None and total.keep_as:
|
||||
w(total.keep_as, sheet).list_totals.append(block)
|
||||
sub = block.rows.subtotals
|
||||
if sub is not None and sub.keep_as:
|
||||
w(sub.keep_as, sheet).list_subtotals.append(block)
|
||||
return out
|
||||
|
||||
|
||||
@@ -395,6 +414,15 @@ def _base_total_fragments(w: _Writers) -> list[NoteFragment]:
|
||||
b.rows.total_row is not None and not b.rows.total_row.keep_as for b in w.lists)
|
||||
if dropped:
|
||||
out.append(NoteFragment("total_not_kept", None, NOTE_TEXT["total_not_kept"]))
|
||||
# 期 5:穿插的小计行不在本表里:另存在哪,或者没导入
|
||||
for b in w.lists:
|
||||
sub = b.rows.subtotals
|
||||
if sub is None:
|
||||
continue
|
||||
if sub.keep_as:
|
||||
_add_unique(out, _frag("subtotal_kept", sub.keep_as, NOTE_TEXT["subtotal_kept"].format(table=_tab(sub.keep_as))))
|
||||
elif all(f.key != "subtotal_not_kept" for f in out):
|
||||
out.append(NoteFragment("subtotal_not_kept", None, NOTE_TEXT["subtotal_not_kept"]))
|
||||
return out
|
||||
|
||||
|
||||
@@ -450,6 +478,13 @@ def _kept_total_fragments(ctx: _Ctx, table: str, w: _Writers) -> list[NoteFragme
|
||||
else:
|
||||
frag = _frag(key, table, NOTE_TEXT[key].format(base=_tab(block.table)))
|
||||
_add_unique(out, frag)
|
||||
for block in w.list_subtotals:
|
||||
# 期 5:另存的小计:T 通过才说「核对一致」,没核对上的(无法核对)照实说部分未能核对;不一致的整份拒收
|
||||
t = ctx.check(table, ctx.plan.t.get(f"{block.id}{SUBTOTAL_KEY}"), f"块「{block.id}」的小计行")
|
||||
if t is None or _blocking(t):
|
||||
continue
|
||||
key = "list_subtotal_passed" if t.status == "passed" else "list_subtotal_unverified"
|
||||
_add_unique(out, _frag(key, table, NOTE_TEXT[key].format(base=_tab(block.table))))
|
||||
return out
|
||||
|
||||
|
||||
@@ -653,10 +688,17 @@ def _date_key(ctx: _Ctx, w: _Writers, table: str, col: ColumnOut) -> str | None:
|
||||
return None
|
||||
|
||||
|
||||
def _period_column_key(col: ColumnOut) -> str | None:
|
||||
"""期 5:系统加的统计期起止两列(role=period)用哪条列说明。"""
|
||||
if col.role != "period":
|
||||
return None
|
||||
return "period_start" if col.name == LIST_PERIOD_START else "period_end" if col.name == LIST_PERIOD_END else None
|
||||
|
||||
|
||||
def _null_eligible(ctx: _Ctx, w: _Writers, table: str, col: ColumnOut) -> bool:
|
||||
"""这一列要不要考虑空值说明:值列;主键不会是空值;表内合计表的空值已由表说明(未能核对)交代。"""
|
||||
value_col = col.role in ("measure", "value") and col.type in ("INTEGER", "REAL")
|
||||
is_total = bool(w.kept_from or w.list_totals)
|
||||
is_total = bool(w.kept_from or w.list_totals or w.list_subtotals)
|
||||
return value_col and not is_total and col.name not in set(ctx.grains.get(table, []))
|
||||
|
||||
|
||||
@@ -668,6 +710,8 @@ def _column_templates(ctx: _Ctx, table: str, null_counts: Mapping[str, Mapping[s
|
||||
frags: list[str] = []
|
||||
if (date := _date_key(ctx, w, table, col)) is not None:
|
||||
frags.append(COLUMN_TEXT[date])
|
||||
if (period := _period_column_key(col)) is not None:
|
||||
frags.append(COLUMN_TEXT[period])
|
||||
if col.unit:
|
||||
frags.append(COLUMN_TEXT["unit"].format(unit=_tok("单位", col.unit)))
|
||||
has_nulls = counts is None or counts.get(col.name, 0) > 0
|
||||
@@ -793,22 +837,27 @@ _FAMILY: dict[str, str] = {
|
||||
"hidden_excluded": "hidden_excluded", "hidden_included": "hidden_included",
|
||||
"ignored_columns": "ignored_columns", "ignored_rows": "ignored_rows", "ignored_cells": "ignored_cells",
|
||||
"rows_after_stop": "rows_after_stop",
|
||||
"subtotal_kept": "subtotal_kept", "subtotal_not_kept": "subtotal_not_kept",
|
||||
"list_subtotal_passed": "list_total", "list_subtotal_unverified": "list_total",
|
||||
"period_conflict_accepted": "period_conflict", "period_conflict_pending": "period_conflict",
|
||||
"period_human": "period_human", "outside_digits": "outside_digits",
|
||||
"note": "note",
|
||||
}
|
||||
#: 合并后的拼接顺序(与 _table_fragments 一致;累积特有的几句排在用户写的表说明之前)
|
||||
_FAMILY_ORDER = ("grain", "sum_eq", "not_comparable", "hour_range", "const", "total_kept", "total_not_kept", "total_k",
|
||||
"list_total", "total_overlap", "hidden_excluded", "hidden_included", "ignored_columns",
|
||||
_FAMILY_ORDER = ("grain", "sum_eq", "not_comparable", "hour_range", "const", "total_kept", "total_not_kept",
|
||||
"subtotal_kept", "subtotal_not_kept", "total_k", "list_total", "total_overlap", "hidden_excluded", "hidden_included", "ignored_columns",
|
||||
"ignored_rows", "ignored_cells", "rows_after_stop", "period_conflict", "period_human",
|
||||
"outside_digits", "union", "note")
|
||||
#: 与核对结论无关、只描述表结构的片段族:用目标配方那一期的;只在早期出现的 subject(退役的表、另存表)用
|
||||
#: 最后一个含它的那一期的
|
||||
_STRUCTURAL = frozenset({"grain", "hour_range", "const", "total_kept", "total_not_kept", "total_overlap", "note"})
|
||||
_STRUCTURAL = frozenset({"grain", "hour_range", "const", "total_kept", "total_not_kept", "total_overlap", "note",
|
||||
"subtotal_kept", "subtotal_not_kept"})
|
||||
#: 「任一期出现就写」的片段族 → 并集里的说法
|
||||
_ANY_UNION = {"hidden_excluded": "hidden_excluded_union", "hidden_included": "hidden_included_union",
|
||||
"ignored_columns": "ignored_columns_union", "ignored_rows": "ignored_rows_union",
|
||||
"ignored_cells": "ignored_cells_union", "outside_digits": "outside_digits_union"}
|
||||
"ignored_cells": "ignored_cells_union", "outside_digits": "outside_digits_union",
|
||||
# 期 5:列表可以按期累积了,空行之后的文字行任一期有就写
|
||||
"rows_after_stop": "rows_after_stop_union"}
|
||||
_K_PASSED = frozenset({"total_k_passed", "total_k_passed_g_unverifiable"})
|
||||
|
||||
|
||||
@@ -943,20 +992,26 @@ def _merge_group(table: str, family: str, subject: str | None, group: dict[int,
|
||||
if family == "period_human":
|
||||
key = "period_human_all" if len(present) == len(having) else "period_human_union"
|
||||
return NoteFragment(key, subject, NOTE_TEXT[key])
|
||||
# list_total、rows_after_stop 只有列表才有,而列表不能按期累积(P3-SPEC 2.2);不认识的片段族同样没有并集说法。
|
||||
# list_total 只在列表的合计行、小计行另存的表上,这种表不能按期累积(accumulate_blockers);不认识的片段族同样
|
||||
# 没有并集说法。
|
||||
# 记 problems(不能保存),绝不沿用某一期的单期说法。消息给人看,不露片段族的键名
|
||||
problems.append(f"表「{table}」:{_UNMERGEABLE.get(family, '有一段说明')}不能按期合并,无法生成说明")
|
||||
return None
|
||||
|
||||
|
||||
#: 没有并集说法的片段族 → 给人看的叫法(problems 里不露键名)
|
||||
_UNMERGEABLE = {"list_total": "列表合计行的核对结论", "rows_after_stop": "列表空行之后的文字行"}
|
||||
_UNMERGEABLE = {"list_total": "列表合计行、小计行的核对结论"}
|
||||
|
||||
|
||||
def _axis_column(recipe: Recipe, table: str) -> str | None:
|
||||
"""各期按哪一列区分(说明里「请按 {col} 筛选」):交叉表的日期轴;期 5 起列表按 period 绑定的日期列,或者
|
||||
系统加的统计期起那一列。"""
|
||||
tables, _ = derive_tables(recipe)
|
||||
axes = [c.name for c in tables.get(table, []) if c.role == "axis"]
|
||||
return axes[0] if len(axes) == 1 else None
|
||||
if len(axes) == 1:
|
||||
return axes[0]
|
||||
spec = list_period_columns(recipe).get(table)
|
||||
return spec[1] if spec else None
|
||||
|
||||
|
||||
def _union_table_fragments(table: str, having: list[int], parts: list[_UPart], ref: int, target: Recipe,
|
||||
@@ -1037,6 +1092,8 @@ def _union_column_templates(table: str, columns: list[ColumnOut], having: list[i
|
||||
frags.append(COLUMN_TEXT["date_year_from_period"])
|
||||
elif "date" in dates or (not with_col and col.role == "axis"):
|
||||
frags.append(COLUMN_TEXT["date"])
|
||||
if (period := _period_column_key(col)) is not None:
|
||||
frags.append(COLUMN_TEXT[period])
|
||||
if target_cols is not None and col.name in target_cols:
|
||||
unit = target_cols[col.name].unit
|
||||
elif with_col:
|
||||
|
||||
@@ -427,6 +427,21 @@ def _bottom_box(ctx: _Ctx, plan: Any, first: int) -> bool:
|
||||
ctx.fail("selection_bottom_not_anchorable",
|
||||
f"框里只有表头和合计行(第 {last} 行),没有数据行:请把框扩大到数据所在的行")
|
||||
return False
|
||||
# 期 5:框里夹在中间、第一列以合计字样开头、下面紧接着还有同一张表的明细的行(起草器 _scan_block 的同一条
|
||||
# 规则),照起草器提议成小计(_settle_subtotals);提议不成的照旧报错
|
||||
mid = [r for r in data[1:] if S._total_word(S._label_of(grid.get(r, c1))) is not None]
|
||||
rest = [r for r in data if r not in mid]
|
||||
if mid and rest and not any(grid.get(r, c) is not None and S._total_word(grid.get(r, c).value) is not None
|
||||
for r in data for c in range(c1 + 1, c2 + 1)):
|
||||
trial = S._ListPlan(header_rows=plan.header_rows, cols=plan.cols, headers=plan.headers, rows=rest,
|
||||
subtotal_rows=list(mid),
|
||||
subtotal_words=[S._total_word(S._label_of(grid.get(r, c1))) or "" for r in mid])
|
||||
types = S._types_of(grid, rest, plan.cols)
|
||||
if all(S._more_rows(grid, r, trial, S._bottom(grid), types) for r in mid):
|
||||
S._settle_subtotals(grid, trial)
|
||||
if trial.subtotal_word is not None:
|
||||
plan.subtotal_rows, plan.subtotal_word = trial.subtotal_rows, trial.subtotal_word
|
||||
data = [r for r in data if r not in mid]
|
||||
if not _no_stray_totals(ctx, data, c1, c2):
|
||||
return False
|
||||
bottom = S._bottom(grid)
|
||||
|
||||
@@ -12,6 +12,11 @@ validate)和执行器干跑(注入的 dry_run),再由人逐条确认之
|
||||
哪些是区域外文字、哪些格没有去处)直接用执行器的结论:同一件事只有执行器一套判法,这里自己定位只是没有
|
||||
干跑时的近似。
|
||||
|
||||
**期 5(下一轮 03 细案 E)。** 有 Excel 表格对象(插入 → 表格)的工作表优先按表格对象出列表块(_lists_from_tables:
|
||||
表头、列范围、汇总行都是表格对象给的,锚点仍按表头文字);夹在明细中间、第一列以同一个合计字样开头的行提议成
|
||||
穿插的小计(_settle_subtotals),不再一律记起草失败;每个列表都问一道「每期怎么更新」(q_list_period),选按期
|
||||
累积时同时绑定每一行属于哪一期(按日期列核对,或加统计期起止两列)。
|
||||
|
||||
**问题的 effects 怎么写。** 每个选项是一组 JSON Patch,路径按传进来的配方写;接口层从 answers_base 起、按
|
||||
问题顺序重放全部回答(P2-SPEC 7.2)。因此:
|
||||
- 关系的两个选项都用整条 replace(「登记」写回完整的 sum_eq / not_comparable,「不登记」写 dismissed),
|
||||
@@ -63,7 +68,7 @@ from app.data.recipe_types import (
|
||||
derive_tables,
|
||||
)
|
||||
from app.data.tabular import DATE_HEADER_MIN, EXCEL_ERRORS
|
||||
from app.data.xlsx_scan import WorkbookScan, cell_ref, col_letter, parse_ref
|
||||
from app.data.xlsx_scan import TableScan, WorkbookScan, cell_ref, col_letter, parse_ref
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -612,8 +617,15 @@ class _ListPlan:
|
||||
#: 按数据行收进来并记起草失败
|
||||
adjacent: list[tuple[int, list[int]]] = field(default_factory=list)
|
||||
#: 不在末尾第一列的合计字样:(行, 列)。第一列在中间或最上面、或者落在别的列上,都无法按合计行核对,
|
||||
#: 记起草失败;不记就会把合计当明细导入,求和翻倍(H2)
|
||||
#: 记起草失败;不记就会把合计当明细导入,求和翻倍(H2)。期 5:第一列里同一个字样的穿插行先记进 subtotal_rows,
|
||||
#: 收完整块再判能不能提议成小计(_settle_subtotals);判不了的才回到这里
|
||||
stray_totals: list[tuple[int, int]] = field(default_factory=list)
|
||||
#: 期 5:穿插在明细中间、第一列以合计字样开头的行(不进 rows)和它们的字样;能提议成小计时 subtotal_word 非空
|
||||
subtotal_rows: list[int] = field(default_factory=list)
|
||||
subtotal_words: list[str] = field(default_factory=list)
|
||||
subtotal_word: str | None = None
|
||||
#: 期 5:按 Excel 的表格对象起草的(xlsx_scan.TableScan 的名字):表名优先用它,卡片里写明
|
||||
table_object: str | None = None
|
||||
# 命名阶段填
|
||||
id: str = ""
|
||||
table: str = ""
|
||||
@@ -808,13 +820,44 @@ def _scan_block(grid: Grid, plan: _ListPlan, bottom: int) -> None:
|
||||
if word is not None and plan.rows and not _more_rows(grid, r, plan, bottom, types):
|
||||
plan.total_row, plan.total_word = r, word
|
||||
break
|
||||
others = [(r, c) for c, cell in zip(cols[1:], cells[1:])
|
||||
if cell is not None and _total_word(cell.value) is not None]
|
||||
plan.stray_totals.extend(others)
|
||||
if word is not None and plan.rows and not others:
|
||||
# 期 5:夹在明细中间的合计字样,先当小计候选:不进数据行,收完整块再判(_settle_subtotals)
|
||||
plan.subtotal_rows.append(r)
|
||||
plan.subtotal_words.append(word)
|
||||
r += 1
|
||||
continue
|
||||
if word is not None:
|
||||
plan.stray_totals.append((r, cols[0]))
|
||||
plan.stray_totals.extend((r, c) for c, cell in zip(cols[1:], cells[1:])
|
||||
if cell is not None and _total_word(cell.value) is not None)
|
||||
plan.rows.append(r)
|
||||
acc.add(r)
|
||||
r += 1
|
||||
_settle_subtotals(grid, plan)
|
||||
|
||||
|
||||
def _settle_subtotals(grid: Grid, plan: _ListPlan) -> None:
|
||||
"""期 5:夹在明细中间的合计字样能不能提议成小计(ListRows.subtotals,规则起草不再把它们记成失败)。
|
||||
|
||||
能的条件:都在第一列、是同一个字样,每一行至少有一格数(没有数的小计核对不了)。末尾那一行和它们字样相同时
|
||||
(表以最后一组的小计结束,没有总计),它也是小计,不是合计行。不能的(字样不同、有一行没有数)照旧记进
|
||||
stray_totals(起草失败,交给人或 AI),这些行回到数据行里——和期 4 的结果一样。"""
|
||||
if not plan.subtotal_rows:
|
||||
return
|
||||
words = set(plan.subtotal_words)
|
||||
numeric = [c for c in plan.cols[1:] if any(_value_num(grid.get(r, c)) is not None for r in plan.rows)]
|
||||
ok = len(words) == 1 and all(any(_value_num(grid.get(r, c)) is not None for c in numeric)
|
||||
for r in plan.subtotal_rows)
|
||||
if ok:
|
||||
plan.subtotal_word = next(iter(words))
|
||||
if plan.total_row is not None and plan.total_word == plan.subtotal_word:
|
||||
plan.subtotal_rows.append(plan.total_row)
|
||||
plan.total_row = plan.total_word = None
|
||||
return
|
||||
plan.stray_totals.extend((r, plan.cols[0]) for r in plan.subtotal_rows)
|
||||
plan.rows = sorted(set(plan.rows) | set(plan.subtotal_rows))
|
||||
plan.subtotal_rows, plan.subtotal_words = [], []
|
||||
|
||||
|
||||
def _list_failures(sheet: str, grid: Grid, plan: _ListPlan) -> list[_Failure]:
|
||||
@@ -898,7 +941,8 @@ def _analyze_lists(grid: Grid) -> _Lists:
|
||||
_finish_block(grid, plan)
|
||||
out.failures.extend(_list_failures(grid.sheet, grid, plan))
|
||||
out.blocks.append(plan)
|
||||
end = max((p.total_row or (p.rows[-1] if p.rows else p.header_rows[-1])) for p in made)
|
||||
end = max(max([p.total_row or 0, *p.subtotal_rows, p.rows[-1] if p.rows else p.header_rows[-1]])
|
||||
for p in made)
|
||||
while r_idx < len(rows) and rows[r_idx] <= end:
|
||||
r_idx += 1
|
||||
return out
|
||||
@@ -945,7 +989,7 @@ def _finish_block(grid: Grid, plan: _ListPlan) -> None:
|
||||
break
|
||||
if cells:
|
||||
break
|
||||
last = plan.total_row or rows[-1]
|
||||
last = max([plan.total_row or 0, rows[-1], *plan.subtotal_rows])
|
||||
for r in range(plan.header_rows[0], last + 1):
|
||||
for c in cols:
|
||||
if grid.get(r, c) is not None or r in plan.header_rows:
|
||||
@@ -975,15 +1019,77 @@ class _Sheet:
|
||||
hidden_cols: list[int] = field(default_factory=list)
|
||||
|
||||
|
||||
def _analyze_sheet(grid: Grid, sid: str) -> _Sheet:
|
||||
def _lists_from_tables(grid: Grid, objects: list[TableScan]) -> _Lists | None:
|
||||
"""期 5:按 Excel 的表格对象(插入 → 表格)起草列表块。表格对象明确给出了表头行、列范围、数据行和汇总行,比按
|
||||
「像不像表头」猜可靠:数字格空着的行不会被当成表头,紧挨着的两张表也分得开。
|
||||
|
||||
出来的块和按规则猜的一样只记文字:锚点仍按表头文字(不记坐标,下一期表格挪了位置照样认得出),汇总行按第一列
|
||||
的文字映射成合计行(total_row),夹在中间的合计字样按小计提议(_settle_subtotals)。表格对象的 ref 写坏了、
|
||||
没有表头行时返回 None,退回按规则猜。"""
|
||||
out = _Lists(sheet=grid.sheet)
|
||||
spans = sorted(((t.span(), t) for t in objects), key=lambda x: (x[0] or (0, 0, 0, 0))[:2])
|
||||
for span, t in spans:
|
||||
if span is None or t.header_row_count < 1:
|
||||
return None
|
||||
r1, c1, r2, c2 = span
|
||||
hrows = list(range(r1, r1 + t.header_row_count))
|
||||
cols = list(range(c1, c2 + 1))
|
||||
texts = _header_texts(grid, hrows, cols)
|
||||
if set(texts) != set(cols):
|
||||
blank = [col_letter(c) for c in cols if c not in texts]
|
||||
msg = (f"工作表「{grid.sheet}」:表格「{t.name}」的表头有空着的列({'、'.join(blank)} 列),"
|
||||
"按列表导入要每一列都有表头文字")
|
||||
out.failures.append(_Failure(msg, msg))
|
||||
continue
|
||||
plan = _ListPlan(header_rows=hrows, cols=cols, headers={c: texts[c] for c in cols}, table_object=t.name)
|
||||
last_data = r2 - max(t.totals_row_count, 0)
|
||||
rows = [r for r in grid.row_numbers() if hrows[-1] < r <= last_data and any(grid.get(r, c) is not None
|
||||
for c in cols)]
|
||||
prev = hrows[-1]
|
||||
for r in rows:
|
||||
if r > prev + 1:
|
||||
plan.skip_blank = True
|
||||
plan.skipped.extend(range(prev + 1, r))
|
||||
prev = r
|
||||
word = _total_word(_label_of(grid.get(r, c1)))
|
||||
if word is not None and plan.rows:
|
||||
plan.subtotal_rows.append(r)
|
||||
plan.subtotal_words.append(word)
|
||||
continue
|
||||
plan.rows.append(r)
|
||||
if not plan.rows:
|
||||
msg = f"工作表「{grid.sheet}」:表格「{t.name}」的表头下面没有数据行"
|
||||
out.failures.append(_Failure(msg, msg))
|
||||
continue
|
||||
if t.totals_row_count > 0:
|
||||
tr = last_data + 1
|
||||
label = _label_of(grid.get(tr, c1))
|
||||
word = _total_word(label) or (label if label and len(label) <= 12 and not any(
|
||||
ch.isdigit() for ch in label) else None)
|
||||
if word is not None:
|
||||
plan.total_row, plan.total_word = tr, word
|
||||
else:
|
||||
why = ("起草只读了表格前面的部分,没读到汇总行" if grid.truncated and tr > max(grid.row_numbers())
|
||||
else "汇总行的第一列没有可用作合计标签的文字(不能含数字、不超过 12 个字)")
|
||||
msg = f"工作表「{grid.sheet}」:表格「{t.name}」的汇总行(第 {tr} 行)无法按合计行核对:{why}"
|
||||
out.failures.append(_Failure(msg, msg))
|
||||
_settle_subtotals(grid, plan)
|
||||
_finish_block(grid, plan)
|
||||
out.failures.extend(_list_failures(grid.sheet, grid, plan))
|
||||
out.blocks.append(plan)
|
||||
return out
|
||||
|
||||
|
||||
def _analyze_sheet(grid: Grid, sid: str, objects: list[TableScan] | None = None) -> _Sheet:
|
||||
sh = _Sheet(sheet=grid.sheet, sid=sid)
|
||||
axis = _axis_row(grid)
|
||||
from_objects = _lists_from_tables(grid, objects) if objects and axis is None else None
|
||||
if axis is not None:
|
||||
sh.cross = _analyze_crosstab(grid, *axis)
|
||||
sh.region = set(sh.cross.region)
|
||||
sh.failures.extend(sh.cross.failures)
|
||||
else:
|
||||
sh.lists = _analyze_lists(grid)
|
||||
sh.lists = from_objects if from_objects is not None else _analyze_lists(grid)
|
||||
sh.failures.extend(sh.lists.failures)
|
||||
for b in sh.lists.blocks:
|
||||
sh.region |= b.region
|
||||
@@ -1111,7 +1217,8 @@ def _measure_column(label: str, pos: int) -> tuple[str, str | None, str | None]:
|
||||
def _analyze(scan: WorkbookScan, grids: dict[str, Grid]) -> _Analysis:
|
||||
visible = _visible_grids(scan, grids)
|
||||
hidden = [s.name for s in scan.sheets if s.state != "visible"]
|
||||
sheets = [_analyze_sheet(g, f"s{i + 1}") for i, g in enumerate(visible)]
|
||||
sheets = [_analyze_sheet(g, f"s{i + 1}", [t for t in scan.tables if t.sheet == g.sheet])
|
||||
for i, g in enumerate(visible)]
|
||||
failures = [f for sh in sheets for f in sh.failures]
|
||||
|
||||
tables_used: set[str] = set()
|
||||
@@ -1266,7 +1373,9 @@ def list_block(sheet: str, b: _ListPlan, *, block_id: str, tables_used: set[str]
|
||||
合计表名撞了已用的名字时照起草器的老行为:不另建表定义,keep_as 仍写这个名字(交给静态校验报)。"""
|
||||
b.id = block_id
|
||||
if table is None:
|
||||
title = b.title if b.title and not any(ch.isdigit() for ch in b.title) else None
|
||||
# 期 5:按表格对象起草的,表名优先用表格对象的名字(不含数字时;Excel 默认的「表1」不用)
|
||||
named = b.table_object if b.table_object and not any(ch.isdigit() for ch in b.table_object) else None
|
||||
title = named or (b.title if b.title and not any(ch.isdigit() for ch in b.title) else None)
|
||||
want = _ENUM_PREFIX.sub("", P.split_unit_suffix(title)[0]) if title else sheet
|
||||
n_fallback = len(tables_used) + 1
|
||||
b.table = _unique(to_sql_name(want, fallback=f"表{n_fallback}") if want else f"表{n_fallback}", tables_used)
|
||||
@@ -1293,6 +1402,11 @@ def list_block(sheet: str, b: _ListPlan, *, block_id: str, tables_used: set[str]
|
||||
total_spec = {"name": keep, "grain": ["合计项"], "kind": "reported_total", "units": num_units, "note": ""}
|
||||
b.keep_table = keep
|
||||
total = {"label_column": b.names[b.cols[0]], "pick": b.total_word, "keep_as": keep}
|
||||
subtotals = None
|
||||
if b.subtotal_word is not None:
|
||||
# 期 5:穿插的小计只核对、不另存(另存的小计表不能按期累积,要另存由人在配方面板里打开)
|
||||
subtotals = {"label_column": b.names[b.cols[0]], "pick": b.subtotal_word, "scope": "since_previous",
|
||||
"group_column": None, "keep_as": None}
|
||||
grain = [b.names[b.grain_col]] if b.grain_col is not None else []
|
||||
main = {"name": b.table, "grain": grain, "kind": "data", "units": dict(b.units), "note": ""}
|
||||
fill = bool(b.merged_fill) and all(r2 > r1 and c1 == c2 and b.types.get(c1) == "TEXT"
|
||||
@@ -1300,7 +1414,8 @@ def list_block(sheet: str, b: _ListPlan, *, block_id: str, tables_used: set[str]
|
||||
block = {
|
||||
"id": b.id, "layout": "list", "table": b.table, "header_rows": len(b.header_rows),
|
||||
"after_title": None, "columns": cols_out, "extra_columns": "reject",
|
||||
"rows": {"blank_rows": "skip" if b.skip_blank else "stop", "total_row": total},
|
||||
"rows": {"blank_rows": "skip" if b.skip_blank else "stop", "total_row": total,
|
||||
**({"subtotals": subtotals} if subtotals is not None else {})},
|
||||
"values": {"placeholders": [{"text": e[0], "meaning": "无数据"} for e in b.placeholders.values()],
|
||||
"blank": "null", "text_number": "reject", "formula": "accept_cached"},
|
||||
"merged_data": "fill" if fill else "reject",
|
||||
@@ -1797,6 +1912,8 @@ class _ListLoc:
|
||||
rows: list[int]
|
||||
total_row: int | None
|
||||
region: set[tuple[int, int]] = field(default_factory=set)
|
||||
#: 期 5:穿插的小计行
|
||||
subtotal_rows: list[int] = field(default_factory=list)
|
||||
|
||||
|
||||
def _find_grid(name: str, grids: dict[str, Grid]) -> Grid | None:
|
||||
@@ -1943,19 +2060,26 @@ def _locate_list(block: ListBlock, grid: Grid, taken: set[int]) -> _ListLoc | No
|
||||
bottom = _bottom(grid)
|
||||
rr = r + 1
|
||||
pick = P.match_key(block.rows.total_row.pick) if block.rows.total_row else None
|
||||
sub_pick = P.match_key(block.rows.subtotals.pick) if block.rows.subtotals else None
|
||||
subs: list[int] = []
|
||||
while rr <= bottom:
|
||||
if all(grid.get(rr, c) is None for c in cols):
|
||||
if block.rows.blank_rows == "skip":
|
||||
rr += 1
|
||||
continue
|
||||
break
|
||||
if pick and P.match_key(_label_of(grid.get(rr, cols[0])) or "").startswith(pick):
|
||||
label = P.match_key(_label_of(grid.get(rr, cols[0])) or "")
|
||||
if pick and label.startswith(pick):
|
||||
total = rr
|
||||
break
|
||||
if sub_pick and label.startswith(sub_pick):
|
||||
subs.append(rr)
|
||||
rr += 1
|
||||
continue
|
||||
rows.append(rr)
|
||||
rr += 1
|
||||
loc = _ListLoc(hrows, cols, rows, total)
|
||||
for row in [*hrows, *rows, *([total] if total else [])]:
|
||||
loc = _ListLoc(hrows, cols, rows, total, subtotal_rows=subs)
|
||||
for row in [*hrows, *rows, *subs, *([total] if total else [])]:
|
||||
loc.region.update((row, c) for c in cols)
|
||||
return loc
|
||||
|
||||
@@ -2130,6 +2254,9 @@ def questions_for(recipe: dict[str, Any], facts: DraftFacts, grids: dict[str, Gr
|
||||
if ext is not None and ext.sheets.skipped_hidden:
|
||||
for nm in _dedupe([item.get("sheet", "") for item in ext.sheets.skipped_hidden]):
|
||||
cards.append(_hidden_sheet_card(nm))
|
||||
# 期 5:列表每期怎么更新的问题排在其余问题之后、q_mode 之前:它只和按期累积有关,不该顶掉期 2 起排在前面的问题
|
||||
questions = ([q for q in questions if not q.id.startswith("q_list_period:")]
|
||||
+ [q for q in questions if q.id.startswith("q_list_period:")])
|
||||
blockers = accumulate_blockers(parsed)
|
||||
cards.append(_mode_card(blockers))
|
||||
if not blockers:
|
||||
@@ -2543,9 +2670,57 @@ def _list_cards(si: int, bi: int, sheet: Any, block: ListBlock, loc: _ListLoc |
|
||||
reason += f",原值另存表「{total.keep_as}」" if total.keep_as else ",只核对不另存"
|
||||
cells = _coords(name, [(loc.total_row, loc.cols[0])]) if loc is not None and loc.total_row else []
|
||||
cards.append(Card(f"total_row:{block.id}", f"{where}是合计行,改作核对", reason, cells))
|
||||
subt = block.rows.subtotals
|
||||
if subt is not None:
|
||||
rows = loc.subtotal_rows if loc is not None else []
|
||||
where = _rows_text(rows) if rows else f"「{subt.pick}」开头的行"
|
||||
scope = ("上一行小计之后的全部明细" if subt.scope == "since_previous"
|
||||
else f"紧挨在它上面、「{subt.group_column}」相同的那几行明细")
|
||||
reason = f"每行小计按列求和与{scope}核对,小计行不进数据"
|
||||
reason += f",原值另存表「{subt.keep_as}」" if subt.keep_as else ",只核对不另存"
|
||||
cells = _coords(name, [(r, loc.cols[0]) for r in rows]) if loc is not None else []
|
||||
cards.append(Card(f"subtotals:{block.id}", f"{where}是穿插的小计行,改作核对", reason, cells))
|
||||
q = _list_period_question(si, bi, sheet, block, recipe)
|
||||
questions.append(q)
|
||||
cards.append(Card(f"list_period:{block.id}", "这份明细每期怎么更新",
|
||||
"每期替换:当前版本只含最新一期。按期累积:每期一份明细加进当前版本、可以跨期查询,要说明每一行"
|
||||
"属于哪一期——选一个日期列,核对每行都在本期统计期内;明细里没有能代表统计期的日期列时,"
|
||||
"选「每行加上本期统计期的起止两列」", [], q.id))
|
||||
return cards, questions
|
||||
|
||||
|
||||
def _list_period_question(si: int, bi: int, sheet: Any, block: ListBlock, recipe: dict[str, Any]) -> Question:
|
||||
"""期 5:列表每期怎么更新、按期累积时每一行属于哪一期(ListBlock.period),一道题定:每期替换(不绑定);按期
|
||||
累积并按某个日期列核对(每个日期列一个选项);按期累积并给每行加统计期起止两列。后两种同时把导入模式改成按期
|
||||
累积,所在工作表没配统计期的一并加上统计期(回执里统计期取自表内,没有就请人录入)。
|
||||
|
||||
为什么不分成「先绑定、再问 q_mode」两道题:问题集合按起草的草稿生成、回答时不变(recipe_imports.answer),
|
||||
列表的草稿还没绑定统计期,不符合按期累积的资格,q_mode 根本不会出现。几个列表各答各的:有的选累积、有的选替换
|
||||
时,静态校验照实报出没绑定的那个列表不能累积。"""
|
||||
base = ["sheets", si, "blocks", bi, "period"]
|
||||
context = [] if sheet.context else _add_ops(recipe, ["sheets", si, "context"], [
|
||||
{"id": "统计期", "kind": "period", "parser": "cn_date_range", "prefer_prefix": None, "cross_check": "filename"}])
|
||||
mode = _add_ops(recipe, ["mode"], "accumulate")
|
||||
# 「每期替换」排第一个:它就是草稿本来的样子(期 4 起各处取第一个选项的地方都不会因此改了配方)
|
||||
options: list[dict[str, Any]] = [{"value": "none", "label": "每期替换", "needs_reason": False}]
|
||||
effects: dict[str, list[dict[str, Any]]] = {"none": _add_ops(recipe, base, None)}
|
||||
for c in block.columns:
|
||||
if c.type != "DATE":
|
||||
continue
|
||||
value = f"column:{c.name}"
|
||||
options.append({"value": value, "label": f"按期累积,按「{c.name}」核对每行都在本期统计期内",
|
||||
"needs_reason": False})
|
||||
effects[value] = context + mode + _add_ops(recipe, base, {"column": c.name, "check": "within_context",
|
||||
"synthesize": None})
|
||||
options.append({"value": "synthesize", "label": "按期累积,每行加上本期统计期的起止两列", "needs_reason": False})
|
||||
effects["synthesize"] = context + mode + _add_ops(recipe, base, {"column": None, "check": "within_context",
|
||||
"synthesize": "统计期"})
|
||||
current = block.period
|
||||
default = ("synthesize" if current is not None and current.synthesize else
|
||||
f"column:{current.column}" if current is not None and current.column else "none")
|
||||
return Question(f"q_list_period:{block.id}", f"列表「{block.id}」每期怎么更新", options, default, effects)
|
||||
|
||||
|
||||
# ==========================================================================
|
||||
# 期 3:按规则重新起草,并把名字对齐到现行配方(P3-SPEC 6.3)
|
||||
# ==========================================================================
|
||||
|
||||
@@ -300,11 +300,32 @@ class TotalRow(_M):
|
||||
keep_as: str | None = None
|
||||
|
||||
|
||||
class Subtotals(_M):
|
||||
"""穿插在明细中间的小计行:不作数据,按它所管的那一组明细逐列求和核对;keep_as 给了就另存一张表。"""
|
||||
|
||||
#: 小计标签所在的列(列名)
|
||||
label_column: str
|
||||
#: 小计行的标签以这个词开头(match_key 比较)。取自实际格子文字的候选词,不能含数字
|
||||
pick: str = Field(min_length=1, max_length=12)
|
||||
#: 一行小计管哪几行明细:since_previous 上一行小计(或表头)之后的全部明细;group_column 紧挨在它上面、
|
||||
#: group_column 这一列取值相同的那几行明细
|
||||
scope: Literal["since_previous", "group_column"] = "since_previous"
|
||||
#: scope=group_column 时按哪一列分组(列名);since_previous 时为 None
|
||||
group_column: str | None = None
|
||||
#: 原表的小计另存成哪张表(kind=reported_total);None 只核对不另存
|
||||
keep_as: str | None = None
|
||||
|
||||
|
||||
class ListRows(_M):
|
||||
#: 数据中间的空行:stop 到此为止(默认);skip 跳过继续(回执记个数)
|
||||
blank_rows: Literal["stop", "skip"] = "stop"
|
||||
#: 合计行:到合计行为止,合计行按列求和核对
|
||||
total_row: TotalRow | None = None
|
||||
#: 期 5:穿插在明细中间的小计行(Subtotals)。默认 None = 期 4 的行为(像小计的行照常是数据或起草失败)
|
||||
subtotals: Subtotals | None = None
|
||||
#: 期 5:人确认过这张列表没有小计行:以合计字样开头的行就是数据(「累计充值」这类键值清单),执行器不再报
|
||||
#: subtotal_suspect,每期也就不用再勾那一项确认。和 subtotals 二选一(静态校验 subtotals_conflict)
|
||||
no_subtotals: bool = False
|
||||
|
||||
|
||||
class ListValues(_M):
|
||||
@@ -317,6 +338,18 @@ class ListValues(_M):
|
||||
formula: Literal["accept_cached", "reject"] = "accept_cached"
|
||||
|
||||
|
||||
class ListPeriod(_M):
|
||||
"""列表按期累积时,每一行属于哪一期,二选一:column 按表里的一个日期列,每行的日期都要落在本期的统计期内;
|
||||
synthesize 每行加上本期统计期的起、止两列(明细里没有日期列,或者日期不代表它属于哪一期时)。"""
|
||||
|
||||
#: 按这个日期列(ListColumn.name,类型 DATE)核对;和 synthesize 二选一
|
||||
column: str | None = None
|
||||
#: 怎么核对:within_context 每行的日期都要落在本期统计期内(空着的也不行)
|
||||
check: Literal["within_context"] = "within_context"
|
||||
#: 加统计期起、止两列(LIST_PERIOD_START、LIST_PERIOD_END);和 column 二选一
|
||||
synthesize: Literal["统计期"] | None = None
|
||||
|
||||
|
||||
class ListBlock(_M):
|
||||
id: str = Field(min_length=1, max_length=32)
|
||||
layout: Literal["list"]
|
||||
@@ -333,6 +366,8 @@ class ListBlock(_M):
|
||||
merged_data: Literal["reject", "fill"] = "reject"
|
||||
#: 期 3:按表头忽略的列(IgnoreColumn),只针对这些列;其余多出的列照 extra_columns。默认空 = 期 2 的行为
|
||||
ignore_columns: list[IgnoreColumn] = Field(default_factory=list, max_length=50)
|
||||
#: 期 5:按期累积时每一行属于哪一期(ListPeriod)。默认 None = 期 4 的行为(列表只能每期替换)
|
||||
period: ListPeriod | None = None
|
||||
|
||||
|
||||
Block = Annotated[Union[CrosstabBlock, ListBlock], Field(discriminator="layout")]
|
||||
@@ -441,9 +476,10 @@ class Problem:
|
||||
fix_args: dict[str, Any] | None = None
|
||||
|
||||
|
||||
#: 修复按钮的九种(P3-SPEC 3.2 ①–⑨)。Problem.fix、FixProposal.kind、确认项 fix:<目标键> 的 kind 都取自这里
|
||||
#: 修复按钮的种类(P3-SPEC 3.2 ①–⑨;期 5 加 declare_subtotals:列表中间像小计的行改作小计核对)。Problem.fix、
|
||||
#: FixProposal.kind、确认项 fix:<目标键> 的 kind 都取自这里
|
||||
FIX_KINDS = ("remove_label", "add_label", "edit_members", "rename_title", "declare_total", "ignore_cells",
|
||||
"declare_placeholder", "declare_hidden", "rename_sheet")
|
||||
"declare_placeholder", "declare_hidden", "rename_sheet", "declare_subtotals")
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -462,13 +498,14 @@ CheckStatus = Literal["passed", "mismatch", "unverifiable", "info"]
|
||||
|
||||
@dataclass
|
||||
class CheckResult:
|
||||
#: 稳定 id:K1… 合计核对、G1… 公式引用、T1… 列表合计、R1… 关系(沿用配方里的 id)、C1 统计期、C2 文件名、
|
||||
#: N1… 行数、P1… 主键;期 3 的并集整体核对 U1 行数、U2 主键、U3 统计期(物化时跑,都是结构类,P3-SPEC 2.7)。
|
||||
#: 稳定 id:K1… 合计核对、G1… 公式引用、T1… 列表合计(期 5 起也含列表中间的小计)、R1… 关系(沿用配方里的
|
||||
#: id)、C1 统计期、C2 文件名、N1… 行数、P1… 主键、D1… 列表的日期落在统计期内(期 5,ListPeriod.column);
|
||||
#: 期 3 的并集整体核对 U1 行数、U2 主键、U3 统计期(物化时跑,都是结构类,P3-SPEC 2.7)。
|
||||
#: F 编号留给系统发现的事实(Fact.id),两者不混用
|
||||
id: str
|
||||
kind: Literal["derived_sum", "formula_refs", "column_sum", "relation_sum_eq", "relation_not_comparable",
|
||||
"context_agree", "filename_period", "row_count", "pk_unique",
|
||||
"union_rows", "union_pk", "union_period"]
|
||||
"union_rows", "union_pk", "union_period", "period_rows"]
|
||||
title: str
|
||||
status: CheckStatus
|
||||
#: 不通过时的类别:structure / data_quality / info
|
||||
@@ -759,9 +796,10 @@ class SheetsOut:
|
||||
#: hidden_excluded = hidden.rows=exclude 排除的隐藏行;blank_skipped = 列表 blank_rows=skip 跳过的空行;
|
||||
#: ignored_rows = 交叉表 ignore_rows 忽略的行;ignored_outside = ignore_outside 忽略了区域外数字格的行;
|
||||
#: after_stop = 列表停止之后没导入的文字行;total_not_kept = 参与核对、但不另存的合计行(交叉表 keep_as 为空的
|
||||
#: derived 段、列表 total_row.keep_as 为空的合计行)
|
||||
#: derived 段、列表 total_row.keep_as 为空的合计行);期 5:subtotal = 列表中间的小计行(参与核对、不进数据,
|
||||
#: 另存与否都记在这里:它们都不在本表里)
|
||||
ExcludedReason = Literal["hidden_excluded", "blank_skipped", "ignored_rows", "ignored_outside", "after_stop",
|
||||
"total_not_kept"]
|
||||
"total_not_kept", "subtotal"]
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -1031,6 +1069,15 @@ def prose_problems(template: str, *, known: dict[str, set[str]] | None = None) -
|
||||
|
||||
#: 列表合计行另存成表时,合计项列的固定列名
|
||||
LIST_TOTAL_DIM = "合计项"
|
||||
#: 期 5:列表的小计行另存成表时,小计项列的固定列名
|
||||
LIST_SUBTOTAL_DIM = "小计项"
|
||||
#: 期 5:列表按期累积、由系统给每行加统计期时(ListPeriod.synthesize)的两列:本期统计期的第一天、最后一天,
|
||||
#: 写法 YYYY-MM-DD(同 DATE 列,存 TEXT)
|
||||
LIST_PERIOD_START = "统计期起"
|
||||
LIST_PERIOD_END = "统计期止"
|
||||
#: 期 5:小计核对的键(核对编号 T、合计格 DerivedItem.segment)= 块 id 加这个后缀。块 id 在整份配方内唯一,
|
||||
#: 同一块的合计行用块 id 本身,两者不相混
|
||||
SUBTOTAL_KEY = "·小计"
|
||||
_SQL_TYPE = {"INTEGER": "INTEGER", "REAL": "REAL", "TEXT": "TEXT", "DATE": "TEXT"}
|
||||
|
||||
|
||||
@@ -1107,16 +1154,42 @@ def derive_tables(recipe: Recipe) -> tuple[dict[str, list[ColumnOut]], list[Reci
|
||||
cols = [ColumnOut(c.name, _SQL_TYPE[c.type], header=c.header,
|
||||
role="measure" if c.type in ("INTEGER", "REAL") else "text")
|
||||
for c in block.columns]
|
||||
if block.period is not None and block.period.synthesize:
|
||||
# 期 5:每行加上本期统计期的起止(role=period:没有格子来历,取值来自统计期)
|
||||
cols += [ColumnOut(LIST_PERIOD_START, "TEXT", role="period"),
|
||||
ColumnOut(LIST_PERIOD_END, "TEXT", role="period")]
|
||||
put(block.table, cols, base)
|
||||
total = block.rows.total_row
|
||||
numeric = [c for c in cols if c.type in ("INTEGER", "REAL")]
|
||||
if total is not None and total.keep_as:
|
||||
tcols = [ColumnOut(LIST_TOTAL_DIM, "TEXT", role="dim")]
|
||||
tcols += [ColumnOut(c.name, c.type, header=c.header, role="value")
|
||||
for c in cols if c.type in ("INTEGER", "REAL")]
|
||||
tcols += [ColumnOut(c.name, c.type, header=c.header, role="value") for c in numeric]
|
||||
put(total.keep_as, tcols, f"{base}/rows/total_row")
|
||||
sub = block.rows.subtotals
|
||||
if sub is not None and sub.keep_as:
|
||||
# 期 5:小计另存:小计项(标签的规范写法)、按组小计时的分组列、各数字列
|
||||
scols = [ColumnOut(LIST_SUBTOTAL_DIM, "TEXT", role="dim")]
|
||||
if sub.scope == "group_column" and sub.group_column:
|
||||
scols.append(ColumnOut(sub.group_column, "TEXT", role="dim"))
|
||||
scols += [ColumnOut(c.name, c.type, header=c.header, role="value") for c in numeric]
|
||||
put(sub.keep_as, scols, f"{base}/rows/subtotals")
|
||||
return out, problems
|
||||
|
||||
|
||||
def list_period_columns(recipe: Recipe) -> dict[str, tuple[str, ...]]:
|
||||
"""期 5:列表写入的表 → 它按什么分期:("column", 日期列) 或 ("synthesize", 统计期起列, 统计期止列)。只列配了
|
||||
ListBlock.period 的列表;累积计划的 U3、单期的 D 核对、说明里「按哪一列筛选」共用这一份。"""
|
||||
out: dict[str, tuple[str, ...]] = {}
|
||||
for sheet in recipe.sheets:
|
||||
for block in sheet.blocks:
|
||||
if isinstance(block, ListBlock) and block.period is not None:
|
||||
if block.period.synthesize:
|
||||
out.setdefault(block.table, ("synthesize", LIST_PERIOD_START, LIST_PERIOD_END))
|
||||
elif block.period.column:
|
||||
out.setdefault(block.table, ("column", block.period.column))
|
||||
return out
|
||||
|
||||
|
||||
# ==========================================================================
|
||||
# 期 3:按期累积的资格(静态校验、起草器、累积计划共用同一份,P3-SPEC 2.2)
|
||||
# ==========================================================================
|
||||
@@ -1130,10 +1203,13 @@ def accumulate_blockers(recipe: Recipe) -> list[RecipeProblem]:
|
||||
1. 至少一个工作表配置了统计期上下文(SheetRecipe.context 非空),否则报 accumulate_needs_period(path=/mode);
|
||||
2. derive_tables 推出的每张表都「按统计期分得开」,否则逐表报 accumulate_unkeyed(path=/tables/{ti},表没在
|
||||
tables 里声明时指向第一个写入它的块),消息写明哪一条不满足:
|
||||
- 只由交叉表块写入(measures、dimension 段,或者 derived 段的 keep_as)——列表写入的表一律不满足;
|
||||
- 写入它的每个交叉表块,axis.checks 都含 covers_context(日期恰好落在统计期内);
|
||||
- 这张表的 grain 含那个块的 axis.name。
|
||||
两条都成立时,各期的日期集合互不相交(统计期不重叠由累积计划保证),主键里不需要另加统计期列。
|
||||
- 交叉表写入的(measures、dimension 段,或者 derived 段的 keep_as):写入它的每个交叉表块,axis.checks 都含
|
||||
covers_context(日期恰好落在统计期内),并且这张表的 grain 含那个块的 axis.name;
|
||||
- 期 5:列表写入的明细表:写入它的每个列表块都配了 period(按日期列核对,或加统计期起止两列),并且所在的
|
||||
工作表配了统计期。列表的合计行、小计行另存的表不能累积:各期的合计项、小计项相同,主键分不开各期;
|
||||
- 同一张表不能既由交叉表又由列表写入。
|
||||
都成立时,各期的行按统计期分得开(统计期不重叠由累积计划保证;列表明细跨期重复的同一条记录由并集的主键核对
|
||||
U2 拒收)。
|
||||
"""
|
||||
problems: list[RecipeProblem] = []
|
||||
if not any(sheet.context for sheet in recipe.sheets):
|
||||
@@ -1144,6 +1220,11 @@ def accumulate_blockers(recipe: Recipe) -> list[RecipeProblem]:
|
||||
writers: dict[str, list[CrosstabBlock | ListBlock]] = {}
|
||||
first_path: dict[str, str] = {}
|
||||
|
||||
#: 列表的合计行、小计行另存的表 → 写入它的块
|
||||
totals: dict[str, list[ListBlock]] = {}
|
||||
#: 列表块 → 所在工作表有没有配统计期
|
||||
ctx_of: dict[int, bool] = {}
|
||||
|
||||
def _add(table: str, block: CrosstabBlock | ListBlock, path: str) -> None:
|
||||
got = writers.setdefault(table, [])
|
||||
if all(b is not block for b in got):
|
||||
@@ -1161,16 +1242,32 @@ def accumulate_blockers(recipe: Recipe) -> list[RecipeProblem]:
|
||||
_add(seg.keep_as.table, block, path)
|
||||
else:
|
||||
_add(block.table, block, path)
|
||||
total = block.rows.total_row
|
||||
if total is not None and total.keep_as:
|
||||
_add(total.keep_as, block, path)
|
||||
ctx_of[id(block)] = bool(sheet.context)
|
||||
for kept in (block.rows.total_row.keep_as if block.rows.total_row else None,
|
||||
block.rows.subtotals.keep_as if block.rows.subtotals else None):
|
||||
if kept:
|
||||
_add(kept, block, path)
|
||||
totals.setdefault(kept, []).append(block)
|
||||
|
||||
index = {t.name: ti for ti, t in enumerate(recipe.tables)}
|
||||
grains = {t.name: list(t.grain) for t in recipe.tables}
|
||||
for name in tables:
|
||||
blocks = writers.get(name, [])
|
||||
if any(isinstance(b, ListBlock) for b in blocks):
|
||||
reasons = ["列表形态的表目前只支持每期替换"]
|
||||
lists = [b for b in blocks if isinstance(b, ListBlock)]
|
||||
if lists and len(lists) != len(blocks):
|
||||
reasons = ["同一张表既由交叉表又由列表写入,各期的行分不开"]
|
||||
elif name in totals:
|
||||
reasons = ["列表的合计行、小计行另存的表不能按期累积:各期的合计项、小计项相同,主键分不开各期"
|
||||
"(可以改为只核对、不另存)"]
|
||||
elif lists:
|
||||
reasons = []
|
||||
unbound = [b.id for b in lists if b.period is None]
|
||||
if unbound:
|
||||
reasons.append("、".join(f"列表「{x}」" for x in unbound)
|
||||
+ "没有指定每一行属于哪一期(按日期列核对,或加统计期起止两列)")
|
||||
no_ctx = [b.id for b in lists if b.period is not None and not ctx_of.get(id(b))]
|
||||
if no_ctx:
|
||||
reasons.append("、".join(f"列表「{x}」" for x in no_ctx) + "所在的工作表没有配置统计期")
|
||||
else:
|
||||
reasons = []
|
||||
uncovered = [b.id for b in blocks if isinstance(b, CrosstabBlock) and "covers_context" not in b.axis.checks]
|
||||
|
||||
@@ -110,6 +110,16 @@ def load_manifest(artifact_id: str | None) -> dict[str, Any] | None:
|
||||
return doc if isinstance(doc, dict) else None
|
||||
|
||||
|
||||
def _signer_verified(artifact_id: str | None) -> bool:
|
||||
"""导入清单里记下的「署名是否认证」;没有清单(简单导入)、读不出来的照实算未认证。"""
|
||||
try:
|
||||
doc = load_manifest(artifact_id)
|
||||
except ValueError:
|
||||
return False
|
||||
signer = (doc or {}).get("signed_by")
|
||||
return bool(signer.get("verified")) if isinstance(signer, dict) else False
|
||||
|
||||
|
||||
def _rows_of(report: dict[str, Any] | None) -> dict[str, int]:
|
||||
"""构建回执里各表的行数(执行器回执、期 1 的解析回执都是 tables: [{name, rows}])。"""
|
||||
out: dict[str, int] = {}
|
||||
@@ -394,7 +404,7 @@ def columns_doc(report: dict[str, Any], periods: list[str]) -> dict[str, dict[st
|
||||
def manifest_doc(*, source_id: str, snapshot_id: str, action: str, recipe: dict[str, Any],
|
||||
union: dict[str, Any], parts: list[Part | dict[str, Any]], report: dict[str, Any],
|
||||
label_sets: list[Any], gaps: list[dict[str, str]], replaced: Any, removed: Any,
|
||||
notes: SchemaNotes, signed_by: str | None) -> dict[str, Any]:
|
||||
notes: SchemaNotes, signed_by: str | None, signer_verified: bool = False) -> dict[str, Any]:
|
||||
"""快照清单(2.9)。parts 按统计期排序,与 report.part_rows 同序;本期写成 dict(导入记录还没落库)。"""
|
||||
part_rows = report.get("part_rows") or []
|
||||
keys = []
|
||||
@@ -422,7 +432,8 @@ def manifest_doc(*, source_id: str, snapshot_id: str, action: str, recipe: dict[
|
||||
"checks": copy.deepcopy(report.get("checks") or []),
|
||||
"notes": {"templates": dict(notes.templates),
|
||||
"rendered": {t: {"comment": n.comment, "columns": dict(n.columns)} for t, n in notes.tables.items()}},
|
||||
"signed_by": {"name": signed_by, "verified": False},
|
||||
# 是否认证照实记(05-已知遗留 (c),同导入清单)
|
||||
"signed_by": {"name": signed_by, "verified": bool(signer_verified and signed_by)},
|
||||
"created_at": utcnow().isoformat(),
|
||||
}
|
||||
|
||||
@@ -684,8 +695,10 @@ async def _current(session: AsyncSession, source: DataSource) -> _Current:
|
||||
mode=snap.mode or "replace", recipe=recipe)
|
||||
|
||||
|
||||
def _revoked_record(reason: str, signed_by: str | None) -> dict[str, Any]:
|
||||
return {"at": utcnow().isoformat(), "reason": reason, "signed_by": signed_by or None, "signed_by_verified": False}
|
||||
def _revoked_record(reason: str, signed_by: str | None, signer_verified: bool = False) -> dict[str, Any]:
|
||||
"""作废接受的记录。是否认证照实记(05-已知遗留 (c)):开了登录、署名是登录的人时为真。"""
|
||||
return {"at": utcnow().isoformat(), "reason": reason, "signed_by": signed_by or None,
|
||||
"signed_by_verified": bool(signer_verified and signed_by)}
|
||||
|
||||
|
||||
async def rollback_target(session: AsyncSession, source: DataSource, import_id: str) -> SourceSnapshot | None:
|
||||
@@ -739,7 +752,8 @@ async def revoke_plan(session: AsyncSession, source: DataSource, imp: TableImpor
|
||||
|
||||
|
||||
async def _rebuild_without(session: AsyncSession, cur: _Current, imp: TableImport, *, expected_current: str | None,
|
||||
kind: str, reason: str, signed_by: str | None, revoke: bool) -> tuple[str, bool]:
|
||||
kind: str, reason: str, signed_by: str | None, revoke: bool,
|
||||
signer_verified: bool = False) -> tuple[str, bool]:
|
||||
"""从当前版本去掉一期、生成(或复用、复活)新快照(7.4)。配方和导入模式不变。返回 (快照 id, 是否复用)。
|
||||
|
||||
revoke=True 时同一个事务里写 table_imports.revoked(7.5):复用路径在调 activate_snapshot 之前改好(随它的提交
|
||||
@@ -758,7 +772,7 @@ async def _rebuild_without(session: AsyncSession, cur: _Current, imp: TableImpor
|
||||
sid = table_versions.snapshot_id(source_id, keep, recipe_types.RECIPE_ENGINE_VER, mode="accumulate",
|
||||
recipe_sha=target_sha)
|
||||
imp_id = imp.id
|
||||
revoked = _revoked_record(reason, signed_by) if revoke else None
|
||||
revoked = _revoked_record(reason, signed_by, signer_verified) if revoke else None
|
||||
activation = {"kind": kind, "import_id": imp_id, "reason": reason, "signed_by": signed_by}
|
||||
|
||||
existing = await session.get(SourceSnapshot, sid)
|
||||
@@ -871,7 +885,7 @@ async def _rebuild_without(session: AsyncSession, cur: _Current, imp: TableImpor
|
||||
"table_hashes": dict((info.union_report or report_json or {}).get("table_hashes") or {}),
|
||||
"union_ver": recipe_types.UNION_VER},
|
||||
parts=parts, report=info.union_report or report_json or {}, label_sets=label_sets, gaps=gaps,
|
||||
replaced=None, removed=removed, notes=notes, signed_by=signed_by)
|
||||
replaced=None, removed=removed, notes=notes, signed_by=signed_by, signer_verified=signer_verified)
|
||||
cache["snapshot_manifest"] = await artifact_store.put_json(
|
||||
doc, kind="snapshot_manifest", meta={"source_id": source_id, "snapshot_id": info.snapshot_id})
|
||||
return cache
|
||||
@@ -931,7 +945,7 @@ def _check_base(cur: _Current, expected_current: str | None) -> None:
|
||||
|
||||
|
||||
async def remove_period(session: AsyncSession, source: DataSource, import_id: str, *, expected_current: str | None,
|
||||
reason: str, signed_by: str | None) -> dict[str, Any]:
|
||||
reason: str, signed_by: str | None, signer_verified: bool = False) -> dict[str, Any]:
|
||||
"""POST …/imports/{id}/remove(7.4)。"""
|
||||
imp = await session.get(TableImport, import_id)
|
||||
if imp is None or imp.source_id != source.id:
|
||||
@@ -947,13 +961,15 @@ async def remove_period(session: AsyncSession, source: DataSource, import_id: st
|
||||
if len(cur.imports) < 2:
|
||||
raise ImportRefused(409, "last_period", _LAST_PERIOD)
|
||||
sid, reused = await _rebuild_without(session, cur, imp, expected_current=expected_current, kind="remove_period",
|
||||
reason=reason, signed_by=signed_by, revoke=False)
|
||||
reason=reason, signed_by=signed_by, revoke=False,
|
||||
signer_verified=signer_verified)
|
||||
return {"snapshot_id": sid, "removed_import_id": import_id, "reused": reused}
|
||||
|
||||
|
||||
async def revoke_acceptance(session: AsyncSession, source: DataSource, import_id: str, *,
|
||||
expected_current: str | None, expected_target: str | None,
|
||||
ack_mask_lost: list[str] | None, reason: str, signed_by: str | None) -> dict[str, Any]:
|
||||
ack_mask_lost: list[str] | None, reason: str, signed_by: str | None,
|
||||
signer_verified: bool = False) -> dict[str, Any]:
|
||||
"""POST …/imports/{id}/revoke-acceptance(7.5)。累积模式(至少两期)= 移除这一期;否则回滚到第 5 节定义的目标。"""
|
||||
imp = await session.get(TableImport, import_id)
|
||||
if imp is None or imp.source_id != source.id:
|
||||
@@ -968,7 +984,7 @@ async def revoke_acceptance(session: AsyncSession, source: DataSource, import_id
|
||||
if _accumulating(cur):
|
||||
sid, _reused = await _rebuild_without(session, cur, imp, expected_current=expected_current,
|
||||
kind="revoke_acceptance", reason=reason, signed_by=signed_by,
|
||||
revoke=True)
|
||||
revoke=True, signer_verified=signer_verified)
|
||||
return {"snapshot_id": sid, "action": "remove_period"}
|
||||
target = await rollback_target(session, source, imp.id)
|
||||
if target is None:
|
||||
@@ -976,7 +992,7 @@ async def revoke_acceptance(session: AsyncSession, source: DataSource, import_id
|
||||
if target.id != expected_target:
|
||||
raise ImportRefused(409, "revoke_target_changed", _TARGET_CHANGED)
|
||||
target_id = target.id
|
||||
imp.revoked = _revoked_record(reason, signed_by)
|
||||
imp.revoked = _revoked_record(reason, signed_by, signer_verified)
|
||||
try:
|
||||
await table_versions.activate_snapshot(
|
||||
session, source, target_id, expected_current=expected_current, kind="revoke_acceptance", reason=reason,
|
||||
@@ -1049,6 +1065,8 @@ async def import_records(session: AsyncSession, source: DataSource, imps: list[T
|
||||
# 按配方导入才有(简单导入为空):配方、统计期、署名,接受的条数
|
||||
"recipe_id": imp.recipe_id, "period_start": imp.period_start, "period_end": imp.period_end,
|
||||
"signed_by": imp.signed_by, "overrides": len(imp.overrides or []), "waivers": len(imp.waivers or []),
|
||||
# 署名是不是认证过的身份:按这一期导入清单里记下的(05-已知遗留 (c);旧清单写的是 False,照记下的说,不改写)
|
||||
"signed_by_verified": _signer_verified(imp.manifest_artifact),
|
||||
# 期 3(7.6)
|
||||
"acceptances": _acceptance_rows(imp), "revoked": imp.revoked, "manifest_artifact": imp.manifest_artifact,
|
||||
"recipe_seq": seqs.get(imp.recipe_id) if imp.recipe_id else None, "in_current": in_current,
|
||||
|
||||
@@ -326,7 +326,7 @@ class _Table:
|
||||
「唯一日期列」和这里认的时间列对得上)。"""
|
||||
meta = next((c for c in self.meta.get("columns") or []
|
||||
if isinstance(c, Mapping) and str(c.get("name")).lower() == column.lower()), None)
|
||||
return catalog.is_date_column(column, (meta or {}).get("type"))
|
||||
return catalog.is_date_column(column, (meta or {}).get("type"), enum_values=(meta or {}).get("enum_values"))
|
||||
|
||||
|
||||
#: 一列解析到的位置:(表, 列的真名)
|
||||
@@ -1084,36 +1084,36 @@ def graph_checks(nodes: Iterable[Any], checkers: Mapping[str, SqlChecker], *,
|
||||
return out
|
||||
|
||||
|
||||
async def load_checkers(session: Any, sources: Iterable[Any]) -> dict[str, SqlChecker]:
|
||||
async def load_checkers(session: Any, sources: Iterable[Any], *, catalog_view: Any = None) -> dict[str, SqlChecker]:
|
||||
"""给一批数据源(DataSource 行)各建一个检查器:{源名: SqlChecker}。没有目录的源不建。
|
||||
|
||||
一次查询读出这些源的全部目录。读不到就当没有目录(返回空 dict):检查是附加的,不能挡住自查和发布本身。
|
||||
一次查询读出这些源的全部目录,读哪一份由 catalog_view 定(data/catalog_view.py:当前目录、目录快照、关;只用已核实
|
||||
的项),不给是当前目录。读不到当前目录就当没有目录(返回空 dict):检查是附加的,不能挡住自查和发布本身;视图指定的
|
||||
目录快照不在了照实报错。
|
||||
"""
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.db.models import CatalogNote
|
||||
from app.data.catalog_view import CatalogSnapshotMissing, CatalogView, read_many
|
||||
|
||||
rows = [s for s in sources if getattr(s, "id", None) and dialect_of(getattr(s, "kind", None))]
|
||||
if not rows:
|
||||
return {}
|
||||
try:
|
||||
notes = (await session.execute(
|
||||
select(CatalogNote.source_id, CatalogNote.table_name, CatalogNote.notes)
|
||||
.where(CatalogNote.source_id.in_([r.id for r in rows]))
|
||||
)).all()
|
||||
entries = await read_many(session, CatalogView.parse(catalog_view), [r.id for r in rows])
|
||||
except CatalogSnapshotMissing:
|
||||
raise
|
||||
except Exception: # noqa: BLE001
|
||||
logger.exception("读取数据目录失败,本次不做基于目录的 SQL 检查")
|
||||
return {}
|
||||
by_source: dict[str, dict[str, Any]] = {}
|
||||
for source_id, table, value in notes:
|
||||
if isinstance(value, dict) and value:
|
||||
by_source.setdefault(source_id, {})[table] = value
|
||||
for source_id, tables in entries.items():
|
||||
for table, entry in tables.items():
|
||||
if isinstance(entry.notes, dict) and entry.notes:
|
||||
by_source.setdefault(source_id, {})[table] = entry.notes
|
||||
return {r.name: SqlChecker(kind=r.kind, schema_cache=r.schema_cache, notes=by_source[r.id], source_name=r.name)
|
||||
for r in rows if r.id in by_source}
|
||||
|
||||
|
||||
async def load_checkers_by_name(session: Any, names: Iterable[str]) -> dict[str, SqlChecker]:
|
||||
"""按数据源名建检查器(只看启用着的源)。发布门禁用:图里写的是源名。"""
|
||||
"""按数据源名建检查器(只看启用着的源)。发布门禁用:图里写的是源名,读的是当前目录。"""
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.db.models import DataSource
|
||||
|
||||
@@ -2130,7 +2130,7 @@ class RawPurge:
|
||||
|
||||
|
||||
async def purge_import_raw(
|
||||
session: AsyncSession, imp: TableImport, *, reason: str, signed_by: str | None
|
||||
session: AsyncSession, imp: TableImport, *, reason: str, signed_by: str | None, signer_verified: bool = False
|
||||
) -> RawPurge:
|
||||
"""清除一次导入的原件(提交)。按两元组解包得到 (是否删了文件, 一并标成已清除的其他导入记录),
|
||||
discarded_stagings 是一并放弃的暂存区。
|
||||
@@ -2160,7 +2160,9 @@ async def purge_import_raw(
|
||||
now = utcnow()
|
||||
record = {
|
||||
"at": now.isoformat(), "reason": reason,
|
||||
"signed_by": (signed_by or "").strip() or None, "signed_by_verified": False,
|
||||
# 是否认证照实记(05-已知遗留 (c)):开了登录、署名是登录的人时为真
|
||||
"signed_by": (signed_by or "").strip() or None,
|
||||
"signed_by_verified": bool(signer_verified and (signed_by or "").strip()),
|
||||
}
|
||||
imp.raw_state, imp.purged = "purged", record
|
||||
for other in others:
|
||||
|
||||
@@ -24,7 +24,7 @@ from collections.abc import Iterator
|
||||
from typing import Any
|
||||
from xml.etree import ElementTree as ET
|
||||
|
||||
from app.data import tabular
|
||||
from app.data import csv_grid, tabular
|
||||
from app.data.recipe_types import Grid, GridCell
|
||||
from app.data.xlsx_scan import SheetScan, UnsupportedTable, WorkbookScan, _broken, cell_ref, parse_ref
|
||||
|
||||
@@ -70,6 +70,14 @@ class Workbooks:
|
||||
self.raw = raw
|
||||
self._values: Any = None
|
||||
self._formulas: Any = None
|
||||
self._csv: Any = None
|
||||
|
||||
def csv(self, delimiter: str | None = None) -> Any:
|
||||
"""CSV 原件解析好的记录(csv_grid.CsvTable),第一次要读时才解析,同一次执行里复用。delimiter 是扫描时定下的
|
||||
分隔符(csv_grid.sheet_delimiter),不给时再嗅一次。"""
|
||||
if self._csv is None:
|
||||
self._csv = csv_grid.parse(self.raw, delimiter)
|
||||
return self._csv
|
||||
|
||||
def values(self) -> Any:
|
||||
if self._values is None:
|
||||
@@ -126,6 +134,11 @@ def sheet_rows(books: Workbooks, sheet: SheetScan, *, min_row: int, max_row: int
|
||||
行数和每行的宽度都一样。文件内容写坏了(见 _READ_ERRORS)抛 UnsupportedTable,日志只记异常类型。
|
||||
"""
|
||||
try:
|
||||
if csv_grid.is_csv_sheet(sheet):
|
||||
# CSV 适配出来的「工作表」:从解析好的记录里取,口径同下(只给非空格、整行为空的不给)。分隔符用扫描时定下的
|
||||
yield from csv_grid.rows(books.csv(csv_grid.sheet_delimiter(sheet)), min_row=min_row, max_row=max_row,
|
||||
min_col=min_col, max_col=max_col)
|
||||
return
|
||||
yield from _sheet_rows(books, sheet, min_row=min_row, max_row=max_row, min_col=min_col, max_col=max_col)
|
||||
except UnsupportedTable:
|
||||
raise
|
||||
|
||||
@@ -216,6 +216,9 @@ class WorkbookScan:
|
||||
tables: list[TableScan] = field(default_factory=list)
|
||||
#: 定义名称 → 原文。工作表级的名称记作「工作表名!名称」,免得同名互相覆盖
|
||||
defined_names: dict[str, str] = field(default_factory=dict)
|
||||
#: 原件的格式:xlsx,或者 csv(csv_grid.scan 适配出来的一张「工作表」,读取层据 SheetScan.path 从 CSV 里读)。
|
||||
#: 老暂存区的扫描结果里没有这个键,按 xlsx 读
|
||||
format: str = "xlsx"
|
||||
|
||||
def sheet(self, name: str) -> SheetScan | None:
|
||||
return next((s for s in self.sheets if s.name == name), None)
|
||||
@@ -1090,4 +1093,5 @@ def scan_from_json(data: dict[str, Any]) -> WorkbookScan:
|
||||
for t in tables:
|
||||
t.columns = list(t.columns)
|
||||
return WorkbookScan(sheets=sheets, full_calc_on_load=bool(data.get("full_calc_on_load")), tables=tables,
|
||||
defined_names=dict(data.get("defined_names") or {}))
|
||||
defined_names=dict(data.get("defined_names") or {}),
|
||||
format=str(data.get("format") or "xlsx"))
|
||||
|
||||
@@ -239,6 +239,12 @@ _COLUMN_MIGRATIONS: list[tuple[str, str, str]] = [
|
||||
("apps", "scope", "VARCHAR(20) DEFAULT 'shared'"),
|
||||
("apps", "owner", "VARCHAR(100)"),
|
||||
("apps", "blueprint_id", "VARCHAR(32)"),
|
||||
# 下一轮(docs/design/next-round/00-总方案.md X4、X9):运行为什么结束在 cancelled;定时的目标种类和 id;
|
||||
# 定时触发建的那次评测。存量行:end_reason 为空,target_kind 是 workflow(target_id 由 scheduler 按 workflow_id 读)
|
||||
("runs", "end_reason", "VARCHAR(20)"),
|
||||
("schedules", "target_kind", "VARCHAR(20) DEFAULT 'workflow'"),
|
||||
("schedules", "target_id", "VARCHAR(32)"),
|
||||
("schedule_fires", "eval_run_id", "VARCHAR(32)"),
|
||||
]
|
||||
|
||||
#: 迁移补上的列要建的索引:(索引名, 表, 列或多列)。create_all 不碰已有的表,表上新列的索引也就不会建。
|
||||
|
||||
@@ -654,6 +654,9 @@ class Run(Base, TimestampMixin):
|
||||
input: Mapped[dict[str, Any]] = mapped_column(default=dict)
|
||||
output: Mapped[dict[str, Any]] = mapped_column(default=dict)
|
||||
error: Mapped[str | None] = mapped_column(Text, default=None)
|
||||
# 运行为什么结束在 cancelled(docs/design/next-round/00-总方案.md X9):rejected 人工驳回、stopped 手动停止。
|
||||
# 其余终态、加这一列之前的运行为空
|
||||
end_reason: Mapped[str | None] = mapped_column(String(20), default=None)
|
||||
usage: Mapped[dict[str, Any]] = mapped_column(default=dict) # tokens / cost / 各节点耗时
|
||||
started_at: Mapped[datetime | None] = mapped_column(default=None)
|
||||
finished_at: Mapped[datetime | None] = mapped_column(default=None)
|
||||
@@ -975,6 +978,12 @@ class Schedule(Base, TimestampMixin):
|
||||
|
||||
id: Mapped[str] = mapped_column(String(32), primary_key=True, default=new_id)
|
||||
name: Mapped[str] = mapped_column(String(200), default="")
|
||||
# 定时的目标(next-round X4,定时只有一套):workflow 发起工作流运行;eval_suite 跑一次评测题集;
|
||||
# source_maintenance 维护一个数据源(探查、剖析、取值观察)。加这一列之前的任务都是 workflow
|
||||
target_kind: Mapped[str] = mapped_column(String(20), default="workflow")
|
||||
# 目标的 id:workflow 时同 workflow_id;其余两种是题集 id、数据源 id
|
||||
target_id: Mapped[str | None] = mapped_column(String(32), default=None, index=True)
|
||||
# 非空(老表的约束,只加不改的迁移放宽不了):目标不是工作流时存空串,读写都经 scheduler/service 里的一处封装
|
||||
workflow_id: Mapped[str] = mapped_column(String(32), index=True)
|
||||
# 保存时的工作流名。工作流还在时接口给它现在的名字,删掉之后才用这一份
|
||||
workflow_name: Mapped[str] = mapped_column(String(200), default="")
|
||||
@@ -1020,6 +1029,8 @@ class ScheduleFire(Base):
|
||||
# started | skipped_overlap | skipped_misfire | failed_to_start
|
||||
status: Mapped[str] = mapped_column(String(30), default="started")
|
||||
run_id: Mapped[str | None] = mapped_column(String(32), default=None, index=True)
|
||||
# 目标是评测题集时,这一次触发建的那次评测(target_kind=eval_suite,next-round X4)
|
||||
eval_run_id: Mapped[str | None] = mapped_column(String(32), default=None)
|
||||
reason: Mapped[str | None] = mapped_column(Text, default=None)
|
||||
# 渲染后的输入,就是交给运行的那一份
|
||||
inputs: Mapped[dict[str, Any]] = mapped_column(default=dict)
|
||||
|
||||
@@ -39,6 +39,9 @@ class Decision:
|
||||
#: 审批卡上点的是「始终允许」:批准这次,并且这个工具以后改由门控把关(tools/trust.py)。
|
||||
#: 只认明确的 True,而且只跟着批准走
|
||||
always: bool = False
|
||||
#: 这次驳回是待办超时、按「超时驳回」替人做的(process/tasks.py 的 tick),不是有人点了驳回。只跟着驳回走,
|
||||
#: 节点据此在 NodeRejected 上记 decision=expired_reject
|
||||
expired: bool = False
|
||||
|
||||
|
||||
def _word(value: Any) -> str:
|
||||
@@ -71,6 +74,7 @@ def read_decision(response: Any) -> Decision:
|
||||
code=code if isinstance(code, str) and code else None,
|
||||
value=response.get("value"),
|
||||
always=verdict is True and response.get("always") is True,
|
||||
expired=verdict is False and response.get("expired") is True,
|
||||
)
|
||||
verdict = _verdict(response)
|
||||
# 不是一个是/否词的字符串("拒绝,这个表是生产库")既算不上批准,本身又
|
||||
|
||||
@@ -10,7 +10,7 @@ from langgraph.graph import END, START, StateGraph
|
||||
|
||||
from app.core.errors import describe_exception, raw_detail
|
||||
from app.core.events import EventType
|
||||
from app.engine.context import NodeContext, NodeError, RunContext
|
||||
from app.engine.context import NodeContext, NodeError, NodeRejected, RunContext
|
||||
from app.engine.expressions import open_template_scope, template_scope
|
||||
from app.engine.nodes import (
|
||||
challenge, control, dual_check, human, io, knowledge, llm, merge, metrics, multi, report, tools,
|
||||
@@ -144,6 +144,13 @@ def _wrap(
|
||||
raise
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except NodeRejected as e:
|
||||
# 人的决定(驳回、超时驳回),不是故障(总方案 X9):「失败重试」「出错时继续」都是给故障准备的。重试只会把
|
||||
# 同一件事再问一遍人;on_error=continue 会把驳回吞掉接着往下走,最后以 succeeded 结束,通知、定时推送、
|
||||
# 发布单都看不到这次驳回。所以不重试、不被容错接住;节点这一步照旧记一条 node.failed(时间线、实例进度
|
||||
# 按它收尾这一步),交给 runner 收成 cancelled + end_reason=rejected
|
||||
last_error = e
|
||||
break
|
||||
except Exception as e: # noqa: BLE001
|
||||
# NodeError 以前在这里直接 raise,于是节点自己抛的错既不重试、
|
||||
# 也不发 node.failed、更不被 on_error=continue 接住。而 NodeError
|
||||
@@ -168,8 +175,8 @@ def _wrap(
|
||||
ctx.emit(EventType.NODE_FAILED, error=message, duration_ms=elapsed, node_id=node.id,
|
||||
detail=_detail(last_error))
|
||||
|
||||
# 容错模式:记下错误继续往下走,而不是让整张图挂掉
|
||||
if node.config.get("on_error") == "continue":
|
||||
# 容错模式:记下错误继续往下走,而不是让整张图挂掉。驳回不算错误,不走这里(见上)
|
||||
if node.config.get("on_error") == "continue" and not isinstance(last_error, NodeRejected):
|
||||
return {
|
||||
"nodes": {node.id: {"error": message, "failed": True}},
|
||||
"trail": [{"node_id": node.id, "error": message, "ts": time.time()}],
|
||||
|
||||
@@ -106,3 +106,33 @@ class NodeError(RuntimeError):
|
||||
super().__init__(message)
|
||||
self.node_id = node_id
|
||||
self.retryable = retryable
|
||||
|
||||
|
||||
#: 驳回发生在哪种关卡上(NodeRejected.kind):人工节点的三种模式、写入前的动作确认、代码执行审批、工具调用审批、双盲裁决
|
||||
REJECTION_KINDS = ("approve", "input", "edit", "action_confirm", "code", "tool", "dual_check")
|
||||
|
||||
|
||||
class NodeRejected(NodeError):
|
||||
"""人的决定让运行到这里结束:驳回、超时驳回(docs/design/next-round/05-已知遗留.md (a))。
|
||||
|
||||
这不是故障。以前六处驳回都抛普通的 NodeError,运行记成 failed:通知推「实例失败」,定时推「运行失败」,发布单把
|
||||
驳回过的试运行说成「试运行失败了」,记录页还给「继续运行」——接着跑只会读到同一个驳回答复、再停一次。现在 runner
|
||||
单独接住它,收成 cancelled + end_reason=rejected,run.cancelled 带上在哪一步、谁、结论和备注,消费方都读这一个字段。
|
||||
|
||||
是 NodeError 的子类:节点这一步照旧记 node.failed(时间线、实例进度按它收尾);但不重试、不被 on_error=continue
|
||||
接住(engine/compiler._execute:那两项是给故障准备的,驳回被吞掉运行就以 succeeded 结束了)。原话(str)照旧写给人看,
|
||||
Run.error 和以前一字不差。kind 见 REJECTION_KINDS;decision 是 reject 或 expired_reject(同待办的结论)。"""
|
||||
|
||||
def __init__(self, node_id: str, message: str, *, kind: str, decision: str = "reject",
|
||||
actor: str | None = None, note: str = "") -> None:
|
||||
super().__init__(node_id, message)
|
||||
self.kind = kind
|
||||
self.decision = decision
|
||||
self.actor = actor
|
||||
self.note = note
|
||||
|
||||
@classmethod
|
||||
def of(cls, ctx: Any, message: str, *, kind: str, decision: Any) -> NodeRejected:
|
||||
"""按节点上下文和读出来的审批结论(engine/approval.Decision)造一个:谁驳回的取 ctx.actor()。"""
|
||||
return cls(ctx.node.id, message, kind=kind, decision="expired_reject" if decision.expired else "reject",
|
||||
actor=ctx.actor(), note=decision.note or "")
|
||||
|
||||
+10
-51
@@ -37,7 +37,6 @@ import re
|
||||
import time
|
||||
from collections.abc import Callable, Iterable
|
||||
from dataclasses import asdict, dataclass
|
||||
from datetime import datetime
|
||||
from typing import Any
|
||||
|
||||
from langchain_core.messages import AIMessage, BaseMessage, HumanMessage, SystemMessage
|
||||
@@ -68,6 +67,7 @@ from app.engine.evidence import (
|
||||
)
|
||||
from app.engine.expressions import CellError, column_kind, locate_cell
|
||||
from app.engine.merge_query import MERGE_SOURCE
|
||||
from app.engine import spend
|
||||
from app.engine.nodes.llm import _split_structured, _usage_of
|
||||
from app.engine.state import message_text
|
||||
from app.providers import catalog as pricing
|
||||
@@ -855,7 +855,7 @@ JUDGE_DEFAULTS: dict[str, Any] = {
|
||||
"click_max_cost_usd": 0.01, # 探索运行里每点一次「请模型判断」的金额上限
|
||||
"daily_max_usd": 2.0, # 全局每日裁判金额上限(按本地日期)
|
||||
}
|
||||
#: 每日计数存在设置表的这一行:{date: 本地日期, usd, calls, unpriced_calls}。不从设置接口进出
|
||||
#: 每日计数存在设置表的这一行:{date: 本地日期, usd, calls, unpriced_calls}。不从设置接口进出(账本见 engine/spend.py)
|
||||
SPEND_KEY = "judge_spend"
|
||||
_COUNTS = frozenset({"report_max_claims"})
|
||||
_SECONDS = frozenset({"report_timeout_s"})
|
||||
@@ -1024,61 +1024,20 @@ async def source_masks(catalog: dict[str, Any], *, loader: Callable[[str], Any]
|
||||
# 每日计数
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
_spend_lock = asyncio.Lock()
|
||||
|
||||
|
||||
def _today() -> str:
|
||||
"""本地日期:每日上限按它滚动。"""
|
||||
return datetime.now().astimezone().date().isoformat()
|
||||
"""裁判这本账的日历:本地日期(同 engine/spend._today)。单独留一个名字:测试换掉它来模拟裁判这边跨天。"""
|
||||
return spend._today()
|
||||
|
||||
|
||||
def _spend_of(row: Any) -> dict[str, Any]:
|
||||
value = row.value if row is not None and isinstance(row.value, dict) else {}
|
||||
today = _today()
|
||||
if value.get("date") != today:
|
||||
return {"date": today, "usd": 0.0, "calls": 0, "unpriced_calls": 0}
|
||||
return {"date": today, "usd": float(value.get("usd") or 0.0), "calls": int(value.get("calls") or 0),
|
||||
"unpriced_calls": int(value.get("unpriced_calls") or 0)}
|
||||
#: 裁判这本每日账(engine/spend.py 的 Ledger:预留、找补、按本地日期滚动和核查、评测是同一套代码,各记各的键)
|
||||
judge_spend = spend.Ledger(SPEND_KEY, today=lambda: _today())
|
||||
_LEDGER = judge_spend
|
||||
|
||||
|
||||
async def daily_spend() -> dict[str, Any]:
|
||||
"""今天(本地日期)裁判已经花了多少:{date, usd, calls, unpriced_calls}。估不出金额的调用只计次数。"""
|
||||
async with SessionLocal() as session:
|
||||
return _spend_of(await session.get(Setting, SPEND_KEY))
|
||||
|
||||
|
||||
async def _charge(amount: float, *, limit: float | None, priced: bool) -> str | None:
|
||||
"""记一次调用、先按估算预留金额。加上这笔会超过每日上限就不记,返回 None;记下了返回记在哪天。"""
|
||||
async with _spend_lock:
|
||||
async with SessionLocal() as session:
|
||||
row = await session.get(Setting, SPEND_KEY)
|
||||
spend = _spend_of(row)
|
||||
if limit is not None and spend["usd"] + amount > limit + 1e-12:
|
||||
return None
|
||||
spend["usd"] = round(spend["usd"] + amount, 8)
|
||||
spend["calls"] += 1
|
||||
spend["unpriced_calls"] += 0 if priced else 1
|
||||
if row is None:
|
||||
session.add(Setting(key=SPEND_KEY, value=spend))
|
||||
else:
|
||||
row.value = spend
|
||||
await session.commit()
|
||||
return spend["date"]
|
||||
|
||||
|
||||
async def _settle(date: str, delta: float) -> None:
|
||||
"""调用结束后按实际用量找补预留的差额。跨了天就不找补:预留记在前一天,今天从零算。"""
|
||||
if not delta:
|
||||
return
|
||||
async with _spend_lock:
|
||||
async with SessionLocal() as session:
|
||||
row = await session.get(Setting, SPEND_KEY)
|
||||
spend = _spend_of(row)
|
||||
if row is None or spend["date"] != date:
|
||||
return
|
||||
spend["usd"] = max(0.0, round(spend["usd"] + delta, 8))
|
||||
row.value = spend
|
||||
await session.commit()
|
||||
return await _LEDGER.daily()
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
@@ -1318,7 +1277,7 @@ async def run_request(request: JudgeRequest, *, spec: ModelSpec, budget: Budget,
|
||||
pending = pending[:len(batch)]
|
||||
pending = pending[len(batch):]
|
||||
charged = cost or 0.0
|
||||
day = await _charge(charged, limit=budget.daily_max_usd if priced else None, priced=priced)
|
||||
day = await _LEDGER.charge(charged, limit=budget.daily_max_usd if priced else None, priced=priced)
|
||||
if day is None: # 估算之后、记账之前别的裁判先花掉了今天剩下的钱
|
||||
stop([*batch, *pending], "daily_max_usd")
|
||||
pending = []
|
||||
@@ -1353,7 +1312,7 @@ async def run_request(request: JudgeRequest, *, spec: ModelSpec, budget: Budget,
|
||||
inp, out = sum(t["input_tokens"] for t in tokens), sum(t["output_tokens"] for t in tokens)
|
||||
# 回复没带用量(有的网关不给、调用超时)就按估算记:宁可多记,不能让每日上限漏掉
|
||||
actual = (pricing.estimate_cost(model_id, inp, out) if inp or out else charged) if priced else 0.0
|
||||
await _settle(day, actual - charged)
|
||||
await _LEDGER.settle(day, actual - charged)
|
||||
spent += actual
|
||||
usage["input_tokens"] += inp
|
||||
usage["output_tokens"] += out
|
||||
|
||||
@@ -67,6 +67,7 @@ from app.core.errors import describe_exception
|
||||
from app.core.events import EventType
|
||||
from app.db.base import SessionLocal
|
||||
from app.db.models import DataSource
|
||||
from app.data import catalog_view
|
||||
from app.engine import deadline, verify_agent
|
||||
from app.engine import judge as judging
|
||||
from app.engine.context import NodeContext, NodeError
|
||||
@@ -767,7 +768,8 @@ class _Review:
|
||||
spec=self.spec, system=CHALLENGE_SYSTEM, task=task_text, sources=live,
|
||||
output_schema=CHALLENGE_SCHEMA, max_queries=self.max_queries, timeout_s=allot.timeout_s,
|
||||
daily_max_usd=self.daily_max_usd, emit=deadline.relabel(ctx.emit, allot), run_id=ctx.run.run_id,
|
||||
node_id=ctx.node.id, data_versions=run_versions(ctx.run), masked_values=known)
|
||||
node_id=ctx.node.id, data_versions=run_versions(ctx.run), masked_values=known,
|
||||
catalog_view=catalog_view.of_run(ctx.run))
|
||||
outcome = await asyncio.to_thread(_masked_rationale, outcome, known, given=[cand.text], kinds=self.kinds,
|
||||
masks=self.masks)
|
||||
if outcome.error and not outcome.model:
|
||||
|
||||
@@ -68,9 +68,10 @@ from app.core.errors import describe_exception
|
||||
from app.core.events import EventType
|
||||
from app.db.base import SessionLocal
|
||||
from app.db.models import DataSource
|
||||
from app.data import catalog_view
|
||||
from app.engine import deadline, recompute
|
||||
from app.engine.approval import read_decision
|
||||
from app.engine.context import NodeContext, NodeError
|
||||
from app.engine.context import NodeContext, NodeError, NodeRejected
|
||||
from app.engine.dual_check_facts import preview, preview_joint, record_choices, upload_dates
|
||||
from app.engine.evidence import build_catalog, is_number, next_exec
|
||||
from app.engine.expressions import ExpressionError, _cell_call, _chain_root, cell_value, column_kind, numeric_text, \
|
||||
@@ -346,9 +347,11 @@ def _skipped(ctx: NodeContext, reason: str) -> dict[str, Any]:
|
||||
|
||||
|
||||
class _Sources:
|
||||
"""一次复算里要读的查询快照、数据源和快照要遮的列:同一份只读一次、只算一次。"""
|
||||
"""一次复算里要读的查询快照、数据源和快照要遮的列:同一份只读一次、只算一次。view 是这次运行的目录视图
|
||||
(data/catalog_view.py):交给盲算的目录、比较两条 SQL 用的中文名都按它读,和运行时工具给模型看的是同一份。"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
def __init__(self, view: Any = None) -> None:
|
||||
self.view = catalog_view.CatalogView.parse(view)
|
||||
self.snapshots: dict[str, tuple[Any, str | None]] = {}
|
||||
self.rows: dict[str, DataSource | None] = {}
|
||||
self.masks: dict[tuple[str, str | None], frozenset[str]] = {}
|
||||
@@ -782,14 +785,14 @@ async def _constant_columns(snapshot: Mapping[str, Any], sources: _Sources) -> f
|
||||
|
||||
async def _catalog_text(sources: _Sources, name: str) -> str:
|
||||
"""交给盲算 agent 的数据目录:表清单(表名、中文名、粒度)和有目录的表的业务说明(业务日期、有效记录条件、去重、
|
||||
字段含义、码值、关联关系)。太长的截断,其余请它用 db_schema 查。"""
|
||||
字段含义、码值、关联关系)。太长的截断,其余请它用 db_schema 查。读哪一份由这次运行的目录视图定(sources.view),
|
||||
和运行时工具给模型看的是同一份。"""
|
||||
from app.data import catalog
|
||||
|
||||
row = await sources.source(name)
|
||||
if row is None:
|
||||
return BLIND_NO_CATALOG_PROMPT
|
||||
async with SessionLocal() as session:
|
||||
entries = await catalog.read_catalog(session, row.id)
|
||||
entries = await catalog_view.read(sources.view, row)
|
||||
cache = row.schema_cache if isinstance(row.schema_cache, dict) else {}
|
||||
tables = cache.get("tables") or {}
|
||||
notes = {table: entry.notes for table, entry in entries.items()}
|
||||
@@ -843,7 +846,7 @@ async def _blind(ctx: NodeContext, jobs: dict[str, dict[str, Any]], options: _Op
|
||||
return {}, None, False, usage
|
||||
async with SessionLocal() as session:
|
||||
spec = await verify_model_spec(session, ctx.config, settings=dict(settings))
|
||||
sources = _Sources()
|
||||
sources = _Sources(catalog_view.of_run(ctx.run))
|
||||
catalogs: dict[str, str] = {}
|
||||
for job in jobs.values():
|
||||
if job["source"] not in catalogs:
|
||||
@@ -870,7 +873,7 @@ async def _blind(ctx: NodeContext, jobs: dict[str, dict[str, Any]], options: _Op
|
||||
spec=spec, system=BLIND_SYSTEM, task=_task_text(job, catalogs[job["source"]]), sources=[job["source"]],
|
||||
output_schema=BLIND_SCHEMA, max_queries=options.max_queries, timeout_s=allot.timeout_s,
|
||||
daily_max_usd=options.daily_max_usd, emit=deadline.relabel(ctx.emit, allot), run_id=ctx.run.run_id,
|
||||
node_id=ctx.node.id, data_versions=run_versions(ctx.run))
|
||||
node_id=ctx.node.id, data_versions=run_versions(ctx.run), catalog_view=sources.view)
|
||||
|
||||
keys = list(jobs)
|
||||
await one(keys[0], jobs[keys[0]])
|
||||
@@ -1078,7 +1081,7 @@ async def _review(state: GraphState, ctx: NodeContext, checked: dict[str, Any],
|
||||
"""
|
||||
rows = [r for r in checked["rows"] if r["status"] == DIFFERENT]
|
||||
results = checked["results"]
|
||||
sources = _Sources()
|
||||
sources = _Sources(catalog_view.of_run(ctx.run))
|
||||
infos: dict[str, dict[str, Any]] = {}
|
||||
metrics: list[dict[str, Any]] = []
|
||||
summary: list[dict[str, Any]] = []
|
||||
@@ -1179,15 +1182,15 @@ async def _source_info(sources: _Sources, name: str) -> dict[str, Any]:
|
||||
(uploaded:导入时日期一律存成文字)和它导入时认作日期的列(upload_dates,按当前版本的配方读,见
|
||||
dual_check_facts.upload_dates)——业务日期据此认上传表格的日期列,认不出的原因照实说(_dated_problem)。两项都进
|
||||
候选:审批卡的预告按发起审批时读的这一份;裁决后的真写按写入时的版本重读 upload_dates,和现读的表结构一起以写入时
|
||||
为准(dual_check_facts.record_choices)。数据源没了就都不给,照样比原文。"""
|
||||
from app.data import catalog
|
||||
为准(dual_check_facts.record_choices)。数据源没了就都不给,照样比原文。目录按这次运行的目录视图读(sources.view),
|
||||
比较两条 SQL 用到的中文名和运行时给模型看的是同一份。"""
|
||||
from app.data.sql_semantics import catalog_labels
|
||||
|
||||
row = await sources.source(name)
|
||||
if row is None:
|
||||
return {"kind": None, "schema": None, "names": {}, "notes": {}, "uploaded": False, "upload_dates": {}}
|
||||
entries = await catalog_view.read(sources.view, row)
|
||||
async with SessionLocal() as session:
|
||||
entries = await catalog.read_catalog(session, row.id)
|
||||
dates = await upload_dates(session, row)
|
||||
notes = {table: entry.notes for table, entry in entries.items()}
|
||||
return {"kind": row.kind, "schema": row.schema_cache if isinstance(row.schema_cache, dict) else None,
|
||||
@@ -1396,7 +1399,7 @@ def _choices(ctx: NodeContext, response: Any, rows: list[dict[str, Any]]) -> tup
|
||||
填写模式一样:这一步没有驳回的出口。"""
|
||||
decision = read_decision(response)
|
||||
if isinstance(response, dict) and decision.rejected:
|
||||
raise NodeError(ctx.node.id, f"人工驳回:{decision.note or '未说明原因'}")
|
||||
raise NodeRejected.of(ctx, f"人工驳回:{decision.note or '未说明原因'}", kind="dual_check", decision=decision)
|
||||
value = decision.value if isinstance(decision.value, dict) else response if isinstance(response, dict) else {}
|
||||
raw = value.get("choices") if isinstance(value.get("choices"), dict) else {}
|
||||
choices = {r["metric_id"]: raw[r["metric_id"]] for r in rows if raw.get(r["metric_id"]) in CHOICES}
|
||||
|
||||
@@ -8,7 +8,7 @@ from typing import Any
|
||||
from app.core.events import EventType
|
||||
from app.db.base import SessionLocal
|
||||
from app.engine.approval import read_decision
|
||||
from app.engine.context import NodeContext, NodeError
|
||||
from app.engine.context import NodeContext, NodeError, NodeRejected
|
||||
from app.engine.replay import ask, once
|
||||
from app.engine.state import GraphState, merge_usage, message_text, template_context
|
||||
from app.providers import catalog
|
||||
@@ -52,7 +52,7 @@ async def run_human(state: GraphState, ctx: NodeContext) -> dict[str, Any]:
|
||||
"__decision__": "approved" if approved else "rejected",
|
||||
}
|
||||
if not approved and ctx.cfg("stop_on_reject", False):
|
||||
raise NodeError(ctx.node.id, f"人工驳回:{result['note'] or '未说明原因'}")
|
||||
raise NodeRejected.of(ctx, f"人工驳回:{result['note'] or '未说明原因'}", kind="approve", decision=decision)
|
||||
else: # edit / input
|
||||
# 驳回在这两种模式下以前是个摆设:__decision__ 硬编码成 approved,
|
||||
# approved 字段看都不看。界面上按钮在、toast 说"已驳回",运行却照常
|
||||
@@ -65,7 +65,7 @@ async def run_human(state: GraphState, ctx: NodeContext) -> dict[str, Any]:
|
||||
# 只有结构体里明说了"不"才算驳回:这两种模式下裸字符串就是内容本身,
|
||||
# 用户在输入框里回一句"no"是在交稿,不是在驳回。
|
||||
if isinstance(response, dict) and decision.rejected:
|
||||
raise NodeError(ctx.node.id, f"人工驳回:{decision.note or '未说明原因'}")
|
||||
raise NodeRejected.of(ctx, f"人工驳回:{decision.note or '未说明原因'}", kind=mode, decision=decision)
|
||||
note = decision.note if isinstance(response, dict) else "" # 裸字符串是内容,不是备注
|
||||
result = {"value": decision.value, "note": note, "__decision__": "approved"}
|
||||
|
||||
|
||||
@@ -48,6 +48,7 @@ from app.providers.factory import (
|
||||
get_chat_model,
|
||||
)
|
||||
from app.engine.sql_problems import event_checks
|
||||
from app.data import catalog_view
|
||||
from app.tools.datasource import QUERY_PREFIX, run_masks_results, run_reads_only, run_scope_fixed, run_versions
|
||||
from app.tools.registry import (
|
||||
ToolArgsError,
|
||||
@@ -341,6 +342,8 @@ async def _resolve_tools(ctx: NodeContext, state: GraphState) -> list[BaseTool]:
|
||||
mask_results=run_masks_results(ctx.run),
|
||||
force_readonly=run_reads_only(ctx.run),
|
||||
fixed_scope=fixed,
|
||||
# 读哪一份数据目录:发起时定下的(data/catalog_view.py),审批恢复、续跑都沿用
|
||||
catalog_view=catalog_view.of_run(ctx.run),
|
||||
# 数据源工具运行中途补固定版本时经它留一条事件(tools/datasource.py 的 _pin_late)
|
||||
emit=ctx.emit,
|
||||
)
|
||||
|
||||
@@ -28,6 +28,7 @@ from app.engine.toolcalls import (
|
||||
from app.providers import catalog
|
||||
from app.providers.factory import ModelSpec, bind_tools_safely, get_chat_model
|
||||
from app.engine.sql_problems import event_checks
|
||||
from app.data import catalog_view
|
||||
from app.tools.datasource import QUERY_PREFIX, run_masks_results, run_reads_only, run_scope_fixed, run_versions
|
||||
from app.tools.registry import (
|
||||
ToolArgsError,
|
||||
@@ -317,6 +318,8 @@ async def run_supervisor(state: GraphState, ctx: NodeContext) -> dict[str, Any]:
|
||||
mask_results=run_masks_results(ctx.run),
|
||||
force_readonly=run_reads_only(ctx.run),
|
||||
fixed_scope=run_scope_fixed(ctx.run),
|
||||
# 读哪一份数据目录:发起时定下的(data/catalog_view.py),审批恢复、续跑都沿用
|
||||
catalog_view=catalog_view.of_run(ctx.run),
|
||||
# 数据源工具运行中途补固定版本时经它留一条事件(tools/datasource.py 的 _pin_late)
|
||||
emit=ctx.emit,
|
||||
)
|
||||
|
||||
@@ -11,7 +11,7 @@ from langgraph.func import task
|
||||
|
||||
from app.core.artifact_store import canonical_json, content_hash
|
||||
from app.engine.approval import read_decision
|
||||
from app.engine.context import NodeContext, NodeError
|
||||
from app.engine.context import NodeContext, NodeError, NodeRejected
|
||||
from app.engine.evidence import ledger_enabled, next_exec, query_entry_fields, schema_ledger_entry
|
||||
from app.engine.replay import ask, once
|
||||
from app.engine.state import GraphState
|
||||
@@ -20,6 +20,7 @@ from app.sandbox.base import SandboxLimits
|
||||
from app.sandbox.manager import sandbox_manager
|
||||
from app.engine.sql_problems import event_checks
|
||||
from app.capabilities.registry import is_platform_query, snapshot_owned
|
||||
from app.data import catalog_view
|
||||
from app.tools.datasource import (DATA_UNAVAILABLE, SCHEMA_PREFIX, run_masks_results, run_reads_only,
|
||||
run_scope_fixed, run_versions)
|
||||
from app.tools.registry import (
|
||||
@@ -54,6 +55,8 @@ def _tool_ctx(ctx: NodeContext) -> ToolContext:
|
||||
mask_results=run_masks_results(ctx.run),
|
||||
force_readonly=run_reads_only(ctx.run),
|
||||
fixed_scope=fixed,
|
||||
# 读哪一份数据目录:发起时定下的(data/catalog_view.py),审批恢复、续跑都沿用
|
||||
catalog_view=catalog_view.of_run(ctx.run),
|
||||
# 数据源工具运行中途补固定版本时经它留一条事件(tools/datasource.py 的 _pin_late)
|
||||
emit=ctx.emit,
|
||||
)
|
||||
@@ -127,8 +130,8 @@ async def run_tool(state: GraphState, ctx: NodeContext) -> dict[str, Any]:
|
||||
await once(ctx, EventType.HUMAN_RESOLVED, tool=name, approved=decision.approved,
|
||||
note=decision.note, actor=ctx.actor(), **({"always": True} if always else {}))
|
||||
if not decision.approved:
|
||||
raise NodeError(ctx.node.id, f"工具 {name} 的调用已被驳回"
|
||||
+ (f":{decision.note}" if decision.note else ""))
|
||||
raise NodeRejected.of(ctx, f"工具 {name} 的调用已被驳回" + (f":{decision.note}" if decision.note else ""),
|
||||
kind="tool", decision=decision)
|
||||
|
||||
limit = limit_of(tool, name, args)
|
||||
ctx.emit(EventType.TOOL_START, tool=name, args=args, **limit_fields(limit))
|
||||
@@ -299,10 +302,11 @@ async def _run_write(state: GraphState, ctx: NodeContext, name: str, args: dict[
|
||||
await once(ctx, EventType.HUMAN_RESOLVED, tool=name, approved=decision.approved, note=decision.note,
|
||||
actor=ctx.actor(), payload_hash=confirmed)
|
||||
if not decision.approved:
|
||||
# 驳回是人的决定,不是故障:运行到这里结束(状态记失败,没有可走的出口),说法写成「未确认(被驳回)」,
|
||||
# 实例页按待办的结论另说(apps/resolve.instance_states)
|
||||
raise NodeError(ctx.node.id, f"「{ctx.node.title}」未确认(被驳回),没有发送"
|
||||
+ (f":{decision.note}" if decision.note else ""))
|
||||
# 驳回是人的决定,不是故障:运行到这里结束(没有可走的出口),收成 cancelled + end_reason=rejected
|
||||
# (NodeRejected),说法写成「未确认(被驳回)」,实例页、通知按同一个结论说(apps/resolve.instance_state)
|
||||
raise NodeRejected.of(ctx, f"「{ctx.node.title}」未确认(被驳回),没有发送"
|
||||
+ (f":{decision.note}" if decision.note else ""),
|
||||
kind="action_confirm", decision=decision)
|
||||
if confirmed != prep.payload_hash:
|
||||
# 动作确认不接受改参数:要改在上游的表单里改。确认之后上游的数据变了,看过的就不是要发的,拒发
|
||||
raise NodeError(ctx.node.id, f"「{name}」确认时看到的内容和现在要发送的不一致(内容指纹对不上),"
|
||||
@@ -380,8 +384,8 @@ async def run_code(state: GraphState, ctx: NodeContext) -> dict[str, Any]:
|
||||
await once(ctx, EventType.HUMAN_RESOLVED, approved=decision.approved, note=decision.note,
|
||||
actor=ctx.actor())
|
||||
if not decision.approved:
|
||||
raise NodeError(ctx.node.id, "代码执行已被驳回"
|
||||
+ (f":{decision.note}" if decision.note else ""))
|
||||
raise NodeRejected.of(ctx, "代码执行已被驳回" + (f":{decision.note}" if decision.note else ""),
|
||||
kind="code", decision=decision)
|
||||
|
||||
# 隔离档位:strict 要硬件级(microVM),fast 要低延迟(Seatbelt/bwrap),
|
||||
# 留空跟随整机默认。要的那档不可用时会安静退回默认,不阻断运行。
|
||||
|
||||
@@ -24,11 +24,34 @@ ORIGIN_KINDS = (
|
||||
"api", # 直接调接口,没说是哪
|
||||
"trial", # 蓝图的试运行(探索运行,docs/design/normal-user-apps.md N4)
|
||||
"personal_app", # 个人应用的刷新(探索运行,N7)
|
||||
# 下一轮(docs/design/next-round/00-总方案.md X1):都只能由服务端写
|
||||
"eval", # 评测:作答和算期望的运行(探索运行)
|
||||
"ledger_issue", # 问数据的答案按已治理口径正式出具(正式运行)
|
||||
"reconcile", # 口径对账:这次的取数和口径在同一份数据上各算一次(探索运行)
|
||||
"recall", # 口径召回:新旧两版口径在同一份数据上各算一次(探索运行)
|
||||
)
|
||||
|
||||
#: 客户端在 POST /api/runs 里能声明的来源
|
||||
CLIENT_ORIGINS = ("chat", "canvas", "launcher", "api")
|
||||
|
||||
#: 系统派生的运行(docs/design/next-round/00-总方案.md X1):评测、口径对账、口径召回都是服务端为了量、为了比替人跑的,
|
||||
#: 不是有人在用。它们是完整的运行(证据、封存都在、点得开),但不进使用统计——探索问题聚类、工具白名单衰减、数据目录的
|
||||
#: 使用次数和「被哪些运行用过」、审阅队列的使用权重、以后的账本统计——否则一跑评测,治理信号和目录的使用热度就失真了。
|
||||
#: 记录页默认也不列(筛选里能选,列表照实写有几次没列出)。ledger_issue 是替用户正式出具,照常计入
|
||||
SYSTEM_ORIGINS = ("eval", "reconcile", "recall")
|
||||
|
||||
|
||||
def counted_runs() -> Any:
|
||||
"""SQL 条件:计入使用统计的运行(排除 SYSTEM_ORIGINS)。所有使用统计都只算它圈进来的运行,别处不另写一份过滤。
|
||||
|
||||
来源按非空比较(coalesce,同 auth/access.hidden_runs):加来源这一列之前的老运行 origin_kind 为空,NOT IN 碰上 NULL
|
||||
会把它们一起筛掉;外连接没对上运行的行(问数据只回了一句、没有运行的那一轮)同理照常计入。"""
|
||||
from sqlalchemy import func
|
||||
|
||||
from app.db.models import Run
|
||||
|
||||
return func.coalesce(Run.origin_kind, "").not_in(SYSTEM_ORIGINS)
|
||||
|
||||
|
||||
def origin_note(kind: str, **data: Any) -> dict[str, Any]:
|
||||
"""run.origin 事件的数据:kind 加上调用方给的字段(app_id、app_version、spec_hash、entry / view、hook、
|
||||
|
||||
@@ -23,7 +23,7 @@ from app.db.base import SessionLocal
|
||||
from app.db.models import Approval, Run, RunEvent, Workflow
|
||||
from app.engine.approval import read_decision
|
||||
from app.engine.compiler import compile_graph, initial_state
|
||||
from app.engine.context import NodeError, RunContext
|
||||
from app.engine.context import NodeError, NodeRejected, RunContext
|
||||
from app.engine.deadline import segment
|
||||
from app.engine.replay import PROTOCOL, PROTOCOL_KEY, protocol_of
|
||||
from app.engine.schema import GraphSpec, NodeType, loop_steps, topology_of, validate_graph
|
||||
@@ -42,9 +42,18 @@ _EPHEMERAL = {EventType.LLM_TOKEN, EventType.LLM_THINKING_DELTA}
|
||||
RUN_OPTIONS_KEY = "run_options"
|
||||
|
||||
|
||||
#: 接着跑被拒的机器码:正式运行带图(F2),有结果未知、还没核实的写入
|
||||
#: 接着跑被拒的机器码:正式运行带图(F2),有结果未知、还没核实的写入,被人驳回了的
|
||||
FORMAL_GRAPH_LOCKED = "formal_graph_locked"
|
||||
ACTION_UNRESOLVED = "action_unresolved"
|
||||
RUN_REJECTED = "run_rejected"
|
||||
|
||||
#: 运行为什么结束在 cancelled(Run.end_reason,docs/design/next-round/00-总方案.md X9)
|
||||
END_REJECTED = "rejected"
|
||||
END_STOPPED = "stopped"
|
||||
#: 被驳回的运行不能继续运行:从断点重跑会读到同一个驳回答复、再停在同一处
|
||||
REJECTED_CONTINUE = "被驳回的运行不能继续运行:改好内容后重新发起"
|
||||
#: 服务重启时还在排队的运行改成失败时写的话(run.error 和补发的 run.failed 是同一句)
|
||||
QUEUED_AT_RESTART = "服务重启时这次运行仍在排队,没有可恢复的进度,请重新发起运行"
|
||||
|
||||
|
||||
class ContinueRefused(ValueError):
|
||||
@@ -94,6 +103,9 @@ class RunManager:
|
||||
stranded = list((await session.execute(
|
||||
select(Run.id, Run.last_seq).where(Run.status == "running")
|
||||
)).all())
|
||||
queued = list((await session.execute(
|
||||
select(Run.id, Run.last_seq).where(Run.status == "queued")
|
||||
)).all())
|
||||
await session.execute(
|
||||
update(Run)
|
||||
.where(Run.status == "running")
|
||||
@@ -102,7 +114,7 @@ class RunManager:
|
||||
await session.execute(
|
||||
update(Run)
|
||||
.where(Run.status == "queued")
|
||||
.values(status="failed", error="服务重启时这次运行仍在排队,没有可恢复的进度,请重新发起运行")
|
||||
.values(status="failed", error=QUEUED_AT_RESTART)
|
||||
)
|
||||
await session.commit()
|
||||
# 进程被强杀时连 server_shutdown 日志都来不及发,事件流停在半路。补上这一条,
|
||||
@@ -113,6 +125,11 @@ class RunManager:
|
||||
"level": "warn", "code": "server_shutdown",
|
||||
"message": "服务上次未正常关闭,这次运行已中断;可从断点继续运行",
|
||||
})
|
||||
# 排队中的改成了失败,也补一条 run.failed:以前只改状态,时间线里没有失败原因这一行(05-已知遗留 (h))。
|
||||
# 不写结束时间、不封存:它从没开始执行,同以前(定时通知按「没有开始时间」认出它,scheduler/webhook.py)
|
||||
for run_id, last_seq in queued:
|
||||
bus.set_seq(run_id, last_seq or 0)
|
||||
await self._emit(run_id, EventType.RUN_FAILED, data={"error": QUEUED_AT_RESTART})
|
||||
|
||||
async def shutdown(self) -> None:
|
||||
self._closing = True
|
||||
@@ -505,8 +522,13 @@ class RunManager:
|
||||
Approval.run_id == run_id, Approval.status.in_(["pending", "answered"])
|
||||
).limit(1)
|
||||
)).first() is not None
|
||||
if run.status == "cancelled" and run.end_reason == END_REJECTED:
|
||||
raise ContinueRefused(REJECTED_CONTINUE, code=RUN_REJECTED)
|
||||
if not (run.status == "failed" or (run.status == "interrupted" and not has_pending)):
|
||||
raise ValueError(_cannot_continue(run.status, has_pending))
|
||||
if run.status == "failed" and await _rejected_before(session, run):
|
||||
# 加 end_reason 之前被驳回的运行记的是 failed(封存不改写):接着跑会读到同一个驳回答复、再停一次
|
||||
raise ContinueRefused(REJECTED_CONTINUE, code=RUN_REJECTED)
|
||||
if graph is not None and run.run_class == "formal":
|
||||
raise ContinueRefused(
|
||||
"正式运行固定在发布的版本上,不能修改配置后继续运行。如需修改,请在画布上修改并发布新版本后重新发起",
|
||||
@@ -613,7 +635,7 @@ class RunManager:
|
||||
# 或者另一个取消先到了,这里就什么都不做
|
||||
claimed = await session.execute(
|
||||
update(Run).where(Run.id == run_id, Run.status == "interrupted")
|
||||
.values(status="cancelled", finished_at=now)
|
||||
.values(status="cancelled", end_reason=END_STOPPED, finished_at=now)
|
||||
)
|
||||
if claimed.rowcount != 1:
|
||||
await session.rollback()
|
||||
@@ -631,7 +653,7 @@ class RunManager:
|
||||
bus.set_seq(run_id, await _seq_floor(session, run_id, run.last_seq))
|
||||
# 和状态同一个事务提交,理由同 _finalize
|
||||
event = self._event(run_id, EventType.RUN_CANCELLED, data={
|
||||
"timing": timing, "actor": actor,
|
||||
"reason": END_STOPPED, "timing": timing, "actor": actor,
|
||||
"message": f"放弃了这次运行,{closed} 条待审批一并关闭" if closed
|
||||
else "放弃了这次挂起的运行",
|
||||
})
|
||||
@@ -684,6 +706,8 @@ class RunManager:
|
||||
#: 终态事件(run.failed / run.cancelled)等 _finalize 算完耗时再发,timing 才带得上
|
||||
terminal: tuple[EventType, dict[str, Any]] | None = None
|
||||
error_node: str | None = None
|
||||
#: 收成 cancelled 的原因(Run.end_reason):rejected 人的驳回,stopped 手动停止
|
||||
end_reason: str | None = None
|
||||
#: 结果未知、挂起等核实的动作(ActionPending),收尾之后发 action.unknown、安排核实
|
||||
pending_action: str | None = None
|
||||
|
||||
@@ -796,7 +820,9 @@ class RunManager:
|
||||
else:
|
||||
status = "cancelled"
|
||||
error = "用户取消"
|
||||
terminal = (EventType.RUN_CANCELLED, {"actor": self._stopped_by.pop(run_id, None)})
|
||||
end_reason = END_STOPPED
|
||||
terminal = (EventType.RUN_CANCELLED, {"reason": END_STOPPED,
|
||||
"actor": self._stopped_by.pop(run_id, None)})
|
||||
raise
|
||||
except ActionPending as e:
|
||||
# 写外部系统的结果未知(app/process/actions.py):不在节点里等,运行收成挂起、不封存,等核实之后
|
||||
@@ -829,6 +855,12 @@ class RunManager:
|
||||
"(AGENTLAB_MAX_GRAPH_STEPS)"
|
||||
)
|
||||
terminal = (EventType.RUN_FAILED, {"error": error})
|
||||
except NodeRejected as e:
|
||||
# 人的决定(驳回、超时驳回),不是故障:收成 cancelled,同手动停止走一样的收尾(关掉并行支路上的审批、
|
||||
# 回收沙箱、关流、封存、不许继续运行)。原话照旧记在 error 上,在哪一步、谁、结论、备注进 run.cancelled
|
||||
status, end_reason = "cancelled", END_REJECTED
|
||||
error, error_node = str(e), e.node_id
|
||||
terminal = (EventType.RUN_CANCELLED, rejected_data(e, spec))
|
||||
except Exception as e: # noqa: BLE001
|
||||
status = "failed"
|
||||
error, terminal = _failure(e, spec)
|
||||
@@ -840,7 +872,7 @@ class RunManager:
|
||||
self._finalizing.add(run_id)
|
||||
try:
|
||||
await self._finalize(run_id, status, error, final_state, elapsed,
|
||||
terminal=terminal, error_node=error_node)
|
||||
terminal=terminal, error_node=error_node, end_reason=end_reason)
|
||||
finally:
|
||||
# 收尾被打断(关停时的取消、库连接出错)也得把自己摘掉。以前写在
|
||||
# _finalize 后面,一被打断就漏掉,留下的任务让下一次 shutdown 的
|
||||
@@ -944,6 +976,7 @@ class RunManager:
|
||||
*,
|
||||
terminal: tuple[EventType, dict[str, Any]] | None = None,
|
||||
error_node: str | None = None,
|
||||
end_reason: str | None = None,
|
||||
) -> None:
|
||||
usage = dict(state.get("usage") or {})
|
||||
output = dict(state.get("output") or {})
|
||||
@@ -1000,6 +1033,7 @@ class RunManager:
|
||||
run.output = output
|
||||
run.usage = usage
|
||||
run.error_node_id = error_node
|
||||
run.end_reason = end_reason
|
||||
if status != "interrupted":
|
||||
run.finished_at = now
|
||||
if status in ("failed", "cancelled"):
|
||||
@@ -1024,11 +1058,12 @@ class RunManager:
|
||||
# 关停时挂起的也要让在线的客户端收尾:进程马上就没了,不说一声,
|
||||
# 界面会一直以为它还在跑
|
||||
await bus.close(run_id)
|
||||
if app_id and status in ("succeeded", "failed"):
|
||||
# 应用实例结束、失败:按应用订阅推送(app/notify/service.py)。放到后台,接收端慢不拖着收尾
|
||||
if app_id and (status in ("succeeded", "failed") or end_reason == END_REJECTED):
|
||||
# 应用实例结束、失败:按应用订阅推送(app/notify/service.py)。放到后台,接收端慢不拖着收尾。被驳回的走
|
||||
# 「实例结束」(next-round X6):发起人要知道结果,订阅「实例失败」的人盯的是故障;手动停止的不推,同以前
|
||||
from app.notify.service import notify_run, spawn
|
||||
|
||||
spawn(notify_run(run_id, "run.finished" if status == "succeeded" else "run.failed"))
|
||||
spawn(notify_run(run_id, "run.failed" if status == "failed" else "run.finished"))
|
||||
|
||||
async def _seal(self, run_id: str) -> None:
|
||||
"""对到此为止的全部事件算清单哈希,记下封到了哪一条。"""
|
||||
@@ -1269,6 +1304,22 @@ def _trust_event(tool_trust: dict[str, str] | None) -> Any:
|
||||
return "formal" if tool_trust.get(FORMAL_MARK) else snapshot_levels(tool_trust)
|
||||
|
||||
|
||||
def rejected_data(exc: NodeRejected, spec: GraphSpec) -> dict[str, Any]:
|
||||
"""被驳回的 run.cancelled 带什么:原因(rejected)、在哪一步(节点 id 和图上的名字)、哪种关卡、结论(驳回 / 超时
|
||||
驳回)、谁、备注。进封存:谁在哪一步驳回、为什么,事后查得到。"""
|
||||
node = spec.node_map().get(exc.node_id)
|
||||
return {"reason": END_REJECTED, "node_id": exc.node_id, "label": node.title if node else exc.node_id,
|
||||
"kind": exc.kind, "decision": exc.decision, "actor": exc.actor, "note": exc.note}
|
||||
|
||||
|
||||
async def _rejected_before(session: Any, run: Run) -> bool:
|
||||
"""加 end_reason 之前被人驳回、记成 failed 的运行:失败的那一步上最近一条处理完的待办是驳回(和实例页认驳回的是
|
||||
同一份判断,apps/resolve.legacy_rejected)。"""
|
||||
from app.apps.resolve import legacy_rejected
|
||||
|
||||
return await legacy_rejected(session, run)
|
||||
|
||||
|
||||
def _failure(exc: BaseException, spec: GraphSpec) -> tuple[str, tuple[EventType, dict[str, Any]]]:
|
||||
"""失败的首行报错(给人看)和 run.failed 的负载(带上定位和原始异常)。"""
|
||||
if isinstance(exc, NodeError):
|
||||
|
||||
+99
-60
@@ -1,11 +1,18 @@
|
||||
"""核查花费账:双盲复算、反方审稿、审批质询里的模型调用,按本地日期记一本每日账。
|
||||
"""模型调用的每日花费账:一本账一个键,按本地日期记,先预留、后找补(docs/design/next-round/01-效果度量与回归评测.md 做法 17)。
|
||||
|
||||
照结论句裁判的每日计数做(judge.py 的 SPEND_KEY、_charge、_settle):调用前按目录价估算、先预留,加上这笔会超过
|
||||
每日上限就不记;调用后按实际用量找补差额。账落在设置表里,按本地日期滚动。
|
||||
调用前按目录价估算、先预留,加上这笔会超过每日上限就不记;调用后按实际用量找补差额。账落在设置表里,按本地日期
|
||||
滚动,不从设置接口进出(api/settings.py 只给只读的「今天花了多少」)。
|
||||
|
||||
**为什么和裁判分开记。** 核查是一个 agent 查库、来回好几轮,一次花的钱比裁判判一份报告多得多。记在同一本账上,
|
||||
核查一多,当天的裁判就全因为「每日上限」判不了——报告的出具会因此降档,而降档的原因和这份报告毫无关系。两本账
|
||||
各管各的上限,互不挤占。
|
||||
**一套代码,几本账。** 以前核查(这个模块)和结论句裁判(judge.py 的 _charge、_settle)各写一份一模一样的读改写,评测
|
||||
要是再抄一份就是第三份。现在都是 Ledger(key) 的实例,各记各的键:
|
||||
|
||||
- verify_spend:核查——双盲复算、反方审稿、审批质询里的模型调用;
|
||||
- judge_spend:结论句裁判(实例在 engine/judge.py,键 judge_spend:日历跟着 judge._today,裁判的测试换它模拟跨天);
|
||||
- eval_spend:评测(预留,第 3 波的评测执行接上;设置和接口在那时加)。
|
||||
|
||||
**为什么各记各的。** 核查是一个 agent 查库、来回好几轮,一次花的钱比裁判判一份报告多得多;评测一次几十道题。记在同一本
|
||||
账上,一边一多,另一边当天就全因为「每日上限」做不了——报告的出具会因此降档,而降档的原因和这份报告毫无关系。几本账
|
||||
各管各的上限,互不挤占。三本按同一个本地日期滚动(_today)。
|
||||
|
||||
目录里没有价格的模型估不出金额:只计次数(unpriced_calls),金额上限对它不起作用,由调用方照实说明。
|
||||
"""
|
||||
@@ -13,27 +20,26 @@ from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import weakref
|
||||
from collections.abc import Callable
|
||||
from datetime import datetime
|
||||
from typing import Any
|
||||
|
||||
from app.db.base import SessionLocal
|
||||
from app.db.models import Setting
|
||||
|
||||
#: 每日计数存在设置表的这一行:{date: 本地日期, usd, calls, unpriced_calls}。不从设置接口进出(api/settings.py)
|
||||
SPEND_KEY = "verify_spend"
|
||||
|
||||
_locks: "weakref.WeakKeyDictionary[asyncio.AbstractEventLoop, asyncio.Lock]" = weakref.WeakKeyDictionary()
|
||||
#: 每本账的读改写锁:{事件循环: {键: 锁}}。asyncio.Lock 不能跨循环用(测试每个用例一个循环),按循环各一组;同一个键
|
||||
#: 同一把——几个核查并行时各自预留,不串行的话两边读到同一个余额,会一起越过上限
|
||||
_locks: "weakref.WeakKeyDictionary[asyncio.AbstractEventLoop, dict[str, asyncio.Lock]]" = weakref.WeakKeyDictionary()
|
||||
|
||||
|
||||
def _lock() -> asyncio.Lock:
|
||||
"""读改写这一行时持的锁,按事件循环各一把(asyncio.Lock 不能跨循环用)。
|
||||
|
||||
几个核查并行时各自预留:不串行的话两边读到同一个余额,会一起越过上限。
|
||||
"""
|
||||
def _lock(key: str) -> asyncio.Lock:
|
||||
loop = asyncio.get_running_loop()
|
||||
lock = _locks.get(loop)
|
||||
group = _locks.get(loop)
|
||||
if group is None:
|
||||
group = _locks[loop] = {}
|
||||
lock = group.get(key)
|
||||
if lock is None:
|
||||
lock = _locks[loop] = asyncio.Lock()
|
||||
lock = group[key] = asyncio.Lock()
|
||||
return lock
|
||||
|
||||
|
||||
@@ -42,57 +48,90 @@ def _today() -> str:
|
||||
return datetime.now().astimezone().date().isoformat()
|
||||
|
||||
|
||||
def _spend_of(row: Any) -> dict[str, Any]:
|
||||
value = row.value if row is not None and isinstance(row.value, dict) else {}
|
||||
today = _today()
|
||||
if value.get("date") != today:
|
||||
return {"date": today, "usd": 0.0, "calls": 0, "unpriced_calls": 0}
|
||||
return {"date": today, "usd": float(value.get("usd") or 0.0), "calls": int(value.get("calls") or 0),
|
||||
"unpriced_calls": int(value.get("unpriced_calls") or 0)}
|
||||
class Ledger:
|
||||
"""一本每日账,存在设置表 key 这一行:{date: 本地日期, usd, calls, unpriced_calls}。
|
||||
|
||||
today 是这本账的日历(测试换掉它来模拟跨天);不给就按本模块的 _today,调用时才取,换掉 _today 对没给 today 的几本
|
||||
账都生效。"""
|
||||
|
||||
def __init__(self, key: str, *, today: Callable[[], str] | None = None) -> None:
|
||||
self.key = key
|
||||
self._clock = today
|
||||
|
||||
def _date(self) -> str:
|
||||
return self._clock() if self._clock is not None else _today()
|
||||
|
||||
def _spend_of(self, row: Any) -> dict[str, Any]:
|
||||
value = row.value if row is not None and isinstance(row.value, dict) else {}
|
||||
today = self._date()
|
||||
if value.get("date") != today:
|
||||
return {"date": today, "usd": 0.0, "calls": 0, "unpriced_calls": 0}
|
||||
return {"date": today, "usd": float(value.get("usd") or 0.0), "calls": int(value.get("calls") or 0),
|
||||
"unpriced_calls": int(value.get("unpriced_calls") or 0)}
|
||||
|
||||
async def daily(self) -> dict[str, Any]:
|
||||
"""今天(本地日期)这本账已经花了多少:{date, usd, calls, unpriced_calls}。估不出金额的调用只计次数。"""
|
||||
async with SessionLocal() as session:
|
||||
return self._spend_of(await session.get(Setting, self.key))
|
||||
|
||||
async def charge(self, amount: float, *, limit: float | None, priced: bool) -> str | None:
|
||||
"""记一次调用、先按估算预留金额。加上这笔会超过每日上限就不记,返回 None;记下了返回记在哪天(找补时要用)。
|
||||
|
||||
limit 为 None 表示不限。估不出金额的调用(priced=False)照记次数,金额记 0,调用方应传 limit=None:
|
||||
拿 0 去比上限,等于说它不花钱。
|
||||
"""
|
||||
async with _lock(self.key):
|
||||
async with SessionLocal() as session:
|
||||
row = await session.get(Setting, self.key)
|
||||
spend = self._spend_of(row)
|
||||
if limit is not None and spend["usd"] + amount > limit + 1e-12:
|
||||
return None
|
||||
spend["usd"] = round(spend["usd"] + amount, 8)
|
||||
spend["calls"] += 1
|
||||
spend["unpriced_calls"] += 0 if priced else 1
|
||||
if row is None:
|
||||
session.add(Setting(key=self.key, value=spend))
|
||||
else:
|
||||
row.value = spend
|
||||
await session.commit()
|
||||
return spend["date"]
|
||||
|
||||
async def settle(self, date: str, delta: float) -> None:
|
||||
"""调用结束后按实际用量找补预留的差额。跨了天就不找补:预留记在前一天,今天从零算。"""
|
||||
if not delta:
|
||||
return
|
||||
async with _lock(self.key):
|
||||
async with SessionLocal() as session:
|
||||
row = await session.get(Setting, self.key)
|
||||
spend = self._spend_of(row)
|
||||
if row is None or spend["date"] != date:
|
||||
return
|
||||
spend["usd"] = max(0.0, round(spend["usd"] + delta, 8))
|
||||
row.value = spend
|
||||
await session.commit()
|
||||
|
||||
|
||||
#: 核查、评测各一本(裁判那本在 engine/judge.py 的 judge_spend)
|
||||
verify_spend = Ledger("verify_spend")
|
||||
eval_spend = Ledger("eval_spend")
|
||||
|
||||
#: 核查那本账的老名字(verify_agent、设置接口、已有测试从这里取)
|
||||
SPEND_KEY = verify_spend.key
|
||||
|
||||
|
||||
async def daily_spend() -> dict[str, Any]:
|
||||
"""今天(本地日期)核查已经花了多少:{date, usd, calls, unpriced_calls}。估不出金额的调用只计次数。"""
|
||||
async with SessionLocal() as session:
|
||||
return _spend_of(await session.get(Setting, SPEND_KEY))
|
||||
"""今天核查已经花了多少(verify_spend.daily)。"""
|
||||
return await verify_spend.daily()
|
||||
|
||||
|
||||
async def charge(amount: float, *, limit: float | None, priced: bool) -> str | None:
|
||||
"""记一次调用、先按估算预留金额。加上这笔会超过每日上限就不记,返回 None;记下了返回记在哪天(找补时要用)。
|
||||
|
||||
limit 为 None 表示不限。估不出金额的调用(priced=False)照记次数,金额记 0,调用方应传 limit=None:
|
||||
拿 0 去比上限,等于说它不花钱。
|
||||
"""
|
||||
async with _lock():
|
||||
async with SessionLocal() as session:
|
||||
row = await session.get(Setting, SPEND_KEY)
|
||||
spend = _spend_of(row)
|
||||
if limit is not None and spend["usd"] + amount > limit + 1e-12:
|
||||
return None
|
||||
spend["usd"] = round(spend["usd"] + amount, 8)
|
||||
spend["calls"] += 1
|
||||
spend["unpriced_calls"] += 0 if priced else 1
|
||||
if row is None:
|
||||
session.add(Setting(key=SPEND_KEY, value=spend))
|
||||
else:
|
||||
row.value = spend
|
||||
await session.commit()
|
||||
return spend["date"]
|
||||
"""核查记一次调用(verify_spend.charge)。"""
|
||||
return await verify_spend.charge(amount, limit=limit, priced=priced)
|
||||
|
||||
|
||||
async def settle(date: str, delta: float) -> None:
|
||||
"""调用结束后按实际用量找补预留的差额。跨了天就不找补:预留记在前一天,今天从零算。"""
|
||||
if not delta:
|
||||
return
|
||||
async with _lock():
|
||||
async with SessionLocal() as session:
|
||||
row = await session.get(Setting, SPEND_KEY)
|
||||
spend = _spend_of(row)
|
||||
if row is None or spend["date"] != date:
|
||||
return
|
||||
spend["usd"] = max(0.0, round(spend["usd"] + delta, 8))
|
||||
row.value = spend
|
||||
await session.commit()
|
||||
"""核查找补(verify_spend.settle)。"""
|
||||
await verify_spend.settle(date, delta)
|
||||
|
||||
|
||||
__all__ = ["SPEND_KEY", "charge", "daily_spend", "settle"]
|
||||
__all__ = ["Ledger", "SPEND_KEY", "charge", "daily_spend", "eval_spend", "settle", "verify_spend"]
|
||||
|
||||
@@ -758,6 +758,7 @@ async def run_verify_agent(
|
||||
node_id: str = "",
|
||||
data_versions: dict[str, Any] | Versions | None = Versions.UNSET,
|
||||
masked_values: Any = None,
|
||||
catalog_view: Any = None,
|
||||
) -> VerifyOutcome:
|
||||
"""让一个只读 agent 查库,按 output_schema 交回有出处的结果。永远不抛。
|
||||
|
||||
@@ -775,6 +776,8 @@ async def run_verify_agent(
|
||||
- masked_values:调用方已经认出的遮罩值(engine/masking 的 _Secrets,审批质询、反方审稿收的报告快照里的值)。
|
||||
查询工具从第一条查询起就按它们按值遮:agent 查到的备注、别的表里写着同一个号码时,模型看不到。随查询时遮掉的值
|
||||
一起交回(VerifyOutcome.masked_values)。不改它。
|
||||
- catalog_view:db_schema 给 agent 看哪一份数据目录(data/catalog_view.py)。运行里的节点传这次运行的视图
|
||||
(catalog_view.of_run);不给是当前目录(质询接口)。
|
||||
"""
|
||||
state = _State(started=time.monotonic(), seen=SeenValues(
|
||||
given=masked_values if masked_values is not None else _Secrets()))
|
||||
@@ -782,7 +785,7 @@ async def run_verify_agent(
|
||||
out = await _verify(state, spec=spec, system=system, task=task, sources=sources,
|
||||
output_schema=output_schema, max_queries=max_queries, timeout_s=timeout_s,
|
||||
max_cost_usd=max_cost_usd, daily_max_usd=daily_max_usd, max_steps=max_steps, emit=emit,
|
||||
run_id=run_id, node_id=node_id, data_versions=data_versions)
|
||||
run_id=run_id, node_id=node_id, data_versions=data_versions, catalog_view=catalog_view)
|
||||
except Exception as e: # noqa: BLE001 - 永远不抛:库写不进去、建工具出错这类意外也落在返回值里
|
||||
logger.warning("核查出错", exc_info=True)
|
||||
out = state.outcome(error=f"核查出错({describe_exception(e)})")
|
||||
@@ -798,6 +801,7 @@ async def run_verify_agent(
|
||||
async def _verify(state: _State, *, spec: ModelSpec, system: str, task: str, sources: Iterable[str],
|
||||
output_schema: dict[str, Any], max_queries: int | None, timeout_s: float | None,
|
||||
max_cost_usd: float | None, daily_max_usd: float | None | Default, max_steps: int | None,
|
||||
catalog_view: Any = None,
|
||||
emit: Emit | None, run_id: str, node_id: str,
|
||||
data_versions: dict[str, Any] | Versions | None) -> VerifyOutcome:
|
||||
deadline = state.started + timeout_s if timeout_s is not None else None
|
||||
@@ -834,7 +838,7 @@ async def _verify(state: _State, *, spec: ModelSpec, system: str, task: str, sou
|
||||
|
||||
# ---- 工具:只读的 db_schema / db_query,查询结果先遮再交给模型(遮掉的值并进 state.seen)
|
||||
wanted = list(dict.fromkeys(s for s in sources if isinstance(s, str) and s))
|
||||
ctx = ToolContext(run_id=run_id, node_id=node_id, data_versions=data_versions, emit=emit,
|
||||
ctx = ToolContext(run_id=run_id, node_id=node_id, data_versions=data_versions, emit=emit, catalog_view=catalog_view,
|
||||
extra={MASKED_VALUES_KEY: state.seen})
|
||||
async with SessionLocal() as session:
|
||||
tools = await build_datasource_tools([f"{p}{s}" for s in wanted for p in (SCHEMA_PREFIX, QUERY_PREFIX)], ctx,
|
||||
|
||||
@@ -159,12 +159,16 @@ class TableKind:
|
||||
|
||||
@classmethod
|
||||
async def load(cls, sources: list[Any], *, graph: Mapping[str, Any] | None = None,
|
||||
picker: Picker | None = None) -> TableKind:
|
||||
"""读各源的数据目录。graph 是现有工作流(改图时的原图、追问时上一轮的图),概览里列出它查过的表。"""
|
||||
from app.tools.datasource import _catalog_of
|
||||
picker: Picker | None = None, catalog_view: Any = None) -> TableKind:
|
||||
"""读各源的数据目录。graph 是现有工作流(改图时的原图、追问时上一轮的图),概览里列出它查过的表。
|
||||
|
||||
读哪一份目录由 catalog_view 定(data/catalog_view.py:当前目录、目录快照、关;只用已核实的项),不给是当前目录。
|
||||
查看结果里的目录和运行时 db_schema 给模型看的、自查用的是同一份——评测的「目录对照」量的才是整份贡献。"""
|
||||
from app.data.catalog_view import CatalogView, read
|
||||
|
||||
view = CatalogView.parse(catalog_view)
|
||||
seeds = graph_tables(graph, sources)
|
||||
plans = [SourcePlan(source=s, entries=await _catalog_of(s) if introspect.table_names(s) else {},
|
||||
plans = [SourcePlan(source=s, entries=await read(view, s) if introspect.table_names(s) else {},
|
||||
seeds=seeds.get(s.name, [])) for s in sources]
|
||||
return cls(plans, picker=picker)
|
||||
|
||||
|
||||
+11
-3
@@ -30,6 +30,7 @@ from app.core.config import settings
|
||||
from app.db.base import init_db
|
||||
from app.engine.runner import run_manager
|
||||
from app.seed import seed_defaults
|
||||
from app.web_security import SecurityHeaders
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
@@ -145,16 +146,20 @@ async def lifespan(_: FastAPI) -> AsyncIterator[None]:
|
||||
|
||||
|
||||
async def _platform_startup() -> None:
|
||||
"""应用平台的启动整理(BE-1):登录设置读进缓存;自定义工具请求头、MCP env 和请求头里的存量明文加密一遍(幂等);
|
||||
存量的内网 MCP 服务改为绑定登记端点(只在升级后第一次启动时做);能力登记同步一次。后三样出错只记日志:最坏是登记
|
||||
暂时不全,不该让服务起不来。"""
|
||||
"""应用平台的启动整理(BE-1):登录设置、对外访问地址读进缓存;自定义工具请求头、MCP env 和请求头里的存量明文加密
|
||||
一遍(幂等);存量的内网 MCP 服务改为绑定登记端点(只在升级后第一次启动时做);能力登记同步一次。后三样出错只记
|
||||
日志:最坏是登记暂时不全,不该让服务起不来。"""
|
||||
from app.auth import current_mode, load_settings
|
||||
from app.capabilities import registry as capability_registry
|
||||
from app.capabilities.secrets import seal_existing
|
||||
from app.core import public_url
|
||||
from app.db.base import SessionLocal
|
||||
|
||||
await load_settings(force=True)
|
||||
logger.info("登录方式:%s", current_mode())
|
||||
# 对外访问地址(core/public_url.py)同样读进缓存:中间件核对来源时不查库
|
||||
if address := await public_url.refresh():
|
||||
logger.info("对外访问地址:%s(%s)", address, "环境变量" if public_url.env_locked() else "设置")
|
||||
try:
|
||||
if sealed := await seal_existing():
|
||||
logger.info("已加密 %d 处存量的明文密钥(自定义工具请求头、MCP 环境变量和请求头)", sealed)
|
||||
@@ -192,6 +197,8 @@ app.add_middleware(
|
||||
allow_methods=["*"],
|
||||
allow_headers=["*"],
|
||||
)
|
||||
# 安全响应头(app/web_security.py):最后加、在最外层,CORS 预检和登录检查自己回的 401 / 403 也带上
|
||||
app.add_middleware(SecurityHeaders)
|
||||
|
||||
# 路由级权限(§3.12)。关登录时这些依赖全部放行;开了登录,中间件先挡掉没登录的,这里再按角色和运行挡
|
||||
_BUILDER = [require_role("builder")]
|
||||
@@ -238,6 +245,7 @@ for router, guard in (
|
||||
(identity_api.router, _ADMIN),
|
||||
# 选人用的名录(用户名、显示名、是否停用、组名):岗位绑定、应用授权由搭建者做,只读
|
||||
(identity_api.directory_router, _BUILDER),
|
||||
(identity_api.candidates_router, _USER),
|
||||
# 能力登记、端点:搭建者只读,改动在接口里要管理员
|
||||
(capabilities_api.router, _BUILDER),
|
||||
# 普通用户建应用:个人应用的接口共用 /api/apps 前缀,要在 apps_api 之前登记;蓝图助手共用 /api/copilot 前缀,
|
||||
|
||||
+102
-21
@@ -7,15 +7,29 @@
|
||||
同一个渠道只推一次。待办改派、升级、催办都是「有一条待办要你处理」,订阅了 task.opened 的渠道一并收到。
|
||||
|
||||
消息里只放标题、状态、链接(P4,沿用定时任务 Webhook 的规矩):不放报告里的数字,待办的标题用节点在图上的名字
|
||||
(搭图时起的),不用按运行数据渲染出来的那一句。链接要设了对外访问地址(设置里的 notify)才有。
|
||||
(搭图时起的),不用按运行数据渲染出来的那一句。链接要有对外访问地址(环境变量或设置,core/public_url.py)才有。
|
||||
|
||||
推送放在后台(spawn):接收端挂住时超时加重试要几十秒,不能拖着开待办、运行收尾。测试用 drain 等它们推完。
|
||||
|
||||
事件只在这里登记(EVENTS,总方案 X6):每个事件一行,名字、界面上的叫法(前端 lib/terms.ts 的 NOTIFY_TEXT.eventLabel
|
||||
逐字一致,test_notify_events 对照)、推给哪几条路(scopes:global 全局渠道自己订阅、user 相关的人自己的个人渠道、app 应用
|
||||
版本 spec.notify 的订阅)。_channels 只走登记了的路,没登记的事件推不出去;接口按 scopes 收订阅,设置页和应用的通知按
|
||||
GET /api/notify/events 列。scopes 是空的,是登记了、还没有推送的事件:不收订阅、界面上不列——勾了却收不到,就是「看着
|
||||
生效、其实没生效」(个人渠道勾了实例结束、失败却收不到,04-对外集成与触达 B1,就是这么错的)。本轮第 1 波把六个新事件
|
||||
一次登记好;实现推送的那一块写自己的 notify_*,只改自己那一行的 scopes,不再改这张表的别处、也不再改前端的叫法表:几块
|
||||
并行改同一处必然冲突。以前存下的订阅不按 scopes 重新核对(api/notify._validated 只核对新勾的)。
|
||||
|
||||
消息的结构,新事件一律照这个写(P4、D9):标题、状态、条数、链接,外加编号(#前 6 位)和搭建时起的名字(应用、工作流、
|
||||
节点在图上的名字,题集、数据源、访问密钥的名字)。条数说的是有几件事要处理(几个变化、通过几题、几条待办),可以放。
|
||||
不放:报告和答案里的数(金额、比例、差值、期望值),表名、列名、取值(码值、样例行),SQL,按运行数据渲染出来的句子。
|
||||
接收端是群机器人、外部系统,消息出了这个服务就不归遮罩管了。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from collections.abc import Coroutine
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import select
|
||||
@@ -26,12 +40,60 @@ from app.notify.core import Message, Outcome, host_of
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
#: 可以订阅的事件
|
||||
EVENTS = ("task.opened", "run.finished", "run.failed", "action.unknown", "review.done")
|
||||
EVENT_LABEL = {"task.opened": "有待办派给我", "run.finished": "实例完成", "run.failed": "实例失败",
|
||||
"action.unknown": "写入结果未知",
|
||||
# 个人应用提交审核之后,搭建者发布了或退回了(docs/design/normal-user-apps.md N7)
|
||||
"review.done": "我提交审核的应用有结果"}
|
||||
@dataclass(frozen=True)
|
||||
class Event:
|
||||
"""一个可订阅的通知事件(见模块说明「事件只在这里登记」)。"""
|
||||
|
||||
#: 界面上的叫法:和前端 NOTIFY_TEXT.eventLabel 同一句
|
||||
label: str
|
||||
#: 推给哪几条路(_channels):global 全局渠道自己订阅,user 相关的人自己的个人渠道,app 应用版本 spec.notify 的订阅。
|
||||
#: 空的:登记了、还没有推送
|
||||
scopes: tuple[str, ...]
|
||||
|
||||
|
||||
_EVERY = ("global", "user", "app")
|
||||
#: 渠道范围在说法里怎么叫(同前端 NOTIFY_TEXT.scopes)
|
||||
SCOPE_TEXT = {"global": "全局", "user": "个人", "app": "应用"}
|
||||
|
||||
#: 可以订阅的事件,只在这里登记
|
||||
EVENTS: dict[str, Event] = {
|
||||
# 待办派出、改派、升级、催办(_AS):全局渠道、派给的人自己的个人渠道、应用的订阅
|
||||
"task.opened": Event("待办打开", _EVERY),
|
||||
# 应用实例结束、失败:个人渠道是发起人自己的(04-对外集成与触达 B1)
|
||||
"run.finished": Event("运行结束", _EVERY),
|
||||
"run.failed": Event("运行失败", _EVERY),
|
||||
# 写入外部系统的结果未知:个人渠道是核实待办派给的人
|
||||
"action.unknown": Event("动作结果未知", _EVERY),
|
||||
# 个人应用提交审核之后,搭建者发布了或退回了(docs/design/normal-user-apps.md N7):发给提交人,不归哪个应用
|
||||
"review.done": Event("我提交审核的应用有结果", ("global", "user")),
|
||||
# ---- 本轮登记、还没有推送(总方案 X6):实现推送的那一块只改自己这一行的 scopes,消息照模块说明的规矩 ----
|
||||
# 数据源有要处理的结构 / 取值变化(03 B3):只推 severity 为 action 的,按源汇总;源名、各级别条数、链接
|
||||
"source.changed": Event("数据源有变化要处理", ()),
|
||||
# 口径升版后召回重算,数真变了(02 第 7 项):答案的主人有 recall 待办;口径名、版本、链接,不放新旧值和差值
|
||||
"ledger.recall": Event("口径升版,答案的数变了", ()),
|
||||
# 定时评测有题从通过变成不通过 / 评测没跑完(01 第 15 项):题集名、通过数、退步题数、链接;不放期望值和答案里的数
|
||||
"eval.regressed": Event("评测有退步", ()),
|
||||
"eval.failed": Event("评测失败", ()),
|
||||
# 每天一条待办汇总(04 B3):只推个人渠道;条数、最早的时限、最多 5 条(节点在图上的名字)、去待办箱的链接
|
||||
"task.digest": Event("每天的待办汇总", ()),
|
||||
# 访问密钥到期前 7 天、1 天各一次(04 B3):个人密钥推本人,服务账号的推全局渠道;密钥名、到期日、链接
|
||||
"key.expiring": Event("访问密钥快到期", ()),
|
||||
}
|
||||
EVENT_LABEL = {name: event.label for name, event in EVENTS.items()}
|
||||
|
||||
|
||||
def subscribe_problem(event: str, scope: str) -> str | None:
|
||||
"""范围是 scope 的渠道能不能订阅 event:不能的回一句为什么(接口原样回 422),能的是 None。"""
|
||||
spec = EVENTS.get(event)
|
||||
if spec is None:
|
||||
offered = "、".join(e.label for e in EVENTS.values() if scope in e.scopes)
|
||||
return f"不认识的通知事件:{event}。可选:{offered}"
|
||||
if not spec.scopes:
|
||||
return f"「{spec.label}」的通知还没有推送,暂时不能订阅"
|
||||
if scope not in spec.scopes:
|
||||
return f"「{spec.label}」不推给范围是「{SCOPE_TEXT.get(scope, scope)}」的渠道"
|
||||
return None
|
||||
|
||||
#: 这几种也是「有一条待办要你处理」,按 task.opened 的订阅推
|
||||
_AS = {"task.reassigned": "task.opened", "task.escalated": "task.opened", "task.reminded": "task.opened"}
|
||||
_TASK_HEAD = {"task.opened": "新的待办", "task.reassigned": "待办已改派给你", "task.escalated": "待办已超时,升级处理",
|
||||
@@ -91,15 +153,19 @@ def generic_body(message: dict[str, Any], msg: Message) -> dict[str, Any]:
|
||||
|
||||
|
||||
async def _base_url(session: Any) -> str | None:
|
||||
from app.api.settings import notify_settings
|
||||
"""拼链接用的对外访问地址:全站同一个(环境变量优先,其次设置),手里有会话就按库里的值对齐缓存。"""
|
||||
from app.core import public_url
|
||||
|
||||
return (await notify_settings(session))["public_base_url"]
|
||||
return await public_url.refresh(session)
|
||||
|
||||
|
||||
async def _channels(session: Any, event: str, run: Run | None, users: list[str]) -> list[NotifyChannel]:
|
||||
wanted = _AS.get(event, event)
|
||||
if (registered := EVENTS.get(wanted)) is None:
|
||||
raise ValueError(f"通知事件 {wanted} 没有登记:先在 notify/service.EVENTS 登记,再推送")
|
||||
paths = registered.scopes
|
||||
found: dict[str, NotifyChannel] = {}
|
||||
if run is not None and run.app_id and run.app_version is not None:
|
||||
if "app" in paths and run is not None and run.app_id and run.app_version is not None:
|
||||
row = (await session.execute(select(AppVersion.spec, AppVersion.bindings).where(
|
||||
AppVersion.app_id == run.app_id, AppVersion.version == run.app_version))).first()
|
||||
spec, bindings = (row[0], row[1]) if row is not None else ({}, {})
|
||||
@@ -115,11 +181,12 @@ async def _channels(session: Any, event: str, run: Run | None, users: list[str])
|
||||
for row in (await session.execute(select(NotifyChannel).where(
|
||||
NotifyChannel.id.in_(subscribed), NotifyChannel.enabled.is_(True)))).scalars():
|
||||
found[row.id] = row
|
||||
for row in (await session.execute(select(NotifyChannel).where(
|
||||
NotifyChannel.scope == "global", NotifyChannel.enabled.is_(True)))).scalars():
|
||||
if wanted in (row.events or []):
|
||||
found.setdefault(row.id, row)
|
||||
if users:
|
||||
if "global" in paths:
|
||||
for row in (await session.execute(select(NotifyChannel).where(
|
||||
NotifyChannel.scope == "global", NotifyChannel.enabled.is_(True)))).scalars():
|
||||
if wanted in (row.events or []):
|
||||
found.setdefault(row.id, row)
|
||||
if "user" in paths and users:
|
||||
owners = list((await session.execute(select(User.id).where(User.username.in_(users)))).scalars())
|
||||
if owners:
|
||||
for row in (await session.execute(select(NotifyChannel).where(
|
||||
@@ -201,22 +268,36 @@ async def notify_task(task_id: str, event: str, *, users: list[str]) -> int:
|
||||
|
||||
|
||||
async def notify_run(run_id: str, event: str) -> int:
|
||||
"""应用实例结束、失败(run.finished / run.failed)。只推应用发起的运行。"""
|
||||
"""应用实例结束、失败(run.finished / run.failed)。只推应用发起的运行。
|
||||
|
||||
状态那一句取自实例页的说法(apps/resolve.instance_state),两边说同一句话:被驳回的(cancelled + end_reason=rejected)
|
||||
走「实例结束」、写「「付款」未确认(被驳回)」;走到「已退回」成果的写「已退回」,不再一律写「已完成」。ok 只在真的
|
||||
办成了(code 是 succeeded)时为真。
|
||||
|
||||
发起人是认证过的身份(run.origin.actor_verified)时,他自己订阅了这件事的个人渠道也收得到(「我发起的实例完成 / 失败」,
|
||||
04-对外集成与触达 B1)。以前这里给 _channels 的 users 是空的,个人渠道勾了这两项永远收不到。自报的署名不算:关登录
|
||||
时谁都能填别人的名字。"""
|
||||
from app.apps.resolve import _trails, instance_state
|
||||
|
||||
async with SessionLocal() as session:
|
||||
run = await session.get(Run, run_id)
|
||||
if run is None or not run.app_id:
|
||||
return 0
|
||||
channels = await _channels(session, event, run, [])
|
||||
verified = bool((run.origin or {}).get("actor_verified")) and bool(run.started_by)
|
||||
channels = await _channels(session, event, run, [run.started_by] if verified else [])
|
||||
if not channels:
|
||||
return 0
|
||||
base = await _base_url(session)
|
||||
where = await _where(session, run)
|
||||
state = instance_state(run, (await _trails(session, [run]))[run.id])
|
||||
finished = event == "run.finished"
|
||||
lines = [("状态", "已完成" if finished else "失败"), ("实例", f"#{run_id[:6]}")]
|
||||
if not finished and run.error_node_id:
|
||||
ok = state["code"] == "succeeded"
|
||||
lines = [("状态", state["text"]), ("实例", f"#{run_id[:6]}")]
|
||||
if not finished and run.error_node_id and state["code"] == "failed":
|
||||
lines.append(("失败节点", f"「{_label(run.graph, run.error_node_id)}」"))
|
||||
message = {"event": event, "status": run.status, "ok": finished,
|
||||
"title": f"{where or '应用'}的实例{'已完成' if finished else '失败'}", "lines": lines,
|
||||
head = ("已完成" if ok else "已结束") if finished else ("已结束" if state["code"] == "cancelled" else "失败")
|
||||
message = {"event": event, "status": run.status, "ok": ok,
|
||||
"title": f"{where or '应用'}的实例{head}", "lines": lines,
|
||||
"links": [("查看实例", f"{base}/apps/{run.app_id}/i/{run_id}")] if base else [],
|
||||
"subject": {"run_id": run_id, "app_id": run.app_id}}
|
||||
return await _push(channels, message, subject_kind="run", subject_id=run_id, event=event, run_id=run_id,
|
||||
|
||||
@@ -181,6 +181,9 @@ PLAIN: dict[str, Plain] = {
|
||||
"plan.trial_failed": Plain(
|
||||
"最近一次试运行失败了", "用真数运行出了错,这样发布出去,每次出数都会是同一个错。", _Y,
|
||||
"回到试运行看出错的原因(可以交给助手修),修好后再用真数试一次。"),
|
||||
"plan.trial_rejected": Plain(
|
||||
"最近一次试运行被驳回、没有走完", "那是你(或审批人)的决定,不是出错;驳回之后的步骤没用真数看过。", _Y,
|
||||
"要看完整结果,再试运行一次并批准;确认不需要的话可以直接发布。"),
|
||||
"plan.untrialed": Plain(
|
||||
"还没用真数试运行过", "没看过真数就发布,可能不是你要的结果。", _Y, "先试运行一次,看过结果再发布。"),
|
||||
"plan.trial_differs": Plain(
|
||||
|
||||
@@ -572,17 +572,18 @@ async def _continue(run_id: str, actor: str | None) -> Any:
|
||||
return None
|
||||
|
||||
|
||||
async def tick(now: datetime | None = None) -> int:
|
||||
"""调度循环每一拍:对登记了 lookup 的结果未知动作带退避地查询。返回查了几条。"""
|
||||
async def tick(now: datetime | None = None) -> list[str]:
|
||||
"""调度循环每一拍:对登记了 lookup 的结果未知动作带退避地查询。交回这一拍查过的动作记录 id(同 tasks.tick)。"""
|
||||
now = now or utcnow()
|
||||
async with SessionLocal() as session:
|
||||
due = list((await session.execute(select(ActionRecord.id, ActionRecord.next_lookup_at).where(
|
||||
ActionRecord.state == "unknown", ActionRecord.next_lookup_at.is_not(None),
|
||||
ActionRecord.next_lookup_at <= now))).all())
|
||||
done = 0
|
||||
done: list[str] = []
|
||||
for action_id, claimed_at in due:
|
||||
try:
|
||||
done += await _lookup(action_id, claimed_at, now)
|
||||
if await _lookup(action_id, claimed_at, now):
|
||||
done.append(action_id)
|
||||
except Exception: # noqa: BLE001 - 一条出错不拖住这一拍的其他动作
|
||||
logger.exception("动作 %s 的结果查询失败,下一拍再试", action_id)
|
||||
return done
|
||||
|
||||
@@ -406,6 +406,22 @@ def _check_hash(card: dict[str, Any], given: Any, *, approving: bool) -> None:
|
||||
status_code=422, code="payload_hash_mismatch")
|
||||
|
||||
|
||||
async def _check_principals(session: AsyncSession, task: Task, principal: Any, values: dict[str, Any]) -> None:
|
||||
"""待办表单里的选人控件:开了登录时按候选池核对(同入口表单,auth/directory.principal_errors)。不属于应用的待办
|
||||
(试运行、画布)没有池,不核对。"""
|
||||
from app.auth.directory import principal_errors
|
||||
|
||||
if not task.app_id or getattr(principal, "mode", "off") == "off":
|
||||
return
|
||||
app = await session.get(App, task.app_id)
|
||||
if app is None:
|
||||
return
|
||||
bindings = await session.scalar(select(AppVersion.bindings).where(
|
||||
AppVersion.app_id == task.app_id, AppVersion.version == task.app_version)) if task.app_version else None
|
||||
if errors := await principal_errors(session, app, bindings, task.form, values, principal.mode):
|
||||
raise FormInvalid(errors)
|
||||
|
||||
|
||||
def _validated(form: dict[str, Any], values: dict[str, Any]) -> dict[str, Any]:
|
||||
from app.apps.forms import validate_form
|
||||
|
||||
@@ -491,6 +507,8 @@ async def respond(task_id: str, principal: Any, *, decision: Any, values: dict[s
|
||||
rows = (await _assignments(session, [task.id])).get(task.id, [])
|
||||
mine = _pick(task, rows, principal)
|
||||
response, verdict = _engine_response(task, approval, decision, values, note, payload_hash)
|
||||
if task.form and verdict in ("approve", "submit") and values:
|
||||
await _check_principals(session, task, principal, values)
|
||||
response = await _bind_files(session, task, principal, response)
|
||||
now = utcnow()
|
||||
marked = await session.execute(
|
||||
@@ -761,8 +779,9 @@ async def _notify(task_id: str, event: str, users: list[str]) -> int:
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def tick(now: datetime | None = None) -> int:
|
||||
"""调度循环每一拍:提醒、升级、超时驳回(永不自动批准)。返回处理的条数。试运行的待办不处理(只显示时限)。"""
|
||||
async def tick(now: datetime | None = None) -> list[str]:
|
||||
"""调度循环每一拍:提醒、升级、超时驳回(永不自动批准)。交回这一拍处理过的待办 id(生效可查:办了哪几条,
|
||||
不只是几条;测试按自己造的那条断言,不受同一个库里别的待办影响)。试运行的待办不处理(只显示时限)。"""
|
||||
now = now or utcnow()
|
||||
action = func.json_extract(Task.on_timeout, "$.action")
|
||||
trial = select(Run.id).where(Run.origin_kind == TRIAL_ORIGIN)
|
||||
@@ -773,10 +792,11 @@ async def tick(now: datetime | None = None) -> int:
|
||||
and_(Task.due_at.is_not(None), Task.due_at <= now, action == "reject"),
|
||||
and_(Task.due_at.is_not(None), Task.due_at <= now, action == "escalate", Task.escalation_level == 0),
|
||||
)).order_by(Task.due_at))).scalars())
|
||||
done = 0
|
||||
done: list[str] = []
|
||||
for task_id in due:
|
||||
try:
|
||||
done += await _on_timeout(task_id, now)
|
||||
if await _on_timeout(task_id, now):
|
||||
done.append(task_id)
|
||||
except Exception: # noqa: BLE001 - 一条出错不拖住这一拍的其他待办
|
||||
logger.exception("待办 %s 的超时处理失败,下一拍再试", task_id)
|
||||
return done
|
||||
@@ -838,7 +858,9 @@ async def _on_timeout(task_id: str, now: datetime) -> int:
|
||||
from app.engine.runner import run_manager
|
||||
|
||||
try:
|
||||
await run_manager.resume(run_id, {"approved": False, "note": EXPIRED_NOTE}, approval_id=approval_id)
|
||||
# expired:节点据此把这次驳回记成超时驳回(engine/approval.Decision.expired → NodeRejected.decision)
|
||||
await run_manager.resume(run_id, {"approved": False, "note": EXPIRED_NOTE, "expired": True},
|
||||
approval_id=approval_id)
|
||||
except (ValueError, KeyError):
|
||||
logger.warning("待办 %s 超时驳回时运行已无法恢复", task_id)
|
||||
async with SessionLocal() as session:
|
||||
|
||||
@@ -18,6 +18,7 @@ engine/nodes/llm.py(cite_fields 的抽取)、data/recipe_ai.py 各有一份
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Awaitable, Callable, Hashable
|
||||
from contextlib import aclosing
|
||||
from typing import Any
|
||||
|
||||
#: 服务方嫌额度太大时报错里会出现的字样:超过模型的输出上限(DeepSeek chat 系只到 8K),
|
||||
@@ -167,19 +168,24 @@ class BudgetFallback:
|
||||
return await (await self._current()).ainvoke(messages, *args, **kwargs)
|
||||
|
||||
async def astream(self, messages: Any, *args: Any, **kwargs: Any):
|
||||
# 里面那条流用 aclosing 包着:调用方读到一半不读了(助手读到 done、look 就关流),关这一层时里面那条也当场关掉、
|
||||
# 模型那一次调用当场收尾(LangChain 的回调这时把已经收到的用量交出来,providers/usage.py)。不包的话里面那条要等
|
||||
# 垃圾回收才关,收尾落在调用方算完账之后
|
||||
used = await self._current()
|
||||
started = False
|
||||
try:
|
||||
async for chunk in used.astream(messages, *args, **kwargs):
|
||||
started = True
|
||||
yield chunk
|
||||
async with aclosing(used.astream(messages, *args, **kwargs)) as stream:
|
||||
async for chunk in stream:
|
||||
started = True
|
||||
yield chunk
|
||||
return
|
||||
except Exception as e:
|
||||
if started or not self._can_fall_back(used) or not self._switch.rejected(e):
|
||||
raise
|
||||
self._switch.fall_back()
|
||||
async for chunk in (await self._current()).astream(messages, *args, **kwargs):
|
||||
yield chunk
|
||||
async with aclosing((await self._current()).astream(messages, *args, **kwargs)) as stream:
|
||||
async for chunk in stream:
|
||||
yield chunk
|
||||
|
||||
def _derived(self, make: Callable[[Any], Any]) -> BudgetFallback:
|
||||
"""对大小两个模型做同一件事(绑工具、要结构化输出),包成共用开关的一层。小的还没建时,建好再做。"""
|
||||
|
||||
@@ -0,0 +1,204 @@
|
||||
"""模型用量计量:助手建图(含挑表、接着写、自查修正)和复核这两段花了多少(细案 01 做法 3)。
|
||||
|
||||
**为什么要。** 运行按 llm.end 事件记用量,问数据一轮里另外两块——助手建图、复核——一直不记:这一轮的花费是低报的,
|
||||
评测也算不出按角色的成本。在每个调用点手写累计(生成、挑表、退回小额度重来、复核各一处)迟早漏掉一处,所以用 LangChain
|
||||
的回调在模型这一层统一截:和 langchain_core.callbacks.get_usage_metadata_callback 同一个机制(一个登记过的上下文变量,
|
||||
模型每次调用时把它挂成回调),按模型累计 input / output tokens 和调用次数。
|
||||
|
||||
**为什么不直接用 get_usage_metadata_callback。** 它是一个 with 块,块里设上下文变量、出块时按 token 复原。生成过程是一个
|
||||
async generator:with 块跨着 yield 的话,变量在两次 yield 之间留在调用方的上下文里——调用方这时去发起运行,运行的任务
|
||||
会继承它,运行里的调用就被重复计一遍;调用方中途不迭代了、在别的任务里收尾,复原还会报「token 不属于这个上下文」。
|
||||
这里只在每一步 __anext__ 期间设上变量、交出操作之前复原(metered):生成过程里起的任务(心跳那一层每一步一个任务)
|
||||
在创建时拷走上下文,照样计到;调用方自己的代码一行都不在计量里。
|
||||
|
||||
**金额。** 按 providers/catalog.estimate_cost 算,和运行的 llm.end 一个口径。估不出的调用记 unpriced_calls:模型不在价格
|
||||
目录里,或者用量没报全而目录价又不是 0。估不出的不当成 0 元,调用方据此写「≥」或「金额无法估算」。用量没报全有两种:
|
||||
回复没带用量(有的网关流式不给);读到一半就不读了——生成过程读到 done、look、ask 那一行就关掉模型的流(后面的闲话不要、
|
||||
也不再为它付钱),用量在流的末尾才报全(Anthropic 的输出 token 在最后一条,OpenAI 兼容系开了 stream_usage 也在最后),
|
||||
关掉的那一次只拿得到已经报了的部分。这一次照样计次数、计已报的 token,记成估不出,不悄悄少算。
|
||||
|
||||
模型名按发起调用时的参数取(invocation_params 的 model / model_name,就是接入里配的模型 id,价格目录按它查),取不到再
|
||||
看回复里的 model_name。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
from collections.abc import AsyncIterator, Iterator
|
||||
from contextlib import contextmanager
|
||||
from contextvars import ContextVar
|
||||
from typing import Any
|
||||
from uuid import UUID
|
||||
|
||||
from langchain_core.callbacks import BaseCallbackHandler
|
||||
from langchain_core.tracers.context import register_configure_hook
|
||||
|
||||
#: 取不到模型名的调用记在这个名下
|
||||
UNKNOWN_MODEL = "unknown"
|
||||
|
||||
|
||||
class Meter(BaseCallbackHandler):
|
||||
"""一次计量:按模型累计用量。线程安全(同步模型的回调可能在线程里来)。"""
|
||||
|
||||
#: 同步模型在线程里调用时也要回调
|
||||
run_inline = True
|
||||
|
||||
def __init__(self) -> None:
|
||||
super().__init__()
|
||||
self._lock = threading.Lock()
|
||||
self._pending: dict[UUID, str] = {}
|
||||
self._models: dict[str, dict[str, int]] = {}
|
||||
|
||||
# ------------------------------------------------------------------ 回调
|
||||
|
||||
def _started(self, run_id: UUID, serialized: dict[str, Any] | None, kwargs: dict[str, Any]) -> None:
|
||||
params = kwargs.get("invocation_params") or {}
|
||||
model = params.get("model") or params.get("model_name") or params.get("model_id")
|
||||
if not model and isinstance(serialized, dict):
|
||||
model = (serialized.get("kwargs") or {}).get("model") or (serialized.get("kwargs") or {}).get("model_name")
|
||||
with self._lock:
|
||||
self._pending[run_id] = str(model) if model else ""
|
||||
|
||||
def on_chat_model_start(self, serialized: dict[str, Any], messages: Any, *, run_id: UUID,
|
||||
**kwargs: Any) -> None:
|
||||
self._started(run_id, serialized, kwargs)
|
||||
|
||||
def on_llm_start(self, serialized: dict[str, Any], prompts: Any, *, run_id: UUID, **kwargs: Any) -> None:
|
||||
self._started(run_id, serialized, kwargs)
|
||||
|
||||
def on_llm_end(self, response: Any, *, run_id: UUID, **kwargs: Any) -> None:
|
||||
message = None
|
||||
try:
|
||||
message = response.generations[0][0].message
|
||||
except (AttributeError, IndexError, TypeError):
|
||||
pass
|
||||
usage = getattr(message, "usage_metadata", None) or None
|
||||
meta = getattr(message, "response_metadata", None) or {}
|
||||
with self._lock:
|
||||
model = self._pending.pop(run_id, "") or str(meta.get("model_name") or meta.get("model") or "") \
|
||||
or UNKNOWN_MODEL
|
||||
row = self._models.setdefault(model, {"input_tokens": 0, "output_tokens": 0, "calls": 0, "unreported": 0})
|
||||
row["calls"] += 1
|
||||
if usage:
|
||||
row["input_tokens"] += int(usage.get("input_tokens") or 0)
|
||||
row["output_tokens"] += int(usage.get("output_tokens") or 0)
|
||||
else:
|
||||
row["unreported"] += 1
|
||||
|
||||
def on_llm_error(self, error: BaseException, *, run_id: UUID, **kwargs: Any) -> None:
|
||||
"""流读到一半停了(生成过程读到 done / look / ask 就关流,或者连接断了):LangChain 照样交来已经收到的那几段。
|
||||
有收到东西的照计一次、记已报的 token,算用量没报全;一段都没收到的(请求刚发出去就被拒了)不计。"""
|
||||
from langchain_core.outputs import ChatGenerationChunk
|
||||
|
||||
response = kwargs.get("response")
|
||||
partial = None
|
||||
try:
|
||||
first = response.generations[0][0] if response is not None and response.generations[0] else None
|
||||
partial = first.message if isinstance(first, ChatGenerationChunk) else None
|
||||
except (AttributeError, IndexError, TypeError):
|
||||
partial = None
|
||||
with self._lock:
|
||||
model = self._pending.pop(run_id, "")
|
||||
if partial is None:
|
||||
return
|
||||
meta = getattr(partial, "response_metadata", None) or {}
|
||||
model = model or str(meta.get("model_name") or meta.get("model") or "") or UNKNOWN_MODEL
|
||||
row = self._models.setdefault(model, {"input_tokens": 0, "output_tokens": 0, "calls": 0, "unreported": 0})
|
||||
row["calls"] += 1
|
||||
row["unreported"] += 1
|
||||
reported = getattr(partial, "usage_metadata", None) or {}
|
||||
row["input_tokens"] += int(reported.get("input_tokens") or 0)
|
||||
row["output_tokens"] += int(reported.get("output_tokens") or 0)
|
||||
|
||||
# ------------------------------------------------------------------ 合并、汇总
|
||||
|
||||
def merge(self, other: Meter) -> None:
|
||||
"""把另一次计量并进来(嵌套的计量出块时并进外层)。"""
|
||||
with other._lock:
|
||||
rows = {k: dict(v) for k, v in other._models.items()}
|
||||
with self._lock:
|
||||
for model, row in rows.items():
|
||||
mine = self._models.setdefault(model, dict.fromkeys(row, 0))
|
||||
for key, value in row.items():
|
||||
mine[key] = mine.get(key, 0) + value
|
||||
|
||||
def summary(self) -> dict[str, Any]:
|
||||
"""{by_model: {模型: {input_tokens, output_tokens, calls, cost_usd, unpriced_calls}}, input_tokens, output_tokens,
|
||||
calls, cost_usd, unpriced_calls}。cost_usd 只是估得出的那部分。"""
|
||||
from app.providers.catalog import estimate_cost, find_pricing
|
||||
|
||||
with self._lock:
|
||||
rows = {k: dict(v) for k, v in self._models.items()}
|
||||
by_model: dict[str, dict[str, Any]] = {}
|
||||
for model, row in sorted(rows.items()):
|
||||
pricing = find_pricing(model)
|
||||
free = pricing is not None and not any(pricing.values())
|
||||
if pricing is None:
|
||||
unpriced = row["calls"]
|
||||
else:
|
||||
unpriced = 0 if free else row["unreported"]
|
||||
by_model[model] = {"input_tokens": row["input_tokens"], "output_tokens": row["output_tokens"],
|
||||
"calls": row["calls"],
|
||||
"cost_usd": round(estimate_cost(model, row["input_tokens"], row["output_tokens"]), 6),
|
||||
"unpriced_calls": unpriced}
|
||||
total = {key: sum(r[key] for r in by_model.values())
|
||||
for key in ("input_tokens", "output_tokens", "calls", "unpriced_calls")}
|
||||
return {"by_model": by_model, **total, "cost_usd": round(sum(r["cost_usd"] for r in by_model.values()), 6)}
|
||||
|
||||
|
||||
#: 计量挂在这个上下文变量上(inheritable:模型调用时 LangChain 按它把 Meter 挂成回调,子任务拷走上下文时一并带走)
|
||||
_CURRENT: ContextVar[Meter | None] = ContextVar("agentlab_usage_meter", default=None)
|
||||
register_configure_hook(_CURRENT, inheritable=True)
|
||||
|
||||
|
||||
def meter() -> Meter:
|
||||
"""新的一次计量。用 metering(一段代码)或 metered(一个生成过程)把它挂上。"""
|
||||
return Meter()
|
||||
|
||||
|
||||
def current() -> Meter | None:
|
||||
return _CURRENT.get()
|
||||
|
||||
|
||||
@contextmanager
|
||||
def metering(m: Meter) -> Iterator[Meter]:
|
||||
"""这一段代码里的模型调用计进 m。只给不跨 yield 的一段用(复核那一次调用);生成过程用 metered。
|
||||
|
||||
嵌套的:出块时并进外层的那一次,外层照样看得到。复原用 set 而不是按 token reset:在别的任务里收尾也不报错。"""
|
||||
outer = _CURRENT.get()
|
||||
_CURRENT.set(m)
|
||||
try:
|
||||
yield m
|
||||
finally:
|
||||
_CURRENT.set(outer)
|
||||
if outer is not None and outer is not m:
|
||||
outer.merge(m)
|
||||
|
||||
|
||||
async def metered(source: AsyncIterator[Any], m: Meter) -> AsyncIterator[Any]:
|
||||
"""迭代一个生成过程,它的模型调用计进 m:每一步 __anext__ 期间挂上,交出操作之前复原(见模块开头)。
|
||||
|
||||
开始时外层已经有一次计量的(评测把一整题的助手用量记成一份):结束时把 m 并进去,同 metering。"""
|
||||
enclosing = _CURRENT.get()
|
||||
iterator = source.__aiter__()
|
||||
try:
|
||||
while True:
|
||||
outer = _CURRENT.get()
|
||||
_CURRENT.set(m)
|
||||
try:
|
||||
item = await iterator.__anext__()
|
||||
except StopAsyncIteration:
|
||||
return
|
||||
finally:
|
||||
_CURRENT.set(outer)
|
||||
yield item
|
||||
finally:
|
||||
if enclosing is not None and enclosing is not m:
|
||||
enclosing.merge(m)
|
||||
|
||||
|
||||
def empty() -> dict[str, Any]:
|
||||
"""一次调用都没有时的汇总(同 Meter().summary())。"""
|
||||
return Meter().summary()
|
||||
|
||||
|
||||
__all__ = ["Meter", "UNKNOWN_MODEL", "current", "empty", "meter", "metered", "metering"]
|
||||
@@ -320,7 +320,8 @@ def build_report(
|
||||
) -> dict[str, Any]:
|
||||
"""页面里的数据(report-data)。
|
||||
|
||||
- run:运行记录的几样(id、workflow_id、workflow_name、run_class、status、started_at、finished_at)
|
||||
- run:运行记录的几样(id、workflow_id、workflow_name、run_class、status、end_reason、started_at、finished_at);
|
||||
end_reason 分开同是 cancelled 的被驳回和手动停止(05-已知遗留 (a))
|
||||
- report_node / doc_artifact / doc_hash_ok:选中的报告节点、它的文档工件、取回时哈希核对的结果
|
||||
- doc:报告文档(正文的服务端渲染另走 render_body,这里不给:数据块只装点开数字要用的东西)
|
||||
- seal:{_sealed 的 seal 字段里的几样}(sealed、ok、manifest_hash、manifest_seq)
|
||||
@@ -334,6 +335,7 @@ def build_report(
|
||||
"meta": {
|
||||
"run_id": run.get("id"), "workflow_id": run.get("workflow_id"),
|
||||
"workflow_name": display_name(run), "run_class": run.get("run_class"), "status": run.get("status"),
|
||||
"end_reason": run.get("end_reason"),
|
||||
"started_at": _iso(run.get("started_at")), "finished_at": _iso(run.get("finished_at")),
|
||||
"report": {"node_id": report_node, "label": (nodes.get(report_node) or {}).get("label") or report_node},
|
||||
"doc_artifact": doc_artifact, "doc_hash_ok": doc_hash_ok,
|
||||
|
||||
@@ -25,7 +25,7 @@
|
||||
metaRun: '运行 ID', metaClass: '运行类别', metaStatus: '运行状态', metaReport: '报告节点',
|
||||
metaStarted: '开始', metaFinished: '结束', metaExported: '导出时间',
|
||||
classLabel: { formal: '正式运行', exploratory: '探索运行' },
|
||||
statusLabel: { succeeded: '已完成', failed: '已失败', cancelled: '已取消', running: '进行中', waiting: '等待审批' },
|
||||
statusLabel: { succeeded: '已完成', failed: '已失败', cancelled: '已取消', rejected: '被驳回', running: '进行中', waiting: '等待审批' },
|
||||
docHashBad: '这份报告文档与哈希不一致(疑似被修改),正文仅供参考,其中的口径和出处未经核对。',
|
||||
sealNone: '这次运行导出时尚未封存:下列出处来自导出那一刻的系统记录,未经封存核对。',
|
||||
sealOk: (hash) => `运行已封存,清单哈希 ${hash}。要向系统外的人证明文件未被修改,请导出「可验证报告」(记录页 · 凭证详情),拿封存哈希去系统对照。`,
|
||||
@@ -147,7 +147,8 @@
|
||||
$('meta').replaceChildren(...[...[
|
||||
[T.metaRun, h('code', {}, String(meta.run_id || ''))],
|
||||
[T.metaClass, label(T.classLabel, meta.run_class)],
|
||||
[T.metaStatus, label(T.statusLabel, meta.status)],
|
||||
// 被驳回(cancelled + end_reason=rejected)照运行记录写「被驳回」,不写成手动停止的「已取消」
|
||||
[T.metaStatus, label(T.statusLabel, meta.status === 'cancelled' && meta.end_reason === 'rejected' ? 'rejected' : meta.status)],
|
||||
[T.metaReport, String(meta.report?.label || meta.report?.node_id || NONE)],
|
||||
[T.metaStarted, when(meta.started_at)],
|
||||
[T.metaFinished, when(meta.finished_at)],
|
||||
|
||||
@@ -130,7 +130,9 @@ async def process_due(now: datetime | None = None) -> None:
|
||||
for name, step in (("待办的时限处理", tasks.tick), ("写入结果的查询", actions.tick),
|
||||
("没有提交的上传文件清理", files_store.cleanup_tick)):
|
||||
try:
|
||||
await step(now)
|
||||
done = await step(now)
|
||||
if isinstance(done, list) and done:
|
||||
logger.info("%s:这一拍处理了 %d 条", name, len(done))
|
||||
except Exception: # noqa: BLE001
|
||||
logger.exception("%s这一拍出错,下一拍照常进行", name)
|
||||
|
||||
|
||||
@@ -3,6 +3,11 @@
|
||||
调度循环(loop.py)到点调 fire_once,接口的「立即运行」也调它:重叠跳过、渲染输入、记触发、发起运行这一套
|
||||
只有一份。发起本身走 engine/launch.py 的 launch_run,和 HTTP 发起是同一套规则——正式运行只能从当前发布版本
|
||||
发起,这一条不会在定时任务这边被绕过。
|
||||
|
||||
**定时的目标**(docs/design/next-round/00-总方案.md X4:定时只有一套):一个定时任务指向一个目标——工作流、评测题集
|
||||
(第 3 波)或数据源维护(第 3 波)。认领、补跑、重叠跳过、触发记录、通知都只有这一套,fire_once 按目标种类分派。
|
||||
老表的 workflow_id 是非空约束,只加不改的迁移放宽不了:目标不是工作流时它存空串。这条规矩只在 target_of /
|
||||
target_columns 这一处,别处读写目标都经它们,不直接碰这两列。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -64,6 +69,33 @@ INTERRUPTED_REASON = "服务在发起运行前停止,这次触发没有发起
|
||||
GENERIC_REASON = "发起运行时出错,详细信息见服务日志"
|
||||
|
||||
|
||||
#: 定时的目标种类(Schedule.target_kind),顺序即以后界面上的顺序。这一波 fire_once 只会发起 workflow
|
||||
TARGET_KINDS = ("workflow", "eval_suite", "source_maintenance")
|
||||
TARGET_LABEL = {"workflow": "工作流", "eval_suite": "评测题集", "source_maintenance": "数据源维护"}
|
||||
#: 目标种类这个版本还发起不了(库里有别的版本建的任务、手改过库):照实记「未能发起」,不当成工作流去跑
|
||||
TARGET_UNSUPPORTED = "这个定时任务的目标是「{kind}」,当前版本还不能按时发起这类目标,这次触发没有发起"
|
||||
|
||||
|
||||
def target_of(schedule: Any) -> tuple[str, str]:
|
||||
"""一个定时任务的目标:(种类, id)。加列之前建的任务 target_kind 缺省 workflow、target_id 为空,目标就是 workflow_id。
|
||||
|
||||
workflow 时 id 以 workflow_id 为准(两列由 target_columns 一起写,对得上;老任务只有这一列)。"""
|
||||
kind = getattr(schedule, "target_kind", None) or "workflow"
|
||||
if kind == "workflow":
|
||||
return kind, schedule.workflow_id or getattr(schedule, "target_id", None) or ""
|
||||
return kind, getattr(schedule, "target_id", None) or ""
|
||||
|
||||
|
||||
def target_columns(kind: str, target_id: str) -> dict[str, str]:
|
||||
"""写目标的几列:{target_kind, target_id, workflow_id}。workflow 时 target_id 同 workflow_id;别的种类 workflow_id 存空串
|
||||
(老表的非空约束)。种类不认识、id 为空抛 ValueError。"""
|
||||
if kind not in TARGET_KINDS:
|
||||
raise ValueError(f"不认识的定时目标种类:{kind}")
|
||||
if not target_id:
|
||||
raise ValueError("定时任务的目标 id 不能为空")
|
||||
return {"target_kind": kind, "target_id": target_id, "workflow_id": target_id if kind == "workflow" else ""}
|
||||
|
||||
|
||||
class TemplateError(ValueError):
|
||||
"""输入模板写错了。str() 是给人看的中文说明,指出是哪个输入。"""
|
||||
|
||||
@@ -282,46 +314,61 @@ async def fire_once(schedule_id: str, fire_at: datetime, kind: str, *, actor: st
|
||||
"""处理一次触发,返回记下的触发记录。同一 (任务, fire_at, kind) 已经有记录(两拍撞在一起、立即运行连点两下)
|
||||
时不再发起,返回已有的那一条;任务不存在返回 None。
|
||||
|
||||
上一次运行还没结束就记 skipped_overlap;输入渲染不了、launch_run 拒绝或出错记 failed_to_start(拒绝写原话,
|
||||
其他异常写通用说明、详细信息进日志);发起成功记 started 和运行 id。
|
||||
按目标种类分派(target_of):这一波只有工作流(_fire_workflow),别的种类记 failed_to_start、写明原因,不发起。
|
||||
"""
|
||||
async with launch_lock():
|
||||
async with SessionLocal() as session:
|
||||
schedule = await session.get(Schedule, schedule_id)
|
||||
if schedule is None:
|
||||
return None
|
||||
name, workflow_id, run_class = schedule.name, schedule.workflow_id, schedule.run_class
|
||||
target_kind, target_id = target_of(schedule)
|
||||
name, run_class = schedule.name, schedule.run_class
|
||||
tz, cron_expr, template = schedule.timezone, schedule.cron, schedule.input_template
|
||||
try:
|
||||
inputs = render_inputs(check_template(template), template_values(fire_at, tz, cron_expr))
|
||||
except (TemplateError, cron.CronError, cron.TimezoneError) as e:
|
||||
# 存进库之后才坏掉的(手改过库、时区库升级删了时区):照实说是哪一项,不发起
|
||||
return await _record(schedule_id, fire_at, kind, "failed_to_start", f"无法生成这次运行的输入:{e}", {})
|
||||
busy = await _active_run(schedule_id)
|
||||
if busy is not None:
|
||||
return await _record(schedule_id, fire_at, kind, "skipped_overlap", overlap_reason(*busy), inputs)
|
||||
fire = await _record(schedule_id, fire_at, kind, "failed_to_start", INTERRUPTED_REASON, inputs, fresh=True)
|
||||
if fire is None:
|
||||
return await _existing(schedule_id, fire_at, kind)
|
||||
from app.engine.launch import LaunchRefused, launch_run
|
||||
if target_kind != "workflow":
|
||||
reason = TARGET_UNSUPPORTED.format(kind=TARGET_LABEL.get(target_kind, target_kind))
|
||||
return await _record(schedule_id, fire_at, kind, "failed_to_start", reason, {})
|
||||
return await _fire_workflow(schedule_id, fire_at, kind, actor=actor, name=name, workflow_id=target_id,
|
||||
run_class=run_class, tz=tz, cron_expr=cron_expr, template=template)
|
||||
|
||||
_LAUNCHING.add(fire.id)
|
||||
|
||||
async def _fire_workflow(schedule_id: str, fire_at: datetime, kind: str, *, actor: str | None, name: str,
|
||||
workflow_id: str, run_class: str, tz: str, cron_expr: str, template: Any,
|
||||
) -> ScheduleFire | None:
|
||||
"""目标是工作流的一次触发(持着 launch_lock)。
|
||||
|
||||
上一次运行还没结束就记 skipped_overlap;输入渲染不了、launch_run 拒绝或出错记 failed_to_start(拒绝写原话,
|
||||
其他异常写通用说明、详细信息进日志);发起成功记 started 和运行 id。
|
||||
"""
|
||||
try:
|
||||
inputs = render_inputs(check_template(template), template_values(fire_at, tz, cron_expr))
|
||||
except (TemplateError, cron.CronError, cron.TimezoneError) as e:
|
||||
# 存进库之后才坏掉的(手改过库、时区库升级删了时区):照实说是哪一项,不发起
|
||||
return await _record(schedule_id, fire_at, kind, "failed_to_start", f"无法生成这次运行的输入:{e}", {})
|
||||
busy = await _active_run(schedule_id)
|
||||
if busy is not None:
|
||||
return await _record(schedule_id, fire_at, kind, "skipped_overlap", overlap_reason(*busy), inputs)
|
||||
fire = await _record(schedule_id, fire_at, kind, "failed_to_start", INTERRUPTED_REASON, inputs, fresh=True)
|
||||
if fire is None:
|
||||
return await _existing(schedule_id, fire_at, kind)
|
||||
from app.engine.launch import LaunchRefused, launch_run
|
||||
|
||||
_LAUNCHING.add(fire.id)
|
||||
try:
|
||||
try:
|
||||
try:
|
||||
async with SessionLocal() as session:
|
||||
run = await launch_run(session, workflow_id=workflow_id, run_class=run_class,
|
||||
input_payload=inputs, actor=actor or actor_label(name),
|
||||
schedule_id=schedule_id, origin_kind="schedule",
|
||||
origin={"schedule_id": schedule_id, "schedule": name,
|
||||
"fire_id": fire.id, "fire_kind": kind})
|
||||
except LaunchRefused as e:
|
||||
return await _finish(fire.id, status="failed_to_start", reason=e.message)
|
||||
except Exception: # noqa: BLE001 - 发起失败不能让调度循环停下;详细信息进日志,记录上写通用说明
|
||||
logger.exception("定时任务「%s」(%s)发起运行失败", name, schedule_id)
|
||||
return await _finish(fire.id, status="failed_to_start", reason=GENERIC_REASON)
|
||||
return await _finish(fire.id, status="started", reason=None, run_id=run.id)
|
||||
finally:
|
||||
_LAUNCHING.discard(fire.id)
|
||||
async with SessionLocal() as session:
|
||||
run = await launch_run(session, workflow_id=workflow_id, run_class=run_class,
|
||||
input_payload=inputs, actor=actor or actor_label(name),
|
||||
schedule_id=schedule_id, origin_kind="schedule",
|
||||
origin={"schedule_id": schedule_id, "schedule": name,
|
||||
"fire_id": fire.id, "fire_kind": kind})
|
||||
except LaunchRefused as e:
|
||||
return await _finish(fire.id, status="failed_to_start", reason=e.message)
|
||||
except Exception: # noqa: BLE001 - 发起失败不能让调度循环停下;详细信息进日志,记录上写通用说明
|
||||
logger.exception("定时任务「%s」(%s)发起运行失败", name, schedule_id)
|
||||
return await _finish(fire.id, status="failed_to_start", reason=GENERIC_REASON)
|
||||
return await _finish(fire.id, status="started", reason=None, run_id=run.id)
|
||||
finally:
|
||||
_LAUNCHING.discard(fire.id)
|
||||
|
||||
|
||||
async def record_skip(schedule_id: str, fire_at: datetime, reason: str) -> ScheduleFire | None:
|
||||
|
||||
@@ -5,7 +5,8 @@
|
||||
再导出,调用方和测试不用改;_transport、RETRY_DELAYS 仍是这个模块自己的(测试照旧替换它们),send 每次调用时读。
|
||||
|
||||
通知里只有任务名、状态、出具档位、缺口条数、运行链接、封存哈希(界面上叫清单哈希)前 12 位和起止时间,失败时
|
||||
另写失败节点的名字和下一步(不许诺能继续运行);运行成功结束后和同一任务上一次成功的运行的对比(auto_compare)只写
|
||||
另写失败节点的名字和下一步(不许诺能继续运行);被人驳回的那一期按「运行结束」推「未出具(被驳回)」、写驳回的那一步,
|
||||
手动停止的不推;运行成功结束后和同一任务上一次成功的运行的对比(auto_compare)只写
|
||||
各类指标的个数和对比页链接。不放报告里的具体数字,也不放失败时的原始报错——报错里可能带着查询到的数据,Webhook
|
||||
推去的群不一定有权看。
|
||||
"""
|
||||
@@ -71,6 +72,11 @@ NEXT_AS_USUAL = "下一次触发会照常发起"
|
||||
#: 从没开始执行、任务又不在排期上(停用了):原因刚写过,记录页上也没有更多可看的,下一步是重新发起。「立即运行」
|
||||
#: 是定时任务页上那个按钮的名字,停用的任务也能点
|
||||
RERUN_ACTION = "如需重新发起,请在 AgentLab 的「定时任务」中点击「立即运行」"
|
||||
#: 被人驳回的那一期(cancelled + end_reason=rejected,next-round X9):是人的决定,不是故障。按「运行结束」推(订阅
|
||||
#: 「运行失败」的人盯的是故障),标题、状态照实写被驳回;驳回的备注不写(可能带着数),驳回的是哪一步写节点名
|
||||
REJECTED_HEADLINE = "未出具(被驳回)"
|
||||
REJECTED_STATUS = "被驳回"
|
||||
REJECTED_ACTION = "驳回的说明请在 AgentLab 的「记录」中查看"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
@@ -110,12 +116,18 @@ class Notice:
|
||||
#: 挂起的原因是写外部系统的结果未知(等核实,核实后自动接着跑),不是服务重启。只在 suspended 时有意义
|
||||
action_pending: bool = False
|
||||
error_node: str | None = None
|
||||
#: 运行结束在 cancelled 的原因(Run.end_reason):rejected 时按「未出具(被驳回)」说,error_node 是驳回的那一步
|
||||
end_reason: str | None = None
|
||||
next_fire: bool = False
|
||||
compare: dict[str, Any] | None = None
|
||||
compare_url: str | None = None
|
||||
fmt: str | None = None
|
||||
signed: bool = False
|
||||
|
||||
@property
|
||||
def rejected(self) -> bool:
|
||||
return self.status == "cancelled" and self.end_reason == "rejected"
|
||||
|
||||
@property
|
||||
def never_started(self) -> bool:
|
||||
"""失败的运行从没开始执行(没有开始时间):服务重启时还在排队、被改成失败的就是这种,没有断点可续。"""
|
||||
@@ -142,6 +154,8 @@ def message(notice: Notice) -> tuple[str, list[tuple[str, str]], str | None]:
|
||||
headline = "未能发起运行"
|
||||
elif notice.suspended:
|
||||
headline = "运行已挂起"
|
||||
elif notice.rejected:
|
||||
headline = REJECTED_HEADLINE
|
||||
else:
|
||||
headline = _HEADLINE.get(notice.event, notice.event)
|
||||
title = f"定时任务「{notice.schedule_name}」{headline}"
|
||||
@@ -163,6 +177,10 @@ def message(notice: Notice) -> tuple[str, list[tuple[str, str]], str | None]:
|
||||
lines += [("原因", ACTION_PENDING_REASON), ("处理", ACTION_PENDING_ACTION)]
|
||||
elif notice.suspended:
|
||||
lines += [("原因", SUSPENDED_REASON), ("处理", SUSPENDED_ACTION)]
|
||||
elif notice.rejected:
|
||||
if notice.error_node:
|
||||
lines.append(("驳回节点", f"「{notice.error_node}」"))
|
||||
lines.append(("处理", f"{REJECTED_ACTION};{NEXT_AS_USUAL}" if notice.next_fire else REJECTED_ACTION))
|
||||
elif notice.status == "failed":
|
||||
# 不写原始报错(可能带着查询到的数据),写失败节点的名字和下一步
|
||||
if notice.error_node:
|
||||
@@ -219,6 +237,8 @@ def _status_label(notice: Notice) -> str:
|
||||
if notice.pending_approvals:
|
||||
return "等待审批"
|
||||
return "已挂起,等待核实" if notice.action_pending else "已挂起,可继续运行"
|
||||
if notice.rejected:
|
||||
return REJECTED_STATUS
|
||||
return _STATUS_LABEL.get(notice.status or "", notice.status or "")
|
||||
|
||||
|
||||
@@ -239,7 +259,9 @@ def _links(notice: Notice) -> list[tuple[str, str]]:
|
||||
|
||||
def _message(notice: Notice) -> core.Message:
|
||||
title, lines, _ = message(notice)
|
||||
return core.Message(title=title, lines=lines, links=_links(notice), ok=notice.event in ("finished", "test"))
|
||||
# 被驳回的那一期按 finished 推,但没出具:不画成成功的颜色
|
||||
return core.Message(title=title, lines=lines, links=_links(notice),
|
||||
ok=notice.event in ("finished", "test") and not notice.rejected)
|
||||
|
||||
|
||||
def _text(notice: Notice) -> str:
|
||||
@@ -332,10 +354,17 @@ async def _record(schedule_id: str, fire_id: str | None, run_id: str | None, eve
|
||||
|
||||
#: 运行到了这些状态才可能要推送
|
||||
_WATCHED = ("succeeded", "failed", "cancelled", "interrupted")
|
||||
#: 结束的运行 → 事件。cancelled 是有人手动停的,不推。interrupted 分停在审批和挂起两种,见 _state_of
|
||||
#: 结束的运行 → 事件。cancelled 是有人手动停的,不推;被驳回的(end_reason=rejected)按 finished 推,见 _event_of。
|
||||
#: interrupted 分停在审批和挂起两种,见 _state_of
|
||||
_EVENT_OF = {"succeeded": "finished", "failed": "failed", "cancelled": None}
|
||||
|
||||
|
||||
def _event_of(run: Run) -> str | None:
|
||||
if run.status == "cancelled" and run.end_reason == "rejected":
|
||||
return "finished"
|
||||
return _EVENT_OF[run.status]
|
||||
|
||||
|
||||
@dataclass
|
||||
class _Job:
|
||||
fmt: str
|
||||
@@ -395,9 +424,10 @@ async def notify_due(now: datetime | None = None) -> int:
|
||||
.order_by(ScheduleFire.created_at)
|
||||
)).all()
|
||||
if rows:
|
||||
from app.api.settings import notify_settings
|
||||
# 链接里的地址:全站同一个对外访问地址(环境变量优先,其次设置,core/public_url.py)
|
||||
from app.core import public_url
|
||||
|
||||
base_url = (await notify_settings(session))["public_base_url"]
|
||||
base_url = await public_url.refresh(session)
|
||||
for fire, run in rows:
|
||||
seen = await _state_of(session, fire, run, now)
|
||||
if seen.state is None or seen.state == fire.notified_status:
|
||||
@@ -469,7 +499,7 @@ async def _state_of(session: Any, fire: ScheduleFire, run: Run | None, now: date
|
||||
since = run.finished_at or run.updated_at or fire.created_at
|
||||
if not sealed and (now - service.as_utc(since)).total_seconds() < SEAL_WAIT_S:
|
||||
return _Seen(None)
|
||||
return _Seen(run.status, _EVENT_OF[run.status], sealed=sealed)
|
||||
return _Seen(run.status, _event_of(run), sealed=sealed)
|
||||
if run.status != "interrupted":
|
||||
return _Seen(None)
|
||||
counts = dict((await session.execute(
|
||||
@@ -516,7 +546,9 @@ async def _notice(session: Any, schedule: Schedule, fire: ScheduleFire, run: Run
|
||||
pending_approvals=pending or None,
|
||||
suspended=run is not None and run.status == "interrupted" and not pending,
|
||||
action_pending=seen.action_pending,
|
||||
error_node=_failed_node(run) if run is not None and run.status == "failed" else None,
|
||||
error_node=_failed_node(run) if run is not None and (run.status == "failed" or run.end_reason == "rejected")
|
||||
else None,
|
||||
end_reason=run.end_reason if run is not None else None,
|
||||
next_fire=bool(schedule.enabled and schedule.next_run_at is not None),
|
||||
)
|
||||
|
||||
|
||||
@@ -277,7 +277,7 @@ async def build_datasource_tools(
|
||||
except table_versions.SnapshotMissing as e:
|
||||
tools.extend(_unavailable_tool(row, prefix, str(e)) for prefix in prefixes)
|
||||
continue
|
||||
entries = await _catalog_of(row)
|
||||
entries = await _catalog_of(row, ctx.catalog_view)
|
||||
for prefix in prefixes:
|
||||
if prefix == QUERY_PREFIX:
|
||||
tools.append(_make_query_tool(source, ctx, fixed=fixed, catalog=entries, force_readonly=force_readonly,
|
||||
@@ -287,20 +287,17 @@ async def build_datasource_tools(
|
||||
return tools
|
||||
|
||||
|
||||
async def _catalog_of(row: DataSource) -> dict[str, data_catalog.CatalogEntry]:
|
||||
async def _catalog_of(row: Any, view: Any = None) -> dict[str, data_catalog.CatalogEntry]:
|
||||
"""这个源的数据目录,建工具时读一次:db_schema 给模型看的和查询时冻结进表结构快照的是同一版。
|
||||
|
||||
读不到就当没有目录,工具照常可用——目录是附加说明,不能挡住查询。用单独的会话读:出错要回滚,
|
||||
而回滚会让调用方会话里已经加载的对象(row)全部过期,之后再读属性就是异步环境里的懒加载。
|
||||
读哪一份由目录视图定(view,data/catalog_view.py:当前目录、目录快照、关;只用已核实的项),不给就是当前目录。
|
||||
读不到就当没有目录,工具照常可用——目录是附加说明,不能挡住查询;视图指定的目录快照不在了照实报错(不悄悄换成当前
|
||||
目录)。用单独的会话读:出错要回滚,而回滚会让调用方会话里已经加载的对象(row)全部过期,之后再读属性就是异步环境里
|
||||
的懒加载。
|
||||
"""
|
||||
from app.db.base import SessionLocal
|
||||
from app.data import catalog_view
|
||||
|
||||
try:
|
||||
async with SessionLocal() as own:
|
||||
return await data_catalog.read_catalog(own, row.id)
|
||||
except Exception: # noqa: BLE001
|
||||
logger.exception("读取数据源 %s 的数据目录失败,本次不附目录", row.name)
|
||||
return {}
|
||||
return await catalog_view.read(catalog_view.CatalogView.parse(view), row)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
@@ -753,6 +750,12 @@ async def _store_schema(source: Any, ctx: ToolContext, *,
|
||||
if frozen := data_catalog.frozen_catalog(catalog or {}, tables):
|
||||
content[CATALOG_SNAPSHOT_KEY] = frozen
|
||||
meta["catalog_tables"] = len(frozen)
|
||||
# 这次运行读的不是缺省的当前目录(评测的对照:关、目录快照、只用已核实的项):记在引用的 meta 里,证据面板写得出
|
||||
# 「本次运行目录:关」。不进内容哈希,缺省视图不记,快照和以前一字不差
|
||||
from app.data.catalog_view import CatalogView, describe
|
||||
|
||||
if (view := describe(CatalogView.parse(getattr(ctx, "catalog_view", None)))) is not None:
|
||||
meta["catalog_view"] = view
|
||||
try:
|
||||
return await put_json(content, kind=SCHEMA_SNAPSHOT, run_id=ctx.run_id or "", node_id=ctx.node_id or "",
|
||||
meta=meta)
|
||||
|
||||
@@ -49,6 +49,9 @@ class ToolContext:
|
||||
#: 个人应用的刷新、个人蓝图的试运行)。记忆、知识库的内容不经遮罩:kb_search 的 collection 参数(调用工具节点写的、
|
||||
#: 参数名纠错改出来的、图级 defaults 补上的、模型挑的)一律拒,只能检索 collection 这一个
|
||||
fixed_scope: bool = False
|
||||
#: 读哪一份数据目录(data/catalog_view.CatalogView):运行里由节点从 Run.options 取来(catalog_view.of_run),db_schema
|
||||
#: 给模型看的、查询冻结进表结构快照的、查询时的 SQL 检查都按它。None 是当前目录(缺省,行为同以前)
|
||||
catalog_view: Any = None
|
||||
extra: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user