W3-a 变化记录的接口与处理、定时维护(next-round 03 B3、B4)

变化没有统一的记录:进不了审阅,发不了通知,口径账本的召回也没有可以订阅的接口;目录只能手动探查、手动剖析。现在:

- api/source_changes.py(骨架已在 datasources 通配之前登记,搭建者可用,访问密钥不放):GET /api/datasources/{id}/changes
  (缺省开着的和知道了的,按「使用次数权重 × 严重程度」排,带各级别计数)、GET /changes/{change_id}、POST
  /changes/{change_id}/resolve。处理动作落到目录已有的写入接口:清理目录里说到被删的列的项(整份写、乐观锁)、加入码值
  (apply_patch 配 _merge_codes,含义待填写,依据写变化编号)、重新起草这张表(模型用不了时只按表结构起草)、重新剖析、
  知道了、误报(必须写理由);每个动作写 resolution,动了目录的记目录版本,建了任务的记任务编号。
- data/source_changes.py 给召回:list_changes(按 seq 游标拉)、affecting(按源和表反查)、resolve、triggers_recall(只认
  结构类、处理时改了目录的、触及有效记录条件的新码值)。
- 通知 source.changed 的发送函数 notify/service.notify_source_changed:只推还要人看的「要处理」级变化,按源汇总,消息里
  只有源名、各级别条数和链接(D9);探查、剖析任务结束时有新的要处理变化就调。这一行的订阅范围第 4 波有了界面才打开,
  现在推不出去(照实)。
- 定时维护(data/source_maintenance.py,X4:就是一条 target_kind=source_maintenance 的定时,不另建表):fire 按顺序建
  探查任务 → 剖析任务(范围 needed / observe / maintenance 按选的步骤),任务上记 trigger=schedule、fire_at、schedule_id;
  剖析等探查进了同源队才建,靠 SourceLanes 串行;剖析没开、数据源停用 / 不存在 / 是上传表格时照实记没执行,同源还有
  探查或剖析在做时记跳过;触发记录的 inputs 写明建了哪几个任务、哪几步没执行。check_target 核对目标。维护步骤存在
  options.catalog_maintenance,经 GET / PUT /api/datasources/{id}/maintenance 改,数据源表单不显示它(api/datasources.py
  的 _MANAGED_OPTIONS 加了这一个键)。

测试:test_source_changes(列表与计数、误报要理由、处理过的 409、清理目录与加入码值记目录版本、剖析没开照实 422、
去重、游标拉取与反查、召回触发、通知只有源名条数链接)、test_source_maintenance(按顺序串行、trigger 与 fire_at、
剖析没开记未执行、停用 / 不存在 / 忙、调度循环只补最近一次且积压只记一条、步骤接口与核对目标)。
This commit is contained in:
ThearchyHelios
2026-10-16 16:48:02 +08:00
parent c95e5b4e84
commit f5c94cf12d
9 changed files with 874 additions and 13 deletions
+3 -2
View File
@@ -140,8 +140,9 @@ def _import_mode(row: DataSource) -> str | None:
#: options 里由别的接口维护的项:数据源表单不显示、不提交它们,整份回写 options 时原样留着(同模型接入 extra 里的
#: last_check)。数据目录的起草要求只经 PUT /catalog/guidance 改(api/catalog.py):表单把认不得的项放进「高级参数」
#: 的单行输入框,多行的要求会被压成一行
#: 探查范围(introspect_scope)同理:只经 PUT /introspect-scope 改,表单整份回写 options 时原样留着
_MANAGED_OPTIONS = (CATALOG_DRAFT_GUIDANCE_OPTION, source_objects.SCOPE_OPTION)
#: 探查范围(introspect_scope)同理:只经 PUT /introspect-scope 改,表单整份回写 options 时原样留着;定时维护的步骤
#: (catalog_maintenance)只经 PUT /maintenance 改(api/source_changes.py)
_MANAGED_OPTIONS = (CATALOG_DRAFT_GUIDANCE_OPTION, source_objects.SCOPE_OPTION, "catalog_maintenance")
def _shown_options(options: dict[str, Any] | None) -> dict[str, Any]:
+119 -3
View File
@@ -1,9 +1,125 @@
"""数据源的变化记录(docs/design/next-round/03-数据接入与目录维护.md B3):结构变化、取值变化,查看与处理。
"""数据源的变化记录(docs/design/next-round/03-数据接入与目录维护.md B3):结构变化、取值变化,查看与处理;以及同一前缀下
数据源定时维护要做哪几步(B4)。
挂在 /api/datasources/{id}/changes 下,要在 datasources 的通配 /{source_id} 之前登记(main.py)。第 3 波 W3-a 填。
挂在 /api/datasources/{id}/… 下,要在 datasources 的通配 /{source_id} 之前登记(main.py):
- GET /{id}/changes 变化列表(缺省开着的和知道了的;status=all 全部):按「使用次数权重 × 严重程度」排,
同分新的在前;带各级别的计数(数据源卡片的角标「N 个变化要处理」按它,第 4 波)
- GET /{id}/changes/{change_id} 一条变化
- POST /{id}/changes/{change_id}/resolve 处理:{action, note?, codes?}。动作落到目录已有的写入接口并记目录版本
(data/source_changes.handle);误报必须写理由(422);已经处理过的 409
- GET /{id}/maintenance 定时维护要做哪几步、剖析开没开、指向这个源的维护定时
- PUT /{id}/maintenance 改要做哪几步:{steps: [introspect | profile_needed | observe]}。定时本身(cron、时区、开关)
在 /api/schedules 里建(target_kind=source_maintenance),这里只管步骤
访问密钥默认不放(auth/keys.py 的 KEY_ROUTES 没有登记这几条)。界面(审阅页「变化」标签页、数据源角标、定时维护设置)在
第 4 波。
"""
from __future__ import annotations
from fastapi import APIRouter
from typing import Any
from fastapi import APIRouter, Depends, Header, HTTPException, Query
from pydantic import BaseModel, Field
from sqlalchemy.ext.asyncio import AsyncSession
from app.api.deps import actor_of
from app.data import catalog, source_changes as sc, source_maintenance
from app.db.base import get_session
from app.db.models import DataSource
router = APIRouter(prefix="/api/datasources", tags=["source-changes"])
#: 一页最多几条
LIMIT_MAX = 200
async def _source(session: AsyncSession, source_id: str) -> DataSource:
row = await session.get(DataSource, source_id)
if row is None:
raise HTTPException(404, "数据源不存在,可能已被删除")
return row
@router.get("/{source_id}/changes")
async def list_changes(source_id: str, status: str = Query("pending"), kind: list[str] | None = Query(None),
table: str | None = Query(None), limit: int = Query(100, ge=1, le=LIMIT_MAX),
offset: int = Query(0, ge=0), session: AsyncSession = Depends(get_session)) -> dict[str, Any]:
"""{items: [变化(agentlab-change/1)], total, counts: {pending, open, by_severity}}。"""
if status not in ("pending", "all", *sc.STATUSES):
raise HTTPException(422, f"不认识的状态:{status}。可选:pending、all、{'、'.join(sc.STATUSES)}")
if kind and (bad := [k for k in kind if k not in sc.KINDS]):
raise HTTPException(422, f"不认识的变化种类:{'、'.join(bad)}")
row = await _source(session, source_id)
rows, total = await sc.list_for_source(session, row, status=status, kinds=kind, table=table, limit=limit,
offset=offset)
return {"items": [sc.change_out(r) for r in rows], "total": total, "counts": await sc.counts(session, row.id)}
@router.get("/{source_id}/changes/{change_id}")
async def get_change(source_id: str, change_id: str, session: AsyncSession = Depends(get_session)) -> dict[str, Any]:
row = await _source(session, source_id)
found = await sc.load(session, row.id, change_id)
if found is None:
raise HTTPException(404, "这条变化不存在,可能已随数据源删除")
return {"change": sc.change_out(found)}
class ResolveIn(BaseModel):
action: str
#: 理由:误报必须写
note: str | None = Field(default=None, max_length=sc.NOTE_MAX)
#: 加入码值时的 {码值: 含义}(含义可以空);不给就加这条变化测到的新码值
codes: dict[str, str] | None = None
@router.post("/{source_id}/changes/{change_id}/resolve")
async def resolve_change(source_id: str, change_id: str, payload: ResolveIn, x_actor: str | None = Header(default=None),
session: AsyncSession = Depends(get_session)) -> dict[str, Any]:
"""处理一条变化:clean_catalog 清理目录里说到这些列的项、add_codes 加入码值、redraft 重新起草这张表、reprofile 重新
剖析、acknowledge 知道了、dismiss 误报(必须写理由)。返回 {change, catalog_version?, job?}。目录被别人刚改过 409。"""
row = await _source(session, source_id)
found = await sc.load(session, row.id, change_id)
if found is None:
raise HTTPException(404, "这条变化不存在,可能已随数据源删除")
try:
return await sc.handle(session, row, found, payload.action, actor=actor_of(x_actor), note=payload.note,
codes=payload.codes)
except sc.ChangeInvalid as e:
await session.rollback()
raise HTTPException(422, str(e)) from e
except sc.ChangeFinished as e:
await session.rollback()
raise HTTPException(409, str(e)) from e
except catalog.CatalogConflict as e:
await session.rollback()
raise HTTPException(409, "这张表的数据目录刚被别人改过,请刷新后再处理") from e
@router.get("/{source_id}/maintenance")
async def get_maintenance(source_id: str, session: AsyncSession = Depends(get_session)) -> dict[str, Any]:
"""{steps, profile_enabled, schedules}:要做哪几步、剖析开没开(没开时剖析、观察两步到点照实记没执行)、指向这个源的
维护定时。"""
from app.data import catalog_profile
row = await _source(session, source_id)
return {"steps": source_maintenance.steps_of(row.options),
"profile_enabled": catalog_profile.profile_settings(row.options).enabled,
"schedules": await source_maintenance.schedules_for(session, row.id)}
class MaintenanceIn(BaseModel):
steps: list[Any]
@router.put("/{source_id}/maintenance")
async def put_maintenance(source_id: str, payload: MaintenanceIn, x_actor: str | None = Header(default=None),
session: AsyncSession = Depends(get_session)) -> dict[str, Any]:
row = await _source(session, source_id)
if (row.origin or "manual") in ("upload", "records"):
raise HTTPException(409, source_maintenance.NOT_MAINTAINED)
if problem := source_maintenance.steps_problem(payload.steps):
raise HTTPException(422, problem)
row.options = source_maintenance.with_steps(row.options, [str(s) for s in payload.steps], actor=actor_of(x_actor))
await session.commit()
return await get_maintenance(source_id, session)
+11
View File
@@ -904,6 +904,16 @@ class _ProfileRun(runtime.JobRun):
await self._write(_close, self.job_id, job_status, job_error, task_status, task_error)
except Exception: # noqa: BLE001
logger.exception("数据剖析任务 %s 收尾时写库出错", self.job_id)
await self._notify()
async def _notify(self) -> None:
"""这次取值观察新记了「要处理」的变化(撤销了已列全):按源推一条 source.changed。"""
from app.data import source_changes
async with SessionLocal() as session:
job = await session.get(CatalogProfileJob, self.job_id)
if job is not None and job.scope in OBSERVING_SCOPES:
await source_changes.notify_after_job(job.source_id, self.job_id)
def stop_text(self) -> str:
return STOPPED_ERROR if self.stop_reason == "user" else INTERRUPTED_ERROR
@@ -966,6 +976,7 @@ class _ProfileRun(runtime.JobRun):
break
await self._settle()
await self._write(self._finish)
await self._notify()
# ---------------------------------------------------------------- 进度回调(引擎调)
+11
View File
@@ -572,6 +572,16 @@ class _IntrospectRun(runtime.JobRun):
await self._write(_close, self.job_id, job_status, job_error, task_error)
except Exception: # noqa: BLE001
logger.exception("探查任务 %s 收尾时写库出错", self.job_id)
await self._notify()
async def _notify(self) -> None:
"""这次新记了「要处理」的结构变化:按源推一条 source.changed(data/source_changes.notify_after_job)。"""
from app.data import source_changes
async with SessionLocal() as session:
job = await session.get(IntrospectJob, self.job_id)
if job is not None:
await source_changes.notify_after_job(job.source_id, self.job_id)
async def _run(self) -> None:
if not await self._write(self._claim):
@@ -600,6 +610,7 @@ class _IntrospectRun(runtime.JobRun):
await self._write(self._save_batch, source_id, schema, batch, got)
await self._settle()
await self._write(self._finish, source_id)
await self._notify()
# ---------------------------------------------------------------- 库操作
+23 -1
View File
@@ -325,6 +325,27 @@ def triggers_recall(change: Mapping[str, Any]) -> bool:
return kind == CODES_NEW and bool((change.get("affects") or {}).get("rules_touched"))
async def notify_after_job(source_id: str, job_id: str) -> int:
"""一次探查、剖析任务结束时:这次新记了还开着的「要处理」级变化,就在后台按源推一条 source.changed(不等它推完)。返回
这次新记了几条要处理的。出错只记日志:通知是附加的。"""
from app.db.base import SessionLocal
try:
async with SessionLocal() as session:
n = int((await session.execute(select(func.count()).select_from(SourceChange).where(
SourceChange.source_id == source_id, SourceChange.severity == "action",
SourceChange.status == "open",
func.json_extract(SourceChange.detected_by, "$.job_id") == job_id))).scalar() or 0)
if n:
from app.notify import service
service.spawn(service.notify_source_changed(source_id))
return n
except Exception: # noqa: BLE001
logger.exception("任务 %s 结束后推送数据源变化的通知时出错", job_id)
return 0
# ==========================================================================
# 处理
# ==========================================================================
@@ -487,6 +508,7 @@ __all__ = [
"ACTIONS", "ACTION_STATUS", "CODES_NEW", "CODES_SHIFT", "COVERAGE_DROP", "ChangeFinished", "ChangeInvalid",
"ChangeNotFound", "DATE_GAP", "DATE_STALLED", "Detected", "KINDS", "KIND_LABEL", "NewChange", "PENDING", "ROWS_DROP",
"ROWS_JUMP", "SCHEMA", "SEVERITIES", "SEVERITY_RANK", "STATUSES", "VALUE_KINDS", "VOLUME_DROP", "affecting",
"change_out", "counts", "fingerprint", "handle", "list_changes", "list_for_source", "load", "record", "record_all",
"change_out", "counts", "fingerprint", "handle", "list_changes", "list_for_source", "load", "notify_after_job",
"record", "record_all",
"resolve", "summary", "triggers_recall", "without_paths",
]
+228 -7
View File
@@ -1,21 +1,242 @@
"""数据源的定时维护:探查 → 剖析需要的表 → 取值观察(docs/design/next-round/03-数据接入与目录维护.md B4)。
"""数据源的定时维护:探查 → 剖析需要的表 → 取值观察(docs/design/next-round/03-数据接入与目录维护.md B4,目录-D5)。
scheduler/service.py 按目标种类导入本模块(_TARGET_MODULES):fire 到点发起一次,check_target 建任务时核对目标。
第 3 波 W3-a 填;填之前照实记「当前版本还不能按时发起这类目标」,不发起。
**为什么要有它。** 以前只能手动探查、手动剖析,目录只在点按钮那一刻是对的;结构变化、取值漂移要靠定期观察才看得到。
**定时只有一套**(总方案 X4):维护就是一条 Schedule,target_kind=source_maintenance、target_id=数据源 id。认领、错过的只补
最近一次、积压只记一条、重叠跳过、触发记录、通知都是 scheduler 那一套(scheduler/service.py 按目标种类导入本模块:fire 到点
发起一次、check_target 建任务时核对目标),不另建表、不另写 cron。按数据源开、缺省关(没有这条定时就不维护),只在设定的
时段跑(cron 自己定)。
**做哪几步**存在数据源的 options.catalog_maintenance({steps: [...], updated_by, updated_at},由 PUT
/api/datasources/{id}/maintenance 维护,数据源表单不显示、不提交它):introspect 探查结构、profile_needed 剖析需要的表、
observe 取值观察,可多选,没设过是三步都做。
**到点**(fire,持着 scheduler 的 launch_lock):按顺序建探查任务 → 剖析任务(范围 needed / observe / maintenance,按选的
步骤),任务上记 trigger=schedule 和 fire_at,靠 SourceLanes 串行:剖析排在同一次触发的探查后面(等探查进了同源队才建剖析
任务),开始时按那时的表结构和目录挑表(挑出 0 张也照建,任务以 0 张表结束、提醒里写明为什么)。预算、止损照剖析设置。
照实记没执行的:数据源不存在、已停用、是上传表格或记录表时整次不执行(failed_to_start);剖析没开时剖析、观察两步不执行、
探查照做;上一次的维护(或别的探查、剖析)还没做完时这次跳过(skipped_overlap)。每次触发记下建了哪几个任务、哪几步没执行、
为什么(触发记录的 inputs:{steps, jobs, skipped})。
AGENTLAB_SCHEDULER 关着时调度循环不起,到点不会触发(拷来的数据目录不替原库查业务库);「立即运行」是人点的,照常执行。
"""
from __future__ import annotations
import asyncio
import logging
from collections.abc import Mapping
from datetime import datetime
from typing import Any
from sqlalchemy import select, update
from app.db.base import SessionLocal, utcnow
logger = logging.getLogger(__name__)
#: options 里维护设置的键
MAINTENANCE_OPTION = "catalog_maintenance"
STEPS = ("introspect", "profile_needed", "observe")
STEP_LABEL = {"introspect": "探查结构", "profile_needed": "剖析需要的表", "observe": "取值观察"}
DEFAULT_STEPS = STEPS
#: 等同一次触发的探查任务进同源队最多等几秒(进了队,剖析任务才排得到它后面)
LANE_WAIT_S = 5.0
#: 先记下这次触发、还没建完任务时的说法:进程在这中间没了,留下的记录照实说没有执行
INTERRUPTED = "服务在建维护任务前停止,这次维护没有执行"
SOURCE_GONE = "数据源不存在,可能已被删除"
SOURCE_DISABLED = "数据源「{name}」已停用,这次维护没有执行"
NOT_MAINTAINED = "上传的表格、记录表的结构随版本或表单走,不需要定时维护"
NO_STEPS = "没有选要做的维护步骤(探查结构、剖析需要的表、取值观察至少选一步)"
BUSY = "数据源「{name}」上一次的维护(或别的探查、剖析)还没做完:{what},这次触发已跳过"
NOT_RUN = "{step}没有执行:{why}"
def steps_of(options: Mapping[str, Any] | None) -> list[str]:
"""这个源定时维护要做哪几步(按 STEPS 的顺序);没设过是三步都做。"""
raw = (options or {}).get(MAINTENANCE_OPTION)
steps = raw.get("steps") if isinstance(raw, Mapping) else None
if not isinstance(steps, list):
return list(DEFAULT_STEPS)
picked = {s for s in steps if s in STEPS}
return [s for s in STEPS if s in picked]
def steps_problem(raw: Any) -> str | None:
"""PUT /maintenance 交来的步骤合不合规:不合规返回一句中文。"""
if not isinstance(raw, list) or not all(isinstance(s, str) for s in raw):
return "维护步骤要是一组文字:introspect(探查结构)、profile_needed(剖析需要的表)、observe(取值观察)"
unknown = [s for s in raw if s not in STEPS]
if unknown:
return f"不认识的维护步骤:{'、'.join(unknown)}。可选:{'、'.join(f'{k}({v})' for k, v in STEP_LABEL.items())}"
if not raw:
return NO_STEPS
return None
def with_steps(options: Mapping[str, Any] | None, steps: list[str], *, actor: str | None) -> dict[str, Any]:
"""options 换上新的维护步骤(不改别的项)。"""
ordered = [s for s in STEPS if s in set(steps)]
return {**dict(options or {}),
MAINTENANCE_OPTION: {"steps": ordered, "updated_by": actor, "updated_at": utcnow().isoformat()}}
def _origin_problem(row: Any) -> str | None:
return NOT_MAINTAINED if (getattr(row, "origin", None) or "manual") in ("upload", "records") else None
async def check_target(session: Any, target_id: str) -> str | None:
"""建定时任务、改目标时核对:数据源要在、不是上传表格或记录表、选了至少一步。停用了、没开剖析不拦(到点照实记没执行)。"""
from app.db.models import DataSource
row = await session.get(DataSource, target_id)
if row is None:
return SOURCE_GONE
if problem := _origin_problem(row):
return problem
if not steps_of(row.options):
return NO_STEPS
return None
async def _busy(session: Any, source_id: str) -> str | None:
"""同源有排队或进行中的探查、剖析任务:说明是哪一种。"""
from app.data import catalog_profile_jobs, introspect_jobs
if await introspect_jobs.active_job(session, source_id) is not None:
return "正在探查结构"
if await catalog_profile_jobs.active_job(session, source_id) is not None:
return "正在剖析"
return None
async def _entered(source_id: str, job_id: str) -> None:
"""等这个探查任务进了同源队(或者已经不在跑了):剖析任务要排在它后面。最多等 LANE_WAIT_S 秒。"""
from app.core import job_runtime as runtime
from app.data import introspect_jobs
loop = asyncio.get_running_loop()
deadline = loop.time() + LANE_WAIT_S
while loop.time() < deadline:
if runtime.lanes.ahead(source_id, job_id) is not None or not introspect_jobs.manager.running(job_id):
return
await asyncio.sleep(0.01)
logger.warning("定时维护:探查任务 %s 过了 %.0f 秒还没进同源队,剖析任务照建", job_id, LANE_WAIT_S)
async def _settle_fire(fire_id: str, status: str, reason: str | None, inputs: dict[str, Any]) -> Any:
from app.db.models import ScheduleFire
async with SessionLocal() as session:
await session.execute(update(ScheduleFire).where(ScheduleFire.id == fire_id)
.values(status=status, reason=reason, inputs=inputs))
await session.commit()
return await session.get(ScheduleFire, fire_id)
async def fire(*, schedule_id: str, fire_at: datetime, kind: str, actor: str | None, target_id: str, name: str,
tz: str, cron_expr: str) -> Any:
"""到点维护一次(持着 launch_lock 调用):建探查任务、剖析任务并在后台开始,记下触发。同一次触发已经记过就交回那一条。"""
from app.db.models import DataSource, ScheduleFire
from app.scheduler import service
reason = service.TARGET_UNSUPPORTED.format(kind=service.TARGET_LABEL["source_maintenance"])
return await service.record_fire(schedule_id, fire_at, kind, "failed_to_start", reason)
async with SessionLocal() as session:
existing = (await session.execute(select(ScheduleFire).where(
ScheduleFire.schedule_id == schedule_id, ScheduleFire.fire_at == fire_at,
ScheduleFire.kind == kind))).scalar_one_or_none()
if existing is not None:
return existing
row = await session.get(DataSource, target_id)
if row is None:
return await service.record_fire(schedule_id, fire_at, kind, "failed_to_start", SOURCE_GONE)
if problem := _origin_problem(row):
return await service.record_fire(schedule_id, fire_at, kind, "failed_to_start", problem)
if row.enabled is False:
return await service.record_fire(schedule_id, fire_at, kind, "failed_to_start",
SOURCE_DISABLED.format(name=row.name))
steps = steps_of(row.options)
if not steps:
return await service.record_fire(schedule_id, fire_at, kind, "failed_to_start", NO_STEPS)
if what := await _busy(session, row.id):
return await service.record_fire(schedule_id, fire_at, kind, "skipped_overlap",
BUSY.format(name=row.name, what=what), {"steps": steps})
placeholder = await service.record_fire(schedule_id, fire_at, kind, "failed_to_start", INTERRUPTED,
{"steps": steps})
if placeholder is None:
return None
try:
jobs, skipped = await _start(target_id, steps, actor=actor or service.actor_label(name),
fire_at=fire_at.isoformat(), schedule_id=schedule_id)
except Exception: # noqa: BLE001 - 建任务出意外不能让调度循环停下:照实记下,详细信息进日志
logger.exception("定时维护(%s)建任务时出错", schedule_id)
return await _settle_fire(placeholder.id, "failed_to_start", "建维护任务时出错,详细信息见服务日志",
{"steps": steps})
reason = ";".join(NOT_RUN.format(step=STEP_LABEL[s["step"]], why=s["reason"]) for s in skipped) or None
return await _settle_fire(placeholder.id, "started" if jobs else "failed_to_start", reason,
{"steps": steps, "jobs": jobs, "skipped": skipped})
async def check_target(session: Any, target_id: str) -> str | None:
return "当前版本还不能按时发起「数据源维护」这类目标"
async def _start(source_id: str, steps: list[str], *, actor: str, fire_at: str,
schedule_id: str) -> tuple[list[dict[str, Any]], list[dict[str, str]]]:
"""按顺序建任务并开始:探查(introspect)→ 剖析 / 观察(一个剖析任务:范围按选的步骤)。返回 (建了的任务, 没执行的步骤
和原因)。"""
from app.data import catalog_profile, catalog_profile_jobs, introspect_jobs, table_versions
from app.db.models import DataSource
jobs: list[dict[str, Any]] = []
skipped: list[dict[str, str]] = []
introspect_id: str | None = None
if "introspect" in steps:
async with SessionLocal() as session:
row = await session.get(DataSource, source_id)
try:
job, n = await introspect_jobs.prepare(session, row, actor=actor, trigger="schedule", fire_at=fire_at,
schedule_id=schedule_id)
except (introspect_jobs.IntrospectRefused, introspect_jobs.IntrospectBusy, introspect_jobs.IntrospectFailed,
introspect_jobs.ScopeTooLarge) as e:
skipped.append({"step": "introspect", "reason": str(e)})
else:
introspect_jobs.manager.start(job.id)
introspect_id = job.id
jobs.append({"step": "introspect", "kind": "introspect", "id": job.id, "tables": n})
wanted = [s for s in steps if s in ("profile_needed", "observe")]
if not wanted:
return jobs, skipped
if introspect_id is not None:
await _entered(source_id, introspect_id)
scope = "maintenance" if len(wanted) == 2 else ("needed" if wanted == ["profile_needed"] else "observe")
async with SessionLocal() as session:
row = await session.get(DataSource, source_id)
try:
settings = catalog_profile.ensure_enabled(row)
except catalog_profile.ProfileDisabled as e:
skipped += [{"step": s, "reason": str(e)} for s in wanted]
return jobs, skipped
try:
view = await table_versions.resolve_source(session, row)
job, tasks = await catalog_profile_jobs.create_job(
session, row, view, tables=None, scope=scope, settings=settings, actor=actor, trigger="schedule",
fire_at=fire_at, schedule_id=schedule_id, allow_empty=True)
except (catalog_profile_jobs.ProfileBusy, catalog_profile_jobs.ProfileNothing,
table_versions.SnapshotMissing) as e:
skipped += [{"step": s, "reason": str(e)} for s in wanted]
return jobs, skipped
catalog_profile_jobs.manager.start(job.id)
jobs.append({"step": "+".join(wanted), "kind": "profile", "id": job.id, "scope": scope, "tables": len(tasks)})
return jobs, skipped
async def schedules_for(session: Any, source_id: str) -> list[dict[str, Any]]:
"""指向这个数据源的维护定时(数据源设置里「定时维护」的快捷入口读它,第 4 波):[{id, name, cron, timezone, enabled,
next_run_at}]。"""
from app.db.models import Schedule
rows = (await session.execute(select(Schedule).where(
Schedule.target_kind == "source_maintenance", Schedule.target_id == source_id)
.order_by(Schedule.created_at))).scalars()
return [{"id": r.id, "name": r.name, "cron": r.cron, "timezone": r.timezone, "enabled": r.enabled,
"next_run_at": r.next_run_at.isoformat() if r.next_run_at else None} for r in rows]
__all__ = ["DEFAULT_STEPS", "MAINTENANCE_OPTION", "STEPS", "STEP_LABEL", "check_target", "fire", "schedules_for",
"steps_of", "steps_problem", "with_steps"]
+30
View File
@@ -357,3 +357,33 @@ async def notify_action(action_id: str, event: str) -> int:
"subject": {"action_id": action_id, "run_id": rec.run_id, "task_id": task_id}}
return await _push(channels, message, subject_kind="action", subject_id=action_id, event=event,
run_id=rec.run_id, task_id=task_id)
async def notify_source_changed(source_id: str) -> int:
"""数据源有要处理的变化(source.changed,03 B3 ④、目录-D9):只推还要人看(开着、知道了还没处理)的「要处理」级变化,
按源汇总;消息里只有源名、各级别条数和链接,不放表名、列名、取值。探查、剖析任务结束时有新的「要处理」变化才调。
这一行的 scopes 第 4 波有了「变化」标签页才打开:现在 _channels 推不出去,这里返回 0(照实:没有界面看不了的通知不发)。"""
from app.data import source_changes
from app.db.models import DataSource
event = "source.changed"
async with SessionLocal() as session:
row = await session.get(DataSource, source_id)
if row is None:
return 0
counts = await source_changes.counts(session, source_id)
by = counts["by_severity"]
if not by.get("action"):
return 0
channels = await _channels(session, event, None, [])
if not channels:
return 0
base = await _base_url(session)
lines = [("数据源", f"「{row.name}」"), ("要处理", f"{by['action']} 条"), ("提醒", f"{by.get('warning', 0)} 条"),
("说明", f"{by.get('info', 0)} 条")]
message = {"event": event, "status": "action", "title": f"数据源「{row.name[:80]}」有 {by['action']} 个变化要处理",
"lines": lines, "links": [("去看看", f"{base}/data/catalog/{source_id}")] if base else [],
"subject": {"source_id": source_id}}
return await _push(channels, message, subject_kind="source", subject_id=source_id, event=event, run_id=None,
task_id=None)
+239
View File
@@ -0,0 +1,239 @@
"""数据源的变化记录:接口、处理动作、给召回的拉取、通知(next-round 03 B3)。
SQLite 夹具(虚构的店铺库)探查建基线、写目录,再删一列、重新探查得到真实的结构变化;取值变化(新码值)直接经
source_changes.record 记一条。守的是:
- GET /changes 缺省列开着的和知道了的,带各级别计数;状态、种类筛选;不认识的状态 422;
- 处理:误报必须写理由(422),处理过的再处理 409;清理目录把影响面里的项从目录里去掉、记目录版本;加入码值走
apply_patch(含义待填写、依据写变化编号)、记目录版本;知道了只改状态;剖析没开时「重新剖析」照实 422;
- 同一个变化再次测到只更新 last_seen_at、seen_count;
- list_changes 按 seq 游标拉、affecting 按表反查、triggers_recall 只认结构类和触及规则的新码值;
- source.changed 的推送:订阅还没打开时推不出去;打开后(测试里临时打开)消息里只有源名、条数和链接。
"""
from __future__ import annotations
import json
import sqlite3
import uuid
import httpx
import pytest
from httpx import ASGITransport, AsyncClient
from sqlalchemy import select
from app.data import catalog, source_changes as sc
from app.db.base import SessionLocal
from app.db.models import DataSource
from app.db.models_sources import SourceChange
from app.main import app
from app.notify import core, service
from tests.fixtures.sources import drop_source, introspect_now
@pytest.fixture
async def client():
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as c:
yield c
@pytest.fixture
async def shop(client, tmp_path):
path = tmp_path / "shop.db"
db = sqlite3.connect(path)
db.executescript("""
CREATE TABLE orders (id INTEGER PRIMARY KEY, status VARCHAR(10), channel VARCHAR(8), amount NUMERIC(10, 2));
INSERT INTO orders VALUES (1, 'paid', 'web', 3), (2, 'void', 'shop', 4);
""")
db.commit()
db.close()
name = f"chg_{uuid.uuid4().hex[:8]}"
r = await client.post("/api/datasources", json={"name": name, "kind": "sqlite", "database": str(path),
"readonly": True})
sid = r.json()["id"]
await introspect_now(client, sid)
async with SessionLocal() as session:
await catalog.write_entry(session, sid, "orders", {
"label": catalog.make_item("订单", "human"),
"columns": {"status": {"codes": catalog.make_item({"paid": "已支付", "void": "作废"}, "human"),
"label": catalog.make_item("状态", "human")},
"channel": {"codes": catalog.make_item({"web": "线上"}, "profile", "proposed")}},
}, if_version=0, actor="测试员")
db = sqlite3.connect(path)
db.execute("ALTER TABLE orders DROP COLUMN status")
db.commit()
db.close()
await introspect_now(client, sid)
yield {"id": sid, "name": name}
await drop_source(sid)
async def _change(sid: str, kind: str) -> SourceChange:
async with SessionLocal() as session:
return (await session.execute(select(SourceChange).where(
SourceChange.source_id == sid, SourceChange.kind == kind))).scalars().first()
async def _codes_new(sid: str, *, rules: bool = False) -> SourceChange:
async with SessionLocal() as session:
row = await session.get(DataSource, sid)
change, new = await sc.record(session, row, sc.NewChange(
kind=sc.CODES_NEW, table="orders", column="channel", severity="warning",
after={"codes": ["app"], "shares": {"app": 0.4}}, basis={"method": "full", "key": "full"},
affects={"catalog": [], "templates": [], "rules_touched": rules}, key=("full", ("app",))),
sc.Detected(job_kind="profile", job_id="job-x", trigger="schedule"))
await session.commit()
assert new
return change
async def test_list_counts_and_filters(client, shop):
sid = shop["id"]
r = await client.get(f"/api/datasources/{sid}/changes")
assert r.status_code == 200, r.text
body = r.json()
[removed] = [c for c in body["items"] if c["kind"] == "column_removed"]
assert removed["schema"] == "agentlab-change/1" and removed["column"] == "status" and removed["status"] == "open"
assert removed["severity"] == "action" and "删除了列" in removed["summary"]
assert body["counts"]["pending"] == body["total"] and body["counts"]["by_severity"]["action"] >= 1
r = await client.get(f"/api/datasources/{sid}/changes", params={"kind": ["column_added"]})
assert r.json()["items"] == []
assert (await client.get(f"/api/datasources/{sid}/changes", params={"status": "nope"})).status_code == 422
one = (await client.get(f"/api/datasources/{sid}/changes/{removed['id']}")).json()["change"]
assert one["id"] == removed["id"]
async def test_dismiss_needs_a_reason_and_finished_changes_are_final(client, shop):
sid = shop["id"]
change = await _change(sid, "column_removed")
url = f"/api/datasources/{sid}/changes/{change.id}/resolve"
r = await client.post(url, json={"action": "dismiss"})
assert r.status_code == 422 and "理由" in r.json()["detail"]
r = await client.post(url, json={"action": "dismiss", "note": "测试库里故意删的列"}, headers={"X-Actor": "tester"})
assert r.status_code == 200, r.text
got = r.json()["change"]
assert got["status"] == "dismissed" and got["resolution"]["note"] == "测试库里故意删的列"
assert got["resolution"]["action"] == "dismiss" and got["resolution"]["by"] == "tester"
assert (await client.post(url, json={"action": "acknowledge"})).status_code == 409
counts = (await client.get(f"/api/datasources/{sid}/changes")).json()["counts"]
assert counts["by_severity"]["action"] == 0, "误报不再算要处理"
async def test_clean_catalog_removes_the_items_and_records_the_catalog_version(client, shop):
sid = shop["id"]
change = await _change(sid, "column_removed")
r = await client.post(f"/api/datasources/{sid}/changes/{change.id}/resolve", json={"action": "clean_catalog"})
assert r.status_code == 200, r.text
body = r.json()
async with SessionLocal() as session:
entry = await catalog.read_entry(session, sid, "orders")
assert "status" not in (entry.notes.get("columns") or {}), "说到被删的列的项都清掉了"
assert entry.notes["label"]["value"] == "订单", "别的项不动"
assert body["catalog_version"] == entry.version == body["change"]["resolution"]["catalog_version"]
assert body["change"]["status"] == "resolved"
async def test_add_codes_goes_through_apply_patch(client, shop):
sid = shop["id"]
change = await _codes_new(sid)
r = await client.post(f"/api/datasources/{sid}/changes/{change.id}/resolve", json={"action": "add_codes"})
assert r.status_code == 200, r.text
async with SessionLocal() as session:
entry = await catalog.read_entry(session, sid, "orders")
item = entry.notes["columns"]["channel"]["codes"]
assert item["value"] == {"web": "线上", "app": ""}, "码值是补充:原来的留着,新码值含义待填写"
assert (item["source"], item["status"]) == ("human", "confirmed")
assert change.id[:6] in item["reason"], "确认时的依据写这条变化的编号"
assert r.json()["catalog_version"] == entry.version
removed = await _change(sid, "column_removed")
r = await client.post(f"/api/datasources/{sid}/changes/{removed.id}/resolve", json={"action": "add_codes"})
assert r.status_code == 422, "只有新码值能加入码值"
async def test_acknowledge_and_reprofile_when_profiling_is_off(client, shop):
sid = shop["id"]
change = await _codes_new(sid)
url = f"/api/datasources/{sid}/changes/{change.id}/resolve"
r = await client.post(url, json={"action": "reprofile"})
assert r.status_code == 422 and "剖析" in r.json()["detail"], "剖析没开:照实说做不了"
r = await client.post(url, json={"action": "acknowledge"})
assert r.status_code == 200 and r.json()["change"]["status"] == "acknowledged"
counts = (await client.get(f"/api/datasources/{sid}/changes")).json()["counts"]
assert counts["pending"] > counts["open"], "知道了还没处理的照样算要人看"
async def test_same_change_detected_again_only_bumps_seen_count(shop):
sid = shop["id"]
first = await _codes_new(sid)
async with SessionLocal() as session:
row = await session.get(DataSource, sid)
again, new = await sc.record(session, row, sc.NewChange(
kind=sc.CODES_NEW, table="orders", column="channel", severity="action", key=("full", ("app",))),
sc.Detected(job_kind="profile"))
await session.commit()
assert not new and again.id == first.id and again.seen_count == 2 and again.severity == "action"
async def test_cursor_pull_affecting_and_recall_triggers(shop):
sid, name = shop["id"], shop["name"]
codes = await _codes_new(sid, rules=True)
async with SessionLocal() as session:
everything = await sc.list_changes(session, source_id=sid)
seqs = [c["seq"] for c in everything]
assert seqs == sorted(seqs) and len(seqs) >= 2
rest = await sc.list_changes(session, source_id=sid, since_seq=seqs[0])
assert [c["seq"] for c in rest] == seqs[1:], "按 seq 游标:大于 since_seq 的"
hit = await sc.affecting(session, name, ["ORDERS"])
assert {c["id"] for c in hit} == {c["id"] for c in everything}, "表名不分大小写"
assert await sc.affecting(session, name, ["refunds"]) == []
assert await sc.affecting(session, name, ["orders"], since=seqs[-1]) == []
by = {c["kind"]: c for c in everything}
assert sc.triggers_recall(by["column_removed"]), "结构类触发召回"
assert sc.triggers_recall(next(c for c in everything if c["id"] == codes.id)), "触及有效记录条件的新码值触发召回"
assert not sc.triggers_recall({"kind": "volume_drop", "affects": {}}), "纯取值漂移只算数据质量提醒"
async def test_source_changed_message_has_only_name_counts_and_link(shop, monkeypatch):
sid, name = shop["id"], shop["name"]
assert await service.notify_source_changed(sid) == 0, "订阅还没打开(第 4 波):推不出去"
got: list[httpx.Request] = []
def handler(request: httpx.Request) -> httpx.Response:
got.append(request)
return httpx.Response(200, json={"ok": True})
monkeypatch.setattr(core, "_transport", httpx.MockTransport(handler))
monkeypatch.setattr(core, "RETRY_DELAYS", (0, 0, 0))
monkeypatch.setitem(service.EVENTS, "source.changed", service.Event("数据源有变化要处理", ("global",)))
async with SessionLocal() as session:
from app.core.crypto import encrypt
from app.db.models import NotifyChannel
channel = NotifyChannel(name=f"变化-{uuid.uuid4().hex[:6]}", scope="global", format="generic",
url=encrypt("https://hooks.example.com/agentlab/changes-7a1c"),
events=["source.changed"])
session.add(channel)
await session.commit()
channel_id = channel.id
try:
assert await service.notify_source_changed(sid) == 1
[request] = got
body = json.loads(request.content)
text = body["title"] + body["text"]
assert body["event"] == "source.changed" and name in body["title"]
assert "要处理" in text and "orders" not in text and "status" not in text and "app" not in text, \
"不放表名、列名、取值"
assert body["subject"] == {"source_id": sid}
finally:
async with SessionLocal() as session:
from app.db.models import NotifyChannel
row = await session.get(NotifyChannel, channel_id)
await session.delete(row)
await session.commit()
def test_maintenance_option_is_hidden_from_the_datasource_form():
from app.api import datasources
from app.data import source_maintenance
assert source_maintenance.MAINTENANCE_OPTION in datasources._MANAGED_OPTIONS
+210
View File
@@ -0,0 +1,210 @@
"""数据源的定时维护(app/data/source_maintenance.py,next-round 03 B4、总方案 X4)。
定时只有一套:维护就是一条 target_kind=source_maintenance 的 Schedule,认领、补跑、积压只记一条都是 scheduler 那一套。
这里守:
- 到点按顺序建探查任务 → 剖析任务(范围按选的步骤),任务上记 trigger=schedule 和 fire_at;剖析排在同一次触发的探查后面
(同源队串行:探查做完剖析才开始);触发记录写明建了哪几个任务;
- 照实记没执行的:剖析没开时剖析、观察两步不执行(探查照做);数据源停用、不存在、是上传表格时整次不执行;上一次的探查
还没做完时这次跳过;
- 调度循环的认领:错过的只补最近一次、积压只记一条;
- 维护步骤的接口:不合规 422,合规存进 options、数据源表单不显示它;核对目标(check_target)。
"""
from __future__ import annotations
import sqlite3
import uuid
from datetime import timedelta
import pytest
from httpx import ASGITransport, AsyncClient
from sqlalchemy import delete, select
from app.data import catalog, catalog_profile_jobs, introspect_jobs, source_maintenance as sm
from app.db.base import SessionLocal, utcnow
from app.db.models import CatalogProfileJob, DataSource, Schedule, ScheduleFire
from app.db.models_sources import IntrospectJob
from app.main import app
from app.scheduler import loop, service
from tests.fixtures.sources import drop_source, introspect_now
@pytest.fixture
async def client():
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as c:
yield c
async def _source(client, path, *, profile: bool = True) -> str:
db = sqlite3.connect(path)
db.executescript("""
CREATE TABLE orders (id INTEGER PRIMARY KEY, status VARCHAR(10), paid_on DATE);
INSERT INTO orders VALUES (1, 'paid', '2026-10-01'), (2, 'void', '2026-10-02');
""")
db.commit()
db.close()
r = await client.post("/api/datasources", json={"name": f"mnt_{uuid.uuid4().hex[:8]}", "kind": "sqlite",
"database": str(path), "readonly": True})
sid = r.json()["id"]
await introspect_now(client, sid)
async with SessionLocal() as session:
row = await session.get(DataSource, sid)
if profile:
row.options = {**(row.options or {}), "catalog_profile": {"enabled": True}}
await session.commit()
await catalog.write_entry(session, sid, "orders", {
"columns": {"status": {"codes": catalog.make_item({"paid": "已支付", "void": "作废"}, "human")}},
}, if_version=0, actor="测试员")
return sid
async def _schedule(sid: str, **over) -> str:
async with SessionLocal() as session:
fields = {"name": f"维护-{uuid.uuid4().hex[:6]}", "cron": "0 3 * * *", "timezone": "Asia/Shanghai",
**service.target_columns("source_maintenance", sid), **over}
row = Schedule(**fields)
session.add(row)
await session.commit()
return row.id
@pytest.fixture
async def made():
ids: list[str] = []
yield ids
async with SessionLocal() as session:
await session.execute(delete(Schedule).where(Schedule.id.in_(ids)))
await session.commit()
async def _wait_jobs(fire: ScheduleFire) -> None:
for job in (fire.inputs or {}).get("jobs") or []:
manager = introspect_jobs.manager if job["kind"] == "introspect" else catalog_profile_jobs.manager
await manager.wait(job["id"], timeout=60)
async def test_fire_builds_introspect_then_profile_in_order(client, tmp_path, made):
sid = await _source(client, tmp_path / "a.db")
try:
schedule_id = await _schedule(sid)
made.append(schedule_id)
at = utcnow().replace(microsecond=0)
fire = await service.fire_once(schedule_id, at, "manual", actor="测试员")
assert fire.status == "started" and fire.reason is None, fire.reason
jobs = fire.inputs["jobs"]
assert [j["kind"] for j in jobs] == ["introspect", "profile"] and jobs[1]["scope"] == "maintenance"
assert fire.inputs["steps"] == list(sm.STEPS)
await _wait_jobs(fire)
async with SessionLocal() as session:
intro = await session.get(IntrospectJob, jobs[0]["id"])
prof = await session.get(CatalogProfileJob, jobs[1]["id"])
assert intro.trigger == "schedule" and intro.scope["fire_at"] == at.isoformat()
assert intro.scope["schedule_id"] == schedule_id and intro.created_by == "测试员"
assert catalog_profile_jobs.trigger_of(prof) == {"trigger": "schedule", "fire_at": at.isoformat(),
"schedule_id": schedule_id}
assert prof.started_at >= intro.finished_at, "同源串行:探查做完,剖析才开始"
assert prof.status in ("succeeded", "partial"), prof.error
# 同一次触发再来一遍:交回已有的那一条,不再建任务
again = await service.fire_once(schedule_id, at, "manual", actor="测试员")
assert again.id == fire.id
finally:
await drop_source(sid)
async def test_profile_off_records_steps_not_run_and_still_introspects(client, tmp_path, made):
sid = await _source(client, tmp_path / "b.db", profile=False)
try:
schedule_id = await _schedule(sid)
made.append(schedule_id)
fire = await service.fire_once(schedule_id, utcnow().replace(microsecond=0), "manual")
assert fire.status == "started"
assert [j["kind"] for j in fire.inputs["jobs"]] == ["introspect"]
assert {s["step"] for s in fire.inputs["skipped"]} == {"profile_needed", "observe"}
assert "剖析需要的表没有执行" in fire.reason and "取值观察没有执行" in fire.reason and "数据剖析" in fire.reason
await _wait_jobs(fire)
# 只选观察一步:剖析没开就整次没执行
async with SessionLocal() as session:
row = await session.get(DataSource, sid)
row.options = sm.with_steps(row.options, ["observe"], actor="测试员")
await session.commit()
fire = await service.fire_once(schedule_id, utcnow().replace(microsecond=0) + timedelta(seconds=1), "manual")
assert fire.status == "failed_to_start" and "取值观察没有执行" in fire.reason
finally:
await drop_source(sid)
async def test_disabled_missing_and_busy_sources(client, tmp_path, made):
sid = await _source(client, tmp_path / "c.db")
try:
schedule_id = await _schedule(sid)
made.append(schedule_id)
async with SessionLocal() as session:
row = await session.get(DataSource, sid)
row.enabled = False
await session.commit()
fire = await service.fire_once(schedule_id, utcnow().replace(microsecond=0), "manual")
assert fire.status == "failed_to_start" and "已停用" in fire.reason
async with SessionLocal() as session:
row = await session.get(DataSource, sid)
row.enabled = True
session.add(IntrospectJob(source_id=sid, status="queued", scope={}))
await session.commit()
fire = await service.fire_once(schedule_id, utcnow().replace(microsecond=0) + timedelta(seconds=2), "manual")
assert fire.status == "skipped_overlap" and "正在探查结构" in fire.reason
async with SessionLocal() as session:
await session.execute(delete(IntrospectJob).where(IntrospectJob.source_id == sid,
IntrospectJob.status == "queued"))
await session.commit()
finally:
await drop_source(sid)
fire = await service.fire_once(schedule_id, utcnow().replace(microsecond=0) + timedelta(seconds=3), "manual")
assert fire.status == "failed_to_start" and fire.reason == sm.SOURCE_GONE
async def test_loop_claims_and_only_the_latest_missed_fire_runs(client, tmp_path, made):
sid = await _source(client, tmp_path / "d.db")
try:
async with SessionLocal() as session:
row = await session.get(DataSource, sid)
row.options = sm.with_steps(row.options, ["introspect"], actor="测试员")
await session.commit()
now = utcnow().replace(second=0, microsecond=0)
# 每 10 分钟一次,停机错过了好几次:只补最近一次,更早的积压只记一条
schedule_id = await _schedule(sid, cron="*/10 * * * *", timezone="UTC",
next_run_at=now - timedelta(minutes=40), misfire_grace_s=3600)
made.append(schedule_id)
fires = await loop.fire_due(now + timedelta(minutes=1))
assert sorted(f.status for f in fires) == ["skipped_misfire", "started"], [(f.kind, f.status) for f in fires]
started = next(f for f in fires if f.status == "started")
assert [j["kind"] for j in started.inputs["jobs"]] == ["introspect"]
await _wait_jobs(started)
async with SessionLocal() as session:
sched = await session.get(Schedule, schedule_id)
count = len(list((await session.execute(select(ScheduleFire).where(
ScheduleFire.schedule_id == schedule_id))).scalars()))
assert sched.next_run_at > now and count == 2
finally:
await drop_source(sid)
async def test_steps_api_and_check_target(client, tmp_path):
sid = await _source(client, tmp_path / "e.db")
try:
url = f"/api/datasources/{sid}/maintenance"
body = (await client.get(url)).json()
assert body["steps"] == list(sm.STEPS) and body["profile_enabled"] is True and body["schedules"] == []
assert (await client.put(url, json={"steps": ["nope"]})).status_code == 422
assert (await client.put(url, json={"steps": []})).status_code == 422
r = await client.put(url, json={"steps": ["observe", "introspect"]})
assert r.status_code == 200 and r.json()["steps"] == ["introspect", "observe"], "按固定顺序存"
[shown] = [s["options"] for s in (await client.get("/api/datasources")).json() if s["id"] == sid]
assert sm.MAINTENANCE_OPTION not in shown, "数据源表单不显示、不提交它"
async with SessionLocal() as session:
assert await sm.check_target(session, sid) is None
assert await sm.check_target(session, "missing") == sm.SOURCE_GONE
row = await session.get(DataSource, sid)
row.origin = "upload"
await session.commit()
assert await sm.check_target(session, sid) == sm.NOT_MAINTAINED
finally:
await drop_source(sid)