diff --git a/.env.example b/.env.example index 76e377ba..28405256 100644 --- a/.env.example +++ b/.env.example @@ -14,6 +14,8 @@ # 设成 PostgreSQL 时整个应用库和断点都放进去（要 [postgres] 额外依赖，environment.yml 已带）；数据目录照样要有。 # 从现有的 SQLite 数据目录搬过去：python -m app.db copy-to-postgres data（README「用 PostgreSQL」） # AGENTLAB_DATABASE_URL=postgresql+asyncpg://agentlab:密码@127.0.0.1:5432/agentlab +# 要 SSL 时加 ?sslmode=require（verify-ca / verify-full 同理）；证书文件用环境变量 PGSSLROOTCERT / PGSSLCERT / PGSSLKEY，不写在地址里 +# AGENTLAB_DATABASE_URL=postgresql+asyncpg://agentlab:密码@db.example.com:5432/agentlab?sslmode=verify-full # 断点放另一个库时才设（不设就和应用库同一个库） # AGENTLAB_CHECKPOINT_URL=postgresql://agentlab:密码@127.0.0.1:5432/agentlab_checkpoints # 连接池：每个进程常驻几条、高峰再多开几条；设成 0 不用连接池（前面有 PgBouncer 时） diff --git a/README.md b/README.md index a6b8bd3c..e69832d3 100644 --- a/README.md +++ b/README.md @@ -1042,16 +1042,30 @@ AGENTLAB_DATABASE_URL=postgresql+asyncpg://agentlab:<密码>@127.0.0.1:5432/agen - 要 `[postgres]` 额外依赖（`environment.yml`、Docker 镜像都已带上）；库要先建好（`CREATE DATABASE agentlab`）， 表由服务启动时建，老版本建的库启动时补列补索引，同 SQLite。启动日志第一句写用的是哪种库、哪台主机哪个库（不写密码）。 +- 没装 `[postgres]` 就设了这一项，启动时直接报要装什么（`pip install -e 'backend[postgres]' -c backend/requirements.lock`）。 +- SSL：地址后加 `?sslmode=require`（或 `verify-ca`、`verify-full`；asyncpg 的写法 `?ssl=require` 也认），启动时换成 asyncpg + 认的参数。证书文件不写在地址里（asyncpg 不收 `sslrootcert` 这类参数，写了启动就报错），设环境变量 `PGSSLROOTCERT`、 + `PGSSLCERT`、`PGSSLKEY`——应用库的 asyncpg 和断点库的 libpq 都认。地址里别的 libpq 参数（`application_name`、 + `connect_timeout` 之类）asyncpg 也不收，同样启动时报出来。 - 断点（LangGraph 的 checkpoints、checkpoint_blobs、checkpoint_writes、checkpoint_migrations 四张表，不和应用的表重名） - 缺省放在同一个库；要放别处设 `AGENTLAB_CHECKPOINT_URL=postgresql://…`。 + 缺省放在同一个库；要放别处设 `AGENTLAB_CHECKPOINT_URL=postgresql://…`（这一项是 libpq 的写法，`sslrootcert` 等可以直接写）。 - 连接池：每个进程常驻 `AGENTLAB_DB_POOL_SIZE`（缺省 10）条、高峰再多开 `AGENTLAB_DB_MAX_OVERFLOW`（缺省 20）条，取连接前探活， 30 分钟换一次；断点库另有一个小池子。前面有 PgBouncer（事务模式）时：`AGENTLAB_DB_POOL_SIZE` 设成 0（不用连接池）、 地址后加 `?prepared_statement_cache_size=0`；断点库用预备语句，要 PgBouncer 1.21 以上开 `max_prepared_statements`， 或者用 `AGENTLAB_CHECKPOINT_URL` 让它直连 PostgreSQL。 +- 连接数的账：每个进程最多 `AGENTLAB_DB_POOL_SIZE + AGENTLAB_DB_MAX_OVERFLOW`（缺省 10 + 20 = 30）条应用库连接，断点库的池子 + 最多 `min(AGENTLAB_MAX_CONCURRENT_RUNS, 20)` 条（至少 4，缺省 20），合起来一个进程最多 50 条；导入命令另占几条。PostgreSQL + 缺省 `max_connections=100`，还要留几条给 `psql`、备份和超级用户（`superuser_reserved_connections` 缺省 3）：一个进程够用， + 起几个进程就是几倍——要么把 `max_connections` 调大（约「进程数 × 50 + 20」），要么调小两边的池子（比如 POOL_SIZE=5、 + MAX_OVERFLOW=10、MAX_CONCURRENT_RUNS=10，一个进程最多 25 条），要么前面放 PgBouncer。 - 和 SQLite 上的行为一致：字符串列不限长（SQLite 本来就不查长度，模型里写的长度只是示意）、文本按字节比大小和排序 （列上带 COLLATE "C"，同 SQLite，不看建库时选的区域设置）、整数列是 BIGINT、JSON 列存原文（PostgreSQL 的 json，不是 JSONB： - JSONB 会重排对象里的键）、时间存不带时区的 UTC、排序时可空列的 NULL 照 SQLite 的位置（升序在前、降序在后）。两处 SQLite 照存而 PostgreSQL 不收的值会被换掉：JSON 里的 NaN / Infinity 写成 null，字符串里的 - NUL 字符去掉。 + JSONB 会重排对象里的键）、时间存不带时区的 UTC、排序时可空列的 NULL 照 SQLite 的位置（升序在前、降序在后；只管直接按列 + 排的，按聚合、coalesce 之类的表达式排的不管，列表都另带了 id 之类的决胜键，并列时两种库排出来一样）、按 JSON 里的键筛选 + 按 JSON 的类型比（true 不等于 1）。两处 SQLite 照存而 PostgreSQL 不收的值会被换掉：JSON 里的 NaN / Infinity 写成 null， + 文本（JSON 里的、普通文本列里的）中的 NUL 字符去掉。 +- 不分大小写的搜索（运行列表按名字、对象索引按注释）：文本列是 COLLATE "C"，列上的 ILIKE 只认 ASCII 的大小写，所以按建库时的 + 区域设置折大小写——建库用 `en_US.UTF-8`、`C.UTF-8` 这类区域，É/é、全角Ａ/ａ 都认；用纯 `C` 区域建的库只认 ASCII。 - `AGENTLAB_DATA_DIR` 照旧要设：工件、上传表格的版本库、原件、附件、记录表、沙箱工作区、`.secret_key` 都还是文件。 **从 SQLite 搬过来。** `python -m app.db copy-to-postgres` 把数据目录里的 `agentlab.db` 和 `checkpoints.db` 导进 PostgreSQL， @@ -1071,12 +1085,18 @@ cd backend && python -m app.db copy-to-postgres ../data postgresql+asyncpg://age 空间，缺省系统临时目录）。读的是 SQLite 备份接口拷出的一份快照，在快照上先补到当前的表结构，来源的内容不改。 - 重跑是幂等的：目标上已有同一主键（或撞了别的唯一约束）的行一律不动，只补上还没有的，不重复、也不覆盖——所以导入之后又在 SQLite 上改过的行，重跑不会带过去；要整份重来就清空目标库再导。断点同理，已有的不重写。 -- 能查到的只有上面两样：服务连着别的数据目录、另一台机器上的服务连着目标库，查不出来，要自己保证。来源旁边的 -wal 是上次 - 强杀留下的，确认服务已停后加 `--assume-stopped`。 +- 能查到的只有上面两样：服务连着别的数据目录、另一台机器上的服务连着目标库，查不出来，要自己保证。服务正常停下时会关掉 + 所有连接、-wal 随之消失；还在的是上次强杀留下的，确认服务已停后加 `--assume-stopped`。 +- 老库里 SQLite 存得下、PostgreSQL 不收的会先在快照上处理、在报告里照实列出：老版本补列时没带缺省值、留着 NULL 的 NOT NULL 列 + 填成模型的缺省值（启动时的迁移在 SQLite 上也这么填，两边存的一样）；**孤儿行**——外键指向已经不在的行（SQLite 当时没强制 + 外键留下的）——外键是 SET NULL 的置空，别的（CASCADE、不可空）不导入，按表列出条数，不算核对失败（父行被删时它们本来就该 + 跟着删）。自增主键的序列接在 SQLite 用过的最大号后面（删过的号不复用）。 - 导完 `agentlab.db` 留在数据目录里也没关系（可以当回退用）：它照样算 AgentLab 自己的文件，不能登记成数据源。 **备份。** PostgreSQL 用 `pg_dump`（应用表和断点在同一个库里就一次带上），数据目录照旧整份拷。两样最好在停服务时一起做； -不停服务的话先 dump 库、再拷数据目录：工件、原件按内容寻址、只增不改，库里引用到的文件拷数据目录时都在。 +不停服务的话先 dump 库、再拷数据目录：工件、原件按内容寻址、只增不改，库里引用到的文件拷数据目录时都在。例外是填报的记录表 +`records/*.db`：SQLite 文件、原地追加写，开着服务直接拷可能拷到写了一半的——要么停服务拷，要么一个个用 SQLite 的备份接口拷 +（`sqlite3 data/records/x.v1.db ".backup /备份目录/x.v1.db"`）。 **多台机器**见下一节[多进程](#多进程)。 diff --git a/backend/app/api/access_keys.py b/backend/app/api/access_keys.py index 23b374f8..f7caecba 100644 --- a/backend/app/api/access_keys.py +++ b/backend/app/api/access_keys.py @@ -151,7 +151,7 @@ async def reference() -> dict[str, Any]: async def list_service_accounts(session: AsyncSession = Depends(get_session)) -> dict[str, Any]: _require_admin() rows = list((await session.execute(select(User).where(User.kind == keys.SERVICE_KIND) - .order_by(User.created_at))).scalars()) + .order_by(User.created_at, User.id))).scalars()) counts: dict[str, int] = {} for owner_id in (await session.execute(select(AccessKey.owner_user_id).where( AccessKey.owner_user_id.in_([r.id for r in rows] or [""]), AccessKey.revoked_at.is_(None)))).scalars(): @@ -232,7 +232,7 @@ async def list_keys(session: AsyncSession = Depends(get_session)) -> dict[str, A principal = current_principal() if principal.mode == "off" or not principal.user_id: return {"items": []} - query = select(AccessKey).order_by(AccessKey.created_at.desc()) + query = select(AccessKey).order_by(AccessKey.created_at.desc(), AccessKey.id.desc()) if not _is_admin(): query = query.where(AccessKey.owner_user_id == principal.user_id) rows = list((await session.execute(query)).scalars()) diff --git a/backend/app/api/actions.py b/backend/app/api/actions.py index 1ece0340..47fe2f3e 100644 --- a/backend/app/api/actions.py +++ b/backend/app/api/actions.py @@ -25,7 +25,7 @@ async def list_actions( limit: int = Query(default=100, ge=1, le=500), ) -> dict[str, Any]: ensure_role("builder") - stmt = select(ActionRecord).order_by(ActionRecord.created_at.desc()).limit(limit) + stmt = select(ActionRecord).order_by(ActionRecord.created_at.desc(), ActionRecord.id.desc()).limit(limit) if run_id: stmt = stmt.where(ActionRecord.run_id == run_id) if app_id: diff --git a/backend/app/api/apps.py b/backend/app/api/apps.py index 5dd1fcea..a4f2bbb3 100644 --- a/backend/app/api/apps.py +++ b/backend/app/api/apps.py @@ -249,7 +249,7 @@ async def list_apps(include_archived: bool = False, scope: str | None = None, from app.blueprints.personal import allowed principal = _principal(x_actor) - query = select(App).order_by(App.updated_at.desc()) + query = select(App).order_by(App.updated_at.desc(), App.id.desc()) if not include_archived: query = query.where(App.status != "archived") if scope == "personal": @@ -434,7 +434,7 @@ async def assistant_conversation(app_id: str, session: AsyncSession = Depends(ge mine = Conversation.created_by == who if who else Conversation.created_by.is_(None) lookup = (select(Conversation).where(Conversation.kind == "app", Conversation.app_id == app.id, Conversation.archived.is_(False), mine) - .order_by(Conversation.last_active_at.desc()).limit(1)) + .order_by(Conversation.last_active_at.desc(), Conversation.id.desc()).limit(1)) # 取或建在进程里按 (应用, 人) 串行：编辑器开发模式下 effect 执行两次、同一个人开两个标签页，都会同时来取。库里有 # (app_id, created_by) 的部分唯一索引（ux_conversations_app_owner）：别处（另一个进程）抢先建了的，这次插入在保存点里 # 被拒，改用那一段——历史才不会被拆开。没署名（关登录又没填）的不受索引约束，靠这把进程内的锁 @@ -692,7 +692,7 @@ async def _hook_state(session: AsyncSession, app: App) -> tuple[dict[str, Any], version = await _previous(session, app) rows: dict[str, AppHook] = {} for row in (await session.execute(select(AppHook).where(AppHook.app_id == app.id) - .order_by(AppHook.created_at))).scalars(): + .order_by(AppHook.created_at, AppHook.id))).scalars(): rows.setdefault(row.name, row) return declarations(app.draft or {}), declarations(version.spec if version is not None else {}), rows diff --git a/backend/app/api/blueprints.py b/backend/app/api/blueprints.py index 0979f507..3c51ad1c 100644 --- a/backend/app/api/blueprints.py +++ b/backend/app/api/blueprints.py @@ -264,13 +264,14 @@ async def list_blueprints(scope: Literal["shared", "personal"] | None = None, # 候选只有自己的、发布成了应用的（要不要列看应用的管理权，shared_listed 逐个应用核对） query = query.where(or_(Blueprint.owner == (principal.username or ""), Blueprint.app_id.is_not(None))) else: - query = query.order_by(Blueprint.updated_at.desc()).limit(_LIST_MAX) + query = query.order_by(Blueprint.updated_at.desc(), Blueprint.id.desc()).limit(_LIST_MAX) rows += await shared_listed(session, principal, list((await session.execute(query)).scalars())) if scope in (None, "personal") and await allowed(session, principal): mine = Blueprint.scope == "personal" if principal.mode != "off": mine = mine & (Blueprint.owner == (principal.username or "")) - rows += list((await session.execute(select(Blueprint).where(mine).order_by(Blueprint.updated_at.desc()) + rows += list((await session.execute(select(Blueprint).where(mine) + .order_by(Blueprint.updated_at.desc(), Blueprint.id.desc()) .limit(_LIST_MAX))).scalars()) rows.sort(key=lambda bp: bp.updated_at or bp.created_at, reverse=True) return {"items": [await blueprint_out(session, bp) for bp in rows[:_LIST_MAX]]} diff --git a/backend/app/api/compare.py b/backend/app/api/compare.py index aa82f7d9..eda5a4a2 100644 --- a/backend/app/api/compare.py +++ b/backend/app/api/compare.py @@ -120,7 +120,7 @@ async def pick_base(session: Any, target: Run, *, schedule_only: bool = False) - Run.created_at < target.created_at) base = (await session.execute( select(Run).where(*earlier, Run.status == "succeeded", Run.manifest_hash.is_not(None)) - .order_by(Run.created_at.desc()).limit(1))).scalars().first() + .order_by(Run.created_at.desc(), Run.id.desc()).limit(1))).scalars().first() if base is None: continue between = await session.scalar(select(func.count()).select_from(Run).where( diff --git a/backend/app/api/conversations.py b/backend/app/api/conversations.py index 4d969ba3..94dc93a1 100644 --- a/backend/app/api/conversations.py +++ b/backend/app/api/conversations.py @@ -390,7 +390,7 @@ async def list_conversations( # 开登录后的 user 角色：只列自己建的；建立人为空的老对话不列 query = query.where(Conversation.created_by == mine) rows = list((await session.execute( - query.order_by(Conversation.last_active_at.desc()) + query.order_by(Conversation.last_active_at.desc(), Conversation.id.desc()) )).scalars()) # 应用的助手对话只列能管理的应用的（一个应用一次判断） managed: dict[str, bool] = {} diff --git a/backend/app/api/datasources.py b/backend/app/api/datasources.py index b1450b87..7a1c2f54 100644 --- a/backend/app/api/datasources.py +++ b/backend/app/api/datasources.py @@ -280,7 +280,7 @@ async def _recipe_infos(session: AsyncSession, rows: list[DataSource]) -> dict[s for st in (await session.execute( select(ImportStaging).where( ImportStaging.source_id.in_(ids), ImportStaging.status.in_(table_versions.STAGING_OPEN), - ).order_by(ImportStaging.created_at) + ).order_by(ImportStaging.created_at, ImportStaging.id) )).scalars(): # 按创建时间升序，后面的覆盖前面的：留下最近的那一个 out.setdefault(st.source_id, {})["open_staging"] = { diff --git a/backend/app/api/evals.py b/backend/app/api/evals.py index 37da3987..325f6ae5 100644 --- a/backend/app/api/evals.py +++ b/backend/app/api/evals.py @@ -285,7 +285,7 @@ async def _overview(session: AsyncSession, suite_ids: list[str]) -> dict[str, di for r in reversed(same)] schedules = (await session.execute(select(Schedule).where( Schedule.target_kind == "eval_suite", Schedule.target_id.in_(suite_ids)) - .order_by(Schedule.created_at))).scalars() + .order_by(Schedule.created_at, Schedule.id))).scalars() for row in schedules: try: text = cron.describe(row.cron, row.timezone) @@ -299,7 +299,7 @@ async def _overview(session: AsyncSession, suite_ids: list[str]) -> dict[str, di @router.get("/suites") async def list_suites(archived: bool = False, session: AsyncSession = Depends(get_session)) -> dict[str, Any]: """题集（归档的不列，archived=true 时一起列）。每个题集另带 latest、trend、schedules（_overview）。""" - query = select(EvalSuite).order_by(EvalSuite.updated_at.desc()) + query = select(EvalSuite).order_by(EvalSuite.updated_at.desc(), EvalSuite.id.desc()) if not archived: query = query.where(EvalSuite.archived.is_(False)) rows = list((await session.execute(query)).scalars()) @@ -457,7 +457,7 @@ async def preview_sql(suite_id: str, request: Request, session: AsyncSession = D async with SessionLocal() as read: refs = (await read.execute(select(Artifact.id).where( Artifact.run_id == outcome.run_ids[0], Artifact.kind == "query_snapshot", - Artifact.node_id == gold_rules.QUERY_NODE).order_by(Artifact.created_at.desc()))).scalars().all() + Artifact.node_id == gold_rules.QUERY_NODE).order_by(Artifact.created_at.desc(), Artifact.id.desc()))).scalars().all() snapshot = None for ref in refs: try: @@ -537,7 +537,7 @@ async def _run_queries(session: AsyncSession, run_id: str) -> list[dict[str, Any from app.db.models import Artifact rows = (await session.execute(select(Artifact).where(Artifact.run_id == run_id, Artifact.kind == QUERY_SNAPSHOT_KIND) - .order_by(Artifact.created_at))).scalars().all() + .order_by(Artifact.created_at, Artifact.id))).scalars().all() out = [] for ref in rows: try: @@ -780,7 +780,8 @@ def _snapshot_error(e: snapshots.SnapshotInvalid) -> CodedHTTPException: @router.get("/catalog-snapshots") async def list_snapshots(session: AsyncSession = Depends(get_session)) -> dict[str, Any]: - rows = list((await session.execute(select(EvalCatalogSnapshot).order_by(EvalCatalogSnapshot.created_at.desc()) + rows = list((await session.execute(select(EvalCatalogSnapshot) + .order_by(EvalCatalogSnapshot.created_at.desc(), EvalCatalogSnapshot.id.desc()) .limit(200))).scalars()) await snapshots.reconcile_stale(session, rows) return {"snapshots": [snapshots.snapshot_out(r) for r in rows]} @@ -976,7 +977,8 @@ async def list_evaluations(suite_id: str, limit: int = 50, session: AsyncSession await _suite(session, suite_id) rows = list((await session.execute(select(EvalRun).where(EvalRun.suite_id == suite_id) - .order_by(EvalRun.created_at.desc()).limit(max(1, min(limit, 200))))).scalars()) + .order_by(EvalRun.created_at.desc(), EvalRun.id.desc()) + .limit(max(1, min(limit, 200))))).scalars()) await runner.reconcile_stale(session, rows) return {"evaluations": [_evaluation_out(r) for r in rows]} @@ -1185,7 +1187,7 @@ async def get_pair(pair_id: str, session: AsyncSession = Depends(get_session)) - from app.eval import compare, runner rows = list((await session.execute(select(EvalRun).where(EvalRun.pair_id == pair_id) - .order_by(EvalRun.created_at))).scalars()) + .order_by(EvalRun.created_at, EvalRun.id))).scalars()) if not rows: raise CodedHTTPException(404, "这次 A/B 不存在", PAIR_NOT_FOUND) await runner.reconcile_stale(session, rows) diff --git a/backend/app/api/governance.py b/backend/app/api/governance.py index a5a73074..9fd5351b 100644 --- a/backend/app/api/governance.py +++ b/backend/app/api/governance.py @@ -52,7 +52,7 @@ async def tool_usage( for (rid,) in await session.execute( select(Run.id) .where(Run.workflow_id == workflow_id, counted_runs()) - .order_by(Run.created_at.desc()) + .order_by(Run.created_at.desc(), Run.id.desc()) .limit(limit_runs) ) ] diff --git a/backend/app/api/identity.py b/backend/app/api/identity.py index 6a7c2530..adb972a8 100644 --- a/backend/app/api/identity.py +++ b/backend/app/api/identity.py @@ -49,7 +49,7 @@ def user_out(row: User) -> dict[str, Any]: @router.get("/users") async def list_users(session: AsyncSession = Depends(get_session)) -> dict[str, Any]: - rows = (await session.execute(select(User).order_by(User.created_at))).scalars() + rows = (await session.execute(select(User).order_by(User.created_at, User.id))).scalars() return {"items": [user_out(r) for r in rows]} @@ -294,7 +294,7 @@ async def candidates(purpose: str, app_id: str | None = None, entry: str | None @router.get("/audit") async def list_audit(subject_kind: str | None = None, subject_id: str | None = None, limit: int = 200, session: AsyncSession = Depends(get_session)) -> dict[str, Any]: - query = select(AuditEvent).order_by(AuditEvent.at.desc()).limit(max(1, min(limit, 1000))) + query = select(AuditEvent).order_by(AuditEvent.at.desc(), AuditEvent.id.desc()).limit(max(1, min(limit, 1000))) if subject_kind: query = query.where(AuditEvent.subject_kind == subject_kind) if subject_id: diff --git a/backend/app/api/knowledge.py b/backend/app/api/knowledge.py index 5e858428..0f2bb700 100644 --- a/backend/app/api/knowledge.py +++ b/backend/app/api/knowledge.py @@ -221,7 +221,7 @@ async def list_documents( collection: str | None = None, session: AsyncSession = Depends(get_session), ) -> list[Document]: - stmt = select(Document).order_by(Document.created_at.desc()).limit(300) + stmt = select(Document).order_by(Document.created_at.desc(), Document.id.desc()).limit(300) if collection: stmt = stmt.where(Document.collection == collection) return list((await session.execute(stmt)).scalars()) @@ -710,7 +710,7 @@ class SkillOut(SkillIn): @skills_router.get("", response_model=list[SkillOut]) async def list_skills(session: AsyncSession = Depends(get_session)) -> list[Skill]: - return list((await session.execute(select(Skill).order_by(Skill.created_at.desc()))).scalars()) + return list((await session.execute(select(Skill).order_by(Skill.created_at.desc(), Skill.id.desc()))).scalars()) @skills_router.post("", response_model=SkillOut, status_code=201) diff --git a/backend/app/api/ledger.py b/backend/app/api/ledger.py index 2fe472ec..c938d611 100644 --- a/backend/app/api/ledger.py +++ b/backend/app/api/ledger.py @@ -86,7 +86,7 @@ async def effects(workflow_id: str | None = None, version: int | None = None, ap query = query.where(ActionRecord.state == state) if capability: query = query.where(ActionRecord.capability == capability) - rows = (await session.execute(query.order_by(ActionRecord.created_at.desc()).limit(limit))).scalars() + rows = (await session.execute(query.order_by(ActionRecord.created_at.desc(), ActionRecord.id.desc()).limit(limit))).scalars() return {"items": [effect_row(r) for r in rows]} diff --git a/backend/app/api/notify.py b/backend/app/api/notify.py index 62592fe3..a120585e 100644 --- a/backend/app/api/notify.py +++ b/backend/app/api/notify.py @@ -127,7 +127,7 @@ async def list_channels() -> dict[str, Any]: from app.db.models import User principal = current_principal() - stmt = select(NotifyChannel).order_by(NotifyChannel.created_at) + stmt = select(NotifyChannel).order_by(NotifyChannel.created_at, NotifyChannel.id) if principal.mode != "off": # 个人渠道只给本人看 stmt = stmt.where(or_(NotifyChannel.scope != "user", NotifyChannel.owner_id == principal.user_id)) @@ -295,7 +295,7 @@ async def list_deliveries( format 是渠道的格式：邮件渠道成功的那条界面上写「已交给邮件服务器」，不写「已送达」（退信不追踪）。""" stmt = (select(NotifyDelivery, NotifyChannel.format) .join(NotifyChannel, NotifyChannel.id == NotifyDelivery.channel_id) - .order_by(NotifyDelivery.created_at.desc()).limit(limit)) + .order_by(NotifyDelivery.created_at.desc(), NotifyDelivery.id.desc()).limit(limit)) principal = current_principal() if principal.mode != "off" and not principal.at_least("builder"): stmt = stmt.where(NotifyChannel.scope == "user", NotifyChannel.owner_id == principal.user_id) diff --git a/backend/app/api/runs.py b/backend/app/api/runs.py index d1b79a1a..fe570b3c 100644 --- a/backend/app/api/runs.py +++ b/backend/app/api/runs.py @@ -213,7 +213,7 @@ async def list_runs( """ stmt = _filtered(select(Run), workflow_id=workflow_id, status=status, run_class=run_class, q=q, schedule_id=schedule_id, origin_kind=origin_kind) - stmt = stmt.order_by(Run.created_at.desc()).limit(limit) + stmt = stmt.order_by(Run.created_at.desc(), Run.id.desc()).limit(limit) if before is not None: stmt = stmt.where(Run.created_at < before) return list((await session.execute(stmt)).scalars()) @@ -265,7 +265,10 @@ def _filtered(stmt: Any, *, workflow_id: str | None, status: str | None, run_cla if classes := _csv(run_class): stmt = stmt.where(Run.run_class.in_(classes)) if q and q.strip(): - stmt = stmt.where(Run.workflow_name.ilike(f"%{_like(q.strip())}%", escape="\\")) + # 不分大小写（db/portable.contains_ci）：PostgreSQL 的文本列按字节排序（COLLATE "C"），列上的 ILIKE 只认 ASCII 的大小写 + from app.db.portable import contains_ci + + stmt = stmt.where(contains_ci(Run.workflow_name, q.strip())) if schedule_id: stmt = stmt.where(Run.schedule_id == schedule_id) if origins := _csv(origin_kind): @@ -281,11 +284,6 @@ def _csv(value: str | None) -> list[str]: return [v.strip() for v in (value or "").split(",") if v.strip()] -def _like(text: str) -> str: - """用户输入里的 % 和 _ 按字面匹配，不当通配符。""" - return text.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_") - - @router.get("/{run_id}", response_model=RunOut) async def get_run(run_id: str, session: AsyncSession = Depends(get_session)) -> RunOut: run = await session.get(Run, run_id) @@ -343,7 +341,7 @@ async def list_artifacts( ) -> list[dict[str, Any]]: """这次运行产出的全部工件引用。审阅出具物时从这里下钻到完整证据。""" rows = await session.execute( - select(Artifact).where(Artifact.run_id == run_id).order_by(Artifact.created_at) + select(Artifact).where(Artifact.run_id == run_id).order_by(Artifact.created_at, Artifact.id) ) return [ { @@ -757,7 +755,7 @@ async def list_approvals( .join(Task, Task.approval_id == Approval.id, isouter=True) # 别人的个人应用刷新、个人蓝图试运行上的审批不列（只给拥有者，同运行列表） .where(or_(Run.id.is_(None), not_(hidden_runs(current_principal())))) - .order_by(Approval.created_at.desc()) + .order_by(Approval.created_at.desc(), Approval.id.desc()) .limit(limit) ) if status != "all" and (statuses := _csv(status)): diff --git a/backend/app/api/schedules.py b/backend/app/api/schedules.py index 393a4998..d9548802 100644 --- a/backend/app/api/schedules.py +++ b/backend/app/api/schedules.py @@ -467,7 +467,7 @@ async def _fire_outs(session: AsyncSession, fires: list[ScheduleFire]) -> list[d if fires: rows = (await session.execute( select(WebhookDelivery).where(WebhookDelivery.fire_id.in_([f.id for f in fires])) - .order_by(WebhookDelivery.created_at) + .order_by(WebhookDelivery.created_at, WebhookDelivery.id) )).scalars() for d in rows: notified.setdefault(d.fire_id, []).append({"event": d.event, "status": d.status, "at": _iso(d.created_at)}) @@ -618,10 +618,10 @@ async def trigger_options(event: str | None = None, q: str | None = None, select async def page(model: Any, *conds: Any) -> tuple[list[Any], bool]: query = select(model).where(*conds) if words: - # 名字里含这几个字，不分大小写：PostgreSQL 上 LIKE 分 ASCII 大小写、SQLite 不分，一律 ilike（转义同 runs._like） - from app.api.runs import _like + # 名字里含这几个字，不分大小写：两种库同一个说法（db/portable.contains_ci，PostgreSQL 上按库的区域设置折大小写） + from app.db.portable import contains_ci - query = query.where(model.name.ilike(f"%{_like(words)}%", escape="\\")) + query = query.where(contains_ci(model.name, words)) rows = list((await session.execute(query.order_by(model.name, model.id).limit(limit + 1))).scalars()) more = len(rows) > limit rows = rows[:limit] @@ -656,7 +656,7 @@ async def trigger_options(event: str | None = None, q: str | None = None, select objects, more = await makers[e]() entries.append({"event": e, "label": triggers.EVENT_LABEL[e], "subject_label": triggers.SUBJECT_LABEL[e], "objects": objects, "more": more}) - apps = (await session.execute(select(App.id, App.name).where(App.status != "archived").order_by(App.name))).all() + apps = (await session.execute(select(App.id, App.name).where(App.status != "archived").order_by(App.name, App.id))).all() return { "events": entries, "origins": [{"kind": k, "label": triggers.ORIGIN_LABEL[k]} for k in triggers.FORMAL_ORIGINS], @@ -737,7 +737,7 @@ async def _preview_event(session: AsyncSession, body: dict[str, Any]) -> dict[st @router.get("") async def list_schedules(session: AsyncSession = Depends(get_session)) -> dict[str, Any]: - rows = list((await session.execute(select(Schedule).order_by(Schedule.created_at.desc()))).scalars()) + rows = list((await session.execute(select(Schedule).order_by(Schedule.created_at.desc(), Schedule.id.desc()))).scalars()) return {"items": await _schedule_outs(session, rows)} diff --git a/backend/app/api/settings.py b/backend/app/api/settings.py index e3217874..a1ba2b51 100644 --- a/backend/app/api/settings.py +++ b/backend/app/api/settings.py @@ -134,7 +134,7 @@ def _to_out(row: Provider) -> ProviderOut: @router.get("/providers", response_model=list[ProviderOut]) async def list_providers(session: AsyncSession = Depends(get_session)) -> list[ProviderOut]: - rows = (await session.execute(select(Provider).order_by(Provider.created_at))).scalars() + rows = (await session.execute(select(Provider).order_by(Provider.created_at, Provider.id))).scalars() return [_to_out(r) for r in rows] diff --git a/backend/app/api/shares.py b/backend/app/api/shares.py index 27006a5a..5f0bd679 100644 --- a/backend/app/api/shares.py +++ b/backend/app/api/shares.py @@ -206,7 +206,7 @@ async def list_shares(app_id: str, session: AsyncSession = Depends(get_session), app, _ = await _managed(session, app_id, x_actor) rows = list((await session.execute(select(ViewShare).where(ViewShare.app_id == app.id) - .order_by(ViewShare.created_at.desc()))).scalars()) + .order_by(ViewShare.created_at.desc(), ViewShare.id.desc()))).scalars()) version = await _previous(session, app) titles = _titles(version.spec if version is not None else None) return {"enabled": await _enabled(session), "embed_origins": bool(settings.embed_origins), diff --git a/backend/app/api/source_changes.py b/backend/app/api/source_changes.py index e899b1e2..f3f5c44f 100644 --- a/backend/app/api/source_changes.py +++ b/backend/app/api/source_changes.py @@ -79,7 +79,7 @@ async def upkeep(session: AsyncSession = Depends(get_session)) -> dict[str, Any] counts = await sc.counts_by_source(session, ids) schedules: dict[str, list[dict[str, Any]]] = {} for r in (await session.execute(select(Schedule).where(Schedule.target_kind == "source_maintenance") - .order_by(Schedule.created_at))).scalars(): + .order_by(Schedule.created_at, Schedule.id))).scalars(): schedules.setdefault(r.target_id or "", []).append({ "id": r.id, "enabled": r.enabled, "cron": r.cron, "timezone": r.timezone, "description_text": _describe(r.cron, r.timezone), @@ -249,7 +249,7 @@ async def get_maintenance(source_id: str, session: AsyncSession = Depends(get_se row = await _source(session, source_id) rows = list((await session.execute(select(Schedule).where( Schedule.target_kind == "source_maintenance", Schedule.target_id == row.id) - .order_by(Schedule.created_at))).scalars()) + .order_by(Schedule.created_at, Schedule.id))).scalars()) outs = await _schedule_outs(session, rows) fire_ids = [o["last_fire"]["id"] for o in outs if o.get("last_fire")] inputs = dict((await session.execute(select(ScheduleFire.id, ScheduleFire.inputs).where( diff --git a/backend/app/api/workflows.py b/backend/app/api/workflows.py index 6355d69f..a536b491 100644 --- a/backend/app/api/workflows.py +++ b/backend/app/api/workflows.py @@ -131,7 +131,7 @@ async def _workflow_out(session: AsyncSession, workflow: Workflow) -> WorkflowOu @router.get("", response_model=list[WorkflowOut]) async def list_workflows(session: AsyncSession = Depends(get_session)) -> list[WorkflowOut]: rows = list( - (await session.execute(select(Workflow).order_by(Workflow.updated_at.desc()))).scalars() + (await session.execute(select(Workflow).order_by(Workflow.updated_at.desc(), Workflow.id.desc()))).scalars() ) counts = await _run_counts(session) levels = await _published_levels(session) diff --git a/backend/app/apps/access.py b/backend/app/apps/access.py index c3121f13..27a9b4f5 100644 --- a/backend/app/apps/access.py +++ b/backend/app/apps/access.py @@ -72,7 +72,7 @@ def role_of(principal: Principal, app: Any, grants: list[Any]) -> str | None: async def grants_of(session: AsyncSession, app_id: str) -> list[AppGrant]: return list((await session.execute(select(AppGrant).where(AppGrant.app_id == app_id) - .order_by(AppGrant.created_at))).scalars()) + .order_by(AppGrant.created_at, AppGrant.id))).scalars()) async def app_role(session: AsyncSession, principal: Principal, app: Any) -> str | None: diff --git a/backend/app/apps/derive.py b/backend/app/apps/derive.py index 831bffe6..307b96df 100644 --- a/backend/app/apps/derive.py +++ b/backend/app/apps/derive.py @@ -65,7 +65,7 @@ async def derive_app(session: AsyncSession, workflow_id: str, version: int, *, n state = PinState(alias=PIN, pin={"workflow_id": workflow_id, "version": version}, workflow=workflow, row=row) schedule = (await session.execute(select(Schedule).where( Schedule.workflow_id == workflow_id, Schedule.run_class == "formal", Schedule.enabled.is_(True)) - .order_by(Schedule.created_at))).scalars().first() + .order_by(Schedule.created_at, Schedule.id))).scalars().first() return await _derive(session, row.graph or {}, title=workflow.name, name=name, description=workflow.description or "", pin={"alias": PIN, "workflow_id": workflow_id, "version": version, "graph_hash": state.hash, diff --git a/backend/app/apps/resolve.py b/backend/app/apps/resolve.py index d04ee995..d20bb73a 100644 --- a/backend/app/apps/resolve.py +++ b/backend/app/apps/resolve.py @@ -163,7 +163,7 @@ async def sealed_runs(session: AsyncSession, workflow_id: Any, version: Any, gra or_(Run.origin_kind.is_(None), Run.origin_kind.in_(SEALED_ORIGINS))) if isinstance(origin, str) and origin.startswith("schedule:"): query = query.where(Run.schedule_id == origin[len("schedule:"):]) - rows = list((await session.execute(query.order_by(Run.finished_at.desc(), Run.created_at.desc()) + rows = list((await session.execute(query.order_by(Run.finished_at.desc(), Run.created_at.desc(), Run.id.desc()) .limit(max(n, 1) * 3 + 5))).scalars()) out: list[Run] = [] for run in rows: @@ -213,7 +213,7 @@ async def trend_runs(session: AsyncSession, app_id: str | None, workflow_id: Any rows = (await session.execute(select(Run).where( Run.app_id == app_id, Run.origin_kind == BACKFILL_ORIGIN, Run.workflow_id == workflow_id, Run.version == version, Run.version_hash == graph_hash, Run.run_class == "formal", - Run.status == "succeeded").order_by(Run.created_at.desc()).limit(max(n, 1) * 3 + 5))).scalars() + Run.status == "succeeded").order_by(Run.created_at.desc(), Run.id.desc()).limit(max(n, 1) * 3 + 5))).scalars() seen = {input_key(r.input) for r in own} for run in rows: key = input_key(run.input) @@ -1064,7 +1064,7 @@ async def _task_states(session: AsyncSession, run: Run) -> dict[str, Any]: from app.db.models_process import Task rows = (await session.execute(select(Task.node_id, Task.status).where(Task.run_id == run.id) - .order_by(Task.created_at))).all() + .order_by(Task.created_at, Task.id))).all() return {str(node): {"status": status} for node, status in rows if node} @@ -1525,7 +1525,7 @@ async def _trails(session: AsyncSession, runs: list[Run]) -> dict[str, _Trail]: if not ids: return out for row in (await session.execute(select(Task.run_id, Task.node_id, Task.kind, Task.status, Task.outcome) - .where(Task.run_id.in_(ids)).order_by(Task.created_at))).all(): + .where(Task.run_id.in_(ids)).order_by(Task.created_at, Task.id))).all(): out[row.run_id].tasks.append(row) for row in (await session.execute(select(ActionRecord).where(ActionRecord.run_id.in_(ids)) .order_by(ActionRecord.created_at, ActionRecord.attempt))).scalars(): @@ -1761,7 +1761,7 @@ async def list_instances(session: AsyncSession, app_id: str, *, mine: str | None query = query.where(Run.started_by == mine) if status: query = query.where(Run.status == status) - rows = list((await session.execute(query.order_by(Run.created_at.desc()).limit(max(1, min(limit, 200)) * 2))) + rows = list((await session.execute(query.order_by(Run.created_at.desc(), Run.id.desc()).limit(max(1, min(limit, 200)) * 2))) .scalars()) if entry: rows = [r for r in rows if (r.origin or {}).get("entry") == entry] @@ -1912,7 +1912,7 @@ async def _task_list(session: AsyncSession, ctx: _Page, props: dict[str, Any]) - | ((TaskAssignment.principal_kind == "user") & (TaskAssignment.principal == (p.username or ""))) | ((TaskAssignment.principal_kind == "group") & TaskAssignment.principal.in_(list(p.groups) or [""]))) query = query.where(Task.id.in_(mine)) - rows = list((await session.execute(query.order_by(Task.created_at.desc()).limit(100))).scalars()) + rows = list((await session.execute(query.order_by(Task.created_at.desc(), Task.id.desc()).limit(100))).scalars()) runs = {run_id: str((origin or {}).get("title") or name or "") for run_id, origin, name in (await session.execute( select(Run.id, Run.origin, Run.workflow_name).where(Run.id.in_({t.run_id for t in rows})))).all()} return {"scope": scope, "items": [_task_row(t, runs.get(t.run_id)) for t in rows]} @@ -1970,11 +1970,11 @@ async def _process_data(session: AsyncSession, ctx: _Page, ctype: Any, kind: str return {"state": "empty"} if kind == "task": task = (await session.execute(select(Task).where(Task.run_id == run.id, Task.node_id == node_id) - .order_by(Task.created_at.desc()))).scalars().first() + .order_by(Task.created_at.desc(), Task.id.desc()))).scalars().first() if task is None: return {"state": "empty", "node": node_id} rows = list((await session.execute(select(TaskAssignment).where(TaskAssignment.task_id == task.id) - .order_by(TaskAssignment.created_at))).scalars()) + .order_by(TaskAssignment.created_at, TaskAssignment.id))).scalars()) reader = principal_with_actor(ctx.principal, ctx.actor) resolved_at = _aware(task.resolved_at).isoformat() if task.resolved_at else None signoffs = ctype == "task_form" and any( diff --git a/backend/app/auth/keys.py b/backend/app/auth/keys.py index 0739e135..b4d82771 100644 --- a/backend/app/auth/keys.py +++ b/backend/app/auth/keys.py @@ -551,7 +551,7 @@ async def started_before(principal: Any, idem_key: str) -> str | None: return await session.scalar(select(Run.id).where( Run.origin_kind == "app_entry", Run.started_by == principal.username, json_eq(Run.origin, "idempotency_key", idem_key), - json_eq(Run.origin, "via", "key")).order_by(Run.created_at.desc()).limit(1)) + json_eq(Run.origin, "via", "key")).order_by(Run.created_at.desc(), Run.id.desc()).limit(1)) # ---------------------------------------------------------------- 请求记录（A6） @@ -660,7 +660,8 @@ async def recent_requests(session: Any, key_id: str, limit: int = 100) -> list[d except Exception: # noqa: BLE001 - 写不进去只是少几条最新的，照常给库里有的 logger.exception("访问密钥的请求记录写库失败") rows = (await session.execute(select(AccessKeyRequest).where(AccessKeyRequest.key_id == key_id) - .order_by(AccessKeyRequest.at.desc()).limit(max(1, min(limit, 200))))).scalars() + .order_by(AccessKeyRequest.at.desc(), AccessKeyRequest.id.desc()) + .limit(max(1, min(limit, 200))))).scalars() return [{"at": r.at.isoformat() if r.at else None, "method": r.method, "route": r.route, "status": r.status, "run_id": r.run_id, "ip": r.ip} for r in rows] diff --git a/backend/app/blueprints/personal.py b/backend/app/blueprints/personal.py index 4eee74b2..99c1b5e2 100644 --- a/backend/app/blueprints/personal.py +++ b/backend/app/blueprints/personal.py @@ -432,7 +432,7 @@ async def latest_refresh(session: AsyncSession, app_id: str, graph_hash: str, *, query = _runs_of(app_id, graph_hash) if active_only: query = query.where(Run.status.in_(("queued", "running", "interrupted"))) - return (await session.execute(query.order_by(Run.created_at.desc()).limit(1))).scalars().first() + return (await session.execute(query.order_by(Run.created_at.desc(), Run.id.desc()).limit(1))).scalars().first() async def exploratory_runs(session: AsyncSession, app_id: str, graph_hash: str | None, n: int, @@ -448,7 +448,7 @@ async def exploratory_runs(session: AsyncSession, app_id: str, graph_hash: str | query = _runs_of(app_id, graph_hash).where(Run.status == "succeeded") if owner: query = query.where(Run.started_by == owner) - rows = list((await session.execute(query.order_by(Run.finished_at.desc(), Run.created_at.desc()) + rows = list((await session.execute(query.order_by(Run.finished_at.desc(), Run.created_at.desc(), Run.id.desc()) .limit(max(n, 1) * 3 + 5))).scalars()) out = [] for run in rows: @@ -471,7 +471,7 @@ async def review_task(session: AsyncSession, blueprint_id: str, *, pending: bool query = select(Task).where(Task.kind == REVIEW, Task.correlation_key == f"{REVIEW_KEY}{blueprint_id}") if pending: query = query.where(Task.status == "pending") - return (await session.execute(query.order_by(Task.created_at.desc()).limit(1))).scalars().first() + return (await session.execute(query.order_by(Task.created_at.desc(), Task.id.desc()).limit(1))).scalars().first() async def _review_targets(session: AsyncSession, to: Any) -> list[tuple[str, str]]: @@ -571,7 +571,7 @@ async def _copy_by(session: AsyncSession, blueprint_id: str, who: str | None) -> query = select(Blueprint).where(Blueprint.copied_from == blueprint_id) query = query.where(Blueprint.owner == who) if who else query.where(Blueprint.owner.is_(None)) - return (await session.execute(query.order_by(Blueprint.created_at.desc()).limit(1))).scalars().first() + return (await session.execute(query.order_by(Blueprint.created_at.desc(), Blueprint.id.desc()).limit(1))).scalars().first() async def review_published(session: AsyncSession, blueprint: Any, *, app_id: str, principal: Any) -> str | None: @@ -611,7 +611,7 @@ async def settle_review(session: AsyncSession, original: Any) -> str | None: return None published = (await session.execute(select(Blueprint).where( Blueprint.copied_from == original.id, Blueprint.status == "published", Blueprint.app_id.is_not(None)) - .order_by(Blueprint.updated_at.desc()).limit(1))).scalars().first() + .order_by(Blueprint.updated_at.desc(), Blueprint.id.desc()).limit(1))).scalars().first() if published is None or await review_task(session, original.id) is None: return None publisher = SimpleNamespace(username=published.owner, verified=False, groups=(), mode="local", @@ -670,7 +670,7 @@ async def review_payload(session: AsyncSession, task: Any) -> dict[str, Any]: .order_by(BlueprintEvent.seq.desc()).limit(1))).scalars().first() copies = (await session.execute(select(Blueprint.id, Blueprint.owner, Blueprint.status, Blueprint.app_id) .where(Blueprint.copied_from == blueprint.id) - .order_by(Blueprint.created_at))).all() + .order_by(Blueprint.created_at, Blueprint.id))).all() return {"kind": REVIEW, "blueprint_id": blueprint.id, "app_id": task.app_id, "name": blueprint.name, "purpose": _purpose(blueprint), "template_id": blueprint.template_id, "submitted_by": submitted.actor if submitted is not None else blueprint.owner, diff --git a/backend/app/chat/questions.py b/backend/app/chat/questions.py index 54d56f50..aaa7efd5 100644 --- a/backend/app/chat/questions.py +++ b/backend/app/chat/questions.py @@ -108,13 +108,13 @@ async def recent_questions(session: AsyncSession, principal: Any, *, limit: int) .join(Conversation, Conversation.id == ConversationTurn.conversation_id) .outerjoin(Run, Run.id == ConversationTurn.run_id) .where(Conversation.kind == "chat", func.trim(ConversationTurn.question) != "", counted_runs(), not_(hidden)) - .order_by(ConversationTurn.created_at.desc()).limit(limit) + .order_by(ConversationTurn.created_at.desc(), ConversationTurn.id.desc()).limit(limit) )).all() runs = (await session.execute( select(Run.id, Run.input, Run.workflow_name, Run.created_at, Run.origin_kind) .where(Run.run_class != "formal", func.coalesce(Run.origin_kind, "") != "chat", counted_runs(), not_(hidden), json_is_string(Run.input, "question")) - .order_by(Run.created_at.desc()).limit(limit) + .order_by(Run.created_at.desc(), Run.id.desc()).limit(limit) )).all() out = [Question(text=question.strip(), at=at, run_id=run_id, turn_id=turn_id, workflow=workflow, origin="chat") for turn_id, question, at, run_id, workflow in turns] diff --git a/backend/app/cluster/requests.py b/backend/app/cluster/requests.py index fc5fd42f..09332874 100644 --- a/backend/app/cluster/requests.py +++ b/backend/app/cluster/requests.py @@ -69,7 +69,7 @@ async def _drain() -> int: me = identity.owner() async with SessionLocal() as session: rows = list((await session.execute(select(ClusterRequest).where(ClusterRequest.owner == me) - .order_by(ClusterRequest.created_at))).scalars()) + .order_by(ClusterRequest.created_at, ClusterRequest.id))).scalars()) if not rows: return 0 await session.execute(delete(ClusterRequest).where(ClusterRequest.id.in_([r.id for r in rows]))) diff --git a/backend/app/cluster/workers.py b/backend/app/cluster/workers.py index d3b7066a..db9dd624 100644 --- a/backend/app/cluster/workers.py +++ b/backend/app/cluster/workers.py @@ -135,7 +135,7 @@ async def listing() -> list[dict[str, Any]]: now = utcnow() async with SessionLocal() as session: - rows = (await session.execute(select(ClusterWorker).order_by(ClusterWorker.started_at))).scalars().all() + rows = (await session.execute(select(ClusterWorker).order_by(ClusterWorker.started_at, ClusterWorker.id))).scalars().all() return [{"id": r.id, "host": r.host, "pid": r.pid, "alive": r.lease_until > now, "me": r.id == identity.owner(), "started_at": r.started_at.isoformat() if r.started_at else None, "heartbeat_at": r.heartbeat_at.isoformat() if r.heartbeat_at else None} for r in rows] diff --git a/backend/app/core/atomic.py b/backend/app/core/atomic.py index 732ef089..8a3eb8e4 100644 --- a/backend/app/core/atomic.py +++ b/backend/app/core/atomic.py @@ -4,7 +4,8 @@ 看不到写了一半的；进程写到一半没了，留下的是一个没人认的临时文件，不是半个正式文件。临时文件名带随机后缀： 两个进程同时写同一个文件（内容寻址的工件撞了同一份内容）各写各的，不会一个把另一个写到一半的截断。 -- write_bytes / write_text：写完 os.replace 到目标（目标已有就替换）； +- write_bytes / write_text：写完 os.replace 到目标（目标已有就替换，沿用它的权限）； +- append_bytes：追加写，不走改名（理由见函数说明）； - create_bytes / create_text：只在目标还没有时建（os.link，已有就不动、返回 False）——给只该生成一次的东西用 （加密密钥、签名私钥）：两个进程同时第一次启动，后到的读先到的那一份，不会各拿一把。 @@ -47,8 +48,25 @@ def _write_tmp(path: Path, data: bytes, *, mode: int | None, fsync: bool) -> Pat return tmp +def fsync_dir(folder: Path) -> None: + """把目录项（新建、改名）落盘。有的平台、文件系统打不开目录或不支持，就算了（同 data/raw_store.fsync_dir）。""" + try: + fd = os.open(folder, os.O_RDONLY) + except OSError: + return + try: + os.fsync(fd) + except OSError: + pass + finally: + os.close(fd) + + def write_bytes(path: Path, data: bytes, *, mode: int | None = None, fsync: bool = False) -> None: - """原子地写（替换）path。""" + """原子地写（替换）path。没给 mode、原来已有这个文件时沿用它的权限（改名换掉的是文件本身，权限不跟着走）。""" + if mode is None: + with contextlib.suppress(OSError): + mode = path.stat().st_mode & 0o7777 tmp = _write_tmp(path, data, mode=mode, fsync=fsync) try: os.replace(tmp, path) @@ -64,16 +82,57 @@ def write_text(path: Path, text: str, *, encoding: str = "utf-8", mode: int | No def create_bytes(path: Path, data: bytes, *, mode: int | None = None, fsync: bool = True) -> bool: - """path 还没有时原子地建出来，返回 True；已经有了（别人先建了）什么都不动，返回 False。""" + """path 还没有时原子地建出来，返回 True；已经有了（别人先建了）什么都不动，返回 False。建成后把目录项也落盘。 + + 硬链接不了的文件系统（exFAT、一部分 SMB / FUSE 挂载报 EPERM、ENOTSUP、EXDEV）退回 O_CREAT | O_EXCL 直接建目标文件： + 仍然只有一个进程建得成，但写的那一小会儿别的进程可能读到不完整的内容——只在头一次生成时有这个窗口。""" tmp = _write_tmp(path, data, mode=mode, fsync=fsync) try: os.link(tmp, path) - return True except FileExistsError: return False + except OSError: + return _create_directly(path, data, mode=mode, fsync=fsync) finally: with contextlib.suppress(OSError): tmp.unlink() + fsync_dir(path.parent) + return True + + +def _create_directly(path: Path, data: bytes, *, mode: int | None, fsync: bool) -> bool: + try: + fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o666 if mode is None else 0o600) + except FileExistsError: + return False + try: + with os.fdopen(fd, "wb") as fh: + fh.write(data) + fh.flush() + if fsync: + os.fsync(fh.fileno()) + if mode is not None: + os.chmod(path, mode) + except BaseException: + with contextlib.suppress(OSError): + path.unlink() + raise + fsync_dir(path.parent) + return True + + +def append_bytes(path: Path, data: bytes, *, fsync: bool = False) -> None: + """追加写（O_APPEND）：不换文件（inode、权限、别人手里开着的句柄都不变），原有内容一个字节都不动。 + + 不走「临时文件再改名」：那得把整份读出来重写，文件越大越慢，还会换掉 inode 和权限。代价是追加到一半时读的一方可能 + 看到半截新内容——它看到的永远是「原有的整份 + 新内容的一个前缀」，原有的内容不会丢。""" + path.parent.mkdir(parents=True, exist_ok=True) + fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_APPEND, 0o666) + with os.fdopen(fd, "ab") as fh: + fh.write(data) + fh.flush() + if fsync: + os.fsync(fh.fileno()) def create_text(path: Path, text: str, *, encoding: str = "utf-8", mode: int | None = None) -> bool: diff --git a/backend/app/core/config.py b/backend/app/core/config.py index 8d5a37fc..bdb311f7 100644 --- a/backend/app/core/config.py +++ b/backend/app/core/config.py @@ -31,6 +31,10 @@ def sanitize_session(session_id: str | None) -> str: return safe[:_SESSION_MAX] or "default" +#: 数据目录里的标记文件（Settings.ensure_dirs 建，deploy/deploy.sh 认） +DATA_DIR_MARKER = ".agentlab-data" + + class Settings(BaseSettings): """全局配置。所有项都可以用 AGENTLAB_ 前缀的环境变量覆盖。""" @@ -327,6 +331,16 @@ class Settings(BaseSettings): def ensure_dirs(self) -> None: for d in (self.data_dir, self.workspace_dir, self.uploads_dir): d.mkdir(parents=True, exist_ok=True) + # 认得出「这是 AgentLab 的数据目录」的标记：应用库放在 PostgreSQL 时这里没有 agentlab.db，设了 AGENTLAB_SECRET_KEY + # 时也没有 .secret_key。deploy.sh 改属主前按它认，免得 DATA_DIR 写错把别人的目录 chown 了。只建一次，不覆盖 + marker = self.data_dir / DATA_DIR_MARKER + if not marker.exists(): + from app.core import atomic + + try: + atomic.create_text(marker, "AgentLab 的数据目录（deploy.sh 按这个文件认，别删）\n") + except OSError: + pass @lru_cache diff --git a/backend/app/core/errors.py b/backend/app/core/errors.py index a64cf211..01355678 100644 --- a/backend/app/core/errors.py +++ b/backend/app/core/errors.py @@ -36,8 +36,8 @@ _UNREACHABLE = ("connection refused", "nodename nor servname", "name or service "all connection attempts failed", "network is unreachable") #: SQLAlchemy 包过的驱动异常开头是「(sqlite3.OperationalError) 」；asyncpg（PostgreSQL）的后面还跟着 -#: 「: 」，一起去掉 -_WRAPPED = re.compile(r"^\([\w.]+\)\s*(?::\s*)?") +#: 「: 」，一起去掉（直接拿 e.orig 的也只剩后一段，同样去掉） +_WRAPPED = re.compile(r"^(?:\([\w.]+\)\s*)?(?::\s*)?") # 内部符号：`a.b.._run()`、``、模块路径。对使用者没有意义 _LOCALS = re.compile(r"[\w.]*\.?[\w.]*(\(\))?") _REPR = re.compile(r"<(?:function|bound method|coroutine|object) [^>]*>") diff --git a/backend/app/data/catalog_jobs.py b/backend/app/data/catalog_jobs.py index fbff46b3..9a0826b3 100644 --- a/backend/app/data/catalog_jobs.py +++ b/backend/app/data/catalog_jobs.py @@ -142,7 +142,7 @@ async def active_job(session: AsyncSession, source_id: str) -> CatalogDraftJob | """排队或进行中的那个（同一个源最多一个）。""" return (await session.execute( select(CatalogDraftJob).where(CatalogDraftJob.source_id == source_id, CatalogDraftJob.status.in_(ACTIVE)) - .order_by(CatalogDraftJob.created_at.desc()).limit(1) + .order_by(CatalogDraftJob.created_at.desc(), CatalogDraftJob.id.desc()).limit(1) )).scalar_one_or_none() @@ -150,7 +150,7 @@ async def latest_job(session: AsyncSession, source_id: str) -> CatalogDraftJob | """最近建的一个（任何状态）。""" return (await session.execute( select(CatalogDraftJob).where(CatalogDraftJob.source_id == source_id) - .order_by(CatalogDraftJob.created_at.desc()).limit(1) + .order_by(CatalogDraftJob.created_at.desc(), CatalogDraftJob.id.desc()).limit(1) )).scalar_one_or_none() @@ -288,7 +288,7 @@ async def last_drafts(session: AsyncSession, source_id: str) -> dict[str, dict[s 中断了、而且没有写进新内容（parts_written 为 0）时照样带着，界面据此继续提示留空的列；之后有起草写进了新内容（停止 前已经写了几段、或者正在写）就是 null：以前的留空列数可能已经不对了。不用模型的起草不改模型留空的列，不算。""" recent = (select(CatalogDraftJob.id).where(CatalogDraftJob.source_id == source_id) - .order_by(CatalogDraftJob.created_at.desc()).limit(LAST_DRAFT_JOBS)).scalar_subquery() + .order_by(CatalogDraftJob.created_at.desc(), CatalogDraftJob.id.desc()).limit(LAST_DRAFT_JOBS)).scalar_subquery() def ranked() -> Any: return func.row_number().over(partition_by=CatalogDraftTask.table_name, diff --git a/backend/app/data/catalog_profile_jobs.py b/backend/app/data/catalog_profile_jobs.py index e3f7ec15..6b25d9a7 100644 --- a/backend/app/data/catalog_profile_jobs.py +++ b/backend/app/data/catalog_profile_jobs.py @@ -127,7 +127,7 @@ async def active_job(session: AsyncSession, source_id: str) -> CatalogProfileJob return (await session.execute( select(CatalogProfileJob).where(CatalogProfileJob.source_id == source_id, CatalogProfileJob.status.in_(ACTIVE)) - .order_by(CatalogProfileJob.created_at.desc()).limit(1) + .order_by(CatalogProfileJob.created_at.desc(), CatalogProfileJob.id.desc()).limit(1) )).scalar_one_or_none() @@ -135,7 +135,7 @@ async def latest_job(session: AsyncSession, source_id: str) -> CatalogProfileJob """最近建的一个（任何状态）。""" return (await session.execute( select(CatalogProfileJob).where(CatalogProfileJob.source_id == source_id) - .order_by(CatalogProfileJob.created_at.desc()).limit(1) + .order_by(CatalogProfileJob.created_at.desc(), CatalogProfileJob.id.desc()).limit(1) )).scalar_one_or_none() diff --git a/backend/app/data/catalog_profile_observe.py b/backend/app/data/catalog_profile_observe.py index 0663312e..ad716a1c 100644 --- a/backend/app/data/catalog_profile_observe.py +++ b/backend/app/data/catalog_profile_observe.py @@ -505,7 +505,7 @@ async def observed_facts(session: AsyncSession, source_id: str, query = query.where(TableObservation.table_name.in_(wanted)) out: dict[str, dict[str, Any]] = {} seen: set[tuple[str, str, str]] = set() - for r in (await session.execute(query.order_by(TableObservation.at.desc()))).scalars(): + for r in (await session.execute(query.order_by(TableObservation.at.desc(), TableObservation.id.desc()))).scalars(): key = (r.table_name, r.kind, r.column_name) if key in seen: continue diff --git a/backend/app/data/introspect_jobs.py b/backend/app/data/introspect_jobs.py index 974f4e96..29fad631 100644 --- a/backend/app/data/introspect_jobs.py +++ b/backend/app/data/introspect_jobs.py @@ -104,13 +104,13 @@ def _iso(value: Any) -> str | None: async def active_job(session: AsyncSession, source_id: str) -> IntrospectJob | None: return (await session.execute( select(IntrospectJob).where(IntrospectJob.source_id == source_id, IntrospectJob.status.in_(ACTIVE)) - .order_by(IntrospectJob.created_at.desc()).limit(1))).scalar_one_or_none() + .order_by(IntrospectJob.created_at.desc(), IntrospectJob.id.desc()).limit(1))).scalar_one_or_none() async def latest_job(session: AsyncSession, source_id: str) -> IntrospectJob | None: return (await session.execute( select(IntrospectJob).where(IntrospectJob.source_id == source_id) - .order_by(IntrospectJob.created_at.desc()).limit(1))).scalar_one_or_none() + .order_by(IntrospectJob.created_at.desc(), IntrospectJob.id.desc()).limit(1))).scalar_one_or_none() async def load(session: AsyncSession, source_id: str, job_id: str) -> IntrospectJob | None: @@ -437,7 +437,7 @@ async def _record_scope_changes(session: AsyncSession, row: Any, job: Introspect select(IntrospectJob).where(IntrospectJob.source_id == job.source_id, IntrospectJob.id != job.id, IntrospectJob.started_at.is_not(None), IntrospectJob.created_at <= job.created_at) - .order_by(IntrospectJob.created_at.desc()).limit(1))).scalar_one_or_none() + .order_by(IntrospectJob.created_at.desc(), IntrospectJob.id.desc()).limit(1))).scalar_one_or_none() before = (previous.scope or {}).get("rules") if previous is not None else None after = (job.scope or {}).get("rules") if isinstance(before, dict) and isinstance(after, dict) and before != after: diff --git a/backend/app/data/source_maintenance.py b/backend/app/data/source_maintenance.py index f9c3c7a3..7c61dcc5 100644 --- a/backend/app/data/source_maintenance.py +++ b/backend/app/data/source_maintenance.py @@ -241,7 +241,7 @@ async def schedules_for(session: Any, source_id: str) -> list[dict[str, Any]]: rows = (await session.execute(select(Schedule).where( Schedule.target_kind == "source_maintenance", Schedule.target_id == source_id) - .order_by(Schedule.created_at))).scalars() + .order_by(Schedule.created_at, Schedule.id))).scalars() return [{"id": r.id, "name": r.name, "cron": r.cron, "timezone": r.timezone, "enabled": r.enabled, "next_run_at": r.next_run_at.isoformat() if r.next_run_at else None} for r in rows] diff --git a/backend/app/data/source_objects.py b/backend/app/data/source_objects.py index 5dfcab0e..9ff8392c 100644 --- a/backend/app/data/source_objects.py +++ b/backend/app/data/source_objects.py @@ -47,6 +47,7 @@ from app.core.errors import describe_exception, first_line from app.data.names import name_key from app.db.base import SessionLocal, utcnow from app.db.models_sources import SourceObject +from app.db.portable import contains_ci logger = logging.getLogger(__name__) @@ -879,9 +880,8 @@ async def search_index(session: AsyncSession, row: Any, query: str, *, limit: in return [], 0 key = name_key(text.rsplit(".", 1)[-1]) filt = (*_scope_filter(row.id, index_schema(row)), SourceObject.removed_at.is_(None), - # 名字比的是 name_key（ASCII 已转小写）；注释要不分大小写：SQLite 的 LIKE 本来就不分（ASCII），PostgreSQL 分 - or_(SourceObject.name_key.contains(key, autoescape=True), - SourceObject.comment.icontains(text, autoescape=True))) + # 名字比的是 name_key（ASCII 已转小写）；注释要不分大小写（db/portable.contains_ci：两种库都折 Unicode 的大小写） + or_(SourceObject.name_key.contains(key, autoescape=True), contains_ci(SourceObject.comment, text))) total = int((await session.execute(select(func.count()).select_from(SourceObject).where(*filt))).scalar() or 0) rows = (await session.execute( select(SourceObject.name, SourceObject.name_key, SourceObject.kind, SourceObject.comment, diff --git a/backend/app/data/table_versions.py b/backend/app/data/table_versions.py index 8f2f2fe4..cdb4e2dd 100644 --- a/backend/app/data/table_versions.py +++ b/backend/app/data/table_versions.py @@ -1996,7 +1996,7 @@ async def _release_raws_locked(session: AsyncSession, shas: set[str]) -> set[str async def _expire_stagings_locked(session: AsyncSession, now: datetime) -> list[str]: """（持锁）过期的暂存区 → expired、瘦身、删试运行库、放掉没人要的原件；结束超过 90 天的行删掉（提交）。""" expired = list((await session.execute( - select(ImportStaging).where(_expiry_due(now)).order_by(ImportStaging.created_at) + select(ImportStaging).where(_expiry_due(now)).order_by(ImportStaging.created_at, ImportStaging.id) )).scalars()) ids, shas = [s.id for s in expired], {s.raw_sha256 for s in expired} trials = [close_staging(s, "expired", now=now) for s in expired] @@ -2177,7 +2177,7 @@ async def purge_import_raw( stagings = list((await session.execute( select(ImportStaging).where( ImportStaging.raw_sha256 == sha, ImportStaging.status.in_(STAGING_OPEN), - ).order_by(ImportStaging.created_at) + ).order_by(ImportStaging.created_at, ImportStaging.id) )).scalars()) if sha else [] now = utcnow() record = { diff --git a/backend/app/db/base.py b/backend/app/db/base.py index a018ff05..ee3edaca 100644 --- a/backend/app/db/base.py +++ b/backend/app/db/base.py @@ -205,6 +205,7 @@ def make_engine(url: str | None = None, *, application_name: str = APPLICATION_N return made from app.db import postgres + _require_postgres_driver("asyncpg") pool: dict[str, Any] = ( {"poolclass": NullPool} if settings.db_pool_size <= 0 else { "pool_size": settings.db_pool_size, "max_overflow": max(settings.db_max_overflow, 0), @@ -215,9 +216,21 @@ def make_engine(url: str | None = None, *, application_name: str = APPLICATION_N connect_args={"server_settings": {"application_name": application_name, "timezone": "UTC"}}, **pool) postgres.install(made.sync_engine.dialect) + event.listen(made.sync_engine, "before_cursor_execute", postgres.strip_nul, retval=True) return made +def _require_postgres_driver(*modules: str) -> None: + """设了 AGENTLAB_DATABASE_URL 却没装 [postgres]：说清楚要装什么，不丢一个光秃秃的 ModuleNotFoundError。""" + import importlib.util + + missing = [m for m in modules if importlib.util.find_spec(m) is None] + if missing: + raise RuntimeError( + f"应用库设成了 PostgreSQL（AGENTLAB_DATABASE_URL），但没装它要的包（{'、'.join(missing)}）：" + "pip install -e 'backend[postgres]' -c backend/requirements.lock") + + def describe_database() -> str: """启动日志里的一句：用的是哪种库、在哪（不写密码）。""" from app.db.urls import describe @@ -364,7 +377,9 @@ _COLUMN_MIGRATIONS: list[tuple[str, str, str]] = [ ("schedule_fires", "eval_run_id", "VARCHAR(32)"), # 编排完善（docs/design/orchestration-round/00-总方案.md §4）：定时任务的触发方式。存量行都是 cron ("schedules", "trigger_kind", "VARCHAR(20) DEFAULT 'cron'"), - ("schedules", "trigger", "JSON"), + # 第一版补这一列时没带缺省值：在那之前建过定时任务的库里这一列是 NULL（读时 or {} 兜着），模型是 NOT NULL。 + # 现在带上缺省值；已经补过、留着 NULL 的由 _NOT_NULL_BACKFILLS 填上 + ("schedules", "trigger", "JSON DEFAULT '{}'"), # 第 2 波：服务账号（users.kind）；剖析上传表格源时的快照 ("users", "kind", "VARCHAR(20) DEFAULT 'person'"), ("catalog_profile_tasks", "snapshot_id", "VARCHAR(32)"), @@ -402,6 +417,42 @@ _INDEX_MIGRATIONS: list[tuple[str, str, str | tuple[str, ...]]] = [ ] +#: 补列时没带缺省值、存量行留着 NULL 的 NOT NULL 列：每次启动按模型的缺省值填上（UPDATE … WHERE 列 IS NULL，填完再跑 +#: 什么都不改）。两种库存的才一样，导入 PostgreSQL 时这一列也不会撞 NOT NULL。以后新加的 NOT NULL 列在迁移表里一律带 +#: DEFAULT（test_db_portable 守着），就不用往这里加 +_NOT_NULL_BACKFILLS: list[tuple[str, str]] = [("schedules", "trigger")] + + +def model_default(column: Any) -> tuple[bool, Any]: + """模型列的缺省值：(有没有, 值)。标量原样，无参的可调用（dict、list、utcnow 之类）调一次；要按执行上下文算的不算。""" + default = getattr(column, "default", None) + if default is None: + return False, None + if default.is_scalar: + return True, default.arg + if default.is_callable: + try: + return True, default.arg(None) + except Exception: # noqa: BLE001 - 要读执行上下文的缺省值：这里给不出 + return False, None + return False, None + + +def _backfill_not_null(conn: Any, columns: dict[str, set[str]]) -> None: + from sqlalchemy import Table, update + + for table_name, column_name in _NOT_NULL_BACKFILLS: + if column_name not in (columns.get(table_name) or set()): + continue + model = Base.metadata.tables[table_name].c[column_name] + known, value = model_default(model) + if not known: + continue + # 只带这一列的表：模型表上 updated_at 的 onupdate 会跟着改（这是补数据，不是改了这一行），老库也未必有那一列 + bare = Table(table_name, MetaData(), Column(column_name, model.type)) + conn.execute(update(bare).where(bare.c[column_name].is_(None)).values({column_name: value})) + + #: 迁移补建的唯一索引：(索引名, 表, 列, 部分索引的条件, 建之前的去重语句)。只在索引还不存在时先去重再建——存量库里可能 #: 已经有并发留下的重复行，直接建会失败。去重的规矩写在各自的语句里：多出的钩子行删掉（只留最早的那一行，它的密钥照旧 #: 能用），多出的助手对话归档（不删，历史还在）。条件和语句里的 {false} / {true} 按方言换成布尔的字面量（SQLite 是 0 / 1） @@ -470,6 +521,7 @@ def _migrate(conn) -> None: # pragma: no cover - 同步回调 conn.execute(text(dedupe.format(**literals))) conn.execute(text(f"CREATE UNIQUE INDEX {name} ON {table} ({', '.join(wanted)})" + (f" WHERE {where.format(**literals)}" if where else ""))) + _backfill_not_null(conn, columns) #: init_db 在 PostgreSQL 上拿的会话级咨询锁的号：几个进程同时启动时，建表和迁移一个一个来（同一个号别处别用） diff --git a/backend/app/db/portable.py b/backend/app/db/portable.py index 4d643d1d..054df487 100644 --- a/backend/app/db/portable.py +++ b/backend/app/db/portable.py @@ -2,12 +2,14 @@ 只给应用库（agentlab.db / AGENTLAB_DATABASE_URL）用；发给业务数据源的 SQL 不走这里。 -**按 JSON 列里的键筛选。** SQLite 用 json_extract：取出来是 SQL 标量（字符串是文本、数字是数字、true 是 1、 -没有这个键或值为 null 是 NULL），和 Python 值比是带类型的比较（数字 3 不等于字符串 "3"）。PostgreSQL 的 JSON 列是 -存原文的 json（db/base.PortableJSON），用 -> / ->> 取，要比较的只把取出来的那一小段转成 JSONB： +**按 JSON 列里的键筛选。** 两种库同一套语义，按 JSON 的类型比（SQLite 的 json_extract 单独用时不分：true 取出来 +是 1，和 1 比相等；PostgreSQL 的 JSON 列是存原文的 json，db/base.PortableJSON，比较时只把取出来的那一小段转成 JSONB）： -- json_eq：PostgreSQL 上按 JSONB 相等比（`(col -> 'k')::jsonb = '3'::jsonb`），同样带类型，和 SQLite 一致； -- json_text：只要文本（字符串原样、数字是它的写法），PostgreSQL 上是 ->>； +- json_eq(列, 键, 值)：字符串只等于同样的字符串；数只等于数（1 和 1.0 相等）；True / False 只等于 JSON 的 true / false + （不等于 1 / 0）；None 是「没有这个键或值为 null」。只收这几种标量； +- json_text：这个键的值当文本——字符串是它本身，数、true / false、数组、对象是它的 JSON 写法（同 PostgreSQL 的 ->>， + SQLite 上用 json_type 分开、非字符串取 -> 的写法），没有这个键或值为 null 是 NULL。所以数 1 的 json_text 是 '1'， + 两种库都是； - json_flag：标记位「真」（true、非零的数）：没有这个键、null、false、0 都算假； - json_is_string：这个键的值是不是字符串（SQLite 的 json_type 叫 text，PostgreSQL 的 jsonb_typeof 叫 string）。 @@ -16,6 +18,11 @@ **插入，撞了唯一键就什么都不做。** insert_ignore 按会话连着的库挑 sqlite / postgresql 的 insert，两边的 on_conflict_do_nothing 是同一个接口。 +**不分大小写地找子串。** contains_ci：SQLite 上是 lower(列) LIKE lower(模式)（同 SQLAlchemy 的 ilike；大小写怎么折 +看 SQLite 的构建，本机 conda 的 3.53 连 É/é、全角Ａ/ａ 都认）。PostgreSQL 上应用库的文本列是 COLLATE "C"（db/postgres.py）， +列上的 ILIKE、lower() 只认 ASCII 的大小写；所以模式带 COLLATE "default"，按建库时的区域设置折大小写（en_US.UTF-8、 +C.UTF-8 都认 Unicode；建库用的是纯 C 区域的话只认 ASCII）。用户输入里的 %、_、反斜杠按字面匹配。 + **撞的是哪个唯一约束。** unique_violation 认 IntegrityError：SQLite 的报错是「UNIQUE constraint failed: 表.列」， PostgreSQL（asyncpg）的是 23505，表名、DETAIL 里的「Key (列)=」分别带着。不靠约束名（命名规则改了也认得出）。 """ @@ -64,12 +71,15 @@ class json_text(FunctionElement[str]): class _JsonEq(FunctionElement[bool]): + """kind 是 text / number / true / false，写进语句（语句缓存按它区分）；true / false 不用绑定参数。""" + type = Boolean() inherit_cache = True name = "json_eq" - def __init__(self, column: Any, key: str, value: Any) -> None: - super().__init__(column, _key(key), bindparam(None, value, type_=_JsonScalar())) + def __init__(self, column: Any, key: str, kind: str, value: Any = None) -> None: + extra = [] if kind in ("true", "false") else [bindparam(None, value, type_=_JsonScalar())] + super().__init__(column, _key(key), literal_column(kind), *extra) class _JsonNull(FunctionElement[bool]): @@ -82,10 +92,18 @@ class _JsonNull(FunctionElement[bool]): def json_eq(column: Any, key: str, value: Any) -> Any: - """JSON 列顶层键的值等于 value（带类型的比较，见模块说明）。value 为 None 时是「没有这个键或值为 null」。 + """JSON 列顶层键的值等于 value（按 JSON 的类型比，见模块说明）。value 为 None 时是「没有这个键或值为 null」。 - None 用另一个表达式：语句缓存不按绑定参数的值区分，同一个表达式不能因为值是 None 编译成两样。""" - return _JsonNull(column, key) if value is None else _JsonEq(column, key, value) + 值的种类写进语句：语句缓存不按绑定参数的值区分，同一个表达式不能因为值的种类不同编译成两样。""" + if value is None: + return _JsonNull(column, key) + if isinstance(value, bool): + return _JsonEq(column, key, "true" if value else "false") + if isinstance(value, (int, float)): + return _JsonEq(column, key, "number", value) + if isinstance(value, str): + return _JsonEq(column, key, "text", value) + raise TypeError(f"json_eq 只比标量（字符串、数、布尔、None），不收 {type(value).__name__}") class json_flag(FunctionElement[bool]): @@ -120,14 +138,23 @@ def _parts(element: Any, compiler: Any, **kw: Any) -> tuple[str, str, list[Any]] @compiles(json_text) def _json_text_sqlite(element: Any, compiler: Any, **kw: Any) -> str: + # 同 PostgreSQL 的 ->>：字符串取它本身，别的取 JSON 写法（-> 要 SQLite 3.38 以上） col, key, _ = _parts(element, compiler, **kw) - return f"json_extract({col}, '$.{key}')" + path = f"'$.{key}'" + return (f"(CASE json_type({col}, {path}) WHEN 'text' THEN json_extract({col}, {path}) WHEN 'null' THEN NULL " + f"ELSE ({col} -> {path}) END)") + + +_SQLITE_KIND = {"text": "= 'text'", "number": "IN ('integer', 'real')", "true": "= 'true'", "false": "= 'false'"} @compiles(_JsonEq) def _json_eq_sqlite(element: Any, compiler: Any, **kw: Any) -> str: - col, key, (value,) = _parts(element, compiler, **kw) - return f"(json_extract({col}, '$.{key}') = {compiler.process(value, **kw)})" + col, key, (kind, *value) = _parts(element, compiler, **kw) + typed = f"json_type({col}, '$.{key}') {_SQLITE_KIND[kind.name]}" + if not value: + return f"({typed})" + return f"({typed} AND json_extract({col}, '$.{key}') = {compiler.process(value[0], **kw)})" @compiles(_JsonNull) @@ -164,8 +191,10 @@ def _json_text_pg(element: Any, compiler: Any, **kw: Any) -> str: @compiles(_JsonEq, "postgresql") def _json_eq_pg(element: Any, compiler: Any, **kw: Any) -> str: - col, key, (value,) = _parts(element, compiler, **kw) - return f"({_got(col, key)} = CAST({compiler.process(value, **kw)} AS JSONB))" + # JSONB 的相等本来就按类型：'1' = '1.0'，'1' ≠ 'true'，'"1"' ≠ '1' + col, key, (kind, *value) = _parts(element, compiler, **kw) + right = f"'{kind.name}'::jsonb" if not value else f"CAST({compiler.process(value[0], **kw)} AS JSONB)" + return f"({_got(col, key)} = {right})" @compiles(_JsonNull, "postgresql") @@ -187,6 +216,38 @@ def _json_is_string_pg(element: Any, compiler: Any, **kw: Any) -> str: return f"(jsonb_typeof({_got(col, key)}) = 'string')" +# ---- 不分大小写的子串 ---- + + +def _escape_like(text: str) -> str: + return text.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_") + + +class _ContainsCI(FunctionElement[bool]): + type = Boolean() + inherit_cache = True + name = "contains_ci" + + +def contains_ci(column: Any, text: str) -> Any: + """列里含有 text，不分大小写（见模块说明）。""" + return _ContainsCI(column, bindparam(None, f"%{_escape_like(text)}%", type_=String())) + + +@compiles(_ContainsCI) +def _contains_ci_sqlite(element: Any, compiler: Any, **kw: Any) -> str: + column, pattern = list(element.clauses) + return (f"(lower({compiler.process(column, **kw)}) LIKE lower({compiler.process(pattern, **kw)}) " + "ESCAPE '\\')") + + +@compiles(_ContainsCI, "postgresql") +def _contains_ci_pg(element: Any, compiler: Any, **kw: Any) -> str: + column, pattern = list(element.clauses) + return (f"({compiler.process(column, **kw)} ILIKE ({compiler.process(pattern, **kw)} COLLATE \"default\") " + "ESCAPE '\\')") + + # ---- 插入，撞了唯一键就跳过 ---- diff --git a/backend/app/db/postgres.py b/backend/app/db/postgres.py index 6a15ab24..3921c60b 100644 --- a/backend/app/db/postgres.py +++ b/backend/app/db/postgres.py @@ -12,13 +12,17 @@ VARCHAR(n) 超长直接报错、INTEGER 只有 32 位。模型里的 String(n) **文本怎么比大小、怎么排序。** SQLite 缺省按字节比（BINARY）；PostgreSQL 按建库时的区域设置（常见的 en_US.UTF-8 不分 大小写地先比字母、忽略标点），同一列排出来的先后、`key < '…'` 这种比较的结果都会变。应用库上的文本列建表时一律带 COLLATE "C"（按字节比），和 SQLite 一致，不看建库时选了什么区域；LIKE 'xx%' 这种前缀匹配也因此用得上普通索引。 -ILIKE、lower() 照旧按库的字符集分大小写。 +代价是列上的 ILIKE、lower() 也按 C 的规则，只认 ASCII 的大小写（É/é、全角Ａ/ａ 不认）：要不分大小写地找，用 +db/portable.contains_ci，模式带 COLLATE "default"，按建库时的区域设置折大小写。 **ORDER BY 里 NULL 的位置。** SQLite 升序 NULL 在前、降序 NULL 在后，PostgreSQL 正好相反。应用里两百多处排序， 可空的列（开始时间、到期时间、发布时间……）排出来的先后要和 SQLite 一致：可空列上没写 nulls_first / nulls_last 的 ASC 补 NULLS FIRST、DESC 补 NULLS LAST，ORDER BY 里直接写列名（隐含 ASC）的同样补上。不可空的列不动—— -补了会让索引用不上，又没有 NULL 可排。窗口函数 OVER (ORDER BY …) 里同样补。原生 text() 里写的 ORDER BY 不经过 -这里，要自己写明（db/base 的去重语句里就写了）。 +补了会让索引用不上，又没有 NULL 可排。窗口函数 OVER (ORDER BY …) 里同样补。 + +只认直接写的列（Column，含它的 asc() / desc()）。这几种不补，两种库排出来的先后可能不同，要紧的地方自己写明 +nulls_first / nulls_last：label 出来的聚合（max(...) 之类）、coalesce / case 这类表达式、外连接进来的表上的列（模型里 +不可空，外连接没连上时却是 NULL）、原生 text() 里的 ORDER BY（db/base 的去重语句里写明了）。 """ from __future__ import annotations @@ -83,6 +87,33 @@ class AppCompiler(PGCompiler_asyncpg): return text +def _has_nul(value: Any) -> bool: + return isinstance(value, str) and "\x00" in value + + +def _clean_row(params: Any) -> Any: + if isinstance(params, dict): + return {k: v.replace("\x00", "") if _has_nul(v) else v for k, v in params.items()} \ + if any(_has_nul(v) for v in params.values()) else params + if isinstance(params, (list, tuple)) and any(_has_nul(v) for v in params): + return type(params)(v.replace("\x00", "") if _has_nul(v) else v for v in params) + return params + + +def strip_nul(conn: Any, cursor: Any, statement: str, parameters: Any, context: Any, + executemany: bool) -> tuple[str, Any]: + """写进 PostgreSQL 之前去掉文本参数里的 NUL 字符（before_cursor_execute，retval=True）。 + + PostgreSQL 的 text / varchar 不收 \\x00（报 invalid byte sequence），SQLite 照存：一段带 NUL 的模型输出、一个从二进制 + 文件里粘出来的问题，在 SQLite 上写得进去，在 PostgreSQL 上整条写入失败。所以和 JSON 列（db/base.pg_json_dumps）一样 + 去掉。只看参数里的字符串，bytes（LargeBinary）不动；没有 NUL 的参数原样交回，不多一次拷贝。""" + if executemany: + if any(_clean_row(row) is not row for row in parameters): + return statement, [_clean_row(row) for row in parameters] + return statement, parameters + return statement, _clean_row(parameters) + + def install(dialect: Any) -> None: """装到应用库引擎的方言实例上。""" dialect.statement_compiler = AppCompiler diff --git a/backend/app/db/to_postgres.py b/backend/app/db/to_postgres.py index 4dec6a25..bb6093cc 100644 --- a/backend/app/db/to_postgres.py +++ b/backend/app/db/to_postgres.py @@ -23,7 +23,13 @@ pending writes 按条数核对，逐项打印；有来源的行目标上没有 **值的转换。** 读写都按模型的列类型走：时间是不带时区的 UTC、布尔是真布尔、JSON 原文照搬（SQL 的 NULL 和 JSON 的 null 分开，不混）。SQLite 不查类型、PostgreSQL 查：整数列里的整数值的浮点、数字字符串换成整数，文本列里的非文本换成 -字符串；文本里的 NUL 字符（PostgreSQL 不收）去掉，JSON 里的 NaN / Infinity 写成 null，各自计数打印。 +字符串；文本里的 NUL 字符（PostgreSQL 不收）去掉，JSON 里的 NaN / Infinity 写成 null；模型里 NOT NULL、有缺省值的列， +老库补列时没带缺省值留下的 NULL 填成缺省值（启动时的迁移在 SQLite 上也这么填）；各自计数打印。 + +**孤儿行。** 外键指向已经不在的行（SQLite 当时没强制外键留下的）：外键是 SET NULL 的置空、计入转换；别的从快照里拿掉、 +不导入，按表列出条数，不算核对失败（settle_orphans）。 + +**自增主键。** 导入后序列接在已用过的最大号后面：目标表的 MAX、SQLite 的 AUTOINCREMENT 计数器（删过的号不复用）取大的。 """ from __future__ import annotations @@ -53,8 +59,10 @@ DEFAULT_BATCH = 1000 @dataclass class TableResult: name: str - #: 来源的行数 + #: 来源的行数（不含孤儿行） source: int = 0 + #: 外键指向已不存在的行、又不能置空的（外键是 CASCADE 或不可空）：SQLite 当时没强制外键留下的，没导入 + orphans: int = 0 #: 来源的这些主键，目标上导入前已经有的（--dry-run 时就是「不用补的」）、导入后有的 present_before: int = 0 present_after: int = 0 @@ -77,6 +85,8 @@ class Report: checkpoints_written: int = 0 threads: int = 0 fixed: dict[str, int] = field(default_factory=dict) + #: 没导入的孤儿行，按表（见 settle_orphans） + orphans: dict[str, int] = field(default_factory=dict) dry_run: bool = False @property @@ -139,6 +149,49 @@ def snapshot(db: Path, into: Path) -> Path: return out +def settle_orphans(path: Path, *, fixed: dict[str, int]) -> dict[str, int]: + """快照上按模型的外键找孤儿行：外键列有值、指向的行却不在（SQLite 当时没强制外键：老版本、或者那时外键还没加上）。 + PostgreSQL 一律拒收，重跑也补不上，所以导入前在快照上（来源不动）先处理： + + - 外键是 ON DELETE SET NULL、列可空的：置空，计入「转换」——父行被删时本来就该这样； + - 别的（CASCADE、不可空）：从快照里拿掉、不导入，按表计数交回——父行被删时本来就该跟着删。拿掉一行可能让下一层又成了 + 孤儿，反复扫到没有为止。 + + 这些不算核对失败：报告里照实列出哪张表、几行。""" + from sqlalchemy import and_, delete, exists, update + + from app.db import models # noqa: F401 + from app.db.base import Base + + orphans: dict[str, int] = {} + engine = create_sync_engine(f"sqlite:///{path}") + try: + while True: + removed = 0 + with engine.begin() as conn: + for table in Base.metadata.sorted_tables: + for fk in table.foreign_key_constraints: + parent = fk.referred_table.alias("parent") + pairs = [(el.parent, parent.c[el.column.name]) for el in fk.elements] + lost = and_(*(child.is_not(None) for child, _ in pairs), + ~exists(select(1).select_from(parent).where( + and_(*(p == child for child, p in pairs))))) + n = int(conn.execute(select(func.count()).select_from(table).where(lost)).scalar() or 0) + if not n: + continue + if (fk.ondelete or "").upper() == "SET NULL" and all(c.nullable for c, _ in pairs): + conn.execute(update(table).where(lost).values({c.name: None for c, _ in pairs})) + fixed["外键指向已不存在的行，置空"] = fixed.get("外键指向已不存在的行，置空", 0) + n + else: + conn.execute(delete(table).where(lost)) + orphans[table.name] = orphans.get(table.name, 0) + n + removed += n + if not removed: + return orphans + finally: + engine.dispose() + + def upgrade_snapshot(path: Path) -> None: """在快照上跑一遍启动时的建表补列：老版本的库按当前的表结构读。""" from app.db import models # noqa: F401 @@ -173,7 +226,7 @@ class _Fixer: def value(self, column: Any, value: Any) -> Any: if value is None: - return None + return self.missing(column) if _is_json(column): return self.json(value) kind = column.type @@ -201,6 +254,19 @@ class _Fixer: value = value.replace("\x00", "") return value + def missing(self, column: Any) -> Any: + """SQL 的 NULL。模型列 NOT NULL、又有缺省值的（老版本补列时没带缺省值，存量行留着 NULL；SQLite 的 ADD COLUMN + 不带 NOT NULL，照存）填成缺省值：PostgreSQL 的表照模型建，带 NOT NULL，不填就整行插不进。""" + from app.db.base import model_default, pg_json_dumps + + if column.nullable or column.primary_key: + return None + known, default = model_default(column) + if not known: + return None + self._count("NOT NULL 列的空值填成缺省值") + return pg_json_dumps(default) if _is_json(column) else default + def json(self, raw: Any) -> str: """JSON 原文 → 写进 PostgreSQL 的原文。平常的原样照搬（键的先后不变）；有 NaN / NUL 的才重写一遍。""" from app.db.base import pg_json_dumps @@ -252,7 +318,7 @@ async def _write_batch(target: Any, statement: Any, rows: list[dict[str, Any]], from app.core.errors import first_line ident = ", ".join(f"{k}={row.get(k)!r}" for k in keys) - refused.append(f"{ident}：{first_line(e.orig or e)}") + refused.append(f"{ident}：{first_line(e)}") def _present(table: Any, keys: list[tuple[Any, ...]]) -> Any: @@ -276,8 +342,12 @@ async def copy_tables(snapshot_db: Path, target: Any, *, batch: int, dry_run: bo source = create_sync_engine(f"sqlite:///{snapshot_db}") try: with source.connect() as src: + # AUTOINCREMENT 表的计数器（删过最后几行时比 MAX 大）：PostgreSQL 的序列要接在它后面，不复用删掉的号—— + # 召回按 source_changes.seq 当游标往后拉，复用了就跳过新变化 + counters = dict(src.execute(text("SELECT name, seq FROM sqlite_sequence")).all()) if src.execute(text( + "SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = 'sqlite_sequence'")).first() else {} for table in Base.metadata.sorted_tables: - result = TableResult(table.name) + result = TableResult(table.name, orphans=report.orphans.get(table.name, 0)) report.tables.append(result) result.source = int(src.execute(select(func.count()).select_from(table)).scalar() or 0) async with target.connect() as conn: @@ -306,7 +376,7 @@ async def copy_tables(snapshot_db: Path, target: Any, *, batch: int, dry_run: bo if dry_run: result.present_after, result.target_after = result.present_before, result.target_before continue - await _reset_sequences(target, table) + await _reset_sequences(target, table, int(counters.get(table.name) or 0)) async with target.connect() as conn: result.target_after = await _count(conn, table) finally: @@ -319,15 +389,22 @@ def _has_table(sync: Any, name: str) -> bool: return inspect(sync).has_table(name) -async def _reset_sequences(target: Any, table: Any) -> None: - """自增主键的序列接到已有的最大值后面：导入时写的是来源的主键值，序列自己没动过，不改的话下一条撞主键。""" +async def _reset_sequences(target: Any, table: Any, floor: int = 0) -> None: + """自增主键的序列接到已用过的最大号后面：导入时写的是来源的主键值，序列自己没动过，不改的话下一条撞主键。 + 已用过的取三者最大：目标表里的 MAX、SQLite 的 AUTOINCREMENT 计数器（floor）、序列现在的值（重跑时不往回拨）。""" for column in table.primary_key.columns: if column is not table.autoincrement_column: continue async with target.begin() as conn: - await conn.execute(text( - f"SELECT setval(pg_get_serial_sequence(:t, :c), GREATEST(COALESCE(MAX({column.name}), 0), 1), " - f"MAX({column.name}) IS NOT NULL) FROM {table.name}"), {"t": table.name, "c": column.name}) + name = (await conn.execute(text("SELECT pg_get_serial_sequence(:t, :c)"), + {"t": table.name, "c": column.name})).scalar() + if not name: + continue + top = int((await conn.execute(select(func.max(column)))).scalar() or 0) + last, called = (await conn.execute(text(f"SELECT last_value, is_called FROM {name}"))).one() + used = max(top, floor, int(last) if called else int(last) - 1) + await conn.execute(text("SELECT setval(:n, :v, :called)"), + {"n": name, "v": max(used, 1), "called": used >= 1}) async def services_on(target: Any) -> int: @@ -425,6 +502,7 @@ async def copy_to_postgres(source: str | Path, target_url: str, *, checkpoint_ur with tempfile.TemporaryDirectory(prefix="agentlab-copy-", dir=work_dir) as tmp: snap = snapshot(db, Path(tmp)) upgrade_snapshot(snap) + report.orphans = settle_orphans(snap, fixed=report.fixed) if not dry_run: await create_schema(target) await copy_tables(snap, target, batch=batch, dry_run=dry_run, report=report, progress=progress) @@ -451,8 +529,14 @@ def render(report: Report) -> list[str]: else: verdict = "一致" + (f"（这次补了 {t.present_after - t.present_before} 行）" if t.present_after > t.present_before else "（都已经有了）") + if t.orphans: + verdict += f"；另有 {t.orphans} 行孤儿行没导入" lines.append(f"{t.name:<28}{t.source:>10}{t.present_before:>10}{t.present_after:>10}{t.target_after:>10}" f" {verdict}") + if report.orphans: + total = sum(report.orphans.values()) + lines.append(f"孤儿行（SQLite 当时没强制外键，外键指向的行已经不在）：{total} 行没导入，不算核对失败——" + + "、".join(f"{name} {n} 行" for name, n in report.orphans.items())) for t in report.tables: for item in t.refused[:20]: lines.append(f" {t.name} 没插进去：{item}") diff --git a/backend/app/db/urls.py b/backend/app/db/urls.py index 62064ac9..cf7f16e1 100644 --- a/backend/app/db/urls.py +++ b/backend/app/db/urls.py @@ -15,6 +15,13 @@ _CHECKPOINT_SCHEMES = {"postgresql", "postgres", "postgresql+psycopg", "postgres #: asyncpg 和 libpq 对 SSL 的参数名不一样：asyncpg 是 ssl=require，libpq 是 sslmode=require（取值同一套） _ASYNCPG_TO_LIBPQ = {"ssl": "sslmode"} +#: libpq 写法的 SSL 证书参数：asyncpg 不收这几个连接参数（SQLAlchemy 原样交过去就是 TypeError），改用同名的环境变量 +_LIBPQ_SSL_FILES = {"sslrootcert": "PGSSLROOTCERT", "sslcert": "PGSSLCERT", "sslkey": "PGSSLKEY", + "sslcrl": "PGSSLCRL", "sslpassword": "PGSSLPASSWORD"} +_SSL_MODES = {"disable", "allow", "prefer", "require", "verify-ca", "verify-full"} +#: 应用库地址里认的参数。SQLAlchemy 把地址里的参数原样交给 asyncpg.connect，它不认的（application_name、connect_timeout +#: 这类 libpq 写法）连的时候才报 TypeError；这里在启动时就说清楚 +_APP_PARAMS = {"ssl", "prepared_statement_cache_size", "target_session_attrs", "async_fallback"} #: 只给 SQLAlchemy / asyncpg 看的参数，libpq 不认（psycopg 连的时候会报「invalid connection option」） _ASYNCPG_ONLY = {"prepared_statement_cache_size", "prepared_statement_name_func", "command_timeout", "async_fallback"} @@ -32,9 +39,26 @@ def _parse(raw: str, schemes: set[str], example: str) -> URL: def normalize_database_url(raw: str) -> str: - """AGENTLAB_DATABASE_URL → postgresql+asyncpg://…（密码原样保留）。认不出的写法抛 ValueError。""" + """AGENTLAB_DATABASE_URL → postgresql+asyncpg://…（密码原样保留）。认不出的写法抛 ValueError。 + + SSL：libpq 的 sslmode= 换成 asyncpg 的 ssl=（取值同一套：disable / allow / prefer / require / verify-ca / verify-full）； + 证书文件（sslrootcert 之类）asyncpg 不收连接参数，要用 PGSSLROOTCERT 等环境变量（asyncpg 和断点库的 libpq 都认）。""" url = _parse(raw, _APP_SCHEMES, "postgresql+asyncpg://用户:密码@主机:5432/库名") - return url.set(drivername="postgresql+asyncpg").render_as_string(hide_password=False) + query = dict(url.query) + if files := [k for k in query if k in _LIBPQ_SSL_FILES]: + raise ValueError("地址里不能写 " + "、".join(files) + "：asyncpg 不收这几个参数，改设环境变量 " + + "、".join(_LIBPQ_SSL_FILES[k] for k in files)) + if "sslmode" in query: + if "ssl" in query: + raise ValueError("ssl= 和 sslmode= 只写一个") + mode = str(query.pop("sslmode")) + if mode not in _SSL_MODES: + raise ValueError(f"认不出 sslmode={mode}（可选 {' / '.join(sorted(_SSL_MODES))}）") + query["ssl"] = mode + if other := sorted(k for k in query if k not in _APP_PARAMS): + raise ValueError("地址里的参数 " + "、".join(other) + " 不认（asyncpg 不收）；能写的只有 " + + "、".join(sorted(_APP_PARAMS | {"sslmode"}))) + return url.set(drivername="postgresql+asyncpg", query=query).render_as_string(hide_password=False) def normalize_checkpoint_url(raw: str) -> str: diff --git a/backend/app/engine/checkpoints.py b/backend/app/engine/checkpoints.py index 73ab7b7e..919ec46c 100644 --- a/backend/app/engine/checkpoints.py +++ b/backend/app/engine/checkpoints.py @@ -32,6 +32,9 @@ def _pool_size() -> int: async def open_postgres_saver(conninfo: str, *, max_size: int | None = None, application_name: str = "agentlab") -> AsyncIterator[BaseCheckpointSaver]: """PostgreSQL 的断点库：开连接池、建表（幂等），退出时关池子。application_name 同应用库（db/base.APPLICATION_NAME）。""" + from app.db.base import _require_postgres_driver + + _require_postgres_driver("psycopg", "psycopg_pool", "langgraph.checkpoint.postgres") from langgraph.checkpoint.postgres.aio import AsyncPostgresSaver from psycopg.rows import dict_row from psycopg_pool import AsyncConnectionPool diff --git a/backend/app/engine/nodes/tools.py b/backend/app/engine/nodes/tools.py index 3f15785b..17b3e257 100644 --- a/backend/app/engine/nodes/tools.py +++ b/backend/app/engine/nodes/tools.py @@ -345,7 +345,7 @@ async def _confirmed_by(run_id: str, node_id: str) -> str | None: async with SessionLocal() as session: return await session.scalar(select(Task.resolved_by).where( Task.run_id == run_id, Task.node_id == node_id, Task.kind == "action_confirm", - Task.status == "resolved").order_by(Task.created_at.desc()).limit(1)) + Task.status == "resolved").order_by(Task.created_at.desc(), Task.id.desc()).limit(1)) def _write_updates(ctx: NodeContext, result: dict[str, Any]) -> dict[str, Any]: diff --git a/backend/app/engine/spend.py b/backend/app/engine/spend.py index f709b4d1..a9d2eaf1 100644 --- a/backend/app/engine/spend.py +++ b/backend/app/engine/spend.py @@ -51,8 +51,7 @@ async def _locked(session: Any, key: str) -> Any: """占住设置表里 key 这一行再读它（见模块说明「多进程」）：没有这一行交回 None。updated_at 不动。""" from sqlalchemy import update - await session.execute(update(Setting).where(Setting.key == key).values(value=Setting.value, - updated_at=Setting.updated_at) + await session.execute(update(Setting).where(Setting.key == key).values(updated_at=Setting.updated_at) .execution_options(synchronize_session=False)) return await session.get(Setting, key, populate_existing=True) diff --git a/backend/app/eval/cleanup.py b/backend/app/eval/cleanup.py index 44a22f69..a52a947e 100644 --- a/backend/app/eval/cleanup.py +++ b/backend/app/eval/cleanup.py @@ -83,7 +83,7 @@ async def clean_suite(suite_id: str, *, keep: int | None = None) -> dict[str, An if keep is None: keep = int((await eval_settings(session))["keep_runs"]) runs = list((await session.execute(select(EvalRun).where(EvalRun.suite_id == suite_id) - .order_by(EvalRun.created_at.desc()))).scalars()) + .order_by(EvalRun.created_at.desc(), EvalRun.id.desc()))).scalars()) kept = keep_set(runs, keep) targets = [r for r in runs if r.id not in kept and r.cleaned_at is None] results = removed = 0 diff --git a/backend/app/eval/compare.py b/backend/app/eval/compare.py index 35d97996..bbf83118 100644 --- a/backend/app/eval/compare.py +++ b/backend/app/eval/compare.py @@ -53,7 +53,7 @@ async def pick_baseline(session: AsyncSession, run: Any, *, explicit: str | None rows = (await session.execute( select(EvalRun).where(EvalRun.suite_id == run.suite_id, EvalRun.id != run.id, EvalRun.created_at < run.created_at, EvalRun.status.in_(BASELINE_STATUSES)) - .order_by(EvalRun.created_at.desc()).limit(200))).scalars().all() + .order_by(EvalRun.created_at.desc(), EvalRun.id.desc()).limit(200))).scalars().all() same = [r for r in rows if (r.fingerprint or {}).get("variant_key") == key] pinned = next((r for r in same if r.pinned), None) return pinned or (same[0] if same else None) diff --git a/backend/app/eval/gold.py b/backend/app/eval/gold.py index b0219398..872aaf32 100644 --- a/backend/app/eval/gold.py +++ b/backend/app/eval/gold.py @@ -241,7 +241,7 @@ async def _artifacts(run_id: str, kind: str) -> list[tuple[str | None, tuple[str async with SessionLocal() as session: rows = (await session.execute(select(Artifact.id, Artifact.node_id).where( - Artifact.run_id == run_id, Artifact.kind == kind).order_by(Artifact.created_at))).all() + Artifact.run_id == run_id, Artifact.kind == kind).order_by(Artifact.created_at, Artifact.id))).all() out: list[tuple[str | None, tuple[str, Any]]] = [] for artifact_id, node_id in rows: try: diff --git a/backend/app/eval/runner.py b/backend/app/eval/runner.py index a7bb8ecf..441430d7 100644 --- a/backend/app/eval/runner.py +++ b/backend/app/eval/runner.py @@ -132,7 +132,7 @@ async def estimate(session: AsyncSession, suite_id: str, items: int) -> dict[str 平均记成 0、却有估不出的调用的，也按估不出算。""" row = (await session.execute(select(EvalRun).where( EvalRun.suite_id == suite_id, EvalRun.status.in_(("succeeded", "partial"))) - .order_by(EvalRun.created_at.desc()).limit(1))).scalar_one_or_none() + .order_by(EvalRun.created_at.desc(), EvalRun.id.desc()).limit(1))).scalar_one_or_none() if row is None: return {"items": items, "per_item_usd": None, "total_usd": None, "basis": None, "note": "这个题集还没有评完过，估不出花费：按每题开评前的实际花费累计，到了上限就停"} @@ -322,7 +322,7 @@ class EvalExecution(runtime.JobRun): async with SessionLocal() as session: leader = await session.get(EvalRun, self.job_id) runs = [leader] if not leader.pair_id else list((await session.execute( - select(EvalRun).where(EvalRun.pair_id == leader.pair_id).order_by(EvalRun.created_at))).scalars()) + select(EvalRun).where(EvalRun.pair_id == leader.pair_id).order_by(EvalRun.created_at, EvalRun.id))).scalars()) suite = await session.get(EvalSuite, leader.suite_id) if suite is None: raise runtime.JobFailed("题集不存在，可能已被删除") @@ -398,7 +398,7 @@ class EvalExecution(runtime.JobRun): unrestricted = True if unrestricted: ids += list((await session.execute(select(DataSource.id).where(DataSource.enabled.is_(True)) - .order_by(DataSource.created_at))).scalars()) + .order_by(DataSource.created_at, DataSource.id))).scalars()) return list(dict.fromkeys(ids)) # ---------------------------------------------------------------- 主循环 diff --git a/backend/app/ledger/certify.py b/backend/app/ledger/certify.py index 3f1d454c..7da2ea32 100644 --- a/backend/app/ledger/certify.py +++ b/backend/app/ledger/certify.py @@ -309,7 +309,7 @@ async def queue(session: Any, *, principal: Any) -> dict[str, Any]: kinds = {name: kind for name, kind in (await session.execute(select(DataSource.name, DataSource.kind))).all()} runs = list((await session.execute(select(Run).where( Run.origin_kind == "chat", Run.status == "succeeded", Run.run_class != "formal", counted_runs()) - .order_by(Run.created_at.desc()).limit(RUN_LIMIT))).scalars()) + .order_by(Run.created_at.desc(), Run.id.desc()).limit(RUN_LIMIT))).scalars()) turns = await _turns(session, runs) by_run = {r.id: r for r in runs} # 每次运行最新的一份比对（store.latest 按事件认这一次封存的；这里只要内容，按工件引用的时间取最新） diff --git a/backend/app/ledger/issue.py b/backend/app/ledger/issue.py index 77308a1c..e7c9181d 100644 --- a/backend/app/ledger/issue.py +++ b/backend/app/ledger/issue.py @@ -154,7 +154,7 @@ async def issued_runs(session: Any, run_id: str) -> list[Run]: （db/portable.json_eq，同 auth/keys 认幂等键的写法）。""" rows = (await session.execute(select(Run).where( Run.origin_kind == ORIGIN, json_eq(Run.origin, "source_run", run_id)) - .order_by(Run.created_at.desc()))).scalars() + .order_by(Run.created_at.desc(), Run.id.desc()))).scalars() return list(rows) @@ -319,7 +319,7 @@ async def _conversation_of(session: Any, run: Run) -> dict[str, str]: if origin.get("conversation_id") and origin.get("turn_id"): return {"conversation_id": str(origin["conversation_id"]), "turn_id": str(origin["turn_id"])} turn = (await session.execute(select(ConversationTurn).where(ConversationTurn.run_id == run.id) - .order_by(ConversationTurn.created_at.desc()))).scalars().first() + .order_by(ConversationTurn.created_at.desc(), ConversationTurn.id.desc()))).scalars().first() return {"conversation_id": turn.conversation_id, "turn_id": turn.id} if turn is not None else {} diff --git a/backend/app/ledger/match.py b/backend/app/ledger/match.py index 1761ec9b..7a1b2fa3 100644 --- a/backend/app/ledger/match.py +++ b/backend/app/ledger/match.py @@ -798,7 +798,7 @@ async def backfill(*, days: int = BACKFILL_DAYS, limit: int = BACKFILL_LIMIT) -> select(Run.id).where(Run.status == "succeeded", Run.run_class != "formal", Run.origin_kind.in_(EXPLORE_ORIGINS), Run.manifest_hash.is_not(None), Run.finished_at >= since) - .order_by(Run.finished_at.desc()).limit(limit))).all()] + .order_by(Run.finished_at.desc(), Run.id.desc()).limit(limit))).all()] done = await store.matched_run_ids(session, ids) count = 0 for run_id in ids: diff --git a/backend/app/ledger/recall.py b/backend/app/ledger/recall.py index 364b138d..5d87fb28 100644 --- a/backend/app/ledger/recall.py +++ b/backend/app/ledger/recall.py @@ -231,7 +231,7 @@ async def recall_runs(session: Any, workflow_id: str, new_version: int | None = json_eq(Run.origin, "workflow_id", workflow_id), not_(json_flag(Run.origin, "probe"))) if new_version is not None: stmt = stmt.where(json_eq(Run.origin, "to_version", int(new_version))) - return list((await session.execute(stmt.order_by(Run.created_at.desc()))).scalars()) + return list((await session.execute(stmt.order_by(Run.created_at.desc(), Run.id.desc()))).scalars()) async def build_plan(session: Any, workflow_id: str, new_version: int) -> Plan: @@ -1546,7 +1546,7 @@ async def _probe_calibers(change: Mapping[str, Any]) -> list[str]: continue latest = (await session.execute(select(Run.input).where( Run.workflow_id == caliber.workflow_id, Run.version == caliber.version, Run.run_class == "formal", - Run.status == "succeeded").order_by(Run.finished_at.desc()).limit(1))).scalar_one_or_none() + Run.status == "succeeded").order_by(Run.finished_at.desc(), Run.id.desc()).limit(1))).scalar_one_or_none() inputs_from = "formal" if latest else "defaults" if not latest: # 这一版还没有正式运行：退回最近一次等价于它的问数回答绑定出的参数，再没有才只用入口的缺省值 diff --git a/backend/app/ledger/reconcile.py b/backend/app/ledger/reconcile.py index e6c281d6..4de86bf5 100644 --- a/backend/app/ledger/reconcile.py +++ b/backend/app/ledger/reconcile.py @@ -147,7 +147,7 @@ async def reconciled_runs(session: Any, run_id: str) -> list[Run]: rows = (await session.execute(select(Run).where( Run.origin_kind == ORIGIN, json_eq(Run.origin, "source_run", run_id)) - .order_by(Run.created_at.desc()))).scalars() + .order_by(Run.created_at.desc(), Run.id.desc()))).scalars() return list(rows) diff --git a/backend/app/ledger/uses.py b/backend/app/ledger/uses.py index b473e110..523720b0 100644 --- a/backend/app/ledger/uses.py +++ b/backend/app/ledger/uses.py @@ -116,7 +116,7 @@ async def _answers(session: AsyncSession, workflow_id: str, below: int | None, o rows = (await session.execute( select(Artifact.run_id, Artifact.meta, Run.finished_at).join(Run, Run.id == Artifact.run_id) .where(Artifact.kind == store.KIND, Run.status == "succeeded", counted_runs()) - .order_by(Run.finished_at.desc()))).all() + .order_by(Run.finished_at.desc(), Run.id.desc()))).all() wanted: dict[str, datetime | None] = {} for run_id, meta, at in rows: keys = [k for k in ((meta or {}).get("calibers") or []) if isinstance(k, str) and k.startswith(prefix)] @@ -179,7 +179,7 @@ async def _chat_owners(session: AsyncSession, found: list[Use]) -> None: if missing: for turn_id, conv, rid in (await session.execute( select(ConversationTurn.id, ConversationTurn.conversation_id, ConversationTurn.run_id) - .where(ConversationTurn.run_id.in_(missing)).order_by(ConversationTurn.created_at))).all(): + .where(ConversationTurn.run_id.in_(missing)).order_by(ConversationTurn.created_at, ConversationTurn.id))).all(): turns.setdefault(rid, (conv, turn_id)) convs = sorted({c for c, _ in turns.values()}) owners = dict((await session.execute(select(Conversation.id, Conversation.created_by) @@ -219,7 +219,7 @@ async def _formal(session: AsyncSession, clause: Any, *, kind_of: Any) -> tuple[ rows = list((await session.execute( select(Run).where(clause, Run.run_class == "formal", Run.status == "succeeded", Run.manifest_hash.is_not(None), counted_runs()) - .order_by(Run.finished_at.desc()).limit(MAX_FORMAL + 1))).scalars()) + .order_by(Run.finished_at.desc(), Run.id.desc()).limit(MAX_FORMAL + 1))).scalars()) truncated = len(rows) > MAX_FORMAL per: dict[tuple[str, str], int] = {} out: list[Use] = [] @@ -325,7 +325,7 @@ async def _exporters(session: AsyncSession, uses: list[Use]) -> None: by: dict[str, list[str]] = {} for rid, actor in (await session.execute(select(AuditEvent.subject_id, AuditEvent.actor).where( AuditEvent.action == EXPORTED, AuditEvent.subject_kind == "run", - AuditEvent.subject_id.in_(ids)).order_by(AuditEvent.at))).all(): + AuditEvent.subject_id.in_(ids)).order_by(AuditEvent.at, AuditEvent.id))).all(): if actor and actor not in by.setdefault(rid, []): by[rid].append(actor) for use in uses: diff --git a/backend/app/look/calibers.py b/backend/app/look/calibers.py index 17cbc5c4..8bae0ca2 100644 --- a/backend/app/look/calibers.py +++ b/backend/app/look/calibers.py @@ -193,7 +193,7 @@ class CaliberKind: from app.db.models import Workflow items: list[CaliberView] = [] - for wf in (await session.execute(select(Workflow).order_by(Workflow.name))).scalars().all(): + for wf in (await session.execute(select(Workflow).order_by(Workflow.name, Workflow.id))).scalars().all(): latest = await app_versions.latest_published(session, wf.id) if latest is not None: items += views_of(wf.id, wf.name, latest) diff --git a/backend/app/look/versions.py b/backend/app/look/versions.py index f24d25de..b2191010 100644 --- a/backend/app/look/versions.py +++ b/backend/app/look/versions.py @@ -70,7 +70,7 @@ class VersionKind: from app.db.models import Workflow items: list[VersionView] = [] - for wf in (await session.execute(select(Workflow).order_by(Workflow.name))).scalars().all(): + for wf in (await session.execute(select(Workflow).order_by(Workflow.name, Workflow.id))).scalars().all(): for v in await app_versions.published_versions(session, wf.id): items.append(VersionView(workflow_id=wf.id, workflow_name=wf.name, description=wf.description or "", version=v)) diff --git a/backend/app/main.py b/backend/app/main.py index 4b2b96bb..b926852f 100644 --- a/backend/app/main.py +++ b/backend/app/main.py @@ -145,9 +145,14 @@ async def lifespan(_: FastAPI) -> AsyncIterator[None]: from app.sandbox.manager import sandbox_manager await sandbox_manager.close() - # 最后交班、删掉本进程的登记：别的进程马上就知道本进程不在了，接手它留下的行 + # 交班、删掉本进程的登记：别的进程马上就知道本进程不在了，接手它留下的行（要用库，在关连接之前） await relay.stop() await cluster.stop() + # 应用库的连接全关掉：SQLite 最后一个连接关掉时才把 -wal 并回、删掉（copy-to-postgres 按它认服务停没停）， + # PostgreSQL 的连接池也正常关 + from app.db.base import engine as db_engine + + await db_engine.dispose() logger.info("已停止") diff --git a/backend/app/memory/store.py b/backend/app/memory/store.py index 44524cd3..534d6f65 100644 --- a/backend/app/memory/store.py +++ b/backend/app/memory/store.py @@ -215,7 +215,7 @@ async def clear_scope(session: AsyncSession, scope: str) -> int: async def list_memories( session: AsyncSession, *, scope: str | None = None, limit: int = 200 ) -> list[MemoryItem]: - stmt = select(MemoryItem).order_by(MemoryItem.created_at.desc()).limit(limit) + stmt = select(MemoryItem).order_by(MemoryItem.created_at.desc(), MemoryItem.id.desc()).limit(limit) if scope: stmt = stmt.where(MemoryItem.scope == scope) return list((await session.execute(stmt)).scalars()) diff --git a/backend/app/process/actions.py b/backend/app/process/actions.py index d0a68d83..6231d047 100644 --- a/backend/app/process/actions.py +++ b/backend/app/process/actions.py @@ -519,7 +519,7 @@ async def _confirm_task(session: AsyncSession, run_id: str, node_id: str) -> str return await session.scalar(select(Task.id).where(Task.run_id == run_id, Task.node_id == node_id, Task.kind == "action_confirm") - .order_by(Task.created_at.desc()).limit(1)) + .order_by(Task.created_at.desc(), Task.id.desc()).limit(1)) # -------------------------------------------------------------------------- diff --git a/backend/app/process/hooks.py b/backend/app/process/hooks.py index 8afee1e5..2973a6e8 100644 --- a/backend/app/process/hooks.py +++ b/backend/app/process/hooks.py @@ -251,7 +251,8 @@ async def _started_before(hook: AppHook, idem_key: str) -> str | None: async with SessionLocal() as session: rows = (await session.execute(select(Run.id, Run.origin).where( - Run.app_id == hook.app_id, Run.origin_kind == "hook").order_by(Run.created_at.desc()).limit(500))).all() + Run.app_id == hook.app_id, Run.origin_kind == "hook") + .order_by(Run.created_at.desc(), Run.id.desc()).limit(500))).all() for run_id, origin in rows: if isinstance(origin, dict) and origin.get("hook") == hook.name and origin.get("hook_key") == idem_key: return run_id @@ -267,12 +268,12 @@ async def _signal(hook: AppHook, request: HookRequest, header_key: str, *, recla async with SessionLocal() as session: task_id = await session.scalar(select(Task.id).where( Task.app_id == hook.app_id, Task.kind == "signal", Task.status == "pending", - Task.correlation_key == key).order_by(Task.created_at).limit(1)) + Task.correlation_key == key).order_by(Task.created_at, Task.id).limit(1)) if task_id is None and reclaimed: # 上一次处理到一半服务停了：在等的待办其实已经解决，交回它，不当成没人在等 done = (await session.execute(select(Task.id, Task.run_id).where( Task.app_id == hook.app_id, Task.kind == "signal", Task.status != "pending", - Task.correlation_key == key).order_by(Task.created_at.desc()).limit(1))).first() + Task.correlation_key == key).order_by(Task.created_at.desc(), Task.id.desc()).limit(1))).first() if done is not None: return {"task_id": done[0], "run_id": done[1]} if task_id is None: @@ -409,7 +410,7 @@ async def recent_receipts(session: AsyncSession, app_id: str, name: str, .scalars()) rows = list((await session.execute( select(HookReceipt).where(HookReceipt.hook_id.in_(ids or [""]), HookReceipt.status != "signature") - .order_by(HookReceipt.received_at.desc()).limit(max(1, min(limit, RECEIPTS_MAX))))).scalars()) + .order_by(HookReceipt.received_at.desc(), HookReceipt.id.desc()).limit(max(1, min(limit, RECEIPTS_MAX))))).scalars()) return {"items": [{"idem_key": r.idem_key, "status": r.status, "received_at": r.received_at.isoformat() if r.received_at else None, "run_id": r.run_id, "task_id": r.task_id, "detail": r.detail} for r in rows], diff --git a/backend/app/process/tasks.py b/backend/app/process/tasks.py index c82baa5d..e1690f40 100644 --- a/backend/app/process/tasks.py +++ b/backend/app/process/tasks.py @@ -922,7 +922,7 @@ async def tick(now: datetime | None = None) -> list[str]: and_(Task.remind_at.is_not(None), Task.remind_at <= now), and_(Task.due_at.is_not(None), Task.due_at <= now, action == "reject"), and_(Task.due_at.is_not(None), Task.due_at <= now, action == "escalate", Task.escalation_level == 0), - )).order_by(Task.due_at))).scalars()) + )).order_by(Task.due_at, Task.id))).scalars()) done: list[str] = [] for task_id in due: try: @@ -1104,7 +1104,8 @@ async def _assignments(session: AsyncSession, task_ids: list[str], *, fresh: boo ) -> dict[str, list[TaskAssignment]]: if not task_ids: return {} - stmt = select(TaskAssignment).where(TaskAssignment.task_id.in_(task_ids)).order_by(TaskAssignment.created_at) + stmt = (select(TaskAssignment).where(TaskAssignment.task_id.in_(task_ids)) + .order_by(TaskAssignment.created_at, TaskAssignment.id)) if fresh: stmt = stmt.execution_options(populate_existing=True) out: dict[str, list[TaskAssignment]] = {} @@ -1135,7 +1136,7 @@ def _mine(principal: Any) -> Any: async def list_tasks(principal: Any, *, assigned: str = "me", statuses: list[str] | None = None, app_id: str | None = None, run_id: str | None = None, limit: int = 100) -> list[dict[str, Any]]: async with SessionLocal() as session: - stmt = select(Task).order_by(Task.created_at.desc()).limit(limit) + stmt = select(Task).order_by(Task.created_at.desc(), Task.id.desc()).limit(limit) if statuses: stmt = stmt.where(Task.status.in_(statuses)) if app_id: diff --git a/backend/app/providers/factory.py b/backend/app/providers/factory.py index e7fbb29b..4389225e 100644 --- a/backend/app/providers/factory.py +++ b/backend/app/providers/factory.py @@ -53,7 +53,7 @@ class ProviderNotConfigured(RuntimeError): async def list_providers(session: AsyncSession) -> list[Provider]: - rows = await session.execute(select(Provider).order_by(Provider.created_at)) + rows = await session.execute(select(Provider).order_by(Provider.created_at, Provider.id)) return list(rows.scalars()) diff --git a/backend/app/sandbox/bubblewrap_sandbox.py b/backend/app/sandbox/bubblewrap_sandbox.py index 56ca78ce..09f13be8 100644 --- a/backend/app/sandbox/bubblewrap_sandbox.py +++ b/backend/app/sandbox/bubblewrap_sandbox.py @@ -132,9 +132,14 @@ class BubblewrapSandbox(Sandbox): return _apply - def _argv(self, workdir: Path, limits: SandboxLimits, cmd: list[str]) -> list[str]: + def _argv(self, workdir: Path, limits: SandboxLimits, cmd: list[str], + env: dict[str, str] | None = None) -> list[str]: argv = [ self._bwrap or "bwrap", + # 先清空环境变量，再一条条设：bwrap 按参数的先后处理，--clearenv 要在所有 --setenv 前面。以前不清，被执行的 + # 代码看得到后端进程的全部环境变量（模型的 API Key、带密码的 AGENTLAB_DATABASE_URL）；local、seatbelt 一直只给 + # 最小的一套 + "--clearenv", "--unshare-all", # mount/pid/ipc/uts/cgroup/net 全部隔离 *(["--share-net"] if limits.network else []), "--die-with-parent", # 父进程没了，沙箱跟着走 @@ -147,8 +152,11 @@ class BubblewrapSandbox(Sandbox): "--setenv", "HOME", "/work", "--setenv", "TMPDIR", "/tmp", "--setenv", "PATH", "/usr/bin:/bin", + "--setenv", "LANG", "C.UTF-8", "--setenv", "PYTHONDONTWRITEBYTECODE", "1", ] + for key, value in (env or {}).items(): + argv += ["--setenv", key, value] for path in _RO_BINDS: if Path(path).exists(): argv += ["--ro-bind", path, path] @@ -185,14 +193,14 @@ class BubblewrapSandbox(Sandbox): target.parent.mkdir(parents=True, exist_ok=True) atomic.write_text(target, content) - argv = self._argv(workdir, limits, cmd) - for key, value in (env or {}).items(): - argv[1:1] = ["--setenv", key, value] + argv = self._argv(workdir, limits, cmd, env) started = time.perf_counter() try: proc = await asyncio.create_subprocess_exec( *argv, + # bwrap 自己也只拿最小的环境：沙箱里的进程靠上面的 --setenv，不从这里继承 + env={"PATH": os.environ.get("PATH", "/usr/bin:/bin")}, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE, preexec_fn=self._preexec(limits), diff --git a/backend/app/scheduler/outbox.py b/backend/app/scheduler/outbox.py index 4df850a5..5ae11922 100644 --- a/backend/app/scheduler/outbox.py +++ b/backend/app/scheduler/outbox.py @@ -388,7 +388,7 @@ async def _drain(schedule_id: str, now: datetime) -> list[ScheduleFire]: select(TriggerDelivery.id, TriggerDelivery.created_at, TriggerEvent) .join(TriggerEvent, TriggerEvent.id == TriggerDelivery.event_id) .where(TriggerDelivery.schedule_id == schedule_id, TriggerDelivery.status == "pending") - .order_by(TriggerDelivery.occurred_at, TriggerEvent.created_at).limit(DRAIN_BATCH))).all() + .order_by(TriggerDelivery.occurred_at, TriggerEvent.created_at, TriggerEvent.id).limit(DRAIN_BATCH))).all() # 等着的时候改了触发条件（换了对象、加了过滤）：已经不合条件的不再派发 stale = [did for did, _, ev in rows if not triggers.matches(trigger, ev.event, ev.subject_id, ev.payload)] if stale: diff --git a/backend/app/scheduler/service.py b/backend/app/scheduler/service.py index 9539b94b..e3e0aa3f 100644 --- a/backend/app/scheduler/service.py +++ b/backend/app/scheduler/service.py @@ -533,7 +533,7 @@ async def _active_run(schedule_id: str) -> tuple[str, str, bool, bool] | None: async with SessionLocal() as session: row = (await session.execute( select(Run.id, Run.status).where(Run.schedule_id == schedule_id, Run.status.in_(ACTIVE_RUN)) - .order_by(Run.created_at.desc()).limit(1))).first() + .order_by(Run.created_at.desc(), Run.id.desc()).limit(1))).first() if row is None: return None run_id, status = row diff --git a/backend/app/scheduler/triggers.py b/backend/app/scheduler/triggers.py index 00ad81a5..f13a4b5e 100644 --- a/backend/app/scheduler/triggers.py +++ b/backend/app/scheduler/triggers.py @@ -547,7 +547,7 @@ async def sample(session: Any, trigger: dict[str, Any]) -> Sample: query = query.where(Run.app_id == trigger["app_id"]) if trigger.get("origin_kind"): query = query.where(Run.origin_kind == trigger["origin_kind"]) - candidates = list((await session.execute(query.order_by(Run.finished_at.desc()).limit(SAMPLE_SCAN))) + candidates = list((await session.execute(query.order_by(Run.finished_at.desc(), Run.id.desc()).limit(SAMPLE_SCAN))) .scalars()) if subject else [] # 和真派发同一套条件（matches）：缺省不认的途经驳回、应用补算跳过，取合条件的最近一次 for run in candidates: diff --git a/backend/app/scheduler/webhook.py b/backend/app/scheduler/webhook.py index 7c32d96f..2db4eb85 100644 --- a/backend/app/scheduler/webhook.py +++ b/backend/app/scheduler/webhook.py @@ -446,7 +446,7 @@ async def notify_due(now: datetime | None = None) -> int: and_(ScheduleFire.status == "started", Run.status.in_(_WATCHED), or_(ScheduleFire.notified_status.is_(None), ScheduleFire.notified_status != Run.status)), )) - .order_by(ScheduleFire.created_at) + .order_by(ScheduleFire.created_at, ScheduleFire.id) )).all() if rows: # 链接里的地址：全站同一个对外访问地址（环境变量优先，其次设置，core/public_url.py） diff --git a/backend/app/tools/builtin/files.py b/backend/app/tools/builtin/files.py index 87c35cae..cd988987 100644 --- a/backend/app/tools/builtin/files.py +++ b/backend/app/tools/builtin/files.py @@ -75,10 +75,12 @@ async def file_write(ctx: ToolContext, **kwargs: Any) -> dict[str, Any]: target = _resolve(args.path, ctx) except ValueError as e: return {"error": str(e)} - # 先写临时文件再改名（core/atomic）：同一个工作目录里正在跑的代码、另一台机器上读共享数据目录的一方，看到的要么是 - # 旧的整份、要么是新的整份。追加也一样：读出原有的、接上新的、整份换掉 - before = target.read_bytes() if args.append and target.is_file() else b"" - atomic.write_bytes(target, before + args.content.encode("utf-8")) + # 覆盖：先写临时文件再改名（core/atomic），读的一方看到的要么是旧的整份、要么是新的整份，权限照旧。追加：原地追加写 + # （O_APPEND），不整份重写——文件的 inode、权限不变，原有内容不动，读的一方最多看到半截新内容 + if args.append: + atomic.append_bytes(target, args.content.encode("utf-8")) + else: + atomic.write_bytes(target, args.content.encode("utf-8")) return {"path": args.path, "bytes": target.stat().st_size, "ok": True} diff --git a/backend/tests/conftest.py b/backend/tests/conftest.py index cd074b0a..0a51b2a2 100644 --- a/backend/tests/conftest.py +++ b/backend/tests/conftest.py @@ -71,6 +71,12 @@ os.environ.setdefault("AGENTLAB_SCHEDULER", "0") import pytest # noqa: E402 +# 整个 JSON 值做比较、去重、分组、排序时编译就报错（tests/jsonguard.py）：PostgreSQL 的 json 不支持，SQLite 的测试也拦下 +from app.db.base import engine as _guarded_engine # noqa: E402 +from tests.jsonguard import install as _json_guard # noqa: E402 + +_json_guard(_guarded_engine.sync_engine.dialect) + @pytest.fixture(scope="session", autouse=True) def _prepare_schema(): diff --git a/backend/tests/jsonguard.py b/backend/tests/jsonguard.py new file mode 100644 index 00000000..6dbb11e3 --- /dev/null +++ b/backend/tests/jsonguard.py @@ -0,0 +1,63 @@ +"""测试时挂在应用库编译器上的守卫：整个 JSON 值做比较、去重、分组、排序就报错（两种库都挂，跑全量时一直生效）。 + +PostgreSQL 的 json 没有相等和大小的运算符：`=`、`!=`、`IN`、`IS DISTINCT FROM`、`<` 之类，SELECT DISTINCT、GROUP BY、 +ORDER BY、UNION（去重的那几种）碰到 json 类型的值都报错；SQLite 照常执行。只在 PostgreSQL 上跑到才发现太晚：这里在编译 +那一刻就拦下，SQLite 的测试也拦得到。按表达式的类型认，不按写法认：跨行的 order_by、变量别名、 +`select(Run.origin).distinct()`、JSON 的下标（`Run.origin["k"] == 1`，取出来的还是 json）都算。按键筛选用 db/portable 的 +json_eq 等（类型是布尔、文本），不受影响。 +""" + +from __future__ import annotations + +from typing import Any + +from sqlalchemy import JSON +from sqlalchemy.sql import operators +from sqlalchemy.sql.elements import UnaryExpression + + +class JsonMisuse(AssertionError): + """整个 JSON 值被拿去比较、去重、分组或排序。""" + + +_COMPARE = {operators.eq, operators.ne, operators.in_op, operators.not_in_op, operators.is_distinct_from, + operators.is_not_distinct_from, operators.lt, operators.le, operators.gt, operators.ge} + + +def _json(element: Any) -> bool: + while isinstance(element, UnaryExpression) and element.modifier is not None: + element = element.element + return isinstance(getattr(element, "type", None), JSON) + + +class JsonGuard: + """混进编译器类里用：class Guarded(JsonGuard, 原来的编译器)。""" + + def visit_binary(self, binary: Any, **kw: Any) -> str: + if binary.operator in _COMPARE and (_json(binary.left) or _json(binary.right)): + raise JsonMisuse(f"整个 JSON 值做了比较（{binary.operator.__name__}）：PostgreSQL 的 json 没有这个运算符，" + "按键筛选用 db/portable.json_eq 等") + return super().visit_binary(binary, **kw) # type: ignore[misc] + + def visit_select(self, select: Any, **kw: Any) -> str: + if select._distinct and any(_json(c) for c in select.selected_columns): + raise JsonMisuse("SELECT DISTINCT 里有 JSON 列：PostgreSQL 的 json 没有相等运算符，去不了重") + if any(_json(c) for c in select._group_by_clauses): + raise JsonMisuse("GROUP BY 了 JSON 值：PostgreSQL 的 json 没有相等运算符") + if any(_json(c) for c in select._order_by_clauses): + raise JsonMisuse("ORDER BY 了 JSON 值：PostgreSQL 的 json 没有大小") + return super().visit_select(select, **kw) # type: ignore[misc] + + def visit_compound_select(self, cs: Any, **kw: Any) -> str: + keyword = str(getattr(cs.keyword, "value", cs.keyword)).upper() + if "ALL" not in keyword and any(_json(c) for c in cs.selected_columns): + raise JsonMisuse(f"{keyword or 'UNION'} 去重时碰到 JSON 列：PostgreSQL 的 json 没有相等运算符") + return super().visit_compound_select(cs, **kw) # type: ignore[misc] + + +def install(dialect: Any) -> None: + """挂到应用库引擎的方言实例上（PostgreSQL 时是 db/postgres 换过的编译器，SQLite 时是原来的）。""" + base = dialect.statement_compiler + if issubclass(base, JsonGuard): + return + dialect.statement_compiler = type(f"JsonGuarded{base.__name__}", (JsonGuard, base), {}) diff --git a/backend/tests/test_atomic_writes.py b/backend/tests/test_atomic_writes.py index 76c326d4..8ac67581 100644 --- a/backend/tests/test_atomic_writes.py +++ b/backend/tests/test_atomic_writes.py @@ -77,16 +77,67 @@ def test_content_addressed_artifacts_are_written_atomically(tmp_path, monkeypatc assert _leftovers(folder) == [] -async def test_file_write_tool_appends_by_replacing_the_whole_file(): +async def test_file_write_tool_overwrites_atomically_and_appends_in_place(): from types import SimpleNamespace from app.tools.builtin.files import file_write ctx = SimpleNamespace(sandbox_session=f"atomic-{os.getpid()}") assert (await file_write(ctx, path="notes/a.txt", content="第一行\n"))["ok"] - out = await file_write(ctx, path="notes/a.txt", content="第二行\n", append=True) from app.core.config import settings path = settings.session_dir(ctx.sandbox_session) / "notes" / "a.txt" + inode = path.stat().st_ino + out = await file_write(ctx, path="notes/a.txt", content="第二行\n", append=True) assert path.read_text(encoding="utf-8") == "第一行\n第二行\n" and out["bytes"] == len("第一行\n第二行\n".encode()) + assert path.stat().st_ino == inode, "追加不换文件" + await file_write(ctx, path="notes/a.txt", content="重写\n") + assert path.read_text(encoding="utf-8") == "重写\n" assert _leftovers(path.parent) == [] + + +def test_create_falls_back_when_the_filesystem_cannot_hard_link(tmp_path, monkeypatch): + """exFAT、一部分 SMB / FUSE 挂载不支持硬链接（EPERM、ENOTSUP、EXDEV）：退回 O_EXCL 直接建，照样只建一次，不崩。""" + import errno + + def no_links(src, dst): + raise OSError(errno.EPERM, "Operation not permitted") + + monkeypatch.setattr(os, "link", no_links) + target = tmp_path / ".secret" + assert atomic.create_text(target, "第一把", mode=0o600) is True + assert atomic.create_text(target, "第二把", mode=0o600) is False + assert target.read_text() == "第一把" and stat.S_IMODE(target.stat().st_mode) == 0o600 + assert _leftovers(tmp_path) == [] + + +def test_replacing_keeps_the_permissions_of_the_old_file(tmp_path): + target = tmp_path / "run.sh" + target.write_text("echo 1") + target.chmod(0o750) + atomic.write_text(target, "echo 2") + assert target.read_text() == "echo 2" and stat.S_IMODE(target.stat().st_mode) == 0o750 + + +def test_appending_keeps_the_same_file(tmp_path): + """追加写不换文件：inode、权限都不变，原有内容不动（以前整份读出来重写、再改名）。""" + target = tmp_path / "log.txt" + target.write_text("一\n") + target.chmod(0o640) + before = target.stat().st_ino + atomic.append_bytes(target, "二\n".encode()) + assert target.read_text() == "一\n二\n" + assert target.stat().st_ino == before and stat.S_IMODE(target.stat().st_mode) == 0o640 + assert _leftovers(tmp_path) == [] + + +def test_the_data_dir_carries_a_marker_deploy_can_recognise(): + """数据目录里有 .agentlab-data：库在 PostgreSQL、密钥由环境变量给时，deploy.sh 也认得出这是 AgentLab 的数据目录。""" + from app.core.config import DATA_DIR_MARKER, settings + + settings.ensure_dirs() + marker = settings.data_dir / DATA_DIR_MARKER + assert marker.is_file() + text = marker.read_text(encoding="utf-8") + settings.ensure_dirs() + assert marker.read_text(encoding="utf-8") == text, "已有的不覆盖" diff --git a/backend/tests/test_bubblewrap_probe.py b/backend/tests/test_bubblewrap_probe.py index 81b1aaf8..55fa91d3 100644 --- a/backend/tests/test_bubblewrap_probe.py +++ b/backend/tests/test_bubblewrap_probe.py @@ -103,3 +103,58 @@ def test_auto_falls_back_to_local_and_says_why(monkeypatch: pytest.MonkeyPatch) # 节点上选 fast 档也不能点到一个起不来的 bubblewrap monkeypatch.setattr(mgr.sys, "platform", "linux") assert manager._resolve_named("fast") is None + + +# -------------------------------------------------------------------------- +# 环境变量：沙箱里的代码看不到后端进程的（模型的 API Key、带密码的 AGENTLAB_DATABASE_URL） +# -------------------------------------------------------------------------- + +_SECRETS = {"AGENTLAB_DATABASE_URL": "postgresql+asyncpg://u:not-a-real-password@h/db", + "ANTHROPIC_API_KEY": "sk-ant-not-real", "OPENAI_API_KEY": "sk-not-real"} + + +def _setenvs(argv: list[str]) -> dict[str, str]: + return {argv[i + 1]: argv[i + 2] for i, a in enumerate(argv) if a == "--setenv"} + + +def test_the_sandbox_starts_from_a_cleared_environment(monkeypatch, tmp_path): + from app.sandbox.base import SandboxLimits + + for key, value in _SECRETS.items(): + monkeypatch.setenv(key, value) + argv = bw.BubblewrapSandbox()._argv(tmp_path, SandboxLimits(), ["/usr/bin/python3", "/work/x.py"], + {"RUN_FLAG": "1", "HOME": "/work/custom"}) + # --clearenv 在所有 --setenv 前面：bwrap 按先后处理，放后面就把设好的也清掉了 + assert argv[1] == "--clearenv" and argv.index("--clearenv") < argv.index("--setenv") + env = _setenvs(argv) + assert env["RUN_FLAG"] == "1" and env["HOME"] == "/work/custom", "调用方给的照旧生效，能盖过缺省的" + assert env["PATH"] == "/usr/bin:/bin" + assert not any(k.startswith("AGENTLAB_") or k.endswith("_API_KEY") for k in env) + assert not any(v in " ".join(argv) for v in _SECRETS.values()) + + +async def test_bwrap_itself_gets_only_a_minimal_environment(monkeypatch, tmp_path): + for key, value in _SECRETS.items(): + monkeypatch.setenv(key, value) + seen: dict[str, Any] = {} + + class Done: + returncode = 0 + pid = 1 + + async def communicate(self): + return b"", b"" + + async def fake_exec(*argv, **kwargs): + seen["argv"], seen["env"] = list(argv), kwargs.get("env") + return Done() + + box = bw.BubblewrapSandbox() + monkeypatch.setattr(bw.BubblewrapSandbox, "available", staticmethod(lambda: True)) + monkeypatch.setattr(box, "_session_dir", lambda _sid: tmp_path) + monkeypatch.setattr(bw.asyncio, "create_subprocess_exec", fake_exec) + result = await box.run("print(1)", env={"RUN_FLAG": "1"}) + assert result.ok, result.error + assert seen["env"] is not None and set(seen["env"]) == {"PATH"}, "bwrap 进程本身不继承后端的环境变量" + assert not any(v in " ".join(seen["argv"]) for v in _SECRETS.values()) + assert _setenvs(seen["argv"])["RUN_FLAG"] == "1" diff --git a/backend/tests/test_copy_to_postgres.py b/backend/tests/test_copy_to_postgres.py index b1391432..23a41350 100644 --- a/backend/tests/test_copy_to_postgres.py +++ b/backend/tests/test_copy_to_postgres.py @@ -14,6 +14,7 @@ import sys from pathlib import Path import pytest +from sqlalchemy import text from app.core.config import settings from app.db import to_postgres @@ -68,6 +69,11 @@ async def main(): s.add(source) await s.flush() s.add(SourceChange(source_id=source.id, kind="column_added", fingerprint="f" * 64, table_name="订单")) + dropped = SourceChange(source_id=source.id, kind="column_removed", fingerprint="d" * 64, table_name="订单") + s.add(dropped) + await s.flush() + # 删掉最后一条：SQLite 的 AUTOINCREMENT 计数器停在 2，表里的 MAX 是 1。导过去之后新的变化要从 3 编起 + await s.delete(dropped) long = Run(workflow_name="长" * 300, last_seq=5_000_000_000, origin={"b": 2, "a": 1}) sql_null = Run(workflow_name="SQL 的空", origin={"x": 1}) json_null = Run(workflow_name="JSON 的空", origin=None) @@ -263,7 +269,7 @@ async def test_a_waiting_run_is_imported_and_finishes_on_postgresql(source_dir): seqs = done["seqs"] assert seqs == sorted(set(seqs)) and seqs[-1] > made["last_seq"], "接着导入前的序号往下编" assert done["artifact"] == {"区域": "虚构", "数": 1} - assert done["new_change_seq"] > 1, "自增主键的序列接在导入的最大值后面" + assert done["new_change_seq"] == 3, "序列接在 SQLite 的 AUTOINCREMENT 计数器后面，删过的号不复用（召回按它当游标）" @needs_pg @@ -294,3 +300,137 @@ async def test_a_target_with_the_service_still_connected_is_refused(source_dir): with pytest.raises(CopyRefused, match="服务的连接"): await copy_to_postgres(data, target, progress=lambda _: None) assert _sql(target, "SELECT count(*) FROM information_schema.tables WHERE table_name = 'runs'") == [(0,)] + + +# -------------------------------------------------------------------------- +# 老库：补列时没带缺省值的 NOT NULL 列、SQLite 当时没强制外键留下的孤儿行 +# -------------------------------------------------------------------------- + + +def _legacy_source(folder: Path) -> Path: + """照模型建一个 SQLite 应用库，再做旧：定时任务的 trigger 列删掉、按第一版迁移（不带缺省值）补回来，存量行是 NULL； + 关着外键写几行孤儿行（对话轮次指向已删的运行、事件指向已删的运行、轮次指向已删的对话）。""" + from datetime import datetime, timezone + + from sqlalchemy import create_engine, insert + + from app.db import models # noqa: F401 + from app.db.base import Base + + folder.mkdir(parents=True, exist_ok=True) + path = folder / "agentlab.db" + engine = create_engine(f"sqlite:///{path}") + tables = Base.metadata.tables + try: + Base.metadata.create_all(engine) + with engine.begin() as conn: + conn.execute(insert(tables["schedules"]).values(id="s1", workflow_id="w1", cron="0 9 * * *", + timezone="Asia/Shanghai")) + conn.execute(insert(tables["schedule_fires"]).values( + id="f1", schedule_id="s1", fire_at=datetime(2026, 1, 1, tzinfo=timezone.utc))) + conn.execute(insert(tables["conversations"]).values(id="c1", title="虚构的对话")) + conn.execute(insert(tables["conversation_turns"]).values(id="t1", conversation_id="c1", seq=0, + run_id="run-gone")) + conn.execute(insert(tables["conversation_turns"]).values(id="t2", conversation_id="conv-gone", seq=0)) + conn.execute(insert(tables["run_events"]).values(id="e1", run_id="run-gone", seq=1, type="log", ts=0.0, + data={})) + with engine.begin() as conn: + conn.execute(text("ALTER TABLE schedules DROP COLUMN \"trigger\"")) + conn.execute(text("ALTER TABLE schedules ADD COLUMN \"trigger\" JSON")) + finally: + engine.dispose() + return path + + +def test_old_sqlite_databases_get_null_not_null_columns_filled(tmp_path): + """SQLite 这边：启动时的迁移把老版本补列留下的 NULL 填成模型的缺省值，两种库存的一样。""" + from sqlalchemy import create_engine + + from app.db.base import _migrate + + path = _legacy_source(tmp_path / "old") + engine = create_engine(f"sqlite:///{path}") + try: + with engine.begin() as conn: + assert conn.execute(text("SELECT \"trigger\" FROM schedules")).scalar_one() is None + _migrate(conn) + assert conn.execute(text("SELECT \"trigger\" FROM schedules")).scalar_one() == "{}" + finally: + engine.dispose() + + +def test_orphan_rows_are_nulled_or_left_out_on_the_snapshot(tmp_path): + path = _legacy_source(tmp_path / "old") + fixed: dict[str, int] = {} + orphans = to_postgres.settle_orphans(path, fixed=fixed) + assert fixed == {"外键指向已不存在的行，置空": 1} + assert orphans == {"conversation_turns": 1, "run_events": 1} + import sqlite3 + + db = sqlite3.connect(path) + try: + assert db.execute("SELECT id, run_id FROM conversation_turns").fetchall() == [("t1", None)] + assert db.execute("SELECT count(*) FROM run_events").fetchone() == (0,) + finally: + db.close() + + +@needs_pg +async def test_an_old_database_with_nulls_and_orphans_imports_cleanly(tmp_path): + """真实开发库上核对不过的那两种：定时任务 trigger 为 NULL（NOT NULL 列），对话轮次指向已删的运行。导入照样核对通过， + 孤儿行照实列出、不算失败，重跑也一样。""" + from tests.appdb import fresh_postgres_database + + folder = tmp_path / "old" + _legacy_source(folder) + with fresh_postgres_database("legacy") as target: + report = await copy_to_postgres(folder, target, progress=lambda _: None) + lines = to_postgres.render(report) + assert report.ok, "\n".join(lines) + assert report.orphans == {"conversation_turns": 1, "run_events": 1} + assert report.fixed.get("外键指向已不存在的行，置空") == 1 + assert any("孤儿行" in line and "run_events 1 行" in line for line in lines) + assert _sql(target, "SELECT CAST(\"trigger\" AS TEXT) FROM schedules") == [("{}",)] + assert _sql(target, "SELECT count(*) FROM schedule_fires") == [(1,)] + assert _sql(target, "SELECT run_id FROM conversation_turns") == [(None,)] + again = await copy_to_postgres(folder, target, progress=lambda _: None) + assert again.ok and again.orphans == report.orphans + + +def test_null_not_null_values_fall_back_to_the_model_default(): + from app.db.base import Base + + schedules = Base.metadata.tables["schedules"] + fixed: dict[str, int] = {} + fix = to_postgres._Fixer(fixed) + assert fix.value(schedules.c["trigger"], None) == "{}" # JSON 列按原文绑定 + assert fix.value(schedules.c["trigger_kind"], None) == "cron" + assert fix.value(schedules.c["target_id"], None) is None # 可空的照旧是 NULL + assert fix.value(schedules.c["cron"], None) is None # 没有缺省值的：照实交给库去拒 + assert fixed == {"NOT NULL 列的空值填成缺省值": 2} + + +# -------------------------------------------------------------------------- +# 停服务之后数据目录里不留 -wal（导入按它认服务停没停） +# -------------------------------------------------------------------------- + +START_STOP = r''' +from fastapi.testclient import TestClient +from app.main import app + +with TestClient(app) as client: + assert client.get("/api/health").status_code == 200 +''' + + +def test_stopping_the_service_leaves_no_wal_behind(tmp_path): + data = tmp_path / "data" + data.mkdir() + env = {**os.environ, "AGENTLAB_DATA_DIR": str(data), "AGENTLAB_DATABASE_URL": "", "AGENTLAB_CHECKPOINT_URL": "", + "AGENTLAB_SCHEDULER": "0", "PYTHONPATH": str(BACKEND)} + done = subprocess.run([sys.executable, "-c", START_STOP], env=env, capture_output=True, text=True, timeout=180, + cwd=str(data)) + assert done.returncode == 0, done.stderr[-4000:] + assert (data / "agentlab.db").is_file() and (data / "checkpoints.db").is_file() + assert sorted(p.name for p in data.iterdir() if p.name.endswith(("-wal", "-shm"))) == [] + assert to_postgres.source_in_use(data / "agentlab.db", data / "checkpoints.db") is None diff --git a/backend/tests/test_db_portable.py b/backend/tests/test_db_portable.py index 6374077f..1d0e134e 100644 --- a/backend/tests/test_db_portable.py +++ b/backend/tests/test_db_portable.py @@ -45,6 +45,21 @@ def test_every_migrated_column_is_in_the_models(): assert missing == [] +def test_every_not_null_migrated_column_gets_a_value_for_old_rows(): + """迁移补的列模型里是 NOT NULL 的，存量行不能留 NULL（PostgreSQL 的表照模型带 NOT NULL，导入时整行插不进）：要么补列时 + 带 DEFAULT 回填，要么登记在 _NOT_NULL_BACKFILLS 里每次启动按模型的缺省值填。""" + bad = [] + for table, column, ddl in db_base._COLUMN_MIGRATIONS: + col = Base.metadata.tables[table].c[column] + if col.nullable: + continue + if not re.search(r"\bDEFAULT\b", ddl, re.I) and (table, column) not in db_base._NOT_NULL_BACKFILLS: + bad.append((table, column)) + assert bad == [] + for table, column in db_base._NOT_NULL_BACKFILLS: + assert db_base.model_default(Base.metadata.tables[table].c[column])[0], (table, column) + + def test_checkpoint_tables_do_not_clash_with_app_tables(): """断点和应用表放在同一个库、同一个 schema：两边的表名不能撞。名字从两个 saver 自己的建表语句里取。""" import inspect as pyinspect @@ -58,22 +73,40 @@ def test_checkpoint_tables_do_not_clash_with_app_tables(): assert created & set(Base.metadata.tables) == set() -def test_no_query_compares_sorts_or_groups_a_whole_json_column(): - """PostgreSQL 的 json 没有相等运算符：整列做 ==、!=、in_、DISTINCT、ORDER BY、GROUP BY 都会报错（SQLite 不报）。 - 按里面某个键筛选走 db/portable 的 json_eq 等。扫一遍应用代码里「模型.JSON列」后面跟着这几样的写法。""" - from pathlib import Path +def test_the_compile_guard_catches_whole_json_values_however_they_are_written(): + """整个 JSON 值做比较、去重、分组、排序：编译时就报错（tests/jsonguard.py，conftest 挂在应用库引擎上，跑全量时一直生效）。 + 按表达式的类型认，逐行正则抓不到的写法也抓得到；按键筛选的 json_eq 等、IS NULL、UNION ALL 不受影响。""" + from sqlalchemy import union, union_all - from sqlalchemy import JSON + from tests.jsonguard import JsonMisuse - columns = sorted({f"{m.class_.__name__}.{a.key}" for m in Base.registry.mappers for a in m.column_attrs - if isinstance(a.columns[0].type, JSON)}) - names = "|".join(re.escape(c) for c in columns) - bad = re.compile(rf"(? bool: + return bool((await session.execute(select(func.count()).select_from(Run).where( + Run.workflow_name.like(f"大小写-{tag}-%"), contains_ci(Run.workflow_name, needle)))).scalar()) + + assert await hit("report") and await hit("REPORT_100%") + assert not await hit("report-100") and not await hit("100%%") + assert await hit("éclair") == bool(folds) and await hit("жук") == bool(folds) and await hit("ａｂ") == bool(folds) + assert await hit("Éclair") and await hit("Жук") + + async def test_timestamps_read_back_as_aware_utc(): from datetime import datetime, timedelta, timezone diff --git a/backend/tests/test_kb_progress.py b/backend/tests/test_kb_progress.py index 3d8055a1..f4ab5c37 100644 --- a/backend/tests/test_kb_progress.py +++ b/backend/tests/test_kb_progress.py @@ -178,7 +178,9 @@ async def test_a_second_reindex_while_one_is_running_is_refused(client, monkeypa async def slow(session, collection=None, *, on_progress=None): await gate.wait() - return await real(session, collection, on_progress=on_progress) + # 放行之后只重建一个空集合：这里测的是「同时只跑一个」，不是重建本身。按全部集合重建要把同一个 xdist 进程里 + # 别的测试留下的几百段都重算一遍，满载时超过 _wait_done 的 10 秒，后面两条测试跟着撞上「正在重建」 + return await real(session, "reindex_gate_empty", on_progress=on_progress) monkeypatch.setattr(kb, "reindex", slow) try: @@ -211,7 +213,7 @@ async def test_two_reindexes_started_at_the_same_moment_do_not_both_run(client, async def slow(session, collection=None, *, on_progress=None): started.append(collection) await gate.wait() - return await real(session, collection, on_progress=on_progress) + return await real(session, "reindex_gate_empty", on_progress=on_progress) # 理由同上一条 monkeypatch.setattr(kb, "reindex", slow) try: diff --git a/deploy/README.md b/deploy/README.md index 22d0bfab..9edbabad 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -58,7 +58,8 @@ deploy.sh 依次：检查 docker 和架构 → 导入镜像（`docker load -i im 所有数据都在 `DATA_DIR`（默认是包目录下的 `data/`）：数据库 `agentlab.db`、执行断点 `checkpoints.db`、 工件、上传文件、沙箱工作区，以及加密密钥 `.secret_key`。**要备份的就是这个目录**， -丢了 `.secret_key`，存进去的 API Key 就解不开了。数据库放在 PostgreSQL 时（下面「用 PostgreSQL」）没有这两个库文件， +丢了 `.secret_key`，存进去的 API Key 就解不开了。目录里的 `.agentlab-data` 是服务启动时建的标记，deploy.sh 改属主前按它 +（或 `agentlab.db`、`.secret_key`）认出这是 AgentLab 的数据目录，别删。数据库放在 PostgreSQL 时（下面「用 PostgreSQL」）没有这两个库文件， 另用 `pg_dump` 备份数据库。 deploy.sh 会把数据目录的属主改成容器里的用户（uid 10001），所以备份、删除要用 sudo。 diff --git a/deploy/deploy.sh b/deploy/deploy.sh index 9e74e29f..0d579dda 100755 --- a/deploy/deploy.sh +++ b/deploy/deploy.sh @@ -266,10 +266,11 @@ fix_data_owner() { owner="$(ls -nd "$DATA_ABS" | awk '{print $3}')" [ "$owner" = "$uid" ] && return 0 # 改属主前确认这确实是一个新目录或者 AgentLab 的数据目录，免得 DATA_DIR 写错改了别的东西 - # 应用库放在 PostgreSQL 时（AGENTLAB_DATABASE_URL）数据目录里没有 agentlab.db，认 .secret_key 和 uploads/ + # 认这三样里的一样：agentlab.db（库在 SQLite）、.secret_key（自动生成的加密密钥）、.agentlab-data（服务启动时建的标记， + # 库放在 PostgreSQL、密钥又由 AGENTLAB_SECRET_KEY 给时只有它）。只有 uploads/ 这类普通目录名的不认，免得 chown 错了别人的目录 if [ -n "$(ls -A "$DATA_ABS" 2>/dev/null)" ] && [ ! -e "$DATA_ABS/agentlab.db" ] \ - && [ ! -e "$DATA_ABS/.secret_key" ] && [ ! -d "$DATA_ABS/uploads" ]; then - die "DATA_DIR 指向的 $DATA_ABS 不是空目录，也不像 AgentLab 的数据目录（里面没有 agentlab.db、.secret_key、uploads）。 + && [ ! -e "$DATA_ABS/.secret_key" ] && [ ! -e "$DATA_ABS/.agentlab-data" ]; then + die "DATA_DIR 指向的 $DATA_ABS 不是空目录，也不像 AgentLab 的数据目录（里面没有 agentlab.db、.secret_key、.agentlab-data）。 为免改错别的文件的属主，请指向一个新目录，或者原来的 AgentLab 数据目录" fi say "把数据目录的属主改成容器里的用户（uid ${uid}），容器里的服务才写得进去"