访问密钥与服务账号的后端(下一轮第 2 波 W2-d,细案 04 A1–A6、总方案 X8)

外部系统(门户、OA、BI)和个人脚本要用一个身份调接口,以前只能拿个人会话 cookie 去模拟。这次加访问密钥:

- 签发(A1):users.kind 区分人和服务账号(svc- 开头、没有口令、不能登录、角色固定使用者,管理员建);access_keys 只存
  sha256 和开头一截,明文 agl_ 加 32 字节随机串只显示一次;到期必填(最长 365 天、缺省 90 天)、可撤销、可轮换(新钥 + 旧钥
  24 小时后到期);个人访问密钥由管理员开关 auth.personal_keys 管,缺省关,关掉时已有的个人密钥一并不认
- 认人与门禁(A2):Principal 加 via / key_id / key_name / scopes / app_ids(全部带默认值);中间件认 Bearer agl_…,不看
  cookie、不查 CSRF、不看 X-Actor,认不出的各回 401 带各自的 code、绝不退回匿名;WebSocket 带密钥一律拒;关登录时 409
  keys_need_login。KEY_ROUTES(方法 + 路由模板 → scope)一张表,FastAPI(dependencies=[Depends(key_gate)]) 挂一次;不在表里的
  403 key_route_forbidden,缺 scope、限定应用、个人应用和试运行各有各的 403;每把密钥每分钟 120 次(429 + Retry-After)
- 封顶(A3):按密钥认出来的一律是使用者(拥有者现在的组),现有的使用者规则和遮罩全部自动生效,管理员的密钥也一样
- 写操作(A4):发起入口、刷新视图必须带 Idempotency-Key,api_receipts 按(拥有者, 键)去重,轮换后照样认得出;状态机从
  process/hooks.py 抽到 process/receipts.py,钩子和密钥共用(钩子现有测试原样通过),另按请求指纹拒掉同一个键办另一件事;
  有文件控件的入口 422;回复待办只给个人密钥、只限审批 / 补充输入 / 编辑稿件;run.origin、task.responded 记 via=key、哪把密钥
- 视图数据(A5):GET /api/apps/{id}/views/{view}/data(agentlab.viewdata/1),印章、清单哈希、口径卡指标原值和渲染文字,
  遮罩的 value 为 null(和页面同一个判断),按需视图只读缓存、不发起运行,只给正式运行
- 留痕(A6):建 / 撤销 / 轮换密钥、建 / 停用服务账号、开关个人密钥写审计;请求记录进程内缓冲、调度循环一拍里批量写、
  保留 90 天;last_used_at 至少隔 1 分钟写一次;钩子收件记录接口(manage),验签失败只在进程内按钩子计数
- 名录与派单:服务账号在名录里标出,岗位绑定、选人候选、提交审核、改派里排除,应用授权照收;服务账号发起的实例里派给发起人
  的待办交给任何人
- 接口说明(A7):GET /api/access-keys/reference 从 KEY_ROUTES 生成(要登录)

测试:签发与各种 401、门禁合约(遍历全站路由)、CSRF 与身份来源、封顶(管理员密钥取工件也是遮过的)、幂等(含轮换后、
半路杀进程)、留痕、待办种类、视图数据(真跑的遮罩会员库)、限流、钩子收件记录。
This commit is contained in:
ThearchyHelios
2026-10-16 13:48:09 +08:00
parent 301a599e9c
commit 19329c873e
25 changed files with 2794 additions and 87 deletions
+68 -33
View File
@@ -35,6 +35,7 @@ import json
import logging
import secrets
import time
from collections import deque
from dataclasses import dataclass
from typing import Any
@@ -44,13 +45,16 @@ from sqlalchemy.ext.asyncio import AsyncSession
from app.db.base import SessionLocal, utcnow
from app.db.models import AppHook, HookReceipt, Task
from app.process import receipts
logger = logging.getLogger(__name__)
#: 签名时间窗(秒)
WINDOW_S = 300
#: 回执停在 processing 超过这么久(秒),同一个幂等键可以重新认领:处理一次请求用不了这么久,多半是中途服务停了
STALE_S = 600
STALE_S = receipts.STALE_S
#: 钩子的收件记录:按 (钩子, 幂等键) 去重的状态机(process/receipts.py,访问密钥的发起也用它)
_BOOK = receipts.Book(HookReceipt, "hook_id", stale_s=STALE_S)
#: 「这个签名认了哪个幂等键」的记录也放在回执表里(idem_key 是这个前缀加时间戳和签名,detail 是认的幂等键):同一个
#: 唯一约束管两件事,不另起表;过了两个时间窗就删
SIGNATURE_PREFIX = "sig:"
@@ -116,7 +120,11 @@ async def receive(hook_id: str, headers: Any, body: bytes) -> dict[str, Any]:
declared = await published_declaration(session, hook.app_id, hook.name)
if not secret:
raise HookRejected("钩子的密钥无法解密(加密密钥可能已更换),请在应用里重新生成钩子", status_code=409)
verify(secret, headers, body)
try:
verify(secret, headers, body)
except HookRejected:
_signature_failed(hook.id)
raise
# 验过签名才说「发布版里没有它」:没验签的请求不该探出应用里声明了哪些钩子
if declared is None:
raise HookRejected("应用当前的发布版本里没有声明这个钩子,不接收请求", status_code=404)
@@ -203,40 +211,17 @@ async def _bind_signature(hook_id: str, ts: str, signature: str, idem_key: str)
async def _claim(hook_id: str, idem_key: str) -> tuple[dict[str, Any] | None, bool]:
"""按 (钩子, 幂等键) 认领这一次,交回 (上次的结果, 是不是重新认领了一条停在处理中的)。处理成功过的交回上次的结果
(duplicate: true);被拒过的重新认领;正在处理的 409,停在处理中超过 STALE_S 的重新认领(条件 UPDATE,并发的两次
只有一次认领得到)。"""
from datetime import timedelta
async with SessionLocal() as session:
session.add(HookReceipt(hook_id=hook_id, idem_key=idem_key, status="processing"))
try:
await session.commit()
return None, False
except IntegrityError:
await session.rollback()
row = await session.scalar(select(HookReceipt).where(HookReceipt.hook_id == hook_id,
HookReceipt.idem_key == idem_key))
if row.status == "accepted":
return {"status": "accepted", "duplicate": True, "run_id": row.run_id, "task_id": row.task_id}, False
stale = row.status == "processing" and row.received_at < utcnow() - timedelta(seconds=STALE_S)
if row.status == "rejected" or stale:
again = await session.execute(update(HookReceipt).where(
HookReceipt.id == row.id, HookReceipt.status == row.status,
HookReceipt.received_at == row.received_at)
.values(status="processing", detail=None, received_at=utcnow()))
await session.commit()
if again.rowcount == 1:
return None, stale
raise HookRejected("同一个幂等键的请求正在处理,请稍后再查", status_code=409)
(duplicate: true);被拒过的重新认领;正在处理的 409,停在处理中超过 STALE_S 的重新认领。状态机和访问密钥的发起
共用(process/receipts.py)。"""
try:
return await _BOOK.claim(hook_id, idem_key)
except receipts.ReceiptBusy:
raise HookRejected("同一个幂等键的请求正在处理,请稍后再查", status_code=409) from None
async def _finish(hook_id: str, idem_key: str, *, status: str, detail: str | None = None,
run_id: str | None = None, task_id: str | None = None) -> None:
async with SessionLocal() as session:
await session.execute(update(HookReceipt).where(HookReceipt.hook_id == hook_id,
HookReceipt.idem_key == idem_key)
.values(status=status, detail=detail, run_id=run_id, task_id=task_id))
await session.commit()
await _BOOK.finish(hook_id, idem_key, status=status, detail=detail, run_id=run_id, task_id=task_id)
def _caller(hook: AppHook) -> Any:
@@ -244,7 +229,7 @@ def _caller(hook: AppHook) -> Any:
from app.api.deps import Principal, auth_mode
return Principal(username=None, role="admin", display_name=f"外部系统({hook.name})", verified=True,
mode=auth_mode())
mode=auth_mode(), via="hook")
async def _start(hook: AppHook, request: HookRequest, entry: Any, *, reclaimed: bool = False) -> dict[str, Any]:
@@ -379,3 +364,53 @@ async def delete_hook(session: AsyncSession, app_id: str, name: str) -> bool:
for row in rows:
await session.delete(row)
return bool(rows)
# --------------------------------------------------------------------------
# 收件记录(给 GET /api/apps/{id}/hooks/{name}/receipts 用,docs/design/next-round/04-对外集成与触达.md A6)
#
# 外部系统说「我发过了」,管理应用的人要查得到收到没有、为什么被拒。验过签名的请求在 hook_receipts 里本来就有一行;
# 验签失败的不落库(匿名的人能拿它把库刷满),只在进程里按钩子记最近一小时的次数,界面上写「近 1 小时验签失败 N 次」。
# 单进程部署(同登录失败退避),重启清零
# --------------------------------------------------------------------------
#: 验签失败按多长时间算(秒)
SIGNATURE_WINDOW_S = 3600
#: 一个钩子最多记多少次失败的时间:被刷的时候只要知道「很多」,不用无限长
_FAILURES_MAX = 10_000
#: 钩子 id → 验签失败的时刻(time.monotonic)
_SIGNATURE_FAILURES: dict[str, deque[float]] = {}
#: 一页收件记录最多几条
RECEIPTS_MAX = 100
def _signature_failed(hook_id: str) -> None:
times = _SIGNATURE_FAILURES.setdefault(hook_id, deque(maxlen=_FAILURES_MAX))
times.append(time.monotonic())
def signature_failures(hook_ids: list[str]) -> int:
"""这几个钩子(同名多出来的行一起算)最近一小时验签失败了几次。顺带丢掉窗口外的。"""
floor = time.monotonic() - SIGNATURE_WINDOW_S
total = 0
for hook_id in hook_ids:
times = _SIGNATURE_FAILURES.get(hook_id)
while times and times[0] < floor:
times.popleft()
total += len(times or ())
return total
async def recent_receipts(session: AsyncSession, app_id: str, name: str,
limit: int = RECEIPTS_MAX) -> dict[str, Any]:
"""这个名字的钩子最近收到的请求(新的在前):幂等键、状态(处理中 / 已接受 / 被拒)、时间、发起的实例或解决的待办、
被拒的原因;另给近 1 小时验签失败的次数。防换键重放的签名记录(status=signature)不是一次请求,不列。"""
ids = list((await session.execute(select(AppHook.id).where(AppHook.app_id == app_id, AppHook.name == name)))
.scalars())
rows = list((await session.execute(
select(HookReceipt).where(HookReceipt.hook_id.in_(ids or [""]), HookReceipt.status != "signature")
.order_by(HookReceipt.received_at.desc()).limit(max(1, min(limit, RECEIPTS_MAX))))).scalars())
return {"items": [{"idem_key": r.idem_key, "status": r.status,
"received_at": r.received_at.isoformat() if r.received_at else None,
"run_id": r.run_id, "task_id": r.task_id, "detail": r.detail} for r in rows],
"signature_failures": signature_failures(ids), "window_s": SIGNATURE_WINDOW_S}
+92
View File
@@ -0,0 +1,92 @@
"""按幂等键去重的收件记录(docs/design/next-round/04-对外集成与触达.md A4):入站钩子(hook_receipts)和访问密钥
(api_receipts)共用这一套状态机,各存各的表(Book 按表参数化)。
一条记录按(谁, 幂等键)唯一:钩子按钩子,访问密钥按拥有者(轮换了密钥,同一个键照样认得出是同一件事)。
- claim:先插一行 processing。插得进去就是头一回,交回 (None, False);插不进去(撞了唯一约束)看已有的那一行:
- accepted:交回上次的结果(duplicate: true),不再处理;
- rejected:重新认领。被拒的(表单不对、没有在等的待办)可以用同一个键重试——外部系统多半会原样重发,拒过一次就永远拒,
它只能换键,去重也就落空了;
- processing 超过 STALE_S:处理一次请求用不了这么久,多半是中途服务停了,重新认领,交回 reclaimed=True。调用方先找这个键
已经办成的事(已经发起的实例、已经解决的待办),找到就交回它,不重复办;
- processing 没超时:正在处理,抛 ReceiptBusy(调用方回 409)。
重新认领用条件 UPDATE(状态、收到时间都对得上才改):并发的两次只有一次认领得到。
- finish:处理完写结果(accepted 带运行或待办,rejected 带原因)。
有指纹的表(api_receipts.fingerprint)另核对同一个键办的是不是同一件事:办成过的、还在办的,指纹对不上抛 ReceiptMismatch,
不把上一件事的结果当成这一件的交回去;被拒过的不算办过,换了内容重试照常认领(指纹换成这一次的)。
"""
from __future__ import annotations
from dataclasses import dataclass
from datetime import timedelta
from typing import Any
from sqlalchemy import select, update
from sqlalchemy.exc import IntegrityError
from app.db.base import SessionLocal, utcnow
#: 回执停在 processing 超过这么久(秒),同一个幂等键可以重新认领
STALE_S = 600
#: 幂等键最长存多少字
KEY_MAX = 200
class ReceiptBusy(Exception):
"""同一个幂等键的请求正在处理(调用方回 409)。"""
class ReceiptMismatch(Exception):
"""同一个幂等键已经用来办过另一件事(指纹对不上,调用方回 422)。"""
@dataclass(frozen=True)
class Book:
"""一张收件记录表:model 是表,owner 是「谁」那一列的名字(hook_id / owner_user_id)。表里要有 idem_key、status、
detail、run_id、task_id、received_at;fingerprint 列可有可无。"""
model: Any
owner: str
stale_s: int = STALE_S
def _key(self, owner: str, idem_key: str) -> list[Any]:
return [getattr(self.model, self.owner) == owner, self.model.idem_key == idem_key]
async def claim(self, owner: str, idem_key: str, *, fingerprint: str | None = None,
extra: dict[str, Any] | None = None) -> tuple[dict[str, Any] | None, bool]:
"""按(谁, 幂等键)认领这一次,交回 (上次的结果, 是不是重新认领了一条停在处理中的)。规则见模块说明。extra 是
认领时一并记下的列(访问密钥记哪一把密钥、哪个接口)。"""
fields = {**(extra or {}), **({"fingerprint": fingerprint} if fingerprint is not None else {})}
async with SessionLocal() as session:
session.add(self.model(**{self.owner: owner}, idem_key=idem_key, status="processing", **fields))
try:
await session.commit()
return None, False
except IntegrityError:
await session.rollback()
row = await session.scalar(select(self.model).where(*self._key(owner, idem_key)))
if row is None:
# 撞了约束、回头又查不到:那一行刚被删掉(钩子被删除时连同记录一起删)。当成正在处理,让调用方稍后重发
raise ReceiptBusy
known = getattr(row, "fingerprint", None)
if fingerprint is not None and known and known != fingerprint and row.status != "rejected":
raise ReceiptMismatch
if row.status == "accepted":
return {"status": "accepted", "duplicate": True, "run_id": row.run_id, "task_id": row.task_id}, False
stale = row.status == "processing" and row.received_at < utcnow() - timedelta(seconds=self.stale_s)
if row.status == "rejected" or stale:
again = await session.execute(update(self.model).where(
self.model.id == row.id, self.model.status == row.status,
self.model.received_at == row.received_at)
.values(status="processing", detail=None, received_at=utcnow(), **fields))
await session.commit()
if again.rowcount == 1:
return None, stale
raise ReceiptBusy
async def finish(self, owner: str, idem_key: str, *, status: str, detail: str | None = None,
run_id: str | None = None, task_id: str | None = None) -> None:
async with SessionLocal() as session:
await session.execute(update(self.model).where(*self._key(owner, idem_key))
.values(status=status, detail=detail, run_id=run_id, task_id=task_id))
await session.commit()
+51 -4
View File
@@ -212,6 +212,42 @@ async def _assign(session: AsyncSession, task: Task, run: Run | None, assignee:
#: 蓝图试运行的运行来源(engine/origin.py):待办派给发起人自己、调度不处理时限
TRIAL_ORIGIN = "trial"
#: 服务账号(users.kind,docs/design/next-round/04-对外集成与触达.md A1):只持有访问密钥、回复不了待办,所以不派给它——
#: 组展开时跳过、发起人是它时交给任何人、表达式里点了它的名字跳过、改派给它拒收
SERVICE_KIND = "service"
async def _service_accounts(session: AsyncSession, names: list[str]) -> set[str]:
"""这几个用户名里哪些是服务账号。"""
wanted = sorted({n for n in names if n})
if not wanted:
return set()
return set((await session.execute(select(User.username).where(
User.username.in_(wanted), User.kind == SERVICE_KIND))).scalars())
def key_trace(principal: Any) -> dict[str, Any]:
"""用访问密钥回复时 task.responded 另记的:via=key、哪一把、叫什么(auth/keys.trace)。别的认法什么都不加。"""
from app.auth.keys import trace
return trace(principal)
def _refuse_key(task: Task, principal: Any) -> None:
"""用访问密钥回复待办(04 细案 A4,集成-I3):只给个人访问密钥,只限审批、补充输入、编辑稿件三种。确认写入、核实写入
结果、双盲复算裁决、工具和代码审批要人在界面上看过确认卡再按,一律 403(key_task_kind)。表单里有文件控件的也不行
(文件要先在应用里上传,422 key_files_unsupported)。不是访问密钥认出来的不管。"""
if getattr(principal, "via", None) != "key":
return
from app.auth import keys
from app.files import store as files_store
if task.kind not in keys.TASK_KINDS:
raise TaskRefused("这种待办不能用访问密钥回复:确认写入、核实写入结果、双盲复算裁决、工具和代码审批要在界面上看过"
"确认卡再处理", status_code=403, code=keys.KEY_TASK_KIND)
if files_store.file_fields(task.form):
raise TaskRefused("这条待办的表单里要传文件,不能用访问密钥回复:请在应用里处理", status_code=422,
code=keys.KEY_FILES_UNSUPPORTED)
async def _targets(session: AsyncSession, task: Task, run: Run | None, assignee: Any) -> list[tuple[str, str, str]]:
@@ -232,10 +268,15 @@ async def _targets(session: AsyncSession, task: Task, run: Run | None, assignee:
else:
out.append(("anyone", "", f"slot:{slot}"))
elif kind == "initiator":
out.append(("user", run.started_by, "initiator") if run is not None and run.started_by
else ("anyone", "", "initiator"))
# 用服务账号的访问密钥发起的实例:服务账号不能被派待办(它回复不了),交给「任何人」(搭建者及以上能处理)
starter = run.started_by if run is not None and run.started_by \
and not await _service_accounts(session, [run.started_by]) else None
out.append(("user", starter, "initiator") if starter else ("anyone", "", "initiator"))
elif kind == "expression":
service = await _service_accounts(session, [str(n) for n in spec.get("names") or []])
for name in spec.get("names") or []:
if str(name) in service:
continue
is_group = await session.scalar(select(Group.id).where(Group.name == str(name)))
out += await _expand(session, task, "group" if is_group else "user", str(name), "expression")
elif kind == "hook":
@@ -261,7 +302,8 @@ async def _expand(session: AsyncSession, task: Task, kind: str, name: str, sourc
members = list((await session.execute(
select(User.username).join(GroupMember, GroupMember.user_id == User.id)
.join(Group, Group.id == GroupMember.group_id)
.where(Group.name == name, User.disabled.is_(False)).order_by(User.username))).scalars())
.where(Group.name == name, User.disabled.is_(False), func.coalesce(User.kind, "person") != SERVICE_KIND)
.order_by(User.username))).scalars())
if not members:
return [("group", name, source)]
session.add(AuditEvent(actor=None, verified=False, action="task.group_expanded", subject_kind="task",
@@ -495,6 +537,7 @@ async def respond(task_id: str, principal: Any, *, decision: Any, values: dict[s
task = await session.get(Task, task_id)
if task is None:
raise TaskRefused("待办不存在,所属的运行可能已被删除", status_code=404)
_refuse_key(task, principal)
if task.kind == "reconcile":
return await _respond_reconcile(session, task, principal, decision, values, note)
if task.kind == REVIEW_KIND:
@@ -545,7 +588,8 @@ async def respond(task_id: str, principal: Any, *, decision: Any, values: dict[s
run_id, approval_id, node_id = task.run_id, task.approval_id, task.node_id
await record(run_id, EventType.TASK_RESPONDED, node_id=node_id, task_id=task_id, actor=principal.username,
verified=bool(principal.verified), decision=verdict, index=responded, of=len(voters))
verified=bool(principal.verified), decision=verdict, index=responded, of=len(voters),
**key_trace(principal))
if decided is None:
return {"task": await detail(task_id, principal), "resolved": False, "run": await _run(run_id)}
@@ -728,6 +772,9 @@ async def reassign(task_id: str, principal: Any, *, to: list[dict[str, str]], no
raise TaskRefused(_closed(task.status))
if not await _can_manage(session, task, principal):
raise TaskRefused("只有管理员或这个应用的管理者可以改派待办", status_code=403)
if service := await _service_accounts(session, [n for k, n in targets if k == "user"]):
raise TaskRefused(f"「{'、'.join(sorted(service))}」是服务账号,不能被派待办:请改派给人或组", status_code=422,
code="reassign_service")
await session.execute(update(TaskAssignment).where(TaskAssignment.task_id == task_id,
TaskAssignment.status == "pending")
.values(status="withdrawn"))