mirror of
https://github.com/ThearchyHelios/AgentLab.git
synced 2026-11-15 04:21:57 +00:00
ThearchyHelios
39e7a5b014
阶段①。产品原先能取数的只有 http_request 和沙箱里自己写代码——面对"分析上季度
华东区销量下滑"这类需求，用户得自己知道数据在哪张表、字段叫什么，再手写 SQL 塞进
代码节点。最难的部分留给了用户。
## 连接层（app/data/）
方言差异全收敛在 engine.py 一处：Oracle 要 service_name/sid、MySQL 要 charset、
各家默认端口不同。三个驱动都是 asyncio 原生且免装客户端——oracledb 走 thin 模式，
不需要 Instant Client。密码复用 app/core/crypto 的 Fernet，和 Provider 同一套；
build_url 默认把密码替换成星号，日志和错误信息里用这个版本。
introspect.py 产出两种粒度：summary()（表名+代表字段，几百 token，给 Copilot 的
system prompt 用）和 describe_table()（单表完整字段，按需取）。分两种是因为一个
中等业务库全量塞进 prompt 轻松几万 token，既贵又会把真正的需求淹掉。
## 守卫（app/data/guard.py）
这是唯一不能出错的地方：SQL 由模型生成，模型会写出什么事前无法穷举，防线不能
建立在"它应该不会这么干"上。三层：
- 只读判定用**白名单**而非黑名单——黑名单永远漏，MERGE/CALL/GRANT/各家私货写不完
- 多语句在到达驱动前就断掉，不赌驱动默认只跑第一条（MySQL 开了
CLIENT_MULTI_STATEMENTS 就会全跑）
- 行数/字节/超时三重上限，一条 SELECT * 不该把内存和模型上下文一起打爆
刻意不引 sqlparse：完整解析 SQL 方言是无底洞，而这里的判断很窄——"首关键字是
什么"和"有没有第二条语句"。窄问题用窄解法。难点是不能误杀：`SELECT ';'` 只是
一条语句，简单 split(";") 会把它切两半然后拒掉，那比漏杀更让人无从下手。
## 工具（app/tools/datasource.py）
每个数据源动态生成一对 db_query__<源> / db_schema__<源>，而不是一个通用工具带
datasource 参数：工具名自带语义，模型不会把查销售库的 SQL 发给财务库；description
里直接塞表清单，省掉"先问有哪些表"那一轮。接进 registry.build_tools 作为第四来源
（前三个是内置/MCP/自定义）。
查询结果落 put_json(kind="query_snapshot")，把 SQL 原文 + 结果集 + 数据源一起存，
出具体系的数字回指能下钻到"这个数是哪条 SQL 查出来的"。
## 实测
真实 SQLite 业务库（600 行订单，埋了华东 6 月断崖的信号）+ 真实模型：
agent 8 轮自主查询——列表 → 查三张表结构 → 探数据时间范围 → 查地区维度 →
按月汇总 → 按地区+月份下钻，准确定位到华东 6 月从 1187 件跌到 354 件。
只读守卫在 agent 路径上同样生效（UPDATE/DROP/多语句全被拒且数据未变）。
测试 55 项：守卫 30 项（含"字符串里的分号不能误杀"）、连接与探查 15 项、
URL 构造 6 项、Copilot 流式 4 项。
313 lines
15 KiB
Python
313 lines
15 KiB
Python
from __future__ import annotations
|
||
|
||
from datetime import datetime
|
||
from typing import Any
|
||
|
||
from sqlalchemy import Float, ForeignKey, Index, Integer, LargeBinary, String, Text, UniqueConstraint
|
||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||
|
||
from app.db.base import Base, TimestampMixin, new_id
|
||
|
||
# --------------------------------------------------------------------------
|
||
# 设置：模型供应商 / 用户偏好 / MCP / 自定义工具
|
||
# --------------------------------------------------------------------------
|
||
|
||
|
||
class Provider(Base, TimestampMixin):
|
||
"""一个模型接入点。api_key 落库前经 app.core.crypto 加密。"""
|
||
|
||
__tablename__ = "providers"
|
||
|
||
id: Mapped[str] = mapped_column(String(32), primary_key=True, default=new_id)
|
||
name: Mapped[str] = mapped_column(String(100), unique=True)
|
||
kind: Mapped[str] = mapped_column(String(32)) # anthropic | openai | openai_compatible | mock
|
||
base_url: Mapped[str | None] = mapped_column(String(500), default=None)
|
||
api_key: Mapped[str | None] = mapped_column(Text, default=None) # 密文
|
||
models: Mapped[list[Any]] = mapped_column(default=list) # [{id, label, context, pricing}]
|
||
default_model: Mapped[str | None] = mapped_column(String(200), default=None)
|
||
enabled: Mapped[bool] = mapped_column(default=True)
|
||
extra: Mapped[dict[str, Any]] = mapped_column(default=dict)
|
||
|
||
|
||
class DataSource(Base, TimestampMixin):
|
||
"""一个数据库接入点。password 落库前经 app.core.crypto 加密，和 Provider 同一套。
|
||
|
||
readonly 默认为真，是刻意的安全基线：这里的 SQL 由模型生成，不是人写的。
|
||
要写库就另建一个显式关掉只读的源，并且写操作会走人工审批——把"能写"这件事
|
||
变成一个需要两次明确动作的决定，而不是默认状态。
|
||
"""
|
||
|
||
__tablename__ = "data_sources"
|
||
|
||
id: Mapped[str] = mapped_column(String(32), primary_key=True, default=new_id)
|
||
name: Mapped[str] = mapped_column(String(100), unique=True)
|
||
kind: Mapped[str] = mapped_column(String(32)) # mysql | postgres | oracle | sqlite
|
||
host: Mapped[str | None] = mapped_column(String(255), default=None)
|
||
port: Mapped[int | None] = mapped_column(Integer, default=None)
|
||
database: Mapped[str | None] = mapped_column(String(255), default=None)
|
||
username: Mapped[str | None] = mapped_column(String(128), default=None)
|
||
password: Mapped[str | None] = mapped_column(Text, default=None) # 密文
|
||
# 驱动差异塞这里：Oracle 的 service_name/sid、MySQL 的 charset、SSL 参数…
|
||
# 不给每种数据库加一列，否则表会长成各家方言的并集
|
||
options: Mapped[dict[str, Any]] = mapped_column(default=dict)
|
||
readonly: Mapped[bool] = mapped_column(default=True)
|
||
# 给 Copilot 看的一句话："销售库，含订单/客户/产品"。它据此判断该查哪个源
|
||
description: Mapped[str] = mapped_column(Text, default="")
|
||
# 探查到的表结构。缓存下来，否则每次建图都要连一次生产库
|
||
schema_cache: Mapped[dict[str, Any]] = mapped_column(default=dict)
|
||
schema_synced_at: Mapped[datetime | None] = mapped_column(default=None)
|
||
enabled: Mapped[bool] = mapped_column(default=True)
|
||
|
||
|
||
class Setting(Base, TimestampMixin):
|
||
"""键值形式的用户设置，前端 Settings 页直接读写。"""
|
||
|
||
__tablename__ = "settings"
|
||
|
||
key: Mapped[str] = mapped_column(String(100), primary_key=True)
|
||
value: Mapped[dict[str, Any]] = mapped_column(default=dict)
|
||
|
||
|
||
class McpServer(Base, TimestampMixin):
|
||
__tablename__ = "mcp_servers"
|
||
|
||
id: Mapped[str] = mapped_column(String(32), primary_key=True, default=new_id)
|
||
name: Mapped[str] = mapped_column(String(100), unique=True)
|
||
transport: Mapped[str] = mapped_column(String(20), default="stdio") # stdio | http
|
||
command: Mapped[str | None] = mapped_column(String(500), default=None)
|
||
args: Mapped[list[Any]] = mapped_column(default=list)
|
||
env: Mapped[dict[str, Any]] = mapped_column(default=dict)
|
||
url: Mapped[str | None] = mapped_column(String(500), default=None)
|
||
enabled: Mapped[bool] = mapped_column(default=True)
|
||
# 最近一次探活的结果与工具清单缓存
|
||
status: Mapped[str] = mapped_column(String(20), default="unknown")
|
||
last_error: Mapped[str | None] = mapped_column(Text, default=None)
|
||
tools_cache: Mapped[list[Any]] = mapped_column(default=list)
|
||
|
||
|
||
class CustomTool(Base, TimestampMixin):
|
||
"""用户自定义工具：HTTP 调用模板，或跑在沙箱里的一段 Python。"""
|
||
|
||
__tablename__ = "custom_tools"
|
||
|
||
id: Mapped[str] = mapped_column(String(32), primary_key=True, default=new_id)
|
||
name: Mapped[str] = mapped_column(String(100), unique=True)
|
||
description: Mapped[str] = mapped_column(Text, default="")
|
||
kind: Mapped[str] = mapped_column(String(20), default="http") # http | python
|
||
parameters: Mapped[dict[str, Any]] = mapped_column(default=dict) # JSON Schema
|
||
config: Mapped[dict[str, Any]] = mapped_column(default=dict)
|
||
enabled: Mapped[bool] = mapped_column(default=True)
|
||
|
||
|
||
# --------------------------------------------------------------------------
|
||
# 工作流与版本
|
||
# --------------------------------------------------------------------------
|
||
|
||
|
||
class Workflow(Base, TimestampMixin):
|
||
__tablename__ = "workflows"
|
||
|
||
id: Mapped[str] = mapped_column(String(32), primary_key=True, default=new_id)
|
||
name: Mapped[str] = mapped_column(String(200))
|
||
description: Mapped[str] = mapped_column(Text, default="")
|
||
graph: Mapped[dict[str, Any]] = mapped_column(default=dict) # {nodes, edges, viewport}
|
||
tags: Mapped[list[Any]] = mapped_column(default=list)
|
||
version: Mapped[int] = mapped_column(Integer, default=1)
|
||
is_template: Mapped[bool] = mapped_column(default=False)
|
||
# 治理状态机：draft（随便改）→ published（有正式版本可跑）→ governed（发布需过治理 lint）
|
||
status: Mapped[str] = mapped_column(String(20), default="draft")
|
||
# 正式运行默认解析到这个版本；为空表示还没发布过
|
||
published_version: Mapped[int | None] = mapped_column(Integer, default=None)
|
||
published_by: Mapped[str | None] = mapped_column(String(100), default=None)
|
||
|
||
versions: Mapped[list["WorkflowVersion"]] = relationship(
|
||
back_populates="workflow", cascade="all, delete-orphan", lazy="selectin"
|
||
)
|
||
|
||
|
||
class WorkflowVersion(Base, TimestampMixin):
|
||
"""每次保存留一份快照，便于回滚和对比不同编排的效果。"""
|
||
|
||
__tablename__ = "workflow_versions"
|
||
__table_args__ = (UniqueConstraint("workflow_id", "version"),)
|
||
|
||
id: Mapped[str] = mapped_column(String(32), primary_key=True, default=new_id)
|
||
workflow_id: Mapped[str] = mapped_column(ForeignKey("workflows.id", ondelete="CASCADE"))
|
||
version: Mapped[int] = mapped_column(Integer)
|
||
graph: Mapped[dict[str, Any]] = mapped_column(default=dict)
|
||
note: Mapped[str] = mapped_column(Text, default="")
|
||
# 图内容的 sha256 指纹（剔除 viewport）。formal run 钉住的就是它。
|
||
graph_hash: Mapped[str | None] = mapped_column(String(64), default=None)
|
||
|
||
workflow: Mapped[Workflow] = relationship(back_populates="versions")
|
||
|
||
|
||
# --------------------------------------------------------------------------
|
||
# 运行、事件、审批
|
||
# --------------------------------------------------------------------------
|
||
|
||
|
||
class Run(Base, TimestampMixin):
|
||
__tablename__ = "runs"
|
||
|
||
id: Mapped[str] = mapped_column(String(32), primary_key=True, default=new_id)
|
||
workflow_id: Mapped[str | None] = mapped_column(
|
||
ForeignKey("workflows.id", ondelete="SET NULL"), default=None
|
||
)
|
||
workflow_name: Mapped[str] = mapped_column(String(200), default="")
|
||
# LangGraph checkpointer 的 thread_id —— 持久化与恢复的锚点
|
||
thread_id: Mapped[str] = mapped_column(String(64), index=True, default=new_id)
|
||
# queued | running | interrupted | succeeded | failed | cancelled
|
||
status: Mapped[str] = mapped_column(String(20), default="queued", index=True)
|
||
graph: Mapped[dict[str, Any]] = mapped_column(default=dict) # 执行时的图快照
|
||
input: Mapped[dict[str, Any]] = mapped_column(default=dict)
|
||
output: Mapped[dict[str, Any]] = mapped_column(default=dict)
|
||
error: Mapped[str | None] = mapped_column(Text, default=None)
|
||
usage: Mapped[dict[str, Any]] = mapped_column(default=dict) # tokens / cost / 各节点耗时
|
||
started_at: Mapped[datetime | None] = mapped_column(default=None)
|
||
finished_at: Mapped[datetime | None] = mapped_column(default=None)
|
||
last_seq: Mapped[int] = mapped_column(Integer, default=0)
|
||
# formal：从不可变的 WorkflowVersion 发起，可复现、可出具
|
||
# exploratory：画布试跑 / 裸 graph / 追问，结果只标探索性
|
||
run_class: Mapped[str] = mapped_column(String(20), default="exploratory")
|
||
version: Mapped[int | None] = mapped_column(Integer, default=None)
|
||
version_hash: Mapped[str | None] = mapped_column(String(64), default=None)
|
||
# 结束时对全部事件计算的清单哈希，事后改动事件流会被它戳穿
|
||
manifest_hash: Mapped[str | None] = mapped_column(String(64), default=None)
|
||
started_by: Mapped[str | None] = mapped_column(String(100), default=None)
|
||
|
||
events: Mapped[list["RunEvent"]] = relationship(
|
||
back_populates="run", cascade="all, delete-orphan"
|
||
)
|
||
|
||
|
||
class RunEvent(Base):
|
||
"""完整事件流落库 —— 这既是 trace，也是页面刷新后重建时间线的依据。"""
|
||
|
||
__tablename__ = "run_events"
|
||
__table_args__ = (Index("ix_run_events_run_seq", "run_id", "seq"),)
|
||
|
||
id: Mapped[str] = mapped_column(String(32), primary_key=True, default=new_id)
|
||
run_id: Mapped[str] = mapped_column(ForeignKey("runs.id", ondelete="CASCADE"))
|
||
seq: Mapped[int] = mapped_column(Integer, default=0)
|
||
type: Mapped[str] = mapped_column(String(40))
|
||
node_id: Mapped[str | None] = mapped_column(String(64), default=None)
|
||
ts: Mapped[float] = mapped_column(Float)
|
||
data: Mapped[dict[str, Any]] = mapped_column(default=dict)
|
||
|
||
run: Mapped[Run] = relationship(back_populates="events")
|
||
|
||
|
||
class Approval(Base, TimestampMixin):
|
||
"""一次人工介入请求。对应 LangGraph 的 interrupt。"""
|
||
|
||
__tablename__ = "approvals"
|
||
|
||
id: Mapped[str] = mapped_column(String(32), primary_key=True, default=new_id)
|
||
run_id: Mapped[str] = mapped_column(ForeignKey("runs.id", ondelete="CASCADE"), index=True)
|
||
node_id: Mapped[str] = mapped_column(String(64))
|
||
interrupt_id: Mapped[str | None] = mapped_column(String(64), default=None)
|
||
mode: Mapped[str] = mapped_column(String(20), default="approve") # approve | input | edit
|
||
title: Mapped[str] = mapped_column(String(300), default="")
|
||
payload: Mapped[dict[str, Any]] = mapped_column(default=dict) # 给人看的上下文
|
||
schema_: Mapped[dict[str, Any]] = mapped_column("schema", default=dict)
|
||
status: Mapped[str] = mapped_column(String(20), default="pending", index=True)
|
||
response: Mapped[dict[str, Any]] = mapped_column(default=dict)
|
||
resolved_at: Mapped[datetime | None] = mapped_column(default=None)
|
||
# 责任归属：谁批的。没有 actor 的治理是空转。
|
||
resolved_by: Mapped[str | None] = mapped_column(String(100), default=None)
|
||
|
||
|
||
class Artifact(Base, TimestampMixin):
|
||
"""工件引用表。
|
||
|
||
文件本体按内容哈希存在 data/artifacts/ 下（同内容只一份）；
|
||
这张表记录"哪个 run 的哪个节点产出了它"——溯源需要的正是这层归属。
|
||
复合主键：同一工件被多个 run 产出时各记一行引用。
|
||
"""
|
||
|
||
__tablename__ = "artifacts"
|
||
__table_args__ = (Index("ix_artifacts_run", "run_id"),)
|
||
|
||
id: Mapped[str] = mapped_column(String(64), primary_key=True)
|
||
run_id: Mapped[str] = mapped_column(String(32), primary_key=True, default="")
|
||
kind: Mapped[str] = mapped_column(String(32), default="node_output") # node_output | tool_snapshot | metric_set
|
||
node_id: Mapped[str | None] = mapped_column(String(64), default=None)
|
||
size: Mapped[int] = mapped_column(Integer, default=0)
|
||
meta: Mapped[dict[str, Any]] = mapped_column(default=dict)
|
||
|
||
|
||
# --------------------------------------------------------------------------
|
||
# 记忆、知识库、Skill
|
||
# --------------------------------------------------------------------------
|
||
|
||
|
||
class MemoryItem(Base, TimestampMixin):
|
||
"""长期记忆。scope 用来隔离不同 agent / 会话的记忆空间。"""
|
||
|
||
__tablename__ = "memories"
|
||
__table_args__ = (Index("ix_memories_scope_kind", "scope", "kind"),)
|
||
|
||
id: Mapped[str] = mapped_column(String(32), primary_key=True, default=new_id)
|
||
scope: Mapped[str] = mapped_column(String(100), default="default")
|
||
kind: Mapped[str] = mapped_column(String(32), default="fact") # fact | preference | episode
|
||
content: Mapped[str] = mapped_column(Text)
|
||
meta: Mapped[dict[str, Any]] = mapped_column(default=dict)
|
||
embedding: Mapped[bytes | None] = mapped_column(LargeBinary, default=None)
|
||
importance: Mapped[float] = mapped_column(Float, default=0.5)
|
||
last_used_at: Mapped[datetime | None] = mapped_column(default=None)
|
||
use_count: Mapped[int] = mapped_column(Integer, default=0)
|
||
|
||
|
||
class Document(Base, TimestampMixin):
|
||
"""知识库里的一份原始文档。"""
|
||
|
||
__tablename__ = "documents"
|
||
|
||
id: Mapped[str] = mapped_column(String(32), primary_key=True, default=new_id)
|
||
collection: Mapped[str] = mapped_column(String(100), default="default", index=True)
|
||
title: Mapped[str] = mapped_column(String(300), default="")
|
||
source: Mapped[str] = mapped_column(String(500), default="") # 文件名 / URL
|
||
mime: Mapped[str] = mapped_column(String(100), default="text/plain")
|
||
content: Mapped[str] = mapped_column(Text, default="")
|
||
meta: Mapped[dict[str, Any]] = mapped_column(default=dict)
|
||
chunk_count: Mapped[int] = mapped_column(Integer, default=0)
|
||
|
||
chunks: Mapped[list["Chunk"]] = relationship(
|
||
back_populates="document", cascade="all, delete-orphan"
|
||
)
|
||
|
||
|
||
class Chunk(Base):
|
||
__tablename__ = "chunks"
|
||
__table_args__ = (Index("ix_chunks_collection", "collection"),)
|
||
|
||
id: Mapped[str] = mapped_column(String(32), primary_key=True, default=new_id)
|
||
document_id: Mapped[str] = mapped_column(ForeignKey("documents.id", ondelete="CASCADE"))
|
||
collection: Mapped[str] = mapped_column(String(100), default="default")
|
||
ordinal: Mapped[int] = mapped_column(Integer, default=0)
|
||
content: Mapped[str] = mapped_column(Text)
|
||
embedding: Mapped[bytes | None] = mapped_column(LargeBinary, default=None)
|
||
meta: Mapped[dict[str, Any]] = mapped_column(default=dict)
|
||
|
||
document: Mapped[Document] = relationship(back_populates="chunks")
|
||
|
||
|
||
class Skill(Base, TimestampMixin):
|
||
"""可复用的方法论：一段指令 + 可选的示例和推荐工具。
|
||
|
||
节点可以挂载 skill，运行时把内容注入 system prompt —— 把"怎么做事"
|
||
从工作流结构里抽出来单独管理和迭代。
|
||
"""
|
||
|
||
__tablename__ = "skills"
|
||
|
||
id: Mapped[str] = mapped_column(String(32), primary_key=True, default=new_id)
|
||
name: Mapped[str] = mapped_column(String(100), unique=True)
|
||
description: Mapped[str] = mapped_column(Text, default="")
|
||
instructions: Mapped[str] = mapped_column(Text, default="")
|
||
examples: Mapped[list[Any]] = mapped_column(default=list)
|
||
suggested_tools: Mapped[list[Any]] = mapped_column(default=list)
|
||
tags: Mapped[list[Any]] = mapped_column(default=list)
|
||
enabled: Mapped[bool] = mapped_column(default=True)
|