mirror of
https://github.com/ThearchyHelios/AgentLab.git
synced 2026-11-19 11:01:58 +00:00
可点击证据第二期:取数链路打通、问数据的答案逐段可点;发布前自动修复
一、可点击证据第二期
取数链路
- 智能体、工具节点、代码节点、知识检索取数时都记进证据台账;报告目录按台账编出 Q1… / K1…
- 报告可以引用快照里的一格([[v:Q1.r0.gmv]]),也可以由系统从快照生成整张表([[table:Q1 cols=… rows=0-4]]),
表里每一格都是可点的片段;取值一律以快照为准,取回快照时复验哈希
- 口径卡表达式新增 cell(x, row, column),lineage 精确到单元格
- 智能体开 cite_fields(配合 output_schema)后,循环结束做一次结构化抽取,每个字段标出取自哪次查询的哪一格,
再到快照里核对:一致记 verified;不一致以快照为准并告警;查不到的记为空值,永远不兜底成 0
- 代码节点新增 evidence_role(source / compute)和 code_sha;沙箱算出来的数不能被报告直接引用,要进口径卡
- 查询快照记下列类型,文本列按文本渲染(年份 "2026" 不再显示成「2,026」)
- 升级前发起的运行(停在审批上的)一律走旧逻辑:新的 task 只追加在循环之后,恢复时不会错号
证据接口与面板
- 片段链接上查询步骤:SQL、被引用的行加前后各 2 行、高亮到格;数据源可设 mask_columns 遮罩列,
遮罩在查询时记进快照,数据源事后改名或删除也照样遮罩(在有身份体系之前,遮罩只减少暴露,不是安全边界)
- 输入来源写明核对结果(与快照一致 / 模型报的值和快照不一致 / 没查到记为空);口径卡缺输入时也给出完整的链
- 受管出具要在契约里声明 cells 才能引用单元格;受管与否记在所钉版本上,改画布不再让受管版本的正式运行降级
- 口径卡可以用 caliber_from 钉住另一个已发布工作流里的口径卡,上游有新版本时按 upgrade_policy 处置,
和子工作流同一套规则;子工作流节点也补上了升版处置的配置项
Copilot、校验与报错
- 问数据的答案里有数字时,搭 input → 智能体 → 报告 → 出口,报告直接引用查过的单元格
- 写 SQL 前核对数据源的表结构,SQL 里出现表结构里没有的列会被打回
- 不再搭「智能体文字 → 转换节点解析 JSON」这种图;转换节点的 JSON 报错分清是上游模型写坏了、
模板里裸插了模型文字(给出 | json 的改法),还是模板本身写错了
- 工具节点遇到「查询失败」「SQL 被拒绝」判为失败;发起运行前检查绑定的工具是否存在,不存在时直接拒绝并给出入口
二、发布前自动修复
- 问题带上机器编号(code)和修复 id;新增只读接口 POST /api/workflows/{id}/publish-check 和 /autofix
- 修复分三类:答案唯一、不降低要求的一键修复;要人拿主意的给选项(不替人选);结构性问题交给 Copilot
- 一律先预览,不自动保存、不自动发布;删节点、删契约、放宽审批、关掉 strict、改级别等降低要求的改动一律拒绝,
Copilot 的方案也要过这一关;应用后重跑检查,错误必须变少且不能冒出新的
- 发布弹窗打开就先检查;问题面板新增「发布前检查」,平时就能看到、就地修掉
- 门禁修正:契约有 report_from 时不再要求 narrative(模板⑨可以发成受管);全图默认审批为「全部放行」时同样拦下
- 修掉画布保存会抹掉全图默认(defaults)的老问题:以前任何一次保存都会把它清空,审批 always 因此被悄悄放宽
验证(沙箱):tsc 干净,生产构建通过,pytest 2268 passed(两遍),check-all 17 项检查、3915 项全过
(新增 check-publish);关键行为做了变异验证、全部抓到。
This commit is contained in:
+368
-21
@@ -18,7 +18,7 @@ from app.core.errors import explain as explain_error, graph_error, not_configure
|
||||
from app.db.base import get_session
|
||||
from app.db.models import Workflow
|
||||
from app.engine.layout import CORRIDOR_MIN, MIN_ROW_GAP, NODE_W, _height, auto_layout
|
||||
from app.engine.schema import GraphSpec, NodeType, ValidationIssue, validate_graph
|
||||
from app.engine.schema import GraphSpec, NodeType, ValidationIssue, text_parse_issues, validate_graph
|
||||
from app.engine.state import message_text
|
||||
from app.providers.factory import ModelSpec, ProviderNotConfigured, get_chat_model
|
||||
from app.tools.registry import all_specs
|
||||
@@ -130,7 +130,11 @@ def _datasource_section(rows: list[Any]) -> str:
|
||||
"- 认准方言:Oracle 用 FETCH FIRST n ROWS ONLY,不是 LIMIT\n"
|
||||
"- 一次只写一条语句;分号拼接会被拒\n"
|
||||
"- 字段拿不准就在图里先放一个 db_schema 工具节点,别猜字段名\n"
|
||||
"- 取数结果要进口径卡(metrics 节点)才能被报告撰写节点(report)引用,别让 llm 节点直接对数字做算术\n"
|
||||
# 真实踩过:列名是照着业务说法猜的,跑到查库那一步才「no such column」
|
||||
"- 列名只照上面结构里列出来的写;结构里只列了表名时先用 db_schema 查字段。搭完自查会拿 SQL 里的列名"
|
||||
"去对结构,对不上会打回来让你改\n"
|
||||
"- 比率、增幅、占比这类派生计算进口径卡(metrics 节点);查询结果里原样的数,报告撰写节点(report)"
|
||||
"可以直接用 [[v:Q1.r0.列名]] 引用。别让 llm 节点直接对数字做算术\n"
|
||||
)
|
||||
|
||||
|
||||
@@ -139,7 +143,12 @@ NODE_REFERENCE = """\
|
||||
- input:入口。config.fields = [{name, required, default, description}]
|
||||
- output:出口,收集最终成果。config.fields = [{name, value}],value 里写模板引用
|
||||
- llm:单次模型调用。config: {system, prompt, model, temperature, max_tokens, assign_to, output_schema}
|
||||
- agent:带工具循环的 agent。config: {system, prompt, tools:[工具名], approval, assign_to}
|
||||
- agent:带工具循环的 agent。config: {system, prompt, tools:[工具名], approval, assign_to, output_schema, cite_fields}
|
||||
**agent 要把查到的数交给口径卡或报告时**,config 加 output_schema(JSON Schema,写要交的字段和类型)和
|
||||
cite_fields: true:循环结束后系统按 Schema 抽一次字段,每个字段都核对到它查过的那一格——以查询快照为准,
|
||||
查不到的记空值、不兜底成 0。assign_to 拿到的就是核对过的对象,口径卡直接写 vars.<assign_to>.字段。
|
||||
只写 output_schema 不开 cite_fields 不生效。**不要用 transform 解析 agent / llm 的文字**:模型写的 JSON
|
||||
常夹着没转义的引号,解析一失败整个节点就失败,也核对不了出处
|
||||
**默认不要写 max_steps、budget_tokens、budget_usd**。平台用的是护栏而不是固定步数:
|
||||
步数只是很高的兜底(跟随设置,默认 100),重复调用、连续几步没有新信息、预算用完、
|
||||
上下文快满时,它会按查到的部分收尾。写死一个小数字(见过 8)会让它查完表结构就没额度回答了。
|
||||
@@ -154,6 +163,8 @@ NODE_REFERENCE = """\
|
||||
成员停不下来等人:需要审批的 MCP / 自定义工具在成员手里不会执行。要用这类工具,交给团队外的 agent 节点
|
||||
- tool:直接调一个工具。config: {tool: 工具名, args: {...}, assign_to}
|
||||
- code:沙箱里跑代码。config: {language: python|bash|node, code, timeout, network, assign_to}
|
||||
evidence_role:source(取数:产出本身就是源数据)/ compute(计算,默认)。负责取数的代码写 source;
|
||||
compute 的产出喂给口径卡会被提示「核对不了出处」——业务计算写进口径卡的表达式。
|
||||
assign_to 拿到的是 stdout(尾部的换行已去掉)。**stdout 是 JSON 时会解析成对象**,
|
||||
这时下游要用 {{ vars.x.字段 }} 取字段,拿它比字符串永远不成立。要判一个简单结论,
|
||||
就 print 一个短字符串(比如 print('ok'))并让下游比它。
|
||||
@@ -172,26 +183,33 @@ NODE_REFERENCE = """\
|
||||
- subgraph:嵌套另一个工作流。config: {workflow_id, input}
|
||||
- metrics:口径卡。报告里要出现的数字、以及所有派生计算(比率、增幅、占比、差值)都登记在这里。
|
||||
config: {caliber, caliber_version, metrics:[{id, name, unit, decimals, format, expression}], on_missing, assign_to}
|
||||
expression 是受限表达式(不是模板):vars.q.rows[0][0]、round((vars.q.rows[0][0] - vars.q.rows[0][1]) / vars.q.rows[0][1] * 100, 1)
|
||||
expression 是受限表达式(不是模板):vars.kpi.gmv、cell(nodes.fetch, 0, 'gmv')、
|
||||
round((vars.kpi.gmv - vars.kpi.gmv_prev) / vars.kpi.gmv_prev * 100, 1)
|
||||
format:thousands(默认,千分位)/ plain(不分组,年份编号这类)/ percent_of_ratio(值是 0.0235 这样的比率,显示成 2.35%);
|
||||
decimals 写 -15 到 15 的整数;percent_of_ratio 的 decimals 按比率算:要显示两位百分比(2.35%)写 4,
|
||||
写 2 只剩「2%」,0.004 这样的小比率还会显示不出来、报告引用不了。on_missing 默认 fail(缺输入整张卡失败);写 null 则缺输入的指标记为空值,交给出具契约判档
|
||||
db_query__* 返回的是 JSON 文本 {columns, rows},口径卡取不到里面的字段:先接一个
|
||||
transform(mode=json,template="{{ nodes.取数节点id }}",assign_to="q"),口径卡再写 vars.q.rows[0][0]
|
||||
取 tool 节点查库的结果用 cell(nodes.取数节点id, 行, '列名')(行号从 0 数,列按列名):cell() 直接认查询工具
|
||||
交回的结果,出处精确到快照里的那一格,不用再接 transform 解析。取 agent 的字段写 vars.<agent 的 assign_to>.字段
|
||||
- report:报告撰写(带引用)。config: {instructions, system, metrics_from:[口径卡id], on_violation, max_repairs, assign_to}
|
||||
**凡是给人看的、带数字的报告、分析、结论,都用 report,不要用 llm**。它自动收集上游口径卡的指标和运行输入,
|
||||
模型只能写 [[m:指标id]]、[[i:输入字段]] 这样的引用标记,数值由系统换成口径卡里的真实值;
|
||||
模型自己写的数字会被打回重写。它只能引用口径卡指标和运行输入,所以报告里要出现的每个数都得先在口径卡里登记。
|
||||
**凡是给人看的、带数字的报告、分析、结论,都用 report,不要用 llm**。它自动收集上游的口径卡指标、查询结果
|
||||
(tool 节点和 agent 查过的库,按查询先后编成 Q1、Q2…)和运行输入,模型只能写引用标记,数值由系统从证据里
|
||||
取出来渲染;模型自己写的数字会被打回重写:
|
||||
- [[m:指标id]]:口径卡指标。比率、增幅、占比、差值这类派生计算只能先在口径卡里登记再引用
|
||||
- [[v:Q1.r0.列名]]:第 1 次查询结果里第 0 行那一列的原值(行号从 0 数)
|
||||
- [[table:Q1 cols=列a,列b rows=0-4]]:把查询结果的几行几列原样生成表格,每一格都点得开出处
|
||||
- [[i:输入字段]]:运行输入
|
||||
metrics_from 不写就取所有上游口径卡;on_violation 不写时探索运行是 flag(标出来、照常产出),正式运行是 fail
|
||||
- output 的出具契约:config.contract = {report_from: 报告节点id, metrics_from:[口径卡id],
|
||||
required:[必需指标id], expected:[期望指标id], strict: true}。成果字段写 {{ nodes.报告节点id.text }},
|
||||
前后不要拼别的字——拼了就没法逐段对应证据,出具会降档
|
||||
前后不要拼别的字——拼了就没法逐段对应证据,出具会降档。受管模板的报告要直接引用查询单元格
|
||||
([[v:]] / [[table:]])的,契约里写 cells: true,否则这些引用按解析不了算
|
||||
|
||||
搭图规则(有数字结论时必须遵守):
|
||||
1. 取数(tool 节点查库,接 transform 解析)→ 口径卡 → report → output(配 report_from 契约)。
|
||||
agent 的回答是自由文本,进不了口径卡:要出可追溯的报告,取数用 tool 节点
|
||||
1. 取数(tool 节点查库;或者 agent 查库并配 output_schema + cite_fields)→ 口径卡 → report → output
|
||||
(要三档出具时配 report_from 契约)。没有比率、增幅这类派生计算时可以不要口径卡,report 直接引用查询单元格
|
||||
2. 能用 SUM / COUNT 做的聚合放在 SQL 里,口径卡只做标量运算
|
||||
3. 不要用 code 节点做业务计算;code 只做格式转换
|
||||
3. 不要用 code 节点做业务计算;code 只做格式转换(负责取数的 code 标 evidence_role: source)
|
||||
4. 不要用 transform 解析 agent / llm 的文字:口径卡读 cell(nodes.查库节点id, 行, '列名') 或 vars.<agent 的 assign_to>.字段
|
||||
|
||||
模型字段(llm / agent / supervisor / report 的 config.model):
|
||||
- **不要填**。留空表示跟随当前供应商的默认模型,这几乎总是对的。
|
||||
@@ -279,10 +297,16 @@ def _user_message(payload: GenerateIn, *, patch: bool) -> str:
|
||||
"- 涉及数据的问题必须用 db_query__* / db_schema__* 去真查,"
|
||||
"不要只对问题本身做文字加工(改写、摘要、分类都不是回答)\n"
|
||||
"- 出口节点给出的应当是这个问题的答案本身\n"
|
||||
# 报告撰写节点只能引用口径卡里登记过的数。agent 查到的结论还进不了口径卡,
|
||||
# 普通问数也套上 report 的话,答案里的数会全部被判成没有出处
|
||||
"- 用户要的是报告、周报,或者要能核对每个数的出处时,按搭图规则走 tool 取数 → 口径卡 → "
|
||||
"report → output(配 report_from 契约);普通的问数照旧用 agent,不必加 report\n"
|
||||
# 二期起报告能直接引用 agent 查过的单元格:答案里的每个数都点得开出处,不必为问数硬塞一张口径卡。
|
||||
# 只问表结构、清单的没有数要核对,多一次写作调用只是多等十几秒
|
||||
"- 答案里有数字时,搭 input → agent → report → output:agent 绑 db_query__ 去查,report 的 instructions 写"
|
||||
"「回答:{{ input.question }}」(可以附上 {{ nodes.agent的id.text }} 当分析思路),数用 [[v:Q1.r0.列名]] "
|
||||
"直接引用 agent 查过的单元格,不必经过口径卡;要比率、增幅再在 agent 和 report 之间加口径卡,agent 配 "
|
||||
"output_schema + cite_fields,口径卡读 vars.<assign_to>.字段。出口字段写 {{ nodes.report的id.text }}\n"
|
||||
"- 只问表结构、清单(有哪些表、有哪些字段、列出名单)的,不加 report,agent → output 就行\n"
|
||||
"- 用户要的是周报、要三档出具时,按搭图规则配口径卡和 report_from 契约\n"
|
||||
"- 不要用 transform 解析 agent / llm 的文字:要把 agent 查到的数交给下游,就给它配 output_schema + "
|
||||
"cite_fields\n"
|
||||
# 只把历史放进 prompt 是不够的:模型拿到上一轮的图,默认仍然会
|
||||
# 重新设计一张"更完整"的。而用户说"再按月份拆一下"时要的是上一张
|
||||
# 图改个 SQL——重建出来的那张经常接到另一张表上,答案对不上前一轮。
|
||||
@@ -943,12 +967,20 @@ def _sse(obj: dict[str, Any]) -> str:
|
||||
|
||||
def _blocking_issues(
|
||||
nodes: dict[str, dict[str, Any]], edges: list[dict[str, Any]], scope: set[str] | None = None,
|
||||
*, sources: list[Any] | None = None, baseline: list[dict[str, Any]] | None = None,
|
||||
level: str | None = None,
|
||||
) -> list[dict[str, Any]] | None:
|
||||
"""按运行时同一套校验,挑出会挡住运行的问题。图本身不成形时返回 None。
|
||||
|
||||
限定了数据源时,用了范围外的库也算:用户点了「只查这个库」,拿别的库的数
|
||||
回答他,比跑不起来更糟——答案看起来是对的。
|
||||
|
||||
另有两类运行时不挡、但助手搭图时要打回的写法(authored_issues):SQL 里的列在数据源
|
||||
的结构里查不到、整形节点按 JSON 解析 agent / llm 写的文字。
|
||||
|
||||
level(published / governed)给了的话,那一档发布门禁的 error 也算进来:助手改完的图
|
||||
能跑还不够,要能发布。不给就和以前一样只管能不能跑。
|
||||
|
||||
每条是 ValidationIssue 的字典形状(node_id、field 都在),和 final.issues 一样:
|
||||
界面靠 node_id 定位卡片、靠 field 落到具体的输入框,不用从一行字里抠节点 id。
|
||||
"""
|
||||
@@ -956,8 +988,178 @@ def _blocking_issues(
|
||||
spec = GraphSpec.model_validate({"nodes": list(nodes.values()), "edges": edges})
|
||||
except Exception: # noqa: BLE001 - 结构不合法留给收尾那一步报
|
||||
return None
|
||||
return ([i.model_dump() for i in validate_graph(spec).issues if i.level == "error"]
|
||||
+ _issue_dicts(scope_issues(list(nodes.values()), scope), "datasource_out_of_scope"))
|
||||
out = ([i.model_dump() for i in validate_graph(spec).issues if i.level == "error"]
|
||||
+ _issue_dicts(scope_issues(list(nodes.values()), scope), "datasource_out_of_scope")
|
||||
+ authored_issues(spec, sources or [], baseline))
|
||||
if level:
|
||||
from app.engine.governance import lint_for_publish
|
||||
|
||||
out += [i.model_dump() for i in lint_for_publish(spec, level=level).issues if i.level == "error"]
|
||||
return out
|
||||
|
||||
|
||||
def _changed(spec: GraphSpec, baseline: list[dict[str, Any]] | None) -> set[str]:
|
||||
"""这一轮新加或改过配置的节点。改图时只查这些:用户自己写的、这一轮没碰的 SQL 不替他改。"""
|
||||
before = {n.get("id"): (n.get("data") or {}).get("config") for n in baseline or [] if isinstance(n, dict)}
|
||||
return {n.id for n in spec.nodes if n.id not in before or before[n.id] != n.config}
|
||||
|
||||
|
||||
def authored_issues(spec: GraphSpec, sources: list[Any], baseline: list[dict[str, Any]] | None = None,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""助手搭图时要打回、运行时却不挡的写法,按 error 交回模型改:
|
||||
|
||||
- sql_unknown_column:调用工具节点的 SQL 里有数据源结构(schema_cache)里查不到的列——凭空
|
||||
猜的列名跑起来只会「no such column」,还白跑前面的步骤
|
||||
- parse_model_text:整形节点按 JSON 解析 agent / llm 写的文字(validate 里只是 warning:旧图照跑)
|
||||
"""
|
||||
changed = _changed(spec, baseline)
|
||||
out: list[dict[str, Any]] = []
|
||||
for issue in text_parse_issues(spec):
|
||||
if issue.node_id in changed:
|
||||
# 模型的文字放在模板引号里的那种,改法是 | json,不是换结构化输出
|
||||
tail = "。不要用整形节点解析 agent / llm 的文字" if "cite_fields" in issue.message else ""
|
||||
out.append({**issue.model_dump(), "level": "error", "code": "parse_model_text",
|
||||
"message": issue.message + tail})
|
||||
by_name = {getattr(r, "name", None): r for r in sources}
|
||||
for node in spec.nodes:
|
||||
if node.id not in changed or node.type != NodeType.TOOL:
|
||||
continue
|
||||
source = by_name.get(_source_of(str(node.config.get("tool") or "")) or "")
|
||||
args = node.config.get("args")
|
||||
sql = args.get("sql") if isinstance(args, dict) else None
|
||||
if source is None or not str(node.config.get("tool") or "").startswith("db_query__") \
|
||||
or not isinstance(sql, str):
|
||||
continue
|
||||
found = sql_unknown_columns(sql, source)
|
||||
if not found:
|
||||
continue
|
||||
unknown, known = found
|
||||
listed = "、".join(known[:20]) + (f" 等 {len(known)} 列" if len(known) > 20 else "")
|
||||
out.append({"level": "error", "node_id": node.id, "edge_id": None, "field": "args.sql",
|
||||
"code": "sql_unknown_column",
|
||||
"message": f"SQL 里的 {'、'.join(unknown[:5])} 在数据源「{source.name}」的结构里查不到(这几张表的列:"
|
||||
f"{listed})。照结构里的列名改;拿不准就先用 db_schema__{source.name} 查表结构,"
|
||||
"别凭空猜列名"})
|
||||
return out
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# SQL 里的列名对照数据源结构
|
||||
#
|
||||
# 只做有把握的判断,宁可漏判不可误判:这条会把图打回去让模型改。CTE、FROM 里的子查询、
|
||||
# 引号括起来的标识符、结构里找不到的表、结构没探查过或者探查被截断的,一律不判。
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
#: @ 开头的是变量(@region、@@session),: 开头的是参数,. 后面的是限定名
|
||||
_SQL_IDENT = re.compile(r"(?<![\w$#.:@])([A-Za-z_][A-Za-z0-9_$#]*)(?![\w$#])")
|
||||
_FROM_ITEMS = re.compile(
|
||||
r"\b(?:from|join)\s+(.+?)(?=\bwhere\b|\bgroup\b|\border\b|\bhaving\b|\blimit\b|\bfetch\b|\boffset\b"
|
||||
r"|\bunion\b|\bintersect\b|\bexcept\b|\bon\b|\busing\b|\b(?:left|right|inner|outer|full|cross|natural)\b"
|
||||
r"|\bjoin\b|\bwindow\b|\)|$)", re.I | re.S)
|
||||
_TABLE_ITEM = re.compile(r"^\s*([A-Za-z_][\w$#]*(?:\.[A-Za-z_][\w$#]*)*)(?:\s+(?:as\s+)?([A-Za-z_][\w$#]*))?\s*$",
|
||||
re.I)
|
||||
_ALIAS_AS = re.compile(r"\bas\s+([A-Za-z_][\w$#]*)", re.I)
|
||||
#: 不写 AS 的列别名:「SUM(amount) total,」「amount total FROM」
|
||||
_ALIAS_BARE = re.compile(r"(\)|\b[A-Za-z_][\w$#]*|\d)\s+([A-Za-z_][\w$#]*)\s*(?=,|\bfrom\b)", re.I)
|
||||
_QUALIFIED = re.compile(r"(?<![\w$#.])([A-Za-z_][\w$#]*)\s*\.\s*([A-Za-z_][\w$#]*|\*)(?!\s*\()")
|
||||
_SQL_WORDS = frozenset("""
|
||||
select distinct all from where and or not in is null like ilike between exists case when then else end as on using
|
||||
join inner left right outer full cross natural group by order having limit offset fetch first next rows row only
|
||||
union intersect except asc desc nulls last with over partition range preceding following unbounded current true
|
||||
false interval year month day hour minute second week quarter dow doy epoch date time timestamp datetime zone
|
||||
cast integer int bigint smallint tinyint decimal numeric real float double precision varchar char character text
|
||||
boolean bool signed unsigned binary top percent escape collate any some values current_date current_time
|
||||
current_timestamp localtime localtimestamp sysdate systimestamp rownum level dual unknown filter within
|
||||
at nocase rtrim glob regexp rlike similar to div mod xor isnull notnull minus prior connect start nocycle siblings
|
||||
grouping sets rollup cube ignore respect lateral apply qualify window exclude include ties others no groups
|
||||
for update of share nowait skip locked lock mode array sounds overlaps language query expansion separator
|
||||
charset utf8 utf8mb4 latin1 ascii unicode years months weeks days hours minutes seconds microsecond microseconds
|
||||
millisecond milliseconds year_month day_hour day_minute day_second hour_minute hour_second minute_second
|
||||
isodow isoyear century decade millennium julian timezone timezone_hour timezone_minute
|
||||
sql_calc_found_rows sql_no_cache sql_cache sql_small_result sql_big_result sql_buffer_result high_priority
|
||||
straight_join distinctrow unique recursive tablesample bernoulli sample repeatable pivot unpivot keep ordinality
|
||||
symmetric asymmetric leading trailing both placing force use index into default without local
|
||||
user current_user session_user system_user current_schema current_catalog current_role
|
||||
""".split())
|
||||
|
||||
#: 后面跟的一定是操作数、不会是别名的词:「SELECT created_at FROM」里 created_at 是列。
|
||||
#: END、NULL、TRUE、CURRENT_DATE 这类「一个值到此为止」的词不在里面——「… END flag,」的 flag 是别名
|
||||
_OPERAND_BEFORE = frozenset("""
|
||||
select distinct distinctrow all unique by as and or not xor is in like ilike glob regexp rlike similar to between
|
||||
when then else case on where having exists div mod escape interval collate using any some from join
|
||||
sql_calc_found_rows sql_no_cache sql_cache sql_small_result sql_big_result sql_buffer_result high_priority
|
||||
straight_join
|
||||
""".split())
|
||||
#: 不是列的名字:排序规则名(COLLATE x)、字符集名(USING x、CHARSET x、CHARACTER SET x)
|
||||
_NAME_AFTER = re.compile(
|
||||
r"\b(?:collate|charset|character\s+set|using(?!\s*\())\s+([A-Za-z_][\w$#]*)", re.I)
|
||||
#: DATEADD(dd, …)、DATEDIFF(wk, …) 的第一个参数是日期部分的缩写,不是列。只在这个位置放过:
|
||||
#: 放进关键字表的话,别处打错成 wk、dd 的列名就查不出来了
|
||||
_DATEPART_ARG = re.compile(
|
||||
r"\b(?:dateadd|datediff|datediff_big|datepart|datename|datetrunc|date_bucket|timestampadd|timestampdiff)"
|
||||
r"\s*\(\s*([A-Za-z_][\w$#]*)", re.I)
|
||||
#: 命名窗口:WINDOW w AS (…)、…, w2 AS (…)
|
||||
_WINDOW_NAME = re.compile(r"(?:\bwindow|,)\s*([A-Za-z_][\w$#]*)\s+as\s*\(", re.I)
|
||||
|
||||
|
||||
def sql_unknown_columns(sql: str, source: Any) -> tuple[list[str], list[str]] | None:
|
||||
"""SQL 里用到、数据源结构里查不到的列:(查不到的列, 这几张表的全部列)。判断不了、或者都查得到时返回 None。"""
|
||||
from app.data import introspect as _introspect
|
||||
from app.data.guard import _blank_quoted, first_verb, strip_comments
|
||||
|
||||
cache = getattr(source, "schema_cache", None) or {}
|
||||
if not cache.get("tables") or cache.get("failed") or first_verb(sql) != "select":
|
||||
return None
|
||||
text = re.sub(r"\{\{.*?\}\}", " 0 ", strip_comments(sql), flags=re.S)
|
||||
if '"' in text.replace("''", "") and _blank_quoted(text) != _blank_quoted(text, identifiers=False):
|
||||
return None # 引号括起来的标识符:大小写、空格都可能,不猜
|
||||
text = _blank_quoted(text)
|
||||
if re.search(r"\bwith\b", text, re.I) or re.search(r"\b(?:from|join)\s*\(", text, re.I) or "`" in text \
|
||||
or "[" in text:
|
||||
return None # [方括号] 标识符、数组下标:同样不猜
|
||||
tables: dict[str, dict[str, Any]] = {} # 表名、别名(小写)→ 表结构
|
||||
spans: list[tuple[int, int]] = []
|
||||
for m in _FROM_ITEMS.finditer(text):
|
||||
spans.append(m.span(1))
|
||||
for item in m.group(1).split(","):
|
||||
parsed = _TABLE_ITEM.match(item)
|
||||
if not parsed:
|
||||
return None
|
||||
name, alias = parsed.group(1), parsed.group(2)
|
||||
meta = _introspect.find_table(source, name)
|
||||
if meta is None:
|
||||
return None # 结构里找不到的表(视图、同义词、截断了的结构)不判
|
||||
tables[name.lower()] = tables[name.rsplit(".", 1)[-1].lower()] = meta
|
||||
if alias and alias.lower() not in _SQL_WORDS:
|
||||
tables[alias.lower()] = meta
|
||||
if not tables:
|
||||
return None
|
||||
known = list(dict.fromkeys(c["name"] for meta in {id(m): m for m in tables.values()}.values()
|
||||
for c in meta.get("columns") or []))
|
||||
columns = {c.lower() for c in known}
|
||||
# 「SELECT created_at FROM」里 created_at 前面是关键字,它是列,不是别名
|
||||
aliases = {a.lower() for a in _ALIAS_AS.findall(text)} | {
|
||||
m.group(2).lower() for m in _ALIAS_BARE.finditer(text) if m.group(1).lower() not in _OPERAND_BEFORE} | {
|
||||
a.lower() for a in _WINDOW_NAME.findall(text)}
|
||||
names = {m.start(1) for pattern in (_NAME_AFTER, _DATEPART_ARG) for m in pattern.finditer(text)}
|
||||
unknown: list[str] = []
|
||||
for m in _QUALIFIED.finditer(text):
|
||||
if any(a <= m.start() < b for a, b in spans):
|
||||
continue
|
||||
meta = tables.get(m.group(1).lower())
|
||||
col = m.group(2)
|
||||
if meta is not None and col != "*" and col.lower() not in {c["name"].lower() for c in meta.get("columns") or []}:
|
||||
unknown.append(f"{m.group(1)}.{col}")
|
||||
for m in _SQL_IDENT.finditer(text):
|
||||
word = m.group(1)
|
||||
low = word.lower()
|
||||
if any(a <= m.start() < b for a, b in spans) or low in _SQL_WORDS or low in columns or low in aliases \
|
||||
or low in tables or m.start() in names or re.match(r"\s*[.(]", text[m.end():]) \
|
||||
or text[m.end():m.end() + 1] == "'": # N'…'、X'…'、_utf8mb4'…':字面量的前缀
|
||||
continue
|
||||
unknown.append(word)
|
||||
unknown = list(dict.fromkeys(unknown))
|
||||
return (unknown, known) if unknown else None
|
||||
|
||||
|
||||
def _source_of(tool: str) -> str | None:
|
||||
@@ -1322,7 +1524,7 @@ async def generate_stream(payload: GenerateIn, session: AsyncSession = Depends(g
|
||||
# 循环 / 分支条件里套了 {{ }}),前面的步骤白跑,报错还停在半路
|
||||
repaired = 0
|
||||
for round_no in range(1, _SELF_CHECK_ROUNDS + 1):
|
||||
errors = _blocking_issues(nodes, edges, scope)
|
||||
errors = _blocking_issues(nodes, edges, scope, sources=sources, baseline=baseline)
|
||||
if not errors:
|
||||
break
|
||||
yield _sse({"op": "check", "status": "repairing", "round": round_no, "issues": errors})
|
||||
@@ -1340,7 +1542,7 @@ async def generate_stream(payload: GenerateIn, session: AsyncSession = Depends(g
|
||||
"message": f"自查修正没跑成:{explain_error(e)[0]}", "detail": raw_error(e)})
|
||||
break
|
||||
repaired = round_no
|
||||
remaining = _blocking_issues(nodes, edges, scope)
|
||||
remaining = _blocking_issues(nodes, edges, scope, sources=sources, baseline=baseline)
|
||||
# 工具绑定变化不挡运行,但要在自查这一步就说出来:工具被改没了的图照样
|
||||
# 能跑,只是跑出来的是模型「假设」查过库的答案
|
||||
changes = tool_changes(baseline, list(nodes.values()))
|
||||
@@ -1363,6 +1565,9 @@ async def generate_stream(payload: GenerateIn, session: AsyncSession = Depends(g
|
||||
issues = [i.model_dump() for i in validate_graph(spec).issues]
|
||||
issues += _issue_dicts(scope_issues(list(nodes.values()), scope),
|
||||
"datasource_out_of_scope")
|
||||
# 自查没改好的猜列名留在画布上:运行时不挡(结构缓存可能旧了),但要看得见
|
||||
issues += [{**i, "level": "warning"} for i in authored_issues(spec, sources, baseline)
|
||||
if i["code"] == "sql_unknown_column"]
|
||||
# 跳过的节点要说出来,不能安静地少一步。code 给前端认:这一类要单独
|
||||
# 提示「少了一步」,不能和普通校验警告混在一起
|
||||
issues += [
|
||||
@@ -1393,6 +1598,148 @@ async def generate_stream(payload: GenerateIn, session: AsyncSession = Depends(g
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 发布前自动修复的兜底:确定性修复修不了的 error 交给 Copilot,产出同样只是预览
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
_LEVEL_WORD = {"published": "已发布", "governed": "受管"}
|
||||
|
||||
_ASSIST_RULES = """\
|
||||
硬性约定(违反任何一条,这次修改整个作废):
|
||||
- 不许降低要求:不删节点、不删连线,add_node 不许用图里已有的节点 id(那等于把旧节点删了重建),也不换节点类型;
|
||||
不删掉或清空出具契约,不删 required 里的指标,不把 strict 改成 false,不往 allow_numbers 里加数,不替人打开 cells;
|
||||
审批策略只能往严里改(每次调用都审批 > 仅危险工具需要审批 > 全部自动放行),不放宽、不删掉,更不改成 never;
|
||||
不取消子工作流钉住的版本;发布级别不是你能改的。结构要动(比如换掉协作团队),输出 question 说明怎么拆,不要自己动手
|
||||
- 要人拿主意的,不替人选、不编:required 该包括哪些指标、几张口径卡该用哪张、钉哪个版本、协作团队拆成哪几个
|
||||
固定步骤……每一处输出一行 {"op":"question","node_id":"节点 id","text":"要问的话,把候选列出来"},那一处不改
|
||||
- 只修下面列出的问题,别的不要动
|
||||
- 可用操作只有 update_node(config 只写要改的字段,没写的原样保留)、add_node、add_edge 和 question;
|
||||
不要输出 remove_node、remove_edge、reply
|
||||
- 第一行 plan,最后一行 done,done 的 explanation 用一两句话说清改了什么、为什么"""
|
||||
|
||||
|
||||
def _publish_fix_request(graph: dict[str, Any], errors: list[dict[str, Any]], level: str) -> str:
|
||||
return (
|
||||
"这是当前的工作流:\n"
|
||||
f"{json.dumps(_slim(graph), ensure_ascii=False, indent=2)}\n\n"
|
||||
f"它要发布成「{_LEVEL_WORD.get(level, level)}」级别,发布前检查(和真正发布同一套规则)还拦着下面这些问题:\n"
|
||||
+ "\n".join(f"- {_issue_line(i)}" for i in errors)
|
||||
+ f"\n\n{_ASSIST_RULES}"
|
||||
)
|
||||
|
||||
|
||||
async def assist_publish_fix(
|
||||
session: AsyncSession, graph: dict[str, Any], *, level: str,
|
||||
provider: str | None = None, model: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""把挡住发布的 error 交给 Copilot 修。模型、工具目录、操作流协议都和画布上改图是同一套。
|
||||
|
||||
拿回的操作在副本上应用,再按同一套规则复核(_blocking_issues 带上发布级别:validate、门禁,
|
||||
以及这一轮改过的节点里有没有凭空猜的列名):error 变少、没有新 error、没有降低要求的改动
|
||||
(autofix.forbidden_changes)才采纳,否则整个作废并写明原因。模型认为要人拿主意的,放进
|
||||
questions 原样交给人,不硬改。
|
||||
|
||||
返回 {accepted, reason, summary, questions(一句一条的文字), graph, ops}。不落库。
|
||||
"""
|
||||
from app.engine.autofix import forbidden_changes, judge
|
||||
|
||||
sources = await _sources(session, None)
|
||||
base = copy.deepcopy(graph)
|
||||
before_nodes = {str(n["id"]): n for n in copy.deepcopy(base.get("nodes") or []) if isinstance(n, dict)}
|
||||
before_edges = [e for e in copy.deepcopy(base.get("edges") or []) if isinstance(e, dict)]
|
||||
baseline = list(copy.deepcopy(before_nodes).values())
|
||||
errors = _blocking_issues(before_nodes, before_edges, sources=sources, baseline=baseline, level=level) or []
|
||||
out: dict[str, Any] = {"accepted": False, "reason": None, "summary": "", "questions": [], "graph": graph,
|
||||
"ops": []}
|
||||
if not errors:
|
||||
out["summary"] = "没有挡住发布的错误,不用交给 Copilot"
|
||||
return out
|
||||
try:
|
||||
spec_ = await copilot_model_spec(
|
||||
session, GenerateIn(instruction="发布前自动修复", provider=provider, model=model))
|
||||
chat, _ = await get_chat_model(session, spec_)
|
||||
except ProviderNotConfigured as e:
|
||||
out["summary"] = _unconfigured(e)
|
||||
return out
|
||||
|
||||
system = (
|
||||
"你是一个 agent 工作流编排专家。现在要做的是:让一张工作流过发布前检查,以操作流的方式输出修改。\n\n"
|
||||
f"{NODE_REFERENCE}\n\n可用工具:\n{_tool_catalog(sources)}{_datasource_section(sources)}\n\n"
|
||||
f"{_STREAM_PROTOCOL}\n\n另外可以输出 question 操作:{{\"op\":\"question\",\"node_id\":\"…\",\"text\":\"…\"}}"
|
||||
)
|
||||
nodes = copy.deepcopy(before_nodes)
|
||||
edges = copy.deepcopy(before_edges)
|
||||
tried: list[dict[str, Any]] = []
|
||||
effective: list[dict[str, Any]] = []
|
||||
plan = ""
|
||||
try:
|
||||
async for op in _iter_ops(chat, [("system", system), ("human", _publish_fix_request(base, errors, level))]):
|
||||
kind = op.get("op")
|
||||
if kind in ("thinking", "heartbeat"):
|
||||
continue
|
||||
if kind == "plan":
|
||||
plan = str(op.get("summary") or "")
|
||||
elif kind == "done":
|
||||
out["summary"] = str(op.get("explanation") or "")
|
||||
elif kind in ("question", "reply"):
|
||||
text = str(op.get("text") or "").strip()
|
||||
# 原样交给人看,前面标上是哪个节点的事(界面上认节点标题,不认 id)
|
||||
where = before_nodes.get(str(op.get("node_id") or ""))
|
||||
title = str((where.get("data") or {}).get("label") or where.get("id")) if where else ""
|
||||
if text:
|
||||
out["questions"].append(f"「{title}」:{text}" if title and title not in text else text)
|
||||
else:
|
||||
tried.append(op)
|
||||
if _apply_op(nodes, edges, op):
|
||||
effective.append(op)
|
||||
except Exception as e: # noqa: BLE001 - 修不成就照实说,确定性修复的结果照样交给人
|
||||
out["summary"] = f"Copilot 这一轮没跑完:{explain_error(e)[0]}"
|
||||
return out
|
||||
out["summary"] = out["summary"] or plan
|
||||
if not effective:
|
||||
# 没动图:要么全是要人拿主意的问题,要么它没给出能用的修改。都不算「拒绝」
|
||||
out["summary"] = out["summary"] or ("Copilot 认为这些问题要你来拿主意" if out["questions"]
|
||||
else "Copilot 这一轮没有给出修改")
|
||||
return out
|
||||
|
||||
proposed = _proposed_graph(base, nodes, edges)
|
||||
reasons = forbidden_changes(base, proposed, tried)
|
||||
after = _blocking_issues(_nodes_by_id(proposed), proposed.get("edges") or [], sources=sources,
|
||||
baseline=baseline, level=level)
|
||||
if reasons:
|
||||
out["reason"] = "Copilot 的修改降低了要求,已作废:" + ";".join(reasons)
|
||||
elif after is None:
|
||||
out["reason"] = "Copilot 改完的图结构读不懂,已作废"
|
||||
elif why := judge(errors, after):
|
||||
out["reason"] = f"Copilot 的修改没有让图变得更好,已作废:{why}"
|
||||
else:
|
||||
out.update(accepted=True, graph=proposed, ops=effective)
|
||||
return out
|
||||
|
||||
|
||||
def _nodes_by_id(graph: dict[str, Any]) -> dict[str, dict[str, Any]]:
|
||||
return {str(n["id"]): n for n in graph.get("nodes") or [] if isinstance(n, dict) and n.get("id")}
|
||||
|
||||
|
||||
def _proposed_graph(base: dict[str, Any], nodes: dict[str, dict[str, Any]],
|
||||
edges: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
"""Copilot 改完的整张图:原图里前端的东西(坐标、连线 id、视口)原样留着,新节点找空处摆。"""
|
||||
out = copy.deepcopy(base)
|
||||
out["nodes"] = list(nodes.values())
|
||||
out["edges"] = edges
|
||||
pinned = _pinned_positions(base)
|
||||
fresh = [nid for nid in nodes if nid not in pinned]
|
||||
if fresh:
|
||||
try:
|
||||
laid, _ = _layout_keeping(GraphSpec.model_validate({"nodes": out["nodes"], "edges": edges}), pinned)
|
||||
where = {n.id: n.position.model_dump() for n in laid.nodes}
|
||||
for nid in fresh:
|
||||
nodes[nid]["position"] = where.get(nid, nodes[nid].get("position"))
|
||||
except Exception: # noqa: BLE001 - 摆不好就留在原点,复核那一步会报结构问题
|
||||
pass
|
||||
return out
|
||||
|
||||
|
||||
class FromRunIn(BaseModel):
|
||||
run_id: str
|
||||
name: str = ""
|
||||
|
||||
@@ -19,8 +19,8 @@ from app.core.config import settings
|
||||
from app.core.crypto import encrypt, mask
|
||||
from app.data import introspect as introspect_mod
|
||||
from app.data.engine import (
|
||||
MAX_QUERY_TIMEOUT_S, QUERY_TIMEOUT_OPTION, SUPPORTED_KINDS, build_url, engine_args, engines,
|
||||
query_timeout_problem,
|
||||
MASK_COLUMNS_OPTION, MAX_QUERY_TIMEOUT_S, QUERY_TIMEOUT_OPTION, SUPPORTED_KINDS, build_url, engine_args,
|
||||
engines, mask_columns_problem, query_timeout_problem,
|
||||
)
|
||||
from app.db.base import get_session
|
||||
from app.db.models import DataSource
|
||||
@@ -155,8 +155,16 @@ _TIMEOUT_FIELD = {
|
||||
}
|
||||
|
||||
|
||||
#: 证据面板展示原始行时遮掉的列。每种库都有:遮罩只看查询结果的列名,和方言无关
|
||||
_MASK_FIELD = {
|
||||
"key": MASK_COLUMNS_OPTION, "label": "遮罩的列", "placeholder": "phone, email",
|
||||
"help": "证据面板展示查询结果的原始行时,这些列的值一律显示成「已遮罩」。在有身份体系之前,"
|
||||
"遮罩只减少暴露,不是安全边界:完整快照仍能按工件取到,SQL 里给列起别名也能绕开",
|
||||
}
|
||||
|
||||
|
||||
def _refuse_bad_options(options: dict[str, Any] | None) -> None:
|
||||
problem = query_timeout_problem(options)
|
||||
problem = query_timeout_problem(options) or mask_columns_problem(options)
|
||||
if problem:
|
||||
raise HTTPException(422, problem)
|
||||
|
||||
@@ -175,11 +183,11 @@ async def list_kinds() -> dict[str, Any]:
|
||||
"needs": ["host", "database", "username", "password"],
|
||||
"hint": "字符集默认 utf8mb4,要换就在「高级连接参数」里加 charset",
|
||||
"advanced": [{"key": "charset", "label": "字符集", "placeholder": "utf8mb4"},
|
||||
_TIMEOUT_FIELD]},
|
||||
_TIMEOUT_FIELD, _MASK_FIELD]},
|
||||
{"value": "postgres", "label": "PostgreSQL", "default_port": 5432,
|
||||
"needs": ["host", "database", "username", "password"],
|
||||
"hint": "「schema」留空就用 public",
|
||||
"advanced": [_TIMEOUT_FIELD]},
|
||||
"advanced": [_TIMEOUT_FIELD, _MASK_FIELD]},
|
||||
{"value": "oracle", "label": "Oracle", "default_port": 1521,
|
||||
"needs": ["host", "username", "password"],
|
||||
"hint": "service_name 和 SID 二选一:一般填 service_name,老库只给了 SID 就切到 SID。"
|
||||
@@ -191,10 +199,11 @@ async def list_kinds() -> dict[str, Any]:
|
||||
"help": "和 SID 二选一"},
|
||||
{"key": "sid", "label": "SID", "placeholder": "ORCL", "help": "和 service_name 二选一"},
|
||||
_TIMEOUT_FIELD,
|
||||
_MASK_FIELD,
|
||||
]},
|
||||
{"value": "sqlite", "label": "SQLite(文件)", "default_port": None,
|
||||
"needs": ["database"], "hint": "「数据库文件路径」填 .db 文件的绝对路径",
|
||||
"advanced": [_TIMEOUT_FIELD]},
|
||||
"advanced": [_TIMEOUT_FIELD, _MASK_FIELD]},
|
||||
],
|
||||
"supported": list(SUPPORTED_KINDS),
|
||||
}
|
||||
|
||||
+229
-22
@@ -5,12 +5,18 @@
|
||||
|
||||
- 报告文档:report.checked 事件里的 doc_artifact
|
||||
- 口径卡的指标集:口径卡节点 node.finished 事件里的 evidence 台账
|
||||
- 查询快照:tool.end 事件里的 query_artifact,或者 node.finished.evidence 里的 query 条目
|
||||
- 成果(哪些字段是报告原文):出口节点 node.finished 事件里的 node_output 工件
|
||||
- 运行输入:入口节点 node.finished 事件里的 node_output 工件
|
||||
|
||||
artifacts 表可以事后插行,run.output 是可以改写的一列,封存之后追加的事件不在
|
||||
核对范围里——这三样都不当来源。本期只解析数字层(口径卡指标、运行输入)的链;
|
||||
旧运行按旧契约的位置信息标出 matched(legacy_contract),不按数值猜来源。
|
||||
核对范围里——这三样都不当来源。解析数字层的链:口径卡指标 → 它的输入 → 查询快照里
|
||||
的那一格,或者单元格引用直接到查询快照;旧运行按旧契约的位置信息标出 matched
|
||||
(legacy_contract),不按数值猜来源。
|
||||
|
||||
查询步骤只给被引用的行和前后各 2 行,完整快照仍走 /api/artifacts/{id}。数据源
|
||||
options.mask_columns 里的列换成「已遮罩」——在有身份体系之前,这只减少暴露,不是
|
||||
安全边界:知道工件 id 的人照样能取到整份快照。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -22,8 +28,9 @@ from sqlalchemy import select
|
||||
|
||||
from app.api.coded import CodedHTTPException
|
||||
from app.core.artifact_store import load
|
||||
from app.data.engine import masked_columns
|
||||
from app.db.base import SessionLocal
|
||||
from app.db.models import Run, RunEvent
|
||||
from app.db.models import DataSource, Run, RunEvent, Workflow
|
||||
from app.engine.evidence import (
|
||||
RenderError,
|
||||
find_segment,
|
||||
@@ -47,6 +54,18 @@ LEGACY_NOTE = "旧版出具:按数值匹配,不是显式引用。同一个
|
||||
NONE_NOTE = "这次运行没有报告文档,也没有出具契约,没有可以展示的证据"
|
||||
NO_DOC_NOTE = "这次运行的出具契约用的是引用模式,但报告没有产出可以核对的文档"
|
||||
|
||||
MASKED = "已遮罩"
|
||||
MASK_NOTE = "这些列在数据源里设了遮罩,面板上不显示原值。在有身份体系之前,遮罩只减少暴露,不是安全边界"
|
||||
#: 查询条目的数据源现在找不到了(改名或删掉了):只能按快照里记下的、查询当时的遮罩来遮
|
||||
SOURCE_GONE = "数据源「{source}」现在找不到了(改名或删掉了),按查询当时记下的遮罩处理"
|
||||
#: 查询步骤给被引用的行前后各带几行
|
||||
WINDOW = 2
|
||||
#: 一个查询步骤最多给多少行:数组字段引用了一大段行时,窗口不能把整份快照搬过来
|
||||
MAX_WINDOW_ROWS = 50
|
||||
UNSEALED_QUERY = "这份查询快照不在封存范围内的任何事件里(没有哪次查询在封存前交回过它),不能当证据展示"
|
||||
TAMPERED_QUERY = "查询快照和它的哈希对不上,疑似被改过,不展示其中的行"
|
||||
MISSING_QUERY = "查询快照在工件库里取不回来"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 封存范围
|
||||
@@ -63,6 +82,10 @@ class _Sealed:
|
||||
ledger: set[tuple[str, str]] = field(default_factory=set)
|
||||
#: 节点 id → 它最后一次 node.finished 的产出工件 id
|
||||
outputs: dict[str, str] = field(default_factory=dict)
|
||||
#: 封存范围内交回过的查询快照:工件 id → {node_id, tool, source, via, call_id, exec}
|
||||
queries: dict[str, dict[str, Any]] = field(default_factory=dict)
|
||||
#: 口径卡节点 id → 封存范围内它的 caliber.upgrade 事件(升版处置)
|
||||
upgrades: dict[str, dict[str, Any]] = field(default_factory=dict)
|
||||
|
||||
@property
|
||||
def sealed(self) -> bool:
|
||||
@@ -110,13 +133,24 @@ async def _sealed(run_id: str) -> _Sealed:
|
||||
"sealed": bool(verdict.get("sealed")), "ok": verdict.get("ok"), "manifest_seq": bound,
|
||||
"legacy": bool(verdict.get("legacy", False))})
|
||||
for _, etype, node_id, data in events:
|
||||
if etype != "node.finished" or not node_id or not isinstance(data, dict):
|
||||
if not node_id or not isinstance(data, dict):
|
||||
continue
|
||||
if etype == "tool.end" and data.get("query_artifact"):
|
||||
out.queries.setdefault(str(data["query_artifact"]), {
|
||||
"node_id": node_id, "tool": data.get("tool"), "via": data.get("artifact")})
|
||||
elif etype == "caliber.upgrade":
|
||||
out.upgrades[node_id] = data
|
||||
if etype != "node.finished":
|
||||
continue
|
||||
if data.get("artifact"):
|
||||
out.outputs[node_id] = data["artifact"]
|
||||
for entry in data.get("evidence") or []:
|
||||
if isinstance(entry, dict) and entry.get("artifact"):
|
||||
out.ledger.add((node_id, entry["artifact"]))
|
||||
if entry.get("kind") == "query":
|
||||
out.queries[str(entry["artifact"])] = {
|
||||
"node_id": node_id, **{k: entry[k] for k in ("tool", "source", "via", "call_id", "exec")
|
||||
if entry.get(k) is not None}}
|
||||
return out
|
||||
|
||||
|
||||
@@ -243,25 +277,63 @@ def _graph_of(report: _Report, sealed: _Sealed, inputs: dict[str, Any]) -> tuple
|
||||
|
||||
evidence = []
|
||||
cards: dict[str, dict[str, Any] | None] = {}
|
||||
for alias, entry in (doc.get("catalog") or {}).items():
|
||||
evidence.append({
|
||||
catalog = doc.get("catalog") or {}
|
||||
cells: list[str] = []
|
||||
for _, unit in iter_units(doc):
|
||||
for seg in unit.get("segments") or []:
|
||||
cite = seg.get("cite") or {}
|
||||
if cite.get("kind") == "cell" and cite.get("status") == "resolved":
|
||||
cells.append(_cell_name(cite.get("alias"), cite.get("locator") or {}))
|
||||
for alias, entry in catalog.items():
|
||||
item = {
|
||||
"alias": alias, "eid": entry.get("eid"), "kind": entry.get("kind"), "label": entry.get("label"),
|
||||
"node_id": entry.get("node_id"), "artifact": entry.get("artifact"),
|
||||
"sealed": _entry_sealed(sealed, entry, inputs), "cited_by": cited_by.get(alias, []),
|
||||
"units": units_of.get(alias, []), "report": report.node_id,
|
||||
})
|
||||
}
|
||||
if entry.get("kind") in ("query", "retrieval"):
|
||||
# 查询、检索条目带上是什么、有多大:图上不点开也看得出这是哪次取数
|
||||
item.update({k: entry[k] for k in ("tool", "source", "columns", "rows", "truncated") if k in entry})
|
||||
evidence.append(item)
|
||||
if entry.get("kind") != "metric" or not entry.get("artifact"):
|
||||
continue
|
||||
artifact = entry["artifact"]
|
||||
if artifact not in cards:
|
||||
cards[artifact] = _load_card(artifact)[0]
|
||||
metric = _metric_in(cards[artifact], (entry.get("locator") or {}).get("metric"))
|
||||
for item in (metric or {}).get("inputs") or []:
|
||||
edges.append({"from": alias, "to": item.get("path"), "rel": "input", "node_id": item.get("node_id"),
|
||||
"report": report.node_id})
|
||||
for source in (metric or {}).get("inputs") or []:
|
||||
edge = {"from": alias, "to": source.get("path"), "rel": "input", "node_id": source.get("node_id"),
|
||||
"report": report.node_id}
|
||||
if cell := _input_cell(source, catalog):
|
||||
edge["cell"] = cell
|
||||
cells.append(cell)
|
||||
edges.append(edge)
|
||||
for cell in dict.fromkeys(c for c in cells if c):
|
||||
edges.append({"from": cell, "to": cell.split(".", 1)[0], "rel": "cell_of", "report": report.node_id})
|
||||
return evidence, edges
|
||||
|
||||
|
||||
def _cell_name(alias: Any, locator: dict[str, Any]) -> str:
|
||||
"""Q1.r0.gmv:报告目录里的全局编号加行列,和 [[v:]] 的写法一致。"""
|
||||
if not alias or not isinstance(locator.get("row"), int) or not locator.get("column"):
|
||||
return ""
|
||||
return f"{alias}.r{locator['row']}.{locator['column']}"
|
||||
|
||||
|
||||
def _query_alias(catalog: dict[str, Any], artifact: Any) -> str | None:
|
||||
"""这份快照在报告目录里的全局编号。agent 字段的 ref 是那个节点内部的编号,靠工件对上。"""
|
||||
if not artifact:
|
||||
return None
|
||||
return next((alias for alias, e in catalog.items() if e.get("kind") == "query" and e.get("artifact") == artifact),
|
||||
None)
|
||||
|
||||
|
||||
def _input_cell(item: dict[str, Any], catalog: dict[str, Any]) -> str:
|
||||
"""口径卡的一个输入落在哪一格(cell() 取的、或者 agent 字段核对到的),按全局编号写。"""
|
||||
locator = item.get("locator") or {}
|
||||
return _cell_name(_query_alias(catalog, item.get("artifact")), locator) if item.get("artifact") else ""
|
||||
|
||||
|
||||
def _load_card(artifact: str) -> tuple[dict[str, Any] | None, bool]:
|
||||
try:
|
||||
card = load(artifact)
|
||||
@@ -312,7 +384,7 @@ def _legacy(sealed: _Sealed, issuance: dict[str, Any]) -> dict[str, Any]:
|
||||
|
||||
@router.get("/{run_id}/evidence/segments/{segment_id}")
|
||||
async def evidence_segment(run_id: str, segment_id: str, report: str | None = None) -> dict[str, Any]:
|
||||
"""点开报告里的一个片段:片段、所在的句子、出处链(本期是指标步骤加它的输入)、封存状态。
|
||||
"""点开报告里的一个片段:片段、所在的句子、出处链(指标 → 输入 → 查询快照,或者单元格直接到查询快照)、封存状态。
|
||||
|
||||
一次运行里有好几份报告时用 ?report=<节点 id> 指定;不指定取成果里标注的那一份。
|
||||
"""
|
||||
@@ -333,8 +405,9 @@ async def evidence_segment(run_id: str, segment_id: str, report: str | None = No
|
||||
doc = chosen.doc
|
||||
markdown = str(doc.get("markdown") or "")
|
||||
start, end = unit.get("span") or [0, 0]
|
||||
chain = _chain(seg, doc, sealed)
|
||||
chain = await _chain(seg, doc, sealed)
|
||||
covered = sealed.trusted and all(step.get("sealed") for step in chain if "sealed" in step)
|
||||
masked = list(dict.fromkeys(c for step in chain if step.get("step") == "query" for c in step.get("masked") or []))
|
||||
return {
|
||||
"report": {"node_id": chosen.node_id, "doc_artifact": chosen.doc_artifact},
|
||||
"segment": {"id": seg["id"], "text": seg.get("text"), "kind": seg.get("kind"), "state": seg.get("state"),
|
||||
@@ -348,8 +421,7 @@ async def evidence_segment(run_id: str, segment_id: str, report: str | None = No
|
||||
"violations": [v for v in doc.get("violations") or []
|
||||
if v.get("segment") == seg["id"] or (v.get("unit") == unit.get("id") and not v.get("segment"))],
|
||||
"seal": {"sealed": sealed.sealed, "ok": sealed.seal["ok"], "covered": covered},
|
||||
# 本期没有查询快照的行,也就没有要遮的列;字段先占位,前端不用等二期再改
|
||||
"redacted": {"columns": []},
|
||||
"redacted": {"columns": masked, "note": MASK_NOTE} if masked else {"columns": []},
|
||||
}
|
||||
|
||||
|
||||
@@ -363,11 +435,21 @@ def _choose(reports: list[_Report], wanted: str | None, sealed: _Sealed) -> _Rep
|
||||
return next((r for r in reports if r.node_id == primary.get("report_node")), reports[0])
|
||||
|
||||
|
||||
def _chain(seg: dict[str, Any], doc: dict[str, Any], sealed: _Sealed) -> list[dict[str, Any]]:
|
||||
async def _chain(seg: dict[str, Any], doc: dict[str, Any], sealed: _Sealed) -> list[dict[str, Any]]:
|
||||
cite = seg.get("cite") or {}
|
||||
entry = (doc.get("catalog") or {}).get(cite.get("alias")) or {}
|
||||
masks = _Masks()
|
||||
if _missing_input(cite, entry):
|
||||
# 指标在卡里、只是这次没有值(显示「—」):照样给出指标、输入和查询步骤,面板才指得出是哪个
|
||||
# 输入空了、那一格在哪次查询里。片段上没记定位和 eid,用目录条目的(它们指的是同一个指标)
|
||||
return await _metric_chain(seg, {**cite, "locator": entry.get("locator") or {}, "eid": entry.get("eid")},
|
||||
entry, doc, sealed, masks)
|
||||
if seg.get("state") != "deterministic" or cite.get("status") != "resolved":
|
||||
return []
|
||||
entry = (doc.get("catalog") or {}).get(cite.get("alias")) or {}
|
||||
if cite.get("kind") == "cell":
|
||||
locator = cite.get("locator") or {}
|
||||
return [await _query_step(str(entry.get("artifact") or ""), [(locator.get("row"), locator.get("column"))],
|
||||
doc, sealed, masks)]
|
||||
if cite.get("kind") == "input":
|
||||
name = (cite.get("locator") or {}).get("field")
|
||||
inputs = _input_payloads(sealed)
|
||||
@@ -378,11 +460,17 @@ def _chain(seg: dict[str, Any], doc: dict[str, Any], sealed: _Sealed) -> list[di
|
||||
"sealed": _entry_sealed(sealed, entry, inputs)}]
|
||||
if cite.get("kind") != "metric":
|
||||
return []
|
||||
return _metric_chain(seg, cite, entry, sealed)
|
||||
return await _metric_chain(seg, cite, entry, doc, sealed, masks)
|
||||
|
||||
|
||||
def _metric_chain(seg: dict[str, Any], cite: dict[str, Any], entry: dict[str, Any],
|
||||
sealed: _Sealed) -> list[dict[str, Any]]:
|
||||
def _missing_input(cite: dict[str, Any], entry: dict[str, Any]) -> bool:
|
||||
"""引用的指标存在,只是这次缺输入没有值。引用写错(目录里没有、同名指标有歧义)的不算。"""
|
||||
return (cite.get("kind") == "metric" and cite.get("status") == "unresolved" and entry.get("kind") == "metric"
|
||||
and entry.get("value") is None and entry.get("status") == "missing_input")
|
||||
|
||||
|
||||
async def _metric_chain(seg: dict[str, Any], cite: dict[str, Any], entry: dict[str, Any], doc: dict[str, Any],
|
||||
sealed: _Sealed, masks: "_Masks") -> list[dict[str, Any]]:
|
||||
"""指标步骤:从封存台账里那件 metric_set 工件取回整张卡,按 id 找到指标,再验三件事——
|
||||
eid 能由工件和定位重算出来、卡里的值按同样的格式渲染出来就是报告上的字、这件工件
|
||||
确实记在口径卡节点封存过的台账里。"""
|
||||
@@ -409,10 +497,127 @@ def _metric_chain(seg: dict[str, Any], cite: dict[str, Any], entry: dict[str, An
|
||||
"hash_ok": hash_ok and metric is not None,
|
||||
"render_ok": rendered_now is not None and rendered_now == seg.get("text"),
|
||||
"sealed": sealed.trusted and (str(entry.get("node_id") or ""), artifact) in sealed.ledger,
|
||||
# 口径卡的版本还没纳入升版处置(那一套目前只管钉了版本的子工作流)
|
||||
"caliber_upgrade": None,
|
||||
"source": await _caliber_source((card or {}).get("source")),
|
||||
"caliber_upgrade": _caliber_upgrade(sealed.upgrades.get(str(entry.get("node_id") or ""))),
|
||||
}
|
||||
return [step, *({"step": "input", **item} for item in (metric or {}).get("inputs") or [])]
|
||||
catalog = doc.get("catalog") or {}
|
||||
inputs: list[dict[str, Any]] = []
|
||||
#: 输入指到的查询快照 → 被引用的格;同一份快照只给一个查询步骤,格合在一起高亮
|
||||
wanted: dict[str, list[tuple[Any, Any]]] = {}
|
||||
for item in (metric or {}).get("inputs") or []:
|
||||
one = {"step": "input", **item}
|
||||
artifact_of = item.get("artifact")
|
||||
if artifact_of and item.get("via") in ("tool_cell", "agent_field"):
|
||||
one["query"] = _query_alias(catalog, artifact_of)
|
||||
if cell := _input_cell(item, catalog):
|
||||
one["cell"] = cell
|
||||
wanted.setdefault(str(artifact_of), []).extend(_located(item.get("locator") or {}))
|
||||
inputs.append(one)
|
||||
queries = [await _query_step(a, cells, doc, sealed, masks) for a, cells in wanted.items()]
|
||||
return [step, *inputs, *queries]
|
||||
|
||||
|
||||
def _located(locator: dict[str, Any]) -> list[tuple[Any, Any]]:
|
||||
"""输入的定位 → 被引用的格。单个格 {row, column};数组字段是一段行 {rows:[a,b], column | columns}。"""
|
||||
if isinstance(locator.get("row"), int):
|
||||
return [(locator["row"], locator.get("column"))]
|
||||
rows = locator.get("rows")
|
||||
if isinstance(rows, list) and len(rows) == 2 and all(isinstance(r, int) for r in rows):
|
||||
cols = [locator["column"]] if locator.get("column") else list((locator.get("columns") or {}).values())
|
||||
return [(r, c) for r in range(rows[0], rows[1] + 1) for c in cols]
|
||||
return []
|
||||
|
||||
|
||||
async def _caliber_source(source: Any) -> dict[str, Any] | None:
|
||||
"""钉住别处口径卡(caliber_from)时,定义取自哪个工作流的哪一版;带上工作流现在的名字方便认。"""
|
||||
if not isinstance(source, dict) or not source.get("workflow_id"):
|
||||
return None
|
||||
async with SessionLocal() as session:
|
||||
workflow = await session.get(Workflow, str(source["workflow_id"]))
|
||||
return {**source, "workflow_name": workflow.name if workflow else None}
|
||||
|
||||
|
||||
def _caliber_upgrade(event: dict[str, Any] | None) -> dict[str, Any] | None:
|
||||
"""这张口径卡在正式运行发起时的升版处置:钉在哪一版、上游最新是哪一版、怎么处置的。"""
|
||||
if not isinstance(event, dict):
|
||||
return None
|
||||
return {k: event.get(k) for k in ("workflow_id", "caliber_node", "pinned", "latest", "policy", "policy_label")}
|
||||
|
||||
|
||||
class _Masks:
|
||||
"""一次请求里查过的数据源遮罩:数据源名 → (要遮的列(小写), 数据源现在还在不在)。"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._memo: dict[str, tuple[set[str], bool]] = {}
|
||||
|
||||
async def of(self, source: Any) -> tuple[set[str], bool]:
|
||||
name = str(source or "")
|
||||
if not name:
|
||||
return set(), True
|
||||
if name not in self._memo:
|
||||
async with SessionLocal() as session:
|
||||
row = (await session.execute(select(DataSource).where(DataSource.name == name))).scalar_one_or_none()
|
||||
self._memo[name] = ({c.lower() for c in masked_columns(row.options if row else None)}, row is not None)
|
||||
return self._memo[name]
|
||||
|
||||
|
||||
async def _query_step(artifact: str, cells: list[tuple[Any, Any]], doc: dict[str, Any], sealed: _Sealed,
|
||||
masks: _Masks) -> dict[str, Any]:
|
||||
"""查询步骤:被引用的格所在的行加前后各 WINDOW 行,高亮那几格,遮掉数据源设了遮罩的列。
|
||||
|
||||
快照只认封存范围内的事件交回过的那些(tool.end.query_artifact、node.finished.evidence 的
|
||||
query 条目);目录里写着、事件里追不到的,照实说不认,一行都不给。
|
||||
"""
|
||||
catalog = doc.get("catalog") or {}
|
||||
alias = _query_alias(catalog, artifact)
|
||||
entry = catalog.get(alias, {}) if alias else {}
|
||||
info = sealed.queries.get(artifact) if artifact else None
|
||||
rows_hit = sorted({r for r, _ in cells if isinstance(r, int)})
|
||||
cols_hit = list(dict.fromkeys(str(c) for _, c in cells if c))
|
||||
step: dict[str, Any] = {
|
||||
"step": "query", "alias": alias, "artifact": artifact or None,
|
||||
"node_id": (info or {}).get("node_id") or entry.get("node_id"),
|
||||
"tool": (info or {}).get("tool") or entry.get("tool"),
|
||||
"source": (info or {}).get("source") or entry.get("source"),
|
||||
"sql": None, "columns": [], "rows": [], "row_offset": 0, "row_index": [], "total_rows": None,
|
||||
"truncated": None,
|
||||
"highlight": {"rows": rows_hit, "cols": cols_hit,
|
||||
"cells": [[r, str(c)] for r, c in dict.fromkeys(cells) if isinstance(r, int) and c]},
|
||||
"masked": [], "hash_ok": None, "sealed": False,
|
||||
}
|
||||
if info is None:
|
||||
return {**step, "note": UNSEALED_QUERY}
|
||||
step["sealed"] = sealed.trusted
|
||||
try:
|
||||
snap = load(artifact)
|
||||
step["hash_ok"] = True if snap is not None else None
|
||||
except ValueError:
|
||||
snap, step["hash_ok"] = None, False
|
||||
if not isinstance(snap, dict) or not isinstance(snap.get("rows"), list):
|
||||
return {**step, "note": TAMPERED_QUERY if step["hash_ok"] is False else MISSING_QUERY}
|
||||
columns = [str(c) for c in snap.get("columns") or []]
|
||||
rows = snap["rows"]
|
||||
step["source"] = step["source"] or snap.get("source")
|
||||
# 遮的是「查询当时记下的」和「数据源现在设的」两者之和:事后加的遮罩照样生效,数据源改名、
|
||||
# 删掉了也不会把当时遮着的列亮出来
|
||||
hidden, found = await masks.of(step["source"])
|
||||
recorded = snap.get("mask_columns")
|
||||
hidden = hidden | ({str(c).lower() for c in recorded if isinstance(c, str)} if isinstance(recorded, list) else set())
|
||||
if not found:
|
||||
step["mask_note"] = SOURCE_GONE.format(source=step["source"])
|
||||
masked = {i for i, c in enumerate(columns) if c.lower() in hidden}
|
||||
index = sorted({i for r in rows_hit for i in range(r - WINDOW, r + WINDOW + 1) if 0 <= i < len(rows)})
|
||||
step.update(sql=snap.get("sql"), columns=columns, total_rows=len(rows), truncated=bool(snap.get("truncated")),
|
||||
masked=[columns[i] for i in sorted(masked)],
|
||||
# 查询时从驱动的原始值记下的列类型(老快照没有):文本列里的 "2026" 按文本显示,不当数
|
||||
column_types=snap.get("column_types") if isinstance(snap.get("column_types"), dict) else {})
|
||||
if len(index) > MAX_WINDOW_ROWS:
|
||||
index, step["window_truncated"] = index[:MAX_WINDOW_ROWS], True
|
||||
step["row_index"] = index
|
||||
step["row_offset"] = index[0] if index else 0
|
||||
step["rows"] = [[MASKED if i in masked else v for i, v in enumerate(rows[r])] if isinstance(rows[r], list)
|
||||
else rows[r] for r in index]
|
||||
return step
|
||||
|
||||
|
||||
def _note(seg: dict[str, Any], unit: dict[str, Any]) -> str:
|
||||
@@ -422,6 +627,8 @@ def _note(seg: dict[str, Any], unit: dict[str, Any]) -> str:
|
||||
return "这个数字没有出处:写作者直接写了数字,没有用引用标记,系统没法核对它"
|
||||
if issue == "unresolved_ref":
|
||||
return f"引用解析不了:{(seg.get('cite') or {}).get('reason') or '证据目录里没有它'}"
|
||||
if seg.get("state") == "deterministic" and (seg.get("cite") or {}).get("kind") == "cell":
|
||||
return "这个值由系统从查询快照里取出、按固定规则渲染,没有经过模型转写"
|
||||
if seg.get("state") == "deterministic":
|
||||
return "数字由系统从证据里取出、按口径卡的格式渲染,没有经过模型转写"
|
||||
kind = seg.get("kind")
|
||||
|
||||
+105
-2
@@ -2,6 +2,7 @@ from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import contextlib
|
||||
import logging
|
||||
from datetime import datetime
|
||||
from typing import Any
|
||||
from urllib.parse import unquote
|
||||
@@ -13,15 +14,20 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from sqlalchemy import select as _select
|
||||
|
||||
from app.api.coded import CodedHTTPException
|
||||
from app.core.bus import bus
|
||||
from app.core import artifact_store
|
||||
from app.db.base import SessionLocal, get_session
|
||||
from app.db.models import Approval, Artifact, Run, RunEvent, Workflow, WorkflowVersion
|
||||
from app.engine.governance import unresolved_caliber_upgrades
|
||||
from app.engine.governance import governance_note, unresolved_caliber_upgrades
|
||||
from app.engine.runner import run_manager
|
||||
from app.engine.schema import GraphSpec
|
||||
from app.engine.schema import GraphSpec, NodeType, type_label
|
||||
|
||||
router = APIRouter(prefix="/api/runs", tags=["runs"])
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
#: 发起时发现绑定的工具在本机不存在(见 missing_tools)
|
||||
TOOL_MISSING = "run_tool_missing"
|
||||
|
||||
|
||||
def actor_of(header: str | None) -> str | None:
|
||||
@@ -87,6 +93,7 @@ async def start_run(
|
||||
name = "未保存的工作流"
|
||||
version: int | None = None
|
||||
version_hash: str | None = None
|
||||
governed = False
|
||||
|
||||
if payload.run_class == "formal":
|
||||
# 正式运行的全部语义就这一条规则:必须引用一个不可变的已发布版本。
|
||||
@@ -126,6 +133,8 @@ async def start_run(
|
||||
graph = snapshot.graph
|
||||
version_hash = snapshot.graph_hash or artifact_store.graph_hash(graph)
|
||||
name = workflow.name
|
||||
# 按这一版发布时记下的级别;记级别之前发布的老版本没有,照旧看工作流现在的 status
|
||||
governed = (snapshot.level == "governed") if snapshot.level else workflow.status == "governed"
|
||||
|
||||
# 口径升版是被迫处置的事件:钉住的方法卡有新版本而未声明策略,拒绝启动
|
||||
spec = GraphSpec.model_validate(graph)
|
||||
@@ -143,6 +152,7 @@ async def start_run(
|
||||
name = workflow.name
|
||||
if not graph:
|
||||
raise HTTPException(400, "没有要运行的内容:请求里既没有工作流 id,也没有工作流内容")
|
||||
await _refuse_missing_tools(session, graph)
|
||||
|
||||
from app.api.settings import resolve_run_scope
|
||||
|
||||
@@ -165,6 +175,11 @@ async def start_run(
|
||||
except ValueError as e:
|
||||
raise HTTPException(400, str(e)) from e
|
||||
|
||||
if payload.run_class == "formal":
|
||||
# 按不按受管出具在发起这一刻定下来、记进封存范围:status 反映的是当前画布,中途改图、
|
||||
# 审批停下再恢复时再去读它,报告节点和出口复核可能各读到不同的值
|
||||
await run_manager.note(run.id, "log", **governance_note(
|
||||
governed=governed, workflow_name=name, version=version))
|
||||
# 已声明策略的升版记入事件流,出具物上能看到"这期换口径了、怎么处置的"
|
||||
for ev in upgrade_events:
|
||||
payload_ev = dict(ev)
|
||||
@@ -173,6 +188,94 @@ async def start_run(
|
||||
return run
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 发起前检查绑定的工具
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _bindings(spec: GraphSpec) -> list[tuple[str, str]]:
|
||||
"""图里每一处绑定的工具:(谁绑的, 工具名)。调用工具节点的 tool、Agent 的 tools、协作成员的 tools。"""
|
||||
out: list[tuple[str, str]] = []
|
||||
for node in spec.nodes:
|
||||
who = f"「{node.title}」({type_label(node.type)})"
|
||||
cfg = node.config
|
||||
if node.type == NodeType.TOOL:
|
||||
names = [cfg.get("tool")]
|
||||
out.extend((who, n) for n in names if isinstance(n, str) and n.strip())
|
||||
elif node.type == NodeType.AGENT:
|
||||
out.extend((who, n) for n in cfg.get("tools") or [] if isinstance(n, str) and n.strip())
|
||||
elif node.type == NodeType.SUPERVISOR:
|
||||
for member in cfg.get("agents") or []:
|
||||
if not isinstance(member, dict):
|
||||
continue
|
||||
label = f"「{node.title}」的成员「{member.get('name') or '?'}」"
|
||||
out.extend((label, n) for n in member.get("tools") or [] if isinstance(n, str) and n.strip())
|
||||
# 模板拼出来的工具名({{ vars.tool }})发起时还不知道是谁,交给运行时
|
||||
return [(who, name.strip()) for who, name in out if "{{" not in name]
|
||||
|
||||
|
||||
async def missing_tools(session: AsyncSession, spec: GraphSpec) -> list[tuple[str, str, str]]:
|
||||
"""本机不存在的绑定:(谁绑的, 工具名, 去哪接入:data / tools)。
|
||||
|
||||
解析和运行时 build_tools 一致:内置(静态注册)、mcp:<服务>/<工具>、db_query__ / db_schema__
|
||||
<数据源>、其余按自定义工具。数据源和自定义工具只认启用着的——停用的运行时同样拿不到。
|
||||
MCP 只看服务登记着、启用着没有:连不连得上、工具还在不在,要连一次才知道,发起时不去等它,
|
||||
上次没连上或者工具清单里没有的只在日志里提一句。
|
||||
"""
|
||||
from app.db.models import CustomTool, DataSource, McpServer
|
||||
from app.tools.datasource import QUERY_PREFIX, SCHEMA_PREFIX
|
||||
from app.tools.registry import all_specs
|
||||
|
||||
bindings = _bindings(spec)
|
||||
if not bindings:
|
||||
return []
|
||||
builtin = all_specs()
|
||||
sources = {n for (n,) in await session.execute(select(DataSource.name).where(DataSource.enabled.is_(True)))}
|
||||
custom = {n for (n,) in await session.execute(select(CustomTool.name).where(CustomTool.enabled.is_(True)))}
|
||||
servers = {r.name: r for r in (await session.execute(select(McpServer))).scalars()}
|
||||
missing: list[tuple[str, str, str]] = []
|
||||
for who, name in bindings:
|
||||
if name.startswith("mcp:"):
|
||||
server, _, tool = name[4:].partition("/")
|
||||
row = servers.get(server)
|
||||
if row is None or not row.enabled:
|
||||
missing.append((who, name, "tools"))
|
||||
elif row.status == "error":
|
||||
logger.warning("%s 绑的 %s:MCP 服务 %s 上次没连上,运行时可能拿不到这个工具", who, name, server)
|
||||
elif tool not in ("", "*") and row.tools_cache and tool not in row.tools_cache:
|
||||
logger.warning("%s 绑的 %s:MCP 服务 %s 上次交回的工具清单里没有 %s", who, name, server, tool)
|
||||
elif name in builtin:
|
||||
continue
|
||||
elif name.startswith((QUERY_PREFIX, SCHEMA_PREFIX)):
|
||||
source = name[len(QUERY_PREFIX):] if name.startswith(QUERY_PREFIX) else name[len(SCHEMA_PREFIX):]
|
||||
if source not in sources:
|
||||
missing.append((who, name, "data"))
|
||||
elif name not in custom:
|
||||
missing.append((who, name, "tools"))
|
||||
return missing
|
||||
|
||||
|
||||
async def _refuse_missing_tools(session: AsyncSession, graph: dict[str, Any]) -> None:
|
||||
"""绑定的工具在本机不存在就不发起:以前要跑到那一步才失败,前面的步骤白跑、钱白花。"""
|
||||
try:
|
||||
spec = GraphSpec.model_validate(graph)
|
||||
except Exception: # noqa: BLE001 - 图本身不成形交给 run_manager.start 报
|
||||
return
|
||||
missing = await missing_tools(session, spec)
|
||||
if not missing:
|
||||
return
|
||||
lines = ";".join(f"{who}绑的 {name}" for who, name, _ in missing[:8])
|
||||
more = f"等 {len(missing)} 处" if len(missing) > 8 else ""
|
||||
places = {where for _, _, where in missing}
|
||||
if places == {"data"}:
|
||||
fix = "去数据页接入,或在节点里重新选"
|
||||
elif places == {"tools"}:
|
||||
fix = "去工具页接入,或在节点里重新选"
|
||||
else:
|
||||
fix = "数据源工具去数据页接入、其他工具去工具页接入,或在节点里重新选"
|
||||
raise CodedHTTPException(422, f"绑定的工具在本机不存在:{lines}{more}。{fix}", TOOL_MISSING)
|
||||
|
||||
|
||||
#: workflow_id 取这个值时只要没挂在任何工作流上的运行:画布上没保存就跑的临时图,
|
||||
#: 以及工作流被删掉之后留下来的运行(外键是 SET NULL)
|
||||
UNSAVED = "__none__"
|
||||
|
||||
@@ -12,12 +12,14 @@ from app.api.runs import actor_of
|
||||
from app.core.artifact_store import graph_hash
|
||||
from app.db.base import get_session
|
||||
from app.db.models import Run, Workflow, WorkflowVersion
|
||||
from app.engine.governance import lint_for_publish
|
||||
from app.engine.schema import GraphSpec, validate_graph
|
||||
from app.engine.governance import publish_issues
|
||||
from app.engine.schema import GraphSpec, NodeType, validate_graph
|
||||
|
||||
router = APIRouter(prefix="/api/workflows", tags=["workflows"])
|
||||
|
||||
_GONE = "这个工作流不存在,可能已经被删了"
|
||||
_LEVELS = ("published", "governed")
|
||||
_BAD_LEVEL = "发布档位只能选「已发布」或「受管」"
|
||||
|
||||
|
||||
class WorkflowIn(BaseModel):
|
||||
@@ -181,6 +183,8 @@ class VersionOut(BaseModel):
|
||||
version: int
|
||||
note: str
|
||||
created_at: Any = None
|
||||
#: 这一版按哪一档发布的(published / governed)。没发布过、或者是记级别之前发布的为 null
|
||||
level: str | None = None
|
||||
|
||||
model_config = {"from_attributes": True}
|
||||
|
||||
@@ -238,7 +242,7 @@ async def get_version(
|
||||
id=snapshot.id, version=snapshot.version, note=snapshot.note,
|
||||
created_at=snapshot.created_at, workflow_id=workflow_id, graph=graph,
|
||||
graph_hash=snapshot.graph_hash or graph_hash(graph), input_fields=fields,
|
||||
published=workflow.published_version == snapshot.version,
|
||||
published=workflow.published_version == snapshot.version, level=snapshot.level,
|
||||
)
|
||||
|
||||
|
||||
@@ -297,8 +301,8 @@ async def publish_workflow(
|
||||
错误会挡住发布——这就是"必经检查点"落地的地方。
|
||||
返回 200 + ok/issues 而不是抛错,让前端能把问题列表渲染出来。
|
||||
"""
|
||||
if payload.level not in ("published", "governed"):
|
||||
raise HTTPException(400, "发布档位只能选「已发布」或「受管」")
|
||||
if payload.level not in _LEVELS:
|
||||
raise HTTPException(400, _BAD_LEVEL)
|
||||
workflow = await session.get(Workflow, workflow_id)
|
||||
if not workflow:
|
||||
raise HTTPException(404, _GONE)
|
||||
@@ -320,13 +324,14 @@ async def publish_workflow(
|
||||
except Exception as e: # noqa: BLE001
|
||||
return {"ok": False, "issues": [{"level": "error", "message": f"工作流的结构读不懂:{graph_error(e)}"}]}
|
||||
|
||||
issues = [i.model_dump() for i in validate_graph(spec).issues]
|
||||
issues += [i.model_dump() for i in lint_for_publish(spec, level=payload.level).issues]
|
||||
issues = [i.model_dump() for i in publish_issues(spec, level=payload.level)]
|
||||
if any(i["level"] == "error" for i in issues):
|
||||
return {"ok": False, "level": payload.level, "version": version, "issues": issues}
|
||||
|
||||
if not snapshot.graph_hash:
|
||||
snapshot.graph_hash = graph_hash(snapshot.graph)
|
||||
# 级别记在这一版上:之后改画布(status 退回 draft),从这一版发起的正式运行照样按发布时的级别出具
|
||||
snapshot.level = payload.level
|
||||
workflow.status = payload.level
|
||||
workflow.published_version = version
|
||||
workflow.published_by = actor_of(x_actor)
|
||||
@@ -341,6 +346,130 @@ async def publish_workflow(
|
||||
}
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 发布前检查与自动修复:两个接口都只读,不改库、不发布
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
class PublishCheckIn(BaseModel):
|
||||
level: str = "published"
|
||||
#: 画布上还没保存的图。不带就用库里当前的草稿
|
||||
graph: dict[str, Any] | None = None
|
||||
|
||||
|
||||
class AutofixIn(PublishCheckIn):
|
||||
#: 要应用的修复 id(publish-check 给的 fixes[].id)
|
||||
apply: list[str] = Field(default_factory=list)
|
||||
#: choice 类修复的选择:{fix_id: 值},可以多选的给列表
|
||||
choices: dict[str, Any] = Field(default_factory=dict)
|
||||
#: 剩下的 error 交给 Copilot 试着修(结果同样只是预览)
|
||||
assist: bool = False
|
||||
provider: str | None = None
|
||||
model: str | None = None
|
||||
|
||||
|
||||
async def _draft(session: AsyncSession, workflow_id: str, payload: PublishCheckIn) -> dict[str, Any]:
|
||||
if payload.level not in _LEVELS:
|
||||
raise HTTPException(400, _BAD_LEVEL)
|
||||
workflow = await session.get(Workflow, workflow_id)
|
||||
if not workflow:
|
||||
raise HTTPException(404, _GONE)
|
||||
return payload.graph if payload.graph is not None else (workflow.graph or {})
|
||||
|
||||
|
||||
async def _gate_context(session: AsyncSession, spec: GraphSpec) -> tuple[dict[str, Any], dict[str, Any]]:
|
||||
"""修复要用、但门禁本身不查库的东西:子工作流当前的发布版本,钉在别处的口径卡有哪些指标。"""
|
||||
versions: dict[str, Any] = {}
|
||||
cards: dict[str, Any] = {}
|
||||
for node in spec.nodes:
|
||||
cfg = node.config
|
||||
if node.type == NodeType.SUBGRAPH and cfg.get("workflow_id") and str(cfg["workflow_id"]) not in versions:
|
||||
upstream = await session.get(Workflow, str(cfg["workflow_id"]))
|
||||
versions[str(cfg["workflow_id"])] = upstream.published_version if upstream else None
|
||||
elif node.type == NodeType.METRICS and isinstance(cfg.get("caliber_from"), dict):
|
||||
ref = cfg["caliber_from"]
|
||||
version = ref.get("workflow_version")
|
||||
if not ref.get("workflow_id") or not str(version or "").isdigit():
|
||||
continue
|
||||
snapshot = (await session.execute(select(WorkflowVersion).where(
|
||||
WorkflowVersion.workflow_id == str(ref["workflow_id"]),
|
||||
WorkflowVersion.version == int(version)))).scalar_one_or_none()
|
||||
card = next((n for n in ((snapshot.graph if snapshot else None) or {}).get("nodes") or []
|
||||
if isinstance(n, dict) and n.get("id") == ref.get("node_id")), None)
|
||||
if card is not None:
|
||||
cards[node.id] = ((card.get("data") or {}).get("config") or {}).get("metrics") or []
|
||||
return versions, cards
|
||||
|
||||
|
||||
def _unreadable(level: str, e: Exception) -> dict[str, Any]:
|
||||
return {"level": level, "ok": False, "fixes": [], "issues": [
|
||||
{"level": "error", "node_id": None, "edge_id": None, "field": None, "code": None, "fix": None,
|
||||
"message": f"工作流的结构读不懂:{graph_error(e)}"}]}
|
||||
|
||||
|
||||
@router.post("/{workflow_id}/publish-check")
|
||||
async def publish_check(
|
||||
workflow_id: str, payload: PublishCheckIn, session: AsyncSession = Depends(get_session)
|
||||
) -> dict[str, Any]:
|
||||
"""发布前检查:和真正发布同一套口径(validate + 门禁)列出问题,以及能怎么修。
|
||||
|
||||
只读:不改库、不发布。发布弹窗一打开就调它,不用等点了发布才知道被拦。
|
||||
"""
|
||||
from app.engine.autofix import check
|
||||
|
||||
graph = await _draft(session, workflow_id, payload)
|
||||
try:
|
||||
spec = GraphSpec.model_validate(graph)
|
||||
except Exception as e: # noqa: BLE001
|
||||
return _unreadable(payload.level, e)
|
||||
versions, cards = await _gate_context(session, spec)
|
||||
return check(spec, level=payload.level, versions=versions, cards=cards)
|
||||
|
||||
|
||||
@router.post("/{workflow_id}/autofix")
|
||||
async def autofix(
|
||||
workflow_id: str, payload: AutofixIn, session: AsyncSession = Depends(get_session)
|
||||
) -> dict[str, Any]:
|
||||
"""应用选中的修复,返回修复后的整张图——只是预览。
|
||||
|
||||
不自动保存、不自动发布:人看过逐项变更再确认,前端走现有的保存接口存草稿,再检查、再发布。
|
||||
每条修复都复核过(降低要求的、错误没变少或者冒出新错误的丢弃并写明原因)。assist 为真时,
|
||||
剩下的 error 交给 Copilot,同一道复核通过才采纳。
|
||||
"""
|
||||
from app.engine.autofix import apply_fixes, diff_changes
|
||||
|
||||
graph = await _draft(session, workflow_id, payload)
|
||||
try:
|
||||
spec = GraphSpec.model_validate(graph)
|
||||
except Exception as e: # noqa: BLE001
|
||||
bad = _unreadable(payload.level, e)
|
||||
return {"graph": graph, "changes": [], "ops": [], "applied": [], "rejected": [
|
||||
{"fix_id": fid, "reason": "工作流的结构读不懂,没法应用修复"} for fid in payload.apply],
|
||||
"remaining": bad["issues"], "fixes": [], "assist": None, "ok": False}
|
||||
versions, cards = await _gate_context(session, spec)
|
||||
out = apply_fixes(graph, payload.apply, payload.choices, level=payload.level, versions=versions, cards=cards)
|
||||
out["assist"] = None
|
||||
if payload.assist:
|
||||
if out["ok"]:
|
||||
out["assist"] = {"ok": True, "summary": "修复之后已经没有挡住发布的错误,没有再交给 Copilot",
|
||||
"questions": []}
|
||||
return out
|
||||
from app.api.copilot import assist_publish_fix
|
||||
|
||||
helped = await assist_publish_fix(session, out["graph"], level=payload.level,
|
||||
provider=payload.provider, model=payload.model)
|
||||
out["assist"] = {"ok": helped["accepted"], "summary": helped["summary"], "questions": helped["questions"]}
|
||||
if helped["accepted"]:
|
||||
fixed = helped["graph"]
|
||||
out["changes"] += diff_changes(out["graph"], fixed, fix_id="assist", label="Copilot 的修改")
|
||||
out["ops"] += helped["ops"]
|
||||
out["applied"].append("assist")
|
||||
after = apply_fixes(fixed, [], level=payload.level, versions=versions, cards=cards)
|
||||
out.update(graph=fixed, remaining=after["remaining"], fixes=after["fixes"], ok=after["ok"])
|
||||
elif helped["reason"]:
|
||||
out["rejected"].append({"fix_id": "assist", "reason": helped["reason"]})
|
||||
return out
|
||||
|
||||
|
||||
class ValidateIn(BaseModel):
|
||||
graph: dict[str, Any]
|
||||
|
||||
|
||||
Reference in New Issue
Block a user