diff --git a/README.md b/README.md index dea36913..759764fe 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ English | [Simplified Chinese](README.zh-CN.md) ## Why use it -- **Numbers you can check**: all arithmetic happens in SQL and in caliber cards (controlled metric sets), and the narrative layer has no right to create numbers. The report-writer node writes only citation markers, the system fills in the numbers from snapshots, and every number opens all the way to the cell in the query result it came from. +- **Numbers you can check**: all arithmetic happens in SQL and in caliber cards (controlled metric sets), and the narrative layer has no right to create numbers. The report-writer node writes only citation markers, the system fills in the numbers from snapshots, and every number opens all the way to the cell in the query result it came from. Judgments such as "mainly from East" or "outside the usual fluctuation" are likewise computed by the system with fixed methods and written as verdict phrases, and the report only cites them. - **Constrained orchestration**: the engine is compiled, so at run time the model can't change the graph structure; its freedom is confined to the inside of agent nodes and to branch decisions. Fully dynamic multi-agent collaboration stays in the exploration layer. - **Pinned, reproducible versions**: formal runs accept only published immutable versions (pinned by graph hash). The complete evidence of every execution goes into a content-addressed artifact store and is sealed, and one click verifies that nothing was altered. - **Human control**: approvals, joint sign-off, time limits, and Reject exits; nothing is ever approved automatically, and a process restart loses no state. @@ -20,7 +20,7 @@ English | [Simplified Chinese](README.zh-CN.md) **Orchestration and runs** -- Canvas: 18 node types, drag and connect; live node highlighting, streaming output, and a timeline track at run time → [Workflows](docs/en/workflows.md) +- Canvas: 19 node types, drag and connect; live node highlighting, streaming output, and a timeline track at run time → [Workflows](docs/en/workflows.md) - Copilot: generate or rewrite a whole graph from one sentence; it looks up schemas on demand and checks its work before delivering → [Copilot](docs/en/ask-data.md#copilot) - Human-in-the-loop: approvals, joint sign-off, time limits, Reject exits → [Human nodes](docs/en/workflows.md#human-nodes) - Agent guardrails: repeated calls, spending, and context each have a guard; the step count is only a backstop → [Guardrails](docs/en/workflows.md#guardrails) @@ -41,6 +41,7 @@ English | [Simplified Chinese](README.zh-CN.md) **Evidence and issuance** - Clickable evidence: numbers, table and field names, and verbatim knowledge-base quotes all open to their provenance → [Evidence and Issuance](docs/en/evidence.md) +- Analytical judgments: change decomposition, fluctuation baselines, trends, difference tests, and comparability checks are computed by the system, which also writes the verdict phrases; reports cite them, and judgment words are checked sentence by sentence → [Analysis](docs/en/analysis.md) - Three-tier issuance: caliber cards plus issuance contracts, checked by value or by citation → [Issuance contracts](docs/en/evidence.md#issuance-contracts-and-tiers) - Sealing and exports: one-click seal verification; self-contained report exports and verifiable reports → [Exports](docs/en/evidence.md#exports) - Review suite: double-blind recomputation, adversarial review, approval inquiry, the what-if sandbox, run comparison → [Review suite](docs/en/evidence.md#the-review-suite) @@ -71,7 +72,7 @@ conda env create -f environment.yml # first time: create the agentlab environm ./scripts/dev.sh # after that: one command starts the frontend and backend ``` -Open http://localhost:5273 . You can try everything without an API key: the built-in Mock provider runs the whole orchestration path end to end; to connect a real model, see [Model access](docs/en/configuration.md#model-access). The 9 built-in templates run as they are. +Open http://localhost:5273 . You can try everything without an API key: the built-in Mock provider runs the whole orchestration path end to end; to connect a real model, see [Model access](docs/en/configuration.md#model-access). The 11 built-in templates run as they are. **Login is off by default**: anyone on the same network who can open the page can use every feature. Turn on login before multiple people use it; see [Login modes](docs/en/identity-and-security.md#login-modes). For more startup options, see [Development](docs/en/development.md#start). @@ -90,6 +91,7 @@ See [Deployment](docs/en/deployment.md) and [deploy/README.md](deploy/README.md) | [Ask Data and Copilot](docs/en/ask-data.md) | What happens after you ask, output review, record retention, how Copilot builds graphs and looks things up | | [Workflows](docs/en/workflows.md) | The canvas, node types, passing data, human nodes and rejection, agent guardrails, subworkflows, publishing and gates, runs and resuming | | [Evidence and Issuance](docs/en/evidence.md) | Caliber cards, issuance contracts and tiers, citation markers, claim sentences, no data in the queried range, sealing and verification, report exports and verifiable reports, the review suite | +| [Analysis](docs/en/analysis.md) | The five analyses and their criteria, when an analysis can't be done, analysis tools for agents and analysis cards, how to cite, the analysis claim check, the analysis scope, settings and limits | | [Data Sources and Spreadsheet Import](docs/en/data-sources.md) | Connecting databases, read-only guarantees, uploading spreadsheets and recipes, accumulating by period, cross-database merges, masked columns | | [Data Catalog](docs/en/data-catalog.md) | What the catalog records, drafting and review, export and import, data profiling, SQL checks, introspection scope for large databases, schema and value changes | | [Tools, MCP, and the Capability Registry](docs/en/tools.md) | Kinds of tools, approval levels, capability governance, limits on write capabilities, registered endpoints, network rules | diff --git a/README.zh-CN.md b/README.zh-CN.md index 2ec3a833..1d53df40 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -10,7 +10,7 @@ ## 为什么用它 -- **数字可核对**：所有算术都在 SQL 和口径卡（受控指标集）里完成，叙述层无权创造数字。报告撰写节点只写引用标记，系统照快照填数，每个数一路点得开到查询结果里的那一格。 +- **数字可核对**：所有算术都在 SQL 和口径卡（受控指标集）里完成，叙述层无权创造数字。报告撰写节点只写引用标记，系统照快照填数，每个数一路点得开到查询结果里的那一格。「主要来自华东」「超出常规波动」这类判断也由系统按固定的方法算、写成判定语，报告只引用。 - **受限编排**：引擎是编译式的，运行时模型改不了图结构，它的自由限在 agent 节点内部和分支决策里；全动态的多 agent 协作只留在探索层。 - **版本固定、可复现**：正式运行只认已发布的不可变版本（按图哈希固定）。每次执行的完整证据落进内容寻址的工件库并封存，一键核对有没有被改过。 - **人工把关**：审批、会签、时限、驳回出口；永远不会自动批准，进程重启不丢状态。 @@ -20,7 +20,7 @@ **编排与运行** -- 画布：18 种节点拖拽连线，运行时逐节点高亮、流式输出、航迹时间轴 → [工作流](docs/zh-CN/workflows.md) +- 画布：19 种节点拖拽连线，运行时逐节点高亮、流式输出、航迹时间轴 → [工作流](docs/zh-CN/workflows.md) - 助手：一句话生成或改写整张图，按需查看表结构，交付前自查 → [助手](docs/zh-CN/ask-data.md#助手) - 人工介入：审批、会签、时限、驳回出口 → [人工节点](docs/zh-CN/workflows.md#人工节点) - Agent 护栏：重复调用、花费、上下文都有挡法，步数只留兜底 → [护栏](docs/zh-CN/workflows.md#护栏) @@ -41,6 +41,7 @@ **证据与出具** - 可点击证据：数字、表名字段名、知识库原话都点得开出处 → [证据与出具](docs/zh-CN/evidence.md) +- 分析判断：变化拆解、波动基线、趋势、差异检验、可比性检查由系统算，判定语由系统写，报告引用、判断词逐句核对 → [分析能力](docs/zh-CN/analysis.md) - 三档出具：口径卡加出具契约，按数值回指或按引用核对 → [出具契约](docs/zh-CN/evidence.md#出具契约与三档) - 封存与导出：一键核对封存；自包含的报告导出和可验证报告 → [导出](docs/zh-CN/evidence.md#导出) - 复核套件：双盲复算、反方审稿、审批质询、数字推演、运行对比 → [复核套件](docs/zh-CN/evidence.md#复核套件) @@ -71,7 +72,7 @@ conda env create -f environment.yml # 首次：创建 agentlab 环境并安装 ./scripts/dev.sh # 之后：一条命令启动前后端 ``` -打开 http://localhost:5273 。不配 API Key 也能完整体验：内置的 Mock 会把编排链路完整走一遍；接入真实模型见[模型接入](docs/zh-CN/configuration.md#模型接入)。内置 9 个模板可以直接运行。 +打开 http://localhost:5273 。不配 API Key 也能完整体验：内置的 Mock 会把编排链路完整走一遍；接入真实模型见[模型接入](docs/zh-CN/configuration.md#模型接入)。内置 11 个模板可以直接运行。 **默认不开登录**：同一网络里打得开页面的人都能用全部功能。多人使用前先开登录，见[登录方式](docs/zh-CN/identity-and-security.md#登录方式)。更多启动选项见[开发](docs/zh-CN/development.md#启动)。 @@ -90,6 +91,7 @@ conda env create -f environment.yml # 首次：创建 agentlab 环境并安装 | [问数据与助手](docs/zh-CN/ask-data.md) | 提问之后发生什么、产出复核、记录保留，助手怎么生成图、怎么按需查看 | | [工作流](docs/zh-CN/workflows.md) | 画布、节点类型、传数据、人工节点与驳回、agent 护栏、子工作流、发布与门禁、运行与断点续跑 | | [证据与出具](docs/zh-CN/evidence.md) | 口径卡、出具契约与三档、引用标记、结论句、查询区间里没有数据、封存与核对、报告导出与可验证报告、复核套件 | +| [分析能力](docs/zh-CN/analysis.md) | 五种分析与判定标准、算不了的时候、Agent 的分析工具与分析卡、引用写法、分析说法核对、分析范围、设置与限制 | | [数据源与表格导入](docs/zh-CN/data-sources.md) | 接入数据库、只读保证、上传表格与取数配方、按期累积、跨库合并、遮罩列 | | [数据目录](docs/zh-CN/data-catalog.md) | 目录记什么、起草与审阅、导出导入、数据剖析、SQL 检查、大库探查范围、结构与取值变化 | | [工具、MCP 与能力登记](docs/zh-CN/tools.md) | 工具种类、审批档位、能力治理、写入能力的限制、登记端点、网络规则 | diff --git a/backend/app/analysis/intent.py b/backend/app/analysis/intent.py new file mode 100644 index 00000000..e5055bb6 --- /dev/null +++ b/backend/app/analysis/intent.py @@ -0,0 +1,200 @@ +"""问题意图：用户问的一句话是不是分析类问题（docs/design/analysis-round/00-总方案.md §3.7）。 + +助手搭问数据的图时自查（api/copilot_gen.GraphChecker 的 analysis_gaps）用它：问题问的是为什么、怎么变了、哪里拖累、谁贡献 +最大、是否异常、正常吗、波动、趋势、持续、是不是真的更高这类，而查库的 Agent 没绑分析工具、图里也没有分析卡时，提一条只挡 +一轮的问题，请助手给 Agent 绑上分析工具——变化从哪来、算不算异常、是不是趋势由平台按固定的方法和阈值算，判定语由系统写， +报告引用它。 + +**确定的词表，带版本**（INTENT_VERSION）：不调模型。只收明确的问法，宁可漏判：漏判只是少一条自查提醒，误判要让助手多改一轮、 +给用不上的 Agent 绑分析工具。按句子认（句号、问号、感叹号、分号、换行分开；逗号不分：「销售额下降了，为什么」是一句）， +一句里命中一种就算，先认先得（原因 → 来源 → 异常 → 波动 → 趋势 → 差异）。各类收什么、不收什么： + +- 原因（why）：「为什么 / 为何 / 为啥 / 怎么会」要和涨跌、变化、这么高、比…低这类说法在同一句（「为什么报错」「为什么查不到 + 数据」不算）；「原因」要是变化的原因（「下降的原因」「什么原因导致订单减少」「销售额下滑了，原因是什么」），「退款原因有哪些」 + 「按取消原因统计」问的是一个字段，不算 +- 来源（source）：「怎么变了 / 有什么变化」「哪里拖累 / 谁拉动 / 什么带动了增长」「贡献最大」「下降主要来自哪」「驱动因素」 + 「把变化拆开」「归因分析」。「订单来自哪些渠道」问的是构成，不算（要先说变化）；「表结构有什么变化」「口径有什么变化」问的 + 不是数，不算 +- 异常（anomaly）：「是否异常 / 异常吗 / 正常吗 / 算不算异常 / 有没有异动 / 是不是暴跌了」这类问句，和「超出正常范围」。 + 「异常订单有多少」「有没有异常记录」「门店是否正常营业」问的是业务状态，「系统正常吗」「连接正常吗」问的是服务，都不算 +- 波动（fluctuation）：「波动大吗 / 变化大不大 / 是不是正常波动 / 只是波动」。「列出价格波动超过 10% 的商品」是筛选，「网络波动大吗」问的 + 是服务，都不算 +- 趋势（trend）：「趋势 / 走势」「持续上升 / 下降会持续吗 / 一直在降 / 连续三周下降 / 越来越少 / 是不是在涨 / 稳步增长」。 + 「活动会持续几天」「促销还在持续吗」不算（「持续」前后要有涨跌） +- 差异（differ）：「是不是真的更高 / 真的比华南高吗」「差异显著吗 / 有没有明显差别」「是不是偶然」「统计上显著」 + +英文只收几种明确的问法（why … drop、what drove、is this normal、trend、significantly higher……），同样宁可漏判。 + +意图判的是用户的问题，不是写作者的句子：说法核对的说法表（engine/analysis_claims）另是一套，两边不共用词表——写作者写 +「主要来自华东」是下判断，用户问「主要来自哪」是在问。 + +整个模块是纯函数，不碰数据库、不调模型。 +""" +from __future__ import annotations + +import re +from dataclasses import dataclass + +#: 词表的版本：加减词、改规则就升版本（评测比两次的提示词时据此认得出自查规则变了） +INTENT_VERSION = 1 + +#: 意图的种类 → 给人看的叫法（自查的那条问题里说「问题问的是……」） +KINDS: dict[str, str] = { + "why": "原因", "source": "变化从哪来", "anomaly": "是否异常", "fluctuation": "波动大不大", "trend": "趋势", + "differ": "差异站不站得住", +} + + +@dataclass(frozen=True) +class Intent: + """一句问题的分析意图：kind 是 KINDS 之一，text 是问题里认出的那几个字（给人看、给助手看）。""" + + kind: str + text: str + version: int = INTENT_VERSION + + @property + def label(self) -> str: + return KINDS[self.kind] + + +# -------------------------------------------------------------------------- +# 词表 v1（正则都写在 re.compile 里：拼词表的常量只放用「|」连起来的词） +# -------------------------------------------------------------------------- + +#: 涨跌和变化（用户问题里的说法）。单字的「升」「降」不收：「升级」「降级」不是变化 +_CHANGE = ("上升|下降|增长|下滑|减少|增加|提升|提高|降低|上涨|下跌|回落|回升|走低|走高|反弹|萎缩|下行|上行|缩水|" + "变化|变动|波动|变了|变多|变少|多了|少了|高了|低了|大了|小了|好转|恶化|变好|变差|暴跌|激增|骤降|飙升|大涨|" + "大跌|突增|突降|翻倍|腰斩|涨|跌") +_WHY = "为什么|为何|为啥|怎么会|咋会|怎么就|怎么突然" +#: 「异常」后面是这些名词的，说的是业务状态（「异常订单」「有没有异常记录」） +_STATUS_NOUN = "订单|记录|单据|单|数据|状态|交易|用户|账户|账号|日志|值|码|件|商品|门店|设备|情况|信息|事件|标记|原因" +#: 「正常」后面是这些的，说的是在不在营业、能不能用（「门店是否正常营业」） +_STATUS_VERB = "营业|运行|运转|工作|使用|登录|发货|开放|上线|启动|显示|访问|打开|连接" + +#: 句子的边界：逗号不分（「销售额下降了，为什么」是一句） +_SENTENCE = re.compile(r"[。；;！!？?\n]|\.(?=\s|$)") +#: 一句里有这些就不是在问数：服务、配置、结构、口径本身（「系统正常吗」「表结构有什么变化」「口径有什么变化」）。只管标了 +#: data_only 的那几条规则 +_NOT_DATA = re.compile(r"系统|服务器|连接|接口|网络|进程|部署|配置|设置|权限|代码|界面|流程|工作流|表结构|字段|列名|" + r"口径|定义|报错|出错") + +#: 原因一类：问原因的词和变化、高低的说法在同一句（_why） +_WHY_RE = re.compile(_WHY) +_WHY_WHAT = re.compile(rf"{_CHANGE}|(?:这么|那么|如此|这样|偏|特别|格外|明显)(?:高|低|多|少|大|小|差|好)" + r"|比.{1,16}?(?:高|低|多|少|大|小|好|差)|差(?:这么|那么)多|差距(?:这么|那么)?大") +#: 「原因是什么」：同一句前面要说过变化（「销售额下滑了，原因是什么」） +_REASON_ASK = re.compile(r"原因(?:是什么|是啥|有哪些|在哪|何在|为何)") +_CHANGE_RE = re.compile(_CHANGE) + +#: (种类, 规则, data_only：这一句有 _NOT_DATA 时不认) +_RULES: list[tuple[str, re.Pattern[str], bool]] = [ + ("why", re.compile(rf"(?:{_CHANGE}).{{0,6}}?的?(?:主要)?原因"), False), + ("why", re.compile(rf"(?:什么|啥|哪些)原因.{{0,12}}?(?:{_CHANGE})"), False), + ("source", re.compile(r"归因分析|做(?:个|一下|下)?归因|归因一下|(?:怎么|如何)归因|驱动(?:因素|力)" + r"|(?:由|靠)(?:什么|哪些|哪个).{0,4}?驱动"), False), + ("source", re.compile(r"怎么变(?:了|的|化)|如何变化|有(?:什么|啥|哪些)变化|变化(?:在哪|如何|怎么样|情况)"), True), + ("source", re.compile(r"(?:哪|谁|什么).{0,10}?(?:拖累|拉低|拉高|拉动|带动|推动|贡献)" + r"|(?:拖累|拉动|带动|推动|贡献)了?.{0,6}?(?:最大|最多|最明显)"), False), + ("source", re.compile(rf"对.{{0,12}}?(?:{_CHANGE}|增量|增幅|降幅).{{0,2}}?贡献"), False), + ("source", re.compile(rf"(?:{_CHANGE}).{{0,12}}?(?:主要)?(?:来自|来源于|出在|集中在|源于|发生在)(?:哪|什么|何)"), + False), + ("source", re.compile(rf"(?:哪|什么).{{0,8}}?(?:导致|造成|引起|带来)了?.{{0,12}}?(?:{_CHANGE})"), False), + ("source", re.compile(rf"(?:拆解|拆分|分解|拆开).{{0,12}}?(?:{_CHANGE})|(?:{_CHANGE}).{{0,12}}?(?:拆解|拆分|分解|拆开)"), + False), + ("anomaly", re.compile(rf"(?:是否|是不是|有没有|算不算|算是|属于)(?:出现)?了?(?:异常|反常|不正常)(?!{_STATUS_NOUN})"), + True), + ("anomaly", re.compile(rf"(?:异常|反常)(?!{_STATUS_NOUN})(?:吗|么|与否)|异常不异常" + rf"|(?:正常|不正常)(?!{_STATUS_VERB})(?:吗|么|与否)|正不正常" + rf"|(?:是否|是不是|算不算|属于)正常(?!{_STATUS_VERB}|的?波动)"), True), + ("anomaly", re.compile(r"(?:超出|超过|偏离)了?(?:正常|常规|历史)的?(?:范围|水平|波动|区间)" + r"|(?:有没有|是否有|是不是)异动|异动分析|异动吗"), True), + ("anomaly", re.compile(r"(?:是否|是不是|算不算|有没有).{0,6}?(?:大幅|骤降|暴跌|激增|飙升|突增|突降|大涨|大跌)"), True), + ("fluctuation", re.compile(r"(?:波动|变化)(?:大吗|大么|大不大|算大|有多大)|波动(?:正常吗|正不正常)" + r"|(?:正常|常规|一般|季节性|随机|自然)的?波动" + r"|(?:只是|仅仅是|是不是|是否|算不算|属于|算是)(?:正常|常规)?的?波动"), True), + ("trend", re.compile(r"趋势|走势"), False), + ("trend", re.compile(r"持续(?:地)?(?:上升|下降|增长|下滑|走高|走低|上涨|下跌|减少|增加|回落|萎缩|改善|恶化|变好|变差|" + r"走强|走弱|攀升)"), False), + ("trend", re.compile(rf"(?:{_CHANGE}).{{0,12}}?(?:是否|是不是|会不会|还会|还在|一直在?|会)持续"), False), + ("trend", re.compile(r"一直(?:在)?(?:涨|跌|上升|下降|增长|下滑|减少|增加|走高|走低|攀升)" + r"|连续.{0,6}?(?:周|月|天|日|期|季度|年).{0,4}?(?:上升|下降|增长|下滑|涨|跌|减少|增加|走高|走低)" + r"|越来越(?:高|低|多|少|好|差|大|小)" + r"|(?:是不是|是否|有没有)(?:一直)?在(?:涨|跌|上升|下降|增长|下滑|减少|增加)" + r"|(?:逐步|稳步|节节)(?:上升|下降|增长|下滑|攀升|走高|走低|回落)"), False), + ("differ", re.compile(r"(?:是不是|是否)真的(?:更|比较)?(?:高|低|多|少|好|差|大|小)" + r"|真的(?:更)?(?:高|低|多|少|好|差)(?:吗|么)|真的比.{1,16}?(?:高|低|多|少|好|差|大|小)" + r"|(?:差异|差别|区别|差距)(?:是否|是不是)?(?:显著|明显|站得住|有意义|真实存在)" + r"|有没有(?:显著|明显|统计学?上?)的?(?:差异|差别|区别)" + r"|(?:差异|差别|区别)(?:大吗|大不大|明显吗|显著吗)" + r"|显著(?:高于|低于|不同|差异|更高|更低)|统计学?上?(?:显著|有意义)" + r"|(?:是不是|是否|会不会)(?:只是)?(?:偶然|巧合|随机|样本(?:波动|太少|太小))|置信区间"), False), +] + +#: 英文：只收几种明确的问法 +_EN_CHANGE = (r"drop|dropped|fall|fell|decline|declined|decrease|decreased|rise|rose|increase|increased|grow|grew|" + r"change|changed|higher|lower|spike|spiked|dip|dipped|jump|jumped|surge|surged|plunge|plunged|so (?:high|low)") +_EN_RULES: list[tuple[str, re.Pattern[str]]] = [ + ("why", re.compile(rf"\bwhy\b.{{0,60}}?\b(?:{_EN_CHANGE})\b|\b(?:{_EN_CHANGE})\b.{{0,40}}?\bwhy\b", re.I)), + ("source", re.compile(r"\bwhat (?:drove|drives|is driving|caused|explains)\b|\bdrivers? of\b" + r"|\bcontribut(?:ed|es|ion)\b.{0,20}\bmost\b|\bwhere did\b.{0,40}\bcome from\b", re.I)), + ("anomaly", re.compile(r"\b(?:is|was|are|were) (?:this|it|that|these) (?:normal|abnormal|unusual|an anomaly|anomalous)\b" + r"|\banomal(?:y|ies|ous)\b|\bout of the ordinary\b", re.I)), + ("trend", re.compile(r"\btrend(?:s|ing)?\b|\bkeeps? (?:rising|falling|growing|declining|increasing|decreasing)\b" + r"|\bconsecutive (?:weeks|months|days)\b", re.I)), + ("differ", re.compile(r"\bsignificant(?:ly)? (?:higher|lower|different|difference)\b|\bstatistically\b" + r"|\breally (?:higher|lower|better|worse)\b", re.I)), +] + +#: 认出的那几个字最多留多长（给人看的问题里引一段） +TEXT_MAX = 24 + + +def _quote(text: str) -> str: + text = text.strip() + return text if len(text) <= TEXT_MAX else text[:TEXT_MAX - 1] + "…" + + +def _why(sentence: str) -> str | None: + """问原因：问原因的词和变化、高低的说法在同一句。交回从先出现的那个到后出现的那个的一段。""" + ask = _WHY_RE.search(sentence) + what = _WHY_WHAT.search(sentence) + if ask is None or what is None: + return None + return sentence[min(ask.start(), what.start()):max(ask.end(), what.end())] + + +def _reason(sentence: str) -> str | None: + """「…下滑了，原因是什么」：问原因的那几个字前面说过变化。""" + ask = _REASON_ASK.search(sentence) + if ask is None: + return None + what = _CHANGE_RE.search(sentence, 0, ask.start()) + return sentence[what.start():ask.end()] if what is not None else None + + +def analysis_intent(question: str | None) -> Intent | None: + """一句问题的分析意图；不是分析类问题（或认不准）为 None。""" + text = str(question or "").strip() + for sentence in (s.strip() for s in _SENTENCE.split(text)): + if not sentence: + continue + if found := _why(sentence) or _reason(sentence): + return Intent("why", _quote(found)) + not_data = _NOT_DATA.search(sentence) is not None + for kind, rule, data_only in _RULES: + if data_only and not_data: + continue + if (m := rule.search(sentence)) is not None: + return Intent(kind, _quote(m.group(0))) + for kind, rule in _EN_RULES: + if (m := rule.search(sentence)) is not None: + return Intent(kind, _quote(m.group(0))) + return None + + +def is_analytical(question: str | None) -> bool: + return analysis_intent(question) is not None + + +__all__ = ["INTENT_VERSION", "KINDS", "Intent", "analysis_intent", "is_analytical"] diff --git a/backend/app/api/copilot.py b/backend/app/api/copilot.py index fb8b7ade..34191d7a 100644 --- a/backend/app/api/copilot.py +++ b/backend/app/api/copilot.py @@ -76,7 +76,7 @@ def _scope_note(rows: list[Any], scoped: bool) -> str: "db_schema__ 工具；前几轮用过别的库，这一轮也不要沿用") -def _tool_catalog(rows: list[Any], *, builtins: frozenset[str] | None = None) -> str: +def _tool_catalog(rows: list[Any], *, builtins: frozenset[str] | None = None, analysis_tools: bool = True) -> str: """给助手看的工具清单。 all_specs() 只有静态注册的内置工具，数据源工具是运行时按库动态生成的 @@ -86,11 +86,14 @@ def _tool_catalog(rows: list[Any], *, builtins: frozenset[str] | None = None) -> 不列表名（look 文档 D3）：以前每个库列前 12 个表名，和这一轮要做什么无关，闲聊也带着；表数写一句，字段要用时 look。 builtins 给了的话内置工具只列这几个：使用者的问数据（_CHAT_USER_PROMPT）只能用只读的那几个，列出来用不了的工具， - 模型照样会绑。 + 模型照样会绑。analysis_tools 为假（评测「关掉分析工具」那一臂，answer_ops）时不列五个分析工具。 """ + from app.tools.analysis import ANALYSIS_TOOLS + lines = [ f"- {name}（{spec.category}）：{spec.description}" - for name, spec in sorted(all_specs().items()) if builtins is None or name in builtins + for name, spec in sorted(all_specs().items()) + if (builtins is None or name in builtins) and (analysis_tools or name not in ANALYSIS_TOOLS) ] from app.data import introspect as _introspect @@ -137,6 +140,12 @@ NODE_REFERENCE = """\ 加标识符引号：MySQL / MariaDB 用反引号（AS `月份`），PostgreSQL、Oracle、SQLite 用双引号（AS "月份"），不要用 单引号（那是字符串）。只给最外层 SELECT 的结果列起别名，WHERE、JOIN、GROUP BY 里照旧写原列名；Oracle 旧版本 的标识符最长 30 字节，别名不超过 10 个字 + **问题问的是为什么、怎么变了、哪里拖累、谁贡献最大、是否异常、波动、趋势、是不是真的更高这类分析问题时**，查库的 agent + 除了 db_query__ / db_schema__ 再绑上五个分析工具（analysis_comparable、analysis_baseline、analysis_trend、 + analysis_decompose、analysis_differ）：变化从哪来、算不算异常、是不是持续在变、差异站不站得住由平台按固定的方法和阈值算， + 判定语由系统写好，报告引用它；agent 只决定算什么（比哪两期、拆哪个维度、往哪一段下钻）。绑了任一个，平台就把分析方法 + 注入 agent 的 system，prompt 里不用再教方法，也不要写阈值。分析工具只分析这个 agent 自己查过的结果，所以只绑给同时绑了 + db_query__ 的 agent；不能放进 tool 节点（模板里每次都要做的固定分析用 analysis 分析卡） **默认不要写 max_steps、budget_tokens、budget_usd**。平台用的是护栏而不是固定步数： 步数只是很高的兜底（跟随设置，默认 100），重复调用、连续几步没有新信息、预算用完、 上下文快满时，它会按查到的部分收尾。写死一个小数字（见过 8）会让它查完表结构就没额度回答了。 @@ -215,6 +224,21 @@ NODE_REFERENCE = """\ 写 2 只剩「2%」，0.004 这样的小比率还会显示不出来、报告引用不了。on_missing 默认 fail（缺输入整张卡失败）；写 null 则缺输入的指标记为空值，交给出具契约判档 取 tool 节点查库的结果用 cell(nodes.取数节点id, 行, '列名')（行号从 0 数，列按列名）：cell() 直接认查询工具 交回的结果，出处精确到快照里的那一格，不用再接 transform 解析。取 agent 的字段写 vars..字段 +- analysis：分析卡。对上游查询节点的结果固定做几项确定性分析，和口径卡平级：用于模板里每次都要做的分析（周报的异动： + 这周的变化在不在常规波动范围内、主要来自哪个地区）。问数据里要看了数才知道拆哪个维度的不用它，给查库的 agent 绑分析工具。 + config: {items:[{id, op, mode, from, params, hypothesis}], overrides, assign_to} + · op：decompose（变化拆解，mode 选 additive 可加 / ratio 比率 / factor 乘积）、baseline（波动基线）、trend（趋势）、 + differ（差异检验，mode 选 rate / mean / rows）、comparable（可比性检查）；hypothesis 一句话写这一项在验证什么 + · from 是要分析的上游查询节点 id：只能是调用 db_query__ 的 tool 节点或 merge 节点，而且连在分析卡之前；agent 不能当 + 输入（它可能查了好几次）。基期、本期是两份查询时不写 from，在 params 里写 base_query、target_query 两个节点 id + · params 和同名分析工具的参数一样（period、measure、by、base、target、headline……，不写 query），可以用 {{ input.x }} + 按运行输入选期次；headline 的两格写成「节点id.r行号.列名」（行号从 0 数，如 weekly.r0.销售额）；下钻的 parent 写 + {"of": 前面一项的 id, "key": [那一段的维度值]}。params 里不写阈值：阈值来自设置，模板要固定阈值时写在 overrides + （键同设置里分析一组的键，如 k_warn、main_share） + · 产出 nodes..results.<项 id> = {status, verdict, direction, values, phrase}：下游 branch 可以按判定分支，如 + nodes.check.results.wow.verdict in ['outside_band', 'far_outside_band'] 时转人工。报告引用它的结论（见 report 里分析 + 结论的写法）；不要把结果交给口径卡：口径卡读分析卡的结果追不到原始数据。一项算不了（历史不够、本期没过完）不让卡 + 失败，那一项照实记成算不了 - dual_check：双盲复算。正式出具前，口径卡上输入来自查询的指标另起一个看不到原 SQL 的只读 agent 独立写 SQL 再算一遍： 一致就放行；不一致时停下来，审批卡并排列出两边的 SQL、差异说明和两个值，由人逐个指标选采用原流程、采用盲算还是 两边都不对。选盲算或都不对的计入缺口、出具降档；两边只差一个方面（业务日期按哪一列、有效记录条件、去重）时，选择 @@ -228,10 +252,11 @@ NODE_REFERENCE = """\ 出具降档）/ fail（审核人没采用原流程的结果就判为失败）；enabled 不写跟随设置里的开关，true / false 强制开关。 不要给它配 retries：它的失败是确定的，重试只会把盲算再做一遍、再请审核人选一次（画布校验会报错） - report：报告撰写（带引用）。config: {instructions, system, metrics_from:[口径卡id], numbers, on_violation, max_repairs, - claims, entities, assign_to} + claims, analysis_claims, analysis_scope, entities, assign_to} **凡是给人看的、带数字的报告、分析、结论，都用 report，不要用 llm**。它自动收集上游的口径卡指标、查询结果 - （tool 节点和 agent 查过的库、merge 的合并结果，按查询先后编成 Q1、Q2…）、查库当时的表结构、知识库检索（按先后编成 K1、K2…） - 和运行输入，模型只能写引用标记，数值由系统从证据里取出来渲染；模型自己写的数字会被打回重写： + （tool 节点和 agent 查过的库、merge 的合并结果，按查询先后编成 Q1、Q2…）、查库当时的表结构、知识库检索（按先后编成 K1、K2…）、 + 分析结论（agent 调分析工具、分析卡做的，按先后编成 A1、A2…）和运行输入，模型只能写引用标记，数值由系统从证据里取出来渲染； + 模型自己写的数字会被打回重写： - [[m:指标id]]：口径卡指标。比率、增幅、占比、差值这类派生计算只能先在口径卡里登记再引用。渲染出来自带单位（口径卡里的 unit，「8条」），instructions 里不要写成「覆盖 [[m:line_count]] 条产线」这样后面再跟一个单位，写「覆盖 [[m:line_count]] 产线」 - [[v:Q1.r0.列名]]：第 1 次查询结果里第 0 行那一列的原值（行号从 0 数） @@ -244,6 +269,12 @@ NODE_REFERENCE = """\ 没有出处；只是拿检索当依据时写 [[see:K1]]。要引原话，在 report 前面接 retrieve 节点 - 句末的 [[see:m:指标id,Q1,K1]]：这句结论的依据，不显示，所以只能放在句末；instructions 里不要写「以 [[see:…]] 为依据」 「根据 [[see:…]]」这样把它当成句子成分的示例，模型照着写会留下「以 为依据」这样的断句 + - 分析结论：[[a:A1]] 渲染成 A1 的判定语（系统按判定写好的一句，连同里面的数）；[[a:A1.top_share|pct]] 是它的一个具名的数 + （换算后缀同 [[m:]]）；[[v:A1.r0.share]]、[[table:A1 cols=地区,delta,share rows=0-4]] 取它的结果表；句末的 [[see:A1]] + 把它当这句的依据。「主要来自」「集中在」「拖累」「异常」「大幅」「持续上升」「趋势」「显著高于」这类判断只能照分析结论的 + 判定写、挂它当依据（最好直接引用判定语），没有对应的分析结论就只陈述数字和变化；拆解只说明变化来自哪里、说明不了为什么， + 写原因要有知识库原话或运行输入作依据，否则写成待核实的推测。这些规矩系统会告诉写作者，instructions 里写「按分析结论写、 + 判断引用判定语」就够了，不要替它把判断写好 - 涉及码值（状态、类型这类用数字编码的取值）时，instructions 里让模型写它的含义（「作废记录」），不要写码值本身： 码值是没有出处的数字，会被数字核对打回重写 metrics_from 不写就取所有上游口径卡；on_violation 不写时探索运行是 flag（标出来、照常产出），正式运行是 fail； @@ -260,6 +291,10 @@ NODE_REFERENCE = """\ 不写 judge 的预算（judge 整个不写，金额、句数、时长上限都取设置里的默认）：正式运行在节点里自动判，探索运行由读的人 点开哪句判哪句，不在节点里花钱；报告要交给人工审批的，探索运行也在节点里判。用户明确说不要核对结论句的不加；只回答一个问题、一次查询的不必加。要按受管级别 发布的模板不适用这条默认，按下面受管级别的规则写预算 + analysis_claims 是分析说法核对：check（默认）判断词没挂对分析结论、原因没有依据的句子交回写作者重写一次，仍有的标在 + 报告里，不影响出具档位；strict 按 max_repairs 重写，仍有的计入出具缺口、出具降档；off 不核。只能往严里改 + （off < check < strict），不要为了少重写一次写成 off。analysis_scope：auto（默认：用到分析结论时，系统在报告末尾附一段 + 「分析范围」，写明拆过什么、排除了什么、哪些候选维度没拆）/ off，一般不写 entities 默认 link （核对表名、字段名），报告里根本不提表名时才写 off - output 的出具契约：config.contract = {report_from: 报告节点id, metrics_from:[口径卡id], @@ -268,10 +303,13 @@ NODE_REFERENCE = """\ （[[v:]] / [[table:]]）的，契约里写 cells: true，否则这些引用按解析不了算；只引单元格、图里没有口径卡的 （问数据的典型图），契约不写 metrics_from，也不写 required / expected 要按受管级别发布的模板（发布前检查不过就发不出去）：报告节点写明 numbers: "strict"、on_violation: "fail"、 + analysis_claims: "strict"、 claims: "require_citation"（或者更严的 claims: "judge"，这时 judge 里必须写 judge.max_cost_usd——写一个金额，或者 写 null 表示不限，预算要用户定，拿不准就问；只有受管级别要这样，普通草稿里默认开的 judge 不写预算、不为预算问人）；契约用 report_from（受管级别不收 narrative），成果字段只取报告节点的正文；给口径卡 供数的 agent 配 output_schema 和 cite_fields: true，code 节点喂口径卡的必须是真取数（evidence_role: "source"）， - 否则把计算写进口径卡；llm、agent、协作团队写的文字不能绕过报告节点直接进出口，llm 也不能喂口径卡 + 否则把计算写进口径卡；llm、agent、协作团队写的文字不能绕过报告节点直接进出口，llm 也不能喂口径卡。分析卡的输入要是 + 调用 db_query__ 的 tool 节点或 merge；分析卡的结论总能引用，agent 用分析工具现场做的分析和它查的格同一个规矩（契约 + 写 cells: true 才引用得了） - challenge：反方审稿。报告写完后挑出几句关键结论（挂了依据的、带比较或因果的优先），每句派一个只读 agent 站在反方 自己写查询找反例，逐句判经受住、被削弱、被推翻或无法判断，理由引用它自己查到的数据；不改报告。 config: {report, max_claims, max_queries, timeout_s, provider, model, on_refuted, enabled} @@ -299,6 +337,9 @@ NODE_REFERENCE = """\ 把它当成上周来写。想让报告说清数据最新到哪天，另取一列 MAX(日期) AS 数据最新日期。 写按周期取数的 SQL 前先看表的日期范围（look 结果里的剖析、取值观察）：只有一天的数据、或者最新日期早于要查的区间的表， 不能用来算这段时间（查出来全是 0）——换有历史的表；没有这样的表，就在说明里照实告诉用户这段时间没有数据 +7. 问题问为什么、怎么变了、哪里拖累、谁贡献最大、是否异常、波动、趋势、是不是真的更高这类分析问题时，查库的 agent 绑上 + 分析工具（判定由平台算，报告引用判定语）；模板里每次都要做的固定分析（周报的异动）用分析卡：取数 → 分析卡 → report → output。 + 分析卡的结果不进口径卡，报告直接引用 模型字段（llm / agent / supervisor / report 的 config.model）： - **不要填**。留空表示跟随当前供应商的默认模型，这几乎总是对的。 @@ -353,6 +394,26 @@ skip_if。这些字段按受限的 Python 表达式求值，变量直接写路 - 一个节点连出多条普通边 = 并行执行，多条边汇入同一节点 = 等待汇聚 """ +#: 节点手册里只讲分析能力的几段：(从这一行起, 到这一行之前)。评测「关掉分析工具」那一臂（answer_ops 的 analysis_tools 为假） +#: 不给这几段——agent 段里「分析类问题绑分析工具」那一条、分析卡一整条、搭图规则第 7 条：助手不知道有分析工具和分析卡， +#: 也不会被教着去用。报告段里分析结论的引用写法、分析说法核对照留（报告节点照样会核判断词） +_ANALYSIS_PROMPT_SPANS = ( + (" **问题问的是为什么、怎么变了", " **默认不要写 max_steps"), + ("- analysis：分析卡。", "- dual_check："), + ("7. 问题问为什么、怎么变了", "\n模型字段"), +) + + +def node_reference(*, analysis_tools: bool = True) -> str: + """给助手的节点手册。analysis_tools 为假时去掉只讲分析能力的那几段（_ANALYSIS_PROMPT_SPANS）。""" + if analysis_tools: + return NODE_REFERENCE + text = NODE_REFERENCE + for start, end in _ANALYSIS_PROMPT_SPANS: + i = text.index(start) + text = text[:i] + text[text.index(end, i):] + return text + class GenerateIn(BaseModel): instruction: str = Field(min_length=1, description="用自然语言描述想要的工作流") @@ -365,10 +426,33 @@ class GenerateIn(BaseModel): conversation_id: str | None = None #: 这一轮只查这几个数据源（id 或名字）。不传或空表示不限 datasource_ids: list[str] | None = None + #: 问数据里用户原本问的那句话。自动重来、重试本轮时 instruction 是带着原因的那段话（「上一次运行没有成功：…」）， + #: 自查按问题判分析意图（app/analysis/intent）时用这一句；不传就用 instruction + question: str | None = None -def _user_message(payload: GenerateIn, *, patch: bool) -> str: - """把用户说的话包成给模型的请求。 +def _question_of(payload: GenerateIn) -> str | None: + """自查判分析意图用的那句问题：只在问数据（intent answer）里有，画布上用户说的是一个流程，不是问题。""" + if payload.intent != "answer": + return None + return (payload.question or "").strip() or payload.instruction + + +#: 问数据的包装里「分析类问题」那一条形状（00-总方案 §3.7）：问原因、来源、异常、趋势、差异的，判定由平台的分析工具算、系统写 +#: 判定语，报告引用它。问题意图另由自查按 app/analysis/intent 的词表再核一遍：分析类问题而查库的 agent 没绑分析工具时交回改一轮 +ANALYSIS_SHAPE_PROMPT = ( + "- 问的是为什么、怎么变了、哪里拖累、谁贡献最大、是否异常、正常吗、波动、趋势、是不是真的更高这类分析问题时，搭 " + "input → agent → report → output：agent 绑 db_query__、db_schema__ 和五个分析工具（analysis_comparable、" + "analysis_baseline、analysis_trend、analysis_decompose、analysis_differ），prompt 里写清要回答的问题、比哪两期" + "（问题没说就比上一个整期），不用教分析方法（平台会注入）、不写阈值；report 的 instructions 写「回答：{{ input.question }}。" + "按分析结论写，判断直接引用判定语；没有分析结论的判断不下，原因没有依据的写成待核实的推测」（可以附上 " + "{{ nodes.agent的id.text }} 当分析思路）：数引用 agent 查过的单元格，判定引用它做过的分析结论。analysis_claims " + "用默认，不要关\n" +) + + +def _user_message(payload: GenerateIn, *, patch: bool, analysis_tools: bool = True) -> str: + """把用户说的话包成给模型的请求。analysis_tools 为假（评测「关掉分析工具」那一臂）时不给分析类问题那一条形状。 两个入口的语义完全不同，包法必须跟着变： @@ -404,7 +488,9 @@ def _user_message(payload: GenerateIn, *, patch: bool) -> str: "「回答：{{ input.question }}」（可以附上 {{ nodes.agent的id.text }} 当分析思路），数用 [[v:Q1.r0.列名]] " "直接引用 agent 查过的单元格，不必经过口径卡；要比率、增幅再在 agent 和 report 之间加口径卡，agent 配 " "output_schema + cite_fields，口径卡读 vars..字段。出口字段写 {{ nodes.report的id.text }}\n" - "- 只问表结构、清单（有哪些表、有哪些字段、列出名单）的，不加 report，agent → output 就行\n" + # 分析类问题那一条形状（ANALYSIS_SHAPE_PROMPT）：评测「关掉分析工具」那一臂不给 + + (ANALYSIS_SHAPE_PROMPT if analysis_tools else "") + + "- 只问表结构、清单（有哪些表、有哪些字段、列出名单）的，不加 report，agent → output 就行\n" # 三期起报告会核对写出来的表名、字段名：instructions 里让它「列出用到的表」时，编出来的名字会被标出来 "- report 提到表名、字段名时写 [[t:表名]] / [[c:表名.字段名]]（英文名也可以放进反引号，中文名不行：" "反引号里的中文名不核对），只写 agent 真查过的名字，" @@ -822,16 +908,18 @@ GRAPH_SCHEMA: dict[str, Any] = { def _graph_system(sources: list[Any], scope: set[str] | None, datasources: str, looks: Any, *, - chat_user: bool = False) -> str: + chat_user: bool = False, analysis_tools: bool = True) -> str: """画布、问数据（流式和非流式）共用的 system：节点手册、工具清单、数据源概览、结构要求、输出协议、查看的用法。 这一份一直用到自查修正结束（接着写、修正轮都不换），模型前后看到的是同一套说法。 - chat_user：开登录后 user 角色的问数据（D-c）。内置工具只列只读的那几个，结构要求后面再加一段使用者的约束。""" + chat_user：开登录后 user 角色的问数据（D-c）。内置工具只列只读的那几个，结构要求后面再加一段使用者的约束。 + analysis_tools 为假（评测「关掉分析工具」那一臂）：节点手册去掉只讲分析能力的几段，工具清单不列分析工具。""" from app.auth.chat_policy import CHAT_BUILTIN_TOOLS + catalog = _tool_catalog(sources, builtins=CHAT_BUILTIN_TOOLS if chat_user else None, analysis_tools=analysis_tools) return ( "你是一个 agent 工作流编排专家。根据用户需求，以操作流的方式逐步搭出一张可执行的工作流图。\n\n" - f"{NODE_REFERENCE}\n\n可用工具：\n{_tool_catalog(sources, builtins=CHAT_BUILTIN_TOOLS if chat_user else None)}" + f"{node_reference(analysis_tools=analysis_tools)}\n\n可用工具：\n{catalog}" f"{datasources}{_scope_note(sources, scope is not None)}\n\n" "结构要求：\n" "1. 必须有且只有一个 input 节点和至少一个 output 节点\n" @@ -883,7 +971,41 @@ def _chat_user_issues(nodes: dict[str, dict[str, Any]], edges: list[dict[str, An "去掉不允许的节点、工具和记忆范围 / 知识库配置"}] -def _graph_request(payload: GenerateIn) -> str: +#: 关掉分析工具时（answer_ops 的 analysis_tools 为假）图里还绑着分析工具的那条问题的 code +ANALYSIS_TOOLS_OFF = "analysis_tools_off" + + +def _analysis_off_issues(nodes: dict[str, dict[str, Any]]) -> list[dict[str, Any]]: + """评测「关掉分析工具」那一臂：这一轮的工具清单里没有分析工具，图里照样绑了的（Agent、协作成员、调用工具节点）按 + 「绑定的工具不存在」挡（同 engine/launch 发起前那一道的说法），交回模型去掉；改不好就不运行。一个节点一条。""" + from app.tools.analysis import ANALYSIS_TOOLS + + out: list[dict[str, Any]] = [] + for nid, node in nodes.items(): + cfg = ((node or {}).get("data") or {}).get("config") or {} + ntype = (node or {}).get("type") + if ntype == NodeType.TOOL: + names = [cfg.get("tool")] + elif ntype == NodeType.AGENT: + names = list(cfg.get("tools") or []) + elif ntype == NodeType.SUPERVISOR: + names = [t for m in cfg.get("agents") or [] if isinstance(m, dict) for t in m.get("tools") or []] + else: + continue + bound = list(dict.fromkeys(n.strip() for n in names if isinstance(n, str) and n.strip() in ANALYSIS_TOOLS)) + if not bound: + continue + title = str(((node or {}).get("data") or {}).get("label") or "").strip() or nid + field = "tool" if ntype == NodeType.TOOL else "agents" if ntype == NodeType.SUPERVISOR else "tools" + out.append({"level": "error", "node_id": nid, "edge_id": None, "field": field, "code": ANALYSIS_TOOLS_OFF, + "message": f"绑定的工具不存在：「{title}」绑定的 {'、'.join(bound)}。这一轮没有分析工具可用，请在节点中去掉它们", + "for_model": f"「{nid}」绑定的 {'、'.join(bound)} 不在这一轮的工具清单里，这一轮没有分析工具可用：从它的" + "工具里去掉这几个，只用上面列出的工具（db_query__、db_schema__ 照留）；数据要怎么比、怎么拆，" + "写进查询"}) + return out + + +def _graph_request(payload: GenerateIn, *, analysis_tools: bool = True) -> str: """这一轮的请求：改现有图时附上原图、只要改动操作；新建时按入口包好（_user_message）。""" if payload.base_graph and payload.base_graph.get("nodes"): return ( @@ -891,7 +1013,7 @@ def _graph_request(payload: GenerateIn) -> str: f"{json.dumps(_slim(payload.base_graph), ensure_ascii=False, indent=2)}\n\n" f"请按下面的要求修改它（只输出改动操作）：\n{payload.instruction}" ) - return _user_message(payload, patch=False) + return _user_message(payload, patch=False, analysis_tools=analysis_tools) def _start_graph(payload: GenerateIn) -> tuple[dict[str, dict[str, Any]], list[dict[str, Any]]]: @@ -932,7 +1054,8 @@ async def generate( baseline = copy.deepcopy((payload.base_graph or {}).get("nodes") or (prev_graph or {}).get("nodes") or []) applier = GraphApplier(nodes, edges, patches=False) gen = Generation(model, system=system, applier=applier, looks=looks, - checker=GraphChecker(applier, scope=scope, sources=sources, baseline=baseline, checkers=checkers)) + checker=GraphChecker(applier, scope=scope, sources=sources, baseline=baseline, checkers=checkers, + question=_question_of(payload))) human = _with_history(_graph_request(payload), await _history_section(session, payload.conversation_id)) try: async for _op in gen.draft(human): @@ -956,6 +1079,7 @@ async def generate( async for _op in gen.repair(): pass covered = await gen.checker.coverage(level="warning") if isinstance(gen.checker, GraphChecker) else [] + covered += gen.checker.analysis_gaps(level="warning") if isinstance(gen.checker, GraphChecker) else [] try: spec = GraphSpec.model_validate({"nodes": list(nodes.values()), "edges": edges}) @@ -1152,12 +1276,14 @@ def _blocking_issues( nodes: dict[str, dict[str, Any]], edges: list[dict[str, Any]], scope: set[str] | None = None, *, sources: list[Any] | None = None, baseline: list[dict[str, Any]] | None = None, level: str | None = None, defaults: dict[str, Any] | None = None, - checkers: dict[str, Any] | None = None, chat_user: bool = False, + checkers: dict[str, Any] | None = None, chat_user: bool = False, analysis_tools: bool = True, ) -> list[dict[str, Any]] | None: """按运行时同一套校验，挑出会挡住运行的问题。图本身不成形时返回 None。 chat_user：图要以开登录后 user 角色的身份在问数据里发起（D-c），使用者不能用的节点、工具、配置也挡（_chat_user_issues）。 + analysis_tools 为假（评测「关掉分析工具」那一臂）：图里还绑着分析工具的也挡（_analysis_off_issues）。 + 限定了数据源时，用了范围外的库也算：用户点了「只查这个库」，拿别的库的数 回答他，比跑不起来更糟——答案看起来是对的。 @@ -1186,7 +1312,8 @@ def _blocking_issues( out = ([i.model_dump() for i in validate_graph(spec).issues if i.level == "error"] + _issue_dicts(scope_issues(list(nodes.values()), scope), "datasource_out_of_scope") + [i for i in authored if i["level"] == "error"] - + (_chat_user_issues(nodes, edges) if chat_user else [])) + + (_chat_user_issues(nodes, edges) if chat_user else []) + + ([] if analysis_tools else _analysis_off_issues(nodes))) if level: from app.engine.governance import lint_for_publish @@ -1237,9 +1364,9 @@ def _numeric_input(field: Any) -> bool: def _evidence_source(node: Any) -> bool: - """报告撰写能从这个节点拿到可引用的证据：口径卡、调用工具、配了工具的 agent、知识检索，以及声明了数值字段的 - 入口（运行输入进证据目录，报告用 [[i:字段]] 引用）。子工作流、协作团队里面是什么看不见，一律算有——这条会把图 - 打回去让模型改，宁可漏判不可误判。 + """报告撰写能从这个节点拿到可引用的证据：口径卡、分析卡（结论总能引用，判定语、具名的数、结果表，00-总方案 D10）、 + 调用工具、配了工具的 agent、知识检索，以及声明了数值字段的入口（运行输入进证据目录，报告用 [[i:字段]] 引用）。 + 子工作流、协作团队里面是什么看不见，一律算有——这条会把图打回去让模型改，宁可漏判不可误判。 入口只声明了文字字段的（问题、主题）不算：报告能照引的只有问题本身，要写的数照样没有出处。画布上的入口字段 不写类型，没写默认值的字段看不出是不是数，也不算——提示里说明了怎么声明（写数值默认值）。""" @@ -1248,8 +1375,8 @@ def _evidence_source(node: Any) -> bool: if node.type == NodeType.INPUT: fields = node.config.get("fields") return isinstance(fields, list) and any(_numeric_input(f) for f in fields) - return node.type in (NodeType.METRICS, NodeType.TOOL, NodeType.MERGE, NodeType.RETRIEVE, NodeType.SUBGRAPH, - NodeType.SUPERVISOR) + return node.type in (NodeType.METRICS, NodeType.ANALYSIS, NodeType.TOOL, NodeType.MERGE, NodeType.RETRIEVE, + NodeType.SUBGRAPH, NodeType.SUPERVISOR) def _unsourced_reports(spec: GraphSpec, changed: set[str]) -> list[dict[str, Any]]: @@ -1681,7 +1808,7 @@ async def generate_stream(payload: GenerateIn): ) -async def answer_ops(payload: GenerateIn, *, catalog_view: Any = None): +async def answer_ops(payload: GenerateIn, *, catalog_view: Any = None, analysis_tools: bool = True): """生成过程：按需求起草、按需查看、自查修正、排版校验，逐条交出操作（op dict）。问数据页（经 SSE）和评测（直接迭代） 走的是这同一条路（细案 01 做法 2）。 @@ -1693,6 +1820,10 @@ async def answer_ops(payload: GenerateIn, *, catalog_view: Any = None): 拒绝在这里抛 HTTPException（调用方没迭代到第一条就知道）。catalog_view 是这一轮读哪一份数据目录（data/catalog_view.py： 查看的表详情、自查的 SQL 检查器都按它），不给是当前目录。 + analysis_tools 为假是评测「关掉分析工具」那一臂（分析工具开、关对照，chat/drive.TurnDriver 的 analysis_tools 透传过来）： + 工具清单不列五个分析工具，节点手册、问数据的包装里只讲分析能力的那几段不给，问题意图那条自查不提；图里照样绑了分析工具 + 的按「绑定的工具不存在」挡（_analysis_off_issues）。缺省为真，和以前一字不差。 + final 带 next（chat/turn.next_step：run 运行、ready 只建不跑、fail 判失败），问数据页照它走。最后多交一条 usage （providers/usage.py）：{op: usage, by_model, input_tokens, output_tokens, calls, cost_usd, unpriced_calls}——这一轮助手建图（含挑表、接着写、自查修正）花了多少。前面每一条和抽出来之前逐条一样。运行不在这里， @@ -1724,7 +1855,8 @@ async def answer_ops(payload: GenerateIn, *, catalog_view: Any = None): except ProviderNotConfigured as e: raise HTTPException(400, _unconfigured(e)) from e - human = _with_history(_graph_request(payload), await _history_section(session, payload.conversation_id)) + human = _with_history(_graph_request(payload, analysis_tools=analysis_tools), + await _history_section(session, payload.conversation_id)) # 兜底用：模型只发了改动操作时，把它们重放到这张图上 prev_graph = await _previous_graph(session, payload.conversation_id) # 比对工具绑定的基准：画布上是改之前的图；问数据页是上一轮的图（沿用它的节点 id） @@ -1738,12 +1870,14 @@ async def answer_ops(payload: GenerateIn, *, catalog_view: Any = None): looks = await _look_service(session, payload, sources, graph=payload.base_graph if ( payload.base_graph or {}).get("nodes") else prev_graph, catalog_view=view, kinds=kinds) # 这一份 system 一直用到最后：接着写、自查修正都沿用，不换 - system = _graph_system(sources, scope, await copilot_context.section(looks), looks, chat_user=chat_user) + system = _graph_system(sources, scope, await copilot_context.section(looks), looks, chat_user=chat_user, + analysis_tools=analysis_tools) applier = GraphApplier(nodes, edges, composer=copilot_context.composer(looks)) started = time.monotonic() gen = Generation(model, system=system, applier=applier, looks=looks, started=started, checker=GraphChecker(applier, scope=scope, sources=sources, baseline=baseline, checkers=checkers, - chat_user=chat_user)) + chat_user=chat_user, question=_question_of(payload), + analysis_tools=analysis_tools)) async def steps(): yield {"op": "model", "model": model_id} @@ -1810,7 +1944,7 @@ async def answer_ops(payload: GenerateIn, *, catalog_view: Any = None): async for op in _with_heartbeat(gen.repair(), phase="repairing"): yield op remaining = _blocking_issues(nodes, edges, scope, sources=sources, baseline=baseline, checkers=checkers, - chat_user=chat_user) + chat_user=chat_user, analysis_tools=analysis_tools) # 工具绑定变化不挡运行，但要在自查这一步就说出来：工具被改没了的图照样 # 能跑，只是跑出来的是模型「假设」查过库的答案 changes = tool_changes(baseline, list(nodes.values())) @@ -1818,6 +1952,8 @@ async def answer_ops(payload: GenerateIn, *, catalog_view: Any = None): # 工具节点写死的 SQL 查的区间里表里没有数据（data/coverage.py）：自查时交回模型改过一轮，仍在的不挡运行，留在检查 # 结果里当提醒（模型照要求在说明里告诉用户） warnings += await gen.checker.coverage(level="warning") if isinstance(gen.checker, GraphChecker) else [] + # 分析类问题、查库的 Agent 没绑分析工具：同上，交回模型改过一轮仍在的不挡运行，留作提醒 + warnings += gen.checker.analysis_gaps(level="warning") if isinstance(gen.checker, GraphChecker) else [] # 建完要不要跑，由模型在 done.run 里表态；问了用户的不跑：要等他回答 autorun = applier.autorun and gen.asked is None if remaining: @@ -1844,6 +1980,8 @@ async def answer_ops(payload: GenerateIn, *, catalog_view: Any = None): issues += [i for i in authored if i["code"] in CHECK_CODES] # 使用者发起不了的图（D-c）：自查没改好的照实列出，界面据此不运行、说清原因 issues += _chat_user_issues(nodes, edges) if chat_user else [] + # 关掉分析工具的那一臂里还绑着分析工具的（自查没改好）：同上，照实列出，不运行 + issues += [] if analysis_tools else _analysis_off_issues(nodes) # 跳过的节点要说出来，不能安静地少一步。code 给前端认：这一类要单独 # 提示「少了一步」，不能和普通校验警告混在一起 issues += [ @@ -1893,8 +2031,9 @@ _NO_LOWERING = """\ 不删掉或清空出具契约，不删 required 里的指标，不把 strict 改成 false，不往 allow_numbers 里加数，不替人打开 cells； 审批策略只能往严里改（每次调用都审批 > 仅危险工具需要审批 > 全部自动放行），不放宽、不删掉，更不改成 never； 不取消子工作流钉住的版本；发布级别不是你能改的。结构要动（比如换掉协作团队），输出 question 说明怎么拆，不要自己动手； - 报告撰写节点的 numbers、on_violation、claims 只能往严里改（numbers: strict、on_violation: fail、 - claims: require_citation），不放宽、不删掉；claims 从宽到严是 off < require_citation < judge，已经是 judge 的不改回 + 报告撰写节点的 numbers、on_violation、claims、analysis_claims 只能往严里改（numbers: strict、on_violation: fail、 + claims: require_citation、analysis_claims: strict），不放宽、不删掉；analysis_claims 从宽到严是 off < check < strict， + 已经是 strict 的不改回 check；claims 从宽到严是 off < require_citation < judge，已经是 judge 的不改回 require_citation；judge 缺预算（judge.max_cost_usd）时输出 question 问写多少、还是写 null 不限，不替人定； judge.on_unsupported 是 withhold 的不改回 degrade、不删掉；update_node 会把 judge 整个换掉，补预算时把原来的 judge（包括从全图默认继承来的）整份带上，只加 max_cost_usd；契约里的 claims 也一样； diff --git a/backend/app/api/copilot_gen.py b/backend/app/api/copilot_gen.py index 68c7a888..8e207c4d 100644 --- a/backend/app/api/copilot_gen.py +++ b/backend/app/api/copilot_gen.py @@ -707,12 +707,19 @@ class GraphChecker: """图的自查：按运行时同一套规则（copilot._blocking_issues）挑出挡路的问题，交回模型改。 known 给了（升级证据）：只把这一轮冒出来的问题交回去（图上原来就有的不归这一轮管），一个修改都没落地时不修。 + + question 给了（问数据：用户问的那句话）：问题是分析类（app/analysis/intent）、查库的 Agent 却没绑分析工具时另提一条 + （analysis_gaps），同 coverage 只交回模型改一轮。analysis_tools 为假（评测「关掉分析工具」那一臂）时这一条不提，图里 + 还绑着分析工具的按挡路的问题算（copilot._analysis_off_issues）。 """ def __init__(self, applier: GraphApplier, *, scope: set[str] | None = None, sources: list[Any] | None = None, baseline: list[dict[str, Any]] | None = None, level: str | None = None, defaults: dict[str, Any] | None = None, checkers: dict[str, Any] | None = None, - known: set[Any] | None = None, chat_user: bool = False) -> None: + known: set[Any] | None = None, chat_user: bool = False, question: str | None = None, + analysis_tools: bool = True) -> None: + from app.analysis.intent import analysis_intent + self.applier = applier #: 图要以开登录后 user 角色的身份在问数据里发起（D-c）：使用者不能用的节点、工具也算挡路 self.chat_user = chat_user @@ -728,13 +735,19 @@ class GraphChecker: self._coverage_cache: Any = None #: 这类问题已经交回模型改过一轮：之后不再挡（修正后仍在的留在检查结果里当提醒，由模型在说明里照实告诉用户） self._coverage_given = False + #: 这一轮给不给分析工具（评测「关掉分析工具」那一臂为假） + self.analysis_tools = analysis_tools + #: 用户问题的分析意图（问为什么、怎么变了、是否异常、趋势……），不是分析类问题、或者这一轮没有分析工具时为 None + self.intent = analysis_intent(question) if question and analysis_tools else None + #: 「分析类问题、Agent 没绑分析工具」已经交回模型改过一轮：之后不再挡，同 _coverage_given + self._analysis_given = False def blocking(self) -> list[dict[str, Any]] | None: from app.api.copilot import _blocking_issues return _blocking_issues(self.applier.nodes, self.applier.edges, self.scope, sources=self.sources, baseline=self.baseline, level=self.level, defaults=self.defaults, - checkers=self.checkers, chat_user=self.chat_user) + checkers=self.checkers, chat_user=self.chat_user, analysis_tools=self.analysis_tools) async def issues(self) -> list[dict[str, Any]]: errors = self.blocking() @@ -743,7 +756,10 @@ class GraphChecker: # 查的区间里表里没有数据：交回模型改一轮（换有这段数据的表，或者在说明里照实告诉用户）；改过一轮仍在的不挡 covered = [] if self._coverage_given or errors is None else await self.coverage() self._coverage_given = self._coverage_given or bool(covered) - return [*refused, *(errors or []), *covered] + # 分析类问题、查库的 Agent 没绑分析工具：同样只交回模型改一轮（用不上分析的，模型在说明里讲一句为什么就行） + analysis = [] if self._analysis_given or errors is None else self.analysis_gaps() + self._analysis_given = self._analysis_given or bool(analysis) + return [*refused, *(errors or []), *covered, *analysis] from app.engine.autofix import _sig if not self.applier.effective: @@ -793,9 +809,49 @@ class GraphChecker: return await coverage.sql_gaps(source, sql, facts=_profile_facts(row), masked=masked_columns(row.options), cache=self._coverage_cache, budget=COVERAGE_BUDGET_S) + def analysis_gaps(self, *, level: str = "error") -> list[dict[str, Any]]: + """问题是分析类（self.intent），图里没有分析卡，查库的 Agent 一个分析工具都没绑：一条问题（形状同自查的问题，code 是 + ANALYSIS_TOOLS_MISSING），指向第一个查库的 Agent。没有查库的 Agent 的图（只用调用工具节点取数、选用了口径）不提：这条 + 说的是「查库的 Agent 要绑分析工具」，别的结构由助手按节点手册定。level 给 warning 时是收尾那一步的提醒。""" + if self.intent is None: + return [] + from app.analysis.contract import TOOL_NAMES + from app.tools.datasource import QUERY_PREFIX + + bound = set(TOOL_NAMES.values()) + querying: tuple[str, str] | None = None + for nid, node in self.applier.nodes.items(): + ntype = (node or {}).get("type") + if ntype == NodeType.ANALYSIS: + return [] + if ntype != NodeType.AGENT: + continue + data = (node or {}).get("data") or {} + tools = {t.strip() for t in (data.get("config") or {}).get("tools") or [] if isinstance(t, str)} + if tools & bound: + return [] + if querying is None and any(t.startswith(QUERY_PREFIX) for t in tools): + querying = (nid, str(data.get("label") or "").strip() or nid) + if querying is None: + return [] + nid, title = querying + asked = f"问题问的是{self.intent.label}（「{self.intent.text}」）" + names = "、".join(TOOL_NAMES[op] for op in ANALYSIS_TOOL_ORDER) + return [{"level": level, "node_id": nid, "edge_id": None, "field": "tools", "code": ANALYSIS_TOOLS_MISSING, + "message": f"{asked}，查库的「{title}」没有绑定分析工具：变化从哪来、是否异常、是不是趋势只能由模型自己判断，" + "报告里的这类判断没有分析结论可依据", + "for_model": f"{asked}。按「分析类问题」的形状改：给查库的 agent「{nid}」的 tools 加上 {names}" + "（db_query__、db_schema__ 照留），平台会把分析方法注入它的 system，prompt 里写清要回答的问题、" + "比哪两期就行；report 的 instructions 让它按分析结论写、判断直接引用判定语。问的其实不是原因、" + "异常、趋势这类、确实用不上分析的，不改这一处，在 done 的 explanation 里说一句为什么"}] + #: 自查时一条 SQL 的覆盖检查最多花多少秒（data/coverage.py）：搭图要等它，比查询时的 BUDGET_S 短 COVERAGE_BUDGET_S = 8.0 +#: 分析类问题、查库的 Agent 没绑分析工具（GraphChecker.analysis_gaps）那条问题的 code +ANALYSIS_TOOLS_MISSING = "analysis_tools_missing" +#: 自查让助手绑分析工具时列出的先后：先定现象（可比性、基线、趋势），再拆、再检验（同分析方法 tools/builtin/analysis.py） +ANALYSIS_TOOL_ORDER = ("comparable", "baseline", "trend", "decompose", "differ") async def _probe_source(row: Any) -> Any: diff --git a/backend/app/chat/drive.py b/backend/app/chat/drive.py index d406b27f..7e1a8041 100644 --- a/backend/app/chat/drive.py +++ b/backend/app/chat/drive.py @@ -123,13 +123,16 @@ class TurnDriver: def __init__(self, question: str, *, datasource_ids: list[str], on_phase: Callable[[str], Awaitable[None]] | None = None, - provider: str | None = None, model: str | None = None) -> None: + provider: str | None = None, model: str | None = None, analysis_tools: bool = True) -> None: self.question = question self.datasource_ids = list(datasource_ids) self.on_phase = on_phase #: 助手模型（生成、复核都用它）：评测按变体覆盖，问数据用设置里的 self.provider = provider self.model = model + #: 这一轮给不给助手分析工具：透传给 api/copilot.answer_ops 的 analysis_tools。只给评测的「分析工具开、关」对照用 + #: （eval/answer.Driver 按变体设），问数据页不暴露这个开关，一直是真 + self.analysis_tools = analysis_tools self.turn = Turn() self.terminal: dict[str, Any] | None = None self.refusal: Exception | None = None @@ -150,11 +153,12 @@ class TurnDriver: # ------------------------------------------------------------------ 子类给的差异 def generate_in(self, instruction: str, base_graph: dict[str, Any] | None) -> Any: - """这一次生成的请求（api/copilot.GenerateIn）。""" + """这一次生成的请求（api/copilot.GenerateIn）。question 是这一轮原本的问题：重来时 instruction 是带着原因的那段话， + 自查按原本的问题判分析意图。""" from app.api.copilot import GenerateIn return GenerateIn(instruction=instruction, base_graph=base_graph, intent="answer", provider=self.provider, - model=self.model, datasource_ids=self.datasource_ids or None) + model=self.model, datasource_ids=self.datasource_ids or None, question=self.question) def catalog_view(self) -> Any: """这一轮读哪一份数据目录（data/catalog_view.py），None 是当前目录。""" @@ -199,7 +203,7 @@ class TurnDriver: payload = self.generate_in(instruction, base_graph) ops: list[dict[str, Any]] = [] try: - async for op in answer_ops(payload, catalog_view=self.catalog_view()): + async for op in answer_ops(payload, catalog_view=self.catalog_view(), analysis_tools=self.analysis_tools): await self.on_op(op) if op.get("op") in ("thinking", "heartbeat"): continue diff --git a/backend/app/chat/job.py b/backend/app/chat/job.py index 6bfdfae9..d45e7365 100644 --- a/backend/app/chat/job.py +++ b/backend/app/chat/job.py @@ -153,7 +153,8 @@ class ChatDriver(TurnDriver): from app.api.copilot import GenerateIn return GenerateIn(instruction=instruction, base_graph=base_graph, intent="answer", - conversation_id=self.execution.conversation_id, datasource_ids=self.datasource_ids or None) + conversation_id=self.execution.conversation_id, datasource_ids=self.datasource_ids or None, + question=self.question) async def on_op(self, op: dict[str, Any]) -> None: kind = op.get("op") diff --git a/backend/app/engine/review.py b/backend/app/engine/review.py index a0387a69..74dba4ba 100644 --- a/backend/app/engine/review.py +++ b/backend/app/engine/review.py @@ -6,9 +6,11 @@ from collections.abc import Iterable, Mapping from dataclasses import dataclass from typing import Any +from app.analysis.contract import OP_LABELS, TOOL_NAMES from app.core.events import EventType from app.engine.issuance import extract_numbers, trace_numbers from app.engine.toolcalls import leaked_markup +from app.tools.analysis import CALL_ERROR_HEAD, REFUSED_CODE # -------------------------------------------------------------------------- # 跑完之后的复核层。 @@ -145,11 +147,15 @@ WARN_CODES: dict[str, tuple[str, str]] = { # 分析能力一轮（00-总方案 §3.7）：只写说明、不自动重来（NOTE_ONLY）。答案照常交付，缺口照实说一句就够了： # 报告里判断词没挂对分析结论、原因没有依据，重写后仍有（check 标在报告里，strict 另记出具缺口） "report_analysis_claims": ("report_analysis_claims", DEGRADED), - # 一份分析算不了（历史不够、本期不完整、截断……）：算不了本身是一条结论，写作者照实写原因 - "analysis_refused": ("analysis_refused", DEGRADED), + # 一份分析算不了（历史不够、本期不完整、截断……）：算不了本身是一条结论，写作者照实写原因。一次运行里几次算不了的 + # 并成一条「有 n 次分析没做成」（_analysis_signal），不一次一条 + REFUSED_CODE: (REFUSED_CODE, DEGRADED), # 口径卡上用到两次查询的指标没做成可比性检查（探查失败、算不了）：指标照常算，两期能不能直接比要人自己核对 "metric_comparability_skipped": ("metric_comparability_skipped", DEGRADED), } +#: 分析调用写错了、后来同一个分析工具一次都没做成（tool.error 带 call_error，见 scan）：这几次分析没做成，同 analysis_refused +#: 是缺口不是断裂。它不是日志代码，是 scan 从工具事件里认出来的信号种类 +ANALYSIS_CALL_FAILED = "analysis_call_failed" # 这些 kind 重跑一次大概率能好：原因说得清，且不是「问题本身没问明白」。 # 工具参数错了、步数不够、图没校验过，都属于改一下再来一遍就行的。 @@ -160,7 +166,9 @@ REBUILD = {"query_no_data", "metric_no_data"} #: 只写说明、不自动重来的信号（00-总方案 §3.7）：都是缺口（DEGRADED）、不进 RETRYABLE / REBUILD。一次运行的信号全是 #: 这几种时复核不叫模型，说明就照信号原话写（review）：答案照常、报告里已经标出，叫一次模型只是把同一句话换个说法 -NOTE_ONLY = frozenset({"report_analysis_claims", "analysis_refused", "metric_comparability_skipped"}) +NOTE_ONLY = frozenset({"report_analysis_claims", REFUSED_CODE, "metric_comparability_skipped", ANALYSIS_CALL_FAILED}) +#: 「有 n 次分析没做成」那条信号里最多点名几次，其余写「……」 +_ANALYSIS_SHOWN = 3 #: 串行工具模式下塞给未执行调用的占位说明。它漏到用户面前当过答案， #: 所以复核要认得出来。从 nodes.llm 导入会绕成循环，这里按内容识别。 @@ -229,12 +237,25 @@ def scan(events: Iterable[Any], output: Mapping[str, Any] | None = None) -> list finals = _final_checks(rows) #: 报过的节点失败 (节点 id, 原话)：子工作流节点跟着子图里那一步失败时认得出是同一件事 failures: list[tuple[str, str]] = [] - for e in rows: + #: 没做成的分析：算不了的（analysis_refused 日志）、调用写错了一直没改对的（带 call_error 的 tool.error）。各并成一条信号， + #: 排在第一次出现的位置（slots：信号种类 → (那时信号清单有多长, 第几条事件)） + refused: list[tuple[str, Mapping[str, Any]]] = [] + unfixed: list[tuple[str, Mapping[str, Any]]] = [] + slots: dict[str, tuple[int, int]] = {} + for seq, e in enumerate(rows): etype = str(_event_field(e, "type", "")) data = _event_field(e, "data") or {} node_id = _event_field(e, "node_id") or "" - if etype == EventType.TOOL_ERROR: + if etype == EventType.TOOL_ERROR and data.get("call_error"): + # 分析调用写错了（contract.CallError：列不存在、比率当可加、这次执行的分析次数用完……）：不记结论，原话交回 Agent + # 照着改。后来同一个分析工具做成了，那是它改对了，不是缺口；一直没做成的，那几次分析没做成——缺口不是断裂， + # 只写说明、不重来（NOTE_ONLY）：重来一遍，Agent 照样可能不做这项分析 + if str(data.get("tool") or "") not in recovered: + slots.setdefault(ANALYSIS_CALL_FAILED, (len(signals), seq)) + unfixed.append((str(node_id), data)) + + elif etype == EventType.TOOL_ERROR: tool = str(data.get("tool") or "工具") err = str(data.get("error") or "").strip()[:200] if tool in recovered: @@ -274,6 +295,10 @@ def scan(events: Iterable[Any], output: Mapping[str, Any] | None = None) -> list q = str(data.get("query") or "").strip()[:60] add("retrieve_empty", f"知识库里没有检索到与「{q}」相关的内容", BROKEN) + elif etype == EventType.LOG and data.get("level") == "warn" and data.get("code") == REFUSED_CODE: + slots.setdefault(REFUSED_CODE, (len(signals), seq)) + refused.append((str(node_id), data)) + elif etype == EventType.LOG and data.get("level") == "warn": message = str(data.get("message") or "").strip() code = str(data.get("code") or "") @@ -286,6 +311,12 @@ def scan(events: Iterable[Any], output: Mapping[str, Any] | None = None) -> list if message: add(kind, message[:300], severity, reason if isinstance(reason, str) and reason else None) + # 先插靠后的那条（位置一样时先插后出现的），靠前那条的位置就不受影响 + for kind in sorted(slots, key=lambda k: slots[k], reverse=True): + items = refused if kind == REFUSED_CODE else unfixed + signals.insert(slots[kind][0], Signal(kind, _analysis_signal(kind, items, titles), DEGRADED)) + kinds.add(kind) + text = answer_text(output) if not text: add("empty_output", "这次运行没有产出任何成果内容", BROKEN) @@ -307,6 +338,48 @@ def _step_name(node_id: str, titles: Mapping[str, str]) -> str: return " › ".join(titles.get("/".join(parts[:i + 1])) or parts[i] for i in range(len(parts))) +#: 分析工具名 → 算子（contract.TOOL_NAMES 反过来）：说「波动基线没有做成」，不写工具名 +_TOOL_OPS = {name: op for op, name in TOOL_NAMES.items()} + + +def _analysis_who(node_id: str, data: Mapping[str, Any], titles: Mapping[str, str]) -> str | None: + """没做成的那次分析是谁做的、哪一种：「「周报分析」的分析项「wow」（波动基线）」「「数据查询」的变化拆解」「「分析团队」的 + 成员「取数员」的趋势判断」。认不出是哪一种分析（老运行、事件里没带）为 None。""" + op = data.get("op") if data.get("op") in OP_LABELS else _TOOL_OPS.get(str(data.get("tool") or "")) + if op is None: + return None + where = f"「{_step_name(node_id, titles)}」" if node_id else "" + if data.get("item"): + return f"{where}的分析项「{data['item']}」（{OP_LABELS[op]}）" + member = f"的成员「{data['agent']}」" if data.get("agent") else "" + return f"{where}{member}的{OP_LABELS[op]}" if where or member else OP_LABELS[op] + + +def _analysis_why(kind: str, data: Mapping[str, Any]) -> str: + """没做成的原因：算不了的照结论里的原因；调用写错的照交回 Agent 的那句（去掉「分析调用有误，没有记成结论：」的开头）。""" + if kind == REFUSED_CODE: + return str(data.get("reason") or "").strip() or "原因未知" + error = str(data.get("error") or "").strip() + return (error[len(CALL_ERROR_HEAD):] if error.startswith(CALL_ERROR_HEAD) else error).strip() or "原因未知" + + +def _analysis_signal(kind: str, items: list[tuple[str, Mapping[str, Any]]], titles: Mapping[str, str]) -> str: + """一次运行里没做成的分析并成的那一句：一次时照实说哪一项、为什么；几次时先说「有 n 次分析没做成」，再点名前几次。 + 认不出是哪一种分析的（老运行的事件）照日志原话。复核模型、只写说明的复核都照这一句说。""" + told: list[tuple[str | None, str, str]] = [] + for node_id, data in items: + told.append((_analysis_who(node_id, data, titles), _analysis_why(kind, data), + str(data.get("message") or data.get("error") or "").strip())) + verb = "没有做成" if kind == REFUSED_CODE else "的调用写错了、一直没改对，没有做成" + if len(told) == 1: + who, why, raw = told[0] + return (f"{who}{verb}：{why}" if who else raw or f"有 1 次分析没有做成：{why}")[:300] + head = f"有 {len(told)} 次分析没做成" if kind == REFUSED_CODE else f"有 {len(told)} 次分析调用写错了、一直没改对，没有做成" + named = "；".join(f"{who}：{why}" if who else raw or why for who, why, raw in told[:_ANALYSIS_SHOWN]) + more = "；……" if len(told) > _ANALYSIS_SHOWN else "" + return f"{head}。{named}{more}"[:300] + + #: 报告节点「改过一次」的信号：按同一个节点最后一次 report.checked 的结果定档（最终通过的不是缺口） _SETTLED_BY_REPORT = ("report_repair", "report_rewrite") #: report.checked 最多带几条违规（报告节点截到前 20 条）：带满了的，后面可能还有，不当作只剩标注 diff --git a/backend/app/seed.py b/backend/app/seed.py index 213bfba2..478a6fb3 100644 --- a/backend/app/seed.py +++ b/backend/app/seed.py @@ -338,6 +338,64 @@ TEMPLATES: list[dict[str, Any]] = [ ], "edges": [_e("start", "review"), _e("review", "approved", "approved"), _e("review", "rejected", "rejected")], }, + { + # 分析能力一轮（docs/design/analysis-round/00-总方案.md §3.7）：模板定骨架——分析卡固定做哪几项分析写在图里，判定由 + # 系统按平台的方法和阈值算、系统写判定语，报告撰写节点只引用。照模板 ⑨ 的写法，拿来就能按受管级别发布 + "name": "⑪ 异动分析（可追溯）", + "description": "周销售额的异动分析，每个判断都可追溯：两个「调用工具」节点取数（整体按周、按地区两周），分析卡固定做" + "两项分析——这一周的变化在不在常规波动范围内、变化主要来自哪个地区（拆开后各段之和和总数核对闭合），报告撰写节点只引用" + "判定语和数，判断词逐句核对。使用前请接入名为 shop 的数据源（含 orders 表的 order_date、region、amount 列），或上传" + "同名表格。SQL 按 SQLite 的写法，按周汇总时期次取每周第一天（周一）的日期；接别的库时换成那个库的写法。报告撰写节点已按" + "受管级别设置核对规则。", + "tags": ["分析", "出具", "周报", "证据"], + "nodes": [ + _n("start", "input", "周期", fields=[ + {"name": "week", "default": "2026-10-12", "description": "本周的第一天（周一）"}, + {"name": "prev_week", "default": "2026-10-05", "description": "上周的第一天（周一）"}]), + # 期次在 SQL 里由日期列算出来（每周第一天的日期）：分析卡据此认出每一期覆盖哪几天，本期过没过完核对得了。 + # 倒序排：第 0 行是本周、第 1 行是上周，变化拆解按这两格核对闭合 + _n("weekly", "tool", "整体按周（26 周）", tool="db_query__shop", args={"sql": ( + "SELECT date(order_date, '-6 days', 'weekday 1') AS \"周\", SUM(amount) AS \"销售额\"\n" + "FROM orders\n" + "WHERE order_date >= date('{{ input.week }}', '-175 days')" + " AND order_date < date('{{ input.week }}', '+7 days')\n" + "GROUP BY date(order_date, '-6 days', 'weekday 1')\n" + "ORDER BY \"周\" DESC")}), + _n("by_region", "tool", "按地区两周", tool="db_query__shop", args={"sql": ( + "SELECT region AS \"地区\", date(order_date, '-6 days', 'weekday 1') AS \"周\", SUM(amount) AS \"销售额\"\n" + "FROM orders\n" + "WHERE order_date >= '{{ input.prev_week }}' AND order_date < date('{{ input.week }}', '+7 days')\n" + "GROUP BY region, date(order_date, '-6 days', 'weekday 1')\n" + "ORDER BY \"地区\", \"周\"")}), + # 阈值不写在图里：取设置里的分析一组（发起时快照）；要固定阈值时在这张卡上写「阈值覆盖」 + _n("check", "analysis", "异动分析卡", items=[ + {"id": "wow", "op": "baseline", "from": "weekly", + "params": {"period": "周", "measure": "销售额", "target": "{{ input.week }}", "grain": "week", + "label": "销售额"}, + "hypothesis": "本周销售额的变化在不在常规波动范围内"}, + {"id": "where", "op": "decompose", "mode": "additive", "from": "by_region", + "params": {"by": ["地区"], "period": "周", "base": "{{ input.prev_week }}", "target": "{{ input.week }}", + "measure": "销售额", "headline": {"base": "weekly.r1.销售额", "target": "weekly.r0.销售额"}, + "label": "销售额"}, + "hypothesis": "本周销售额的变化主要来自哪个地区"}, + ]), + # 受管出具要求报告撰写节点写明最严的几项（发布门禁 G3，含分析说法核对设为严格），模板照着写 + _n("write", "report", "报告撰写", numbers="strict", on_violation="fail", claims="require_citation", + analysis_claims="strict", + system="你是经营分析撰写人：结论先行，判断照分析结论写，只写有数据支撑的内容。", + instructions="为 {{ input.week }} 开始的这一周写一份销售额异动分析：先用一句话说本周销售额和上周比变了多少，" + "再按分析卡的两项结论说这个变化在不在常规波动范围内、主要来自哪个地区，最后用一张表列出各地区的" + "变化。判断照分析结论写、直接引用判定语，没有分析结论的判断不下；有一项没有做成的，照实写原因。"), + # 图里没有口径卡：报告直接引用查询和分析结论的格，契约开「单元格引用」（受管级别要写明） + _n("done", "output", "出具", + fields=[{"name": "异动分析", "value": "{{ nodes.write.text }}"}], + contract={"report_from": "write", "cells": True, "strict": True}), + ], + "edges": [ + _e("start", "weekly"), _e("weekly", "by_region"), _e("by_region", "check"), + _e("check", "write"), _e("write", "done"), + ], + }, ] diff --git a/backend/app/tools/builtin/analysis.py b/backend/app/tools/builtin/analysis.py index 89213a2a..43efd152 100644 --- a/backend/app/tools/builtin/analysis.py +++ b/backend/app/tools/builtin/analysis.py @@ -22,20 +22,22 @@ from app.tools.registry import ToolContext, ToolGuide, register CATEGORY = "分析" -#: 「分析方法」：五个分析工具共用，绑了任一个就注入 system（第一版，第 2 波定稿）。改了方法就改 id 里的版本号：评测的 +#: 「分析方法」：五个分析工具共用，绑了任一个就注入 system（第 2 波定稿为第二版）。改了方法就改 id 里的版本号：评测的 #: prompt_fingerprint 认得出提示词变了 ANALYSIS_METHOD_PROMPT = """\ # 分析方法（平台内置，用分析工具时照这个做） -1. 先定现象：取本期和问题对应的基期，用 analysis_comparable 确认两期能直接比；历史够长（缺省至少 8 期）就用 analysis_baseline 放回历史看——是常规波动就照实说，不硬找原因。 -2. 指标是乘积或比率时，先用 analysis_decompose 的 factor、ratio 模式拆因子、拆结构。 -3. 按数据目录里的候选维度由粗到细拆（additive 模式），每次用 headline 给出总数核对拆开后是否闭合；找到主要来源再用 parent 下钻一层：下钻的 SQL 由你写（只查那一段），系统按上一层那一段的值核对闭合。 -4. 每次调用在 hypothesis 里写明这次在验证什么；拆了没发现集中也是结论，照实写。 -5. 不作因果断言：数据只说明变化来自哪里；原因要有知识库原话或输入作依据，否则写成待核实的推测。 -6. 阈值和方法由平台定，工具参数里没有阈值；算不了时照实说算不了的原因，不要换个说法硬下判断。 +1. 先按问题定基期：「这周怎么样」「环比」比上一个整期（上周、上月）；「同比」「和去年比」比去年同期（波动基线的 lag 周数据写 52、月数据写 12，历史要相应够长）；「和目标比」比库里的目标或预算（目标和实际查进同一份结果，用一列区分两者当期次，label 写清是和目标比；库里没有目标就照实说，不自己定目标）。问题没说时比上一个整期，并写明比的是哪一期。本期取最近一个已经过完的整期：这周没过完，就比上周和上上周。 +2. 期次在 SQL 里由日期列算出来：按周汇总写每周第一天的日期（SQLite 写 date(日期列, '-6 days', 'weekday 1')），或 ISO 周「YYYY-Www」——SQLite 的 %W、MySQL 的 %u 算的不是 ISO 周，同一个「2026-W42」落在别的日子上。直接取表里现成的周号文本时，系统认不出它的算法，本期齐不齐核对不了。 +3. 先定现象：用 analysis_comparable 确认两期能直接比；历史够长（缺省至少 8 期）就用 analysis_baseline 放回历史看，要看是不是持续在变用 analysis_trend。是常规波动就照实说，不硬找原因。 +4. 再拆：指标是乘积（销售额 = 订单数 × 客单价）先用 analysis_decompose 的 factor 拆因子，是比率（退款率、客单价）先用 ratio 拆结构和各段自身的变化；可加的量按数据目录里的候选维度由粗到细用 additive 拆，每次用 headline 给出总数，核对拆开后是否闭合。找到主要来源再用 parent 下钻一层：下钻的 SQL 由你写（只查那一段），系统按上一层那一段的值核对闭合。 +5. 每次调用在 hypothesis 里写明这次在验证什么。拆了没发现集中（分散、持平）也是结论，照实写，不换着维度硬凑「主要来自」。两组、两期的差异要说站不站得住时用 analysis_differ。 +6. 不作因果断言：数据只说明变化来自哪里，说明不了为什么。原因要有知识库原话或运行输入作依据，否则写成待核实的推测。 +7. 阈值和方法由平台定，工具参数里没有阈值。算不了（历史不够、本期不完整、截断、不可比）时照实说原因，不要换个说法硬下判断；调用写错时照交回的说明改了再调。 +8. 最后交给写作者的文字里列出做过的分析：「【分析 An】」的编号、分析的是什么、判定（判定语原话）或算不了的原因，没做的分析不要写成做过。 分析结论的编号「【分析 An】」只在本节点有效：下钻时 parent 写它（如 {"of": "A1", "key": ["华东"]}）。判定语由系统写好了，结论里照它的意思说，不要改写判断的强弱。""" -METHOD_GUIDE = ToolGuide(id="analysis.method@1", text=ANALYSIS_METHOD_PROMPT) +METHOD_GUIDE = ToolGuide(id="analysis.method@2", text=ANALYSIS_METHOD_PROMPT) #: 交回 agent 的说法：这个节点里没有可分析的查询容器（调用工具节点、工具库试用、没记台账的老运行） NO_SCOPE = ("分析工具只能由 Agent 在本节点查过数之后调用：它分析的是本节点交回过的查询结果（【证据 Qn】的编号）。" diff --git a/backend/tests/test_analysis_intent.py b/backend/tests/test_analysis_intent.py new file mode 100644 index 00000000..811ef234 --- /dev/null +++ b/backend/tests/test_analysis_intent.py @@ -0,0 +1,135 @@ +"""问题意图（app/analysis/intent.py）：一句问题是不是分析类问题，给助手搭问数据的图时自查用。 + +只收明确的问法、宁可漏判：漏判只是少一条自查提醒，误判要让助手多改一轮。所以反例和正例一样多——问一个字段（退款原因）、 +问业务状态（异常订单）、问服务（系统正常吗）、筛选（波动超过 10%）、问时长（持续几天）都不算。问法都是虚构的零售场景。 +""" +from __future__ import annotations + +import pytest + +from app.analysis.intent import INTENT_VERSION, KINDS, Intent, analysis_intent, is_analytical + +POSITIVE = [ + # 原因 + ("上周销售额为什么下降了", "why"), + ("华东的退款率为什么这么高", "why"), + ("为什么华东比华南的客单价低", "why"), + ("销售额下降了，为什么", "why"), + ("这周订单怎么会少了这么多", "why"), + ("销售额下滑的主要原因是什么", "why"), + ("什么原因导致线上订单减少", "why"), + ("本周销售额下滑了，原因是什么？", "why"), + # 来源 + ("这周和上周比，销售额怎么变了", "source"), + ("上周各渠道的销售额有什么变化", "source"), + ("哪个地区拖累了整体销售额", "source"), + ("谁贡献最大", "source"), + ("增长主要来自哪个地区", "source"), + ("各地区对销售额增长的贡献是多少", "source"), + ("销售额增长的驱动因素有哪些", "source"), + ("什么带动了订单增长", "source"), + ("把销售额的变化按地区拆开看看", "source"), + ("做个归因分析", "source"), + # 异常 + ("本周销售额是否异常", "anomaly"), + ("这周退款单数异常吗？", "anomaly"), + ("下降 12% 正常吗", "anomaly"), + ("这个降幅算不算异常", "anomaly"), + ("上周订单有没有异动", "anomaly"), + ("取消单是不是暴增了，有没有激增", "anomaly"), + ("这周的销售额是不是超出了正常范围", "anomaly"), + # 波动 + ("最近销售额波动大吗", "fluctuation"), + ("这是不是正常波动", "fluctuation"), + ("销售额同比变化大不大", "fluctuation"), + # 趋势 + ("线上销售额的趋势怎么样", "trend"), + ("看一下近半年订单数的走势", "trend"), + ("线上销售额是不是在持续上升", "trend"), + ("下降会持续吗", "trend"), + ("华南的销售额连续三周下降了吗", "trend"), + ("客单价是不是越来越低", "trend"), + ("退款率是不是一直在涨", "trend"), + # 差异 + ("华东的退款率是不是真的比华南高", "differ"), + ("线上和线下的转化率差异显著吗", "differ"), + ("两组的客单价有没有明显差别", "differ"), + ("这个差距是不是偶然", "differ"), + # 英文 + ("Why did sales drop last week?", "why"), + ("What drove the decline in online orders?", "source"), + ("Is this normal for a Monday?", "anomaly"), + ("Show me the trend of refunds", "trend"), + ("Is the refund rate in East significantly higher than South?", "differ"), +] + +NEGATIVE = [ + "上周销售额是多少", + "列出华东、华南、华北、西南四个地区的订单数", + "shop 里都有哪些表", + "退款原因有哪些", + "按取消原因统计上周的订单数", + "订单主要来自哪些渠道", + "有多少异常订单", + "有没有异常记录", + "列出状态为异常的订单", + "系统正常吗", + "数据源连接正常吗", + "门店是否正常营业", + "列出价格波动超过 10% 的商品", + "网络波动大吗", + "促销活动持续了几天", + "这个活动会持续多久", + "为什么报错了", + "为什么查不到数据", + "表结构有什么变化", + "口径有什么变化", + "按地区拆分订单", + "导出上周的销售明细", + "", + " ", + "How many orders did we get last week?", + "Why is the query slow?", +] + + +@pytest.mark.parametrize("question, kind", POSITIVE) +def test_analysis_questions_are_recognized(question, kind): + found = analysis_intent(question) + assert found is not None, question + assert found.kind == kind, (question, found) + assert found.text and found.text.rstrip("…") in question, found + assert is_analytical(question) + + +@pytest.mark.parametrize("question", NEGATIVE) +def test_other_questions_are_left_alone(question): + assert analysis_intent(question) is None, (question, analysis_intent(question)) + assert not is_analytical(question) + + +def test_the_lexicon_is_versioned_and_every_kind_has_a_label(): + assert INTENT_VERSION == 1 + assert {kind for _, kind in POSITIVE} == set(KINDS) + found = analysis_intent("上周销售额为什么下降了") + assert isinstance(found, Intent) and found.version == INTENT_VERSION and found.label == "原因" + + +def test_sentences_are_read_one_at_a_time(): + """问原因的词和变化要在同一句：前一句问报错、后一句问销售额，不拼在一起认；逗号不分句。""" + assert analysis_intent("为什么报错了？上周销售额是多少") is None + assert analysis_intent("上周销售额是多少？跟前一周比为什么少了").kind == "why" + # 只有那一句是在问服务的，别的句子照认 + assert analysis_intent("系统正常吗？另外这周销售额是否异常").kind == "anomaly" + + +def test_the_quoted_text_is_short(): + question = "为什么" + "华东" * 30 + "的销售额下降" + found = analysis_intent(question) + assert found is not None and len(found.text) <= 24 and found.text.endswith("…") + + +def test_the_intent_lexicon_is_not_the_claims_lexicon(): + """意图判用户的问题，说法核对判写作者的句子：「主要来自哪」是在问（意图认），「主要来自华东」是下判断（意图不认）。""" + assert analysis_intent("下降主要来自哪个地区").kind == "source" + assert analysis_intent("本周订单主要来自线上渠道") is None diff --git a/backend/tests/test_analysis_method_guide.py b/backend/tests/test_analysis_method_guide.py new file mode 100644 index 00000000..0c937ac8 --- /dev/null +++ b/backend/tests/test_analysis_method_guide.py @@ -0,0 +1,47 @@ +"""平台内置的分析方法（tools/builtin/analysis.py 的 ANALYSIS_METHOD_PROMPT，第 2 波定稿，guide id analysis.method@2）。 + +方法随代码升版、绑了任一个分析工具就注入 Agent 的 system（注入本身见 tests/test_analysis_tools.py）。这里守方法里该写清的几件事 +都在：按问题选基期、周序列的期次怎么写、先现象后拆解再下钻的顺序、写明在验证什么、拆了不集中也是结论、不作因果断言、阈值 +不由模型定、算不了照实说、最后列出做过的分析；方法里点名的工具都是真有的那五个。 +""" +from __future__ import annotations + +import re + +from app.analysis import contract +from app.tools.builtin.analysis import ANALYSIS_METHOD_PROMPT, METHOD_GUIDE +from app.tools.registry import all_specs + + +def test_the_guide_is_the_second_version_and_shared_by_all_five_tools(): + assert METHOD_GUIDE.id == "analysis.method@2" and METHOD_GUIDE.text == ANALYSIS_METHOD_PROMPT + specs = all_specs() + assert {name for name, spec in specs.items() if spec.guide is METHOD_GUIDE} == set(contract.TOOL_NAMES.values()) + + +def test_the_method_says_how_to_pick_the_base_period(): + first = ANALYSIS_METHOD_PROMPT.split("\n2.")[0] + for words in ("上周", "去年同期", "和目标比", "上一个整期", "lag", "52", "库里没有目标就照实说", "已经过完的整期"): + assert words in first, words + + +def test_the_method_says_how_to_write_weekly_periods(): + for words in ("每周第一天的日期", "ISO 周「YYYY-Www」", "SQLite 的 %W", "不是 ISO 周", "date(日期列, '-6 days', 'weekday 1')", + "核对不了"): + assert words in ANALYSIS_METHOD_PROMPT, words + + +def test_the_method_orders_the_steps_and_names_only_real_tools(): + text = ANALYSIS_METHOD_PROMPT + # 先可比性和基线，再拆因子与结构，再按维度由粗到细，找到主要来源再下钻一层 + order = [text.index(w) for w in ("analysis_comparable", "analysis_baseline", "factor", "ratio", "additive", "parent")] + assert order == sorted(order), order + named = set(re.findall(r"\banalysis_[a-z]+\b", text)) + assert named == set(contract.TOOL_NAMES.values()), named + assert "由粗到细" in text and "下钻一层" in text and "headline" in text + + +def test_the_method_keeps_the_round_rules(): + for words in ("hypothesis", "拆了没发现集中", "也是结论", "不作因果断言", "待核实的推测", "工具参数里没有阈值", + "算不了", "照实说原因", "【分析 An】", "判定语原话", "没做的分析不要写成做过"): + assert words in ANALYSIS_METHOD_PROMPT, words diff --git a/backend/tests/test_copilot_analysis.py b/backend/tests/test_copilot_analysis.py new file mode 100644 index 00000000..6f4aae53 --- /dev/null +++ b/backend/tests/test_copilot_analysis.py @@ -0,0 +1,360 @@ +"""助手搭图认分析能力（00-总方案 §3.7）：节点手册教分析卡和分析工具、报告的分析结论引用写法；问数据的包装多一种「分析类 +问题」的形状；自查时问题是分析类（app/analysis/intent）、查库的 Agent 却没绑分析工具，交回模型改一轮（只挡一轮，照 coverage +的样板），改过一轮仍在的不挡运行、留作提醒；报告的上游只有分析卡也算有可引用的来源。评测「分析工具开、关」对照的「关」那一臂 +（answer_ops 的 analysis_tools、chat/drive.TurnDriver 的同名属性）：工具清单、节点手册、问数据的包装里都没有分析，图里绑了 +也按「绑定的工具不存在」挡，别的和开着时一样。 + +手册里举的引用标记交给真正的解析器，确认写法和系统认的是同一套（同 tests/test_copilot_entity_guide.py）。问法、节点、 +表都是虚构的零售场景。 +""" +from __future__ import annotations + +import json +import re + +import pytest +from langchain_core.messages import AIMessageChunk + +from app.analysis import contract +from app.api.copilot import (_ASSIST_RULES, ANALYSIS_SHAPE_PROMPT, ANALYSIS_TOOLS_OFF, NODE_REFERENCE, GenerateIn, + _evidence_source, _question_of, _tool_catalog, _user_message, node_reference) +from app.api.copilot_gen import ANALYSIS_TOOLS_MISSING, GraphApplier, GraphChecker +from app.engine.evidence import _CELL_REF, _ROW_REF, SUPPORTED_KINDS, parse_markers +from app.engine.schema import ANALYSIS_CLAIMS_VALUES, ANALYSIS_SCOPE_VALUES, GraphNode, NodeType +from tests.test_copilot_answer_ops import _sse, _use, client # noqa: F401 + +TOOLS = list(contract.TOOL_NAMES.values()) + + +def _section(start: str, end: str) -> str: + i = NODE_REFERENCE.index(start) + return NODE_REFERENCE[i:NODE_REFERENCE.index(end, i)] + + +def _report() -> str: + return _section("- report：", "\n- output") + + +# -------------------------------------------------------------------------- +# 节点手册 +# -------------------------------------------------------------------------- + + +def test_the_card_entry_teaches_the_analysis_card(): + card = _section("- analysis：分析卡", "\n- dual_check") + for words in (["items", "overrides", "from", "params", "hypothesis"], # 配置 + ["模板里每次都要做的分析", "周报的异动"], # 用途 + ["nodes..results.<项 id>", "verdict", "branch"], # 产出、按判定分支 + ["tool 节点或 merge", "agent 不能当", "headline", "parent"], # 输入只认查询节点 + ["不要把结果交给口径卡", "追不到原始数据"]): # D16 + assert all(w in card for w in words), [w for w in words if w not in card] + assert all(op in card for op in contract.OPS) + assert all(mode in card for modes in contract.MODES.values() for mode in modes) + # 例子里的阈值键、判定码都是契约里的 + assert all(key in contract.DEFAULTS for key in re.findall(r"\b(k_warn|main_share|k_alert|flat_pct)\b", card)) + branch = re.search(r"verdict in \[([^\]]+)\]", card).group(1) + assert {v.strip(" '") for v in branch.split(",")} <= set(contract.VERDICTS["baseline"]) + + +def test_the_agent_entry_names_the_five_analysis_tools(): + agent = _section("- agent：", "\n- supervisor") + assert all(name in agent for name in TOOLS) + for words in ("分析方法", "注入", "不要写阈值", "只绑给同时绑了", "db_query__", "不能放进 tool 节点"): + assert words in agent, words + + +def test_the_markers_of_analysis_conclusions_parse_as_the_system_reads_them(): + markers = [m for m in parse_markers(_report()) if "A1" in m["body"]] + kinds = {m["kind"] for m in markers} + assert {contract.MARKER_KIND, "v", "table", "see"} <= kinds <= SUPPORTED_KINDS + by_kind = {m["kind"]: m for m in markers} + plain = [m for m in markers if m["kind"] == contract.MARKER_KIND and m["ref"] == "A1"] + named = [m for m in markers if m["kind"] == contract.MARKER_KIND and m["ref"] != "A1"] + assert plain and named and named[0]["ref"] == "A1.top_share" and named[0]["conv"] == "pct" + cell = _CELL_REF.match(by_kind["v"]["ref"]) + assert cell and cell.group("alias") == "A1" and cell.group("column") == "share" + assert _ROW_REF.match(by_kind["table"]["ref"]).group("alias") == "A1" + assert by_kind["see"]["refs"] == ["A1"] + # 拆解结果表的列 id（判定写法里举的 delta、share）是真结果表的列 + from app.analysis.decompose import _FIXED_ADDITIVE + + cols = by_kind["table"]["options"]["cols"].split(",") + assert set(cols[1:]) <= set(_FIXED_ADDITIVE), cols + + +def test_the_report_entry_spells_out_analysis_claims_and_scope(): + report = _report() + assert "analysis_claims" in report and "analysis_scope" in report + assert all(v in report for v in ANALYSIS_CLAIMS_VALUES) and all(v in report for v in ANALYSIS_SCOPE_VALUES) + assert "off < check < strict" in report and "只能往严里改" in report + for words in ("主要来自", "异常", "持续上升", "待核实的推测", "说明不了为什么"): + assert words in report, words + + +def test_governed_templates_and_the_publish_assist_keep_analysis_claims_strict(): + governed = NODE_REFERENCE[NODE_REFERENCE.index("要按受管级别发布的模板"):] + assert 'analysis_claims: "strict"' in governed[:400] + rules = " ".join(_ASSIST_RULES.split()) + assert re.search(r"analysis_claims.{0,40}只能往严里改|只能往严里改.{0,80}analysis_claims: strict", rules), rules + assert "off < check < strict" in rules and "不改回 check" in rules + + +def test_the_build_rules_bind_analysis_tools_for_analysis_questions(): + rules = NODE_REFERENCE[NODE_REFERENCE.index("搭图规则"):NODE_REFERENCE.index("模型字段")] + seventh = rules[rules.index("\n7."):] + assert "分析工具" in seventh and "分析卡" in seventh and "不进口径卡" in seventh + + +# -------------------------------------------------------------------------- +# 问数据的包装 +# -------------------------------------------------------------------------- + + +def test_the_answer_request_has_an_analysis_shape(): + text = _user_message(GenerateIn(instruction="上周销售额为什么下降", intent="answer"), patch=False) + shape = next(line for line in text.splitlines() if line.startswith("- 问的是为什么")) + assert "input → agent → report → output" in shape + assert all(name in shape for name in TOOLS) and "db_query__" in shape and "db_schema__" in shape + assert "按分析结论写" in shape and "判定语" in shape and "不写阈值" in shape + # 问数是探索运行：不替人打开结论句策略（test_copilot_entity_guide 守着），说法核对用默认、不关 + assert not re.search(r"(? dict: + return {"id": nid, "type": ntype, "position": {"x": 0, "y": 0}, "data": {"label": nid, "config": config}} + + +def _graph(*, tools=("db_query__shop", "db_schema__shop"), card=False): + nodes = {"start": _n("start", "input"), + "bot": _n("bot", "agent", prompt="{{ input.question }}", tools=list(tools)), + "write": _n("write", "report", instructions="回答：{{ input.question }}"), + "out": _n("out", "output", fields=[{"name": "回答", "value": "{{ nodes.write.text }}"}])} + if card: + nodes["check"] = _n("check", "analysis", items=[]) + order = ["start", "bot", "write", "out"] + return nodes, [{"source": a, "target": b} for a, b in zip(order, order[1:])] + + +async def test_an_analysis_question_without_analysis_tools_is_returned_once(): + nodes, edges = _graph() + checker = GraphChecker(GraphApplier(nodes, edges), question="上周销售额为什么下降") + [issue] = [i for i in await checker.issues() if i["code"] == ANALYSIS_TOOLS_MISSING] + assert (issue["level"], issue["node_id"], issue["field"]) == ("error", "bot", "tools") + assert issue["message"].startswith("问题问的是原因（「为什么下降」），查库的「bot」没有绑定分析工具") + assert all(name in issue["for_model"] for name in TOOLS) and "explanation" in issue["for_model"] + # 交回模型改过一轮：之后不再挡；收尾时作为提醒 + assert not [i for i in await checker.issues() if i["code"] == ANALYSIS_TOOLS_MISSING] + [warning] = checker.analysis_gaps(level="warning") + assert (warning["level"], warning["code"]) == ("warning", ANALYSIS_TOOLS_MISSING) + + +@pytest.mark.parametrize("question, graph", [ + ("上周销售额为什么下降", _graph(tools=("db_query__shop", "db_schema__shop", "analysis_decompose"))), # 绑了一个就算 + ("上周销售额为什么下降", _graph(card=True)), # 有分析卡 + ("上周销售额为什么下降", _graph(tools=("current_time",))), # 没有查库的 Agent + ("上周销售额是多少", _graph()), # 不是分析类问题 + (None, _graph()), # 画布：没有问题 +]) +async def test_the_rule_stays_quiet_otherwise(question, graph): + nodes, edges = graph + checker = GraphChecker(GraphApplier(nodes, edges), question=question) + assert not [i for i in await checker.issues() if i["code"] == ANALYSIS_TOOLS_MISSING] + assert checker.analysis_gaps(level="warning") == [] + + +def _ops(tools: list[str]) -> list[dict]: + nodes, edges = _graph(tools=tools) + return [{"op": "plan", "summary": "查销售额"}, + *({"op": "add_node", "node": {"id": n["id"], "type": n["type"], "label": n["id"], + "config": n["data"]["config"]}} for n in nodes.values()), + *({"op": "add_edge", "edge": e} for e in edges), {"op": "done", "explanation": "查一下", "run": True}] + + +def _scripted(first: list[dict], fix: list[dict], seen: list[str]): + class Scripted: + async def astream(self, messages): + human = next(text for role, text in messages if role == "human") + seen.append(human) + for op in (fix if "上线前自查" in human else first): + yield AIMessageChunk(content=json.dumps(op, ensure_ascii=False) + "\n") + + return Scripted() + + +async def test_the_stream_asks_for_one_repair_and_the_model_binds_the_tools(client, monkeypatch): # noqa: F811 + seen: list[str] = [] + fix = [{"op": "update_node", "id": "bot", "config": {"tools": ["db_query__shop", "db_schema__shop", *TOOLS]}}, + {"op": "done", "explanation": "绑上了分析工具"}] + _use(monkeypatch, _scripted(_ops(["db_query__shop", "db_schema__shop"]), fix, seen)) + got = await _sse(client, {"instruction": "上周销售额为什么下降", "intent": "answer"}) + [repairing] = [op for op in got if op["op"] == "check" and op["status"] == "repairing"] + assert ANALYSIS_TOOLS_MISSING in [i["code"] for i in repairing["issues"]] + assert any("上线前自查" in h and "analysis_decompose" in h for h in seen), "交回模型改一轮" + [final] = [op for op in got if op["op"] == "final"] + bot = next(n for n in final["graph"]["nodes"] if n["id"] == "bot") + assert set(TOOLS) <= set(bot["data"]["config"]["tools"]) + assert not [i for i in final["issues"] if i["code"] == ANALYSIS_TOOLS_MISSING] + + +async def test_a_model_that_keeps_its_graph_still_runs_with_a_warning(client, monkeypatch): # noqa: F811 + seen: list[str] = [] + keep = [{"op": "done", "explanation": "这个问题其实只要一个数，不用分析"}] + _use(monkeypatch, _scripted(_ops(["db_query__shop", "db_schema__shop"]), keep, seen)) + got = await _sse(client, {"instruction": "上周销售额为什么下降", "intent": "answer"}) + assert sum("上线前自查" in h for h in seen) == 1, "只挡一轮" + [final] = [op for op in got if op["op"] == "final"] + assert final["next"] == "run", "改过一轮仍在的不挡运行" + assert [i["level"] for i in final["issues"] if i["code"] == ANALYSIS_TOOLS_MISSING] == ["warning"] + + +async def test_a_retry_reads_the_original_question(client, monkeypatch): # noqa: F811 + """重来时 instruction 是带着原因的那段话（里面有「原因」两个字）：按原问题判，原问题不是分析类就不提。""" + seen: list[str] = [] + _use(monkeypatch, _scripted(_ops(["db_query__shop", "db_schema__shop"]), [], seen)) + retry = "这个工作流上一次运行没有成功：用满了步数。\n\n请针对这个原因改它，改完继续回答原来的问题：上周销售额多少" + got = await _sse(client, {"instruction": retry, "intent": "answer", "question": "上周销售额多少"}) + assert not [op for op in got if op["op"] == "check" and op["status"] == "repairing"] + [final] = [op for op in got if op["op"] == "final"] + assert not [i for i in final["issues"] if i["code"] == ANALYSIS_TOOLS_MISSING] + + +# -------------------------------------------------------------------------- +# 关掉分析工具（评测「分析工具开、关」对照的「关」那一臂：answer_ops 的 analysis_tools、TurnDriver 的同名属性） +# -------------------------------------------------------------------------- + + +def test_the_catalog_and_the_manual_leave_analysis_out_when_switched_off(): + assert all(f"- {name}（" in _tool_catalog([]) for name in TOOLS) + off = _tool_catalog([], analysis_tools=False) + assert not [line for line in off.splitlines() if line.startswith("- analysis_")] + assert off == "\n".join(line for line in _tool_catalog([]).splitlines() if not line.startswith("- analysis_")) + assert node_reference() is NODE_REFERENCE + manual = node_reference(analysis_tools=False) + assert not any(name in manual for name in TOOLS) + # 去掉的是 agent 段里「分析类问题绑分析工具」那一条、分析卡一整条、搭图规则第 7 条；报告段的引用写法、受管规则照留 + assert "- analysis：" not in manual and "\n7. " not in manual and "**问题问的是为什么" not in manual + # 别的段落一字不少：只去掉那三段 + for kept in (" **默认不要写 max_steps", "- dual_check：", "\n模型字段", "- report：", "[[a:A1]]", "analysis_claims"): + assert kept in manual, kept + request = GenerateIn(instruction="上周销售额为什么下降", intent="answer") + on, off_text = _user_message(request, patch=False), _user_message(request, patch=False, analysis_tools=False) + assert ANALYSIS_SHAPE_PROMPT in on and ANALYSIS_SHAPE_PROMPT not in off_text + assert on.replace(ANALYSIS_SHAPE_PROMPT, "") == off_text + + +def _recording(first: list[dict], fix: list[dict], seen: list[list[tuple[str, str]]]): + class Recording: + async def astream(self, messages): + seen.append(list(messages)) + human = next(text for role, text in messages if role == "human") + for op in (fix if "上线前自查" in human else first): + yield AIMessageChunk(content=json.dumps(op, ensure_ascii=False) + "\n") + + return Recording() + + +async def _ops_of(question: str, *, analysis_tools: bool) -> list[dict]: + from app.api.copilot import answer_ops + + return [op async for op in answer_ops(GenerateIn(instruction=question, intent="answer"), + analysis_tools=analysis_tools) + if op.get("op") not in ("heartbeat", "usage")] + + +async def test_switched_off_the_copilot_sees_no_analysis_and_bound_tools_are_blocked(monkeypatch): + seen: list[list[tuple[str, str]]] = [] + fix = [{"op": "update_node", "id": "bot", "config": {"tools": ["db_query__shop", "db_schema__shop"]}}, + {"op": "done", "explanation": "去掉了分析工具"}] + _use(monkeypatch, _recording(_ops(["db_query__shop", "db_schema__shop", "analysis_decompose"]), fix, seen)) + got = await _ops_of("上周销售额为什么下降", analysis_tools=False) + system = seen[0][0][1] + assert seen[0][0][0] == "system" and not any(name in system for name in TOOLS) + assert ANALYSIS_TOOLS_MISSING not in json.dumps(got, ensure_ascii=False), "问题意图那条自查不提" + [repairing] = [op for op in got if op["op"] == "check" and op["status"] == "repairing"] + [issue] = [i for i in repairing["issues"] if i["code"] == ANALYSIS_TOOLS_OFF] + assert issue["node_id"] == "bot" and issue["message"].startswith("绑定的工具不存在：「bot」绑定的 analysis_decompose") + [final] = [op for op in got if op["op"] == "final"] + assert final["next"] == "run" and not [i for i in final["issues"] if i["code"] == ANALYSIS_TOOLS_OFF] + + # 改不好：照实列出、不运行 + seen.clear() + _use(monkeypatch, _recording(_ops(["db_query__shop", "analysis_decompose"]), [{"op": "done", "explanation": "x"}], + seen)) + got = await _ops_of("上周销售额为什么下降", analysis_tools=False) + [final] = [op for op in got if op["op"] == "final"] + assert final["next"] == "fail" and [i["level"] for i in final["issues"] if i["code"] == ANALYSIS_TOOLS_OFF] == ["error"] + + +async def test_switched_on_bound_analysis_tools_are_fine(monkeypatch): + """缺省开着：绑了分析工具不挡，和以前一样直接运行。""" + seen: list[list[tuple[str, str]]] = [] + _use(monkeypatch, _recording(_ops(["db_query__shop", "db_schema__shop", *TOOLS]), [], seen)) + got = await _ops_of("上周销售额为什么下降", analysis_tools=True) + assert all(name in seen[0][0][1] for name in TOOLS) + assert not [op for op in got if op["op"] == "check" and op["status"] == "repairing"] + [final] = [op for op in got if op["op"] == "final"] + assert final["next"] == "run" + + +async def test_other_turns_go_the_same_either_way(monkeypatch): + """没碰分析工具的一轮（不是分析类问题、图里没绑分析工具）：开、关两臂交出的操作逐条一样，差的只是给助手看的那几段。""" + runs = {} + for flag in (True, False): + seen: list[list[tuple[str, str]]] = [] + _use(monkeypatch, _recording(_ops(["db_query__shop", "db_schema__shop"]), [], seen)) + runs[flag] = (await _ops_of("上周销售额是多少", analysis_tools=flag), seen[0]) + assert runs[True][0] == runs[False][0] + (_, system_on), (_, human_on) = runs[True][1] + (_, system_off), (_, human_off) = runs[False][1] + assert human_on.replace(ANALYSIS_SHAPE_PROMPT, "") == human_off and system_on != system_off + + +async def test_the_turn_driver_passes_the_switch_through(monkeypatch): + import app.api.copilot as copilot + from app.chat.drive import TurnDriver + + calls: list[dict] = [] + + async def fake_ops(payload, **kwargs): + calls.append({"question": payload.question, **kwargs}) + yield {"op": "reply", "text": "上一轮已经查过了。"} + + monkeypatch.setattr(copilot, "answer_ops", fake_ops) + assert TurnDriver("上周销售额为什么下降", datasource_ids=[]).analysis_tools is True + driver = TurnDriver("上周销售额为什么下降", datasource_ids=[], analysis_tools=False) + ops, refused = await driver._generate("上周销售额为什么下降", None) + assert refused is None and ops == [{"op": "reply", "text": "上一轮已经查过了。"}] + assert calls == [{"question": "上周销售额为什么下降", "catalog_view": None, "analysis_tools": False}] diff --git a/backend/tests/test_review.py b/backend/tests/test_review.py index fc1c8e20..301b291e 100644 --- a/backend/tests/test_review.py +++ b/backend/tests/test_review.py @@ -494,7 +494,8 @@ def test_analysis_signals_are_registered_as_gaps_that_only_need_a_note(): assert [(s.kind, s.severity) for s in signals] == [ ("report_analysis_claims", rv.DEGRADED), ("analysis_refused", rv.DEGRADED), ("metric_comparability_skipped", rv.DEGRADED)] - assert {s.kind for s in signals} == rv.NOTE_ONLY + # NOTE_ONLY 另有一种不是日志代码的：分析调用写错了一直没改对（analysis_call_failed，见 tests/test_review_analysis.py） + assert {s.kind for s in signals} == rv.NOTE_ONLY - {rv.ANALYSIS_CALL_FAILED} assert not rv.should_retry(signals) and not rv.NOTE_ONLY & (rv.RETRYABLE | rv.REBUILD) diff --git a/backend/tests/test_review_analysis.py b/backend/tests/test_review_analysis.py new file mode 100644 index 00000000..f5b9f3f7 --- /dev/null +++ b/backend/tests/test_review_analysis.py @@ -0,0 +1,144 @@ +"""复核认分析能力一轮的信号（engine/review.py，00-总方案 §3.7）：只写说明、不自动重来。 + +- 几次算不了的分析（analysis_refused）并成一条「有 n 次分析没做成」，点名是哪一项、为什么（叫法取契约的 OP_LABELS，不写工具名）； +- 分析调用写错了（tool.error 带 call_error）：后来同一个分析工具做成了是 Agent 改对了，不报；一直没做成的并成一条 + analysis_call_failed，同样是缺口、不重来——以前它落成 tool_error（严重、可重来），一次写错的分析会让整轮问数据重来； +- 判断词核对重写后仍有的（report_analysis_claims）照报告节点的原话说「报告有 n 处…重写 k 次仍未改掉」； +- 这几种信号都在 NOTE_ONLY 里：全是它们时复核不叫模型，照信号原话写说明。 + +事件是手写的，名字、数都是虚构的。 +""" +from __future__ import annotations + +import pytest + +from app.core.events import EventType +from app.engine import review as rv +from app.tools.analysis import CALL_ERROR_HEAD, REFUSED_CODE + +OK_OUTPUT = {"结果": "周报：销售额 45,678.5元。"} + + +def ev(etype, **data): + return {"type": str(etype), "node_id": data.pop("node_id", ""), "data": data} + + +def started(node_id, label): + return ev(EventType.NODE_STARTED, node_id=node_id, label=label) + + +def card_refused(item="wow", op="baseline", reason="历史只有 5 期，至少要 8 期"): + return ev(EventType.LOG, node_id="card", level="warn", code=REFUSED_CODE, item=item, op=op, reason_code="history_short", + reason=reason, message=f"「周报分析」的分析项「{item}」没有做成：{reason}") + + +def agent_refused(tool="analysis_decompose", reason="本期数据只到 2026-10-16，还差 2 天"): + return ev(EventType.LOG, node_id="bot", level="warn", code=REFUSED_CODE, call_id="c1", tool=tool, + reason_code="partial_target", reason=reason, message=f"{tool} 没有做成：{reason}") + + +def call_error(tool="analysis_decompose", message="列「区域」在 Q1 里没有：可用的列是 地区、周、销售额", node_id="bot"): + text = f"{CALL_ERROR_HEAD}{message}" + return ev(EventType.TOOL_ERROR, node_id=node_id, tool=tool, call_id="c2", error=text, call_error="unknown_column") + + +def tool_end(tool="analysis_decompose", node_id="bot"): + return ev(EventType.TOOL_END, node_id=node_id, tool=tool, call_id="c3") + + +TITLES = [started("card", "周报分析"), started("bot", "数据查询")] + + +def test_one_refused_analysis_names_the_item_and_the_reason(): + [signal] = rv.scan([*TITLES, agent_refused()], OK_OUTPUT) + assert (signal.kind, signal.severity) == (REFUSED_CODE, rv.DEGRADED) + # 叫法照契约写「变化拆解」，不写工具名 + assert signal.detail == "「数据查询」的变化拆解没有做成：本期数据只到 2026-10-16，还差 2 天" + assert signal.reason is None + [signal] = rv.scan([*TITLES, card_refused()], OK_OUTPUT) + assert signal.detail == "「周报分析」的分析项「wow」（波动基线）没有做成：历史只有 5 期，至少要 8 期" + + +def test_several_refused_analyses_become_one_count(): + rows = [*TITLES, card_refused(), agent_refused(), + ev(EventType.LOG, node_id="team", level="warn", code=REFUSED_CODE, agent="取数员", tool="analysis_trend", + reason="中间缺了 2026-W30", message="取数员 的 analysis_trend 没有做成：中间缺了 2026-W30"), + started("team", "分析团队"), + card_refused(item="where", op="decompose", reason="按地区只有 1 段，至少要 2 段才拆得开")] + [signal] = rv.scan(rows, OK_OUTPUT) + assert signal.kind == REFUSED_CODE + assert signal.detail.startswith("有 4 次分析没做成。「周报分析」的分析项「wow」（波动基线）：历史只有 5 期，至少要 8 期；" + "「数据查询」的变化拆解：本期数据只到 2026-10-16，还差 2 天；") + assert "「分析团队」的成员「取数员」的趋势判断：中间缺了 2026-W30" in signal.detail + assert signal.detail.endswith("；……") and len(signal.detail) <= 300 + + +def test_old_events_without_the_structured_fields_keep_their_wording(): + plain = ev(EventType.LOG, level="warn", code=REFUSED_CODE, message="「周报分析」的波动基线没有做成：历史只有 5 期") + assert [s.detail for s in rv.scan([plain], OK_OUTPUT)] == ["「周报分析」的波动基线没有做成：历史只有 5 期"] + two = rv.scan([plain, ev(EventType.LOG, level="warn", code=REFUSED_CODE, message="趋势判断没有做成：中间缺期")], OK_OUTPUT) + assert [s.detail for s in two] == ["有 2 次分析没做成。「周报分析」的波动基线没有做成：历史只有 5 期；趋势判断没有做成：中间缺期"] + + +def test_the_merged_signal_keeps_its_place_among_the_others(): + skipped = ev(EventType.LOG, level="warn", code="metric_comparability_skipped", message="指标「销售额环比」没做成可比性检查") + claims = ev(EventType.LOG, level="warn", code="report_analysis_claims", message="报告有 1 处分析说法未通过核对") + kinds = [s.kind for s in rv.scan([*TITLES, claims, agent_refused(), skipped, card_refused()], OK_OUTPUT)] + assert kinds == ["report_analysis_claims", REFUSED_CODE, "metric_comparability_skipped"] + kinds = [s.kind for s in rv.scan([*TITLES, call_error(), claims, agent_refused()], OK_OUTPUT)] + assert kinds == [rv.ANALYSIS_CALL_FAILED, "report_analysis_claims", REFUSED_CODE] + + +def test_a_call_error_the_agent_fixed_is_not_a_gap(): + """Agent 照交回的说明改了参数、同一个分析工具做成了：不报（以前记成 tool_args_fixed，为一次改对的调用叫一次复核模型）。""" + assert rv.scan([*TITLES, call_error(), tool_end()], OK_OUTPUT) == [] + + +def test_a_call_error_never_fixed_is_a_note_not_a_retry(): + """一直没改对：那次分析没做成，是缺口不是断裂；不落成 tool_error（严重、可重来）——重来一遍 Agent 照样可能不做这项分析。""" + [signal] = rv.scan([*TITLES, call_error()], OK_OUTPUT) + assert (signal.kind, signal.severity) == (rv.ANALYSIS_CALL_FAILED, rv.DEGRADED) + assert signal.detail == ("「数据查询」的变化拆解的调用写错了、一直没改对，没有做成：列「区域」在 Q1 里没有：可用的列是 地区、周、" + "销售额") + assert not rv.should_retry([signal]) and rv.ANALYSIS_CALL_FAILED in rv.NOTE_ONLY + exhausted = call_error(tool="analysis_baseline", message="这个 Agent 这次执行最多做 12 次分析，已经用完：用已有的分析结论写，不再新做分析") + [signal] = rv.scan([*TITLES, call_error(), exhausted], OK_OUTPUT) + assert signal.detail.startswith("有 2 次分析调用写错了、一直没改对，没有做成。「数据查询」的变化拆解：列「区域」") + assert "「数据查询」的波动基线：这个 Agent 这次执行最多做 12 次分析，已经用完" in signal.detail + + +def test_other_failures_of_an_analysis_tool_stay_tool_errors(): + """不是调用写错（没有 call_error：快照取不回来这类）的照旧是工具失败。""" + broken = ev(EventType.TOOL_ERROR, node_id="bot", tool="analysis_decompose", error="工具执行失败：Q1 的查询快照已不存在") + assert [(s.kind, s.severity) for s in rv.scan([broken], OK_OUTPUT)] == [("tool_error", rv.BROKEN)] + + +def test_the_claims_leftover_says_how_many_are_left_after_the_rewrite(): + message = ("报告有 2 处分析说法未通过核对（1 处判断没挂分析结论、1 处原因没有依据：「主要来自」「由于」），已要求模型重写 1 次仍未" + "改掉：判断要挂对应的分析结论并照判定写，原因要有知识库原话或运行输入作依据。报告照常产出，这几处已标注") + [signal] = rv.scan([ev(EventType.LOG, node_id="write", level="warn", code="report_analysis_claims", count=2, + mode="check", message=message)], OK_OUTPUT) + assert signal.kind == "report_analysis_claims" and "报告有 2 处分析说法未通过核对" in signal.detail + assert "重写 1 次仍未改掉" in signal.detail + + +@pytest.mark.asyncio +async def test_a_run_with_only_analysis_gaps_gets_a_note_without_the_model(): + class Never: + def with_structured_output(self, *_a, **_k): + raise AssertionError("不该叫模型") + + signals = rv.scan([*TITLES, agent_refused(), card_refused(), call_error()], OK_OUTPUT) + result = await rv.review(Never(), question="上周销售额为什么下降", answer="答案", signals=signals, locked=True) + assert result.verdict == "annotated" and result.retry is False and result.severity == rv.DEGRADED + assert result.note.startswith("有 2 次分析没做成。") and "「数据查询」的变化拆解的调用写错了" in result.note + + +def test_every_new_log_code_of_the_analysis_round_is_registered(): + """第 1 波各包新加的分析日志代码都登记了，严重程度是缺口，不在重来的两组里。""" + from app.engine.nodes.metrics import COMPARABILITY_SKIPPED + + for code in (REFUSED_CODE, COMPARABILITY_SKIPPED, "report_analysis_claims"): + assert rv.WARN_CODES[code] == (code, rv.DEGRADED), code + assert code in rv.NOTE_ONLY and code not in rv.RETRYABLE | rv.REBUILD + assert rv.ANALYSIS_CALL_FAILED not in rv.RETRYABLE | rv.REBUILD diff --git a/backend/tests/test_template_anomaly.py b/backend/tests/test_template_anomaly.py new file mode 100644 index 00000000..7877941e --- /dev/null +++ b/backend/tests/test_template_anomaly.py @@ -0,0 +1,156 @@ +"""内置模板「⑪ 异动分析（可追溯）」：取数（整体按周 26 周、按地区两周）→ 分析卡（波动基线 + 带总数的变化拆解）→ 报告撰写 +（分析说法严格核对）→ 出具（引用模式，单元格引用）。 + +模板既是示范也是活文档（同 tests/test_template_traceable.py 守模板 ⑨）：要通过画布校验、能直接按受管级别发布（发布检查 +没有错误），接上一份虚构的零售库 shop（orders(order_date, region, amount)：华东、华南、华北、西南四个地区，26 周每天一笔） +要能从头到尾运行、两项分析都做成、完整出具。期次是每周第一天的日期，分析卡认得出每一期覆盖哪几天，本期过没过完核对得了。 +""" +from __future__ import annotations + +import asyncio +import copy +import sqlite3 +from datetime import date, timedelta + +import pytest +from langchain_core.messages import AIMessage +from sqlalchemy import select + +from app.db.base import SessionLocal +from app.db.models import Run, RunEvent +from app.engine.governance import lint_for_publish, publish_issues +from app.engine.runner import run_manager +from app.engine.schema import GraphSpec, validate_graph +from app.seed import TEMPLATES + +NAME = "⑪ 异动分析（可追溯）" +REGIONS = {"华东": 400, "华南": 300, "华北": 200, "西南": 100} + + +def template() -> dict: + return next(t for t in TEMPLATES if t["name"] == NAME) + + +def graph() -> dict: + t = template() + return copy.deepcopy({"nodes": t["nodes"], "edges": t["edges"]}) + + +def test_the_template_is_listed_once_after_the_others(): + names = [t["name"] for t in TEMPLATES] + assert names.count(NAME) == 1 and names[-1] == NAME + assert any(n.startswith("⑨ 可追溯周报") for n in names) + + +def test_the_template_passes_validation(): + report = validate_graph(GraphSpec.model_validate(graph())) + assert report.ok, [i.message for i in report.issues if i.level == "error"] + types = {n["id"]: n["type"] for n in template()["nodes"]} + assert list(types.values()).count("analysis") == 1 and "metrics" not in types.values() + card = next(n for n in template()["nodes"] if n["type"] == "analysis")["data"]["config"] + assert [(i["id"], i["op"], i.get("mode")) for i in card["items"]] == [("wow", "baseline", None), + ("where", "decompose", "additive")] + assert card["items"][1]["params"]["headline"] == {"base": "weekly.r1.销售额", "target": "weekly.r0.销售额"} + assert "overrides" not in card, "阈值取设置，模板不写死" + + +def test_the_template_publishes_as_governed(): + """拿来就能按受管级别发布：发布检查（和真正发布同一套）没有错误，证据门禁 G1–G5 和分析卡的受管规则都过。""" + spec = GraphSpec.model_validate(graph()) + errors = [i for i in publish_issues(spec, level="governed") if i.level == "error"] + assert errors == [], [i.message for i in errors] + report = next(n for n in template()["nodes"] if n["type"] == "report")["data"]["config"] + assert {k: report[k] for k in ("numbers", "on_violation", "claims", "analysis_claims")} == { + "numbers": "strict", "on_violation": "fail", "claims": "require_citation", "analysis_claims": "strict"} + out = next(n for n in template()["nodes"] if n["type"] == "output")["data"]["config"] + assert out["contract"] == {"report_from": "write", "cells": True, "strict": True} + assert out["fields"] == [{"name": "异动分析", "value": "{{ nodes.write.text }}"}] + + +def test_template_nine_also_publishes_as_governed(): + """模板 ⑨ 在第 1 波补了分析说法核对设为严格：同样直接按受管级别发布，没有错误。""" + nine = next(t for t in TEMPLATES if t["name"].startswith("⑨")) + spec = GraphSpec.model_validate(copy.deepcopy({"nodes": nine["nodes"], "edges": nine["edges"]})) + errors = [i for i in publish_issues(spec, level="governed") if i.level == "error"] + assert errors == [], [i.message for i in errors] + assert not [i for i in lint_for_publish(spec, level="governed").issues if i.code == "governed.report_policy"] + + +# -------------------------------------------------------------------------- +# 接一份虚构的零售库从头到尾运行 +# -------------------------------------------------------------------------- + + +def _last_week() -> date: + """最近一个已经过完的整周的周一（运行当天往前，本周没过完）。""" + today = date.today() + return today - timedelta(days=today.weekday() + 7) + + +@pytest.fixture +async def shop(tmp_path, scratch_source): + """26 周、每个地区每天一笔：各周略有起伏；最后一周（最近一个过完的整周）华东掉了四成，别的地区照常。""" + monday = _last_week() + first = monday - timedelta(weeks=25) + path = tmp_path / "shop.db" + db = sqlite3.connect(path) + db.executescript("CREATE TABLE orders (id INTEGER PRIMARY KEY, order_date TEXT, region TEXT, amount REAL);") + rows, n = [], 0 + for week in range(26): + for day in range(7): + when = first + timedelta(weeks=week, days=day) + for k, (region, base) in enumerate(REGIONS.items()): + amount = base + (week * 7 + day * 3 + k * 5) % 11 + if week == 25 and region == "华东": + amount = round(amount * 0.6, 2) + n += 1 + rows.append((n, when.isoformat(), region, float(amount))) + db.executemany("INSERT INTO orders VALUES (?, ?, ?, ?)", rows) + db.commit() + db.close() + await scratch_source("shop", path) + return monday + + +@pytest.fixture +async def engine_up(): + await run_manager.setup() + yield + await run_manager.shutdown() + + +async def _wait(run_id: str) -> Run: + for _ in range(600): + await asyncio.sleep(0.05) + async with SessionLocal() as session: + row = await session.get(Run, run_id) + if row.status in ("succeeded", "failed") and row.manifest_seq is not None: + return row + raise AssertionError(f"run {run_id} 没有结束：{row.status}") + + +async def test_the_template_runs_end_to_end_and_issues_formally(engine_up, shop, monkeypatch): + from app.providers import mock_model + + text = ("## 本周异动\n\n本周销售额 [[v:Q1.r0.销售额]]，上周 [[v:Q1.r1.销售额]]。[[see:Q1]]\n\n" + "[[a:A1]]。[[see:A1]]\n\n[[a:A2]]。[[see:A2]]\n\n[[table:A2 cols=地区,base,target,delta]]") + monkeypatch.setattr(mock_model.MockChatModel, "_decide", lambda self, messages: AIMessage(content=text)) + monday = shop + run = await run_manager.start(graph=graph(), input_payload={ + "week": monday.isoformat(), "prev_week": (monday - timedelta(weeks=1)).isoformat()}) + row = await _wait(run.id) + assert row.status == "succeeded", row.error + async with SessionLocal() as session: + ends = [e.data for e in (await session.execute( + select(RunEvent).where(RunEvent.run_id == run.id, RunEvent.type == "analysis.end") + .order_by(RunEvent.seq))).scalars()] + by_item = {e["item"]: e for e in ends} + # 两项都做成了：本周的变化放回 25 周的历史里看（期次认得出、本期过完了），华东主导了这次下降（各段之和和总数闭合） + assert by_item["wow"]["status"] == "ok", by_item["wow"] + assert by_item["wow"]["verdict"] in ("outside_band", "far_outside_band"), by_item["wow"] + assert (by_item["where"]["status"], by_item["where"]["verdict"]) == ("ok", "driver"), by_item["where"] + assert "主要来自华东" in by_item["where"]["phrase"] + issuance = row.output["_issuance"] + assert issuance["mode"] == "citations" and issuance["tier"] == "formal", issuance + report = row.output["异动分析"] + assert by_item["where"]["phrase"] in report and by_item["wow"]["phrase"] in report diff --git a/docs/en/analysis.md b/docs/en/analysis.md new file mode 100644 index 00000000..f023a838 --- /dev/null +++ b/docs/en/analysis.md @@ -0,0 +1,204 @@ +English | [Simplified Chinese](../zh-CN/analysis.md) + +# Analysis + +This page covers where the analytical judgments in a report come from: how statements such as "mainly from East", "outside the usual fluctuation", "rising steadily", or "the difference holds up" are computed by the system, written into reports, and checked. It covers what each of the five analyses answers, the criteria it judges by, what it says when it can't be done, the two entry points (analysis tools for agents and analysis cards), how reports cite conclusions, the check on judgment words, the Analysis Scope section at the end of a report, and the settings and limits. It is for builders and for people who read reports. The user interface is currently available in Simplified Chinese only. + +## What it is and why + +The system fills the numbers in a report from snapshots, and the narrative layer does no arithmetic (see [Evidence and Issuance](evidence.md#the-basic-idea)). But a report says more than numbers; it also makes judgments: where a change mainly came from, whether this week counts as unusual, whether something keeps rising, whether a gap between two groups is real. These judgments used to be made by the writing model on its own; it could pick its evidence, and readers had no way to check. + +Analysis brings judgments back into the system as well: + +- **The system computes the verdicts**: the five analyses are deterministic platform code. They read only query snapshots returned during this run and compute verdicts with fixed methods and thresholds. The model decides only what to compute (which two periods to compare, which dimension to break down, which segment to drill into); it can't change the method or the thresholds. +- **The system writes the verdict phrases**: every analysis conclusion carries a verdict phrase written by the system from a template, and reports cite it (`[[a:A1]]`); like a number, it opens to its provenance. When the writer writes judgment words such as "mainly from" or "unusual" itself, the system checks sentence by sentence that the sentence cites a matching analysis conclusion and agrees with its verdict (see [Analysis claim check](#analysis-claim-check)). +- **Saying it as it is**: when an analysis can't be done, it says why (not enough history, the current period isn't over, the query was truncated, the two periods aren't comparable, too few samples) instead of forcing a conclusion. "Can't be done" is a conclusion too, and the writer reports it as such. +- **Leaving a trace**: every analysis conclusion, together with its method and version, parameters, the thresholds actually used, the input cells, the table ranges checked live, the results, and the verdict, is stored as a content-addressed artifact, recorded in the evidence ledger, and sealed with the run. The evidence panel, the report export, and the verifiable report all recompute it with the same function (see [What you see when you click a number](evidence.md#what-you-see-when-you-click-a-number)). + +A breakdown shows only where a change came from, not why. A cause needs a verbatim knowledge-base quote or a run input as its basis; otherwise it is written as a conjecture to be verified. + +## The five analyses + +Each analysis produces one analysis conclusion, numbered A1, A2, … in order in the writing catalog (the same numbering scheme as Q1 for queries and K1 for knowledge-base chunks). A conclusion has a verdict (verdict code, the segments it names, a direction), a verdict phrase, a criterion (one sentence stating the thresholds actually used), named values (shares, the total change, and so on), a result table, and caveats (offsetting, Simpson's paradox, small samples, a record high, and so on; at most two are appended to the verdict phrase). + +The thresholds in parentheses below are defaults, changed under Settings → Preferences → Analysis (see [Analysis settings](configuration.md#analysis-settings)). + +### Change decomposition + +Answers "where did the change come from". Three modes: + +| Mode | Used for | How it breaks down | +|---|---|---| +| Additive | Quantities that add up, such as sales or order counts | By dimension (one or two columns), each segment's contribution to the total change and its share | +| Ratio | Ratios such as refund rate or average order value | Into a mix effect (segment weights changed) and each segment's own change; the two add up exactly to the change in the ratio | +| Factor | Sales = orders × average order value | Into each factor's contribution, by logarithmic mean (LMDI) | + +- Verdicts: one segment accounting for at least half of the total change (0.5) is the main source ("mainly from East"); without such a segment, the top few segments together accounting for at least 80% (0.8) count as concentrated (at most a fifth of all segments, and at least 2); a total change under 0.5% of the base counts as essentially flat; anything else is dispersed. A ratio decomposition is judged as mainly from the mix, mainly from the segments' own changes, or both; a factor decomposition is judged by which factor it mainly comes from. +- When totals are given, the segments must add up to them (within a tolerance based on the precision the totals are written with) before the decomposition judges where the change came from. If they don't, it doesn't judge, and the verdict phrase states the gap. Without totals, the sum of the segments is the total, and the criterion says so. +- When the segments moving the other way add up to more than a quarter of the total change (0.25), a caveat says the changes partly offset each other. In a ratio decomposition, when most segments fall while the whole rises (or the other way round), a caveat explains Simpson's paradox. Segments with a denominator under 30 are marked as small and left out of that judgment. +- Drilling down: one segment of a decomposition can be broken down one level further (East by channel). The segments of that level must add up to the value of the segment above, at most 3 levels deep. The agent writes the drill-down query itself, querying only that segment, and the system checks that it closes against the value of the segment above. +- An example verdict phrase (translated): "Sales fell 194 (-9.2%), mainly from East (-120, 61.9% of the change)". + +### Fluctuation baseline + +Answers "is this change large, does it count as unusual": it puts the current period back into its own history. + +- The input is one series ordered by period (day, week, month, quarter). By default it compares the rate of change against the previous period; daily data is compared with the same day last week, to avoid weekday effects. Stocks and ratios can compare the value itself. For year over year, compare with 52 weeks or 12 months earlier, with correspondingly longer history. +- It uses the median and the median absolute deviation of the history (robust standard deviation = 1.4826 × median absolute deviation): a current period within 2 robust standard deviations of the median is a usual fluctuation, 2 to 3 is outside, and 3 or more is far outside. It uses the median rather than the mean because extremes such as promotion weeks widen the mean and standard deviation, making normal weeks and unusual weeks alike look ordinary. +- History needs at least 8 periods and uses at most 52. When the current period is the highest or lowest in its history, a caveat says it is a record high or low. +- An example verdict phrase (translated): "Sales changed -15.8% this period, far outside the usual fluctuation of the past 24 weeks (-0.5% to +0.4%); this period's sales are the lowest of the past 24 weeks". + +### Trend test + +Answers "is it changing steadily, or was it a one-off". + +- It takes the last contiguous stretch (at least 6 periods): when the two-sided p-value of the Mann–Kendall test is below 0.05, the sign of the Theil–Sen slope decides rising or falling; otherwise there is no clear trend. When the last 3 or more periods all move the same way, the verdict phrase adds "down 3 weeks in a row", or "but up the last 3 weeks in a row" when that runs against the trend. +- Mann–Kendall assumes the periods are independent, while business series are often autocorrelated, which makes p too small. So the verdict phrase says only "trending up", never "significantly up". Daily data gets a caveat about weekday effects suggesting weekly aggregation. +- An example verdict phrase (translated): "Online sales have trended up over the past 12 weeks (about +2.1% per week), up 4 weeks in a row". + +### Difference test + +Answers "are these two groups (or two periods of one group) really different, or is it sampling noise". + +- For rates (conversion rate, refund rate), give the successes and the trials: each group gets a Wilson interval; when every cell's expected frequency is at least 5, it uses the pooled-proportion z-test, otherwise Fisher's exact test. +- For means, give each group's mean, standard deviation, and sample size, or its sum, sum of squares, and sample size (SQLite has no standard deviation function, so write it this way), and it uses Welch's t-test. You can also give the values row by row (at most 1000 rows, not truncated); the system computes the statistics itself and also reruns the test without the most distant value as a sensitivity check. +- A two-sided p-value below 0.05 means the difference holds up; several pairs in one call are corrected with Holm's method. A caveat about small samples is added when a group has fewer than 30 samples for means, or when Fisher's test was used for rates. +- Verdict phrases avoid terms such as "p-value" or "significance": a difference that holds up is described as such, and otherwise the phrase says the difference may be just sampling noise, with the sample sizes (230 and 40, say). P-values and intervals are in the result table and the evidence panel. + +### Comparability check + +Answers "can these two periods be compared directly". Each item passes, warns, fails, or can't be checked: + +| Item | How it is judged | +|---|---| +| Same definition | The two queries differ only in their dates; when their structure differs, or they also filter something else (East versus South), it fails | +| Window length | Both periods have the same number of days; for an additive flow a mismatch fails, for ratios and stocks it only warns | +| Whole period | The current period is over and the table's data has arrived; for a flow, missing data fails ("the current period has only 3 days of data"), otherwise it warns | +| Weekday mix | For daily windows whose length isn't a multiple of 7, both periods have the same number of weekend days; otherwise it warns | +| Data problems | No data in the queried range fails; SQL check errors warn | + +When everything passes (or can't be checked), the periods are comparable; with only warnings they are conditionally comparable; with any failure they aren't comparable. Items that can't be checked never make the periods incomparable, and the verdict phrase names them. The table's actual date range is checked live (read-only, time-limited, masked columns are never checked, falling back to the range recorded by profiling on failure); the range found is recorded in the conclusion, and recomputation uses the recorded values. + +When a change decomposition is given two queries, it runs this check automatically first; when the fluctuation baseline, the trend test, or a change decomposition over a single query checks whether the current period is complete, it uses the whole-period item of this check. It also runs once automatically when a caliber card metric uses two queries of the same shape that differ only in their dates (this week versus last week), with the result attached to the metric (see [Caliber cards](evidence.md#caliber-cards)). + +## When an analysis can't be done + +An analysis that can't be done doesn't fail the agent or the node. It is recorded as a conclusion, with a verdict phrase along the lines of "Fluctuation baseline not done: only 5 periods of history, at least 8 needed", and it goes into the ledger and can be cited and checked: + +| Reason | When | +|---|---| +| Truncated | The input query returned only its first rows, so the sum of the segments isn't the full sum | +| Too few segments | Fewer than 2 segments after the breakdown | +| History too short | Fewer valid periods of history than the minimum | +| Current period incomplete | The current period isn't over, data hasn't arrived, it isn't a whole week, or the query covered only part of it | +| Gap | A period is missing between the current period and its comparison period; the last periods used by the trend test aren't contiguous | +| Periods unreadable | The period column's format can't be read (times of day, mixed formats) | +| Zero base | The comparison period is 0, so no rate of change can be computed | +| Non-positive values | Factor decomposition requires every value to be positive | +| Not the same quantity | The two queries differ in structure, so they aren't two periods of the same quantity, and it doesn't break them down | +| No p-value | The difference test can't compute a p-value for any pair (each group has only 1 sample, or a sample size is 0) | +| Upper level not done, or missing from this data | In an analysis card, the level a drill-down sits under couldn't be done, or this period's data lacks a period, segment, group, or total cell that is needed | + +A wrong call isn't recorded as a conclusion: a column that doesn't exist, a masked column used as a dimension or measure, a ratio treated as additive, several rows for the same segment and period, wrong parameters, or this agent having used up its analyses for this execution. The agent gets a sentence on how to fix the call and calls again; in an analysis card it is the template that is wrong, so the whole card fails and says which item. + +Periods are read in these formats: dates, `YYYY-MM`, `YYYY-Www`, `YYYY-Qn`, and integer sequence numbers. For weekly aggregation, it is best to compute the period from a date column in the SQL, as the date of the first day of each week, or as an ISO week `YYYY-Www`. The same `2026-W42` lands on different days under SQLite's `%W`, MySQL's `%u`, and the ISO week; the system recognizes the scheme from the format string in the SQL and works out the real start and end. When the period is a ready-made week-number text column taken straight from the table, the scheme can't be recognized, and whether the current period is complete can't be checked. + +## Two entry points + +Both entry points share the same analyses, the same shape of conclusion, and the same table of verdict phrases. + +### Analysis tools for agents + +Agents in Ask Data and on the canvas can be given five analysis tools: `analysis_decompose` (change decomposition), `analysis_baseline` (fluctuation baseline), `analysis_trend` (trend test), `analysis_differ` (difference test), and `analysis_comparable` (comparability check). Use them when you only know which dimension to break down after seeing the numbers. + +- The parameters cover only what to compute: which result this node queried (`Q3`, the same number as in the "Evidence Q3" handed back to the model), the role of each column, which two periods to compare, the two total cells, the level to drill down from, what to call the measure, and one sentence on what this call is testing (shown only in the evidence panel, never in the report body). There are no threshold parameters at all. +- Binding any one of them injects the platform's built-in analysis method into the agent's system prompt: first set the base period from the question, run the comparability check and the fluctuation baseline, then break down factors and mix, break down by dimension from coarse to fine, and drill down one level once the main source is found; a breakdown that finds no concentration is a conclusion too; make no causal claims; say plainly when something can't be done; finally, list the analyses done and their verdicts. The method is versioned with the code. +- What goes back to the agent is "Analysis A2" with the verdict, the verdict phrase, the key values, and caveats or the reason it couldn't be done; the full conclusion goes into the evidence ledger. +- Each agent can run at most 12 analyses per execution and drill down at most 3 levels. +- The analysis tools analyze only results that the agent they are bound to queried itself, and can't be put in a Call Tool node (for fixed analyses in a template, use an analysis card). Members of a collaboration team can use them too, but their conclusions don't go into the evidence ledger, so reports can't cite them. +- Users (the user role) can use them in Ask Data too: the analysis tools don't query the database; they read only query results already masked, and masked columns are never used as dimensions or measures. + +### Analysis cards + +An analysis card (the `analysis` node, a peer of the caliber card) runs a fixed set of analyses on the results of upstream query nodes. It is for analyses a template always does, such as the anomaly part of a weekly report: is this week's change within the usual fluctuation, and which region did it mainly come from. + +- Each item states the analysis method and mode, which upstream query node to analyze, the parameters, and what the item is testing. Inputs can only be Call Tool nodes that call a database query tool, or Merge Query nodes, connected before the analysis card; an agent can't be an input, since it may have queried several times. The parameters are the same as the analysis tools', and `{{ input.x }}` can pick periods from run inputs. +- Thresholds aren't written in the items. When a template needs fixed thresholds, write them in the card's Threshold Overrides, pinned with the version; when they conflict with this run's analysis settings, the card fails and says where the conflict is. +- The output is `nodes..results.`: status, verdict code, direction, named values, and verdict phrase. A downstream branch can route on the verdict, for example sending a run to a person when the change is outside the usual fluctuation. +- An item that can't be done doesn't fail the card: that item is recorded as not done, and the card moves on to the next one. +- Analysis card results are not caliber card inputs: a caliber card that reads them is treated as unable to trace back to raw data. Take metrics in the caliber card from query result cells, and cite analysis conclusions directly in the report. +- The governed level requires analysis card inputs to be Call Tool nodes that call a database query tool, or Merge Query nodes, so that it is settled at publish time that the card analyzes queries the platform executed. +- Run comparison aligns analysis cards by node and item and lists both runs' verdicts (see [Run comparison](evidence.md#run-comparison)). + +The built-in template 11, Anomaly Analysis (Traceable), is built this way: fetch the weekly totals and two weeks by region → analysis card (fluctuation baseline, change decomposition with totals) → report writer → issuance. It can be published at the governed level as it is (see [Built-in templates](workflows.md#built-in-templates)). + +## Citing in reports + +``` +[[a:A1]] A1's verdict phrase, rendered by the system from the verdict; one clickable segment +[[a:A1.top_share|pct]] one named value of A1; unit conversion as for [[m:]] +[[v:A1.r0.share]] one cell of A1's result table +[[table:A1 cols=region,delta,share rows=0-4]] A1's result table +[[see:A1]] basis of the sentence; attached at the end of the sentence, not rendered +``` + +- The Analysis Conclusions section of the writing catalog lists each conclusion's number, verdict phrase, named values, the top 10 segments of its result table, and caveats or the reason it couldn't be done. +- Conclusions from analysis cards can always be cited. In formal issuance at the governed level, analyses an agent ran on the spot with analysis tools follow the same rule as the cells it queried: they can be cited only when the issuance contract turns on Cell Citations (they can still serve as the basis at the end of a sentence). +- The measure name in a verdict phrase comes from the agent or the template; the system first strips digits, percent signs, units, and citation markers from it and cuts it to 20 characters, so text written by the system never smuggles in numbers written by the model. + +## Analysis claim check + +The report-writer node's Analysis Claim Check (`analysis_claims`) checks judgment words sentence by sentence, with a fixed lexicon and binding rules, without calling a model. + +| Class of judgment word | Examples (translated) | Analysis conclusion and verdict it needs | +|---|---|---| +| Source | mainly from, concentrated in, dragged down, drove, contributed most | Change decomposition judged a main source or concentrated; the segments named must be in the verdict, and a segment said to drag down must be falling | +| Dispersed, offset, mix | spread across regions, one up and one down, structural change | Change decomposition judged dispersed, offsetting, or mainly from the mix | +| Unusual, record, normal | unusual, sharply, surged, record high, normal fluctuation | Fluctuation baseline judged outside or far outside (direction matching), record high or low, or usual fluctuation | +| Trend, no trend | rising steadily, down three weeks in a row, no clear trend | Trend test, with matching direction and run length | +| Difference, no difference | significantly higher, really lower, little difference | Difference test judged as holding up (direction matching) or possibly sampling noise | +| Causal | because, due to, led to, affected by | A verbatim knowledge-base quote or a run input as the basis in the same sentence, or phrasing as a conjecture (possibly, to be verified) | + +- A sentence's basis is the `[[a:…]]`, `[[v:A…]]`, and `[[table:A…]]` in it plus the `[[see:A…]]` at its end. Negation flips the class: "not unusual" is checked as normal. +- Not checked: statements about composition rather than change ("this week's orders came mainly from the online channel"), suggestions and plans, "abnormal" as a business status ("some orders are abnormal"), verbatim knowledge-base quotes, system-rendered verdict phrases, headings, and tables. +- The line between source and cause: "the decline was mainly caused by East", citing a breakdown by region and naming one of its segments, is checked as a source, the bookkeeping sense of "came from"; "the decline was due to the end of a promotion" names no segment and is checked as causation. +- Three kinds of violation: a judgment word without a matching analysis conclusion; a conclusion cited, but the verdict, the segments named, or the direction don't match (citing a conclusion that couldn't be done counts too); a cause without a basis. + +| Value | What it does | +|---|---| +| `check` (default, also used by Ask Data) | Sends the report back to the writer for one rewrite; what remains is marked in the report and doesn't affect the issuance tier | +| `strict` | Rewrites up to the maximum number of repairs; what remains counts as an issuance gap and downgrades the issuance; required at the governed level (gate G3) | +| `off` | No check | + +- When the writing catalog has analysis conclusions, the writer is told to follow the verdicts and preferably cite verdict phrases directly; when it has none, the writer is told not to make such judgments and to state only numbers and changes. +- An exit in citation mode checks again independently, with the same check used at assembly. The report document records which disposition and which version of the lexicon it was checked with: only documents that carry this record are checked for it on verification, so verification results for reports from before the upgrade don't change. +- It errs on the side of missing cases: only strong judgment words are collected. The finer points it misses are left to the claim-sentence judge (see [Claim sentences](evidence.md#claim-sentences)); when the judge's excerpt includes an analysis conclusion, the judge is also told that the verdict was computed by the system and to judge only whether the sentence states it correctly. + +## Analysis scope + +When a report uses analysis conclusions (when this run has any), the system appends an Analysis Scope section at the end of the report. The writer can't change it; it is sealed with the document, and its numbers are clickable: + +- **Phenomena**: verdict phrases of comparability checks, fluctuation baselines, and trend tests. +- **Broken down**: each change decomposition, listed along its drill-down chain with its dimensions and verdict. +- **Ruled out**: decompositions judged dispersed, and difference tests where the difference may be just sampling noise. A breakdown that finds no concentration is a conclusion too. +- **Not done**: the reason for each analysis that couldn't be done. +- **Not broken down**: candidate dimensions that weren't broken down. Candidate dimensions are computed deterministically from the data catalog: columns on the fact table used by the decomposition's query that are marked as an attribute or status, aren't masked, and have a code-value table (at most 50 values, or marked complete), plus the same kind of columns on dimension tables related to it. When the columns used by the decomposition don't match the catalog, it says only that candidate dimensions can't be checked, rather than listing guesses. +- **Tests**: how many difference tests were run this time, and whether they were corrected for the number of tests. + +The sentence an agent writes about what a call is testing doesn't go into this section; it appears only in the evidence panel. The report-writer node's Analysis Scope (`analysis_scope`) defaults to appending the section when there are analysis conclusions, and can be set not to. + +## Settings + +Under Settings → Preferences → Analysis you can change the thresholds used for verdicts, how many analyses an agent may run per execution, how many levels it may drill down, and how many segments are listed. For each item's label and default, see [Analysis settings](configuration.md#analysis-settings). + +- The group is snapshotted into the run when the run starts (only when the graph has an analysis card or an agent with analysis tools), and resuming or continuing uses the same snapshot. +- Agents can't change thresholds: the analysis tools have no threshold parameters. A template can write threshold overrides on an analysis card. +- Every conclusion records the thresholds actually used, and its criterion is written from them, for example "a segment accounting for at least half of the total change is the main source". + +## Limits + +- Not in this round: forecasting; correlation analysis; dimension hierarchies in the data catalog; profiling of numeric columns; analysis tools for the verification agent of double-blind recomputation, adversarial review, and approval inquiry; analysis conclusions in caliber ledger matching and recalls (numbers in analysis conclusions get no caliber label; see [Caliber ledger](evaluation-and-ledger.md#caliber-ledger)). +- The trend test assumes independent periods; for autocorrelated series, p comes out too small. +- Candidate dimensions come only from code-value tables: profiling doesn't yet record each column's number of distinct values, so attribute columns without a code-value table aren't candidates. +- The lexicon collects only strong judgment words, so a different wording may slip through; what slips through is left to the claim-sentence judge and to the people reading the report. +- Analyses done by members of a collaboration team don't go into the evidence ledger, so reports can't cite them. diff --git a/docs/en/ask-data.md b/docs/en/ask-data.md index 22076108..73cefbb0 100644 --- a/docs/en/ask-data.md +++ b/docs/en/ask-data.md @@ -25,6 +25,17 @@ A turn covers: Copilot builds the graph, the run starts, the result is reviewed, - When Copilot returns only thinking and no operations, the turn is retried once automatically. - The cost of each turn includes the model calls for Copilot's graph building, table selection, and review. +## Analytical questions + +When the question asks why, how something changed, what dragged it down, what contributed most, whether something is unusual, how much it fluctuates, whether it keeps rising, or whether one is really higher than the other, the agent that queries data in the graph Copilot builds is also given the five analysis tools (see [Analysis](analysis.md)): + +- The agent first settles which two periods to compare from the question (the previous whole period when the question doesn't say), then follows the platform's built-in analysis method: comparability check, putting the period back into its history, breaking down by dimension, and drilling down one level once the main source is found. The platform computes the verdicts with fixed methods and thresholds; the agent decides only what to compute. +- The report is written from the analysis conclusions, citing the system-written verdict phrases directly for its judgments. When the writer writes judgment words such as "mainly from" or "unusual" itself, the system checks each sentence: sentences that don't cite a matching analysis conclusion go back for one rewrite, and what remains is marked in the answer without affecting the issuance tier (see [Analysis claim check](analysis.md#analysis-claim-check)). +- At the end of an answer that uses analysis conclusions, the system appends an Analysis Scope section: what was broken down, what was ruled out, which candidate dimensions weren't broken down, and what couldn't be done (see [Analysis scope](analysis.md#analysis-scope)). +- A breakdown shows only where a change came from, not why: causes in an answer need a verbatim knowledge-base quote as their basis, or else they are written as conjectures to be verified. +- During its self-check, Copilot recognizes analytical questions with a fixed lexicon that collects only explicit phrasings. When the question is analytical but the agent that queries data has no analysis tools, the problem goes back to Copilot for one round of fixes; if the tools are still missing after that round, the graph runs as usual with a warning. +- When an analysis couldn't be done (not enough history, the current period isn't over, and so on), or judgment words still don't cite a matching conclusion after the rewrite, the review only writes a note and doesn't start over automatically (see [Output review](#output-review)). + ## Output review After a run ends, rules scan the event stream first, and a model review runs only when they find an anomaly. The review may restructure the answer, but it **must not introduce numbers that aren't in the original answer**. @@ -32,10 +43,11 @@ After a run ends, rules scan the event stream first, and a model review runs onl - Ask Data for users (the user role) is reviewed too. - When a step fails inside a subworkflow, automatic review reports it only once. - Continue Run in Ask Data is limited to builders and above. +- The gaps from analysis (some analyses couldn't be done, an analysis call was wrong and never fixed, judgment words still failed the check after the rewrite, the two periods of a caliber card metric couldn't be checked for comparability) only get a note and never trigger an automatic redo. When a run has only these, the review calls no model and writes the note from their wording as is. ## Calibers in an answer -A caliber is the agreed definition of how a metric is fetched and computed. It lives in a caliber card, with a name and a version. After an answer is sealed, the system compares every number in it with the published calibers in the background and labels each number with a caliber; each turn gets a one-line summary. When the whole answer is equivalent to a governed caliber, the system automatically starts a formal run by that caliber. See [Evaluation and the Caliber Ledger](evaluation-and-ledger.md#caliber-ledger). +A caliber is the agreed definition of how a metric is fetched and computed. It lives in a caliber card, with a name and a version. After an answer is sealed, the system compares every number in it with the published calibers in the background and labels each number with a caliber; each turn gets a one-line summary. When the whole answer is equivalent to a governed caliber, the system automatically starts a formal run by that caliber. Numbers in analysis conclusions (shares, changes, fluctuation ranges) are derived from query results, so they aren't compared with calibers, get no caliber label, and aren't counted in the summary line. See [Evaluation and the Caliber Ledger](evaluation-and-ledger.md#caliber-ledger). ## From an answer to a workflow or an app @@ -60,6 +72,7 @@ Copilot generates or rewrites a whole graph from one sentence and lays it out au - When it edits a graph, it outputs only the change operations instead of rewriting the whole graph. - Before delivering, it checks its work with the same rules the runtime uses. Problems that would make the run fail go back to it to fix first; if they can't be fixed, nothing runs automatically. +- In Ask Data, the self-check also looks at the question itself: when the question is analytical but the agent that queries data has no analysis tools, it gets one round of reminders (see [Analytical questions](#analytical-questions)). - While building graphs, it records facts worth remembering in long-term memory. ### Looking on demand diff --git a/docs/en/configuration.md b/docs/en/configuration.md index c396fb31..188f3838 100644 --- a/docs/en/configuration.md +++ b/docs/en/configuration.md @@ -67,7 +67,7 @@ For the app database, checkpoint database, connection pool, PgBouncer, and multi | Location | Contents | |---|---| | Settings → Model Access | Providers and keys | -| Settings → Preferences | Run defaults (default memory scope, default knowledge base, require approval for dangerous tools by default); the evidence judge (judge model and limits); verification (verification model, the switches and limits for double-blind recomputation, adversarial review, and approval inquiry, and the daily spending cap; see [The review suite](evidence.md#the-review-suite)); evaluation concurrency, per-question time limit, and per-evaluation and daily spending; run concurrency (admins); the caliber ledger (automatic issuance by caliber and its daily cap, the caliber matching time limit, the recall limit `ledger.recall_limit`; see [Caliber ledger](evaluation-and-ledger.md#caliber-ledger)); drafting model; apps (the daily cap on on-demand runs per app, letting users create personal apps, allowing share links); interface theme | +| Settings → Preferences | Run defaults (default memory scope, default knowledge base, require approval for dangerous tools by default); the evidence judge (judge model and limits); verification (verification model, the switches and limits for double-blind recomputation, adversarial review, and approval inquiry, and the daily spending cap; see [The review suite](evidence.md#the-review-suite)); evaluation concurrency, per-question time limit, and per-evaluation and daily spending; analysis (the thresholds used for verdicts, how many analyses an agent may run per execution, the drill-down depth, how many segments are listed; see [Analysis settings](#analysis-settings)); run concurrency (admins); the caliber ledger (automatic issuance by caliber and its daily cap, the caliber matching time limit, the recall limit `ledger.recall_limit`; see [Caliber ledger](evaluation-and-ledger.md#caliber-ledger)); drafting model; apps (the daily cap on on-demand runs per app, letting users create personal apps, allowing share links); interface theme | | Settings → General | Public URL | | Settings → Runtime Environment | Sandbox candidates and their availability, the current choice and its reason; current values such as time limits | | Settings → Identity | Login mode; enabling Ask Data for users | @@ -77,6 +77,30 @@ For the app database, checkpoint database, connection pool, PgBouncer, and multi Other settings: `notify.digest_at` (the time of the daily task digest), and the agent token budget, spending budget, and step count. +### Analysis settings + +The Analysis section under Settings → Preferences governs how change decomposition, the fluctuation baseline, the trend test, and the difference test judge, and how many analyses an agent may run (see [Analysis](analysis.md)). It is snapshotted into a run when the run starts, and resuming or continuing uses the same snapshot; an analysis card can carry threshold overrides, and agents can't change any of it. + +| Setting | Default | Range | What it does | +|---|---|---|---| +| Main-source share | 0.5 | 0.3–0.9 | A segment (a factor, an effect) accounting for this share of the total change is the main source | +| Concentration share | 0.8 | 0.5–0.99 | The top few segments together accounting for this share count as concentrated; can't be below the main-source share | +| Offset ratio | 0.25 | 0.05–1 | When the segments moving the other way add up to more than this share of the total change, a caveat says they offset each other | +| Flat threshold | 0.005 | 0–0.1 | A total change below this share of the base counts as essentially flat | +| Outside-fluctuation multiple | 2 | 1–5 | Fluctuation baseline: this many robust standard deviations from the historical median counts as outside the usual fluctuation | +| Far-outside multiple | 3 | 1.5–8 | This many counts as far outside; must be greater than the outside-fluctuation multiple | +| Minimum history periods | 8 | 4–104 | With less history than this, the fluctuation baseline can't be done | +| Maximum history periods | 52 | 8–260 | The fluctuation baseline and the trend test use at most this many periods; can't be below the minimum | +| Minimum trend periods | 6 | 4–104 | The trend test needs at least this many periods | +| Trend significance level | 0.05 | 0.001–0.2 | Rising or falling is judged only when p is below this | +| Difference-test significance level | 0.05 | 0.001–0.2 | A corrected p below this means the difference holds up | +| Minimum sample size for means | 30 | 2–10000 | When a group has fewer samples in a comparison of means, a caveat says the sample is small | +| Minimum expected frequency for rates | 5 | 1–50 | When a cell's expected frequency is below this, Fisher's exact test is used | +| Minimum denominator for ratio breakdowns | 30 | 1–100000 | Segments with a smaller denominator are marked small and left out of the Simpson's paradox judgment | +| Maximum analyses per agent execution | 12 | 1–50 | Once used up, further analysis tool calls from the agent are told the analyses are used up | +| Maximum drill-down depth | 3 | 1–6 | How many levels a change decomposition can drill down | +| Segments listed | 10 | 3–30 | How many segments of a result table the writing catalog and the reply to the agent list | + ## Sandbox backend Settings → Runtime Environment lists every candidate and its availability, and marks the current choice and its selection reason. Automatic selection works in this order: diff --git a/docs/en/evaluation-and-ledger.md b/docs/en/evaluation-and-ledger.md index 86cda868..0591da55 100644 --- a/docs/en/evaluation-and-ledger.md +++ b/docs/en/evaluation-and-ledger.md @@ -16,13 +16,18 @@ Each question in a test suite is a question plus an expected answer. The expecta - A metric of a published caliber. - A query identified from an Ask Data run. - "No data", or a clarifying question back. +- The main source of a change (contribution): an expected SQL statement plus the parameters of one change decomposition. The mode is additive, ratio, or factor, the parameter names are the same as the analysis tools', and you also set how many top segments to name (1–10, default 1); the expectation is the computed main source and direction. +- A verdict: an expected SQL statement plus parameters for a fluctuation baseline, a trend test, a difference test (with its mode), or a comparability check; the expectation is the computed class of verdict: usual fluctuation or outside it; rising, falling, or no clear trend; a difference that holds up or not enough to conclude; comparable or not comparable (see [The five analyses](analysis.md#the-five-analyses)). Rules: -- Numeric values may not be typed in by hand. +- Numeric values may not be typed in by hand. Nor may the segments, directions, and verdicts in contribution and verdict expectations: they are computed on the snapshot of the expected SQL with the same analyses the answer uses, once before and once after the answer. +- The parameters of contribution and verdict expectations include no input (the input is the expected SQL's result), no totals, no drill-down, and no thresholds; text parameters may use the schedule variables `{{ schedule.* }}`. When the expected SQL returns only two periods, the base and current periods can be left out: the earlier one is the base. Caliber questions and questions that expect a clarifying question back can't have these two kinds. +- Thresholds come from the analysis settings snapshotted when the evaluation starts: computing expectations and every arm's answers use the same snapshot, which is recorded in the evaluation's fingerprint ("analysis settings"). An expectation that can't be computed (not enough history, an incomplete current period, and so on) makes that item ungradable, with the reason stated. - Text key points are judged by the judge, listed separately, and not counted toward accuracy. - After a caliber version bump, questions that use it show a notice at the top. Each question takes one of three options: switch to pinning the new version, compute one expectation for each of the old and new versions, or archive the old question and create a new one. Until then, the question is skipped in evaluations. - Test suites can be imported and exported. +- The repository includes a fictional retail database and a test suite for analytical questions: in the `backend` directory, run `python -m tests.fixtures.acceptance.retail_trends [--today YYYY-MM-DD]` to generate 26 weeks of orders up to yesterday (with a fixed random seed), register it as a read-only data source named `retail_trends`, then import `tests/fixtures/eval/analysis_suite.json` (eight questions, one for each planted effect). ### Run an evaluation @@ -42,7 +47,16 @@ Rules: - **Comparison**: against the previous evaluation or a baseline, with per-question transitions and attribution. - **A/B paired table**: exact McNemar test; display only, not a gate. -Evaluation runs are left out of usage statistics. The scoring rules are at version 2; old evaluations can be re-scored under the new rules from the evaluation details (reading only the sealed records, without calling models). +Contribution and verdict expectations are scored like this: + +- **Correct and traceable**: the answer cites an analysis conclusion whose verdict matches (the same analysis; for a main source, it names the expected segments with the same direction). If the sentence citing it says the opposite, it is wrong. +- **Correct but uncited**: no matching conclusion is cited, but the body states it correctly by the lexicon (the same judgment words and binding rules; see [Analysis claim check](analysis.md#analysis-claim-check)). Comparability has no judgment words, so only a citation counts. +- **Wrong**: the cited conclusion's verdict doesn't match, the body says the opposite (calling a usual fluctuation unusual, rising falling, or naming another segment as the main source), the analysis was done but not stated, or it wasn't done at all. +- **Ungradable**: the expectation can't be computed. + +The system-written Analysis Scope doesn't count as a citation by the answer. When an answer cites analysis conclusions, the per-question E1 compares their input queries. + +Evaluation runs are left out of usage statistics. The scoring rules are at version 3 (which adds contribution and verdict expectations); old evaluations can be re-scored under the new rules from the evaluation details (reading only the sealed records, without calling models). Copilot's prompt fingerprint includes the analysis method and the writer's analysis rules, so a comparison with the previous evaluation flags "Prompt changed" when they change. ### Scheduled evaluations @@ -68,6 +82,7 @@ After an exploratory run is sealed, the system compares the answer with publishe - Judging is number by number: when a report cites several columns of one query (for example sales, order count, first and last day), each number is compared on its own column against the caliber's metric column, and extra queried columns don't affect it. This applies only when removing the other columns wouldn't change the number; queries with DISTINCT, truncation after sorting by another column, or extra grouping columns are still compared as a whole. - When a multi-row query has no `ORDER BY`, citing one of its cells doesn't count as "from a caliber", and recall and reconciliation mark it "Row order undefined". - An answer is issued automatically only when every number in it that comes from data is proven to come from the same caliber. +- Numbers in analysis conclusions (shares, changes, fluctuation ranges) are derived from query results, so they aren't compared or labeled with a caliber and aren't counted in the summary line. When writing a caliber's data-fetch SQL, it is best not to round (`ROUND(SUM(x), 2)`) or wrap values in `COALESCE(…, 0)`. Leave rounding and defaults to the caliber card's decimal places and display settings; otherwise Ask Data queries written without rounding will never be recognized as equivalent (and where they differ is stated plainly). diff --git a/docs/en/evidence.md b/docs/en/evidence.md index 27e0bf98..3e437a2b 100644 --- a/docs/en/evidence.md +++ b/docs/en/evidence.md @@ -2,7 +2,7 @@ English | [Simplified Chinese](../zh-CN/evidence.md) # Evidence and Issuance -This page covers where the numbers in a report come from and how to check them: caliber cards and issuance contracts, the three issuance tiers, citation markers in the report-writer node, what happens when the queried range has no data, what you see when you click a number, sealing and verification, the two kinds of export, and the review tools: double-blind recomputation, adversarial review, approval inquiry, the what-if sandbox, and run comparison. It is for builders, approvers, and people who read reports. The user interface is currently available in Simplified Chinese only. +This page covers where the numbers in a report come from and how to check them: caliber cards and issuance contracts, the three issuance tiers, citation markers in the report-writer node (including analysis conclusions), what happens when the queried range has no data, what you see when you click a number or an analysis conclusion, sealing and verification, the two kinds of export, and the review tools: double-blind recomputation, adversarial review, approval inquiry, the what-if sandbox, run comparison, and the analysis claim check. It is for builders, approvers, and people who read reports. For how analytical judgments themselves are computed, see [Analysis](analysis.md). The user interface is currently available in Simplified Chinese only. ## The basic idea @@ -45,6 +45,9 @@ Measured: every number in body text written by Opus 5 traced back successfully, When the contract sets `report_from`, the check goes by citation: the hash is verified again when the report document is retrieved, everything is recomputed independently, and only when every number carries a citation is the issuance judged complete. +- The inputs of caliber card metrics the report cites must trace back to raw data (cells of query snapshots, run inputs): in formal runs at the governed level, inputs that can't be traced count as gaps and downgrade the issuance; otherwise they are stated in the issuance declaration, and the evidence panel still marks them "Can't trace back to raw data". +- Analysis claims in the report are checked again here independently, with the same check used when the report is assembled: when the report-writer node is set to strict, remaining violations count as issuance gaps (see [Analysis claim check](#analysis-claim-check)). + ### Metric issuance A pure metrics dashboard (the contract sets only `metrics_from`, there is no report-writer node upstream, and the output contains no model-written text) is issued by its metrics. With no narrative to check, the metrics themselves are checked: @@ -62,6 +65,8 @@ When the source query of a contract metric has SQL check errors, the issuance is - An expression can be written as `cell(nodes.pull, 0, 'gmv')`, which makes the provenance precise to the cell. - Leave rounding and default values to the caliber card's decimal places and display settings instead of writing them into the data-fetch SQL (for why, see [Caliber Ledger](evaluation-and-ledger.md#a-caliber-label-for-each-number)). - When publishing at the governed level, caliber card inputs can't come from sandbox code in the compute role (gate G4). +- When a metric uses two queries of the same shape that differ only in their dates (this week versus last week), the system runs a comparability check once automatically (see [Comparability check](analysis.md#comparability-check)) and attaches the result to the metric: the writing catalog states it plainly; at an exit with a contract, incomparable periods count as a gap and downgrade the issuance, and conditionally comparable ones are only flagged. +- Analysis card results are not caliber card inputs: a caliber card that reads them is treated as unable to trace back to raw data. ### Caliber version bumps @@ -88,9 +93,13 @@ In the report-writer node (`report`), the model writes only citation markers, an [[table:Q1 cols=region,amount rows=0-4]] the system builds the whole table from the snapshot; every cell is clickable [[t:orders]] [[c:orders.week]] table name, field name; names written in backticks are checked too [[q:K1|]] verbatim knowledge-base quote; must appear word for word in the chunks that retrieval hit -[[see:m:gmv,Q1]] basis of a claim sentence; attached at the end of the sentence, not rendered +[[a:A1]] [[a:A1.top_share|pct]] an analysis conclusion's verdict phrase and a named value, both rendered by the system +[[v:A1.r0.share]] [[table:A1 rows=0-4]] one cell and the whole of an analysis conclusion's result table +[[see:m:gmv,Q1,A1]] basis of a claim sentence; attached at the end of the sentence, not rendered ``` +Analysis conclusions are numbered A1, A2, … in ledger order, and the writing catalog has a separate Analysis Conclusions section. Judgment words such as "mainly from", "unusual", or "rising steadily" must cite a matching analysis conclusion and agree with its verdict; see [Analysis claim check](analysis.md#analysis-claim-check). + ### Handling violations - Bare numbers and citations that can't be parsed are flagged as violations on the spot and sent back for rewriting (`max_repairs`, default 1). @@ -99,6 +108,7 @@ In the report-writer node (`report`), the model writes only citation markers, an ## The data retrieval chain - Data fetched by agents, tool nodes, code nodes, and knowledge retrieval is recorded in the evidence ledger. The report's source list is numbered from the ledger: Q1… for queries, K1… for knowledge-base chunks. +- Analysis conclusions from agents' analysis tools and from analysis cards are recorded in the evidence ledger too, numbered A1…. Analyses read only query snapshots the platform executed during this run, and each conclusion is sealed together with its inputs (see [Analysis](analysis.md)). - When an agent turns on `cite_fields` (together with `output_schema`), each field records which cell of which query it came from, and that is checked against the snapshot: on a mismatch the snapshot wins and a warning is raised; anything not found is recorded as null, not defaulted to 0. - A code node's `evidence_role` is `source` (data retrieval) or `compute` (computation, the default). Numbers computed in the sandbox can't be cited directly by a report; they must go through a caliber card. - Cells of uploaded spreadsheets can be traced back to the cell in the original sheet (an inferred source). @@ -126,6 +136,7 @@ Query a table that keeps only the latest day (fully reloaded every day) for last - **Numbers**: the metric's original formula, substituted formula, and recomputed result; whether the inputs match the snapshot; the query's SQL; the cited row plus 2 rows before and after, highlighted down to the cell; and finally the seal status. Only artifacts returned by events inside the sealed scope are accepted. The provenance chain also starts with a Caliber step (see [Caliber Ledger](evaluation-and-ledger.md#a-caliber-label-for-each-number)). - **Table and field names**: which query they appear in, their type, and the table they belong to. A name found nowhere in the schema snapshot, the SQL, or the result columns is marked as a suspicious entity ("May be a fabricated name", with the 3 closest matches). When a snapshot was truncated and can't be checked, it only says so, without wrongly calling the name fabricated. - **Knowledge-base quotes**: must appear word for word in the chunks that retrieval hit. +- **Analysis conclusions** (verdict phrases, named values, result table cells): in order, the verdict (verdict phrase, criterion, segments named and direction, who did it, caveats, and the agent's note on what the call was testing), the method (method and version, parameters, thresholds actually used, table ranges checked live), the calculation (the result table and the substituted formula for the key value, such as "share = East's change / total change = -120 / -194", with the cited cell highlighted), the inputs (which columns and rows of which Q, the closure check against the two total cells, the level a drill-down sits under, followed by the steps of the input queries), the recomputation (rerun from the sealed inputs with the same function; it verifies only when every field is identical, and when the method has a new version, it says which version the run used), and the seal. In the evidence graph, an analysis conclusion has a "derived from" edge to its inputs and to the level above it. ## Claim sentences @@ -165,12 +176,14 @@ The two exports serve different purposes: to prove nothing was altered, use the - The body is rendered on the server, so it is readable without running scripts (email attachment previews often don't run scripts). - With scripts running, every number, every number without evidence, and every suspicious table or field name is clickable. The panel shows the same provenance chain as the evidence panel on the web page (caliber, source, calculation, SQL, cited rows), with the seal status marked item by item. - Masked columns say "Masked", following the same rules as the web page. +- Verdict phrases, named values, and result table cells of analysis conclusions are clickable too, and the panel shows the verdict, method, calculation, and inputs in turn; at export time the server recomputes each conclusion from the sealed inputs and writes the result into the panel. The Analysis Scope at the end of the report is rendered from what the document recorded, with clickable numbers. - Unsealed runs export as usual, for the same reason you can read their reports on the web. **Verifiable report**: Export Verifiable Report under Provenance Details on the Records page (`GET /api/runs/{id}/bundle.html`) produces a seal verification document. - The recipient recomputes the hashes in the browser, without connecting to the service or installing anything, and checks that no number, document, or run record has been altered. - The provenance panel for a number likewise shows the caliber, original formula, substituted formula, inputs, and recomputation; the query snapshots that caliber card inputs point to come attached to the file. +- Analysis conclusions the report cites (including those named in the Analysis Scope) come attached to the file byte for byte, together with their input snapshots and the conclusions they drill down from; at export time the server recomputes them from the sealed inputs and records the result on each conclusion's entry. The browser only verifies hashes and shows the server's recomputation; the page doesn't recompute statistics itself. When any input snapshot can't be shown to leave masked columns untouched, the conclusion is given by id only. - It proves only that the file is internally consistent: numbers, documents, and run records agree with one another and haven't been altered. To check authenticity, compare the seal hash on the page with the seal verification of this run in the system (`GET /api/runs/{id}/verify`); the page says so prominently. - Events appended after sealing (annotations such as on-demand judging and approval inquiries) are listed separately, with only their type, sequence number, and a content summary, and aren't part of the manifest. - When a data source has masked columns, query snapshots that can't be shown to leave them untouched, and caliber cards with inputs from masked columns, are given by id only, without content. Masking only reduces exposure and is not a security boundary. @@ -246,4 +259,14 @@ For two runs of the same workflow (this week's and last week's weekly report, sa - When the caliber and data both changed and a strict split is possible, the caliber effect and the data effect are given separately, and they add up to the total change. - For metrics in the data-only, parameters-only, and caliber-and-data classes, when attribution is possible, the dimension values (regions, say) with the largest changes are listed, up to 10. Every item traces back to the query snapshots on both sides; queries run by verification take no part in the comparison. - Values from masked columns never appear in the result, and metrics that use masked columns aren't split. +- When the graph has analysis cards, the two runs' analysis cards are aligned by node and item, listing each run's status, verdict, direction, and verdict phrase; for cards executed several times in a loop, the last execution counts. - When a scheduled run finishes successfully and is sealed, it is compared once automatically with the same schedule's last successful run; the trigger history and notifications show only which run was the baseline and how many metrics fall into each class, never the actual numbers. + +### Analysis claim check + +Judgment words in a report, such as "mainly from East", "outside the usual fluctuation", "rising steadily", or "the difference holds up", must cite a matching analysis conclusion and agree with its verdict; a stated cause needs a verbatim knowledge-base quote or a run input as its basis, or else it is written as a conjecture to be verified. + +- The report-writer node checks this sentence by sentence while writing, with a fixed lexicon and binding rules: no model call, no cost. An exit in citation mode checks again independently. +- By default, the report goes back for one rewrite and what remains is marked in the report (visible in the evidence panel, the audit, and exports), without affecting the issuance tier; set to strict, it rewrites up to the maximum number of repairs, and what remains counts as an issuance gap and downgrades the issuance. The governed level requires strict (gate G3). +- Only new report documents that carry this check's record are checked; verification results for reports from before the upgrade don't change. +- For the classes of judgment words, the analysis each one needs, and which wordings aren't checked, see [Analysis claim check](analysis.md#analysis-claim-check). diff --git a/docs/en/tools.md b/docs/en/tools.md index 58fc43fd..b8ddc562 100644 --- a/docs/en/tools.md +++ b/docs/en/tools.md @@ -2,17 +2,27 @@ English | [Simplified Chinese](../zh-CN/tools.md) # Tools, MCP, and the Capability Registry -This page covers the tools workflows can call (built-in tools, custom tools, and MCP), their approval levels, how admins govern them in the capability registry, and how to register endpoints for external systems. It is for builders and admins. The user interface is currently available in Simplified Chinese only. +This page covers the tools workflows can call (built-in tools, including the analysis tools, custom tools, and MCP), their approval levels, how admins govern them in the capability registry, and how to register endpoints for external systems. It is for builders and admins. The user interface is currently available in Simplified Chinese only. ## Kinds of tools -- **Built-in tools**: 15 of them. +- **Built-in tools**: 23 of them, 5 of which are analysis tools (see below). - **Custom tools**: an HTTP template, or Python in the sandbox. - **MCP**: connect MCP servers (stdio or http). - **Data source tools**: each data source's query tool, `db_query__`, plus `db_schema__` for looking up schemas at run time (see [Data Sources](data-sources.md)). Who can configure what: MCP stdio, endpoints and credentials, and capability governance can be changed only by admins; builders can read them, and users can't see them (see the [permission table](identity-and-security.md#permission-table)). +## Analysis tools + +The five analysis tools are the platform's deterministic analyses: `analysis_decompose` (change decomposition), `analysis_baseline` (fluctuation baseline), `analysis_trend` (trend test), `analysis_differ` (difference test), and `analysis_comparable` (comparability check). They don't query any database; they analyze only results that the agent they are bound to queried during this execution. The platform computes the verdicts with fixed methods and thresholds, and the system writes the verdict phrases (see [Analysis tools for agents](analysis.md#analysis-tools-for-agents)). + +- Their parameters include no thresholds. Binding any one of them injects the platform's built-in analysis method into the agent's system prompt, once. +- They are read-only, write nothing, and aren't dangerous tools, so under the default approval policy they don't wait for approval; users can use them in Ask Data too. +- They can be bound only to agents: putting one in a Call Tool node is a canvas validation error. For fixed analyses in a template, use an analysis card (see [Analysis cards](analysis.md#analysis-cards)). +- Each agent can run at most 12 analyses per execution (changed under Settings → Preferences → Analysis). +- They don't connect to data sources: they read snapshots already queried during this execution, and what they return is an analysis conclusion, not a query result. + ## Approval levels The platform can't tell what MCP and custom tools do, so by default every call to them waits for approval. There are three approval levels: `ask`, `gated`, and `always`. diff --git a/docs/en/upgrade-notes.md b/docs/en/upgrade-notes.md index 92d3f9d8..946fa378 100644 --- a/docs/en/upgrade-notes.md +++ b/docs/en/upgrade-notes.md @@ -4,6 +4,22 @@ English | [Simplified Chinese](../zh-CN/upgrade-notes.md) This page lists, newest first, the changes in each update that affect users and operators, and what to do when upgrading. It is for operators and admins. The features themselves are described in the topic documents. The user interface is currently available in Simplified Chinese only. +## 2026-10, the analysis round + +**What to do**: nothing is required, and the database schema doesn't change. For report templates you will publish again at the governed level, set the report-writer node's Analysis Claim Check to strict; the pre-publish check points it out, and it can be fixed in one click. + +- New analysis capability (see [Analysis](analysis.md)): five deterministic analyses (change decomposition, fluctuation baseline, trend test, difference test, comparability check). The system computes the verdicts and writes the verdict phrases, and reports cite them with `[[a:A1]]`; they are clickable and can be recomputed. +- A new node, the analysis card (`analysis`), and five analysis tools; built-in tools go from 18 to 23. A new template, 11 Anomaly Analysis (Traceable), is added to the template list automatically on the first start after upgrading. Templates are installed once by name: in existing deployments, template 9, Traceable Weekly Report, isn't updated, so when publishing it at the governed level, add the analysis claim check as the pre-publish check suggests. +- Report-writer nodes check analysis claims by default (rewrite once, then mark): sentences whose judgment words such as "mainly from", "unusual", or "rising steadily" don't cite a matching analysis conclusion, or whose causes have no basis, go back to the writer for one rewrite; what remains is marked in the report without affecting the issuance tier. The writing rules gain a passage about judgment words, including for reports in graphs without any analysis: when there are no analysis conclusions, the writer is told not to make such judgments and to state only numbers and changes. To skip the check, set the report-writer node's Analysis Claim Check to off. +- The governed level requires the Analysis Claim Check to be strict (one more item under gate G3). Versions already published keep running as usual, checked with the default rewrite-once-then-mark; the next publish is checked against the new rule. +- Reports that use analysis conclusions get a system-written Analysis Scope section at the end, outside the body. Report documents also record which disposition and which version of the lexicon their analysis claims were checked with: only new documents that carry this record are checked for it on verification, so verification results for reports from before the upgrade don't change. +- An exit in citation mode now collects the metrics the report cites: inputs that can't be traced back to raw data count as gaps and downgrade the issuance in formal runs at the governed level; at other levels they are still only stated in the issuance declaration. +- A caliber card metric that uses two queries differing only in their dates gets a comparability check automatically: at an exit with a contract, incomparable periods count as a gap and downgrade the issuance, and conditionally comparable ones are only flagged. +- The report export and the verifiable report include analysis conclusions and their input snapshots, recomputed from the sealed inputs for verification. +- Settings gain a Preferences → Analysis section (see [Analysis settings](configuration.md#analysis-settings)). It is snapshotted into a run only when the graph has an analysis card or an agent with analysis tools, so the start records of other runs don't change at all. +- Ask Data: Copilot gains a graph shape for analytical questions, and its self-check reminds it to give the querying agent the analysis tools (one round only). The review merges several analyses that couldn't be done into one "n analyses couldn't be done" note, without an automatic redo; analysis calls that were wrong but later fixed no longer count as a gap, and ones never fixed no longer trigger a redo of the whole turn. +- Evaluation gains contribution and verdict expectations (computed from the expected SQL with the same analyses, never typed in by hand), and the scoring rules move to version 3, so old evaluations can be re-scored under the new rules; Copilot's prompt fingerprint now includes the analysis method, so comparisons with evaluations from before the upgrade flag "Prompt changed" (see [Regression evaluation](evaluation-and-ledger.md#regression-evaluation)). + ## 2026-10, orchestration improvements wave 2: PostgreSQL and multiple processes **What to do**: in a multi-process deployment, stop all worker processes before upgrading; pass values that code nodes need explicitly; don't delete `.agentlab-data` in the data directory. diff --git a/docs/en/workflows.md b/docs/en/workflows.md index b49ca16c..90fe53de 100644 --- a/docs/en/workflows.md +++ b/docs/en/workflows.md @@ -40,7 +40,7 @@ When motion is turned off at the system level (`prefers-reduced-motion`), the li ### Built-in templates -There are 9 built-in templates, from Minimal Q&A to Traceable Weekly Report. They run as they are and also serve as executable documentation of how each kind of node is used. +There are 11 built-in templates, from Minimal Q&A to Anomaly Analysis (Traceable). They run as they are and also serve as executable documentation of how each kind of node is used. Templates 9, Traceable Weekly Report, and 11, Anomaly Analysis (Traceable), can be published at the governed level as they are: the first demonstrates caliber cards and issuance by citation; the second demonstrates the analysis card, which takes two queries, judges whether this week's change is within the usual fluctuation and which region it mainly came from, and leaves the report to cite only the verdict phrases (see [Analysis](analysis.md#analysis-cards)). ## Node types @@ -51,9 +51,9 @@ There are 9 built-in templates, from Minimal Q&A to Traceable Weekly Report. The | Execution | `tool` `code` `merge` | Call a tool directly / execute code in the sandbox / merge the results of several queries (aligned by key across databases) | | Control | `branch` `loop` `subgraph` | Conditional or semantic branch / iteration and conditional loop / nested workflow | | Context | `memory` `retrieve` `transform` | Long-term memory read/write / knowledge retrieval / data shaping | -| Gatekeeping | `human` `validate` `metrics` | Human-in-the-loop / Schema validation and automatic rework / caliber card | +| Gatekeeping | `human` `validate` `metrics` `analysis` | Human-in-the-loop / Schema validation and automatic rework / caliber card / analysis card (a fixed set of deterministic analyses on upstream query results) | -The `output` node validates structured output against a JSON Schema; when the output doesn't conform, the errors go back to the model for automatic rework. For `metrics` (caliber card) and `report` (report writer), see [Evidence and Issuance](evidence.md); for `merge` (merge query), see [Data Sources](data-sources.md#cross-database-merge). +The `output` node validates structured output against a JSON Schema; when the output doesn't conform, the errors go back to the model for automatic rework. For `metrics` (caliber card) and `report` (report writer), see [Evidence and Issuance](evidence.md); for `analysis` (analysis card), see [Analysis](analysis.md#analysis-cards); for `merge` (merge query), see [Data Sources](data-sources.md#cross-database-merge). ## Passing data between nodes @@ -125,7 +125,7 @@ Approvals, supplementary input, draft editing, action confirmations, code and to ## Agent nodes -An `agent` node is a model loop with tools. An allowlist on the node decides which tools it can use (see [Tools](tools.md)). An agent with `output_schema` can turn on `cite_fields`, so each field records which cell of which query it came from (see [Evidence and Issuance](evidence.md#the-data-retrieval-chain)). +An `agent` node is a model loop with tools. An allowlist on the node decides which tools it can use (see [Tools](tools.md)). An agent with `output_schema` can turn on `cite_fields`, so each field records which cell of which query it came from (see [Evidence and Issuance](evidence.md#the-data-retrieval-chain)). An agent that queries data can also be given the analysis tools: breaking down changes and judging anomalies and trends is computed by the platform with fixed methods and thresholds, the system writes the verdict phrases, and reports cite them (see [Analysis tools for agents](analysis.md#analysis-tools-for-agents)). ### Guardrails @@ -186,7 +186,7 @@ On top of the governance check, the governed level adds evidence gates. They loo - **G1** At an exit with a contract, the text in output fields may come only from report-writer nodes. - **G2** Model-written text doesn't bypass the report-writer node to flow straight into the exit. -- **G3** The report-writer node sets `numbers: strict`, `on_violation: fail`, and `claims: require_citation` (or `judge`, with a budget). +- **G3** The report-writer node sets `numbers: strict`, `on_violation: fail`, `claims: require_citation` (or `judge`, with a budget), and `analysis_claims: strict` (the analysis claim check set to strict; see [Analysis claim check](analysis.md#analysis-claim-check)). - **G4** Caliber card inputs don't come from sandbox code in the compute role. - **G5** Agents that feed a caliber card have `output_schema` set and `cite_fields` turned on. diff --git a/docs/zh-CN/analysis.md b/docs/zh-CN/analysis.md new file mode 100644 index 00000000..d0f442b6 --- /dev/null +++ b/docs/zh-CN/analysis.md @@ -0,0 +1,204 @@ +[English](../en/analysis.md) | 简体中文 + +# 分析能力 + +这篇讲报告里的分析判断从哪来：「主要来自华东」「超出常规波动」「持续上升」「差异站得住」这类判断怎么由系统算出来、怎么写进报告、怎么核对。包括五种分析各回答什么问题、按什么标准判、算不了时怎么说，Agent 的分析工具和分析卡两个入口，报告里的引用写法，判断词的核对，报告末尾的「分析范围」，以及设置和限制。给搭建者和看报告的人看。 + +## 是什么、为什么 + +报告里的数字由系统照快照填进去，叙述层不做算术（见[证据与出具](evidence.md#基本思路)）。可报告要说的不只是数，还有判断：变化主要来自哪里、这次算不算异常、是不是一直在涨、两组的差别是不是真的。以前这些判断由写报告的模型自己下，它挑着证据说，读的人核对不了。 + +分析能力把判断也收回系统： + +- **判定由系统算**：五种分析是平台的确定性代码，只读这次运行里交回过的查询快照，按固定的方法和阈值算出判定。模型只决定算什么（比哪两期、拆哪个维度、往哪一段下钻），改不了方法和阈值。 +- **判定语由系统写**：每份分析结论带一句系统按模板写好的判定语，报告引用它（`[[a:A1]]`），和数字一样点得开出处。写作者自己写「主要来自」「异常」这类判断时，系统逐句核对它挂没挂对应的分析结论、判定对不对得上（见[分析说法核对](#分析说法核对)）。 +- **照实说**：算不了就说为什么——历史不够、本期没过完、查询被截断、两期不可比、样本太少——不硬给结论。算不了本身也是一条结论，写作者照实写。 +- **留痕**：每份分析结论连同方法与版本、参数、实际用的阈值、输入的格、现查的表范围、结果和判定，存成内容寻址的工件，进证据台账、随运行封存；证据面板、报告导出、可验证报告都按同一个函数复算核对（见[点开一个数看到什么](evidence.md#点开一个数看到什么)）。 + +拆解只说明变化来自哪里，说明不了为什么。原因要有知识库原话或运行输入作依据，否则写成待核实的推测。 + +## 五种分析 + +每做一次分析产出一份分析结论，写作目录里按先后编成 A1、A2……（和查询的 Q1、知识库的 K1 同一套编号规则）。一份结论有：判定（判定码、点名的段、方向）、判定语、判定标准（照实际阈值写的一句）、具名的数（占比、总变化这类）、结果表、附带说明（抵消、辛普森悖论、样本少、历史新高……，最多两条接在判定语后面）。 + +下面括号里的阈值都是缺省值，在「设置 → 偏好设置 → 分析」里改（见[分析设置](configuration.md#分析设置)）。 + +### 变化拆解 + +回答「变化从哪来」。三种模式： + +| 模式 | 用在 | 怎么拆 | +|---|---|---| +| 可加 | 销售额、订单数这类能相加的量 | 按维度（一到两列）算各段对总变化的贡献和占比 | +| 比率 | 退款率、客单价这类比率 | 拆成结构变化（各段的占比变了）和各段自身的变化，两者之和精确等于比率的变化 | +| 乘积 | 销售额 = 订单数 × 客单价 | 按对数平均（LMDI）拆成各因子的贡献 | + +- 判定：一段占总变化的一半以上（0.5）算主要来源（「主要来自华东」）；没有这样的一段、前几段合计占八成以上（0.8）算集中（段数最多是全部段数的两成，至少 2 段）；总变化不到基期的 0.5% 算基本持平；其余算分散。比率拆解判主要来自结构变化、主要来自各段自身的变化，或两者都有影响；乘积拆解判主要来自哪个因子。 +- 给了总数时，各段之和要和总数对得上（按总数的书写精度容差）才判断变化来自哪里；对不上就不判，判定语写差多少。没给总数时以各段之和为总数，判定标准里写明。 +- 反向的段合计超过总变化的四分之一（0.25）时附一句互相抵消了一部分；比率拆解里各段大多在降、整体却在升（或反过来）时附一句辛普森悖论的说明；分母不到 30 的段标「分母小」，不参与这个判断。 +- 下钻：一次拆解里的某一段可以接着往下拆一层（华东按渠道），这一层各段之和要等于上一层那一段的值，最多下钻 3 层。下钻的查询由 Agent 自己写、只查那一段，系统按上一层那一段的值核对闭合。 +- 判定语举例：「销售额下降 194（-9.2%），主要来自华东（-120，占 61.9%）」。 + +### 波动基线 + +回答「这次的变化大不大、算不算异常」：把本期放回它自己的历史里看。 + +- 输入一条按期次排的序列（日、周、月、季）。缺省比的是和上一期比的变化率，日数据和上周同一天比，避开星期效应；存量、比率可以比值本身；要同比就和 52 周前、12 个月前比，历史要相应够长。 +- 用历史的中位数和中位绝对偏差（稳健标准差 = 1.4826 × 中位绝对偏差）：本期和中位数的差不到 2 倍稳健标准差算常规波动，2 到 3 倍算超出，3 倍以上算明显超出。用中位数不用均值：促销周这类极端值会把均值和标准差拉宽，正常的周、异常的周都显得不异常。 +- 历史至少 8 期、最多用 52 期；本期是历史里最高或最低的，附一句历史新高、新低。 +- 判定语举例：「销售额本期变化 -15.8%，明显超出过去 24 周的常规波动范围（-0.5% 至 +0.4%）；本期的销售额是过去 24 周里最低的」。 + +### 趋势判断 + +回答「是在持续变，还是一次波动」。 + +- 取最后连着的一段（至少 6 期）：Mann–Kendall 检验的双侧 p 值小于 0.05 时，按 Theil–Sen 斜率的正负判上升或下降，否则判没有明显趋势。末尾连续 3 期以上同向时，判定语接「已连续 3 周下降」，和趋势反向时写「但最近已连续 3 周上升」。 +- Mann–Kendall 假设各期独立，业务序列常有自相关，p 会偏小，所以判定语只说「呈上升趋势」，不说「显著上升」。日数据附一句星期效应，建议按周汇总。 +- 判定语举例：「过去 12 周线上销售额呈上升趋势（每周约 +2.1%），已连续 4 周上升」。 + +### 差异检验 + +回答「这两组（或同一组的两期）真的不一样，还是样本波动」。 + +- 比率（转化率、退款率）给成功数和试验数：每组算 Wilson 区间；各格的期望频数都不小于 5 时用合并比例的 z 检验，否则用 Fisher 精确检验。 +- 均值给每组的均值、标准差、样本数，或者合计、平方和、样本数（SQLite 没有标准差函数时这样写），用 Welch t 检验；也可以给逐行的值（不超过 1000 行、没截断），系统自己算，另做一遍去掉最远那个值的敏感性检查。 +- 双侧 p 值小于 0.05 算差异站得住；一次比了好几对时按 Holm 校正。均值有一组样本不到 30、比率用了 Fisher 时附一句样本少。 +- 判定语不出现「p 值」「显著性」这些词：站得住写「差异站得住」，否则写「差异可能只是样本波动（样本量 230、40）」。p 值和区间在结果表和证据面板里。 + +### 可比性检查 + +回答「这两期能不能直接比」。每项判通过、提醒、不通过或核对不了： + +| 项 | 怎么判 | +|---|---| +| 同口径 | 两份查询只差日期才算同口径；结构不同，或者除了日期还筛了别的（华东对华南），不通过 | +| 窗口长度 | 两期天数一样；可加的流量不一样就不通过，比率、存量只提醒 | +| 整期 | 本期过完了、表里的数据到了；流量没到齐不通过（「本期只有 3 天数据」），其余提醒 | +| 星期构成 | 按天取、天数不是 7 的倍数时，两期的周末天数要一样，不一样提醒 | +| 数据问题 | 查询区间里没有数据不通过，SQL 检查有错误提醒 | + +全部通过（或核对不了）判可比，只有提醒判有条件可比，有一项不通过判不可比。核对不了的项不判不可比，判定语里写明哪几项核对不了。表的实际日期范围现查（只读、限时、遮罩列不查，失败退回剖析记下的范围），查到的范围记进结论，复算按记下的值。 + +变化拆解给的是两份查询时，先自动做一次可比性检查；波动基线、趋势判断和单份查询的变化拆解看本期齐不齐时，用的是其中「整期」那一项。口径卡里一个指标用到两份同形、只差日期的查询（本周对上周）时，也自动做一次，结果挂在指标上（见[口径卡](evidence.md#口径卡)）。 + +## 算不了的时候 + +算不了不让 Agent、节点失败，记成一条结论，判定语写「波动基线没有做成：历史只有 5 期，至少要 8 期」这样的一句，进台账、能引用、能核对： + +| 原因 | 什么时候 | +|---|---| +| 截断 | 输入的查询只取回了前一部分行，各段之和不是完整的和 | +| 段太少 | 拆开后不到 2 段 | +| 历史不够 | 有效的历史期数不到最少期数 | +| 本期不完整 | 本期没过完、数据没到、不到整周，或者查询只取了本期的一部分 | +| 缺期 | 本期和比较期之间缺期；趋势判断用的最后几期不连着 | +| 期次认不出 | 期次列的写法认不出（带时刻的、写法混着的） | +| 比较期为 0 | 算不了变化率 | +| 有值不为正 | 乘积拆解要求各值都为正 | +| 两边不是同一个量 | 两份查询结构不同，不是同一个量的两期，不拆 | +| 拿不到 p 值 | 差异检验一对都算不出 p 值（两组各只有 1 个样本、样本量为 0） | +| 上一层没做成、这次的数据里没有 | 分析卡里下钻的上一层算不了，或者这期的数据里没有要用的期次、段、组、总数那一格 | + +调用写错不记结论：列不存在、拿遮罩列当维度或度量、比率当成可加的量、同一段同一期有好几行、参数写错、这个 Agent 这次执行的分析次数用完了。Agent 收到一句怎么改的说明，改了再调；分析卡里是模板写错了，整张卡失败并说清是哪一项。 + +期次认这几种写法：日期、`YYYY-MM`、`YYYY-Www`、`YYYY-Qn`、整数序号。按周汇总时，期次最好在 SQL 里由日期列算出来，写每周第一天的日期，或 ISO 周 `YYYY-Www`。同样写成 `2026-W42`，SQLite 的 `%W`、MySQL 的 `%u` 算出的和 ISO 周落在不同的日子上，系统按 SQL 里的格式串认出算法、算出真实的起止；直接取表里现成的周号文本时认不出算法，本期齐不齐核对不了。 + +## 两个入口 + +两个入口共用同一套分析、同一种结论、同一张判定语表。 + +### Agent 的分析工具 + +问数据和画布上的 agent 可以绑五个分析工具：`analysis_decompose`（变化拆解）、`analysis_baseline`（波动基线）、`analysis_trend`（趋势判断）、`analysis_differ`（差异检验）、`analysis_comparable`（可比性检查）。要看了数才知道拆哪个维度时用它。 + +- 参数只有「算什么」：分析本节点查过的哪个结果（`Q3`，和交回模型的「【证据 Q3】」同一个编号）、各列的角色、比哪两期、总数那两格、下钻的上一层、度量的叫法，和一句这次在验证什么（只在证据面板显示，不进报告正文）。没有任何阈值参数。 +- 绑了任一个，平台就把内置的分析方法注入 agent 的系统提示：先按问题定基期，做可比性检查和波动基线，再拆因子与结构、按维度由粗到细拆，找到主要来源再下钻一层；拆了不集中也是结论；不作因果断言；算不了照实说；最后列出做过的分析和判定。方法随版本升级。 +- 交回 agent 的是「【分析 A2】」加判定、判定语、关键的数、附带说明或算不了的原因；整份结论进证据台账。 +- 每个 Agent 每次执行最多分析 12 次、最多下钻 3 层。 +- 分析工具只分析绑了它的 agent 自己查过的结果，不能放进「调用工具」节点（模板里要固定做的分析用分析卡）。协作团队的成员也能用，但结论不进证据台账，报告引用不了。 +- 使用者（user 角色）的问数据里也能用：分析工具不查库，只读已经按遮罩处理过的查询结果，遮罩列不当维度、不当度量。 + +### 分析卡 + +分析卡（`analysis` 节点，和口径卡平级）对上游查询节点的结果固定做几项分析，用于模板里每次都要做的分析，比如周报的异动：这一周的变化在不在常规波动范围内、主要来自哪个地区。 + +- 每一项写明分析方法和模式、要分析哪个上游查询节点、参数和这一项在验证什么。输入只能是调用数据库查询工具的「调用工具」节点或「合并查询」，而且连在分析卡之前；Agent 可能查了好几次，不能当输入。参数和分析工具的一样，可以用 `{{ input.x }}` 按运行输入选期次。 +- 阈值不写在分析项里。模板要固定阈值时写在卡上的「阈值覆盖」里，随版本固定；它和这次运行的分析设置冲突时，卡失败并说清冲突在哪。 +- 产出 `nodes.<卡>.results.<项>`：状态、判定码、方向、具名的数、判定语。下游条件分支可以按判定走，比如超出常规波动就转人工。 +- 一项算不了不让卡失败，那一项照实记成算不了，接着做下一项。 +- 分析卡的结果不作口径卡的输入：口径卡读它，按追不到原始数据处理。指标在口径卡里按格从查询结果取，分析结论在报告里直接引用。 +- 受管级别要求分析卡的输入是调用数据库查询工具的「调用工具」节点或「合并查询」，发布时就能确定分析的是平台执行的查询。 +- 运行对比按分析卡的节点和项对齐，列出两次的判定（见[运行对比](evidence.md#运行对比)）。 + +内置模板 ⑪「异动分析（可追溯）」就是这样搭的：整体按周、按地区两周取数 → 分析卡（波动基线、带总数的变化拆解）→ 报告撰写 → 出具，拿来就能按受管级别发布（见[内置模板](workflows.md#内置模板)）。 + +## 在报告里引用 + +``` +[[a:A1]] A1 的判定语，由系统按判定渲染，一个点得开的片段 +[[a:A1.top_share|pct]] A1 的一个具名的数，换算后缀同 [[m:]] +[[v:A1.r0.share]] A1 结果表里的一格 +[[table:A1 cols=地区,delta,share rows=0-4]] A1 的结果表 +[[see:A1]] 这句的依据，挂在句末，不渲染 +``` + +- 写作目录里「分析结论」一段列出每份结论的编号、判定语、具名的数、结果表的前 10 段、附带说明或算不了的原因。 +- 分析卡的结论总能引用。受管级别的正式出具里，Agent 用分析工具现场做的分析和它查的格同一个规矩：出具契约开了「单元格引用」才能引用（句末依据照样能挂）。 +- 判定语里的度量叫法是 Agent 或模板给的，系统先去掉里面的数字、百分号、单位和引用标记，截到 20 字以内：系统写的字里不会夹带模型写的数。 + +## 分析说法核对 + +报告撰写节点的「分析说法核对」（`analysis_claims`）逐句核对判断词：确定的词表加绑定，不调用模型。 + +| 判断词的类 | 例子 | 要挂的分析结论和判定 | +|---|---|---| +| 来源 | 主要来自、集中在、拖累、带动、贡献最大 | 变化拆解，判主要来源或集中；点名的段要在判定里，说拖累的那一段要在降 | +| 分散、抵消、结构 | 分散在各地区、此消彼长、结构变化 | 变化拆解，判分散、有抵消、主要来自结构变化 | +| 异常、新高新低、正常 | 异常、大幅、激增、历史新高、正常波动 | 波动基线，判超出或明显超出（方向对得上）、新高新低、常规波动 | +| 趋势、无趋势 | 持续上升、连续三周下降、没有明显趋势 | 趋势判断，方向、连续期数对得上 | +| 差异、无差异 | 显著高于、确实更低、差异不大 | 差异检验，差异站得住（方向对得上）或可能只是样本波动 | +| 因果 | 因为、由于、导致、受…影响 | 同一句有知识库原话或运行输入作依据，或者写成推测（可能、待核实） | + +- 一句的依据是句中的 `[[a:…]]`、`[[v:A…]]`、`[[table:A…]]` 和句末的 `[[see:A…]]`。否定会翻到反面：「并不异常」按正常核。 +- 不核的：说的是构成而不是变化（「本周订单主要来自线上渠道」）、建议和计划、业务状态里的「异常」（「部分订单异常」）；知识库原话、系统渲染的判定语、标题和表格。 +- 来源和因果的分界：「下降主要由华东造成」挂了按地区的拆解、点名的是拆解里的段，按来源核，这是记账意义上的「来自」；「由于促销结束导致下降」点名的不是段，按因果核。 +- 违规三种：判断词没挂对应的分析结论；挂了，但判定、点名的段或方向对不上（挂了算不了的结论也算）；原因没有依据。 + +| 取值 | 做法 | +|---|---| +| `check`（缺省，问数据也是它） | 交回写作者重写一次，仍有的标在报告里，不影响出具档位 | +| `strict` | 按最多重写次数重写，仍有的计入出具缺口、出具降档；受管级别要求它（门禁 G3） | +| `off` | 不核 | + +- 写作目录里有分析结论时，写作者被告知判断照判定写、最好直接引用判定语；没有时被告知不下这类判断，只陈述数字和变化。 +- 引用模式的出口独立再核一遍，和组装时用同一个核对。报告文档记下按哪种处置、哪一版说法表核：只有带这一项的文档在复核时核这一项，升级前的报告复核结果不变。 +- 宁可漏报：只收强判断词。漏掉的细处由结论句裁判兜底（见[结论句](evidence.md#结论句)）；裁判摘录里有分析结论时，裁判另被告知判定是系统算的，只判句子说得对不对。 + +## 分析范围 + +报告用到了分析结论（这次运行有分析结论）时，系统在报告末尾附一段「分析范围」。写作者改不了，随文档封存，编号点得开： + +- **现象**：可比性检查、波动基线、趋势判断的判定语。 +- **拆过**：每份变化拆解按下钻链列出维度和判定。 +- **排除**：判定为分散的拆解、差异可能只是样本波动的检验——拆了没发现集中也是结论。 +- **算不了**：各项算不了的原因。 +- **没拆**：候选维度里没拆过的。候选维度从数据目录确定地算：拆解所用查询里的事实表上标成属性、状态，不遮罩、有码值表（取值不超过 50 个，或者已列全）的列，加上它关联的维度表上同样的列。拆解用的列对不上目录时只说候选维度核对不了，不乱列。 +- **检验**：这次做了几次差异检验，是否已按次数校正。 + +Agent 写的「这次在验证什么」不进这一段，只在证据面板显示。报告撰写节点的「分析范围」（`analysis_scope`）缺省是有分析结论时附上，可以设成不附。 + +## 设置 + +「设置 → 偏好设置 → 分析」里改判定用的阈值、Agent 每次执行最多分析几次、最多下钻几层、列出前几段，各项的叫法和缺省见[分析设置](configuration.md#分析设置)。 + +- 发起运行时把这一组快照进运行（只在图里有分析卡、或者 agent 绑了分析工具时），恢复和接着跑沿用同一份。 +- Agent 改不了阈值：分析工具的参数里没有阈值。模板可以在分析卡上写阈值覆盖。 +- 每份结论记下实际用的阈值，判定标准照这些值写出来，例如「一段占总变化的一半以上算主要来源」。 + +## 限制 + +- 这一轮没有：预测；相关分析；维度层级进数据目录；数值列的剖析；双盲复算、反方审稿、审批质询的核查 agent 使用分析工具；分析结论参与口径账本的比对和召回（分析结论里的数不标口径，见[口径账本](evaluation-and-ledger.md#口径账本)）。 +- 趋势判断假设各期独立，自相关的序列 p 会偏小。 +- 候选维度只认码值表：剖析目前不记每列的取值数，没有码值表的属性列不进候选。 +- 说法表只收强判断词，换个说法可能漏过，漏过的由结论句裁判和读报告的人兜底。 +- 协作团队成员做的分析不进证据台账，报告引用不了。 diff --git a/docs/zh-CN/ask-data.md b/docs/zh-CN/ask-data.md index b5246221..1459c2f8 100644 --- a/docs/zh-CN/ask-data.md +++ b/docs/zh-CN/ask-data.md @@ -25,6 +25,17 @@ - 助手只给了思考、一个操作都没给时，自动重来一次。 - 每一轮的花费包括助手建图、挑表和复核的模型调用。 +## 分析类问题 + +问的是为什么、怎么变了、哪里拖累、谁贡献最大、是否异常、波动大不大、是不是持续在涨、是不是真的更高这类问题时，助手搭的图里，查库的 agent 另绑五个分析工具（见[分析能力](analysis.md)）： + +- agent 先按问题定比哪两期（问题没说就比上一个整期），再照平台内置的分析方法做：可比性检查、放回历史看、按维度拆解，找到主要来源再下钻一层。判定由平台按固定的方法和阈值算，agent 只决定算什么。 +- 报告按分析结论写，判断直接引用系统写好的判定语。写作者自己写了「主要来自」「异常」这类判断的，系统逐句核对：没挂对分析结论的交回重写一次，仍有的标在答案里，不影响出具档位（见[分析说法核对](analysis.md#分析说法核对)）。 +- 用到分析结论的答案末尾，系统附一段「分析范围」：拆过什么、排除了什么、哪些候选维度没拆、哪些算不了（见[分析范围](analysis.md#分析范围)）。 +- 拆解只说明变化来自哪里，说明不了为什么：答案里的原因要有知识库原话作依据，否则写成待核实的推测。 +- 助手自查时按一张确定的词表认问题是不是分析类，只收明确的问法。是分析类、查库的 agent 却没绑分析工具时，交回助手改一轮；改过一轮仍没绑的照常运行，留一条提醒。 +- 分析没做成（历史不够、本期没过完……）、判断词重写后仍没挂对的，复核只写一句说明，不自动重来（见[产出复核](#产出复核)）。 + ## 产出复核 运行结束后，规则先扫一遍事件流，发现异常才调用模型复核。复核可以重组答案，但**不得引入原答案里没有的数字**。 @@ -32,10 +43,11 @@ - 使用者（user 角色）的问数据也会被复核。 - 子工作流里一步失败时，自动复核只报一条。 - 问数据里的「继续运行」只给搭建者以上。 +- 分析的几种缺口——有几次分析没做成、分析调用写错了一直没改对、判断词核对重写后仍有几处、口径卡指标的两期没做成可比性检查——只写说明、不自动重来。一次运行只有这几种时，复核不调用模型，照这几句原话写说明。 ## 答案里的口径 -口径是一个指标怎么取数、怎么算的约定，固定在口径卡里，带名字和版本。答案封存后，系统在后台把答案里的每个数和已发布的口径比对、标上口径，每一轮底下有一行摘要。整题等价于一份受管口径时，系统自动按口径发起一次正式运行。详见[回归评测与口径账本](evaluation-and-ledger.md#口径账本)。 +口径是一个指标怎么取数、怎么算的约定，固定在口径卡里，带名字和版本。答案封存后，系统在后台把答案里的每个数和已发布的口径比对、标上口径，每一轮底下有一行摘要。整题等价于一份受管口径时，系统自动按口径发起一次正式运行。分析结论里的数（占比、变化、波动范围）是从查询结果派生的，不和口径比对、不标口径，摘要行也不计它们。详见[回归评测与口径账本](evaluation-and-ledger.md#口径账本)。 ## 从答案到工作流和应用 @@ -60,6 +72,7 @@ - 改图时只给出改动操作，不重写整张图。 - 交付前按运行时同一套规则自查。查出会运行失败的问题，先交回去改；改不好就不自动运行。 +- 问数据里自查还看问题本身：分析类问题、查库的 agent 却没绑分析工具时，提醒一轮（见[分析类问题](#分析类问题)）。 - 建图时主动把应当记住的事实写进长期记忆。 ### 按需查看 diff --git a/docs/zh-CN/configuration.md b/docs/zh-CN/configuration.md index c279bb65..b0fff952 100644 --- a/docs/zh-CN/configuration.md +++ b/docs/zh-CN/configuration.md @@ -67,7 +67,7 @@ AgentLab 放在反向代理后面、或者要给外部系统回调时，设对 | 位置 | 内容 | |---|---| | 设置 → 模型接入 | provider 和密钥 | -| 设置 → 偏好设置 | 运行默认值（默认记忆作用域、默认知识库、危险工具默认需要人工审批）；证据裁判（裁判模型和各项上限）；核查（核查模型，双盲复算、反方审稿、审批质询的开关和上限，每日金额上限，见[复核套件](evidence.md#复核套件)）；评测的并发、每题时限、单次和每日金额；运行并发（管理员）；口径账本（自动按口径出具的开关和每日上限、口径比对时限、召回上限 `ledger.recall_limit`，见[口径账本](evaluation-and-ledger.md#口径账本)）；起草模型；应用（每个应用每天的按需运行上限、让使用者建个人应用、允许建分享链接）；界面主题 | +| 设置 → 偏好设置 | 运行默认值（默认记忆作用域、默认知识库、危险工具默认需要人工审批）；证据裁判（裁判模型和各项上限）；核查（核查模型，双盲复算、反方审稿、审批质询的开关和上限，每日金额上限，见[复核套件](evidence.md#复核套件)）；评测的并发、每题时限、单次和每日金额；分析（判定用的阈值、Agent 每次执行最多分析几次、最多下钻几层、列出前几段，见[分析设置](#分析设置)）；运行并发（管理员）；口径账本（自动按口径出具的开关和每日上限、口径比对时限、召回上限 `ledger.recall_limit`，见[口径账本](evaluation-and-ledger.md#口径账本)）；起草模型；应用（每个应用每天的按需运行上限、让使用者建个人应用、允许建分享链接）；界面主题 | | 设置 → 通用 | 对外访问地址 | | 设置 → 运行环境 | 沙箱候选和可用性、当前选中项和选择原因；时限等当前值 | | 设置 → 身份 | 登录方式；给使用者开启问数据 | @@ -77,6 +77,30 @@ AgentLab 放在反向代理后面、或者要给外部系统回调时，设对 其他设置项：`notify.digest_at`（每天待办汇总的时间），以及 agent 的令牌预算、金额预算和步数。 +### 分析设置 + +「设置 → 偏好设置 → 分析」一节，管变化拆解、波动基线、趋势判断、差异检验怎么判，以及 Agent 能分析多少次（见[分析能力](analysis.md)）。发起运行时快照进运行，恢复和接着跑沿用同一份；分析卡上可以写阈值覆盖，Agent 改不了。 + +| 设置项 | 缺省 | 取值范围 | 作用 | +|---|---|---|---| +| 「主要来源」的占比 | 0.5 | 0.3–0.9 | 一段（一个因子、一种效应）占总变化的比重到这么多，算主要来源 | +| 「集中」的合计占比 | 0.8 | 0.5–0.99 | 前几段合计占到这么多算集中；不能小于「主要来源」的占比 | +| 记为「抵消」的比例 | 0.25 | 0.05–1 | 反向的段合计超过总变化的这个比例时，附一句互相抵消 | +| 「基本持平」的幅度 | 0.005 | 0–0.1 | 总变化不到基期的这个比例，算基本持平 | +| 「超出常规波动」的倍数 | 2 | 1–5 | 波动基线：本期离历史中位数这么多倍稳健标准差，算超出常规波动 | +| 「明显超出」的倍数 | 3 | 1.5–8 | 到这么多倍算明显超出；要大于「超出常规波动」的倍数 | +| 至少几期历史 | 8 | 4–104 | 波动基线的历史不到这么多期，算不了 | +| 最多用几期历史 | 52 | 8–260 | 波动基线、趋势判断最多用这么多期；不能小于至少几期历史 | +| 趋势至少几期 | 6 | 4–104 | 趋势判断最少要这么多期 | +| 趋势的显著性水平 | 0.05 | 0.001–0.2 | p 值小于它才判上升或下降 | +| 差异检验的显著性水平 | 0.05 | 0.001–0.2 | 校正后的 p 值小于它算差异站得住 | +| 均值比较的最小样本数 | 30 | 2–10000 | 均值比较有一组样本不到这么多，附一句样本少 | +| 比率比较的最小期望频数 | 5 | 1–50 | 有格子的期望频数不到它时改用 Fisher 精确检验 | +| 比率拆解的最小分母 | 30 | 1–100000 | 分母不到它的段标「分母小」，不参与辛普森悖论的判断 | +| Agent 每次执行最多分析几次 | 12 | 1–50 | 用完了，Agent 再调分析工具会收到「已经用完」 | +| 最多下钻几层 | 3 | 1–6 | 变化拆解往下钻的层数上限 | +| 列出前几段 | 10 | 3–30 | 写作目录、交回 Agent 的结果表列出前几段 | + ## 沙箱后端 「设置 → 运行环境」列出全部候选和它们的可用性，并标出当前选中项和「选择原因」。自动选择的顺序： diff --git a/docs/zh-CN/evaluation-and-ledger.md b/docs/zh-CN/evaluation-and-ledger.md index 87990aca..cb55d33d 100644 --- a/docs/zh-CN/evaluation-and-ledger.md +++ b/docs/zh-CN/evaluation-and-ledger.md @@ -16,13 +16,18 @@ - 一份已发布口径的指标。 - 从某次问数据运行里认定的查询。 - 「没有数据」，或者期望反问。 +- 一个变化的主要来源（贡献）：期望 SQL 加一次变化拆解的参数。模式选可加、比率或乘积，参数名和分析工具的一样，另写点名前几段（1–10，缺省 1）；期望是算出的主要来源和方向。 +- 一个判定：期望 SQL 加波动基线、趋势判断、差异检验（要写模式）或可比性检查的参数；期望是算出的判定类——常规波动还是超出，上升、下降还是没有明显趋势，差异站得住还是不足以下结论，可比还是不可比（见[五种分析](analysis.md#五种分析)）。 规则： -- 数值不许人填。 +- 数值不许人填。贡献、判定两种期望里的段、方向、判定同样不许人填：在期望 SQL 的快照上，用和作答同一套分析算出来，作答前后各算一次。 +- 贡献、判定的参数里不写输入（就是期望 SQL 的结果）、总数、下钻和阈值；文字参数可以带定时变量 `{{ schedule.* }}`。期望 SQL 只返回两期时可以不写基期、本期：先的一期是基期。口径题、期望反问的题配不了这两种。 +- 阈值用这次评测开评时拍下的分析设置：算期望、各臂作答都按这一份，记进评测的指纹（「分析设置」）。期望算不了（历史不够、本期不完整……）的这一项不能评分，写明原因。 - 文字要点由裁判判，单列，不计入准确率。 - 口径升版后，用到它的题在顶部提示。每道题三选一：改为固定新版本、新旧两版各算一份期望、旧题归档另建新题。处置之前，评测时跳过这道题。 - 题集能导入导出。 +- 仓库带一份分析题用的虚构零售库和题集：在 `backend` 目录下运行 `python -m tests.fixtures.acceptance.retail_trends <库文件> [--today YYYY-MM-DD]`，生成截止昨天的 26 周订单（随机数种子固定），登记成名为 `retail_trends` 的只读数据源，再导入 `tests/fixtures/eval/analysis_suite.json`（八题，每个埋下的效应一题）。 ### 发起评测 @@ -42,7 +47,16 @@ - **对比**：和上一次或基线比，看逐题转移与归因。 - **A/B 配对表**：精确 McNemar 检验，只展示，不卡门。 -评测运行不进使用统计。评分规则现在是第 2 版；老评测可以在评测详情里按新规则重评（只读封存，不调模型）。 +贡献、判定两种期望这样评： + +- **对且可追溯**：答案引用了判定对得上的分析结论（同一种分析，主要来源点名的是期望的那几段、方向一致）；引用它的那句话却写反了的算错。 +- **对但没引用**：没引用对得上的结论，但正文按说法表写对了（同一套判断词和绑定规则，见[分析说法核对](analysis.md#分析说法核对)）。可比性没有对应的判断词，只能靠引用。 +- **错**：引用的结论判定对不上、正文写反了（把常规波动说成异常、把上升说成下降、主要来源说成别的段）、做了分析没说，或者根本没做。 +- **不能评分**：期望算不了。 + +系统写的「分析范围」不算答案的引用。答案引用分析结论时，逐题 E1 按它的输入查询比。 + +评测运行不进使用统计。评分规则现在是第 3 版（加了贡献、判定两种期望）；老评测可以在评测详情里按新规则重评（只读封存，不调模型）。助手提示词的指纹包含分析方法和写作者的分析规则，和上一次比时这些改了会标「提示词变了」。 ### 定时评测 @@ -68,6 +82,7 @@ - 逐个数判：报告引用了一条查询的几列（例如销售额、订单数、首末日）时，每个数按它那一列单独和口径的指标列比，多查的列不影响这个数。只在去掉别的列不会改变这个数时才这样比；有 DISTINCT、按别的列排序后截断、多出来分组列的，照旧整条比。 - 多行查询没有 `ORDER BY` 时，引用其中一格不算「来自口径」，召回和对账里照实标「行序不定」。 - 整题要每个来自数据的数都被证明来自同一份口径，才自动出具。 +- 分析结论里的数（占比、变化、波动范围）是从查询结果派生的，不比对、不标口径，摘要行也不计它们。 写口径的取数 SQL 时，最好不要取整（`ROUND(SUM(x), 2)`），也不要套 `COALESCE(…, 0)`。取整和缺省值交给口径卡的小数位和展示；否则问数据里没取整的写法永远认不成等价（差在哪会照实写出来）。 diff --git a/docs/zh-CN/evidence.md b/docs/zh-CN/evidence.md index cf7e69ad..731e6b69 100644 --- a/docs/zh-CN/evidence.md +++ b/docs/zh-CN/evidence.md @@ -2,7 +2,7 @@ # 证据与出具 -这篇讲报告里的数字从哪来、怎么核对：口径卡和出具契约、三档出具、报告撰写节点的引用标记、查询区间里没有数据时怎么办、点开一个数能看到什么、封存与核对、两种导出，以及双盲复算、反方审稿、审批质询、数字推演、运行对比这组复核工具。给搭建者、审批人和看报告的人看。 +这篇讲报告里的数字从哪来、怎么核对：口径卡和出具契约、三档出具、报告撰写节点的引用标记（含分析结论）、查询区间里没有数据时怎么办、点开一个数或一个分析结论能看到什么、封存与核对、两种导出，以及双盲复算、反方审稿、审批质询、数字推演、运行对比、分析说法核对这组复核工具。给搭建者、审批人和看报告的人看。分析判断本身怎么算见[分析能力](analysis.md)。 ## 基本思路 @@ -45,6 +45,9 @@ flowchart LR 契约写了 `report_from` 时按引用核对：取回报告文档时复验哈希、独立重算一遍，全部数字都带引用才判为完整出具。 +- 报告引用的口径卡指标，输入要追得到原始数据（查询快照的格、运行输入）：受管级别的正式运行里追不到的记缺口、出具降档，其余照实写进出具声明，证据面板照旧标「追不到原始数据」。 +- 报告里的分析说法在这里独立再核一遍，和组装报告时用同一个核对：报告撰写节点设成严格时，仍有的违规计入出具缺口（见[分析说法核对](#分析说法核对)）。 + ### 指标出具 纯指标看板（契约只写 `metrics_from`，上游没有报告撰写节点，成果里也没有模型写的话）按指标出具。没有叙述可核，就核指标本身： @@ -62,6 +65,8 @@ flowchart LR - 表达式可以写 `cell(nodes.pull, 0, 'gmv')`，出处精确到单元格。 - 取整和缺省值交给口径卡的小数位和展示，不要写进取数 SQL（原因见[口径账本](evaluation-and-ledger.md#逐个数标口径)）。 - 受管级发布时，口径卡的输入不能来自计算角色的沙箱代码（门禁 G4）。 +- 一个指标用到两份同形、只差日期的查询（本周对上周）时，系统自动做一次可比性检查（见[可比性检查](analysis.md#可比性检查)），结果挂在指标上：写作目录照实提醒；带契约的出口里，不可比的记缺口、出具降档，有条件可比的只标注。 +- 分析卡的结果不作口径卡的输入：口径卡读它，按追不到原始数据处理。 ### 口径升版 @@ -88,9 +93,13 @@ flowchart LR [[table:Q1 cols=region,amount rows=0-4]] 系统照快照生成整张表，每一格都点得开 [[t:orders]] [[c:orders.week]] 表名、字段名；写在反引号里的名字同样核对 [[q:K1|原话]] 知识库原话，须在那次检索命中的片段里逐字出现 -[[see:m:gmv,Q1]] 结论句的依据，挂在句末，不渲染 +[[a:A1]] [[a:A1.top_share|pct]] 分析结论的判定语、具名的数，都由系统按结论渲染 +[[v:A1.r0.share]] [[table:A1 rows=0-4]] 分析结论结果表里的一格、整张表 +[[see:m:gmv,Q1,A1]] 结论句的依据，挂在句末，不渲染 ``` +分析结论按台账先后编成 A1、A2……，写作目录里另有一段「分析结论」。「主要来自」「异常」「持续上升」这类判断词要挂对应的分析结论、判定对得上，见[分析说法核对](analysis.md#分析说法核对)。 + ### 违规怎么处理 - 裸数字、解析不了的引用当场判违规，交回去重写（`max_repairs`，默认 1 次）。 @@ -99,6 +108,7 @@ flowchart LR ## 取数链路 - agent、工具节点、代码节点、知识检索取数时都记进证据台账。报告的来源清单按台账编号：查询是 Q1…，知识库片段是 K1…。 +- agent 的分析工具、分析卡做的分析结论也记进证据台账，编号 A1…。分析只读这次运行里平台执行的查询快照，结论连同它的输入一起封存（见[分析能力](analysis.md)）。 - agent 打开 `cite_fields`（配合 `output_schema`）后，每个字段标出取自哪次查询的哪一格，再到快照里核对：对不上以快照为准并告警；查不到的记为空值，不兜底成 0。 - 代码节点的 `evidence_role`：`source` 是取数，`compute`（默认）是计算。沙箱算出来的数不能被报告直接引用，要进口径卡。 - 上传表格的单元格可以追到原表的格子（推断的来源）。 @@ -126,6 +136,7 @@ flowchart LR - **数字**：指标的原式、代入式和复算结果；输入与快照是否一致；查询的 SQL；被引用的行加前后各 2 行，高亮到格；最后是封存状态。只认封存范围内的事件交回过的工件。出处链最上面还有一步「口径」（见[口径账本](evaluation-and-ledger.md#逐个数标口径)）。 - **表名、字段名**：出现在哪次查询、类型、所属的表。表结构快照、SQL、结果列里都没有的名字标成可疑实体（「可能是编造的名字」，附最接近的 3 个）。快照被截断、核对不了的，只说核对不了，不冤枉成编造。 - **知识库原话**：须在那次检索命中的片段里逐字出现。 +- **分析结论**（判定语、具名的数、结果表的格）：依次是判定（判定语、判定标准、点名的段与方向、谁做的、附带说明、Agent 写的这次在验证什么）、方法（方法与版本、参数、实际用的阈值、现查的表范围）、计算（结果表，关键数的代入式，如「占比 = 华东的变化 / 总变化 = -120 / -194」，引用的那一格高亮）、输入（落到哪个 Q 的哪几列、哪几行，总数那两格的闭合核对，下钻的上一层，接着是输入查询的步骤）、复算（按封存的输入用同一个函数重算，逐字段相同才算核对通过；方法升了版本，照实写运行时用的是哪一版）、封存。证据图里，分析结论连一条「派生自」的边到它的输入和下钻的上一层。 ## 结论句 @@ -165,12 +176,14 @@ flowchart LR - 正文由服务端渲染，不执行脚本也读得到（邮件附件预览常常不执行脚本）。 - 执行脚本后，每个数字、无证据的数字、可疑的表名字段名都点得开。面板是网页上证据面板同一个拼法的出处链（口径、来源、计算方式、SQL、被引用的行），封存状态逐项标明。 - 遮罩列照网页上的规矩写「已遮罩」。 +- 分析结论的判定语、具名的数、结果表的格同样点得开，面板依次是判定、方法、计算、输入；导出时服务端按封存的输入复算一遍，复算结果写在面板里。报告末尾的「分析范围」照文档里记下的渲染，编号点得开。 - 没封存的运行照常导出，和在网页上读报告一个道理。 **可验证报告**：记录页「凭证详情」里的「导出可验证报告」，接口 `GET /api/runs/{id}/bundle.html`，是一份封存核对件。 - 收件人不连服务、不装任何东西，在浏览器里重算哈希，核对每个数字、文档和运行记录没被改过。 - 数字的出处面板同样带口径、原式、代入式、输入和复算；口径卡输入指向的查询快照也随文件附带。 +- 报告引用的分析结论（含分析范围里点名的）按原字节随文件附带，连同它的输入快照和下钻的上一层结论；导出时服务端按封存的输入复算一遍，结果记在结论的条目上。浏览器里只核哈希、展示服务端的复算结果，不在页面里重算统计。任一份输入快照证明不了没碰遮罩列的，结论也只给 id。 - 它只证明文件内部一致：数字、文档、运行记录彼此对得上，而且没被改过。真伪要拿页面上的封存哈希和系统里这次运行的封存核对（`GET /api/runs/{id}/verify`）对照，页面上醒目地写着这一点。 - 封存之后追加的事件（按需裁判、审批质询这类批注）单列，只有类型、序号和内容摘要，不进清单。 - 数据源设了遮罩时，证明不了没碰遮罩列的查询快照、有输入取自遮罩列的口径卡只给 id，不附内容。遮罩只减少暴露，不是安全边界。 @@ -246,4 +259,14 @@ flowchart LR - 口径与数据都变、又能严格拆分的，分别给出口径效应和数据效应，两者之和等于总变化。 - 只数据变、只参数变、口径与数据都变的指标，能归因时按维度（比如区域）列出变化最大的前 10 项。每一项都追得到两边的查询快照；核查执行的查询不参与对比。 - 遮罩列的值不出现在结果里，用到遮罩列的指标不拆分。 +- 图里有分析卡的，两次运行的分析卡按节点和项对齐，列出两次的状态、判定、方向和判定语；循环里执行了几次的取最后一次。 - 定时任务的运行成功结束、封存后，自动和同一任务上一次成功的运行比一次；触发历史和通知里只给基准是哪一次、各类指标各几个，不给具体数字。 + +### 分析说法核对 + +报告里的判断词——「主要来自华东」「超出常规波动」「持续上升」「差异站得住」——要挂对应的分析结论、判定对得上；写原因要有知识库原话或运行输入作依据，否则写成待核实的推测。 + +- 由报告撰写节点在写作时逐句核对：确定的词表加绑定，不调用模型，不花钱。引用模式的出口独立再核一遍。 +- 缺省重写一次、仍有的标在报告里（证据面板、审计、导出都看得到），不影响出具档位；设成严格时按最多重写次数重写，仍有的计入出具缺口、出具降档。受管级别要求严格（门禁 G3）。 +- 只核带这一项记录的新报告文档，升级前的报告复核结果不变。 +- 判断词分哪几类、各要挂哪种分析、哪些写法不核，见[分析说法核对](analysis.md#分析说法核对)。 diff --git a/docs/zh-CN/tools.md b/docs/zh-CN/tools.md index f668d38c..5e05328c 100644 --- a/docs/zh-CN/tools.md +++ b/docs/zh-CN/tools.md @@ -2,17 +2,27 @@ # 工具、MCP 与能力登记 -这篇讲工作流能调用的工具：内置工具、自定义工具和 MCP，它们的审批档位，管理员怎么在能力登记里管住它们，以及怎么登记外部系统的端点。给搭建者和管理员看。 +这篇讲工作流能调用的工具：内置工具（含分析工具）、自定义工具和 MCP，它们的审批档位，管理员怎么在能力登记里管住它们，以及怎么登记外部系统的端点。给搭建者和管理员看。 ## 工具种类 -- **内置工具**：15 个。 +- **内置工具**：23 个，其中 5 个是分析工具（见下文）。 - **自定义工具**：HTTP 模板，或沙箱里的 Python。 - **MCP**：接入 MCP server（stdio 或 http）。 - **数据源工具**：每个数据源的查询工具 `db_query__<源>`，以及运行时查结构的 `db_schema__`（见[数据源](data-sources.md)）。 谁能配：MCP 的 stdio、端点和凭据、能力治理只有管理员能改，搭建者只读，使用者看不到（见[权限表](identity-and-security.md#权限表)）。 +## 分析工具 + +五个分析工具是平台的确定性分析：`analysis_decompose`（变化拆解）、`analysis_baseline`（波动基线）、`analysis_trend`（趋势判断）、`analysis_differ`（差异检验）、`analysis_comparable`（可比性检查）。它们不查库，只分析绑了它的 agent 这次执行里查过的结果，判定由平台按固定的方法和阈值算，判定语由系统写（见[Agent 的分析工具](analysis.md#agent-的分析工具)）。 + +- 参数里没有阈值。绑了任一个，平台就把内置的分析方法注入 agent 的系统提示，同一份只注入一次。 +- 只读、不写任何东西，不算危险工具：缺省的审批策略下不用等审批；使用者的问数据里也能用。 +- 只能绑给 agent：放进「调用工具」节点时画布校验报错，模板里要固定做的分析用分析卡（见[分析卡](analysis.md#分析卡)）。 +- 每个 Agent 每次执行最多分析 12 次（「设置 → 偏好设置 → 分析」里改）。 +- 它们不连数据源：读的是这次执行里已经查回来的快照，交回的是分析结论，不是查询结果。 + ## 审批档位 平台看不出 MCP 和自定义工具做什么，所以它们默认每次调用都等审批。审批档位有三档：`ask`、`gated`、`always`。 diff --git a/docs/zh-CN/upgrade-notes.md b/docs/zh-CN/upgrade-notes.md index e0e02b4a..eced8ed4 100644 --- a/docs/zh-CN/upgrade-notes.md +++ b/docs/zh-CN/upgrade-notes.md @@ -4,6 +4,22 @@ 这篇按时间倒序列出各次更新里，对使用者和运维有影响的变化，以及升级时要做的事。给运维和管理员看。功能本身的说明在各主题文档里。 +## 2026-10 分析能力一轮 + +**要做的事**：没有必须的操作，数据库表结构没有变化。要按受管级别重新发布的报告模板，报告撰写节点的「分析说法核对」要设为严格，发布前检查会提示，也能一键修复。 + +- 新增分析能力（见[分析能力](analysis.md)）：变化拆解、波动基线、趋势判断、差异检验、可比性检查五种确定性分析，判定由系统算、判定语由系统写，报告用 `[[a:A1]]` 引用，点得开、能复算。 +- 新节点「分析卡」（`analysis`）和五个分析工具，内置工具从 18 个变成 23 个。新模板「⑪ 异动分析（可追溯）」，升级后第一次启动自动加进模板列表。模板按名字只装一次：已有部署里的「⑨ 可追溯周报」不会跟着更新，按受管级别发布时照发布前检查的提示补上分析说法核对。 +- 报告撰写节点缺省核对分析说法（「重写一次后标出」）：「主要来自」「异常」「持续上升」这类判断词没挂对分析结论、原因没有依据的句子，交回写作者重写一次，仍有的标在报告里，不影响出具档位。写作规则多了一段讲判断词的话，图里没有分析的报告也一样：没有分析结论时，写作者被告知不下这类判断、只陈述数字和变化。不想核的，把报告撰写节点的「分析说法核对」设成不核对。 +- 受管级别要求「分析说法核对」设为严格（门禁 G3 多一条）。已经发布的版本照常运行，按缺省的「重写一次后标出」核；下次发布时按新规则检查。 +- 用到分析结论的报告末尾多一段系统写的「分析范围」，不进正文。报告文档多记一项分析说法按哪种处置、哪一版说法表核：只有带它的新文档在复核时核这一项，升级前的报告复核结果不变。 +- 引用模式的出口收齐报告引用的指标：输入追不到原始数据的，受管级别的正式运行记缺口、出具降档；别的级别照旧只写进出具声明。 +- 口径卡里用到两份只差日期的查询的指标，自动做一次可比性检查：带契约的出口里，不可比的记缺口降档，有条件可比的只标注。 +- 报告导出、可验证报告带上分析结论和它的输入快照，按封存的输入复算核对。 +- 设置多了「偏好设置 → 分析」一节（见[分析设置](configuration.md#分析设置)）。只在图里有分析卡、或者 agent 绑了分析工具时，发起运行才把这一组快照进运行，别的运行的发起记录一字不变。 +- 问数据：助手建图多了分析类问题的形状，自查时提醒查库的 agent 绑分析工具（只提醒一轮）。复核把几次没做成的分析并成一条「有 n 次分析没做成」，只写说明、不自动重来；分析调用写错了、后来改对的不再记成缺口，一直没改对的也不再触发整轮重来。 +- 评测多了贡献、判定两种期望（由期望 SQL 和同一套分析算出，不许人填），评分规则升到第 3 版，老评测可以按新规则重评；助手提示词的指纹包含了分析方法，和升级前的评测比会标「提示词变了」（见[回归评测](evaluation-and-ledger.md#回归评测)）。 + ## 2026-10 编排完善第 2 波：PostgreSQL 与多进程 **要做的事**：多进程部署升级前，先停掉全部工作进程；代码节点要用的值改成显式传入；别删数据目录里的 `.agentlab-data`。 diff --git a/docs/zh-CN/workflows.md b/docs/zh-CN/workflows.md index 297dc551..669effcd 100644 --- a/docs/zh-CN/workflows.md +++ b/docs/zh-CN/workflows.md @@ -40,7 +40,7 @@ ### 内置模板 -内置 9 个模板，从「最小问答」到「可追溯周报」。它们可以直接运行，也是各类节点用法的可执行文档。 +内置 11 个模板，从「最小问答」到「异动分析（可追溯）」。它们可以直接运行，也是各类节点用法的可执行文档。「⑨ 可追溯周报」和「⑪ 异动分析（可追溯）」拿来就能按受管级别发布：前者示范口径卡和按引用出具，后者示范分析卡——两次查询取数，分析卡判这一周的变化在不在常规波动范围内、主要来自哪个地区，报告只引用判定语（见[分析能力](analysis.md#分析卡)）。 ## 节点类型 @@ -51,9 +51,9 @@ | 执行 | `tool` `code` `merge` | 直接调用工具 / 沙箱内执行代码 / 合并几次查询的结果（跨库按键对齐） | | 控制 | `branch` `loop` `subgraph` | 条件或语义分支 / 遍历与条件循环 / 嵌套工作流 | | 上下文 | `memory` `retrieve` `transform` | 长期记忆读写 / 知识检索 / 数据整形 | -| 把关 | `human` `validate` `metrics` | 人工介入 / Schema 校验与自动返工 / 口径卡 | +| 把关 | `human` `validate` `metrics` `analysis` | 人工介入 / Schema 校验与自动返工 / 口径卡 / 分析卡（对上游查询结果固定做几项确定性分析） | -`output` 节点按 JSON Schema 校验结构化成果，不合格时把错误交回模型自动返工。`metrics`（口径卡）和 `report`（报告撰写）见[证据与出具](evidence.md)；`merge`（合并查询）见[数据源](data-sources.md#跨库合并)。 +`output` 节点按 JSON Schema 校验结构化成果，不合格时把错误交回模型自动返工。`metrics`（口径卡）和 `report`（报告撰写）见[证据与出具](evidence.md)；`analysis`（分析卡）见[分析能力](analysis.md#分析卡)；`merge`（合并查询）见[数据源](data-sources.md#跨库合并)。 ## 节点之间怎么传数据 @@ -125,7 +125,7 @@ ## Agent 节点 -`agent` 节点是一个带工具的模型循环，能用哪些工具由节点上的白名单决定（见[工具](tools.md)）。配了 `output_schema` 的 agent 可以打开 `cite_fields`，让每个字段标出取自哪次查询的哪一格（见[证据与出具](evidence.md#取数链路)）。 +`agent` 节点是一个带工具的模型循环，能用哪些工具由节点上的白名单决定（见[工具](tools.md)）。配了 `output_schema` 的 agent 可以打开 `cite_fields`，让每个字段标出取自哪次查询的哪一格（见[证据与出具](evidence.md#取数链路)）。查库的 agent 可以再绑分析工具：拆变化、判异常、判趋势由平台按固定的方法和阈值算，判定语由系统写，报告引用它（见[Agent 的分析工具](analysis.md#agent-的分析工具)）。 ### 护栏 @@ -186,7 +186,7 @@ Excel / CSV 作为数据源上传，会变成可以用 SQL 查的表；放进知 - **G1** 带契约的出口，成果字段的文字只能来自报告撰写节点。 - **G2** 模型写的文字不绕过报告撰写节点直接流进出口。 -- **G3** 报告撰写节点写明 `numbers: strict`、`on_violation: fail`、`claims: require_citation`（或 `judge`，并写预算）。 +- **G3** 报告撰写节点写明 `numbers: strict`、`on_violation: fail`、`claims: require_citation`（或 `judge`，并写预算）、`analysis_claims: strict`（分析说法核对设为严格，见[分析说法核对](analysis.md#分析说法核对)）。 - **G4** 口径卡的输入不来自计算角色的沙箱代码。 - **G5** 给口径卡供数的 agent 配 `output_schema` 并开 `cite_fields`。 diff --git a/scripts/copy-allow.json b/scripts/copy-allow.json index ca847547..f6339f03 100644 --- a/scripts/copy-allow.json +++ b/scripts/copy-allow.json @@ -55,6 +55,12 @@ "rule": "半角标点", "reason": "模板里「写代码」节点的系统提示（写给模型），print(json.dumps(...)) 是代码示例，半角括号和省略号是 Python 语法" }, + { + "file": "backend/app/seed.py", + "text": "AS \"周\"", + "rule": "半角标点", + "reason": "模板 ⑪「异动分析（可追溯）」两个取数节点的 SQL：结果列按节点手册的规矩起中文别名、SQLite 的写法加双引号（AS \"周\"），引号和逗号是 SQL 语法；中文列名让分析卡的判定语写「分散在各地区」、报告的表头读得懂" + }, { "file": "backend/app/api/copilot.py", "text": "报告按 [[i:字段名]]",