mirror of
https://github.com/ThearchyHelios/AgentLab.git
synced 2026-11-15 00:01:58 +00:00
文档:新增 analysis 一篇(中英各一,标题层级一致)——分析能力是什么、为什么(判定由系统算、判定语由系统写、照实说、留痕)、五种分析各回答什么问题与判定标准(括号里是缺省阈值)、算不了的情形与调用写错、期次的写法、两个入口(Agent 的分析工具、分析卡)、引用写法、分析说法核对(判断词的类、check / strict / off、来源与因果的分界)、分析范围、设置、限制。更新:证据与出具(按引用核对的血缘口子与分析说法复核、口径卡指标的两期可比性、引用标记加 a: 与 v:A / table:A、点开一个分析结论看到什么、两种导出认分析结论、运行对比按分析卡对齐、复核套件加分析说法核对)、工作流(内置模板 11 个、节点类型表加分析卡、G3 加分析说法核对、agent 可绑分析工具)、问数据(分析类问题怎么走、复核只写说明的几种缺口、分析结论的数不标口径、自查提醒)、回归评测与口径账本(贡献、判定两种期望与评分,细节待评测包交付后校对;分析结论的数不比对口径)、配置(偏好设置里的分析一节,17 项的叫法、缺省、范围)、工具(内置工具 23 个、分析工具一节;原来写的 15 个早已对不上)、升级说明(本轮);README 中英加一条主要功能、文档索引加 analysis,节点 19 种、模板 11 个。development.md 的检查项数没动,等前端包的数
This commit is contained in:
@@ -10,7 +10,7 @@ English | [Simplified Chinese](README.zh-CN.md)
|
|||||||
|
|
||||||
## Why use it
|
## Why use it
|
||||||
|
|
||||||
- **Numbers you can check**: all arithmetic happens in SQL and in caliber cards (controlled metric sets), and the narrative layer has no right to create numbers. The report-writer node writes only citation markers, the system fills in the numbers from snapshots, and every number opens all the way to the cell in the query result it came from.
|
- **Numbers you can check**: all arithmetic happens in SQL and in caliber cards (controlled metric sets), and the narrative layer has no right to create numbers. The report-writer node writes only citation markers, the system fills in the numbers from snapshots, and every number opens all the way to the cell in the query result it came from. Judgments such as "mainly from East" or "outside the usual fluctuation" are likewise computed by the system with fixed methods and written as verdict phrases, and the report only cites them.
|
||||||
- **Constrained orchestration**: the engine is compiled, so at run time the model can't change the graph structure; its freedom is confined to the inside of agent nodes and to branch decisions. Fully dynamic multi-agent collaboration stays in the exploration layer.
|
- **Constrained orchestration**: the engine is compiled, so at run time the model can't change the graph structure; its freedom is confined to the inside of agent nodes and to branch decisions. Fully dynamic multi-agent collaboration stays in the exploration layer.
|
||||||
- **Pinned, reproducible versions**: formal runs accept only published immutable versions (pinned by graph hash). The complete evidence of every execution goes into a content-addressed artifact store and is sealed, and one click verifies that nothing was altered.
|
- **Pinned, reproducible versions**: formal runs accept only published immutable versions (pinned by graph hash). The complete evidence of every execution goes into a content-addressed artifact store and is sealed, and one click verifies that nothing was altered.
|
||||||
- **Human control**: approvals, joint sign-off, time limits, and Reject exits; nothing is ever approved automatically, and a process restart loses no state.
|
- **Human control**: approvals, joint sign-off, time limits, and Reject exits; nothing is ever approved automatically, and a process restart loses no state.
|
||||||
@@ -20,7 +20,7 @@ English | [Simplified Chinese](README.zh-CN.md)
|
|||||||
|
|
||||||
**Orchestration and runs**
|
**Orchestration and runs**
|
||||||
|
|
||||||
- Canvas: 18 node types, drag and connect; live node highlighting, streaming output, and a timeline track at run time → [Workflows](docs/en/workflows.md)
|
- Canvas: 19 node types, drag and connect; live node highlighting, streaming output, and a timeline track at run time → [Workflows](docs/en/workflows.md)
|
||||||
- Copilot: generate or rewrite a whole graph from one sentence; it looks up schemas on demand and checks its work before delivering → [Copilot](docs/en/ask-data.md#copilot)
|
- Copilot: generate or rewrite a whole graph from one sentence; it looks up schemas on demand and checks its work before delivering → [Copilot](docs/en/ask-data.md#copilot)
|
||||||
- Human-in-the-loop: approvals, joint sign-off, time limits, Reject exits → [Human nodes](docs/en/workflows.md#human-nodes)
|
- Human-in-the-loop: approvals, joint sign-off, time limits, Reject exits → [Human nodes](docs/en/workflows.md#human-nodes)
|
||||||
- Agent guardrails: repeated calls, spending, and context each have a guard; the step count is only a backstop → [Guardrails](docs/en/workflows.md#guardrails)
|
- Agent guardrails: repeated calls, spending, and context each have a guard; the step count is only a backstop → [Guardrails](docs/en/workflows.md#guardrails)
|
||||||
@@ -41,6 +41,7 @@ English | [Simplified Chinese](README.zh-CN.md)
|
|||||||
**Evidence and issuance**
|
**Evidence and issuance**
|
||||||
|
|
||||||
- Clickable evidence: numbers, table and field names, and verbatim knowledge-base quotes all open to their provenance → [Evidence and Issuance](docs/en/evidence.md)
|
- Clickable evidence: numbers, table and field names, and verbatim knowledge-base quotes all open to their provenance → [Evidence and Issuance](docs/en/evidence.md)
|
||||||
|
- Analytical judgments: change decomposition, fluctuation baselines, trends, difference tests, and comparability checks are computed by the system, which also writes the verdict phrases; reports cite them, and judgment words are checked sentence by sentence → [Analysis](docs/en/analysis.md)
|
||||||
- Three-tier issuance: caliber cards plus issuance contracts, checked by value or by citation → [Issuance contracts](docs/en/evidence.md#issuance-contracts-and-tiers)
|
- Three-tier issuance: caliber cards plus issuance contracts, checked by value or by citation → [Issuance contracts](docs/en/evidence.md#issuance-contracts-and-tiers)
|
||||||
- Sealing and exports: one-click seal verification; self-contained report exports and verifiable reports → [Exports](docs/en/evidence.md#exports)
|
- Sealing and exports: one-click seal verification; self-contained report exports and verifiable reports → [Exports](docs/en/evidence.md#exports)
|
||||||
- Review suite: double-blind recomputation, adversarial review, approval inquiry, the what-if sandbox, run comparison → [Review suite](docs/en/evidence.md#the-review-suite)
|
- Review suite: double-blind recomputation, adversarial review, approval inquiry, the what-if sandbox, run comparison → [Review suite](docs/en/evidence.md#the-review-suite)
|
||||||
@@ -71,7 +72,7 @@ conda env create -f environment.yml # first time: create the agentlab environm
|
|||||||
./scripts/dev.sh # after that: one command starts the frontend and backend
|
./scripts/dev.sh # after that: one command starts the frontend and backend
|
||||||
```
|
```
|
||||||
|
|
||||||
Open http://localhost:5273 . You can try everything without an API key: the built-in Mock provider runs the whole orchestration path end to end; to connect a real model, see [Model access](docs/en/configuration.md#model-access). The 9 built-in templates run as they are.
|
Open http://localhost:5273 . You can try everything without an API key: the built-in Mock provider runs the whole orchestration path end to end; to connect a real model, see [Model access](docs/en/configuration.md#model-access). The 11 built-in templates run as they are.
|
||||||
|
|
||||||
**Login is off by default**: anyone on the same network who can open the page can use every feature. Turn on login before multiple people use it; see [Login modes](docs/en/identity-and-security.md#login-modes). For more startup options, see [Development](docs/en/development.md#start).
|
**Login is off by default**: anyone on the same network who can open the page can use every feature. Turn on login before multiple people use it; see [Login modes](docs/en/identity-and-security.md#login-modes). For more startup options, see [Development](docs/en/development.md#start).
|
||||||
|
|
||||||
@@ -90,6 +91,7 @@ See [Deployment](docs/en/deployment.md) and [deploy/README.md](deploy/README.md)
|
|||||||
| [Ask Data and Copilot](docs/en/ask-data.md) | What happens after you ask, output review, record retention, how Copilot builds graphs and looks things up |
|
| [Ask Data and Copilot](docs/en/ask-data.md) | What happens after you ask, output review, record retention, how Copilot builds graphs and looks things up |
|
||||||
| [Workflows](docs/en/workflows.md) | The canvas, node types, passing data, human nodes and rejection, agent guardrails, subworkflows, publishing and gates, runs and resuming |
|
| [Workflows](docs/en/workflows.md) | The canvas, node types, passing data, human nodes and rejection, agent guardrails, subworkflows, publishing and gates, runs and resuming |
|
||||||
| [Evidence and Issuance](docs/en/evidence.md) | Caliber cards, issuance contracts and tiers, citation markers, claim sentences, no data in the queried range, sealing and verification, report exports and verifiable reports, the review suite |
|
| [Evidence and Issuance](docs/en/evidence.md) | Caliber cards, issuance contracts and tiers, citation markers, claim sentences, no data in the queried range, sealing and verification, report exports and verifiable reports, the review suite |
|
||||||
|
| [Analysis](docs/en/analysis.md) | The five analyses and their criteria, when an analysis can't be done, analysis tools for agents and analysis cards, how to cite, the analysis claim check, the analysis scope, settings and limits |
|
||||||
| [Data Sources and Spreadsheet Import](docs/en/data-sources.md) | Connecting databases, read-only guarantees, uploading spreadsheets and recipes, accumulating by period, cross-database merges, masked columns |
|
| [Data Sources and Spreadsheet Import](docs/en/data-sources.md) | Connecting databases, read-only guarantees, uploading spreadsheets and recipes, accumulating by period, cross-database merges, masked columns |
|
||||||
| [Data Catalog](docs/en/data-catalog.md) | What the catalog records, drafting and review, export and import, data profiling, SQL checks, introspection scope for large databases, schema and value changes |
|
| [Data Catalog](docs/en/data-catalog.md) | What the catalog records, drafting and review, export and import, data profiling, SQL checks, introspection scope for large databases, schema and value changes |
|
||||||
| [Tools, MCP, and the Capability Registry](docs/en/tools.md) | Kinds of tools, approval levels, capability governance, limits on write capabilities, registered endpoints, network rules |
|
| [Tools, MCP, and the Capability Registry](docs/en/tools.md) | Kinds of tools, approval levels, capability governance, limits on write capabilities, registered endpoints, network rules |
|
||||||
|
|||||||
+5
-3
@@ -10,7 +10,7 @@
|
|||||||
|
|
||||||
## 为什么用它
|
## 为什么用它
|
||||||
|
|
||||||
- **数字可核对**:所有算术都在 SQL 和口径卡(受控指标集)里完成,叙述层无权创造数字。报告撰写节点只写引用标记,系统照快照填数,每个数一路点得开到查询结果里的那一格。
|
- **数字可核对**:所有算术都在 SQL 和口径卡(受控指标集)里完成,叙述层无权创造数字。报告撰写节点只写引用标记,系统照快照填数,每个数一路点得开到查询结果里的那一格。「主要来自华东」「超出常规波动」这类判断也由系统按固定的方法算、写成判定语,报告只引用。
|
||||||
- **受限编排**:引擎是编译式的,运行时模型改不了图结构,它的自由限在 agent 节点内部和分支决策里;全动态的多 agent 协作只留在探索层。
|
- **受限编排**:引擎是编译式的,运行时模型改不了图结构,它的自由限在 agent 节点内部和分支决策里;全动态的多 agent 协作只留在探索层。
|
||||||
- **版本固定、可复现**:正式运行只认已发布的不可变版本(按图哈希固定)。每次执行的完整证据落进内容寻址的工件库并封存,一键核对有没有被改过。
|
- **版本固定、可复现**:正式运行只认已发布的不可变版本(按图哈希固定)。每次执行的完整证据落进内容寻址的工件库并封存,一键核对有没有被改过。
|
||||||
- **人工把关**:审批、会签、时限、驳回出口;永远不会自动批准,进程重启不丢状态。
|
- **人工把关**:审批、会签、时限、驳回出口;永远不会自动批准,进程重启不丢状态。
|
||||||
@@ -20,7 +20,7 @@
|
|||||||
|
|
||||||
**编排与运行**
|
**编排与运行**
|
||||||
|
|
||||||
- 画布:18 种节点拖拽连线,运行时逐节点高亮、流式输出、航迹时间轴 → [工作流](docs/zh-CN/workflows.md)
|
- 画布:19 种节点拖拽连线,运行时逐节点高亮、流式输出、航迹时间轴 → [工作流](docs/zh-CN/workflows.md)
|
||||||
- 助手:一句话生成或改写整张图,按需查看表结构,交付前自查 → [助手](docs/zh-CN/ask-data.md#助手)
|
- 助手:一句话生成或改写整张图,按需查看表结构,交付前自查 → [助手](docs/zh-CN/ask-data.md#助手)
|
||||||
- 人工介入:审批、会签、时限、驳回出口 → [人工节点](docs/zh-CN/workflows.md#人工节点)
|
- 人工介入:审批、会签、时限、驳回出口 → [人工节点](docs/zh-CN/workflows.md#人工节点)
|
||||||
- Agent 护栏:重复调用、花费、上下文都有挡法,步数只留兜底 → [护栏](docs/zh-CN/workflows.md#护栏)
|
- Agent 护栏:重复调用、花费、上下文都有挡法,步数只留兜底 → [护栏](docs/zh-CN/workflows.md#护栏)
|
||||||
@@ -41,6 +41,7 @@
|
|||||||
**证据与出具**
|
**证据与出具**
|
||||||
|
|
||||||
- 可点击证据:数字、表名字段名、知识库原话都点得开出处 → [证据与出具](docs/zh-CN/evidence.md)
|
- 可点击证据:数字、表名字段名、知识库原话都点得开出处 → [证据与出具](docs/zh-CN/evidence.md)
|
||||||
|
- 分析判断:变化拆解、波动基线、趋势、差异检验、可比性检查由系统算,判定语由系统写,报告引用、判断词逐句核对 → [分析能力](docs/zh-CN/analysis.md)
|
||||||
- 三档出具:口径卡加出具契约,按数值回指或按引用核对 → [出具契约](docs/zh-CN/evidence.md#出具契约与三档)
|
- 三档出具:口径卡加出具契约,按数值回指或按引用核对 → [出具契约](docs/zh-CN/evidence.md#出具契约与三档)
|
||||||
- 封存与导出:一键核对封存;自包含的报告导出和可验证报告 → [导出](docs/zh-CN/evidence.md#导出)
|
- 封存与导出:一键核对封存;自包含的报告导出和可验证报告 → [导出](docs/zh-CN/evidence.md#导出)
|
||||||
- 复核套件:双盲复算、反方审稿、审批质询、数字推演、运行对比 → [复核套件](docs/zh-CN/evidence.md#复核套件)
|
- 复核套件:双盲复算、反方审稿、审批质询、数字推演、运行对比 → [复核套件](docs/zh-CN/evidence.md#复核套件)
|
||||||
@@ -71,7 +72,7 @@ conda env create -f environment.yml # 首次:创建 agentlab 环境并安装
|
|||||||
./scripts/dev.sh # 之后:一条命令启动前后端
|
./scripts/dev.sh # 之后:一条命令启动前后端
|
||||||
```
|
```
|
||||||
|
|
||||||
打开 http://localhost:5273 。不配 API Key 也能完整体验:内置的 Mock 会把编排链路完整走一遍;接入真实模型见[模型接入](docs/zh-CN/configuration.md#模型接入)。内置 9 个模板可以直接运行。
|
打开 http://localhost:5273 。不配 API Key 也能完整体验:内置的 Mock 会把编排链路完整走一遍;接入真实模型见[模型接入](docs/zh-CN/configuration.md#模型接入)。内置 11 个模板可以直接运行。
|
||||||
|
|
||||||
**默认不开登录**:同一网络里打得开页面的人都能用全部功能。多人使用前先开登录,见[登录方式](docs/zh-CN/identity-and-security.md#登录方式)。更多启动选项见[开发](docs/zh-CN/development.md#启动)。
|
**默认不开登录**:同一网络里打得开页面的人都能用全部功能。多人使用前先开登录,见[登录方式](docs/zh-CN/identity-and-security.md#登录方式)。更多启动选项见[开发](docs/zh-CN/development.md#启动)。
|
||||||
|
|
||||||
@@ -90,6 +91,7 @@ conda env create -f environment.yml # 首次:创建 agentlab 环境并安装
|
|||||||
| [问数据与助手](docs/zh-CN/ask-data.md) | 提问之后发生什么、产出复核、记录保留,助手怎么生成图、怎么按需查看 |
|
| [问数据与助手](docs/zh-CN/ask-data.md) | 提问之后发生什么、产出复核、记录保留,助手怎么生成图、怎么按需查看 |
|
||||||
| [工作流](docs/zh-CN/workflows.md) | 画布、节点类型、传数据、人工节点与驳回、agent 护栏、子工作流、发布与门禁、运行与断点续跑 |
|
| [工作流](docs/zh-CN/workflows.md) | 画布、节点类型、传数据、人工节点与驳回、agent 护栏、子工作流、发布与门禁、运行与断点续跑 |
|
||||||
| [证据与出具](docs/zh-CN/evidence.md) | 口径卡、出具契约与三档、引用标记、结论句、查询区间里没有数据、封存与核对、报告导出与可验证报告、复核套件 |
|
| [证据与出具](docs/zh-CN/evidence.md) | 口径卡、出具契约与三档、引用标记、结论句、查询区间里没有数据、封存与核对、报告导出与可验证报告、复核套件 |
|
||||||
|
| [分析能力](docs/zh-CN/analysis.md) | 五种分析与判定标准、算不了的时候、Agent 的分析工具与分析卡、引用写法、分析说法核对、分析范围、设置与限制 |
|
||||||
| [数据源与表格导入](docs/zh-CN/data-sources.md) | 接入数据库、只读保证、上传表格与取数配方、按期累积、跨库合并、遮罩列 |
|
| [数据源与表格导入](docs/zh-CN/data-sources.md) | 接入数据库、只读保证、上传表格与取数配方、按期累积、跨库合并、遮罩列 |
|
||||||
| [数据目录](docs/zh-CN/data-catalog.md) | 目录记什么、起草与审阅、导出导入、数据剖析、SQL 检查、大库探查范围、结构与取值变化 |
|
| [数据目录](docs/zh-CN/data-catalog.md) | 目录记什么、起草与审阅、导出导入、数据剖析、SQL 检查、大库探查范围、结构与取值变化 |
|
||||||
| [工具、MCP 与能力登记](docs/zh-CN/tools.md) | 工具种类、审批档位、能力治理、写入能力的限制、登记端点、网络规则 |
|
| [工具、MCP 与能力登记](docs/zh-CN/tools.md) | 工具种类、审批档位、能力治理、写入能力的限制、登记端点、网络规则 |
|
||||||
|
|||||||
@@ -0,0 +1,204 @@
|
|||||||
|
English | [Simplified Chinese](../zh-CN/analysis.md)
|
||||||
|
|
||||||
|
# Analysis
|
||||||
|
|
||||||
|
This page covers where the analytical judgments in a report come from: how statements such as "mainly from East", "outside the usual fluctuation", "rising steadily", or "the difference holds up" are computed by the system, written into reports, and checked. It covers what each of the five analyses answers, the criteria it judges by, what it says when it can't be done, the two entry points (analysis tools for agents and analysis cards), how reports cite conclusions, the check on judgment words, the Analysis Scope section at the end of a report, and the settings and limits. It is for builders and for people who read reports. The user interface is currently available in Simplified Chinese only.
|
||||||
|
|
||||||
|
## What it is and why
|
||||||
|
|
||||||
|
The system fills the numbers in a report from snapshots, and the narrative layer does no arithmetic (see [Evidence and Issuance](evidence.md#the-basic-idea)). But a report says more than numbers; it also makes judgments: where a change mainly came from, whether this week counts as unusual, whether something keeps rising, whether a gap between two groups is real. These judgments used to be made by the writing model on its own; it could pick its evidence, and readers had no way to check.
|
||||||
|
|
||||||
|
Analysis brings judgments back into the system as well:
|
||||||
|
|
||||||
|
- **The system computes the verdicts**: the five analyses are deterministic platform code. They read only query snapshots returned during this run and compute verdicts with fixed methods and thresholds. The model decides only what to compute (which two periods to compare, which dimension to break down, which segment to drill into); it can't change the method or the thresholds.
|
||||||
|
- **The system writes the verdict phrases**: every analysis conclusion carries a verdict phrase written by the system from a template, and reports cite it (`[[a:A1]]`); like a number, it opens to its provenance. When the writer writes judgment words such as "mainly from" or "unusual" itself, the system checks sentence by sentence that the sentence cites a matching analysis conclusion and agrees with its verdict (see [Analysis claim check](#analysis-claim-check)).
|
||||||
|
- **Saying it as it is**: when an analysis can't be done, it says why (not enough history, the current period isn't over, the query was truncated, the two periods aren't comparable, too few samples) instead of forcing a conclusion. "Can't be done" is a conclusion too, and the writer reports it as such.
|
||||||
|
- **Leaving a trace**: every analysis conclusion, together with its method and version, parameters, the thresholds actually used, the input cells, the table ranges checked live, the results, and the verdict, is stored as a content-addressed artifact, recorded in the evidence ledger, and sealed with the run. The evidence panel, the report export, and the verifiable report all recompute it with the same function (see [What you see when you click a number](evidence.md#what-you-see-when-you-click-a-number)).
|
||||||
|
|
||||||
|
A breakdown shows only where a change came from, not why. A cause needs a verbatim knowledge-base quote or a run input as its basis; otherwise it is written as a conjecture to be verified.
|
||||||
|
|
||||||
|
## The five analyses
|
||||||
|
|
||||||
|
Each analysis produces one analysis conclusion, numbered A1, A2, … in order in the writing catalog (the same numbering scheme as Q1 for queries and K1 for knowledge-base chunks). A conclusion has a verdict (verdict code, the segments it names, a direction), a verdict phrase, a criterion (one sentence stating the thresholds actually used), named values (shares, the total change, and so on), a result table, and caveats (offsetting, Simpson's paradox, small samples, a record high, and so on; at most two are appended to the verdict phrase).
|
||||||
|
|
||||||
|
The thresholds in parentheses below are defaults, changed under Settings → Preferences → Analysis (see [Analysis settings](configuration.md#analysis-settings)).
|
||||||
|
|
||||||
|
### Change decomposition
|
||||||
|
|
||||||
|
Answers "where did the change come from". Three modes:
|
||||||
|
|
||||||
|
| Mode | Used for | How it breaks down |
|
||||||
|
|---|---|---|
|
||||||
|
| Additive | Quantities that add up, such as sales or order counts | By dimension (one or two columns), each segment's contribution to the total change and its share |
|
||||||
|
| Ratio | Ratios such as refund rate or average order value | Into a mix effect (segment weights changed) and each segment's own change; the two add up exactly to the change in the ratio |
|
||||||
|
| Factor | Sales = orders × average order value | Into each factor's contribution, by logarithmic mean (LMDI) |
|
||||||
|
|
||||||
|
- Verdicts: one segment accounting for at least half of the total change (0.5) is the main source ("mainly from East"); without such a segment, the top few segments together accounting for at least 80% (0.8) count as concentrated (at most a fifth of all segments, and at least 2); a total change under 0.5% of the base counts as essentially flat; anything else is dispersed. A ratio decomposition is judged as mainly from the mix, mainly from the segments' own changes, or both; a factor decomposition is judged by which factor it mainly comes from.
|
||||||
|
- When totals are given, the segments must add up to them (within a tolerance based on the precision the totals are written with) before the decomposition judges where the change came from. If they don't, it doesn't judge, and the verdict phrase states the gap. Without totals, the sum of the segments is the total, and the criterion says so.
|
||||||
|
- When the segments moving the other way add up to more than a quarter of the total change (0.25), a caveat says the changes partly offset each other. In a ratio decomposition, when most segments fall while the whole rises (or the other way round), a caveat explains Simpson's paradox. Segments with a denominator under 30 are marked as small and left out of that judgment.
|
||||||
|
- Drilling down: one segment of a decomposition can be broken down one level further (East by channel). The segments of that level must add up to the value of the segment above, at most 3 levels deep. The agent writes the drill-down query itself, querying only that segment, and the system checks that it closes against the value of the segment above.
|
||||||
|
- An example verdict phrase (translated): "Sales fell 194 (-9.2%), mainly from East (-120, 61.9% of the change)".
|
||||||
|
|
||||||
|
### Fluctuation baseline
|
||||||
|
|
||||||
|
Answers "is this change large, does it count as unusual": it puts the current period back into its own history.
|
||||||
|
|
||||||
|
- The input is one series ordered by period (day, week, month, quarter). By default it compares the rate of change against the previous period; daily data is compared with the same day last week, to avoid weekday effects. Stocks and ratios can compare the value itself. For year over year, compare with 52 weeks or 12 months earlier, with correspondingly longer history.
|
||||||
|
- It uses the median and the median absolute deviation of the history (robust standard deviation = 1.4826 × median absolute deviation): a current period within 2 robust standard deviations of the median is a usual fluctuation, 2 to 3 is outside, and 3 or more is far outside. It uses the median rather than the mean because extremes such as promotion weeks widen the mean and standard deviation, making normal weeks and unusual weeks alike look ordinary.
|
||||||
|
- History needs at least 8 periods and uses at most 52. When the current period is the highest or lowest in its history, a caveat says it is a record high or low.
|
||||||
|
- An example verdict phrase (translated): "Sales changed -15.8% this period, far outside the usual fluctuation of the past 24 weeks (-0.5% to +0.4%); this period's sales are the lowest of the past 24 weeks".
|
||||||
|
|
||||||
|
### Trend test
|
||||||
|
|
||||||
|
Answers "is it changing steadily, or was it a one-off".
|
||||||
|
|
||||||
|
- It takes the last contiguous stretch (at least 6 periods): when the two-sided p-value of the Mann–Kendall test is below 0.05, the sign of the Theil–Sen slope decides rising or falling; otherwise there is no clear trend. When the last 3 or more periods all move the same way, the verdict phrase adds "down 3 weeks in a row", or "but up the last 3 weeks in a row" when that runs against the trend.
|
||||||
|
- Mann–Kendall assumes the periods are independent, while business series are often autocorrelated, which makes p too small. So the verdict phrase says only "trending up", never "significantly up". Daily data gets a caveat about weekday effects suggesting weekly aggregation.
|
||||||
|
- An example verdict phrase (translated): "Online sales have trended up over the past 12 weeks (about +2.1% per week), up 4 weeks in a row".
|
||||||
|
|
||||||
|
### Difference test
|
||||||
|
|
||||||
|
Answers "are these two groups (or two periods of one group) really different, or is it sampling noise".
|
||||||
|
|
||||||
|
- For rates (conversion rate, refund rate), give the successes and the trials: each group gets a Wilson interval; when every cell's expected frequency is at least 5, it uses the pooled-proportion z-test, otherwise Fisher's exact test.
|
||||||
|
- For means, give each group's mean, standard deviation, and sample size, or its sum, sum of squares, and sample size (SQLite has no standard deviation function, so write it this way), and it uses Welch's t-test. You can also give the values row by row (at most 1000 rows, not truncated); the system computes the statistics itself and also reruns the test without the most distant value as a sensitivity check.
|
||||||
|
- A two-sided p-value below 0.05 means the difference holds up; several pairs in one call are corrected with Holm's method. A caveat about small samples is added when a group has fewer than 30 samples for means, or when Fisher's test was used for rates.
|
||||||
|
- Verdict phrases avoid terms such as "p-value" or "significance": a difference that holds up is described as such, and otherwise the phrase says the difference may be just sampling noise, with the sample sizes (230 and 40, say). P-values and intervals are in the result table and the evidence panel.
|
||||||
|
|
||||||
|
### Comparability check
|
||||||
|
|
||||||
|
Answers "can these two periods be compared directly". Each item passes, warns, fails, or can't be checked:
|
||||||
|
|
||||||
|
| Item | How it is judged |
|
||||||
|
|---|---|
|
||||||
|
| Same definition | The two queries differ only in their dates; when their structure differs, or they also filter something else (East versus South), it fails |
|
||||||
|
| Window length | Both periods have the same number of days; for an additive flow a mismatch fails, for ratios and stocks it only warns |
|
||||||
|
| Whole period | The current period is over and the table's data has arrived; for a flow, missing data fails ("the current period has only 3 days of data"), otherwise it warns |
|
||||||
|
| Weekday mix | For daily windows whose length isn't a multiple of 7, both periods have the same number of weekend days; otherwise it warns |
|
||||||
|
| Data problems | No data in the queried range fails; SQL check errors warn |
|
||||||
|
|
||||||
|
When everything passes (or can't be checked), the periods are comparable; with only warnings they are conditionally comparable; with any failure they aren't comparable. Items that can't be checked never make the periods incomparable, and the verdict phrase names them. The table's actual date range is checked live (read-only, time-limited, masked columns are never checked, falling back to the range recorded by profiling on failure); the range found is recorded in the conclusion, and recomputation uses the recorded values.
|
||||||
|
|
||||||
|
When a change decomposition is given two queries, it runs this check automatically first; when the fluctuation baseline, the trend test, or a change decomposition over a single query checks whether the current period is complete, it uses the whole-period item of this check. It also runs once automatically when a caliber card metric uses two queries of the same shape that differ only in their dates (this week versus last week), with the result attached to the metric (see [Caliber cards](evidence.md#caliber-cards)).
|
||||||
|
|
||||||
|
## When an analysis can't be done
|
||||||
|
|
||||||
|
An analysis that can't be done doesn't fail the agent or the node. It is recorded as a conclusion, with a verdict phrase along the lines of "Fluctuation baseline not done: only 5 periods of history, at least 8 needed", and it goes into the ledger and can be cited and checked:
|
||||||
|
|
||||||
|
| Reason | When |
|
||||||
|
|---|---|
|
||||||
|
| Truncated | The input query returned only its first rows, so the sum of the segments isn't the full sum |
|
||||||
|
| Too few segments | Fewer than 2 segments after the breakdown |
|
||||||
|
| History too short | Fewer valid periods of history than the minimum |
|
||||||
|
| Current period incomplete | The current period isn't over, data hasn't arrived, it isn't a whole week, or the query covered only part of it |
|
||||||
|
| Gap | A period is missing between the current period and its comparison period; the last periods used by the trend test aren't contiguous |
|
||||||
|
| Periods unreadable | The period column's format can't be read (times of day, mixed formats) |
|
||||||
|
| Zero base | The comparison period is 0, so no rate of change can be computed |
|
||||||
|
| Non-positive values | Factor decomposition requires every value to be positive |
|
||||||
|
| Not the same quantity | The two queries differ in structure, so they aren't two periods of the same quantity, and it doesn't break them down |
|
||||||
|
| No p-value | The difference test can't compute a p-value for any pair (each group has only 1 sample, or a sample size is 0) |
|
||||||
|
| Upper level not done, or missing from this data | In an analysis card, the level a drill-down sits under couldn't be done, or this period's data lacks a period, segment, group, or total cell that is needed |
|
||||||
|
|
||||||
|
A wrong call isn't recorded as a conclusion: a column that doesn't exist, a masked column used as a dimension or measure, a ratio treated as additive, several rows for the same segment and period, wrong parameters, or this agent having used up its analyses for this execution. The agent gets a sentence on how to fix the call and calls again; in an analysis card it is the template that is wrong, so the whole card fails and says which item.
|
||||||
|
|
||||||
|
Periods are read in these formats: dates, `YYYY-MM`, `YYYY-Www`, `YYYY-Qn`, and integer sequence numbers. For weekly aggregation, it is best to compute the period from a date column in the SQL, as the date of the first day of each week, or as an ISO week `YYYY-Www`. The same `2026-W42` lands on different days under SQLite's `%W`, MySQL's `%u`, and the ISO week; the system recognizes the scheme from the format string in the SQL and works out the real start and end. When the period is a ready-made week-number text column taken straight from the table, the scheme can't be recognized, and whether the current period is complete can't be checked.
|
||||||
|
|
||||||
|
## Two entry points
|
||||||
|
|
||||||
|
Both entry points share the same analyses, the same shape of conclusion, and the same table of verdict phrases.
|
||||||
|
|
||||||
|
### Analysis tools for agents
|
||||||
|
|
||||||
|
Agents in Ask Data and on the canvas can be given five analysis tools: `analysis_decompose` (change decomposition), `analysis_baseline` (fluctuation baseline), `analysis_trend` (trend test), `analysis_differ` (difference test), and `analysis_comparable` (comparability check). Use them when you only know which dimension to break down after seeing the numbers.
|
||||||
|
|
||||||
|
- The parameters cover only what to compute: which result this node queried (`Q3`, the same number as in the "Evidence Q3" handed back to the model), the role of each column, which two periods to compare, the two total cells, the level to drill down from, what to call the measure, and one sentence on what this call is testing (shown only in the evidence panel, never in the report body). There are no threshold parameters at all.
|
||||||
|
- Binding any one of them injects the platform's built-in analysis method into the agent's system prompt: first set the base period from the question, run the comparability check and the fluctuation baseline, then break down factors and mix, break down by dimension from coarse to fine, and drill down one level once the main source is found; a breakdown that finds no concentration is a conclusion too; make no causal claims; say plainly when something can't be done; finally, list the analyses done and their verdicts. The method is versioned with the code.
|
||||||
|
- What goes back to the agent is "Analysis A2" with the verdict, the verdict phrase, the key values, and caveats or the reason it couldn't be done; the full conclusion goes into the evidence ledger.
|
||||||
|
- Each agent can run at most 12 analyses per execution and drill down at most 3 levels.
|
||||||
|
- The analysis tools analyze only results that the agent they are bound to queried itself, and can't be put in a Call Tool node (for fixed analyses in a template, use an analysis card). Members of a collaboration team can use them too, but their conclusions don't go into the evidence ledger, so reports can't cite them.
|
||||||
|
- Users (the user role) can use them in Ask Data too: the analysis tools don't query the database; they read only query results already masked, and masked columns are never used as dimensions or measures.
|
||||||
|
|
||||||
|
### Analysis cards
|
||||||
|
|
||||||
|
An analysis card (the `analysis` node, a peer of the caliber card) runs a fixed set of analyses on the results of upstream query nodes. It is for analyses a template always does, such as the anomaly part of a weekly report: is this week's change within the usual fluctuation, and which region did it mainly come from.
|
||||||
|
|
||||||
|
- Each item states the analysis method and mode, which upstream query node to analyze, the parameters, and what the item is testing. Inputs can only be Call Tool nodes that call a database query tool, or Merge Query nodes, connected before the analysis card; an agent can't be an input, since it may have queried several times. The parameters are the same as the analysis tools', and `{{ input.x }}` can pick periods from run inputs.
|
||||||
|
- Thresholds aren't written in the items. When a template needs fixed thresholds, write them in the card's Threshold Overrides, pinned with the version; when they conflict with this run's analysis settings, the card fails and says where the conflict is.
|
||||||
|
- The output is `nodes.<card>.results.<item>`: status, verdict code, direction, named values, and verdict phrase. A downstream branch can route on the verdict, for example sending a run to a person when the change is outside the usual fluctuation.
|
||||||
|
- An item that can't be done doesn't fail the card: that item is recorded as not done, and the card moves on to the next one.
|
||||||
|
- Analysis card results are not caliber card inputs: a caliber card that reads them is treated as unable to trace back to raw data. Take metrics in the caliber card from query result cells, and cite analysis conclusions directly in the report.
|
||||||
|
- The governed level requires analysis card inputs to be Call Tool nodes that call a database query tool, or Merge Query nodes, so that it is settled at publish time that the card analyzes queries the platform executed.
|
||||||
|
- Run comparison aligns analysis cards by node and item and lists both runs' verdicts (see [Run comparison](evidence.md#run-comparison)).
|
||||||
|
|
||||||
|
The built-in template 11, Anomaly Analysis (Traceable), is built this way: fetch the weekly totals and two weeks by region → analysis card (fluctuation baseline, change decomposition with totals) → report writer → issuance. It can be published at the governed level as it is (see [Built-in templates](workflows.md#built-in-templates)).
|
||||||
|
|
||||||
|
## Citing in reports
|
||||||
|
|
||||||
|
```
|
||||||
|
[[a:A1]] A1's verdict phrase, rendered by the system from the verdict; one clickable segment
|
||||||
|
[[a:A1.top_share|pct]] one named value of A1; unit conversion as for [[m:]]
|
||||||
|
[[v:A1.r0.share]] one cell of A1's result table
|
||||||
|
[[table:A1 cols=region,delta,share rows=0-4]] A1's result table
|
||||||
|
[[see:A1]] basis of the sentence; attached at the end of the sentence, not rendered
|
||||||
|
```
|
||||||
|
|
||||||
|
- The Analysis Conclusions section of the writing catalog lists each conclusion's number, verdict phrase, named values, the top 10 segments of its result table, and caveats or the reason it couldn't be done.
|
||||||
|
- Conclusions from analysis cards can always be cited. In formal issuance at the governed level, analyses an agent ran on the spot with analysis tools follow the same rule as the cells it queried: they can be cited only when the issuance contract turns on Cell Citations (they can still serve as the basis at the end of a sentence).
|
||||||
|
- The measure name in a verdict phrase comes from the agent or the template; the system first strips digits, percent signs, units, and citation markers from it and cuts it to 20 characters, so text written by the system never smuggles in numbers written by the model.
|
||||||
|
|
||||||
|
## Analysis claim check
|
||||||
|
|
||||||
|
The report-writer node's Analysis Claim Check (`analysis_claims`) checks judgment words sentence by sentence, with a fixed lexicon and binding rules, without calling a model.
|
||||||
|
|
||||||
|
| Class of judgment word | Examples (translated) | Analysis conclusion and verdict it needs |
|
||||||
|
|---|---|---|
|
||||||
|
| Source | mainly from, concentrated in, dragged down, drove, contributed most | Change decomposition judged a main source or concentrated; the segments named must be in the verdict, and a segment said to drag down must be falling |
|
||||||
|
| Dispersed, offset, mix | spread across regions, one up and one down, structural change | Change decomposition judged dispersed, offsetting, or mainly from the mix |
|
||||||
|
| Unusual, record, normal | unusual, sharply, surged, record high, normal fluctuation | Fluctuation baseline judged outside or far outside (direction matching), record high or low, or usual fluctuation |
|
||||||
|
| Trend, no trend | rising steadily, down three weeks in a row, no clear trend | Trend test, with matching direction and run length |
|
||||||
|
| Difference, no difference | significantly higher, really lower, little difference | Difference test judged as holding up (direction matching) or possibly sampling noise |
|
||||||
|
| Causal | because, due to, led to, affected by | A verbatim knowledge-base quote or a run input as the basis in the same sentence, or phrasing as a conjecture (possibly, to be verified) |
|
||||||
|
|
||||||
|
- A sentence's basis is the `[[a:…]]`, `[[v:A…]]`, and `[[table:A…]]` in it plus the `[[see:A…]]` at its end. Negation flips the class: "not unusual" is checked as normal.
|
||||||
|
- Not checked: statements about composition rather than change ("this week's orders came mainly from the online channel"), suggestions and plans, "abnormal" as a business status ("some orders are abnormal"), verbatim knowledge-base quotes, system-rendered verdict phrases, headings, and tables.
|
||||||
|
- The line between source and cause: "the decline was mainly caused by East", citing a breakdown by region and naming one of its segments, is checked as a source, the bookkeeping sense of "came from"; "the decline was due to the end of a promotion" names no segment and is checked as causation.
|
||||||
|
- Three kinds of violation: a judgment word without a matching analysis conclusion; a conclusion cited, but the verdict, the segments named, or the direction don't match (citing a conclusion that couldn't be done counts too); a cause without a basis.
|
||||||
|
|
||||||
|
| Value | What it does |
|
||||||
|
|---|---|
|
||||||
|
| `check` (default, also used by Ask Data) | Sends the report back to the writer for one rewrite; what remains is marked in the report and doesn't affect the issuance tier |
|
||||||
|
| `strict` | Rewrites up to the maximum number of repairs; what remains counts as an issuance gap and downgrades the issuance; required at the governed level (gate G3) |
|
||||||
|
| `off` | No check |
|
||||||
|
|
||||||
|
- When the writing catalog has analysis conclusions, the writer is told to follow the verdicts and preferably cite verdict phrases directly; when it has none, the writer is told not to make such judgments and to state only numbers and changes.
|
||||||
|
- An exit in citation mode checks again independently, with the same check used at assembly. The report document records which disposition and which version of the lexicon it was checked with: only documents that carry this record are checked for it on verification, so verification results for reports from before the upgrade don't change.
|
||||||
|
- It errs on the side of missing cases: only strong judgment words are collected. The finer points it misses are left to the claim-sentence judge (see [Claim sentences](evidence.md#claim-sentences)); when the judge's excerpt includes an analysis conclusion, the judge is also told that the verdict was computed by the system and to judge only whether the sentence states it correctly.
|
||||||
|
|
||||||
|
## Analysis scope
|
||||||
|
|
||||||
|
When a report uses analysis conclusions (when this run has any), the system appends an Analysis Scope section at the end of the report. The writer can't change it; it is sealed with the document, and its numbers are clickable:
|
||||||
|
|
||||||
|
- **Phenomena**: verdict phrases of comparability checks, fluctuation baselines, and trend tests.
|
||||||
|
- **Broken down**: each change decomposition, listed along its drill-down chain with its dimensions and verdict.
|
||||||
|
- **Ruled out**: decompositions judged dispersed, and difference tests where the difference may be just sampling noise. A breakdown that finds no concentration is a conclusion too.
|
||||||
|
- **Not done**: the reason for each analysis that couldn't be done.
|
||||||
|
- **Not broken down**: candidate dimensions that weren't broken down. Candidate dimensions are computed deterministically from the data catalog: columns on the fact table used by the decomposition's query that are marked as an attribute or status, aren't masked, and have a code-value table (at most 50 values, or marked complete), plus the same kind of columns on dimension tables related to it. When the columns used by the decomposition don't match the catalog, it says only that candidate dimensions can't be checked, rather than listing guesses.
|
||||||
|
- **Tests**: how many difference tests were run this time, and whether they were corrected for the number of tests.
|
||||||
|
|
||||||
|
The sentence an agent writes about what a call is testing doesn't go into this section; it appears only in the evidence panel. The report-writer node's Analysis Scope (`analysis_scope`) defaults to appending the section when there are analysis conclusions, and can be set not to.
|
||||||
|
|
||||||
|
## Settings
|
||||||
|
|
||||||
|
Under Settings → Preferences → Analysis you can change the thresholds used for verdicts, how many analyses an agent may run per execution, how many levels it may drill down, and how many segments are listed. For each item's label and default, see [Analysis settings](configuration.md#analysis-settings).
|
||||||
|
|
||||||
|
- The group is snapshotted into the run when the run starts (only when the graph has an analysis card or an agent with analysis tools), and resuming or continuing uses the same snapshot.
|
||||||
|
- Agents can't change thresholds: the analysis tools have no threshold parameters. A template can write threshold overrides on an analysis card.
|
||||||
|
- Every conclusion records the thresholds actually used, and its criterion is written from them, for example "a segment accounting for at least half of the total change is the main source".
|
||||||
|
|
||||||
|
## Limits
|
||||||
|
|
||||||
|
- Not in this round: forecasting; correlation analysis; dimension hierarchies in the data catalog; profiling of numeric columns; analysis tools for the verification agent of double-blind recomputation, adversarial review, and approval inquiry; analysis conclusions in caliber ledger matching and recalls (numbers in analysis conclusions get no caliber label; see [Caliber ledger](evaluation-and-ledger.md#caliber-ledger)).
|
||||||
|
- The trend test assumes independent periods; for autocorrelated series, p comes out too small.
|
||||||
|
- Candidate dimensions come only from code-value tables: profiling doesn't yet record each column's number of distinct values, so attribute columns without a code-value table aren't candidates.
|
||||||
|
- The lexicon collects only strong judgment words, so a different wording may slip through; what slips through is left to the claim-sentence judge and to the people reading the report.
|
||||||
|
- Analyses done by members of a collaboration team don't go into the evidence ledger, so reports can't cite them.
|
||||||
+14
-1
@@ -25,6 +25,17 @@ A turn covers: Copilot builds the graph, the run starts, the result is reviewed,
|
|||||||
- When Copilot returns only thinking and no operations, the turn is retried once automatically.
|
- When Copilot returns only thinking and no operations, the turn is retried once automatically.
|
||||||
- The cost of each turn includes the model calls for Copilot's graph building, table selection, and review.
|
- The cost of each turn includes the model calls for Copilot's graph building, table selection, and review.
|
||||||
|
|
||||||
|
## Analytical questions
|
||||||
|
|
||||||
|
When the question asks why, how something changed, what dragged it down, what contributed most, whether something is unusual, how much it fluctuates, whether it keeps rising, or whether one is really higher than the other, the agent that queries data in the graph Copilot builds is also given the five analysis tools (see [Analysis](analysis.md)):
|
||||||
|
|
||||||
|
- The agent first settles which two periods to compare from the question (the previous whole period when the question doesn't say), then follows the platform's built-in analysis method: comparability check, putting the period back into its history, breaking down by dimension, and drilling down one level once the main source is found. The platform computes the verdicts with fixed methods and thresholds; the agent decides only what to compute.
|
||||||
|
- The report is written from the analysis conclusions, citing the system-written verdict phrases directly for its judgments. When the writer writes judgment words such as "mainly from" or "unusual" itself, the system checks each sentence: sentences that don't cite a matching analysis conclusion go back for one rewrite, and what remains is marked in the answer without affecting the issuance tier (see [Analysis claim check](analysis.md#analysis-claim-check)).
|
||||||
|
- At the end of an answer that uses analysis conclusions, the system appends an Analysis Scope section: what was broken down, what was ruled out, which candidate dimensions weren't broken down, and what couldn't be done (see [Analysis scope](analysis.md#analysis-scope)).
|
||||||
|
- A breakdown shows only where a change came from, not why: causes in an answer need a verbatim knowledge-base quote as their basis, or else they are written as conjectures to be verified.
|
||||||
|
- During its self-check, Copilot recognizes analytical questions with a fixed lexicon that collects only explicit phrasings. When the question is analytical but the agent that queries data has no analysis tools, the problem goes back to Copilot for one round of fixes; if the tools are still missing after that round, the graph runs as usual with a warning.
|
||||||
|
- When an analysis couldn't be done (not enough history, the current period isn't over, and so on), or judgment words still don't cite a matching conclusion after the rewrite, the review only writes a note and doesn't start over automatically (see [Output review](#output-review)).
|
||||||
|
|
||||||
## Output review
|
## Output review
|
||||||
|
|
||||||
After a run ends, rules scan the event stream first, and a model review runs only when they find an anomaly. The review may restructure the answer, but it **must not introduce numbers that aren't in the original answer**.
|
After a run ends, rules scan the event stream first, and a model review runs only when they find an anomaly. The review may restructure the answer, but it **must not introduce numbers that aren't in the original answer**.
|
||||||
@@ -32,10 +43,11 @@ After a run ends, rules scan the event stream first, and a model review runs onl
|
|||||||
- Ask Data for users (the user role) is reviewed too.
|
- Ask Data for users (the user role) is reviewed too.
|
||||||
- When a step fails inside a subworkflow, automatic review reports it only once.
|
- When a step fails inside a subworkflow, automatic review reports it only once.
|
||||||
- Continue Run in Ask Data is limited to builders and above.
|
- Continue Run in Ask Data is limited to builders and above.
|
||||||
|
- The gaps from analysis (some analyses couldn't be done, an analysis call was wrong and never fixed, judgment words still failed the check after the rewrite, the two periods of a caliber card metric couldn't be checked for comparability) only get a note and never trigger an automatic redo. When a run has only these, the review calls no model and writes the note from their wording as is.
|
||||||
|
|
||||||
## Calibers in an answer
|
## Calibers in an answer
|
||||||
|
|
||||||
A caliber is the agreed definition of how a metric is fetched and computed. It lives in a caliber card, with a name and a version. After an answer is sealed, the system compares every number in it with the published calibers in the background and labels each number with a caliber; each turn gets a one-line summary. When the whole answer is equivalent to a governed caliber, the system automatically starts a formal run by that caliber. See [Evaluation and the Caliber Ledger](evaluation-and-ledger.md#caliber-ledger).
|
A caliber is the agreed definition of how a metric is fetched and computed. It lives in a caliber card, with a name and a version. After an answer is sealed, the system compares every number in it with the published calibers in the background and labels each number with a caliber; each turn gets a one-line summary. When the whole answer is equivalent to a governed caliber, the system automatically starts a formal run by that caliber. Numbers in analysis conclusions (shares, changes, fluctuation ranges) are derived from query results, so they aren't compared with calibers, get no caliber label, and aren't counted in the summary line. See [Evaluation and the Caliber Ledger](evaluation-and-ledger.md#caliber-ledger).
|
||||||
|
|
||||||
## From an answer to a workflow or an app
|
## From an answer to a workflow or an app
|
||||||
|
|
||||||
@@ -60,6 +72,7 @@ Copilot generates or rewrites a whole graph from one sentence and lays it out au
|
|||||||
|
|
||||||
- When it edits a graph, it outputs only the change operations instead of rewriting the whole graph.
|
- When it edits a graph, it outputs only the change operations instead of rewriting the whole graph.
|
||||||
- Before delivering, it checks its work with the same rules the runtime uses. Problems that would make the run fail go back to it to fix first; if they can't be fixed, nothing runs automatically.
|
- Before delivering, it checks its work with the same rules the runtime uses. Problems that would make the run fail go back to it to fix first; if they can't be fixed, nothing runs automatically.
|
||||||
|
- In Ask Data, the self-check also looks at the question itself: when the question is analytical but the agent that queries data has no analysis tools, it gets one round of reminders (see [Analytical questions](#analytical-questions)).
|
||||||
- While building graphs, it records facts worth remembering in long-term memory.
|
- While building graphs, it records facts worth remembering in long-term memory.
|
||||||
|
|
||||||
### Looking on demand
|
### Looking on demand
|
||||||
|
|||||||
@@ -67,7 +67,7 @@ For the app database, checkpoint database, connection pool, PgBouncer, and multi
|
|||||||
| Location | Contents |
|
| Location | Contents |
|
||||||
|---|---|
|
|---|---|
|
||||||
| Settings → Model Access | Providers and keys |
|
| Settings → Model Access | Providers and keys |
|
||||||
| Settings → Preferences | Run defaults (default memory scope, default knowledge base, require approval for dangerous tools by default); the evidence judge (judge model and limits); verification (verification model, the switches and limits for double-blind recomputation, adversarial review, and approval inquiry, and the daily spending cap; see [The review suite](evidence.md#the-review-suite)); evaluation concurrency, per-question time limit, and per-evaluation and daily spending; run concurrency (admins); the caliber ledger (automatic issuance by caliber and its daily cap, the caliber matching time limit, the recall limit `ledger.recall_limit`; see [Caliber ledger](evaluation-and-ledger.md#caliber-ledger)); drafting model; apps (the daily cap on on-demand runs per app, letting users create personal apps, allowing share links); interface theme |
|
| Settings → Preferences | Run defaults (default memory scope, default knowledge base, require approval for dangerous tools by default); the evidence judge (judge model and limits); verification (verification model, the switches and limits for double-blind recomputation, adversarial review, and approval inquiry, and the daily spending cap; see [The review suite](evidence.md#the-review-suite)); evaluation concurrency, per-question time limit, and per-evaluation and daily spending; analysis (the thresholds used for verdicts, how many analyses an agent may run per execution, the drill-down depth, how many segments are listed; see [Analysis settings](#analysis-settings)); run concurrency (admins); the caliber ledger (automatic issuance by caliber and its daily cap, the caliber matching time limit, the recall limit `ledger.recall_limit`; see [Caliber ledger](evaluation-and-ledger.md#caliber-ledger)); drafting model; apps (the daily cap on on-demand runs per app, letting users create personal apps, allowing share links); interface theme |
|
||||||
| Settings → General | Public URL |
|
| Settings → General | Public URL |
|
||||||
| Settings → Runtime Environment | Sandbox candidates and their availability, the current choice and its reason; current values such as time limits |
|
| Settings → Runtime Environment | Sandbox candidates and their availability, the current choice and its reason; current values such as time limits |
|
||||||
| Settings → Identity | Login mode; enabling Ask Data for users |
|
| Settings → Identity | Login mode; enabling Ask Data for users |
|
||||||
@@ -77,6 +77,30 @@ For the app database, checkpoint database, connection pool, PgBouncer, and multi
|
|||||||
|
|
||||||
Other settings: `notify.digest_at` (the time of the daily task digest), and the agent token budget, spending budget, and step count.
|
Other settings: `notify.digest_at` (the time of the daily task digest), and the agent token budget, spending budget, and step count.
|
||||||
|
|
||||||
|
### Analysis settings
|
||||||
|
|
||||||
|
The Analysis section under Settings → Preferences governs how change decomposition, the fluctuation baseline, the trend test, and the difference test judge, and how many analyses an agent may run (see [Analysis](analysis.md)). It is snapshotted into a run when the run starts, and resuming or continuing uses the same snapshot; an analysis card can carry threshold overrides, and agents can't change any of it.
|
||||||
|
|
||||||
|
| Setting | Default | Range | What it does |
|
||||||
|
|---|---|---|---|
|
||||||
|
| Main-source share | 0.5 | 0.3–0.9 | A segment (a factor, an effect) accounting for this share of the total change is the main source |
|
||||||
|
| Concentration share | 0.8 | 0.5–0.99 | The top few segments together accounting for this share count as concentrated; can't be below the main-source share |
|
||||||
|
| Offset ratio | 0.25 | 0.05–1 | When the segments moving the other way add up to more than this share of the total change, a caveat says they offset each other |
|
||||||
|
| Flat threshold | 0.005 | 0–0.1 | A total change below this share of the base counts as essentially flat |
|
||||||
|
| Outside-fluctuation multiple | 2 | 1–5 | Fluctuation baseline: this many robust standard deviations from the historical median counts as outside the usual fluctuation |
|
||||||
|
| Far-outside multiple | 3 | 1.5–8 | This many counts as far outside; must be greater than the outside-fluctuation multiple |
|
||||||
|
| Minimum history periods | 8 | 4–104 | With less history than this, the fluctuation baseline can't be done |
|
||||||
|
| Maximum history periods | 52 | 8–260 | The fluctuation baseline and the trend test use at most this many periods; can't be below the minimum |
|
||||||
|
| Minimum trend periods | 6 | 4–104 | The trend test needs at least this many periods |
|
||||||
|
| Trend significance level | 0.05 | 0.001–0.2 | Rising or falling is judged only when p is below this |
|
||||||
|
| Difference-test significance level | 0.05 | 0.001–0.2 | A corrected p below this means the difference holds up |
|
||||||
|
| Minimum sample size for means | 30 | 2–10000 | When a group has fewer samples in a comparison of means, a caveat says the sample is small |
|
||||||
|
| Minimum expected frequency for rates | 5 | 1–50 | When a cell's expected frequency is below this, Fisher's exact test is used |
|
||||||
|
| Minimum denominator for ratio breakdowns | 30 | 1–100000 | Segments with a smaller denominator are marked small and left out of the Simpson's paradox judgment |
|
||||||
|
| Maximum analyses per agent execution | 12 | 1–50 | Once used up, further analysis tool calls from the agent are told the analyses are used up |
|
||||||
|
| Maximum drill-down depth | 3 | 1–6 | How many levels a change decomposition can drill down |
|
||||||
|
| Segments listed | 10 | 3–30 | How many segments of a result table the writing catalog and the reply to the agent list |
|
||||||
|
|
||||||
## Sandbox backend
|
## Sandbox backend
|
||||||
|
|
||||||
Settings → Runtime Environment lists every candidate and its availability, and marks the current choice and its selection reason. Automatic selection works in this order:
|
Settings → Runtime Environment lists every candidate and its availability, and marks the current choice and its selection reason. Automatic selection works in this order:
|
||||||
|
|||||||
@@ -16,10 +16,12 @@ Each question in a test suite is a question plus an expected answer. The expecta
|
|||||||
- A metric of a published caliber.
|
- A metric of a published caliber.
|
||||||
- A query identified from an Ask Data run.
|
- A query identified from an Ask Data run.
|
||||||
- "No data", or a clarifying question back.
|
- "No data", or a clarifying question back.
|
||||||
|
- The main source of a change (contribution): an expected SQL statement plus change decomposition parameters; the main source (one segment or the top few) and the direction are computed with the same analyses the answer uses.
|
||||||
|
- A verdict: an expected SQL statement plus parameters for a fluctuation baseline, trend test, difference test, or comparability check; the class of verdict is computed with the same analyses: usual fluctuation or unusual; rising, falling, or no clear trend; a difference that holds up or not enough to conclude; comparable or not comparable (see [Analysis](analysis.md#the-five-analyses)).
|
||||||
|
|
||||||
Rules:
|
Rules:
|
||||||
|
|
||||||
- Numeric values may not be typed in by hand.
|
- Numeric values may not be typed in by hand; nor may the segments, directions, and verdicts in contribution and verdict expectations, which are computed from the expected SQL with the same analyses, once before and once after the answer.
|
||||||
- Text key points are judged by the judge, listed separately, and not counted toward accuracy.
|
- Text key points are judged by the judge, listed separately, and not counted toward accuracy.
|
||||||
- After a caliber version bump, questions that use it show a notice at the top. Each question takes one of three options: switch to pinning the new version, compute one expectation for each of the old and new versions, or archive the old question and create a new one. Until then, the question is skipped in evaluations.
|
- After a caliber version bump, questions that use it show a notice at the top. Each question takes one of three options: switch to pinning the new version, compute one expectation for each of the old and new versions, or archive the old question and create a new one. Until then, the question is skipped in evaluations.
|
||||||
- Test suites can be imported and exported.
|
- Test suites can be imported and exported.
|
||||||
@@ -42,7 +44,9 @@ Rules:
|
|||||||
- **Comparison**: against the previous evaluation or a baseline, with per-question transitions and attribution.
|
- **Comparison**: against the previous evaluation or a baseline, with per-question transitions and attribution.
|
||||||
- **A/B paired table**: exact McNemar test; display only, not a gate.
|
- **A/B paired table**: exact McNemar test; display only, not a gate.
|
||||||
|
|
||||||
Evaluation runs are left out of usage statistics. The scoring rules are at version 2; old evaluations can be re-scored under the new rules from the evaluation details (reading only the sealed records, without calling models).
|
Contribution and verdict expectations are scored like this: an answer that cites an analysis conclusion whose verdict matches is correct and traceable; body text that states it correctly by the lexicon without citing it is correct but uncited; a source or verdict that doesn't match, or the opposite answer, is wrong; an expectation that itself can't be computed (the expected SQL returns too little history, say) can't be scored. The input queries of analysis conclusions also count as queries the answer cited.
|
||||||
|
|
||||||
|
Evaluation runs are left out of usage statistics. The scoring rules are at version 3 (which adds contribution and verdict expectations); old evaluations can be re-scored under the new rules from the evaluation details (reading only the sealed records, without calling models).
|
||||||
|
|
||||||
### Scheduled evaluations
|
### Scheduled evaluations
|
||||||
|
|
||||||
@@ -68,6 +72,7 @@ After an exploratory run is sealed, the system compares the answer with publishe
|
|||||||
- Judging is number by number: when a report cites several columns of one query (for example sales, order count, first and last day), each number is compared on its own column against the caliber's metric column, and extra queried columns don't affect it. This applies only when removing the other columns wouldn't change the number; queries with DISTINCT, truncation after sorting by another column, or extra grouping columns are still compared as a whole.
|
- Judging is number by number: when a report cites several columns of one query (for example sales, order count, first and last day), each number is compared on its own column against the caliber's metric column, and extra queried columns don't affect it. This applies only when removing the other columns wouldn't change the number; queries with DISTINCT, truncation after sorting by another column, or extra grouping columns are still compared as a whole.
|
||||||
- When a multi-row query has no `ORDER BY`, citing one of its cells doesn't count as "from a caliber", and recall and reconciliation mark it "Row order undefined".
|
- When a multi-row query has no `ORDER BY`, citing one of its cells doesn't count as "from a caliber", and recall and reconciliation mark it "Row order undefined".
|
||||||
- An answer is issued automatically only when every number in it that comes from data is proven to come from the same caliber.
|
- An answer is issued automatically only when every number in it that comes from data is proven to come from the same caliber.
|
||||||
|
- Numbers in analysis conclusions (shares, changes, fluctuation ranges) are derived from query results, so they aren't compared or labeled with a caliber and aren't counted in the summary line.
|
||||||
|
|
||||||
When writing a caliber's data-fetch SQL, it is best not to round (`ROUND(SUM(x), 2)`) or wrap values in `COALESCE(…, 0)`. Leave rounding and defaults to the caliber card's decimal places and display settings; otherwise Ask Data queries written without rounding will never be recognized as equivalent (and where they differ is stated plainly).
|
When writing a caliber's data-fetch SQL, it is best not to round (`ROUND(SUM(x), 2)`) or wrap values in `COALESCE(…, 0)`. Leave rounding and defaults to the caliber card's decimal places and display settings; otherwise Ask Data queries written without rounding will never be recognized as equivalent (and where they differ is stated plainly).
|
||||||
|
|
||||||
|
|||||||
+25
-2
@@ -2,7 +2,7 @@ English | [Simplified Chinese](../zh-CN/evidence.md)
|
|||||||
|
|
||||||
# Evidence and Issuance
|
# Evidence and Issuance
|
||||||
|
|
||||||
This page covers where the numbers in a report come from and how to check them: caliber cards and issuance contracts, the three issuance tiers, citation markers in the report-writer node, what happens when the queried range has no data, what you see when you click a number, sealing and verification, the two kinds of export, and the review tools: double-blind recomputation, adversarial review, approval inquiry, the what-if sandbox, and run comparison. It is for builders, approvers, and people who read reports. The user interface is currently available in Simplified Chinese only.
|
This page covers where the numbers in a report come from and how to check them: caliber cards and issuance contracts, the three issuance tiers, citation markers in the report-writer node (including analysis conclusions), what happens when the queried range has no data, what you see when you click a number or an analysis conclusion, sealing and verification, the two kinds of export, and the review tools: double-blind recomputation, adversarial review, approval inquiry, the what-if sandbox, run comparison, and the analysis claim check. It is for builders, approvers, and people who read reports. For how analytical judgments themselves are computed, see [Analysis](analysis.md). The user interface is currently available in Simplified Chinese only.
|
||||||
|
|
||||||
## The basic idea
|
## The basic idea
|
||||||
|
|
||||||
@@ -45,6 +45,9 @@ Measured: every number in body text written by Opus 5 traced back successfully,
|
|||||||
|
|
||||||
When the contract sets `report_from`, the check goes by citation: the hash is verified again when the report document is retrieved, everything is recomputed independently, and only when every number carries a citation is the issuance judged complete.
|
When the contract sets `report_from`, the check goes by citation: the hash is verified again when the report document is retrieved, everything is recomputed independently, and only when every number carries a citation is the issuance judged complete.
|
||||||
|
|
||||||
|
- The inputs of caliber card metrics the report cites must trace back to raw data (cells of query snapshots, run inputs): in formal runs at the governed level, inputs that can't be traced count as gaps and downgrade the issuance; otherwise they are stated in the issuance declaration, and the evidence panel still marks them "Can't trace back to raw data".
|
||||||
|
- Analysis claims in the report are checked again here independently, with the same check used when the report is assembled: when the report-writer node is set to strict, remaining violations count as issuance gaps (see [Analysis claim check](#analysis-claim-check)).
|
||||||
|
|
||||||
### Metric issuance
|
### Metric issuance
|
||||||
|
|
||||||
A pure metrics dashboard (the contract sets only `metrics_from`, there is no report-writer node upstream, and the output contains no model-written text) is issued by its metrics. With no narrative to check, the metrics themselves are checked:
|
A pure metrics dashboard (the contract sets only `metrics_from`, there is no report-writer node upstream, and the output contains no model-written text) is issued by its metrics. With no narrative to check, the metrics themselves are checked:
|
||||||
@@ -62,6 +65,8 @@ When the source query of a contract metric has SQL check errors, the issuance is
|
|||||||
- An expression can be written as `cell(nodes.pull, 0, 'gmv')`, which makes the provenance precise to the cell.
|
- An expression can be written as `cell(nodes.pull, 0, 'gmv')`, which makes the provenance precise to the cell.
|
||||||
- Leave rounding and default values to the caliber card's decimal places and display settings instead of writing them into the data-fetch SQL (for why, see [Caliber Ledger](evaluation-and-ledger.md#a-caliber-label-for-each-number)).
|
- Leave rounding and default values to the caliber card's decimal places and display settings instead of writing them into the data-fetch SQL (for why, see [Caliber Ledger](evaluation-and-ledger.md#a-caliber-label-for-each-number)).
|
||||||
- When publishing at the governed level, caliber card inputs can't come from sandbox code in the compute role (gate G4).
|
- When publishing at the governed level, caliber card inputs can't come from sandbox code in the compute role (gate G4).
|
||||||
|
- When a metric uses two queries of the same shape that differ only in their dates (this week versus last week), the system runs a comparability check once automatically (see [Comparability check](analysis.md#comparability-check)) and attaches the result to the metric: the writing catalog states it plainly; at an exit with a contract, incomparable periods count as a gap and downgrade the issuance, and conditionally comparable ones are only flagged.
|
||||||
|
- Analysis card results are not caliber card inputs: a caliber card that reads them is treated as unable to trace back to raw data.
|
||||||
|
|
||||||
### Caliber version bumps
|
### Caliber version bumps
|
||||||
|
|
||||||
@@ -88,9 +93,13 @@ In the report-writer node (`report`), the model writes only citation markers, an
|
|||||||
[[table:Q1 cols=region,amount rows=0-4]] the system builds the whole table from the snapshot; every cell is clickable
|
[[table:Q1 cols=region,amount rows=0-4]] the system builds the whole table from the snapshot; every cell is clickable
|
||||||
[[t:orders]] [[c:orders.week]] table name, field name; names written in backticks are checked too
|
[[t:orders]] [[c:orders.week]] table name, field name; names written in backticks are checked too
|
||||||
[[q:K1|<quote>]] verbatim knowledge-base quote; must appear word for word in the chunks that retrieval hit
|
[[q:K1|<quote>]] verbatim knowledge-base quote; must appear word for word in the chunks that retrieval hit
|
||||||
[[see:m:gmv,Q1]] basis of a claim sentence; attached at the end of the sentence, not rendered
|
[[a:A1]] [[a:A1.top_share|pct]] an analysis conclusion's verdict phrase and a named value, both rendered by the system
|
||||||
|
[[v:A1.r0.share]] [[table:A1 rows=0-4]] one cell and the whole of an analysis conclusion's result table
|
||||||
|
[[see:m:gmv,Q1,A1]] basis of a claim sentence; attached at the end of the sentence, not rendered
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Analysis conclusions are numbered A1, A2, … in ledger order, and the writing catalog has a separate Analysis Conclusions section. Judgment words such as "mainly from", "unusual", or "rising steadily" must cite a matching analysis conclusion and agree with its verdict; see [Analysis claim check](analysis.md#analysis-claim-check).
|
||||||
|
|
||||||
### Handling violations
|
### Handling violations
|
||||||
|
|
||||||
- Bare numbers and citations that can't be parsed are flagged as violations on the spot and sent back for rewriting (`max_repairs`, default 1).
|
- Bare numbers and citations that can't be parsed are flagged as violations on the spot and sent back for rewriting (`max_repairs`, default 1).
|
||||||
@@ -99,6 +108,7 @@ In the report-writer node (`report`), the model writes only citation markers, an
|
|||||||
## The data retrieval chain
|
## The data retrieval chain
|
||||||
|
|
||||||
- Data fetched by agents, tool nodes, code nodes, and knowledge retrieval is recorded in the evidence ledger. The report's source list is numbered from the ledger: Q1… for queries, K1… for knowledge-base chunks.
|
- Data fetched by agents, tool nodes, code nodes, and knowledge retrieval is recorded in the evidence ledger. The report's source list is numbered from the ledger: Q1… for queries, K1… for knowledge-base chunks.
|
||||||
|
- Analysis conclusions from agents' analysis tools and from analysis cards are recorded in the evidence ledger too, numbered A1…. Analyses read only query snapshots the platform executed during this run, and each conclusion is sealed together with its inputs (see [Analysis](analysis.md)).
|
||||||
- When an agent turns on `cite_fields` (together with `output_schema`), each field records which cell of which query it came from, and that is checked against the snapshot: on a mismatch the snapshot wins and a warning is raised; anything not found is recorded as null, not defaulted to 0.
|
- When an agent turns on `cite_fields` (together with `output_schema`), each field records which cell of which query it came from, and that is checked against the snapshot: on a mismatch the snapshot wins and a warning is raised; anything not found is recorded as null, not defaulted to 0.
|
||||||
- A code node's `evidence_role` is `source` (data retrieval) or `compute` (computation, the default). Numbers computed in the sandbox can't be cited directly by a report; they must go through a caliber card.
|
- A code node's `evidence_role` is `source` (data retrieval) or `compute` (computation, the default). Numbers computed in the sandbox can't be cited directly by a report; they must go through a caliber card.
|
||||||
- Cells of uploaded spreadsheets can be traced back to the cell in the original sheet (an inferred source).
|
- Cells of uploaded spreadsheets can be traced back to the cell in the original sheet (an inferred source).
|
||||||
@@ -126,6 +136,7 @@ Query a table that keeps only the latest day (fully reloaded every day) for last
|
|||||||
- **Numbers**: the metric's original formula, substituted formula, and recomputed result; whether the inputs match the snapshot; the query's SQL; the cited row plus 2 rows before and after, highlighted down to the cell; and finally the seal status. Only artifacts returned by events inside the sealed scope are accepted. The provenance chain also starts with a Caliber step (see [Caliber Ledger](evaluation-and-ledger.md#a-caliber-label-for-each-number)).
|
- **Numbers**: the metric's original formula, substituted formula, and recomputed result; whether the inputs match the snapshot; the query's SQL; the cited row plus 2 rows before and after, highlighted down to the cell; and finally the seal status. Only artifacts returned by events inside the sealed scope are accepted. The provenance chain also starts with a Caliber step (see [Caliber Ledger](evaluation-and-ledger.md#a-caliber-label-for-each-number)).
|
||||||
- **Table and field names**: which query they appear in, their type, and the table they belong to. A name found nowhere in the schema snapshot, the SQL, or the result columns is marked as a suspicious entity ("May be a fabricated name", with the 3 closest matches). When a snapshot was truncated and can't be checked, it only says so, without wrongly calling the name fabricated.
|
- **Table and field names**: which query they appear in, their type, and the table they belong to. A name found nowhere in the schema snapshot, the SQL, or the result columns is marked as a suspicious entity ("May be a fabricated name", with the 3 closest matches). When a snapshot was truncated and can't be checked, it only says so, without wrongly calling the name fabricated.
|
||||||
- **Knowledge-base quotes**: must appear word for word in the chunks that retrieval hit.
|
- **Knowledge-base quotes**: must appear word for word in the chunks that retrieval hit.
|
||||||
|
- **Analysis conclusions** (verdict phrases, named values, result table cells): in order, the verdict (verdict phrase, criterion, segments named and direction, who did it, caveats, and the agent's note on what the call was testing), the method (method and version, parameters, thresholds actually used, table ranges checked live), the calculation (the result table and the substituted formula for the key value, such as "share = East's change / total change = -120 / -194", with the cited cell highlighted), the inputs (which columns and rows of which Q, the closure check against the two total cells, the level a drill-down sits under, followed by the steps of the input queries), the recomputation (rerun from the sealed inputs with the same function; it verifies only when every field is identical, and when the method has a new version, it says which version the run used), and the seal. In the evidence graph, an analysis conclusion has a "derived from" edge to its inputs and to the level above it.
|
||||||
|
|
||||||
## Claim sentences
|
## Claim sentences
|
||||||
|
|
||||||
@@ -165,12 +176,14 @@ The two exports serve different purposes: to prove nothing was altered, use the
|
|||||||
- The body is rendered on the server, so it is readable without running scripts (email attachment previews often don't run scripts).
|
- The body is rendered on the server, so it is readable without running scripts (email attachment previews often don't run scripts).
|
||||||
- With scripts running, every number, every number without evidence, and every suspicious table or field name is clickable. The panel shows the same provenance chain as the evidence panel on the web page (caliber, source, calculation, SQL, cited rows), with the seal status marked item by item.
|
- With scripts running, every number, every number without evidence, and every suspicious table or field name is clickable. The panel shows the same provenance chain as the evidence panel on the web page (caliber, source, calculation, SQL, cited rows), with the seal status marked item by item.
|
||||||
- Masked columns say "Masked", following the same rules as the web page.
|
- Masked columns say "Masked", following the same rules as the web page.
|
||||||
|
- Verdict phrases, named values, and result table cells of analysis conclusions are clickable too, and the panel shows the verdict, method, calculation, and inputs in turn; at export time the server recomputes each conclusion from the sealed inputs and writes the result into the panel. The Analysis Scope at the end of the report is rendered from what the document recorded, with clickable numbers.
|
||||||
- Unsealed runs export as usual, for the same reason you can read their reports on the web.
|
- Unsealed runs export as usual, for the same reason you can read their reports on the web.
|
||||||
|
|
||||||
**Verifiable report**: Export Verifiable Report under Provenance Details on the Records page (`GET /api/runs/{id}/bundle.html`) produces a seal verification document.
|
**Verifiable report**: Export Verifiable Report under Provenance Details on the Records page (`GET /api/runs/{id}/bundle.html`) produces a seal verification document.
|
||||||
|
|
||||||
- The recipient recomputes the hashes in the browser, without connecting to the service or installing anything, and checks that no number, document, or run record has been altered.
|
- The recipient recomputes the hashes in the browser, without connecting to the service or installing anything, and checks that no number, document, or run record has been altered.
|
||||||
- The provenance panel for a number likewise shows the caliber, original formula, substituted formula, inputs, and recomputation; the query snapshots that caliber card inputs point to come attached to the file.
|
- The provenance panel for a number likewise shows the caliber, original formula, substituted formula, inputs, and recomputation; the query snapshots that caliber card inputs point to come attached to the file.
|
||||||
|
- Analysis conclusions the report cites (including those named in the Analysis Scope) come attached to the file byte for byte, together with their input snapshots and the conclusions they drill down from; at export time the server recomputes them from the sealed inputs and records the result on each conclusion's entry. The browser only verifies hashes and shows the server's recomputation; the page doesn't recompute statistics itself. When any input snapshot can't be shown to leave masked columns untouched, the conclusion is given by id only.
|
||||||
- It proves only that the file is internally consistent: numbers, documents, and run records agree with one another and haven't been altered. To check authenticity, compare the seal hash on the page with the seal verification of this run in the system (`GET /api/runs/{id}/verify`); the page says so prominently.
|
- It proves only that the file is internally consistent: numbers, documents, and run records agree with one another and haven't been altered. To check authenticity, compare the seal hash on the page with the seal verification of this run in the system (`GET /api/runs/{id}/verify`); the page says so prominently.
|
||||||
- Events appended after sealing (annotations such as on-demand judging and approval inquiries) are listed separately, with only their type, sequence number, and a content summary, and aren't part of the manifest.
|
- Events appended after sealing (annotations such as on-demand judging and approval inquiries) are listed separately, with only their type, sequence number, and a content summary, and aren't part of the manifest.
|
||||||
- When a data source has masked columns, query snapshots that can't be shown to leave them untouched, and caliber cards with inputs from masked columns, are given by id only, without content. Masking only reduces exposure and is not a security boundary.
|
- When a data source has masked columns, query snapshots that can't be shown to leave them untouched, and caliber cards with inputs from masked columns, are given by id only, without content. Masking only reduces exposure and is not a security boundary.
|
||||||
@@ -246,4 +259,14 @@ For two runs of the same workflow (this week's and last week's weekly report, sa
|
|||||||
- When the caliber and data both changed and a strict split is possible, the caliber effect and the data effect are given separately, and they add up to the total change.
|
- When the caliber and data both changed and a strict split is possible, the caliber effect and the data effect are given separately, and they add up to the total change.
|
||||||
- For metrics in the data-only, parameters-only, and caliber-and-data classes, when attribution is possible, the dimension values (regions, say) with the largest changes are listed, up to 10. Every item traces back to the query snapshots on both sides; queries run by verification take no part in the comparison.
|
- For metrics in the data-only, parameters-only, and caliber-and-data classes, when attribution is possible, the dimension values (regions, say) with the largest changes are listed, up to 10. Every item traces back to the query snapshots on both sides; queries run by verification take no part in the comparison.
|
||||||
- Values from masked columns never appear in the result, and metrics that use masked columns aren't split.
|
- Values from masked columns never appear in the result, and metrics that use masked columns aren't split.
|
||||||
|
- When the graph has analysis cards, the two runs' analysis cards are aligned by node and item, listing each run's status, verdict, direction, and verdict phrase; for cards executed several times in a loop, the last execution counts.
|
||||||
- When a scheduled run finishes successfully and is sealed, it is compared once automatically with the same schedule's last successful run; the trigger history and notifications show only which run was the baseline and how many metrics fall into each class, never the actual numbers.
|
- When a scheduled run finishes successfully and is sealed, it is compared once automatically with the same schedule's last successful run; the trigger history and notifications show only which run was the baseline and how many metrics fall into each class, never the actual numbers.
|
||||||
|
|
||||||
|
### Analysis claim check
|
||||||
|
|
||||||
|
Judgment words in a report, such as "mainly from East", "outside the usual fluctuation", "rising steadily", or "the difference holds up", must cite a matching analysis conclusion and agree with its verdict; a stated cause needs a verbatim knowledge-base quote or a run input as its basis, or else it is written as a conjecture to be verified.
|
||||||
|
|
||||||
|
- The report-writer node checks this sentence by sentence while writing, with a fixed lexicon and binding rules: no model call, no cost. An exit in citation mode checks again independently.
|
||||||
|
- By default, the report goes back for one rewrite and what remains is marked in the report (visible in the evidence panel, the audit, and exports), without affecting the issuance tier; set to strict, it rewrites up to the maximum number of repairs, and what remains counts as an issuance gap and downgrades the issuance. The governed level requires strict (gate G3).
|
||||||
|
- Only new report documents that carry this check's record are checked; verification results for reports from before the upgrade don't change.
|
||||||
|
- For the classes of judgment words, the analysis each one needs, and which wordings aren't checked, see [Analysis claim check](analysis.md#analysis-claim-check).
|
||||||
|
|||||||
+12
-2
@@ -2,17 +2,27 @@ English | [Simplified Chinese](../zh-CN/tools.md)
|
|||||||
|
|
||||||
# Tools, MCP, and the Capability Registry
|
# Tools, MCP, and the Capability Registry
|
||||||
|
|
||||||
This page covers the tools workflows can call (built-in tools, custom tools, and MCP), their approval levels, how admins govern them in the capability registry, and how to register endpoints for external systems. It is for builders and admins. The user interface is currently available in Simplified Chinese only.
|
This page covers the tools workflows can call (built-in tools, including the analysis tools, custom tools, and MCP), their approval levels, how admins govern them in the capability registry, and how to register endpoints for external systems. It is for builders and admins. The user interface is currently available in Simplified Chinese only.
|
||||||
|
|
||||||
## Kinds of tools
|
## Kinds of tools
|
||||||
|
|
||||||
- **Built-in tools**: 15 of them.
|
- **Built-in tools**: 23 of them, 5 of which are analysis tools (see below).
|
||||||
- **Custom tools**: an HTTP template, or Python in the sandbox.
|
- **Custom tools**: an HTTP template, or Python in the sandbox.
|
||||||
- **MCP**: connect MCP servers (stdio or http).
|
- **MCP**: connect MCP servers (stdio or http).
|
||||||
- **Data source tools**: each data source's query tool, `db_query__<source>`, plus `db_schema__` for looking up schemas at run time (see [Data Sources](data-sources.md)).
|
- **Data source tools**: each data source's query tool, `db_query__<source>`, plus `db_schema__` for looking up schemas at run time (see [Data Sources](data-sources.md)).
|
||||||
|
|
||||||
Who can configure what: MCP stdio, endpoints and credentials, and capability governance can be changed only by admins; builders can read them, and users can't see them (see the [permission table](identity-and-security.md#permission-table)).
|
Who can configure what: MCP stdio, endpoints and credentials, and capability governance can be changed only by admins; builders can read them, and users can't see them (see the [permission table](identity-and-security.md#permission-table)).
|
||||||
|
|
||||||
|
## Analysis tools
|
||||||
|
|
||||||
|
The five analysis tools are the platform's deterministic analyses: `analysis_decompose` (change decomposition), `analysis_baseline` (fluctuation baseline), `analysis_trend` (trend test), `analysis_differ` (difference test), and `analysis_comparable` (comparability check). They don't query any database; they analyze only results that the agent they are bound to queried during this execution. The platform computes the verdicts with fixed methods and thresholds, and the system writes the verdict phrases (see [Analysis tools for agents](analysis.md#analysis-tools-for-agents)).
|
||||||
|
|
||||||
|
- Their parameters include no thresholds. Binding any one of them injects the platform's built-in analysis method into the agent's system prompt, once.
|
||||||
|
- They are read-only, write nothing, and aren't dangerous tools, so under the default approval policy they don't wait for approval; users can use them in Ask Data too.
|
||||||
|
- They can be bound only to agents: putting one in a Call Tool node is a canvas validation error. For fixed analyses in a template, use an analysis card (see [Analysis cards](analysis.md#analysis-cards)).
|
||||||
|
- Each agent can run at most 12 analyses per execution (changed under Settings → Preferences → Analysis).
|
||||||
|
- They don't connect to data sources: they read snapshots already queried during this execution, and what they return is an analysis conclusion, not a query result.
|
||||||
|
|
||||||
## Approval levels
|
## Approval levels
|
||||||
|
|
||||||
The platform can't tell what MCP and custom tools do, so by default every call to them waits for approval. There are three approval levels: `ask`, `gated`, and `always`.
|
The platform can't tell what MCP and custom tools do, so by default every call to them waits for approval. There are three approval levels: `ask`, `gated`, and `always`.
|
||||||
|
|||||||
@@ -4,6 +4,22 @@ English | [Simplified Chinese](../zh-CN/upgrade-notes.md)
|
|||||||
|
|
||||||
This page lists, newest first, the changes in each update that affect users and operators, and what to do when upgrading. It is for operators and admins. The features themselves are described in the topic documents. The user interface is currently available in Simplified Chinese only.
|
This page lists, newest first, the changes in each update that affect users and operators, and what to do when upgrading. It is for operators and admins. The features themselves are described in the topic documents. The user interface is currently available in Simplified Chinese only.
|
||||||
|
|
||||||
|
## 2026-10, the analysis round
|
||||||
|
|
||||||
|
**What to do**: nothing is required, and the database schema doesn't change. For report templates you will publish again at the governed level, set the report-writer node's Analysis Claim Check to strict; the pre-publish check points it out, and it can be fixed in one click.
|
||||||
|
|
||||||
|
- New analysis capability (see [Analysis](analysis.md)): five deterministic analyses (change decomposition, fluctuation baseline, trend test, difference test, comparability check). The system computes the verdicts and writes the verdict phrases, and reports cite them with `[[a:A1]]`; they are clickable and can be recomputed.
|
||||||
|
- A new node, the analysis card (`analysis`), and five analysis tools; built-in tools go from 18 to 23. A new template, 11 Anomaly Analysis (Traceable), is added to the template list automatically on the first start after upgrading. Templates are installed once by name: in existing deployments, template 9, Traceable Weekly Report, isn't updated, so when publishing it at the governed level, add the analysis claim check as the pre-publish check suggests.
|
||||||
|
- Report-writer nodes check analysis claims by default (rewrite once, then mark): sentences whose judgment words such as "mainly from", "unusual", or "rising steadily" don't cite a matching analysis conclusion, or whose causes have no basis, go back to the writer for one rewrite; what remains is marked in the report without affecting the issuance tier. The writing rules gain a passage about judgment words, including for reports in graphs without any analysis: when there are no analysis conclusions, the writer is told not to make such judgments and to state only numbers and changes. To skip the check, set the report-writer node's Analysis Claim Check to off.
|
||||||
|
- The governed level requires the Analysis Claim Check to be strict (one more item under gate G3). Versions already published keep running as usual, checked with the default rewrite-once-then-mark; the next publish is checked against the new rule.
|
||||||
|
- Reports that use analysis conclusions get a system-written Analysis Scope section at the end, outside the body. Report documents also record which disposition and which version of the lexicon their analysis claims were checked with: only new documents that carry this record are checked for it on verification, so verification results for reports from before the upgrade don't change.
|
||||||
|
- An exit in citation mode now collects the metrics the report cites: inputs that can't be traced back to raw data count as gaps and downgrade the issuance in formal runs at the governed level; at other levels they are still only stated in the issuance declaration.
|
||||||
|
- A caliber card metric that uses two queries differing only in their dates gets a comparability check automatically: at an exit with a contract, incomparable periods count as a gap and downgrade the issuance, and conditionally comparable ones are only flagged.
|
||||||
|
- The report export and the verifiable report include analysis conclusions and their input snapshots, recomputed from the sealed inputs for verification.
|
||||||
|
- Settings gain a Preferences → Analysis section (see [Analysis settings](configuration.md#analysis-settings)). It is snapshotted into a run only when the graph has an analysis card or an agent with analysis tools, so the start records of other runs don't change at all.
|
||||||
|
- Ask Data: Copilot gains a graph shape for analytical questions, and its self-check reminds it to give the querying agent the analysis tools (one round only). The review merges several analyses that couldn't be done into one "n analyses couldn't be done" note, without an automatic redo; analysis calls that were wrong but later fixed no longer count as a gap, and ones never fixed no longer trigger a redo of the whole turn.
|
||||||
|
- Evaluation gains contribution and verdict expectations, and the scoring rules move to version 3 (see [Regression evaluation](evaluation-and-ledger.md#regression-evaluation)).
|
||||||
|
|
||||||
## 2026-10, orchestration improvements wave 2: PostgreSQL and multiple processes
|
## 2026-10, orchestration improvements wave 2: PostgreSQL and multiple processes
|
||||||
|
|
||||||
**What to do**: in a multi-process deployment, stop all worker processes before upgrading; pass values that code nodes need explicitly; don't delete `.agentlab-data` in the data directory.
|
**What to do**: in a multi-process deployment, stop all worker processes before upgrading; pass values that code nodes need explicitly; don't delete `.agentlab-data` in the data directory.
|
||||||
|
|||||||
@@ -40,7 +40,7 @@ When motion is turned off at the system level (`prefers-reduced-motion`), the li
|
|||||||
|
|
||||||
### Built-in templates
|
### Built-in templates
|
||||||
|
|
||||||
There are 9 built-in templates, from Minimal Q&A to Traceable Weekly Report. They run as they are and also serve as executable documentation of how each kind of node is used.
|
There are 11 built-in templates, from Minimal Q&A to Anomaly Analysis (Traceable). They run as they are and also serve as executable documentation of how each kind of node is used. Templates 9, Traceable Weekly Report, and 11, Anomaly Analysis (Traceable), can be published at the governed level as they are: the first demonstrates caliber cards and issuance by citation; the second demonstrates the analysis card, which takes two queries, judges whether this week's change is within the usual fluctuation and which region it mainly came from, and leaves the report to cite only the verdict phrases (see [Analysis](analysis.md#analysis-cards)).
|
||||||
|
|
||||||
## Node types
|
## Node types
|
||||||
|
|
||||||
@@ -51,9 +51,9 @@ There are 9 built-in templates, from Minimal Q&A to Traceable Weekly Report. The
|
|||||||
| Execution | `tool` `code` `merge` | Call a tool directly / execute code in the sandbox / merge the results of several queries (aligned by key across databases) |
|
| Execution | `tool` `code` `merge` | Call a tool directly / execute code in the sandbox / merge the results of several queries (aligned by key across databases) |
|
||||||
| Control | `branch` `loop` `subgraph` | Conditional or semantic branch / iteration and conditional loop / nested workflow |
|
| Control | `branch` `loop` `subgraph` | Conditional or semantic branch / iteration and conditional loop / nested workflow |
|
||||||
| Context | `memory` `retrieve` `transform` | Long-term memory read/write / knowledge retrieval / data shaping |
|
| Context | `memory` `retrieve` `transform` | Long-term memory read/write / knowledge retrieval / data shaping |
|
||||||
| Gatekeeping | `human` `validate` `metrics` | Human-in-the-loop / Schema validation and automatic rework / caliber card |
|
| Gatekeeping | `human` `validate` `metrics` `analysis` | Human-in-the-loop / Schema validation and automatic rework / caliber card / analysis card (a fixed set of deterministic analyses on upstream query results) |
|
||||||
|
|
||||||
The `output` node validates structured output against a JSON Schema; when the output doesn't conform, the errors go back to the model for automatic rework. For `metrics` (caliber card) and `report` (report writer), see [Evidence and Issuance](evidence.md); for `merge` (merge query), see [Data Sources](data-sources.md#cross-database-merge).
|
The `output` node validates structured output against a JSON Schema; when the output doesn't conform, the errors go back to the model for automatic rework. For `metrics` (caliber card) and `report` (report writer), see [Evidence and Issuance](evidence.md); for `analysis` (analysis card), see [Analysis](analysis.md#analysis-cards); for `merge` (merge query), see [Data Sources](data-sources.md#cross-database-merge).
|
||||||
|
|
||||||
## Passing data between nodes
|
## Passing data between nodes
|
||||||
|
|
||||||
@@ -125,7 +125,7 @@ Approvals, supplementary input, draft editing, action confirmations, code and to
|
|||||||
|
|
||||||
## Agent nodes
|
## Agent nodes
|
||||||
|
|
||||||
An `agent` node is a model loop with tools. An allowlist on the node decides which tools it can use (see [Tools](tools.md)). An agent with `output_schema` can turn on `cite_fields`, so each field records which cell of which query it came from (see [Evidence and Issuance](evidence.md#the-data-retrieval-chain)).
|
An `agent` node is a model loop with tools. An allowlist on the node decides which tools it can use (see [Tools](tools.md)). An agent with `output_schema` can turn on `cite_fields`, so each field records which cell of which query it came from (see [Evidence and Issuance](evidence.md#the-data-retrieval-chain)). An agent that queries data can also be given the analysis tools: breaking down changes and judging anomalies and trends is computed by the platform with fixed methods and thresholds, the system writes the verdict phrases, and reports cite them (see [Analysis tools for agents](analysis.md#analysis-tools-for-agents)).
|
||||||
|
|
||||||
### Guardrails
|
### Guardrails
|
||||||
|
|
||||||
@@ -186,7 +186,7 @@ On top of the governance check, the governed level adds evidence gates. They loo
|
|||||||
|
|
||||||
- **G1** At an exit with a contract, the text in output fields may come only from report-writer nodes.
|
- **G1** At an exit with a contract, the text in output fields may come only from report-writer nodes.
|
||||||
- **G2** Model-written text doesn't bypass the report-writer node to flow straight into the exit.
|
- **G2** Model-written text doesn't bypass the report-writer node to flow straight into the exit.
|
||||||
- **G3** The report-writer node sets `numbers: strict`, `on_violation: fail`, and `claims: require_citation` (or `judge`, with a budget).
|
- **G3** The report-writer node sets `numbers: strict`, `on_violation: fail`, `claims: require_citation` (or `judge`, with a budget), and `analysis_claims: strict` (the analysis claim check set to strict; see [Analysis claim check](analysis.md#analysis-claim-check)).
|
||||||
- **G4** Caliber card inputs don't come from sandbox code in the compute role.
|
- **G4** Caliber card inputs don't come from sandbox code in the compute role.
|
||||||
- **G5** Agents that feed a caliber card have `output_schema` set and `cite_fields` turned on.
|
- **G5** Agents that feed a caliber card have `output_schema` set and `cite_fields` turned on.
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,204 @@
|
|||||||
|
[English](../en/analysis.md) | 简体中文
|
||||||
|
|
||||||
|
# 分析能力
|
||||||
|
|
||||||
|
这篇讲报告里的分析判断从哪来:「主要来自华东」「超出常规波动」「持续上升」「差异站得住」这类判断怎么由系统算出来、怎么写进报告、怎么核对。包括五种分析各回答什么问题、按什么标准判、算不了时怎么说,Agent 的分析工具和分析卡两个入口,报告里的引用写法,判断词的核对,报告末尾的「分析范围」,以及设置和限制。给搭建者和看报告的人看。
|
||||||
|
|
||||||
|
## 是什么、为什么
|
||||||
|
|
||||||
|
报告里的数字由系统照快照填进去,叙述层不做算术(见[证据与出具](evidence.md#基本思路))。可报告要说的不只是数,还有判断:变化主要来自哪里、这次算不算异常、是不是一直在涨、两组的差别是不是真的。以前这些判断由写报告的模型自己下,它挑着证据说,读的人核对不了。
|
||||||
|
|
||||||
|
分析能力把判断也收回系统:
|
||||||
|
|
||||||
|
- **判定由系统算**:五种分析是平台的确定性代码,只读这次运行里交回过的查询快照,按固定的方法和阈值算出判定。模型只决定算什么(比哪两期、拆哪个维度、往哪一段下钻),改不了方法和阈值。
|
||||||
|
- **判定语由系统写**:每份分析结论带一句系统按模板写好的判定语,报告引用它(`[[a:A1]]`),和数字一样点得开出处。写作者自己写「主要来自」「异常」这类判断时,系统逐句核对它挂没挂对应的分析结论、判定对不对得上(见[分析说法核对](#分析说法核对))。
|
||||||
|
- **照实说**:算不了就说为什么——历史不够、本期没过完、查询被截断、两期不可比、样本太少——不硬给结论。算不了本身也是一条结论,写作者照实写。
|
||||||
|
- **留痕**:每份分析结论连同方法与版本、参数、实际用的阈值、输入的格、现查的表范围、结果和判定,存成内容寻址的工件,进证据台账、随运行封存;证据面板、报告导出、可验证报告都按同一个函数复算核对(见[点开一个数看到什么](evidence.md#点开一个数看到什么))。
|
||||||
|
|
||||||
|
拆解只说明变化来自哪里,说明不了为什么。原因要有知识库原话或运行输入作依据,否则写成待核实的推测。
|
||||||
|
|
||||||
|
## 五种分析
|
||||||
|
|
||||||
|
每做一次分析产出一份分析结论,写作目录里按先后编成 A1、A2……(和查询的 Q1、知识库的 K1 同一套编号规则)。一份结论有:判定(判定码、点名的段、方向)、判定语、判定标准(照实际阈值写的一句)、具名的数(占比、总变化这类)、结果表、附带说明(抵消、辛普森悖论、样本少、历史新高……,最多两条接在判定语后面)。
|
||||||
|
|
||||||
|
下面括号里的阈值都是缺省值,在「设置 → 偏好设置 → 分析」里改(见[分析设置](configuration.md#分析设置))。
|
||||||
|
|
||||||
|
### 变化拆解
|
||||||
|
|
||||||
|
回答「变化从哪来」。三种模式:
|
||||||
|
|
||||||
|
| 模式 | 用在 | 怎么拆 |
|
||||||
|
|---|---|---|
|
||||||
|
| 可加 | 销售额、订单数这类能相加的量 | 按维度(一到两列)算各段对总变化的贡献和占比 |
|
||||||
|
| 比率 | 退款率、客单价这类比率 | 拆成结构变化(各段的占比变了)和各段自身的变化,两者之和精确等于比率的变化 |
|
||||||
|
| 乘积 | 销售额 = 订单数 × 客单价 | 按对数平均(LMDI)拆成各因子的贡献 |
|
||||||
|
|
||||||
|
- 判定:一段占总变化的一半以上(0.5)算主要来源(「主要来自华东」);没有这样的一段、前几段合计占八成以上(0.8)算集中(段数最多是全部段数的两成,至少 2 段);总变化不到基期的 0.5% 算基本持平;其余算分散。比率拆解判主要来自结构变化、主要来自各段自身的变化,或两者都有影响;乘积拆解判主要来自哪个因子。
|
||||||
|
- 给了总数时,各段之和要和总数对得上(按总数的书写精度容差)才判断变化来自哪里;对不上就不判,判定语写差多少。没给总数时以各段之和为总数,判定标准里写明。
|
||||||
|
- 反向的段合计超过总变化的四分之一(0.25)时附一句互相抵消了一部分;比率拆解里各段大多在降、整体却在升(或反过来)时附一句辛普森悖论的说明;分母不到 30 的段标「分母小」,不参与这个判断。
|
||||||
|
- 下钻:一次拆解里的某一段可以接着往下拆一层(华东按渠道),这一层各段之和要等于上一层那一段的值,最多下钻 3 层。下钻的查询由 Agent 自己写、只查那一段,系统按上一层那一段的值核对闭合。
|
||||||
|
- 判定语举例:「销售额下降 194(-9.2%),主要来自华东(-120,占 61.9%)」。
|
||||||
|
|
||||||
|
### 波动基线
|
||||||
|
|
||||||
|
回答「这次的变化大不大、算不算异常」:把本期放回它自己的历史里看。
|
||||||
|
|
||||||
|
- 输入一条按期次排的序列(日、周、月、季)。缺省比的是和上一期比的变化率,日数据和上周同一天比,避开星期效应;存量、比率可以比值本身;要同比就和 52 周前、12 个月前比,历史要相应够长。
|
||||||
|
- 用历史的中位数和中位绝对偏差(稳健标准差 = 1.4826 × 中位绝对偏差):本期和中位数的差不到 2 倍稳健标准差算常规波动,2 到 3 倍算超出,3 倍以上算明显超出。用中位数不用均值:促销周这类极端值会把均值和标准差拉宽,正常的周、异常的周都显得不异常。
|
||||||
|
- 历史至少 8 期、最多用 52 期;本期是历史里最高或最低的,附一句历史新高、新低。
|
||||||
|
- 判定语举例:「销售额本期变化 -15.8%,明显超出过去 24 周的常规波动范围(-0.5% 至 +0.4%);本期的销售额是过去 24 周里最低的」。
|
||||||
|
|
||||||
|
### 趋势判断
|
||||||
|
|
||||||
|
回答「是在持续变,还是一次波动」。
|
||||||
|
|
||||||
|
- 取最后连着的一段(至少 6 期):Mann–Kendall 检验的双侧 p 值小于 0.05 时,按 Theil–Sen 斜率的正负判上升或下降,否则判没有明显趋势。末尾连续 3 期以上同向时,判定语接「已连续 3 周下降」,和趋势反向时写「但最近已连续 3 周上升」。
|
||||||
|
- Mann–Kendall 假设各期独立,业务序列常有自相关,p 会偏小,所以判定语只说「呈上升趋势」,不说「显著上升」。日数据附一句星期效应,建议按周汇总。
|
||||||
|
- 判定语举例:「过去 12 周线上销售额呈上升趋势(每周约 +2.1%),已连续 4 周上升」。
|
||||||
|
|
||||||
|
### 差异检验
|
||||||
|
|
||||||
|
回答「这两组(或同一组的两期)真的不一样,还是样本波动」。
|
||||||
|
|
||||||
|
- 比率(转化率、退款率)给成功数和试验数:每组算 Wilson 区间;各格的期望频数都不小于 5 时用合并比例的 z 检验,否则用 Fisher 精确检验。
|
||||||
|
- 均值给每组的均值、标准差、样本数,或者合计、平方和、样本数(SQLite 没有标准差函数时这样写),用 Welch t 检验;也可以给逐行的值(不超过 1000 行、没截断),系统自己算,另做一遍去掉最远那个值的敏感性检查。
|
||||||
|
- 双侧 p 值小于 0.05 算差异站得住;一次比了好几对时按 Holm 校正。均值有一组样本不到 30、比率用了 Fisher 时附一句样本少。
|
||||||
|
- 判定语不出现「p 值」「显著性」这些词:站得住写「差异站得住」,否则写「差异可能只是样本波动(样本量 230、40)」。p 值和区间在结果表和证据面板里。
|
||||||
|
|
||||||
|
### 可比性检查
|
||||||
|
|
||||||
|
回答「这两期能不能直接比」。每项判通过、提醒、不通过或核对不了:
|
||||||
|
|
||||||
|
| 项 | 怎么判 |
|
||||||
|
|---|---|
|
||||||
|
| 同口径 | 两份查询只差日期才算同口径;结构不同,或者除了日期还筛了别的(华东对华南),不通过 |
|
||||||
|
| 窗口长度 | 两期天数一样;可加的流量不一样就不通过,比率、存量只提醒 |
|
||||||
|
| 整期 | 本期过完了、表里的数据到了;流量没到齐不通过(「本期只有 3 天数据」),其余提醒 |
|
||||||
|
| 星期构成 | 按天取、天数不是 7 的倍数时,两期的周末天数要一样,不一样提醒 |
|
||||||
|
| 数据问题 | 查询区间里没有数据不通过,SQL 检查有错误提醒 |
|
||||||
|
|
||||||
|
全部通过(或核对不了)判可比,只有提醒判有条件可比,有一项不通过判不可比。核对不了的项不判不可比,判定语里写明哪几项核对不了。表的实际日期范围现查(只读、限时、遮罩列不查,失败退回剖析记下的范围),查到的范围记进结论,复算按记下的值。
|
||||||
|
|
||||||
|
变化拆解给的是两份查询时,先自动做一次可比性检查;波动基线、趋势判断和单份查询的变化拆解看本期齐不齐时,用的是其中「整期」那一项。口径卡里一个指标用到两份同形、只差日期的查询(本周对上周)时,也自动做一次,结果挂在指标上(见[口径卡](evidence.md#口径卡))。
|
||||||
|
|
||||||
|
## 算不了的时候
|
||||||
|
|
||||||
|
算不了不让 Agent、节点失败,记成一条结论,判定语写「波动基线没有做成:历史只有 5 期,至少要 8 期」这样的一句,进台账、能引用、能核对:
|
||||||
|
|
||||||
|
| 原因 | 什么时候 |
|
||||||
|
|---|---|
|
||||||
|
| 截断 | 输入的查询只取回了前一部分行,各段之和不是完整的和 |
|
||||||
|
| 段太少 | 拆开后不到 2 段 |
|
||||||
|
| 历史不够 | 有效的历史期数不到最少期数 |
|
||||||
|
| 本期不完整 | 本期没过完、数据没到、不到整周,或者查询只取了本期的一部分 |
|
||||||
|
| 缺期 | 本期和比较期之间缺期;趋势判断用的最后几期不连着 |
|
||||||
|
| 期次认不出 | 期次列的写法认不出(带时刻的、写法混着的) |
|
||||||
|
| 比较期为 0 | 算不了变化率 |
|
||||||
|
| 有值不为正 | 乘积拆解要求各值都为正 |
|
||||||
|
| 两边不是同一个量 | 两份查询结构不同,不是同一个量的两期,不拆 |
|
||||||
|
| 拿不到 p 值 | 差异检验一对都算不出 p 值(两组各只有 1 个样本、样本量为 0) |
|
||||||
|
| 上一层没做成、这次的数据里没有 | 分析卡里下钻的上一层算不了,或者这期的数据里没有要用的期次、段、组、总数那一格 |
|
||||||
|
|
||||||
|
调用写错不记结论:列不存在、拿遮罩列当维度或度量、比率当成可加的量、同一段同一期有好几行、参数写错、这个 Agent 这次执行的分析次数用完了。Agent 收到一句怎么改的说明,改了再调;分析卡里是模板写错了,整张卡失败并说清是哪一项。
|
||||||
|
|
||||||
|
期次认这几种写法:日期、`YYYY-MM`、`YYYY-Www`、`YYYY-Qn`、整数序号。按周汇总时,期次最好在 SQL 里由日期列算出来,写每周第一天的日期,或 ISO 周 `YYYY-Www`。同样写成 `2026-W42`,SQLite 的 `%W`、MySQL 的 `%u` 算出的和 ISO 周落在不同的日子上,系统按 SQL 里的格式串认出算法、算出真实的起止;直接取表里现成的周号文本时认不出算法,本期齐不齐核对不了。
|
||||||
|
|
||||||
|
## 两个入口
|
||||||
|
|
||||||
|
两个入口共用同一套分析、同一种结论、同一张判定语表。
|
||||||
|
|
||||||
|
### Agent 的分析工具
|
||||||
|
|
||||||
|
问数据和画布上的 agent 可以绑五个分析工具:`analysis_decompose`(变化拆解)、`analysis_baseline`(波动基线)、`analysis_trend`(趋势判断)、`analysis_differ`(差异检验)、`analysis_comparable`(可比性检查)。要看了数才知道拆哪个维度时用它。
|
||||||
|
|
||||||
|
- 参数只有「算什么」:分析本节点查过的哪个结果(`Q3`,和交回模型的「【证据 Q3】」同一个编号)、各列的角色、比哪两期、总数那两格、下钻的上一层、度量的叫法,和一句这次在验证什么(只在证据面板显示,不进报告正文)。没有任何阈值参数。
|
||||||
|
- 绑了任一个,平台就把内置的分析方法注入 agent 的系统提示:先按问题定基期,做可比性检查和波动基线,再拆因子与结构、按维度由粗到细拆,找到主要来源再下钻一层;拆了不集中也是结论;不作因果断言;算不了照实说;最后列出做过的分析和判定。方法随版本升级。
|
||||||
|
- 交回 agent 的是「【分析 A2】」加判定、判定语、关键的数、附带说明或算不了的原因;整份结论进证据台账。
|
||||||
|
- 每个 Agent 每次执行最多分析 12 次、最多下钻 3 层。
|
||||||
|
- 分析工具只分析绑了它的 agent 自己查过的结果,不能放进「调用工具」节点(模板里要固定做的分析用分析卡)。协作团队的成员也能用,但结论不进证据台账,报告引用不了。
|
||||||
|
- 使用者(user 角色)的问数据里也能用:分析工具不查库,只读已经按遮罩处理过的查询结果,遮罩列不当维度、不当度量。
|
||||||
|
|
||||||
|
### 分析卡
|
||||||
|
|
||||||
|
分析卡(`analysis` 节点,和口径卡平级)对上游查询节点的结果固定做几项分析,用于模板里每次都要做的分析,比如周报的异动:这一周的变化在不在常规波动范围内、主要来自哪个地区。
|
||||||
|
|
||||||
|
- 每一项写明分析方法和模式、要分析哪个上游查询节点、参数和这一项在验证什么。输入只能是调用数据库查询工具的「调用工具」节点或「合并查询」,而且连在分析卡之前;Agent 可能查了好几次,不能当输入。参数和分析工具的一样,可以用 `{{ input.x }}` 按运行输入选期次。
|
||||||
|
- 阈值不写在分析项里。模板要固定阈值时写在卡上的「阈值覆盖」里,随版本固定;它和这次运行的分析设置冲突时,卡失败并说清冲突在哪。
|
||||||
|
- 产出 `nodes.<卡>.results.<项>`:状态、判定码、方向、具名的数、判定语。下游条件分支可以按判定走,比如超出常规波动就转人工。
|
||||||
|
- 一项算不了不让卡失败,那一项照实记成算不了,接着做下一项。
|
||||||
|
- 分析卡的结果不作口径卡的输入:口径卡读它,按追不到原始数据处理。指标在口径卡里按格从查询结果取,分析结论在报告里直接引用。
|
||||||
|
- 受管级别要求分析卡的输入是调用数据库查询工具的「调用工具」节点或「合并查询」,发布时就能确定分析的是平台执行的查询。
|
||||||
|
- 运行对比按分析卡的节点和项对齐,列出两次的判定(见[运行对比](evidence.md#运行对比))。
|
||||||
|
|
||||||
|
内置模板 ⑪「异动分析(可追溯)」就是这样搭的:整体按周、按地区两周取数 → 分析卡(波动基线、带总数的变化拆解)→ 报告撰写 → 出具,拿来就能按受管级别发布(见[内置模板](workflows.md#内置模板))。
|
||||||
|
|
||||||
|
## 在报告里引用
|
||||||
|
|
||||||
|
```
|
||||||
|
[[a:A1]] A1 的判定语,由系统按判定渲染,一个点得开的片段
|
||||||
|
[[a:A1.top_share|pct]] A1 的一个具名的数,换算后缀同 [[m:]]
|
||||||
|
[[v:A1.r0.share]] A1 结果表里的一格
|
||||||
|
[[table:A1 cols=地区,delta,share rows=0-4]] A1 的结果表
|
||||||
|
[[see:A1]] 这句的依据,挂在句末,不渲染
|
||||||
|
```
|
||||||
|
|
||||||
|
- 写作目录里「分析结论」一段列出每份结论的编号、判定语、具名的数、结果表的前 10 段、附带说明或算不了的原因。
|
||||||
|
- 分析卡的结论总能引用。受管级别的正式出具里,Agent 用分析工具现场做的分析和它查的格同一个规矩:出具契约开了「单元格引用」才能引用(句末依据照样能挂)。
|
||||||
|
- 判定语里的度量叫法是 Agent 或模板给的,系统先去掉里面的数字、百分号、单位和引用标记,截到 20 字以内:系统写的字里不会夹带模型写的数。
|
||||||
|
|
||||||
|
## 分析说法核对
|
||||||
|
|
||||||
|
报告撰写节点的「分析说法核对」(`analysis_claims`)逐句核对判断词:确定的词表加绑定,不调用模型。
|
||||||
|
|
||||||
|
| 判断词的类 | 例子 | 要挂的分析结论和判定 |
|
||||||
|
|---|---|---|
|
||||||
|
| 来源 | 主要来自、集中在、拖累、带动、贡献最大 | 变化拆解,判主要来源或集中;点名的段要在判定里,说拖累的那一段要在降 |
|
||||||
|
| 分散、抵消、结构 | 分散在各地区、此消彼长、结构变化 | 变化拆解,判分散、有抵消、主要来自结构变化 |
|
||||||
|
| 异常、新高新低、正常 | 异常、大幅、激增、历史新高、正常波动 | 波动基线,判超出或明显超出(方向对得上)、新高新低、常规波动 |
|
||||||
|
| 趋势、无趋势 | 持续上升、连续三周下降、没有明显趋势 | 趋势判断,方向、连续期数对得上 |
|
||||||
|
| 差异、无差异 | 显著高于、确实更低、差异不大 | 差异检验,差异站得住(方向对得上)或可能只是样本波动 |
|
||||||
|
| 因果 | 因为、由于、导致、受…影响 | 同一句有知识库原话或运行输入作依据,或者写成推测(可能、待核实) |
|
||||||
|
|
||||||
|
- 一句的依据是句中的 `[[a:…]]`、`[[v:A…]]`、`[[table:A…]]` 和句末的 `[[see:A…]]`。否定会翻到反面:「并不异常」按正常核。
|
||||||
|
- 不核的:说的是构成而不是变化(「本周订单主要来自线上渠道」)、建议和计划、业务状态里的「异常」(「部分订单异常」);知识库原话、系统渲染的判定语、标题和表格。
|
||||||
|
- 来源和因果的分界:「下降主要由华东造成」挂了按地区的拆解、点名的是拆解里的段,按来源核,这是记账意义上的「来自」;「由于促销结束导致下降」点名的不是段,按因果核。
|
||||||
|
- 违规三种:判断词没挂对应的分析结论;挂了,但判定、点名的段或方向对不上(挂了算不了的结论也算);原因没有依据。
|
||||||
|
|
||||||
|
| 取值 | 做法 |
|
||||||
|
|---|---|
|
||||||
|
| `check`(缺省,问数据也是它) | 交回写作者重写一次,仍有的标在报告里,不影响出具档位 |
|
||||||
|
| `strict` | 按最多重写次数重写,仍有的计入出具缺口、出具降档;受管级别要求它(门禁 G3) |
|
||||||
|
| `off` | 不核 |
|
||||||
|
|
||||||
|
- 写作目录里有分析结论时,写作者被告知判断照判定写、最好直接引用判定语;没有时被告知不下这类判断,只陈述数字和变化。
|
||||||
|
- 引用模式的出口独立再核一遍,和组装时用同一个核对。报告文档记下按哪种处置、哪一版说法表核:只有带这一项的文档在复核时核这一项,升级前的报告复核结果不变。
|
||||||
|
- 宁可漏报:只收强判断词。漏掉的细处由结论句裁判兜底(见[结论句](evidence.md#结论句));裁判摘录里有分析结论时,裁判另被告知判定是系统算的,只判句子说得对不对。
|
||||||
|
|
||||||
|
## 分析范围
|
||||||
|
|
||||||
|
报告用到了分析结论(这次运行有分析结论)时,系统在报告末尾附一段「分析范围」。写作者改不了,随文档封存,编号点得开:
|
||||||
|
|
||||||
|
- **现象**:可比性检查、波动基线、趋势判断的判定语。
|
||||||
|
- **拆过**:每份变化拆解按下钻链列出维度和判定。
|
||||||
|
- **排除**:判定为分散的拆解、差异可能只是样本波动的检验——拆了没发现集中也是结论。
|
||||||
|
- **算不了**:各项算不了的原因。
|
||||||
|
- **没拆**:候选维度里没拆过的。候选维度从数据目录确定地算:拆解所用查询里的事实表上标成属性、状态,不遮罩、有码值表(取值不超过 50 个,或者已列全)的列,加上它关联的维度表上同样的列。拆解用的列对不上目录时只说候选维度核对不了,不乱列。
|
||||||
|
- **检验**:这次做了几次差异检验,是否已按次数校正。
|
||||||
|
|
||||||
|
Agent 写的「这次在验证什么」不进这一段,只在证据面板显示。报告撰写节点的「分析范围」(`analysis_scope`)缺省是有分析结论时附上,可以设成不附。
|
||||||
|
|
||||||
|
## 设置
|
||||||
|
|
||||||
|
「设置 → 偏好设置 → 分析」里改判定用的阈值、Agent 每次执行最多分析几次、最多下钻几层、列出前几段,各项的叫法和缺省见[分析设置](configuration.md#分析设置)。
|
||||||
|
|
||||||
|
- 发起运行时把这一组快照进运行(只在图里有分析卡、或者 agent 绑了分析工具时),恢复和接着跑沿用同一份。
|
||||||
|
- Agent 改不了阈值:分析工具的参数里没有阈值。模板可以在分析卡上写阈值覆盖。
|
||||||
|
- 每份结论记下实际用的阈值,判定标准照这些值写出来,例如「一段占总变化的一半以上算主要来源」。
|
||||||
|
|
||||||
|
## 限制
|
||||||
|
|
||||||
|
- 这一轮没有:预测;相关分析;维度层级进数据目录;数值列的剖析;双盲复算、反方审稿、审批质询的核查 agent 使用分析工具;分析结论参与口径账本的比对和召回(分析结论里的数不标口径,见[口径账本](evaluation-and-ledger.md#口径账本))。
|
||||||
|
- 趋势判断假设各期独立,自相关的序列 p 会偏小。
|
||||||
|
- 候选维度只认码值表:剖析目前不记每列的取值数,没有码值表的属性列不进候选。
|
||||||
|
- 说法表只收强判断词,换个说法可能漏过,漏过的由结论句裁判和读报告的人兜底。
|
||||||
|
- 协作团队成员做的分析不进证据台账,报告引用不了。
|
||||||
+14
-1
@@ -25,6 +25,17 @@
|
|||||||
- 助手只给了思考、一个操作都没给时,自动重来一次。
|
- 助手只给了思考、一个操作都没给时,自动重来一次。
|
||||||
- 每一轮的花费包括助手建图、挑表和复核的模型调用。
|
- 每一轮的花费包括助手建图、挑表和复核的模型调用。
|
||||||
|
|
||||||
|
## 分析类问题
|
||||||
|
|
||||||
|
问的是为什么、怎么变了、哪里拖累、谁贡献最大、是否异常、波动大不大、是不是持续在涨、是不是真的更高这类问题时,助手搭的图里,查库的 agent 另绑五个分析工具(见[分析能力](analysis.md)):
|
||||||
|
|
||||||
|
- agent 先按问题定比哪两期(问题没说就比上一个整期),再照平台内置的分析方法做:可比性检查、放回历史看、按维度拆解,找到主要来源再下钻一层。判定由平台按固定的方法和阈值算,agent 只决定算什么。
|
||||||
|
- 报告按分析结论写,判断直接引用系统写好的判定语。写作者自己写了「主要来自」「异常」这类判断的,系统逐句核对:没挂对分析结论的交回重写一次,仍有的标在答案里,不影响出具档位(见[分析说法核对](analysis.md#分析说法核对))。
|
||||||
|
- 用到分析结论的答案末尾,系统附一段「分析范围」:拆过什么、排除了什么、哪些候选维度没拆、哪些算不了(见[分析范围](analysis.md#分析范围))。
|
||||||
|
- 拆解只说明变化来自哪里,说明不了为什么:答案里的原因要有知识库原话作依据,否则写成待核实的推测。
|
||||||
|
- 助手自查时按一张确定的词表认问题是不是分析类,只收明确的问法。是分析类、查库的 agent 却没绑分析工具时,交回助手改一轮;改过一轮仍没绑的照常运行,留一条提醒。
|
||||||
|
- 分析没做成(历史不够、本期没过完……)、判断词重写后仍没挂对的,复核只写一句说明,不自动重来(见[产出复核](#产出复核))。
|
||||||
|
|
||||||
## 产出复核
|
## 产出复核
|
||||||
|
|
||||||
运行结束后,规则先扫一遍事件流,发现异常才调用模型复核。复核可以重组答案,但**不得引入原答案里没有的数字**。
|
运行结束后,规则先扫一遍事件流,发现异常才调用模型复核。复核可以重组答案,但**不得引入原答案里没有的数字**。
|
||||||
@@ -32,10 +43,11 @@
|
|||||||
- 使用者(user 角色)的问数据也会被复核。
|
- 使用者(user 角色)的问数据也会被复核。
|
||||||
- 子工作流里一步失败时,自动复核只报一条。
|
- 子工作流里一步失败时,自动复核只报一条。
|
||||||
- 问数据里的「继续运行」只给搭建者以上。
|
- 问数据里的「继续运行」只给搭建者以上。
|
||||||
|
- 分析的几种缺口——有几次分析没做成、分析调用写错了一直没改对、判断词核对重写后仍有几处、口径卡指标的两期没做成可比性检查——只写说明、不自动重来。一次运行只有这几种时,复核不调用模型,照这几句原话写说明。
|
||||||
|
|
||||||
## 答案里的口径
|
## 答案里的口径
|
||||||
|
|
||||||
口径是一个指标怎么取数、怎么算的约定,固定在口径卡里,带名字和版本。答案封存后,系统在后台把答案里的每个数和已发布的口径比对、标上口径,每一轮底下有一行摘要。整题等价于一份受管口径时,系统自动按口径发起一次正式运行。详见[回归评测与口径账本](evaluation-and-ledger.md#口径账本)。
|
口径是一个指标怎么取数、怎么算的约定,固定在口径卡里,带名字和版本。答案封存后,系统在后台把答案里的每个数和已发布的口径比对、标上口径,每一轮底下有一行摘要。整题等价于一份受管口径时,系统自动按口径发起一次正式运行。分析结论里的数(占比、变化、波动范围)是从查询结果派生的,不和口径比对、不标口径,摘要行也不计它们。详见[回归评测与口径账本](evaluation-and-ledger.md#口径账本)。
|
||||||
|
|
||||||
## 从答案到工作流和应用
|
## 从答案到工作流和应用
|
||||||
|
|
||||||
@@ -60,6 +72,7 @@
|
|||||||
|
|
||||||
- 改图时只给出改动操作,不重写整张图。
|
- 改图时只给出改动操作,不重写整张图。
|
||||||
- 交付前按运行时同一套规则自查。查出会运行失败的问题,先交回去改;改不好就不自动运行。
|
- 交付前按运行时同一套规则自查。查出会运行失败的问题,先交回去改;改不好就不自动运行。
|
||||||
|
- 问数据里自查还看问题本身:分析类问题、查库的 agent 却没绑分析工具时,提醒一轮(见[分析类问题](#分析类问题))。
|
||||||
- 建图时主动把应当记住的事实写进长期记忆。
|
- 建图时主动把应当记住的事实写进长期记忆。
|
||||||
|
|
||||||
### 按需查看
|
### 按需查看
|
||||||
|
|||||||
@@ -67,7 +67,7 @@ AgentLab 放在反向代理后面、或者要给外部系统回调时,设对
|
|||||||
| 位置 | 内容 |
|
| 位置 | 内容 |
|
||||||
|---|---|
|
|---|---|
|
||||||
| 设置 → 模型接入 | provider 和密钥 |
|
| 设置 → 模型接入 | provider 和密钥 |
|
||||||
| 设置 → 偏好设置 | 运行默认值(默认记忆作用域、默认知识库、危险工具默认需要人工审批);证据裁判(裁判模型和各项上限);核查(核查模型,双盲复算、反方审稿、审批质询的开关和上限,每日金额上限,见[复核套件](evidence.md#复核套件));评测的并发、每题时限、单次和每日金额;运行并发(管理员);口径账本(自动按口径出具的开关和每日上限、口径比对时限、召回上限 `ledger.recall_limit`,见[口径账本](evaluation-and-ledger.md#口径账本));起草模型;应用(每个应用每天的按需运行上限、让使用者建个人应用、允许建分享链接);界面主题 |
|
| 设置 → 偏好设置 | 运行默认值(默认记忆作用域、默认知识库、危险工具默认需要人工审批);证据裁判(裁判模型和各项上限);核查(核查模型,双盲复算、反方审稿、审批质询的开关和上限,每日金额上限,见[复核套件](evidence.md#复核套件));评测的并发、每题时限、单次和每日金额;分析(判定用的阈值、Agent 每次执行最多分析几次、最多下钻几层、列出前几段,见[分析设置](#分析设置));运行并发(管理员);口径账本(自动按口径出具的开关和每日上限、口径比对时限、召回上限 `ledger.recall_limit`,见[口径账本](evaluation-and-ledger.md#口径账本));起草模型;应用(每个应用每天的按需运行上限、让使用者建个人应用、允许建分享链接);界面主题 |
|
||||||
| 设置 → 通用 | 对外访问地址 |
|
| 设置 → 通用 | 对外访问地址 |
|
||||||
| 设置 → 运行环境 | 沙箱候选和可用性、当前选中项和选择原因;时限等当前值 |
|
| 设置 → 运行环境 | 沙箱候选和可用性、当前选中项和选择原因;时限等当前值 |
|
||||||
| 设置 → 身份 | 登录方式;给使用者开启问数据 |
|
| 设置 → 身份 | 登录方式;给使用者开启问数据 |
|
||||||
@@ -77,6 +77,30 @@ AgentLab 放在反向代理后面、或者要给外部系统回调时,设对
|
|||||||
|
|
||||||
其他设置项:`notify.digest_at`(每天待办汇总的时间),以及 agent 的令牌预算、金额预算和步数。
|
其他设置项:`notify.digest_at`(每天待办汇总的时间),以及 agent 的令牌预算、金额预算和步数。
|
||||||
|
|
||||||
|
### 分析设置
|
||||||
|
|
||||||
|
「设置 → 偏好设置 → 分析」一节,管变化拆解、波动基线、趋势判断、差异检验怎么判,以及 Agent 能分析多少次(见[分析能力](analysis.md))。发起运行时快照进运行,恢复和接着跑沿用同一份;分析卡上可以写阈值覆盖,Agent 改不了。
|
||||||
|
|
||||||
|
| 设置项 | 缺省 | 取值范围 | 作用 |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 「主要来源」的占比 | 0.5 | 0.3–0.9 | 一段(一个因子、一种效应)占总变化的比重到这么多,算主要来源 |
|
||||||
|
| 「集中」的合计占比 | 0.8 | 0.5–0.99 | 前几段合计占到这么多算集中;不能小于「主要来源」的占比 |
|
||||||
|
| 记为「抵消」的比例 | 0.25 | 0.05–1 | 反向的段合计超过总变化的这个比例时,附一句互相抵消 |
|
||||||
|
| 「基本持平」的幅度 | 0.005 | 0–0.1 | 总变化不到基期的这个比例,算基本持平 |
|
||||||
|
| 「超出常规波动」的倍数 | 2 | 1–5 | 波动基线:本期离历史中位数这么多倍稳健标准差,算超出常规波动 |
|
||||||
|
| 「明显超出」的倍数 | 3 | 1.5–8 | 到这么多倍算明显超出;要大于「超出常规波动」的倍数 |
|
||||||
|
| 至少几期历史 | 8 | 4–104 | 波动基线的历史不到这么多期,算不了 |
|
||||||
|
| 最多用几期历史 | 52 | 8–260 | 波动基线、趋势判断最多用这么多期;不能小于至少几期历史 |
|
||||||
|
| 趋势至少几期 | 6 | 4–104 | 趋势判断最少要这么多期 |
|
||||||
|
| 趋势的显著性水平 | 0.05 | 0.001–0.2 | p 值小于它才判上升或下降 |
|
||||||
|
| 差异检验的显著性水平 | 0.05 | 0.001–0.2 | 校正后的 p 值小于它算差异站得住 |
|
||||||
|
| 均值比较的最小样本数 | 30 | 2–10000 | 均值比较有一组样本不到这么多,附一句样本少 |
|
||||||
|
| 比率比较的最小期望频数 | 5 | 1–50 | 有格子的期望频数不到它时改用 Fisher 精确检验 |
|
||||||
|
| 比率拆解的最小分母 | 30 | 1–100000 | 分母不到它的段标「分母小」,不参与辛普森悖论的判断 |
|
||||||
|
| Agent 每次执行最多分析几次 | 12 | 1–50 | 用完了,Agent 再调分析工具会收到「已经用完」 |
|
||||||
|
| 最多下钻几层 | 3 | 1–6 | 变化拆解往下钻的层数上限 |
|
||||||
|
| 列出前几段 | 10 | 3–30 | 写作目录、交回 Agent 的结果表列出前几段 |
|
||||||
|
|
||||||
## 沙箱后端
|
## 沙箱后端
|
||||||
|
|
||||||
「设置 → 运行环境」列出全部候选和它们的可用性,并标出当前选中项和「选择原因」。自动选择的顺序:
|
「设置 → 运行环境」列出全部候选和它们的可用性,并标出当前选中项和「选择原因」。自动选择的顺序:
|
||||||
|
|||||||
@@ -16,10 +16,12 @@
|
|||||||
- 一份已发布口径的指标。
|
- 一份已发布口径的指标。
|
||||||
- 从某次问数据运行里认定的查询。
|
- 从某次问数据运行里认定的查询。
|
||||||
- 「没有数据」,或者期望反问。
|
- 「没有数据」,或者期望反问。
|
||||||
|
- 一个变化的主要来源(贡献):期望 SQL 加变化拆解的参数,用和作答同一套分析算出主要来源(一段或前几段)和方向。
|
||||||
|
- 一个判定:期望 SQL 加波动基线、趋势判断、差异检验或可比性检查的参数,用同一套分析算出判定的类——常规波动还是异常,上升、下降还是没有明显趋势,差异站得住还是不足以下结论,可比还是不可比(见[分析能力](analysis.md#五种分析))。
|
||||||
|
|
||||||
规则:
|
规则:
|
||||||
|
|
||||||
- 数值不许人填。
|
- 数值不许人填;贡献和判定两种期望里的段、方向、判定同样不许人填,由期望 SQL 和同一套分析算出,作答前后各算一次。
|
||||||
- 文字要点由裁判判,单列,不计入准确率。
|
- 文字要点由裁判判,单列,不计入准确率。
|
||||||
- 口径升版后,用到它的题在顶部提示。每道题三选一:改为固定新版本、新旧两版各算一份期望、旧题归档另建新题。处置之前,评测时跳过这道题。
|
- 口径升版后,用到它的题在顶部提示。每道题三选一:改为固定新版本、新旧两版各算一份期望、旧题归档另建新题。处置之前,评测时跳过这道题。
|
||||||
- 题集能导入导出。
|
- 题集能导入导出。
|
||||||
@@ -42,7 +44,9 @@
|
|||||||
- **对比**:和上一次或基线比,看逐题转移与归因。
|
- **对比**:和上一次或基线比,看逐题转移与归因。
|
||||||
- **A/B 配对表**:精确 McNemar 检验,只展示,不卡门。
|
- **A/B 配对表**:精确 McNemar 检验,只展示,不卡门。
|
||||||
|
|
||||||
评测运行不进使用统计。评分规则现在是第 2 版;老评测可以在评测详情里按新规则重评(只读封存,不调模型)。
|
贡献和判定两种期望这样评:答案引用了判定对得上的分析结论,算对且可追溯;正文照说法表写对了、但没有引用,算对但没引用;来源、判定对不上或答成相反,算错;期望本身算不了(比如期望 SQL 查出来的历史不够),不能评分。分析结论的输入查询也算答案引用过的查询。
|
||||||
|
|
||||||
|
评测运行不进使用统计。评分规则现在是第 3 版(加了贡献、判定两种期望);老评测可以在评测详情里按新规则重评(只读封存,不调模型)。
|
||||||
|
|
||||||
### 定时评测
|
### 定时评测
|
||||||
|
|
||||||
@@ -68,6 +72,7 @@
|
|||||||
- 逐个数判:报告引用了一条查询的几列(例如销售额、订单数、首末日)时,每个数按它那一列单独和口径的指标列比,多查的列不影响这个数。只在去掉别的列不会改变这个数时才这样比;有 DISTINCT、按别的列排序后截断、多出来分组列的,照旧整条比。
|
- 逐个数判:报告引用了一条查询的几列(例如销售额、订单数、首末日)时,每个数按它那一列单独和口径的指标列比,多查的列不影响这个数。只在去掉别的列不会改变这个数时才这样比;有 DISTINCT、按别的列排序后截断、多出来分组列的,照旧整条比。
|
||||||
- 多行查询没有 `ORDER BY` 时,引用其中一格不算「来自口径」,召回和对账里照实标「行序不定」。
|
- 多行查询没有 `ORDER BY` 时,引用其中一格不算「来自口径」,召回和对账里照实标「行序不定」。
|
||||||
- 整题要每个来自数据的数都被证明来自同一份口径,才自动出具。
|
- 整题要每个来自数据的数都被证明来自同一份口径,才自动出具。
|
||||||
|
- 分析结论里的数(占比、变化、波动范围)是从查询结果派生的,不比对、不标口径,摘要行也不计它们。
|
||||||
|
|
||||||
写口径的取数 SQL 时,最好不要取整(`ROUND(SUM(x), 2)`),也不要套 `COALESCE(…, 0)`。取整和缺省值交给口径卡的小数位和展示;否则问数据里没取整的写法永远认不成等价(差在哪会照实写出来)。
|
写口径的取数 SQL 时,最好不要取整(`ROUND(SUM(x), 2)`),也不要套 `COALESCE(…, 0)`。取整和缺省值交给口径卡的小数位和展示;否则问数据里没取整的写法永远认不成等价(差在哪会照实写出来)。
|
||||||
|
|
||||||
|
|||||||
+25
-2
@@ -2,7 +2,7 @@
|
|||||||
|
|
||||||
# 证据与出具
|
# 证据与出具
|
||||||
|
|
||||||
这篇讲报告里的数字从哪来、怎么核对:口径卡和出具契约、三档出具、报告撰写节点的引用标记、查询区间里没有数据时怎么办、点开一个数能看到什么、封存与核对、两种导出,以及双盲复算、反方审稿、审批质询、数字推演、运行对比这组复核工具。给搭建者、审批人和看报告的人看。
|
这篇讲报告里的数字从哪来、怎么核对:口径卡和出具契约、三档出具、报告撰写节点的引用标记(含分析结论)、查询区间里没有数据时怎么办、点开一个数或一个分析结论能看到什么、封存与核对、两种导出,以及双盲复算、反方审稿、审批质询、数字推演、运行对比、分析说法核对这组复核工具。给搭建者、审批人和看报告的人看。分析判断本身怎么算见[分析能力](analysis.md)。
|
||||||
|
|
||||||
## 基本思路
|
## 基本思路
|
||||||
|
|
||||||
@@ -45,6 +45,9 @@ flowchart LR
|
|||||||
|
|
||||||
契约写了 `report_from` 时按引用核对:取回报告文档时复验哈希、独立重算一遍,全部数字都带引用才判为完整出具。
|
契约写了 `report_from` 时按引用核对:取回报告文档时复验哈希、独立重算一遍,全部数字都带引用才判为完整出具。
|
||||||
|
|
||||||
|
- 报告引用的口径卡指标,输入要追得到原始数据(查询快照的格、运行输入):受管级别的正式运行里追不到的记缺口、出具降档,其余照实写进出具声明,证据面板照旧标「追不到原始数据」。
|
||||||
|
- 报告里的分析说法在这里独立再核一遍,和组装报告时用同一个核对:报告撰写节点设成严格时,仍有的违规计入出具缺口(见[分析说法核对](#分析说法核对))。
|
||||||
|
|
||||||
### 指标出具
|
### 指标出具
|
||||||
|
|
||||||
纯指标看板(契约只写 `metrics_from`,上游没有报告撰写节点,成果里也没有模型写的话)按指标出具。没有叙述可核,就核指标本身:
|
纯指标看板(契约只写 `metrics_from`,上游没有报告撰写节点,成果里也没有模型写的话)按指标出具。没有叙述可核,就核指标本身:
|
||||||
@@ -62,6 +65,8 @@ flowchart LR
|
|||||||
- 表达式可以写 `cell(nodes.pull, 0, 'gmv')`,出处精确到单元格。
|
- 表达式可以写 `cell(nodes.pull, 0, 'gmv')`,出处精确到单元格。
|
||||||
- 取整和缺省值交给口径卡的小数位和展示,不要写进取数 SQL(原因见[口径账本](evaluation-and-ledger.md#逐个数标口径))。
|
- 取整和缺省值交给口径卡的小数位和展示,不要写进取数 SQL(原因见[口径账本](evaluation-and-ledger.md#逐个数标口径))。
|
||||||
- 受管级发布时,口径卡的输入不能来自计算角色的沙箱代码(门禁 G4)。
|
- 受管级发布时,口径卡的输入不能来自计算角色的沙箱代码(门禁 G4)。
|
||||||
|
- 一个指标用到两份同形、只差日期的查询(本周对上周)时,系统自动做一次可比性检查(见[可比性检查](analysis.md#可比性检查)),结果挂在指标上:写作目录照实提醒;带契约的出口里,不可比的记缺口、出具降档,有条件可比的只标注。
|
||||||
|
- 分析卡的结果不作口径卡的输入:口径卡读它,按追不到原始数据处理。
|
||||||
|
|
||||||
### 口径升版
|
### 口径升版
|
||||||
|
|
||||||
@@ -88,9 +93,13 @@ flowchart LR
|
|||||||
[[table:Q1 cols=region,amount rows=0-4]] 系统照快照生成整张表,每一格都点得开
|
[[table:Q1 cols=region,amount rows=0-4]] 系统照快照生成整张表,每一格都点得开
|
||||||
[[t:orders]] [[c:orders.week]] 表名、字段名;写在反引号里的名字同样核对
|
[[t:orders]] [[c:orders.week]] 表名、字段名;写在反引号里的名字同样核对
|
||||||
[[q:K1|原话]] 知识库原话,须在那次检索命中的片段里逐字出现
|
[[q:K1|原话]] 知识库原话,须在那次检索命中的片段里逐字出现
|
||||||
[[see:m:gmv,Q1]] 结论句的依据,挂在句末,不渲染
|
[[a:A1]] [[a:A1.top_share|pct]] 分析结论的判定语、具名的数,都由系统按结论渲染
|
||||||
|
[[v:A1.r0.share]] [[table:A1 rows=0-4]] 分析结论结果表里的一格、整张表
|
||||||
|
[[see:m:gmv,Q1,A1]] 结论句的依据,挂在句末,不渲染
|
||||||
```
|
```
|
||||||
|
|
||||||
|
分析结论按台账先后编成 A1、A2……,写作目录里另有一段「分析结论」。「主要来自」「异常」「持续上升」这类判断词要挂对应的分析结论、判定对得上,见[分析说法核对](analysis.md#分析说法核对)。
|
||||||
|
|
||||||
### 违规怎么处理
|
### 违规怎么处理
|
||||||
|
|
||||||
- 裸数字、解析不了的引用当场判违规,交回去重写(`max_repairs`,默认 1 次)。
|
- 裸数字、解析不了的引用当场判违规,交回去重写(`max_repairs`,默认 1 次)。
|
||||||
@@ -99,6 +108,7 @@ flowchart LR
|
|||||||
## 取数链路
|
## 取数链路
|
||||||
|
|
||||||
- agent、工具节点、代码节点、知识检索取数时都记进证据台账。报告的来源清单按台账编号:查询是 Q1…,知识库片段是 K1…。
|
- agent、工具节点、代码节点、知识检索取数时都记进证据台账。报告的来源清单按台账编号:查询是 Q1…,知识库片段是 K1…。
|
||||||
|
- agent 的分析工具、分析卡做的分析结论也记进证据台账,编号 A1…。分析只读这次运行里平台执行的查询快照,结论连同它的输入一起封存(见[分析能力](analysis.md))。
|
||||||
- agent 打开 `cite_fields`(配合 `output_schema`)后,每个字段标出取自哪次查询的哪一格,再到快照里核对:对不上以快照为准并告警;查不到的记为空值,不兜底成 0。
|
- agent 打开 `cite_fields`(配合 `output_schema`)后,每个字段标出取自哪次查询的哪一格,再到快照里核对:对不上以快照为准并告警;查不到的记为空值,不兜底成 0。
|
||||||
- 代码节点的 `evidence_role`:`source` 是取数,`compute`(默认)是计算。沙箱算出来的数不能被报告直接引用,要进口径卡。
|
- 代码节点的 `evidence_role`:`source` 是取数,`compute`(默认)是计算。沙箱算出来的数不能被报告直接引用,要进口径卡。
|
||||||
- 上传表格的单元格可以追到原表的格子(推断的来源)。
|
- 上传表格的单元格可以追到原表的格子(推断的来源)。
|
||||||
@@ -126,6 +136,7 @@ flowchart LR
|
|||||||
- **数字**:指标的原式、代入式和复算结果;输入与快照是否一致;查询的 SQL;被引用的行加前后各 2 行,高亮到格;最后是封存状态。只认封存范围内的事件交回过的工件。出处链最上面还有一步「口径」(见[口径账本](evaluation-and-ledger.md#逐个数标口径))。
|
- **数字**:指标的原式、代入式和复算结果;输入与快照是否一致;查询的 SQL;被引用的行加前后各 2 行,高亮到格;最后是封存状态。只认封存范围内的事件交回过的工件。出处链最上面还有一步「口径」(见[口径账本](evaluation-and-ledger.md#逐个数标口径))。
|
||||||
- **表名、字段名**:出现在哪次查询、类型、所属的表。表结构快照、SQL、结果列里都没有的名字标成可疑实体(「可能是编造的名字」,附最接近的 3 个)。快照被截断、核对不了的,只说核对不了,不冤枉成编造。
|
- **表名、字段名**:出现在哪次查询、类型、所属的表。表结构快照、SQL、结果列里都没有的名字标成可疑实体(「可能是编造的名字」,附最接近的 3 个)。快照被截断、核对不了的,只说核对不了,不冤枉成编造。
|
||||||
- **知识库原话**:须在那次检索命中的片段里逐字出现。
|
- **知识库原话**:须在那次检索命中的片段里逐字出现。
|
||||||
|
- **分析结论**(判定语、具名的数、结果表的格):依次是判定(判定语、判定标准、点名的段与方向、谁做的、附带说明、Agent 写的这次在验证什么)、方法(方法与版本、参数、实际用的阈值、现查的表范围)、计算(结果表,关键数的代入式,如「占比 = 华东的变化 / 总变化 = -120 / -194」,引用的那一格高亮)、输入(落到哪个 Q 的哪几列、哪几行,总数那两格的闭合核对,下钻的上一层,接着是输入查询的步骤)、复算(按封存的输入用同一个函数重算,逐字段相同才算核对通过;方法升了版本,照实写运行时用的是哪一版)、封存。证据图里,分析结论连一条「派生自」的边到它的输入和下钻的上一层。
|
||||||
|
|
||||||
## 结论句
|
## 结论句
|
||||||
|
|
||||||
@@ -165,12 +176,14 @@ flowchart LR
|
|||||||
- 正文由服务端渲染,不执行脚本也读得到(邮件附件预览常常不执行脚本)。
|
- 正文由服务端渲染,不执行脚本也读得到(邮件附件预览常常不执行脚本)。
|
||||||
- 执行脚本后,每个数字、无证据的数字、可疑的表名字段名都点得开。面板是网页上证据面板同一个拼法的出处链(口径、来源、计算方式、SQL、被引用的行),封存状态逐项标明。
|
- 执行脚本后,每个数字、无证据的数字、可疑的表名字段名都点得开。面板是网页上证据面板同一个拼法的出处链(口径、来源、计算方式、SQL、被引用的行),封存状态逐项标明。
|
||||||
- 遮罩列照网页上的规矩写「已遮罩」。
|
- 遮罩列照网页上的规矩写「已遮罩」。
|
||||||
|
- 分析结论的判定语、具名的数、结果表的格同样点得开,面板依次是判定、方法、计算、输入;导出时服务端按封存的输入复算一遍,复算结果写在面板里。报告末尾的「分析范围」照文档里记下的渲染,编号点得开。
|
||||||
- 没封存的运行照常导出,和在网页上读报告一个道理。
|
- 没封存的运行照常导出,和在网页上读报告一个道理。
|
||||||
|
|
||||||
**可验证报告**:记录页「凭证详情」里的「导出可验证报告」,接口 `GET /api/runs/{id}/bundle.html`,是一份封存核对件。
|
**可验证报告**:记录页「凭证详情」里的「导出可验证报告」,接口 `GET /api/runs/{id}/bundle.html`,是一份封存核对件。
|
||||||
|
|
||||||
- 收件人不连服务、不装任何东西,在浏览器里重算哈希,核对每个数字、文档和运行记录没被改过。
|
- 收件人不连服务、不装任何东西,在浏览器里重算哈希,核对每个数字、文档和运行记录没被改过。
|
||||||
- 数字的出处面板同样带口径、原式、代入式、输入和复算;口径卡输入指向的查询快照也随文件附带。
|
- 数字的出处面板同样带口径、原式、代入式、输入和复算;口径卡输入指向的查询快照也随文件附带。
|
||||||
|
- 报告引用的分析结论(含分析范围里点名的)按原字节随文件附带,连同它的输入快照和下钻的上一层结论;导出时服务端按封存的输入复算一遍,结果记在结论的条目上。浏览器里只核哈希、展示服务端的复算结果,不在页面里重算统计。任一份输入快照证明不了没碰遮罩列的,结论也只给 id。
|
||||||
- 它只证明文件内部一致:数字、文档、运行记录彼此对得上,而且没被改过。真伪要拿页面上的封存哈希和系统里这次运行的封存核对(`GET /api/runs/{id}/verify`)对照,页面上醒目地写着这一点。
|
- 它只证明文件内部一致:数字、文档、运行记录彼此对得上,而且没被改过。真伪要拿页面上的封存哈希和系统里这次运行的封存核对(`GET /api/runs/{id}/verify`)对照,页面上醒目地写着这一点。
|
||||||
- 封存之后追加的事件(按需裁判、审批质询这类批注)单列,只有类型、序号和内容摘要,不进清单。
|
- 封存之后追加的事件(按需裁判、审批质询这类批注)单列,只有类型、序号和内容摘要,不进清单。
|
||||||
- 数据源设了遮罩时,证明不了没碰遮罩列的查询快照、有输入取自遮罩列的口径卡只给 id,不附内容。遮罩只减少暴露,不是安全边界。
|
- 数据源设了遮罩时,证明不了没碰遮罩列的查询快照、有输入取自遮罩列的口径卡只给 id,不附内容。遮罩只减少暴露,不是安全边界。
|
||||||
@@ -246,4 +259,14 @@ flowchart LR
|
|||||||
- 口径与数据都变、又能严格拆分的,分别给出口径效应和数据效应,两者之和等于总变化。
|
- 口径与数据都变、又能严格拆分的,分别给出口径效应和数据效应,两者之和等于总变化。
|
||||||
- 只数据变、只参数变、口径与数据都变的指标,能归因时按维度(比如区域)列出变化最大的前 10 项。每一项都追得到两边的查询快照;核查执行的查询不参与对比。
|
- 只数据变、只参数变、口径与数据都变的指标,能归因时按维度(比如区域)列出变化最大的前 10 项。每一项都追得到两边的查询快照;核查执行的查询不参与对比。
|
||||||
- 遮罩列的值不出现在结果里,用到遮罩列的指标不拆分。
|
- 遮罩列的值不出现在结果里,用到遮罩列的指标不拆分。
|
||||||
|
- 图里有分析卡的,两次运行的分析卡按节点和项对齐,列出两次的状态、判定、方向和判定语;循环里执行了几次的取最后一次。
|
||||||
- 定时任务的运行成功结束、封存后,自动和同一任务上一次成功的运行比一次;触发历史和通知里只给基准是哪一次、各类指标各几个,不给具体数字。
|
- 定时任务的运行成功结束、封存后,自动和同一任务上一次成功的运行比一次;触发历史和通知里只给基准是哪一次、各类指标各几个,不给具体数字。
|
||||||
|
|
||||||
|
### 分析说法核对
|
||||||
|
|
||||||
|
报告里的判断词——「主要来自华东」「超出常规波动」「持续上升」「差异站得住」——要挂对应的分析结论、判定对得上;写原因要有知识库原话或运行输入作依据,否则写成待核实的推测。
|
||||||
|
|
||||||
|
- 由报告撰写节点在写作时逐句核对:确定的词表加绑定,不调用模型,不花钱。引用模式的出口独立再核一遍。
|
||||||
|
- 缺省重写一次、仍有的标在报告里(证据面板、审计、导出都看得到),不影响出具档位;设成严格时按最多重写次数重写,仍有的计入出具缺口、出具降档。受管级别要求严格(门禁 G3)。
|
||||||
|
- 只核带这一项记录的新报告文档,升级前的报告复核结果不变。
|
||||||
|
- 判断词分哪几类、各要挂哪种分析、哪些写法不核,见[分析说法核对](analysis.md#分析说法核对)。
|
||||||
|
|||||||
+12
-2
@@ -2,17 +2,27 @@
|
|||||||
|
|
||||||
# 工具、MCP 与能力登记
|
# 工具、MCP 与能力登记
|
||||||
|
|
||||||
这篇讲工作流能调用的工具:内置工具、自定义工具和 MCP,它们的审批档位,管理员怎么在能力登记里管住它们,以及怎么登记外部系统的端点。给搭建者和管理员看。
|
这篇讲工作流能调用的工具:内置工具(含分析工具)、自定义工具和 MCP,它们的审批档位,管理员怎么在能力登记里管住它们,以及怎么登记外部系统的端点。给搭建者和管理员看。
|
||||||
|
|
||||||
## 工具种类
|
## 工具种类
|
||||||
|
|
||||||
- **内置工具**:15 个。
|
- **内置工具**:23 个,其中 5 个是分析工具(见下文)。
|
||||||
- **自定义工具**:HTTP 模板,或沙箱里的 Python。
|
- **自定义工具**:HTTP 模板,或沙箱里的 Python。
|
||||||
- **MCP**:接入 MCP server(stdio 或 http)。
|
- **MCP**:接入 MCP server(stdio 或 http)。
|
||||||
- **数据源工具**:每个数据源的查询工具 `db_query__<源>`,以及运行时查结构的 `db_schema__`(见[数据源](data-sources.md))。
|
- **数据源工具**:每个数据源的查询工具 `db_query__<源>`,以及运行时查结构的 `db_schema__`(见[数据源](data-sources.md))。
|
||||||
|
|
||||||
谁能配:MCP 的 stdio、端点和凭据、能力治理只有管理员能改,搭建者只读,使用者看不到(见[权限表](identity-and-security.md#权限表))。
|
谁能配:MCP 的 stdio、端点和凭据、能力治理只有管理员能改,搭建者只读,使用者看不到(见[权限表](identity-and-security.md#权限表))。
|
||||||
|
|
||||||
|
## 分析工具
|
||||||
|
|
||||||
|
五个分析工具是平台的确定性分析:`analysis_decompose`(变化拆解)、`analysis_baseline`(波动基线)、`analysis_trend`(趋势判断)、`analysis_differ`(差异检验)、`analysis_comparable`(可比性检查)。它们不查库,只分析绑了它的 agent 这次执行里查过的结果,判定由平台按固定的方法和阈值算,判定语由系统写(见[Agent 的分析工具](analysis.md#agent-的分析工具))。
|
||||||
|
|
||||||
|
- 参数里没有阈值。绑了任一个,平台就把内置的分析方法注入 agent 的系统提示,同一份只注入一次。
|
||||||
|
- 只读、不写任何东西,不算危险工具:缺省的审批策略下不用等审批;使用者的问数据里也能用。
|
||||||
|
- 只能绑给 agent:放进「调用工具」节点时画布校验报错,模板里要固定做的分析用分析卡(见[分析卡](analysis.md#分析卡))。
|
||||||
|
- 每个 Agent 每次执行最多分析 12 次(「设置 → 偏好设置 → 分析」里改)。
|
||||||
|
- 它们不连数据源:读的是这次执行里已经查回来的快照,交回的是分析结论,不是查询结果。
|
||||||
|
|
||||||
## 审批档位
|
## 审批档位
|
||||||
|
|
||||||
平台看不出 MCP 和自定义工具做什么,所以它们默认每次调用都等审批。审批档位有三档:`ask`、`gated`、`always`。
|
平台看不出 MCP 和自定义工具做什么,所以它们默认每次调用都等审批。审批档位有三档:`ask`、`gated`、`always`。
|
||||||
|
|||||||
@@ -4,6 +4,22 @@
|
|||||||
|
|
||||||
这篇按时间倒序列出各次更新里,对使用者和运维有影响的变化,以及升级时要做的事。给运维和管理员看。功能本身的说明在各主题文档里。
|
这篇按时间倒序列出各次更新里,对使用者和运维有影响的变化,以及升级时要做的事。给运维和管理员看。功能本身的说明在各主题文档里。
|
||||||
|
|
||||||
|
## 2026-10 分析能力一轮
|
||||||
|
|
||||||
|
**要做的事**:没有必须的操作,数据库表结构没有变化。要按受管级别重新发布的报告模板,报告撰写节点的「分析说法核对」要设为严格,发布前检查会提示,也能一键修复。
|
||||||
|
|
||||||
|
- 新增分析能力(见[分析能力](analysis.md)):变化拆解、波动基线、趋势判断、差异检验、可比性检查五种确定性分析,判定由系统算、判定语由系统写,报告用 `[[a:A1]]` 引用,点得开、能复算。
|
||||||
|
- 新节点「分析卡」(`analysis`)和五个分析工具,内置工具从 18 个变成 23 个。新模板「⑪ 异动分析(可追溯)」,升级后第一次启动自动加进模板列表。模板按名字只装一次:已有部署里的「⑨ 可追溯周报」不会跟着更新,按受管级别发布时照发布前检查的提示补上分析说法核对。
|
||||||
|
- 报告撰写节点缺省核对分析说法(「重写一次后标出」):「主要来自」「异常」「持续上升」这类判断词没挂对分析结论、原因没有依据的句子,交回写作者重写一次,仍有的标在报告里,不影响出具档位。写作规则多了一段讲判断词的话,图里没有分析的报告也一样:没有分析结论时,写作者被告知不下这类判断、只陈述数字和变化。不想核的,把报告撰写节点的「分析说法核对」设成不核对。
|
||||||
|
- 受管级别要求「分析说法核对」设为严格(门禁 G3 多一条)。已经发布的版本照常运行,按缺省的「重写一次后标出」核;下次发布时按新规则检查。
|
||||||
|
- 用到分析结论的报告末尾多一段系统写的「分析范围」,不进正文。报告文档多记一项分析说法按哪种处置、哪一版说法表核:只有带它的新文档在复核时核这一项,升级前的报告复核结果不变。
|
||||||
|
- 引用模式的出口收齐报告引用的指标:输入追不到原始数据的,受管级别的正式运行记缺口、出具降档;别的级别照旧只写进出具声明。
|
||||||
|
- 口径卡里用到两份只差日期的查询的指标,自动做一次可比性检查:带契约的出口里,不可比的记缺口降档,有条件可比的只标注。
|
||||||
|
- 报告导出、可验证报告带上分析结论和它的输入快照,按封存的输入复算核对。
|
||||||
|
- 设置多了「偏好设置 → 分析」一节(见[分析设置](configuration.md#分析设置))。只在图里有分析卡、或者 agent 绑了分析工具时,发起运行才把这一组快照进运行,别的运行的发起记录一字不变。
|
||||||
|
- 问数据:助手建图多了分析类问题的形状,自查时提醒查库的 agent 绑分析工具(只提醒一轮)。复核把几次没做成的分析并成一条「有 n 次分析没做成」,只写说明、不自动重来;分析调用写错了、后来改对的不再记成缺口,一直没改对的也不再触发整轮重来。
|
||||||
|
- 评测多了贡献、判定两种期望,评分规则升到第 3 版(见[回归评测](evaluation-and-ledger.md#回归评测))。
|
||||||
|
|
||||||
## 2026-10 编排完善第 2 波:PostgreSQL 与多进程
|
## 2026-10 编排完善第 2 波:PostgreSQL 与多进程
|
||||||
|
|
||||||
**要做的事**:多进程部署升级前,先停掉全部工作进程;代码节点要用的值改成显式传入;别删数据目录里的 `.agentlab-data`。
|
**要做的事**:多进程部署升级前,先停掉全部工作进程;代码节点要用的值改成显式传入;别删数据目录里的 `.agentlab-data`。
|
||||||
|
|||||||
@@ -40,7 +40,7 @@
|
|||||||
|
|
||||||
### 内置模板
|
### 内置模板
|
||||||
|
|
||||||
内置 9 个模板,从「最小问答」到「可追溯周报」。它们可以直接运行,也是各类节点用法的可执行文档。
|
内置 11 个模板,从「最小问答」到「异动分析(可追溯)」。它们可以直接运行,也是各类节点用法的可执行文档。「⑨ 可追溯周报」和「⑪ 异动分析(可追溯)」拿来就能按受管级别发布:前者示范口径卡和按引用出具,后者示范分析卡——两次查询取数,分析卡判这一周的变化在不在常规波动范围内、主要来自哪个地区,报告只引用判定语(见[分析能力](analysis.md#分析卡))。
|
||||||
|
|
||||||
## 节点类型
|
## 节点类型
|
||||||
|
|
||||||
@@ -51,9 +51,9 @@
|
|||||||
| 执行 | `tool` `code` `merge` | 直接调用工具 / 沙箱内执行代码 / 合并几次查询的结果(跨库按键对齐) |
|
| 执行 | `tool` `code` `merge` | 直接调用工具 / 沙箱内执行代码 / 合并几次查询的结果(跨库按键对齐) |
|
||||||
| 控制 | `branch` `loop` `subgraph` | 条件或语义分支 / 遍历与条件循环 / 嵌套工作流 |
|
| 控制 | `branch` `loop` `subgraph` | 条件或语义分支 / 遍历与条件循环 / 嵌套工作流 |
|
||||||
| 上下文 | `memory` `retrieve` `transform` | 长期记忆读写 / 知识检索 / 数据整形 |
|
| 上下文 | `memory` `retrieve` `transform` | 长期记忆读写 / 知识检索 / 数据整形 |
|
||||||
| 把关 | `human` `validate` `metrics` | 人工介入 / Schema 校验与自动返工 / 口径卡 |
|
| 把关 | `human` `validate` `metrics` `analysis` | 人工介入 / Schema 校验与自动返工 / 口径卡 / 分析卡(对上游查询结果固定做几项确定性分析) |
|
||||||
|
|
||||||
`output` 节点按 JSON Schema 校验结构化成果,不合格时把错误交回模型自动返工。`metrics`(口径卡)和 `report`(报告撰写)见[证据与出具](evidence.md);`merge`(合并查询)见[数据源](data-sources.md#跨库合并)。
|
`output` 节点按 JSON Schema 校验结构化成果,不合格时把错误交回模型自动返工。`metrics`(口径卡)和 `report`(报告撰写)见[证据与出具](evidence.md);`analysis`(分析卡)见[分析能力](analysis.md#分析卡);`merge`(合并查询)见[数据源](data-sources.md#跨库合并)。
|
||||||
|
|
||||||
## 节点之间怎么传数据
|
## 节点之间怎么传数据
|
||||||
|
|
||||||
@@ -125,7 +125,7 @@
|
|||||||
|
|
||||||
## Agent 节点
|
## Agent 节点
|
||||||
|
|
||||||
`agent` 节点是一个带工具的模型循环,能用哪些工具由节点上的白名单决定(见[工具](tools.md))。配了 `output_schema` 的 agent 可以打开 `cite_fields`,让每个字段标出取自哪次查询的哪一格(见[证据与出具](evidence.md#取数链路))。
|
`agent` 节点是一个带工具的模型循环,能用哪些工具由节点上的白名单决定(见[工具](tools.md))。配了 `output_schema` 的 agent 可以打开 `cite_fields`,让每个字段标出取自哪次查询的哪一格(见[证据与出具](evidence.md#取数链路))。查库的 agent 可以再绑分析工具:拆变化、判异常、判趋势由平台按固定的方法和阈值算,判定语由系统写,报告引用它(见[Agent 的分析工具](analysis.md#agent-的分析工具))。
|
||||||
|
|
||||||
### 护栏
|
### 护栏
|
||||||
|
|
||||||
@@ -186,7 +186,7 @@ Excel / CSV 作为数据源上传,会变成可以用 SQL 查的表;放进知
|
|||||||
|
|
||||||
- **G1** 带契约的出口,成果字段的文字只能来自报告撰写节点。
|
- **G1** 带契约的出口,成果字段的文字只能来自报告撰写节点。
|
||||||
- **G2** 模型写的文字不绕过报告撰写节点直接流进出口。
|
- **G2** 模型写的文字不绕过报告撰写节点直接流进出口。
|
||||||
- **G3** 报告撰写节点写明 `numbers: strict`、`on_violation: fail`、`claims: require_citation`(或 `judge`,并写预算)。
|
- **G3** 报告撰写节点写明 `numbers: strict`、`on_violation: fail`、`claims: require_citation`(或 `judge`,并写预算)、`analysis_claims: strict`(分析说法核对设为严格,见[分析说法核对](analysis.md#分析说法核对))。
|
||||||
- **G4** 口径卡的输入不来自计算角色的沙箱代码。
|
- **G4** 口径卡的输入不来自计算角色的沙箱代码。
|
||||||
- **G5** 给口径卡供数的 agent 配 `output_schema` 并开 `cite_fields`。
|
- **G5** 给口径卡供数的 agent 配 `output_schema` 并开 `cite_fields`。
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user