mirror of
https://github.com/ThearchyHelios/AgentLab.git
synced 2026-11-18 18:02:04 +00:00
文档:新增 analysis 一篇(中英各一,标题层级一致)——分析能力是什么、为什么(判定由系统算、判定语由系统写、照实说、留痕)、五种分析各回答什么问题与判定标准(括号里是缺省阈值)、算不了的情形与调用写错、期次的写法、两个入口(Agent 的分析工具、分析卡)、引用写法、分析说法核对(判断词的类、check / strict / off、来源与因果的分界)、分析范围、设置、限制。更新:证据与出具(按引用核对的血缘口子与分析说法复核、口径卡指标的两期可比性、引用标记加 a: 与 v:A / table:A、点开一个分析结论看到什么、两种导出认分析结论、运行对比按分析卡对齐、复核套件加分析说法核对)、工作流(内置模板 11 个、节点类型表加分析卡、G3 加分析说法核对、agent 可绑分析工具)、问数据(分析类问题怎么走、复核只写说明的几种缺口、分析结论的数不标口径、自查提醒)、回归评测与口径账本(贡献、判定两种期望与评分,细节待评测包交付后校对;分析结论的数不比对口径)、配置(偏好设置里的分析一节,17 项的叫法、缺省、范围)、工具(内置工具 23 个、分析工具一节;原来写的 15 个早已对不上)、升级说明(本轮);README 中英加一条主要功能、文档索引加 analysis,节点 19 种、模板 11 个。development.md 的检查项数没动,等前端包的数
This commit is contained in:
@@ -0,0 +1,204 @@
|
||||
English | [Simplified Chinese](../zh-CN/analysis.md)
|
||||
|
||||
# Analysis
|
||||
|
||||
This page covers where the analytical judgments in a report come from: how statements such as "mainly from East", "outside the usual fluctuation", "rising steadily", or "the difference holds up" are computed by the system, written into reports, and checked. It covers what each of the five analyses answers, the criteria it judges by, what it says when it can't be done, the two entry points (analysis tools for agents and analysis cards), how reports cite conclusions, the check on judgment words, the Analysis Scope section at the end of a report, and the settings and limits. It is for builders and for people who read reports. The user interface is currently available in Simplified Chinese only.
|
||||
|
||||
## What it is and why
|
||||
|
||||
The system fills the numbers in a report from snapshots, and the narrative layer does no arithmetic (see [Evidence and Issuance](evidence.md#the-basic-idea)). But a report says more than numbers; it also makes judgments: where a change mainly came from, whether this week counts as unusual, whether something keeps rising, whether a gap between two groups is real. These judgments used to be made by the writing model on its own; it could pick its evidence, and readers had no way to check.
|
||||
|
||||
Analysis brings judgments back into the system as well:
|
||||
|
||||
- **The system computes the verdicts**: the five analyses are deterministic platform code. They read only query snapshots returned during this run and compute verdicts with fixed methods and thresholds. The model decides only what to compute (which two periods to compare, which dimension to break down, which segment to drill into); it can't change the method or the thresholds.
|
||||
- **The system writes the verdict phrases**: every analysis conclusion carries a verdict phrase written by the system from a template, and reports cite it (`[[a:A1]]`); like a number, it opens to its provenance. When the writer writes judgment words such as "mainly from" or "unusual" itself, the system checks sentence by sentence that the sentence cites a matching analysis conclusion and agrees with its verdict (see [Analysis claim check](#analysis-claim-check)).
|
||||
- **Saying it as it is**: when an analysis can't be done, it says why (not enough history, the current period isn't over, the query was truncated, the two periods aren't comparable, too few samples) instead of forcing a conclusion. "Can't be done" is a conclusion too, and the writer reports it as such.
|
||||
- **Leaving a trace**: every analysis conclusion, together with its method and version, parameters, the thresholds actually used, the input cells, the table ranges checked live, the results, and the verdict, is stored as a content-addressed artifact, recorded in the evidence ledger, and sealed with the run. The evidence panel, the report export, and the verifiable report all recompute it with the same function (see [What you see when you click a number](evidence.md#what-you-see-when-you-click-a-number)).
|
||||
|
||||
A breakdown shows only where a change came from, not why. A cause needs a verbatim knowledge-base quote or a run input as its basis; otherwise it is written as a conjecture to be verified.
|
||||
|
||||
## The five analyses
|
||||
|
||||
Each analysis produces one analysis conclusion, numbered A1, A2, … in order in the writing catalog (the same numbering scheme as Q1 for queries and K1 for knowledge-base chunks). A conclusion has a verdict (verdict code, the segments it names, a direction), a verdict phrase, a criterion (one sentence stating the thresholds actually used), named values (shares, the total change, and so on), a result table, and caveats (offsetting, Simpson's paradox, small samples, a record high, and so on; at most two are appended to the verdict phrase).
|
||||
|
||||
The thresholds in parentheses below are defaults, changed under Settings → Preferences → Analysis (see [Analysis settings](configuration.md#analysis-settings)).
|
||||
|
||||
### Change decomposition
|
||||
|
||||
Answers "where did the change come from". Three modes:
|
||||
|
||||
| Mode | Used for | How it breaks down |
|
||||
|---|---|---|
|
||||
| Additive | Quantities that add up, such as sales or order counts | By dimension (one or two columns), each segment's contribution to the total change and its share |
|
||||
| Ratio | Ratios such as refund rate or average order value | Into a mix effect (segment weights changed) and each segment's own change; the two add up exactly to the change in the ratio |
|
||||
| Factor | Sales = orders × average order value | Into each factor's contribution, by logarithmic mean (LMDI) |
|
||||
|
||||
- Verdicts: one segment accounting for at least half of the total change (0.5) is the main source ("mainly from East"); without such a segment, the top few segments together accounting for at least 80% (0.8) count as concentrated (at most a fifth of all segments, and at least 2); a total change under 0.5% of the base counts as essentially flat; anything else is dispersed. A ratio decomposition is judged as mainly from the mix, mainly from the segments' own changes, or both; a factor decomposition is judged by which factor it mainly comes from.
|
||||
- When totals are given, the segments must add up to them (within a tolerance based on the precision the totals are written with) before the decomposition judges where the change came from. If they don't, it doesn't judge, and the verdict phrase states the gap. Without totals, the sum of the segments is the total, and the criterion says so.
|
||||
- When the segments moving the other way add up to more than a quarter of the total change (0.25), a caveat says the changes partly offset each other. In a ratio decomposition, when most segments fall while the whole rises (or the other way round), a caveat explains Simpson's paradox. Segments with a denominator under 30 are marked as small and left out of that judgment.
|
||||
- Drilling down: one segment of a decomposition can be broken down one level further (East by channel). The segments of that level must add up to the value of the segment above, at most 3 levels deep. The agent writes the drill-down query itself, querying only that segment, and the system checks that it closes against the value of the segment above.
|
||||
- An example verdict phrase (translated): "Sales fell 194 (-9.2%), mainly from East (-120, 61.9% of the change)".
|
||||
|
||||
### Fluctuation baseline
|
||||
|
||||
Answers "is this change large, does it count as unusual": it puts the current period back into its own history.
|
||||
|
||||
- The input is one series ordered by period (day, week, month, quarter). By default it compares the rate of change against the previous period; daily data is compared with the same day last week, to avoid weekday effects. Stocks and ratios can compare the value itself. For year over year, compare with 52 weeks or 12 months earlier, with correspondingly longer history.
|
||||
- It uses the median and the median absolute deviation of the history (robust standard deviation = 1.4826 × median absolute deviation): a current period within 2 robust standard deviations of the median is a usual fluctuation, 2 to 3 is outside, and 3 or more is far outside. It uses the median rather than the mean because extremes such as promotion weeks widen the mean and standard deviation, making normal weeks and unusual weeks alike look ordinary.
|
||||
- History needs at least 8 periods and uses at most 52. When the current period is the highest or lowest in its history, a caveat says it is a record high or low.
|
||||
- An example verdict phrase (translated): "Sales changed -15.8% this period, far outside the usual fluctuation of the past 24 weeks (-0.5% to +0.4%); this period's sales are the lowest of the past 24 weeks".
|
||||
|
||||
### Trend test
|
||||
|
||||
Answers "is it changing steadily, or was it a one-off".
|
||||
|
||||
- It takes the last contiguous stretch (at least 6 periods): when the two-sided p-value of the Mann–Kendall test is below 0.05, the sign of the Theil–Sen slope decides rising or falling; otherwise there is no clear trend. When the last 3 or more periods all move the same way, the verdict phrase adds "down 3 weeks in a row", or "but up the last 3 weeks in a row" when that runs against the trend.
|
||||
- Mann–Kendall assumes the periods are independent, while business series are often autocorrelated, which makes p too small. So the verdict phrase says only "trending up", never "significantly up". Daily data gets a caveat about weekday effects suggesting weekly aggregation.
|
||||
- An example verdict phrase (translated): "Online sales have trended up over the past 12 weeks (about +2.1% per week), up 4 weeks in a row".
|
||||
|
||||
### Difference test
|
||||
|
||||
Answers "are these two groups (or two periods of one group) really different, or is it sampling noise".
|
||||
|
||||
- For rates (conversion rate, refund rate), give the successes and the trials: each group gets a Wilson interval; when every cell's expected frequency is at least 5, it uses the pooled-proportion z-test, otherwise Fisher's exact test.
|
||||
- For means, give each group's mean, standard deviation, and sample size, or its sum, sum of squares, and sample size (SQLite has no standard deviation function, so write it this way), and it uses Welch's t-test. You can also give the values row by row (at most 1000 rows, not truncated); the system computes the statistics itself and also reruns the test without the most distant value as a sensitivity check.
|
||||
- A two-sided p-value below 0.05 means the difference holds up; several pairs in one call are corrected with Holm's method. A caveat about small samples is added when a group has fewer than 30 samples for means, or when Fisher's test was used for rates.
|
||||
- Verdict phrases avoid terms such as "p-value" or "significance": a difference that holds up is described as such, and otherwise the phrase says the difference may be just sampling noise, with the sample sizes (230 and 40, say). P-values and intervals are in the result table and the evidence panel.
|
||||
|
||||
### Comparability check
|
||||
|
||||
Answers "can these two periods be compared directly". Each item passes, warns, fails, or can't be checked:
|
||||
|
||||
| Item | How it is judged |
|
||||
|---|---|
|
||||
| Same definition | The two queries differ only in their dates; when their structure differs, or they also filter something else (East versus South), it fails |
|
||||
| Window length | Both periods have the same number of days; for an additive flow a mismatch fails, for ratios and stocks it only warns |
|
||||
| Whole period | The current period is over and the table's data has arrived; for a flow, missing data fails ("the current period has only 3 days of data"), otherwise it warns |
|
||||
| Weekday mix | For daily windows whose length isn't a multiple of 7, both periods have the same number of weekend days; otherwise it warns |
|
||||
| Data problems | No data in the queried range fails; SQL check errors warn |
|
||||
|
||||
When everything passes (or can't be checked), the periods are comparable; with only warnings they are conditionally comparable; with any failure they aren't comparable. Items that can't be checked never make the periods incomparable, and the verdict phrase names them. The table's actual date range is checked live (read-only, time-limited, masked columns are never checked, falling back to the range recorded by profiling on failure); the range found is recorded in the conclusion, and recomputation uses the recorded values.
|
||||
|
||||
When a change decomposition is given two queries, it runs this check automatically first; when the fluctuation baseline, the trend test, or a change decomposition over a single query checks whether the current period is complete, it uses the whole-period item of this check. It also runs once automatically when a caliber card metric uses two queries of the same shape that differ only in their dates (this week versus last week), with the result attached to the metric (see [Caliber cards](evidence.md#caliber-cards)).
|
||||
|
||||
## When an analysis can't be done
|
||||
|
||||
An analysis that can't be done doesn't fail the agent or the node. It is recorded as a conclusion, with a verdict phrase along the lines of "Fluctuation baseline not done: only 5 periods of history, at least 8 needed", and it goes into the ledger and can be cited and checked:
|
||||
|
||||
| Reason | When |
|
||||
|---|---|
|
||||
| Truncated | The input query returned only its first rows, so the sum of the segments isn't the full sum |
|
||||
| Too few segments | Fewer than 2 segments after the breakdown |
|
||||
| History too short | Fewer valid periods of history than the minimum |
|
||||
| Current period incomplete | The current period isn't over, data hasn't arrived, it isn't a whole week, or the query covered only part of it |
|
||||
| Gap | A period is missing between the current period and its comparison period; the last periods used by the trend test aren't contiguous |
|
||||
| Periods unreadable | The period column's format can't be read (times of day, mixed formats) |
|
||||
| Zero base | The comparison period is 0, so no rate of change can be computed |
|
||||
| Non-positive values | Factor decomposition requires every value to be positive |
|
||||
| Not the same quantity | The two queries differ in structure, so they aren't two periods of the same quantity, and it doesn't break them down |
|
||||
| No p-value | The difference test can't compute a p-value for any pair (each group has only 1 sample, or a sample size is 0) |
|
||||
| Upper level not done, or missing from this data | In an analysis card, the level a drill-down sits under couldn't be done, or this period's data lacks a period, segment, group, or total cell that is needed |
|
||||
|
||||
A wrong call isn't recorded as a conclusion: a column that doesn't exist, a masked column used as a dimension or measure, a ratio treated as additive, several rows for the same segment and period, wrong parameters, or this agent having used up its analyses for this execution. The agent gets a sentence on how to fix the call and calls again; in an analysis card it is the template that is wrong, so the whole card fails and says which item.
|
||||
|
||||
Periods are read in these formats: dates, `YYYY-MM`, `YYYY-Www`, `YYYY-Qn`, and integer sequence numbers. For weekly aggregation, it is best to compute the period from a date column in the SQL, as the date of the first day of each week, or as an ISO week `YYYY-Www`. The same `2026-W42` lands on different days under SQLite's `%W`, MySQL's `%u`, and the ISO week; the system recognizes the scheme from the format string in the SQL and works out the real start and end. When the period is a ready-made week-number text column taken straight from the table, the scheme can't be recognized, and whether the current period is complete can't be checked.
|
||||
|
||||
## Two entry points
|
||||
|
||||
Both entry points share the same analyses, the same shape of conclusion, and the same table of verdict phrases.
|
||||
|
||||
### Analysis tools for agents
|
||||
|
||||
Agents in Ask Data and on the canvas can be given five analysis tools: `analysis_decompose` (change decomposition), `analysis_baseline` (fluctuation baseline), `analysis_trend` (trend test), `analysis_differ` (difference test), and `analysis_comparable` (comparability check). Use them when you only know which dimension to break down after seeing the numbers.
|
||||
|
||||
- The parameters cover only what to compute: which result this node queried (`Q3`, the same number as in the "Evidence Q3" handed back to the model), the role of each column, which two periods to compare, the two total cells, the level to drill down from, what to call the measure, and one sentence on what this call is testing (shown only in the evidence panel, never in the report body). There are no threshold parameters at all.
|
||||
- Binding any one of them injects the platform's built-in analysis method into the agent's system prompt: first set the base period from the question, run the comparability check and the fluctuation baseline, then break down factors and mix, break down by dimension from coarse to fine, and drill down one level once the main source is found; a breakdown that finds no concentration is a conclusion too; make no causal claims; say plainly when something can't be done; finally, list the analyses done and their verdicts. The method is versioned with the code.
|
||||
- What goes back to the agent is "Analysis A2" with the verdict, the verdict phrase, the key values, and caveats or the reason it couldn't be done; the full conclusion goes into the evidence ledger.
|
||||
- Each agent can run at most 12 analyses per execution and drill down at most 3 levels.
|
||||
- The analysis tools analyze only results that the agent they are bound to queried itself, and can't be put in a Call Tool node (for fixed analyses in a template, use an analysis card). Members of a collaboration team can use them too, but their conclusions don't go into the evidence ledger, so reports can't cite them.
|
||||
- Users (the user role) can use them in Ask Data too: the analysis tools don't query the database; they read only query results already masked, and masked columns are never used as dimensions or measures.
|
||||
|
||||
### Analysis cards
|
||||
|
||||
An analysis card (the `analysis` node, a peer of the caliber card) runs a fixed set of analyses on the results of upstream query nodes. It is for analyses a template always does, such as the anomaly part of a weekly report: is this week's change within the usual fluctuation, and which region did it mainly come from.
|
||||
|
||||
- Each item states the analysis method and mode, which upstream query node to analyze, the parameters, and what the item is testing. Inputs can only be Call Tool nodes that call a database query tool, or Merge Query nodes, connected before the analysis card; an agent can't be an input, since it may have queried several times. The parameters are the same as the analysis tools', and `{{ input.x }}` can pick periods from run inputs.
|
||||
- Thresholds aren't written in the items. When a template needs fixed thresholds, write them in the card's Threshold Overrides, pinned with the version; when they conflict with this run's analysis settings, the card fails and says where the conflict is.
|
||||
- The output is `nodes.<card>.results.<item>`: status, verdict code, direction, named values, and verdict phrase. A downstream branch can route on the verdict, for example sending a run to a person when the change is outside the usual fluctuation.
|
||||
- An item that can't be done doesn't fail the card: that item is recorded as not done, and the card moves on to the next one.
|
||||
- Analysis card results are not caliber card inputs: a caliber card that reads them is treated as unable to trace back to raw data. Take metrics in the caliber card from query result cells, and cite analysis conclusions directly in the report.
|
||||
- The governed level requires analysis card inputs to be Call Tool nodes that call a database query tool, or Merge Query nodes, so that it is settled at publish time that the card analyzes queries the platform executed.
|
||||
- Run comparison aligns analysis cards by node and item and lists both runs' verdicts (see [Run comparison](evidence.md#run-comparison)).
|
||||
|
||||
The built-in template 11, Anomaly Analysis (Traceable), is built this way: fetch the weekly totals and two weeks by region → analysis card (fluctuation baseline, change decomposition with totals) → report writer → issuance. It can be published at the governed level as it is (see [Built-in templates](workflows.md#built-in-templates)).
|
||||
|
||||
## Citing in reports
|
||||
|
||||
```
|
||||
[[a:A1]] A1's verdict phrase, rendered by the system from the verdict; one clickable segment
|
||||
[[a:A1.top_share|pct]] one named value of A1; unit conversion as for [[m:]]
|
||||
[[v:A1.r0.share]] one cell of A1's result table
|
||||
[[table:A1 cols=region,delta,share rows=0-4]] A1's result table
|
||||
[[see:A1]] basis of the sentence; attached at the end of the sentence, not rendered
|
||||
```
|
||||
|
||||
- The Analysis Conclusions section of the writing catalog lists each conclusion's number, verdict phrase, named values, the top 10 segments of its result table, and caveats or the reason it couldn't be done.
|
||||
- Conclusions from analysis cards can always be cited. In formal issuance at the governed level, analyses an agent ran on the spot with analysis tools follow the same rule as the cells it queried: they can be cited only when the issuance contract turns on Cell Citations (they can still serve as the basis at the end of a sentence).
|
||||
- The measure name in a verdict phrase comes from the agent or the template; the system first strips digits, percent signs, units, and citation markers from it and cuts it to 20 characters, so text written by the system never smuggles in numbers written by the model.
|
||||
|
||||
## Analysis claim check
|
||||
|
||||
The report-writer node's Analysis Claim Check (`analysis_claims`) checks judgment words sentence by sentence, with a fixed lexicon and binding rules, without calling a model.
|
||||
|
||||
| Class of judgment word | Examples (translated) | Analysis conclusion and verdict it needs |
|
||||
|---|---|---|
|
||||
| Source | mainly from, concentrated in, dragged down, drove, contributed most | Change decomposition judged a main source or concentrated; the segments named must be in the verdict, and a segment said to drag down must be falling |
|
||||
| Dispersed, offset, mix | spread across regions, one up and one down, structural change | Change decomposition judged dispersed, offsetting, or mainly from the mix |
|
||||
| Unusual, record, normal | unusual, sharply, surged, record high, normal fluctuation | Fluctuation baseline judged outside or far outside (direction matching), record high or low, or usual fluctuation |
|
||||
| Trend, no trend | rising steadily, down three weeks in a row, no clear trend | Trend test, with matching direction and run length |
|
||||
| Difference, no difference | significantly higher, really lower, little difference | Difference test judged as holding up (direction matching) or possibly sampling noise |
|
||||
| Causal | because, due to, led to, affected by | A verbatim knowledge-base quote or a run input as the basis in the same sentence, or phrasing as a conjecture (possibly, to be verified) |
|
||||
|
||||
- A sentence's basis is the `[[a:…]]`, `[[v:A…]]`, and `[[table:A…]]` in it plus the `[[see:A…]]` at its end. Negation flips the class: "not unusual" is checked as normal.
|
||||
- Not checked: statements about composition rather than change ("this week's orders came mainly from the online channel"), suggestions and plans, "abnormal" as a business status ("some orders are abnormal"), verbatim knowledge-base quotes, system-rendered verdict phrases, headings, and tables.
|
||||
- The line between source and cause: "the decline was mainly caused by East", citing a breakdown by region and naming one of its segments, is checked as a source, the bookkeeping sense of "came from"; "the decline was due to the end of a promotion" names no segment and is checked as causation.
|
||||
- Three kinds of violation: a judgment word without a matching analysis conclusion; a conclusion cited, but the verdict, the segments named, or the direction don't match (citing a conclusion that couldn't be done counts too); a cause without a basis.
|
||||
|
||||
| Value | What it does |
|
||||
|---|---|
|
||||
| `check` (default, also used by Ask Data) | Sends the report back to the writer for one rewrite; what remains is marked in the report and doesn't affect the issuance tier |
|
||||
| `strict` | Rewrites up to the maximum number of repairs; what remains counts as an issuance gap and downgrades the issuance; required at the governed level (gate G3) |
|
||||
| `off` | No check |
|
||||
|
||||
- When the writing catalog has analysis conclusions, the writer is told to follow the verdicts and preferably cite verdict phrases directly; when it has none, the writer is told not to make such judgments and to state only numbers and changes.
|
||||
- An exit in citation mode checks again independently, with the same check used at assembly. The report document records which disposition and which version of the lexicon it was checked with: only documents that carry this record are checked for it on verification, so verification results for reports from before the upgrade don't change.
|
||||
- It errs on the side of missing cases: only strong judgment words are collected. The finer points it misses are left to the claim-sentence judge (see [Claim sentences](evidence.md#claim-sentences)); when the judge's excerpt includes an analysis conclusion, the judge is also told that the verdict was computed by the system and to judge only whether the sentence states it correctly.
|
||||
|
||||
## Analysis scope
|
||||
|
||||
When a report uses analysis conclusions (when this run has any), the system appends an Analysis Scope section at the end of the report. The writer can't change it; it is sealed with the document, and its numbers are clickable:
|
||||
|
||||
- **Phenomena**: verdict phrases of comparability checks, fluctuation baselines, and trend tests.
|
||||
- **Broken down**: each change decomposition, listed along its drill-down chain with its dimensions and verdict.
|
||||
- **Ruled out**: decompositions judged dispersed, and difference tests where the difference may be just sampling noise. A breakdown that finds no concentration is a conclusion too.
|
||||
- **Not done**: the reason for each analysis that couldn't be done.
|
||||
- **Not broken down**: candidate dimensions that weren't broken down. Candidate dimensions are computed deterministically from the data catalog: columns on the fact table used by the decomposition's query that are marked as an attribute or status, aren't masked, and have a code-value table (at most 50 values, or marked complete), plus the same kind of columns on dimension tables related to it. When the columns used by the decomposition don't match the catalog, it says only that candidate dimensions can't be checked, rather than listing guesses.
|
||||
- **Tests**: how many difference tests were run this time, and whether they were corrected for the number of tests.
|
||||
|
||||
The sentence an agent writes about what a call is testing doesn't go into this section; it appears only in the evidence panel. The report-writer node's Analysis Scope (`analysis_scope`) defaults to appending the section when there are analysis conclusions, and can be set not to.
|
||||
|
||||
## Settings
|
||||
|
||||
Under Settings → Preferences → Analysis you can change the thresholds used for verdicts, how many analyses an agent may run per execution, how many levels it may drill down, and how many segments are listed. For each item's label and default, see [Analysis settings](configuration.md#analysis-settings).
|
||||
|
||||
- The group is snapshotted into the run when the run starts (only when the graph has an analysis card or an agent with analysis tools), and resuming or continuing uses the same snapshot.
|
||||
- Agents can't change thresholds: the analysis tools have no threshold parameters. A template can write threshold overrides on an analysis card.
|
||||
- Every conclusion records the thresholds actually used, and its criterion is written from them, for example "a segment accounting for at least half of the total change is the main source".
|
||||
|
||||
## Limits
|
||||
|
||||
- Not in this round: forecasting; correlation analysis; dimension hierarchies in the data catalog; profiling of numeric columns; analysis tools for the verification agent of double-blind recomputation, adversarial review, and approval inquiry; analysis conclusions in caliber ledger matching and recalls (numbers in analysis conclusions get no caliber label; see [Caliber ledger](evaluation-and-ledger.md#caliber-ledger)).
|
||||
- The trend test assumes independent periods; for autocorrelated series, p comes out too small.
|
||||
- Candidate dimensions come only from code-value tables: profiling doesn't yet record each column's number of distinct values, so attribute columns without a code-value table aren't candidates.
|
||||
- The lexicon collects only strong judgment words, so a different wording may slip through; what slips through is left to the claim-sentence judge and to the people reading the report.
|
||||
- Analyses done by members of a collaboration team don't go into the evidence ledger, so reports can't cite them.
|
||||
+14
-1
@@ -25,6 +25,17 @@ A turn covers: Copilot builds the graph, the run starts, the result is reviewed,
|
||||
- When Copilot returns only thinking and no operations, the turn is retried once automatically.
|
||||
- The cost of each turn includes the model calls for Copilot's graph building, table selection, and review.
|
||||
|
||||
## Analytical questions
|
||||
|
||||
When the question asks why, how something changed, what dragged it down, what contributed most, whether something is unusual, how much it fluctuates, whether it keeps rising, or whether one is really higher than the other, the agent that queries data in the graph Copilot builds is also given the five analysis tools (see [Analysis](analysis.md)):
|
||||
|
||||
- The agent first settles which two periods to compare from the question (the previous whole period when the question doesn't say), then follows the platform's built-in analysis method: comparability check, putting the period back into its history, breaking down by dimension, and drilling down one level once the main source is found. The platform computes the verdicts with fixed methods and thresholds; the agent decides only what to compute.
|
||||
- The report is written from the analysis conclusions, citing the system-written verdict phrases directly for its judgments. When the writer writes judgment words such as "mainly from" or "unusual" itself, the system checks each sentence: sentences that don't cite a matching analysis conclusion go back for one rewrite, and what remains is marked in the answer without affecting the issuance tier (see [Analysis claim check](analysis.md#analysis-claim-check)).
|
||||
- At the end of an answer that uses analysis conclusions, the system appends an Analysis Scope section: what was broken down, what was ruled out, which candidate dimensions weren't broken down, and what couldn't be done (see [Analysis scope](analysis.md#analysis-scope)).
|
||||
- A breakdown shows only where a change came from, not why: causes in an answer need a verbatim knowledge-base quote as their basis, or else they are written as conjectures to be verified.
|
||||
- During its self-check, Copilot recognizes analytical questions with a fixed lexicon that collects only explicit phrasings. When the question is analytical but the agent that queries data has no analysis tools, the problem goes back to Copilot for one round of fixes; if the tools are still missing after that round, the graph runs as usual with a warning.
|
||||
- When an analysis couldn't be done (not enough history, the current period isn't over, and so on), or judgment words still don't cite a matching conclusion after the rewrite, the review only writes a note and doesn't start over automatically (see [Output review](#output-review)).
|
||||
|
||||
## Output review
|
||||
|
||||
After a run ends, rules scan the event stream first, and a model review runs only when they find an anomaly. The review may restructure the answer, but it **must not introduce numbers that aren't in the original answer**.
|
||||
@@ -32,10 +43,11 @@ After a run ends, rules scan the event stream first, and a model review runs onl
|
||||
- Ask Data for users (the user role) is reviewed too.
|
||||
- When a step fails inside a subworkflow, automatic review reports it only once.
|
||||
- Continue Run in Ask Data is limited to builders and above.
|
||||
- The gaps from analysis (some analyses couldn't be done, an analysis call was wrong and never fixed, judgment words still failed the check after the rewrite, the two periods of a caliber card metric couldn't be checked for comparability) only get a note and never trigger an automatic redo. When a run has only these, the review calls no model and writes the note from their wording as is.
|
||||
|
||||
## Calibers in an answer
|
||||
|
||||
A caliber is the agreed definition of how a metric is fetched and computed. It lives in a caliber card, with a name and a version. After an answer is sealed, the system compares every number in it with the published calibers in the background and labels each number with a caliber; each turn gets a one-line summary. When the whole answer is equivalent to a governed caliber, the system automatically starts a formal run by that caliber. See [Evaluation and the Caliber Ledger](evaluation-and-ledger.md#caliber-ledger).
|
||||
A caliber is the agreed definition of how a metric is fetched and computed. It lives in a caliber card, with a name and a version. After an answer is sealed, the system compares every number in it with the published calibers in the background and labels each number with a caliber; each turn gets a one-line summary. When the whole answer is equivalent to a governed caliber, the system automatically starts a formal run by that caliber. Numbers in analysis conclusions (shares, changes, fluctuation ranges) are derived from query results, so they aren't compared with calibers, get no caliber label, and aren't counted in the summary line. See [Evaluation and the Caliber Ledger](evaluation-and-ledger.md#caliber-ledger).
|
||||
|
||||
## From an answer to a workflow or an app
|
||||
|
||||
@@ -60,6 +72,7 @@ Copilot generates or rewrites a whole graph from one sentence and lays it out au
|
||||
|
||||
- When it edits a graph, it outputs only the change operations instead of rewriting the whole graph.
|
||||
- Before delivering, it checks its work with the same rules the runtime uses. Problems that would make the run fail go back to it to fix first; if they can't be fixed, nothing runs automatically.
|
||||
- In Ask Data, the self-check also looks at the question itself: when the question is analytical but the agent that queries data has no analysis tools, it gets one round of reminders (see [Analytical questions](#analytical-questions)).
|
||||
- While building graphs, it records facts worth remembering in long-term memory.
|
||||
|
||||
### Looking on demand
|
||||
|
||||
@@ -67,7 +67,7 @@ For the app database, checkpoint database, connection pool, PgBouncer, and multi
|
||||
| Location | Contents |
|
||||
|---|---|
|
||||
| Settings → Model Access | Providers and keys |
|
||||
| Settings → Preferences | Run defaults (default memory scope, default knowledge base, require approval for dangerous tools by default); the evidence judge (judge model and limits); verification (verification model, the switches and limits for double-blind recomputation, adversarial review, and approval inquiry, and the daily spending cap; see [The review suite](evidence.md#the-review-suite)); evaluation concurrency, per-question time limit, and per-evaluation and daily spending; run concurrency (admins); the caliber ledger (automatic issuance by caliber and its daily cap, the caliber matching time limit, the recall limit `ledger.recall_limit`; see [Caliber ledger](evaluation-and-ledger.md#caliber-ledger)); drafting model; apps (the daily cap on on-demand runs per app, letting users create personal apps, allowing share links); interface theme |
|
||||
| Settings → Preferences | Run defaults (default memory scope, default knowledge base, require approval for dangerous tools by default); the evidence judge (judge model and limits); verification (verification model, the switches and limits for double-blind recomputation, adversarial review, and approval inquiry, and the daily spending cap; see [The review suite](evidence.md#the-review-suite)); evaluation concurrency, per-question time limit, and per-evaluation and daily spending; analysis (the thresholds used for verdicts, how many analyses an agent may run per execution, the drill-down depth, how many segments are listed; see [Analysis settings](#analysis-settings)); run concurrency (admins); the caliber ledger (automatic issuance by caliber and its daily cap, the caliber matching time limit, the recall limit `ledger.recall_limit`; see [Caliber ledger](evaluation-and-ledger.md#caliber-ledger)); drafting model; apps (the daily cap on on-demand runs per app, letting users create personal apps, allowing share links); interface theme |
|
||||
| Settings → General | Public URL |
|
||||
| Settings → Runtime Environment | Sandbox candidates and their availability, the current choice and its reason; current values such as time limits |
|
||||
| Settings → Identity | Login mode; enabling Ask Data for users |
|
||||
@@ -77,6 +77,30 @@ For the app database, checkpoint database, connection pool, PgBouncer, and multi
|
||||
|
||||
Other settings: `notify.digest_at` (the time of the daily task digest), and the agent token budget, spending budget, and step count.
|
||||
|
||||
### Analysis settings
|
||||
|
||||
The Analysis section under Settings → Preferences governs how change decomposition, the fluctuation baseline, the trend test, and the difference test judge, and how many analyses an agent may run (see [Analysis](analysis.md)). It is snapshotted into a run when the run starts, and resuming or continuing uses the same snapshot; an analysis card can carry threshold overrides, and agents can't change any of it.
|
||||
|
||||
| Setting | Default | Range | What it does |
|
||||
|---|---|---|---|
|
||||
| Main-source share | 0.5 | 0.3–0.9 | A segment (a factor, an effect) accounting for this share of the total change is the main source |
|
||||
| Concentration share | 0.8 | 0.5–0.99 | The top few segments together accounting for this share count as concentrated; can't be below the main-source share |
|
||||
| Offset ratio | 0.25 | 0.05–1 | When the segments moving the other way add up to more than this share of the total change, a caveat says they offset each other |
|
||||
| Flat threshold | 0.005 | 0–0.1 | A total change below this share of the base counts as essentially flat |
|
||||
| Outside-fluctuation multiple | 2 | 1–5 | Fluctuation baseline: this many robust standard deviations from the historical median counts as outside the usual fluctuation |
|
||||
| Far-outside multiple | 3 | 1.5–8 | This many counts as far outside; must be greater than the outside-fluctuation multiple |
|
||||
| Minimum history periods | 8 | 4–104 | With less history than this, the fluctuation baseline can't be done |
|
||||
| Maximum history periods | 52 | 8–260 | The fluctuation baseline and the trend test use at most this many periods; can't be below the minimum |
|
||||
| Minimum trend periods | 6 | 4–104 | The trend test needs at least this many periods |
|
||||
| Trend significance level | 0.05 | 0.001–0.2 | Rising or falling is judged only when p is below this |
|
||||
| Difference-test significance level | 0.05 | 0.001–0.2 | A corrected p below this means the difference holds up |
|
||||
| Minimum sample size for means | 30 | 2–10000 | When a group has fewer samples in a comparison of means, a caveat says the sample is small |
|
||||
| Minimum expected frequency for rates | 5 | 1–50 | When a cell's expected frequency is below this, Fisher's exact test is used |
|
||||
| Minimum denominator for ratio breakdowns | 30 | 1–100000 | Segments with a smaller denominator are marked small and left out of the Simpson's paradox judgment |
|
||||
| Maximum analyses per agent execution | 12 | 1–50 | Once used up, further analysis tool calls from the agent are told the analyses are used up |
|
||||
| Maximum drill-down depth | 3 | 1–6 | How many levels a change decomposition can drill down |
|
||||
| Segments listed | 10 | 3–30 | How many segments of a result table the writing catalog and the reply to the agent list |
|
||||
|
||||
## Sandbox backend
|
||||
|
||||
Settings → Runtime Environment lists every candidate and its availability, and marks the current choice and its selection reason. Automatic selection works in this order:
|
||||
|
||||
@@ -16,10 +16,12 @@ Each question in a test suite is a question plus an expected answer. The expecta
|
||||
- A metric of a published caliber.
|
||||
- A query identified from an Ask Data run.
|
||||
- "No data", or a clarifying question back.
|
||||
- The main source of a change (contribution): an expected SQL statement plus change decomposition parameters; the main source (one segment or the top few) and the direction are computed with the same analyses the answer uses.
|
||||
- A verdict: an expected SQL statement plus parameters for a fluctuation baseline, trend test, difference test, or comparability check; the class of verdict is computed with the same analyses: usual fluctuation or unusual; rising, falling, or no clear trend; a difference that holds up or not enough to conclude; comparable or not comparable (see [Analysis](analysis.md#the-five-analyses)).
|
||||
|
||||
Rules:
|
||||
|
||||
- Numeric values may not be typed in by hand.
|
||||
- Numeric values may not be typed in by hand; nor may the segments, directions, and verdicts in contribution and verdict expectations, which are computed from the expected SQL with the same analyses, once before and once after the answer.
|
||||
- Text key points are judged by the judge, listed separately, and not counted toward accuracy.
|
||||
- After a caliber version bump, questions that use it show a notice at the top. Each question takes one of three options: switch to pinning the new version, compute one expectation for each of the old and new versions, or archive the old question and create a new one. Until then, the question is skipped in evaluations.
|
||||
- Test suites can be imported and exported.
|
||||
@@ -42,7 +44,9 @@ Rules:
|
||||
- **Comparison**: against the previous evaluation or a baseline, with per-question transitions and attribution.
|
||||
- **A/B paired table**: exact McNemar test; display only, not a gate.
|
||||
|
||||
Evaluation runs are left out of usage statistics. The scoring rules are at version 2; old evaluations can be re-scored under the new rules from the evaluation details (reading only the sealed records, without calling models).
|
||||
Contribution and verdict expectations are scored like this: an answer that cites an analysis conclusion whose verdict matches is correct and traceable; body text that states it correctly by the lexicon without citing it is correct but uncited; a source or verdict that doesn't match, or the opposite answer, is wrong; an expectation that itself can't be computed (the expected SQL returns too little history, say) can't be scored. The input queries of analysis conclusions also count as queries the answer cited.
|
||||
|
||||
Evaluation runs are left out of usage statistics. The scoring rules are at version 3 (which adds contribution and verdict expectations); old evaluations can be re-scored under the new rules from the evaluation details (reading only the sealed records, without calling models).
|
||||
|
||||
### Scheduled evaluations
|
||||
|
||||
@@ -68,6 +72,7 @@ After an exploratory run is sealed, the system compares the answer with publishe
|
||||
- Judging is number by number: when a report cites several columns of one query (for example sales, order count, first and last day), each number is compared on its own column against the caliber's metric column, and extra queried columns don't affect it. This applies only when removing the other columns wouldn't change the number; queries with DISTINCT, truncation after sorting by another column, or extra grouping columns are still compared as a whole.
|
||||
- When a multi-row query has no `ORDER BY`, citing one of its cells doesn't count as "from a caliber", and recall and reconciliation mark it "Row order undefined".
|
||||
- An answer is issued automatically only when every number in it that comes from data is proven to come from the same caliber.
|
||||
- Numbers in analysis conclusions (shares, changes, fluctuation ranges) are derived from query results, so they aren't compared or labeled with a caliber and aren't counted in the summary line.
|
||||
|
||||
When writing a caliber's data-fetch SQL, it is best not to round (`ROUND(SUM(x), 2)`) or wrap values in `COALESCE(…, 0)`. Leave rounding and defaults to the caliber card's decimal places and display settings; otherwise Ask Data queries written without rounding will never be recognized as equivalent (and where they differ is stated plainly).
|
||||
|
||||
|
||||
+25
-2
@@ -2,7 +2,7 @@ English | [Simplified Chinese](../zh-CN/evidence.md)
|
||||
|
||||
# Evidence and Issuance
|
||||
|
||||
This page covers where the numbers in a report come from and how to check them: caliber cards and issuance contracts, the three issuance tiers, citation markers in the report-writer node, what happens when the queried range has no data, what you see when you click a number, sealing and verification, the two kinds of export, and the review tools: double-blind recomputation, adversarial review, approval inquiry, the what-if sandbox, and run comparison. It is for builders, approvers, and people who read reports. The user interface is currently available in Simplified Chinese only.
|
||||
This page covers where the numbers in a report come from and how to check them: caliber cards and issuance contracts, the three issuance tiers, citation markers in the report-writer node (including analysis conclusions), what happens when the queried range has no data, what you see when you click a number or an analysis conclusion, sealing and verification, the two kinds of export, and the review tools: double-blind recomputation, adversarial review, approval inquiry, the what-if sandbox, run comparison, and the analysis claim check. It is for builders, approvers, and people who read reports. For how analytical judgments themselves are computed, see [Analysis](analysis.md). The user interface is currently available in Simplified Chinese only.
|
||||
|
||||
## The basic idea
|
||||
|
||||
@@ -45,6 +45,9 @@ Measured: every number in body text written by Opus 5 traced back successfully,
|
||||
|
||||
When the contract sets `report_from`, the check goes by citation: the hash is verified again when the report document is retrieved, everything is recomputed independently, and only when every number carries a citation is the issuance judged complete.
|
||||
|
||||
- The inputs of caliber card metrics the report cites must trace back to raw data (cells of query snapshots, run inputs): in formal runs at the governed level, inputs that can't be traced count as gaps and downgrade the issuance; otherwise they are stated in the issuance declaration, and the evidence panel still marks them "Can't trace back to raw data".
|
||||
- Analysis claims in the report are checked again here independently, with the same check used when the report is assembled: when the report-writer node is set to strict, remaining violations count as issuance gaps (see [Analysis claim check](#analysis-claim-check)).
|
||||
|
||||
### Metric issuance
|
||||
|
||||
A pure metrics dashboard (the contract sets only `metrics_from`, there is no report-writer node upstream, and the output contains no model-written text) is issued by its metrics. With no narrative to check, the metrics themselves are checked:
|
||||
@@ -62,6 +65,8 @@ When the source query of a contract metric has SQL check errors, the issuance is
|
||||
- An expression can be written as `cell(nodes.pull, 0, 'gmv')`, which makes the provenance precise to the cell.
|
||||
- Leave rounding and default values to the caliber card's decimal places and display settings instead of writing them into the data-fetch SQL (for why, see [Caliber Ledger](evaluation-and-ledger.md#a-caliber-label-for-each-number)).
|
||||
- When publishing at the governed level, caliber card inputs can't come from sandbox code in the compute role (gate G4).
|
||||
- When a metric uses two queries of the same shape that differ only in their dates (this week versus last week), the system runs a comparability check once automatically (see [Comparability check](analysis.md#comparability-check)) and attaches the result to the metric: the writing catalog states it plainly; at an exit with a contract, incomparable periods count as a gap and downgrade the issuance, and conditionally comparable ones are only flagged.
|
||||
- Analysis card results are not caliber card inputs: a caliber card that reads them is treated as unable to trace back to raw data.
|
||||
|
||||
### Caliber version bumps
|
||||
|
||||
@@ -88,9 +93,13 @@ In the report-writer node (`report`), the model writes only citation markers, an
|
||||
[[table:Q1 cols=region,amount rows=0-4]] the system builds the whole table from the snapshot; every cell is clickable
|
||||
[[t:orders]] [[c:orders.week]] table name, field name; names written in backticks are checked too
|
||||
[[q:K1|<quote>]] verbatim knowledge-base quote; must appear word for word in the chunks that retrieval hit
|
||||
[[see:m:gmv,Q1]] basis of a claim sentence; attached at the end of the sentence, not rendered
|
||||
[[a:A1]] [[a:A1.top_share|pct]] an analysis conclusion's verdict phrase and a named value, both rendered by the system
|
||||
[[v:A1.r0.share]] [[table:A1 rows=0-4]] one cell and the whole of an analysis conclusion's result table
|
||||
[[see:m:gmv,Q1,A1]] basis of a claim sentence; attached at the end of the sentence, not rendered
|
||||
```
|
||||
|
||||
Analysis conclusions are numbered A1, A2, … in ledger order, and the writing catalog has a separate Analysis Conclusions section. Judgment words such as "mainly from", "unusual", or "rising steadily" must cite a matching analysis conclusion and agree with its verdict; see [Analysis claim check](analysis.md#analysis-claim-check).
|
||||
|
||||
### Handling violations
|
||||
|
||||
- Bare numbers and citations that can't be parsed are flagged as violations on the spot and sent back for rewriting (`max_repairs`, default 1).
|
||||
@@ -99,6 +108,7 @@ In the report-writer node (`report`), the model writes only citation markers, an
|
||||
## The data retrieval chain
|
||||
|
||||
- Data fetched by agents, tool nodes, code nodes, and knowledge retrieval is recorded in the evidence ledger. The report's source list is numbered from the ledger: Q1… for queries, K1… for knowledge-base chunks.
|
||||
- Analysis conclusions from agents' analysis tools and from analysis cards are recorded in the evidence ledger too, numbered A1…. Analyses read only query snapshots the platform executed during this run, and each conclusion is sealed together with its inputs (see [Analysis](analysis.md)).
|
||||
- When an agent turns on `cite_fields` (together with `output_schema`), each field records which cell of which query it came from, and that is checked against the snapshot: on a mismatch the snapshot wins and a warning is raised; anything not found is recorded as null, not defaulted to 0.
|
||||
- A code node's `evidence_role` is `source` (data retrieval) or `compute` (computation, the default). Numbers computed in the sandbox can't be cited directly by a report; they must go through a caliber card.
|
||||
- Cells of uploaded spreadsheets can be traced back to the cell in the original sheet (an inferred source).
|
||||
@@ -126,6 +136,7 @@ Query a table that keeps only the latest day (fully reloaded every day) for last
|
||||
- **Numbers**: the metric's original formula, substituted formula, and recomputed result; whether the inputs match the snapshot; the query's SQL; the cited row plus 2 rows before and after, highlighted down to the cell; and finally the seal status. Only artifacts returned by events inside the sealed scope are accepted. The provenance chain also starts with a Caliber step (see [Caliber Ledger](evaluation-and-ledger.md#a-caliber-label-for-each-number)).
|
||||
- **Table and field names**: which query they appear in, their type, and the table they belong to. A name found nowhere in the schema snapshot, the SQL, or the result columns is marked as a suspicious entity ("May be a fabricated name", with the 3 closest matches). When a snapshot was truncated and can't be checked, it only says so, without wrongly calling the name fabricated.
|
||||
- **Knowledge-base quotes**: must appear word for word in the chunks that retrieval hit.
|
||||
- **Analysis conclusions** (verdict phrases, named values, result table cells): in order, the verdict (verdict phrase, criterion, segments named and direction, who did it, caveats, and the agent's note on what the call was testing), the method (method and version, parameters, thresholds actually used, table ranges checked live), the calculation (the result table and the substituted formula for the key value, such as "share = East's change / total change = -120 / -194", with the cited cell highlighted), the inputs (which columns and rows of which Q, the closure check against the two total cells, the level a drill-down sits under, followed by the steps of the input queries), the recomputation (rerun from the sealed inputs with the same function; it verifies only when every field is identical, and when the method has a new version, it says which version the run used), and the seal. In the evidence graph, an analysis conclusion has a "derived from" edge to its inputs and to the level above it.
|
||||
|
||||
## Claim sentences
|
||||
|
||||
@@ -165,12 +176,14 @@ The two exports serve different purposes: to prove nothing was altered, use the
|
||||
- The body is rendered on the server, so it is readable without running scripts (email attachment previews often don't run scripts).
|
||||
- With scripts running, every number, every number without evidence, and every suspicious table or field name is clickable. The panel shows the same provenance chain as the evidence panel on the web page (caliber, source, calculation, SQL, cited rows), with the seal status marked item by item.
|
||||
- Masked columns say "Masked", following the same rules as the web page.
|
||||
- Verdict phrases, named values, and result table cells of analysis conclusions are clickable too, and the panel shows the verdict, method, calculation, and inputs in turn; at export time the server recomputes each conclusion from the sealed inputs and writes the result into the panel. The Analysis Scope at the end of the report is rendered from what the document recorded, with clickable numbers.
|
||||
- Unsealed runs export as usual, for the same reason you can read their reports on the web.
|
||||
|
||||
**Verifiable report**: Export Verifiable Report under Provenance Details on the Records page (`GET /api/runs/{id}/bundle.html`) produces a seal verification document.
|
||||
|
||||
- The recipient recomputes the hashes in the browser, without connecting to the service or installing anything, and checks that no number, document, or run record has been altered.
|
||||
- The provenance panel for a number likewise shows the caliber, original formula, substituted formula, inputs, and recomputation; the query snapshots that caliber card inputs point to come attached to the file.
|
||||
- Analysis conclusions the report cites (including those named in the Analysis Scope) come attached to the file byte for byte, together with their input snapshots and the conclusions they drill down from; at export time the server recomputes them from the sealed inputs and records the result on each conclusion's entry. The browser only verifies hashes and shows the server's recomputation; the page doesn't recompute statistics itself. When any input snapshot can't be shown to leave masked columns untouched, the conclusion is given by id only.
|
||||
- It proves only that the file is internally consistent: numbers, documents, and run records agree with one another and haven't been altered. To check authenticity, compare the seal hash on the page with the seal verification of this run in the system (`GET /api/runs/{id}/verify`); the page says so prominently.
|
||||
- Events appended after sealing (annotations such as on-demand judging and approval inquiries) are listed separately, with only their type, sequence number, and a content summary, and aren't part of the manifest.
|
||||
- When a data source has masked columns, query snapshots that can't be shown to leave them untouched, and caliber cards with inputs from masked columns, are given by id only, without content. Masking only reduces exposure and is not a security boundary.
|
||||
@@ -246,4 +259,14 @@ For two runs of the same workflow (this week's and last week's weekly report, sa
|
||||
- When the caliber and data both changed and a strict split is possible, the caliber effect and the data effect are given separately, and they add up to the total change.
|
||||
- For metrics in the data-only, parameters-only, and caliber-and-data classes, when attribution is possible, the dimension values (regions, say) with the largest changes are listed, up to 10. Every item traces back to the query snapshots on both sides; queries run by verification take no part in the comparison.
|
||||
- Values from masked columns never appear in the result, and metrics that use masked columns aren't split.
|
||||
- When the graph has analysis cards, the two runs' analysis cards are aligned by node and item, listing each run's status, verdict, direction, and verdict phrase; for cards executed several times in a loop, the last execution counts.
|
||||
- When a scheduled run finishes successfully and is sealed, it is compared once automatically with the same schedule's last successful run; the trigger history and notifications show only which run was the baseline and how many metrics fall into each class, never the actual numbers.
|
||||
|
||||
### Analysis claim check
|
||||
|
||||
Judgment words in a report, such as "mainly from East", "outside the usual fluctuation", "rising steadily", or "the difference holds up", must cite a matching analysis conclusion and agree with its verdict; a stated cause needs a verbatim knowledge-base quote or a run input as its basis, or else it is written as a conjecture to be verified.
|
||||
|
||||
- The report-writer node checks this sentence by sentence while writing, with a fixed lexicon and binding rules: no model call, no cost. An exit in citation mode checks again independently.
|
||||
- By default, the report goes back for one rewrite and what remains is marked in the report (visible in the evidence panel, the audit, and exports), without affecting the issuance tier; set to strict, it rewrites up to the maximum number of repairs, and what remains counts as an issuance gap and downgrades the issuance. The governed level requires strict (gate G3).
|
||||
- Only new report documents that carry this check's record are checked; verification results for reports from before the upgrade don't change.
|
||||
- For the classes of judgment words, the analysis each one needs, and which wordings aren't checked, see [Analysis claim check](analysis.md#analysis-claim-check).
|
||||
|
||||
+12
-2
@@ -2,17 +2,27 @@ English | [Simplified Chinese](../zh-CN/tools.md)
|
||||
|
||||
# Tools, MCP, and the Capability Registry
|
||||
|
||||
This page covers the tools workflows can call (built-in tools, custom tools, and MCP), their approval levels, how admins govern them in the capability registry, and how to register endpoints for external systems. It is for builders and admins. The user interface is currently available in Simplified Chinese only.
|
||||
This page covers the tools workflows can call (built-in tools, including the analysis tools, custom tools, and MCP), their approval levels, how admins govern them in the capability registry, and how to register endpoints for external systems. It is for builders and admins. The user interface is currently available in Simplified Chinese only.
|
||||
|
||||
## Kinds of tools
|
||||
|
||||
- **Built-in tools**: 15 of them.
|
||||
- **Built-in tools**: 23 of them, 5 of which are analysis tools (see below).
|
||||
- **Custom tools**: an HTTP template, or Python in the sandbox.
|
||||
- **MCP**: connect MCP servers (stdio or http).
|
||||
- **Data source tools**: each data source's query tool, `db_query__<source>`, plus `db_schema__` for looking up schemas at run time (see [Data Sources](data-sources.md)).
|
||||
|
||||
Who can configure what: MCP stdio, endpoints and credentials, and capability governance can be changed only by admins; builders can read them, and users can't see them (see the [permission table](identity-and-security.md#permission-table)).
|
||||
|
||||
## Analysis tools
|
||||
|
||||
The five analysis tools are the platform's deterministic analyses: `analysis_decompose` (change decomposition), `analysis_baseline` (fluctuation baseline), `analysis_trend` (trend test), `analysis_differ` (difference test), and `analysis_comparable` (comparability check). They don't query any database; they analyze only results that the agent they are bound to queried during this execution. The platform computes the verdicts with fixed methods and thresholds, and the system writes the verdict phrases (see [Analysis tools for agents](analysis.md#analysis-tools-for-agents)).
|
||||
|
||||
- Their parameters include no thresholds. Binding any one of them injects the platform's built-in analysis method into the agent's system prompt, once.
|
||||
- They are read-only, write nothing, and aren't dangerous tools, so under the default approval policy they don't wait for approval; users can use them in Ask Data too.
|
||||
- They can be bound only to agents: putting one in a Call Tool node is a canvas validation error. For fixed analyses in a template, use an analysis card (see [Analysis cards](analysis.md#analysis-cards)).
|
||||
- Each agent can run at most 12 analyses per execution (changed under Settings → Preferences → Analysis).
|
||||
- They don't connect to data sources: they read snapshots already queried during this execution, and what they return is an analysis conclusion, not a query result.
|
||||
|
||||
## Approval levels
|
||||
|
||||
The platform can't tell what MCP and custom tools do, so by default every call to them waits for approval. There are three approval levels: `ask`, `gated`, and `always`.
|
||||
|
||||
@@ -4,6 +4,22 @@ English | [Simplified Chinese](../zh-CN/upgrade-notes.md)
|
||||
|
||||
This page lists, newest first, the changes in each update that affect users and operators, and what to do when upgrading. It is for operators and admins. The features themselves are described in the topic documents. The user interface is currently available in Simplified Chinese only.
|
||||
|
||||
## 2026-10, the analysis round
|
||||
|
||||
**What to do**: nothing is required, and the database schema doesn't change. For report templates you will publish again at the governed level, set the report-writer node's Analysis Claim Check to strict; the pre-publish check points it out, and it can be fixed in one click.
|
||||
|
||||
- New analysis capability (see [Analysis](analysis.md)): five deterministic analyses (change decomposition, fluctuation baseline, trend test, difference test, comparability check). The system computes the verdicts and writes the verdict phrases, and reports cite them with `[[a:A1]]`; they are clickable and can be recomputed.
|
||||
- A new node, the analysis card (`analysis`), and five analysis tools; built-in tools go from 18 to 23. A new template, 11 Anomaly Analysis (Traceable), is added to the template list automatically on the first start after upgrading. Templates are installed once by name: in existing deployments, template 9, Traceable Weekly Report, isn't updated, so when publishing it at the governed level, add the analysis claim check as the pre-publish check suggests.
|
||||
- Report-writer nodes check analysis claims by default (rewrite once, then mark): sentences whose judgment words such as "mainly from", "unusual", or "rising steadily" don't cite a matching analysis conclusion, or whose causes have no basis, go back to the writer for one rewrite; what remains is marked in the report without affecting the issuance tier. The writing rules gain a passage about judgment words, including for reports in graphs without any analysis: when there are no analysis conclusions, the writer is told not to make such judgments and to state only numbers and changes. To skip the check, set the report-writer node's Analysis Claim Check to off.
|
||||
- The governed level requires the Analysis Claim Check to be strict (one more item under gate G3). Versions already published keep running as usual, checked with the default rewrite-once-then-mark; the next publish is checked against the new rule.
|
||||
- Reports that use analysis conclusions get a system-written Analysis Scope section at the end, outside the body. Report documents also record which disposition and which version of the lexicon their analysis claims were checked with: only new documents that carry this record are checked for it on verification, so verification results for reports from before the upgrade don't change.
|
||||
- An exit in citation mode now collects the metrics the report cites: inputs that can't be traced back to raw data count as gaps and downgrade the issuance in formal runs at the governed level; at other levels they are still only stated in the issuance declaration.
|
||||
- A caliber card metric that uses two queries differing only in their dates gets a comparability check automatically: at an exit with a contract, incomparable periods count as a gap and downgrade the issuance, and conditionally comparable ones are only flagged.
|
||||
- The report export and the verifiable report include analysis conclusions and their input snapshots, recomputed from the sealed inputs for verification.
|
||||
- Settings gain a Preferences → Analysis section (see [Analysis settings](configuration.md#analysis-settings)). It is snapshotted into a run only when the graph has an analysis card or an agent with analysis tools, so the start records of other runs don't change at all.
|
||||
- Ask Data: Copilot gains a graph shape for analytical questions, and its self-check reminds it to give the querying agent the analysis tools (one round only). The review merges several analyses that couldn't be done into one "n analyses couldn't be done" note, without an automatic redo; analysis calls that were wrong but later fixed no longer count as a gap, and ones never fixed no longer trigger a redo of the whole turn.
|
||||
- Evaluation gains contribution and verdict expectations, and the scoring rules move to version 3 (see [Regression evaluation](evaluation-and-ledger.md#regression-evaluation)).
|
||||
|
||||
## 2026-10, orchestration improvements wave 2: PostgreSQL and multiple processes
|
||||
|
||||
**What to do**: in a multi-process deployment, stop all worker processes before upgrading; pass values that code nodes need explicitly; don't delete `.agentlab-data` in the data directory.
|
||||
|
||||
@@ -40,7 +40,7 @@ When motion is turned off at the system level (`prefers-reduced-motion`), the li
|
||||
|
||||
### Built-in templates
|
||||
|
||||
There are 9 built-in templates, from Minimal Q&A to Traceable Weekly Report. They run as they are and also serve as executable documentation of how each kind of node is used.
|
||||
There are 11 built-in templates, from Minimal Q&A to Anomaly Analysis (Traceable). They run as they are and also serve as executable documentation of how each kind of node is used. Templates 9, Traceable Weekly Report, and 11, Anomaly Analysis (Traceable), can be published at the governed level as they are: the first demonstrates caliber cards and issuance by citation; the second demonstrates the analysis card, which takes two queries, judges whether this week's change is within the usual fluctuation and which region it mainly came from, and leaves the report to cite only the verdict phrases (see [Analysis](analysis.md#analysis-cards)).
|
||||
|
||||
## Node types
|
||||
|
||||
@@ -51,9 +51,9 @@ There are 9 built-in templates, from Minimal Q&A to Traceable Weekly Report. The
|
||||
| Execution | `tool` `code` `merge` | Call a tool directly / execute code in the sandbox / merge the results of several queries (aligned by key across databases) |
|
||||
| Control | `branch` `loop` `subgraph` | Conditional or semantic branch / iteration and conditional loop / nested workflow |
|
||||
| Context | `memory` `retrieve` `transform` | Long-term memory read/write / knowledge retrieval / data shaping |
|
||||
| Gatekeeping | `human` `validate` `metrics` | Human-in-the-loop / Schema validation and automatic rework / caliber card |
|
||||
| Gatekeeping | `human` `validate` `metrics` `analysis` | Human-in-the-loop / Schema validation and automatic rework / caliber card / analysis card (a fixed set of deterministic analyses on upstream query results) |
|
||||
|
||||
The `output` node validates structured output against a JSON Schema; when the output doesn't conform, the errors go back to the model for automatic rework. For `metrics` (caliber card) and `report` (report writer), see [Evidence and Issuance](evidence.md); for `merge` (merge query), see [Data Sources](data-sources.md#cross-database-merge).
|
||||
The `output` node validates structured output against a JSON Schema; when the output doesn't conform, the errors go back to the model for automatic rework. For `metrics` (caliber card) and `report` (report writer), see [Evidence and Issuance](evidence.md); for `analysis` (analysis card), see [Analysis](analysis.md#analysis-cards); for `merge` (merge query), see [Data Sources](data-sources.md#cross-database-merge).
|
||||
|
||||
## Passing data between nodes
|
||||
|
||||
@@ -125,7 +125,7 @@ Approvals, supplementary input, draft editing, action confirmations, code and to
|
||||
|
||||
## Agent nodes
|
||||
|
||||
An `agent` node is a model loop with tools. An allowlist on the node decides which tools it can use (see [Tools](tools.md)). An agent with `output_schema` can turn on `cite_fields`, so each field records which cell of which query it came from (see [Evidence and Issuance](evidence.md#the-data-retrieval-chain)).
|
||||
An `agent` node is a model loop with tools. An allowlist on the node decides which tools it can use (see [Tools](tools.md)). An agent with `output_schema` can turn on `cite_fields`, so each field records which cell of which query it came from (see [Evidence and Issuance](evidence.md#the-data-retrieval-chain)). An agent that queries data can also be given the analysis tools: breaking down changes and judging anomalies and trends is computed by the platform with fixed methods and thresholds, the system writes the verdict phrases, and reports cite them (see [Analysis tools for agents](analysis.md#analysis-tools-for-agents)).
|
||||
|
||||
### Guardrails
|
||||
|
||||
@@ -186,7 +186,7 @@ On top of the governance check, the governed level adds evidence gates. They loo
|
||||
|
||||
- **G1** At an exit with a contract, the text in output fields may come only from report-writer nodes.
|
||||
- **G2** Model-written text doesn't bypass the report-writer node to flow straight into the exit.
|
||||
- **G3** The report-writer node sets `numbers: strict`, `on_violation: fail`, and `claims: require_citation` (or `judge`, with a budget).
|
||||
- **G3** The report-writer node sets `numbers: strict`, `on_violation: fail`, `claims: require_citation` (or `judge`, with a budget), and `analysis_claims: strict` (the analysis claim check set to strict; see [Analysis claim check](analysis.md#analysis-claim-check)).
|
||||
- **G4** Caliber card inputs don't come from sandbox code in the compute role.
|
||||
- **G5** Agents that feed a caliber card have `output_schema` set and `cite_fields` turned on.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user