mirror of
https://github.com/ThearchyHelios/AgentLab.git
synced 2026-11-15 00:21:58 +00:00
文档:新增 analysis 一篇(中英各一,标题层级一致)——分析能力是什么、为什么(判定由系统算、判定语由系统写、照实说、留痕)、五种分析各回答什么问题与判定标准(括号里是缺省阈值)、算不了的情形与调用写错、期次的写法、两个入口(Agent 的分析工具、分析卡)、引用写法、分析说法核对(判断词的类、check / strict / off、来源与因果的分界)、分析范围、设置、限制。更新:证据与出具(按引用核对的血缘口子与分析说法复核、口径卡指标的两期可比性、引用标记加 a: 与 v:A / table:A、点开一个分析结论看到什么、两种导出认分析结论、运行对比按分析卡对齐、复核套件加分析说法核对)、工作流(内置模板 11 个、节点类型表加分析卡、G3 加分析说法核对、agent 可绑分析工具)、问数据(分析类问题怎么走、复核只写说明的几种缺口、分析结论的数不标口径、自查提醒)、回归评测与口径账本(贡献、判定两种期望与评分,细节待评测包交付后校对;分析结论的数不比对口径)、配置(偏好设置里的分析一节,17 项的叫法、缺省、范围)、工具(内置工具 23 个、分析工具一节;原来写的 15 个早已对不上)、升级说明(本轮);README 中英加一条主要功能、文档索引加 analysis,节点 19 种、模板 11 个。development.md 的检查项数没动,等前端包的数
This commit is contained in:
@@ -16,10 +16,12 @@ Each question in a test suite is a question plus an expected answer. The expecta
|
||||
- A metric of a published caliber.
|
||||
- A query identified from an Ask Data run.
|
||||
- "No data", or a clarifying question back.
|
||||
- The main source of a change (contribution): an expected SQL statement plus change decomposition parameters; the main source (one segment or the top few) and the direction are computed with the same analyses the answer uses.
|
||||
- A verdict: an expected SQL statement plus parameters for a fluctuation baseline, trend test, difference test, or comparability check; the class of verdict is computed with the same analyses: usual fluctuation or unusual; rising, falling, or no clear trend; a difference that holds up or not enough to conclude; comparable or not comparable (see [Analysis](analysis.md#the-five-analyses)).
|
||||
|
||||
Rules:
|
||||
|
||||
- Numeric values may not be typed in by hand.
|
||||
- Numeric values may not be typed in by hand; nor may the segments, directions, and verdicts in contribution and verdict expectations, which are computed from the expected SQL with the same analyses, once before and once after the answer.
|
||||
- Text key points are judged by the judge, listed separately, and not counted toward accuracy.
|
||||
- After a caliber version bump, questions that use it show a notice at the top. Each question takes one of three options: switch to pinning the new version, compute one expectation for each of the old and new versions, or archive the old question and create a new one. Until then, the question is skipped in evaluations.
|
||||
- Test suites can be imported and exported.
|
||||
@@ -42,7 +44,9 @@ Rules:
|
||||
- **Comparison**: against the previous evaluation or a baseline, with per-question transitions and attribution.
|
||||
- **A/B paired table**: exact McNemar test; display only, not a gate.
|
||||
|
||||
Evaluation runs are left out of usage statistics. The scoring rules are at version 2; old evaluations can be re-scored under the new rules from the evaluation details (reading only the sealed records, without calling models).
|
||||
Contribution and verdict expectations are scored like this: an answer that cites an analysis conclusion whose verdict matches is correct and traceable; body text that states it correctly by the lexicon without citing it is correct but uncited; a source or verdict that doesn't match, or the opposite answer, is wrong; an expectation that itself can't be computed (the expected SQL returns too little history, say) can't be scored. The input queries of analysis conclusions also count as queries the answer cited.
|
||||
|
||||
Evaluation runs are left out of usage statistics. The scoring rules are at version 3 (which adds contribution and verdict expectations); old evaluations can be re-scored under the new rules from the evaluation details (reading only the sealed records, without calling models).
|
||||
|
||||
### Scheduled evaluations
|
||||
|
||||
@@ -68,6 +72,7 @@ After an exploratory run is sealed, the system compares the answer with publishe
|
||||
- Judging is number by number: when a report cites several columns of one query (for example sales, order count, first and last day), each number is compared on its own column against the caliber's metric column, and extra queried columns don't affect it. This applies only when removing the other columns wouldn't change the number; queries with DISTINCT, truncation after sorting by another column, or extra grouping columns are still compared as a whole.
|
||||
- When a multi-row query has no `ORDER BY`, citing one of its cells doesn't count as "from a caliber", and recall and reconciliation mark it "Row order undefined".
|
||||
- An answer is issued automatically only when every number in it that comes from data is proven to come from the same caliber.
|
||||
- Numbers in analysis conclusions (shares, changes, fluctuation ranges) are derived from query results, so they aren't compared or labeled with a caliber and aren't counted in the summary line.
|
||||
|
||||
When writing a caliber's data-fetch SQL, it is best not to round (`ROUND(SUM(x), 2)`) or wrap values in `COALESCE(…, 0)`. Leave rounding and defaults to the caliber card's decimal places and display settings; otherwise Ask Data queries written without rounding will never be recognized as equivalent (and where they differ is stated plainly).
|
||||
|
||||
|
||||
Reference in New Issue
Block a user