mirror of
https://github.com/ThearchyHelios/AgentLab.git
synced 2026-11-15 00:01:58 +00:00
文档:回归评测「发起评测」的变体补「分析工具」开 / 关(缺省开)和「分析工具贡献」预设(开 vs 关);关的那一臂助手建图看不到分析工具和分析卡、工作流题不受影响、开关不同算不同变体、对比写「分析工具:开 → 关」。「看结果」里可比性那句改成:期望不可比时,正文明确说了两期不能直接比的也算对但没引用,期望可比的只能靠引用。升级说明的评测那一条补上这个开关和预设(开着的和升级前的评测算同一变体)。中英两边标题层级不变,check-docs 0 问题
This commit is contained in:
@@ -31,11 +31,12 @@ Rules:
|
||||
|
||||
### Run an evaluation
|
||||
|
||||
1. Choose the variants: Copilot model, writing model, judge on or off, catalog (current / snapshot / verified only / off), and how many repetitions.
|
||||
2. Check the estimated cost, then start. The Catalog contribution preset runs "current vs. off" in one go.
|
||||
1. Choose the variants: Copilot model, writing model, judge on or off, catalog (current / snapshot / verified only / off), analysis tools (on / off, on by default), and how many repetitions.
|
||||
2. Check the estimated cost, then start. The Catalog contribution preset runs "current vs. off" in one go, and the Analysis tools contribution preset runs "analysis tools on vs. off".
|
||||
|
||||
- Evaluations run in the background, and each question goes exactly the way a user's question would in Ask Data (graph building, run, review). Questions that stop at an approval aren't approved on anyone's behalf and are recorded as they are.
|
||||
- When Copilot returns only thinking and no operations, the question is retried once automatically.
|
||||
- In an arm with analysis tools off, Copilot doesn't see the analysis tools or the analysis card when building the graph, and a graph that still binds analysis tools is sent back to be fixed; workflow questions (which test a pinned version of a published workflow) build no graph and aren't affected. Analysis tools on and off count as different variants, and a comparison with the previous evaluation shows "Analysis tools: on → off".
|
||||
- Concurrency, per-question time limits, and per-evaluation and daily spending all have caps, set under Settings → Preferences → Evaluation. Evaluation spending is tallied separately.
|
||||
- A question's time limit doesn't include time spent queuing. Under full load, evaluations queue behind Ask Data (see [Run concurrency](postgresql-and-scaling.md#run-concurrency)).
|
||||
- An evaluation can be stopped, and you can retry only the unfinished questions. A service restart marks it interrupted, and it doesn't resume by itself.
|
||||
@@ -50,7 +51,7 @@ Rules:
|
||||
Contribution and verdict expectations are scored like this:
|
||||
|
||||
- **Correct and traceable**: the answer cites an analysis conclusion whose verdict matches (the same analysis; for a main source, it names the expected segments with the same direction). If the sentence citing it says the opposite, it is wrong.
|
||||
- **Correct but uncited**: no matching conclusion is cited, but the body states it correctly by the lexicon (the same judgment words and binding rules; see [Analysis claim check](analysis.md#analysis-claim-check)). Comparability has no judgment words, so only a citation counts.
|
||||
- **Correct but uncited**: no matching conclusion is cited, but the body states it correctly by the lexicon (the same judgment words and binding rules; see [Analysis claim check](analysis.md#analysis-claim-check)). Comparability isn't in the lexicon: when the expectation is not comparable, a body that plainly says the two periods can't be compared directly (the current period is incomplete or not over yet, the numbers of days differ, the calibers differ, and so on; statements that are hypothetical, questions, or negated don't count) also falls here; when the expectation is comparable, only a citation counts.
|
||||
- **Wrong**: the cited conclusion's verdict doesn't match, the body says the opposite (calling a usual fluctuation unusual, rising falling, or naming another segment as the main source), the analysis was done but not stated, or it wasn't done at all.
|
||||
- **Ungradable**: the expectation can't be computed.
|
||||
|
||||
|
||||
@@ -18,7 +18,7 @@ This page lists, newest first, the changes in each update that affect users and
|
||||
- The report export and the verifiable report include analysis conclusions and their input snapshots, recomputed from the sealed inputs for verification.
|
||||
- Settings gain a Preferences → Analysis section (see [Analysis settings](configuration.md#analysis-settings)). It is snapshotted into a run only when the graph has an analysis card or an agent with analysis tools, so the start records of other runs don't change at all.
|
||||
- Ask Data: Copilot gains a graph shape for analytical questions, and its self-check reminds it to give the querying agent the analysis tools (one round only). The review merges several analyses that couldn't be done into one "n analyses couldn't be done" note, without an automatic redo; analysis calls that were wrong but later fixed no longer count as a gap, and ones never fixed no longer trigger a redo of the whole turn.
|
||||
- Evaluation gains contribution and verdict expectations (computed from the expected SQL with the same analyses, never typed in by hand), and the scoring rules move to version 3, so old evaluations can be re-scored under the new rules; Copilot's prompt fingerprint now includes the analysis method, so comparisons with evaluations from before the upgrade flag "Prompt changed" (see [Regression evaluation](evaluation-and-ledger.md#regression-evaluation)).
|
||||
- Evaluation gains contribution and verdict expectations (computed from the expected SQL with the same analyses, never typed in by hand), and the scoring rules move to version 3, so old evaluations can be re-scored under the new rules; Copilot's prompt fingerprint now includes the analysis method, so comparisons with evaluations from before the upgrade flag "Prompt changed"; variants gain an analysis tools switch (on by default) and an Analysis tools contribution preset, and a variant with analysis tools on counts as the same variant as evaluations from before the upgrade (see [Regression evaluation](evaluation-and-ledger.md#regression-evaluation)).
|
||||
|
||||
## 2026-10, orchestration improvements wave 2: PostgreSQL and multiple processes
|
||||
|
||||
|
||||
@@ -31,11 +31,12 @@
|
||||
|
||||
### 发起评测
|
||||
|
||||
1. 选变体:助手模型、写作模型、裁判开关、目录(当前 / 快照 / 只用已核实的 / 关)、重复几次。
|
||||
2. 看预估花费,确认发起。「目录贡献」预设一次跑「当前 vs 关」。
|
||||
1. 选变体:助手模型、写作模型、裁判开关、目录(当前 / 快照 / 只用已核实的 / 关)、分析工具(开 / 关,缺省开)、重复几次。
|
||||
2. 看预估花费,确认发起。「目录贡献」预设一次跑「当前 vs 关」,「分析工具贡献」预设一次跑「分析工具开 vs 关」。
|
||||
|
||||
- 评测在后台跑,每题怎么走和用户在问数据里拿到的一样(建图、运行、复核)。停在审批上的不代批,照实记。
|
||||
- 助手只给了思考、一个操作都没给时,自动重来一次。
|
||||
- 分析工具关掉的那一臂,助手建图时看不到分析工具和分析卡,图里还绑着分析工具的会被挡回去改;工作流题(考一个已发布工作流的固定版本)不建图,不受这一项影响。分析工具开、关不同算不同的变体,和上一次比时写「分析工具:开 → 关」。
|
||||
- 并发、每题时限、单次和每日金额都有上限,在「设置 → 偏好设置 → 评测」里改。评测的花费单独记账。
|
||||
- 每题的时限不算排队的时间。满载时评测会排在问数据后面(见[运行并发](postgresql-and-scaling.md#运行并发))。
|
||||
- 能停,能只重试没做完的题。服务重启时标成中断,不会自己续跑。
|
||||
@@ -50,7 +51,7 @@
|
||||
贡献、判定两种期望这样评:
|
||||
|
||||
- **对且可追溯**:答案引用了判定对得上的分析结论(同一种分析,主要来源点名的是期望的那几段、方向一致);引用它的那句话却写反了的算错。
|
||||
- **对但没引用**:没引用对得上的结论,但正文按说法表写对了(同一套判断词和绑定规则,见[分析说法核对](analysis.md#分析说法核对))。可比性没有对应的判断词,只能靠引用。
|
||||
- **对但没引用**:没引用对得上的结论,但正文按说法表写对了(同一套判断词和绑定规则,见[分析说法核对](analysis.md#分析说法核对))。可比性不进说法表:期望是不可比的,正文明确说了两期不能直接比(本期不完整、还没过完、天数不同、口径不同……,带假设、疑问或否定的不算)也算这一种;期望是可比的只能靠引用。
|
||||
- **错**:引用的结论判定对不上、正文写反了(把常规波动说成异常、把上升说成下降、主要来源说成别的段)、做了分析没说,或者根本没做。
|
||||
- **不能评分**:期望算不了。
|
||||
|
||||
|
||||
@@ -18,7 +18,7 @@
|
||||
- 报告导出、可验证报告带上分析结论和它的输入快照,按封存的输入复算核对。
|
||||
- 设置多了「偏好设置 → 分析」一节(见[分析设置](configuration.md#分析设置))。只在图里有分析卡、或者 agent 绑了分析工具时,发起运行才把这一组快照进运行,别的运行的发起记录一字不变。
|
||||
- 问数据:助手建图多了分析类问题的形状,自查时提醒查库的 agent 绑分析工具(只提醒一轮)。复核把几次没做成的分析并成一条「有 n 次分析没做成」,只写说明、不自动重来;分析调用写错了、后来改对的不再记成缺口,一直没改对的也不再触发整轮重来。
|
||||
- 评测多了贡献、判定两种期望(由期望 SQL 和同一套分析算出,不许人填),评分规则升到第 3 版,老评测可以按新规则重评;助手提示词的指纹包含了分析方法,和升级前的评测比会标「提示词变了」(见[回归评测](evaluation-and-ledger.md#回归评测))。
|
||||
- 评测多了贡献、判定两种期望(由期望 SQL 和同一套分析算出,不许人填),评分规则升到第 3 版,老评测可以按新规则重评;助手提示词的指纹包含了分析方法,和升级前的评测比会标「提示词变了」;变体多了「分析工具」开关(缺省开)和「分析工具贡献」预设对照,开着的和升级前的评测算同一变体(见[回归评测](evaluation-and-ledger.md#回归评测))。
|
||||
|
||||
## 2026-10 编排完善第 2 波:PostgreSQL 与多进程
|
||||
|
||||
|
||||
Reference in New Issue
Block a user