ThearchyHelios
2371e75f17
评测执行的主干:后台评测、无头作答、评分、统计、对比与 A/B、清理(下一轮第 3 波 W3-c 第一段)
为什么:第 2 波只有题集和期望答案的来源,评测本身跑不起来。这一段把细案 01 做法 9–12、16、17 的后端接上,
量的是用户在问数据页拿到的同一轮,而不是另抄一份编排。
- eval/answer.py:无头作答,串起 answer_ops → chat/turn 的决定 → launch_run(origin=eval,带目录视图、只读、遮罩)
→ 等终态(停在审批的取消、记「评测不代批」;驳回记路径「被驳回」)→ review_of_run → 按同一规则重来一次;
工作流题按钉住版本的图跑探索运行(不发正式运行,免得被 pick_base 和应用视图取到)。用量按角色记
- eval/runner.py:用 core/job_runtime,一次评测一个任务,题 × 重复 × 臂一项;A/B 几臂共用期望、同一题同时开跑;
开评拍目录快照、记上传表格版本和表结构哈希的前后;并发不超过 max_parallel 和全局并发的四分之一;每题时限;
评测单独一本账(开跑前估算、每题预留后找补、触顶不再开新题);止损;能停、只重试没做完的、重启标中断不续跑
- eval/score.py:六种期望、五种结论(量纲不同单列)、维度配对与弱匹配、路径不对直接不通过、口径 dual 两版都报;
逐题 E1 用口径账本的 compare_queries(总方案 X2,不另写 signals);SCORER_VERSION 可重评(eval/rescore.py,只读封存)
- eval/judging.py:结论裁判走按需裁判同一条路(evidence._judge_now),文字要点由裁判判、单列不计准确率;花费经
judge.charged_to 记在评测那本账上(engine/judge.py 只加了一个上下文变量和取账本的一处)
- eval/stats.py、eval/compare.py:准确率与可追溯准确率 + Wilson 区间、各项分布、P50/P90、按角色成本、引用率、逐题 E1;
选基准、逐题转移与归因、指纹差异、A/B 配对表与精确 McNemar;「目录贡献」预设对照(当前 vs 关,可加只用已核实)
- eval/cleanup.py + core/artifact_store.forget:保留最近 N 次 + 钉住的基线 + A/B 另几臂,删结果、明细工件、底下的运行和
引用行,工件文件一条引用都不剩才删;总表永久保留
- engine/tolerance.py:双盲复算的 _agree / _delta_pct 原样搬来共用(dual_check 改 import,行为不变)
- 设置加 eval 一组(builder 读、admin 写)和 GET /api/settings/eval/spend;题集预算在全站缺省上覆盖,每日上限只能往下收
- api/evals.py:发起(单臂 / A/B / 目录贡献预设)、估算、进度、逐题、停止、重试、重评、钉基线、和上一次比、配对表、清理
- main.py 的 lifespan:影子起草和评测的启动整理、关停(评测先于运行停,停时取消自己发起的运行)
- 测试:test_eval_e2e(dual_check_shop + 剧本模型:按支付日 330 通过且 SQL 等价,按下单日 310 不通过、差在 time;
目录对照 1:0;和上一次比找到退步、归因做法变);通知事件、定时接口形状的老测试按新事件和新字段更新
2026-10-16 16:38:09 +08:00
..
2026-10-16 13:48:09 +08:00
2026-10-16 15:38:11 +08:00
2026-10-15 12:09:29 +08:00
2026-10-16 12:46:48 +08:00
2026-10-04 23:40:20 +08:00
2026-10-16 16:38:09 +08:00
2026-10-16 14:23:41 +08:00
2026-10-06 11:44:58 +08:00
2026-10-16 13:48:09 +08:00
2026-10-16 14:15:51 +08:00
2026-10-16 14:15:51 +08:00
2026-10-14 21:55:51 +08:00
2026-09-30 08:05:15 +08:00
2026-09-30 08:05:15 +08:00
2026-09-30 08:05:15 +08:00
2026-09-25 15:20:46 +08:00
2026-09-30 08:05:15 +08:00
2026-10-14 21:55:51 +08:00
2026-10-14 21:55:51 +08:00
2026-10-15 12:09:29 +08:00
2026-10-14 21:55:51 +08:00
2026-10-15 07:15:37 +08:00
2026-10-16 10:14:27 +08:00
2026-10-15 07:15:37 +08:00
2026-10-14 21:55:51 +08:00
2026-10-15 08:35:28 +08:00
2026-10-15 12:09:29 +08:00
2026-10-15 13:55:19 +08:00
2026-10-15 08:35:28 +08:00
2026-09-29 05:26:34 +08:00
2026-10-04 12:03:14 +08:00
2026-10-16 10:14:27 +08:00
2026-10-15 08:35:28 +08:00
2026-10-15 07:15:37 +08:00
2026-10-14 20:05:23 +08:00
2026-09-30 08:05:15 +08:00
2026-09-30 08:05:15 +08:00
2026-09-30 08:05:15 +08:00
2026-10-15 14:32:25 +08:00
2026-10-15 13:55:19 +08:00
2026-10-15 17:31:20 +08:00
2026-10-15 13:55:19 +08:00
2026-10-15 13:55:19 +08:00
2026-10-16 10:34:18 +08:00
2026-10-16 10:14:27 +08:00
2026-09-30 09:48:16 +08:00
2026-10-16 12:46:48 +08:00
2026-10-16 12:46:48 +08:00
2026-10-12 17:23:27 +08:00
2026-10-12 19:09:50 +08:00
2026-10-06 19:21:23 +08:00
2026-10-12 19:09:50 +08:00
2026-10-16 12:46:48 +08:00
2026-10-12 23:01:09 +08:00
2026-09-17 18:29:16 +08:00
2026-10-14 22:08:33 +08:00
2026-10-15 14:44:26 +08:00
2026-10-14 21:55:51 +08:00
2026-10-16 14:17:06 +08:00
2026-10-16 14:17:06 +08:00
2026-10-14 09:59:19 +08:00
2026-10-16 10:25:28 +08:00
2026-10-14 14:07:14 +08:00
2026-10-14 22:17:00 +08:00
2026-10-13 20:30:02 +08:00
2026-10-14 20:05:23 +08:00
2026-10-13 20:30:02 +08:00
2026-10-16 10:45:45 +08:00
2026-10-15 19:43:29 +08:00
2026-10-16 10:25:28 +08:00
2026-10-16 14:17:06 +08:00
2026-10-04 01:07:15 +08:00
2026-10-14 16:11:04 +08:00
2026-10-04 02:00:17 +08:00
2026-10-14 09:51:49 +08:00
2026-10-03 11:09:09 +08:00
2026-10-16 14:17:06 +08:00
2026-10-16 12:02:41 +08:00
2026-10-04 02:19:26 +08:00
2026-10-03 12:14:17 +08:00
2026-10-14 16:11:04 +08:00
2026-10-16 10:28:01 +08:00
2026-10-07 01:09:36 +08:00
2026-10-14 20:05:23 +08:00
2026-10-12 15:55:19 +08:00
2026-10-16 10:59:15 +08:00
2026-09-29 11:10:53 +08:00
2026-09-30 08:05:15 +08:00
2026-09-24 23:15:19 +08:00
2026-10-06 16:42:29 +08:00
2026-10-05 16:41:59 +08:00
2026-10-05 16:41:59 +08:00
2026-09-30 08:05:15 +08:00
2026-09-30 08:05:15 +08:00
2026-09-30 08:05:15 +08:00
2026-09-30 08:05:15 +08:00
2026-10-06 19:51:29 +08:00
2026-09-30 10:04:52 +08:00
2026-10-15 13:55:19 +08:00
2026-10-16 12:46:36 +08:00
2026-10-14 21:55:51 +08:00
2026-10-16 10:45:45 +08:00
2026-10-16 10:59:15 +08:00
2026-10-04 11:07:40 +08:00
2026-10-15 13:55:19 +08:00
2026-10-05 15:08:53 +08:00
2026-10-04 11:10:28 +08:00
2026-10-14 20:05:23 +08:00
2026-09-29 11:10:53 +08:00
2026-10-16 10:45:45 +08:00
2026-10-16 10:45:45 +08:00
2026-10-16 10:45:45 +08:00
2026-10-16 10:45:45 +08:00
2026-10-16 10:45:45 +08:00
2026-10-16 10:45:45 +08:00
2026-10-16 10:45:45 +08:00
2026-10-14 20:05:23 +08:00
2026-10-16 10:45:45 +08:00
2026-09-25 15:41:30 +08:00
2026-10-15 14:59:40 +08:00
2026-10-16 10:45:45 +08:00
2026-10-16 12:23:19 +08:00
2026-09-27 10:16:32 +08:00
2026-09-30 08:05:15 +08:00
2026-10-05 00:26:04 +08:00
2026-10-16 14:17:06 +08:00
2026-09-30 08:05:15 +08:00
2026-10-02 07:38:44 +08:00
2026-10-02 14:38:10 +08:00
2026-10-16 10:24:22 +08:00
2026-10-05 15:54:03 +08:00
2026-10-05 15:54:03 +08:00
2026-10-14 15:38:15 +08:00
2026-10-07 01:34:59 +08:00
2026-10-16 10:14:27 +08:00
2026-10-12 12:24:14 +08:00
2026-10-06 18:03:12 +08:00
2026-09-18 09:39:24 +08:00
2026-10-15 07:15:37 +08:00
2026-10-01 13:42:04 +08:00
2026-10-16 13:58:36 +08:00
2026-10-16 16:38:09 +08:00
2026-10-16 14:06:29 +08:00
2026-10-16 10:18:12 +08:00
2026-10-16 14:05:22 +08:00
2026-10-16 14:05:22 +08:00
2026-09-28 11:47:23 +08:00
2026-09-30 08:05:15 +08:00
2026-10-06 12:02:03 +08:00
2026-09-30 08:05:15 +08:00
2026-09-30 08:05:15 +08:00
2026-10-06 12:58:37 +08:00
2026-10-16 11:04:41 +08:00
2026-09-29 05:26:34 +08:00
2026-09-29 05:26:34 +08:00
2026-09-30 08:05:15 +08:00
2026-10-12 13:50:31 +08:00
2026-10-12 17:23:27 +08:00
2026-10-12 13:50:31 +08:00
2026-10-12 17:23:27 +08:00
2026-10-12 10:48:19 +08:00
2026-10-04 02:35:33 +08:00
2026-09-30 08:05:15 +08:00
2026-09-30 08:05:15 +08:00
2026-10-12 12:24:14 +08:00
2026-09-30 15:03:31 +08:00
2026-10-01 13:42:04 +08:00
2026-09-30 08:05:15 +08:00
2026-10-06 11:37:22 +08:00
2026-10-01 13:42:04 +08:00
2026-09-30 08:05:15 +08:00
2026-10-04 11:02:06 +08:00
2026-09-25 14:33:56 +08:00
2026-10-15 13:55:19 +08:00
2026-10-02 07:38:44 +08:00
2026-10-15 13:55:19 +08:00
2026-09-30 08:05:15 +08:00
2026-09-30 10:04:52 +08:00
2026-09-30 08:05:15 +08:00
2026-09-27 10:16:32 +08:00
2026-09-30 08:05:15 +08:00
2026-09-27 10:16:32 +08:00
2026-10-12 15:55:19 +08:00
2026-10-07 01:09:36 +08:00
2026-10-06 22:52:28 +08:00
2026-10-12 15:55:19 +08:00
2026-10-05 01:18:03 +08:00
2026-10-16 10:33:41 +08:00
2026-10-01 13:42:04 +08:00
2026-10-03 09:30:50 +08:00
2026-10-16 15:36:21 +08:00
2026-09-28 11:47:23 +08:00
2026-09-28 11:47:23 +08:00
2026-10-05 18:45:59 +08:00
2026-09-25 15:30:32 +08:00
2026-10-16 09:59:44 +08:00
2026-09-29 16:07:19 +08:00
2026-10-12 13:50:31 +08:00
2026-10-02 14:38:10 +08:00
2026-09-30 10:04:52 +08:00
2026-09-30 10:04:52 +08:00
2026-09-30 10:04:52 +08:00
2026-10-06 12:58:37 +08:00
2026-10-06 14:30:15 +08:00
2026-09-27 10:16:32 +08:00
2026-09-28 11:47:45 +08:00
2026-09-25 12:47:07 +08:00
2026-10-16 14:23:41 +08:00
2026-10-16 13:49:53 +08:00
2026-10-16 14:23:41 +08:00
2026-09-30 08:05:15 +08:00
2026-10-14 20:05:23 +08:00
2026-10-16 14:26:04 +08:00
2026-09-30 08:05:15 +08:00
2026-09-30 08:05:15 +08:00
2026-10-12 10:48:19 +08:00
2026-10-12 10:48:19 +08:00
2026-10-07 01:09:36 +08:00
2026-10-12 13:14:36 +08:00
2026-10-06 19:11:14 +08:00
2026-10-06 21:50:48 +08:00
2026-10-12 10:50:40 +08:00
2026-10-13 16:59:58 +08:00
2026-10-12 12:18:34 +08:00
2026-10-07 01:09:36 +08:00
2026-10-14 20:05:23 +08:00
2026-10-04 01:08:42 +08:00
2026-10-04 02:17:32 +08:00
2026-10-03 09:36:20 +08:00
2026-10-03 10:40:47 +08:00
2026-09-28 11:47:23 +08:00
2026-10-15 07:15:37 +08:00
2026-10-05 03:26:43 +08:00
2026-10-15 07:15:37 +08:00
2026-09-30 11:19:25 +08:00
2026-10-06 14:29:48 +08:00
2026-09-30 08:05:15 +08:00
2026-10-01 13:42:04 +08:00
2026-09-25 23:14:50 +08:00
2026-09-30 08:05:15 +08:00
2026-10-14 20:05:23 +08:00
2026-10-16 16:38:09 +08:00
2026-10-16 10:14:27 +08:00
2026-10-04 12:27:38 +08:00
2026-10-02 07:38:44 +08:00
2026-10-16 10:18:12 +08:00
2026-10-15 12:09:29 +08:00
2026-10-15 07:15:37 +08:00
2026-10-16 10:14:27 +08:00
2026-10-14 22:08:33 +08:00
2026-10-16 10:14:27 +08:00
2026-10-16 12:02:41 +08:00
2026-10-16 12:06:24 +08:00
2026-10-16 14:26:04 +08:00
2026-10-02 14:38:10 +08:00
2026-10-02 14:38:10 +08:00
2026-10-03 09:36:20 +08:00
2026-10-02 14:38:10 +08:00
2026-10-12 10:17:40 +08:00
2026-10-02 14:38:10 +08:00
2026-10-06 19:43:44 +08:00
2026-10-06 19:43:44 +08:00
2026-10-06 17:29:18 +08:00
2026-10-06 19:43:44 +08:00
2026-10-06 19:43:44 +08:00
2026-10-16 10:18:12 +08:00
2026-09-30 08:05:15 +08:00
2026-10-16 10:34:18 +08:00
2026-10-04 02:52:43 +08:00
2026-09-28 21:20:04 +08:00
2026-09-30 08:05:15 +08:00
2026-09-25 14:44:56 +08:00
2026-10-02 07:38:44 +08:00
2026-10-02 07:38:44 +08:00
2026-10-02 07:38:44 +08:00
2026-10-01 22:00:07 +08:00
2026-10-01 22:00:07 +08:00
2026-10-02 07:38:44 +08:00
2026-10-02 07:38:44 +08:00
2026-10-02 07:38:44 +08:00
2026-10-16 11:16:33 +08:00
2026-10-02 07:38:44 +08:00
2026-10-01 22:00:07 +08:00
2026-10-02 07:38:44 +08:00
2026-10-02 14:38:10 +08:00
2026-10-02 07:38:44 +08:00
2026-10-16 11:16:33 +08:00
2026-10-16 11:16:33 +08:00
2026-10-02 07:38:44 +08:00
2026-10-02 07:38:44 +08:00
2026-10-16 12:23:19 +08:00
2026-10-01 22:00:07 +08:00
2026-10-16 11:09:39 +08:00
2026-10-02 07:38:44 +08:00
2026-10-16 11:09:39 +08:00
2026-10-16 12:23:19 +08:00
2026-10-02 07:38:44 +08:00
2026-10-16 11:09:39 +08:00
2026-10-16 11:09:39 +08:00
2026-10-16 11:09:39 +08:00
2026-10-16 11:09:39 +08:00
2026-10-05 18:50:34 +08:00
2026-10-06 09:20:09 +08:00
2026-10-16 10:14:27 +08:00
2026-10-14 20:05:23 +08:00
2026-10-04 11:28:23 +08:00
2026-10-04 12:28:07 +08:00
2026-09-30 08:05:15 +08:00
2026-09-30 08:05:15 +08:00
2026-10-04 11:33:09 +08:00
2026-10-12 14:26:43 +08:00
2026-10-13 10:04:30 +08:00
2026-10-10 12:59:21 +08:00
2026-10-06 13:32:38 +08:00
2026-09-30 10:04:52 +08:00
2026-10-12 13:50:31 +08:00
2026-10-12 23:01:09 +08:00
2026-10-12 12:33:32 +08:00
2026-10-12 23:01:09 +08:00
2026-10-12 23:01:09 +08:00
2026-10-12 19:09:50 +08:00
2026-10-12 17:23:27 +08:00
2026-10-12 13:50:31 +08:00
2026-10-12 17:23:27 +08:00
2026-09-30 15:03:31 +08:00
2026-09-30 08:05:15 +08:00
2026-10-04 02:24:49 +08:00
2026-10-04 02:33:15 +08:00
2026-09-30 08:05:15 +08:00
2026-10-06 09:31:00 +08:00
2026-09-30 08:05:15 +08:00
2026-10-05 01:18:03 +08:00
2026-10-05 18:45:59 +08:00
2026-10-06 14:32:30 +08:00
2026-09-26 23:11:31 +08:00
2026-10-14 22:08:33 +08:00
2026-10-05 00:07:07 +08:00
2026-10-16 14:02:55 +08:00
2026-09-30 08:05:15 +08:00
2026-10-16 12:46:35 +08:00
2026-09-30 08:05:15 +08:00
2026-09-30 08:05:15 +08:00
2026-10-14 20:05:23 +08:00
2026-09-27 10:16:32 +08:00
2026-10-14 20:05:23 +08:00
2026-10-14 21:55:51 +08:00
2026-10-04 01:52:24 +08:00
2026-09-27 23:37:32 +08:00
2026-10-07 00:02:15 +08:00
2026-10-16 16:38:09 +08:00
2026-10-06 08:48:56 +08:00
2026-10-06 16:42:29 +08:00
2026-10-05 08:50:57 +08:00
2026-10-05 08:37:57 +08:00
2026-10-15 07:15:37 +08:00
2026-10-05 18:39:10 +08:00
2026-10-06 08:48:56 +08:00
2026-10-16 10:14:27 +08:00
2026-10-16 16:38:09 +08:00
2026-09-30 08:05:15 +08:00
2026-10-15 12:09:29 +08:00
2026-10-05 08:37:57 +08:00
2026-09-25 16:05:14 +08:00
2026-10-16 10:54:30 +08:00
2026-10-02 07:38:44 +08:00
2026-10-16 11:04:41 +08:00
2026-10-16 13:39:15 +08:00
2026-10-16 13:39:15 +08:00
2026-09-29 05:26:34 +08:00
2026-09-25 14:44:56 +08:00
2026-10-07 01:34:59 +08:00
2026-10-07 01:34:59 +08:00
2026-10-16 12:53:07 +08:00
2026-10-01 13:42:04 +08:00
2026-09-30 08:05:15 +08:00
2026-09-30 08:05:15 +08:00
2026-10-02 07:38:44 +08:00
2026-10-02 07:38:44 +08:00
2026-10-02 07:38:44 +08:00
2026-10-02 07:38:44 +08:00
2026-10-02 07:38:44 +08:00
2026-10-01 22:00:07 +08:00
2026-10-14 21:55:51 +08:00
2026-09-30 08:05:15 +08:00
2026-09-26 23:11:31 +08:00
2026-10-15 07:15:37 +08:00
2026-09-28 21:20:04 +08:00
2026-09-18 17:57:20 +08:00
2026-10-07 00:02:15 +08:00
2026-10-14 20:05:23 +08:00
2026-10-15 07:15:37 +08:00
2026-10-14 21:55:51 +08:00
2026-09-30 08:05:15 +08:00
2026-09-30 08:05:15 +08:00
2026-10-03 11:09:09 +08:00
2026-09-30 08:05:15 +08:00
2026-09-30 08:05:15 +08:00
2026-10-16 10:45:45 +08:00
2026-10-15 13:55:19 +08:00
2026-10-16 14:26:04 +08:00
2026-09-27 10:16:32 +08:00
2026-09-30 08:05:15 +08:00
2026-09-30 08:05:15 +08:00
2026-10-06 21:50:48 +08:00
2026-10-05 01:46:55 +08:00
2026-10-06 14:32:30 +08:00
2026-10-05 18:45:44 +08:00
2026-10-12 12:24:14 +08:00
2026-10-07 00:02:15 +08:00
2026-10-14 22:17:00 +08:00
2026-10-07 01:19:26 +08:00
2026-10-05 00:06:03 +08:00
2026-10-06 14:30:03 +08:00
2026-09-29 05:26:34 +08:00
2026-09-30 09:48:16 +08:00
2026-10-16 10:17:37 +08:00
2026-10-15 07:15:37 +08:00
2026-10-12 17:23:27 +08:00
2026-10-05 06:49:33 +08:00
2026-09-26 23:11:31 +08:00
2026-10-14 21:55:51 +08:00
2026-09-28 02:42:40 +08:00
2026-10-06 23:56:31 +08:00
2026-10-01 22:00:07 +08:00
2026-10-01 13:42:04 +08:00