mirror of
https://github.com/ThearchyHelios/AgentLab.git
synced 2026-11-17 19:22:02 +00:00
2371e75f17
为什么:第 2 波只有题集和期望答案的来源,评测本身跑不起来。这一段把细案 01 做法 9–12、16、17 的后端接上, 量的是用户在问数据页拿到的同一轮,而不是另抄一份编排。 - eval/answer.py:无头作答,串起 answer_ops → chat/turn 的决定 → launch_run(origin=eval,带目录视图、只读、遮罩) → 等终态(停在审批的取消、记「评测不代批」;驳回记路径「被驳回」)→ review_of_run → 按同一规则重来一次; 工作流题按钉住版本的图跑探索运行(不发正式运行,免得被 pick_base 和应用视图取到)。用量按角色记 - eval/runner.py:用 core/job_runtime,一次评测一个任务,题 × 重复 × 臂一项;A/B 几臂共用期望、同一题同时开跑; 开评拍目录快照、记上传表格版本和表结构哈希的前后;并发不超过 max_parallel 和全局并发的四分之一;每题时限; 评测单独一本账(开跑前估算、每题预留后找补、触顶不再开新题);止损;能停、只重试没做完的、重启标中断不续跑 - eval/score.py:六种期望、五种结论(量纲不同单列)、维度配对与弱匹配、路径不对直接不通过、口径 dual 两版都报; 逐题 E1 用口径账本的 compare_queries(总方案 X2,不另写 signals);SCORER_VERSION 可重评(eval/rescore.py,只读封存) - eval/judging.py:结论裁判走按需裁判同一条路(evidence._judge_now),文字要点由裁判判、单列不计准确率;花费经 judge.charged_to 记在评测那本账上(engine/judge.py 只加了一个上下文变量和取账本的一处) - eval/stats.py、eval/compare.py:准确率与可追溯准确率 + Wilson 区间、各项分布、P50/P90、按角色成本、引用率、逐题 E1; 选基准、逐题转移与归因、指纹差异、A/B 配对表与精确 McNemar;「目录贡献」预设对照(当前 vs 关,可加只用已核实) - eval/cleanup.py + core/artifact_store.forget:保留最近 N 次 + 钉住的基线 + A/B 另几臂,删结果、明细工件、底下的运行和 引用行,工件文件一条引用都不剩才删;总表永久保留 - engine/tolerance.py:双盲复算的 _agree / _delta_pct 原样搬来共用(dual_check 改 import,行为不变) - 设置加 eval 一组(builder 读、admin 写)和 GET /api/settings/eval/spend;题集预算在全站缺省上覆盖,每日上限只能往下收 - api/evals.py:发起(单臂 / A/B / 目录贡献预设)、估算、进度、逐题、停止、重试、重评、钉基线、和上一次比、配对表、清理 - main.py 的 lifespan:影子起草和评测的启动整理、关停(评测先于运行停,停时取消自己发起的运行) - 测试:test_eval_e2e(dual_check_shop + 剧本模型:按支付日 330 通过且 SQL 等价,按下单日 310 不通过、差在 time; 目录对照 1:0;和上一次比找到退步、归因做法变);通知事件、定时接口形状的老测试按新事件和新字段更新
156 lines
8.9 KiB
Python
156 lines
8.9 KiB
Python
"""评测的汇总与统计(细案 01 做法 11、12;评测-D10)。所有汇总都在后端算好,界面直接显示(界面层不放逻辑:同一个数不能
|
||
有两套算法)。纯计算,输入是一次评测的逐题结果(EvalResult 的列,result_row 的形状)。
|
||
|
||
- 准确率和可追溯准确率:分母是能判的题(通过 + 不通过);不能评分、跳过的单列,既不算对也不算错。都带 Wilson 95% 区间和
|
||
样本数——题集小、模型有随机性,差两三题就可能被读成「变好了」「变差了」(评测-D10:只展示、不卡门)。
|
||
- 作答路径、出具档位(声明了出具的按 formal / degraded / withheld,没声明的单列「探索问答」)、复核结论(verdict × severity)
|
||
和自动重来的次数、复核用时超过问数据页等待(25 秒)的题数。
|
||
- 结论裁判(打开了才有):supported / partial / contradicted / insufficient / 未判的句数;文字要点的判断单列。
|
||
- look:每题平均几次(分模型和系统),用满这一轮上限的比例;主模型(助手)调用次数。
|
||
- 耗时:建图、运行(墙钟、执行)、复核、整题的 P50 / P90。
|
||
- token 和成本按角色:助手 = 建图 + 挑表 + 复核;写作 = 答案运行里的模型调用(按模型分);裁判;期望探针不调模型,为 0。
|
||
- 报告引用率:numbers_cited / numbers,另给 uncited_numbers、unresolved、unknown_entities。
|
||
- 逐题 E1:答案的查询和期望的取数等价的比例、差异方面的分布(score.sql_match,口径账本同一个比对函数)。
|
||
- 重复跑 k 次的题:每题的通过率。
|
||
|
||
McNemar(精确、双侧)给 A/B 的配对表用(compare.py):只看「只 A 对」「只 B 对」的那几题,二项分布 p=0.5 的双侧概率。
|
||
"""
|
||
from __future__ import annotations
|
||
|
||
import math
|
||
from collections import Counter
|
||
from collections.abc import Iterable, Mapping, Sequence
|
||
from typing import Any
|
||
|
||
#: 95% 区间的 z
|
||
Z95 = 1.959963984540054
|
||
#: 问数据页等复核最多这么久(毫秒,同 answer.REVIEW_PAGE_WAIT_S)
|
||
REVIEW_PAGE_WAIT_MS = 25_000
|
||
TIERS = ("formal", "degraded", "withheld", "explore")
|
||
TIMING_KEYS = ("build_ms", "run_wall_ms", "run_active_ms", "review_ms", "total_ms")
|
||
|
||
|
||
def wilson(passed: int, n: int, z: float = Z95) -> tuple[float, float] | None:
|
||
"""Wilson 区间(缺省 95%)。n 为 0 时没有区间。"""
|
||
if n <= 0:
|
||
return None
|
||
p = passed / n
|
||
denom = 1 + z * z / n
|
||
centre = (p + z * z / (2 * n)) / denom
|
||
half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / denom
|
||
return round(max(0.0, centre - half), 4), round(min(1.0, centre + half), 4)
|
||
|
||
|
||
def mcnemar_exact(only_a: int, only_b: int) -> float:
|
||
"""精确 McNemar 双侧 p 值:不一致的那几题里,「只 A 对」服从 Binomial(b + c, 0.5)。没有不一致的题时为 1。"""
|
||
n = only_a + only_b
|
||
if n == 0:
|
||
return 1.0
|
||
k = min(only_a, only_b)
|
||
tail = sum(math.comb(n, i) for i in range(k + 1)) / (2 ** n)
|
||
return round(min(1.0, 2 * tail), 6)
|
||
|
||
|
||
def percentile(values: Sequence[float], q: float) -> float | None:
|
||
"""最近秩法的百分位(P50、P90)。没有值时为 None。"""
|
||
data = sorted(v for v in values if v is not None)
|
||
if not data:
|
||
return None
|
||
rank = max(1, math.ceil(q / 100 * len(data)))
|
||
return data[min(rank, len(data)) - 1]
|
||
|
||
|
||
def rate(part: int, whole: int) -> dict[str, Any]:
|
||
"""{passed, n, rate, ci}:比例和它的 Wilson 区间。"""
|
||
return {"passed": part, "n": whole, "rate": round(part / whole, 4) if whole else None, "ci": wilson(part, whole)}
|
||
|
||
|
||
def _usage_total(rows: Iterable[Mapping[str, Any] | None]) -> dict[str, Any]:
|
||
from app.eval.answer import add_usage
|
||
|
||
out = add_usage(None, None)
|
||
for row in rows:
|
||
out = add_usage(out, row)
|
||
return out
|
||
|
||
|
||
def summarize(results: Sequence[Mapping[str, Any]], *, judge_on: bool = False) -> dict[str, Any]:
|
||
"""一次评测(一臂)的总表。results 是逐题结果(result_row 的形状),只算已经结束的那些。"""
|
||
from app.eval.score import SCORER_VERSION
|
||
|
||
done = [r for r in results if r.get("status") in ("ok", "skipped", "failed", "cancelled", "partial")]
|
||
verdicts = Counter(r.get("verdict") or "none" for r in done)
|
||
graded = [r for r in done if r.get("verdict") in ("pass", "fail")]
|
||
passed = [r for r in graded if r.get("verdict") == "pass"]
|
||
traced = [r for r in passed if r.get("traced")]
|
||
outcomes = Counter(s.get("outcome") for r in done for s in r.get("scores") or []
|
||
if s.get("required") and s.get("kind") != "text")
|
||
text = Counter(s.get("outcome") for r in done for s in r.get("scores") or [] if s.get("kind") == "text")
|
||
answered = [r for r in done if r.get("path")]
|
||
tiers = Counter((r.get("tier") or "explore") for r in answered if r.get("path") == "ran")
|
||
reviews = [r.get("review") or {} for r in answered if r.get("review")]
|
||
looks = [r.get("looks") or {} for r in answered]
|
||
timings = {k: [float((r.get("timings") or {}).get(k)) for r in answered if (r.get("timings") or {}).get(k) is not None]
|
||
for k in TIMING_KEYS}
|
||
usage = {role: _usage_total((r.get("usage") or {}).get(role) for r in done)
|
||
for role in ("assistant", "writer", "judge")}
|
||
cost = round(sum(u["cost_usd"] for u in usage.values()), 6)
|
||
unpriced = sum(int(u.get("unpriced_calls") or 0) for u in usage.values())
|
||
stats = [((r.get("answer_stats") or {})) for r in answered]
|
||
numbers = sum(int(s.get("numbers") or 0) for s in stats)
|
||
cited = sum(int(s.get("numbers_cited") or 0) for s in stats)
|
||
e1 = [r.get("sql_match") for r in done if isinstance(r.get("sql_match"), Mapping)
|
||
and (r["sql_match"].get("status") in ("proven", "related", "none"))]
|
||
aspects = Counter(a for m in e1 if m.get("status") == "related" for a in m.get("aspects") or [])
|
||
claims = Counter()
|
||
for r in done:
|
||
for key, n in (((r.get("judge") or {}).get("claims") or {}).get("counts") or {}).items():
|
||
claims[key] += int(n or 0)
|
||
by_case: dict[str, list[Mapping[str, Any]]] = {}
|
||
for r in graded:
|
||
by_case.setdefault(str(r.get("case_id")), []).append(r)
|
||
repeated = {cid: rate(sum(1 for r in rows if r.get("verdict") == "pass"), len(rows))
|
||
for cid, rows in by_case.items() if len(rows) > 1}
|
||
return {
|
||
"scorer_version": SCORER_VERSION,
|
||
"items": len(results), "done": len(done),
|
||
"cases": len({r.get("case_id") for r in results}),
|
||
"verdicts": {k: verdicts.get(k, 0) for k in ("pass", "fail", "ungradable", "skipped")},
|
||
"accuracy": rate(len(passed), len(graded)),
|
||
"traced_accuracy": rate(len(traced), len(graded)),
|
||
"outcomes": dict(outcomes),
|
||
"text": dict(text),
|
||
"paths": dict(Counter(r.get("path") for r in answered)),
|
||
"tiers": {k: tiers.get(k, 0) for k in TIERS},
|
||
"review": {
|
||
"verdicts": dict(Counter(str(v.get("verdict") or "ok") for v in reviews)),
|
||
"severity": dict(Counter(str(v.get("severity")) for v in reviews if v.get("severity"))),
|
||
"retries": sum(1 for r in answered if int(r.get("attempts") or 0) > 0),
|
||
"slow": sum(1 for r in answered if ((r.get("timings") or {}).get("review_ms") or 0) > REVIEW_PAGE_WAIT_MS),
|
||
},
|
||
"judge": {"on": judge_on, "claims": dict(claims) if judge_on else None},
|
||
"looks": {
|
||
"model": sum(int(x.get("model") or 0) for x in looks),
|
||
"system": sum(int(x.get("system") or 0) for x in looks),
|
||
"per_case": round(sum(int(x.get("model") or 0) + int(x.get("system") or 0) for x in looks) / len(looks), 2)
|
||
if looks else None,
|
||
"limit_hit": rate(sum(1 for x in looks if x.get("limit_hit")), len(looks))["rate"] if looks else None,
|
||
},
|
||
"calls": {"assistant": int(usage["assistant"].get("calls") or 0),
|
||
"per_case": round(int(usage["assistant"].get("calls") or 0) / len(answered), 2) if answered else None},
|
||
"timings": {k: {"p50": percentile(v, 50), "p90": percentile(v, 90), "n": len(v)} for k, v in timings.items()},
|
||
"usage": {**usage, "gold": {"cost_usd": 0.0, "calls": 0}},
|
||
"cost_usd": cost, "unpriced_calls": unpriced,
|
||
"per_case_cost_usd": round(cost / len(done), 6) if done else None,
|
||
"citations": {"numbers": numbers, "numbers_cited": cited,
|
||
"uncited_numbers": sum(int(s.get("uncited_numbers") or 0) for s in stats),
|
||
"unresolved": sum(int(s.get("unresolved") or 0) for s in stats),
|
||
"unknown_entities": sum(int(s.get("unknown_entities") or 0) for s in stats),
|
||
"rate": round(cited / numbers, 4) if numbers else None},
|
||
"e1": {**rate(sum(1 for m in e1 if m.get("status") == "proven"), len(e1)), "aspects": dict(aspects)},
|
||
"pass_rates": repeated,
|
||
}
|
||
|
||
|
||
__all__ = ["REVIEW_PAGE_WAIT_MS", "TIERS", "Z95", "mcnemar_exact", "percentile", "rate", "summarize", "wilson"]
|