mirror of
https://github.com/ThearchyHelios/AgentLab.git
synced 2026-11-18 22:42:00 +00:00
2371e75f17
为什么:第 2 波只有题集和期望答案的来源,评测本身跑不起来。这一段把细案 01 做法 9–12、16、17 的后端接上, 量的是用户在问数据页拿到的同一轮,而不是另抄一份编排。 - eval/answer.py:无头作答,串起 answer_ops → chat/turn 的决定 → launch_run(origin=eval,带目录视图、只读、遮罩) → 等终态(停在审批的取消、记「评测不代批」;驳回记路径「被驳回」)→ review_of_run → 按同一规则重来一次; 工作流题按钉住版本的图跑探索运行(不发正式运行,免得被 pick_base 和应用视图取到)。用量按角色记 - eval/runner.py:用 core/job_runtime,一次评测一个任务,题 × 重复 × 臂一项;A/B 几臂共用期望、同一题同时开跑; 开评拍目录快照、记上传表格版本和表结构哈希的前后;并发不超过 max_parallel 和全局并发的四分之一;每题时限; 评测单独一本账(开跑前估算、每题预留后找补、触顶不再开新题);止损;能停、只重试没做完的、重启标中断不续跑 - eval/score.py:六种期望、五种结论(量纲不同单列)、维度配对与弱匹配、路径不对直接不通过、口径 dual 两版都报; 逐题 E1 用口径账本的 compare_queries(总方案 X2,不另写 signals);SCORER_VERSION 可重评(eval/rescore.py,只读封存) - eval/judging.py:结论裁判走按需裁判同一条路(evidence._judge_now),文字要点由裁判判、单列不计准确率;花费经 judge.charged_to 记在评测那本账上(engine/judge.py 只加了一个上下文变量和取账本的一处) - eval/stats.py、eval/compare.py:准确率与可追溯准确率 + Wilson 区间、各项分布、P50/P90、按角色成本、引用率、逐题 E1; 选基准、逐题转移与归因、指纹差异、A/B 配对表与精确 McNemar;「目录贡献」预设对照(当前 vs 关,可加只用已核实) - eval/cleanup.py + core/artifact_store.forget:保留最近 N 次 + 钉住的基线 + A/B 另几臂,删结果、明细工件、底下的运行和 引用行,工件文件一条引用都不剩才删;总表永久保留 - engine/tolerance.py:双盲复算的 _agree / _delta_pct 原样搬来共用(dual_check 改 import,行为不变) - 设置加 eval 一组(builder 读、admin 写)和 GET /api/settings/eval/spend;题集预算在全站缺省上覆盖,每日上限只能往下收 - api/evals.py:发起(单臂 / A/B / 目录贡献预设)、估算、进度、逐题、停止、重试、重评、钉基线、和上一次比、配对表、清理 - main.py 的 lifespan:影子起草和评测的启动整理、关停(评测先于运行停,停时取消自己发起的运行) - 测试:test_eval_e2e(dual_check_shop + 剧本模型:按支付日 330 通过且 SQL 等价,按下单日 310 不通过、差在 time; 目录对照 1:0;和上一次比找到退步、归因做法变);通知事件、定时接口形状的老测试按新事件和新字段更新
75 lines
3.4 KiB
Python
75 lines
3.4 KiB
Python
"""按新规则重新评分(细案 01 做法 10):评分规则改了(score.SCORER_VERSION 加一),已经评完的评测不必重跑一遍。
|
|
|
|
**只读封存,不调模型、不查业务库。** 每一项:期望用存下的那一份(EvalResult.gold:作答前后两次和结论),答案从作答那次运行的
|
|
封存里重新读(score.read_answer),文字要点用当时裁判给的判断(EvalResult.judge,不再请裁判),逐题 SQL 差异按同一个比对函数
|
|
再比一次(纯计算)。作答前就算不出期望、没作答的,照旧按存下的期望结论。总表跟着重算(runner.refresh_summary)。
|
|
|
|
明细(作答经过、直接回答的原文)在明细工件里:清理掉了的评测不能重评(接口先拦)。
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import logging
|
|
from typing import Any
|
|
|
|
from sqlalchemy import select
|
|
|
|
from app.db.base import SessionLocal
|
|
from app.db.models_eval import EvalCase, EvalResult, EvalRun
|
|
|
|
logger = logging.getLogger(__name__)
|
|
|
|
|
|
async def rescore(eval_run_id: str) -> int:
|
|
"""重评一次评测(只这一臂)。交回结论变了的项数。"""
|
|
from app.core import artifact_store
|
|
from app.eval import runner, score
|
|
from app.eval.cases import case_fields
|
|
|
|
async with SessionLocal() as session:
|
|
run = await session.get(EvalRun, eval_run_id)
|
|
if run is None:
|
|
return 0
|
|
rows = list((await session.execute(select(EvalResult).where(EvalResult.eval_run_id == eval_run_id)
|
|
.order_by(EvalResult.seq))).scalars())
|
|
cases = {c.id: case_fields(c) for c in (await session.execute(
|
|
select(EvalCase).where(EvalCase.id.in_({r.case_id for r in rows if r.case_id})))).scalars()}
|
|
changed = 0
|
|
updates: dict[str, dict[str, Any]] = {}
|
|
for row in rows:
|
|
case = cases.get(row.case_id or "")
|
|
if row.status != "ok" or case is None:
|
|
continue
|
|
details: dict[str, Any] = {}
|
|
if row.details_artifact:
|
|
try:
|
|
details = artifact_store.load(row.details_artifact) or {}
|
|
except ValueError:
|
|
details = {}
|
|
turn = details.get("turn") or {}
|
|
if row.path is None:
|
|
graded = score.score_case(case, row.gold, score.Answer(run_id=None), path=None)
|
|
match = row.sql_match
|
|
else:
|
|
answer = await score.read_answer(row.run_id if row.path == "ran" else None, reply=turn.get("reply"))
|
|
graded = score.score_case(case, row.gold, answer, path=row.path, path_reason=turn.get("reason"),
|
|
judgement=(row.judge or {}).get("text"))
|
|
match = await score.sql_match(case, row.gold, answer)
|
|
if graded.verdict != row.verdict:
|
|
changed += 1
|
|
updates[row.id] = {"verdict": graded.verdict, "scores": graded.scores, "sql_match": match,
|
|
"error": graded.reason if graded.verdict != "pass" else None}
|
|
async with SessionLocal() as session:
|
|
for result_id, values in updates.items():
|
|
row = await session.get(EvalResult, result_id)
|
|
if row is not None:
|
|
for key, value in values.items():
|
|
setattr(row, key, value)
|
|
await session.flush()
|
|
run = await session.get(EvalRun, eval_run_id)
|
|
await runner.refresh_summary(session, run)
|
|
await session.commit()
|
|
return changed
|
|
|
|
|
|
__all__ = ["rescore"]
|