mirror of
https://github.com/ThearchyHelios/AgentLab.git
synced 2026-11-15 05:21:57 +00:00
ThearchyHelios
45c1ede402
为什么：前三波评测只有接口，这一波是收口——每样能力都要有界面、能走通（细案 01 第 18 项、第 15 项定时的前端部分，评测-D8）。 界面层不放逻辑：比率、结论、趋势、分档、能不能做都由服务端给（上一次提交在 api/evals.py 补的 latest、trend、actions、 preview、signals、逐题花费），界面只显示。 - 导航：「评测」排在设置前面、⌥0，搭建者以上可见（使用者的页面表在 visiblePages 里收窄，手输地址被送回应用首页）。导航从 9 项 变 10 项，在 600、520、480 像素高的窗口里重新量过：≤520px 那档的导航项从 32px 收到 30px（17px 图标 + 2px + 11px 字）， 不然 480 高时遥测点超出视口 12px；App.tsx 的说明跟着改。 - 评测页 pages/evals/*（类型 typesEval.ts、接口 api/evals.ts、文案 lib/termsEval.ts 的 EVAL_TEXT）：题集列表（最近一次、同一 变体的趋势、定时，没有定时给「设定时」带着题集去新建）；新建、导入（先预览）、导出、归档、删除、题集设置（时区、默认变体—— 定时评测按它评、预算只填要改的项）；题目表（启用、试算、编辑、删除），口径升版待处置的顶部提示和三选一处置（叫法由服务端给， 没有缺省）；加题四种方式：写 SQL 定期望（选数据源、写 SQL、预览、点选格 / 列 / 配对列、按维度找行、试算，遮罩列拒收的原话 落在 SQL 下面）、选口径（工作流｜口径卡｜指标，不能复用的照列写原因，口径的输入按那一版的入口字段填）、从问数据运行认定 （选运行、选查询、选取哪一列，用到遮罩列的查询不给认定）、期望没有数据或期望反问；可另加文字要点。数值期望不许人填：只点选 取哪一格。 - 发起评测：评一次 / A/B 对照（两到三臂）/ 目录贡献对照（预设，可加「只用已核实和已确认的项」），变体有助手、写作模型、结论 裁判、目录（当前 / 只用已核实的 / 关 / 快照）、重复次数；估算和发起是同一份请求体，项数、按上一次估的花费、生效的预算、今天 已花照服务端写；可点名和哪一次比。 - 评测详情 /evals/runs/:id：进度和每题正在做什么、停止、只重试没做完的、按新规则重评、设为基线（按钮照服务端的 actions 摆）； 总览卡片（准确率和可追溯准确率带区间、花费、逐题 E1）和各项分布（结论、作答路径、出具档位、复核、结论裁判、引用率、查看、 耗时、按角色的用量）；逐题表（结论、期望和答案里的数、弱匹配、SQL 差在哪、查看、耗时、花费，展开看原因、去运行详情； ?case= 定位那道题）；和上一次比（换基准、指纹差异、转移和归因）；A/B 配对表（都对 / 只 A 对 / 只 B 对 / 都错、p 值）。 - 「口径命中与重复度」页签：E1、E3 这一窗口的值和档位、参考线、往前几个窗口的趋势（GET /api/evals/signals），明细直接调口径 账本的 GET /api/ledger/stats（总方案 X2）；成员不少于 3 的问题类能加入题集（带着问法去加题）、提取模板（从最近一次的运行提取， 去画布）。目录快照页签：拍当前目录、影子起草（先估算再开始，能停）。 - 定时页：新建时先选定时什么（工作流 / 评测题集；数据源维护在数据源页建），评测题集的表单（pages/schedules/TargetEditor.tsx：名称、题集、 FreqPicker、补发宽限，建题集目标用 ScheduleTargetCreate），评测题集的定时在详情里能改名称、频率、时区、补发宽限（SchedulePatch， 只交改了的、带版本）；数据源维护的定时只读、链到数据源页，第 3 波「只能查看，不能修改」那句改成说清在数据源页设置；触发历史里 的评测链到评测详情。纯函数 targetFormOf / targetCreateBody / targetPatchBody / targetProblems 放 model.ts。 - 记录页：评测运行旁的「评测」标记写是哪一步（作答、算期望……），点了回到评测里那道题；运行详情页头有键盘到得了的链接 （pages/runs/EvalSource.tsx）。 - 设置：偏好设置里「评测」一节（pages/EvalSettings.tsx，SettingsPage 里加一行）：四项预算缺省、保留几次、缺省开不开结论裁判、 今天已花；自己存、只交 eval 一组，取值范围只在服务端判。 - 问数据用量（评测-D14 的界面一半）：复核慢过问数据页的等待上限时，答案照原样先交出去，复核那一次的用量晚到了照记进这一轮、 再落一次库（以前直接丢掉）；开发期 window.__chatReviewWait 让检查脚本把上限调短。 检查：check-evals.mjs 换成真检查（假接口加夹具，写请求在浏览器层拦下并断言请求体，98 项）；check-shell 导航十项、⌥0、三种窗口高； check-schedules 目标种类（纯函数和页面）；check-chat 用量那一段加复核晚到。
223 lines
13 KiB
Python
223 lines
13 KiB
Python
"""评测页（第 4 波，细案 01 第 18 项）在 app/api/evals.py 里补的几样：统计、能不能做都由服务端给，界面只显示。
|
|
|
|
- 题集列表每个题集带最近一次评测、同一变体最近几次评完的准确率（旧的在前，换了变体的不连成一条线）、目标是它的定时任务；
|
|
- 评测详情带 actions：只重试没做完的、重新评分、钉住 / 取消基线现在能不能做，和接口拒收的条件同一份；
|
|
- 写 SQL 定期望的预览：交回结果的列和前几行；用到遮罩列的照保存时那一道拒收，没过只读守卫的指明是哪一格（都不回 422）；
|
|
- 「口径命中与重复度」：判据、窗口趋势（出自 ledger/stats.compute_stats）、成员不少于 3 的问题类（出自 chat/questions），
|
|
评测运行不进问题类。
|
|
|
|
库、表、题集名、问法都是虚构的。
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import uuid
|
|
from datetime import timedelta
|
|
|
|
import pytest
|
|
from httpx import ASGITransport, AsyncClient
|
|
from sqlalchemy import delete
|
|
|
|
from app.db.base import SessionLocal, utcnow
|
|
from app.db.models import Conversation, ConversationTurn, Run, Schedule
|
|
from app.db.models_eval import EvalResult, EvalRun
|
|
from app.engine.runner import run_manager
|
|
from app.main import app
|
|
from tests import eval_samples as es
|
|
from tests.fixtures import dual_check_shop as shop
|
|
from tests.fixtures.sources import drop_source
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
async def engine_up():
|
|
await run_manager.setup()
|
|
yield
|
|
await run_manager.shutdown()
|
|
|
|
|
|
@pytest.fixture
|
|
async def client():
|
|
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as c:
|
|
yield c
|
|
|
|
|
|
@pytest.fixture
|
|
async def source(tmp_path):
|
|
name, sid = await shop.add_source(shop.build(tmp_path / "shop.db"))
|
|
yield {"name": name, "id": sid}
|
|
await drop_source(sid)
|
|
|
|
|
|
def _summary(passed: int, n: int) -> dict:
|
|
return {"accuracy": {"passed": passed, "n": n, "rate": round(passed / n, 4) if n else None, "ci": None},
|
|
"traced_accuracy": {"passed": passed, "n": n, "rate": round(passed / n, 4) if n else None, "ci": None}}
|
|
|
|
|
|
async def _evaluation(suite_id: str, *, status: str, key: str, passed: int, n: int, minutes_ago: int,
|
|
arm: str | None = None, pinned: bool = False, cleaned: bool = False) -> str:
|
|
async with SessionLocal() as session:
|
|
row = EvalRun(suite_id=suite_id, status=status, trigger="manual", variant={}, resolved_models={},
|
|
fingerprint={"variant_key": key}, summary=_summary(passed, n) if status != "queued" else {},
|
|
counts={"total": n, "done": n}, arm=arm, pinned=pinned,
|
|
cleaned_at=utcnow() if cleaned else None)
|
|
session.add(row)
|
|
await session.flush()
|
|
row.created_at = utcnow() - timedelta(minutes=minutes_ago)
|
|
await session.commit()
|
|
return row.id
|
|
|
|
|
|
async def test_the_suite_list_carries_the_latest_run_a_trend_of_the_same_variant_and_its_schedules(client, source):
|
|
suite_id = await es.make_suite(source["id"], name=f"虚构题集_{uuid.uuid4().hex[:6]}")
|
|
old = await _evaluation(suite_id, status="succeeded", key="A", passed=1, n=4, minutes_ago=50)
|
|
other = await _evaluation(suite_id, status="succeeded", key="B", passed=4, n=4, minutes_ago=40)
|
|
failed = await _evaluation(suite_id, status="failed", key="A", passed=0, n=4, minutes_ago=30)
|
|
mid = await _evaluation(suite_id, status="partial", key="A", passed=2, n=4, minutes_ago=20)
|
|
new = await _evaluation(suite_id, status="succeeded", key="A", passed=3, n=4, minutes_ago=10)
|
|
async with SessionLocal() as session:
|
|
sched = Schedule(name="虚构每周回归", workflow_id="", workflow_name="虚构题集", target_kind="eval_suite",
|
|
target_id=suite_id, cron="0 9 * * 1", timezone="Asia/Shanghai", enabled=True)
|
|
session.add(sched)
|
|
await session.commit()
|
|
sched_id = sched.id
|
|
try:
|
|
listed = {s["id"]: s for s in (await client.get("/api/evals/suites")).json()["suites"]}
|
|
got = listed[suite_id]
|
|
assert got["latest"]["id"] == new and got["latest"]["accuracy"]["rate"] == 0.75
|
|
# 趋势只连同一变体、评完了的：B 那一次、失败的那一次都不进；旧的在前
|
|
assert [p["id"] for p in got["trend"]] == [old, mid, new], got["trend"]
|
|
assert [p["rate"] for p in got["trend"]] == [0.25, 0.5, 0.75]
|
|
assert other not in [p["id"] for p in got["trend"]] and failed not in [p["id"] for p in got["trend"]]
|
|
[schedule] = got["schedules"]
|
|
assert schedule["id"] == sched_id and schedule["enabled"] is True and "09:00" in schedule["description_text"]
|
|
detail = (await client.get(f"/api/evals/suites/{suite_id}")).json()
|
|
assert detail["latest"]["id"] == new and [s["id"] for s in detail["schedules"]] == [sched_id]
|
|
finally:
|
|
async with SessionLocal() as session:
|
|
await session.execute(delete(Schedule).where(Schedule.id == sched_id))
|
|
await session.commit()
|
|
await es.drop_suites()
|
|
|
|
|
|
async def test_the_evaluation_says_what_can_be_done_now(client, source):
|
|
suite_id = await es.make_suite(source["id"], name=f"虚构题集_{uuid.uuid4().hex[:6]}")
|
|
try:
|
|
run_id = await _evaluation(suite_id, status="partial", key="A", passed=1, n=2, minutes_ago=5)
|
|
async with SessionLocal() as session:
|
|
session.add(EvalResult(eval_run_id=run_id, seq=1, case_id=None, case_seq=1, case_hash="h", repeat=0,
|
|
status="ok", verdict="pass"))
|
|
session.add(EvalResult(eval_run_id=run_id, seq=2, case_id=None, case_seq=2, case_hash="h", repeat=0,
|
|
status="cancelled", error="评测已停止"))
|
|
await session.commit()
|
|
actions = (await client.get(f"/api/evals/evaluations/{run_id}")).json()["actions"]
|
|
assert actions == {"stop": False, "retry": True, "rescore": True, "pin": True, "unpin": False, "open": 1,
|
|
"cleaned": False}
|
|
pinned = await client.post(f"/api/evals/evaluations/{run_id}/pin", json={"pinned": True})
|
|
assert pinned.status_code == 200
|
|
actions = (await client.get(f"/api/evals/evaluations/{run_id}")).json()["actions"]
|
|
assert actions["pin"] is False and actions["unpin"] is True
|
|
|
|
done = await _evaluation(suite_id, status="succeeded", key="A", passed=2, n=2, minutes_ago=4, cleaned=True)
|
|
actions = (await client.get(f"/api/evals/evaluations/{done}")).json()["actions"]
|
|
# 明细清理了的：不能重试、不能重评（和接口拒收的条件一致）
|
|
assert actions["retry"] is False and actions["rescore"] is False and actions["cleaned"] is True
|
|
assert (await client.post(f"/api/evals/evaluations/{done}/rescore")).status_code == 409
|
|
finally:
|
|
await es.drop_suites()
|
|
|
|
|
|
async def test_previewing_an_expected_sql_gives_columns_and_rows(client, source):
|
|
made = await client.post("/api/evals/suites", json={"name": f"虚构题集_{uuid.uuid4().hex[:6]}"})
|
|
suite_id = made.json()["id"]
|
|
try:
|
|
body = {"datasource_id": source["id"],
|
|
"sql": "SELECT status AS 状态, SUM(amount) AS gmv FROM orders GROUP BY status ORDER BY status"}
|
|
got = (await client.post(f"/api/evals/suites/{suite_id}/cases/preview", json=body)).json()
|
|
assert got["status"] == "ok" and got["columns"] == ["状态", "gmv"], got
|
|
assert got["rows"] and all(len(r) == 2 for r in got["rows"]) and got["run_id"]
|
|
assert got["rendered"]["sql"].startswith("SELECT status")
|
|
# 没过只读守卫：指明是 SQL 那一格，不回 422
|
|
res = await client.post(f"/api/evals/suites/{suite_id}/cases/preview",
|
|
json={"datasource_id": source["id"], "sql": "DELETE FROM orders"})
|
|
assert res.status_code == 200 and res.json()["status"] == "invalid" and res.json()["field"] == "gold.sql"
|
|
res = await client.post(f"/api/evals/suites/{suite_id}/cases/preview", json={"sql": "SELECT 1"})
|
|
assert res.json()["status"] == "invalid" and res.json()["field"] == "gold.datasource_id"
|
|
finally:
|
|
await es.drop_suites()
|
|
|
|
|
|
async def test_a_preview_touching_a_masked_column_is_refused(client, tmp_path):
|
|
name, sid = await shop.add_source(shop.build(tmp_path / "masked.db"), options={"mask_columns": ["region"]})
|
|
made = await client.post("/api/evals/suites", json={"name": f"虚构题集_{uuid.uuid4().hex[:6]}"})
|
|
try:
|
|
got = (await client.post(f"/api/evals/suites/{made.json()['id']}/cases/preview", json={
|
|
"datasource_id": sid, "sql": "SELECT region AS 地区, SUM(amount) AS gmv FROM orders GROUP BY region"})).json()
|
|
assert got["status"] == "refused" and got["code"] == "eval_gold_masked" and "遮罩" in got["reason"], got
|
|
assert got["rows"] == [] and got["columns"] == [], "遮罩列的值一格都不能交出去"
|
|
finally:
|
|
await es.drop_suites()
|
|
await drop_source(sid)
|
|
|
|
|
|
async def _asked(question: str, n: int, *, origin: str = "chat") -> list[str]:
|
|
async with SessionLocal() as session:
|
|
conv = Conversation(kind="chat", title="虚构对话")
|
|
session.add(conv)
|
|
await session.flush()
|
|
ids = []
|
|
for i in range(n):
|
|
run = Run(workflow_name="问数据", status="succeeded", graph={}, input={"question": question},
|
|
origin_kind=origin, run_class="exploratory")
|
|
session.add(run)
|
|
await session.flush()
|
|
session.add(ConversationTurn(conversation_id=conv.id, seq=i, question=question, run_id=run.id,
|
|
status="done"))
|
|
ids.append(run.id)
|
|
await session.commit()
|
|
return ids
|
|
|
|
|
|
async def test_signals_give_criteria_trend_and_repeated_question_clusters(client):
|
|
question = f"虚构门店 {uuid.uuid4().hex[:6]} 上周的销售额是多少？"
|
|
runs = await _asked(question, 3)
|
|
eval_question = f"虚构评测题 {uuid.uuid4().hex[:6]} 本月退款率？"
|
|
await _asked(eval_question, 3, origin="eval")
|
|
got = (await client.get("/api/evals/signals", params={"days": 30, "points": 3})).json()
|
|
assert [c["band"] for c in got["criteria"]["e1"]] == ["route", "normalize", "reconcile"]
|
|
assert got["criteria"]["e3"][0]["at_least"] == 0.2
|
|
assert len(got["trend"]) == 3 and got["current"] == got["trend"][-1]
|
|
assert got["trend"][0]["until"] <= got["trend"][1]["since"] or got["trend"][0]["until"] == got["trend"][1]["since"]
|
|
assert got["current"]["e3"]["n"] >= 3 and got["current"]["e3"]["band"] in ("worth", "not_yet")
|
|
mine = [c for c in got["clusters"] if c["representative"] == question]
|
|
assert mine and mine[0]["count"] == 3 and mine[0]["run_id"] in runs, got["clusters"]
|
|
assert not [c for c in got["clusters"] if c["representative"] == eval_question], "评测运行不进问题类"
|
|
assert got["threshold"] == 3 and got["embedder"]
|
|
assert (await client.get("/api/evals/signals", params={"days": 0})).status_code == 422
|
|
|
|
|
|
def test_bands_follow_the_reference_lines():
|
|
from app.api import evals
|
|
|
|
assert [evals._e1_band(x) for x in (None, 0.05, 0.1, 0.29, 0.3)] == [None, "reconcile", "normalize", "normalize",
|
|
"route"]
|
|
assert [evals._e3_band(x) for x in (None, 0.19, 0.2)] == [None, "not_yet", "worth"]
|
|
|
|
|
|
async def test_each_result_carries_its_total_cost(client, source):
|
|
"""逐题结果带这一题的花费：助手（已含复核）+ 写作 + 结论裁判，估不出的另计次数——界面直接显示，不自己加。"""
|
|
suite_id = await es.make_suite(source["id"], name=f"虚构题集_{uuid.uuid4().hex[:6]}")
|
|
try:
|
|
run_id = await _evaluation(suite_id, status="succeeded", key="A", passed=1, n=1, minutes_ago=1)
|
|
async with SessionLocal() as session:
|
|
session.add(EvalResult(eval_run_id=run_id, seq=1, case_id=None, case_seq=1, case_hash="h", repeat=0,
|
|
status="ok", verdict="pass", usage={
|
|
"assistant": {"cost_usd": 0.012, "unpriced_calls": 0},
|
|
"review": {"cost_usd": 0.002, "unpriced_calls": 0},
|
|
"writer": {"cost_usd": 0.03, "unpriced_calls": 1},
|
|
"judge": {"cost_usd": 0.001, "calls": 1}}))
|
|
await session.commit()
|
|
[row] = (await client.get(f"/api/evals/evaluations/{run_id}/results")).json()["results"]
|
|
# 复核已经算在助手里（runner 记的 assistant = 建图 + 复核），不重复加
|
|
assert row["cost"] == {"cost_usd": 0.043, "unpriced_calls": 1}, row["cost"]
|
|
finally:
|
|
await es.drop_suites()
|