mirror of
https://github.com/ThearchyHelios/AgentLab.git
synced 2026-11-18 17:41:57 +00:00
评测停止、中断、到了上限后的计数照实分开:没跑的题单独计 not_run,stop_reason 写停下的原因
验收:评测停止或中断后,没跑的题计入 errors、done = total、stop_reason 为 null,读起来像全评完了、出了很多错。 原因:runner._counts 把 cancelled、skipped 都算进 done(TASK_DONE),cancelled 又算进 errors;停止、中断走 _close → _conclude(status, None, error),原因写进了 error、stop_reason 是 None;服务重启后的整理(startup、reconcile_stale) 和排队中就停掉(cancel_queued)只改了逐项状态,计数还是旧的、stop_reason 也没写。 改法: - _counts:done 只算真的跑过的(ok、partial、failed);停止、中断时没评完的(cancelled)和到了上限、止损之后没开评 的(skipped)单独计 not_run;errors 只算 failed;skipped 只算评了、但期望算不出、不评分的(verdict),不再混进没开评 的。done + not_run + running + pending = total。 - 停止、中断是「停下」不是「出错」:_close 把原因记进 stop_reason(STOPPED / INTERRUPTED),error 留空;整体出错照旧 记 error。到了上限、止损本来就写 stop_reason,不变。还没备好几臂就停下的那一支(bare)同样写 stop_reason 并重算计数。 - core/job_runtime 的 close_job、cancel_queued 加一个可选的 finish:状态改好、提交之前在同一个事务里补一步。评测的 startup、reconcile_stale、stop 用它记 stop_reason、按改好的逐项状态重算计数(_closing)。起草、剖析、探查不传, 行为不变(它们的测试全过)。 - stats.summarize 的 not_run 也只算没跑的(cancelled、skipped),出错的另列 errors(以前 not_run 里混着出错的)。 前端:评测页总览「N 项没评(停止、中断,或者到了上限、止损之后没开评)」和「M 项出错」分两行;类型补上 not_run、 errors 的说明。停下的原因本来就照 stop_reason 显示。 测试(test_eval_runner):停止之后 stop_reason 是「评测已停止」、error 为空,errors 为 0、done 是评完的题数、not_run 是其余,总表的 not_run、errors 同样分开;重启整理成 interrupted 时 stop_reason 是中断的说法、计数重算(not_run 1、 running 0);新增一条排队中就停掉的:not_run 等于题数、pending 归零;到了金额上限没开评的那题记 not_run、不记跳过。 去掉修复回跑挂 4 条。check-evals 的停止夹具照服务端改,核对停下的原因、不当成出错、没评的单列。评测、起草、剖析、 探查、定时维护相关的测试 548 条全过;tsc、check-copy 通过,check-evals 103 项全过。
This commit is contained in:
@@ -146,10 +146,14 @@ async def test_stop_then_retry_only_the_unfinished(monkeypatch):
|
||||
await asyncio.sleep(0.1)
|
||||
assert await runner.stop(run.id)
|
||||
row = await get(run.id)
|
||||
assert row.status == "cancelled" and row.error == runner.STOPPED
|
||||
# 停止是「停下」不是「出错」:原因记在 stop_reason;没跑的题单独计 not_run,不算进 errors,done 不再等于 total
|
||||
assert row.status == "cancelled" and row.stop_reason == runner.STOPPED and row.error is None
|
||||
before = {r.id: (r.status, r.finished_at) for r in await results(run.id)}
|
||||
done = [rid for rid, (status, _) in before.items() if status == "ok"]
|
||||
assert done and any(status == "cancelled" for status, _ in before.values())
|
||||
assert row.counts["errors"] == 0 and row.counts["done"] == len(done)
|
||||
assert row.counts["not_run"] == 4 - len(done) and row.counts["done"] + row.counts["not_run"] == row.counts["total"]
|
||||
assert row.summary["not_run"] == 4 - len(done) and row.summary["errors"] == 0
|
||||
assert not await runner.stop(run.id), "已经结束的停不了"
|
||||
|
||||
stub(monkeypatch, delay=0.01)
|
||||
@@ -175,12 +179,25 @@ async def test_a_restart_marks_unfinished_evaluations_interrupted_and_does_not_r
|
||||
run_id = row.id
|
||||
assert await runner.startup() >= 1
|
||||
row = await get(run_id)
|
||||
assert row.status == "interrupted" and row.error == runner.INTERRUPTED
|
||||
assert row.status == "interrupted" and row.stop_reason == runner.INTERRUPTED and row.error is None
|
||||
assert row.counts["not_run"] == 1 and row.counts["running"] == 0 and row.counts["errors"] == 0, row.counts
|
||||
[item] = await results(run_id)
|
||||
assert item.status == "cancelled"
|
||||
assert not runner.manager.running(run_id), "不自动续跑"
|
||||
|
||||
|
||||
async def test_stopping_a_queued_evaluation_counts_everything_as_not_run(monkeypatch):
|
||||
"""排队中就停掉(还没开始):每一题都没跑,stop_reason 写停止,计数跟着重算(以前还是建的时候那份:pending 照旧)。"""
|
||||
stub(monkeypatch)
|
||||
suite_id = await es.make_suite("none", cases=ask_cases(2))
|
||||
monkeypatch.setattr(runner.manager, "start", lambda *_a, **_k: None) # 不让它开跑:停在排队
|
||||
[run] = await start(suite_id)
|
||||
assert await runner.stop(run.id)
|
||||
row = await get(run.id)
|
||||
assert row.status == "cancelled" and row.stop_reason == runner.STOPPED
|
||||
assert (row.counts["not_run"], row.counts["pending"], row.counts["done"]) == (2, 0, 0), row.counts
|
||||
|
||||
|
||||
async def test_the_per_run_cap_stops_new_items(monkeypatch):
|
||||
stub(monkeypatch, cost=1.0)
|
||||
suite_id = await es.make_suite("none", cases=ask_cases(3), budget={"max_parallel": 1, "per_run_max_usd": 1.5})
|
||||
@@ -190,6 +207,8 @@ async def test_the_per_run_cap_stops_new_items(monkeypatch):
|
||||
items = await results(run.id)
|
||||
assert [r.status for r in items] == ["ok", "ok", "skipped"]
|
||||
assert row.status == "partial" and row.stop_reason == runner.LIMIT_RUN.format(limit=1.5)
|
||||
assert (row.counts["done"], row.counts["not_run"], row.counts["errors"], row.counts["skipped"]) == (2, 1, 0, 0), \
|
||||
"到了上限没开评的是没跑,不是跳过(跳过是评了、但期望算不出、不评分)"
|
||||
assert "金额上限" in items[2].error and row.summary["cost_usd"] == 2.0
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user