评测停止、中断、到了上限后的计数照实分开：没跑的题单独计 not_run，stop_reason 写停下的原因

验收：评测停止或中断后，没跑的题计入 errors、done = total、stop_reason 为 null，读起来像全评完了、出了很多错。原因：runner._counts 把 cancelled、skipped 都算进 done（TASK_DONE），cancelled 又算进 errors；停止、中断走 _close → _conclude(status, None, error)，原因写进了 error、stop_reason 是 None；服务重启后的整理（startup、reconcile_stale）和排队中就停掉（cancel_queued）只改了逐项状态，计数还是旧的、stop_reason 也没写。改法： - _counts：done 只算真的跑过的（ok、partial、failed）；停止、中断时没评完的（cancelled）和到了上限、止损之后没开评的（skipped）单独计 not_run；errors 只算 failed；skipped 只算评了、但期望算不出、不评分的（verdict），不再混进没开评的。done + not_run + running + pending = total。 - 停止、中断是「停下」不是「出错」：_close 把原因记进 stop_reason（STOPPED / INTERRUPTED），error 留空；整体出错照旧记 error。到了上限、止损本来就写 stop_reason，不变。还没备好几臂就停下的那一支（bare）同样写 stop_reason 并重算计数。 - core/job_runtime 的 close_job、cancel_queued 加一个可选的 finish：状态改好、提交之前在同一个事务里补一步。评测的 startup、reconcile_stale、stop 用它记 stop_reason、按改好的逐项状态重算计数（_closing）。起草、剖析、探查不传，行为不变（它们的测试全过）。 - stats.summarize 的 not_run 也只算没跑的（cancelled、skipped），出错的另列 errors（以前 not_run 里混着出错的）。前端：评测页总览「N 项没评（停止、中断，或者到了上限、止损之后没开评）」和「M 项出错」分两行；类型补上 not_run、 errors 的说明。停下的原因本来就照 stop_reason 显示。测试（test_eval_runner）：停止之后 stop_reason 是「评测已停止」、error 为空，errors 为 0、done 是评完的题数、not_run 是其余，总表的 not_run、errors 同样分开；重启整理成 interrupted 时 stop_reason 是中断的说法、计数重算（not_run 1、 running 0）；新增一条排队中就停掉的：not_run 等于题数、pending 归零；到了金额上限没开评的那题记 not_run、不记跳过。去掉修复回跑挂 4 条。check-evals 的停止夹具照服务端改，核对停下的原因、不当成出错、没评的单列。评测、起草、剖析、探查、定时维护相关的测试 548 条全过；tsc、check-copy 通过，check-evals 103 项全过。
2026-11-15 00:31:58 +00:00 · 2026-10-17 09:10:52 +08:00
parent a4486af90b
commit ccb2b2baa8
8 changed files with 111 additions and 30 deletions
@@ -22,8 +22,9 @@
  说明源上正有别的事，接口回 409；占到了，执行期间新建的任务排在它后面。**队里的事是开放的**：运行时不认具体哪几种，
  每种在自己的模块里 register_lane_kind 登记一次（kind → 叫法），进队时只核对登记过——第 2 波的后台探查（introspect）
  登记一行就排进同一条队，不用改这里；拼错的 kind 当场报错，不会悄悄排成一种谁都不认识的事。
- **close_job / interrupt_on_startup**：收尾（还没结束的表记成某个状态，任务记成某个状态），和服务启动时的整理：排队、
-  进行中的标 interrupted，没结束的表记 cancelled。**不自动续跑**：复制出来的数据目录（检查脚本、验收副本）启动时不能
+- **close_job / interrupt_on_startup**：收尾（还没结束的表记成某个状态，任务记成某个状态；可选的 finish 在同一个事务里
+  补一步，评测拿它按改好的逐项状态重算计数、记停下的原因），和服务启动时的整理：排队、进行中的标 interrupted，没结束的
+  表记 cancelled。**不自动续跑**：复制出来的数据目录（检查脚本、验收副本）启动时不能
  自己去调模型、查业务库，界面让人点「重试」接着做。

 进程内登记、进程内的队：服务端是单进程部署，多进程部署时要换成库里的锁。
@@ -243,9 +244,16 @@ lanes = SourceLanes()
 # ==========================================================================


+#: 收尾时在同一个事务里补写的那一步（close_job、cancel_queued 的 finish）：表和任务的状态已经改好、还没提交
+Finish = Callable[[AsyncSession], Awaitable[None]]
+
+
 async def close_job(session: AsyncSession, job_model: Any, task_model: Any, job_id: str, job_status: str,
-                    job_error: str | None, task_status: str, task_error: str | None) -> None:
-    """收尾：还没结束的表记 task_status（原因 task_error），任务（还没结束的话）记 job_status。提交。"""
+                    job_error: str | None, task_status: str, task_error: str | None, *,
+                    finish: Finish | None = None) -> None:
+    """收尾：还没结束的表记 task_status（原因 task_error），任务（还没结束的话）记 job_status。提交。
+
+    finish：提交前在同一个事务里再做的一步（评测按改好的逐项状态重算计数、记停下的原因）。起草、剖析、探查不传，照旧。"""
    now = utcnow()
    await session.execute(
        update(task_model).where(task_model.job_id == job_id, task_model.status.in_(OPEN))
@@ -253,12 +261,15 @@ async def close_job(session: AsyncSession, job_model: Any, task_model: Any, job_
    await session.execute(
        update(job_model).where(job_model.id == job_id, job_model.status.in_(ACTIVE))
        .values(status=job_status, error=job_error, finished_at=now).execution_options(synchronize_session=False))
+    if finish is not None:
+        await finish(session)
    await session.commit()


-async def cancel_queued(session: AsyncSession, job_model: Any, task_model: Any, job_id: str) -> bool:
+async def cancel_queued(session: AsyncSession, job_model: Any, task_model: Any, job_id: str, *,
+                        finish: Finish | None = None) -> bool:
    """停止一个排队中的任务：任务和没结束的表直接记 cancelled。已经被后台认领（开始了）返回 False，交给调用方按进行中
-    的停。"""
+    的停。finish 同 close_job（只在真停了时做）。"""
    now = utcnow()
    claimed = await session.execute(
        update(job_model).where(job_model.id == job_id, job_model.status == "queued")
@@ -269,6 +280,8 @@ async def cancel_queued(session: AsyncSession, job_model: Any, task_model: Any,
    await session.execute(
        update(task_model).where(task_model.job_id == job_id, task_model.status.in_(OPEN))
        .values(status="cancelled", finished_at=now).execution_options(synchronize_session=False))
+    if finish is not None:
+        await finish(session)
    await session.commit()
    return True

@@ -708,20 +708,26 @@ class EvalExecution(runtime.JobRun):
                                  .execution_options(synchronize_session=False))
            await session.commit()

+        # 停止、中断是「停下」不是「出错」：原因记 stop_reason（error 留空）；整体出错记 error
+        stopped = status in ("cancelled", "interrupted")
        try:
            await self._write(write)
            if not self.arms:
                async def bare(session: AsyncSession) -> None:
-                    await session.execute(update(EvalRun).where(
+                    ids = list((await session.execute(select(EvalRun.id).where(
                        (EvalRun.id == self.job_id) | (EvalRun.pair_id == self.job_id),
-                        EvalRun.status.in_(runtime.ACTIVE))
-                        .values(status=status, error=error, finished_at=utcnow())
-                        .execution_options(synchronize_session=False))
+                        EvalRun.status.in_(runtime.ACTIVE)))).scalars())
+                    await session.execute(update(EvalRun).where(EvalRun.id.in_(ids))
+                                          .values(status=status, error=None if stopped else error,
+                                                  stop_reason=error if stopped else None, finished_at=utcnow())
+                                          .execution_options(synchronize_session=False))
+                    for run_id in ids:
+                        await _recount(session, run_id)
                    await session.commit()

                await self._write(bare)
                return
-            await self._conclude(status, None, error)
+            await self._conclude(status, error if stopped else None, None if stopped else error)
        except Exception:  # noqa: BLE001
            logger.exception("评测 %s 收尾时写库出错", self.job_id)

@@ -793,16 +799,31 @@ def _sql_shapes(answer: Any) -> list[dict[str, Any]]:
    return out


+#: 逐项结束了、却没有跑出结果的状态：cancelled 停止、中断时还没评完的，skipped 到了金额上限、止损之后没开评的
+NOT_RUN_STATUSES = ("cancelled", "skipped")
+
+
 def _counts(rows: Sequence[Mapping[str, Any]]) -> dict[str, int]:
-    out = {"total": len(rows), "done": 0, "running": 0, "pending": 0, "passed": 0, "failed": 0, "ungradable": 0,
-           "skipped": 0, "errors": 0}
+    """一臂的计数（EvalRun.counts）。几样各管各的，加起来不重：
+
+    - done：评完了的（ok、partial）和评的时候出错的（failed）——真的跑过的；running、pending 是还在评、还没轮到的；
+    - not_run：停止、中断时还没评完，到了上限、止损之后没开评的（NOT_RUN_STATUSES）。以前这些算进 done 和 errors，停下的
+      评测就成了 done = total、一堆「出错」，读起来像全评完了、出了很多错；
+    - errors：出错的（failed：题目没了、评的时候出了意外），不含没跑的；
+    - passed / failed / ungradable / skipped：评完了的按结论分（skipped 是评了、但期望算不出、不评分，不含没开评的）。
+
+    done + not_run + running + pending = total。"""
+    out = {"total": len(rows), "done": 0, "running": 0, "pending": 0, "not_run": 0, "passed": 0, "failed": 0,
+           "ungradable": 0, "skipped": 0, "errors": 0}
    for r in rows:
        status = r.get("status")
-        if status in runtime.TASK_DONE:
+        if status in NOT_RUN_STATUSES:
+            out["not_run"] += 1
+        elif status in runtime.TASK_DONE:
            out["done"] += 1
        elif status in ("running", "pending"):
            out[status] += 1
-        if status in ("failed", "cancelled"):
+        if status == "failed":
            out["errors"] += 1
        verdict = r.get("verdict")
        if verdict == "pass":
@@ -811,11 +832,21 @@ def _counts(rows: Sequence[Mapping[str, Any]]) -> dict[str, int]:
            out["failed"] += 1
        elif verdict == "ungradable":
            out["ungradable"] += 1
-        elif verdict == "skipped" or status == "skipped":
+        elif verdict == "skipped":
            out["skipped"] += 1
    return out


+def _closing(run_id: str, reason: str) -> runtime.Finish:
+    """runtime.close_job / cancel_queued 的 finish：同一个事务里记停下的原因（停止、中断），按改好的逐项状态重算计数。"""
+    async def finish(session: AsyncSession) -> None:
+        await session.execute(update(EvalRun).where(EvalRun.id == run_id).values(stop_reason=reason)
+                              .execution_options(synchronize_session=False))
+        await _recount(session, run_id)
+
+    return finish
+
+
 async def _recount(session: AsyncSession, run_id: str) -> None:
    rows = (await session.execute(select(EvalResult.status, EvalResult.verdict).where(
        EvalResult.eval_run_id == run_id))).all()
@@ -910,16 +941,16 @@ async def reconcile_stale(session: AsyncSession, rows: Sequence[EvalRun]) -> Non
    """读接口当场整理：记着排队、进行中，这个进程里却没在跑（服务重启过）的，照实标中断。"""
    stale = [r for r in rows if r.status in runtime.ACTIVE and not running(r)]
    for row in stale:
-        await runtime.close_job(session, EvalRun, EvalResult, row.id, "interrupted", INTERRUPTED, "cancelled",
-                                INTERRUPTED)
+        await runtime.close_job(session, EvalRun, EvalResult, row.id, "interrupted", None, "cancelled",
+                                INTERRUPTED, finish=_closing(row.id, INTERRUPTED))
        await session.refresh(row)


 async def startup() -> int:
    """服务启动时：没结束的评测标中断（没结束的项记停止），不自动续跑。"""
    async def close(session: AsyncSession, job_id: str) -> None:
-        await runtime.close_job(session, EvalRun, EvalResult, job_id, "interrupted", INTERRUPTED, "cancelled",
-                                INTERRUPTED)
+        await runtime.close_job(session, EvalRun, EvalResult, job_id, "interrupted", None, "cancelled",
+                                INTERRUPTED, finish=_closing(job_id, INTERRUPTED))

    return await runtime.interrupt_on_startup(EvalRun, close, running=manager.running)

@@ -938,7 +969,8 @@ async def stop(run_id: str) -> bool:
            (EvalRun.id == leader) | (EvalRun.pair_id == leader)))).scalars())
        done = False
        for arm in arms:
-            done = await runtime.cancel_queued(session, EvalRun, EvalResult, arm) or done
+            done = await runtime.cancel_queued(session, EvalRun, EvalResult, arm,
+                                               finish=_closing(arm, STOPPED)) or done
        return done


@@ -1,8 +1,8 @@
 """评测的汇总与统计（细案 01 做法 11、12；评测-D10）。所有汇总都在后端算好，界面直接显示（界面层不放逻辑：同一个数不能
 有两套算法）。纯计算，输入是一次评测的逐题结果（EvalResult 的列，result_row 的形状）。

- 准确率和可追溯准确率：分母是能判的题（通过 + 不通过）；不能评分、跳过的单列，既不算对也不算错，没开评、停下、出错的
-  另列 not_run。都带 Wilson 95% 区间和样本数——题集小、模型有随机性，差两三题就可能被读成「变好了」「变差了」（评测-D10：只展示、不卡门）。
+- 准确率和可追溯准确率：分母是能判的题（通过 + 不通过）；不能评分、跳过的单列，既不算对也不算错，没跑的（停止、中断、
+  到了上限、止损没开评）另列 not_run，出错的另列 errors。都带 Wilson 95% 区间和样本数——题集小、模型有随机性，差两三题就可能被读成「变好了」「变差了」（评测-D10：只展示、不卡门）。
 - 作答路径、出具档位（声明了出具的按 formal / degraded / withheld，没声明的单列「探索问答」）、复核结论（verdict × severity）
  和自动重来的次数、复核用时超过问数据页等待（25 秒）的题数。
 - 结论裁判（打开了才有）：supported / partial / contradicted / insufficient / 未判的句数；文字要点的判断单列。
@@ -79,7 +79,7 @@ def summarize(results: Sequence[Mapping[str, Any]], *, judge_on: bool = False) -
    from app.eval.score import SCORER_VERSION

    done = [r for r in results if r.get("status") in ("ok", "skipped", "failed", "cancelled", "partial")]
-    #: 评完了的（status=ok）；没开评、停下、出错的单列成 not_run
+    #: 评完了的（status=ok）；没跑的（停止、中断、到了上限、止损没开评）单列成 not_run，出错的单列成 errors（同 runner._counts）
    evaluated = [r for r in done if r.get("status") == "ok"]
    verdicts = Counter(r.get("verdict") or "none" for r in done)
    graded = [r for r in done if r.get("verdict") in ("pass", "fail")]
@@ -118,7 +118,8 @@ def summarize(results: Sequence[Mapping[str, Any]], *, judge_on: bool = False) -
        "items": len(results), "done": len(done),
        "cases": len({r.get("case_id") for r in results}),
        "verdicts": {k: verdicts.get(k, 0) for k in ("pass", "fail", "ungradable", "skipped")},
-        "not_run": len(done) - len(evaluated),
+        "not_run": sum(1 for r in done if r.get("status") in ("cancelled", "skipped")),
+        "errors": sum(1 for r in done if r.get("status") == "failed"),
        "accuracy": rate(len(passed), len(graded)),
        "traced_accuracy": rate(len(traced), len(graded)),
        "outcomes": dict(outcomes),
@@ -146,10 +146,14 @@ async def test_stop_then_retry_only_the_unfinished(monkeypatch):
    await asyncio.sleep(0.1)
    assert await runner.stop(run.id)
    row = await get(run.id)
-    assert row.status == "cancelled" and row.error == runner.STOPPED
+    # 停止是「停下」不是「出错」：原因记在 stop_reason；没跑的题单独计 not_run，不算进 errors，done 不再等于 total
+    assert row.status == "cancelled" and row.stop_reason == runner.STOPPED and row.error is None
    before = {r.id: (r.status, r.finished_at) for r in await results(run.id)}
    done = [rid for rid, (status, _) in before.items() if status == "ok"]
    assert done and any(status == "cancelled" for status, _ in before.values())
+    assert row.counts["errors"] == 0 and row.counts["done"] == len(done)
+    assert row.counts["not_run"] == 4 - len(done) and row.counts["done"] + row.counts["not_run"] == row.counts["total"]
+    assert row.summary["not_run"] == 4 - len(done) and row.summary["errors"] == 0
    assert not await runner.stop(run.id), "已经结束的停不了"

    stub(monkeypatch, delay=0.01)
@@ -175,12 +179,25 @@ async def test_a_restart_marks_unfinished_evaluations_interrupted_and_does_not_r
        run_id = row.id
    assert await runner.startup() >= 1
    row = await get(run_id)
-    assert row.status == "interrupted" and row.error == runner.INTERRUPTED
+    assert row.status == "interrupted" and row.stop_reason == runner.INTERRUPTED and row.error is None
+    assert row.counts["not_run"] == 1 and row.counts["running"] == 0 and row.counts["errors"] == 0, row.counts
    [item] = await results(run_id)
    assert item.status == "cancelled"
    assert not runner.manager.running(run_id), "不自动续跑"


+async def test_stopping_a_queued_evaluation_counts_everything_as_not_run(monkeypatch):
+    """排队中就停掉（还没开始）：每一题都没跑，stop_reason 写停止，计数跟着重算（以前还是建的时候那份：pending 照旧）。"""
+    stub(monkeypatch)
+    suite_id = await es.make_suite("none", cases=ask_cases(2))
+    monkeypatch.setattr(runner.manager, "start", lambda *_a, **_k: None)  # 不让它开跑：停在排队
+    [run] = await start(suite_id)
+    assert await runner.stop(run.id)
+    row = await get(run.id)
+    assert row.status == "cancelled" and row.stop_reason == runner.STOPPED
+    assert (row.counts["not_run"], row.counts["pending"], row.counts["done"]) == (2, 0, 0), row.counts
+
+
 async def test_the_per_run_cap_stops_new_items(monkeypatch):
    stub(monkeypatch, cost=1.0)
    suite_id = await es.make_suite("none", cases=ask_cases(3), budget={"max_parallel": 1, "per_run_max_usd": 1.5})
@@ -190,6 +207,8 @@ async def test_the_per_run_cap_stops_new_items(monkeypatch):
    items = await results(run.id)
    assert [r.status for r in items] == ["ok", "ok", "skipped"]
    assert row.status == "partial" and row.stop_reason == runner.LIMIT_RUN.format(limit=1.5)
+    assert (row.counts["done"], row.counts["not_run"], row.counts["errors"], row.counts["skipped"]) == (2, 1, 0, 0), \
+        "到了上限没开评的是没跑，不是跳过（跳过是评了、但期望算不出、不评分）"
    assert "金额上限" in items[2].error and row.summary["cost_usd"] == 2.0


@@ -411,7 +411,8 @@ export const EVAL_TEXT = {
    traced: '可追溯准确率',
    tracedHint: '必需的期望全部经引用对上（答案里的数点得开、来自查询的那一格或口径卡的指标）',
    verdicts: '结论',
-    notRun: (n: number) => `${formatNumber(n)} 项没评（停下、出错、到了上限）`,
+    notRun: (n: number) => `${formatNumber(n)} 项没评（停止、中断，或者到了上限、止损之后没开评）`,
+    errors: (n: number) => `${formatNumber(n)} 项出错`,
    cost: '花费',
    costHint: '助手（建图、挑表、复核）+ 写作（答案运行里的模型调用）+ 结论裁判；算期望不调模型',
    perCase: (v: string) => `每题约 ${v}`,
@@ -221,7 +221,8 @@ function Overview({ run }: { run: EvalRun }) {
      </div>
      <Section title={O.verdicts} attr="verdicts">
        <p>{distText(s.verdicts as Record<string, number>, T.verdict)}</p>
-        {!!s.not_run && <p className="text-dim">{O.notRun(s.not_run)}</p>}
+        {!!s.not_run && <p className="text-dim" data-eval-not-run={s.not_run}>{O.notRun(s.not_run)}</p>}
+        {!!s.errors && <p className="text-dim" data-eval-errors={s.errors}>{O.errors(s.errors)}</p>}
        <p className="text-dim">{O.outcomes}：{distText(s.outcomes as Record<string, number>, T.outcome)}</p>
        {s.text && Object.keys(s.text).length > 0 && <p className="text-dim">{O.text}：{distText(s.text as Record<string, number>, T.textOutcome)}</p>}
      </Section>
@@ -393,11 +393,14 @@ export interface EvalSpend {
 export type EvalStatus = 'queued' | 'running' | 'succeeded' | 'partial' | 'failed' | 'cancelled' | 'interrupted'
 export type EvalTrigger = 'manual' | 'schedule' | 'pair'

+/** 一臂的计数（服务端 runner._counts）：done 是真的跑过的（评完、出错），not_run 是停止、中断、到了上限、止损没跑的，
+ * errors 是出错的（不含没跑的）；done + not_run + running + pending = total */
 export interface EvalCounts {
  total?: number
  done?: number
  running?: number
  pending?: number
+  not_run?: number
  passed?: number
  failed?: number
  ungradable?: number
@@ -428,7 +431,9 @@ export interface EvalSummary {
  done?: number
  cases?: number
  verdicts?: Partial<Record<EvalVerdict, number>>
+  /** 没跑的（停止、中断、到了上限、止损没开评）；出错的另在 errors */
  not_run?: number
+  errors?: number
  accuracy?: EvalRate
  traced_accuracy?: EvalRate
  outcomes?: Partial<Record<EvalOutcome, number>>
@@ -362,7 +362,11 @@ function reply(state, method, path, url, body) {
    }
    if (method === 'POST' && rest === '/stop') {
      write()
-      Object.assign(ev, { status: 'cancelled', running: false, current: [], stop_reason: null,
+      // 服务端（runner._counts、_close）：停下的原因记 stop_reason、不当成出错；没跑的题单独计 not_run，不进 errors，done 不等于 total
+      Object.assign(ev, { status: 'cancelled', running: false, current: [], stop_reason: '评测已停止', error: null,
+        counts: { total: 3, done: 1, not_run: 2, running: 0, pending: 0, passed: 1, failed: 0, ungradable: 0, skipped: 0, errors: 0 },
+        summary: { ...clone(SUMMARY), items: 3, done: 3, verdicts: { pass: 1, fail: 0, ungradable: 0, skipped: 0 }, not_run: 2, errors: 0,
+          accuracy: RATE(1, 1), traced_accuracy: RATE(1, 1) },
        actions: { stop: false, retry: true, rescore: true, pin: false, unpin: false, open: 2, cleaned: false } })
      return { json: ev }
    }
@@ -743,6 +747,11 @@ await section('评测详情：进度、正在做什么、停止和重试、重
  await page.locator('[data-eval-retry]').waitFor()
  check('停止之后：服务端说能重试，给「重试没做完的」，悬停写几项', (await page.locator('[data-eval-retry]').getAttribute('title')).includes('2 项')
    && writes(state, /\/stop$/).length === 1)
+  check('……停下的原因照服务端写「评测已停止」，不当成出错；没评的 2 项单列，不算出错', await until(async () =>
+    (await page.locator('[data-eval-stop-reason]').count()) === 1)
+    && (await text(page, '[data-eval-stop-reason]')).includes('评测已停止') && await page.locator('[data-eval-error]').count() === 0
+    && (await text(page, '[data-eval-not-run]')).startsWith('2 项没评') && await page.locator('[data-eval-errors]').count() === 0,
+  await text(page, '[data-eval-run-head]'))
  await page.locator('[data-eval-retry]').click()
  await until(() => writes(state, /\/retry$/).length === 1)
  check('重试：发了一次', writes(state, /\/retry$/).length === 1)