mirror of
https://github.com/ThearchyHelios/AgentLab.git
synced 2026-11-15 23:41:58 +00:00
ThearchyHelios
271bc6e782
真实模型下问「上周的销售额是多少」，助手写 SUM(amount) AS 销售额, COUNT(*) AS 订单数, MIN / MAX(paid_date) AS 区间首日 /
末日，报告四列都引用了，口径「周销售」只取 SUM(amount)。按「全部引用列」投影后还是 4 列，对不上口径的 1 列，整条比、
四个数都标相关口径，说明整段兜底。
- ledger/match._cells / _cell（algo 4）：整条（含按全部引用列投影）没证明、又引用了这条查询两列以上的，被引用的每一列单独
投影到那一列，和每份口径的每个指标读的那一列（card_reads；口径版本没有口径卡的按取数的每一列）逐个比，仍只走
match_template / explain_sql_diff，上次的投影规则照用：投影不了（分组列、DISTINCT……）、过了时限、不可比的格照用整条
查询的那一个候选。结果记在 queries[].cells {列名: {status, candidates}}，另记 one_row（sql_semantics.one_row：不分组的
聚合，一定只出一行）。整条这一层的结论、候选一字不改；老工件没有 cells，照旧按整条读。
- ledger/labels：单元格按它那一格的结论标（_view）；指标用了同一条查询的好几格时，只有那条查询一定只出一行才把逐格的
结论合起来用（多行的结果里每一列各自对得上，不等于每一行的几列一起对得上）。整题照拍板 L1 不变：每个数都证明来自
同一份口径才等价；一格证明了、别的格只是相关的整题不等价，原因写「有的数和相关口径不一致，不来自口径」（以前一律说
「这次的推理」）。出具入口照 ledger/issue 现有判法只在整题等价时给，没有放宽。
- 逐格证明了的口径各处都算上（labels.candidates_of）：工件 meta 的 calibers（召回按旧版本反查）、ledger/uses 的使用记录、
召回时找原答案对应的那一格、E4；E1 另数各种结论各几格（cells），E2 的兜底比例按标签上给人看的那一份说明数。
- GET /runs/{id}/ledger 的 queries 带上逐格的 cells（遮罩同候选）；证据片段的 ledger 键、轮次摘要的计数本来就按每个数的
标签，数据变了、形状不变。前端只补类型；check-ledger 加一段：同一条查询一格来自口径（正文挂小标记）、一格相关（差在哪
先说多查了几列），整题不等价不给出具。
- 评测逐题比 SQL（eval/score.sql_match）取和期望配上的那一格（score.matched_cells：评分时配上的格），没给时取逐格证明了
的那一格；结果带 column。runner、rescore 传它。
测试：tests/test_ledger_cells.py（同一条查询一列等价、一列不等价各自标对；两份口径只证明了一份标那一份；口径多个指标按列
配对；指标跨两格只在一行的查询上用逐格结论；分组列、DISTINCT 逐格也不投影；照真实例子造的虚构库真跑一遍：销售额标等价
已证明，订单数、首末日标相关，整题不等价、不给出具、不自动出具，证据片段的 ledger 键一致；评测取配上的那一格）。去掉逐格
再跑，10 条失败；去掉「一行才合起来用」、去掉评测取格，各 1 条失败。
727 lines
41 KiB
Python
727 lines
41 KiB
Python
"""评分：拿期望和答案比，给出每个期望的结论和这道题过没过（细案 01 做法 10；评测-D4）。
|
||
|
||
**只读封存，不调模型、不查库。** 答案从作答那次运行封存范围内的报告文档、查询快照里取（api/evidence 的封存读法，和证据
|
||
面板、口径账本同一份）；期望是作答前后算好的那一份（EvalResult.gold，gold.settle 的形状）；文字要点的判断是作答时裁判给的
|
||
（EvalResult.judge，judging.py）。所以可以「按新规则重新评分」：规则改了 SCORER_VERSION 就加一，重评只把这几样再读一遍。
|
||
|
||
**从答案里取数。**
|
||
- 报告文档里引用解析成的数（segment.cite.role=value、status=resolved）：取 cite.value 原值（显示时的换算、取整不影响）。
|
||
引用的是查询的一格时，读那份快照里这一行的非数值列当维度；引用的是口径卡的指标时没有维度。
|
||
- 正文裸写的数（uncited_number）、没有报告时成果里的文字、直接回答的原文：按出具校验同一套认法取（engine/issuance 的
|
||
extract_numbers：千分位、万 / 亿后缀、百分号、中文数字），容差按写出的精度推。叙述里自己算出来的数也在这里——只能算
|
||
「对但没引用」，不算可追溯（叙述层无算术）。
|
||
|
||
**每个期望五种结论**（OUTCOMES）：traced 对且可追溯（引用的格或指标对上了）、untraced 对但没引用（只有正文裸写的对上了）、
|
||
wrong 错、scale 量纲不同（差 10 的整数次幂，比如元和万元、比率和百分数写反了：单列，也不算对）、ungradable 不能评分（期望
|
||
没算出来、作答期间数据变了、答案引用的是遮罩格、封存核对不上）。
|
||
- scalar：数值按「允许的相对误差」比（engine/tolerance.agree，和双盲复算同一个函数）；期望没写容差时，引用的原值按
|
||
DEFAULT_TOLERANCE，裸写的数按它写出的精度（3.5 万 ±500）。维度配对：期望写了 where（或期望那一行本身有维度）时，答案
|
||
那一格所在行的非数值列要有这个取值；答案那一格没有维度（按条件筛过、只剩一个数）不算对不上。期望值是 0–10 的小整数
|
||
又没有维度可配时标「弱匹配」（巧合的数：3 家门店正好等于期望的 3）。答错时「答案里的数」给和这一项配对上的那一格（列名、
|
||
指标、说明同名，或者维度对得上的只有一个数），配不上就不给、原因里照实写配不上，列出答案引用了哪几个数。
|
||
- table：逐行按 keys 的取值配对，每个要比的值同 scalar；ordered 时按答案里各行键值出现的先后核对顺序。
|
||
- set：名单里每一项都要在答案里出现（ordered 时按顺序）；每一项都出现在答案查过的结果里算可追溯。
|
||
- none：期望答「没有数据」：答案里没有数，而且查过的查询里有返回 0 行的（可追溯）；没有数、但也没有一条查询返回 0 行的
|
||
算「对但没引用」。「给了数」只认引用的数值格和叙述里写的数：引用文字格（地区名这种维度）、空值格（NULL）不算——「华中区
|
||
没有数据」引用了地区那一格，说的仍是没有数据。
|
||
- ask：期望反问：作答路径是反问才对。
|
||
- text：文字要点，由裁判判（covered / partial / missing / unjudged），单列「模型判断」，不进准确率（评测-D4）。
|
||
|
||
**题目结论**（verdict）：路径不对（要答出数的题直接回答了、反问了、只建不跑、建图或运行失败）直接不通过、写明原因；
|
||
必需的期望全是 traced / untraced 就通过，另报「可追溯通过」（全是 traced）；有 wrong / scale 的不通过；剩下的（有不能评分的、
|
||
没有错的）不能评分。期望没算出来按 gold 的结论：口径升版待处置是跳过，别的是不能评分——不混进「答错」。被驳回（总方案 X9）
|
||
记路径「被驳回」，不算失败：不能评分。只有文字要点的题不能评分（单列，不计入准确率）。
|
||
|
||
**逐题的 E1**（sql_match）：产出答案的那条查询和期望的取数（SQL 题是期望 SQL，口径题是口径卡的取数节点）用口径账本同一个
|
||
比对函数（ledger/match.compare_queries：先绑参数、再由 explain_sql_diff 判等价）比一次，给出等价与否和差在哪个方面（总方案
|
||
X2：E1 只有一份实现）。哪条是产出答案的：答案引用了就是被引用的；没引用、只查了一条就是那一条（写明依据）；没引用、查了
|
||
几条的不硬比（照实写没法对上是哪条，不计入 E1）。
|
||
"""
|
||
from __future__ import annotations
|
||
|
||
import logging
|
||
import math
|
||
from collections.abc import Mapping, Sequence
|
||
from dataclasses import dataclass, field
|
||
from typing import Any
|
||
|
||
from app.engine.tolerance import agree, is_finite_number
|
||
|
||
logger = logging.getLogger(__name__)
|
||
|
||
#: 评分规则的版本：规则改了就加一，结果里记着按哪一版评的，可以按新规则重评。
|
||
#: 2：期望「没有数据」时引用文字格、空值格不算给了数；判错时「答案里的数」给配对上的那一格；答案没引用查询时逐题 E1
|
||
#: 不硬比第一条查询
|
||
SCORER_VERSION = 2
|
||
|
||
OUTCOMES = ("traced", "untraced", "wrong", "scale", "ungradable")
|
||
OUTCOME_LABEL = {"traced": "对且可追溯", "untraced": "对但没引用", "wrong": "错", "scale": "量纲不同",
|
||
"ungradable": "不能评分"}
|
||
#: 文字要点的判断（裁判给的）
|
||
TEXT_OUTCOMES = ("covered", "partial", "missing", "unjudged")
|
||
TEXT_LABEL = {"covered": "都答到了", "partial": "答到一部分", "missing": "没答到", "unjudged": "没有判"}
|
||
VERDICT_LABEL = {"pass": "通过", "fail": "不通过", "ungradable": "不能评分", "skipped": "跳过"}
|
||
PATH_LABEL = {"reply": "直接回答", "asked": "反问", "ready": "只建图未运行", "build_failed": "建图失败",
|
||
"run_failed": "运行失败", "ran": "运行完成", "rejected": "被驳回"}
|
||
#: 期望没写容差时，引用的原值按这个相对误差比（同双盲复算的缺省）
|
||
DEFAULT_TOLERANCE = 1e-6
|
||
#: 量纲不同：差这几个倍数（元 / 万元、比率 / 百分数……）
|
||
SCALE_FACTORS = (10.0, 100.0, 1000.0, 1e4, 1e5, 1e6, 1e8)
|
||
#: 弱匹配：期望是这么小的整数、又没有维度可配
|
||
WEAK_MAX = 10
|
||
#: 判错、又配不上时原因里列出答案引用的前几个数
|
||
PAIR_SHOWN = 3
|
||
|
||
#: 路径不对时给人看的原因
|
||
WRONG_PATH = {
|
||
"reply": "该查不查：助手直接回答了，没有查数",
|
||
"asked": "该答不答：助手反问了",
|
||
"ready": "只建了图、没有运行",
|
||
"build_failed": "建图失败",
|
||
"run_failed": "运行失败",
|
||
}
|
||
|
||
|
||
# ==========================================================================
|
||
# 从封存的运行里读答案
|
||
# ==========================================================================
|
||
|
||
|
||
def _numbers(text: str) -> list[dict[str, Any]]:
|
||
from app.engine.issuance import _tolerance, extract_numbers
|
||
|
||
out = []
|
||
for token in extract_numbers(text or ""):
|
||
out.append({"raw": token.raw, "value": token.value, "decimals": token.decimals, "percent": token.is_percent,
|
||
"tolerance": _tolerance(token)})
|
||
return out
|
||
|
||
|
||
def _snapshot(artifact: str | None, cache: dict[str, Any]) -> dict[str, Any] | None:
|
||
from app.core import artifact_store
|
||
|
||
if not artifact:
|
||
return None
|
||
if artifact not in cache:
|
||
try:
|
||
content = artifact_store.load(artifact)
|
||
except ValueError:
|
||
content = None
|
||
cache[artifact] = content if isinstance(content, dict) else None
|
||
return cache[artifact]
|
||
|
||
|
||
def _masked_columns(snapshot: Mapping[str, Any]) -> set[str]:
|
||
return {str(c) for c in snapshot.get("masked_at_query") or []}
|
||
|
||
|
||
def _cell_facts(cite: Mapping[str, Any], catalog: Mapping[str, Any], cache: dict[str, Any]) -> dict[str, Any]:
|
||
"""引用的一格：哪份快照、第几行哪一列、这一行的维度（非数值列）、是不是遮罩格。"""
|
||
entry = catalog.get(cite.get("alias") or "") or {}
|
||
locator = cite.get("locator") or {}
|
||
artifact = entry.get("artifact")
|
||
out: dict[str, Any] = {"artifact": artifact, "row": locator.get("row"), "column": locator.get("column"), "dims": {}}
|
||
snapshot = _snapshot(artifact, cache)
|
||
if snapshot is None:
|
||
return out
|
||
columns = [str(c) for c in snapshot.get("columns") or []]
|
||
masked = _masked_columns(snapshot)
|
||
out["masked"] = bool(snapshot.get("masked_whole")) or str(locator.get("column")) in masked
|
||
row_index = locator.get("row")
|
||
rows = snapshot.get("rows") or []
|
||
if isinstance(row_index, int) and 0 <= row_index < len(rows) and isinstance(rows[row_index], (list, tuple)):
|
||
row = rows[row_index]
|
||
out["dims"] = {c: row[i] for i, c in enumerate(columns)
|
||
if i < len(row) and c != locator.get("column") and c not in masked and row[i] is not None
|
||
and not is_finite_number(row[i]) and not isinstance(row[i], bool)}
|
||
return out
|
||
|
||
|
||
@dataclass
|
||
class Answer:
|
||
"""从封存的作答运行里读出来的答案（评分只看这些）。problem 不为空时整份答案读不了（封存核对不上、没有封存）。"""
|
||
|
||
run_id: str | None
|
||
text: str = ""
|
||
values: list[dict[str, Any]] = field(default_factory=list)
|
||
bare: list[dict[str, Any]] = field(default_factory=list)
|
||
queries: list[dict[str, Any]] = field(default_factory=list)
|
||
cited: list[str] = field(default_factory=list)
|
||
stats: dict[str, Any] = field(default_factory=dict)
|
||
tier: str | None = None
|
||
report: str | None = None
|
||
problem: str | None = None
|
||
|
||
def to_dict(self) -> dict[str, Any]:
|
||
return {"run_id": self.run_id, "text": self.text, "values": self.values, "bare": self.bare,
|
||
"queries": self.queries, "cited": self.cited, "stats": self.stats, "tier": self.tier,
|
||
"report": self.report, "problem": self.problem}
|
||
|
||
@classmethod
|
||
def from_dict(cls, raw: Mapping[str, Any]) -> Answer:
|
||
return cls(run_id=raw.get("run_id"), text=str(raw.get("text") or ""), values=list(raw.get("values") or []),
|
||
bare=list(raw.get("bare") or []), queries=list(raw.get("queries") or []),
|
||
cited=list(raw.get("cited") or []), stats=dict(raw.get("stats") or {}), tier=raw.get("tier"),
|
||
report=raw.get("report"), problem=raw.get("problem"))
|
||
|
||
|
||
def _output_text(output: Any) -> str:
|
||
"""成果里给人看的文字（没有报告文档时从这里取数）：字段值里的文字，_ 开头的元数据不算。"""
|
||
if not isinstance(output, Mapping):
|
||
return str(output or "")
|
||
parts = []
|
||
for key, value in output.items():
|
||
if str(key).startswith("_"):
|
||
continue
|
||
if isinstance(value, str):
|
||
parts.append(value)
|
||
elif isinstance(value, (int, float)) and not isinstance(value, bool):
|
||
parts.append(str(value))
|
||
return "\n".join(parts)
|
||
|
||
|
||
async def read_answer(run_id: str | None, *, reply: str | None = None) -> Answer:
|
||
"""读一次作答运行的答案。没有运行（直接回答、反问）的只有原文：reply。"""
|
||
from fastapi import HTTPException
|
||
|
||
from app.api.evidence import _choose, _reports, _sealed
|
||
|
||
if not run_id:
|
||
text = reply or ""
|
||
return Answer(run_id=None, text=text, bare=_numbers(text))
|
||
try:
|
||
sealed = await _sealed(run_id)
|
||
except HTTPException as e:
|
||
return Answer(run_id=run_id, problem=str(e.detail))
|
||
if not sealed.sealed:
|
||
return Answer(run_id=run_id, problem="作答的运行没有封存，答案无从核对")
|
||
if not sealed.trusted:
|
||
return Answer(run_id=run_id, problem="作答的运行封存核对不一致，答案不能当证据")
|
||
cache: dict[str, Any] = {}
|
||
answer = Answer(run_id=run_id)
|
||
for artifact, info in sealed.queries.items():
|
||
snap = _snapshot(artifact, cache)
|
||
if snap is None:
|
||
continue
|
||
rows = snap.get("rows") or []
|
||
answer.queries.append({"artifact": artifact, "node_id": (info or {}).get("node_id"),
|
||
"source": snap.get("source") or (info or {}).get("source"), "sql": snap.get("sql"),
|
||
"columns": [str(c) for c in snap.get("columns") or []],
|
||
"row_count": snap.get("row_count", len(rows)), "truncated": bool(snap.get("truncated")),
|
||
"role": snap.get("role")})
|
||
from app.db.base import SessionLocal
|
||
from app.db.models import Run
|
||
|
||
async with SessionLocal() as session:
|
||
row = await session.get(Run, run_id)
|
||
final = row.output if row is not None else None
|
||
issuance = (final or {}).get("_issuance") if isinstance(final, Mapping) else None
|
||
answer.tier = (issuance or {}).get("tier") if isinstance(issuance, Mapping) else None
|
||
reports = _reports(sealed)
|
||
chosen = _choose(reports, None, sealed) if reports else None
|
||
if chosen is None or chosen.doc is None or chosen.hash_ok is False:
|
||
answer.text = _output_text(final)
|
||
answer.bare = _numbers(answer.text)
|
||
if chosen is not None and chosen.hash_ok is False:
|
||
answer.problem = "报告文档与哈希不一致，疑似被修改，答案不能当证据"
|
||
return answer
|
||
doc = chosen.doc
|
||
catalog = doc.get("catalog") or {}
|
||
answer.report = chosen.node_id
|
||
answer.text = str(doc.get("markdown") or "")
|
||
answer.stats = dict(doc.get("stats") or {})
|
||
from app.engine.evidence import iter_segments
|
||
|
||
cited: list[str] = []
|
||
for seg in iter_segments(doc):
|
||
cite = seg.get("cite") or {}
|
||
if cite.get("role") == "value" and cite.get("status") == "resolved":
|
||
item: dict[str, Any] = {"value": cite.get("value"), "kind": cite.get("kind"), "text": seg.get("text"),
|
||
"segment": seg.get("id"), "alias": cite.get("alias")}
|
||
if cite.get("kind") == "metric":
|
||
entry = catalog.get(cite.get("alias") or "") or {}
|
||
item.update(metric=(cite.get("locator") or {}).get("metric"), artifact=entry.get("artifact"), dims={})
|
||
else:
|
||
item.update(_cell_facts(cite, catalog, cache))
|
||
answer.values.append(item)
|
||
if item.get("artifact"):
|
||
cited.append(item["artifact"])
|
||
elif seg.get("issue") == "uncited_number" or (seg.get("kind") == "number" and not cite):
|
||
answer.bare += _numbers(str(seg.get("text") or ""))
|
||
answer.cited = list(dict.fromkeys(cited))
|
||
return answer
|
||
|
||
|
||
# ==========================================================================
|
||
# 比
|
||
# ==========================================================================
|
||
|
||
|
||
def _close(expected: Any, got: Any, tolerance: float) -> bool:
|
||
if is_finite_number(expected) and is_finite_number(got):
|
||
return agree(float(expected), float(got), tolerance)
|
||
return expected is not None and got is not None and str(expected).strip() == str(got).strip()
|
||
|
||
|
||
def _bare_close(expected: Any, token: Mapping[str, Any], tolerance: float | None) -> bool:
|
||
"""裸写的数对不对得上：容差取期望写的、或者这个数写出的精度（大的那个）。百分数和比率互认。"""
|
||
if not is_finite_number(expected):
|
||
return False
|
||
value = float(token["value"])
|
||
allowed = max(float(token.get("tolerance") or 0), abs(float(expected)) * (tolerance or 0))
|
||
if abs(value - float(expected)) <= allowed + 1e-9:
|
||
return True
|
||
return bool(token.get("percent")) and abs(value - float(expected) * 100) <= allowed + 1e-9
|
||
|
||
|
||
def _scaled(expected: Any, got: float) -> bool:
|
||
"""差 10 的整数次幂（量纲不同）。"""
|
||
if not is_finite_number(expected) or expected == 0 or got == 0:
|
||
return False
|
||
ratio = abs(got / float(expected))
|
||
return any(math.isclose(ratio, f, rel_tol=1e-6) or math.isclose(ratio, 1 / f, rel_tol=1e-6) for f in SCALE_FACTORS)
|
||
|
||
|
||
def _dims_ok(wanted: Mapping[str, Any], dims: Mapping[str, Any]) -> bool:
|
||
"""维度配对：答案那一格所在行有维度时，每个期望的键值都要出现在这一行的非数值列里（按取值认，列名可以不同）。"""
|
||
if not wanted or not dims:
|
||
return True
|
||
shown = {str(v).strip() for v in dims.values()}
|
||
return all(str(v).strip() in shown for v in wanted.values())
|
||
|
||
|
||
def _key_of(exp: Mapping[str, Any], gold: Mapping[str, Any]) -> dict[str, Any]:
|
||
return dict(exp.get("where") or gold.get("dims") or {})
|
||
|
||
|
||
def _score_scalar(exp: Mapping[str, Any], gold: Mapping[str, Any], answer: Answer) -> dict[str, Any]:
|
||
expected = gold.get("value")
|
||
tolerance = exp.get("tolerance")
|
||
key = _key_of(exp, gold)
|
||
out: dict[str, Any] = {"expected": expected}
|
||
if key:
|
||
out["key"] = key
|
||
if is_finite_number(expected) and float(expected).is_integer() and 0 <= float(expected) <= WEAK_MAX and not key:
|
||
out["weak"] = True
|
||
usable = [v for v in answer.values if _dims_ok(key, v.get("dims") or {})]
|
||
masked = [v for v in usable if v.get("masked")]
|
||
clean = [v for v in usable if not v.get("masked")]
|
||
for v in clean:
|
||
if _close(expected, v.get("value"), tolerance if tolerance is not None else DEFAULT_TOLERANCE):
|
||
return {**out, "outcome": "traced", "got": v.get("value"), "cell": _cell_ref(v)}
|
||
for token in answer.bare:
|
||
if _bare_close(expected, token, tolerance):
|
||
return {**out, "outcome": "untraced", "got": token["value"], "raw": token["raw"]}
|
||
if not is_finite_number(expected) and expected is not None and str(expected).strip() and \
|
||
str(expected).strip() in answer.text:
|
||
return {**out, "outcome": "untraced", "got": str(expected)}
|
||
if masked and not clean:
|
||
return {**out, "outcome": "ungradable", "reason": "答案引用的是遮罩的格，不评分"}
|
||
for v in clean:
|
||
if is_finite_number(v.get("value")) and _scaled(expected, float(v["value"])):
|
||
return {**out, "outcome": "scale", "got": v.get("value"), "cell": _cell_ref(v)}
|
||
for token in answer.bare:
|
||
if _scaled(expected, float(token["value"])):
|
||
return {**out, "outcome": "scale", "got": token["value"], "raw": token["raw"]}
|
||
numeric = [v for v in clean if is_finite_number(v.get("value"))]
|
||
if not numeric and not answer.bare:
|
||
return {**out, "outcome": "wrong", "reason": "答案里没有对得上的数"}
|
||
paired = _paired(numeric, exp, gold, key)
|
||
if paired is not None:
|
||
return {**out, "outcome": "wrong", "got": paired.get("value"), "cell": _cell_ref(paired),
|
||
"reason": "答案里的数和期望对不上"}
|
||
if numeric:
|
||
shown = "、".join(f"{_value_name(v)} {_plain(v.get('value'))}" for v in numeric[:PAIR_SHOWN])
|
||
more = f" 等 {len(numeric)} 个" if len(numeric) > PAIR_SHOWN else ""
|
||
return {**out, "outcome": "wrong",
|
||
"reason": f"答案里的数和期望对不上：答案引用的数按列名、维度都配不上这一项（引用了 {shown}{more}）"}
|
||
return {**out, "outcome": "wrong", "reason": "答案里的数和期望对不上"}
|
||
|
||
|
||
def _norm(name: Any) -> str:
|
||
return str(name).strip().strip('"`[]').lower() if name is not None else ""
|
||
|
||
|
||
def _paired(numeric: Sequence[Mapping[str, Any]], exp: Mapping[str, Any], gold: Mapping[str, Any],
|
||
key: Mapping[str, Any]) -> Mapping[str, Any] | None:
|
||
"""答错时答案里和这一项配对上的那一格（「答案里的数」给它）：按名字配——列名（或口径卡的指标）是期望取的那一列、
|
||
那个指标，或者和期望的说明（label）同名；名字配不上时按维度配——期望写了维度、答案里维度对得上的数恰好只有一个。
|
||
都配不上为 None：以前取答案引用的第一个值，退款金额那一项显示成了答案里的订单数，甚至显示成一格文字。"""
|
||
names = {_norm(n) for n in (exp.get("column"), gold.get("column"), exp.get("metric"), gold.get("metric"),
|
||
gold.get("name"), exp.get("label")) if n not in (None, "")}
|
||
named = [v for v in numeric if _norm(v.get("column")) in names or _norm(v.get("metric")) in names]
|
||
if named:
|
||
return named[0]
|
||
if key:
|
||
on_dims = [v for v in numeric if v.get("dims") and _dims_ok(key, v["dims"])]
|
||
if len(on_dims) == 1:
|
||
return on_dims[0]
|
||
return None
|
||
|
||
|
||
def _value_name(value: Mapping[str, Any]) -> str:
|
||
return str(value.get("column") or value.get("metric") or value.get("alias") or "（没有列名）")
|
||
|
||
|
||
def _plain(value: Any) -> str:
|
||
return str(int(value)) if is_finite_number(value) and float(value).is_integer() else str(value)
|
||
|
||
|
||
def _cell_ref(value: Mapping[str, Any]) -> dict[str, Any]:
|
||
return {k: value.get(k) for k in ("artifact", "row", "column", "metric", "segment") if value.get(k) is not None}
|
||
|
||
|
||
def _first_position(text: str, word: Any) -> int:
|
||
return text.find(str(word).strip()) if word is not None and str(word).strip() else -1
|
||
|
||
|
||
def _score_table(exp: Mapping[str, Any], gold: Mapping[str, Any], answer: Answer) -> dict[str, Any]:
|
||
keys = list(gold.get("keys") or [])
|
||
columns = [c for c in gold.get("columns") or [] if c not in keys]
|
||
rows = list(gold.get("rows") or [])
|
||
tolerance = exp.get("tolerance")
|
||
cells: list[str] = []
|
||
for row in rows:
|
||
key = {k: row.get(k) for k in keys}
|
||
for column in columns:
|
||
sub = _score_scalar({"tolerance": tolerance, "where": key}, {"value": row.get(column)}, answer)
|
||
cells.append(sub["outcome"])
|
||
outcome = _combine(cells)
|
||
out: dict[str, Any] = {"rows": len(rows), "cells": dict(_count(cells)), "outcome": outcome}
|
||
if outcome in ("traced", "untraced") and gold.get("ordered"):
|
||
positions = [_first_position(answer.text, row.get(keys[0])) if keys else -1 for row in rows]
|
||
if any(p < 0 for p in positions) or positions != sorted(positions):
|
||
out.update(outcome="wrong", reason="行的先后和期望不一致")
|
||
return out
|
||
|
||
|
||
def _score_set(exp: Mapping[str, Any], gold: Mapping[str, Any], answer: Answer, cache: dict[str, Any]) -> dict[str, Any]:
|
||
values = [v for v in gold.get("values") or [] if v is not None]
|
||
positions = [_first_position(answer.text, v) for v in values]
|
||
missing = [v for v, p in zip(values, positions) if p < 0]
|
||
if missing:
|
||
return {"outcome": "wrong", "missing": missing[:20], "reason": f"答案里少了 {len(missing)} 项"}
|
||
if gold.get("ordered") and positions != sorted(positions):
|
||
return {"outcome": "wrong", "reason": "名单的先后和期望不一致"}
|
||
seen: set[str] = set()
|
||
for query in answer.queries:
|
||
snap = _snapshot(query.get("artifact"), cache) or {}
|
||
masked = _masked_columns(snap)
|
||
columns = [str(c) for c in snap.get("columns") or []]
|
||
for row in snap.get("rows") or []:
|
||
if isinstance(row, (list, tuple)):
|
||
seen |= {str(c).strip() for i, c in enumerate(row) if c is not None and
|
||
(i >= len(columns) or columns[i] not in masked)}
|
||
traced = all(str(v).strip() in seen for v in values)
|
||
return {"outcome": "traced" if traced else "untraced", "items": len(values)}
|
||
|
||
|
||
def _score_none(answer: Answer) -> dict[str, Any]:
|
||
"""期望答「没有数据」。「给了数」只认引用的数值格、叙述里写的数：引用的文字格（地区名这种维度）、空值格（NULL）不是数——
|
||
「华中区（[[v:Q1.r0.地区]]）没有数据」说的正是没有数据，以前因为它引用了一格就判成「给了数」。"""
|
||
numeric = next((v for v in answer.values if is_finite_number(v.get("value"))), None)
|
||
if numeric is not None:
|
||
return {"outcome": "wrong", "got": numeric.get("value"), "cell": _cell_ref(numeric),
|
||
"reason": "期望答「没有数据」，答案里却引用了数"}
|
||
if answer.bare:
|
||
return {"outcome": "wrong", "got": answer.bare[0]["value"], "raw": answer.bare[0]["raw"],
|
||
"reason": "期望答「没有数据」，答案的叙述里却写了数"}
|
||
if any(int(q.get("row_count") or 0) == 0 and q.get("role") != "verify" for q in answer.queries):
|
||
return {"outcome": "traced"}
|
||
return {"outcome": "untraced", "reason": "答案没有给数，但没有一条查询返回 0 行可以说明「没有数据」"}
|
||
|
||
|
||
def _combine(outcomes: Sequence[str]) -> str:
|
||
"""几个结论合成一个：有错就错（全是量纲不同的算量纲不同），有不能评分的不能评分，全可追溯才可追溯。"""
|
||
if not outcomes:
|
||
return "ungradable"
|
||
bad = [o for o in outcomes if o in ("wrong", "scale")]
|
||
if bad:
|
||
return "scale" if all(o == "scale" for o in bad) else "wrong"
|
||
if "ungradable" in outcomes:
|
||
return "ungradable"
|
||
return "traced" if all(o == "traced" for o in outcomes) else "untraced"
|
||
|
||
|
||
def _count(outcomes: Sequence[str]) -> dict[str, int]:
|
||
out: dict[str, int] = {}
|
||
for o in outcomes:
|
||
out[o] = out.get(o, 0) + 1
|
||
return out
|
||
|
||
|
||
def _text_outcome(exp: Mapping[str, Any], judgement: Mapping[str, Any] | None) -> dict[str, Any]:
|
||
"""文字要点：裁判对每条要点的判断（judging.judge_points 的结果），合成一个。"""
|
||
points = list(exp.get("points") or [])
|
||
found = ((judgement or {}).get("expectations") or {}).get(exp["id"]) if judgement else None
|
||
if not found:
|
||
reason = (judgement or {}).get("reason") or "这次评测没有打开结论裁判，文字要点没有判"
|
||
return {"outcome": "unjudged", "points": len(points), "reason": reason}
|
||
verdicts = [p.get("verdict") for p in found.get("points") or []]
|
||
if not verdicts or any(v not in ("covered", "partial", "missing") for v in verdicts):
|
||
return {"outcome": "unjudged", "points": len(points), "reason": found.get("reason") or "裁判没有给出判断"}
|
||
outcome = "covered" if all(v == "covered" for v in verdicts) else \
|
||
"missing" if all(v == "missing" for v in verdicts) else "partial"
|
||
return {"outcome": outcome, "points": len(points), "detail": found.get("points")}
|
||
|
||
|
||
# ==========================================================================
|
||
# 一道题
|
||
# ==========================================================================
|
||
|
||
|
||
@dataclass
|
||
class CaseScore:
|
||
verdict: str
|
||
traced: bool
|
||
reason: str | None
|
||
scores: list[dict[str, Any]]
|
||
#: 口径题选了 dual 时新版那一份期望的结论（旧的那一份就是 scores）
|
||
by_version: list[dict[str, Any]] = field(default_factory=list)
|
||
|
||
def to_dict(self) -> dict[str, Any]:
|
||
return {"verdict": self.verdict, "traced": self.traced, "reason": self.reason, "scores": self.scores,
|
||
"by_version": self.by_version, "scorer_version": SCORER_VERSION}
|
||
|
||
|
||
def _expectation_scores(expectations: Sequence[Mapping[str, Any]], values: Mapping[str, Any], answer: Answer,
|
||
path: str | None, judgement: Mapping[str, Any] | None, cache: dict[str, Any]
|
||
) -> list[dict[str, Any]]:
|
||
out = []
|
||
for exp in expectations:
|
||
kind = exp.get("kind")
|
||
base = {"id": exp["id"], "kind": kind, "required": bool(exp.get("required", True))}
|
||
if exp.get("label"):
|
||
base["label"] = exp["label"]
|
||
if kind == "text":
|
||
out.append({**base, **_text_outcome(exp, judgement)})
|
||
continue
|
||
if kind == "ask":
|
||
ok = path == "asked"
|
||
out.append({**base, "outcome": "traced" if ok else "wrong",
|
||
**({} if ok else {"reason": "期望反问（问法含糊），助手却没有反问"})})
|
||
continue
|
||
gold = values.get(exp["id"])
|
||
if gold is None:
|
||
out.append({**base, "outcome": "ungradable", "reason": "期望没有算出这一项"})
|
||
continue
|
||
if kind == "scalar":
|
||
out.append({**base, **_score_scalar(exp, gold, answer)})
|
||
elif kind == "table":
|
||
out.append({**base, **_score_table(exp, gold, answer)})
|
||
elif kind == "set":
|
||
out.append({**base, **_score_set(exp, gold, answer, cache)})
|
||
elif kind == "none":
|
||
out.append({**base, **_score_none(answer)})
|
||
return out
|
||
|
||
|
||
def _verdict(scores: Sequence[Mapping[str, Any]]) -> tuple[str, bool, str | None]:
|
||
counted = [s for s in scores if s.get("required") and s.get("kind") != "text"]
|
||
if not counted:
|
||
return "ungradable", False, "这道题只有文字要点：由裁判判、单列，不计入准确率"
|
||
outcomes = [s["outcome"] for s in counted]
|
||
if any(o in ("wrong", "scale") for o in outcomes):
|
||
first = next(s for s in counted if s["outcome"] in ("wrong", "scale"))
|
||
return "fail", False, first.get("reason") or OUTCOME_LABEL[first["outcome"]]
|
||
if any(o == "ungradable" for o in outcomes):
|
||
first = next(s for s in counted if s["outcome"] == "ungradable")
|
||
return "ungradable", False, first.get("reason") or OUTCOME_LABEL["ungradable"]
|
||
return "pass", all(o == "traced" for o in outcomes), None
|
||
|
||
|
||
def score_case(case: Mapping[str, Any], gold: Mapping[str, Any] | None, answer: Answer, *, path: str | None,
|
||
path_reason: str | None = None, judgement: Mapping[str, Any] | None = None) -> CaseScore:
|
||
"""一道题的评分。gold 是 gold.Settled.to_dict() 的形状；answer 是 read_answer 读出来的。纯计算。"""
|
||
from app.eval.cases import NEEDS_GOLD
|
||
|
||
expectations = list(case.get("expectations") or [])
|
||
needs_numbers = any(e.get("kind") in NEEDS_GOLD for e in expectations)
|
||
gold = dict(gold or {})
|
||
if needs_numbers and gold.get("status") != "ok":
|
||
verdict = "skipped" if gold.get("status") == "skipped" else "ungradable"
|
||
reason = gold.get("reason") or "期望没有算出来"
|
||
scores = [{"id": e["id"], "kind": e.get("kind"), "required": bool(e.get("required", True)),
|
||
"outcome": "ungradable", "reason": reason} for e in expectations if e.get("kind") != "text"]
|
||
return CaseScore(verdict, False, reason, scores)
|
||
if path == "rejected":
|
||
return CaseScore("ungradable", False, path_reason or "运行被驳回（人的决定，不算失败）", [])
|
||
ask_only = any(e.get("kind") == "ask" for e in expectations)
|
||
if needs_numbers and path != "ran":
|
||
reason = WRONG_PATH.get(path or "", "没有得到答案")
|
||
if path_reason:
|
||
reason = f"{reason}：{path_reason}"
|
||
return CaseScore("fail", False, reason, [{"id": e["id"], "kind": e.get("kind"),
|
||
"required": bool(e.get("required", True)), "outcome": "wrong",
|
||
"reason": reason} for e in expectations if e.get("kind") != "text"])
|
||
if answer.problem and needs_numbers:
|
||
return CaseScore("ungradable", False, answer.problem, [])
|
||
cache: dict[str, Any] = {}
|
||
sets = list(((gold.get("before") or {}).get("sets") or [])) if needs_numbers else [{}]
|
||
first = sets[0] if sets else {}
|
||
scores = _expectation_scores(expectations, (first or {}).get("values") or {}, answer, path, judgement, cache)
|
||
verdict, traced, reason = _verdict(scores)
|
||
if ask_only and path != "asked":
|
||
verdict, traced, reason = "fail", False, "期望反问（问法含糊），助手却没有反问"
|
||
if path in ("build_failed", "run_failed"):
|
||
# 没走到能不能反问那一步（建图、运行失败、超时）：照实写是哪一步、为什么
|
||
reason = WRONG_PATH[path] + (f"：{path_reason}" if path_reason else "")
|
||
by_version = []
|
||
for extra in sets[1:]:
|
||
more = _expectation_scores(expectations, (extra or {}).get("values") or {}, answer, path, judgement, cache)
|
||
v, t, r = _verdict(more)
|
||
by_version.append({"version": (extra or {}).get("version"), "verdict": v, "traced": t, "reason": r,
|
||
"scores": more})
|
||
return CaseScore(verdict, traced, reason, scores, by_version)
|
||
|
||
|
||
# ==========================================================================
|
||
# 逐题的 E1：答案的查询和期望的取数比一次（ledger/match 的比对函数）
|
||
# ==========================================================================
|
||
|
||
|
||
async def gold_calibers(case: Mapping[str, Any], gold: Mapping[str, Any] | None) -> list[Any]:
|
||
"""期望的取数，包成口径账本比对用的 Caliber：SQL 题是一条期望 SQL（按截至时间渲染好的那一条，不带参数），口径题是
|
||
钉住版本里口径卡读的取数节点（带模板参数，由 match_template 绑定）。拿不到的为空。"""
|
||
from app.db.base import SessionLocal
|
||
from app.db.models import DataSource, WorkflowVersion
|
||
from app.eval.gold import QUERY_NODE
|
||
from app.ledger.match import Caliber, Fetch, _table_keys, caliber_of
|
||
|
||
spec = (case.get("gold") or {})
|
||
if not spec or (gold or {}).get("status") not in ("ok", "drifted"):
|
||
return []
|
||
before = (gold or {}).get("before") or {}
|
||
if spec.get("kind") in ("sql", "from_run"):
|
||
sql = str((before.get("rendered") or {}).get("sql") or "")
|
||
async with SessionLocal() as session:
|
||
source = await session.get(DataSource, spec.get("datasource_id"))
|
||
if not sql or source is None:
|
||
return []
|
||
return [Caliber(workflow_id="gold", version=0, level="", graph_hash=None,
|
||
fetches=(Fetch(QUERY_NODE, source.name, sql, _table_keys(sql)),))]
|
||
if spec.get("kind") == "caliber":
|
||
from sqlalchemy import select
|
||
|
||
async with SessionLocal() as session:
|
||
row = (await session.execute(select(WorkflowVersion).where(
|
||
WorkflowVersion.workflow_id == spec.get("workflow_id"),
|
||
WorkflowVersion.version == spec.get("version")))).scalar_one_or_none()
|
||
found = caliber_of(str(spec.get("workflow_id")), row) if row is not None else None
|
||
return [found] if found is not None else []
|
||
return []
|
||
|
||
|
||
async def sql_match(case: Mapping[str, Any], gold: Mapping[str, Any] | None, answer: Answer, *,
|
||
timeout: float = 8.0, cells: Sequence[Any] = ()) -> dict[str, Any] | None:
|
||
"""逐题的 E1：{status: proven|related|none|skipped, query, column?, aspects, diff, reason, cited, basis}。没有作答运行、
|
||
没有期望取数的为 None。比的是哪条查询见 _answering：basis 是 cited（答案引用的）或 only_query（答案没引用、运行只查了
|
||
这一条）。比对在线程里跑（纯计算，和账本一样限时）。
|
||
|
||
和账本同一个口径逐格判（ledger/match._cells）：引用了好几列、整条比不上的查询，每一列各比一次。cells 是评分时和期望
|
||
配上的那几格（各项期望的 cell：{artifact, column}）：取那一格的结论（column 写是哪一列）；没给、配上的格没逐格比过时，
|
||
逐格证明了的取那一格，再不然照旧按整条查询。"""
|
||
import asyncio
|
||
import time
|
||
|
||
from app.api.evidence import _sealed
|
||
from app.ledger.match import _kinds, compare_queries, read_queries, referenced_of
|
||
|
||
if not answer.run_id or answer.problem:
|
||
return None
|
||
calibers = await gold_calibers(case, gold)
|
||
if not calibers:
|
||
return None
|
||
try:
|
||
sealed = await _sealed(answer.run_id)
|
||
except Exception: # noqa: BLE001 - 读不出封存就不比
|
||
return None
|
||
from app.db.base import SessionLocal
|
||
|
||
async with SessionLocal() as session:
|
||
kinds = await _kinds(session)
|
||
|
||
def work() -> list[dict[str, Any]]:
|
||
queries = [q for q in read_queries(sealed, kinds)]
|
||
# 和账本同一个口径：报告引用了查询的哪几列，整条比不等价时按这几列投影再比（ledger/match._projected）。答案的
|
||
# SQL 多查了几列（订单数、日期常量）不该让这一题的 E1 变成「相关」
|
||
return compare_queries(queries, calibers, deadline=time.monotonic() + timeout, referenced=referenced_of(sealed))
|
||
|
||
try:
|
||
rows = await asyncio.wait_for(asyncio.to_thread(work), timeout=timeout + 2)
|
||
except asyncio.TimeoutError:
|
||
return {"status": "skipped", "reason": "比对超过时限，没有做完"}
|
||
except Exception: # noqa: BLE001 - 比对是附加的一步，出错照实记没比
|
||
logger.exception("评测：逐题比 SQL 出错")
|
||
return {"status": "skipped", "reason": "比对出错"}
|
||
pool, basis = _answering(rows, answer)
|
||
if isinstance(pool, dict):
|
||
return pool
|
||
column, cell = _cell_of(pool, cells)
|
||
proven = next((r for r in pool if r.get("status") == "proven"), None)
|
||
chosen = cell or proven or next((r for r in pool if r.get("status") == "related"), None) or \
|
||
next((r for r in pool if r.get("status") == "skipped"), None) or pool[0]
|
||
candidate = next((c for c in chosen.get("candidates") or [] if c.get("status") == chosen.get("status")), None) or \
|
||
next(iter(chosen.get("candidates") or []), None) or {}
|
||
diff = candidate.get("diff") or {}
|
||
# projection 那一条只是说「多查了几列，不影响这个数」，不是差在哪个方面（账本的 E1 差异类型也不数它）
|
||
aspects = sorted({i.get("aspect") for i in diff.get("items") or [] if i.get("aspect") not in (None, "", "projection")})
|
||
sql = next((q.get("sql") for q in answer.queries if q.get("artifact") == chosen.get("artifact")), None)
|
||
return {"status": chosen.get("status"), "query": chosen.get("artifact"), **({"column": column} if column else {}),
|
||
"sql": sql, "aspects": aspects, "diff": diff or None, "via": candidate.get("via"),
|
||
"binding": candidate.get("binding"), "reason": chosen.get("reason") or candidate.get("reason"),
|
||
"cited": basis == "cited", "basis": basis}
|
||
|
||
|
||
def matched_cells(graded: "CaseScore") -> list[dict[str, Any]]:
|
||
"""评分时各项期望和答案里配上的那一格（答对的、量纲不同的、答错时配对上的）：逐题 E1 取它们的逐格结论。"""
|
||
return [dict(item["cell"]) for item in graded.scores
|
||
if isinstance(item, Mapping) and isinstance(item.get("cell"), dict)]
|
||
|
||
|
||
def _cell_of(pool: list[dict[str, Any]], cells: Sequence[Any]) -> tuple[str | None, dict[str, Any] | None]:
|
||
"""逐格比过的格里挑一格：评分时和期望配上的那几格先（其中证明了的先），再不然逐格证明了的那一格。交回 (列名, 那一格的
|
||
结论，带上查询快照 artifact)；没有可挑的为 (None, None)。"""
|
||
hinted = [(str(c.get("artifact")), str(c.get("column"))) for c in cells
|
||
if isinstance(c, Mapping) and c.get("artifact") and c.get("column") is not None]
|
||
found: list[tuple[str, dict[str, Any]]] = []
|
||
for row in pool:
|
||
for name, entry in (row.get("cells") or {}).items() if isinstance(row.get("cells"), dict) else []:
|
||
if isinstance(entry, dict):
|
||
found.append((name, {**entry, "artifact": row.get("artifact"), "_hinted": (row.get("artifact"), name)
|
||
in hinted}))
|
||
ranked = sorted(found, key=lambda f: (not f[1]["_hinted"], f[1].get("status") != "proven"))
|
||
for name, entry in ranked:
|
||
if entry["_hinted"] or entry.get("status") == "proven":
|
||
return name, {k: v for k, v in entry.items() if k != "_hinted"}
|
||
return None, None
|
||
|
||
|
||
def _answering(rows: list[dict[str, Any]], answer: Answer) -> tuple[list[dict[str, Any]] | dict[str, Any], str]:
|
||
"""逐题 E1 拿哪几条查询比：产出答案的那几条。交回 (查询, 依据)，确定不了时交回 (照实的结论, "")。
|
||
|
||
- 答案引用了查询（报告里引用的格、指标所在的查询）：比被引用的那几条（依据 cited）；
|
||
- 没引用、这次运行（核查用的不算）只查了一条：答案的数只可能来自它，按它比（依据 only_query）；
|
||
- 没引用、查了几条：不比——以前取第一条相关的，取到的往往是探索查询（看看有哪些地区那种），说「差在筛选条件」和答案的数
|
||
毫无关系。照实写没法对上是哪条，不计入 E1。"""
|
||
cited = set(answer.cited)
|
||
if cited:
|
||
pool = [r for r in rows if r.get("artifact") in cited]
|
||
if pool:
|
||
return pool, "cited"
|
||
return {"status": "skipped", "cited": True,
|
||
"reason": "答案引用的查询不在可比对的查询里（合并查询不比：它的数都来自它的输入）"}, ""
|
||
roles = {q.get("artifact"): q.get("role") for q in answer.queries}
|
||
pool = [r for r in rows if roles.get(r.get("artifact")) != "verify"]
|
||
if not pool:
|
||
return {"status": "none", "reason": "答案没有查询"}, ""
|
||
if len(pool) > 1:
|
||
return {"status": "skipped", "cited": False,
|
||
"reason": f"答案没有引用查询，这次运行执行了 {len(pool)} 条查询，没法对上是哪条产出了答案，不比"}, ""
|
||
return pool, "only_query"
|
||
|
||
|
||
__all__ = ["Answer", "CaseScore", "DEFAULT_TOLERANCE", "OUTCOMES", "OUTCOME_LABEL", "PATH_LABEL", "SCORER_VERSION",
|
||
"TEXT_LABEL", "TEXT_OUTCOMES", "VERDICT_LABEL", "gold_calibers", "matched_cells", "read_answer",
|
||
"score_case", "sql_match"]
|