From 1ad087c387c6ea0c9719811a490ff4d9b53c8db2 Mon Sep 17 00:00:00 2001 From: ThearchyHelios Date: Wed, 21 Oct 2026 10:51:15 +0800 Subject: [PATCH] =?UTF-8?q?=E6=96=87=E6=A1=A3=E8=A1=A5=E5=85=85=EF=BC=9A?= =?UTF-8?q?=E5=A4=8D=E6=A0=B8=E5=A5=97=E4=BB=B6=E9=80=90=E9=A1=B9=E5=86=99?= =?UTF-8?q?=E6=B8=85=EF=BC=88=E5=8F=8C=E7=9B=B2=E5=A4=8D=E7=AE=97=E3=80=81?= =?UTF-8?q?=E5=8F=8D=E6=96=B9=E5=AE=A1=E7=A8=BF=E3=80=81=E5=AE=A1=E6=89=B9?= =?UTF-8?q?=E8=B4=A8=E8=AF=A2=E3=80=81=E6=95=B0=E5=AD=97=E6=8E=A8=E6=BC=94?= =?UTF-8?q?=E6=B2=99=E7=9B=98=E3=80=81=E8=BF=90=E8=A1=8C=E5=AF=B9=E6=AF=94?= =?UTF-8?q?=EF=BC=89=EF=BC=8C=E6=96=B0=E5=A2=9E=E3=80=8C=E6=9F=A5=E8=AF=A2?= =?UTF-8?q?=E5=8C=BA=E9=97=B4=E9=87=8C=E6=B2=A1=E6=9C=89=E6=95=B0=E6=8D=AE?= =?UTF-8?q?=E3=80=8D=E4=B8=80=E8=8A=82=EF=BC=8C=E8=A1=A5=E5=85=A8=E8=AE=BF?= =?UTF-8?q?=E9=97=AE=E5=AF=86=E9=92=A5=E5=88=B0=E6=9C=9F=E6=8F=90=E9=86=92?= =?UTF-8?q?=E5=8F=91=E5=88=B0=E5=93=AA=E9=87=8C=EF=BC=9B=E4=B8=AD=E8=8B=B1?= =?UTF-8?q?=E6=96=87=E5=90=8C=E6=AD=A5?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - evidence.md「复核套件」：开头讲三项共用的只读核查 agent 和「设置 → 偏好设置 → 核查」，再按代码模块文档逐项写解决什么、怎么用、结果、局限；可验证报告的细节并进「导出」一节（只证明内部一致、封存后追加的单列、遮罩、何时能导出），复核套件里链过去 - evidence.md 新增「查询区间里没有数据」：整列是 0 时现查表里实际的日期范围，口径卡标「区间内无数据」、出具降档、写作提醒、问数据按复核重来、助手自查、宁可漏报；data-catalog.md 的数据剖析一节链过去 - schedules.md：访问密钥到期提醒写明个人密钥发本人的个人渠道、服务账号的密钥发全局渠道，以及消息里有什么；定时运行结束后的自动对比链到运行对比 - README 两版：功能列表加复核套件一条，文档索引里证据与出具一行补上新加的两节 scripts/check-docs.py：英文文档中文字符 0 处，相对链接 383 条全通，中英文档 18 对标题层级一致 --- README.md | 3 +- README.zh-CN.md | 3 +- docs/en/data-catalog.md | 1 + docs/en/evidence.md | 88 ++++++++++++++++++++++++++++++++++++-- docs/en/schedules.md | 3 +- docs/zh-CN/data-catalog.md | 1 + docs/zh-CN/evidence.md | 88 ++++++++++++++++++++++++++++++++++++-- docs/zh-CN/schedules.md | 3 +- 8 files changed, 180 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 3bb21a80..dea36913 100644 --- a/README.md +++ b/README.md @@ -43,6 +43,7 @@ English | [Simplified Chinese](README.zh-CN.md) - Clickable evidence: numbers, table and field names, and verbatim knowledge-base quotes all open to their provenance → [Evidence and Issuance](docs/en/evidence.md) - Three-tier issuance: caliber cards plus issuance contracts, checked by value or by citation → [Issuance contracts](docs/en/evidence.md#issuance-contracts-and-tiers) - Sealing and exports: one-click seal verification; self-contained report exports and verifiable reports → [Exports](docs/en/evidence.md#exports) +- Review suite: double-blind recomputation, adversarial review, approval inquiry, the what-if sandbox, run comparison → [Review suite](docs/en/evidence.md#the-review-suite) **Quality** @@ -88,7 +89,7 @@ See [Deployment](docs/en/deployment.md) and [deploy/README.md](deploy/README.md) | **Using and building** | | | [Ask Data and Copilot](docs/en/ask-data.md) | What happens after you ask, output review, record retention, how Copilot builds graphs and looks things up | | [Workflows](docs/en/workflows.md) | The canvas, node types, passing data, human nodes and rejection, agent guardrails, subworkflows, publishing and gates, runs and resuming | -| [Evidence and Issuance](docs/en/evidence.md) | Caliber cards, issuance contracts and tiers, citation markers, claim sentences, sealing and verification, report exports and verifiable reports | +| [Evidence and Issuance](docs/en/evidence.md) | Caliber cards, issuance contracts and tiers, citation markers, claim sentences, no data in the queried range, sealing and verification, report exports and verifiable reports, the review suite | | [Data Sources and Spreadsheet Import](docs/en/data-sources.md) | Connecting databases, read-only guarantees, uploading spreadsheets and recipes, accumulating by period, cross-database merges, masked columns | | [Data Catalog](docs/en/data-catalog.md) | What the catalog records, drafting and review, export and import, data profiling, SQL checks, introspection scope for large databases, schema and value changes | | [Tools, MCP, and the Capability Registry](docs/en/tools.md) | Kinds of tools, approval levels, capability governance, limits on write capabilities, registered endpoints, network rules | diff --git a/README.zh-CN.md b/README.zh-CN.md index 3b0a53a7..2ec3a833 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -43,6 +43,7 @@ - 可点击证据：数字、表名字段名、知识库原话都点得开出处 → [证据与出具](docs/zh-CN/evidence.md) - 三档出具：口径卡加出具契约，按数值回指或按引用核对 → [出具契约](docs/zh-CN/evidence.md#出具契约与三档) - 封存与导出：一键核对封存；自包含的报告导出和可验证报告 → [导出](docs/zh-CN/evidence.md#导出) +- 复核套件：双盲复算、反方审稿、审批质询、数字推演、运行对比 → [复核套件](docs/zh-CN/evidence.md#复核套件) **质量** @@ -88,7 +89,7 @@ conda env create -f environment.yml # 首次：创建 agentlab 环境并安装 | **使用与搭建** | | | [问数据与助手](docs/zh-CN/ask-data.md) | 提问之后发生什么、产出复核、记录保留，助手怎么生成图、怎么按需查看 | | [工作流](docs/zh-CN/workflows.md) | 画布、节点类型、传数据、人工节点与驳回、agent 护栏、子工作流、发布与门禁、运行与断点续跑 | -| [证据与出具](docs/zh-CN/evidence.md) | 口径卡、出具契约与三档、引用标记、结论句、封存与核对、报告导出与可验证报告 | +| [证据与出具](docs/zh-CN/evidence.md) | 口径卡、出具契约与三档、引用标记、结论句、查询区间里没有数据、封存与核对、报告导出与可验证报告、复核套件 | | [数据源与表格导入](docs/zh-CN/data-sources.md) | 接入数据库、只读保证、上传表格与取数配方、按期累积、跨库合并、遮罩列 | | [数据目录](docs/zh-CN/data-catalog.md) | 目录记什么、起草与审阅、导出导入、数据剖析、SQL 检查、大库探查范围、结构与取值变化 | | [工具、MCP 与能力登记](docs/zh-CN/tools.md) | 工具种类、审批档位、能力治理、写入能力的限制、登记端点、网络规则 | diff --git a/docs/en/data-catalog.md b/docs/en/data-catalog.md index 3bfd603f..91547d3e 100644 --- a/docs/en/data-catalog.md +++ b/docs/en/data-catalog.md @@ -121,6 +121,7 @@ Profiling runs read-only queries to check the catalog's inferences and fill in s - The "Checked key count" profiling setting: `DISTINCT … LIMIT` doesn't take a random sample. - On MySQL / MariaDB, profiling notes for text columns state the collation, ENUM columns gain the values that don't appear in the data, and date columns state how many invalid dates were excluded. - The table list has a Profiling column, and the table details show the last profiling result. +- The date ranges profiling records aren't reliable for tables fully reloaded every day, so whether the queried range has data is checked live at run time; the ranges recorded by profiling and value observation serve only as a fallback when the live check fails (see [No data in the queried range](evidence.md#no-data-in-the-queried-range)). By default, it profiles only "tables that need profiling": those never profiled, not finished last time, or that have since gained relationships awaiting verification or new code-value or date columns. On the same data source, drafting and profiling queue up one after the other, and the scope is set from the catalog as it stands when profiling starts, so you can click profile right after drafting finishes. diff --git a/docs/en/evidence.md b/docs/en/evidence.md index a0439a40..27e0bf98 100644 --- a/docs/en/evidence.md +++ b/docs/en/evidence.md @@ -2,7 +2,7 @@ English | [Simplified Chinese](../zh-CN/evidence.md) # Evidence and Issuance -This page covers where the numbers in a report come from and how to check them: caliber cards and issuance contracts, the three issuance tiers, citation markers in the report-writer node, what you see when you click a number, sealing and verification, and the two kinds of export. It is for builders and for people who read reports. The user interface is currently available in Simplified Chinese only. +This page covers where the numbers in a report come from and how to check them: caliber cards and issuance contracts, the three issuance tiers, citation markers in the report-writer node, what happens when the queried range has no data, what you see when you click a number, sealing and verification, the two kinds of export, and the review tools: double-blind recomputation, adversarial review, approval inquiry, the what-if sandbox, and run comparison. It is for builders, approvers, and people who read reports. The user interface is currently available in Simplified Chinese only. ## The basic idea @@ -105,6 +105,18 @@ In the report-writer node (`report`), the model writes only citation markers, an - Tables read back from registered endpoints can serve as evidence too (see [Tools](tools.md#endpoint-reads-can-serve-as-evidence)), and a subworkflow's provenance travels back to the parent graph (see [Workflows](workflows.md#subworkflows)). - A data source can set masked columns (`mask_columns`). For builders, admins, and everyone when login is off, masking only reduces exposure and is not a security boundary; for users with login on, it is a boundary (see [Which roles can see raw values](identity-and-security.md#which-roles-can-see-raw-values)). +## No data in the queried range + +Query a table that keeps only the latest day (fully reloaded every day) for last week and the week before, and whole columns come back as 0. Those zeros don't mean nothing happened; the table simply has no data for that period. Keep computing with them, and the report ends up saying "0 last week" and "0.00% week over week". + +- When a query result has a numeric column that is entirely 0 or empty, the system narrows the SQL to the expression that computes that column, extracts the tables and time conditions it uses, checks on the spot the earliest and latest dates actually present in that column of the table, and compares them with the queried range. It counts as no data only when they don't overlap at all; when a column uses several ranges (this week minus last week), every range must fail to overlap. +- What it finds is written into the query result and the snapshot for people and the model to read: which period was queried, which period the table actually has data for, and that these zeros don't mean the period was really 0. The timeline records a warning, and the query step in the evidence panel lists those columns. +- The caliber card computes as usual; metrics taken from those columns are marked "No data in range", with the reason (columns with data in the same query are unaffected). The issuance contract counts them as gaps and downgrades the issuance, and the same goes for a report that cites such a cell directly. For outputs without an issuance contract, the app's stamp states plainly that these zeros don't mean 0. +- The report-writer node is reminded not to write 0, flat, or 0%, but to state plainly that the table has no data for the period and up to which date data exists. +- In Ask Data, the review records it as a gap worth one rebuild, so the turn can start over per the review with a table that has data for the period (see [Output review](ask-data.md#output-review)). When Copilot checks a graph it is building, it also checks hard-coded SQL; if it finds this problem, it is told to switch tables, or, if there is no such table, to tell the user plainly. +- Double-blind recomputation records inputs on those columns as "Can't recompute". +- It errs on the side of missing cases: the live checks are time-limited (10 seconds each, 20 seconds in total); on a timeout or failure it falls back to the range recorded by profiling or value observation and notes which day's profiling or observation it used (see [Data profiling](data-catalog.md#data-profiling)), and with neither it doesn't judge. SQL it can't parse and date formats it doesn't recognize aren't judged, and masked columns aren't checked. A result with no rows at all is outside its scope. + ## What you see when you click a number ![Evidence panel](../images/evidence-panel.png) @@ -157,11 +169,81 @@ The two exports serve different purposes: to prove nothing was altered, use the **Verifiable report**: Export Verifiable Report under Provenance Details on the Records page (`GET /api/runs/{id}/bundle.html`) produces a seal verification document. -- The recipient recomputes the hashes in the browser without connecting to the service, and checks that no number, document, or run record has been altered. +- The recipient recomputes the hashes in the browser, without connecting to the service or installing anything, and checks that no number, document, or run record has been altered. - The provenance panel for a number likewise shows the caliber, original formula, substituted formula, inputs, and recomputation; the query snapshots that caliber card inputs point to come attached to the file. +- It proves only that the file is internally consistent: numbers, documents, and run records agree with one another and haven't been altered. To check authenticity, compare the seal hash on the page with the seal verification of this run in the system (`GET /api/runs/{id}/verify`); the page says so prominently. +- Events appended after sealing (annotations such as on-demand judging and approval inquiries) are listed separately, with only their type, sequence number, and a content summary, and aren't part of the manifest. +- When a data source has masked columns, query snapshots that can't be shown to leave them untouched, and caliber cards with inputs from masked columns, are given by id only, without content. Masking only reduces exposure and is not a security boundary. +- Export is available only after the run has finished and been sealed. If a run that ended in failure is continued later, it is sealed again, and the exported file no longer matches the seal in the system. Each export records an audit entry, `report.exported`. ## The review suite -Beyond the run itself, there is a set of review tools: double-blind recomputation, adversarial review, approval inquiry, and the what-if sandbox. The adjudication in double-blind recomputation is a human step: it can reject and can take a Reject exit (see [Workflows](workflows.md#rejection)). With multiple processes, the daily spending of the judge and of verification is counted site-wide (see [PostgreSQL and Scaling](postgresql-and-scaling.md#state-shared-across-processes)). +Before a report is issued, you can have another model query the database to check it; afterwards, you can change inputs or pick a baseline to see how the numbers move. + +- Double-blind recomputation, adversarial review, and approval inquiry share one read-only verification agent. It gets only read-only query tools for the data sources, and write statements are refused even on writable sources; cells taken from masked columns are replaced with "Masked" before they reach the model. Numbers it writes itself aren't trusted: the system takes every number by cell reference from the agent's own query snapshots. +- Queries from double-blind recomputation and adversarial review are sealed with the run and can be drilled into from the evidence panel; inquiry queries are listed under the Q&A. None of them enter the report's writing catalog, so writers never cite them. The comparisons from double-blind recomputation, the review records, and the inquiry answers form the Verification Records group on the Artifacts tab of run details. +- Under Settings → Preferences → Verification, the three share a verification model and a daily spending cap ($2 by default), and each has its own switch and caps; anything set on a node takes precedence. When no verification model is set, the evidence judge's model is used. It's better to pick one that differs from the models that write the report and query the data: a model has a hard time spotting its own mistakes. +- Double-blind recomputation and adversarial review follow the verification settings captured when the run started: if the settings change while the run waits on an approval, it resumes with the original ones. With multiple processes, the daily spending of the judge and of verification is counted site-wide (see [PostgreSQL and Scaling](postgresql-and-scaling.md#state-shared-across-processes)). +- The what-if sandbox and run comparison only recompute from the run records: they call no model, don't query business databases, and don't change the original run. The verifiable report is an export; see [Exports](#exports). + +### Double-blind recomputation + +A caliber card's formula is deterministic and can be recomputed; what goes wrong is the numbers fed into it: which table was queried, which day a record is attributed to, whether refunds were excluded. Before issuance, double-blind recomputation hands every input of a metric that comes from a query to a verification agent that can't see the original SQL, has it compute the input independently, then plugs the result back into the original formula and compares it with the original value. + +- Usage: a Double-Blind Recomputation node on the canvas, placed after the caliber card and before the output. Choose the caliber card to recompute (no need when there is only one upstream) and the metrics to recompute (by default, every metric whose inputs come from queries), then write a caliber description for each metric, such as "Sales are counted by payment time, excluding refunds". The blind side can write its queries only from the metric name, this description, and the data catalog, so the more specific the description, the more meaningful the comparison. +- The blind side sees neither the original SQL nor the original result, nor the column names the original flow chose: a column name would reveal which column the original attributed by, the blind side would compute the same number by following it, and traps like order date versus payment date would slip through. +- The two sides agree when they differ by no more than the Allowed Relative Error (one in a million by default; absolute error when the original value is 0). When everything agrees, the run passes. +- On a disagreement, the run pauses, and the approval card explains, input by input, where the two SQL statements differ. For each metric, the reviewer chooses "Use the original flow's result", "Use the blind result", or "Both are wrong". Choosing the blind result doesn't replace the number in the report: the latter two choices, and metrics left without a choice, count as issuance gaps and downgrade the issuance; with When the Original Flow Isn't Adopted set to Fail, the node fails instead. This step can take a Reject exit (see [Rejection](workflows.md#rejection)). +- When the two sides differ in exactly one respect that matches a data catalog fact (filtering on different date columns, one extra condition excluding invalid records, deduplicating or not), the reviewer's choice is written into the catalog as a manually confirmed business date, valid-record condition, or dedup rule; a metric's own filter scope doesn't count. The approval card previews what each choice would write; when in doubt, nothing is written and the reason is stated. +- Recorded as "Can't recompute" and counted as gaps: the blind side hit a cap, errored, or returned no number; an input comes from a masked column; an input lands on a column whose queried range has no data (see [No data in the queried range](#no-data-in-the-queried-range)); the run's time limit was near and there was no time left. When time runs short, select fewer metrics, or put a human approval before the double-blind node (time limits are counted separately before and after an approval). +- Only metrics whose inputs come from query results are recomputed; the node doesn't support retry on failure; and when the blind model is the same as the original flow's, the run warns that the recomputation lacks independence. + +### Adversarial review + +The claim-sentence judge looks only at the evidence the writer attached, but that evidence may have been cherry-picked: sales queried with refunds included, or only one month looked at. Adversarial review works from the other direction and writes its own queries to look for counterexamples. + +- Usage: an Adversarial Review node on the canvas, wired as report writer → adversarial review → output. On a side branch or after the output, the downgrade can't happen, and the canvas warns about it. With only one report upstream, there's no need to choose Report to Review. +- From each report it picks up to 5 key claim sentences (those with a basis, or with comparisons or causation, come first; the same report always yields the same sentences) and sends a read-only verification agent to each in turn, querying the data sources that sentence cites (for a sentence without a basis, the data sources the report cites). The agent first works out "if this sentence were false, what would the data show", then goes looking. +- Each sentence is ruled as holds up, weakened, refuted, or inconclusive, and numbers in the rationale may only cite cells from the agent's own queries. A refuted ruling must cite the agent's own query results in its rationale, or it isn't accepted; a sentence for which no query succeeded is inconclusive. +- When a Claim Is Refuted offers Downgrade Issuance (the default; the refuted claim counts as an issuance gap), Flag Only, and Fail and Stop the Run. +- When there is a human approval downstream, its card carries a summary of the review, stating whether the issuance will be downgraded and how many sentences couldn't be checked. Each sentence's ruling, rationale, and queries are saved in the review record and sealed with the run. +- Default caps: 4 queries and 90 seconds per sentence, with sentences checked one after another. When the run's time limit is near, the remaining sentences aren't started and are recorded as not checked. +- The rulings come from the adversarial model, not from a system check. The canvas and the run both warn when the adversarial model is the same as the writing model. + +### Approval inquiry + +An approval card shows only the report body and a summary of the checks. When an approver has a question about a sentence ("does this number include refunds?", "what if test accounts are excluded?"), there's no need to reject and rerun or to query the database yourself: you can ask on the spot. + +- Usage: on the report awaiting approval, click Inquire at the end of a sentence, select a passage of the text, or use Ask About the Whole Report. The verification agent answers by querying, read-only, the data sources this run used; it usually takes a few dozen seconds. +- The answer leads with the conclusion. Its numbers may only cite cells from the agent's own queries, metrics already in the report, or run inputs, and all of them open to their provenance; bare numbers are flagged. The answer notes that it was given by the verification model and is not a system check. +- The report isn't changed by a single character. The question and answer go into the run record and are sealed with the run. +- Only one inquiry per run is answered at a time, and the run can't be approved or rejected while an answer is pending. +- After sealing, only exploratory runs whose seal verifies can still be questioned: the Q&A is appended after sealing, as an annotation rather than evidence, and the agent queries the data as it is now, which the end of the answer notes. Formal runs accept no more inquiries once sealed. +- Default caps: 6 queries and 120 seconds per inquiry; hitting a cap or a failed query is noted at the end of the answer. With Approval Inquiry · Enabled turned off, there's no entry point, and existing Q&A is still listed. +- Masked values stay out of answers: an assembled answer in which a masked value can still be recognized isn't handed out. + +### What-if sandbox + +Use it when you want to know what a number would become if an input took a different value or a few rows were removed, and which places in the report would change with it, without touching the original run. + +- Usage: in the evidence panel, open a number on a caliber card and click What-If. Change the value of an input, or tick rows to exclude in the query results behind it (ticked rows are treated as absent; rows the metric doesn't use can't be ticked). +- The result shows the new value, the what-if substituted formula, and the change; it lists the places in the report that would change (you can jump to each) and the downstream caliber cards that might be affected: they cite this card's metrics, and the what-if doesn't recompute them. +- It computes the way the caliber card does: the same original formula, the same evaluator, the same rounding. +- A what-if is never an issuance: nothing goes into the run record, the report isn't changed, and only the person who ran it sees the result. +- Runs still in progress or paused on an approval can be explored too, and the page says "The run isn't sealed, and the data may still change"; when seal verification fails, it says the data behind the what-if may have been altered. +- When a what-if isn't possible, the page states why instead of saying "No change": inputs from masked columns show no value and can't be changed (change an input or exclude a row, compare the old and new values, and the masked cell can be worked out), and such a metric's original value also shows as "Masked"; a card that didn't record all its inputs, a container input whose full value can't be recovered (old runs didn't save it), or a query snapshot that can't be read also rules out a what-if. + +### Run comparison + +For two runs of the same workflow (this week's and last week's weekly report, say), it breaks down each metric's change: did the data change, the caliber, or the template or SQL? + +- Usage: Compare With Previous Run in the More menu of run details. When the previous run didn't succeed, the menu says Compare With Last Successful Run; with no default baseline, it's Choose a Baseline…. On the comparison page you can change the baseline or swap the two sides. +- Default baseline: for a run started by a schedule, the last run of the same schedule that finished successfully and was sealed; for other runs, or when the schedule has no earlier successful run, the same within the same workflow and run class. +- Both runs must be finished, sealed, and pass seal verification, and only runs of the same saved workflow can be compared. +- Each metric falls into one class: data only, caliber and data, SQL structure changed (including changed configuration of an upstream node), parameters only (such as different dates), added, removed, incomparable, or unchanged. The caliber covers the formula, the caliber version, decimal places, units, and whether the metric is a ratio; turning thousands separators on or off doesn't count. +- When the caliber and data both changed and a strict split is possible, the caliber effect and the data effect are given separately, and they add up to the total change. +- For metrics in the data-only, parameters-only, and caliber-and-data classes, when attribution is possible, the dimension values (regions, say) with the largest changes are listed, up to 10. Every item traces back to the query snapshots on both sides; queries run by verification take no part in the comparison. +- Values from masked columns never appear in the result, and metrics that use masked columns aren't split. +- When a scheduled run finishes successfully and is sealed, it is compared once automatically with the same schedule's last successful run; the trigger history and notifications show only which run was the baseline and how many metrics fall into each class, never the actual numbers. diff --git a/docs/en/schedules.md b/docs/en/schedules.md index b364170b..af03be40 100644 --- a/docs/en/schedules.md +++ b/docs/en/schedules.md @@ -11,6 +11,7 @@ A schedule starts formal runs on time, and the origin of each run is recorded as - The target can be a workflow, a data source (that is, scheduled maintenance of the data source; see [Data Catalog](data-catalog.md#scheduled-maintenance)), or a test suite (that is, a scheduled evaluation; see [Regression evaluation](evaluation-and-ledger.md#scheduled-evaluations)). - App views prefer to bind to existing schedules. A dashboard reads the schedule's latest sealed formal run, and its stamp names the schedule and the run number. - When a run is rejected by a person, the schedule pushes "Not issued (rejected)". +- When a run finishes successfully and is sealed, it is compared once automatically with the same schedule's last successful run, and the trigger history offers Compare With Previous Run (see [Run comparison](evidence.md#run-comparison)). - Schedules that target a data source or a test suite don't accept webhooks (they return 422); their notifications go through channel subscriptions under Settings → Notifications. - For run concurrency, schedules belong to the scheduled class (see [Run concurrency](postgresql-and-scaling.md#run-concurrency)). - The scheduler loop ticks every `AGENTLAB_SCHEDULER_TICK_S`, 15 seconds by default. @@ -54,7 +55,7 @@ An admin configures SMTP under Settings → Notifications → Mail Service; the | "Data source has changes to handle" | Global channels | Only the source name, the count, and a link; no table names, column names, or values | | "Caliber version bumped, answer numbers changed" | Global channels and the answer owner's personal channels | No numbers; `ledger.recall` is pushed at the same time | | Daily task digest | Personal channels subscribed to it | Sent at `notify.digest_at`, default 09:00 (server time zone) | -| Access key expiry reminder | — | Once 7 days and once 1 day before expiry | +| Access key expiry reminder | Personal keys go to the owner's personal channels subscribed to it; service account keys go to global channels subscribed to it | Once 7 days and once 1 day before expiry; only the key name, its prefix, the owner, the expiry time, and a link to rotate it, never the key itself | The daily digest and access key reminders are sent only when login is on. diff --git a/docs/zh-CN/data-catalog.md b/docs/zh-CN/data-catalog.md index 7891da57..ae8e12da 100644 --- a/docs/zh-CN/data-catalog.md +++ b/docs/zh-CN/data-catalog.md @@ -121,6 +121,7 @@ - 剖析设置里的「核对键值数」：`DISTINCT … LIMIT` 取的不是随机样本。 - MySQL / MariaDB 上，文字列的剖析说明写明排序规则；ENUM 列补上数据里没出现的取值；日期列写明排除了几个无效日期。 - 表清单有「剖析」一列，表详情里有上次剖析的结果。 +- 剖析记下的日期范围对每天整表重灌的表靠不住，所以查询区间里有没有数据是运行时现查的；剖析和取值观察记下的范围只在现查不成时当退路（见[查询区间里没有数据](evidence.md#查询区间里没有数据)）。 缺省只剖析「需要剖析的表」：从没剖析过、上次没做完，或者之后又出现了待核实的关联、新的码值列或日期列的。同一个数据源上，起草和剖析排队串行；范围在开始剖析时按当时的目录定，所以起草完接着点剖析就行。 diff --git a/docs/zh-CN/evidence.md b/docs/zh-CN/evidence.md index 81eb023d..cf7e69ad 100644 --- a/docs/zh-CN/evidence.md +++ b/docs/zh-CN/evidence.md @@ -2,7 +2,7 @@ # 证据与出具 -这篇讲报告里的数字从哪来、怎么核对：口径卡和出具契约、三档出具、报告撰写节点的引用标记、点开一个数能看到什么、封存与核对、两种导出。给搭建者和看报告的人看。 +这篇讲报告里的数字从哪来、怎么核对：口径卡和出具契约、三档出具、报告撰写节点的引用标记、查询区间里没有数据时怎么办、点开一个数能看到什么、封存与核对、两种导出，以及双盲复算、反方审稿、审批质询、数字推演、运行对比这组复核工具。给搭建者、审批人和看报告的人看。 ## 基本思路 @@ -105,6 +105,18 @@ flowchart LR - 登记端点读回的表格也能作证（见[工具](tools.md#端点读取可以作证)）；子工作流的出处一路带回父图（见[工作流](workflows.md#子工作流)）。 - 数据源可以设遮罩列（`mask_columns`）。对搭建者、管理员和未开启登录时的所有人，遮罩只减少暴露，不是安全边界；对开了登录的使用者是边界（见[哪些角色看得到原值](identity-and-security.md#哪些角色看得到原值)）。 +## 查询区间里没有数据 + +一张只存最新一天的表（每天整表重灌），按上周、上上周去查，结果整列是 0。这些 0 不是「没有发生」，是这段时间表里根本没有数据；照算下去，报告就会写出「上周为 0」「环比 0.00%」。 + +- 查询结果里有整列都是 0 或空的数值列时，系统把 SQL 收窄到算出这一列的那一项，取出用到的表和时间条件，当场查这一列在表里实际的最早、最晚日期，和查询的区间比。完全不重叠才算没有数据；一列用到几段区间（本周减上周）时，每一段都不重叠才算。 +- 查出来的照实写进查询结果和快照，给人和模型看：查的是哪段时间、表里实际有哪段时间的数据、这些 0 不代表这段时间真的是 0。时间线上记一条提醒，证据面板的查询步骤列出这几列。 +- 口径卡照算，取自这些列的指标标「区间内无数据」并写明原因（同一条查询里有数的列不受影响）。出具契约把它计入缺口、出具降档，报告直接引用这样的格也一样。没配出具契约的成果，应用的印章上照实写这些 0 不代表真的是 0。 +- 报告撰写节点会收到提醒：不要写成 0、持平或 0%，照实写这段时间表里没有数据、数据只到哪天。 +- 问数据里，复核把它记成值得重建一次的缺口，可以按复核重来，换一张有这段数据的表（见[产出复核](ask-data.md#产出复核)）。助手搭图自查时也查写死的 SQL，查出来就提醒它换表，没有这样的表就照实告诉用户。 +- 双盲复算把落在这些列上的输入记为无法复算。 +- 宁可漏报：现查限时（每条 10 秒，合计 20 秒），超时或失败时退回剖析、取值观察记下的范围，并注明按的是哪天的剖析或取值观察（见[数据剖析](data-catalog.md#数据剖析)），都没有就不判；解析不了的写法、认不出的日期写法不判；遮罩列不查。结果一行都没有的不归这里管。 + ## 点开一个数看到什么 ![证据面板](../images/evidence-panel.png) @@ -157,11 +169,81 @@ flowchart LR **可验证报告**：记录页「凭证详情」里的「导出可验证报告」，接口 `GET /api/runs/{id}/bundle.html`，是一份封存核对件。 -- 收件人不连服务，在浏览器里重算哈希，核对每个数字、文档和运行记录没被改过。 +- 收件人不连服务、不装任何东西，在浏览器里重算哈希，核对每个数字、文档和运行记录没被改过。 - 数字的出处面板同样带口径、原式、代入式、输入和复算；口径卡输入指向的查询快照也随文件附带。 +- 它只证明文件内部一致：数字、文档、运行记录彼此对得上，而且没被改过。真伪要拿页面上的封存哈希和系统里这次运行的封存核对（`GET /api/runs/{id}/verify`）对照，页面上醒目地写着这一点。 +- 封存之后追加的事件（按需裁判、审批质询这类批注）单列，只有类型、序号和内容摘要，不进清单。 +- 数据源设了遮罩时，证明不了没碰遮罩列的查询快照、有输入取自遮罩列的口径卡只给 id，不附内容。遮罩只减少暴露，不是安全边界。 +- 运行结束并封存后才能导出。以失败结束的运行，之后若继续运行会重新封存，导出的文件就和系统里的封存对不上了。 两种导出各记一条审计 `report.exported`。 ## 复核套件 -运行之外还有一组复核工具：双盲复算、反方审稿、审批质询、数字推演沙盘。双盲复算的裁决是人工步骤，可以驳回，也可以接驳回出口（见[工作流](workflows.md#驳回)）。裁判和核查的每日花费在多进程时全站一起算（见 [PostgreSQL 与多进程](postgresql-and-scaling.md#进程间一致的状态)）。 +报告出具之前，可以另请一个模型查库核对；事后也可以改输入、换基准，看数字怎么变。 + +- 双盲复算、反方审稿、审批质询共用一个只读的核查 agent：只给它数据源的只读查询工具，可写的源上写语句也被拒绝；取自遮罩列的格子在交给模型之前就换成「已遮罩」。它自己写的数字不采信，数都由系统按格子引用，从它自己的查询快照里取。 +- 双盲复算和反方审稿的查询随运行封存，证据面板能下钻；质询的查询列在问答下面。这些查询都不进报告的写作目录，写作者不会引用它们。双盲复算的对照、反方审稿记录、质询回答在运行详情「工件」页签的「核查记录」一组。 +- 这三项在「设置 → 偏好设置 → 核查」里共用核查模型和每日金额上限（缺省 $2），各自单独开关、设上限；节点上写了的以节点为准。核查模型没单独配置时用证据裁判的模型；建议另选一个和写报告、查数据都不同的模型，同一模型不易发现自身的错误。 +- 双盲复算和反方审稿按发起运行时的核查设置走：停在审批上期间改了设置，恢复后仍按发起时的。裁判和核查的每日花费在多进程时全站一起算（见 [PostgreSQL 与多进程](postgresql-and-scaling.md#进程间一致的状态)）。 +- 数字推演和运行对比只按运行记录重算，不调用模型、不查业务库，也不改原运行。可验证报告是一种导出，见[导出](#导出)。 + +### 双盲复算 + +口径卡的式子本身是确定的、可以复算，会错的是喂给它的数：查哪张表、按哪天归属、排没排除退款。双盲复算在出具之前，把指标里来自查询的每个输入交给一个看不到原 SQL 的核查 agent 独立算一遍，再代回原式，和原值比。 + +- 用法：画布上的「双盲复算」节点，接在口径卡之后、成果之前。选要复算的口径卡（上游只有一张时不用选）和复算的指标（缺省是输入来自查询的全部指标），再给每个指标写一段口径说明，例如「销售额按支付时间统计，不含退款」。盲算只能依据指标名、这段说明和数据目录写查询，说明越具体，比对越有意义。 +- 盲算看不到原 SQL 和原结果，也看不到原流程取的列名：列名会泄露按哪一列归属，照着算出同一个数，下单日和支付日这类陷阱就漏过去了。 +- 两边相差不超过「允许的相对误差」（缺省百万分之一；原值为 0 时按绝对误差）算一致。全部一致就放行。 +- 有不一致时运行停下，审批卡上逐个输入说清两条 SQL 差在哪，审核人逐个指标选「采用原流程的结果」「采用盲算的结果」或「两边都不对」。选盲算不会替换报告里的数：后两种和没选的都计入出具缺口、出具降档；「未采用原流程时」选「判为失败」时节点失败。这一步可以接驳回出口（见[驳回](workflows.md#驳回)）。 +- 两边只差一处、又正好对得上数据目录的一项事实（按不同的时间列筛、多一个排除无效记录的条件、去不去重）时，审核人的选择写进目录，成为人工确认过的业务日期、有效记录条件或去重规则；指标自己的筛选范围不算。审批卡上预告选哪边会写什么；拿不准就不写，照实说原因。 +- 记为「无法复算」并计入缺口的：盲算触顶、出错或交不出数；输入取自遮罩列；输入落在查询区间里没有数据的列上（见[查询区间里没有数据](#查询区间里没有数据)）；运行时长上限将至、来不及做的。来不及时，可以少选几个指标，或者在双盲复算前加一道人工审批（时限按审批前后分段算）。 +- 只复算输入来自查询结果的指标；节点不支持失败重试；盲算模型和原流程的模型相同时，运行里会提醒复算缺乏独立性。 + +### 反方审稿 + +结论句的裁判只看写作者挂的证据，可证据本身可能是挑过的：只查了含退款的销售额，只看了一个月。反方审稿换个方向，自己写查询去找反例。 + +- 用法：画布上的「反方审稿」节点，按「报告撰写 → 反方审稿 → 成果」接；接在侧枝上或成果之后，降档不会发生，画布会提醒。上游只有一份报告时，不用选「要审的报告」。 +- 每份报告挑最多 5 句关键结论（挂了依据的、带比较或因果的在前；同一份报告每次选出同样的句子），逐句派一个只读的核查 agent，在这句引用的数据源里查（没挂依据的，在报告引用的数据源里查）。它先想清楚「如果这句不成立，数据里会看到什么」，再去找。 +- 每句判为经受住、被削弱、被推翻或无法判断，理由里的数字只能引用它自己查询的格子。判「被推翻」的，理由里必须引用到自己的查询结果，否则不采信；一条查询都没成功的，记为无法判断。 +- 「结论被推翻时」可选出具降档（缺省，被推翻的结论计入出具缺口）、只标注、报错并停止运行。 +- 下游有人工审批时，审批卡上附审稿概况，写明出具会不会降档、几句未能反查。每句的结论、理由和查询存进审稿记录，随运行封存。 +- 上限缺省每句 4 次查询、90 秒，逐句串行。运行时长上限将至时，后面的句子不再开始，记为未能反查。 +- 判断由反方模型给出，不是系统核对。反方模型和写作模型相同时，画布和运行里都会提醒。 + +### 审批质询 + +审批卡上只有报告正文和核对概况。审批人对一句话有疑问时（「这个数含不含退款」「排除测试账号会怎样」），不必驳回重跑，也不必自己去查库，可以当场问。 + +- 用法：在待审的报告上点句末的「质询」、划选一段原文，或者「就整份报告提问」。核查 agent 只读查询这次运行用过的数据源作答，通常要几十秒。 +- 回答先给结论。数字只能引用它自己查询的格子、报告里已有的指标或运行输入，都点得开出处；裸数字会被标出来。回答上注明由核查模型给出，并非系统核对。 +- 报告一个字不改。问答写进运行记录，随运行一起封存。 +- 同一次运行同时只答一条；回答期间不能批准或驳回。 +- 封存之后，只有封存核对一致的探索运行还能问：问答追加在封存之后，是批注不是证据；查的是数据源现在的数据，回答末尾会注明。正式运行封存之后不再追加。 +- 上限缺省每次 6 次查询、120 秒，触顶、查询失败都在回答末尾注明。「审批质询 · 启用」关掉后不给入口，已有的问答照列。 +- 遮罩值不进回答：组装好的回答里还认得出遮罩值的，不交出去。 + +### 数字推演沙盘 + +想知道某个输入换个值、去掉几行数据，这个数会变成多少、报告里哪几处跟着变，又不想动原运行时，用推演。 + +- 用法：在证据面板里点开口径卡上的一个数，点「推演」，改某个输入的值，或者在它背后的查询结果里勾选要排除的行（勾选的行视为不存在；这个指标没用到的行不能勾）。 +- 结果给出推演后的值、推演代入式和变化，列出报告中会变的地方（可以定位过去），以及可能受影响的下游口径卡：它们引用了这张卡的指标，推演不会重算它们。 +- 算法和口径卡相同：同一条原式、同一个求值器、同样的取整。 +- 推演永远不是出具：不写进运行记录，不改报告，结果只给发起推演的人。 +- 还在跑、停在审批上的运行也能推演，页面写明「运行未封存，数据可能还会变」；封存核对没通过的，写明推演所依据的数据可能已被改动。 +- 推演不了的照实写原因，不说「无变化」：取自遮罩列的输入不给值、也不能改（改一个输入、排除一行，新旧一比就能算出遮住的那一格），这样的指标原值也写「已遮罩」；卡上没记全输入、容器输入的完整值找不回来（老运行当时没保存）、查询快照读不出来的，也推演不了。 + +### 运行对比 + +同一个工作流的两次运行（比如本周和上周的周报），把每个指标的变化拆清楚：是数据变了、口径变了，还是模板或 SQL 变了。 + +- 用法：运行详情「更多」菜单里的「与上一次对比」。上一次没成功时，菜单写「与上一次成功的运行对比」；没有默认基准时是「选择基准对比…」。对比页上可以更换基准、互换两边。 +- 默认基准：定时任务发起的运行，取同一定时任务上一次成功结束并封存的运行；不是定时任务发起的，或者同一任务之前没有成功的，取同一工作流、同一运行类别里的。 +- 两次运行都要已结束、封存且封存核对一致；只比同一个已保存的工作流。 +- 每个指标归一类：只数据变、口径与数据都变、SQL 结构变（含上游节点改了配置）、只参数变（如日期不同）、新增、移除、无法比较、未变。口径包括计算式、口径版本、小数位、单位和是不是比率；千分位显示与否不算。 +- 口径与数据都变、又能严格拆分的，分别给出口径效应和数据效应，两者之和等于总变化。 +- 只数据变、只参数变、口径与数据都变的指标，能归因时按维度（比如区域）列出变化最大的前 10 项。每一项都追得到两边的查询快照；核查执行的查询不参与对比。 +- 遮罩列的值不出现在结果里，用到遮罩列的指标不拆分。 +- 定时任务的运行成功结束、封存后，自动和同一任务上一次成功的运行比一次；触发历史和通知里只给基准是哪一次、各类指标各几个，不给具体数字。 diff --git a/docs/zh-CN/schedules.md b/docs/zh-CN/schedules.md index 0542c371..46bc9a89 100644 --- a/docs/zh-CN/schedules.md +++ b/docs/zh-CN/schedules.md @@ -11,6 +11,7 @@ - 目标可以是工作流、数据源（即数据源的定时维护，见[数据目录](data-catalog.md#定时维护)）或评测题集（即定时评测，见[回归评测](evaluation-and-ledger.md#定时评测)）。 - 应用的视图优先绑定已有的定时任务；看板读定时任务最近一次封存的正式运行，印章写明「定时任务「…」第 n 次」。 - 运行被人工驳回时，定时任务推「未出具（被驳回）」。 +- 运行成功结束、封存后，自动和同一定时任务上一次成功的运行对比一次，触发历史里有「与上一次对比」（见[运行对比](evidence.md#运行对比)）。 - 以数据源和题集为目标的定时任务不收 Webhook（回 422），通知走「设置 → 通知」的渠道订阅。 - 在运行并发里，定时任务属于「定时」一类（见[运行并发](postgresql-and-scaling.md#运行并发)）。 - 调度循环每一拍的间隔是 `AGENTLAB_SCHEDULER_TICK_S`，缺省 15 秒。 @@ -54,7 +55,7 @@ | 「数据源有变化要处理」 | 全局渠道 | 只有源名、条数和链接，不放表名、列名和取值 | | 「口径升版，答案的数变了」 | 全局渠道、答案主人的个人渠道 | 不放数字；同时推 `ledger.recall` | | 每天的待办汇总 | 订阅了它的个人渠道 | 按 `notify.digest_at` 发，缺省 09:00（服务端时区） | -| 访问密钥到期提醒 | — | 到期前 7 天、1 天各一次 | +| 访问密钥到期提醒 | 个人密钥发给本人订阅了它的个人渠道；服务账号的密钥发到订阅了它的全局渠道 | 到期前 7 天、1 天各一次；只有密钥名、开头一截、拥有者、到期时间和去轮换的链接，不放密钥本身 | 待办汇总和访问密钥提醒只在开了登录时发。