定时任务自动对比复核整改(后端):只比成功的运行、按发起先后挑上一次成功的基准、和两边的封存绑定,对比挪到认领之前在锁外做并限时限并发;失败通知的处理行不再重复原因

- 只比成功结束的运行,基准也只取成功的:失败、取消的运行指标不全,和上一次比只会得出「移除 N 个」,拿失败的那次当
  基准每个指标都成了「新增」。基准改成自己查同一定时任务、同一工作流、比这次先发起的运行里成功并封存的最近一次,
  不再用对比接口的 default_base(按结束时间挑:老的失败运行事后被接着跑完,会拿后一次触发当它的「上一次」)。
  中间隔着失败、取消的运行时,通知写「与上一次成功的运行(#短编号,日期)相比」,Fire.compare 多一个 previous
- 和封存绑定:运行的封存要盖住最后一条结束事件才算封存完(sealed_now)。runner 先写终态、再另起会话封存,失败后
  接着跑完的运行在两步之间还带着失败时的清单哈希,以前这一拍会直接认领、按旧封存比、通知里写旧的清单哈希;现在
  等新的封存,等不来就照常推送但不比、不写哈希。停在审批上的通知也不再写失败时留下的哈希。对比工件连同两边的清单
  哈希一起存,触发历史发现任何一边不在终态或清单哈希变了就给 null;对比途中封存变了不记(条件写)
- 时机和时限:锁里只挑出状态有变化的触发,认领挪到对比之后、锁外按条件 UPDATE 做。对比最多等 20 秒(含排队),
  同一时刻最多 2 个;超时只记警告,通知不带对比照常推送。停机时扫描在对比途中被取消,这次触发还没认领,下次照常
  比、照常推(以前先认领、后比,这条通知会永远丢掉)。同一份封存已经比过的直接用记下的结果
- 失败通知:从没开始执行的,原因刚写过,处理行不再让人「查看原因」,任务在排期上写「下一次触发会照常发起」,停用
  的写怎么重新发起;开始执行过的改成「请在 AgentLab 的「记录」中查看失败原因」,和挂起那句指向同一个导航入口
- 新增 test_scheduler_compare_guards.py(封存绑定、时限、取消、并发、复用),test_scheduler_compare.py 补真实缺
  指标的失败运行、上一次失败、补跑老运行几种用例;去掉实现逐项验证过会变红
This commit is contained in:
ThearchyHelios
2026-10-06 08:48:56 +08:00
parent f8ac8dc1d6
commit 45fe01effe
8 changed files with 766 additions and 182 deletions
+149 -47
View File
@@ -1,17 +1,20 @@
"""定时运行结束后的自动对比(app/scheduler/auto_compare.py,由通知扫描 webhook.notify_due 在推送之前触发):
"""定时运行结束后的自动对比(app/scheduler/auto_compare.py,由通知扫描 webhook.notify_due 在认领、推送之前触发):
- 运行进入终态且已封存、同一定时任务有更早的已封存运行时,用运行对比的现成函数(api/compare.py 的 default_base、
compare_runs)和它比一次,整份结果存成工件(kind=run_compare,不挂在运行名下、不进运行的工件列表),工件 id 记进
触发记录的 compare_artifact。基准是同一定时任务的上一次:同一工作流更近的手动运行、别的定时任务的运行都不算
- 这个任务的第一次(只有同一工作流别的运行可比)、还没封存时不比;对比出错(基准的封存核对不一致、意外的异常)只记
日志,通知照常推送、不带对比
- 接口:Fire 多一个 compare: {base_run_id, summary} 或 null,summary 同对比接口的计数;运行又被继续执行、不在终态
时为 null(对比页只认封存的运行)
- 通知:结束、失败的通知多一行「与上一次相比:只数据变 2 个、口径与数据都变 0 个、SQL 结构变 0 个」,配了对外访问
地址时附对比页链接(/runs/compare?target=…&base=…);只给个数,报告、指标、查询里的具体数字一个不出现
- 只比成功结束并封存的运行:失败、取消的指标不全,和上一次比只会得出「移除 N 个」,误导收到通知的人
- 基准是同一定时任务、同一工作流、比这次先发起的运行里成功结束并封存的最近一次:同一工作流更近的手动运行、别的
定时任务的运行都不算;失败的那一次不当基准(拿它当基准每个指标都成了「新增」);按发起先后挑,不按结束时间——
老的失败运行事后被接着跑完,结束得比后一次晚,它的上一次仍是它前面那次。紧挨着的上一次不是成功的时,通知和触发
历史写明比的是哪一次
- 整份结果连同两边的清单哈希存成工件(kind=run_compare,不挂在运行名下、不进运行的工件列表),工件 id 记进触发
记录的 compare_artifact。这个任务的第一次(只有同一工作流别的运行可比)、还没封存时不比;对比出错(基准的封存核对
不一致、意外的异常)只记日志,通知照常推送、不带对比
- 接口:Fire 多一个 compare: {base_run_id, summary, previous} 或 null,summary 同对比接口的计数,previous 说基准是不是
紧挨着的上一次;基准被删、被继续执行(不在终态)时为 null(对比页只认封存的运行)
- 通知:结束的通知多一行「与上一次相比:只数据变 2 个、口径与数据都变 0 个、SQL 结构变 0 个」,配了对外访问地址时附
对比页链接(/runs/compare?target=…&base=…);只给个数,报告、指标、查询里的具体数字一个不出现
两次周报运行用 tests/fixtures/compare_runs.py 的虚构运行(区域、金额、日期都是编的),按 runner 的算法封存,不真的
执行;Webhook 用 httpx.MockTransport 接住,不连外网。
封存绑定、时限、取消这几道保护另见 test_scheduler_compare_guards.py。两次周报运行用 tests/fixtures/compare_runs.py 的
虚构运行(区域、金额、日期都是编的),按 runner 的算法封存,不真的执行;Webhook 用 httpx.MockTransport 接住,不连外网。
"""
from __future__ import annotations
@@ -32,7 +35,7 @@ from app.db.base import SessionLocal
from app.db.models import Artifact, Run, RunEvent, Schedule, ScheduleFire, Setting, Workflow
from app.main import app
from app.scheduler import auto_compare, webhook
from tests.fixtures.compare_runs import NEXT_ROWS, Fake, week
from tests.fixtures.compare_runs import NEXT_ROWS, Fake, region_sql, week
UTC = timezone.utc
T0 = datetime(2026, 9, 7, 1, 0, tzinfo=UTC)
@@ -111,8 +114,9 @@ async def new_schedule(workflow_id: str, *, fmt: str = "generic", url: str | Non
async def persist(fake: Fake, workflow_id: str, *, finished: datetime, schedule_id: str | None = None,
status: str = "succeeded", seal: bool = True) -> str:
"""把一次虚构运行落库:工件进工件库,事件进运行记录,按 runner 的算法封存(同 test_compare_api.py)。"""
status: str = "succeeded", seal: bool = True, seal_upto: int | None = None) -> str:
"""把一次虚构运行落库:工件进工件库,事件进运行记录,按 runner 的算法封存(同 test_compare_api.py)。
seal_upto 只封到第几条事件:失败时封存过、接着跑完又写下了终态,新的封存还没补上的那一刻。"""
for content in fake.store.values():
await put_json(content, kind="compare_fixture")
async with SessionLocal() as session:
@@ -128,11 +132,39 @@ async def persist(fake: Fake, workflow_id: str, *, finished: datetime, schedule_
rows.append((seq, etype, node_id, data))
run.last_seq = len(rows)
if seal:
run.manifest_hash, run.manifest_seq = manifest_hash(rows), rows[-1][0]
covered = [r for r in rows if seal_upto is None or r[0] <= seal_upto]
run.manifest_hash, run.manifest_seq = manifest_hash(covered), covered[-1][0]
await session.commit()
return run.id
async def reseal(run_id: str, fake: Fake, *, status: str, finished: datetime) -> None:
"""一次运行被接着跑完(或者被重新封存):事件换成 fake 的那一份,按新的事件封存,状态、结束时间一并改。"""
for content in fake.store.values():
await put_json(content, kind="compare_fixture")
async with SessionLocal() as session:
await session.execute(delete(RunEvent).where(RunEvent.run_id == run_id))
rows = []
for seq, etype, node_id, data in fake.events:
session.add(RunEvent(run_id=run_id, seq=seq, type=etype, node_id=node_id, ts=float(seq), data=data))
rows.append((seq, etype, node_id, data))
run = await session.get(Run, run_id)
run.last_seq = len(rows)
run.manifest_hash, run.manifest_seq = manifest_hash(rows), rows[-1][0]
run.status, run.finished_at = status, finished
await session.commit()
def failed_week(run_id: str, rows=NEXT_ROWS) -> Fake:
"""在 Agent 节点失败的一次周报:取数做完了,Agent 没交回结果,口径卡没有执行,一个指标都没有。"""
run = Fake(run_id)
run.tool("fetch", region_sql(), rows=rows)
run.add("node.started", "bot", {"node_type": "agent"})
run.add("node.failed", "bot", {"error": "模型接入超时"})
run.add("run.failed", None, {"error": "节点「区域分析」失败"})
return run
async def seal(run_id: str) -> None:
async with SessionLocal() as session:
run = await session.get(Run, run_id)
@@ -206,8 +238,14 @@ async def test_finished_run_is_compared_with_the_previous_run_of_its_schedule(re
from app.api.compare import compare_runs
stored = load(artifact)
assert stored == json.loads(canonical_json(await compare_runs(target_id, base_id)))
assert stored["base"]["id"] == base_id and stored["summary"] == SUMMARY
# 整份对比结果,连同比的时候两边的清单哈希(封存变了,这份结果就不再作数)
assert stored["result"] == json.loads(canonical_json(await compare_runs(target_id, base_id)))
assert stored["result"]["base"]["id"] == base_id and stored["result"]["summary"] == SUMMARY
async with SessionLocal() as session:
runs = await session.scalars(select(Run).where(Run.id.in_((target_id, base_id))))
seals = {r.id: r.manifest_hash for r in runs}
assert (stored["target_manifest"], stored["base_manifest"], stored["previous"]) == (
seals[target_id], seals[base_id], True)
async with SessionLocal() as session:
rows = list((await session.execute(select(Artifact).where(Artifact.id == artifact))).scalars())
# 对比结果不是运行产出的:不挂在运行名下,不进这次运行的工件列表
@@ -215,7 +253,7 @@ async def test_finished_run_is_compared_with_the_previous_run_of_its_schedule(re
assert artifact not in [a["id"] for a in (await client.get(f"/api/runs/{target_id}/artifacts")).json()]
items = {f["id"]: f for f in (await client.get(f"/api/schedules/{sid}/fires")).json()["items"]}
assert items[fire_id]["compare"] == {"base_run_id": base_id, "summary": SUMMARY}
assert items[fire_id]["compare"] == {"base_run_id": base_id, "summary": SUMMARY, "previous": True}
assert items[base_fire]["compare"] is None
[listed] = [s for s in (await client.get("/api/schedules")).json()["items"] if s["id"] == sid]
assert listed["last_fire"]["id"] == fire_id and listed["last_fire"]["compare"]["base_run_id"] == base_id
@@ -223,7 +261,7 @@ async def test_finished_run_is_compared_with_the_previous_run_of_its_schedule(re
[body] = recorder.bodies()
link = f"{BASE_URL}/runs/compare?target={target_id}&base={base_id}"
assert body["event"] == "finished" and LINE in body["text"] and f"查看对比:{link}" in body["text"]
assert body["compare"] == {"base_run_id": base_id, "summary": SUMMARY, "url": link}
assert body["compare"] == {"base_run_id": base_id, "summary": SUMMARY, "previous": True, "url": link}
query = parse_qs(urlsplit(body["compare"]["url"]).query)
assert query == {"target": [target_id], "base": [base_id]}
assert no_numbers(recorder.requests[0].content.decode()) == []
@@ -231,25 +269,79 @@ async def test_finished_run_is_compared_with_the_previous_run_of_its_schedule(re
assert await webhook.notify_due() == 0 and (await fire_row(fire_id)).compare_artifact == artifact
@pytest.mark.parametrize("fmt, url", [("feishu", FEISHU), ("generic", GENERIC)])
async def test_failed_run_notice_carries_the_counts_too(recorder, fmt, url):
"""失败的运行也封存:和上一次比一次,失败通知同样带计数摘要"""
@pytest.mark.parametrize("status, fmt, url", [("failed", "feishu", FEISHU), ("failed", "generic", GENERIC),
("cancelled", "generic", GENERIC)])
async def test_failed_or_cancelled_run_is_not_compared(recorder, client, status, fmt, url):
"""失败、取消的运行指标不全(这次在 Agent 节点失败,口径卡没有执行):和上一次比只会得出「移除 2 个」,收到通知
的人会以为报告删了两个指标。不比:触发记录上没有对比,失败通知不带计数那一行和对比页链接"""
await public_address()
_, base_id, target_id, fire_id = await weekly_pair(target_status="failed", fmt=fmt, url=url)
assert await webhook.notify_due() == 1
assert (await fire_row(fire_id)).compare_artifact
raw = recorder.requests[0].content.decode()
text = json.loads(raw)["content"]["text"] if fmt == "feishu" else json.loads(raw)["text"]
assert "运行失败" in text and LINE in text and f"base={base_id}" in text and f"target={target_id}" in text
assert no_numbers(raw) == []
async def test_cancelled_run_is_compared_but_not_pushed(recorder, client):
"""取消的运行不推送(有人手动停的),对比照样算:触发历史里一样能点「对比上一次」"""
sid, base_id, _, fire_id = await weekly_pair(target_status="cancelled")
assert await webhook.notify_due() == 0 and recorder.requests == []
workflow = await new_workflow()
sid = await new_schedule(workflow, fmt=fmt, url=url)
base_id = await persist(week("r1"), workflow, finished=T0, schedule_id=sid)
await fire_for(sid, base_id, T0 - timedelta(minutes=2), notified="succeeded")
target_id = await persist(failed_week("r2"), workflow, finished=T0 + WEEK, schedule_id=sid, status=status)
fire_id = await fire_for(sid, target_id, T0 + WEEK - timedelta(minutes=2))
# 取消的不推送(有人手动停的)
assert await webhook.notify_due() == (1 if status == "failed" else 0)
assert (await fire_row(fire_id)).compare_artifact is None
items = {f["id"]: f for f in (await client.get(f"/api/schedules/{sid}/fires")).json()["items"]}
assert items[fire_id]["compare"]["base_run_id"] == base_id
assert items[fire_id]["compare"] is None
if status == "failed":
raw = recorder.requests[0].content.decode()
text = json.loads(raw)["content"]["text"] if fmt == "feishu" else json.loads(raw)["text"]
assert "运行失败" in text and f"{BASE_URL}/runs/{target_id}" in text
for absent in ("相比", "移除", "查看对比", "/runs/compare"):
assert absent not in text, text
if fmt == "generic":
assert json.loads(raw)["compare"] is None
async def test_after_a_failed_run_the_base_is_the_last_successful_one(recorder, client):
"""上一次失败了:基准取再往前、成功的那一次——拿失败的那次当基准,每个指标都成了「新增」。紧挨着的上一次不是
它,通知和触发历史写明比的是哪一次(短编号和发起的日期,按定时任务的时区)"""
await public_address()
workflow = await new_workflow()
sid = await new_schedule(workflow)
first = await persist(week("r1"), workflow, finished=T0, schedule_id=sid)
await fire_for(sid, first, T0 - timedelta(minutes=2), notified="succeeded")
failed = await persist(failed_week("r2"), workflow, finished=T0 + WEEK, schedule_id=sid, status="failed")
await fire_for(sid, failed, T0 + WEEK - timedelta(minutes=2), notified="failed")
target_id = await persist(week("r3", NEXT_ROWS), workflow, finished=T0 + 2 * WEEK, schedule_id=sid)
fire_id = await fire_for(sid, target_id, T0 + 2 * WEEK - timedelta(minutes=2))
assert await webhook.notify_due() == 1
stored = load((await fire_row(fire_id)).compare_artifact)
assert stored["result"]["base"]["id"] == first and stored["previous"] is False
[body] = recorder.bodies()
# 第一次发起于北京时间 2026-09-07 08:58
line = f"与上一次成功的运行(#{first[:6]},2026-09-07)相比:只数据变 2 个、口径与数据都变 0 个、SQL 结构变 0 个"
assert line in body["text"] and "新增" not in body["text"], body["text"]
assert f"base={first}" in body["text"] and failed not in body["text"]
assert (body["compare"]["base_run_id"], body["compare"]["previous"]) == (first, False)
items = {f["id"]: f for f in (await client.get(f"/api/schedules/{sid}/fires")).json()["items"]}
assert items[fire_id]["compare"] == {"base_run_id": first, "summary": SUMMARY, "previous": False}
async def test_base_follows_launch_order_not_finish_time(recorder):
"""第 2 周失败、第 3 周照常成功之后,有人把第 2 周接着跑完:它结束得比第 3 周晚,但它的上一次仍是第 1 周。以前
按结束时间挑,拿后一次触发的第 3 周当基准,「与上一次相比」的变化方向是反的。第 3 周已经比过的那一份不受影响"""
workflow = await new_workflow()
sid = await new_schedule(workflow, url=None)
first = await persist(week("r1"), workflow, finished=T0, schedule_id=sid)
await fire_for(sid, first, T0 - timedelta(minutes=2), notified="succeeded")
second = await persist(failed_week("r2"), workflow, finished=T0 + WEEK, schedule_id=sid, status="failed")
second_fire = await fire_for(sid, second, T0 + WEEK - timedelta(minutes=2))
third = await persist(week("r3", NEXT_ROWS), workflow, finished=T0 + 2 * WEEK, schedule_id=sid)
third_fire = await fire_for(sid, third, T0 + 2 * WEEK - timedelta(minutes=2))
await webhook.notify_due()
third_artifact = (await fire_row(third_fire)).compare_artifact
assert load(third_artifact)["result"]["base"]["id"] == first
await reseal(second, week("r2", NEXT_ROWS), status="succeeded", finished=T0 + 2 * WEEK + timedelta(days=1))
await webhook.notify_due()
stored = load((await fire_row(second_fire)).compare_artifact)
assert stored["result"]["base"]["id"] == first and stored["previous"] is True
assert (await fire_row(third_fire)).compare_artifact == third_artifact
async def test_first_run_of_a_schedule_has_no_compare(recorder, client):
@@ -276,7 +368,7 @@ async def test_waits_for_the_seal_before_comparing(recorder):
assert (await fire_row(fire_id)).compare_artifact is None and recorder.requests == []
await seal(target_id)
assert await webhook.notify_due() == 1
assert load((await fire_row(fire_id)).compare_artifact)["base"]["id"] == base_id
assert load((await fire_row(fire_id)).compare_artifact)["result"]["base"]["id"] == base_id
assert LINE in recorder.bodies()[0]["text"]
@@ -327,18 +419,28 @@ async def test_compare_failure_only_logs_and_the_notice_still_goes_out(recorder,
assert "封存核对不一致" in caplog.text
async def test_compare_is_hidden_once_the_run_is_running_again(client):
"""失败后被继续执行(不在终态):对比页只认封存的运行,接口不给 compare;再次结束时重新比"""
sid, base_id, target_id, fire_id = await weekly_pair(target_status="failed", url=None)
async def test_a_failed_run_is_compared_once_it_is_continued_to_success(client):
"""失败时不比;接着跑(不在终态)时也没有;跑完、成功并重新封存之后按新的封存比一次"""
workflow = await new_workflow()
sid = await new_schedule(workflow, url=None)
base_id = await persist(week("r1"), workflow, finished=T0, schedule_id=sid)
await fire_for(sid, base_id, T0 - timedelta(minutes=2), notified="succeeded")
target_id = await persist(failed_week("r2"), workflow, finished=T0 + WEEK, schedule_id=sid, status="failed")
fire_id = await fire_for(sid, target_id, T0 + WEEK - timedelta(minutes=2))
async def shown():
items = (await client.get(f"/api/schedules/{sid}/fires")).json()["items"]
return {f["id"]: f for f in items}[fire_id]
await webhook.notify_due()
assert (await fire_row(fire_id)).compare_artifact
assert (await fire_row(fire_id)).compare_artifact is None and (await shown())["compare"] is None
await set_run(target_id, status="running")
[item] = [f for f in (await client.get(f"/api/schedules/{sid}/fires")).json()["items"] if f["id"] == fire_id]
assert item["run_status"] == "running" and item["compare"] is None
await set_run(target_id, status="succeeded", finished_at=T0 + WEEK + timedelta(hours=1))
await webhook.notify_due()
[item] = [f for f in (await client.get(f"/api/schedules/{sid}/fires")).json()["items"] if f["id"] == fire_id]
assert item["compare"] == {"base_run_id": base_id, "summary": SUMMARY}
item = await shown()
assert item["run_status"] == "running" and item["compare"] is None
await reseal(target_id, week("r2", NEXT_ROWS), status="succeeded", finished=T0 + WEEK + timedelta(hours=1))
await webhook.notify_due()
assert (await shown())["compare"] == {"base_run_id": base_id, "summary": SUMMARY, "previous": True}
async def test_compare_is_hidden_once_the_base_run_is_gone_or_running(client):
@@ -0,0 +1,312 @@
"""定时运行自动对比的几道保护(app/scheduler/auto_compare.py、webhook.notify_due):
- 和封存绑定:运行的封存要盖住它最后一条结束事件才算封存完。runner 先写终态、再另起会话封存,失败后接着跑完的运行
在两步之间还带着失败时的清单哈希——这时去比,只比得到旧封存范围里的事件,通知里的清单哈希也是旧的。等不来新的
封存(两步之间进程没了)时照常推送,不带对比、不写旧的哈希
- 对比结果记下两边的清单哈希:任何一边后来变了,触发历史就不再给这份对比(悬停的计数和点进去的对比页对不上);
对比途中封存变了,结果不记
- 时限:对比放在认领之前、锁外做,最多等 WAIT_S 秒,同一时刻最多 SLOTS 个。慢的对比不压住这条通知,也不压住别的
定时任务的通知;扫描在对比途中被取消(停机),这次触发还没认领,下次照常比、照常推。同一份封存已经比过的(比完
还没推送就停机了),下次直接用记下的结果
运行、任务名、金额都是虚构的(tests/fixtures/compare_runs.py),不真的执行;Webhook 用 httpx.MockTransport 接住。
"""
from __future__ import annotations
import asyncio
import logging
import time
from datetime import datetime, timedelta, timezone
import httpx
import pytest
from httpx import ASGITransport, AsyncClient
from sqlalchemy import select
from app import scheduler
from app.api import compare as compare_api
from app.core.artifact_store import load
from app.core.config import Settings
from app.db.base import SessionLocal
from app.db.models import Run, RunEvent
from app.main import app
from app.scheduler import auto_compare, webhook
from tests.fixtures.compare_runs import BASE_ROWS, NEXT_ROWS, Fake, default_metrics, region_sql, week
from tests.test_scheduler_compare import ( # noqa: F401 - clean、recorder 是夹具,按名字引入
BASE_URL, LINE, T0, WEEK, Recorder, clean, fire_for, fire_row, new_schedule, new_workflow, persist,
public_address, recorder, reseal, seal, set_run, weekly_pair,
)
UTC = timezone.utc
@pytest.fixture
async def client():
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as c:
yield c
@pytest.fixture
def slow_compare(monkeypatch):
"""对比要 5 秒:真的对比之前先等着。返回原来的函数,用例里可以换回去"""
real = compare_api.compare_runs
async def slow(*args, **kwargs):
await asyncio.sleep(5)
return await real(*args, **kwargs)
monkeypatch.setattr(compare_api, "compare_runs", slow)
return real
def failed_then_continued(run_id: str, rows=NEXT_ROWS) -> tuple[Fake, int]:
"""在 Agent 节点失败、之后从断点接着跑完的一次周报。返回 (运行, 失败时封存到第几条事件)。"""
run = Fake(run_id)
sql = region_sql()
fetch = run.tool("fetch", sql, rows=rows)
run.add("node.started", "bot", {"node_type": "agent"})
run.add("node.failed", "bot", {"error": "模型接入超时"})
run.add("run.failed", None, {"error": "节点「区域分析」失败"})
failed_at = len(run.events)
run.add("run.resumed", None, {"from": ["bot"], "message": "从「区域分析」继续运行"})
bot = run.agent("bot", sql, rows=rows)
run.card(default_metrics(fetch, bot, rows, rows))
run.add("run.finished", None, {"output": {}})
return run, failed_at
async def previous_week(*, url: str | None = None):
"""一个定时任务和它上一次成功、已推送过的周报。返回 (工作流, 任务, 上一次)。"""
workflow = await new_workflow()
sid = await new_schedule(workflow, url=url) if url else await new_schedule(workflow)
base_id = await persist(week("r1"), workflow, finished=T0, schedule_id=sid)
await fire_for(sid, base_id, T0 - timedelta(minutes=2), notified="succeeded")
return workflow, sid, base_id
async def seal_of(run_id: str) -> str | None:
async with SessionLocal() as session:
return (await session.get(Run, run_id)).manifest_hash
# --------------------------------------------------------------------------
# 和封存绑定
# --------------------------------------------------------------------------
async def test_waits_for_the_seal_of_this_ending_not_the_failed_one(recorder):
"""失败后接着跑完:终态和结束时间已经写下,封存还是失败时那一份(只盖到 run.failed)。这一拍不比、不推,新的
封存补上之后再比,比的是完整的一次,通知里的清单哈希是新的"""
await public_address()
workflow, sid, base_id = await previous_week()
fake, failed_at = failed_then_continued("r2")
target_id = await persist(fake, workflow, finished=datetime.now(UTC), schedule_id=sid, seal_upto=failed_at)
fire_id = await fire_for(sid, target_id, T0 + WEEK - timedelta(minutes=2))
stale = await seal_of(target_id)
assert await webhook.notify_due() == 0
assert recorder.requests == [] and (await fire_row(fire_id)).compare_artifact is None
await seal(target_id)
fresh = await seal_of(target_id)
assert fresh != stale
assert await webhook.notify_due() == 1
stored = load((await fire_row(fire_id)).compare_artifact)
assert (stored["target_manifest"], stored["result"]["base"]["id"]) == (fresh, base_id)
[body] = recorder.bodies()
assert LINE in body["text"] and body["manifest_hash"] == fresh[:12]
async def test_a_seal_that_never_catches_up_is_not_used(recorder):
"""新的封存一直没补上(写完终态、封存之前进程没了):等过时限照常推送,但不比,也不写失败时那份旧的清单哈希"""
await public_address()
workflow, sid, _ = await previous_week()
fake, failed_at = failed_then_continued("r2")
target_id = await persist(fake, workflow, finished=datetime.now(UTC), schedule_id=sid, seal_upto=failed_at)
fire_id = await fire_for(sid, target_id, T0 + WEEK - timedelta(minutes=2))
later = datetime.now(UTC) + timedelta(seconds=webhook.SEAL_WAIT_S + 5)
assert await webhook.notify_due(later) == 1
[body] = recorder.bodies()
assert (body["event"], body["manifest_hash"], body["compare"]) == ("finished", None, None)
assert "清单哈希" not in body["text"] and "相比" not in body["text"]
assert (await fire_row(fire_id)).compare_artifact is None
async def test_approval_notice_does_not_carry_the_seal_of_an_earlier_failure(recorder):
"""失败后接着跑、停在审批上:运行上还留着失败时的清单哈希,那不是这一次的封存,「等待审批」里不写"""
workflow, sid, _ = await previous_week()
fake, failed_at = failed_then_continued("r2")
target_id = await persist(fake, workflow, finished=T0 + WEEK, schedule_id=sid, status="interrupted",
seal_upto=failed_at)
await set_run(target_id, finished_at=None)
async with SessionLocal() as session:
from app.db.models import Approval
session.add(Approval(run_id=target_id, node_id="card", title="审核周报口径", status="pending"))
await session.commit()
await fire_for(sid, target_id, T0 + WEEK - timedelta(minutes=2))
assert await webhook.notify_due() == 1
[body] = recorder.bodies()
assert (body["event"], body["manifest_hash"]) == ("awaiting_approval", None)
assert "清单哈希" not in body["text"]
@pytest.mark.parametrize("side", ["target", "base"])
async def test_fire_history_drops_a_compare_whose_seal_changed(client, side):
"""比过之后哪一边被重新封存(内容变了):记下的计数已经不是对比页现场算出来的那样,触发历史不再给这份对比"""
sid, base_id, target_id, fire_id = await weekly_pair(url=None)
await webhook.notify_due()
async def shown():
items = (await client.get(f"/api/schedules/{sid}/fires")).json()["items"]
return {f["id"]: f for f in items}[fire_id]["compare"]
assert (await shown())["base_run_id"] == base_id
# 换成和另一边一样的数:现场再比就是「未变」
run_id, rows = (target_id, BASE_ROWS) if side == "target" else (base_id, NEXT_ROWS)
async with SessionLocal() as session:
finished = (await session.get(Run, run_id)).finished_at
await reseal(run_id, week("rx", rows), status="succeeded", finished=finished)
assert await shown() is None
# 对比页现场算出来的和记下的确实不一样
assert (await compare_api.compare_runs(target_id, base_id))["summary"]["unchanged"] == 2
async def test_result_is_dropped_when_a_seal_changes_mid_compare(recorder, monkeypatch):
"""对比途中基准被重新封存:比出来的是旧封存的结果,不记进触发记录、不写进通知"""
await public_address()
_, base_id, _, fire_id = await weekly_pair()
real = compare_api.compare_runs
async def resealing(run_id, other):
out = await real(run_id, other)
await set_run(base_id, manifest_hash="ab" * 32)
return out
monkeypatch.setattr(compare_api, "compare_runs", resealing)
assert await webhook.notify_due() == 1
assert (await fire_row(fire_id)).compare_artifact is None
[body] = recorder.bodies()
assert body["compare"] is None and "相比" not in body["text"]
# --------------------------------------------------------------------------
# 时限、取消、并发
# --------------------------------------------------------------------------
async def test_a_slow_compare_is_cut_off_and_the_notice_goes_out(recorder, monkeypatch, caplog, slow_compare):
"""对比超过时限:通知不等它,照常推送、不带对比,记一条警告"""
monkeypatch.setattr(auto_compare, "WAIT_S", 0.3, raising=False)
await public_address()
_, _, target_id, fire_id = await weekly_pair()
began = time.monotonic()
with caplog.at_level(logging.WARNING, logger="app.scheduler"):
assert await webhook.notify_due() == 1
assert time.monotonic() - began < 2
[body] = recorder.bodies()
assert (body["event"], body["compare"]) == ("finished", None) and "相比" not in body["text"]
assert f"{BASE_URL}/runs/{target_id}" in body["text"]
assert (await fire_row(fire_id)).compare_artifact is None
assert "超过" in caplog.text and target_id in caplog.text
async def test_scan_cancelled_mid_compare_pushes_next_time(recorder, monkeypatch, slow_compare):
"""停机时通知扫描只等 1.5 秒就被取消:撞在对比途中的那次触发还没认领,下次启动照常比、照常推。以前先认领、后比,
取消之后这条「运行结束」再也不会推"""
await public_address()
_, _, _, fire_id = await weekly_pair()
with pytest.raises(TimeoutError):
await asyncio.wait_for(webhook.notify_due(), timeout=0.5)
assert (await fire_row(fire_id)).notified_status is None and recorder.requests == []
monkeypatch.setattr(compare_api, "compare_runs", slow_compare)
assert await webhook.notify_due() == 1
[body] = recorder.bodies()
assert LINE in body["text"]
assert (await fire_row(fire_id)).notified_status == "succeeded"
async def test_a_slow_compare_does_not_hold_other_schedules(monkeypatch, slow_compare):
"""任务 A 的对比慢:调度循环里通知扫描同一时刻只跑一份,以前 A 比多久,之后才结束的任务 B(第一次,不用比)
的通知就要等多久。现在 A 的对比等过时限就不等了,B 的通知在下一拍推出"""
rec = Recorder()
monkeypatch.setattr(webhook, "_transport", httpx.MockTransport(rec))
monkeypatch.setattr(webhook, "RETRY_DELAYS", (0, 0, 0))
monkeypatch.setattr(auto_compare, "WAIT_S", 0.5, raising=False)
await weekly_pair()
on = Settings(_env_file=None, scheduler=True)
assert await scheduler.start(interval=0.05, config=on) is True
try:
await asyncio.sleep(0.2)
workflow = await new_workflow("仓库日报")
sid = await new_schedule(workflow, name="仓库日报 · 每天")
run_id = await persist(week("b1"), workflow, finished=T0, schedule_id=sid)
await fire_for(sid, run_id, T0 - timedelta(minutes=2))
began = time.monotonic()
while not any(b.get("run_id") == run_id for b in rec.bodies()) and time.monotonic() - began < 8:
await asyncio.sleep(0.02)
waited = time.monotonic() - began
finally:
await scheduler.stop()
assert any(b.get("run_id") == run_id for b in rec.bodies())
assert waited < 2, f"任务 B 的通知等了 {waited:.2f} 秒"
async def test_compares_take_turns(monkeypatch):
"""同一拍里好几个任务都要比:同一时刻最多 SLOTS 个,不一起占满线程池(接口请求也要用)"""
monkeypatch.setattr(auto_compare, "SLOTS", 2, raising=False)
real = compare_api.compare_runs
active = peak = 0
async def counted(*args, **kwargs):
nonlocal active, peak
active += 1
peak = max(peak, active)
try:
await asyncio.sleep(0.2)
return await real(*args, **kwargs)
finally:
active -= 1
monkeypatch.setattr(compare_api, "compare_runs", counted)
fires = [(await weekly_pair(url=None))[3] for _ in range(4)]
await webhook.notify_due()
assert peak == 2
assert all([(await fire_row(f)).compare_artifact for f in fires])
async def test_a_compare_already_made_for_this_seal_is_reused(recorder, monkeypatch):
"""比完、还没认领就停机了,结果已经记在触发记录上:下次直接用,不再算一遍"""
await public_address()
_, _, target_id, fire_id = await weekly_pair()
assert await auto_compare.compare_fire(fire_id, target_id) is not None
artifact = (await fire_row(fire_id)).compare_artifact
assert artifact
calls = 0
async def counted(*args, **kwargs):
nonlocal calls
calls += 1
raise RuntimeError("不该再算一遍")
monkeypatch.setattr(compare_api, "compare_runs", counted)
auto_compare._BRIEFS.clear()
assert await webhook.notify_due() == 1
assert calls == 0 and LINE in recorder.bodies()[0]["text"]
assert (await fire_row(fire_id)).compare_artifact == artifact
async def test_seal_covers_the_last_ending():
"""封存盖没盖住最后一条结束事件:老运行(没有 manifest_seq)只看有没有哈希"""
workflow = await new_workflow()
fake, failed_at = failed_then_continued("r9")
partial = await persist(fake, workflow, finished=T0, seal_upto=failed_at)
full = await persist(failed_then_continued("r8")[0], workflow, finished=T0)
legacy = await persist(week("r7"), workflow, finished=T0)
await set_run(legacy, manifest_seq=None)
unsealed = await persist(week("r6"), workflow, finished=T0, seal=False)
async with SessionLocal() as session:
got = {rid: await auto_compare.sealed_now(session, await session.get(Run, rid))
for rid in (partial, full, legacy, unsealed)}
assert await session.scalar(select(RunEvent.seq).where(
RunEvent.run_id == partial, RunEvent.type == "run.finished")) > failed_at
assert got == {partial: False, full: True, legacy: True, unsealed: False}
+17 -11
View File
@@ -1,9 +1,10 @@
"""定时任务通知(app/scheduler/webhook.py)的两条规则:
- 失败通知的「处理」行不许诺能继续运行:服务重启时还在排队、从没开始执行的运行(被改成失败、没有开始时间)没有
断点,「修正后可继续运行」是错的。按有没有开始执行分开写——没开始的另写一行原因(没有可继续的进度),处理一律是
中性的「请在运行记录中查看原因」;任务还在排期上时补一句「下一次触发会照常发起」:失败的运行不挡之后的触发,
和挂起的那种(之后的触发都会跳过)说清区别
断点,「修正后可继续运行」是错的。按有没有开始执行分开写——没开始的另写一行原因(没有可继续的进度),处理行不再
让人去「查看原因」(原因刚写过),任务还在排期上时只写「下一次触发会照常发起」,不在排期上时写怎么重新发起;开始
执行过的写「请在 AgentLab 的「记录」中查看失败原因」(和挂起那句一样指到导航里的「记录」),任务还在排期上时补
「下一次触发会照常发起」:失败的运行不挡之后的触发,和挂起的那种(之后的触发都会跳过)说清区别
- 扫描窗口(SCAN_DAYS)之外的触发:停在审批、有待审批的运行不受窗口限制——创建 30 天以后才停到审批上的照样推送
「等待审批」;运行记录在窗口内有变化的(等了一个多月的审批批完、跑完)照样推送结束。早就结束、记录多日没有变化
的旧触发仍不扫
@@ -27,6 +28,8 @@ from app.scheduler import webhook
UTC = timezone.utc
GENERIC = "https://hooks.example.com/agentlab/inbox-3e9d"
NOT_STARTED = "运行在开始执行前就已中止(通常是服务重启时仍在排队),没有可继续的进度"
LOOK = "请在 AgentLab 的「记录」中查看失败原因"
RERUN = "如需重新发起,请在 AgentLab 的「定时任务」中点击「立即运行」"
class Recorder:
@@ -106,12 +109,15 @@ async def notified(fire_id: str) -> str | None:
def test_a_run_that_never_started_is_not_promised_a_continue():
"""服务重启时还在排队的运行没有断点:不写「修正后可继续运行」,另写一行原因说没有可继续的进度"""
"""服务重启时还在排队的运行没有断点:不写「修正后可继续运行」,另写一行原因说没有可继续的进度。处理行不再让人
去「查看原因」(原因刚写过):还在排期上的只说下一次照常,不在排期上的说怎么重新发起"""
_, lines, _ = webhook.message(notice(started_at=None, next_fire=True))
shown = dict(lines)
assert shown["原因"] == NOT_STARTED
assert shown["处理"] == "请在 AgentLab 的运行记录中查看原因;下一次触发会照常发起"
assert not any("继续运行" in value for _, value in lines), lines
assert shown["处理"] == "下一次触发会照常发起"
assert not any("继续运行" in value or "查看" in value for _, value in lines), lines
_, lines, _ = webhook.message(notice(started_at=None, next_fire=False))
assert dict(lines)["处理"] == RERUN
def test_a_failed_run_gets_a_neutral_action_without_promising_a_continue():
@@ -120,14 +126,14 @@ def test_a_failed_run_gets_a_neutral_action_without_promising_a_continue():
_, lines, _ = webhook.message(notice(error_node="汇总库存", next_fire=True))
shown = dict(lines)
assert shown["失败节点"] == "「汇总库存」" and "原因" not in shown
assert shown["处理"] == "请在 AgentLab 的运行记录中查看原因;下一次触发会照常发起"
assert shown["处理"] == f"{LOOK};下一次触发会照常发起"
assert "修正后可继续运行" not in webhook._text(notice(error_node="汇总库存"))
def test_next_fire_is_only_promised_while_the_schedule_is_on():
"""任务已暂停(或往后不再触发)时不说「下一次触发会照常发起」"""
_, lines, _ = webhook.message(notice(next_fire=False))
assert dict(lines)["处理"] == "请在 AgentLab 的运行记录中查看原因"
assert dict(lines)["处理"] == LOOK
async def test_restart_failure_notice_end_to_end(recorder):
@@ -140,12 +146,12 @@ async def test_restart_failure_notice_end_to_end(recorder):
by_fire = {b["fire_id"]: b for b in recorder.bodies()}
on, off = by_fire[on_fire], by_fire[off_fire]
assert (on["event"], on["status"], on["reason"]) == ("failed", "failed", NOT_STARTED)
assert "处理:请在 AgentLab 的运行记录中查看原因;下一次触发会照常发起" in on["text"]
assert "处理:请在 AgentLab 的运行记录中查看原因\n" in off["text"] + "\n" and "照常发起" not in off["text"]
assert "处理:下一次触发会照常发起" in on["text"]
assert f"处理:{RERUN}" in off["text"] and "照常发起" not in off["text"]
for body in (on, off):
assert "继续运行" not in body["text"], body["text"]
# 运行记录上的原始报错不进通知
assert "请重新发起运行" not in body["text"]
assert "没有可恢复的进度" not in body["text"]
# --------------------------------------------------------------------------
+1 -1
View File
@@ -179,7 +179,7 @@ def test_failed_run_text_points_to_the_node_without_repeating_itself():
shown = dict(lines)
assert title.endswith("运行失败") and shown["状态"] == "失败"
# 处理只说去哪里看,不许诺能继续运行(没有断点的运行续不了,见 test_scheduler_notice_rules.py)
assert shown["失败节点"] == "「查询门店销售」" and shown["处理"] == "请在 AgentLab 的运行记录中查看原因"
assert shown["失败节点"] == "「查询门店销售」" and shown["处理"] == "请在 AgentLab 的「记录」中查看失败原因"
# 不再写「原因:运行失败,详情请查看运行记录」这种同义反复
assert "原因" not in shown