mirror of
https://github.com/ThearchyHelios/AgentLab.git
synced 2026-11-20 16:52:02 +00:00
定时任务自动对比复核整改(后端):只比成功的运行、按发起先后挑上一次成功的基准、和两边的封存绑定,对比挪到认领之前在锁外做并限时限并发;失败通知的处理行不再重复原因
- 只比成功结束的运行,基准也只取成功的:失败、取消的运行指标不全,和上一次比只会得出「移除 N 个」,拿失败的那次当 基准每个指标都成了「新增」。基准改成自己查同一定时任务、同一工作流、比这次先发起的运行里成功并封存的最近一次, 不再用对比接口的 default_base(按结束时间挑:老的失败运行事后被接着跑完,会拿后一次触发当它的「上一次」)。 中间隔着失败、取消的运行时,通知写「与上一次成功的运行(#短编号,日期)相比」,Fire.compare 多一个 previous - 和封存绑定:运行的封存要盖住最后一条结束事件才算封存完(sealed_now)。runner 先写终态、再另起会话封存,失败后 接着跑完的运行在两步之间还带着失败时的清单哈希,以前这一拍会直接认领、按旧封存比、通知里写旧的清单哈希;现在 等新的封存,等不来就照常推送但不比、不写哈希。停在审批上的通知也不再写失败时留下的哈希。对比工件连同两边的清单 哈希一起存,触发历史发现任何一边不在终态或清单哈希变了就给 null;对比途中封存变了不记(条件写) - 时机和时限:锁里只挑出状态有变化的触发,认领挪到对比之后、锁外按条件 UPDATE 做。对比最多等 20 秒(含排队), 同一时刻最多 2 个;超时只记警告,通知不带对比照常推送。停机时扫描在对比途中被取消,这次触发还没认领,下次照常 比、照常推(以前先认领、后比,这条通知会永远丢掉)。同一份封存已经比过的直接用记下的结果 - 失败通知:从没开始执行的,原因刚写过,处理行不再让人「查看原因」,任务在排期上写「下一次触发会照常发起」,停用 的写怎么重新发起;开始执行过的改成「请在 AgentLab 的「记录」中查看失败原因」,和挂起那句指向同一个导航入口 - 新增 test_scheduler_compare_guards.py(封存绑定、时限、取消、并发、复用),test_scheduler_compare.py 补真实缺 指标的失败运行、上一次失败、补跑老运行几种用例;去掉实现逐项验证过会变红
This commit is contained in:
@@ -1,17 +1,20 @@
|
||||
"""定时运行结束后的自动对比(app/scheduler/auto_compare.py,由通知扫描 webhook.notify_due 在推送之前触发):
|
||||
"""定时运行结束后的自动对比(app/scheduler/auto_compare.py,由通知扫描 webhook.notify_due 在认领、推送之前触发):
|
||||
|
||||
- 运行进入终态且已封存、同一定时任务有更早的已封存运行时,用运行对比的现成函数(api/compare.py 的 default_base、
|
||||
compare_runs)和它比一次,整份结果存成工件(kind=run_compare,不挂在运行名下、不进运行的工件列表),工件 id 记进
|
||||
触发记录的 compare_artifact。基准是同一定时任务的上一次:同一工作流更近的手动运行、别的定时任务的运行都不算
|
||||
- 这个任务的第一次(只有同一工作流别的运行可比)、还没封存时不比;对比出错(基准的封存核对不一致、意外的异常)只记
|
||||
日志,通知照常推送、不带对比
|
||||
- 接口:Fire 多一个 compare: {base_run_id, summary} 或 null,summary 同对比接口的计数;运行又被继续执行、不在终态
|
||||
时为 null(对比页只认封存的运行)
|
||||
- 通知:结束、失败的通知多一行「与上一次相比:只数据变 2 个、口径与数据都变 0 个、SQL 结构变 0 个」,配了对外访问
|
||||
地址时附对比页链接(/runs/compare?target=…&base=…);只给个数,报告、指标、查询里的具体数字一个不出现
|
||||
- 只比成功结束并封存的运行:失败、取消的指标不全,和上一次比只会得出「移除 N 个」,误导收到通知的人
|
||||
- 基准是同一定时任务、同一工作流、比这次先发起的运行里成功结束并封存的最近一次:同一工作流更近的手动运行、别的
|
||||
定时任务的运行都不算;失败的那一次不当基准(拿它当基准每个指标都成了「新增」);按发起先后挑,不按结束时间——
|
||||
老的失败运行事后被接着跑完,结束得比后一次晚,它的上一次仍是它前面那次。紧挨着的上一次不是成功的时,通知和触发
|
||||
历史写明比的是哪一次
|
||||
- 整份结果连同两边的清单哈希存成工件(kind=run_compare,不挂在运行名下、不进运行的工件列表),工件 id 记进触发
|
||||
记录的 compare_artifact。这个任务的第一次(只有同一工作流别的运行可比)、还没封存时不比;对比出错(基准的封存核对
|
||||
不一致、意外的异常)只记日志,通知照常推送、不带对比
|
||||
- 接口:Fire 多一个 compare: {base_run_id, summary, previous} 或 null,summary 同对比接口的计数,previous 说基准是不是
|
||||
紧挨着的上一次;基准被删、被继续执行(不在终态)时为 null(对比页只认封存的运行)
|
||||
- 通知:结束的通知多一行「与上一次相比:只数据变 2 个、口径与数据都变 0 个、SQL 结构变 0 个」,配了对外访问地址时附
|
||||
对比页链接(/runs/compare?target=…&base=…);只给个数,报告、指标、查询里的具体数字一个不出现
|
||||
|
||||
两次周报运行用 tests/fixtures/compare_runs.py 的虚构运行(区域、金额、日期都是编的),按 runner 的算法封存,不真的
|
||||
执行;Webhook 用 httpx.MockTransport 接住,不连外网。
|
||||
封存绑定、时限、取消这几道保护另见 test_scheduler_compare_guards.py。两次周报运行用 tests/fixtures/compare_runs.py 的
|
||||
虚构运行(区域、金额、日期都是编的),按 runner 的算法封存,不真的执行;Webhook 用 httpx.MockTransport 接住,不连外网。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -32,7 +35,7 @@ from app.db.base import SessionLocal
|
||||
from app.db.models import Artifact, Run, RunEvent, Schedule, ScheduleFire, Setting, Workflow
|
||||
from app.main import app
|
||||
from app.scheduler import auto_compare, webhook
|
||||
from tests.fixtures.compare_runs import NEXT_ROWS, Fake, week
|
||||
from tests.fixtures.compare_runs import NEXT_ROWS, Fake, region_sql, week
|
||||
|
||||
UTC = timezone.utc
|
||||
T0 = datetime(2026, 9, 7, 1, 0, tzinfo=UTC)
|
||||
@@ -111,8 +114,9 @@ async def new_schedule(workflow_id: str, *, fmt: str = "generic", url: str | Non
|
||||
|
||||
|
||||
async def persist(fake: Fake, workflow_id: str, *, finished: datetime, schedule_id: str | None = None,
|
||||
status: str = "succeeded", seal: bool = True) -> str:
|
||||
"""把一次虚构运行落库:工件进工件库,事件进运行记录,按 runner 的算法封存(同 test_compare_api.py)。"""
|
||||
status: str = "succeeded", seal: bool = True, seal_upto: int | None = None) -> str:
|
||||
"""把一次虚构运行落库:工件进工件库,事件进运行记录,按 runner 的算法封存(同 test_compare_api.py)。
|
||||
seal_upto 只封到第几条事件:失败时封存过、接着跑完又写下了终态,新的封存还没补上的那一刻。"""
|
||||
for content in fake.store.values():
|
||||
await put_json(content, kind="compare_fixture")
|
||||
async with SessionLocal() as session:
|
||||
@@ -128,11 +132,39 @@ async def persist(fake: Fake, workflow_id: str, *, finished: datetime, schedule_
|
||||
rows.append((seq, etype, node_id, data))
|
||||
run.last_seq = len(rows)
|
||||
if seal:
|
||||
run.manifest_hash, run.manifest_seq = manifest_hash(rows), rows[-1][0]
|
||||
covered = [r for r in rows if seal_upto is None or r[0] <= seal_upto]
|
||||
run.manifest_hash, run.manifest_seq = manifest_hash(covered), covered[-1][0]
|
||||
await session.commit()
|
||||
return run.id
|
||||
|
||||
|
||||
async def reseal(run_id: str, fake: Fake, *, status: str, finished: datetime) -> None:
|
||||
"""一次运行被接着跑完(或者被重新封存):事件换成 fake 的那一份,按新的事件封存,状态、结束时间一并改。"""
|
||||
for content in fake.store.values():
|
||||
await put_json(content, kind="compare_fixture")
|
||||
async with SessionLocal() as session:
|
||||
await session.execute(delete(RunEvent).where(RunEvent.run_id == run_id))
|
||||
rows = []
|
||||
for seq, etype, node_id, data in fake.events:
|
||||
session.add(RunEvent(run_id=run_id, seq=seq, type=etype, node_id=node_id, ts=float(seq), data=data))
|
||||
rows.append((seq, etype, node_id, data))
|
||||
run = await session.get(Run, run_id)
|
||||
run.last_seq = len(rows)
|
||||
run.manifest_hash, run.manifest_seq = manifest_hash(rows), rows[-1][0]
|
||||
run.status, run.finished_at = status, finished
|
||||
await session.commit()
|
||||
|
||||
|
||||
def failed_week(run_id: str, rows=NEXT_ROWS) -> Fake:
|
||||
"""在 Agent 节点失败的一次周报:取数做完了,Agent 没交回结果,口径卡没有执行,一个指标都没有。"""
|
||||
run = Fake(run_id)
|
||||
run.tool("fetch", region_sql(), rows=rows)
|
||||
run.add("node.started", "bot", {"node_type": "agent"})
|
||||
run.add("node.failed", "bot", {"error": "模型接入超时"})
|
||||
run.add("run.failed", None, {"error": "节点「区域分析」失败"})
|
||||
return run
|
||||
|
||||
|
||||
async def seal(run_id: str) -> None:
|
||||
async with SessionLocal() as session:
|
||||
run = await session.get(Run, run_id)
|
||||
@@ -206,8 +238,14 @@ async def test_finished_run_is_compared_with_the_previous_run_of_its_schedule(re
|
||||
from app.api.compare import compare_runs
|
||||
|
||||
stored = load(artifact)
|
||||
assert stored == json.loads(canonical_json(await compare_runs(target_id, base_id)))
|
||||
assert stored["base"]["id"] == base_id and stored["summary"] == SUMMARY
|
||||
# 整份对比结果,连同比的时候两边的清单哈希(封存变了,这份结果就不再作数)
|
||||
assert stored["result"] == json.loads(canonical_json(await compare_runs(target_id, base_id)))
|
||||
assert stored["result"]["base"]["id"] == base_id and stored["result"]["summary"] == SUMMARY
|
||||
async with SessionLocal() as session:
|
||||
runs = await session.scalars(select(Run).where(Run.id.in_((target_id, base_id))))
|
||||
seals = {r.id: r.manifest_hash for r in runs}
|
||||
assert (stored["target_manifest"], stored["base_manifest"], stored["previous"]) == (
|
||||
seals[target_id], seals[base_id], True)
|
||||
async with SessionLocal() as session:
|
||||
rows = list((await session.execute(select(Artifact).where(Artifact.id == artifact))).scalars())
|
||||
# 对比结果不是运行产出的:不挂在运行名下,不进这次运行的工件列表
|
||||
@@ -215,7 +253,7 @@ async def test_finished_run_is_compared_with_the_previous_run_of_its_schedule(re
|
||||
assert artifact not in [a["id"] for a in (await client.get(f"/api/runs/{target_id}/artifacts")).json()]
|
||||
|
||||
items = {f["id"]: f for f in (await client.get(f"/api/schedules/{sid}/fires")).json()["items"]}
|
||||
assert items[fire_id]["compare"] == {"base_run_id": base_id, "summary": SUMMARY}
|
||||
assert items[fire_id]["compare"] == {"base_run_id": base_id, "summary": SUMMARY, "previous": True}
|
||||
assert items[base_fire]["compare"] is None
|
||||
[listed] = [s for s in (await client.get("/api/schedules")).json()["items"] if s["id"] == sid]
|
||||
assert listed["last_fire"]["id"] == fire_id and listed["last_fire"]["compare"]["base_run_id"] == base_id
|
||||
@@ -223,7 +261,7 @@ async def test_finished_run_is_compared_with_the_previous_run_of_its_schedule(re
|
||||
[body] = recorder.bodies()
|
||||
link = f"{BASE_URL}/runs/compare?target={target_id}&base={base_id}"
|
||||
assert body["event"] == "finished" and LINE in body["text"] and f"查看对比:{link}" in body["text"]
|
||||
assert body["compare"] == {"base_run_id": base_id, "summary": SUMMARY, "url": link}
|
||||
assert body["compare"] == {"base_run_id": base_id, "summary": SUMMARY, "previous": True, "url": link}
|
||||
query = parse_qs(urlsplit(body["compare"]["url"]).query)
|
||||
assert query == {"target": [target_id], "base": [base_id]}
|
||||
assert no_numbers(recorder.requests[0].content.decode()) == []
|
||||
@@ -231,25 +269,79 @@ async def test_finished_run_is_compared_with_the_previous_run_of_its_schedule(re
|
||||
assert await webhook.notify_due() == 0 and (await fire_row(fire_id)).compare_artifact == artifact
|
||||
|
||||
|
||||
@pytest.mark.parametrize("fmt, url", [("feishu", FEISHU), ("generic", GENERIC)])
|
||||
async def test_failed_run_notice_carries_the_counts_too(recorder, fmt, url):
|
||||
"""失败的运行也封存:和上一次比一次,失败通知同样带计数摘要"""
|
||||
@pytest.mark.parametrize("status, fmt, url", [("failed", "feishu", FEISHU), ("failed", "generic", GENERIC),
|
||||
("cancelled", "generic", GENERIC)])
|
||||
async def test_failed_or_cancelled_run_is_not_compared(recorder, client, status, fmt, url):
|
||||
"""失败、取消的运行指标不全(这次在 Agent 节点失败,口径卡没有执行):和上一次比只会得出「移除 2 个」,收到通知
|
||||
的人会以为报告删了两个指标。不比:触发记录上没有对比,失败通知不带计数那一行和对比页链接"""
|
||||
await public_address()
|
||||
_, base_id, target_id, fire_id = await weekly_pair(target_status="failed", fmt=fmt, url=url)
|
||||
assert await webhook.notify_due() == 1
|
||||
assert (await fire_row(fire_id)).compare_artifact
|
||||
raw = recorder.requests[0].content.decode()
|
||||
text = json.loads(raw)["content"]["text"] if fmt == "feishu" else json.loads(raw)["text"]
|
||||
assert "运行失败" in text and LINE in text and f"base={base_id}" in text and f"target={target_id}" in text
|
||||
assert no_numbers(raw) == []
|
||||
|
||||
|
||||
async def test_cancelled_run_is_compared_but_not_pushed(recorder, client):
|
||||
"""取消的运行不推送(有人手动停的),对比照样算:触发历史里一样能点「对比上一次」"""
|
||||
sid, base_id, _, fire_id = await weekly_pair(target_status="cancelled")
|
||||
assert await webhook.notify_due() == 0 and recorder.requests == []
|
||||
workflow = await new_workflow()
|
||||
sid = await new_schedule(workflow, fmt=fmt, url=url)
|
||||
base_id = await persist(week("r1"), workflow, finished=T0, schedule_id=sid)
|
||||
await fire_for(sid, base_id, T0 - timedelta(minutes=2), notified="succeeded")
|
||||
target_id = await persist(failed_week("r2"), workflow, finished=T0 + WEEK, schedule_id=sid, status=status)
|
||||
fire_id = await fire_for(sid, target_id, T0 + WEEK - timedelta(minutes=2))
|
||||
# 取消的不推送(有人手动停的)
|
||||
assert await webhook.notify_due() == (1 if status == "failed" else 0)
|
||||
assert (await fire_row(fire_id)).compare_artifact is None
|
||||
items = {f["id"]: f for f in (await client.get(f"/api/schedules/{sid}/fires")).json()["items"]}
|
||||
assert items[fire_id]["compare"]["base_run_id"] == base_id
|
||||
assert items[fire_id]["compare"] is None
|
||||
if status == "failed":
|
||||
raw = recorder.requests[0].content.decode()
|
||||
text = json.loads(raw)["content"]["text"] if fmt == "feishu" else json.loads(raw)["text"]
|
||||
assert "运行失败" in text and f"{BASE_URL}/runs/{target_id}" in text
|
||||
for absent in ("相比", "移除", "查看对比", "/runs/compare"):
|
||||
assert absent not in text, text
|
||||
if fmt == "generic":
|
||||
assert json.loads(raw)["compare"] is None
|
||||
|
||||
|
||||
async def test_after_a_failed_run_the_base_is_the_last_successful_one(recorder, client):
|
||||
"""上一次失败了:基准取再往前、成功的那一次——拿失败的那次当基准,每个指标都成了「新增」。紧挨着的上一次不是
|
||||
它,通知和触发历史写明比的是哪一次(短编号和发起的日期,按定时任务的时区)"""
|
||||
await public_address()
|
||||
workflow = await new_workflow()
|
||||
sid = await new_schedule(workflow)
|
||||
first = await persist(week("r1"), workflow, finished=T0, schedule_id=sid)
|
||||
await fire_for(sid, first, T0 - timedelta(minutes=2), notified="succeeded")
|
||||
failed = await persist(failed_week("r2"), workflow, finished=T0 + WEEK, schedule_id=sid, status="failed")
|
||||
await fire_for(sid, failed, T0 + WEEK - timedelta(minutes=2), notified="failed")
|
||||
target_id = await persist(week("r3", NEXT_ROWS), workflow, finished=T0 + 2 * WEEK, schedule_id=sid)
|
||||
fire_id = await fire_for(sid, target_id, T0 + 2 * WEEK - timedelta(minutes=2))
|
||||
|
||||
assert await webhook.notify_due() == 1
|
||||
stored = load((await fire_row(fire_id)).compare_artifact)
|
||||
assert stored["result"]["base"]["id"] == first and stored["previous"] is False
|
||||
[body] = recorder.bodies()
|
||||
# 第一次发起于北京时间 2026-09-07 08:58
|
||||
line = f"与上一次成功的运行(#{first[:6]},2026-09-07)相比:只数据变 2 个、口径与数据都变 0 个、SQL 结构变 0 个"
|
||||
assert line in body["text"] and "新增" not in body["text"], body["text"]
|
||||
assert f"base={first}" in body["text"] and failed not in body["text"]
|
||||
assert (body["compare"]["base_run_id"], body["compare"]["previous"]) == (first, False)
|
||||
items = {f["id"]: f for f in (await client.get(f"/api/schedules/{sid}/fires")).json()["items"]}
|
||||
assert items[fire_id]["compare"] == {"base_run_id": first, "summary": SUMMARY, "previous": False}
|
||||
|
||||
|
||||
async def test_base_follows_launch_order_not_finish_time(recorder):
|
||||
"""第 2 周失败、第 3 周照常成功之后,有人把第 2 周接着跑完:它结束得比第 3 周晚,但它的上一次仍是第 1 周。以前
|
||||
按结束时间挑,拿后一次触发的第 3 周当基准,「与上一次相比」的变化方向是反的。第 3 周已经比过的那一份不受影响"""
|
||||
workflow = await new_workflow()
|
||||
sid = await new_schedule(workflow, url=None)
|
||||
first = await persist(week("r1"), workflow, finished=T0, schedule_id=sid)
|
||||
await fire_for(sid, first, T0 - timedelta(minutes=2), notified="succeeded")
|
||||
second = await persist(failed_week("r2"), workflow, finished=T0 + WEEK, schedule_id=sid, status="failed")
|
||||
second_fire = await fire_for(sid, second, T0 + WEEK - timedelta(minutes=2))
|
||||
third = await persist(week("r3", NEXT_ROWS), workflow, finished=T0 + 2 * WEEK, schedule_id=sid)
|
||||
third_fire = await fire_for(sid, third, T0 + 2 * WEEK - timedelta(minutes=2))
|
||||
await webhook.notify_due()
|
||||
third_artifact = (await fire_row(third_fire)).compare_artifact
|
||||
assert load(third_artifact)["result"]["base"]["id"] == first
|
||||
|
||||
await reseal(second, week("r2", NEXT_ROWS), status="succeeded", finished=T0 + 2 * WEEK + timedelta(days=1))
|
||||
await webhook.notify_due()
|
||||
stored = load((await fire_row(second_fire)).compare_artifact)
|
||||
assert stored["result"]["base"]["id"] == first and stored["previous"] is True
|
||||
assert (await fire_row(third_fire)).compare_artifact == third_artifact
|
||||
|
||||
|
||||
async def test_first_run_of_a_schedule_has_no_compare(recorder, client):
|
||||
@@ -276,7 +368,7 @@ async def test_waits_for_the_seal_before_comparing(recorder):
|
||||
assert (await fire_row(fire_id)).compare_artifact is None and recorder.requests == []
|
||||
await seal(target_id)
|
||||
assert await webhook.notify_due() == 1
|
||||
assert load((await fire_row(fire_id)).compare_artifact)["base"]["id"] == base_id
|
||||
assert load((await fire_row(fire_id)).compare_artifact)["result"]["base"]["id"] == base_id
|
||||
assert LINE in recorder.bodies()[0]["text"]
|
||||
|
||||
|
||||
@@ -327,18 +419,28 @@ async def test_compare_failure_only_logs_and_the_notice_still_goes_out(recorder,
|
||||
assert "封存核对不一致" in caplog.text
|
||||
|
||||
|
||||
async def test_compare_is_hidden_once_the_run_is_running_again(client):
|
||||
"""失败后被继续执行(不在终态):对比页只认封存的运行,接口不给 compare;再次结束时重新比"""
|
||||
sid, base_id, target_id, fire_id = await weekly_pair(target_status="failed", url=None)
|
||||
async def test_a_failed_run_is_compared_once_it_is_continued_to_success(client):
|
||||
"""失败时不比;接着跑(不在终态)时也没有;跑完、成功并重新封存之后按新的封存比一次"""
|
||||
workflow = await new_workflow()
|
||||
sid = await new_schedule(workflow, url=None)
|
||||
base_id = await persist(week("r1"), workflow, finished=T0, schedule_id=sid)
|
||||
await fire_for(sid, base_id, T0 - timedelta(minutes=2), notified="succeeded")
|
||||
target_id = await persist(failed_week("r2"), workflow, finished=T0 + WEEK, schedule_id=sid, status="failed")
|
||||
fire_id = await fire_for(sid, target_id, T0 + WEEK - timedelta(minutes=2))
|
||||
|
||||
async def shown():
|
||||
items = (await client.get(f"/api/schedules/{sid}/fires")).json()["items"]
|
||||
return {f["id"]: f for f in items}[fire_id]
|
||||
|
||||
await webhook.notify_due()
|
||||
assert (await fire_row(fire_id)).compare_artifact
|
||||
assert (await fire_row(fire_id)).compare_artifact is None and (await shown())["compare"] is None
|
||||
await set_run(target_id, status="running")
|
||||
[item] = [f for f in (await client.get(f"/api/schedules/{sid}/fires")).json()["items"] if f["id"] == fire_id]
|
||||
assert item["run_status"] == "running" and item["compare"] is None
|
||||
await set_run(target_id, status="succeeded", finished_at=T0 + WEEK + timedelta(hours=1))
|
||||
await webhook.notify_due()
|
||||
[item] = [f for f in (await client.get(f"/api/schedules/{sid}/fires")).json()["items"] if f["id"] == fire_id]
|
||||
assert item["compare"] == {"base_run_id": base_id, "summary": SUMMARY}
|
||||
item = await shown()
|
||||
assert item["run_status"] == "running" and item["compare"] is None
|
||||
await reseal(target_id, week("r2", NEXT_ROWS), status="succeeded", finished=T0 + WEEK + timedelta(hours=1))
|
||||
await webhook.notify_due()
|
||||
assert (await shown())["compare"] == {"base_run_id": base_id, "summary": SUMMARY, "previous": True}
|
||||
|
||||
|
||||
async def test_compare_is_hidden_once_the_base_run_is_gone_or_running(client):
|
||||
|
||||
@@ -0,0 +1,312 @@
|
||||
"""定时运行自动对比的几道保护(app/scheduler/auto_compare.py、webhook.notify_due):
|
||||
|
||||
- 和封存绑定:运行的封存要盖住它最后一条结束事件才算封存完。runner 先写终态、再另起会话封存,失败后接着跑完的运行
|
||||
在两步之间还带着失败时的清单哈希——这时去比,只比得到旧封存范围里的事件,通知里的清单哈希也是旧的。等不来新的
|
||||
封存(两步之间进程没了)时照常推送,不带对比、不写旧的哈希
|
||||
- 对比结果记下两边的清单哈希:任何一边后来变了,触发历史就不再给这份对比(悬停的计数和点进去的对比页对不上);
|
||||
对比途中封存变了,结果不记
|
||||
- 时限:对比放在认领之前、锁外做,最多等 WAIT_S 秒,同一时刻最多 SLOTS 个。慢的对比不压住这条通知,也不压住别的
|
||||
定时任务的通知;扫描在对比途中被取消(停机),这次触发还没认领,下次照常比、照常推。同一份封存已经比过的(比完
|
||||
还没推送就停机了),下次直接用记下的结果
|
||||
|
||||
运行、任务名、金额都是虚构的(tests/fixtures/compare_runs.py),不真的执行;Webhook 用 httpx.MockTransport 接住。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import time
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
from sqlalchemy import select
|
||||
|
||||
from app import scheduler
|
||||
from app.api import compare as compare_api
|
||||
from app.core.artifact_store import load
|
||||
from app.core.config import Settings
|
||||
from app.db.base import SessionLocal
|
||||
from app.db.models import Run, RunEvent
|
||||
from app.main import app
|
||||
from app.scheduler import auto_compare, webhook
|
||||
from tests.fixtures.compare_runs import BASE_ROWS, NEXT_ROWS, Fake, default_metrics, region_sql, week
|
||||
from tests.test_scheduler_compare import ( # noqa: F401 - clean、recorder 是夹具,按名字引入
|
||||
BASE_URL, LINE, T0, WEEK, Recorder, clean, fire_for, fire_row, new_schedule, new_workflow, persist,
|
||||
public_address, recorder, reseal, seal, set_run, weekly_pair,
|
||||
)
|
||||
|
||||
UTC = timezone.utc
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
async def client():
|
||||
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as c:
|
||||
yield c
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def slow_compare(monkeypatch):
|
||||
"""对比要 5 秒:真的对比之前先等着。返回原来的函数,用例里可以换回去"""
|
||||
real = compare_api.compare_runs
|
||||
|
||||
async def slow(*args, **kwargs):
|
||||
await asyncio.sleep(5)
|
||||
return await real(*args, **kwargs)
|
||||
|
||||
monkeypatch.setattr(compare_api, "compare_runs", slow)
|
||||
return real
|
||||
|
||||
|
||||
def failed_then_continued(run_id: str, rows=NEXT_ROWS) -> tuple[Fake, int]:
|
||||
"""在 Agent 节点失败、之后从断点接着跑完的一次周报。返回 (运行, 失败时封存到第几条事件)。"""
|
||||
run = Fake(run_id)
|
||||
sql = region_sql()
|
||||
fetch = run.tool("fetch", sql, rows=rows)
|
||||
run.add("node.started", "bot", {"node_type": "agent"})
|
||||
run.add("node.failed", "bot", {"error": "模型接入超时"})
|
||||
run.add("run.failed", None, {"error": "节点「区域分析」失败"})
|
||||
failed_at = len(run.events)
|
||||
run.add("run.resumed", None, {"from": ["bot"], "message": "从「区域分析」继续运行"})
|
||||
bot = run.agent("bot", sql, rows=rows)
|
||||
run.card(default_metrics(fetch, bot, rows, rows))
|
||||
run.add("run.finished", None, {"output": {}})
|
||||
return run, failed_at
|
||||
|
||||
|
||||
async def previous_week(*, url: str | None = None):
|
||||
"""一个定时任务和它上一次成功、已推送过的周报。返回 (工作流, 任务, 上一次)。"""
|
||||
workflow = await new_workflow()
|
||||
sid = await new_schedule(workflow, url=url) if url else await new_schedule(workflow)
|
||||
base_id = await persist(week("r1"), workflow, finished=T0, schedule_id=sid)
|
||||
await fire_for(sid, base_id, T0 - timedelta(minutes=2), notified="succeeded")
|
||||
return workflow, sid, base_id
|
||||
|
||||
|
||||
async def seal_of(run_id: str) -> str | None:
|
||||
async with SessionLocal() as session:
|
||||
return (await session.get(Run, run_id)).manifest_hash
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 和封存绑定
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_waits_for_the_seal_of_this_ending_not_the_failed_one(recorder):
|
||||
"""失败后接着跑完:终态和结束时间已经写下,封存还是失败时那一份(只盖到 run.failed)。这一拍不比、不推,新的
|
||||
封存补上之后再比,比的是完整的一次,通知里的清单哈希是新的"""
|
||||
await public_address()
|
||||
workflow, sid, base_id = await previous_week()
|
||||
fake, failed_at = failed_then_continued("r2")
|
||||
target_id = await persist(fake, workflow, finished=datetime.now(UTC), schedule_id=sid, seal_upto=failed_at)
|
||||
fire_id = await fire_for(sid, target_id, T0 + WEEK - timedelta(minutes=2))
|
||||
stale = await seal_of(target_id)
|
||||
|
||||
assert await webhook.notify_due() == 0
|
||||
assert recorder.requests == [] and (await fire_row(fire_id)).compare_artifact is None
|
||||
await seal(target_id)
|
||||
fresh = await seal_of(target_id)
|
||||
assert fresh != stale
|
||||
assert await webhook.notify_due() == 1
|
||||
stored = load((await fire_row(fire_id)).compare_artifact)
|
||||
assert (stored["target_manifest"], stored["result"]["base"]["id"]) == (fresh, base_id)
|
||||
[body] = recorder.bodies()
|
||||
assert LINE in body["text"] and body["manifest_hash"] == fresh[:12]
|
||||
|
||||
|
||||
async def test_a_seal_that_never_catches_up_is_not_used(recorder):
|
||||
"""新的封存一直没补上(写完终态、封存之前进程没了):等过时限照常推送,但不比,也不写失败时那份旧的清单哈希"""
|
||||
await public_address()
|
||||
workflow, sid, _ = await previous_week()
|
||||
fake, failed_at = failed_then_continued("r2")
|
||||
target_id = await persist(fake, workflow, finished=datetime.now(UTC), schedule_id=sid, seal_upto=failed_at)
|
||||
fire_id = await fire_for(sid, target_id, T0 + WEEK - timedelta(minutes=2))
|
||||
later = datetime.now(UTC) + timedelta(seconds=webhook.SEAL_WAIT_S + 5)
|
||||
assert await webhook.notify_due(later) == 1
|
||||
[body] = recorder.bodies()
|
||||
assert (body["event"], body["manifest_hash"], body["compare"]) == ("finished", None, None)
|
||||
assert "清单哈希" not in body["text"] and "相比" not in body["text"]
|
||||
assert (await fire_row(fire_id)).compare_artifact is None
|
||||
|
||||
|
||||
async def test_approval_notice_does_not_carry_the_seal_of_an_earlier_failure(recorder):
|
||||
"""失败后接着跑、停在审批上:运行上还留着失败时的清单哈希,那不是这一次的封存,「等待审批」里不写"""
|
||||
workflow, sid, _ = await previous_week()
|
||||
fake, failed_at = failed_then_continued("r2")
|
||||
target_id = await persist(fake, workflow, finished=T0 + WEEK, schedule_id=sid, status="interrupted",
|
||||
seal_upto=failed_at)
|
||||
await set_run(target_id, finished_at=None)
|
||||
async with SessionLocal() as session:
|
||||
from app.db.models import Approval
|
||||
|
||||
session.add(Approval(run_id=target_id, node_id="card", title="审核周报口径", status="pending"))
|
||||
await session.commit()
|
||||
await fire_for(sid, target_id, T0 + WEEK - timedelta(minutes=2))
|
||||
assert await webhook.notify_due() == 1
|
||||
[body] = recorder.bodies()
|
||||
assert (body["event"], body["manifest_hash"]) == ("awaiting_approval", None)
|
||||
assert "清单哈希" not in body["text"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("side", ["target", "base"])
|
||||
async def test_fire_history_drops_a_compare_whose_seal_changed(client, side):
|
||||
"""比过之后哪一边被重新封存(内容变了):记下的计数已经不是对比页现场算出来的那样,触发历史不再给这份对比"""
|
||||
sid, base_id, target_id, fire_id = await weekly_pair(url=None)
|
||||
await webhook.notify_due()
|
||||
|
||||
async def shown():
|
||||
items = (await client.get(f"/api/schedules/{sid}/fires")).json()["items"]
|
||||
return {f["id"]: f for f in items}[fire_id]["compare"]
|
||||
|
||||
assert (await shown())["base_run_id"] == base_id
|
||||
# 换成和另一边一样的数:现场再比就是「未变」
|
||||
run_id, rows = (target_id, BASE_ROWS) if side == "target" else (base_id, NEXT_ROWS)
|
||||
async with SessionLocal() as session:
|
||||
finished = (await session.get(Run, run_id)).finished_at
|
||||
await reseal(run_id, week("rx", rows), status="succeeded", finished=finished)
|
||||
assert await shown() is None
|
||||
# 对比页现场算出来的和记下的确实不一样
|
||||
assert (await compare_api.compare_runs(target_id, base_id))["summary"]["unchanged"] == 2
|
||||
|
||||
|
||||
async def test_result_is_dropped_when_a_seal_changes_mid_compare(recorder, monkeypatch):
|
||||
"""对比途中基准被重新封存:比出来的是旧封存的结果,不记进触发记录、不写进通知"""
|
||||
await public_address()
|
||||
_, base_id, _, fire_id = await weekly_pair()
|
||||
real = compare_api.compare_runs
|
||||
|
||||
async def resealing(run_id, other):
|
||||
out = await real(run_id, other)
|
||||
await set_run(base_id, manifest_hash="ab" * 32)
|
||||
return out
|
||||
|
||||
monkeypatch.setattr(compare_api, "compare_runs", resealing)
|
||||
assert await webhook.notify_due() == 1
|
||||
assert (await fire_row(fire_id)).compare_artifact is None
|
||||
[body] = recorder.bodies()
|
||||
assert body["compare"] is None and "相比" not in body["text"]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 时限、取消、并发
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_a_slow_compare_is_cut_off_and_the_notice_goes_out(recorder, monkeypatch, caplog, slow_compare):
|
||||
"""对比超过时限:通知不等它,照常推送、不带对比,记一条警告"""
|
||||
monkeypatch.setattr(auto_compare, "WAIT_S", 0.3, raising=False)
|
||||
await public_address()
|
||||
_, _, target_id, fire_id = await weekly_pair()
|
||||
began = time.monotonic()
|
||||
with caplog.at_level(logging.WARNING, logger="app.scheduler"):
|
||||
assert await webhook.notify_due() == 1
|
||||
assert time.monotonic() - began < 2
|
||||
[body] = recorder.bodies()
|
||||
assert (body["event"], body["compare"]) == ("finished", None) and "相比" not in body["text"]
|
||||
assert f"{BASE_URL}/runs/{target_id}" in body["text"]
|
||||
assert (await fire_row(fire_id)).compare_artifact is None
|
||||
assert "超过" in caplog.text and target_id in caplog.text
|
||||
|
||||
|
||||
async def test_scan_cancelled_mid_compare_pushes_next_time(recorder, monkeypatch, slow_compare):
|
||||
"""停机时通知扫描只等 1.5 秒就被取消:撞在对比途中的那次触发还没认领,下次启动照常比、照常推。以前先认领、后比,
|
||||
取消之后这条「运行结束」再也不会推"""
|
||||
await public_address()
|
||||
_, _, _, fire_id = await weekly_pair()
|
||||
with pytest.raises(TimeoutError):
|
||||
await asyncio.wait_for(webhook.notify_due(), timeout=0.5)
|
||||
assert (await fire_row(fire_id)).notified_status is None and recorder.requests == []
|
||||
monkeypatch.setattr(compare_api, "compare_runs", slow_compare)
|
||||
assert await webhook.notify_due() == 1
|
||||
[body] = recorder.bodies()
|
||||
assert LINE in body["text"]
|
||||
assert (await fire_row(fire_id)).notified_status == "succeeded"
|
||||
|
||||
|
||||
async def test_a_slow_compare_does_not_hold_other_schedules(monkeypatch, slow_compare):
|
||||
"""任务 A 的对比慢:调度循环里通知扫描同一时刻只跑一份,以前 A 比多久,之后才结束的任务 B(第一次,不用比)
|
||||
的通知就要等多久。现在 A 的对比等过时限就不等了,B 的通知在下一拍推出"""
|
||||
rec = Recorder()
|
||||
monkeypatch.setattr(webhook, "_transport", httpx.MockTransport(rec))
|
||||
monkeypatch.setattr(webhook, "RETRY_DELAYS", (0, 0, 0))
|
||||
monkeypatch.setattr(auto_compare, "WAIT_S", 0.5, raising=False)
|
||||
await weekly_pair()
|
||||
on = Settings(_env_file=None, scheduler=True)
|
||||
assert await scheduler.start(interval=0.05, config=on) is True
|
||||
try:
|
||||
await asyncio.sleep(0.2)
|
||||
workflow = await new_workflow("仓库日报")
|
||||
sid = await new_schedule(workflow, name="仓库日报 · 每天")
|
||||
run_id = await persist(week("b1"), workflow, finished=T0, schedule_id=sid)
|
||||
await fire_for(sid, run_id, T0 - timedelta(minutes=2))
|
||||
began = time.monotonic()
|
||||
while not any(b.get("run_id") == run_id for b in rec.bodies()) and time.monotonic() - began < 8:
|
||||
await asyncio.sleep(0.02)
|
||||
waited = time.monotonic() - began
|
||||
finally:
|
||||
await scheduler.stop()
|
||||
assert any(b.get("run_id") == run_id for b in rec.bodies())
|
||||
assert waited < 2, f"任务 B 的通知等了 {waited:.2f} 秒"
|
||||
|
||||
|
||||
async def test_compares_take_turns(monkeypatch):
|
||||
"""同一拍里好几个任务都要比:同一时刻最多 SLOTS 个,不一起占满线程池(接口请求也要用)"""
|
||||
monkeypatch.setattr(auto_compare, "SLOTS", 2, raising=False)
|
||||
real = compare_api.compare_runs
|
||||
active = peak = 0
|
||||
|
||||
async def counted(*args, **kwargs):
|
||||
nonlocal active, peak
|
||||
active += 1
|
||||
peak = max(peak, active)
|
||||
try:
|
||||
await asyncio.sleep(0.2)
|
||||
return await real(*args, **kwargs)
|
||||
finally:
|
||||
active -= 1
|
||||
|
||||
monkeypatch.setattr(compare_api, "compare_runs", counted)
|
||||
fires = [(await weekly_pair(url=None))[3] for _ in range(4)]
|
||||
await webhook.notify_due()
|
||||
assert peak == 2
|
||||
assert all([(await fire_row(f)).compare_artifact for f in fires])
|
||||
|
||||
|
||||
async def test_a_compare_already_made_for_this_seal_is_reused(recorder, monkeypatch):
|
||||
"""比完、还没认领就停机了,结果已经记在触发记录上:下次直接用,不再算一遍"""
|
||||
await public_address()
|
||||
_, _, target_id, fire_id = await weekly_pair()
|
||||
assert await auto_compare.compare_fire(fire_id, target_id) is not None
|
||||
artifact = (await fire_row(fire_id)).compare_artifact
|
||||
assert artifact
|
||||
calls = 0
|
||||
|
||||
async def counted(*args, **kwargs):
|
||||
nonlocal calls
|
||||
calls += 1
|
||||
raise RuntimeError("不该再算一遍")
|
||||
|
||||
monkeypatch.setattr(compare_api, "compare_runs", counted)
|
||||
auto_compare._BRIEFS.clear()
|
||||
assert await webhook.notify_due() == 1
|
||||
assert calls == 0 and LINE in recorder.bodies()[0]["text"]
|
||||
assert (await fire_row(fire_id)).compare_artifact == artifact
|
||||
|
||||
|
||||
async def test_seal_covers_the_last_ending():
|
||||
"""封存盖没盖住最后一条结束事件:老运行(没有 manifest_seq)只看有没有哈希"""
|
||||
workflow = await new_workflow()
|
||||
fake, failed_at = failed_then_continued("r9")
|
||||
partial = await persist(fake, workflow, finished=T0, seal_upto=failed_at)
|
||||
full = await persist(failed_then_continued("r8")[0], workflow, finished=T0)
|
||||
legacy = await persist(week("r7"), workflow, finished=T0)
|
||||
await set_run(legacy, manifest_seq=None)
|
||||
unsealed = await persist(week("r6"), workflow, finished=T0, seal=False)
|
||||
async with SessionLocal() as session:
|
||||
got = {rid: await auto_compare.sealed_now(session, await session.get(Run, rid))
|
||||
for rid in (partial, full, legacy, unsealed)}
|
||||
assert await session.scalar(select(RunEvent.seq).where(
|
||||
RunEvent.run_id == partial, RunEvent.type == "run.finished")) > failed_at
|
||||
assert got == {partial: False, full: True, legacy: True, unsealed: False}
|
||||
@@ -1,9 +1,10 @@
|
||||
"""定时任务通知(app/scheduler/webhook.py)的两条规则:
|
||||
|
||||
- 失败通知的「处理」行不许诺能继续运行:服务重启时还在排队、从没开始执行的运行(被改成失败、没有开始时间)没有
|
||||
断点,「修正后可继续运行」是错的。按有没有开始执行分开写——没开始的另写一行原因(没有可继续的进度),处理一律是
|
||||
中性的「请在运行记录中查看原因」;任务还在排期上时补一句「下一次触发会照常发起」:失败的运行不挡之后的触发,
|
||||
和挂起的那种(之后的触发都会跳过)说清区别
|
||||
断点,「修正后可继续运行」是错的。按有没有开始执行分开写——没开始的另写一行原因(没有可继续的进度),处理行不再
|
||||
让人去「查看原因」(原因刚写过),任务还在排期上时只写「下一次触发会照常发起」,不在排期上时写怎么重新发起;开始
|
||||
执行过的写「请在 AgentLab 的「记录」中查看失败原因」(和挂起那句一样指到导航里的「记录」),任务还在排期上时补
|
||||
「下一次触发会照常发起」:失败的运行不挡之后的触发,和挂起的那种(之后的触发都会跳过)说清区别
|
||||
- 扫描窗口(SCAN_DAYS)之外的触发:停在审批、有待审批的运行不受窗口限制——创建 30 天以后才停到审批上的照样推送
|
||||
「等待审批」;运行记录在窗口内有变化的(等了一个多月的审批批完、跑完)照样推送结束。早就结束、记录多日没有变化
|
||||
的旧触发仍不扫
|
||||
@@ -27,6 +28,8 @@ from app.scheduler import webhook
|
||||
UTC = timezone.utc
|
||||
GENERIC = "https://hooks.example.com/agentlab/inbox-3e9d"
|
||||
NOT_STARTED = "运行在开始执行前就已中止(通常是服务重启时仍在排队),没有可继续的进度"
|
||||
LOOK = "请在 AgentLab 的「记录」中查看失败原因"
|
||||
RERUN = "如需重新发起,请在 AgentLab 的「定时任务」中点击「立即运行」"
|
||||
|
||||
|
||||
class Recorder:
|
||||
@@ -106,12 +109,15 @@ async def notified(fire_id: str) -> str | None:
|
||||
|
||||
|
||||
def test_a_run_that_never_started_is_not_promised_a_continue():
|
||||
"""服务重启时还在排队的运行没有断点:不写「修正后可继续运行」,另写一行原因说没有可继续的进度"""
|
||||
"""服务重启时还在排队的运行没有断点:不写「修正后可继续运行」,另写一行原因说没有可继续的进度。处理行不再让人
|
||||
去「查看原因」(原因刚写过):还在排期上的只说下一次照常,不在排期上的说怎么重新发起"""
|
||||
_, lines, _ = webhook.message(notice(started_at=None, next_fire=True))
|
||||
shown = dict(lines)
|
||||
assert shown["原因"] == NOT_STARTED
|
||||
assert shown["处理"] == "请在 AgentLab 的运行记录中查看原因;下一次触发会照常发起"
|
||||
assert not any("继续运行" in value for _, value in lines), lines
|
||||
assert shown["处理"] == "下一次触发会照常发起"
|
||||
assert not any("继续运行" in value or "查看" in value for _, value in lines), lines
|
||||
_, lines, _ = webhook.message(notice(started_at=None, next_fire=False))
|
||||
assert dict(lines)["处理"] == RERUN
|
||||
|
||||
|
||||
def test_a_failed_run_gets_a_neutral_action_without_promising_a_continue():
|
||||
@@ -120,14 +126,14 @@ def test_a_failed_run_gets_a_neutral_action_without_promising_a_continue():
|
||||
_, lines, _ = webhook.message(notice(error_node="汇总库存", next_fire=True))
|
||||
shown = dict(lines)
|
||||
assert shown["失败节点"] == "「汇总库存」" and "原因" not in shown
|
||||
assert shown["处理"] == "请在 AgentLab 的运行记录中查看原因;下一次触发会照常发起"
|
||||
assert shown["处理"] == f"{LOOK};下一次触发会照常发起"
|
||||
assert "修正后可继续运行" not in webhook._text(notice(error_node="汇总库存"))
|
||||
|
||||
|
||||
def test_next_fire_is_only_promised_while_the_schedule_is_on():
|
||||
"""任务已暂停(或往后不再触发)时不说「下一次触发会照常发起」"""
|
||||
_, lines, _ = webhook.message(notice(next_fire=False))
|
||||
assert dict(lines)["处理"] == "请在 AgentLab 的运行记录中查看原因"
|
||||
assert dict(lines)["处理"] == LOOK
|
||||
|
||||
|
||||
async def test_restart_failure_notice_end_to_end(recorder):
|
||||
@@ -140,12 +146,12 @@ async def test_restart_failure_notice_end_to_end(recorder):
|
||||
by_fire = {b["fire_id"]: b for b in recorder.bodies()}
|
||||
on, off = by_fire[on_fire], by_fire[off_fire]
|
||||
assert (on["event"], on["status"], on["reason"]) == ("failed", "failed", NOT_STARTED)
|
||||
assert "处理:请在 AgentLab 的运行记录中查看原因;下一次触发会照常发起" in on["text"]
|
||||
assert "处理:请在 AgentLab 的运行记录中查看原因\n" in off["text"] + "\n" and "照常发起" not in off["text"]
|
||||
assert "处理:下一次触发会照常发起" in on["text"]
|
||||
assert f"处理:{RERUN}" in off["text"] and "照常发起" not in off["text"]
|
||||
for body in (on, off):
|
||||
assert "继续运行" not in body["text"], body["text"]
|
||||
# 运行记录上的原始报错不进通知
|
||||
assert "请重新发起运行" not in body["text"]
|
||||
assert "没有可恢复的进度" not in body["text"]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
@@ -179,7 +179,7 @@ def test_failed_run_text_points_to_the_node_without_repeating_itself():
|
||||
shown = dict(lines)
|
||||
assert title.endswith("运行失败") and shown["状态"] == "失败"
|
||||
# 处理只说去哪里看,不许诺能继续运行(没有断点的运行续不了,见 test_scheduler_notice_rules.py)
|
||||
assert shown["失败节点"] == "「查询门店销售」" and shown["处理"] == "请在 AgentLab 的运行记录中查看原因"
|
||||
assert shown["失败节点"] == "「查询门店销售」" and shown["处理"] == "请在 AgentLab 的「记录」中查看失败原因"
|
||||
# 不再写「原因:运行失败,详情请查看运行记录」这种同义反复
|
||||
assert "原因" not in shown
|
||||
|
||||
|
||||
Reference in New Issue
Block a user