mirror of
https://github.com/ThearchyHelios/AgentLab.git
synced 2026-11-15 00:31:58 +00:00
助手只写了思考、一个操作都没给时自动重来一次:问数据和评测走同一个决定
真实模型验收里 deepseek 偶发只出思考、不出操作,生成过程交出 error「助手本轮未完成:模型没有给出任何修改,也没有说明 原因」,问数据这一轮照实报了,但不会自动重来——而这种偶发情况重来一次多半就好,「重来一次」的规矩本来就有 (chat/turn 的 MAX_ATTEMPTS = 2,含第一次)。 - chat/turn:error 也由 next_step 判:机读码是 EMPTY_GENERATION(copilot_empty)的判 retry,别的仍判失败(被截断的不重来: 额度不变,重来还是截断);error_reason 取重来时交给助手的原因(error 带的 reason,没有就去掉「助手本轮未完成:」前缀); - api/copilot.answer_ops:一个操作都没给时 CopilotIncomplete 带上这个码;error 操作多带 reason 和 next(retry | fail),和 final 的 next 一样由服务端定; - 问数据页(store/chat.ts)照 error.next 走:retry 就带着原因重来一次(同 retry_instruction 的从零重建措辞,次数照 MAX_ATTEMPTS),试满了才照实报;进度写「助手上一次没有给出任何操作,正在重新生成工作流…」(lib/terms 的 CHAT_RETRY_TEXT); 第一次建图的用量照样记进这一轮; - 评测(eval/answer.Driver.drive)用同一个 next_step / error_reason / retry_instruction:先重来一次,还不行才记建图失败、 助手调用失败;每次自动重来的原因照实记在 turn.retries(没过检查、复核判不可用的重来也记),评测页逐题明细写出来。 测试:test_chat_turn_policy.py 的 test_next_step_branches(error 的两支和 error_reason)、 test_the_frontend_retries_an_error_the_server_says_to_retry、test_answer_ops_says_to_retry_an_empty_generation、 test_the_eval_driver_retries_an_empty_generation_once(重来后回答 / 重来还不行,两次的用量都算); test_copilot_answer_ops.py 的 SSE 对照多认 error 上新加的 next、reason、code。去掉修复回跑都失败。 浏览器:check-chat 新加 emptyretry 一节(重来一次后照常回答、交回去的话带原因、两次用量都记;两次都不行照实报、不试第三次), check-chat 840 项全过。
This commit is contained in:
@@ -1751,14 +1751,20 @@ async def answer_ops(payload: GenerateIn, *, catalog_view: Any = None):
|
||||
# 流正常结束,却没有 done、也没有一个操作能认(格式跑偏,每一行都被跳过):
|
||||
# 往下走就是拿原图去自查、交付,界面上写「无需修改」——而模型什么都没说
|
||||
if gen.asked is None and all(o.get("op") == "plan" for o in applier.seen):
|
||||
# 只写了思考(deepseek 这类推理模型偶发)、或者格式跑偏:带机读码,问数据页和评测按 chat/turn 的规矩自动
|
||||
# 重来一次(error.next = retry),还不行再照实报
|
||||
raise CopilotIncomplete("模型没有给出任何修改,也没有说明原因",
|
||||
hint="画布未修改。可以重试,或把需求描述得更具体")
|
||||
hint="画布未修改。可以重试,或把需求描述得更具体", code=turn.EMPTY_GENERATION)
|
||||
except CopilotIncomplete as e:
|
||||
yield {"op": "error", "message": f"助手本轮未完成:{e}", "hint": e.hint, "detail": e.detail}
|
||||
failed = {"op": "error", "message": f"助手本轮未完成:{e}", "hint": e.hint, "detail": e.detail,
|
||||
"reason": str(e), **({"code": e.code} if e.code else {})}
|
||||
yield {**failed, "next": turn.next_step(failed)}
|
||||
return
|
||||
except Exception as e: # noqa: BLE001
|
||||
reason, hint = explain_error(e)
|
||||
yield {"op": "error", "message": f"助手本轮未完成:{reason}", "hint": hint, "detail": raw_error(e)}
|
||||
failed = {"op": "error", "message": f"助手本轮未完成:{reason}", "hint": hint, "detail": raw_error(e),
|
||||
"reason": reason}
|
||||
yield {**failed, "next": turn.next_step(failed)}
|
||||
return
|
||||
|
||||
# 兜底:一个操作都没落地,而上一轮有图。
|
||||
|
||||
@@ -69,12 +69,14 @@ def _parse_op_line(line: str) -> dict[str, Any] | None:
|
||||
|
||||
|
||||
class CopilotIncomplete(Exception):
|
||||
"""模型这一轮没说完:输出被截断,或者一个操作都没给。画布不能按「无需修改」交付。"""
|
||||
"""模型这一轮没说完:输出被截断,或者一个操作都没给。画布不能按「无需修改」交付。code 是机读码:一个操作都没给的是
|
||||
chat/turn.EMPTY_GENERATION(问数据一轮按它自动重来一次),截断的不给。"""
|
||||
|
||||
def __init__(self, reason: str, *, hint: str = "", detail: str = "") -> None:
|
||||
def __init__(self, reason: str, *, hint: str = "", detail: str = "", code: str = "") -> None:
|
||||
super().__init__(reason)
|
||||
self.hint = hint or "画布未修改。可以重试,或把需求拆成几步分别提出"
|
||||
self.detail = detail
|
||||
self.code = code
|
||||
|
||||
|
||||
async def _drain(stream: Any, already: int) -> None:
|
||||
|
||||
@@ -5,13 +5,13 @@
|
||||
后端跑不了一轮问数据:评测要么照着抄一份(迟早走样,量出来的不是用户拿到的那一轮),要么做不了;这也违反「界面层不放
|
||||
逻辑」。现在决定写在这里,是纯函数:
|
||||
|
||||
- next_step:生成过程交出结局之后做什么;
|
||||
- next_step:生成过程交出结局之后做什么(含生成过程一个操作都没给时自动重来一次);
|
||||
- fill_input:用户的问题填进这张图的入口字段;
|
||||
- retry_after_review:复核之后要不要带着原图重来一次;
|
||||
- retry_instruction:重来时交给助手的那句话(两种措辞)。
|
||||
|
||||
接口把决定交给问数据页:/copilot/generate-stream 的 final 带 next(run | ready | fail),/copilot/review 的结果带
|
||||
retry_instruction(要重来时是那句话,不重来是 null)。问数据页照后端给的走;老后端没有这两个字段时退回它自己那份同样的
|
||||
接口把决定交给问数据页:/copilot/generate-stream 的 final 带 next(run | ready | fail),error 带 next(retry | fail),
|
||||
/copilot/review 的结果带 retry_instruction(要重来时是那句话,不重来是 null)。问数据页照后端给的走;老后端没有这两个字段时退回它自己那份同样的
|
||||
规则(tests/test_chat_turn_policy.py 逐条对照两边的常量和措辞)。评测(第 3 波)用同一组函数把一轮串起来。
|
||||
|
||||
只搬「决定」:重连、断点续看、取消、复核等多久这些仍在问数据页(把整轮编排搬到后端要重做这些,单独立项)。
|
||||
@@ -28,9 +28,19 @@ MAX_ATTEMPTS = 2
|
||||
#: final 之后的三种走法(final.next):运行、只建不跑(摆着等用户点运行)、判失败(没过检查的图不拿去跑)
|
||||
NEXT_RUN, NEXT_READY, NEXT_FAIL = "run", "ready", "fail"
|
||||
NEXT_STEPS = (NEXT_RUN, NEXT_READY, NEXT_FAIL)
|
||||
#: 别的结局:直接回答了(reply)、助手反问了等用户答(asked)、没自查过的老服务端交来没过检查的图时重来一次(retry)
|
||||
#: 别的结局:直接回答了(reply)、助手反问了等用户答(asked)、重来一次(retry:没自查过的老服务端交来没过检查的图,或者
|
||||
#: 生成过程一个操作都没给)
|
||||
NEXT_REPLY, NEXT_ASKED, NEXT_RETRY = "reply", "asked", "retry"
|
||||
|
||||
#: 生成过程一个操作都没给就结束了(error 操作的 code):模型只写了思考(deepseek 这类推理模型偶发)、或者格式跑偏每一行都被
|
||||
#: 跳过。这是偶发的,重来一次多半就好,按「重来一次」的规矩自动重来(最多 MAX_ATTEMPTS 次,含第一次);被截断的不重来——额度
|
||||
#: 不变,重来还是截断
|
||||
EMPTY_GENERATION = "copilot_empty"
|
||||
#: 自动重来的 error(code)
|
||||
RETRY_ERRORS = frozenset({EMPTY_GENERATION})
|
||||
#: error 操作的 message 前缀(api/copilot.answer_ops):重来时交给助手的原因不带它
|
||||
_ERROR_PREFIX = "助手本轮未完成:"
|
||||
|
||||
#: 问题往哪个入口字段放:名字像「问题」的优先(同 chat.ts fillInput 的正则,大小写不敏感)
|
||||
QUESTION_FIELD = re.compile(r"question|query|topic|input|问题|需求|主题", re.IGNORECASE)
|
||||
|
||||
@@ -51,7 +61,9 @@ def next_step(terminal: Mapping[str, Any] | None, *, self_checked: bool = True)
|
||||
"""生成过程交出结局之后做什么。terminal 是最后那条结局操作(final / reply / error);流结束时只有反问(ask)、没有
|
||||
别的结局的给 None。
|
||||
|
||||
- reply:直接回答,这一轮到此为止;error:判失败;只有反问:等用户回答;
|
||||
- reply:直接回答,这一轮到此为止;只有反问:等用户回答;
|
||||
- error:生成过程一个操作都没给(code 是 EMPTY_GENERATION)时重来一次(retry,次数由调用方按 MAX_ATTEMPTS 管,原因
|
||||
取 error_reason),别的判失败;
|
||||
- final:有 error 级问题时,服务端已经自查过(收到过 check 的结论)就判失败——没过的问题已经交回模型在同一轮里改过
|
||||
了,再整轮重来是再烧一次完整生成、改的还是同一批问题;没自查过(老服务端)才重来一次(retry);模型说不运行
|
||||
(autorun=false:用户要的是流程本身、或者问了用户)只建不跑;其余运行。
|
||||
@@ -61,6 +73,8 @@ def next_step(terminal: Mapping[str, Any] | None, *, self_checked: bool = True)
|
||||
kind = terminal.get("op")
|
||||
if kind == "reply":
|
||||
return NEXT_REPLY
|
||||
if kind == "error":
|
||||
return NEXT_RETRY if terminal.get("code") in RETRY_ERRORS else NEXT_FAIL
|
||||
if kind != "final":
|
||||
return NEXT_FAIL
|
||||
if _blockers(terminal):
|
||||
@@ -70,6 +84,15 @@ def next_step(terminal: Mapping[str, Any] | None, *, self_checked: bool = True)
|
||||
return NEXT_RUN
|
||||
|
||||
|
||||
def error_reason(error: Mapping[str, Any]) -> str:
|
||||
"""生成过程交出 error 时重来的原因(交给助手的那句话里用):error 带的 reason,没有就是 message 去掉「助手本轮未完成:」。"""
|
||||
reason = str(error.get("reason") or "").strip()
|
||||
if reason:
|
||||
return reason
|
||||
message = str(error.get("message") or "").strip()
|
||||
return message[len(_ERROR_PREFIX):] if message.startswith(_ERROR_PREFIX) else message
|
||||
|
||||
|
||||
def blocked_reason(final: Mapping[str, Any]) -> str:
|
||||
"""没过检查的图重来时的原因(只有老服务端那条路用得到)。"""
|
||||
return "生成的工作流没有通过校验:" + ";".join(_blockers(final))
|
||||
@@ -124,6 +147,7 @@ def has_nodes(graph: Any) -> bool:
|
||||
return isinstance(graph, Mapping) and bool(graph.get("nodes"))
|
||||
|
||||
|
||||
__all__ = ["MAX_ATTEMPTS", "NEXT_ASKED", "NEXT_FAIL", "NEXT_READY", "NEXT_REPLY", "NEXT_RETRY", "NEXT_RUN", "NEXT_STEPS",
|
||||
"QUESTION_FIELD", "RETRY_FROM_SCRATCH", "RETRY_WITH_BASE", "blocked_reason", "fill_input", "has_nodes",
|
||||
"next_step", "retry_after_review", "retry_instruction", "review_reason"]
|
||||
__all__ = ["EMPTY_GENERATION", "MAX_ATTEMPTS", "NEXT_ASKED", "NEXT_FAIL", "NEXT_READY", "NEXT_REPLY", "NEXT_RETRY",
|
||||
"NEXT_RUN", "NEXT_STEPS", "QUESTION_FIELD", "RETRY_ERRORS", "RETRY_FROM_SCRATCH", "RETRY_WITH_BASE",
|
||||
"blocked_reason", "error_reason", "fill_input", "has_nodes", "next_step", "retry_after_review",
|
||||
"retry_instruction", "review_reason"]
|
||||
|
||||
@@ -10,7 +10,8 @@
|
||||
复核跟着换);读哪份目录按变体(answer_ops 的 catalog_view,查看的表详情、自查的 SQL 检查器都按它)。
|
||||
- 结局:reply 直接回答、asked 反问、fail 判失败(建图失败)、ready 只建不跑、retry(老服务端那条路:没自查过的图带着原因
|
||||
重来一次)、run 运行。生成过程交出 error(助手本轮未完成)、开流前就拒绝(模型没配好、数据源不在)都是建图失败,另记
|
||||
「助手调用失败」——连续几题都这样就止损(runner.py)。
|
||||
「助手调用失败」——连续几题都这样就止损(runner.py);只有生成过程一个操作都没给(只写了思考,error.next=retry)时按
|
||||
同一个规矩先带着原因重来一次,还不行才记建图失败。每次重来的原因照实记在 retries。
|
||||
- 运行:fill_input 填入口字段(单轮,history 为空),来源 eval,名字「评测:题集名·第 n 题」,选项带目录视图(Run.options
|
||||
的 catalog_view:运行时的 db_schema、冻结快照、自查都按它)、只读和遮罩(同使用者的问数据运行:评测页会显示答案里的数,
|
||||
遮罩值不能从这里露出来;评测不写业务库)。写作模型按变体写进图的 defaults:节点自己点了模型的不覆盖,照实记在
|
||||
@@ -79,6 +80,8 @@ class Turn:
|
||||
#: 这一题发起过的全部作答运行(重来的话两次),失败的也在
|
||||
runs: list[str] = field(default_factory=list)
|
||||
attempts: int = 0
|
||||
#: 每次自动重来的原因(和 attempts 一一对应):没过检查、复核说不可信、生成过程一个操作都没给
|
||||
retries: list[str] = field(default_factory=list)
|
||||
reason: str | None = None
|
||||
#: 助手调用失败(生成过程交出 error、开流前就被拒):runner 据此止损
|
||||
copilot_failed: bool = False
|
||||
@@ -101,7 +104,7 @@ class Turn:
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return {"path": self.path, "run_id": self.run_id, "runs": list(self.runs), "attempts": self.attempts,
|
||||
"reason": self.reason, "copilot_failed": self.copilot_failed, "timed_out": self.timed_out,
|
||||
"retries": list(self.retries), "reason": self.reason, "copilot_failed": self.copilot_failed, "timed_out": self.timed_out,
|
||||
"reply": self.reply, "asked": self.asked, "review": self.review, "looks": list(self.looks),
|
||||
"assistant_model": self.assistant_model, "usage": copy.deepcopy(self.usage),
|
||||
"timings": dict(self.timings), "own_models": list(self.own_models)}
|
||||
@@ -381,6 +384,14 @@ class Driver:
|
||||
return t
|
||||
kind = terminal.get("op")
|
||||
if kind == "error":
|
||||
# 生成过程一个操作都没给(只写了思考):按重来一次的规矩带着原因重来(同问数据页,chat/turn.next_step)
|
||||
step = terminal.get("next") or policy.next_step(terminal)
|
||||
if step == policy.NEXT_RETRY and t.attempts + 1 < policy.MAX_ATTEMPTS:
|
||||
reason = policy.error_reason(terminal)
|
||||
t.attempts += 1
|
||||
t.retries.append(reason)
|
||||
instruction = policy.retry_instruction(self.question, reason, policy.has_nodes(base_graph))
|
||||
continue
|
||||
t.path, t.reason, t.copilot_failed = "build_failed", str(terminal.get("message") or "助手本轮未完成"), True
|
||||
return t
|
||||
if kind == "reply":
|
||||
@@ -395,6 +406,7 @@ class Driver:
|
||||
if step == policy.NEXT_RETRY:
|
||||
if t.attempts + 1 < policy.MAX_ATTEMPTS:
|
||||
t.attempts += 1
|
||||
t.retries.append(policy.blocked_reason(terminal))
|
||||
instruction = policy.retry_instruction(self.question, policy.blocked_reason(terminal),
|
||||
policy.has_nodes(graph))
|
||||
base_graph = graph if policy.has_nodes(graph) else None
|
||||
@@ -423,7 +435,10 @@ class Driver:
|
||||
t.review = {k: v for k, v in review.items() if k != "usage"}
|
||||
t.usage["review"] = _add_usage(t.usage.get("review"), review.get("usage"))
|
||||
if review.get("retry_instruction"):
|
||||
from app.chat.turn import review_reason
|
||||
|
||||
t.attempts += 1
|
||||
t.retries.append(review_reason(review) or "复核判答案不可用")
|
||||
instruction = str(review["retry_instruction"])
|
||||
base_graph = row.graph if policy.has_nodes(row.graph) else None
|
||||
continue
|
||||
|
||||
@@ -48,6 +48,11 @@ def test_next_step_branches():
|
||||
assert turn.next_step(_final(issues=[ERROR], autorun=False)) == "fail", "没过检查比只建不跑优先"
|
||||
assert turn.next_step({"op": "reply", "text": "已经查过了"}) == "reply"
|
||||
assert turn.next_step({"op": "error", "message": "x"}) == "fail"
|
||||
empty = {"op": "error", "message": "助手本轮未完成:模型没有给出任何修改,也没有说明原因",
|
||||
"code": turn.EMPTY_GENERATION, "reason": "模型没有给出任何修改,也没有说明原因"}
|
||||
assert turn.next_step(empty) == "retry", "只写了思考、一个操作都没给:按重来一次的规矩自动重来"
|
||||
assert turn.error_reason(empty) == "模型没有给出任何修改,也没有说明原因"
|
||||
assert turn.error_reason({"op": "error", "message": "助手本轮未完成:接入返回了 502"}) == "接入返回了 502"
|
||||
assert turn.next_step(None) == "asked", "流结束时只有反问"
|
||||
assert set(turn.NEXT_STEPS) == {"run", "ready", "fail"}
|
||||
|
||||
@@ -167,6 +172,74 @@ async def _run(output: dict, graph: dict, code: str) -> str:
|
||||
return run.id
|
||||
|
||||
|
||||
def test_the_frontend_retries_an_error_the_server_says_to_retry():
|
||||
"""问数据页照服务端给的 next 走:error 操作带 next=retry(只写了思考、一个操作都没给)时带着原因重来一次,不再直接报
|
||||
「助手本轮未完成」。读 chat.ts 里 error 那一支(行为另由 scripts/check-chat.mjs 的 emptyretry 一节在浏览器里守)。"""
|
||||
branch = re.search(r"case 'error':(.*?)break\n", _ts(), re.S)
|
||||
assert branch, "chat.ts 里找不到 error 那一支"
|
||||
assert re.search(r"op\.next === 'retry'", branch.group(1)) and "retry(" in branch.group(1), branch.group(1)
|
||||
|
||||
|
||||
async def test_answer_ops_says_to_retry_an_empty_generation(client, monkeypatch):
|
||||
"""生成过程一个操作都没给(deepseek 偶发只出思考):error 操作带机读码 copilot_empty、原因、next=retry。"""
|
||||
import json
|
||||
|
||||
import app.api.copilot as copilot
|
||||
from tests.test_usage_meter import Scripted
|
||||
|
||||
model = Scripted(model="claude-opus-5", replies=[[""]])
|
||||
|
||||
async def _model(*_a, **_k):
|
||||
return model, "claude-opus-5"
|
||||
|
||||
monkeypatch.setattr(copilot, "get_chat_model", _model)
|
||||
events = []
|
||||
async with client.stream("POST", "/api/copilot/generate-stream",
|
||||
json={"instruction": "上周各地区销售额", "intent": "answer"}) as r:
|
||||
async for line in r.aiter_lines():
|
||||
if line.startswith("data: "):
|
||||
events.append(json.loads(line[6:]))
|
||||
[err] = [e for e in events if e["op"] == "error"]
|
||||
assert err["code"] == turn.EMPTY_GENERATION and err["next"] == "retry", err
|
||||
assert err["reason"] == "模型没有给出任何修改,也没有说明原因" and err["message"].startswith("助手本轮未完成")
|
||||
|
||||
|
||||
async def test_the_eval_driver_retries_an_empty_generation_once(monkeypatch):
|
||||
"""评测的作答(app/eval/answer.Driver)走同一个决定:第一次一个操作都没给就带着原因重来一次,原因照实记在 retries;
|
||||
重来还不行才照实报建图失败(助手调用失败),不再试第三次。"""
|
||||
import json
|
||||
|
||||
import app.api.copilot as copilot
|
||||
from app.eval import answer
|
||||
from tests.test_usage_meter import Scripted
|
||||
|
||||
reply = json.dumps({"op": "reply", "text": "上周没有华中区的数据"}, ensure_ascii=False)
|
||||
for replies, path in (([[""], [reply]], "reply"), ([[""]], "build_failed")):
|
||||
model = Scripted(model="claude-opus-5", replies=replies)
|
||||
seen: list[str] = []
|
||||
|
||||
async def _model(*_a, _m=model, **_k):
|
||||
return _m, "claude-opus-5"
|
||||
|
||||
monkeypatch.setattr(copilot, "get_chat_model", _model)
|
||||
real = answer.Driver._generate
|
||||
|
||||
async def spy(self, instruction, base_graph, _seen=seen, _real=real):
|
||||
_seen.append(instruction)
|
||||
return await _real(self, instruction, base_graph)
|
||||
|
||||
monkeypatch.setattr(answer.Driver, "_generate", spy)
|
||||
driver = answer.Driver("上周华中区销售额", datasource_ids=[], ctx=answer.TurnContext(
|
||||
suite_id="s", suite_name="虚构题集", eval_run_id="e", case_id=None, case_seq=1, actor=None))
|
||||
t = await driver.drive()
|
||||
assert t.path == path and t.attempts == 1 and model.calls == 2, (t.path, t.attempts, model.calls, t.reason)
|
||||
assert t.retries == ["模型没有给出任何修改,也没有说明原因"], t.retries
|
||||
assert seen[1] == turn.retry_instruction("上周华中区销售额", t.retries[0], False), seen
|
||||
if path == "build_failed":
|
||||
assert t.copilot_failed and t.reason.startswith("助手本轮未完成"), t.reason
|
||||
assert t.usage["assistant"]["calls"] == 2, "两次生成都是这一题花的"
|
||||
|
||||
|
||||
async def test_review_says_whether_and_how_to_retry(client, monkeypatch):
|
||||
import app.api.copilot as copilot
|
||||
from app.providers.factory import ProviderNotConfigured
|
||||
|
||||
@@ -5,7 +5,8 @@
|
||||
|
||||
- 几种结局(建图交付、直接回答、只问不改、自查改不好、模型出错、没给操作、输出被截断)的 SSE 输出,和抽出来之前
|
||||
逐条一样(剧本见 SCENARIOS,抽出来之前录下的输出在 fixtures/copilot_answer_ops_baseline.json),末尾只多一条 usage,
|
||||
final 只多一个 next(问数据一轮的决定,chat/turn.py);
|
||||
final 只多一个 next(问数据一轮的决定,chat/turn.py);error 多一个 next 和原因 reason(一个操作都没给的另带机读码 code,
|
||||
next 是 retry:问数据页和评测据此自动重来一次);
|
||||
- 直接迭代 answer_ops 和 SSE 收到的是同一串;
|
||||
- 开流前的拒绝(模型没配好、限定的数据源都不在)照旧是 HTTP 错误,不进流。
|
||||
|
||||
@@ -133,6 +134,8 @@ BODY = {"instruction": "数一数门店", "intent": "answer"}
|
||||
|
||||
#: final 带的 next(chat/turn.next_step):问数据页照它走
|
||||
NEXT = {"graph_ok": "run", "graph_fixed": "run", "check_failed": "fail", "ask_only": "ready"}
|
||||
#: error 带的 (next, code):只有一个操作都没给的重来一次
|
||||
ERROR_NEXT = {"error": ("fail", None), "no_ops": ("retry", "copilot_empty"), "truncated": ("fail", None)}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("name", list(SCENARIOS))
|
||||
@@ -144,6 +147,11 @@ async def test_the_sse_output_is_unchanged_but_for_a_trailing_usage(client, monk
|
||||
# final 多了 next(问数据一轮的决定下沉到后端),别的一个字不变
|
||||
finals = [op for op in got if op["op"] == "final"]
|
||||
assert [op.pop("next") for op in finals] == ([NEXT[name]] if name in NEXT else [])
|
||||
errors = [op for op in got if op["op"] == "error"]
|
||||
assert [(op.pop("next"), op.pop("code", None)) for op in errors] == \
|
||||
([ERROR_NEXT[name]] if name in ERROR_NEXT else [])
|
||||
for op in errors:
|
||||
assert op["message"] == f"助手本轮未完成:{op.pop('reason')}", op
|
||||
assert got[:-1] == expected
|
||||
|
||||
|
||||
|
||||
@@ -774,6 +774,12 @@ export const TURN_COST_TEXT = {
|
||||
`其中 ${formatNumber(n)} 次调用的金额无法估算(模型不在价格目录中,或用量没有报全),未计入,实际花费不低于此数`,
|
||||
} as const
|
||||
|
||||
/** 问数据一轮自动重来一次时的进度说明(服务端 chat/turn.next_step 判的 retry) */
|
||||
export const CHAT_RETRY_TEXT = {
|
||||
/** 助手上一次只写了思考、一个操作都没给(error.next = retry) */
|
||||
empty: '助手上一次没有给出任何操作,正在重新生成工作流…',
|
||||
} as const
|
||||
|
||||
export const JUDGE_SETTING_TEXT = {
|
||||
label: '证据裁判',
|
||||
hint: '报告中的结论句(如「增长主要来自新客首单」)由另一个模型按证据逐句判断:正式运行由报告撰写节点当场裁判,探索运行在点击时逐句裁判。判断由模型给出,不是系统核对,界面上一律标注「非确定」。',
|
||||
|
||||
@@ -505,6 +505,8 @@ export const EVAL_TEXT = {
|
||||
filter: '只看',
|
||||
repeat: (n: number) => `第 ${n + 1} 次`,
|
||||
attempts: (n: number) => `这一题自动重来了 ${n} 次`,
|
||||
/** 每次自动重来的原因(服务端 eval/answer 的 turn.retries) */
|
||||
retriesWhy: (reasons: string[]) => `:${reasons.join(';')}`,
|
||||
turnReason: '作答的经过',
|
||||
answer: '答案原文',
|
||||
expand: '展开',
|
||||
|
||||
@@ -463,7 +463,11 @@ function ResultDetail({ evaluationId, resultId }: { evaluationId: string; result
|
||||
const R = T.results
|
||||
return (
|
||||
<div className="space-y-1" data-eval-result-details="">
|
||||
{(turn.attempts ?? 0) > 0 && <p className="text-faint">{R.attempts(turn.attempts)}</p>}
|
||||
{(turn.attempts ?? 0) > 0 && (
|
||||
<p className="text-faint" data-eval-retries={turn.retries?.length ?? 0}>
|
||||
{R.attempts(turn.attempts)}{turn.retries?.length ? R.retriesWhy(turn.retries) : ''}
|
||||
</p>
|
||||
)}
|
||||
{turn.reason && <p className="text-dim"><span className="text-faint">{R.turnReason}:</span>{turn.reason}</p>}
|
||||
{answer && (
|
||||
<details>
|
||||
|
||||
@@ -2,7 +2,7 @@ import { create } from 'zustand'
|
||||
import { ApiError, api, streamCopilot, streamRun } from '../api/client'
|
||||
import type { CopilotOp, FixKind, RunFinal } from '../run/decode'
|
||||
import { copilotContext, copilotLook, decodeRun, looksOf } from '../run/decode'
|
||||
import { CHAT_BLUEPRINT_TEXT, LEDGER_TEXT, LOOK_TEXT, REJECTED_TEXT } from '../lib/terms'
|
||||
import { CHAT_BLUEPRINT_TEXT, CHAT_RETRY_TEXT, LEDGER_TEXT, LOOK_TEXT, REJECTED_TEXT } from '../lib/terms'
|
||||
import type { LookEvent } from '../typesApp'
|
||||
import { errorMessage, humanizeError, isNetworkError } from '../lib/errors'
|
||||
import { guardReview, hasEvidence } from '../lib/evidence'
|
||||
@@ -1419,14 +1419,14 @@ function plan(flow: Flow, args: PlanArgs) {
|
||||
* 这里会清空这一次的过程和答案——它是自动补救,用户还没拿到过答案。用户主动
|
||||
* 点的重试走 retryTurn,上一次要留档。
|
||||
*/
|
||||
const retry = (reason: string, instruction?: string) => {
|
||||
const retry = (reason: string, instruction?: string, status?: string) => {
|
||||
// 服务端给了那句话(复核的 retry_instruction)就是它已经按次数判过可以重来;老服务端由这里按次数判
|
||||
if ((!instruction && attempt + 1 >= MAX_ATTEMPTS) || !flow.alive()) return false
|
||||
const base = built && (built.nodes?.length ?? 0) > 0 ? built : null
|
||||
patch((t) => ({
|
||||
attempt: attempt + 1,
|
||||
phase: 'planning',
|
||||
status: base ? '上次运行未成功,正在调整工作流…' : '上次运行未成功,正在重新生成工作流…',
|
||||
status: status ?? (base ? '上次运行未成功,正在调整工作流…' : '上次运行未成功,正在重新生成工作流…'),
|
||||
ops: [], events: [], run: null, output: null, final: undefined, runStatus: undefined,
|
||||
review: null, reviewRaw: null, rawOutput: null, lastSeq: 0, failure: null, error: '',
|
||||
// 换下来的那次运行花的钱还是这一轮的
|
||||
@@ -1586,9 +1586,12 @@ function plan(flow: Flow, args: PlanArgs) {
|
||||
void flow.persist()
|
||||
break
|
||||
case 'error':
|
||||
settled = true
|
||||
// 生成过程一个操作都没给(模型只写了思考):服务端说重来一次(error.next = retry,backend/app/chat/turn.py 的
|
||||
// next_step),照它带着原因重来,次数照 MAX_ATTEMPTS;试满了才照实报。这一次建图的用量随后那条 usage 照样记进这一轮
|
||||
if (op.next === 'retry' && retry(String(op.reason ?? op.message ?? ''), undefined, CHAT_RETRY_TEXT.empty)) break
|
||||
// 后端给的是三段:message 人话、hint 怎么办、detail 原始异常。
|
||||
// 原文不再截成一行甩给用户,收进可展开、可复制的技术细节
|
||||
settled = true
|
||||
fail({
|
||||
...failureOf(op.message ?? '生成失败'),
|
||||
...(op.hint ? { hint: String(op.hint) } : {}),
|
||||
|
||||
+49
-1
@@ -29,7 +29,8 @@
|
||||
// 14. 助手反问(normal-user-apps N11):只问不改的一轮记成「助手反问」,问题作为这一轮的内容、落库进历史,不再一直停在
|
||||
// 「正在理解需求」;输入框提示回答助手的问题;刷新后照样标「助手反问」,不当成没查库的回答;
|
||||
// 15. 回答上的「做成应用」带着这次运行进蓝图(建运行起点的蓝图),不再提取工作流、跳画布的发布弹窗;
|
||||
// 16. 服务端这一轮已经自查修正过(收到 check 的结论)还没过的:不再整轮重新生成一遍,直接说没过、给补救。
|
||||
// 16. 服务端这一轮已经自查修正过(收到 check 的结论)还没过的:不再整轮重新生成一遍,直接说没过、给补救;助手只写了思考、
|
||||
// 一个操作都没给(error.next=retry)的照服务端的决定自动重来一次,还不行再照实报。
|
||||
// 17. 口径账本(next-round 02):交付了的轮次底部写「正在比对口径」,按退避再取到比对出结果、比完不再取;一行摘要
|
||||
// (几个数来自哪份口径)、按口径正式出具和对账按钮照服务端给的动作摆;出具、对账的写请求在浏览器层拦下并核对请求体,
|
||||
// 出具过的正式运行挂在轮次下面(档位、打开运行)。
|
||||
@@ -532,6 +533,26 @@ async function fakeApi(route) {
|
||||
return route.fulfill({ status: 200, contentType: 'text/event-stream',
|
||||
body: ops.map((o) => `data: ${JSON.stringify(o)}\n\n`).join('') })
|
||||
}
|
||||
if (String(b.instruction ?? '').includes('只写了思考')) {
|
||||
// 生成过程一个操作都没给(deepseek 偶发只出思考):服务端的 error 带 next=retry(backend/app/chat/turn.py 的 next_step)。
|
||||
// 重来那一次(指令里缀了「上一次没有成功」)照常直接回答;「两次都」那一问每次都只写思考
|
||||
const again = String(b.instruction).includes('上一次没有成功') && !String(b.instruction).includes('两次都')
|
||||
const spent = { by_model: { m: { input_tokens: 500, output_tokens: 40, calls: 1, cost_usd: 0.001, unpriced_calls: 0 } },
|
||||
input_tokens: 500, output_tokens: 40, calls: 1, cost_usd: 0.001, unpriced_calls: 0 }
|
||||
const ops = again ? [
|
||||
{ op: 'heartbeat', phase: 'planning', elapsed_ms: 200 },
|
||||
{ op: 'reply', text: '已经查过了:一共 3 家门店。' },
|
||||
{ op: 'usage', ...spent },
|
||||
] : [
|
||||
{ op: 'heartbeat', phase: 'planning', elapsed_ms: 200 },
|
||||
{ op: 'thinking', delta: '先想一想要查哪张表……' },
|
||||
{ op: 'error', message: '助手本轮未完成:模型没有给出任何修改,也没有说明原因', hint: '画布未修改。可以重试,或把需求描述得更具体',
|
||||
detail: '', reason: '模型没有给出任何修改,也没有说明原因', code: 'copilot_empty', next: 'retry' },
|
||||
{ op: 'usage', ...spent },
|
||||
]
|
||||
return route.fulfill({ status: 200, contentType: 'text/event-stream',
|
||||
body: ops.map((o) => `data: ${JSON.stringify(o)}\n\n`).join('') })
|
||||
}
|
||||
if (String(b.instruction ?? '').includes('照服务端的决定') || String(b.instruction ?? '').includes(DECIDE_RETRY)) {
|
||||
// 新服务端:final 带 next(backend/app/chat/turn.py),最后一条是这一轮建图的用量(answer_ops 的 usage)。
|
||||
// 「只建不跑」那一问:autorun 没写 false,next 却是 ready——前端得照 next 走
|
||||
@@ -1858,6 +1879,33 @@ for (const theme of THEMES) {
|
||||
await ctx.close()
|
||||
})
|
||||
|
||||
await section('emptyretry', '助手只写了思考、一个操作都没给:照服务端的 next=retry 自动重来一次,还不行再照实报', async () => {
|
||||
const { page, ctx, errors } = await open(theme)
|
||||
await goto(page, 'c0emptg')
|
||||
let gens = log.gens.length
|
||||
await send(page, '只写了思考:有几家门店')
|
||||
const done = await page.waitForFunction(() => window.__chat.getState().byConversation.c0emptg?.at(-1)?.phase === 'done',
|
||||
null, { timeout: 6000 }).then(() => true, () => false)
|
||||
await page.waitForTimeout(300)
|
||||
const turn = (await chatState(page)).byConversation.c0emptg.at(-1)
|
||||
check('第一次没给操作:自动重来一次,第二次照常回答', done && log.gens.length - gens === 2
|
||||
&& String(turn.output?.answer ?? '').includes('3 家'), `${log.gens.length - gens} 次生成 · ${JSON.stringify(turn.output)}`)
|
||||
check('重来时交回去的话带着原因(同服务端 retry_instruction 的从零重建措辞)',
|
||||
log.gens.at(-1)?.instruction === '只写了思考:有几家门店\n\n(上一次没有成功:模型没有给出任何修改,也没有说明原因。请针对这个原因调整,重新给出完整的工作流。)',
|
||||
log.gens.at(-1)?.instruction)
|
||||
check('两次建图的用量都记进这一轮', Math.abs((turn.usage?.copilot?.cost_usd ?? 0) - 0.002) < 1e-9, JSON.stringify(turn.usage))
|
||||
gens = log.gens.length
|
||||
await send(page, '两次都只写了思考:有几家门店')
|
||||
const failed = await page.waitForFunction(() => window.__chat.getState().byConversation.c0emptg?.at(-1)?.phase === 'error',
|
||||
null, { timeout: 6000 }).then(() => true, () => false)
|
||||
await page.waitForTimeout(500)
|
||||
const last = (await chatState(page)).byConversation.c0emptg.at(-1)
|
||||
check('重来还不行:照实报「助手本轮未完成」,不再试第三次', failed && log.gens.length - gens === 2
|
||||
&& JSON.stringify(last.failure ?? {}).includes('助手本轮未完成'), `${log.gens.length - gens} 次生成 · ${JSON.stringify(last.failure)}`)
|
||||
check('没有运行时报错', errors.length === 0, errors[0] ?? '')
|
||||
await ctx.close()
|
||||
})
|
||||
|
||||
await section('decide', '新服务端:照 final.next 和复核的 retry_instruction 走,这一轮的花费 = 助手 + 运行 + 复核', async () => {
|
||||
const { page, ctx, errors } = await open(theme)
|
||||
await goto(page, 'c0emptg')
|
||||
|
||||
Reference in New Issue
Block a user