feat: 长任务无敌补齐三件:连续失败上限 + worker退出审计 + 配额恢复自动续跑

1. host goal 循环连续失败护栏:3 轮连续失败即中断出口,
   任务记 interrupted、可断点续跑,不再无限空转
   (2026-10-29 实弹:断点后 274 轮全失败)。
2. worker 退出杀残留路径补审计 worker_shutdown_interrupted,
   detail 为任务 id,与 sweep_interrupted 同口径。
3. 配额中断自动恢复:每轮现查配额,仍超限/查不到时留
   在 interrupted 不重排队,下轮再查;现查为 None 时
   (新月份/额度上调/无价上限重置)才 CAS 回 queued。

单测:配额仍超限不动、配额恢复续跑、连续 3 轮失败停转。
This commit is contained in:
host committed 2026-10-29 13:07:57 +08:00
1 parent 88fcfeebd3
commit ae3aff8602
4 files changed
+100 -6

No files matched your search

+13
View File
@@ -1944,6 +1944,12 @@ def run_goal_loop(server, state, state_path, task, goal_text, verify,
# max_rounds=0 是不限轮次:while 循环只在模型宣告、预算耗尽
# 或取消时结束;非 0 时按旧语义穷尽轮次即 unmet。
round_no = int(start_round) - 1
# 连续失败轮次护栏:回合瞬间失败(如密钥缺失)时不限轮次
# 循环会无限空转(2026-10-29 实弹:断点后 274 轮全失败,
# state 文件持续增长)。3 轮连续失败即走中断出口,任务
# 记 interrupted、可断点续跑,不再原地空转;偶发单轮失败
# 不受影响(计数在成功轮清零)。
consecutive_failed = 0
while True:
round_no += 1
if int(max_rounds) != 0 and round_no > int(max_rounds):
@@ -2002,6 +2008,13 @@ def run_goal_loop(server, state, state_path, task, goal_text, verify,
goal_usage = _add_usage(goal_usage, result.get("usage"))
last_turn_status = result["status"]
last_turn_text = result.get("text", "") or ""
if last_turn_status == "failed":
consecutive_failed += 1
if consecutive_failed >= 3:
return _interrupt_loop(
Exception("连续 3 轮回合失败,停止空转"))
else:
consecutive_failed = 0
if verify is None:
# 新形态没有验收命令可跑,v_rc 记 -(轮次行格式不变,
# 云端解析只读 verify_out 做 unmet 详情,不受影响)。