feat: auto-resume platform-interrupted non-goal tasks
Deploy restarts kill running tasks: the worker marks them interrupted (worker_shutdown on graceful exit, sweep on stale reclaim), and auto_resume_interrupted used to accept goal tasks only, so every non-goal task killed by a deploy stayed interrupted forever unless a human retried it (the 2026-10-30 demo task died this way). Now worker_shutdown/sweep interruptions are re-queued for non-goal tasks too: the platform killed the task, so the platform re-queues it. The re-queue is a fresh run, not a breakpoint resume: goal_resume_pending is forced to 0 and any stale resume_message is cleared, so the new run cannot be hijacked by the goal-resume path or an old continue message; files in the same worktree persist. All existing guards stay: user pause, coordinator orphans, quota (still goal-only), backoff, the 3-strike stall guard (non-goal tasks have no checkpoints, so each platform re-queue counts as no-progress), and the auto_resume_max cap. Other non-goal reasons (NULL, quota, user cancel) still do not auto-resume.
This commit is contained in:
1 parent
e2ce04ce4f
commit
4428a721a6
2 files changed
+86
-12
No files matched your search
+40
-11
@@ -923,7 +923,8 @@ class Worker:
|
||||
pass
|
||||
|
||||
def auto_resume_interrupted(self):
|
||||
"""长任务自动恢复:只接系统中断的 goal 任务。
|
||||
"""长任务自动恢复:接系统中断的 goal 任务,以及平台
|
||||
中断的非 goal 任务。
|
||||
|
||||
只有用户暂停才暂停:paused_by_user=1(/pause、/cancel)
|
||||
的任务绝不自动动;协调员孤儿(coordinator_orphan)
|
||||
@@ -934,8 +935,18 @@ class Worker:
|
||||
下一轮再查;查不到配额或仍超限都不接。恢复的判定
|
||||
是现查结果为 None(新月份、额度上调、无价上限重置
|
||||
都算恢复),不是猜时间。
|
||||
非 goal 任务不接:它没有断点语义,重跑等于从头再来,
|
||||
可能重复副作用,留给人工 /retry 决定。
|
||||
非 goal 任务只接平台中断:interrupted_reason 为
|
||||
worker_shutdown(worker 关停,如部署重启)或 sweep
|
||||
(server 侧陈旧回收,如 worker 被强杀)时,平台杀了
|
||||
任务、平台负责重排;重排是全新跑(不置
|
||||
goal_resume_pending、不走断点,同一 worktree 里文件
|
||||
还在),并清掉可能残留的 resume_message,防旧续跑
|
||||
消息劫持新跑的 prompt。其它原因(NULL、用户取消等)
|
||||
的非 goal 任务仍不接,留给人工 /retry 决定。
|
||||
注意:非 goal 任务没有断点,断点轮次恒为 -1,因此
|
||||
每次平台中断重排都会计一次无进展——3 次重排后第 4
|
||||
次平台中断即 stall 停住等人工,这是防 crash-重排死
|
||||
循环的同一道闸,不是 bug。
|
||||
接到的任务 CAS 回 queued 并置 goal_resume_pending=1,
|
||||
下一轮认领即走 G10 断点续跑(同一线程、同一 worktree、
|
||||
从 goal_rounds_done 下一轮起),同时清旧结果、清
|
||||
@@ -962,15 +973,18 @@ class Worker:
|
||||
"SELECT id, project_id, created_by, parent_task_id,"
|
||||
" model, interrupted_reason, resume_after,"
|
||||
" auto_resume_stall_count,"
|
||||
" checkpoint_round_at_last_resume FROM tasks"
|
||||
" checkpoint_round_at_last_resume,"
|
||||
" goal_mode, verify_command, goal FROM tasks"
|
||||
" WHERE status = 'interrupted'"
|
||||
" AND (paused_by_user IS NULL OR paused_by_user = 0)"
|
||||
" AND (interrupted_reason IS NULL OR"
|
||||
" interrupted_reason NOT IN"
|
||||
" ('coordinator_orphan'))"
|
||||
" AND parent_task_id IS NULL"
|
||||
" AND (goal_mode = 1 OR verify_command IS NOT NULL"
|
||||
" AND ((goal_mode = 1 OR verify_command IS NOT NULL"
|
||||
" OR goal IS NOT NULL)"
|
||||
" OR interrupted_reason IN"
|
||||
" ('worker_shutdown', 'sweep'))"
|
||||
" AND (auto_resume_count IS NULL OR"
|
||||
" auto_resume_count < ?)",
|
||||
(self.auto_resume_max,),
|
||||
@@ -1002,6 +1016,8 @@ class Worker:
|
||||
except Exception:
|
||||
cp_round = -1
|
||||
prev = row[8]
|
||||
is_goal = (row[9] == 1 or row[10] is not None
|
||||
or row[11] is not None)
|
||||
stall = 0
|
||||
if prev is not None:
|
||||
stall = (row[7] or 0) + 1 if cp_round == prev else 0
|
||||
@@ -1019,10 +1035,17 @@ class Worker:
|
||||
task_id)
|
||||
continue
|
||||
backoff = min(300, 10 * (2 ** stall))
|
||||
if is_goal:
|
||||
resume_set = "goal_resume_pending = 1"
|
||||
else:
|
||||
# 全新跑:不置断点标记,并清掉残留的续跑
|
||||
# 消息,防它劫持新跑的 prompt。
|
||||
resume_set = ("goal_resume_pending = 0,"
|
||||
" resume_message = NULL")
|
||||
cur = conn.execute(
|
||||
"UPDATE tasks SET status = 'queued',"
|
||||
" goal_resume_pending = 1,"
|
||||
" result_text = NULL, error_text = NULL,"
|
||||
"UPDATE tasks SET status = 'queued', "
|
||||
+ resume_set +
|
||||
", result_text = NULL, error_text = NULL,"
|
||||
" actual_model = NULL,"
|
||||
" interrupted_reason = NULL,"
|
||||
" auto_resume_count ="
|
||||
@@ -1040,10 +1063,16 @@ class Worker:
|
||||
resumed.append(task_id)
|
||||
self._audit(project_id, "worker_auto_resumed",
|
||||
task_id)
|
||||
if is_goal:
|
||||
summary = ("自动续跑(无进展计数 %d,"
|
||||
"下次退避 %d 秒)"
|
||||
% (stall, backoff))
|
||||
else:
|
||||
summary = ("平台中断自动重排(%s,全新跑,"
|
||||
"无进展计数 %d,下次退避 %d 秒)"
|
||||
% (row[5], stall, backoff))
|
||||
self._record_step(
|
||||
conn, task_id, "auto_resume",
|
||||
"自动续跑(无进展计数 %d,下次退避 %d 秒)"
|
||||
% (stall, backoff))
|
||||
conn, task_id, "auto_resume", summary)
|
||||
conn.commit()
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
Reference in new issue
Block a user