feat: auto-resume platform-interrupted non-goal tasks

Deploy restarts kill running tasks: the worker marks them
interrupted (worker_shutdown on graceful exit, sweep on stale
reclaim), and auto_resume_interrupted used to accept goal tasks
only, so every non-goal task killed by a deploy stayed
interrupted forever unless a human retried it (the 2026-10-30
demo task died this way).

Now worker_shutdown/sweep interruptions are re-queued for
non-goal tasks too: the platform killed the task, so the
platform re-queues it. The re-queue is a fresh run, not a
breakpoint resume: goal_resume_pending is forced to 0 and any
stale resume_message is cleared, so the new run cannot be
hijacked by the goal-resume path or an old continue message;
files in the same worktree persist. All existing guards stay:
user pause, coordinator orphans, quota (still goal-only),
backoff, the 3-strike stall guard (non-goal tasks have no
checkpoints, so each platform re-queue counts as no-progress),
and the auto_resume_max cap. Other non-goal reasons (NULL,
quota, user cancel) still do not auto-resume.
This commit is contained in:
host committed 2026-10-30 15:39:55 +08:00
1 parent e2ce04ce4f
commit 4428a721a6
2 files changed
+86 -12

No files matched your search

+40 -11
View File
@@ -923,7 +923,8 @@ class Worker:
pass
def auto_resume_interrupted(self):
"""长任务自动恢复:只接系统中断的 goal 任务。
"""长任务自动恢复:接系统中断的 goal 任务,以及平台
中断的非 goal 任务。
只有用户暂停才暂停:paused_by_user=1(/pause、/cancel)
的任务绝不自动动;协调员孤儿(coordinator_orphan)
@@ -934,8 +935,18 @@ class Worker:
下一轮再查;查不到配额或仍超限都不接。恢复的判定
是现查结果为 None(新月份、额度上调、无价上限重置
都算恢复),不是猜时间。
非 goal 任务不接:它没有断点语义,重跑等于从头再来,
可能重复副作用,留给人工 /retry 决定。
非 goal 任务只接平台中断:interrupted_reason 为
worker_shutdown(worker 关停,如部署重启)或 sweep
(server 侧陈旧回收,如 worker 被强杀)时,平台杀了
任务、平台负责重排;重排是全新跑(不置
goal_resume_pending、不走断点,同一 worktree 里文件
还在),并清掉可能残留的 resume_message,防旧续跑
消息劫持新跑的 prompt。其它原因(NULL、用户取消等)
的非 goal 任务仍不接,留给人工 /retry 决定。
注意:非 goal 任务没有断点,断点轮次恒为 -1,因此
每次平台中断重排都会计一次无进展——3 次重排后第 4
次平台中断即 stall 停住等人工,这是防 crash-重排死
循环的同一道闸,不是 bug。
接到的任务 CAS 回 queued 并置 goal_resume_pending=1,
下一轮认领即走 G10 断点续跑(同一线程、同一 worktree、
从 goal_rounds_done 下一轮起),同时清旧结果、清
@@ -962,15 +973,18 @@ class Worker:
"SELECT id, project_id, created_by, parent_task_id,"
" model, interrupted_reason, resume_after,"
" auto_resume_stall_count,"
" checkpoint_round_at_last_resume FROM tasks"
" checkpoint_round_at_last_resume,"
" goal_mode, verify_command, goal FROM tasks"
" WHERE status = 'interrupted'"
" AND (paused_by_user IS NULL OR paused_by_user = 0)"
" AND (interrupted_reason IS NULL OR"
" interrupted_reason NOT IN"
" ('coordinator_orphan'))"
" AND parent_task_id IS NULL"
" AND (goal_mode = 1 OR verify_command IS NOT NULL"
" AND ((goal_mode = 1 OR verify_command IS NOT NULL"
" OR goal IS NOT NULL)"
" OR interrupted_reason IN"
" ('worker_shutdown', 'sweep'))"
" AND (auto_resume_count IS NULL OR"
" auto_resume_count < ?)",
(self.auto_resume_max,),
@@ -1002,6 +1016,8 @@ class Worker:
except Exception:
cp_round = -1
prev = row[8]
is_goal = (row[9] == 1 or row[10] is not None
or row[11] is not None)
stall = 0
if prev is not None:
stall = (row[7] or 0) + 1 if cp_round == prev else 0
@@ -1019,10 +1035,17 @@ class Worker:
task_id)
continue
backoff = min(300, 10 * (2 ** stall))
if is_goal:
resume_set = "goal_resume_pending = 1"
else:
# 全新跑:不置断点标记,并清掉残留的续跑
# 消息,防它劫持新跑的 prompt。
resume_set = ("goal_resume_pending = 0,"
" resume_message = NULL")
cur = conn.execute(
"UPDATE tasks SET status = 'queued',"
" goal_resume_pending = 1,"
" result_text = NULL, error_text = NULL,"
"UPDATE tasks SET status = 'queued', "
+ resume_set +
", result_text = NULL, error_text = NULL,"
" actual_model = NULL,"
" interrupted_reason = NULL,"
" auto_resume_count ="
@@ -1040,10 +1063,16 @@ class Worker:
resumed.append(task_id)
self._audit(project_id, "worker_auto_resumed",
task_id)
if is_goal:
summary = ("自动续跑(无进展计数 %d,"
"下次退避 %d 秒)"
% (stall, backoff))
else:
summary = ("平台中断自动重排(%s,全新跑,"
"无进展计数 %d,下次退避 %d 秒)"
% (row[5], stall, backoff))
self._record_step(
conn, task_id, "auto_resume",
"自动续跑(无进展计数 %d,下次退避 %d 秒)"
% (stall, backoff))
conn, task_id, "auto_resume", summary)
conn.commit()
finally:
conn.close()