fix: worker 代码变化自退 + 部署时清孤儿 worker
2026-10-27 暂存清理实弹:05:51 的旧 docker worker 成孤儿后 继续用旧代码抢任务、按老路失败且不删暂存;autoscaler 用 start_new_session=True 拉 worker、部署重启杀不到非 unit 进程。 两层修: 1. worker 常驻循环每轮边界比对 worker.py 文件签名 (mtime_ns + size),部署换目录后必变,变了就以 75 (EX_TEMPFAIL,失败类)退出——检查点在轮次边界,绝不 中断在跑任务。退出后:unit 托管的由 systemd on-failure 重拉新版,autoscaler 托管的由 reconcile 重拉,孤儿直接 消失。附带好处:以后部署不用再手动重启本地 worker。 2. host_deploy.sh 换目录后、启动服务前清孤儿:按 cgroup 过滤杀非 .service 的 cloud.worker 进程(桥接修补前就存 在的孤儿),先 TERM(跑完当前任务再退),30 秒未退的 再 KILL;unit 托管的(如 host-1)不动。pgrep 用 [c]loud.worker 写法,避免检查命令自匹配。 测试:test_worker.py 新增 2 个(签名变了则 75 退出且不认领、 签名不变则跑到 max_seconds);反向验证:撤掉 worker 修复 时正好前者变红。清扫过滤器主机干跑:两个现役 worker 均 为 unit cgroup,全 KEEP,杀 0 个。全量 deploy 19、host 62、cloud 619 全过。
This commit is contained in:
1 parent
c007fafcc7
commit
57f8852634
3 files changed
+86
No files matched your search
@@ -185,6 +185,41 @@ if id codexsvc >/dev/null 2>&1; then
|
||||
chown -R codexsvc:codexsvc "$APP_DIR"
|
||||
fi
|
||||
|
||||
# --- 5b. 清孤儿 worker ---------------------------------------------------
|
||||
# 非 systemd unit cgroup 的 cloud.worker 进程(手动/setsid 拉起的旧
|
||||
# worker):部署重启杀不到它(2026-10-27 实弹:05:51 的旧 docker
|
||||
# worker 成孤儿后继续用旧代码抢任务、按老路失败且不删暂存)。
|
||||
# unit 托管的 worker(cgroup 含 .service,如 codex-host-worker)
|
||||
# 不动:它们靠新版 worker 的代码变化自退 + systemd on-failure 重拉。
|
||||
# 先 TERM(worker 收到后停认领、跑完当前任务再退),30 秒内未退
|
||||
# 的再 KILL;被中断的在跑任务由陈旧回收标 interrupted,可续跑。
|
||||
for _pid in $(pgrep -f "[c]loud.worker" 2>/dev/null || true); do
|
||||
_cg=$(cat /proc/$_pid/cgroup 2>/dev/null || true)
|
||||
case "$_cg" in
|
||||
*".service"*) ;;
|
||||
*) echo "清孤儿 worker:pid=$_pid"; kill $_pid 2>/dev/null || true ;;
|
||||
esac
|
||||
done
|
||||
for _i in $(seq 1 30); do
|
||||
_left=0
|
||||
for _pid in $(pgrep -f "[c]loud.worker" 2>/dev/null || true); do
|
||||
_cg=$(cat /proc/$_pid/cgroup 2>/dev/null || true)
|
||||
case "$_cg" in
|
||||
*".service"*) ;;
|
||||
*) _left=1 ;;
|
||||
esac
|
||||
done
|
||||
[ "$_left" = "0" ] && break
|
||||
sleep 1
|
||||
done
|
||||
for _pid in $(pgrep -f "[c]loud.worker" 2>/dev/null || true); do
|
||||
_cg=$(cat /proc/$_pid/cgroup 2>/dev/null || true)
|
||||
case "$_cg" in
|
||||
*".service"*) ;;
|
||||
*) echo "孤儿 worker 未退,KILL:pid=$_pid"; kill -9 $_pid 2>/dev/null || true ;;
|
||||
esac
|
||||
done
|
||||
|
||||
rollback() {
|
||||
echo "新代码 health 未通过,开始回滚到备份:$BACKUP_FILE" >&2
|
||||
systemctl stop "$WORKER_UNIT" $SERVER_UNITS || true
|
||||
|
||||
Reference in new issue
Block a user