Autoscale workers by queue depth (adaptive threads + process scaler)

Layer 1, in-worker: run_all pool thread count is now adaptive
per round (min(capacity, ready_count)) instead of always
capacity threads; ready_count counts only dependency-satisfied
queued tasks via a shared _ready_tasks() helper (pick_next
refactored onto it), so blocked tasks never inflate the pool.

Layer 2, process scaler (cloud/autoscaler.py): a systemd-managed
daemon that starts/stops cloud.worker subprocesses by ready
queue depth. desired = ceil(depth / per_worker_capacity)
clamped to [min_workers, max_workers]; scale-up is immediate,
scale-down waits scale_down_after consecutive low checks
(default 3) to avoid flapping on bursty completions. Scale-down
is LIFO (youngest worker first), SIGTERM for graceful drain
(worker run_loop drains up to drain_timeout), SIGKILL of the
process group only after drain timeout. Depth read failure
keeps the current worker count (never scale on unknown state).
Worker command forces --loop/--worker-id/--worker-capacity
(appended last so argparse last-wins); --db comes from scaler
or worker args; workers inherit scaler env (Postgres etc.).
State file (depth/desired/actual/workers) written atomically
for observability. deploy.sh now restarts codex-autoscaler
instead of the static worker unit when the scaler is active,
so the two never poll the same queue at once.
Tests: 8 autoscaler tests (desired math, ready-only depth,
cycle=0, scale up/down, cooldown streak, forced worker flags).

deploy/autoscaler/codex-autoscaler.service

@@ -0,0 +1,25 @@
+[Unit]
+Description=Codex cloud autoscaler (scales worker processes by queue depth)
+After=codex-cloud-server.service
+Wants=codex-cloud-server.service
+
+[Service]
+Type=simple
+User=codexsvc
+Group=codexsvc
+WorkingDirectory=/opt/codex-app-server
+EnvironmentFile=/etc/codex-app-server/env
+# scaler 自身参数在前，`--` 之后是传给 cloud.worker 的参数。
+# 不要在这里写 --worker-id / --worker-capacity / --loop，
+# scaler 会按实例强制追加；--db 只给 scaler，scaler 会补给 worker。
+# 注意：启用 autoscaler 后必须停掉静态 codex-cloud-worker，
+# 否则两边同时派任务（scaler 只管理 autoscale-* 实例）。
+ExecStart=/usr/bin/python3 -m cloud.autoscaler --db /var/lib/codex-app-server/control.db --interval 30 --min-workers 1 --max-workers 2 --per-worker-capacity 2 --scale-down-after 3 --drain-timeout 60 --state-file /var/lib/codex-app-server/autoscaler-state.json --log-dir /var/lib/codex-app-server/autoscaler-logs -- --interval 10 --config /opt/codex-app-server/host-config-docker.json --executor docker --docker-image codex-app-server:latest --docker-user 1003:1004 --docker-cpus 2 --docker-memory 2g --docker-extra-mounts /opt/codex-app-server/work:/app/work,/var/lib/codex-app-server/codex-home:/var/lib/codex-app-server/codex-home,/var/lib/codex-app-server/repos:/var/lib/codex-app-server/repos,/var/lib/codex-app-server/test-repo:/var/lib/codex-app-server/test-repo --repos-dir /var/lib/codex-app-server/repos
+Restart=on-failure
+RestartSec=10
+# 停 scaler 时要等它把 worker 优雅排空（drain 60 秒 + 缓冲），
+# 默认 90 秒不够，放到 180 秒；超时后 systemd 再 SIGKILL。
+TimeoutStopSec=180
+
+[Install]
+WantedBy=multi-user.target

deploy/deploy.sh

@@ -138,7 +138,15 @@ tar -xzf /tmp/codex-app-server-code.tar.gz -C "$APP_DIR"
 chown -R codexsvc:codexsvc "$APP_DIR"
 
 # 3. 重启并验 health
-systemctl restart codex-cloud-server codex-cloud-worker
+# autoscaler 接管后，worker 由它拉起（autoscale-* 实例），
+# 静态 codex-cloud-worker 必须保持停用，否则两边同时派任务。
+# 这里按 autoscaler 是否在跑决定重启哪一边；回滚路径同理。
+if systemctl is-active -q codex-autoscaler; then
+  WORKER_UNIT=codex-autoscaler
+else
+  WORKER_UNIT=codex-cloud-worker
+fi
+systemctl restart codex-cloud-server $WORKER_UNIT
 if wait_health; then
   echo "部署成功，health 通过"
   exit 0
@@ -146,11 +154,11 @@ fi
 
 # 4. 失败：恢复备份并重启，再验一次
 echo "新代码 health 未通过，开始回滚到备份：$BACKUP_FILE"
-systemctl stop codex-cloud-worker codex-cloud-server || true
+systemctl stop $WORKER_UNIT codex-cloud-server || true
 find "$APP_DIR" -mindepth 1 -maxdepth 1 ! -name work -exec rm -rf {} +
 tar -xzf "$BACKUP_FILE" -C /opt
 chown -R codexsvc:codexsvc "$APP_DIR"
-systemctl start codex-cloud-server codex-cloud-worker
+systemctl start codex-cloud-server $WORKER_UNIT
 if wait_health; then
   echo "部署失败，已回滚到备份，health 已恢复（新代码未上线）" >&2
   exit 1