Autoscale workers by queue depth (adaptive threads + process scaler)

Layer 1, in-worker: run_all pool thread count is now adaptive
per round (min(capacity, ready_count)) instead of always
capacity threads; ready_count counts only dependency-satisfied
queued tasks via a shared _ready_tasks() helper (pick_next
refactored onto it), so blocked tasks never inflate the pool.

Layer 2, process scaler (cloud/autoscaler.py): a systemd-managed
daemon that starts/stops cloud.worker subprocesses by ready
queue depth. desired = ceil(depth / per_worker_capacity)
clamped to [min_workers, max_workers]; scale-up is immediate,
scale-down waits scale_down_after consecutive low checks
(default 3) to avoid flapping on bursty completions. Scale-down
is LIFO (youngest worker first), SIGTERM for graceful drain
(worker run_loop drains up to drain_timeout), SIGKILL of the
process group only after drain timeout. Depth read failure
keeps the current worker count (never scale on unknown state).
Worker command forces --loop/--worker-id/--worker-capacity
(appended last so argparse last-wins); --db comes from scaler
or worker args; workers inherit scaler env (Postgres etc.).
State file (depth/desired/actual/workers) written atomically
for observability. deploy.sh now restarts codex-autoscaler
instead of the static worker unit when the scaler is active,
so the two never poll the same queue at once.
Tests: 8 autoscaler tests (desired math, ready-only depth,
cycle=0, scale up/down, cooldown streak, forced worker flags).

deploy/autoscaler/codex-autoscaler.service

@@ -0,0 +1,25 @@
+[Unit]
+Description=Codex cloud autoscaler (scales worker processes by queue depth)
+After=codex-cloud-server.service
+Wants=codex-cloud-server.service
+
+[Service]
+Type=simple
+User=codexsvc
+Group=codexsvc
+WorkingDirectory=/opt/codex-app-server
+EnvironmentFile=/etc/codex-app-server/env
+# scaler 自身参数在前，`--` 之后是传给 cloud.worker 的参数。
+# 不要在这里写 --worker-id / --worker-capacity / --loop，
+# scaler 会按实例强制追加；--db 只给 scaler，scaler 会补给 worker。
+# 注意：启用 autoscaler 后必须停掉静态 codex-cloud-worker，
+# 否则两边同时派任务（scaler 只管理 autoscale-* 实例）。
+ExecStart=/usr/bin/python3 -m cloud.autoscaler --db /var/lib/codex-app-server/control.db --interval 30 --min-workers 1 --max-workers 2 --per-worker-capacity 2 --scale-down-after 3 --drain-timeout 60 --state-file /var/lib/codex-app-server/autoscaler-state.json --log-dir /var/lib/codex-app-server/autoscaler-logs -- --interval 10 --config /opt/codex-app-server/host-config-docker.json --executor docker --docker-image codex-app-server:latest --docker-user 1003:1004 --docker-cpus 2 --docker-memory 2g --docker-extra-mounts /opt/codex-app-server/work:/app/work,/var/lib/codex-app-server/codex-home:/var/lib/codex-app-server/codex-home,/var/lib/codex-app-server/repos:/var/lib/codex-app-server/repos,/var/lib/codex-app-server/test-repo:/var/lib/codex-app-server/test-repo --repos-dir /var/lib/codex-app-server/repos
+Restart=on-failure
+RestartSec=10
+# 停 scaler 时要等它把 worker 优雅排空（drain 60 秒 + 缓冲），
+# 默认 90 秒不够，放到 180 秒；超时后 systemd 再 SIGKILL。
+TimeoutStopSec=180
+
+[Install]
+WantedBy=multi-user.target