host
7e147be9ed
Layer 1, in-worker: run_all pool thread count is now adaptive per round (min(capacity, ready_count)) instead of always capacity threads; ready_count counts only dependency-satisfied queued tasks via a shared _ready_tasks() helper (pick_next refactored onto it), so blocked tasks never inflate the pool. Layer 2, process scaler (cloud/autoscaler.py): a systemd-managed daemon that starts/stops cloud.worker subprocesses by ready queue depth. desired = ceil(depth / per_worker_capacity) clamped to [min_workers, max_workers]; scale-up is immediate, scale-down waits scale_down_after consecutive low checks (default 3) to avoid flapping on bursty completions. Scale-down is LIFO (youngest worker first), SIGTERM for graceful drain (worker run_loop drains up to drain_timeout), SIGKILL of the process group only after drain timeout. Depth read failure keeps the current worker count (never scale on unknown state). Worker command forces --loop/--worker-id/--worker-capacity (appended last so argparse last-wins); --db comes from scaler or worker args; workers inherit scaler env (Postgres etc.). State file (depth/desired/actual/workers) written atomically for observability. deploy.sh now restarts codex-autoscaler instead of the static worker unit when the scaler is active, so the two never poll the same queue at once. Tests: 8 autoscaler tests (desired math, ready-only depth, cycle=0, scale up/down, cooldown streak, forced worker flags).
26 lines
1.7 KiB
Desktop File
26 lines
1.7 KiB
Desktop File
[Unit]
|
||
Description=Codex cloud autoscaler (scales worker processes by queue depth)
|
||
After=codex-cloud-server.service
|
||
Wants=codex-cloud-server.service
|
||
|
||
[Service]
|
||
Type=simple
|
||
User=codexsvc
|
||
Group=codexsvc
|
||
WorkingDirectory=/opt/codex-app-server
|
||
EnvironmentFile=/etc/codex-app-server/env
|
||
# scaler 自身参数在前，`--` 之后是传给 cloud.worker 的参数。
|
||
# 不要在这里写 --worker-id / --worker-capacity / --loop，
|
||
# scaler 会按实例强制追加；--db 只给 scaler，scaler 会补给 worker。
|
||
# 注意：启用 autoscaler 后必须停掉静态 codex-cloud-worker，
|
||
# 否则两边同时派任务（scaler 只管理 autoscale-* 实例）。
|
||
ExecStart=/usr/bin/python3 -m cloud.autoscaler --db /var/lib/codex-app-server/control.db --interval 30 --min-workers 1 --max-workers 2 --per-worker-capacity 2 --scale-down-after 3 --drain-timeout 60 --state-file /var/lib/codex-app-server/autoscaler-state.json --log-dir /var/lib/codex-app-server/autoscaler-logs -- --interval 10 --config /opt/codex-app-server/host-config-docker.json --executor docker --docker-image codex-app-server:latest --docker-user 1003:1004 --docker-cpus 2 --docker-memory 2g --docker-extra-mounts /opt/codex-app-server/work:/app/work,/var/lib/codex-app-server/codex-home:/var/lib/codex-app-server/codex-home,/var/lib/codex-app-server/repos:/var/lib/codex-app-server/repos,/var/lib/codex-app-server/test-repo:/var/lib/codex-app-server/test-repo --repos-dir /var/lib/codex-app-server/repos
|
||
Restart=on-failure
|
||
RestartSec=10
|
||
# 停 scaler 时要等它把 worker 优雅排空（drain 60 秒 + 缓冲），
|
||
# 默认 90 秒不够，放到 180 秒；超时后 systemd 再 SIGKILL。
|
||
TimeoutStopSec=180
|
||
|
||
[Install]
|
||
WantedBy=multi-user.target
|