Two veto layers in reconcile(), worker.py untouched:
- Global live-coordinator veto: a running task whose
coordinator_parts parse (via cloud.coordinator.parse_parts)
to a non-empty list and whose last_heartbeat is within
120s blocks all scale-down this round; low_streak is
frozen, not reset, so scale-down resumes on the next
low-depth round once the veto lifts.
- Per-victim busy veto: a LIFO victim whose workers row
is alive with running>0 is skipped, but only when the
DB also has a live running task (ghost registry rows
with no live DB task do not block).
- Coordinator-veto query failure fails closed (no stop,
guard error recorded); victim-veto read failure fails
open for that layer only.
- A missing/stale workers row is treated as idle.
Tests: 7 new ScaleDownGuardTestCase cases (live/stale
coordinator, non-coordinator, invalid parts, busy/idle
victims, ghost registry, streak freeze); cloud suite
1077 tests OK (4 skipped).
queue_depth turned a NULL depends_on into [None], and
ready_tasks then required None to be in completed_ids, so
such tasks never counted as ready. Worker pick_next was
unaffected (_task_from_row maps NULL to []), which is why
tasks still ran while the scaler always saw depth 0 and
never scaled up (proven live: depth probe with 1 queued
task returned 0). NULL now means no dependencies, with a
regression test.
setup_logging() takes a log LEVEL, not a logger name; passing
'cloud.autoscaler' raised ValueError at startup, so the scaler
crash-looped under systemd and started zero workers (the host
had no worker at all until this fix). The '/opt/.../-m'
FileNotFoundError in the same journal lines is only Ubuntu's
apport hook misfiring on 'python3 -m', not a second bug.
Regression test: main() with mocked run_loop/signal must reach
the loop (this would have caught the crash before deploy).
Layer 1, in-worker: run_all pool thread count is now adaptive
per round (min(capacity, ready_count)) instead of always
capacity threads; ready_count counts only dependency-satisfied
queued tasks via a shared _ready_tasks() helper (pick_next
refactored onto it), so blocked tasks never inflate the pool.
Layer 2, process scaler (cloud/autoscaler.py): a systemd-managed
daemon that starts/stops cloud.worker subprocesses by ready
queue depth. desired = ceil(depth / per_worker_capacity)
clamped to [min_workers, max_workers]; scale-up is immediate,
scale-down waits scale_down_after consecutive low checks
(default 3) to avoid flapping on bursty completions. Scale-down
is LIFO (youngest worker first), SIGTERM for graceful drain
(worker run_loop drains up to drain_timeout), SIGKILL of the
process group only after drain timeout. Depth read failure
keeps the current worker count (never scale on unknown state).
Worker command forces --loop/--worker-id/--worker-capacity
(appended last so argparse last-wins); --db comes from scaler
or worker args; workers inherit scaler env (Postgres etc.).
State file (depth/desired/actual/workers) written atomically
for observability. deploy.sh now restarts codex-autoscaler
instead of the static worker unit when the scaler is active,
so the two never poll the same queue at once.
Tests: 8 autoscaler tests (desired math, ready-only depth,
cycle=0, scale up/down, cooldown streak, forced worker flags).
host