Worker previously only recorded a model_fallback event on a
fallbackable failure and failed the task; ModelRouter.next_after
was never used to actually run the task on another model.
With --fallback-models (default empty = off, old behaviour
unchanged), a fallbackable failure (404/429/5xx/timeout; 401/400
and cancellations never fall back) now re-runs the same task,
same worktree/branch, on the first fallback model via host.py's
top-level --model flag (no host.py signature change). The second
attempt's result is final: actual_model is the fallback model,
fallback_count=1, and a double failure keeps both reasons in
error_text. LocalExecutor/DockerExecutor pass task model through
to host.py; CloudExecutor includes model only when set.
Worker pool: workers table + register/heartbeat (heartbeat is
registration, no 404), GET /workers with alive (120s default),
POST /workers/heartbeat (X-API-Key, missing worker_id 400).
Tasks gain executor (local/docker/cloud, None = any worker);
worker claim filters by executor in pick_next and in the CAS
WHERE clause, CAS semantics unchanged; CloudExecutor _name
now correctly returns cloud, not local.
CloudExecutor: generic HTTP runner interface (endpoint from
explicit arg or CLOUD_RUNNER_ENDPOINT, auth header name only,
token value only from env into the request header, never into
logs/body/errors); no endpoint configured = explicit failure,
never silent downgrade. Vendor adapters (ECS/Cloud Run) are
out of scope, documented as such.
Auto PR: projects.auto_pr (0/1, create-time only, no PATCH
route exists); POST /tasks/{id}/pr (maintainer+ via
approve_merge) only for merged tasks (else 409); skip reasons
in order: auto_pr_disabled, no_worktree, no_remote, no_gh,
gh_not_authenticated — all return 200 with pr_skipped, task
stays merged, no push attempted; push/pr-create failure
returns 502 with step, task stays merged; success writes
pr_url/pr_number and audits pr_created. No default auto-push
from the worker. Remote URLs in errors are scrubbed of
userinfo; gh credentials are managed by gh itself, never
stored by the control plane.
Tests: host 27 pass, cloud 323 pass (8 skipped are
environment-dependent real-Postgres tests).
Worker startup cleanup previously deleted every container whose
name started with codex-, including the codex-postgres database
container. On the external host the first worker restart after
the prod-grade deploy deleted codex-postgres, leaving the
control plane unable to connect to Postgres (connection refused
on 127.0.0.1:5432, server auto-restart loop).
Fix: cleanup only deletes codex-<32-hex task id> containers,
which is exactly the name DockerExecutor gives task containers;
infrastructure names like codex-postgres never match and are
never deleted. Regression test covers postgres + non-hex names.
GitHub webhook: delivery-id dedup persisted in webhook_deliveries,
/fix-review creates a queued task with audit (no model run in the
webhook itself), check_run/check_suite parsing without mutating
task state (CI backfill still via pr-checks).
Control plane: Postgres pg_dump backup branch + POST /admin/backup
(maintainer+, password only via PGPASSWORD env), per-key/IP rate
limit (default 120/min, 429), health returns backend + db_ok,
metrics by_status always has all seven status keys.
Worker: per-task timeout (default 1800s, --task-timeout), timeout
is failed not interrupted, limited retries (default 1, backoff,
fallback failures not re-retried), stale codex- container cleanup
on loop/CLI start.
Tests: cloud 271 (267 pass, 4 skipped are pre-existing real-
Postgres tests), host 27 pass.
- DockerExecutor: extra mounts (host.py work/state, repo, codex-home
at same paths), worker --docker-extra-mounts, container user
configurable; Dockerfile copies codex-resources beside the codex
binary and puts bwrap on PATH (previous /usr/local/lib layout
made codex report bubblewrap unavailable inside the container).
- Real Docker task on external host: triple added in ~18s, merged
inside a container, main repo now add+mul2+triple, --rm cleanup.
Honest boundary: this host/kernel forbids unprivileged user
namespaces, so bwrap fails on host and in containers; production
mode here is Docker isolation + inner danger-full-access, not
double sandboxing.
- Cloud merge status: POST /tasks/{id}/merge syncs status only
(merge itself stays with host.py merge); completed -> merged,
maintainer+ (approve_merge), other states and repeat calls 409,
audit merge_task. Tests: cloud 223, host 27, all passing.
host