PR #145 added docker_run_* lines only to _run_in_container, but
goal/goal-resume/goal-resume-breakpoint call _call_runner directly
(to keep full stdout for goal conclusions), so the production live
test (a goal task) produced zero docker_run_ lines. Extract a
_docker_call_staged helper and route all four task paths through it;
error lines log the class name then re-raise so each caller's
kill+failed handling is unchanged.
Also fix audit attribution: audit_log has no task_id column, and
worker_exec_done/error details carried no task id, so per-task
queries missed them (the rows existed). All three exec details now
start with task_id=. Tests: 2 new goal-path stage cases + a done
detail task-id assertion (8 total in test_prestart_stages).
Claim-to-dispatch stalls (cf. 2026-10-29 task A: 61s to interrupted
with zero detail) were invisible because docker run blocks with
captured output and watchdog kills were silent. Add three layers:
- worker_exec_start/done/error audit rows in _run_attempt (PG,
survives --rm), start/done pairing shows whether an attempt died
inside the executor call;
- docker_run_start/done/error log lines in DockerExecutor (container
+ image only, never the inner command/prompt);
- stdout tail (last 2000 chars) to the worker log on failed or
interrupted results, and a log line on both watchdog kill paths.
Tests: cloud/tests/test_prestart_stages.py, 6 cases incl. a
no-prompt-in-logs assertion.
DockerExecutor gains a runtime parameter (passed as docker --runtime
in build_run_command, after --pids-limit); worker CLI gains
--docker-runtime, wired through _build_cli_executor. Unset or empty
means no --runtime flag, so existing tasks keep docker default (runc);
an unregistered runtime fails container start (fail closed), it never
silently falls back. Tests: runsc appears in the command, default/empty
omit it, CLI wiring passes through. README documents the flag and
corrects a stale multi-tenant note (host kernel does not ban userns;
bwrap fails inside Docker containers on nested-pidns proc mount).
Inner codex sandbox stays danger-full-access by choice (bwrap
path is a separate decision); this hardens everything around it:
- DockerExecutor: --security-opt plumbing (repeatable, dedups
no-new-privileges), --pids-limit (CLI default 512, fork-bomb
guard), per-call extra_env (-e NAME=VALUE, non-secrets only,
e.g. proxy URLs; the model key still goes by -e NAME only).
- Worker: --egress-proxy-port (0 = off). When on, Docker task
containers get HTTP(S)_PROXY pointing at the host filtering
proxy via their network gateway IP (tenant network gateway
or default bridge gateway). Gateway unresolvable => the
attempt fails instead of running unproxied (fail closed,
same rule as tenant network creation failure).
- AppArmor profile deploy/apparmor/docker-codex: docker-default
base + deny network raw/packet, deny ptrace, deny mount/
umount/pivot_root, deny dangerous capabilities, deny writes
to /proc/sys, /sys, /proc/keys. Deliberately NO userns rule
(that would enable the bwrap path, not this one). File rules
mirror docker-default (broad + sensitive denies); a strict
file allowlist would need profiling codex's file access set
and is not claimed here.
- Egress proxy deploy/egress/: Tinyproxy with
FilterDefaultDeny + anchored domain allowlist (relay,
github, pypi, npm). Anchoring matters: an unanchored
'github.com' regex would also match 'evilgithub.com'.
- Seccomp honesty (README): Docker default seccomp already
enforces on all task containers; no new stricter allowlist
is claimed in this change (building one from memory risks
silently breaking the toolchain; it needs syscall profiling
first).
Tests: 3 outer-hardening tests (security-opt dedup, pids
limit, proxy env without leaking the key value, gateway
parse/fail, proxy fail-closed). Full suites pass.
Container-side isolation was project-blind: every task mounted
the shared /app/work (all worktrees + host-state), the shared
codex-home (all thread history), the shared repos dir and the
shared test-repo, and all containers shared the default bridge
network. With --tenant-isolation (Docker only; Local runs on
the host under one user and has no container boundary to
enforce), each project gets its own host dirs under
tenants_root/<project>/{work,codex-home,test-repo,<project>}:
- mounts: container paths stay identical (/app/work,
codex-home, repos/<pid>, test-repo) so host.py config needs
no change; only the host mount sources are per-project.
Static --docker-extra-mounts are replaced per task (not
appended), so a shared mount cannot leak back in.
- codex-home seeds ONLY the template's config.toml (relay
endpoint/provider names, no secret values); thread_history
is never copied (it holds other tasks' prompts/results).
test-repo is a per-project copy of the fixture (worktree
creation needs a writable repo; a shared writable fixture
would let one tenant poison the next task's base code).
- repos: repo_url checkouts go to the tenant dir and task
repo is set to the container path; no-repo_url tasks get
the tenant test-repo copy instead of the shared fixture.
- network: one Docker network per project (codex-<pid>,
created idempotently, already-exists is OK); network
creation failure fails the task instead of silently
falling back to the default network (a silent downgrade
would make isolation fail quietly).
- hardening for all Docker runs: --cap-drop=ALL and
--security-opt=no-new-privileges (non-root user was already
the case); verified live by a good-model task, not assumed.
Honest limit, kept in README: the inner codex sandbox is
still danger-full-access (host kernel blocks user namespaces,
bwrap unusable), so the boundary is container + mounts +
network, not the inner sandbox.
Tests: 7 tenant tests (seeding incl. no-history, per-project
mounts, network sanitization, per-call override, ensure
network ok/fail). Full suites pass.
1. Concurrency: capacity is now a real limit, not display-only.
run_all with capacity>1 runs a manual thread pool (threading,
not concurrent.futures: a test in this repo puts cloud/ on
sys.path, shadowing stdlib queue and breaking
concurrent.futures' SimpleQueue import). Claim stays CAS, so
two threads racing one task still yield one owner; max_tasks
counts dispatched+running to avoid overshoot; stop signals
stop new claims only, running tasks are drained by run_loop.
Running state moved from single fields to a _running dict
(proc/container per task) so cancel kills the right task.
Docker timeout is now a per-call argument, never a mutated
shared executor field. CostStore opens a fresh connection per
record/report (shared sqlite connection is thread-bound and
raised ProgrammingError under the pool; :memory: keeps the
shared connection because a fresh one is an empty DB).
Final status write is CAS on status='running' so a background
thread cannot overwrite an interrupted task with completed.
2. Survival: run_loop starts a daemon heartbeat thread (every
30s, refreshes running tasks' last_heartbeat) and sweeps stale
tasks every round (threshold 300s, best-effort); live tasks
never go stale now, dead workers' tasks are reclaimed without
manual /admin/sweep. On exit, run_loop waits up to
drain_timeout (CLI --drain-timeout, default 60s, inside
systemd's 90s stop window) for running tasks before killing
leftovers and marking interrupted.
3. Retry: POST /tasks/{id}/retry (maintainer+, 404/403/409):
failed/interrupted/merge-failed go back to queued with
result/error/model cleared, prompt/model/deps kept, CAS
update, audit task_retried.
Tests: 5 concurrency tests (barrier overlap, serial capacity=1,
max_tasks, wait_for_running, CAS final write), 3 retry tests;
full suites pass.
Worker previously only recorded a model_fallback event on a
fallbackable failure and failed the task; ModelRouter.next_after
was never used to actually run the task on another model.
With --fallback-models (default empty = off, old behaviour
unchanged), a fallbackable failure (404/429/5xx/timeout; 401/400
and cancellations never fall back) now re-runs the same task,
same worktree/branch, on the first fallback model via host.py's
top-level --model flag (no host.py signature change). The second
attempt's result is final: actual_model is the fallback model,
fallback_count=1, and a double failure keeps both reasons in
error_text. LocalExecutor/DockerExecutor pass task model through
to host.py; CloudExecutor includes model only when set.
Worker pool: workers table + register/heartbeat (heartbeat is
registration, no 404), GET /workers with alive (120s default),
POST /workers/heartbeat (X-API-Key, missing worker_id 400).
Tasks gain executor (local/docker/cloud, None = any worker);
worker claim filters by executor in pick_next and in the CAS
WHERE clause, CAS semantics unchanged; CloudExecutor _name
now correctly returns cloud, not local.
CloudExecutor: generic HTTP runner interface (endpoint from
explicit arg or CLOUD_RUNNER_ENDPOINT, auth header name only,
token value only from env into the request header, never into
logs/body/errors); no endpoint configured = explicit failure,
never silent downgrade. Vendor adapters (ECS/Cloud Run) are
out of scope, documented as such.
Auto PR: projects.auto_pr (0/1, create-time only, no PATCH
route exists); POST /tasks/{id}/pr (maintainer+ via
approve_merge) only for merged tasks (else 409); skip reasons
in order: auto_pr_disabled, no_worktree, no_remote, no_gh,
gh_not_authenticated — all return 200 with pr_skipped, task
stays merged, no push attempted; push/pr-create failure
returns 502 with step, task stays merged; success writes
pr_url/pr_number and audits pr_created. No default auto-push
from the worker. Remote URLs in errors are scrubbed of
userinfo; gh credentials are managed by gh itself, never
stored by the control plane.
Tests: host 27 pass, cloud 323 pass (8 skipped are
environment-dependent real-Postgres tests).
Worker startup cleanup previously deleted every container whose
name started with codex-, including the codex-postgres database
container. On the external host the first worker restart after
the prod-grade deploy deleted codex-postgres, leaving the
control plane unable to connect to Postgres (connection refused
on 127.0.0.1:5432, server auto-restart loop).
Fix: cleanup only deletes codex-<32-hex task id> containers,
which is exactly the name DockerExecutor gives task containers;
infrastructure names like codex-postgres never match and are
never deleted. Regression test covers postgres + non-hex names.
GitHub webhook: delivery-id dedup persisted in webhook_deliveries,
/fix-review creates a queued task with audit (no model run in the
webhook itself), check_run/check_suite parsing without mutating
task state (CI backfill still via pr-checks).
Control plane: Postgres pg_dump backup branch + POST /admin/backup
(maintainer+, password only via PGPASSWORD env), per-key/IP rate
limit (default 120/min, 429), health returns backend + db_ok,
metrics by_status always has all seven status keys.
Worker: per-task timeout (default 1800s, --task-timeout), timeout
is failed not interrupted, limited retries (default 1, backoff,
fallback failures not re-retried), stale codex- container cleanup
on loop/CLI start.
Tests: cloud 271 (267 pass, 4 skipped are pre-existing real-
Postgres tests), host 27 pass.
- DockerExecutor: extra mounts (host.py work/state, repo, codex-home
at same paths), worker --docker-extra-mounts, container user
configurable; Dockerfile copies codex-resources beside the codex
binary and puts bwrap on PATH (previous /usr/local/lib layout
made codex report bubblewrap unavailable inside the container).
- Real Docker task on external host: triple added in ~18s, merged
inside a container, main repo now add+mul2+triple, --rm cleanup.
Honest boundary: this host/kernel forbids unprivileged user
namespaces, so bwrap fails on host and in containers; production
mode here is Docker isolation + inner danger-full-access, not
double sandboxing.
- Cloud merge status: POST /tasks/{id}/merge syncs status only
(merge itself stays with host.py merge); completed -> merged,
maintainer+ (approve_merge), other states and repeat calls 409,
audit merge_task. Tests: cloud 223, host 27, all passing.
host