- CODEX_GLOBAL_ADMINS replaces maintainer-derived global admin;
admin endpoints (quotas, sweep, backup, push dispatch) fail-closed
403 when unset, with audit on allow and deny.
- continue/followups/messages/retry check both caller and original
creator quotas (USD + unpriced token cap), 402 on either overrun.
- Quota-less users now get the #86 default quota instead of unlimited;
init_db backfills and OAuth create/bind/existing-login branches
INSERT OR IGNORE a default row; explicit NULL row stays unlimited.
user_quotas added to migration tables.
- /fix-review: untrusted authors rejected before dedup claim, no task;
trusted tasks force executor=docker, created_by=project owner, owner
quota gate, prompt is a fixed template plus a 2000-char excerpt.
- GitHub App setup state only issued by global admins, state rows bind
issued_by (ALTER-compatible for old DBs), callback rejects issuers
no longer in the admin list without consuming the state.
- Cancel: popen poll checks DB status every second, Docker path runs a
cancel watchdog that kills by container name, write-back is CAS-guarded
so late results never overwrite a new status.
- SHELL_ENV_EXCLUDE adds *_TOKEN, *_AUTH*, *_CREDENTIAL*, *API_KEY*;
comment states the residual risk: same-UID /proc environ remains
readable, true isolation needs UID separation or hidepid.
- Tests: new cloud/tests/test_p0_security.py (17 cases); old tests
updated to the new semantics (admin env, default quota, trusted
fix-review author) without weakening any gate.
- claim-time gate: queued task whose quota was spent while waiting is
marked interrupted with a quota reason, no cost row, resume kept
- retry/fallback boundaries and every followup round check DB month
usage plus this run's accumulated usage (priced via cost table)
- local popen poll loop kills the process group on quota (30s throttle);
Docker/sync paths get boundary checks only (documented)
- quota stops finalize as interrupted + quota_exceeded error, keep the
actual cost row, no push, followups restored to pending
DockerExecutor gains a runtime parameter (passed as docker --runtime
in build_run_command, after --pids-limit); worker CLI gains
--docker-runtime, wired through _build_cli_executor. Unset or empty
means no --runtime flag, so existing tasks keep docker default (runc);
an unregistered runtime fails container start (fail closed), it never
silently falls back. Tests: runsc appears in the command, default/empty
omit it, CLI wiring passes through. README documents the flag and
corrects a stale multi-tenant note (host kernel does not ban userns;
bwrap fails inside Docker containers on nested-pidns proc mount).
Docker outer layer and bwrap inner layer are mutually exclusive
on this host (fresh proc mounts fail with kernel EPERM inside
Docker's nested pidns, but work on the host under a userns
profile). So tasks that need an inner sandbox run on the host:
- Worker --claim-executor-only: strict executor claim (only
executor == own type, never NULL). The host worker MUST use
it, or it would steal NULL-executor tasks meant for Docker
workers (those tasks would lose the Docker outer layer).
Default behavior unchanged (NULL claimable by any worker).
- host-config-host.json: sandbox workspace-write, state
work/host-state-host.json (distinct from Docker tasks'
host-state.json), same relay home and test repo as Docker.
- deploy/apparmor/codex-host: docker-codex base + userns,
mount, pivot_root for bwrap; still denies raw network,
ptrace, dangerous capabilities, /proc/sys writes.
- deploy/host-worker/codex-host-worker.service: aa-exec -p
codex-host, --executor local --claim-executor-only,
capacity 2, proxy env 127.0.0.1:8888 so host tasks' relay/git
egress also goes through the Tinyproxy allowlist (host direct
egress would bypass it).
POST /tasks already accepts executor (local/docker/cloud), so
no API change was needed for routing.
Tests: 4 host-sandbox-routing tests (strict claim incl. CAS,
default still takes NULL, config sandbox/state, profile
allows bwrap without deny mount). Full suites pass.
Inner codex sandbox stays danger-full-access by choice (bwrap
path is a separate decision); this hardens everything around it:
- DockerExecutor: --security-opt plumbing (repeatable, dedups
no-new-privileges), --pids-limit (CLI default 512, fork-bomb
guard), per-call extra_env (-e NAME=VALUE, non-secrets only,
e.g. proxy URLs; the model key still goes by -e NAME only).
- Worker: --egress-proxy-port (0 = off). When on, Docker task
containers get HTTP(S)_PROXY pointing at the host filtering
proxy via their network gateway IP (tenant network gateway
or default bridge gateway). Gateway unresolvable => the
attempt fails instead of running unproxied (fail closed,
same rule as tenant network creation failure).
- AppArmor profile deploy/apparmor/docker-codex: docker-default
base + deny network raw/packet, deny ptrace, deny mount/
umount/pivot_root, deny dangerous capabilities, deny writes
to /proc/sys, /sys, /proc/keys. Deliberately NO userns rule
(that would enable the bwrap path, not this one). File rules
mirror docker-default (broad + sensitive denies); a strict
file allowlist would need profiling codex's file access set
and is not claimed here.
- Egress proxy deploy/egress/: Tinyproxy with
FilterDefaultDeny + anchored domain allowlist (relay,
github, pypi, npm). Anchoring matters: an unanchored
'github.com' regex would also match 'evilgithub.com'.
- Seccomp honesty (README): Docker default seccomp already
enforces on all task containers; no new stricter allowlist
is claimed in this change (building one from memory risks
silently breaking the toolchain; it needs syscall profiling
first).
Tests: 3 outer-hardening tests (security-opt dedup, pids
limit, proxy env without leaking the key value, gateway
parse/fail, proxy fail-closed). Full suites pass.
Container-side isolation was project-blind: every task mounted
the shared /app/work (all worktrees + host-state), the shared
codex-home (all thread history), the shared repos dir and the
shared test-repo, and all containers shared the default bridge
network. With --tenant-isolation (Docker only; Local runs on
the host under one user and has no container boundary to
enforce), each project gets its own host dirs under
tenants_root/<project>/{work,codex-home,test-repo,<project>}:
- mounts: container paths stay identical (/app/work,
codex-home, repos/<pid>, test-repo) so host.py config needs
no change; only the host mount sources are per-project.
Static --docker-extra-mounts are replaced per task (not
appended), so a shared mount cannot leak back in.
- codex-home seeds ONLY the template's config.toml (relay
endpoint/provider names, no secret values); thread_history
is never copied (it holds other tasks' prompts/results).
test-repo is a per-project copy of the fixture (worktree
creation needs a writable repo; a shared writable fixture
would let one tenant poison the next task's base code).
- repos: repo_url checkouts go to the tenant dir and task
repo is set to the container path; no-repo_url tasks get
the tenant test-repo copy instead of the shared fixture.
- network: one Docker network per project (codex-<pid>,
created idempotently, already-exists is OK); network
creation failure fails the task instead of silently
falling back to the default network (a silent downgrade
would make isolation fail quietly).
- hardening for all Docker runs: --cap-drop=ALL and
--security-opt=no-new-privileges (non-root user was already
the case); verified live by a good-model task, not assumed.
Honest limit, kept in README: the inner codex sandbox is
still danger-full-access (host kernel blocks user namespaces,
bwrap unusable), so the boundary is container + mounts +
network, not the inner sandbox.
Tests: 7 tenant tests (seeding incl. no-history, per-project
mounts, network sanitization, per-call override, ensure
network ok/fail). Full suites pass.
Layer 1, in-worker: run_all pool thread count is now adaptive
per round (min(capacity, ready_count)) instead of always
capacity threads; ready_count counts only dependency-satisfied
queued tasks via a shared _ready_tasks() helper (pick_next
refactored onto it), so blocked tasks never inflate the pool.
Layer 2, process scaler (cloud/autoscaler.py): a systemd-managed
daemon that starts/stops cloud.worker subprocesses by ready
queue depth. desired = ceil(depth / per_worker_capacity)
clamped to [min_workers, max_workers]; scale-up is immediate,
scale-down waits scale_down_after consecutive low checks
(default 3) to avoid flapping on bursty completions. Scale-down
is LIFO (youngest worker first), SIGTERM for graceful drain
(worker run_loop drains up to drain_timeout), SIGKILL of the
process group only after drain timeout. Depth read failure
keeps the current worker count (never scale on unknown state).
Worker command forces --loop/--worker-id/--worker-capacity
(appended last so argparse last-wins); --db comes from scaler
or worker args; workers inherit scaler env (Postgres etc.).
State file (depth/desired/actual/workers) written atomically
for observability. deploy.sh now restarts codex-autoscaler
instead of the static worker unit when the scaler is active,
so the two never poll the same queue at once.
Tests: 8 autoscaler tests (desired math, ready-only depth,
cycle=0, scale up/down, cooldown streak, forced worker flags).
1. Concurrency: capacity is now a real limit, not display-only.
run_all with capacity>1 runs a manual thread pool (threading,
not concurrent.futures: a test in this repo puts cloud/ on
sys.path, shadowing stdlib queue and breaking
concurrent.futures' SimpleQueue import). Claim stays CAS, so
two threads racing one task still yield one owner; max_tasks
counts dispatched+running to avoid overshoot; stop signals
stop new claims only, running tasks are drained by run_loop.
Running state moved from single fields to a _running dict
(proc/container per task) so cancel kills the right task.
Docker timeout is now a per-call argument, never a mutated
shared executor field. CostStore opens a fresh connection per
record/report (shared sqlite connection is thread-bound and
raised ProgrammingError under the pool; :memory: keeps the
shared connection because a fresh one is an empty DB).
Final status write is CAS on status='running' so a background
thread cannot overwrite an interrupted task with completed.
2. Survival: run_loop starts a daemon heartbeat thread (every
30s, refreshes running tasks' last_heartbeat) and sweeps stale
tasks every round (threshold 300s, best-effort); live tasks
never go stale now, dead workers' tasks are reclaimed without
manual /admin/sweep. On exit, run_loop waits up to
drain_timeout (CLI --drain-timeout, default 60s, inside
systemd's 90s stop window) for running tasks before killing
leftovers and marking interrupted.
3. Retry: POST /tasks/{id}/retry (maintainer+, 404/403/409):
failed/interrupted/merge-failed go back to queued with
result/error/model cleared, prompt/model/deps kept, CAS
update, audit task_retried.
Tests: 5 concurrency tests (barrier overlap, serial capacity=1,
max_tasks, wait_for_running, CAS final write), 3 retry tests;
full suites pass.
Worker polls the DB every interval (production 30s) and every
poll ran SELECT * FROM tasks: full rows including result_text
and error_text (up to 4000 chars each) for every task ever,
growing with history. load_tasks now selects only the columns
pick_next actually needs (id/project/prompt/status/depends/
priority/timestamps/executor) and only queued/completed/merged
rows; running/failed rows are neither ready nor completed, so
excluding them cannot change pick_next. get_task still does
SELECT * for the single claimed task. Plus CREATE INDEX IF NOT
EXISTS idx_tasks_status on tasks(status) in both SQLite
init_db and the Postgres schema, so the poll filter stops being
a full table scan on old databases too. Test: load_tasks returns
only queued+completed out of queued/completed/running/failed.
Real bad-model turn: relay returns 404, codex retries sampling
5 times, turn fails, but app-server's turn/completed event
carries status=failed with NO error and NO text, so worker
never saw any fallbackable signal. The 404 actually lives in
codex's thread_history_1.sqlite (thread_turns.error_json).
host.py Server.run_turn now best-effort recovers that message
(read-only query by thread_id+turn_id, any failure -> None,
never breaks the turn itself), and dispatch CLI prints the
error field (run CLI already printed the full result). Worker
_attempt_error prefers the error field when text is empty, so
a double failure keeps the real reasons, not just status codes.
Tests: host TurnErrorFromHistoryTest (recovery + missing/corrupt
DB -> None), full host + cloud suites pass.
1. Worker _fallback_token correction: real host.py never returns
status='404'; a failed turn is status='failed' with the real
error (e.g. '404 model not found') in text/error. The old code
returned host_status verbatim, so 'failed' blocked every real
fallback and the old event-only branch did too. Now: if
host_status itself is a fallbackable code it is used; otherwise
the code is extracted from turn JSON text/error, stdout, stderr
or runner error, and 'failed' alone never counts as a signal.
Tests: failed-turn JSON with 404 in text falls back and
succeeds; with 401 it never falls back.
2. Docker config codex_home: /var/lib/codex-app-server/codex-home,
mounted at the same path into the container, is the only home
that holds the real relay config (xcmapi provider, base_url,
env_key). Pointing codex_home at a fresh empty dir made codex
POST to api.openai.com with no auth header and loop on connect
timeouts until host.py's 600s turn timeout. The /app/work
codex-home from PR #17 was necessary to get past initialize
but insufficient for a real turn; this replaces it.
Two live-test findings in one fix:
1. DockerExecutor has no repo fallback: when a project has no
repo_url, worker leaves task['repo'] unset and the container
host.py dispatch fails instantly with 'missing --repo' because
host-config-docker.json had no repo field either. Add the host
test-repo as the config default; projects with repo_url still
override it via worker's explicit --repo.
2. Worker discarded stderr from the final error_text, so every
container/host crash surfaced only as the generic 'host
execution failed' and was undebuggable from the task record.
When stdout is empty, error_text now keeps the stderr tail
(last 1000 chars, where the real error line lives).
Live test found the gap: a nonexistent model fails in
thread/start, host.py exits non-zero without JSON, so the
executor result has host_status=None and the worker never
fell back (task failed, no worker_fallback audit). When
host_status is absent, worker now extracts an explicit signal
from stdout/stderr/runner error: 404/429/5xx codes, or
'model not found' style text (counted as 404). Generic
failures with no explicit code still do not fall back, and
401 in stderr still never falls back. Tests: fallback 9 pass,
cloud 333 pass, host 27 pass.
Worker previously only recorded a model_fallback event on a
fallbackable failure and failed the task; ModelRouter.next_after
was never used to actually run the task on another model.
With --fallback-models (default empty = off, old behaviour
unchanged), a fallbackable failure (404/429/5xx/timeout; 401/400
and cancellations never fall back) now re-runs the same task,
same worktree/branch, on the first fallback model via host.py's
top-level --model flag (no host.py signature change). The second
attempt's result is final: actual_model is the fallback model,
fallback_count=1, and a double failure keeps both reasons in
error_text. LocalExecutor/DockerExecutor pass task model through
to host.py; CloudExecutor includes model only when set.
Worker pool: workers table + register/heartbeat (heartbeat is
registration, no 404), GET /workers with alive (120s default),
POST /workers/heartbeat (X-API-Key, missing worker_id 400).
Tasks gain executor (local/docker/cloud, None = any worker);
worker claim filters by executor in pick_next and in the CAS
WHERE clause, CAS semantics unchanged; CloudExecutor _name
now correctly returns cloud, not local.
CloudExecutor: generic HTTP runner interface (endpoint from
explicit arg or CLOUD_RUNNER_ENDPOINT, auth header name only,
token value only from env into the request header, never into
logs/body/errors); no endpoint configured = explicit failure,
never silent downgrade. Vendor adapters (ECS/Cloud Run) are
out of scope, documented as such.
Auto PR: projects.auto_pr (0/1, create-time only, no PATCH
route exists); POST /tasks/{id}/pr (maintainer+ via
approve_merge) only for merged tasks (else 409); skip reasons
in order: auto_pr_disabled, no_worktree, no_remote, no_gh,
gh_not_authenticated — all return 200 with pr_skipped, task
stays merged, no push attempted; push/pr-create failure
returns 502 with step, task stays merged; success writes
pr_url/pr_number and audits pr_created. No default auto-push
from the worker. Remote URLs in errors are scrubbed of
userinfo; gh credentials are managed by gh itself, never
stored by the control plane.
Tests: host 27 pass, cloud 323 pass (8 skipped are
environment-dependent real-Postgres tests).
Deploy: deploy.sh builds tarball, backs up current code (keep 3),
deploys, restarts, health-gates 60s (ok/backend=postgres/db_ok),
auto-rolls back on failure; rollback.sh restores latest backup.
Rollback is code-only, DB migrations are additive-only so old
code can still read new columns.
Backup: restore_postgres refuses to restore into the production
DB name unless explicitly allowed (drill never allows it);
restore-drill.sh backs up fresh, restores to a test DB, compares
tasks/projects/users/audit row counts, drops the test DB.
Monitoring: worker writes heartbeat file every loop (atomic
replace, failure only logged); GET /health adds
worker_last_seen_seconds_ago + worker_ok (180s default, env
tunable, still HTTP 200 when worker is stale); monitor.sh
checks server/worker/health/pg_isready/docker-postgres/disk/
app dir as one JSON line, exit non-zero on any fail, plus
systemd timer every 5 minutes. Push alerting to chat still
needs a separate channel decision (SSH key or webhook URL).
Multi-user: verify-multiuser.py creates test users/projects
in memory-key only, checks RBAC via real HTTP API (owner/
maintainer/contributor/viewer, cross-project isolation,
merge 200/403/409, no-key 401), test tasks carry a never-
satisfiable dependency so the production worker never claims
them, cleanup deletes only multiuser-test- rows (audit log
is append-only and keeps test entries by design).
Tests: host 27 pass, cloud 298 pass (8 skipped are
environment-dependent real-Postgres tests).
Projects gain repo_url (validated: https:// or git@ only, no
userinfo/password, no file:// or local paths; old DBs get the
column via idempotent ALTER in both SQLite and Postgres init).
Worker resolves a project's repo_url to a local checkout under
--repos-dir (default ./repos) before dispatch: clone if absent,
verify remote origin matches and fetch if present, never force
reset/clean, never push (push needs write credentials the
control plane does not store). Resolve failure fails the task
with a clear error, never completed. Docker workers need
repos_dir under a same-path extra mount (e.g.
/var/lib/codex-app-server/repos).
Also: driver-missing test is now mocked instead of depending
on the host really having no psycopg driver installed.
Deploy package: Caddyfile template exposing only /webhooks/*
and /oauth/* (tasks/projects/admin stay 403) plus a
webhook-secret generator script that never prints the secret.
Tests: host 27 pass, cloud 286 pass (8 skipped are
environment-dependent real-Postgres tests).
GitHub webhook: delivery-id dedup persisted in webhook_deliveries,
/fix-review creates a queued task with audit (no model run in the
webhook itself), check_run/check_suite parsing without mutating
task state (CI backfill still via pr-checks).
Control plane: Postgres pg_dump backup branch + POST /admin/backup
(maintainer+, password only via PGPASSWORD env), per-key/IP rate
limit (default 120/min, 429), health returns backend + db_ok,
metrics by_status always has all seven status keys.
Worker: per-task timeout (default 1800s, --task-timeout), timeout
is failed not interrupted, limited retries (default 1, backoff,
fallback failures not re-retried), stale codex- container cleanup
on loop/CLI start.
Tests: cloud 271 (267 pass, 4 skipped are pre-existing real-
Postgres tests), host 27 pass.
- DockerExecutor: extra mounts (host.py work/state, repo, codex-home
at same paths), worker --docker-extra-mounts, container user
configurable; Dockerfile copies codex-resources beside the codex
binary and puts bwrap on PATH (previous /usr/local/lib layout
made codex report bubblewrap unavailable inside the container).
- Real Docker task on external host: triple added in ~18s, merged
inside a container, main repo now add+mul2+triple, --rm cleanup.
Honest boundary: this host/kernel forbids unprivileged user
namespaces, so bwrap fails on host and in containers; production
mode here is Docker isolation + inner danger-full-access, not
double sandboxing.
- Cloud merge status: POST /tasks/{id}/merge syncs status only
(merge itself stays with host.py merge); completed -> merged,
maintainer+ (approve_merge), other states and repeat calls 409,
audit merge_task. Tests: cloud 223, host 27, all passing.
host