POST /tasks/{id}/pr used to push the branch as-is, so uncommitted
worktree changes never reached the PR (empty diff). Now, after all
skip checks pass, the endpoint commits the worktree on host/<task>
(guarded: refuses when the worktree is on another branch), cleans
__pycache__/*.pyc like host.py merge does, and falls back to the
cloud identity when no git identity is configured. Nothing-to-commit
proceeds. Docs corrected: PR comes after status-merge and before
any local git merge (which deletes branch+worktree).
One-off worker verification passed before this flip (2026-10-26):
production-shaped docker task (apparmor=docker-codex, pids-limit 512,
tenant isolation, egress proxy 8888, --docker-runtime runsc) completed
in 60s with gpt-6.1-sol, worktree quadruple + proof verified, template
and tenant repo copies clean. README now states the production state.
DockerExecutor gains a runtime parameter (passed as docker --runtime
in build_run_command, after --pids-limit); worker CLI gains
--docker-runtime, wired through _build_cli_executor. Unset or empty
means no --runtime flag, so existing tasks keep docker default (runc);
an unregistered runtime fails container start (fail closed), it never
silently falls back. Tests: runsc appears in the command, default/empty
omit it, CLI wiring passes through. README documents the flag and
corrects a stale multi-tenant note (host kernel does not ban userns;
bwrap fails inside Docker containers on nested-pidns proc mount).
Two bwrap failures found by live host tasks, both in the
AppArmor profile (not in codex or the worker):
1. deny capability sys_admin (etc.) breaks bwrap at userns
creation: codex-host with the deny list failed, the same
profile with capability, allowed it. codexsvc holds no host
capabilities, so the allow only covers fake-root caps inside
bwrap's own user namespace (scoped, harmless to the host);
network/ptrace/file denies still hold independently.
2. bwrap brings up loopback in its new net namespace via a
NETLINK_ROUTE socket: without network netlink raw, codex
workspace-write fails with 'Failed to bind NETLINK_ROUTE
socket' and no tool runs. And a bare 'deny network raw'
overrides the netlink allow (deny wins over allow), so the
raw deny must name inet/inet6 explicitly (verified by an
isolated netlink bind test under each profile variant).
README correction in the same change: codex workspace-write
does NOT isolate /tmp -- an agent /tmp write lands on host
/tmp (verified). The verified boundary is: own worktree
writable, everything else read-only, /tmp shared.
Docker outer layer and bwrap inner layer are mutually exclusive
on this host (fresh proc mounts fail with kernel EPERM inside
Docker's nested pidns, but work on the host under a userns
profile). So tasks that need an inner sandbox run on the host:
- Worker --claim-executor-only: strict executor claim (only
executor == own type, never NULL). The host worker MUST use
it, or it would steal NULL-executor tasks meant for Docker
workers (those tasks would lose the Docker outer layer).
Default behavior unchanged (NULL claimable by any worker).
- host-config-host.json: sandbox workspace-write, state
work/host-state-host.json (distinct from Docker tasks'
host-state.json), same relay home and test repo as Docker.
- deploy/apparmor/codex-host: docker-codex base + userns,
mount, pivot_root for bwrap; still denies raw network,
ptrace, dangerous capabilities, /proc/sys writes.
- deploy/host-worker/codex-host-worker.service: aa-exec -p
codex-host, --executor local --claim-executor-only,
capacity 2, proxy env 127.0.0.1:8888 so host tasks' relay/git
egress also goes through the Tinyproxy allowlist (host direct
egress would bypass it).
POST /tasks already accepts executor (local/docker/cloud), so
no API change was needed for routing.
Tests: 4 host-sandbox-routing tests (strict claim incl. CAS,
default still takes NULL, config sandbox/state, profile
allows bwrap without deny mount). Full suites pass.
Inner codex sandbox stays danger-full-access by choice (bwrap
path is a separate decision); this hardens everything around it:
- DockerExecutor: --security-opt plumbing (repeatable, dedups
no-new-privileges), --pids-limit (CLI default 512, fork-bomb
guard), per-call extra_env (-e NAME=VALUE, non-secrets only,
e.g. proxy URLs; the model key still goes by -e NAME only).
- Worker: --egress-proxy-port (0 = off). When on, Docker task
containers get HTTP(S)_PROXY pointing at the host filtering
proxy via their network gateway IP (tenant network gateway
or default bridge gateway). Gateway unresolvable => the
attempt fails instead of running unproxied (fail closed,
same rule as tenant network creation failure).
- AppArmor profile deploy/apparmor/docker-codex: docker-default
base + deny network raw/packet, deny ptrace, deny mount/
umount/pivot_root, deny dangerous capabilities, deny writes
to /proc/sys, /sys, /proc/keys. Deliberately NO userns rule
(that would enable the bwrap path, not this one). File rules
mirror docker-default (broad + sensitive denies); a strict
file allowlist would need profiling codex's file access set
and is not claimed here.
- Egress proxy deploy/egress/: Tinyproxy with
FilterDefaultDeny + anchored domain allowlist (relay,
github, pypi, npm). Anchoring matters: an unanchored
'github.com' regex would also match 'evilgithub.com'.
- Seccomp honesty (README): Docker default seccomp already
enforces on all task containers; no new stricter allowlist
is claimed in this change (building one from memory risks
silently breaking the toolchain; it needs syscall profiling
first).
Tests: 3 outer-hardening tests (security-opt dedup, pids
limit, proxy env without leaking the key value, gateway
parse/fail, proxy fail-closed). Full suites pass.
Container-side isolation was project-blind: every task mounted
the shared /app/work (all worktrees + host-state), the shared
codex-home (all thread history), the shared repos dir and the
shared test-repo, and all containers shared the default bridge
network. With --tenant-isolation (Docker only; Local runs on
the host under one user and has no container boundary to
enforce), each project gets its own host dirs under
tenants_root/<project>/{work,codex-home,test-repo,<project>}:
- mounts: container paths stay identical (/app/work,
codex-home, repos/<pid>, test-repo) so host.py config needs
no change; only the host mount sources are per-project.
Static --docker-extra-mounts are replaced per task (not
appended), so a shared mount cannot leak back in.
- codex-home seeds ONLY the template's config.toml (relay
endpoint/provider names, no secret values); thread_history
is never copied (it holds other tasks' prompts/results).
test-repo is a per-project copy of the fixture (worktree
creation needs a writable repo; a shared writable fixture
would let one tenant poison the next task's base code).
- repos: repo_url checkouts go to the tenant dir and task
repo is set to the container path; no-repo_url tasks get
the tenant test-repo copy instead of the shared fixture.
- network: one Docker network per project (codex-<pid>,
created idempotently, already-exists is OK); network
creation failure fails the task instead of silently
falling back to the default network (a silent downgrade
would make isolation fail quietly).
- hardening for all Docker runs: --cap-drop=ALL and
--security-opt=no-new-privileges (non-root user was already
the case); verified live by a good-model task, not assumed.
Honest limit, kept in README: the inner codex sandbox is
still danger-full-access (host kernel blocks user namespaces,
bwrap unusable), so the boundary is container + mounts +
network, not the inner sandbox.
Tests: 7 tenant tests (seeding incl. no-history, per-project
mounts, network sanitization, per-call override, ensure
network ok/fail). Full suites pass.
Worker previously only recorded a model_fallback event on a
fallbackable failure and failed the task; ModelRouter.next_after
was never used to actually run the task on another model.
With --fallback-models (default empty = off, old behaviour
unchanged), a fallbackable failure (404/429/5xx/timeout; 401/400
and cancellations never fall back) now re-runs the same task,
same worktree/branch, on the first fallback model via host.py's
top-level --model flag (no host.py signature change). The second
attempt's result is final: actual_model is the fallback model,
fallback_count=1, and a double failure keeps both reasons in
error_text. LocalExecutor/DockerExecutor pass task model through
to host.py; CloudExecutor includes model only when set.
Worker pool: workers table + register/heartbeat (heartbeat is
registration, no 404), GET /workers with alive (120s default),
POST /workers/heartbeat (X-API-Key, missing worker_id 400).
Tasks gain executor (local/docker/cloud, None = any worker);
worker claim filters by executor in pick_next and in the CAS
WHERE clause, CAS semantics unchanged; CloudExecutor _name
now correctly returns cloud, not local.
CloudExecutor: generic HTTP runner interface (endpoint from
explicit arg or CLOUD_RUNNER_ENDPOINT, auth header name only,
token value only from env into the request header, never into
logs/body/errors); no endpoint configured = explicit failure,
never silent downgrade. Vendor adapters (ECS/Cloud Run) are
out of scope, documented as such.
Auto PR: projects.auto_pr (0/1, create-time only, no PATCH
route exists); POST /tasks/{id}/pr (maintainer+ via
approve_merge) only for merged tasks (else 409); skip reasons
in order: auto_pr_disabled, no_worktree, no_remote, no_gh,
gh_not_authenticated — all return 200 with pr_skipped, task
stays merged, no push attempted; push/pr-create failure
returns 502 with step, task stays merged; success writes
pr_url/pr_number and audits pr_created. No default auto-push
from the worker. Remote URLs in errors are scrubbed of
userinfo; gh credentials are managed by gh itself, never
stored by the control plane.
Tests: host 27 pass, cloud 323 pass (8 skipped are
environment-dependent real-Postgres tests).
Deploy: deploy.sh builds tarball, backs up current code (keep 3),
deploys, restarts, health-gates 60s (ok/backend=postgres/db_ok),
auto-rolls back on failure; rollback.sh restores latest backup.
Rollback is code-only, DB migrations are additive-only so old
code can still read new columns.
Backup: restore_postgres refuses to restore into the production
DB name unless explicitly allowed (drill never allows it);
restore-drill.sh backs up fresh, restores to a test DB, compares
tasks/projects/users/audit row counts, drops the test DB.
Monitoring: worker writes heartbeat file every loop (atomic
replace, failure only logged); GET /health adds
worker_last_seen_seconds_ago + worker_ok (180s default, env
tunable, still HTTP 200 when worker is stale); monitor.sh
checks server/worker/health/pg_isready/docker-postgres/disk/
app dir as one JSON line, exit non-zero on any fail, plus
systemd timer every 5 minutes. Push alerting to chat still
needs a separate channel decision (SSH key or webhook URL).
Multi-user: verify-multiuser.py creates test users/projects
in memory-key only, checks RBAC via real HTTP API (owner/
maintainer/contributor/viewer, cross-project isolation,
merge 200/403/409, no-key 401), test tasks carry a never-
satisfiable dependency so the production worker never claims
them, cleanup deletes only multiuser-test- rows (audit log
is append-only and keeps test entries by design).
Tests: host 27 pass, cloud 298 pass (8 skipped are
environment-dependent real-Postgres tests).
Projects gain repo_url (validated: https:// or git@ only, no
userinfo/password, no file:// or local paths; old DBs get the
column via idempotent ALTER in both SQLite and Postgres init).
Worker resolves a project's repo_url to a local checkout under
--repos-dir (default ./repos) before dispatch: clone if absent,
verify remote origin matches and fetch if present, never force
reset/clean, never push (push needs write credentials the
control plane does not store). Resolve failure fails the task
with a clear error, never completed. Docker workers need
repos_dir under a same-path extra mount (e.g.
/var/lib/codex-app-server/repos).
Also: driver-missing test is now mocked instead of depending
on the host really having no psycopg driver installed.
Deploy package: Caddyfile template exposing only /webhooks/*
and /oauth/* (tasks/projects/admin stay 403) plus a
webhook-secret generator script that never prints the secret.
Tests: host 27 pass, cloud 286 pass (8 skipped are
environment-dependent real-Postgres tests).
GitHub webhook: delivery-id dedup persisted in webhook_deliveries,
/fix-review creates a queued task with audit (no model run in the
webhook itself), check_run/check_suite parsing without mutating
task state (CI backfill still via pr-checks).
Control plane: Postgres pg_dump backup branch + POST /admin/backup
(maintainer+, password only via PGPASSWORD env), per-key/IP rate
limit (default 120/min, 429), health returns backend + db_ok,
metrics by_status always has all seven status keys.
Worker: per-task timeout (default 1800s, --task-timeout), timeout
is failed not interrupted, limited retries (default 1, backoff,
fallback failures not re-retried), stale codex- container cleanup
on loop/CLI start.
Tests: cloud 271 (267 pass, 4 skipped are pre-existing real-
Postgres tests), host 27 pass.
Add GET /oauth/{provider}/callback (code+state from query, client_id
and redirect_uri recovered from the state record) alongside the
existing POST callback. Callback now fetches real userinfo
(GitHub api.github.com/user, Google openidconnect userinfo) and
issues a usable local API key on success: new users get the
create_user key, existing users rotate (old key dead, audit
oauth_login + api_key_rotated). Key plaintext appears only once
on the success page/response; failures are 400 HTML/JSON without
secrets, and a missing client_secret fails before any network call.
State remains one-time, 10 minutes, provider/redirect_uri matched.
Tests: 10 new OAuth real-login tests pass; host 27, cloud 249
(4 skipped are the pre-existing real-Postgres tests).
Real end-to-end still needs a registered OAuth App and a public
HTTPS callback entry (server still binds localhost only).
Replace the PostgresDatabase skeleton with a real implementation:
psycopg (v3) first, psycopg2 second, lazy load, missing driver
still raises PostgresDriverMissingError with no silent SQLite
fallback. A sqlite3-style adapter (PgConnection/PgCursor/PgRow)
translates ? placeholders to %s outside string literals, maps
UNIQUE/FK violations, lock errors and connection failures to
explicit Postgres error types, and handles INSERT OR REPLACE,
PRAGMA table_info, AUTOINCREMENT and FTS5 refusal. POSTGRES_SCHEMA
covers all business tables with tasks result columns; server
init_db uses it under CLOUD_DB_BACKEND=postgres, SQLite path
unchanged. migrate_sqlite_to_postgres.py copies rows with
ON CONFLICT DO NOTHING and reseeds identity sequences.
Tests: cloud 239 (235 pass, 4 skipped without a real server),
host 27 pass; real-server tests skip unless CLOUD_TEST_PG_DSN
is set.
- DockerExecutor: extra mounts (host.py work/state, repo, codex-home
at same paths), worker --docker-extra-mounts, container user
configurable; Dockerfile copies codex-resources beside the codex
binary and puts bwrap on PATH (previous /usr/local/lib layout
made codex report bubblewrap unavailable inside the container).
- Real Docker task on external host: triple added in ~18s, merged
inside a container, main repo now add+mul2+triple, --rm cleanup.
Honest boundary: this host/kernel forbids unprivileged user
namespaces, so bwrap fails on host and in containers; production
mode here is Docker isolation + inner danger-full-access, not
double sandboxing.
- Cloud merge status: POST /tasks/{id}/merge syncs status only
(merge itself stays with host.py merge); completed -> merged,
maintainer+ (approve_merge), other states and repeat calls 409,
audit merge_task. Tests: cloud 223, host 27, all passing.
host