Since the autoscaler takeover (static codex-cloud-worker stopped at
2026-10-26 07:02:55, disabled by design), the monitor's worker check
(systemctl is-active codex-cloud-worker) has failed every 5 minutes
while the real workers were healthy (health worker_ok kept passing,
GET /workers shows live autoscale-1 + host-1). A monitor that always
fails masks real failures. Fix: pass if either codex-autoscaler or
codex-cloud-worker is active, covering both deployment modes; the
health check still covers worker heartbeat freshness.
One-off worker verification passed before this flip (2026-10-26):
production-shaped docker task (apparmor=docker-codex, pids-limit 512,
tenant isolation, egress proxy 8888, --docker-runtime runsc) completed
in 60s with gpt-6.1-sol, worktree quadruple + proof verified, template
and tenant repo copies clean. README now states the production state.
Two bwrap failures found by live host tasks, both in the
AppArmor profile (not in codex or the worker):
1. deny capability sys_admin (etc.) breaks bwrap at userns
creation: codex-host with the deny list failed, the same
profile with capability, allowed it. codexsvc holds no host
capabilities, so the allow only covers fake-root caps inside
bwrap's own user namespace (scoped, harmless to the host);
network/ptrace/file denies still hold independently.
2. bwrap brings up loopback in its new net namespace via a
NETLINK_ROUTE socket: without network netlink raw, codex
workspace-write fails with 'Failed to bind NETLINK_ROUTE
socket' and no tool runs. And a bare 'deny network raw'
overrides the netlink allow (deny wins over allow), so the
raw deny must name inet/inet6 explicitly (verified by an
isolated netlink bind test under each profile variant).
README correction in the same change: codex workspace-write
does NOT isolate /tmp -- an agent /tmp write lands on host
/tmp (verified). The verified boundary is: own worktree
writable, everything else read-only, /tmp shared.
Docker outer layer and bwrap inner layer are mutually exclusive
on this host (fresh proc mounts fail with kernel EPERM inside
Docker's nested pidns, but work on the host under a userns
profile). So tasks that need an inner sandbox run on the host:
- Worker --claim-executor-only: strict executor claim (only
executor == own type, never NULL). The host worker MUST use
it, or it would steal NULL-executor tasks meant for Docker
workers (those tasks would lose the Docker outer layer).
Default behavior unchanged (NULL claimable by any worker).
- host-config-host.json: sandbox workspace-write, state
work/host-state-host.json (distinct from Docker tasks'
host-state.json), same relay home and test repo as Docker.
- deploy/apparmor/codex-host: docker-codex base + userns,
mount, pivot_root for bwrap; still denies raw network,
ptrace, dangerous capabilities, /proc/sys writes.
- deploy/host-worker/codex-host-worker.service: aa-exec -p
codex-host, --executor local --claim-executor-only,
capacity 2, proxy env 127.0.0.1:8888 so host tasks' relay/git
egress also goes through the Tinyproxy allowlist (host direct
egress would bypass it).
POST /tasks already accepts executor (local/docker/cloud), so
no API change was needed for routing.
Tests: 4 host-sandbox-routing tests (strict claim incl. CAS,
default still takes NULL, config sandbox/state, profile
allows bwrap without deny mount). Full suites pass.
- AppArmor network rules must use socket types (stream/dgram),
not protocol names (tcp/udp/icmp): the parser rejected the
profile (Invalid network entry) until fixed.
- Tinyproxy has no 'Upstream none' syntax; the line failed the
service at startup. Direct connection is the default, so the
line is removed.
- Tinyproxy Filter is POSIX BASIC regex, not extended: the
extended-style anchored patterns denied even allowed domains
(verified: xcmapi.org got 403). Rewritten in basic syntax
(\(...\)* groups); the anchored form still blocks
evilgithub.com and xcmapi.org.evil.com (verified on host).
- Filter file must be world-readable (was 640 root:root, so
tinyproxy could not read it and exited on reload); install
steps now chmod 644 (documented in the filter header via
the install note in tinyproxy.conf).
Host already runs these exact files (proxy allow/block
verified: xcmapi/github/api.github 200, example/evilgithub/
xcmapi.org.evil.com 403).
Inner codex sandbox stays danger-full-access by choice (bwrap
path is a separate decision); this hardens everything around it:
- DockerExecutor: --security-opt plumbing (repeatable, dedups
no-new-privileges), --pids-limit (CLI default 512, fork-bomb
guard), per-call extra_env (-e NAME=VALUE, non-secrets only,
e.g. proxy URLs; the model key still goes by -e NAME only).
- Worker: --egress-proxy-port (0 = off). When on, Docker task
containers get HTTP(S)_PROXY pointing at the host filtering
proxy via their network gateway IP (tenant network gateway
or default bridge gateway). Gateway unresolvable => the
attempt fails instead of running unproxied (fail closed,
same rule as tenant network creation failure).
- AppArmor profile deploy/apparmor/docker-codex: docker-default
base + deny network raw/packet, deny ptrace, deny mount/
umount/pivot_root, deny dangerous capabilities, deny writes
to /proc/sys, /sys, /proc/keys. Deliberately NO userns rule
(that would enable the bwrap path, not this one). File rules
mirror docker-default (broad + sensitive denies); a strict
file allowlist would need profiling codex's file access set
and is not claimed here.
- Egress proxy deploy/egress/: Tinyproxy with
FilterDefaultDeny + anchored domain allowlist (relay,
github, pypi, npm). Anchoring matters: an unanchored
'github.com' regex would also match 'evilgithub.com'.
- Seccomp honesty (README): Docker default seccomp already
enforces on all task containers; no new stricter allowlist
is claimed in this change (building one from memory risks
silently breaking the toolchain; it needs syscall profiling
first).
Tests: 3 outer-hardening tests (security-opt dedup, pids
limit, proxy env without leaking the key value, gateway
parse/fail, proxy fail-closed). Full suites pass.
Container-side isolation was project-blind: every task mounted
the shared /app/work (all worktrees + host-state), the shared
codex-home (all thread history), the shared repos dir and the
shared test-repo, and all containers shared the default bridge
network. With --tenant-isolation (Docker only; Local runs on
the host under one user and has no container boundary to
enforce), each project gets its own host dirs under
tenants_root/<project>/{work,codex-home,test-repo,<project>}:
- mounts: container paths stay identical (/app/work,
codex-home, repos/<pid>, test-repo) so host.py config needs
no change; only the host mount sources are per-project.
Static --docker-extra-mounts are replaced per task (not
appended), so a shared mount cannot leak back in.
- codex-home seeds ONLY the template's config.toml (relay
endpoint/provider names, no secret values); thread_history
is never copied (it holds other tasks' prompts/results).
test-repo is a per-project copy of the fixture (worktree
creation needs a writable repo; a shared writable fixture
would let one tenant poison the next task's base code).
- repos: repo_url checkouts go to the tenant dir and task
repo is set to the container path; no-repo_url tasks get
the tenant test-repo copy instead of the shared fixture.
- network: one Docker network per project (codex-<pid>,
created idempotently, already-exists is OK); network
creation failure fails the task instead of silently
falling back to the default network (a silent downgrade
would make isolation fail quietly).
- hardening for all Docker runs: --cap-drop=ALL and
--security-opt=no-new-privileges (non-root user was already
the case); verified live by a good-model task, not assumed.
Honest limit, kept in README: the inner codex sandbox is
still danger-full-access (host kernel blocks user namespaces,
bwrap unusable), so the boundary is container + mounts +
network, not the inner sandbox.
Tests: 7 tenant tests (seeding incl. no-history, per-project
mounts, network sanitization, per-call override, ensure
network ok/fail). Full suites pass.
The host backup tar runs with -C /opt codex-app-server, so its
members are named codex-app-server/..., and the old patterns
./work and ./worker-heartbeat.json never matched anything.
The live work/ dir (host-state, worktrees rewritten by running
workers) was therefore included, tar hit 'file changed as we
read it', exited 1, and set -e aborted the deploy. Patterns
are now codex-app-server/work and
codex-app-server/worker-heartbeat.json, which actually match.
The worker rewrites ./worker-heartbeat.json every poll interval
(10s); the host backup tar reads the same directory, so it
could hit 'file changed as we read it', exit 1, and abort the
deploy under set -e (this killed the PR #22 deploy). The
heartbeat file is regenerable runtime state, not code, so
excluding it from the backup is correct, not a workaround.
Layer 1, in-worker: run_all pool thread count is now adaptive
per round (min(capacity, ready_count)) instead of always
capacity threads; ready_count counts only dependency-satisfied
queued tasks via a shared _ready_tasks() helper (pick_next
refactored onto it), so blocked tasks never inflate the pool.
Layer 2, process scaler (cloud/autoscaler.py): a systemd-managed
daemon that starts/stops cloud.worker subprocesses by ready
queue depth. desired = ceil(depth / per_worker_capacity)
clamped to [min_workers, max_workers]; scale-up is immediate,
scale-down waits scale_down_after consecutive low checks
(default 3) to avoid flapping on bursty completions. Scale-down
is LIFO (youngest worker first), SIGTERM for graceful drain
(worker run_loop drains up to drain_timeout), SIGKILL of the
process group only after drain timeout. Depth read failure
keeps the current worker count (never scale on unknown state).
Worker command forces --loop/--worker-id/--worker-capacity
(appended last so argparse last-wins); --db comes from scaler
or worker args; workers inherit scaler env (Postgres etc.).
State file (depth/desired/actual/workers) written atomically
for observability. deploy.sh now restarts codex-autoscaler
instead of the static worker unit when the scaler is active,
so the two never poll the same queue at once.
Tests: 8 autoscaler tests (desired math, ready-only depth,
cycle=0, scale up/down, cooldown streak, forced worker flags).
Overlay-only extract left files added by the newer version behind
after a rollback (2026-10-26: after rolling back ops-batch,
deploy/ new scripts were still present). Before every extract,
delete app-dir contents except work/ (task worktrees), then
extract; backup tarballs also exclude work/, so worktrees are
preserved on both deploy and rollback.
Deploy: deploy.sh builds tarball, backs up current code (keep 3),
deploys, restarts, health-gates 60s (ok/backend=postgres/db_ok),
auto-rolls back on failure; rollback.sh restores latest backup.
Rollback is code-only, DB migrations are additive-only so old
code can still read new columns.
Backup: restore_postgres refuses to restore into the production
DB name unless explicitly allowed (drill never allows it);
restore-drill.sh backs up fresh, restores to a test DB, compares
tasks/projects/users/audit row counts, drops the test DB.
Monitoring: worker writes heartbeat file every loop (atomic
replace, failure only logged); GET /health adds
worker_last_seen_seconds_ago + worker_ok (180s default, env
tunable, still HTTP 200 when worker is stale); monitor.sh
checks server/worker/health/pg_isready/docker-postgres/disk/
app dir as one JSON line, exit non-zero on any fail, plus
systemd timer every 5 minutes. Push alerting to chat still
needs a separate channel decision (SSH key or webhook URL).
Multi-user: verify-multiuser.py creates test users/projects
in memory-key only, checks RBAC via real HTTP API (owner/
maintainer/contributor/viewer, cross-project isolation,
merge 200/403/409, no-key 401), test tasks carry a never-
satisfiable dependency so the production worker never claims
them, cleanup deletes only multiuser-test- rows (audit log
is append-only and keeps test entries by design).
Tests: host 27 pass, cloud 298 pass (8 skipped are
environment-dependent real-Postgres tests).
Projects gain repo_url (validated: https:// or git@ only, no
userinfo/password, no file:// or local paths; old DBs get the
column via idempotent ALTER in both SQLite and Postgres init).
Worker resolves a project's repo_url to a local checkout under
--repos-dir (default ./repos) before dispatch: clone if absent,
verify remote origin matches and fetch if present, never force
reset/clean, never push (push needs write credentials the
control plane does not store). Resolve failure fails the task
with a clear error, never completed. Docker workers need
repos_dir under a same-path extra mount (e.g.
/var/lib/codex-app-server/repos).
Also: driver-missing test is now mocked instead of depending
on the host really having no psycopg driver installed.
Deploy package: Caddyfile template exposing only /webhooks/*
and /oauth/* (tasks/projects/admin stay 403) plus a
webhook-secret generator script that never prints the secret.
Tests: host 27 pass, cloud 286 pass (8 skipped are
environment-dependent real-Postgres tests).
host