- claim-time gate: queued task whose quota was spent while waiting is
marked interrupted with a quota reason, no cost row, resume kept
- retry/fallback boundaries and every followup round check DB month
usage plus this run's accumulated usage (priced via cost table)
- local popen poll loop kills the process group on quota (30s throttle);
Docker/sync paths get boundary checks only (documented)
- quota stops finalize as interrupted + quota_exceeded error, keep the
actual cost row, no push, followups restored to pending
New external_identities table (UNIQUE(provider, external_id),
UNIQUE(provider, user_id)) + bind/initiate, bindings list/unbind
endpoints. Callback branches on bind_user_id in state: bind success
shows no key and no user id; conflicts are 409
(identity_bound_to_other / user_already_bound); repeat bind to the
same user+identity is idempotent. Login resolution checks bindings
first: a bound identity lands in the bound account and rotates its
key like any OAuth login; unbound identities keep the deterministic
oauth:<provider>:<login> mapping. Unbind restores oauth mapping.
Tests: new cloud/tests/test_oauth_bind.py (11). Full suites: host
29 OK, cloud 389 OK (4 pre-existing skips).
POST /tasks/{id}/pr used to push the branch as-is, so uncommitted
worktree changes never reached the PR (empty diff). Now, after all
skip checks pass, the endpoint commits the worktree on host/<task>
(guarded: refuses when the worktree is on another branch), cleans
__pycache__/*.pyc like host.py merge does, and falls back to the
cloud identity when no git identity is configured. Nothing-to-commit
proceeds. Docs corrected: PR comes after status-merge and before
any local git merge (which deletes branch+worktree).
DockerExecutor gains a runtime parameter (passed as docker --runtime
in build_run_command, after --pids-limit); worker CLI gains
--docker-runtime, wired through _build_cli_executor. Unset or empty
means no --runtime flag, so existing tasks keep docker default (runc);
an unregistered runtime fails container start (fail closed), it never
silently falls back. Tests: runsc appears in the command, default/empty
omit it, CLI wiring passes through. README documents the flag and
corrects a stale multi-tenant note (host kernel does not ban userns;
bwrap fails inside Docker containers on nested-pidns proc mount).
Two bwrap failures found by live host tasks, both in the
AppArmor profile (not in codex or the worker):
1. deny capability sys_admin (etc.) breaks bwrap at userns
creation: codex-host with the deny list failed, the same
profile with capability, allowed it. codexsvc holds no host
capabilities, so the allow only covers fake-root caps inside
bwrap's own user namespace (scoped, harmless to the host);
network/ptrace/file denies still hold independently.
2. bwrap brings up loopback in its new net namespace via a
NETLINK_ROUTE socket: without network netlink raw, codex
workspace-write fails with 'Failed to bind NETLINK_ROUTE
socket' and no tool runs. And a bare 'deny network raw'
overrides the netlink allow (deny wins over allow), so the
raw deny must name inet/inet6 explicitly (verified by an
isolated netlink bind test under each profile variant).
README correction in the same change: codex workspace-write
does NOT isolate /tmp -- an agent /tmp write lands on host
/tmp (verified). The verified boundary is: own worktree
writable, everything else read-only, /tmp shared.
Docker outer layer and bwrap inner layer are mutually exclusive
on this host (fresh proc mounts fail with kernel EPERM inside
Docker's nested pidns, but work on the host under a userns
profile). So tasks that need an inner sandbox run on the host:
- Worker --claim-executor-only: strict executor claim (only
executor == own type, never NULL). The host worker MUST use
it, or it would steal NULL-executor tasks meant for Docker
workers (those tasks would lose the Docker outer layer).
Default behavior unchanged (NULL claimable by any worker).
- host-config-host.json: sandbox workspace-write, state
work/host-state-host.json (distinct from Docker tasks'
host-state.json), same relay home and test repo as Docker.
- deploy/apparmor/codex-host: docker-codex base + userns,
mount, pivot_root for bwrap; still denies raw network,
ptrace, dangerous capabilities, /proc/sys writes.
- deploy/host-worker/codex-host-worker.service: aa-exec -p
codex-host, --executor local --claim-executor-only,
capacity 2, proxy env 127.0.0.1:8888 so host tasks' relay/git
egress also goes through the Tinyproxy allowlist (host direct
egress would bypass it).
POST /tasks already accepts executor (local/docker/cloud), so
no API change was needed for routing.
Tests: 4 host-sandbox-routing tests (strict claim incl. CAS,
default still takes NULL, config sandbox/state, profile
allows bwrap without deny mount). Full suites pass.
Inner codex sandbox stays danger-full-access by choice (bwrap
path is a separate decision); this hardens everything around it:
- DockerExecutor: --security-opt plumbing (repeatable, dedups
no-new-privileges), --pids-limit (CLI default 512, fork-bomb
guard), per-call extra_env (-e NAME=VALUE, non-secrets only,
e.g. proxy URLs; the model key still goes by -e NAME only).
- Worker: --egress-proxy-port (0 = off). When on, Docker task
containers get HTTP(S)_PROXY pointing at the host filtering
proxy via their network gateway IP (tenant network gateway
or default bridge gateway). Gateway unresolvable => the
attempt fails instead of running unproxied (fail closed,
same rule as tenant network creation failure).
- AppArmor profile deploy/apparmor/docker-codex: docker-default
base + deny network raw/packet, deny ptrace, deny mount/
umount/pivot_root, deny dangerous capabilities, deny writes
to /proc/sys, /sys, /proc/keys. Deliberately NO userns rule
(that would enable the bwrap path, not this one). File rules
mirror docker-default (broad + sensitive denies); a strict
file allowlist would need profiling codex's file access set
and is not claimed here.
- Egress proxy deploy/egress/: Tinyproxy with
FilterDefaultDeny + anchored domain allowlist (relay,
github, pypi, npm). Anchoring matters: an unanchored
'github.com' regex would also match 'evilgithub.com'.
- Seccomp honesty (README): Docker default seccomp already
enforces on all task containers; no new stricter allowlist
is claimed in this change (building one from memory risks
silently breaking the toolchain; it needs syscall profiling
first).
Tests: 3 outer-hardening tests (security-opt dedup, pids
limit, proxy env without leaking the key value, gateway
parse/fail, proxy fail-closed). Full suites pass.
Container-side isolation was project-blind: every task mounted
the shared /app/work (all worktrees + host-state), the shared
codex-home (all thread history), the shared repos dir and the
shared test-repo, and all containers shared the default bridge
network. With --tenant-isolation (Docker only; Local runs on
the host under one user and has no container boundary to
enforce), each project gets its own host dirs under
tenants_root/<project>/{work,codex-home,test-repo,<project>}:
- mounts: container paths stay identical (/app/work,
codex-home, repos/<pid>, test-repo) so host.py config needs
no change; only the host mount sources are per-project.
Static --docker-extra-mounts are replaced per task (not
appended), so a shared mount cannot leak back in.
- codex-home seeds ONLY the template's config.toml (relay
endpoint/provider names, no secret values); thread_history
is never copied (it holds other tasks' prompts/results).
test-repo is a per-project copy of the fixture (worktree
creation needs a writable repo; a shared writable fixture
would let one tenant poison the next task's base code).
- repos: repo_url checkouts go to the tenant dir and task
repo is set to the container path; no-repo_url tasks get
the tenant test-repo copy instead of the shared fixture.
- network: one Docker network per project (codex-<pid>,
created idempotently, already-exists is OK); network
creation failure fails the task instead of silently
falling back to the default network (a silent downgrade
would make isolation fail quietly).
- hardening for all Docker runs: --cap-drop=ALL and
--security-opt=no-new-privileges (non-root user was already
the case); verified live by a good-model task, not assumed.
Honest limit, kept in README: the inner codex sandbox is
still danger-full-access (host kernel blocks user namespaces,
bwrap unusable), so the boundary is container + mounts +
network, not the inner sandbox.
Tests: 7 tenant tests (seeding incl. no-history, per-project
mounts, network sanitization, per-call override, ensure
network ok/fail). Full suites pass.
queue_depth turned a NULL depends_on into [None], and
ready_tasks then required None to be in completed_ids, so
such tasks never counted as ready. Worker pick_next was
unaffected (_task_from_row maps NULL to []), which is why
tasks still ran while the scaler always saw depth 0 and
never scaled up (proven live: depth probe with 1 queued
task returned 0). NULL now means no dependencies, with a
regression test.
setup_logging() takes a log LEVEL, not a logger name; passing
'cloud.autoscaler' raised ValueError at startup, so the scaler
crash-looped under systemd and started zero workers (the host
had no worker at all until this fix). The '/opt/.../-m'
FileNotFoundError in the same journal lines is only Ubuntu's
apport hook misfiring on 'python3 -m', not a second bug.
Regression test: main() with mocked run_loop/signal must reach
the loop (this would have caught the crash before deploy).
Layer 1, in-worker: run_all pool thread count is now adaptive
per round (min(capacity, ready_count)) instead of always
capacity threads; ready_count counts only dependency-satisfied
queued tasks via a shared _ready_tasks() helper (pick_next
refactored onto it), so blocked tasks never inflate the pool.
Layer 2, process scaler (cloud/autoscaler.py): a systemd-managed
daemon that starts/stops cloud.worker subprocesses by ready
queue depth. desired = ceil(depth / per_worker_capacity)
clamped to [min_workers, max_workers]; scale-up is immediate,
scale-down waits scale_down_after consecutive low checks
(default 3) to avoid flapping on bursty completions. Scale-down
is LIFO (youngest worker first), SIGTERM for graceful drain
(worker run_loop drains up to drain_timeout), SIGKILL of the
process group only after drain timeout. Depth read failure
keeps the current worker count (never scale on unknown state).
Worker command forces --loop/--worker-id/--worker-capacity
(appended last so argparse last-wins); --db comes from scaler
or worker args; workers inherit scaler env (Postgres etc.).
State file (depth/desired/actual/workers) written atomically
for observability. deploy.sh now restarts codex-autoscaler
instead of the static worker unit when the scaler is active,
so the two never poll the same queue at once.
Tests: 8 autoscaler tests (desired math, ready-only depth,
cycle=0, scale up/down, cooldown streak, forced worker flags).
host