run_all previously sized its pool from the ready snapshot at round start (min(capacity, ready_count)). If a round started with only 1 ready task (e.g. a just-claimed PM), only 1 thread ran; that thread entered a marker wait (up to 3600s) after its model turn, and children created/retried mid-round had no free thread to claim them. run_loop is blocked joining that round, so no new round starts; if the other worker had already scaled down, the queued child waited out the full wait timeout unclaimed. Fix: always start capacity threads. Idle threads exit immediately when no task is ready and nothing is active, and poll every 0.2s while something is active, so mid-round arrivals are claimed. Test: run_all with capacity=2, 1 initial task whose mocked run_one creates a child then polls for it; without the fix the PM poll times out in 5s (1 result), with the fix both tasks dispatch in the same round (0.2s).
Marker waits run after _run_attempt has cleared _running, so the run_loop heartbeat thread no longer refreshes the waiting parent. sweep_stale_tasks (default 300s) then kills a parent waiting up to the 3600s marker timeout, as seen in the marker live test (PM2 swept mid wait-threads). Poll and wait loops now heartbeat the parent each iteration (interval clamped to <=60s, well under the 300s threshold), best-effort, without touching _running, so no new capacity leak and cancel still works (empty _running entry has no proc/container to kill). Tests: 2 new sweep-survival tests (stale heartbeat at -299s, 2s wait, sweep must spare the parent); both fail 2/2 on pre-fix code and pass with the fix.
Host can now fetch code directly from GitHub instead of receiving a VM-built tarball, bypassing the VM->host scp/kex failures and stale-tar risk. Design: VM sends only two small scripts (host_git_fetch.sh + host_deploy.sh, sha-verified via the existing transfer chain, now with a chunk_key fallback that does not need SSH_PASSWORD); host clones via a read-only deploy key (SSH only, HTTPS refused, key must exist with mode 0600), checks out the exact 40-char commit, verifies HEAD, builds a tar in the same format as the VM release (DEPLOYED_COMMIT + RELEASE.json, member names without ./ prefix so host_deploy.sh tar -O works), then calls the unchanged host_deploy.sh for staging swap, backup, health and rollback. Git fetch tolerates the known intermittent host->GitHub degradation: 120s timeout, 3 attempts, 10s sleep, live untouched until host_deploy runs. VM gates: clean tree, tests, commit must be on origin/main, SSH repo only, full sha only; --source tar remains the default. Tests: deploy suite OK, including 6 new host_git_fetch simulations (success with clean tar members, unknown commit, key permissions/missing, HTTPS refusal, short sha) and 4 new deploy.py git-mode helpers plus chunk_key fallback; git-mode dry-run verified against HEAD 630d14f.
Coordinator tasks bypass _run_attempt, so their _claim placeholder in _running was never cleared: each completed coordinator permanently reduced the worker process effective capacity until it could claim nothing, while heartbeat stayed fresh and the capacity gate returned silently. This caused the 2026-10-28 40-person stall (autoscale-1 had one leaked slot, then claimed coordinator A and filled up) and the old 18-person C-6 never-claim (autoscale-1 leaked both A and B). Fix: wrap the run_one coordinator branch in try/finally _running_clear, so normal and exception paths both release. Also log capacity-gate refusals (worker_claim_refused_capacity) throttled to once per 30s; throttling applies to the log only, refusal always returns False. Tests: coordinator completion releases slot, next claim succeeds, capacity-log throttle; both leak tests fail on pre-fix code.
Two veto layers in reconcile(), worker.py untouched:
- Global live-coordinator veto: a running task whose
coordinator_parts parse (via cloud.coordinator.parse_parts)
to a non-empty list and whose last_heartbeat is within
120s blocks all scale-down this round; low_streak is
frozen, not reset, so scale-down resumes on the next
low-depth round once the veto lifts.
- Per-victim busy veto: a LIFO victim whose workers row
is alive with running>0 is skipped, but only when the
DB also has a live running task (ghost registry rows
with no live DB task do not block).
- Coordinator-veto query failure fails closed (no stop,
guard error recorded); victim-veto read failure fails
open for that layer only.
- A missing/stale workers row is treated as idle.
Tests: 7 new ScaleDownGuardTestCase cases (live/stale
coordinator, non-coordinator, invalid parts, busy/idle
victims, ghost registry, streak freeze); cloud suite
1077 tests OK (4 skipped).
Postgres 翻译层对未登记主键的表把 INSERT OR REPLACE 退化为
ON CONFLICT DO NOTHING，致使 memory_settings、task_memory_flags、
memory_extraction_log 首写之后的所有更新静默丢弃（2026-10-28
生产实测：提取日志永远停在首写原因）。补三张表主键登记，并
在翻译层加回归测试（SQLite 原生 OR REPLACE 测不出来）。
Docker tasks do not inherit the worker process environment, they only
get extra_env (-e). SEARXNG_URL in the worker unit EnvironmentFile
therefore never reached codex -> MCP inside the container, so the
auto chain skipped SearXNG and only DuckDuckGo was tried (live e2e:
'all search providers failed: duckduckgo (DNS)').
_docker_task_context now merges any set SEARXNG_URL /
PERPLEXITY_API_KEY / EXA_API_KEY / TAVILY_API_KEY / BRAVE_API_KEY
from the worker environment into extra_env (values never logged;
paid keys follow the same extra_env path as gateway keys).
Tests: 2 new docker-env tests; websearch + tenant isolation = 25
passing; full cloud suite 1038 passing (4 skipped).
Worker --websearch-mcp (default off) seeds tenant codex-home with
websearch MCP on _tenant_dirs (container path); old tenants merge
on next task. Autoscaler unit gains the flag. Tinyproxy filter
adds Perplexity/Exa/Tavily/Brave/DuckDuckGo + exact
searx.diante.me (no diante.me wildcard, control plane stays out).
Tests: test_websearch_worker 4 + websearch_mcp 12 + tenant 7 = 23 OK.
Not pushed, not merged, not deployed; SearXNG host/DNS/nginx and
tinyproxy reload are host steps.
web_search auto chain Perplexity/Exa/Tavily/Brave/SearXNG/DuckDuckGo
with pinned-provider no-fallback, web_fetch with head+tail truncation,
codex-home seeding helper (idempotent, never writes keys).
Tests: cloud/tests/test_websearch_mcp.py 12/12 (mocked HTTP).
Not wired into worker, not deployed, paid backends not live-tested.
- migrations: MIGRATION_TABLES now equals POSTGRES_SCHEMA tables
(uploads, task_attachments, task_followups, environments added);
SEQUENCE_TABLES matches IDENTITY tables; task_followups
sender_task_id column ensured before insert; UNMIGRATED_TABLES
and migration_coverage() report tables with no PG schema instead
of silently skipping them; CLI prints the coverage report.
- deploy: DEFAULT_TEST_SUITES gains deploy/tests (discover -s only,
never package path); regression tests pin the default list.
- monitor: MONITOR_SINGLE_HOST=1 makes the server check single-host;
default stays dual to avoid silent downgrade; server_mode is in
the JSON. Optional fail-only alert outlet (webhook or mail,
default off, alert body is pass/fail JSON only, no secrets).
- backup/rollback: README and restore-drill document what is
recoverable (DB, code) vs not (worktrees, traces, uploads, host
configs); rollback.sh shares the server-unit list and dual-port
health success standard with host_deploy.sh (sync test guards it).
- public entry: Caddyfile uses native {$DOMAIN}/{$ACME_EMAIL}
variables plus a caddy.service template; README documents the
steps and the current BT-nginx production status.
- CODEX_GLOBAL_ADMINS replaces maintainer-derived global admin;
admin endpoints (quotas, sweep, backup, push dispatch) fail-closed
403 when unset, with audit on allow and deny.
- continue/followups/messages/retry check both caller and original
creator quotas (USD + unpriced token cap), 402 on either overrun.
- Quota-less users now get the #86 default quota instead of unlimited;
init_db backfills and OAuth create/bind/existing-login branches
INSERT OR IGNORE a default row; explicit NULL row stays unlimited.
user_quotas added to migration tables.
- /fix-review: untrusted authors rejected before dedup claim, no task;
trusted tasks force executor=docker, created_by=project owner, owner
quota gate, prompt is a fixed template plus a 2000-char excerpt.
- GitHub App setup state only issued by global admins, state rows bind
issued_by (ALTER-compatible for old DBs), callback rejects issuers
no longer in the admin list without consuming the state.
- Cancel: popen poll checks DB status every second, Docker path runs a
cancel watchdog that kills by container name, write-back is CAS-guarded
so late results never overwrite a new status.
- SHELL_ENV_EXCLUDE adds *_TOKEN, *_AUTH*, *_CREDENTIAL*, *API_KEY*;
comment states the residual risk: same-UID /proc environ remains
readable, true isolation needs UID separation or hidepid.
- Tests: new cloud/tests/test_p0_security.py (17 cases); old tests
updated to the new semantics (admin env, default quota, trusted
fix-review author) without weakening any gate.
- claim-time gate: queued task whose quota was spent while waiting is
marked interrupted with a quota reason, no cost row, resume kept
- retry/fallback boundaries and every followup round check DB month
usage plus this run's accumulated usage (priced via cost table)
- local popen poll loop kills the process group on quota (30s throttle);
Docker/sync paths get boundary checks only (documented)
- quota stops finalize as interrupted + quota_exceeded error, keep the
actual cost row, no push, followups restored to pending
New external_identities table (UNIQUE(provider, external_id),
UNIQUE(provider, user_id)) + bind/initiate, bindings list/unbind
endpoints. Callback branches on bind_user_id in state: bind success
shows no key and no user id; conflicts are 409
(identity_bound_to_other / user_already_bound); repeat bind to the
same user+identity is idempotent. Login resolution checks bindings
first: a bound identity lands in the bound account and rotates its
key like any OAuth login; unbound identities keep the deterministic
oauth:<provider>:<login> mapping. Unbind restores oauth mapping.
Tests: new cloud/tests/test_oauth_bind.py (11). Full suites: host
29 OK, cloud 389 OK (4 pre-existing skips).
POST /tasks/{id}/pr used to push the branch as-is, so uncommitted
worktree changes never reached the PR (empty diff). Now, after all
skip checks pass, the endpoint commits the worktree on host/<task>
(guarded: refuses when the worktree is on another branch), cleans
__pycache__/*.pyc like host.py merge does, and falls back to the
cloud identity when no git identity is configured. Nothing-to-commit
proceeds. Docs corrected: PR comes after status-merge and before
any local git merge (which deletes branch+worktree).
Since the autoscaler takeover (static codex-cloud-worker stopped at
2026-10-26 07:02:55, disabled by design), the monitor's worker check
(systemctl is-active codex-cloud-worker) has failed every 5 minutes
while the real workers were healthy (health worker_ok kept passing,
GET /workers shows live autoscale-1 + host-1). A monitor that always
fails masks real failures. Fix: pass if either codex-autoscaler or
codex-cloud-worker is active, covering both deployment modes; the
health check still covers worker heartbeat freshness.
One-off worker verification passed before this flip (2026-10-26):
production-shaped docker task (apparmor=docker-codex, pids-limit 512,
tenant isolation, egress proxy 8888, --docker-runtime runsc) completed
in 60s with gpt-6.1-sol, worktree quadruple + proof verified, template
and tenant repo copies clean. README now states the production state.
DockerExecutor gains a runtime parameter (passed as docker --runtime
in build_run_command, after --pids-limit); worker CLI gains
--docker-runtime, wired through _build_cli_executor. Unset or empty
means no --runtime flag, so existing tasks keep docker default (runc);
an unregistered runtime fails container start (fail closed), it never
silently falls back. Tests: runsc appears in the command, default/empty
omit it, CLI wiring passes through. README documents the flag and
corrects a stale multi-tenant note (host kernel does not ban userns;
bwrap fails inside Docker containers on nested-pidns proc mount).
Two bwrap failures found by live host tasks, both in the
AppArmor profile (not in codex or the worker):
1. deny capability sys_admin (etc.) breaks bwrap at userns
creation: codex-host with the deny list failed, the same
profile with capability, allowed it. codexsvc holds no host
capabilities, so the allow only covers fake-root caps inside
bwrap's own user namespace (scoped, harmless to the host);
network/ptrace/file denies still hold independently.
2. bwrap brings up loopback in its new net namespace via a
NETLINK_ROUTE socket: without network netlink raw, codex
workspace-write fails with 'Failed to bind NETLINK_ROUTE
socket' and no tool runs. And a bare 'deny network raw'
overrides the netlink allow (deny wins over allow), so the
raw deny must name inet/inet6 explicitly (verified by an
isolated netlink bind test under each profile variant).
README correction in the same change: codex workspace-write
does NOT isolate /tmp -- an agent /tmp write lands on host
/tmp (verified). The verified boundary is: own worktree
writable, everything else read-only, /tmp shared.
Docker outer layer and bwrap inner layer are mutually exclusive
on this host (fresh proc mounts fail with kernel EPERM inside
Docker's nested pidns, but work on the host under a userns
profile). So tasks that need an inner sandbox run on the host:
- Worker --claim-executor-only: strict executor claim (only
executor == own type, never NULL). The host worker MUST use
it, or it would steal NULL-executor tasks meant for Docker
workers (those tasks would lose the Docker outer layer).
Default behavior unchanged (NULL claimable by any worker).
- host-config-host.json: sandbox workspace-write, state
work/host-state-host.json (distinct from Docker tasks'
host-state.json), same relay home and test repo as Docker.
- deploy/apparmor/codex-host: docker-codex base + userns,
mount, pivot_root for bwrap; still denies raw network,
ptrace, dangerous capabilities, /proc/sys writes.
- deploy/host-worker/codex-host-worker.service: aa-exec -p
codex-host, --executor local --claim-executor-only,
capacity 2, proxy env 127.0.0.1:8888 so host tasks' relay/git
egress also goes through the Tinyproxy allowlist (host direct
egress would bypass it).
POST /tasks already accepts executor (local/docker/cloud), so
no API change was needed for routing.
Tests: 4 host-sandbox-routing tests (strict claim incl. CAS,
default still takes NULL, config sandbox/state, profile
allows bwrap without deny mount). Full suites pass.
- AppArmor network rules must use socket types (stream/dgram),
not protocol names (tcp/udp/icmp): the parser rejected the
profile (Invalid network entry) until fixed.
- Tinyproxy has no 'Upstream none' syntax; the line failed the
service at startup. Direct connection is the default, so the
line is removed.
- Tinyproxy Filter is POSIX BASIC regex, not extended: the
extended-style anchored patterns denied even allowed domains
(verified: xcmapi.org got 403). Rewritten in basic syntax
(\(...\)* groups); the anchored form still blocks
evilgithub.com and xcmapi.org.evil.com (verified on host).
- Filter file must be world-readable (was 640 root:root, so
tinyproxy could not read it and exited on reload); install
steps now chmod 644 (documented in the filter header via
the install note in tinyproxy.conf).
Host already runs these exact files (proxy allow/block
verified: xcmapi/github/api.github 200, example/evilgithub/
xcmapi.org.evil.com 403).
Inner codex sandbox stays danger-full-access by choice (bwrap
path is a separate decision); this hardens everything around it:
- DockerExecutor: --security-opt plumbing (repeatable, dedups
no-new-privileges), --pids-limit (CLI default 512, fork-bomb
guard), per-call extra_env (-e NAME=VALUE, non-secrets only,
e.g. proxy URLs; the model key still goes by -e NAME only).
- Worker: --egress-proxy-port (0 = off). When on, Docker task
containers get HTTP(S)_PROXY pointing at the host filtering
proxy via their network gateway IP (tenant network gateway
or default bridge gateway). Gateway unresolvable => the
attempt fails instead of running unproxied (fail closed,
same rule as tenant network creation failure).
- AppArmor profile deploy/apparmor/docker-codex: docker-default
base + deny network raw/packet, deny ptrace, deny mount/
umount/pivot_root, deny dangerous capabilities, deny writes
to /proc/sys, /sys, /proc/keys. Deliberately NO userns rule
(that would enable the bwrap path, not this one). File rules
mirror docker-default (broad + sensitive denies); a strict
file allowlist would need profiling codex's file access set
and is not claimed here.
- Egress proxy deploy/egress/: Tinyproxy with
FilterDefaultDeny + anchored domain allowlist (relay,
github, pypi, npm). Anchoring matters: an unanchored
'github.com' regex would also match 'evilgithub.com'.
- Seccomp honesty (README): Docker default seccomp already
enforces on all task containers; no new stricter allowlist
is claimed in this change (building one from memory risks
silently breaking the toolchain; it needs syscall profiling
first).
Tests: 3 outer-hardening tests (security-opt dedup, pids
limit, proxy env without leaking the key value, gateway
parse/fail, proxy fail-closed). Full suites pass.
Container-side isolation was project-blind: every task mounted
the shared /app/work (all worktrees + host-state), the shared
codex-home (all thread history), the shared repos dir and the
shared test-repo, and all containers shared the default bridge
network. With --tenant-isolation (Docker only; Local runs on
the host under one user and has no container boundary to
enforce), each project gets its own host dirs under
tenants_root/<project>/{work,codex-home,test-repo,<project>}:
- mounts: container paths stay identical (/app/work,
codex-home, repos/<pid>, test-repo) so host.py config needs
no change; only the host mount sources are per-project.
Static --docker-extra-mounts are replaced per task (not
appended), so a shared mount cannot leak back in.
- codex-home seeds ONLY the template's config.toml (relay
endpoint/provider names, no secret values); thread_history
is never copied (it holds other tasks' prompts/results).
test-repo is a per-project copy of the fixture (worktree
creation needs a writable repo; a shared writable fixture
would let one tenant poison the next task's base code).
- repos: repo_url checkouts go to the tenant dir and task
repo is set to the container path; no-repo_url tasks get
the tenant test-repo copy instead of the shared fixture.
- network: one Docker network per project (codex-<pid>,
created idempotently, already-exists is OK); network
creation failure fails the task instead of silently
falling back to the default network (a silent downgrade
would make isolation fail quietly).
- hardening for all Docker runs: --cap-drop=ALL and
--security-opt=no-new-privileges (non-root user was already
the case); verified live by a good-model task, not assumed.
Honest limit, kept in README: the inner codex sandbox is
still danger-full-access (host kernel blocks user namespaces,
bwrap unusable), so the boundary is container + mounts +
network, not the inner sandbox.
Tests: 7 tenant tests (seeding incl. no-history, per-project
mounts, network sanitization, per-call override, ensure
network ok/fail). Full suites pass.
queue_depth turned a NULL depends_on into [None], and
ready_tasks then required None to be in completed_ids, so
such tasks never counted as ready. Worker pick_next was
unaffected (_task_from_row maps NULL to []), which is why
tasks still ran while the scaler always saw depth 0 and
never scaled up (proven live: depth probe with 1 queued
task returned 0). NULL now means no dependencies, with a
regression test.
setup_logging() takes a log LEVEL, not a logger name; passing
'cloud.autoscaler' raised ValueError at startup, so the scaler
crash-looped under systemd and started zero workers (the host
had no worker at all until this fix). The '/opt/.../-m'
FileNotFoundError in the same journal lines is only Ubuntu's
apport hook misfiring on 'python3 -m', not a second bug.
Regression test: main() with mocked run_loop/signal must reach
the loop (this would have caught the crash before deploy).
The host backup tar runs with -C /opt codex-app-server, so its
members are named codex-app-server/..., and the old patterns
./work and ./worker-heartbeat.json never matched anything.
The live work/ dir (host-state, worktrees rewritten by running
workers) was therefore included, tar hit 'file changed as we
read it', exited 1, and set -e aborted the deploy. Patterns
are now codex-app-server/work and
codex-app-server/worker-heartbeat.json, which actually match.
The worker rewrites ./worker-heartbeat.json every poll interval
(10s); the host backup tar reads the same directory, so it
could hit 'file changed as we read it', exit 1, and abort the
deploy under set -e (this killed the PR #22 deploy). The
heartbeat file is regenerable runtime state, not code, so
excluding it from the backup is correct, not a workaround.
host