PR #145 added docker_run_* lines only to _run_in_container, but
goal/goal-resume/goal-resume-breakpoint call _call_runner directly
(to keep full stdout for goal conclusions), so the production live
test (a goal task) produced zero docker_run_ lines. Extract a
_docker_call_staged helper and route all four task paths through it;
error lines log the class name then re-raise so each caller's
kill+failed handling is unchanged.
Also fix audit attribution: audit_log has no task_id column, and
worker_exec_done/error details carried no task id, so per-task
queries missed them (the rows existed). All three exec details now
start with task_id=. Tests: 2 new goal-path stage cases + a done
detail task-id assertion (8 total in test_prestart_stages).
Claim-to-dispatch stalls (cf. 2026-10-29 task A: 61s to interrupted
with zero detail) were invisible because docker run blocks with
captured output and watchdog kills were silent. Add three layers:
- worker_exec_start/done/error audit rows in _run_attempt (PG,
survives --rm), start/done pairing shows whether an attempt died
inside the executor call;
- docker_run_start/done/error log lines in DockerExecutor (container
+ image only, never the inner command/prompt);
- stdout tail (last 2000 chars) to the worker log on failed or
interrupted results, and a log line on both watchdog kill paths.
Tests: cloud/tests/test_prestart_stages.py, 6 cases incl. a
no-prompt-in-logs assertion.
2026-10-29 live test: task A was interrupted 61s after claim but had
never written anything (no host-state entry, no worktree, no trace,
no checkpoint); auto-resume ran goal-resume, which host rightly
refused, and the task failed. Refusal only makes sense for tasks
with progress but a lost breakpoint (those go through the P0
checkpoint rebuild); for a never-started task the only meaningful
outcome is a fresh run, identical to its first attempt (same task
id, same goal, worktree does not exist yet so nothing is
overwritten).
Change (claim path only, via _resolve_breakpoint_resume):
- rebuild from checkpoint first, unchanged;
- if rebuild fails AND _task_never_started holds (state entry,
checkpoint row, worktree dir, trace file all absent), clear the
breakpoint flag so the run takes the fresh goal path, with a
worker_fresh_restart audit event;
- any evidence of prior execution, or any check failure,
keeps the old refusal path (fail-closed); the auto-resume
count cap and no-progress stall guard still bound retries.
Tests: cloud/tests/test_neverstart_fresh.py, 10 items (fallback
holds, entry/checkpoint/worktree/trace presence each block it,
corrupt state blocks it, rebuild still wins when a checkpoint
exists); test_goal_resume's breakpoint-routing test now sets up
a task with a checkpoint so it still exercises goal-resume.
Full suites: cloud 1178 (skipped 4), host 104, deploy 39, green.
run_all previously sized its pool from the ready snapshot at round start (min(capacity, ready_count)). If a round started with only 1 ready task (e.g. a just-claimed PM), only 1 thread ran; that thread entered a marker wait (up to 3600s) after its model turn, and children created/retried mid-round had no free thread to claim them. run_loop is blocked joining that round, so no new round starts; if the other worker had already scaled down, the queued child waited out the full wait timeout unclaimed. Fix: always start capacity threads. Idle threads exit immediately when no task is ready and nothing is active, and poll every 0.2s while something is active, so mid-round arrivals are claimed. Test: run_all with capacity=2, 1 initial task whose mocked run_one creates a child then polls for it; without the fix the PM poll times out in 5s (1 result), with the fix both tasks dispatch in the same round (0.2s).
Marker waits run after _run_attempt has cleared _running, so the run_loop heartbeat thread no longer refreshes the waiting parent. sweep_stale_tasks (default 300s) then kills a parent waiting up to the 3600s marker timeout, as seen in the marker live test (PM2 swept mid wait-threads). Poll and wait loops now heartbeat the parent each iteration (interval clamped to <=60s, well under the 300s threshold), best-effort, without touching _running, so no new capacity leak and cancel still works (empty _running entry has no proc/container to kill). Tests: 2 new sweep-survival tests (stale heartbeat at -299s, 2s wait, sweep must spare the parent); both fail 2/2 on pre-fix code and pass with the fix.
Coordinator tasks bypass _run_attempt, so their _claim placeholder in _running was never cleared: each completed coordinator permanently reduced the worker process effective capacity until it could claim nothing, while heartbeat stayed fresh and the capacity gate returned silently. This caused the 2026-10-28 40-person stall (autoscale-1 had one leaked slot, then claimed coordinator A and filled up) and the old 18-person C-6 never-claim (autoscale-1 leaked both A and B). Fix: wrap the run_one coordinator branch in try/finally _running_clear, so normal and exception paths both release. Also log capacity-gate refusals (worker_claim_refused_capacity) throttled to once per 30s; throttling applies to the log only, refusal always returns False. Tests: coordinator completion releases slot, next claim succeeds, capacity-log throttle; both leak tests fail on pre-fix code.
Docker tasks do not inherit the worker process environment, they only
get extra_env (-e). SEARXNG_URL in the worker unit EnvironmentFile
therefore never reached codex -> MCP inside the container, so the
auto chain skipped SearXNG and only DuckDuckGo was tried (live e2e:
'all search providers failed: duckduckgo (DNS)').
_docker_task_context now merges any set SEARXNG_URL /
PERPLEXITY_API_KEY / EXA_API_KEY / TAVILY_API_KEY / BRAVE_API_KEY
from the worker environment into extra_env (values never logged;
paid keys follow the same extra_env path as gateway keys).
Tests: 2 new docker-env tests; websearch + tenant isolation = 25
passing; full cloud suite 1038 passing (4 skipped).
Worker --websearch-mcp (default off) seeds tenant codex-home with
websearch MCP on _tenant_dirs (container path); old tenants merge
on next task. Autoscaler unit gains the flag. Tinyproxy filter
adds Perplexity/Exa/Tavily/Brave/DuckDuckGo + exact
searx.diante.me (no diante.me wildcard, control plane stays out).
Tests: test_websearch_worker 4 + websearch_mcp 12 + tenant 7 = 23 OK.
Not pushed, not merged, not deployed; SearXNG host/DNS/nginx and
tinyproxy reload are host steps.
- CODEX_GLOBAL_ADMINS replaces maintainer-derived global admin;
admin endpoints (quotas, sweep, backup, push dispatch) fail-closed
403 when unset, with audit on allow and deny.
- continue/followups/messages/retry check both caller and original
creator quotas (USD + unpriced token cap), 402 on either overrun.
- Quota-less users now get the #86 default quota instead of unlimited;
init_db backfills and OAuth create/bind/existing-login branches
INSERT OR IGNORE a default row; explicit NULL row stays unlimited.
user_quotas added to migration tables.
- /fix-review: untrusted authors rejected before dedup claim, no task;
trusted tasks force executor=docker, created_by=project owner, owner
quota gate, prompt is a fixed template plus a 2000-char excerpt.
- GitHub App setup state only issued by global admins, state rows bind
issued_by (ALTER-compatible for old DBs), callback rejects issuers
no longer in the admin list without consuming the state.
- Cancel: popen poll checks DB status every second, Docker path runs a
cancel watchdog that kills by container name, write-back is CAS-guarded
so late results never overwrite a new status.
- SHELL_ENV_EXCLUDE adds *_TOKEN, *_AUTH*, *_CREDENTIAL*, *API_KEY*;
comment states the residual risk: same-UID /proc environ remains
readable, true isolation needs UID separation or hidepid.
- Tests: new cloud/tests/test_p0_security.py (17 cases); old tests
updated to the new semantics (admin env, default quota, trusted
fix-review author) without weakening any gate.
- claim-time gate: queued task whose quota was spent while waiting is
marked interrupted with a quota reason, no cost row, resume kept
- retry/fallback boundaries and every followup round check DB month
usage plus this run's accumulated usage (priced via cost table)
- local popen poll loop kills the process group on quota (30s throttle);
Docker/sync paths get boundary checks only (documented)
- quota stops finalize as interrupted + quota_exceeded error, keep the
actual cost row, no push, followups restored to pending
DockerExecutor gains a runtime parameter (passed as docker --runtime
in build_run_command, after --pids-limit); worker CLI gains
--docker-runtime, wired through _build_cli_executor. Unset or empty
means no --runtime flag, so existing tasks keep docker default (runc);
an unregistered runtime fails container start (fail closed), it never
silently falls back. Tests: runsc appears in the command, default/empty
omit it, CLI wiring passes through. README documents the flag and
corrects a stale multi-tenant note (host kernel does not ban userns;
bwrap fails inside Docker containers on nested-pidns proc mount).
Docker outer layer and bwrap inner layer are mutually exclusive
on this host (fresh proc mounts fail with kernel EPERM inside
Docker's nested pidns, but work on the host under a userns
profile). So tasks that need an inner sandbox run on the host:
- Worker --claim-executor-only: strict executor claim (only
executor == own type, never NULL). The host worker MUST use
it, or it would steal NULL-executor tasks meant for Docker
workers (those tasks would lose the Docker outer layer).
Default behavior unchanged (NULL claimable by any worker).
- host-config-host.json: sandbox workspace-write, state
work/host-state-host.json (distinct from Docker tasks'
host-state.json), same relay home and test repo as Docker.
- deploy/apparmor/codex-host: docker-codex base + userns,
mount, pivot_root for bwrap; still denies raw network,
ptrace, dangerous capabilities, /proc/sys writes.
- deploy/host-worker/codex-host-worker.service: aa-exec -p
codex-host, --executor local --claim-executor-only,
capacity 2, proxy env 127.0.0.1:8888 so host tasks' relay/git
egress also goes through the Tinyproxy allowlist (host direct
egress would bypass it).
POST /tasks already accepts executor (local/docker/cloud), so
no API change was needed for routing.
Tests: 4 host-sandbox-routing tests (strict claim incl. CAS,
default still takes NULL, config sandbox/state, profile
allows bwrap without deny mount). Full suites pass.
host