PR #145 added docker_run_* lines only to _run_in_container, but
goal/goal-resume/goal-resume-breakpoint call _call_runner directly
(to keep full stdout for goal conclusions), so the production live
test (a goal task) produced zero docker_run_ lines. Extract a
_docker_call_staged helper and route all four task paths through it;
error lines log the class name then re-raise so each caller's
kill+failed handling is unchanged.
Also fix audit attribution: audit_log has no task_id column, and
worker_exec_done/error details carried no task id, so per-task
queries missed them (the rows existed). All three exec details now
start with task_id=. Tests: 2 new goal-path stage cases + a done
detail task-id assertion (8 total in test_prestart_stages).
Claim-to-dispatch stalls (cf. 2026-10-29 task A: 61s to interrupted
with zero detail) were invisible because docker run blocks with
captured output and watchdog kills were silent. Add three layers:
- worker_exec_start/done/error audit rows in _run_attempt (PG,
survives --rm), start/done pairing shows whether an attempt died
inside the executor call;
- docker_run_start/done/error log lines in DockerExecutor (container
+ image only, never the inner command/prompt);
- stdout tail (last 2000 chars) to the worker log on failed or
interrupted results, and a log line on both watchdog kill paths.
Tests: cloud/tests/test_prestart_stages.py, 6 cases incl. a
no-prompt-in-logs assertion.
2026-10-29 live test: task A was interrupted 61s after claim but had
never written anything (no host-state entry, no worktree, no trace,
no checkpoint); auto-resume ran goal-resume, which host rightly
refused, and the task failed. Refusal only makes sense for tasks
with progress but a lost breakpoint (those go through the P0
checkpoint rebuild); for a never-started task the only meaningful
outcome is a fresh run, identical to its first attempt (same task
id, same goal, worktree does not exist yet so nothing is
overwritten).
Change (claim path only, via _resolve_breakpoint_resume):
- rebuild from checkpoint first, unchanged;
- if rebuild fails AND _task_never_started holds (state entry,
checkpoint row, worktree dir, trace file all absent), clear the
breakpoint flag so the run takes the fresh goal path, with a
worker_fresh_restart audit event;
- any evidence of prior execution, or any check failure,
keeps the old refusal path (fail-closed); the auto-resume
count cap and no-progress stall guard still bound retries.
Tests: cloud/tests/test_neverstart_fresh.py, 10 items (fallback
holds, entry/checkpoint/worktree/trace presence each block it,
corrupt state blocks it, rebuild still wins when a checkpoint
exists); test_goal_resume's breakpoint-routing test now sets up
a task with a checkpoint so it still exercises goal-resume.
Full suites: cloud 1178 (skipped 4), host 104, deploy 39, green.
run_all previously sized its pool from the ready snapshot at round start (min(capacity, ready_count)). If a round started with only 1 ready task (e.g. a just-claimed PM), only 1 thread ran; that thread entered a marker wait (up to 3600s) after its model turn, and children created/retried mid-round had no free thread to claim them. run_loop is blocked joining that round, so no new round starts; if the other worker had already scaled down, the queued child waited out the full wait timeout unclaimed. Fix: always start capacity threads. Idle threads exit immediately when no task is ready and nothing is active, and poll every 0.2s while something is active, so mid-round arrivals are claimed. Test: run_all with capacity=2, 1 initial task whose mocked run_one creates a child then polls for it; without the fix the PM poll times out in 5s (1 result), with the fix both tasks dispatch in the same round (0.2s).
Marker waits run after _run_attempt has cleared _running, so the run_loop heartbeat thread no longer refreshes the waiting parent. sweep_stale_tasks (default 300s) then kills a parent waiting up to the 3600s marker timeout, as seen in the marker live test (PM2 swept mid wait-threads). Poll and wait loops now heartbeat the parent each iteration (interval clamped to <=60s, well under the 300s threshold), best-effort, without touching _running, so no new capacity leak and cancel still works (empty _running entry has no proc/container to kill). Tests: 2 new sweep-survival tests (stale heartbeat at -299s, 2s wait, sweep must spare the parent); both fail 2/2 on pre-fix code and pass with the fix.
Coordinator tasks bypass _run_attempt, so their _claim placeholder in _running was never cleared: each completed coordinator permanently reduced the worker process effective capacity until it could claim nothing, while heartbeat stayed fresh and the capacity gate returned silently. This caused the 2026-10-28 40-person stall (autoscale-1 had one leaked slot, then claimed coordinator A and filled up) and the old 18-person C-6 never-claim (autoscale-1 leaked both A and B). Fix: wrap the run_one coordinator branch in try/finally _running_clear, so normal and exception paths both release. Also log capacity-gate refusals (worker_claim_refused_capacity) throttled to once per 30s; throttling applies to the log only, refusal always returns False. Tests: coordinator completion releases slot, next claim succeeds, capacity-log throttle; both leak tests fail on pre-fix code.
Docker tasks do not inherit the worker process environment, they only
get extra_env (-e). SEARXNG_URL in the worker unit EnvironmentFile
therefore never reached codex -> MCP inside the container, so the
auto chain skipped SearXNG and only DuckDuckGo was tried (live e2e:
'all search providers failed: duckduckgo (DNS)').
_docker_task_context now merges any set SEARXNG_URL /
PERPLEXITY_API_KEY / EXA_API_KEY / TAVILY_API_KEY / BRAVE_API_KEY
from the worker environment into extra_env (values never logged;
paid keys follow the same extra_env path as gateway keys).
Tests: 2 new docker-env tests; websearch + tenant isolation = 25
passing; full cloud suite 1038 passing (4 skipped).
Worker --websearch-mcp (default off) seeds tenant codex-home with
websearch MCP on _tenant_dirs (container path); old tenants merge
on next task. Autoscaler unit gains the flag. Tinyproxy filter
adds Perplexity/Exa/Tavily/Brave/DuckDuckGo + exact
searx.diante.me (no diante.me wildcard, control plane stays out).
Tests: test_websearch_worker 4 + websearch_mcp 12 + tenant 7 = 23 OK.
Not pushed, not merged, not deployed; SearXNG host/DNS/nginx and
tinyproxy reload are host steps.
- CODEX_GLOBAL_ADMINS replaces maintainer-derived global admin;
admin endpoints (quotas, sweep, backup, push dispatch) fail-closed
403 when unset, with audit on allow and deny.
- continue/followups/messages/retry check both caller and original
creator quotas (USD + unpriced token cap), 402 on either overrun.
- Quota-less users now get the #86 default quota instead of unlimited;
init_db backfills and OAuth create/bind/existing-login branches
INSERT OR IGNORE a default row; explicit NULL row stays unlimited.
user_quotas added to migration tables.
- /fix-review: untrusted authors rejected before dedup claim, no task;
trusted tasks force executor=docker, created_by=project owner, owner
quota gate, prompt is a fixed template plus a 2000-char excerpt.
- GitHub App setup state only issued by global admins, state rows bind
issued_by (ALTER-compatible for old DBs), callback rejects issuers
no longer in the admin list without consuming the state.
- Cancel: popen poll checks DB status every second, Docker path runs a
cancel watchdog that kills by container name, write-back is CAS-guarded
so late results never overwrite a new status.
- SHELL_ENV_EXCLUDE adds *_TOKEN, *_AUTH*, *_CREDENTIAL*, *API_KEY*;
comment states the residual risk: same-UID /proc environ remains
readable, true isolation needs UID separation or hidepid.
- Tests: new cloud/tests/test_p0_security.py (17 cases); old tests
updated to the new semantics (admin env, default quota, trusted
fix-review author) without weakening any gate.
- claim-time gate: queued task whose quota was spent while waiting is
marked interrupted with a quota reason, no cost row, resume kept
- retry/fallback boundaries and every followup round check DB month
usage plus this run's accumulated usage (priced via cost table)
- local popen poll loop kills the process group on quota (30s throttle);
Docker/sync paths get boundary checks only (documented)
- quota stops finalize as interrupted + quota_exceeded error, keep the
actual cost row, no push, followups restored to pending
DockerExecutor gains a runtime parameter (passed as docker --runtime
in build_run_command, after --pids-limit); worker CLI gains
--docker-runtime, wired through _build_cli_executor. Unset or empty
means no --runtime flag, so existing tasks keep docker default (runc);
an unregistered runtime fails container start (fail closed), it never
silently falls back. Tests: runsc appears in the command, default/empty
omit it, CLI wiring passes through. README documents the flag and
corrects a stale multi-tenant note (host kernel does not ban userns;
bwrap fails inside Docker containers on nested-pidns proc mount).
host