2026-10-29 live test: task A was interrupted 61s after claim but had
never written anything (no host-state entry, no worktree, no trace,
no checkpoint); auto-resume ran goal-resume, which host rightly
refused, and the task failed. Refusal only makes sense for tasks
with progress but a lost breakpoint (those go through the P0
checkpoint rebuild); for a never-started task the only meaningful
outcome is a fresh run, identical to its first attempt (same task
id, same goal, worktree does not exist yet so nothing is
overwritten).
Change (claim path only, via _resolve_breakpoint_resume):
- rebuild from checkpoint first, unchanged;
- if rebuild fails AND _task_never_started holds (state entry,
checkpoint row, worktree dir, trace file all absent), clear the
breakpoint flag so the run takes the fresh goal path, with a
worker_fresh_restart audit event;
- any evidence of prior execution, or any check failure,
keeps the old refusal path (fail-closed); the auto-resume
count cap and no-progress stall guard still bound retries.
Tests: cloud/tests/test_neverstart_fresh.py, 10 items (fallback
holds, entry/checkpoint/worktree/trace presence each block it,
corrupt state blocks it, rebuild still wins when a checkpoint
exists); test_goal_resume's breakpoint-routing test now sets up
a task with a checkpoint so it still exercises goal-resume.
Full suites: cloud 1178 (skipped 4), host 104, deploy 39, green.
- select_final trivial condition now k==1 and single-candidate pool,
regardless of whether a selector is provided (previously gated on
selector is None, so wired runs always paid for a useless selector
task at k=1)
- Budget gains max_sessions gate (probe + stage-attempt sessions;
fmt2 corrections and selector/reflector sessions exempt), the
backstop for unpriced models where the USD gate never binds
- validate_run_tag (charset, no leading '-', no s<N> seed prefix)
enforced at run_improvement_via_tasks and the CLI
- CLI --check preflight (base harness, source dir, prompt, tag,
control-plane /health probe) without creating tasks
- ModeResult.to_dict now includes rollout summaries so a lost probe
is visible in full_result instead of only on the server
- CLI plan counts the selector only when k>1
Port Google Research's Harness^2 method layer (model weights frozen,
only the surrounding harness is optimized): Propose-Probe-Compose with
INV/DOM/AGG editors, parallel/sequential modes, per-domain experience
library, anonymous-product selection, k<=3 hard cap and budget gate.
Execution goes only through the existing control-plane API (upload ->
task with idempotency key -> poll -> trace/cost -> cancel on timeout);
no new server endpoints, no codex flags composed, tasks inherit the
normal executor/sandbox/quota/audit path.
Wiring lessons from four real rounds against production (project
wwww, ledger task): HttpTaskClient retries transient disconnects and
502/503/429 (3 attempts); editors get a worked file-block example, an
explicit <no-change> marker and one feedback retry (.fmt2) instead of
silently accepting notes-only replies; run_task unwraps the server's
result_text JSON envelope (text field holds the real reply) — without
this, line-anchored file-block parsing could never see blocks the
agents were actually emitting. Final round (real4) fully green:
all stages landed, selection non-fallback, experience入库, probe
delivers the correct signed total 85.15.
Tests: 35 harness tests (17 method + 18 wiring incl. fake-control-plane
full round, idempotent replay, retry, format retry, envelope round).
Deliberate deviations from upstream are documented in docs/harness2.md
(no network-unrestricted/sandbox-disabled defaults, auto_apply off,
no benchmark/scorer, probe deliverable = final reply text).
run_all previously sized its pool from the ready snapshot at round start (min(capacity, ready_count)). If a round started with only 1 ready task (e.g. a just-claimed PM), only 1 thread ran; that thread entered a marker wait (up to 3600s) after its model turn, and children created/retried mid-round had no free thread to claim them. run_loop is blocked joining that round, so no new round starts; if the other worker had already scaled down, the queued child waited out the full wait timeout unclaimed. Fix: always start capacity threads. Idle threads exit immediately when no task is ready and nothing is active, and poll every 0.2s while something is active, so mid-round arrivals are claimed. Test: run_all with capacity=2, 1 initial task whose mocked run_one creates a child then polls for it; without the fix the PM poll times out in 5s (1 result), with the fix both tasks dispatch in the same round (0.2s).
Marker waits run after _run_attempt has cleared _running, so the run_loop heartbeat thread no longer refreshes the waiting parent. sweep_stale_tasks (default 300s) then kills a parent waiting up to the 3600s marker timeout, as seen in the marker live test (PM2 swept mid wait-threads). Poll and wait loops now heartbeat the parent each iteration (interval clamped to <=60s, well under the 300s threshold), best-effort, without touching _running, so no new capacity leak and cancel still works (empty _running entry has no proc/container to kill). Tests: 2 new sweep-survival tests (stale heartbeat at -299s, 2s wait, sweep must spare the parent); both fail 2/2 on pre-fix code and pass with the fix.
Coordinator tasks bypass _run_attempt, so their _claim placeholder in _running was never cleared: each completed coordinator permanently reduced the worker process effective capacity until it could claim nothing, while heartbeat stayed fresh and the capacity gate returned silently. This caused the 2026-10-28 40-person stall (autoscale-1 had one leaked slot, then claimed coordinator A and filled up) and the old 18-person C-6 never-claim (autoscale-1 leaked both A and B). Fix: wrap the run_one coordinator branch in try/finally _running_clear, so normal and exception paths both release. Also log capacity-gate refusals (worker_claim_refused_capacity) throttled to once per 30s; throttling applies to the log only, refusal always returns False. Tests: coordinator completion releases slot, next claim succeeds, capacity-log throttle; both leak tests fail on pre-fix code.
Two veto layers in reconcile(), worker.py untouched:
- Global live-coordinator veto: a running task whose
coordinator_parts parse (via cloud.coordinator.parse_parts)
to a non-empty list and whose last_heartbeat is within
120s blocks all scale-down this round; low_streak is
frozen, not reset, so scale-down resumes on the next
low-depth round once the veto lifts.
- Per-victim busy veto: a LIFO victim whose workers row
is alive with running>0 is skipped, but only when the
DB also has a live running task (ghost registry rows
with no live DB task do not block).
- Coordinator-veto query failure fails closed (no stop,
guard error recorded); victim-veto read failure fails
open for that layer only.
- A missing/stale workers row is treated as idle.
Tests: 7 new ScaleDownGuardTestCase cases (live/stale
coordinator, non-coordinator, invalid parts, busy/idle
victims, ghost registry, streak freeze); cloud suite
1077 tests OK (4 skipped).
Postgres 翻译层对未登记主键的表把 INSERT OR REPLACE 退化为
ON CONFLICT DO NOTHING，致使 memory_settings、task_memory_flags、
memory_extraction_log 首写之后的所有更新静默丢弃（2026-10-28
生产实测：提取日志永远停在首写原因）。补三张表主键登记，并
在翻译层加回归测试（SQLite 原生 OR REPLACE 测不出来）。
Docker tasks do not inherit the worker process environment, they only
get extra_env (-e). SEARXNG_URL in the worker unit EnvironmentFile
therefore never reached codex -> MCP inside the container, so the
auto chain skipped SearXNG and only DuckDuckGo was tried (live e2e:
'all search providers failed: duckduckgo (DNS)').
_docker_task_context now merges any set SEARXNG_URL /
PERPLEXITY_API_KEY / EXA_API_KEY / TAVILY_API_KEY / BRAVE_API_KEY
from the worker environment into extra_env (values never logged;
paid keys follow the same extra_env path as gateway keys).
Tests: 2 new docker-env tests; websearch + tenant isolation = 25
passing; full cloud suite 1038 passing (4 skipped).
Worker --websearch-mcp (default off) seeds tenant codex-home with
websearch MCP on _tenant_dirs (container path); old tenants merge
on next task. Autoscaler unit gains the flag. Tinyproxy filter
adds Perplexity/Exa/Tavily/Brave/DuckDuckGo + exact
searx.diante.me (no diante.me wildcard, control plane stays out).
Tests: test_websearch_worker 4 + websearch_mcp 12 + tenant 7 = 23 OK.
Not pushed, not merged, not deployed; SearXNG host/DNS/nginx and
tinyproxy reload are host steps.
web_search auto chain Perplexity/Exa/Tavily/Brave/SearXNG/DuckDuckGo
with pinned-provider no-fallback, web_fetch with head+tail truncation,
codex-home seeding helper (idempotent, never writes keys).
Tests: cloud/tests/test_websearch_mcp.py 12/12 (mocked HTTP).
Not wired into worker, not deployed, paid backends not live-tested.
- migrations: MIGRATION_TABLES now equals POSTGRES_SCHEMA tables
(uploads, task_attachments, task_followups, environments added);
SEQUENCE_TABLES matches IDENTITY tables; task_followups
sender_task_id column ensured before insert; UNMIGRATED_TABLES
and migration_coverage() report tables with no PG schema instead
of silently skipping them; CLI prints the coverage report.
- deploy: DEFAULT_TEST_SUITES gains deploy/tests (discover -s only,
never package path); regression tests pin the default list.
- monitor: MONITOR_SINGLE_HOST=1 makes the server check single-host;
default stays dual to avoid silent downgrade; server_mode is in
the JSON. Optional fail-only alert outlet (webhook or mail,
default off, alert body is pass/fail JSON only, no secrets).
- backup/rollback: README and restore-drill document what is
recoverable (DB, code) vs not (worktrees, traces, uploads, host
configs); rollback.sh shares the server-unit list and dual-port
health success standard with host_deploy.sh (sync test guards it).
- public entry: Caddyfile uses native {$DOMAIN}/{$ACME_EMAIL}
variables plus a caddy.service template; README documents the
steps and the current BT-nginx production status.
host