check_permission now returns False for roles not in _ROLE_LEVEL
(e.g. a directly-written 'member' row, which set_member itself
rejects but legacy/direct DB rows can still carry), so the failure
mode is PermissionError/403, never an unhandled 500. Regression test
test_unknown_role_denied_not_500 fails with KeyError on old code.
GET /tasks/{id}/files previously returned 409 for all docker tasks
on the grounds that tenant docker shares /app/work per project.
Production inspection shows the worker creates a per-task
host-worktrees/<task-id> subdirectory under the tenant work dir
(62 such dirs under the wwww tenant), so per-task reads are
attributable after all: the endpoint now resolves tenant docker
tasks via tenants/<project>/work/host-worktrees/<task-id> (local
worktree as fallback), never serves the shared /app/work mount
itself, and still 409s cloud/cross-host tasks. Docs and the
harness2_tasks boundary notes are updated to match.
DispatchSection 提交时删掉 full 的 window.confirm：下拉里
选 full 本身就是显式选择，提交直接送 approval，不再二次
确认。计划模式与 full 互斥的拦截不动，其它确认框（取消/
合并/提 PR 等）不动。
构建产物 cloud/console.html 由 npm run build（tsc + vite
singlefile）重新生成，产物里已无“确认提交吗”，仅剩下拉
选项本身的“完全访问”文案。
DispatchSection gains a 'goal 模式无敌版' checkbox next to 'goal 模式':
checking it sends goal_mode=1 plus no_time_limit=1 (server already
accepts the field; the console just never exposed it). Plain goal
checkbox behavior is unchanged; invincible wins when both are
checked. Plan/coordinator mutual exclusion now covers invincible
too (it is a goal variant). TaskDetailPanel shows
'goal 模式无敌版（不限时）' when a task carries no_time_limit.
Help text states the honest boundary: worker/docker total timeouts
are lifted, the 600s single-turn host guard stays. Built with
npm run build (tsc+vite pass); console.html synced verbatim from
dist/index.html.
PR #145 added docker_run_* lines only to _run_in_container, but
goal/goal-resume/goal-resume-breakpoint call _call_runner directly
(to keep full stdout for goal conclusions), so the production live
test (a goal task) produced zero docker_run_ lines. Extract a
_docker_call_staged helper and route all four task paths through it;
error lines log the class name then re-raise so each caller's
kill+failed handling is unchanged.
Also fix audit attribution: audit_log has no task_id column, and
worker_exec_done/error details carried no task id, so per-task
queries missed them (the rows existed). All three exec details now
start with task_id=. Tests: 2 new goal-path stage cases + a done
detail task-id assertion (8 total in test_prestart_stages).
Claim-to-dispatch stalls (cf. 2026-10-29 task A: 61s to interrupted
with zero detail) were invisible because docker run blocks with
captured output and watchdog kills were silent. Add three layers:
- worker_exec_start/done/error audit rows in _run_attempt (PG,
survives --rm), start/done pairing shows whether an attempt died
inside the executor call;
- docker_run_start/done/error log lines in DockerExecutor (container
+ image only, never the inner command/prompt);
- stdout tail (last 2000 chars) to the worker log on failed or
interrupted results, and a log line on both watchdog kill paths.
Tests: cloud/tests/test_prestart_stages.py, 6 cases incl. a
no-prompt-in-logs assertion.
2026-10-29 live test: task A was interrupted 61s after claim but had
never written anything (no host-state entry, no worktree, no trace,
no checkpoint); auto-resume ran goal-resume, which host rightly
refused, and the task failed. Refusal only makes sense for tasks
with progress but a lost breakpoint (those go through the P0
checkpoint rebuild); for a never-started task the only meaningful
outcome is a fresh run, identical to its first attempt (same task
id, same goal, worktree does not exist yet so nothing is
overwritten).
Change (claim path only, via _resolve_breakpoint_resume):
- rebuild from checkpoint first, unchanged;
- if rebuild fails AND _task_never_started holds (state entry,
checkpoint row, worktree dir, trace file all absent), clear the
breakpoint flag so the run takes the fresh goal path, with a
worker_fresh_restart audit event;
- any evidence of prior execution, or any check failure,
keeps the old refusal path (fail-closed); the auto-resume
count cap and no-progress stall guard still bound retries.
Tests: cloud/tests/test_neverstart_fresh.py, 10 items (fallback
holds, entry/checkpoint/worktree/trace presence each block it,
corrupt state blocks it, rebuild still wins when a checkpoint
exists); test_goal_resume's breakpoint-routing test now sets up
a task with a checkpoint so it still exercises goal-resume.
Full suites: cloud 1178 (skipped 4), host 104, deploy 39, green.
- Role models: run_improvement_via_tasks gains role_models
(probe/editor/selector/reflector) plus --model-<role> CLI flags;
missing/empty falls back to shared --model, both empty means
server default, so default behavior is unchanged.
- Sweep: run_sweep + examples/harness2_sweep.py manifest CLI with
shared library_root (library restore), per-run failure isolation
and sweep-report.json summary; preflight gates task creation.
- Deliverable assets: GET /tasks/{id}/files for local-executor
tasks (safe path resolution, UTF-8 text <=256KB, listing <=200;
docker/cloud/cross-host 409 with explicit reason, since tenant
Docker shares one /app/work per project and per-task attribution
would misattribute); probe rollout deliverable_files from the
endpoint plus DELIVERABLE FILE reply blocks (endpoint wins on
collision, bad blocks dropped), surfaced in
deliverables/<label>/files/ context and full_result summary.
- Docs: harness2.md comparison table updated for all three items.
2026-10-29 task 2 lost its host-state entry (version 390 -> single
digits) while running under Docker/gVisor. Root cause, proven live:
- save_state tmp name contained only pid, and every containerized
host.py runs as pid 1, so concurrent writers shared one tmp file
and interleaved writes produced corrupt JSON;
- fcntl.flock does not exclude across runsc containers sharing a
bind-mounted lock file (both entered at 0.0s in a live probe);
- _read_state_file silently treated corrupt JSON as empty state, so
the next save legitimately rewrote the file with only the
surviving writer's tasks.
Fix:
- tmp name now hostname + pid + thread + uuid4 (backup tmp same);
- mkdir atomic lock replaces flock (atomic on the host fs, holds
across bind mounts/gofer), with stale-lock recovery after 120s
and a loud timeout instead of unlocked writes; reads need no lock
because publication is tmp + os.replace;
- save path uses a strict disk read: corrupt JSON, non-object top
level, or wrong-shaped tasks/automations now raise instead of
overwriting (recovery stays with restore_state + .bak).
Tests: tests/test_host_state_lock.py, 7 items (tmp uniqueness in
process and across processes, 8-process concurrent saves all
survive with version 8, stale lock recovery, fresh lock timeout
raises, corrupt/wrong-shaped disk refuses to overwrite).
Full suites: cloud 1168 (skipped 4), host 104, deploy 39, all green.
- select_final trivial condition now k==1 and single-candidate pool,
regardless of whether a selector is provided (previously gated on
selector is None, so wired runs always paid for a useless selector
task at k=1)
- Budget gains max_sessions gate (probe + stage-attempt sessions;
fmt2 corrections and selector/reflector sessions exempt), the
backstop for unpriced models where the USD gate never binds
- validate_run_tag (charset, no leading '-', no s<N> seed prefix)
enforced at run_improvement_via_tasks and the CLI
- CLI --check preflight (base harness, source dir, prompt, tag,
control-plane /health probe) without creating tasks
- ModeResult.to_dict now includes rollout summaries so a lost probe
is visible in full_result instead of only on the server
- CLI plan counts the selector only when k>1
Port Google Research's Harness^2 method layer (model weights frozen,
only the surrounding harness is optimized): Propose-Probe-Compose with
INV/DOM/AGG editors, parallel/sequential modes, per-domain experience
library, anonymous-product selection, k<=3 hard cap and budget gate.
Execution goes only through the existing control-plane API (upload ->
task with idempotency key -> poll -> trace/cost -> cancel on timeout);
no new server endpoints, no codex flags composed, tasks inherit the
normal executor/sandbox/quota/audit path.
Wiring lessons from four real rounds against production (project
wwww, ledger task): HttpTaskClient retries transient disconnects and
502/503/429 (3 attempts); editors get a worked file-block example, an
explicit <no-change> marker and one feedback retry (.fmt2) instead of
silently accepting notes-only replies; run_task unwraps the server's
result_text JSON envelope (text field holds the real reply) — without
this, line-anchored file-block parsing could never see blocks the
agents were actually emitting. Final round (real4) fully green:
all stages landed, selection non-fallback, experience入库, probe
delivers the correct signed total 85.15.
Tests: 35 harness tests (17 method + 18 wiring incl. fake-control-plane
full round, idempotent replay, retry, format retry, envelope round).
Deliberate deviations from upstream are documented in docs/harness2.md
(no network-unrestricted/sandbox-disabled defaults, auto_apply off,
no benchmark/scorer, probe deliverable = final reply text).
run_all previously sized its pool from the ready snapshot at round start (min(capacity, ready_count)). If a round started with only 1 ready task (e.g. a just-claimed PM), only 1 thread ran; that thread entered a marker wait (up to 3600s) after its model turn, and children created/retried mid-round had no free thread to claim them. run_loop is blocked joining that round, so no new round starts; if the other worker had already scaled down, the queued child waited out the full wait timeout unclaimed. Fix: always start capacity threads. Idle threads exit immediately when no task is ready and nothing is active, and poll every 0.2s while something is active, so mid-round arrivals are claimed. Test: run_all with capacity=2, 1 initial task whose mocked run_one creates a child then polls for it; without the fix the PM poll times out in 5s (1 result), with the fix both tasks dispatch in the same round (0.2s).
Marker waits run after _run_attempt has cleared _running, so the run_loop heartbeat thread no longer refreshes the waiting parent. sweep_stale_tasks (default 300s) then kills a parent waiting up to the 3600s marker timeout, as seen in the marker live test (PM2 swept mid wait-threads). Poll and wait loops now heartbeat the parent each iteration (interval clamped to <=60s, well under the 300s threshold), best-effort, without touching _running, so no new capacity leak and cancel still works (empty _running entry has no proc/container to kill). Tests: 2 new sweep-survival tests (stale heartbeat at -299s, 2s wait, sweep must spare the parent); both fail 2/2 on pre-fix code and pass with the fix.
Host can now fetch code directly from GitHub instead of receiving a VM-built tarball, bypassing the VM->host scp/kex failures and stale-tar risk. Design: VM sends only two small scripts (host_git_fetch.sh + host_deploy.sh, sha-verified via the existing transfer chain, now with a chunk_key fallback that does not need SSH_PASSWORD); host clones via a read-only deploy key (SSH only, HTTPS refused, key must exist with mode 0600), checks out the exact 40-char commit, verifies HEAD, builds a tar in the same format as the VM release (DEPLOYED_COMMIT + RELEASE.json, member names without ./ prefix so host_deploy.sh tar -O works), then calls the unchanged host_deploy.sh for staging swap, backup, health and rollback. Git fetch tolerates the known intermittent host->GitHub degradation: 120s timeout, 3 attempts, 10s sleep, live untouched until host_deploy runs. VM gates: clean tree, tests, commit must be on origin/main, SSH repo only, full sha only; --source tar remains the default. Tests: deploy suite OK, including 6 new host_git_fetch simulations (success with clean tar members, unknown commit, key permissions/missing, HTTPS refusal, short sha) and 4 new deploy.py git-mode helpers plus chunk_key fallback; git-mode dry-run verified against HEAD 630d14f.
Coordinator tasks bypass _run_attempt, so their _claim placeholder in _running was never cleared: each completed coordinator permanently reduced the worker process effective capacity until it could claim nothing, while heartbeat stayed fresh and the capacity gate returned silently. This caused the 2026-10-28 40-person stall (autoscale-1 had one leaked slot, then claimed coordinator A and filled up) and the old 18-person C-6 never-claim (autoscale-1 leaked both A and B). Fix: wrap the run_one coordinator branch in try/finally _running_clear, so normal and exception paths both release. Also log capacity-gate refusals (worker_claim_refused_capacity) throttled to once per 30s; throttling applies to the log only, refusal always returns False. Tests: coordinator completion releases slot, next claim succeeds, capacity-log throttle; both leak tests fail on pre-fix code.
Two veto layers in reconcile(), worker.py untouched:
- Global live-coordinator veto: a running task whose
coordinator_parts parse (via cloud.coordinator.parse_parts)
to a non-empty list and whose last_heartbeat is within
120s blocks all scale-down this round; low_streak is
frozen, not reset, so scale-down resumes on the next
low-depth round once the veto lifts.
- Per-victim busy veto: a LIFO victim whose workers row
is alive with running>0 is skipped, but only when the
DB also has a live running task (ghost registry rows
with no live DB task do not block).
- Coordinator-veto query failure fails closed (no stop,
guard error recorded); victim-veto read failure fails
open for that layer only.
- A missing/stale workers row is treated as idle.
Tests: 7 new ScaleDownGuardTestCase cases (live/stale
coordinator, non-coordinator, invalid parts, busy/idle
victims, ghost registry, streak freeze); cloud suite
1077 tests OK (4 skipped).
Postgres 翻译层对未登记主键的表把 INSERT OR REPLACE 退化为
ON CONFLICT DO NOTHING，致使 memory_settings、task_memory_flags、
memory_extraction_log 首写之后的所有更新静默丢弃（2026-10-28
生产实测：提取日志永远停在首写原因）。补三张表主键登记，并
在翻译层加回归测试（SQLite 原生 OR REPLACE 测不出来）。
Docker tasks do not inherit the worker process environment, they only
get extra_env (-e). SEARXNG_URL in the worker unit EnvironmentFile
therefore never reached codex -> MCP inside the container, so the
auto chain skipped SearXNG and only DuckDuckGo was tried (live e2e:
'all search providers failed: duckduckgo (DNS)').
_docker_task_context now merges any set SEARXNG_URL /
PERPLEXITY_API_KEY / EXA_API_KEY / TAVILY_API_KEY / BRAVE_API_KEY
from the worker environment into extra_env (values never logged;
paid keys follow the same extra_env path as gateway keys).
Tests: 2 new docker-env tests; websearch + tenant isolation = 25
passing; full cloud suite 1038 passing (4 skipped).
Worker --websearch-mcp (default off) seeds tenant codex-home with
websearch MCP on _tenant_dirs (container path); old tenants merge
on next task. Autoscaler unit gains the flag. Tinyproxy filter
adds Perplexity/Exa/Tavily/Brave/DuckDuckGo + exact
searx.diante.me (no diante.me wildcard, control plane stays out).
Tests: test_websearch_worker 4 + websearch_mcp 12 + tenant 7 = 23 OK.
Not pushed, not merged, not deployed; SearXNG host/DNS/nginx and
tinyproxy reload are host steps.
web_search auto chain Perplexity/Exa/Tavily/Brave/SearXNG/DuckDuckGo
with pinned-provider no-fallback, web_fetch with head+tail truncation,
codex-home seeding helper (idempotent, never writes keys).
Tests: cloud/tests/test_websearch_mcp.py 12/12 (mocked HTTP).
Not wired into worker, not deployed, paid backends not live-tested.
- migrations: MIGRATION_TABLES now equals POSTGRES_SCHEMA tables
(uploads, task_attachments, task_followups, environments added);
SEQUENCE_TABLES matches IDENTITY tables; task_followups
sender_task_id column ensured before insert; UNMIGRATED_TABLES
and migration_coverage() report tables with no PG schema instead
of silently skipping them; CLI prints the coverage report.
- deploy: DEFAULT_TEST_SUITES gains deploy/tests (discover -s only,
never package path); regression tests pin the default list.
- monitor: MONITOR_SINGLE_HOST=1 makes the server check single-host;
default stays dual to avoid silent downgrade; server_mode is in
the JSON. Optional fail-only alert outlet (webhook or mail,
default off, alert body is pass/fail JSON only, no secrets).
- backup/rollback: README and restore-drill document what is
recoverable (DB, code) vs not (worktrees, traces, uploads, host
configs); rollback.sh shares the server-unit list and dual-port
health success standard with host_deploy.sh (sync test guards it).
- public entry: Caddyfile uses native {$DOMAIN}/{$ACME_EMAIL}
variables plus a caddy.service template; README documents the
steps and the current BT-nginx production status.
- CODEX_GLOBAL_ADMINS replaces maintainer-derived global admin;
admin endpoints (quotas, sweep, backup, push dispatch) fail-closed
403 when unset, with audit on allow and deny.
- continue/followups/messages/retry check both caller and original
creator quotas (USD + unpriced token cap), 402 on either overrun.
- Quota-less users now get the #86 default quota instead of unlimited;
init_db backfills and OAuth create/bind/existing-login branches
INSERT OR IGNORE a default row; explicit NULL row stays unlimited.
user_quotas added to migration tables.
- /fix-review: untrusted authors rejected before dedup claim, no task;
trusted tasks force executor=docker, created_by=project owner, owner
quota gate, prompt is a fixed template plus a 2000-char excerpt.
- GitHub App setup state only issued by global admins, state rows bind
issued_by (ALTER-compatible for old DBs), callback rejects issuers
no longer in the admin list without consuming the state.
- Cancel: popen poll checks DB status every second, Docker path runs a
cancel watchdog that kills by container name, write-back is CAS-guarded
so late results never overwrite a new status.
- SHELL_ENV_EXCLUDE adds *_TOKEN, *_AUTH*, *_CREDENTIAL*, *API_KEY*;
comment states the residual risk: same-UID /proc environ remains
readable, true isolation needs UID separation or hidepid.
- Tests: new cloud/tests/test_p0_security.py (17 cases); old tests
updated to the new semantics (admin env, default quota, trusted
fix-review author) without weakening any gate.
- claim-time gate: queued task whose quota was spent while waiting is
marked interrupted with a quota reason, no cost row, resume kept
- retry/fallback boundaries and every followup round check DB month
usage plus this run's accumulated usage (priced via cost table)
- local popen poll loop kills the process group on quota (30s throttle);
Docker/sync paths get boundary checks only (documented)
- quota stops finalize as interrupted + quota_exceeded error, keep the
actual cost row, no push, followups restored to pending
host