5 Commits
Author SHA1 Message Date
host 26d9edac33 Perf: light worker poll query + tasks(status) index
Worker polls the DB every interval (production 30s) and every
poll ran SELECT * FROM tasks: full rows including result_text
and error_text (up to 4000 chars each) for every task ever,
growing with history. load_tasks now selects only the columns
pick_next actually needs (id/project/prompt/status/depends/
priority/timestamps/executor) and only queued/completed/merged
rows; running/failed rows are neither ready nor completed, so
excluding them cannot change pick_next. get_task still does
SELECT * for the single claimed task. Plus CREATE INDEX IF NOT
EXISTS idx_tasks_status on tasks(status) in both SQLite
init_db and the Postgres schema, so the poll filter stops being
a full table scan on old databases too. Test: load_tasks returns
only queued+completed out of queued/completed/running/failed.
2026-10-26 06:19:01 +08:00
host 5827716a3a Fallback: extract HTTP code from failed-turn JSON; Docker codex_home points at real relay home
1. Worker _fallback_token correction: real host.py never returns
status='404'; a failed turn is status='failed' with the real
error (e.g. '404 model not found') in text/error. The old code
returned host_status verbatim, so 'failed' blocked every real
fallback and the old event-only branch did too. Now: if
host_status itself is a fallbackable code it is used; otherwise
the code is extracted from turn JSON text/error, stdout, stderr
or runner error, and 'failed' alone never counts as a signal.
Tests: failed-turn JSON with 404 in text falls back and
succeeds; with 401 it never falls back.
2. Docker config codex_home: /var/lib/codex-app-server/codex-home,
mounted at the same path into the container, is the only home
that holds the real relay config (xcmapi provider, base_url,
env_key). Pointing codex_home at a fresh empty dir made codex
POST to api.openai.com with no auth header and loop on connect
timeouts until host.py's 600s turn timeout. The /app/work
codex-home from PR #17 was necessary to get past initialize
but insufficient for a real turn; this replaces it.
2026-10-26 06:07:07 +08:00
host 87a2ad0066 Docker default repo + keep stderr in worker error_text
Two live-test findings in one fix:
1. DockerExecutor has no repo fallback: when a project has no
repo_url, worker leaves task['repo'] unset and the container
host.py dispatch fails instantly with 'missing --repo' because
host-config-docker.json had no repo field either. Add the host
test-repo as the config default; projects with repo_url still
override it via worker's explicit --repo.
2. Worker discarded stderr from the final error_text, so every
container/host crash surfaced only as the generic 'host
execution failed' and was undebuggable from the task record.
When stdout is empty, error_text now keeps the stderr tail
(last 1000 chars, where the real error line lives).
2026-10-26 05:45:51 +08:00
host 6cee452750 Fallback also triggers on thread/start crash with explicit HTTP code
Live test found the gap: a nonexistent model fails in
thread/start, host.py exits non-zero without JSON, so the
executor result has host_status=None and the worker never
fell back (task failed, no worker_fallback audit). When
host_status is absent, worker now extracts an explicit signal
from stdout/stderr/runner error: 404/429/5xx codes, or
'model not found' style text (counted as 404). Generic
failures with no explicit code still do not fall back, and
401 in stderr still never falls back. Tests: fallback 9 pass,
cloud 333 pass, host 27 pass.
2026-10-26 05:30:04 +08:00
host 7ce603b610 Real model fallback retry (opt-in, at most once)
Worker previously only recorded a model_fallback event on a
fallbackable failure and failed the task; ModelRouter.next_after
was never used to actually run the task on another model.
With --fallback-models (default empty = off, old behaviour
unchanged), a fallbackable failure (404/429/5xx/timeout; 401/400
and cancellations never fall back) now re-runs the same task,
same worktree/branch, on the first fallback model via host.py's
top-level --model flag (no host.py signature change). The second
attempt's result is final: actual_model is the fallback model,
fallback_count=1, and a double failure keeps both reasons in
error_text. LocalExecutor/DockerExecutor pass task model through
to host.py; CloudExecutor includes model only when set.
2026-10-26 05:23:52 +08:00