Worker concurrency (capacity enforced), survival (heartbeat/sweep/drain), retry endpoint
1. Concurrency: capacity is now a real limit, not display-only.
run_all with capacity>1 runs a manual thread pool (threading,
not concurrent.futures: a test in this repo puts cloud/ on
sys.path, shadowing stdlib queue and breaking
concurrent.futures' SimpleQueue import). Claim stays CAS, so
two threads racing one task still yield one owner; max_tasks
counts dispatched+running to avoid overshoot; stop signals
stop new claims only, running tasks are drained by run_loop.
Running state moved from single fields to a _running dict
(proc/container per task) so cancel kills the right task.
Docker timeout is now a per-call argument, never a mutated
shared executor field. CostStore opens a fresh connection per
record/report (shared sqlite connection is thread-bound and
raised ProgrammingError under the pool; :memory: keeps the
shared connection because a fresh one is an empty DB).
Final status write is CAS on status='running' so a background
thread cannot overwrite an interrupted task with completed.
2. Survival: run_loop starts a daemon heartbeat thread (every
30s, refreshes running tasks' last_heartbeat) and sweeps stale
tasks every round (threshold 300s, best-effort); live tasks
never go stale now, dead workers' tasks are reclaimed without
manual /admin/sweep. On exit, run_loop waits up to
drain_timeout (CLI --drain-timeout, default 60s, inside
systemd's 90s stop window) for running tasks before killing
leftovers and marking interrupted.
3. Retry: POST /tasks/{id}/retry (maintainer+, 404/403/409):
failed/interrupted/merge-failed go back to queued with
result/error/model cleared, prompt/model/deps kept, CAS
update, audit task_retried.
Tests: 5 concurrency tests (barrier overlap, serial capacity=1,
max_tasks, wait_for_running, CAS final write), 3 retry tests;
full suites pass.
This commit is contained in:
1 parent
341f4a83e1
commit
d51c06a900
7 files changed
+793
-116
No files matched your search
@@ -289,8 +289,7 @@ class WorkerDockerTestCase(unittest.TestCase):
|
||||
worker = Worker(self.db_path, "/fake/host.py",
|
||||
env={"XCM_API_KEY": "placeholder-not-a-real-key"}, executor=ex)
|
||||
# 模拟 dispatch 阻塞期间:worker 已把容器名记下来。
|
||||
worker._running_task_id = "d3"
|
||||
worker._running_container = "codex-d3"
|
||||
worker._running_set("d3", container="codex-d3")
|
||||
conn = sqlite3.connect(self.db_path)
|
||||
try:
|
||||
conn.execute(
|
||||
|
||||
Reference in new issue
Block a user