Worker concurrency (capacity enforced), survival (heartbeat/sweep/drain), retry endpoint

1. Concurrency: capacity is now a real limit, not display-only.
   run_all with capacity>1 runs a manual thread pool (threading,
   not concurrent.futures: a test in this repo puts cloud/ on
   sys.path, shadowing stdlib queue and breaking
   concurrent.futures' SimpleQueue import). Claim stays CAS, so
   two threads racing one task still yield one owner; max_tasks
   counts dispatched+running to avoid overshoot; stop signals
   stop new claims only, running tasks are drained by run_loop.
   Running state moved from single fields to a _running dict
   (proc/container per task) so cancel kills the right task.
   Docker timeout is now a per-call argument, never a mutated
   shared executor field. CostStore opens a fresh connection per
   record/report (shared sqlite connection is thread-bound and
   raised ProgrammingError under the pool; :memory: keeps the
   shared connection because a fresh one is an empty DB).
   Final status write is CAS on status='running' so a background
   thread cannot overwrite an interrupted task with completed.
2. Survival: run_loop starts a daemon heartbeat thread (every
   30s, refreshes running tasks' last_heartbeat) and sweeps stale
   tasks every round (threshold 300s, best-effort); live tasks
   never go stale now, dead workers' tasks are reclaimed without
   manual /admin/sweep. On exit, run_loop waits up to
   drain_timeout (CLI --drain-timeout, default 60s, inside
   systemd's 90s stop window) for running tasks before killing
   leftovers and marking interrupted.
3. Retry: POST /tasks/{id}/retry (maintainer+, 404/403/409):
   failed/interrupted/merge-failed go back to queued with
   result/error/model cleared, prompt/model/deps kept, CAS
   update, audit task_retried.
Tests: 5 concurrency tests (barrier overlap, serial capacity=1,
max_tasks, wait_for_running, CAS final write), 3 retry tests;
full suites pass.

cloud/sandbox.py

@@ -420,10 +420,17 @@ class DockerExecutor(Executor):
         return cmd
 
     # -- runner 调用 / 清理 --------------------------------------------
-    def _call_runner(self, cmd):
-        """调 runner：优先传 timeout，假 runner 只接受 cmd 时回退。"""
+    def _call_runner(self, cmd, timeout=None):
+        """调 runner：优先传 timeout，假 runner 只接受 cmd 时回退。
+
+        timeout 参数只影响本次调用（并发下两个任务各用各的
+        超时）；不传时用实例的 self.timeout。绝不能在调用方
+        临时改 self.timeout 再恢复，那在并发下会互相覆盖。
+        """
         try:
-            return self.runner(cmd, timeout=self.timeout)
+            return self.runner(
+                cmd,
+                timeout=self.timeout if timeout is None else timeout)
         except TypeError:
             return self.runner(cmd)
 
@@ -493,20 +500,20 @@ class DockerExecutor(Executor):
                 continue
         return ok
 
-    def _run_in_container(self, task, inner_cmd):
+    def _run_in_container(self, task, inner_cmd, timeout=None):
         """在容器里跑一轮内层命令，返回 LocalExecutor 同款结果 dict。
 
         超时：runner 抛 subprocess.TimeoutExpired（或同名异常）
         时先 kill+rm 容器，再返回 failed（stdout 截断 + timeout
         说明，returncode 为 None）；其它 runner 异常也如实返回
-        failed，不编造成功。
+        failed，不编造成功。timeout 只影响本次调用，不改实例。
         """
         task_id = task.get("id") if isinstance(task, dict) else None
         if task_id is None:
             raise ValueError("任务缺少 id")
         cmd = self.build_run_command(task, inner_cmd)
         try:
-            result = self._call_runner(cmd)
+            result = self._call_runner(cmd, timeout=timeout)
         except Exception as exc:
             if exc.__class__.__name__ == "TimeoutExpired":
                 self.kill_container(task)
@@ -529,7 +536,7 @@ class DockerExecutor(Executor):
         parsed["container"] = self.container_name(task)
         return parsed
 
-    def dispatch_task(self, task) -> dict:
+    def dispatch_task(self, task, timeout=None) -> dict:
         """容器内调 host.py dispatch：建 host 任务并跑一轮 turn。"""
         if not isinstance(task, dict) or task.get("id") is None:
             raise ValueError("任务缺少 id")
@@ -541,9 +548,9 @@ class DockerExecutor(Executor):
         repo = task.get("repo")
         if repo:
             inner += ["--repo", repo]
-        return self._run_in_container(task, inner)
+        return self._run_in_container(task, inner, timeout=timeout)
 
-    def run_task(self, task) -> dict:
+    def run_task(self, task, timeout=None) -> dict:
         """容器内调 host.py run：在已有 host 任务上再跑一轮 turn。"""
         if not isinstance(task, dict) or task.get("id") is None:
             raise ValueError("任务缺少 id")
@@ -552,7 +559,7 @@ class DockerExecutor(Executor):
         inner = self._inner_host_command(
             "run", "--task", self._host_name(task),
             "--prompt", task["prompt"], model=task.get("model"))
-        return self._run_in_container(task, inner)
+        return self._run_in_container(task, inner, timeout=timeout)
 
     def execute(self, task) -> dict:
         """只读执行：容器内调 host.py status，不直接跑模型回合。