Worker concurrency (capacity enforced), survival (heartbeat/sweep/drain), retry endpoint

1. Concurrency: capacity is now a real limit, not display-only.
   run_all with capacity>1 runs a manual thread pool (threading,
   not concurrent.futures: a test in this repo puts cloud/ on
   sys.path, shadowing stdlib queue and breaking
   concurrent.futures' SimpleQueue import). Claim stays CAS, so
   two threads racing one task still yield one owner; max_tasks
   counts dispatched+running to avoid overshoot; stop signals
   stop new claims only, running tasks are drained by run_loop.
   Running state moved from single fields to a _running dict
   (proc/container per task) so cancel kills the right task.
   Docker timeout is now a per-call argument, never a mutated
   shared executor field. CostStore opens a fresh connection per
   record/report (shared sqlite connection is thread-bound and
   raised ProgrammingError under the pool; :memory: keeps the
   shared connection because a fresh one is an empty DB).
   Final status write is CAS on status='running' so a background
   thread cannot overwrite an interrupted task with completed.
2. Survival: run_loop starts a daemon heartbeat thread (every
   30s, refreshes running tasks' last_heartbeat) and sweeps stale
   tasks every round (threshold 300s, best-effort); live tasks
   never go stale now, dead workers' tasks are reclaimed without
   manual /admin/sweep. On exit, run_loop waits up to
   drain_timeout (CLI --drain-timeout, default 60s, inside
   systemd's 90s stop window) for running tasks before killing
   leftovers and marking interrupted.
3. Retry: POST /tasks/{id}/retry (maintainer+, 404/403/409):
   failed/interrupted/merge-failed go back to queued with
   result/error/model cleared, prompt/model/deps kept, CAS
   update, audit task_retried.
Tests: 5 concurrency tests (barrier overlap, serial capacity=1,
max_tasks, wait_for_running, CAS final write), 3 retry tests;
full suites pass.

cloud/server.py

@@ -624,6 +624,8 @@ class CloudControlPlaneHandler(BaseHTTPRequestHandler):
                 self._merge_task(user_id, rest[: -len("/merge")])
             elif rest.endswith("/pr") and method == "POST":
                 self._task_pr(user_id, rest[: -len("/pr")])
+            elif rest.endswith("/retry") and method == "POST":
+                self._retry_task(user_id, rest[: -len("/retry")])
             elif rest.endswith("/heartbeat") and method == "POST":
                 self._heartbeat_task(user_id, rest[: -len("/heartbeat")])
             else:
@@ -1321,6 +1323,67 @@ class CloudControlPlaneHandler(BaseHTTPRequestHandler):
             store.audit(user_id, task["project_id"], "merge_task", task_id)
         self._send_json(200, task)
 
+    def _retry_task(self, user_id, task_id):
+        """把终态失败任务重新排队：只改状态，不直接执行。
+
+        规则：任务必须存在（404）、调用方必须是项目 maintainer
+        以上（approve_merge，403——重跑要再花模型钱，不能让
+        只有读权限或普通贡献者随便点）、任务必须是 failed /
+        interrupted / merge-failed（其它状态一律 409：running
+        本来就在跑，queued 本来就在等，completed/merged 已有
+        结果，重跑应是新任务而不是覆盖结果）。
+        成功后状态回 queued，上一轮的 result_text/error_text/
+        actual_model 清空（否则新一轮的结果会和旧结果混在一起
+        无法分辨）；prompt、模型、优先级、依赖原样保留，依赖
+        已完成时重跑任务会立刻就绪。写审计 task_retried。
+        更新用 CAS（WHERE status IN (...)），避免与正在完成的
+        回合互相覆盖：若 CAS 没改到行，说明状态已变，返回
+        409 而不是假装成功。
+        """
+        row = self._fetch_task(task_id)
+        if row is None:
+            self._send_json(404, {"error": "task not found"})
+            return
+        task = _task_to_dict(row)
+        with _auth_store(self.server.db_path) as store:
+            if not store.check_permission(
+                    user_id, task["project_id"], "approve_merge"):
+                self._send_json(403, {"error": "forbidden"})
+                return
+        if task["status"] not in ("failed", "interrupted", "merge-failed"):
+            self._send_json(409, {
+                "error": "only failed/interrupted tasks can be retried",
+                "status": task["status"],
+            })
+            return
+        now = time.time()
+        conn = self._db()
+        try:
+            cur = conn.execute(
+                "UPDATE tasks SET status = 'queued', result_text = NULL,"
+                " error_text = NULL, actual_model = NULL, updated_at = ?"
+                " WHERE id = ? AND status IN"
+                " ('failed', 'interrupted', 'merge-failed')",
+                (now, task_id),
+            )
+            conn.commit()
+            if cur.rowcount != 1:
+                self._send_json(409, {
+                    "error": "task status changed, retry aborted",
+                    "status": task["status"],
+                })
+                return
+            task = _task_to_dict(
+                conn.execute(
+                    "SELECT * FROM tasks WHERE id = ?", (task_id,)
+                ).fetchone()
+            )
+        finally:
+            conn.close()
+        with _auth_store(self.server.db_path) as store:
+            store.audit(user_id, task["project_id"], "task_retried", task_id)
+        self._send_json(200, task)
+
     # -- worker 池 -----------------------------------------------------
     def _workers_list(self, user_id):
         """列出注册 worker：任何已认证用户可读（worker 是基础设施