Worker concurrency (capacity enforced), survival (heartbeat/sweep/drain), retry endpoint

1. Concurrency: capacity is now a real limit, not display-only.
   run_all with capacity>1 runs a manual thread pool (threading,
   not concurrent.futures: a test in this repo puts cloud/ on
   sys.path, shadowing stdlib queue and breaking
   concurrent.futures' SimpleQueue import). Claim stays CAS, so
   two threads racing one task still yield one owner; max_tasks
   counts dispatched+running to avoid overshoot; stop signals
   stop new claims only, running tasks are drained by run_loop.
   Running state moved from single fields to a _running dict
   (proc/container per task) so cancel kills the right task.
   Docker timeout is now a per-call argument, never a mutated
   shared executor field. CostStore opens a fresh connection per
   record/report (shared sqlite connection is thread-bound and
   raised ProgrammingError under the pool; :memory: keeps the
   shared connection because a fresh one is an empty DB).
   Final status write is CAS on status='running' so a background
   thread cannot overwrite an interrupted task with completed.
2. Survival: run_loop starts a daemon heartbeat thread (every
   30s, refreshes running tasks' last_heartbeat) and sweeps stale
   tasks every round (threshold 300s, best-effort); live tasks
   never go stale now, dead workers' tasks are reclaimed without
   manual /admin/sweep. On exit, run_loop waits up to
   drain_timeout (CLI --drain-timeout, default 60s, inside
   systemd's 90s stop window) for running tasks before killing
   leftovers and marking interrupted.
3. Retry: POST /tasks/{id}/retry (maintainer+, 404/403/409):
   failed/interrupted/merge-failed go back to queued with
   result/error/model cleared, prompt/model/deps kept, CAS
   update, audit task_retried.
Tests: 5 concurrency tests (barrier overlap, serial capacity=1,
max_tasks, wait_for_running, CAS final write), 3 retry tests;
full suites pass.

cloud/cost.py

@@ -76,6 +76,41 @@ class CostStore:
         )
         self._conn.commit()
 
+    def _new_conn(self):
+        """每次写/读开一个新连接，用完即关。
+
+        旧实现在 __init__ 里建一条连接并终身共用，单线程没问题；
+        worker 并发后多个线程共用同一条 sqlite 连接会直接报
+        ProgrammingError（连接是线程绑定的）。其它 store（Worker
+        的 _set_status、_audit 等）本来就是每次开新连接，这里
+        向它们看齐。_conn 仍保留（建表 + 旧测试直接用它读），
+        但 record/_report 不再走它。
+        例外：db_path 为 :memory: 时新连接是另一个空库，必须
+        复用 _conn（:memory: 只用于单线程单测，并发 worker
+        从不用它）。
+        """
+        if self.db_path == ":memory:":
+            return self._conn
+        conn = _db_connect(self.db_path)
+        conn.execute(
+            """CREATE TABLE IF NOT EXISTS cost_records (
+                id INTEGER PRIMARY KEY AUTOINCREMENT,
+                task_id TEXT,
+                project_id TEXT,
+                model TEXT,
+                input_tokens INTEGER,
+                output_tokens INTEGER,
+                total_tokens INTEGER,
+                usd_micros INTEGER,
+                source TEXT,
+                fallback_count INTEGER DEFAULT 0,
+                day TEXT,
+                created_at REAL
+            )"""
+        )
+        conn.commit()
+        return conn
+
     def record(self, task_id, project_id, model, input_tokens=0,
                output_tokens=0, usd_micros=None, source="actual",
                fallback_count=0):
@@ -93,18 +128,24 @@ class CostStore:
         total_tokens = input_tokens + output_tokens
         created_at = time.time()
         day = datetime.fromtimestamp(created_at, tz=timezone.utc).strftime("%Y-%m-%d")
-        cursor = self._conn.execute(
-            """INSERT INTO cost_records
-               (task_id, project_id, model, input_tokens, output_tokens,
-                total_tokens, usd_micros, source, fallback_count, day,
-                created_at)
-               VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
-            (task_id, project_id, model, input_tokens, output_tokens,
-             total_tokens, usd_micros, source, int(fallback_count or 0),
-             day, created_at),
-        )
-        self._conn.commit()
-        return cursor.lastrowid
+        conn = self._new_conn()
+        try:
+            cursor = conn.execute(
+                """INSERT INTO cost_records
+                   (task_id, project_id, model, input_tokens, output_tokens,
+                    total_tokens, usd_micros, source, fallback_count, day,
+                    created_at)
+                   VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
+                (task_id, project_id, model, input_tokens, output_tokens,
+                 total_tokens, usd_micros, source, int(fallback_count or 0),
+                 day, created_at),
+            )
+            conn.commit()
+            return cursor.lastrowid
+        finally:
+            # :memory: 时 conn 就是共享的 _conn，绝不能关掉它。
+            if conn is not self._conn:
+                conn.close()
 
     def record_estimated(self, task_id, project_id, model, prompt_text,
                          output_text):
@@ -123,19 +164,24 @@ class CostStore:
         )
 
     def _report(self, where, params):
-        row = self._conn.execute(
-            """SELECT
-                 COALESCE(SUM(total_tokens), 0),
-                 COALESCE(SUM(usd_micros), 0),
-                 COUNT(*),
-                 COALESCE(SUM(fallback_count), 0),
-                 COALESCE(SUM(CASE WHEN source = 'actual'
-                                   THEN total_tokens ELSE 0 END), 0),
-                 COALESCE(SUM(CASE WHEN source = 'estimated'
-                                   THEN total_tokens ELSE 0 END), 0)
-               FROM cost_records WHERE %s""" % where,
-            params,
-        ).fetchone()
+        conn = self._new_conn()
+        try:
+            row = conn.execute(
+                """SELECT
+                     COALESCE(SUM(total_tokens), 0),
+                     COALESCE(SUM(usd_micros), 0),
+                     COUNT(*),
+                     COALESCE(SUM(fallback_count), 0),
+                     COALESCE(SUM(CASE WHEN source = 'actual'
+                                       THEN total_tokens ELSE 0 END), 0),
+                     COALESCE(SUM(CASE WHEN source = 'estimated'
+                                       THEN total_tokens ELSE 0 END), 0)
+                   FROM cost_records WHERE %s""" % where,
+                params,
+            ).fetchone()
+        finally:
+            if conn is not self._conn:
+                conn.close()
         total_tokens, usd_micros, turns, fallback_count, actual, estimated = row
         return {
             "total_tokens": int(total_tokens),