Ops batch: deploy automation, backup restore drill, monitoring, multi-user verification

Deploy: deploy.sh builds tarball, backs up current code (keep 3),
deploys, restarts, health-gates 60s (ok/backend=postgres/db_ok),
auto-rolls back on failure; rollback.sh restores latest backup.
Rollback is code-only, DB migrations are additive-only so old
code can still read new columns.
Backup: restore_postgres refuses to restore into the production
DB name unless explicitly allowed (drill never allows it);
restore-drill.sh backs up fresh, restores to a test DB, compares
tasks/projects/users/audit row counts, drops the test DB.
Monitoring: worker writes heartbeat file every loop (atomic
replace, failure only logged); GET /health adds
worker_last_seen_seconds_ago + worker_ok (180s default, env
tunable, still HTTP 200 when worker is stale); monitor.sh
checks server/worker/health/pg_isready/docker-postgres/disk/
app dir as one JSON line, exit non-zero on any fail, plus
systemd timer every 5 minutes. Push alerting to chat still
needs a separate channel decision (SSH key or webhook URL).
Multi-user: verify-multiuser.py creates test users/projects
in memory-key only, checks RBAC via real HTTP API (owner/
maintainer/contributor/viewer, cross-project isolation,
merge 200/403/409, no-key 401), test tasks carry a never-
satisfiable dependency so the production worker never claims
them, cleanup deletes only multiuser-test- rows (audit log
is append-only and keeps test entries by design).
Tests: host 27 pass, cloud 298 pass (8 skipped are
environment-dependent real-Postgres tests).

cloud/server.py

@@ -156,6 +156,47 @@ def _db_backend_name():
     return (os.environ.get("CLOUD_DB_BACKEND") or "sqlite").strip().lower()
 
 
+def _worker_heartbeat_status(server=None):
+    """读 worker 运维心跳，返回 (age_seconds_or_None, worker_ok)。
+
+    心跳文件路径：测试注入优先（server.worker_heartbeat_file），
+    其次环境变量 CLOUD_WORKER_HEARTBEAT_FILE，未设则默认
+    ./worker-heartbeat.json——与 worker 的默认路径保持一致，主机
+    部署时两边都要指到同一个主机路径（见 deploy/README）。
+    阈值：server.worker_heartbeat_max_age_seconds（测试注入）或
+    环境变量 CLOUD_WORKER_HEARTBEAT_MAX_AGE_SECONDS，默认 180 秒。
+    文件不存在、读失败、JSON 非法、last_seen 非数字时一律返回
+    (None, False)，不抛异常、不影响 health 本身的 200。
+    """
+    path = None
+    max_age = 180
+    if server is not None:
+        path = getattr(server, "worker_heartbeat_file", None)
+        override = getattr(server, "worker_heartbeat_max_age_seconds", None)
+        if override is not None:
+            try:
+                max_age = float(override)
+            except (TypeError, ValueError):
+                max_age = 180
+    if not path:
+        path = os.environ.get("CLOUD_WORKER_HEARTBEAT_FILE") or \
+            "./worker-heartbeat.json"
+    if max_age == 180:
+        try:
+            max_age = float(os.environ.get(
+                "CLOUD_WORKER_HEARTBEAT_MAX_AGE_SECONDS") or 180)
+        except (TypeError, ValueError):
+            max_age = 180
+    try:
+        with open(path, encoding="utf-8") as f:
+            payload = json.load(f)
+        last_seen = float(payload.get("last_seen"))
+    except Exception:
+        return None, False
+    age = time.time() - last_seen
+    return round(age, 1), bool(age <= max_age)
+
+
 def _connect(db_path):
     """每个请求新建一个连接；后端按 CLOUD_DB_BACKEND 分发。
 
@@ -448,9 +489,16 @@ class CloudControlPlaneHandler(BaseHTTPRequestHandler):
                     conn.close()
             except Exception:
                 db_ok = False
+            # worker 字段：worker_ok 只表示“worker 心跳新鲜”，worker
+            # 不 ok 不代表 server 挂了，所以 health 本身仍返回 200，
+            # 调用方看 worker_ok 字段判断，不要把 health 改成 503
+            # 破坏已有调用（monitor.sh 也是按字段判断，不按状态码）。
+            age, worker_ok = _worker_heartbeat_status(self.server)
             self._send_json(200, {
                 "ok": True, "service": "cloud-control-plane",
                 "backend": _db_backend_name(), "db_ok": db_ok,
+                "worker_last_seen_seconds_ago": age,
+                "worker_ok": worker_ok,
             })
             return
         if path.startswith("/oauth/"):