Ops batch: deploy automation, backup restore drill, monitoring, multi-user verification

Deploy: deploy.sh builds tarball, backs up current code (keep 3),
deploys, restarts, health-gates 60s (ok/backend=postgres/db_ok),
auto-rolls back on failure; rollback.sh restores latest backup.
Rollback is code-only, DB migrations are additive-only so old
code can still read new columns.
Backup: restore_postgres refuses to restore into the production
DB name unless explicitly allowed (drill never allows it);
restore-drill.sh backs up fresh, restores to a test DB, compares
tasks/projects/users/audit row counts, drops the test DB.
Monitoring: worker writes heartbeat file every loop (atomic
replace, failure only logged); GET /health adds
worker_last_seen_seconds_ago + worker_ok (180s default, env
tunable, still HTTP 200 when worker is stale); monitor.sh
checks server/worker/health/pg_isready/docker-postgres/disk/
app dir as one JSON line, exit non-zero on any fail, plus
systemd timer every 5 minutes. Push alerting to chat still
needs a separate channel decision (SSH key or webhook URL).
Multi-user: verify-multiuser.py creates test users/projects
in memory-key only, checks RBAC via real HTTP API (owner/
maintainer/contributor/viewer, cross-project isolation,
merge 200/403/409, no-key 401), test tasks carry a never-
satisfiable dependency so the production worker never claims
them, cleanup deletes only multiuser-test- rows (audit log
is append-only and keeps test entries by design).
Tests: host 27 pass, cloud 298 pass (8 skipped are
environment-dependent real-Postgres tests).

cloud/backup.py

@@ -101,6 +101,61 @@ def backup_postgres(out_dir, keep=3, timestamp=None, runner=None,
     return dest
 
 
+def restore_postgres(backup_file, target_dbname, runner=None,
+                     psql_cmd="psql", allow_production=False):
+    """Postgres 恢复：用 psql 把 .sql 备份恢复到目标库，返回目标库名。
+
+    安全规则（恢复演练的核心，绝不放宽）：
+    - 目标库名绝不能等于生产库名（CLOUD_DB_NAME 环境变量读到的值），
+      相等时直接报错拒绝；只有调用方显式传 allow_production=True
+      才放行，而部署脚本（deploy/restore-drill.sh）永远不设这个值。
+    - 恢复前要求备份文件存在且非空，否则报错，不跑 psql。
+    - 连接参数与 backup_postgres 同款：host/port/user 走 CLOUD_DB_*
+      环境变量，密码只以 PGPASSWORD 环境变量传给子进程，绝不
+      出现在命令行参数、返回值或日志里。
+    - 目标库必须已存在（调用方先 CREATE DATABASE，见恢复演练
+      脚本）；psql 非零退出时明确报错（stderr 截断，不含密码）。
+    - 本函数只恢复，不删库、不改生产库；删测试库是调用方的事。
+    """
+    if not backup_file or not os.path.isfile(backup_file):
+        raise FileNotFoundError("备份文件不存在：%s" % backup_file)
+    if os.path.getsize(backup_file) == 0:
+        raise ValueError("备份文件为空，拒绝恢复：%s" % backup_file)
+    if not target_dbname or not isinstance(target_dbname, str) \
+            or not target_dbname.strip() \
+            or any(c.isspace() for c in target_dbname):
+        raise ValueError("目标库名非法：%r" % (target_dbname,))
+    production = os.environ.get("CLOUD_DB_NAME") or ""
+    if production and target_dbname == production \
+            and not allow_production:
+        raise ValueError(
+            "拒绝恢复到生产库 %s：目标库名与 CLOUD_DB_NAME 相同；"
+            "恢复演练必须用测试库名" % production)
+    host = os.environ.get("CLOUD_DB_HOST") or "127.0.0.1"
+    port = os.environ.get("CLOUD_DB_PORT") or "5432"
+    user = os.environ.get("CLOUD_DB_USER")
+    if not user:
+        raise RuntimeError("Postgres 恢复需要 CLOUD_DB_USER")
+    if shutil.which(psql_cmd) is None and runner is None:
+        raise RuntimeError("psql 不可用：本机找不到 %s" % psql_cmd)
+    cmd = [psql_cmd, "-h", host, "-p", str(port), "-U", user,
+           "-d", target_dbname, "-v", "ON_ERROR_STOP=1",
+           "-f", backup_file]
+    env = dict(os.environ)
+    password = os.environ.get("CLOUD_DB_PASSWORD")
+    if password:
+        env["PGPASSWORD"] = password
+    if runner is not None:
+        result = runner(cmd, env)
+    else:
+        result = subprocess.run(cmd, capture_output=True, text=True,
+                                env=env)
+    if getattr(result, "returncode", 1) != 0:
+        err = (getattr(result, "stderr", "") or "")[:500]
+        raise RuntimeError("psql 恢复失败：%s" % err)
+    return target_dbname
+
+
 def main(argv=None):
     parser = argparse.ArgumentParser(description="cloud control-plane backup")
     parser.add_argument("--db", default=None,