Ops batch: deploy automation, backup restore drill, monitoring, multi-user verification

Deploy: deploy.sh builds tarball, backs up current code (keep 3),
deploys, restarts, health-gates 60s (ok/backend=postgres/db_ok),
auto-rolls back on failure; rollback.sh restores latest backup.
Rollback is code-only, DB migrations are additive-only so old
code can still read new columns.
Backup: restore_postgres refuses to restore into the production
DB name unless explicitly allowed (drill never allows it);
restore-drill.sh backs up fresh, restores to a test DB, compares
tasks/projects/users/audit row counts, drops the test DB.
Monitoring: worker writes heartbeat file every loop (atomic
replace, failure only logged); GET /health adds
worker_last_seen_seconds_ago + worker_ok (180s default, env
tunable, still HTTP 200 when worker is stale); monitor.sh
checks server/worker/health/pg_isready/docker-postgres/disk/
app dir as one JSON line, exit non-zero on any fail, plus
systemd timer every 5 minutes. Push alerting to chat still
needs a separate channel decision (SSH key or webhook URL).
Multi-user: verify-multiuser.py creates test users/projects
in memory-key only, checks RBAC via real HTTP API (owner/
maintainer/contributor/viewer, cross-project isolation,
merge 200/403/409, no-key 401), test tasks carry a never-
satisfiable dependency so the production worker never claims
them, cleanup deletes only multiuser-test- rows (audit log
is append-only and keeps test entries by design).
Tests: host 27 pass, cloud 298 pass (8 skipped are
environment-dependent real-Postgres tests).

deploy/rollback.sh

@@ -0,0 +1,79 @@
+#!/bin/sh
+# 回滚：VM 侧运行，把主机恢复到最近一份代码备份包并重启验证。
+#
+# 用法：deploy/rollback.sh <host> [user]
+#   host：主机 IP 或域名（必填）；user：SSH 用户（默认 root）。
+#
+# 密码：与 deploy.sh 同款规则——绝不接受密码参数、不读 env 文件、
+# 不打印任何密钥；SSH 密码由交互提示或调用方通道提供；需要代理时
+# 用环境变量 SSH_PROXY_CMD 传入 ProxyCommand，未设则直连。
+#
+# 回滚边界：只回代码，不回数据库（理由见 deploy.sh 头注释与
+# deploy/README.md：数据库迁移是只加列的幂等操作，老代码仍能读新列）。
+# 备份包来自 deploy.sh 每次部署前在主机上打的
+# /opt/codex-app-server-backups/codex-app-server-<时间戳>.tar.gz，
+# 本脚本恢复其中最新的一份；主机上没有备份包时直接报错退出，不编造。
+set -eu
+
+HOST=${1:-}
+USER_ARG=${2:-root}
+if [ -z "$HOST" ]; then
+  echo "用法：deploy/rollback.sh <host> [user]" >&2
+  exit 2
+fi
+
+ssh_cmd() {
+  if [ -n "${SSH_PROXY_CMD:-}" ]; then
+    ssh -o "ProxyCommand $SSH_PROXY_CMD" -o StrictHostKeyChecking=no \
+      -o UserKnownHostsFile=/dev/null "$@"
+  else
+    ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null "$@"
+  fi
+}
+
+ssh_cmd "$USER_ARG@$HOST" 'sh -s' <<'HOST_SCRIPT'
+set -eu
+APP_DIR=/opt/codex-app-server
+BACKUP_DIR=/opt/codex-app-server-backups
+
+check_health() {
+  OUT=$(curl -s --max-time 5 http://127.0.0.1:8765/health || true)
+  [ -n "$OUT" ] && printf '%s' "$OUT" | python3 -c '
+import json, sys
+try:
+    d = json.load(sys.stdin)
+except Exception:
+    sys.exit(1)
+sys.exit(0 if (d.get("ok") is True and d.get("backend") == "postgres"
+                and d.get("db_ok") is True) else 1)
+'
+}
+
+wait_health() {
+  i=0
+  while [ "$i" -lt 12 ]; do
+    if check_health; then return 0; fi
+    sleep 5
+    i=$((i + 1))
+  done
+  return 1
+}
+
+BACKUP_FILE=$(ls -t "$BACKUP_DIR"/codex-app-server-*.tar.gz 2>/dev/null \
+  | head -n 1 || true)
+if [ -z "$BACKUP_FILE" ]; then
+  echo "主机上没有代码备份包，无法回滚：$BACKUP_DIR" >&2
+  exit 1
+fi
+echo "将恢复备份包：$BACKUP_FILE"
+systemctl stop codex-cloud-worker codex-cloud-server || true
+tar -xzf "$BACKUP_FILE" -C /opt
+chown -R codexsvc:codexsvc "$APP_DIR"
+systemctl start codex-cloud-server codex-cloud-worker
+if wait_health; then
+  echo "回滚成功，health 通过，当前代码为备份包：$BACKUP_FILE"
+  exit 0
+fi
+echo "回滚后 health 仍失败，需人工介入" >&2
+exit 1
+HOST_SCRIPT