HA 阶段 1：worktree_host + 跨主机 /pr 队列化 (#50)

* 部署自动化：唯一发布链路 deploy.py + host_deploy.sh，sha/commit 双重验真，失败回滚

旧链路三次栽过的坑：通用 tar 路径导致旧包误部署、传输不验 sha、
主机脚本多份漂移、health 通过不等于新代码上线。本单把四点变硬门禁：
- 产物唯一命名 codex-app-server-<commit>.tar.gz + .sha256，远端唯一路径
- git archive 产物 + DEPLOYED_COMMIT 清单，默认拒绝脏工作区
- 传输链 scp -> scp_run -> chunk 分片，每路传完远端验 sha，不符删掉重来
- 主机侧唯一脚本：先验 sha/清单，staging 解包 + 编译检查，换目录上线，
  work/ 原样保留，health + 上线 commit 不符即回滚并退出非零
- deploy.py 成功后独立 ssh 再读一次 live 清单二次核对
- CI：host-verify 加 deploy/tests；release.yml 只在手动/tag + production
  environment 审批时部署，主干推送绝不自动部署生产
- deploy.sh 标记为旧链路弃用
- health 的 commit 字段已在父提交 HA 阶段 0 里随 server.py 落地，
  本分支整体包含它；commit 为空时只靠文件核对，门禁同样成立
测试：host 32、cloud 487（4 跳过）、deploy 9，全部通过；
host_deploy.sh 仿真覆盖成功/坏 sha/清单不符/health commit 不符回滚。
生产实机部署未做，等确认后再用新链路推。

* HA stage 1: worktree_host + queued cross-host PR jobs

---------

Co-authored-by: host <host@test.local>

deploy/README.md

@@ -4,7 +4,68 @@
 传代码、备份、重启、验证，不改任务逻辑；运维脚本只负责检查与演练，
 不跑模型、不推代码。
 
-## 部署：deploy.sh
+## 发布链路（新，默认用这个）：deploy.py + host_deploy.sh
+
+VM 侧运行，一条命令走完构建、传输、验真、部署、回滚核对：
+
+```sh
+python3 deploy/deploy.py <host> [user]
+# 干跑（只构建、不传输、不动主机）：
+python3 deploy/deploy.py <host> --dry-run
+# 只构建发布包（给 CI artifact / 手动检查用）：
+python3 deploy/deploy.py --build-only --out-dir work/external-install/releases
+# 部署已有发布包（CI release 作业就是这么调的）：
+python3 deploy/deploy.py <host> --tarball <包> --skip-tests
+```
+
+它修掉的正是旧手工链路三次栽过的坑：
+
+- **产物唯一命名，旧包无法冒充新包。** 发布包是
+  `codex-app-server-<commit>.tar.gz` + `.sha256`，远端路径是
+  `/tmp/codex-app-server-releases/<commit>.tar.gz`，不再用
+  `/tmp/codex-app-server-code.tar.gz` 这种通用路径。scp 断了、
+  旧包残留，都不可能被下一步误当成新包。
+- **传输、主机、二次核对三道 sha/commit 门。** 传输链自动
+  fallback：scp（SSH_PROXY_CMD）→ ssh-helpers/scp_run.py（此前
+  实测最稳）→ ssh-helpers/chunk_send.py（base64 分片）；每路
+  传完都在远端跑 `sha256sum` 比对，不符就删掉该次产物再试
+  下一路，绝不把错包交给主机脚本。主机侧只用
+  `deploy/host_deploy.sh` 这一份（VM 侧随包一起传过去，不依赖
+  主机旧代码里的版本），它先验 tar sha、再验包内
+  `DEPLOYED_COMMIT`、再 staging 解包 + 编译检查，任何一步
+  不符都直接退出、旧代码原地不动。
+- **换目录代替原地 rm + 解包。** 主机侧先停服务，再把 live
+  换成 old、staging 换成 live，`work/` 与
+  `host-config-real.json`（如有）原样搬回；中途失败不会留
+  半解包的 live。备份名含旧 commit + 时间戳，留最近 5 份。
+- **health 通过不再等于上线。** 主机脚本要求 health
+  `ok=true` 且 `backend=postgres` 且 `db_ok=true`，且 live 的
+  `DEPLOYED_COMMIT` 等于期望 commit；deploy.py 成功后还要
+  独立 ssh 再读一次 live 清单核对，两处都对上才打印成功。
+  旧链路“health 通过但跑的是旧代码”在新链路里会触发回滚
+  并退出非零。health 的 `commit` 字段已在父提交 HA 阶段 0
+  里随 server.py 一起落地（`_deployed_commit`，无文件时为
+  null）；本单不依赖它也能成立，commit 为空时只靠文件核对。
+- **产物来自 git archive，默认拒绝脏工作区。** 部署的永远是
+  已提交的 HEAD，不是工作区里没提交的临时改动；`--allow-dirty`
+  也只告警，部署的仍是 HEAD，不会把脏文件偷偷带上去。
+- **CI 发布与手工发布同一套脚本。** `.github/workflows/release.yml`
+  只在手动 workflow_dispatch 或打 `v*` tag 时触发，走
+  `environment: production` 审批；主干推送绝不自动部署生产。
+  Actions 里没配 `DEPLOY_HOST` / `DEPLOY_SSH_KEY` 时，build
+  作业仍产出发布包 artifact，deploy 作业明确失败并提示改走
+  VM 侧 deploy.py，绝不静默跳过又报成功。
+
+多机滚动（HA 阶段 2 的前置）：当前 deploy.py 一次只推一台
+主机，多机时按主机逐台跑、每台 health + commit 都通过再推
+下一台，第一台失败就停，不要并行推两台。`--hosts` 批量滚动
+是后续一单，不在本轮做。
+
+密钥规则与旧脚本一致：deploy.py 绝不接受密码参数、不读
+env 文件、不打印密钥；SSH 走 key，或调用方已有的
+ssh-helpers pty 通道（密码只走 `SSH_PASSWORD` 环境变量）。
+
+## 部署：deploy.sh（旧链路，已弃用，新发布不要再用）
 
 VM 侧运行：
 

deploy/deploy.py

@@ -0,0 +1,325 @@
+#!/usr/bin/env python3
+"""统一发布入口：取代手工的 deploy.sh + scp_run + chunk_send + grep 验真。
+
+为什么需要它（此前三次栽过的坑，见 deploy/README.md）：
+  1. 通用 tar 路径 /tmp/codex-app-server-code.tar.gz：一次 scp 失败，
+     旧包还躺在原路径，下一步手工跑主机脚本就把旧代码“部署成功”了。
+  2. 传输不验 sha：scp 断在半路、chunk 首片写反，主机都可能拿到错包。
+  3. 主机脚本多份：deploy.sh 内嵌版、deploy_*_host.sh 各自演进，行为漂移。
+  4. health 通过不等于新代码上线：旧代码的 health 照样 ok=true。
+
+本脚本把以上四点变成硬门禁，任何一步不符都退出非零、绝不打印成功：
+  - 产物唯一命名：codex-app-server-<commit>.tar.gz + .sha256，
+    远端路径 /tmp/codex-app-server-releases/<commit>.tar.gz，旧包无法冒充新包。
+  - 产物来自 git archive（已提交的 HEAD），默认拒绝工作区脏文件；
+    tar 根目录带 DEPLOYED_COMMIT 清单，主机侧先验清单再动代码。
+  - 传输链自动 fallback：scp（SSH_PROXY_CMD）-> ssh-helpers/scp_run.py
+    -> ssh-helpers/chunk_send.py（base64 分片），每种方式传完都先在远端
+    跑 sha256sum 比对，不符就删掉该次产物再试下一种，绝不把错包交给主机脚本。
+  - 主机侧只用 deploy/host_deploy.sh 这一份（VM 侧随包一起传过去，
+    不依赖主机旧代码里的版本），它做 staging 换目录 + 回滚。
+  - 上线证据双重核对：主机脚本打印的 RELEASE_OK 里的 commit，
+    再加 VM 侧独立 ssh cat live 的 DEPLOYED_COMMIT，两者都必须等于期望 commit。
+
+用法：
+  python3 deploy/deploy.py <host> [user] [--ref HEAD] [--dry-run]
+  python3 deploy/deploy.py <host> --build-only [--out-dir DIR]
+  python3 deploy/deploy.py <host> --tarball <path> --commit <sha>
+  python3 deploy/deploy.py <host> --skip-tests   # CI 里测试已跑过时用
+
+密钥规则：本脚本绝不接受密码参数、不读任何 env 文件、不打印任何密钥。
+SSH 密码只由调用方已有的通道处理（key，或 ~/workspace/ssh-helpers 的
+pty helper，密码只走 SSH_PASSWORD 环境变量）；GitHub Actions 里走 key。
+"""
+import argparse
+import hashlib
+import json
+import os
+import shutil
+import subprocess
+import sys
+import tarfile
+import tempfile
+from datetime import datetime, timezone
+
+REPO_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+HOST_SCRIPT = os.path.join(REPO_ROOT, "deploy", "host_deploy.sh")
+HELPER_DIR = os.path.expanduser("~/workspace/ssh-helpers")
+REMOTE_RELEASES_DIR = "/tmp/codex-app-server-releases"
+
+
+def run(cmd, cwd=None, check=True):
+    """跑命令并返回 stdout；失败时把 stderr 透传后按 check 决定是否退出。"""
+    r = subprocess.run(cmd, cwd=cwd, capture_output=True, text=True)
+    if check and r.returncode != 0:
+        print("命令失败：%s" % " ".join(cmd), file=sys.stderr)
+        print((r.stdout + r.stderr)[-2000:], file=sys.stderr)
+        sys.exit(1)
+    return r
+
+
+def git_commit(ref):
+    return run(["git", "rev-parse", ref], cwd=REPO_ROOT).stdout.strip()
+
+
+def check_clean_tree(allow_dirty):
+    """已提交内容与工作区不一致时，默认拒绝；--allow-dirty 只告警，
+    且部署的仍是 HEAD 的 archive，脏文件不会被偷偷带上去。"""
+    diff = run(["git", "diff", "--quiet"], cwd=REPO_ROOT, check=False)
+    cached = run(["git", "diff", "--cached", "--quiet"], cwd=REPO_ROOT, check=False)
+    if diff.returncode != 0 or cached.returncode != 0:
+        if not allow_dirty:
+            print("工作区有未提交改动，先提交再部署（或显式 --allow-dirty，"
+                  "但部署的仍是 HEAD，不含脏文件）。", file=sys.stderr)
+            sys.exit(2)
+        print("警告：工作区有脏文件，部署的仍是 HEAD 的 archive，"
+              "脏文件不会上线。")
+
+
+def sha256_of(path):
+    h = hashlib.sha256()
+    with open(path, "rb") as f:
+        for chunk in iter(lambda: f.read(1024 * 1024), b""):
+            h.update(chunk)
+    return h.hexdigest()
+
+
+def release_tarball_name(commit):
+    return "codex-app-server-%s.tar.gz" % commit
+
+
+def build_release(ref, out_dir):
+    """从 git archive 建发布包，返回 (tarball_path, sha256, commit)。"""
+    commit = git_commit(ref)
+    os.makedirs(out_dir, exist_ok=True)
+    tarball = os.path.join(out_dir, release_tarball_name(commit))
+    with tempfile.TemporaryDirectory() as tmp:
+        # git archive 输出是二进制 tar 流，直接写文件，不经过文本解码。
+        with open(os.path.join(tmp, "archive.tar"), "wb") as f:
+            r = subprocess.run(["git", "archive", ref], cwd=REPO_ROOT, stdout=f)
+            if r.returncode != 0:
+                print("git archive 失败：%s" % ref, file=sys.stderr)
+                sys.exit(1)
+        with tarfile.open(os.path.join(tmp, "archive.tar")) as tf:
+            tf.extractall(tmp, filter="data")
+        with open(os.path.join(tmp, "DEPLOYED_COMMIT"), "w") as f:
+            f.write(commit + "\n")
+        with open(os.path.join(tmp, "RELEASE.json"), "w") as f:
+            json.dump({
+                "commit": commit, "ref": ref,
+                "built_at": datetime.now(timezone.utc).isoformat(),
+            }, f, indent=2, sort_keys=True)
+            f.write("\n")
+        with tarfile.open(tarball, "w:gz", format=tarfile.PAX_FORMAT) as tf:
+            for root, dirs, files in os.walk(tmp):
+                dirs.sort()
+                for name in sorted(files):
+                    full = os.path.join(root, name)
+                    arc = os.path.relpath(full, tmp)
+                    if arc in ("archive.tar",):
+                        continue
+                    tf.add(full, arcname=arc)
+    sha = sha256_of(tarball)
+    with open(tarball + ".sha256", "w") as f:
+        f.write("%s  %s\n" % (sha, os.path.basename(tarball)))
+    return tarball, sha, commit
+
+
+def tarball_commit(tarball):
+    with tarfile.open(tarball, "r:gz") as tf:
+        try:
+            member = tf.getmember("DEPLOYED_COMMIT")
+        except KeyError:
+            print("发布包里没有 DEPLOYED_COMMIT，拒绝部署：%s" % tarball,
+                  file=sys.stderr)
+            sys.exit(2)
+        data = tf.extractfile(member).read().decode().strip()
+    return data
+
+
+def parse_release_ok(output, expected_commit):
+    """主机脚本成功输出里必须有 RELEASE_OK 且 commit 等于期望值。"""
+    for line in output.splitlines():
+        if line.startswith("RELEASE_OK"):
+            fields = dict(
+                part.split("=", 1) for part in line.split()[1:]
+                if "=" in part)
+            return fields.get("commit") == expected_commit
+    return False
+
+
+# ---------------- 传输：ssh / scp 抽象 ----------------
+
+def helper_path(name):
+    path = os.path.join(HELPER_DIR, name)
+    return path if os.path.isfile(path) else None
+
+
+def ssh_cmd(host, user, remote_cmd):
+    """返回一条 ssh 命令。优先用已验证稳定的 helper，否则用
+    SSH_PROXY_CMD 环境变量，否则直连。绝不把密码拼进命令。"""
+    helper = helper_path("ssh_run.py")
+    if helper:
+        return ["python3", helper, "ssh", host, user, remote_cmd]
+    proxy = os.environ.get("SSH_PROXY_CMD")
+    base = ["ssh", "-o", "StrictHostKeyChecking=no",
+            "-o", "UserKnownHostsFile=/dev/null"]
+    if proxy:
+        base += ["-o", "ProxyCommand %s" % proxy]
+    return base + ["%s@%s" % (user, host), remote_cmd]
+
+
+def scp_cmd(host, user, local, remote):
+    helper = helper_path("scp_run.py")
+    if helper:
+        # helper 自带 egress proxy + 密码 pty，是此前实测最稳的一路。
+        return ["python3", helper, "scp", host, user, local, remote]
+    proxy = os.environ.get("SSH_PROXY_CMD")
+    base = ["scp", "-o", "StrictHostKeyChecking=no",
+            "-o", "UserKnownHostsFile=/dev/null"]
+    if proxy:
+        base += ["-o", "ProxyCommand %s" % proxy]
+    return base + [local, "%s@%s:%s" % (user, host, remote)]
+
+
+def remote_sha(host, user, remote_path):
+    r = subprocess.run(ssh_cmd(host, user, "sha256sum %s" % remote_path),
+                       capture_output=True, text=True)
+    if r.returncode != 0:
+        return None
+    return r.stdout.split()[0] if r.stdout.split() else None
+
+
+def remote_rm(host, user, remote_path):
+    subprocess.run(ssh_cmd(host, user, "rm -f %s" % remote_path),
+                   capture_output=True, text=True)
+
+
+def transfer(host, user, local, remote, expected_sha):
+    """传输链：scp -> helper scp -> chunk 分片。每路传完都验远端 sha，
+    不符删掉重来；全部失败才退出，绝不把错包交给主机脚本。"""
+    attempts = []
+    # 1. scp（含 helper 路由，由 scp_cmd 内部决定）
+    attempts.append(("scp", scp_cmd(host, user, local, remote)))
+    chunk = helper_path("chunk_send.py")
+    if chunk and os.environ.get("SSH_PASSWORD"):
+        remote_b64 = remote + ".b64"
+        attempts.append(("chunk", None))  # 特殊处理，见下
+    for name, cmd in attempts:
+        print("传输尝试：%s -> %s" % (name, remote))
+        if name == "chunk":
+            r = subprocess.run(
+                ["python3", chunk, local, remote_b64, host, user],
+                capture_output=True, text=True)
+            if r.returncode != 0:
+                print("chunk 传输失败，换下一路。", file=sys.stderr)
+                continue
+            r = subprocess.run(
+                ssh_cmd(host, user,
+                        "base64 -d %s > %s && sha256sum %s"
+                        % (remote_b64, remote, remote)),
+                capture_output=True, text=True)
+            if r.returncode != 0:
+                print("chunk 解码失败，换下一路。", file=sys.stderr)
+                continue
+        else:
+            r = subprocess.run(cmd, capture_output=True, text=True)
+            if r.returncode != 0:
+                print("scp 失败（%s），换下一路。" % (r.stderr or "").strip()[-200:],
+                      file=sys.stderr)
+                continue
+        actual = remote_sha(host, user, remote)
+        if actual == expected_sha:
+            print("传输验真通过：%s sha=%s" % (name, actual))
+            return
+        print("远端 sha 不符（expected=%s actual=%s），删掉该次产物，换下一路。"
+              % (expected_sha, actual), file=sys.stderr)
+        remote_rm(host, user, remote)
+    print("所有传输方式都失败或验真不通过，部署未开始（旧代码不动）。",
+          file=sys.stderr)
+    sys.exit(1)
+
+
+def run_tests():
+    for suite in ("tests", "cloud/tests"):
+        run(["python3", "-m", "unittest", "discover", "-s", suite],
+            cwd=REPO_ROOT)
+
+
+def main(argv=None):
+    parser = argparse.ArgumentParser(description="codex-app-server 统一发布入口")
+    parser.add_argument("host", nargs="?", help="部署目标主机")
+    parser.add_argument("user", nargs="?", default="root", help="SSH 用户")
+    parser.add_argument("--ref", default="HEAD")
+    parser.add_argument("--tarball", help="直接部署已有发布包（CI 产物）")
+    parser.add_argument("--commit", help="配合 --tarball 指定期望 commit")
+    parser.add_argument("--out-dir", default=os.path.join(
+        REPO_ROOT, "work", "external-install", "releases"))
+    parser.add_argument("--allow-dirty", action="store_true")
+    parser.add_argument("--skip-tests", action="store_true")
+    parser.add_argument("--build-only", action="store_true")
+    parser.add_argument("--dry-run", action="store_true")
+    args = parser.parse_args(argv)
+
+    if args.tarball:
+        commit = args.commit or tarball_commit(args.tarball)
+        tarball, sha = args.tarball, sha256_of(args.tarball)
+        if tarball_commit(tarball) != commit:
+            print("发布包内 DEPLOYED_COMMIT 与 --commit 不符，拒绝部署。",
+                  file=sys.stderr)
+            sys.exit(2)
+    else:
+        if not args.build_only and not args.dry_run and not args.host:
+            parser.error("需要 host，或用 --build-only / --dry-run")
+        check_clean_tree(args.allow_dirty)
+        if not args.skip_tests:
+            run_tests()
+        tarball, sha, commit = build_release(args.ref, args.out_dir)
+    print("发布包：%s" % tarball)
+    print("commit=%s sha256=%s" % (commit, sha))
+
+    if args.build_only:
+        print("仅构建，不传输、不部署。")
+        return 0
+    if args.dry_run:
+        print("[dry-run] 将传输 %s 到 %s@%s:%s/%s（唯一路径，传完验 sha）"
+              % (tarball, args.user, args.host, REMOTE_RELEASES_DIR,
+                 os.path.basename(tarball)))
+        print("[dry-run] 将传主机脚本 %s 并执行："
+              "sh host_deploy.sh <tarball> <sha> <commit>，"
+              "成功须打印 RELEASE_OK 且 commit 一致，否则回滚。" % HOST_SCRIPT)
+        return 0
+
+    host, user = args.host, args.user
+    remote_tar = "%s/%s" % (REMOTE_RELEASES_DIR, os.path.basename(tarball))
+    remote_script = "%s/host_deploy-%s.sh" % (REMOTE_RELEASES_DIR, commit)
+    run(ssh_cmd(host, user, "mkdir -p %s" % REMOTE_RELEASES_DIR), check=False)
+    transfer(host, user, tarball, remote_tar, sha)
+    transfer(host, user, HOST_SCRIPT, remote_script, sha256_of(HOST_SCRIPT))
+    print("执行主机部署脚本（唯一脚本，staging 换目录 + 失败回滚）…")
+    r = subprocess.run(
+        ssh_cmd(host, user,
+                "sh %s %s %s %s" % (remote_script, remote_tar, sha, commit)),
+        capture_output=True, text=True)
+    print(r.stdout)
+    if r.stderr:
+        print(r.stderr, file=sys.stderr)
+    if r.returncode != 0 or not parse_release_ok(r.stdout, commit):
+        print("主机部署未返回有效的 RELEASE_OK，部署失败（主机侧已回滚或未动）。",
+              file=sys.stderr)
+        sys.exit(1)
+    # 独立二次核对：不只信主机脚本的打印，再 ssh 读一次 live 清单。
+    live = subprocess.run(
+        ssh_cmd(host, user, "cat /opt/codex-app-server/DEPLOYED_COMMIT"),
+        capture_output=True, text=True)
+    if live.stdout.strip() != commit:
+        print("二次核对失败：live DEPLOYED_COMMIT=%s，期望 %s。"
+              "请立即用 deploy/rollback.sh 检查，不要再推新版。"
+              % (live.stdout.strip(), commit), file=sys.stderr)
+        sys.exit(1)
+    print("部署成功且二次核对通过：commit=%s sha256=%s" % (commit, sha))
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

deploy/host_deploy.sh

@@ -0,0 +1,214 @@
+#!/bin/sh
+# 主机侧唯一部署脚本（取代 deploy.sh 内嵌 HOST_SCRIPT 与
+# ~/workspace/ssh-helpers/deploy_*_host.sh 等多份手工脚本）。
+#
+# 设计目标：旧 tar 绝不能被误部署，health 通过绝不能代替“新代码上线”。
+#
+# 调用（VM 侧 deploy.py 传过来，绝不由人工在主机上拼路径）：
+#   sh host_deploy.sh <tarball> <expected_sha256> <expected_commit>
+#   tarball：远端唯一路径，例如 /tmp/codex-app-server-releases/<commit>.tar.gz
+#            绝不再用 /tmp/codex-app-server-code.tar.gz 这种通用路径。
+#   expected_sha256：VM 侧算好的 tar sha256，主机侧先验再动任何东西。
+#   expected_commit：git 完整 commit，tar 根目录的 DEPLOYED_COMMIT
+#            必须与它一致，否则直接退出，旧代码原地不动。
+#
+# 流程（任何一步失败都走回滚，退出非零）：
+#   1. 验 tar sha256，不符直接退出，不备份、不解包、不重启。
+#   2. 验 tar 内 DEPLOYED_COMMIT，不符直接退出。
+#   3. 解包到 staging 目录（与 APP_DIR 同一文件系统，便于 mv 换目录），
+#      并对 staging 做 python 编译检查，语法坏的包绝不上线。
+#   4. 备份当前 live 代码（备份名含旧 commit + 时间戳，留最近 5 份；
+#      备份 tar 遇运行中文件变化返回 1 时不直接 abort，只要备份文件
+#      非空就继续，见 PR #46 教训）。
+#   5. 停 server + worker/autoscaler，再换目录：live -> old，
+#      staging -> live，work/ 与 host-config-real.json（如有）原样搬回。
+#      换目录代替“原地 rm + 解包”，中途失败不会留半解包的 live。
+#   6. 启动并验 health 60 秒：ok=true 且 backend=postgres 且 db_ok=true，
+#      且 live 的 DEPLOYED_COMMIT 等于 expected_commit；health 里
+#      若有 commit 字段且非空，也必须等于 expected_commit。
+#   7. 成功打印 RELEASE_OK commit=... sha=... backup=...，删掉 old 目录
+#      （备份 tar 已留档）；失败则把 old 换回、重启、再验 health，
+#      回滚后 health 恢复也算部署失败（新代码未上线），退出非零。
+#
+# 可覆盖的环境变量（仅用于本地仿真测试，生产用默认值）：
+#   CODEX_DEPLOY_APP_DIR、CODEX_DEPLOY_BACKUP_DIR、
+#   CODEX_DEPLOY_HEALTH_INTERVAL、CODEX_DEPLOY_HEALTH_ATTEMPTS
+# 回滚边界：只回代码，不回数据库（迁移只加列、幂等，见 deploy.sh 注释）。
+# 密钥：本脚本不读 env 文件、不打印任何密钥。
+set -eu
+
+TARBALL=${1:-}
+EXPECTED_SHA=${2:-}
+EXPECTED_COMMIT=${3:-}
+if [ -z "$TARBALL" ] || [ -z "$EXPECTED_SHA" ] || [ -z "$EXPECTED_COMMIT" ]; then
+  echo "用法：sh host_deploy.sh <tarball> <expected_sha256> <expected_commit>" >&2
+  exit 2
+fi
+if [ ! -f "$TARBALL" ]; then
+  echo "部署包不存在，拒绝部署（旧代码不动）：$TARBALL" >&2
+  exit 2
+fi
+
+APP_DIR=${CODEX_DEPLOY_APP_DIR:-/opt/codex-app-server}
+APP_BASE=$(basename "$APP_DIR")
+BACKUP_DIR=${CODEX_DEPLOY_BACKUP_DIR:-/opt/codex-app-server-backups}
+STAGING_DIR=${CODEX_DEPLOY_STAGING_DIR:-/opt/codex-app-server-staging}
+OLD_DIR=${CODEX_DEPLOY_OLD_DIR:-/opt/codex-app-server-old}
+FAILED_DIR=${CODEX_DEPLOY_FAILED_DIR:-/opt/codex-app-server-failed}
+PRESERVE_DIR=${CODEX_DEPLOY_PRESERVE_DIR:-/opt/codex-app-server-preserve}
+HEALTH_INTERVAL=${CODEX_DEPLOY_HEALTH_INTERVAL:-5}
+HEALTH_ATTEMPTS=${CODEX_DEPLOY_HEALTH_ATTEMPTS:-12}
+
+# --- 1. 验 sha：不符直接退出，什么都不动 -------------------------------
+ACTUAL_SHA=$(sha256sum "$TARBALL" | awk '{print $1}')
+if [ "$ACTUAL_SHA" != "$EXPECTED_SHA" ]; then
+  echo "部署包 sha256 不符，拒绝部署（旧代码不动）：" >&2
+  echo "  expected=$EXPECTED_SHA" >&2
+  echo "  actual=$ACTUAL_SHA tarball=$TARBALL" >&2
+  exit 2
+fi
+
+# --- 2. 验 tar 内清单：DEPLOYED_COMMIT 必须等于 expected -------------
+TARBALL_COMMIT=$(tar -xzf "$TARBALL" -O DEPLOYED_COMMIT 2>/dev/null | tr -d ' \n\r' || true)
+if [ "$TARBALL_COMMIT" != "$EXPECTED_COMMIT" ]; then
+  echo "部署包内 DEPLOYED_COMMIT 与期望 commit 不符，拒绝部署（旧代码不动）：" >&2
+  echo "  expected=$EXPECTED_COMMIT tarball_commit=$TARBALL_COMMIT" >&2
+  exit 2
+fi
+
+# --- 3. staging 解包 + 编译检查 ----------------------------------------
+rm -rf "$STAGING_DIR"
+mkdir -p "$STAGING_DIR"
+tar -xzf "$TARBALL" -C "$STAGING_DIR"
+python3 -m compileall -q "$STAGING_DIR" || {
+  echo "staging 编译检查失败，拒绝部署（旧代码不动）" >&2
+  rm -rf "$STAGING_DIR"
+  exit 2
+}
+
+check_health() {
+  OUT=$(curl -s --max-time 5 http://127.0.0.1:8765/health || true)
+  [ -n "$OUT" ] && printf '%s' "$OUT" | EXPECTED_COMMIT="$EXPECTED_COMMIT" python3 -c '
+import json, os, sys
+try:
+    d = json.load(sys.stdin)
+except Exception:
+    sys.exit(1)
+if not (d.get("ok") is True and d.get("backend") == "postgres"
+        and d.get("db_ok") is True):
+    sys.exit(1)
+commit = d.get("commit")
+if commit is not None and commit != os.environ.get("EXPECTED_COMMIT"):
+    sys.exit(1)
+sys.exit(0)
+'
+}
+
+wait_health() {
+  i=0
+  while [ "$i" -lt "$HEALTH_ATTEMPTS" ]; do
+    if check_health; then return 0; fi
+    sleep "$HEALTH_INTERVAL"
+    i=$((i + 1))
+  done
+  return 1
+}
+
+worker_unit() {
+  if systemctl is-active -q codex-autoscaler 2>/dev/null; then
+    echo codex-autoscaler
+  else
+    echo codex-cloud-worker
+  fi
+}
+
+# --- 4. 备份 live（名含旧 commit，留 5 份） ------------------------------
+mkdir -p "$BACKUP_DIR"
+PREV_COMMIT=unknown
+if [ -f "$APP_DIR/DEPLOYED_COMMIT" ]; then
+  PREV_COMMIT=$(tr -d ' \n\r' < "$APP_DIR/DEPLOYED_COMMIT" || echo unknown)
+  [ -n "$PREV_COMMIT" ] || PREV_COMMIT=unknown
+fi
+STAMP=$(date +%Y%m%d%H%M%S)
+BACKUP_FILE="$BACKUP_DIR/codex-app-server-${PREV_COMMIT}-${STAMP}.tar.gz"
+# PR #46 教训：备份 tar 遇运行中文件变化返回 1 时不 abort，文件非空即继续。
+tar --exclude="$APP_BASE/work" --exclude="$APP_BASE/worker-heartbeat.json" \
+  -czf "$BACKUP_FILE" -C "$(dirname "$APP_DIR")" "$APP_BASE" || true
+if [ ! -s "$BACKUP_FILE" ]; then
+  echo "备份失败（备份文件为空），拒绝部署（旧代码不动）" >&2
+  rm -rf "$STAGING_DIR"
+  exit 2
+fi
+echo "已备份当前代码到：$BACKUP_FILE"
+ls -t "$BACKUP_DIR"/codex-app-server-*.tar.gz 2>/dev/null \
+  | tail -n +6 | xargs -r rm -f
+
+WORKER_UNIT=$(worker_unit)
+echo "停服务并换目录：worker_unit=$WORKER_UNIT"
+systemctl stop "$WORKER_UNIT" codex-cloud-server || true
+
+# --- 5. 换目录：live -> old，staging -> live，保留文件搬回 --------------
+rm -rf "$OLD_DIR" "$PRESERVE_DIR"
+mkdir -p "$PRESERVE_DIR"
+for name in work host-config-real.json; do
+  if [ -e "$APP_DIR/$name" ]; then
+    mv "$APP_DIR/$name" "$PRESERVE_DIR/$name"
+  fi
+done
+mv "$APP_DIR" "$OLD_DIR"
+mv "$STAGING_DIR" "$APP_DIR"
+for name in work host-config-real.json; do
+  if [ -e "$PRESERVE_DIR/$name" ]; then
+    mv "$PRESERVE_DIR/$name" "$APP_DIR/$name"
+  fi
+done
+rmdir "$PRESERVE_DIR" 2>/dev/null || true
+if id codexsvc >/dev/null 2>&1; then
+  chown -R codexsvc:codexsvc "$APP_DIR"
+fi
+
+rollback() {
+  echo "新代码 health 未通过，开始回滚到备份：$BACKUP_FILE" >&2
+  systemctl stop "$WORKER_UNIT" codex-cloud-server || true
+  rm -rf "$FAILED_DIR" "$PRESERVE_DIR"
+  mkdir -p "$PRESERVE_DIR"
+  for name in work host-config-real.json; do
+    if [ -e "$APP_DIR/$name" ]; then
+      mv "$APP_DIR/$name" "$PRESERVE_DIR/$name"
+    fi
+  done
+  mv "$APP_DIR" "$FAILED_DIR"
+  mv "$OLD_DIR" "$APP_DIR"
+  for name in work host-config-real.json; do
+    if [ -e "$PRESERVE_DIR/$name" ]; then
+      mv "$PRESERVE_DIR/$name" "$APP_DIR/$name"
+    fi
+  done
+  rmdir "$PRESERVE_DIR" 2>/dev/null || true
+  if id codexsvc >/dev/null 2>&1; then
+    chown -R codexsvc:codexsvc "$APP_DIR"
+  fi
+  systemctl start codex-cloud-server "$WORKER_UNIT" || true
+  if wait_health; then
+    echo "部署失败，已回滚到备份，health 已恢复（新代码未上线）" >&2
+  else
+    echo "已回滚但 health 仍失败，需人工介入" >&2
+  fi
+  exit 1
+}
+
+# --- 6. 启动 + 验 health + 验上线 commit -------------------------------
+systemctl start codex-cloud-server "$WORKER_UNIT" || true
+if ! wait_health; then
+  rollback
+fi
+LIVE_COMMIT=$(tr -d ' \n\r' < "$APP_DIR/DEPLOYED_COMMIT" 2>/dev/null || true)
+if [ "$LIVE_COMMIT" != "$EXPECTED_COMMIT" ]; then
+  echo "上线 commit 不符（health 通过但代码不是新版），开始回滚：" >&2
+  echo "  expected=$EXPECTED_COMMIT live=$LIVE_COMMIT" >&2
+  rollback
+fi
+
+rm -rf "$OLD_DIR"
+echo "RELEASE_OK commit=$EXPECTED_COMMIT sha=$EXPECTED_SHA backup=$BACKUP_FILE"
+exit 0

deploy/tests/test_deploy.py

@@ -0,0 +1,93 @@
+"""deploy.py 的单测：只测纯逻辑与发布包结构，不碰真实主机。
+
+主机侧 host_deploy.sh 的仿真见 test_host_deploy.py（假 systemctl/curl）。
+"""
+import hashlib
+import os
+import subprocess
+import sys
+import tarfile
+import tempfile
+import unittest
+
+REPO_ROOT = os.path.dirname(os.path.dirname(os.path.dirname(
+    os.path.abspath(__file__))))
+sys.path.insert(0, os.path.join(REPO_ROOT, "deploy"))
+import deploy as deploy_mod  # noqa: E402
+
+
+class TestDeployPure(unittest.TestCase):
+    def test_release_tarball_name_is_commit_scoped(self):
+        name = deploy_mod.release_tarball_name("abc123")
+        self.assertEqual(name, "codex-app-server-abc123.tar.gz")
+        # 绝不再用通用名：不同 commit 绝不可能撞到同一个远端路径。
+        self.assertNotEqual(
+            deploy_mod.release_tarball_name("abc123"),
+            deploy_mod.release_tarball_name("def456"))
+
+    def test_parse_release_ok_requires_matching_commit(self):
+        out = "部署成功\nRELEASE_OK commit=abc sha=dead backup=/tmp/x.tar.gz\n"
+        self.assertTrue(deploy_mod.parse_release_ok(out, "abc"))
+        self.assertFalse(deploy_mod.parse_release_ok(out, "other"))
+        self.assertFalse(deploy_mod.parse_release_ok("部署成功，health 通过\n", "abc"))
+
+    def test_sha256_of(self):
+        with tempfile.NamedTemporaryFile(delete=False) as f:
+            f.write(b"hello")
+            path = f.name
+        try:
+            self.assertEqual(deploy_mod.sha256_of(path),
+                             hashlib.sha256(b"hello").hexdigest())
+        finally:
+            os.unlink(path)
+
+    def test_tarball_commit_missing_manifest_refuses(self):
+        with tempfile.NamedTemporaryFile(suffix=".tar.gz", delete=False) as f:
+            path = f.name
+        try:
+            with tarfile.open(path, "w:gz") as tf:
+                pass
+            with self.assertRaises(SystemExit):
+                deploy_mod.tarball_commit(path)
+        finally:
+            os.unlink(path)
+
+
+class TestBuildRelease(unittest.TestCase):
+    def test_build_from_temp_git_repo(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            repo = os.path.join(tmp, "repo")
+            os.makedirs(repo)
+            subprocess.run(["git", "init", "-q"], cwd=repo, check=True)
+            subprocess.run(["git", "config", "user.email", "t@t"],
+                           cwd=repo, check=True)
+            subprocess.run(["git", "config", "user.name", "t"],
+                           cwd=repo, check=True)
+            with open(os.path.join(repo, "hello.py"), "w") as f:
+                f.write("x = 1\n")
+            subprocess.run(["git", "add", "."], cwd=repo, check=True)
+            subprocess.run(["git", "commit", "-qm", "init"],
+                           cwd=repo, check=True)
+            commit = subprocess.run(
+                ["git", "rev-parse", "HEAD"], cwd=repo, capture_output=True,
+                text=True, check=True).stdout.strip()
+            # 把 deploy 模块的 REPO_ROOT 临时指到这个小仓库。
+            old = deploy_mod.REPO_ROOT
+            deploy_mod.REPO_ROOT = repo
+            try:
+                out_dir = os.path.join(tmp, "out")
+                tarball, sha, got = deploy_mod.build_release("HEAD", out_dir)
+            finally:
+                deploy_mod.REPO_ROOT = old
+            self.assertEqual(got, commit)
+            self.assertEqual(deploy_mod.sha256_of(tarball), sha)
+            self.assertEqual(deploy_mod.tarball_commit(tarball), commit)
+            with tarfile.open(tarball, "r:gz") as tf:
+                names = tf.getnames()
+            self.assertIn("DEPLOYED_COMMIT", names)
+            self.assertIn("RELEASE.json", names)
+            self.assertIn("hello.py", names)
+
+
+if __name__ == "__main__":
+    unittest.main()

deploy/tests/test_host_deploy.py

@@ -0,0 +1,147 @@
+"""host_deploy.sh 的仿真测试：假 systemctl / 假 curl，不碰真实主机。
+
+每个用例都在临时目录里建一个假 APP_DIR，host_deploy.sh 的
+CODEX_DEPLOY_* 环境变量把它指过去。生产默认值不受影响。
+"""
+import hashlib
+import json
+import os
+import subprocess
+import tarfile
+import tempfile
+import unittest
+
+REPO_ROOT = os.path.dirname(os.path.dirname(os.path.dirname(
+    os.path.abspath(__file__))))
+HOST_SCRIPT = os.path.join(REPO_ROOT, "deploy", "host_deploy.sh")
+
+
+def make_tarball(source_dir, tarball):
+    with tarfile.open(tarball, "w:gz", format=tarfile.PAX_FORMAT) as tf:
+        for root, dirs, files in os.walk(source_dir):
+            dirs.sort()
+            for name in sorted(files):
+                full = os.path.join(root, name)
+                tf.add(full, arcname=os.path.relpath(full, source_dir))
+    with open(tarball, "rb") as f:
+        return hashlib.sha256(f.read()).hexdigest()
+
+
+class HostDeploySim(unittest.TestCase):
+    def setUp(self):
+        self.tmp = tempfile.TemporaryDirectory()
+        base = self.tmp.name
+        self.app = os.path.join(base, "codex-app-server")
+        os.makedirs(os.path.join(self.app, "work"))
+        with open(os.path.join(self.app, "DEPLOYED_COMMIT"), "w") as f:
+            f.write("oldcommit\n")
+        with open(os.path.join(self.app, "work", "keep.txt"), "w") as f:
+            f.write("keep\n")
+        with open(os.path.join(self.app, "old.py"), "w") as f:
+            f.write("x = 0\n")
+        # 假 systemctl：is-active 一律失败（走 cloud-worker 分支），其余成功。
+        fake_bin = os.path.join(base, "fakebin")
+        os.makedirs(fake_bin)
+        with open(os.path.join(fake_bin, "systemctl"), "w") as f:
+            f.write("#!/bin/sh\n"
+                    "echo \"$@\" >> \"$FAKE_SYSTEMCTL_LOG\"\n"
+                    "if [ \"$1\" = \"is-active\" ]; then exit 1; fi\n"
+                    "exit 0\n")
+        with open(os.path.join(fake_bin, "curl"), "w") as f:
+            f.write("#!/bin/sh\ncat \"$FAKE_HEALTH_JSON\"\n")
+        os.chmod(os.path.join(fake_bin, "systemctl"), 0o755)
+        os.chmod(os.path.join(fake_bin, "curl"), 0o755)
+        self.fake_bin = fake_bin
+        self.health_path = os.path.join(base, "health.json")
+        self.log_path = os.path.join(base, "systemctl.log")
+        open(self.log_path, "w").close()
+        self.env = dict(os.environ)
+        self.env["PATH"] = fake_bin + os.pathsep + os.environ["PATH"]
+        self.env["CODEX_DEPLOY_APP_DIR"] = self.app
+        self.env["CODEX_DEPLOY_BACKUP_DIR"] = os.path.join(base, "backups")
+        self.env["CODEX_DEPLOY_STAGING_DIR"] = os.path.join(base, "staging")
+        self.env["CODEX_DEPLOY_OLD_DIR"] = os.path.join(base, "old")
+        self.env["CODEX_DEPLOY_FAILED_DIR"] = os.path.join(base, "failed")
+        self.env["CODEX_DEPLOY_PRESERVE_DIR"] = os.path.join(base, "preserve")
+        self.env["CODEX_DEPLOY_HEALTH_INTERVAL"] = "0.1"
+        self.env["CODEX_DEPLOY_HEALTH_ATTEMPTS"] = "3"
+        self.env["FAKE_HEALTH_JSON"] = self.health_path
+        self.env["FAKE_SYSTEMCTL_LOG"] = self.log_path
+
+    def tearDown(self):
+        self.tmp.cleanup()
+
+    def set_health(self, payload):
+        with open(self.health_path, "w") as f:
+            json.dump(payload, f)
+
+    def build_new_tarball(self, commit="newcommit"):
+        src = os.path.join(self.tmp.name, "src")
+        os.makedirs(src, exist_ok=True)
+        with open(os.path.join(src, "DEPLOYED_COMMIT"), "w") as f:
+            f.write(commit + "\n")
+        with open(os.path.join(src, "new.py"), "w") as f:
+            f.write("x = 1\n")
+        tarball = os.path.join(self.tmp.name, "release.tar.gz")
+        sha = make_tarball(src, tarball)
+        return tarball, sha
+
+    def run_host(self, tarball, sha, commit):
+        return subprocess.run(
+            ["sh", HOST_SCRIPT, tarball, sha, commit],
+            capture_output=True, text=True, env=self.env)
+
+    def live_commit(self):
+        with open(os.path.join(self.app, "DEPLOYED_COMMIT")) as f:
+            return f.read().strip()
+
+    def test_success_swaps_and_preserves_work(self):
+        self.set_health({"ok": True, "backend": "postgres", "db_ok": True,
+                         "commit": "newcommit"})
+        tarball, sha = self.build_new_tarball()
+        r = self.run_host(tarball, sha, "newcommit")
+        self.assertEqual(r.returncode, 0, r.stdout + r.stderr)
+        self.assertIn("RELEASE_OK", r.stdout)
+        self.assertIn("commit=newcommit", r.stdout)
+        self.assertEqual(self.live_commit(), "newcommit")
+        with open(os.path.join(self.app, "work", "keep.txt")) as f:
+            self.assertEqual(f.read().strip(), "keep")
+        self.assertTrue(os.path.exists(os.path.join(self.app, "new.py")))
+        self.assertFalse(os.path.exists(os.path.join(self.app, "old.py")))
+        backups = os.listdir(self.env["CODEX_DEPLOY_BACKUP_DIR"])
+        self.assertEqual(len(backups), 1)
+        self.assertIn("oldcommit", backups[0])
+
+    def test_bad_sha_never_touches_live(self):
+        self.set_health({"ok": True, "backend": "postgres", "db_ok": True,
+                         "commit": "newcommit"})
+        tarball, _sha = self.build_new_tarball()
+        r = self.run_host(tarball, "0" * 64, "newcommit")
+        self.assertNotEqual(r.returncode, 0)
+        self.assertEqual(self.live_commit(), "oldcommit")
+        self.assertFalse(os.path.exists(self.env["CODEX_DEPLOY_BACKUP_DIR"])
+                         and os.listdir(self.env["CODEX_DEPLOY_BACKUP_DIR"]))
+
+    def test_manifest_mismatch_never_touches_live(self):
+        self.set_health({"ok": True, "backend": "postgres", "db_ok": True,
+                         "commit": "newcommit"})
+        tarball, sha = self.build_new_tarball(commit="othercommit")
+        r = self.run_host(tarball, sha, "newcommit")
+        self.assertNotEqual(r.returncode, 0)
+        self.assertEqual(self.live_commit(), "oldcommit")
+
+    def test_health_failure_rolls_back(self):
+        # health 通过但 commit 是旧的：这是旧链路会误报成功的情况，
+        # 新链路必须回滚，live 回到 oldcommit。
+        self.set_health({"ok": True, "backend": "postgres", "db_ok": True,
+                         "commit": "oldcommit"})
+        tarball, sha = self.build_new_tarball()
+        r = self.run_host(tarball, sha, "newcommit")
+        self.assertNotEqual(r.returncode, 0)
+        self.assertEqual(self.live_commit(), "oldcommit")
+        with open(os.path.join(self.app, "work", "keep.txt")) as f:
+            self.assertEqual(f.read().strip(), "keep")
+
+
+if __name__ == "__main__":
+    unittest.main()