Commit Graph
4 Commits
Author SHA1 Message Date
host 64995fafab P2-ops: migration alignment, deploy test list, monitor alerts/single-host, backup/rollback closure, Caddy template
- migrations: MIGRATION_TABLES now equals POSTGRES_SCHEMA tables
  (uploads, task_attachments, task_followups, environments added);
  SEQUENCE_TABLES matches IDENTITY tables; task_followups
  sender_task_id column ensured before insert; UNMIGRATED_TABLES
  and migration_coverage() report tables with no PG schema instead
  of silently skipping them; CLI prints the coverage report.
- deploy: DEFAULT_TEST_SUITES gains deploy/tests (discover -s only,
  never package path); regression tests pin the default list.
- monitor: MONITOR_SINGLE_HOST=1 makes the server check single-host;
  default stays dual to avoid silent downgrade; server_mode is in
  the JSON. Optional fail-only alert outlet (webhook or mail,
  default off, alert body is pass/fail JSON only, no secrets).
- backup/rollback: README and restore-drill document what is
  recoverable (DB, code) vs not (worktrees, traces, uploads, host
  configs); rollback.sh shares the server-unit list and dual-port
  health success standard with host_deploy.sh (sync test guards it).
- public entry: Caddyfile uses native {$DOMAIN}/{$ACME_EMAIL}
  variables plus a caddy.service template; README documents the
  steps and the current BT-nginx production status.
2026-10-27 16:37:28 +00:00
host 57f8852634 fix: worker 代码变化自退 + 部署时清孤儿 worker
2026-10-27 暂存清理实弹:05:51 的旧 docker worker 成孤儿后
继续用旧代码抢任务、按老路失败且不删暂存;autoscaler 用
start_new_session=True 拉 worker、部署重启杀不到非 unit 进程。

两层修:
1. worker 常驻循环每轮边界比对 worker.py 文件签名
   (mtime_ns + size),部署换目录后必变,变了就以 75
   (EX_TEMPFAIL,失败类)退出——检查点在轮次边界,绝不
   中断在跑任务。退出后:unit 托管的由 systemd on-failure
   重拉新版,autoscaler 托管的由 reconcile 重拉,孤儿直接
   消失。附带好处:以后部署不用再手动重启本地 worker。
2. host_deploy.sh 换目录后、启动服务前清孤儿:按 cgroup
   过滤杀非 .service 的 cloud.worker 进程(桥接修补前就存
   在的孤儿),先 TERM(跑完当前任务再退),30 秒未退的
   再 KILL;unit 托管的(如 host-1)不动。pgrep 用
   [c]loud.worker 写法,避免检查命令自匹配。

测试:test_worker.py 新增 2 个(签名变了则 75 退出且不认领、
签名不变则跑到 max_seconds);反向验证:撤掉 worker 修复
时正好前者变红。清扫过滤器主机干跑:两个现役 worker 均
为 unit cgroup,全 KEEP,杀 0 个。全量 deploy 19、host
62、cloud 619 全过。
2026-10-27 11:23:49 +08:00
openqjandhost c6e633d162 fix: host_deploy stops/starts and health-checks server-2 too (#56)
Co-authored-by: host <host@test.local>
2026-10-27 00:50:33 +08:00
openqjandhost fc8e337d0f HA 阶段 1:worktree_host + 跨主机 /pr 队列化 (#50)
* 部署自动化:唯一发布链路 deploy.py + host_deploy.sh,sha/commit 双重验真,失败回滚

旧链路三次栽过的坑:通用 tar 路径导致旧包误部署、传输不验 sha、
主机脚本多份漂移、health 通过不等于新代码上线。本单把四点变硬门禁:
- 产物唯一命名 codex-app-server-<commit>.tar.gz + .sha256,远端唯一路径
- git archive 产物 + DEPLOYED_COMMIT 清单,默认拒绝脏工作区
- 传输链 scp -> scp_run -> chunk 分片,每路传完远端验 sha,不符删掉重来
- 主机侧唯一脚本:先验 sha/清单,staging 解包 + 编译检查,换目录上线,
  work/ 原样保留,health + 上线 commit 不符即回滚并退出非零
- deploy.py 成功后独立 ssh 再读一次 live 清单二次核对
- CI:host-verify 加 deploy/tests;release.yml 只在手动/tag + production
  environment 审批时部署,主干推送绝不自动部署生产
- deploy.sh 标记为旧链路弃用
- health 的 commit 字段已在父提交 HA 阶段 0 里随 server.py 落地,
  本分支整体包含它;commit 为空时只靠文件核对,门禁同样成立
测试:host 32、cloud 487(4 跳过)、deploy 9,全部通过;
host_deploy.sh 仿真覆盖成功/坏 sha/清单不符/health commit 不符回滚。
生产实机部署未做,等确认后再用新链路推。

* HA stage 1: worktree_host + queued cross-host PR jobs

---------

Co-authored-by: host <host@test.local>
2026-10-27 00:05:18 +08:00