feat: CHV gap fill — worker breakpoint wiring, pool reconcile, per-VM host keys, egress validation, guest build script
Stack: #216 -> #221 -> this. Worker breakpoint branch now calls any executor providing goal_resume_breakpoint_task (Local keeps its original path, gateway tasks still loud-fail); WarmPool reconcile_orphans is called on CHV worker startup and close with audit-on-failure; per-VM SSH host keys are provisioned before VMM start via cloud-init NoCloud seed (keyscan kept only as weak fallback, CHV_HOST_KEY_MODE=provisioned|keyscan); CHV_NET_JSON is validated in preflight and explicit direct egress is audited; chv-guest/build.sh is now real steps with --dry-run and fails fast without /dev/kvm. Tests: new test_chv_gaps.py (18), cloud full 1514 OK (skipped=4). No real VM verification: no KVM host.
This commit is contained in:
1 parent
fde7bf4c3e
commit
22b83f118a
8 files changed
+922
-59
No files matched your search
+35
-11
@@ -1,11 +1,12 @@
|
||||
# Cloud Hypervisor 客机镜像(chv-guest)——内容清单与版本化构建说明
|
||||
|
||||
> 本目录只有清单与构建脚本骨架,镜像本身不在本轨内实际构建
|
||||
> 发布。原因写在前面,不藏着:当前没有带 /dev/kvm 的宿主
|
||||
> 可以构建并实测 VM 镜像(生产主机是 CVM、无嵌套虚拟化;
|
||||
> 本 VM 容器内同样无 /dev/kvm)。构建与实测见
|
||||
> `docs/chv-production-readiness.md` 的“需镜像构建 / 需 KVM
|
||||
> 宿主实测”项,绝不在本轨声称镜像已建成、已发布、已跑通。
|
||||
> 本目录有清单与真实构建步骤(build.sh,补缺口轨已实做),
|
||||
> 但镜像本身不在本轨内实际构建发布。原因写在前面,不藏着:
|
||||
> 当前没有带 /dev/kvm 的宿主可以构建并实测 VM 镜像(生产
|
||||
> 主机是 CVM、无嵌套虚拟化;本 VM 容器内同样无 /dev/kvm)。
|
||||
> 构建与实测见 `docs/chv-production-readiness.md` 的“需镜像
|
||||
> 构建 / 需 KVM 宿主实测”项,绝不在本轨声称镜像已建成、
|
||||
> 已发布、已跑通。
|
||||
|
||||
## 1. 镜像定位
|
||||
|
||||
@@ -53,12 +54,35 @@ chv-guest-<ubuntu 版本>-<codex 版本>-<仓库 commit 短 hash>
|
||||
- `BUILDINFO.json`:`{ubuntu, codex_version, repo_commit, built_at,
|
||||
contents: [清单逐项版本]}`
|
||||
|
||||
## 4. 构建脚本骨架
|
||||
## 4. 构建脚本(补缺口轨:已从骨架改成真实步骤)
|
||||
|
||||
见 `build.sh`:只做参数校验、产物路径与版本 tag 计算、以及
|
||||
“需要 KVM 宿主 / virt-customize 才能执行的步骤”占位。脚本在
|
||||
无 KVM 的机器上运行时应在第一步就明确退出(`CHV_GUEST_BUILD
|
||||
needs a KVM host`),绝不假装构建成功。
|
||||
见 `build.sh`:参数校验 → 下载 Ubuntu 云镜像 → qemu-img
|
||||
转 raw → virt-customize 装 §2 清单、拷入 codex 成套二进制
|
||||
与仓库代码到 /app、写 host key 不重生成配置 →
|
||||
virt-get-kernel 取内核 → 生成并校验 BUILDINFO.json。
|
||||
`./build.sh --dry-run out/` 只打印命令、不执行、不要求
|
||||
KVM;全量模式在无 /dev/kvm 的机器上第一步就明确退出
|
||||
(`CHV_GUEST_BUILD needs a KVM host`),绝不假装构建成功。
|
||||
本轨未在真机上跑过全量模式,镜像未建成、未发布,仍属
|
||||
“需 KVM 宿主实测”项。
|
||||
|
||||
## 6. Host key 首启注入硬性要求(补缺口轨新增)
|
||||
|
||||
执行器的 per-VM 预置方案(见 `cloud/chv/hostkeys.py`)能
|
||||
跑通的前提是客机镜像必须满足以下全部,否则预置在真机
|
||||
上跑不通,执行器会 fail-fast,不回落成弱方案:
|
||||
- 装 cloud-init 且支持 NoCloud seed(seed 磁盘/目录由
|
||||
执行器在 VM 启动前生成,含 meta-data 与 user-data);
|
||||
- 首启时先应用 seed 再启动 sshd,seed 里的 user-data
|
||||
把该 VM 的 host key 对写进 /etc/ssh/ 并重启 sshd;
|
||||
- 镜像里已钉死 `ssh_deletekeys: false` 与
|
||||
`ssh_genkeytypes: []`(build.sh 已写),首启不得无条件
|
||||
重新生成 host key——重生成会使预置的 known_hosts 立即
|
||||
失效,SSH 钉死直接拒绝,这是设计,不是 bug;
|
||||
- sshd 只允许 key 登录,禁止密码登录;
|
||||
- seed 磁盘/目录是 per-VM 敏感物,执行器在 VM 关闭/
|
||||
回收时删除,镜像构建方不得把任何一把 host key 固化
|
||||
进镜像(所有 VM 同 key 是安全事故)。
|
||||
|
||||
## 5. 与执行器的对接
|
||||
|
||||
|
||||
+110
-19
@@ -1,26 +1,52 @@
|
||||
#!/bin/bash
|
||||
# chv-guest 镜像构建脚本骨架(未在无 KVM 机器上实际执行过)。
|
||||
# chv-guest 镜像构建脚本(补缺口轨:从 TODO 骨架改成真实步骤)。
|
||||
#
|
||||
# 用法(需要带 /dev/kvm 的宿主,见 README):
|
||||
# 用法(必须在带 /dev/kvm 的宿主上跑全量模式):
|
||||
# CODEX_BIN_DIR=bin UBUNTU_VERSION=24.04 CODEX_VERSION=0.164.0-alpha.9 \
|
||||
# REPO_COMMIT=$(git rev-parse --short HEAD) ./build.sh out/
|
||||
# ./build.sh --dry-run out/ # 只打印命令,不执行、不要求 KVM
|
||||
#
|
||||
# 本脚本只做三件事,绝不假装已建成镜像:
|
||||
# 1. 参数校验(缺 CODEX_BIN_DIR 成套二进制、缺版本、缺 /dev/kvm
|
||||
# 即退出,退出码非 0);
|
||||
# 2. 算出版本 tag(chv-guest-<ubuntu>-<codex>-<commit>);
|
||||
# 3. 列出仍需在 KVM 宿主上执行的步骤(下载 Ubuntu 云镜像、
|
||||
# virt-customize 装清单软件、拷入 /app 代码、生成 vmlinux /
|
||||
# disk.raw / BUILDINFO.json),每一步在骨架里以 TODO 标记,
|
||||
# 不在本轨内静默跳过或谎称完成。
|
||||
# 产物(out/ 内):
|
||||
# vmlinux # virt-get-kernel 从云镜像取出的内核(CHV --kernel 指向它)
|
||||
# disk.raw # qemu-img 转成的 raw 根文件系统镜像(CHV_ROOTFS 指向它)
|
||||
# BUILDINFO.json # {tag, ubuntu, codex_version, repo_commit, built_at, contents}
|
||||
# 绝不进镜像的东西:任何模型 key、系统 key、用户账号数据、
|
||||
# 生产凭证;模型 key 由客机 cloud-init 预置的运行环境变量名
|
||||
# 注入,不写进镜像层。
|
||||
#
|
||||
# 客机 host key 硬性要求(见 README §6,补缺口轨新增):
|
||||
# 客机必须装 cloud-init 且支持 NoCloud seed,首启先应用
|
||||
# seed 再启动 sshd,且不得无条件重新生成 host key,否则
|
||||
# 执行器的 per-VM 预置钉死即失效。本脚本在镜像里写
|
||||
# /etc/cloud/cloud.cfg.d/99-chv-hostkey.cfg(ssh_deletekeys:
|
||||
# false、ssh_genkeytypes: []),把“不重生成”钉进镜像。
|
||||
set -eu
|
||||
|
||||
DRY_RUN=0
|
||||
if [ "${1:-}" = "--dry-run" ]; then
|
||||
DRY_RUN=1
|
||||
shift
|
||||
fi
|
||||
OUT="${1:-out}"
|
||||
: "${CODEX_BIN_DIR:?CODEX_BIN_DIR 未设:需 npm 安装 @openai/codex 后的配对二进制目录}"
|
||||
: "${UBUNTU_VERSION:?UBUNTU_VERSION 未设}"
|
||||
: "${CODEX_VERSION:?CODEX_VERSION 未设}"
|
||||
: "${REPO_COMMIT:?REPO_COMMIT 未设:构建时仓库 commit 短 hash}"
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
REPO_DIR="${REPO_DIR:-$(cd "$SCRIPT_DIR/.." && pwd)}"
|
||||
TAG="chv-guest-${UBUNTU_VERSION}-${CODEX_VERSION}-${REPO_COMMIT}"
|
||||
UBUNTU_IMAGE_URL="${UBUNTU_IMAGE_URL:-https://cloud-images.ubuntu.com/releases/${UBUNTU_VERSION}/release/ubuntu-${UBUNTU_VERSION}-server-cloudimg-amd64.img}"
|
||||
|
||||
run() {
|
||||
if [ "$DRY_RUN" = "1" ]; then
|
||||
echo "DRY-RUN: $*"
|
||||
else
|
||||
echo "+ $*"
|
||||
"$@"
|
||||
fi
|
||||
}
|
||||
|
||||
# 参数校验(干跑与全量都做,不依赖 KVM)。
|
||||
for bin in codex codex-code-mode-host; do
|
||||
if [ ! -x "$CODEX_BIN_DIR/$bin" ]; then
|
||||
echo "缺少配对二进制:$CODEX_BIN_DIR/$bin,构建失败" >&2
|
||||
@@ -31,17 +57,82 @@ if [ ! -d "$CODEX_BIN_DIR/codex-resources" ]; then
|
||||
echo "缺少 codex-resources(bwrap 载体),构建失败" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [ ! -e /dev/kvm ]; then
|
||||
echo "CHV_GUEST_BUILD needs a KVM host: /dev/kvm 不存在" >&2
|
||||
if [ ! -f "$REPO_DIR/host.py" ] || [ ! -d "$REPO_DIR/cloud" ]; then
|
||||
echo "REPO_DIR 不是 codex-app-server 仓库根(缺 host.py/cloud/):$REPO_DIR" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [ "$DRY_RUN" = "0" ]; then
|
||||
if [ ! -e /dev/kvm ]; then
|
||||
echo "CHV_GUEST_BUILD needs a KVM host: /dev/kvm 不存在" >&2
|
||||
exit 1
|
||||
fi
|
||||
for tool in curl qemu-img virt-customize virt-get-kernel python3; do
|
||||
if ! command -v "$tool" >/dev/null 2>&1; then
|
||||
echo "缺少构建工具:$tool,构建失败" >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
TAG="chv-guest-${UBUNTU_VERSION}-${CODEX_VERSION}-${REPO_COMMIT}"
|
||||
mkdir -p "$OUT"
|
||||
run mkdir -p "$OUT"
|
||||
IMG="$OUT/ubuntu-${UBUNTU_VERSION}-server-cloudimg-amd64.img"
|
||||
if [ "$DRY_RUN" = "1" ]; then
|
||||
echo "DRY-RUN: curl -fL $UBUNTU_IMAGE_URL -o $IMG"
|
||||
echo "DRY-RUN: [ -s $IMG ] || exit 1"
|
||||
else
|
||||
if [ ! -s "$IMG" ]; then
|
||||
curl -fL "$UBUNTU_IMAGE_URL" -o "$IMG"
|
||||
fi
|
||||
[ -s "$IMG" ] || { echo "下载的云镜像为空:$IMG" >&2; exit 1; }
|
||||
if [ -n "${UBUNTU_IMAGE_SHA256:-}" ]; then
|
||||
echo "$UBUNTU_IMAGE_SHA256 $IMG" | sha256sum -c -
|
||||
fi
|
||||
fi
|
||||
run qemu-img convert -f qcow2 -O raw "$IMG" "$OUT/disk.raw"
|
||||
# 清单软件(照抄 cloud/docker/Dockerfile 的 apt/pip 清单,
|
||||
# 中文 LaTeX xelatex+CJK 故意不装,见 README;Playwright
|
||||
# 默认不装,PLAYWRIGHT=1 时才加装,属可选步骤)。
|
||||
run virt-customize -a "$OUT/disk.raw" \
|
||||
--install git,pandoc,poppler-utils,latexmk,texlive-latex-base,openssh-server,cloud-init,qemu-guest-agent,python3,python3-pip \
|
||||
--run-command 'pip3 install --no-cache-dir --break-system-packages python-docx openpyxl python-pptx PyMuPDF Pillow || pip3 install --no-cache-dir python-docx openpyxl python-pptx PyMuPDF Pillow' \
|
||||
--copy-in "$CODEX_BIN_DIR/codex:/usr/local/bin/" \
|
||||
--copy-in "$CODEX_BIN_DIR/codex-code-mode-host:/usr/local/bin/" \
|
||||
--copy-in "$CODEX_BIN_DIR/codex-resources:/usr/local/bin/" \
|
||||
--run-command 'chmod +x /usr/local/bin/codex /usr/local/bin/codex-code-mode-host; ln -sf /usr/local/bin/codex-resources/bwrap /usr/local/bin/bwrap' \
|
||||
--mkdir /app \
|
||||
--copy-in "$REPO_DIR/host.py:/app/" \
|
||||
--copy-in "$REPO_DIR/cloud:/app/" \
|
||||
--run-command 'printf "ssh_deletekeys: false\nssh_genkeytypes: []\n" > /etc/cloud/cloud.cfg.d/99-chv-hostkey.cfg; systemctl enable ssh || true'
|
||||
if [ "${PLAYWRIGHT:-0}" = "1" ]; then
|
||||
run virt-customize -a "$OUT/disk.raw" \
|
||||
--run-command 'pip3 install --no-cache-dir playwright; python3 -m playwright install --with-deps chromium'
|
||||
fi
|
||||
run virt-get-kernel -a "$OUT/disk.raw" -o "$OUT/vmlinux"
|
||||
# 产物校验(干跑只打印,不执行校验命令的副作用部分)。
|
||||
if [ "$DRY_RUN" = "1" ]; then
|
||||
echo "DRY-RUN: test -s $OUT/vmlinux && test -s $OUT/disk.raw"
|
||||
echo "DRY-RUN: python3 生成 $OUT/BUILDINFO.json 并校验 tag=$TAG"
|
||||
else
|
||||
test -s "$OUT/vmlinux" || { echo "内核产物为空:$OUT/vmlinux" >&2; exit 1; }
|
||||
test -s "$OUT/disk.raw" || { echo "磁盘产物为空:$OUT/disk.raw" >&2; exit 1; }
|
||||
python3 - "$OUT/BUILDINFO.json" "$TAG" "$UBUNTU_VERSION" "$CODEX_VERSION" "$REPO_COMMIT" <<'PYEOF'
|
||||
import json, sys, time
|
||||
path, tag, ubuntu, codex, commit = sys.argv[1:6]
|
||||
info = {"tag": tag, "ubuntu": ubuntu, "codex_version": codex,
|
||||
"repo_commit": commit, "built_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
|
||||
"contents": ["git", "pandoc", "poppler-utils", "latexmk", "texlive-latex-base",
|
||||
"openssh-server", "cloud-init", "qemu-guest-agent",
|
||||
"python-docx", "openpyxl", "python-pptx", "PyMuPDF", "Pillow",
|
||||
"codex", "codex-code-mode-host", "codex-resources/bwrap", "/app/host.py"]}
|
||||
with open(path, "w") as fh:
|
||||
json.dump(info, fh, indent=2)
|
||||
PYEOF
|
||||
python3 -c "import json; d=json.load(open('$OUT/BUILDINFO.json')); assert d['tag'] == '$TAG', d"
|
||||
fi
|
||||
echo "tag=$TAG"
|
||||
echo "out=$OUT"
|
||||
echo "TODO(KVM host): 下载 Ubuntu ${UBUNTU_VERSION} 云镜像并转 raw"
|
||||
echo "TODO(KVM host): virt-customize 装 README §2 清单(git/python/pandoc/poppler/latex/codex/playwright/openssh)"
|
||||
echo "TODO(KVM host): 拷入仓库代码到 /app(版本 $REPO_COMMIT),绝不拷入任何 key"
|
||||
echo "TODO(KVM host): 生成 vmlinux + disk.raw + BUILDINFO.json,校验 tag 与清单版本"
|
||||
echo "构建骨架结束:以上 TODO 步骤未执行,镜像未建成(见 README 与生产就绪清单)"
|
||||
if [ "$DRY_RUN" = "1" ]; then
|
||||
echo "干跑结束:未执行任何构建步骤,镜像未建成"
|
||||
else
|
||||
echo "构建完成:$OUT/vmlinux $OUT/disk.raw $OUT/BUILDINFO.json"
|
||||
fi
|
||||
@@ -106,6 +106,12 @@ def build_chv_config(params, env):
|
||||
disk_gb = float(disk_gb_raw) if disk_gb_raw else None
|
||||
egress_proxy = env.get("CHV_EGRESS_PROXY")
|
||||
allow_direct_egress = env.get("CHV_ALLOW_DIRECT_EGRESS") == "1"
|
||||
host_key_mode = params.get("host_key_mode") \
|
||||
or env.get("CHV_HOST_KEY_MODE") or "keyscan"
|
||||
if host_key_mode not in ("provisioned", "keyscan"):
|
||||
raise ValueError(
|
||||
"CHV_HOST_KEY_MODE 只能是 provisioned 或 keyscan:%s"
|
||||
% host_key_mode)
|
||||
task_timeout_raw = env.get("CHV_TASK_TIMEOUT")
|
||||
task_timeout = float(task_timeout_raw) \
|
||||
if task_timeout_raw else None
|
||||
@@ -141,6 +147,7 @@ def build_chv_config(params, env):
|
||||
max_vms=max_vms, disk_mode=disk_mode, disk_gb=disk_gb,
|
||||
egress_proxy=egress_proxy,
|
||||
allow_direct_egress=allow_direct_egress,
|
||||
host_key_mode=host_key_mode,
|
||||
task_timeout=task_timeout, snapshot_url=snapshot_url,
|
||||
warm_pool_size=warm_pool_size, api_timeout=api_timeout,
|
||||
ready_timeout=ready_timeout, ready_interval=ready_interval)
|
||||
+99
-17
@@ -164,10 +164,10 @@ class CloudHypervisorExecutor(Executor):
|
||||
preflight 直接失败。完整清单见
|
||||
docs/chv-production-readiness.md。
|
||||
- worker 层现状:通用路径已能走 dispatch_task/goal_task;
|
||||
断点续跑(goal-resume-breakpoint)在 worker 层对非
|
||||
Local 执行器仍明确失败(见 worker.py),本类的
|
||||
goal_resume_breakpoint_task 已实现,供直接调用与后续
|
||||
worker 接线,不通过 worker 迂回假装已通。
|
||||
断点续跑(goal-resume-breakpoint)已由补缺口轨接线:
|
||||
worker 通用分支对任何提供该方法的执行器调用它,
|
||||
Local 仍走原路径;网关任务仍被 worker 侧 loud fail
|
||||
守卫拦住,不放宽网关规则。
|
||||
- 网关现状:worker 通用路径在 sub2api 网关下对非 Local
|
||||
执行器直接 loud fail(与 CloudExecutor 同款),v1 不
|
||||
改这条门,所以 CHV + sub2api 组合目前在 worker 层就
|
||||
@@ -231,7 +231,7 @@ class CloudHypervisorExecutor(Executor):
|
||||
cpuinfo_path="/proc/cpuinfo", env=None,
|
||||
model=None, ch_remote=None, api_client=None,
|
||||
sleeper=None, api_timeout=None, ready_timeout=None,
|
||||
ready_interval=None):
|
||||
ready_interval=None, host_key_mode=None):
|
||||
self._env = env if env is not None else os.environ
|
||||
# 配置解析已拆到 cloud/chv/config.py(参数优先于环境
|
||||
# 变量,默认值与 #216 一致;显式空用户/越界端口的收紧
|
||||
@@ -247,7 +247,8 @@ class CloudHypervisorExecutor(Executor):
|
||||
"cpuinfo_path": cpuinfo_path, "ch_remote": ch_remote,
|
||||
"model": model, "api_timeout": api_timeout,
|
||||
"ready_timeout": ready_timeout,
|
||||
"ready_interval": ready_interval}, self._env)
|
||||
"ready_interval": ready_interval,
|
||||
"host_key_mode": host_key_mode}, self._env)
|
||||
for _field in (
|
||||
"binary", "kernel", "rootfs", "cpus", "memory",
|
||||
"state_dir", "ssh_key", "ssh_user", "ssh_port",
|
||||
@@ -256,7 +257,7 @@ class CloudHypervisorExecutor(Executor):
|
||||
"max_vms", "disk_mode", "disk_gb", "egress_proxy",
|
||||
"allow_direct_egress", "task_timeout", "snapshot_url",
|
||||
"warm_pool_size", "api_timeout", "ready_timeout",
|
||||
"ready_interval"):
|
||||
"ready_interval", "host_key_mode"):
|
||||
setattr(self, _field, getattr(_cfg, _field))
|
||||
self.api_client = api_client if api_client is not None \
|
||||
else _unix_socket_api
|
||||
@@ -391,6 +392,22 @@ class CloudHypervisorExecutor(Executor):
|
||||
missing.append(("egress_not_configured",
|
||||
"CHV_EGRESS_PROXY 未设且未显式允许直连:"
|
||||
"VM 出口必须过白名单代理,默认不给直连外网"))
|
||||
# 补缺口轨 (4):tap/网络配置校验前置到 preflight。
|
||||
# CHV_NET_JSON 非法 JSON 或非 list/dict 时直接报
|
||||
# net_config_invalid,不等到 vm.create 才炸;直连
|
||||
# 只在显式 CHV_ALLOW_DIRECT_EGRESS=1 时放行,且
|
||||
# wake_vm 里发 egress_direct_allowed 审计。tap 桥接
|
||||
# 本身是否真只走代理仍需宿主实测,代码不声称已验证。
|
||||
if self.net_json:
|
||||
try:
|
||||
_net = json.loads(self.net_json)
|
||||
except ValueError:
|
||||
missing.append(("net_config_invalid",
|
||||
"CHV_NET_JSON 不是合法 JSON"))
|
||||
else:
|
||||
if not isinstance(_net, (list, dict)):
|
||||
missing.append(("net_config_invalid",
|
||||
"CHV_NET_JSON 须为 JSON list 或 dict"))
|
||||
if missing:
|
||||
kind = missing[0][0]
|
||||
detail = "; ".join(text for _, text in missing)
|
||||
@@ -446,12 +463,45 @@ class CloudHypervisorExecutor(Executor):
|
||||
return _known_hosts_path(self.state_dir,
|
||||
self.vm_name(task_or_name))
|
||||
|
||||
def provision_host_key(self, task):
|
||||
"""按 VM 预置 host key(补缺口轨强方案,见 hostkeys.py):
|
||||
启动前为该 VM 生成一次性 ed25519 host key 对,公钥
|
||||
立即写进该 VM 的 known_hosts(钉死),私钥只写进
|
||||
per-VM seed 目录的 cloud-init user-data,随 VM 销毁
|
||||
而删。返回 {paths, seed_dir, known_hosts}。keygen
|
||||
失败即 raise,provisioned 模式下绝不回落 keyscan;
|
||||
不同 VM 的 key 对不同(state_dir/<vm> 隔离),同一
|
||||
VM 重启复用已有对(否则钉死立即失效)。"""
|
||||
from cloud.chv.hostkeys import (
|
||||
generate_host_keypair, host_key_paths, known_hosts_line,
|
||||
write_seed)
|
||||
from cloud.chv.ssh_transport import write_known_hosts
|
||||
vm = self.vm_name(task)
|
||||
paths = generate_host_keypair(
|
||||
self.state_dir, vm,
|
||||
runner=lambda cmd: self._call_runner(cmd))
|
||||
with open(paths["public"], "r") as fh:
|
||||
public_text = fh.read()
|
||||
with open(paths["private"], "r") as fh:
|
||||
private_text = fh.read()
|
||||
write_known_hosts(
|
||||
paths["known_hosts"],
|
||||
[known_hosts_line(self.ssh_port, public_text)])
|
||||
seed_dir = write_seed(paths["seed_dir"], vm,
|
||||
private_text, public_text)
|
||||
return {"paths": paths, "seed_dir": seed_dir,
|
||||
"known_hosts": paths["known_hosts"]}
|
||||
|
||||
def capture_host_key(self, task):
|
||||
"""首次捕获客机 host key(TOFU,边界见 ssh_transport.py):
|
||||
ssh-keyscan 经本机回环端口抓 key 行,写入该 VM 的
|
||||
known_hosts 文件(0600)。抓不到可用 key 行时返回
|
||||
False(不写文件、不报错),由 wait_guest_ready 下一轮
|
||||
再试;sshd 还没起来时这就是常态,不是失败。"""
|
||||
"""首次捕获客机 host key(TOFU,弱回落,边界见
|
||||
ssh_transport.py 与 hostkeys.py):ssh-keyscan 经本机
|
||||
回环端口抓 key 行,写入该 VM 的 known_hosts 文件
|
||||
(0600)。生产部署必须用 provisioned 预置模式,
|
||||
本方法只在 CHV_HOST_KEY_MODE=keyscan 时作回落,
|
||||
它只能防首次之后的替换,不能验证首次本身。
|
||||
抓不到可用 key 行时返回 False(不写文件、不报错),
|
||||
由 wait_guest_ready 下一轮再试;sshd 还没起来时
|
||||
这就是常态,不是失败。"""
|
||||
try:
|
||||
result = self._call_runner(
|
||||
["ssh-keyscan", "-p", str(self.ssh_port),
|
||||
@@ -699,7 +749,18 @@ class CloudHypervisorExecutor(Executor):
|
||||
self._kill_proc(old)
|
||||
task_disk = self._provision_task_disk(task)
|
||||
socket_path = self.api_socket(task)
|
||||
if self.allow_direct_egress:
|
||||
# (4) 直连 egress 只在显式 CHV_ALLOW_DIRECT_EGRESS=1
|
||||
# 时放行,且必须审计;默认 fail-closed 在 preflight
|
||||
# 里已拦(无代理且未显式允许即缺项)。tap 桥接
|
||||
# 本身是否真只走代理仍需宿主实测,代码不声称已验证。
|
||||
self._emit("egress_direct_allowed", task,
|
||||
"vm=%s CHV_ALLOW_DIRECT_EGRESS=1 explicit" % name)
|
||||
try:
|
||||
if getattr(self, "host_key_mode", "keyscan") == "provisioned":
|
||||
# 强方案:启动前预置每 VM host key 并钉死,
|
||||
# 失败即 fail-fast,绝不回落 keyscan。
|
||||
self.provision_host_key(task)
|
||||
self._launch_api_only(task)
|
||||
self.wait_api_ready(task)
|
||||
status, _ = self._api_call(
|
||||
@@ -865,6 +926,22 @@ class CloudHypervisorExecutor(Executor):
|
||||
os.remove(socket_path)
|
||||
except Exception:
|
||||
pass
|
||||
# 补缺口轨:per-VM seed 目录与 host 私钥随 VM
|
||||
# 关闭而删,绝不留给下一个 VM 复用;known_hosts
|
||||
# 公钥文件保留供事后审计,删失败不炸关闭。下次
|
||||
# 同名 VM 预置时若私钥已删即重新生成并覆盖
|
||||
# known_hosts,不会用旧钉死去校验新 VM。
|
||||
try:
|
||||
seed_dir = os.path.join(self.state_dir, name, "seed")
|
||||
if os.path.isdir(seed_dir):
|
||||
shutil.rmtree(seed_dir)
|
||||
for _suffix in ("ssh_host_ed25519_key",
|
||||
"ssh_host_ed25519_key.pub"):
|
||||
_p = os.path.join(self.state_dir, name, _suffix)
|
||||
if os.path.exists(_p):
|
||||
os.remove(_p)
|
||||
except Exception:
|
||||
pass
|
||||
task = task_or_name if isinstance(task_or_name, dict) \
|
||||
else {"id": task_or_name}
|
||||
if had:
|
||||
@@ -1195,17 +1272,22 @@ class CloudHypervisorExecutor(Executor):
|
||||
task, args, goal=True, model=task.get("model"),
|
||||
effort=task.get("effort"), approval=task.get("approval"))
|
||||
|
||||
def goal_resume_breakpoint_task(self, task) -> dict:
|
||||
def goal_resume_breakpoint_task(self, task,
|
||||
model_provider=None) -> dict:
|
||||
"""客内 host.py goal-resume 断点续跑:无 --goal-text、
|
||||
无 --max-rounds,host 沿用原任务从断点轮次续跑。
|
||||
注意 worker 层对非 Local 执行器的断点续跑目前仍明确
|
||||
失败,本方法供直接调用与后续 worker 接线(见类
|
||||
docstring),不要理解成 worker 路径已通。"""
|
||||
worker 层已接线(补缺口轨):非网关的 CHV 断点任务
|
||||
经 worker 的通用断点分支调到这里,model_provider
|
||||
按名透传给客内 host.py(与 Local/Docker 同款口径,
|
||||
首轮 thread 的 provider 必须一致);网关任务在 worker
|
||||
侧仍被 loud fail 守卫拦住,不到这里。模型 key
|
||||
本身绝不经本方法传值,只按环境变量名注入。"""
|
||||
if not isinstance(task, dict) or task.get("id") is None:
|
||||
raise ValueError("任务缺少 id")
|
||||
return self._run_with_vm(
|
||||
task, ("goal-resume", "--task", self._host_name(task)),
|
||||
goal=True, model=task.get("model"),
|
||||
effort=task.get("effort"), approval=task.get("approval"))
|
||||
effort=task.get("effort"), approval=task.get("approval"),
|
||||
model_provider=model_provider)
|
||||
|
||||
|
||||
@@ -0,0 +1,147 @@
|
||||
"""按 VM 生成的 SSH host key 预置(替代 TOFU 的强方案)。
|
||||
|
||||
安全模型(补缺口轨,替代 #221 的 ssh-keyscan 首次捕获):
|
||||
- 绝不用镜像固定同一把 host key:所有 VM 共用一把 key 时,
|
||||
一台 VM 泄漏即全网可冒充,属安全事故。
|
||||
- 每 VM 一次性生成:宿主在 VMM 启动前为该 VM 生成一对
|
||||
ed25519 host key(私钥只活在该 VM 的 seed 里,随 VM
|
||||
销毁而删,绝不跨 VM 复用、绝不写进镜像、绝不进已知
|
||||
主机文件之外的共享位置)。
|
||||
- 公钥预置钉死:生成后立即把公钥写进该 VM 的
|
||||
state_dir/<vm>/known_hosts(格式 [127.0.0.1]:<port> ...),
|
||||
执行器在 VM 启动前就已钉死,启动后 SSH 直接用该文件
|
||||
校验,换 key 即拒绝——不再靠启动后 ssh-keyscan 抓到的
|
||||
key 来“事后钉死”。
|
||||
- 私钥注入通道(选 cloud-init NoCloud,写清唯一通道):
|
||||
seed 目录 state_dir/<vm>/seed/ 含 meta-data(instance-id
|
||||
即 VM 名)、user-data(write_files 写 /etc/ssh/ 的 host
|
||||
key 对并重启 sshd)。客机镜像硬性要求(见
|
||||
chv-guest/README.md §6):必须装 cloud-init 且支持
|
||||
NoCloud seed、首启时先应用 seed 再启动 sshd、且不得在
|
||||
seed 应用前/后无条件重新生成 host key(否则预置即失效)。
|
||||
seed 磁盘/目录是 per-VM 敏感物,VM 关闭/回收时必须删。
|
||||
- 回落边界:ssh-keyscan 首次捕获只作弱回落
|
||||
(CHV_HOST_KEY_MODE=keyscan,且文档标为弱方案):
|
||||
它只能防首次之后的替换,不能验证首次本身。生产部署
|
||||
必须用 provisioned 模式;provisioned 模式下 seed 工具
|
||||
或 keygen 不可用时直接 fail-fast,绝不静默回落 keyscan。
|
||||
|
||||
本模块只做纯函数与文件操作,不跑 VM、不联网、不含
|
||||
任何私钥值(测试用假 runner 时私钥由测试自己写文件)。
|
||||
"""
|
||||
|
||||
import os
|
||||
|
||||
|
||||
def host_key_paths(state_dir, vm_name):
|
||||
"""该 VM 的 host key 文件路径(私钥/公钥/known_hosts)。"""
|
||||
base = os.path.join(state_dir, vm_name)
|
||||
return {
|
||||
"private": os.path.join(base, "ssh_host_ed25519_key"),
|
||||
"public": os.path.join(base, "ssh_host_ed25519_key.pub"),
|
||||
"known_hosts": os.path.join(base, "known_hosts"),
|
||||
"seed_dir": os.path.join(base, "seed"),
|
||||
}
|
||||
|
||||
|
||||
def known_hosts_line(port, public_key_text):
|
||||
"""公钥文件内容转 known_hosts 行:只取前两列(类型+数据)。
|
||||
|
||||
公钥文件形如 'ssh-ed25519 AAAA... [comment]',known_hosts
|
||||
行形如 '[127.0.0.1]:<port> ssh-ed25519 AAAA...'。非法
|
||||
内容 ValueError,绝不把陌生行原样写进文件。"""
|
||||
parts = (public_key_text or "").strip().split()
|
||||
if len(parts) < 2 or not parts[0].startswith("ssh-"):
|
||||
raise ValueError("host 公钥格式非法")
|
||||
return "[127.0.0.1]:%s %s %s" % (port, parts[0], parts[1])
|
||||
|
||||
|
||||
def build_seed_user_data(private_key_text, public_key_text):
|
||||
"""cloud-init user-data:把 host key 对写进客机 /etc/ssh。
|
||||
|
||||
注意:user-data 本身含私钥,只写进 per-VM seed 目录
|
||||
(0600),随 VM 销毁而删,绝不进镜像、绝不进日志、
|
||||
绝不进任务 prompt 落库。"""
|
||||
if not private_key_text or not public_key_text:
|
||||
raise ValueError("host key 对缺失,无法生成 seed")
|
||||
lines = [
|
||||
"#cloud-config",
|
||||
"ssh_deletekeys: false",
|
||||
"write_files:",
|
||||
" - path: /etc/ssh/ssh_host_ed25519_key",
|
||||
" permissions: '0600'",
|
||||
" owner: root:root",
|
||||
" content: |",
|
||||
]
|
||||
for line in private_key_text.rstrip("\n").split("\n"):
|
||||
lines.append(" " + line)
|
||||
lines += [
|
||||
" - path: /etc/ssh/ssh_host_ed25519_key.pub",
|
||||
" permissions: '0644'",
|
||||
" owner: root:root",
|
||||
" content: |",
|
||||
]
|
||||
for line in public_key_text.rstrip("\n").split("\n"):
|
||||
lines.append(" " + line)
|
||||
lines += [
|
||||
"runcmd:",
|
||||
" - chmod 600 /etc/ssh/ssh_host_ed25519_key",
|
||||
" - systemctl restart ssh || service ssh restart || true",
|
||||
]
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def write_seed(seed_dir, vm_name, private_key_text, public_key_text):
|
||||
"""写 seed 目录:meta-data + user-data(均为 0600)。
|
||||
|
||||
返回 seed 目录路径。seed 目录本身不含已知主机之外的
|
||||
共享数据,调用方(执行器关闭/回收)负责删除它。"""
|
||||
os.makedirs(seed_dir, exist_ok=True)
|
||||
meta_path = os.path.join(seed_dir, "meta-data")
|
||||
user_path = os.path.join(seed_dir, "user-data")
|
||||
with open(meta_path, "w") as fh:
|
||||
fh.write("instance-id: %s\nlocal-hostname: %s\n"
|
||||
% (vm_name, vm_name))
|
||||
os.chmod(meta_path, 0o600)
|
||||
with open(user_path, "w") as fh:
|
||||
fh.write(build_seed_user_data(private_key_text, public_key_text))
|
||||
os.chmod(user_path, 0o600)
|
||||
return seed_dir
|
||||
|
||||
|
||||
def generate_host_keypair(state_dir, vm_name, runner=None,
|
||||
keygen="ssh-keygen"):
|
||||
"""为该 VM 生成(或复用本 VM 已有的)host key 对。
|
||||
|
||||
同一 VM 重启/恢复时复用已有对(否则钉死的 known_hosts
|
||||
立即失效);不同 VM 的 state_dir/<vm> 不同,绝不共用。
|
||||
runner 不给时用 subprocess,测试可注入假 runner(假
|
||||
runner 负责把私钥/公钥文件写出来)。keygen 失败即
|
||||
RuntimeError,调用方不得回落 keyscan(provisioned
|
||||
模式)或必须显式标弱回落(keyscan 模式,见执行器)。"""
|
||||
paths = host_key_paths(state_dir, vm_name)
|
||||
if (os.path.exists(paths["private"])
|
||||
and os.path.exists(paths["public"])):
|
||||
return paths
|
||||
base = os.path.dirname(paths["private"])
|
||||
os.makedirs(base, exist_ok=True)
|
||||
cmd = [keygen, "-t", "ed25519", "-N", "",
|
||||
"-f", paths["private"]]
|
||||
if runner is not None:
|
||||
result = runner(cmd)
|
||||
rc = getattr(result, "returncode", 0)
|
||||
if rc not in (0, None):
|
||||
raise RuntimeError("ssh-keygen 失败:returncode=%s" % rc)
|
||||
else:
|
||||
import subprocess
|
||||
result = subprocess.run(cmd, capture_output=True, text=True)
|
||||
if result.returncode != 0:
|
||||
raise RuntimeError("ssh-keygen 失败:%s" % result.stderr)
|
||||
if not (os.path.exists(paths["private"])
|
||||
and os.path.exists(paths["public"])):
|
||||
raise RuntimeError("ssh-keygen 未生成 host key 对")
|
||||
try:
|
||||
os.chmod(paths["private"], 0o600)
|
||||
except OSError:
|
||||
pass
|
||||
return paths
|
||||
@@ -0,0 +1,417 @@
|
||||
"""CHV 补缺口轨测试(堆叠于 #221,base=feat-chv-optimize)。
|
||||
|
||||
只用标准库与 mock runner/launcher/api_client,绝不启动真实
|
||||
VM:当前环境没有可跑真 VM 的宿主(生产主机是 CVM、无
|
||||
/dev/kvm、无嵌套虚拟化;本 VM 容器内同样无 /dev/kvm)。
|
||||
真实 VM 启动、快照恢复、压测一律不在本文件声称已验证。
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
|
||||
try:
|
||||
from cloud.chv.hostkeys import (
|
||||
build_seed_user_data,
|
||||
generate_host_keypair,
|
||||
host_key_paths,
|
||||
known_hosts_line,
|
||||
write_seed,
|
||||
)
|
||||
from cloud.chv.pool import WarmPool
|
||||
from cloud.sandbox import (
|
||||
ChvFailure,
|
||||
CloudHypervisorExecutor,
|
||||
_parse_memory_bytes,
|
||||
)
|
||||
from cloud.worker import Worker, _call_goal_resume_breakpoint
|
||||
except ImportError:
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
from cloud.chv.hostkeys import (
|
||||
build_seed_user_data,
|
||||
generate_host_keypair,
|
||||
host_key_paths,
|
||||
known_hosts_line,
|
||||
write_seed,
|
||||
)
|
||||
from cloud.chv.pool import WarmPool
|
||||
from cloud.sandbox import (
|
||||
ChvFailure,
|
||||
CloudHypervisorExecutor,
|
||||
_parse_memory_bytes,
|
||||
)
|
||||
from cloud.worker import Worker, _call_goal_resume_breakpoint
|
||||
|
||||
|
||||
class FakeResult:
|
||||
def __init__(self, returncode=0, stdout="", stderr=""):
|
||||
self.returncode = returncode
|
||||
self.stdout = stdout
|
||||
self.stderr = stderr
|
||||
|
||||
|
||||
class FakeProc:
|
||||
def __init__(self, alive=True):
|
||||
self._alive = alive
|
||||
|
||||
def poll(self):
|
||||
return None if self._alive else 1
|
||||
|
||||
def terminate(self):
|
||||
self._alive = False
|
||||
|
||||
def wait(self, timeout=None):
|
||||
return 0
|
||||
|
||||
def kill(self):
|
||||
self._alive = False
|
||||
|
||||
|
||||
def _write(path, content=b"x"):
|
||||
with open(path, "wb") as fh:
|
||||
fh.write(content)
|
||||
os.chmod(path, 0o755)
|
||||
return path
|
||||
|
||||
|
||||
class ChvGapCase(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.dir = self.tmp.name
|
||||
self.binary = _write(os.path.join(self.dir, "cloud-hypervisor"))
|
||||
self.ch_remote = _write(os.path.join(self.dir, "ch-remote"))
|
||||
self.kernel = _write(os.path.join(self.dir, "vmlinux"))
|
||||
self.rootfs = _write(os.path.join(self.dir, "disk.raw"),
|
||||
b"R" * 100)
|
||||
self.ssh_key = _write(os.path.join(self.dir, "id_chv"))
|
||||
self.kvm = _write(os.path.join(self.dir, "kvm"))
|
||||
self.cpuinfo = os.path.join(self.dir, "cpuinfo")
|
||||
with open(self.cpuinfo, "w") as fh:
|
||||
fh.write("flags : fpu vmx sse\n")
|
||||
self.state_dir = os.path.join(self.dir, "state")
|
||||
self.calls = []
|
||||
self.audit = []
|
||||
|
||||
def tearDown(self):
|
||||
self.tmp.cleanup()
|
||||
|
||||
def _default_runner(self, cmd, timeout=None):
|
||||
self.calls.append(list(cmd))
|
||||
if cmd and cmd[0] == "ssh":
|
||||
if cmd[-1] == "true":
|
||||
return FakeResult(0, "", "")
|
||||
if "goal-resume" in cmd:
|
||||
return FakeResult(0, "goal met after rounds\n", "")
|
||||
if "goal" in cmd:
|
||||
return FakeResult(0, "goal met after rounds\n", "")
|
||||
return FakeResult(0, "", "")
|
||||
return FakeResult(0, "", "")
|
||||
|
||||
def _default_api(self, socket_path, method, path, payload=None):
|
||||
return (200, {})
|
||||
|
||||
def _default_launcher(self, cmd):
|
||||
return FakeProc(alive=True)
|
||||
|
||||
def make_exec(self, runner=None, env_extra=None, **overrides):
|
||||
env = {"CHV_EGRESS_PROXY": "http://10.0.0.1:8888"}
|
||||
if env_extra:
|
||||
env.update(env_extra)
|
||||
kwargs = dict(
|
||||
binary=self.binary, ch_remote=self.ch_remote,
|
||||
kernel=self.kernel, rootfs=self.rootfs,
|
||||
ssh_key=self.ssh_key, kvm_path=self.kvm,
|
||||
cpuinfo_path=self.cpuinfo, state_dir=self.state_dir,
|
||||
runner=runner or self._default_runner,
|
||||
launcher=self._default_launcher,
|
||||
api_client=self._default_api,
|
||||
sleeper=lambda s: None, env=env)
|
||||
kwargs.update(overrides)
|
||||
ex = CloudHypervisorExecutor(**kwargs)
|
||||
ex.audit_callback = lambda e, t, d, k=None: self.audit.append(
|
||||
(e, t.get("id") if isinstance(t, dict) else t, d, k))
|
||||
return ex
|
||||
|
||||
|
||||
# -- (1) worker 断点接线 ---------------------------------------------
|
||||
class BreakpointWiringTest(ChvGapCase):
|
||||
def test_call_helper_passes_model_provider_when_accepted(self):
|
||||
seen = {}
|
||||
|
||||
class FakeChv:
|
||||
def goal_resume_breakpoint_task(self, task, model_provider=None):
|
||||
seen["provider"] = model_provider
|
||||
return {"status": "completed", "task_id": task["id"]}
|
||||
|
||||
out = _call_goal_resume_breakpoint(
|
||||
FakeChv(), {"id": "bp1"}, model_provider="prov-x")
|
||||
self.assertEqual(out["status"], "completed")
|
||||
self.assertEqual(seen["provider"], "prov-x")
|
||||
|
||||
def test_call_helper_single_arg_executor_still_works(self):
|
||||
class FakeOld:
|
||||
def goal_resume_breakpoint_task(self, task):
|
||||
return {"status": "completed", "task_id": task["id"]}
|
||||
|
||||
out = _call_goal_resume_breakpoint(
|
||||
FakeOld(), {"id": "bp2"}, model_provider="prov-x")
|
||||
self.assertEqual(out["status"], "completed")
|
||||
|
||||
def test_chv_breakpoint_accepts_model_provider(self):
|
||||
ex = self.make_exec()
|
||||
out = ex.goal_resume_breakpoint_task(
|
||||
{"id": "g1", "prompt": "续跑"}, model_provider="prov-x")
|
||||
self.assertEqual(out["status"], "completed")
|
||||
self.assertTrue(out["goal_met"])
|
||||
# 客内命令带 --model-provider(网关 key 本身不进命令值)。
|
||||
ssh_calls = [c for c in self.calls if c and c[0] == "ssh"]
|
||||
self.assertTrue(any("--model-provider" in c for c in ssh_calls))
|
||||
|
||||
def test_worker_branch_uses_hasattr_not_only_local(self):
|
||||
# worker.py 通用分支的文字证据:Local 优先、其次 hasattr。
|
||||
worker_path = Path(__file__).resolve().parents[1] / "worker.py"
|
||||
text = worker_path.read_text()
|
||||
self.assertIn("_call_goal_resume_breakpoint", text)
|
||||
self.assertIn("hasattr(", text)
|
||||
self.assertIn("goal_resume_breakpoint_task", text)
|
||||
|
||||
|
||||
# -- (2) reconcile_orphans 接线 ---------------------------------------
|
||||
class ReconcileWiringTest(ChvGapCase):
|
||||
def test_worker_startup_and_close_call_reconcile(self):
|
||||
from cloud.server import init_db
|
||||
db_path = os.path.join(self.dir, "control.db")
|
||||
init_db(db_path)
|
||||
ex = self.make_exec()
|
||||
# 手工造一个 orphan:先建池写 registry,再新开池载入。
|
||||
pool1 = WarmPool(ex)
|
||||
pool1._entries["chv-old"] = {
|
||||
"state": "warm", "vm": "chv-old",
|
||||
"task": {"id": "old"}, "snapshot_url": None}
|
||||
pool1._save_registry()
|
||||
self.assertTrue(
|
||||
os.path.exists(os.path.join(self.state_dir, "warm-pool.json")))
|
||||
# Worker 构造即 startup reconcile:用假 shutdown 记录。
|
||||
shutdowns = []
|
||||
orig_shutdown = ex.shutdown_vm
|
||||
ex.shutdown_vm = lambda name: shutdowns.append(name) or True
|
||||
worker = Worker(db_path, "/fake/host.py", executor=ex,
|
||||
env={"XCM_API_KEY": "test-placeholder"})
|
||||
self.assertIn("chv-old", shutdowns)
|
||||
self.assertEqual(worker._warm_pool.orphan_count(), 0)
|
||||
worker.close()
|
||||
# close 时再次 reconcile 不炸(池已空,返回 [] 即可)。
|
||||
self.assertEqual(worker._reconcile_warm_pool("shutdown"), [])
|
||||
|
||||
def test_reconcile_failure_does_not_break_close(self):
|
||||
from cloud.server import init_db
|
||||
db_path = os.path.join(self.dir, "control.db")
|
||||
init_db(db_path)
|
||||
ex = self.make_exec()
|
||||
worker = Worker(db_path, "/fake/host.py", executor=ex,
|
||||
env={"XCM_API_KEY": "test-placeholder"})
|
||||
# 池的 reconcile 抛错时 close 仍不炸。
|
||||
worker._warm_pool.reconcile_orphans = lambda: (_ for _ in ()).throw(
|
||||
RuntimeError("boom"))
|
||||
worker.close() # 不抛即通过
|
||||
|
||||
|
||||
# -- (3) SSH host key 预置 ---------------------------------------------
|
||||
class HostKeyProvisionTest(ChvGapCase):
|
||||
def _fake_keygen_runner(self, cmd, timeout=None):
|
||||
self.calls.append(list(cmd))
|
||||
if cmd and cmd[0] == "ssh-keygen":
|
||||
priv = cmd[cmd.index("-f") + 1]
|
||||
with open(priv, "w") as fh:
|
||||
fh.write("-----BEGIN OPENSSH PRIVATE KEY-----\nFAKE\n")
|
||||
os.chmod(priv, 0o600)
|
||||
with open(priv + ".pub", "w") as fh:
|
||||
fh.write("ssh-ed25519 AAAAfake1 vm-key\n")
|
||||
return FakeResult(0, "", "")
|
||||
return self._default_runner(cmd, timeout)
|
||||
|
||||
def test_per_vm_keys_differ_and_known_hosts_pinned(self):
|
||||
ex = self.make_exec(runner=self._fake_keygen_runner)
|
||||
r1 = ex.provision_host_key({"id": "vm-a"})
|
||||
r2 = ex.provision_host_key({"id": "vm-b"})
|
||||
# 不同 VM 的私钥文件不同路径、known_hosts 不同文件。
|
||||
self.assertNotEqual(r1["paths"]["private"],
|
||||
r2["paths"]["private"])
|
||||
self.assertNotEqual(r1["known_hosts"], r2["known_hosts"])
|
||||
for r in (r1, r2):
|
||||
with open(r["known_hosts"]) as fh:
|
||||
line = fh.read().strip()
|
||||
self.assertTrue(line.startswith("[127.0.0.1]:"))
|
||||
self.assertIn("ssh-ed25519 AAAAfake1", line)
|
||||
self.assertNotIn("PRIVATE KEY", line)
|
||||
# 同一 VM 再次预置复用已有对(不重生成、不改 known_hosts)。
|
||||
with open(r1["known_hosts"]) as fh:
|
||||
before = fh.read()
|
||||
ex.provision_host_key({"id": "vm-a"})
|
||||
with open(r1["known_hosts"]) as fh:
|
||||
self.assertEqual(fh.read(), before)
|
||||
|
||||
def test_seed_contains_private_key_and_is_deleted_on_stop(self):
|
||||
ex = self.make_exec(runner=self._fake_keygen_runner)
|
||||
r = ex.provision_host_key({"id": "vm-c"})
|
||||
seed_user = os.path.join(r["seed_dir"], "user-data")
|
||||
self.assertTrue(os.path.exists(seed_user))
|
||||
with open(seed_user) as fh:
|
||||
seed_text = fh.read()
|
||||
self.assertIn("ssh_host_ed25519_key", seed_text)
|
||||
self.assertIn("ssh_deletekeys: false", seed_text)
|
||||
# stop 时 seed 目录与私钥删掉,known_hosts 公钥保留审计。
|
||||
ex.stop_vm({"id": "vm-c"})
|
||||
self.assertFalse(os.path.exists(r["seed_dir"]))
|
||||
self.assertFalse(os.path.exists(r["paths"]["private"]))
|
||||
self.assertTrue(os.path.exists(r["known_hosts"]))
|
||||
|
||||
def test_known_hosts_line_rejects_bad_key(self):
|
||||
with self.assertRaises(ValueError):
|
||||
known_hosts_line(2222, "not-a-key")
|
||||
self.assertEqual(
|
||||
known_hosts_line(2222, "ssh-ed25519 AAAAx comment"),
|
||||
"[127.0.0.1]:2222 ssh-ed25519 AAAAx")
|
||||
|
||||
def test_seed_user_data_shape(self):
|
||||
text = build_seed_user_data("PRIV\n", "ssh-ed25519 AAAAx\n")
|
||||
self.assertIn("/etc/ssh/ssh_host_ed25519_key", text)
|
||||
self.assertIn("ssh_deletekeys: false", text)
|
||||
with self.assertRaises(ValueError):
|
||||
build_seed_user_data("", "ssh-ed25519 AAAAx")
|
||||
|
||||
def test_keyscan_is_weak_fallback_only(self):
|
||||
# provisioned 模式下 wake 前必须预置;keyscan 不应被
|
||||
# 当成生产路径:配置值本身可解析,且文档/代码里标弱。
|
||||
ex = self.make_exec(env_extra={"CHV_HOST_KEY_MODE": "provisioned"})
|
||||
self.assertEqual(ex.host_key_mode, "provisioned")
|
||||
with self.assertRaises(ValueError):
|
||||
self.make_exec(env_extra={"CHV_HOST_KEY_MODE": "bogus"})
|
||||
|
||||
|
||||
# -- (4) egress tap 校验 -------------------------------------------------
|
||||
class EgressValidationTest(ChvGapCase):
|
||||
def test_direct_egress_without_flag_is_fail_closed(self):
|
||||
ex = self.make_exec()
|
||||
ex.egress_proxy = None
|
||||
ex.allow_direct_egress = False
|
||||
with self.assertRaises(ChvFailure) as ctx:
|
||||
ex.preflight()
|
||||
self.assertEqual(ctx.exception.kind, "egress_not_configured")
|
||||
|
||||
def test_direct_egress_with_flag_passes_and_is_audited(self):
|
||||
ex = self.make_exec(env_extra={"CHV_ALLOW_DIRECT_EGRESS": "1"})
|
||||
# preflight 本身不拦显式直连(守卫在 wake 的审计里)。
|
||||
self.assertTrue(ex.preflight())
|
||||
out = ex.goal_task({"id": "g1", "prompt": "做目标",
|
||||
"goal_mode": True})
|
||||
self.assertEqual(out["status"], "completed")
|
||||
self.assertTrue(any(e == "egress_direct_allowed"
|
||||
for e, _, _, _ in self.audit))
|
||||
|
||||
def test_bad_net_json_is_preflight_error(self):
|
||||
ex = self.make_exec(env_extra={"CHV_NET_JSON": "{not json"})
|
||||
with self.assertRaises(ChvFailure) as ctx:
|
||||
ex.preflight()
|
||||
self.assertEqual(ctx.exception.kind, "net_config_invalid")
|
||||
|
||||
|
||||
# -- (5) 构建脚本 --------------------------------------------------------
|
||||
class BuildScriptTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.dir = self.tmp.name
|
||||
self.bin_dir = os.path.join(self.dir, "bin")
|
||||
os.makedirs(os.path.join(self.bin_dir, "codex-resources"))
|
||||
for name in ("codex", "codex-code-mode-host"):
|
||||
p = os.path.join(self.bin_dir, name)
|
||||
with open(p, "w") as fh:
|
||||
fh.write("#!/bin/sh\n")
|
||||
os.chmod(p, 0o755)
|
||||
self.repo = os.path.join(self.dir, "repo")
|
||||
os.makedirs(os.path.join(self.repo, "cloud"))
|
||||
with open(os.path.join(self.repo, "host.py"), "w") as fh:
|
||||
fh.write("# host\n")
|
||||
self.script = str(Path(__file__).resolve().parents[2]
|
||||
/ "chv-guest" / "build.sh")
|
||||
|
||||
def tearDown(self):
|
||||
self.tmp.cleanup()
|
||||
|
||||
def _run(self, args, env_extra=None):
|
||||
env = dict(os.environ)
|
||||
env.update({"CODEX_BIN_DIR": self.bin_dir,
|
||||
"UBUNTU_VERSION": "24.04",
|
||||
"CODEX_VERSION": "0.164.0-alpha.9",
|
||||
"REPO_COMMIT": "test123",
|
||||
"REPO_DIR": self.repo})
|
||||
if env_extra:
|
||||
env.update(env_extra)
|
||||
return subprocess.run(["bash", self.script] + args,
|
||||
capture_output=True, text=True, env=env)
|
||||
|
||||
def test_dry_run_prints_real_steps_and_builds_nothing(self):
|
||||
out_dir = os.path.join(self.dir, "out")
|
||||
r = self._run(["--dry-run", out_dir])
|
||||
self.assertEqual(r.returncode, 0, r.stderr)
|
||||
self.assertIn("qemu-img convert", r.stdout)
|
||||
self.assertIn("virt-customize", r.stdout)
|
||||
self.assertIn("virt-get-kernel", r.stdout)
|
||||
self.assertIn("BUILDINFO", r.stdout)
|
||||
self.assertIn("chv-guest-24.04-0.164.0-alpha.9-test123", r.stdout)
|
||||
self.assertFalse(os.path.exists(os.path.join(out_dir, "disk.raw")))
|
||||
|
||||
def test_full_mode_without_kvm_fails_fast(self):
|
||||
# 本 VM 容器内无 /dev/kvm,全量模式必须明确失败,
|
||||
# 绝不声称已建成镜像。
|
||||
r = self._run([os.path.join(self.dir, "out2")])
|
||||
self.assertNotEqual(r.returncode, 0)
|
||||
self.assertIn("/dev/kvm", r.stderr + r.stdout)
|
||||
|
||||
def test_missing_binary_fails_fast(self):
|
||||
os.remove(os.path.join(self.bin_dir, "codex"))
|
||||
r = self._run(["--dry-run", os.path.join(self.dir, "out3")])
|
||||
self.assertNotEqual(r.returncode, 0)
|
||||
|
||||
|
||||
# -- (6) 中途死亡 → interrupted → 断点续跑闭环 ---------------------------
|
||||
class MidDeathLoopTest(ChvGapCase):
|
||||
def test_vm_death_is_interrupted_then_breakpoint_resumes(self):
|
||||
state = {"dead": False}
|
||||
|
||||
def runner(cmd, timeout=None):
|
||||
self.calls.append(list(cmd))
|
||||
if cmd and cmd[0] == "ssh":
|
||||
if cmd[-1] == "true":
|
||||
return FakeResult(0, "", "")
|
||||
if "goal-resume" in cmd and state["dead"]:
|
||||
# 第二次断点调用:VM 已换新,健康返回。
|
||||
return FakeResult(0, "goal met after rounds\n", "")
|
||||
if "goal" in cmd:
|
||||
# 第一次 goal:模拟传输中断,并把 VMM 进程标死。
|
||||
state["dead"] = True
|
||||
# 找到登记的 VMM 进程并标死(wake 已登记)。
|
||||
for proc in list(ex._vms.values()):
|
||||
proc._alive = False
|
||||
raise RuntimeError("connection lost mid-task")
|
||||
return FakeResult(0, "", "")
|
||||
|
||||
ex = self.make_exec(runner=runner)
|
||||
# 第一次 goal:中途死应判 interrupted,不是 failed 死信。
|
||||
out = ex.goal_task({"id": "g9", "prompt": "做目标",
|
||||
"goal_mode": True})
|
||||
self.assertEqual(out["status"], "interrupted")
|
||||
# 断点续跑:同一执行器的断点方法能走通并完成。
|
||||
state["dead"] = True # 标记已进入断点阶段(runner 按此分支)
|
||||
out2 = ex.goal_resume_breakpoint_task(
|
||||
{"id": "g9", "prompt": "续跑"})
|
||||
self.assertEqual(out2["status"], "completed")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
+86
-1
@@ -120,6 +120,7 @@ Docker 下 repos_dir 必须放在 extra-mounts 已挂载的目录下,
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import inspect
|
||||
import json
|
||||
import os
|
||||
from datetime import datetime, timezone
|
||||
@@ -332,6 +333,24 @@ def _merge_env(base, override):
|
||||
return merged
|
||||
|
||||
|
||||
def _call_goal_resume_breakpoint(executor, task, model_provider=None):
|
||||
"""通用断点调用:执行器提供 goal_resume_breakpoint_task 时调用它。
|
||||
|
||||
Local 仍走原路径(调用方直接传 model_provider,不经过这里,
|
||||
以免改生产语义);这里只用于非 Local 执行器:签名接受
|
||||
model_provider 时传它(CHV/Docker 均接受),否则只传 task
|
||||
(旧假执行器单参)。网关 key 绝不经这里传值:worker 侧
|
||||
gateway_env 只按名注入,见 _run_attempt 的网关守卫。"""
|
||||
method = getattr(executor, "goal_resume_breakpoint_task")
|
||||
try:
|
||||
params = inspect.signature(method).parameters
|
||||
except (TypeError, ValueError):
|
||||
return method(task)
|
||||
if "model_provider" in params:
|
||||
return method(task, model_provider=model_provider)
|
||||
return method(task)
|
||||
|
||||
|
||||
def _resolve_model(model=None, config_path=None):
|
||||
"""模型名:显式参数 > host 配置文件里的 model > host 默认值。"""
|
||||
if model:
|
||||
@@ -525,6 +544,19 @@ class Worker:
|
||||
self.executor.audit_callback = _chv_audit
|
||||
except Exception:
|
||||
pass
|
||||
# CHV 补缺口轨:WarmPool 显式接线。池构造本身不自动
|
||||
# 清理(单活池假设,见 pool.py),worker 启动是显式
|
||||
# 入口,在这里建池并调 reconcile_orphans();关闭时
|
||||
# close() 再调一次。调用失败不炸 worker 构造/关闭,
|
||||
# 但要进审计(_audit 本身 best-effort)。
|
||||
self._warm_pool = None
|
||||
if isinstance(self.executor, CloudHypervisorExecutor):
|
||||
try:
|
||||
from cloud.chv.pool import WarmPool as _WarmPool
|
||||
self._warm_pool = _WarmPool(self.executor)
|
||||
self._reconcile_warm_pool("startup")
|
||||
except Exception:
|
||||
pass
|
||||
self._cost_store = cost_store
|
||||
self._cancel_requested = set()
|
||||
# 真取消用:process_factory 可注入(单测用它返回一个假长跑
|
||||
@@ -759,8 +791,49 @@ class Worker:
|
||||
self._cost_store = CostStore(self.db_path)
|
||||
return self._cost_store
|
||||
|
||||
def _reconcile_warm_pool(self, when):
|
||||
"""显式调 WarmPool.reconcile_orphans(),返回已清理 VM 名。
|
||||
|
||||
仅 CHV worker 且池已建时有效;池未建时若执行器是 CHV
|
||||
则现建现调(供测试与手动清理用)。调用失败不炸主流程,
|
||||
但进审计(失败本身也审计,不静默丢弃)。orphan 条目
|
||||
绝不发给任务,reconcile 只做清理。"""
|
||||
pool = getattr(self, "_warm_pool", None)
|
||||
if pool is None:
|
||||
executor = getattr(self, "executor", None)
|
||||
if not isinstance(executor, CloudHypervisorExecutor):
|
||||
return []
|
||||
try:
|
||||
from cloud.chv.pool import WarmPool as _WarmPool
|
||||
pool = _WarmPool(executor)
|
||||
self._warm_pool = pool
|
||||
except Exception:
|
||||
return []
|
||||
try:
|
||||
cleaned = pool.reconcile_orphans() or []
|
||||
except Exception as exc:
|
||||
try:
|
||||
self._audit(None, "chv_warm_pool_reconcile_failed",
|
||||
"when=%s error=%s" % (when, exc))
|
||||
except Exception:
|
||||
pass
|
||||
return []
|
||||
if cleaned:
|
||||
try:
|
||||
self._audit(None, "chv_warm_pool_reconcile",
|
||||
"when=%s cleaned=%s" % (when, ",".join(cleaned)))
|
||||
except Exception:
|
||||
pass
|
||||
return cleaned
|
||||
|
||||
def close(self):
|
||||
"""关闭惰性建的成本存储连接(注入的 cost_store 由调用方管)。"""
|
||||
"""关闭惰性建的成本存储连接(注入的 cost_store 由调用方管)。
|
||||
CHV worker 关闭时先显式调一次 warm-pool reconcile,
|
||||
失败不炸关闭流程。"""
|
||||
try:
|
||||
self._reconcile_warm_pool("shutdown")
|
||||
except Exception:
|
||||
pass
|
||||
if self._cost_store is not None:
|
||||
try:
|
||||
self._cost_store._conn.close()
|
||||
@@ -4712,6 +4785,18 @@ class Worker:
|
||||
self.executor, LocalExecutor):
|
||||
result = self.executor.goal_resume_breakpoint_task(
|
||||
task, model_provider=gateway_provider)
|
||||
elif task.get("_goal_resume_breakpoint") and hasattr(
|
||||
self.executor,
|
||||
"goal_resume_breakpoint_task"):
|
||||
# CHV 补缺口轨:任何提供断点方法的执行器
|
||||
# 都可走断点(CHV 的 goal_resume_breakpoint_task
|
||||
# 已接线;Docker 走上面 is_docker 分支,不到
|
||||
# 这里)。网关任务在进入这里之前已被
|
||||
# “网关不支持该执行器”守卫 loud fail,
|
||||
# 故这里不放宽网关规则。
|
||||
result = _call_goal_resume_breakpoint(
|
||||
self.executor, task,
|
||||
model_provider=gateway_provider)
|
||||
elif task.get("_goal_resume_breakpoint"):
|
||||
# 假执行器/CloudExecutor 没有断点方法时
|
||||
# 绝不回落成 goal(会撞已存在)或
|
||||
|
||||
@@ -49,10 +49,15 @@
|
||||
非 root 客机用户(CHV_SSH_USER,显式空串直接拒绝)与端口
|
||||
(CHV_SSH_PORT,1-65535)为显式配置。mock 只验证捕获→
|
||||
写文件→钉死选项的管线,不验证首次密钥真实性。
|
||||
- 🔒 首次信任边界未闭合:首次捕获是 TOFU,只防首次之后的
|
||||
替换,不验证首次本身;更强的方案是镜像构建/cloud-init 时
|
||||
把客机 host key 预置进 known_hosts,仍是待办,安全审计时
|
||||
须单独立项确认。
|
||||
- ✅🧪 Host key 预置方案(补缺口轨,替代 TOFU 为生产路径):
|
||||
宿主在启动前为每 VM 生成一次性 ed25519 host key 对,
|
||||
公钥立即写进该 VM 的 known_hosts 钉死,私钥只进 per-VM
|
||||
cloud-init seed(随 VM 关闭/回收而删),客机镜像硬性
|
||||
要求见 chv-guest/README.md §6(cloud-init NoCloud、
|
||||
首启先应用 seed 再起 sshd、不得无条件重生成 host key)。
|
||||
ssh-keyscan 首次捕获只保留为弱回落(CHV_HOST_KEY_MODE
|
||||
=keyscan),生产部署必须用 provisioned 模式;TOFU 首次
|
||||
真实性仍待预置 key 方案在真机上验证,安全审计时须单独立项确认。
|
||||
- 🔒 需安全审计:VM 逃逸面(virtio 设备、tap 桥接、SSH 配置)、
|
||||
客机 root 提权面、租户间东西向隔离的宿主网络配置,须由
|
||||
安全审计逐项确认,代码自查不能代替审计。
|
||||
@@ -114,8 +119,10 @@
|
||||
pandoc/poppler/latex、python-docx/openpyxl/pptx/PyMuPDF/
|
||||
Pillow、codex 0.164.0-alpha.9 成套、Playwright + Chromium、
|
||||
openssh-server、cloud-init)。
|
||||
- ✅ 构建脚本骨架:`chv-guest/build.sh`(参数校验、版本 tag
|
||||
格式、KVM 宿主检查、TODO 步骤清单)。
|
||||
- ✅ 构建脚本实做(补缺口轨):`chv-guest/build.sh`(参数校验、
|
||||
KVM 宿主检查、下载 Ubuntu 云镜像、qemu-img 转 raw、
|
||||
virt-customize 装清单、拷入 /app 代码、virt-get-kernel、
|
||||
BUILDINFO.json 生成与 tag 校验;--dry-run 只打印命令)。
|
||||
- 🏗️ 需镜像构建:镜像本身未构建、未发布;BUILDINFO 版本核对、
|
||||
浏览器版本钉死、中文 LaTeX 单独一版,均待构建时做。
|
||||
- ⏳ 需 KVM 宿主实测:镜像在真 VM 里启动、codex/Playwright/
|
||||
@@ -168,11 +175,14 @@
|
||||
egress_not_configured/api_timeout/guest_not_ready/vm_create_failed/
|
||||
vm_boot_failed/host_capacity_exceeded/disk_quota_exceeded),
|
||||
不笼统 failed。
|
||||
- ✅ 断点对齐:VM 快照/休眠与现有 task_checkpoints/task_attempts
|
||||
语义对齐(recycle 保留两者,丢失临时文件);goal 断点续跑
|
||||
方法(goal_resume_breakpoint_task)已实现,worker 层对非
|
||||
Local 执行器的断点续跑目前仍明确失败(见执行器 docstring),
|
||||
后续 worker 接线是单独一轨,不在本轨假装已通。
|
||||
- ✅ 断点对齐(补缺口轨已接线):VM 快照/休眠与现有
|
||||
task_checkpoints/task_attempts 语义对齐(recycle 保留两者,
|
||||
丢失临时文件);worker 通用分支对任何提供
|
||||
goal_resume_breakpoint_task 的执行器调用它(Local 仍走
|
||||
原路径,网关任务仍被 loud fail 守卫拦住);WarmPool
|
||||
reconcile_orphans 已由 worker 启动与关闭时显式调用,
|
||||
失败不炸主流程但进审计;VM 中途死亡改判 interrupted
|
||||
后走同一断点路径,不再落 failed 死信(见 §3 与新增测试)。
|
||||
- ⏳ 需 KVM 宿主实测:审计事件在真机任务 timeline 里的完整
|
||||
呈现、失败分类与真实故障的对应关系,均未验证。
|
||||
|
||||
|
||||
Reference in new issue
Block a user