fix(chv-hybrid): p2 第三轮:attach 轮询等挂盘落地 + detach 走 sudo 窄规则 umount
ci / test (push) Failing after 47m51s

第 2 轮复测(0dc8328)p2 仍 FAIL 的两处根因:
1. 挂盘时序竞态:add-disk 返回 06:59:10,root hotplug 路径
   (cloud-init-hotplugd → cloud-final runcmd → chv-data-mount)
   06:59:14–16 才挂上;旧 attach 以 chv 调 chv-data-mount 并判
   rc==0,但 /dev/vd? 属 root:disk、chv 的 blkid/mkfs 必败,
   脚本又静默 exit 0,rc 检查是空洞的成功假象。
2. 卸载权限:挂载是 root 做的,chv 的 plain umount 恒 rc=32
   must be superuser,release 按现写法永远失败、VM 留置。

修法选 (a) 轮询,不选 (b) root 特权挂载:挂盘本身已有 root
路径在跑,host 侧不另开新的 root 挂载通道(新特权面只扩
攻击面);轮询以 chv 只读判定挂载态,零新增特权。
- attach:删 chv 调 chv-data-mount,改为轮询 mountpoint -q
  /home/chv/workspace && test -w(默认 60 秒/2 秒,可经
  CHV_DATA_MOUNT_TIMEOUT_SECONDS / CHV_DATA_MOUNT_INTERVAL_SECONDS
  注入短值);超时记 data_disk_mount_timeout 审计并抛同名
  kind,不写归属标记。attach 返回成功 = 挂盘真实发生。
  resume_seconds 因此真实包含挂盘等待(§2.1 口径),≤2 秒
  目标不改口径、不改目标。
- detach:改走 sudo -n /usr/bin/umount /home/chv/workspace;
  失败即 False,release 留置 VM 不回收(fail-closed)。
- chv-data-mount.sh:盘已找到后 mkfs/mount 失败改 exit 1
  (原全部失败路径 exit 0);无盘仍 exit 0(启动期 no-op,
  不挡 boot)。本脚本只由 root 跑(service/runcmd)。
- build.sh:apt 加 sudo;写 /etc/sudoers.d/chv-data-umount
  窄规则(chv 仅能以 root 跑 umount /home/chv/workspace
  这一条精确 argv,NOPASSWD,无 shell/其它命令/挂载权),
  0440 + visudo -cf 作构建门;buildinfo/lockfile 登记
  sudo 与 chv-data-umount。

特权边界:本轮唯一新增提权面就是上述一条 sudoers 规则;
attach 零提权(只读轮询),挂载权仍只在 root 的
hotplug/service 路径手里。旧镜像(无 sudo/无规则)上
detach 返回 False、release 留置,须重建镜像才恢复回收。

测试:新增 10 项(attach 等三次轮询才成功、超时 loud 失败、
永不以 chv 调 chv-data-mount、detach 精确 sudo argv、
sudo 失败 False、卸载失败 release 留置、客机脚本 exit 码、
build.sh sudo/窄规则/buildinfo 登记);另修 2 处旧断言
(attach 不再调挂载脚本)与 stage4 lockfile 夹具(补 sudo
dpkg 条目),并把 attach 热插段抽为 _hotplug_tenant_disk
(50 行函数上限)。全量 Ran 4688,OK (skipped=15,均为
环境 skip:无 PG DSN/驱动、缺 SearXNG_URL、fallback state
文件在位)。
This commit is contained in:
host committed 2026-11-09 08:17:01 +00:00
1 parent 0dc8328ecc
commit a859d731d5
8 files changed
+351 -37

No files matched your search

+12 -1
View File
@@ -277,7 +277,7 @@ run qemu-img convert -f qcow2 -O raw "$IMG" "$OUT/disk.raw"
# 写进 /app/guest-versions-chromium.txt)。 # 写进 /app/guest-versions-chromium.txt)。
GT_INSTALL_CMD="curl -fsSL -o /tmp/gt-install.sh https://gitea.com/gitea/cli/raw/branch/main/scripts/install.sh && echo \"${GT_INSTALL_SH_SHA256} /tmp/gt-install.sh\" | sha256sum -c - && GITEA_CLI_VERSION=${GT_VERSION} GITEA_CLI_INSTALL_DIR=/usr/local/bin sh /tmp/gt-install.sh && rm -f /tmp/gt-install.sh && /usr/local/bin/gt version" GT_INSTALL_CMD="curl -fsSL -o /tmp/gt-install.sh https://gitea.com/gitea/cli/raw/branch/main/scripts/install.sh && echo \"${GT_INSTALL_SH_SHA256} /tmp/gt-install.sh\" | sha256sum -c - && GITEA_CLI_VERSION=${GT_VERSION} GITEA_CLI_INSTALL_DIR=/usr/local/bin sh /tmp/gt-install.sh && rm -f /tmp/gt-install.sh && /usr/local/bin/gt version"
VC_ARGS=(-a "$OUT/disk.raw" VC_ARGS=(-a "$OUT/disk.raw"
--run-command 'apt-get update -qq && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends git openssh-server openssh-client cloud-init python3 python3-pip curl ca-certificates xz-utils fonts-wqy-microhei fontconfig systemd-container iproute2 iptables && apt-get clean' --run-command 'apt-get update -qq && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends git openssh-server openssh-client cloud-init python3 python3-pip curl ca-certificates xz-utils fonts-wqy-microhei fontconfig systemd-container iproute2 iptables sudo && apt-get clean'
--copy-in "$CODEX_BIN_DIR/codex:/usr/local/bin/" --copy-in "$CODEX_BIN_DIR/codex:/usr/local/bin/"
--copy-in "$CODEX_BIN_DIR/codex-code-mode-host:/usr/local/bin/" --copy-in "$CODEX_BIN_DIR/codex-code-mode-host:/usr/local/bin/"
--copy-in "$CODEX_BIN_DIR/codex-resources:/usr/local/bin/" --copy-in "$CODEX_BIN_DIR/codex-resources:/usr/local/bin/"
@@ -298,6 +298,17 @@ VC_ARGS=(-a "$OUT/disk.raw"
--run-command 'mv /usr/local/bin/chv-sync.sh /usr/local/bin/chv-sync && chmod +x /usr/local/bin/chv-sync' --run-command 'mv /usr/local/bin/chv-sync.sh /usr/local/bin/chv-sync && chmod +x /usr/local/bin/chv-sync'
--copy-in "$SCRIPT_DIR/chv-data-mount.sh:/usr/local/bin/" --copy-in "$SCRIPT_DIR/chv-data-mount.sh:/usr/local/bin/"
--run-command 'mv /usr/local/bin/chv-data-mount.sh /usr/local/bin/chv-data-mount && chmod +x /usr/local/bin/chv-data-mount' --run-command 'mv /usr/local/bin/chv-data-mount.sh /usr/local/bin/chv-data-mount && chmod +x /usr/local/bin/chv-data-mount'
# p2 第三轮(2026-11-09 第 2 轮复测根因 2):挂载是 root
# 做的,chv 身份 plain umount 恒 rc=32。唯一提权入口是这条
# sudoers 窄规则:chv 仅能以 root 跑 /usr/bin/umount
# /home/chv/workspace 这一条精确 argv(NOPASSWD,无密码;
# 无 shell、无其它命令、无挂载权——挂载仍只走 root 的
# hotplug/service 路径,host 侧 attach 轮询等它落地,不
# 另开 root 挂载通道)。0440 + visudo -cf 是构建门:
# 规则写错/权限不对即构建失败,不把坏 sudoers 带进镜像
# (坏 sudoers 会让整台客机 sudo 报错)。executor 的
# detach 以 `sudo -n` 调它(-n 绝不交互问密码)。
--run-command 'printf "chv ALL=(root) NOPASSWD: /usr/bin/umount /home/chv/workspace\n" > /etc/sudoers.d/chv-data-umount && chown root:root /etc/sudoers.d/chv-data-umount && chmod 0440 /etc/sudoers.d/chv-data-umount && visudo -cf /etc/sudoers.d/chv-data-umount'
--copy-in "$SCRIPT_DIR/chv-cell.py:/usr/local/bin/" --copy-in "$SCRIPT_DIR/chv-cell.py:/usr/local/bin/"
--copy-in "$SCRIPT_DIR/chv-broker.py:/usr/local/bin/" --copy-in "$SCRIPT_DIR/chv-broker.py:/usr/local/bin/"
--copy-in "$SCRIPT_DIR/chv-browser.py:/usr/local/bin/" --copy-in "$SCRIPT_DIR/chv-browser.py:/usr/local/bin/"
+4
View File
@@ -37,6 +37,10 @@ BASE_CONTENTS = [
"xz-utils", "fonts-wqy-microhei", "fontconfig", "codex", "xz-utils", "fonts-wqy-microhei", "fontconfig", "codex",
"codex-code-mode-host", "codex-resources/bwrap", "codex-code-mode-host", "codex-resources/bwrap",
"/app/host.py", "gt", "chv-sync", "/app/host.py", "gt", "chv-sync",
# p2 第三轮:sudo(dpkg 包,版本走 dpkg freeze)+
# chv-data-umount sudoers 窄规则(版本随 repo_commit,
# 见 check/generate_lockfile.py 的 DECLARED_SOURCES)。
"sudo", "chv-data-umount",
# 对齐 Muse P2:nspawn 单元启动器 + broker + 浏览器子代理。 # 对齐 Muse P2:nspawn 单元启动器 + broker + 浏览器子代理。
"systemd-container", "chv-cell", "chv-broker", "chv-browser", "systemd-container", "chv-cell", "chv-broker", "chv-browser",
# feat-chv-opt-landing(阶段 1):数据盘挂载脚本。 # feat-chv-opt-landing(阶段 1):数据盘挂载脚本。
+3
View File
@@ -33,6 +33,9 @@ DECLARED_SOURCES = {
# feat-chv-opt-landing(阶段 1):数据盘挂载脚本,版本随 # feat-chv-opt-landing(阶段 1):数据盘挂载脚本,版本随
# repo_commit(与 chv-sync 同款)。 # repo_commit(与 chv-sync 同款)。
"chv-data-mount": "repo_commit", "chv-data-mount": "repo_commit",
# p2 第三轮:卸载 sudoers 窄规则(build.sh 内联写入,
# 内容随仓库走,版本随 repo_commit)。
"chv-data-umount": "repo_commit",
# feat-chv-opt-landing(阶段 2):客机运行时三分服务,版本 # feat-chv-opt-landing(阶段 2):客机运行时三分服务,版本
# 随 repo_commit(与 chv-sync 同款)。 # 随 repo_commit(与 chv-sync 同款)。
"chv-daemon": "repo_commit", "chv-daemon": "repo_commit",
+13 -2
View File
@@ -36,13 +36,24 @@ if [ -z "$DEV" ]; then
done done
fi fi
# p2 第三轮退出码契约(2026-11-09 第 2 轮复测):无数据盘
# 即静默退出 0(启动期 no-op:干净池 VM 与 copy 模式旧 VM
# 本来就没有第三块盘,boot 服务绝不能因此失败);但盘已
# 找到后 mkfs/mount 任一步失败即 exit 1 并记日志——旧版
# 全部失败路径 exit 0 是“成功假象”的另一半:host 侧曾以
# chv 调本脚本并判 rc==0,chv 无权读 /dev/vd?、mkfs 必败
# 却被报成成功。host 侧现已改走轮询等 root hotplug 路径
# 落地(见 executor.py attach),不再以 chv 调本脚本;
# 本脚本只由 root 跑(chv-data-mount.service 与
# cloud-final runcmd),失败 loud 不挡启动(runcmd 行带
# || true,service 失败可见于 systemctl status)。
[ -n "$DEV" ] || { log "no data disk found, skip"; exit 0; } [ -n "$DEV" ] || { log "no data disk found, skip"; exit 0; }
if [ "$(blkid -o value -s LABEL "$DEV" 2>/dev/null || echo "")" != "$LABEL" ]; then if [ "$(blkid -o value -s LABEL "$DEV" 2>/dev/null || echo "")" != "$LABEL" ]; then
log "formatting $DEV as $LABEL" log "formatting $DEV as $LABEL"
mkfs.ext4 -L "$LABEL" "$DEV" || { log "mkfs failed on $DEV"; exit 0; } mkfs.ext4 -L "$LABEL" "$DEV" || { log "mkfs failed on $DEV"; exit 1; }
fi fi
mkdir -p "$MOUNT" mkdir -p "$MOUNT"
mount "LABEL=$LABEL" "$MOUNT" || { log "mount failed"; exit 0; } mount "LABEL=$LABEL" "$MOUNT" || { log "mount failed"; exit 1; }
chown chv:chv "$MOUNT" || true chown chv:chv "$MOUNT" || true
# feat-chv-opt-landing(阶段 3):浏览器持久 profile 落数据盘。 # feat-chv-opt-landing(阶段 3):浏览器持久 profile 落数据盘。
# chv-browser 以 chv-browser 身份跑,工作区根属 chv、它写不进; # chv-browser 以 chv-browser 身份跑,工作区根属 chv、它写不进;
+3
View File
@@ -54,6 +54,9 @@ DECLARED_SOURCES = {
# feat-chv-opt-landing(阶段 1):数据盘挂载脚本,版本随 # feat-chv-opt-landing(阶段 1):数据盘挂载脚本,版本随
# repo_commit(与 chv-sync 同款)。 # repo_commit(与 chv-sync 同款)。
"chv-data-mount": "repo_commit", "chv-data-mount": "repo_commit",
# p2 第三轮:卸载 sudoers 窄规则(build.sh 内联写入,
# 内容随仓库走,版本随 repo_commit)。
"chv-data-umount": "repo_commit",
# feat-chv-opt-landing(阶段 2):客机运行时三分服务,版本 # feat-chv-opt-landing(阶段 2):客机运行时三分服务,版本
# 随 repo_commit(与 chv-sync 同款)。 # 随 repo_commit(与 chv-sync 同款)。
"chv-daemon": "repo_commit", "chv-daemon": "repo_commit",
+123 -30
View File
@@ -263,6 +263,32 @@ HYBRID_COLD_RESTORE_TARGET_SECONDS = 30
HYBRID_SNAPSHOT_PER_TENANT = 1 HYBRID_SNAPSHOT_PER_TENANT = 1
HYBRID_SNAPSHOT_RETENTION_DAYS = 30 HYBRID_SNAPSHOT_RETENTION_DAYS = 30
HYBRID_RESIDENT_TIERS = ("paid", "heavy") HYBRID_RESIDENT_TIERS = ("paid", "heavy")
# p2 第三轮(feat-chv-hybrid,2026-11-09 第 2 轮复测):
# 数据盘挂盘等待上限/轮询间隔。真机时序:attach 的 add-disk
# 返回 06:59:10,root 的 hotplug 路径(cloud-init-hotplugd →
# cloud-final runcmd → chv-data-mount)06:59:14–16 才把盘挂
# 上——热插后挂盘不是同步发生的,须轮询等它落地。60 秒上限
# 覆盖 hotplug + cloud-final 重跑的实测 ~6 秒并留足余量;
# 测试经 CHV_DATA_MOUNT_TIMEOUT_SECONDS /
# CHV_DATA_MOUNT_INTERVAL_SECONDS 注入短值(调用时现读,
# 与 _hybrid_env_float 同款非法值回默认)。≤2 秒热恢复目标
# 不变(DESIGN-HYBRID §2.1),本轮只修“attach 返回成功 =
# 挂盘真实发生”的语义,不改口径、不改目标。
DATA_MOUNT_TIMEOUT_SECONDS = 60
DATA_MOUNT_POLL_INTERVAL_SECONDS = 2
# 客内工作区挂载点(chv-data-mount 的挂载目标,LABEL=CHVDATA)。
DATA_MOUNT_POINT = "/home/chv/workspace"
# 卸载特权路径(p2 第三轮):挂载是 root 做的,chv 身份的
# plain umount 恒报 rc=32 must be superuser。唯一提权入口是
# 镜像里的 sudoers 窄规则(/etc/sudoers.d/chv-data-umount,
# 见 chv-guest/build.sh):chv ALL=(root) NOPASSWD: 仅
# /usr/bin/umount /home/chv/workspace 这一条精确 argv,
# 无 shell、无其它命令、无挂载权。sudo -n 绝不交互问密码,
# 规则缺失/镜像未重建时直接失败(detach 返回 False →
# release 不回收、VM 留置待人工),fail-closed,不回落
# plain umount(那条路恒失败,回落只是噪音)。
DATA_UMOUNT_PRIVILEGED_COMMAND = (
"sudo", "-n", "/usr/bin/umount", DATA_MOUNT_POINT)
CLEAN_POOL_DIR_NAME = "clean_pool" CLEAN_POOL_DIR_NAME = "clean_pool"
# 干净池预热时拒绝的租户同步键(Gitea 同步变量):池 VM 出生 # 干净池预热时拒绝的租户同步键(Gitea 同步变量):池 VM 出生
# 时不带租户,显式带这些键进预热即调用方错误,fail-closed。 # 时不带租户,显式带这些键进预热即调用方错误,fail-closed。
@@ -3820,28 +3846,57 @@ class CloudHypervisorExecutor(Executor):
"租户数据盘的归属标记是 %s,与请求方 %s 不一致," "租户数据盘的归属标记是 %s,与请求方 %s 不一致,"
"拒绝挂载" % (owner, tenant)) "拒绝挂载" % (owner, tenant))
def attach_tenant_data_disk(self, task: dict, vm: str, def _data_mount_poll_params(self):
data_disk: str | None = None) -> dict: """挂盘等待参数(超时,间隔):测试可经
"""分配时挂载租户数据盘(§3.1 首选方案):供给/复用该租户 CHV_DATA_MOUNT_TIMEOUT_SECONDS /
的数据盘 → 过任务盘来源不变量 → CHV 热挂载(PUT CHV_DATA_MOUNT_INTERVAL_SECONDS 注入短值;非法/
/vm.add-disk,CHV 54 的热插盘端点)→ 客内挂载脚本 缺失/<=0 即回生产默认,绝不因配错而停表(与
(chv-data-mount,按 LABEL=CHVDATA 挂到 _hybrid_env_float 同款)。"""
/home/chv/workspace)→ 写 VM 归属标记。任一步失败即 return (
ChvFailure(fail-closed,调用方回落冷启或拒绝,不把 self._hybrid_env_float(
未挂盘的 VM 当已绑定发给任务)。返回 {vm, tenant_id, "CHV_DATA_MOUNT_TIMEOUT_SECONDS",
data_disk}。热挂载 body 与 create 时盘同款:path + DATA_MOUNT_TIMEOUT_SECONDS),
image_type(CHV v54 已移除盘格式自动探测,只发 path self._hybrid_env_float(
时真机回 400,见 p2 复测);非 2xx 时 CHV 的 body "CHV_DATA_MOUNT_INTERVAL_SECONDS",
原样记入 data_disk_attach_failed 审计事件与异常 DATA_MOUNT_POLL_INTERVAL_SECONDS))
detail(超 500 字符截断,见 _api_error_body_text),
不再丢弃。""" def _wait_data_disk_mounted(self, task: dict, vm: str) -> bool:
tenant = self._tenant_id(task) """等客内数据盘真正挂上 /home/chv/workspace:轮询
if not tenant: `mountpoint -q <挂载点> && test -w <挂载点>`(chv
raise ChvFailure( 身份只读判定,不需特权:mountpoint 读内核挂载态,
"task_disk_source_invalid", test -w 验任务随后真能写进去)。每轮一次 SSH,上
"挂载数据盘须带 tenant_id:无归属的任务不能绑盘") 限见 _data_mount_poll_params;超时即 False(调用方
disk = data_disk or self._provision_tenant_data_disk(tenant) fail-closed,绝不把未挂盘的 VM 当已绑定发给任务)。
self._verify_task_disk_source(vm, tenant, disk) 注意:轮询不等 chv-data-mount 脚本本身——第 2 轮
复测已证实该脚本以 chv 跑时 /dev/vd? 属 root:disk,
blkid 读不到、mkfs 必败,旧 attach 的 rc==0 检查是
空洞的;真正挂盘的是 root 的 hotplug 路径
(cloud-init-hotplugd → cloud-final runcmd →
chv-data-mount),轮询等的就是它落地。"""
timeout, interval = self._data_mount_poll_params()
attempts = max(1, int(timeout / interval))
for attempt in range(attempts):
try:
result = self._call_runner(self.build_ssh_command(
{"id": vm},
["sh", "-c",
"mountpoint -q %s && test -w %s"
% (DATA_MOUNT_POINT, DATA_MOUNT_POINT)]))
if getattr(result, "returncode", 1) == 0:
return True
except Exception:
pass
if attempt + 1 < attempts:
self.sleeper(interval)
return False
def _hotplug_tenant_disk(self, task: dict, vm: str, disk: str):
"""CHV 热插盘(PUT /vm.add-disk,CHV 54 端点):body
与 create 时盘同款 path + image_type(只发 path 时
真机回 400 "Image type required for disk",见 p2 复测);
非 2xx 时 CHV 的 body 原样记入
data_disk_attach_failed 审计事件与异常 detail(超 500
字符截断,见 _api_error_body_text),不再丢弃。"""
status, body = self._api_call( status, body = self._api_call(
self.api_socket({"id": vm}), "PUT", "/vm.add-disk", self.api_socket({"id": vm}), "PUT", "/vm.add-disk",
{"path": disk, "image_type": _disk_image_type(disk)}) {"path": disk, "image_type": _disk_image_type(disk)})
@@ -3855,13 +3910,40 @@ class CloudHypervisorExecutor(Executor):
"data_disk_attach_failed", "data_disk_attach_failed",
"数据盘热挂载失败:vm=%s status=%s body=%s" "数据盘热挂载失败:vm=%s status=%s body=%s"
% (vm, status, body_text)) % (vm, status, body_text))
result = self._call_runner(self.build_ssh_command(
{"id": vm}, ["/usr/local/bin/chv-data-mount"])) def attach_tenant_data_disk(self, task: dict, vm: str,
if getattr(result, "returncode", 1) != 0: data_disk: str | None = None) -> dict:
"""分配时挂载租户数据盘(§3.1 首选方案):供给/复用该租户
的数据盘 → 过任务盘来源不变量 → CHV 热挂载(PUT
/vm.add-disk)→ 轮询等客内挂盘真正落地
(_wait_data_disk_mounted,修法 (a):挂盘已有 root
hotplug 路径在跑,不另开 root 挂载通道;旧的 chv 调
chv-data-mount 判 rc==0 已删除,那是静默 exit 0 的
成功假象)→ 写 VM 归属标记。任一步失败即 ChvFailure
(fail-closed,不把未挂盘的 VM 当已绑定发给任务)。
超时记 data_disk_mount_timeout 审计并抛同名 kind,
不写归属标记;resume_seconds 因此真实包含挂盘等待
(§2.1 口径),≤2 秒目标不改口径。"""
tenant = self._tenant_id(task)
if not tenant:
raise ChvFailure( raise ChvFailure(
"data_disk_mount_failed", "task_disk_source_invalid",
"客内挂载数据盘失败:vm=%s rc=%s" "挂载数据盘须带 tenant_id:无归属的任务不能绑盘")
% (vm, getattr(result, "returncode", None))) disk = data_disk or self._provision_tenant_data_disk(tenant)
self._verify_task_disk_source(vm, tenant, disk)
self._hotplug_tenant_disk(task, vm, disk)
timeout, _interval = self._data_mount_poll_params()
if not self._wait_data_disk_mounted(task, vm):
self._emit("data_disk_mount_timeout", task,
"vm=%s timeout=%s data_disk=%s"
% (vm, timeout, disk),
"data_disk_mount_timeout")
raise ChvFailure(
"data_disk_mount_timeout",
"数据盘热挂载后 %s 秒内客内 /home/chv/workspace "
"仍未挂上(vm=%s,热插已成功、root hotplug 挂盘"
"未落地),已拒绝绑定,不写归属标记"
% (timeout, vm))
write_vm_tenant(self.state_dir, vm, tenant, disk=disk) write_vm_tenant(self.state_dir, vm, tenant, disk=disk)
return {"vm": vm, "tenant_id": tenant, "data_disk": disk} return {"vm": vm, "tenant_id": tenant, "data_disk": disk}
@@ -3869,10 +3951,21 @@ class CloudHypervisorExecutor(Executor):
"""客内卸载租户数据盘(umount /home/chv/workspace,落盘 """客内卸载租户数据盘(umount /home/chv/workspace,落盘
回外部存储):成功 True;失败 False(调用方不得继续销毁 回外部存储):成功 True;失败 False(调用方不得继续销毁
VM,见 release_pool_vm——同步/卸载失败的 VM 不回收, VM,见 release_pool_vm——同步/卸载失败的 VM 不回收,
等人工/重试,绝不丢改动后静默回收)。""" 等人工/重试,绝不丢改动后静默回收)。
特权边界(p2 第三轮):挂载是 root 做的,chv 身份的
plain umount 恒报 rc=32 must be superuser(第 2 轮复
测实测),故此处走 sudo -n 窄规则(见
DATA_UMOUNT_PRIVILEGED_COMMAND 与 build.sh 的
/etc/sudoers.d/chv-data-umount):仅允许 chv 以 root
跑 `/usr/bin/umount /home/chv/workspace` 这一条精确
argv,无密码、无 shell、无其它命令、无挂载权;sudo
-n 失败即 False,不回落 plain umount。旧镜像(无
sudo/无该规则)上同样返回 False、release 留置 VM,
fail-closed,须重建镜像才恢复回收。"""
try: try:
result = self._call_runner(self.build_ssh_command( result = self._call_runner(self.build_ssh_command(
{"id": vm}, ["umount", "/home/chv/workspace"])) {"id": vm},
list(DATA_UMOUNT_PRIVILEGED_COMMAND)))
except Exception: except Exception:
return False return False
return getattr(result, "returncode", 1) == 0 return getattr(result, "returncode", 1) == 0
+192 -4
View File
@@ -33,6 +33,8 @@ try:
write_persistent_vm, write_persistent_vm,
) )
from cloud.chv.executor import ( from cloud.chv.executor import (
DATA_MOUNT_POLL_INTERVAL_SECONDS,
DATA_MOUNT_TIMEOUT_SECONDS,
HYBRID_COLD_RESTORE_TARGET_SECONDS, HYBRID_COLD_RESTORE_TARGET_SECONDS,
HYBRID_QUEUE_TIMEOUT_SECONDS, HYBRID_QUEUE_TIMEOUT_SECONDS,
HYBRID_SNAPSHOT_PER_TENANT, HYBRID_SNAPSHOT_PER_TENANT,
@@ -51,6 +53,8 @@ except ImportError: # discover -s cloud/tests 时把项目根目录补进 sys.p
write_persistent_vm, write_persistent_vm,
) )
from cloud.chv.executor import ( from cloud.chv.executor import (
DATA_MOUNT_POLL_INTERVAL_SECONDS,
DATA_MOUNT_TIMEOUT_SECONDS,
HYBRID_COLD_RESTORE_TARGET_SECONDS, HYBRID_COLD_RESTORE_TARGET_SECONDS,
HYBRID_QUEUE_TIMEOUT_SECONDS, HYBRID_QUEUE_TIMEOUT_SECONDS,
HYBRID_SNAPSHOT_PER_TENANT, HYBRID_SNAPSHOT_PER_TENANT,
@@ -492,9 +496,15 @@ class DataDiskAttachTest(HybridACase):
# CHV v54 已移除盘格式自动探测:热插盘 body 须与 create # CHV v54 已移除盘格式自动探测:热插盘 body 须与 create
# 时盘同款带 image_type(租户数据盘是 raw 稀疏盘 → Raw)。 # 时盘同款带 image_type(租户数据盘是 raw 稀疏盘 → Raw)。
self.assertEqual(payload["image_type"], "Raw") self.assertEqual(payload["image_type"], "Raw")
# 客内挂载脚本被调过(chv-data-mount)。 # p2 第三轮:attach 不再以 chv 调 chv-data-mount(那条
# 路恒因权限失败、脚本又静默 exit 0),改为轮询等
# root hotplug 路径把盘挂上:ssh 里须出现 mountpoint
# 轮询,且绝不能出现 chv-data-mount 调用。
self.assertTrue(any( self.assertTrue(any(
"/usr/local/bin/chv-data-mount" in call "mountpoint -q /home/chv/workspace" in " ".join(call)
for call in self.calls if call and call[0] == "ssh"))
self.assertFalse(any(
"/usr/local/bin/chv-data-mount" in " ".join(call)
for call in self.calls if call and call[0] == "ssh")) for call in self.calls if call and call[0] == "ssh"))
self.assertEqual( self.assertEqual(
read_vm_tenant(self.state_dir, "chv-pool-0"), "tenant-a") read_vm_tenant(self.state_dir, "chv-pool-0"), "tenant-a")
@@ -519,9 +529,9 @@ class DataDiskAttachTest(HybridACase):
self.assertEqual(len(details), 1) self.assertEqual(len(details), 1)
self.assertIn("Image type required for disk", details[0]) self.assertIn("Image type required for disk", details[0])
self.assertIn("status=400", details[0]) self.assertIn("status=400", details[0])
# 挂载脚本没跑、归属标记没写。 # 挂载轮询没跑(add-disk 已失败)、归属标记没写。
self.assertFalse(any( self.assertFalse(any(
"/usr/local/bin/chv-data-mount" in call "mountpoint -q /home/chv/workspace" in " ".join(call)
for call in self.calls if call and call[0] == "ssh")) for call in self.calls if call and call[0] == "ssh"))
self.assertIsNone(read_vm_tenant(self.state_dir, "chv-pool-0")) self.assertIsNone(read_vm_tenant(self.state_dir, "chv-pool-0"))
@@ -761,5 +771,183 @@ class DispatchWiringTest(HybridACase):
self.assertNotIn("route", out) self.assertNotIn("route", out)
# -- p2 第三轮:挂盘等待 + 卸载提权(2026-11-09 第 2 轮复测)---------
class DataMountWaitAndPrivilegedUmountTest(HybridACase):
"""覆盖第 2 轮复测的两个根因:
- 挂盘时序竞态:attach 须等客内 workspace 真挂上才返回
成功,超时 loud 失败;且绝不再以 chv 调 chv-data-mount
并判 rc==0(静默 exit 0 的成功假象)。
- 卸载权限:detach 须走 sudo -n 窄规则 umount,plain
umount 恒 rc=32 的老路不得再出现;提权失败即 False,
release 留置 VM 不回收。
只 mock 成功/失败路径的 SSH 返回码,不启动真实 VM。"""
def _poll_calls(self):
return [call for call in self.calls
if call and call[0] == "ssh"
and "mountpoint -q /home/chv/workspace"
in " ".join(call)]
def _mount_runner(self, succeed_after):
"""轮询专用 runner:前 succeed_after-1 次轮询报
未挂上(rc=1,第 2 轮真机:attach 返回时盘尚未挂),
第 succeed_after 次起报已挂上;非轮询 ssh 一律 rc=0。
succeed_after=None 即永远未挂上。"""
state = {"n": 0}
def runner(cmd, timeout=None):
self.calls.append(list(cmd))
if cmd and cmd[0] == "ssh" and any(
"mountpoint -q" in str(tok) for tok in cmd):
state["n"] += 1
if succeed_after is not None \
and state["n"] >= succeed_after:
return FakeResult(0, "", "")
return FakeResult(1, "", "")
return FakeResult(0, "", "")
return runner
def test_mount_poll_defaults(self):
self.assertEqual(DATA_MOUNT_TIMEOUT_SECONDS, 60)
self.assertEqual(DATA_MOUNT_POLL_INTERVAL_SECONDS, 2)
def test_attach_waits_for_mount_before_returning(self):
# 真机时序复现:add-disk 成功后前两次轮询盘未挂,
# 第三次才挂上——attach 必须等到第三次才返回成功,
# 而不是 add-disk 一成功就返回。
ex = self.make_exec(
runner=self._mount_runner(succeed_after=3),
env_extra={"CHV_DATA_MOUNT_TIMEOUT_SECONDS": "10",
"CHV_DATA_MOUNT_INTERVAL_SECONDS": "1"})
out = ex.attach_tenant_data_disk(
self._task(project="tenant-a"), "chv-pool-0")
self.assertEqual(out["tenant_id"], "tenant-a")
self.assertEqual(len(self._poll_calls()), 3)
self.assertEqual(
read_vm_tenant(self.state_dir, "chv-pool-0"), "tenant-a")
def test_attach_mount_timeout_fails_loud(self):
# 盘一直不挂:超时后抛 data_disk_mount_timeout(kind
# 同名),记同名审计事件,不写归属标记,不谎报成功。
ex = self.make_exec(
runner=self._mount_runner(succeed_after=None),
env_extra={"CHV_DATA_MOUNT_TIMEOUT_SECONDS": "3",
"CHV_DATA_MOUNT_INTERVAL_SECONDS": "1"})
with self.assertRaises(ChvFailure) as ctx:
ex.attach_tenant_data_disk(
self._task(project="tenant-a"), "chv-pool-0")
self.assertEqual(ctx.exception.kind, "data_disk_mount_timeout")
self.assertIn("3.0", str(ctx.exception))
self.assertEqual(len(self._poll_calls()), 3)
details = self._event_details("data_disk_mount_timeout")
self.assertEqual(len(details), 1)
self.assertIn("chv-pool-0", details[0])
self.assertIsNone(read_vm_tenant(self.state_dir, "chv-pool-0"))
def test_attach_never_runs_data_mount_as_chv(self):
# 回归:静默 exit 0 的成功假象不得再出现——attach 全程
# 不得以 chv 身份调 /usr/local/bin/chv-data-mount。
ex = self.make_exec()
ex.attach_tenant_data_disk(
self._task(project="tenant-a"), "chv-pool-0")
self.assertFalse(any(
"/usr/local/bin/chv-data-mount" in " ".join(call)
for call in self.calls if call and call[0] == "ssh"))
def test_detach_uses_privileged_umount(self):
ex = self.make_exec()
self.assertTrue(ex.detach_tenant_data_disk(
self._task(project="tenant-a"), "chv-pool-0"))
sudo_calls = [call for call in self.calls
if call and call[0] == "ssh"
and "sudo" in call]
self.assertEqual(len(sudo_calls), 1)
call = sudo_calls[0]
# 精确 argv:sudo -n /usr/bin/umount /home/chv/workspace,
# 且同一调用里不得出现不带 sudo 的 plain umount。
self.assertIn("sudo", call)
self.assertIn("-n", call)
self.assertIn("/usr/bin/umount", call)
self.assertIn("/home/chv/workspace", call)
self.assertLess(call.index("sudo"),
call.index("/usr/bin/umount"))
self.assertFalse(any(
call and call[0] == "ssh" and "sudo" not in call
and "umount" in call for call in self.calls))
def test_detach_privilege_failure_returns_false(self):
# sudo 规则缺失/镜像未重建(sudo 报 rc=1)即 False,
# 不谎报卸载成功。
def runner(cmd, timeout=None):
self.calls.append(list(cmd))
if cmd and cmd[0] == "ssh" and "sudo" in cmd:
return FakeResult(1, "", "sudo: not found")
return FakeResult(0, "", "")
ex = self.make_exec(runner=runner)
self.assertFalse(ex.detach_tenant_data_disk(
self._task(project="tenant-a"), "chv-pool-0"))
def test_release_retains_vm_when_unmount_fails(self):
# 第 2 轮根因 2 的终局:卸载失败 → release 返回
# released=False + reason=unmount_failed,记审计,
# VM 不销毁(不丢改动后静默回收)。
def runner(cmd, timeout=None):
self.calls.append(list(cmd))
if cmd and cmd[0] == "ssh" and "sudo" in cmd:
return FakeResult(32, "",
"umount: must be superuser")
return FakeResult(0, "", "")
ex = self.make_exec(runner=runner)
out = ex.release_pool_vm(
self._task(project="tenant-a"), "chv-pool-0")
self.assertFalse(out["released"])
self.assertEqual(out["reason"], "unmount_failed")
self.assertIn("warm_pool_release_failed", self._event_names())
details = self._event_details("warm_pool_release_failed")
self.assertTrue(any("unmount_failed" in d for d in details))
class DataMountGuestWiringTest(unittest.TestCase):
"""客机侧接线(纯文本断言,不启动 VM):chv-data-mount
失败 loud、build.sh 装 sudo + 写窄规则。"""
def setUp(self):
self.root = Path(__file__).resolve().parents[2]
def test_data_mount_failures_are_loud(self):
text = (self.root / "chv-guest"
/ "chv-data-mount.sh").read_text()
# 盘已找到后 mkfs/mount 失败必须 exit 1,不再静默 0。
self.assertIn('mkfs failed on $DEV"; exit 1', text)
self.assertIn('mount failed"; exit 1', text)
# 无盘仍是启动期 no-op:exit 0,不挡 boot。
self.assertIn('no data disk found, skip"; exit 0', text)
def test_build_sh_installs_sudo_and_narrow_rule(self):
text = (self.root / "chv-guest" / "build.sh").read_text()
self.assertIn("iptables sudo", text)
self.assertIn(
"chv ALL=(root) NOPASSWD: "
"/usr/bin/umount /home/chv/workspace", text)
self.assertIn("/etc/sudoers.d/chv-data-umount", text)
self.assertIn("chmod 0440 /etc/sudoers.d/chv-data-umount",
text)
self.assertIn("visudo -cf /etc/sudoers.d/chv-data-umount",
text)
# 窄规则只放行 umount 这一条:不得出现放宽到 mount、
# ALL 命令或带 shell 的写法。
idx = text.find("/etc/sudoers.d/chv-data-umount")
block = text[max(0, idx - 1500):idx + 500]
self.assertNotIn("NOPASSWD: ALL", block)
self.assertNotIn("/usr/bin/mount", block)
def test_buildinfo_lists_sudo_and_umount_rule(self):
text = (self.root / "chv-guest" / "buildinfo.py").read_text()
self.assertIn('"sudo", "chv-data-umount"', text)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+1
View File
@@ -533,6 +533,7 @@ class BuildWiringTest(unittest.TestCase):
"openssh-server 1:9.6p1-3ubuntu13", "openssh-server 1:9.6p1-3ubuntu13",
"python3 3.12.3-0ubuntu2", "python3 3.12.3-0ubuntu2",
"python3-pip 24.0+dfsg-1ubuntu1", "python3-pip 24.0+dfsg-1ubuntu1",
"sudo 1.9.15p5-3ubuntu5",
"systemd-container 255.4-1ubuntu8", "systemd-container 255.4-1ubuntu8",
"xz-utils 5.6.1+really5.4.5-1build0.1"], "xz-utils 5.6.1+really5.4.5-1build0.1"],
"pip": ["playwright==1.49.1", "pip": ["playwright==1.49.1",