fix(chv): V-7 复测缺陷复修 D1/D2 + F1 瞬时 reset 重试
D1:snapshot_vm 在配额预检后、ch-remote 前建好 file:// 目标 目录(新 helper _ensure_snapshot_dir;fresh URL 直接 snapshot 曾 rc=1 Destination is not a directory)。建目录只在 snapshot_vm 一处,hibernate 经此调用;配额拒绝仍先于任何文件改动。 D2:keep_disk=True 时 stop 保留 seed.iso(_remove_vm_seed_files 抽出;快照 config.json 的 disks 同时引用 disk.raw 与 seed.iso), 且 cleanup_stale_secrets 不清有登记快照的 VM 的 seed(与 gc_stale_disks 的盘保护同款逻辑);正常 stop 仍删两者。 F1:首条任务 SSH 撞 kex 阶段瞬时 reset(rc=255 + kex/reset 标记) 时重试恰好 1 次(_call_ssh_with_transient_retry,_run_guest 与 _run_guest_goal 共用;host key 拒绝/认证失败/远端失败不重试; 审计 vm_ssh_transient_retry)。§5“执行器内部不重试”的唯一例外, 结论写入 docs/chv-production-readiness.md §3/§5/§7。 测试:test_chv_v3_fixes 新增 12 项(D1×3、D2×4、F1×5);旧代码 上 12 项中 9 项失败(余 3 项为守卫既有行为)。全量 test_chv* 470 测试通过。
This commit is contained in:
1 parent
65504ae8cb
commit
1836c5af32
3 files changed
+387
-23
No files matched your search
+146
-21
@@ -165,6 +165,52 @@ def _assert_snapshot_dir_persisted(snapshot_url):
|
|||||||
"快照未落盘或为空目录:%s" % snapshot_url)
|
"快照未落盘或为空目录:%s" % snapshot_url)
|
||||||
|
|
||||||
|
|
||||||
|
# SSH 连接建立阶段瞬时 reset 的 stderr 标记(V-7/F1):
|
||||||
|
# kex 阶段被 reset 时 OpenSSH 报 kex_exchange_identification
|
||||||
|
# + Connection reset,远端命令根本没开始执行。只认这组
|
||||||
|
# 标记:host key 拒绝(同为 rc=255)与远端命令失败不在此列。
|
||||||
|
_SSH_TRANSIENT_RESET_MARKERS = (
|
||||||
|
"kex_exchange_identification",
|
||||||
|
"Connection reset",
|
||||||
|
"Connection closed by",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _is_transient_ssh_reset(result) -> bool:
|
||||||
|
"""单次 SSH 是否撞上连接建立阶段的瞬时 reset:returncode
|
||||||
|
255 + stderr 含 kex/reset 标记。stderr 可能是 bytes,
|
||||||
|
按文本解码后比对;无 stderr 即不认,不猜。"""
|
||||||
|
if getattr(result, "returncode", 0) != 255:
|
||||||
|
return False
|
||||||
|
stderr = getattr(result, "stderr", "") or ""
|
||||||
|
if isinstance(stderr, bytes):
|
||||||
|
stderr = stderr.decode("utf-8", "replace")
|
||||||
|
return any(marker in stderr
|
||||||
|
for marker in _SSH_TRANSIENT_RESET_MARKERS)
|
||||||
|
|
||||||
|
|
||||||
|
def _ensure_snapshot_dir(snapshot_url, snapshot_path) -> None:
|
||||||
|
"""file:// 快照目标目录建好(V-7/D1):服务端要求目标是
|
||||||
|
已存在的目录,fresh URL 直接 snapshot 会 rc=1
|
||||||
|
(Destination is not a directory),mkdir 后同一 URL 即
|
||||||
|
通。非 file:// 目标(snapshot_path 为 None)跳过。目标
|
||||||
|
已存在且不是目录、或建目录失败时报 snapshot_not_persisted
|
||||||
|
(ch-remote 之前 fail-fast,不调 ch-remote)。"""
|
||||||
|
if snapshot_path is None:
|
||||||
|
return
|
||||||
|
if os.path.exists(snapshot_path) \
|
||||||
|
and not os.path.isdir(snapshot_path):
|
||||||
|
raise ChvFailure(
|
||||||
|
"snapshot_not_persisted",
|
||||||
|
"快照目标已存在且不是目录:%s" % snapshot_url)
|
||||||
|
try:
|
||||||
|
os.makedirs(snapshot_path, exist_ok=True)
|
||||||
|
except OSError as exc:
|
||||||
|
raise ChvFailure(
|
||||||
|
"snapshot_not_persisted",
|
||||||
|
"快照目录无法创建:%s:%s" % (snapshot_url, exc))
|
||||||
|
|
||||||
|
|
||||||
def _default_popen_launcher(cmd):
|
def _default_popen_launcher(cmd):
|
||||||
"""默认 VMM 启动器:独立会话子进程,输出丢弃(纯移动)。"""
|
"""默认 VMM 启动器:独立会话子进程,输出丢弃(纯移动)。"""
|
||||||
return subprocess.Popen(
|
return subprocess.Popen(
|
||||||
@@ -1334,6 +1380,36 @@ class CloudHypervisorExecutor(Executor):
|
|||||||
return True
|
return True
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
def _call_ssh_with_transient_retry(self, task, ssh_cmd,
|
||||||
|
timeout=None):
|
||||||
|
"""任务 SSH 的连接建立阶段瞬时 reset 重试(V-7/F1,
|
||||||
|
执行器内部唯一允许的重试——§5“执行器内部不重试”
|
||||||
|
的例外,见 _run_with_vm 的注):
|
||||||
|
|
||||||
|
崩溃后重 wake 的首条任务 SSH 曾撞 1 次 kex 阶段
|
||||||
|
Connection reset(rc=255),下一条即通;与防火墙
|
||||||
|
冒烟第一轮 cloud-init 收尾重启 sshd 的瞬态故障同款
|
||||||
|
签名。wait_guest_ready 的轮询本就容忍这类抖动,但
|
||||||
|
首条任务 SSH 是单次调用:不重试就把一次秒级抖动
|
||||||
|
放大成 worker 级重试 = 整台 VM 冷启动重跑(分钟级
|
||||||
|
+ 重复计费)。故只对“连接未建立即 reset”(rc=255
|
||||||
|
+ kex/reset 标记,见 _is_transient_ssh_reset)重试
|
||||||
|
恰好 1 次,重试前 sleeper(1);第二次仍失败即原样
|
||||||
|
返回,不再追。host key 拒绝、认证失败、远端命令
|
||||||
|
失败一律不重试;误重试的最坏后果也被 host.py 兜住:
|
||||||
|
dispatch 同名任务已存在即 loud fail(task 已存在),
|
||||||
|
不会静默重跑一轮 turn。重试发 vm_ssh_transient_retry
|
||||||
|
审计。"""
|
||||||
|
result = self._call_runner(ssh_cmd, timeout=timeout)
|
||||||
|
if _is_transient_ssh_reset(result):
|
||||||
|
self._emit("vm_ssh_transient_retry", task,
|
||||||
|
"vm=%s ssh 瞬时 reset(kex 阶段),1 秒后重试 1 次"
|
||||||
|
% self.vm_name(task),
|
||||||
|
"ssh_transient_retry")
|
||||||
|
self.sleeper(1)
|
||||||
|
result = self._call_runner(ssh_cmd, timeout=timeout)
|
||||||
|
return result
|
||||||
|
|
||||||
def wait_guest_ready(self, task: dict) -> bool:
|
def wait_guest_ready(self, task: dict) -> bool:
|
||||||
"""等 guest SSH 就绪:轮询 ssh "true",returncode 0 即就绪;
|
"""等 guest SSH 就绪:轮询 ssh "true",returncode 0 即就绪;
|
||||||
每轮先尽力捕获 host key(keyscan 模式),超时先关 VM
|
每轮先尽力捕获 host key(keyscan 模式),超时先关 VM
|
||||||
@@ -1806,10 +1882,12 @@ class CloudHypervisorExecutor(Executor):
|
|||||||
feat-chv-secfix(L2/M3):ch-remote 之前先过 URL
|
feat-chv-secfix(L2/M3):ch-remote 之前先过 URL
|
||||||
白名单(file:// 须在 state_dir/CHV_SNAPSHOT_DIR 内,
|
白名单(file:// 须在 state_dir/CHV_SNAPSHOT_DIR 内,
|
||||||
远端须显式放行)与数量/总量/保留期上限;成功后登
|
远端须显式放行)与数量/总量/保留期上限;成功后登
|
||||||
记并在总量超帽时按最旧优先剪枝(删动作审计)。"""
|
记并在总量超帽时按最旧优先剪枝(删动作审计)。
|
||||||
|
feat-chv-v3-repair(V-7/D1):file:// 目标目录在
|
||||||
|
配额预检后、ch-remote 前建好(见 _ensure_snapshot_dir)。"""
|
||||||
if not snapshot_url:
|
if not snapshot_url:
|
||||||
raise ValueError("snapshot_url 不能为空")
|
raise ValueError("snapshot_url 不能为空")
|
||||||
validate_snapshot_url(
|
snapshot_path = validate_snapshot_url(
|
||||||
snapshot_url, self.state_dir,
|
snapshot_url, self.state_dir,
|
||||||
snapshot_dir=getattr(self, "snapshot_dir", None),
|
snapshot_dir=getattr(self, "snapshot_dir", None),
|
||||||
allow_remote=getattr(self, "allow_remote_snapshot", False))
|
allow_remote=getattr(self, "allow_remote_snapshot", False))
|
||||||
@@ -1819,6 +1897,11 @@ class CloudHypervisorExecutor(Executor):
|
|||||||
getattr(self, "snapshot_max_count", 10),
|
getattr(self, "snapshot_max_count", 10),
|
||||||
getattr(self, "snapshot_max_total_gb", 20.0),
|
getattr(self, "snapshot_max_total_gb", 20.0),
|
||||||
getattr(self, "snapshot_retention_days", 30.0))
|
getattr(self, "snapshot_retention_days", 30.0))
|
||||||
|
# feat-chv-v3-repair(V-7/D1):建目录放在配额预检
|
||||||
|
# 之后——配额拒绝先于任何文件改动,且目标是文件时
|
||||||
|
# 也不抢占配额错误的 kind;只在 snapshot_vm 这一处
|
||||||
|
# 建,hibernate_vm 经此调用,不重不漏。
|
||||||
|
_ensure_snapshot_dir(snapshot_url, snapshot_path)
|
||||||
self._ch_remote(task_or_name, "snapshot", snapshot_url)
|
self._ch_remote(task_or_name, "snapshot", snapshot_url)
|
||||||
# feat-chv-fix(P1-7)/feat-chv-v3:只查 returncode
|
# feat-chv-fix(P1-7)/feat-chv-v3:只查 returncode
|
||||||
# 不够,见 _assert_snapshot_dir_persisted——file://
|
# 不够,见 _assert_snapshot_dir_persisted——file://
|
||||||
@@ -1827,6 +1910,13 @@ class CloudHypervisorExecutor(Executor):
|
|||||||
# 不 shutdown、不销毁原 VM;非 file:// 目标无法从宿
|
# 不 shutdown、不销毁原 VM;非 file:// 目标无法从宿
|
||||||
# 主验证落盘,仍只认 returncode,但文档明写该边界。
|
# 主验证落盘,仍只认 returncode,但文档明写该边界。
|
||||||
_assert_snapshot_dir_persisted(snapshot_url)
|
_assert_snapshot_dir_persisted(snapshot_url)
|
||||||
|
return self._record_snapshot_and_emit(
|
||||||
|
task_or_name, vm_name, snapshot_url)
|
||||||
|
|
||||||
|
def _record_snapshot_and_emit(self, task_or_name, vm_name,
|
||||||
|
snapshot_url):
|
||||||
|
"""快照成功后登记 + 总量超帽时最旧优先剪枝(删动作
|
||||||
|
审计)+ vm_snapshot 审计,返回 snapshot_url。"""
|
||||||
task = task_or_name if isinstance(task_or_name, dict) \
|
task = task_or_name if isinstance(task_or_name, dict) \
|
||||||
else {"id": task_or_name}
|
else {"id": task_or_name}
|
||||||
_entries, _removed = record_snapshot(
|
_entries, _removed = record_snapshot(
|
||||||
@@ -2010,7 +2100,14 @@ class CloudHypervisorExecutor(Executor):
|
|||||||
留下的盘由 gc_stale_disks 的快照登记保护(有登记
|
留下的盘由 gc_stale_disks 的快照登记保护(有登记
|
||||||
快照的 VM 不 GC 盘),快照被剪干净后盘才成孤儿可删。
|
快照的 VM 不 GC 盘),快照被剪干净后盘才成孤儿可删。
|
||||||
restore 成功后 VM 再走正常 stop,盘照常删,不留
|
restore 成功后 VM 再走正常 stop,盘照常删,不留
|
||||||
长期残留。"""
|
长期残留。
|
||||||
|
feat-chv-v3-repair(V-7/D2):keep_disk=True 同样留
|
||||||
|
seed.iso——config.json 的 disks 同时引用 disk.raw 与
|
||||||
|
seed.iso(readonly),只留盘不留 seed.iso 时 restore
|
||||||
|
仍在服务端失败(Cannot open disk path …/seed.iso),
|
||||||
|
预检 snapshot_disk_missing 先拒掉不是过严;有登记
|
||||||
|
快照的 VM 的 seed.iso 也不被 cleanup_stale_secrets
|
||||||
|
当孤儿清掉(见 _clean_one_vm_secrets)。"""
|
||||||
self.pause_vm(task_or_name)
|
self.pause_vm(task_or_name)
|
||||||
self.snapshot_vm(task_or_name, snapshot_url)
|
self.snapshot_vm(task_or_name, snapshot_url)
|
||||||
self.stop_vm(task_or_name, keep_disk=True)
|
self.stop_vm(task_or_name, keep_disk=True)
|
||||||
@@ -2038,26 +2135,41 @@ class CloudHypervisorExecutor(Executor):
|
|||||||
if alive:
|
if alive:
|
||||||
self._kill_proc(proc)
|
self._kill_proc(proc)
|
||||||
|
|
||||||
def _cleanup_vm_files(self, name, task, keep_disk=False):
|
def _remove_vm_seed_files(self, name, keep_disk):
|
||||||
"""stop 的三块文件清理:seed/私钥、任务磁盘、net 登记。
|
"""删 seed 目录 + 宿主私钥 + seed.iso(stop 的第一块
|
||||||
|
清理,见 _cleanup_vm_files):keep_disk=True 时 seed.iso
|
||||||
任一块失败记对应审计并返回 True(脏标记由调用方写);
|
留——快照 config.json 的 disks 含 seed.iso(readonly),
|
||||||
seed/私钥与磁盘删除的审计名与文案逐字不变。
|
删了 restore 必在服务端失败(Cannot open disk path
|
||||||
keep_disk=True 时跳过任务盘删除(hibernate 专用:
|
…/seed.iso / No such file or directory),预检
|
||||||
快照 config.json 引用原盘路径,盘删了 restore 即
|
snapshot_disk_missing 不是过严(V-7/D2)。seed 目录
|
||||||
失败——见 hibernate_vm 的评估注释)。"""
|
与宿主私钥仍删,config.json 不引用它们。"""
|
||||||
from cloud.chv.hostkeys import dirty_marker_path # noqa: F401
|
|
||||||
failed = False
|
|
||||||
try:
|
|
||||||
seed_dir = os.path.join(self.state_dir, name, "seed")
|
seed_dir = os.path.join(self.state_dir, name, "seed")
|
||||||
if os.path.isdir(seed_dir):
|
if os.path.isdir(seed_dir):
|
||||||
shutil.rmtree(seed_dir)
|
shutil.rmtree(seed_dir)
|
||||||
for _suffix in ("ssh_host_ed25519_key",
|
for _suffix in ("ssh_host_ed25519_key",
|
||||||
"ssh_host_ed25519_key.pub",
|
"ssh_host_ed25519_key.pub",
|
||||||
"seed.iso"):
|
"seed.iso"):
|
||||||
|
if keep_disk and _suffix == "seed.iso":
|
||||||
|
continue
|
||||||
_p = os.path.join(self.state_dir, name, _suffix)
|
_p = os.path.join(self.state_dir, name, _suffix)
|
||||||
if os.path.exists(_p):
|
if os.path.exists(_p):
|
||||||
os.remove(_p)
|
os.remove(_p)
|
||||||
|
|
||||||
|
def _cleanup_vm_files(self, name, task, keep_disk=False):
|
||||||
|
"""stop 的三块文件清理:seed/私钥、任务磁盘、net 登记。
|
||||||
|
|
||||||
|
任一块失败记对应审计并返回 True(脏标记由调用方写);
|
||||||
|
seed/私钥与磁盘删除的审计名与文案逐字不变。
|
||||||
|
keep_disk=True 时跳过任务盘删除,并跳过 seed.iso
|
||||||
|
删除(hibernate 专用:快照 config.json 引用原盘路径
|
||||||
|
与 seed.iso,任一删了 restore 即失败——见
|
||||||
|
hibernate_vm 的评估注释与 V-7/D2;seed 目录与宿主
|
||||||
|
私钥仍删,config.json 不引用它们,私钥明文已嵌进
|
||||||
|
seed.iso、宿主侧不留)。"""
|
||||||
|
from cloud.chv.hostkeys import dirty_marker_path # noqa: F401
|
||||||
|
failed = False
|
||||||
|
try:
|
||||||
|
self._remove_vm_seed_files(name, keep_disk)
|
||||||
except Exception as _cleanup_exc:
|
except Exception as _cleanup_exc:
|
||||||
failed = True
|
failed = True
|
||||||
self._emit("vm_key_cleanup_failed", task,
|
self._emit("vm_key_cleanup_failed", task,
|
||||||
@@ -2127,7 +2239,9 @@ class CloudHypervisorExecutor(Executor):
|
|||||||
feat-chv-v3(F2):新实例的 _vms 里无登记时,旧 VMM
|
feat-chv-v3(F2):新实例的 _vms 里无登记时,旧 VMM
|
||||||
也要按 socket 进程扫描清掉(_kill_vmm_by_socket),
|
也要按 socket 进程扫描清掉(_kill_vmm_by_socket),
|
||||||
否则 socket/磁盘已删、VMM 空壳进程仍活着占 tap。
|
否则 socket/磁盘已删、VMM 空壳进程仍活着占 tap。
|
||||||
keep_disk=True 时不删任务盘(hibernate 专用)。"""
|
keep_disk=True 时不删任务盘与 seed.iso(hibernate 专用,
|
||||||
|
见 hibernate_vm:快照 config.json 的 disks 同时引用
|
||||||
|
两者)。"""
|
||||||
name = self.vm_name(task_or_name)
|
name = self.vm_name(task_or_name)
|
||||||
socket_path = os.path.join(self.state_dir, name + ".sock")
|
socket_path = os.path.join(self.state_dir, name + ".sock")
|
||||||
with self._vms_lock:
|
with self._vms_lock:
|
||||||
@@ -2221,7 +2335,11 @@ class CloudHypervisorExecutor(Executor):
|
|||||||
def _clean_one_vm_secrets(self, entry, vm_dir):
|
def _clean_one_vm_secrets(self, entry, vm_dir):
|
||||||
"""单 VM 残留清理:私钥一律删(live VM 私钥在 seed 盘
|
"""单 VM 残留清理:私钥一律删(live VM 私钥在 seed 盘
|
||||||
建好后即删,残留只可能是崩溃遗留);非 live VM 的
|
建好后即删,残留只可能是崩溃遗留);非 live VM 的
|
||||||
seed.iso/seed 目录同理删。返回 (删私钥, 删 seed)。"""
|
seed.iso/seed 目录同理删,但有登记快照的 VM 除外
|
||||||
|
(feat-chv-v3-repair,V-7/D2:那是休眠 VM 的 restore
|
||||||
|
输入,seed.iso 被快照 config.json 的 disks 引用,与
|
||||||
|
gc_stale_disks 的盘保护同款逻辑;私钥仍删,restore
|
||||||
|
不读宿主私钥文件)。返回 (删私钥, 删 seed)。"""
|
||||||
removed_priv = False
|
removed_priv = False
|
||||||
priv = os.path.join(vm_dir, "ssh_host_ed25519_key")
|
priv = os.path.join(vm_dir, "ssh_host_ed25519_key")
|
||||||
if os.path.exists(priv):
|
if os.path.exists(priv):
|
||||||
@@ -2231,7 +2349,8 @@ class CloudHypervisorExecutor(Executor):
|
|||||||
except Exception:
|
except Exception:
|
||||||
pass
|
pass
|
||||||
removed_seed = False
|
removed_seed = False
|
||||||
if not self._is_live_vm(entry):
|
if not self._is_live_vm(entry) \
|
||||||
|
and not has_recorded_snapshots(self.state_dir, entry):
|
||||||
for _suffix in ("seed.iso", "seed"):
|
for _suffix in ("seed.iso", "seed"):
|
||||||
_p = os.path.join(vm_dir, _suffix)
|
_p = os.path.join(vm_dir, _suffix)
|
||||||
try:
|
try:
|
||||||
@@ -2247,7 +2366,9 @@ class CloudHypervisorExecutor(Executor):
|
|||||||
|
|
||||||
def cleanup_stale_secrets(self) -> list:
|
def cleanup_stale_secrets(self) -> list:
|
||||||
"""崩溃残留扫描(H5):私钥一律删、非 live VM 的 seed
|
"""崩溃残留扫描(H5):私钥一律删、非 live VM 的 seed
|
||||||
同理删并审计;返回已清理 VM 名。worker 启动与
|
同理删并审计;有登记快照的 VM 的 seed 不删(V-7/D2,
|
||||||
|
休眠 VM 的 restore 输入,见 _clean_one_vm_secrets);
|
||||||
|
返回已清理 VM 名。worker 启动与
|
||||||
cleanup_stale_vms 时调它,不依赖 socket 是否存在。"""
|
cleanup_stale_vms 时调它,不依赖 socket 是否存在。"""
|
||||||
cleaned: list = []
|
cleaned: list = []
|
||||||
try:
|
try:
|
||||||
@@ -2368,7 +2489,8 @@ class CloudHypervisorExecutor(Executor):
|
|||||||
ssh_cmd = self.build_ssh_command(
|
ssh_cmd = self.build_ssh_command(
|
||||||
task, self._guest_host_command(*inner_args, **kwargs),
|
task, self._guest_host_command(*inner_args, **kwargs),
|
||||||
extra_env=extra_env)
|
extra_env=extra_env)
|
||||||
result = self._call_runner(ssh_cmd, timeout=self.task_timeout)
|
result = self._call_ssh_with_transient_retry(
|
||||||
|
task, ssh_cmd, timeout=self.task_timeout)
|
||||||
# 方法内 import:见模块头注释的循环导入说明。
|
# 方法内 import:见模块头注释的循环导入说明。
|
||||||
from cloud.sandbox import SimpleNamespaceShim, _parse_host_result
|
from cloud.sandbox import SimpleNamespaceShim, _parse_host_result
|
||||||
parsed = _parse_host_result(task["id"],
|
parsed = _parse_host_result(task["id"],
|
||||||
@@ -2403,7 +2525,8 @@ class CloudHypervisorExecutor(Executor):
|
|||||||
extra_env=extra_env)
|
extra_env=extra_env)
|
||||||
return self._goal_result(
|
return self._goal_result(
|
||||||
task["id"],
|
task["id"],
|
||||||
self._call_runner(ssh_cmd, timeout=self.task_timeout))
|
self._call_ssh_with_transient_retry(
|
||||||
|
task, ssh_cmd, timeout=self.task_timeout))
|
||||||
|
|
||||||
def _vm_proc_dead(self, task_or_name):
|
def _vm_proc_dead(self, task_or_name):
|
||||||
"""该 VM 的登记进程是否已死:只认本实例登记的进程
|
"""该 VM 的登记进程是否已死:只认本实例登记的进程
|
||||||
@@ -2481,7 +2604,9 @@ class CloudHypervisorExecutor(Executor):
|
|||||||
wake 失败时不跑
|
wake 失败时不跑
|
||||||
任务、直接抛分类失败。本方法不做内部重试:重试是
|
任务、直接抛分类失败。本方法不做内部重试:重试是
|
||||||
worker 的 max_retries 语义,执行器内部重试会把一次
|
worker 的 max_retries 语义,执行器内部重试会把一次
|
||||||
失败放大成多次计费与多次 VM 启动。
|
失败放大成多次计费与多次 VM 启动。唯一例外:首条任务 SSH
|
||||||
|
的连接建立阶段瞬时 reset 由 _run_guest 经
|
||||||
|
_call_ssh_with_transient_retry 重试 1 次(传输抖动,非任务重试)。
|
||||||
|
|
||||||
中途死亡分类(本轨新增):客内命令抛传输异常、或
|
中途死亡分类(本轨新增):客内命令抛传输异常、或
|
||||||
非 goal 结果已 failed 时,若登记的 VMM 进程已死,
|
非 goal 结果已 failed 时,若登记的 VMM 进程已死,
|
||||||
|
|||||||
@@ -13,6 +13,11 @@
|
|||||||
guest-env 归 SSH 用户所有;
|
guest-env 归 SSH 用户所有;
|
||||||
- V-1 附带:build.sh 预建 /app/work(1777 粘滞位 + 断言);
|
- V-1 附带:build.sh 预建 /app/work(1777 粘滞位 + 断言);
|
||||||
- V-1 附带:host.py default_codex_bin() 走 PATH 查找。
|
- V-1 附带:host.py default_codex_bin() 走 PATH 查找。
|
||||||
|
- V-7 复修(feat-chv-v3-repair):D1 snapshot 先建目录(旧
|
||||||
|
fixture 预建目录 + mock ch-remote 不校验,458 测试全过也
|
||||||
|
盖不住缺失目录的真实路径)、D2 hibernate 留 seed.iso +
|
||||||
|
残留清理保护有登记快照的 VM 的 seed、F1 任务 SSH 连接建立
|
||||||
|
阶段瞬时 reset 重试恰好 1 次。
|
||||||
|
|
||||||
全部用假 runner/launcher/api_client + 真实 /proc 扫描(F2
|
全部用假 runner/launcher/api_client + 真实 /proc 扫描(F2
|
||||||
的被杀进程是真实 spawn 的 sleep 进程,argv[0] 伪装成
|
的被杀进程是真实 spawn 的 sleep 进程,argv[0] 伪装成
|
||||||
@@ -545,5 +550,217 @@ class GuestBuildAndHostBinV3Test(unittest.TestCase):
|
|||||||
"/opt/custom/codex")
|
"/opt/custom/codex")
|
||||||
|
|
||||||
|
|
||||||
|
# -- V-7/D1:snapshot 先建目标目录 ---------------------------------------
|
||||||
|
class SnapshotMkdirV7Test(V3Case):
|
||||||
|
def _snapshot_creating_runner(self):
|
||||||
|
"""模拟真实 ch-remote:往目标目录写快照文件。目录不
|
||||||
|
存在时 open 即 FileNotFoundError——若执行器没先
|
||||||
|
mkdir,测试直接失败,这正是旧 fixture 盖不住的点
|
||||||
|
(旧 fixture 预建目录,mock 又不写文件)。"""
|
||||||
|
def runner(cmd, timeout=None):
|
||||||
|
if cmd and cmd[0] == self.ch_remote and "snapshot" in cmd:
|
||||||
|
self.calls.append(list(cmd))
|
||||||
|
target = cmd[-1][len("file://"):]
|
||||||
|
for fname, content in (
|
||||||
|
("config.json", b"{}"),
|
||||||
|
("state.json", b"{}"),
|
||||||
|
("memory-ranges", b"m")):
|
||||||
|
with open(os.path.join(target, fname), "wb") as fh:
|
||||||
|
fh.write(content)
|
||||||
|
return FakeResult(0, "", "")
|
||||||
|
return self._default_runner(cmd, timeout)
|
||||||
|
return runner
|
||||||
|
|
||||||
|
def test_snapshot_creates_missing_snapshot_dir(self):
|
||||||
|
ex = self.make_exec(runner=self._snapshot_creating_runner())
|
||||||
|
url = "file://" + os.path.join(self.state_dir, "fresh-snap")
|
||||||
|
self.assertFalse(os.path.exists(url[len("file://"):]))
|
||||||
|
out = ex.snapshot_vm(self._task(), url)
|
||||||
|
self.assertEqual(out, url)
|
||||||
|
snap_path = url[len("file://"):]
|
||||||
|
self.assertTrue(os.path.isdir(snap_path))
|
||||||
|
self.assertIn("config.json", os.listdir(snap_path))
|
||||||
|
with open(os.path.join(self.state_dir, "chv-abc123",
|
||||||
|
"snapshots.json")) as fh:
|
||||||
|
entries = json.load(fh)
|
||||||
|
self.assertEqual(entries[0]["path"], snap_path)
|
||||||
|
|
||||||
|
def test_hibernate_creates_missing_snapshot_dir(self):
|
||||||
|
ex = self.make_exec(runner=self._snapshot_creating_runner())
|
||||||
|
ex.wake_vm(self._task())
|
||||||
|
url = "file://" + os.path.join(self.state_dir, "fresh-snap")
|
||||||
|
out = ex.hibernate_vm(self._task(), url)
|
||||||
|
self.assertEqual(out["state"], "hibernated")
|
||||||
|
self.assertTrue(os.path.isdir(url[len("file://"):]))
|
||||||
|
|
||||||
|
def test_snapshot_target_existing_file_rejected(self):
|
||||||
|
ex = self.make_exec()
|
||||||
|
target = os.path.join(self.state_dir, "not-a-dir")
|
||||||
|
os.makedirs(self.state_dir, exist_ok=True)
|
||||||
|
with open(target, "wb") as fh:
|
||||||
|
fh.write(b"x")
|
||||||
|
with self.assertRaises(ChvFailure) as ctx:
|
||||||
|
ex.snapshot_vm(self._task(), "file://" + target)
|
||||||
|
self.assertEqual(ctx.exception.kind, "snapshot_not_persisted")
|
||||||
|
self.assertEqual(self.calls, [],
|
||||||
|
"目标不是目录时 ch-remote 根本不应被调")
|
||||||
|
|
||||||
|
|
||||||
|
# -- V-7/D2:hibernate 留 seed.iso ----------------------------------------
|
||||||
|
class HibernateSeedIsoV7Test(V3Case):
|
||||||
|
def _prov_exec(self):
|
||||||
|
# keyscan 模式不制 seed.iso;provisioned 模式 wake 即
|
||||||
|
# 建 seed.iso(真机休眠的形态),用它走全链路。
|
||||||
|
return self.make_exec(
|
||||||
|
env_extra={"CHV_HOST_KEY_MODE": "provisioned"})
|
||||||
|
|
||||||
|
def _seed_iso(self):
|
||||||
|
return os.path.join(self.state_dir, "chv-abc123", "seed.iso")
|
||||||
|
|
||||||
|
def _disk(self):
|
||||||
|
return os.path.join(self.state_dir, "chv-abc123", "disk.raw")
|
||||||
|
|
||||||
|
def _snap_with_disks(self):
|
||||||
|
"""快照 config.json 同时引用 disk.raw 与 seed.iso
|
||||||
|
(真机快照的 disks 形态)。"""
|
||||||
|
return self._snap_dir(files={
|
||||||
|
"config.json": json.dumps(
|
||||||
|
{"disks": [{"path": self._disk()},
|
||||||
|
{"path": self._seed_iso()}]}).encode(),
|
||||||
|
"state.json": b"{}"})
|
||||||
|
|
||||||
|
def test_hibernate_keeps_seed_iso(self):
|
||||||
|
ex = self._prov_exec()
|
||||||
|
ex.wake_vm(self._task())
|
||||||
|
self.assertTrue(os.path.exists(self._seed_iso()))
|
||||||
|
url, _ = self._snap_dir()
|
||||||
|
out = ex.hibernate_vm(self._task(), url)
|
||||||
|
self.assertEqual(out["state"], "hibernated")
|
||||||
|
self.assertTrue(os.path.exists(self._disk()),
|
||||||
|
"hibernate 仍须留 disk.raw")
|
||||||
|
self.assertTrue(os.path.exists(self._seed_iso()),
|
||||||
|
"hibernate 须留 seed.iso:快照 config.json "
|
||||||
|
"的 disks 引用它,删了 restore 即失败")
|
||||||
|
# 只留 iso:seed 目录(制盘输入)与宿主私钥仍删,
|
||||||
|
# config.json 不引用它们,私钥明文不留宿主。
|
||||||
|
self.assertFalse(os.path.exists(
|
||||||
|
os.path.join(self.state_dir, "chv-abc123", "seed")))
|
||||||
|
self.assertFalse(os.path.exists(
|
||||||
|
os.path.join(self.state_dir, "chv-abc123",
|
||||||
|
"ssh_host_ed25519_key")))
|
||||||
|
|
||||||
|
def test_stop_without_keep_disk_removes_seed_iso(self):
|
||||||
|
ex = self._prov_exec()
|
||||||
|
ex.wake_vm(self._task())
|
||||||
|
self.assertTrue(ex.stop_vm(self._task()))
|
||||||
|
self.assertFalse(os.path.exists(self._seed_iso()),
|
||||||
|
"正常 stop 仍须删 seed.iso:它含宿主私钥"
|
||||||
|
"明文,不得随 VM 残留")
|
||||||
|
self.assertFalse(os.path.exists(self._disk()))
|
||||||
|
|
||||||
|
def test_cleanup_stale_secrets_spares_seed_iso_while_snapshot_recorded(self):
|
||||||
|
ex = self._prov_exec()
|
||||||
|
ex.wake_vm(self._task())
|
||||||
|
url, _ = self._snap_dir()
|
||||||
|
ex.hibernate_vm(self._task(), url)
|
||||||
|
cleaned = ex.cleanup_stale_secrets()
|
||||||
|
self.assertNotIn("chv-abc123", cleaned)
|
||||||
|
self.assertTrue(os.path.exists(self._seed_iso()),
|
||||||
|
"有登记快照的 VM 的 seed.iso 不是孤儿,"
|
||||||
|
"残留清理不得清掉(与 gc_stale_disks 的"
|
||||||
|
"盘保护同款逻辑)")
|
||||||
|
# 快照登记清掉后,seed.iso 成孤儿,才可清。
|
||||||
|
os.remove(os.path.join(self.state_dir, "chv-abc123",
|
||||||
|
"snapshots.json"))
|
||||||
|
cleaned = ex.cleanup_stale_secrets()
|
||||||
|
self.assertIn("chv-abc123", cleaned)
|
||||||
|
self.assertFalse(os.path.exists(self._seed_iso()))
|
||||||
|
|
||||||
|
def test_restore_after_hibernate_with_seed_iso_passes_precheck(self):
|
||||||
|
ex = self._prov_exec()
|
||||||
|
ex.wake_vm(self._task())
|
||||||
|
url, _ = self._snap_with_disks()
|
||||||
|
ex.hibernate_vm(self._task(), url)
|
||||||
|
out = ex.restore_vm(self._task(), url)
|
||||||
|
self.assertEqual(out["mode"], "restored")
|
||||||
|
|
||||||
|
|
||||||
|
# -- V-7/F1:任务 SSH 连接建立阶段瞬时 reset 重试 1 次 --------------------
|
||||||
|
class SshTransientRetryV7Test(V3Case):
|
||||||
|
def _dispatch_runner(self, first, second=None):
|
||||||
|
"""首条 dispatch SSH 返回 first,之后返回 second
|
||||||
|
(默认成功)。只记 dispatch SSH 的调用次数。"""
|
||||||
|
state = {"n": 0}
|
||||||
|
|
||||||
|
def runner(cmd, timeout=None):
|
||||||
|
if cmd and cmd[0] == "ssh" and "dispatch" in cmd:
|
||||||
|
state["n"] += 1
|
||||||
|
if state["n"] == 1:
|
||||||
|
return first
|
||||||
|
return second or FakeResult(
|
||||||
|
0, json.dumps({"status": "completed",
|
||||||
|
"text": "ok"}), "")
|
||||||
|
return self._default_runner(cmd, timeout)
|
||||||
|
runner.state = state
|
||||||
|
return runner
|
||||||
|
|
||||||
|
def test_dispatch_retries_once_on_kex_reset_then_succeeds(self):
|
||||||
|
runner = self._dispatch_runner(FakeResult(
|
||||||
|
255, "",
|
||||||
|
"kex_exchange_identification: Connection reset by peer"))
|
||||||
|
ex = self.make_exec(runner=runner)
|
||||||
|
out = ex.dispatch_task(self._task())
|
||||||
|
self.assertEqual(out["status"], "completed")
|
||||||
|
self.assertEqual(runner.state["n"], 2,
|
||||||
|
"kex 阶段瞬时 reset 只重试恰好 1 次")
|
||||||
|
self.assertTrue(any(e == "vm_ssh_transient_retry"
|
||||||
|
for e, _t, _d, _k in self.audit))
|
||||||
|
|
||||||
|
def test_dispatch_does_not_retry_on_host_key_mismatch(self):
|
||||||
|
runner = self._dispatch_runner(FakeResult(
|
||||||
|
255, "", "Host key verification failed."))
|
||||||
|
ex = self.make_exec(runner=runner)
|
||||||
|
out = ex.dispatch_task(self._task())
|
||||||
|
self.assertEqual(out["status"], "failed")
|
||||||
|
self.assertEqual(runner.state["n"], 1,
|
||||||
|
"host key 拒绝(同为 255)绝不重试")
|
||||||
|
self.assertFalse(any(e == "vm_ssh_transient_retry"
|
||||||
|
for e, _t, _d, _k in self.audit))
|
||||||
|
|
||||||
|
def test_dispatch_does_not_retry_on_remote_failure(self):
|
||||||
|
runner = self._dispatch_runner(FakeResult(
|
||||||
|
1, json.dumps({"status": "failed", "text": "boom"}), ""))
|
||||||
|
ex = self.make_exec(runner=runner)
|
||||||
|
out = ex.dispatch_task(self._task())
|
||||||
|
self.assertEqual(out["status"], "failed")
|
||||||
|
self.assertEqual(runner.state["n"], 1, "远端命令失败不重试")
|
||||||
|
|
||||||
|
def test_dispatch_retries_at_most_once(self):
|
||||||
|
runner = self._dispatch_runner(
|
||||||
|
FakeResult(255, "", "Connection reset by peer"),
|
||||||
|
second=FakeResult(255, "", "Connection reset by peer"))
|
||||||
|
ex = self.make_exec(runner=runner)
|
||||||
|
out = ex.dispatch_task(self._task())
|
||||||
|
self.assertEqual(out["status"], "failed")
|
||||||
|
self.assertEqual(runner.state["n"], 2, "第二次失败即停,不再追")
|
||||||
|
|
||||||
|
def test_transient_retry_helper_retries_once(self):
|
||||||
|
"""helper 本身(_run_guest 与 _run_guest_goal 共用):
|
||||||
|
首条 255 + reset 标记即重试 1 次,第二次成功即返回。"""
|
||||||
|
calls = []
|
||||||
|
|
||||||
|
def runner(cmd, timeout=None):
|
||||||
|
calls.append(list(cmd))
|
||||||
|
if len(calls) == 1:
|
||||||
|
return FakeResult(255, "", "Connection reset by peer")
|
||||||
|
return FakeResult(0, "ok", "")
|
||||||
|
|
||||||
|
ex = self.make_exec(runner=runner)
|
||||||
|
result = ex._call_ssh_with_transient_retry(
|
||||||
|
self._task(), ["ssh", "true"])
|
||||||
|
self.assertEqual(result.returncode, 0)
|
||||||
|
self.assertEqual(len(calls), 2)
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
unittest.main()
|
unittest.main()
|
||||||
@@ -98,7 +98,18 @@
|
|||||||
SSH 就绪),恢复后默认 paused 的坑已在代码与测试里钉住。
|
SSH 就绪),恢复后默认 paused 的坑已在代码与测试里钉住。
|
||||||
- ✅🧪 hibernate:CHV 无原生 hibernate 端点,本轨用 pause +
|
- ✅🧪 hibernate:CHV 无原生 hibernate 端点,本轨用 pause +
|
||||||
snapshot + shutdown 等价实现,文档与代码双写“等价实现”,
|
snapshot + shutdown 等价实现,文档与代码双写“等价实现”,
|
||||||
不声称接了原生端点;唤醒走 restore_vm。
|
不声称接了原生端点;唤醒走 restore_vm。hibernate 的 stop
|
||||||
|
用 keep_disk=True:disk.raw 与 seed.iso 都留(快照
|
||||||
|
config.json 的 disks 同时引用两者,删其一 restore 即在
|
||||||
|
服务端失败,见 V-7/D2);有登记快照的 VM 的盘与 seed.iso
|
||||||
|
也不被 gc_stale_disks / cleanup_stale_secrets 当孤儿清掉,
|
||||||
|
登记剪干净后才成孤儿可删;正常 stop(keep_disk=False)
|
||||||
|
仍删两者(seed.iso 含宿主私钥明文,不得残留)。
|
||||||
|
- ✅🧪 snapshot 目标目录:snapshot_vm 在 ch-remote 之前建好
|
||||||
|
file:// 目标目录(V-7/D1):服务端要求目标是已存在的
|
||||||
|
目录,fresh URL 直接 snapshot 会 rc=1(Destination is not
|
||||||
|
a directory),mkdir 后即通;目标已存在且不是目录时报
|
||||||
|
snapshot_not_persisted,不调 ch-remote。
|
||||||
- ✅🧪 shutdown-destroy / recycle:shutdown_vm(ch-remote
|
- ✅🧪 shutdown-destroy / recycle:shutdown_vm(ch-remote
|
||||||
shutdown + 杀进程 + 删 socket)、recycle_vm(保留
|
shutdown + 杀进程 + 删 socket)、recycle_vm(保留
|
||||||
task_checkpoints/task_attempts,丢失 guest /tmp 与未拷出
|
task_checkpoints/task_attempts,丢失 guest /tmp 与未拷出
|
||||||
@@ -149,6 +160,17 @@
|
|||||||
- ✅ 超时与最大轮次:SSH 单次超时(CHV_TASK_TIMEOUT)+
|
- ✅ 超时与最大轮次:SSH 单次超时(CHV_TASK_TIMEOUT)+
|
||||||
worker task_timeout 外层护栏;goal 最大轮次 100(与 Local
|
worker task_timeout 外层护栏;goal 最大轮次 100(与 Local
|
||||||
同款),执行器内部不重试(重试是 worker max_retries 语义)。
|
同款),执行器内部不重试(重试是 worker max_retries 语义)。
|
||||||
|
唯一例外(V-7/F1,评估结论:值得改,已改):首条任务 SSH
|
||||||
|
撞连接建立阶段瞬时 reset(rc=255 + kex/reset 标记,崩溃后
|
||||||
|
重 wake 实测 1 次、下一条即通,与 cloud-init 收尾重启 sshd
|
||||||
|
的瞬态故障同款签名)时,_run_guest / _run_guest_goal 经
|
||||||
|
_call_ssh_with_transient_retry 重试恰好 1 次(重试前等 1
|
||||||
|
秒,发 vm_ssh_transient_retry 审计)。wait_guest_ready 的
|
||||||
|
轮询本就容忍这类抖动,缺口只在单次调用的首条任务 SSH:
|
||||||
|
不重试会把秒级抖动放大成 worker 级重试 = 整台 VM 冷启动
|
||||||
|
重跑(分钟级 + 重复计费)。host key 拒绝、认证失败、远端
|
||||||
|
命令失败一律不重试;误重试的最坏后果也被 host.py 兜住
|
||||||
|
(dispatch 同名任务已存在即 loud fail,不静默重跑一轮)。
|
||||||
- ✅ 成本复用:不另起一套计费;worker 现有 quota/成本语义
|
- ✅ 成本复用:不另起一套计费;worker 现有 quota/成本语义
|
||||||
(per-user USD 配额、quota 看门狗、成本写库)仍是唯一口径,
|
(per-user USD 配额、quota 看门狗、成本写库)仍是唯一口径,
|
||||||
执行器只把 host.py 用量原样返回。
|
执行器只把 host.py 用量原样返回。
|
||||||
@@ -190,7 +212,7 @@
|
|||||||
release 均经 audit_callback 发出;worker 接线后转 AuthStore
|
release 均经 audit_callback 发出;worker 接线后转 AuthStore
|
||||||
审计(_audit 本身 best-effort)。feat-chv-secfix 起补齐:
|
审计(_audit 本身 best-effort)。feat-chv-secfix 起补齐:
|
||||||
host_key_generated/host_key_captured/host_key_mismatch、
|
host_key_generated/host_key_captured/host_key_mismatch、
|
||||||
preflight_failed、wake 前置失败(容量/配额/net 冲突)、
|
vm_ssh_transient_retry、preflight_failed、wake 前置失败(容量/配额/net 冲突)、
|
||||||
vm_disk_deleted/vm_disk_delete_failed/vm_disk_gc、
|
vm_disk_deleted/vm_disk_delete_failed/vm_disk_gc、
|
||||||
host_secret_residue_cleaned、net_config_dual、快照剪枝;
|
host_secret_residue_cleaned、net_config_dual、快照剪枝;
|
||||||
审计回调失败记宿主 error 日志,不再双重静默。
|
审计回调失败记宿主 error 日志,不再双重静默。
|
||||||
|
|||||||
Reference in new issue
Block a user