feat(chv-muse-p1): 常驻 VM + WarmPool 扶正 + 气球 + vsock,对齐 Muse
每用户一台常驻 VM(暂停态,resume 到 SSH 5 秒内为待验证目标), 池空/失败回落冷启动;CHV 配置加 virtio-balloon + free-page-reporting 与 vsock notify 最小实现(SSH 仍是命令通道与就绪判定)。 新模块 resident_pool.py / vsock.py + 客内 vsock_notify.py; _config 增加 CHV_RESIDENT_IDLE_MINUTES / CHV_BALLOON / CHV_VSOCK; worker_adapter 接常驻孤儿清理 facade;ADR-17 差距表见 docs/chv-muse-p1-gap.md。 恢复收尾(分离后回归,见 gap.md §4):VmDeathTest 死亡模拟改杀 实际跑命令的 VM;两处冷路径旧测试显式关常驻;_balloon_vsock_payload 与 listen_notify 拆到 50 行内;resident_pool.py:208 mypy 修掉。 executor.py out 未注解与 marker_tools.py:142 是基线旧错,未动。 测试:全量 cloud 套件 1934 OK(skipped=8);mypy 在改动文件上 干净(仅基线旧错 4 处)。真机验证待第三棒让出 WSL 测试机。
This commit is contained in:
1 parent
06c3e172fe
commit
78098c5594
13 files changed
+2527
-36
No files matched your search
+61
-5
@@ -259,6 +259,60 @@ def _gitea_fragment(env: dict) -> dict:
|
|||||||
return {"gitea_sync_env": config_gitea_sync_env(env)}
|
return {"gitea_sync_env": config_gitea_sync_env(env)}
|
||||||
|
|
||||||
|
|
||||||
|
def _to_switch(raw: object, name: str) -> bool | None:
|
||||||
|
"""CHV_* 开关解析(P1 气球/vsock 用):未设 → None(调用方
|
||||||
|
再定默认值);"1" 开、"0" 关;其它值一律 ValueError,绝不
|
||||||
|
静默当关(与 CHV_SSH_FORWARD 的 `== "1"` 旧口径不同,本轨
|
||||||
|
新开关按差距表要求非法值即报错)。"""
|
||||||
|
if raw is None:
|
||||||
|
return None
|
||||||
|
if raw == "1":
|
||||||
|
return True
|
||||||
|
if raw == "0":
|
||||||
|
return False
|
||||||
|
raise ValueError("%s 只能是 0 或 1:%r" % (name, raw))
|
||||||
|
|
||||||
|
|
||||||
|
def _balloon_vsock_fragment(env: dict) -> dict:
|
||||||
|
"""气球/vsock 片段(P1,feat-chv-muse-p1):virtio-balloon
|
||||||
|
+ free-page-reporting 开关、vsock notify 通道开关与客 CID。
|
||||||
|
|
||||||
|
- CHV_BALLOON:默认 1(开)。只认 "1"/"0",其它值
|
||||||
|
ValueError。
|
||||||
|
- CHV_VSOCK:默认 1(开)。只认 "1"/"0",其它值
|
||||||
|
ValueError。
|
||||||
|
- CHV_VSOCK_CID:客机 vsock CID,可选,默认 3。必须是
|
||||||
|
整数且在 3-4294967295 之间:0/1/2 是 vsock 保留 CID
|
||||||
|
(Hypervisor/Loopback/Host,CHV 的 VmConfig 校验会
|
||||||
|
以 VsockSpecialCid 拒绝),-1(随机 CID)CHV 的
|
||||||
|
vm.create 同样 400 拒绝(上游 discussion #7531),
|
||||||
|
所以这里不接受保留值/负值;显式设了非法 CID 即
|
||||||
|
ValueError,哪怕 CHV_VSOCK=0(配错早炸,不留到起 VM
|
||||||
|
才暴露)。CID 由运维指定、不是 CHV 分配的:同一宿主
|
||||||
|
上各 VM 的 vsock socket 文件不同,CID 相同(默认 3)
|
||||||
|
并不冲突。"""
|
||||||
|
balloon = _to_switch(env.get("CHV_BALLOON"), "CHV_BALLOON")
|
||||||
|
if balloon is None:
|
||||||
|
balloon = True
|
||||||
|
vsock = _to_switch(env.get("CHV_VSOCK"), "CHV_VSOCK")
|
||||||
|
if vsock is None:
|
||||||
|
vsock = True
|
||||||
|
raw_cid = env.get("CHV_VSOCK_CID")
|
||||||
|
if raw_cid is None:
|
||||||
|
vsock_cid = 3
|
||||||
|
else:
|
||||||
|
try:
|
||||||
|
vsock_cid = _to_int(raw_cid)
|
||||||
|
except ValueError:
|
||||||
|
raise ValueError("CHV_VSOCK_CID 必须是整数")
|
||||||
|
if not 3 <= vsock_cid <= 0xFFFFFFFF:
|
||||||
|
raise ValueError(
|
||||||
|
"CHV_VSOCK_CID 必须在 3-4294967295 之间:%s"
|
||||||
|
% vsock_cid)
|
||||||
|
return {"balloon": balloon, "vsock": vsock,
|
||||||
|
"vsock_cid": vsock_cid}
|
||||||
|
|
||||||
|
|
||||||
def _seed_fragment(params: dict, env: dict) -> dict:
|
def _seed_fragment(params: dict, env: dict) -> dict:
|
||||||
"""seed 附加项片段:只存宿主文件路径,不读文件内容
|
"""seed 附加项片段:只存宿主文件路径,不读文件内容
|
||||||
(读文件在 wake 时 fail-closed,见执行器 _seed_extras;
|
(读文件在 wake 时 fail-closed,见执行器 _seed_extras;
|
||||||
@@ -295,17 +349,19 @@ def build_chv_config(params: dict, env: dict) -> "ChvConfig":
|
|||||||
"""把显式构造参数与环境变量解析成 ChvConfig。
|
"""把显式构造参数与环境变量解析成 ChvConfig。
|
||||||
|
|
||||||
片段调用顺序即旧校验顺序(SSH → 配额 → 磁盘 →
|
片段调用顺序即旧校验顺序(SSH → 配额 → 磁盘 →
|
||||||
超时),只有一处可观测的顺序变化:warm_pool_size
|
气球/vsock → 超时),只有两处可观测的顺序变化:
|
||||||
从旧超时片段移到配额片段,因此“CHV_WARM_POOL_SIZE
|
warm_pool_size 从旧超时片段移到配额片段,因此
|
||||||
非法 + 磁盘非法同时出现”时先报前者;单项非法时
|
“CHV_WARM_POOL_SIZE 非法 + 磁盘非法同时出现”时先报
|
||||||
报错与旧代码逐字一致。字段集合与取值与旧代码一致,
|
前者;单项非法时报错与旧代码逐字一致。字段集合 =
|
||||||
由测试的字段集断言钉住。"""
|
旧代码字段 + P1 气球/vsock 三字段(balloon/vsock/
|
||||||
|
vsock_cid),由测试的字段集断言钉住。"""
|
||||||
env = env if env is not None else {}
|
env = env if env is not None else {}
|
||||||
return _assemble_chv_config([
|
return _assemble_chv_config([
|
||||||
_base_fragment(params, env),
|
_base_fragment(params, env),
|
||||||
_ssh_fragment(params, env),
|
_ssh_fragment(params, env),
|
||||||
_quota_fragment(env),
|
_quota_fragment(env),
|
||||||
_disk_fragment(env),
|
_disk_fragment(env),
|
||||||
|
_balloon_vsock_fragment(env),
|
||||||
_network_fragment(params, env),
|
_network_fragment(params, env),
|
||||||
_timeout_fragment(params, env),
|
_timeout_fragment(params, env),
|
||||||
_gitea_fragment(env),
|
_gitea_fragment(env),
|
||||||
|
|||||||
+227
-25
@@ -97,6 +97,10 @@ class CloudHypervisorExecutor(Executor):
|
|||||||
SSH 直接复用 OpenSSH 的密钥认证与端口转发,云镜像自带
|
SSH 直接复用 OpenSSH 的密钥认证与端口转发,云镜像自带
|
||||||
sshd,无需在客内再写一个 vsock agent、也无需在宿主再写
|
sshd,无需在客内再写一个 vsock agent、也无需在宿主再写
|
||||||
一个 vsock 代理去翻译 host.py 协议。
|
一个 vsock 代理去翻译 host.py 协议。
|
||||||
|
- P1(feat-chv-muse-p1)补充:vsock 只做最小 notify
|
||||||
|
(客机发 ready/心跳,见 cloud/chv/vsock.py 与
|
||||||
|
cloud/chv/guest/vsock_notify.py),与 SSH 并存;SSH
|
||||||
|
仍是唯一命令通道,host key 钉死不动。
|
||||||
- 放弃 vsock 的代价:需要运维提供从宿主到客机的 SSH 通路
|
- 放弃 vsock 的代价:需要运维提供从宿主到客机的 SSH 通路
|
||||||
(tap/桥接 + 端口转发,见下文“VM 生命周期”)。
|
(tap/桥接 + 端口转发,见下文“VM 生命周期”)。
|
||||||
- Host key 钉死(本轨 #216 后的加固,默认行为变化):
|
- Host key 钉死(本轨 #216 后的加固,默认行为变化):
|
||||||
@@ -131,8 +135,8 @@ class CloudHypervisorExecutor(Executor):
|
|||||||
“true”命令,直到 CHV_READY_TIMEOUT(默认 120 秒,
|
“true”命令,直到 CHV_READY_TIMEOUT(默认 120 秒,
|
||||||
轮询间隔 CHV_READY_INTERVAL,默认 5 秒)内成功才跑
|
轮询间隔 CHV_READY_INTERVAL,默认 5 秒)内成功才跑
|
||||||
host.py/任务;超时则关 VM 并报失败,绝不在未就绪时
|
host.py/任务;超时则关 VM 并报失败,绝不在未就绪时
|
||||||
就派任务。vsock 放弃理由同客内通道一节(需客内 agent
|
就派任务。P1 的 vsock ready 只做就绪辅助信号
|
||||||
+ 宿主代理两套新代码,v1 不拼半吊子)。
|
(cloud/chv/vsock.py),绝不取代 SSH 就绪判定。
|
||||||
- 暂停与快照唤醒是两条独立代码分支:
|
- 暂停与快照唤醒是两条独立代码分支:
|
||||||
pause_vm()/resume_vm() 走 ch-remote pause/resume;
|
pause_vm()/resume_vm() 走 ch-remote pause/resume;
|
||||||
restore_vm() 先 ch-remote restore(source_url=...),
|
restore_vm() 先 ch-remote restore(source_url=...),
|
||||||
@@ -230,6 +234,12 @@ class CloudHypervisorExecutor(Executor):
|
|||||||
- CHV_API_TIMEOUT:等 vmm.ping 通的上限秒数(默认 30)
|
- CHV_API_TIMEOUT:等 vmm.ping 通的上限秒数(默认 30)
|
||||||
- CHV_READY_TIMEOUT:等 guest SSH 就绪的上限秒数(默认 120)
|
- CHV_READY_TIMEOUT:等 guest SSH 就绪的上限秒数(默认 120)
|
||||||
- CHV_READY_INTERVAL:SSH 就绪轮询间隔秒数(默认 5)
|
- CHV_READY_INTERVAL:SSH 就绪轮询间隔秒数(默认 5)
|
||||||
|
- CHV_BALLOON:virtio-balloon + free-page-reporting 开关
|
||||||
|
(默认 1 开;只认 0/1,非法值 ValueError)
|
||||||
|
- CHV_VSOCK:vsock notify 通道开关(默认 1 开;只认
|
||||||
|
0/1,非法值 ValueError)
|
||||||
|
- CHV_VSOCK_CID:客机 vsock CID(默认 3;须在
|
||||||
|
3-4294967295 内,0/1/2 为保留 CID)
|
||||||
|
|
||||||
前置校验(preflight,执行入口必调,缺一即 fail-fast):
|
前置校验(preflight,执行入口必调,缺一即 fail-fast):
|
||||||
收集全部缺失项一次报出具体缺哪一项,绝不笼统报错:
|
收集全部缺失项一次报出具体缺哪一项,绝不笼统报错:
|
||||||
@@ -282,6 +292,9 @@ class CloudHypervisorExecutor(Executor):
|
|||||||
task_timeout: float | None
|
task_timeout: float | None
|
||||||
snapshot_url: str | None
|
snapshot_url: str | None
|
||||||
warm_pool_size: int
|
warm_pool_size: int
|
||||||
|
balloon: bool
|
||||||
|
vsock: bool
|
||||||
|
vsock_cid: int
|
||||||
api_timeout: float
|
api_timeout: float
|
||||||
ready_timeout: float
|
ready_timeout: float
|
||||||
ready_interval: float
|
ready_interval: float
|
||||||
@@ -431,7 +444,8 @@ class CloudHypervisorExecutor(Executor):
|
|||||||
"snapshot_max_total_gb", "snapshot_retention_days",
|
"snapshot_max_total_gb", "snapshot_retention_days",
|
||||||
"snapshot_dir", "allow_remote_snapshot",
|
"snapshot_dir", "allow_remote_snapshot",
|
||||||
"gitea_sync_env", "seed_user_data_extra_file",
|
"gitea_sync_env", "seed_user_data_extra_file",
|
||||||
"seed_network_config_file"):
|
"seed_network_config_file", "balloon", "vsock",
|
||||||
|
"vsock_cid"):
|
||||||
setattr(self, _field, getattr(_cfg, _field))
|
setattr(self, _field, getattr(_cfg, _field))
|
||||||
|
|
||||||
# -- 名字 / 路径 ---------------------------------------------------
|
# -- 名字 / 路径 ---------------------------------------------------
|
||||||
@@ -1024,7 +1038,8 @@ class CloudHypervisorExecutor(Executor):
|
|||||||
"image_type": "Raw"})
|
"image_type": "Raw"})
|
||||||
return disks
|
return disks
|
||||||
|
|
||||||
def build_vm_config(self, task_disk: str | None = None, seed_disk: str | None = None) -> dict:
|
def build_vm_config(self, task_disk: str | None = None, seed_disk: str | None = None,
|
||||||
|
vsock_socket: str | None = None) -> dict:
|
||||||
"""API 模式的 VmConfig(vm.create 请求体,纯函数):
|
"""API 模式的 VmConfig(vm.create 请求体,纯函数):
|
||||||
cpus{boot_vcpus,max_vcpus}、memory{size: 字节}、
|
cpus{boot_vcpus,max_vcpus}、memory{size: 字节}、
|
||||||
payload{kernel, cmdline}、disks[{path: 每任务磁盘}]、
|
payload{kernel, cmdline}、disks[{path: 每任务磁盘}]、
|
||||||
@@ -1037,7 +1052,11 @@ class CloudHypervisorExecutor(Executor):
|
|||||||
旧版顶层 kernel{path}/cmdline{args} 会被服务端忽略,
|
旧版顶层 kernel{path}/cmdline{args} 会被服务端忽略,
|
||||||
vm.create 照常 204、vm.boot 报 400 "No bootitem
|
vm.create 照常 204、vm.boot 报 400 "No bootitem
|
||||||
provided: neither firmware nor kernel"(2026-11-07
|
provided: neither firmware nor kernel"(2026-11-07
|
||||||
WSL 真机 CHV 54.0.0 实测)。"""
|
WSL 真机 CHV 54.0.0 实测)。
|
||||||
|
P1(feat-chv-muse-p1):默认带 balloon/vsock 设备
|
||||||
|
(见 _balloon_vsock_payload;CHV_BALLOON/CHV_VSOCK
|
||||||
|
可关)。vsock_socket 显式给时用它,否则按任务盘
|
||||||
|
位置派生每 VM 唯一路径(_boot_vm 未传时即走派生)。"""
|
||||||
try:
|
try:
|
||||||
cpus = int(self.cpus)
|
cpus = int(self.cpus)
|
||||||
except ValueError:
|
except ValueError:
|
||||||
@@ -1054,6 +1073,8 @@ class CloudHypervisorExecutor(Executor):
|
|||||||
"disks": self._vm_config_disks(task_disk, seed_disk),
|
"disks": self._vm_config_disks(task_disk, seed_disk),
|
||||||
"rng": {"src": "/dev/urandom"},
|
"rng": {"src": "/dev/urandom"},
|
||||||
}
|
}
|
||||||
|
config.update(self._balloon_vsock_payload(
|
||||||
|
vsock_socket=vsock_socket, task_disk=task_disk))
|
||||||
if self.net_json:
|
if self.net_json:
|
||||||
try:
|
try:
|
||||||
net = json.loads(self.net_json)
|
net = json.loads(self.net_json)
|
||||||
@@ -1062,6 +1083,67 @@ class CloudHypervisorExecutor(Executor):
|
|||||||
config["net"] = net if isinstance(net, list) else [net]
|
config["net"] = net if isinstance(net, list) else [net]
|
||||||
return config
|
return config
|
||||||
|
|
||||||
|
def _balloon_vsock_payload(self, vsock_socket=None,
|
||||||
|
task_disk=None) -> dict:
|
||||||
|
"""P1(feat-chv-muse-p1):VmConfig 的 balloon/vsock
|
||||||
|
设备(CHV 54 API,VmConfig 顶层键;字段名以 CHV
|
||||||
|
OpenAPI 生成的客户端为准:BalloonConfig{size,
|
||||||
|
deflate_on_oom, free_page_reporting}、VsockConfig{
|
||||||
|
cid, socket};本机无 /dev/kvm,尚未真机验证,见下)。
|
||||||
|
|
||||||
|
balloon(默认开):{"size": 0, "deflate_on_oom": true,
|
||||||
|
"free_page_reporting": true}。size 是目标气球尺寸
|
||||||
|
(字节):0 = 启动时不预扣客机内存,只靠
|
||||||
|
free_page_reporting 让客机持续上报空闲页、宿主
|
||||||
|
MADV_DONTNEED 回收——被动回收,不主动 inflate。
|
||||||
|
deflate_on_oom 是安全网:日后若经 vm.resize 充气
|
||||||
|
后客机承压,气球自动放气而不 OOM。待真机验证:
|
||||||
|
size=0 是否被 CHV 54 接受(若拒绝,改 size 为 VM
|
||||||
|
内存字节数并重新验证,见汇报);气球实际省内存量。
|
||||||
|
vsock(默认开):{"cid": <CHV_VSOCK_CID,默认 3>,
|
||||||
|
"socket": <宿主侧 Unix socket 路径>}。cid 是客机
|
||||||
|
CID(0/1/2 保留,宿主永远是 CID 2,客机连宿主用
|
||||||
|
CID 2;CHV 不分配 CID,由配置指定)。socket 路径
|
||||||
|
派生:显式给 vsock_socket 时用它;否则任务盘在
|
||||||
|
state_dir 内时落任务盘同目录(state_dir/<vm>/
|
||||||
|
vsock.sock,每 VM 唯一);reuse/lab(任务盘即基础
|
||||||
|
镜像或未给)时落 state_dir/vsock.sock——单 VM 可
|
||||||
|
用,多 VM 并发会在 vm.create 撞路径(大声失败,
|
||||||
|
不静默),_boot_vm 后续接线应显式传每 VM 路径。
|
||||||
|
宿主侧监听在 <socket>_<port> 上(port 见
|
||||||
|
cloud/chv/vsock.py 的 VSOCK_NOTIFY_PORT),客机
|
||||||
|
连 CID 2 发 ready/心跳(cloud/chv/guest/
|
||||||
|
vsock_notify.py)。开关关时对应键整个不出现
|
||||||
|
(不是发 false),旧行为(无设备)原样保留。"""
|
||||||
|
extra: dict = {}
|
||||||
|
if getattr(self, "balloon", True):
|
||||||
|
extra["balloon"] = {
|
||||||
|
"size": 0, "deflate_on_oom": True,
|
||||||
|
"free_page_reporting": True,
|
||||||
|
}
|
||||||
|
if getattr(self, "vsock", True):
|
||||||
|
extra["vsock"] = {
|
||||||
|
"cid": getattr(self, "vsock_cid", 3),
|
||||||
|
"socket": self._resolve_vsock_socket(
|
||||||
|
vsock_socket, task_disk),
|
||||||
|
}
|
||||||
|
return extra
|
||||||
|
|
||||||
|
def _resolve_vsock_socket(self, vsock_socket, task_disk):
|
||||||
|
"""vsock socket 路径派生(_balloon_vsock_payload 的
|
||||||
|
子步,单拆只为 50 行规矩,行为不动):显式给定时
|
||||||
|
用它;否则任务盘在 state_dir 内时落任务盘同目录
|
||||||
|
(每 VM 唯一),reuse/lab 时落 state_dir/vsock.sock
|
||||||
|
(多 VM 并发会撞路径,大声失败)。"""
|
||||||
|
if vsock_socket is not None:
|
||||||
|
return vsock_socket
|
||||||
|
if task_disk and os.path.realpath(
|
||||||
|
task_disk).startswith(
|
||||||
|
os.path.realpath(self.state_dir) + os.sep):
|
||||||
|
return os.path.join(
|
||||||
|
os.path.dirname(task_disk), "vsock.sock")
|
||||||
|
return os.path.join(self.state_dir, "vsock.sock")
|
||||||
|
|
||||||
def _api_call(self, socket_path, method, path, payload=None):
|
def _api_call(self, socket_path, method, path, payload=None):
|
||||||
"""API 调用归一化(实现在 cloud/chv/api_client.py)。"""
|
"""API 调用归一化(实现在 cloud/chv/api_client.py)。"""
|
||||||
return normalize_api_result(
|
return normalize_api_result(
|
||||||
@@ -2245,27 +2327,118 @@ class CloudHypervisorExecutor(Executor):
|
|||||||
"interrupted_reason": reason,
|
"interrupted_reason": reason,
|
||||||
}
|
}
|
||||||
|
|
||||||
def _run_with_vm(self, task, inner_args, goal=False, extra_env=None,
|
def _resident_enabled(self):
|
||||||
**kwargs):
|
"""常驻默认开;CHV_RESIDENT_ENABLED=0/false/no/off 关。
|
||||||
"""每任务 VM 生命周期(冷启动正式路径,feat-chv-arch95
|
优先认执行器上已有的 resident_enabled 属性(P1-B 若
|
||||||
定案):wake_vm(API 模式 + SSH 就绪)→ 跑客内命令 →
|
经 config 落属性,直接复用),否则惰性读 env;__init__
|
||||||
finally shutdown_vm。本方法绝不查 WarmPool:预热池是
|
不动,避免与 P1-B 的 __init__ 改动撞车。"""
|
||||||
实验性的(见 experimental_pool.py 头定案与 ADR-5),dispatch 永远
|
flag = getattr(self, "resident_enabled", None)
|
||||||
冷启动,warm_pool_wired_to_dispatch 恒 False。
|
if flag is None:
|
||||||
wake 失败时不跑
|
env = getattr(self, "_env", None)
|
||||||
任务、直接抛分类失败。本方法不做内部重试:重试是
|
if env is None:
|
||||||
worker 的 max_retries 语义,执行器内部重试会把一次
|
env = os.environ
|
||||||
失败放大成多次计费与多次 VM 启动。
|
flag = env.get("CHV_RESIDENT_ENABLED", "1")
|
||||||
|
if isinstance(flag, str):
|
||||||
|
return flag.strip().lower() not in (
|
||||||
|
"0", "false", "no", "off")
|
||||||
|
return bool(flag)
|
||||||
|
|
||||||
中途死亡分类(本轨新增):客内命令抛传输异常、或
|
def _resident_pool_instance(self):
|
||||||
非 goal 结果已 failed 时,若登记的 VMM 进程已死,
|
"""惰性建唯一常驻池并缓存(方法内 import:执行器顶层
|
||||||
判 interrupted(断点可续跑)而不是笼统 failed 或
|
绝不 import 池/orphans,见 ADR-12 的 AST 钉子)。
|
||||||
原样抛异常;VM 还活着时的异常/失败原样处理,不
|
同一 state_dir 只活一个池实例:registry 只在构造时
|
||||||
误判。goal 结果本身已有 interrupted 分类(见
|
读一次,重启后条目一律 orphan(见 resident_pool)。"""
|
||||||
parse_goal_result),这里不再二次改写 goal 结论。
|
pool = getattr(self, "_resident_pool", None)
|
||||||
feat-chv-secfix(H2):extra_env 透传给 wake_vm(密钥
|
if pool is not None:
|
||||||
子集在 provisioned 模式下随 seed 投递),再透传给
|
return pool
|
||||||
_run_guest(远程命令 source 客内密钥文件)。"""
|
with self._vms_lock:
|
||||||
|
pool = getattr(self, "_resident_pool", None)
|
||||||
|
if pool is None:
|
||||||
|
from cloud.chv.resident_pool import ResidentPool
|
||||||
|
pool = ResidentPool(self)
|
||||||
|
self._resident_pool = pool
|
||||||
|
return pool
|
||||||
|
|
||||||
|
def _resident_acquire(self, task, extra_env=None):
|
||||||
|
"""常驻获取(best-effort,一律不抛,失败回 None):
|
||||||
|
未启用/带密钥 env → None(走冷启动;密钥须随 seed
|
||||||
|
投递,常驻 VM 的 seed 是建池时定的,带密钥任务只能
|
||||||
|
冷启动)。有 paused 条目 → resume + SSH 就绪;池空/
|
||||||
|
失败 → 惰性 provision 一次再取,仍无 → None。返回
|
||||||
|
{"task": 常驻伪任务, "pool": 池, "user_id": 用户}。"""
|
||||||
|
try:
|
||||||
|
if not self._resident_enabled():
|
||||||
|
return None
|
||||||
|
secrets, _plain = split_secret_env(extra_env)
|
||||||
|
if secrets:
|
||||||
|
return None
|
||||||
|
pool = self._resident_pool_instance()
|
||||||
|
user_id = task.get("user_id") \
|
||||||
|
if isinstance(task, dict) else None
|
||||||
|
acquired = pool.acquire(user_id)
|
||||||
|
if acquired.get("mode") == "resident":
|
||||||
|
return {"task": acquired["task"], "pool": pool,
|
||||||
|
"user_id": user_id}
|
||||||
|
provisioned = pool.provision(user_id)
|
||||||
|
if provisioned.get("mode") != "resident":
|
||||||
|
return None
|
||||||
|
acquired = pool.acquire(user_id)
|
||||||
|
if acquired.get("mode") != "resident":
|
||||||
|
return None
|
||||||
|
return {"task": acquired["task"], "pool": pool,
|
||||||
|
"user_id": user_id}
|
||||||
|
except Exception:
|
||||||
|
return None
|
||||||
|
|
||||||
|
def _run_on_resident(self, task, resident_task, inner_args,
|
||||||
|
goal=False, extra_env=None, **kwargs):
|
||||||
|
"""常驻路径执行(_run_with_vm 的常驻分支本体):
|
||||||
|
_run_guest/_run_guest_goal 一律走常驻伪任务传参(SSH
|
||||||
|
known_hosts、api-socket、密钥投递记录、VMM 登记都按
|
||||||
|
常驻 VM 名算),返回后把 task_id 改回真实任务 id;
|
||||||
|
VM 中途死亡判据同样按常驻伪任务查登记进程,死亡
|
||||||
|
结论里的任务 id 用真实任务。finally 的 pause 在调
|
||||||
|
用方(_run_with_vm)做,这里只跑命令。"""
|
||||||
|
try:
|
||||||
|
if goal:
|
||||||
|
result = self._run_guest_goal(
|
||||||
|
resident_task, inner_args, extra_env=extra_env,
|
||||||
|
**kwargs)
|
||||||
|
else:
|
||||||
|
result = self._run_guest(
|
||||||
|
resident_task, inner_args, extra_env=extra_env,
|
||||||
|
**kwargs)
|
||||||
|
except Exception as exc:
|
||||||
|
if self._vm_proc_dead(resident_task):
|
||||||
|
return self._vm_death_result(
|
||||||
|
task, goal, str(exc)[:500])
|
||||||
|
raise
|
||||||
|
if not goal and result.get("status") == "failed" \
|
||||||
|
and self._vm_proc_dead(resident_task):
|
||||||
|
result = dict(result)
|
||||||
|
result["status"] = "interrupted"
|
||||||
|
result["vm_dead"] = True
|
||||||
|
result["interrupted_reason"] = (
|
||||||
|
"VM 中途死亡,任务未完成、断点可续跑:"
|
||||||
|
"客内命令已 failed 且登记的 VMM 进程已死")
|
||||||
|
if isinstance(result, dict):
|
||||||
|
result = dict(result)
|
||||||
|
result["task_id"] = task["id"]
|
||||||
|
return result
|
||||||
|
|
||||||
|
def _run_cold(self, task, inner_args, goal=False, extra_env=None,
|
||||||
|
**kwargs):
|
||||||
|
"""冷启动路径(原 _run_with_vm 本体,逐字搬运):
|
||||||
|
wake_vm(API 模式 + SSH 就绪)→ 跑客内命令 →
|
||||||
|
finally shutdown_vm。wake 失败时不跑任务、直接抛
|
||||||
|
分类失败;本方法不做内部重试(重试是 worker 的
|
||||||
|
max_retries 语义)。中途死亡分类:客内命令抛传输
|
||||||
|
异常、或非 goal 结果已 failed 时,若登记的 VMM 进程
|
||||||
|
已死,判 interrupted(断点可续跑);VM 还活着时的
|
||||||
|
异常/失败原样处理。feat-chv-secfix(H2):extra_env
|
||||||
|
透传给 wake_vm(密钥子集在 provisioned 模式下随 seed
|
||||||
|
投递),再透传给 _run_guest(远程命令 source 客内
|
||||||
|
密钥文件)。"""
|
||||||
self.wake_vm(task, extra_env=extra_env)
|
self.wake_vm(task, extra_env=extra_env)
|
||||||
try:
|
try:
|
||||||
try:
|
try:
|
||||||
@@ -2294,6 +2467,35 @@ class CloudHypervisorExecutor(Executor):
|
|||||||
except Exception:
|
except Exception:
|
||||||
pass
|
pass
|
||||||
|
|
||||||
|
def _run_with_vm(self, task, inner_args, goal=False, extra_env=None,
|
||||||
|
**kwargs):
|
||||||
|
"""任务 VM 生命周期(feat-chv-muse-p1,见 ADR-17):
|
||||||
|
常驻启用时先 _resident_acquire——paused 常驻 VM 经
|
||||||
|
resume + SSH 就绪即跑,跑完 finally 里 release(pause,
|
||||||
|
不 shutdown);池空/失败/未启用/带密钥 env 时走冷启动
|
||||||
|
(_run_cold:wake_vm → 跑客内命令 → finally shutdown)。
|
||||||
|
常驻失败一律回落冷启动,绝不因池坏而丢任务;WarmPool
|
||||||
|
仍绝不查(实验模块,wired_to_dispatch 恒 False)。
|
||||||
|
wake 失败时不跑任务、直接抛分类失败;本方法不做内部
|
||||||
|
重试(重试是 worker 的 max_retries 语义)。中途死亡
|
||||||
|
判 interrupted(断点可续跑),见 _run_cold 与
|
||||||
|
_run_on_resident。dispatch/run/goal/goal_resume/断点
|
||||||
|
续跑都走本方法,签名不动。"""
|
||||||
|
resident = self._resident_acquire(task, extra_env=extra_env)
|
||||||
|
if resident is not None:
|
||||||
|
pool = resident["pool"]
|
||||||
|
try:
|
||||||
|
return self._run_on_resident(
|
||||||
|
task, resident["task"], inner_args, goal=goal,
|
||||||
|
extra_env=extra_env, **kwargs)
|
||||||
|
finally:
|
||||||
|
try:
|
||||||
|
pool.release(resident["user_id"])
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
return self._run_cold(task, inner_args, goal=goal,
|
||||||
|
extra_env=extra_env, **kwargs)
|
||||||
|
|
||||||
def execute(self, task: dict) -> dict:
|
def execute(self, task: dict) -> dict:
|
||||||
"""只读执行:客内 host.py status,不直接跑模型回合
|
"""只读执行:客内 host.py status,不直接跑模型回合
|
||||||
(与 Local/Docker 的 execute 同款语义)。
|
(与 Local/Docker 的 execute 同款语义)。
|
||||||
|
|||||||
Executable
+136
@@ -0,0 +1,136 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""客内 vsock notify 脚本(P1,feat-chv-muse-p1)。
|
||||||
|
|
||||||
|
启动后连宿主 CID 2 的 1024 端口发 ready,之后每 30 秒发一次
|
||||||
|
heartbeat。消息是 JSON 行:
|
||||||
|
|
||||||
|
{"type": "ready"|"heartbeat", "vm": "<vm 名>", "ts": <unix 秒>}
|
||||||
|
|
||||||
|
宿主在 <vsock socket>_1024 这个 Unix socket 上监听接收
|
||||||
|
(cloud/chv/vsock.py 的 wait_ready/listen_notify)。
|
||||||
|
|
||||||
|
失败一律静默退出(exit 0,不打日志、不留 traceback):vsock
|
||||||
|
只是就绪辅助信号,SSH 仍是唯一命令通道与就绪判定——本脚本
|
||||||
|
绝不能因为连不上、发失败而影响 sshd 或任务。调试时设
|
||||||
|
VSOCK_NOTIFY_DEBUG=1 可在 stderr 看到异常。
|
||||||
|
|
||||||
|
================ P2 镜像集成接入说明(本轨只写脚本,不改构建) ================
|
||||||
|
本文件不由 chv-guest/build.sh 安装,镜像集成是 P2 的范围。
|
||||||
|
P2 接入时做这三件:
|
||||||
|
|
||||||
|
1. 把本文件拷进镜像,例如 /app/vsock_notify.py(须 python3,
|
||||||
|
客机镜像已有 Python;若最小镜像无 python3,P2 须先补);
|
||||||
|
2. 加 systemd 单元,例如 /etc/systemd/system/vsock-notify.service:
|
||||||
|
|
||||||
|
[Unit]
|
||||||
|
Description=vsock notify to host (ready/heartbeat)
|
||||||
|
After=multi-user.target
|
||||||
|
[Service]
|
||||||
|
Type=simple
|
||||||
|
ExecStart=/usr/bin/python3 /app/vsock_notify.py %H
|
||||||
|
Restart=no
|
||||||
|
[Install]
|
||||||
|
WantedBy=multi-user.target
|
||||||
|
|
||||||
|
(%H 是客机 hostname;也可直接写 VM 名,或由 cloud-init
|
||||||
|
写进环境变量 CHV_VM_NAME。Restart=no:本脚本失败即静默
|
||||||
|
退出,不由 systemd 反复拉起打扰宿主监听。)
|
||||||
|
3. 前置条件(缺一则本脚本静默退出,任务仍走 SSH,不报错):
|
||||||
|
- 客机内核须含 CONFIG_VIRTIO_VSOCKETS;
|
||||||
|
- 起 VM 时 CHV_VSOCK=1(默认开),VmConfig 带 vsock 设备
|
||||||
|
(cloud/chv/executor.py 的 _balloon_vsock_payload);
|
||||||
|
- 宿主侧须在 VM boot 前后监听 <socket>_1024(否则客机
|
||||||
|
首次 connect 失败,本脚本在 30 秒内重试,仍失败即退出)。
|
||||||
|
|
||||||
|
端口 1024 与宿主侧 cloud/chv/vsock.py 的 VSOCK_NOTIFY_PORT
|
||||||
|
须一致;改一处必须改另一处。
|
||||||
|
===============================================================================
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import socket
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
|
||||||
|
# 宿主 CID(vsock 众所周知值,CHV 官方 docs/vsock.md)。
|
||||||
|
HOST_CID = 2
|
||||||
|
# notify 端口(与宿主侧 VSOCK_NOTIFY_PORT 一致)。
|
||||||
|
NOTIFY_PORT = 1024
|
||||||
|
# 首次 connect 重试窗口(秒):宿主监听可能比客机启动晚到。
|
||||||
|
CONNECT_RETRY_SECONDS = 30
|
||||||
|
# 心跳间隔(秒)。
|
||||||
|
HEARTBEAT_INTERVAL = 30
|
||||||
|
|
||||||
|
|
||||||
|
def _vm_name(argv) -> str:
|
||||||
|
"""VM 名:argv[1] > CHV_VM_NAME > hostname。"""
|
||||||
|
if len(argv) > 1 and argv[1]:
|
||||||
|
return argv[1]
|
||||||
|
env_name = os.environ.get("CHV_VM_NAME") or os.environ.get(
|
||||||
|
"VSOCK_VM_NAME")
|
||||||
|
if env_name:
|
||||||
|
return env_name
|
||||||
|
try:
|
||||||
|
return socket.gethostname()
|
||||||
|
except OSError:
|
||||||
|
return "unknown"
|
||||||
|
|
||||||
|
|
||||||
|
def _send(sock, message_type: str, vm: str) -> None:
|
||||||
|
line = json.dumps({"type": message_type, "vm": vm,
|
||||||
|
"ts": time.time()},
|
||||||
|
separators=(",", ":"))
|
||||||
|
sock.sendall((line + "\n").encode("utf-8"))
|
||||||
|
|
||||||
|
|
||||||
|
def _connect(deadline: float):
|
||||||
|
"""在 deadline 前反复连 (HOST_CID, NOTIFY_PORT),成功返回
|
||||||
|
socket,超时返回 None。"""
|
||||||
|
while True:
|
||||||
|
remaining = deadline - time.monotonic()
|
||||||
|
if remaining <= 0:
|
||||||
|
return None
|
||||||
|
sock = socket.socket(socket.AF_VSOCK, socket.SOCK_STREAM)
|
||||||
|
try:
|
||||||
|
sock.settimeout(min(remaining, 5))
|
||||||
|
sock.connect((HOST_CID, NOTIFY_PORT))
|
||||||
|
sock.settimeout(None)
|
||||||
|
return sock
|
||||||
|
except OSError as exc:
|
||||||
|
try:
|
||||||
|
sock.close()
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
if os.environ.get("VSOCK_NOTIFY_DEBUG") == "1":
|
||||||
|
print("vsock_notify connect 失败:%r" % (exc,),
|
||||||
|
file=sys.stderr, flush=True)
|
||||||
|
time.sleep(1)
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv) -> int:
|
||||||
|
vm = _vm_name(argv)
|
||||||
|
deadline = time.monotonic() + CONNECT_RETRY_SECONDS
|
||||||
|
sock = _connect(deadline)
|
||||||
|
if sock is None:
|
||||||
|
return 0
|
||||||
|
try:
|
||||||
|
_send(sock, "ready", vm)
|
||||||
|
while True:
|
||||||
|
time.sleep(HEARTBEAT_INTERVAL)
|
||||||
|
_send(sock, "heartbeat", vm)
|
||||||
|
except (OSError, ValueError):
|
||||||
|
# 任何发送失败都静默退出,不影响 sshd/任务。
|
||||||
|
return 0
|
||||||
|
except KeyboardInterrupt:
|
||||||
|
return 0
|
||||||
|
finally:
|
||||||
|
try:
|
||||||
|
sock.close()
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main(sys.argv))
|
||||||
@@ -0,0 +1,515 @@
|
|||||||
|
"""每用户常驻 VM 池(生产模块,P1 对齐 Muse,见 ADR-17)。
|
||||||
|
|
||||||
|
每用户一台常驻 VM:任务到时 resume 已暂停的 VM,跑完
|
||||||
|
再 pause,不销毁;池空或任何一步失败都回落冷启动,
|
||||||
|
绝不因池坏而丢任务。常驻 VM 的盘是每用户持久工作区盘
|
||||||
|
(state_dir/chv-resident-<用户>/disk.raw),不是每任务
|
||||||
|
新盘——隔离单元在常驻路径上从“每任务”变成“每用户”
|
||||||
|
(单租户可接受,多租户重审前不得把常驻当租户隔离)。
|
||||||
|
|
||||||
|
状态机:cold → provisioning → paused → running → paused。
|
||||||
|
acquire(user):有 paused 条目时 resume_vm + 等 SSH 就绪
|
||||||
|
(复用执行器 wait_guest_ready,不另写判定),返回
|
||||||
|
mode=resident;无可用/失败时返回 mode=cold。release(user):
|
||||||
|
pause_vm,不销毁;pause 失败时条目留在 running(不删
|
||||||
|
不销毁,worker 关闭时 reconcile 清理)。provision(user):
|
||||||
|
wake_vm + pause_vm 建 paused 条目(惰性建池,失败回落)。
|
||||||
|
reclaim_idle():paused 超 CHV_RESIDENT_IDLE_MINUTES(默认
|
||||||
|
0=不自动关机)时 hibernate(pause + snapshot + shutdown)
|
||||||
|
并删条目,快照文件留作恢复物,restore 是后续工作。
|
||||||
|
|
||||||
|
Registry(state_dir/resident-pool.json):文件 IO 与损坏
|
||||||
|
语义复用 cloud.chv.orphans(原子写、损坏 quarantine、
|
||||||
|
单条目非法只丢该条);重启后读到的条目一律 orphan,不
|
||||||
|
复用,只能 reconcile 清理。条目按用户键隔离:release
|
||||||
|
绝不把用过的 VM 放回给另一用户;同一用户同一时间只跑
|
||||||
|
一个常驻任务,并发第二个任务回落冷启动。
|
||||||
|
|
||||||
|
诚实边界:resume 失败时 wait_guest_ready 的失败路径会
|
||||||
|
stop_vm(含删工作区盘),条目随之丢弃——该路径的工作区
|
||||||
|
以 Gitea 同步为准;带密钥 extra_env 的任务不走常驻
|
||||||
|
(seed 是建池时定的,密钥只能冷启动随 seed 投递)。
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import threading
|
||||||
|
import time
|
||||||
|
|
||||||
|
from cloud.chv.orphans import _read_registry, _write_registry
|
||||||
|
|
||||||
|
_LOGGER = logging.getLogger("cloud.cloud_hypervisor")
|
||||||
|
|
||||||
|
_REGISTRY_FILENAME = "resident-pool.json"
|
||||||
|
_RESIDENT_STATES = ("paused", "running", "provisioning")
|
||||||
|
_DEFAULT_USER = "default"
|
||||||
|
|
||||||
|
|
||||||
|
def resident_user_key(user_id: str | None) -> str:
|
||||||
|
"""用户键:非空即原样消毒(非法字符变 -),空/缺省
|
||||||
|
即 "default"(当前单租户:任务 dict 无 user_id 时全
|
||||||
|
部落到同一常驻 VM;多租户重审前不得当租户边界用)。"""
|
||||||
|
raw = _DEFAULT_USER if user_id is None else str(user_id)
|
||||||
|
key = re.sub(r"[^A-Za-z0-9_.-]", "-", raw).strip(".-")
|
||||||
|
return key or _DEFAULT_USER
|
||||||
|
|
||||||
|
|
||||||
|
def resident_task_for_user(user_id: str | None) -> dict:
|
||||||
|
"""常驻伪任务:执行器按它算 VM 名/known_hosts/socket。
|
||||||
|
vm_name({"id": "resident-alice"}) = "chv-resident-alice"。"""
|
||||||
|
return {"id": "resident-" + resident_user_key(user_id)}
|
||||||
|
|
||||||
|
|
||||||
|
def resident_registry_path_for_executor(executor: object) -> str | None:
|
||||||
|
"""常驻 registry 路径:state_dir/resident-pool.json。
|
||||||
|
无 state_dir 时返回 None(调用方按“无残留”处理)。"""
|
||||||
|
state_dir = getattr(executor, "state_dir", None)
|
||||||
|
if not state_dir:
|
||||||
|
return None
|
||||||
|
return os.path.join(state_dir, _REGISTRY_FILENAME)
|
||||||
|
|
||||||
|
|
||||||
|
def reconcile_resident_orphans_for_executor(executor: object) -> list[str]:
|
||||||
|
"""生产路径的常驻残留清理入口:返回已清理的 VM 名。
|
||||||
|
只做清理:读 resident-pool.json → 每个上一世条目尽力
|
||||||
|
shutdown_vm → 落盘为空 → best-effort 审计。绝不构造
|
||||||
|
ResidentPool、绝不把任何条目发给任务。无 registry 路
|
||||||
|
径时返回 []。文件 IO 与损坏语义复用 orphans。"""
|
||||||
|
path = resident_registry_path_for_executor(executor)
|
||||||
|
if not path:
|
||||||
|
return []
|
||||||
|
data, _corrupt = _read_registry(path)
|
||||||
|
if data is None:
|
||||||
|
return []
|
||||||
|
cleaned: list[str] = []
|
||||||
|
for entry in data.values():
|
||||||
|
if not isinstance(entry, dict):
|
||||||
|
continue
|
||||||
|
if not entry.get("vm"):
|
||||||
|
continue
|
||||||
|
if entry.get("state") not in _RESIDENT_STATES:
|
||||||
|
continue
|
||||||
|
vm = str(entry["vm"])
|
||||||
|
try:
|
||||||
|
executor.shutdown_vm(vm) # type: ignore[attr-defined]
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
cleaned.append(vm)
|
||||||
|
if cleaned:
|
||||||
|
_write_registry(path, {})
|
||||||
|
try:
|
||||||
|
executor._emit( # type: ignore[attr-defined]
|
||||||
|
"resident_pool_reconcile", {"id": None},
|
||||||
|
"cleaned=%s" % ",".join(cleaned))
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
return cleaned
|
||||||
|
|
||||||
|
|
||||||
|
class ResidentPool:
|
||||||
|
"""每用户常驻 VM 池(生产路径,见模块 docstring)。
|
||||||
|
|
||||||
|
同一 state_dir 只活一个池实例:registry 只在构造时
|
||||||
|
读一次,重启后条目一律 orphan(见 _load_registry)。
|
||||||
|
公共方法全部加锁:并发 acquire/release/provision 只
|
||||||
|
放行一路,其余回 cold/False,不把同一 VM 发给两个任务。
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, executor: object, idle_minutes: int | None = None,
|
||||||
|
registry_path: str | None = None) -> None:
|
||||||
|
"""建池并读 registry:读到的条目一律 orphan(见模块
|
||||||
|
docstring)。idle_minutes 未给时读执行器属性或
|
||||||
|
CHV_RESIDENT_IDLE_MINUTES,非法/负数即 0(不自动关机)。"""
|
||||||
|
self.executor = executor
|
||||||
|
self._lock = threading.Lock()
|
||||||
|
self._busy: set[str] = set()
|
||||||
|
self._entries: dict = {}
|
||||||
|
self.registry_path = registry_path
|
||||||
|
if self.registry_path is None:
|
||||||
|
self.registry_path = \
|
||||||
|
resident_registry_path_for_executor(executor)
|
||||||
|
if idle_minutes is None:
|
||||||
|
idle_minutes = self._idle_minutes_from_env()
|
||||||
|
self.idle_minutes = idle_minutes if idle_minutes > 0 else 0
|
||||||
|
self.registry_corrupt = False
|
||||||
|
self.registry_corrupt_entries = 0
|
||||||
|
self._load_registry()
|
||||||
|
|
||||||
|
# -- registry ------------------------------------------------------
|
||||||
|
def _load_registry(self) -> None:
|
||||||
|
"""载入 registry:读到的条目一律标 orphan(绝不直接
|
||||||
|
当 paused 发给任务,旧 VMM 进程已随旧 worker 消失)。
|
||||||
|
无文件时为空池;损坏时 quarantine 并从空开始;单条
|
||||||
|
目非法只丢该条并计数,不连坐其它条目。"""
|
||||||
|
if not self.registry_path:
|
||||||
|
return
|
||||||
|
data, corrupt = _read_registry(self.registry_path)
|
||||||
|
if corrupt:
|
||||||
|
self.registry_corrupt = True
|
||||||
|
return
|
||||||
|
if data is None:
|
||||||
|
return
|
||||||
|
for key, entry in data.items():
|
||||||
|
if not isinstance(entry, dict) \
|
||||||
|
or not entry.get("vm") \
|
||||||
|
or entry.get("state") not in _RESIDENT_STATES:
|
||||||
|
self.registry_corrupt_entries += 1
|
||||||
|
continue
|
||||||
|
self._entries[key] = {
|
||||||
|
"state": "orphan",
|
||||||
|
"vm": entry["vm"],
|
||||||
|
"user_id": entry.get("user_id") or key,
|
||||||
|
"paused_at": None,
|
||||||
|
"last_used": None,
|
||||||
|
}
|
||||||
|
|
||||||
|
def _save_registry(self) -> None:
|
||||||
|
"""落盘当前条目(只写 paused/running/provisioning;
|
||||||
|
orphan 是待清理的上一世条目,reconcile 后即删,不写
|
||||||
|
回)。调用方须已持锁;写失败只记日志,不炸任务流。"""
|
||||||
|
if not self.registry_path:
|
||||||
|
return
|
||||||
|
data = {}
|
||||||
|
for key, entry in self._entries.items():
|
||||||
|
if entry["state"] not in _RESIDENT_STATES:
|
||||||
|
continue
|
||||||
|
data[key] = {
|
||||||
|
"state": entry["state"],
|
||||||
|
"vm": entry.get("vm"),
|
||||||
|
"user_id": entry.get("user_id"),
|
||||||
|
"paused_at": entry.get("paused_at"),
|
||||||
|
"last_used": entry.get("last_used"),
|
||||||
|
}
|
||||||
|
_write_registry(self.registry_path, data)
|
||||||
|
|
||||||
|
def _emit(self, event: str, task: dict, detail: str) -> None:
|
||||||
|
"""best-effort 审计:执行器无 _emit 或抛错都不炸
|
||||||
|
池操作(与 WarmPool 同款)。"""
|
||||||
|
emit = getattr(self.executor, "_emit", None)
|
||||||
|
if not callable(emit):
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
emit(event, task, detail)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
def _idle_minutes_from_env(self) -> int:
|
||||||
|
"""闲置阈值:执行器属性优先(P1-B 若经 config 落
|
||||||
|
属性,直接复用),否则读 CHV_RESIDENT_IDLE_MINUTES;
|
||||||
|
非数字/负数一律 0(不自动关机,不炸建池)。"""
|
||||||
|
raw = getattr(self.executor, "resident_idle_minutes", None)
|
||||||
|
if raw is None:
|
||||||
|
env = getattr(self.executor, "_env", None)
|
||||||
|
if env is None:
|
||||||
|
env = os.environ
|
||||||
|
raw = env.get("CHV_RESIDENT_IDLE_MINUTES", 0)
|
||||||
|
if raw is None:
|
||||||
|
return 0
|
||||||
|
try:
|
||||||
|
minutes = int(raw)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return 0
|
||||||
|
return minutes if minutes > 0 else 0
|
||||||
|
|
||||||
|
def _vm_name(self, resident_task: dict) -> str:
|
||||||
|
"""常驻 VM 名:优先执行器 vm_name(与冷启动同款派
|
||||||
|
生),鸭子型执行器无该方法时退回 chv- 前缀拼法。"""
|
||||||
|
vm_name = getattr(self.executor, "vm_name", None)
|
||||||
|
if callable(vm_name):
|
||||||
|
try:
|
||||||
|
return str(vm_name(resident_task))
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
return "chv-" + str(resident_task.get("id"))
|
||||||
|
|
||||||
|
# -- 计数 ----------------------------------------------------------
|
||||||
|
def paused_count(self) -> int:
|
||||||
|
"""当前 paused(可分配)的常驻条目数。"""
|
||||||
|
with self._lock:
|
||||||
|
return sum(1 for e in self._entries.values()
|
||||||
|
if e.get("state") == "paused")
|
||||||
|
|
||||||
|
def orphan_count(self) -> int:
|
||||||
|
"""上一世残留条目数(待 reconcile,不可分配)。"""
|
||||||
|
with self._lock:
|
||||||
|
return sum(1 for e in self._entries.values()
|
||||||
|
if e.get("state") == "orphan")
|
||||||
|
|
||||||
|
# -- acquire / release ----------------------------------------------
|
||||||
|
def _claim_paused(self, user_key: str) -> dict | None:
|
||||||
|
"""占一条 paused 条目:加忙标记并返回其快照;无条目/
|
||||||
|
非 paused/正忙时返回 None(调用方回落冷启动)。"""
|
||||||
|
with self._lock:
|
||||||
|
entry = self._entries.get(user_key)
|
||||||
|
if entry is None or entry.get("state") != "paused":
|
||||||
|
return None
|
||||||
|
if user_key in self._busy:
|
||||||
|
return None
|
||||||
|
self._busy.add(user_key)
|
||||||
|
return dict(entry)
|
||||||
|
|
||||||
|
def _resume_and_wait(self, resident_task: dict) -> bool:
|
||||||
|
"""resume + 等 SSH 就绪(复用执行器判定)。
|
||||||
|
resume 缺失的鸭子型执行器跳过 resume(与 WarmPool
|
||||||
|
同款宽容);SSH 未就绪即 False(wait_guest_ready
|
||||||
|
的失败路径已自行 stop_vm,调用方只须删条目)。"""
|
||||||
|
resume = getattr(self.executor, "resume_vm", None)
|
||||||
|
if callable(resume):
|
||||||
|
try:
|
||||||
|
resume(resident_task)
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
wait = getattr(self.executor, "wait_guest_ready", None)
|
||||||
|
if not callable(wait):
|
||||||
|
return True
|
||||||
|
try:
|
||||||
|
return bool(wait(resident_task))
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
|
||||||
|
def _settle_acquire(self, user_key: str, ok: bool) -> None:
|
||||||
|
"""acquire 收尾:成功标 running,失败删条目;都清
|
||||||
|
忙标记并落盘。"""
|
||||||
|
with self._lock:
|
||||||
|
self._busy.discard(user_key)
|
||||||
|
entry = self._entries.get(user_key)
|
||||||
|
if entry is None:
|
||||||
|
return
|
||||||
|
if ok:
|
||||||
|
entry["state"] = "running"
|
||||||
|
entry["last_used"] = time.time()
|
||||||
|
else:
|
||||||
|
self._entries.pop(user_key, None)
|
||||||
|
self._save_registry()
|
||||||
|
|
||||||
|
def acquire(self, user_id: str | None) -> dict:
|
||||||
|
"""取常驻 VM:paused 条目 → resume + SSH 就绪 →
|
||||||
|
mode=resident;无条目/正忙/失败 → mode=cold(调用
|
||||||
|
方回落冷启动)。失败条目直接丢弃(VM 本身已被
|
||||||
|
wait_guest_ready 的失败路径关掉,或 resume 已拒)。"""
|
||||||
|
user_key = resident_user_key(user_id)
|
||||||
|
entry = self._claim_paused(user_key)
|
||||||
|
if entry is None:
|
||||||
|
return {"mode": "cold", "vm": None, "task": None}
|
||||||
|
resident_task = resident_task_for_user(user_id)
|
||||||
|
ok = self._resume_and_wait(resident_task)
|
||||||
|
self._settle_acquire(user_key, ok)
|
||||||
|
if not ok:
|
||||||
|
return {"mode": "cold", "vm": None, "task": None}
|
||||||
|
self._emit("resident_pool_acquire", resident_task,
|
||||||
|
"vm=%s user=%s" % (entry.get("vm"), user_key))
|
||||||
|
return {"mode": "resident", "vm": entry.get("vm"),
|
||||||
|
"task": resident_task}
|
||||||
|
|
||||||
|
def _pause(self, resident_task: dict) -> bool:
|
||||||
|
"""pause_vm 尽力而为:失败返回 False,不抛。"""
|
||||||
|
pause = getattr(self.executor, "pause_vm", None)
|
||||||
|
if not callable(pause):
|
||||||
|
return False
|
||||||
|
try:
|
||||||
|
pause(resident_task)
|
||||||
|
return True
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
|
||||||
|
def release(self, user_id: str | None) -> bool:
|
||||||
|
"""还常驻 VM:running 条目 → pause_vm,不销毁。
|
||||||
|
无条目/非 running/正忙 → False;pause 失败时条目
|
||||||
|
留在 running(不删不销毁,worker 关闭时 reconcile
|
||||||
|
清理),返回 False。只动本用户的条目。"""
|
||||||
|
user_key = resident_user_key(user_id)
|
||||||
|
resident_task = resident_task_for_user(user_id)
|
||||||
|
with self._lock:
|
||||||
|
entry = self._entries.get(user_key)
|
||||||
|
if entry is None or entry.get("state") != "running":
|
||||||
|
return False
|
||||||
|
if user_key in self._busy:
|
||||||
|
return False
|
||||||
|
self._busy.add(user_key)
|
||||||
|
paused = self._pause(resident_task)
|
||||||
|
with self._lock:
|
||||||
|
self._busy.discard(user_key)
|
||||||
|
entry = self._entries.get(user_key)
|
||||||
|
if entry is None or not paused:
|
||||||
|
return False
|
||||||
|
entry["state"] = "paused"
|
||||||
|
entry["paused_at"] = time.time()
|
||||||
|
self._save_registry()
|
||||||
|
self._emit("resident_pool_release", resident_task,
|
||||||
|
"vm=%s user=%s" % (entry.get("vm"), user_key))
|
||||||
|
return True
|
||||||
|
|
||||||
|
# -- provision -------------------------------------------------------
|
||||||
|
def _wake_and_pause(self, resident_task: dict) -> bool:
|
||||||
|
"""建池:wake_vm(冷启动全套,含 SSH 就绪)再
|
||||||
|
pause_vm。任一步失败都尽力 shutdown,不留半醒的
|
||||||
|
VM;调用方随后删条目回落冷启动。"""
|
||||||
|
try:
|
||||||
|
wake = getattr(self.executor, "wake_vm", None)
|
||||||
|
pause = getattr(self.executor, "pause_vm", None)
|
||||||
|
if not callable(wake) or not callable(pause):
|
||||||
|
return False
|
||||||
|
wake(resident_task)
|
||||||
|
pause(resident_task)
|
||||||
|
return True
|
||||||
|
except Exception:
|
||||||
|
shutdown = getattr(self.executor, "shutdown_vm", None)
|
||||||
|
if callable(shutdown):
|
||||||
|
try:
|
||||||
|
shutdown(resident_task)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
return False
|
||||||
|
|
||||||
|
def _finish_provision(self, user_key: str, ok: bool) -> str | None:
|
||||||
|
"""建池收尾:成功标 paused 并记时间,失败删条目;
|
||||||
|
都清忙标记并落盘。返回 VM 名或 None。"""
|
||||||
|
with self._lock:
|
||||||
|
self._busy.discard(user_key)
|
||||||
|
entry = self._entries.get(user_key)
|
||||||
|
if entry is None:
|
||||||
|
return None
|
||||||
|
if not ok:
|
||||||
|
self._entries.pop(user_key, None)
|
||||||
|
self._save_registry()
|
||||||
|
return None
|
||||||
|
entry["state"] = "paused"
|
||||||
|
entry["paused_at"] = time.time()
|
||||||
|
self._save_registry()
|
||||||
|
return str(entry.get("vm"))
|
||||||
|
|
||||||
|
def provision(self, user_id: str | None) -> dict:
|
||||||
|
"""建常驻 VM:已有 paused/running 条目直接返回(不
|
||||||
|
重 wake);否则 wake + pause,成功落 paused 条目,
|
||||||
|
失败删条目回 cold。并发建池只放行一路,其余回 cold。"""
|
||||||
|
user_key = resident_user_key(user_id)
|
||||||
|
resident_task = resident_task_for_user(user_id)
|
||||||
|
with self._lock:
|
||||||
|
entry = self._entries.get(user_key)
|
||||||
|
if entry is not None and entry.get("state") in (
|
||||||
|
"paused", "running"):
|
||||||
|
return {"mode": "resident", "vm": entry.get("vm"),
|
||||||
|
"task": resident_task}
|
||||||
|
if user_key in self._busy:
|
||||||
|
return {"mode": "cold", "vm": None, "task": None}
|
||||||
|
self._busy.add(user_key)
|
||||||
|
self._entries[user_key] = {
|
||||||
|
"state": "provisioning",
|
||||||
|
"vm": self._vm_name(resident_task),
|
||||||
|
"user_id": user_id if user_id is not None
|
||||||
|
else _DEFAULT_USER,
|
||||||
|
"paused_at": None, "last_used": None}
|
||||||
|
self._save_registry()
|
||||||
|
ok = self._wake_and_pause(resident_task)
|
||||||
|
vm = self._finish_provision(user_key, ok)
|
||||||
|
if vm is None:
|
||||||
|
return {"mode": "cold", "vm": None, "task": None}
|
||||||
|
self._emit("resident_pool_provision", resident_task,
|
||||||
|
"vm=%s user=%s" % (vm, user_key))
|
||||||
|
return {"mode": "resident", "vm": vm, "task": resident_task}
|
||||||
|
|
||||||
|
# -- 空闲回收 --------------------------------------------------------
|
||||||
|
def _hibernate_one(self, user_key: str) -> str | None:
|
||||||
|
"""单条目休眠:hibernate_vm(快照到 state_dir 内
|
||||||
|
resident.snapshot,成功后删条目)。失败留条目下次
|
||||||
|
再试,返回 None;无 state_dir 或无 hibernate 入口
|
||||||
|
同样留条目。"""
|
||||||
|
with self._lock:
|
||||||
|
entry = self._entries.get(user_key)
|
||||||
|
if entry is None or entry.get("state") != "paused":
|
||||||
|
return None
|
||||||
|
if user_key in self._busy:
|
||||||
|
return None
|
||||||
|
self._busy.add(user_key)
|
||||||
|
resident_task = resident_task_for_user(
|
||||||
|
entry.get("user_id"))
|
||||||
|
vm = str(entry.get("vm"))
|
||||||
|
state_dir = getattr(self.executor, "state_dir", None)
|
||||||
|
hibernate = getattr(self.executor, "hibernate_vm", None)
|
||||||
|
if not state_dir or not callable(hibernate):
|
||||||
|
with self._lock:
|
||||||
|
self._busy.discard(user_key)
|
||||||
|
return None
|
||||||
|
snapshot_url = "file://" + os.path.join(
|
||||||
|
state_dir, vm, "resident.snapshot")
|
||||||
|
try:
|
||||||
|
hibernate(resident_task, snapshot_url)
|
||||||
|
except Exception:
|
||||||
|
with self._lock:
|
||||||
|
self._busy.discard(user_key)
|
||||||
|
return None
|
||||||
|
with self._lock:
|
||||||
|
self._busy.discard(user_key)
|
||||||
|
self._entries.pop(user_key, None)
|
||||||
|
self._save_registry()
|
||||||
|
self._emit("resident_pool_reclaim", resident_task,
|
||||||
|
"vm=%s snapshot=%s" % (vm, snapshot_url))
|
||||||
|
return vm
|
||||||
|
|
||||||
|
def reclaim_idle(self, now: float | None = None) -> list[str]:
|
||||||
|
"""空闲回收:paused 超阈值的条目 hibernate(pause +
|
||||||
|
snapshot + shutdown)并删条目。阈值 0 即关闭回收;
|
||||||
|
未到阈值/正忙/无 paused_at 的条目不动。返回回收的
|
||||||
|
VM 名。"""
|
||||||
|
if self.idle_minutes <= 0:
|
||||||
|
return []
|
||||||
|
if now is None:
|
||||||
|
now = time.time()
|
||||||
|
due: list[str] = []
|
||||||
|
with self._lock:
|
||||||
|
for key, entry in self._entries.items():
|
||||||
|
if entry.get("state") != "paused":
|
||||||
|
continue
|
||||||
|
if key in self._busy:
|
||||||
|
continue
|
||||||
|
paused_at = entry.get("paused_at")
|
||||||
|
if paused_at is None:
|
||||||
|
continue
|
||||||
|
if now - paused_at > self.idle_minutes * 60:
|
||||||
|
due.append(key)
|
||||||
|
reclaimed: list[str] = []
|
||||||
|
for key in due:
|
||||||
|
vm = self._hibernate_one(key)
|
||||||
|
if vm is not None:
|
||||||
|
reclaimed.append(vm)
|
||||||
|
return reclaimed
|
||||||
|
|
||||||
|
# -- 孤儿清理 --------------------------------------------------------
|
||||||
|
def _shutdown_one(self, user_key: str) -> str | None:
|
||||||
|
"""单条目尽力 shutdown 并从内存删掉(失败仍计入
|
||||||
|
已清理,与 orphans 旧语义一致:不因单条卡住整批)。
|
||||||
|
返回 VM 名或 None(条目已不在)。"""
|
||||||
|
with self._lock:
|
||||||
|
entry = self._entries.get(user_key)
|
||||||
|
if entry is None:
|
||||||
|
return None
|
||||||
|
vm = str(entry.get("vm"))
|
||||||
|
shutdown = getattr(self.executor, "shutdown_vm", None)
|
||||||
|
if callable(shutdown):
|
||||||
|
try:
|
||||||
|
shutdown(vm)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
with self._lock:
|
||||||
|
self._entries.pop(user_key, None)
|
||||||
|
return vm
|
||||||
|
|
||||||
|
def reconcile_orphans(self) -> list[str]:
|
||||||
|
"""清理本池内存里的 orphan 条目(重启后读到的上一
|
||||||
|
世条目):每个尽力 shutdown_vm,条目清掉并落盘。
|
||||||
|
生产启动/关闭不走本方法,走模块级 facade。"""
|
||||||
|
with self._lock:
|
||||||
|
orphans = [k for k, e in self._entries.items()
|
||||||
|
if e.get("state") == "orphan"]
|
||||||
|
cleaned: list[str] = []
|
||||||
|
for key in orphans:
|
||||||
|
vm = self._shutdown_one(key)
|
||||||
|
if vm is not None:
|
||||||
|
cleaned.append(vm)
|
||||||
|
if cleaned:
|
||||||
|
with self._lock:
|
||||||
|
self._save_registry()
|
||||||
|
self._emit("resident_pool_reconcile", {"id": None},
|
||||||
|
"cleaned=%s" % ",".join(cleaned))
|
||||||
|
return cleaned
|
||||||
@@ -0,0 +1,223 @@
|
|||||||
|
"""宿主侧 vsock notify 最小实现(P1,feat-chv-muse-p1)。
|
||||||
|
|
||||||
|
方向只一个:客机 → 宿主。客机连宿主 CID 2 的
|
||||||
|
VSOCK_NOTIFY_PORT 端口发 ready/心跳(客内脚本
|
||||||
|
cloud/chv/guest/vsock_notify.py);宿主在
|
||||||
|
``<vsock socket>_<port>`` 这个 Unix socket 上监听接收。
|
||||||
|
|
||||||
|
形态依据(CHV 官方 docs/vsock.md,upstream 原文):
|
||||||
|
- ``--vsock cid=3,socket=/tmp/ch.vsock``:cid 是客机 CID,
|
||||||
|
socket 是宿主侧 Unix socket 路径;
|
||||||
|
- 客 → 宿主:宿主须先在 ``<socket>_<port>`` 上监听
|
||||||
|
(如 ``/tmp/ch.vsock_1234``),客机连众所周知的 CID 2
|
||||||
|
(Host);
|
||||||
|
- 只支持 stream socket。
|
||||||
|
|
||||||
|
只做就绪辅助:wait_ready() 收到 ready 即返回,供
|
||||||
|
_run_with_vm/常驻路径在 SSH 轮询之外多一个早期信号;
|
||||||
|
SSH 仍是唯一命令通道与就绪判定——vsock 超时、无消息、
|
||||||
|
监听失败一律返回 None,绝不因此判任务失败、绝不抛到
|
||||||
|
调用方。消息是 JSON 行:
|
||||||
|
|
||||||
|
{"type": "ready"|"heartbeat", "vm": "<vm 名>", "ts": <unix 秒>}
|
||||||
|
|
||||||
|
不做宿主 → 客机方向(客机不监听,host.py 协议仍走 SSH)。
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import logging
|
||||||
|
import os
|
||||||
|
import socket
|
||||||
|
import time
|
||||||
|
|
||||||
|
_LOGGER = logging.getLogger("cloud.cloud_hypervisor")
|
||||||
|
|
||||||
|
# 宿主 CID(vsock 众所周知值,CHV 官方 docs/vsock.md)。
|
||||||
|
HOST_CID = 2
|
||||||
|
# 客机 CID 默认值(与 CHV_VSOCK_CID 默认值一致;0/1/2 保留)。
|
||||||
|
DEFAULT_GUEST_CID = 3
|
||||||
|
# notify 端口:客机连 (CID 2, 本端口) 发 ready/心跳;
|
||||||
|
# 宿主监听 <socket>_<本端口>。与客内脚本的端口须一致。
|
||||||
|
VSOCK_NOTIFY_PORT = 1024
|
||||||
|
# 单行上限:客机发来的行超长即丢弃该连接(防脏数据撑内存)。
|
||||||
|
MAX_LINE_BYTES = 65536
|
||||||
|
# 合法消息类型。
|
||||||
|
VALID_TYPES = ("ready", "heartbeat")
|
||||||
|
|
||||||
|
|
||||||
|
def vsock_socket_path(state_dir: str, vm_name: str) -> str:
|
||||||
|
"""该 VM 的 vsock socket 路径(放进 VmConfig 的值)。
|
||||||
|
|
||||||
|
与 build_vm_config 的派生一致:任务盘同目录的
|
||||||
|
``state_dir/<vm>/vsock.sock``。vm_name 须与
|
||||||
|
api_socket/任务盘用的名字相同(executor.vm_name(task)
|
||||||
|
的返回值,含 chv- 前缀),这里不二次加前缀。"""
|
||||||
|
return os.path.join(state_dir, vm_name, "vsock.sock")
|
||||||
|
|
||||||
|
|
||||||
|
def notify_socket_path(socket_path: str,
|
||||||
|
port: int = VSOCK_NOTIFY_PORT) -> str:
|
||||||
|
"""宿主侧监听路径:``<socket>_<port>``(CHV 官方约定)。"""
|
||||||
|
return "%s_%d" % (socket_path, port)
|
||||||
|
|
||||||
|
|
||||||
|
def encode_notify(message_type: str, vm: str,
|
||||||
|
ts: float | None = None) -> bytes:
|
||||||
|
"""编码一条 notify 消息为 JSON 行(含换行)。
|
||||||
|
|
||||||
|
type 非法或 vm 为空即 ValueError(调用方是可信的客内
|
||||||
|
脚本/宿主代码,编码错误是编程错误,早炸)。"""
|
||||||
|
if message_type not in VALID_TYPES:
|
||||||
|
raise ValueError("vsock notify 类型非法:%r" % (message_type,))
|
||||||
|
if not vm:
|
||||||
|
raise ValueError("vsock notify vm 不能为空")
|
||||||
|
if ts is None:
|
||||||
|
ts = time.time()
|
||||||
|
line = json.dumps({"type": message_type, "vm": vm, "ts": ts},
|
||||||
|
separators=(",", ":"))
|
||||||
|
return (line + "\n").encode("utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
def decode_notify(line: bytes | str) -> dict | None:
|
||||||
|
"""解码一行 notify 消息:合法返回 dict,否则返回 None。
|
||||||
|
|
||||||
|
客机字节不可信,任何畸形(非 JSON、非对象、类型不在
|
||||||
|
ready/heartbeat 内、vm 空/非字符串、ts 非数字)一律
|
||||||
|
None,绝不抛异常。"""
|
||||||
|
try:
|
||||||
|
if isinstance(line, bytes):
|
||||||
|
line = line.decode("utf-8")
|
||||||
|
obj = json.loads(line)
|
||||||
|
except (UnicodeDecodeError, ValueError):
|
||||||
|
return None
|
||||||
|
if not isinstance(obj, dict):
|
||||||
|
return None
|
||||||
|
if obj.get("type") not in VALID_TYPES:
|
||||||
|
return None
|
||||||
|
if not isinstance(obj.get("vm"), str) or not obj.get("vm"):
|
||||||
|
return None
|
||||||
|
if not isinstance(obj.get("ts"), (int, float)) \
|
||||||
|
or isinstance(obj.get("ts"), bool):
|
||||||
|
return None
|
||||||
|
return obj
|
||||||
|
|
||||||
|
|
||||||
|
def _bind_notify_listener(listener, notify_path: str) -> bool:
|
||||||
|
"""bind notify socket:残留文件先删一次再重试一次;
|
||||||
|
两次都失败即 False(调用方记 warning 后返回 None,
|
||||||
|
与原来直抛到外层 except 的行为一致)。"""
|
||||||
|
try:
|
||||||
|
listener.bind(notify_path)
|
||||||
|
return True
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
try:
|
||||||
|
os.unlink(notify_path)
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
try:
|
||||||
|
listener.bind(notify_path)
|
||||||
|
return True
|
||||||
|
except OSError:
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def listen_notify(notify_path: str, timeout: float,
|
||||||
|
want_type: str | None = None) -> dict | None:
|
||||||
|
"""在 notify_path(Unix socket)上监听,等一条 notify。
|
||||||
|
|
||||||
|
收到首条合法消息即返回(want_type 给定时跳过其它类型
|
||||||
|
继续等,直到超时);超时、客机断开未发合法消息、监听
|
||||||
|
失败一律返回 None。监听用的 socket 文件用完即删(尽力
|
||||||
|
而为);bind 前已存在的同名残留文件先删一次再 bind。
|
||||||
|
timeout <= 0 即不等,直接返回 None。"""
|
||||||
|
if timeout is None or timeout <= 0:
|
||||||
|
return None
|
||||||
|
deadline = time.monotonic() + timeout
|
||||||
|
listener = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
|
||||||
|
try:
|
||||||
|
if not _bind_notify_listener(listener, notify_path):
|
||||||
|
_LOGGER.warning("vsock notify 监听失败:%s", notify_path)
|
||||||
|
return None
|
||||||
|
listener.listen(1)
|
||||||
|
while True:
|
||||||
|
remaining = deadline - time.monotonic()
|
||||||
|
if remaining <= 0:
|
||||||
|
return None
|
||||||
|
listener.settimeout(remaining)
|
||||||
|
try:
|
||||||
|
conn, _ = listener.accept()
|
||||||
|
except socket.timeout:
|
||||||
|
return None
|
||||||
|
except OSError:
|
||||||
|
_LOGGER.warning("vsock notify accept 失败:%s",
|
||||||
|
notify_path)
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
return _read_notify(conn, deadline, want_type)
|
||||||
|
finally:
|
||||||
|
try:
|
||||||
|
conn.close()
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
except OSError:
|
||||||
|
_LOGGER.warning("vsock notify 监听失败:%s", notify_path)
|
||||||
|
return None
|
||||||
|
finally:
|
||||||
|
try:
|
||||||
|
listener.close()
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
try:
|
||||||
|
os.unlink(notify_path)
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def _read_notify(conn, deadline: float,
|
||||||
|
want_type: str | None) -> dict | None:
|
||||||
|
"""从一条已接受的连接上读 JSON 行,直到合法消息或超时/
|
||||||
|
断开。want_type 给定时跳过其它合法类型继续读。"""
|
||||||
|
try:
|
||||||
|
stream = conn.makefile("rb")
|
||||||
|
except OSError:
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
while True:
|
||||||
|
remaining = deadline - time.monotonic()
|
||||||
|
if remaining <= 0:
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
conn.settimeout(remaining)
|
||||||
|
line = stream.readline(MAX_LINE_BYTES + 1)
|
||||||
|
except (OSError, ValueError):
|
||||||
|
return None
|
||||||
|
if not line:
|
||||||
|
return None
|
||||||
|
if len(line) > MAX_LINE_BYTES:
|
||||||
|
return None
|
||||||
|
msg = decode_notify(line)
|
||||||
|
if msg is None:
|
||||||
|
continue
|
||||||
|
if want_type is not None and msg["type"] != want_type:
|
||||||
|
continue
|
||||||
|
return msg
|
||||||
|
finally:
|
||||||
|
try:
|
||||||
|
stream.close()
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def wait_ready(state_dir: str, vm_name: str,
|
||||||
|
timeout: float = 30,
|
||||||
|
port: int = VSOCK_NOTIFY_PORT) -> dict | None:
|
||||||
|
"""等一台 VM 的 vsock ready(就绪辅助,非 SSH 判定)。
|
||||||
|
|
||||||
|
在 ``state_dir/<vm>/vsock.sock_<port>`` 上监听,直到
|
||||||
|
收到 {"type": "ready"} 或超时。返回 ready 消息 dict,
|
||||||
|
超时/失败返回 None。调用方(_run_with_vm/常驻路径)
|
||||||
|
拿到 None 时继续走 SSH 就绪轮询,绝不因此失败。"""
|
||||||
|
path = notify_socket_path(
|
||||||
|
vsock_socket_path(state_dir, vm_name), port)
|
||||||
|
return listen_notify(path, timeout, want_type="ready")
|
||||||
@@ -12,6 +12,9 @@ cloud.logging_config,不 import cloud.worker,绝无循环。
|
|||||||
(原 Worker.__init__ 里的闭包本体)。
|
(原 Worker.__init__ 里的闭包本体)。
|
||||||
- reconcile_warm_pool:生产残留清理 facade 调用 + 审计
|
- reconcile_warm_pool:生产残留清理 facade 调用 + 审计
|
||||||
(原 Worker._reconcile_warm_pool 的本体)。
|
(原 Worker._reconcile_warm_pool 的本体)。
|
||||||
|
- reconcile_resident_pool:常驻池残留清理 facade 调用 +
|
||||||
|
审计(新,feat-chv-muse-p1;调 resident_pool 的模块级
|
||||||
|
facade,不构造 ResidentPool、不碰实验接口)。
|
||||||
- cancel_watchdog_loop / quota_watchdog_loop:同步路径
|
- cancel_watchdog_loop / quota_watchdog_loop:同步路径
|
||||||
两条看门狗循环(原 Worker._chv_*_watchdog 的本体,
|
两条看门狗循环(原 Worker._chv_*_watchdog 的本体,
|
||||||
self 依赖全部改成回调参数)。
|
self 依赖全部改成回调参数)。
|
||||||
@@ -115,6 +118,38 @@ def reconcile_warm_pool(executor: object, audit_fn: Callable,
|
|||||||
return cleaned
|
return cleaned
|
||||||
|
|
||||||
|
|
||||||
|
def reconcile_resident_pool(executor: object, audit_fn: Callable,
|
||||||
|
when: str) -> list:
|
||||||
|
"""显式调常驻残留清理 facade,返回已清理 VM 名。
|
||||||
|
|
||||||
|
仅 CHV worker 有效(其它执行器返回 [])。实现是
|
||||||
|
cloud.chv.resident_pool.reconcile_resident_orphans_
|
||||||
|
for_executor():直接读 resident-pool.json 做清理,不
|
||||||
|
构造 ResidentPool、不碰实验接口。调用失败不炸主流程,
|
||||||
|
但进审计(失败本身也审计,不静默丢弃)。orphan 条目
|
||||||
|
绝不发给任务,reconcile 只做清理。"""
|
||||||
|
if not isinstance(executor, CloudHypervisorExecutor):
|
||||||
|
return []
|
||||||
|
try:
|
||||||
|
from cloud.chv.resident_pool import (
|
||||||
|
reconcile_resident_orphans_for_executor as _reconcile)
|
||||||
|
cleaned = _reconcile(executor) or []
|
||||||
|
except Exception as exc:
|
||||||
|
try:
|
||||||
|
audit_fn(None, "chv_resident_pool_reconcile_failed",
|
||||||
|
"when=%s error=%s" % (when, exc))
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
return []
|
||||||
|
if cleaned:
|
||||||
|
try:
|
||||||
|
audit_fn(None, "chv_resident_pool_reconcile",
|
||||||
|
"when=%s cleaned=%s" % (when, ",".join(cleaned)))
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
return cleaned
|
||||||
|
|
||||||
|
|
||||||
def cancel_watchdog_loop(task_id: str, vm_name: str | None,
|
def cancel_watchdog_loop(task_id: str, vm_name: str | None,
|
||||||
stop_event: threading.Event,
|
stop_event: threading.Event,
|
||||||
cancel_requested: set,
|
cancel_requested: set,
|
||||||
|
|||||||
@@ -31,6 +31,7 @@ try:
|
|||||||
from cloud.chv.compat215 import REPLACEMENT_POINTS
|
from cloud.chv.compat215 import REPLACEMENT_POINTS
|
||||||
from cloud.chv.compat215 import resolve_inside, tmpfs_isolated
|
from cloud.chv.compat215 import resolve_inside, tmpfs_isolated
|
||||||
from cloud.chv.config import _assemble_chv_config
|
from cloud.chv.config import _assemble_chv_config
|
||||||
|
from cloud.chv.config import _balloon_vsock_fragment
|
||||||
from cloud.chv.config import _base_fragment
|
from cloud.chv.config import _base_fragment
|
||||||
from cloud.chv.config import _disk_fragment
|
from cloud.chv.config import _disk_fragment
|
||||||
from cloud.chv.config import _network_fragment
|
from cloud.chv.config import _network_fragment
|
||||||
@@ -48,6 +49,7 @@ except ImportError:
|
|||||||
from cloud.chv.compat215 import REPLACEMENT_POINTS
|
from cloud.chv.compat215 import REPLACEMENT_POINTS
|
||||||
from cloud.chv.compat215 import resolve_inside, tmpfs_isolated
|
from cloud.chv.compat215 import resolve_inside, tmpfs_isolated
|
||||||
from cloud.chv.config import _assemble_chv_config
|
from cloud.chv.config import _assemble_chv_config
|
||||||
|
from cloud.chv.config import _balloon_vsock_fragment
|
||||||
from cloud.chv.config import _base_fragment
|
from cloud.chv.config import _base_fragment
|
||||||
from cloud.chv.config import _disk_fragment
|
from cloud.chv.config import _disk_fragment
|
||||||
from cloud.chv.config import _network_fragment
|
from cloud.chv.config import _network_fragment
|
||||||
@@ -76,6 +78,8 @@ EXPECTED_FIELDS = {
|
|||||||
"snapshot_max_total_gb", "snapshot_retention_days",
|
"snapshot_max_total_gb", "snapshot_retention_days",
|
||||||
"snapshot_dir", "allow_remote_snapshot", "snapshot_url",
|
"snapshot_dir", "allow_remote_snapshot", "snapshot_url",
|
||||||
"task_timeout", "api_timeout", "ready_timeout", "ready_interval",
|
"task_timeout", "api_timeout", "ready_timeout", "ready_interval",
|
||||||
|
# P1-B(feat-chv-muse-p1):气球/vsock 三字段。
|
||||||
|
"balloon", "vsock", "vsock_cid",
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
@@ -249,13 +253,15 @@ class ConfigFragmentsTest(unittest.TestCase):
|
|||||||
def test_fragment_keys_disjoint_and_union_is_field_set(self):
|
def test_fragment_keys_disjoint_and_union_is_field_set(self):
|
||||||
frags = [_base_fragment({}, {}), _ssh_fragment({}, {}),
|
frags = [_base_fragment({}, {}), _ssh_fragment({}, {}),
|
||||||
_network_fragment({}, {}), _quota_fragment({}),
|
_network_fragment({}, {}), _quota_fragment({}),
|
||||||
_disk_fragment({}), _timeout_fragment({}, {})]
|
_disk_fragment({}), _timeout_fragment({}, {}),
|
||||||
|
# P1-B:新片段加入并集断言(字段 38 → 41)。
|
||||||
|
_balloon_vsock_fragment({})]
|
||||||
seen = set()
|
seen = set()
|
||||||
for frag in frags:
|
for frag in frags:
|
||||||
self.assertFalse(seen & set(frag), "片段键重叠")
|
self.assertFalse(seen & set(frag), "片段键重叠")
|
||||||
seen |= set(frag)
|
seen |= set(frag)
|
||||||
self.assertEqual(seen, EXPECTED_FIELDS)
|
self.assertEqual(seen, EXPECTED_FIELDS)
|
||||||
self.assertEqual(len(EXPECTED_FIELDS), 38)
|
self.assertEqual(len(EXPECTED_FIELDS), 41)
|
||||||
|
|
||||||
def test_defaults_and_overrides(self):
|
def test_defaults_and_overrides(self):
|
||||||
cfg = build_chv_config({}, {})
|
cfg = build_chv_config({}, {})
|
||||||
|
|||||||
@@ -598,8 +598,16 @@ class VmDeathTest(CloseupCase):
|
|||||||
def runner(cmd, timeout=None):
|
def runner(cmd, timeout=None):
|
||||||
self.calls.append(list(cmd))
|
self.calls.append(list(cmd))
|
||||||
if cmd[0] == "ssh" and "dispatch" in cmd:
|
if cmd[0] == "ssh" and "dispatch" in cmd:
|
||||||
proc = ex._vms.get("chv-abc123")
|
# 死亡模拟须杀当时已登记的 VM(即实际跑命令
|
||||||
if proc is not None:
|
# 的那台):P1 常驻路由下 dispatch 跑在常驻 VM
|
||||||
|
#(chv-resident-*)上,任务自己的 chv-abc123
|
||||||
|
# 从未 wake、登记里根本没有,盯着它杀是空操
|
||||||
|
# 作、模拟落空(探针实证:登记里只有
|
||||||
|
# chv-resident-default 且活着,异常直抛)。
|
||||||
|
# 常驻 VM 真死时 _run_on_resident 的判定已验
|
||||||
|
# 证正确(两态皆 interrupted),这里只修模拟
|
||||||
|
# 器的靶子,不动断言与判定逻辑。
|
||||||
|
for proc in list(ex._vms.values()):
|
||||||
proc._alive = False
|
proc._alive = False
|
||||||
if mode == "raise":
|
if mode == "raise":
|
||||||
raise RuntimeError("ssh broken")
|
raise RuntimeError("ssh broken")
|
||||||
|
|||||||
@@ -0,0 +1,423 @@
|
|||||||
|
"""P1-B(feat-chv-muse-p1):气球/vsock 配置、payload 与 vsock 编解码测试。
|
||||||
|
|
||||||
|
覆盖三件:
|
||||||
|
|
||||||
|
- config:CHV_BALLOON/CHV_VSOCK 开关默认开、只认 0/1,
|
||||||
|
CHV_VSOCK_CID 默认 3 且拒绝保留 CID(0/1/2)与非法值;
|
||||||
|
- executor:build_vm_config 默认带 balloon/vsock 设备,开关
|
||||||
|
关时对应键整个不出现,vsock socket 路径按任务盘派生;
|
||||||
|
- vsock(宿主侧)与客内脚本:JSON 行编解码、监听/超时、
|
||||||
|
客内脚本静默失败(连不上宿主即 exit 0,不影响 sshd/任务)。
|
||||||
|
|
||||||
|
全部用假执行器与本地 Unix socket,绝不连真实网络、不起
|
||||||
|
真实 VM。CHV 54 的 balloon size=0 与 vsock 端到端收发尚未
|
||||||
|
真机验证(本机无 /dev/kvm,第三棒占着 WSL 测试机),见
|
||||||
|
汇报的待验证项——这里只钉住配置层 + payload 形态。
|
||||||
|
"""
|
||||||
|
|
||||||
|
import ast
|
||||||
|
import importlib.util
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import socket
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import threading
|
||||||
|
import time
|
||||||
|
import unittest
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from cloud.chv import vsock as chv_vsock
|
||||||
|
from cloud.chv.config import _balloon_vsock_fragment
|
||||||
|
from cloud.chv.config import _base_fragment
|
||||||
|
from cloud.chv.config import _disk_fragment
|
||||||
|
from cloud.chv.config import _network_fragment
|
||||||
|
from cloud.chv.config import _quota_fragment
|
||||||
|
from cloud.chv.config import _ssh_fragment
|
||||||
|
from cloud.chv.config import _timeout_fragment
|
||||||
|
from cloud.chv.config import build_chv_config
|
||||||
|
from cloud.chv.executor import CloudHypervisorExecutor
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parents[2]
|
||||||
|
GUEST_SCRIPT = ROOT / "cloud" / "chv" / "guest" / "vsock_notify.py"
|
||||||
|
|
||||||
|
|
||||||
|
def _load_guest_module():
|
||||||
|
spec = importlib.util.spec_from_file_location(
|
||||||
|
"vsock_notify_guest", str(GUEST_SCRIPT))
|
||||||
|
module = importlib.util.module_from_spec(spec)
|
||||||
|
spec.loader.exec_module(module)
|
||||||
|
return module
|
||||||
|
|
||||||
|
|
||||||
|
def _stdlib_only(path):
|
||||||
|
tree = ast.parse(Path(path).read_text())
|
||||||
|
mods = set()
|
||||||
|
for node in ast.walk(tree):
|
||||||
|
if isinstance(node, ast.Import):
|
||||||
|
mods.update(a.name.split(".")[0] for a in node.names)
|
||||||
|
elif isinstance(node, ast.ImportFrom) and node.module:
|
||||||
|
mods.add(node.module.split(".")[0])
|
||||||
|
return mods <= set(sys.stdlib_module_names)
|
||||||
|
|
||||||
|
|
||||||
|
class BalloonVsockConfigTest(unittest.TestCase):
|
||||||
|
def test_defaults_on(self):
|
||||||
|
cfg = build_chv_config({}, {})
|
||||||
|
self.assertTrue(cfg.balloon)
|
||||||
|
self.assertTrue(cfg.vsock)
|
||||||
|
self.assertEqual(cfg.vsock_cid, 3)
|
||||||
|
|
||||||
|
def test_switches_off(self):
|
||||||
|
cfg = build_chv_config(
|
||||||
|
{}, {"CHV_BALLOON": "0", "CHV_VSOCK": "0"})
|
||||||
|
self.assertFalse(cfg.balloon)
|
||||||
|
self.assertFalse(cfg.vsock)
|
||||||
|
self.assertEqual(cfg.vsock_cid, 3)
|
||||||
|
|
||||||
|
def test_switches_on_explicit(self):
|
||||||
|
cfg = build_chv_config(
|
||||||
|
{}, {"CHV_BALLOON": "1", "CHV_VSOCK": "1"})
|
||||||
|
self.assertTrue(cfg.balloon)
|
||||||
|
self.assertTrue(cfg.vsock)
|
||||||
|
|
||||||
|
def test_illegal_switch_values_raise(self):
|
||||||
|
for env in ({"CHV_BALLOON": "2"}, {"CHV_BALLOON": "yes"},
|
||||||
|
{"CHV_BALLOON": "true"}, {"CHV_VSOCK": "on"},
|
||||||
|
{"CHV_VSOCK": ""}):
|
||||||
|
with self.assertRaises(ValueError):
|
||||||
|
build_chv_config({}, env)
|
||||||
|
|
||||||
|
def test_vsock_cid_override(self):
|
||||||
|
cfg = build_chv_config({}, {"CHV_VSOCK_CID": "7"})
|
||||||
|
self.assertEqual(cfg.vsock_cid, 7)
|
||||||
|
cfg2 = build_chv_config({}, {"CHV_VSOCK_CID": "4294967295"})
|
||||||
|
self.assertEqual(cfg2.vsock_cid, 4294967295)
|
||||||
|
|
||||||
|
def test_vsock_cid_reserved_and_illegal_raise(self):
|
||||||
|
# 0/1/2 是 vsock 保留 CID(Hypervisor/Loopback/Host),
|
||||||
|
# -1 随机 CID 也被 CHV vm.create 400 拒绝。
|
||||||
|
for cid in ("0", "1", "2", "-1", "4294967296", "abc", ""):
|
||||||
|
with self.assertRaises(ValueError, msg=cid):
|
||||||
|
build_chv_config({}, {"CHV_VSOCK_CID": cid})
|
||||||
|
|
||||||
|
def test_fragment_keys_disjoint(self):
|
||||||
|
frag = _balloon_vsock_fragment({})
|
||||||
|
self.assertEqual(set(frag), {"balloon", "vsock", "vsock_cid"})
|
||||||
|
others = [_base_fragment({}, {}), _ssh_fragment({}, {}),
|
||||||
|
_network_fragment({}, {}), _quota_fragment({}),
|
||||||
|
_disk_fragment({}), _timeout_fragment({}, {})]
|
||||||
|
seen = set()
|
||||||
|
for other in others:
|
||||||
|
seen |= set(other)
|
||||||
|
self.assertFalse(seen & set(frag), "新片段键与旧片段冲突")
|
||||||
|
|
||||||
|
|
||||||
|
class _PayloadCase(unittest.TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
self.dir = tempfile.mkdtemp(prefix="chv-p1b-")
|
||||||
|
self.state_dir = os.path.join(self.dir, "state")
|
||||||
|
os.makedirs(self.state_dir)
|
||||||
|
|
||||||
|
def tearDown(self):
|
||||||
|
import shutil
|
||||||
|
shutil.rmtree(self.dir, ignore_errors=True)
|
||||||
|
|
||||||
|
def make_exec(self, env_extra=None):
|
||||||
|
env = dict(env_extra or {})
|
||||||
|
return CloudHypervisorExecutor(
|
||||||
|
kernel=os.path.join(self.dir, "vmlinux"),
|
||||||
|
rootfs=os.path.join(self.dir, "root.raw"),
|
||||||
|
ssh_key=os.path.join(self.dir, "id_ed25519"),
|
||||||
|
state_dir=self.state_dir,
|
||||||
|
runner=lambda cmd, timeout=None: None,
|
||||||
|
launcher=lambda cmd: None,
|
||||||
|
api_client=lambda *a: (200, {}),
|
||||||
|
sleeper=lambda s: None, env=env)
|
||||||
|
|
||||||
|
|
||||||
|
class BalloonVsockPayloadTest(_PayloadCase):
|
||||||
|
def test_default_payload_has_balloon_and_vsock(self):
|
||||||
|
ex = self.make_exec()
|
||||||
|
cfg = ex.build_vm_config()
|
||||||
|
# CHV 54 BalloonConfig:size 是目标气球尺寸(字节),
|
||||||
|
# 0 = 启动不预扣,只靠 free_page_reporting 被动回收。
|
||||||
|
self.assertEqual(cfg["balloon"], {
|
||||||
|
"size": 0, "deflate_on_oom": True,
|
||||||
|
"free_page_reporting": True})
|
||||||
|
self.assertEqual(cfg["vsock"], {
|
||||||
|
"cid": 3,
|
||||||
|
"socket": os.path.join(self.state_dir, "vsock.sock")})
|
||||||
|
|
||||||
|
def test_switches_off_drop_keys(self):
|
||||||
|
ex = self.make_exec({"CHV_BALLOON": "0", "CHV_VSOCK": "0"})
|
||||||
|
cfg = ex.build_vm_config()
|
||||||
|
self.assertNotIn("balloon", cfg)
|
||||||
|
self.assertNotIn("vsock", cfg)
|
||||||
|
# 旧键原样保留。
|
||||||
|
self.assertIn("cpus", cfg)
|
||||||
|
self.assertIn("memory", cfg)
|
||||||
|
self.assertIn("payload", cfg)
|
||||||
|
self.assertIn("disks", cfg)
|
||||||
|
self.assertIn("rng", cfg)
|
||||||
|
|
||||||
|
def test_balloon_off_only(self):
|
||||||
|
ex = self.make_exec({"CHV_BALLOON": "0"})
|
||||||
|
cfg = ex.build_vm_config()
|
||||||
|
self.assertNotIn("balloon", cfg)
|
||||||
|
self.assertIn("vsock", cfg)
|
||||||
|
|
||||||
|
def test_vsock_cid_honored(self):
|
||||||
|
ex = self.make_exec({"CHV_VSOCK_CID": "9"})
|
||||||
|
cfg = ex.build_vm_config()
|
||||||
|
self.assertEqual(cfg["vsock"]["cid"], 9)
|
||||||
|
|
||||||
|
def test_vsock_socket_derived_from_task_disk_dir(self):
|
||||||
|
ex = self.make_exec()
|
||||||
|
disk = os.path.join(self.state_dir, "chv-t1", "disk.raw")
|
||||||
|
cfg = ex.build_vm_config(disk)
|
||||||
|
self.assertEqual(
|
||||||
|
cfg["vsock"]["socket"],
|
||||||
|
os.path.join(self.state_dir, "chv-t1", "vsock.sock"))
|
||||||
|
|
||||||
|
def test_vsock_socket_falls_back_when_disk_outside_state(self):
|
||||||
|
# reuse/lab:任务盘即基础镜像(state_dir 外),socket
|
||||||
|
# 落 state_dir/vsock.sock(单 VM 可用,见 helper 注释)。
|
||||||
|
ex = self.make_exec()
|
||||||
|
cfg = ex.build_vm_config()
|
||||||
|
self.assertEqual(
|
||||||
|
cfg["vsock"]["socket"],
|
||||||
|
os.path.join(self.state_dir, "vsock.sock"))
|
||||||
|
|
||||||
|
def test_explicit_vsock_socket_wins(self):
|
||||||
|
ex = self.make_exec()
|
||||||
|
cfg = ex.build_vm_config(
|
||||||
|
vsock_socket="/tmp/per-vm/v.sock")
|
||||||
|
self.assertEqual(cfg["vsock"]["socket"], "/tmp/per-vm/v.sock")
|
||||||
|
|
||||||
|
def test_helper_direct(self):
|
||||||
|
ex = self.make_exec({"CHV_BALLOON": "0"})
|
||||||
|
extra = ex._balloon_vsock_payload(
|
||||||
|
vsock_socket="/tmp/s.sock")
|
||||||
|
self.assertEqual(extra, {"vsock": {"cid": 3,
|
||||||
|
"socket": "/tmp/s.sock"}})
|
||||||
|
|
||||||
|
|
||||||
|
class VsockCodecTest(unittest.TestCase):
|
||||||
|
def test_encode_decode_round_trip(self):
|
||||||
|
raw = chv_vsock.encode_notify("ready", "chv-t1", ts=1234.5)
|
||||||
|
self.assertTrue(raw.endswith(b"\n"))
|
||||||
|
msg = chv_vsock.decode_notify(raw)
|
||||||
|
self.assertEqual(msg, {"type": "ready", "vm": "chv-t1",
|
||||||
|
"ts": 1234.5})
|
||||||
|
|
||||||
|
def test_encode_rejects_bad_type_or_vm(self):
|
||||||
|
with self.assertRaises(ValueError):
|
||||||
|
chv_vsock.encode_notify("bogus", "chv-t1")
|
||||||
|
with self.assertRaises(ValueError):
|
||||||
|
chv_vsock.encode_notify("ready", "")
|
||||||
|
|
||||||
|
def test_decode_rejects_garbage(self):
|
||||||
|
cases = [b"not json\n", b"[1, 2]\n", b"123\n", b"\n",
|
||||||
|
b'{"type": "bogus", "vm": "v", "ts": 1}\n',
|
||||||
|
b'{"type": "ready", "vm": "", "ts": 1}\n',
|
||||||
|
b'{"type": "ready", "vm": 7, "ts": 1}\n',
|
||||||
|
b'{"type": "ready", "vm": "v"}\n',
|
||||||
|
b'{"type": "ready", "vm": "v", "ts": "x"}\n',
|
||||||
|
b'{"type": "ready", "vm": "v", "ts": true}\n',
|
||||||
|
b'\xff\xfe bad utf8\n']
|
||||||
|
for line in cases:
|
||||||
|
self.assertIsNone(chv_vsock.decode_notify(line), line)
|
||||||
|
|
||||||
|
def test_decode_accepts_int_ts(self):
|
||||||
|
msg = chv_vsock.decode_notify(
|
||||||
|
b'{"type": "heartbeat", "vm": "v", "ts": 7}')
|
||||||
|
self.assertEqual(msg["ts"], 7)
|
||||||
|
|
||||||
|
def test_paths(self):
|
||||||
|
self.assertEqual(
|
||||||
|
chv_vsock.vsock_socket_path("/s", "chv-t1"),
|
||||||
|
"/s/chv-t1/vsock.sock")
|
||||||
|
self.assertEqual(
|
||||||
|
chv_vsock.notify_socket_path("/s/chv-t1/vsock.sock"),
|
||||||
|
"/s/chv-t1/vsock.sock_1024")
|
||||||
|
self.assertEqual(chv_vsock.HOST_CID, 2)
|
||||||
|
self.assertEqual(chv_vsock.VSOCK_NOTIFY_PORT, 1024)
|
||||||
|
|
||||||
|
|
||||||
|
def _wait_for_socket(path, timeout=2.0):
|
||||||
|
deadline = time.monotonic() + timeout
|
||||||
|
while time.monotonic() < deadline:
|
||||||
|
if os.path.exists(path):
|
||||||
|
return True
|
||||||
|
time.sleep(0.02)
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
class VsockListenTest(unittest.TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
self.dir = tempfile.mkdtemp(prefix="chv-p1b-vsock-")
|
||||||
|
|
||||||
|
def tearDown(self):
|
||||||
|
import shutil
|
||||||
|
shutil.rmtree(self.dir, ignore_errors=True)
|
||||||
|
|
||||||
|
def _client(self, path, lines, delay=0.05):
|
||||||
|
"""后台线程:等 socket 出现,连上发 lines 后关闭。
|
||||||
|
|
||||||
|
connect 带重试:监听方 bind 与 listen 之间、以及删
|
||||||
|
残留文件重 bind 的窗口里,单次 connect 可能被拒;
|
||||||
|
须重试到成功或超时,否则测试会偶发失败。"""
|
||||||
|
def run():
|
||||||
|
if not _wait_for_socket(path):
|
||||||
|
return
|
||||||
|
deadline = time.monotonic() + 5
|
||||||
|
sock = None
|
||||||
|
while time.monotonic() < deadline:
|
||||||
|
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
|
||||||
|
try:
|
||||||
|
sock.connect(path)
|
||||||
|
break
|
||||||
|
except OSError:
|
||||||
|
try:
|
||||||
|
sock.close()
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
sock = None
|
||||||
|
time.sleep(0.02)
|
||||||
|
if sock is None:
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
time.sleep(delay)
|
||||||
|
for line in lines:
|
||||||
|
sock.sendall(line)
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
finally:
|
||||||
|
try:
|
||||||
|
sock.close()
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
thread = threading.Thread(target=run, daemon=True)
|
||||||
|
thread.start()
|
||||||
|
return thread
|
||||||
|
|
||||||
|
def test_listen_returns_ready(self):
|
||||||
|
path = os.path.join(self.dir, "n.sock_1024")
|
||||||
|
thread = self._client(
|
||||||
|
path, [chv_vsock.encode_notify("ready", "chv-t1")])
|
||||||
|
msg = chv_vsock.listen_notify(path, timeout=5)
|
||||||
|
thread.join(timeout=5)
|
||||||
|
self.assertIsNotNone(msg)
|
||||||
|
self.assertEqual(msg["type"], "ready")
|
||||||
|
self.assertEqual(msg["vm"], "chv-t1")
|
||||||
|
self.assertFalse(os.path.exists(path), "监听完须清 socket 文件")
|
||||||
|
|
||||||
|
def test_listen_timeout_returns_none(self):
|
||||||
|
path = os.path.join(self.dir, "t.sock_1024")
|
||||||
|
start = time.monotonic()
|
||||||
|
self.assertIsNone(chv_vsock.listen_notify(path, timeout=0.3))
|
||||||
|
self.assertLess(time.monotonic() - start, 2)
|
||||||
|
self.assertFalse(os.path.exists(path))
|
||||||
|
|
||||||
|
def test_listen_skips_invalid_lines_and_wrong_type(self):
|
||||||
|
path = os.path.join(self.dir, "w.sock_1024")
|
||||||
|
thread = self._client(path, [
|
||||||
|
b"not json\n",
|
||||||
|
chv_vsock.encode_notify("heartbeat", "chv-t1"),
|
||||||
|
chv_vsock.encode_notify("ready", "chv-t1"),
|
||||||
|
])
|
||||||
|
msg = chv_vsock.listen_notify(path, timeout=5,
|
||||||
|
want_type="ready")
|
||||||
|
thread.join(timeout=5)
|
||||||
|
self.assertIsNotNone(msg)
|
||||||
|
self.assertEqual(msg["type"], "ready")
|
||||||
|
|
||||||
|
def test_listen_survives_stale_socket_file(self):
|
||||||
|
path = os.path.join(self.dir, "s.sock_1024")
|
||||||
|
with open(path, "w") as fh:
|
||||||
|
fh.write("stale")
|
||||||
|
thread = self._client(
|
||||||
|
path, [chv_vsock.encode_notify("ready", "chv-t1")])
|
||||||
|
msg = chv_vsock.listen_notify(path, timeout=5)
|
||||||
|
thread.join(timeout=5)
|
||||||
|
self.assertIsNotNone(msg)
|
||||||
|
self.assertEqual(msg["type"], "ready")
|
||||||
|
|
||||||
|
def test_wait_ready_end_to_end(self):
|
||||||
|
vm_dir = os.path.join(self.dir, "chv-t1")
|
||||||
|
os.makedirs(vm_dir)
|
||||||
|
path = chv_vsock.notify_socket_path(
|
||||||
|
chv_vsock.vsock_socket_path(self.dir, "chv-t1"))
|
||||||
|
thread = self._client(
|
||||||
|
path, [chv_vsock.encode_notify("ready", "chv-t1")])
|
||||||
|
msg = chv_vsock.wait_ready(self.dir, "chv-t1", timeout=5)
|
||||||
|
thread.join(timeout=5)
|
||||||
|
self.assertIsNotNone(msg)
|
||||||
|
self.assertEqual(msg["type"], "ready")
|
||||||
|
self.assertEqual(msg["vm"], "chv-t1")
|
||||||
|
|
||||||
|
def test_wait_ready_timeout_returns_none(self):
|
||||||
|
vm_dir = os.path.join(self.dir, "chv-t2")
|
||||||
|
os.makedirs(vm_dir)
|
||||||
|
self.assertIsNone(
|
||||||
|
chv_vsock.wait_ready(self.dir, "chv-t2", timeout=0.3))
|
||||||
|
|
||||||
|
|
||||||
|
class GuestNotifyScriptTest(unittest.TestCase):
|
||||||
|
def test_script_exists_and_stdlib_only(self):
|
||||||
|
self.assertTrue(GUEST_SCRIPT.is_file())
|
||||||
|
self.assertTrue(_stdlib_only(GUEST_SCRIPT))
|
||||||
|
self.assertTrue(
|
||||||
|
_stdlib_only(ROOT / "cloud" / "chv" / "vsock.py"))
|
||||||
|
|
||||||
|
def test_vm_name_sources(self):
|
||||||
|
mod = _load_guest_module()
|
||||||
|
self.assertEqual(mod._vm_name(["prog", "chv-t1"]), "chv-t1")
|
||||||
|
old = os.environ.pop("CHV_VM_NAME", None)
|
||||||
|
try:
|
||||||
|
os.environ["CHV_VM_NAME"] = "chv-env"
|
||||||
|
self.assertEqual(mod._vm_name(["prog"]), "chv-env")
|
||||||
|
finally:
|
||||||
|
if old is None:
|
||||||
|
os.environ.pop("CHV_VM_NAME", None)
|
||||||
|
else:
|
||||||
|
os.environ["CHV_VM_NAME"] = old
|
||||||
|
name = mod._vm_name(["prog"])
|
||||||
|
self.assertIsInstance(name, str)
|
||||||
|
self.assertTrue(name)
|
||||||
|
|
||||||
|
def test_send_writes_json_line(self):
|
||||||
|
mod = _load_guest_module()
|
||||||
|
client, server = socket.socketpair()
|
||||||
|
try:
|
||||||
|
mod._send(client, "heartbeat", "chv-t1")
|
||||||
|
server.settimeout(2)
|
||||||
|
data = server.recv(4096)
|
||||||
|
msg = json.loads(data.decode("utf-8"))
|
||||||
|
self.assertEqual(msg["type"], "heartbeat")
|
||||||
|
self.assertEqual(msg["vm"], "chv-t1")
|
||||||
|
self.assertIsInstance(msg["ts"], float)
|
||||||
|
finally:
|
||||||
|
client.close()
|
||||||
|
server.close()
|
||||||
|
|
||||||
|
def test_main_exits_silently_when_host_unreachable(self):
|
||||||
|
# 本机无 vsock 监听:main 应在重试窗口内静默返回 0,
|
||||||
|
# 绝不抛异常、不影响调用方。
|
||||||
|
mod = _load_guest_module()
|
||||||
|
old = mod.CONNECT_RETRY_SECONDS
|
||||||
|
mod.CONNECT_RETRY_SECONDS = 0
|
||||||
|
try:
|
||||||
|
self.assertEqual(mod.main(["prog", "chv-t1"]), 0)
|
||||||
|
finally:
|
||||||
|
mod.CONNECT_RETRY_SECONDS = old
|
||||||
|
|
||||||
|
def test_ports_agree(self):
|
||||||
|
mod = _load_guest_module()
|
||||||
|
self.assertEqual(mod.NOTIFY_PORT, chv_vsock.VSOCK_NOTIFY_PORT)
|
||||||
|
self.assertEqual(mod.HOST_CID, chv_vsock.HOST_CID)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -0,0 +1,730 @@
|
|||||||
|
"""CHV 常驻池(feat-chv-muse-p1)测试。
|
||||||
|
|
||||||
|
覆盖 cloud/chv/resident_pool.py、executor 的常驻接线
|
||||||
|
(_run_with_vm 常驻/冷启动两路、finally 语义)与
|
||||||
|
worker_adapter.reconcile_resident_pool facade。全部用假
|
||||||
|
执行器/假 VMM,绝不起真实 VM、不连真实网络。
|
||||||
|
|
||||||
|
断言点(按任务书):
|
||||||
|
- acquire/resume 失败回落冷启动,失败条目丢弃;
|
||||||
|
- release 只 pause 不销毁,pause 失败条目留 running;
|
||||||
|
- registry 损坏 quarantine、单坏条目不连坐;
|
||||||
|
- 重启后条目一律 orphan 不复用,只能 reconcile 清理;
|
||||||
|
- _run_with_vm 常驻命中跑完 pause、冷启动跑完 shutdown,
|
||||||
|
客内抛错时两路 finally 都执行;
|
||||||
|
- 带密钥 env / 未启用 / 建池失败时回落冷启动,任务不丢;
|
||||||
|
- 每用户隔离:release 只动本用户条目;
|
||||||
|
- 空闲回收只动超阈值 paused 条目,阈值 0 即关闭;
|
||||||
|
- WarmPool 实验模块不动(wired_to_dispatch 仍 False)。
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import tempfile
|
||||||
|
import unittest
|
||||||
|
from types import SimpleNamespace
|
||||||
|
from unittest import mock
|
||||||
|
|
||||||
|
from cloud.chv import worker_adapter
|
||||||
|
from cloud.chv.errors import ChvFailure
|
||||||
|
from cloud.chv.executor import CloudHypervisorExecutor
|
||||||
|
from cloud.chv.experimental_pool import WarmPool
|
||||||
|
from cloud.chv.resident_pool import (
|
||||||
|
ResidentPool,
|
||||||
|
reconcile_resident_orphans_for_executor,
|
||||||
|
resident_task_for_user,
|
||||||
|
resident_user_key,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class _FakeExecutor:
|
||||||
|
"""ResidentPool 用的鸭子型执行器:记录调用,可注入失败。
|
||||||
|
|
||||||
|
wait 失败时模拟真实语义:先自行 shutdown 再抛错
|
||||||
|
(wait_guest_ready 的失败路径就是 stop_vm 后 raise)。
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, state_dir, env=None):
|
||||||
|
self.state_dir = state_dir
|
||||||
|
self._env = env or {}
|
||||||
|
self.resumes = []
|
||||||
|
self.pauses = []
|
||||||
|
self.wakes = []
|
||||||
|
self.waits = []
|
||||||
|
self.shutdowns = []
|
||||||
|
self.hibernates = []
|
||||||
|
self.emits = []
|
||||||
|
self.resume_fail = False
|
||||||
|
self.wait_fail = False
|
||||||
|
self.wake_fail = False
|
||||||
|
self.pause_fail = False
|
||||||
|
self.hibernate_fail = False
|
||||||
|
self.shutdown_fail = False
|
||||||
|
|
||||||
|
def vm_name(self, task_or_name):
|
||||||
|
raw = task_or_name.get("id") if isinstance(
|
||||||
|
task_or_name, dict) else str(task_or_name)
|
||||||
|
safe = re.sub(r"[^A-Za-z0-9_.-]", "-", raw)
|
||||||
|
if safe.startswith("chv-"):
|
||||||
|
return safe
|
||||||
|
return "chv-" + safe
|
||||||
|
|
||||||
|
def resume_vm(self, task_or_name):
|
||||||
|
self.resumes.append(self.vm_name(task_or_name))
|
||||||
|
if self.resume_fail:
|
||||||
|
raise RuntimeError("resume boom")
|
||||||
|
return True
|
||||||
|
|
||||||
|
def pause_vm(self, task_or_name):
|
||||||
|
self.pauses.append(self.vm_name(task_or_name))
|
||||||
|
if self.pause_fail:
|
||||||
|
raise RuntimeError("pause boom")
|
||||||
|
return True
|
||||||
|
|
||||||
|
def wake_vm(self, task_or_name):
|
||||||
|
self.wakes.append(self.vm_name(task_or_name))
|
||||||
|
if self.wake_fail:
|
||||||
|
raise RuntimeError("wake boom")
|
||||||
|
return {"vm": self.vm_name(task_or_name), "mode": "cold"}
|
||||||
|
|
||||||
|
def wait_guest_ready(self, task):
|
||||||
|
self.waits.append(self.vm_name(task))
|
||||||
|
if self.wait_fail:
|
||||||
|
self.shutdown_vm(task)
|
||||||
|
raise ChvFailure("guest_not_ready", "ssh 未就绪")
|
||||||
|
return True
|
||||||
|
|
||||||
|
def shutdown_vm(self, task_or_name):
|
||||||
|
self.shutdowns.append(self.vm_name(task_or_name))
|
||||||
|
if self.shutdown_fail:
|
||||||
|
raise RuntimeError("shutdown boom")
|
||||||
|
return True
|
||||||
|
|
||||||
|
def hibernate_vm(self, task_or_name, snapshot_url):
|
||||||
|
self.hibernates.append(
|
||||||
|
(self.vm_name(task_or_name), snapshot_url))
|
||||||
|
if self.hibernate_fail:
|
||||||
|
raise RuntimeError("hibernate boom")
|
||||||
|
return {"vm": self.vm_name(task_or_name),
|
||||||
|
"state": "hibernated", "snapshot": snapshot_url}
|
||||||
|
|
||||||
|
def _emit(self, event, task, detail="", kind=None):
|
||||||
|
self.emits.append(event)
|
||||||
|
|
||||||
|
|
||||||
|
def _write(path, content):
|
||||||
|
with open(path, "w") as fh:
|
||||||
|
fh.write(content)
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
def _registry_path(state_dir):
|
||||||
|
return os.path.join(state_dir, "resident-pool.json")
|
||||||
|
|
||||||
|
|
||||||
|
class ResidentKeyTest(unittest.TestCase):
|
||||||
|
def test_user_key_sanitizes_and_defaults(self):
|
||||||
|
self.assertEqual(resident_user_key(None), "default")
|
||||||
|
self.assertEqual(resident_user_key(""), "default")
|
||||||
|
self.assertEqual(resident_user_key("alice"), "alice")
|
||||||
|
self.assertEqual(resident_user_key("a/b c"), "a-b-c")
|
||||||
|
self.assertEqual(resident_task_for_user("alice"),
|
||||||
|
{"id": "resident-alice"})
|
||||||
|
self.assertEqual(resident_task_for_user(None),
|
||||||
|
{"id": "resident-default"})
|
||||||
|
|
||||||
|
|
||||||
|
class ResidentPoolAcquireTest(unittest.TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
self.tmp = tempfile.TemporaryDirectory()
|
||||||
|
self.fake = _FakeExecutor(self.tmp.name)
|
||||||
|
|
||||||
|
def tearDown(self):
|
||||||
|
self.tmp.cleanup()
|
||||||
|
|
||||||
|
def _provisioned_pool(self, user="alice"):
|
||||||
|
pool = ResidentPool(self.fake)
|
||||||
|
got = pool.provision(user)
|
||||||
|
self.assertEqual(got["mode"], "resident")
|
||||||
|
return pool
|
||||||
|
|
||||||
|
def test_provision_then_acquire_then_release(self):
|
||||||
|
pool = self._provisioned_pool()
|
||||||
|
self.assertEqual(self.fake.wakes, ["chv-resident-alice"])
|
||||||
|
self.assertEqual(self.fake.pauses, ["chv-resident-alice"])
|
||||||
|
self.assertEqual(pool.paused_count(), 1)
|
||||||
|
got = pool.acquire("alice")
|
||||||
|
self.assertEqual(got["mode"], "resident")
|
||||||
|
self.assertEqual(got["vm"], "chv-resident-alice")
|
||||||
|
self.assertEqual(got["task"], {"id": "resident-alice"})
|
||||||
|
self.assertEqual(self.fake.resumes, ["chv-resident-alice"])
|
||||||
|
self.assertEqual(self.fake.waits, ["chv-resident-alice"])
|
||||||
|
self.assertEqual(pool.paused_count(), 0)
|
||||||
|
# running 中的条目不接受第二个 acquire(回落冷启动)。
|
||||||
|
self.assertEqual(pool.acquire("alice")["mode"], "cold")
|
||||||
|
self.assertEqual(len(self.fake.resumes), 1)
|
||||||
|
# release 只 pause,不销毁。
|
||||||
|
self.assertTrue(pool.release("alice"))
|
||||||
|
self.assertEqual(
|
||||||
|
self.fake.pauses,
|
||||||
|
["chv-resident-alice", "chv-resident-alice"])
|
||||||
|
self.assertEqual(self.fake.shutdowns, [])
|
||||||
|
self.assertEqual(pool.paused_count(), 1)
|
||||||
|
# 再取仍是同一台 VM。
|
||||||
|
got = pool.acquire("alice")
|
||||||
|
self.assertEqual(got["vm"], "chv-resident-alice")
|
||||||
|
|
||||||
|
def test_resume_failure_falls_back_to_cold_and_drops_entry(self):
|
||||||
|
pool = self._provisioned_pool()
|
||||||
|
self.fake.resume_fail = True
|
||||||
|
got = pool.acquire("alice")
|
||||||
|
self.assertEqual(got, {"mode": "cold", "vm": None,
|
||||||
|
"task": None})
|
||||||
|
self.assertEqual(pool.paused_count(), 0)
|
||||||
|
with open(_registry_path(self.tmp.name)) as fh:
|
||||||
|
self.assertEqual(json.load(fh), {})
|
||||||
|
|
||||||
|
def test_wait_failure_falls_back_to_cold_and_drops_entry(self):
|
||||||
|
pool = self._provisioned_pool()
|
||||||
|
self.fake.wait_fail = True
|
||||||
|
got = pool.acquire("alice")
|
||||||
|
self.assertEqual(got["mode"], "cold")
|
||||||
|
self.assertEqual(pool.paused_count(), 0)
|
||||||
|
# wait 的失败路径已自行 stop_vm(模拟真实语义)。
|
||||||
|
self.assertEqual(self.fake.shutdowns,
|
||||||
|
["chv-resident-alice"])
|
||||||
|
with open(_registry_path(self.tmp.name)) as fh:
|
||||||
|
self.assertEqual(json.load(fh), {})
|
||||||
|
|
||||||
|
def test_release_without_running_entry_is_false(self):
|
||||||
|
pool = self._provisioned_pool()
|
||||||
|
self.assertFalse(pool.release("alice"))
|
||||||
|
self.assertFalse(pool.release("nobody"))
|
||||||
|
self.assertEqual(self.fake.pauses, ["chv-resident-alice"])
|
||||||
|
|
||||||
|
def test_release_pause_failure_keeps_entry_running(self):
|
||||||
|
pool = self._provisioned_pool()
|
||||||
|
pool.acquire("alice")
|
||||||
|
self.fake.pause_fail = True
|
||||||
|
self.assertFalse(pool.release("alice"))
|
||||||
|
# 不删条目、不销毁:下次 acquire 仍是 running→cold,
|
||||||
|
# VM 本身没被 shutdown,留待 worker 关闭时清理。
|
||||||
|
self.assertEqual(self.fake.shutdowns, [])
|
||||||
|
self.assertEqual(pool.acquire("alice")["mode"], "cold")
|
||||||
|
|
||||||
|
def test_provision_is_idempotent_and_per_user_isolated(self):
|
||||||
|
pool = ResidentPool(self.fake)
|
||||||
|
self.assertEqual(pool.provision("alice")["mode"], "resident")
|
||||||
|
self.assertEqual(pool.provision("bob")["mode"], "resident")
|
||||||
|
self.assertEqual(pool.provision("alice")["mode"], "resident")
|
||||||
|
self.assertEqual(self.fake.wakes.count("chv-resident-alice"), 1)
|
||||||
|
self.assertEqual(self.fake.wakes.count("chv-resident-bob"), 1)
|
||||||
|
self.assertEqual(pool.paused_count(), 2)
|
||||||
|
# release 只动本用户条目。
|
||||||
|
self.assertFalse(pool.release("bob")) # bob 未 running
|
||||||
|
pool.acquire("alice")
|
||||||
|
self.assertTrue(pool.release("alice"))
|
||||||
|
self.assertEqual(pool.paused_count(), 2)
|
||||||
|
got = pool.acquire("bob")
|
||||||
|
self.assertEqual(got["vm"], "chv-resident-bob")
|
||||||
|
self.assertEqual(pool.paused_count(), 1)
|
||||||
|
self.assertEqual(self.fake.resumes, ["chv-resident-alice",
|
||||||
|
"chv-resident-bob"])
|
||||||
|
|
||||||
|
def test_provision_wake_failure_falls_back_and_cleans_up(self):
|
||||||
|
pool = ResidentPool(self.fake)
|
||||||
|
self.fake.wake_fail = True
|
||||||
|
got = pool.provision("alice")
|
||||||
|
self.assertEqual(got, {"mode": "cold", "vm": None,
|
||||||
|
"task": None})
|
||||||
|
# 建池失败尽力 shutdown,不留半醒 VM 与条目。
|
||||||
|
self.assertEqual(self.fake.shutdowns,
|
||||||
|
["chv-resident-alice"])
|
||||||
|
self.assertEqual(pool.paused_count(), 0)
|
||||||
|
with open(_registry_path(self.tmp.name)) as fh:
|
||||||
|
self.assertEqual(json.load(fh), {})
|
||||||
|
|
||||||
|
|
||||||
|
class ResidentPoolRegistryTest(unittest.TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
self.tmp = tempfile.TemporaryDirectory()
|
||||||
|
self.fake = _FakeExecutor(self.tmp.name)
|
||||||
|
|
||||||
|
def tearDown(self):
|
||||||
|
self.tmp.cleanup()
|
||||||
|
|
||||||
|
def test_corrupt_registry_is_quarantined(self):
|
||||||
|
path = _write(_registry_path(self.tmp.name), "{not json")
|
||||||
|
pool = ResidentPool(self.fake)
|
||||||
|
self.assertTrue(pool.registry_corrupt)
|
||||||
|
self.assertEqual(pool.paused_count(), 0)
|
||||||
|
self.assertEqual(pool.orphan_count(), 0)
|
||||||
|
self.assertTrue(os.path.exists(path + ".corrupt"))
|
||||||
|
self.assertFalse(os.path.exists(path))
|
||||||
|
|
||||||
|
def test_top_level_non_dict_is_corrupt(self):
|
||||||
|
path = _write(_registry_path(self.tmp.name), "[1, 2]")
|
||||||
|
pool = ResidentPool(self.fake)
|
||||||
|
self.assertTrue(pool.registry_corrupt)
|
||||||
|
self.assertTrue(os.path.exists(path + ".corrupt"))
|
||||||
|
|
||||||
|
def test_bad_entries_dropped_without_tainting_good_ones(self):
|
||||||
|
_write(_registry_path(self.tmp.name), json.dumps({
|
||||||
|
"good": {"state": "paused", "vm": "chv-resident-good",
|
||||||
|
"user_id": "good"},
|
||||||
|
"no-vm": {"state": "paused"},
|
||||||
|
"not-a-dict": "x",
|
||||||
|
"wrong-state": {"state": "flying",
|
||||||
|
"vm": "chv-resident-x"},
|
||||||
|
}))
|
||||||
|
pool = ResidentPool(self.fake)
|
||||||
|
self.assertEqual(pool.registry_corrupt_entries, 3)
|
||||||
|
self.assertEqual(pool.orphan_count(), 1)
|
||||||
|
self.assertEqual(pool.paused_count(), 0)
|
||||||
|
|
||||||
|
def test_restart_never_reuses_entries(self):
|
||||||
|
pool1 = ResidentPool(self.fake)
|
||||||
|
pool1.provision("alice")
|
||||||
|
pool2 = ResidentPool(self.fake)
|
||||||
|
self.assertEqual(pool2.orphan_count(), 1)
|
||||||
|
self.assertEqual(pool2.paused_count(), 0)
|
||||||
|
# 新池绝不把上一世条目发给任务。
|
||||||
|
self.assertEqual(pool2.acquire("alice")["mode"], "cold")
|
||||||
|
self.assertEqual(self.fake.resumes, [])
|
||||||
|
cleaned = pool2.reconcile_orphans()
|
||||||
|
self.assertEqual(cleaned, ["chv-resident-alice"])
|
||||||
|
self.assertEqual(self.fake.shutdowns,
|
||||||
|
["chv-resident-alice"])
|
||||||
|
with open(_registry_path(self.tmp.name)) as fh:
|
||||||
|
self.assertEqual(json.load(fh), {})
|
||||||
|
|
||||||
|
def test_idle_minutes_env_parsing(self):
|
||||||
|
self.assertEqual(
|
||||||
|
ResidentPool(self.fake).idle_minutes, 0)
|
||||||
|
fake = _FakeExecutor(
|
||||||
|
self.tmp.name, env={"CHV_RESIDENT_IDLE_MINUTES": "45"})
|
||||||
|
self.assertEqual(ResidentPool(fake).idle_minutes, 45)
|
||||||
|
fake = _FakeExecutor(
|
||||||
|
self.tmp.name, env={"CHV_RESIDENT_IDLE_MINUTES": "junk"})
|
||||||
|
self.assertEqual(ResidentPool(fake).idle_minutes, 0)
|
||||||
|
fake = _FakeExecutor(
|
||||||
|
self.tmp.name, env={"CHV_RESIDENT_IDLE_MINUTES": "-5"})
|
||||||
|
self.assertEqual(ResidentPool(fake).idle_minutes, 0)
|
||||||
|
pool = ResidentPool(self.fake, idle_minutes=10)
|
||||||
|
self.assertEqual(pool.idle_minutes, 10)
|
||||||
|
|
||||||
|
|
||||||
|
class ResidentPoolIdleTest(unittest.TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
self.tmp = tempfile.TemporaryDirectory()
|
||||||
|
self.fake = _FakeExecutor(self.tmp.name)
|
||||||
|
|
||||||
|
def tearDown(self):
|
||||||
|
self.tmp.cleanup()
|
||||||
|
|
||||||
|
def test_idle_disabled_by_default(self):
|
||||||
|
pool = ResidentPool(self.fake)
|
||||||
|
pool.provision("alice")
|
||||||
|
import time
|
||||||
|
self.assertEqual(
|
||||||
|
pool.reclaim_idle(now=time.time() + 10 ** 9), [])
|
||||||
|
self.assertEqual(self.fake.hibernates, [])
|
||||||
|
|
||||||
|
def test_overdue_paused_entry_is_hibernated(self):
|
||||||
|
import time
|
||||||
|
pool = ResidentPool(self.fake, idle_minutes=30)
|
||||||
|
pool.provision("alice")
|
||||||
|
pool.provision("bob")
|
||||||
|
now = time.time() + 31 * 60
|
||||||
|
reclaimed = pool.reclaim_idle(now=now)
|
||||||
|
self.assertEqual(reclaimed, ["chv-resident-alice",
|
||||||
|
"chv-resident-bob"])
|
||||||
|
self.assertEqual(len(self.fake.hibernates), 2)
|
||||||
|
for vm, url in self.fake.hibernates:
|
||||||
|
self.assertTrue(url.startswith("file://"))
|
||||||
|
self.assertIn(self.tmp.name, url)
|
||||||
|
self.assertTrue(url.endswith(
|
||||||
|
os.path.join(vm, "resident.snapshot")))
|
||||||
|
self.assertEqual(pool.paused_count(), 0)
|
||||||
|
with open(_registry_path(self.tmp.name)) as fh:
|
||||||
|
self.assertEqual(json.load(fh), {})
|
||||||
|
|
||||||
|
def test_not_yet_idle_entry_is_kept(self):
|
||||||
|
import time
|
||||||
|
pool = ResidentPool(self.fake, idle_minutes=30)
|
||||||
|
pool.provision("alice")
|
||||||
|
self.assertEqual(
|
||||||
|
pool.reclaim_idle(now=time.time() + 10 * 60), [])
|
||||||
|
self.assertEqual(pool.paused_count(), 1)
|
||||||
|
|
||||||
|
def test_hibernate_failure_keeps_entry_for_retry(self):
|
||||||
|
import time
|
||||||
|
pool = ResidentPool(self.fake, idle_minutes=30)
|
||||||
|
pool.provision("alice")
|
||||||
|
self.fake.hibernate_fail = True
|
||||||
|
self.assertEqual(
|
||||||
|
pool.reclaim_idle(now=time.time() + 31 * 60), [])
|
||||||
|
self.assertEqual(pool.paused_count(), 1)
|
||||||
|
|
||||||
|
|
||||||
|
class ResidentFacadeTest(unittest.TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
self.tmp = tempfile.TemporaryDirectory()
|
||||||
|
self.fake = _FakeExecutor(self.tmp.name)
|
||||||
|
|
||||||
|
def tearDown(self):
|
||||||
|
self.tmp.cleanup()
|
||||||
|
|
||||||
|
def test_module_facade_cleans_and_empties_registry(self):
|
||||||
|
_write(_registry_path(self.tmp.name), json.dumps({
|
||||||
|
"a": {"state": "paused", "vm": "chv-resident-a"},
|
||||||
|
"b": {"state": "running", "vm": "chv-resident-b"},
|
||||||
|
"c": {"state": "provisioning", "vm": "chv-resident-c"},
|
||||||
|
"d": {"state": "warm", "vm": "chv-resident-d"},
|
||||||
|
"e": {"state": "paused"},
|
||||||
|
"f": "not-a-dict",
|
||||||
|
}))
|
||||||
|
cleaned = reconcile_resident_orphans_for_executor(self.fake)
|
||||||
|
self.assertEqual(cleaned, ["chv-resident-a", "chv-resident-b",
|
||||||
|
"chv-resident-c"])
|
||||||
|
self.assertEqual(self.fake.shutdowns, cleaned)
|
||||||
|
with open(_registry_path(self.tmp.name)) as fh:
|
||||||
|
self.assertEqual(json.load(fh), {})
|
||||||
|
self.assertIn("resident_pool_reconcile", self.fake.emits)
|
||||||
|
|
||||||
|
def test_module_facade_without_registry_is_empty(self):
|
||||||
|
self.assertEqual(
|
||||||
|
reconcile_resident_orphans_for_executor(self.fake), [])
|
||||||
|
self.assertEqual(self.fake.shutdowns, [])
|
||||||
|
|
||||||
|
def test_module_facade_without_state_dir_is_empty(self):
|
||||||
|
fake = _FakeExecutor(None)
|
||||||
|
fake.state_dir = None
|
||||||
|
self.assertEqual(
|
||||||
|
reconcile_resident_orphans_for_executor(fake), [])
|
||||||
|
|
||||||
|
class WorkerAdapterFacadeTest(unittest.TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
self.tmp = tempfile.TemporaryDirectory()
|
||||||
|
self.fake = _FakeExecutor(self.tmp.name)
|
||||||
|
self.audits = []
|
||||||
|
|
||||||
|
def tearDown(self):
|
||||||
|
self.tmp.cleanup()
|
||||||
|
|
||||||
|
def _audit(self, project_id, event, detail):
|
||||||
|
self.audits.append((project_id, event, detail))
|
||||||
|
|
||||||
|
def test_facade_cleans_resident_registry(self):
|
||||||
|
# worker_adapter 的 isinstance 门要求真执行器:用假
|
||||||
|
# 的 shutdown 替身包一层真执行器实例(只借类型)。
|
||||||
|
ex = _RealExecCase.make_executor(self.tmp.name, self.tmp.name)
|
||||||
|
ex.shutdown_vm = self.fake.shutdown_vm
|
||||||
|
ex._emit = self.fake._emit
|
||||||
|
_write(_registry_path(self.tmp.name), json.dumps({
|
||||||
|
"a": {"state": "paused", "vm": "chv-resident-a"},
|
||||||
|
"b": {"state": "running", "vm": "chv-resident-b"},
|
||||||
|
}))
|
||||||
|
cleaned = worker_adapter.reconcile_resident_pool(
|
||||||
|
ex, self._audit, "startup")
|
||||||
|
self.assertEqual(cleaned, ["chv-resident-a", "chv-resident-b"])
|
||||||
|
self.assertEqual(self.fake.shutdowns, cleaned)
|
||||||
|
self.assertTrue(any(
|
||||||
|
event == "chv_resident_pool_reconcile"
|
||||||
|
for _p, event, _d in self.audits))
|
||||||
|
with open(_registry_path(self.tmp.name)) as fh:
|
||||||
|
self.assertEqual(json.load(fh), {})
|
||||||
|
|
||||||
|
def test_facade_never_constructs_resident_pool(self):
|
||||||
|
ex = _RealExecCase.make_executor(self.tmp.name, self.tmp.name)
|
||||||
|
ex.shutdown_vm = self.fake.shutdown_vm
|
||||||
|
ex._emit = self.fake._emit
|
||||||
|
_write(_registry_path(self.tmp.name), json.dumps({
|
||||||
|
"a": {"state": "paused", "vm": "chv-resident-a"},
|
||||||
|
}))
|
||||||
|
with mock.patch(
|
||||||
|
"cloud.chv.resident_pool.ResidentPool",
|
||||||
|
side_effect=AssertionError("must not construct")):
|
||||||
|
cleaned = worker_adapter.reconcile_resident_pool(
|
||||||
|
ex, self._audit, "shutdown")
|
||||||
|
self.assertEqual(cleaned, ["chv-resident-a"])
|
||||||
|
|
||||||
|
def test_facade_ignores_non_chv_executor(self):
|
||||||
|
cleaned = worker_adapter.reconcile_resident_pool(
|
||||||
|
object(), self._audit, "startup")
|
||||||
|
self.assertEqual(cleaned, [])
|
||||||
|
self.assertEqual(self.audits, [])
|
||||||
|
|
||||||
|
def test_facade_does_not_touch_warm_pool_reconcile(self):
|
||||||
|
# reconcile_warm_pool 仍走 warm-pool.json,与常驻无关。
|
||||||
|
ex = _RealExecCase.make_executor(self.tmp.name, self.tmp.name)
|
||||||
|
ex.shutdown_vm = self.fake.shutdown_vm
|
||||||
|
ex._emit = self.fake._emit
|
||||||
|
warm_path = os.path.join(self.tmp.name, "warm-pool.json")
|
||||||
|
_write(warm_path, json.dumps({
|
||||||
|
"w": {"state": "warm", "vm": "chv-warm-1",
|
||||||
|
"task_id": "w"},
|
||||||
|
}))
|
||||||
|
cleaned = worker_adapter.reconcile_warm_pool(
|
||||||
|
ex, self._audit, "startup")
|
||||||
|
self.assertEqual(cleaned, ["chv-warm-1"])
|
||||||
|
self.assertFalse(os.path.exists(_registry_path(self.tmp.name)))
|
||||||
|
|
||||||
|
|
||||||
|
def _write_file(path, content=b"x"):
|
||||||
|
with open(path, "wb") as fh:
|
||||||
|
fh.write(content)
|
||||||
|
os.chmod(path, 0o755)
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
class _RealExecCase(unittest.TestCase):
|
||||||
|
"""真执行器 + 全方法替身:测 _run_with_vm 的接线本身,
|
||||||
|
不测 VMM。"""
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def make_executor(state_dir, base_dir, env_extra=None):
|
||||||
|
env = {"CHV_EGRESS_PROXY": "http://10.0.0.1:8888",
|
||||||
|
"CHV_HOST_KEY_MODE": "keyscan",
|
||||||
|
"CHV_SSH_FORWARD": "1"}
|
||||||
|
if env_extra:
|
||||||
|
env.update(env_extra)
|
||||||
|
cpuinfo = os.path.join(base_dir, "cpuinfo")
|
||||||
|
with open(cpuinfo, "w") as fh:
|
||||||
|
fh.write("flags : fpu vmx sse\n")
|
||||||
|
ex = CloudHypervisorExecutor(
|
||||||
|
binary=_write_file(os.path.join(base_dir,
|
||||||
|
"cloud-hypervisor")),
|
||||||
|
ch_remote=_write_file(os.path.join(base_dir, "ch-remote")),
|
||||||
|
kernel=_write_file(os.path.join(base_dir, "vmlinux")),
|
||||||
|
rootfs=_write_file(os.path.join(base_dir, "disk.raw"),
|
||||||
|
b"R" * 100),
|
||||||
|
ssh_key=_write_file(os.path.join(base_dir, "id_chv")),
|
||||||
|
kvm_path=_write_file(os.path.join(base_dir, "kvm")),
|
||||||
|
cpuinfo_path=cpuinfo,
|
||||||
|
state_dir=state_dir,
|
||||||
|
sleeper=lambda s: None, env=env)
|
||||||
|
ex.audit_callback = lambda e, t, d, k=None: None
|
||||||
|
return ex
|
||||||
|
|
||||||
|
def setUp(self):
|
||||||
|
self.tmp = tempfile.TemporaryDirectory()
|
||||||
|
self.state_dir = os.path.join(self.tmp.name, "state")
|
||||||
|
self.calls = []
|
||||||
|
|
||||||
|
def tearDown(self):
|
||||||
|
self.tmp.cleanup()
|
||||||
|
|
||||||
|
def _stubbed_exec(self, env_extra=None, wake_fail_resident=False):
|
||||||
|
ex = self.make_executor(self.state_dir, self.tmp.name, env_extra)
|
||||||
|
calls = self.calls
|
||||||
|
|
||||||
|
def wake(task, extra_env=None):
|
||||||
|
calls.append(("wake", ex.vm_name(task)))
|
||||||
|
if wake_fail_resident and ex.vm_name(task).startswith(
|
||||||
|
"chv-resident-"):
|
||||||
|
raise RuntimeError("provision boom")
|
||||||
|
return {"vm": ex.vm_name(task), "mode": "cold"}
|
||||||
|
|
||||||
|
def pause(task_or_name):
|
||||||
|
calls.append(("pause", ex.vm_name(task_or_name)))
|
||||||
|
return True
|
||||||
|
|
||||||
|
def resume(task_or_name):
|
||||||
|
calls.append(("resume", ex.vm_name(task_or_name)))
|
||||||
|
return True
|
||||||
|
|
||||||
|
def wait(task):
|
||||||
|
calls.append(("wait", ex.vm_name(task)))
|
||||||
|
return True
|
||||||
|
|
||||||
|
def shutdown(task_or_name):
|
||||||
|
calls.append(("shutdown", ex.vm_name(task_or_name)))
|
||||||
|
return True
|
||||||
|
|
||||||
|
def run_guest(task, inner_args, extra_env=None, **kwargs):
|
||||||
|
calls.append(("guest", task["id"], list(inner_args)))
|
||||||
|
return {"task_id": task["id"], "status": "completed",
|
||||||
|
"stdout": "ok", "vm": ex.vm_name(task)}
|
||||||
|
|
||||||
|
def run_guest_goal(task, inner_args, extra_env=None, **kwargs):
|
||||||
|
calls.append(("guest-goal", task["id"], list(inner_args)))
|
||||||
|
return {"task_id": task["id"], "status": "completed",
|
||||||
|
"goal_met": True, "vm": ex.vm_name(task)}
|
||||||
|
|
||||||
|
ex.wake_vm = wake
|
||||||
|
ex.pause_vm = pause
|
||||||
|
ex.resume_vm = resume
|
||||||
|
ex.wait_guest_ready = wait
|
||||||
|
ex.shutdown_vm = shutdown
|
||||||
|
ex._run_guest = run_guest
|
||||||
|
ex._run_guest_goal = run_guest_goal
|
||||||
|
return ex
|
||||||
|
|
||||||
|
|
||||||
|
class RunWithVmResidentTest(_RealExecCase):
|
||||||
|
def test_first_task_provisions_and_second_reuses(self):
|
||||||
|
ex = self._stubbed_exec()
|
||||||
|
task = {"id": "task-1", "prompt": "hi"}
|
||||||
|
result = ex._run_with_vm(task, ["dispatch", "--task", "hi"])
|
||||||
|
self.assertEqual(result["status"], "completed")
|
||||||
|
# 常驻伪任务跑命令,但 task_id 改回真实任务。
|
||||||
|
self.assertEqual(result["task_id"], "task-1")
|
||||||
|
self.assertEqual(result["vm"], "chv-resident-default")
|
||||||
|
kinds = [c[0] for c in self.calls]
|
||||||
|
self.assertEqual(kinds, ["wake", "pause", "resume", "wait",
|
||||||
|
"guest", "pause"])
|
||||||
|
self.assertEqual(self.calls[0][1], "chv-resident-default")
|
||||||
|
self.assertEqual(self.calls[4][1], "resident-default")
|
||||||
|
# 第二个任务复用同一台常驻 VM,不再 wake。
|
||||||
|
self.calls.clear()
|
||||||
|
task2 = {"id": "task-2", "prompt": "hi2"}
|
||||||
|
result2 = ex._run_with_vm(task2, ["dispatch", "--task", "hi2"])
|
||||||
|
self.assertEqual(result2["task_id"], "task-2")
|
||||||
|
self.assertEqual(result2["vm"], "chv-resident-default")
|
||||||
|
self.assertEqual([c[0] for c in self.calls],
|
||||||
|
["resume", "wait", "guest", "pause"])
|
||||||
|
# 落盘条目是 paused,可供下次 acquire。
|
||||||
|
with open(_registry_path(self.state_dir)) as fh:
|
||||||
|
data = json.load(fh)
|
||||||
|
self.assertEqual(data["default"]["state"], "paused")
|
||||||
|
self.assertEqual(data["default"]["vm"], "chv-resident-default")
|
||||||
|
|
||||||
|
def test_goal_path_runs_on_resident_and_rewrites_task_id(self):
|
||||||
|
ex = self._stubbed_exec()
|
||||||
|
task = {"id": "goal-1", "prompt": "g"}
|
||||||
|
result = ex._run_with_vm(task, ["goal", "--task", "g"],
|
||||||
|
goal=True)
|
||||||
|
self.assertEqual(result["status"], "completed")
|
||||||
|
self.assertEqual(result["task_id"], "goal-1")
|
||||||
|
self.assertTrue(result["goal_met"])
|
||||||
|
self.assertIn(("guest-goal", "resident-default",
|
||||||
|
["goal", "--task", "g"]), self.calls)
|
||||||
|
|
||||||
|
def test_dispatch_task_flows_through_resident(self):
|
||||||
|
ex = self._stubbed_exec()
|
||||||
|
result = ex.dispatch_task({"id": "d-1", "prompt": "hello"})
|
||||||
|
self.assertEqual(result["status"], "completed")
|
||||||
|
self.assertEqual(result["task_id"], "d-1")
|
||||||
|
self.assertEqual(result["vm"], "chv-resident-default")
|
||||||
|
|
||||||
|
def test_per_user_vms_are_isolated(self):
|
||||||
|
ex = self._stubbed_exec()
|
||||||
|
ex._run_with_vm({"id": "t-a", "user_id": "alice",
|
||||||
|
"prompt": "x"}, ["dispatch"])
|
||||||
|
ex._run_with_vm({"id": "t-b", "user_id": "bob",
|
||||||
|
"prompt": "x"}, ["dispatch"])
|
||||||
|
wakes = [c[1] for c in self.calls if c[0] == "wake"]
|
||||||
|
self.assertEqual(wakes, ["chv-resident-alice",
|
||||||
|
"chv-resident-bob"])
|
||||||
|
with open(_registry_path(self.state_dir)) as fh:
|
||||||
|
data = json.load(fh)
|
||||||
|
self.assertEqual(set(data), {"alice", "bob"})
|
||||||
|
|
||||||
|
def test_disabled_flag_goes_cold(self):
|
||||||
|
ex = self._stubbed_exec(
|
||||||
|
env_extra={"CHV_RESIDENT_ENABLED": "0"})
|
||||||
|
result = ex._run_with_vm({"id": "task-1", "prompt": "hi"},
|
||||||
|
["dispatch"])
|
||||||
|
self.assertEqual(result["status"], "completed")
|
||||||
|
self.assertEqual(result["task_id"], "task-1")
|
||||||
|
kinds = [c[0] for c in self.calls]
|
||||||
|
self.assertEqual(kinds, ["wake", "guest", "shutdown"])
|
||||||
|
self.assertEqual(self.calls[0][1], "chv-task-1")
|
||||||
|
self.assertIsNone(getattr(ex, "_resident_pool", None))
|
||||||
|
self.assertFalse(os.path.exists(
|
||||||
|
_registry_path(self.state_dir)))
|
||||||
|
|
||||||
|
def test_secret_env_goes_cold(self):
|
||||||
|
ex = self._stubbed_exec()
|
||||||
|
result = ex._run_with_vm(
|
||||||
|
{"id": "task-1", "prompt": "hi"}, ["dispatch"],
|
||||||
|
extra_env={"SUB2API_API_KEY": "sekret"})
|
||||||
|
self.assertEqual(result["status"], "completed")
|
||||||
|
kinds = [c[0] for c in self.calls]
|
||||||
|
self.assertEqual(kinds, ["wake", "guest", "shutdown"])
|
||||||
|
self.assertFalse(os.path.exists(
|
||||||
|
_registry_path(self.state_dir)))
|
||||||
|
|
||||||
|
def test_provision_failure_falls_back_to_cold(self):
|
||||||
|
ex = self._stubbed_exec(wake_fail_resident=True)
|
||||||
|
result = ex._run_with_vm({"id": "task-1", "prompt": "hi"},
|
||||||
|
["dispatch"])
|
||||||
|
self.assertEqual(result["status"], "completed")
|
||||||
|
self.assertEqual(result["task_id"], "task-1")
|
||||||
|
kinds = [c[0] for c in self.calls]
|
||||||
|
# 建池 wake 失败(尽力 shutdown)→ 冷启动照常跑任务。
|
||||||
|
self.assertEqual(kinds, ["wake", "shutdown", "wake",
|
||||||
|
"guest", "shutdown"])
|
||||||
|
self.assertEqual(self.calls[1][1], "chv-resident-default")
|
||||||
|
self.assertEqual(self.calls[2][1], "chv-task-1")
|
||||||
|
|
||||||
|
def test_guest_error_still_pauses_resident_in_finally(self):
|
||||||
|
ex = self._stubbed_exec()
|
||||||
|
|
||||||
|
def boom(task, inner_args, extra_env=None, **kwargs):
|
||||||
|
raise RuntimeError("guest boom")
|
||||||
|
|
||||||
|
ex._run_guest = boom
|
||||||
|
with self.assertRaises(RuntimeError):
|
||||||
|
ex._run_with_vm({"id": "task-1", "prompt": "hi"},
|
||||||
|
["dispatch"])
|
||||||
|
kinds = [c[0] for c in self.calls]
|
||||||
|
self.assertEqual(kinds, ["wake", "pause", "resume", "wait",
|
||||||
|
"pause"])
|
||||||
|
self.assertEqual(self.calls[-1][1], "chv-resident-default")
|
||||||
|
|
||||||
|
def test_guest_error_still_shuts_down_cold_in_finally(self):
|
||||||
|
ex = self._stubbed_exec(
|
||||||
|
env_extra={"CHV_RESIDENT_ENABLED": "0"})
|
||||||
|
|
||||||
|
def boom(task, inner_args, extra_env=None, **kwargs):
|
||||||
|
raise RuntimeError("guest boom")
|
||||||
|
|
||||||
|
ex._run_guest = boom
|
||||||
|
with self.assertRaises(RuntimeError):
|
||||||
|
ex._run_with_vm({"id": "task-1", "prompt": "hi"},
|
||||||
|
["dispatch"])
|
||||||
|
self.assertEqual([c[0] for c in self.calls],
|
||||||
|
["wake", "shutdown"])
|
||||||
|
|
||||||
|
def test_wake_failure_propagates_without_shutdown(self):
|
||||||
|
ex = self._stubbed_exec(
|
||||||
|
env_extra={"CHV_RESIDENT_ENABLED": "0"})
|
||||||
|
|
||||||
|
def wake_fail(task, extra_env=None):
|
||||||
|
raise ChvFailure("vm_create_failed", "no vm")
|
||||||
|
|
||||||
|
ex.wake_vm = wake_fail
|
||||||
|
with self.assertRaises(ChvFailure):
|
||||||
|
ex._run_with_vm({"id": "task-1", "prompt": "hi"},
|
||||||
|
["dispatch"])
|
||||||
|
self.assertEqual(self.calls, [])
|
||||||
|
|
||||||
|
|
||||||
|
class WarmPoolUntouchedTest(unittest.TestCase):
|
||||||
|
def test_wired_to_dispatch_still_false(self):
|
||||||
|
self.assertFalse(WarmPool.wired_to_dispatch)
|
||||||
|
|
||||||
|
def test_executor_source_never_imports_experimental(self):
|
||||||
|
import ast
|
||||||
|
from pathlib import Path
|
||||||
|
tree = ast.parse((Path(__file__).resolve().parents[2]
|
||||||
|
/ "cloud" / "chv" / "executor.py").read_text())
|
||||||
|
for node in ast.walk(tree):
|
||||||
|
if isinstance(node, ast.Import):
|
||||||
|
for alias in node.names:
|
||||||
|
self.assertNotIn(alias.name, (
|
||||||
|
"cloud.chv.pool",
|
||||||
|
"cloud.chv.experimental_pool",
|
||||||
|
"cloud.chv.orphans"))
|
||||||
|
if isinstance(node, ast.ImportFrom):
|
||||||
|
self.assertNotIn(node.module, (
|
||||||
|
"cloud.chv.pool",
|
||||||
|
"cloud.chv.experimental_pool",
|
||||||
|
"cloud.chv.orphans"))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -227,7 +227,15 @@ class PreflightTest(ChvCase):
|
|||||||
self.assertIn("内核", text)
|
self.assertIn("内核", text)
|
||||||
|
|
||||||
def test_dispatch_missing_dependency_does_not_call_runner(self):
|
def test_dispatch_missing_dependency_does_not_call_runner(self):
|
||||||
ex = self.make_exec(binary="/tmp/no-chv")
|
# P1 常驻默认开:这里显式关常驻,测的仍是冷启动
|
||||||
|
# preflight 路径(缺依赖 → 零 runner 调用)。不开
|
||||||
|
# 关的话,_resident_acquire 会先试建池,wake 失败
|
||||||
|
# 后的清理 shutdown 会多一次 ch-remote 调用——那是
|
||||||
|
# 建池失败清理的既定行为(见 P1 的 provision 失败
|
||||||
|
# 回落测试),不是本测试要守的“缺依赖零调用”。
|
||||||
|
ex = self.make_exec(
|
||||||
|
binary="/tmp/no-chv",
|
||||||
|
env_extra={"CHV_RESIDENT_ENABLED": "0"})
|
||||||
with self.assertRaises(ChvFailure):
|
with self.assertRaises(ChvFailure):
|
||||||
ex.dispatch_task(self._task())
|
ex.dispatch_task(self._task())
|
||||||
self.assertEqual(self.calls, [])
|
self.assertEqual(self.calls, [])
|
||||||
@@ -538,7 +546,12 @@ class WarmPoolTest(ChvCase):
|
|||||||
# -- dispatch / goal(wake → 跑 → finally 关) ---------------------------
|
# -- dispatch / goal(wake → 跑 → finally 关) ---------------------------
|
||||||
class DispatchTest(ChvCase):
|
class DispatchTest(ChvCase):
|
||||||
def test_dispatch_wakes_runs_shuts_down(self):
|
def test_dispatch_wakes_runs_shuts_down(self):
|
||||||
ex = self.make_exec()
|
# 本测试钉的是冷启动 dispatch 序列(wake chv-abc123
|
||||||
|
# → 跑 → finally shutdown);P1 起常驻默认开,须显
|
||||||
|
# 式关常驻才走这条路。常驻 dispatch 由
|
||||||
|
# test_chv_muse_p1_resident 覆盖。
|
||||||
|
ex = self.make_exec(
|
||||||
|
env_extra={"CHV_RESIDENT_ENABLED": "0"})
|
||||||
out = ex.dispatch_task(self._task())
|
out = ex.dispatch_task(self._task())
|
||||||
self.assertEqual(out["executor"], "cloud-hypervisor")
|
self.assertEqual(out["executor"], "cloud-hypervisor")
|
||||||
self.assertEqual(out["status"], "completed")
|
self.assertEqual(out["status"], "completed")
|
||||||
|
|||||||
@@ -57,6 +57,60 @@
|
|||||||
- 重启条件:将来要重新接线,必须先实现任务盘重供给并补
|
- 重启条件:将来要重新接线,必须先实现任务盘重供给并补
|
||||||
“任务盘来源”不变量测试,否则不得把 wired 改成 True。
|
“任务盘来源”不变量测试,否则不得把 wired 改成 True。
|
||||||
|
|
||||||
|
## ADR-17 常驻 VM 池取代 WarmPool 接线(feat-chv-muse-p1)
|
||||||
|
|
||||||
|
- 决策:不再接线 WarmPool,改走新建的生产模块
|
||||||
|
`cloud/chv/resident_pool.py`(ResidentPool):每用户一台
|
||||||
|
常驻 VM,任务到时 resume 已暂停的 VM,跑完 pause,不
|
||||||
|
销毁;池空或任何一步失败都回落冷启动。WarmPool 实验模块
|
||||||
|
一字不动,`wired_to_dispatch` 仍恒 False,ADR-5 的定案
|
||||||
|
(预热盘是干净快照盘、不是任务盘)仍然成立、只是不再
|
||||||
|
是接线障碍——常驻 VM 的盘本来就是每用户持久工作区盘,
|
||||||
|
不是每任务新盘。
|
||||||
|
- 隔离单元变了:常驻路径下隔离单元从“每任务”变成“每用户”
|
||||||
|
——同一用户的任务共享同一台常驻 VM 与同一块工作区盘,
|
||||||
|
每任务新盘的不变量在常驻路径上不再成立。冷启动路径
|
||||||
|
(每任务新盘)保留作回落与隔离严格场景。当前是单租户,
|
||||||
|
这个取舍可接受;多租户重审前不得把常驻路径当多租户隔离。
|
||||||
|
任务 dict 目前无 user_id,全落到 "default" 键;user_id
|
||||||
|
出现时按键隔离,release 绝不把用过的 VM 放回给另一用户。
|
||||||
|
- 接线:`_run_with_vm` 先常驻 acquire,命中走常驻路径
|
||||||
|
(finally 里 release=pause),失败/未启用走原冷启动路径
|
||||||
|
(finally shutdown 不变,冷启动本体逐字移到 `_run_cold`);
|
||||||
|
dispatch/run/goal/goal_resume/断点续跑都走 `_run_with_vm`,
|
||||||
|
签名不动。常驻默认开,`CHV_RESIDENT_ENABLED=0` 可关;
|
||||||
|
开关用 getattr/env 惰性读,`__init__` 不动。池空时惰性
|
||||||
|
provision 一次(wake + pause)再取,仍无则冷启动——首个
|
||||||
|
任务自举常驻 VM,不依赖 worker 启动时预热。
|
||||||
|
- 常驻执行细节:`_run_guest`/`_run_guest_goal` 一律走常驻伪
|
||||||
|
任务 `{"id": "resident-<用户>"}` 传参(SSH known_hosts、
|
||||||
|
api-socket、密钥投递记录、VMM 登记都按常驻 VM 名算),
|
||||||
|
返回后把 `task_id` 改回真实任务 id;中途死亡判据同样按
|
||||||
|
常驻伪任务查登记进程。带密钥 extra_env 的任务不走常驻
|
||||||
|
(seed 是建池时定的,密钥只能冷启动随 seed 投递),直接
|
||||||
|
冷启动。同一用户同一时间只跑一个常驻任务,并发第二个
|
||||||
|
任务回落冷启动。
|
||||||
|
- Registry:`state_dir/resident-pool.json`,文件 IO 与损坏
|
||||||
|
语义复用 orphans(原子写、损坏 quarantine、单条目非法
|
||||||
|
只丢该条);重启后读到的条目一律 orphan 不复用、只清理。
|
||||||
|
空闲回收:paused 超 `CHV_RESIDENT_IDLE_MINUTES`(默认 0=
|
||||||
|
不自动关机)时 hibernate(pause + snapshot + shutdown)
|
||||||
|
并删条目,快照文件留作恢复物,restore 是后续工作。
|
||||||
|
- 依赖方向(ADR-12 的禁令全部保留):executor 仍绝不 import
|
||||||
|
orphans/experimental_pool(常驻池用方法内 import 拿
|
||||||
|
ResidentPool);resident_pool → orphans(只借 registry
|
||||||
|
文件 IO,与 experimental_pool 同款方向);worker 启动/
|
||||||
|
关闭调 `worker_adapter.reconcile_resident_pool` facade,
|
||||||
|
它只读 resident-pool.json 做清理,不构造 ResidentPool。
|
||||||
|
`reconcile_warm_pool` 不动。
|
||||||
|
- 诚实边界(未验证,不宣称):resume 耗时、到 SSH 5 秒内、
|
||||||
|
气球压内存——无真机验证(第三棒占着 WSL 测试机),一律
|
||||||
|
记入真机待验证清单;resume 失败时 `wait_guest_ready` 的
|
||||||
|
失败路径会 stop_vm(含删工作区盘),该路径的工作区以
|
||||||
|
Gitea 同步为准;pause 失败时条目留在 running(不删不销
|
||||||
|
毁),worker 关闭时 reconcile 清理,在此之前该用户只走
|
||||||
|
冷启动。
|
||||||
|
|
||||||
## ADR-6 Docker goal 解析分叉:共享正典 + 同步守卫(本轨定案)
|
## ADR-6 Docker goal 解析分叉:共享正典 + 同步守卫(本轨定案)
|
||||||
|
|
||||||
- 决策:Local/CHV 共用 parse_goal_result 正典;Docker goal_task
|
- 决策:Local/CHV 共用 parse_goal_result 正典;Docker goal_task
|
||||||
|
|||||||
@@ -0,0 +1,90 @@
|
|||||||
|
# 对齐 Muse P1 差距表:常驻 VM + WarmPool + 气球 + vsock
|
||||||
|
|
||||||
|
依据:`~/workspace/research_notes/meta-muse-ai-architecture-20261101-0755/report.md`
|
||||||
|
(官方以 Meta security.muse.ai 2026-09-08 为准;第三方拆解只作参考,
|
||||||
|
逆向细节不得写成官方架构)。Muse 模型一句话:每用户一台常驻 Secure VM,
|
||||||
|
闲置靠气球压内存,宿主–客机走 vsock 通知通道。
|
||||||
|
|
||||||
|
本轨范围只限 P1:常驻 VM、WarmPool 扶正、内存气球、vsock 通道。
|
||||||
|
不碰 P2(客机 nspawn 单元、浏览器子代理独立)、不碰 P3(authd 替身凭证、
|
||||||
|
Sentinel 式出口审批)。分支 `feat-chv-muse-p1`(从 `feat-chv-cover100`
|
||||||
|
分出,基线 HEAD `06c3e17`)。
|
||||||
|
|
||||||
|
## 1. 现状 vs Muse
|
||||||
|
|
||||||
|
| 项 | 现状(实测/代码为准) | Muse 目标 | 本轨做法 |
|
||||||
|
|---|---|---|---|
|
||||||
|
| VM 生命周期 | 1 任务 = 1 VM,用完销毁;冷启动到 SSH 实测 11 秒 | 每用户一台常驻 Secure VM(hatchling),任务来了直接用 | 新增常驻服务:每用户一台常驻 VM(暂停态),任务到时 resume;池空回落冷启动(11 秒路径不动) |
|
||||||
|
| 派发查池 | `_run_with_vm` 绝不查池,`warm_pool_wired_to_dispatch` 恒 False;worker 派发前查池未接线 | 常驻 VM 就是派发目标,无冷启动 | `_run_with_vm` 接常驻路径:先 acquire 常驻 VM(resume + 等 SSH 就绪),无可用时回落 `wake_vm` |
|
||||||
|
| 预热池状态 | `experimental_pool.WarmPool` 纯实验,生产禁 import(ADR-12);ADR-5 定案:预热盘是干净快照盘、不是任务盘,接线须先实现任务盘重供给 | —(Muse 无此概念:常驻 VM 的盘就是用户工作区盘) | WarmPool 实验模块不动(仍禁生产 import);新建生产模块 `resident_pool.py`。常驻 VM 的盘是**每用户持久工作区盘**,不是每任务新盘 |
|
||||||
|
| 内存 | 固定 2G,无气球 | 气球 + 空闲页上报,闲置 8G 压到约 2G 工作集 | CHV 配置加 virtio-balloon + free-page-reporting(默认开,可关) |
|
||||||
|
| 宿主–客机通道 | 只 SSH;executor 文档明确“只选 SSH,不做 vsock 第二套”(旧决策) | vsock 通知通道(另有 Unix socket + SO_PEERCRED 的单元内 IPC,属 P2/P3 层) | vsock 只做最小 notify(就绪/心跳上报),与 SSH 并存;SSH 仍是命令通道与就绪判定,host key 钉死不动 |
|
||||||
|
| 池持久化/孤儿 | WarmPool registry `state_dir/warm-pool.json` 已有(原子写、损坏 quarantine、重启后条目一律 orphan 不复用) | — | 常驻池复用同款 registry 语义,另建 `resident-pool.json`;worker 启动/关闭走 `orphans` facade 清理,不自动复用上一世条目 |
|
||||||
|
|
||||||
|
## 2. 改了什么(本轨交付物)
|
||||||
|
|
||||||
|
- `cloud/chv/resident_pool.py`(新):常驻池。状态机 cold → provisioning →
|
||||||
|
paused → running → paused;`acquire(user)` resume + SSH 就绪,
|
||||||
|
`release(user)` pause 不销毁;空闲回收策略(pause 后常驻,超闲置阈值
|
||||||
|
关机+快照,阈值可配,默认不开自动关机);registry 原子落盘、损坏
|
||||||
|
quarantine、重启后不复用。
|
||||||
|
- `cloud/chv/executor.py`:`_run_with_vm` 接常驻路径(默认开,可配关;
|
||||||
|
常驻失败一律回落冷启动,绝不因池坏而丢任务);`_create` payload 加
|
||||||
|
balloon/vsock 设备。
|
||||||
|
- `cloud/chv/config.py`:`CHV_RESIDENT_IDLE_MINUTES`、`CHV_BALLOON`、
|
||||||
|
`CHV_VSOCK`(开关 + CID),非法值 ValueError,不静默 fallback。
|
||||||
|
- `cloud/chv/worker_adapter.py`:常驻池 reconcile facade(接 `orphans`)。
|
||||||
|
- `cloud/chv/vsock.py`(新,宿主侧):vsock notify 客户端/监听最小实现。
|
||||||
|
- `cloud/chv/guest/vsock_notify.py`(新,客内脚本):连宿主 CID 2 发
|
||||||
|
ready/心跳。**只写脚本文件,不改 `chv-guest/build.sh`**——镜像集成
|
||||||
|
是 P2 的范围,本轨只留接入说明。
|
||||||
|
- `cloud/tests/test_chv_muse_p1_*.py`(新):常驻池、气球/vsock 配置与
|
||||||
|
payload、回落语义的单测;全量 cloud 套件全过。
|
||||||
|
- `docs/chv-architecture.md`:ADR-5/ADR-12 补充新决策(常驻路径取代
|
||||||
|
WarmPool 接线问题),旧定案不删、只追加。
|
||||||
|
|
||||||
|
## 3. 诚实边界(本轮不宣称的东西)
|
||||||
|
|
||||||
|
1. **隔离单元变了**:常驻路径下隔离单元从“每任务”变成“每用户”——同一
|
||||||
|
用户的任务共享同一台常驻 VM 与同一块工作区盘,每任务新盘的不变量在
|
||||||
|
常驻路径上不再成立。冷启动路径(每任务新盘)保留作回落与隔离严格场景。
|
||||||
|
当前是单租户,这个取舍可接受;多租户重审前不得把常驻路径当多租户隔离。
|
||||||
|
2. **性能数字全是待验证目标,不是结果**:resume 1–2 秒、到 SSH 5 秒内、
|
||||||
|
气球压内存——本轮无真机验证(第三棒 V-1~V-9 占着 WSL 测试机),一律
|
||||||
|
记入“真机待验证清单”,不在代码/文档里写成已测得。
|
||||||
|
3. **气球 API 形态须核对**:CHV 54 的 balloon/vsock payload 字段以
|
||||||
|
Cloud Hypervisor 官方文档/二进制为准;本机无 /dev/kvm,无法本地起
|
||||||
|
VM 校验,只做配置层 + payload 单测。
|
||||||
|
4. **vsock 端到端未通**:本轮只做到宿主侧 + 客内脚本;客内脚本进镜像、
|
||||||
|
真机收发验证都待 P2 集成后、测试机空闲时再做。SSH 不动,vsock 坏了
|
||||||
|
不影响任务。
|
||||||
|
5. **WarmPool 实验模块不动**:不把它翻成生产模块,不改
|
||||||
|
`wired_to_dispatch`;常驻是新模块、新路径,旧实验代码与测试原样保留。
|
||||||
|
|
||||||
|
## 4. 恢复收尾(分离后在本分支修掉的回归)
|
||||||
|
|
||||||
|
分离时 chv 子集 3 失败 + 1 错误,全量另有 2 失败,均已修
|
||||||
|
掉、未删测试、未改断言:
|
||||||
|
|
||||||
|
- **VmDeathTest 两处**:根因不是生产判定错——探针实证,
|
||||||
|
常驻 VM 真死时 `_run_on_resident` 两态都正确判
|
||||||
|
interrupted;错的是旧测试的死亡模拟:它杀
|
||||||
|
`ex._vms["chv-abc123"]`,P1 常驻路由下该 VM 从未 wake、
|
||||||
|
登记里只有 `chv-resident-default`,模拟是空操作。修
|
||||||
|
法:模拟器改杀 dispatch 时已登记的 VM(即实际跑命令
|
||||||
|
的那台),断言不动。
|
||||||
|
- **test_cloud_hypervisor 两处冷路径测试**
|
||||||
|
(`test_dispatch_wakes_runs_shuts_down`、
|
||||||
|
`test_dispatch_missing_dependency_does_not_call_runner`):
|
||||||
|
P1 把常驻改成默认开后,它们测的冷启动序列不再是默认
|
||||||
|
路径;修法:两测试显式 `CHV_RESIDENT_ENABLED=0`,
|
||||||
|
继续钉冷启动行为,断言不动。常驻 dispatch 由 P1 新测
|
||||||
|
试覆盖。附带说明:建池 wake 失败后的清理 shutdown 是
|
||||||
|
既定行为(P1 回落测试钉住),不是回归。
|
||||||
|
- **超 50 行**:`_balloon_vsock_payload`(54)拆出
|
||||||
|
`_resolve_vsock_socket`;`vsock.listen_notify`(55)
|
||||||
|
拆出 `_bind_notify_listener`;行为不动。
|
||||||
|
- **mypy**:`resident_pool.py:208` 加 None 守卫后干净;
|
||||||
|
`executor.py` 的 `out` 未注解错与 `marker_tools.py:142`
|
||||||
|
是基线 06c3e17 已有(`git show` 核对,P1 diff 零行碰
|
||||||
|
`out`),如实记录、未动。
|
||||||
Reference in new issue
Block a user