feat(chv-muse-p1): 常驻 VM + WarmPool 扶正 + 气球 + vsock,对齐 Muse
每用户一台常驻 VM(暂停态,resume 到 SSH 5 秒内为待验证目标), 池空/失败回落冷启动;CHV 配置加 virtio-balloon + free-page-reporting 与 vsock notify 最小实现(SSH 仍是命令通道与就绪判定)。 新模块 resident_pool.py / vsock.py + 客内 vsock_notify.py; _config 增加 CHV_RESIDENT_IDLE_MINUTES / CHV_BALLOON / CHV_VSOCK; worker_adapter 接常驻孤儿清理 facade;ADR-17 差距表见 docs/chv-muse-p1-gap.md。 恢复收尾(分离后回归,见 gap.md §4):VmDeathTest 死亡模拟改杀 实际跑命令的 VM;两处冷路径旧测试显式关常驻;_balloon_vsock_payload 与 listen_notify 拆到 50 行内;resident_pool.py:208 mypy 修掉。 executor.py out 未注解与 marker_tools.py:142 是基线旧错,未动。 测试:全量 cloud 套件 1934 OK(skipped=8);mypy 在改动文件上 干净(仅基线旧错 4 处)。真机验证待第三棒让出 WSL 测试机。
This commit is contained in:
1 parent
06c3e172fe
commit
78098c5594
13 files changed
+2527
-36
No files matched your search
+227
-25
@@ -97,6 +97,10 @@ class CloudHypervisorExecutor(Executor):
|
||||
SSH 直接复用 OpenSSH 的密钥认证与端口转发,云镜像自带
|
||||
sshd,无需在客内再写一个 vsock agent、也无需在宿主再写
|
||||
一个 vsock 代理去翻译 host.py 协议。
|
||||
- P1(feat-chv-muse-p1)补充:vsock 只做最小 notify
|
||||
(客机发 ready/心跳,见 cloud/chv/vsock.py 与
|
||||
cloud/chv/guest/vsock_notify.py),与 SSH 并存;SSH
|
||||
仍是唯一命令通道,host key 钉死不动。
|
||||
- 放弃 vsock 的代价:需要运维提供从宿主到客机的 SSH 通路
|
||||
(tap/桥接 + 端口转发,见下文“VM 生命周期”)。
|
||||
- Host key 钉死(本轨 #216 后的加固,默认行为变化):
|
||||
@@ -131,8 +135,8 @@ class CloudHypervisorExecutor(Executor):
|
||||
“true”命令,直到 CHV_READY_TIMEOUT(默认 120 秒,
|
||||
轮询间隔 CHV_READY_INTERVAL,默认 5 秒)内成功才跑
|
||||
host.py/任务;超时则关 VM 并报失败,绝不在未就绪时
|
||||
就派任务。vsock 放弃理由同客内通道一节(需客内 agent
|
||||
+ 宿主代理两套新代码,v1 不拼半吊子)。
|
||||
就派任务。P1 的 vsock ready 只做就绪辅助信号
|
||||
(cloud/chv/vsock.py),绝不取代 SSH 就绪判定。
|
||||
- 暂停与快照唤醒是两条独立代码分支:
|
||||
pause_vm()/resume_vm() 走 ch-remote pause/resume;
|
||||
restore_vm() 先 ch-remote restore(source_url=...),
|
||||
@@ -230,6 +234,12 @@ class CloudHypervisorExecutor(Executor):
|
||||
- CHV_API_TIMEOUT:等 vmm.ping 通的上限秒数(默认 30)
|
||||
- CHV_READY_TIMEOUT:等 guest SSH 就绪的上限秒数(默认 120)
|
||||
- CHV_READY_INTERVAL:SSH 就绪轮询间隔秒数(默认 5)
|
||||
- CHV_BALLOON:virtio-balloon + free-page-reporting 开关
|
||||
(默认 1 开;只认 0/1,非法值 ValueError)
|
||||
- CHV_VSOCK:vsock notify 通道开关(默认 1 开;只认
|
||||
0/1,非法值 ValueError)
|
||||
- CHV_VSOCK_CID:客机 vsock CID(默认 3;须在
|
||||
3-4294967295 内,0/1/2 为保留 CID)
|
||||
|
||||
前置校验(preflight,执行入口必调,缺一即 fail-fast):
|
||||
收集全部缺失项一次报出具体缺哪一项,绝不笼统报错:
|
||||
@@ -282,6 +292,9 @@ class CloudHypervisorExecutor(Executor):
|
||||
task_timeout: float | None
|
||||
snapshot_url: str | None
|
||||
warm_pool_size: int
|
||||
balloon: bool
|
||||
vsock: bool
|
||||
vsock_cid: int
|
||||
api_timeout: float
|
||||
ready_timeout: float
|
||||
ready_interval: float
|
||||
@@ -431,7 +444,8 @@ class CloudHypervisorExecutor(Executor):
|
||||
"snapshot_max_total_gb", "snapshot_retention_days",
|
||||
"snapshot_dir", "allow_remote_snapshot",
|
||||
"gitea_sync_env", "seed_user_data_extra_file",
|
||||
"seed_network_config_file"):
|
||||
"seed_network_config_file", "balloon", "vsock",
|
||||
"vsock_cid"):
|
||||
setattr(self, _field, getattr(_cfg, _field))
|
||||
|
||||
# -- 名字 / 路径 ---------------------------------------------------
|
||||
@@ -1024,7 +1038,8 @@ class CloudHypervisorExecutor(Executor):
|
||||
"image_type": "Raw"})
|
||||
return disks
|
||||
|
||||
def build_vm_config(self, task_disk: str | None = None, seed_disk: str | None = None) -> dict:
|
||||
def build_vm_config(self, task_disk: str | None = None, seed_disk: str | None = None,
|
||||
vsock_socket: str | None = None) -> dict:
|
||||
"""API 模式的 VmConfig(vm.create 请求体,纯函数):
|
||||
cpus{boot_vcpus,max_vcpus}、memory{size: 字节}、
|
||||
payload{kernel, cmdline}、disks[{path: 每任务磁盘}]、
|
||||
@@ -1037,7 +1052,11 @@ class CloudHypervisorExecutor(Executor):
|
||||
旧版顶层 kernel{path}/cmdline{args} 会被服务端忽略,
|
||||
vm.create 照常 204、vm.boot 报 400 "No bootitem
|
||||
provided: neither firmware nor kernel"(2026-11-07
|
||||
WSL 真机 CHV 54.0.0 实测)。"""
|
||||
WSL 真机 CHV 54.0.0 实测)。
|
||||
P1(feat-chv-muse-p1):默认带 balloon/vsock 设备
|
||||
(见 _balloon_vsock_payload;CHV_BALLOON/CHV_VSOCK
|
||||
可关)。vsock_socket 显式给时用它,否则按任务盘
|
||||
位置派生每 VM 唯一路径(_boot_vm 未传时即走派生)。"""
|
||||
try:
|
||||
cpus = int(self.cpus)
|
||||
except ValueError:
|
||||
@@ -1054,6 +1073,8 @@ class CloudHypervisorExecutor(Executor):
|
||||
"disks": self._vm_config_disks(task_disk, seed_disk),
|
||||
"rng": {"src": "/dev/urandom"},
|
||||
}
|
||||
config.update(self._balloon_vsock_payload(
|
||||
vsock_socket=vsock_socket, task_disk=task_disk))
|
||||
if self.net_json:
|
||||
try:
|
||||
net = json.loads(self.net_json)
|
||||
@@ -1062,6 +1083,67 @@ class CloudHypervisorExecutor(Executor):
|
||||
config["net"] = net if isinstance(net, list) else [net]
|
||||
return config
|
||||
|
||||
def _balloon_vsock_payload(self, vsock_socket=None,
|
||||
task_disk=None) -> dict:
|
||||
"""P1(feat-chv-muse-p1):VmConfig 的 balloon/vsock
|
||||
设备(CHV 54 API,VmConfig 顶层键;字段名以 CHV
|
||||
OpenAPI 生成的客户端为准:BalloonConfig{size,
|
||||
deflate_on_oom, free_page_reporting}、VsockConfig{
|
||||
cid, socket};本机无 /dev/kvm,尚未真机验证,见下)。
|
||||
|
||||
balloon(默认开):{"size": 0, "deflate_on_oom": true,
|
||||
"free_page_reporting": true}。size 是目标气球尺寸
|
||||
(字节):0 = 启动时不预扣客机内存,只靠
|
||||
free_page_reporting 让客机持续上报空闲页、宿主
|
||||
MADV_DONTNEED 回收——被动回收,不主动 inflate。
|
||||
deflate_on_oom 是安全网:日后若经 vm.resize 充气
|
||||
后客机承压,气球自动放气而不 OOM。待真机验证:
|
||||
size=0 是否被 CHV 54 接受(若拒绝,改 size 为 VM
|
||||
内存字节数并重新验证,见汇报);气球实际省内存量。
|
||||
vsock(默认开):{"cid": <CHV_VSOCK_CID,默认 3>,
|
||||
"socket": <宿主侧 Unix socket 路径>}。cid 是客机
|
||||
CID(0/1/2 保留,宿主永远是 CID 2,客机连宿主用
|
||||
CID 2;CHV 不分配 CID,由配置指定)。socket 路径
|
||||
派生:显式给 vsock_socket 时用它;否则任务盘在
|
||||
state_dir 内时落任务盘同目录(state_dir/<vm>/
|
||||
vsock.sock,每 VM 唯一);reuse/lab(任务盘即基础
|
||||
镜像或未给)时落 state_dir/vsock.sock——单 VM 可
|
||||
用,多 VM 并发会在 vm.create 撞路径(大声失败,
|
||||
不静默),_boot_vm 后续接线应显式传每 VM 路径。
|
||||
宿主侧监听在 <socket>_<port> 上(port 见
|
||||
cloud/chv/vsock.py 的 VSOCK_NOTIFY_PORT),客机
|
||||
连 CID 2 发 ready/心跳(cloud/chv/guest/
|
||||
vsock_notify.py)。开关关时对应键整个不出现
|
||||
(不是发 false),旧行为(无设备)原样保留。"""
|
||||
extra: dict = {}
|
||||
if getattr(self, "balloon", True):
|
||||
extra["balloon"] = {
|
||||
"size": 0, "deflate_on_oom": True,
|
||||
"free_page_reporting": True,
|
||||
}
|
||||
if getattr(self, "vsock", True):
|
||||
extra["vsock"] = {
|
||||
"cid": getattr(self, "vsock_cid", 3),
|
||||
"socket": self._resolve_vsock_socket(
|
||||
vsock_socket, task_disk),
|
||||
}
|
||||
return extra
|
||||
|
||||
def _resolve_vsock_socket(self, vsock_socket, task_disk):
|
||||
"""vsock socket 路径派生(_balloon_vsock_payload 的
|
||||
子步,单拆只为 50 行规矩,行为不动):显式给定时
|
||||
用它;否则任务盘在 state_dir 内时落任务盘同目录
|
||||
(每 VM 唯一),reuse/lab 时落 state_dir/vsock.sock
|
||||
(多 VM 并发会撞路径,大声失败)。"""
|
||||
if vsock_socket is not None:
|
||||
return vsock_socket
|
||||
if task_disk and os.path.realpath(
|
||||
task_disk).startswith(
|
||||
os.path.realpath(self.state_dir) + os.sep):
|
||||
return os.path.join(
|
||||
os.path.dirname(task_disk), "vsock.sock")
|
||||
return os.path.join(self.state_dir, "vsock.sock")
|
||||
|
||||
def _api_call(self, socket_path, method, path, payload=None):
|
||||
"""API 调用归一化(实现在 cloud/chv/api_client.py)。"""
|
||||
return normalize_api_result(
|
||||
@@ -2245,27 +2327,118 @@ class CloudHypervisorExecutor(Executor):
|
||||
"interrupted_reason": reason,
|
||||
}
|
||||
|
||||
def _run_with_vm(self, task, inner_args, goal=False, extra_env=None,
|
||||
**kwargs):
|
||||
"""每任务 VM 生命周期(冷启动正式路径,feat-chv-arch95
|
||||
定案):wake_vm(API 模式 + SSH 就绪)→ 跑客内命令 →
|
||||
finally shutdown_vm。本方法绝不查 WarmPool:预热池是
|
||||
实验性的(见 experimental_pool.py 头定案与 ADR-5),dispatch 永远
|
||||
冷启动,warm_pool_wired_to_dispatch 恒 False。
|
||||
wake 失败时不跑
|
||||
任务、直接抛分类失败。本方法不做内部重试:重试是
|
||||
worker 的 max_retries 语义,执行器内部重试会把一次
|
||||
失败放大成多次计费与多次 VM 启动。
|
||||
def _resident_enabled(self):
|
||||
"""常驻默认开;CHV_RESIDENT_ENABLED=0/false/no/off 关。
|
||||
优先认执行器上已有的 resident_enabled 属性(P1-B 若
|
||||
经 config 落属性,直接复用),否则惰性读 env;__init__
|
||||
不动,避免与 P1-B 的 __init__ 改动撞车。"""
|
||||
flag = getattr(self, "resident_enabled", None)
|
||||
if flag is None:
|
||||
env = getattr(self, "_env", None)
|
||||
if env is None:
|
||||
env = os.environ
|
||||
flag = env.get("CHV_RESIDENT_ENABLED", "1")
|
||||
if isinstance(flag, str):
|
||||
return flag.strip().lower() not in (
|
||||
"0", "false", "no", "off")
|
||||
return bool(flag)
|
||||
|
||||
中途死亡分类(本轨新增):客内命令抛传输异常、或
|
||||
非 goal 结果已 failed 时,若登记的 VMM 进程已死,
|
||||
判 interrupted(断点可续跑)而不是笼统 failed 或
|
||||
原样抛异常;VM 还活着时的异常/失败原样处理,不
|
||||
误判。goal 结果本身已有 interrupted 分类(见
|
||||
parse_goal_result),这里不再二次改写 goal 结论。
|
||||
feat-chv-secfix(H2):extra_env 透传给 wake_vm(密钥
|
||||
子集在 provisioned 模式下随 seed 投递),再透传给
|
||||
_run_guest(远程命令 source 客内密钥文件)。"""
|
||||
def _resident_pool_instance(self):
|
||||
"""惰性建唯一常驻池并缓存(方法内 import:执行器顶层
|
||||
绝不 import 池/orphans,见 ADR-12 的 AST 钉子)。
|
||||
同一 state_dir 只活一个池实例:registry 只在构造时
|
||||
读一次,重启后条目一律 orphan(见 resident_pool)。"""
|
||||
pool = getattr(self, "_resident_pool", None)
|
||||
if pool is not None:
|
||||
return pool
|
||||
with self._vms_lock:
|
||||
pool = getattr(self, "_resident_pool", None)
|
||||
if pool is None:
|
||||
from cloud.chv.resident_pool import ResidentPool
|
||||
pool = ResidentPool(self)
|
||||
self._resident_pool = pool
|
||||
return pool
|
||||
|
||||
def _resident_acquire(self, task, extra_env=None):
|
||||
"""常驻获取(best-effort,一律不抛,失败回 None):
|
||||
未启用/带密钥 env → None(走冷启动;密钥须随 seed
|
||||
投递,常驻 VM 的 seed 是建池时定的,带密钥任务只能
|
||||
冷启动)。有 paused 条目 → resume + SSH 就绪;池空/
|
||||
失败 → 惰性 provision 一次再取,仍无 → None。返回
|
||||
{"task": 常驻伪任务, "pool": 池, "user_id": 用户}。"""
|
||||
try:
|
||||
if not self._resident_enabled():
|
||||
return None
|
||||
secrets, _plain = split_secret_env(extra_env)
|
||||
if secrets:
|
||||
return None
|
||||
pool = self._resident_pool_instance()
|
||||
user_id = task.get("user_id") \
|
||||
if isinstance(task, dict) else None
|
||||
acquired = pool.acquire(user_id)
|
||||
if acquired.get("mode") == "resident":
|
||||
return {"task": acquired["task"], "pool": pool,
|
||||
"user_id": user_id}
|
||||
provisioned = pool.provision(user_id)
|
||||
if provisioned.get("mode") != "resident":
|
||||
return None
|
||||
acquired = pool.acquire(user_id)
|
||||
if acquired.get("mode") != "resident":
|
||||
return None
|
||||
return {"task": acquired["task"], "pool": pool,
|
||||
"user_id": user_id}
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
def _run_on_resident(self, task, resident_task, inner_args,
|
||||
goal=False, extra_env=None, **kwargs):
|
||||
"""常驻路径执行(_run_with_vm 的常驻分支本体):
|
||||
_run_guest/_run_guest_goal 一律走常驻伪任务传参(SSH
|
||||
known_hosts、api-socket、密钥投递记录、VMM 登记都按
|
||||
常驻 VM 名算),返回后把 task_id 改回真实任务 id;
|
||||
VM 中途死亡判据同样按常驻伪任务查登记进程,死亡
|
||||
结论里的任务 id 用真实任务。finally 的 pause 在调
|
||||
用方(_run_with_vm)做,这里只跑命令。"""
|
||||
try:
|
||||
if goal:
|
||||
result = self._run_guest_goal(
|
||||
resident_task, inner_args, extra_env=extra_env,
|
||||
**kwargs)
|
||||
else:
|
||||
result = self._run_guest(
|
||||
resident_task, inner_args, extra_env=extra_env,
|
||||
**kwargs)
|
||||
except Exception as exc:
|
||||
if self._vm_proc_dead(resident_task):
|
||||
return self._vm_death_result(
|
||||
task, goal, str(exc)[:500])
|
||||
raise
|
||||
if not goal and result.get("status") == "failed" \
|
||||
and self._vm_proc_dead(resident_task):
|
||||
result = dict(result)
|
||||
result["status"] = "interrupted"
|
||||
result["vm_dead"] = True
|
||||
result["interrupted_reason"] = (
|
||||
"VM 中途死亡,任务未完成、断点可续跑:"
|
||||
"客内命令已 failed 且登记的 VMM 进程已死")
|
||||
if isinstance(result, dict):
|
||||
result = dict(result)
|
||||
result["task_id"] = task["id"]
|
||||
return result
|
||||
|
||||
def _run_cold(self, task, inner_args, goal=False, extra_env=None,
|
||||
**kwargs):
|
||||
"""冷启动路径(原 _run_with_vm 本体,逐字搬运):
|
||||
wake_vm(API 模式 + SSH 就绪)→ 跑客内命令 →
|
||||
finally shutdown_vm。wake 失败时不跑任务、直接抛
|
||||
分类失败;本方法不做内部重试(重试是 worker 的
|
||||
max_retries 语义)。中途死亡分类:客内命令抛传输
|
||||
异常、或非 goal 结果已 failed 时,若登记的 VMM 进程
|
||||
已死,判 interrupted(断点可续跑);VM 还活着时的
|
||||
异常/失败原样处理。feat-chv-secfix(H2):extra_env
|
||||
透传给 wake_vm(密钥子集在 provisioned 模式下随 seed
|
||||
投递),再透传给 _run_guest(远程命令 source 客内
|
||||
密钥文件)。"""
|
||||
self.wake_vm(task, extra_env=extra_env)
|
||||
try:
|
||||
try:
|
||||
@@ -2294,6 +2467,35 @@ class CloudHypervisorExecutor(Executor):
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def _run_with_vm(self, task, inner_args, goal=False, extra_env=None,
|
||||
**kwargs):
|
||||
"""任务 VM 生命周期(feat-chv-muse-p1,见 ADR-17):
|
||||
常驻启用时先 _resident_acquire——paused 常驻 VM 经
|
||||
resume + SSH 就绪即跑,跑完 finally 里 release(pause,
|
||||
不 shutdown);池空/失败/未启用/带密钥 env 时走冷启动
|
||||
(_run_cold:wake_vm → 跑客内命令 → finally shutdown)。
|
||||
常驻失败一律回落冷启动,绝不因池坏而丢任务;WarmPool
|
||||
仍绝不查(实验模块,wired_to_dispatch 恒 False)。
|
||||
wake 失败时不跑任务、直接抛分类失败;本方法不做内部
|
||||
重试(重试是 worker 的 max_retries 语义)。中途死亡
|
||||
判 interrupted(断点可续跑),见 _run_cold 与
|
||||
_run_on_resident。dispatch/run/goal/goal_resume/断点
|
||||
续跑都走本方法,签名不动。"""
|
||||
resident = self._resident_acquire(task, extra_env=extra_env)
|
||||
if resident is not None:
|
||||
pool = resident["pool"]
|
||||
try:
|
||||
return self._run_on_resident(
|
||||
task, resident["task"], inner_args, goal=goal,
|
||||
extra_env=extra_env, **kwargs)
|
||||
finally:
|
||||
try:
|
||||
pool.release(resident["user_id"])
|
||||
except Exception:
|
||||
pass
|
||||
return self._run_cold(task, inner_args, goal=goal,
|
||||
extra_env=extra_env, **kwargs)
|
||||
|
||||
def execute(self, task: dict) -> dict:
|
||||
"""只读执行:客内 host.py status,不直接跑模型回合
|
||||
(与 Local/Docker 的 execute 同款语义)。
|
||||
|
||||
Reference in new issue
Block a user