feat: CHV 执行器优化(拆模块/去重复/SSH 钉死/池持久化/中途死亡分类)
- 拆模块:CHV 从 cloud/sandbox.py 巨类拆到 cloud/chv/(executor/ pool/config/api_client/ssh_transport/disk/errors),sandbox 只留 兼容 re-export,旧导入路径与新路径是同一对象 - 去重复:goal 解析、host 顶层参数、runner 超时抽成共享 helper, Local 与 CHV 在矩阵输入下结论全等(Docker goal-resume 变体 有意保留内联版,见 PR 描述) - 安全:SSH host key 按 VM 钉死(state_dir/<vm>/known_hosts, ssh-keyscan 首次捕获 TOFU),废除 no + /dev/null;CHV_SSH_USER 空串与 CHV_SSH_PORT 越界直接拒绝,不静默 fallback - execute 语义:只读已运行 VM,未运行报 vm_not_running,不为 status 起 VM - 预热池:registry 落盘、重启载入只当孤儿、reconcile 清理、 损坏 quarantine;条目仍一次性,绝不复用脏 VM - 长任务:VM 中途死亡判 interrupted(断点可续跑);中途心跳 经评估不做(无客内进度信号可采) - #215 未合并:只列复用清单,不复制模块 测试:原 42 项 + 新 24 项全过;相关旧套件与 cloud 全量 1496 项 全过(skipped=4)。验证仍是 mock + 干跑 + 单元测试,未做真机、 压测、安全审计;堆叠于 #216 之上,base 为 feat-cloud-hypervisor。
This commit is contained in:
1 parent
c371bbdf1e
commit
fde7bf4c3e
14 files changed
+2706
-1408
No files matched your search
+21
-3
@@ -1288,8 +1288,9 @@ interrupted 且终态任务 cancel 返回 400、心跳后 sweep 不扫刚心跳
|
||||
错误文本或返回结果。worker CLI 的 `--executor` 新增 cloud 选项,
|
||||
构造 CloudExecutor;worker 的 run_one 走 dispatch_task,
|
||||
CloudExecutor 的 dispatch_task/run_task 直接委托给 execute。
|
||||
- CloudHypervisorExecutor(生产级 v1,`cloud/sandbox.py`,
|
||||
执行器名 `cloud-hypervisor`,绝不用 `cloud` 缩写):
|
||||
- CloudHypervisorExecutor(生产级 v1,`cloud/chv/`,旧路径
|
||||
`cloud/sandbox.py` 只留兼容 re-export,执行器名
|
||||
`cloud-hypervisor`,绝不用 `cloud` 缩写):
|
||||
单任务单 VM:一个任务一个 VM、独立内核/文件系统/网络,
|
||||
任务结束销毁,租户之间无共享路径、无东西向互访;VM 内
|
||||
agent 可有 root,但只能影响本 VM。密钥按现有惯例走运行
|
||||
@@ -1307,7 +1308,24 @@ interrupted 且终态任务 cancel 返回 400、心跳后 sweep 不扫刚心跳
|
||||
用 pause + snapshot + shutdown 等价实现,恢复走 restore。
|
||||
预热池 `WarmPool`:预热条目是一次性干净快照 VM,acquire
|
||||
消耗、release 一律销毁,绝不跨任务复用脏 VM;池空回落
|
||||
冷启动,绝不降级 Docker/Local。关闭与清理:任务结束/
|
||||
冷启动,绝不降级 Docker/Local;池状态另落
|
||||
`state_dir/warm-pool.json` registry,worker 重启后载入的
|
||||
条目一律当孤儿、只能 `reconcile_orphans()` 清理,绝不
|
||||
重新发给任务,registry 损坏时 quarantine 成 `.corrupt` 并
|
||||
从空池开始。SSH host key 按 VM 钉死(#216 的
|
||||
`StrictHostKeyChecking=no` + `/dev/null` 已废除):每 VM
|
||||
在 `state_dir/<vm>/known_hosts` 有独立文件,首次启动经
|
||||
ssh-keyscan 在本机回环端口捕获 key 写入该文件(首次信任
|
||||
TOFU,只防首次之后的替换,不验证首次本身),此后 SSH
|
||||
一律 `StrictHostKeyChecking=yes` + 该文件校验;客机用户
|
||||
(`CHV_SSH_USER`,默认 root,显式空串直接拒绝)与端口
|
||||
(`CHV_SSH_PORT`,默认 2222,1-65535)为显式配置。
|
||||
`execute()` 只读已运行 VM:未运行报 `vm_not_running`
|
||||
分类失败,不为一次 status 起 VM。长任务中途死亡
|
||||
(传输异常、或结果已 failed 且登记 VMM 进程已死)判
|
||||
`interrupted` 而非笼统 failed;中途心跳经评估未做
|
||||
(host.py 单次 SSH 长回合无客内进度信号可采,另起轮询
|
||||
只能证明 VM 活着)。关闭与清理:任务结束/
|
||||
失败/取消时 `shutdown_vm`(ch-remote shutdown + 杀进程 +
|
||||
删 socket),`cleanup_stale_vms` 处理已死/已关/socket 残留
|
||||
三种情况。审计:VM 的 create/boot/ready/pause/resume/
|
||||
|
||||
@@ -0,0 +1,18 @@
|
||||
"""Cloud Hypervisor 执行器包(从 cloud/sandbox.py 巨类拆出)。
|
||||
|
||||
公开名与 #216 一致,cloud/sandbox.py 会 re-export 它们,
|
||||
保证旧导入路径可用:
|
||||
- CloudHypervisorExecutor:执行器本体(executor.py)
|
||||
- WarmPool:预热池 + registry 持久化与孤儿清理(pool.py)
|
||||
- ChvFailure:分类失败(errors.py)
|
||||
|
||||
子模块分工见 executor.py 头注释。本包顶层 import 不触
|
||||
cloud.sandbox(执行器方法内按需 import 共享 helper),
|
||||
所以任何导入顺序都不会循环。
|
||||
"""
|
||||
|
||||
from cloud.chv.errors import ChvFailure
|
||||
from cloud.chv.executor import CloudHypervisorExecutor
|
||||
from cloud.chv.pool import WarmPool
|
||||
|
||||
__all__ = ["ChvFailure", "CloudHypervisorExecutor", "WarmPool"]
|
||||
@@ -0,0 +1,74 @@
|
||||
"""Cloud Hypervisor REST API 传输(叶子模块,不依赖
|
||||
cloud.sandbox,避免循环导入)。
|
||||
|
||||
只用标准库 raw HTTP over Unix socket,无第三方依赖。
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import socket
|
||||
|
||||
|
||||
def _unix_socket_api(socket_path, method, path, payload=None):
|
||||
"""Cloud Hypervisor REST API 的默认传输:标准库 raw HTTP
|
||||
over Unix socket(无第三方依赖)。返回 (status, body),
|
||||
body 能解析 JSON 时为 dict,否则为原文本。socket 不存在
|
||||
或连接失败时 raise RuntimeError(调用方据此判 api 超时
|
||||
/失败,不编造 200)。"""
|
||||
if not os.path.exists(socket_path):
|
||||
raise RuntimeError("api socket 不存在:%s" % socket_path)
|
||||
body = b""
|
||||
if payload is not None:
|
||||
body = json.dumps(payload).encode("utf-8")
|
||||
req_lines = ["%s /api/v1%s HTTP/1.1" % (method, path),
|
||||
"Host: localhost",
|
||||
"Accept: application/json",
|
||||
"Content-Type: application/json",
|
||||
"Content-Length: %d" % len(body),
|
||||
"Connection: close", "", ""]
|
||||
raw = "\r\n".join(req_lines).encode("utf-8") + body
|
||||
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
|
||||
sock.settimeout(5)
|
||||
try:
|
||||
sock.connect(socket_path)
|
||||
sock.sendall(raw)
|
||||
chunks = []
|
||||
while True:
|
||||
try:
|
||||
data = sock.recv(65536)
|
||||
except socket.timeout:
|
||||
break
|
||||
if not data:
|
||||
break
|
||||
chunks.append(data)
|
||||
finally:
|
||||
sock.close()
|
||||
text = b"".join(chunks).decode("utf-8", errors="replace")
|
||||
status = 0
|
||||
if text.startswith("HTTP/"):
|
||||
try:
|
||||
status = int(text.split()[1])
|
||||
except (IndexError, ValueError):
|
||||
status = 0
|
||||
body_text = text.split("\r\n\r\n", 1)[1] if "\r\n\r\n" in text \
|
||||
else ""
|
||||
try:
|
||||
parsed = json.loads(body_text) if body_text.strip() else {}
|
||||
except ValueError:
|
||||
parsed = body_text
|
||||
return status, parsed
|
||||
|
||||
|
||||
def normalize_api_result(result):
|
||||
"""API 调用归一化:fake api_client 可返回 (status, body)
|
||||
或 {"status":..., "body":...} 或带 .status 的对象,本
|
||||
函数统一成 (status:int, body)。"""
|
||||
if isinstance(result, tuple) and len(result) == 2:
|
||||
return int(result[0]), result[1]
|
||||
if isinstance(result, dict):
|
||||
status = result.get("status", result.get("status_code"))
|
||||
body = result.get("body", result)
|
||||
return (int(status) if status is not None else 0), body
|
||||
status = getattr(result, "status", None) \
|
||||
or getattr(result, "status_code", None)
|
||||
return (int(status) if status is not None else 0), result
|
||||
@@ -0,0 +1,146 @@
|
||||
"""Cloud Hypervisor 执行器配置解析(叶子模块,不依赖
|
||||
cloud.sandbox,避免循环导入)。
|
||||
|
||||
职责只有两件:内存字符串解析,以及把构造参数 + 环境变量
|
||||
解析成一份显式配置。参数优先于环境变量、环境变量优先于
|
||||
默认值——与 #216 原 __init__ 的 `or` 链完全一致,只收紧了
|
||||
两处原来会静默 fallback 的 nonsense 输入(见下文),收紧点
|
||||
在 PR 描述里明写、单测钉住:
|
||||
|
||||
- CHV_SSH_USER 显式传空字符串:原来 `or` 链会静默 fallback
|
||||
成 root(调用方以为没给用户、实际跑了 root),现在直接
|
||||
ValueError。
|
||||
- CHV_SSH_PORT 显式传 0、或解析后超出 1-65535:原来 0 会
|
||||
静默 fallback 成 2222、其它越界值会原样进 ssh 命令,现在
|
||||
直接 ValueError。
|
||||
|
||||
与 #215 的关系(只列清单,不复制模块):#215 未合并,
|
||||
本轨不假装复用它。#215 合并后应再复用的点:超时类
|
||||
(api_timeout/ready_timeout/ready_interval/task_timeout →
|
||||
timeouts 模块)、门禁类(preflight fail-fast、容量门、磁盘
|
||||
配额门、egress fail-closed → gates 模块)、配额类
|
||||
(max_vms、disk_gb、任务成本走 worker quota → quota 模块)、
|
||||
隔离类(单任务单 VM、磁盘 copy 模式、egress 代理、SSH 钉死
|
||||
→ sandbox_policy 模块)。见 PR 描述的复用清单。
|
||||
"""
|
||||
|
||||
import re
|
||||
|
||||
|
||||
def _parse_memory_bytes(value):
|
||||
"""内存字符串转字节:2G/512M/1024K/2048(无单位=字节)。
|
||||
非法值 ValueError,绝不静默按默认算。"""
|
||||
m = re.match(r"^\s*(\d+)\s*([KMGkmg]?)\s*[Bb]?\s*$",
|
||||
str(value))
|
||||
if not m:
|
||||
raise ValueError("CHV_MEM 格式非法:%r" % (value,))
|
||||
mult = {"": 1, "K": 1024, "M": 1024 ** 2, "G": 1024 ** 3}
|
||||
return int(m.group(1)) * mult[m.group(2).upper()]
|
||||
|
||||
|
||||
class ChvConfig:
|
||||
"""解析后的 CHV 配置:纯数据对象,字段名与
|
||||
CloudHypervisorExecutor 原 __init__ 的属性名一一对应,
|
||||
执行器 __init__ 直接把字段抄到 self 上,旧测试读属性
|
||||
的方式全部不变。"""
|
||||
|
||||
def __init__(self, **fields):
|
||||
for key, value in fields.items():
|
||||
setattr(self, key, value)
|
||||
|
||||
|
||||
def build_chv_config(params, env):
|
||||
"""把显式构造参数(params,None=未给)与环境变量(env,
|
||||
默认为 os.environ 的调用方传入 dict)解析成 ChvConfig。
|
||||
|
||||
显式参数只有 __init__ 本来就有的那些;max_vms、磁盘模式、
|
||||
磁盘配额、egress、快照、预热池目标数本来就没有显式参数,
|
||||
仍只走环境变量(与 #216 一致,不悄悄加新参数面)。"""
|
||||
env = env if env is not None else {}
|
||||
binary = params.get("binary") or env.get("CHV_BINARY") \
|
||||
or "cloud-hypervisor"
|
||||
kernel = params.get("kernel") or env.get("CHV_KERNEL")
|
||||
rootfs = params.get("rootfs") or params.get("image") \
|
||||
or env.get("CHV_ROOTFS") or env.get("CHV_IMAGE")
|
||||
cpus = str(params.get("cpus") or env.get("CHV_CPU") or "2")
|
||||
memory = params.get("memory") or env.get("CHV_MEM") or "2G"
|
||||
state_dir = params.get("state_dir") or env.get("CHV_STATE_DIR") \
|
||||
or "/tmp/chv-vms"
|
||||
ssh_key = params.get("ssh_key") or env.get("CHV_SSH_KEY")
|
||||
if params.get("ssh_user") is not None \
|
||||
and params.get("ssh_user") == "":
|
||||
raise ValueError("CHV_SSH_USER 不能为空字符串")
|
||||
ssh_user = params.get("ssh_user") or env.get("CHV_SSH_USER") \
|
||||
or "root"
|
||||
if params.get("ssh_port") is not None:
|
||||
raw_port = params.get("ssh_port")
|
||||
else:
|
||||
raw_port = env.get("CHV_SSH_PORT") or 2222
|
||||
try:
|
||||
ssh_port = int(raw_port)
|
||||
except (TypeError, ValueError):
|
||||
raise ValueError("CHV_SSH_PORT 必须是整数")
|
||||
if not 1 <= ssh_port <= 65535:
|
||||
raise ValueError("CHV_SSH_PORT 必须在 1-65535 之间")
|
||||
guest_host_py = params.get("guest_host_py") or "/app/host.py"
|
||||
cmdline = params.get("cmdline") or env.get("CHV_CMDLINE") \
|
||||
or "console=hvc0 root=/dev/vda1 rw"
|
||||
net = params.get("net") or env.get("CHV_NET")
|
||||
net_json = params.get("net_json") or env.get("CHV_NET_JSON")
|
||||
kvm_path = params.get("kvm_path") \
|
||||
if params.get("kvm_path") is not None else "/dev/kvm"
|
||||
cpuinfo_path = params.get("cpuinfo_path") \
|
||||
if params.get("cpuinfo_path") is not None else "/proc/cpuinfo"
|
||||
ch_remote = params.get("ch_remote") or env.get("CHV_REMOTE") \
|
||||
or "ch-remote"
|
||||
model = params.get("model")
|
||||
try:
|
||||
max_vms = int(env.get("CHV_MAX_VMS") or 4)
|
||||
except ValueError:
|
||||
raise ValueError("CHV_MAX_VMS 必须是整数")
|
||||
disk_mode = env.get("CHV_DISK_MODE") or "copy"
|
||||
if disk_mode not in ("copy", "reuse"):
|
||||
raise ValueError(
|
||||
"CHV_DISK_MODE 只能是 copy 或 reuse:%s" % disk_mode)
|
||||
disk_gb_raw = env.get("CHV_DISK_GB")
|
||||
disk_gb = float(disk_gb_raw) if disk_gb_raw else None
|
||||
egress_proxy = env.get("CHV_EGRESS_PROXY")
|
||||
allow_direct_egress = env.get("CHV_ALLOW_DIRECT_EGRESS") == "1"
|
||||
task_timeout_raw = env.get("CHV_TASK_TIMEOUT")
|
||||
task_timeout = float(task_timeout_raw) \
|
||||
if task_timeout_raw else None
|
||||
snapshot_url = env.get("CHV_SNAPSHOT_URL")
|
||||
try:
|
||||
warm_pool_size = int(env.get("CHV_WARM_POOL_SIZE") or 0)
|
||||
except ValueError:
|
||||
raise ValueError("CHV_WARM_POOL_SIZE 必须是整数")
|
||||
try:
|
||||
api_timeout = float(
|
||||
params.get("api_timeout")
|
||||
if params.get("api_timeout") is not None
|
||||
else env.get("CHV_API_TIMEOUT") or 30)
|
||||
ready_timeout = float(
|
||||
params.get("ready_timeout")
|
||||
if params.get("ready_timeout") is not None
|
||||
else env.get("CHV_READY_TIMEOUT") or 120)
|
||||
ready_interval = float(
|
||||
params.get("ready_interval")
|
||||
if params.get("ready_interval") is not None
|
||||
else env.get("CHV_READY_INTERVAL") or 5)
|
||||
except ValueError:
|
||||
raise ValueError("CHV_*_TIMEOUT/INTERVAL 必须是数字")
|
||||
if ready_interval <= 0:
|
||||
ready_interval = 5
|
||||
return ChvConfig(
|
||||
binary=binary, kernel=kernel, rootfs=rootfs, cpus=cpus,
|
||||
memory=memory, state_dir=state_dir, ssh_key=ssh_key,
|
||||
ssh_user=ssh_user, ssh_port=ssh_port,
|
||||
guest_host_py=guest_host_py, cmdline=cmdline, net=net,
|
||||
net_json=net_json, kvm_path=kvm_path,
|
||||
cpuinfo_path=cpuinfo_path, ch_remote=ch_remote, model=model,
|
||||
max_vms=max_vms, disk_mode=disk_mode, disk_gb=disk_gb,
|
||||
egress_proxy=egress_proxy,
|
||||
allow_direct_egress=allow_direct_egress,
|
||||
task_timeout=task_timeout, snapshot_url=snapshot_url,
|
||||
warm_pool_size=warm_pool_size, api_timeout=api_timeout,
|
||||
ready_timeout=ready_timeout, ready_interval=ready_interval)
|
||||
@@ -0,0 +1,43 @@
|
||||
"""Cloud Hypervisor 每任务磁盘供给(叶子模块)。
|
||||
|
||||
单任务单 VM 的文件系统独立性就靠这里:copy 模式每任务
|
||||
复制一份基础镜像,两个任务绝不写同一文件;reuse 模式
|
||||
直接返回基础镜像路径(仅单租户实验,生产禁用,见
|
||||
CloudHypervisorExecutor 的类 docstring)。
|
||||
"""
|
||||
|
||||
import os
|
||||
import shutil
|
||||
|
||||
from cloud.chv.errors import ChvFailure
|
||||
|
||||
|
||||
def check_disk_quota(rootfs, disk_gb):
|
||||
"""磁盘配额门:CHV_DISK_GB 已设且基础镜像超过该 GB 数
|
||||
即拒绝(kind=disk_quota_exceeded)。未设则不检查;
|
||||
镜像读不到时不拦(preflight 自会报 image_missing,
|
||||
这里不抢它的错)。"""
|
||||
if disk_gb is None or not rootfs:
|
||||
return
|
||||
try:
|
||||
size = os.path.getsize(rootfs)
|
||||
except OSError:
|
||||
return
|
||||
if size > disk_gb * 1024 ** 3:
|
||||
raise ChvFailure(
|
||||
"disk_quota_exceeded",
|
||||
"磁盘镜像 %d 字节超过配额 %.1f GB" % (size, disk_gb))
|
||||
|
||||
|
||||
def provision_task_disk(state_dir, vm_name, rootfs, disk_mode):
|
||||
"""每任务磁盘路径:copy 模式复制到
|
||||
state_dir/<vm>/disk.raw(已存在则复用,不重复复制);
|
||||
reuse 模式直接返回基础镜像路径。"""
|
||||
if disk_mode == "reuse":
|
||||
return rootfs
|
||||
task_dir = os.path.join(state_dir, vm_name)
|
||||
os.makedirs(task_dir, exist_ok=True)
|
||||
task_disk = os.path.join(task_dir, "disk.raw")
|
||||
if not os.path.exists(task_disk):
|
||||
shutil.copyfile(rootfs, task_disk)
|
||||
return task_disk
|
||||
@@ -0,0 +1,12 @@
|
||||
"""Cloud Hypervisor 执行器的分类失败(叶子模块,不依赖
|
||||
cloud.sandbox,避免循环导入)。"""
|
||||
|
||||
class ChvFailure(RuntimeError):
|
||||
"""Cloud Hypervisor 分类失败:RuntimeError 的子类(worker
|
||||
现有 except Exception 路径照常捕获),另带 kind 字段供
|
||||
审计与失败分类用,绝不笼统只写 failed。kind 取值见
|
||||
CloudHypervisorExecutor.preflight / wake_vm 的注释。"""
|
||||
|
||||
def __init__(self, kind, message):
|
||||
super().__init__("chv_failure:%s: %s" % (kind, message))
|
||||
self.kind = kind
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,257 @@
|
||||
"""快照预热池 WarmPool(含 state_dir registry 持久化与孤儿清理)。
|
||||
|
||||
隔离红线(与 #216 一致,绝不妥协):预热条目是一次性的
|
||||
干净快照 VM,acquire 消耗掉它,release 一律销毁,绝不把
|
||||
一个任务用过的 VM(带任务文件、带 agent 痕迹)翻出来给
|
||||
下一个任务/租户复用;池子空了就回落冷启动(wake_vm),
|
||||
绝不因为池空就降级 Docker/Local。
|
||||
|
||||
Registry(本轨新增):
|
||||
- 文件:state_dir/warm-pool.json(registry_path 可覆盖),
|
||||
内容为 {<合成名>: {"state", "vm", "task_id", "snapshot_url"}}。
|
||||
每次 prewarm/acquire/release 后立即落盘(临时文件 + 原子
|
||||
rename),worker 崩溃后重启仍能发现上一世的条目。
|
||||
- 载入时绝不复用:新池实例从 registry 读到的条目一律标为
|
||||
orphan(不计入 warm_count,acquire 绝不发给任务),因为
|
||||
旧 VMM 进程已随旧 worker 消失,无法验证它还是当初那个
|
||||
干净快照 VM。它们只能走 reconcile_orphans() 清理。
|
||||
- reconcile_orphans():worker 启动/清理时的显式入口,对
|
||||
每个 orphan/assigned 条目尽力调 executor.shutdown_vm
|
||||
(清 socket 与残留登记),条目清掉并落盘,返回已清理的
|
||||
VM 名列表。registry 里没有的残留 socket 仍由执行器的
|
||||
cleanup_stale_vms() 扫 state_dir 处理,两者互补,不互相
|
||||
替代;reconcile 绝不在构造时自动执行(同一 state_dir 只
|
||||
能有一个活池,构造时自动清理会误杀另一个活池的 VM)。
|
||||
- 损坏 fail-safe:registry 不是合法 JSON / 顶层不是 dict 时,
|
||||
整个文件 quarantine 成 <path>.corrupt(尽力而为),池子
|
||||
从空开始,绝不用损坏数据复用 VM;单条目结构非法时只
|
||||
丢该条(记 registry_corrupt_entries),不连坐其它条目。
|
||||
registry 写失败时只记日志警告、不炸任务流(池子内存态
|
||||
仍可用,重启后孤儿改由 socket 扫描兜底)。
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
|
||||
_LOGGER = logging.getLogger("cloud.cloud_hypervisor")
|
||||
|
||||
_REGISTRY_FILENAME = "warm-pool.json"
|
||||
|
||||
|
||||
class WarmPool:
|
||||
"""快照预热池(快速启动设计,接口与状态机在代码里)。
|
||||
|
||||
目标:冷启动(preflight + 起 VMM + create + boot + SSH
|
||||
就绪)是慢路径;预热池提前把干净 VM 起好并暂停,任务
|
||||
来时 acquire 直接 resume,省掉冷启动的大头。
|
||||
状态机:cold(未预热)→ prewarming(wake 中)→ warm
|
||||
(已暂停待分配)→ assigned(已分配给某任务)→
|
||||
running(已 resume)→ recycled/destroyed(release 后)。
|
||||
prewarm 的实现:对每个条目调 executor.wake_vm(冷启动)
|
||||
再 pause_vm,得到 paused 的干净 VM;mock 单测里 wake/
|
||||
pause 均可注入假实现,真机上这一步需要 KVM 宿主(见
|
||||
就绪清单“需 KVM 宿主实测”)。snapshot_url 未设时
|
||||
prewarm 返回 [](无基准快照可预热,不是静默造快照)。
|
||||
"""
|
||||
|
||||
def __init__(self, executor, size=None, snapshot_url=None,
|
||||
registry_path=None):
|
||||
self.executor = executor
|
||||
self.size = int(size if size is not None
|
||||
else getattr(executor, "warm_pool_size", 0))
|
||||
self.snapshot_url = snapshot_url \
|
||||
if snapshot_url is not None \
|
||||
else getattr(executor, "snapshot_url", None)
|
||||
self._entries = {} # 合成名 -> {"state", "vm", "task", ...}
|
||||
self.registry_path = registry_path
|
||||
if self.registry_path is None:
|
||||
state_dir = getattr(executor, "state_dir", None)
|
||||
if state_dir:
|
||||
self.registry_path = os.path.join(
|
||||
state_dir, _REGISTRY_FILENAME)
|
||||
self.registry_corrupt = False
|
||||
self.registry_corrupt_entries = 0
|
||||
self._load_registry()
|
||||
|
||||
# -- registry ------------------------------------------------------
|
||||
def _load_registry(self):
|
||||
"""载入 registry:读到的条目一律标 orphan(见模块
|
||||
docstring,绝不直接当 warm 发给任务)。无文件时为空
|
||||
池;损坏时 quarantine 并从空开始。"""
|
||||
if not self.registry_path:
|
||||
return
|
||||
try:
|
||||
with open(self.registry_path, "r") as fh:
|
||||
data = json.load(fh)
|
||||
except FileNotFoundError:
|
||||
return
|
||||
except (OSError, ValueError):
|
||||
self.registry_corrupt = True
|
||||
self._quarantine_registry()
|
||||
return
|
||||
if not isinstance(data, dict):
|
||||
self.registry_corrupt = True
|
||||
self._quarantine_registry()
|
||||
return
|
||||
for name, entry in data.items():
|
||||
if not isinstance(entry, dict) \
|
||||
or not entry.get("vm") \
|
||||
or entry.get("state") not in ("warm", "assigned"):
|
||||
self.registry_corrupt_entries += 1
|
||||
continue
|
||||
self._entries[name] = {
|
||||
"state": "orphan",
|
||||
"vm": entry["vm"],
|
||||
"task": {"id": entry.get("task_id") or name},
|
||||
"snapshot_url": entry.get("snapshot_url"),
|
||||
}
|
||||
|
||||
def _quarantine_registry(self):
|
||||
"""把损坏的 registry 文件挪到 <path>.corrupt 旁(尽力
|
||||
而为),原路径不再留损坏数据给下一次载入。"""
|
||||
if not self.registry_path:
|
||||
return
|
||||
try:
|
||||
os.replace(self.registry_path,
|
||||
self.registry_path + ".corrupt")
|
||||
except OSError:
|
||||
_LOGGER.warning("chv_pool_registry_quarantine_failed "
|
||||
"path=%s", self.registry_path)
|
||||
|
||||
def _save_registry(self):
|
||||
"""落盘当前条目(只写 warm/assigned 态;orphan 是
|
||||
待清理的上一世条目,reconcile 后即删,不写回)。
|
||||
写失败只记日志,不炸任务流。"""
|
||||
if not self.registry_path:
|
||||
return
|
||||
data = {}
|
||||
for name, entry in self._entries.items():
|
||||
if entry["state"] not in ("warm", "assigned"):
|
||||
continue
|
||||
task = entry.get("task") or {}
|
||||
task_id = task.get("id") if isinstance(task, dict) \
|
||||
else None
|
||||
data[name] = {
|
||||
"state": entry["state"],
|
||||
"vm": entry.get("vm"),
|
||||
"task_id": task_id,
|
||||
"snapshot_url": entry.get("snapshot_url"),
|
||||
}
|
||||
try:
|
||||
directory = os.path.dirname(self.registry_path)
|
||||
if directory:
|
||||
os.makedirs(directory, exist_ok=True)
|
||||
tmp_path = self.registry_path + ".tmp"
|
||||
with open(tmp_path, "w") as fh:
|
||||
json.dump(data, fh)
|
||||
os.replace(tmp_path, self.registry_path)
|
||||
except OSError:
|
||||
_LOGGER.warning("chv_pool_registry_save_failed path=%s",
|
||||
self.registry_path)
|
||||
|
||||
def reconcile_orphans(self):
|
||||
"""清理上一世残留条目:对每个 orphan/assigned 条目
|
||||
尽力 shutdown_vm(清 socket 与残留登记),条目清掉
|
||||
并落盘,返回已清理的 VM 名列表。worker 启动/清理时
|
||||
显式调它(见模块 docstring 的单活池假设)。"""
|
||||
cleaned = []
|
||||
for name, entry in list(self._entries.items()):
|
||||
if entry["state"] not in ("orphan", "assigned"):
|
||||
continue
|
||||
vm = entry.get("vm") or name
|
||||
try:
|
||||
self.executor.shutdown_vm(vm)
|
||||
except Exception:
|
||||
pass
|
||||
self._entries.pop(name, None)
|
||||
cleaned.append(vm)
|
||||
if cleaned:
|
||||
self._save_registry()
|
||||
try:
|
||||
self.executor._emit(
|
||||
"warm_pool_reconcile", {"id": None},
|
||||
"cleaned=%s" % ",".join(cleaned))
|
||||
except Exception:
|
||||
pass
|
||||
return cleaned
|
||||
|
||||
# -- 池操作 ----------------------------------------------------------
|
||||
def warm_count(self):
|
||||
return sum(1 for e in self._entries.values()
|
||||
if e["state"] == "warm")
|
||||
|
||||
def orphan_count(self):
|
||||
"""上一世残留条目数(待 reconcile,不可分配)。"""
|
||||
return sum(1 for e in self._entries.values()
|
||||
if e["state"] in ("orphan", "assigned"))
|
||||
|
||||
def prewarm(self, count=None):
|
||||
"""预热到目标数:已有 warm 条目不重复建;每个新条目
|
||||
用一个合成任务名 wake + pause(合成任务 id 为
|
||||
warm-<序号>,绝不与真实任务 id 撞名)。返回新增的
|
||||
warm VM 名列表。"""
|
||||
if self.snapshot_url is None:
|
||||
return []
|
||||
target = int(count) if count is not None else self.size
|
||||
added = []
|
||||
i = 0
|
||||
while self.warm_count() < target:
|
||||
i += 1
|
||||
name = "warm-%d" % i
|
||||
if name in self._entries:
|
||||
continue
|
||||
task = {"id": name}
|
||||
self._entries[name] = {"state": "prewarming",
|
||||
"task": task}
|
||||
try:
|
||||
self.executor.wake_vm(task)
|
||||
self.executor.pause_vm(task)
|
||||
except Exception:
|
||||
self._entries.pop(name, None)
|
||||
raise
|
||||
vm = self.executor.vm_name(task)
|
||||
self._entries[name] = {
|
||||
"state": "warm", "task": task, "vm": vm,
|
||||
"snapshot_url": self.snapshot_url}
|
||||
added.append(name)
|
||||
if added:
|
||||
self._save_registry()
|
||||
return added
|
||||
|
||||
def acquire(self, task):
|
||||
"""分配一个预热 VM 给任务:有 warm 条目时消耗一个
|
||||
(状态 assigned,返回 mode=warm + 该条目的合成任务
|
||||
名,供调用方 resume 后使用);无 warm 条目时返回
|
||||
mode=cold(调用方回落 wake_vm 冷启动)。"""
|
||||
for name, entry in list(self._entries.items()):
|
||||
if entry["state"] == "warm":
|
||||
entry["state"] = "assigned"
|
||||
entry["task"] = task
|
||||
self._save_registry()
|
||||
self.executor._emit(
|
||||
"warm_pool_acquire", task,
|
||||
"vm=%s" % name)
|
||||
return {"mode": "warm", "vm": name,
|
||||
"pool_task": entry["task"]}
|
||||
return {"mode": "cold", "vm": None, "pool_task": None}
|
||||
|
||||
def release(self, task, pool_vm=None):
|
||||
"""释放:无论 warm 还是 cold 路径,VM 一律销毁
|
||||
(executor.shutdown_vm),池条目清掉;绝不把用过
|
||||
的 VM 放回 warm 池。返回 True 表示确实销毁了一个
|
||||
条目或 VM。"""
|
||||
destroyed = False
|
||||
if pool_vm and pool_vm in self._entries:
|
||||
self._entries.pop(pool_vm, None)
|
||||
destroyed = True
|
||||
try:
|
||||
if self.executor.shutdown_vm(task):
|
||||
destroyed = True
|
||||
except Exception:
|
||||
pass
|
||||
self._save_registry()
|
||||
self.executor._emit(
|
||||
"warm_pool_release", task,
|
||||
"vm=%s destroyed=%s" % (pool_vm, destroyed))
|
||||
return destroyed
|
||||
@@ -0,0 +1,93 @@
|
||||
"""Cloud Hypervisor 客机 SSH 传输(叶子模块,不依赖
|
||||
cloud.sandbox,避免循环导入)。
|
||||
|
||||
安全模型(本轨的核心变化,替代 #216 的
|
||||
StrictHostKeyChecking=no + UserKnownHostsFile=/dev/null):
|
||||
|
||||
- 每 VM 一个独立 known_hosts 文件:
|
||||
state_dir/<vm 名>/known_hosts(与每任务磁盘同目录,
|
||||
任务销毁时不自动删——known_hosts 留在宿主 state_dir 里,
|
||||
供事后审计该 VM 当时钉死的 key;VM 名本身已含任务 id,
|
||||
不会串到别的 VM 上)。
|
||||
- 首次信任(TOFU,边界必须写清,不假装是强验证):
|
||||
首次启动时客机 host key 未知,执行器用 ssh-keyscan 经
|
||||
宿主转发端口(127.0.0.1:<port>,只走本机回环)抓一次
|
||||
当时呈现的 key,写入该 VM 的 known_hosts 文件;此后
|
||||
同一 VM 的每次 SSH 都用 StrictHostKeyChecking=yes +
|
||||
该文件校验,VM 被替换 / 中间人换 key 时 SSH 直接拒绝,
|
||||
不再静默接受。首次捕获本身不验证“这台客机就是我们要
|
||||
的那台”——它只防首次之后的替换;更强的方案是镜像构建
|
||||
/cloud-init 时把客机 host key 预置进 known_hosts(未来
|
||||
工作,见 docs/chv-production-readiness.md 的待办),本轨
|
||||
在 mock 测试里只能验证“捕获→写文件→后续命令带钉死
|
||||
选项”的管线,绝不声称 mock 验证了首次密钥的真实性。
|
||||
- 密钥红线不变:argv 里只有 key 文件路径,绝无 key 值;
|
||||
known_hosts 文件只含 host key 公钥行,不含任何私钥。
|
||||
"""
|
||||
|
||||
import os
|
||||
|
||||
|
||||
def known_hosts_path(state_dir, vm_name):
|
||||
"""该 VM 的 known_hosts 文件路径:state_dir/<vm>/known_hosts。"""
|
||||
return os.path.join(state_dir, vm_name, "known_hosts")
|
||||
|
||||
|
||||
def build_ssh_command(ssh_port, ssh_key, ssh_user, egress_proxy,
|
||||
known_hosts_file, inner_cmd):
|
||||
"""构造 SSH 命令:ssh -p <端口> -i <key 文件> + 钉死选项
|
||||
+ user@127.0.0.1 + 客内命令。
|
||||
|
||||
与 #216 的区别只有 host key 选项:StrictHostKeyChecking=yes
|
||||
+ UserKnownHostsFile=<每 VM 文件>,不再是 no + /dev/null。
|
||||
BatchMode=yes 不变(绝不交互问 host key / 口令)。
|
||||
egress_proxy 已设时客内命令前加 env 前缀(代理地址本身
|
||||
不是密钥;客机出口只走该白名单代理)。"""
|
||||
inner = list(inner_cmd)
|
||||
if egress_proxy:
|
||||
inner = ["env",
|
||||
"HTTPS_PROXY=%s" % egress_proxy,
|
||||
"HTTP_PROXY=%s" % egress_proxy,
|
||||
"NO_PROXY=localhost,127.0.0.1"] + inner
|
||||
return ["ssh", "-p", str(ssh_port),
|
||||
"-i", ssh_key,
|
||||
"-o", "StrictHostKeyChecking=yes",
|
||||
"-o", "UserKnownHostsFile=%s" % known_hosts_file,
|
||||
"-o", "BatchMode=yes",
|
||||
"%s@127.0.0.1" % ssh_user] + inner
|
||||
|
||||
|
||||
def parse_keyscan_output(text):
|
||||
"""从 ssh-keyscan 的 stdout 里挑出本机回环端口的 host key
|
||||
行:形如 "[127.0.0.1]:<port> ssh-ed25519 AAAA..."(-p 时
|
||||
的标准写法)或 "127.0.0.1 ssh-rsa AAAA..."。其它主机行、
|
||||
注释行、空行一律丢掉,绝不把陌生主机的 key 写进文件。"""
|
||||
kept = []
|
||||
for line in (text or "").splitlines():
|
||||
line = line.strip()
|
||||
if not line or line.startswith("#"):
|
||||
continue
|
||||
parts = line.split()
|
||||
if len(parts) < 3:
|
||||
continue
|
||||
host, keytype = parts[0], parts[1]
|
||||
if host != "127.0.0.1" \
|
||||
and not host.startswith("[127.0.0.1]:"):
|
||||
continue
|
||||
if not keytype.startswith("ssh-"):
|
||||
continue
|
||||
kept.append(line)
|
||||
return kept
|
||||
|
||||
|
||||
def write_known_hosts(path, key_lines):
|
||||
"""原子写 known_hosts:先写同目录临时文件再 rename,
|
||||
文件权限 0600(只含公钥行,仍按敏感文件对待)。"""
|
||||
directory = os.path.dirname(path)
|
||||
if directory:
|
||||
os.makedirs(directory, exist_ok=True)
|
||||
tmp_path = path + ".tmp"
|
||||
with open(tmp_path, "w") as fh:
|
||||
fh.write("\n".join(key_lines) + "\n")
|
||||
os.chmod(tmp_path, 0o600)
|
||||
os.replace(tmp_path, path)
|
||||
@@ -0,0 +1,30 @@
|
||||
"""执行器基类(叶子模块,不依赖 cloud.sandbox 与 cloud.chv,
|
||||
避免循环导入)。
|
||||
|
||||
原来在 cloud/sandbox.py 里;拆分后 Local/Docker/Cloud
|
||||
执行器(sandbox.py 内)与 CloudHypervisorExecutor
|
||||
(cloud/chv/executor.py)共用这一个类对象,isinstance
|
||||
判断与旧路径 `from cloud.sandbox import Executor` 都不变
|
||||
(sandbox.py 会 re-export 它)。
|
||||
"""
|
||||
|
||||
|
||||
class Executor:
|
||||
"""执行器基类。
|
||||
|
||||
task 是 dict,含 id、project_id、repo、prompt、branch(可选)。
|
||||
execute(task) -> dict,至少含 executor、task_id、status、stdout。
|
||||
"""
|
||||
|
||||
def execute(self, task) -> dict:
|
||||
raise NotImplementedError("子类必须实现 execute(task)")
|
||||
|
||||
def goal_task(self, task) -> dict:
|
||||
"""目标循环(host.py goal):turn→验测→喂回重试。
|
||||
|
||||
子类实现;默认不支持(CloudExecutor 的通用 HTTP 协议里
|
||||
没有 goal/验测概念,调它会如实失败,不静默降级成普通
|
||||
dispatch)。结果 dict 除 execute() 的字段外另含:
|
||||
goal_met(True/False/None=无结论)、goal_error。
|
||||
"""
|
||||
raise NotImplementedError("该执行器不支持 goal 模式")
|
||||
+175
-1405
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,586 @@
|
||||
"""Cloud Hypervisor 优化轨(feat-chv-optimize,堆叠于 #216 之上)测试。
|
||||
|
||||
只用标准库与 mock runner/launcher/api_client,绝不启动真实
|
||||
VM(与 test_cloud_hypervisor.py 同款边界)。覆盖:
|
||||
- 拆模块:cloud/chv/* 新路径与 cloud/sandbox 旧路径是同一对象
|
||||
- 去重复:Local 与 CHV 的 goal 解析在矩阵输入下结论全等;
|
||||
三执行器 host 顶层参数顺序全等;runner 超时 helper
|
||||
- 安全:每 VM known_hosts 钉死、ssh-keyscan 首次捕获(TOFU)、
|
||||
非 root 用户与端口显式配置与收紧校验
|
||||
- execute 语义:只读已运行 VM,未运行报 vm_not_running
|
||||
- 预热池:registry 持久化、重启载入只当孤儿、reconcile 清理、
|
||||
损坏 quarantine、一 次性不复用
|
||||
- 长任务:VM 中途死亡判 interrupted 而非 failed/抛异常
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
from cloud.chv import ChvFailure
|
||||
from cloud.chv import CloudHypervisorExecutor
|
||||
from cloud.chv import WarmPool
|
||||
from cloud.chv.api_client import _unix_socket_api as _chv_api
|
||||
from cloud.chv.config import _parse_memory_bytes as _chv_mem
|
||||
from cloud.sandbox import ChvFailure as _sb_failure
|
||||
from cloud.sandbox import CloudHypervisorExecutor as _sb_exec
|
||||
from cloud.sandbox import DockerExecutor
|
||||
from cloud.sandbox import LocalExecutor
|
||||
from cloud.sandbox import WarmPool as _sb_pool
|
||||
from cloud.sandbox import _call_runner_with_timeout
|
||||
from cloud.sandbox import _parse_memory_bytes as _sb_mem
|
||||
from cloud.sandbox import _unix_socket_api as _sb_api
|
||||
except ImportError: # discover -s cloud/tests 时把项目根目录补进 sys.path
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
from cloud.chv import ChvFailure
|
||||
from cloud.chv import CloudHypervisorExecutor
|
||||
from cloud.chv import WarmPool
|
||||
from cloud.chv.api_client import _unix_socket_api as _chv_api
|
||||
from cloud.chv.config import _parse_memory_bytes as _chv_mem
|
||||
from cloud.sandbox import ChvFailure as _sb_failure
|
||||
from cloud.sandbox import CloudHypervisorExecutor as _sb_exec
|
||||
from cloud.sandbox import DockerExecutor
|
||||
from cloud.sandbox import LocalExecutor
|
||||
from cloud.sandbox import WarmPool as _sb_pool
|
||||
from cloud.sandbox import _call_runner_with_timeout
|
||||
from cloud.sandbox import _parse_memory_bytes as _sb_mem
|
||||
from cloud.sandbox import _unix_socket_api as _sb_api
|
||||
|
||||
|
||||
class FakeResult:
|
||||
def __init__(self, returncode=0, stdout="", stderr=""):
|
||||
self.returncode = returncode
|
||||
self.stdout = stdout
|
||||
self.stderr = stderr
|
||||
|
||||
|
||||
class FakeProc:
|
||||
def __init__(self, alive=True):
|
||||
self._alive = alive
|
||||
self.terminated = False
|
||||
|
||||
def poll(self):
|
||||
return None if self._alive else 1
|
||||
|
||||
def terminate(self):
|
||||
self.terminated = True
|
||||
self._alive = False
|
||||
|
||||
def wait(self, timeout=None):
|
||||
return 0
|
||||
|
||||
def kill(self):
|
||||
self._alive = False
|
||||
|
||||
|
||||
def _write(path, content=b"x"):
|
||||
with open(path, "wb") as fh:
|
||||
fh.write(content)
|
||||
os.chmod(path, 0o755)
|
||||
return path
|
||||
|
||||
|
||||
class ChvOptCase(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.dir = self.tmp.name
|
||||
self.binary = _write(os.path.join(self.dir, "cloud-hypervisor"))
|
||||
self.ch_remote = _write(os.path.join(self.dir, "ch-remote"))
|
||||
self.kernel = _write(os.path.join(self.dir, "vmlinux"))
|
||||
self.rootfs = _write(os.path.join(self.dir, "disk.raw"),
|
||||
b"R" * 100)
|
||||
self.ssh_key = _write(os.path.join(self.dir, "id_chv"))
|
||||
self.kvm = _write(os.path.join(self.dir, "kvm"))
|
||||
self.cpuinfo = os.path.join(self.dir, "cpuinfo")
|
||||
with open(self.cpuinfo, "w") as fh:
|
||||
fh.write("flags : fpu vmx sse\n")
|
||||
self.state_dir = os.path.join(self.dir, "state")
|
||||
self.calls = []
|
||||
self.api_calls = []
|
||||
self.launches = []
|
||||
|
||||
def tearDown(self):
|
||||
self.tmp.cleanup()
|
||||
|
||||
def make_exec(self, runner=None, api_client=None, launcher=None,
|
||||
env_extra=None, **overrides):
|
||||
env = {"CHV_EGRESS_PROXY": "http://10.0.0.1:8888"}
|
||||
if env_extra:
|
||||
env.update(env_extra)
|
||||
if runner is None:
|
||||
runner = self._default_runner
|
||||
if api_client is None:
|
||||
api_client = self._default_api
|
||||
if launcher is None:
|
||||
launcher = self._default_launcher
|
||||
kwargs = dict(
|
||||
binary=self.binary, ch_remote=self.ch_remote,
|
||||
kernel=self.kernel, rootfs=self.rootfs,
|
||||
ssh_key=self.ssh_key, kvm_path=self.kvm,
|
||||
cpuinfo_path=self.cpuinfo, state_dir=self.state_dir,
|
||||
runner=runner, launcher=launcher, api_client=api_client,
|
||||
sleeper=lambda s: None, env=env)
|
||||
kwargs.update(overrides)
|
||||
return CloudHypervisorExecutor(**kwargs)
|
||||
|
||||
def _default_runner(self, cmd, timeout=None):
|
||||
self.calls.append(list(cmd))
|
||||
if cmd[0] == "ssh":
|
||||
if cmd[-1] == "true":
|
||||
return FakeResult(0, "", "")
|
||||
if "dispatch" in cmd:
|
||||
return FakeResult(0, json.dumps(
|
||||
{"status": "completed", "text": "ok-dispatch"}), "")
|
||||
if "goal" in cmd:
|
||||
return FakeResult(0, "goal met after rounds\n", "")
|
||||
if "status" in cmd:
|
||||
return FakeResult(0, json.dumps(
|
||||
{"status": "completed", "text": "ok-status"}), "")
|
||||
return FakeResult(0, "", "")
|
||||
return FakeResult(0, "", "")
|
||||
|
||||
def _default_api(self, socket_path, method, path, payload=None):
|
||||
self.api_calls.append((method, path, payload))
|
||||
return (200, {})
|
||||
|
||||
def _default_launcher(self, cmd):
|
||||
self.launches.append(list(cmd))
|
||||
return FakeProc(alive=True)
|
||||
|
||||
def _task(self, task_id="abc123"):
|
||||
return {"id": task_id, "prompt": "做一个加法函数",
|
||||
"project_id": "proj1"}
|
||||
|
||||
|
||||
# -- 拆模块 ---------------------------------------------------------------
|
||||
class SplitTest(unittest.TestCase):
|
||||
def test_new_and_old_import_paths_are_same_objects(self):
|
||||
self.assertIs(CloudHypervisorExecutor, _sb_exec)
|
||||
self.assertIs(WarmPool, _sb_pool)
|
||||
self.assertIs(ChvFailure, _sb_failure)
|
||||
self.assertIs(_chv_mem, _sb_mem)
|
||||
self.assertIs(_chv_api, _sb_api)
|
||||
|
||||
def test_classes_live_in_chv_modules(self):
|
||||
self.assertEqual(CloudHypervisorExecutor.__module__,
|
||||
"cloud.chv.executor")
|
||||
self.assertEqual(WarmPool.__module__, "cloud.chv.pool")
|
||||
self.assertEqual(ChvFailure.__module__, "cloud.chv.errors")
|
||||
|
||||
|
||||
# -- 去重复:goal 解析与 host 命令 ------------------------------------------
|
||||
class SharedGoalTest(unittest.TestCase):
|
||||
CASES = [
|
||||
# (returncode, stdout, stderr)
|
||||
(0, "goal met after rounds\n", ""),
|
||||
(0, "round 1 verify_out=bad\ngoal unmet after rounds\n", ""),
|
||||
(1, "goal interrupted after rounds: boom\n", ""),
|
||||
(1, "KeyError: 'x'\n", ""),
|
||||
(1, "some odd failure\n", ""),
|
||||
(0, "nothing conclusive\n", ""),
|
||||
(1, "goal-resume 拒绝:断点丢失\n", ""),
|
||||
(1, "任务已存在\n", ""),
|
||||
(1, "goal met after rounds\n", ""),
|
||||
(1, "goal unmet after rounds\n", ""),
|
||||
(1, "", "HTTP 503 Service Unavailable\n"),
|
||||
]
|
||||
|
||||
def test_local_and_chv_goal_conclusions_match(self):
|
||||
local = LocalExecutor("/tmp/host.py")
|
||||
chv = ChvOptCase()
|
||||
chv.setUp()
|
||||
try:
|
||||
ex = chv.make_exec()
|
||||
for rc, out, err in self.CASES:
|
||||
res = FakeResult(rc, out, err)
|
||||
a = local._goal_result_from_runner("t1", res)
|
||||
b = ex._goal_result("t1", res)
|
||||
self.assertEqual(
|
||||
(a["status"], a["goal_met"]),
|
||||
(b["status"], b["goal_met"]),
|
||||
"rc=%s stdout=%r" % (rc, out))
|
||||
self.assertEqual(a["executor"], "local")
|
||||
self.assertEqual(b["executor"], "cloud-hypervisor")
|
||||
finally:
|
||||
chv.tearDown()
|
||||
|
||||
def test_already_exists_message_points_to_continue(self):
|
||||
chv = ChvOptCase()
|
||||
chv.setUp()
|
||||
try:
|
||||
ex = chv.make_exec()
|
||||
out = ex._goal_result(
|
||||
"t1", FakeResult(1, "任务已存在\n", ""))
|
||||
self.assertEqual(out["status"], "failed")
|
||||
self.assertIn("继续", out["goal_error"])
|
||||
finally:
|
||||
chv.tearDown()
|
||||
|
||||
def test_host_command_options_identical_across_executors(self):
|
||||
chv = ChvOptCase()
|
||||
chv.setUp()
|
||||
try:
|
||||
ex = chv.make_exec()
|
||||
local = LocalExecutor("/tmp/host.py")
|
||||
docker = DockerExecutor()
|
||||
for kwargs in (
|
||||
{"model": "m1", "effort": "low", "approval": "ask"},
|
||||
{"model": "m1", "effort": "low", "approval": "full",
|
||||
"sandbox": "read-only",
|
||||
"model_provider": "sub2api"},
|
||||
):
|
||||
lcmd = local._base_command(**kwargs)
|
||||
dcmd = docker._inner_host_command("status", **kwargs)
|
||||
ccmd = ex._guest_host_command("status", **kwargs)
|
||||
self.assertEqual(lcmd[2:], dcmd[2:-1])
|
||||
self.assertEqual(lcmd[2:], ccmd[2:-1])
|
||||
finally:
|
||||
chv.tearDown()
|
||||
|
||||
def test_runner_timeout_helper(self):
|
||||
seen = []
|
||||
|
||||
def only_cmd(cmd):
|
||||
seen.append(("cmd-only", list(cmd)))
|
||||
return FakeResult(0, "", "")
|
||||
|
||||
out = _call_runner_with_timeout(only_cmd, ["x"], timeout=5)
|
||||
self.assertEqual(out.returncode, 0)
|
||||
self.assertEqual(seen, [("cmd-only", ["x"])])
|
||||
|
||||
def with_timeout(cmd, timeout=None):
|
||||
seen.append(("with-timeout", timeout))
|
||||
return FakeResult(0, "", "")
|
||||
|
||||
_call_runner_with_timeout(with_timeout, ["x"], timeout=7)
|
||||
self.assertEqual(seen[-1], ("with-timeout", 7))
|
||||
_call_runner_with_timeout(with_timeout, ["x"], timeout=None)
|
||||
self.assertEqual(seen[-1], ("with-timeout", None))
|
||||
|
||||
|
||||
# -- 安全:known_hosts 钉死 -------------------------------------------------
|
||||
class SshPinningTest(ChvOptCase):
|
||||
def test_known_hosts_path_is_per_vm(self):
|
||||
ex = self.make_exec()
|
||||
a = ex.known_hosts_path({"id": "t-a"})
|
||||
b = ex.known_hosts_path({"id": "t-b"})
|
||||
self.assertNotEqual(a, b)
|
||||
self.assertEqual(a, os.path.join(self.state_dir, "chv-t-a",
|
||||
"known_hosts"))
|
||||
|
||||
def test_capture_host_key_writes_pinned_file_0600(self):
|
||||
def keyscan(cmd, timeout=None):
|
||||
self.calls.append(list(cmd))
|
||||
return FakeResult(
|
||||
0, "[127.0.0.1]:2222 ssh-ed25519 AAAAB3test\n"
|
||||
"evil.example.com ssh-rsa AAAAEvil\n"
|
||||
"# comment\n", "")
|
||||
ex = self.make_exec(runner=keyscan)
|
||||
self.assertTrue(ex.capture_host_key(self._task()))
|
||||
path = ex.known_hosts_path(self._task())
|
||||
with open(path) as fh:
|
||||
content = fh.read()
|
||||
self.assertIn("[127.0.0.1]:2222 ssh-ed25519 AAAAB3test",
|
||||
content)
|
||||
self.assertNotIn("evil.example.com", content)
|
||||
self.assertEqual(os.stat(path).st_mode & 0o777, 0o600)
|
||||
|
||||
def test_capture_host_key_empty_is_false_not_error(self):
|
||||
ex = self.make_exec(
|
||||
runner=lambda cmd, timeout=None: FakeResult(0, "", ""))
|
||||
self.assertFalse(ex.capture_host_key(self._task()))
|
||||
self.assertFalse(os.path.exists(
|
||||
ex.known_hosts_path(self._task())))
|
||||
|
||||
def test_wait_guest_ready_captures_before_ssh(self):
|
||||
def runner(cmd, timeout=None):
|
||||
self.calls.append(list(cmd))
|
||||
if cmd[0] == "ssh-keyscan":
|
||||
return FakeResult(
|
||||
0, "[127.0.0.1]:2222 ssh-ed25519 AAAAB3test\n",
|
||||
"")
|
||||
if cmd[0] == "ssh" and cmd[-1] == "true":
|
||||
return FakeResult(0, "", "")
|
||||
return FakeResult(0, "", "")
|
||||
ex = self.make_exec(runner=runner)
|
||||
self.assertTrue(ex.wait_guest_ready(self._task()))
|
||||
self.assertTrue(os.path.exists(
|
||||
ex.known_hosts_path(self._task())))
|
||||
keyscan_idx = next(i for i, c in enumerate(self.calls)
|
||||
if c[0] == "ssh-keyscan")
|
||||
ssh_idx = next(i for i, c in enumerate(self.calls)
|
||||
if c[0] == "ssh" and c[-1] == "true")
|
||||
self.assertLess(keyscan_idx, ssh_idx)
|
||||
|
||||
def test_ssh_user_port_explicit_config(self):
|
||||
ex = self.make_exec(ssh_user="codex", ssh_port=2201)
|
||||
cmd = ex.build_ssh_command(self._task(), ["true"])
|
||||
joined = " ".join(cmd)
|
||||
self.assertIn("codex@127.0.0.1", joined)
|
||||
self.assertNotIn("root@127.0.0.1", joined)
|
||||
self.assertIn("2201", joined)
|
||||
|
||||
def test_ssh_user_port_tightened_validation(self):
|
||||
with self.assertRaises(ValueError):
|
||||
self.make_exec(ssh_user="")
|
||||
with self.assertRaises(ValueError):
|
||||
self.make_exec(ssh_port=0)
|
||||
with self.assertRaises(ValueError):
|
||||
self.make_exec(ssh_port=70000)
|
||||
with self.assertRaises(ValueError):
|
||||
self.make_exec(ssh_port="abc")
|
||||
with self.assertRaises(ValueError):
|
||||
self.make_exec(env_extra={"CHV_SSH_PORT": "0"})
|
||||
|
||||
|
||||
# -- execute 语义 -------------------------------------------------------------
|
||||
class ExecuteGateTest(ChvOptCase):
|
||||
def test_execute_without_running_vm_is_classified(self):
|
||||
ex = self.make_exec()
|
||||
with self.assertRaises(ChvFailure) as ctx:
|
||||
ex.execute(self._task())
|
||||
self.assertEqual(ctx.exception.kind, "vm_not_running")
|
||||
self.assertEqual(self.launches, [])
|
||||
self.assertFalse(any(c[0] == "ssh" for c in self.calls))
|
||||
|
||||
def test_execute_with_dead_proc_and_no_socket_is_classified(self):
|
||||
ex = self.make_exec()
|
||||
ex._vms["chv-abc123"] = FakeProc(alive=False)
|
||||
with self.assertRaises(ChvFailure) as ctx:
|
||||
ex.execute(self._task())
|
||||
self.assertEqual(ctx.exception.kind, "vm_not_running")
|
||||
|
||||
def test_execute_with_socket_but_no_registry_proceeds(self):
|
||||
ex = self.make_exec()
|
||||
os.makedirs(self.state_dir, exist_ok=True)
|
||||
with open(os.path.join(self.state_dir,
|
||||
"chv-abc123.sock"), "w") as fh:
|
||||
fh.write("")
|
||||
out = ex.execute(self._task())
|
||||
self.assertIn("ok-status", out["stdout"])
|
||||
self.assertEqual(self.launches, [])
|
||||
|
||||
|
||||
# -- 预热池持久化与孤儿清理 ----------------------------------------------------
|
||||
class StubExecutor:
|
||||
"""WarmPool 测试替身:只实现池子调用的鸭子接口。"""
|
||||
|
||||
def __init__(self, state_dir):
|
||||
self.state_dir = state_dir
|
||||
self.warm_pool_size = 2
|
||||
self.snapshot_url = "file:///snap/base"
|
||||
self.wake_calls = []
|
||||
self.pause_calls = []
|
||||
self.shutdown_calls = []
|
||||
self.audit = []
|
||||
|
||||
def vm_name(self, task):
|
||||
raw = task.get("id") if isinstance(task, dict) else str(task)
|
||||
return raw if raw.startswith("chv-") else "chv-" + raw
|
||||
|
||||
def wake_vm(self, task):
|
||||
self.wake_calls.append(task["id"])
|
||||
return {"vm": self.vm_name(task), "state": "running"}
|
||||
|
||||
def pause_vm(self, task):
|
||||
self.pause_calls.append(task["id"])
|
||||
return True
|
||||
|
||||
def shutdown_vm(self, task_or_name):
|
||||
name = task_or_name if isinstance(task_or_name, str) \
|
||||
else self.vm_name(task_or_name)
|
||||
self.shutdown_calls.append(name)
|
||||
return True
|
||||
|
||||
def _emit(self, event, task, detail="", kind=None):
|
||||
self.audit.append(event)
|
||||
|
||||
|
||||
class PoolRegistryTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.state_dir = os.path.join(self.tmp.name, "state")
|
||||
os.makedirs(self.state_dir, exist_ok=True)
|
||||
self.registry = os.path.join(self.state_dir, "warm-pool.json")
|
||||
|
||||
def tearDown(self):
|
||||
self.tmp.cleanup()
|
||||
|
||||
def test_persist_load_orphan_reconcile(self):
|
||||
stub = StubExecutor(self.state_dir)
|
||||
pool = WarmPool(stub, size=2, snapshot_url="file:///snap/base")
|
||||
added = pool.prewarm()
|
||||
self.assertEqual(len(added), 2)
|
||||
self.assertTrue(os.path.exists(self.registry))
|
||||
with open(self.registry) as fh:
|
||||
data = json.load(fh)
|
||||
self.assertEqual(len(data), 2)
|
||||
# 新池实例载入:上一世条目只当孤儿,绝不发给任务:
|
||||
stub2 = StubExecutor(self.state_dir)
|
||||
pool2 = WarmPool(stub2, size=2,
|
||||
snapshot_url="file:///snap/base")
|
||||
self.assertEqual(pool2.warm_count(), 0)
|
||||
self.assertEqual(pool2.orphan_count(), 2)
|
||||
self.assertEqual(pool2.acquire({"id": "t1"})["mode"], "cold")
|
||||
cleaned = pool2.reconcile_orphans()
|
||||
self.assertEqual(len(cleaned), 2)
|
||||
self.assertEqual(sorted(stub2.shutdown_calls), sorted(cleaned))
|
||||
self.assertEqual(pool2.orphan_count(), 0)
|
||||
with open(self.registry) as fh:
|
||||
self.assertEqual(json.load(fh), {})
|
||||
# 第三实例:无残留:
|
||||
pool3 = WarmPool(StubExecutor(self.state_dir), size=2,
|
||||
snapshot_url="file:///snap/base")
|
||||
self.assertEqual(pool3.orphan_count(), 0)
|
||||
|
||||
def test_corrupt_registry_quarantines_and_starts_empty(self):
|
||||
with open(self.registry, "w") as fh:
|
||||
fh.write("{not json")
|
||||
pool = WarmPool(StubExecutor(self.state_dir), size=2,
|
||||
snapshot_url="file:///snap/base")
|
||||
self.assertTrue(pool.registry_corrupt)
|
||||
self.assertEqual(pool.warm_count(), 0)
|
||||
self.assertEqual(pool.orphan_count(), 0)
|
||||
self.assertTrue(os.path.exists(self.registry + ".corrupt"))
|
||||
self.assertFalse(os.path.exists(self.registry))
|
||||
# 损坏后仍可正常预热(registry 重建):
|
||||
self.assertEqual(len(pool.prewarm()), 2)
|
||||
self.assertTrue(os.path.exists(self.registry))
|
||||
|
||||
def test_assigned_entry_on_load_is_never_served(self):
|
||||
with open(self.registry, "w") as fh:
|
||||
json.dump({"warm-1": {"state": "assigned",
|
||||
"vm": "chv-warm-1",
|
||||
"task_id": "old-task",
|
||||
"snapshot_url": "file:///snap/base"}},
|
||||
fh)
|
||||
pool = WarmPool(StubExecutor(self.state_dir), size=2,
|
||||
snapshot_url="file:///snap/base")
|
||||
self.assertEqual(pool.warm_count(), 0)
|
||||
self.assertEqual(pool.acquire({"id": "t1"})["mode"], "cold")
|
||||
self.assertEqual(pool.reconcile_orphans(), ["chv-warm-1"])
|
||||
|
||||
def test_release_never_rewarms_and_registry_clears(self):
|
||||
stub = StubExecutor(self.state_dir)
|
||||
pool = WarmPool(stub, size=1, snapshot_url="file:///snap/base")
|
||||
added = pool.prewarm()
|
||||
got = pool.acquire({"id": "t1"})
|
||||
self.assertEqual(got["mode"], "warm")
|
||||
self.assertTrue(pool.release({"id": "t1"}, pool_vm=added[0]))
|
||||
self.assertEqual(pool.warm_count(), 0)
|
||||
with open(self.registry) as fh:
|
||||
self.assertEqual(json.load(fh), {})
|
||||
|
||||
|
||||
# -- 长任务中途死亡 --------------------------------------------------------------
|
||||
class MidDeathTest(ChvOptCase):
|
||||
def _dispatch_with(self, runner):
|
||||
ex = self.make_exec(runner=runner)
|
||||
return ex.dispatch_task(self._task())
|
||||
|
||||
def test_failed_result_with_dead_vm_is_interrupted(self):
|
||||
proc_holder = []
|
||||
|
||||
def launcher(cmd):
|
||||
proc = FakeProc(alive=True)
|
||||
proc_holder.append(proc)
|
||||
self.launches.append(list(cmd))
|
||||
return proc
|
||||
|
||||
def runner(cmd, timeout=None):
|
||||
self.calls.append(list(cmd))
|
||||
if cmd[0] == "ssh" and "dispatch" in cmd:
|
||||
proc_holder[0]._alive = False
|
||||
return FakeResult(1, "",
|
||||
"Connection closed by remote host")
|
||||
if cmd[0] == "ssh":
|
||||
return FakeResult(0, "", "")
|
||||
return FakeResult(0, "", "")
|
||||
|
||||
ex = self.make_exec(runner=runner, launcher=launcher)
|
||||
out = ex.dispatch_task(self._task())
|
||||
self.assertEqual(out["status"], "interrupted")
|
||||
self.assertTrue(out["vm_dead"])
|
||||
|
||||
def test_failed_result_with_live_vm_stays_failed(self):
|
||||
def runner(cmd, timeout=None):
|
||||
self.calls.append(list(cmd))
|
||||
if cmd[0] == "ssh" and "dispatch" in cmd:
|
||||
return FakeResult(1, "", "host dispatch failed")
|
||||
if cmd[0] == "ssh":
|
||||
return FakeResult(0, "", "")
|
||||
return FakeResult(0, "", "")
|
||||
|
||||
ex = self.make_exec(runner=runner)
|
||||
out = ex.dispatch_task(self._task())
|
||||
self.assertEqual(out["status"], "failed")
|
||||
self.assertNotIn("vm_dead", out)
|
||||
|
||||
def test_exception_with_dead_vm_is_interrupted_not_raised(self):
|
||||
proc_holder = []
|
||||
|
||||
def launcher(cmd):
|
||||
proc = FakeProc(alive=True)
|
||||
proc_holder.append(proc)
|
||||
self.launches.append(list(cmd))
|
||||
return proc
|
||||
|
||||
def runner(cmd, timeout=None):
|
||||
self.calls.append(list(cmd))
|
||||
if cmd[0] == "ssh" and "dispatch" in cmd:
|
||||
proc_holder[0]._alive = False
|
||||
raise RuntimeError("ssh transport broken")
|
||||
if cmd[0] == "ssh":
|
||||
return FakeResult(0, "", "")
|
||||
return FakeResult(0, "", "")
|
||||
|
||||
ex = self.make_exec(runner=runner, launcher=launcher)
|
||||
out = ex.dispatch_task(self._task())
|
||||
self.assertEqual(out["status"], "interrupted")
|
||||
self.assertTrue(out["vm_dead"])
|
||||
|
||||
def test_exception_with_live_vm_propagates(self):
|
||||
def runner(cmd, timeout=None):
|
||||
self.calls.append(list(cmd))
|
||||
if cmd[0] == "ssh" and "dispatch" in cmd:
|
||||
raise RuntimeError("ssh transport broken")
|
||||
if cmd[0] == "ssh":
|
||||
return FakeResult(0, "", "")
|
||||
return FakeResult(0, "", "")
|
||||
|
||||
ex = self.make_exec(runner=runner)
|
||||
with self.assertRaises(RuntimeError):
|
||||
ex.dispatch_task(self._task())
|
||||
|
||||
def test_goal_exception_with_dead_vm_is_interrupted(self):
|
||||
proc_holder = []
|
||||
|
||||
def launcher(cmd):
|
||||
proc = FakeProc(alive=True)
|
||||
proc_holder.append(proc)
|
||||
self.launches.append(list(cmd))
|
||||
return proc
|
||||
|
||||
def runner(cmd, timeout=None):
|
||||
self.calls.append(list(cmd))
|
||||
if cmd[0] == "ssh" and "goal" in cmd:
|
||||
proc_holder[0]._alive = False
|
||||
raise RuntimeError("ssh transport broken")
|
||||
if cmd[0] == "ssh":
|
||||
return FakeResult(0, "", "")
|
||||
return FakeResult(0, "", "")
|
||||
|
||||
ex = self.make_exec(runner=runner, launcher=launcher)
|
||||
out = ex.goal_task({"id": "g1", "prompt": "做目标",
|
||||
"goal_mode": True})
|
||||
self.assertEqual(out["status"], "interrupted")
|
||||
self.assertIsNone(out["goal_met"])
|
||||
self.assertTrue(out["vm_dead"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -279,6 +279,14 @@ class ConfigCommandTest(ChvCase):
|
||||
self.assertNotIn("SECRET", joined)
|
||||
# egress 代理地址在 env 前缀里(不是密钥),且只在代理已设时出现:
|
||||
self.assertIn("HTTPS_PROXY=http://10.0.0.1:8888", joined)
|
||||
# 本轨安全默认变化(#216 的 no + /dev/null 已废除):
|
||||
# host key 钉死在每 VM 的 known_hosts 文件上:
|
||||
self.assertIn("StrictHostKeyChecking=yes", joined)
|
||||
self.assertNotIn("StrictHostKeyChecking=no", joined)
|
||||
self.assertNotIn("/dev/null", joined)
|
||||
self.assertIn(os.path.join(self.state_dir,
|
||||
"chv-abc123", "known_hosts"),
|
||||
joined)
|
||||
|
||||
def test_isolation_names_and_disks_differ(self):
|
||||
ex = self.make_exec()
|
||||
@@ -534,6 +542,10 @@ class DispatchTest(ChvCase):
|
||||
|
||||
def test_execute_is_ssh_status_only(self):
|
||||
ex = self.make_exec()
|
||||
# 本轨 execute 语义门禁:只读已运行 VM。先登记一个活
|
||||
# 的 VMM 进程表示 VM 已醒(未登记时 execute 报
|
||||
# vm_not_running,见 test_chv_optimize.py):
|
||||
ex._vms["chv-abc123"] = FakeProc(alive=True)
|
||||
out = ex.execute(self._task())
|
||||
self.assertEqual(out["executor"], "cloud-hypervisor")
|
||||
self.assertIn("ok-status", out["stdout"])
|
||||
|
||||
@@ -42,6 +42,17 @@
|
||||
- ✅🧪 密钥不进镜像/代码/prompt:SSH argv 只有 key 文件路径;
|
||||
模型 key 由客机 cloud-init 预置的环境变量名注入;测试断言
|
||||
命令中不含哨兵密钥值。
|
||||
- ✅🧪 SSH host key 按 VM 钉死(优化轨,#216 的 no + /dev/null
|
||||
已废除):每 VM 在 state_dir/<vm>/known_hosts 有独立文件,
|
||||
首次启动经 ssh-keyscan 在本机回环端口捕获 key 写入该文件,
|
||||
此后 SSH 一律 StrictHostKeyChecking=yes + 该文件校验;
|
||||
非 root 客机用户(CHV_SSH_USER,显式空串直接拒绝)与端口
|
||||
(CHV_SSH_PORT,1-65535)为显式配置。mock 只验证捕获→
|
||||
写文件→钉死选项的管线,不验证首次密钥真实性。
|
||||
- 🔒 首次信任边界未闭合:首次捕获是 TOFU,只防首次之后的
|
||||
替换,不验证首次本身;更强的方案是镜像构建/cloud-init 时
|
||||
把客机 host key 预置进 known_hosts,仍是待办,安全审计时
|
||||
须单独立项确认。
|
||||
- 🔒 需安全审计:VM 逃逸面(virtio 设备、tap 桥接、SSH 配置)、
|
||||
客机 root 提权面、租户间东西向隔离的宿主网络配置,须由
|
||||
安全审计逐项确认,代码自查不能代替审计。
|
||||
@@ -55,6 +66,12 @@
|
||||
冷启动,绝不降级 Docker/Local。
|
||||
- ✅🧪 预热条目一次性:acquire 消耗、release 一律销毁,绝不跨
|
||||
任务复用脏 VM。
|
||||
- ✅🧪 预热池 registry 与孤儿清理(优化轨):池状态落
|
||||
state_dir/warm-pool.json(原子写),worker 重启后载入的
|
||||
条目一律当孤儿、只能 reconcile_orphans() 清理,绝不重新
|
||||
发给任务;registry 损坏时 quarantine 成 .corrupt 并从空池
|
||||
开始;registry 之外的残留 socket 仍由 cleanup_stale_vms()
|
||||
扫描处理。
|
||||
- ⏳ 需 KVM 宿主实测:预热池的真实冷启动耗时、resume 耗时、
|
||||
池大小与命中率、快照来源一致性,均需真机数据,mock 数字
|
||||
不能当性能结论。
|
||||
@@ -75,6 +92,17 @@
|
||||
shutdown + 杀进程 + 删 socket)、recycle_vm(保留
|
||||
task_checkpoints/task_attempts,丢失 guest /tmp 与未拷出
|
||||
的 worktree 临时物,绝不声称恢复临时文件)。
|
||||
- ✅🧪 execute 只读已运行 VM(优化轨):execute() 先查 VM
|
||||
是否在运行(登记活进程或 api-socket 存在),未运行报
|
||||
ChvFailure(kind=vm_not_running),不为一次 status 起 VM,
|
||||
也不向不存在的端口发 SSH 再拿连接失败冒充任务状态。
|
||||
- ✅🧪 长任务中途死亡判 interrupted(优化轨):客内命令抛
|
||||
传输异常、或非 goal 结果已 failed 且登记 VMM 进程已死时,
|
||||
_run_with_vm 改判 interrupted(断点可续跑)而非笼统
|
||||
failed;VM 还活着时的异常/失败原样处理。中途心跳未做:
|
||||
host.py 单次 SSH 长回合没有客内进度信号可采,另起一路
|
||||
SSH 轮询只能证明 VM 活着、不能证明任务在进展,评估后
|
||||
认为风险大于收益,宁可只做死亡分类。
|
||||
- ⏳ 需 KVM 宿主实测:上述全部生命周期在真机上的状态转换、
|
||||
快照恢复后 guest 状态一致性、休眠后文件/worktree 保留,
|
||||
均未在真机验证。
|
||||
|
||||
Reference in new issue
Block a user