feat: Cloud Hypervisor 执行器生产级 v1（cloud-hypervisor）
单任务单 VM：API 唤醒（preflight→vmm.ping→vm.create→vm.boot→SSH 就绪）、 暂停/快照/休眠等价/关闭/回收、预热快照池、每任务磁盘 copy、容量与磁盘配额门、 受控 egress fail-closed、VM 事件审计与分类失败；worker/server/autoscaler 接线 （executor=cloud-hypervisor，任务内 LLM 不可自选）；默认关闭，不进生产路由。 未做真实 VM 验证：生产主机与本 VM 均无 /dev/kvm、无嵌套虚拟化，验证为 mock + 干跑命令 + 单元测试；真 VM、快照恢复、压测、安全审计见 docs/chv-production-readiness.md。CI 账单故障仍是硬门禁，本提交只到 PR OPEN。
This commit is contained in:
host committed
1 parent
aebc6238a6
commit
c371bbdf1e
9 files changed
+2364
-11
No files matched your search
@@ -0,0 +1,71 @@
|
||||
# Cloud Hypervisor 客机镜像（chv-guest）——内容清单与版本化构建说明
|
||||
|
||||
> 本目录只有清单与构建脚本骨架，镜像本身不在本轨内实际构建
|
||||
> 发布。原因写在前面，不藏着：当前没有带 /dev/kvm 的宿主
|
||||
> 可以构建并实测 VM 镜像（生产主机是 CVM、无嵌套虚拟化；
|
||||
> 本 VM 容器内同样无 /dev/kvm）。构建与实测见
|
||||
> `docs/chv-production-readiness.md` 的“需镜像构建 / 需 KVM
|
||||
> 宿主实测”项，绝不在本轨声称镜像已建成、已发布、已跑通。
|
||||
|
||||
## 1. 镜像定位
|
||||
|
||||
- 基础：Ubuntu 云镜像（与 `cloud/docker/Dockerfile` 的任务镜像
|
||||
对齐能力，不是另起一套环境口径）。
|
||||
- 载体：Cloud Hypervisor 直接内核启动（vmlinux）+ raw 磁盘
|
||||
镜像，执行器用 `CHV_KERNEL` / `CHV_ROOTFS` 指向它们。
|
||||
- 客内 host.py 位置：`/app/host.py`（与 Docker 的 `/app:ro`
|
||||
同款约定，只是载体从容器只读挂载换成 VM 镜像内置代码；
|
||||
代码版本 = 构建时仓库 commit，见 §3 版本化）。
|
||||
- 绝不进镜像的东西：任何模型 key、系统 key、用户账号数据、
|
||||
生产凭证。模型 key 由客机 cloud-init 预置的运行环境变量名
|
||||
注入（与 Docker 的 `-e <名>` 同款惯例），不写进镜像层。
|
||||
|
||||
## 2. 内容清单（按现有 Docker 任务镜像能力对齐）
|
||||
|
||||
| 类别 | 内容 | 来源/版本口径 |
|
||||
| --- | --- | --- |
|
||||
| 语言与工具链 | Python 3.12、git、Node.js（Playwright 驱动用） | 与任务镜像同款大版本，构建时钉小版本并记入构建日志 |
|
||||
| 文档/图片工具 | pandoc、poppler-utils（pdftotext）、latexmk、texlive-latex-base、python-docx、openpyxl、python-pptx、PyMuPDF、Pillow | 逐项照抄 `cloud/docker/Dockerfile` 的 apt/pip 清单，不擅自加减 |
|
||||
| agent 引擎 | codex、codex-code-mode-host、codex-resources/bwrap（成套，约 367M，缺 sibling 直接构建失败） | codex 0.164.0-alpha.9（与生产任务镜像一致）；`CODEX_BIN_DIR` 由构建方提供，bin/ 不进仓库 |
|
||||
| 浏览器 | Playwright + Chromium（浏览器自动化 MCP 用） | 与生产 playwright 镜像同款：浏览器装在固定路径、供租户用户可读；本骨架只列清单，浏览器版本在构建时钉死 |
|
||||
| 客机基础 | openssh-server（SSH 通道用）、cloud-init、qemu-guest-agent（可选，宿主信息用） | sshd 默认只允许 key 登录，禁止密码登录 |
|
||||
| 任务代码 | codex-app-server 仓库代码置于 `/app`（host.py 入口） | 版本见 §3，不挂载宿主代码目录（VM 与宿主无共享路径） |
|
||||
|
||||
中文 LaTeX 说明：与 Docker 镜像同款口径——xelatex + CJK
|
||||
字体会再增数百 MB，本清单不默认包含；需要时单独一版镜像，
|
||||
不在本轨悄悄加。
|
||||
|
||||
## 3. 版本化
|
||||
|
||||
镜像 tag 格式（构建时三项缺一即失败，不生成无版本镜像）：
|
||||
|
||||
```
|
||||
chv-guest-<ubuntu 版本>-<codex 版本>-<仓库 commit 短 hash>
|
||||
```
|
||||
|
||||
例如（示例格式，非已建成）：`chv-guest-24.04-0.164.0-alpha.9-aebc623`。
|
||||
|
||||
构建产物清单（构建方必须输出，执行器用它做 preflight 之外的
|
||||
版本核对，v1 执行器暂不校验 tag，见就绪清单）：
|
||||
|
||||
- `vmlinux`（与 Ubuntu 云镜像配套的内核）
|
||||
- `disk.raw`（根文件系统镜像）
|
||||
- `BUILDINFO.json`：`{ubuntu, codex_version, repo_commit, built_at,
|
||||
contents: [清单逐项版本]}`
|
||||
|
||||
## 4. 构建脚本骨架
|
||||
|
||||
见 `build.sh`：只做参数校验、产物路径与版本 tag 计算、以及
|
||||
“需要 KVM 宿主 / virt-customize 才能执行的步骤”占位。脚本在
|
||||
无 KVM 的机器上运行时应在第一步就明确退出（`CHV_GUEST_BUILD
|
||||
needs a KVM host`），绝不假装构建成功。
|
||||
|
||||
## 5. 与执行器的对接
|
||||
|
||||
- 执行器 preflight 只校验内核/镜像文件存在且可读，不校验镜像
|
||||
内容版本——内容版本核对（BUILDINFO）是镜像发布流程的责任，
|
||||
在就绪清单里标为“需镜像构建”项，不在本轨假装已做。
|
||||
- 每任务磁盘：执行器 copy 模式下把 `disk.raw` 复制为每任务
|
||||
磁盘，任务之间文件系统独立；reuse 模式仅单租户实验，生产禁用。
|
||||
- 快照基准：`CHV_SNAPSHOT_URL` 指向的快照必须由本镜像的干净
|
||||
启动状态制成（pause 后 snapshot），预热池用它做干净 VM 来源。
|
||||
Executable
+47
@@ -0,0 +1,47 @@
|
||||
#!/bin/bash
|
||||
# chv-guest 镜像构建脚本骨架（未在无 KVM 机器上实际执行过）。
|
||||
#
|
||||
# 用法（需要带 /dev/kvm 的宿主，见 README）：
|
||||
# CODEX_BIN_DIR=bin UBUNTU_VERSION=24.04 CODEX_VERSION=0.164.0-alpha.9 \
|
||||
# REPO_COMMIT=$(git rev-parse --short HEAD) ./build.sh out/
|
||||
#
|
||||
# 本脚本只做三件事，绝不假装已建成镜像：
|
||||
# 1. 参数校验（缺 CODEX_BIN_DIR 成套二进制、缺版本、缺 /dev/kvm
|
||||
# 即退出，退出码非 0）；
|
||||
# 2. 算出版本 tag（chv-guest-<ubuntu>-<codex>-<commit>）；
|
||||
# 3. 列出仍需在 KVM 宿主上执行的步骤（下载 Ubuntu 云镜像、
|
||||
# virt-customize 装清单软件、拷入 /app 代码、生成 vmlinux /
|
||||
# disk.raw / BUILDINFO.json），每一步在骨架里以 TODO 标记，
|
||||
# 不在本轨内静默跳过或谎称完成。
|
||||
set -eu
|
||||
|
||||
OUT="${1:-out}"
|
||||
: "${CODEX_BIN_DIR:?CODEX_BIN_DIR 未设：需 npm 安装 @openai/codex 后的配对二进制目录}"
|
||||
: "${UBUNTU_VERSION:?UBUNTU_VERSION 未设}"
|
||||
: "${CODEX_VERSION:?CODEX_VERSION 未设}"
|
||||
: "${REPO_COMMIT:?REPO_COMMIT 未设：构建时仓库 commit 短 hash}"
|
||||
|
||||
for bin in codex codex-code-mode-host; do
|
||||
if [ ! -x "$CODEX_BIN_DIR/$bin" ]; then
|
||||
echo "缺少配对二进制：$CODEX_BIN_DIR/$bin，构建失败" >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
if [ ! -d "$CODEX_BIN_DIR/codex-resources" ]; then
|
||||
echo "缺少 codex-resources（bwrap 载体），构建失败" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [ ! -e /dev/kvm ]; then
|
||||
echo "CHV_GUEST_BUILD needs a KVM host: /dev/kvm 不存在" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
TAG="chv-guest-${UBUNTU_VERSION}-${CODEX_VERSION}-${REPO_COMMIT}"
|
||||
mkdir -p "$OUT"
|
||||
echo "tag=$TAG"
|
||||
echo "out=$OUT"
|
||||
echo "TODO(KVM host): 下载 Ubuntu ${UBUNTU_VERSION} 云镜像并转 raw"
|
||||
echo "TODO(KVM host): virt-customize 装 README §2 清单（git/python/pandoc/poppler/latex/codex/playwright/openssh）"
|
||||
echo "TODO(KVM host): 拷入仓库代码到 /app（版本 $REPO_COMMIT），绝不拷入任何 key"
|
||||
echo "TODO(KVM host): 生成 vmlinux + disk.raw + BUILDINFO.json，校验 tag 与清单版本"
|
||||
echo "构建骨架结束：以上 TODO 步骤未执行，镜像未建成（见 README 与生产就绪清单）"
|
||||
@@ -1288,6 +1288,57 @@ interrupted 且终态任务 cancel 返回 400、心跳后 sweep 不扫刚心跳
|
||||
错误文本或返回结果。worker CLI 的 `--executor` 新增 cloud 选项，
|
||||
构造 CloudExecutor；worker 的 run_one 走 dispatch_task，
|
||||
CloudExecutor 的 dispatch_task/run_task 直接委托给 execute。
|
||||
- CloudHypervisorExecutor（生产级 v1，`cloud/sandbox.py`，
|
||||
执行器名 `cloud-hypervisor`，绝不用 `cloud` 缩写）：
|
||||
单任务单 VM：一个任务一个 VM、独立内核/文件系统/网络，
|
||||
任务结束销毁，租户之间无共享路径、无东西向互访；VM 内
|
||||
agent 可有 root，但只能影响本 VM。密钥按现有惯例走运行
|
||||
时环境变量名注入，不进镜像、不进代码、不进任务 prompt
|
||||
落库（SSH argv 里只有 key 文件路径，绝无 key 值）。
|
||||
唤醒是 API 模式：preflight（见下）→ 容量门 → 磁盘配额门
|
||||
→ 每任务磁盘供给（copy 模式复制基础镜像）→ 只以
|
||||
`--api-socket` 起 VMM → 等 `vmm.ping` 通 → `PUT vm.create`
|
||||
（VmConfig）→ `PUT vm.boot` → 等 guest SSH 就绪（默认
|
||||
120 秒）才跑 host.py；超时关 VM 并报分类失败。直接启动
|
||||
命令（`build_vm_command`）只作手动等价写法，执行器不走它。
|
||||
暂停/快照/休眠：`pause_vm`/`resume_vm` 走 ch-remote；
|
||||
`restore_vm` 先 restore 再显式 resume（恢复后默认 paused，
|
||||
不 resume 不算醒）；CHV 无原生 hibernate 端点，`hibernate_vm`
|
||||
用 pause + snapshot + shutdown 等价实现，恢复走 restore。
|
||||
预热池 `WarmPool`：预热条目是一次性干净快照 VM，acquire
|
||||
消耗、release 一律销毁，绝不跨任务复用脏 VM；池空回落
|
||||
冷启动，绝不降级 Docker/Local。关闭与清理：任务结束/
|
||||
失败/取消时 `shutdown_vm`（ch-remote shutdown + 杀进程 +
|
||||
删 socket），`cleanup_stale_vms` 处理已死/已关/socket 残留
|
||||
三种情况。审计：VM 的 create/boot/ready/pause/resume/
|
||||
snapshot/restore/shutdown/destroy/failed/recycled 均经
|
||||
audit_callback 发出，worker 接线后转 AuthStore 审计；失败
|
||||
带 kind 分类（kvm_missing/image_missing/guest_not_ready/
|
||||
api_timeout/host_capacity_exceeded/disk_quota_exceeded 等），
|
||||
不笼统 failed。资源与成本：每 VM 的 CPU/内存/磁盘配额、
|
||||
超时、最大轮次（goal 上限 100）复用 worker 现有 quota 语义，
|
||||
不另起一套计费；空闲休眠省成本；autoscaler 按 executor 过
|
||||
滤队列深度并透传 `--executor` 给拉起的 worker。网络与治理：
|
||||
VM 出口默认必须过 CHV_EGRESS_PROXY 白名单代理（未设且未
|
||||
显式允许直连时 preflight 直接失败）；审批门仍走任务的
|
||||
approval 与 server 的 merge_gate，VM 内 agent 不能自批。
|
||||
配置（环境变量，见该类 docstring 全表）：CHV_BINARY、
|
||||
CHV_KERNEL、CHV_ROOTFS（或 CHV_IMAGE 别名）、CHV_CPU、
|
||||
CHV_MEM、CHV_STATE_DIR、CHV_SSH_KEY/USER/PORT、CHV_CMDLINE、
|
||||
CHV_NET（直接命令用）/CHV_NET_JSON（API 用）、CHV_REMOTE、
|
||||
CHV_MAX_VMS、CHV_DISK_MODE（copy 默认/reuse 仅实验）、
|
||||
CHV_DISK_GB、CHV_EGRESS_PROXY、CHV_TASK_TIMEOUT、
|
||||
CHV_SNAPSHOT_URL、CHV_WARM_POOL_SIZE、CHV_API_TIMEOUT、
|
||||
CHV_READY_TIMEOUT/INTERVAL。默认关闭：server 默认执行器仍
|
||||
是 docker，worker 默认 local，CHV 只在任务创建时显式选
|
||||
`executor=cloud-hypervisor` 且有 CHV worker 认领时才跑；
|
||||
任务内 LLM 绝不能发现/调用它（调度层选择，只在创建层可见：
|
||||
POST /tasks 可选值、本文档、控制台选项，不新增任务内 MCP/
|
||||
工具）。真实验证边界：未做真实 VM 启动验证——生产主机
|
||||
本身是 CVM、无 /dev/kvm、无嵌套虚拟化，本 VM 容器内同样
|
||||
无 /dev/kvm；真 VM、快照恢复、压测、安全审计均在
|
||||
`docs/chv-production-readiness.md` 里标为待办，绝不声称
|
||||
已生产可用。客机镜像内容清单与构建骨架见 `chv-guest/`。
|
||||
- 自动 PR（默认关闭，显式触发，`POST /tasks/{id}/pr`，需项目
|
||||
maintainer 以上权限 approve_merge）：仅当任务状态为 merged 时
|
||||
可调（其它状态 409；任务不存在 404；权限不足 403）。逻辑按顺序
|
||||
|
||||
@@ -304,6 +304,14 @@ class Autoscaler:
|
||||
if not any(a == "--host-name" or a.startswith("--host-name=")
|
||||
for a in self.worker_args):
|
||||
cmd.extend(["--host-name", socket.gethostname()])
|
||||
# 执行器透传：scaler 的 --executor 只算该类型队列深度，
|
||||
# 不透传时拉起的 worker 默认 local，永远认领不到该类型
|
||||
# 任务（cloud-hypervisor 池就是这样被饿死的）。worker_args
|
||||
# 里已显式给 --executor 时以显式为准，不覆盖。
|
||||
if self.executor and not any(
|
||||
a == "--executor" or a.startswith("--executor=")
|
||||
for a in self.worker_args):
|
||||
cmd.extend(["--executor", self.executor])
|
||||
return cmd
|
||||
|
||||
def _start_worker(self, worker_id):
|
||||
@@ -553,6 +561,8 @@ def main(argv=None):
|
||||
help="各 worker 子进程的日志目录")
|
||||
parser.add_argument("--executor", default=None,
|
||||
help="只按该执行器类型的就绪任务算深度；"
|
||||
"同时透传给拉起的 worker（worker_args 里"
|
||||
"已显式给 --executor 时以显式为准）；"
|
||||
"不给时按全部任务算（可能轻微多扩，"
|
||||
"受 max-workers 封顶）")
|
||||
parser.add_argument("worker_args", nargs=argparse.REMAINDER,
|
||||
|
||||
+1289
File diff suppressed because it is too large.
Load diff
+22
-6
@@ -494,6 +494,9 @@ def _task_worktree_for_files(server, task):
|
||||
"""
|
||||
if task.get("executor") == "cloud":
|
||||
return None, (409, "cloud task worktrees are not on this host")
|
||||
if task.get("executor") == "cloud-hypervisor":
|
||||
return None, (409, "cloud-hypervisor task worktrees are "
|
||||
"inside the VM, not on this host")
|
||||
worktree_host = task.get("worktree_host")
|
||||
if worktree_host and worktree_host != socket.gethostname():
|
||||
return None, (409, "task worktree is on host %s;"
|
||||
@@ -1550,7 +1553,7 @@ def _default_executor():
|
||||
raw = raw.strip()
|
||||
if raw == "":
|
||||
return None
|
||||
if raw in ("local", "docker", "cloud"):
|
||||
if raw in ("local", "docker", "cloud", "cloud-hypervisor"):
|
||||
return raw
|
||||
return "docker"
|
||||
|
||||
@@ -2492,9 +2495,11 @@ class CloudControlPlaneHandler(BaseHTTPRequestHandler):
|
||||
self._send_json(400, {"error": "name too long"})
|
||||
return
|
||||
executor = body.get("executor")
|
||||
if executor is not None and executor not in ("local", "docker"):
|
||||
if executor is not None and executor not in (
|
||||
"local", "docker", "cloud-hypervisor"):
|
||||
self._send_json(400, {
|
||||
"error": "executor must be local or docker"})
|
||||
"error": "executor must be local, docker or "
|
||||
"cloud-hypervisor"})
|
||||
return
|
||||
try:
|
||||
branch = _validate_env_branch(body.get("branch"))
|
||||
@@ -4248,12 +4253,15 @@ class CloudControlPlaneHandler(BaseHTTPRequestHandler):
|
||||
self._send_json(400, {"error": "priority must be an integer"})
|
||||
return
|
||||
# executor 可选：None 表示任意 worker 可认领；指定时只能是
|
||||
# local/docker/cloud，非法值 400 且不建任务。
|
||||
# local/docker/cloud/cloud-hypervisor，非法值 400 且不建任务。
|
||||
# cloud-hypervisor 是创建层可见的调度选择（任务内 LLM
|
||||
# 绝不能自选它，见 sandbox.py 该类 docstring）。
|
||||
executor = body.get("executor")
|
||||
if executor is not None and executor not in (
|
||||
"local", "docker", "cloud"):
|
||||
"local", "docker", "cloud", "cloud-hypervisor"):
|
||||
self._send_json(400, {
|
||||
"error": "executor must be local, docker or cloud"})
|
||||
"error": "executor must be local, docker, cloud or "
|
||||
"cloud-hypervisor"})
|
||||
return
|
||||
# 任务级选项：model 不做白名单（中转目录会变，下拉只是
|
||||
# 建议值）；effort 必须是 codex 的 effort 档位；approval
|
||||
@@ -4481,6 +4489,10 @@ class CloudControlPlaneHandler(BaseHTTPRequestHandler):
|
||||
self._send_json(400, {
|
||||
"error": "附件不支持 cloud 执行器"})
|
||||
return
|
||||
if executor == "cloud-hypervisor":
|
||||
self._send_json(400, {
|
||||
"error": "附件不支持 cloud-hypervisor 执行器"})
|
||||
return
|
||||
conn0 = self._db()
|
||||
try:
|
||||
brow = conn0.execute(
|
||||
@@ -5893,6 +5905,10 @@ class CloudControlPlaneHandler(BaseHTTPRequestHandler):
|
||||
if executor == "cloud":
|
||||
return (409, "thread ops are local-executor only;"
|
||||
" cloud task threads are not on this host")
|
||||
if executor == "cloud-hypervisor":
|
||||
return (409, "thread ops are local-executor only;"
|
||||
" cloud-hypervisor task threads live inside"
|
||||
" the VM, not on this host")
|
||||
worktree_host = task.get("worktree_host")
|
||||
if worktree_host and worktree_host != socket.gethostname():
|
||||
return (409, "task worktree is on host %s;"
|
||||
|
||||
@@ -0,0 +1,665 @@
|
||||
"""Cloud Hypervisor 执行器 v1 + 生产级唤醒/生命周期测试。
|
||||
|
||||
只用标准库与 mock runner/launcher/api_client，绝不启动真实
|
||||
VM：当前环境没有可跑真 VM 的宿主（生产主机是 CVM、无
|
||||
/dev/kvm、无嵌套虚拟化；本 VM 容器内同样无 /dev/kvm），
|
||||
所以本文件验证的是：preflight 分类、API 唤醒顺序、VmConfig、
|
||||
SSH 就绪、暂停/快照/休眠/关闭/清理、预热池、单任务单 VM
|
||||
隔离、容量门、磁盘门、worker/server/autoscaler 接线。
|
||||
真实 VM 启动、快照恢复、压测一律不在本文件声称已验证。
|
||||
"""
|
||||
|
||||
import http.client
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
try:
|
||||
from cloud.sandbox import (
|
||||
ChvFailure,
|
||||
CloudHypervisorExecutor,
|
||||
WarmPool,
|
||||
_parse_memory_bytes,
|
||||
)
|
||||
except ImportError: # discover -s cloud/tests 时把项目根目录补进 sys.path
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
from cloud.sandbox import (
|
||||
ChvFailure,
|
||||
CloudHypervisorExecutor,
|
||||
WarmPool,
|
||||
_parse_memory_bytes,
|
||||
)
|
||||
|
||||
|
||||
class FakeResult:
|
||||
def __init__(self, returncode=0, stdout="", stderr=""):
|
||||
self.returncode = returncode
|
||||
self.stdout = stdout
|
||||
self.stderr = stderr
|
||||
|
||||
|
||||
class FakeProc:
|
||||
"""假 VMM 进程：poll() 返回 None=活着，数字=已死。"""
|
||||
|
||||
def __init__(self, alive=True):
|
||||
self._alive = alive
|
||||
self.terminated = False
|
||||
self.killed = False
|
||||
|
||||
def poll(self):
|
||||
return None if self._alive else 1
|
||||
|
||||
def terminate(self):
|
||||
self.terminated = True
|
||||
self._alive = False
|
||||
|
||||
def wait(self, timeout=None):
|
||||
return 0
|
||||
|
||||
def kill(self):
|
||||
self.killed = True
|
||||
self._alive = False
|
||||
|
||||
|
||||
def _write(path, content=b"x"):
|
||||
with open(path, "wb") as fh:
|
||||
fh.write(content)
|
||||
os.chmod(path, 0o755)
|
||||
return path
|
||||
|
||||
|
||||
class ChvCase(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.dir = self.tmp.name
|
||||
self.binary = _write(os.path.join(self.dir, "cloud-hypervisor"))
|
||||
self.ch_remote = _write(os.path.join(self.dir, "ch-remote"))
|
||||
self.kernel = _write(os.path.join(self.dir, "vmlinux"))
|
||||
self.rootfs = _write(os.path.join(self.dir, "disk.raw"),
|
||||
b"R" * 100)
|
||||
self.ssh_key = _write(os.path.join(self.dir, "id_chv"))
|
||||
self.kvm = _write(os.path.join(self.dir, "kvm"))
|
||||
self.cpuinfo = os.path.join(self.dir, "cpuinfo")
|
||||
with open(self.cpuinfo, "w") as fh:
|
||||
fh.write("flags : fpu vmx sse\n")
|
||||
self.state_dir = os.path.join(self.dir, "state")
|
||||
self.calls = [] # runner 调用记录
|
||||
self.api_calls = [] # api_client 调用记录
|
||||
self.launches = [] # launcher 调用记录
|
||||
self.audit = [] # audit 事件记录
|
||||
|
||||
def tearDown(self):
|
||||
self.tmp.cleanup()
|
||||
|
||||
def make_exec(self, runner=None, api_client=None, launcher=None,
|
||||
env_extra=None, **overrides):
|
||||
env = {"CHV_EGRESS_PROXY": "http://10.0.0.1:8888"}
|
||||
if env_extra:
|
||||
env.update(env_extra)
|
||||
if runner is None:
|
||||
runner = self._default_runner
|
||||
if api_client is None:
|
||||
api_client = self._default_api
|
||||
if launcher is None:
|
||||
launcher = self._default_launcher
|
||||
kwargs = dict(
|
||||
binary=self.binary, ch_remote=self.ch_remote,
|
||||
kernel=self.kernel, rootfs=self.rootfs,
|
||||
ssh_key=self.ssh_key, kvm_path=self.kvm,
|
||||
cpuinfo_path=self.cpuinfo, state_dir=self.state_dir,
|
||||
runner=runner, launcher=launcher, api_client=api_client,
|
||||
sleeper=lambda s: None, env=env)
|
||||
kwargs.update(overrides)
|
||||
ex = CloudHypervisorExecutor(**kwargs)
|
||||
ex.audit_callback = lambda e, t, d, k=None: self.audit.append(
|
||||
(e, t.get("id") if isinstance(t, dict) else t, d, k))
|
||||
return ex
|
||||
|
||||
def _default_runner(self, cmd, timeout=None):
|
||||
self.calls.append(list(cmd))
|
||||
if cmd[0] == "ssh":
|
||||
if cmd[-1] == "true":
|
||||
return FakeResult(0, "", "")
|
||||
if "dispatch" in cmd:
|
||||
return FakeResult(0, json.dumps(
|
||||
{"status": "completed", "text": "ok-dispatch"}), "")
|
||||
if "goal" in cmd or "goal-resume" in cmd:
|
||||
return FakeResult(0, "goal met after rounds\n", "")
|
||||
if "status" in cmd:
|
||||
return FakeResult(0, json.dumps(
|
||||
{"status": "completed", "text": "ok-status"}), "")
|
||||
return FakeResult(0, "", "")
|
||||
return FakeResult(0, "", "")
|
||||
|
||||
def _default_api(self, socket_path, method, path, payload=None):
|
||||
self.api_calls.append((method, path, payload))
|
||||
return (200, {})
|
||||
|
||||
def _default_launcher(self, cmd):
|
||||
self.launches.append(list(cmd))
|
||||
return FakeProc(alive=True)
|
||||
|
||||
def _task(self, task_id="abc123"):
|
||||
return {"id": task_id, "prompt": "做一个加法函数",
|
||||
"project_id": "proj1"}
|
||||
|
||||
|
||||
# -- preflight ---------------------------------------------------------
|
||||
class PreflightTest(ChvCase):
|
||||
def test_all_present_passes(self):
|
||||
self.assertTrue(self.make_exec().preflight())
|
||||
|
||||
def test_missing_binary_reports_it(self):
|
||||
ex = self.make_exec(binary="/tmp/no-such-chv-binary")
|
||||
with self.assertRaises(ChvFailure) as ctx:
|
||||
ex.preflight()
|
||||
self.assertEqual(ctx.exception.kind, "binary_missing")
|
||||
self.assertIn("cloud-hypervisor", str(ctx.exception))
|
||||
|
||||
def test_missing_ch_remote_reports_it(self):
|
||||
ex = self.make_exec(ch_remote="/tmp/no-such-ch-remote")
|
||||
with self.assertRaises(ChvFailure) as ctx:
|
||||
ex.preflight()
|
||||
self.assertEqual(ctx.exception.kind, "binary_missing")
|
||||
self.assertIn("ch-remote", str(ctx.exception))
|
||||
|
||||
def test_missing_kvm_reports_it(self):
|
||||
ex = self.make_exec(kvm_path="/tmp/no-such-kvm")
|
||||
with self.assertRaises(ChvFailure) as ctx:
|
||||
ex.preflight()
|
||||
self.assertEqual(ctx.exception.kind, "kvm_missing")
|
||||
self.assertIn("/tmp/no-such-kvm", str(ctx.exception))
|
||||
|
||||
def test_missing_cpu_flag_reports_it(self):
|
||||
with open(self.cpuinfo, "w") as fh:
|
||||
fh.write("flags : fpu sse\n")
|
||||
with self.assertRaises(ChvFailure) as ctx:
|
||||
self.make_exec().preflight()
|
||||
self.assertEqual(ctx.exception.kind, "cpu_flag_missing")
|
||||
self.assertIn("vmx", str(ctx.exception))
|
||||
|
||||
def test_missing_kernel_reports_it(self):
|
||||
ex = self.make_exec(kernel="/tmp/no-such-vmlinux")
|
||||
with self.assertRaises(ChvFailure) as ctx:
|
||||
ex.preflight()
|
||||
self.assertEqual(ctx.exception.kind, "kernel_missing")
|
||||
|
||||
def test_missing_image_reports_it(self):
|
||||
ex = self.make_exec(rootfs="/tmp/no-such-disk.raw")
|
||||
with self.assertRaises(ChvFailure) as ctx:
|
||||
ex.preflight()
|
||||
self.assertEqual(ctx.exception.kind, "image_missing")
|
||||
|
||||
def test_missing_ssh_key_reports_it(self):
|
||||
ex = self.make_exec(ssh_key="/tmp/no-such-key")
|
||||
with self.assertRaises(ChvFailure) as ctx:
|
||||
ex.preflight()
|
||||
self.assertEqual(ctx.exception.kind, "ssh_key_missing")
|
||||
|
||||
def test_missing_egress_is_fail_closed(self):
|
||||
ex = self.make_exec(env_extra={"CHV_EGRESS_PROXY": ""})
|
||||
# env_extra 空值会被 __init__ 的 or 链忽略，故改用显式 env：
|
||||
ex.egress_proxy = None
|
||||
ex.allow_direct_egress = False
|
||||
with self.assertRaises(ChvFailure) as ctx:
|
||||
ex.preflight()
|
||||
self.assertEqual(ctx.exception.kind, "egress_not_configured")
|
||||
|
||||
def test_multiple_missing_lists_all(self):
|
||||
ex = self.make_exec(binary="/tmp/no-chv", kernel="/tmp/no-kern")
|
||||
with self.assertRaises(ChvFailure) as ctx:
|
||||
ex.preflight()
|
||||
text = str(ctx.exception)
|
||||
self.assertIn("cloud-hypervisor", text)
|
||||
self.assertIn("内核", text)
|
||||
|
||||
def test_dispatch_missing_dependency_does_not_call_runner(self):
|
||||
ex = self.make_exec(binary="/tmp/no-chv")
|
||||
with self.assertRaises(ChvFailure):
|
||||
ex.dispatch_task(self._task())
|
||||
self.assertEqual(self.calls, [])
|
||||
|
||||
|
||||
# -- 配置与命令构造 -----------------------------------------------------
|
||||
class ConfigCommandTest(ChvCase):
|
||||
def test_vm_config_shape_and_memory_bytes(self):
|
||||
ex = self.make_exec()
|
||||
cfg = ex.build_vm_config("/tmp/x/disk.raw")
|
||||
self.assertEqual(cfg["cpus"], {"boot_vcpus": 2,
|
||||
"max_vcpus": 2})
|
||||
self.assertEqual(cfg["memory"]["size"], 2 * 1024 ** 3)
|
||||
self.assertEqual(cfg["kernel"]["path"], self.kernel)
|
||||
self.assertEqual(cfg["disks"], [{"path": "/tmp/x/disk.raw"}])
|
||||
self.assertEqual(cfg["cmdline"]["args"],
|
||||
"console=hvc0 root=/dev/vda1 rw")
|
||||
|
||||
def test_memory_parsing(self):
|
||||
self.assertEqual(_parse_memory_bytes("512M"), 512 * 1024 ** 2)
|
||||
self.assertEqual(_parse_memory_bytes("1024"), 1024)
|
||||
self.assertEqual(_parse_memory_bytes("1K"), 1024)
|
||||
with self.assertRaises(ValueError):
|
||||
_parse_memory_bytes("abc")
|
||||
|
||||
def test_rootfs_image_alias(self):
|
||||
ex = self.make_exec(rootfs=None, env_extra={})
|
||||
# rootfs=None 时 __init__ 仍从 env 取不到（env 无 CHV_*），
|
||||
# 故用 env 含 CHV_IMAGE 的构造验证别名：
|
||||
env = {"CHV_EGRESS_PROXY": "http://10.0.0.1:8888",
|
||||
"CHV_IMAGE": self.rootfs}
|
||||
ex2 = CloudHypervisorExecutor(
|
||||
binary=self.binary, ch_remote=self.ch_remote,
|
||||
kernel=self.kernel, ssh_key=self.ssh_key,
|
||||
kvm_path=self.kvm, cpuinfo_path=self.cpuinfo,
|
||||
state_dir=self.state_dir, env=env,
|
||||
sleeper=lambda s: None)
|
||||
self.assertEqual(ex2.rootfs, self.rootfs)
|
||||
|
||||
def test_manual_vm_command_shape(self):
|
||||
ex = self.make_exec()
|
||||
cmd = ex.build_vm_command(self._task())
|
||||
self.assertEqual(cmd[0], self.binary)
|
||||
self.assertIn("--api-socket", cmd)
|
||||
self.assertIn("--cpus", cmd)
|
||||
self.assertIn("boot=2", cmd)
|
||||
self.assertIn("size=2G", cmd)
|
||||
self.assertIn("--kernel", cmd)
|
||||
self.assertIn("--disk", cmd)
|
||||
|
||||
def test_ssh_command_has_key_path_not_secret_value(self):
|
||||
ex = self.make_exec()
|
||||
cmd = ex.build_ssh_command(self._task(), ["true"])
|
||||
joined = " ".join(cmd)
|
||||
self.assertIn(self.ssh_key, joined)
|
||||
self.assertIn("BatchMode=yes", joined)
|
||||
self.assertNotIn("SECRET", joined)
|
||||
# egress 代理地址在 env 前缀里（不是密钥），且只在代理已设时出现：
|
||||
self.assertIn("HTTPS_PROXY=http://10.0.0.1:8888", joined)
|
||||
|
||||
def test_isolation_names_and_disks_differ(self):
|
||||
ex = self.make_exec()
|
||||
a, b = {"id": "task-a"}, {"id": "task-b"}
|
||||
self.assertNotEqual(ex.vm_name(a), ex.vm_name(b))
|
||||
self.assertNotEqual(ex.api_socket(a), ex.api_socket(b))
|
||||
self.assertNotEqual(ex._provision_task_disk(a),
|
||||
ex._provision_task_disk(b))
|
||||
# copy 模式：每任务磁盘是基础镜像的真实副本，内容一致、
|
||||
# 路径不同（两个任务绝不写同一文件）：
|
||||
with open(ex._provision_task_disk(a), "rb") as fh:
|
||||
self.assertEqual(fh.read(), b"R" * 100)
|
||||
|
||||
def test_disk_quota_gate(self):
|
||||
ex = self.make_exec(env_extra={"CHV_DISK_GB": "0.00000001"})
|
||||
with self.assertRaises(ChvFailure) as ctx:
|
||||
ex.wake_vm(self._task())
|
||||
self.assertEqual(ctx.exception.kind, "disk_quota_exceeded")
|
||||
|
||||
def test_capacity_gate_rejects_not_fallback(self):
|
||||
ex = self.make_exec(env_extra={"CHV_MAX_VMS": "1"})
|
||||
ex._vms["chv-other"] = FakeProc(alive=True)
|
||||
with self.assertRaises(ChvFailure) as ctx:
|
||||
ex.wake_vm(self._task())
|
||||
self.assertEqual(ctx.exception.kind, "host_capacity_exceeded")
|
||||
self.assertEqual(self.launches, [])
|
||||
|
||||
|
||||
# -- API 唤醒 -----------------------------------------------------------
|
||||
class WakeApiTest(ChvCase):
|
||||
def test_wake_uses_api_mode_not_direct_command(self):
|
||||
ex = self.make_exec()
|
||||
out = ex.wake_vm(self._task())
|
||||
self.assertEqual(out["state"], "running")
|
||||
self.assertTrue(out["ready"])
|
||||
self.assertEqual(out["mode"], "cold")
|
||||
# launcher 只带 --api-socket，不带内核/磁盘（API 模式证明）：
|
||||
self.assertEqual(len(self.launches), 1)
|
||||
launch = self.launches[0]
|
||||
self.assertIn("--api-socket", launch)
|
||||
self.assertNotIn("--kernel", launch)
|
||||
self.assertNotIn("--disk", launch)
|
||||
# API 调用顺序：ping → create → boot：
|
||||
kinds = [(m, p) for m, p, _ in self.api_calls]
|
||||
self.assertEqual(kinds[0], ("GET", "/vmm.ping"))
|
||||
self.assertEqual(kinds[1], ("PUT", "/vm.create"))
|
||||
self.assertEqual(kinds[2], ("PUT", "/vm.boot"))
|
||||
create_payload = self.api_calls[1][2]
|
||||
self.assertEqual(create_payload["kernel"]["path"], self.kernel)
|
||||
self.assertTrue(create_payload["disks"][0]["path"].endswith(
|
||||
"disk.raw"))
|
||||
events = [e for e, _, _, _ in self.audit]
|
||||
self.assertIn("vm_create", events)
|
||||
self.assertIn("vm_boot", events)
|
||||
self.assertIn("vm_ready", events)
|
||||
|
||||
def test_wake_api_timeout_kills_proc(self):
|
||||
def never_ok(socket_path, method, path, payload=None):
|
||||
self.api_calls.append((method, path, payload))
|
||||
return (500, {})
|
||||
ex = self.make_exec(api_client=never_ok, api_timeout=3)
|
||||
proc_holder = []
|
||||
|
||||
def launcher(cmd):
|
||||
proc = FakeProc(alive=True)
|
||||
proc_holder.append(proc)
|
||||
self.launches.append(list(cmd))
|
||||
return proc
|
||||
ex.launcher = launcher
|
||||
with self.assertRaises(ChvFailure) as ctx:
|
||||
ex.wake_vm(self._task())
|
||||
self.assertEqual(ctx.exception.kind, "api_timeout")
|
||||
self.assertTrue(proc_holder[0].terminated)
|
||||
self.assertNotIn("chv-abc123", ex._vms)
|
||||
|
||||
def test_guest_ready_timeout_shuts_down(self):
|
||||
def ssh_never(cmd, timeout=None):
|
||||
self.calls.append(list(cmd))
|
||||
if cmd[0] == "ssh":
|
||||
return FakeResult(1, "", "")
|
||||
return FakeResult(0, "", "")
|
||||
ex = self.make_exec(runner=ssh_never, ready_timeout=10,
|
||||
ready_interval=5)
|
||||
with self.assertRaises(ChvFailure) as ctx:
|
||||
ex.wake_vm(self._task())
|
||||
self.assertEqual(ctx.exception.kind, "guest_not_ready")
|
||||
# 超时后关 VM：runner 里有 ch-remote shutdown：
|
||||
self.assertTrue(any("shutdown" in c for c in self.calls))
|
||||
|
||||
|
||||
# -- 暂停 / 快照 / 休眠 / 关闭 / 清理 -----------------------------------
|
||||
class LifecycleTest(ChvCase):
|
||||
def test_pause_resume(self):
|
||||
ex = self.make_exec()
|
||||
self.assertTrue(ex.pause_vm(self._task()))
|
||||
self.assertTrue(ex.resume_vm(self._task()))
|
||||
kinds = [c[-1] for c in self.calls if c[0] == self.ch_remote]
|
||||
self.assertEqual(kinds, ["pause", "resume"])
|
||||
|
||||
def test_restore_then_explicit_resume_order(self):
|
||||
ex = self.make_exec()
|
||||
out = ex.restore_vm(self._task(), "file:///snap/1")
|
||||
self.assertEqual(out["mode"], "restored")
|
||||
self.assertTrue(out["ready"])
|
||||
kinds = [c[-1] if c[-1] != "restore" else c[-1]
|
||||
for c in self.calls if c[0] == self.ch_remote]
|
||||
# ch-remote 调用里 restore 必须在 resume 之前：
|
||||
flat = []
|
||||
for c in self.calls:
|
||||
if c[0] == self.ch_remote:
|
||||
flat.append(c[-1] if c[-1].startswith("source_url")
|
||||
else c[-1])
|
||||
self.assertIn("resume", flat)
|
||||
restore_idx = next(i for i, c in enumerate(self.calls)
|
||||
if c[0] == self.ch_remote
|
||||
and any(a.startswith("source_url")
|
||||
for a in c))
|
||||
resume_idx = next(i for i, c in enumerate(self.calls)
|
||||
if c[0] == self.ch_remote and c[-1] == "resume")
|
||||
self.assertLess(restore_idx, resume_idx)
|
||||
|
||||
def test_hibernate_is_pause_snapshot_shutdown(self):
|
||||
ex = self.make_exec()
|
||||
out = ex.hibernate_vm(self._task(), "file:///snap/h1")
|
||||
self.assertEqual(out["state"], "hibernated")
|
||||
self.assertEqual(out["snapshot"], "file:///snap/h1")
|
||||
kinds = [c[-1] for c in self.calls if c[0] == self.ch_remote]
|
||||
self.assertEqual(kinds[0], "pause")
|
||||
self.assertTrue(any(a.startswith("source_url")
|
||||
for c in self.calls for a in c))
|
||||
self.assertIn("shutdown", kinds)
|
||||
# 从休眠唤醒 = restore + resume：
|
||||
ex2 = self.make_exec()
|
||||
out2 = ex2.wake_from_hibernate(self._task(), "file:///snap/h1")
|
||||
self.assertEqual(out2["mode"], "restored")
|
||||
|
||||
def test_stop_dead_proc(self):
|
||||
ex = self.make_exec()
|
||||
proc = FakeProc(alive=False)
|
||||
ex._vms["chv-abc123"] = proc
|
||||
sock = os.path.join(self.state_dir, "chv-abc123.sock")
|
||||
os.makedirs(self.state_dir, exist_ok=True)
|
||||
with open(sock, "w") as _fh:
|
||||
_fh.write("")
|
||||
self.assertTrue(ex.stop_vm(self._task()))
|
||||
self.assertFalse(os.path.exists(sock))
|
||||
self.assertNotIn("chv-abc123", ex._vms)
|
||||
|
||||
def test_stop_already_closed(self):
|
||||
ex = self.make_exec()
|
||||
self.assertFalse(ex.stop_vm(self._task()))
|
||||
# 已关仍尽力调过一次 ch-remote shutdown：
|
||||
self.assertTrue(any("shutdown" in c for c in self.calls))
|
||||
|
||||
def test_stop_socket_residual(self):
|
||||
ex = self.make_exec()
|
||||
sock = os.path.join(self.state_dir, "chv-abc123.sock")
|
||||
os.makedirs(self.state_dir, exist_ok=True)
|
||||
with open(sock, "w") as _fh:
|
||||
_fh.write("")
|
||||
self.assertTrue(ex.stop_vm(self._task()))
|
||||
self.assertFalse(os.path.exists(sock))
|
||||
|
||||
def test_cleanup_skips_live_and_non_chv(self):
|
||||
ex = self.make_exec()
|
||||
os.makedirs(self.state_dir, exist_ok=True)
|
||||
live = os.path.join(self.state_dir, "chv-live.sock")
|
||||
with open(live, "w") as _fh:
|
||||
_fh.write("")
|
||||
ex._vms["chv-live"] = FakeProc(alive=True)
|
||||
dead = os.path.join(self.state_dir, "chv-dead.sock")
|
||||
with open(dead, "w") as _fh:
|
||||
_fh.write("")
|
||||
other = os.path.join(self.state_dir, "other.sock")
|
||||
with open(other, "w") as _fh:
|
||||
_fh.write("")
|
||||
cleaned = ex.cleanup_stale_vms()
|
||||
self.assertEqual(cleaned, ["chv-dead"])
|
||||
self.assertTrue(os.path.exists(live))
|
||||
self.assertTrue(os.path.exists(other))
|
||||
self.assertFalse(os.path.exists(dead))
|
||||
|
||||
def test_cleanup_missing_state_dir(self):
|
||||
ex = self.make_exec()
|
||||
self.assertEqual(ex.cleanup_stale_vms(), [])
|
||||
|
||||
def test_recycle_preserves_checkpoints_not_temp(self):
|
||||
ex = self.make_exec()
|
||||
out = ex.recycle_vm(self._task())
|
||||
self.assertEqual(out["state"], "recycled")
|
||||
self.assertIn("task_checkpoints", out["preserved"])
|
||||
self.assertIn("task_attempts", out["preserved"])
|
||||
self.assertIn("guest /tmp", out["lost"])
|
||||
|
||||
|
||||
# -- 预热池 --------------------------------------------------------------
|
||||
class WarmPoolTest(ChvCase):
|
||||
def test_prewarm_acquire_release(self):
|
||||
ex = self.make_exec()
|
||||
pool = WarmPool(ex, size=2, snapshot_url="file:///snap/base")
|
||||
added = pool.prewarm()
|
||||
self.assertEqual(len(added), 2)
|
||||
self.assertEqual(pool.warm_count(), 2)
|
||||
first = pool.acquire(self._task("t1"))
|
||||
self.assertEqual(first["mode"], "warm")
|
||||
second = pool.acquire(self._task("t2"))
|
||||
self.assertEqual(second["mode"], "warm")
|
||||
third = pool.acquire(self._task("t3"))
|
||||
self.assertEqual(third["mode"], "cold")
|
||||
# release 一律销毁，不放回池：
|
||||
self.assertTrue(pool.release(self._task("t1"),
|
||||
pool_vm=added[0]))
|
||||
self.assertEqual(pool.warm_count(), 0)
|
||||
|
||||
def test_prewarm_without_snapshot_is_empty(self):
|
||||
ex = self.make_exec()
|
||||
pool = WarmPool(ex, size=2, snapshot_url=None)
|
||||
self.assertEqual(pool.prewarm(), [])
|
||||
self.assertEqual(pool.acquire(self._task())["mode"], "cold")
|
||||
|
||||
|
||||
# -- dispatch / goal（wake → 跑 → finally 关） ---------------------------
|
||||
class DispatchTest(ChvCase):
|
||||
def test_dispatch_wakes_runs_shuts_down(self):
|
||||
ex = self.make_exec()
|
||||
out = ex.dispatch_task(self._task())
|
||||
self.assertEqual(out["executor"], "cloud-hypervisor")
|
||||
self.assertEqual(out["status"], "completed")
|
||||
self.assertIn("ok-dispatch", out["stdout"])
|
||||
self.assertEqual(out["vm"], "chv-abc123")
|
||||
# finally 关 VM：ch-remote shutdown 在 dispatch 之后出现：
|
||||
shutdown_idx = next(i for i, c in enumerate(self.calls)
|
||||
if c[0] == self.ch_remote
|
||||
and c[-1] == "shutdown")
|
||||
ssh_idx = next(i for i, c in enumerate(self.calls)
|
||||
if c[0] == "ssh" and "dispatch" in c)
|
||||
self.assertLess(ssh_idx, shutdown_idx)
|
||||
self.assertNotIn("chv-abc123", ex._vms)
|
||||
|
||||
def test_goal_met(self):
|
||||
ex = self.make_exec()
|
||||
out = ex.goal_task({"id": "g1", "prompt": "做目标",
|
||||
"goal_mode": True})
|
||||
self.assertEqual(out["status"], "completed")
|
||||
self.assertTrue(out["goal_met"])
|
||||
|
||||
def test_goal_resume_breakpoint(self):
|
||||
ex = self.make_exec()
|
||||
out = ex.goal_resume_breakpoint_task({"id": "g1",
|
||||
"prompt": "续跑"})
|
||||
self.assertEqual(out["status"], "completed")
|
||||
self.assertTrue(out["goal_met"])
|
||||
|
||||
def test_execute_is_ssh_status_only(self):
|
||||
ex = self.make_exec()
|
||||
out = ex.execute(self._task())
|
||||
self.assertEqual(out["executor"], "cloud-hypervisor")
|
||||
self.assertIn("ok-status", out["stdout"])
|
||||
# execute 不 wake（launcher 未调用）：
|
||||
self.assertEqual(self.launches, [])
|
||||
|
||||
|
||||
# -- worker / autoscaler / server 接线 -----------------------------------
|
||||
class WiringTest(ChvCase):
|
||||
def test_worker_executor_name_and_cli_builder(self):
|
||||
from cloud import worker as worker_mod
|
||||
from cloud.sandbox import CloudHypervisorExecutor as CHV
|
||||
ex = self.make_exec()
|
||||
w = worker_mod.Worker.__new__(worker_mod.Worker)
|
||||
w.executor = ex
|
||||
self.assertEqual(w._executor_name(), "cloud-hypervisor")
|
||||
|
||||
class Args:
|
||||
executor = "cloud-hypervisor"
|
||||
built = worker_mod._build_cli_executor(Args(), None)
|
||||
self.assertIsInstance(built, CHV)
|
||||
|
||||
def test_autoscaler_passthrough(self):
|
||||
from cloud.autoscaler import Autoscaler
|
||||
scaler = Autoscaler("/tmp/no.db", [], executor="cloud-hypervisor")
|
||||
cmd = scaler._worker_command("w1")
|
||||
self.assertIn("--executor", cmd)
|
||||
self.assertEqual(cmd[cmd.index("--executor") + 1],
|
||||
"cloud-hypervisor")
|
||||
scaler2 = Autoscaler("/tmp/no.db",
|
||||
["--executor", "docker"],
|
||||
executor="cloud-hypervisor")
|
||||
cmd2 = scaler2._worker_command("w1")
|
||||
self.assertEqual(cmd2.count("--executor"), 1)
|
||||
self.assertEqual(cmd2[cmd2.index("--executor") + 1], "docker")
|
||||
|
||||
def test_server_default_executor(self):
|
||||
from cloud.server import _default_executor
|
||||
with mock.patch.dict(os.environ,
|
||||
{"CLOUD_DEFAULT_EXECUTOR":
|
||||
"cloud-hypervisor"}):
|
||||
self.assertEqual(_default_executor(), "cloud-hypervisor")
|
||||
with mock.patch.dict(os.environ,
|
||||
{"CLOUD_DEFAULT_EXECUTOR": "bogus"}):
|
||||
self.assertEqual(_default_executor(), "docker")
|
||||
|
||||
|
||||
class ServerWhitelistTest(unittest.TestCase):
|
||||
"""server 白名单端到端：真实 HTTP + 临时库 + 临时端口。"""
|
||||
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.db_path = os.path.join(self.tmp.name, "control.db")
|
||||
try:
|
||||
from cloud.auth import AuthStore
|
||||
from cloud.server import create_server
|
||||
except ImportError:
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
from cloud.auth import AuthStore
|
||||
from cloud.server import create_server
|
||||
self.auth = AuthStore(self.db_path)
|
||||
self.owner_id, self.owner_key = self.auth.create_user("owner")
|
||||
old = os.environ.get("CODEX_GLOBAL_ADMINS")
|
||||
self._old_admins = old
|
||||
os.environ["CODEX_GLOBAL_ADMINS"] = self.owner_id
|
||||
self.server = create_server(self.db_path, "127.0.0.1", 0)
|
||||
self.port = self.server.server_address[1]
|
||||
self.thread = threading.Thread(
|
||||
target=self.server.serve_forever, daemon=True)
|
||||
self.thread.start()
|
||||
status, self.project = self._req(
|
||||
"POST", "/projects", {"name": "demo"}, self.owner_key)
|
||||
self.assertEqual(status, 201, self.project)
|
||||
|
||||
def tearDown(self):
|
||||
self.server.shutdown()
|
||||
self.server.server_close()
|
||||
self.thread.join(timeout=5)
|
||||
try:
|
||||
self.auth.conn.close()
|
||||
except Exception:
|
||||
pass
|
||||
self.tmp.cleanup()
|
||||
if self._old_admins is None:
|
||||
os.environ.pop("CODEX_GLOBAL_ADMINS", None)
|
||||
else:
|
||||
os.environ["CODEX_GLOBAL_ADMINS"] = self._old_admins
|
||||
|
||||
def _req(self, method, path, body=None, key=None):
|
||||
conn = http.client.HTTPConnection("127.0.0.1", self.port,
|
||||
timeout=10)
|
||||
hdrs = {}
|
||||
payload = None
|
||||
if body is not None:
|
||||
payload = json.dumps(body).encode("utf-8")
|
||||
hdrs["Content-Type"] = "application/json"
|
||||
if key is not None:
|
||||
hdrs["X-API-Key"] = key
|
||||
try:
|
||||
conn.request(method, path, body=payload, headers=hdrs)
|
||||
resp = conn.getresponse()
|
||||
data = resp.read()
|
||||
finally:
|
||||
conn.close()
|
||||
try:
|
||||
parsed = json.loads(data.decode("utf-8")) if data else None
|
||||
except ValueError:
|
||||
parsed = None
|
||||
return resp.status, parsed
|
||||
|
||||
def test_task_executor_cloud_hypervisor_accepted(self):
|
||||
status, task = self._req(
|
||||
"POST", "/tasks",
|
||||
{"project_id": self.project["id"], "prompt": "x",
|
||||
"executor": "cloud-hypervisor"}, self.owner_key)
|
||||
self.assertEqual(status, 201, task)
|
||||
self.assertEqual(task["executor"], "cloud-hypervisor")
|
||||
|
||||
def test_task_executor_bogus_rejected(self):
|
||||
status, body = self._req(
|
||||
"POST", "/tasks",
|
||||
{"project_id": self.project["id"], "prompt": "x",
|
||||
"executor": "bogus"}, self.owner_key)
|
||||
self.assertEqual(status, 400)
|
||||
self.assertIn("cloud-hypervisor", body["error"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
+36
-5
@@ -195,7 +195,8 @@ from cloud.pr_exec import (claim_pr_job, execute_task_pr, finish_pr_job,
|
||||
mark_pr_job_needs_local_worker)
|
||||
from cloud.queue import budget_exceeded, ready_tasks
|
||||
from cloud.repos import RepoError, checkout_dir, resolve_repo
|
||||
from cloud.sandbox import CloudExecutor, DockerExecutor, LocalExecutor
|
||||
from cloud.sandbox import CloudExecutor, CloudHypervisorExecutor, \
|
||||
DockerExecutor, LocalExecutor
|
||||
from cloud.uploads import upload_root as _upload_root
|
||||
from cloud.sandbox import _default_runner as _sandbox_default_runner
|
||||
from cloud.caps import effective_capacity, resolve_capacity
|
||||
@@ -506,6 +507,24 @@ class Worker:
|
||||
self.executor.ignore_host_budget = self.ignore_host_budget
|
||||
except Exception:
|
||||
pass
|
||||
# CHV 审计转发：VM 生命周期事件（启动/暂停/恢复/快照/
|
||||
# 销毁/失败）经本回调进 AuthStore 审计（_audit 本身
|
||||
# best-effort，失败不影响任务）。时间线展示 VM 事件时
|
||||
# 读审计表即可，不另起一套 VM 事件表；成本/配额仍走
|
||||
# worker 现有 quota 语义，本回调绝不记账。
|
||||
if isinstance(self.executor, CloudHypervisorExecutor):
|
||||
def _chv_audit(event, task, detail, kind=None):
|
||||
try:
|
||||
project_id = task.get("project_id") \
|
||||
if isinstance(task, dict) else None
|
||||
self._audit(project_id, "chv_vm_" + event,
|
||||
detail or "")
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
self.executor.audit_callback = _chv_audit
|
||||
except Exception:
|
||||
pass
|
||||
self._cost_store = cost_store
|
||||
self._cancel_requested = set()
|
||||
# 真取消用：process_factory 可注入（单测用它返回一个假长跑
|
||||
@@ -2409,12 +2428,15 @@ class Worker:
|
||||
# -- 缺 key 门 ------------------------------------------------------
|
||||
def _executor_name(self):
|
||||
"""执行器名：DockerExecutor -> docker，CloudExecutor -> cloud，
|
||||
其它 -> local。cloud 绝不能回落成 local，否则 cloud worker
|
||||
CloudHypervisorExecutor -> cloud-hypervisor，其它 -> local。
|
||||
cloud/cloud-hypervisor 绝不能回落成 local，否则该 worker
|
||||
会认领 local 池的任务，池过滤就失效了。"""
|
||||
if isinstance(self.executor, DockerExecutor):
|
||||
return "docker"
|
||||
if isinstance(self.executor, CloudExecutor):
|
||||
return "cloud"
|
||||
if isinstance(self.executor, CloudHypervisorExecutor):
|
||||
return "cloud-hypervisor"
|
||||
return "local"
|
||||
def key_available(self):
|
||||
"""只查存在性：名在环境里且值非空即 True；值本身绝不返回。"""
|
||||
@@ -6756,7 +6778,12 @@ def _build_cli_executor(args, host_py):
|
||||
docker 时返回 DockerExecutor（env_key 名用 Worker 的解析规则，
|
||||
值本身绝不从这里传——DockerExecutor 只查存在性，-e 只传名），
|
||||
cloud 时返回 CloudExecutor（通用 HTTP runner，未配 endpoint
|
||||
时执行即明确失败，绝不降级）。"""
|
||||
时执行即明确失败，绝不降级），cloud-hypervisor 时返回
|
||||
CloudHypervisorExecutor（配置走 CHV_* 环境变量，缺依赖时
|
||||
执行即明确失败，绝不降级；任务内 LLM 不可自选它，见
|
||||
sandbox.py 该类 docstring）。"""
|
||||
if getattr(args, "executor", "local") == "cloud-hypervisor":
|
||||
return CloudHypervisorExecutor()
|
||||
if getattr(args, "executor", "local") == "cloud":
|
||||
return CloudExecutor()
|
||||
if getattr(args, "executor", "local") != "docker":
|
||||
@@ -6834,11 +6861,15 @@ def main(argv=None):
|
||||
"的 max_daily_turns 预算照常生效（双重门）")
|
||||
parser.add_argument("--max-tasks", type=int, default=None)
|
||||
parser.add_argument("--executor", default="local",
|
||||
choices=["local", "docker", "cloud"],
|
||||
choices=["local", "docker", "cloud",
|
||||
"cloud-hypervisor"],
|
||||
help="执行器：local 调 host.py 子进程；docker"
|
||||
" 调 DockerExecutor（无 docker 时任务"
|
||||
"明确失败，不静默降级）；cloud 调通用"
|
||||
" HTTP runner（需 CLOUD_RUNNER_ENDPOINT）")
|
||||
" HTTP runner（需 CLOUD_RUNNER_ENDPOINT）；"
|
||||
"cloud-hypervisor 调 CloudHypervisorExecutor"
|
||||
"（VM 内经 SSH 调 host.py，配置走 CHV_*"
|
||||
" 环境变量，缺依赖时任务明确失败）")
|
||||
parser.add_argument("--worker-id", default=None,
|
||||
help="多 worker 池身份（默认：主机名-进程号；"
|
||||
"生产部署应配稳定 id，否则重启后旧行变 stale）")
|
||||
|
||||
@@ -0,0 +1,173 @@
|
||||
# Cloud Hypervisor 执行器生产就绪清单
|
||||
|
||||
> 状态图例：✅ 代码已完成 · 🧪 mock 测试已覆盖（无真 VM） ·
|
||||
> ⏳ 需 KVM 宿主实测 · 🏗️ 需镜像构建 · 📈 需压测 · 🔒 需安全审计
|
||||
>
|
||||
> 本清单是 PR 的一部分，不是事后补的口号。凡是标 ⏳/🏗️/📈/🔒
|
||||
> 的项，在本轨内一律不声称已验证、已建成、已通过。
|
||||
|
||||
## 0. 交付点卡在哪一关（必读，先于逐项清单）
|
||||
|
||||
- **CI 门禁（未过，交付点 = PR OPEN）**：CI 必须通过才能合并。
|
||||
此前全仓 CI 因 GitHub 账单问题 0 步骤失败；若账单未恢复、
|
||||
CI 仍失败，本 PR 只到 OPEN，不合并，更不为变绿改 CI 配置
|
||||
或绕过检查。
|
||||
- **真 VM 门禁（未过，执行器默认关闭）**：生产启用（把真实任务
|
||||
路由到 cloud-hypervisor、部署到生产主机）必须先有带 /dev/kvm
|
||||
的宿主并完成真实 VM 启动、快照恢复、任务端到端验证。
|
||||
当前事实：生产主机 124.222.165.170 本身就是 CVM
|
||||
（systemd-detect-virt=kvm，DMI product=CVM），/dev/kvm 不存在、
|
||||
无 vmx/svm 标志、无 kvm 模块、未装 cloud-hypervisor，无嵌套
|
||||
虚拟化透传；本 VM 容器内同样无 /dev/kvm。
|
||||
故真 VM、快照恢复、压测在本轨内一律不声称已验证；执行器
|
||||
默认关闭（server 默认仍是 docker，worker 默认仍是 local，
|
||||
CHV 只在任务创建时显式选 `executor=cloud-hypervisor` 且有
|
||||
CHV worker 认领时才跑），不进入生产任务路由。
|
||||
- **还差什么才能真正上生产**（按顺序）：① 用户侧开通带 /dev/kvm
|
||||
的裸金属或开启嵌套虚拟化的实例；② 在该宿主上构建并发布
|
||||
chv-guest 镜像（见 §4）；③ 在该宿主上跑通真实 VM 启动、
|
||||
快照恢复、任务端到端、预热池与压测；④ 安全审计通过（§6）；
|
||||
⑤ CI 账单恢复且本 PR CI 变绿后，按当时 main 头复核 diff 再合并、
|
||||
再部署。缺任何一项，都不把 CHV 推上生产。
|
||||
|
||||
## 1. 单任务单 VM 隔离
|
||||
|
||||
- ✅🧪 一个任务一个 VM：dispatch/run/goal 均走 wake → 跑 → finally
|
||||
shutdown，同一任务重复 wake 先关旧 VM；VM 名、api-socket、
|
||||
每任务磁盘路径均按任务派生，互不相同（测试断言）。
|
||||
- ✅🧪 每任务磁盘独立：copy 模式复制基础镜像为每任务磁盘；
|
||||
reuse 模式仅单租户实验，生产禁用（代码与文档双写）。
|
||||
- ✅🧪 租户无共享路径：执行器不挂载宿主根、宿主代码目录或含
|
||||
密钥目录；客机代码在镜像内 `/app`，不在宿主共享挂载上。
|
||||
- ✅🧪 密钥不进镜像/代码/prompt：SSH argv 只有 key 文件路径；
|
||||
模型 key 由客机 cloud-init 预置的环境变量名注入；测试断言
|
||||
命令中不含哨兵密钥值。
|
||||
- 🔒 需安全审计：VM 逃逸面（virtio 设备、tap 桥接、SSH 配置）、
|
||||
客机 root 提权面、租户间东西向隔离的宿主网络配置，须由
|
||||
安全审计逐项确认，代码自查不能代替审计。
|
||||
|
||||
## 2. 快速启动：预热快照池 + 镜像缓存
|
||||
|
||||
- ✅🧪 快照/恢复接口：snapshot_vm / restore_vm（restore 后显式
|
||||
resume，见 §3），WarmPool 接口与状态机（cold → prewarming →
|
||||
warm → assigned → running → recycled/destroyed）。
|
||||
- ✅🧪 冷启动回落：池空时 acquire 返回 cold，调用方回落 wake_vm
|
||||
冷启动，绝不降级 Docker/Local。
|
||||
- ✅🧪 预热条目一次性：acquire 消耗、release 一律销毁，绝不跨
|
||||
任务复用脏 VM。
|
||||
- ⏳ 需 KVM 宿主实测：预热池的真实冷启动耗时、resume 耗时、
|
||||
池大小与命中率、快照来源一致性，均需真机数据，mock 数字
|
||||
不能当性能结论。
|
||||
- 📈 需压测：多 VM 并行预热/分配/销毁的宿主 CPU/内存/磁盘
|
||||
压力、快照存储增长，均未测。
|
||||
|
||||
## 3. 完整生命周期
|
||||
|
||||
- ✅🧪 create / boot：API 模式 wake_vm（vmm.ping → vm.create →
|
||||
vm.boot → SSH 就绪）。
|
||||
- ✅🧪 pause-resume（sleep-wake）：pause_vm / resume_vm（ch-remote）。
|
||||
- ✅🧪 snapshot-restore：restore_vm（restore + 显式 resume +
|
||||
SSH 就绪），恢复后默认 paused 的坑已在代码与测试里钉住。
|
||||
- ✅🧪 hibernate：CHV 无原生 hibernate 端点，本轨用 pause +
|
||||
snapshot + shutdown 等价实现，文档与代码双写“等价实现”，
|
||||
不声称接了原生端点；唤醒走 restore_vm。
|
||||
- ✅🧪 shutdown-destroy / recycle：shutdown_vm（ch-remote
|
||||
shutdown + 杀进程 + 删 socket）、recycle_vm（保留
|
||||
task_checkpoints/task_attempts，丢失 guest /tmp 与未拷出
|
||||
的 worktree 临时物，绝不声称恢复临时文件）。
|
||||
- ⏳ 需 KVM 宿主实测：上述全部生命周期在真机上的状态转换、
|
||||
快照恢复后 guest 状态一致性、休眠后文件/worktree 保留，
|
||||
均未在真机验证。
|
||||
|
||||
## 4. 完整开发环境（客机镜像）
|
||||
|
||||
- ✅ 客机镜像内容清单：`chv-guest/README.md` 按
|
||||
`cloud/docker/Dockerfile` 能力对齐（Python 3.12、git、
|
||||
pandoc/poppler/latex、python-docx/openpyxl/pptx/PyMuPDF/
|
||||
Pillow、codex 0.164.0-alpha.9 成套、Playwright + Chromium、
|
||||
openssh-server、cloud-init）。
|
||||
- ✅ 构建脚本骨架：`chv-guest/build.sh`（参数校验、版本 tag
|
||||
格式、KVM 宿主检查、TODO 步骤清单）。
|
||||
- 🏗️ 需镜像构建：镜像本身未构建、未发布；BUILDINFO 版本核对、
|
||||
浏览器版本钉死、中文 LaTeX 单独一版，均待构建时做。
|
||||
- ⏳ 需 KVM 宿主实测：镜像在真 VM 里启动、codex/Playwright/
|
||||
git/文档工具链逐项跑通，均未验证。
|
||||
|
||||
## 5. 资源与成本
|
||||
|
||||
- ✅🧪 每 VM 配额：CPU/内存进 VmConfig（CHV_CPU/CHV_MEM），
|
||||
磁盘配额门（CHV_DISK_GB，超限拒绝），宿主容量门
|
||||
（CHV_MAX_VMS，超限明确拒绝排队，不降级）。
|
||||
- ✅ 超时与最大轮次：SSH 单次超时（CHV_TASK_TIMEOUT）+
|
||||
worker task_timeout 外层护栏；goal 最大轮次 100（与 Local
|
||||
同款），执行器内部不重试（重试是 worker max_retries 语义）。
|
||||
- ✅ 成本复用：不另起一套计费；worker 现有 quota/成本语义
|
||||
（per-user USD 配额、quota 看门狗、成本写库）仍是唯一口径，
|
||||
执行器只把 host.py 用量原样返回。
|
||||
- ✅ 空闲休眠省成本：hibernate_vm（pause + snapshot +
|
||||
shutdown）+ WarmPool 释放即销毁，空闲 VM 不占 vCPU/内存
|
||||
（快照只占磁盘，见 §2 压测项）。
|
||||
- ✅ autoscaler：按 executor 过滤队列深度（已有）+ 本轨透传
|
||||
`--executor` 给拉起的 worker（否则 CHV 池永远饿死）。
|
||||
- 📈 需压测：配额下的真实资源占用、超限拒绝的排队行为、
|
||||
成本与配额在长任务下的实际表现，均未测。
|
||||
|
||||
## 6. 网络与治理
|
||||
|
||||
- ✅🧪 受控 egress：默认 fail-closed（CHV_EGRESS_PROXY 未设且
|
||||
未显式允许直连时 preflight 直接失败）；客机命令加 env
|
||||
代理前缀，出口只走白名单代理（Tinyproxy 思路复用）。
|
||||
- ⏳🔒 需宿主网络配置 + 安全审计：tap 桥接是否真的只路由到
|
||||
白名单代理、VM 能否绕过代理直连外网，是宿主网络配置问题，
|
||||
执行器代码不能自证，必须在 KVM 宿主上实测并审计。
|
||||
- ✅ 审批门：任务 approval 经 `_approval_cli_args` 进客内 host.py
|
||||
顶层参数，VM 内 agent 不能自改；delete/deploy/spend 类动作
|
||||
仍走 server 的 approval/merge_gate，不由 VM 内 agent 自批
|
||||
（本轨不改 approval/merge_gate 语义，只复用）。
|
||||
- ✅ 工具与系统调用范围收敛：VM 边界本身就是系统调用边界
|
||||
（客机独立内核）；plan 任务走 read-only 沙箱参数；最大步数
|
||||
（goal 100 轮）、时间（task_timeout）、重试（worker 1 次）
|
||||
超限即停并审计（vm_failed 带 kind）。
|
||||
|
||||
## 7. 可观测与审计
|
||||
|
||||
- ✅🧪 VM 事件审计：create/boot/ready/pause/resume/snapshot/
|
||||
restore/shutdown/destroy/failed/recycled/warm_pool_acquire/
|
||||
release 均经 audit_callback 发出；worker 接线后转 AuthStore
|
||||
审计（_audit 本身 best-effort）。
|
||||
- ✅🧪 失败分类：ChvFailure 带 kind（kvm_missing/binary_missing/
|
||||
kernel_missing/image_missing/ssh_key_missing/cpu_flag_missing/
|
||||
egress_not_configured/api_timeout/guest_not_ready/vm_create_failed/
|
||||
vm_boot_failed/host_capacity_exceeded/disk_quota_exceeded），
|
||||
不笼统 failed。
|
||||
- ✅ 断点对齐：VM 快照/休眠与现有 task_checkpoints/task_attempts
|
||||
语义对齐（recycle 保留两者，丢失临时文件）；goal 断点续跑
|
||||
方法（goal_resume_breakpoint_task）已实现，worker 层对非
|
||||
Local 执行器的断点续跑目前仍明确失败（见执行器 docstring），
|
||||
后续 worker 接线是单独一轨，不在本轨假装已通。
|
||||
- ⏳ 需 KVM 宿主实测：审计事件在真机任务 timeline 里的完整
|
||||
呈现、失败分类与真实故障的对应关系，均未验证。
|
||||
|
||||
## 8. 多 VM 并发与容量
|
||||
|
||||
- ✅🧪 worker 注册：`_executor_name()` 返回 cloud-hypervisor，
|
||||
worker register/heartbeat 已上报 executor + capacity（现有
|
||||
机制复用，无需新表）。
|
||||
- ✅🧪 多 VM 并行：`_vms` 登记 + 锁保护，wake 按任务名登记，
|
||||
不同任务互不覆盖；容量门在 wake 前检查。
|
||||
- ✅🧪 残留 VM 清理：cleanup_stale_vms 处理已死/已关/socket
|
||||
残留三种情况，只清 chv- 前缀 socket。
|
||||
- ✅🧪 容量不足明确拒绝：kind=host_capacity_exceeded，调用方
|
||||
排队/拒绝，绝不降级 Docker/Local（测试断言 launcher 未调用）。
|
||||
- 📈 需压测：多 VM 并行下的宿主容量、VM 启动并发上限、残留
|
||||
清理在高压下的表现，均未测。
|
||||
|
||||
## 9. 默认关闭与生产启用条件（重申）
|
||||
|
||||
- ✅ 默认关闭：server 默认执行器仍是 docker，worker 默认 local；
|
||||
CHV 只在显式选 executor 且有 CHV worker 时才跑；无 CHV_*
|
||||
配置时 preflight 直接失败。
|
||||
- ⏳ 生产启用前置（全部未完成）：带 /dev/kvm 的宿主开通 →
|
||||
chv-guest 镜像构建发布 → 真实 VM 启动/快照恢复/任务端到端
|
||||
验证 → 压测 → 安全审计 → CI 变绿后按当时 main 头复核 diff
|
||||
再合并部署。缺一不可。
|
||||
Reference in new issue
Block a user