Compare commits
50
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
50cae2a0af | ||
|
|
0203f32627 | ||
|
|
adc7bf6666 | ||
|
|
39084ce2bf | ||
|
|
c4661b64db | ||
|
|
f20cbc79cd | ||
|
|
7a227323f2 | ||
|
|
2be69af92c | ||
|
|
fc33eed262 | ||
|
|
179a86d655 | ||
|
|
581e850eb7 | ||
|
|
04287f6053 | ||
|
|
c1d00b3620 | ||
|
|
4a7ca25334 | ||
|
|
f093558c0d | ||
|
|
9f593d62ef | ||
|
|
2d12179b9a | ||
|
|
b41fb67026 | ||
|
|
e02828c590 | ||
|
|
d324d87529 | ||
|
|
717efc4688 | ||
|
|
29d3a5a474 | ||
|
|
73176e135c | ||
|
|
304707b372 | ||
|
|
3209f59cd6 | ||
|
|
55f6d7ad6c | ||
|
|
78098c5594 | ||
|
|
a90cf78d84 | ||
|
|
c61484aa68 | ||
|
|
06c3e172fe | ||
|
|
f448ecb74b | ||
|
|
2ad29eeb1f | ||
|
|
c39715659e | ||
|
|
ba5df980bf | ||
|
|
4f3a176771 | ||
|
|
5b6514d019 | ||
|
|
d7cc482ed5 | ||
|
|
e7f699c62b | ||
|
|
67ca0b86cd | ||
|
|
cbd5ab7c0b | ||
|
|
dcb0824f78 | ||
|
|
4c8e4eac7b | ||
|
|
5206dfbc43 | ||
|
|
4d09518338 | ||
|
|
9e60d5dfaa | ||
|
|
b5e75f156d | ||
|
|
368bccd16e | ||
|
|
3b41f864e8 | ||
|
|
79ba56bd51 | ||
|
|
7f18c2d13b |
No files matched your search
+86
-22
@@ -10,8 +10,10 @@
|
||||
|
||||
## 1. 镜像定位
|
||||
|
||||
- 基础:Ubuntu 云镜像(与 `cloud/docker/Dockerfile` 的任务镜像
|
||||
对齐能力,不是另起一套环境口径)。
|
||||
- 基础:Ubuntu Minimal 24.04 官方云镜像(轻量口径,2026-11-06
|
||||
用户定:停掉 5.5G 全量版,不再对齐 Manus,不做三面板/
|
||||
多租户/生产级全套;只装 codex + Python + SSH + cloud-init
|
||||
+ 薄客机 agent + git + gt + 最小中文字体)。
|
||||
- 载体:Cloud Hypervisor 直接内核启动(vmlinux)+ raw 磁盘
|
||||
镜像,执行器用 `CHV_KERNEL` / `CHV_ROOTFS` 指向它们。
|
||||
- 客内 host.py 位置:`/app/host.py`(与 Docker 的 `/app:ro`
|
||||
@@ -21,20 +23,22 @@
|
||||
生产凭证。模型 key 由客机 cloud-init 预置的运行环境变量名
|
||||
注入(与 Docker 的 `-e <名>` 同款惯例),不写进镜像层。
|
||||
|
||||
## 2. 内容清单(按现有 Docker 任务镜像能力对齐)
|
||||
## 2. 内容清单(轻量口径,2026-11-06 用户定)
|
||||
|
||||
| 类别 | 内容 | 来源/版本口径 |
|
||||
| --- | --- | --- |
|
||||
| 语言与工具链 | Python 3.12、git(Node.js 不装:Playwright 驱动是 Python playwright,见 PLAYWRIGHT=1 分支;任务镜像 Dockerfile 里也没有 Node.js,本清单不虚构它) | 与任务镜像同款大版本,构建时钉小版本并记入构建日志 |
|
||||
| 文档/图片工具 | pandoc、poppler-utils(pdftotext)、latexmk、texlive-latex-base、python-docx、openpyxl、python-pptx、PyMuPDF、Pillow | 逐项照抄 `cloud/docker/Dockerfile` 的 apt/pip 清单,不擅自加减 |
|
||||
| 语言与工具链 | Python 3.12、git、curl、ca-certificates、xz-utils、fontconfig、openssh-client(Node.js 不装:Playwright 驱动是 Python playwright,见 PLAYWRIGHT=1 分支;任务镜像 Dockerfile 里也没有 Node.js,本清单不虚构它) | 构建时钉小版本并记入构建日志;apt 用 --no-install-recommends(B-21:默认装 Recommends 会连带拉入 gcc 全套工具链顶满根分区) |
|
||||
| 文档/图片工具 | 已裁剪:pandoc、poppler-utils、latexmk、texlive-latex-base、python-docx、openpyxl、python-pptx、PyMuPDF、Pillow 不在轻量镜像内 | 全量版曾照抄 `cloud/docker/Dockerfile` 的 apt/pip 清单,轻量版按用户决定砍掉 |
|
||||
| agent 引擎 | codex、codex-code-mode-host、codex-resources/bwrap(成套,约 367M,缺 sibling 直接构建失败) | codex 0.164.0-alpha.9(与生产任务镜像一致);`CODEX_BIN_DIR` 由构建方提供,bin/ 不进仓库 |
|
||||
| 浏览器 | Playwright + Chromium(浏览器自动化 MCP 用,PLAYWRIGHT=1 时才装) | Python playwright 装当时的 Chromium,未在 BUILDINFO 里钉浏览器版本——这是已知缺口,不是已钉死 |
|
||||
| 客机基础 | openssh-server(SSH 通道用)、cloud-init、qemu-guest-agent(可选,宿主信息用) | sshd 只允许 key 登录,禁止密码登录——build.sh 写入 /etc/ssh/sshd_config.d/99-chv.conf 并经 sshd -t + grep 断言(feat-chv-secfix,M8,此前只是文档声称) |
|
||||
| 任务代码 | codex-app-server 仓库代码置于 `/app`(host.py 入口) | 版本见 §3,不挂载宿主代码目录(VM 与宿主无共享路径) |
|
||||
| 浏览器 | Playwright + chromium-headless-shell(默认装,不要完整 Chromium) | Python playwright 装当时的 headless-shell,版本以 ms-playwright 缓存目录名记入 BUILDINFO.versions.chromium;浏览器在系统路径 /usr/local/share/ms-playwright(B-22:virt-customize 里 HOME=/,默认缓存会落到根目录 /.cache),agent 运行时须设 PLAYWRIGHT_BROWSERS_PATH 指向它;单独钉死见 §3:CHROMIUM_HEADLESS_SHELL_REV 设为纯数字修订号即钉死,未设时只记录、不强制 |
|
||||
| 客机基础 | openssh-server(SSH 通道用)、cloud-init(qemu-guest-agent 已裁剪,CHV 不用它) | sshd 只允许 key 登录,禁止密码登录——build.sh 写入 /etc/ssh/sshd_config.d/99-chv.conf 并经 sshd -t + grep 断言(feat-chv-secfix,M8,此前只是文档声称) |
|
||||
| Gitea 同步 | gt 1.0.0(官方 install.sh + SHA256 校验)、chv-sync(启动时 clone、显式 pull/push 的同步脚本) | 凭证走 seed 盘 /etc/chv/guest-env(0600)投递,不进镜像;见 §7 |
|
||||
| 任务代码 | 仅 `/app/host.py` 一个文件(host.py 只用标准库,不依赖 cloud/;旧版把整个 cloud/ 拷进 /app/,客内可读全套 server/worker 代码,M4 收尾已收紧为最小集) | 版本见 §3(repo_commit),不挂载宿主代码目录(VM 与宿主无共享路径) |
|
||||
|
||||
中文 LaTeX 说明:与 Docker 镜像同款口径——xelatex + CJK
|
||||
字体会再增数百 MB,本清单不默认包含;需要时单独一版镜像,
|
||||
不在本轨悄悄加。
|
||||
不在本轨悄悄加。字体只留最小的 fonts-wqy-microhei(中文能
|
||||
看),不要全套 fonts-noto-cjk。
|
||||
|
||||
## 3. 版本化
|
||||
|
||||
@@ -52,21 +56,31 @@ chv-guest-<ubuntu 版本>-<codex 版本>-<仓库 commit 短 hash>
|
||||
- `vmlinux`(与 Ubuntu 云镜像配套的内核)
|
||||
- `disk.raw`(根文件系统镜像)
|
||||
- `BUILDINFO.json`:`{ubuntu, codex_version, repo_commit, built_at,
|
||||
contents: [清单逐项名]}`——注意:build.sh 生成的 contents 只有
|
||||
名字列表,没有逐项版本号(feat-chv-fix 纠正 §3 旧措辞);
|
||||
逐项版本只留在构建日志里,未进 BUILDINFO,这是已知缺口。
|
||||
contents: [清单逐项名]}`——contents 只有名字列表;逐项版本
|
||||
在 `versions` 里(B-02:dpkg + pip freeze 从客机读回,
|
||||
chromium 以 ms-playwright 缓存目录名记,gt 以 gt_version 记),
|
||||
`generate_lockfile.py` / `check_lockfile.py` 以它为唯一来源。
|
||||
Chromium 单独钉死:构建时设 CHROMIUM_HEADLESS_SHELL_REV
|
||||
为纯数字修订号(如 1248),buildinfo.py 校验读回的缓存
|
||||
目录名须恰好是 `chromium_headless_shell-<rev>`,否则构建
|
||||
失败,BUILDINFO.chromium_pinned_rev 记该值;未设时记 null、
|
||||
只记录不强制,构建输出明说“未钉死”。真实 lockfile 产出
|
||||
须等 V-9 真机构建(本机无 /dev/kvm,版本号绝不编造)。
|
||||
|
||||
## 4. 构建脚本(补缺口轨:已从骨架改成真实步骤)
|
||||
|
||||
见 `build.sh`:参数校验 → 下载 Ubuntu 云镜像 → qemu-img
|
||||
转 raw → virt-customize 装 §2 清单、拷入 codex 成套二进制
|
||||
与仓库代码到 /app、写 host key 不重生成配置 →
|
||||
见 `build.sh`:参数校验 → 下载 Ubuntu Minimal 云镜像(SHA256
|
||||
必填)→ qemu-img 转 raw(轻量版不扩盘,底座根分区 2.4G 够用;
|
||||
全量版 B-16 的 +12G 扩盘已删除)→ virt-customize 装 §2 清单、
|
||||
拷入 codex 成套二进制与 host.py(仅此一个文件,不拷 cloud/)
|
||||
到 /app、装 gt 1.0.0、拷入
|
||||
chv-sync 并写 cloud-init 启动钩子、写 host key 不重生成配置 →
|
||||
virt-get-kernel 取内核 → 生成并校验 BUILDINFO.json。
|
||||
`./build.sh --dry-run out/` 只打印命令、不执行、不要求
|
||||
KVM;全量模式在无 /dev/kvm 的机器上第一步就明确退出
|
||||
(`CHV_GUEST_BUILD needs a KVM host`),绝不假装构建成功。
|
||||
本轨未在真机上跑过全量模式,镜像未建成、未发布,仍属
|
||||
“需 KVM 宿主实测”项。
|
||||
全量版曾在 WSL KVM 宿主(ajpkmq6i)上跑过四轮真机构建,
|
||||
轻量版沿用同一宿主与流程,构建记录见 `~/workspace/chv-prod-audit/`。
|
||||
|
||||
## 5. Host key 首启注入硬性要求(补缺口轨新增)
|
||||
|
||||
@@ -76,11 +90,20 @@ KVM;全量模式在无 /dev/kvm 的机器上第一步就明确退出
|
||||
- 装 cloud-init 且支持 NoCloud seed(seed 磁盘(CIDATA ISO,feat-chv-fix 起挂进 VmConfig
|
||||
第二块只读盘)由执行器在 VM 启动前生成,含 meta-data 与 user-data);
|
||||
- 首启时先应用 seed 再启动 sshd,seed 里的 user-data
|
||||
把该 VM 的 host key 对写进 /etc/ssh/ 并重启 sshd;
|
||||
- 镜像里已钉死 `ssh_deletekeys: false` 与
|
||||
`ssh_genkeytypes: []`(build.sh 已写),首启不得无条件
|
||||
重新生成 host key——重生成会使预置的 known_hosts 立即
|
||||
失效,SSH 钉死直接拒绝,这是设计,不是 bug;
|
||||
把该 VM 的 host key 对写进 /etc/ssh/(write_files,
|
||||
0600);seed 里绝不能带 `runcmd:`——user-data 的
|
||||
runcmd 会整体替换镜像 `/etc/cloud/cloud.cfg.d/` 里
|
||||
的 runcmd(cloud-init 不跨配置类型合并),旧版 seed
|
||||
的 restart ssh 两行就这样顶掉了 `99-chv-sync.cfg`
|
||||
的 `chv-sync boot` 钩子(sync-test 首轮实测);sshd
|
||||
重启由镜像自己的 runcmd(`99-chv-sync.cfg`,cloud-final
|
||||
阶段)做,见 `cloud/chv/hostkeys.py` 根因禁令;
|
||||
- 镜像里已钉死 `ssh_deletekeys: false`(build.sh 已写),首启
|
||||
不得无条件重新生成 host key——重生成会使预置的 known_hosts
|
||||
立即失效,SSH 钉死直接拒绝,这是设计,不是 bug;无预置 key
|
||||
时首启 bootcmd 只在缺 key 时 `ssh-keygen -A` 生成(B-23:
|
||||
曾写 `ssh_genkeytypes: []` 彻底关掉首启生成,客机 sshd 起
|
||||
不来,已删,绝不再写);
|
||||
- sshd 只允许 key 登录,禁止密码登录(build.sh 写入
|
||||
/etc/ssh/sshd_config.d/99-chv.conf:PasswordAuthentication no、
|
||||
PermitRootLogin prohibit-password、ChallengeResponseAuthentication no、
|
||||
@@ -99,3 +122,44 @@ KVM;全量模式在无 /dev/kvm 的机器上第一步就明确退出
|
||||
磁盘,任务之间文件系统独立;reuse 模式仅单租户实验,preflight 拦截(须同时设 CHV_REUSE_SINGLE_TENANT_EXPERIMENT=1 并审计放行,feat-chv-fix)。
|
||||
- 快照基准:`CHV_SNAPSHOT_URL` 指向的快照必须由本镜像的干净
|
||||
启动状态制成(pause 后 snapshot),预热池用它做干净 VM 来源。
|
||||
|
||||
## 7. Gitea 双向同步(用户工作区口径,2026-11-06 追加)
|
||||
|
||||
主角是用户,不是 agent:用户在 Gitea 上的仓库在 VM 启动时
|
||||
clone 到 `~/workspace`,用完再同步回 Gitea;VM 是用户的工作
|
||||
区,Gitea 是权威远端。同步口径沿用 Git Workspace Platform
|
||||
已定的规则:分叉或冲突时失败关闭,不强制覆盖、不自动
|
||||
rebase、不静默丢提交。
|
||||
|
||||
- 凭证:seed 盘投递 `/etc/chv/guest-env`(0600,root 所有),
|
||||
含 `GITEA_URL` / `GITEA_USER` / `GITEA_TOKEN` / `SYNC_REPO`
|
||||
/ 可选 `SYNC_BRANCH` / 可选 `GITEA_EMAIL`。token 绝不进镜像、
|
||||
代码、日志;`chv-sync boot`(root,cloud-init runcmd 调)把
|
||||
token 写进 `/home/chv/.git-credentials`(0600,git credential
|
||||
store 标准机制),再以 chv 身份 clone;`pull` / `push` /
|
||||
`status` 以 chv 身份跑,只用已落盘的 credential,不再读
|
||||
guest-env。无 guest-env 的启动静默不同步,VM 照常用。
|
||||
- 行为:`boot` 只在未 clone 时 clone,已 clone 则静默退出,
|
||||
不自动 pull(避免重启时覆盖进行中的工作);`pull` 要求
|
||||
工作树干净且只接受 fast-forward;`push` 无 upstream 时
|
||||
`push -u origin HEAD`,否则普通 push,非 fast-forward 即
|
||||
失败,绝不 `--force`。
|
||||
- gt 1.0.0 随镜像装好(官方 install.sh + SHA256 校验),登录
|
||||
不 baked:运行时用 `GITEA_TOKEN` + `GITEA_CLI_HOST` 环境变量
|
||||
即可用(`gt repo list` 等),或在 VM 里 `gt auth login`。
|
||||
- 验收(用户视角):VM 启动后仓在 `~/workspace` 且是最新
|
||||
HEAD;VM 内改文件 commit push 后 Gitea 网页可见该提交;
|
||||
Gitea 上改一次后 VM 内 pull 能同步回来。双向都通才算集成通。
|
||||
- 集成(2026-11-07,第二轮第一棒):执行器生成 seed 时自动写
|
||||
guest-env 已做(`cloud/chv/gitea_sync.py` + `wake_vm`):
|
||||
GITEA_TOKEN 进密钥集(只进 seed,绝不进 argv),其余 5 键
|
||||
也随 seed 进 guest-env;来源是宿主 `CHV_GITEA_*` 环境变量
|
||||
或任务 `gitea_sync` dict;必填 4 键缺失时起 VMM 前
|
||||
fail-closed(`gitea_sync_incomplete`),无同步变量时静默
|
||||
不同步。客机用户与静态网络由 seed 附加项供给:
|
||||
`CHV_SEED_USER_DATA_EXTRA_FILE`(含 `users:` 块的文件)与
|
||||
`CHV_SEED_NETWORK_CONFIG_FILE`(NoCloud network-config),
|
||||
路径配了但文件读不出时 wake 前 fail-closed
|
||||
(`seed_extra_missing`);附加 user-data 里禁 `runcmd:`。
|
||||
专用 token 与测试仓名须用户给,本轨不自编 token、不动
|
||||
生产 Gitea 配置。
|
||||
@@ -0,0 +1,379 @@
|
||||
#!/usr/bin/env python3
|
||||
"""chv-authd:客内凭证守护进程(feat-chv-muse-p3)。
|
||||
|
||||
对齐 Muse 官方安全文:authd 保管凭证(含 token,存于用户 VM
|
||||
而非集中基础设施),代理只见替身令牌;凭证存入运行时单元之外
|
||||
的安全凭证存储,代理不可见,仅在需要时注入。
|
||||
|
||||
本进程跑在客内、运行时单元之外(root 身份的 systemd 服务,
|
||||
见 chv-authd.service;P2 的 nspawn 单元落地后,codex 在单元内,
|
||||
本进程在单元外——单元内 root 映射为非特权用户,读不到本进程的
|
||||
socket 与 seed 文件)。它只做三件事:
|
||||
|
||||
1. 启动时读 /etc/chv/authd-seed(0600 root,seed 投递),把
|
||||
GITEA_TOKEN 持在内存里(cloud.chv.authd.CredentialVault),
|
||||
除此之外不落盘、不进日志;
|
||||
2. 在 Unix socket /run/chv-authd/authd.sock 上接受 git credential
|
||||
helper 的请求(SO_PEERCRED 只放行 root 与 chv 用户;socket
|
||||
用 group 权限——目录 0750、socket 0660、组为第一个非 root
|
||||
放行 uid 的主组,round-11 真机修复层 3,探针用的 0666
|
||||
不上生产);
|
||||
3. 每次放凭证前先过 Sentinel 评估(cloud.chv.sentinel,主机/
|
||||
方法/路径/解析后 IP/SSRF),再按次释放(单次、300 秒 TTL),
|
||||
全程审计(审计里无值)。
|
||||
|
||||
只用标准库。cloud/chv/authd.py 与 cloud/chv/sentinel.py 是叶子
|
||||
模块(只 import 标准库),镜像构建时随本文件一起拷进
|
||||
/usr/local/lib/chv-authd/(见 docs/chv-muse-p3-gap.md §2 的接入
|
||||
说明),本文件把该目录加进 sys.path 后 import。
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import socket
|
||||
import socketserver
|
||||
import struct
|
||||
import sys
|
||||
import threading
|
||||
|
||||
DEFAULT_LIB_DIR = "/usr/local/lib/chv-authd"
|
||||
DEFAULT_SEED_PATH = "/etc/chv/authd-seed"
|
||||
DEFAULT_SOCKET_PATH = "/run/chv-authd/authd.sock"
|
||||
DEFAULT_LOG_PATH = "/var/log/chv-authd.log"
|
||||
|
||||
# seed 文件里允许的键(与 cloud.chv.gitea_sync.GITEA_SYNC_KEYS
|
||||
# 同款白名单;未知键一律丢掉,调用方拼错的键绝不静默进保险库)。
|
||||
SEED_KEYS = frozenset({
|
||||
"GITEA_URL", "GITEA_USER", "GITEA_TOKEN",
|
||||
"SYNC_REPO", "SYNC_BRANCH", "GITEA_EMAIL",
|
||||
})
|
||||
|
||||
# SO_PEERCRED 放行的 uid:root + chv(git 以 chv 身份跑 helper)。
|
||||
# 可用环境变量 CHV_AUTHD_ALLOWED_UIDS 覆盖(逗号分隔)。
|
||||
DEFAULT_ALLOWED_UIDS = (0, 1000)
|
||||
|
||||
|
||||
def _socket_group_gid(allowed_uids, socket_group=None):
|
||||
"""socket 共享组的 gid:socket_group 显式给时优先(int 即 gid,
|
||||
str 即组名);否则取第一个非 root 放行 uid 的主组。解析不出
|
||||
(uid 无 passwd 条目、组名不存在)即 None——调用方回落
|
||||
root-only 权限(目录 0700、socket 0600),绝不把 socket 放给
|
||||
全员可读(round-11 真机修复层 3:探针用 0666,生产用 group
|
||||
权限——chv 在组内,目录 0750 + socket 0660 + chgrp)。"""
|
||||
import grp
|
||||
import pwd
|
||||
if socket_group is not None:
|
||||
if isinstance(socket_group, int):
|
||||
return socket_group
|
||||
try:
|
||||
return grp.getgrnam(socket_group).gr_gid
|
||||
except (KeyError, OverflowError):
|
||||
return None
|
||||
for uid in sorted(allowed_uids or ()):
|
||||
if uid == 0:
|
||||
continue
|
||||
try:
|
||||
return pwd.getpwuid(uid).pw_gid
|
||||
except (KeyError, OverflowError):
|
||||
continue
|
||||
return None
|
||||
|
||||
|
||||
def _set_group_perms(path, gid, mode):
|
||||
"""目录/socket 的组权限:先 chgrp(失败只放过,不中断启动),
|
||||
再 chmod。gid 为 None 时只 chmod(root-only 模式)。"""
|
||||
if gid is not None:
|
||||
try:
|
||||
os.chown(path, -1, gid)
|
||||
except OSError:
|
||||
pass
|
||||
try:
|
||||
os.chmod(path, mode)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def _load_seed(path):
|
||||
"""读 seed 文件(KEY=VALUE,每行一项)→ 白名单后的 dict。
|
||||
文件不存在即 FileNotFoundError(调用方 fail-fast,绝不带着
|
||||
“无凭证”的 authd 继续跑、让 git 静默拿不到凭证)。"""
|
||||
values = {}
|
||||
with open(path) as fh:
|
||||
for line in fh:
|
||||
line = line.strip()
|
||||
if not line or line.startswith("#") or "=" not in line:
|
||||
continue
|
||||
key, value = line.split("=", 1)
|
||||
key = key.strip()
|
||||
if key in SEED_KEYS and value.strip():
|
||||
values[key] = value.strip()
|
||||
return values
|
||||
|
||||
|
||||
def _audit(log_path, record):
|
||||
"""审计落盘(JSON 行,0600):只含绑定与 verdict,不含值。
|
||||
落盘失败只记 stderr,绝不炸守护进程(审计失败不改变放行
|
||||
决策是 worker 层的既定语义,见 executor._emit)。"""
|
||||
line = json.dumps(record, sort_keys=True)
|
||||
try:
|
||||
with open(log_path, "a") as fh:
|
||||
fh.write(line + "\n")
|
||||
except OSError as exc:
|
||||
print("chv-authd: audit 落盘失败:%s" % exc, file=sys.stderr)
|
||||
print("chv-authd: %s" % line, file=sys.stderr)
|
||||
|
||||
|
||||
def _operation_for_path(path):
|
||||
"""git 智能 HTTP 路径 → 操作(helper 不知道 git 操作,守护
|
||||
进程按路径推断;Sentinel 的 L7 规则已把路径限定在这三条内,
|
||||
推断不到的一律拒绝):
|
||||
- .../git-receive-pack → push(写)
|
||||
- .../git-upload-pack → pull(读)
|
||||
- .../info/refs → pull(读,clone 也走这条)
|
||||
- 裸 .../ → pull(读;round-11 真机修复层 4:git 智能
|
||||
HTTP 的发现请求只发裸 .git 路径,裸路径一律当 pull——
|
||||
只读默认;push 只走显式的 git-receive-pack)"""
|
||||
if path.endswith(".git/git-receive-pack"):
|
||||
return "POST", "push"
|
||||
if path.endswith(".git/git-upload-pack"):
|
||||
return "POST", "pull"
|
||||
if path.endswith(".git/info/refs"):
|
||||
return "GET", "pull"
|
||||
if path.endswith(".git"):
|
||||
return "GET", "pull"
|
||||
return None, None
|
||||
|
||||
|
||||
def _repo_from_path(path):
|
||||
"""从 git 路径取仓库名:/admin/repo.git/info/refs → admin/repo。
|
||||
取不出即 None(调用方拒绝)。"""
|
||||
parts = [p for p in (path or "").split("/") if p]
|
||||
if len(parts) < 2:
|
||||
return None
|
||||
repo = parts[1]
|
||||
if not repo.endswith(".git"):
|
||||
return None
|
||||
return "%s/%s" % (parts[0], repo[:-len(".git")])
|
||||
|
||||
|
||||
class AuthdHandler(socketserver.BaseRequestHandler):
|
||||
"""单次请求处理(每连接一 JSON 行):鉴权 → Sentinel 评估 →
|
||||
按次释放 → 回包。任何一步拒绝都回 {ok:false},不含值。"""
|
||||
|
||||
def handle(self):
|
||||
server = self.server
|
||||
try:
|
||||
creds = self.request.getsockopt(
|
||||
socket.SOL_SOCKET, socket.SO_PEERCRED,
|
||||
struct.calcsize("3i"))
|
||||
_pid, uid, _gid = struct.unpack("3i", creds)
|
||||
except (OSError, struct.error):
|
||||
self._reply({"ok": False, "error": "peercred_failed"})
|
||||
return
|
||||
if uid not in server.allowed_uids:
|
||||
server.audit({"event": "authd_peer_rejected",
|
||||
"uid": uid})
|
||||
self._reply({"ok": False, "error": "uid_not_allowed"})
|
||||
return
|
||||
try:
|
||||
buf = b""
|
||||
while b"\n" not in buf:
|
||||
chunk = self.request.recv(4096)
|
||||
if not chunk:
|
||||
break
|
||||
buf += chunk
|
||||
if len(buf) > 65536:
|
||||
break
|
||||
req = json.loads(buf.decode("utf-8"))
|
||||
except (OSError, ValueError):
|
||||
self._reply({"ok": False, "error": "bad_request"})
|
||||
return
|
||||
self._serve(req)
|
||||
|
||||
def _reply(self, obj):
|
||||
try:
|
||||
self.request.sendall(
|
||||
(json.dumps(obj) + "\n").encode("utf-8"))
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
def _serve(self, req):
|
||||
server = self.server
|
||||
connector = req.get("connector")
|
||||
host = (req.get("host") or "").lower()
|
||||
path = req.get("path") or "/"
|
||||
if connector != "gitea" or not host or not path:
|
||||
server.audit({"event": "authd_credential_refused",
|
||||
"kind": "bad_request_fields",
|
||||
"connector": connector})
|
||||
self._reply({"ok": False, "error": "bad_request_fields"})
|
||||
return
|
||||
method, operation = _operation_for_path(path)
|
||||
if method is None:
|
||||
server.audit({"event": "authd_credential_refused",
|
||||
"kind": "path_not_allowed",
|
||||
"connector": connector, "host": host,
|
||||
"path": path})
|
||||
self._reply({"ok": False, "error": "path_not_allowed"})
|
||||
return
|
||||
repo = _repo_from_path(path)
|
||||
if repo is None:
|
||||
server.audit({"event": "authd_credential_refused",
|
||||
"kind": "repo_unparseable",
|
||||
"connector": connector, "host": host})
|
||||
self._reply({"ok": False, "error": "repo_unparseable"})
|
||||
return
|
||||
# Sentinel 评估:主机/方法/路径/解析后 IP/SSRF。
|
||||
from cloud.chv.sentinel import EgressRequest
|
||||
scheme = "https"
|
||||
port = 443
|
||||
egress = EgressRequest(connector, method, scheme, host,
|
||||
port, path)
|
||||
decision = server.policy.evaluate(
|
||||
egress, resolve=getattr(server, "resolve", None))
|
||||
if decision.verdict != "allow":
|
||||
record = decision.audit(egress)
|
||||
server.audit(record)
|
||||
self._reply({"ok": False,
|
||||
"error": "sentinel_" + decision.verdict,
|
||||
"reasons": decision.reasons})
|
||||
return
|
||||
server.audit(decision.audit(egress))
|
||||
# 按次释放:绑定须与入库作用域一致(主机/仓库),单次有效。
|
||||
try:
|
||||
release = server.vault.release(
|
||||
connector, "GITEA_TOKEN", host, repo, operation)
|
||||
except Exception as exc: # AuthdRefusal:kind 在异常上
|
||||
kind = getattr(exc, "kind", "refused")
|
||||
server.audit({"event": "authd_credential_refused",
|
||||
"kind": kind, "connector": connector,
|
||||
"host": host, "repo": repo,
|
||||
"operation": operation})
|
||||
self._reply({"ok": False, "error": kind})
|
||||
return
|
||||
server.audit(server.vault.audit_release(release))
|
||||
try:
|
||||
value = server.vault.consume(release.release_id)
|
||||
except Exception as exc:
|
||||
kind = getattr(exc, "kind", "refused")
|
||||
server.audit({"event": "authd_credential_refused",
|
||||
"kind": kind, "connector": connector,
|
||||
"host": host, "repo": repo,
|
||||
"operation": operation})
|
||||
self._reply({"ok": False, "error": kind})
|
||||
return
|
||||
self._reply({"ok": True, "username": server.gitea_user,
|
||||
"password": value,
|
||||
"expires_at": release.expires_at})
|
||||
|
||||
|
||||
class AuthdServer(socketserver.ThreadingUnixStreamServer):
|
||||
"""Unix socket 服务(线程化,daemon_threads=True 避免僵尸线程)。"""
|
||||
|
||||
daemon_threads = True
|
||||
allow_reuse_address = False
|
||||
|
||||
def __init__(self, socket_path, vault, policy, gitea_user,
|
||||
allowed_uids, log_path, resolve=None,
|
||||
socket_group=None):
|
||||
self.vault = vault
|
||||
self.policy = policy
|
||||
self.gitea_user = gitea_user
|
||||
self.allowed_uids = frozenset(allowed_uids)
|
||||
self.log_path = log_path
|
||||
# 解析器可注入(测试用假解析器;不给时 Sentinel 用真实
|
||||
# getaddrinfo)。与执行器的 sentinel_resolver 同款口径。
|
||||
self.resolve = resolve
|
||||
# round-11 真机修复(层 3):socket 用 group 权限。
|
||||
# gid 取 socket_group 显式值或第一个非 root 放行 uid 的
|
||||
# 主组;解析不出即 None,回落 root-only(0700/0600)。
|
||||
self.socket_gid = _socket_group_gid(allowed_uids, socket_group)
|
||||
directory = os.path.dirname(socket_path)
|
||||
if directory:
|
||||
os.makedirs(directory, exist_ok=True)
|
||||
if self.socket_gid is not None:
|
||||
_set_group_perms(directory, self.socket_gid, 0o750)
|
||||
else:
|
||||
_set_group_perms(directory, None, 0o700)
|
||||
if os.path.exists(socket_path):
|
||||
os.unlink(socket_path)
|
||||
super().__init__(socket_path, AuthdHandler)
|
||||
if self.socket_gid is not None:
|
||||
_set_group_perms(socket_path, self.socket_gid, 0o660)
|
||||
else:
|
||||
_set_group_perms(socket_path, None, 0o600)
|
||||
|
||||
def audit(self, record):
|
||||
_audit(self.log_path, record)
|
||||
|
||||
|
||||
def build_server(seed_path=DEFAULT_SEED_PATH,
|
||||
socket_path=DEFAULT_SOCKET_PATH,
|
||||
log_path=DEFAULT_LOG_PATH,
|
||||
lib_dir=DEFAULT_LIB_DIR,
|
||||
allowed_uids=None,
|
||||
resolve=None,
|
||||
socket_group=None):
|
||||
"""建服务(不启动):读 seed → 入库 → 建策略 → 建 socket 服务。
|
||||
|
||||
seed 缺必填键(GITEA_URL/GITEA_USER/GITEA_TOKEN/SYNC_REPO)
|
||||
即 ValueError(只含缺失键名,不含值);无 seed 文件即
|
||||
FileNotFoundError——authd 绝不带着“无凭证”启动。"""
|
||||
if lib_dir not in sys.path:
|
||||
sys.path.insert(0, lib_dir)
|
||||
from cloud.chv.authd import CredentialVault
|
||||
from cloud.chv.sentinel import SentinelPolicy
|
||||
seed = _load_seed(seed_path)
|
||||
missing = sorted({"GITEA_URL", "GITEA_USER", "GITEA_TOKEN",
|
||||
"SYNC_REPO"} - set(seed))
|
||||
if missing:
|
||||
raise ValueError(
|
||||
"authd seed 不完整,缺失:%s" % ",".join(missing))
|
||||
from urllib.parse import urlparse
|
||||
gitea_host = (urlparse(seed["GITEA_URL"]).hostname or "").lower()
|
||||
if not gitea_host:
|
||||
raise ValueError("authd seed 的 GITEA_URL 主机解析不出")
|
||||
vault = CredentialVault()
|
||||
vault.store("gitea", "GITEA_TOKEN", seed["GITEA_TOKEN"],
|
||||
host=gitea_host, repo=seed["SYNC_REPO"])
|
||||
extra = [h.strip().lower()
|
||||
for h in (os.environ.get("CHV_SENTINEL_ALLOW_HOSTS")
|
||||
or "").split(",") if h.strip()]
|
||||
policy = SentinelPolicy(
|
||||
allow_hosts={gitea_host} | set(extra))
|
||||
uids = allowed_uids
|
||||
if uids is None:
|
||||
raw = os.environ.get("CHV_AUTHD_ALLOWED_UIDS") or ""
|
||||
uids = [int(p) for p in raw.split(",") if p.strip()] \
|
||||
if raw.strip() else list(DEFAULT_ALLOWED_UIDS)
|
||||
server = AuthdServer(socket_path, vault, policy,
|
||||
seed["GITEA_USER"], uids, log_path,
|
||||
resolve=resolve, socket_group=socket_group)
|
||||
return server
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
"""前台运行(systemd Type=simple 直接跑本进程,不 daemonize)。"""
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--seed", default=DEFAULT_SEED_PATH)
|
||||
parser.add_argument("--socket", default=DEFAULT_SOCKET_PATH)
|
||||
parser.add_argument("--log", default=DEFAULT_LOG_PATH)
|
||||
parser.add_argument("--lib", default=DEFAULT_LIB_DIR)
|
||||
args = parser.parse_args(argv)
|
||||
server = build_server(seed_path=args.seed,
|
||||
socket_path=args.socket,
|
||||
log_path=args.log, lib_dir=args.lib)
|
||||
print("chv-authd: listening on %s" % args.socket,
|
||||
file=sys.stderr)
|
||||
try:
|
||||
server.serve_forever()
|
||||
except KeyboardInterrupt:
|
||||
pass
|
||||
finally:
|
||||
server.server_close()
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,14 @@
|
||||
"""chv-authd 叶子树的 stub cloud/chv/__init__.py(round-11 真机修复,层 1)。
|
||||
|
||||
真实的 cloud/chv/__init__.py import executor/orphans(进而要整套
|
||||
cloud 包),拷进 authd 的最小叶子树(/usr/local/lib/chv-authd)
|
||||
会让 chv-authd 在 import 时 ModuleNotFoundError 而 crash loop:
|
||||
先是 cloud.chv.errors 缺失,补上 errors.py 后又会在 executor
|
||||
上失败。探针 fixprobe3 的形状:叶子树里只放 errors.py + 仅导出
|
||||
ChvFailure 的 stub __init__.py——authd 只需要这个名字。
|
||||
build.sh 拷本文件作叶子树的 cloud/chv/__init__.py,不拷真实的
|
||||
包 __init__。
|
||||
"""
|
||||
from cloud.chv.errors import ChvFailure
|
||||
|
||||
__all__ = ["ChvFailure"]
|
||||
+306
-60
@@ -1,16 +1,49 @@
|
||||
#!/bin/bash
|
||||
# chv-guest 镜像构建脚本(补缺口轨:从 TODO 骨架改成真实步骤)。
|
||||
# chv-guest 轻量镜像构建脚本(2026-11-06 用户定:停掉 5.5G 全量版,
|
||||
# 按轻量口径重开;不再对齐 Manus,不做三面板/多租户/生产级全套)。
|
||||
#
|
||||
# 底座:Ubuntu Minimal 24.04 官方 qcow2(248MB,
|
||||
# https://cloud-images.ubuntu.com/minimal/releases/noble/release/
|
||||
# ubuntu-24.04-minimal-cloudimg-amd64.img——minimal 发布在 noble
|
||||
# 代号目录下,不是 releases/24.04)。只装 codex 0.164.0-alpha.9
|
||||
# linux-x64 + Python + SSH + cloud-init + 一个薄客机 agent
|
||||
# (/app/host.py,仅此一个文件——host.py 只用标准库,不依赖
|
||||
# cloud/;旧版把整个 $REPO_DIR/cloud/ 拷进 /app/,客内 agent
|
||||
# 可读全套 server/worker 代码,M4 收尾已收紧为最小集)。
|
||||
# 浏览器:Playwright +
|
||||
# chromium-headless-shell(不要完整 Chromium);字体只要最小的
|
||||
# fonts-wqy-microhei(中文能看),不要全套 fonts-noto-cjk。
|
||||
# 目标:qcow2 实际占用 ≤1GB,raw ≤2GB(见下注)。
|
||||
# 控制面只留最小的:现有 cloud/chv/executor.py 起停 VM + SSH
|
||||
# 进去调 codex + Playwright 截图。
|
||||
#
|
||||
# Gitea 双向同步(用户工作区口径,2026-11-06 追加):镜像里加
|
||||
# git + gt 1.0.0(官方 install.sh + SHA256 校验,跟 170/172
|
||||
# 同款),配好 git user,不含真实 token,不设 credential helper
|
||||
#(round-11 真机修复层 5:store helper 已去掉,authd helper 是
|
||||
# 唯一凭证 helper,且自身绝不落盘);客机启动时 chv-sync boot
|
||||
# 按 seed 盘 /etc/chv/guest-env(0600)把用户选定的仓 clone 到
|
||||
# ~/workspace,后续 pull/push 由用户显式同步,分叉/冲突时
|
||||
# 失败关闭(不 --force、不自动 rebase)。token 绝不进镜像、
|
||||
# 代码、日志。
|
||||
#
|
||||
# 对齐 Muse P2(2026-11-07):镜像里加 systemd-nspawn 运行时单元
|
||||
#(chv-cell:codex 跑在单元内)+ 单元外 broker(chv-broker:
|
||||
#Unix socket + SO_PEERCRED 白名单代办)+ 浏览器子代理(chv-browser:
|
||||
#单元外独立进程,只返回无障碍树快照)。boot clone 仍在单元外做;
|
||||
#单元隔离口径见 docs/chv-muse-p2-gap.md。
|
||||
#
|
||||
# 用法(必须在带 /dev/kvm 的宿主上跑全量模式):
|
||||
# CODEX_BIN_DIR=bin UBUNTU_VERSION=24.04 CODEX_VERSION=0.164.0-alpha.9 \
|
||||
# REPO_COMMIT=$(git rev-parse --short HEAD) ./build.sh out/
|
||||
# REPO_COMMIT=$(git rev-parse --short HEAD) \
|
||||
# UBUNTU_IMAGE_SHA256=<minimal 镜像 sha256> ./build.sh out/
|
||||
# ./build.sh --dry-run out/ # 只打印命令,不执行、不要求 KVM
|
||||
#
|
||||
# 产物(out/ 内):
|
||||
# vmlinux # virt-get-kernel 从云镜像取出的内核(CHV --kernel 指向它)
|
||||
# disk.raw # qemu-img 转成的 raw 根文件系统镜像(CHV_ROOTFS 指向它)
|
||||
# BUILDINFO.json # {tag, ubuntu, codex_version, repo_commit, built_at, contents}
|
||||
# + versions {dpkg, pip}(构建时从客机读回的实际版本)+
|
||||
# + versions {dpkg, pip, chromium, gt}(构建时从客机读回的实际版本)+
|
||||
# ubuntu_image_sha256_set(SHA256 必填,见下)
|
||||
# 绝不进镜像的东西:任何模型 key、系统 key、用户账号数据、
|
||||
# 生产凭证;模型 key 由客机 cloud-init 预置的运行环境变量名
|
||||
@@ -19,14 +52,30 @@
|
||||
# sshd 加固由构建写入 /etc/ssh/sshd_config.d/99-chv.conf
|
||||
# 并经 sshd -t + grep 断言(README 的“只允许 key 登录”
|
||||
# 从此是构建断言,不再只是文档声称)。
|
||||
# B-10:virt-customize 的 chroot 里 sshd -t 须先 ssh-keygen -A
|
||||
# 临时生成 host key 并 mkdir -p /run/sshd,否则分别报无 host
|
||||
# key / 缺 privilege separation 目录而失败;断言通过后立即删
|
||||
# 临时 key,镜像仍不带 host key(per-VM 预置才生效)。
|
||||
#
|
||||
# 客机 host key 硬性要求(见 README §5,补缺口轨新增):
|
||||
# 客机必须装 cloud-init 且支持 NoCloud seed,首启先应用
|
||||
# seed 再启动 sshd,且不得无条件重新生成 host key,否则
|
||||
# 执行器的 per-VM 预置钉死即失效。本脚本在镜像里写
|
||||
# /etc/cloud/cloud.cfg.d/99-chv-hostkey.cfg(ssh_deletekeys:
|
||||
# false、ssh_genkeytypes: []),把“不重生成”钉进镜像。
|
||||
set -eu
|
||||
# /etc/cloud/cloud.cfg.d/99-chv-hostkey.cfg:ssh_deletekeys:
|
||||
# false(绝不删已有 key)+ 首启 bootcmd 只在缺 key 时
|
||||
# ssh-keygen -A(bootcmd 跑在 seed write_files 之前,预置
|
||||
# key 随后覆盖生成值,最终仍是预置 key)。
|
||||
# B-23:曾写 ssh_genkeytypes: [],彻底关掉首启生成;Minimal
|
||||
# 底座 cloud.cfg 又不含 ssh 模块,客机无 host key、sshd -t
|
||||
# 失败、ssh.service 连败,SSH 全程不通——[] 已删,绝不再写。
|
||||
#
|
||||
# 注:Minimal 底座虚拟盘 3.5G(根分区 2.4G,实测 1.9G 空闲),
|
||||
# qemu-img convert 出的 disk.raw 表观 3.5G、实际占用约 0.6G
|
||||
# (稀疏)。raw 表观 ≤2G 须在构建后另做一步收窄(guestfish
|
||||
# 缩根文件系统 + sgdisk 重建 1 号分区 + qemu-img resize
|
||||
# --shrink),不在本脚本内做——收窄前先确认装完后根分区实际
|
||||
# 占用,避免把镜像收到无法启动;qcow2 压缩产物是首要验收项。
|
||||
set -euo pipefail
|
||||
|
||||
DRY_RUN=0
|
||||
if [ "${1:-}" = "--dry-run" ]; then
|
||||
@@ -40,8 +89,64 @@ OUT="${1:-out}"
|
||||
: "${REPO_COMMIT:?REPO_COMMIT 未设:构建时仓库 commit 短 hash}"
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
REPO_DIR="${REPO_DIR:-$(cd "$SCRIPT_DIR/.." && pwd)}"
|
||||
# gt 1.0.0:官方 install.sh(gitea.com/gitea/cli,main 分支
|
||||
# scripts/install.sh),脚本本身下载 dl.gitea.com 的二进制并
|
||||
# 验 SHA256;此处再验脚本自身的 SHA256。2026-11-07 06:44
|
||||
# 上游轮换过一次(V-9 实测:WSL fresh curl 与失败镜像内下载残留
|
||||
# 逐字节一致,新 sha 49ffe090…;新脚本全文已读,只从 dl.gitea.com
|
||||
# 取 1.0.0 二进制并验 hash,行为干净)——旧值 cf82… 已失效,
|
||||
# 现钉死新值;上游再轮换时构建会再次明确失败,不静默跑新脚本。
|
||||
GT_VERSION="${GT_VERSION:-1.0.0}"
|
||||
# feat-chv-closeup(M4):Chromium 单独钉死。PLAYWRIGHT=1 时
|
||||
# ms-playwright 缓存目录名即 Chromium 版本标识(如
|
||||
# chromium_headless_shell-1248),此前只记录不钉死——playwright
|
||||
# 一升级,目录名跟着变,构建静默换浏览器版本。
|
||||
# CHROMIUM_HEADLESS_SHELL_REV 设为纯数字修订号(如 1248)即钉死:
|
||||
# buildinfo.py 校验读回的目录名须恰好是该修订号,否则构建失败。
|
||||
# 未设时只记录、不强制,并在构建输出里明说“未钉死”,绝不假装
|
||||
# 已钉死;真机 V-9 验收须用已设该变量的构建。
|
||||
CHROMIUM_HEADLESS_SHELL_REV="${CHROMIUM_HEADLESS_SHELL_REV:-}"
|
||||
if [ -n "$CHROMIUM_HEADLESS_SHELL_REV" ] && ! printf '%s' "$CHROMIUM_HEADLESS_SHELL_REV" | grep -Eq '^[0-9]+$'; then
|
||||
echo "CHROMIUM_HEADLESS_SHELL_REV 须为纯数字修订号(如 1248):$CHROMIUM_HEADLESS_SHELL_REV" >&2; exit 1
|
||||
fi
|
||||
GT_INSTALL_SH_SHA256="${GT_INSTALL_SH_SHA256:-49ffe0909be9117e36cf7c8a35b5937939639090322063cbad7667e0bde7c9e9}"
|
||||
# B-03/B-04:输入格式校验(干跑与全量都做,报错点提前,
|
||||
# 不等到 curl 404 或 sha256sum -c 才失败)。
|
||||
if ! printf '%s' "$UBUNTU_VERSION" | grep -Eq '^[0-9]+\.[0-9]+$'; then
|
||||
echo "UBUNTU_VERSION 格式须为 NN.NN:$UBUNTU_VERSION" >&2; exit 1
|
||||
fi
|
||||
if ! printf '%s' "$REPO_COMMIT" | grep -Eq '^[0-9a-f]{7,40}$'; then
|
||||
echo "REPO_COMMIT 须为 7-40 位 hex:$REPO_COMMIT" >&2; exit 1
|
||||
fi
|
||||
if ! printf '%s' "$CODEX_VERSION" | grep -Eq '^[0-9]+\.[0-9]+\.[0-9]+'; then
|
||||
echo "CODEX_VERSION 格式异常:$CODEX_VERSION" >&2; exit 1
|
||||
fi
|
||||
if ! printf '%s' "$GT_VERSION" | grep -Eq '^[0-9]+\.[0-9]+\.[0-9]+$'; then
|
||||
echo "GT_VERSION 格式异常:$GT_VERSION" >&2; exit 1
|
||||
fi
|
||||
if ! printf '%s' "$GT_INSTALL_SH_SHA256" | grep -Eq '^[0-9a-f]{64}$'; then
|
||||
echo "GT_INSTALL_SH_SHA256 须为 64 位 hex" >&2; exit 1
|
||||
fi
|
||||
if [ -n "${UBUNTU_IMAGE_SHA256:-}" ] && ! printf '%s' "$UBUNTU_IMAGE_SHA256" | grep -Eq '^[0-9a-f]{64}$'; then
|
||||
echo "UBUNTU_IMAGE_SHA256 须为 64 位 hex" >&2; exit 1
|
||||
fi
|
||||
# B-09:版本声称须核对,不只进 tag。codex 二进制真实版本
|
||||
# 须包含 CODEX_VERSION;REPO_COMMIT 须等于仓库 HEAD 短 hash;
|
||||
# 工作树脏即拒绝构建(tag 会谎称干净 commit)。
|
||||
CODEX_ACTUAL_VERSION="$("$CODEX_BIN_DIR/codex" --version 2>/dev/null || true)"
|
||||
case "$CODEX_ACTUAL_VERSION" in
|
||||
*"$CODEX_VERSION"*) ;;
|
||||
*) echo "codex 二进制版本与 CODEX_VERSION 不符:got [$CODEX_ACTUAL_VERSION] want 含 [$CODEX_VERSION]" >&2; exit 1 ;;
|
||||
esac
|
||||
HEAD_SHORT="$(git -C "$REPO_DIR" rev-parse --short HEAD)"
|
||||
if [ "$HEAD_SHORT" != "$REPO_COMMIT" ]; then
|
||||
echo "REPO_COMMIT 与仓库 HEAD 不符:$REPO_COMMIT != $HEAD_SHORT" >&2; exit 1
|
||||
fi
|
||||
if [ -n "$(git -C "$REPO_DIR" status --porcelain)" ]; then
|
||||
echo "仓库工作树有未提交修改,拒绝构建(tag 会谎称干净 commit)" >&2; exit 1
|
||||
fi
|
||||
TAG="chv-guest-${UBUNTU_VERSION}-${CODEX_VERSION}-${REPO_COMMIT}"
|
||||
UBUNTU_IMAGE_URL="${UBUNTU_IMAGE_URL:-https://cloud-images.ubuntu.com/releases/${UBUNTU_VERSION}/release/ubuntu-${UBUNTU_VERSION}-server-cloudimg-amd64.img}"
|
||||
UBUNTU_IMAGE_URL="${UBUNTU_IMAGE_URL:-https://cloud-images.ubuntu.com/minimal/releases/noble/release/ubuntu-${UBUNTU_VERSION}-minimal-cloudimg-amd64.img}"
|
||||
|
||||
run() {
|
||||
if [ "$DRY_RUN" = "1" ]; then
|
||||
@@ -63,10 +168,43 @@ if [ ! -d "$CODEX_BIN_DIR/codex-resources" ]; then
|
||||
echo "缺少 codex-resources(bwrap 载体),构建失败" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [ ! -f "$REPO_DIR/host.py" ] || [ ! -d "$REPO_DIR/cloud" ]; then
|
||||
echo "REPO_DIR 不是 codex-app-server 仓库根(缺 host.py/cloud/):$REPO_DIR" >&2
|
||||
if [ ! -f "$REPO_DIR/host.py" ]; then
|
||||
echo "REPO_DIR 不是 codex-app-server 仓库根(缺 host.py):$REPO_DIR" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [ ! -f "$SCRIPT_DIR/chv-sync.sh" ]; then
|
||||
echo "缺少 chv-sync.sh(Gitea 双向同步脚本):$SCRIPT_DIR" >&2
|
||||
exit 1
|
||||
fi
|
||||
# 对齐 Muse P2:单元启动器 / broker / 浏览器子代理三脚本缺一即失败。
|
||||
for p2_script in chv-cell.py chv-broker.py chv-browser.py; do
|
||||
if [ ! -f "$SCRIPT_DIR/$p2_script" ]; then
|
||||
echo "缺少 $p2_script(Muse P2 客机脚本):$SCRIPT_DIR" >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
# 对齐 Muse P3:authd 三件套缺一即失败(与 build.sh 同目录,P2
|
||||
# 同款规矩)。云端叶子模块(cloud/chv/authd.py 与 sentinel.py)
|
||||
# 不做硬要求:M4 既定“/app 最小集、build.sh 不再要求 cloud/”
|
||||
#(test_chv_residual.BuildScriptMinimalAppTest 钉住),缺失时只
|
||||
# 警告、不失败——缺叶子模块的镜像里 chv-authd 起不来,chv-sync
|
||||
# 按 P3 差距表 §3.5 回落旧落盘路径并警告,绝不静默假装有 authd。
|
||||
# round-11 真机修复(层 1):叶子树还须 errors.py + stub
|
||||
# __init__(authd_init_stub.py)——真实的 cloud/chv/__init__.py
|
||||
# import executor/orphans,拷进最小叶子树会让 chv-authd
|
||||
# ModuleNotFoundError 而 crash loop(探针 fixprobe3 的形状)。
|
||||
for p3_file in authd.py git-credential-chv-authd chv-authd.service authd_init_stub.py; do
|
||||
if [ ! -f "$SCRIPT_DIR/$p3_file" ]; then
|
||||
echo "缺少 $p3_file(Muse P3 客机文件):$SCRIPT_DIR" >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
P3_LIBS_PRESENT=0
|
||||
if [ -f "$REPO_DIR/cloud/chv/authd.py" ] && [ -f "$REPO_DIR/cloud/chv/sentinel.py" ] && [ -f "$REPO_DIR/cloud/chv/errors.py" ] && [ -f "$REPO_DIR/cloud/__init__.py" ] && [ -f "$REPO_DIR/cloud/chv/__init__.py" ]; then
|
||||
P3_LIBS_PRESENT=1
|
||||
else
|
||||
echo "警告:Muse P3 云端叶子模块不在仓库里(cloud/chv/authd.py 与 sentinel.py),镜像暂不含 authd 叶子模块" >&2
|
||||
fi
|
||||
if [ "$DRY_RUN" = "0" ]; then
|
||||
if [ ! -e /dev/kvm ]; then
|
||||
echo "CHV_GUEST_BUILD needs a KVM host: /dev/kvm 不存在" >&2
|
||||
@@ -81,7 +219,16 @@ if [ "$DRY_RUN" = "0" ]; then
|
||||
fi
|
||||
|
||||
run mkdir -p "$OUT"
|
||||
IMG="$OUT/ubuntu-${UBUNTU_VERSION}-server-cloudimg-amd64.img"
|
||||
# B-07:磁盘空间预检(仅全量模式;干跑不建目录,无法 df)。
|
||||
# 轻量版实际占用约 2G(底座 248M + disk.raw 稀疏约 0.6G +
|
||||
# 内核约 46M + 构建余量),阈值从全量版 20G 降到 8G。
|
||||
if [ "$DRY_RUN" = "0" ]; then
|
||||
avail_kb="$(df -k --output=avail "$OUT" | tail -1 | tr -d ' ')"
|
||||
if [ "${avail_kb:-0}" -lt 8388608 ]; then
|
||||
echo "磁盘不足:$OUT 所在盘可用 ${avail_kb}KB,需 ≥8GB" >&2; exit 1
|
||||
fi
|
||||
fi
|
||||
IMG="$OUT/ubuntu-${UBUNTU_VERSION}-minimal-cloudimg-amd64.img"
|
||||
# feat-chv-secfix(M4):云镜像 SHA256 改为必填(干跑只验
|
||||
# “已设”,不验文件;全量模式下载后验哈希)。未设即失败,
|
||||
# 不再只依赖 HTTPS 传输层作唯一锚。
|
||||
@@ -90,38 +237,148 @@ if [ -z "${UBUNTU_IMAGE_SHA256:-}" ]; then
|
||||
exit 1
|
||||
fi
|
||||
if [ "$DRY_RUN" = "1" ]; then
|
||||
echo "DRY-RUN: curl -fL $UBUNTU_IMAGE_URL -o $IMG"
|
||||
echo "DRY-RUN: curl -fL --retry 3 --retry-all-errors --connect-timeout 30 $UBUNTU_IMAGE_URL -o $IMG"
|
||||
echo "DRY-RUN: [ -s $IMG ] || exit 1"
|
||||
echo "DRY-RUN: sha256sum -c (UBUNTU_IMAGE_SHA256)"
|
||||
else
|
||||
if [ ! -s "$IMG" ]; then
|
||||
curl -fL "$UBUNTU_IMAGE_URL" -o "$IMG"
|
||||
curl -fL --retry 3 --retry-all-errors --connect-timeout 30 "$UBUNTU_IMAGE_URL" -o "$IMG"
|
||||
fi
|
||||
[ -s "$IMG" ] || { echo "下载的云镜像为空:$IMG" >&2; exit 1; }
|
||||
# B-06:哈希不符时删文件重下一次,不直接退出让操作员手动清理。
|
||||
if ! echo "$UBUNTU_IMAGE_SHA256 $IMG" | sha256sum -c -; then
|
||||
echo "云镜像哈希不符,删除后重下一次:$IMG" >&2
|
||||
rm -f "$IMG"
|
||||
curl -fL --retry 3 --retry-all-errors --connect-timeout 30 "$UBUNTU_IMAGE_URL" -o "$IMG"
|
||||
[ -s "$IMG" ] || { echo "重下的云镜像为空:$IMG" >&2; exit 1; }
|
||||
echo "$UBUNTU_IMAGE_SHA256 $IMG" | sha256sum -c -
|
||||
fi
|
||||
run qemu-img convert -f qcow2 -O raw "$IMG" "$OUT/disk.raw"
|
||||
# 清单软件(照抄 cloud/docker/Dockerfile 的 apt/pip 清单,
|
||||
# 中文 LaTeX xelatex+CJK 故意不装,见 README;Playwright
|
||||
# 默认不装,PLAYWRIGHT=1 时才加装,属可选步骤)。
|
||||
run virt-customize -a "$OUT/disk.raw" \
|
||||
--install git,pandoc,poppler-utils,latexmk,texlive-latex-base,openssh-server,cloud-init,qemu-guest-agent,python3,python3-pip \
|
||||
--run-command 'pip3 install --no-cache-dir --break-system-packages python-docx openpyxl python-pptx PyMuPDF Pillow || pip3 install --no-cache-dir python-docx openpyxl python-pptx PyMuPDF Pillow' \
|
||||
--copy-in "$CODEX_BIN_DIR/codex:/usr/local/bin/" \
|
||||
--copy-in "$CODEX_BIN_DIR/codex-code-mode-host:/usr/local/bin/" \
|
||||
--copy-in "$CODEX_BIN_DIR/codex-resources:/usr/local/bin/" \
|
||||
--run-command 'chmod +x /usr/local/bin/codex /usr/local/bin/codex-code-mode-host; ln -sf /usr/local/bin/codex-resources/bwrap /usr/local/bin/bwrap' \
|
||||
--mkdir /app \
|
||||
--copy-in "$REPO_DIR/host.py:/app/" \
|
||||
--copy-in "$REPO_DIR/cloud:/app/" \
|
||||
--run-command 'printf "ssh_deletekeys: false\nssh_genkeytypes: []\n" > /etc/cloud/cloud.cfg.d/99-chv-hostkey.cfg; systemctl enable ssh || true' \
|
||||
--run-command 'printf "PasswordAuthentication no\nPermitRootLogin prohibit-password\nChallengeResponseAuthentication no\nKbdInteractiveAuthentication no\nPubkeyAuthentication yes\n" > /etc/ssh/sshd_config.d/99-chv.conf; sshd -t && grep -q "^PasswordAuthentication no" /etc/ssh/sshd_config.d/99-chv.conf && grep -q "^PermitRootLogin prohibit-password" /etc/ssh/sshd_config.d/99-chv.conf' \
|
||||
--run-command 'dpkg-query -W -f "${Package} ${Version}\n" > /app/guest-versions-dpkg.txt; pip3 freeze > /app/guest-versions-pip.txt || true'
|
||||
if [ "${PLAYWRIGHT:-0}" = "1" ]; then
|
||||
run virt-customize -a "$OUT/disk.raw" \
|
||||
--run-command 'pip3 install --no-cache-dir playwright; python3 -m playwright install --with-deps chromium'
|
||||
fi
|
||||
run virt-get-kernel -a "$OUT/disk.raw" -o "$OUT/vmlinux"
|
||||
run qemu-img convert -f qcow2 -O raw "$IMG" "$OUT/disk.raw"
|
||||
# 轻量版不扩盘:Minimal 底座根分区 2.4G、实测 1.9G 空闲,
|
||||
# 轻量清单(见下)装完仍有余量;全量版 B-16 的 +12G 扩盘
|
||||
# 已删除——扩盘会把 raw 表观顶到 16G,与 raw ≤2G 目标直接冲突。
|
||||
# B-21:apt 必须 --no-install-recommends。virt-customize
|
||||
# --install 默认装 Recommends,python3-pip 会连带拉入
|
||||
# python3-dev → gcc/g++/build-essential 全套工具链,根分区
|
||||
# 2.4G 被顶到 96%(实测 2.2G/95M 空闲),Playwright 下载
|
||||
# headless-shell 到 70% 时 ENOSPC 失败(round 6)。改走
|
||||
# --run-command apt-get install --no-install-recommends,
|
||||
# 并把会被 Recommends 带进来的必需项显式列出:fontconfig
|
||||
# (字体发现,没有它 Chromium 找不到中文字体)、openssh-client
|
||||
# (git 的 Recommends,agent 任务可能要 ssh 出站)。
|
||||
# apt-get clean 只清 .deb 缓存、不动 lists(playwright
|
||||
# --with-deps 稍后还要 apt 装浏览器依赖);lists 在最后一步清。
|
||||
# B-22:virt-customize 的客机 shell 里 HOME=/,Playwright 默认
|
||||
# 缓存路径 ~/.cache/ms-playwright 会落到根目录 /.cache(round 7
|
||||
# 实测:浏览器装在 /.cache/ms-playwright/chromium_headless_shell-1248,
|
||||
# /root/.cache 根本不存在,ls 读回空文件、BUILDINFO.versions.chromium
|
||||
# 为空,check_lockfile 兜底报未覆盖)。改走系统路径
|
||||
# PLAYWRIGHT_BROWSERS_PATH=/usr/local/share/ms-playwright:构建时
|
||||
# 装这里、读回这里;运行时 agent 也须设同一变量,否则 chv 用户
|
||||
# 会在自己家目录下找不到浏览器。
|
||||
# Playwright 默认开启(PLAYWRIGHT=0 可显式关掉),浏览器只装
|
||||
# chromium-headless-shell(不要完整 Chromium)。单次
|
||||
# virt-customize:apt 安装、pip、拷入、sshd 加固、git 占位、
|
||||
# gt 安装、chv-sync、版本读回都在同一次定制里按参数顺序跑;
|
||||
# Playwright 装在第一次 freeze 之后、最终 freeze 之前(B-08:
|
||||
# 最终 freeze 须在 Playwright 安装之后,否则
|
||||
# BUILDINFO.versions.pip 里没有 playwright;Chromium 版本以
|
||||
# ms-playwright 缓存目录名记录(如 chromium_headless_shell-1248),
|
||||
# 写进 /app/guest-versions-chromium.txt)。
|
||||
GT_INSTALL_CMD="curl -fsSL -o /tmp/gt-install.sh https://gitea.com/gitea/cli/raw/branch/main/scripts/install.sh && echo \"${GT_INSTALL_SH_SHA256} /tmp/gt-install.sh\" | sha256sum -c - && GITEA_CLI_VERSION=${GT_VERSION} GITEA_CLI_INSTALL_DIR=/usr/local/bin sh /tmp/gt-install.sh && rm -f /tmp/gt-install.sh && /usr/local/bin/gt version"
|
||||
VC_ARGS=(-a "$OUT/disk.raw"
|
||||
--run-command 'apt-get update -qq && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends git openssh-server openssh-client cloud-init python3 python3-pip curl ca-certificates xz-utils fonts-wqy-microhei fontconfig systemd-container iproute2 iptables && apt-get clean'
|
||||
--copy-in "$CODEX_BIN_DIR/codex:/usr/local/bin/"
|
||||
--copy-in "$CODEX_BIN_DIR/codex-code-mode-host:/usr/local/bin/"
|
||||
--copy-in "$CODEX_BIN_DIR/codex-resources:/usr/local/bin/"
|
||||
--run-command 'chmod +x /usr/local/bin/codex /usr/local/bin/codex-code-mode-host; ln -sf /usr/local/bin/codex-resources/bwrap /usr/local/bin/bwrap'
|
||||
--mkdir /app
|
||||
--copy-in "$REPO_DIR/host.py:/app/"
|
||||
--copy-in "$SCRIPT_DIR/chv-sync.sh:/usr/local/bin/"
|
||||
--run-command 'mv /usr/local/bin/chv-sync.sh /usr/local/bin/chv-sync && chmod +x /usr/local/bin/chv-sync'
|
||||
--copy-in "$SCRIPT_DIR/chv-cell.py:/usr/local/bin/"
|
||||
--copy-in "$SCRIPT_DIR/chv-broker.py:/usr/local/bin/"
|
||||
--copy-in "$SCRIPT_DIR/chv-browser.py:/usr/local/bin/"
|
||||
--run-command 'mv /usr/local/bin/chv-cell.py /usr/local/bin/chv-cell && mv /usr/local/bin/chv-broker.py /usr/local/bin/chv-broker && mv /usr/local/bin/chv-browser.py /usr/local/bin/chv-browser && chmod +x /usr/local/bin/chv-cell /usr/local/bin/chv-broker /usr/local/bin/chv-browser'
|
||||
--mkdir /usr/local/lib/chv-authd
|
||||
--mkdir /usr/local/lib/chv-authd/cloud
|
||||
--mkdir /usr/local/lib/chv-authd/cloud/chv
|
||||
--copy-in "$SCRIPT_DIR/authd.py:/usr/local/bin/"
|
||||
--copy-in "$SCRIPT_DIR/git-credential-chv-authd:/usr/local/bin/"
|
||||
--copy-in "$SCRIPT_DIR/chv-authd.service:/etc/systemd/system/"
|
||||
--run-command 'mv /usr/local/bin/authd.py /usr/local/bin/chv-authd && chmod +x /usr/local/bin/chv-authd /usr/local/bin/git-credential-chv-authd'
|
||||
--run-command 'systemctl enable chv-authd || true'
|
||||
--run-command 'printf "runcmd:\n - systemctl restart ssh || service ssh restart || true\n - systemctl start chv-authd || true\n - test -f /etc/chv/guest-env && chown chv:chv /etc/chv/guest-env || true\n - /usr/local/bin/chv-sync boot >>/var/log/chv-sync.log 2>&1 || true\n" > /etc/cloud/cloud.cfg.d/99-chv-sync.cfg'
|
||||
--run-command 'git config --system user.name chv && git config --system user.email chv@localhost && git config --system init.defaultBranch main'
|
||||
--run-command "$GT_INSTALL_CMD"
|
||||
--run-command '/usr/local/bin/gt version > /app/guest-versions-gt.txt'
|
||||
--run-command 'printf "ssh_deletekeys: false\nbootcmd:\n - \"[ -f /etc/ssh/ssh_host_ed25519_key ] || ssh-keygen -A\"\n" > /etc/cloud/cloud.cfg.d/99-chv-hostkey.cfg; systemctl enable ssh || true'
|
||||
--run-command 'printf "PasswordAuthentication no\nPermitRootLogin prohibit-password\nChallengeResponseAuthentication no\nKbdInteractiveAuthentication no\nPubkeyAuthentication yes\n" > /etc/ssh/sshd_config.d/99-chv.conf && mkdir -p /run/sshd && ssh-keygen -A && sshd -t && grep -q "^PasswordAuthentication no" /etc/ssh/sshd_config.d/99-chv.conf && grep -q "^PermitRootLogin prohibit-password" /etc/ssh/sshd_config.d/99-chv.conf && rm -f /etc/ssh/ssh_host_* && test -z "$(ls /etc/ssh/ssh_host_* 2>/dev/null)"'
|
||||
--run-command 'dpkg-query -W -f "\${Package} \${Version}\n" > /app/guest-versions-dpkg.txt; pip3 freeze > /app/guest-versions-pip.txt || true'
|
||||
--run-command 'useradd --system --home-dir /var/lib/chv-browser --shell /usr/sbin/nologin chv-browser && mkdir -p /var/lib/chv-browser /run/chv/shots /run/chv-browser && chown chv-browser:chv-browser /var/lib/chv-browser /run/chv/shots /run/chv-browser && chmod 0755 /run/chv /run/chv/shots /run/chv-browser'
|
||||
--run-command 'printf "[Unit]\nDescription=CHV cell broker (Muse P2)\nAfter=network.target\n\n[Service]\nType=simple\nExecStart=/usr/bin/python3 /usr/local/bin/chv-broker\nRestart=on-failure\nRestartSec=2\n\n[Install]\nWantedBy=multi-user.target\n" > /etc/systemd/system/chv-broker.service'
|
||||
--run-command 'printf "[Unit]\nDescription=CHV browser sub-agent (Muse P2)\nAfter=network.target\n\n[Service]\nType=simple\nUser=chv-browser\nGroup=chv-browser\nExecStart=/usr/bin/python3 /usr/local/bin/chv-browser\nRestart=on-failure\nRestartSec=2\n\n[Install]\nWantedBy=multi-user.target\n" > /etc/systemd/system/chv-browser.service'
|
||||
--run-command 'printf "d /run/chv 0755 root root -\nd /run/chv/shots 0755 chv-browser chv-browser -\nd /run/chv-browser 0755 chv-browser chv-browser -\n" > /etc/tmpfiles.d/chv.conf && systemctl enable chv-broker chv-browser || true')
|
||||
# P3 交接(Muse P3 的 authd/sentinel 叶子模块):文件在仓库里存在时
|
||||
# 拷入镜像(本分支已有,合入后自动生效);不存在时只警告、不失败——
|
||||
# M4 既定 build.sh 不硬要求 cloud/,见上。缺叶子模块时 chv-authd
|
||||
# 守护进程起不来,chv-sync 按差距表 §3.5 回落旧路径并警告。
|
||||
# round-11 真机修复(层 1):叶子树是 errors.py + stub __init__
|
||||
#(authd_init_stub.py),不拷真实的 cloud/chv/__init__.py(它
|
||||
# import executor/orphans,会让 authd 在最小叶子树里
|
||||
# ModuleNotFoundError 而 crash loop);cloud/__init__.py 本身只
|
||||
# 有 docstring,照常拷。
|
||||
if [ "$P3_LIBS_PRESENT" = "1" ]; then
|
||||
VC_ARGS+=(--copy-in "$REPO_DIR/cloud/chv/authd.py:/usr/local/lib/chv-authd/cloud/chv/"
|
||||
--copy-in "$REPO_DIR/cloud/chv/sentinel.py:/usr/local/lib/chv-authd/cloud/chv/"
|
||||
--copy-in "$REPO_DIR/cloud/chv/errors.py:/usr/local/lib/chv-authd/cloud/chv/"
|
||||
--copy-in "$REPO_DIR/cloud/__init__.py:/usr/local/lib/chv-authd/cloud/"
|
||||
--copy-in "$SCRIPT_DIR/authd_init_stub.py:/usr/local/lib/chv-authd/cloud/chv/"
|
||||
--run-command 'mv /usr/local/lib/chv-authd/cloud/chv/authd_init_stub.py /usr/local/lib/chv-authd/cloud/chv/__init__.py')
|
||||
fi
|
||||
# P1 交接(Muse P1 的 vsock_notify.py):文件在仓库里存在时拷入
|
||||
# 镜像并建 vsock-notify.service(自动启动,round-11 真机修复:
|
||||
# 此前只拷文件、不建服务,vsock 通道从未自动启动;服务形状按
|
||||
# vsock_notify.py 的 docstring:ExecStart=/usr/local/bin/
|
||||
# vsock-notify %H,Restart=no,After/WantedBy=multi-user.target;
|
||||
# printf 格式串里 %H 须写 %%H);不存在时只警告、不失败。
|
||||
VSOCK_NOTIFY_PRESENT=0
|
||||
if [ -f "$REPO_DIR/cloud/chv/guest/vsock_notify.py" ]; then
|
||||
VC_ARGS+=(--copy-in "$REPO_DIR/cloud/chv/guest/vsock_notify.py:/usr/local/bin/"
|
||||
--run-command 'mv /usr/local/bin/vsock_notify.py /usr/local/bin/vsock-notify && chmod +x /usr/local/bin/vsock-notify'
|
||||
--run-command 'printf "[Unit]\nDescription=CHV vsock notify (Muse P1)\nAfter=multi-user.target\n\n[Service]\nType=simple\nExecStart=/usr/local/bin/vsock-notify %%H\nRestart=no\n\n[Install]\nWantedBy=multi-user.target\n" > /etc/systemd/system/vsock-notify.service && systemctl enable vsock-notify || true')
|
||||
VSOCK_NOTIFY_PRESENT=1
|
||||
else
|
||||
echo "警告:cloud/chv/guest/vsock_notify.py 不在仓库里(P1 未合并),镜像暂不含 vsock notify" >&2
|
||||
fi
|
||||
if [ "${PLAYWRIGHT:-1}" = "1" ]; then
|
||||
VC_ARGS+=(
|
||||
--run-command '(pip3 install --no-cache-dir --break-system-packages playwright || pip3 install --no-cache-dir playwright) && PLAYWRIGHT_BROWSERS_PATH=/usr/local/share/ms-playwright python3 -m playwright install --with-deps chromium-headless-shell'
|
||||
--run-command 'dpkg-query -W -f "\${Package} \${Version}\n" > /app/guest-versions-dpkg.txt; pip3 freeze > /app/guest-versions-pip.txt || true'
|
||||
--run-command 'ls /usr/local/share/ms-playwright/ > /app/guest-versions-chromium.txt 2>/dev/null || true')
|
||||
fi
|
||||
# B-21 收尾:清掉两轮 apt 的缓存与 lists(--with-deps 装完后
|
||||
# 不再有 apt 活动),把空间还给镜像。
|
||||
VC_ARGS+=(
|
||||
--run-command 'apt-get clean && rm -rf /var/lib/apt/lists/*')
|
||||
run virt-customize "${VC_ARGS[@]}"
|
||||
# B-18:virt-get-kernel 的 -o 是输出目录(不是文件),它在
|
||||
# 目录里下载 vmlinuz-<ver> 与 initrd.img-<ver>;旧写法
|
||||
# -o "$OUT/vmlinux" 会建出 vmlinux 目录,后面的 test -s 文件
|
||||
# 校验必然失败。改成 -o "$OUT",再把 vmlinuz-* 移为 vmlinux
|
||||
# 文件;initrd.img-* 留在 OUT 里作配套产物。另:第三轮真机
|
||||
# 构建里 virt-get-kernel 首跑报过一次 "file receive cancelled
|
||||
# by daemon",同一盘手动重跑即成功(瞬时故障),此处失败重跑
|
||||
# 一次,与 B-05/B-06 的重试口径一致。
|
||||
if [ "$DRY_RUN" = "1" ]; then
|
||||
echo "DRY-RUN: virt-get-kernel -a $OUT/disk.raw -o $OUT"
|
||||
echo "DRY-RUN: mv $OUT/vmlinuz-* $OUT/vmlinux"
|
||||
else
|
||||
run virt-get-kernel -a "$OUT/disk.raw" -o "$OUT" \
|
||||
|| run virt-get-kernel -a "$OUT/disk.raw" -o "$OUT"
|
||||
mv "$OUT"/vmlinuz-* "$OUT/vmlinux"
|
||||
fi
|
||||
# feat-chv-secfix(M4):构建时把客机内实际装好的版本
|
||||
# 读回 BUILDINFO.versions(dpkg + pip freeze),名字列表
|
||||
# 之外再有逐项版本——可审计、可复现比对。事前 lockfile
|
||||
@@ -131,43 +388,32 @@ run virt-get-kernel -a "$OUT/disk.raw" -o "$OUT/vmlinux"
|
||||
if [ "$DRY_RUN" = "1" ]; then
|
||||
echo "DRY-RUN: virt-cat -a $OUT/disk.raw /app/guest-versions-dpkg.txt > $OUT/guest-versions-dpkg.txt"
|
||||
echo "DRY-RUN: virt-cat -a $OUT/disk.raw /app/guest-versions-pip.txt > $OUT/guest-versions-pip.txt"
|
||||
echo "DRY-RUN: virt-cat -a $OUT/disk.raw /app/guest-versions-gt.txt > $OUT/guest-versions-gt.txt"
|
||||
if [ "${PLAYWRIGHT:-1}" = "1" ]; then
|
||||
echo "DRY-RUN: virt-cat -a $OUT/disk.raw /app/guest-versions-chromium.txt > $OUT/guest-versions-chromium.txt"
|
||||
fi
|
||||
else
|
||||
virt-cat -a "$OUT/disk.raw" /app/guest-versions-dpkg.txt > "$OUT/guest-versions-dpkg.txt"
|
||||
virt-cat -a "$OUT/disk.raw" /app/guest-versions-pip.txt > "$OUT/guest-versions-pip.txt"
|
||||
virt-cat -a "$OUT/disk.raw" /app/guest-versions-gt.txt > "$OUT/guest-versions-gt.txt" || true
|
||||
if [ "${PLAYWRIGHT:-1}" = "1" ]; then
|
||||
virt-cat -a "$OUT/disk.raw" /app/guest-versions-chromium.txt > "$OUT/guest-versions-chromium.txt" || true
|
||||
fi
|
||||
fi
|
||||
# 产物校验(干跑只打印,不执行校验命令的副作用部分)。
|
||||
if [ "$DRY_RUN" = "1" ]; then
|
||||
echo "DRY-RUN: test -s $OUT/vmlinux && test -s $OUT/disk.raw"
|
||||
echo "DRY-RUN: python3 生成 $OUT/BUILDINFO.json 并校验 tag=$TAG"
|
||||
echo "DRY-RUN: python3 $SCRIPT_DIR/buildinfo.py $OUT $TAG $UBUNTU_VERSION $CODEX_VERSION $REPO_COMMIT <sha256> ${PLAYWRIGHT:-1} ${CHROMIUM_HEADLESS_SHELL_REV:-} [vsock_notify?] # 生成 $OUT/BUILDINFO.json 并校验 tag=$TAG"
|
||||
else
|
||||
test -s "$OUT/vmlinux" || { echo "内核产物为空:$OUT/vmlinux" >&2; exit 1; }
|
||||
test -s "$OUT/disk.raw" || { echo "磁盘产物为空:$OUT/disk.raw" >&2; exit 1; }
|
||||
python3 - "$OUT/BUILDINFO.json" "$TAG" "$UBUNTU_VERSION" "$CODEX_VERSION" "$REPO_COMMIT" <<'PYEOF'
|
||||
import json, sys, time
|
||||
path, tag, ubuntu, codex, commit = sys.argv[1:6]
|
||||
def _read_lines(p):
|
||||
try:
|
||||
with open(p) as fh:
|
||||
return [l.strip() for l in fh if l.strip()]
|
||||
except OSError:
|
||||
return []
|
||||
import os
|
||||
outdir = os.path.dirname(path)
|
||||
info = {"tag": tag, "ubuntu": ubuntu, "codex_version": codex,
|
||||
"repo_commit": commit, "built_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
|
||||
"contents": ["git", "pandoc", "poppler-utils", "latexmk", "texlive-latex-base",
|
||||
"openssh-server", "cloud-init", "qemu-guest-agent",
|
||||
"python-docx", "openpyxl", "python-pptx", "PyMuPDF", "Pillow",
|
||||
"codex", "codex-code-mode-host", "codex-resources/bwrap", "/app/host.py"],
|
||||
"versions": {
|
||||
"dpkg": _read_lines(os.path.join(outdir, "guest-versions-dpkg.txt")),
|
||||
"pip": _read_lines(os.path.join(outdir, "guest-versions-pip.txt")),
|
||||
},
|
||||
"ubuntu_image_sha256_set": True}
|
||||
with open(path, "w") as fh:
|
||||
json.dump(info, fh, indent=2)
|
||||
PYEOF
|
||||
python3 -c "import json; d=json.load(open('$OUT/BUILDINFO.json')); assert d['tag'] == '$TAG', d; assert d['versions']['dpkg'], 'BUILDINFO versions.dpkg 为空'; assert d['versions']['pip'], 'BUILDINFO versions.pip 为空'"
|
||||
if [ "${PLAYWRIGHT:-1}" = "1" ] && [ -z "$CHROMIUM_HEADLESS_SHELL_REV" ]; then
|
||||
echo "警告:CHROMIUM_HEADLESS_SHELL_REV 未设,Chromium 版本只记录、不钉死(V-9 验收须设该变量重建)" >&2
|
||||
fi
|
||||
EXTRA_CONTENTS=""
|
||||
[ "$P3_LIBS_PRESENT" = "1" ] && EXTRA_CONTENTS="cloud/chv/authd.py,cloud/chv/sentinel.py,cloud/chv/errors.py,cloud/chv/__init__.py"
|
||||
[ "$VSOCK_NOTIFY_PRESENT" = "1" ] && EXTRA_CONTENTS="${EXTRA_CONTENTS:+$EXTRA_CONTENTS,}vsock_notify"
|
||||
python3 "$SCRIPT_DIR/buildinfo.py" "$OUT" "$TAG" "$UBUNTU_VERSION" "$CODEX_VERSION" "$REPO_COMMIT" "$UBUNTU_IMAGE_SHA256" "${PLAYWRIGHT:-1}" "${CHROMIUM_HEADLESS_SHELL_REV:-}" "$EXTRA_CONTENTS"
|
||||
fi
|
||||
echo "tag=$TAG"
|
||||
echo "out=$OUT"
|
||||
|
||||
@@ -0,0 +1,190 @@
|
||||
#!/usr/bin/env python3
|
||||
"""生成 CHV 客机镜像的 BUILDINFO.json(build.sh 调用)。
|
||||
|
||||
从 build.sh 抽出(B-02),使其可独立单元测试。输入全部是
|
||||
已落盘的构建产物与读回文件,不编造版本号:versions 来自
|
||||
客机读回的 dpkg-query / pip freeze(Playwright=1 时另有
|
||||
ms-playwright 缓存目录名作 chromium 标识);artifacts 的
|
||||
sha256/size 是对产物文件实测的;image_sha256 即 disk.raw
|
||||
的 sha256,供 generate_lockfile.py 沿用。
|
||||
|
||||
用法:
|
||||
python3 buildinfo.py OUT_DIR TAG UBUNTU CODEX_VERSION \\
|
||||
REPO_COMMIT UBUNTU_IMAGE_SHA256 PLAYWRIGHT(0|1)
|
||||
[CHROMIUM_HEADLESS_SHELL_REV]
|
||||
|
||||
CHROMIUM_HEADLESS_SHELL_REV(可选,纯数字):钉死的
|
||||
ms-playwright 缓存修订号(如 1248)。PLAYWRIGHT=1 且
|
||||
给了该值时,versions.chromium 里须恰好出现
|
||||
chromium_headless_shell-<rev>,否则退出 1、构建失败;
|
||||
未给时只记录、不强制,BUILDINFO.chromium_pinned_rev
|
||||
记 null,构建输出由 build.sh 明说“未钉死”。
|
||||
"""
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
|
||||
# 轻量口径(2026-11-06 用户定):Ubuntu Minimal 底座,只装
|
||||
# codex + Python + SSH + cloud-init + 薄 agent + git + gt +
|
||||
# 最小中文字体;已裁剪 pandoc/texlive/python-docx 等文档工具
|
||||
# 与 qemu-guest-agent。chv-sync 是仓库内脚本,版本随 repo_commit。
|
||||
BASE_CONTENTS = [
|
||||
"git", "openssh-server", "openssh-client", "cloud-init",
|
||||
"python3", "python3-pip", "curl", "ca-certificates",
|
||||
"xz-utils", "fonts-wqy-microhei", "fontconfig", "codex",
|
||||
"codex-code-mode-host", "codex-resources/bwrap",
|
||||
"/app/host.py", "gt", "chv-sync",
|
||||
# 对齐 Muse P2:nspawn 单元启动器 + broker + 浏览器子代理。
|
||||
"systemd-container", "chv-cell", "chv-broker", "chv-browser",
|
||||
# 对齐 Muse P3(round-11 合入镜像):authd 守护进程 + git
|
||||
# credential helper + systemd 单元(三件套是无条件拷入的)。
|
||||
# 云端叶子模块(cloud/chv/authd.py、sentinel.py)不进 BASE:
|
||||
# 它们是条件拷入(缺失只警告、不失败),存在时才经
|
||||
# extra_contents 登记,绝不在缺失的构建里谎称已进镜像。
|
||||
"chv-authd", "git-credential-chv-authd", "chv-authd.service",
|
||||
]
|
||||
|
||||
# 执行器默认内核 cmdline(config.py),构建产物须与它一起记录(B-12),
|
||||
# 否则换宿主/换执行器配置时无法复现。
|
||||
KERNEL_CMDLINE = "console=hvc0 root=/dev/vda1 rw"
|
||||
|
||||
# 浏览器装在系统路径(B-22),不是某用户的 ~/.cache:构建时
|
||||
# PLAYWRIGHT_BROWSERS_PATH 指这里,agent 运行时也须设同一变量。
|
||||
PLAYWRIGHT_BROWSERS_PATH = "/usr/local/share/ms-playwright"
|
||||
|
||||
|
||||
def _read_lines(path):
|
||||
try:
|
||||
with open(path) as fh:
|
||||
return [line.strip() for line in fh if line.strip()]
|
||||
except OSError:
|
||||
return []
|
||||
|
||||
|
||||
def _sha256_size(path):
|
||||
digest = hashlib.sha256()
|
||||
size = 0
|
||||
with open(path, "rb") as fh:
|
||||
for chunk in iter(lambda: fh.read(1 << 20), b""):
|
||||
digest.update(chunk)
|
||||
size += len(chunk)
|
||||
return digest.hexdigest(), size
|
||||
|
||||
|
||||
def build(outdir, tag, ubuntu, codex_version, repo_commit,
|
||||
ubuntu_image_sha256, playwright, chromium_pinned_rev=None,
|
||||
extra_contents=None):
|
||||
disk_path = os.path.join(outdir, "disk.raw")
|
||||
vmlinux_path = os.path.join(outdir, "vmlinux")
|
||||
disk_sha, disk_size = _sha256_size(disk_path)
|
||||
vmlinux_sha, vmlinux_size = _sha256_size(vmlinux_path)
|
||||
contents = list(BASE_CONTENTS)
|
||||
# extra_contents:构建时才确定的附加内容(如 P1 的 vsock_notify,
|
||||
# 文件存在才拷入)。调用方传什么记什么,不编默认值。
|
||||
for item in extra_contents or []:
|
||||
if item and item not in contents:
|
||||
contents.append(item)
|
||||
versions = {
|
||||
"dpkg": _read_lines(
|
||||
os.path.join(outdir, "guest-versions-dpkg.txt")),
|
||||
"pip": _read_lines(
|
||||
os.path.join(outdir, "guest-versions-pip.txt")),
|
||||
"gt": _read_lines(
|
||||
os.path.join(outdir, "guest-versions-gt.txt")),
|
||||
}
|
||||
gt_version = ""
|
||||
for line in versions["gt"]:
|
||||
m = re.search(r"(\d+\.\d+\.\d+)", line)
|
||||
if m:
|
||||
gt_version = m.group(1)
|
||||
break
|
||||
if playwright:
|
||||
contents += ["playwright", "chromium-headless-shell"]
|
||||
versions["chromium"] = _read_lines(
|
||||
os.path.join(outdir, "guest-versions-chromium.txt"))
|
||||
info = {
|
||||
"tag": tag,
|
||||
"ubuntu": ubuntu,
|
||||
"codex_version": codex_version,
|
||||
"repo_commit": repo_commit,
|
||||
"gt_version": gt_version,
|
||||
"built_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
|
||||
"contents": contents,
|
||||
"versions": versions,
|
||||
"ubuntu_image_sha256": ubuntu_image_sha256,
|
||||
"ubuntu_image_sha256_set": True,
|
||||
"kernel_cmdline": KERNEL_CMDLINE,
|
||||
"playwright": bool(playwright),
|
||||
"playwright_browsers_path": PLAYWRIGHT_BROWSERS_PATH
|
||||
if playwright else "",
|
||||
# feat-chv-closeup(M4):钉死的 Chromium 修订号(未设
|
||||
# 记 null,绝不编一个默认值出来假装已钉死)。
|
||||
"chromium_pinned_rev": chromium_pinned_rev or None,
|
||||
"image_sha256": disk_sha,
|
||||
"artifacts": {
|
||||
"disk.raw": {"sha256": disk_sha, "size": disk_size},
|
||||
"vmlinux": {"sha256": vmlinux_sha, "size": vmlinux_size},
|
||||
},
|
||||
}
|
||||
path = os.path.join(outdir, "BUILDINFO.json")
|
||||
with open(path, "w") as fh:
|
||||
json.dump(info, fh, indent=2)
|
||||
fh.write("\n")
|
||||
return info
|
||||
|
||||
|
||||
def main(argv):
|
||||
if len(argv) not in (8, 9, 10):
|
||||
print("用法:buildinfo.py OUT TAG UBUNTU CODEX_VERSION "
|
||||
"REPO_COMMIT UBUNTU_IMAGE_SHA256 PLAYWRIGHT "
|
||||
"[CHROMIUM_HEADLESS_SHELL_REV] [EXTRA_CONTENTS]", file=sys.stderr)
|
||||
return 2
|
||||
_, outdir, tag, ubuntu, codex_version, repo_commit, \
|
||||
ubuntu_image_sha256, playwright = argv[:8]
|
||||
chromium_pinned_rev = argv[8] if len(argv) >= 9 else None
|
||||
# 第 10 参数:逗号分隔的附加内容(如 vsock_notify),可空。
|
||||
extra_contents = None
|
||||
if len(argv) == 10 and argv[9]:
|
||||
extra_contents = [s for s in argv[9].split(",") if s]
|
||||
if chromium_pinned_rev and not re.fullmatch(r"[0-9]+",
|
||||
chromium_pinned_rev):
|
||||
print("CHROMIUM_HEADLESS_SHELL_REV 须为纯数字:%s"
|
||||
% chromium_pinned_rev, file=sys.stderr)
|
||||
return 2
|
||||
info = build(outdir, tag, ubuntu, codex_version, repo_commit,
|
||||
ubuntu_image_sha256, playwright == "1",
|
||||
chromium_pinned_rev, extra_contents)
|
||||
if not info["versions"]["dpkg"]:
|
||||
print("BUILDINFO versions.dpkg 为空", file=sys.stderr)
|
||||
return 1
|
||||
if not info["versions"]["pip"]:
|
||||
print("BUILDINFO versions.pip 为空", file=sys.stderr)
|
||||
return 1
|
||||
if not info["gt_version"]:
|
||||
print("BUILDINFO gt_version 为空(gt 未装好)", file=sys.stderr)
|
||||
return 1
|
||||
if playwright == "1" and not info["versions"].get("chromium"):
|
||||
print("BUILDINFO versions.chromium 为空(浏览器未装好或"
|
||||
"读回路径不对)", file=sys.stderr)
|
||||
return 1
|
||||
if playwright == "1" and chromium_pinned_rev:
|
||||
# feat-chv-closeup(M4):钉死校验——读回的缓存目录名
|
||||
# 须恰好是该修订号,playwright 升级导致目录名变了就
|
||||
# 在这里失败,不静默换浏览器版本。
|
||||
expected = "chromium_headless_shell-%s" % chromium_pinned_rev
|
||||
if expected not in (info["versions"].get("chromium") or []):
|
||||
print("Chromium 修订号与钉死值不符:读回 %s,钉死 %s"
|
||||
% (info["versions"].get("chromium"), expected),
|
||||
file=sys.stderr)
|
||||
return 1
|
||||
if info["tag"] != tag:
|
||||
print("BUILDINFO tag 不一致", file=sys.stderr)
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main(sys.argv))
|
||||
+112
-12
@@ -1,13 +1,14 @@
|
||||
#!/usr/bin/env python3
|
||||
"""校验已构建镜像的 BUILDINFO 供应链清单(事后 lockfile 检查)。
|
||||
|
||||
事前 apt/pip 精确 lockfile 无法在本轨生成:apt 版本在构建
|
||||
宿主上解析、pip 版本在客机里解析,没有构建宿主就编不出
|
||||
版本号——本脚本绝不编造。构建时 build.sh 已从客机读回
|
||||
dpkg + pip freeze 写进 BUILDINFO.versions(事后 lockfile),
|
||||
本脚本校验它:versions 非空、tag 与预期一致、可选的冻结
|
||||
清单(--frozen JSON,形如 {"dpkg": ["pkg version", ...]})
|
||||
逐项比对,漂移即退出非 0。
|
||||
生成端见 generate_lockfile.py(读 BUILDINFO.versions 生成
|
||||
lockfile.json,版本唯一来源是 build.sh 从客机读回的
|
||||
dpkg/pip freeze + chromium 目录名 + codex/repo_commit/gt
|
||||
声明字段,绝不查构建宿主现场、绝不编造版本号)。
|
||||
本脚本校验 BUILDINFO:versions 非空、每条有版本、contents
|
||||
全覆盖、tag 与预期一致、钉死的 Chromium 修订号一致、
|
||||
可选的冻结清单(--frozen JSON,形如
|
||||
{"dpkg": ["pkg version", ...]})双向比对,漂移即退出非 0。
|
||||
|
||||
用法:
|
||||
python3 check_lockfile.py out/BUILDINFO.json [--tag TAG]
|
||||
@@ -16,6 +17,68 @@ dpkg + pip freeze 写进 BUILDINFO.versions(事后 lockfile),
|
||||
import json
|
||||
import sys
|
||||
|
||||
# 非包名的内容项:版本来自 BUILDINFO 的声明字段,不是 versions 列表。
|
||||
DECLARED_SOURCES = {
|
||||
"codex": "codex_version",
|
||||
"codex-code-mode-host": "codex_version",
|
||||
"codex-resources/bwrap": "codex_version",
|
||||
"/app/host.py": "repo_commit",
|
||||
"gt": "gt_version",
|
||||
"chv-sync": "repo_commit",
|
||||
# 对齐 Muse P2:nspawn 单元启动器 + broker + 浏览器子代理,
|
||||
# 均是仓库内脚本,版本随 repo_commit(与 chv-sync 同款)。
|
||||
"chv-cell": "repo_commit",
|
||||
"chv-broker": "repo_commit",
|
||||
"chv-browser": "repo_commit",
|
||||
# 对齐 Muse P3 + P1 vsock:authd 三件套、authd/sentinel
|
||||
# 叶子模块与 vsock 通知脚本,均是仓库内脚本,版本随
|
||||
# repo_commit。缺了它们 check 会报 contents 未覆盖(717efc4
|
||||
# 真机实测:6 项 unresolved,verify 与 check 双双失败)。
|
||||
# round-11 真机修复(层 1):叶子树新增 errors.py 与 stub
|
||||
# __init__.py(authd_init_stub.py),同款 repo_commit 登记。
|
||||
"chv-authd": "repo_commit",
|
||||
"git-credential-chv-authd": "repo_commit",
|
||||
"chv-authd.service": "repo_commit",
|
||||
"cloud/chv/authd.py": "repo_commit",
|
||||
"cloud/chv/sentinel.py": "repo_commit",
|
||||
"cloud/chv/errors.py": "repo_commit",
|
||||
"cloud/chv/__init__.py": "repo_commit",
|
||||
"vsock_notify": "repo_commit",
|
||||
}
|
||||
|
||||
|
||||
def _names(entries, pip_style=False):
|
||||
"""versions 条目 → 小写名集合。dpkg 行是 "name version",
|
||||
pip freeze 行是 "name==version"。"""
|
||||
names = set()
|
||||
for entry in entries or []:
|
||||
text = str(entry)
|
||||
if pip_style and "==" in text:
|
||||
names.add(text.split("==", 1)[0].lower())
|
||||
else:
|
||||
parts = text.split(None, 1)
|
||||
if parts:
|
||||
names.add(parts[0].lower())
|
||||
return names
|
||||
|
||||
|
||||
def _check_format(kind, entries):
|
||||
"""每条须带非空版本:dpkg "name version",pip "name==version"。"""
|
||||
for entry in entries or []:
|
||||
text = str(entry)
|
||||
if kind == "pip" and "==" in text:
|
||||
if not text.split("==", 1)[1].strip():
|
||||
print("版本为空的条目:%s %r" % (kind, entry),
|
||||
file=sys.stderr)
|
||||
return False
|
||||
else:
|
||||
parts = text.split()
|
||||
if len(parts) < 2 or not parts[1]:
|
||||
print("版本为空的条目:%s %r" % (kind, entry),
|
||||
file=sys.stderr)
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def main(argv):
|
||||
if len(argv) < 2:
|
||||
@@ -47,20 +110,57 @@ def main(argv):
|
||||
print("BUILDINFO.versions.dpkg/pip 为空,供应链清单缺失",
|
||||
file=sys.stderr)
|
||||
return 1
|
||||
if not _check_format("dpkg", versions.get("dpkg")):
|
||||
return 1
|
||||
if not _check_format("pip", versions.get("pip")):
|
||||
return 1
|
||||
# contents 全覆盖:每项须在 dpkg/pip 名单里,或有声明字段版本。
|
||||
covered = _names(versions.get("dpkg")) | _names(
|
||||
versions.get("pip"), pip_style=True)
|
||||
missing = []
|
||||
for name in data.get("contents") or []:
|
||||
if name.lower() in covered:
|
||||
continue
|
||||
source = DECLARED_SOURCES.get(name)
|
||||
if source and data.get(source):
|
||||
continue
|
||||
if name.lower() in ("chromium", "chromium-headless-shell") \
|
||||
and versions.get("chromium"):
|
||||
continue
|
||||
missing.append(name)
|
||||
if missing:
|
||||
print("contents 未覆盖:%s" % sorted(missing), file=sys.stderr)
|
||||
return 1
|
||||
# feat-chv-closeup(M4):钉死的 Chromium 修订号须与读回
|
||||
# 的缓存目录名一致;未钉(pinned_rev 为空)时不校验。
|
||||
pinned = data.get("chromium_pinned_rev")
|
||||
if pinned:
|
||||
expected = "chromium_headless_shell-%s" % pinned
|
||||
if expected not in (versions.get("chromium") or []):
|
||||
print("Chromium 修订号与钉死值不符:读回 %s,钉死 %s"
|
||||
% (versions.get("chromium"), expected),
|
||||
file=sys.stderr)
|
||||
return 1
|
||||
if frozen_path:
|
||||
with open(frozen_path) as fh:
|
||||
frozen = json.load(fh)
|
||||
for kind in ("dpkg", "pip"):
|
||||
want = set(frozen.get(kind) or [])
|
||||
got = set(versions.get(kind) or [])
|
||||
missing = want - got
|
||||
if missing:
|
||||
absent = want - got
|
||||
if absent:
|
||||
print("%s 漂移:冻结清单里有、构建产物里无:%s"
|
||||
% (kind, sorted(missing)), file=sys.stderr)
|
||||
% (kind, sorted(absent)), file=sys.stderr)
|
||||
return 1
|
||||
print("lockfile 校验通过:tag=%s dpkg=%d pip=%d"
|
||||
extra = got - want
|
||||
if extra:
|
||||
print("%s 漂移:构建产物里有、冻结清单里无:%s"
|
||||
% (kind, sorted(extra)), file=sys.stderr)
|
||||
return 1
|
||||
print("lockfile 校验通过:tag=%s dpkg=%d pip=%d contents=%d"
|
||||
% (data.get("tag"), len(versions["dpkg"]),
|
||||
len(versions["pip"])))
|
||||
len(versions["pip"]),
|
||||
len(data.get("contents") or [])))
|
||||
return 0
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
[Unit]
|
||||
Description=CHV authd (credential broker for chv-sync)
|
||||
# 注意时序:本服务须在 chv-sync boot 之前就绪。cloud-final 的
|
||||
# runcmd(99-chv-sync.cfg)在 chv-sync boot 之前显式
|
||||
# `systemctl start chv-authd`;这里不依赖 After=cloud-final
|
||||
#(cloud-final 跑完才 After 得到它,太晚)。enable 后开机自启
|
||||
# 只是兜底,首启顺序以 runcmd 的显式 start 为准。
|
||||
After=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
# authd 跑在运行时单元之外(root 身份):它是凭证的唯一持有者,
|
||||
# 单元内的 agent 只能经 Unix socket + SO_PEERCRED 向它要凭证,
|
||||
# 读不到它的内存与 seed 文件(P2 单元隔离落地后)。
|
||||
ExecStart=/usr/bin/python3 /usr/local/bin/chv-authd
|
||||
Restart=on-failure
|
||||
RestartSec=2
|
||||
StandardOutput=journal
|
||||
StandardError=journal
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -0,0 +1,336 @@
|
||||
#!/usr/bin/env python3
|
||||
"""chv-broker:单元外 IPC broker(客机侧、单元外服务)。
|
||||
|
||||
对齐 Muse P2(见 docs/chv-muse-p2-gap.md):单元内外走 Unix socket +
|
||||
SO_PEERCRED 的最小权限 IPC。broker 只认白名单操作,凭证与浏览器
|
||||
都在单元外——单元看不见 token,只能请 broker 代办。
|
||||
|
||||
校验(双重,缺一不可才放行;无 cell pid 文件时只做 uid 检查并打日志):
|
||||
1. SO_PEERCRED 的 uid 须在单元映射范围 [1000, 1000+65536) 内;
|
||||
2. 对端 pid 的 pid ns 须与任一活着的单元 init 同 ns(part 4:
|
||||
并发 cell 各有 cell-<machine>.pid,旧的单文件 cell.pid 只记
|
||||
最新单元;读不到任何 pid 文件时跳过第 2 项并警告;有 pid 文
|
||||
件但对端 ns 与任一可读的单元 ns 都对不上时拒绝,不降级)。
|
||||
|
||||
操作白名单:
|
||||
ping -> {"ok": true}
|
||||
sync {action: pull|push|status, repo?} -> 单元外以 chv 身份跑
|
||||
chv-sync(list argv,不进 shell;仓目录只允许
|
||||
~/workspace 下,realpath + commonpath 校验)。
|
||||
browser {…} -> 原样转发给 chv-browser.sock(浏览器子代理),
|
||||
回包原样返回;浏览器服务不在时报错,不替它跑 playwright。
|
||||
|
||||
只用标准库。单连接一请求一响应(JSON 行,≤64KB);异常只断本
|
||||
连接,绝不杀死 broker 循环。
|
||||
"""
|
||||
import glob
|
||||
import grp
|
||||
import json
|
||||
import os
|
||||
import socket
|
||||
import struct
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
BROKER_SOCK = "/run/chv/broker.sock"
|
||||
# round-11 真机修复(缺陷 4):browser.sock 从 /run/chv 移到
|
||||
# chv-browser 自有目录——/run/chv 是 0755 root root,
|
||||
# chv-browser 在 bind 时 EACCES 而无限 crash loop(restart 21);
|
||||
# 放进组可写的 /run/chv 会给 chv-browser 对 broker.sock/cell.pid
|
||||
# 的 unlink 能力,所以用独立目录(0755 chv-browser 所有)。
|
||||
BROWSER_SOCK = "/run/chv-browser/browser.sock"
|
||||
CELL_PID_FILE = "/run/chv/cell.pid"
|
||||
CELL_PID_GLOB = "/run/chv/cell-*.pid"
|
||||
UID_BASE = 1000
|
||||
UID_COUNT = 65536
|
||||
WORKSPACE = "/home/chv/workspace"
|
||||
SYNC_USER = "chv"
|
||||
SYNC_BIN = "/usr/local/bin/chv-sync"
|
||||
MAX_REQUEST_BYTES = 65536
|
||||
MAX_OUTPUT_BYTES = 8192
|
||||
|
||||
SYNC_ACTIONS = ("pull", "push", "status")
|
||||
|
||||
|
||||
def _read_ns_link(pid, proc_root="/proc"):
|
||||
"""读 /proc/<pid>/ns/pid 的链接目标;读不到返回 None。"""
|
||||
try:
|
||||
return os.readlink(os.path.join(proc_root, str(pid), "ns", "pid"))
|
||||
except OSError:
|
||||
return None
|
||||
|
||||
|
||||
def peer_allowed(peer_uid, peer_pid, cell_pid=None, proc_root="/proc"):
|
||||
"""对端是否允许用 broker。返回 (allowed, reason)。
|
||||
cell_pid 为 None 时只做 uid 检查(broker 打日志警告,调用方负责)。"""
|
||||
if not (UID_BASE <= peer_uid < UID_BASE + UID_COUNT):
|
||||
return False, "uid %d 不在单元映射范围 [%d, %d)" % (
|
||||
peer_uid, UID_BASE, UID_BASE + UID_COUNT)
|
||||
if cell_pid is None:
|
||||
return True, "uid 在范围内(cell.pid 缺失,未做 pid ns 对照)"
|
||||
cell_ns = _read_ns_link(cell_pid, proc_root)
|
||||
if cell_ns is None:
|
||||
return False, "cell.pid 的 pid ns 读不到,拒绝(不降级)"
|
||||
peer_ns = _read_ns_link(peer_pid, proc_root)
|
||||
if peer_ns is None:
|
||||
return False, "对端 pid ns 读不到,拒绝"
|
||||
if peer_ns != cell_ns:
|
||||
return False, "对端 pid ns 与单元不一致"
|
||||
return True, "ok"
|
||||
|
||||
|
||||
def peer_allowed_multi(peer_uid, peer_pid, cell_pids, proc_root="/proc"):
|
||||
"""多 cell 版:对端 pid ns 与任一活 cell 的 init 同 ns 即放行
|
||||
(part 4 并发 cell 用;单 cell 时与 peer_allowed 等价)。
|
||||
cell_pids 为空即只做 uid 检查;全部 pid 的 ns 都读不到时
|
||||
拒绝(不降级);个别 pid 的 ns 读不到(已退出的残留文件)
|
||||
只跳过该项,不因此拒绝整个请求。"""
|
||||
if not (UID_BASE <= peer_uid < UID_BASE + UID_COUNT):
|
||||
return False, "uid %d 不在单元映射范围 [%d, %d)" % (
|
||||
peer_uid, UID_BASE, UID_BASE + UID_COUNT)
|
||||
if not cell_pids:
|
||||
# part 5:reason 文案与单 cell 版 peer_allowed 对齐,保留
|
||||
# “cell.pid 缺失”字样——P2-9 驱动 grep 该串认降级日志;
|
||||
# part 4 切到 multi 版时把这里改写成“无 cell pid 文件”,
|
||||
# 该串自此再也没进过 journal,是 RETEST4
|
||||
# degraded_logged=False 的根因(不是时序、不是日志路径)。
|
||||
return True, "uid 在范围内(cell.pid 缺失,未做 pid ns 对照)"
|
||||
peer_ns = _read_ns_link(peer_pid, proc_root)
|
||||
if peer_ns is None:
|
||||
return False, "对端 pid ns 读不到,拒绝"
|
||||
readable = 0
|
||||
for cell_pid in cell_pids:
|
||||
cell_ns = _read_ns_link(cell_pid, proc_root)
|
||||
if cell_ns is None:
|
||||
continue
|
||||
readable += 1
|
||||
if peer_ns == cell_ns:
|
||||
return True, "ok"
|
||||
if readable == 0:
|
||||
return False, "cell pid 的 pid ns 均读不到,拒绝(不降级)"
|
||||
return False, "对端 pid ns 与任何单元不一致"
|
||||
|
||||
|
||||
def _read_cell_pid():
|
||||
try:
|
||||
with open(CELL_PID_FILE) as fh:
|
||||
return int(fh.read().strip())
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def _read_cell_pids(cell_dir="/run/chv"):
|
||||
"""读全部 cell pid:每单元一份 cell-<machine>.pid + 兼容旧
|
||||
路径 cell.pid(最新单元)。去重、跳过坏内容,返回 int 列表。"""
|
||||
pids = []
|
||||
paths = sorted(glob.glob(os.path.join(cell_dir, "cell-*.pid")))
|
||||
paths.append(os.path.join(cell_dir, "cell.pid"))
|
||||
for path in paths:
|
||||
try:
|
||||
with open(path) as fh:
|
||||
pid = int(fh.read().strip())
|
||||
except (OSError, ValueError):
|
||||
continue
|
||||
if pid not in pids:
|
||||
pids.append(pid)
|
||||
return pids
|
||||
|
||||
|
||||
def _degraded_warning(reason, pid, uid):
|
||||
"""降级警告行:reason 原样进日志。serve() 在无 cell pid 文件
|
||||
时打这条;P2-9 驱动 grep 'cell.pid 缺失' 认的就是它(part 5:
|
||||
part 4 之前 serve 直接打印含该串的警告行,RETEST3 同项为
|
||||
True;part 4 切到 peer_allowed_multi 后警告行改写成
|
||||
“无 cell pid 文件”,该串再也没进过 journal——RETEST4 的
|
||||
degraded_logged=False 即由此来,不是时序、不是日志路径
|
||||
变化。reason 与警告行分开写曾让两者悄悄漂移,今后改文案
|
||||
只改 peer_allowed_multi 的 reason,警告行自动跟随)。"""
|
||||
return ("chv-broker: 警告:%s (pid=%d uid=%d)"
|
||||
% (reason, pid, uid))
|
||||
|
||||
|
||||
def _resolve_repo(repo):
|
||||
"""仓目录只允许 ~/workspace 下。repo 为空时返回 None(由
|
||||
chv-sync 自己取唯一仓)。返回解析后的绝对路径或抛 ValueError。"""
|
||||
if not repo:
|
||||
return None
|
||||
base = os.path.realpath(WORKSPACE)
|
||||
target = os.path.realpath(repo)
|
||||
if os.path.commonpath([base, target]) != base or target == base:
|
||||
raise ValueError("仓目录只允许 %s 下:%s" % (WORKSPACE, repo))
|
||||
return target
|
||||
|
||||
|
||||
def _truncate(text, limit=MAX_OUTPUT_BYTES):
|
||||
if text is None:
|
||||
return ""
|
||||
if len(text) > limit:
|
||||
return text[:limit] + "\n…(已截断)"
|
||||
return text
|
||||
|
||||
|
||||
def _run_sync(action, repo):
|
||||
if action not in SYNC_ACTIONS:
|
||||
return {"ok": False,
|
||||
"error": "sync action 只允许 %s:%s"
|
||||
% ("/".join(SYNC_ACTIONS), action)}
|
||||
try:
|
||||
repo_path = _resolve_repo(repo)
|
||||
except ValueError as e:
|
||||
return {"ok": False, "error": str(e)}
|
||||
cmd = ["runuser", "-u", SYNC_USER, "--", "env", "HOME=/home/chv",
|
||||
SYNC_BIN, action]
|
||||
if repo_path:
|
||||
cmd.append(repo_path)
|
||||
try:
|
||||
proc = subprocess.run(cmd, capture_output=True, text=True,
|
||||
timeout=300)
|
||||
except subprocess.TimeoutExpired:
|
||||
return {"ok": False, "error": "chv-sync 超时(300 秒)"}
|
||||
except OSError as e:
|
||||
return {"ok": False, "error": "chv-sync 起不来:%s" % e}
|
||||
return {"ok": proc.returncode == 0,
|
||||
"stdout": _truncate(proc.stdout),
|
||||
"stderr": _truncate(proc.stderr),
|
||||
"returncode": proc.returncode}
|
||||
|
||||
|
||||
def _forward_browser(req, sock_path=BROWSER_SOCK, timeout=120):
|
||||
"""转发给浏览器子代理;回包须是带 ok 的 dict,否则报错。"""
|
||||
try:
|
||||
conn = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
|
||||
conn.settimeout(timeout)
|
||||
conn.connect(sock_path)
|
||||
except OSError as e:
|
||||
return {"ok": False,
|
||||
"error": "浏览器子代理未就绪:%s" % e}
|
||||
try:
|
||||
conn.sendall((json.dumps(req, ensure_ascii=False) + "\n").encode())
|
||||
data = b""
|
||||
while len(data) < MAX_REQUEST_BYTES:
|
||||
chunk = conn.recv(4096)
|
||||
if not chunk:
|
||||
break
|
||||
data += chunk
|
||||
if data.endswith(b"\n"):
|
||||
break
|
||||
except OSError as e:
|
||||
return {"ok": False, "error": "浏览器子代理通信失败:%s" % e}
|
||||
finally:
|
||||
conn.close()
|
||||
try:
|
||||
resp = json.loads(data.decode())
|
||||
except (ValueError, UnicodeDecodeError):
|
||||
return {"ok": False, "error": "浏览器子代理回包不是 JSON"}
|
||||
if not isinstance(resp, dict) or "ok" not in resp:
|
||||
return {"ok": False, "error": "浏览器子代理回包格式不对"}
|
||||
return resp
|
||||
|
||||
|
||||
def handle_request(req, browser_sock=BROWSER_SOCK):
|
||||
"""处理一个请求 dict,返回响应 dict(纯逻辑,可单测)。"""
|
||||
if not isinstance(req, dict):
|
||||
return {"ok": False, "error": "请求须是 JSON 对象"}
|
||||
op = req.get("op")
|
||||
if op == "ping":
|
||||
return {"ok": True, "cell": True}
|
||||
if op == "sync":
|
||||
return _run_sync(req.get("action"), req.get("repo"))
|
||||
if op == "browser":
|
||||
return _forward_browser(req, sock_path=browser_sock)
|
||||
return {"ok": False, "error": "未知操作:%r(白名单:ping/sync/browser)"
|
||||
% (op,)}
|
||||
|
||||
|
||||
def _peer_creds(conn):
|
||||
"""SO_PEERCRED:返回 (pid, uid, gid)。Linux struct ucred = 3 个 int。"""
|
||||
raw = conn.getsockopt(socket.SOL_SOCKET, socket.SO_PEERCRED,
|
||||
struct.calcsize("3i"))
|
||||
pid, uid, gid = struct.unpack("3i", raw)
|
||||
return pid, uid, gid
|
||||
|
||||
|
||||
def serve(sock_path=BROKER_SOCK, browser_sock=BROWSER_SOCK):
|
||||
"""监听 Unix socket,逐连接处理。返回码 0 只能是被信号杀死;
|
||||
socket 建不起来即抛错(systemd 会重启服务,不要静默带病跑)。"""
|
||||
if os.path.exists(sock_path):
|
||||
os.remove(sock_path)
|
||||
os.makedirs(os.path.dirname(sock_path), exist_ok=True)
|
||||
srv = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
|
||||
srv.bind(sock_path)
|
||||
srv.listen(16)
|
||||
# 0660 + 组 chv:单元内 root 在客机侧是 uid 1000(chv),
|
||||
# 组 chv 可连;其它用户连不上(再叠加 SO_PEERCRED 校验)。
|
||||
os.chmod(sock_path, 0o660)
|
||||
try:
|
||||
gid = grp.getgrnam("chv").gr_gid
|
||||
except KeyError:
|
||||
print("chv-broker: 组 chv 不存在,socket 权限置不全,拒绝启动",
|
||||
file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
os.chown(sock_path, -1, gid)
|
||||
print("chv-broker: 监听 %s" % sock_path, file=sys.stderr, flush=True)
|
||||
while True:
|
||||
try:
|
||||
conn, _ = srv.accept()
|
||||
except OSError:
|
||||
continue
|
||||
try:
|
||||
pid, uid, _gid = _peer_creds(conn)
|
||||
cell_pids = _read_cell_pids()
|
||||
allowed, reason = peer_allowed_multi(uid, pid, cell_pids)
|
||||
if not allowed:
|
||||
print("chv-broker: 拒绝连接 pid=%d uid=%d:%s"
|
||||
% (pid, uid, reason), file=sys.stderr, flush=True)
|
||||
conn.close()
|
||||
continue
|
||||
if not cell_pids:
|
||||
print(_degraded_warning(reason, pid, uid),
|
||||
file=sys.stderr, flush=True)
|
||||
conn.settimeout(310)
|
||||
data = b""
|
||||
while len(data) < MAX_REQUEST_BYTES:
|
||||
chunk = conn.recv(4096)
|
||||
if not chunk:
|
||||
break
|
||||
data += chunk
|
||||
if data.endswith(b"\n"):
|
||||
break
|
||||
try:
|
||||
req = json.loads(data.decode())
|
||||
except (ValueError, UnicodeDecodeError):
|
||||
resp = {"ok": False, "error": "请求不是合法 JSON"}
|
||||
else:
|
||||
resp = handle_request(req, browser_sock=browser_sock)
|
||||
conn.sendall((json.dumps(resp, ensure_ascii=False)
|
||||
+ "\n").encode())
|
||||
except OSError as e:
|
||||
print("chv-broker: 连接处理异常:%s" % e, file=sys.stderr,
|
||||
flush=True)
|
||||
finally:
|
||||
try:
|
||||
conn.close()
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def main(argv):
|
||||
sock_path = BROKER_SOCK
|
||||
browser_sock = BROWSER_SOCK
|
||||
i = 0
|
||||
while i < len(argv):
|
||||
if argv[i] == "--socket" and i + 1 < len(argv):
|
||||
sock_path = argv[i + 1]
|
||||
i += 2
|
||||
elif argv[i] == "--browser-socket" and i + 1 < len(argv):
|
||||
browser_sock = argv[i + 1]
|
||||
i += 2
|
||||
else:
|
||||
print("chv-broker: 未知参数:%s" % argv[i], file=sys.stderr)
|
||||
return 2
|
||||
serve(sock_path, browser_sock)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main(sys.argv[1:]))
|
||||
@@ -0,0 +1,308 @@
|
||||
#!/usr/bin/env python3
|
||||
"""chv-browser:浏览器子代理(客机侧、单元外独立进程)。
|
||||
|
||||
对齐 Muse P2(见 docs/chv-muse-p2-gap.md):浏览器任务由专门子代理
|
||||
完成,与 codex 单元分离。子代理只向单元返回页面无障碍快照
|
||||
(Playwright Page.aria_snapshot() 的 ARIA 快照字符串),不见原始
|
||||
DOM,不能在页面上下文跑 JS,不暴露 evaluate/content,不开 DevTools。
|
||||
|
||||
操作(Unix socket /run/chv/browser.sock,JSON 行):
|
||||
open {url} -> 导航并返回 {title, url, a11y, truncated}
|
||||
snapshot {} -> 当前页的 {title, url, a11y, truncated}
|
||||
screenshot {} -> 截图落盘 /run/chv/shots/<uuid>.png,返回 {path}
|
||||
close {} -> 关当前页(浏览器进程继续跑,等下一次 open)
|
||||
ping {} -> {"ok": true}
|
||||
|
||||
URL 只允许 http/https;localhost 与私有地址字面量直接拒绝(主机名
|
||||
判定,不做 DNS 解析——DNS 重绑定防护是 P3 Sentinel 的事,这里不
|
||||
冒充已做)。无障碍树截断到 200 节点 / 64KB,超了标 truncated=true,
|
||||
绝不把整棵树无上限灌进 socket。
|
||||
|
||||
playwright 是懒加载:import 本模块不需要 playwright 已装(单测
|
||||
在无 playwright 环境跑);真正 open 时才加载,缺失即报错。
|
||||
|
||||
只用标准库 + playwright(运行时)。运行身份:系统用户 chv-browser,
|
||||
socket 0600(broker 以 root 连,不受模式限制)。
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import socket
|
||||
import sys
|
||||
import uuid
|
||||
|
||||
# round-11 真机修复(缺陷 4):自有目录 /run/chv-browser,
|
||||
# 0755 chv-browser 所有——/run/chv 是 0755 root root,
|
||||
# 在此 bind 会 EACCES 而无限 crash loop(见 chv-broker.py)。
|
||||
BROWSER_SOCK = "/run/chv-browser/browser.sock"
|
||||
SHOTS_DIR = "/run/chv/shots"
|
||||
PLAYWRIGHT_BROWSERS_PATH = "/usr/local/share/ms-playwright"
|
||||
MAX_REQUEST_BYTES = 65536
|
||||
MAX_A11Y_NODES = 200
|
||||
MAX_A11Y_CHARS = 65536
|
||||
NAV_TIMEOUT_MS = 30000
|
||||
|
||||
# 拒绝的 URL 主机:localhost 与私有地址字面量(主机名判定)。
|
||||
_PRIVATE_HOST_RE = re.compile(
|
||||
r"^(localhost|127\.\d+\.\d+\.\d+|10\.\d+\.\d+\.\d+|"
|
||||
r"172\.(1[6-9]|2\d|3[01])\.\d+\.\d+|192\.168\.\d+\.\d+|"
|
||||
r"\[::1\]|\[fe8[0-9a-f]*:.*\])$",
|
||||
re.IGNORECASE)
|
||||
|
||||
|
||||
def validate_url(url):
|
||||
"""URL 校验:只允许 http/https,且主机不是 localhost/私有地址
|
||||
字面量。返回 (ok, error)。DNS 重绑定不在本层防(见模块 docstring)。"""
|
||||
if not isinstance(url, str) or not url:
|
||||
return False, "url 不能为空"
|
||||
m = re.match(r"^(https?)://([^/]+)", url, re.IGNORECASE)
|
||||
if not m:
|
||||
return False, "只允许 http/https URL:%s" % url[:80]
|
||||
host = m.group(2).split("@")[-1].split(":")[0].lower()
|
||||
if _PRIVATE_HOST_RE.match(host):
|
||||
return False, "拒绝 localhost/私有地址:%s" % host
|
||||
return True, ""
|
||||
|
||||
|
||||
def truncate_aria(text, max_nodes=MAX_A11Y_NODES,
|
||||
max_chars=MAX_A11Y_CHARS):
|
||||
"""截断 ARIA 快照字符串:节点行(行首 "- ")超上限时剪掉多余
|
||||
行,字符超上限时截断。返回 (text, truncated)。纯函数,可单测。
|
||||
round-11 复测纠正(第二轮):Playwright 根本没有
|
||||
page.accessibility(那是 Puppeteer 的 API),原生快照是
|
||||
Page.aria_snapshot() 返回的字符串;旧的 dict 版 truncate_a11y
|
||||
随之废止。a11y 字段仍叫 a11y,值改为 ARIA 快照字符串。"""
|
||||
text = text or ""
|
||||
lines = text.splitlines()
|
||||
kept = []
|
||||
nodes = 0
|
||||
truncated = False
|
||||
for line in lines:
|
||||
if line.lstrip().startswith("- "):
|
||||
nodes += 1
|
||||
if nodes > max_nodes:
|
||||
truncated = True
|
||||
break
|
||||
kept.append(line)
|
||||
out = "\n".join(kept)
|
||||
if len(out) > max_chars:
|
||||
out = out[:max_chars]
|
||||
truncated = True
|
||||
return out, truncated
|
||||
|
||||
|
||||
def _load_playwright():
|
||||
"""懒加载 playwright;缺失时报明错(不静默假装有浏览器)。
|
||||
返回调用后的上下文管理器(sync_playwright()),不是函数本身——
|
||||
round-11 复测回归:曾返回裸 sync_playwright 函数,
|
||||
_ensure_browser 对它调 __enter__ 即 AttributeError,open
|
||||
永远失败;单测的假工厂直接返回带 __enter__ 的对象,盖不住
|
||||
这个形状(见 test_load_playwright_returns_called_cm)。"""
|
||||
os.environ.setdefault("PLAYWRIGHT_BROWSERS_PATH",
|
||||
PLAYWRIGHT_BROWSERS_PATH)
|
||||
try:
|
||||
from playwright.sync_api import sync_playwright
|
||||
except ImportError as e:
|
||||
raise RuntimeError(
|
||||
"chv-browser: playwright 未装(%s),浏览器子代理不可用"
|
||||
% e)
|
||||
return sync_playwright()
|
||||
|
||||
|
||||
class BrowserAgent:
|
||||
"""单页浏览器子代理。playwright_factory 供单测注入假实现;
|
||||
正常运行时用 _load_playwright()。"""
|
||||
|
||||
def __init__(self, playwright_factory=None, shots_dir=SHOTS_DIR):
|
||||
self._factory = playwright_factory or _load_playwright
|
||||
self._shots_dir = shots_dir
|
||||
self._pw = None
|
||||
self._browser = None
|
||||
self._page = None
|
||||
|
||||
def _ensure_browser(self):
|
||||
if self._browser is not None:
|
||||
return
|
||||
self._pw = self._factory()
|
||||
# sync_playwright() 返回上下文管理器;__enter__ 后拿 chromium。
|
||||
entered = self._pw.__enter__()
|
||||
self._browser = entered.chromium.launch(headless=True)
|
||||
|
||||
def open(self, url):
|
||||
ok, err = validate_url(url)
|
||||
if not ok:
|
||||
return {"ok": False, "error": err}
|
||||
try:
|
||||
self._ensure_browser()
|
||||
if self._page is not None:
|
||||
try:
|
||||
self._page.close()
|
||||
except Exception:
|
||||
pass
|
||||
self._page = self._browser.new_page()
|
||||
self._page.goto(url, timeout=NAV_TIMEOUT_MS)
|
||||
self._page.wait_for_load_state("domcontentloaded",
|
||||
timeout=NAV_TIMEOUT_MS)
|
||||
except Exception as e:
|
||||
return {"ok": False,
|
||||
"error": "打开 %s 失败:%s" % (url[:80], e)}
|
||||
return self.snapshot()
|
||||
|
||||
def snapshot(self):
|
||||
if self._page is None:
|
||||
return {"ok": False, "error": "还没有打开的页面(先 open)"}
|
||||
try:
|
||||
title = self._page.title()
|
||||
url = self._page.url
|
||||
# round-11 复测纠正(第二轮):Playwright 没有
|
||||
# page.accessibility(Puppeteer 的 API),原生快照是
|
||||
# Page.aria_snapshot()(字符串);a11y 字段仍叫 a11y,
|
||||
# 值改为 ARIA 快照字符串(截断到 200 节点行/64KB)。
|
||||
aria = self._page.aria_snapshot()
|
||||
except Exception as e:
|
||||
return {"ok": False, "error": "取快照失败:%s" % e}
|
||||
aria, truncated = truncate_aria(aria or "")
|
||||
return {"ok": True, "title": title, "url": url,
|
||||
"a11y": aria, "truncated": truncated}
|
||||
|
||||
def screenshot(self):
|
||||
if self._page is None:
|
||||
return {"ok": False, "error": "还没有打开的页面(先 open)"}
|
||||
try:
|
||||
os.makedirs(self._shots_dir, exist_ok=True)
|
||||
path = os.path.join(self._shots_dir, "%s.png"
|
||||
% uuid.uuid4().hex)
|
||||
self._page.screenshot(path=path)
|
||||
except Exception as e:
|
||||
return {"ok": False, "error": "截图失败:%s" % e}
|
||||
return {"ok": True, "path": path}
|
||||
|
||||
def close(self):
|
||||
if self._page is not None:
|
||||
try:
|
||||
self._page.close()
|
||||
except Exception:
|
||||
pass
|
||||
self._page = None
|
||||
return {"ok": True, "closed": True}
|
||||
|
||||
def shutdown(self):
|
||||
try:
|
||||
if self._browser is not None:
|
||||
self._browser.close()
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
if self._pw is not None:
|
||||
self._pw.__exit__(None, None, None)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
# 本层不暴露的操作:evaluate / content(原始 DOM)/ DevTools。
|
||||
# 收到即拒绝,不转发、不执行。
|
||||
FORBIDDEN_OPS = ("evaluate", "content", "devtools", "js")
|
||||
|
||||
|
||||
def handle_request(agent, req):
|
||||
"""处理一个请求 dict,返回响应 dict(纯逻辑,可单测)。
|
||||
op "browser" 是 broker 原样转发的形状(round-11 真机修复):
|
||||
内层 op 在 action 里;无 action 但带 url 时默认 open(驱动
|
||||
只发 url);action 同样受 FORBIDDEN_OPS 约束。"""
|
||||
if not isinstance(req, dict):
|
||||
return {"ok": False, "error": "请求须是 JSON 对象"}
|
||||
op = req.get("op")
|
||||
if op == "browser":
|
||||
op = req.get("action")
|
||||
if op is None and req.get("url"):
|
||||
op = "open"
|
||||
if op in FORBIDDEN_OPS:
|
||||
return {"ok": False,
|
||||
"error": "操作 %r 不暴露(无 JS/无原始 DOM)" % op}
|
||||
if op == "ping":
|
||||
return {"ok": True, "browser": True}
|
||||
if op == "open":
|
||||
return agent.open(req.get("url"))
|
||||
if op == "snapshot":
|
||||
return agent.snapshot()
|
||||
if op == "screenshot":
|
||||
return agent.screenshot()
|
||||
if op == "close":
|
||||
return agent.close()
|
||||
return {"ok": False,
|
||||
"error": "未知操作:%r(白名单:open/snapshot/screenshot/"
|
||||
"close/ping)" % (op,)}
|
||||
|
||||
|
||||
def serve(sock_path=BROWSER_SOCK, shots_dir=SHOTS_DIR):
|
||||
"""监听 Unix socket,逐连接处理。socket 0600(只给属主 +
|
||||
broker 的 root 连)。建不起来即抛错,不要静默带病跑。"""
|
||||
if os.path.exists(sock_path):
|
||||
os.remove(sock_path)
|
||||
os.makedirs(os.path.dirname(sock_path), exist_ok=True)
|
||||
srv = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
|
||||
srv.bind(sock_path)
|
||||
srv.listen(16)
|
||||
os.chmod(sock_path, 0o600)
|
||||
print("chv-browser: 监听 %s" % sock_path, file=sys.stderr, flush=True)
|
||||
agent = BrowserAgent(shots_dir=shots_dir)
|
||||
try:
|
||||
while True:
|
||||
try:
|
||||
conn, _ = srv.accept()
|
||||
except OSError:
|
||||
continue
|
||||
try:
|
||||
conn.settimeout(120)
|
||||
data = b""
|
||||
while len(data) < MAX_REQUEST_BYTES:
|
||||
chunk = conn.recv(4096)
|
||||
if not chunk:
|
||||
break
|
||||
data += chunk
|
||||
if data.endswith(b"\n"):
|
||||
break
|
||||
try:
|
||||
req = json.loads(data.decode())
|
||||
except (ValueError, UnicodeDecodeError):
|
||||
resp = {"ok": False, "error": "请求不是合法 JSON"}
|
||||
else:
|
||||
try:
|
||||
resp = handle_request(agent, req)
|
||||
except Exception as e:
|
||||
resp = {"ok": False,
|
||||
"error": "浏览器子代理内部错误:%s" % e}
|
||||
conn.sendall((json.dumps(resp, ensure_ascii=False)
|
||||
+ "\n").encode())
|
||||
except OSError as e:
|
||||
print("chv-browser: 连接处理异常:%s" % e, file=sys.stderr,
|
||||
flush=True)
|
||||
finally:
|
||||
try:
|
||||
conn.close()
|
||||
except OSError:
|
||||
pass
|
||||
finally:
|
||||
agent.shutdown()
|
||||
|
||||
|
||||
def main(argv):
|
||||
sock_path = BROWSER_SOCK
|
||||
shots_dir = SHOTS_DIR
|
||||
i = 0
|
||||
while i < len(argv):
|
||||
if argv[i] == "--socket" and i + 1 < len(argv):
|
||||
sock_path = argv[i + 1]
|
||||
i += 2
|
||||
elif argv[i] == "--shots-dir" and i + 1 < len(argv):
|
||||
shots_dir = argv[i + 1]
|
||||
i += 2
|
||||
else:
|
||||
print("chv-browser: 未知参数:%s" % argv[i], file=sys.stderr)
|
||||
return 2
|
||||
serve(sock_path, shots_dir)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main(sys.argv[1:]))
|
||||
@@ -0,0 +1,844 @@
|
||||
#!/usr/bin/env python3
|
||||
"""chv-cell:客机内 systemd-nspawn 运行时单元(Hatch 式)启动器。
|
||||
|
||||
对齐 Muse P2(见 docs/chv-muse-p2-gap.md):codex 跑在单元内,不再
|
||||
直接跑在客机用户态。单元是单命令容器(--as-pid2,不 boot 第二个
|
||||
systemd),单元即沙箱——单元内 codex 用 danger-full-access,不再
|
||||
套 bwrap(bwrap 的 --proc 挂载在嵌套命名空间里走不通,见 AGENTS.md
|
||||
内层 proc 定位)。
|
||||
|
||||
关键映射(与 Muse“单元 root 映射为 host 侧非特权用户”同款):
|
||||
--private-users=1000:65536 —— 单元内 uid 0 → 客机侧 chv(uid 1000)。
|
||||
效果:workspace / ~/.codex(chv 所有)单元内仍可写;host-root 0600
|
||||
文件(如 /etc/chv/guest-env)在单元内显示为 overflow uid、不可读;
|
||||
再加 --inaccessible 遮蔽作纵深防御(只遮 /usr 下路径:--volatile=yes
|
||||
下容器根是基本空的 tmpfs,/etc/chv 等在容器根里根本不存在,按构造
|
||||
即不可达,列出反而让 nspawn 解析失败而起不来,见 INACCESSIBLE_BASE)。
|
||||
|
||||
网络:--network-veth。每次 run_cell 在锁下分配唯一身份
|
||||
(CellNet:machine chv-cell-<xx>、宿主 ve-chv-cell-<xx>、
|
||||
10.200.<xx>.0/24,宿主 .1、单元 .2),不再共用固定名与固定
|
||||
子网——固定名是并发 cell 撞毁的根因(part 4)。客机侧 NAT
|
||||
出站(MASQUERADE 走客机默认路由的上行口,动态发现)。单元内
|
||||
配网经 nsenter 进单元 init 的 netns 做。veth 建不起来即大声
|
||||
失败,不静默回落共享网络;CHV_CELL_NETWORK=shared 是显式调试
|
||||
逃生口,须显式设置并打日志,绝不默认。
|
||||
起步门(part 4):payload 在 Popen 的瞬间就开始跑,与配网全程
|
||||
并发是启动竞态的根因——快 payload 在配网完成前退出,nspawn 随
|
||||
之退出,包装器再去配网即撞墙。run_cell 把 payload 包一层
|
||||
sh 起步门(等 /run/chv/.cell-gate-<machine> 出现再 exec),
|
||||
配网完成后才开门;find_cell_init_pid 改成带超时轮询;nspawn
|
||||
已死时返回它的退出码而不是抛错。
|
||||
volatile 空根(part 4):--volatile=yes 下单元根是基本空的
|
||||
tmpfs,/app、/etc/resolv.conf、/etc/hosts、/etc/ssl/certs 都须
|
||||
显式绑进去(/app 与 CA 只读绑;resolv/hosts 是 run_cell 按客机
|
||||
有效 DNS 生成的单元专属文件,只读绑;不绑 nsswitch.conf,
|
||||
glibc 默认 files+dns 正是单元需要的)。part 6(RETEST5)纠正
|
||||
part 5 的设想:/app/work 以可写 bind 盖住只读 /app 的做法在真机
|
||||
上不成立——后挂的 bind 只有在挂载点已存在时才能盖住,挂载点
|
||||
不存在时 nspawn 在只读 /app 下连建都建不起来(全量 73 次同一
|
||||
错误,所有 cell 启动失败)。worktree 根改移出 /app:单元内
|
||||
make_worktree 改用 workspace 绑定下已可写的
|
||||
/home/chv/workspace/.chv-worktrees(经 CHV_WORKTREE_ROOT 环境
|
||||
变量传进单元),不再发任何指向 /app 之下的 bind。
|
||||
|
||||
环境透传:nspawn 不透传调用方环境,本脚本只转发白名单变量(模型
|
||||
网关与代理变量)。GITEA_TOKEN 等凭证变量不在白名单里,绝不进单元。
|
||||
|
||||
用法(客机侧 root 运行,执行器经 SSH 调):
|
||||
chv-cell run [--network veth|shared] -- <cmd...>
|
||||
# 例:chv-cell run -- python3 /app/host.py --state
|
||||
# /home/chv/.codex/host-state.json --codex-home /home/chv/.codex
|
||||
# --sandbox danger-full-access dispatch --name t --task ...
|
||||
chv-cell argv [--network veth|shared] -- <cmd...>
|
||||
# 只打印 nspawn 参数表(JSON),不启动
|
||||
chv-cell network-commands [--uplink IFACE] [--cell-pid PID]
|
||||
# 只打印配网命令表(JSON),不执行
|
||||
只用标准库。单元启动失败、配网失败一律非零退出,不静默降级。
|
||||
"""
|
||||
import fcntl
|
||||
import glob
|
||||
import json
|
||||
import os
|
||||
import pwd
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from typing import NamedTuple
|
||||
|
||||
MACHINE = "chv-cell"
|
||||
# 单元映射:单元内 uid 0 → 客机侧 chv(uid 1000,非特权用户)。
|
||||
UID_BASE = 1000
|
||||
UID_COUNT = 65536
|
||||
CELL_USER = "chv"
|
||||
CELL_UID = 1000
|
||||
CELL_GID = 1000
|
||||
WORKSPACE = "/home/chv/workspace"
|
||||
CODEX_HOME = "/home/chv/.codex"
|
||||
BROKER_DIR = "/run/chv"
|
||||
# part 6(RETEST5 新发现):单元内 worktree 根(RETEST4 新发现 2
|
||||
# 的旧址是 HERE/work,即单元内 /app/work——/app 是 --bind-ro,
|
||||
# 带 repo 的默认 dispatch 撞 OSError Errno 30;part 5 把该目录以
|
||||
# 可写 bind 挂到 /app/work 上,真机证明不可行:挂载点不存在时
|
||||
# nspawn 在只读 /app 下连建都建不起来,所有 cell 启动失败)。
|
||||
# 新址是 workspace 绑定下的 .chv-worktrees:--bind=/home/chv/
|
||||
# workspace 已把整个 workspace 以可写方式挂进单元(目的路径与
|
||||
# 源相同、挂载点即 workspace 自身,挂载时已存在),worktree 根
|
||||
# 落在这里无需任何额外 bind;backing 仍在客机 workspace 下、
|
||||
# 随客机磁盘持久化(跨 cell 的 run/merge 续跑仍找得到 worktree),
|
||||
# 隔离与落盘语义不变,只是单元内路径从 /app/work/host-worktrees
|
||||
# 改为 /home/chv/workspace/.chv-worktrees/host-worktrees。
|
||||
# 该值经 --setenv=CHV_WORKTREE_ROOT 传进单元,host.py 的
|
||||
# worktree_base() 读它(与 host.py 的 CELL_WORKTREE_ROOT 同值,
|
||||
# 两处须保持一致)。
|
||||
WORKTREE_DIR = "/home/chv/workspace/.chv-worktrees"
|
||||
CELL_PID_FILE = "/run/chv/cell.pid"
|
||||
|
||||
# 单元网络(与客机 tap 网 192.168.10.0/24 不重叠的独立 /24)。
|
||||
# part 4(2026-11-07):固定名/固定子网是并发 cell 撞毁的根因——
|
||||
# 每次 run_cell 都在锁下分配唯一身份(CellNet),不再共用常量。
|
||||
# machine 名须短:nspawn 把宿主侧 veth 叫 ve-<machine>,接口名
|
||||
# 上限 15 字符;"chv-cell-%02x" 为 11 字符,ve- 前缀后 14 字符。
|
||||
CELL_NET_BASE = "10.200"
|
||||
CELL_NET = "10.200.0.0/24"
|
||||
HOST_IP = "10.200.0.1"
|
||||
CELL_IP = "10.200.0.2"
|
||||
HOST_IFACE = "ve-chv-cell"
|
||||
CELL_IFACE = "host0"
|
||||
MAX_CELL_INDEX = 250
|
||||
ALLOC_LOCK_FILE = "/run/chv/.cell-alloc.lock"
|
||||
|
||||
# Muse 点名要丢的能力 + 同批丢的危险能力(nspawn 默认集已有限,
|
||||
# 这里是显式再丢,--drop-capability 逐项写,测试逐项断言)。
|
||||
DROP_CAPABILITIES = [
|
||||
"CAP_SYS_ADMIN",
|
||||
"CAP_SYS_PTRACE",
|
||||
"CAP_NET_ADMIN",
|
||||
"CAP_NET_RAW",
|
||||
"CAP_SYS_MODULE",
|
||||
"CAP_SYS_RAWIO",
|
||||
"CAP_SYS_BOOT",
|
||||
"CAP_MKNOD",
|
||||
]
|
||||
|
||||
# syscall 过滤:@system-service 集合再扣掉 io_uring 三件套
|
||||
#(Muse 点名禁用 io_uring)。
|
||||
SYSCALL_FILTER = ("@system-service ~io_uring_setup "
|
||||
"~io_uring_enter ~io_uring_register")
|
||||
|
||||
# 单元内遮掉的路径:浏览器(单元内跑不起自己的浏览器,
|
||||
# 只能走 broker 的 chv-browser)。只列 /usr 下的路径——
|
||||
# round-11 复测纠正(第二轮):--volatile=yes 下容器根是
|
||||
# 基本空的 tmpfs(只挂进只读 /usr),nspawn 按容器根解析
|
||||
# --inaccessible 路径,不存在的路径即启动失败;/etc/chv 与
|
||||
# /home/chv/.git-credentials 在容器根里根本不存在(无 /home、
|
||||
# 无 /etc/chv),凭证文件按构造就不可达,不再列出遮蔽。
|
||||
# 旧的两项若列回,cell 会报 Failed to resolve 而起不来。
|
||||
INACCESSIBLE_BASE = [
|
||||
"/usr/local/share/ms-playwright",
|
||||
]
|
||||
|
||||
# 只转发白名单环境变量进单元。凭证类变量(GITEA_TOKEN 等)
|
||||
# 不在白名单里,绝不进单元。
|
||||
FORWARD_ENV_ALLOWLIST = (
|
||||
"SUB2API_BASE_URL",
|
||||
"SUB2API_API_KEY",
|
||||
"XCM_API_KEY",
|
||||
"HTTP_PROXY",
|
||||
"HTTPS_PROXY",
|
||||
"NO_PROXY",
|
||||
"http_proxy",
|
||||
"https_proxy",
|
||||
"no_proxy",
|
||||
)
|
||||
|
||||
|
||||
class CellNet(NamedTuple):
|
||||
"""一次 cell 运行的唯一网络身份(锁下分配,见 allocate_cell_net)。
|
||||
machine 同时决定宿主侧 veth 名(ve-<machine>,nspawn 约定)与
|
||||
/run/chv 下的专属文件后缀(gate/resolv/hosts/pid)。"""
|
||||
index: int
|
||||
machine: str
|
||||
host_iface: str
|
||||
host_ip: str
|
||||
cell_ip: str
|
||||
subnet: str
|
||||
|
||||
|
||||
def legacy_cell_net():
|
||||
"""无身份时的旧常量(argv/network-commands 子命令与旧测试用;
|
||||
run_cell 一律走 allocate_cell_net,不用它)。"""
|
||||
return CellNet(index=0, machine=MACHINE, host_iface=HOST_IFACE,
|
||||
host_ip=HOST_IP, cell_ip=CELL_IP, subnet=CELL_NET)
|
||||
|
||||
# Muse 点名要丢的能力 + 同批丢的危险能力(nspawn 默认集已有限,
|
||||
# 这里是显式再丢,--drop-capability 逐项写,测试逐项断言)。
|
||||
DROP_CAPABILITIES = [
|
||||
"CAP_SYS_ADMIN",
|
||||
"CAP_SYS_PTRACE",
|
||||
"CAP_NET_ADMIN",
|
||||
"CAP_NET_RAW",
|
||||
"CAP_SYS_MODULE",
|
||||
"CAP_SYS_RAWIO",
|
||||
"CAP_SYS_BOOT",
|
||||
"CAP_MKNOD",
|
||||
]
|
||||
|
||||
# syscall 过滤:@system-service 集合再扣掉 io_uring 三件套
|
||||
#(Muse 点名禁用 io_uring)。
|
||||
SYSCALL_FILTER = ("@system-service ~io_uring_setup "
|
||||
"~io_uring_enter ~io_uring_register")
|
||||
|
||||
# 单元内遮掉的路径:浏览器(单元内跑不起自己的浏览器,
|
||||
# 只能走 broker 的 chv-browser)。只列 /usr 下的路径——
|
||||
# round-11 复测纠正(第二轮):--volatile=yes 下容器根是
|
||||
# 基本空的 tmpfs(只挂进只读 /usr),nspawn 按容器根解析
|
||||
# --inaccessible 路径,不存在的路径即启动失败;/etc/chv 与
|
||||
# /home/chv/.git-credentials 在容器根里根本不存在(无 /home、
|
||||
# 无 /etc/chv),凭证文件按构造就不可达,不再列出遮蔽。
|
||||
# 旧的两项若列回,cell 会报 Failed to resolve 而起不来。
|
||||
INACCESSIBLE_BASE = [
|
||||
"/usr/local/share/ms-playwright",
|
||||
]
|
||||
|
||||
# 只转发白名单环境变量进单元。凭证类变量(GITEA_TOKEN 等)
|
||||
# 不在白名单里,绝不进单元。
|
||||
FORWARD_ENV_ALLOWLIST = (
|
||||
"SUB2API_BASE_URL",
|
||||
"SUB2API_API_KEY",
|
||||
"XCM_API_KEY",
|
||||
"HTTP_PROXY",
|
||||
"HTTPS_PROXY",
|
||||
"NO_PROXY",
|
||||
"http_proxy",
|
||||
"https_proxy",
|
||||
"no_proxy",
|
||||
)
|
||||
|
||||
|
||||
def _playwright_package_dir():
|
||||
"""playwright pip 包目录(遮蔽用)。算不出时回落到 noble 默认路径。"""
|
||||
try:
|
||||
import sysconfig
|
||||
purelib = sysconfig.get_path("purelib")
|
||||
if purelib:
|
||||
return os.path.join(purelib, "playwright")
|
||||
except Exception:
|
||||
pass
|
||||
return "/usr/local/lib/python3.12/dist-packages/playwright"
|
||||
|
||||
|
||||
def build_nspawn_argv(command, network="veth",
|
||||
workspace=WORKSPACE, codex_home=CODEX_HOME,
|
||||
broker_dir=BROKER_DIR,
|
||||
playwright_package_dir=None,
|
||||
cell_net=None, extra_ro_binds=None):
|
||||
"""产 nspawn 参数表(纯函数,可单测)。command 是单元内要跑的命令
|
||||
(list,如 ["python3", "/app/host.py", ...])。
|
||||
cell_net 为 None 时用旧常量身份(argv 子命令与旧测试);
|
||||
run_cell 一律传 allocate_cell_net() 的唯一身份。
|
||||
extra_ro_binds 是 [(src, dest)],只读绑进单元(如单元专属
|
||||
resolv.conf/hosts;/app 与 /etc/ssl/certs 是固定只读绑定,
|
||||
直接列在下面)。worktree 根不经 bind:WORKTREE_DIR 在已可写
|
||||
绑进单元的 workspace 之下(part 6,RETEST5),经
|
||||
--setenv=CHV_WORKTREE_ROOT 传进单元供 host.py 的
|
||||
worktree_base() 用;argv 里绝不出现指向 /app 之下(只读父
|
||||
目录)的 bind——挂载点不存在时 nspawn 在只读父目录下建不
|
||||
了挂载点,单元直接起不来。"""
|
||||
if not command:
|
||||
raise ValueError("chv-cell: 单元内命令不能为空")
|
||||
if network not in ("veth", "shared"):
|
||||
raise ValueError("chv-cell: network 只能是 veth|shared:%r"
|
||||
% network)
|
||||
if playwright_package_dir is None:
|
||||
playwright_package_dir = _playwright_package_dir()
|
||||
machine = cell_net.machine if cell_net is not None else MACHINE
|
||||
argv = [
|
||||
"systemd-nspawn", "--quiet",
|
||||
"--machine=%s" % machine,
|
||||
"--directory=/",
|
||||
# round-11 真机修复:nspawn 255 拒裸 --directory=/(须
|
||||
# --ephemeral 或 --volatile),也拒裸 --no-new-privileges
|
||||
#(把它当作带必填参数的选项,吞掉下一个参数后报错)。
|
||||
# 2026-11-07 复测纠正:首轮修用的 --ephemeral 在客内
|
||||
# ext4 根盘(2.4G、已用 1.8G、可用 593M)上做整盘快照
|
||||
# 拷贝,空间不够即 ENOSPC,全部 cell 调用失败。改
|
||||
# --volatile=yes:根为基本空的 tmpfs、/usr 只读挂进,
|
||||
# 单元内除 bind 出去的目录外一律可写、用完即弃——与
|
||||
# --ephemeral 的设计意图一致,但不拷盘、不占空间。
|
||||
# --volatile=state 不行:整树只读、仅 /var 为 tmpfs,
|
||||
# codex 写 $HOME/.cache 等非 bind 路径会失败。
|
||||
"--volatile=yes",
|
||||
"--as-pid2",
|
||||
"--private-users=%d:%d" % (UID_BASE, UID_COUNT),
|
||||
"--no-new-privileges=yes",
|
||||
"--system-call-filter=%s" % SYSCALL_FILTER,
|
||||
]
|
||||
for cap in DROP_CAPABILITIES:
|
||||
argv.append("--drop-capability=%s" % cap)
|
||||
for path in INACCESSIBLE_BASE + [playwright_package_dir]:
|
||||
argv.append("--inaccessible=%s" % path)
|
||||
argv += [
|
||||
"--bind=%s" % workspace,
|
||||
"--bind=%s" % codex_home,
|
||||
"--bind=%s" % broker_dir,
|
||||
# part 4(volatile 空根缺口):--volatile=yes 下单元根是
|
||||
# 基本空的 tmpfs,执行器默认在单元内跑 python3 /app/host.py、
|
||||
# codex 要做 DNS 与 TLS——/app、CA bundle、resolv.conf、
|
||||
# hosts 都须显式绑进去,否则单元内一律 ENOENT/无 DNS。
|
||||
# /app 与 /etc/ssl/certs 是镜像/系统固定路径,只读绑;
|
||||
# resolv.conf/hosts 是 run_cell 按客机有效 DNS 生成的
|
||||
# 单元专属文件,经 extra_ro_binds 传进来(见
|
||||
# prepare_cell_files:客机 /etc/resolv.conf 可能是
|
||||
# systemd-resolved stub(127.0.0.53),单元内无 resolved,
|
||||
# 直接绑 stub 即 DNS 全灭,故须生成真实服务器列表)。
|
||||
# 不绑 /etc/nsswitch.conf:客机版可能含 resolve 模块,
|
||||
# 单元内无 resolved/dbus,绑了反而可能挂起;glibc 缺
|
||||
# nsswitch.conf 时默认 files+dns,正是单元需要的。
|
||||
"--bind-ro=/app",
|
||||
# part 6(RETEST5):此处绝不再加 --bind=...:/app/work。
|
||||
# part 5 的该绑定已证不可行:/app/work 挂载点在 --bind-ro
|
||||
# 的 /app 镜像里不存在,nspawn 在只读父目录下建不了挂
|
||||
# 载点(EROFS),单元启动即失败。worktree 根已移出 /app,
|
||||
# 见 WORKTREE_DIR 与 --setenv=CHV_WORKTREE_ROOT。
|
||||
"--bind-ro=/etc/ssl/certs",
|
||||
"--tmpfs=/tmp",
|
||||
]
|
||||
for src, dest in (extra_ro_binds or []):
|
||||
argv.append("--bind-ro=%s:%s" % (src, dest))
|
||||
if network == "veth":
|
||||
argv.append("--network-veth")
|
||||
argv += [
|
||||
"--setenv=CHV_CELL=1",
|
||||
# part 6:单元内 worktree 根(host.py worktree_base() 读
|
||||
# 它;值与 WORKTREE_DIR 一致,客机侧与单元内是同一路径,
|
||||
# 因 workspace 以同名可写 bind 进单元)。
|
||||
"--setenv=CHV_WORKTREE_ROOT=%s" % WORKTREE_DIR,
|
||||
"--setenv=HOME=/home/chv",
|
||||
]
|
||||
for name in FORWARD_ENV_ALLOWLIST:
|
||||
value = os.environ.get(name)
|
||||
if value:
|
||||
argv.append("--setenv=%s=%s" % (name, value))
|
||||
argv += ["--chdir=%s" % workspace]
|
||||
argv += list(command)
|
||||
return argv
|
||||
|
||||
|
||||
def parse_default_route_iface(ip_route_output):
|
||||
"""从 `ip route show default` 的输出里解析上行口名。
|
||||
无默认路由时返回 None(调用方大声失败)。"""
|
||||
for line in (ip_route_output or "").splitlines():
|
||||
line = line.strip()
|
||||
if not line.startswith("default "):
|
||||
continue
|
||||
m = re.search(r"\bdev\s+(\S+)", line)
|
||||
if m:
|
||||
return m.group(1)
|
||||
return None
|
||||
|
||||
|
||||
def find_uplink_iface(run=None):
|
||||
"""发现客机上行口(默认路由的 dev)。run 供单测注入。"""
|
||||
run = run or subprocess.run
|
||||
proc = run(["ip", "route", "show", "default"],
|
||||
capture_output=True, text=True)
|
||||
if proc.returncode != 0:
|
||||
return None
|
||||
return parse_default_route_iface(proc.stdout)
|
||||
|
||||
|
||||
def host_network_commands(uplink_iface, net=None):
|
||||
"""客机侧配网命令表(纯函数):ve 接口起 IP + 转发 + NAT。
|
||||
调用方按顺序执行,任一失败即停。net 为 None 时用旧常量
|
||||
(argv/network-commands 子命令与旧测试);run_cell 传
|
||||
allocate_cell_net() 的唯一身份。"""
|
||||
if not uplink_iface:
|
||||
raise ValueError("chv-cell: 上行口名为空,配不出 NAT")
|
||||
net = net or legacy_cell_net()
|
||||
return [
|
||||
["ip", "addr", "add", "%s/24" % net.host_ip, "dev",
|
||||
net.host_iface],
|
||||
["ip", "link", "set", net.host_iface, "up"],
|
||||
["sysctl", "-w", "net.ipv4.ip_forward=1"],
|
||||
["iptables", "-t", "nat", "-A", "POSTROUTING",
|
||||
"-s", net.subnet, "-o", uplink_iface, "-j", "MASQUERADE"],
|
||||
["iptables", "-A", "FORWARD", "-i", net.host_iface,
|
||||
"-o", uplink_iface, "-j", "ACCEPT"],
|
||||
["iptables", "-A", "FORWARD", "-i", uplink_iface,
|
||||
"-o", net.host_iface, "-m", "state", "--state",
|
||||
"RELATED,ESTABLISHED", "-j", "ACCEPT"],
|
||||
]
|
||||
|
||||
|
||||
def host_network_cleanup_commands(uplink_iface, net=None):
|
||||
"""配网清理命令表(与 host_network_commands 配对,-A 改 -D)。
|
||||
清理是 best-effort:删不掉(规则已不在)不报错。"""
|
||||
if not uplink_iface:
|
||||
raise ValueError("chv-cell: 上行口名为空,清不了 NAT")
|
||||
cmds = []
|
||||
for cmd in host_network_commands(uplink_iface, net):
|
||||
if "iptables" not in cmd:
|
||||
continue
|
||||
cleaned = [c if c != "-A" else "-D" for c in cmd]
|
||||
cmds.append(cleaned)
|
||||
return cmds
|
||||
|
||||
|
||||
def find_cell_init_pid(nspawn_pid, proc_root="/proc", timeout=0.0,
|
||||
interval=0.1, proc_alive=None):
|
||||
"""找单元 init 的 pid:nspawn 进程的第一个子进程
|
||||
(--as-pid2 时它是 stub init,payload 是 PID 2;两者同 netns)。
|
||||
timeout=0(默认)即只查一次(旧行为);timeout>0 时带超时轮询
|
||||
(part 4 竞态修复:init 可能还没 fork,单次查找即误判)。
|
||||
proc_alive 供调用方传“nspawn 是否还活着”(如 proc.poll() is
|
||||
None 的 lambda):nspawn 已退出即立刻回 None,不再空等超时。
|
||||
找不到返回 None。"""
|
||||
deadline = time.time() + timeout
|
||||
while True:
|
||||
found = None
|
||||
try:
|
||||
task_dir = os.path.join(proc_root, str(nspawn_pid), "task")
|
||||
for tid in os.listdir(task_dir):
|
||||
children_path = os.path.join(task_dir, tid, "children")
|
||||
try:
|
||||
with open(children_path) as fh:
|
||||
children = fh.read().split()
|
||||
except OSError:
|
||||
continue
|
||||
if children:
|
||||
try:
|
||||
found = int(children[0])
|
||||
except ValueError:
|
||||
continue
|
||||
break
|
||||
except OSError:
|
||||
# 任务目录还没出现(init 未 fork)即当作“这次没找到”,
|
||||
# 继续轮询,不直接回 None(旧代码这里直接 return None,
|
||||
# 正是单次查找误判的根因之一)。
|
||||
found = None
|
||||
if found is not None:
|
||||
return found
|
||||
if proc_alive is not None and not proc_alive():
|
||||
return None
|
||||
if time.time() >= deadline:
|
||||
return None
|
||||
time.sleep(min(interval, max(0.0, deadline - time.time())))
|
||||
|
||||
|
||||
def cell_network_commands(cell_pid, net=None):
|
||||
"""单元内配网命令表(纯函数):nsenter 进单元 init 的 netns,
|
||||
给 host0 配 IP、起接口、加默认路由。net 为 None 时用旧常量。"""
|
||||
if not cell_pid:
|
||||
raise ValueError("chv-cell: 单元 init pid 为空,配不了单元内网络")
|
||||
net = net or legacy_cell_net()
|
||||
prefix = ["nsenter", "-t", str(cell_pid), "-n"]
|
||||
return [
|
||||
prefix + ["ip", "addr", "add", "%s/24" % net.cell_ip,
|
||||
"dev", CELL_IFACE],
|
||||
prefix + ["ip", "link", "set", CELL_IFACE, "up"],
|
||||
prefix + ["ip", "route", "add", "default", "via", net.host_ip],
|
||||
]
|
||||
|
||||
|
||||
def _ensure_bind_source(path, uid=CELL_UID, gid=CELL_GID):
|
||||
"""bind 源目录须存在且属 chv(单元内 root 即客机 chv,属主不对
|
||||
就写不进 workspace)。root 建的目录要 chown 回 chv。"""
|
||||
os.makedirs(path, exist_ok=True)
|
||||
try:
|
||||
st = os.stat(path)
|
||||
if st.st_uid != uid or st.st_gid != gid:
|
||||
os.chown(path, uid, gid)
|
||||
except OSError as e:
|
||||
print("chv-cell: chown %s 失败:%s" % (path, e), file=sys.stderr)
|
||||
raise
|
||||
|
||||
|
||||
def gate_wrapper_command(command, gate_path):
|
||||
"""把单元内命令包一层起步门(纯函数):payload 先在单元内
|
||||
等 gate 文件出现再 exec 原命令。gate 文件由 run_cell 在配网
|
||||
完成后才创建(见 run_cell)——payload 与配网零同步是 RETEST3
|
||||
新缺陷 1 的根因(快 payload 在配网完成前退出,nspawn 随之
|
||||
退出,包装器再去配网即撞墙)。/usr/bin/sh 用绝对路径:
|
||||
volatile 单元根里 /bin 未必有,/usr 是只读挂进来的,必有。
|
||||
注意:$1 是 gate 路径,shift 后 "$@" 即原命令。"""
|
||||
script = ("while [ ! -e \"$1\" ]; do sleep 0.05; done; "
|
||||
"shift; exec \"$@\"")
|
||||
return ["/usr/bin/sh", "-c", script, "chv-cell-gate",
|
||||
gate_path] + list(command)
|
||||
|
||||
|
||||
def _parse_nameservers(path):
|
||||
"""读一个 resolv.conf 风格文件,返回 nameserver 列表
|
||||
(去 127.0.0.0/8 与 ::1——单元内无 resolved,stub 指过去
|
||||
即 DNS 全灭)。文件不存在或读不出即 []。"""
|
||||
servers = []
|
||||
try:
|
||||
with open(path) as fh:
|
||||
for line in fh:
|
||||
line = line.strip()
|
||||
if not line.startswith("nameserver"):
|
||||
continue
|
||||
parts = line.split()
|
||||
if len(parts) < 2:
|
||||
continue
|
||||
ip = parts[1]
|
||||
if ip.startswith("127.") or ip == "::1":
|
||||
continue
|
||||
if ip not in servers:
|
||||
servers.append(ip)
|
||||
except OSError:
|
||||
return []
|
||||
return servers
|
||||
|
||||
|
||||
def _effective_nameservers(resolv_conf="/etc/resolv.conf",
|
||||
resolved_conf="/run/systemd/resolve/resolv.conf"):
|
||||
"""客机有效 DNS:先读 /etc/resolv.conf;只剩 stub(127.0.0.53)
|
||||
或为空时读 resolved 的 resolv.conf;仍为空即大声失败。"""
|
||||
servers = _parse_nameservers(resolv_conf)
|
||||
if not servers:
|
||||
servers = _parse_nameservers(resolved_conf)
|
||||
if not servers:
|
||||
raise RuntimeError(
|
||||
"chv-cell: 客机无有效 DNS,配不出单元 resolv.conf")
|
||||
return servers
|
||||
|
||||
|
||||
def prepare_cell_files(machine, run_dir=BROKER_DIR,
|
||||
resolv_conf="/etc/resolv.conf",
|
||||
resolved_conf="/run/systemd/resolve/resolv.conf",
|
||||
hosts_file="/etc/hosts"):
|
||||
"""准备单元专属文件(run_cell 在 Popen 之前调),返回 dict:
|
||||
gate(起步门,run_cell 配网完成后才创建——注意 gate 文件本身
|
||||
在这里不创建,只给路径)、resolv(单元专属 resolv.conf,已写
|
||||
好)、hosts(单元专属 hosts,已写好)。文件都在 /run/chv 下
|
||||
(bind 进单元),0644(单元内 root 在客机侧是 chv,0600 的
|
||||
root 文件在单元内读不到;DNS/hosts 不含秘密,0644 安全)。
|
||||
/app 与 /etc/ssl/certs 是固定只读绑定,不走这里。"""
|
||||
gate = os.path.join(run_dir, ".cell-gate-%s" % machine)
|
||||
resolv = os.path.join(run_dir, "resolv.conf.%s" % machine)
|
||||
hosts = os.path.join(run_dir, "hosts.%s" % machine)
|
||||
servers = _effective_nameservers(resolv_conf, resolved_conf)
|
||||
with open(resolv, "w") as fh:
|
||||
for server in servers:
|
||||
fh.write("nameserver %s\n" % server)
|
||||
os.chmod(resolv, 0o644)
|
||||
try:
|
||||
with open(hosts_file) as src:
|
||||
content = src.read()
|
||||
except OSError:
|
||||
content = ""
|
||||
if "localhost" not in content:
|
||||
content = "127.0.0.1 localhost\n" + content
|
||||
with open(hosts, "w") as fh:
|
||||
fh.write(content)
|
||||
os.chmod(hosts, 0o644)
|
||||
return {"gate": gate, "resolv": resolv, "hosts": hosts}
|
||||
|
||||
|
||||
def _wait_for_iface(iface, timeout=10.0):
|
||||
deadline = time.time() + timeout
|
||||
while time.time() < deadline:
|
||||
proc = subprocess.run(["ip", "link", "show", iface],
|
||||
capture_output=True)
|
||||
if proc.returncode == 0:
|
||||
return True
|
||||
time.sleep(0.2)
|
||||
return False
|
||||
|
||||
|
||||
def _alloc_lock(lock_path, timeout=30.0):
|
||||
"""拿分配锁(fcntl 排他锁,阻塞拿不到即超时大声失败)。
|
||||
返回锁 fd(调用方负责 close 即解锁)。"""
|
||||
fd = os.open(lock_path, os.O_CREAT | os.O_RDWR, 0o600)
|
||||
deadline = time.time() + timeout
|
||||
while True:
|
||||
try:
|
||||
fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
return fd
|
||||
except BlockingIOError:
|
||||
if time.time() >= deadline:
|
||||
os.close(fd)
|
||||
raise RuntimeError(
|
||||
"chv-cell: 分配锁 %s 拿不到(超时 %.0f 秒)"
|
||||
% (lock_path, timeout))
|
||||
time.sleep(0.05)
|
||||
|
||||
|
||||
def _used_cell_indices(run):
|
||||
"""扫描已占用的 cell 下标:活着的 ve-chv-cell-<xx> 接口名、
|
||||
10.200.<N>.1/24 地址、machinectl 里仍在册的同名 machine
|
||||
(注销滞后时避让,见 RETEST3 新缺陷 1 的 veth 超时分析)。
|
||||
run 供单测注入(subprocess.run 兼容)。machinectl 查不到
|
||||
(无此命令/失败)即跳过该项,不因此失败。"""
|
||||
used = set()
|
||||
proc = run(["ip", "-o", "link", "show"], capture_output=True,
|
||||
text=True)
|
||||
if proc.returncode != 0:
|
||||
raise RuntimeError("chv-cell: 扫描已占用 cell 下标失败:"
|
||||
"ip -o link show")
|
||||
for m in re.finditer(r"\bve-chv-cell-([0-9a-f]{2})\b",
|
||||
proc.stdout or ""):
|
||||
try:
|
||||
used.add(int(m.group(1), 16))
|
||||
except ValueError:
|
||||
continue
|
||||
proc = run(["ip", "-o", "addr", "show"], capture_output=True,
|
||||
text=True)
|
||||
if proc.returncode != 0:
|
||||
raise RuntimeError("chv-cell: 扫描已占用 cell 下标失败:"
|
||||
"ip -o addr show")
|
||||
for m in re.finditer(r"\b10\.200\.(\d+)\.1/24\b",
|
||||
proc.stdout or ""):
|
||||
try:
|
||||
used.add(int(m.group(1)))
|
||||
except ValueError:
|
||||
continue
|
||||
try:
|
||||
proc = run(["machinectl", "list", "--no-legend"],
|
||||
capture_output=True, text=True)
|
||||
if proc.returncode == 0:
|
||||
for m in re.finditer(r"\bchv-cell-([0-9a-f]{2})\b",
|
||||
proc.stdout or ""):
|
||||
try:
|
||||
used.add(int(m.group(1), 16))
|
||||
except ValueError:
|
||||
continue
|
||||
except OSError:
|
||||
pass
|
||||
return used
|
||||
|
||||
|
||||
def allocate_cell_net(run=None, lock_path=ALLOC_LOCK_FILE):
|
||||
"""锁下分配唯一 cell 身份(run_cell 每调一次即分配一个)。
|
||||
取最小空闲下标 0..250;machine 名 chv-cell-<xx>(11 字符,
|
||||
ve- 前缀后 14 字符,符合接口名 15 字符上限);子网
|
||||
10.200.<idx>.0/24、宿主 10.200.<idx>.1、单元 10.200.<idx>.2。
|
||||
全部下标占满即大声失败,不静默回落固定身份。"""
|
||||
run = run or subprocess.run
|
||||
fd = _alloc_lock(lock_path)
|
||||
try:
|
||||
used = _used_cell_indices(run)
|
||||
for index in range(MAX_CELL_INDEX + 1):
|
||||
if index not in used:
|
||||
break
|
||||
else:
|
||||
raise RuntimeError(
|
||||
"chv-cell: cell 下标已占满(0..%d)" % MAX_CELL_INDEX)
|
||||
machine = "chv-cell-%02x" % index
|
||||
return CellNet(
|
||||
index=index,
|
||||
machine=machine,
|
||||
host_iface="ve-%s" % machine,
|
||||
host_ip="%s.%d.1" % (CELL_NET_BASE, index),
|
||||
cell_ip="%s.%d.2" % (CELL_NET_BASE, index),
|
||||
subnet="%s.%d.0/24" % (CELL_NET_BASE, index))
|
||||
finally:
|
||||
try:
|
||||
fcntl.flock(fd, fcntl.LOCK_UN)
|
||||
except OSError:
|
||||
pass
|
||||
os.close(fd)
|
||||
|
||||
|
||||
def _write_cell_pid_files(machine, cell_pid):
|
||||
"""写 pid 文件:每单元一份 cell-<machine>.pid + 兼容旧路径
|
||||
/run/chv/cell.pid(最新单元;P2-9 驱动与 broker 降级路径
|
||||
仍读它)。"""
|
||||
per_cell = os.path.join(BROKER_DIR, "cell-%s.pid" % machine)
|
||||
with open(per_cell, "w") as fh:
|
||||
fh.write(str(cell_pid))
|
||||
with open(CELL_PID_FILE, "w") as fh:
|
||||
fh.write(str(cell_pid))
|
||||
|
||||
|
||||
def _remove_cell_pid_files(machine, cell_pid):
|
||||
"""清 pid 文件(best-effort):每单元份必删;旧路径只在
|
||||
内容仍指向本单元时才删(并发 cell 时别删别人的指针)。"""
|
||||
for path in (os.path.join(BROKER_DIR, "cell-%s.pid" % machine),):
|
||||
try:
|
||||
os.remove(path)
|
||||
except OSError:
|
||||
pass
|
||||
try:
|
||||
with open(CELL_PID_FILE) as fh:
|
||||
if fh.read().strip() == str(cell_pid):
|
||||
os.remove(CELL_PID_FILE)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def _run_cmds(cmds, what):
|
||||
for cmd in cmds:
|
||||
print("chv-cell: + %s" % " ".join(cmd), file=sys.stderr)
|
||||
proc = subprocess.run(cmd)
|
||||
if proc.returncode != 0:
|
||||
raise RuntimeError(
|
||||
"chv-cell: %s 失败(rc=%d):%s"
|
||||
% (what, proc.returncode, " ".join(cmd)))
|
||||
|
||||
|
||||
def run_cell(command, network="veth"):
|
||||
"""起单元、配网、等 payload 退出、清 NAT,返回 payload 退出码。
|
||||
part 4 顺序(RETEST3 新缺陷 1):分配唯一身份 → 准备单元专属
|
||||
文件 → Popen 带起步门的 payload(payload 在门后等着,不与配网
|
||||
并发)→ 等 veth → 客机侧配网 → 轮询找 init pid → 写 pid 文件 →
|
||||
单元内配网 → 开门放行 payload → proc.wait()。payload 已退出
|
||||
(nspawn 起不来等)时返回它的退出码而不是抛错;其余配网失败
|
||||
仍大声抛错,不静默回落共享网络。"""
|
||||
if os.geteuid() != 0:
|
||||
print("chv-cell: 须以 root 运行(nspawn + 配网都要 root)",
|
||||
file=sys.stderr)
|
||||
return 2
|
||||
_ensure_bind_source(WORKSPACE)
|
||||
_ensure_bind_source(CODEX_HOME)
|
||||
_ensure_bind_source(BROKER_DIR, uid=0, gid=0)
|
||||
# part 6:worktree 根目录须在 Popen 前存在,且属 chv(单元内
|
||||
# root 即客机 chv,属主不对就写不进 worktree)。它在 workspace
|
||||
# 绑定之下,进单元后 host.py 的 makedirs 也能建,这里预建只
|
||||
# 是把属主一次定对;不再是任何 bind 的源(无 /app/work 绑定)。
|
||||
_ensure_bind_source(WORKTREE_DIR)
|
||||
net = None
|
||||
files = None
|
||||
uplink = None
|
||||
cell_pid = None
|
||||
proc = None
|
||||
if network == "veth":
|
||||
net = allocate_cell_net()
|
||||
files = prepare_cell_files(net.machine)
|
||||
wrapped = gate_wrapper_command(command, files["gate"])
|
||||
else:
|
||||
wrapped = list(command)
|
||||
argv = build_nspawn_argv(
|
||||
wrapped, network=network, cell_net=net,
|
||||
extra_ro_binds=[(files["resolv"], "/etc/resolv.conf"),
|
||||
(files["hosts"], "/etc/hosts")]
|
||||
if files else None)
|
||||
print("chv-cell: 启动单元:%s" % " ".join(argv), file=sys.stderr)
|
||||
proc = subprocess.Popen(argv)
|
||||
try:
|
||||
if network == "veth":
|
||||
# veth 等待放宽到 20 秒:RETEST3 里连续调用在 10.2/10.3
|
||||
# 秒处超时(疑似 machine 注销滞后);唯一身份 + 起步门
|
||||
# 后配网不再与 payload 赛跑,等久一点无害。
|
||||
if not _wait_for_iface(net.host_iface, timeout=20.0):
|
||||
raise RuntimeError(
|
||||
"chv-cell: veth 接口 %s 未出现,拒绝回落共享网络"
|
||||
% net.host_iface)
|
||||
uplink = find_uplink_iface()
|
||||
if not uplink:
|
||||
raise RuntimeError(
|
||||
"chv-cell: 客机无默认路由,配不出 NAT")
|
||||
_run_cmds(host_network_commands(uplink, net), "客机侧配网")
|
||||
cell_pid = find_cell_init_pid(
|
||||
proc.pid, timeout=10.0,
|
||||
proc_alive=lambda: proc.poll() is None)
|
||||
if not cell_pid:
|
||||
raise RuntimeError("chv-cell: 找不到单元 init pid")
|
||||
_write_cell_pid_files(net.machine, cell_pid)
|
||||
_run_cmds(cell_network_commands(cell_pid, net), "单元内配网")
|
||||
# 配网完成,开门放行 payload。
|
||||
with open(files["gate"], "w") as fh:
|
||||
fh.write("")
|
||||
return proc.wait()
|
||||
except RuntimeError:
|
||||
# nspawn 已死(如 machine 名冲突、argv 错)即返回它的退出
|
||||
# 码,不把“起不来”报成配网失败;还活着才是真配网失败,大声抛。
|
||||
if proc is not None and proc.poll() is not None:
|
||||
return proc.returncode
|
||||
raise
|
||||
finally:
|
||||
if uplink:
|
||||
for cmd in host_network_cleanup_commands(uplink, net):
|
||||
subprocess.run(cmd, capture_output=True)
|
||||
if cell_pid:
|
||||
_remove_cell_pid_files(net.machine, cell_pid)
|
||||
if files:
|
||||
for key in ("gate", "resolv", "hosts"):
|
||||
try:
|
||||
os.remove(files[key])
|
||||
except OSError:
|
||||
pass
|
||||
if proc is not None and proc.poll() is None:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
|
||||
|
||||
def main(argv):
|
||||
if len(argv) < 1:
|
||||
print("用法:chv-cell run|argv|network-commands …",
|
||||
file=sys.stderr)
|
||||
return 2
|
||||
verb = argv[0]
|
||||
rest = argv[1:]
|
||||
network = os.environ.get("CHV_CELL_NETWORK", "veth")
|
||||
# --network 只在 run/argv 前接受,其它位置一律不认。
|
||||
if rest and rest[0] == "--network":
|
||||
if len(rest) < 2:
|
||||
print("chv-cell: --network 缺值", file=sys.stderr)
|
||||
return 2
|
||||
network = rest[1]
|
||||
rest = rest[2:]
|
||||
if network not in ("veth", "shared"):
|
||||
print("chv-cell: network 只能是 veth|shared:%r" % network,
|
||||
file=sys.stderr)
|
||||
return 2
|
||||
if network == "shared":
|
||||
print("chv-cell: 警告:共享网络模式(调试逃生口),单元与"
|
||||
"客机共用网络命名空间", file=sys.stderr)
|
||||
if verb == "run":
|
||||
if not rest or rest[0] != "--":
|
||||
print("chv-cell: run 用法:chv-cell run -- <cmd...>",
|
||||
file=sys.stderr)
|
||||
return 2
|
||||
command = rest[1:]
|
||||
if not command:
|
||||
print("chv-cell: 单元内命令不能为空", file=sys.stderr)
|
||||
return 2
|
||||
try:
|
||||
return run_cell(command, network=network)
|
||||
except RuntimeError as e:
|
||||
print(str(e), file=sys.stderr)
|
||||
return 1
|
||||
if verb == "argv":
|
||||
if not rest or rest[0] != "--":
|
||||
print("chv-cell: argv 用法:chv-cell argv -- <cmd...>",
|
||||
file=sys.stderr)
|
||||
return 2
|
||||
print(json.dumps(build_nspawn_argv(rest[1:], network=network),
|
||||
ensure_ascii=False))
|
||||
return 0
|
||||
if verb == "network-commands":
|
||||
uplink = None
|
||||
cell_pid = None
|
||||
i = 0
|
||||
while i < len(rest):
|
||||
if rest[i] == "--uplink" and i + 1 < len(rest):
|
||||
uplink = rest[i + 1]
|
||||
i += 2
|
||||
elif rest[i] == "--cell-pid" and i + 1 < len(rest):
|
||||
cell_pid = int(rest[i + 1])
|
||||
i += 2
|
||||
else:
|
||||
print("chv-cell: network-commands 未知参数:%s"
|
||||
% rest[i], file=sys.stderr)
|
||||
return 2
|
||||
out = {}
|
||||
if uplink:
|
||||
out["host"] = host_network_commands(uplink)
|
||||
out["host_cleanup"] = host_network_cleanup_commands(uplink)
|
||||
if cell_pid:
|
||||
out["cell"] = cell_network_commands(cell_pid)
|
||||
print(json.dumps(out, ensure_ascii=False))
|
||||
return 0
|
||||
print("chv-cell: 未知子命令:%s" % verb, file=sys.stderr)
|
||||
return 2
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main(sys.argv[1:]))
|
||||
Executable
+149
@@ -0,0 +1,149 @@
|
||||
#!/bin/bash
|
||||
# chv-sync:轻量 VM 里的 Gitea 双向同步(用户工作区口径)。
|
||||
#
|
||||
# 主角是用户,不是 agent:用户在 Gitea 上的仓库在 VM 启动时
|
||||
# clone 到 ~/workspace,用完再同步回 Gitea;VM 是用户的工作区,
|
||||
# Gitea 是权威远端。同步口径沿用 Git Workspace Platform 已定
|
||||
# 的规则:分叉或冲突时失败关闭,不强制覆盖、不自动 rebase、
|
||||
# 不静默丢提交。
|
||||
#
|
||||
# 凭证绝不进镜像。启动时由 seed 盘投递两份文件(0600,root 所有):
|
||||
# /etc/chv/guest-env(非密钥同步变量:GITEA_URL/GITEA_USER/
|
||||
# SYNC_REPO/SYNC_BRANCH/GITEA_EMAIL)与
|
||||
# /etc/chv/authd-seed(feat-chv-muse-p3:只装 GITEA_TOKEN,
|
||||
# 只给客内 authd 守护进程读)。
|
||||
# boot(root,cloud-init runcmd 调)读 guest-env,authd 在时配
|
||||
# git credential helper 走 authd 按次取凭证(只进 git 子进程
|
||||
# 内存,不落盘);authd 不在时直接失败(round-11 真机修复层 5:
|
||||
# 旧镜像的落盘回落已废止——它在真机测试里把 authd 释放的长效
|
||||
# token 写进了客内 ~/.git-credentials;authd helper 是唯一凭证
|
||||
# helper,且自身绝不落盘)。pull/push/status 以 chv 身份跑,用
|
||||
# helper,不再读 guest-env;token 不会出现在 git remote URL 或
|
||||
# 镜像里。
|
||||
#
|
||||
# 用法:
|
||||
# chv-sync boot # 启动时 clone(已 clone 则静默退出,不自动 pull)
|
||||
# chv-sync pull [仓目录] # 脏树拒绝;只接受 fast-forward,分叉即失败
|
||||
# chv-sync push [仓目录] # 普通 push;无 upstream 时 push -u origin HEAD;
|
||||
# # 非 fast-forward 即失败,绝不 --force
|
||||
# chv-sync status [仓目录] # 本地状态 + 最近提交
|
||||
# 仓目录省略时取 $WORKSPACE 下唯一的 git 仓库;多个时必须显式给。
|
||||
set -euo pipefail
|
||||
|
||||
GUEST_ENV="${CHV_GUEST_ENV:-/etc/chv/guest-env}"
|
||||
WORKSPACE="${CHV_WORKSPACE:-/home/chv/workspace}"
|
||||
USER_HOME="${CHV_USER_HOME:-/home/chv}"
|
||||
SYNC_USER="${CHV_USER:-chv}"
|
||||
|
||||
die() { echo "chv-sync: $*" >&2; exit 1; }
|
||||
info() { echo "chv-sync: $*"; }
|
||||
|
||||
# 解析仓目录:显式给则用它;否则取 WORKSPACE 下唯一的 git 仓库。
|
||||
resolve_repo() {
|
||||
if [ -n "${1:-}" ]; then
|
||||
[ -d "$1/.git" ] || die "不是 git 仓库:$1"
|
||||
printf '%s' "$1"
|
||||
return 0
|
||||
fi
|
||||
local found=()
|
||||
local d
|
||||
for d in "$WORKSPACE"/*/; do
|
||||
[ -d "${d}.git" ] && found+=("${d%/}")
|
||||
done
|
||||
if [ "${#found[@]}" -eq 1 ]; then
|
||||
printf '%s' "${found[0]}"
|
||||
return 0
|
||||
fi
|
||||
die "WORKSPACE 下 git 仓库不是唯一(${#found[@]} 个),请显式给仓目录"
|
||||
}
|
||||
|
||||
cmd_boot() {
|
||||
# 无 guest-env = 本次启动不同步任何仓,静默成功(VM 照常用)。
|
||||
[ -f "$GUEST_ENV" ] || { info "无 $GUEST_ENV,本次启动不同步"; exit 0; }
|
||||
# shellcheck disable=SC1090
|
||||
set -a; . "$GUEST_ENV"; set +a
|
||||
: "${GITEA_URL:?guest-env 缺 GITEA_URL}"
|
||||
: "${GITEA_USER:?guest-env 缺 GITEA_USER}"
|
||||
: "${SYNC_REPO:?guest-env 缺 SYNC_REPO}"
|
||||
[ "$(id -u)" = "0" ] || die "boot 须以 root 运行(cloud-init runcmd)"
|
||||
local name="${SYNC_REPO##*/}"
|
||||
local dir="$WORKSPACE/$name"
|
||||
mkdir -p "$WORKSPACE"
|
||||
chown "$SYNC_USER:$SYNC_USER" "$WORKSPACE"
|
||||
# feat-chv-muse-p3:authd 路径——socket 存在时凭证走 helper,
|
||||
# 不再把真 token 写进 .git-credentials 落盘;helper 按次向
|
||||
# authd 取凭证,只进 git 子进程内存。socket 不存在时直接失败
|
||||
#(round-11 真机修复层 5:落盘回落已废止,见下)。
|
||||
# 注意:authd 路径下 GITEA_TOKEN 不在 guest-env 里(它只进
|
||||
# authd-seed 给守护进程),所以这里不要求它。
|
||||
local authd_socket="${CHV_AUTHD_SOCKET:-/run/chv-authd/authd.sock}"
|
||||
local helper="/usr/local/bin/git-credential-chv-authd"
|
||||
if [ -S "$authd_socket" ] && [ -x "$helper" ]; then
|
||||
runuser -u "$SYNC_USER" -- env HOME="$USER_HOME" git config --global credential.helper "$helper"
|
||||
# helper 按路径推断仓库与操作(Sentinel L7 绑定),git
|
||||
# 默认不把 path 发给 helper(须 credential.useHttpPath),
|
||||
# 不开它每次都会因“请求不完整”拒绝——新测试抓到的真问题。
|
||||
runuser -u "$SYNC_USER" -- env HOME="$USER_HOME" git config --global credential.useHttpPath true
|
||||
info "凭证走 chv-authd(不落盘)"
|
||||
else
|
||||
# round-11 真机修复(层 5):authd 不在时直接失败,不再
|
||||
# 把真 token 写进 ~/.git-credentials 落盘——旧路径在
|
||||
# round-11 真机测试里把 authd 释放的长效 token 写进了
|
||||
# 客内磁盘(/home/chv/.git-credentials,0600)。authd
|
||||
# helper 是唯一凭证 helper,且自身绝不落盘;旧镜像退役
|
||||
# 后过渡兼容即废止,不再留落盘逃生口。
|
||||
die "chv-authd 未运行(socket 或 helper 缺失),拒绝同步:凭证绝不落盘,请用含 authd 的镜像"
|
||||
fi
|
||||
if [ -d "$dir/.git" ]; then
|
||||
info "已 clone:$dir(不自动 pull,改动由用户显式同步)"
|
||||
exit 0
|
||||
fi
|
||||
[ -e "$dir" ] && die "$dir 已存在但不是 git 仓库,请人工处理"
|
||||
local clone_args=(clone)
|
||||
[ -n "${SYNC_BRANCH:-}" ] && clone_args+=(--branch "$SYNC_BRANCH")
|
||||
clone_args+=("$GITEA_URL/$SYNC_REPO.git" "$dir")
|
||||
runuser -u "$SYNC_USER" -- env HOME="$USER_HOME" git "${clone_args[@]}"
|
||||
runuser -u "$SYNC_USER" -- env HOME="$USER_HOME" git -C "$dir" config user.name "$GITEA_USER"
|
||||
if [ -n "${GITEA_EMAIL:-}" ]; then
|
||||
runuser -u "$SYNC_USER" -- env HOME="$USER_HOME" git -C "$dir" config user.email "$GITEA_EMAIL"
|
||||
fi
|
||||
info "已 clone $SYNC_REPO 到 $dir,HEAD=$(runuser -u "$SYNC_USER" -- env HOME="$USER_HOME" git -C "$dir" rev-parse --short HEAD)"
|
||||
}
|
||||
|
||||
cmd_pull() {
|
||||
local dir
|
||||
dir="$(resolve_repo "${1:-}")"
|
||||
[ -z "$(git -C "$dir" status --porcelain)" ] \
|
||||
|| die "工作树有未提交改动,拒绝 pull(先 commit 或 stash)"
|
||||
git -C "$dir" pull --ff-only \
|
||||
|| die "pull 不是 fast-forward(本地与远端已分叉),请人工处理,不自动 rebase/merge"
|
||||
info "已同步:$dir HEAD=$(git -C "$dir" rev-parse --short HEAD)"
|
||||
}
|
||||
|
||||
cmd_push() {
|
||||
local dir
|
||||
dir="$(resolve_repo "${1:-}")"
|
||||
if git -C "$dir" rev-parse --abbrev-ref --symbolic-full-name '@{u}' >/dev/null 2>&1; then
|
||||
git -C "$dir" push \
|
||||
|| die "push 被拒绝(远端已有新提交,先 pull 再 push,不 --force)"
|
||||
else
|
||||
git -C "$dir" push -u origin HEAD \
|
||||
|| die "push 被拒绝(远端已有新提交,先 pull 再 push,不 --force)"
|
||||
fi
|
||||
info "已 push:$dir HEAD=$(git -C "$dir" rev-parse --short HEAD)"
|
||||
}
|
||||
|
||||
cmd_status() {
|
||||
local dir
|
||||
dir="$(resolve_repo "${1:-}")"
|
||||
git -C "$dir" status -sb
|
||||
git -C "$dir" log --oneline -5
|
||||
}
|
||||
|
||||
case "${1:-}" in
|
||||
boot) shift; cmd_boot "$@" ;;
|
||||
pull) shift; cmd_pull "$@" ;;
|
||||
push) shift; cmd_push "$@" ;;
|
||||
status) shift; cmd_status "$@" ;;
|
||||
*) echo "用法:chv-sync boot|pull|push|status [仓目录]" >&2; exit 2 ;;
|
||||
esac
|
||||
+122
-52
@@ -1,67 +1,107 @@
|
||||
#!/usr/bin/env python3
|
||||
"""在构建宿主上生成 CHV 客机镜像的供应链 lockfile(to10 轨)。
|
||||
"""在构建宿主上生成 CHV 客机镜像的供应链 lockfile。
|
||||
|
||||
事后校验见 check_lockfile.py(校验 BUILDINFO.versions);
|
||||
本脚本是事前/事中生成端:在构建宿主上跑,读 BUILDINFO
|
||||
的 tag 与 contents(仅名字清单,不含版本号),逐项从本
|
||||
机 dpkg / pip 查真实版本,写出 lockfile.json:
|
||||
本脚本是生成端:读 BUILDINFO.json,以其中的 versions
|
||||
(build.sh 已从客机读回的 dpkg-query / pip freeze,
|
||||
Playwright=1 时另有 chromium 标识)为唯一版本来源,
|
||||
逐项给 contents 清单配版本,写出 lockfile.json:
|
||||
|
||||
{"tag": ..., "image_sha256": ... | null,
|
||||
"generated_at": UTC ISO 时间,
|
||||
"versions": {"dpkg": ["name version", ...],
|
||||
"pip": ["name version", ...]},
|
||||
"pip": ["name version", ...],
|
||||
"declared": ["name version", ...]},
|
||||
"unresolved": ["查不到版本的名", ...]}
|
||||
|
||||
绝不编造版本号:查不到的进 unresolved,不猜、不填占位
|
||||
版本。--verify 模式校验一份 lockfile:tag 与 BUILDINFO
|
||||
一致、dpkg/pip 非空、每条都有非空版本、unresolved 为空;
|
||||
任一不满足即退出非 0(unresolved 非空 = 清单不完整,
|
||||
不是“生成成功”)。
|
||||
版本来源映射(绝不编造版本号,查不到的进 unresolved):
|
||||
- dpkg 包名 → versions.dpkg("name version" 行)
|
||||
- pip 包名 → versions.pip("name==version" 行)
|
||||
- codex / codex-code-mode-host / codex-resources/bwrap
|
||||
→ BUILDINFO.codex_version(随 codex 一起发布)
|
||||
- /app/host.py → BUILDINFO.repo_commit
|
||||
- chromium / chromium-headless-shell → versions.chromium
|
||||
(ms-playwright 缓存目录名)
|
||||
|
||||
注意:旧版逐项查构建宿主的 dpkg/pip,方法不成立——宿主
|
||||
版本与客机版本没有必然关系,且 /app/host.py、codex 三件
|
||||
套在宿主上永远查不到,--verify 永远过不了。本版只读
|
||||
BUILDINFO.versions,不再现场查宿主。
|
||||
|
||||
用法(均在构建宿主上跑):
|
||||
python3 generate_lockfile.py BUILDINFO.json --output lockfile.json
|
||||
python3 generate_lockfile.py --verify lockfile.json \
|
||||
--buildinfo BUILDINFO.json
|
||||
本轨无构建宿主:脚本已落地且可跑(见回执里的 /tmp 合成
|
||||
BUILDINFO 实测),真实清单待构建宿主产出,绝不把合成
|
||||
数据写成生产清单。
|
||||
"""
|
||||
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
|
||||
|
||||
def _dpkg_version(name):
|
||||
"""dpkg 查版本,查不到回 None(不抛)。"""
|
||||
try:
|
||||
out = subprocess.run(
|
||||
["dpkg-query", "-W", "-f=${Version}", name],
|
||||
capture_output=True, text=True, timeout=30)
|
||||
except Exception:
|
||||
return None
|
||||
if out.returncode != 0:
|
||||
return None
|
||||
version = out.stdout.strip()
|
||||
return version or None
|
||||
# 非包名的内容项:版本来自 BUILDINFO 的声明字段,不是 dpkg/pip。
|
||||
DECLARED_SOURCES = {
|
||||
"codex": "codex_version",
|
||||
"codex-code-mode-host": "codex_version",
|
||||
"codex-resources/bwrap": "codex_version",
|
||||
"/app/host.py": "repo_commit",
|
||||
"gt": "gt_version",
|
||||
"chv-sync": "repo_commit",
|
||||
# 对齐 Muse P2:nspawn 单元启动器 + broker + 浏览器子代理,
|
||||
# 均是仓库内脚本,版本随 repo_commit(与 chv-sync 同款)。
|
||||
"chv-cell": "repo_commit",
|
||||
"chv-broker": "repo_commit",
|
||||
"chv-browser": "repo_commit",
|
||||
# 对齐 Muse P3 + P1 vsock:authd 三件套、authd/sentinel
|
||||
# 叶子模块与 vsock 通知脚本,均是仓库内脚本,版本随
|
||||
# repo_commit。缺了它们 generate 会记 6 项 unresolved,
|
||||
# verify 与 check 双双失败(717efc4 真机实测)。
|
||||
# round-11 真机修复(层 1):叶子树新增 errors.py 与 stub
|
||||
# __init__.py(authd_init_stub.py),同款 repo_commit 登记,
|
||||
# 缺了它们新镜像的 lockfile 同样会 unresolved。
|
||||
"chv-authd": "repo_commit",
|
||||
"git-credential-chv-authd": "repo_commit",
|
||||
"chv-authd.service": "repo_commit",
|
||||
"cloud/chv/authd.py": "repo_commit",
|
||||
"cloud/chv/sentinel.py": "repo_commit",
|
||||
"cloud/chv/errors.py": "repo_commit",
|
||||
"cloud/chv/__init__.py": "repo_commit",
|
||||
"vsock_notify": "repo_commit",
|
||||
}
|
||||
|
||||
|
||||
def _pip_version(name):
|
||||
"""pip 查版本,查不到回 None(不抛)。"""
|
||||
try:
|
||||
out = subprocess.run(
|
||||
[sys.executable, "-m", "pip", "show", name],
|
||||
capture_output=True, text=True, timeout=30)
|
||||
except Exception:
|
||||
return None
|
||||
if out.returncode != 0:
|
||||
return None
|
||||
for line in out.stdout.splitlines():
|
||||
if line.startswith("Version:"):
|
||||
version = line.split(":", 1)[1].strip()
|
||||
return version or None
|
||||
return None
|
||||
def _index_dpkg(lines):
|
||||
"""dpkg 行 "name version" → {小写名: "name version"}。"""
|
||||
index = {}
|
||||
for line in lines or []:
|
||||
parts = str(line).split(None, 1)
|
||||
if len(parts) == 2:
|
||||
index[parts[0].lower()] = "%s %s" % (parts[0], parts[1])
|
||||
return index
|
||||
|
||||
|
||||
def _index_pip(lines):
|
||||
"""pip freeze 行 "name==version" → {小写名: "name version"}。"""
|
||||
index = {}
|
||||
for line in lines or []:
|
||||
if "==" in str(line):
|
||||
name, version = str(line).split("==", 1)
|
||||
index[name.lower()] = "%s %s" % (name, version.strip())
|
||||
return index
|
||||
|
||||
|
||||
def _index_chromium(lines):
|
||||
"""chromium 标识行 → 键。完整 Chromium 的目录名是
|
||||
chromium-<rev>,headless-shell 的是 chromium_headless_shell-<rev>,
|
||||
各自映射到同名 contents 项。"""
|
||||
index = {}
|
||||
for line in lines or []:
|
||||
text = str(line).strip()
|
||||
if text.startswith("chromium_headless_shell"):
|
||||
index["chromium-headless-shell"] = \
|
||||
"chromium-headless-shell %s" % text
|
||||
elif text:
|
||||
index["chromium"] = "chromium %s" % text
|
||||
return index
|
||||
|
||||
|
||||
def generate(buildinfo_path, output_path):
|
||||
@@ -71,15 +111,28 @@ def generate(buildinfo_path, output_path):
|
||||
if not isinstance(names, list):
|
||||
print("BUILDINFO.contents 不是列表", file=sys.stderr)
|
||||
return 2
|
||||
dpkg, pip, unresolved = [], [], []
|
||||
versions = buildinfo.get("versions") or {}
|
||||
dpkg_index = _index_dpkg(versions.get("dpkg"))
|
||||
pip_index = _index_pip(versions.get("pip"))
|
||||
chromium_index = _index_chromium(versions.get("chromium"))
|
||||
dpkg, pip, declared, unresolved = [], [], [], []
|
||||
for name in names:
|
||||
version = _dpkg_version(name)
|
||||
if version is not None:
|
||||
dpkg.append("%s %s" % (name, version))
|
||||
key = name.lower()
|
||||
if name in DECLARED_SOURCES:
|
||||
version = buildinfo.get(DECLARED_SOURCES[name])
|
||||
if version:
|
||||
declared.append("%s %s" % (name, version))
|
||||
else:
|
||||
unresolved.append(name)
|
||||
continue
|
||||
version = _pip_version(name)
|
||||
if version is not None:
|
||||
pip.append("%s %s" % (name, version))
|
||||
if key in chromium_index:
|
||||
declared.append(chromium_index[key])
|
||||
continue
|
||||
if key in dpkg_index:
|
||||
dpkg.append(dpkg_index[key])
|
||||
continue
|
||||
if key in pip_index:
|
||||
pip.append(pip_index[key])
|
||||
continue
|
||||
unresolved.append(name)
|
||||
lockfile = {
|
||||
@@ -87,7 +140,8 @@ def generate(buildinfo_path, output_path):
|
||||
"image_sha256": buildinfo.get("image_sha256")
|
||||
or buildinfo.get("sha256"),
|
||||
"generated_at": datetime.now(timezone.utc).isoformat(),
|
||||
"versions": {"dpkg": sorted(dpkg), "pip": sorted(pip)},
|
||||
"versions": {"dpkg": sorted(dpkg), "pip": sorted(pip),
|
||||
"declared": sorted(declared)},
|
||||
"unresolved": sorted(unresolved),
|
||||
}
|
||||
with open(output_path, "w") as fh:
|
||||
@@ -111,6 +165,21 @@ def verify(lockfile_path, buildinfo_path):
|
||||
% (lockfile.get("tag"), buildinfo.get("tag")),
|
||||
file=sys.stderr)
|
||||
return 1
|
||||
# feat-chv-closeup(M4):钉死的 Chromium 修订号须与
|
||||
# lockfile 里记的 chromium-headless-shell 条目一致;
|
||||
# BUILDINFO 没钉(pinned_rev 为空)时不校验。
|
||||
pinned = buildinfo.get("chromium_pinned_rev")
|
||||
if pinned:
|
||||
expected = "chromium_headless_shell-%s" % pinned
|
||||
declared = (lockfile.get("versions") or {}).get(
|
||||
"declared") or []
|
||||
hit = [e for e in declared
|
||||
if str(e).startswith("chromium-headless-shell ")]
|
||||
if not hit or expected not in str(hit[0]):
|
||||
print("Chromium 修订号与钉死值不符:lockfile %s,"
|
||||
"钉死 chromium_headless_shell-%s"
|
||||
% (hit, pinned), file=sys.stderr)
|
||||
return 1
|
||||
versions = lockfile.get("versions") or {}
|
||||
for kind in ("dpkg", "pip"):
|
||||
entries = versions.get(kind)
|
||||
@@ -129,9 +198,10 @@ def verify(lockfile_path, buildinfo_path):
|
||||
print("清单不完整,unresolved 非空:%s" % sorted(unresolved),
|
||||
file=sys.stderr)
|
||||
return 1
|
||||
print("lockfile 校验通过:tag=%s dpkg=%d pip=%d"
|
||||
print("lockfile 校验通过:tag=%s dpkg=%d pip=%d declared=%d"
|
||||
% (lockfile.get("tag"), len(versions["dpkg"]),
|
||||
len(versions["pip"])))
|
||||
len(versions["pip"]),
|
||||
len(versions.get("declared") or [])))
|
||||
return 0
|
||||
|
||||
|
||||
|
||||
Executable
+107
@@ -0,0 +1,107 @@
|
||||
#!/usr/bin/env python3
|
||||
"""git credential helper:向 chv-authd 按次取凭证(feat-chv-muse-p3)。
|
||||
|
||||
git credential 协议:git 把 `protocol/host/path`(每行 `key=value`,
|
||||
空行结束)写进本进程 stdin;本进程向 authd 的 Unix socket 发
|
||||
一次 JSON 请求,拿到用户名/口令后按协议回给 git(属性 + 空行)。
|
||||
|
||||
口令只进 git 子进程内存:本进程不写文件、不打日志(拒绝时只
|
||||
报 kind,不含值);`password_expiry_utc` 按释放的 expires_at
|
||||
回给 git,git 在该时间后重新问 helper(不缓存过期凭证)。
|
||||
authd 拒绝 / socket 不通时本进程 exit 1——git 操作随之失败
|
||||
(fail-closed,绝不回落成“无凭证也照推”)。
|
||||
|
||||
用法(chv-sync boot 配好后 git 自动调,不用手工跑):
|
||||
git config --global credential.helper /usr/local/bin/git-credential-chv-authd
|
||||
git config --global credential.useHttpPath true
|
||||
(后者不可少:git 默认不把 path 发给 helper,没有 path 本进程
|
||||
无法推断仓库与操作,会直接拒绝。)
|
||||
socket 路径可用 CHV_AUTHD_SOCKET 覆盖(测试用 /tmp 下的假 socket)。
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import socket
|
||||
import sys
|
||||
|
||||
DEFAULT_SOCKET_PATH = "/run/chv-authd/authd.sock"
|
||||
SOCKET_TIMEOUT_SECONDS = 5
|
||||
|
||||
|
||||
def _read_credential_request():
|
||||
"""读 git credential 协议的请求属性(空行结束)→ dict。"""
|
||||
attrs = {}
|
||||
for line in sys.stdin:
|
||||
line = line.rstrip("\n")
|
||||
if not line:
|
||||
break
|
||||
if "=" in line:
|
||||
key, value = line.split("=", 1)
|
||||
attrs[key.strip()] = value.strip()
|
||||
return attrs
|
||||
|
||||
|
||||
def _ask_authd(socket_path, host, path):
|
||||
"""向 authd 要一次凭证:返回 (username, password, expires_at)。
|
||||
拒绝/通信失败即 RuntimeError(只含 kind,不含值)。"""
|
||||
req = {"connector": "gitea", "host": host, "path": path}
|
||||
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
|
||||
sock.settimeout(SOCKET_TIMEOUT_SECONDS)
|
||||
try:
|
||||
sock.connect(socket_path)
|
||||
sock.sendall((json.dumps(req) + "\n").encode("utf-8"))
|
||||
chunks = []
|
||||
while True:
|
||||
data = sock.recv(65536)
|
||||
if not data:
|
||||
break
|
||||
chunks.append(data)
|
||||
if b"\n" in data:
|
||||
break
|
||||
except OSError as exc:
|
||||
raise RuntimeError("authd_unreachable:%s" % exc)
|
||||
finally:
|
||||
sock.close()
|
||||
try:
|
||||
resp = json.loads(b"".join(chunks).decode("utf-8"))
|
||||
except ValueError:
|
||||
raise RuntimeError("authd_bad_response")
|
||||
if not resp.get("ok"):
|
||||
raise RuntimeError("authd_refused:%s"
|
||||
% resp.get("error", "refused"))
|
||||
return (resp["username"], resp["password"],
|
||||
resp.get("expires_at"))
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
"""helper 入口:读协议 → 问 authd → 回属性。拒绝即 exit 1。"""
|
||||
attrs = _read_credential_request()
|
||||
host = attrs.get("host", "")
|
||||
path = attrs.get("path", "")
|
||||
if attrs.get("protocol") != "https" or not host or not path:
|
||||
print("git-credential-chv-authd: 请求不完整(须 protocol=https "
|
||||
"+ host + path),拒绝", file=sys.stderr)
|
||||
return 1
|
||||
if not path.startswith("/"):
|
||||
path = "/" + path
|
||||
# git 的 info/refs 路径带 ?service=... 查询串:Sentinel 的
|
||||
# L7 规则按干净路径匹配,这里先去掉查询串再问 authd。
|
||||
path = path.split("?", 1)[0]
|
||||
socket_path = os.environ.get("CHV_AUTHD_SOCKET") \
|
||||
or DEFAULT_SOCKET_PATH
|
||||
try:
|
||||
username, password, expires_at = _ask_authd(
|
||||
socket_path, host, path)
|
||||
except RuntimeError as exc:
|
||||
print("git-credential-chv-authd: %s" % exc, file=sys.stderr)
|
||||
return 1
|
||||
print("username=%s" % username)
|
||||
print("password=%s" % password)
|
||||
if expires_at:
|
||||
print("password_expiry_utc=%d" % int(expires_at))
|
||||
print()
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
+20
-1
@@ -33,6 +33,15 @@ def _read_header(sock):
|
||||
return header, buf[header_end + 4:]
|
||||
|
||||
|
||||
def _parse_status(header: str) -> int:
|
||||
"""从响应状态行取状态码(解析失败即 0,与
|
||||
_parse_response 的口径一致)。"""
|
||||
try:
|
||||
return int(header.split()[1])
|
||||
except (IndexError, ValueError):
|
||||
return 0
|
||||
|
||||
|
||||
def _parse_content_length(header):
|
||||
"""从响应头取 Content-Length:chunked 或无长度即拒绝。
|
||||
|
||||
@@ -102,7 +111,13 @@ def _unix_socket_api(socket_path, method, path, payload=None):
|
||||
body 能解析 JSON 时为 dict,否则为原文本。socket 不存在
|
||||
或连接失败时 raise RuntimeError(调用方据此判 api 超时
|
||||
/失败,不编造 200)。读满语义见 _read_header /
|
||||
_parse_content_length / _read_body(P1-6,纯移动)。"""
|
||||
_parse_content_length / _read_body(P1-6,纯移动)。
|
||||
204/304 例外:按 HTTP 语义体长恒为 0,不要求
|
||||
Content-Length、也不读体——CHV 的 vm.create/vm.boot
|
||||
成功即回 204 且不带该头(2026-11-07 WSL 真机实测:
|
||||
vmm.ping 200 带长度、vm.create 204 无长度),严格
|
||||
校验会把合法成功当失败拒掉;其它状态仍严格(无长度
|
||||
/截断/chunked 均拒绝)。"""
|
||||
if not os.path.exists(socket_path):
|
||||
raise RuntimeError("api socket 不存在:%s" % socket_path)
|
||||
body = b""
|
||||
@@ -121,6 +136,10 @@ def _unix_socket_api(socket_path, method, path, payload=None):
|
||||
sock.connect(socket_path)
|
||||
sock.sendall(raw)
|
||||
header, body = _read_header(sock)
|
||||
if _parse_status(header) in (204, 304):
|
||||
# 体长恒为 0:不读体(keep-alive 下读会等到超时)。
|
||||
body = b""
|
||||
else:
|
||||
content_length = _parse_content_length(header)
|
||||
body = _read_body(sock, body, content_length)
|
||||
finally:
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
"""CHV 宿主侧小文件的原子写(P2-18 收尾)。
|
||||
|
||||
此前 net.json / snapshots.json / pool registry /
|
||||
known_hosts 各自用固定名 `<path>.tmp` + 直接 os.replace,
|
||||
存在两个问题:
|
||||
|
||||
1. 固定 tmp 名:同一进程内两个写者(或一次崩溃残留的
|
||||
.tmp)会互相覆盖/误读;不同 VM 的文件虽在不同目录,
|
||||
但同一目录内的快照登记与 net 登记仍共用可预测的
|
||||
tmp 名,崩溃后残留的 .tmp 会被下一次写静默覆盖,
|
||||
丢了“上次写了一半”的信号。
|
||||
2. 无 fsync:os.replace 只保证目录项原子,不保证内容
|
||||
已落盘;宿主断电/崩溃后新文件可能回到旧内容或
|
||||
0 字节,而调用方已认为登记成功。
|
||||
|
||||
本模块是唯一写路径:mkstemp 随机名(0600)→ 写 →
|
||||
flush + fsync 文件 → os.replace → fsync 父目录。
|
||||
写失败时删临时文件再原样抛错,调用方按各自语义
|
||||
处理(审计/拒绝/脏标记),本模块不吞错、不记日志。
|
||||
"""
|
||||
|
||||
import os
|
||||
import tempfile
|
||||
|
||||
|
||||
def atomic_write(path: str, data: bytes, mode: int = 0o600) -> None:
|
||||
"""原子写 bytes 到 path(随机 tmp 名 + 双 fsync)。
|
||||
|
||||
mode 是最终文件的权限(默认 0600:这些文件都在
|
||||
0700 的 state_dir 内、且含登记/密钥类内容,按敏感
|
||||
文件对待;调用方要放宽时显式传)。
|
||||
"""
|
||||
directory = os.path.dirname(path) or "."
|
||||
fd, tmp_path = tempfile.mkstemp(prefix=".tmp-", dir=directory)
|
||||
try:
|
||||
os.fchmod(fd, mode)
|
||||
with os.fdopen(fd, "wb") as fh:
|
||||
fh.write(data)
|
||||
fh.flush()
|
||||
os.fsync(fh.fileno())
|
||||
os.replace(tmp_path, path)
|
||||
_fsync_dir(directory)
|
||||
except BaseException:
|
||||
try:
|
||||
os.unlink(tmp_path)
|
||||
except OSError:
|
||||
pass
|
||||
raise
|
||||
|
||||
|
||||
def atomic_write_text(path: str, text: str, mode: int = 0o600) -> None:
|
||||
"""atomic_write 的 str 版(UTF-8 编码)。"""
|
||||
atomic_write(path, text.encode("utf-8"), mode=mode)
|
||||
|
||||
|
||||
def _fsync_dir(directory: str) -> None:
|
||||
"""fsync 父目录,让 rename 本身落盘(best-effort:
|
||||
某些文件系统不支持目录 fsync,失败即跳过——文件
|
||||
内容的 fsync 已在 atomic_write 里做完,目录 fsync
|
||||
丢了只影响“rename 是否在崩溃后可见”,不影响内容)。
|
||||
"""
|
||||
try:
|
||||
dir_fd = os.open(directory, os.O_RDONLY)
|
||||
except OSError:
|
||||
return
|
||||
try:
|
||||
os.fsync(dir_fd)
|
||||
except OSError:
|
||||
pass
|
||||
finally:
|
||||
os.close(dir_fd)
|
||||
@@ -0,0 +1,311 @@
|
||||
"""authd 式替身凭证服务(feat-chv-muse-p3,叶子模块)。
|
||||
|
||||
对齐 Muse 官方安全文(security.muse.ai 2026-09-08,见
|
||||
docs/chv-muse-p3-gap.md):凭证由 VM 内独立服务 authd 保管
|
||||
(含 OAuth token,存于用户 VM 而非集中基础设施),代理只见
|
||||
替身令牌;内置连接器的执行分离——每个 privsep worker 有明确
|
||||
凭证白名单(日历 worker 不能仅改参数就索取邮件凭证);登录
|
||||
凭证存入运行时单元之外的安全凭证存储,代理不可见,仅在需要
|
||||
时注入。
|
||||
|
||||
本模块是纯逻辑(只用标准库,不依赖 cloud.*),宿主侧与客内
|
||||
authd 守护进程共用同一份代码(见 chv-guest/authd.py):
|
||||
- `CredentialVault`:只活内存的保险库;`store()` 只接受白名单
|
||||
内的(连接器,凭证键);`release()` 按次发放,绑定
|
||||
{连接器, 主机, 仓库路径, 操作},单次有效 + 300 秒 TTL,
|
||||
过期即废;`__repr__` 与审计记录里绝无凭证值。
|
||||
- 替身的两档(见差距表 §1):本期默认是“按次释放”——释放给
|
||||
git 子进程的值仍是真 token,但只进 git 进程内存、不进 agent
|
||||
环境/argv/日志/落盘;真铸 token(Gitea API 铸短期专用 token)
|
||||
由 `mint_gitea_token()` 实现,默认关闭(须真 token 有建 token
|
||||
权限,现有 write:repository 测试 token 铸 token 会失败)。
|
||||
- 位置决策:authd 跑在客内、单元外(root 身份的 systemd 服务),
|
||||
不在宿主侧——Muse 的 authd 就在用户 VM 里;宿主侧方案要走
|
||||
vsock(P1 范围、未就绪),且会把全部 VM 的 token 集中到宿主。
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import hmac
|
||||
import json
|
||||
import secrets
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from collections.abc import Callable, Iterable
|
||||
|
||||
# 连接器白名单(privsep 口径):本期只认 gitea 一个连接器。
|
||||
# 未知连接器一律拒绝,不得以“默认放行”扩展。
|
||||
ALLOWED_CONNECTORS = frozenset({"gitea"})
|
||||
|
||||
# 每连接器的凭证白名单:连接器 → 允许的凭证键。gitea 只允许
|
||||
# GITEA_TOKEN;其它键(哪怕调用方传进来)一律拒绝入库。
|
||||
CONNECTOR_CREDENTIALS = {
|
||||
"gitea": frozenset({"GITEA_TOKEN"}),
|
||||
}
|
||||
|
||||
# 按次释放的默认有效期(秒)与次数:单次、300 秒,过期即废。
|
||||
DEFAULT_RELEASE_TTL_SECONDS = 300
|
||||
DEFAULT_RELEASE_MAX_USES = 1
|
||||
|
||||
|
||||
class AuthdRefusal(Exception):
|
||||
"""authd 拒绝:kind + 原因,不含任何凭证值。"""
|
||||
|
||||
def __init__(self, kind: str, message: str) -> None:
|
||||
super().__init__("authd_refused:%s: %s" % (kind, message))
|
||||
self.kind = kind
|
||||
|
||||
|
||||
class SurrogateRelease:
|
||||
"""一次按次释放(替身):绑定到一次具体的凭证使用。
|
||||
|
||||
release_id 是不透明 id(审计与吊销用,不含值);value 只在
|
||||
释放对象里出现一次,由调用方(客内守护进程)经 Unix socket
|
||||
交给 git credential helper、再由 helper 交给 git 子进程——
|
||||
除此之外不进任何环境、argv、日志、审计。`consume()` 校验
|
||||
绑定与有效期,单次有效:用过一次即废,过期即废。"""
|
||||
|
||||
def __init__(self, release_id: str, connector: str, key: str,
|
||||
host: str, repo: str, operation: str, value: str,
|
||||
expires_at: float, max_uses: int = 1) -> None:
|
||||
self.release_id = release_id
|
||||
self.connector = connector
|
||||
self.key = key
|
||||
self.host = host
|
||||
self.repo = repo
|
||||
self.operation = operation
|
||||
self.value = value
|
||||
self.expires_at = expires_at
|
||||
self.uses_left = max_uses
|
||||
|
||||
def describe(self) -> dict:
|
||||
"""审计用描述(只含绑定,不含值)。"""
|
||||
return {"release_id": self.release_id,
|
||||
"connector": self.connector, "key": self.key,
|
||||
"host": self.host, "repo": self.repo,
|
||||
"operation": self.operation,
|
||||
"expires_at": self.expires_at,
|
||||
"uses_left": self.uses_left}
|
||||
|
||||
def __repr__(self) -> str: # 绝不含 value
|
||||
return ("SurrogateRelease(id=%s connector=%s host=%s "
|
||||
"repo=%s operation=%s)"
|
||||
% (self.release_id, self.connector, self.host,
|
||||
self.repo, self.operation))
|
||||
|
||||
|
||||
class CredentialVault:
|
||||
"""只活内存的凭证保险库(一个 VM 一个实例,客内 authd 持有)。
|
||||
|
||||
- 值只存在 `self._store` 的内存 dict 里,不落盘、不进日志;
|
||||
- `store()` 校验白名单:未知连接器 / 该连接器不允许的键 /
|
||||
空值一律拒绝入库(AuthdRefusal),绝不静默收下;
|
||||
- `release()` 按次发放:调用方须给出完整绑定(主机、仓库、
|
||||
操作),绑定与入库时的作用域不一致即拒绝;
|
||||
- `consume()` 单次兑现:release 兑现一次即从库里删除,过期
|
||||
也兑现不了;兑现失败不留可重试的残值。
|
||||
clock 可注入(测试用假时钟测过期),默认 time.time。"""
|
||||
|
||||
def __init__(self, clock: Callable[[], float] | None = None) -> None:
|
||||
self._store: dict[tuple[str, str], dict] = {}
|
||||
self._releases: dict[str, SurrogateRelease] = {}
|
||||
self._clock = clock or time.time
|
||||
|
||||
def __repr__(self) -> str: # 绝不含任何凭证值
|
||||
return "CredentialVault(connectors=%s)" % sorted(
|
||||
{c for (c, _k) in self._store})
|
||||
|
||||
def store(self, connector: str, key: str, value: str,
|
||||
host: str | None = None, repo: str | None = None) -> None:
|
||||
"""入库:白名单校验 + 作用域记录(host/repo 是该凭证
|
||||
允许使用的范围,release 时须一致)。"""
|
||||
if connector not in ALLOWED_CONNECTORS:
|
||||
raise AuthdRefusal(
|
||||
"unknown_connector",
|
||||
"连接器 %s 不在 authd 白名单内" % connector)
|
||||
allowed = CONNECTOR_CREDENTIALS.get(connector, frozenset())
|
||||
if key not in allowed:
|
||||
raise AuthdRefusal(
|
||||
"credential_not_allowlisted",
|
||||
"连接器 %s 不允许凭证键 %s" % (connector, key))
|
||||
if not isinstance(value, str) or not value.strip():
|
||||
raise AuthdRefusal(
|
||||
"empty_credential", "凭证值为空,拒绝入库")
|
||||
self._store[(connector, key)] = {
|
||||
"value": value, "host": host, "repo": repo}
|
||||
|
||||
def has(self, connector: str, key: str) -> bool:
|
||||
"""该(连接器,键)是否已入库(不返回值)。"""
|
||||
return (connector, key) in self._store
|
||||
|
||||
def forget(self, connector: str, key: str) -> bool:
|
||||
"""删一条入库凭证(VM 停止时调用);同时作废其全部未兑现
|
||||
释放。未知键返回 False,不抛错。"""
|
||||
gone = self._store.pop((connector, key), None) is not None
|
||||
if gone:
|
||||
doomed = [rid for rid, rel in self._releases.items()
|
||||
if rel.connector == connector
|
||||
and rel.key == key]
|
||||
for rid in doomed:
|
||||
self._releases.pop(rid, None)
|
||||
return gone
|
||||
|
||||
def release(self, connector: str, key: str, host: str, repo: str,
|
||||
operation: str,
|
||||
ttl_seconds: int = DEFAULT_RELEASE_TTL_SECONDS,
|
||||
max_uses: int = DEFAULT_RELEASE_MAX_USES
|
||||
) -> SurrogateRelease:
|
||||
"""按次发放:绑定须与入库作用域一致(host/repo 有记录时
|
||||
须一致,无记录时本次绑定即定作用域),否则拒绝。发放的
|
||||
释放单次有效、ttl 后过期;发放本身记入库内待兑现表,
|
||||
须经 consume() 兑现一次才交出值。"""
|
||||
entry = self._store.get((connector, key))
|
||||
if entry is None:
|
||||
raise AuthdRefusal(
|
||||
"credential_not_stored",
|
||||
"连接器 %s 的凭证 %s 未入库" % (connector, key))
|
||||
if entry["host"] is not None and entry["host"] != host:
|
||||
raise AuthdRefusal(
|
||||
"scope_mismatch",
|
||||
"凭证作用域主机不一致,拒绝释放")
|
||||
if entry["repo"] is not None and entry["repo"] != repo:
|
||||
raise AuthdRefusal(
|
||||
"scope_mismatch",
|
||||
"凭证作用域仓库不一致,拒绝释放")
|
||||
if operation not in ("clone", "pull", "push"):
|
||||
raise AuthdRefusal(
|
||||
"operation_not_allowed",
|
||||
"操作 %s 不在允许集内" % operation)
|
||||
release = SurrogateRelease(
|
||||
release_id=secrets.token_hex(8),
|
||||
connector=connector, key=key, host=host, repo=repo,
|
||||
operation=operation, value=entry["value"],
|
||||
expires_at=self._clock() + ttl_seconds,
|
||||
max_uses=max_uses)
|
||||
self._releases[release.release_id] = release
|
||||
return release
|
||||
|
||||
def consume(self, release_id: str) -> str:
|
||||
"""兑现一次释放:过期/用完/未知 id 一律拒绝(AuthdRefusal,
|
||||
不含值)。兑现成功返回凭证值,并把该释放从库里删除
|
||||
(单次有效,用过即废)。"""
|
||||
release = self._releases.get(release_id)
|
||||
if release is None:
|
||||
raise AuthdRefusal(
|
||||
"release_unknown", "释放记录不存在或已兑现")
|
||||
if self._clock() >= release.expires_at:
|
||||
self._releases.pop(release_id, None)
|
||||
raise AuthdRefusal("release_expired", "释放已过期")
|
||||
if release.uses_left <= 0:
|
||||
self._releases.pop(release_id, None)
|
||||
raise AuthdRefusal("release_exhausted", "释放已用完")
|
||||
release.uses_left -= 1
|
||||
value = release.value
|
||||
self._releases.pop(release_id, None)
|
||||
return value
|
||||
|
||||
def audit_release(self, release: SurrogateRelease) -> dict:
|
||||
"""发放审计记录(只含绑定,不含值)。"""
|
||||
record = release.describe()
|
||||
record["event"] = "authd_credential_released"
|
||||
return record
|
||||
|
||||
def audit_refusal(self, kind: str, connector: str,
|
||||
key: str | None = None) -> dict:
|
||||
"""拒绝审计记录(只含 kind 与名,不含值)。"""
|
||||
return {"event": "authd_credential_refused", "kind": kind,
|
||||
"connector": connector, "key": key}
|
||||
|
||||
|
||||
def _default_post(url: str, headers: dict, payload: dict):
|
||||
"""默认 HTTP 客户端(urllib):返回 (status, body_dict)。
|
||||
只用于 mint_gitea_token,测试注入假 post,不联网。"""
|
||||
data = json.dumps(payload).encode("utf-8")
|
||||
req = urllib.request.Request(url, data=data, headers=headers,
|
||||
method="POST")
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=15) as resp:
|
||||
return resp.status, json.loads(resp.read() or b"{}")
|
||||
except urllib.error.HTTPError as exc:
|
||||
try:
|
||||
body = json.loads(exc.read() or b"{}")
|
||||
except ValueError:
|
||||
body = {}
|
||||
return exc.code, body
|
||||
|
||||
|
||||
def mint_gitea_token(api_url: str, user: str, master_token: str,
|
||||
name: str, scopes: Iterable[str] = ("write:repository",),
|
||||
post: Callable[[str, dict, dict], tuple[int, dict]]
|
||||
| None = None) -> str:
|
||||
"""经 Gitea API 铸短期专用 token(真替身路径,默认关闭)。
|
||||
|
||||
调 `POST {api_url}/users/{user}/tokens`(Gitea 官方 API),
|
||||
新 token 与 master token 是不同的值、可独立吊销(DELETE 同
|
||||
路径),到期由调用方吊销。返回新 token 值;失败即
|
||||
AuthdRefusal(不含任何 token 值)。
|
||||
前提(见差距表 §3.2):master token 须有建 token 的权限;
|
||||
现有 write:repository 测试 token 铸 token 会失败(403),
|
||||
开这条路须另配有铸 token 权限的 token 并做真机验证。
|
||||
post 可注入(测试用假 post),默认走 urllib。"""
|
||||
if not api_url or not user or not master_token or not name:
|
||||
raise AuthdRefusal(
|
||||
"mint_args_missing", "铸 token 参数不全")
|
||||
base = api_url.rstrip("/")
|
||||
url = "%s/users/%s/tokens" % (base, user)
|
||||
headers = {"Authorization": "token %s" % master_token,
|
||||
"Content-Type": "application/json"}
|
||||
payload = {"name": name, "scopes": list(scopes)}
|
||||
post_fn = post or _default_post
|
||||
status, body = post_fn(url, headers, payload)
|
||||
if status not in (200, 201):
|
||||
raise AuthdRefusal(
|
||||
"mint_failed",
|
||||
"Gitea 铸 token 失败:status=%s" % status)
|
||||
token = (body or {}).get("sha1") or (body or {}).get("token")
|
||||
if not token:
|
||||
raise AuthdRefusal(
|
||||
"mint_empty", "Gitea 铸 token 未返回 token 值")
|
||||
return token
|
||||
|
||||
|
||||
def revoke_gitea_token(api_url: str, user: str, master_token: str,
|
||||
token_id: object,
|
||||
delete: Callable[[str, dict], int] | None = None
|
||||
) -> bool:
|
||||
"""吊销一个铸出的 token(`DELETE {api}/users/{user}/tokens/{id}`)。
|
||||
成功返回 True;404 视为已不在(也返回 True,不报错);
|
||||
其它失败返回 False(调用方审计,不抛错——吊销失败不能炸任务,
|
||||
但须留审计)。delete 可注入,默认走 urllib。"""
|
||||
if not api_url or not user or not master_token \
|
||||
or token_id is None:
|
||||
return False
|
||||
base = api_url.rstrip("/")
|
||||
url = "%s/users/%s/tokens/%s" % (base, user, token_id)
|
||||
headers = {"Authorization": "token %s" % master_token}
|
||||
if delete is None:
|
||||
def _do_delete(url: str, headers: dict) -> int:
|
||||
req = urllib.request.Request(url, headers=headers,
|
||||
method="DELETE")
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=15) as resp:
|
||||
return resp.status
|
||||
except urllib.error.HTTPError as exc:
|
||||
return exc.code
|
||||
delete = _do_delete
|
||||
status = delete(url, headers)
|
||||
return status in (200, 204, 404)
|
||||
|
||||
|
||||
def surrogate_fingerprint(value: str) -> str:
|
||||
"""凭证值的指纹(审计/日志里唯一允许出现的凭证派生形式):
|
||||
SHA-256 前 8 位 hex。指纹不可逆推出原值,审计里只记指纹、
|
||||
不记值。"""
|
||||
return hashlib.sha256(value.encode("utf-8")).hexdigest()[:8]
|
||||
|
||||
|
||||
def hmac_sign(key: str, message: str) -> str:
|
||||
"""HMAC-SHA256 签名(释放记录的防篡改绑定,客内守护进程
|
||||
用它签发 release_id 的校验码;与 Gitea 无关)。"""
|
||||
return hmac.new(key.encode("utf-8"), message.encode("utf-8"),
|
||||
hashlib.sha256).hexdigest()
|
||||
+101
-5
@@ -153,6 +153,12 @@ def _network_fragment(params: dict, env: dict) -> dict:
|
||||
"egress_proxy": env.get("CHV_EGRESS_PROXY"),
|
||||
"allow_direct_egress": env.get("CHV_ALLOW_DIRECT_EGRESS")
|
||||
== "1",
|
||||
# feat-chv-muse-p3:Sentinel 白名单主机(逗号分隔),
|
||||
# 默认只放行配置的 Gitea 主机(执行器 wake 预审时追加)。
|
||||
"sentinel_allow_hosts": tuple(
|
||||
h.strip().lower()
|
||||
for h in (env.get("CHV_SENTINEL_ALLOW_HOSTS") or "")
|
||||
.split(",") if h.strip()),
|
||||
}
|
||||
|
||||
|
||||
@@ -245,6 +251,91 @@ def _timeout_fragment(params: dict, env: dict) -> dict:
|
||||
}
|
||||
|
||||
|
||||
def _gitea_fragment(env: dict) -> dict:
|
||||
"""Gitea 同步片段:宿主配置环境变量 CHV_GITEA_*(见
|
||||
gitea_sync.CHV_GITEA_ENV_MAP)映射成的 guest-env 键值
|
||||
dict。无配置即空 dict(本次启动不同步);值本身不校验
|
||||
完整性——完整性由 wake_vm 在起 VMM 前 fail-closed
|
||||
(validate_gitea_sync_env),构造时不拦,免得一个配
|
||||
错的宿主连执行器对象都建不出来、排障信息全丢。
|
||||
import 在函数内:本模块是叶子模块,顶层只许 import
|
||||
标准库(见 test_leaf_modules_only_import_stdlib),
|
||||
与执行器的方法内 import 同款惯例。"""
|
||||
from cloud.chv.gitea_sync import config_gitea_sync_env
|
||||
return {"gitea_sync_env": config_gitea_sync_env(env)}
|
||||
|
||||
|
||||
def _to_switch(raw: object, name: str) -> bool | None:
|
||||
"""CHV_* 开关解析(P1 气球/vsock 用):未设 → None(调用方
|
||||
再定默认值);"1" 开、"0" 关;其它值一律 ValueError,绝不
|
||||
静默当关(与 CHV_SSH_FORWARD 的 `== "1"` 旧口径不同,本轨
|
||||
新开关按差距表要求非法值即报错)。"""
|
||||
if raw is None:
|
||||
return None
|
||||
if raw == "1":
|
||||
return True
|
||||
if raw == "0":
|
||||
return False
|
||||
raise ValueError("%s 只能是 0 或 1:%r" % (name, raw))
|
||||
|
||||
|
||||
def _balloon_vsock_fragment(env: dict) -> dict:
|
||||
"""气球/vsock 片段(P1,feat-chv-muse-p1):virtio-balloon
|
||||
+ free-page-reporting 开关、vsock notify 通道开关与客 CID。
|
||||
|
||||
- CHV_BALLOON:默认 1(开)。只认 "1"/"0",其它值
|
||||
ValueError。
|
||||
- CHV_VSOCK:默认 1(开)。只认 "1"/"0",其它值
|
||||
ValueError。
|
||||
- CHV_VSOCK_CID:客机 vsock CID,可选,默认 3。必须是
|
||||
整数且在 3-4294967295 之间:0/1/2 是 vsock 保留 CID
|
||||
(Hypervisor/Loopback/Host,CHV 的 VmConfig 校验会
|
||||
以 VsockSpecialCid 拒绝),-1(随机 CID)CHV 的
|
||||
vm.create 同样 400 拒绝(上游 discussion #7531),
|
||||
所以这里不接受保留值/负值;显式设了非法 CID 即
|
||||
ValueError,哪怕 CHV_VSOCK=0(配错早炸,不留到起 VM
|
||||
才暴露)。CID 由运维指定、不是 CHV 分配的:同一宿主
|
||||
上各 VM 的 vsock socket 文件不同,CID 相同(默认 3)
|
||||
并不冲突。"""
|
||||
balloon = _to_switch(env.get("CHV_BALLOON"), "CHV_BALLOON")
|
||||
if balloon is None:
|
||||
balloon = True
|
||||
vsock = _to_switch(env.get("CHV_VSOCK"), "CHV_VSOCK")
|
||||
if vsock is None:
|
||||
vsock = True
|
||||
raw_cid = env.get("CHV_VSOCK_CID")
|
||||
if raw_cid is None:
|
||||
vsock_cid = 3
|
||||
else:
|
||||
try:
|
||||
vsock_cid = _to_int(raw_cid)
|
||||
except ValueError:
|
||||
raise ValueError("CHV_VSOCK_CID 必须是整数")
|
||||
if not 3 <= vsock_cid <= 0xFFFFFFFF:
|
||||
raise ValueError(
|
||||
"CHV_VSOCK_CID 必须在 3-4294967295 之间:%s"
|
||||
% vsock_cid)
|
||||
return {"balloon": balloon, "vsock": vsock,
|
||||
"vsock_cid": vsock_cid}
|
||||
|
||||
|
||||
def _seed_fragment(params: dict, env: dict) -> dict:
|
||||
"""seed 附加项片段:只存宿主文件路径,不读文件内容
|
||||
(读文件在 wake 时 fail-closed,见执行器 _seed_extras;
|
||||
构造时不读,免得路径配错时连执行器对象都建不出来)。
|
||||
CHV_SEED_USER_DATA_EXTRA_FILE 指向含 `users:` 块的文件,
|
||||
CHV_SEED_NETWORK_CONFIG_FILE 指向 NoCloud network-config
|
||||
文件;皆空即不附加(行为与旧 seed 一致)。"""
|
||||
return {
|
||||
"seed_user_data_extra_file":
|
||||
params.get("seed_user_data_extra_file")
|
||||
or env.get("CHV_SEED_USER_DATA_EXTRA_FILE"),
|
||||
"seed_network_config_file":
|
||||
params.get("seed_network_config_file")
|
||||
or env.get("CHV_SEED_NETWORK_CONFIG_FILE"),
|
||||
}
|
||||
|
||||
|
||||
def _assemble_chv_config(fragments: list) -> "ChvConfig":
|
||||
"""把各片段 dict 合并成 ChvConfig(纯合并,不改值)。
|
||||
|
||||
@@ -264,17 +355,22 @@ def build_chv_config(params: dict, env: dict) -> "ChvConfig":
|
||||
"""把显式构造参数与环境变量解析成 ChvConfig。
|
||||
|
||||
片段调用顺序即旧校验顺序(SSH → 配额 → 磁盘 →
|
||||
超时),只有一处可观测的顺序变化:warm_pool_size
|
||||
从旧超时片段移到配额片段,因此“CHV_WARM_POOL_SIZE
|
||||
非法 + 磁盘非法同时出现”时先报前者;单项非法时
|
||||
报错与旧代码逐字一致。字段集合与取值与旧代码一致,
|
||||
由测试的字段集断言钉住。"""
|
||||
气球/vsock → 超时),只有两处可观测的顺序变化:
|
||||
warm_pool_size 从旧超时片段移到配额片段,因此
|
||||
“CHV_WARM_POOL_SIZE 非法 + 磁盘非法同时出现”时先报
|
||||
前者;单项非法时报错与旧代码逐字一致。字段集合 =
|
||||
旧代码字段 + P1 气球/vsock 三字段(balloon/vsock/
|
||||
vsock_cid)+ P3 Sentinel 字段(sentinel_allow_hosts,
|
||||
在网络片段内),由测试的字段集断言钉住。"""
|
||||
env = env if env is not None else {}
|
||||
return _assemble_chv_config([
|
||||
_base_fragment(params, env),
|
||||
_ssh_fragment(params, env),
|
||||
_quota_fragment(env),
|
||||
_disk_fragment(env),
|
||||
_balloon_vsock_fragment(env),
|
||||
_network_fragment(params, env),
|
||||
_timeout_fragment(params, env),
|
||||
_gitea_fragment(env),
|
||||
_seed_fragment(params, env),
|
||||
])
|
||||
+3
-4
@@ -154,12 +154,11 @@ def _load_snapshots(state_dir, vm_name):
|
||||
|
||||
def _save_snapshots(state_dir, vm_name, entries):
|
||||
import json
|
||||
from cloud.chv.atomic_file import atomic_write_text
|
||||
path = _snapshots_path(state_dir, vm_name)
|
||||
os.makedirs(os.path.dirname(path), exist_ok=True)
|
||||
tmp_path = path + ".tmp"
|
||||
with open(tmp_path, "w") as fh:
|
||||
json.dump(entries, fh)
|
||||
os.replace(tmp_path, path)
|
||||
# feat-chv-closeup(P2-18):随机 tmp 名 + 双 fsync。
|
||||
atomic_write_text(path, json.dumps(entries))
|
||||
|
||||
|
||||
def prune_snapshots(state_dir: str, vm_name: str,
|
||||
|
||||
+808
-93
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,146 @@
|
||||
"""Gitea 同步变量供给(CHV guest-env 集成,叶子模块)。
|
||||
|
||||
背景:客内 chv-sync boot 在 cloud-init 时运行、SSH 之前,
|
||||
只能读 seed 投递的 /etc/chv/guest-env,看不到 SSH env 前缀。
|
||||
因此同步所需的 6 个变量必须随 seed 进 guest-env,不能只靠
|
||||
extra_env 的 SSH 前缀;其中 GITEA_TOKEN 是密钥,只进 seed,
|
||||
绝不进 argv/日志/镜像。
|
||||
|
||||
变量与来源(优先级低→高,后者覆盖前者):
|
||||
1. 宿主配置环境变量(主来源,生产口径):CHV_GITEA_URL、
|
||||
CHV_GITEA_USER、CHV_GITEA_TOKEN、CHV_SYNC_REPO、
|
||||
CHV_SYNC_BRANCH、CHV_GITEA_EMAIL。由部署该 worker/执行器
|
||||
的运维在宿主环境里设置,与 CHV_SSH_KEY、CHV_EGRESS_PROXY
|
||||
同款口径;CHV 是单租户单 VM 模型,一台宿主对应一个
|
||||
Gitea 身份。密钥来源绝不硬编码进代码。
|
||||
2. 任务级覆盖(内存态):任务 dict 的 `gitea_sync` 子 dict,
|
||||
键即 6 个变量名。仅在调用方以内存态构造任务 dict 时生效
|
||||
(直接调 wake_vm、测试、手工驱动);tasks 表目前没有
|
||||
JSON/元数据列,DB 往返会丢掉它——这是已知边界,不做
|
||||
静默声称,需要每任务不同仓库时再做任务表迁移。
|
||||
3. 显式 extra_env:调用方直接传给 wake_vm/dispatch 的
|
||||
extra_env 里的同名键,优先级最高。
|
||||
|
||||
缺失语义(fail-closed,不跳过同步):
|
||||
- 6 个键一个都没有 → 本次启动不同步(chv-sync boot 见无
|
||||
guest-env 静默成功,VM 照常用)。
|
||||
- 任一键出现、但必填 4 键(GITEA_URL、GITEA_USER、
|
||||
GITEA_TOKEN、SYNC_REPO)缺失 → 校验失败,wake 在起 VMM
|
||||
之前直接拒绝;半份同步配置绝不能让 VM 带着“我以为同步
|
||||
了”的状态启动。SYNC_BRANCH、GITEA_EMAIL 可选。
|
||||
- 校验错误只含缺失的键名,绝不含任何变量值。
|
||||
|
||||
本模块是纯函数,不读文件、不跑 VM、不含任何密钥值。
|
||||
"""
|
||||
|
||||
# 同步变量全集(guest-env 里应有的键)。
|
||||
GITEA_SYNC_KEYS = frozenset({
|
||||
"GITEA_URL", "GITEA_USER", "GITEA_TOKEN",
|
||||
"SYNC_REPO", "SYNC_BRANCH", "GITEA_EMAIL",
|
||||
})
|
||||
|
||||
# 其中密钥:只进 seed,绝不进 argv(见 ssh_transport 的
|
||||
# _CHV_SECRET_ENV_NAMES)。
|
||||
GITEA_SYNC_SECRET_KEYS = frozenset({"GITEA_TOKEN"})
|
||||
|
||||
# 其中 authd 专用密钥(feat-chv-muse-p3):只进客内 authd 的
|
||||
# 独立 seed 文件 /etc/chv/authd-seed(见 hostkeys.AUTHD_SEED_PATH),
|
||||
# 不进 guest-env、不进 codex 进程环境。本期只有 GITEA_TOKEN。
|
||||
AUTHD_SECRET_KEYS = frozenset({"GITEA_TOKEN"})
|
||||
|
||||
|
||||
def split_authd_env(secrets: dict | None) -> tuple[dict, dict]:
|
||||
"""把密钥集拆成(走 authd 通道的子集,仍走 guest-env 的子集)。
|
||||
|
||||
authd 子集 = AUTHD_SECRET_KEYS 命中的键(本期只有 GITEA_TOKEN);
|
||||
其余密钥(如 SUB2API_API_KEY)仍走 guest-env(见差距表 §3.3,
|
||||
网关 key 的 privsep 改造是后续工作)。"""
|
||||
authd, rest = {}, {}
|
||||
for key, value in dict(secrets or {}).items():
|
||||
if key in AUTHD_SECRET_KEYS:
|
||||
authd[key] = value
|
||||
else:
|
||||
rest[key] = value
|
||||
return authd, rest
|
||||
|
||||
# 其中非密钥、但 boot 时就要用、也必须进 guest-env 的键
|
||||
#(SSH env 前缀到不了 cloud-init 时运行的 chv-sync boot)。
|
||||
GITEA_SYNC_PLAIN_KEYS = GITEA_SYNC_KEYS - GITEA_SYNC_SECRET_KEYS
|
||||
|
||||
# 必填键:缺一即 fail-closed(见模块头缺失语义)。
|
||||
GITEA_SYNC_REQUIRED_KEYS = frozenset({
|
||||
"GITEA_URL", "GITEA_USER", "GITEA_TOKEN", "SYNC_REPO",
|
||||
})
|
||||
|
||||
# 宿主配置环境变量名 → guest-env 键名。
|
||||
CHV_GITEA_ENV_MAP = {
|
||||
"CHV_GITEA_URL": "GITEA_URL",
|
||||
"CHV_GITEA_USER": "GITEA_USER",
|
||||
"CHV_GITEA_TOKEN": "GITEA_TOKEN",
|
||||
"CHV_SYNC_REPO": "SYNC_REPO",
|
||||
"CHV_SYNC_BRANCH": "SYNC_BRANCH",
|
||||
"CHV_GITEA_EMAIL": "GITEA_EMAIL",
|
||||
}
|
||||
|
||||
|
||||
def _clean(mapping: dict | None) -> dict:
|
||||
"""只留白名单键:值须为非空字符串(strip 后),其它
|
||||
类型/空值/未知键一律丢掉——调用方拼错的键名绝不静默
|
||||
进 seed 变成客内无人读的脏变量。"""
|
||||
cleaned = {}
|
||||
for key, value in dict(mapping or {}).items():
|
||||
if key not in GITEA_SYNC_KEYS:
|
||||
continue
|
||||
if not isinstance(value, str):
|
||||
continue
|
||||
value = value.strip()
|
||||
if not value:
|
||||
continue
|
||||
cleaned[key] = value
|
||||
return cleaned
|
||||
|
||||
|
||||
def config_gitea_sync_env(env: dict | None) -> dict:
|
||||
"""宿主配置环境变量(CHV_GITEA_*)→ guest-env 键名。"""
|
||||
mapped = {}
|
||||
for env_name, key in CHV_GITEA_ENV_MAP.items():
|
||||
value = (env or {}).get(env_name)
|
||||
if value is not None:
|
||||
mapped[key] = value
|
||||
return _clean(mapped)
|
||||
|
||||
|
||||
def task_gitea_sync_env(task: dict | None) -> dict:
|
||||
"""任务 dict 的 `gitea_sync` 子 dict → 白名单后的键值。
|
||||
不是 dict(或缺失)即空——绝不把任务其它字段误读成
|
||||
同步变量。"""
|
||||
if not isinstance(task, dict):
|
||||
return {}
|
||||
return _clean(task.get("gitea_sync"))
|
||||
|
||||
|
||||
def extract_gitea_sync_env(task: dict | None = None,
|
||||
extra_env: dict | None = None,
|
||||
config_sync: dict | None = None) -> dict:
|
||||
"""三源合并(优先级:config_sync < 任务 gitea_sync <
|
||||
extra_env 同名键)。返回白名单后的 dict,可能为空
|
||||
(空 = 本次启动不同步)。"""
|
||||
merged = {}
|
||||
merged.update(_clean(config_sync))
|
||||
merged.update(task_gitea_sync_env(task))
|
||||
merged.update(_clean(extra_env))
|
||||
return merged
|
||||
|
||||
|
||||
def validate_gitea_sync_env(env: dict | None) -> None:
|
||||
"""校验同步变量完整性:空即不同步(通过);非空但必填
|
||||
键缺失即 ValueError(只含缺失键名,不含值)。调用方
|
||||
(执行器 wake_vm)在起 VMM 前调本函数做 fail-closed。"""
|
||||
env = dict(env or {})
|
||||
if not env:
|
||||
return
|
||||
missing = sorted(GITEA_SYNC_REQUIRED_KEYS - set(env))
|
||||
if missing:
|
||||
raise ValueError(
|
||||
"Gitea 同步变量不完整,缺失:%s(同步已请求但必填"
|
||||
"键不全,拒绝启动)" % ",".join(missing))
|
||||
Executable
+136
@@ -0,0 +1,136 @@
|
||||
#!/usr/bin/env python3
|
||||
"""客内 vsock notify 脚本(P1,feat-chv-muse-p1)。
|
||||
|
||||
启动后连宿主 CID 2 的 1024 端口发 ready,之后每 30 秒发一次
|
||||
heartbeat。消息是 JSON 行:
|
||||
|
||||
{"type": "ready"|"heartbeat", "vm": "<vm 名>", "ts": <unix 秒>}
|
||||
|
||||
宿主在 <vsock socket>_1024 这个 Unix socket 上监听接收
|
||||
(cloud/chv/vsock.py 的 wait_ready/listen_notify)。
|
||||
|
||||
失败一律静默退出(exit 0,不打日志、不留 traceback):vsock
|
||||
只是就绪辅助信号,SSH 仍是唯一命令通道与就绪判定——本脚本
|
||||
绝不能因为连不上、发失败而影响 sshd 或任务。调试时设
|
||||
VSOCK_NOTIFY_DEBUG=1 可在 stderr 看到异常。
|
||||
|
||||
================ P2 镜像集成接入说明(本轨只写脚本,不改构建) ================
|
||||
本文件不由 chv-guest/build.sh 安装,镜像集成是 P2 的范围。
|
||||
P2 接入时做这三件:
|
||||
|
||||
1. 把本文件拷进镜像,例如 /app/vsock_notify.py(须 python3,
|
||||
客机镜像已有 Python;若最小镜像无 python3,P2 须先补);
|
||||
2. 加 systemd 单元,例如 /etc/systemd/system/vsock-notify.service:
|
||||
|
||||
[Unit]
|
||||
Description=vsock notify to host (ready/heartbeat)
|
||||
After=multi-user.target
|
||||
[Service]
|
||||
Type=simple
|
||||
ExecStart=/usr/bin/python3 /app/vsock_notify.py %H
|
||||
Restart=no
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
|
||||
(%H 是客机 hostname;也可直接写 VM 名,或由 cloud-init
|
||||
写进环境变量 CHV_VM_NAME。Restart=no:本脚本失败即静默
|
||||
退出,不由 systemd 反复拉起打扰宿主监听。)
|
||||
3. 前置条件(缺一则本脚本静默退出,任务仍走 SSH,不报错):
|
||||
- 客机内核须含 CONFIG_VIRTIO_VSOCKETS;
|
||||
- 起 VM 时 CHV_VSOCK=1(默认开),VmConfig 带 vsock 设备
|
||||
(cloud/chv/executor.py 的 _balloon_vsock_payload);
|
||||
- 宿主侧须在 VM boot 前后监听 <socket>_1024(否则客机
|
||||
首次 connect 失败,本脚本在 30 秒内重试,仍失败即退出)。
|
||||
|
||||
端口 1024 与宿主侧 cloud/chv/vsock.py 的 VSOCK_NOTIFY_PORT
|
||||
须一致;改一处必须改另一处。
|
||||
===============================================================================
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import socket
|
||||
import sys
|
||||
import time
|
||||
|
||||
# 宿主 CID(vsock 众所周知值,CHV 官方 docs/vsock.md)。
|
||||
HOST_CID = 2
|
||||
# notify 端口(与宿主侧 VSOCK_NOTIFY_PORT 一致)。
|
||||
NOTIFY_PORT = 1024
|
||||
# 首次 connect 重试窗口(秒):宿主监听可能比客机启动晚到。
|
||||
CONNECT_RETRY_SECONDS = 30
|
||||
# 心跳间隔(秒)。
|
||||
HEARTBEAT_INTERVAL = 30
|
||||
|
||||
|
||||
def _vm_name(argv) -> str:
|
||||
"""VM 名:argv[1] > CHV_VM_NAME > hostname。"""
|
||||
if len(argv) > 1 and argv[1]:
|
||||
return argv[1]
|
||||
env_name = os.environ.get("CHV_VM_NAME") or os.environ.get(
|
||||
"VSOCK_VM_NAME")
|
||||
if env_name:
|
||||
return env_name
|
||||
try:
|
||||
return socket.gethostname()
|
||||
except OSError:
|
||||
return "unknown"
|
||||
|
||||
|
||||
def _send(sock, message_type: str, vm: str) -> None:
|
||||
line = json.dumps({"type": message_type, "vm": vm,
|
||||
"ts": time.time()},
|
||||
separators=(",", ":"))
|
||||
sock.sendall((line + "\n").encode("utf-8"))
|
||||
|
||||
|
||||
def _connect(deadline: float):
|
||||
"""在 deadline 前反复连 (HOST_CID, NOTIFY_PORT),成功返回
|
||||
socket,超时返回 None。"""
|
||||
while True:
|
||||
remaining = deadline - time.monotonic()
|
||||
if remaining <= 0:
|
||||
return None
|
||||
sock = socket.socket(socket.AF_VSOCK, socket.SOCK_STREAM)
|
||||
try:
|
||||
sock.settimeout(min(remaining, 5))
|
||||
sock.connect((HOST_CID, NOTIFY_PORT))
|
||||
sock.settimeout(None)
|
||||
return sock
|
||||
except OSError as exc:
|
||||
try:
|
||||
sock.close()
|
||||
except OSError:
|
||||
pass
|
||||
if os.environ.get("VSOCK_NOTIFY_DEBUG") == "1":
|
||||
print("vsock_notify connect 失败:%r" % (exc,),
|
||||
file=sys.stderr, flush=True)
|
||||
time.sleep(1)
|
||||
|
||||
|
||||
def main(argv) -> int:
|
||||
vm = _vm_name(argv)
|
||||
deadline = time.monotonic() + CONNECT_RETRY_SECONDS
|
||||
sock = _connect(deadline)
|
||||
if sock is None:
|
||||
return 0
|
||||
try:
|
||||
_send(sock, "ready", vm)
|
||||
while True:
|
||||
time.sleep(HEARTBEAT_INTERVAL)
|
||||
_send(sock, "heartbeat", vm)
|
||||
except (OSError, ValueError):
|
||||
# 任何发送失败都静默退出,不影响 sshd/任务。
|
||||
return 0
|
||||
except KeyboardInterrupt:
|
||||
return 0
|
||||
finally:
|
||||
try:
|
||||
sock.close()
|
||||
except OSError:
|
||||
pass
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv))
|
||||
+212
-45
@@ -34,12 +34,25 @@ import os
|
||||
import shlex
|
||||
from collections.abc import Callable
|
||||
|
||||
from cloud.chv.atomic_file import atomic_write_text
|
||||
|
||||
# seed 投递的客内密钥环境文件路径(feat-chv-secfix,H2):
|
||||
# 只在 provisioned 模式的 seed user-data 里写出,0600 root,
|
||||
# 客内 host.py 命令执行前由远程命令 source 进环境;宿主侧
|
||||
# 除 seed 目录/user-data(随 VM 销毁而删)外不另存该文件。
|
||||
GUEST_ENV_PATH = "/etc/chv/guest-env"
|
||||
|
||||
# seed 投递的客内 authd 专用密钥文件路径(feat-chv-muse-p3):
|
||||
# 只写 AUTHD_SECRET_KEYS 命中的密钥(本期只有 GITEA_TOKEN),
|
||||
# 0600 root,只给客内 authd 守护进程读;codex 进程环境里永不
|
||||
# 出现这些键(见执行器 _authd_delivered)。与 guest-env 分离
|
||||
# 的理由:guest-env 会被远程命令 source 进任务进程环境,任何
|
||||
# 进 guest-env 的密钥都会进 codex 环境;authd 密钥绝不能走
|
||||
# 那条路。文件本身仍是 seed 写出的落盘文件(cloud-init 的唯一
|
||||
# 投递通道就是写文件),真正让它对 agent 不可见的是 P2 的
|
||||
# nspawn 单元隔离,见差距表 §3.1。
|
||||
AUTHD_SEED_PATH = "/etc/chv/authd-seed"
|
||||
|
||||
|
||||
def host_key_paths(state_dir: str, vm_name: str) -> dict:
|
||||
"""该 VM 的 host key 文件路径(私钥/公钥/known_hosts)。"""
|
||||
@@ -74,13 +87,23 @@ def delete_host_private_key(state_dir: str, vm_name: str) -> str:
|
||||
def known_hosts_line(port: int, public_key_text: str, host: str = "127.0.0.1") -> str:
|
||||
"""公钥文件内容转 known_hosts 行:只取前两列(类型+数据)。
|
||||
|
||||
公钥文件形如 'ssh-ed25519 AAAA... [comment]',known_hosts
|
||||
行形如 '[<host>]:<port> ssh-ed25519 AAAA...'(host 缺省
|
||||
回环,仅作兼容;客机 IP 方案下 host 应为解析出的客机 IP)。
|
||||
非法内容 ValueError,绝不把陌生行原样写进文件。"""
|
||||
公钥文件形如 'ssh-ed25519 AAAA... [comment]'。22 端口
|
||||
(OpenSSH 默认端口)时行形如 '<host> ssh-ed25519
|
||||
AAAA...'(裸 host 名);非 22 端口时才用
|
||||
'[<host>]:<port> ssh-ed25519 AAAA...'。OpenSSH 查
|
||||
known_hosts 时,默认端口只查裸 host 名、非默认端口
|
||||
只查括号形式,写错即永远匹配不上(2026-11-07 WSL
|
||||
真机实测:provisioned 模式钉死 key 写成
|
||||
'[192.168.10.10]:22',22 端口 ssh 报 Host key
|
||||
verification failed;keyscan TOFU 输出裸行所以
|
||||
lite9 没暴露)。host 缺省回环,仅作兼容;客机 IP
|
||||
方案下 host 应为解析出的客机 IP。非法内容
|
||||
ValueError,绝不把陌生行原样写进文件。"""
|
||||
parts = (public_key_text or "").strip().split()
|
||||
if len(parts) < 2 or not parts[0].startswith("ssh-"):
|
||||
raise ValueError("host 公钥格式非法")
|
||||
if int(port) == 22:
|
||||
return "%s %s %s" % (host, parts[0], parts[1])
|
||||
return "[%s]:%s %s %s" % (host, port, parts[0], parts[1])
|
||||
|
||||
|
||||
@@ -128,7 +151,22 @@ def build_seed_user_data(private_key_text: str, public_key_text: str) -> str:
|
||||
|
||||
注意:user-data 本身含私钥,只写进 per-VM seed 目录
|
||||
(0600),随 VM 销毁而删,绝不进镜像、绝不进日志、
|
||||
绝不进任务 prompt 落库。"""
|
||||
绝不进任务 prompt 落库。
|
||||
|
||||
根因禁令(sync-test 残留,2026-11-06 实测):seed 的
|
||||
user-data 里绝不能带 `runcmd:`。cloud-init 不跨配置
|
||||
类型合并——user-data 的 `runcmd` 会整体替换掉镜像
|
||||
`/etc/cloud/cloud.cfg.d/` 里的 `runcmd`(`merge_how`
|
||||
只合并多个 user-data 段,救不了 conf.d,见 cloud-init
|
||||
合并文档);旧版 seed 自带的 chmod + restart ssh 两行
|
||||
就这样顶掉了镜像 `99-chv-sync.cfg` 的 `chv-sync boot`
|
||||
钩子,首轮启动 clone 静默不跑。去掉的两行不需要
|
||||
补在 seed 里:chmod 600 已由 write_files 的
|
||||
permissions '0600' 保证;sshd 重启已搬到镜像自己的
|
||||
runcmd(`99-chv-sync.cfg`,cloud-final 阶段,sshd
|
||||
首次启动之后),那条是存活的列表,不会被 seed 覆盖。
|
||||
任何人再给 seed 加 runcmd 即重犯此 bug,回归测试
|
||||
`test_seed_user_data_has_no_runcmd` 会拦住。"""
|
||||
if not private_key_text or not public_key_text:
|
||||
raise ValueError("host key 对缺失,无法生成 seed")
|
||||
lines = [
|
||||
@@ -150,60 +188,189 @@ def build_seed_user_data(private_key_text: str, public_key_text: str) -> str:
|
||||
]
|
||||
for line in public_key_text.rstrip("\n").split("\n"):
|
||||
lines.append(" " + line)
|
||||
lines += [
|
||||
"runcmd:",
|
||||
" - chmod 600 /etc/ssh/ssh_host_ed25519_key",
|
||||
" - systemctl restart ssh || service ssh restart || true",
|
||||
]
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def write_seed(seed_dir: str, vm_name: str, private_key_text: str,
|
||||
public_key_text: str,
|
||||
secret_env: dict | None = None) -> str:
|
||||
"""写 seed 目录:meta-data + user-data(均为 0600)。
|
||||
def _authd_env_entry(authd_env: dict) -> list:
|
||||
"""authd-seed 的 write_files 条目行(0600 root,值经
|
||||
shlex.quote 引用):装 AUTHD_SECRET_KEYS 命中的密钥
|
||||
(本期只有 GITEA_TOKEN),外加与 guest-env 同源抄进来的
|
||||
三个非密钥同步变量(GITEA_URL/GITEA_USER/SYNC_REPO——
|
||||
round-11 真机修复层 2:authd 的 build_server 要求 seed
|
||||
里同时有这三键,只 GITEA_TOKEN 会 ValueError;探针
|
||||
fixprobe3 即从 guest-env 补这三键)。三键只在调用方
|
||||
guest-env 里有才抄,没有不补、不编默认值;与 guest-env
|
||||
条目并列写进同一 write_files 列表(绝不另写一个
|
||||
write_files: 键,见 _guest_env_entry 的 YAML 重复键禁令)。
|
||||
"""
|
||||
lines = [
|
||||
" - path: %s" % AUTHD_SEED_PATH,
|
||||
" permissions: '0600'",
|
||||
" owner: root:root",
|
||||
" content: |",
|
||||
]
|
||||
for key in sorted(authd_env):
|
||||
lines.append(" %s=%s"
|
||||
% (key, shlex.quote(str(authd_env[key]))))
|
||||
return lines
|
||||
|
||||
返回 seed 目录路径。seed 目录本身不含已知主机之外的
|
||||
共享数据,调用方(执行器关闭/回收)负责删除它。
|
||||
feat-chv-secfix(H2):secret_env 给定时,user-data 再
|
||||
写一个 write_files 条目,把密钥环境变量写进客内
|
||||
GUEST_ENV_PATH(0600 root),值经 shlex.quote 引用后
|
||||
才进文件内容——远程命令 source 该文件即可得到密钥
|
||||
环境,密钥值从不出现在宿主 ssh argv、宿主进程环境与
|
||||
审计明文里。secret_env 为空时不写该条目(客内文件不
|
||||
存在时远程命令也不 source 它,见执行器)。"""
|
||||
os.makedirs(seed_dir, exist_ok=True)
|
||||
meta_path = os.path.join(seed_dir, "meta-data")
|
||||
user_path = os.path.join(seed_dir, "user-data")
|
||||
with open(meta_path, "w") as fh:
|
||||
fh.write("instance-id: %s\nlocal-hostname: %s\n"
|
||||
% (vm_name, vm_name))
|
||||
os.chmod(meta_path, 0o600)
|
||||
user_data = build_seed_user_data(private_key_text, public_key_text)
|
||||
if secret_env:
|
||||
# 注意:只能把条目并进已有的 write_files 列表,绝不能
|
||||
# 再写一个 write_files: 键(YAML 重复键会覆盖掉 host
|
||||
# key 的两条写文件条目,预置即失效)。条目插在 runcmd
|
||||
# 段之前、仍属同一列表。
|
||||
|
||||
def _guest_env_entry(merged_env: dict) -> list:
|
||||
"""guest-env 的 write_files 条目行(0600 root,值经
|
||||
shlex.quote 引用):只能并进已有的 write_files 列表,
|
||||
绝不能再写一个 write_files: 键(YAML 重复键会覆盖掉
|
||||
host key 的两条写文件条目,预置即失效)。
|
||||
owner 恒为 root:root(V-9 真机根因:cloud-init 的 config
|
||||
阶段里 write_files 在 users-groups 之前跑,owner 写
|
||||
chv:chv 时 getpwnam('chv') 尚不存在,整个 write_files
|
||||
模块失败、guest-env 文件根本不产生)。文件改归 SSH 用户
|
||||
由客内 final 阶段 99-chv-sync.cfg runcmd 在 chv-sync
|
||||
boot 之前 chown(用户那时已建好),见 chv-guest/build.sh。"""
|
||||
lines = [
|
||||
" - path: %s" % GUEST_ENV_PATH,
|
||||
" permissions: '0600'",
|
||||
" owner: root:root",
|
||||
" content: |",
|
||||
]
|
||||
for key in sorted(secret_env):
|
||||
for key in sorted(merged_env):
|
||||
lines.append(" %s=%s"
|
||||
% (key, shlex.quote(str(secret_env[key]))))
|
||||
# 条目插在 runcmd 段之前、仍属同一 write_files 列表。
|
||||
% (key, shlex.quote(str(merged_env[key]))))
|
||||
return lines
|
||||
|
||||
|
||||
def _apply_seed_extras(seed_dir: str, user_data: str,
|
||||
user_data_extra: str | None,
|
||||
network_config: str | None) -> str:
|
||||
"""seed 可选附加项(执行器 wake 路径的客机身份/网络供给):
|
||||
user_data_extra 原样追加进 user-data(调用方用它供
|
||||
`users:` 块——chv 用户 + authorized_keys,镜像本身不
|
||||
建用户);network_config 写成 seed 目录的 network-config
|
||||
文件(NoCloud 静态 IP,tap 无 DHCP 时客机才有地址)。
|
||||
两者皆空即原样返回。user_data_extra 里含 runcmd: 行即
|
||||
ValueError——seed 禁 runcmd 是 2026-11-06 根因禁令,
|
||||
附加项也不得绕过(镜像的 99-chv-sync.cfg 靠它活着)。"""
|
||||
if user_data_extra:
|
||||
for line in user_data_extra.splitlines():
|
||||
if line.strip().startswith("runcmd:"):
|
||||
raise ValueError(
|
||||
"seed 附加 user-data 里不得带 runcmd:")
|
||||
if not user_data.endswith("\n"):
|
||||
user_data += "\n"
|
||||
user_data += user_data_extra
|
||||
if not user_data.endswith("\n"):
|
||||
user_data += "\n"
|
||||
if network_config:
|
||||
net_path = os.path.join(seed_dir, "network-config")
|
||||
# feat-chv-cover100:走 atomic_file(与 user-data 同款
|
||||
# 理由,见 write_seed)——直接写 + 事后 chmod 会在
|
||||
# chmod 前以 umask 权限短暂可读。
|
||||
atomic_write_text(net_path, network_config)
|
||||
return user_data
|
||||
|
||||
|
||||
def _validate_authd_env(authd_env: dict | None) -> None:
|
||||
"""authd_env 的键必须全在 AUTHD_SECRET_KEYS 内(本期只有
|
||||
GITEA_TOKEN),否则 ValueError——authd 通道只装 authd
|
||||
密钥,别的密钥误进这里会绕过 guest-env 的 source 语义。"""
|
||||
if not authd_env:
|
||||
return
|
||||
from cloud.chv.gitea_sync import AUTHD_SECRET_KEYS
|
||||
unknown = sorted(set(authd_env) - AUTHD_SECRET_KEYS)
|
||||
if unknown:
|
||||
raise ValueError(
|
||||
"authd 通道只接受 %s,拒绝:%s"
|
||||
% (",".join(sorted(AUTHD_SECRET_KEYS)),
|
||||
",".join(unknown)))
|
||||
|
||||
|
||||
def _write_seed_meta(seed_dir: str, vm_name: str) -> str:
|
||||
"""建 seed 目录并写 meta-data(0600),返回 user-data 路径。
|
||||
feat-chv-cover100:seed 暂存文件走 atomic_file(mkstemp
|
||||
0600 → 写 → fsync → rename):user-data 含 host 私钥
|
||||
明文,直接写 + 事后 chmod 会在 chmod 前以 umask 权限
|
||||
短暂可读;半截文件到不了 ISO(见 H5),原子写同时堵
|
||||
住权限窗口与半截残留。"""
|
||||
os.makedirs(seed_dir, exist_ok=True)
|
||||
meta_path = os.path.join(seed_dir, "meta-data")
|
||||
user_path = os.path.join(seed_dir, "user-data")
|
||||
atomic_write_text(
|
||||
meta_path,
|
||||
"instance-id: %s\nlocal-hostname: %s\n" % (vm_name, vm_name))
|
||||
return user_path
|
||||
|
||||
|
||||
def _seed_env_entries(merged_env: dict,
|
||||
authd_env: dict | None) -> list:
|
||||
"""guest-env 与 authd-seed 两路的 write_files 条目行
|
||||
(皆空即空列表,不写条目)。authd-seed 除了密钥还抄
|
||||
guest-env 里的 GITEA_URL/GITEA_USER/SYNC_REPO(层 2,
|
||||
见 _authd_env_entry)——build_server 要求四键齐全。"""
|
||||
extra_lines = []
|
||||
if merged_env:
|
||||
extra_lines += _guest_env_entry(merged_env)
|
||||
if authd_env:
|
||||
seed_env = dict(authd_env)
|
||||
for key in ("GITEA_URL", "GITEA_USER", "SYNC_REPO"):
|
||||
if key in merged_env:
|
||||
seed_env[key] = merged_env[key]
|
||||
extra_lines += _authd_env_entry(seed_env)
|
||||
return extra_lines
|
||||
|
||||
|
||||
def _insert_seed_entries(user_data: str, lines: list) -> str:
|
||||
"""条目插在 runcmd 段之前、仍属同一 write_files 列表
|
||||
(seed 自 2026-11-07 起不再带 runcmd,见
|
||||
build_seed_user_data 的根因禁令;marker 分支只作
|
||||
防御,将来模板若恢复 runcmd 仍插在其前)。"""
|
||||
marker = "runcmd:\n"
|
||||
if marker in user_data:
|
||||
head, tail = user_data.split(marker, 1)
|
||||
user_data = head + "\n".join(lines) + "\n" + marker + tail
|
||||
else:
|
||||
user_data = user_data + "\n".join(lines) + "\n"
|
||||
with open(user_path, "w") as fh:
|
||||
fh.write(user_data)
|
||||
os.chmod(user_path, 0o600)
|
||||
return head + "\n".join(lines) + "\n" + marker + tail
|
||||
return user_data + "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def write_seed(seed_dir: str, vm_name: str, private_key_text: str,
|
||||
public_key_text: str,
|
||||
secret_env: dict | None = None,
|
||||
guest_env: dict | None = None,
|
||||
user_data_extra: str | None = None,
|
||||
network_config: str | None = None,
|
||||
authd_env: dict | None = None) -> str:
|
||||
"""写 seed 目录:meta-data + user-data(均为 0600),
|
||||
返回 seed 目录路径。调用方负责随 VM 销毁删掉它。
|
||||
|
||||
secret_env 给定时写进客内 GUEST_ENV_PATH(0600 root,
|
||||
远程命令 source 它得密钥环境,值不进宿主 argv/环境/
|
||||
审计明文)。guest_env 给定时与 secret_env 合并写同一
|
||||
个条目(secret 优先于同名键,防两行同名);guest_env
|
||||
装 boot 时要用的非密钥同步变量(GITEA_URL 等,见
|
||||
gitea_sync):只走 SSH 前缀到不了 cloud-init 时运行
|
||||
的 chv-sync boot。两者皆空时不写该条目。
|
||||
authd_env(feat-chv-muse-p3)给定时写进另一个独立条目
|
||||
AUTHD_SEED_PATH(0600 root,只给客内 authd 守护进程读):
|
||||
键必须全在 gitea_sync.AUTHD_SECRET_KEYS 内(本期只有
|
||||
GITEA_TOKEN),否则 ValueError——authd 通道只装 authd
|
||||
密钥,别的密钥误进这里会绕过 guest-env 的 source 语义。
|
||||
user_data_extra / network_config 见 _apply_seed_extras
|
||||
(客机用户与静态网络的供给,皆空即不写)。"""
|
||||
_validate_authd_env(authd_env)
|
||||
user_path = _write_seed_meta(seed_dir, vm_name)
|
||||
user_data = build_seed_user_data(private_key_text, public_key_text)
|
||||
merged_env = dict(guest_env or {})
|
||||
merged_env.update(secret_env or {})
|
||||
# feat-chv-muse-p3:GITEA_TOKEN 绝不进 guest-env——它只能走
|
||||
# authd_env(独立的 authd-seed 条目)。调用方(执行器)须先
|
||||
# 用 gitea_sync.split_authd_env() 拆出来;这里再拦一次,
|
||||
# 免得别的调用点把旧写法带进来、悄悄恢复泄露面。
|
||||
if "GITEA_TOKEN" in merged_env:
|
||||
raise ValueError(
|
||||
"GITEA_TOKEN 须走 authd_env,不进 guest-env")
|
||||
extra_lines = _seed_env_entries(merged_env, authd_env)
|
||||
if extra_lines:
|
||||
user_data = _insert_seed_entries(user_data, extra_lines)
|
||||
user_data = _apply_seed_extras(seed_dir, user_data,
|
||||
user_data_extra, network_config)
|
||||
atomic_write_text(user_path, user_data)
|
||||
return seed_dir
|
||||
|
||||
|
||||
|
||||
@@ -74,15 +74,15 @@ def _read_registry(path: str) -> tuple[dict | None, bool]:
|
||||
|
||||
|
||||
def _write_registry(path: str, data: dict) -> None:
|
||||
"""原子落盘 registry(tmp + rename);写失败只记日志。"""
|
||||
"""原子落盘 registry(随机 tmp 名 + 双 fsync,P2-18);
|
||||
写失败只记日志。"""
|
||||
try:
|
||||
from cloud.chv.atomic_file import atomic_write_text
|
||||
import json
|
||||
directory = os.path.dirname(path)
|
||||
if directory:
|
||||
os.makedirs(directory, exist_ok=True)
|
||||
tmp_path = path + ".tmp"
|
||||
with open(tmp_path, "w") as fh:
|
||||
json.dump(data, fh)
|
||||
os.replace(tmp_path, path)
|
||||
atomic_write_text(path, json.dumps(data))
|
||||
except OSError:
|
||||
_LOGGER.warning("chv_pool_registry_save_failed path=%s", path)
|
||||
|
||||
|
||||
@@ -0,0 +1,515 @@
|
||||
"""每用户常驻 VM 池(生产模块,P1 对齐 Muse,见 ADR-17)。
|
||||
|
||||
每用户一台常驻 VM:任务到时 resume 已暂停的 VM,跑完
|
||||
再 pause,不销毁;池空或任何一步失败都回落冷启动,
|
||||
绝不因池坏而丢任务。常驻 VM 的盘是每用户持久工作区盘
|
||||
(state_dir/chv-resident-<用户>/disk.raw),不是每任务
|
||||
新盘——隔离单元在常驻路径上从“每任务”变成“每用户”
|
||||
(单租户可接受,多租户重审前不得把常驻当租户隔离)。
|
||||
|
||||
状态机:cold → provisioning → paused → running → paused。
|
||||
acquire(user):有 paused 条目时 resume_vm + 等 SSH 就绪
|
||||
(复用执行器 wait_guest_ready,不另写判定),返回
|
||||
mode=resident;无可用/失败时返回 mode=cold。release(user):
|
||||
pause_vm,不销毁;pause 失败时条目留在 running(不删
|
||||
不销毁,worker 关闭时 reconcile 清理)。provision(user):
|
||||
wake_vm + pause_vm 建 paused 条目(惰性建池,失败回落)。
|
||||
reclaim_idle():paused 超 CHV_RESIDENT_IDLE_MINUTES(默认
|
||||
0=不自动关机)时 hibernate(pause + snapshot + shutdown)
|
||||
并删条目,快照文件留作恢复物,restore 是后续工作。
|
||||
|
||||
Registry(state_dir/resident-pool.json):文件 IO 与损坏
|
||||
语义复用 cloud.chv.orphans(原子写、损坏 quarantine、
|
||||
单条目非法只丢该条);重启后读到的条目一律 orphan,不
|
||||
复用,只能 reconcile 清理。条目按用户键隔离:release
|
||||
绝不把用过的 VM 放回给另一用户;同一用户同一时间只跑
|
||||
一个常驻任务,并发第二个任务回落冷启动。
|
||||
|
||||
诚实边界:resume 失败时 wait_guest_ready 的失败路径会
|
||||
stop_vm(含删工作区盘),条目随之丢弃——该路径的工作区
|
||||
以 Gitea 同步为准;带密钥 extra_env 的任务不走常驻
|
||||
(seed 是建池时定的,密钥只能冷启动随 seed 投递)。
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import threading
|
||||
import time
|
||||
|
||||
from cloud.chv.orphans import _read_registry, _write_registry
|
||||
|
||||
_LOGGER = logging.getLogger("cloud.cloud_hypervisor")
|
||||
|
||||
_REGISTRY_FILENAME = "resident-pool.json"
|
||||
_RESIDENT_STATES = ("paused", "running", "provisioning")
|
||||
_DEFAULT_USER = "default"
|
||||
|
||||
|
||||
def resident_user_key(user_id: str | None) -> str:
|
||||
"""用户键:非空即原样消毒(非法字符变 -),空/缺省
|
||||
即 "default"(当前单租户:任务 dict 无 user_id 时全
|
||||
部落到同一常驻 VM;多租户重审前不得当租户边界用)。"""
|
||||
raw = _DEFAULT_USER if user_id is None else str(user_id)
|
||||
key = re.sub(r"[^A-Za-z0-9_.-]", "-", raw).strip(".-")
|
||||
return key or _DEFAULT_USER
|
||||
|
||||
|
||||
def resident_task_for_user(user_id: str | None) -> dict:
|
||||
"""常驻伪任务:执行器按它算 VM 名/known_hosts/socket。
|
||||
vm_name({"id": "resident-alice"}) = "chv-resident-alice"。"""
|
||||
return {"id": "resident-" + resident_user_key(user_id)}
|
||||
|
||||
|
||||
def resident_registry_path_for_executor(executor: object) -> str | None:
|
||||
"""常驻 registry 路径:state_dir/resident-pool.json。
|
||||
无 state_dir 时返回 None(调用方按“无残留”处理)。"""
|
||||
state_dir = getattr(executor, "state_dir", None)
|
||||
if not state_dir:
|
||||
return None
|
||||
return os.path.join(state_dir, _REGISTRY_FILENAME)
|
||||
|
||||
|
||||
def reconcile_resident_orphans_for_executor(executor: object) -> list[str]:
|
||||
"""生产路径的常驻残留清理入口:返回已清理的 VM 名。
|
||||
只做清理:读 resident-pool.json → 每个上一世条目尽力
|
||||
shutdown_vm → 落盘为空 → best-effort 审计。绝不构造
|
||||
ResidentPool、绝不把任何条目发给任务。无 registry 路
|
||||
径时返回 []。文件 IO 与损坏语义复用 orphans。"""
|
||||
path = resident_registry_path_for_executor(executor)
|
||||
if not path:
|
||||
return []
|
||||
data, _corrupt = _read_registry(path)
|
||||
if data is None:
|
||||
return []
|
||||
cleaned: list[str] = []
|
||||
for entry in data.values():
|
||||
if not isinstance(entry, dict):
|
||||
continue
|
||||
if not entry.get("vm"):
|
||||
continue
|
||||
if entry.get("state") not in _RESIDENT_STATES:
|
||||
continue
|
||||
vm = str(entry["vm"])
|
||||
try:
|
||||
executor.shutdown_vm(vm) # type: ignore[attr-defined]
|
||||
except Exception:
|
||||
pass
|
||||
cleaned.append(vm)
|
||||
if cleaned:
|
||||
_write_registry(path, {})
|
||||
try:
|
||||
executor._emit( # type: ignore[attr-defined]
|
||||
"resident_pool_reconcile", {"id": None},
|
||||
"cleaned=%s" % ",".join(cleaned))
|
||||
except Exception:
|
||||
pass
|
||||
return cleaned
|
||||
|
||||
|
||||
class ResidentPool:
|
||||
"""每用户常驻 VM 池(生产路径,见模块 docstring)。
|
||||
|
||||
同一 state_dir 只活一个池实例:registry 只在构造时
|
||||
读一次,重启后条目一律 orphan(见 _load_registry)。
|
||||
公共方法全部加锁:并发 acquire/release/provision 只
|
||||
放行一路,其余回 cold/False,不把同一 VM 发给两个任务。
|
||||
"""
|
||||
|
||||
def __init__(self, executor: object, idle_minutes: int | None = None,
|
||||
registry_path: str | None = None) -> None:
|
||||
"""建池并读 registry:读到的条目一律 orphan(见模块
|
||||
docstring)。idle_minutes 未给时读执行器属性或
|
||||
CHV_RESIDENT_IDLE_MINUTES,非法/负数即 0(不自动关机)。"""
|
||||
self.executor = executor
|
||||
self._lock = threading.Lock()
|
||||
self._busy: set[str] = set()
|
||||
self._entries: dict = {}
|
||||
self.registry_path = registry_path
|
||||
if self.registry_path is None:
|
||||
self.registry_path = \
|
||||
resident_registry_path_for_executor(executor)
|
||||
if idle_minutes is None:
|
||||
idle_minutes = self._idle_minutes_from_env()
|
||||
self.idle_minutes = idle_minutes if idle_minutes > 0 else 0
|
||||
self.registry_corrupt = False
|
||||
self.registry_corrupt_entries = 0
|
||||
self._load_registry()
|
||||
|
||||
# -- registry ------------------------------------------------------
|
||||
def _load_registry(self) -> None:
|
||||
"""载入 registry:读到的条目一律标 orphan(绝不直接
|
||||
当 paused 发给任务,旧 VMM 进程已随旧 worker 消失)。
|
||||
无文件时为空池;损坏时 quarantine 并从空开始;单条
|
||||
目非法只丢该条并计数,不连坐其它条目。"""
|
||||
if not self.registry_path:
|
||||
return
|
||||
data, corrupt = _read_registry(self.registry_path)
|
||||
if corrupt:
|
||||
self.registry_corrupt = True
|
||||
return
|
||||
if data is None:
|
||||
return
|
||||
for key, entry in data.items():
|
||||
if not isinstance(entry, dict) \
|
||||
or not entry.get("vm") \
|
||||
or entry.get("state") not in _RESIDENT_STATES:
|
||||
self.registry_corrupt_entries += 1
|
||||
continue
|
||||
self._entries[key] = {
|
||||
"state": "orphan",
|
||||
"vm": entry["vm"],
|
||||
"user_id": entry.get("user_id") or key,
|
||||
"paused_at": None,
|
||||
"last_used": None,
|
||||
}
|
||||
|
||||
def _save_registry(self) -> None:
|
||||
"""落盘当前条目(只写 paused/running/provisioning;
|
||||
orphan 是待清理的上一世条目,reconcile 后即删,不写
|
||||
回)。调用方须已持锁;写失败只记日志,不炸任务流。"""
|
||||
if not self.registry_path:
|
||||
return
|
||||
data = {}
|
||||
for key, entry in self._entries.items():
|
||||
if entry["state"] not in _RESIDENT_STATES:
|
||||
continue
|
||||
data[key] = {
|
||||
"state": entry["state"],
|
||||
"vm": entry.get("vm"),
|
||||
"user_id": entry.get("user_id"),
|
||||
"paused_at": entry.get("paused_at"),
|
||||
"last_used": entry.get("last_used"),
|
||||
}
|
||||
_write_registry(self.registry_path, data)
|
||||
|
||||
def _emit(self, event: str, task: dict, detail: str) -> None:
|
||||
"""best-effort 审计:执行器无 _emit 或抛错都不炸
|
||||
池操作(与 WarmPool 同款)。"""
|
||||
emit = getattr(self.executor, "_emit", None)
|
||||
if not callable(emit):
|
||||
return
|
||||
try:
|
||||
emit(event, task, detail)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def _idle_minutes_from_env(self) -> int:
|
||||
"""闲置阈值:执行器属性优先(P1-B 若经 config 落
|
||||
属性,直接复用),否则读 CHV_RESIDENT_IDLE_MINUTES;
|
||||
非数字/负数一律 0(不自动关机,不炸建池)。"""
|
||||
raw = getattr(self.executor, "resident_idle_minutes", None)
|
||||
if raw is None:
|
||||
env = getattr(self.executor, "_env", None)
|
||||
if env is None:
|
||||
env = os.environ
|
||||
raw = env.get("CHV_RESIDENT_IDLE_MINUTES", 0)
|
||||
if raw is None:
|
||||
return 0
|
||||
try:
|
||||
minutes = int(raw)
|
||||
except (TypeError, ValueError):
|
||||
return 0
|
||||
return minutes if minutes > 0 else 0
|
||||
|
||||
def _vm_name(self, resident_task: dict) -> str:
|
||||
"""常驻 VM 名:优先执行器 vm_name(与冷启动同款派
|
||||
生),鸭子型执行器无该方法时退回 chv- 前缀拼法。"""
|
||||
vm_name = getattr(self.executor, "vm_name", None)
|
||||
if callable(vm_name):
|
||||
try:
|
||||
return str(vm_name(resident_task))
|
||||
except Exception:
|
||||
pass
|
||||
return "chv-" + str(resident_task.get("id"))
|
||||
|
||||
# -- 计数 ----------------------------------------------------------
|
||||
def paused_count(self) -> int:
|
||||
"""当前 paused(可分配)的常驻条目数。"""
|
||||
with self._lock:
|
||||
return sum(1 for e in self._entries.values()
|
||||
if e.get("state") == "paused")
|
||||
|
||||
def orphan_count(self) -> int:
|
||||
"""上一世残留条目数(待 reconcile,不可分配)。"""
|
||||
with self._lock:
|
||||
return sum(1 for e in self._entries.values()
|
||||
if e.get("state") == "orphan")
|
||||
|
||||
# -- acquire / release ----------------------------------------------
|
||||
def _claim_paused(self, user_key: str) -> dict | None:
|
||||
"""占一条 paused 条目:加忙标记并返回其快照;无条目/
|
||||
非 paused/正忙时返回 None(调用方回落冷启动)。"""
|
||||
with self._lock:
|
||||
entry = self._entries.get(user_key)
|
||||
if entry is None or entry.get("state") != "paused":
|
||||
return None
|
||||
if user_key in self._busy:
|
||||
return None
|
||||
self._busy.add(user_key)
|
||||
return dict(entry)
|
||||
|
||||
def _resume_and_wait(self, resident_task: dict) -> bool:
|
||||
"""resume + 等 SSH 就绪(复用执行器判定)。
|
||||
resume 缺失的鸭子型执行器跳过 resume(与 WarmPool
|
||||
同款宽容);SSH 未就绪即 False(wait_guest_ready
|
||||
的失败路径已自行 stop_vm,调用方只须删条目)。"""
|
||||
resume = getattr(self.executor, "resume_vm", None)
|
||||
if callable(resume):
|
||||
try:
|
||||
resume(resident_task)
|
||||
except Exception:
|
||||
return False
|
||||
wait = getattr(self.executor, "wait_guest_ready", None)
|
||||
if not callable(wait):
|
||||
return True
|
||||
try:
|
||||
return bool(wait(resident_task))
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
def _settle_acquire(self, user_key: str, ok: bool) -> None:
|
||||
"""acquire 收尾:成功标 running,失败删条目;都清
|
||||
忙标记并落盘。"""
|
||||
with self._lock:
|
||||
self._busy.discard(user_key)
|
||||
entry = self._entries.get(user_key)
|
||||
if entry is None:
|
||||
return
|
||||
if ok:
|
||||
entry["state"] = "running"
|
||||
entry["last_used"] = time.time()
|
||||
else:
|
||||
self._entries.pop(user_key, None)
|
||||
self._save_registry()
|
||||
|
||||
def acquire(self, user_id: str | None) -> dict:
|
||||
"""取常驻 VM:paused 条目 → resume + SSH 就绪 →
|
||||
mode=resident;无条目/正忙/失败 → mode=cold(调用
|
||||
方回落冷启动)。失败条目直接丢弃(VM 本身已被
|
||||
wait_guest_ready 的失败路径关掉,或 resume 已拒)。"""
|
||||
user_key = resident_user_key(user_id)
|
||||
entry = self._claim_paused(user_key)
|
||||
if entry is None:
|
||||
return {"mode": "cold", "vm": None, "task": None}
|
||||
resident_task = resident_task_for_user(user_id)
|
||||
ok = self._resume_and_wait(resident_task)
|
||||
self._settle_acquire(user_key, ok)
|
||||
if not ok:
|
||||
return {"mode": "cold", "vm": None, "task": None}
|
||||
self._emit("resident_pool_acquire", resident_task,
|
||||
"vm=%s user=%s" % (entry.get("vm"), user_key))
|
||||
return {"mode": "resident", "vm": entry.get("vm"),
|
||||
"task": resident_task}
|
||||
|
||||
def _pause(self, resident_task: dict) -> bool:
|
||||
"""pause_vm 尽力而为:失败返回 False,不抛。"""
|
||||
pause = getattr(self.executor, "pause_vm", None)
|
||||
if not callable(pause):
|
||||
return False
|
||||
try:
|
||||
pause(resident_task)
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
def release(self, user_id: str | None) -> bool:
|
||||
"""还常驻 VM:running 条目 → pause_vm,不销毁。
|
||||
无条目/非 running/正忙 → False;pause 失败时条目
|
||||
留在 running(不删不销毁,worker 关闭时 reconcile
|
||||
清理),返回 False。只动本用户的条目。"""
|
||||
user_key = resident_user_key(user_id)
|
||||
resident_task = resident_task_for_user(user_id)
|
||||
with self._lock:
|
||||
entry = self._entries.get(user_key)
|
||||
if entry is None or entry.get("state") != "running":
|
||||
return False
|
||||
if user_key in self._busy:
|
||||
return False
|
||||
self._busy.add(user_key)
|
||||
paused = self._pause(resident_task)
|
||||
with self._lock:
|
||||
self._busy.discard(user_key)
|
||||
entry = self._entries.get(user_key)
|
||||
if entry is None or not paused:
|
||||
return False
|
||||
entry["state"] = "paused"
|
||||
entry["paused_at"] = time.time()
|
||||
self._save_registry()
|
||||
self._emit("resident_pool_release", resident_task,
|
||||
"vm=%s user=%s" % (entry.get("vm"), user_key))
|
||||
return True
|
||||
|
||||
# -- provision -------------------------------------------------------
|
||||
def _wake_and_pause(self, resident_task: dict) -> bool:
|
||||
"""建池:wake_vm(冷启动全套,含 SSH 就绪)再
|
||||
pause_vm。任一步失败都尽力 shutdown,不留半醒的
|
||||
VM;调用方随后删条目回落冷启动。"""
|
||||
try:
|
||||
wake = getattr(self.executor, "wake_vm", None)
|
||||
pause = getattr(self.executor, "pause_vm", None)
|
||||
if not callable(wake) or not callable(pause):
|
||||
return False
|
||||
wake(resident_task)
|
||||
pause(resident_task)
|
||||
return True
|
||||
except Exception:
|
||||
shutdown = getattr(self.executor, "shutdown_vm", None)
|
||||
if callable(shutdown):
|
||||
try:
|
||||
shutdown(resident_task)
|
||||
except Exception:
|
||||
pass
|
||||
return False
|
||||
|
||||
def _finish_provision(self, user_key: str, ok: bool) -> str | None:
|
||||
"""建池收尾:成功标 paused 并记时间,失败删条目;
|
||||
都清忙标记并落盘。返回 VM 名或 None。"""
|
||||
with self._lock:
|
||||
self._busy.discard(user_key)
|
||||
entry = self._entries.get(user_key)
|
||||
if entry is None:
|
||||
return None
|
||||
if not ok:
|
||||
self._entries.pop(user_key, None)
|
||||
self._save_registry()
|
||||
return None
|
||||
entry["state"] = "paused"
|
||||
entry["paused_at"] = time.time()
|
||||
self._save_registry()
|
||||
return str(entry.get("vm"))
|
||||
|
||||
def provision(self, user_id: str | None) -> dict:
|
||||
"""建常驻 VM:已有 paused/running 条目直接返回(不
|
||||
重 wake);否则 wake + pause,成功落 paused 条目,
|
||||
失败删条目回 cold。并发建池只放行一路,其余回 cold。"""
|
||||
user_key = resident_user_key(user_id)
|
||||
resident_task = resident_task_for_user(user_id)
|
||||
with self._lock:
|
||||
entry = self._entries.get(user_key)
|
||||
if entry is not None and entry.get("state") in (
|
||||
"paused", "running"):
|
||||
return {"mode": "resident", "vm": entry.get("vm"),
|
||||
"task": resident_task}
|
||||
if user_key in self._busy:
|
||||
return {"mode": "cold", "vm": None, "task": None}
|
||||
self._busy.add(user_key)
|
||||
self._entries[user_key] = {
|
||||
"state": "provisioning",
|
||||
"vm": self._vm_name(resident_task),
|
||||
"user_id": user_id if user_id is not None
|
||||
else _DEFAULT_USER,
|
||||
"paused_at": None, "last_used": None}
|
||||
self._save_registry()
|
||||
ok = self._wake_and_pause(resident_task)
|
||||
vm = self._finish_provision(user_key, ok)
|
||||
if vm is None:
|
||||
return {"mode": "cold", "vm": None, "task": None}
|
||||
self._emit("resident_pool_provision", resident_task,
|
||||
"vm=%s user=%s" % (vm, user_key))
|
||||
return {"mode": "resident", "vm": vm, "task": resident_task}
|
||||
|
||||
# -- 空闲回收 --------------------------------------------------------
|
||||
def _hibernate_one(self, user_key: str) -> str | None:
|
||||
"""单条目休眠:hibernate_vm(快照到 state_dir 内
|
||||
resident.snapshot,成功后删条目)。失败留条目下次
|
||||
再试,返回 None;无 state_dir 或无 hibernate 入口
|
||||
同样留条目。"""
|
||||
with self._lock:
|
||||
entry = self._entries.get(user_key)
|
||||
if entry is None or entry.get("state") != "paused":
|
||||
return None
|
||||
if user_key in self._busy:
|
||||
return None
|
||||
self._busy.add(user_key)
|
||||
resident_task = resident_task_for_user(
|
||||
entry.get("user_id"))
|
||||
vm = str(entry.get("vm"))
|
||||
state_dir = getattr(self.executor, "state_dir", None)
|
||||
hibernate = getattr(self.executor, "hibernate_vm", None)
|
||||
if not state_dir or not callable(hibernate):
|
||||
with self._lock:
|
||||
self._busy.discard(user_key)
|
||||
return None
|
||||
snapshot_url = "file://" + os.path.join(
|
||||
state_dir, vm, "resident.snapshot")
|
||||
try:
|
||||
hibernate(resident_task, snapshot_url)
|
||||
except Exception:
|
||||
with self._lock:
|
||||
self._busy.discard(user_key)
|
||||
return None
|
||||
with self._lock:
|
||||
self._busy.discard(user_key)
|
||||
self._entries.pop(user_key, None)
|
||||
self._save_registry()
|
||||
self._emit("resident_pool_reclaim", resident_task,
|
||||
"vm=%s snapshot=%s" % (vm, snapshot_url))
|
||||
return vm
|
||||
|
||||
def reclaim_idle(self, now: float | None = None) -> list[str]:
|
||||
"""空闲回收:paused 超阈值的条目 hibernate(pause +
|
||||
snapshot + shutdown)并删条目。阈值 0 即关闭回收;
|
||||
未到阈值/正忙/无 paused_at 的条目不动。返回回收的
|
||||
VM 名。"""
|
||||
if self.idle_minutes <= 0:
|
||||
return []
|
||||
if now is None:
|
||||
now = time.time()
|
||||
due: list[str] = []
|
||||
with self._lock:
|
||||
for key, entry in self._entries.items():
|
||||
if entry.get("state") != "paused":
|
||||
continue
|
||||
if key in self._busy:
|
||||
continue
|
||||
paused_at = entry.get("paused_at")
|
||||
if paused_at is None:
|
||||
continue
|
||||
if now - paused_at > self.idle_minutes * 60:
|
||||
due.append(key)
|
||||
reclaimed: list[str] = []
|
||||
for key in due:
|
||||
vm = self._hibernate_one(key)
|
||||
if vm is not None:
|
||||
reclaimed.append(vm)
|
||||
return reclaimed
|
||||
|
||||
# -- 孤儿清理 --------------------------------------------------------
|
||||
def _shutdown_one(self, user_key: str) -> str | None:
|
||||
"""单条目尽力 shutdown 并从内存删掉(失败仍计入
|
||||
已清理,与 orphans 旧语义一致:不因单条卡住整批)。
|
||||
返回 VM 名或 None(条目已不在)。"""
|
||||
with self._lock:
|
||||
entry = self._entries.get(user_key)
|
||||
if entry is None:
|
||||
return None
|
||||
vm = str(entry.get("vm"))
|
||||
shutdown = getattr(self.executor, "shutdown_vm", None)
|
||||
if callable(shutdown):
|
||||
try:
|
||||
shutdown(vm)
|
||||
except Exception:
|
||||
pass
|
||||
with self._lock:
|
||||
self._entries.pop(user_key, None)
|
||||
return vm
|
||||
|
||||
def reconcile_orphans(self) -> list[str]:
|
||||
"""清理本池内存里的 orphan 条目(重启后读到的上一
|
||||
世条目):每个尽力 shutdown_vm,条目清掉并落盘。
|
||||
生产启动/关闭不走本方法,走模块级 facade。"""
|
||||
with self._lock:
|
||||
orphans = [k for k, e in self._entries.items()
|
||||
if e.get("state") == "orphan"]
|
||||
cleaned: list[str] = []
|
||||
for key in orphans:
|
||||
vm = self._shutdown_one(key)
|
||||
if vm is not None:
|
||||
cleaned.append(vm)
|
||||
if cleaned:
|
||||
with self._lock:
|
||||
self._save_registry()
|
||||
self._emit("resident_pool_reconcile", {"id": None},
|
||||
"cleaned=%s" % ",".join(cleaned))
|
||||
return cleaned
|
||||
@@ -0,0 +1,345 @@
|
||||
"""Sentinel 式出口审批决策引擎(feat-chv-muse-p3,叶子模块)。
|
||||
|
||||
对齐 Muse 官方安全文(security.muse.ai 2026-09-08,见
|
||||
docs/chv-muse-p3-gap.md):Sentinel 是连接器动作与全部网络出口的
|
||||
唯一权限机构;L4 与 L7 评估(主机名、解析后与最终目的 IP、端口、
|
||||
协议、HTTP 方法、路径、解码后的实际请求);SSRF 防护(防止公开
|
||||
主机名在 DNS 解析后指向私有基础设施);审批是严格能力
|
||||
(strict capabilities),绑定到特定连接器/目的地与用例,支持
|
||||
一次性/会话级/任务级/时限级授权;审批 UI 不在对话内。
|
||||
|
||||
本模块只是决策层(纯函数 + 内存态队列),不是网络层强制:
|
||||
- 它不拦截客内流量。客内直连的强制仍靠 Tinyproxy 白名单 +
|
||||
宿主防火墙(第三棒落地中);eBPF cgroup 拦截 + LSM hook 污点
|
||||
传播是后续工作,见差距表 §3.4,不得把本模块写成“出口已全部
|
||||
经 Sentinel”。
|
||||
- 本期两个接线点:① 执行器 wake 时的 Gitea 出口预审(起 VMM 前
|
||||
fail-closed);② 客内 authd 放凭证前评估。两者都调本模块的
|
||||
`evaluate()`,决策记录进审计(无值)。
|
||||
- `ask` 的语义:主机不在白名单、也不在拒绝名单时返回 ask,并记入
|
||||
`ApprovalQueue` 待审批;调用方遇到 ask 必须停止执行、不得自动
|
||||
放行。审批 UI 不在对话内——本期只留队列,不实现任何对话内审批
|
||||
入口(见差距表 §1)。
|
||||
|
||||
本模块只用标准库,不依赖 cloud.*,客内 authd 守护进程可直接
|
||||
import 它(见 chv-guest/authd.py)。解析器可注入(`resolve`
|
||||
参数),默认 `socket.getaddrinfo`;测试用假解析器,绝不联网。
|
||||
审计记录里绝不含凭证值、也绝不含解析出的 IP 以外的内容——
|
||||
reason 只含网段名与策略名。
|
||||
"""
|
||||
|
||||
import ipaddress
|
||||
import socket
|
||||
import time
|
||||
import uuid
|
||||
from collections.abc import Callable, Iterable
|
||||
|
||||
# SSRF 挡截:解析到这些网段的 IP 一律 deny。公开主机名解析到
|
||||
# 私有地址即挡(Muse 安全文的 SSRF 防护口径);解析失败也
|
||||
# deny(fail-closed,不把“解析不出”当成“可以放行”)。
|
||||
_BLOCKED_NETWORKS = tuple(
|
||||
ipaddress.ip_network(net) for net in (
|
||||
"0.0.0.0/8", # 本网络 / 未指定
|
||||
"10.0.0.0/8", # RFC1918
|
||||
"100.64.0.0/10", # CGNAT 共享地址空间
|
||||
"127.0.0.0/8", # 回环
|
||||
"169.254.0.0/16", # 链路本地
|
||||
"172.16.0.0/12", # RFC1918
|
||||
"192.0.0.0/24", # IETF 协议分配
|
||||
"192.0.2.0/24", # 文档 TEST-NET-1
|
||||
"192.88.99.0/24", # 6to4 中继(已废弃)
|
||||
"192.168.0.0/16", # RFC1918
|
||||
# 注意:198.18.0.0/15(基准测试)故意不在表里——本部署的
|
||||
# egress/DNS 基础设施自用该网段,gitea.tose.top 在此环境
|
||||
# 真实解析到 198.18.139.152;挡掉它会把真 Gitea 主机也拒了。
|
||||
"198.51.100.0/24", # 文档 TEST-NET-2
|
||||
"203.0.113.0/24", # 文档 TEST-NET-3
|
||||
"224.0.0.0/4", # 组播
|
||||
"240.0.0.0/4", # 保留
|
||||
"255.255.255.255/32",
|
||||
"::1/128", # 回环
|
||||
"::/128", # 未指定
|
||||
"fc00::/7", # 唯一本地
|
||||
"fe80::/10", # 链路本地
|
||||
"ff00::/8", # 组播
|
||||
))
|
||||
|
||||
# 每连接器的 L7 路径策略:(方法, 匹配方式, 模式)。
|
||||
# gitea:API 只放 /api/v1/;git 智能 HTTP 只放三条(clone/pull/push
|
||||
# 的实际路径,见 git http-protocol)加裸 .git(round-11 真机修复
|
||||
# 层 4:git 智能 HTTP 的发现请求只发裸 .git 路径,裸路径只放 GET、
|
||||
# 一律当 pull;POST 裸 .git 仍拒绝)。其它路径(Web UI、登录页、
|
||||
# 任意管理接口)一律不在本连接器的能力内。
|
||||
_CONNECTOR_PATH_RULES = {
|
||||
"gitea": (
|
||||
("GET", "prefix", "/api/v1/"),
|
||||
("POST", "prefix", "/api/v1/"),
|
||||
("GET", "suffix", ".git/info/refs"),
|
||||
("GET", "suffix", ".git"),
|
||||
("POST", "suffix", ".git/git-upload-pack"),
|
||||
("POST", "suffix", ".git/git-receive-pack"),
|
||||
),
|
||||
}
|
||||
|
||||
# 每连接器允许的 scheme/端口(默认:只 https/443;http 只在
|
||||
# 显式放行时才出现,本期没有任何连接器放 http)。
|
||||
_CONNECTOR_SCHEMES = {"gitea": frozenset({"https"})}
|
||||
_CONNECTOR_PORTS = {"gitea": frozenset({443})}
|
||||
|
||||
# 默认放凭证有效期(秒):与 authd 按次释放同款 300 秒。
|
||||
DEFAULT_CAPABILITY_TTL_SECONDS = 300
|
||||
|
||||
|
||||
def _blocked_network_for(ip_text: str):
|
||||
"""该 IP 落在哪个挡截网段里(无即 None)。非法 IP 文本
|
||||
按“挡不住也说不清”处理:返回 0.0.0.0/8(deny),绝不
|
||||
把解析异常当成放行信号。"""
|
||||
try:
|
||||
addr = ipaddress.ip_address(ip_text)
|
||||
except ValueError:
|
||||
return _BLOCKED_NETWORKS[0]
|
||||
for net in _BLOCKED_NETWORKS:
|
||||
if addr.version == net.version and addr in net:
|
||||
return net
|
||||
return None
|
||||
|
||||
|
||||
def _default_resolve(host: str, port: int) -> list:
|
||||
"""默认解析器:socket.getaddrinfo 去重后的 IP 列表。"""
|
||||
infos = socket.getaddrinfo(host, port, proto=socket.IPPROTO_TCP)
|
||||
out = []
|
||||
for info in infos:
|
||||
ip = info[4][0]
|
||||
if ip not in out:
|
||||
out.append(ip)
|
||||
return out
|
||||
|
||||
|
||||
class EgressRequest:
|
||||
"""一次出口请求(决策输入):连接器 + 方法 + 目标 + 解析后
|
||||
IP(解析在 evaluate 里做,这里只记目标描述,不记任何凭证值)。"""
|
||||
|
||||
def __init__(self, connector: str, method: str, scheme: str,
|
||||
host: str, port: int, path: str) -> None:
|
||||
self.connector = connector
|
||||
self.method = (method or "").upper()
|
||||
self.scheme = (scheme or "").lower()
|
||||
self.host = (host or "").lower()
|
||||
self.port = int(port)
|
||||
self.path = path or "/"
|
||||
|
||||
def describe(self) -> dict:
|
||||
"""审计用描述(只含连接器/目标/方法,不含值)。"""
|
||||
return {"connector": self.connector, "method": self.method,
|
||||
"scheme": self.scheme, "host": self.host,
|
||||
"port": self.port, "path": self.path}
|
||||
|
||||
|
||||
class Capability:
|
||||
"""严格能力(Muse 口径:绑定到特定连接器/目的地与用例)。
|
||||
|
||||
一次 evaluate().capability 只对:一个连接器、一个主机、声明
|
||||
的方法与路径模式、在有效期内、剩余次数内有效。越界使用
|
||||
(换主机、换方法、换路径、过期、用完)一律 `matches()` 为
|
||||
False,调用方必须重新走 evaluate,不得自行续期。"""
|
||||
|
||||
def __init__(self, connector: str, host: str,
|
||||
methods: Iterable[str],
|
||||
path_rules: Iterable[tuple[str, str, str]],
|
||||
ttl_seconds: int = DEFAULT_CAPABILITY_TTL_SECONDS,
|
||||
max_uses: int = 1, now: float | None = None) -> None:
|
||||
self.id = uuid.uuid4().hex[:16]
|
||||
self.connector = connector
|
||||
self.host = (host or "").lower()
|
||||
self.methods = frozenset(methods)
|
||||
self.path_rules = tuple(path_rules)
|
||||
self.expires_at = (now if now is not None else time.time()) \
|
||||
+ ttl_seconds
|
||||
self.max_uses = max_uses
|
||||
self.uses_left = max_uses
|
||||
|
||||
def matches(self, request: EgressRequest,
|
||||
now: float | None = None) -> bool:
|
||||
"""该请求是否在能力绑定内(不消耗次数,只判定)。"""
|
||||
if request.connector != self.connector:
|
||||
return False
|
||||
if request.host != self.host:
|
||||
return False
|
||||
if request.method not in self.methods:
|
||||
return False
|
||||
if not _path_allowed(request, self.path_rules):
|
||||
return False
|
||||
at = now if now is not None else time.time()
|
||||
if at >= self.expires_at:
|
||||
return False
|
||||
if self.uses_left <= 0:
|
||||
return False
|
||||
return True
|
||||
|
||||
def use(self, request: EgressRequest,
|
||||
now: float | None = None) -> bool:
|
||||
"""消耗一次能力:绑定内才扣次数并返回 True,否则 False
|
||||
(不抛错,调用方按 deny 处理)。"""
|
||||
if not self.matches(request, now=now):
|
||||
return False
|
||||
self.uses_left -= 1
|
||||
return True
|
||||
|
||||
|
||||
class Decision:
|
||||
"""一次审批决策:allow / deny / ask + 原因 + 能力(仅 allow
|
||||
时有)+ 审计记录(无值)。"""
|
||||
|
||||
def __init__(self, verdict: str, reasons: Iterable[str],
|
||||
capability: Capability | None = None) -> None:
|
||||
assert verdict in ("allow", "deny", "ask")
|
||||
self.verdict = verdict
|
||||
self.reasons = list(reasons)
|
||||
self.capability = capability
|
||||
|
||||
def audit(self, request: EgressRequest) -> dict:
|
||||
"""审计记录:事件名按 verdict,detail 只含目标描述与原因。"""
|
||||
record = request.describe()
|
||||
record["event"] = "sentinel_egress_" + self.verdict
|
||||
record["verdict"] = self.verdict
|
||||
record["reasons"] = list(self.reasons)
|
||||
if self.capability is not None:
|
||||
record["capability_id"] = self.capability.id
|
||||
return record
|
||||
|
||||
|
||||
def _path_allowed(request: EgressRequest, rules) -> bool:
|
||||
"""路径是否在该连接器的 L7 规则内。"""
|
||||
for method, kind, pattern in rules:
|
||||
if request.method != method:
|
||||
continue
|
||||
if kind == "prefix" and request.path.startswith(pattern):
|
||||
return True
|
||||
if kind == "suffix" and request.path.endswith(pattern):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
class SentinelPolicy:
|
||||
"""Sentinel 策略:白名单主机 + 拒绝主机 + 连接器 L7 规则。
|
||||
|
||||
默认拒绝(default-deny):主机不在白名单、也不在拒绝名单时
|
||||
返回 ask(不是 allow);拒绝名单命中、SSRF 命中、scheme/
|
||||
端口/路径越界时返回 deny。白名单只放行“已知且解析干净”的
|
||||
目标——白名单不是“解析后是什么都行”,SSRF 检查对白名单
|
||||
主机同样生效(DNS 可被投毒,白名单主机解析到私网 IP 仍 deny)。"""
|
||||
|
||||
def __init__(self, allow_hosts: Iterable[str] = (),
|
||||
deny_hosts: Iterable[str] = (),
|
||||
connector_paths: dict[str, tuple] | None = None
|
||||
) -> None:
|
||||
self.allow_hosts = frozenset(
|
||||
h.lower() for h in (allow_hosts or ()))
|
||||
self.deny_hosts = frozenset(
|
||||
h.lower() for h in (deny_hosts or ()))
|
||||
self.connector_paths = dict(
|
||||
connector_paths if connector_paths is not None
|
||||
else _CONNECTOR_PATH_RULES)
|
||||
|
||||
@classmethod
|
||||
def from_env(cls: type["SentinelPolicy"], env: dict | None,
|
||||
extra_allow_hosts: Iterable[str] = ()
|
||||
) -> "SentinelPolicy":
|
||||
"""从宿主环境变量建策略:CHV_SENTINEL_ALLOW_HOSTS(逗号
|
||||
分隔主机名)+ 调用方追加的白名单(执行器 wake 预审时把
|
||||
配置的 Gitea 主机追加进来——只放行配置的 Gitea 主机,
|
||||
不是放行任意主机)。"""
|
||||
raw = (env or {}).get("CHV_SENTINEL_ALLOW_HOSTS") or ""
|
||||
hosts = [h.strip().lower() for h in raw.split(",") if h.strip()]
|
||||
hosts += [h.lower() for h in (extra_allow_hosts or ())]
|
||||
return cls(allow_hosts=hosts)
|
||||
|
||||
def evaluate(self, request: EgressRequest,
|
||||
resolve: Callable[[str, int], list] | None = None,
|
||||
now: float | None = None) -> Decision:
|
||||
"""评估一次出口请求(按顺序,首个命中即定 verdict):
|
||||
|
||||
1. 未知连接器 → deny(没有策略的连接器不放行);
|
||||
2. scheme 不在该连接器允许集 → deny;
|
||||
3. 端口不在该连接器允许集 → deny;
|
||||
4. 主机在拒绝名单 → deny;
|
||||
5. 解析失败 → deny(fail-closed);
|
||||
6. 任一解析 IP 落在挡截网段 → deny(SSRF);
|
||||
7. 主机不在白名单 → ask(记入待审批队列,调用方停止执行);
|
||||
8. 方法/路径不在连接器 L7 规则内 → deny;
|
||||
9. 全部通过 → allow + 严格能力(单次、300 秒)。
|
||||
"""
|
||||
rules = self.connector_paths.get(request.connector)
|
||||
if rules is None:
|
||||
return Decision("deny", ["unknown_connector"])
|
||||
if request.scheme not in _CONNECTOR_SCHEMES.get(
|
||||
request.connector, frozenset()):
|
||||
return Decision("deny", ["scheme_not_allowed"])
|
||||
if request.port not in _CONNECTOR_PORTS.get(
|
||||
request.connector, frozenset()):
|
||||
return Decision("deny", ["port_not_allowed"])
|
||||
if request.host in self.deny_hosts:
|
||||
return Decision("deny", ["host_denied"])
|
||||
resolver = resolve or _default_resolve
|
||||
try:
|
||||
ips = resolver(request.host, request.port)
|
||||
except Exception:
|
||||
return Decision("deny", ["dns_resolution_failed"])
|
||||
if not ips:
|
||||
return Decision("deny", ["dns_resolution_failed"])
|
||||
for ip in ips:
|
||||
net = _blocked_network_for(ip)
|
||||
if net is not None:
|
||||
return Decision(
|
||||
"deny", ["ssrf_private_ip:%s" % net])
|
||||
if request.host not in self.allow_hosts:
|
||||
return Decision("ask", ["host_not_allowlisted"])
|
||||
if not _path_allowed(request, rules):
|
||||
return Decision("deny", ["path_not_allowed"])
|
||||
capability = Capability(
|
||||
request.connector, request.host,
|
||||
{request.method}, rules,
|
||||
ttl_seconds=DEFAULT_CAPABILITY_TTL_SECONDS,
|
||||
max_uses=1, now=now)
|
||||
return Decision("allow", ["policy_allow"],
|
||||
capability=capability)
|
||||
|
||||
|
||||
class ApprovalQueue:
|
||||
"""ask 待审批队列(内存态):ask 出现即记入,执行停止。
|
||||
|
||||
设计约束(Muse 口径):审批 UI 不在对话内。本期只留队列与
|
||||
查询/标记接口,不实现任何对话内审批入口,也不自动放行。
|
||||
队列是内存态,进程重启即清——ask 本来就意味着“这次执行
|
||||
已停止”,重启后须重新申请。"""
|
||||
|
||||
def __init__(self, clock: Callable[[], float] | None = None) -> None:
|
||||
self._pending: dict[str, dict] = {}
|
||||
self._clock = clock or time.time
|
||||
|
||||
def record(self, request: EgressRequest,
|
||||
reasons: Iterable[str]) -> str:
|
||||
"""记入一条待审批,返回 pending id(只含目标描述与原因)。"""
|
||||
pending_id = uuid.uuid4().hex[:16]
|
||||
record = request.describe()
|
||||
record["pending_id"] = pending_id
|
||||
record["reasons"] = list(reasons)
|
||||
record["ts"] = self._clock()
|
||||
self._pending[pending_id] = record
|
||||
return pending_id
|
||||
|
||||
def list_pending(self) -> list:
|
||||
"""全部待审批(副本,按记录时间排序)。"""
|
||||
return [dict(v) for _, v in sorted(
|
||||
self._pending.items(), key=lambda kv: kv[1]["ts"])]
|
||||
|
||||
def resolve(self, pending_id: str, verdict: str) -> bool:
|
||||
"""标记一条待审批已处理(verdict 只记 allow/deny,不在
|
||||
这里执行放行——放行须调用方重新走 evaluate)。未知 id
|
||||
返回 False,不抛错。"""
|
||||
if pending_id not in self._pending:
|
||||
return False
|
||||
if verdict not in ("allow", "deny"):
|
||||
return False
|
||||
self._pending.pop(pending_id)
|
||||
return True
|
||||
+95
-19
@@ -44,12 +44,26 @@ StrictHostKeyChecking=no + UserKnownHostsFile=/dev/null):
|
||||
import os
|
||||
import shlex
|
||||
|
||||
from cloud.chv.atomic_file import atomic_write_text
|
||||
from cloud.chv.gitea_sync import (
|
||||
GITEA_SYNC_PLAIN_KEYS,
|
||||
GITEA_SYNC_SECRET_KEYS,
|
||||
)
|
||||
|
||||
# CHV 密钥类环境变量名:值绝不进 ssh argv。Docker 线是
|
||||
# “-e 只传名、值由 CLI 子进程环境提供”(值仍在宿主进程
|
||||
# 环境里,/proc/<pid>/environ 对同用户可见);CHV 线是
|
||||
# “值只进 seed 盘客内文件”(宿主进程表与宿主进程环境里
|
||||
# 都没有值),比 Docker 线更严——差异见执行器文档。
|
||||
_CHV_SECRET_ENV_NAMES = frozenset({"SUB2API_API_KEY"})
|
||||
_CHV_SECRET_ENV_NAMES = frozenset({"SUB2API_API_KEY"}) \
|
||||
| GITEA_SYNC_SECRET_KEYS
|
||||
|
||||
# CHV guest-env 类非密钥变量名:值可进 SSH env 前缀(按定义
|
||||
# 不是密钥),但 chv-sync boot 在 cloud-init 时运行、SSH 之
|
||||
# 前,只能读 seed 投递的 /etc/chv/guest-env,所以这些键也
|
||||
# 必须随 seed 写进 guest-env(见 split_guest_env 与执行器
|
||||
# wake_vm),只走 SSH 前缀是不够的。
|
||||
_CHV_GUEST_ENV_NAMES = frozenset(GITEA_SYNC_PLAIN_KEYS)
|
||||
|
||||
|
||||
def split_secret_env(extra_env: dict | None) -> tuple[dict, dict]:
|
||||
@@ -63,6 +77,80 @@ def split_secret_env(extra_env: dict | None) -> tuple[dict, dict]:
|
||||
return secrets, plain
|
||||
|
||||
|
||||
def split_guest_env(extra_env: dict | None) -> tuple[dict, dict]:
|
||||
"""把 extra_env 拆成(须写进 seed guest-env 的子集,
|
||||
非密钥子集)。
|
||||
|
||||
guest-env 子集 = 密钥子集 + _CHV_GUEST_ENV_NAMES 命中的
|
||||
非密钥同步变量(GITEA_URL 等 boot 时就要用的键);第二
|
||||
个返回仍是全部非密钥子集(含 guest-env 命中键——它们
|
||||
任务期也要经 SSH env 前缀进客机环境,seed 只解决
|
||||
boot 时可见性,不替代任务期前缀)。"""
|
||||
secrets, plain = split_secret_env(extra_env)
|
||||
guest_env = dict(secrets)
|
||||
for key, value in plain.items():
|
||||
if key in _CHV_GUEST_ENV_NAMES:
|
||||
guest_env[key] = value
|
||||
return guest_env, plain
|
||||
|
||||
|
||||
def _merge_no_proxy(extra_env, egress_proxy):
|
||||
"""调用方 NO_PROXY/no_proxy 与 egress 的 localhost、
|
||||
127.0.0.1 去重合并(单条 NO_PROXY 的值)。"""
|
||||
parts = []
|
||||
if extra_env:
|
||||
for key in ("NO_PROXY", "no_proxy"):
|
||||
val = extra_env.get(key)
|
||||
if val:
|
||||
parts += [p.strip() for p in str(val).split(",")
|
||||
if p.strip()]
|
||||
merged = []
|
||||
for part in parts:
|
||||
if part not in merged:
|
||||
merged.append(part)
|
||||
if egress_proxy:
|
||||
for part in ("localhost", "127.0.0.1"):
|
||||
if part not in merged:
|
||||
merged.append(part)
|
||||
return merged
|
||||
|
||||
|
||||
def _env_assignments(extra_env, egress_proxy):
|
||||
"""env 前缀的赋值列表(feat-chv-closeup,H1 附带)。
|
||||
|
||||
NO_PROXY 覆盖顺序已核实并修掉:旧拼装是 extra_env 的
|
||||
赋值在前、egress 的 NO_PROXY 在后,`env` 遇重复赋值以
|
||||
后者为准(实测 env NO_PROXY=a NO_PROXY=b → b),调用方
|
||||
传的 NO_PROXY 被 egress 的 localhost,127.0.0.1 静默覆盖。
|
||||
现改为合并(见 _merge_no_proxy),单条 NO_PROXY 发出。
|
||||
egress 代理本身须赢:egress 已设时调用方的
|
||||
HTTPS_PROXY/HTTP_PROXY(含小写,curl 等工具读小写优先,
|
||||
留着会架空 egress)一律不发出,只发 egress 的三条;
|
||||
无 egress 时调用方的值原样保留(只去重,不丢)。"""
|
||||
skip = {"NO_PROXY", "no_proxy"}
|
||||
if egress_proxy:
|
||||
skip |= {"HTTPS_PROXY", "HTTP_PROXY",
|
||||
"https_proxy", "http_proxy"}
|
||||
assignments = []
|
||||
if extra_env:
|
||||
for key in sorted(extra_env):
|
||||
if key in skip:
|
||||
continue
|
||||
assignments.append("%s=%s"
|
||||
% (key, shlex.quote(
|
||||
str(extra_env[key]))))
|
||||
merged = _merge_no_proxy(extra_env, egress_proxy)
|
||||
if egress_proxy:
|
||||
assignments += [
|
||||
"HTTPS_PROXY=%s" % shlex.quote(egress_proxy),
|
||||
"HTTP_PROXY=%s" % shlex.quote(egress_proxy),
|
||||
"NO_PROXY=%s" % shlex.quote(",".join(merged))]
|
||||
elif merged:
|
||||
assignments += [
|
||||
"NO_PROXY=%s" % shlex.quote(",".join(merged))]
|
||||
return assignments
|
||||
|
||||
|
||||
def remote_command(inner_cmd: list, egress_proxy: str | None = None,
|
||||
extra_env: dict | None = None,
|
||||
guest_env_file: str | None = None) -> list:
|
||||
@@ -74,7 +162,8 @@ def remote_command(inner_cmd: list, egress_proxy: str | None = None,
|
||||
set +a`(把 seed 投递的密钥文件 source 进环境并 export),
|
||||
再接 env 前缀与客内命令;文件路径本身不是密钥。
|
||||
extra_env 含密钥名时 ValueError(调用方须先 split 出
|
||||
密钥子集走 seed 投递,见 split_secret_env)。"""
|
||||
密钥子集走 seed 投递,见 split_secret_env)。env 前缀的
|
||||
拼装见 _env_assignments(NO_PROXY 合并、egress 优先)。"""
|
||||
if extra_env:
|
||||
secrets, _ = split_secret_env(extra_env)
|
||||
if secrets:
|
||||
@@ -85,17 +174,7 @@ def remote_command(inner_cmd: list, egress_proxy: str | None = None,
|
||||
if guest_env_file:
|
||||
tokens += ["set", "-a;", ".",
|
||||
shlex.quote(guest_env_file) + ";", "set", "+a;"]
|
||||
env_assignments = []
|
||||
if extra_env:
|
||||
for key in sorted(extra_env):
|
||||
env_assignments.append("%s=%s"
|
||||
% (key, shlex.quote(
|
||||
str(extra_env[key]))))
|
||||
if egress_proxy:
|
||||
env_assignments += [
|
||||
"HTTPS_PROXY=%s" % shlex.quote(egress_proxy),
|
||||
"HTTP_PROXY=%s" % shlex.quote(egress_proxy),
|
||||
"NO_PROXY=localhost,127.0.0.1"]
|
||||
env_assignments = _env_assignments(extra_env, egress_proxy)
|
||||
inner = [shlex.quote(str(arg)) for arg in inner_cmd]
|
||||
if env_assignments:
|
||||
tokens += ["env"] + env_assignments + inner
|
||||
@@ -168,13 +247,10 @@ def parse_keyscan_output(text: str, host: str = "127.0.0.1") -> list:
|
||||
|
||||
|
||||
def write_known_hosts(path: str, key_lines: list) -> None:
|
||||
"""原子写 known_hosts:先写同目录临时文件再 rename,
|
||||
"""原子写 known_hosts:随机 tmp 名 + fsync 后 rename
|
||||
(feat-chv-closeup,P2-18:旧固定 `<path>.tmp` 无 fsync),
|
||||
文件权限 0600(只含公钥行,仍按敏感文件对待)。"""
|
||||
directory = os.path.dirname(path)
|
||||
if directory:
|
||||
os.makedirs(directory, exist_ok=True)
|
||||
tmp_path = path + ".tmp"
|
||||
with open(tmp_path, "w") as fh:
|
||||
fh.write("\n".join(key_lines) + "\n")
|
||||
os.chmod(tmp_path, 0o600)
|
||||
os.replace(tmp_path, path)
|
||||
atomic_write_text(path, "\n".join(key_lines) + "\n", mode=0o600)
|
||||
@@ -0,0 +1,223 @@
|
||||
"""宿主侧 vsock notify 最小实现(P1,feat-chv-muse-p1)。
|
||||
|
||||
方向只一个:客机 → 宿主。客机连宿主 CID 2 的
|
||||
VSOCK_NOTIFY_PORT 端口发 ready/心跳(客内脚本
|
||||
cloud/chv/guest/vsock_notify.py);宿主在
|
||||
``<vsock socket>_<port>`` 这个 Unix socket 上监听接收。
|
||||
|
||||
形态依据(CHV 官方 docs/vsock.md,upstream 原文):
|
||||
- ``--vsock cid=3,socket=/tmp/ch.vsock``:cid 是客机 CID,
|
||||
socket 是宿主侧 Unix socket 路径;
|
||||
- 客 → 宿主:宿主须先在 ``<socket>_<port>`` 上监听
|
||||
(如 ``/tmp/ch.vsock_1234``),客机连众所周知的 CID 2
|
||||
(Host);
|
||||
- 只支持 stream socket。
|
||||
|
||||
只做就绪辅助:wait_ready() 收到 ready 即返回,供
|
||||
_run_with_vm/常驻路径在 SSH 轮询之外多一个早期信号;
|
||||
SSH 仍是唯一命令通道与就绪判定——vsock 超时、无消息、
|
||||
监听失败一律返回 None,绝不因此判任务失败、绝不抛到
|
||||
调用方。消息是 JSON 行:
|
||||
|
||||
{"type": "ready"|"heartbeat", "vm": "<vm 名>", "ts": <unix 秒>}
|
||||
|
||||
不做宿主 → 客机方向(客机不监听,host.py 协议仍走 SSH)。
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import socket
|
||||
import time
|
||||
|
||||
_LOGGER = logging.getLogger("cloud.cloud_hypervisor")
|
||||
|
||||
# 宿主 CID(vsock 众所周知值,CHV 官方 docs/vsock.md)。
|
||||
HOST_CID = 2
|
||||
# 客机 CID 默认值(与 CHV_VSOCK_CID 默认值一致;0/1/2 保留)。
|
||||
DEFAULT_GUEST_CID = 3
|
||||
# notify 端口:客机连 (CID 2, 本端口) 发 ready/心跳;
|
||||
# 宿主监听 <socket>_<本端口>。与客内脚本的端口须一致。
|
||||
VSOCK_NOTIFY_PORT = 1024
|
||||
# 单行上限:客机发来的行超长即丢弃该连接(防脏数据撑内存)。
|
||||
MAX_LINE_BYTES = 65536
|
||||
# 合法消息类型。
|
||||
VALID_TYPES = ("ready", "heartbeat")
|
||||
|
||||
|
||||
def vsock_socket_path(state_dir: str, vm_name: str) -> str:
|
||||
"""该 VM 的 vsock socket 路径(放进 VmConfig 的值)。
|
||||
|
||||
与 build_vm_config 的派生一致:任务盘同目录的
|
||||
``state_dir/<vm>/vsock.sock``。vm_name 须与
|
||||
api_socket/任务盘用的名字相同(executor.vm_name(task)
|
||||
的返回值,含 chv- 前缀),这里不二次加前缀。"""
|
||||
return os.path.join(state_dir, vm_name, "vsock.sock")
|
||||
|
||||
|
||||
def notify_socket_path(socket_path: str,
|
||||
port: int = VSOCK_NOTIFY_PORT) -> str:
|
||||
"""宿主侧监听路径:``<socket>_<port>``(CHV 官方约定)。"""
|
||||
return "%s_%d" % (socket_path, port)
|
||||
|
||||
|
||||
def encode_notify(message_type: str, vm: str,
|
||||
ts: float | None = None) -> bytes:
|
||||
"""编码一条 notify 消息为 JSON 行(含换行)。
|
||||
|
||||
type 非法或 vm 为空即 ValueError(调用方是可信的客内
|
||||
脚本/宿主代码,编码错误是编程错误,早炸)。"""
|
||||
if message_type not in VALID_TYPES:
|
||||
raise ValueError("vsock notify 类型非法:%r" % (message_type,))
|
||||
if not vm:
|
||||
raise ValueError("vsock notify vm 不能为空")
|
||||
if ts is None:
|
||||
ts = time.time()
|
||||
line = json.dumps({"type": message_type, "vm": vm, "ts": ts},
|
||||
separators=(",", ":"))
|
||||
return (line + "\n").encode("utf-8")
|
||||
|
||||
|
||||
def decode_notify(line: bytes | str) -> dict | None:
|
||||
"""解码一行 notify 消息:合法返回 dict,否则返回 None。
|
||||
|
||||
客机字节不可信,任何畸形(非 JSON、非对象、类型不在
|
||||
ready/heartbeat 内、vm 空/非字符串、ts 非数字)一律
|
||||
None,绝不抛异常。"""
|
||||
try:
|
||||
if isinstance(line, bytes):
|
||||
line = line.decode("utf-8")
|
||||
obj = json.loads(line)
|
||||
except (UnicodeDecodeError, ValueError):
|
||||
return None
|
||||
if not isinstance(obj, dict):
|
||||
return None
|
||||
if obj.get("type") not in VALID_TYPES:
|
||||
return None
|
||||
if not isinstance(obj.get("vm"), str) or not obj.get("vm"):
|
||||
return None
|
||||
if not isinstance(obj.get("ts"), (int, float)) \
|
||||
or isinstance(obj.get("ts"), bool):
|
||||
return None
|
||||
return obj
|
||||
|
||||
|
||||
def _bind_notify_listener(listener, notify_path: str) -> bool:
|
||||
"""bind notify socket:残留文件先删一次再重试一次;
|
||||
两次都失败即 False(调用方记 warning 后返回 None,
|
||||
与原来直抛到外层 except 的行为一致)。"""
|
||||
try:
|
||||
listener.bind(notify_path)
|
||||
return True
|
||||
except OSError:
|
||||
pass
|
||||
try:
|
||||
os.unlink(notify_path)
|
||||
except OSError:
|
||||
pass
|
||||
try:
|
||||
listener.bind(notify_path)
|
||||
return True
|
||||
except OSError:
|
||||
return False
|
||||
|
||||
|
||||
def listen_notify(notify_path: str, timeout: float,
|
||||
want_type: str | None = None) -> dict | None:
|
||||
"""在 notify_path(Unix socket)上监听,等一条 notify。
|
||||
|
||||
收到首条合法消息即返回(want_type 给定时跳过其它类型
|
||||
继续等,直到超时);超时、客机断开未发合法消息、监听
|
||||
失败一律返回 None。监听用的 socket 文件用完即删(尽力
|
||||
而为);bind 前已存在的同名残留文件先删一次再 bind。
|
||||
timeout <= 0 即不等,直接返回 None。"""
|
||||
if timeout is None or timeout <= 0:
|
||||
return None
|
||||
deadline = time.monotonic() + timeout
|
||||
listener = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
|
||||
try:
|
||||
if not _bind_notify_listener(listener, notify_path):
|
||||
_LOGGER.warning("vsock notify 监听失败:%s", notify_path)
|
||||
return None
|
||||
listener.listen(1)
|
||||
while True:
|
||||
remaining = deadline - time.monotonic()
|
||||
if remaining <= 0:
|
||||
return None
|
||||
listener.settimeout(remaining)
|
||||
try:
|
||||
conn, _ = listener.accept()
|
||||
except socket.timeout:
|
||||
return None
|
||||
except OSError:
|
||||
_LOGGER.warning("vsock notify accept 失败:%s",
|
||||
notify_path)
|
||||
return None
|
||||
try:
|
||||
return _read_notify(conn, deadline, want_type)
|
||||
finally:
|
||||
try:
|
||||
conn.close()
|
||||
except OSError:
|
||||
pass
|
||||
except OSError:
|
||||
_LOGGER.warning("vsock notify 监听失败:%s", notify_path)
|
||||
return None
|
||||
finally:
|
||||
try:
|
||||
listener.close()
|
||||
except OSError:
|
||||
pass
|
||||
try:
|
||||
os.unlink(notify_path)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def _read_notify(conn, deadline: float,
|
||||
want_type: str | None) -> dict | None:
|
||||
"""从一条已接受的连接上读 JSON 行,直到合法消息或超时/
|
||||
断开。want_type 给定时跳过其它合法类型继续读。"""
|
||||
try:
|
||||
stream = conn.makefile("rb")
|
||||
except OSError:
|
||||
return None
|
||||
try:
|
||||
while True:
|
||||
remaining = deadline - time.monotonic()
|
||||
if remaining <= 0:
|
||||
return None
|
||||
try:
|
||||
conn.settimeout(remaining)
|
||||
line = stream.readline(MAX_LINE_BYTES + 1)
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
if not line:
|
||||
return None
|
||||
if len(line) > MAX_LINE_BYTES:
|
||||
return None
|
||||
msg = decode_notify(line)
|
||||
if msg is None:
|
||||
continue
|
||||
if want_type is not None and msg["type"] != want_type:
|
||||
continue
|
||||
return msg
|
||||
finally:
|
||||
try:
|
||||
stream.close()
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def wait_ready(state_dir: str, vm_name: str,
|
||||
timeout: float = 30,
|
||||
port: int = VSOCK_NOTIFY_PORT) -> dict | None:
|
||||
"""等一台 VM 的 vsock ready(就绪辅助,非 SSH 判定)。
|
||||
|
||||
在 ``state_dir/<vm>/vsock.sock_<port>`` 上监听,直到
|
||||
收到 {"type": "ready"} 或超时。返回 ready 消息 dict,
|
||||
超时/失败返回 None。调用方(_run_with_vm/常驻路径)
|
||||
拿到 None 时继续走 SSH 就绪轮询,绝不因此失败。"""
|
||||
path = notify_socket_path(
|
||||
vsock_socket_path(state_dir, vm_name), port)
|
||||
return listen_notify(path, timeout, want_type="ready")
|
||||
@@ -12,6 +12,9 @@ cloud.logging_config,不 import cloud.worker,绝无循环。
|
||||
(原 Worker.__init__ 里的闭包本体)。
|
||||
- reconcile_warm_pool:生产残留清理 facade 调用 + 审计
|
||||
(原 Worker._reconcile_warm_pool 的本体)。
|
||||
- reconcile_resident_pool:常驻池残留清理 facade 调用 +
|
||||
审计(新,feat-chv-muse-p1;调 resident_pool 的模块级
|
||||
facade,不构造 ResidentPool、不碰实验接口)。
|
||||
- cancel_watchdog_loop / quota_watchdog_loop:同步路径
|
||||
两条看门狗循环(原 Worker._chv_*_watchdog 的本体,
|
||||
self 依赖全部改成回调参数)。
|
||||
@@ -115,6 +118,38 @@ def reconcile_warm_pool(executor: object, audit_fn: Callable,
|
||||
return cleaned
|
||||
|
||||
|
||||
def reconcile_resident_pool(executor: object, audit_fn: Callable,
|
||||
when: str) -> list:
|
||||
"""显式调常驻残留清理 facade,返回已清理 VM 名。
|
||||
|
||||
仅 CHV worker 有效(其它执行器返回 [])。实现是
|
||||
cloud.chv.resident_pool.reconcile_resident_orphans_
|
||||
for_executor():直接读 resident-pool.json 做清理,不
|
||||
构造 ResidentPool、不碰实验接口。调用失败不炸主流程,
|
||||
但进审计(失败本身也审计,不静默丢弃)。orphan 条目
|
||||
绝不发给任务,reconcile 只做清理。"""
|
||||
if not isinstance(executor, CloudHypervisorExecutor):
|
||||
return []
|
||||
try:
|
||||
from cloud.chv.resident_pool import (
|
||||
reconcile_resident_orphans_for_executor as _reconcile)
|
||||
cleaned = _reconcile(executor) or []
|
||||
except Exception as exc:
|
||||
try:
|
||||
audit_fn(None, "chv_resident_pool_reconcile_failed",
|
||||
"when=%s error=%s" % (when, exc))
|
||||
except Exception:
|
||||
pass
|
||||
return []
|
||||
if cleaned:
|
||||
try:
|
||||
audit_fn(None, "chv_resident_pool_reconcile",
|
||||
"when=%s cleaned=%s" % (when, ",".join(cleaned)))
|
||||
except Exception:
|
||||
pass
|
||||
return cleaned
|
||||
|
||||
|
||||
def cancel_watchdog_loop(task_id: str, vm_name: str | None,
|
||||
stop_event: threading.Event,
|
||||
cancel_requested: set,
|
||||
|
||||
@@ -111,7 +111,7 @@ class Arch95Case(unittest.TestCase):
|
||||
"第一盘必须是任务盘")
|
||||
self.assertTrue(disks[1].endswith("seed.iso"),
|
||||
"第二盘必须是 seed 盘,缺 seed 即假保证")
|
||||
self.assertTrue(payload.get("kernel", {}).get("path"))
|
||||
self.assertTrue(payload.get("payload", {}).get("kernel"))
|
||||
nets = payload.get("net", [])
|
||||
taps = [n.get("tap") for n in nets]
|
||||
self.assertEqual(len(taps), len(set(taps)), "net tap 重复")
|
||||
|
||||
@@ -31,6 +31,7 @@ try:
|
||||
from cloud.chv.compat215 import REPLACEMENT_POINTS
|
||||
from cloud.chv.compat215 import resolve_inside, tmpfs_isolated
|
||||
from cloud.chv.config import _assemble_chv_config
|
||||
from cloud.chv.config import _balloon_vsock_fragment
|
||||
from cloud.chv.config import _base_fragment
|
||||
from cloud.chv.config import _disk_fragment
|
||||
from cloud.chv.config import _network_fragment
|
||||
@@ -48,6 +49,7 @@ except ImportError:
|
||||
from cloud.chv.compat215 import REPLACEMENT_POINTS
|
||||
from cloud.chv.compat215 import resolve_inside, tmpfs_isolated
|
||||
from cloud.chv.config import _assemble_chv_config
|
||||
from cloud.chv.config import _balloon_vsock_fragment
|
||||
from cloud.chv.config import _base_fragment
|
||||
from cloud.chv.config import _disk_fragment
|
||||
from cloud.chv.config import _network_fragment
|
||||
@@ -76,6 +78,10 @@ EXPECTED_FIELDS = {
|
||||
"snapshot_max_total_gb", "snapshot_retention_days",
|
||||
"snapshot_dir", "allow_remote_snapshot", "snapshot_url",
|
||||
"task_timeout", "api_timeout", "ready_timeout", "ready_interval",
|
||||
# P1-B(feat-chv-muse-p1):气球/vsock 三字段。
|
||||
"balloon", "vsock", "vsock_cid",
|
||||
# P3(feat-chv-muse-p3):Sentinel 白名单主机。
|
||||
"sentinel_allow_hosts",
|
||||
}
|
||||
|
||||
|
||||
@@ -249,13 +255,17 @@ class ConfigFragmentsTest(unittest.TestCase):
|
||||
def test_fragment_keys_disjoint_and_union_is_field_set(self):
|
||||
frags = [_base_fragment({}, {}), _ssh_fragment({}, {}),
|
||||
_network_fragment({}, {}), _quota_fragment({}),
|
||||
_disk_fragment({}), _timeout_fragment({}, {})]
|
||||
_disk_fragment({}), _timeout_fragment({}, {}),
|
||||
# P1-B + P3(round-11 并集):新片段加入并集断言
|
||||
#(字段 38 → 42:P1 气球/vsock 三字段 + P3
|
||||
# sentinel_allow_hosts)。
|
||||
_balloon_vsock_fragment({})]
|
||||
seen = set()
|
||||
for frag in frags:
|
||||
self.assertFalse(seen & set(frag), "片段键重叠")
|
||||
seen |= set(frag)
|
||||
self.assertEqual(seen, EXPECTED_FIELDS)
|
||||
self.assertEqual(len(EXPECTED_FIELDS), 38)
|
||||
self.assertEqual(len(EXPECTED_FIELDS), 42)
|
||||
|
||||
def test_defaults_and_overrides(self):
|
||||
cfg = build_chv_config({}, {})
|
||||
|
||||
@@ -213,6 +213,42 @@ class WakeFailureTest(CloseupCase):
|
||||
with self.assertRaises(ChvFailure) as ctx:
|
||||
ex.wake_vm(self._task())
|
||||
self.assertEqual(ctx.exception.kind, "vm_boot_failed")
|
||||
|
||||
def test_wake_vm_boot_failed_records_body(self):
|
||||
# part 4:vm.boot 500 只留 status 即无法定位——body 要进
|
||||
# 失败消息(截断后,见 _truncate_api_body)。
|
||||
def api(socket_path, method, path, payload=None):
|
||||
if method == "PUT" and path == "/vm.boot":
|
||||
return (500, {"error": "boom-detail-500"})
|
||||
return (200, {})
|
||||
ex = self.make_exec(api_client=api)
|
||||
with self.assertRaises(ChvFailure) as ctx:
|
||||
ex.wake_vm(self._task())
|
||||
self.assertEqual(ctx.exception.kind, "vm_boot_failed")
|
||||
self.assertIn("boom-detail-500", str(ctx.exception))
|
||||
self.assertIn("status=500", str(ctx.exception))
|
||||
|
||||
def test_wake_vm_create_failed_records_body(self):
|
||||
def api(socket_path, method, path, payload=None):
|
||||
if method == "PUT" and path == "/vm.create":
|
||||
return (500, "No bootitem named default")
|
||||
return (200, {})
|
||||
ex = self.make_exec(api_client=api)
|
||||
with self.assertRaises(ChvFailure) as ctx:
|
||||
ex.wake_vm(self._task())
|
||||
self.assertEqual(ctx.exception.kind, "vm_create_failed")
|
||||
self.assertIn("No bootitem named default", str(ctx.exception))
|
||||
|
||||
def test_wake_vm_boot_failed_body_truncated(self):
|
||||
def api(socket_path, method, path, payload=None):
|
||||
if method == "PUT" and path == "/vm.boot":
|
||||
return (500, {"error": "x" * 2000})
|
||||
return (200, {})
|
||||
ex = self.make_exec(api_client=api)
|
||||
with self.assertRaises(ChvFailure) as ctx:
|
||||
ex.wake_vm(self._task())
|
||||
self.assertIn("已截断", str(ctx.exception))
|
||||
self.assertLess(len(str(ctx.exception)), 1200)
|
||||
self.assertIn("vm_failed", self._audit_events())
|
||||
self.assertEqual(ex._vms, {})
|
||||
|
||||
@@ -312,6 +348,77 @@ class RestoreSnapshotTest(CloseupCase):
|
||||
out2 = ex.wake_from_hibernate(self._task(), url)
|
||||
self.assertEqual(out2["mode"], "restored")
|
||||
|
||||
def _register_other_vm_net(self, vm="chv-other"):
|
||||
vm_dir = os.path.join(self.state_dir, vm)
|
||||
os.makedirs(vm_dir, exist_ok=True)
|
||||
with open(os.path.join(vm_dir, "net.json"), "w") as fh:
|
||||
json.dump([{"tap": "tap0",
|
||||
"mac": "aa:bb:cc:dd:ee:01",
|
||||
"ip": "192.168.10.10",
|
||||
"mask": ""}], fh)
|
||||
|
||||
def test_restore_net_conflict_refused_before_launch(self):
|
||||
# feat-chv-cover100:restore 也须过 H7 冲突校验——
|
||||
# 此前 restore 路径不校验不登记,恢复出的 VM 会与
|
||||
# 已登记 VM 静默复用 tap/mac/ip,且自身不登记、后
|
||||
# 来的 wake 也看不到它,两边都漏判。
|
||||
url, _ = self._snap_url()
|
||||
self._register_other_vm_net()
|
||||
ex = self.make_exec(env_extra={
|
||||
"CHV_NET_JSON": '[{"tap": "tap0", '
|
||||
'"mac": "aa:bb:cc:dd:ee:02"}]'})
|
||||
with self.assertRaises(ChvFailure) as ctx:
|
||||
ex.restore_vm(self._task(), url)
|
||||
self.assertEqual(ctx.exception.kind, "net_tap_conflict")
|
||||
self.assertIn("vm_failed", self._audit_events())
|
||||
self.assertEqual(self.launches, [])
|
||||
self.assertFalse(os.path.exists(
|
||||
os.path.join(self.state_dir, "chv-abc123", "net.json")))
|
||||
|
||||
def test_restore_registers_net_entries(self):
|
||||
# feat-chv-cover100:恢复成功后登记 net.json,后来
|
||||
# 的 wake 才能看到恢复出的 VM 占用的 tap/mac/ip。
|
||||
url, _ = self._snap_url()
|
||||
ex = self.make_exec(env_extra={
|
||||
"CHV_NET_JSON": '[{"tap": "tap0", '
|
||||
'"mac": "aa:bb:cc:dd:ee:01", '
|
||||
'"ip": "192.168.10.10"}]'})
|
||||
out = ex.restore_vm(self._task(), url)
|
||||
self.assertEqual(out["mode"], "restored")
|
||||
with open(os.path.join(self.state_dir, "chv-abc123",
|
||||
"net.json")) as fh:
|
||||
entries = json.load(fh)
|
||||
self.assertEqual(entries[0]["tap"], "tap0")
|
||||
self.assertEqual(entries[0]["mac"], "aa:bb:cc:dd:ee:01")
|
||||
self.assertEqual(entries[0]["ip"], "192.168.10.10")
|
||||
|
||||
def test_restore_net_register_failure_refuses(self):
|
||||
# feat-chv-cover100:登记失败与 wake 同款审计+拒绝
|
||||
#(P2-19 语义),restore_vm 的 except 会 stop_vm
|
||||
# 清掉半恢复 VM(shutdown 已发出)。
|
||||
url, _ = self._snap_url()
|
||||
blocker = os.path.join(self.state_dir, "chv-abc123")
|
||||
os.makedirs(self.state_dir, exist_ok=True)
|
||||
with open(blocker, "w") as fh:
|
||||
fh.write("x")
|
||||
ex = self.make_exec(env_extra={
|
||||
"CHV_NET_JSON": '[{"tap": "tap0"}]'})
|
||||
with self.assertRaises(ChvFailure) as ctx:
|
||||
ex.restore_vm(self._task(), url)
|
||||
self.assertEqual(ctx.exception.kind,
|
||||
"net_registry_write_failed")
|
||||
self.assertIn("net_registry_write_failed",
|
||||
self._audit_events())
|
||||
self.assertTrue(any(c[-1] == "shutdown" for c in self.calls))
|
||||
|
||||
def test_restore_without_net_json_writes_no_registry(self):
|
||||
url, _ = self._snap_url()
|
||||
ex = self.make_exec()
|
||||
out = ex.restore_vm(self._task(), url)
|
||||
self.assertEqual(out["mode"], "restored")
|
||||
self.assertFalse(os.path.exists(
|
||||
os.path.join(self.state_dir, "chv-abc123", "net.json")))
|
||||
|
||||
def test_restore_provisioned_missing_known_hosts(self):
|
||||
url, _ = self._snap_url()
|
||||
ex = self.make_exec(env_extra={"CHV_HOST_KEY_MODE": "provisioned"})
|
||||
@@ -527,8 +634,16 @@ class VmDeathTest(CloseupCase):
|
||||
def runner(cmd, timeout=None):
|
||||
self.calls.append(list(cmd))
|
||||
if cmd[0] == "ssh" and "dispatch" in cmd:
|
||||
proc = ex._vms.get("chv-abc123")
|
||||
if proc is not None:
|
||||
# 死亡模拟须杀当时已登记的 VM(即实际跑命令
|
||||
# 的那台):P1 常驻路由下 dispatch 跑在常驻 VM
|
||||
#(chv-resident-*)上,任务自己的 chv-abc123
|
||||
# 从未 wake、登记里根本没有,盯着它杀是空操
|
||||
# 作、模拟落空(探针实证:登记里只有
|
||||
# chv-resident-default 且活着,异常直抛)。
|
||||
# 常驻 VM 真死时 _run_on_resident 的判定已验
|
||||
# 证正确(两态皆 interrupted),这里只修模拟
|
||||
# 器的靶子,不动断言与判定逻辑。
|
||||
for proc in list(ex._vms.values()):
|
||||
proc._alive = False
|
||||
if mode == "raise":
|
||||
raise RuntimeError("ssh broken")
|
||||
@@ -1047,12 +1162,18 @@ class DefensiveBranchTest(CloseupCase):
|
||||
marker = dirty_marker_path(self.state_dir, "chv-abc123")
|
||||
self.assertTrue(os.path.exists(marker))
|
||||
|
||||
def test_write_dirty_marker_failure_swallowed(self):
|
||||
def test_write_dirty_marker_failure_audited_not_raised(self):
|
||||
# feat-chv-closeup(P2-17):脏标记写失败仍不抛(stop
|
||||
# 契约不变),但不再全静默——返回 False + 独立审计
|
||||
# dirty_marker_write_failed。
|
||||
blocker = os.path.join(self.dir, "blocker2")
|
||||
with open(blocker, "w") as fh:
|
||||
fh.write("x")
|
||||
ex = self.make_exec(state_dir=blocker)
|
||||
ex._write_dirty_marker("chv-abc123", True) # 不抛
|
||||
ok = ex._write_dirty_marker("chv-abc123", True) # 不抛
|
||||
self.assertFalse(ok)
|
||||
self.assertIn("dirty_marker_write_failed",
|
||||
self._audit_events())
|
||||
|
||||
def test_stop_socket_remove_failure_swallowed(self):
|
||||
ex = self.make_exec()
|
||||
@@ -1290,7 +1411,10 @@ class ExecutorLastLinesTest(CloseupCase):
|
||||
ex.wait_guest_ready(self._task())
|
||||
self.assertEqual(ctx.exception.kind, "guest_not_ready")
|
||||
|
||||
def test_record_net_entries_write_failure_only_warns(self):
|
||||
def test_record_net_entries_write_failure_audited_and_reported(self):
|
||||
# feat-chv-closeup(P2-19):登记失败不再“只 warning”——
|
||||
# 返回 False + 独立审计 net_registry_write_failed,调用方
|
||||
# _boot_vm 据此拒起(见 test_chv_residual)。
|
||||
ex = self.make_exec()
|
||||
blocker = os.path.join(self.state_dir, "chv-x")
|
||||
os.makedirs(self.state_dir, exist_ok=True)
|
||||
@@ -1298,9 +1422,12 @@ class ExecutorLastLinesTest(CloseupCase):
|
||||
fh.write("x")
|
||||
with self.assertLogs("cloud.cloud_hypervisor",
|
||||
level="WARNING") as logs:
|
||||
ex._record_net_entries("chv-x", [{"tap": "tap0"}])
|
||||
ok = ex._record_net_entries("chv-x", [{"tap": "tap0"}])
|
||||
self.assertFalse(ok)
|
||||
self.assertTrue(any("net_registry_write_failed" in line
|
||||
for line in logs.output))
|
||||
self.assertIn("net_registry_write_failed",
|
||||
self._audit_events())
|
||||
|
||||
def test_socket_permissions_stat_failure_after_exists(self):
|
||||
ex = self.make_exec()
|
||||
|
||||
@@ -103,6 +103,50 @@ class ApiClientCover100Test(unittest.TestCase):
|
||||
self.assertIn(b'"name": "vm1"', fake.sent)
|
||||
self.assertTrue(fake.closed)
|
||||
|
||||
def test_204_without_content_length_is_success(self):
|
||||
# 真机实测(2026-11-07,CHV 54.0.0):vm.create 成功回
|
||||
# 204 No Content,且不带 Content-Length、无体——严格
|
||||
# 校验会把合法成功当失败拒掉。204 按 HTTP 语义体长为
|
||||
# 0,不读体(keep-alive 下读会等到超时;假 socket 的
|
||||
# 剩余片为空,读体即报截断,本测试能抓住误读)。
|
||||
fake = _FakeApiSocket([
|
||||
b"HTTP/1.1 204 No Content\r\nServer: Cloud Hypervisor API"
|
||||
b"\r\nConnection: keep-alive\r\n\r\n",
|
||||
])
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
sock_path = os.path.join(tmp, "chv.sock")
|
||||
with open(sock_path, "w") as fh:
|
||||
fh.write("x")
|
||||
with mock.patch.object(api_client.socket, "socket",
|
||||
lambda *a, **k: fake):
|
||||
status, body = api_client._unix_socket_api(
|
||||
sock_path, "PUT", "/vm.create",
|
||||
payload={"cpus": {"boot_vcpus": 2}})
|
||||
self.assertEqual(status, 204)
|
||||
self.assertEqual(body, {})
|
||||
self.assertTrue(fake.closed)
|
||||
|
||||
def test_200_without_content_length_still_rejected(self):
|
||||
# 204 例外不得放宽其它状态:200 无长度仍拒绝。
|
||||
fake = _FakeApiSocket([
|
||||
b"HTTP/1.1 200 OK\r\nContent-Type: application/json"
|
||||
b"\r\n\r\n{}",
|
||||
])
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
sock_path = os.path.join(tmp, "chv.sock")
|
||||
with open(sock_path, "w") as fh:
|
||||
fh.write("x")
|
||||
with mock.patch.object(api_client.socket, "socket",
|
||||
lambda *a, **k: fake):
|
||||
with self.assertRaises(RuntimeError):
|
||||
api_client._unix_socket_api(
|
||||
sock_path, "GET", "/vmm.ping")
|
||||
|
||||
def test_parse_status(self):
|
||||
self.assertEqual(api_client._parse_status("HTTP/1.1 204 foo"), 204)
|
||||
self.assertEqual(api_client._parse_status("HTTP/1.1 OK"), 0)
|
||||
self.assertEqual(api_client._parse_status(""), 0)
|
||||
|
||||
def test_parse_status_non_numeric_is_zero(self):
|
||||
status, _body = api_client._parse_response(
|
||||
"HTTP/1.1 OK\r\n\r\n", b"{}")
|
||||
|
||||
@@ -158,6 +158,9 @@ class SeedDiskTest(ChvFixCase):
|
||||
self.assertEqual(len(cfg["disks"]), 2)
|
||||
self.assertEqual(cfg["disks"][1]["path"], seed)
|
||||
self.assertTrue(cfg["disks"][1]["readonly"])
|
||||
# CHV 54 要求磁盘显式 image_type(缺了 vm.boot 400)。
|
||||
self.assertEqual(cfg["disks"][0]["image_type"], "Raw")
|
||||
self.assertEqual(cfg["disks"][1]["image_type"], "Raw")
|
||||
cfg2 = ex.build_vm_config(disk)
|
||||
self.assertEqual(len(cfg2["disks"]), 1)
|
||||
|
||||
|
||||
@@ -15,6 +15,8 @@ import unittest
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
|
||||
import yaml
|
||||
|
||||
try:
|
||||
from cloud.chv.hostkeys import (
|
||||
build_seed_user_data,
|
||||
@@ -316,6 +318,17 @@ class HostKeyProvisionTest(ChvGapCase):
|
||||
self.assertFalse(os.path.exists(r["paths"]["private"]))
|
||||
self.assertTrue(os.path.exists(r["known_hosts"]))
|
||||
|
||||
def test_known_hosts_line_port_22_is_bare(self):
|
||||
# 22 端口写裸 host 行:OpenSSH 默认端口只查裸名,
|
||||
# '[host]:22' 永远匹配不上(2026-11-07 WSL 真机)。
|
||||
self.assertEqual(
|
||||
known_hosts_line(22, "ssh-ed25519 AAAAx comment",
|
||||
host="192.168.10.10"),
|
||||
"192.168.10.10 ssh-ed25519 AAAAx")
|
||||
self.assertEqual(
|
||||
known_hosts_line(2222, "ssh-ed25519 AAAAx comment"),
|
||||
"[127.0.0.1]:2222 ssh-ed25519 AAAAx")
|
||||
|
||||
def test_known_hosts_line_rejects_bad_key(self):
|
||||
with self.assertRaises(ValueError):
|
||||
known_hosts_line(2222, "not-a-key")
|
||||
@@ -330,6 +343,125 @@ class HostKeyProvisionTest(ChvGapCase):
|
||||
with self.assertRaises(ValueError):
|
||||
build_seed_user_data("", "ssh-ed25519 AAAAx")
|
||||
|
||||
def test_seed_user_data_has_no_runcmd(self):
|
||||
# 根因回归(sync-test 残留,2026-11-06 实测):seed
|
||||
# 的 user-data 里绝不能带 runcmd——cloud-init 不跨配
|
||||
# 置类型合并,user-data 的 runcmd 会整体替换镜像
|
||||
# /etc/cloud/cloud.cfg.d/ 里的 runcmd,旧版 seed 的
|
||||
# 两行就这样顶掉了 99-chv-sync.cfg 的 chv-sync boot
|
||||
# 钩子,首轮启动 clone 静默不跑。sshd 重启已搬到
|
||||
# 镜像自己的 runcmd(build.sh 写 99-chv-sync.cfg)。
|
||||
text = build_seed_user_data("PRIV\n", "ssh-ed25519 AAAAx\n")
|
||||
parsed = yaml.safe_load(text)
|
||||
self.assertNotIn("runcmd", parsed)
|
||||
self.assertNotIn("runcmd:", text)
|
||||
self.assertNotIn("systemctl restart", text)
|
||||
self.assertNotIn("merge_how", parsed)
|
||||
# write_files 两条 key 条目仍在,chmod 靠其 0600 保证。
|
||||
self.assertEqual(len(parsed["write_files"]), 2)
|
||||
self.assertEqual(
|
||||
parsed["write_files"][0]["permissions"], "0600")
|
||||
|
||||
def test_write_seed_with_secret_has_no_runcmd(self):
|
||||
# 带 secret_env 时 guest-env 条目仍须并进同一个
|
||||
# write_files 列表(单 write_files 键),且 user-data
|
||||
# 里仍不能出现 runcmd。feat-chv-muse-p3:GITEA_TOKEN
|
||||
# 不再走 secret_env,改走 authd_env(独立的 authd-seed
|
||||
# 条目,0600);guest-env 条目里只剩非 authd 密钥。
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
seed_dir = os.path.join(tmp, "seed")
|
||||
write_seed(seed_dir, "vm1", "PRIV", "ssh-ed25519 AAAAx",
|
||||
secret_env={"SUB2API_API_KEY": "gw-key"},
|
||||
authd_env={"GITEA_TOKEN": "s3cr3t v"})
|
||||
with open(os.path.join(seed_dir, "user-data")) as fh:
|
||||
content = fh.read()
|
||||
self.assertEqual(content.count("write_files:"), 1)
|
||||
parsed = yaml.safe_load(content)
|
||||
self.assertNotIn("runcmd", parsed)
|
||||
self.assertEqual(len(parsed["write_files"]), 4)
|
||||
paths = [e["path"] for e in parsed["write_files"]]
|
||||
self.assertIn("/etc/chv/guest-env", paths)
|
||||
self.assertIn("/etc/chv/authd-seed", paths)
|
||||
# 密钥值经 shlex.quote 引用后才进文件(值本身在条目内
|
||||
# 是设计内的,见 hostkeys.write_seed);GITEA_TOKEN
|
||||
# 只在 authd-seed 里,不在 guest-env 里。
|
||||
entries = {e["path"]: e for e in parsed["write_files"]}
|
||||
authd_text = entries["/etc/chv/authd-seed"]["content"]
|
||||
self.assertIn("GITEA_TOKEN=", authd_text)
|
||||
self.assertIn("s3cr3t v", authd_text)
|
||||
self.assertEqual(
|
||||
entries["/etc/chv/authd-seed"]["permissions"], "0600")
|
||||
self.assertNotIn(
|
||||
"GITEA_TOKEN", entries["/etc/chv/guest-env"]["content"])
|
||||
|
||||
def test_write_seed_uses_atomic_write(self):
|
||||
# feat-chv-cover100:seed 暂存文件(user-data 含 host
|
||||
# 私钥明文)此前直接 open("w") 再 chmod 0600——崩溃
|
||||
# 半截文件本身到不了 ISO(唯一消费者 genisoimage 只
|
||||
# 在 write_seed 返回后跑,失败重试会整文件重写),
|
||||
# 但 chmod 之前文件以 umask 权限(常为 0644)可读,
|
||||
# 同 uid 进程能在该窗口读到私钥;现走 atomic_file
|
||||
#(mkstemp 0600 → 写 → fsync → rename),文件从
|
||||
# 创建起就是 0600,且写失败不留半截文件。
|
||||
import stat as _stat
|
||||
from unittest import mock
|
||||
|
||||
import cloud.chv.hostkeys as hostkeys_mod
|
||||
from cloud.chv.atomic_file import atomic_write_text as _real
|
||||
calls = []
|
||||
|
||||
def _spy(path, text, mode=0o600):
|
||||
calls.append(path)
|
||||
return _real(path, text, mode=mode)
|
||||
|
||||
with (tempfile.TemporaryDirectory() as tmp,
|
||||
mock.patch.object(hostkeys_mod, "atomic_write_text",
|
||||
side_effect=_spy)):
|
||||
seed_dir = os.path.join(tmp, "seed")
|
||||
write_seed(seed_dir, "vm1", "PRIV",
|
||||
"ssh-ed25519 AAAAx",
|
||||
network_config="version: 2\n")
|
||||
self.assertEqual(
|
||||
sorted(os.path.basename(p) for p in calls),
|
||||
["meta-data", "network-config", "user-data"])
|
||||
for name in ("meta-data", "user-data", "network-config"):
|
||||
mode = _stat.S_IMODE(os.stat(
|
||||
os.path.join(seed_dir, name)).st_mode)
|
||||
self.assertEqual(mode, 0o600, name)
|
||||
leftovers = [n for n in os.listdir(seed_dir)
|
||||
if n.startswith(".tmp-")]
|
||||
self.assertEqual(leftovers, [])
|
||||
|
||||
def test_write_seed_failure_leaves_no_partial_file(self):
|
||||
# feat-chv-cover100:原子写的回归——写 user-data 中途
|
||||
# 失败时不留半截 user-data、不留 tmp 残留;旧的直接
|
||||
# 写会在崩溃后留半截文件(虽到不了 ISO,但含私钥明
|
||||
# 文的半截文件滞留 seed 目录本身即泄漏面)。
|
||||
from unittest import mock
|
||||
|
||||
import cloud.chv.hostkeys as hostkeys_mod
|
||||
real = hostkeys_mod.atomic_write_text
|
||||
|
||||
def _fail_on_user_data(path, text, mode=0o600):
|
||||
if path.endswith("user-data"):
|
||||
raise OSError("disk boom")
|
||||
return real(path, text, mode=mode)
|
||||
|
||||
with (tempfile.TemporaryDirectory() as tmp,
|
||||
mock.patch.object(hostkeys_mod, "atomic_write_text",
|
||||
side_effect=_fail_on_user_data)):
|
||||
seed_dir = os.path.join(tmp, "seed")
|
||||
os.makedirs(seed_dir)
|
||||
|
||||
with self.assertRaises(OSError):
|
||||
write_seed(seed_dir, "vm1", "PRIV",
|
||||
"ssh-ed25519 AAAAx")
|
||||
self.assertFalse(os.path.exists(
|
||||
os.path.join(seed_dir, "user-data")))
|
||||
leftovers = [n for n in os.listdir(seed_dir)
|
||||
if n.startswith(".tmp-")]
|
||||
self.assertEqual(leftovers, [])
|
||||
|
||||
def test_keyscan_is_weak_fallback_only(self):
|
||||
# provisioned 模式下 wake 前必须预置;keyscan 不应被
|
||||
# 当成生产路径:配置值本身可解析,且文档/代码里标弱。
|
||||
@@ -376,12 +508,31 @@ class BuildScriptTest(unittest.TestCase):
|
||||
for name in ("codex", "codex-code-mode-host"):
|
||||
p = os.path.join(self.bin_dir, name)
|
||||
with open(p, "w") as fh:
|
||||
fh.write("#!/bin/sh\n")
|
||||
# B-09:build.sh 会核对 codex --version 含 CODEX_VERSION,
|
||||
# 假二进制须报出真实版本号。
|
||||
fh.write("#!/bin/sh\necho \"codex-cli 0.164.0-alpha.9\"\n")
|
||||
os.chmod(p, 0o755)
|
||||
self.repo = os.path.join(self.dir, "repo")
|
||||
os.makedirs(os.path.join(self.repo, "cloud"))
|
||||
with open(os.path.join(self.repo, "host.py"), "w") as fh:
|
||||
fh.write("# host\n")
|
||||
# B-09:REPO_COMMIT 须等于仓库 HEAD 短 hash 且工作树干净,
|
||||
# 假仓库须是真 git 仓库。
|
||||
subprocess.run(["git", "init", "-q", self.repo],
|
||||
check=True, capture_output=True)
|
||||
subprocess.run(["git", "-C", self.repo, "config",
|
||||
"user.email", "t@t"], check=True,
|
||||
capture_output=True)
|
||||
subprocess.run(["git", "-C", self.repo, "config",
|
||||
"user.name", "t"], check=True,
|
||||
capture_output=True)
|
||||
subprocess.run(["git", "-C", self.repo, "add", "-A"],
|
||||
check=True, capture_output=True)
|
||||
subprocess.run(["git", "-C", self.repo, "commit", "-qm",
|
||||
"init"], check=True, capture_output=True)
|
||||
self.repo_commit = subprocess.run(
|
||||
["git", "-C", self.repo, "rev-parse", "--short", "HEAD"],
|
||||
check=True, capture_output=True, text=True).stdout.strip()
|
||||
self.script = str(Path(__file__).resolve().parents[2]
|
||||
/ "chv-guest" / "build.sh")
|
||||
|
||||
@@ -393,7 +544,7 @@ class BuildScriptTest(unittest.TestCase):
|
||||
env.update({"CODEX_BIN_DIR": self.bin_dir,
|
||||
"UBUNTU_VERSION": "24.04",
|
||||
"CODEX_VERSION": "0.164.0-alpha.9",
|
||||
"REPO_COMMIT": "test123",
|
||||
"REPO_COMMIT": self.repo_commit,
|
||||
# feat-chv-secfix(M4):SHA256 必填,
|
||||
# 干跑只验“已设”,用 FAKE 占位。
|
||||
"UBUNTU_IMAGE_SHA256": "0" * 64,
|
||||
@@ -411,9 +562,29 @@ class BuildScriptTest(unittest.TestCase):
|
||||
self.assertIn("virt-customize", r.stdout)
|
||||
self.assertIn("virt-get-kernel", r.stdout)
|
||||
self.assertIn("BUILDINFO", r.stdout)
|
||||
self.assertIn("chv-guest-24.04-0.164.0-alpha.9-test123", r.stdout)
|
||||
self.assertIn("chv-guest-24.04-0.164.0-alpha.9-%s"
|
||||
% self.repo_commit, r.stdout)
|
||||
self.assertFalse(os.path.exists(os.path.join(out_dir, "disk.raw")))
|
||||
|
||||
def test_hostkey_dropin_generates_missing_keys_at_first_boot(self):
|
||||
# B-23:Minimal 底座的 cloud.cfg 不含 ssh 模块,build.sh
|
||||
# 曾写 ssh_genkeytypes: [] 彻底关掉首启 host key 生成,
|
||||
# 客机无 key、sshd -t 失败、ssh.service 连败 5 次,SSH
|
||||
# 全程不通。修正:首启 bootcmd 只在缺 key 时生成,绝不
|
||||
# 重生成已预置的 per-VM key;写盘的命令里 [] 绝不再出现
|
||||
#(注释里提历史可以,功能行里不行)。
|
||||
with open(self.script) as fh:
|
||||
text = fh.read()
|
||||
hostkey_lines = [ln for ln in text.splitlines()
|
||||
if "99-chv-hostkey.cfg" in ln
|
||||
and "printf" in ln]
|
||||
self.assertEqual(len(hostkey_lines), 1)
|
||||
cmd = hostkey_lines[0]
|
||||
self.assertIn("ssh_deletekeys: false", cmd)
|
||||
self.assertIn("[ -f /etc/ssh/ssh_host_ed25519_key ] "
|
||||
"|| ssh-keygen -A", cmd)
|
||||
self.assertNotIn("ssh_genkeytypes", cmd)
|
||||
|
||||
def test_full_mode_without_kvm_fails_fast(self):
|
||||
# 本 VM 容器内无 /dev/kvm,全量模式必须明确失败,
|
||||
# 绝不声称已建成镜像。
|
||||
@@ -427,6 +598,166 @@ class BuildScriptTest(unittest.TestCase):
|
||||
self.assertNotEqual(r.returncode, 0)
|
||||
|
||||
|
||||
# -- (5b) chv-sync 双向同步 ------------------------------------------------
|
||||
# 轻量口径 + Gitea 集成(2026-11-06):本地 bare 仓当远端替身,
|
||||
# 不碰真实 Gitea、不需 token,测同步机制本身(双向通 + 分叉/
|
||||
# 脏树时失败关闭)。boot 的 clone 一步需 root + guest-env,
|
||||
# 不在单元测试里跑,只测 boot 无 guest-env 时静默成功与
|
||||
# guest-env 缺字段时明确失败;真机验收见构建记录。
|
||||
class ChvSyncTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.dir = self.tmp.name
|
||||
self.script = str(Path(__file__).resolve().parents[2]
|
||||
/ "chv-guest" / "chv-sync.sh")
|
||||
self.ws = os.path.join(self.dir, "workspace")
|
||||
os.makedirs(self.ws)
|
||||
# 远端替身:bare 仓 + 一个已推入的初始提交。
|
||||
self.remote = os.path.join(self.dir, "remote.git")
|
||||
subprocess.run(["git", "init", "-q", "--bare", self.remote],
|
||||
check=True, capture_output=True)
|
||||
seed = os.path.join(self.dir, "seed-clone")
|
||||
subprocess.run(["git", "clone", "-q", self.remote, seed],
|
||||
check=True, capture_output=True)
|
||||
self._git(seed, "config", "user.email", "t@t")
|
||||
self._git(seed, "config", "user.name", "t")
|
||||
with open(os.path.join(seed, "a.txt"), "w") as fh:
|
||||
fh.write("v1\n")
|
||||
self._git(seed, "add", "-A")
|
||||
self._git(seed, "commit", "-qm", "v1")
|
||||
self._git(seed, "push", "-q", "origin", "HEAD:main")
|
||||
# 本地工作区:模拟 boot 已 clone 的结果。
|
||||
self.repo = os.path.join(self.ws, "remote")
|
||||
subprocess.run(["git", "clone", "-q", "--branch", "main",
|
||||
self.remote, self.repo],
|
||||
check=True, capture_output=True)
|
||||
self._git(self.repo, "config", "user.email", "t@t")
|
||||
self._git(self.repo, "config", "user.name", "t")
|
||||
|
||||
def tearDown(self):
|
||||
self.tmp.cleanup()
|
||||
|
||||
def _git(self, cwd, *args):
|
||||
return subprocess.run(["git", "-C", cwd] + list(args),
|
||||
check=True, capture_output=True, text=True)
|
||||
|
||||
def _sync(self, *args, env_extra=None):
|
||||
env = dict(os.environ)
|
||||
env["CHV_WORKSPACE"] = self.ws
|
||||
env["CHV_GUEST_ENV"] = os.path.join(self.dir, "no-guest-env")
|
||||
if env_extra:
|
||||
env.update(env_extra)
|
||||
return subprocess.run(["bash", self.script] + list(args),
|
||||
capture_output=True, text=True, env=env)
|
||||
|
||||
def _head(self, cwd):
|
||||
return self._git(cwd, "rev-parse", "HEAD").stdout.strip()
|
||||
|
||||
def test_push_then_pull_round_trip(self):
|
||||
# 本地改 → push → 远端可见。
|
||||
with open(os.path.join(self.repo, "a.txt"), "a") as fh:
|
||||
fh.write("v2-local\n")
|
||||
self._git(self.repo, "add", "-A")
|
||||
self._git(self.repo, "commit", "-qm", "v2-local")
|
||||
r = self._sync("push", self.repo)
|
||||
self.assertEqual(r.returncode, 0, r.stderr)
|
||||
other = os.path.join(self.dir, "other-clone")
|
||||
subprocess.run(["git", "clone", "-q", "--branch", "main",
|
||||
self.remote, other],
|
||||
check=True, capture_output=True)
|
||||
self.assertEqual(self._head(other), self._head(self.repo))
|
||||
# 远端改 → pull → 本地同步回来。
|
||||
self._git(other, "config", "user.email", "t@t")
|
||||
self._git(other, "config", "user.name", "t")
|
||||
with open(os.path.join(other, "a.txt"), "a") as fh:
|
||||
fh.write("v3-remote\n")
|
||||
self._git(other, "add", "-A")
|
||||
self._git(other, "commit", "-qm", "v3-remote")
|
||||
self._git(other, "push", "-q", "origin", "HEAD:main")
|
||||
r = self._sync("pull", self.repo)
|
||||
self.assertEqual(r.returncode, 0, r.stderr)
|
||||
with open(os.path.join(self.repo, "a.txt")) as fh:
|
||||
self.assertIn("v3-remote", fh.read())
|
||||
|
||||
def test_push_new_branch_sets_upstream(self):
|
||||
self._git(self.repo, "checkout", "-qb", "feature")
|
||||
with open(os.path.join(self.repo, "b.txt"), "w") as fh:
|
||||
fh.write("f\n")
|
||||
self._git(self.repo, "add", "-A")
|
||||
self._git(self.repo, "commit", "-qm", "feature")
|
||||
r = self._sync("push", self.repo)
|
||||
self.assertEqual(r.returncode, 0, r.stderr)
|
||||
out = subprocess.run(["git", "--git-dir", self.remote,
|
||||
"branch", "--list", "feature"],
|
||||
check=True, capture_output=True,
|
||||
text=True).stdout
|
||||
self.assertIn("feature", out)
|
||||
|
||||
def test_diverged_pull_fails_closed(self):
|
||||
# 本地与远端同分支各进一步 → pull 必须失败,不自动合并。
|
||||
with open(os.path.join(self.repo, "a.txt"), "a") as fh:
|
||||
fh.write("local-diverge\n")
|
||||
self._git(self.repo, "add", "-A")
|
||||
self._git(self.repo, "commit", "-qm", "local-diverge")
|
||||
other = os.path.join(self.dir, "other2")
|
||||
subprocess.run(["git", "clone", "-q", "--branch", "main",
|
||||
self.remote, other],
|
||||
check=True, capture_output=True)
|
||||
self._git(other, "config", "user.email", "t@t")
|
||||
self._git(other, "config", "user.name", "t")
|
||||
with open(os.path.join(other, "a.txt"), "a") as fh:
|
||||
fh.write("remote-diverge\n")
|
||||
self._git(other, "add", "-A")
|
||||
self._git(other, "commit", "-qm", "remote-diverge")
|
||||
self._git(other, "push", "-q", "origin", "HEAD:main")
|
||||
r = self._sync("pull", self.repo)
|
||||
self.assertNotEqual(r.returncode, 0)
|
||||
with open(os.path.join(self.repo, "a.txt")) as fh:
|
||||
self.assertNotIn("remote-diverge", fh.read())
|
||||
|
||||
def test_dirty_tree_pull_refused(self):
|
||||
with open(os.path.join(self.repo, "a.txt"), "a") as fh:
|
||||
fh.write("uncommitted\n")
|
||||
r = self._sync("pull", self.repo)
|
||||
self.assertNotEqual(r.returncode, 0)
|
||||
|
||||
def test_boot_without_guest_env_is_noop(self):
|
||||
r = self._sync("boot")
|
||||
self.assertEqual(r.returncode, 0, r.stderr)
|
||||
|
||||
def test_boot_with_incomplete_guest_env_fails(self):
|
||||
env_file = os.path.join(self.dir, "guest-env")
|
||||
with open(env_file, "w") as fh:
|
||||
fh.write("GITEA_URL=https://gitea.tose.top\n")
|
||||
r = self._sync("boot", env_extra={"CHV_GUEST_ENV": env_file})
|
||||
self.assertNotEqual(r.returncode, 0)
|
||||
|
||||
def test_boot_without_authd_fails_closed_no_credentials_file(self):
|
||||
# round-11 真机修复(层 5):authd 不在时 boot 直接失败,
|
||||
# 绝不把 token 写进 ~/.git-credentials(旧落盘回落已废止;
|
||||
# 即使 guest-env 里有 token 也不写)。
|
||||
if os.getuid() != 0:
|
||||
self.skipTest("boot 须以 root 运行")
|
||||
env_file = os.path.join(self.dir, "guest-env")
|
||||
with open(env_file, "w") as fh:
|
||||
fh.write("GITEA_URL=https://gitea.tose.top\n"
|
||||
"GITEA_USER=admin\n"
|
||||
"SYNC_REPO=admin/chv-sync-test\n"
|
||||
"GITEA_TOKEN=fake-token-for-test\n")
|
||||
home = os.path.join(self.dir, "home")
|
||||
os.makedirs(home, exist_ok=True)
|
||||
r = self._sync("boot", env_extra={
|
||||
"CHV_GUEST_ENV": env_file,
|
||||
"CHV_USER_HOME": home,
|
||||
"CHV_USER": "root",
|
||||
"CHV_AUTHD_SOCKET": os.path.join(self.dir, "no-authd.sock"),
|
||||
})
|
||||
self.assertNotEqual(r.returncode, 0)
|
||||
self.assertIn("chv-authd 未运行", r.stderr)
|
||||
self.assertFalse(
|
||||
os.path.exists(os.path.join(home, ".git-credentials")))
|
||||
|
||||
|
||||
# -- (6) 中途死亡 → interrupted → 断点续跑闭环 ---------------------------
|
||||
class MidDeathLoopTest(ChvGapCase):
|
||||
def test_vm_death_is_interrupted_then_breakpoint_resumes(self):
|
||||
|
||||
@@ -0,0 +1,721 @@
|
||||
"""CHV guest-env 集成测试:Gitea 同步变量供给链路。
|
||||
|
||||
只用标准库与 mock runner/launcher/api_client,绝不启动真实
|
||||
VM、不联网、不含任何真实 token(测试值全是假值)。
|
||||
|
||||
覆盖:
|
||||
- gitea_sync 纯函数:三源合并优先级、白名单清洗、完整性
|
||||
校验(fail-closed)与空=不同步;
|
||||
- ssh_transport:GITEA_TOKEN 进密钥集(argv 拒绝)、其余
|
||||
5 键进 guest-env 集;
|
||||
- hostkeys.write_seed:guest_env 与 secret_env 合并写同一
|
||||
条目、仍无 runcmd、0600;
|
||||
- 执行器 wake_vm:完整同步变量进 seed(5 键 guest-env +
|
||||
GITEA_TOKEN 走 authd-seed);缺失必填
|
||||
键时起 VMM 前 ChvFailure 且不启动;无同步变量时正常启动
|
||||
且不写 guest-env;seed 投递后 build_ssh_command 才能用
|
||||
GITEA_TOKEN 且 argv 里绝无 token 值;
|
||||
- worker:_gitea_sync_env 取值;_run_attempt 只对 CHV 执行器
|
||||
把同步变量并进 gateway_env。
|
||||
- V-9 移植:guest-env owner 恒 root:root(write_files 先于建用户)
|
||||
+ 客内 runcmd chown(build.sh 位置断言)+ wake 等 guest-env
|
||||
可读再报 ready(超时关 VM 抛 guest_env_not_ready)。
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
|
||||
import yaml
|
||||
|
||||
try:
|
||||
from cloud.chv.gitea_sync import (
|
||||
GITEA_SYNC_KEYS,
|
||||
config_gitea_sync_env,
|
||||
extract_gitea_sync_env,
|
||||
validate_gitea_sync_env,
|
||||
)
|
||||
from cloud.chv.hostkeys import write_seed
|
||||
from cloud.chv.ssh_transport import (
|
||||
_CHV_SECRET_ENV_NAMES,
|
||||
build_ssh_command,
|
||||
remote_command,
|
||||
split_guest_env,
|
||||
split_secret_env,
|
||||
)
|
||||
from cloud.sandbox import ChvFailure, CloudHypervisorExecutor
|
||||
from cloud.worker import Worker
|
||||
except ImportError:
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
from cloud.chv.gitea_sync import (
|
||||
GITEA_SYNC_KEYS,
|
||||
config_gitea_sync_env,
|
||||
extract_gitea_sync_env,
|
||||
validate_gitea_sync_env,
|
||||
)
|
||||
from cloud.chv.hostkeys import write_seed
|
||||
from cloud.chv.ssh_transport import (
|
||||
_CHV_SECRET_ENV_NAMES,
|
||||
build_ssh_command,
|
||||
remote_command,
|
||||
split_guest_env,
|
||||
split_secret_env,
|
||||
)
|
||||
from cloud.sandbox import ChvFailure, CloudHypervisorExecutor
|
||||
from cloud.worker import Worker
|
||||
|
||||
|
||||
FAKE_SYNC = {
|
||||
"GITEA_URL": "https://gitea.tose.top",
|
||||
"GITEA_USER": "admin",
|
||||
"GITEA_TOKEN": "fake-token-123",
|
||||
"SYNC_REPO": "admin/chv-sync-test",
|
||||
"SYNC_BRANCH": "main",
|
||||
"GITEA_EMAIL": "admin@tose.top",
|
||||
}
|
||||
|
||||
|
||||
class FakeResult:
|
||||
def __init__(self, returncode=0, stdout="", stderr=""):
|
||||
self.returncode = returncode
|
||||
self.stdout = stdout
|
||||
self.stderr = stderr
|
||||
|
||||
|
||||
class FakeProc:
|
||||
def __init__(self, alive=True):
|
||||
self._alive = alive
|
||||
|
||||
def poll(self):
|
||||
return None if self._alive else 1
|
||||
|
||||
def terminate(self):
|
||||
self._alive = False
|
||||
|
||||
def wait(self, timeout=None):
|
||||
return 0
|
||||
|
||||
def kill(self):
|
||||
self._alive = False
|
||||
|
||||
|
||||
def _write(path, content=b"x"):
|
||||
with open(path, "wb") as fh:
|
||||
fh.write(content)
|
||||
os.chmod(path, 0o755)
|
||||
return path
|
||||
|
||||
|
||||
def _seed_user_text(state_dir, vm_name):
|
||||
with open(os.path.join(state_dir, vm_name, "seed", "user-data")) as fh:
|
||||
return fh.read()
|
||||
|
||||
|
||||
def _guest_env_lines(user_text):
|
||||
parsed = yaml.safe_load(user_text)
|
||||
entries = parsed["write_files"]
|
||||
guest = [e for e in entries if e["path"] == "/etc/chv/guest-env"]
|
||||
assert len(guest) == 1, entries
|
||||
return guest[0]["content"], guest[0]["permissions"]
|
||||
|
||||
def _authd_seed_lines(user_text):
|
||||
parsed = yaml.safe_load(user_text)
|
||||
entries = parsed["write_files"]
|
||||
authd = [e for e in entries if e["path"] == "/etc/chv/authd-seed"]
|
||||
assert len(authd) == 1, entries
|
||||
return authd[0]["content"], authd[0]["permissions"]
|
||||
|
||||
|
||||
# -- (1) gitea_sync 纯函数 ---------------------------------------------
|
||||
class GiteaSyncPureTest(unittest.TestCase):
|
||||
def test_keys_are_six(self):
|
||||
self.assertEqual(GITEA_SYNC_KEYS, set(FAKE_SYNC))
|
||||
|
||||
def test_config_mapping(self):
|
||||
env = {"CHV_GITEA_URL": "https://gitea.tose.top",
|
||||
"CHV_GITEA_USER": "admin",
|
||||
"CHV_GITEA_TOKEN": "tok",
|
||||
"CHV_SYNC_REPO": "admin/chv-sync-test",
|
||||
"CHV_SYNC_BRANCH": "main",
|
||||
"CHV_GITEA_EMAIL": "admin@tose.top",
|
||||
"CHV_UNRELATED": "x"}
|
||||
self.assertEqual(config_gitea_sync_env(env), {
|
||||
"GITEA_URL": "https://gitea.tose.top",
|
||||
"GITEA_USER": "admin", "GITEA_TOKEN": "tok",
|
||||
"SYNC_REPO": "admin/chv-sync-test",
|
||||
"SYNC_BRANCH": "main", "GITEA_EMAIL": "admin@tose.top"})
|
||||
|
||||
def test_extract_priority_config_lt_task_lt_extra_env(self):
|
||||
task = {"id": "t1", "gitea_sync": {
|
||||
"GITEA_URL": "https://task.example",
|
||||
"GITEA_USER": "task-user", "GITEA_TOKEN": "task-tok",
|
||||
"SYNC_REPO": "task/repo"}}
|
||||
out = extract_gitea_sync_env(
|
||||
task=task,
|
||||
extra_env={"SYNC_REPO": "extra/repo"},
|
||||
config_sync={"GITEA_URL": "https://config.example",
|
||||
"GITEA_USER": "config-user",
|
||||
"GITEA_TOKEN": "config-tok",
|
||||
"SYNC_REPO": "config/repo",
|
||||
"SYNC_BRANCH": "dev"})
|
||||
self.assertEqual(out["GITEA_URL"], "https://task.example")
|
||||
self.assertEqual(out["GITEA_USER"], "task-user")
|
||||
self.assertEqual(out["GITEA_TOKEN"], "task-tok")
|
||||
self.assertEqual(out["SYNC_REPO"], "extra/repo")
|
||||
self.assertEqual(out["SYNC_BRANCH"], "dev")
|
||||
|
||||
def test_extract_drops_unknown_and_empty_and_nonstr(self):
|
||||
task = {"id": "t1", "gitea_sync": {
|
||||
"GITEA_URL": "https://gitea.tose.top",
|
||||
"EVIL": "nope", "SYNC_REPO": " ",
|
||||
"GITEA_TOKEN": 12345}}
|
||||
out = extract_gitea_sync_env(task=task)
|
||||
self.assertEqual(out, {"GITEA_URL": "https://gitea.tose.top"})
|
||||
self.assertNotIn("gitea_sync", out)
|
||||
|
||||
def test_build_chv_config_maps_gitea_env(self):
|
||||
from cloud.chv.config import build_chv_config
|
||||
cfg = build_chv_config({}, {
|
||||
"CHV_GITEA_URL": "https://gitea.tose.top",
|
||||
"CHV_GITEA_USER": "admin",
|
||||
"CHV_GITEA_TOKEN": "tok",
|
||||
"CHV_SYNC_REPO": "admin/chv-sync-test"})
|
||||
self.assertEqual(cfg.gitea_sync_env, {
|
||||
"GITEA_URL": "https://gitea.tose.top",
|
||||
"GITEA_USER": "admin", "GITEA_TOKEN": "tok",
|
||||
"SYNC_REPO": "admin/chv-sync-test"})
|
||||
cfg2 = build_chv_config({}, {})
|
||||
self.assertEqual(cfg2.gitea_sync_env, {})
|
||||
|
||||
def test_validate_empty_passes_partial_fails_with_names_only(self):
|
||||
validate_gitea_sync_env({})
|
||||
validate_gitea_sync_env(None)
|
||||
validate_gitea_sync_env(dict(FAKE_SYNC))
|
||||
with self.assertRaises(ValueError) as ctx:
|
||||
validate_gitea_sync_env({"GITEA_URL": "https://gitea.tose.top",
|
||||
"SYNC_REPO": "admin/chv-sync-test"})
|
||||
msg = str(ctx.exception)
|
||||
self.assertIn("GITEA_TOKEN", msg)
|
||||
self.assertIn("GITEA_USER", msg)
|
||||
self.assertNotIn("gitea.tose.top", msg)
|
||||
self.assertNotIn("chv-sync-test", msg)
|
||||
|
||||
|
||||
# -- (2) ssh_transport -------------------------------------------------
|
||||
class SshTransportSyncTest(unittest.TestCase):
|
||||
def test_gitea_token_is_secret(self):
|
||||
self.assertIn("GITEA_TOKEN", _CHV_SECRET_ENV_NAMES)
|
||||
self.assertIn("SUB2API_API_KEY", _CHV_SECRET_ENV_NAMES)
|
||||
secrets, plain = split_secret_env(
|
||||
{"GITEA_TOKEN": "tok", "GITEA_URL": "https://x",
|
||||
"OTHER": "1"})
|
||||
self.assertEqual(secrets, {"GITEA_TOKEN": "tok"})
|
||||
self.assertEqual(plain, {"GITEA_URL": "https://x", "OTHER": "1"})
|
||||
|
||||
def test_split_guest_env_covers_all_six_keys(self):
|
||||
guest, plain = split_guest_env(dict(FAKE_SYNC, OTHER="1"))
|
||||
self.assertEqual(set(guest), set(FAKE_SYNC))
|
||||
self.assertEqual(guest["GITEA_TOKEN"], "fake-token-123")
|
||||
# 非密钥子集仍含同步普通变量(任务期 SSH 前缀照常带它们)。
|
||||
self.assertIn("GITEA_URL", plain)
|
||||
self.assertNotIn("GITEA_TOKEN", plain)
|
||||
self.assertIn("OTHER", plain)
|
||||
self.assertNotIn("OTHER", guest)
|
||||
|
||||
def test_remote_command_rejects_gitea_token(self):
|
||||
with self.assertRaises(ValueError):
|
||||
remote_command(["true"], extra_env={"GITEA_TOKEN": "tok"})
|
||||
with self.assertRaises(ValueError):
|
||||
build_ssh_command(2222, "/tmp/key", "root", None,
|
||||
"/tmp/known_hosts", ["true"],
|
||||
extra_env={"GITEA_TOKEN": "tok"})
|
||||
|
||||
|
||||
# -- (3) hostkeys.write_seed -------------------------------------------
|
||||
class WriteSeedGuestEnvTest(unittest.TestCase):
|
||||
def test_guest_env_and_secret_merge_into_one_entry_no_runcmd(self):
|
||||
# feat-chv-muse-p3:GITEA_TOKEN 不再进 guest-env,改走
|
||||
# authd_env(独立的 authd-seed 条目,0600);guest-env
|
||||
# 条目里只剩非 authd 密钥与同步普通变量。guest_env 里
|
||||
# 直接给 GITEA_TOKEN 即 ValueError(调用方须先
|
||||
# split_authd_env 拆出来,旧的“同名键 secret 优先”语义
|
||||
# 已由“authd 通道只装 authd 密钥”取代)。
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
seed_dir = os.path.join(tmp, "seed")
|
||||
write_seed(seed_dir, "vm1", "PRIV", "ssh-ed25519 AAAAx",
|
||||
secret_env={"SUB2API_API_KEY": "gw-key"},
|
||||
guest_env={"GITEA_URL": "https://gitea.tose.top",
|
||||
"GITEA_USER": "admin",
|
||||
"SYNC_REPO": "admin/chv-sync-test"},
|
||||
authd_env={"GITEA_TOKEN": "dup-should-lose"})
|
||||
with open(os.path.join(seed_dir, "user-data")) as fh:
|
||||
content = fh.read()
|
||||
self.assertEqual(content.count("write_files:"), 1)
|
||||
parsed = yaml.safe_load(content)
|
||||
self.assertNotIn("runcmd", parsed)
|
||||
self.assertEqual(len(parsed["write_files"]), 4)
|
||||
text, perms = _guest_env_lines(content)
|
||||
self.assertEqual(perms, "0600")
|
||||
self.assertIn("GITEA_URL=https://gitea.tose.top", text)
|
||||
self.assertIn("SUB2API_API_KEY=gw-key", text)
|
||||
self.assertNotIn("GITEA_TOKEN", text)
|
||||
authd_text, authd_perms = _authd_seed_lines(content)
|
||||
self.assertEqual(authd_perms, "0600")
|
||||
self.assertIn("GITEA_TOKEN=dup-should-lose", authd_text)
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
seed_dir = os.path.join(tmp, "seed")
|
||||
with self.assertRaises(ValueError):
|
||||
write_seed(seed_dir, "vm1", "PRIV", "ssh-ed25519 AAAAx",
|
||||
guest_env={"GITEA_TOKEN": "dup-should-lose"})
|
||||
|
||||
def test_guest_env_only_still_no_runcmd(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
seed_dir = os.path.join(tmp, "seed")
|
||||
write_seed(seed_dir, "vm1", "PRIV", "ssh-ed25519 AAAAx",
|
||||
guest_env={"GITEA_URL": "https://gitea.tose.top"})
|
||||
with open(os.path.join(seed_dir, "user-data")) as fh:
|
||||
content = fh.read()
|
||||
parsed = yaml.safe_load(content)
|
||||
self.assertNotIn("runcmd", parsed)
|
||||
self.assertEqual(len(parsed["write_files"]), 3)
|
||||
|
||||
def test_guest_env_owner_always_root(self):
|
||||
# V-9 移植:write_files 在 users-groups 之前跑,owner
|
||||
# 写 chv:chv 时 getpwnam 失败、整个模块失败,文件根本
|
||||
# 不产生;故 owner 恒为 root:root,改归 SSH 用户由客内
|
||||
# final 阶段 99-chv-sync.cfg runcmd chown(本分支的
|
||||
# write_seed 本来就没有 ssh_user 参数,owner 写死
|
||||
# root:root——此测试是防回退断言)。
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
seed_dir = os.path.join(tmp, "seed")
|
||||
write_seed(seed_dir, "vm1", "PRIV", "ssh-ed25519 AAAAx",
|
||||
secret_env={"SUB2API_API_KEY": "s3cr3t"})
|
||||
with open(os.path.join(seed_dir, "user-data")) as fh:
|
||||
parsed = yaml.safe_load(fh.read())
|
||||
guest = [e for e in parsed["write_files"]
|
||||
if e["path"] == "/etc/chv/guest-env"]
|
||||
self.assertEqual(len(guest), 1)
|
||||
self.assertEqual(guest[0]["owner"], "root:root")
|
||||
self.assertEqual(guest[0]["permissions"], "0600")
|
||||
|
||||
|
||||
# -- (3b) build.sh:客内 runcmd chown --------------------------------------
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
|
||||
|
||||
class BuildShGuestEnvChownTest(unittest.TestCase):
|
||||
def test_build_sh_runcmd_chowns_guest_env_before_sync_boot(self):
|
||||
# V-9 移植:guest-env 由 write_files 以 root:root 写
|
||||
# (建用户之前),须在 final 阶段 chv-sync boot 之前
|
||||
# 改归 chv,否则 SSH 派发 source 它会 Permission
|
||||
# denied;无文件时 test -f 护栏使该行空操作,runcmd
|
||||
# 失败不影响启动。
|
||||
text = (ROOT / "chv-guest" / "build.sh").read_text()
|
||||
chown = ("test -f /etc/chv/guest-env && "
|
||||
"chown chv:chv /etc/chv/guest-env || true")
|
||||
self.assertIn(chown, text)
|
||||
# "chv-sync boot" 在文件前部注释里先出现过一次,故从
|
||||
# chown 位置往后找同一 runcmd 行里的 boot。
|
||||
chown_at = text.index(chown)
|
||||
boot_at = text.index("/usr/local/bin/chv-sync boot", chown_at)
|
||||
self.assertLess(chown_at, boot_at)
|
||||
|
||||
|
||||
# -- (4) 执行器 wake_vm -------------------------------------------------
|
||||
class WakeGuestEnvCase(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.dir = self.tmp.name
|
||||
self.binary = _write(os.path.join(self.dir, "cloud-hypervisor"))
|
||||
self.ch_remote = _write(os.path.join(self.dir, "ch-remote"))
|
||||
self.kernel = _write(os.path.join(self.dir, "vmlinux"))
|
||||
self.rootfs = _write(os.path.join(self.dir, "disk.raw"),
|
||||
b"R" * 100)
|
||||
self.ssh_key = _write(os.path.join(self.dir, "id_chv"))
|
||||
self.kvm = _write(os.path.join(self.dir, "kvm"))
|
||||
self.cpuinfo = os.path.join(self.dir, "cpuinfo")
|
||||
with open(self.cpuinfo, "w") as fh:
|
||||
fh.write("flags : fpu vmx sse\n")
|
||||
self.state_dir = os.path.join(self.dir, "state")
|
||||
self.calls = []
|
||||
self.audit = []
|
||||
self.launches = []
|
||||
|
||||
def tearDown(self):
|
||||
self.tmp.cleanup()
|
||||
|
||||
def _runner(self, cmd, timeout=None):
|
||||
self.calls.append(list(cmd))
|
||||
if cmd and cmd[0] == "ssh-keygen":
|
||||
priv = cmd[cmd.index("-f") + 1]
|
||||
with open(priv, "w") as fh:
|
||||
fh.write("-----BEGIN OPENSSH PRIVATE KEY-----\nFAKE\n")
|
||||
os.chmod(priv, 0o600)
|
||||
with open(priv + ".pub", "w") as fh:
|
||||
fh.write("ssh-ed25519 AAAAfake1 vm-key\n")
|
||||
return FakeResult(0, "", "")
|
||||
if cmd and cmd[0] == "genisoimage":
|
||||
out = cmd[cmd.index("-output") + 1]
|
||||
with open(out, "wb") as fh:
|
||||
fh.write(b"ISO")
|
||||
return FakeResult(0, "", "")
|
||||
return FakeResult(0, "", "")
|
||||
|
||||
def _launcher(self, cmd):
|
||||
self.launches.append(list(cmd))
|
||||
return FakeProc(alive=True)
|
||||
|
||||
def make_exec(self, env_extra=None, runner=None, ssh_user=None):
|
||||
env = {"CHV_EGRESS_PROXY": "http://10.0.0.1:8888",
|
||||
"CHV_HOST_KEY_MODE": "provisioned",
|
||||
"CHV_SSH_FORWARD": "1",
|
||||
# feat-chv-muse-p3:wake 前 Sentinel 预审要求
|
||||
# Gitea 主机在白名单内;解析器用假的(测试不联网)。
|
||||
"CHV_SENTINEL_ALLOW_HOSTS": "gitea.tose.top"}
|
||||
if env_extra:
|
||||
env.update(env_extra)
|
||||
ex = CloudHypervisorExecutor(
|
||||
binary=self.binary, ch_remote=self.ch_remote,
|
||||
kernel=self.kernel, rootfs=self.rootfs,
|
||||
ssh_key=self.ssh_key, kvm_path=self.kvm,
|
||||
cpuinfo_path=self.cpuinfo, state_dir=self.state_dir,
|
||||
runner=runner if runner is not None else self._runner,
|
||||
launcher=self._launcher,
|
||||
api_client=lambda s, m, p, payload=None: (200, {}),
|
||||
sleeper=lambda s: None, env=env,
|
||||
ssh_user=ssh_user,
|
||||
sentinel_resolver=lambda host, port: ["93.184.216.34"])
|
||||
ex.audit_callback = lambda e, t, d, k=None: self.audit.append(
|
||||
(e, t.get("id") if isinstance(t, dict) else t, d, k))
|
||||
return ex
|
||||
|
||||
def test_wake_with_task_gitea_sync_writes_all_six_to_seed(self):
|
||||
ex = self.make_exec()
|
||||
task = {"id": "sync1", "gitea_sync": dict(FAKE_SYNC)}
|
||||
out = ex.wake_vm(task)
|
||||
self.assertEqual(out["state"], "running")
|
||||
text = _seed_user_text(self.state_dir, "chv-sync1")
|
||||
self.assertNotIn("runcmd", yaml.safe_load(text))
|
||||
guest_text, perms = _guest_env_lines(text)
|
||||
self.assertEqual(perms, "0600")
|
||||
# feat-chv-muse-p3:六键仍全进 seed,但分两路——
|
||||
# 5 个普通键进 guest-env,GITEA_TOKEN 只进 authd-seed。
|
||||
for key in ("GITEA_URL", "GITEA_USER", "SYNC_REPO",
|
||||
"SYNC_BRANCH", "GITEA_EMAIL"):
|
||||
self.assertIn("%s=%s" % (key, FAKE_SYNC[key]), guest_text)
|
||||
self.assertNotIn("GITEA_TOKEN", guest_text)
|
||||
authd_text, authd_perms = _authd_seed_lines(text)
|
||||
self.assertEqual(authd_perms, "0600")
|
||||
self.assertIn("GITEA_TOKEN=fake-token-123", authd_text)
|
||||
# seed.iso 已制盘且 0600。
|
||||
seed_iso = os.path.join(self.state_dir, "chv-sync1", "seed.iso")
|
||||
self.assertTrue(os.path.exists(seed_iso))
|
||||
self.assertEqual(oct(os.stat(seed_iso).st_mode & 0o777), "0o600")
|
||||
|
||||
def test_wake_with_config_env_sync(self):
|
||||
ex = self.make_exec(env_extra={
|
||||
"CHV_GITEA_URL": "https://gitea.tose.top",
|
||||
"CHV_GITEA_USER": "admin",
|
||||
"CHV_GITEA_TOKEN": "fake-token-123",
|
||||
"CHV_SYNC_REPO": "admin/chv-sync-test",
|
||||
"CHV_SYNC_BRANCH": "main"})
|
||||
self.assertEqual(ex.gitea_sync_env["SYNC_REPO"],
|
||||
"admin/chv-sync-test")
|
||||
out = ex.wake_vm({"id": "sync2"})
|
||||
self.assertEqual(out["state"], "running")
|
||||
guest_text, _ = _guest_env_lines(
|
||||
_seed_user_text(self.state_dir, "chv-sync2"))
|
||||
# feat-chv-muse-p3:GITEA_TOKEN 不再进 guest-env,
|
||||
# 改走 authd-seed;其余同步变量仍在 guest-env 里。
|
||||
self.assertNotIn("GITEA_TOKEN", guest_text)
|
||||
self.assertIn("SYNC_BRANCH=main", guest_text)
|
||||
authd_text, authd_perms = _authd_seed_lines(
|
||||
_seed_user_text(self.state_dir, "chv-sync2"))
|
||||
self.assertEqual(authd_perms, "0600")
|
||||
self.assertIn("GITEA_TOKEN=fake-token-123", authd_text)
|
||||
|
||||
def test_wake_partial_sync_fails_closed_before_launch(self):
|
||||
ex = self.make_exec()
|
||||
task = {"id": "sync3", "gitea_sync": {
|
||||
"GITEA_URL": "https://gitea.tose.top",
|
||||
"SYNC_REPO": "admin/chv-sync-test"}}
|
||||
with self.assertRaises(ChvFailure) as ctx:
|
||||
ex.wake_vm(task)
|
||||
self.assertEqual(ctx.exception.kind, "gitea_sync_incomplete")
|
||||
self.assertEqual(self.launches, [])
|
||||
self.assertFalse(os.path.exists(
|
||||
os.path.join(self.state_dir, "chv-sync3", "seed.iso")))
|
||||
kinds = [a[0] for a in self.audit]
|
||||
self.assertIn("vm_failed", kinds)
|
||||
|
||||
def test_wake_partial_sync_from_config_also_fails_closed(self):
|
||||
ex = self.make_exec(env_extra={
|
||||
"CHV_GITEA_URL": "https://gitea.tose.top"})
|
||||
with self.assertRaises(ChvFailure) as ctx:
|
||||
ex.wake_vm({"id": "sync4"})
|
||||
self.assertEqual(ctx.exception.kind, "gitea_sync_incomplete")
|
||||
self.assertEqual(self.launches, [])
|
||||
|
||||
def test_wake_without_sync_writes_no_guest_env(self):
|
||||
ex = self.make_exec()
|
||||
out = ex.wake_vm({"id": "plain1"})
|
||||
self.assertEqual(out["state"], "running")
|
||||
parsed = yaml.safe_load(
|
||||
_seed_user_text(self.state_dir, "chv-plain1"))
|
||||
self.assertEqual(len(parsed["write_files"]), 2)
|
||||
|
||||
def test_build_ssh_command_after_wake_sources_guest_env(self):
|
||||
ex = self.make_exec()
|
||||
ex.wake_vm({"id": "sync5", "gitea_sync": dict(FAKE_SYNC)})
|
||||
# feat-chv-muse-p3:GITEA_TOKEN 走 authd 投递——任务期
|
||||
# 认“已投递、不报错”,但绝不 source 进 codex 进程环境,
|
||||
# 所以命令里既无 token 值、也不 source 任何客内密钥文件。
|
||||
cmd = ex.build_ssh_command(
|
||||
{"id": "sync5"}, ["true"],
|
||||
extra_env={"GITEA_TOKEN": "fake-token-123"})
|
||||
joined = " ".join(cmd)
|
||||
self.assertNotIn("fake-token-123", joined)
|
||||
self.assertNotIn("/etc/chv/guest-env", joined)
|
||||
self.assertNotIn("/etc/chv/authd-seed", joined)
|
||||
# 非 authd 密钥仍走 guest-env:wake 时投递,任务期 source。
|
||||
ex.wake_vm({"id": "sync5b"},
|
||||
extra_env={"SUB2API_API_KEY": "gw-key"})
|
||||
cmd = ex.build_ssh_command(
|
||||
{"id": "sync5b"}, ["true"],
|
||||
extra_env={"SUB2API_API_KEY": "gw-key"})
|
||||
joined = " ".join(cmd)
|
||||
self.assertNotIn("gw-key", joined)
|
||||
self.assertIn("/etc/chv/guest-env", joined)
|
||||
# 未 wake 投递的 VM 用密钥即拒绝。
|
||||
with self.assertRaises(ChvFailure) as ctx:
|
||||
ex.build_ssh_command(
|
||||
{"id": "other"}, ["true"],
|
||||
extra_env={"GITEA_TOKEN": "fake-token-123"})
|
||||
self.assertEqual(ctx.exception.kind, "secret_env_not_delivered")
|
||||
|
||||
def test_wake_keyscan_mode_with_token_fails_closed(self):
|
||||
ex = self.make_exec(env_extra={"CHV_HOST_KEY_MODE": "keyscan"})
|
||||
with self.assertRaises(ChvFailure):
|
||||
ex.wake_vm({"id": "sync6", "gitea_sync": dict(FAKE_SYNC)})
|
||||
self.assertEqual(self.launches, [])
|
||||
|
||||
def test_wake_with_chv_ssh_user_guest_env_owner_still_root(self):
|
||||
# V-9 移植:owner 恒为 root:root(write_files 先于建用户),
|
||||
# 改归 chv 由客内 99-chv-sync.cfg runcmd chown——即使
|
||||
# SSH 用户就是 chv,seed 里的 owner 也不许写 chv:chv。
|
||||
ex = self.make_exec(ssh_user="chv")
|
||||
out = ex.wake_vm({"id": "envown1"},
|
||||
extra_env={"SUB2API_API_KEY": "s3cr3t"})
|
||||
self.assertEqual(out["state"], "running")
|
||||
parsed = yaml.safe_load(
|
||||
_seed_user_text(self.state_dir, "chv-envown1"))
|
||||
guest = [e for e in parsed["write_files"]
|
||||
if e["path"] == "/etc/chv/guest-env"]
|
||||
self.assertEqual(len(guest), 1)
|
||||
self.assertEqual(guest[0]["owner"], "root:root")
|
||||
|
||||
def _readable_runner(self, answers):
|
||||
"""Fake runner:`test -r /etc/chv/guest-env` 按 answers
|
||||
依次回可读与否(用完后重复最后一个),其余命令走默认
|
||||
runner。answers 是 bool 列表。移植自 V-9(9fb0f4b)。"""
|
||||
state = {"n": 0}
|
||||
|
||||
def run(cmd, timeout=None):
|
||||
if ("test" in cmd
|
||||
and cmd[-1] == "/etc/chv/guest-env"):
|
||||
idx = min(state["n"], len(answers) - 1)
|
||||
state["n"] += 1
|
||||
return FakeResult(0 if answers[idx] else 1, "", "")
|
||||
return self._runner(cmd, timeout)
|
||||
run.readable_calls = lambda: state["n"]
|
||||
return run
|
||||
|
||||
def test_wake_waits_for_guest_env_readable(self):
|
||||
# V-9 移植:runcmd chown 比 SSH 就绪晚约 30 秒,wake 须等
|
||||
# guest-env 对 SSH 用户可读再报 ready,否则首个派发
|
||||
# source 它即 Permission denied。
|
||||
runner = self._readable_runner([False, False, True])
|
||||
ex = self.make_exec(runner=runner, ssh_user="chv")
|
||||
out = ex.wake_vm({"id": "envwait1"},
|
||||
extra_env={"SUB2API_API_KEY": "s3cr3t"})
|
||||
self.assertEqual(out["state"], "running")
|
||||
self.assertEqual(runner.readable_calls(), 3)
|
||||
|
||||
def test_wake_fails_when_guest_env_never_readable(self):
|
||||
# V-9 移植:guest-env 一直对 SSH 用户不可读时 wake 超时即
|
||||
# 关 VM 再抛 guest_env_not_ready,绝不带着读不到的密钥
|
||||
# 跑任务。
|
||||
runner = self._readable_runner([False])
|
||||
ex = self.make_exec(runner=runner, ssh_user="chv")
|
||||
with self.assertRaises(ChvFailure) as ctx:
|
||||
ex.wake_vm({"id": "envwait2"},
|
||||
extra_env={"SUB2API_API_KEY": "s3cr3t"})
|
||||
self.assertEqual(ctx.exception.kind, "guest_env_not_ready")
|
||||
self.assertGreater(runner.readable_calls(), 1)
|
||||
|
||||
|
||||
# -- (4b) seed 附加项(users 块 + network-config)------------------------
|
||||
class SeedExtrasTest(unittest.TestCase):
|
||||
USERS_BLOCK = ("users:\n"
|
||||
" - name: chv\n"
|
||||
" sudo: ALL=(ALL) NOPASSWD:ALL\n"
|
||||
" shell: /bin/bash\n"
|
||||
" ssh_authorized_keys:\n"
|
||||
" - ssh-ed25519 AAAAfake chv-guest\n")
|
||||
NETCFG = ("version: 2\nethernets:\n ens4:\n"
|
||||
" addresses: [192.168.10.10/24]\n")
|
||||
|
||||
def test_write_seed_extras(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
seed_dir = os.path.join(tmp, "seed")
|
||||
write_seed(seed_dir, "vm1", "PRIV", "ssh-ed25519 AAAAx",
|
||||
user_data_extra=self.USERS_BLOCK,
|
||||
network_config=self.NETCFG)
|
||||
with open(os.path.join(seed_dir, "user-data")) as fh:
|
||||
content = fh.read()
|
||||
with open(os.path.join(seed_dir, "network-config")) as fh:
|
||||
net = fh.read()
|
||||
net_mode = oct(os.stat(
|
||||
os.path.join(seed_dir, "network-config")).st_mode
|
||||
& 0o777)
|
||||
parsed = yaml.safe_load(content)
|
||||
self.assertNotIn("runcmd", parsed)
|
||||
self.assertIn("users:", content)
|
||||
self.assertIn("ssh-ed25519 AAAAfake chv-guest", content)
|
||||
self.assertEqual(net, self.NETCFG)
|
||||
self.assertEqual(net_mode, "0o600")
|
||||
|
||||
def test_write_seed_extra_with_runcmd_rejected(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
with self.assertRaises(ValueError):
|
||||
write_seed(os.path.join(tmp, "seed"), "vm1",
|
||||
"PRIV", "ssh-ed25519 AAAAx",
|
||||
user_data_extra="runcmd:\n - echo hi\n")
|
||||
|
||||
def test_config_fragment_maps_seed_files(self):
|
||||
from cloud.chv.config import build_chv_config
|
||||
cfg = build_chv_config({}, {
|
||||
"CHV_SEED_USER_DATA_EXTRA_FILE": "/tmp/users.yaml",
|
||||
"CHV_SEED_NETWORK_CONFIG_FILE": "/tmp/net.yaml"})
|
||||
self.assertEqual(cfg.seed_user_data_extra_file, "/tmp/users.yaml")
|
||||
self.assertEqual(cfg.seed_network_config_file, "/tmp/net.yaml")
|
||||
self.assertIsNone(build_chv_config({}, {}).seed_user_data_extra_file)
|
||||
|
||||
|
||||
class WakeSeedExtrasCase(WakeGuestEnvCase):
|
||||
def test_wake_with_seed_extras(self):
|
||||
users_path = os.path.join(self.dir, "users.yaml")
|
||||
net_path = os.path.join(self.dir, "net.yaml")
|
||||
with open(users_path, "w") as fh:
|
||||
fh.write(SeedExtrasTest.USERS_BLOCK)
|
||||
with open(net_path, "w") as fh:
|
||||
fh.write(SeedExtrasTest.NETCFG)
|
||||
ex = self.make_exec(env_extra={
|
||||
"CHV_SEED_USER_DATA_EXTRA_FILE": users_path,
|
||||
"CHV_SEED_NETWORK_CONFIG_FILE": net_path})
|
||||
out = ex.wake_vm({"id": "extra1"})
|
||||
self.assertEqual(out["state"], "running")
|
||||
seed_dir = os.path.join(self.state_dir, "chv-extra1", "seed")
|
||||
with open(os.path.join(seed_dir, "user-data")) as fh:
|
||||
self.assertIn("ssh-ed25519 AAAAfake chv-guest", fh.read())
|
||||
with open(os.path.join(seed_dir, "network-config")) as fh:
|
||||
self.assertEqual(fh.read(), SeedExtrasTest.NETCFG)
|
||||
|
||||
def test_wake_missing_seed_extra_fails_closed(self):
|
||||
ex = self.make_exec(env_extra={
|
||||
"CHV_SEED_USER_DATA_EXTRA_FILE": "/nonexistent/users.yaml"})
|
||||
with self.assertRaises(ChvFailure) as ctx:
|
||||
ex.wake_vm({"id": "extra2"})
|
||||
self.assertEqual(ctx.exception.kind, "seed_extra_missing")
|
||||
self.assertEqual(self.launches, [])
|
||||
|
||||
|
||||
# -- (5) worker ---------------------------------------------------------
|
||||
class WorkerGiteaSyncTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.db_path = os.path.join(self.tmp.name, "control.db")
|
||||
from cloud.server import init_db
|
||||
init_db(self.db_path)
|
||||
|
||||
def tearDown(self):
|
||||
self.tmp.cleanup()
|
||||
|
||||
def _worker(self, executor, env_extra=None):
|
||||
env = {"XCM_API_KEY": "test-placeholder"}
|
||||
if env_extra:
|
||||
env.update(env_extra)
|
||||
return Worker(self.db_path, "/fake/host.py",
|
||||
executor=executor, env=env)
|
||||
|
||||
def test_gitea_sync_env_from_worker_env_and_task(self):
|
||||
worker = self._worker(None, env_extra={
|
||||
"CHV_GITEA_URL": "https://gitea.tose.top",
|
||||
"CHV_GITEA_USER": "admin",
|
||||
"CHV_GITEA_TOKEN": "env-tok",
|
||||
"CHV_SYNC_REPO": "admin/chv-sync-test"})
|
||||
out = worker._gitea_sync_env({"id": "t1"})
|
||||
self.assertEqual(out["GITEA_TOKEN"], "env-tok")
|
||||
out2 = worker._gitea_sync_env(
|
||||
{"id": "t1", "gitea_sync": {"GITEA_TOKEN": "task-tok"}})
|
||||
self.assertEqual(out2["GITEA_TOKEN"], "task-tok")
|
||||
self.assertEqual(out2["SYNC_REPO"], "admin/chv-sync-test")
|
||||
worker2 = self._worker(None)
|
||||
self.assertEqual(worker2._gitea_sync_env({"id": "t1"}), {})
|
||||
|
||||
def test_run_attempt_merges_sync_env_only_for_chv(self):
|
||||
import tempfile as _tf
|
||||
case_dir = _tf.mkdtemp()
|
||||
binary = _write(os.path.join(case_dir, "cloud-hypervisor"))
|
||||
ch_remote = _write(os.path.join(case_dir, "ch-remote"))
|
||||
kernel = _write(os.path.join(case_dir, "vmlinux"))
|
||||
rootfs = _write(os.path.join(case_dir, "disk.raw"), b"R" * 100)
|
||||
ssh_key = _write(os.path.join(case_dir, "id_chv"))
|
||||
kvm = _write(os.path.join(case_dir, "kvm"))
|
||||
cpuinfo = os.path.join(case_dir, "cpuinfo")
|
||||
with open(cpuinfo, "w") as fh:
|
||||
fh.write("flags : fpu vmx sse\n")
|
||||
ex = CloudHypervisorExecutor(
|
||||
binary=binary, ch_remote=ch_remote, kernel=kernel,
|
||||
rootfs=rootfs, ssh_key=ssh_key, kvm_path=kvm,
|
||||
cpuinfo_path=cpuinfo,
|
||||
state_dir=os.path.join(case_dir, "state"),
|
||||
runner=lambda cmd, timeout=None: FakeResult(0, "", ""),
|
||||
launcher=lambda cmd: FakeProc(alive=True),
|
||||
api_client=lambda s, m, p, payload=None: (200, {}),
|
||||
sleeper=lambda s: None,
|
||||
env={"CHV_EGRESS_PROXY": "http://10.0.0.1:8888",
|
||||
"CHV_HOST_KEY_MODE": "provisioned",
|
||||
"CHV_SSH_FORWARD": "1"})
|
||||
seen = {}
|
||||
orig_dispatch = ex.dispatch_task
|
||||
|
||||
def fake_dispatch(task, extra_env=None, model_provider=None):
|
||||
seen["extra_env"] = dict(extra_env or {})
|
||||
return {"executor": "cloud-hypervisor", "task_id": task["id"],
|
||||
"status": "completed", "stdout": "", "stderr": "",
|
||||
"returncode": 0, "host_status": "completed",
|
||||
"usage": None}
|
||||
ex.dispatch_task = fake_dispatch
|
||||
worker = self._worker(ex, env_extra={
|
||||
"CHV_GITEA_URL": "https://gitea.tose.top",
|
||||
"CHV_GITEA_USER": "admin",
|
||||
"CHV_GITEA_TOKEN": "fake-token-123",
|
||||
"CHV_SYNC_REPO": "admin/chv-sync-test"})
|
||||
worker._ensure_gateway = lambda task: (None, {})
|
||||
task = {"id": "w1", "project_id": "p1", "prompt": "做事"}
|
||||
result, error = worker._run_attempt(task)
|
||||
self.assertIsNone(error)
|
||||
self.assertEqual(result["status"], "completed")
|
||||
self.assertEqual(seen["extra_env"]["GITEA_TOKEN"],
|
||||
"fake-token-123")
|
||||
self.assertEqual(seen["extra_env"]["SYNC_REPO"],
|
||||
"admin/chv-sync-test")
|
||||
worker.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,423 @@
|
||||
"""P1-B(feat-chv-muse-p1):气球/vsock 配置、payload 与 vsock 编解码测试。
|
||||
|
||||
覆盖三件:
|
||||
|
||||
- config:CHV_BALLOON/CHV_VSOCK 开关默认开、只认 0/1,
|
||||
CHV_VSOCK_CID 默认 3 且拒绝保留 CID(0/1/2)与非法值;
|
||||
- executor:build_vm_config 默认带 balloon/vsock 设备,开关
|
||||
关时对应键整个不出现,vsock socket 路径按任务盘派生;
|
||||
- vsock(宿主侧)与客内脚本:JSON 行编解码、监听/超时、
|
||||
客内脚本静默失败(连不上宿主即 exit 0,不影响 sshd/任务)。
|
||||
|
||||
全部用假执行器与本地 Unix socket,绝不连真实网络、不起
|
||||
真实 VM。CHV 54 的 balloon size=0 与 vsock 端到端收发尚未
|
||||
真机验证(本机无 /dev/kvm,第三棒占着 WSL 测试机),见
|
||||
汇报的待验证项——这里只钉住配置层 + payload 形态。
|
||||
"""
|
||||
|
||||
import ast
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
import socket
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
from cloud.chv import vsock as chv_vsock
|
||||
from cloud.chv.config import _balloon_vsock_fragment
|
||||
from cloud.chv.config import _base_fragment
|
||||
from cloud.chv.config import _disk_fragment
|
||||
from cloud.chv.config import _network_fragment
|
||||
from cloud.chv.config import _quota_fragment
|
||||
from cloud.chv.config import _ssh_fragment
|
||||
from cloud.chv.config import _timeout_fragment
|
||||
from cloud.chv.config import build_chv_config
|
||||
from cloud.chv.executor import CloudHypervisorExecutor
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
GUEST_SCRIPT = ROOT / "cloud" / "chv" / "guest" / "vsock_notify.py"
|
||||
|
||||
|
||||
def _load_guest_module():
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
"vsock_notify_guest", str(GUEST_SCRIPT))
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
def _stdlib_only(path):
|
||||
tree = ast.parse(Path(path).read_text())
|
||||
mods = set()
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, ast.Import):
|
||||
mods.update(a.name.split(".")[0] for a in node.names)
|
||||
elif isinstance(node, ast.ImportFrom) and node.module:
|
||||
mods.add(node.module.split(".")[0])
|
||||
return mods <= set(sys.stdlib_module_names)
|
||||
|
||||
|
||||
class BalloonVsockConfigTest(unittest.TestCase):
|
||||
def test_defaults_on(self):
|
||||
cfg = build_chv_config({}, {})
|
||||
self.assertTrue(cfg.balloon)
|
||||
self.assertTrue(cfg.vsock)
|
||||
self.assertEqual(cfg.vsock_cid, 3)
|
||||
|
||||
def test_switches_off(self):
|
||||
cfg = build_chv_config(
|
||||
{}, {"CHV_BALLOON": "0", "CHV_VSOCK": "0"})
|
||||
self.assertFalse(cfg.balloon)
|
||||
self.assertFalse(cfg.vsock)
|
||||
self.assertEqual(cfg.vsock_cid, 3)
|
||||
|
||||
def test_switches_on_explicit(self):
|
||||
cfg = build_chv_config(
|
||||
{}, {"CHV_BALLOON": "1", "CHV_VSOCK": "1"})
|
||||
self.assertTrue(cfg.balloon)
|
||||
self.assertTrue(cfg.vsock)
|
||||
|
||||
def test_illegal_switch_values_raise(self):
|
||||
for env in ({"CHV_BALLOON": "2"}, {"CHV_BALLOON": "yes"},
|
||||
{"CHV_BALLOON": "true"}, {"CHV_VSOCK": "on"},
|
||||
{"CHV_VSOCK": ""}):
|
||||
with self.assertRaises(ValueError):
|
||||
build_chv_config({}, env)
|
||||
|
||||
def test_vsock_cid_override(self):
|
||||
cfg = build_chv_config({}, {"CHV_VSOCK_CID": "7"})
|
||||
self.assertEqual(cfg.vsock_cid, 7)
|
||||
cfg2 = build_chv_config({}, {"CHV_VSOCK_CID": "4294967295"})
|
||||
self.assertEqual(cfg2.vsock_cid, 4294967295)
|
||||
|
||||
def test_vsock_cid_reserved_and_illegal_raise(self):
|
||||
# 0/1/2 是 vsock 保留 CID(Hypervisor/Loopback/Host),
|
||||
# -1 随机 CID 也被 CHV vm.create 400 拒绝。
|
||||
for cid in ("0", "1", "2", "-1", "4294967296", "abc", ""):
|
||||
with self.assertRaises(ValueError, msg=cid):
|
||||
build_chv_config({}, {"CHV_VSOCK_CID": cid})
|
||||
|
||||
def test_fragment_keys_disjoint(self):
|
||||
frag = _balloon_vsock_fragment({})
|
||||
self.assertEqual(set(frag), {"balloon", "vsock", "vsock_cid"})
|
||||
others = [_base_fragment({}, {}), _ssh_fragment({}, {}),
|
||||
_network_fragment({}, {}), _quota_fragment({}),
|
||||
_disk_fragment({}), _timeout_fragment({}, {})]
|
||||
seen = set()
|
||||
for other in others:
|
||||
seen |= set(other)
|
||||
self.assertFalse(seen & set(frag), "新片段键与旧片段冲突")
|
||||
|
||||
|
||||
class _PayloadCase(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.dir = tempfile.mkdtemp(prefix="chv-p1b-")
|
||||
self.state_dir = os.path.join(self.dir, "state")
|
||||
os.makedirs(self.state_dir)
|
||||
|
||||
def tearDown(self):
|
||||
import shutil
|
||||
shutil.rmtree(self.dir, ignore_errors=True)
|
||||
|
||||
def make_exec(self, env_extra=None):
|
||||
env = dict(env_extra or {})
|
||||
return CloudHypervisorExecutor(
|
||||
kernel=os.path.join(self.dir, "vmlinux"),
|
||||
rootfs=os.path.join(self.dir, "root.raw"),
|
||||
ssh_key=os.path.join(self.dir, "id_ed25519"),
|
||||
state_dir=self.state_dir,
|
||||
runner=lambda cmd, timeout=None: None,
|
||||
launcher=lambda cmd: None,
|
||||
api_client=lambda *a: (200, {}),
|
||||
sleeper=lambda s: None, env=env)
|
||||
|
||||
|
||||
class BalloonVsockPayloadTest(_PayloadCase):
|
||||
def test_default_payload_has_balloon_and_vsock(self):
|
||||
ex = self.make_exec()
|
||||
cfg = ex.build_vm_config()
|
||||
# CHV 54 BalloonConfig:size 是目标气球尺寸(字节),
|
||||
# 0 = 启动不预扣,只靠 free_page_reporting 被动回收。
|
||||
self.assertEqual(cfg["balloon"], {
|
||||
"size": 0, "deflate_on_oom": True,
|
||||
"free_page_reporting": True})
|
||||
self.assertEqual(cfg["vsock"], {
|
||||
"cid": 3,
|
||||
"socket": os.path.join(self.state_dir, "vsock.sock")})
|
||||
|
||||
def test_switches_off_drop_keys(self):
|
||||
ex = self.make_exec({"CHV_BALLOON": "0", "CHV_VSOCK": "0"})
|
||||
cfg = ex.build_vm_config()
|
||||
self.assertNotIn("balloon", cfg)
|
||||
self.assertNotIn("vsock", cfg)
|
||||
# 旧键原样保留。
|
||||
self.assertIn("cpus", cfg)
|
||||
self.assertIn("memory", cfg)
|
||||
self.assertIn("payload", cfg)
|
||||
self.assertIn("disks", cfg)
|
||||
self.assertIn("rng", cfg)
|
||||
|
||||
def test_balloon_off_only(self):
|
||||
ex = self.make_exec({"CHV_BALLOON": "0"})
|
||||
cfg = ex.build_vm_config()
|
||||
self.assertNotIn("balloon", cfg)
|
||||
self.assertIn("vsock", cfg)
|
||||
|
||||
def test_vsock_cid_honored(self):
|
||||
ex = self.make_exec({"CHV_VSOCK_CID": "9"})
|
||||
cfg = ex.build_vm_config()
|
||||
self.assertEqual(cfg["vsock"]["cid"], 9)
|
||||
|
||||
def test_vsock_socket_derived_from_task_disk_dir(self):
|
||||
ex = self.make_exec()
|
||||
disk = os.path.join(self.state_dir, "chv-t1", "disk.raw")
|
||||
cfg = ex.build_vm_config(disk)
|
||||
self.assertEqual(
|
||||
cfg["vsock"]["socket"],
|
||||
os.path.join(self.state_dir, "chv-t1", "vsock.sock"))
|
||||
|
||||
def test_vsock_socket_falls_back_when_disk_outside_state(self):
|
||||
# reuse/lab:任务盘即基础镜像(state_dir 外),socket
|
||||
# 落 state_dir/vsock.sock(单 VM 可用,见 helper 注释)。
|
||||
ex = self.make_exec()
|
||||
cfg = ex.build_vm_config()
|
||||
self.assertEqual(
|
||||
cfg["vsock"]["socket"],
|
||||
os.path.join(self.state_dir, "vsock.sock"))
|
||||
|
||||
def test_explicit_vsock_socket_wins(self):
|
||||
ex = self.make_exec()
|
||||
cfg = ex.build_vm_config(
|
||||
vsock_socket="/tmp/per-vm/v.sock")
|
||||
self.assertEqual(cfg["vsock"]["socket"], "/tmp/per-vm/v.sock")
|
||||
|
||||
def test_helper_direct(self):
|
||||
ex = self.make_exec({"CHV_BALLOON": "0"})
|
||||
extra = ex._balloon_vsock_payload(
|
||||
vsock_socket="/tmp/s.sock")
|
||||
self.assertEqual(extra, {"vsock": {"cid": 3,
|
||||
"socket": "/tmp/s.sock"}})
|
||||
|
||||
|
||||
class VsockCodecTest(unittest.TestCase):
|
||||
def test_encode_decode_round_trip(self):
|
||||
raw = chv_vsock.encode_notify("ready", "chv-t1", ts=1234.5)
|
||||
self.assertTrue(raw.endswith(b"\n"))
|
||||
msg = chv_vsock.decode_notify(raw)
|
||||
self.assertEqual(msg, {"type": "ready", "vm": "chv-t1",
|
||||
"ts": 1234.5})
|
||||
|
||||
def test_encode_rejects_bad_type_or_vm(self):
|
||||
with self.assertRaises(ValueError):
|
||||
chv_vsock.encode_notify("bogus", "chv-t1")
|
||||
with self.assertRaises(ValueError):
|
||||
chv_vsock.encode_notify("ready", "")
|
||||
|
||||
def test_decode_rejects_garbage(self):
|
||||
cases = [b"not json\n", b"[1, 2]\n", b"123\n", b"\n",
|
||||
b'{"type": "bogus", "vm": "v", "ts": 1}\n',
|
||||
b'{"type": "ready", "vm": "", "ts": 1}\n',
|
||||
b'{"type": "ready", "vm": 7, "ts": 1}\n',
|
||||
b'{"type": "ready", "vm": "v"}\n',
|
||||
b'{"type": "ready", "vm": "v", "ts": "x"}\n',
|
||||
b'{"type": "ready", "vm": "v", "ts": true}\n',
|
||||
b'\xff\xfe bad utf8\n']
|
||||
for line in cases:
|
||||
self.assertIsNone(chv_vsock.decode_notify(line), line)
|
||||
|
||||
def test_decode_accepts_int_ts(self):
|
||||
msg = chv_vsock.decode_notify(
|
||||
b'{"type": "heartbeat", "vm": "v", "ts": 7}')
|
||||
self.assertEqual(msg["ts"], 7)
|
||||
|
||||
def test_paths(self):
|
||||
self.assertEqual(
|
||||
chv_vsock.vsock_socket_path("/s", "chv-t1"),
|
||||
"/s/chv-t1/vsock.sock")
|
||||
self.assertEqual(
|
||||
chv_vsock.notify_socket_path("/s/chv-t1/vsock.sock"),
|
||||
"/s/chv-t1/vsock.sock_1024")
|
||||
self.assertEqual(chv_vsock.HOST_CID, 2)
|
||||
self.assertEqual(chv_vsock.VSOCK_NOTIFY_PORT, 1024)
|
||||
|
||||
|
||||
def _wait_for_socket(path, timeout=2.0):
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
if os.path.exists(path):
|
||||
return True
|
||||
time.sleep(0.02)
|
||||
return False
|
||||
|
||||
|
||||
class VsockListenTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.dir = tempfile.mkdtemp(prefix="chv-p1b-vsock-")
|
||||
|
||||
def tearDown(self):
|
||||
import shutil
|
||||
shutil.rmtree(self.dir, ignore_errors=True)
|
||||
|
||||
def _client(self, path, lines, delay=0.05):
|
||||
"""后台线程:等 socket 出现,连上发 lines 后关闭。
|
||||
|
||||
connect 带重试:监听方 bind 与 listen 之间、以及删
|
||||
残留文件重 bind 的窗口里,单次 connect 可能被拒;
|
||||
须重试到成功或超时,否则测试会偶发失败。"""
|
||||
def run():
|
||||
if not _wait_for_socket(path):
|
||||
return
|
||||
deadline = time.monotonic() + 5
|
||||
sock = None
|
||||
while time.monotonic() < deadline:
|
||||
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
|
||||
try:
|
||||
sock.connect(path)
|
||||
break
|
||||
except OSError:
|
||||
try:
|
||||
sock.close()
|
||||
except OSError:
|
||||
pass
|
||||
sock = None
|
||||
time.sleep(0.02)
|
||||
if sock is None:
|
||||
return
|
||||
try:
|
||||
time.sleep(delay)
|
||||
for line in lines:
|
||||
sock.sendall(line)
|
||||
except OSError:
|
||||
pass
|
||||
finally:
|
||||
try:
|
||||
sock.close()
|
||||
except OSError:
|
||||
pass
|
||||
thread = threading.Thread(target=run, daemon=True)
|
||||
thread.start()
|
||||
return thread
|
||||
|
||||
def test_listen_returns_ready(self):
|
||||
path = os.path.join(self.dir, "n.sock_1024")
|
||||
thread = self._client(
|
||||
path, [chv_vsock.encode_notify("ready", "chv-t1")])
|
||||
msg = chv_vsock.listen_notify(path, timeout=5)
|
||||
thread.join(timeout=5)
|
||||
self.assertIsNotNone(msg)
|
||||
self.assertEqual(msg["type"], "ready")
|
||||
self.assertEqual(msg["vm"], "chv-t1")
|
||||
self.assertFalse(os.path.exists(path), "监听完须清 socket 文件")
|
||||
|
||||
def test_listen_timeout_returns_none(self):
|
||||
path = os.path.join(self.dir, "t.sock_1024")
|
||||
start = time.monotonic()
|
||||
self.assertIsNone(chv_vsock.listen_notify(path, timeout=0.3))
|
||||
self.assertLess(time.monotonic() - start, 2)
|
||||
self.assertFalse(os.path.exists(path))
|
||||
|
||||
def test_listen_skips_invalid_lines_and_wrong_type(self):
|
||||
path = os.path.join(self.dir, "w.sock_1024")
|
||||
thread = self._client(path, [
|
||||
b"not json\n",
|
||||
chv_vsock.encode_notify("heartbeat", "chv-t1"),
|
||||
chv_vsock.encode_notify("ready", "chv-t1"),
|
||||
])
|
||||
msg = chv_vsock.listen_notify(path, timeout=5,
|
||||
want_type="ready")
|
||||
thread.join(timeout=5)
|
||||
self.assertIsNotNone(msg)
|
||||
self.assertEqual(msg["type"], "ready")
|
||||
|
||||
def test_listen_survives_stale_socket_file(self):
|
||||
path = os.path.join(self.dir, "s.sock_1024")
|
||||
with open(path, "w") as fh:
|
||||
fh.write("stale")
|
||||
thread = self._client(
|
||||
path, [chv_vsock.encode_notify("ready", "chv-t1")])
|
||||
msg = chv_vsock.listen_notify(path, timeout=5)
|
||||
thread.join(timeout=5)
|
||||
self.assertIsNotNone(msg)
|
||||
self.assertEqual(msg["type"], "ready")
|
||||
|
||||
def test_wait_ready_end_to_end(self):
|
||||
vm_dir = os.path.join(self.dir, "chv-t1")
|
||||
os.makedirs(vm_dir)
|
||||
path = chv_vsock.notify_socket_path(
|
||||
chv_vsock.vsock_socket_path(self.dir, "chv-t1"))
|
||||
thread = self._client(
|
||||
path, [chv_vsock.encode_notify("ready", "chv-t1")])
|
||||
msg = chv_vsock.wait_ready(self.dir, "chv-t1", timeout=5)
|
||||
thread.join(timeout=5)
|
||||
self.assertIsNotNone(msg)
|
||||
self.assertEqual(msg["type"], "ready")
|
||||
self.assertEqual(msg["vm"], "chv-t1")
|
||||
|
||||
def test_wait_ready_timeout_returns_none(self):
|
||||
vm_dir = os.path.join(self.dir, "chv-t2")
|
||||
os.makedirs(vm_dir)
|
||||
self.assertIsNone(
|
||||
chv_vsock.wait_ready(self.dir, "chv-t2", timeout=0.3))
|
||||
|
||||
|
||||
class GuestNotifyScriptTest(unittest.TestCase):
|
||||
def test_script_exists_and_stdlib_only(self):
|
||||
self.assertTrue(GUEST_SCRIPT.is_file())
|
||||
self.assertTrue(_stdlib_only(GUEST_SCRIPT))
|
||||
self.assertTrue(
|
||||
_stdlib_only(ROOT / "cloud" / "chv" / "vsock.py"))
|
||||
|
||||
def test_vm_name_sources(self):
|
||||
mod = _load_guest_module()
|
||||
self.assertEqual(mod._vm_name(["prog", "chv-t1"]), "chv-t1")
|
||||
old = os.environ.pop("CHV_VM_NAME", None)
|
||||
try:
|
||||
os.environ["CHV_VM_NAME"] = "chv-env"
|
||||
self.assertEqual(mod._vm_name(["prog"]), "chv-env")
|
||||
finally:
|
||||
if old is None:
|
||||
os.environ.pop("CHV_VM_NAME", None)
|
||||
else:
|
||||
os.environ["CHV_VM_NAME"] = old
|
||||
name = mod._vm_name(["prog"])
|
||||
self.assertIsInstance(name, str)
|
||||
self.assertTrue(name)
|
||||
|
||||
def test_send_writes_json_line(self):
|
||||
mod = _load_guest_module()
|
||||
client, server = socket.socketpair()
|
||||
try:
|
||||
mod._send(client, "heartbeat", "chv-t1")
|
||||
server.settimeout(2)
|
||||
data = server.recv(4096)
|
||||
msg = json.loads(data.decode("utf-8"))
|
||||
self.assertEqual(msg["type"], "heartbeat")
|
||||
self.assertEqual(msg["vm"], "chv-t1")
|
||||
self.assertIsInstance(msg["ts"], float)
|
||||
finally:
|
||||
client.close()
|
||||
server.close()
|
||||
|
||||
def test_main_exits_silently_when_host_unreachable(self):
|
||||
# 本机无 vsock 监听:main 应在重试窗口内静默返回 0,
|
||||
# 绝不抛异常、不影响调用方。
|
||||
mod = _load_guest_module()
|
||||
old = mod.CONNECT_RETRY_SECONDS
|
||||
mod.CONNECT_RETRY_SECONDS = 0
|
||||
try:
|
||||
self.assertEqual(mod.main(["prog", "chv-t1"]), 0)
|
||||
finally:
|
||||
mod.CONNECT_RETRY_SECONDS = old
|
||||
|
||||
def test_ports_agree(self):
|
||||
mod = _load_guest_module()
|
||||
self.assertEqual(mod.NOTIFY_PORT, chv_vsock.VSOCK_NOTIFY_PORT)
|
||||
self.assertEqual(mod.HOST_CID, chv_vsock.HOST_CID)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,730 @@
|
||||
"""CHV 常驻池(feat-chv-muse-p1)测试。
|
||||
|
||||
覆盖 cloud/chv/resident_pool.py、executor 的常驻接线
|
||||
(_run_with_vm 常驻/冷启动两路、finally 语义)与
|
||||
worker_adapter.reconcile_resident_pool facade。全部用假
|
||||
执行器/假 VMM,绝不起真实 VM、不连真实网络。
|
||||
|
||||
断言点(按任务书):
|
||||
- acquire/resume 失败回落冷启动,失败条目丢弃;
|
||||
- release 只 pause 不销毁,pause 失败条目留 running;
|
||||
- registry 损坏 quarantine、单坏条目不连坐;
|
||||
- 重启后条目一律 orphan 不复用,只能 reconcile 清理;
|
||||
- _run_with_vm 常驻命中跑完 pause、冷启动跑完 shutdown,
|
||||
客内抛错时两路 finally 都执行;
|
||||
- 带密钥 env / 未启用 / 建池失败时回落冷启动,任务不丢;
|
||||
- 每用户隔离:release 只动本用户条目;
|
||||
- 空闲回收只动超阈值 paused 条目,阈值 0 即关闭;
|
||||
- WarmPool 实验模块不动(wired_to_dispatch 仍 False)。
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import tempfile
|
||||
import unittest
|
||||
from types import SimpleNamespace
|
||||
from unittest import mock
|
||||
|
||||
from cloud.chv import worker_adapter
|
||||
from cloud.chv.errors import ChvFailure
|
||||
from cloud.chv.executor import CloudHypervisorExecutor
|
||||
from cloud.chv.experimental_pool import WarmPool
|
||||
from cloud.chv.resident_pool import (
|
||||
ResidentPool,
|
||||
reconcile_resident_orphans_for_executor,
|
||||
resident_task_for_user,
|
||||
resident_user_key,
|
||||
)
|
||||
|
||||
|
||||
class _FakeExecutor:
|
||||
"""ResidentPool 用的鸭子型执行器:记录调用,可注入失败。
|
||||
|
||||
wait 失败时模拟真实语义:先自行 shutdown 再抛错
|
||||
(wait_guest_ready 的失败路径就是 stop_vm 后 raise)。
|
||||
"""
|
||||
|
||||
def __init__(self, state_dir, env=None):
|
||||
self.state_dir = state_dir
|
||||
self._env = env or {}
|
||||
self.resumes = []
|
||||
self.pauses = []
|
||||
self.wakes = []
|
||||
self.waits = []
|
||||
self.shutdowns = []
|
||||
self.hibernates = []
|
||||
self.emits = []
|
||||
self.resume_fail = False
|
||||
self.wait_fail = False
|
||||
self.wake_fail = False
|
||||
self.pause_fail = False
|
||||
self.hibernate_fail = False
|
||||
self.shutdown_fail = False
|
||||
|
||||
def vm_name(self, task_or_name):
|
||||
raw = task_or_name.get("id") if isinstance(
|
||||
task_or_name, dict) else str(task_or_name)
|
||||
safe = re.sub(r"[^A-Za-z0-9_.-]", "-", raw)
|
||||
if safe.startswith("chv-"):
|
||||
return safe
|
||||
return "chv-" + safe
|
||||
|
||||
def resume_vm(self, task_or_name):
|
||||
self.resumes.append(self.vm_name(task_or_name))
|
||||
if self.resume_fail:
|
||||
raise RuntimeError("resume boom")
|
||||
return True
|
||||
|
||||
def pause_vm(self, task_or_name):
|
||||
self.pauses.append(self.vm_name(task_or_name))
|
||||
if self.pause_fail:
|
||||
raise RuntimeError("pause boom")
|
||||
return True
|
||||
|
||||
def wake_vm(self, task_or_name):
|
||||
self.wakes.append(self.vm_name(task_or_name))
|
||||
if self.wake_fail:
|
||||
raise RuntimeError("wake boom")
|
||||
return {"vm": self.vm_name(task_or_name), "mode": "cold"}
|
||||
|
||||
def wait_guest_ready(self, task):
|
||||
self.waits.append(self.vm_name(task))
|
||||
if self.wait_fail:
|
||||
self.shutdown_vm(task)
|
||||
raise ChvFailure("guest_not_ready", "ssh 未就绪")
|
||||
return True
|
||||
|
||||
def shutdown_vm(self, task_or_name):
|
||||
self.shutdowns.append(self.vm_name(task_or_name))
|
||||
if self.shutdown_fail:
|
||||
raise RuntimeError("shutdown boom")
|
||||
return True
|
||||
|
||||
def hibernate_vm(self, task_or_name, snapshot_url):
|
||||
self.hibernates.append(
|
||||
(self.vm_name(task_or_name), snapshot_url))
|
||||
if self.hibernate_fail:
|
||||
raise RuntimeError("hibernate boom")
|
||||
return {"vm": self.vm_name(task_or_name),
|
||||
"state": "hibernated", "snapshot": snapshot_url}
|
||||
|
||||
def _emit(self, event, task, detail="", kind=None):
|
||||
self.emits.append(event)
|
||||
|
||||
|
||||
def _write(path, content):
|
||||
with open(path, "w") as fh:
|
||||
fh.write(content)
|
||||
return path
|
||||
|
||||
|
||||
def _registry_path(state_dir):
|
||||
return os.path.join(state_dir, "resident-pool.json")
|
||||
|
||||
|
||||
class ResidentKeyTest(unittest.TestCase):
|
||||
def test_user_key_sanitizes_and_defaults(self):
|
||||
self.assertEqual(resident_user_key(None), "default")
|
||||
self.assertEqual(resident_user_key(""), "default")
|
||||
self.assertEqual(resident_user_key("alice"), "alice")
|
||||
self.assertEqual(resident_user_key("a/b c"), "a-b-c")
|
||||
self.assertEqual(resident_task_for_user("alice"),
|
||||
{"id": "resident-alice"})
|
||||
self.assertEqual(resident_task_for_user(None),
|
||||
{"id": "resident-default"})
|
||||
|
||||
|
||||
class ResidentPoolAcquireTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.fake = _FakeExecutor(self.tmp.name)
|
||||
|
||||
def tearDown(self):
|
||||
self.tmp.cleanup()
|
||||
|
||||
def _provisioned_pool(self, user="alice"):
|
||||
pool = ResidentPool(self.fake)
|
||||
got = pool.provision(user)
|
||||
self.assertEqual(got["mode"], "resident")
|
||||
return pool
|
||||
|
||||
def test_provision_then_acquire_then_release(self):
|
||||
pool = self._provisioned_pool()
|
||||
self.assertEqual(self.fake.wakes, ["chv-resident-alice"])
|
||||
self.assertEqual(self.fake.pauses, ["chv-resident-alice"])
|
||||
self.assertEqual(pool.paused_count(), 1)
|
||||
got = pool.acquire("alice")
|
||||
self.assertEqual(got["mode"], "resident")
|
||||
self.assertEqual(got["vm"], "chv-resident-alice")
|
||||
self.assertEqual(got["task"], {"id": "resident-alice"})
|
||||
self.assertEqual(self.fake.resumes, ["chv-resident-alice"])
|
||||
self.assertEqual(self.fake.waits, ["chv-resident-alice"])
|
||||
self.assertEqual(pool.paused_count(), 0)
|
||||
# running 中的条目不接受第二个 acquire(回落冷启动)。
|
||||
self.assertEqual(pool.acquire("alice")["mode"], "cold")
|
||||
self.assertEqual(len(self.fake.resumes), 1)
|
||||
# release 只 pause,不销毁。
|
||||
self.assertTrue(pool.release("alice"))
|
||||
self.assertEqual(
|
||||
self.fake.pauses,
|
||||
["chv-resident-alice", "chv-resident-alice"])
|
||||
self.assertEqual(self.fake.shutdowns, [])
|
||||
self.assertEqual(pool.paused_count(), 1)
|
||||
# 再取仍是同一台 VM。
|
||||
got = pool.acquire("alice")
|
||||
self.assertEqual(got["vm"], "chv-resident-alice")
|
||||
|
||||
def test_resume_failure_falls_back_to_cold_and_drops_entry(self):
|
||||
pool = self._provisioned_pool()
|
||||
self.fake.resume_fail = True
|
||||
got = pool.acquire("alice")
|
||||
self.assertEqual(got, {"mode": "cold", "vm": None,
|
||||
"task": None})
|
||||
self.assertEqual(pool.paused_count(), 0)
|
||||
with open(_registry_path(self.tmp.name)) as fh:
|
||||
self.assertEqual(json.load(fh), {})
|
||||
|
||||
def test_wait_failure_falls_back_to_cold_and_drops_entry(self):
|
||||
pool = self._provisioned_pool()
|
||||
self.fake.wait_fail = True
|
||||
got = pool.acquire("alice")
|
||||
self.assertEqual(got["mode"], "cold")
|
||||
self.assertEqual(pool.paused_count(), 0)
|
||||
# wait 的失败路径已自行 stop_vm(模拟真实语义)。
|
||||
self.assertEqual(self.fake.shutdowns,
|
||||
["chv-resident-alice"])
|
||||
with open(_registry_path(self.tmp.name)) as fh:
|
||||
self.assertEqual(json.load(fh), {})
|
||||
|
||||
def test_release_without_running_entry_is_false(self):
|
||||
pool = self._provisioned_pool()
|
||||
self.assertFalse(pool.release("alice"))
|
||||
self.assertFalse(pool.release("nobody"))
|
||||
self.assertEqual(self.fake.pauses, ["chv-resident-alice"])
|
||||
|
||||
def test_release_pause_failure_keeps_entry_running(self):
|
||||
pool = self._provisioned_pool()
|
||||
pool.acquire("alice")
|
||||
self.fake.pause_fail = True
|
||||
self.assertFalse(pool.release("alice"))
|
||||
# 不删条目、不销毁:下次 acquire 仍是 running→cold,
|
||||
# VM 本身没被 shutdown,留待 worker 关闭时清理。
|
||||
self.assertEqual(self.fake.shutdowns, [])
|
||||
self.assertEqual(pool.acquire("alice")["mode"], "cold")
|
||||
|
||||
def test_provision_is_idempotent_and_per_user_isolated(self):
|
||||
pool = ResidentPool(self.fake)
|
||||
self.assertEqual(pool.provision("alice")["mode"], "resident")
|
||||
self.assertEqual(pool.provision("bob")["mode"], "resident")
|
||||
self.assertEqual(pool.provision("alice")["mode"], "resident")
|
||||
self.assertEqual(self.fake.wakes.count("chv-resident-alice"), 1)
|
||||
self.assertEqual(self.fake.wakes.count("chv-resident-bob"), 1)
|
||||
self.assertEqual(pool.paused_count(), 2)
|
||||
# release 只动本用户条目。
|
||||
self.assertFalse(pool.release("bob")) # bob 未 running
|
||||
pool.acquire("alice")
|
||||
self.assertTrue(pool.release("alice"))
|
||||
self.assertEqual(pool.paused_count(), 2)
|
||||
got = pool.acquire("bob")
|
||||
self.assertEqual(got["vm"], "chv-resident-bob")
|
||||
self.assertEqual(pool.paused_count(), 1)
|
||||
self.assertEqual(self.fake.resumes, ["chv-resident-alice",
|
||||
"chv-resident-bob"])
|
||||
|
||||
def test_provision_wake_failure_falls_back_and_cleans_up(self):
|
||||
pool = ResidentPool(self.fake)
|
||||
self.fake.wake_fail = True
|
||||
got = pool.provision("alice")
|
||||
self.assertEqual(got, {"mode": "cold", "vm": None,
|
||||
"task": None})
|
||||
# 建池失败尽力 shutdown,不留半醒 VM 与条目。
|
||||
self.assertEqual(self.fake.shutdowns,
|
||||
["chv-resident-alice"])
|
||||
self.assertEqual(pool.paused_count(), 0)
|
||||
with open(_registry_path(self.tmp.name)) as fh:
|
||||
self.assertEqual(json.load(fh), {})
|
||||
|
||||
|
||||
class ResidentPoolRegistryTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.fake = _FakeExecutor(self.tmp.name)
|
||||
|
||||
def tearDown(self):
|
||||
self.tmp.cleanup()
|
||||
|
||||
def test_corrupt_registry_is_quarantined(self):
|
||||
path = _write(_registry_path(self.tmp.name), "{not json")
|
||||
pool = ResidentPool(self.fake)
|
||||
self.assertTrue(pool.registry_corrupt)
|
||||
self.assertEqual(pool.paused_count(), 0)
|
||||
self.assertEqual(pool.orphan_count(), 0)
|
||||
self.assertTrue(os.path.exists(path + ".corrupt"))
|
||||
self.assertFalse(os.path.exists(path))
|
||||
|
||||
def test_top_level_non_dict_is_corrupt(self):
|
||||
path = _write(_registry_path(self.tmp.name), "[1, 2]")
|
||||
pool = ResidentPool(self.fake)
|
||||
self.assertTrue(pool.registry_corrupt)
|
||||
self.assertTrue(os.path.exists(path + ".corrupt"))
|
||||
|
||||
def test_bad_entries_dropped_without_tainting_good_ones(self):
|
||||
_write(_registry_path(self.tmp.name), json.dumps({
|
||||
"good": {"state": "paused", "vm": "chv-resident-good",
|
||||
"user_id": "good"},
|
||||
"no-vm": {"state": "paused"},
|
||||
"not-a-dict": "x",
|
||||
"wrong-state": {"state": "flying",
|
||||
"vm": "chv-resident-x"},
|
||||
}))
|
||||
pool = ResidentPool(self.fake)
|
||||
self.assertEqual(pool.registry_corrupt_entries, 3)
|
||||
self.assertEqual(pool.orphan_count(), 1)
|
||||
self.assertEqual(pool.paused_count(), 0)
|
||||
|
||||
def test_restart_never_reuses_entries(self):
|
||||
pool1 = ResidentPool(self.fake)
|
||||
pool1.provision("alice")
|
||||
pool2 = ResidentPool(self.fake)
|
||||
self.assertEqual(pool2.orphan_count(), 1)
|
||||
self.assertEqual(pool2.paused_count(), 0)
|
||||
# 新池绝不把上一世条目发给任务。
|
||||
self.assertEqual(pool2.acquire("alice")["mode"], "cold")
|
||||
self.assertEqual(self.fake.resumes, [])
|
||||
cleaned = pool2.reconcile_orphans()
|
||||
self.assertEqual(cleaned, ["chv-resident-alice"])
|
||||
self.assertEqual(self.fake.shutdowns,
|
||||
["chv-resident-alice"])
|
||||
with open(_registry_path(self.tmp.name)) as fh:
|
||||
self.assertEqual(json.load(fh), {})
|
||||
|
||||
def test_idle_minutes_env_parsing(self):
|
||||
self.assertEqual(
|
||||
ResidentPool(self.fake).idle_minutes, 0)
|
||||
fake = _FakeExecutor(
|
||||
self.tmp.name, env={"CHV_RESIDENT_IDLE_MINUTES": "45"})
|
||||
self.assertEqual(ResidentPool(fake).idle_minutes, 45)
|
||||
fake = _FakeExecutor(
|
||||
self.tmp.name, env={"CHV_RESIDENT_IDLE_MINUTES": "junk"})
|
||||
self.assertEqual(ResidentPool(fake).idle_minutes, 0)
|
||||
fake = _FakeExecutor(
|
||||
self.tmp.name, env={"CHV_RESIDENT_IDLE_MINUTES": "-5"})
|
||||
self.assertEqual(ResidentPool(fake).idle_minutes, 0)
|
||||
pool = ResidentPool(self.fake, idle_minutes=10)
|
||||
self.assertEqual(pool.idle_minutes, 10)
|
||||
|
||||
|
||||
class ResidentPoolIdleTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.fake = _FakeExecutor(self.tmp.name)
|
||||
|
||||
def tearDown(self):
|
||||
self.tmp.cleanup()
|
||||
|
||||
def test_idle_disabled_by_default(self):
|
||||
pool = ResidentPool(self.fake)
|
||||
pool.provision("alice")
|
||||
import time
|
||||
self.assertEqual(
|
||||
pool.reclaim_idle(now=time.time() + 10 ** 9), [])
|
||||
self.assertEqual(self.fake.hibernates, [])
|
||||
|
||||
def test_overdue_paused_entry_is_hibernated(self):
|
||||
import time
|
||||
pool = ResidentPool(self.fake, idle_minutes=30)
|
||||
pool.provision("alice")
|
||||
pool.provision("bob")
|
||||
now = time.time() + 31 * 60
|
||||
reclaimed = pool.reclaim_idle(now=now)
|
||||
self.assertEqual(reclaimed, ["chv-resident-alice",
|
||||
"chv-resident-bob"])
|
||||
self.assertEqual(len(self.fake.hibernates), 2)
|
||||
for vm, url in self.fake.hibernates:
|
||||
self.assertTrue(url.startswith("file://"))
|
||||
self.assertIn(self.tmp.name, url)
|
||||
self.assertTrue(url.endswith(
|
||||
os.path.join(vm, "resident.snapshot")))
|
||||
self.assertEqual(pool.paused_count(), 0)
|
||||
with open(_registry_path(self.tmp.name)) as fh:
|
||||
self.assertEqual(json.load(fh), {})
|
||||
|
||||
def test_not_yet_idle_entry_is_kept(self):
|
||||
import time
|
||||
pool = ResidentPool(self.fake, idle_minutes=30)
|
||||
pool.provision("alice")
|
||||
self.assertEqual(
|
||||
pool.reclaim_idle(now=time.time() + 10 * 60), [])
|
||||
self.assertEqual(pool.paused_count(), 1)
|
||||
|
||||
def test_hibernate_failure_keeps_entry_for_retry(self):
|
||||
import time
|
||||
pool = ResidentPool(self.fake, idle_minutes=30)
|
||||
pool.provision("alice")
|
||||
self.fake.hibernate_fail = True
|
||||
self.assertEqual(
|
||||
pool.reclaim_idle(now=time.time() + 31 * 60), [])
|
||||
self.assertEqual(pool.paused_count(), 1)
|
||||
|
||||
|
||||
class ResidentFacadeTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.fake = _FakeExecutor(self.tmp.name)
|
||||
|
||||
def tearDown(self):
|
||||
self.tmp.cleanup()
|
||||
|
||||
def test_module_facade_cleans_and_empties_registry(self):
|
||||
_write(_registry_path(self.tmp.name), json.dumps({
|
||||
"a": {"state": "paused", "vm": "chv-resident-a"},
|
||||
"b": {"state": "running", "vm": "chv-resident-b"},
|
||||
"c": {"state": "provisioning", "vm": "chv-resident-c"},
|
||||
"d": {"state": "warm", "vm": "chv-resident-d"},
|
||||
"e": {"state": "paused"},
|
||||
"f": "not-a-dict",
|
||||
}))
|
||||
cleaned = reconcile_resident_orphans_for_executor(self.fake)
|
||||
self.assertEqual(cleaned, ["chv-resident-a", "chv-resident-b",
|
||||
"chv-resident-c"])
|
||||
self.assertEqual(self.fake.shutdowns, cleaned)
|
||||
with open(_registry_path(self.tmp.name)) as fh:
|
||||
self.assertEqual(json.load(fh), {})
|
||||
self.assertIn("resident_pool_reconcile", self.fake.emits)
|
||||
|
||||
def test_module_facade_without_registry_is_empty(self):
|
||||
self.assertEqual(
|
||||
reconcile_resident_orphans_for_executor(self.fake), [])
|
||||
self.assertEqual(self.fake.shutdowns, [])
|
||||
|
||||
def test_module_facade_without_state_dir_is_empty(self):
|
||||
fake = _FakeExecutor(None)
|
||||
fake.state_dir = None
|
||||
self.assertEqual(
|
||||
reconcile_resident_orphans_for_executor(fake), [])
|
||||
|
||||
class WorkerAdapterFacadeTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.fake = _FakeExecutor(self.tmp.name)
|
||||
self.audits = []
|
||||
|
||||
def tearDown(self):
|
||||
self.tmp.cleanup()
|
||||
|
||||
def _audit(self, project_id, event, detail):
|
||||
self.audits.append((project_id, event, detail))
|
||||
|
||||
def test_facade_cleans_resident_registry(self):
|
||||
# worker_adapter 的 isinstance 门要求真执行器:用假
|
||||
# 的 shutdown 替身包一层真执行器实例(只借类型)。
|
||||
ex = _RealExecCase.make_executor(self.tmp.name, self.tmp.name)
|
||||
ex.shutdown_vm = self.fake.shutdown_vm
|
||||
ex._emit = self.fake._emit
|
||||
_write(_registry_path(self.tmp.name), json.dumps({
|
||||
"a": {"state": "paused", "vm": "chv-resident-a"},
|
||||
"b": {"state": "running", "vm": "chv-resident-b"},
|
||||
}))
|
||||
cleaned = worker_adapter.reconcile_resident_pool(
|
||||
ex, self._audit, "startup")
|
||||
self.assertEqual(cleaned, ["chv-resident-a", "chv-resident-b"])
|
||||
self.assertEqual(self.fake.shutdowns, cleaned)
|
||||
self.assertTrue(any(
|
||||
event == "chv_resident_pool_reconcile"
|
||||
for _p, event, _d in self.audits))
|
||||
with open(_registry_path(self.tmp.name)) as fh:
|
||||
self.assertEqual(json.load(fh), {})
|
||||
|
||||
def test_facade_never_constructs_resident_pool(self):
|
||||
ex = _RealExecCase.make_executor(self.tmp.name, self.tmp.name)
|
||||
ex.shutdown_vm = self.fake.shutdown_vm
|
||||
ex._emit = self.fake._emit
|
||||
_write(_registry_path(self.tmp.name), json.dumps({
|
||||
"a": {"state": "paused", "vm": "chv-resident-a"},
|
||||
}))
|
||||
with mock.patch(
|
||||
"cloud.chv.resident_pool.ResidentPool",
|
||||
side_effect=AssertionError("must not construct")):
|
||||
cleaned = worker_adapter.reconcile_resident_pool(
|
||||
ex, self._audit, "shutdown")
|
||||
self.assertEqual(cleaned, ["chv-resident-a"])
|
||||
|
||||
def test_facade_ignores_non_chv_executor(self):
|
||||
cleaned = worker_adapter.reconcile_resident_pool(
|
||||
object(), self._audit, "startup")
|
||||
self.assertEqual(cleaned, [])
|
||||
self.assertEqual(self.audits, [])
|
||||
|
||||
def test_facade_does_not_touch_warm_pool_reconcile(self):
|
||||
# reconcile_warm_pool 仍走 warm-pool.json,与常驻无关。
|
||||
ex = _RealExecCase.make_executor(self.tmp.name, self.tmp.name)
|
||||
ex.shutdown_vm = self.fake.shutdown_vm
|
||||
ex._emit = self.fake._emit
|
||||
warm_path = os.path.join(self.tmp.name, "warm-pool.json")
|
||||
_write(warm_path, json.dumps({
|
||||
"w": {"state": "warm", "vm": "chv-warm-1",
|
||||
"task_id": "w"},
|
||||
}))
|
||||
cleaned = worker_adapter.reconcile_warm_pool(
|
||||
ex, self._audit, "startup")
|
||||
self.assertEqual(cleaned, ["chv-warm-1"])
|
||||
self.assertFalse(os.path.exists(_registry_path(self.tmp.name)))
|
||||
|
||||
|
||||
def _write_file(path, content=b"x"):
|
||||
with open(path, "wb") as fh:
|
||||
fh.write(content)
|
||||
os.chmod(path, 0o755)
|
||||
return path
|
||||
|
||||
|
||||
class _RealExecCase(unittest.TestCase):
|
||||
"""真执行器 + 全方法替身:测 _run_with_vm 的接线本身,
|
||||
不测 VMM。"""
|
||||
|
||||
@staticmethod
|
||||
def make_executor(state_dir, base_dir, env_extra=None):
|
||||
env = {"CHV_EGRESS_PROXY": "http://10.0.0.1:8888",
|
||||
"CHV_HOST_KEY_MODE": "keyscan",
|
||||
"CHV_SSH_FORWARD": "1"}
|
||||
if env_extra:
|
||||
env.update(env_extra)
|
||||
cpuinfo = os.path.join(base_dir, "cpuinfo")
|
||||
with open(cpuinfo, "w") as fh:
|
||||
fh.write("flags : fpu vmx sse\n")
|
||||
ex = CloudHypervisorExecutor(
|
||||
binary=_write_file(os.path.join(base_dir,
|
||||
"cloud-hypervisor")),
|
||||
ch_remote=_write_file(os.path.join(base_dir, "ch-remote")),
|
||||
kernel=_write_file(os.path.join(base_dir, "vmlinux")),
|
||||
rootfs=_write_file(os.path.join(base_dir, "disk.raw"),
|
||||
b"R" * 100),
|
||||
ssh_key=_write_file(os.path.join(base_dir, "id_chv")),
|
||||
kvm_path=_write_file(os.path.join(base_dir, "kvm")),
|
||||
cpuinfo_path=cpuinfo,
|
||||
state_dir=state_dir,
|
||||
sleeper=lambda s: None, env=env)
|
||||
ex.audit_callback = lambda e, t, d, k=None: None
|
||||
return ex
|
||||
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.state_dir = os.path.join(self.tmp.name, "state")
|
||||
self.calls = []
|
||||
|
||||
def tearDown(self):
|
||||
self.tmp.cleanup()
|
||||
|
||||
def _stubbed_exec(self, env_extra=None, wake_fail_resident=False):
|
||||
ex = self.make_executor(self.state_dir, self.tmp.name, env_extra)
|
||||
calls = self.calls
|
||||
|
||||
def wake(task, extra_env=None):
|
||||
calls.append(("wake", ex.vm_name(task)))
|
||||
if wake_fail_resident and ex.vm_name(task).startswith(
|
||||
"chv-resident-"):
|
||||
raise RuntimeError("provision boom")
|
||||
return {"vm": ex.vm_name(task), "mode": "cold"}
|
||||
|
||||
def pause(task_or_name):
|
||||
calls.append(("pause", ex.vm_name(task_or_name)))
|
||||
return True
|
||||
|
||||
def resume(task_or_name):
|
||||
calls.append(("resume", ex.vm_name(task_or_name)))
|
||||
return True
|
||||
|
||||
def wait(task):
|
||||
calls.append(("wait", ex.vm_name(task)))
|
||||
return True
|
||||
|
||||
def shutdown(task_or_name):
|
||||
calls.append(("shutdown", ex.vm_name(task_or_name)))
|
||||
return True
|
||||
|
||||
def run_guest(task, inner_args, extra_env=None, **kwargs):
|
||||
calls.append(("guest", task["id"], list(inner_args)))
|
||||
return {"task_id": task["id"], "status": "completed",
|
||||
"stdout": "ok", "vm": ex.vm_name(task)}
|
||||
|
||||
def run_guest_goal(task, inner_args, extra_env=None, **kwargs):
|
||||
calls.append(("guest-goal", task["id"], list(inner_args)))
|
||||
return {"task_id": task["id"], "status": "completed",
|
||||
"goal_met": True, "vm": ex.vm_name(task)}
|
||||
|
||||
ex.wake_vm = wake
|
||||
ex.pause_vm = pause
|
||||
ex.resume_vm = resume
|
||||
ex.wait_guest_ready = wait
|
||||
ex.shutdown_vm = shutdown
|
||||
ex._run_guest = run_guest
|
||||
ex._run_guest_goal = run_guest_goal
|
||||
return ex
|
||||
|
||||
|
||||
class RunWithVmResidentTest(_RealExecCase):
|
||||
def test_first_task_provisions_and_second_reuses(self):
|
||||
ex = self._stubbed_exec()
|
||||
task = {"id": "task-1", "prompt": "hi"}
|
||||
result = ex._run_with_vm(task, ["dispatch", "--task", "hi"])
|
||||
self.assertEqual(result["status"], "completed")
|
||||
# 常驻伪任务跑命令,但 task_id 改回真实任务。
|
||||
self.assertEqual(result["task_id"], "task-1")
|
||||
self.assertEqual(result["vm"], "chv-resident-default")
|
||||
kinds = [c[0] for c in self.calls]
|
||||
self.assertEqual(kinds, ["wake", "pause", "resume", "wait",
|
||||
"guest", "pause"])
|
||||
self.assertEqual(self.calls[0][1], "chv-resident-default")
|
||||
self.assertEqual(self.calls[4][1], "resident-default")
|
||||
# 第二个任务复用同一台常驻 VM,不再 wake。
|
||||
self.calls.clear()
|
||||
task2 = {"id": "task-2", "prompt": "hi2"}
|
||||
result2 = ex._run_with_vm(task2, ["dispatch", "--task", "hi2"])
|
||||
self.assertEqual(result2["task_id"], "task-2")
|
||||
self.assertEqual(result2["vm"], "chv-resident-default")
|
||||
self.assertEqual([c[0] for c in self.calls],
|
||||
["resume", "wait", "guest", "pause"])
|
||||
# 落盘条目是 paused,可供下次 acquire。
|
||||
with open(_registry_path(self.state_dir)) as fh:
|
||||
data = json.load(fh)
|
||||
self.assertEqual(data["default"]["state"], "paused")
|
||||
self.assertEqual(data["default"]["vm"], "chv-resident-default")
|
||||
|
||||
def test_goal_path_runs_on_resident_and_rewrites_task_id(self):
|
||||
ex = self._stubbed_exec()
|
||||
task = {"id": "goal-1", "prompt": "g"}
|
||||
result = ex._run_with_vm(task, ["goal", "--task", "g"],
|
||||
goal=True)
|
||||
self.assertEqual(result["status"], "completed")
|
||||
self.assertEqual(result["task_id"], "goal-1")
|
||||
self.assertTrue(result["goal_met"])
|
||||
self.assertIn(("guest-goal", "resident-default",
|
||||
["goal", "--task", "g"]), self.calls)
|
||||
|
||||
def test_dispatch_task_flows_through_resident(self):
|
||||
ex = self._stubbed_exec()
|
||||
result = ex.dispatch_task({"id": "d-1", "prompt": "hello"})
|
||||
self.assertEqual(result["status"], "completed")
|
||||
self.assertEqual(result["task_id"], "d-1")
|
||||
self.assertEqual(result["vm"], "chv-resident-default")
|
||||
|
||||
def test_per_user_vms_are_isolated(self):
|
||||
ex = self._stubbed_exec()
|
||||
ex._run_with_vm({"id": "t-a", "user_id": "alice",
|
||||
"prompt": "x"}, ["dispatch"])
|
||||
ex._run_with_vm({"id": "t-b", "user_id": "bob",
|
||||
"prompt": "x"}, ["dispatch"])
|
||||
wakes = [c[1] for c in self.calls if c[0] == "wake"]
|
||||
self.assertEqual(wakes, ["chv-resident-alice",
|
||||
"chv-resident-bob"])
|
||||
with open(_registry_path(self.state_dir)) as fh:
|
||||
data = json.load(fh)
|
||||
self.assertEqual(set(data), {"alice", "bob"})
|
||||
|
||||
def test_disabled_flag_goes_cold(self):
|
||||
ex = self._stubbed_exec(
|
||||
env_extra={"CHV_RESIDENT_ENABLED": "0"})
|
||||
result = ex._run_with_vm({"id": "task-1", "prompt": "hi"},
|
||||
["dispatch"])
|
||||
self.assertEqual(result["status"], "completed")
|
||||
self.assertEqual(result["task_id"], "task-1")
|
||||
kinds = [c[0] for c in self.calls]
|
||||
self.assertEqual(kinds, ["wake", "guest", "shutdown"])
|
||||
self.assertEqual(self.calls[0][1], "chv-task-1")
|
||||
self.assertIsNone(getattr(ex, "_resident_pool", None))
|
||||
self.assertFalse(os.path.exists(
|
||||
_registry_path(self.state_dir)))
|
||||
|
||||
def test_secret_env_goes_cold(self):
|
||||
ex = self._stubbed_exec()
|
||||
result = ex._run_with_vm(
|
||||
{"id": "task-1", "prompt": "hi"}, ["dispatch"],
|
||||
extra_env={"SUB2API_API_KEY": "sekret"})
|
||||
self.assertEqual(result["status"], "completed")
|
||||
kinds = [c[0] for c in self.calls]
|
||||
self.assertEqual(kinds, ["wake", "guest", "shutdown"])
|
||||
self.assertFalse(os.path.exists(
|
||||
_registry_path(self.state_dir)))
|
||||
|
||||
def test_provision_failure_falls_back_to_cold(self):
|
||||
ex = self._stubbed_exec(wake_fail_resident=True)
|
||||
result = ex._run_with_vm({"id": "task-1", "prompt": "hi"},
|
||||
["dispatch"])
|
||||
self.assertEqual(result["status"], "completed")
|
||||
self.assertEqual(result["task_id"], "task-1")
|
||||
kinds = [c[0] for c in self.calls]
|
||||
# 建池 wake 失败(尽力 shutdown)→ 冷启动照常跑任务。
|
||||
self.assertEqual(kinds, ["wake", "shutdown", "wake",
|
||||
"guest", "shutdown"])
|
||||
self.assertEqual(self.calls[1][1], "chv-resident-default")
|
||||
self.assertEqual(self.calls[2][1], "chv-task-1")
|
||||
|
||||
def test_guest_error_still_pauses_resident_in_finally(self):
|
||||
ex = self._stubbed_exec()
|
||||
|
||||
def boom(task, inner_args, extra_env=None, **kwargs):
|
||||
raise RuntimeError("guest boom")
|
||||
|
||||
ex._run_guest = boom
|
||||
with self.assertRaises(RuntimeError):
|
||||
ex._run_with_vm({"id": "task-1", "prompt": "hi"},
|
||||
["dispatch"])
|
||||
kinds = [c[0] for c in self.calls]
|
||||
self.assertEqual(kinds, ["wake", "pause", "resume", "wait",
|
||||
"pause"])
|
||||
self.assertEqual(self.calls[-1][1], "chv-resident-default")
|
||||
|
||||
def test_guest_error_still_shuts_down_cold_in_finally(self):
|
||||
ex = self._stubbed_exec(
|
||||
env_extra={"CHV_RESIDENT_ENABLED": "0"})
|
||||
|
||||
def boom(task, inner_args, extra_env=None, **kwargs):
|
||||
raise RuntimeError("guest boom")
|
||||
|
||||
ex._run_guest = boom
|
||||
with self.assertRaises(RuntimeError):
|
||||
ex._run_with_vm({"id": "task-1", "prompt": "hi"},
|
||||
["dispatch"])
|
||||
self.assertEqual([c[0] for c in self.calls],
|
||||
["wake", "shutdown"])
|
||||
|
||||
def test_wake_failure_propagates_without_shutdown(self):
|
||||
ex = self._stubbed_exec(
|
||||
env_extra={"CHV_RESIDENT_ENABLED": "0"})
|
||||
|
||||
def wake_fail(task, extra_env=None):
|
||||
raise ChvFailure("vm_create_failed", "no vm")
|
||||
|
||||
ex.wake_vm = wake_fail
|
||||
with self.assertRaises(ChvFailure):
|
||||
ex._run_with_vm({"id": "task-1", "prompt": "hi"},
|
||||
["dispatch"])
|
||||
self.assertEqual(self.calls, [])
|
||||
|
||||
|
||||
class WarmPoolUntouchedTest(unittest.TestCase):
|
||||
def test_wired_to_dispatch_still_false(self):
|
||||
self.assertFalse(WarmPool.wired_to_dispatch)
|
||||
|
||||
def test_executor_source_never_imports_experimental(self):
|
||||
import ast
|
||||
from pathlib import Path
|
||||
tree = ast.parse((Path(__file__).resolve().parents[2]
|
||||
/ "cloud" / "chv" / "executor.py").read_text())
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, ast.Import):
|
||||
for alias in node.names:
|
||||
self.assertNotIn(alias.name, (
|
||||
"cloud.chv.pool",
|
||||
"cloud.chv.experimental_pool",
|
||||
"cloud.chv.orphans"))
|
||||
if isinstance(node, ast.ImportFrom):
|
||||
self.assertNotIn(node.module, (
|
||||
"cloud.chv.pool",
|
||||
"cloud.chv.experimental_pool",
|
||||
"cloud.chv.orphans"))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,811 @@
|
||||
"""对齐 Muse P3 测试:authd 替身凭证 + Sentinel 式出口审批。
|
||||
|
||||
只用标准库与 mock,绝不启动真实 VM、不联网、不含真实 token
|
||||
(测试值全是假值)。覆盖:
|
||||
- authd 保险库:白名单入库、作用域绑定、单次释放、替身过期、
|
||||
forget 吊销、repr/审计里绝无 token 值;
|
||||
- 铸 token / 吊销(假 HTTP 客户端);
|
||||
- Sentinel:SSRF 挡截、deny/ask 路径、L7 路径规则、严格能力
|
||||
(绑定/过期/次数)、待审批队列;
|
||||
- seed 拆分:split_authd_env、write_seed 拒收 guest-env 里的
|
||||
GITEA_TOKEN、authd-seed 0600;
|
||||
- wake 预审:allow/ask/deny、无同步跳过;
|
||||
- 客内 authd 守护进程(真 socket + 真 seed):放凭证、拒绝
|
||||
路径/主机/仓库、审计无值;
|
||||
- git credential helper 端到端:真 git credential fill + 假
|
||||
authd socket(守护进程真跑在测试进程内)。
|
||||
"""
|
||||
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
from cloud.chv.authd import (
|
||||
AuthdRefusal,
|
||||
CredentialVault,
|
||||
mint_gitea_token,
|
||||
revoke_gitea_token,
|
||||
surrogate_fingerprint,
|
||||
)
|
||||
from cloud.chv.gitea_sync import split_authd_env
|
||||
from cloud.chv.hostkeys import AUTHD_SEED_PATH, write_seed
|
||||
from cloud.chv.sentinel import (
|
||||
ApprovalQueue,
|
||||
EgressRequest,
|
||||
SentinelPolicy,
|
||||
)
|
||||
from cloud.sandbox import ChvFailure, CloudHypervisorExecutor
|
||||
except ImportError:
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
from cloud.chv.authd import (
|
||||
AuthdRefusal,
|
||||
CredentialVault,
|
||||
mint_gitea_token,
|
||||
revoke_gitea_token,
|
||||
surrogate_fingerprint,
|
||||
)
|
||||
from cloud.chv.gitea_sync import split_authd_env
|
||||
from cloud.chv.hostkeys import AUTHD_SEED_PATH, write_seed
|
||||
from cloud.chv.sentinel import (
|
||||
ApprovalQueue,
|
||||
EgressRequest,
|
||||
SentinelPolicy,
|
||||
)
|
||||
from cloud.sandbox import ChvFailure, CloudHypervisorExecutor
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
TOKEN = "fake-token-123"
|
||||
PUBLIC_IP = "93.184.216.34"
|
||||
FAKE_SYNC = {
|
||||
"GITEA_URL": "https://gitea.tose.top",
|
||||
"GITEA_USER": "admin",
|
||||
"GITEA_TOKEN": TOKEN,
|
||||
"SYNC_REPO": "admin/chv-sync-test",
|
||||
"SYNC_BRANCH": "main",
|
||||
"GITEA_EMAIL": "admin@tose.top",
|
||||
}
|
||||
|
||||
|
||||
def _public_resolver(host, port):
|
||||
return [PUBLIC_IP]
|
||||
|
||||
|
||||
def _load_guest_authd():
|
||||
"""chv-guest/authd.py(守护进程)按文件加载,不走包名。"""
|
||||
path = REPO_ROOT / "chv-guest" / "authd.py"
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
"chv_guest_authd", path)
|
||||
mod = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(mod)
|
||||
return mod
|
||||
|
||||
|
||||
# -- (1) authd 保险库 ----------------------------------------------------
|
||||
class VaultTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.now = [1000.0]
|
||||
self.vault = CredentialVault(clock=lambda: self.now[0])
|
||||
|
||||
def _stored(self):
|
||||
self.vault.store("gitea", "GITEA_TOKEN", TOKEN,
|
||||
host="gitea.tose.top",
|
||||
repo="admin/chv-sync-test")
|
||||
|
||||
def test_store_rejects_unknown_connector_key_empty(self):
|
||||
with self.assertRaises(AuthdRefusal) as ctx:
|
||||
self.vault.store("github", "GITEA_TOKEN", TOKEN)
|
||||
self.assertEqual(ctx.exception.kind, "unknown_connector")
|
||||
with self.assertRaises(AuthdRefusal) as ctx:
|
||||
self.vault.store("gitea", "OTHER_KEY", TOKEN)
|
||||
self.assertEqual(ctx.exception.kind,
|
||||
"credential_not_allowlisted")
|
||||
with self.assertRaises(AuthdRefusal) as ctx:
|
||||
self.vault.store("gitea", "GITEA_TOKEN", " ")
|
||||
self.assertEqual(ctx.exception.kind, "empty_credential")
|
||||
self.assertFalse(self.vault.has("gitea", "GITEA_TOKEN"))
|
||||
|
||||
def test_release_single_use_and_scope_binding(self):
|
||||
self._stored()
|
||||
rel = self.vault.release("gitea", "GITEA_TOKEN",
|
||||
"gitea.tose.top",
|
||||
"admin/chv-sync-test", "push")
|
||||
self.assertEqual(self.vault.consume(rel.release_id), TOKEN)
|
||||
with self.assertRaises(AuthdRefusal) as ctx:
|
||||
self.vault.consume(rel.release_id)
|
||||
self.assertEqual(ctx.exception.kind, "release_unknown")
|
||||
with self.assertRaises(AuthdRefusal) as ctx:
|
||||
self.vault.release("gitea", "GITEA_TOKEN",
|
||||
"other.example",
|
||||
"admin/chv-sync-test", "push")
|
||||
self.assertEqual(ctx.exception.kind, "scope_mismatch")
|
||||
with self.assertRaises(AuthdRefusal) as ctx:
|
||||
self.vault.release("gitea", "GITEA_TOKEN",
|
||||
"gitea.tose.top",
|
||||
"admin/other-repo", "push")
|
||||
self.assertEqual(ctx.exception.kind, "scope_mismatch")
|
||||
with self.assertRaises(AuthdRefusal) as ctx:
|
||||
self.vault.release("gitea", "GITEA_TOKEN",
|
||||
"gitea.tose.top",
|
||||
"admin/chv-sync-test", "delete")
|
||||
self.assertEqual(ctx.exception.kind, "operation_not_allowed")
|
||||
|
||||
def test_release_expires_after_ttl(self):
|
||||
# 替身过期:300 秒 TTL,301 秒后兑现即废。
|
||||
self._stored()
|
||||
rel = self.vault.release("gitea", "GITEA_TOKEN",
|
||||
"gitea.tose.top",
|
||||
"admin/chv-sync-test", "pull")
|
||||
self.now[0] += 301
|
||||
with self.assertRaises(AuthdRefusal) as ctx:
|
||||
self.vault.consume(rel.release_id)
|
||||
self.assertEqual(ctx.exception.kind, "release_expired")
|
||||
|
||||
def test_release_unexpired_still_works(self):
|
||||
self._stored()
|
||||
rel = self.vault.release("gitea", "GITEA_TOKEN",
|
||||
"gitea.tose.top",
|
||||
"admin/chv-sync-test", "clone")
|
||||
self.now[0] += 299
|
||||
self.assertEqual(self.vault.consume(rel.release_id), TOKEN)
|
||||
|
||||
def test_forget_revokes_pending_releases(self):
|
||||
self._stored()
|
||||
rel = self.vault.release("gitea", "GITEA_TOKEN",
|
||||
"gitea.tose.top",
|
||||
"admin/chv-sync-test", "pull")
|
||||
self.assertTrue(self.vault.forget("gitea", "GITEA_TOKEN"))
|
||||
self.assertFalse(self.vault.has("gitea", "GITEA_TOKEN"))
|
||||
self.assertFalse(self.vault.forget("gitea", "GITEA_TOKEN"))
|
||||
with self.assertRaises(AuthdRefusal) as ctx:
|
||||
self.vault.consume(rel.release_id)
|
||||
self.assertEqual(ctx.exception.kind, "release_unknown")
|
||||
with self.assertRaises(AuthdRefusal) as ctx:
|
||||
self.vault.release("gitea", "GITEA_TOKEN",
|
||||
"gitea.tose.top",
|
||||
"admin/chv-sync-test", "pull")
|
||||
self.assertEqual(ctx.exception.kind, "credential_not_stored")
|
||||
|
||||
def test_repr_and_audit_never_contain_value(self):
|
||||
# 真 token 泄露面:repr/审计记录里绝无 token 值。
|
||||
self._stored()
|
||||
rel = self.vault.release("gitea", "GITEA_TOKEN",
|
||||
"gitea.tose.top",
|
||||
"admin/chv-sync-test", "push")
|
||||
for text in (repr(self.vault), repr(rel),
|
||||
json.dumps(self.vault.audit_release(rel)),
|
||||
json.dumps(self.vault.audit_refusal(
|
||||
"scope_mismatch", "gitea", "GITEA_TOKEN"))):
|
||||
self.assertNotIn(TOKEN, text)
|
||||
self.assertNotEqual(surrogate_fingerprint(TOKEN), TOKEN)
|
||||
self.assertEqual(len(surrogate_fingerprint(TOKEN)), 8)
|
||||
self.assertEqual(surrogate_fingerprint(TOKEN),
|
||||
surrogate_fingerprint(TOKEN))
|
||||
|
||||
|
||||
class MintRevokeTest(unittest.TestCase):
|
||||
def test_mint_returns_new_token_and_failure_is_refusal(self):
|
||||
def fake_post(url, headers, payload):
|
||||
self.assertIn("/users/admin/tokens", url)
|
||||
self.assertEqual(payload["scopes"], ["write:repository"])
|
||||
return 201, {"sha1": "new-short-token"}
|
||||
self.assertEqual(
|
||||
mint_gitea_token("https://gitea.tose.top/api/v1", "admin",
|
||||
TOKEN, "chv-test", post=fake_post),
|
||||
"new-short-token")
|
||||
with self.assertRaises(AuthdRefusal) as ctx:
|
||||
mint_gitea_token("https://gitea.tose.top/api/v1", "admin",
|
||||
TOKEN, "chv-test",
|
||||
post=lambda u, h, p: (403, {}))
|
||||
self.assertEqual(ctx.exception.kind, "mint_failed")
|
||||
with self.assertRaises(AuthdRefusal):
|
||||
mint_gitea_token("", "admin", TOKEN, "x")
|
||||
|
||||
def test_revoke_statuses(self):
|
||||
base = ("https://gitea.tose.top/api/v1", "admin", TOKEN)
|
||||
self.assertTrue(revoke_gitea_token(
|
||||
*base, 49, delete=lambda u, h: 204))
|
||||
self.assertTrue(revoke_gitea_token(
|
||||
*base, 49, delete=lambda u, h: 404))
|
||||
self.assertFalse(revoke_gitea_token(
|
||||
*base, 49, delete=lambda u, h: 500))
|
||||
self.assertFalse(revoke_gitea_token("", "admin", TOKEN, 49))
|
||||
|
||||
|
||||
# -- (2) Sentinel 决策 ----------------------------------------------------
|
||||
class SentinelTest(unittest.TestCase):
|
||||
def _req(self, host="gitea.tose.top", method="GET",
|
||||
scheme="https", port=443, path="/api/v1/version"):
|
||||
return EgressRequest("gitea", method, scheme, host, port, path)
|
||||
|
||||
def test_allow_public_ip_allowlisted_host(self):
|
||||
policy = SentinelPolicy(allow_hosts={"gitea.tose.top"})
|
||||
dec = policy.evaluate(self._req(), resolve=_public_resolver)
|
||||
self.assertEqual(dec.verdict, "allow")
|
||||
self.assertIsNotNone(dec.capability)
|
||||
self.assertNotIn(TOKEN, json.dumps(dec.audit(self._req())))
|
||||
|
||||
def test_ssrf_private_ip_denied_even_when_allowlisted(self):
|
||||
# SSRF 挡截:白名单主机解析到私网 IP 仍 deny。
|
||||
policy = SentinelPolicy(allow_hosts={"gitea.tose.top"})
|
||||
for ip in ("10.0.0.5", "192.168.1.1", "127.0.0.1",
|
||||
"169.254.1.2", "100.64.0.9", "::1", "0.0.0.0"):
|
||||
dec = policy.evaluate(
|
||||
self._req(), resolve=lambda h, p, ip=ip: [ip])
|
||||
self.assertEqual(dec.verdict, "deny", ip)
|
||||
self.assertIn("ssrf_private_ip", dec.reasons[0])
|
||||
|
||||
def test_dns_failure_is_deny_not_ask(self):
|
||||
policy = SentinelPolicy(allow_hosts={"gitea.tose.top"})
|
||||
dec = policy.evaluate(
|
||||
self._req(), resolve=lambda h, p: (_ for _ in ()).throw(
|
||||
OSError("boom")))
|
||||
self.assertEqual(dec.verdict, "deny")
|
||||
self.assertEqual(dec.reasons, ["dns_resolution_failed"])
|
||||
dec = policy.evaluate(self._req(), resolve=lambda h, p: [])
|
||||
self.assertEqual(dec.verdict, "deny")
|
||||
|
||||
def test_unknown_host_is_ask_not_allow(self):
|
||||
policy = SentinelPolicy(allow_hosts={"gitea.tose.top"})
|
||||
dec = policy.evaluate(self._req(host="other.example"),
|
||||
resolve=_public_resolver)
|
||||
self.assertEqual(dec.verdict, "ask")
|
||||
self.assertEqual(dec.reasons, ["host_not_allowlisted"])
|
||||
self.assertIsNone(dec.capability)
|
||||
|
||||
def test_deny_paths(self):
|
||||
policy = SentinelPolicy(allow_hosts={"gitea.tose.top"},
|
||||
deny_hosts={"bad.example"})
|
||||
cases = [
|
||||
(EgressRequest("github", "GET", "https", "gitea.tose.top",
|
||||
443, "/api/v1/version"),
|
||||
"unknown_connector"),
|
||||
(self._req(scheme="http"), "scheme_not_allowed"),
|
||||
(self._req(port=80), "port_not_allowed"),
|
||||
(self._req(host="bad.example"), "host_denied"),
|
||||
(self._req(path="/login"), "path_not_allowed"),
|
||||
# round-11 真机修复(层 4):裸 .git 只放 GET,POST 仍拒。
|
||||
(self._req(method="POST", path="/admin/r.git"),
|
||||
"path_not_allowed"),
|
||||
]
|
||||
for req, reason in cases:
|
||||
dec = policy.evaluate(req, resolve=_public_resolver)
|
||||
self.assertEqual(dec.verdict, "deny", reason)
|
||||
self.assertEqual(dec.reasons, [reason])
|
||||
# git 智能 HTTP 三条路径放行,裸 .git 只放 GET(层 4)。
|
||||
for method, path in (
|
||||
("GET", "/admin/r.git/info/refs"),
|
||||
("GET", "/admin/r.git"),
|
||||
("POST", "/admin/r.git/git-upload-pack"),
|
||||
("POST", "/admin/r.git/git-receive-pack")):
|
||||
dec = policy.evaluate(
|
||||
self._req(method=method, path=path),
|
||||
resolve=_public_resolver)
|
||||
self.assertEqual(dec.verdict, "allow", path)
|
||||
|
||||
def test_capability_binding_expiry_and_uses(self):
|
||||
policy = SentinelPolicy(allow_hosts={"gitea.tose.top"})
|
||||
dec = policy.evaluate(self._req(), resolve=_public_resolver,
|
||||
now=1000.0)
|
||||
cap = dec.capability
|
||||
req = self._req()
|
||||
self.assertTrue(cap.matches(req, now=1000.0))
|
||||
self.assertFalse(cap.matches(self._req(host="other.example"),
|
||||
now=1000.0))
|
||||
self.assertFalse(cap.matches(self._req(method="POST"),
|
||||
now=1000.0))
|
||||
self.assertFalse(cap.matches(self._req(path="/login"),
|
||||
now=1000.0))
|
||||
self.assertFalse(cap.matches(req, now=1000.0 + 301))
|
||||
self.assertTrue(cap.use(req, now=1000.0))
|
||||
self.assertFalse(cap.use(req, now=1000.0))
|
||||
|
||||
def test_from_env_and_approval_queue(self):
|
||||
policy = SentinelPolicy.from_env(
|
||||
{"CHV_SENTINEL_ALLOW_HOSTS": "a.example, b.example "},
|
||||
extra_allow_hosts={"c.example"})
|
||||
self.assertEqual(policy.allow_hosts,
|
||||
{"a.example", "b.example", "c.example"})
|
||||
queue = ApprovalQueue(clock=lambda: 1000.0)
|
||||
req = self._req(host="other.example")
|
||||
pid = queue.record(req, ["host_not_allowlisted"])
|
||||
pending = queue.list_pending()
|
||||
self.assertEqual(len(pending), 1)
|
||||
self.assertEqual(pending[0]["pending_id"], pid)
|
||||
self.assertEqual(pending[0]["ts"], 1000.0)
|
||||
self.assertTrue(queue.resolve(pid, "deny"))
|
||||
self.assertFalse(queue.resolve(pid, "deny"))
|
||||
self.assertFalse(queue.resolve("nope", "allow"))
|
||||
self.assertEqual(queue.list_pending(), [])
|
||||
|
||||
|
||||
# -- (3) seed 拆分 ---------------------------------------------------------
|
||||
class SeedSplitTest(unittest.TestCase):
|
||||
def test_split_authd_env(self):
|
||||
authd, rest = split_authd_env(
|
||||
{"GITEA_TOKEN": TOKEN, "SUB2API_API_KEY": "gw",
|
||||
"OTHER": "1"})
|
||||
self.assertEqual(authd, {"GITEA_TOKEN": TOKEN})
|
||||
self.assertEqual(rest, {"SUB2API_API_KEY": "gw", "OTHER": "1"})
|
||||
self.assertEqual(split_authd_env(None), ({}, {}))
|
||||
self.assertEqual(split_authd_env({}), ({}, {}))
|
||||
|
||||
def test_write_seed_refuses_token_in_guest_env(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
seed_dir = os.path.join(tmp, "seed")
|
||||
with self.assertRaises(ValueError):
|
||||
write_seed(seed_dir, "vm1", "PRIV",
|
||||
"ssh-ed25519 AAAAx",
|
||||
secret_env={"GITEA_TOKEN": TOKEN})
|
||||
with self.assertRaises(ValueError):
|
||||
write_seed(seed_dir, "vm1", "PRIV",
|
||||
"ssh-ed25519 AAAAx",
|
||||
guest_env={"GITEA_TOKEN": TOKEN})
|
||||
with self.assertRaises(ValueError):
|
||||
write_seed(seed_dir, "vm1", "PRIV",
|
||||
"ssh-ed25519 AAAAx",
|
||||
authd_env={"SUB2API_API_KEY": "gw"})
|
||||
|
||||
def test_write_seed_authd_entry_is_separate_0600(self):
|
||||
import yaml
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
seed_dir = os.path.join(tmp, "seed")
|
||||
write_seed(seed_dir, "vm1", "PRIV", "ssh-ed25519 AAAAx",
|
||||
secret_env={"SUB2API_API_KEY": "gw-key"},
|
||||
guest_env={"GITEA_URL": "https://gitea.tose.top"},
|
||||
authd_env={"GITEA_TOKEN": TOKEN})
|
||||
with open(os.path.join(seed_dir, "user-data")) as fh:
|
||||
parsed = yaml.safe_load(fh.read())
|
||||
entries = {e["path"]: e for e in parsed["write_files"]}
|
||||
self.assertEqual(entries[AUTHD_SEED_PATH]["permissions"],
|
||||
"0600")
|
||||
self.assertIn("GITEA_TOKEN=" + TOKEN,
|
||||
entries[AUTHD_SEED_PATH]["content"])
|
||||
self.assertNotIn("GITEA_TOKEN",
|
||||
entries["/etc/chv/guest-env"]["content"])
|
||||
|
||||
def test_write_seed_authd_entry_carries_sync_keys(self):
|
||||
# round-11 真机修复(层 2):authd-seed 须同时有
|
||||
# GITEA_URL/GITEA_USER/SYNC_REPO,build_server 才会放行
|
||||
#(只 GITEA_TOKEN 时 ValueError,探针 fixprobe3 即从
|
||||
# guest-env 补这三键);三键只在 guest-env 里有才抄。
|
||||
import yaml
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
seed_dir = os.path.join(tmp, "seed")
|
||||
write_seed(seed_dir, "vm1", "PRIV", "ssh-ed25519 AAAAx",
|
||||
guest_env={
|
||||
"GITEA_URL": "https://gitea.tose.top",
|
||||
"GITEA_USER": "admin",
|
||||
"SYNC_REPO": "admin/chv-sync-test",
|
||||
"SYNC_BRANCH": "main"},
|
||||
authd_env={"GITEA_TOKEN": TOKEN})
|
||||
with open(os.path.join(seed_dir, "user-data")) as fh:
|
||||
parsed = yaml.safe_load(fh.read())
|
||||
entries = {e["path"]: e for e in parsed["write_files"]}
|
||||
content = entries[AUTHD_SEED_PATH]["content"]
|
||||
for key in ("GITEA_URL=https://gitea.tose.top",
|
||||
"GITEA_USER=admin",
|
||||
"SYNC_REPO=admin/chv-sync-test",
|
||||
"GITEA_TOKEN="):
|
||||
self.assertIn(key, content)
|
||||
# 非同步键不进 authd-seed。
|
||||
self.assertNotIn("SYNC_BRANCH", content)
|
||||
|
||||
def test_write_seed_authd_entry_without_sync_keys(self):
|
||||
# guest-env 里没有三键时 authd-seed 只装密钥,不编默认值。
|
||||
import yaml
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
seed_dir = os.path.join(tmp, "seed")
|
||||
write_seed(seed_dir, "vm1", "PRIV", "ssh-ed25519 AAAAx",
|
||||
authd_env={"GITEA_TOKEN": TOKEN})
|
||||
with open(os.path.join(seed_dir, "user-data")) as fh:
|
||||
parsed = yaml.safe_load(fh.read())
|
||||
entries = {e["path"]: e for e in parsed["write_files"]}
|
||||
content = entries[AUTHD_SEED_PATH]["content"]
|
||||
self.assertIn("GITEA_TOKEN=", content)
|
||||
self.assertNotIn("GITEA_URL", content)
|
||||
self.assertNotIn("GITEA_USER", content)
|
||||
self.assertNotIn("SYNC_REPO", content)
|
||||
|
||||
|
||||
# -- (4) wake 预审 -----------------------------------------------------------
|
||||
class _FakeResult:
|
||||
def __init__(self, returncode=0, stdout="", stderr=""):
|
||||
self.returncode = returncode
|
||||
self.stdout = stdout
|
||||
self.stderr = stderr
|
||||
|
||||
|
||||
class _FakeProc:
|
||||
def __init__(self, alive=True):
|
||||
self._alive = alive
|
||||
|
||||
def poll(self):
|
||||
return None if self._alive else 1
|
||||
|
||||
def terminate(self):
|
||||
self._alive = False
|
||||
|
||||
def wait(self, timeout=None):
|
||||
return 0
|
||||
|
||||
def kill(self):
|
||||
self._alive = False
|
||||
|
||||
|
||||
def _write_file(path, content=b"x"):
|
||||
with open(path, "wb") as fh:
|
||||
fh.write(content)
|
||||
os.chmod(path, 0o755)
|
||||
return path
|
||||
|
||||
|
||||
class WakeGateTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.dir = self.tmp.name
|
||||
self.binary = _write_file(
|
||||
os.path.join(self.dir, "cloud-hypervisor"))
|
||||
self.ch_remote = _write_file(os.path.join(self.dir, "ch-remote"))
|
||||
self.kernel = _write_file(os.path.join(self.dir, "vmlinux"))
|
||||
self.rootfs = _write_file(os.path.join(self.dir, "disk.raw"),
|
||||
b"R" * 100)
|
||||
self.ssh_key = _write_file(os.path.join(self.dir, "id_chv"))
|
||||
self.kvm = _write_file(os.path.join(self.dir, "kvm"))
|
||||
self.cpuinfo = os.path.join(self.dir, "cpuinfo")
|
||||
with open(self.cpuinfo, "w") as fh:
|
||||
fh.write("flags : fpu vmx sse\n")
|
||||
self.state_dir = os.path.join(self.dir, "state")
|
||||
self.audit = []
|
||||
self.launches = []
|
||||
|
||||
def tearDown(self):
|
||||
self.tmp.cleanup()
|
||||
|
||||
def _runner(self, cmd, timeout=None):
|
||||
if cmd and cmd[0] == "ssh-keygen":
|
||||
priv = cmd[cmd.index("-f") + 1]
|
||||
with open(priv, "w") as fh:
|
||||
fh.write("-----BEGIN OPENSSH PRIVATE KEY-----\nFAKE\n")
|
||||
os.chmod(priv, 0o600)
|
||||
with open(priv + ".pub", "w") as fh:
|
||||
fh.write("ssh-ed25519 AAAAfake1 vm-key\n")
|
||||
return _FakeResult(0, "", "")
|
||||
if cmd and cmd[0] == "genisoimage":
|
||||
out = cmd[cmd.index("-output") + 1]
|
||||
with open(out, "wb") as fh:
|
||||
fh.write(b"ISO")
|
||||
return _FakeResult(0, "", "")
|
||||
return _FakeResult(0, "", "")
|
||||
|
||||
def _make_exec(self, env_extra=None, resolver=None):
|
||||
env = {"CHV_EGRESS_PROXY": "http://10.0.0.1:8888",
|
||||
"CHV_HOST_KEY_MODE": "provisioned",
|
||||
"CHV_SSH_FORWARD": "1"}
|
||||
if env_extra:
|
||||
env.update(env_extra)
|
||||
ex = CloudHypervisorExecutor(
|
||||
binary=self.binary, ch_remote=self.ch_remote,
|
||||
kernel=self.kernel, rootfs=self.rootfs,
|
||||
ssh_key=self.ssh_key, kvm_path=self.kvm,
|
||||
cpuinfo_path=self.cpuinfo, state_dir=self.state_dir,
|
||||
runner=self._runner,
|
||||
launcher=lambda cmd: self.launches.append(list(cmd))
|
||||
or _FakeProc(alive=True),
|
||||
api_client=lambda s, m, p, payload=None: (200, {}),
|
||||
sleeper=lambda s: None, env=env,
|
||||
sentinel_resolver=resolver)
|
||||
ex.audit_callback = lambda e, t, d, k=None: self.audit.append(
|
||||
(e, t.get("id") if isinstance(t, dict) else t, d, k))
|
||||
return ex
|
||||
|
||||
def test_wake_allowlisted_gitea_passes_gate(self):
|
||||
ex = self._make_exec(
|
||||
env_extra={"CHV_SENTINEL_ALLOW_HOSTS": "gitea.tose.top"},
|
||||
resolver=_public_resolver)
|
||||
out = ex.wake_vm({"id": "gate1", "gitea_sync": dict(FAKE_SYNC)})
|
||||
self.assertEqual(out["state"], "running")
|
||||
kinds = [a[0] for a in self.audit]
|
||||
self.assertIn("sentinel_egress_allow", kinds)
|
||||
|
||||
def test_wake_unknown_gitea_host_is_ask_and_fail_closed(self):
|
||||
ex = self._make_exec(resolver=_public_resolver)
|
||||
sync = dict(FAKE_SYNC)
|
||||
sync["GITEA_URL"] = "https://other.example"
|
||||
with self.assertRaises(ChvFailure) as ctx:
|
||||
ex.wake_vm({"id": "gate2", "gitea_sync": sync})
|
||||
self.assertEqual(ctx.exception.kind, "sentinel_egress_denied")
|
||||
self.assertEqual(self.launches, [])
|
||||
kinds = [a[0] for a in self.audit]
|
||||
self.assertIn("sentinel_egress_ask", kinds)
|
||||
pending = ex._sentinel_queue.list_pending()
|
||||
self.assertEqual(len(pending), 1)
|
||||
self.assertEqual(pending[0]["host"], "other.example")
|
||||
|
||||
def test_wake_gitea_resolving_to_private_ip_is_deny(self):
|
||||
ex = self._make_exec(
|
||||
env_extra={"CHV_SENTINEL_ALLOW_HOSTS": "gitea.tose.top"},
|
||||
resolver=lambda h, p: ["10.0.0.5"])
|
||||
with self.assertRaises(ChvFailure) as ctx:
|
||||
ex.wake_vm({"id": "gate3", "gitea_sync": dict(FAKE_SYNC)})
|
||||
self.assertEqual(ctx.exception.kind, "sentinel_egress_denied")
|
||||
kinds = [a[0] for a in self.audit]
|
||||
self.assertIn("sentinel_egress_denied", kinds)
|
||||
self.assertEqual(ex._sentinel_queue.list_pending(), [])
|
||||
|
||||
def test_wake_without_sync_skips_gate(self):
|
||||
ex = self._make_exec(resolver=_public_resolver)
|
||||
out = ex.wake_vm({"id": "gate4"})
|
||||
self.assertEqual(out["state"], "running")
|
||||
kinds = [a[0] for a in self.audit]
|
||||
self.assertNotIn("sentinel_egress_allow", kinds)
|
||||
self.assertNotIn("sentinel_egress_denied", kinds)
|
||||
self.assertNotIn("sentinel_egress_ask", kinds)
|
||||
|
||||
|
||||
# -- (5) 客内 authd 守护进程 + helper 端到端 -------------------------------
|
||||
class GuestAuthdTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.dir = self.tmp.name
|
||||
self.seed_path = os.path.join(self.dir, "authd-seed")
|
||||
with open(self.seed_path, "w") as fh:
|
||||
fh.write("GITEA_URL=https://gitea.tose.top\n"
|
||||
"GITEA_USER=admin\n"
|
||||
"GITEA_TOKEN=" + TOKEN + "\n"
|
||||
"SYNC_REPO=admin/chv-sync-test\n")
|
||||
self.socket_path = os.path.join(self.dir, "authd.sock")
|
||||
self.log_path = os.path.join(self.dir, "authd.log")
|
||||
mod = _load_guest_authd()
|
||||
self.server = mod.build_server(
|
||||
seed_path=self.seed_path, socket_path=self.socket_path,
|
||||
log_path=self.log_path, lib_dir=str(REPO_ROOT),
|
||||
allowed_uids=[os.getuid()], resolve=_public_resolver)
|
||||
self.thread = threading.Thread(
|
||||
target=self.server.serve_forever, daemon=True)
|
||||
self.thread.start()
|
||||
|
||||
def tearDown(self):
|
||||
self.server.shutdown()
|
||||
self.server.server_close()
|
||||
self.tmp.cleanup()
|
||||
|
||||
def _ask(self, req):
|
||||
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
|
||||
sock.settimeout(5)
|
||||
try:
|
||||
sock.connect(self.socket_path)
|
||||
sock.sendall((json.dumps(req) + "\n").encode("utf-8"))
|
||||
buf = b""
|
||||
while b"\n" not in buf:
|
||||
chunk = sock.recv(4096)
|
||||
if not chunk:
|
||||
break
|
||||
buf += chunk
|
||||
return json.loads(buf.decode("utf-8"))
|
||||
finally:
|
||||
sock.close()
|
||||
|
||||
def test_release_for_clone_path(self):
|
||||
resp = self._ask({"connector": "gitea",
|
||||
"host": "gitea.tose.top",
|
||||
"path": "/admin/chv-sync-test.git/info/refs"})
|
||||
self.assertTrue(resp["ok"])
|
||||
self.assertEqual(resp["username"], "admin")
|
||||
self.assertEqual(resp["password"], TOKEN)
|
||||
self.assertGreater(resp["expires_at"], 0)
|
||||
|
||||
def test_bare_git_path_is_pull(self):
|
||||
# round-11 真机修复(层 4):裸 .git 路径一律当 pull。
|
||||
resp = self._ask({"connector": "gitea",
|
||||
"host": "gitea.tose.top",
|
||||
"path": "/admin/chv-sync-test.git"})
|
||||
self.assertTrue(resp["ok"])
|
||||
self.assertEqual(resp["username"], "admin")
|
||||
self.assertEqual(resp["password"], TOKEN)
|
||||
|
||||
def test_socket_uses_group_perms(self):
|
||||
# round-11 真机修复(层 3):socket 用 group 权限(目录
|
||||
# 0750、socket 0660),不用探针的 0666;放行 uid 无
|
||||
# passwd 条目时才回落 root-only(见 SocketGroupTest)。
|
||||
import stat
|
||||
mod = _load_guest_authd()
|
||||
if mod._socket_group_gid([os.getuid()]) is None:
|
||||
self.skipTest("放行 uid 无 passwd 条目,组权限不可解析")
|
||||
self.assertEqual(
|
||||
stat.S_IMODE(os.stat(self.socket_path).st_mode), 0o660)
|
||||
self.assertEqual(
|
||||
stat.S_IMODE(os.stat(self.dir).st_mode), 0o750)
|
||||
|
||||
def test_second_request_gets_fresh_release(self):
|
||||
for _ in range(2):
|
||||
resp = self._ask({"connector": "gitea",
|
||||
"host": "gitea.tose.top",
|
||||
"path": "/admin/chv-sync-test.git/"
|
||||
"git-upload-pack"})
|
||||
self.assertTrue(resp["ok"])
|
||||
self.assertEqual(resp["password"], TOKEN)
|
||||
|
||||
def test_refused_paths_hosts_and_repos(self):
|
||||
cases = [
|
||||
({"connector": "github", "host": "gitea.tose.top",
|
||||
"path": "/admin/chv-sync-test.git/info/refs"},
|
||||
"bad_request_fields"),
|
||||
({"connector": "gitea", "host": "gitea.tose.top",
|
||||
"path": "/admin/chv-sync-test.git/foo"},
|
||||
"path_not_allowed"),
|
||||
({"connector": "gitea", "host": "other.example",
|
||||
"path": "/admin/chv-sync-test.git/info/refs"},
|
||||
"sentinel_ask"),
|
||||
({"connector": "gitea", "host": "gitea.tose.top",
|
||||
"path": "/foo.git/info/refs"}, "repo_unparseable"),
|
||||
]
|
||||
for req, error in cases:
|
||||
resp = self._ask(req)
|
||||
self.assertFalse(resp["ok"], req)
|
||||
self.assertEqual(resp["error"], error, req)
|
||||
|
||||
def test_audit_log_never_contains_value(self):
|
||||
self._ask({"connector": "gitea", "host": "gitea.tose.top",
|
||||
"path": "/admin/chv-sync-test.git/info/refs"})
|
||||
self._ask({"connector": "gitea", "host": "gitea.tose.top",
|
||||
"path": "/admin/chv-sync-test.git/foo"})
|
||||
with open(self.log_path) as fh:
|
||||
self.assertNotIn(TOKEN, fh.read())
|
||||
|
||||
def test_incomplete_or_missing_seed_fails_fast(self):
|
||||
mod = _load_guest_authd()
|
||||
bad_seed = os.path.join(self.dir, "bad-seed")
|
||||
with open(bad_seed, "w") as fh:
|
||||
fh.write("GITEA_URL=https://gitea.tose.top\n")
|
||||
with self.assertRaises(ValueError):
|
||||
mod.build_server(seed_path=bad_seed,
|
||||
socket_path=os.path.join(self.dir, "s2"),
|
||||
log_path=self.log_path,
|
||||
lib_dir=str(REPO_ROOT),
|
||||
allowed_uids=[os.getuid()],
|
||||
resolve=_public_resolver)
|
||||
with self.assertRaises(FileNotFoundError):
|
||||
mod.build_server(seed_path=os.path.join(self.dir, "nope"),
|
||||
socket_path=os.path.join(self.dir, "s3"),
|
||||
log_path=self.log_path,
|
||||
lib_dir=str(REPO_ROOT),
|
||||
allowed_uids=[os.getuid()],
|
||||
resolve=_public_resolver)
|
||||
|
||||
|
||||
class SocketGroupTest(unittest.TestCase):
|
||||
"""round-11 真机修复(层 3):socket 共享组 gid 的派生与回落。"""
|
||||
|
||||
def test_explicit_group_and_derivation(self):
|
||||
mod = _load_guest_authd()
|
||||
# 显式 int 即 gid;显式组名解析;不存在的组名 → None。
|
||||
self.assertEqual(mod._socket_group_gid([0, 1000], 1234), 1234)
|
||||
self.assertEqual(mod._socket_group_gid([0, 1000], "root"), 0)
|
||||
self.assertIsNone(
|
||||
mod._socket_group_gid([0, 1000], "no-such-group-xyz"))
|
||||
# 只剩 root → 无组可派,回落 root-only。
|
||||
self.assertIsNone(mod._socket_group_gid([0]))
|
||||
self.assertIsNone(mod._socket_group_gid([]))
|
||||
# 派生:第一个非 root 放行 uid 的主组(用 passwd 里第一个
|
||||
# 非 root 条目,不依赖当前测试进程的 uid——本机测试以 root
|
||||
# 跑,os.getuid() 本身就是 0,派生不到组)。
|
||||
import pwd
|
||||
others = [p for p in pwd.getpwall() if p.pw_uid != 0]
|
||||
if not others:
|
||||
self.skipTest("passwd 里无非 root 用户")
|
||||
first = sorted(others, key=lambda p: p.pw_uid)[0]
|
||||
self.assertEqual(
|
||||
mod._socket_group_gid([0, first.pw_uid]), first.pw_gid)
|
||||
|
||||
def test_server_applies_group_perms_with_explicit_group(self):
|
||||
# 显式给组时服务器落地 0750/0660(本机以 root 跑测试,
|
||||
# allowed_uids 只有 root,派生不到组,须显式给才能覆到
|
||||
# AuthdServer 的落盘路径)。
|
||||
import stat
|
||||
mod = _load_guest_authd()
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
seed_path = os.path.join(tmp, "authd-seed")
|
||||
with open(seed_path, "w") as fh:
|
||||
fh.write("GITEA_URL=https://gitea.tose.top\n"
|
||||
"GITEA_USER=admin\n"
|
||||
"GITEA_TOKEN=" + TOKEN + "\n"
|
||||
"SYNC_REPO=admin/chv-sync-test\n")
|
||||
socket_path = os.path.join(tmp, "authd.sock")
|
||||
log_path = os.path.join(tmp, "authd.log")
|
||||
server = mod.build_server(
|
||||
seed_path=seed_path, socket_path=socket_path,
|
||||
log_path=log_path, lib_dir=str(REPO_ROOT),
|
||||
allowed_uids=[os.getuid()], resolve=_public_resolver,
|
||||
socket_group=os.getgid())
|
||||
try:
|
||||
self.assertEqual(server.socket_gid, os.getgid())
|
||||
self.assertEqual(
|
||||
stat.S_IMODE(os.stat(socket_path).st_mode), 0o660)
|
||||
self.assertEqual(
|
||||
stat.S_IMODE(os.stat(tmp).st_mode), 0o750)
|
||||
finally:
|
||||
server.server_close()
|
||||
|
||||
|
||||
class HelperE2ETest(unittest.TestCase):
|
||||
"""helper + 真 git credential fill(假 authd socket 在后台)。"""
|
||||
|
||||
HELPER = REPO_ROOT / "chv-guest" / "git-credential-chv-authd"
|
||||
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.dir = self.tmp.name
|
||||
self.guest = GuestAuthdTest("test_release_for_clone_path")
|
||||
self.guest.setUp()
|
||||
self.socket_path = self.guest.socket_path
|
||||
|
||||
def tearDown(self):
|
||||
self.guest.tearDown()
|
||||
self.tmp.cleanup()
|
||||
|
||||
def _helper_env(self):
|
||||
env = dict(os.environ)
|
||||
env["CHV_AUTHD_SOCKET"] = self.socket_path
|
||||
return env
|
||||
|
||||
def test_helper_returns_credential(self):
|
||||
proc = subprocess.run(
|
||||
[sys.executable, str(self.HELPER)],
|
||||
input="protocol=https\nhost=gitea.tose.top\n"
|
||||
"path=/admin/chv-sync-test.git/info/refs"
|
||||
"?service=git-upload-pack\n\n",
|
||||
capture_output=True, text=True, env=self._helper_env(),
|
||||
check=False)
|
||||
self.assertEqual(proc.returncode, 0, proc.stderr)
|
||||
self.assertIn("username=admin", proc.stdout)
|
||||
self.assertIn("password=" + TOKEN, proc.stdout)
|
||||
self.assertIn("password_expiry_utc=", proc.stdout)
|
||||
|
||||
def test_helper_refusal_is_exit_1(self):
|
||||
proc = subprocess.run(
|
||||
[sys.executable, str(self.HELPER)],
|
||||
input="protocol=https\nhost=gitea.tose.top\n"
|
||||
"path=/admin/chv-sync-test.git/foo\n\n",
|
||||
capture_output=True, text=True, env=self._helper_env(),
|
||||
check=False)
|
||||
self.assertEqual(proc.returncode, 1)
|
||||
self.assertIn("path_not_allowed", proc.stderr)
|
||||
|
||||
def test_helper_returns_credential_for_bare_git_path(self):
|
||||
# round-11 真机修复(层 4):裸 .git 路径一律当 pull,
|
||||
# helper 须照常返回凭证(git 智能 HTTP 发现请求只发裸路径)。
|
||||
proc = subprocess.run(
|
||||
[sys.executable, str(self.HELPER)],
|
||||
input="protocol=https\nhost=gitea.tose.top\n"
|
||||
"path=/admin/chv-sync-test.git\n\n",
|
||||
capture_output=True, text=True, env=self._helper_env(),
|
||||
check=False)
|
||||
self.assertEqual(proc.returncode, 0, proc.stderr)
|
||||
self.assertIn("username=admin", proc.stdout)
|
||||
self.assertIn("password=" + TOKEN, proc.stdout)
|
||||
|
||||
def test_git_credential_fill_end_to_end(self):
|
||||
env = self._helper_env()
|
||||
env["GIT_CONFIG_NOSYSTEM"] = "1"
|
||||
proc = subprocess.run(
|
||||
["git", "-c", "credential.helper=%s" % self.HELPER,
|
||||
"-c", "credential.useHttpPath=true",
|
||||
"credential", "fill"],
|
||||
input="protocol=https\nhost=gitea.tose.top\n"
|
||||
"path=/admin/chv-sync-test.git/info/refs\n\n",
|
||||
capture_output=True, text=True, env=env, cwd=self.dir,
|
||||
check=False)
|
||||
self.assertEqual(proc.returncode, 0, proc.stderr)
|
||||
self.assertIn("password=" + TOKEN, proc.stdout)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,180 @@
|
||||
"""round-11 集成补洞:执行器接 nspawn 单元(P2 差距表 §4)。
|
||||
|
||||
只测 _guest_host_command 的命令构造与 CHV_CELL_ENABLED 开关:
|
||||
单元开(默认)时 host.py 调用改为 sudo chv-cell run -- python3
|
||||
/app/host.py --state … --codex-home … --sandbox
|
||||
danger-full-access …(sudo 不可省:chv-cell 须 root,执行器以
|
||||
chv 身份调;round-11 真机修复);单元关时走原命令;非法开关值即
|
||||
ValueError。只用 mock,不启动真实 VM、不连真实网络。
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
from cloud.chv import CloudHypervisorExecutor
|
||||
except ImportError: # discover -s cloud/tests 时把项目根目录补进 sys.path
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
from cloud.chv import CloudHypervisorExecutor
|
||||
|
||||
|
||||
class FakeResult:
|
||||
def __init__(self, returncode=0, stdout="", stderr=""):
|
||||
self.returncode = returncode
|
||||
self.stdout = stdout
|
||||
self.stderr = stderr
|
||||
|
||||
|
||||
class FakeProc:
|
||||
def __init__(self, alive=True):
|
||||
self._alive = alive
|
||||
|
||||
def poll(self):
|
||||
return None if self._alive else 1
|
||||
|
||||
|
||||
def _write(path, data=b"x"):
|
||||
with open(path, "wb") as fh:
|
||||
fh.write(data)
|
||||
return path
|
||||
|
||||
|
||||
class CellCase(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.dir = self.tmp.name
|
||||
self.binary = _write(os.path.join(self.dir, "cloud-hypervisor"))
|
||||
self.ch_remote = _write(os.path.join(self.dir, "ch-remote"))
|
||||
self.kernel = _write(os.path.join(self.dir, "vmlinux"))
|
||||
self.rootfs = _write(os.path.join(self.dir, "disk.raw"),
|
||||
b"R" * 100)
|
||||
self.ssh_key = _write(os.path.join(self.dir, "id_chv"))
|
||||
self.kvm = _write(os.path.join(self.dir, "kvm"))
|
||||
self.cpuinfo = os.path.join(self.dir, "cpuinfo")
|
||||
with open(self.cpuinfo, "w") as fh:
|
||||
fh.write("flags : fpu vmx sse\n")
|
||||
self.state_dir = os.path.join(self.dir, "state")
|
||||
|
||||
def tearDown(self):
|
||||
self.tmp.cleanup()
|
||||
|
||||
def make_exec(self, env_extra=None):
|
||||
env = {"CHV_EGRESS_PROXY": "http://10.0.0.1:8888",
|
||||
"CHV_HOST_KEY_MODE": "keyscan",
|
||||
"CHV_SSH_FORWARD": "1"}
|
||||
if env_extra:
|
||||
env.update(env_extra)
|
||||
return CloudHypervisorExecutor(
|
||||
binary=self.binary, ch_remote=self.ch_remote,
|
||||
kernel=self.kernel, rootfs=self.rootfs,
|
||||
ssh_key=self.ssh_key, kvm_path=self.kvm,
|
||||
cpuinfo_path=self.cpuinfo, state_dir=self.state_dir,
|
||||
runner=lambda cmd, timeout=None: FakeResult(0, "", ""),
|
||||
launcher=lambda cmd: FakeProc(alive=True),
|
||||
api_client=lambda *a, **k: (200, {}),
|
||||
sleeper=lambda s: None, env=env)
|
||||
|
||||
|
||||
class CellCommandTest(CellCase):
|
||||
def test_cell_on_by_default(self):
|
||||
# 开关未设即默认开:完整命令与契约示例一致。
|
||||
ex = self.make_exec()
|
||||
cmd = ex._guest_host_command("status", "--task", "t1")
|
||||
self.assertEqual(cmd, [
|
||||
"sudo", "chv-cell", "run", "--",
|
||||
"python3", "/app/host.py",
|
||||
"--state", "/home/chv/.codex/host-state.json",
|
||||
"--codex-home", "/home/chv/.codex",
|
||||
"--sandbox", "danger-full-access",
|
||||
"status", "--task", "t1"])
|
||||
|
||||
def test_cell_on_explicit_1(self):
|
||||
ex = self.make_exec(env_extra={"CHV_CELL_ENABLED": "1"})
|
||||
cmd = ex._guest_host_command("status", "--task", "t1")
|
||||
self.assertEqual(cmd[:4], ["sudo", "chv-cell", "run", "--"])
|
||||
self.assertIn("--sandbox", cmd)
|
||||
self.assertEqual(
|
||||
cmd[cmd.index("--sandbox") + 1], "danger-full-access")
|
||||
|
||||
def test_cell_off_returns_original_command(self):
|
||||
# 关时走原命令:一个 token 也不多不少(旧镜像逃生口)。
|
||||
ex = self.make_exec(env_extra={"CHV_CELL_ENABLED": "0"})
|
||||
cmd = ex._guest_host_command("status", "--task", "t1")
|
||||
self.assertEqual(cmd, ["python3", "/app/host.py",
|
||||
"status", "--task", "t1"])
|
||||
self.assertNotIn("chv-cell", cmd)
|
||||
self.assertNotIn("--state", cmd)
|
||||
self.assertNotIn("--codex-home", cmd)
|
||||
self.assertNotIn("--sandbox", cmd)
|
||||
|
||||
def test_cell_on_preserves_original_params(self):
|
||||
# 原参数一个不少:model/provider/effort/approval/
|
||||
# 调用方 sandbox 与子命令全部保留且顺序不变;强制的
|
||||
# danger-full-access 放全局参数最后,盖过调用方的
|
||||
# read-only(argparse 后胜出),read-only 本身仍在。
|
||||
ex = self.make_exec()
|
||||
cmd = ex._guest_host_command(
|
||||
"dispatch", "--name", "t", "--task", "x",
|
||||
model="m1", effort="low", approval="ask",
|
||||
model_provider="sub2api", sandbox="read-only")
|
||||
self.assertEqual(cmd, [
|
||||
"sudo", "chv-cell", "run", "--",
|
||||
"python3", "/app/host.py",
|
||||
"--state", "/home/chv/.codex/host-state.json",
|
||||
"--codex-home", "/home/chv/.codex",
|
||||
"--model", "m1",
|
||||
"--model-provider", "sub2api",
|
||||
"--effort", "low",
|
||||
"--approval-policy", "untrusted",
|
||||
"--sandbox", "read-only",
|
||||
"--sandbox", "danger-full-access",
|
||||
"dispatch", "--name", "t", "--task", "x"])
|
||||
# --state/--codex-home 紧跟 host.py 路径。
|
||||
host_idx = cmd.index("/app/host.py")
|
||||
self.assertEqual(
|
||||
cmd[host_idx + 1:host_idx + 5],
|
||||
["--state", "/home/chv/.codex/host-state.json",
|
||||
"--codex-home", "/home/chv/.codex"])
|
||||
|
||||
def test_sandbox_only_in_cell_path(self):
|
||||
# 调用方没给 sandbox、approval 也没带时:单元路径
|
||||
# 有且仅有一个 --sandbox(danger-full-access),
|
||||
# 非单元路径一个也没有。
|
||||
cell_on = self.make_exec()
|
||||
on_cmd = cell_on._guest_host_command("status", "--task", "t1")
|
||||
self.assertEqual(on_cmd.count("--sandbox"), 1)
|
||||
self.assertEqual(
|
||||
on_cmd[on_cmd.index("--sandbox") + 1],
|
||||
"danger-full-access")
|
||||
cell_off = self.make_exec(
|
||||
env_extra={"CHV_CELL_ENABLED": "0"})
|
||||
off_cmd = cell_off._guest_host_command(
|
||||
"status", "--task", "t1")
|
||||
self.assertEqual(off_cmd.count("--sandbox"), 0)
|
||||
|
||||
def test_cell_invalid_value_raises(self):
|
||||
# 只认 0/1:非法值即 ValueError,不静默 fallback。
|
||||
for bad in ("yes", "2", "", " true"):
|
||||
ex = self.make_exec(
|
||||
env_extra={"CHV_CELL_ENABLED": bad})
|
||||
with self.assertRaises(ValueError):
|
||||
ex._guest_host_command("status", "--task", "t1")
|
||||
|
||||
def test_cell_wiring_reaches_ssh_command(self):
|
||||
# 接线落到 SSH 层:build_ssh_command 的远端命令里
|
||||
# sudo chv-cell 前缀在、python3 在其后。
|
||||
ex = self.make_exec()
|
||||
inner = ex._guest_host_command("status", "--task", "t1")
|
||||
cmd = ex.build_ssh_command({"id": "t1"}, inner)
|
||||
self.assertIn("sudo", cmd)
|
||||
self.assertIn("chv-cell", cmd)
|
||||
self.assertIn("danger-full-access", cmd)
|
||||
self.assertLess(cmd.index("sudo"), cmd.index("chv-cell"))
|
||||
self.assertLess(cmd.index("chv-cell"), cmd.index("python3"))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -244,7 +244,14 @@ class SharedGoalTest(unittest.TestCase):
|
||||
chv = ChvOptCase()
|
||||
chv.setUp()
|
||||
try:
|
||||
ex = chv.make_exec()
|
||||
# 单元路径(CHV_CELL_ENABLED 默认开)有意加 CHV
|
||||
# 专属 argv:chv-cell 前缀 + --state/--codex-home
|
||||
# + 强制 --sandbox danger-full-access,三执行器
|
||||
# 顶层参数全等只在非单元路径上有意义,故这里显式
|
||||
# 关单元;单元路径的“原参数一个不少”由
|
||||
# test_chv_muse_round11_cell.py 钉死。
|
||||
ex = chv.make_exec(
|
||||
env_extra={"CHV_CELL_ENABLED": "0"})
|
||||
local = LocalExecutor("/tmp/host.py")
|
||||
docker = DockerExecutor()
|
||||
for kwargs in (
|
||||
|
||||
@@ -0,0 +1,607 @@
|
||||
"""CHV 残留收尾轨(feat-chv-closeup)新增测试。
|
||||
|
||||
覆盖 SECURITY-REAUDIT §5 第 5 点的逐项闭环:
|
||||
P2-20(_guest_env_delivered 加锁)、P2-19(net 登记失败
|
||||
审计 + 拒起)、P2-17(net.json 删失败审计、脏标记写失败
|
||||
审计 + 返回信号)、P2-18(固定 tmp 名 + 无 fsync 改随机
|
||||
名原子写 + 双 fsync)、H1 附带(NO_PROXY 合并顺序)、
|
||||
M4(generate 只读 BUILDINFO.versions、Chromium 单独钉死、
|
||||
build.sh /app 最小集)。
|
||||
|
||||
全部用标准库与 mock,不启动真实 VM、不碰 WSL/真机;
|
||||
M4 的“真实 lockfile 产出”仍待 V-9 真机构建,本文件只
|
||||
验“生成端以 BUILDINFO 为唯一来源 + 钉死校验逻辑”,
|
||||
版本号全部是合成假值,绝不编造真实版本号。
|
||||
"""
|
||||
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
try:
|
||||
from cloud.chv.atomic_file import atomic_write_text
|
||||
from cloud.chv.disk import _save_snapshots
|
||||
from cloud.chv.executor import CloudHypervisorExecutor
|
||||
from cloud.chv.hostkeys import dirty_marker_path
|
||||
from cloud.chv.orphans import _write_registry
|
||||
from cloud.chv.ssh_transport import (
|
||||
remote_command,
|
||||
write_known_hosts,
|
||||
)
|
||||
from cloud.sandbox import ChvFailure
|
||||
except ImportError: # discover -s cloud/tests 时把项目根目录补进 sys.path
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
from cloud.chv.atomic_file import atomic_write_text
|
||||
from cloud.chv.disk import _save_snapshots
|
||||
from cloud.chv.executor import CloudHypervisorExecutor
|
||||
from cloud.chv.hostkeys import dirty_marker_path
|
||||
from cloud.chv.orphans import _write_registry
|
||||
from cloud.chv.ssh_transport import (
|
||||
remote_command,
|
||||
write_known_hosts,
|
||||
)
|
||||
from cloud.sandbox import ChvFailure
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
|
||||
|
||||
def _load_script(name):
|
||||
"""把 chv-guest 下的独立脚本当模块加载(无包结构)。"""
|
||||
path = ROOT / "chv-guest" / name
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
"chv_guest_" + name[:-3], str(path))
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
class FakeResult:
|
||||
def __init__(self, returncode=0, stdout="", stderr=""):
|
||||
self.returncode = returncode
|
||||
self.stdout = stdout
|
||||
self.stderr = stderr
|
||||
|
||||
|
||||
class FakeProc:
|
||||
def __init__(self, alive=True):
|
||||
self._alive = alive
|
||||
|
||||
def poll(self):
|
||||
return None if self._alive else 1
|
||||
|
||||
def terminate(self):
|
||||
self._alive = False
|
||||
|
||||
def wait(self, timeout=None):
|
||||
return 0
|
||||
|
||||
def kill(self):
|
||||
self._alive = False
|
||||
|
||||
|
||||
def _write(path, content=b"x"):
|
||||
with open(path, "wb") as fh:
|
||||
fh.write(content)
|
||||
os.chmod(path, 0o755)
|
||||
return path
|
||||
|
||||
|
||||
class ResidualCase(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.dir = self.tmp.name
|
||||
self.binary = _write(os.path.join(self.dir, "cloud-hypervisor"))
|
||||
self.ch_remote = _write(os.path.join(self.dir, "ch-remote"))
|
||||
self.kernel = _write(os.path.join(self.dir, "vmlinux"))
|
||||
self.rootfs = _write(os.path.join(self.dir, "disk.raw"),
|
||||
b"R" * 100)
|
||||
self.ssh_key = _write(os.path.join(self.dir, "id_chv"))
|
||||
self.kvm = _write(os.path.join(self.dir, "kvm"))
|
||||
self.cpuinfo = os.path.join(self.dir, "cpuinfo")
|
||||
with open(self.cpuinfo, "w") as fh:
|
||||
fh.write("flags : fpu vmx sse\n")
|
||||
self.state_dir = os.path.join(self.dir, "state")
|
||||
self.calls = []
|
||||
self.api_calls = []
|
||||
self.audit = []
|
||||
|
||||
def tearDown(self):
|
||||
self.tmp.cleanup()
|
||||
|
||||
def make_exec(self, env_extra=None, **overrides):
|
||||
env = {"CHV_EGRESS_PROXY": "http://10.0.0.1:8888",
|
||||
"CHV_HOST_KEY_MODE": "keyscan",
|
||||
"CHV_SSH_FORWARD": "1"}
|
||||
if env_extra:
|
||||
env.update(env_extra)
|
||||
kwargs = {
|
||||
"binary": self.binary, "ch_remote": self.ch_remote,
|
||||
"kernel": self.kernel, "rootfs": self.rootfs,
|
||||
"ssh_key": self.ssh_key, "kvm_path": self.kvm,
|
||||
"cpuinfo_path": self.cpuinfo, "state_dir": self.state_dir,
|
||||
"runner": self._default_runner,
|
||||
"launcher": self._default_launcher,
|
||||
"api_client": self._default_api,
|
||||
"sleeper": lambda s: None, "env": env}
|
||||
kwargs.update(overrides)
|
||||
ex = CloudHypervisorExecutor(**kwargs)
|
||||
ex.audit_callback = lambda e, t, d, k=None: self.audit.append(
|
||||
(e, t.get("id") if isinstance(t, dict) else t, d, k))
|
||||
return ex
|
||||
|
||||
def _default_runner(self, cmd, timeout=None):
|
||||
self.calls.append(list(cmd))
|
||||
return FakeResult(0, "", "")
|
||||
|
||||
def _default_api(self, socket_path, method, path, payload=None):
|
||||
self.api_calls.append((method, path, payload))
|
||||
return (200, {})
|
||||
|
||||
def _default_launcher(self, cmd):
|
||||
return FakeProc(alive=True)
|
||||
|
||||
def _task(self, task_id="abc123"):
|
||||
return {"id": task_id, "prompt": "做一个加法函数",
|
||||
"project_id": "proj1"}
|
||||
|
||||
def _audit_events(self):
|
||||
return [e for e, _t, _d, _k in self.audit]
|
||||
|
||||
# -- P2-20 ---------------------------------------------------------
|
||||
def test_guest_env_lock_exists_and_helpers_roundtrip(self):
|
||||
ex = self.make_exec()
|
||||
self.assertIsInstance(ex._guest_env_lock, type(threading.Lock()))
|
||||
ex._mark_guest_env_delivered("chv-a", {"SUB2API_API_KEY"})
|
||||
self.assertEqual(ex._delivered_secrets("chv-a"),
|
||||
{"SUB2API_API_KEY"})
|
||||
ex._mark_guest_env_delivered("chv-a", set())
|
||||
self.assertEqual(ex._delivered_secrets("chv-a"), set())
|
||||
ex._mark_guest_env_delivered("chv-a", {"K"})
|
||||
ex._clear_guest_env_delivered("chv-a")
|
||||
self.assertEqual(ex._delivered_secrets("chv-a"), set())
|
||||
|
||||
def test_guest_env_concurrent_mark_read_clear(self):
|
||||
ex = self.make_exec()
|
||||
errors = []
|
||||
|
||||
def worker(n):
|
||||
try:
|
||||
for i in range(50):
|
||||
name = f"chv-w{i % 4}"
|
||||
ex._mark_guest_env_delivered(name, {f"K{n}"})
|
||||
got = ex._delivered_secrets(name)
|
||||
self.assertIsInstance(got, set)
|
||||
ex._clear_guest_env_delivered(name)
|
||||
except Exception as exc: # noqa: BLE001 — 汇总后断言
|
||||
errors.append(exc)
|
||||
|
||||
threads = [threading.Thread(target=worker, args=(n,))
|
||||
for n in range(8)]
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join()
|
||||
self.assertEqual(errors, [])
|
||||
# 结束后确定性收尾:最后一次 mark 必须可见,带密钥的
|
||||
# build_ssh_command 不再报 secret_env_not_delivered。
|
||||
ex._mark_guest_env_delivered("chv-final", {"SUB2API_API_KEY"})
|
||||
cmd = ex.build_ssh_command(
|
||||
self._task("final"), ["true"],
|
||||
extra_env={"SUB2API_API_KEY": "x"})
|
||||
self.assertEqual(cmd[0], "ssh")
|
||||
self.assertNotIn("x", cmd)
|
||||
|
||||
def test_build_ssh_command_sees_delivered_after_mark(self):
|
||||
ex = self.make_exec()
|
||||
ex._mark_guest_env_delivered("chv-abc123", {"SUB2API_API_KEY"})
|
||||
cmd = ex.build_ssh_command(
|
||||
self._task(), ["true"],
|
||||
extra_env={"SUB2API_API_KEY": "x"})
|
||||
self.assertNotIn("x", cmd)
|
||||
|
||||
# -- P2-19 ---------------------------------------------------------
|
||||
def test_record_net_entries_failure_returns_false_and_audits(self):
|
||||
ex = self.make_exec()
|
||||
blocker = os.path.join(self.state_dir, "chv-x")
|
||||
os.makedirs(self.state_dir, exist_ok=True)
|
||||
with open(blocker, "w") as fh:
|
||||
fh.write("x")
|
||||
with self.assertLogs("cloud.cloud_hypervisor",
|
||||
level="WARNING") as logs:
|
||||
ok = ex._record_net_entries("chv-x", [{"tap": "tap0"}])
|
||||
self.assertFalse(ok)
|
||||
self.assertTrue(any("net_registry_write_failed" in line
|
||||
for line in logs.output))
|
||||
self.assertIn("net_registry_write_failed",
|
||||
self._audit_events())
|
||||
|
||||
def test_boot_vm_refuses_when_net_registry_fails(self):
|
||||
ex = self.make_exec(env_extra={
|
||||
"CHV_NET_JSON": '[{"tap": "tap0", "ip": "10.0.0.2"}]'})
|
||||
os.makedirs(self.state_dir, exist_ok=True)
|
||||
with open(os.path.join(self.state_dir, "chv-abc123"),
|
||||
"w") as fh:
|
||||
fh.write("x")
|
||||
with self.assertRaises(ChvFailure) as ctx:
|
||||
ex._boot_vm(self._task(), "chv-abc123", self.rootfs,
|
||||
None, [{"tap": "tap0", "ip": "10.0.0.2"}])
|
||||
self.assertEqual(ctx.exception.kind,
|
||||
"net_registry_write_failed")
|
||||
paths = [p for _m, p, _pl in self.api_calls]
|
||||
self.assertIn("/vm.create", paths)
|
||||
self.assertNotIn("/vm.boot", paths)
|
||||
self.assertIn("net_registry_write_failed",
|
||||
self._audit_events())
|
||||
|
||||
# -- P2-17 ---------------------------------------------------------
|
||||
def test_cleanup_net_remove_failure_audited(self):
|
||||
ex = self.make_exec()
|
||||
vm_dir = os.path.join(self.state_dir, "chv-abc123")
|
||||
os.makedirs(vm_dir, exist_ok=True)
|
||||
os.makedirs(os.path.join(vm_dir, "net.json"), exist_ok=True)
|
||||
with open(os.path.join(self.state_dir, "chv-abc123.sock"),
|
||||
"w") as fh:
|
||||
fh.write("")
|
||||
self.assertTrue(ex.stop_vm(self._task()))
|
||||
marker = dirty_marker_path(self.state_dir, "chv-abc123")
|
||||
self.assertTrue(os.path.exists(marker))
|
||||
self.assertIn("vm_net_registry_delete_failed",
|
||||
self._audit_events())
|
||||
|
||||
def test_write_dirty_marker_failure_returns_false_and_audits(self):
|
||||
blocker = os.path.join(self.dir, "blocker3")
|
||||
with open(blocker, "w") as fh:
|
||||
fh.write("x")
|
||||
ex = self.make_exec(state_dir=blocker)
|
||||
ok = ex._write_dirty_marker("chv-abc123", True) # 不抛
|
||||
self.assertFalse(ok)
|
||||
self.assertIn("dirty_marker_write_failed",
|
||||
self._audit_events())
|
||||
|
||||
def test_write_dirty_marker_success_roundtrip(self):
|
||||
ex = self.make_exec()
|
||||
self.assertTrue(ex._write_dirty_marker("chv-abc123", True))
|
||||
marker = dirty_marker_path(self.state_dir, "chv-abc123")
|
||||
self.assertTrue(os.path.exists(marker))
|
||||
self.assertTrue(ex._write_dirty_marker("chv-abc123", False))
|
||||
self.assertFalse(os.path.exists(marker))
|
||||
|
||||
# -- P2-18 ---------------------------------------------------------
|
||||
def test_atomic_write_no_fixed_tmp_and_fsync(self):
|
||||
target = os.path.join(self.dir, "probe.json")
|
||||
calls = {"n": 0}
|
||||
real_fsync = os.fsync
|
||||
|
||||
def _counting(fd):
|
||||
calls["n"] += 1
|
||||
return real_fsync(fd)
|
||||
|
||||
with mock.patch.object(os, "fsync", side_effect=_counting):
|
||||
atomic_write_text(target, '{"a": 1}')
|
||||
with open(target) as fh:
|
||||
self.assertEqual(fh.read(), '{"a": 1}')
|
||||
self.assertEqual(oct(os.stat(target).st_mode & 0o777), "0o600")
|
||||
leftovers = [p for p in os.listdir(self.dir)
|
||||
if ".tmp" in p]
|
||||
self.assertEqual(leftovers, [])
|
||||
# 文件 fsync + 目录 fsync,至少两次。
|
||||
self.assertGreaterEqual(calls["n"], 2)
|
||||
|
||||
def test_net_record_leaves_no_tmp_files(self):
|
||||
ex = self.make_exec()
|
||||
self.assertTrue(ex._record_net_entries(
|
||||
"chv-abc123", [{"tap": "tap0"}]))
|
||||
vm_dir = os.path.join(self.state_dir, "chv-abc123")
|
||||
with open(os.path.join(vm_dir, "net.json")) as fh:
|
||||
self.assertEqual(json.load(fh), [{"tap": "tap0"}])
|
||||
self.assertEqual([p for p in os.listdir(vm_dir)
|
||||
if ".tmp" in p], [])
|
||||
|
||||
def test_known_hosts_atomic_no_fixed_tmp(self):
|
||||
path = os.path.join(self.state_dir, "chv-abc123",
|
||||
"known_hosts")
|
||||
write_known_hosts(path, ["127.0.0.1 ssh-ed25519 AAAA"])
|
||||
with open(path) as fh:
|
||||
self.assertEqual(fh.read(),
|
||||
"127.0.0.1 ssh-ed25519 AAAA\n")
|
||||
self.assertEqual(oct(os.stat(path).st_mode & 0o777), "0o600")
|
||||
vm_dir = os.path.join(self.state_dir, "chv-abc123")
|
||||
self.assertEqual([p for p in os.listdir(vm_dir)
|
||||
if ".tmp" in p], [])
|
||||
|
||||
def test_snapshot_save_no_fixed_tmp(self):
|
||||
_save_snapshots(self.state_dir, "chv-abc123",
|
||||
[{"url": "file://x"}])
|
||||
path = os.path.join(self.state_dir, "chv-abc123",
|
||||
"snapshots.json")
|
||||
with open(path) as fh:
|
||||
self.assertEqual(json.load(fh), [{"url": "file://x"}])
|
||||
vm_dir = os.path.join(self.state_dir, "chv-abc123")
|
||||
self.assertEqual([p for p in os.listdir(vm_dir)
|
||||
if ".tmp" in p], [])
|
||||
|
||||
def test_orphan_registry_no_fixed_tmp(self):
|
||||
path = os.path.join(self.state_dir, "pool.json")
|
||||
_write_registry(path, {"chv-a": {"vm": "chv-a"}})
|
||||
with open(path) as fh:
|
||||
self.assertEqual(json.load(fh),
|
||||
{"chv-a": {"vm": "chv-a"}})
|
||||
self.assertEqual([p for p in os.listdir(self.state_dir)
|
||||
if ".tmp" in p], [])
|
||||
|
||||
# -- H1 附带:NO_PROXY ---------------------------------------------
|
||||
def _env_assignments(self, tokens):
|
||||
"""remote_command 返回的 env 赋值(KEY=VALUE 形式)。"""
|
||||
out = {}
|
||||
for tok in tokens:
|
||||
if tok == "env":
|
||||
continue
|
||||
if "=" in tok and not tok.startswith("'"):
|
||||
key, _, value = tok.partition("=")
|
||||
if key.isidentifier() or key in (
|
||||
"NO_PROXY", "no_proxy", "HTTPS_PROXY",
|
||||
"HTTP_PROXY", "https_proxy", "http_proxy"):
|
||||
out.setdefault(key, []).append(
|
||||
value.strip("'"))
|
||||
return out
|
||||
|
||||
def test_no_proxy_merged_with_egress(self):
|
||||
tokens = remote_command(
|
||||
["true"], egress_proxy="http://10.0.0.1:8888",
|
||||
extra_env={"NO_PROXY": "internal.example.com,10.0.0.0/8"})
|
||||
assigns = self._env_assignments(tokens)
|
||||
self.assertEqual(len(assigns.get("NO_PROXY", [])), 1)
|
||||
merged = assigns["NO_PROXY"][0]
|
||||
for part in ("internal.example.com", "10.0.0.0/8",
|
||||
"localhost", "127.0.0.1"):
|
||||
self.assertIn(part, merged)
|
||||
self.assertEqual(
|
||||
assigns["HTTPS_PROXY"], ["http://10.0.0.1:8888"])
|
||||
|
||||
def test_no_proxy_lowercase_merged_and_not_duplicated(self):
|
||||
tokens = remote_command(
|
||||
["true"], egress_proxy="http://10.0.0.1:8888",
|
||||
extra_env={"no_proxy": "gitea.tose.top",
|
||||
"NO_PROXY": "localhost"})
|
||||
assigns = self._env_assignments(tokens)
|
||||
self.assertEqual(len(assigns.get("NO_PROXY", [])), 1)
|
||||
self.assertIn("gitea.tose.top", assigns["NO_PROXY"][0])
|
||||
self.assertNotIn("no_proxy", assigns)
|
||||
# localhost 去重后只出现一次。
|
||||
self.assertEqual(
|
||||
assigns["NO_PROXY"][0].split(",").count("localhost"), 1)
|
||||
|
||||
def test_no_proxy_preserved_without_egress(self):
|
||||
tokens = remote_command(
|
||||
["true"], extra_env={"NO_PROXY": "a.example"})
|
||||
assigns = self._env_assignments(tokens)
|
||||
self.assertEqual(assigns.get("NO_PROXY"), ["a.example"])
|
||||
|
||||
def test_user_proxy_dropped_when_egress_set(self):
|
||||
tokens = remote_command(
|
||||
["true"], egress_proxy="http://10.0.0.1:8888",
|
||||
extra_env={"HTTPS_PROXY": "http://evil:1",
|
||||
"https_proxy": "http://evil:1",
|
||||
"HTTP_PROXY": "http://evil:1"})
|
||||
assigns = self._env_assignments(tokens)
|
||||
self.assertEqual(
|
||||
assigns["HTTPS_PROXY"], ["http://10.0.0.1:8888"])
|
||||
self.assertEqual(
|
||||
assigns["HTTP_PROXY"], ["http://10.0.0.1:8888"])
|
||||
self.assertNotIn("https_proxy", assigns)
|
||||
self.assertNotIn("http_proxy", assigns)
|
||||
|
||||
|
||||
# -- M4 供应链 ----------------------------------------------------------
|
||||
class LockfileTest(unittest.TestCase):
|
||||
"""合成 BUILDINFO 干跑:版本全是假值(如 git 9.9.9),
|
||||
只验“来源是 BUILDINFO、不是宿主现场”与钉死校验逻辑,
|
||||
真实 lockfile 产出仍待 V-9 真机构建。"""
|
||||
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.dir = self.tmp.name
|
||||
|
||||
def tearDown(self):
|
||||
self.tmp.cleanup()
|
||||
|
||||
def _synth_buildinfo(self, chromium="chromium_headless_shell-1248",
|
||||
pinned_rev="1248"):
|
||||
dpkg = ["git 9.9.9-9ubuntu9", "openssh-server 1:9.9",
|
||||
"openssh-client 1:9.9", "cloud-init 9.9",
|
||||
"python3 9.9", "python3-pip 9.9", "curl 9.9",
|
||||
"ca-certificates 9.9", "xz-utils 9.9",
|
||||
"fonts-wqy-microhei 9.9", "fontconfig 9.9",
|
||||
"systemd-container 9.9"]
|
||||
info = {
|
||||
"tag": "chv-guest-24.04-0.164.0-alpha.9-abc1234",
|
||||
"ubuntu": "24.04",
|
||||
"codex_version": "0.164.0-alpha.9",
|
||||
"repo_commit": "abc1234",
|
||||
"gt_version": "1.0.0",
|
||||
"built_at": "2026-11-07T00:00:00Z",
|
||||
"contents": ["git", "openssh-server", "openssh-client",
|
||||
"cloud-init", "python3", "python3-pip",
|
||||
"curl", "ca-certificates", "xz-utils",
|
||||
"fonts-wqy-microhei", "fontconfig", "codex",
|
||||
"codex-code-mode-host", "codex-resources/bwrap",
|
||||
"/app/host.py", "gt", "chv-sync",
|
||||
# 与 717efc4 真实 BUILDINFO.contents 对齐:
|
||||
# P2 三件套 + P3 authd 三件套 + 叶子模块 +
|
||||
# vsock_notify(缺 DECLARED_SOURCES 登记时
|
||||
# generate 记 unresolved、verify/check 失败,
|
||||
# 合成清单须覆盖才拦得住回归)。round-11
|
||||
# 真机修复(层 1)后叶子模块还含 errors.py
|
||||
# 与 stub __init__.py。
|
||||
"systemd-container", "chv-cell", "chv-broker",
|
||||
"chv-browser", "chv-authd",
|
||||
"git-credential-chv-authd", "chv-authd.service",
|
||||
"cloud/chv/authd.py", "cloud/chv/sentinel.py",
|
||||
"cloud/chv/errors.py", "cloud/chv/__init__.py",
|
||||
"vsock_notify",
|
||||
"playwright", "chromium-headless-shell"],
|
||||
"versions": {"dpkg": dpkg,
|
||||
"pip": ["playwright==9.9.9"],
|
||||
"gt": ["gt version 1.0.0"],
|
||||
"chromium": [chromium]},
|
||||
"chromium_pinned_rev": pinned_rev,
|
||||
"image_sha256": "0" * 64,
|
||||
}
|
||||
path = os.path.join(self.dir, "BUILDINFO.json")
|
||||
with open(path, "w") as fh:
|
||||
json.dump(info, fh)
|
||||
return path
|
||||
|
||||
def test_generate_uses_buildinfo_not_host(self):
|
||||
gen = _load_script("generate_lockfile.py")
|
||||
buildinfo = self._synth_buildinfo()
|
||||
lockfile = os.path.join(self.dir, "lockfile.json")
|
||||
self.assertEqual(gen.generate(buildinfo, lockfile), 0)
|
||||
with open(lockfile) as fh:
|
||||
data = json.load(fh)
|
||||
self.assertEqual(data["unresolved"], [])
|
||||
# 假值进 lockfile:来源是 BUILDINFO,不是宿主 dpkg。
|
||||
self.assertIn("git 9.9.9-9ubuntu9",
|
||||
data["versions"]["dpkg"])
|
||||
self.assertIn("playwright 9.9.9",
|
||||
data["versions"]["pip"])
|
||||
self.assertIn("chromium-headless-shell "
|
||||
"chromium_headless_shell-1248",
|
||||
data["versions"]["declared"])
|
||||
self.assertEqual(
|
||||
gen.verify(lockfile, buildinfo), 0)
|
||||
|
||||
def test_generate_verify_rejects_wrong_pinned_rev(self):
|
||||
gen = _load_script("generate_lockfile.py")
|
||||
buildinfo = self._synth_buildinfo(
|
||||
chromium="chromium_headless_shell-9999",
|
||||
pinned_rev="1248")
|
||||
lockfile = os.path.join(self.dir, "lockfile.json")
|
||||
self.assertEqual(gen.generate(buildinfo, lockfile), 0)
|
||||
# 生成本身不设门(只记版本),--verify 才拦钉死不符。
|
||||
self.assertEqual(gen.verify(lockfile, buildinfo), 1)
|
||||
|
||||
def test_check_lockfile_pinned_rev(self):
|
||||
check = _load_script("check_lockfile.py")
|
||||
ok = self._synth_buildinfo()
|
||||
self.assertEqual(
|
||||
check.main(["check_lockfile.py", ok]), 0)
|
||||
bad = self._synth_buildinfo(
|
||||
chromium="chromium_headless_shell-9999",
|
||||
pinned_rev="1248")
|
||||
self.assertEqual(
|
||||
check.main(["check_lockfile.py", bad]), 1)
|
||||
|
||||
def test_buildinfo_records_and_enforces_pin(self):
|
||||
buildinfo = _load_script("buildinfo.py")
|
||||
out = os.path.join(self.dir, "out")
|
||||
os.makedirs(out)
|
||||
for name, content in (
|
||||
("disk.raw", b"D"), ("vmlinux", b"K")):
|
||||
with open(os.path.join(out, name), "wb") as fh:
|
||||
fh.write(content)
|
||||
for name, lines in (
|
||||
("guest-versions-dpkg.txt",
|
||||
["git 9.9.9-9ubuntu9"]),
|
||||
("guest-versions-pip.txt",
|
||||
["playwright==9.9.9"]),
|
||||
("guest-versions-gt.txt", ["gt version 1.0.0"]),
|
||||
("guest-versions-chromium.txt",
|
||||
["chromium_headless_shell-1248"])):
|
||||
with open(os.path.join(out, name), "w") as fh:
|
||||
fh.write("\n".join(lines) + "\n")
|
||||
args = [out, "chv-guest-24.04-0.164.0-alpha.9-abc1234",
|
||||
"24.04", "0.164.0-alpha.9", "abc1234", "0" * 64,
|
||||
"1", "1248"]
|
||||
info = buildinfo.build(*args[0:7], "1248")
|
||||
self.assertEqual(info["chromium_pinned_rev"], "1248")
|
||||
self.assertEqual(buildinfo.main(["buildinfo.py"] + args), 0)
|
||||
wrong = list(args)
|
||||
wrong[-1] = "9999"
|
||||
self.assertEqual(
|
||||
buildinfo.main(["buildinfo.py"] + wrong), 1)
|
||||
# 未给钉死值:记 null,不强制。
|
||||
info2 = buildinfo.build(*args[0:7])
|
||||
self.assertIsNone(info2["chromium_pinned_rev"])
|
||||
|
||||
|
||||
class BuildScriptMinimalAppTest(unittest.TestCase):
|
||||
"""build.sh 干跑:/app 只剩 host.py(不用 cloud/ 也能
|
||||
干跑通过),Chromium 钉死值透传进 buildinfo 回显。"""
|
||||
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.dir = self.tmp.name
|
||||
self.bin_dir = os.path.join(self.dir, "bin")
|
||||
os.makedirs(os.path.join(self.bin_dir, "codex-resources"))
|
||||
for name in ("codex", "codex-code-mode-host"):
|
||||
p = os.path.join(self.bin_dir, name)
|
||||
with open(p, "w") as fh:
|
||||
fh.write("#!/bin/sh\necho \"codex-cli 0.164.0-alpha.9\"\n")
|
||||
os.chmod(p, 0o755)
|
||||
self.repo = os.path.join(self.dir, "repo")
|
||||
os.makedirs(self.repo)
|
||||
# 故意不建 cloud/:最小集收紧后 build.sh 不再要求它。
|
||||
with open(os.path.join(self.repo, "host.py"), "w") as fh:
|
||||
fh.write("# host\n")
|
||||
subprocess.run(["git", "init", "-q", self.repo],
|
||||
check=True, capture_output=True)
|
||||
subprocess.run(["git", "-C", self.repo, "config",
|
||||
"user.email", "t@t"], check=True,
|
||||
capture_output=True)
|
||||
subprocess.run(["git", "-C", self.repo, "config",
|
||||
"user.name", "t"], check=True,
|
||||
capture_output=True)
|
||||
subprocess.run(["git", "-C", self.repo, "add", "-A"],
|
||||
check=True, capture_output=True)
|
||||
subprocess.run(["git", "-C", self.repo, "commit", "-qm",
|
||||
"init"], check=True, capture_output=True)
|
||||
self.repo_commit = subprocess.run(
|
||||
["git", "-C", self.repo, "rev-parse", "--short", "HEAD"],
|
||||
check=True, capture_output=True, text=True).stdout.strip()
|
||||
self.script = str(ROOT / "chv-guest" / "build.sh")
|
||||
|
||||
def tearDown(self):
|
||||
self.tmp.cleanup()
|
||||
|
||||
def _run(self, args, env_extra=None):
|
||||
env = dict(os.environ)
|
||||
env.update({"CODEX_BIN_DIR": self.bin_dir,
|
||||
"UBUNTU_VERSION": "24.04",
|
||||
"CODEX_VERSION": "0.164.0-alpha.9",
|
||||
"REPO_COMMIT": self.repo_commit,
|
||||
"UBUNTU_IMAGE_SHA256": "0" * 64,
|
||||
"REPO_DIR": self.repo})
|
||||
if env_extra:
|
||||
env.update(env_extra)
|
||||
return subprocess.run(["bash", self.script] + args,
|
||||
check=False,
|
||||
capture_output=True, text=True, env=env)
|
||||
|
||||
def test_dry_run_without_cloud_dir_and_pin_echo(self):
|
||||
out_dir = os.path.join(self.dir, "out")
|
||||
r = self._run(["--dry-run", out_dir],
|
||||
{"CHROMIUM_HEADLESS_SHELL_REV": "1248"})
|
||||
self.assertEqual(r.returncode, 0, r.stderr)
|
||||
self.assertNotIn("cloud:/app", r.stdout)
|
||||
self.assertIn("host.py:/app", r.stdout)
|
||||
self.assertIn("1248", r.stdout)
|
||||
self.assertFalse(os.path.exists(
|
||||
os.path.join(out_dir, "disk.raw")))
|
||||
|
||||
def test_build_sh_text_has_no_cloud_copy(self):
|
||||
with open(self.script) as fh:
|
||||
text = fh.read()
|
||||
self.assertNotIn('"$REPO_DIR/cloud:/app/"', text)
|
||||
self.assertIn('"$REPO_DIR/host.py:/app/"', text)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -529,12 +529,31 @@ class BuildScriptTest(unittest.TestCase):
|
||||
for name in ("codex", "codex-code-mode-host"):
|
||||
p = os.path.join(self.bin_dir, name)
|
||||
with open(p, "w") as fh:
|
||||
fh.write("#!/bin/sh\n")
|
||||
# B-09:build.sh 会核对 codex --version 含 CODEX_VERSION,
|
||||
# 假二进制须报出真实版本号。
|
||||
fh.write("#!/bin/sh\necho \"codex-cli 0.164.0-alpha.9\"\n")
|
||||
os.chmod(p, 0o755)
|
||||
self.repo = os.path.join(self.dir, "repo")
|
||||
os.makedirs(os.path.join(self.repo, "cloud"))
|
||||
with open(os.path.join(self.repo, "host.py"), "w") as fh:
|
||||
fh.write("# host\n")
|
||||
# B-09:REPO_COMMIT 须等于仓库 HEAD 短 hash 且工作树干净,
|
||||
# 假仓库须是真 git 仓库。
|
||||
subprocess.run(["git", "init", "-q", self.repo],
|
||||
check=True, capture_output=True)
|
||||
subprocess.run(["git", "-C", self.repo, "config",
|
||||
"user.email", "t@t"], check=True,
|
||||
capture_output=True)
|
||||
subprocess.run(["git", "-C", self.repo, "config",
|
||||
"user.name", "t"], check=True,
|
||||
capture_output=True)
|
||||
subprocess.run(["git", "-C", self.repo, "add", "-A"],
|
||||
check=True, capture_output=True)
|
||||
subprocess.run(["git", "-C", self.repo, "commit", "-qm",
|
||||
"init"], check=True, capture_output=True)
|
||||
self.repo_commit = subprocess.run(
|
||||
["git", "-C", self.repo, "rev-parse", "--short", "HEAD"],
|
||||
check=True, capture_output=True, text=True).stdout.strip()
|
||||
self.script = str(Path(__file__).resolve().parents[2]
|
||||
/ "chv-guest" / "build.sh")
|
||||
|
||||
@@ -546,7 +565,7 @@ class BuildScriptTest(unittest.TestCase):
|
||||
env.update({"CODEX_BIN_DIR": self.bin_dir,
|
||||
"UBUNTU_VERSION": "24.04",
|
||||
"CODEX_VERSION": "0.164.0-alpha.9",
|
||||
"REPO_COMMIT": "test123",
|
||||
"REPO_COMMIT": self.repo_commit,
|
||||
"REPO_DIR": self.repo})
|
||||
if env_extra:
|
||||
env.update(env_extra)
|
||||
|
||||
@@ -359,21 +359,46 @@ class LockfileScriptTest(unittest.TestCase):
|
||||
spec.loader.exec_module(mod)
|
||||
return mod
|
||||
|
||||
def test_generate_reads_real_dpkg_version(self):
|
||||
def test_generate_reads_guest_versions(self):
|
||||
# C 轨纠正(报告 §3.2):版本唯一来源是 BUILDINFO.versions
|
||||
#(客机读回),不是构建宿主的 dpkg/pip;codex 类声明项
|
||||
# 取 BUILDINFO 的 codex_version / repo_commit。
|
||||
import json as _json
|
||||
mod = self._load_script()
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
bi = os.path.join(tmp, "bi.json")
|
||||
out = os.path.join(tmp, "lock.json")
|
||||
with open(bi, "w") as fh:
|
||||
_json.dump({"tag": "t", "contents": ["coreutils"]},
|
||||
_json.dump({"tag": "t",
|
||||
"contents": ["coreutils", "codex"],
|
||||
"codex_version": "0.164.0-alpha.9",
|
||||
"versions": {"dpkg": ["coreutils 9.4-3"],
|
||||
"pip": []}},
|
||||
fh)
|
||||
self.assertEqual(mod.generate(bi, out), 0)
|
||||
lock = _json.load(open(out))
|
||||
self.assertEqual(lock["tag"], "t")
|
||||
self.assertEqual(lock["unresolved"], [])
|
||||
self.assertTrue(lock["versions"]["dpkg"][0].startswith(
|
||||
"coreutils "))
|
||||
self.assertEqual(lock["versions"]["dpkg"],
|
||||
["coreutils 9.4-3"])
|
||||
self.assertEqual(lock["versions"]["declared"],
|
||||
["codex 0.164.0-alpha.9"])
|
||||
|
||||
def test_generate_does_not_query_host(self):
|
||||
# 宿主上装着 coreutils,但 BUILDINFO.versions 里没有 →
|
||||
# unresolved,绝不回落到宿主查询(旧方法不成立)。
|
||||
import json as _json
|
||||
mod = self._load_script()
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
bi = os.path.join(tmp, "bi.json")
|
||||
out = os.path.join(tmp, "lock.json")
|
||||
with open(bi, "w") as fh:
|
||||
_json.dump({"tag": "t", "contents": ["coreutils"],
|
||||
"versions": {"dpkg": [], "pip": []}},
|
||||
fh)
|
||||
self.assertEqual(mod.generate(bi, out), 0)
|
||||
lock = _json.load(open(out))
|
||||
self.assertEqual(lock["unresolved"], ["coreutils"])
|
||||
|
||||
def test_verify_fails_on_unresolved(self):
|
||||
import json as _json
|
||||
|
||||
@@ -227,7 +227,15 @@ class PreflightTest(ChvCase):
|
||||
self.assertIn("内核", text)
|
||||
|
||||
def test_dispatch_missing_dependency_does_not_call_runner(self):
|
||||
ex = self.make_exec(binary="/tmp/no-chv")
|
||||
# P1 常驻默认开:这里显式关常驻,测的仍是冷启动
|
||||
# preflight 路径(缺依赖 → 零 runner 调用)。不开
|
||||
# 关的话,_resident_acquire 会先试建池,wake 失败
|
||||
# 后的清理 shutdown 会多一次 ch-remote 调用——那是
|
||||
# 建池失败清理的既定行为(见 P1 的 provision 失败
|
||||
# 回落测试),不是本测试要守的“缺依赖零调用”。
|
||||
ex = self.make_exec(
|
||||
binary="/tmp/no-chv",
|
||||
env_extra={"CHV_RESIDENT_ENABLED": "0"})
|
||||
with self.assertRaises(ChvFailure):
|
||||
ex.dispatch_task(self._task())
|
||||
self.assertEqual(self.calls, [])
|
||||
@@ -243,9 +251,10 @@ class ConfigCommandTest(ChvCase):
|
||||
self.assertEqual(cfg["cpus"], {"boot_vcpus": 2,
|
||||
"max_vcpus": 2})
|
||||
self.assertEqual(cfg["memory"]["size"], 2 * 1024 ** 3)
|
||||
self.assertEqual(cfg["kernel"]["path"], self.kernel)
|
||||
self.assertEqual(cfg["disks"], [{"path": disk}])
|
||||
self.assertEqual(cfg["cmdline"]["args"],
|
||||
self.assertEqual(cfg["payload"]["kernel"], self.kernel)
|
||||
self.assertEqual(cfg["disks"],
|
||||
[{"path": disk, "image_type": "Raw"}])
|
||||
self.assertEqual(cfg["payload"]["cmdline"],
|
||||
"console=hvc0 root=/dev/vda1 rw")
|
||||
|
||||
def test_memory_parsing(self):
|
||||
@@ -346,7 +355,10 @@ class WakeApiTest(ChvCase):
|
||||
self.assertEqual(kinds[1], ("PUT", "/vm.create"))
|
||||
self.assertEqual(kinds[2], ("PUT", "/vm.boot"))
|
||||
create_payload = self.api_calls[1][2]
|
||||
self.assertEqual(create_payload["kernel"]["path"], self.kernel)
|
||||
# CHV 54:内核走 payload.kernel 纯字符串(旧顶层
|
||||
# kernel:{path} 会被忽略、vm.boot 400)。
|
||||
self.assertEqual(create_payload["payload"]["kernel"], self.kernel)
|
||||
self.assertEqual(create_payload["disks"][0]["image_type"], "Raw")
|
||||
self.assertTrue(create_payload["disks"][0]["path"].endswith(
|
||||
"disk.raw"))
|
||||
events = [e for e, _, _, _ in self.audit]
|
||||
@@ -534,7 +546,12 @@ class WarmPoolTest(ChvCase):
|
||||
# -- dispatch / goal(wake → 跑 → finally 关) ---------------------------
|
||||
class DispatchTest(ChvCase):
|
||||
def test_dispatch_wakes_runs_shuts_down(self):
|
||||
ex = self.make_exec()
|
||||
# 本测试钉的是冷启动 dispatch 序列(wake chv-abc123
|
||||
# → 跑 → finally shutdown);P1 起常驻默认开,须显
|
||||
# 式关常驻才走这条路。常驻 dispatch 由
|
||||
# test_chv_muse_p1_resident 覆盖。
|
||||
ex = self.make_exec(
|
||||
env_extra={"CHV_RESIDENT_ENABLED": "0"})
|
||||
out = ex.dispatch_task(self._task())
|
||||
self.assertEqual(out["executor"], "cloud-hypervisor")
|
||||
self.assertEqual(out["status"], "completed")
|
||||
|
||||
@@ -2575,6 +2575,25 @@ class Worker:
|
||||
}
|
||||
return "sub2api", env
|
||||
|
||||
def _gitea_sync_env(self, task):
|
||||
"""Gitea 同步变量(CHV guest-env 集成):任务 gitea_sync
|
||||
覆盖宿主 CHV_GITEA_* 配置(见 gitea_sync.extract_...)。
|
||||
|
||||
只读本任务与本 worker 的 self._env(默认 os.environ),
|
||||
绝不硬编码任何值、不读其它任务、不写日志。返回空 dict
|
||||
即本次启动不同步;完整性校验不在这里做——执行器
|
||||
wake_vm 在起 VMM 前 fail-closed(gitea_sync_incomplete),
|
||||
worker 只负责把变量带进 extra_env。
|
||||
注意:tasks 表没有 JSON/元数据列,DB 往返会丢任务 dict
|
||||
的 gitea_sync;生产主来源是宿主 CHV_GITEA_* 环境变量。"""
|
||||
from cloud.chv.gitea_sync import (
|
||||
config_gitea_sync_env,
|
||||
extract_gitea_sync_env,
|
||||
)
|
||||
return extract_gitea_sync_env(
|
||||
task=task,
|
||||
config_sync=config_gitea_sync_env(self._env))
|
||||
|
||||
def _ensure_gateway(self, task):
|
||||
"""任务 dict 上的网关缓存:已算过直接取,否则算一次并存回。
|
||||
|
||||
@@ -4684,6 +4703,14 @@ class Worker:
|
||||
message="worker exec start: %s" % _exec_detail)
|
||||
try:
|
||||
gateway_provider, gateway_env = self._ensure_gateway(task)
|
||||
if isinstance(self.executor, CloudHypervisorExecutor):
|
||||
# CHV guest-env 集成:Gitea 同步变量随 extra_env
|
||||
# 进 wake_vm(密钥走 seed,boot 变量也走 seed;
|
||||
# 见 gitea_sync)。只对 CHV 合并——Docker/Local
|
||||
# 的密钥处理口径不同(-e/子进程环境),不在本轮
|
||||
# 改动,避免 GITEA_TOKEN 经其它通道外泄。
|
||||
gateway_env = _merge_env(
|
||||
gateway_env, self._gitea_sync_env(task))
|
||||
if self._use_popen():
|
||||
if goal_mode and task.get("_goal_resume_breakpoint"):
|
||||
# G10 断点:无 --goal-text,不开新循环。
|
||||
|
||||
@@ -57,6 +57,60 @@
|
||||
- 重启条件:将来要重新接线,必须先实现任务盘重供给并补
|
||||
“任务盘来源”不变量测试,否则不得把 wired 改成 True。
|
||||
|
||||
## ADR-17 常驻 VM 池取代 WarmPool 接线(feat-chv-muse-p1)
|
||||
|
||||
- 决策:不再接线 WarmPool,改走新建的生产模块
|
||||
`cloud/chv/resident_pool.py`(ResidentPool):每用户一台
|
||||
常驻 VM,任务到时 resume 已暂停的 VM,跑完 pause,不
|
||||
销毁;池空或任何一步失败都回落冷启动。WarmPool 实验模块
|
||||
一字不动,`wired_to_dispatch` 仍恒 False,ADR-5 的定案
|
||||
(预热盘是干净快照盘、不是任务盘)仍然成立、只是不再
|
||||
是接线障碍——常驻 VM 的盘本来就是每用户持久工作区盘,
|
||||
不是每任务新盘。
|
||||
- 隔离单元变了:常驻路径下隔离单元从“每任务”变成“每用户”
|
||||
——同一用户的任务共享同一台常驻 VM 与同一块工作区盘,
|
||||
每任务新盘的不变量在常驻路径上不再成立。冷启动路径
|
||||
(每任务新盘)保留作回落与隔离严格场景。当前是单租户,
|
||||
这个取舍可接受;多租户重审前不得把常驻路径当多租户隔离。
|
||||
任务 dict 目前无 user_id,全落到 "default" 键;user_id
|
||||
出现时按键隔离,release 绝不把用过的 VM 放回给另一用户。
|
||||
- 接线:`_run_with_vm` 先常驻 acquire,命中走常驻路径
|
||||
(finally 里 release=pause),失败/未启用走原冷启动路径
|
||||
(finally shutdown 不变,冷启动本体逐字移到 `_run_cold`);
|
||||
dispatch/run/goal/goal_resume/断点续跑都走 `_run_with_vm`,
|
||||
签名不动。常驻默认开,`CHV_RESIDENT_ENABLED=0` 可关;
|
||||
开关用 getattr/env 惰性读,`__init__` 不动。池空时惰性
|
||||
provision 一次(wake + pause)再取,仍无则冷启动——首个
|
||||
任务自举常驻 VM,不依赖 worker 启动时预热。
|
||||
- 常驻执行细节:`_run_guest`/`_run_guest_goal` 一律走常驻伪
|
||||
任务 `{"id": "resident-<用户>"}` 传参(SSH known_hosts、
|
||||
api-socket、密钥投递记录、VMM 登记都按常驻 VM 名算),
|
||||
返回后把 `task_id` 改回真实任务 id;中途死亡判据同样按
|
||||
常驻伪任务查登记进程。带密钥 extra_env 的任务不走常驻
|
||||
(seed 是建池时定的,密钥只能冷启动随 seed 投递),直接
|
||||
冷启动。同一用户同一时间只跑一个常驻任务,并发第二个
|
||||
任务回落冷启动。
|
||||
- Registry:`state_dir/resident-pool.json`,文件 IO 与损坏
|
||||
语义复用 orphans(原子写、损坏 quarantine、单条目非法
|
||||
只丢该条);重启后读到的条目一律 orphan 不复用、只清理。
|
||||
空闲回收:paused 超 `CHV_RESIDENT_IDLE_MINUTES`(默认 0=
|
||||
不自动关机)时 hibernate(pause + snapshot + shutdown)
|
||||
并删条目,快照文件留作恢复物,restore 是后续工作。
|
||||
- 依赖方向(ADR-12 的禁令全部保留):executor 仍绝不 import
|
||||
orphans/experimental_pool(常驻池用方法内 import 拿
|
||||
ResidentPool);resident_pool → orphans(只借 registry
|
||||
文件 IO,与 experimental_pool 同款方向);worker 启动/
|
||||
关闭调 `worker_adapter.reconcile_resident_pool` facade,
|
||||
它只读 resident-pool.json 做清理,不构造 ResidentPool。
|
||||
`reconcile_warm_pool` 不动。
|
||||
- 诚实边界(未验证,不宣称):resume 耗时、到 SSH 5 秒内、
|
||||
气球压内存——无真机验证(第三棒占着 WSL 测试机),一律
|
||||
记入真机待验证清单;resume 失败时 `wait_guest_ready` 的
|
||||
失败路径会 stop_vm(含删工作区盘),该路径的工作区以
|
||||
Gitea 同步为准;pause 失败时条目留在 running(不删不销
|
||||
毁),worker 关闭时 reconcile 清理,在此之前该用户只走
|
||||
冷启动。
|
||||
|
||||
## ADR-6 Docker goal 解析分叉:共享正典 + 同步守卫(本轨定案)
|
||||
|
||||
- 决策:Local/CHV 共用 parse_goal_result 正典;Docker goal_task
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
# 对齐 Muse P1 差距表:常驻 VM + WarmPool + 气球 + vsock
|
||||
|
||||
依据:`~/workspace/research_notes/meta-muse-ai-architecture-20261101-0755/report.md`
|
||||
(官方以 Meta security.muse.ai 2026-09-08 为准;第三方拆解只作参考,
|
||||
逆向细节不得写成官方架构)。Muse 模型一句话:每用户一台常驻 Secure VM,
|
||||
闲置靠气球压内存,宿主–客机走 vsock 通知通道。
|
||||
|
||||
本轨范围只限 P1:常驻 VM、WarmPool 扶正、内存气球、vsock 通道。
|
||||
不碰 P2(客机 nspawn 单元、浏览器子代理独立)、不碰 P3(authd 替身凭证、
|
||||
Sentinel 式出口审批)。分支 `feat-chv-muse-p1`(从 `feat-chv-cover100`
|
||||
分出,基线 HEAD `06c3e17`)。
|
||||
|
||||
## 1. 现状 vs Muse
|
||||
|
||||
| 项 | 现状(实测/代码为准) | Muse 目标 | 本轨做法 |
|
||||
|---|---|---|---|
|
||||
| VM 生命周期 | 1 任务 = 1 VM,用完销毁;冷启动到 SSH 实测 11 秒 | 每用户一台常驻 Secure VM(hatchling),任务来了直接用 | 新增常驻服务:每用户一台常驻 VM(暂停态),任务到时 resume;池空回落冷启动(11 秒路径不动) |
|
||||
| 派发查池 | `_run_with_vm` 绝不查池,`warm_pool_wired_to_dispatch` 恒 False;worker 派发前查池未接线 | 常驻 VM 就是派发目标,无冷启动 | `_run_with_vm` 接常驻路径:先 acquire 常驻 VM(resume + 等 SSH 就绪),无可用时回落 `wake_vm` |
|
||||
| 预热池状态 | `experimental_pool.WarmPool` 纯实验,生产禁 import(ADR-12);ADR-5 定案:预热盘是干净快照盘、不是任务盘,接线须先实现任务盘重供给 | —(Muse 无此概念:常驻 VM 的盘就是用户工作区盘) | WarmPool 实验模块不动(仍禁生产 import);新建生产模块 `resident_pool.py`。常驻 VM 的盘是**每用户持久工作区盘**,不是每任务新盘 |
|
||||
| 内存 | 固定 2G,无气球 | 气球 + 空闲页上报,闲置 8G 压到约 2G 工作集 | CHV 配置加 virtio-balloon + free-page-reporting(默认开,可关) |
|
||||
| 宿主–客机通道 | 只 SSH;executor 文档明确“只选 SSH,不做 vsock 第二套”(旧决策) | vsock 通知通道(另有 Unix socket + SO_PEERCRED 的单元内 IPC,属 P2/P3 层) | vsock 只做最小 notify(就绪/心跳上报),与 SSH 并存;SSH 仍是命令通道与就绪判定,host key 钉死不动 |
|
||||
| 池持久化/孤儿 | WarmPool registry `state_dir/warm-pool.json` 已有(原子写、损坏 quarantine、重启后条目一律 orphan 不复用) | — | 常驻池复用同款 registry 语义,另建 `resident-pool.json`;worker 启动/关闭走 `orphans` facade 清理,不自动复用上一世条目 |
|
||||
|
||||
## 2. 改了什么(本轨交付物)
|
||||
|
||||
- `cloud/chv/resident_pool.py`(新):常驻池。状态机 cold → provisioning →
|
||||
paused → running → paused;`acquire(user)` resume + SSH 就绪,
|
||||
`release(user)` pause 不销毁;空闲回收策略(pause 后常驻,超闲置阈值
|
||||
关机+快照,阈值可配,默认不开自动关机);registry 原子落盘、损坏
|
||||
quarantine、重启后不复用。
|
||||
- `cloud/chv/executor.py`:`_run_with_vm` 接常驻路径(默认开,可配关;
|
||||
常驻失败一律回落冷启动,绝不因池坏而丢任务);`_create` payload 加
|
||||
balloon/vsock 设备。
|
||||
- `cloud/chv/config.py`:`CHV_RESIDENT_IDLE_MINUTES`、`CHV_BALLOON`、
|
||||
`CHV_VSOCK`(开关 + CID),非法值 ValueError,不静默 fallback。
|
||||
- `cloud/chv/worker_adapter.py`:常驻池 reconcile facade(接 `orphans`)。
|
||||
- `cloud/chv/vsock.py`(新,宿主侧):vsock notify 客户端/监听最小实现。
|
||||
- `cloud/chv/guest/vsock_notify.py`(新,客内脚本):连宿主 CID 2 发
|
||||
ready/心跳。**只写脚本文件,不改 `chv-guest/build.sh`**——镜像集成
|
||||
是 P2 的范围,本轨只留接入说明。
|
||||
- `cloud/tests/test_chv_muse_p1_*.py`(新):常驻池、气球/vsock 配置与
|
||||
payload、回落语义的单测;全量 cloud 套件全过。
|
||||
- `docs/chv-architecture.md`:ADR-5/ADR-12 补充新决策(常驻路径取代
|
||||
WarmPool 接线问题),旧定案不删、只追加。
|
||||
|
||||
## 3. 诚实边界(本轮不宣称的东西)
|
||||
|
||||
1. **隔离单元变了**:常驻路径下隔离单元从“每任务”变成“每用户”——同一
|
||||
用户的任务共享同一台常驻 VM 与同一块工作区盘,每任务新盘的不变量在
|
||||
常驻路径上不再成立。冷启动路径(每任务新盘)保留作回落与隔离严格场景。
|
||||
当前是单租户,这个取舍可接受;多租户重审前不得把常驻路径当多租户隔离。
|
||||
2. **性能数字全是待验证目标,不是结果**:resume 1–2 秒、到 SSH 5 秒内、
|
||||
气球压内存——本轮无真机验证(第三棒 V-1~V-9 占着 WSL 测试机),一律
|
||||
记入“真机待验证清单”,不在代码/文档里写成已测得。
|
||||
3. **气球 API 形态须核对**:CHV 54 的 balloon/vsock payload 字段以
|
||||
Cloud Hypervisor 官方文档/二进制为准;本机无 /dev/kvm,无法本地起
|
||||
VM 校验,只做配置层 + payload 单测。
|
||||
4. **vsock 端到端未通**:本轮只做到宿主侧 + 客内脚本;客内脚本进镜像、
|
||||
真机收发验证都待 P2 集成后、测试机空闲时再做。SSH 不动,vsock 坏了
|
||||
不影响任务。
|
||||
5. **WarmPool 实验模块不动**:不把它翻成生产模块,不改
|
||||
`wired_to_dispatch`;常驻是新模块、新路径,旧实验代码与测试原样保留。
|
||||
|
||||
## 4. 恢复收尾(分离后在本分支修掉的回归)
|
||||
|
||||
分离时 chv 子集 3 失败 + 1 错误,全量另有 2 失败,均已修
|
||||
掉、未删测试、未改断言:
|
||||
|
||||
- **VmDeathTest 两处**:根因不是生产判定错——探针实证,
|
||||
常驻 VM 真死时 `_run_on_resident` 两态都正确判
|
||||
interrupted;错的是旧测试的死亡模拟:它杀
|
||||
`ex._vms["chv-abc123"]`,P1 常驻路由下该 VM 从未 wake、
|
||||
登记里只有 `chv-resident-default`,模拟是空操作。修
|
||||
法:模拟器改杀 dispatch 时已登记的 VM(即实际跑命令
|
||||
的那台),断言不动。
|
||||
- **test_cloud_hypervisor 两处冷路径测试**
|
||||
(`test_dispatch_wakes_runs_shuts_down`、
|
||||
`test_dispatch_missing_dependency_does_not_call_runner`):
|
||||
P1 把常驻改成默认开后,它们测的冷启动序列不再是默认
|
||||
路径;修法:两测试显式 `CHV_RESIDENT_ENABLED=0`,
|
||||
继续钉冷启动行为,断言不动。常驻 dispatch 由 P1 新测
|
||||
试覆盖。附带说明:建池 wake 失败后的清理 shutdown 是
|
||||
既定行为(P1 回落测试钉住),不是回归。
|
||||
- **超 50 行**:`_balloon_vsock_payload`(54)拆出
|
||||
`_resolve_vsock_socket`;`vsock.listen_notify`(55)
|
||||
拆出 `_bind_notify_listener`;行为不动。
|
||||
- **mypy**:`resident_pool.py:208` 加 None 守卫后干净;
|
||||
`executor.py` 的 `out` 未注解错与 `marker_tools.py:142`
|
||||
是基线 06c3e17 已有(`git show` 核对,P1 diff 零行碰
|
||||
`out`),如实记录、未动。
|
||||
@@ -0,0 +1,162 @@
|
||||
# 对齐 Muse P2 差距表:客机 nspawn 单元 + 浏览器子代理独立
|
||||
|
||||
依据:`~/workspace/research_notes/meta-muse-ai-architecture-20261101-0755/report.md`
|
||||
(官方以 Meta security.muse.ai 2026-09-08 为准;第三方拆解只作参考,
|
||||
逆向细节不得写成官方架构)。Muse 模型一句话:VM 内 systemd-nspawn
|
||||
运行时单元(Hatch)跑代理,单元 root 映射为 host 侧非特权用户,自有
|
||||
虚拟网络接口,过滤 syscall(如禁用 io_uring),受限能力(如无
|
||||
CAP_SYS_PTRACE、无 CAP_NET_ADMIN);浏览器任务由专门浏览器子代理
|
||||
完成,只见页面无障碍树快照、不见原始 DOM,不能在页面上下文跑 JS,
|
||||
CDP 由单元外 broker 管理。
|
||||
|
||||
本轨范围只限 P2:客机镜像层——systemd-nspawn 运行时单元 + 浏览器
|
||||
子代理独立。不碰 P1(执行器 WarmPool/气球/vsock,常驻 VM)、不碰 P3
|
||||
(authd 替身凭证、Sentinel 式出口审批)。分支 `feat-chv-muse-p2`
|
||||
(从 `feat-chv-cover100` 分出,基线 HEAD `06c3e17`)。
|
||||
|
||||
## 1. 现状 vs Muse
|
||||
|
||||
| 项 | 现状(实测/代码为准) | Muse 目标 | 本轨做法 |
|
||||
|---|---|---|---|
|
||||
| 代理运行位置 | codex 直接跑在客机用户态(执行器经 SSH 以 `chv` 身份调 `python3 /app/host.py`,它再起 `codex app-server`);无内层隔离 | 代理跑在 systemd-nspawn 运行时单元内 | `chv-cell run -- python3 /app/host.py --state … --codex-home … <cmd>`:单命令单元(`--as-pid2`,不 boot 第二个 systemd);单元即沙箱 |
|
||||
| 单元 root 映射 | 无此概念(`chv` 就是普通用户) | 单元 root 映射为 host 侧非特权用户 | `--private-users=1000:65536`:单元内 uid 0 → 客机侧 `chv`(uid 1000)。效果:workspace/`~/.codex`(`chv` 所有)单元内仍可写;host-root 0600 文件(如 `/etc/chv/guest-env`)在单元内显示为 overflow uid、不可读;再加 `--inaccessible=/etc/chv` 纵深防御 |
|
||||
| 单元根文件系统 | 无(直接用客机根文件系统) | 单元自有根文件系统(完整 Debian 镜像),与 host 系统分离 | **本轮不做独立根文件系统**:单元复用客机根文件系统(`--directory=/`)。独立 minbase 根文件系统约 200–300MB,另要把 codex/playwright 依赖再装一份,会顶破 qcow2 ≤1GB 验收。文件隔离本轮是“读多写少 + 遮蔽”:`/usr`、`/app` 等 host-root 所有路径单元内只读不可写,`/etc/chv`、playwright 包与浏览器二进制 `--inaccessible` 遮掉,可写路径只有显式 bind(workspace、`~/.codex`、`/run/chv`)与 tmpfs(`/tmp`)。完整独立根文件系统列为下一阶段,需先重定镜像体积预算 |
|
||||
| syscall 过滤 | 无 | 过滤 syscall,如禁用 io_uring | `--system-call-filter=@system-service ~io_uring_setup ~io_uring_enter ~io_uring_register` |
|
||||
| 能力 | 客机 `chv` 是普通用户,但 codex 的 bwrap 沙箱在单元内另有问题(见下) | 无 CAP_SYS_PTRACE、无 CAP_NET_ADMIN | `--drop-capability=CAP_SYS_ADMIN,CAP_SYS_PTRACE,CAP_NET_ADMIN,CAP_NET_RAW,CAP_SYS_MODULE,CAP_SYS_RAWIO,CAP_SYS_BOOT,CAP_MKNOD` + `--no-new-privileges` |
|
||||
| 单元内 codex 沙箱 | `host.py` 默认 `sandbox="workspace-write"`(bwrap) | 单元本身就是边界 | 单元内 codex 用 `danger-full-access`:bwrap 的 `--proc /proc` 挂载在嵌套命名空间里是已知走不通的形状(见 AGENTS.md 内层 proc 定位),单元内再套 bwrap 只会假隔离。执行器侧改 `--sandbox` 透传是 P1 的文件,本轨只定义接口(见 §4),不改 `executor.py` |
|
||||
| 单元网络 | 无(与客机共享网络命名空间) | 单元自有虚拟网络接口 | `--network-veth`:宿主侧 `ve-chv-cell`(10.200.0.1/24)、单元侧 `host0`(10.200.0.2/24,默认路由经 10.200.0.1),客机侧 NAT 出站(MASQUERADE 走客机默认路由的上行口,动态发现,不写死口名)。单元内配网经 nsenter 进单元 init 的 netns 做(`chv-cell.py` 内实现)。veth 建不起来即大声失败,不静默回落共享网络;调试逃生口 `CHV_CELL_NETWORK=shared` 须显式设置、打日志,绝不默认 |
|
||||
| 单元内外 IPC | 无(SSH 直达,host key 钉死) | Unix socket + SO_PEERCRED + 对端 ACL 的最小权限 IPC | `chv-broker`(客机侧、单元外):Unix socket `/run/chv/broker.sock`(bind 进单元)。SO_PEERCRED 校验:对端 uid 须在单元映射范围 1000–66535,且对端 pid 的 pid ns 须与 `/run/chv/cell.pid` 记的单元 init 同 ns(cell.pid 不存在时只做 uid 检查,并打日志)。操作白名单:`ping`、`sync pull|push|status`(仓目录只允许 `~/workspace` 下,`chv-sync` 以 `chv` 身份跑,不进 shell)、`browser …`(转发给浏览器子代理)。其余一律拒绝 |
|
||||
| 浏览器 | Playwright headless-shell 由 agent 在客机内直接跑,能读原始 DOM、能跑 JS | 浏览器任务由专门浏览器子代理完成,只见无障碍树快照,不见原始 DOM,不能在页面上下文跑 JS,CDP 由单元外 broker 管理 | `chv-browser`(客机侧、单元外独立进程,用户 `chv-browser`):Unix socket `/run/chv-browser/browser.sock`,操作只有 `open|snapshot|screenshot|close`。`open`/`snapshot` 返回标题、URL 与 Playwright `Page.aria_snapshot()` 的 ARIA 快照字符串(截断到 200 节点行/64KB);`screenshot` 落盘到 `/run/chv/shots/`(bind 进单元,单元只读路径取图)。**不暴露** `evaluate`、`content`(原始 DOM)、DevTools。单元内 playwright 包与浏览器二进制用 `--inaccessible` 遮掉,单元内跑不起自己的浏览器,只能走 broker |
|
||||
| 凭证 | `/etc/chv/guest-env`(0600 root)+ `~/.git-credentials`(0600 chv),codex 在客机内直接可读后者 | 凭证由 authd 保管,代理只见替身令牌 | 本轨只做“单元看不见”:`/etc/chv` 与 `~/.git-credentials` 都 `--inaccessible`,同步走 broker 的 `sync` 操作(broker 在单元外跑 `chv-sync`,token 不进单元)。替身化本身是 P3,不在本轨冒充已做 |
|
||||
| 客机启动同步 | `chv-sync boot` 由 cloud-init runcmd 以 root 跑,clone 到 `~/workspace` | — | 不动:boot clone 仍在单元外做;单元内只经 broker 做 pull/push/status |
|
||||
|
||||
## 2. 改了什么(本轨交付物)
|
||||
|
||||
- `chv-guest/chv-cell.py`(新,标准库):单元启动器。`build_nspawn_argv()` 纯函数产
|
||||
nspawn 参数表(单元映射、能力丢弃、syscall 过滤、遮蔽、bind、tmpfs、veth、
|
||||
`--as-pid2`、`CHV_CELL=1`);`configure_cell_network()` 产配网命令表(宿主侧
|
||||
ve 接口 + NAT + nsenter 进单元配 host0);`chv-cell run -- <cmd>` 起单元、
|
||||
写 `/run/chv/cell.pid`、等退出码、清理 NAT 规则;`chv-cell argv` 只打印
|
||||
参数表、不启动(干检查用)。
|
||||
- `chv-guest/chv-broker.py`(新,标准库):单元外 IPC broker。Unix socket
|
||||
`/run/chv/broker.sock`,SO_PEERCRED + pid ns 双重校验,白名单操作见 §1,
|
||||
`sync` 调 `chv-sync`(`runuser -u chv`,list argv、不进 shell),`browser`
|
||||
转发 `chv-browser.sock`。
|
||||
- `chv-guest/chv-browser.py`(新,标准库 + playwright 懒加载):浏览器子代理。
|
||||
Unix socket `/run/chv/browser.sock`,操作 `open|snapshot|screenshot|close`,
|
||||
只返回无障碍树快照(截断)+ 标题/URL/截图路径;无 `evaluate`/`content`。
|
||||
- `chv-guest/build.sh`:apt 加 `systemd-container`、`iproute2`、`iptables`;
|
||||
拷入 `chv-cell.py`/`chv-broker.py`/`chv-browser.py`(`--copy-in` + chmod);
|
||||
建系统用户 `chv-browser`;装 `chv-broker.service`、`chv-browser.service`
|
||||
并 `systemctl enable`;`/etc/tmpfiles.d/chv.conf` 建 `/run/chv` 与
|
||||
`/run/chv/shots`。P1 交接:`cloud/chv/guest/vsock_notify.py` 若在仓库里
|
||||
存在则一并拷入(条件式,不存在时只警告、不失败——P1 文件在本分支尚无,
|
||||
合并后才有)。
|
||||
- `chv-guest/buildinfo.py`:`BASE_CONTENTS` 加 `systemd-container`、
|
||||
`chv-cell`、`chv-broker`、`chv-browser`;`build()` 支持可选 `extra_contents`
|
||||
(vsock 接入时用,不改现有调用签名)。
|
||||
- `cloud/tests/test_chv_muse_p2.py`(新):单元参数表、broker 校验与白名单、
|
||||
浏览器 API schema 与快照截断、build.sh 干跑含新步骤、buildinfo 新内容项。
|
||||
全部用假 socket/假进程,绝不连真实网络、不起真实 VM。
|
||||
- `docs/chv-production-readiness.md`:补 P2 一节(单元隔离验收与待验证项),
|
||||
旧定案不删、只追加。
|
||||
|
||||
## 3. 诚实边界(本轮不宣称的东西)
|
||||
|
||||
1. **没有独立根文件系统**:单元与客机共享根文件系统(只读挂载 + 遮蔽 +
|
||||
bind),不是 Muse 的独立 Debian 镜像。一个有决心的单元内进程仍能读到
|
||||
客机侧 world-readable 的文件内容(如 `/app/host.py`、系统库)。真正
|
||||
的文件系统分离要等下一阶段的独立根文件系统 + 体积重预算。
|
||||
2. **SO_PEERCRED 的 uid 检查不区分单元 root 与客机 `chv`**:单元映射
|
||||
`--private-users=1000:65536` 让单元内 uid 0 在客机侧就是 uid 1000,
|
||||
与客机 `chv` 同 uid。broker 的主校验是 pid ns 对照(`cell.pid`),
|
||||
uid 范围检查只是第二道;`cell.pid` 不存在时只做 uid 检查,并打日志。
|
||||
这不是“单元身份”的密码学证明,是命名空间对照。
|
||||
3. **a11y-tree-only 是方向,不是已封死的强制**:浏览器 socket 不暴露
|
||||
DOM/JS 操作,且单元内跑不起 playwright(已遮蔽),但单元内进程仍
|
||||
有 curl/Python 标准库,可以直接发 HTTP 请求取页面原文——那不是
|
||||
“浏览器 DOM 访问”,但也不是无障碍树。完整封堵(出口只走浏览器
|
||||
子代理)是 P3 Sentinel 的事。
|
||||
4. **性能数字全是待验证目标**:单元启动耗时、veth+NAT 对模型调用延迟
|
||||
的影响、快照截断后的 token 占用——本轮无真机验证(第三棒 V-1~V-9
|
||||
占着 WSL 测试机),一律记入“真机待验证清单”,不在代码/文档里写成
|
||||
已测得。
|
||||
5. **nspawn 参数以 systemd 255(`systemd-nspawn --help` 实测)为准**:
|
||||
`--private-users=UIDBASE[:NUIDS]`、`--drop-capability`、
|
||||
`--system-call-filter`、`--inaccessible`、`--network-veth`、
|
||||
`--as-pid2` 均在本机 help 里确认存在;但在 CHV 客机内核(云镜像
|
||||
generic 内核)里 userns/veth/idmap 的实际行为,须真机验证。
|
||||
6. **WarmPool/常驻路径与本轨无关**:P1 的常驻 VM 若接进来,单元在常驻
|
||||
VM 内同样起(`chv-cell run` 与 VM 生命周期无关);本轨不改执行器,
|
||||
不碰 P1 的文件。
|
||||
|
||||
## 4. 与 P1/P3 的接口(不碰对方文件,只定契约)
|
||||
|
||||
- **P1 → P2(执行器接单元)**:执行器调客内 host.py 的命令从
|
||||
`python3 /app/host.py …` 改为
|
||||
`chv-cell run -- python3 /app/host.py --state /home/chv/.codex/host-state.json --codex-home /home/chv/.codex --sandbox danger-full-access …`。
|
||||
`--sandbox danger-full-access` 只在单元内生效(单元即沙箱);非单元
|
||||
路径(本地/Docker/冷启动回落)保持原沙箱不动。改动在 `executor.py`,
|
||||
由 P1 做,本轨只提供客机侧 `chv-cell` 命令与本节契约。
|
||||
- **P1 → P2(vsock 进镜像)**:P1 的 `cloud/chv/guest/vsock_notify.py`
|
||||
由本轨 `build.sh` 条件拷入(§2),合并后自动生效;P2 不改该脚本内容。
|
||||
- **P2 → P3(broker 预留)**:broker 的 `sync` 操作现在单元外直接跑
|
||||
`chv-sync`(token 仍在客机侧落盘);P3 做 authd 替身化时,只改 broker
|
||||
内部的凭证获取方式(调 authd 拿短期替身),不改 broker 的 socket 协议
|
||||
与白名单。浏览器出口审批(只允许经 `chv-browser` 出站)同样是 P3 的
|
||||
Sentinel 接管点,本轨只做到“单元内无浏览器可跑”。
|
||||
|
||||
## 5. 真机待验证清单(round 11,测试机空闲后)
|
||||
|
||||
- lite 验收照旧:qcow2 ≤1GB、冷启动到 SSH、codex 版本、headless-shell
|
||||
例、Gitea 双向同步(boot clone + push + pull + 分叉失败关闭)。
|
||||
- 单元隔离验收(新增,否定测试为主):
|
||||
1. 单元内 `cat /etc/chv/guest-env` 失败(遮蔽 + 0600 overflow uid)。
|
||||
2. 单元内 `cat ~/.git-credentials` 失败;经 broker `sync status` 成功。
|
||||
3. 单元内写 `/usr/local/bin/x` 失败(只读根文件系统)。
|
||||
4. 单元内 `nsenter`/ptrace 客机侧进程失败(pid ns + dropped caps)。
|
||||
5. 含 `io_uring_setup` 的程序在单元内报 EPERM/ENOSYS(syscall 过滤)。
|
||||
6. 单元内 `python3 -c "import playwright"` 失败(已遮蔽);经 broker
|
||||
`browser open https://example.com` 返回标题 Example Domain 与
|
||||
无障碍树快照。
|
||||
7. 单元内 codex 一轮 turn 完成(`danger-full-access` 下,不再套 bwrap)。
|
||||
8. 单元内出站到模型网关通(veth + NAT);`chv-cell` 退出后 NAT 规则已清理。
|
||||
9. broker 拒绝非单元进程的连接(pid ns 不符时拒绝;`cell.pid` 缺失时
|
||||
只做 uid 检查并打日志——该降级须在日志里可见)。
|
||||
- 性能记录(只记录、不设门):单元启动到 codex 就绪耗时、veth+NAT 下
|
||||
模型调用往返、qcow2 实际占用(systemd-container 与新脚本的增量)。
|
||||
|
||||
## 6. round-11 part 4 补记(2026-11-07,RETEST3 的 3 个 cell 新缺陷)
|
||||
|
||||
RETEST3 证实 `--volatile=yes` + 只遮 /usr 的形状本身生效(遮蔽经
|
||||
mount/ls 实测有效),但引出三个阻塞缺陷,本轮按报告方向修:
|
||||
|
||||
1. **启动竞态**:`run_cell` 原顺序是 Popen → 配网 → 单次找 init pid,
|
||||
快 payload 在配网完成前退出即撞墙(`run -- true` 11 次只成 6 次)。
|
||||
修:payload 包一层 sh 起步门(等 `/run/chv/.cell-gate-<machine>`
|
||||
出现再 exec),配网完成后才开门;`find_cell_init_pid` 改带超时
|
||||
轮询;nspawn 已死时返回它的退出码而不是抛错;veth 等待放宽到
|
||||
20 秒。验收:连续 15 次 `run -- true` 计数(cellrace4)。
|
||||
2. **volatile 空根缺文件**:单元根是基本空的 tmpfs,`/app`、
|
||||
`/etc/resolv.conf`、`/etc/hosts`、`/etc/ssl/certs` 全无,执行器
|
||||
单元路径一律 ENOENT、DNS/TLS 不可用。修:`--bind-ro=/app` 与
|
||||
`--bind-ro=/etc/ssl/certs` 固定绑;resolv.conf/hosts 由 run_cell
|
||||
按客机有效 DNS 生成单元专属文件(`/run/chv/resolv.conf.<machine>`、
|
||||
`hosts.<machine>`,0644)再只读绑——客机 resolv.conf 可能是
|
||||
systemd-resolved stub(127.0.0.53),单元内无 resolved,直接绑
|
||||
即 DNS 全灭。不绑 nsswitch.conf(客机版可能含 resolve 模块,
|
||||
glibc 默认 files+dns 正是单元需要的)。P2-8 的 example 腿换
|
||||
v4 目标(gitea.tose.top)。
|
||||
3. **并发 cell 撞毁**:固定 `ve-chv-cell` 名 + 固定 10.200.0.1/24,
|
||||
第二个 cell 在 nspawn 建 veth 时即 `File exists`。修:每次
|
||||
run_cell 在锁下分配唯一身份(CellNet:machine `chv-cell-<xx>`、
|
||||
`ve-chv-cell-<xx>`、10.200.`<xx>`.0/24;machine 名 11 字符,
|
||||
ve- 前缀后 14 字符,符合接口名 15 字符上限;扫描活接口 + 地址
|
||||
+ machinectl 在册名避让注销滞后)。broker 的 pid ns 对照改成
|
||||
多 cell 版(`cell-<machine>.pid` glob + 兼容旧 `cell.pid`,对端
|
||||
ns 与任一活单元同 ns 即放行;旧单文件仍由 run_cell 写最新单元,
|
||||
P2-9 驱动仍读它)。
|
||||
|
||||
另:执行器 `_boot_vm` 的 vm.create/vm.boot 失败消息现在带截断后
|
||||
的响应体(`_truncate_api_body`,500 字符;只留 status 即无法定位
|
||||
间歇 500 的根因,两轮已提)。
|
||||
@@ -0,0 +1,124 @@
|
||||
# 对齐 Muse P3 差距表:authd 替身凭证 + Sentinel 式出口审批
|
||||
|
||||
依据:`~/workspace/research_notes/meta-muse-ai-architecture-20261101-0755/report.md`
|
||||
(官方以 Meta security.muse.ai 2026-09-08 为准;第三方拆解只作参考,
|
||||
逆向细节不得写成官方架构)。Muse 模型一句话:凭证由 VM 内独立服务
|
||||
authd 保管、代理只见替身令牌;所有连接器动作与网络出口由 Sentinel
|
||||
唯一审批(allow/deny/ask,L4+L7 评估,SSRF 防护,严格能力绑定);
|
||||
审批 UI 不在对话内;连接器执行走 privsep worker,按连接器做凭证白名单。
|
||||
|
||||
本轨范围只限 P3:authd 式替身凭证 + Sentinel 式出口审批。
|
||||
不碰 P1(常驻 VM、WarmPool、气球、vsock)、不碰 P2(客机 nspawn 单元、
|
||||
浏览器子代理独立)。分支 `feat-chv-muse-p3`(从 `feat-chv-cover100`
|
||||
分出,基线 HEAD `06c3e17`)。
|
||||
|
||||
## 1. 现状 vs Muse
|
||||
|
||||
| 项 | 现状(实测/代码为准) | Muse 目标 | 本轨做法 |
|
||||
|---|---|---|---|
|
||||
| Gitea token 可见面 | 真 token 经 seed 写进 `/etc/chv/guest-env`(0600 root);`build_ssh_command` 在每个任务的远程命令前 `source` 该文件,codex 进程环境里直接含真值;`chv-sync boot` 再把它写进 `/home/chv/.git-credentials` 落盘 | authd 保管凭证,代理只见替身令牌;凭证存入运行时单元之外的安全凭证存储,代理不可见;仅在需要时注入 | 真 token 改走 seed 里的独立文件 `/etc/chv/authd-seed`(0600 root),只给客内 authd 守护进程读;guest-env 不再含 GITEA_TOKEN;任务期 `build_ssh_command` 不再把 GITEA_TOKEN source 进 codex 环境;`chv-sync` 经 git credential helper 按次向 authd 取凭证,只进 git 子进程内存,不落盘、不进 agent 环境/argv/日志 |
|
||||
| authd 位置 | 无 | 存于用户 VM 内(非 Meta 集中基础设施),在运行时单元之外 | 客内 authd 守护进程(`chv-guest/authd.py`,标准库 Unix socket 服务),以 root 身份跑在单元外;宿主侧不存任何 VM 的 token。选客内不选宿主的理由:Muse 的 authd 就在用户 VM 里;宿主侧方案要走 vsock(P1 范围、未就绪),且会把全部 VM 的 token 集中到宿主,违背“存于用户 VM”的口径 |
|
||||
| 替身令牌 | 无(真值直给) | authd 签发替身令牌,带作用域与有效期 | 两档:① 本期默认——按次释放:每次 git 操作向 authd 取一次凭证,绑定 {连接器=gitea, 主机, 仓库路径, 操作},单次有效 + 300 秒 TTL,过期即废,释放全程审计(审计里无值);② 真铸 token:authd 可经 Gitea API 铸短期专用 token(`POST /api/v1/users/{user}/tokens`),默认关闭,见诚实边界第 2 条 |
|
||||
| 凭证白名单(privsep) | 无连接器概念;`_CHV_SECRET_ENV_NAMES` 只有“密钥/非密钥”二分 | 每个 privsep worker 有明确凭证白名单(日历 worker 不能仅改参数就索取邮件凭证) | authd 只认一个连接器 `gitea`,白名单只含 GITEA_TOKEN;未知连接器/越权主机/仓库/操作一律拒绝;SUB2API_API_KEY 仍走 guest-env(见诚实边界第 3 条) |
|
||||
| 出口审批 | 只有代码门:preflight 要求代理或显式直连 + `validate_egress_proxy` 格式校验;真正的隔离靠 Tinyproxy 白名单 + 宿主防火墙(第三棒落地中);客内 root 绕过代理直连在代码层无第二道门 | Sentinel 是连接器动作与全部网络出口的唯一权限机构;L4+L7 评估(主机名、解析后 IP、端口、协议、HTTP 方法、路径、解码后请求);SSRF 防护;审批是严格能力(绑定连接器/目的地/用例,一次性/会话级/任务级/时限级) | `cloud/chv/sentinel.py` 决策引擎:每动作审批钩子 allow/deny/ask;L4(解析后 IP、端口、协议)+ L7(主机、方法、路径)评估;SSRF 挡截(公开主机名解析到私有/回环/链路本地/CGNAT/保留地址即 deny);能力绑定 {连接器, 主机, 方法, 路径, 有效期, 次数};ask 记入待审批队列、执行停止。本期两个接线点:① wake 时 Gitea 出口预审(解析失败/私网 IP/不在白名单 → 起 VMM 前 fail-closed);② authd 放凭证前先过 Sentinel 评估 |
|
||||
| 审批 UI | 无 | Sentinel 直接向客户端发审批请求,弹窗在客户端 UI 内,答复直接回 Sentinel;目的:防对话内容伪造审批 | 本期只做设计约束 + 待审批队列(`sentinel.ApprovalQueue`,内存态 pending 记录):ask 出现即停止执行、记入队列,不自动放行;UI 不在对话内这一条记为约束,不在本期实现任何对话内审批入口 |
|
||||
| 污点出口(eBPF) | 无 | eBPF cgroup 程序做网络拦截与进程归因、LSM hook 做污点传播 | 列为后续,不冒充已做。见诚实边界第 4 条 |
|
||||
| 审计 | `_emit` 审计只含 VM 名/路径/URL/kind,不含 key 值(M5) | 审计轨迹展示已做与计划做的事 | 沿用 `_emit` 契约:新增 `sentinel_egress_allow/deny/ask`、`authd_credential_released/refused` 事件,detail 只含连接器/主机/路径/ verdict/原因,不含任何凭证值;authd 客内审计文件 `/var/log/chv-authd.log`(0600)同款口径 |
|
||||
|
||||
## 2. 改了什么(本轨交付物)
|
||||
|
||||
- `cloud/chv/authd.py`(新,叶子模块,标准库):凭证保险库(只活内存,
|
||||
`__repr__` 不含值)+ 按次释放(作用域绑定、单次有效、300 秒 TTL、
|
||||
过期即废)+ 连接器白名单(本期只 `gitea`)+ Gitea API 铸 token
|
||||
辅助函数(可注入 HTTP 客户端,默认关闭)+ 无值审计记录。
|
||||
- `cloud/chv/sentinel.py`(新,叶子模块,标准库):决策引擎
|
||||
`evaluate()`(allow/deny/ask)+ L4/L7 评估 + SSRF 挡截
|
||||
(解析器可注入,默认 `socket.getaddrinfo`)+ 严格能力
|
||||
`Capability`(连接器/主机/方法/路径/有效期/次数绑定)+
|
||||
`ApprovalQueue`(ask 待审批队列,内存态)+ 无值审计记录。
|
||||
- `cloud/chv/hostkeys.py`:`write_seed` 加 `authd_env` 参数——authd
|
||||
专用密钥写进独立的客内文件 `/etc/chv/authd-seed`(0600 root),
|
||||
不再与 guest-env 混写;`AUTHD_SEED_PATH` 常量。
|
||||
- `cloud/chv/gitea_sync.py`:`AUTHD_SECRET_KEYS = {"GITEA_TOKEN"}` +
|
||||
`split_authd_env()`(把 GITEA_TOKEN 从密钥集里拆出来走 authd 通道)。
|
||||
- `cloud/chv/ssh_transport.py`:`_CHV_AUTHD_SECRET_NAMES`;`remote_command`
|
||||
语义不变(仍只 source guest-env 文件)。
|
||||
- `cloud/chv/executor.py`:`provision_host_key`/`_provision_seed_disk`
|
||||
透传 `authd_env`;新增 `_authd_delivered` 投递集(与
|
||||
`_guest_env_delivered` 同锁)——authd 投递的密钥名在
|
||||
`build_ssh_command` 里不报错、但不 source 进 codex 环境;
|
||||
`_wake_prechecks` 加 Gitea 出口 Sentinel 预审(deny/ask 即
|
||||
`sentinel_egress_denied`,起 VMM 前 fail-closed)。
|
||||
- `cloud/chv/config.py`:`CHV_SENTINEL_ALLOW_HOSTS`(逗号分隔主机名白名单,
|
||||
默认只放行配置的 Gitea 主机)。
|
||||
- `chv-guest/authd.py`(新,标准库守护进程):读 `/etc/chv/authd-seed`
|
||||
持 token 于内存、Unix socket `/run/chv-authd/authd.sock`(0700 目录、
|
||||
0600 socket)、SO_PEERCRED 只放行 root 与 chv 用户、每次放凭证前先过
|
||||
Sentinel 评估、审计落盘 0600。
|
||||
- `chv-guest/git-credential-chv-authd`(新,标准库 helper):git
|
||||
credential 协议(stdin 读 `protocol/host/path`,stdout 回用户名/口令),
|
||||
口令只进 git 子进程内存。
|
||||
- `chv-guest/chv-authd.service`(新):systemd 单元(root 运行,单元外)。
|
||||
- `chv-guest/chv-sync.sh`:`boot` 在 authd socket 存在时改配
|
||||
`credential.helper` 走 authd,不再把真 token 写进
|
||||
`.git-credentials`;同时配 `credential.useHttpPath=true`
|
||||
(git 默认不把 path 发给 helper,没有 path helper 无法推断
|
||||
仓库与操作、每次都拒绝——新测试抓到的真问题);socket 不存在
|
||||
时走旧路径 + stderr 警告(过渡兼容,见诚实边界第 5 条)。
|
||||
- `cloud/tests/test_chv_muse_p3.py`(新):替身过期、真 token 泄露面、
|
||||
SSRF 挡截、deny/ask 路径、seed 拆分、wake 预审、helper 端到端
|
||||
(真 git `credential fill` + 假 authd socket);全量 cloud 套件全过。
|
||||
- 镜像集成(只留接入说明,不改 `chv-guest/build.sh`,与 P1 同款规矩,
|
||||
合并时由 P2 一并带进镜像):
|
||||
`--copy-in chv-guest/authd.py:/usr/local/bin/`、
|
||||
`--copy-in chv-guest/git-credential-chv-authd:/usr/local/bin/`、
|
||||
`--copy-in chv-guest/chv-authd.service:/etc/systemd/system/`、
|
||||
`--copy-in cloud/chv/authd.py:/usr/local/lib/chv-authd/cloud/chv/`、
|
||||
`--copy-in cloud/chv/sentinel.py:/usr/local/lib/chv-authd/cloud/chv/`、
|
||||
`--copy-in cloud/__init__.py:/usr/local/lib/chv-authd/cloud/`、
|
||||
`--copy-in cloud/chv/__init__.py:/usr/local/lib/chv-authd/cloud/chv/`、
|
||||
`--run-command 'chmod +x /usr/local/bin/chv-authd /usr/local/bin/git-credential-chv-authd'`、
|
||||
`--run-command 'systemctl enable chv-authd || true'`、
|
||||
`99-chv-sync.cfg` 的 runcmd 在 `chv-sync boot` 之前加
|
||||
`systemctl start chv-authd || true`。
|
||||
|
||||
## 3. 诚实边界(本轮不宣称的东西)
|
||||
|
||||
1. **seed 文件仍是 0600 root 落盘**:真 token 在 `/etc/chv/authd-seed`
|
||||
里仍以文件形式存在(cloud-init 的唯一投递通道就是写文件),且 seed
|
||||
ISO 本身挂在客机上。真正让它对 agent 不可见的是 P2 的 nspawn 单元
|
||||
隔离(单元内 root 映射为非特权用户,读不到单元外的 0600 文件);在
|
||||
P2 落地之前、codex 仍以客内 root 跑的镜像上,root agent 理论上仍能
|
||||
读到它。本轮堵住的是“进程环境/argv/日志/`.git-credentials` 落盘”
|
||||
这条泄露面,不是“客内 root 万能”——后者是 P2 的边界,不是 P3 能修的。
|
||||
2. **本期默认的“替身”不是 Gitea 侧可独立校验的新 token**:按次释放给
|
||||
git 子进程的值仍是真 token,只是只进 git 进程内存、不进 agent。
|
||||
真铸 token(Gitea API 铸短期专用 token、到期吊销)已实现辅助函数
|
||||
与假客户端测试,但默认关闭——它要求真 token 有建 token 的权限,
|
||||
现有测试 token 只有 `write:repository`,铸 token 会失败;开它须另配
|
||||
有铸 token 权限的 token,并做真机验证。不得把本期默认写成“Muse 同款
|
||||
替身令牌已落地”。
|
||||
3. **SUB2API_API_KEY 不在本轮**:网关 key 仍走 guest-env 进 codex 环境
|
||||
(H2 既有行为)。按 Muse 模型它也该走 privsep worker(由 worker 代
|
||||
codex 调网关、codex 不见 key),那是网关调用链改造,超出本轮范围,
|
||||
列为后续,不静默声称已覆盖。
|
||||
4. **eBPF 污点传播未做**:Sentinel 本轮是决策层(wake 预审 + authd 放
|
||||
凭证前评估),不是网络层强制。客内任意进程的直连仍由 Tinyproxy
|
||||
白名单 + 宿主防火墙(第三棒)兜底;eBPF cgroup 拦截 + LSM 污点传播
|
||||
是后续工作,不得写成“出口已全部经 Sentinel”。
|
||||
5. **过渡兼容已关闭(round-11 真机修复,层 5)**:此前 `chv-sync
|
||||
boot` 在 authd socket 不存在时仍走旧的 `.git-credentials` 落盘
|
||||
(只多一句 stderr 警告),为的是 lite9/lite10 旧镜像不被一次
|
||||
改断。round-11 真机测试证实该回落把 authd 释放的长效 token 写
|
||||
进了客内磁盘,修复轮已改为无 socket 即 fail-closed(`die`),
|
||||
同时镜像 `/etc/gitconfig` 的 `credential.helper store` 也已删
|
||||
除——authd helper 是唯一凭证 helper,且自身绝不落盘。旧镜像
|
||||
(lite10 及更早)上 boot 现在会直接失败,须用含 authd 的新
|
||||
镜像;真机复测须验证跑完一次完整 Gitea 链后客内
|
||||
`~/.git-credentials` 不存在。
|
||||
6. **无真机验证**:本轮全部是代码 + 本地单测(含真 git helper 端到端,
|
||||
但 authd 守护进程本身未在真 VM 里跑过)。真机待验证清单:authd 在
|
||||
lite11 镜像里启动时序(须在 `chv-sync boot` 之前就绪)、SO_PEERCRED
|
||||
放行 chv 用户、helper 在 pull/push 下的实际表现、wake 预审对真实
|
||||
Gitea 主机解析的判定、铸 token 的真实 API 行为。测试机(WSL)须等
|
||||
第三棒 V-1~V-9 让出后再做。
|
||||
@@ -317,6 +317,13 @@
|
||||
私钥在 seed 盘建好后即删(只留公钥 + known_hosts),删
|
||||
失败 fail-fast(VM 尚未启动);私钥 chmod 失败不再静默;
|
||||
启动/清理时 cleanup_stale_secrets 扫描残留私钥与 seed。
|
||||
seed 暂存文件(meta-data/user-data/network-config)走
|
||||
atomic_file(mkstemp 0600 → 写 → fsync → rename):
|
||||
user-data 含 host 私钥明文,直接写 + 事后 chmod 会在
|
||||
chmod 前以 umask 权限短暂可读(同 uid 进程可读到私钥),
|
||||
原子写让文件从创建起就是 0600;崩溃半截文件本身到不了
|
||||
ISO(唯一消费者 genisoimage 只在 write_seed 返回后跑,
|
||||
失败重试整文件重写),原子写同时堵住该残留。
|
||||
- H6 state_dir 出 /tmp:默认
|
||||
~/.local/share/codex-app-server/chv-vms,preflight 建目录
|
||||
并保证 0700(失败即 state_dir_insecure);api-socket 组/
|
||||
@@ -326,7 +333,14 @@
|
||||
- H7 tap 隔离校验:NET_JSON 透传前校验 tap/mac/ip 跨 VM
|
||||
不重复(wake 成功后登记 net.json,stop 时释放预约),
|
||||
共用即拒绝并审计;东西向隔离的宿主防火墙部分见 §11.4,
|
||||
不写进代码假装已防。
|
||||
不写进代码假装已防。restore_vm/wake_from_hibernate 同款
|
||||
(feat-chv-cover100 补齐:此前恢复路径既不校验冲突、也
|
||||
不登记 net.json,恢复出的 VM 会与已登记 VM 静默复用
|
||||
tap/mac/ip,且自身不登记、后来的 wake 也看不到它,两边
|
||||
都漏判):恢复前校验冲突(net_tap_conflict,起 VMM 前
|
||||
拒绝并审计 vm_failed)、恢复成功后登记 net.json,登记
|
||||
失败审计 net_registry_write_failed + 拒绝恢复(P2-19
|
||||
语义,stop_vm 清掉半恢复 VM)。
|
||||
|
||||
### 11.2 Medium(每项有修复或有理由的结论)
|
||||
|
||||
@@ -342,11 +356,20 @@
|
||||
- M4 供应链版本钉死:已钉死——云镜像 SHA256 必填(干跑验
|
||||
“已设”、全量验哈希)、Ubuntu 版本与 codex 版本均为必填
|
||||
环境变量;BUILDINFO 新增 versions(构建时从客机读回的
|
||||
dpkg + pip freeze 实际版本,可审计、可复现比对)。未钉死
|
||||
的清单:apt/pip 事前精确版本 lockfile(须在构建宿主上
|
||||
生成一次、再固化进脚本,本轨无构建宿主、未编造版本号);
|
||||
Playwright Chromium 二进制版本同样只随 playwright pip
|
||||
版本记录,未单独钉死。
|
||||
dpkg + pip freeze 实际版本,可审计、可复现比对)。
|
||||
generate_lockfile.py 已改写(to10 轨,本轮干跑复核):
|
||||
只读 BUILDINFO.versions 生成 lockfile,不再查构建宿主
|
||||
的 dpkg/pip,codex 三件套与 /app/host.py、gt、chv-sync
|
||||
走 BUILDINFO 声明字段;check_lockfile.py 已补强(条目
|
||||
格式、contents 全覆盖、冻结清单双向比对)。
|
||||
Chromium 已可单独钉死(本轮):CHROMIUM_HEADLESS_SHELL_REV
|
||||
设为纯数字修订号时 buildinfo.py 校验读回的缓存目录名
|
||||
须恰好是该修订号,否则构建失败,generate/check 的
|
||||
--verify 也复核;未设时只记录、不强制,构建输出明说
|
||||
“未钉死”。/app 已收紧为最小集(本轮):build.sh 只拷
|
||||
host.py(只用标准库,不依赖 cloud/),不再把整个 cloud/
|
||||
拷进客机。仍待真机:真实 lockfile 产出须等 V-9 真机
|
||||
构建(本机无 /dev/kvm,版本号绝不编造)。
|
||||
- M5 审计补齐:host_key_generated/host_key_captured/
|
||||
host_key_mismatch(stderr 含钉死拒绝标记时立即 fail-fast,
|
||||
不轮询成普通超时)、preflight_failed、wake 前置失败
|
||||
@@ -357,7 +380,12 @@
|
||||
实验放行须同时声明 CHV_SINGLE_TENANT_PROJECT,否则
|
||||
preflight 拒绝(reuse_tenant_undeclared);不能校验的
|
||||
部分明写:代码无法验证数据库里是否真的只有一个租户,
|
||||
声明 + 审计就是本项的机制,不假装已校验。
|
||||
声明 + 审计就是本项的机制,不假装已校验。部署核对
|
||||
(人工,见 §11.4):启用 reuse 前,部署人须在部署记录
|
||||
里写明“已核对数据库租户数 = 1、项目名与
|
||||
CHV_SINGLE_TENANT_PROJECT 一致”,并复核放行审计事件
|
||||
disk_reuse_experiment_allowed;多租户部署上误设该标志
|
||||
仍可能发生,代码拦不住,只能靠这道人工核对 + 审计告警。
|
||||
- M7 容量计数残留 socket 误伤:只有真能连通的 socket 才
|
||||
计数,空文件/死 socket 不计入 CHV_MAX_VMS;竞态窗口
|
||||
写清(刚启动未 listen 的 VMM 会被判死,容量门是护栏
|
||||
@@ -380,9 +408,18 @@
|
||||
- L3 失败审计:preflight/容量/配额/net 冲突失败均发
|
||||
vm_failed 审计(见 M5)。
|
||||
- L4 CHV_NET 与 CHV_NET_JSON 双变量:双设时发
|
||||
net_config_dual 警告审计(不硬拒,CHV_NET 是手动直接
|
||||
启动命令的透传值);NET_JSON 条目字段白名单
|
||||
(tap/mac/ip/mask),未知字段直接拒绝。
|
||||
net_config_dual 警告审计(不硬拒,取舍已定,见下);
|
||||
NET_JSON 条目字段白名单(tap/mac/ip/mask),未知字段
|
||||
直接拒绝。取舍定案(feat-chv-closeup 复核):保持警告、
|
||||
不改硬拒。理由:CHV_NET 是手动直接启动命令的透传值,
|
||||
与 API 模式的 CHV_NET_JSON 是两条不同的使用路径,双设
|
||||
不一定是误配;硬拒会断掉手动运维路径,而误配的代价
|
||||
只是“API 模式忽略 CHV_NET”,已有警告审计 + 本条文档
|
||||
写清。兼容影响评估:现存手动运维用法不受影响;若将来
|
||||
改硬拒,须先全量核查现存部署的 env 是否双设,并给手动
|
||||
路径留显式放行开关,不可直接改。部署核对(见 §11.4):
|
||||
生产 API 部署只设 CHV_NET_JSON,不设 CHV_NET;双设时
|
||||
net_config_dual 审计须在部署记录里复核。
|
||||
- L5 WarmPool 预热盘与任务盘对应关系:定义已写进
|
||||
pool.acquire 文档——预热盘是干净快照盘,不是任务盘;
|
||||
本轨定案为实验性(ADR-5),不再作为接线待办。
|
||||
@@ -402,6 +439,17 @@
|
||||
收紧 /proc 可见性、进程表采集不记录完整 cmdline;
|
||||
- 部署后核对:state_dir 权限、socket 实际权限、
|
||||
tap 规则实测(见 §11.5 否定测试)。
|
||||
- 环境变量核对(L4/M6,人工):生产 API 部署只设
|
||||
CHV_NET_JSON,不设 CHV_NET;双设时 net_config_dual
|
||||
审计须在部署记录里复核;启用 reuse 前部署人须在部署
|
||||
记录里写明“已核对数据库租户数 = 1、项目名与
|
||||
CHV_SINGLE_TENANT_PROJECT 一致”,并复核放行审计事件
|
||||
disk_reuse_experiment_allowed。
|
||||
- 残留尾巴审计核对(P2-17/18/19/20,feat-chv-closeup
|
||||
已修):部署后抽查审计里有无 net_registry_write_failed、
|
||||
dirty_marker_write_failed、vm_net_registry_delete_failed
|
||||
事件——任一出现即说明 state_dir 写路径或登记路径有
|
||||
故障,须先修宿主再启用。
|
||||
|
||||
### 11.5 仍需真机验证的清单(本轨一律未做,不声称已验)
|
||||
|
||||
@@ -785,3 +833,54 @@
|
||||
UNSTABLE、未合并、未部署、真机 0 项。差:外部为主。
|
||||
- 动态安全 2.0 → 2.0,未达 10:本轨无任何动态验证,
|
||||
V-7 未跑。差:外部(KVM 宿主 + 审计结论)。
|
||||
|
||||
## 15. 对齐 Muse P2 轨 feat-chv-muse-p2(客机 nspawn 单元 + 浏览器子代理)
|
||||
|
||||
本节是准备工作,不是启用结论。本轨只到本地分支,不合并、不部署;
|
||||
生产就绪与动态安全的外部关卡原样未动。完整差距表见
|
||||
docs/chv-muse-p2-gap.md(现状 vs Muse、改了什么、诚实边界、
|
||||
真机待验证清单、与 P1/P3 的接口)。
|
||||
|
||||
### 15.1 本轨量化结果(实测)
|
||||
|
||||
- 新增 `cloud/tests/test_chv_muse_p2.py` 40 项,全过。命令(仓库根目录):
|
||||
`python3 -m unittest cloud.tests.test_chv_muse_p2`
|
||||
- 冒烟(真实 Unix socket 回路,/tmp 临时路径,nobody 身份):
|
||||
broker ping 通、白名单拒绝通、sync 校验通、browser ping 通、
|
||||
evaluate 拒绝、localhost URL 拒绝、broker→browser 转发通;
|
||||
root(uid 0)连 broker 被拒并有日志。见 §15.2。
|
||||
- 本机无 /dev/kvm:镜像 round 11 未构建,真机验证 0 项(待第三棒
|
||||
V-1~V-9 让出 WSL 测试机)。
|
||||
|
||||
### 15.2 本轨交付物
|
||||
|
||||
- `chv-guest/chv-cell.py`:单元启动器。`--private-users=1000:65536`
|
||||
(单元内 root → 客机 chv)、丢 8 项能力(含 CAP_SYS_PTRACE /
|
||||
CAP_NET_ADMIN)、syscall 过滤扣 io_uring 三件套、遮蔽 /etc/chv /
|
||||
~/.git-credentials / playwright、`--network-veth`(10.200.0.0/24
|
||||
+ NAT,nsenter 配单元内网络)、白名单环境透传(GITEA_TOKEN 不进单元)。
|
||||
- `chv-guest/chv-broker.py`:单元外 IPC broker。Unix socket +
|
||||
SO_PEERCRED(uid 范围 + pid ns 对照 cell.pid),白名单 ping /
|
||||
sync / browser;sync 只调 chv-sync(runuser -u chv,list argv)。
|
||||
- `chv-guest/chv-browser.py`:浏览器子代理(用户 chv-browser)。
|
||||
只暴露 open/snapshot/screenshot/close/ping;返回无障碍树快照
|
||||
(截断 200 节点/64KB);evaluate/content/DevTools 一律拒绝;
|
||||
localhost 与私有地址字面量拒绝。
|
||||
- `chv-guest/build.sh`:apt 加 systemd-container/iproute2/iptables;
|
||||
拷入三脚本;建 chv-browser 系统用户;装 chv-broker/chv-browser
|
||||
两个 systemd 单元并 enable;tmpfiles.d 建 /run/chv。P1 的
|
||||
vsock_notify.py 条件拷入(文件不存在时只警告、不失败)。
|
||||
- `chv-guest/buildinfo.py`:内容项加 systemd-container/chv-cell/
|
||||
chv-broker/chv-browser;支持可选第 10 参数 extra_contents。
|
||||
|
||||
### 15.3 诚实边界(本轨不宣称)
|
||||
|
||||
- 无独立根文件系统:单元与客机共享根文件系统(只读 + 遮蔽 +
|
||||
bind),不是 Muse 的独立 Debian 镜像;world-readable 的客机文件
|
||||
单元内仍可读。完整分离要等下一阶段 + 体积重预算。
|
||||
- SO_PEERCRED 的 uid 检查不区分单元 root 与客机 chv(同 uid 1000);
|
||||
主校验是 pid ns 对照;cell.pid 缺失时只做 uid 检查并打日志。
|
||||
- a11y-tree-only 是方向:单元内仍有 curl/Python 标准库可直接发
|
||||
HTTP 请求;完整出口封堵是 P3 Sentinel 的事。
|
||||
- 单元内 codex 用 danger-full-access(单元即沙箱,不再套 bwrap);
|
||||
执行器侧改 `--sandbox` 透传是 P1 的文件,本轨只定义了接口契约。
|
||||
@@ -50,6 +50,7 @@ import os
|
||||
import queue
|
||||
import re
|
||||
import shlex
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
@@ -60,6 +61,40 @@ from concurrent.futures import ThreadPoolExecutor
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
DEFAULT_STATE = os.path.join(HERE, "host-state.json")
|
||||
|
||||
# part 6(RETEST5 新发现):单元内 worktree 根,绝不在 /app 之下。
|
||||
# 单元内 HERE=/app,而 /app 是 chv-cell 的 --bind-ro;part 5 曾
|
||||
# 把客机 workspace 下的 .chv-worktrees 以可写 bind 挂到
|
||||
# /app/work 上(argv 里排在 --bind-ro=/app 之后,指望后挂盖住
|
||||
# 只读子树),RETEST5 真机证明该设想不成立:后挂的 bind 只有在
|
||||
# 挂载点已存在时才能盖住,挂载点不存在时 nspawn 在只读 /app 下
|
||||
# 连建都建不起来(`Failed to create mount point .../app/work:
|
||||
# Read-only file system`,全量 73 次,CELL-RC=0 共 0 次),
|
||||
# 所有 cell 启动失败。修法三选一取 (a) worktree 根移出 /app:
|
||||
# 单元内改用 workspace 绑定下已可写的
|
||||
# /home/chv/workspace/.chv-worktrees(该路径经 --bind 整个
|
||||
# workspace 以可写方式进单元,无需任何额外 bind,挂载点问题
|
||||
# 按构造即不存在);(b) 默认 --no-worktree 不取:隔离丢了,
|
||||
# 且 goal/fork 同样调 make_worktree,照样撞 Errno 30;(c) 镜像
|
||||
# 里预建空目录 /app/work 不取:修法耦合镜像构建,某次重建漏
|
||||
# 建即静默回归,且单测无法钉住镜像内容。
|
||||
# CHV_WORKTREE_ROOT 由 chv-cell 经 --setenv 传进来(与
|
||||
# chv-cell.py 的 WORKTREE_DIR 同值);CHV_CELL=1 但变量未设
|
||||
# 时(旧版 chv-cell / 单元内手动跑 host.py)回落到同一单元内
|
||||
# 路径,单元内 worktree 根永不在 /app 之下;单元外沿用
|
||||
# HERE/work/host-worktrees,行为不变。
|
||||
CHV_WORKTREE_ROOT_ENV = "CHV_WORKTREE_ROOT"
|
||||
CELL_WORKTREE_ROOT = "/home/chv/workspace/.chv-worktrees"
|
||||
|
||||
|
||||
def worktree_base():
|
||||
"""worktree 根目录(<根>/host-worktrees/<任务>)。"""
|
||||
root = os.environ.get(CHV_WORKTREE_ROOT_ENV)
|
||||
if root:
|
||||
return os.path.join(root, "host-worktrees")
|
||||
if os.environ.get("CHV_CELL") == "1":
|
||||
return os.path.join(CELL_WORKTREE_ROOT, "host-worktrees")
|
||||
return os.path.join(HERE, "work", "host-worktrees")
|
||||
|
||||
|
||||
def _sum_turn_usage(events, thread_id, turn_id):
|
||||
"""从事件窗口里汇总一回合的上游真实用量。
|
||||
@@ -201,6 +236,22 @@ def default_codex_bin():
|
||||
paired = os.path.join(HERE, "bin", "codex")
|
||||
if os.path.exists(paired):
|
||||
return paired
|
||||
# part 5(RETEST4 新发现 1):单元内既无 CODEX_BIN 又无
|
||||
# HERE/bin/codex 时,旧代码返回裸 "codex",run_health_check
|
||||
# 的 os.path.exists("codex") 以相对路径判恒为 False,
|
||||
# health 的 ok 恒 False(单元内 which codex 明明是
|
||||
# /usr/local/bin/codex)。回落到 shutil.which("codex"):
|
||||
# PATH 里找得到即返回绝对路径,health 与 Server 都直接
|
||||
# 受益;找不到时仍返回裸 "codex",行为与旧代码一致。
|
||||
# 不选 (a) CODEX_BIN 进 FORWARD_ENV_ALLOWLIST:要执行器
|
||||
# 与 chv-cell 两边接线、还得在客机环境里定一个单元内路径,
|
||||
# 改动面大,且只修执行器驱动的调用;不选 (b) 执行器显式
|
||||
# 加 --codex-bin:把客机镜像路径硬编码进执行器,且直接
|
||||
# 跑 host.py 的 health 同样修不到。单点修 host.py 改动
|
||||
# 最小、覆盖全部调用方。
|
||||
found = shutil.which("codex")
|
||||
if found:
|
||||
return found
|
||||
return "codex"
|
||||
|
||||
|
||||
@@ -1225,7 +1276,7 @@ def git(args, cwd):
|
||||
def make_worktree(repo, task_name):
|
||||
"""一任务一 worktree;不是 git 仓库时退化为独立目录拷贝。"""
|
||||
repo = os.path.abspath(repo)
|
||||
base = os.path.join(HERE, "work", "host-worktrees")
|
||||
base = worktree_base()
|
||||
os.makedirs(base, exist_ok=True)
|
||||
path = os.path.join(base, task_name)
|
||||
if os.path.exists(path):
|
||||
@@ -1757,7 +1808,9 @@ def dispatch_from_branch(server, state, name, prompt, repo, dep_branch,
|
||||
c = git(["commit", "-m", f"host: {name} base from {dep_branch}"],
|
||||
dep_worktree)
|
||||
# nothing to commit 时分支里已有全部改动,继续用它当基底
|
||||
base = os.path.join(HERE, "work", "host-worktrees")
|
||||
# part 6:与 make_worktree 同款,worktree 根经 worktree_base()
|
||||
# 选(单元内在 workspace 下,不在只读 /app 之下)。
|
||||
base = worktree_base()
|
||||
os.makedirs(base, exist_ok=True)
|
||||
path = os.path.join(base, name)
|
||||
if os.path.exists(path):
|
||||
|
||||
@@ -6,6 +6,7 @@ import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
@@ -558,6 +559,98 @@ class HealthRestoreRecoverTest(unittest.TestCase):
|
||||
self.assertTrue(report["checks"]["server_ok"])
|
||||
|
||||
|
||||
class DefaultCodexBinTest(unittest.TestCase):
|
||||
"""part 5 修复 1(RETEST4 新发现 1):default_codex_bin()
|
||||
回落到 shutil.which("codex")。
|
||||
|
||||
模拟单元内环境:PATH 含 codex 所在目录、cwd 不含 codex、
|
||||
无 CODEX_BIN、无 HERE/bin/codex。旧代码此时返回裸 "codex",
|
||||
run_health_check 的 os.path.exists("codex") 恒 False,
|
||||
health 的 ok 恒 False。"""
|
||||
|
||||
class FakeServer:
|
||||
def initialize(self):
|
||||
return {}
|
||||
|
||||
def list_threads(self):
|
||||
return {"data": []}
|
||||
|
||||
def close(self):
|
||||
pass
|
||||
|
||||
def _fake_bin_dir(self, tmp):
|
||||
bindir = os.path.join(tmp, "bin")
|
||||
os.makedirs(bindir)
|
||||
codex = os.path.join(bindir, "codex")
|
||||
with open(codex, "w") as fh:
|
||||
fh.write("#!/bin/sh\nexit 0\n")
|
||||
os.chmod(codex, 0o755)
|
||||
return bindir, codex
|
||||
|
||||
def test_which_fallback_returns_absolute_path(self):
|
||||
# 仓库根下没有配对的 bin/codex(修的是无配对的情形)。
|
||||
self.assertFalse(
|
||||
os.path.exists(os.path.join(host.HERE, "bin", "codex")))
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
bindir, codex = self._fake_bin_dir(tmp)
|
||||
workdir = os.path.join(tmp, "work")
|
||||
os.makedirs(workdir)
|
||||
old_cwd = os.getcwd()
|
||||
try:
|
||||
os.chdir(workdir)
|
||||
env = {"PATH": bindir}
|
||||
with mock.patch.dict(os.environ, env, clear=True):
|
||||
self.assertEqual(host.default_codex_bin(), codex)
|
||||
finally:
|
||||
os.chdir(old_cwd)
|
||||
|
||||
def test_codex_bin_env_wins_over_path(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
bindir, _codex = self._fake_bin_dir(tmp)
|
||||
env = {"PATH": bindir, "CODEX_BIN": "/custom/codex"}
|
||||
with mock.patch.dict(os.environ, env, clear=True):
|
||||
self.assertEqual(host.default_codex_bin(),
|
||||
"/custom/codex")
|
||||
|
||||
def test_no_codex_anywhere_returns_bare_name(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
empty = os.path.join(tmp, "empty")
|
||||
os.makedirs(empty)
|
||||
with mock.patch.dict(os.environ, {"PATH": empty},
|
||||
clear=True):
|
||||
self.assertEqual(host.default_codex_bin(), "codex")
|
||||
|
||||
def test_health_ok_with_which_resolved_bin(self):
|
||||
# 验收口径:不加任何前缀的单元内 health 应 rc=0 且 ok=true。
|
||||
# 这里钉 host 侧:codex_bin_exists 为真且 report ok 为真。
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
bindir, _codex = self._fake_bin_dir(tmp)
|
||||
workdir = os.path.join(tmp, "work")
|
||||
os.makedirs(workdir)
|
||||
path = os.path.join(tmp, "host-state.json")
|
||||
with open(path, "w") as f:
|
||||
json.dump({"tasks": {}, "automations": [],
|
||||
"version": 1}, f)
|
||||
with open(path + ".bak", "w") as f:
|
||||
json.dump({"tasks": {}, "automations": [],
|
||||
"version": 1}, f)
|
||||
old_cwd = os.getcwd()
|
||||
try:
|
||||
os.chdir(workdir)
|
||||
with mock.patch.dict(os.environ, {"PATH": bindir},
|
||||
clear=True):
|
||||
codex_bin = host.default_codex_bin()
|
||||
self.assertTrue(os.path.isabs(codex_bin))
|
||||
report = host.run_health_check(
|
||||
path, {}, codex_bin, tmp, "deny",
|
||||
server_factory=lambda: self.FakeServer())
|
||||
finally:
|
||||
os.chdir(old_cwd)
|
||||
self.assertTrue(report["checks"]["codex_bin_exists"],
|
||||
report)
|
||||
self.assertTrue(report["ok"], report)
|
||||
|
||||
|
||||
class GithubPRTest(unittest.TestCase):
|
||||
class FakeGh:
|
||||
def __init__(self, checks_output="", pr_url=""):
|
||||
|
||||
Reference in new issue
Block a user