feat: 沙箱规模——/tmp 顺序加严、注册表寻址与失联精确清理、超时边界抽取
- Docker /tmp tmpfs 移到全部 -v 之前(父挂载在前,seed 子挂载在后); 宿主级 bwrap /tmp 仍宣告未隔离,口径见 tmp_isolation_status() - workers 表补 host/pid/address(老库幂等补列),心跳透传与保留语义, _claim 写 claimed_by_worker;orphaned_containers 精确识别失联容器, Worker.cleanup_orphaned_containers 默认只杀 pid_verified 的精确容器名 - docker_run_timeout() 纯函数抽取(行为与原内联一致),README 补边界与 未验证清单;新增 test_sandbox_scale.py 20 项
This commit is contained in:
1 parent
143ab30094
commit
06710cf47b
7 files changed
+897
-48
No files matched your search
@@ -1252,6 +1252,55 @@ interrupted 且终态任务 cancel 返回 400、心跳后 sweep 不扫刚心跳
|
||||
仍是串行一轮一任务)。两个端点都要 X-API-Key(与其它已认证
|
||||
端点一致);worker 直接写库注册时不需要 key,HTTP 心跳是给有
|
||||
key 的外部 worker 用的另一条路。
|
||||
- 沙箱规模(本轮新增,见 `cloud/tests/test_sandbox_scale.py`):
|
||||
- /tmp 隔离分层表(唯一口径 `sandbox.tmp_isolation_status()`):
|
||||
Docker 任务容器 `--tmpfs /tmp` + `--tmpfs /var/tmp`(容器随
|
||||
`--rm` 消失,/tmp 也随之消失,绝不落宿主 /tmp);且 tmpfs
|
||||
挂载在全部 `-v` 之前,附件 seed 的 `/tmp/codex-seed:ro` 子挂载
|
||||
压在 tmpfs 之上(旧顺序把 tmpfs 放最后,本 VM 无 docker 未
|
||||
证实旧顺序确实遮挡,故写为顺序加严而非已证实 bug)。
|
||||
宿主级 codex workspace-write(bwrap)仍不隔离:挂载由 codex
|
||||
二进制内部决定,host.py 只传 sandboxPolicy 类型、无挂载旋钮,
|
||||
本仓库改不了它;需要 /tmp 隔离的任务请走 docker 执行器,
|
||||
不要拿 Docker 层的隔离给宿主级任务背书。
|
||||
- workers 注册表寻址:workers 表新增 host(worker 的
|
||||
`--host-name`,容器内 worker 绝不能用容器名冒充宿主名)、
|
||||
pid(worker 进程号)、address(CLI `--worker-address`,显式给
|
||||
才有,不含任何凭证);老库幂等补列,老行三列为 NULL 按未知
|
||||
处理。`POST /workers/heartbeat` 同样接受三字段(host 非字符串、
|
||||
pid 非整数、address 非字符串时 400);心跳未给时保留旧值,
|
||||
worker 每次心跳都显式刷新本机 host 与本进程 pid,重启复用
|
||||
同 worker_id 时不残留旧 pid。
|
||||
- 失联容器精确清理(`workers.orphaned_containers()` +
|
||||
`Worker.cleanup_orphaned_containers()`,只写识别与杀容器,
|
||||
不改任务状态):容器名必须精确匹配 `codex-<32 位 hex 任务 id>`,
|
||||
且任务行存在、状态为 running/interrupted、认领记录
|
||||
`tasks.claimed_by_worker`(`_claim` 在 CAS 认领时写)指向
|
||||
心跳过期的 worker,才算失联;基础设施容器(如 codex-postgres)、
|
||||
非任务名、已完成任务、无认领记录、owner 仍存活的一律排除,
|
||||
绝不按 `codex-` 前缀删。默认只杀 pid_verified 的容器(本机 host
|
||||
+ 已知 pid 且 pid 明确已死);远端 host / pid 未知 / 未验证的
|
||||
只列不杀,显式 `verified_only=False` 才杀(仅单机语义已确认
|
||||
时开,跨主机场景绝不能默认开)。旧的
|
||||
`cleanup_stale_containers()`(任务正则兜底、不区分 owner)保留
|
||||
不改行为,供认领记录缺失的老任务容器用;任务状态本身由
|
||||
`sweep_stale_tasks`(心跳过期 -> interrupted)负责,两者分工。
|
||||
本轮只写代码与测试,绝不在本轮动生产容器、不杀任何生产任务。
|
||||
- 长任务天花板与并行验证的代码支撑(不宣称已大规模实测):
|
||||
超时边界抽成纯函数 `worker.docker_run_timeout()`(取较小值、
|
||||
None 语义、no_time_limit 语义、非法值回落,行为与原内联一致);
|
||||
取消 -> interrupted(执行结果绝不覆盖)、超时 -> failed、
|
||||
恢复靠断点续跑(goal-resume / run 续跑),三者状态语义不变。
|
||||
天花板验证方法(待真实大规模跑时按此执行,非本轮已做):
|
||||
睡眠循环任务(每轮 sleep + 一行回复)测轮次/墙钟上限、
|
||||
取消与超时各走一遍终态、owner 崩溃后 sweep + 精确清理各走
|
||||
一遍;多端并行只做 web 自适应跨视口:console 已含 viewport
|
||||
meta 与小屏 media 查询(单测只守基线),手工验证矩阵为
|
||||
320 / 375 / 768 / 1024 / 1440 宽度(无横向滚动、导航可收起、
|
||||
任务列表与详情可读),iOS/Android 原生明确不在本轮范围。
|
||||
未验证项(诚实清单):真实 40 人并行、真实大规模长任务天花板、
|
||||
真实 docker 下 tmpfs 顺序是否曾遮挡 seed、宿主级 bwrap /tmp
|
||||
隔离(本仓库无修复手段,见上)。
|
||||
- 任务按执行器池派发:tasks 表新增 `executor` 列(可空,老库幂等
|
||||
补列);`POST /tasks` 接受可选 executor(local/docker/cloud,
|
||||
非法值 400),未传时为 None 表示任意 worker 可认领;worker
|
||||
|
||||
@@ -1115,6 +1115,7 @@ def init_postgres_schema(conn):
|
||||
for _name, _type in (("executor", "TEXT"), ("pr_url", "TEXT"),
|
||||
("pr_number", "INTEGER"),
|
||||
("worktree_host", "TEXT"),
|
||||
("claimed_by_worker", "TEXT"),
|
||||
("goal_mode", "INTEGER"),
|
||||
("environment_id", "TEXT"),
|
||||
("child_model", "TEXT"),
|
||||
|
||||
+49
-12
@@ -41,6 +41,33 @@ from cloud.marker_tools import extract_wait_threads as _extract_wait_threads
|
||||
from cloud.peer_messages import extract_peer_messages as _extract_peer_messages
|
||||
|
||||
MAX_STDOUT_CHARS = 2000
|
||||
|
||||
|
||||
def tmp_isolation_status():
|
||||
"""各层 /tmp 隔离现状的唯一口径(README 与单测都以它为准)。
|
||||
|
||||
返回 {docker: bool, host_bwrap: bool, ...}:
|
||||
- docker=True:DockerExecutor 给每个任务容器挂 --tmpfs /tmp
|
||||
与 --tmpfs /var/tmp(容器随 --rm 消失,/tmp 也随之消失,
|
||||
绝不落宿主 /tmp;见 build_run_command 的顺序注释)。
|
||||
- host_bwrap=False:宿主级 codex workspace-write(bwrap)
|
||||
的挂载由 codex 二进制内部决定,host.py 只传
|
||||
sandboxPolicy 类型、没有任何挂载旋钮,所以本仓库改不
|
||||
了它;2026-10-26 宿主实测 agent 写 /tmp 落到宿主 /tmp,
|
||||
本轮未在宿主重测、也不宣称已修复。需要 /tmp 隔离的
|
||||
任务请走 docker 执行器,不要走宿主级沙箱 worker。
|
||||
本函数是状态宣告,不是修复开关:调用方绝不能拿它的
|
||||
docker=True 去给宿主级任务背书。
|
||||
"""
|
||||
return {
|
||||
"docker": True,
|
||||
"host_bwrap": False,
|
||||
"docker_detail": "per-container tmpfs /tmp + /var/tmp, removed with --rm",
|
||||
"host_bwrap_detail": "bwrap mounts are owned by the codex binary;"
|
||||
" host.py passes only a sandboxPolicy type"
|
||||
" and has no mount knob; /tmp is shared"
|
||||
" with the host (2026-10-26 host test)",
|
||||
}
|
||||
# 任务结果文本上限:2000 字截断的是 host.py 打印的整段 JSON,
|
||||
# text 超过 2000 字时 JSON 本身被截断,worker 再解析 stdout
|
||||
# 取 text 必然失败,存库的就变成截断的 JSON(2026-10-30
|
||||
@@ -1111,23 +1138,33 @@ class DockerExecutor(Executor):
|
||||
cmd += ["--runtime", self.runtime]
|
||||
if network:
|
||||
cmd += ["--network", network]
|
||||
cmd += ["-v", "%s:/work" % self._worktree_for(task),
|
||||
"-v", "%s:/app:ro" % self._app_dir(),
|
||||
"-w", "/work"]
|
||||
# 额外挂载放在基础挂载之后:/app/work 这类子路径挂载会
|
||||
# 覆盖 /app:ro 的对应子路径,让容器内 host.py 能写
|
||||
# work/ 和 host-state.json,同时代码主体仍只读。
|
||||
mounts = self.extra_mounts if extra_mounts is None \
|
||||
else self._parse_extra_mounts(extra_mounts)
|
||||
for host_p, cont_p, mode in mounts:
|
||||
cmd += ["-v", "%s:%s:%s" % (host_p, cont_p, mode)]
|
||||
# /tmp 固定为 tmpfs:容器内 /tmp 绝不落在宿主 /tmp,也不受
|
||||
# 额外挂载影响(/tmp 目标的额外挂载已在解析时拒绝)。
|
||||
# /tmp 固定为 tmpfs,且必须放在所有 -v 挂载之前:父挂载
|
||||
# 在前、子挂载在后是 Linux 挂载顺序的基本要求——附件
|
||||
# seed 以 /tmp/codex-seed:ro 挂进容器(见 dispatch_task),
|
||||
# 它是 /tmp 之下的子挂载,只有 tmpfs 先挂、seed 后挂,
|
||||
# seed 才不会被后挂的 tmpfs 遮掉。旧顺序把 --tmpfs 放在
|
||||
# 全部 -v 之后,本 VM 无 docker、未在真实 docker 上证实
|
||||
# 旧顺序确实遮挡(docker 可能按父子关系重排),所以这里
|
||||
# 写为顺序加严而非“已证实 bug 修复”;单测只断言顺序,
|
||||
# 不宣称旧顺序一定失效。
|
||||
# 容器内 /tmp 绝不落在宿主 /tmp,也不受额外挂载影响
|
||||
# (/tmp 目标的额外挂载已在解析时拒绝)。
|
||||
# 诚实边界:这是 Docker 外层能做的;内层 codex 的 bwrap
|
||||
# workspace-write 仍由 codex 二进制控制,本仓库改不了它
|
||||
# 的 --proc 挂载,宿主级 bwrap 的 /tmp 不隔离问题也不在
|
||||
# 本仓库的修复范围内——不要把本改动说成“内层沙箱已修”。
|
||||
cmd += ["--tmpfs", "/tmp", "--tmpfs", "/var/tmp"]
|
||||
cmd += ["-v", "%s:/work" % self._worktree_for(task),
|
||||
"-v", "%s:/app:ro" % self._app_dir(),
|
||||
"-w", "/work"]
|
||||
# 额外挂载放在基础挂载之后:/app/work 这类子路径挂载会
|
||||
# 覆盖 /app:ro 的对应子路径,让容器内 host.py 能写
|
||||
# work/ 和 host-state.json,同时代码主体仍只读;seed 的
|
||||
# /tmp/codex-seed:ro 同样在此后挂载,压在 tmpfs 之上。
|
||||
mounts = self.extra_mounts if extra_mounts is None \
|
||||
else self._parse_extra_mounts(extra_mounts)
|
||||
for host_p, cont_p, mode in mounts:
|
||||
cmd += ["-v", "%s:%s:%s" % (host_p, cont_p, mode)]
|
||||
if self.env_key:
|
||||
cmd += ["-e", self.env_key]
|
||||
merged_env = dict(self.extra_env)
|
||||
|
||||
+11
-6
@@ -1189,6 +1189,10 @@ def migrate_tasks_table(conn):
|
||||
# HA 阶段 1:认领任务的 worker 所在主机名;/pr 据此判断
|
||||
# worktree 在本机(同步跑)还是别机(排 pr 任务给该主机)。
|
||||
"worktree_host": "TEXT",
|
||||
# 沙箱规模:认领任务的 worker_id(worker._claim 在 CAS
|
||||
# 认领时写);失联容器识别的 owner 依据,见
|
||||
# workers.orphaned_containers。老行 NULL = 无 owner 记录。
|
||||
"claimed_by_worker": "TEXT",
|
||||
# 任务级选项(console 选模型/强度/权限用):NULL = 跟随
|
||||
# worker 配置默认值;model 不做白名单(中转目录会变,
|
||||
# 下拉只是建议值),effort/approval 由 _create_task 校验。
|
||||
@@ -1959,6 +1963,7 @@ def _task_to_dict(row, attachments=None, total_tokens=None):
|
||||
"attachments": list(attachments or []),
|
||||
"resume_message": _row_get(row, "resume_message"),
|
||||
"worktree_host": _row_get(row, "worktree_host"),
|
||||
"claimed_by_worker": _row_get(row, "claimed_by_worker"),
|
||||
"model": _row_get(row, "model"),
|
||||
"effort": _row_get(row, "effort"),
|
||||
"child_model": _row_get(row, "child_model"),
|
||||
@@ -9068,9 +9073,11 @@ class CloudControlPlaneHandler(BaseHTTPRequestHandler):
|
||||
executor/capacity/running/host/pid/address。心跳本身就是
|
||||
注册:worker_id 不存在时自动注册,绝不 404。running 是
|
||||
worker 上报的当前占用槽数(启动中占位也算);未给时保留
|
||||
旧值。host/pid/address 只存调用方显式给的值:远程 worker
|
||||
的 host/pid 绝不由 server 本机推导(server 的主机名/进程
|
||||
号不是该 worker 的),未给时保留旧值。"""
|
||||
旧值。host/pid/address 只存调用方显式给的值:远程 worker
|
||||
的 host/pid 绝不由 server 本机推导(server 的主机名/进程
|
||||
号不是该 worker 的),未给时保留旧值;host 须为字符串、
|
||||
pid 须为整数、address 须为字符串,否则 400(注册表里的
|
||||
寻址信息错了比没有更危险,失联判断会拿错 host/pid 当证据)。"""
|
||||
body, err = self._read_json()
|
||||
if err:
|
||||
self._send_json(400, {"error": err})
|
||||
@@ -9103,9 +9110,7 @@ class CloudControlPlaneHandler(BaseHTTPRequestHandler):
|
||||
return
|
||||
pid = body.get("pid")
|
||||
if pid is not None:
|
||||
try:
|
||||
pid = int(pid)
|
||||
except (TypeError, ValueError):
|
||||
if isinstance(pid, bool) or not isinstance(pid, int):
|
||||
self._send_json(400, {"error": "pid must be an integer"})
|
||||
return
|
||||
if pid <= 0:
|
||||
|
||||
@@ -0,0 +1,435 @@
|
||||
"""沙箱规模单测:/tmp 隔离顺序、注册表寻址、失联识别、超时边界。
|
||||
|
||||
运行(在 ~/workspace/codex-app-server-sandboxscale 下):
|
||||
python3 -m unittest discover -s cloud/tests -p "test_sandbox_scale.py" -v
|
||||
|
||||
说明:全部用注入的假 runner / 假 pid 检查,不调用真实 docker、
|
||||
不跑真实模型回合、不联网;env 映射里的值只是占位假名,
|
||||
绝不是真实密钥。任务 id 用 32 位 hex(容器名正则的精确
|
||||
形状),不用短 id——短 id 的容器名本来就不在失联识别范围内。
|
||||
|
||||
覆盖(与本轮改动一一对应):
|
||||
- Docker /tmp tmpfs 在所有 -v 挂载之前(父挂载在前,seed 子挂载
|
||||
在后,不被遮掉);host_bwrap 仍宣告未隔离(tmp_isolation_status)。
|
||||
- workers 注册/心跳:host/pid/address 写进库、心跳未给保留旧值、
|
||||
老库(无三列)迁移补列;stale_workers 只列心跳过期行。
|
||||
- 失联容器识别:只认精确 codex-<32 位 hex> + 认领到失联 worker
|
||||
+ running/interrupted;基础设施容器、非任务名、已完成任务、
|
||||
无认领、owner 存活的一律排除;pid_verified 只在“本机 host +
|
||||
已知 pid + pid 明确已死”时为真。
|
||||
- Worker 精确清理:verified_only 默认只杀已验证的精确容器名,
|
||||
未验证的只列不杀;显式 verified_only=False 才杀未验证的。
|
||||
- 认领写 claimed_by_worker(CAS 语义不变:二次认领仍失败)。
|
||||
- 超时边界纯函数 docker_run_timeout:取较小值、None 语义、
|
||||
不限时语义、非法值回落(行为与原内联一致)。
|
||||
- web 自适应基线:console.html 含 viewport meta 与小屏 media
|
||||
查询(跨视口手工验证矩阵见 cloud/README.md 新增节,不宣称
|
||||
本文件已做浏览器实测)。
|
||||
"""
|
||||
|
||||
import http.client
|
||||
import json
|
||||
import os
|
||||
import socket
|
||||
import sqlite3
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
from unittest import mock
|
||||
|
||||
try:
|
||||
from cloud.auth import AuthStore
|
||||
from cloud.sandbox import DockerExecutor, tmp_isolation_status
|
||||
from cloud.server import create_server, init_db, sweep_stale_tasks
|
||||
from cloud.worker import Worker, docker_run_timeout
|
||||
from cloud.workers import (default_pid_alive, heartbeat_worker,
|
||||
list_workers, orphaned_containers,
|
||||
register_worker, stale_workers)
|
||||
except ImportError: # discover -s cloud/tests 时把项目根目录补进 sys.path
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
from cloud.auth import AuthStore
|
||||
from cloud.sandbox import DockerExecutor, tmp_isolation_status
|
||||
from cloud.server import create_server, init_db, sweep_stale_tasks
|
||||
from cloud.worker import Worker, docker_run_timeout
|
||||
from cloud.workers import (default_pid_alive, heartbeat_worker,
|
||||
list_workers, orphaned_containers,
|
||||
register_worker, stale_workers)
|
||||
|
||||
DEAD_TASK = "aa" * 16
|
||||
LIVE_TASK = "bb" * 16
|
||||
DONE_TASK = "cc" * 16
|
||||
UNCLAIMED_TASK = "dd" * 16
|
||||
MISSING_TASK = "ee" * 16
|
||||
|
||||
|
||||
def _insert_task(db_path, task_id, status="queued", claimed=None,
|
||||
heartbeat_ago=None):
|
||||
now = time.time()
|
||||
heartbeat = None
|
||||
if heartbeat_ago is not None:
|
||||
heartbeat = now - heartbeat_ago
|
||||
elif status == "running":
|
||||
heartbeat = now
|
||||
conn = sqlite3.connect(db_path)
|
||||
try:
|
||||
conn.execute(
|
||||
"INSERT INTO tasks(id, project_id, prompt, status,"
|
||||
" idempotency_key, depends_on, priority, created_by,"
|
||||
" created_at, updated_at, last_heartbeat, claimed_by_worker)"
|
||||
" VALUES(?, 'p1', 'do work', ?, NULL, '[]', 0, 'tester',"
|
||||
" ?, ?, ?, ?)",
|
||||
(task_id, status, now, now, heartbeat, claimed),
|
||||
)
|
||||
conn.commit()
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def _set_worker_old(db_path, worker_id, seconds_ago):
|
||||
conn = sqlite3.connect(db_path)
|
||||
try:
|
||||
conn.execute(
|
||||
"UPDATE workers SET last_seen = ? WHERE worker_id = ?",
|
||||
(time.time() - seconds_ago, worker_id))
|
||||
conn.commit()
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def _seed_orphan_db(db_path, local_host):
|
||||
"""失联场景:dead worker(本机 host、pid 已死)与 live worker
|
||||
各认领一个 running 任务,另有已完成、无认领任务各一。"""
|
||||
init_db(db_path)
|
||||
register_worker(db_path, "dead-worker", executor="docker",
|
||||
host=local_host, pid=999999, address="h1:9001")
|
||||
register_worker(db_path, "live-worker", executor="docker",
|
||||
host=local_host, pid=123456, address="h1:9002")
|
||||
_set_worker_old(db_path, "dead-worker", 10000)
|
||||
_insert_task(db_path, DEAD_TASK, status="running",
|
||||
claimed="dead-worker", heartbeat_ago=10000)
|
||||
_insert_task(db_path, LIVE_TASK, status="running",
|
||||
claimed="live-worker")
|
||||
_insert_task(db_path, DONE_TASK, status="completed",
|
||||
claimed="dead-worker")
|
||||
_insert_task(db_path, UNCLAIMED_TASK, status="running",
|
||||
claimed=None)
|
||||
|
||||
|
||||
class TmpIsolationTestCase(unittest.TestCase):
|
||||
def test_tmpfs_before_all_volume_mounts(self):
|
||||
seed_dir = tempfile.mkdtemp()
|
||||
try:
|
||||
with mock.patch("cloud.sandbox.shutil.which",
|
||||
return_value="/usr/bin/docker"):
|
||||
ex = DockerExecutor(
|
||||
"/fake/host.py",
|
||||
runner=lambda cmd, timeout=None: SimpleNamespace(
|
||||
returncode=0, stdout="{}", stderr=""),
|
||||
env={"XCM_API_KEY": "placeholder-not-a-real-key"})
|
||||
cmd = ex.build_run_command(
|
||||
{"id": "t1", "worktree": "/tmp/wt"},
|
||||
["echo", "hi"],
|
||||
extra_mounts=["%s:/tmp/codex-seed:ro" % seed_dir])
|
||||
tmpfs_idx = cmd.index("--tmpfs")
|
||||
self.assertEqual(cmd[tmpfs_idx + 1], "/tmp")
|
||||
seed_idx = cmd.index("%s:/tmp/codex-seed:ro" % seed_dir)
|
||||
self.assertLess(tmpfs_idx, seed_idx)
|
||||
self.assertIn("/var/tmp", cmd)
|
||||
finally:
|
||||
os.rmdir(seed_dir)
|
||||
|
||||
def test_host_bwrap_still_declared_not_isolated(self):
|
||||
status = tmp_isolation_status()
|
||||
self.assertTrue(status["docker"])
|
||||
self.assertFalse(status["host_bwrap"])
|
||||
|
||||
|
||||
class RegistryAddressTestCase(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.db_path = os.path.join(self.tmp.name, "control.db")
|
||||
|
||||
def tearDown(self):
|
||||
self.tmp.cleanup()
|
||||
|
||||
def test_register_and_heartbeat_address_fields(self):
|
||||
row = register_worker(self.db_path, "w1", executor="docker",
|
||||
host="host-a", pid=111, address="h:1")
|
||||
self.assertEqual(row["host"], "host-a")
|
||||
self.assertEqual(row["pid"], 111)
|
||||
self.assertEqual(row["address"], "h:1")
|
||||
# 心跳未给 host/pid/address 时保留旧值,不洗成 None。
|
||||
row = heartbeat_worker(self.db_path, "w1", executor="docker")
|
||||
self.assertEqual(row["host"], "host-a")
|
||||
self.assertEqual(row["pid"], 111)
|
||||
self.assertEqual(row["address"], "h:1")
|
||||
# 心跳显式给新 pid 时刷新(重启后同 worker_id 复用不残留旧 pid)。
|
||||
row = heartbeat_worker(self.db_path, "w1", pid=222)
|
||||
self.assertEqual(row["pid"], 222)
|
||||
self.assertEqual(row["host"], "host-a")
|
||||
|
||||
def test_old_workers_table_migrates_address_columns(self):
|
||||
conn = sqlite3.connect(self.db_path)
|
||||
try:
|
||||
conn.execute(
|
||||
"CREATE TABLE workers(worker_id TEXT PRIMARY KEY,"
|
||||
" executor TEXT, capacity INTEGER,"
|
||||
" last_seen DOUBLE PRECISION,"
|
||||
" created_at DOUBLE PRECISION)")
|
||||
conn.commit()
|
||||
finally:
|
||||
conn.close()
|
||||
row = register_worker(self.db_path, "w-old", host="host-b",
|
||||
pid=333)
|
||||
self.assertEqual(row["host"], "host-b")
|
||||
self.assertEqual(row["pid"], 333)
|
||||
self.assertIsNone(row["address"])
|
||||
|
||||
def test_stale_workers_only_expired(self):
|
||||
register_worker(self.db_path, "fresh", host="h")
|
||||
register_worker(self.db_path, "old", host="h")
|
||||
_set_worker_old(self.db_path, "old", 10000)
|
||||
stale = stale_workers(self.db_path, alive_seconds=120)
|
||||
self.assertEqual([w["worker_id"] for w in stale], ["old"])
|
||||
self.assertFalse(stale[0]["alive"])
|
||||
live = list_workers(self.db_path, alive_seconds=120)
|
||||
self.assertTrue([w for w in live
|
||||
if w["worker_id"] == "fresh"][0]["alive"])
|
||||
|
||||
def test_default_pid_alive_current_process(self):
|
||||
self.assertTrue(default_pid_alive(os.getpid()))
|
||||
self.assertIsNone(default_pid_alive("not-a-pid"))
|
||||
self.assertIsNone(default_pid_alive(None))
|
||||
|
||||
|
||||
class OrphanIdentifyTestCase(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.db_path = os.path.join(self.tmp.name, "control.db")
|
||||
self.host = socket.gethostname()
|
||||
_seed_orphan_db(self.db_path, self.host)
|
||||
|
||||
def tearDown(self):
|
||||
self.tmp.cleanup()
|
||||
|
||||
def _names(self):
|
||||
return ["codex-" + DEAD_TASK, "codex-" + LIVE_TASK,
|
||||
"codex-" + DONE_TASK, "codex-" + UNCLAIMED_TASK,
|
||||
"codex-" + MISSING_TASK, "codex-postgres",
|
||||
"codex-not-a-task", "other-" + DEAD_TASK]
|
||||
|
||||
def test_only_dead_owner_exact_container_verified(self):
|
||||
found = orphaned_containers(
|
||||
self.db_path, self._names(), alive_seconds=120,
|
||||
pid_alive=lambda pid: False)
|
||||
self.assertEqual(len(found), 1)
|
||||
self.assertEqual(found[0]["container"], "codex-" + DEAD_TASK)
|
||||
self.assertEqual(found[0]["task_id"], DEAD_TASK)
|
||||
self.assertEqual(found[0]["worker_id"], "dead-worker")
|
||||
self.assertTrue(found[0]["pid_verified"])
|
||||
|
||||
def test_live_pid_not_verified_but_still_listed(self):
|
||||
found = orphaned_containers(
|
||||
self.db_path, self._names(), alive_seconds=120,
|
||||
pid_alive=lambda pid: True)
|
||||
self.assertEqual(len(found), 1)
|
||||
self.assertFalse(found[0]["pid_verified"])
|
||||
|
||||
def test_remote_host_never_pid_verified(self):
|
||||
conn = sqlite3.connect(self.db_path)
|
||||
try:
|
||||
conn.execute(
|
||||
"UPDATE workers SET host = ? WHERE worker_id = ?",
|
||||
("some-other-host", "dead-worker"))
|
||||
conn.commit()
|
||||
finally:
|
||||
conn.close()
|
||||
found = orphaned_containers(
|
||||
self.db_path, self._names(), alive_seconds=120,
|
||||
pid_alive=lambda pid: False)
|
||||
self.assertEqual(len(found), 1)
|
||||
self.assertFalse(found[0]["pid_verified"])
|
||||
|
||||
def test_sweep_marks_dead_owner_task_interrupted(self):
|
||||
swept = sweep_stale_tasks(self.db_path, threshold_seconds=300)
|
||||
self.assertIn(DEAD_TASK, swept)
|
||||
self.assertNotIn(LIVE_TASK, swept)
|
||||
# sweep 后失联容器仍可识别(interrupted 状态也算失联)。
|
||||
found = orphaned_containers(
|
||||
self.db_path, self._names(), alive_seconds=120,
|
||||
pid_alive=lambda pid: False)
|
||||
self.assertEqual(len(found), 1)
|
||||
|
||||
|
||||
class WorkerOrphanCleanupTestCase(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.db_path = os.path.join(self.tmp.name, "control.db")
|
||||
self.host = socket.gethostname()
|
||||
_seed_orphan_db(self.db_path, self.host)
|
||||
self.killed = []
|
||||
|
||||
def fake(cmd, timeout=None):
|
||||
if len(cmd) >= 2 and cmd[1] == "ps":
|
||||
return SimpleNamespace(
|
||||
returncode=0,
|
||||
stdout="\n".join(
|
||||
["codex-" + DEAD_TASK, "codex-" + LIVE_TASK,
|
||||
"codex-postgres"]) + "\n",
|
||||
stderr="")
|
||||
if len(cmd) >= 2 and cmd[1] in ("kill", "rm"):
|
||||
self.killed.append(cmd[-1])
|
||||
return SimpleNamespace(returncode=0, stdout="", stderr="")
|
||||
|
||||
self.ex = DockerExecutor(
|
||||
"/fake/host.py", runner=fake,
|
||||
env={"XCM_API_KEY": "placeholder-not-a-real-key"})
|
||||
self.worker = Worker(
|
||||
self.db_path, "/fake/host.py",
|
||||
env={"XCM_API_KEY": "placeholder-not-a-real-key"},
|
||||
executor=self.ex, worker_id="cleaner",
|
||||
worker_address="cleaner-host:9009")
|
||||
|
||||
def tearDown(self):
|
||||
self.tmp.cleanup()
|
||||
|
||||
def test_verified_only_kills_exact_dead_container(self):
|
||||
cleaned = self.worker.cleanup_orphaned_containers(
|
||||
pid_alive=lambda pid: False, alive_seconds=120)
|
||||
self.assertEqual(cleaned, ["codex-" + DEAD_TASK])
|
||||
self.assertIn("codex-" + DEAD_TASK, self.killed)
|
||||
self.assertNotIn("codex-" + LIVE_TASK, self.killed)
|
||||
self.assertNotIn("codex-postgres", self.killed)
|
||||
|
||||
def test_unverified_skipped_by_default(self):
|
||||
cleaned = self.worker.cleanup_orphaned_containers(
|
||||
pid_alive=lambda pid: True, alive_seconds=120)
|
||||
self.assertEqual(cleaned, [])
|
||||
self.assertEqual(self.killed, [])
|
||||
|
||||
def test_explicit_unverified_opt_in_kills(self):
|
||||
cleaned = self.worker.cleanup_orphaned_containers(
|
||||
verified_only=False, pid_alive=lambda pid: True,
|
||||
alive_seconds=120)
|
||||
self.assertEqual(cleaned, ["codex-" + DEAD_TASK])
|
||||
|
||||
def test_claim_writes_claimed_by_worker(self):
|
||||
_insert_task(self.db_path, "f1" * 16, status="queued")
|
||||
worker = Worker(
|
||||
self.db_path, "/fake/host.py",
|
||||
env={"XCM_API_KEY": "placeholder-not-a-real-key"},
|
||||
worker_id="claimer-1")
|
||||
self.assertTrue(worker._claim("f1" * 16))
|
||||
task = worker.get_task("f1" * 16)
|
||||
self.assertEqual(task["claimed_by_worker"], "claimer-1")
|
||||
# CAS 语义不变:已 running 的任务二次认领仍失败。
|
||||
self.assertFalse(worker._claim("f1" * 16))
|
||||
|
||||
|
||||
class TimeoutBoundaryTestCase(unittest.TestCase):
|
||||
def test_min_wins_and_none_semantics(self):
|
||||
self.assertEqual(docker_run_timeout(1800, None), 1800)
|
||||
self.assertEqual(docker_run_timeout(1800, 600), 600)
|
||||
self.assertEqual(docker_run_timeout(600, 1800), 600)
|
||||
self.assertIsNone(docker_run_timeout(None, 600))
|
||||
self.assertIsNone(docker_run_timeout(None, None))
|
||||
|
||||
def test_no_time_limit_overrides_both(self):
|
||||
self.assertIsNone(docker_run_timeout(1800, 600,
|
||||
no_time_limit=True))
|
||||
|
||||
def test_invalid_value_falls_back_to_task_timeout(self):
|
||||
self.assertEqual(docker_run_timeout(1800, "bad"), 1800)
|
||||
|
||||
|
||||
class ViewportBaselineTestCase(unittest.TestCase):
|
||||
def test_console_has_viewport_and_small_screen_query(self):
|
||||
console = (Path(__file__).resolve().parents[2]
|
||||
/ "cloud" / "console.html")
|
||||
text = console.read_text(encoding="utf-8")
|
||||
self.assertIn('name="viewport"', text)
|
||||
self.assertIn("@media(max-width:", text)
|
||||
|
||||
|
||||
class HeartbeatHttpTestCase(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.db_path = os.path.join(self.tmp.name, "control.db")
|
||||
self.auth = AuthStore(self.db_path)
|
||||
self.owner_id, self.owner_key = self.auth.create_user("owner")
|
||||
self._old_admins = os.environ.get("CODEX_GLOBAL_ADMINS")
|
||||
os.environ["CODEX_GLOBAL_ADMINS"] = self.owner_id
|
||||
self.server = create_server(self.db_path, "127.0.0.1", 0)
|
||||
self.port = self.server.server_address[1]
|
||||
self.thread = threading.Thread(
|
||||
target=self.server.serve_forever, daemon=True)
|
||||
self.thread.start()
|
||||
|
||||
def tearDown(self):
|
||||
self.server.shutdown()
|
||||
self.server.server_close()
|
||||
self.thread.join(timeout=5)
|
||||
try:
|
||||
self.auth.conn.close()
|
||||
except Exception:
|
||||
pass
|
||||
self.tmp.cleanup()
|
||||
if self._old_admins is None:
|
||||
os.environ.pop("CODEX_GLOBAL_ADMINS", None)
|
||||
else:
|
||||
os.environ["CODEX_GLOBAL_ADMINS"] = self._old_admins
|
||||
|
||||
def _req(self, method, path, body=None, key=None):
|
||||
conn = http.client.HTTPConnection("127.0.0.1", self.port,
|
||||
timeout=10)
|
||||
hdrs = {}
|
||||
payload = None
|
||||
if body is not None:
|
||||
payload = json.dumps(body).encode("utf-8")
|
||||
hdrs["Content-Type"] = "application/json"
|
||||
if key is not None:
|
||||
hdrs["X-API-Key"] = key
|
||||
try:
|
||||
conn.request(method, path, body=payload, headers=hdrs)
|
||||
resp = conn.getresponse()
|
||||
data = resp.read()
|
||||
finally:
|
||||
conn.close()
|
||||
try:
|
||||
parsed = json.loads(data.decode("utf-8")) if data else None
|
||||
except ValueError:
|
||||
parsed = None
|
||||
return resp.status, parsed
|
||||
|
||||
def test_heartbeat_address_fields_roundtrip(self):
|
||||
status, body = self._req(
|
||||
"POST", "/workers/heartbeat",
|
||||
{"worker_id": "hw1", "host": "host-x", "pid": 4242,
|
||||
"address": "host-x:9001"},
|
||||
self.owner_key)
|
||||
self.assertEqual(status, 200, body)
|
||||
self.assertEqual(body["host"], "host-x")
|
||||
self.assertEqual(body["pid"], 4242)
|
||||
self.assertEqual(body["address"], "host-x:9001")
|
||||
status, body = self._req("GET", "/workers",
|
||||
key=self.owner_key)
|
||||
self.assertEqual(status, 200, body)
|
||||
row = [w for w in body["workers"]
|
||||
if w["worker_id"] == "hw1"][0]
|
||||
self.assertEqual(row["host"], "host-x")
|
||||
self.assertEqual(row["pid"], 4242)
|
||||
|
||||
def test_heartbeat_invalid_pid_400(self):
|
||||
status, _ = self._req(
|
||||
"POST", "/workers/heartbeat",
|
||||
{"worker_id": "hw2", "pid": "not-an-int"},
|
||||
self.owner_key)
|
||||
self.assertEqual(status, 400)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
+138
-22
@@ -199,7 +199,8 @@ from cloud.sandbox import CloudExecutor, DockerExecutor, LocalExecutor
|
||||
from cloud.uploads import upload_root as _upload_root
|
||||
from cloud.sandbox import _default_runner as _sandbox_default_runner
|
||||
from cloud.caps import effective_capacity, resolve_capacity
|
||||
from cloud.workers import heartbeat_worker, register_worker
|
||||
from cloud.workers import (heartbeat_worker, orphaned_containers,
|
||||
register_worker)
|
||||
|
||||
# 与 server.TERMINAL_STATUSES 对齐:这些状态的任务不再被认领或取消。
|
||||
TERMINAL_STATUSES = frozenset(
|
||||
@@ -232,6 +233,39 @@ def _uuid_hex():
|
||||
return uuid.uuid4().hex
|
||||
|
||||
|
||||
def docker_run_timeout(task_timeout, docker_timeout,
|
||||
no_time_limit=False):
|
||||
"""本次 docker run 超时(纯函数,可单测,长任务天花板边界)。
|
||||
|
||||
规则(与 _run_attempt 原内联逻辑完全一致,本轮只抽取,
|
||||
不改行为):
|
||||
- no_time_limit=1(任务不限时)时一律 None:不限的是任务
|
||||
总墙钟与总轮次,不是单轮无限——host 单轮 turn_timeout
|
||||
仍是护栏,不在本函数里。
|
||||
- task_timeout 为 None 时 None(Docker 路径此时也不再用
|
||||
executor.timeout 兜底?注意:原内联只在 task_timeout
|
||||
非 None 时算 min,task_timeout 为 None 时 effective 恒
|
||||
为 None——本函数忠实保留该语义,绝不悄悄把 docker_timeout
|
||||
扶正为有效超时)。
|
||||
- 否则取 min(task_timeout, docker_timeout)(docker_timeout
|
||||
为 None 时取 task_timeout);任一值非法时回落 task_timeout。
|
||||
取消与超时的状态语义不在此:超时 -> failed(不是取消),
|
||||
取消 -> interrupted(执行结果绝不覆盖它),见 _run_attempt
|
||||
与 cancel();恢复靠断点续跑(goal-resume / run 续跑),
|
||||
不是靠把超时任务悄悄重跑一遍。
|
||||
"""
|
||||
if no_time_limit:
|
||||
return None
|
||||
if task_timeout is None:
|
||||
return None
|
||||
if docker_timeout is None:
|
||||
return task_timeout
|
||||
try:
|
||||
return min(float(task_timeout), float(docker_timeout))
|
||||
except (TypeError, ValueError):
|
||||
return task_timeout
|
||||
|
||||
|
||||
def _worker_host_name(executor, task):
|
||||
"""host 任务名:优先 executor._host_name(与 sandbox 同款
|
||||
派生),没有 executor 时回落任务 id(uuid hex 本就是安全名)。"""
|
||||
@@ -405,6 +439,9 @@ def _task_from_row(row):
|
||||
"interrupted_reason": _col("interrupted_reason"),
|
||||
"auto_resume_count": _col("auto_resume_count") or 0,
|
||||
"no_time_limit": _col("no_time_limit") or 0,
|
||||
# 认领 owner:老行缺列时 None(= 认领前或老任务,无 owner
|
||||
# 可查,失联识别宁可漏不猜)。
|
||||
"claimed_by_worker": _col("claimed_by_worker"),
|
||||
# 协调员:老行缺列时 coordinator_parts 为 [](= 普通任务),
|
||||
# parent_task_id/coordinator_part 缺列时为 None。
|
||||
"coordinator_parts": _coord_parse_parts(_col("coordinator_parts")),
|
||||
@@ -442,7 +479,8 @@ class Worker:
|
||||
outbox_max_attempts=3,
|
||||
quota_poll_interval_seconds=30,
|
||||
auto_resume_max=50,
|
||||
ignore_host_budget=True):
|
||||
ignore_host_budget=True,
|
||||
worker_address=None):
|
||||
self.db_path = str(db_path)
|
||||
self.host_py_path = host_py_path
|
||||
self.config_path = config_path
|
||||
@@ -638,6 +676,11 @@ class Worker:
|
||||
# 自然变 stale,不要用随机 uuid 让注册表无法对应同一台机器。
|
||||
self.worker_id = worker_id or "%s-%d" % (
|
||||
socket.gethostname(), os.getpid())
|
||||
# 寻址串:调用方显式给才有(如运维约定的 host:port),
|
||||
# 默认 None(不猜、不从环境变量里拼密钥类地址);注册/
|
||||
# 心跳时原样写进 workers.address,供失联排查时定位
|
||||
# owner 在哪台主机、哪个入口,绝不是访问凭证。
|
||||
self.worker_address = worker_address
|
||||
# 并发上限配置来源(见 cloud/caps.py):显式构造参数 /
|
||||
# CLI --worker-capacity > 环境变量 CLOUD_WORKER_CAPACITY >
|
||||
# 默认 1;非法值归一为 1。self._env 在上面已定好,单测
|
||||
@@ -700,7 +743,8 @@ class Worker:
|
||||
self.db_path, self.worker_id,
|
||||
executor=self._executor_name(), capacity=self.capacity,
|
||||
running=self._running_count(),
|
||||
host=self.host_name, pid=os.getpid())
|
||||
host=self.host_name, pid=os.getpid(),
|
||||
address=self.worker_address)
|
||||
except Exception as exc:
|
||||
log_event(self.logger, "worker_register_failed",
|
||||
message="worker register failed: %s" % exc)
|
||||
@@ -712,13 +756,17 @@ class Worker:
|
||||
时退化为 socket.gethostname(),容器内即容器名,与
|
||||
worktree_host 同款 caveat),pid 用本进程号;两者每轮
|
||||
刷新,供失联判定做本机 + pid 已死的 pid_verified 校验。
|
||||
"""
|
||||
每次都显式传本机 host 与本进程 pid:worker 重启后
|
||||
同 worker_id 复用时,旧 pid 不会残留在注册表里误导
|
||||
失联判断(旧 pid 残留 + 心跳新鲜 = 活着,尚可;旧 pid
|
||||
残留 + 心跳过期 = pid 校验会误判——每次刷新杜绝此例)。 """
|
||||
try:
|
||||
heartbeat_worker(
|
||||
self.db_path, self.worker_id,
|
||||
executor=self._executor_name(), capacity=self.capacity,
|
||||
running=self._running_count(),
|
||||
host=self.host_name, pid=os.getpid())
|
||||
host=self.host_name, pid=os.getpid(),
|
||||
address=self.worker_address)
|
||||
except Exception as exc:
|
||||
log_event(self.logger, "worker_registry_heartbeat_failed",
|
||||
message="worker registry heartbeat failed: %s" % exc)
|
||||
@@ -859,31 +907,37 @@ class Worker:
|
||||
conn = _db_connect(self.db_path)
|
||||
try:
|
||||
self._ensure_result_columns(conn)
|
||||
# 认领同时写 claimed_by_worker:失联容器识别的唯一
|
||||
# owner 依据(见 workers.orphaned_containers)。CAS
|
||||
# 语义不变(仍只在 queued 时改行),认领失败时行不
|
||||
# 动、旧认领记录也不被覆盖。
|
||||
if executor is None:
|
||||
cur = conn.execute(
|
||||
"UPDATE tasks SET status = 'running',"
|
||||
" last_heartbeat = ?, updated_at = ?,"
|
||||
" worktree_host = ?"
|
||||
" worktree_host = ?, claimed_by_worker = ?"
|
||||
" WHERE id = ? AND status = 'queued'",
|
||||
(now, now, self.host_name, task_id),
|
||||
(now, now, self.host_name, self.worker_id, task_id),
|
||||
)
|
||||
elif self.claim_executor_only:
|
||||
cur = conn.execute(
|
||||
"UPDATE tasks SET status = 'running',"
|
||||
" last_heartbeat = ?, updated_at = ?,"
|
||||
" worktree_host = ?"
|
||||
" worktree_host = ?, claimed_by_worker = ?"
|
||||
" WHERE id = ? AND status = 'queued'"
|
||||
" AND executor = ?",
|
||||
(now, now, self.host_name, task_id, executor),
|
||||
(now, now, self.host_name, self.worker_id, task_id,
|
||||
executor),
|
||||
)
|
||||
else:
|
||||
cur = conn.execute(
|
||||
"UPDATE tasks SET status = 'running',"
|
||||
" last_heartbeat = ?, updated_at = ?,"
|
||||
" worktree_host = ?"
|
||||
" worktree_host = ?, claimed_by_worker = ?"
|
||||
" WHERE id = ? AND status = 'queued'"
|
||||
" AND (executor IS NULL OR executor = ?)",
|
||||
(now, now, self.host_name, task_id, executor),
|
||||
(now, now, self.host_name, self.worker_id, task_id,
|
||||
executor),
|
||||
)
|
||||
conn.commit()
|
||||
claimed = cur.rowcount == 1
|
||||
@@ -1431,6 +1485,7 @@ class Worker:
|
||||
"goal_max_rounds": "INTEGER",
|
||||
"goal_mode": "INTEGER",
|
||||
"worktree_host": "TEXT",
|
||||
"claimed_by_worker": "TEXT",
|
||||
"model": "TEXT",
|
||||
"effort": "TEXT",
|
||||
"child_model": "TEXT",
|
||||
@@ -2382,12 +2437,71 @@ class Worker:
|
||||
pass
|
||||
return
|
||||
|
||||
def cleanup_orphaned_containers(self, verified_only=True,
|
||||
pid_alive=None, alive_seconds=None):
|
||||
"""精确失联清理:只杀 workers.orphaned_containers() 认定的
|
||||
容器(按精确容器名逐个杀,绝不按前缀展开)。
|
||||
|
||||
verified_only=True(默认):只杀 pid_verified 的容器——
|
||||
本机 host + 已知 pid 且 pid 明确已死,owner 死亡有本地
|
||||
证据;远端 host / pid 未知 / 未验证的只列不杀(返回里
|
||||
不含它们,日志里记 unverified 数量供运维人工处理)。
|
||||
verified_only=False:把未验证的也杀掉——只应在“本机
|
||||
单机部署、注册表 pid 列是老版本缺失”这类调用方已确认
|
||||
单机语义的场景显式开,跨主机场景开它等于按心跳过期
|
||||
猜死,绝不能默认开。
|
||||
本方法只写识别与杀容器,不改任务状态:任务状态由
|
||||
sweep_stale_tasks(心跳过期 -> interrupted)负责,两者
|
||||
分工,互不越权。执行器无 list_containers/kill_container
|
||||
时返回 [](Local/Cloud 执行器本来就没有容器可清)。
|
||||
返回已清理的精确容器名列表。
|
||||
"""
|
||||
list_fn = getattr(self.executor, "list_containers", None)
|
||||
kill_fn = getattr(self.executor, "kill_container", None)
|
||||
if list_fn is None or kill_fn is None:
|
||||
return []
|
||||
try:
|
||||
names = list_fn()
|
||||
except Exception as exc:
|
||||
log_event(self.logger, "worker_orphan_list_failed",
|
||||
message="orphan container list failed: %s" % exc)
|
||||
return []
|
||||
try:
|
||||
orphans = orphaned_containers(
|
||||
self.db_path, names, alive_seconds=alive_seconds,
|
||||
pid_alive=pid_alive)
|
||||
except Exception as exc:
|
||||
log_event(self.logger, "worker_orphan_identify_failed",
|
||||
message="orphan identify failed: %s" % exc)
|
||||
return []
|
||||
cleaned = []
|
||||
unverified = 0
|
||||
for item in orphans:
|
||||
if verified_only and not item.get("pid_verified"):
|
||||
unverified += 1
|
||||
continue
|
||||
try:
|
||||
if kill_fn(item["container"]):
|
||||
cleaned.append(item["container"])
|
||||
except Exception:
|
||||
continue
|
||||
if cleaned or unverified:
|
||||
log_event(self.logger, "worker_orphan_cleanup",
|
||||
message="cleaned orphan containers: %s;"
|
||||
" unverified skipped: %d"
|
||||
% (",".join(cleaned), unverified))
|
||||
return cleaned
|
||||
|
||||
def cleanup_stale_containers(self):
|
||||
"""启动清理兜底:清掉上一轮残留的 codex- 容器。
|
||||
|
||||
只在执行器有 cleanup_stale_containers 时做(Docker);
|
||||
列出或删除失败只记日志,绝不抛异常炸掉 worker。
|
||||
返回已清理的容器名列表(非 Docker 执行器返回 [])。
|
||||
注意:这是旧的任务正则兜底(删所有 codex-<任务 id>
|
||||
容器,不区分 owner),精确失联清理请用
|
||||
cleanup_orphaned_containers();本方法保留是为兼容
|
||||
认领记录缺失的老任务容器,不改其行为。
|
||||
"""
|
||||
cleanup = getattr(self.executor, "cleanup_stale_containers", None)
|
||||
if cleanup is None:
|
||||
@@ -4522,18 +4636,15 @@ class Worker:
|
||||
# Docker 超时取较小值:两个超时都不应静默延长另一个。
|
||||
# 注意绝不能改共享的 executor.timeout(并发下两个任务会
|
||||
# 互相覆盖对方的超时);超时只作为本次调用的参数传下去。
|
||||
effective_docker_timeout = None
|
||||
# 规则抽到模块级 docker_run_timeout()(纯函数可单测),
|
||||
# 这里只接线,行为与原内联完全一致。
|
||||
is_docker = isinstance(self.executor, DockerExecutor)
|
||||
if is_docker and _effective_task_timeout is not None:
|
||||
if self.executor.timeout is None:
|
||||
effective_docker_timeout = _effective_task_timeout
|
||||
else:
|
||||
try:
|
||||
effective_docker_timeout = min(
|
||||
float(_effective_task_timeout),
|
||||
float(self.executor.timeout))
|
||||
except (TypeError, ValueError):
|
||||
effective_docker_timeout = _effective_task_timeout
|
||||
effective_docker_timeout = (
|
||||
docker_run_timeout(
|
||||
_effective_task_timeout,
|
||||
self.executor.timeout if is_docker else None,
|
||||
no_time_limit=_no_time_limit)
|
||||
if is_docker else None)
|
||||
# P0 安全批后:Docker 同步路径起取消看门狗线程
|
||||
# (阻塞在 docker run 里时,库状态变 interrupted 就
|
||||
# 按容器名真杀;非 Docker 路径不启动,避免空线程)。
|
||||
@@ -6932,6 +7043,10 @@ def main(argv=None):
|
||||
parser.add_argument("--worker-id", default=None,
|
||||
help="多 worker 池身份(默认:主机名-进程号;"
|
||||
"生产部署应配稳定 id,否则重启后旧行变 stale)")
|
||||
parser.add_argument("--worker-address", default=None,
|
||||
help="worker 寻址串(可选,显式给才写进注册表"
|
||||
" workers.address,如运维约定的 host:port;"
|
||||
"不给则为 NULL,绝不猜、不含任何凭证)")
|
||||
parser.add_argument("--worker-capacity", type=int, default=None,
|
||||
help="worker 并发上限:同一 worker 内最多同时跑"
|
||||
"几个任务(未给时读环境变量"
|
||||
@@ -7065,6 +7180,7 @@ def main(argv=None):
|
||||
repos_dir=args.repos_dir,
|
||||
heartbeat_file=args.heartbeat_file,
|
||||
worker_id=args.worker_id,
|
||||
worker_address=args.worker_address,
|
||||
capacity=args.worker_capacity,
|
||||
host_name=args.host_name,
|
||||
pr_jobs_only=args.pr_jobs_only,
|
||||
|
||||
+214
-8
@@ -15,6 +15,21 @@
|
||||
存在就 UPDATE,绝不因“未注册”返回 404 报错。
|
||||
- alive 定义:last_seen 在阈值内(默认 120 秒,可经环境变量
|
||||
CLOUD_WORKER_ALIVE_SECONDS 调,测试可传参覆盖)。
|
||||
- 寻址信息(本轮新增):host(worker 所在主机名,worker 的
|
||||
--host-name,容器内 worker 绝不能用容器名冒充宿主名)、
|
||||
pid(worker 进程号,owner 死活的本地验证依据)、address
|
||||
(可选寻址串,如运维侧约定的 host:port,调用方显式给才
|
||||
有,绝不从环境变量里猜密钥类地址)。老行三列为 NULL,
|
||||
按“未知”处理,绝不猜成活着或死了。
|
||||
- 失联识别(本轮新增):stale_workers() 列心跳过期行;
|
||||
orphaned_containers() 把“容器名 codex-<任务 id>”精确映射
|
||||
到 tasks.claimed_by_worker(认领时 worker 写),只认领到
|
||||
失联 worker 且状态仍为 running/interrupted 的任务容器才
|
||||
算失联容器;名字不匹配任务 id 正则、任务行缺失、无认领
|
||||
记录、owner 仍存活的一律不算——绝不按 "codex-" 前缀删。
|
||||
pid_verified 另算:本机 host + 已知 pid 时用 pid_alive
|
||||
校验(默认 os.kill(pid, 0)),远端 host 或 pid 未知时
|
||||
为 False(列出来给运维看,不自动杀)。
|
||||
- 本模块不存、不打任何密钥:注册信息只有 worker_id、executor、
|
||||
capacity、last_seen,没有任何凭证字段。
|
||||
- host/pid/address(本轨新增):新库建表直接含三列;老库由
|
||||
@@ -46,6 +61,7 @@
|
||||
"""
|
||||
|
||||
import os
|
||||
import re
|
||||
import signal
|
||||
import socket
|
||||
import time
|
||||
@@ -78,6 +94,54 @@ def alive_threshold_seconds(override=None):
|
||||
return float(DEFAULT_ALIVE_SECONDS)
|
||||
|
||||
|
||||
def _existing_worker_columns(conn):
|
||||
"""workers 表现有列名集合;查不到时返回 None(调用方跳过补列)。
|
||||
|
||||
PG 走 information_schema,SQLite 走 PRAGMA table_info;两种
|
||||
查询失败都只回滚、不抛异常(老库迁移兜底,绝不因查列炸掉
|
||||
注册/心跳主路径)。
|
||||
"""
|
||||
try:
|
||||
rows = conn.execute(
|
||||
"SELECT column_name FROM information_schema.columns "
|
||||
"WHERE table_name = 'workers'").fetchall()
|
||||
return {r[0] for r in rows}
|
||||
except Exception:
|
||||
try:
|
||||
conn.rollback()
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
rows = conn.execute("PRAGMA table_info(workers)").fetchall()
|
||||
return {r[1] for r in rows}
|
||||
except Exception:
|
||||
try:
|
||||
conn.rollback()
|
||||
except Exception:
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
def _ensure_worker_address_columns(conn):
|
||||
"""老库补 host/pid/address 三列(缺哪个补哪个,幂等)。"""
|
||||
cols = _existing_worker_columns(conn)
|
||||
if cols is None:
|
||||
return
|
||||
for name, coltype in (("host", "TEXT"), ("pid", "INTEGER"),
|
||||
("address", "TEXT")):
|
||||
if name in cols:
|
||||
continue
|
||||
try:
|
||||
conn.execute(
|
||||
"ALTER TABLE workers ADD COLUMN %s %s" % (name, coltype))
|
||||
conn.commit()
|
||||
except Exception:
|
||||
try:
|
||||
conn.rollback()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def ensure_workers_table(db_path):
|
||||
"""建 workers 表(不存在时建,已存在时不动,老库补精度)。
|
||||
|
||||
@@ -107,6 +171,7 @@ def ensure_workers_table(db_path):
|
||||
"created_at DOUBLE PRECISION)"
|
||||
)
|
||||
conn.commit()
|
||||
_ensure_worker_address_columns(conn)
|
||||
_ensure_running_column(conn)
|
||||
_ensure_hostpid_columns(conn)
|
||||
# 老库升精度只在确实是 Postgres REAL 列时做一次:每次心跳都
|
||||
@@ -501,6 +566,21 @@ def reap_stale_workers(db_path, stale_seconds=None, local_host=None,
|
||||
return results
|
||||
|
||||
|
||||
def _coerce_pid(pid):
|
||||
"""pid 归一化:None/空串 -> None;可转 int 的转 int;否则 None。
|
||||
|
||||
绝不把非法 pid 写进库(写进去了后续的本地存活校验会拿
|
||||
它当进程号查,错 pid 比缺 pid 更危险)。
|
||||
"""
|
||||
if pid is None or pid == "":
|
||||
return None
|
||||
if isinstance(pid, bool):
|
||||
return None
|
||||
try:
|
||||
return int(pid)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
def register_worker(db_path, worker_id, executor=None, capacity=1,
|
||||
running=None, host=None, pid=None, address=None):
|
||||
"""注册(或刷新)一个 worker:不存在 INSERT,存在 UPDATE。
|
||||
@@ -509,8 +589,7 @@ def register_worker(db_path, worker_id, executor=None, capacity=1,
|
||||
(server 的 heartbeat 端点)据此返回 400。新列语义:新行
|
||||
按给定值写(未给为 NULL);已存在行上给定值覆盖、未给
|
||||
保留旧值——绝不因一次不带 host/pid 的注册把已有的身份
|
||||
洗成 NULL,也不编造 host/pid/address。
|
||||
"""
|
||||
洗成 NULL,也不编造 host/pid/address。 """
|
||||
if not worker_id or not isinstance(worker_id, str):
|
||||
raise ValueError("worker_id required")
|
||||
pid = _normalize_pid(pid)
|
||||
@@ -541,8 +620,7 @@ def register_worker(db_path, worker_id, executor=None, capacity=1,
|
||||
" VALUES(?, ?, ?, ?, ?, ?, ?, ?, ?)",
|
||||
(worker_id, executor, capacity,
|
||||
running if running is not None else 0,
|
||||
host, pid, address, now, now),
|
||||
)
|
||||
host, pid, address, now, now), )
|
||||
conn.commit()
|
||||
row = conn.execute(
|
||||
"SELECT * FROM workers WHERE worker_id = ?",
|
||||
@@ -562,6 +640,7 @@ def heartbeat_worker(db_path, worker_id, executor=None, capacity=None,
|
||||
应传本机 host 与本进程 pid,见 worker._registry_heartbeat),
|
||||
未给则保留旧值;address 没有新来源,绝不由本函数推导
|
||||
(例如不从 worker_id 或 socket 猜地址),只存显式给的值。
|
||||
显式给 None 与未给在心跳里无法区分(均按未给保留处理)。
|
||||
"""
|
||||
if not worker_id or not isinstance(worker_id, str):
|
||||
raise ValueError("worker_id required")
|
||||
@@ -580,8 +659,7 @@ def heartbeat_worker(db_path, worker_id, executor=None, capacity=None,
|
||||
" VALUES(?, ?, ?, ?, ?, ?, ?, ?, ?)",
|
||||
(worker_id, executor, capacity if capacity is not None
|
||||
else 1, running if running is not None else 0,
|
||||
host, pid, address, now, now),
|
||||
)
|
||||
host, pid, address, now, now), )
|
||||
else:
|
||||
old = _worker_to_dict(row)
|
||||
conn.execute(
|
||||
@@ -594,8 +672,7 @@ def heartbeat_worker(db_path, worker_id, executor=None, capacity=None,
|
||||
host if host is not None else old["host"],
|
||||
pid if pid is not None else old["pid"],
|
||||
address if address is not None else old["address"],
|
||||
now, worker_id),
|
||||
)
|
||||
now, worker_id), )
|
||||
conn.commit()
|
||||
row = conn.execute(
|
||||
"SELECT * FROM workers WHERE worker_id = ?",
|
||||
@@ -643,4 +720,133 @@ def _worker_to_dict(row):
|
||||
"address": _get("address"),
|
||||
"last_seen": _get("last_seen"),
|
||||
"created_at": _get("created_at"),
|
||||
"host": _get("host"),
|
||||
"pid": _get("pid"),
|
||||
"address": _get("address"),
|
||||
}
|
||||
|
||||
|
||||
def stale_workers(db_path, alive_seconds=None):
|
||||
"""心跳过期(失联)的 worker 行列表(每行已含 alive=False)。
|
||||
|
||||
只按 last_seen 判,不查 pid:pid 校验是 orphaned_containers
|
||||
里对“能不能自动杀”的加严,不是失联本身的定义(失联的
|
||||
定义与 sweep_stale_tasks 的任务心跳过期同款语义)。
|
||||
"""
|
||||
return [w for w in list_workers(db_path, alive_seconds)
|
||||
if not w.get("alive")]
|
||||
|
||||
|
||||
def default_pid_alive(pid):
|
||||
"""本地进程存活检查:os.kill(pid, 0) 成功或无权限都算活着
|
||||
(PermissionError 说明进程存在,只是归属别人);只有
|
||||
查无此进程算死了;其它异常算未知(返回 None,不猜)。
|
||||
pid 非法时返回 None。"""
|
||||
pid = _coerce_pid(pid)
|
||||
if pid is None:
|
||||
return None
|
||||
try:
|
||||
os.kill(pid, 0)
|
||||
return True
|
||||
except ProcessLookupError:
|
||||
return False
|
||||
except PermissionError:
|
||||
return True
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def _is_local_host(host):
|
||||
"""host 是否本机:空值按“无法确认”处理,不按本机猜。
|
||||
|
||||
注意与 pid_verified 的关系:host 为空时即使 pid 给了也
|
||||
不能断定该 pid 是本机的——跨主机同 pid 号毫无意义,
|
||||
所以空 host 一律 unverified。
|
||||
"""
|
||||
if not host or not isinstance(host, str):
|
||||
return False
|
||||
try:
|
||||
return host == socket.gethostname()
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
_TASK_CONTAINER_RE = re.compile(r"^codex-([0-9a-f]{32})$")
|
||||
|
||||
|
||||
def orphaned_containers(db_path, container_names,
|
||||
alive_seconds=None, pid_alive=None):
|
||||
"""识别失联 owner 的任务容器(纯识别,不杀任何容器)。
|
||||
|
||||
返回 list,每项 {container, task_id, worker_id, pid_verified}:
|
||||
- container:精确的容器名(原样返回,调用方按此名杀,
|
||||
绝不按前缀展开);
|
||||
- task_id:从容器名正则里精确抠出的 32 位 hex(不是前缀
|
||||
切出来的任意串);
|
||||
- worker_id:tasks.claimed_by_worker(认领时写);
|
||||
- pid_verified:True = 本机 host + 已知 pid 且 pid_alive
|
||||
明确报死(可自动杀);False = 未验证(远端 host、pid
|
||||
未知、pid_alive 未给或返回未知),只列不自动杀。
|
||||
排除项(任一即排除):容器名不匹配任务正则;tasks 表或
|
||||
claimed_by_worker 列不存在(无法映射时宁可漏,不猜);
|
||||
任务行缺失;任务状态不在 running/interrupted;owner 行
|
||||
缺失或仍存活(alive)。调用方杀之前仍须按返回的精确
|
||||
容器名逐个杀,绝不把本函数的返回再按前缀展开。
|
||||
"""
|
||||
if pid_alive is None:
|
||||
pid_alive = default_pid_alive
|
||||
stale = {w["worker_id"]: w for w in stale_workers(
|
||||
db_path, alive_seconds)}
|
||||
if not stale:
|
||||
return []
|
||||
conn = _db_connect(str(db_path))
|
||||
try:
|
||||
try:
|
||||
rows = conn.execute(
|
||||
"SELECT id, status, claimed_by_worker FROM tasks"
|
||||
).fetchall()
|
||||
except Exception:
|
||||
try:
|
||||
conn.rollback()
|
||||
except Exception:
|
||||
pass
|
||||
return []
|
||||
finally:
|
||||
conn.close()
|
||||
tasks = {}
|
||||
for row in rows:
|
||||
try:
|
||||
tasks[row["id"]] = (row["status"], row["claimed_by_worker"])
|
||||
except (KeyError, IndexError, TypeError):
|
||||
continue
|
||||
out = []
|
||||
for name in container_names or []:
|
||||
if not isinstance(name, str):
|
||||
continue
|
||||
m = _TASK_CONTAINER_RE.match(name)
|
||||
if not m:
|
||||
continue
|
||||
task_id = m.group(1)
|
||||
info = tasks.get(task_id)
|
||||
if info is None:
|
||||
continue
|
||||
status, claimed = info
|
||||
if status not in ("running", "interrupted"):
|
||||
continue
|
||||
if not claimed or claimed not in stale:
|
||||
continue
|
||||
worker = stale[claimed]
|
||||
verified = False
|
||||
pid = worker.get("pid")
|
||||
if _is_local_host(worker.get("host")) and pid is not None:
|
||||
try:
|
||||
verified = pid_alive(pid) is False
|
||||
except Exception:
|
||||
verified = False
|
||||
out.append({
|
||||
"container": name,
|
||||
"task_id": task_id,
|
||||
"worker_id": claimed,
|
||||
"pid_verified": verified,
|
||||
})
|
||||
return out
|
||||
Reference in new issue
Block a user