6089 lines
310 KiB
Python
6089 lines
310 KiB
Python
"""云端 worker：把控制面的 queued 任务接通到 host.py 执行。
|
||
|
||
零第三方依赖，只用 Python 标准库；不改 host.py 行为、不动
|
||
~/.codex 配置、不碰 wechat-bridge。
|
||
|
||
流程（与 server.py 的任务状态机对齐）：
|
||
queued -> running -> completed / failed / interrupted
|
||
|
||
1. 认领：pick_next() 用 queue.ready_tasks() 按依赖拓扑 + 优先级
|
||
排序，跳过依赖未完成（未 completed/merged）的任务；再过
|
||
queue.budget_exceeded() 预算门，超限不派发、任务保持 queued。
|
||
claim() 用 `WHERE status='queued'` 的 CAS 更新把任务标 running
|
||
并记 last_heartbeat / updated_at，避免两个 worker 重复认领。
|
||
2. 缺 key 门（执行前必查）：只检查配置的 env_key 名在进程环境
|
||
里是否存在且非空，绝不读出、打印、写进文件或日志 key 的值。
|
||
缺 key 时任务直接标 failed，错误固定为
|
||
`Missing environment variable: <env_key 名>`，与之前
|
||
real-nightly-tests automation 的失败模式一致，绝不静默假装跑了。
|
||
env_key 名的来源：显式参数 > codex 配置文件里的 env_key 行
|
||
（只读名字）> 默认 XCM_API_KEY（中转站 relay 的约定名）。
|
||
3. 执行：经 LocalExecutor 调 host.py 真实子命令。dispatch
|
||
（建 host 任务并跑一轮 turn）是首次派发；dispatch 报
|
||
“任务已存在”时改用 run（在已有 host 任务上再跑一轮）续跑。
|
||
host 任务名由 cloud 任务 id 派生（见 sandbox._host_name），
|
||
同一任务每次算出同一名字，续跑才能找到同一个 host 任务。
|
||
4. 写回：server 的 tasks 表没有 result/error 列，所以执行结果
|
||
的持久化分三处，各复用已有结构、不另起字段名：
|
||
- tasks 表：最终状态 + updated_at（取消时为 interrupted）；
|
||
- cost.py CostStore：每轮实际执行记一条成本记录；执行器带回
|
||
上游真实用量（host.py 从 tokenUsage/updated 事件汇总）时记
|
||
source="actual"（input/output/cached 全是实测值，usd 按价目
|
||
表算、无价模型记 None）；没带回用量时回落 cost.py 估算记
|
||
estimated，估算绝不冒充 actual；total_tokens 由
|
||
CostStore.record 按 input+output 算；
|
||
- models.py：可 fallback 的失败（404/429/5xx/timeout，401/400
|
||
不算）。默认（未配 --fallback-models）时 worker 只记录
|
||
fallback 事件（record_fallback() 记 model_fallback 事件，
|
||
fallback_count 写进成本记录），不自动换模型重试——自动重试
|
||
等于未经批准烧 relay 额度。显式配了 --fallback-models 时，
|
||
第一次失败且可 fallback 就真的换列表里的第一个模型再跑一次
|
||
（最多 1 次，同一任务、同一 worktree/分支，dispatch 报
|
||
“任务已存在”时走现有的 run 续跑路径，模型经 host.py 的
|
||
顶层 --model 参数传过去，不改 host.py 签名）；第二次的结果
|
||
就是最终结果，actual_model 写成最后跑的模型，fallback_count=1。
|
||
审计：认领/完成/失败/取消/缺 key 各记一条 AuthStore 审计
|
||
（user_id="system"，与 server.sweep_stale_tasks 的做法一致），
|
||
审计失败不影响任务状态（best-effort）。
|
||
|
||
取消：worker 的 cancel() 走 server 的 cancel 语义：把
|
||
queued/running 任务直接标 interrupted（已终态的任务不动），
|
||
且正在跑的子进程会真杀——子进程以进程组启动（start_new_session），
|
||
取消时先 SIGTERM 给整个进程组、等 2 秒、仍活着再 SIGKILL，
|
||
杀完后任务保持 interrupted，执行结果不覆盖它。run_one() 在
|
||
执行返回后会先查任务是否已被取消，已被取消的一律保持
|
||
interrupted，不用执行结果覆盖它。
|
||
|
||
任务级超时与失败重试（生产级，见 Worker.__init__ 参数）：
|
||
- task_timeout（默认 1800 秒，CLI --task-timeout 可调）是
|
||
worker 级的每任务截止时间，与下面两个参数不要混淆：
|
||
--docker-timeout 是执行器级的单次 docker run 超时（Docker
|
||
Executor 自身的 timeout），--max-seconds 是 run_loop 的
|
||
整体墙钟上限（不是每任务）。三者互不覆盖：Docker 路径下
|
||
worker 取 min(task_timeout, docker_timeout)（docker_timeout
|
||
未设时取 task_timeout）作为本次 docker run 的超时，超时后
|
||
真杀容器、任务置 failed（超时是失败，不是取消，绝不标
|
||
interrupted）；Local popen 路径下 worker 在轮询循环里到
|
||
截止时间就杀进程组、同样置 failed；Local 同步路径（假
|
||
runner / 无超时的 subprocess.run）无法在中途抢占一个已阻塞
|
||
的调用，超时只能靠执行器自身的 timeout，这是诚实边界，
|
||
不要假装同步路径也能中途杀掉。
|
||
- 对齐 host.py 的 turn_timeout 约定：task_timeout 传 0 时
|
||
归一化为 None（不限时，worker 级不设截止），与 host 的
|
||
“0 = 不限时”同约定；默认 1800 不变，不改默认就不会悄悄
|
||
挂起自动化。worker 级不限时时，Docker 路径仍受执行器
|
||
docker_timeout 约束（timeout=None 时 _call_runner 用
|
||
executor 自身的 timeout），不是全链路无限。
|
||
- max_retries（默认 1 次重试，CLI --max-retries 可调，0 表示
|
||
不重试）是对模型任务失败的有限重试：取消不重试；可 fallback
|
||
的失败优先走真实 fallback（见下），不再做同模型重试（fallback
|
||
本身就是模型级的重试，再同模型重试会重复烧额度）；其它失败
|
||
（含 worker 超时）重试前按 retry_backoff_seconds（默认 5 秒，
|
||
测试可注入 0）等待，重试次数记在审计（worker_retry）与
|
||
最终 error_text（[retries=N] 前缀）里，不另起新表；重试
|
||
耗尽后任务才最终 failed，成本记录与 usage_turns 只记一次
|
||
- 真实 fallback（CLI --fallback-models，默认空 = 关闭）：一次
|
||
尝试失败且可 fallback（404/429/5xx/timeout，401/400 不算，
|
||
取消不算）时，若配了 fallback 列表，就记 model_fallback 事件
|
||
与 worker_fallback 审计，然后换列表里的第一个模型再跑一次
|
||
（最多 1 次，与 max_retries 无关，max_retries=0 时 fallback
|
||
照样跑）；第二次的结果就是最终结果，actual_model 写成最后
|
||
跑的模型，fallback_count=1，两次都失败时 error_text 同时写清
|
||
两次原因；未配列表时保持老行为（只记事件、直接 failed）。
|
||
（按最终结果记，不按尝试次数重复记）。
|
||
- 容器清理兜底：Worker.cleanup_stale_containers() 列出
|
||
codex- 前缀的残留容器并逐个 kill+rm，失败只记日志不炸
|
||
worker；run_loop 启动时与 CLI 启动时各调一次（run_all /
|
||
__init__ 里不调，避免单测构造 Worker 时多余 I/O）。
|
||
|
||
可注入：runner（假 CLI，见 sandbox.LocalExecutor）与 env 映射
|
||
均可注入，单测不跑真实模型回合、不联网、不登录任何服务。
|
||
|
||
执行器选择：Worker 默认用 LocalExecutor；传入 DockerExecutor
|
||
（或 CLI --executor docker）时，派发走容器内 host.py，取消时
|
||
按任务派生的容器名 docker kill，不只改数据库状态；本机无
|
||
docker 时 DockerExecutor 在调 runner 前就明确报错，任务标
|
||
failed 且 error_text 含 docker 缺失，绝不静默降级为本地执行。
|
||
|
||
项目仓库：run_one 派发前查任务所属项目的 repo_url（projects
|
||
表），非空时经 cloud.repos.resolve_repo 在 repos_dir（CLI
|
||
--repos-dir，默认 ./repos）下 clone/fetch 该项目仓库，设
|
||
task["repo"] 再交给执行器；Local 与 Docker 执行器都已读
|
||
task["repo"]，同一条路径。repo_url 为空时走原有默认 repo；
|
||
解析失败时任务直接 failed，错误文本写清是仓库解析失败。
|
||
Docker 下 repos_dir 必须放在 extra-mounts 已挂载的目录下，
|
||
否则容器内 repo 路径不存在（见 cloud/README.md）。
|
||
推送到远端不在本轮范围：任务在本地检出上执行、本地 worktree
|
||
合并仍按 host.py 现有流程，git push 需要仓库写凭证，控制面
|
||
不存、不推。
|
||
"""
|
||
|
||
import argparse
|
||
import hashlib
|
||
import json
|
||
import os
|
||
from datetime import datetime, timezone
|
||
import re
|
||
import shutil
|
||
import signal
|
||
import socket
|
||
import subprocess
|
||
import sys
|
||
import threading
|
||
import time
|
||
import uuid
|
||
from types import SimpleNamespace
|
||
|
||
from cloud.auth import AuthStore
|
||
from cloud.coordinator import build_report as _coord_build_report
|
||
from cloud.coordinator import DEFAULT_POLL_INTERVAL as _COORD_POLL_INTERVAL
|
||
from cloud.coordinator import DEFAULT_TIMEOUT_SECONDS as _COORD_TIMEOUT
|
||
from cloud.coordinator import ensure_columns as _coord_ensure_columns
|
||
from cloud.coordinator import parse_parts as _coord_parse_parts
|
||
from cloud.coordinator import nonterminal_children as _coord_nonterminal
|
||
from cloud.cost import CostStore, estimate_tokens, price_usd_micros, \
|
||
sum_usage, unpriced_token_cap
|
||
from cloud.db import connect as _db_connect
|
||
from cloud.logging_config import get_logger, log_event, setup_logging
|
||
from cloud.models import ModelRouter, is_fallbackable, record_fallback
|
||
from cloud.auto_memory import AutoMemoryStore
|
||
from cloud.auto_memory import build_injection_prefix as _auto_prefix
|
||
from cloud.auto_memory import process_due_tasks as _auto_process_due
|
||
from cloud.marker_tools import ancestry_relation as _marker_ancestry
|
||
from cloud.marker_tools import check_control_gate as _marker_gate
|
||
from cloud.marker_tools import count_direct_spawn_children as _marker_count_direct
|
||
from cloud.marker_tools import fetch_task_snapshot as _marker_snapshot
|
||
from cloud.marker_tools import format_snapshot as _marker_format
|
||
from cloud.marker_tools import marker_report_header as _marker_header
|
||
from cloud.marker_tools import new_child_id as _marker_new_id
|
||
from cloud.marker_tools import parent_spawn_generation as _marker_generation
|
||
from cloud.marker_tools import MAX_DIRECT_CHILDREN as _MARKER_MAX_DIRECT
|
||
from cloud.marker_tools import MAX_SPAWN_DEPTH as _MARKER_MAX_DEPTH
|
||
from cloud.marker_tools import TERMINAL_STATUSES as _MARKER_TERMINAL
|
||
from cloud.peer_messages import deliver_peer_message as _deliver_peer_message
|
||
from cloud.peer_messages import ensure_followups_table as _ensure_followups_table
|
||
from cloud.peer_messages import peer_message_prefix as _peer_prefix
|
||
from cloud.peer_messages import row_get as _row_get
|
||
from cloud.sub2api import ensure_user_gateway_key as _ensure_sub2api_key
|
||
from cloud.websearch import ensure_codex_home_websearch as _ensure_websearch_mcp
|
||
from cloud.websearch import ensure_websearch_mcp_env as _ensure_websearch_mcp_env
|
||
from cloud.imagegen import ensure_codex_home_imagegen as _ensure_imagegen_mcp
|
||
from cloud.imagegen import ensure_imagegen_mcp_env as _ensure_imagegen_mcp_env
|
||
from cloud.skill_seed import ensure_codex_home_skills as _ensure_skills
|
||
from cloud.sub2api import gateway_name as _gateway_name
|
||
from cloud.sub2api import gateway_origin as _gateway_origin
|
||
from cloud.sub2api import list_gateway_models as _list_gateway_models
|
||
from cloud.pr import PrSkipped, PrStepError
|
||
from cloud.pr_exec import (claim_pr_job, execute_task_pr, finish_pr_job,
|
||
mark_pr_job_needs_local_worker)
|
||
from cloud.queue import budget_exceeded, ready_tasks
|
||
from cloud.repos import RepoError, checkout_dir, resolve_repo
|
||
from cloud.sandbox import CloudExecutor, DockerExecutor, LocalExecutor
|
||
from cloud.uploads import upload_root as _upload_root
|
||
from cloud.sandbox import _default_runner as _sandbox_default_runner
|
||
from cloud.caps import effective_capacity, resolve_capacity
|
||
from cloud.workers import heartbeat_worker, register_worker
|
||
|
||
# 与 server.TERMINAL_STATUSES 对齐：这些状态的任务不再被认领或取消。
|
||
TERMINAL_STATUSES = frozenset(
|
||
{"completed", "failed", "interrupted", "merged", "merge-failed"})
|
||
COMPLETED_STATUSES = frozenset({"completed", "merged"})
|
||
DEFAULT_ENV_KEY = "XCM_API_KEY"
|
||
# 默认模型（2026-10-27 起为 gpt-5.6-terra）：此前默认的
|
||
# gpt-6.1-sol 路由当日实测持续 503，terra 同日实测 1 秒完成；
|
||
# 任务未指定模型时 worker 与 host.py 都落到这里。
|
||
DEFAULT_MODEL = "gpt-5.6-terra"
|
||
# P1 执行面：goal 轮次上限（有限轮次形态）。goal_mode=1 的 0 轮
|
||
# （模型宣告才结束）不在此限——它的上界是配额（配额看门狗与
|
||
# 认领/边界门），不是轮次；旧验收形态的 goal_max_rounds 必须
|
||
# 在 1..GOAL_MAX_ROUNDS_CAP 之间，超限的任务 worker 在认领后
|
||
# 直接 failed（建任务时的 400 校验在 server.py，属另一路文件，
|
||
# 本路只做执行侧防线，见 run_one 的 goal 上限门）。
|
||
GOAL_MAX_ROUNDS_CAP = 100
|
||
# P1 执行面：无配额行时的默认配额（微美元），与 server.py
|
||
# _default_quota_usd_micros() 同值（$10/月，环境变量
|
||
# CLOUD_DEFAULT_QUOTA_USD_MICROS 可覆盖）。P0 前“无行 =
|
||
# 不限额”已废：worker 侧查不到行时按此默认额度算，显式
|
||
# NULL 行才是不限额（与服务端 _quota_status 同口径）。
|
||
# host.py dispatch 的原话是 "task 已存在: <name>，先换名或 discard 它"
|
||
#（英文 task + 已存在），取共有子串做判断，不依赖标点。
|
||
_ALREADY_EXISTS_MARKER = "已存在"
|
||
|
||
|
||
def _uuid_hex():
|
||
"""uuid hex（task_steps id 用，两后端都不依赖自增语义）。"""
|
||
return uuid.uuid4().hex
|
||
|
||
|
||
def _worker_host_name(executor, task):
|
||
"""host 任务名：优先 executor._host_name（与 sandbox 同款
|
||
派生），没有 executor 时回落任务 id（uuid hex 本就是安全名）。"""
|
||
fn = getattr(executor, "_host_name", None)
|
||
if callable(fn):
|
||
try:
|
||
return fn(task)
|
||
except Exception:
|
||
pass
|
||
return str(task.get("id"))
|
||
|
||
|
||
def _default_quota_usd_micros():
|
||
"""无配额行时的默认配额（微美元），与 server 侧同值。
|
||
|
||
server.py 的 _default_quota_usd_micros() 在另一路文件里，
|
||
worker 不跨模块 import 它（避免循环 import），这里复刻
|
||
同款解析：环境变量 CLOUD_DEFAULT_QUOTA_USD_MICROS，非法
|
||
值回落 10_000_000。"""
|
||
try:
|
||
return max(0, int(os.environ.get(
|
||
"CLOUD_DEFAULT_QUOTA_USD_MICROS", "10000000")))
|
||
except (TypeError, ValueError):
|
||
return 10000000
|
||
|
||
|
||
def _code_signature(host_py_path=None):
|
||
"""多文件签名：worker/sandbox/coordinator/cost/server/host
|
||
各取 (mtime_ns, size) 组成的元组；任一文件取不到即 None。
|
||
|
||
P1 执行面扩面：旧版只签 worker.py，部署只换了 sandbox 或
|
||
host.py 时旧 worker 会继续用旧代码抢任务。host.py 取
|
||
host_py_path（真实文件时用它，否则用 cloud/ 上一级的
|
||
host.py）；单测常传 /fake/host.py 这种不存在的路径，此时
|
||
回落到仓库 host.py，不因此判签名缺失。取不到时返回 None，
|
||
调用方必须 fail-closed（不认领、不执行），绝不按“没变”
|
||
放行——旧版“取不到当没变”是 fail-open，已废。"""
|
||
here = os.path.dirname(os.path.abspath(__file__))
|
||
candidates = [
|
||
os.path.join(here, "worker.py"),
|
||
os.path.join(here, "sandbox.py"),
|
||
os.path.join(here, "coordinator.py"),
|
||
os.path.join(here, "cost.py"),
|
||
os.path.join(here, "server.py"),
|
||
]
|
||
if host_py_path and os.path.isfile(host_py_path):
|
||
candidates.append(os.path.abspath(host_py_path))
|
||
else:
|
||
candidates.append(os.path.join(os.path.dirname(here), "host.py"))
|
||
sig = []
|
||
try:
|
||
for path in candidates:
|
||
st = os.stat(path)
|
||
sig.append((st.st_mtime_ns, st.st_size))
|
||
except OSError:
|
||
return None
|
||
return tuple(sig)
|
||
|
||
|
||
def _missing_key_error(env_key):
|
||
"""缺 key 的标准错误：只含 env_key 名，与 relay 侧的报错一致。"""
|
||
return "Missing environment variable: %s" % env_key
|
||
|
||
|
||
def resolve_env_key(env_key=None, codex_config_path=None):
|
||
"""确定模型 key 对应的环境变量名（只返回名字，从不返回值）。
|
||
|
||
显式参数优先；其次读 codex 配置文件（如 relay 的 config.toml）
|
||
里的 `env_key = "..."` 行，只取名字；文件不存在或无此行时
|
||
用默认名。绝不读 ~/.codex/config.toml 之外的默认位置——
|
||
调用方要读哪个配置文件，必须显式传 codex_config_path。
|
||
"""
|
||
if env_key:
|
||
return env_key
|
||
if codex_config_path:
|
||
try:
|
||
with open(codex_config_path, encoding="utf-8") as f:
|
||
text = f.read()
|
||
except (OSError, UnicodeDecodeError):
|
||
text = ""
|
||
match = re.search(
|
||
r'^\s*env_key\s*=\s*"([^"]+)"', text, re.MULTILINE)
|
||
if match:
|
||
return match.group(1)
|
||
return DEFAULT_ENV_KEY
|
||
|
||
|
||
def _merge_env(base, override):
|
||
"""两份 env 合并：override 优先；任一为空时返回另一份原样
|
||
（保持旧调用在双 None 时传 None，不改执行器行为）。"""
|
||
if not base:
|
||
return override
|
||
if not override:
|
||
return base
|
||
merged = dict(base)
|
||
merged.update(override)
|
||
return merged
|
||
|
||
|
||
def _resolve_model(model=None, config_path=None):
|
||
"""模型名：显式参数 > host 配置文件里的 model > host 默认值。"""
|
||
if model:
|
||
return model
|
||
if config_path:
|
||
try:
|
||
with open(config_path, encoding="utf-8") as f:
|
||
cfg = json.load(f)
|
||
except (OSError, ValueError):
|
||
cfg = {}
|
||
if isinstance(cfg, dict) and cfg.get("model"):
|
||
return cfg["model"]
|
||
return DEFAULT_MODEL
|
||
|
||
|
||
def _task_from_row(row):
|
||
depends_on = row["depends_on"]
|
||
try:
|
||
depends_on = json.loads(depends_on) if depends_on else []
|
||
except (TypeError, ValueError):
|
||
depends_on = []
|
||
if not isinstance(depends_on, list):
|
||
depends_on = [depends_on]
|
||
def _col(name, default=None):
|
||
try:
|
||
return row[name]
|
||
except (KeyError, IndexError, TypeError):
|
||
return default
|
||
return {
|
||
"id": row["id"],
|
||
"project_id": row["project_id"],
|
||
"prompt": row["prompt"],
|
||
"status": row["status"],
|
||
"idempotency_key": row["idempotency_key"],
|
||
"depends_on": depends_on,
|
||
"priority": row["priority"],
|
||
"created_by": row["created_by"],
|
||
"created_at": row["created_at"],
|
||
"updated_at": row["updated_at"],
|
||
"last_heartbeat": row["last_heartbeat"],
|
||
# 老库行缺 executor 列时给 None（表示任意 worker 可认领），
|
||
# 不炸；列本身由 _ensure_result_columns 在写回时补。
|
||
"executor": _col("executor"),
|
||
# goal 字段同款 _col 容错：老库行与 load_tasks 的显式列清单
|
||
# 里没有时给 None（= 普通任务），不炸；run_one 认领后
|
||
# get_task 重读 SELECT *，goal 任务一定能拿到完整字段。
|
||
"goal": _col("goal"),
|
||
"verify_command": _col("verify_command"),
|
||
"goal_max_rounds": _col("goal_max_rounds"),
|
||
# 新 goal 语义：goal_mode=1 时 prompt 即目标，无验收命令、
|
||
# 不限轮次；老行缺列时 None = 按 verify_command 判定。
|
||
"goal_mode": _col("goal_mode"),
|
||
# 任务级选项：老行缺列时 None = 跟随配置（见 server 侧）。
|
||
"model": _col("model"),
|
||
"effort": _col("effort"),
|
||
# 派生默认：老行缺列时 None = 没有单独声明，派生时
|
||
# 先取它、再取本任务的 model / effort。
|
||
"child_model": _col("child_model"),
|
||
"child_effort": _col("child_effort"),
|
||
"approval": _col("approval"),
|
||
"plan": _col("plan"),
|
||
"forked_from": _col("forked_from"),
|
||
# 环境：老行缺列时 None = 不指定环境，走项目仓库路径。
|
||
"environment_id": _col("environment_id"),
|
||
# 续跑消息：老行缺列时 None = 没有待续跑消息。
|
||
"resume_message": _col("resume_message"),
|
||
# G10：goal 断点续跑标记，老行缺列时 0 = 普通排队。
|
||
"goal_resume_pending": _col("goal_resume_pending") or 0,
|
||
# 长任务自动恢复：用户暂停标记、中断原因、自动续跑
|
||
# 次数、不限时标记，老行缺列时按默认（可恢复、不限时关）。
|
||
"paused_by_user": _col("paused_by_user") or 0,
|
||
"interrupted_reason": _col("interrupted_reason"),
|
||
"auto_resume_count": _col("auto_resume_count") or 0,
|
||
"no_time_limit": _col("no_time_limit") or 0,
|
||
# 协调员：老行缺列时 coordinator_parts 为 []（= 普通任务），
|
||
# parent_task_id/coordinator_part 缺列时为 None。
|
||
"coordinator_parts": _coord_parse_parts(_col("coordinator_parts")),
|
||
"parent_task_id": _col("parent_task_id"),
|
||
"coordinator_part": _col("coordinator_part"),
|
||
}
|
||
|
||
|
||
class Worker:
|
||
"""单机 worker：一轮认领一个 queued 任务并执行到底。
|
||
|
||
前置条件：db_path 须已由 cloud.server.init_db() 建好 tasks 表；
|
||
本模块不自建 server 的表（成本表与审计表由 CostStore/AuthStore
|
||
在首次使用时自建，与 server 的做法一致）。
|
||
"""
|
||
|
||
def __init__(self, db_path, host_py_path, config_path=None, repo=None,
|
||
runner=None, env=None, env_key=None,
|
||
codex_config_path=None, model=None, router=None,
|
||
max_turns=None, max_cost_usd=None, executor=None,
|
||
cost_store=None, process_factory=None,
|
||
task_timeout=1800, max_retries=1,
|
||
retry_backoff_seconds=5.0, repos_dir="./repos",
|
||
repos_runner=None, heartbeat_file=None,
|
||
worker_id=None, capacity=None, fallback_models=None,
|
||
tenant_isolation=False, tenants_root=None,
|
||
tenant_template_home=None, tenant_template_repo=None,
|
||
egress_proxy_port=0, claim_executor_only=False,
|
||
websearch_mcp=False, seed_skills=False,
|
||
imagegen_mcp=False,
|
||
host_name=None, pr_jobs_only=False,
|
||
upload_dir=None,
|
||
digest_interval_seconds=3600,
|
||
outbox_backoff_seconds=60,
|
||
outbox_max_attempts=3,
|
||
quota_poll_interval_seconds=30,
|
||
auto_resume_max=50,
|
||
ignore_host_budget=True):
|
||
self.db_path = str(db_path)
|
||
self.host_py_path = host_py_path
|
||
self.config_path = config_path
|
||
self.repo = repo
|
||
# G11/G12：摘要与 outbox 的后台节奏。digest_interval 是
|
||
# worker 循环里两次摘要发送的最短间隔（默认 3600 秒，
|
||
# 与 push.DIGEST_INTERVAL_SECONDS 同值）；outbox 的重试
|
||
# 节奏由每行 next_attempt_at 控（默认 60 秒退避），
|
||
# 这里的两个参数只透传给 process_outbox_due。
|
||
self.digest_interval_seconds = digest_interval_seconds
|
||
self.outbox_backoff_seconds = outbox_backoff_seconds
|
||
self.outbox_max_attempts = outbox_max_attempts
|
||
# G8：配额轮询间隔（秒），Local popen 与 Docker 路径共用，
|
||
# 缺省 30；单测可传很小的值加速，生产不要改小（会打爆库）。
|
||
# None 当 30 用。
|
||
self.quota_poll_interval_seconds = (
|
||
30 if quota_poll_interval_seconds is None
|
||
else quota_poll_interval_seconds)
|
||
# G9：worker 路径默认忽略 host 的 max_daily_turns 预算。
|
||
# 云端已有 per-user USD 配额 + worker 认领/轮询门，host
|
||
# 的单机 turn 数预算再拦一次只会误拦云端已批的任务；
|
||
# standalone 直接跑 host.py 时不受此开关影响，预算照常。
|
||
self.ignore_host_budget = bool(ignore_host_budget)
|
||
self._last_digest_sent = 0.0
|
||
# env 只用于 key 存在性检查：默认接进程环境，单测可注入
|
||
# 一个普通 dict；worker 永远不把 env 里的值写出。
|
||
self._env = env if env is not None else os.environ
|
||
self.env_key = resolve_env_key(env_key, codex_config_path)
|
||
self.model = _resolve_model(model, config_path)
|
||
if router is not None and not isinstance(router, ModelRouter):
|
||
raise ValueError("router 必须是 cloud.models.ModelRouter")
|
||
self.router = router
|
||
# 真实 fallback 换模型列表：默认空 = 关闭，只记事件不换模型
|
||
# （老行为，生产 worker 未配该参数时绝不突然多花模型钱）。
|
||
# 归一化：去空白、去重保序、去掉与主模型相同的项（换成同一
|
||
# 个模型不叫 fallback，只会白烧一次额度）；fallback 总次数
|
||
# 最多 1 次，所以运行期只用列表的第一个元素，后面的只是
|
||
# 备用声明，不会被依次试下去。
|
||
self.fallback_models = []
|
||
for _m in (fallback_models or []):
|
||
if not isinstance(_m, str):
|
||
continue
|
||
_m = _m.strip()
|
||
if _m and _m != self.model and _m not in self.fallback_models:
|
||
self.fallback_models.append(_m)
|
||
self.max_turns = max_turns
|
||
self.max_cost_usd = max_cost_usd
|
||
self.usage_turns = 0
|
||
self.usage_cost_usd = 0.0
|
||
if executor is not None:
|
||
self.executor = executor
|
||
else:
|
||
self.executor = LocalExecutor(
|
||
host_py_path, config_path=config_path, runner=runner,
|
||
repo=repo, ignore_host_budget=self.ignore_host_budget)
|
||
# 外部传入的执行器（CLI 的 DockerExecutor 等）同样同步
|
||
# G9 开关：worker 的 ignore_host_budget 是唯一口径，执行
|
||
# 器自己的缺省值不另起一套。假执行器没有该属性时跳过。
|
||
if hasattr(self.executor, "ignore_host_budget"):
|
||
try:
|
||
self.executor.ignore_host_budget = self.ignore_host_budget
|
||
except Exception:
|
||
pass
|
||
self._cost_store = cost_store
|
||
self._cancel_requested = set()
|
||
# 真取消用：process_factory 可注入（单测用它返回一个假长跑
|
||
# 子进程，真实执行时走 LocalExecutor.popen_task 进程组启动）。
|
||
self.process_factory = process_factory
|
||
# 并发运行状态：task_id -> {"proc": Popen|None,
|
||
# "container": 容器名|None}，受 _running_lock 保护。
|
||
# 旧版只有单任务三个字段（_running_proc 等），capacity=1
|
||
# 时够用；capacity>1 后多个任务同时跑，单字段会被后
|
||
# 一个任务覆盖，导致 cancel 杀错、退出时漏杀。所有读写
|
||
# 都走下面的 _running_set/_running_clear/_running_snapshot。
|
||
self._running = {}
|
||
self._running_lock = threading.Lock()
|
||
# 容量门被拦日志的节流时间戳：_claim 的容量门在池线程
|
||
# 自旋时会高频触发，不节流会把日志打爆；30 秒内只记
|
||
# 一次，够排查“worker 活着却再也认领不了”即可。
|
||
self._capacity_refused_logged_at = 0.0
|
||
# DB 写锁：cost 记录共用一条连接（CostStore._conn），
|
||
# usage 计数也是共享变量；模型回合本身在子进程/容器里
|
||
# 跑不受这把锁影响，锁只覆盖毫秒级的写库操作。
|
||
self._db_lock = threading.RLock()
|
||
self._loop_stop = False
|
||
self.logger = get_logger("cloud.worker")
|
||
# 任务级超时 / 失败重试（见模块 docstring 的生产级说明）。
|
||
# 对齐 host turn_timeout：0 归一化为 None（不限时），
|
||
# 负数同理；默认 1800 不变。归一化后 _run_attempt 里
|
||
# task_timeout is None 的分支本来就走“无 worker 级截止”
|
||
#（Docker 路径仍受 executor docker_timeout 约束），
|
||
# 不用再改各执行路径；不归一化的话 0 会在 popen 轮询
|
||
# 里第一轮就触发“已到截止时间”而立即超时，是反直觉的 bug。
|
||
if task_timeout is not None and task_timeout <= 0:
|
||
task_timeout = None
|
||
self.task_timeout = task_timeout
|
||
self.max_retries = max_retries
|
||
self.retry_backoff_seconds = retry_backoff_seconds
|
||
# 长任务自动恢复上限（次数，不是时间）：同一任务被
|
||
# 系统中断后自动续跑超过该次数即停住、等人工处理，
|
||
# 防确定性失败（每次续跑立刻又中断）无限烧额度；
|
||
# 正常部署重启/偶发中断远达不到 50 次。
|
||
self.auto_resume_max = auto_resume_max
|
||
# 项目仓库检出根目录：CLI --repos-dir 可调（默认 ./repos）；
|
||
# Docker 执行器下它必须放在 extra-mounts 已挂载的目录下，
|
||
# 否则容器内 task["repo"] 路径不存在——见 cloud/README.md。
|
||
# repos_runner 可注入（单测用假 git runner），缺省走真实 git。
|
||
self.repos_dir = repos_dir
|
||
self.repos_runner = repos_runner
|
||
# 多租户隔离（仅 Docker 执行器有效，Local 执行器在本机
|
||
# 直接跑 host.py，同一用户下本来就没有容器边界，flag
|
||
# 对它不起作用）：开启后每个项目用独立的宿主目录
|
||
# tenants_root/<project_id>/{work,codex-home,test-repo,
|
||
# <project_id>}，容器内路径保持与原来完全一致（/app/work、
|
||
# /var/lib/.../codex-home 等），只是宿主侧的挂载源按
|
||
# 项目分开——容器内看不到其它项目的任何文件。codex-home
|
||
# 只播种 config.toml（relay 配置，不含任何密钥值），
|
||
# 绝不复制 thread_history（那是其它任务的 prompt 与结果）。
|
||
self.tenant_isolation = tenant_isolation
|
||
self.tenants_root = tenants_root or \
|
||
"/var/lib/codex-app-server/tenants"
|
||
self.tenant_template_home = tenant_template_home or \
|
||
"/var/lib/codex-app-server/codex-home"
|
||
self.tenant_template_repo = tenant_template_repo or \
|
||
"/var/lib/codex-app-server/test-repo"
|
||
# egress 代理端口（0 = 不启用）：启用后 Docker 任务容器
|
||
# 的 HTTP(S) 出口全部走宿主上的过滤代理（网关 IP + 该
|
||
# 端口），代理只放行白名单域名；网关查不到时任务直接
|
||
# failed，绝不无代理跑（无代理等于绕过白名单）。
|
||
self.egress_proxy_port = egress_proxy_port or 0
|
||
# 严格 executor 认领：只认领 executor 恰好等于本 worker
|
||
# 类型的任务，绝不认领 executor 为 NULL 的任务。默认
|
||
# False（老行为：NULL 任务任意 worker 可认领）。宿主级
|
||
# 沙箱 worker 必须开这个，否则它会抢走本该进 Docker
|
||
# 容器的 NULL 任务（那些任务在宿主上跑就丢了 Docker
|
||
# 外层，违背“按任务二选一”的路由意图）。
|
||
self.claim_executor_only = bool(claim_executor_only)
|
||
# 网页搜索 MCP：默认关（老行为：租户 codex-home 只有
|
||
# 模板 config.toml）。开启后 _tenant_dirs 在播种 config
|
||
# 之后调用 websearch 的播种助手，把 websearch.py /
|
||
# websearch_mcp.py 拷进租户 codex-home 并追加
|
||
# [mcp_servers.websearch]（容器内路径，见 _tenant_dirs）。
|
||
# 老租户下一次任务自动合并，已有 section 不重写。
|
||
self.websearch_mcp = bool(websearch_mcp)
|
||
# 图片生成 MCP：默认关（老行为：租户 codex-home 只有
|
||
# 模板 config.toml + 可选的 websearch MCP）。开启后
|
||
# _tenant_dirs 在 websearch 播种之后调用 imagegen 的
|
||
# 播种助手，把 imagegen.py / imagegen_mcp.py 拷进租户
|
||
# codex-home 并追加 [mcp_servers.imagegen]；_prepare_tenant
|
||
# 在容器启动前写 [mcp_servers.imagegen.env]（系统级
|
||
# IMAGE_GEN_API_KEY + 代理 + NO_PROXY 网关 host）。
|
||
# 未配 IMAGE_GEN_API_KEY 时工具会明确报错“未配置”，
|
||
# 不会静默回退到任务自己的分组 2 key（那会 404）。
|
||
self.imagegen_mcp = bool(imagegen_mcp)
|
||
# 技能播种：默认关（老行为：租户 codex-home 只有模板
|
||
# config.toml + 可选的 websearch MCP）。开启后
|
||
# _tenant_dirs 在 websearch 播种之后调用 skill_seed
|
||
# 的播种助手，把 cloud/skills/ 的 SKILL.md 拷进租户
|
||
# codex-home/skills/（用户级技能路径，实测见
|
||
# skill_seed.py 的路径说明）。老租户下一次任务自动
|
||
# 合并；租户改过的技能文件绝不覆盖，只上报 differs。
|
||
self.seed_skills = bool(seed_skills)
|
||
# HA 阶段 1：本 worker 所在主机名。认领任务时写进
|
||
# tasks.worktree_host，/pr 据此判断 worktree 在哪台主机。
|
||
# 容器内 worker 的 socket.gethostname() 是容器名、不是宿主名，
|
||
# 所以 autoscaler 必须显式传 --host-name（宿主名），见
|
||
# autoscaler._worker_command；单机/测试默认本机名即可。
|
||
self.host_name = host_name or socket.gethostname()
|
||
# 只跑 pr 任务（HA 阶段 1 的跨主机 /pr 演练与专用 pr-runner
|
||
# 用）：不认领普通任务，只处理本主机的 pr_jobs。
|
||
self.pr_jobs_only = bool(pr_jobs_only)
|
||
# 附件暂存根：显式传入 > 环境变量 > 默认路径（与 server 侧
|
||
# 同款解析；两边不一致时 worker 找不到包，派发会静默无附件）。
|
||
self.upload_dir = str(upload_dir) if upload_dir else _upload_root()
|
||
# 运维心跳：路径优先显式参数，其次环境变量，默认当前目录下
|
||
# 的 worker-heartbeat.json；主机路径不在代码里硬编码，主机
|
||
# 部署时用 --heartbeat-file 或环境变量传入（见 deploy/README）。
|
||
# 文件里只有 last_seen 与 pid，绝无任何密钥。
|
||
self.heartbeat_file = heartbeat_file or \
|
||
os.environ.get("CLOUD_WORKER_HEARTBEAT_FILE") or \
|
||
"./worker-heartbeat.json"
|
||
# 多 worker 池身份：显式 --worker-id 优先（生产部署应配稳定
|
||
# id）；未给时用“主机名-进程号”，重启后进程号会变，旧行
|
||
# 自然变 stale，不要用随机 uuid 让注册表无法对应同一台机器。
|
||
self.worker_id = worker_id or "%s-%d" % (
|
||
socket.gethostname(), os.getpid())
|
||
# 并发上限配置来源（见 cloud/caps.py）：显式构造参数 /
|
||
# CLI --worker-capacity > 环境变量 CLOUD_WORKER_CAPACITY >
|
||
# 默认 1；非法值归一为 1。self._env 在上面已定好，单测
|
||
# 注入的普通 dict 同样可读环境变量（无该键时默认 1）。
|
||
self.capacity = resolve_capacity(capacity, env=self._env)
|
||
# 常驻启动时的代码签名：run_loop 每轮边界比对，变了就自退
|
||
# （见 run_loop 的代码变化自退检查）；_claim 每次认领前
|
||
# 也比对（见 _code_signature_valid），不要只在轮次边界查。
|
||
self._code_signature = _code_signature(host_py_path)
|
||
# 配额看门狗命中过的任务（Docker 同步路径用）：看门狗杀
|
||
# 容器后，同步返回的 failed 要被 _run_attempt 改写成
|
||
# quota_exceeded，否则 run_one 会把它当普通失败重试，
|
||
# 配额中断就续跑下去了。受 _quota_hit_lock 保护。
|
||
self._quota_hit_tasks = set()
|
||
self._quota_hit_lock = threading.Lock()
|
||
# 协调员子任务因配额跳过建任务时已记过审计的分工
|
||
# （(parent_id, part_name)）：协调员每轮都会重调
|
||
# _coord_sync_children，超配额期间每轮都记审计会刷屏，
|
||
# 同一 worker 进程内同一分工只记一次。
|
||
self._coord_quota_skipped = set()
|
||
|
||
def _code_changed(self):
|
||
"""启动后关键代码是否被换过（部署换目录/手动改代码）：
|
||
签名取不到时返回 False（是否“变了”无从判断），但这
|
||
不等于放行——放行与否看 _code_signature_valid。"""
|
||
now = _code_signature(self.host_py_path)
|
||
return (now is not None and self._code_signature is not None
|
||
and now != self._code_signature)
|
||
|
||
def _code_signature_valid(self):
|
||
"""签名是否可信且未变：启动签名与现签名都必须取得到
|
||
且相等，否则 False。False 时 _claim 拒绝认领、run_loop
|
||
自退（75）、_run_pr_jobs 不碰任务——fail-closed，绝不
|
||
让签名缺失的 worker 抢任务（旧版取不到当没变是 fail-open）。"""
|
||
now = _code_signature(self.host_py_path)
|
||
return (now is not None and self._code_signature is not None
|
||
and now == self._code_signature)
|
||
|
||
def _mark_quota_hit(self, task_id):
|
||
with self._quota_hit_lock:
|
||
self._quota_hit_tasks.add(task_id)
|
||
|
||
def _take_quota_hit(self, task_id):
|
||
"""取走一次配额看门狗命中（命中即消费，避免旧命中污染
|
||
同一任务的下一轮）：返回 True 表示本轮确被看门狗杀过。"""
|
||
with self._quota_hit_lock:
|
||
if task_id in self._quota_hit_tasks:
|
||
self._quota_hit_tasks.discard(task_id)
|
||
return True
|
||
return False
|
||
|
||
def register(self):
|
||
"""向控制面注册本 worker（写 workers 表，失败只记日志）。
|
||
|
||
注册与心跳共用同一张表：心跳就是带 last_seen 刷新的注册，
|
||
未注册的 worker 发心跳时自动注册，不 404。
|
||
"""
|
||
try:
|
||
register_worker(
|
||
self.db_path, self.worker_id,
|
||
executor=self._executor_name(), capacity=self.capacity,
|
||
running=self._running_count())
|
||
except Exception as exc:
|
||
log_event(self.logger, "worker_register_failed",
|
||
message="worker register failed: %s" % exc)
|
||
|
||
def _registry_heartbeat(self):
|
||
"""向控制面写注册表心跳（与文件心跳并行，失败只记日志）。"""
|
||
try:
|
||
heartbeat_worker(
|
||
self.db_path, self.worker_id,
|
||
executor=self._executor_name(), capacity=self.capacity,
|
||
running=self._running_count())
|
||
except Exception as exc:
|
||
log_event(self.logger, "worker_registry_heartbeat_failed",
|
||
message="worker registry heartbeat failed: %s" % exc)
|
||
|
||
def _write_heartbeat(self):
|
||
"""写一次运维心跳：写失败只记日志，绝不炸 worker。
|
||
|
||
先写临时文件再 os.replace，保证 server 读到的要么是旧的
|
||
完整文件、要么是新的完整文件，不会读到写一半的 JSON。
|
||
"""
|
||
try:
|
||
payload = {"last_seen": time.time(), "pid": os.getpid()}
|
||
tmp = "%s.tmp" % self.heartbeat_file
|
||
parent = os.path.dirname(os.path.abspath(self.heartbeat_file))
|
||
os.makedirs(parent, exist_ok=True)
|
||
with open(tmp, "w", encoding="utf-8") as f:
|
||
f.write(json.dumps(payload))
|
||
os.replace(tmp, self.heartbeat_file)
|
||
except Exception as exc:
|
||
log_event(self.logger, "worker_heartbeat_failed",
|
||
message="worker heartbeat write failed: %s" % exc)
|
||
|
||
# -- 成本存储（惰性建表，避免只认领不执行时也建表）--------------
|
||
@property
|
||
def costs(self):
|
||
if self._cost_store is None:
|
||
self._cost_store = CostStore(self.db_path)
|
||
return self._cost_store
|
||
|
||
def close(self):
|
||
"""关闭惰性建的成本存储连接（注入的 cost_store 由调用方管）。"""
|
||
if self._cost_store is not None:
|
||
try:
|
||
self._cost_store._conn.close()
|
||
except Exception:
|
||
pass
|
||
|
||
# -- DB 读写 ------------------------------------------------------
|
||
def load_tasks(self):
|
||
"""pick_next 的输入：只取 queued/completed/merged 三类任务。
|
||
|
||
旧实现是 SELECT * 全表：prompt/result_text/error_text
|
||
（result_text 可达 21000 字、error_text 可达 4000 字）
|
||
全量进内存，且每轮 poll 都来一次；
|
||
任务表随时间变大后这是纯浪费。pick_next 只需要 queued
|
||
任务和已完成 id（依赖判断），running/failed/interrupted
|
||
的任务对它既不是就绪、也不是已完成，排除它们不影响结果。
|
||
列清单必须覆盖 _task_from_row 直接取的全部字段（prompt、
|
||
idempotency_key 等），缺一列就会 KeyError；result_text、
|
||
error_text 等重字段不在此列，真正需要它们的 get_task
|
||
仍是 SELECT * 单行取。
|
||
"""
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
rows = conn.execute(
|
||
"SELECT id, project_id, prompt, status, idempotency_key,"
|
||
" depends_on, priority, created_by, created_at, updated_at,"
|
||
" last_heartbeat, executor, goal, verify_command,"
|
||
" goal_max_rounds, goal_mode, environment_id FROM tasks"
|
||
" WHERE status IN ('queued', 'completed', 'merged')"
|
||
" ORDER BY created_at, id").fetchall()
|
||
finally:
|
||
conn.close()
|
||
return [_task_from_row(r) for r in rows]
|
||
|
||
def get_task(self, task_id):
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
row = conn.execute(
|
||
"SELECT * FROM tasks WHERE id = ?", (task_id,)).fetchone()
|
||
finally:
|
||
conn.close()
|
||
return _task_from_row(row) if row is not None else None
|
||
|
||
def _claim(self, task_id, executor=None):
|
||
"""CAS 认领：仅当任务仍为 queued 时标 running 并记心跳。
|
||
|
||
executor 过滤（多 worker 池）：executor 给定时，WHERE 里
|
||
再加 (executor IS NULL OR executor = ?)，即只认领未指定
|
||
执行器、或指定为本 worker 执行器类型的任务；CAS 语义不变，
|
||
仍是“仅当 queued 时标 running”，两个同池 worker 抢同一
|
||
任务时只有一个能拿到 rowcount == 1。
|
||
claim_executor_only 开启时（宿主级沙箱 worker 用）：WHERE
|
||
只加 executor = ?，NULL 任务绝不认领（否则宿主 worker
|
||
会抢走本该进 Docker 的任务）。
|
||
|
||
并发上限门（caps）：CAS 之前先在 _running_lock 下占位。
|
||
running 数达到 self.capacity 时直接返回 False，任务保持
|
||
queued，不碰数据库；占位成功才走 CAS。同一任务已在
|
||
_running 里时不重复计数（重复占位不增槽）。CAS 失败
|
||
（已被别的 worker 抢走）时，只清掉本次新建的占位，绝
|
||
不动已在跑任务的条目。占位条目在执行路径的 finally
|
||
里清（本地 _execute_via_process / Docker 同步路径），
|
||
认领后提前返回的三道门（缺 key / 仓库 / 配额）在
|
||
run_one 里显式清占位；CAS 语义本身一句未改。
|
||
P1 签名门：占位之前先比对多文件签名，签名缺失或已变
|
||
时直接返回 False（fail-closed，不抢任务、不占槽），
|
||
并记日志；不要只在 run_loop 轮次边界查——并行池线程
|
||
直接调 run_one/_claim，轮次边界查不到它们。
|
||
"""
|
||
if not self._code_signature_valid():
|
||
log_event(self.logger, "worker_claim_refused_code_signature",
|
||
task_id=task_id,
|
||
message="code signature missing or changed,"
|
||
" refusing claim")
|
||
return False
|
||
capacity_refused = False
|
||
capacity_log = None
|
||
with self._running_lock:
|
||
already_running = task_id in self._running
|
||
if not already_running and \
|
||
len(self._running) >= effective_capacity(self.capacity):
|
||
# 容量门被拦以前是静默返回：worker 活着、心跳
|
||
# 新鲜、日志无错，却再也认领不了任务（协调员
|
||
# 槽位泄漏就是这么被掩盖的）。注意：节流的只
|
||
# 是日志，拒绝本身恒返回 False——若把节流套在
|
||
# 拒绝上，节流窗口内的任务会被 CAS 认领却不进
|
||
# _running，比静默更糟。
|
||
capacity_refused = True
|
||
now_gate = time.time()
|
||
last_gate = getattr(
|
||
self, "_capacity_refused_logged_at", 0.0)
|
||
if now_gate - last_gate >= 30.0:
|
||
self._capacity_refused_logged_at = now_gate
|
||
capacity_log = (
|
||
len(self._running),
|
||
effective_capacity(self.capacity))
|
||
else:
|
||
self._running.setdefault(task_id, {})
|
||
if capacity_log is not None:
|
||
log_event(self.logger, "worker_claim_refused_capacity",
|
||
task_id=task_id,
|
||
message="worker claim refused: capacity full,"
|
||
" running=%d capacity=%d" % capacity_log)
|
||
if capacity_refused:
|
||
return False
|
||
now = time.time()
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
self._ensure_result_columns(conn)
|
||
if executor is None:
|
||
cur = conn.execute(
|
||
"UPDATE tasks SET status = 'running',"
|
||
" last_heartbeat = ?, updated_at = ?,"
|
||
" worktree_host = ?"
|
||
" WHERE id = ? AND status = 'queued'",
|
||
(now, now, self.host_name, task_id),
|
||
)
|
||
elif self.claim_executor_only:
|
||
cur = conn.execute(
|
||
"UPDATE tasks SET status = 'running',"
|
||
" last_heartbeat = ?, updated_at = ?,"
|
||
" worktree_host = ?"
|
||
" WHERE id = ? AND status = 'queued'"
|
||
" AND executor = ?",
|
||
(now, now, self.host_name, task_id, executor),
|
||
)
|
||
else:
|
||
cur = conn.execute(
|
||
"UPDATE tasks SET status = 'running',"
|
||
" last_heartbeat = ?, updated_at = ?,"
|
||
" worktree_host = ?"
|
||
" WHERE id = ? AND status = 'queued'"
|
||
" AND (executor IS NULL OR executor = ?)",
|
||
(now, now, self.host_name, task_id, executor),
|
||
)
|
||
conn.commit()
|
||
claimed = cur.rowcount == 1
|
||
if not claimed and not already_running:
|
||
self._running_clear(task_id)
|
||
return claimed
|
||
finally:
|
||
conn.close()
|
||
|
||
def _set_status(self, task_id, status):
|
||
now = time.time()
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
conn.execute(
|
||
"UPDATE tasks SET status = ?, last_heartbeat = ?,"
|
||
" updated_at = ? WHERE id = ?",
|
||
(status, now, now, task_id),
|
||
)
|
||
conn.commit()
|
||
finally:
|
||
conn.close()
|
||
|
||
def _mark_interrupted_reason(self, task_id, reason):
|
||
"""给已是 interrupted 的任务补中断原因（不改状态本身）。
|
||
|
||
只写 reason，不碰 paused_by_user：调用方是系统路径
|
||
（配额、sweep 已在 server 侧写、worker 退出、协调员
|
||
孤儿），用户暂停的 paused 标记由 server /pause、
|
||
/cancel 写，worker 侧绝不覆盖它。
|
||
老库缺列时先迁移，迁移失败则静默跳过（不炸终局路径）。
|
||
"""
|
||
try:
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
self._ensure_result_columns(conn)
|
||
conn.execute(
|
||
"UPDATE tasks SET interrupted_reason = ?"
|
||
" WHERE id = ? AND status = 'interrupted'",
|
||
(reason, task_id),
|
||
)
|
||
conn.commit()
|
||
finally:
|
||
conn.close()
|
||
except Exception:
|
||
pass
|
||
|
||
def auto_resume_interrupted(self):
|
||
"""长任务自动恢复：只接系统中断的 goal 任务。
|
||
|
||
只有用户暂停才暂停：paused_by_user=1（/pause、/cancel）
|
||
的任务绝不自动动；协调员孤儿（coordinator_orphan）
|
||
不接——要由父协调员收口，自动重跑会跑丢父子关系。
|
||
配额中断（quota）只在配额已恢复时接：每轮先跑
|
||
_quota_exceeded 现查，仍超限就留在 interrupted 不
|
||
重排队（不触发 autoscaler 拉 worker、不烧钱），等
|
||
下一轮再查；查不到配额或仍超限都不接。恢复的判定
|
||
是现查结果为 None（新月份、额度上调、无价上限重置
|
||
都算恢复），不是猜时间。
|
||
非 goal 任务不接：它没有断点语义，重跑等于从头再来，
|
||
可能重复副作用，留给人工 /retry 决定。
|
||
接到的任务 CAS 回 queued 并置 goal_resume_pending=1，
|
||
下一轮认领即走 G10 断点续跑（同一线程、同一 worktree、
|
||
从 goal_rounds_done 下一轮起），同时清旧结果、清
|
||
interrupted_reason、auto_resume_count+1；上限
|
||
self.auto_resume_max 到顶的任务停住并记审计，等人工。
|
||
P0 三件其二的熔断（与次数上限并存，防确定性失败空转）：
|
||
- 退避：重排队时写 resume_after = now + min(300,
|
||
10 * 2**stall)，未到时间不重排队，不再 12 秒一轮烧次数。
|
||
- 无进展熔断：库中断点轮次（task_checkpoints 最大
|
||
round_no）与上次重排队时记的基线相同 → stall_count+1；
|
||
stall 到 3 时停住，interrupted_reason 标
|
||
auto_resume_stalled 并记审计，等人工（任务 2 的
|
||
KeyError 空转：分类已把永久错误判 failed；万一分类
|
||
漏网，无进展熔断是第二道闸）。
|
||
返回自动续跑的任务 id 列表；任何异常只记日志，不炸
|
||
run_loop（恢复是辅助路径，不能带走常驻 worker）。
|
||
"""
|
||
resumed = []
|
||
try:
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
self._ensure_result_columns(conn)
|
||
rows = conn.execute(
|
||
"SELECT id, project_id, created_by, parent_task_id,"
|
||
" model, interrupted_reason, resume_after,"
|
||
" auto_resume_stall_count,"
|
||
" checkpoint_round_at_last_resume FROM tasks"
|
||
" WHERE status = 'interrupted'"
|
||
" AND (paused_by_user IS NULL OR paused_by_user = 0)"
|
||
" AND (interrupted_reason IS NULL OR"
|
||
" interrupted_reason NOT IN"
|
||
" ('coordinator_orphan'))"
|
||
" AND parent_task_id IS NULL"
|
||
" AND (goal_mode = 1 OR verify_command IS NOT NULL"
|
||
" OR goal IS NOT NULL)"
|
||
" AND (auto_resume_count IS NULL OR"
|
||
" auto_resume_count < ?)",
|
||
(self.auto_resume_max,),
|
||
).fetchall()
|
||
now = time.time()
|
||
for row in rows:
|
||
task_id = row[0]
|
||
project_id = row[1]
|
||
if row[6] and now < row[6]:
|
||
continue
|
||
if row[5] == "quota":
|
||
# 配额中断：现查配额，仍超限或查不到
|
||
# 就留在 interrupted，下轮再查，不重排队。
|
||
try:
|
||
_qerr = self._quota_exceeded(
|
||
{"created_by": row[2],
|
||
"parent_task_id": row[3],
|
||
"model": row[4]},
|
||
None, on_error="closed")
|
||
except Exception:
|
||
_qerr = "quota_check_failed"
|
||
if _qerr is not None:
|
||
continue
|
||
try:
|
||
cp = conn.execute(
|
||
"SELECT MAX(round_no) FROM task_checkpoints"
|
||
" WHERE task_id = ?", (task_id,)).fetchone()
|
||
cp_round = cp[0] if cp and cp[0] is not None else -1
|
||
except Exception:
|
||
cp_round = -1
|
||
prev = row[8]
|
||
stall = 0
|
||
if prev is not None:
|
||
stall = (row[7] or 0) + 1 if cp_round == prev else 0
|
||
if stall >= 3:
|
||
cur = conn.execute(
|
||
"UPDATE tasks SET interrupted_reason ="
|
||
" 'auto_resume_stalled',"
|
||
" auto_resume_stall_count = ?,"
|
||
" updated_at = ?"
|
||
" WHERE id = ? AND status = 'interrupted'",
|
||
(stall, now, task_id))
|
||
if cur.rowcount == 1:
|
||
self._audit(
|
||
project_id, "worker_auto_resume_stalled",
|
||
task_id)
|
||
continue
|
||
backoff = min(300, 10 * (2 ** stall))
|
||
cur = conn.execute(
|
||
"UPDATE tasks SET status = 'queued',"
|
||
" goal_resume_pending = 1,"
|
||
" result_text = NULL, error_text = NULL,"
|
||
" actual_model = NULL,"
|
||
" interrupted_reason = NULL,"
|
||
" auto_resume_count ="
|
||
" COALESCE(auto_resume_count, 0) + 1,"
|
||
" auto_resume_stall_count = ?,"
|
||
" checkpoint_round_at_last_resume = ?,"
|
||
" resume_after = ?,"
|
||
" updated_at = ?"
|
||
" WHERE id = ? AND status = 'interrupted'"
|
||
" AND (paused_by_user IS NULL"
|
||
" OR paused_by_user = 0)",
|
||
(stall, cp_round, now + backoff, now, task_id),
|
||
)
|
||
if cur.rowcount == 1:
|
||
resumed.append(task_id)
|
||
self._audit(project_id, "worker_auto_resumed",
|
||
task_id)
|
||
self._record_step(
|
||
conn, task_id, "auto_resume",
|
||
"自动续跑（无进展计数 %d，下次退避 %d 秒）"
|
||
% (stall, backoff))
|
||
conn.commit()
|
||
finally:
|
||
conn.close()
|
||
except Exception as exc:
|
||
log_event(self.logger, "worker_auto_resume_failed",
|
||
message="auto resume failed: %s" % exc)
|
||
return resumed
|
||
|
||
def _record_step(self, conn, task_id, kind, summary, step_no=None,
|
||
state_sha=None):
|
||
"""向 task_steps 写一行（P0 三件其三，时间线事实源）。
|
||
|
||
best-effort：写失败只记日志，绝不影响调用方（认领/
|
||
恢复/终局）的正常流程。step_no 未给时按该任务已有
|
||
最大 step_no + 1 自增（kind=round 的调用方显式传
|
||
round_no，保证 host 轮次与时间线 step_no 一致）。
|
||
"""
|
||
try:
|
||
if step_no is None:
|
||
row = conn.execute(
|
||
"SELECT MAX(step_no) FROM task_steps"
|
||
" WHERE task_id = ?", (task_id,)).fetchone()
|
||
step_no = (row[0] or 0) + 1 if row and row[0] else 1
|
||
conn.execute(
|
||
"INSERT INTO task_steps(id, task_id, step_no, kind,"
|
||
" summary, state_sha, created_at) VALUES(?, ?, ?, ?,"
|
||
" ?, ?, ?)",
|
||
(_uuid_hex(), task_id, step_no, kind, summary[:500]
|
||
if summary else None, state_sha, time.time()))
|
||
except Exception as exc:
|
||
log_event(self.logger, "task_step_record_failed",
|
||
message="step record failed: %s" % exc)
|
||
|
||
def _checkpoint_round(self, conn, task_id):
|
||
"""库中断点最大轮次（无断点时 -1）；表缺失时也返回 -1。"""
|
||
try:
|
||
row = conn.execute(
|
||
"SELECT MAX(round_no) FROM task_checkpoints"
|
||
" WHERE task_id = ?", (task_id,)).fetchone()
|
||
return row[0] if row and row[0] is not None else -1
|
||
except Exception:
|
||
return -1
|
||
|
||
def _checkpoint_mirror_loop(self, task, stop):
|
||
"""断点镜像线程（P0 三件其一）：goal 任务执行期间每 30 秒
|
||
把磁盘 host-state 断点镜像进库一次；执行结束由
|
||
_run_attempt 的 finally 再补一次终局镜像。线程本身绝不
|
||
抛（_mirror_checkpoint_once 已全量兜底），stop 后即退。
|
||
"""
|
||
try:
|
||
self._mirror_checkpoint_once(task)
|
||
except Exception:
|
||
pass
|
||
while not stop.wait(30):
|
||
try:
|
||
self._mirror_checkpoint_once(task)
|
||
except Exception:
|
||
pass
|
||
|
||
def _host_state_path_for_task(self, task):
|
||
"""断点镜像的 host-state 文件路径（P0 三件其一）。
|
||
|
||
租户 Docker：tenants/<safe>/work/host-state.json；
|
||
非租户 Docker：executor extra_mounts 里 dest=/app/work
|
||
的宿主路径（若无则回落 /opt/codex-app-server/work）；
|
||
Local：读 executor config 的 state 项（相对配置目录）。
|
||
找不到时返回 None，镜像静默跳过（不影响任务执行）。
|
||
"""
|
||
try:
|
||
project_id = task.get("project_id")
|
||
if getattr(self, "tenant_isolation", False) and project_id:
|
||
safe = self._tenant_safe_name(project_id)
|
||
base = os.path.join(self.tenants_root, safe, "work")
|
||
return os.path.join(base, "host-state.json")
|
||
mounts = getattr(self.executor, "extra_mounts", None) or []
|
||
for item in mounts:
|
||
src, dst = (item if isinstance(item, tuple)
|
||
else (None, None))
|
||
if dst == "/app/work" and src:
|
||
return os.path.join(src, "host-state.json")
|
||
cfg = getattr(self.executor, "config_path", None)
|
||
if cfg and os.path.exists(cfg):
|
||
with open(cfg, encoding="utf-8") as fh:
|
||
data = json.load(fh)
|
||
state = data.get("state") or "host-state.json"
|
||
if not os.path.isabs(state):
|
||
state = os.path.join(
|
||
os.path.dirname(os.path.abspath(cfg)), state)
|
||
return state
|
||
return os.path.join("/opt/codex-app-server/work",
|
||
"host-state.json")
|
||
except Exception:
|
||
return None
|
||
|
||
def _mirror_checkpoint_once(self, task):
|
||
"""把磁盘 host-state 里的本任务断点镜像进库（单次）。
|
||
|
||
读 state → 取 tasks[host 名] 条目 → (task_id, round_no)
|
||
upsert 进 task_checkpoints（state_sha 是条目 JSON 的
|
||
sha256，轮次相同但线程/状态变了也能看出来）→ 同时
|
||
导入同目录 host-steps/<任务>.jsonl 里的 round 步骤。
|
||
任何异常静默（镜像是辅助路径，执行本身不受影响）。
|
||
返回镜像到的轮次（-1 = 没镜像到）。
|
||
"""
|
||
try:
|
||
state_path = self._host_state_path_for_task(task)
|
||
if not state_path or not os.path.exists(state_path):
|
||
return -1
|
||
with open(state_path, encoding="utf-8") as fh:
|
||
state = json.load(fh)
|
||
host_name = _worker_host_name(self.executor, task)
|
||
entry = (state.get("tasks") or {}).get(host_name)
|
||
if not isinstance(entry, dict):
|
||
return -1
|
||
rounds = int(entry.get("goal_rounds_done") or 0)
|
||
# host-state 条目写的是 status（goal-running/goal-met…），
|
||
# 不是 goal_status：实弹里镜像到的 goal_status 全是
|
||
# None——取 status 作回落，标签才不是空的（轮次与
|
||
# thread_id 等关键字段不受影响，熔断只看轮次）。
|
||
goal_status = entry.get("goal_status") or entry.get("status")
|
||
state_sha = hashlib.sha256(
|
||
json.dumps(entry, sort_keys=True,
|
||
default=str).encode("utf-8")).hexdigest()
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
self._ensure_result_columns(conn)
|
||
existing = conn.execute(
|
||
"SELECT 1 FROM task_checkpoints"
|
||
" WHERE task_id = ? AND round_no = ?",
|
||
(task["id"], rounds)).fetchone()
|
||
if existing:
|
||
conn.execute(
|
||
"UPDATE task_checkpoints SET thread_id = ?,"
|
||
" goal_status = ?, state_sha = ?,"
|
||
" verify_out = ?, created_at = ?"
|
||
" WHERE task_id = ? AND round_no = ?",
|
||
(entry.get("thread_id"),
|
||
goal_status,
|
||
state_sha,
|
||
(entry.get("goal_last_verify") or "")[:500],
|
||
time.time(), task["id"], rounds))
|
||
else:
|
||
conn.execute(
|
||
"INSERT INTO task_checkpoints(task_id, round_no,"
|
||
" thread_id, goal_status, state_sha, verify_out,"
|
||
" created_at) VALUES(?, ?, ?, ?, ?, ?, ?)",
|
||
(task["id"], rounds, entry.get("thread_id"),
|
||
goal_status, state_sha,
|
||
(entry.get("goal_last_verify") or "")[:500],
|
||
time.time()))
|
||
self._import_host_steps(conn, task, state_path)
|
||
conn.commit()
|
||
finally:
|
||
conn.close()
|
||
return rounds
|
||
except Exception:
|
||
return -1
|
||
|
||
def _import_host_steps(self, conn, task, state_path):
|
||
"""把 host-steps/<任务>.jsonl 的 round 行导入 task_steps。
|
||
|
||
按 (task_id, kind='round', step_no) 去重：已入库的
|
||
step_no 跳过；文件不存在或格式坏行都跳过不炸。
|
||
"""
|
||
try:
|
||
steps_path = os.path.join(
|
||
os.path.dirname(os.path.abspath(state_path)),
|
||
"host-steps", "%s.jsonl" % _worker_host_name(
|
||
self.executor, task))
|
||
if not os.path.exists(steps_path):
|
||
return
|
||
existing = {row[0] for row in conn.execute(
|
||
"SELECT step_no FROM task_steps WHERE task_id = ?"
|
||
" AND kind = 'round'", (task["id"],)).fetchall()}
|
||
with open(steps_path, encoding="utf-8") as fh:
|
||
for line in fh:
|
||
try:
|
||
item = json.loads(line)
|
||
except Exception:
|
||
continue
|
||
step_no = item.get("round")
|
||
if not isinstance(step_no, int) or step_no in existing:
|
||
continue
|
||
conn.execute(
|
||
"INSERT INTO task_steps(id, task_id, step_no,"
|
||
" kind, summary, state_sha, created_at)"
|
||
" VALUES(?, ?, ?, 'round', ?, NULL, ?)",
|
||
(_uuid_hex(), task["id"], step_no,
|
||
("第 %d 轮 turn=%s verify=%s" % (
|
||
step_no, item.get("turn_status"),
|
||
(item.get("verify_out") or "")[:200])),
|
||
item.get("ts") or time.time()))
|
||
existing.add(step_no)
|
||
except Exception:
|
||
pass
|
||
|
||
def _rebuild_host_state_from_checkpoint(self, task):
|
||
"""断点丢失时的重建（P0 三件其一，任务 2 类故障的解法）。
|
||
|
||
host-state 里没有本任务条目、但库里有断点时：从库里
|
||
最新断点 + tasks 行的 goal/verify_command/max_rounds
|
||
拼一个最小 host-state 条目写回（goal_status=interrupted，
|
||
轮次与 thread_id 以库为准），再由正常 goal-resume
|
||
路径续跑。拼不出来（库里也无断点）返回 False，
|
||
调用方走原有拒绝/失败路径，不造假断点。
|
||
"""
|
||
try:
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
self._ensure_result_columns(conn)
|
||
row = conn.execute(
|
||
"SELECT round_no, thread_id, goal_status"
|
||
" FROM task_checkpoints WHERE task_id = ?"
|
||
" ORDER BY round_no DESC LIMIT 1",
|
||
(task["id"],)).fetchone()
|
||
if not row:
|
||
return False
|
||
trow = conn.execute(
|
||
"SELECT goal, verify_command, goal_max_rounds"
|
||
" FROM tasks WHERE id = ?", (task["id"],)).fetchone()
|
||
if not trow:
|
||
return False
|
||
finally:
|
||
conn.close()
|
||
state_path = self._host_state_path_for_task(task)
|
||
if not state_path:
|
||
return False
|
||
host_name = _worker_host_name(self.executor, task)
|
||
if os.path.exists(state_path):
|
||
with open(state_path, encoding="utf-8") as fh:
|
||
state = json.load(fh)
|
||
else:
|
||
state = {"version": 1, "tasks": {}}
|
||
tasks_map = state.setdefault("tasks", {})
|
||
if host_name in tasks_map:
|
||
return True
|
||
tasks_map[host_name] = {
|
||
"goal": trow[0],
|
||
"verify_command": trow[1],
|
||
"goal_max_rounds": trow[2] or 0,
|
||
"goal_rounds_done": row[0],
|
||
"thread_id": row[1],
|
||
"goal_status": "interrupted",
|
||
"goal_last_verify": "",
|
||
"rebuilt_from_checkpoint": True,
|
||
}
|
||
os.makedirs(os.path.dirname(
|
||
os.path.abspath(state_path)), exist_ok=True)
|
||
tmp_path = state_path + ".tmp"
|
||
with open(tmp_path, "w", encoding="utf-8") as fh:
|
||
json.dump(state, fh)
|
||
os.replace(tmp_path, state_path)
|
||
return True
|
||
except Exception:
|
||
return False
|
||
|
||
def _task_never_started(self, task):
|
||
"""从未启动判据（2026-10-29 实测任务 A 的教训）。
|
||
|
||
四者全无才算从未启动：host-state 里无本任务条目、
|
||
库里无断点行、state 同目录 host-worktrees 下无本任务
|
||
worktree、host-traces 下无本任务链路文件。任一存在
|
||
都说明执行过，绝不能当成全新任务重跑（会撞已存在
|
||
或丢进度），调用方保持原有拒绝/重建路径。
|
||
任何一步查不出来（state 路径不明、文件损坏、库异常）
|
||
都返回 False——fail-closed，宁可拒绝也不误重跑。
|
||
"""
|
||
try:
|
||
state_path = self._host_state_path_for_task(task)
|
||
if not state_path:
|
||
return False
|
||
host_name = _worker_host_name(self.executor, task)
|
||
if os.path.exists(state_path):
|
||
with open(state_path, encoding="utf-8") as fh:
|
||
state = json.load(fh)
|
||
if host_name in (state.get("tasks") or {}):
|
||
return False
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
self._ensure_result_columns(conn)
|
||
row = conn.execute(
|
||
"SELECT 1 FROM task_checkpoints WHERE task_id = ?"
|
||
" LIMIT 1", (task["id"],)).fetchone()
|
||
if row:
|
||
return False
|
||
finally:
|
||
conn.close()
|
||
state_dir = os.path.dirname(os.path.abspath(state_path))
|
||
if os.path.isdir(os.path.join(state_dir, "host-worktrees",
|
||
host_name)):
|
||
return False
|
||
if os.path.exists(os.path.join(state_dir, "host-traces",
|
||
host_name + ".jsonl")):
|
||
return False
|
||
return True
|
||
except Exception:
|
||
return False
|
||
|
||
def _resolve_breakpoint_resume(self, task):
|
||
"""断点续跑解析：先重建，建不出且从未启动时回落全新跑。
|
||
|
||
正常情况（条目在，或库里有断点可重建）原样返回，
|
||
调用方走 goal-resume 断点续跑。只有重建失败（库里
|
||
也无断点）且 _task_never_started 全无执行痕迹时，
|
||
才清掉断点标记、改走全新 goal：此时 goal-resume 注定
|
||
被 host 拒绝 → failed，而全新跑与首轮完全相同
|
||
（worktree 本就不存在，不会覆盖任何进度）。
|
||
次数上限与无进展熔断仍在上游卡着，不会空转。
|
||
手动 /goal-resume 同样走这里：对从未跑过的任务，
|
||
全新跑也是唯一有意义的结果。
|
||
"""
|
||
try:
|
||
rebuilt = self._rebuild_host_state_from_checkpoint(task)
|
||
except Exception:
|
||
rebuilt = False
|
||
if not rebuilt and self._task_never_started(task):
|
||
task = dict(task)
|
||
task["_goal_resume_breakpoint"] = False
|
||
self._audit(task.get("project_id"),
|
||
"worker_fresh_restart", task["id"])
|
||
try:
|
||
log_event(self.logger, "worker_fresh_restart",
|
||
task_id=task["id"],
|
||
project_id=task.get("project_id"))
|
||
except Exception:
|
||
pass
|
||
return task
|
||
|
||
def _ensure_result_columns(self, conn):
|
||
"""worker 侧的迁移兜底：与 server.migrate_tasks_table 同列。
|
||
|
||
worker 的库理论上已由 server.init_db 迁过；这里再查一次
|
||
是为了 worker 直连老库（测试里手工建的表）时也不炸。
|
||
"""
|
||
try:
|
||
cols = {row[1] for row in conn.execute(
|
||
"PRAGMA table_info(tasks)").fetchall()}
|
||
except Exception:
|
||
return
|
||
for name, coltype in {
|
||
"result_text": "TEXT", "result_text_full": "TEXT",
|
||
"error_text": "TEXT",
|
||
"actual_model": "TEXT", "started_at": "REAL",
|
||
"finished_at": "REAL", "child_pid": "INTEGER",
|
||
"executor": "TEXT", "pr_url": "TEXT",
|
||
"pr_number": "INTEGER", "goal": "TEXT",
|
||
"verify_command": "TEXT",
|
||
"goal_max_rounds": "INTEGER",
|
||
"goal_mode": "INTEGER",
|
||
"worktree_host": "TEXT",
|
||
"model": "TEXT",
|
||
"effort": "TEXT",
|
||
"child_model": "TEXT",
|
||
"child_effort": "TEXT",
|
||
"approval": "TEXT",
|
||
"plan": "INTEGER",
|
||
"forked_from": "TEXT",
|
||
"environment_id": "TEXT",
|
||
"resume_message": "TEXT",
|
||
# 任务列表显示 turn 用：本任务最近一轮产出过
|
||
# turn_id 的回合编号（local/Docker 两路都经
|
||
# _result_from_runner/_parse_host_result 提取）。
|
||
"turn_id": "TEXT",
|
||
"root_turn_id": "TEXT",
|
||
"coordinator_parts": "TEXT",
|
||
"parent_task_id": "TEXT",
|
||
"coordinator_part": "TEXT",
|
||
"goal_resume_pending": "INTEGER DEFAULT 0",
|
||
"paused_by_user": "INTEGER DEFAULT 0",
|
||
"interrupted_reason": "TEXT",
|
||
"auto_resume_count": "INTEGER DEFAULT 0",
|
||
"no_time_limit": "INTEGER DEFAULT 0",
|
||
"resume_after": "DOUBLE PRECISION DEFAULT 0",
|
||
"auto_resume_stall_count": "INTEGER DEFAULT 0",
|
||
"checkpoint_round_at_last_resume": "INTEGER",
|
||
"last_error_signature": "TEXT"}.items():
|
||
if name not in cols:
|
||
try:
|
||
conn.execute(
|
||
"ALTER TABLE tasks ADD COLUMN %s %s"
|
||
% (name, coltype))
|
||
except Exception:
|
||
pass
|
||
# P0 三件的表（worker 直连老库时也补，与 server ensure 同 DDL）。
|
||
try:
|
||
conn.execute(
|
||
"CREATE TABLE IF NOT EXISTS task_checkpoints("
|
||
"task_id TEXT, "
|
||
"round_no INTEGER, "
|
||
"thread_id TEXT, "
|
||
"goal_status TEXT, "
|
||
"state_sha TEXT, "
|
||
"verify_out TEXT, "
|
||
"created_at DOUBLE PRECISION, "
|
||
"PRIMARY KEY(task_id, round_no))")
|
||
conn.execute(
|
||
"CREATE TABLE IF NOT EXISTS task_steps("
|
||
"id TEXT PRIMARY KEY, "
|
||
"task_id TEXT, "
|
||
"step_no INTEGER, "
|
||
"kind TEXT, "
|
||
"summary TEXT, "
|
||
"state_sha TEXT, "
|
||
"created_at DOUBLE PRECISION)")
|
||
conn.execute(
|
||
"CREATE INDEX IF NOT EXISTS idx_task_steps_task"
|
||
" ON task_steps(task_id, step_no)")
|
||
except Exception:
|
||
pass
|
||
|
||
def _set_status_if_running(self, task_id, status):
|
||
"""仅当任务仍为 running 时改状态，返回是否改成功。
|
||
|
||
并发 + 优雅排空下，后台线程可能在任务已被取消
|
||
（标 interrupted）后才跑完；此时最终写回绝不能把
|
||
interrupted 盖回 completed/failed。正常流程里任务
|
||
到最终写回时仍是 running，CAS 一定成功。
|
||
"""
|
||
now = time.time()
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
cur = conn.execute(
|
||
"UPDATE tasks SET status = ?, last_heartbeat = ?,"
|
||
" updated_at = ? WHERE id = ? AND status = 'running'",
|
||
(status, now, now, task_id),
|
||
)
|
||
conn.commit()
|
||
return cur.rowcount == 1
|
||
finally:
|
||
conn.close()
|
||
|
||
def _write_execution_result(self, task_id, result_text=None,
|
||
error_text=None, actual_model=None,
|
||
started_at=None, finished_at=None,
|
||
child_pid=None, turn_id=None,
|
||
root_turn_id=None, only_if_status=None,
|
||
result_full=None):
|
||
# result_full：完整 stdout（dispatch/docker 路由 sandbox
|
||
# 的 stdout_full）；截断版 result_text 保持 4000 不变作
|
||
# 摘要，完整版另存 result_text_full 供详情/端点取。
|
||
"""把执行结果写进 tasks 新列；老库缺列时先迁移再写。
|
||
|
||
turn_id/root_turn_id 只在新值非空时覆盖：某轮没产出
|
||
turn（如提前失败门、解析不到 JSON）时保留上一轮的值，
|
||
列表里始终显示最近一个真实回合的编号，而不是被空值
|
||
洗掉。
|
||
P0 安全批后：only_if_status 非空时写回带状态守卫
|
||
（UPDATE ... WHERE id = ? AND status = ?），状态已
|
||
变（例如迟到的旧进程在任务已被取消/新一轮认领后才
|
||
写回）时不覆盖新结果，返回 False；不传时保持旧行为
|
||
并返回 True。调用方在终局路径必须传守卫，不能只靠
|
||
写回前的 _set_status_if_running（两者之间仍有竞态窗）。
|
||
"""
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
self._ensure_result_columns(conn)
|
||
status_clause = ""
|
||
extra = ()
|
||
if only_if_status is not None:
|
||
status_clause = " AND status = ?"
|
||
extra = (only_if_status,)
|
||
full = (result_full[:200000] if result_full else None)
|
||
short = ((result_text or "")[:21000] if result_text else None)
|
||
err_short = ((error_text or "")[:4000]
|
||
if error_text else None)
|
||
now = time.time()
|
||
if child_pid is None:
|
||
sql_full = (
|
||
"UPDATE tasks SET result_text = ?,"
|
||
" result_text_full = ?, error_text = ?,"
|
||
" actual_model = ?, started_at = ?, finished_at = ?,"
|
||
" turn_id = COALESCE(?, turn_id),"
|
||
" root_turn_id = COALESCE(?, root_turn_id),"
|
||
" updated_at = ? WHERE id = ?" + status_clause)
|
||
params_full = (short, full, err_short, actual_model,
|
||
started_at, finished_at, turn_id,
|
||
root_turn_id, now, task_id) + extra
|
||
sql_old = sql_full.replace(
|
||
" result_text_full = ?,", "")
|
||
params_old = (short, err_short, actual_model,
|
||
started_at, finished_at, turn_id,
|
||
root_turn_id, now, task_id) + extra
|
||
else:
|
||
sql_full = (
|
||
"UPDATE tasks SET result_text = ?,"
|
||
" result_text_full = ?, error_text = ?,"
|
||
" actual_model = ?, started_at = ?, finished_at = ?,"
|
||
" child_pid = ?,"
|
||
" turn_id = COALESCE(?, turn_id),"
|
||
" root_turn_id = COALESCE(?, root_turn_id),"
|
||
" updated_at = ? WHERE id = ?" + status_clause)
|
||
params_full = (short, full, err_short, actual_model,
|
||
started_at, finished_at, child_pid,
|
||
turn_id, root_turn_id, now, task_id) + extra
|
||
sql_old = sql_full.replace(
|
||
" result_text_full = ?,", "")
|
||
params_old = (short, err_short, actual_model,
|
||
started_at, finished_at, child_pid,
|
||
turn_id, root_turn_id, now, task_id) + extra
|
||
try:
|
||
cur = conn.execute(sql_full, params_full)
|
||
except Exception as exc:
|
||
# 老库/被 mock 掉迁移的库没有 result_text_full 列：
|
||
# 完整版丢弃、摘要照写，绝不让新列拖垮终局写回
|
||
# （P0 取消守卫测试就是 mock 掉迁移的手工表）。
|
||
if "result_text_full" not in str(exc):
|
||
raise
|
||
try:
|
||
conn.rollback()
|
||
except Exception:
|
||
pass
|
||
cur = conn.execute(sql_old, params_old)
|
||
conn.commit()
|
||
return cur.rowcount == 1
|
||
finally:
|
||
conn.close()
|
||
|
||
def _set_child_pid(self, task_id, pid):
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
self._ensure_result_columns(conn)
|
||
conn.execute(
|
||
"UPDATE tasks SET child_pid = ? WHERE id = ?",
|
||
(pid, task_id),
|
||
)
|
||
conn.commit()
|
||
finally:
|
||
conn.close()
|
||
|
||
def _kill_proc_group(self, proc):
|
||
"""真杀：先 SIGTERM 给整个进程组，等 2 秒，仍活着再 SIGKILL。
|
||
|
||
proc 须是 start_new_session 启动的 Popen（自成进程组），
|
||
杀组才能连同它拉起的子孙一起杀掉，不只杀父进程。
|
||
"""
|
||
try:
|
||
pgid = os.getpgid(proc.pid)
|
||
except Exception:
|
||
return
|
||
try:
|
||
os.killpg(pgid, signal.SIGTERM)
|
||
except (ProcessLookupError, PermissionError, OSError):
|
||
return
|
||
deadline = time.time() + 2.0
|
||
while time.time() < deadline:
|
||
try:
|
||
os.killpg(pgid, 0)
|
||
except (ProcessLookupError, PermissionError, OSError):
|
||
return
|
||
time.sleep(0.05)
|
||
try:
|
||
os.killpg(pgid, signal.SIGKILL)
|
||
except (ProcessLookupError, PermissionError, OSError):
|
||
pass
|
||
|
||
def heartbeat(self, task_id):
|
||
"""刷新 running 任务的心跳；任务不在 running 时返回 False。"""
|
||
now = time.time()
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
cur = conn.execute(
|
||
"UPDATE tasks SET last_heartbeat = ?, updated_at = ?"
|
||
" WHERE id = ? AND status = 'running'",
|
||
(now, now, task_id),
|
||
)
|
||
conn.commit()
|
||
return cur.rowcount == 1
|
||
finally:
|
||
conn.close()
|
||
|
||
def _audit(self, project_id, action, detail=""):
|
||
"""best-effort 审计：失败不影响任务状态。"""
|
||
try:
|
||
store = AuthStore(self.db_path)
|
||
try:
|
||
store.audit("system", project_id, action, detail)
|
||
finally:
|
||
try:
|
||
store.conn.close()
|
||
except Exception:
|
||
pass
|
||
except Exception:
|
||
pass
|
||
|
||
def _notify_task_finished(self, task_id, project_id, event_type,
|
||
error=None):
|
||
"""任务终态推送：best-effort，绝不因通知失败影响任务状态。
|
||
|
||
只发终态事件：completed/failed/goal-unmet
|
||
（push.ALLOWED_PUSH_EVENTS 的子集；interrupted 不在
|
||
事件集合里，取消不通知）。无订阅时
|
||
dispatch 直接返回空 sent，不联网。同步发送、单次 10 秒
|
||
超时：慢订阅会拖住 worker 循环最多几秒，换取投递结果
|
||
可观测（失败记日志，不静默丢）；后台队列是后续优化，
|
||
不在本轮。
|
||
"""
|
||
try:
|
||
from cloud.push import dispatch_event
|
||
result = dispatch_event(
|
||
self.db_path, event_type, project_id,
|
||
{"task_id": task_id, "status": event_type,
|
||
"executor": self._executor_name(), "error": error})
|
||
if result.get("errors"):
|
||
log_event(self.logger, "worker_notify_failed",
|
||
task_id=task_id, project_id=project_id,
|
||
status=event_type,
|
||
message="notify errors: %s" % (result["errors"],))
|
||
except Exception as exc: # noqa: BLE001 -- 通知失败绝不炸任务
|
||
log_event(self.logger, "worker_notify_failed",
|
||
task_id=task_id, project_id=project_id,
|
||
status=event_type,
|
||
message="notify exception: %s" % exc)
|
||
|
||
# -- 自动记忆 ------------------------------------------------------
|
||
# 注入失败绝不挡任务（fail-open：无记忆跑也比不跑强）；提取
|
||
# 失败只记日志（run_loop 轮次边界已包一层，见 process 方法）。
|
||
|
||
def _apply_auto_memory_injection(self, task):
|
||
"""首轮注入记忆：只改内存副本，原 prompt/goal 不落库。
|
||
|
||
跳过：协调员任务（不跑模型回合）、续跑与断点续跑（已有
|
||
会话历史）、无属主、use_memories 关、无记忆。goal 任务
|
||
同时改内存里的 goal（执行器取 goal 优先于 prompt），
|
||
前缀里写明“仅供参考，不得改写本任务目标”。
|
||
"""
|
||
if task.get("coordinator_parts"):
|
||
return task
|
||
if task.get("_resume") or task.get("_goal_resume_breakpoint"):
|
||
return task
|
||
owner = task.get("created_by")
|
||
if not owner:
|
||
return task
|
||
try:
|
||
store = AutoMemoryStore(self.db_path)
|
||
try:
|
||
flags = store.get_task_flags(task["id"])
|
||
finally:
|
||
store.close()
|
||
prefix = _auto_prefix(
|
||
self.db_path, owner, task["project_id"], flags=flags)
|
||
except Exception as exc:
|
||
try:
|
||
self.logger.warning("auto memory injection failed: %s",
|
||
exc)
|
||
except Exception:
|
||
pass
|
||
return task
|
||
if not prefix:
|
||
return task
|
||
new_task = dict(task)
|
||
prompt = task.get("prompt") or ""
|
||
new_task["prompt"] = prefix + "\n\n" + prompt
|
||
if task.get("goal_mode") or task.get("verify_command"):
|
||
new_task["goal"] = prefix + "\n\n" + (
|
||
task.get("goal") or prompt)
|
||
new_task["_memory_injected"] = True
|
||
self._audit(task["project_id"], "worker_memory_injected",
|
||
task["id"])
|
||
return new_task
|
||
|
||
def process_auto_memories(self, now=None, force=False,
|
||
owner_id=None):
|
||
"""扫 completed 任务并提取记忆（run_loop 轮次边界调用）。
|
||
|
||
返回 process_due_tasks 的结果列表；提取本身失败时返回
|
||
空列表并只记日志，不炸循环。提取成功的任务各补一条
|
||
worker 侧审计（服务端手动触发路径另有自己的审计）。
|
||
"""
|
||
try:
|
||
results = _auto_process_due(
|
||
self.db_path, now=now, force=force, owner_id=owner_id)
|
||
except Exception as exc:
|
||
try:
|
||
self.logger.warning("auto memory sweep failed: %s", exc)
|
||
except Exception:
|
||
pass
|
||
return []
|
||
for item in results:
|
||
if item.get("status") == "extracted":
|
||
# project_id 不在结果里：审计 detail 只带任务 id，
|
||
# project 字段用空串（与系统审计同款占位）。
|
||
self._audit("", "worker_memory_extracted",
|
||
item.get("task_id") or "")
|
||
return results
|
||
|
||
# -- 认领 ----------------------------------------------------------
|
||
def _budget_exceeded(self):
|
||
return budget_exceeded(self.usage_turns, self.max_turns,
|
||
self.usage_cost_usd, self.max_cost_usd)
|
||
|
||
def _quota_limit_and_used(self, owner):
|
||
"""查某人当月配额与已记账用量，返回
|
||
(quota, used, month, unpriced_used)。
|
||
|
||
quota 为 None 表示不限额（只限行里 quota 为 NULL 的
|
||
显式不限额；无配额行按默认配额
|
||
_default_quota_usd_micros() 算，与服务端 _quota_status
|
||
的 P0 口径一致）。查询失败时抛异常，由调用方按
|
||
fail-open / fail-closed 分层决定——本函数自己绝不吞
|
||
异常后冒充“不限额”，否则调用方分不清“真不限额”与
|
||
“查不出来”。
|
||
unpriced_used 是当月 usd IS NULL 且 source='actual'
|
||
的成本行 input + output 之和（与服务端
|
||
_check_unpriced_token_cap 同口径，估算行不计）；
|
||
quota 为 None 时它仍返回实数，但调用方
|
||
_quota_exceeded 在 quota 为 None 时先返回 None，
|
||
所以显式不限额用户不受无价上限约束，与服务端一致。
|
||
"""
|
||
month = datetime.now(timezone.utc).strftime("%Y-%m")
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
conn.execute(
|
||
"CREATE TABLE IF NOT EXISTS user_quotas("
|
||
"user_id TEXT PRIMARY KEY, quota_usd_micros BIGINT,"
|
||
" updated_at DOUBLE PRECISION)")
|
||
conn.commit()
|
||
row = conn.execute(
|
||
"SELECT quota_usd_micros FROM user_quotas"
|
||
" WHERE user_id = ?",
|
||
(owner,)).fetchone()
|
||
if row is None:
|
||
# P0/P1 口径：无行按默认额度，不是不限额（只有
|
||
# 显式 NULL 行才是不限额，见下）。
|
||
quota = _default_quota_usd_micros()
|
||
else:
|
||
quota = row[0]
|
||
if quota is None:
|
||
return None, 0, month, 0
|
||
# cost_records 由 CostStore 懒建：全新库上它还不存
|
||
# 在，直接 JOIN 会报错——先实例化一次建表再查（与
|
||
# 服务端 _quota_status 同款做法），否则 quota=0
|
||
#（一分钱也不能花）在全新库上会被误放行。
|
||
_cost_store = CostStore(self.db_path)
|
||
try:
|
||
_cost_store._conn.close()
|
||
except Exception:
|
||
pass
|
||
used_row = conn.execute(
|
||
"SELECT COALESCE(SUM(c.usd_micros), 0)"
|
||
" FROM cost_records c JOIN tasks t"
|
||
" ON c.task_id = t.id"
|
||
" WHERE t.created_by = ? AND c.day LIKE ?",
|
||
(owner, month + "%")).fetchone()
|
||
used = used_row[0] if used_row else 0
|
||
unpriced_row = conn.execute(
|
||
"SELECT COALESCE(SUM(c.input_tokens + c.output_tokens), 0)"
|
||
" FROM cost_records c JOIN tasks t"
|
||
" ON c.task_id = t.id"
|
||
" WHERE t.created_by = ? AND c.day LIKE ?"
|
||
" AND c.usd_micros IS NULL AND c.source = 'actual'",
|
||
(owner, month + "%")).fetchone()
|
||
unpriced_used = unpriced_row[0] if unpriced_row else 0
|
||
return quota, int(used or 0), month, int(unpriced_used or 0)
|
||
finally:
|
||
conn.close()
|
||
|
||
def _quota_exceeded(self, task, usage_total, on_error="open"):
|
||
"""配额加固的执行时检查：返回超限时的错误文本，否则 None。
|
||
|
||
口径与服务端准入（_quota_status）一致：按任务创建者算，
|
||
当月（UTC）已记账用量 + 本轮已累计用量（按价目表换算，
|
||
无价模型按 0 计，与服务端 SUM（usd NULL 不计）一致）。
|
||
另有一道无价 token 上限（与服务端
|
||
_check_unpriced_token_cap 同口径）：USD 配额不计量无价
|
||
模型，默认模型 terra 无价，若无此上限则配额对默认模型
|
||
永远不生效；有额度约束的用户（无行按默认额度）当月无价
|
||
token（input +
|
||
output，已记账 + 本轮）达到上限即停。无价上限只计
|
||
source='actual' 的行：估算行（estimated）不是真实花费，
|
||
计入上限会把“没花钱”算成“花超了”，且与 cost.py
|
||
_report 的 unpriced_actual_tokens 口径一致。
|
||
属主与额度（P1 口径，与服务端 _quota_status 一致）：
|
||
- 属主：任务 created_by 为空时先查父任务（协调员子任务
|
||
行可能无 created_by，属主跟父任务走）；仍查不到时
|
||
拒绝新回合，不再返回 None 放行。created_by 字面为
|
||
"system" 的系统任务按普通属主算，不另设无限额的
|
||
系统预算（G14 的 system 豁免与 P1 冲突，本次未合入）。
|
||
- 无配额行：按默认配额 _default_quota_usd_micros()
|
||
（$10/月）算；只有显式 NULL 行才是不限额。
|
||
查库失败分层（G7 + P1）：认领门传 on_error="closed"，
|
||
返回 "quota_check_unavailable: ..."（调用方标
|
||
interrupted，可重试/续跑）；默认 "open" 返回
|
||
"quota_check_failed: ..."——新回合边界（直接调用）
|
||
见此串即拒绝，执行中轮询经 _quota_hit 闭包过滤此串、
|
||
只记警告不杀当前轮。查询抖动不应杀掉已在跑的回合，
|
||
但下一轮边界仍会拒绝。
|
||
"""
|
||
owner = (task or {}).get("created_by")
|
||
if not owner and (task or {}).get("parent_task_id"):
|
||
# 协调员子任务属主跟父任务走（与 _gateway_context
|
||
# 同款回查）；查不到时不猜，直接走下面的拒绝分支。
|
||
try:
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
row = conn.execute(
|
||
"SELECT created_by FROM tasks WHERE id = ?",
|
||
(task["parent_task_id"],)).fetchone()
|
||
finally:
|
||
conn.close()
|
||
if row is not None:
|
||
try:
|
||
owner = row[0]
|
||
except (TypeError, IndexError):
|
||
owner = None
|
||
except Exception:
|
||
owner = None
|
||
if not owner:
|
||
return ("quota_exceeded: 任务缺少属主，无法查配额，"
|
||
"拒绝新回合（请归属真实用户后再排队）")
|
||
try:
|
||
quota, used, month, unpriced_used = \
|
||
self._quota_limit_and_used(owner)
|
||
except Exception as exc:
|
||
# 查库失败分层（见本函数 docstring）：closed（认领门）
|
||
# 回 quota_check_unavailable，调用方标 interrupted；
|
||
# open 回 quota_check_failed，新回合边界拒绝，执行中
|
||
# 轮询经 _quota_hit 闭包过滤、只警告不杀轮。
|
||
try:
|
||
self.logger.warning("quota check failed: %s", exc)
|
||
except Exception:
|
||
pass
|
||
if on_error == "closed":
|
||
return "quota_check_unavailable: 配额查询失败，任务未开跑"
|
||
return ("quota_check_failed: 配额查询失败，拒绝新回合"
|
||
"（查库异常，非配额用完）")
|
||
if quota is None:
|
||
return None
|
||
if usage_total:
|
||
model = (task or {}).get("model") or self.model
|
||
micros = price_usd_micros(
|
||
model, usage_total.get("input_tokens", 0),
|
||
usage_total.get("cached_input_tokens", 0),
|
||
usage_total.get("output_tokens", 0))
|
||
used += micros or 0
|
||
if micros is None:
|
||
# 无价模型：本轮 token 计入无价上限（USD 口
|
||
# 径里它是 0，不会在上面那道里被拦住）。
|
||
unpriced_used += (
|
||
max(0, int(usage_total.get("input_tokens", 0)))
|
||
+ max(0, int(usage_total.get("output_tokens", 0))))
|
||
if used >= quota:
|
||
return ("quota_exceeded: 本月配额已用完（已用 %d / "
|
||
"配额 %d 微美元，%s）" % (used, quota, month))
|
||
cap = unpriced_token_cap()
|
||
if unpriced_used >= cap:
|
||
return ("quota_exceeded: 本月无价模型 token 已用完"
|
||
"（已用 %d / 上限 %d，%s）"
|
||
% (unpriced_used, cap, month))
|
||
return None
|
||
|
||
def _ready_tasks(self):
|
||
"""当前就绪任务列表（pick_next 的批量版）。
|
||
|
||
逻辑与 pick_next 完全一致（预算门、依赖满足、执行器
|
||
过滤），只是返回全部而不是第一个。自动扩缩容用它算
|
||
队列深度：只数真正能立刻派的任务，被依赖卡住的
|
||
queued 任务不算，否则 scaler 会为永远派不出去的任务
|
||
扩容。依赖成环时返回 []（与 pick_next 的 None 一致）。
|
||
"""
|
||
if self._budget_exceeded():
|
||
return []
|
||
tasks = self.load_tasks()
|
||
completed_ids = {t["id"] for t in tasks
|
||
if t["status"] in COMPLETED_STATUSES}
|
||
try:
|
||
ready = ready_tasks(tasks, completed_ids)
|
||
except ValueError:
|
||
return []
|
||
mine = self._executor_name()
|
||
if self.claim_executor_only:
|
||
return [t for t in ready if t.get("executor") == mine]
|
||
return [t for t in ready
|
||
if t.get("executor") in (None, "", mine)]
|
||
|
||
def pick_next(self):
|
||
"""下一个可派任务，或 None（无就绪任务 / 预算超限 / 依赖成环）。
|
||
|
||
依赖满足的定义：依赖任务状态为 completed 或 merged。
|
||
依赖成环时 ready_tasks 抛 ValueError，worker 视为无就绪任务
|
||
（成环任务本就永远派不出去，run_all 不应因此崩溃）。
|
||
"""
|
||
ready = self._ready_tasks()
|
||
return ready[0] if ready else None
|
||
|
||
# -- 并发运行状态 --------------------------------------------------
|
||
def _running_set(self, task_id, proc=None, container=None):
|
||
with self._running_lock:
|
||
entry = self._running.setdefault(task_id, {})
|
||
if proc is not None:
|
||
entry["proc"] = proc
|
||
if container is not None:
|
||
entry["container"] = container
|
||
|
||
def _running_clear(self, task_id):
|
||
with self._running_lock:
|
||
self._running.pop(task_id, None)
|
||
|
||
def _running_snapshot(self):
|
||
with self._running_lock:
|
||
return {tid: dict(entry)
|
||
for tid, entry in self._running.items()}
|
||
|
||
def _running_count(self):
|
||
"""当前占用槽位数（_running_lock 下计数）。
|
||
|
||
启动中的任务已在 _claim 里占位（空条目），所以这里数
|
||
到的既包括已起进程 / 容器的任务，也包括已认领未起
|
||
进程的任务；注册表心跳与认领门共用这个口径。
|
||
"""
|
||
with self._running_lock:
|
||
return len(self._running)
|
||
|
||
def _wait_for_running(self, timeout=None):
|
||
"""等 _running 变空；timeout 为 None 时无限等。
|
||
|
||
返回 True 表示已空，False 表示超时仍有任务在跑。
|
||
run_loop 退出时的优雅排空用它：先等任务自然结束，
|
||
超时后调用方再杀残留并标 interrupted。
|
||
"""
|
||
deadline = None if timeout is None else time.time() + timeout
|
||
while True:
|
||
with self._running_lock:
|
||
if not self._running:
|
||
return True
|
||
if deadline is not None and time.time() >= deadline:
|
||
return False
|
||
time.sleep(0.2)
|
||
|
||
def _heartbeat_running_tasks(self):
|
||
"""给所有正在跑的任务刷 last_heartbeat。
|
||
|
||
run_loop 的心跳线程调它：任务在跑时心跳保持新鲜，
|
||
自动陈旧回收（sweep）才不会误杀活任务；worker 真死后
|
||
心跳停了，任务超过阈值才会被别的 worker 回收。
|
||
"""
|
||
for task_id in self._running_snapshot():
|
||
try:
|
||
self.heartbeat(task_id)
|
||
except Exception:
|
||
pass
|
||
|
||
# -- 取消（server cancel 语义，见模块 docstring）-------------------
|
||
def cancel(self, task_id):
|
||
"""把 queued/running 任务标 interrupted；已终态或不存在返回 False。
|
||
|
||
正在跑的子进程要真杀：先杀进程组（见 _kill_proc_group），
|
||
再标状态；已终态或不存在返回 False。
|
||
"""
|
||
task = self.get_task(task_id)
|
||
if task is None or task["status"] in TERMINAL_STATUSES:
|
||
return False
|
||
self._cancel_requested.add(task_id)
|
||
entry = self._running_snapshot().get(task_id) or {}
|
||
if entry.get("proc") is not None:
|
||
self._kill_proc_group(entry["proc"])
|
||
try:
|
||
entry["proc"].wait(timeout=3)
|
||
except Exception:
|
||
pass
|
||
# Docker 取消：按任务派生的容器名真杀，不只改数据库状态。
|
||
# 同步路径（runner 阻塞在 docker run 里）下 proc 为空，
|
||
# 此时靠容器名杀容器；kill 尽力而为，失败也不影响后续
|
||
# 的状态标记。
|
||
if entry.get("container") is not None and \
|
||
hasattr(self.executor, "kill_container"):
|
||
try:
|
||
self.executor.kill_container(entry["container"])
|
||
except Exception:
|
||
pass
|
||
self._running_clear(task_id)
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
cur = conn.execute(
|
||
"UPDATE tasks SET status = 'interrupted', updated_at = ?"
|
||
" WHERE id = ? AND status NOT IN"
|
||
" ('completed', 'failed', 'interrupted', 'merged',"
|
||
" 'merge-failed')",
|
||
(time.time(), task_id),
|
||
)
|
||
conn.commit()
|
||
claimed = cur.rowcount == 1
|
||
finally:
|
||
conn.close()
|
||
if not claimed:
|
||
# 并发竞态：run_one 的轮询已看到本次取消的标记、先一步
|
||
# 把任务标成 interrupted——中断确实由本次取消触发，返回 True。
|
||
current = self.get_task(task_id)
|
||
if current is not None and current["status"] == "interrupted":
|
||
claimed = True
|
||
if claimed:
|
||
self._audit(task["project_id"], "worker_cancelled", task_id)
|
||
log_event(self.logger, "worker_cancelled", task_id=task_id,
|
||
project_id=task["project_id"], status="interrupted")
|
||
return claimed
|
||
|
||
def _docker_cancel_watchdog(self, task_id, container_name,
|
||
stop_event):
|
||
"""Docker 同步路径的取消看门狗（_run_attempt 起的线程）。
|
||
|
||
同步路径阻塞在 docker run 里，worker 的 cancel() 在
|
||
另一线程只能改库、清 _running 项，阻塞中的 run 本身
|
||
不会醒——P0 安全批前这就是“取消杀不掉容器”的根因。
|
||
本线程每 1 秒查一次库：任务变 interrupted 或本进程
|
||
收到取消标志时，按容器名真杀容器（kill+rm，尽力而
|
||
为），然后退出；调用方在 _run_attempt 结束时置
|
||
stop_event 并 join。查库失败时当没取消，不误杀。
|
||
"""
|
||
while not stop_event.wait(1.0):
|
||
try:
|
||
if task_id in self._cancel_requested:
|
||
cancelled = True
|
||
else:
|
||
db_task = self.get_task(task_id)
|
||
cancelled = (db_task is not None and
|
||
db_task.get("status") == "interrupted")
|
||
except Exception:
|
||
cancelled = False
|
||
if not cancelled:
|
||
continue
|
||
try:
|
||
kill = getattr(self.executor, "kill_container", None)
|
||
if kill is not None and container_name:
|
||
kill(container_name)
|
||
log_event(self.logger, "worker_docker_killed",
|
||
task_id=task_id,
|
||
message="cancel watchdog killed container "
|
||
"%s (task interrupted/cancelled)"
|
||
% container_name)
|
||
except Exception:
|
||
pass
|
||
return
|
||
|
||
def _docker_quota_watchdog(self, task_id, container_name,
|
||
stop_event, quota_check, interval=30.0):
|
||
"""Docker 同步路径的配额看门狗（P1，复用取消看门狗模式）。
|
||
|
||
同步路径阻塞在 docker run 里，run_one 的边界检查要等
|
||
本轮返回才跑得到——配额在别处被花完时，本轮会继续烧
|
||
到自然结束。本线程每 interval 秒（默认 30 秒，不超过
|
||
30 秒）用 quota_check 查一次已记账配额：命中时按容器
|
||
名真杀容器并记 _quota_hit_tasks，_run_attempt 在同步
|
||
返回后把结果改写成 quota_exceeded，run_one 走统一的
|
||
配额中断收尾（interrupted + 配额原因 + 已花成本照记，
|
||
下一轮不再续跑）。查库失败时 quota_check 返回 False
|
||
（run_one 的 _quota_hit 会把 quota_check_failed 前缀
|
||
过滤掉，见 _quota_exceeded docstring），本线程只继续
|
||
轮询、不杀当前轮；下一轮边界仍会拒绝。诚实边界：看
|
||
门狗只能看到已记账用量 + 往轮累计用量，本轮容器内尚
|
||
未回传的 token 它看不到——那部分在结果返回后由边界
|
||
门拦住，不会悄悄续跑下一轮。
|
||
"""
|
||
while not stop_event.wait(interval):
|
||
try:
|
||
over = bool(quota_check()) if quota_check else False
|
||
except Exception:
|
||
over = False
|
||
if not over:
|
||
continue
|
||
self._mark_quota_hit(task_id)
|
||
try:
|
||
kill = getattr(self.executor, "kill_container", None)
|
||
if kill is not None and container_name:
|
||
kill(container_name)
|
||
log_event(self.logger, "worker_docker_killed",
|
||
task_id=task_id,
|
||
message="quota watchdog killed container %s "
|
||
"(quota exceeded)" % container_name)
|
||
except Exception:
|
||
pass
|
||
return
|
||
|
||
def cleanup_stale_containers(self):
|
||
"""启动清理兜底：清掉上一轮残留的 codex- 容器。
|
||
|
||
只在执行器有 cleanup_stale_containers 时做（Docker）；
|
||
列出或删除失败只记日志，绝不抛异常炸掉 worker。
|
||
返回已清理的容器名列表（非 Docker 执行器返回 []）。
|
||
"""
|
||
cleanup = getattr(self.executor, "cleanup_stale_containers", None)
|
||
if cleanup is None:
|
||
return []
|
||
try:
|
||
cleaned = cleanup()
|
||
except Exception as exc:
|
||
log_event(self.logger, "worker_cleanup_failed",
|
||
message="cleanup stale containers failed: %s" % exc)
|
||
return []
|
||
if cleaned:
|
||
log_event(self.logger, "worker_cleanup",
|
||
message="cleaned stale containers: %s"
|
||
% ",".join(cleaned))
|
||
return cleaned
|
||
|
||
def _was_cancelled(self, task_id):
|
||
if task_id in self._cancel_requested:
|
||
return True
|
||
task = self.get_task(task_id)
|
||
return task is not None and task["status"] == "interrupted"
|
||
|
||
# -- 缺 key 门 ------------------------------------------------------
|
||
def _executor_name(self):
|
||
"""执行器名：DockerExecutor -> docker，CloudExecutor -> cloud，
|
||
其它 -> local。cloud 绝不能回落成 local，否则 cloud worker
|
||
会认领 local 池的任务，池过滤就失效了。"""
|
||
if isinstance(self.executor, DockerExecutor):
|
||
return "docker"
|
||
if isinstance(self.executor, CloudExecutor):
|
||
return "cloud"
|
||
return "local"
|
||
def key_available(self):
|
||
"""只查存在性：名在环境里且值非空即 True；值本身绝不返回。"""
|
||
try:
|
||
value = self._env.get(self.env_key)
|
||
except Exception:
|
||
return False
|
||
return value is not None and value != ""
|
||
|
||
def _use_popen(self):
|
||
"""是否走进程模式：注入了 process_factory，或执行器是
|
||
默认 runner 的 LocalExecutor（真实 subprocess）。"""
|
||
if self.process_factory is not None:
|
||
return True
|
||
return (isinstance(self.executor, LocalExecutor)
|
||
and getattr(self.executor, "runner", None)
|
||
is _sandbox_default_runner
|
||
and hasattr(self.executor, "popen_task"))
|
||
|
||
def _gateway_context(self, task):
|
||
"""网关注入：xcmapi -> (None, None)；sub2api -> (provider, env)。
|
||
|
||
sub2api 时按任务属主（tasks.created_by）幂等开通并取 key；
|
||
开通失败直接抛（任务失败，不跑错网关）。env 含
|
||
SUB2API_API_KEY、SUB2API_BASE_URL，并把网关 host 追加进
|
||
NO_PROXY/no_proxy（否则 codex 经 egress 代理访问内网网关
|
||
会被挡；proxy_env 的 NO_PROXY 会被这份 env 覆盖，所以这里
|
||
直接给超集 localhost,127.0.0.1,<网关 host>）。
|
||
"""
|
||
if _gateway_name() != "sub2api":
|
||
return None, None
|
||
origin = _gateway_origin()
|
||
if not origin:
|
||
raise RuntimeError("网关为 sub2api 但 SUB2API_BASE_URL 未配置")
|
||
owner = task.get("created_by")
|
||
if not owner and task.get("parent_task_id"):
|
||
# 协调员子任务：属主跟父任务走（子任务行可能无 created_by）。
|
||
try:
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
row = conn.execute(
|
||
"SELECT created_by FROM tasks WHERE id = ?",
|
||
(task["parent_task_id"],)).fetchone()
|
||
finally:
|
||
conn.close()
|
||
if row is not None:
|
||
try:
|
||
owner = row[0]
|
||
except (TypeError, IndexError):
|
||
owner = None
|
||
except Exception:
|
||
owner = None
|
||
if not owner:
|
||
raise RuntimeError("任务缺少 created_by，无法解析网关属主")
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
key = _ensure_sub2api_key(conn, owner)
|
||
finally:
|
||
conn.close()
|
||
host = origin.split("://", 1)[-1].split("/", 1)[0]
|
||
host = host.split(":", 1)[0]
|
||
no_proxy = "localhost,127.0.0.1"
|
||
if host and host not in ("localhost", "127.0.0.1"):
|
||
no_proxy += "," + host
|
||
env = {
|
||
"SUB2API_API_KEY": key,
|
||
"SUB2API_BASE_URL": origin,
|
||
"NO_PROXY": no_proxy,
|
||
"no_proxy": no_proxy,
|
||
}
|
||
return "sub2api", env
|
||
|
||
def _ensure_gateway(self, task):
|
||
"""任务 dict 上的网关缓存：已算过直接取，否则算一次并存回。
|
||
|
||
键是 _gateway_provider/_gateway_env（与 _resume 同款下划线
|
||
私有键约定，不进库）。run_one 算过后 _run_attempt 不再
|
||
重算；_run_fork 直达 _run_attempt 时在这里补算。
|
||
"""
|
||
if "_gateway_provider" in task:
|
||
return task["_gateway_provider"], task["_gateway_env"]
|
||
provider, env = self._gateway_context(task)
|
||
task["_gateway_provider"] = provider
|
||
task["_gateway_env"] = env
|
||
return provider, env
|
||
|
||
def _sub2api_default_model(self, gateway_env):
|
||
"""sub2api 默认模型：环境变量优先，否则取该用户网关模型首个。
|
||
|
||
网关模型目录因用户而异（key 绑定的组决定），worker 构造时
|
||
没有属主，只能到任务时现取；取不到（组里无模型）就 loud
|
||
fail，不让任务拿着 xcmapi 的模型名去撞 sub2api。
|
||
"""
|
||
raw = os.environ.get("CLOUD_SUB2API_DEFAULT_MODEL", "").strip()
|
||
if raw:
|
||
return raw
|
||
origin = _gateway_origin()
|
||
key = (gateway_env or {}).get("SUB2API_API_KEY")
|
||
if not origin or not key:
|
||
raise RuntimeError("sub2api 默认模型无法解析：缺少网关 key")
|
||
models = _list_gateway_models(origin, key)
|
||
if not models:
|
||
raise RuntimeError(
|
||
"sub2api 网关暂无可用模型（组里未挂上游账号？）")
|
||
return models[0]
|
||
|
||
def _task_seed_dir(self, task):
|
||
"""任务附件暂存目录（派发时 seed 用）：无附件返回 None；
|
||
有附件行但目录已不在（如终局后已清理）也返回 None——
|
||
续跑任务不需要重 seed（文件已在 worktree 里），静默跳过
|
||
比报错更正确（重试走已存在续跑时包早已删了）。"""
|
||
try:
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
row = conn.execute(
|
||
"SELECT bundle_id FROM task_attachments"
|
||
" WHERE task_id = ? LIMIT 1",
|
||
(task["id"],),
|
||
).fetchone()
|
||
finally:
|
||
conn.close()
|
||
except Exception:
|
||
return None
|
||
if row is None:
|
||
return None
|
||
bundle_id = row[0] if isinstance(row, tuple) else row["bundle_id"]
|
||
bundle_dir = os.path.join(self.upload_dir, bundle_id)
|
||
return bundle_dir if os.path.isdir(bundle_dir) else None
|
||
|
||
def _cleanup_task_seed(self, task):
|
||
"""删任务附件暂存目录：终局清理的唯一入口。
|
||
|
||
正常终局、缺 key 门、仓库解析失败三路都要走这里——
|
||
后两路在 run_one 里提前返回，够不到正常终局后的清理
|
||
（2026-10-27 实弹：仓库解析失败的任务暂存目录残留）。
|
||
取消走 server 侧删，这里删不到也不报错；删失败不影响
|
||
任务结果本身。
|
||
"""
|
||
try:
|
||
_seed = self._task_seed_dir(task)
|
||
if _seed:
|
||
shutil.rmtree(_seed, ignore_errors=True)
|
||
except Exception:
|
||
pass
|
||
|
||
def _execute_via_process(self, task, kind="dispatch",
|
||
deadline_seconds=None, seed_host_dir=None,
|
||
quota_check=None):
|
||
"""进程模式执行一轮：启动、轮询、取消时杀进程组。
|
||
|
||
返回与同步路径同款 result dict（status/stdout/stderr/
|
||
host_status/returncode）。取消时返回 status='cancelled'
|
||
的标记，调用方据此保持 interrupted；deadline_seconds
|
||
到期时杀进程组并返回 status='timeout' 的标记，调用方
|
||
据此置 failed（超时是失败，不是取消）。quota_check 不为
|
||
空时每 quota_poll_interval_seconds（缺省 30 秒）查一次
|
||
配额（轮询本身 50 毫秒一圈，逐圈查会把库打爆），超限时
|
||
同样杀进程组并返回 status='quota_exceeded' 的内部标记
|
||
——调用方据此走配额中断收尾，绝不能把这个标记原样写
|
||
进 tasks.status（不在合法状态集里）。Docker 路径见
|
||
_docker_call_with_quota_poll（同款轮询）；同步路径没有
|
||
轮询钩子，只走边界检查。
|
||
"""
|
||
task_id = task["id"]
|
||
gateway_provider, gateway_env = self._ensure_gateway(task)
|
||
if self.process_factory is not None:
|
||
proc = self.process_factory(task)
|
||
else:
|
||
# seed 只对 dispatch/goal 首轮有意义：run 续跑不传
|
||
# （文件已在 worktree 里）；popen_task 内部同样只在
|
||
# dispatch/goal 命令上追加 --seed-dir。
|
||
proc = self.executor.popen_task(
|
||
task, kind=kind, seed_dir=seed_host_dir,
|
||
extra_env=gateway_env, model_provider=gateway_provider)
|
||
self._running_set(task_id, proc=proc)
|
||
try:
|
||
self._set_child_pid(task_id, getattr(proc, "pid", None))
|
||
except Exception:
|
||
pass
|
||
cancelled = False
|
||
timed_out = False
|
||
quota_hit = False
|
||
started = time.time()
|
||
last_quota_check = 0.0
|
||
last_db_cancel_check = 0.0
|
||
quota_interval = getattr(
|
||
self, "quota_poll_interval_seconds", 30)
|
||
if quota_interval is None:
|
||
quota_interval = 30
|
||
try:
|
||
while proc.poll() is None:
|
||
if task_id in self._cancel_requested or self._loop_stop:
|
||
cancelled = True
|
||
self._kill_proc_group(proc)
|
||
break
|
||
# P0 安全批后：popen 轮询也要轮询库状态，不只
|
||
# 看本进程标志——控制台取消 / sweep 是另一进程
|
||
# 改库，旧代码只看本进程标志时杀不掉本进程的
|
||
# 子进程。节流到每 1 秒查一次库，避免 0.05 秒
|
||
# 轮询把库打爆；查库失败时当没取消，不误杀。
|
||
if time.time() - last_db_cancel_check >= 1.0:
|
||
last_db_cancel_check = time.time()
|
||
try:
|
||
db_task = self.get_task(task_id)
|
||
except Exception:
|
||
db_task = None
|
||
if db_task is not None and \
|
||
db_task.get("status") == "interrupted":
|
||
cancelled = True
|
||
self._kill_proc_group(proc)
|
||
break
|
||
if deadline_seconds is not None and \
|
||
time.time() - started >= deadline_seconds:
|
||
timed_out = True
|
||
self._kill_proc_group(proc)
|
||
break
|
||
if quota_check is not None and \
|
||
time.time() - last_quota_check >= quota_interval:
|
||
last_quota_check = time.time()
|
||
try:
|
||
over = quota_check()
|
||
except Exception:
|
||
over = False
|
||
if over:
|
||
quota_hit = True
|
||
self._kill_proc_group(proc)
|
||
break
|
||
time.sleep(0.05)
|
||
try:
|
||
out, err = proc.communicate(timeout=5)
|
||
except Exception:
|
||
try:
|
||
proc.kill()
|
||
except Exception:
|
||
pass
|
||
out, err = proc.communicate()
|
||
finally:
|
||
self._running_clear(task_id)
|
||
if cancelled:
|
||
return {"executor": "local", "task_id": task_id,
|
||
"status": "cancelled", "stdout": out or "",
|
||
"stderr": err or "", "returncode": proc.returncode,
|
||
"host_status": None}
|
||
if quota_hit:
|
||
return {"executor": "local", "task_id": task_id,
|
||
"status": "quota_exceeded", "stdout": out or "",
|
||
"stderr": err or "", "returncode": proc.returncode,
|
||
"host_status": None}
|
||
if timed_out:
|
||
return {"executor": "local", "task_id": task_id,
|
||
"status": "timeout", "stdout": out or "",
|
||
"stderr": err or "", "returncode": proc.returncode,
|
||
"host_status": None}
|
||
fake = SimpleNamespace(returncode=proc.returncode,
|
||
stdout=out or "", stderr=err or "")
|
||
if kind in ("goal", "goal-resume", "goal-resume-breakpoint") \
|
||
and hasattr(
|
||
self.executor, "_goal_result_from_runner"):
|
||
return self.executor._goal_result_from_runner(task_id, fake)
|
||
if hasattr(self.executor, "_result_from_runner"):
|
||
return self.executor._result_from_runner(task_id, fake)
|
||
return {"executor": "local", "task_id": task_id,
|
||
"status": "completed" if proc.returncode == 0 else "failed",
|
||
"stdout": out or "", "stderr": err or "",
|
||
"returncode": proc.returncode, "host_status": None}
|
||
|
||
# -- 多租户隔离 ----------------------------------------------------
|
||
def _tenant_safe_name(self, project_id):
|
||
"""项目 id 的安全目录名：与 repos.checkout_dir 同规则。"""
|
||
if project_id is None:
|
||
raise ValueError("project_id 缺失，无法确定租户目录")
|
||
raw = str(project_id)
|
||
safe = re.sub(r"[^A-Za-z0-9_.-]", "-", raw).strip(".")
|
||
if not safe or safe in (".", ".."):
|
||
raise ValueError("project_id 无法映射为安全的租户目录名: %r"
|
||
% raw)
|
||
return safe
|
||
|
||
def _tenant_dirs(self, project_id):
|
||
"""某项目的租户宿主目录（不存在时创建并播种）。
|
||
|
||
返回 dict：work / codex_home / repo_checkout / test_repo
|
||
（均为宿主绝对路径）。播种规则：
|
||
- codex-home：只复制模板的 config.toml（relay 端点与
|
||
provider 名，不含密钥值）；模板缺 config.toml 时直接
|
||
报错——空 home 会让 codex 去 api.openai.com 且无 auth，
|
||
绝不能静默放行（见 PR #18 的教训）。
|
||
- test-repo：整目录复制模板（它是 sample.py 夹具，含
|
||
.git 历史也无妨，本来就是公开的起始代码）。
|
||
- work / repo_checkout：只建目录。
|
||
并发下两个同项目任务同时播种是安全的：mkdir 用
|
||
exist_ok，复制前判存在且复制内容相同（模板），即使
|
||
race 到重复复制也不会写坏租户自己的数据（租户数据
|
||
在复制完成后才产生）。
|
||
"""
|
||
import shutil
|
||
|
||
safe = self._tenant_safe_name(project_id)
|
||
base = os.path.join(self.tenants_root, safe)
|
||
dirs = {
|
||
"work": os.path.join(base, "work"),
|
||
"codex_home": os.path.join(base, "codex-home"),
|
||
"repo_checkout": os.path.join(base, safe),
|
||
"test_repo": os.path.join(base, "test-repo"),
|
||
}
|
||
# 注意绝不 mkdir repo_checkout：repo_url 任务的检出由
|
||
# resolve_repo 负责 clone，空目录先占位会导致它误判
|
||
# “目录已存在但不是 git 仓库”而拒绝 clone；_tenant_mounts
|
||
# 里按该目录是否存在决定是否挂载它（不存在说明任务
|
||
# 不用它，走 test-repo）。
|
||
for key in ("work", "codex_home", "test_repo"):
|
||
os.makedirs(dirs[key], exist_ok=True)
|
||
home_config = os.path.join(dirs["codex_home"], "config.toml")
|
||
if not os.path.exists(home_config):
|
||
template = os.path.join(self.tenant_template_home,
|
||
"config.toml")
|
||
if not os.path.exists(template):
|
||
raise RuntimeError(
|
||
"租户 codex-home 模板缺 config.toml：%s（空 home"
|
||
" 会让 codex 去 api.openai.com 且无 auth，拒绝"
|
||
" 派发）" % template)
|
||
shutil.copyfile(template, home_config)
|
||
if self.websearch_mcp:
|
||
# Docker 租户任务的 codex-home 会原样挂载进容器，
|
||
# 所以这里写容器内路径；宿主侧脚本文件由播种助手
|
||
# 从运行中代码的 cloud/ 目录拷过来（生产即
|
||
# /opt/codex-app-server/cloud/）。播种失败直接
|
||
# 抛错、任务不派发——静默无搜索比 loud 失败更糟，
|
||
# 尤其对已宣传“有搜索”的任务。
|
||
_ensure_websearch_mcp(
|
||
dirs["codex_home"],
|
||
command_args=[
|
||
"python3",
|
||
"/var/lib/codex-app-server/codex-home/"
|
||
"websearch_mcp.py"])
|
||
if self.imagegen_mcp:
|
||
# 与 websearch 同款：播种失败直接抛错、任务不派发
|
||
# ——静默无生图比 loud 失败更糟，尤其对已宣传
|
||
# “能生图”的任务。key 不在这里写，只写
|
||
# command/args；key 由 _prepare_tenant 的 env 块给。
|
||
_ensure_imagegen_mcp(
|
||
dirs["codex_home"],
|
||
command_args=[
|
||
"python3",
|
||
"/var/lib/codex-app-server/codex-home/"
|
||
"imagegen_mcp.py"])
|
||
if self.seed_skills:
|
||
# 技能播种失败绝不阻塞任务：技能是可选的流程
|
||
# 说明，不是 websearch 那种“已宣传就必须有”的
|
||
# 工具；播种助手本身对缺包、已存在、租户改动
|
||
# 都是静默处理，只在文件系统本身出错时才抛，
|
||
# 那种情况下任务仍应派发（技能缺失只影响流程，
|
||
# 不影响 agent 本身跑起来）。
|
||
try:
|
||
_ensure_skills(dirs["codex_home"])
|
||
except Exception as exc:
|
||
# 只记日志不阻塞：技能缺失只影响流程说明，
|
||
# 不应把一次可选播种失败变成任务派发失败。
|
||
# 但绝不静默吞掉——日志里要留下来供排查。
|
||
try:
|
||
self.logger.warning(
|
||
"skill seeding failed: %s", exc)
|
||
except Exception:
|
||
pass
|
||
if not os.path.exists(os.path.join(dirs["test_repo"], ".git")) \
|
||
and not os.path.exists(dirs["test_repo"] + "/sample.py"):
|
||
# 目标为空目录（刚 mkdir 出来的）才播种；已有内容
|
||
# 时绝不覆盖（那是该项目自己的改动或上次播种结果）。
|
||
if not os.listdir(dirs["test_repo"]):
|
||
if not os.path.exists(self.tenant_template_repo):
|
||
raise RuntimeError(
|
||
"租户 test-repo 模板不存在：%s"
|
||
% self.tenant_template_repo)
|
||
os.rmdir(dirs["test_repo"])
|
||
shutil.copytree(self.tenant_template_repo,
|
||
dirs["test_repo"])
|
||
return dirs
|
||
|
||
def _tenant_mounts(self, project_id):
|
||
"""某项目任务的 Docker 挂载列表（字符串，供执行器解析）。
|
||
|
||
容器内路径与非租户模式完全一致，只是宿主源按项目分开：
|
||
/app/work、codex-home、repos/<pid>、test-repo 全部指向
|
||
该项目的租户目录；代码目录 /app:ro 由执行器基础挂载
|
||
提供，不在这里重复。静态 --docker-extra-mounts 在租户
|
||
模式下被这份列表替换（否则共享挂载会继续泄漏）。
|
||
repos 挂载只在检出目录已存在时才加（repo_url 任务在
|
||
派发前已由 _apply_project_repo clone 好；test-repo 任务
|
||
不用它，不存在时硬挂载会被执行器的存在性检查拒绝）。
|
||
"""
|
||
dirs = self._tenant_dirs(project_id)
|
||
mounts = [
|
||
"%s:/app/work:rw" % dirs["work"],
|
||
"%s:/var/lib/codex-app-server/codex-home:rw"
|
||
% dirs["codex_home"],
|
||
"%s:/var/lib/codex-app-server/test-repo:rw"
|
||
% dirs["test_repo"],
|
||
]
|
||
if os.path.exists(dirs["repo_checkout"]):
|
||
mounts.append(
|
||
"%s:/var/lib/codex-app-server/repos/%s:rw"
|
||
% (dirs["repo_checkout"],
|
||
self._tenant_safe_name(project_id)))
|
||
# 环境检出：整目录挂载（按项目分，与 repo_checkout 同款
|
||
# 隔离语义）。环境任务派发前 worker 已准备好检出，目录
|
||
# 存在时才挂载；不存在说明该项目暂无环境，不硬挂载。
|
||
# base = work 的父目录（即租户项目根目录）。
|
||
envs_dir = os.path.join(os.path.dirname(dirs["work"]), "envs")
|
||
if os.path.isdir(envs_dir):
|
||
mounts.append(
|
||
"%s:/var/lib/codex-app-server/envs:rw" % envs_dir)
|
||
return mounts
|
||
|
||
def _tenant_network(self, project_id):
|
||
"""某项目的 Docker 网络名（容器跨项目网络隔离用）。"""
|
||
return "codex-" + self._tenant_safe_name(project_id)
|
||
|
||
# -- 环境 ----------------------------------------------------------
|
||
def _env_host_dir(self, env):
|
||
"""环境的宿主检出目录（worker 准备检出时写这里）。
|
||
|
||
非租户：repos_dir/envs/<env_id>（与项目检出同根，Docker
|
||
非租户挂载 repos 根目录时它自然在容器可见路径下）。
|
||
租户 + Docker：tenants_root/<项目>/envs/<env_id>，由
|
||
_tenant_mounts 挂到容器 /var/lib/codex-app-server/envs。
|
||
env_id 是 server 生成的 hex，仍按 checkout_dir 同款规则
|
||
过滤安全字符，防路径穿越。"""
|
||
if self.tenant_isolation and isinstance(
|
||
self.executor, DockerExecutor):
|
||
safe_proj = self._tenant_safe_name(env["project_id"])
|
||
return checkout_dir(
|
||
os.path.join(self.tenants_root, safe_proj, "envs"),
|
||
env["id"])
|
||
return checkout_dir(
|
||
os.path.join(self.repos_dir, "envs"), env["id"])
|
||
|
||
def _env_repo_path(self, env):
|
||
"""任务派发时传给执行器的 repo 路径：租户 Docker 用容器
|
||
内路径，其余用宿主路径——与 _apply_project_repo 的项目
|
||
检出路径约定完全一致，执行器不用为环境另起分支。"""
|
||
if self.tenant_isolation and isinstance(
|
||
self.executor, DockerExecutor):
|
||
return "/var/lib/codex-app-server/envs/%s" % env["id"]
|
||
return self._env_host_dir(env)
|
||
|
||
def _fetch_environment(self, env_id):
|
||
"""读环境行；表不存在（老库未跑 server init_db）或查不到
|
||
时返回 None，调用方按“环境不存在”处理，绝不炸 worker。"""
|
||
try:
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
row = conn.execute(
|
||
"SELECT * FROM environments WHERE id = ?",
|
||
(env_id,)).fetchone()
|
||
finally:
|
||
conn.close()
|
||
except Exception:
|
||
return None
|
||
if row is None:
|
||
return None
|
||
|
||
def _get(name):
|
||
try:
|
||
return row[name]
|
||
except (KeyError, IndexError, TypeError):
|
||
return None
|
||
|
||
try:
|
||
return {
|
||
"id": row["id"],
|
||
"project_id": row["project_id"],
|
||
"name": row["name"],
|
||
"branch": _get("branch"),
|
||
"executor": _get("executor"),
|
||
"status": row["status"],
|
||
"error_text": _get("error_text"),
|
||
"repo_url": _get("repo_url"),
|
||
}
|
||
except (KeyError, IndexError, TypeError):
|
||
return None
|
||
|
||
def _run_git(self, cmd):
|
||
"""跑一条 git 命令：repos_runner 可注入（单测用假 runner，
|
||
接受 cmd 或 (cmd, timeout) 两种签名，与 repos._run 同款
|
||
兼容）；缺省走真实 subprocess。返回带 returncode/stdout/
|
||
stderr 的结果对象。"""
|
||
if self.repos_runner is not None:
|
||
try:
|
||
return self.repos_runner(cmd)
|
||
except TypeError:
|
||
return self.repos_runner(cmd, timeout=None)
|
||
return subprocess.run(cmd, capture_output=True, text=True)
|
||
|
||
def _prepare_environment(self, env):
|
||
"""准备一个环境的检出，返回错误文本或 None。
|
||
|
||
幂等：目录不存在时 clone，存在时先校验它是 git 仓库且
|
||
origin 与环境 repo_url 一致（不一致绝不强制改 remote，
|
||
直接失败），再 fetch；分支非空时 checkout 该分支（不存
|
||
在则失败，绝不自动建分支——分支名是用户选的，不应由
|
||
准备过程发明）。检出本身不做 reset/clean：任务 worktree
|
||
由 host.py 从检出 HEAD 新建，检出保持干净即可。"""
|
||
repo_url = env.get("repo_url")
|
||
if not repo_url:
|
||
return "environment has no repo_url"
|
||
target = self._env_host_dir(env)
|
||
if not os.path.exists(target):
|
||
os.makedirs(os.path.dirname(target), exist_ok=True)
|
||
result = self._run_git(
|
||
["git", "clone", repo_url, target])
|
||
if getattr(result, "returncode", 1) != 0:
|
||
err = (getattr(result, "stderr", "") or "").strip()
|
||
return "git clone 失败: %s" % (err or repo_url)
|
||
else:
|
||
git_path = os.path.join(target, ".git")
|
||
if not (os.path.isdir(git_path)
|
||
or os.path.isfile(git_path)):
|
||
return "检出目录已存在但不是 git 仓库，拒绝覆盖: %s" \
|
||
% target
|
||
origin = self._run_git(
|
||
["git", "-C", target, "remote", "get-url", "origin"])
|
||
if getattr(origin, "returncode", 1) != 0:
|
||
return "检出目录 %s 取不到 remote origin，拒绝准备" \
|
||
% target
|
||
if (getattr(origin, "stdout", "") or "").strip() != repo_url:
|
||
return "检出目录 origin 与环境 repo_url 不一致，拒绝准备"
|
||
fetched = self._run_git(
|
||
["git", "-C", target, "fetch", "origin"])
|
||
if getattr(fetched, "returncode", 1) != 0:
|
||
err = (getattr(fetched, "stderr", "") or "").strip()
|
||
return "git fetch origin 失败: %s" % (err or target)
|
||
branch = env.get("branch")
|
||
if branch:
|
||
checked = self._run_git(
|
||
["git", "-C", target, "checkout", branch])
|
||
if getattr(checked, "returncode", 1) != 0:
|
||
err = (getattr(checked, "stderr", "") or "").strip()
|
||
return "git checkout %s 失败: %s" % (branch, err or target)
|
||
return None
|
||
|
||
def _prepare_pending_environments(self, limit=5):
|
||
"""准备所有 status=creating 的环境（run_all 每轮开头调）。
|
||
|
||
每条环境独立 try：一条失败只把自己标 failed，不挡其它
|
||
环境和任务。状态写回用 CAS（WHERE status='creating'）：
|
||
准备期间用户归档了环境时 CAS 改不到行，准备结果就此
|
||
作废、状态保持 archived，绝不把归档覆盖回 ready。worker
|
||
中途死了、环境仍是 creating 时下一轮会重新准备（准备
|
||
本身幂等），没有“preparing 卡死”这种中间态。返回
|
||
[(env_id, status, error_or_None), ...] 供日志与单测。"""
|
||
try:
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
rows = conn.execute(
|
||
"SELECT * FROM environments WHERE status = 'creating'"
|
||
" ORDER BY created_at, id LIMIT ?",
|
||
(limit,)).fetchall()
|
||
finally:
|
||
conn.close()
|
||
except Exception as exc:
|
||
log_event(self.logger, "worker_env_list_failed",
|
||
message="worker env list failed: %s" % exc)
|
||
return []
|
||
results = []
|
||
for row in rows:
|
||
try:
|
||
env = {
|
||
"id": row["id"],
|
||
"project_id": row["project_id"],
|
||
"name": row["name"],
|
||
"branch": row["branch"] if "branch" in row.keys()
|
||
else None,
|
||
"executor": row["executor"] if "executor" in row.keys()
|
||
else None,
|
||
"repo_url": row["repo_url"] if "repo_url" in row.keys()
|
||
else None,
|
||
}
|
||
except Exception:
|
||
continue
|
||
try:
|
||
error = self._prepare_environment(env)
|
||
except Exception as exc:
|
||
error = str(exc)
|
||
status = "failed" if error else "ready"
|
||
try:
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
cur = conn.execute(
|
||
"UPDATE environments SET status = ?,"
|
||
" error_text = ?, updated_at = ?"
|
||
" WHERE id = ? AND status = 'creating'",
|
||
(status, error, time.time(), env["id"]))
|
||
conn.commit()
|
||
if cur.rowcount == 0:
|
||
# 准备期间被归档：状态以归档为准，不覆盖。
|
||
status = "archived"
|
||
finally:
|
||
conn.close()
|
||
except Exception as exc:
|
||
log_event(self.logger, "worker_env_update_failed",
|
||
message="worker env update failed: %s" % exc)
|
||
continue
|
||
results.append((env["id"], status, error))
|
||
return results
|
||
|
||
def _apply_environment_repo(self, task):
|
||
"""任务带 environment_id 时的派发：环境必须已 ready 且
|
||
属于本任务项目，执行器冲突时拒绝；通过后 task["repo"]
|
||
指到环境检出（租户 Docker 为容器内路径），后续执行器
|
||
与无环境任务完全同款——host.py 从检出 HEAD 为任务建
|
||
worktree，任务隔离、合并、提 PR 都不变。环境未 ready
|
||
时返回错误文本，调用方把任务标 failed（仓库解析失败
|
||
绝不重试，见 _apply_project_repo 的约定）。"""
|
||
env_id = task.get("environment_id")
|
||
env = self._fetch_environment(env_id)
|
||
if env is None:
|
||
return "environment not found: %s" % env_id
|
||
if env["project_id"] != task.get("project_id"):
|
||
return "environment belongs to another project"
|
||
if env["status"] != "ready":
|
||
return "environment not ready: %s" % env["status"]
|
||
task_executor = task.get("executor")
|
||
if env["executor"] and task_executor \
|
||
and env["executor"] != task_executor:
|
||
return "executor does not match environment executor"
|
||
task["repo"] = self._env_repo_path(env)
|
||
return None
|
||
|
||
# -- 项目仓库 ------------------------------------------------------
|
||
def _project_repo_url(self, project_id):
|
||
"""查任务所属项目的 repo_url；查不到项目、列缺失、表缺失时
|
||
一律返回 None（走原有默认 repo，不报错、不炸掉派发）。
|
||
repo_url 本身只在 projects 表里，绝不从任务 prompt 里猜。"""
|
||
try:
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
row = conn.execute(
|
||
"SELECT repo_url FROM projects WHERE id = ?",
|
||
(project_id,)).fetchone()
|
||
finally:
|
||
conn.close()
|
||
except Exception:
|
||
return None
|
||
if row is None:
|
||
return None
|
||
try:
|
||
url = row["repo_url"]
|
||
except (KeyError, IndexError, TypeError):
|
||
return None
|
||
return url or None
|
||
|
||
def _apply_project_repo(self, task):
|
||
"""派发前按项目 repo_url 解析仓库，返回错误文本或 None。
|
||
|
||
repo_url 为空：非租户模式不动 task，返回 None（走原有
|
||
默认 repo）；租户模式则把 task["repo"] 指到该项目的
|
||
test-repo 副本（容器内路径与原来一致，宿主源已按项目
|
||
分开），否则容器会 fallback 到共享 test-repo。
|
||
repo_url 非空：调 cloud.repos.resolve_repo 得到本地检出，
|
||
设 task["repo"] 再交给执行器——LocalExecutor 与
|
||
DockerExecutor 的 dispatch_task 都已读 task["repo"]，同一条
|
||
路径，不另起分支。租户模式下检出放在租户目录，task
|
||
["repo"] 设为容器内路径（挂载目标），而不是宿主检出
|
||
路径（worker 侧不再用 task["repo"] 当宿主路径）。
|
||
resolve 失败返回错误文本，调用方据此把任务置 failed
|
||
（仓库解析失败绝不标 completed，也不在此重试，避免对
|
||
不可用的仓库反复 clone 烧时间）。
|
||
|
||
任务带 environment_id 时不走项目仓库路径，直接走
|
||
_apply_environment_repo（环境检出已由 worker 准备好，
|
||
这里只校验 ready/归属/执行器并指过去，不再 clone）。
|
||
"""
|
||
if task.get("environment_id"):
|
||
return self._apply_environment_repo(task)
|
||
use_tenant = self.tenant_isolation and \
|
||
isinstance(self.executor, DockerExecutor)
|
||
repo_url = self._project_repo_url(task.get("project_id"))
|
||
if not repo_url:
|
||
if use_tenant:
|
||
try:
|
||
self._tenant_dirs(task.get("project_id"))
|
||
except (ValueError, RuntimeError) as exc:
|
||
return "租户目录准备失败: %s" % exc
|
||
task["repo"] = "/var/lib/codex-app-server/test-repo"
|
||
return None
|
||
try:
|
||
if use_tenant:
|
||
safe = self._tenant_safe_name(task.get("project_id"))
|
||
resolve_repo(
|
||
repo_url, os.path.join(self.tenants_root, safe),
|
||
task.get("project_id"), runner=self.repos_runner)
|
||
task["repo"] = \
|
||
"/var/lib/codex-app-server/repos/%s" % safe
|
||
else:
|
||
task["repo"] = resolve_repo(
|
||
repo_url, self.repos_dir, task.get("project_id"),
|
||
runner=self.repos_runner)
|
||
except (RepoError, ValueError) as exc:
|
||
return "仓库解析失败: %s" % exc
|
||
except Exception as exc:
|
||
return "仓库解析失败: %s" % exc
|
||
return None
|
||
|
||
# -- 执行 ----------------------------------------------------------
|
||
def _docker_task_context(self, task):
|
||
"""Docker 任务的租户挂载/网络/代理准备（_run_attempt 与
|
||
_run_fork 共用）。成功返回 (True, mounts, network,
|
||
proxy_env, None)；失败返回 (False, None, None, None,
|
||
error_text)。网络建失败绝不降级默认网络跑容器（降级会
|
||
让隔离悄悄失效）。"""
|
||
tenant_mounts = None
|
||
tenant_network = None
|
||
proxy_env = None
|
||
if self.tenant_isolation:
|
||
try:
|
||
tenant_mounts = self._tenant_mounts(
|
||
task.get("project_id"))
|
||
tenant_network = self._tenant_network(
|
||
task.get("project_id"))
|
||
except (ValueError, RuntimeError) as exc:
|
||
return (False, None, None, None,
|
||
"租户隔离准备失败: %s" % exc)
|
||
if not self.executor.ensure_network(tenant_network):
|
||
return (False, None, None, None,
|
||
"租户 Docker 网络创建失败: %s" % tenant_network)
|
||
if self.egress_proxy_port:
|
||
# 代理在宿主上，容器经所在网络的网关访问；租户模式
|
||
# 用项目网络，非租户用默认 bridge。
|
||
gw = self.executor.network_gateway(tenant_network)
|
||
if not gw:
|
||
return (False, None, None, None,
|
||
"egress 代理网关解析失败，拒绝无代理"
|
||
"运行（避免绕过出口白名单）")
|
||
proxy_url = "http://%s:%s" % (gw, self.egress_proxy_port)
|
||
proxy_env = {
|
||
"HTTP_PROXY": proxy_url,
|
||
"HTTPS_PROXY": proxy_url,
|
||
"http_proxy": proxy_url,
|
||
"https_proxy": proxy_url,
|
||
"NO_PROXY": "localhost,127.0.0.1",
|
||
"no_proxy": "localhost,127.0.0.1",
|
||
}
|
||
# 搜索 env 透传：Docker 容器不继承 worker 进程环境，只拿
|
||
# extra_env（-e），所以 worker 环境里的搜索配置若不显式
|
||
# 放进这里，容器内 codex -> MCP 永远看不到 SEARXNG_URL，
|
||
# auto 链会跳过 SearXNG、只剩 DuckDuckGo（实弹 e2e 已证实）。
|
||
# 只透传已设置的项，值绝不进日志；付费 key 与网关 key 一样
|
||
# 走 extra_env（容器内进程本就需要它调 API）。
|
||
search_env = {
|
||
k: os.environ[k]
|
||
for k in ("SEARXNG_URL", "SEARXNG_ENGINES",
|
||
"PERPLEXITY_API_KEY", "EXA_API_KEY",
|
||
"TAVILY_API_KEY", "BRAVE_API_KEY")
|
||
if os.environ.get(k)
|
||
}
|
||
if search_env:
|
||
proxy_env = _merge_env(proxy_env, search_env)
|
||
# MCP env must live in config.toml, not just docker -e:
|
||
# Codex's MCP subprocess does not inherit the Codex
|
||
# process env (2026-10-30实测：Codex bash 里有
|
||
# SEARXNG_URL + proxy，MCP 链却只有 duckduckgo，
|
||
# DNS 瞬间失败；给 config 加 [mcp_servers.websearch.env]
|
||
# 后同一任务立刻返回真实 SearXNG 结果）。网关在此时
|
||
# 才已知，所以这里、容器启动前写租户 config，绝不
|
||
# 在播种时写死一个网关。写失败只记日志，不阻塞任务。
|
||
if self.tenant_isolation and proxy_env:
|
||
try:
|
||
home = self._tenant_dirs(
|
||
task.get("project_id"))["codex_home"]
|
||
_ensure_websearch_mcp_env(home, proxy_env)
|
||
except Exception as exc:
|
||
self.logger.warning(
|
||
"websearch MCP env write failed: %s" % exc)
|
||
if self.tenant_isolation and self.imagegen_mcp:
|
||
# 图片生成 MCP env：系统级 IMAGE_GEN_*（worker 环境，
|
||
# 即主机 env 文件）+ 代理（供下载返回的图片 URL）+
|
||
# NO_PROXY（供直连本地网关 origin，不走 egress 代理）。
|
||
# 未配 IMAGE_GEN_API_KEY 时仍写 base/model，工具会
|
||
# 明确报“未配置”；绝不把任务自己的分组 2 key 塞进来。
|
||
try:
|
||
imagegen_env = dict(proxy_env) if proxy_env else {}
|
||
for key in ("IMAGE_GEN_BASE_URL", "IMAGE_GEN_API_KEY",
|
||
"IMAGE_GEN_MODEL"):
|
||
if os.environ.get(key):
|
||
imagegen_env[key] = os.environ[key]
|
||
base = (os.environ.get("IMAGE_GEN_BASE_URL", "")
|
||
or os.environ.get("SUB2API_BASE_URL", ""))
|
||
host = base.split("://", 1)[-1].split("/", 1)[0]
|
||
host = host.split(":", 1)[0]
|
||
no_proxy = imagegen_env.get("NO_PROXY", "")
|
||
parts = [p for p in no_proxy.split(",") if p]
|
||
for item in ("localhost", "127.0.0.1", host):
|
||
if item and item not in parts:
|
||
parts.append(item)
|
||
imagegen_env["NO_PROXY"] = ",".join(parts)
|
||
imagegen_env["no_proxy"] = ",".join(parts)
|
||
home = self._tenant_dirs(
|
||
task.get("project_id"))["codex_home"]
|
||
_ensure_imagegen_mcp_env(home, imagegen_env)
|
||
except Exception as exc:
|
||
self.logger.warning(
|
||
"imagegen MCP env write failed: %s" % exc)
|
||
return (True, tenant_mounts, tenant_network, proxy_env, None)
|
||
|
||
def _run_fork(self, task, quota_check=None):
|
||
"""分支任务执行：host.py fork 建分支 thread，再跑首轮。
|
||
返回 (result, error)，语义同 _run_attempt。fork 本身不
|
||
跑模型；首轮走 _run_attempt（dispatch 撞已存在 → run
|
||
续跑，三路取消/超时处理与普通任务一致）。首轮失败不
|
||
自动重试/fallback：失败后走重试入口重跑首轮（已存在
|
||
续跑），不重复 fork。"""
|
||
task_id = task["id"]
|
||
source_id = task.get("forked_from")
|
||
if not hasattr(self.executor, "fork_task"):
|
||
return ({"executor": self._executor_name(),
|
||
"task_id": task_id,
|
||
"status": "failed", "stdout": "",
|
||
"returncode": None, "host_status": None,
|
||
"stderr": ""},
|
||
"executor 不支持 fork")
|
||
is_docker = isinstance(self.executor, DockerExecutor)
|
||
if is_docker:
|
||
(ok, tenant_mounts, tenant_network, proxy_env,
|
||
prep_err) = self._docker_task_context(task)
|
||
if not ok:
|
||
return ({"executor": "docker",
|
||
"task_id": task_id,
|
||
"status": "failed", "stdout": "",
|
||
"returncode": None, "host_status": None,
|
||
"stderr": ""},
|
||
prep_err)
|
||
else:
|
||
tenant_mounts = tenant_network = proxy_env = None
|
||
try:
|
||
fork_res = self.executor.fork_task(
|
||
task, source_id,
|
||
timeout=(self.task_timeout if is_docker else None),
|
||
extra_mounts=tenant_mounts,
|
||
network=tenant_network,
|
||
extra_env=proxy_env)
|
||
except Exception as exc:
|
||
return ({"executor": self._executor_name(),
|
||
"task_id": task_id,
|
||
"status": "failed", "stdout": "",
|
||
"returncode": None, "host_status": None,
|
||
"stderr": ""},
|
||
str(exc))
|
||
if (fork_res or {}).get("status") != "completed":
|
||
err = ((fork_res or {}).get("stderr", "") or
|
||
(fork_res or {}).get("stdout", "") or "fork failed")
|
||
err = err.strip()
|
||
return (fork_res,
|
||
err[-1000:] if err else "fork failed")
|
||
return self._run_attempt(task, quota_check=quota_check)
|
||
|
||
def _claim_next_followup(self, task_id):
|
||
"""取下一条后续消息：direct 优先、队内按创建时间先来
|
||
先服务。CAS 认领（pending → running），抢不到或表不
|
||
在返回 None（表由 server init_db 建；worker 直连老库
|
||
时缺表不炸 run_one，只当没有后续消息）。"""
|
||
try:
|
||
conn = _db_connect(self.db_path)
|
||
except Exception:
|
||
return None
|
||
try:
|
||
row = conn.execute(
|
||
"SELECT * FROM task_followups"
|
||
" WHERE task_id = ? AND status = 'pending'"
|
||
" ORDER BY CASE WHEN mode = 'direct' THEN 0 ELSE 1 END,"
|
||
" created_at, id LIMIT 1",
|
||
(task_id,)).fetchone()
|
||
if row is None:
|
||
return None
|
||
cur = conn.execute(
|
||
"UPDATE task_followups SET status = 'running',"
|
||
" updated_at = ? WHERE id = ? AND status = 'pending'",
|
||
(time.time(), row["id"]))
|
||
conn.commit()
|
||
if cur.rowcount != 1:
|
||
return None
|
||
return {"id": row["id"], "body": row["body"],
|
||
"mode": row["mode"],
|
||
"sender_task_id": _row_get(row, "sender_task_id")}
|
||
except Exception:
|
||
return None
|
||
finally:
|
||
conn.close()
|
||
|
||
def _finish_followup(self, fu_id, status):
|
||
try:
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
conn.execute(
|
||
"UPDATE task_followups SET status = ?,"
|
||
" updated_at = ? WHERE id = ?",
|
||
(status, time.time(), fu_id))
|
||
conn.commit()
|
||
finally:
|
||
conn.close()
|
||
except Exception:
|
||
pass
|
||
|
||
def _restore_followup(self, fu_id):
|
||
"""取消时把跑中的后续消息放回 pending，留给重试入口。"""
|
||
try:
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
conn.execute(
|
||
"UPDATE task_followups SET status = 'pending',"
|
||
" updated_at = ? WHERE id = ? AND status = 'running'",
|
||
(time.time(), fu_id))
|
||
conn.commit()
|
||
finally:
|
||
conn.close()
|
||
except Exception:
|
||
pass
|
||
|
||
def _queue_self_followup(self, task, body):
|
||
"""给父任务自己排一条系统后续消息（标记工具报告 / 轮询快照）。
|
||
|
||
sender_task_id 置空，排空时不加“来自任务 X 的消息”前缀，
|
||
正文自带 [任务工具报告] 抬头即可区分。排空循环（run_one
|
||
的后续消息段）会在轮次边界认领它：父任务运行中直接跑下
|
||
一轮，父任务已终态时下次认领走续跑——与任务间消息的复活
|
||
语义同款，只是不经过 peer 的“不能发给自己”拒绝。"""
|
||
try:
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
_ensure_followups_table(conn)
|
||
conn.execute(
|
||
"INSERT INTO task_followups(id, task_id, project_id,"
|
||
" body, mode, status, created_by, sender_task_id,"
|
||
" created_at, updated_at)"
|
||
" VALUES(?, ?, ?, ?, 'queued', 'pending', ?, NULL,"
|
||
" ?, ?)",
|
||
(uuid.uuid4().hex, task["id"],
|
||
task["project_id"], body, task.get("created_by"),
|
||
time.time(), time.time()),
|
||
)
|
||
conn.commit()
|
||
return True
|
||
finally:
|
||
conn.close()
|
||
except Exception:
|
||
return False
|
||
|
||
def _poll_task_snapshot(self, task, poll):
|
||
"""轮询一个目标任务：每 interval 查一次库状态，直到终态、
|
||
超时或父任务被取消。等待期间不烧模型费，但占着本 worker
|
||
的槽位（与协调员等齐同款代价）。返回报告正文。"""
|
||
target_id = poll["task_id"]
|
||
interval = poll["interval_seconds"]
|
||
timeout = poll["timeout_seconds"]
|
||
chars = poll["chars"]
|
||
started = time.time()
|
||
snapshot = None
|
||
timed_out = False
|
||
cancelled = False
|
||
while True:
|
||
if self._was_cancelled(task["id"]):
|
||
cancelled = True
|
||
break
|
||
# 等候中父任务自己刷心跳：标记等待在 _run_attempt 返回后
|
||
# 才跑，那时 _running 已被清掉，run_loop 的心跳线程刷
|
||
# 不到它；不自刷的话，sweep（默认 300 秒）会在父任务
|
||
# 等到 3600 秒上限之前把它当死任务误杀。每次循环刷一次，
|
||
# 间隔至多 interval（钳制后 <=60 秒），远小于 300 秒阈值。
|
||
try:
|
||
self.heartbeat(task["id"])
|
||
except Exception:
|
||
pass
|
||
try:
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
snapshot = _marker_snapshot(conn, target_id)
|
||
finally:
|
||
conn.close()
|
||
except Exception:
|
||
snapshot = None
|
||
if snapshot is None:
|
||
return "轮询任务 %s：不存在（同项目内也查不到）。" % target_id
|
||
if snapshot["project_id"] != task["project_id"]:
|
||
return ("轮询任务 %s：拒绝，跨项目任务不可读。"
|
||
% target_id)
|
||
if target_id == task["id"]:
|
||
return "轮询任务 %s：拒绝，不能轮询自己。" % target_id
|
||
if snapshot["status"] in _MARKER_TERMINAL:
|
||
break
|
||
if timeout <= 0:
|
||
break
|
||
elapsed = time.time() - started
|
||
if elapsed >= timeout:
|
||
timed_out = True
|
||
break
|
||
time.sleep(min(interval, max(timeout - elapsed, 0.05)))
|
||
if snapshot is None:
|
||
return "轮询任务 %s：读取失败。" % target_id
|
||
return _marker_format(snapshot, chars, timed_out=timed_out,
|
||
cancelled=cancelled)
|
||
|
||
def _wait_task_targets(self, task, wait):
|
||
"""等一批目标任务：每 interval 并行查一次全部目标，直到
|
||
全部终态、超时或父任务被取消。缺失、跨项目、自等待的
|
||
目标当场记为已处理（写清原因），不让它们把整批拖到超时。
|
||
轮询/等待目标是只读的，goal 任务也可以等；取消与重试
|
||
的 goal 拒绝只在控制门禁里，不在这里。返回报告行列表。"""
|
||
target_ids = wait["task_ids"]
|
||
interval = wait["interval_seconds"]
|
||
timeout = wait["timeout_seconds"]
|
||
chars = wait["chars"]
|
||
started = time.time()
|
||
timed_out = False
|
||
cancelled = False
|
||
resolved = {} # task_id -> ("snapshot", snap) | ("note", text)
|
||
latest = {} # task_id -> 最近一次快照（超时报告用）
|
||
while True:
|
||
if self._was_cancelled(task["id"]):
|
||
cancelled = True
|
||
break
|
||
# 与 _poll_task_snapshot 同款：等候中父任务自己刷心跳，
|
||
# 否则 sweep（默认 300 秒）会误杀最长可等 3600 秒的父任务。
|
||
try:
|
||
self.heartbeat(task["id"])
|
||
except Exception:
|
||
pass
|
||
pending = [tid for tid in target_ids if tid not in resolved]
|
||
if not pending:
|
||
break
|
||
try:
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
for tid in pending:
|
||
snap = _marker_snapshot(conn, tid)
|
||
if snap is None:
|
||
resolved[tid] = (
|
||
"note",
|
||
"等待任务 %s：不存在（同项目内也查不到）。"
|
||
% tid)
|
||
elif snap["project_id"] != task["project_id"]:
|
||
resolved[tid] = (
|
||
"note",
|
||
"等待任务 %s：拒绝，跨项目任务不可读。"
|
||
% tid)
|
||
elif tid == task["id"]:
|
||
resolved[tid] = (
|
||
"note",
|
||
"等待任务 %s：拒绝，不能等待自己。" % tid)
|
||
else:
|
||
latest[tid] = snap
|
||
if snap["status"] in _MARKER_TERMINAL:
|
||
resolved[tid] = ("snapshot", snap)
|
||
finally:
|
||
conn.close()
|
||
except Exception:
|
||
pass
|
||
if all(tid in resolved for tid in target_ids):
|
||
break
|
||
if timeout <= 0:
|
||
break
|
||
elapsed = time.time() - started
|
||
if elapsed >= timeout:
|
||
timed_out = True
|
||
break
|
||
time.sleep(min(interval, max(timeout - elapsed, 0.05)))
|
||
lines = ["[等待 %d 个任务]" % len(target_ids)]
|
||
for tid in target_ids:
|
||
kind, value = resolved.get(tid, (None, None))
|
||
if kind == "note":
|
||
lines.append(value)
|
||
elif kind == "snapshot":
|
||
lines.append(_marker_format(value, chars))
|
||
else:
|
||
snap = latest.get(tid)
|
||
if snap is None:
|
||
lines.append("等待任务 %s：读取失败。" % tid)
|
||
else:
|
||
lines.append(_marker_format(
|
||
snap, chars, timed_out=timed_out,
|
||
cancelled=cancelled))
|
||
if cancelled:
|
||
lines.append("说明：等待期间父任务被取消，等待已中止。")
|
||
elif timed_out:
|
||
lines.append("说明：等待超时，仍有目标未终态，可再等一次。")
|
||
return lines
|
||
|
||
def _marker_auth(self, task, action):
|
||
"""标记控制操作的权限层，返回 (ok_or_None, reason)。
|
||
|
||
ok=True：发送方创建人有该权限；False：明确无权限；
|
||
None：权限检查本身抛异常（查不到），调用方必须收紧到
|
||
仅派生链内可用，绝不能静默放宽。"""
|
||
creator = task.get("created_by")
|
||
if not creator:
|
||
return False, "发送方无创建人，无法查权限"
|
||
try:
|
||
store = AuthStore(self.db_path)
|
||
try:
|
||
ok = store.check_permission(
|
||
creator, task["project_id"], action)
|
||
finally:
|
||
try:
|
||
store.conn.close()
|
||
except Exception:
|
||
pass
|
||
except Exception as exc:
|
||
return None, "权限检查失败：%s" % exc
|
||
return bool(ok), ""
|
||
|
||
def _deliver_marker_tools(self, task, result):
|
||
"""执行本轮 agent 输出里的标记工具块（见 marker_tools）。
|
||
|
||
只执行 completed 轮次的块；goal 任务与协调员任务不解析
|
||
（只收不发，与 peer 同款）；父任务被取消时不执行。
|
||
执行顺序：创建 → 取消 → 重试 → 轮询 → 等待——创建在前
|
||
让同一轮刚创建的任务立刻能被轮询，控制操作在快照之前
|
||
让轮询/等待读到取消或重试后的新状态。全部报告拼成一条
|
||
系统后续消息排给父任务自己，由排空循环在轮次边界跑下
|
||
一轮；拒绝只记报告与审计，不炸 run_one。"""
|
||
if not result or result.get("status") != "completed":
|
||
return
|
||
if self._was_cancelled(task["id"]):
|
||
return
|
||
if task.get("verify_command") or task.get("goal_mode"):
|
||
return
|
||
if task.get("coordinator_parts"):
|
||
return
|
||
creates = result.get("task_creates") or []
|
||
polls = result.get("task_polls") or []
|
||
waits = result.get("wait_threads") or []
|
||
cancels = result.get("task_cancels") or []
|
||
retries = result.get("task_retries") or []
|
||
if not (creates or polls or waits or cancels or retries):
|
||
return
|
||
reports = []
|
||
created = 0
|
||
rejected = 0
|
||
if creates:
|
||
quota_error = None
|
||
try:
|
||
quota_error = self._quota_exceeded(task, None)
|
||
except Exception as exc:
|
||
quota_error = "quota_check_failed: %s" % exc
|
||
if quota_error is not None:
|
||
reports.append("创建跳过：父任务配额检查未通过（%s），"
|
||
"本轮 %d 个创建块全部跳过。"
|
||
% (quota_error, len(creates)))
|
||
rejected += len(creates)
|
||
else:
|
||
try:
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
self._ensure_result_columns(conn)
|
||
generation = _marker_generation(conn, task["id"])
|
||
direct = _marker_count_direct(conn, task["id"])
|
||
now = time.time()
|
||
for create in creates:
|
||
new_depth = generation + 1
|
||
if new_depth > _MARKER_MAX_DEPTH:
|
||
reports.append(
|
||
"创建跳过：派生深度超限（上限 %d 代）。"
|
||
% _MARKER_MAX_DEPTH)
|
||
rejected += 1
|
||
continue
|
||
if direct >= _MARKER_MAX_DIRECT:
|
||
reports.append(
|
||
"创建跳过：直接派生子任务已达上限 %d 个。"
|
||
% _MARKER_MAX_DIRECT)
|
||
rejected += 1
|
||
continue
|
||
child_id = _marker_new_id()
|
||
# 派生模型/强度优先级：块里显式写 >
|
||
# 父任务的派生默认 child_model /
|
||
# child_effort > 父任务自身的
|
||
# model / effort > NULL（认领时走
|
||
# worker 默认）。子任务行上同时记下
|
||
# 父任务的派生默认，隔代派生不丢。
|
||
child_model = (create["model"]
|
||
or task.get("child_model")
|
||
or task.get("model"))
|
||
child_effort = (create["effort"]
|
||
or task.get("child_effort")
|
||
or task.get("effort"))
|
||
conn.execute(
|
||
"INSERT INTO tasks(id, project_id, prompt,"
|
||
" status, idempotency_key, depends_on,"
|
||
" priority, created_by, created_at,"
|
||
" updated_at, last_heartbeat, executor,"
|
||
" model, effort, environment_id,"
|
||
" parent_task_id, coordinator_part,"
|
||
" child_model, child_effort)"
|
||
" VALUES(?, ?, ?, 'queued', NULL, '[]',"
|
||
" ?, ?, ?, ?, NULL, ?, ?, ?, ?, ?, NULL,"
|
||
" ?, ?)",
|
||
(child_id, task["project_id"],
|
||
create["prompt"],
|
||
task.get("priority", 0) or 0,
|
||
task.get("created_by"), now, now,
|
||
create["executor"]
|
||
or task.get("executor"),
|
||
child_model,
|
||
child_effort,
|
||
task.get("environment_id"),
|
||
task["id"],
|
||
task.get("child_model"),
|
||
task.get("child_effort")),
|
||
)
|
||
direct += 1
|
||
created += 1
|
||
label = create["name"] or "未命名"
|
||
model_note = ""
|
||
if child_model or child_effort:
|
||
model_note = "，模型 %s，强度 %s" % (
|
||
child_model or "默认",
|
||
child_effort or "默认")
|
||
reports.append(
|
||
"已创建任务 %s（%s），状态 queued%s，"
|
||
"可用 task-poll 按该 id 轮询。"
|
||
% (child_id, label, model_note))
|
||
conn.commit()
|
||
finally:
|
||
conn.close()
|
||
except Exception as exc:
|
||
reports.append("创建失败：%s" % exc)
|
||
rejected += len(creates) - created
|
||
for kind, blocks, action in (("取消", cancels, "create_task"),
|
||
("重试", retries, "approve_merge")):
|
||
for block in blocks:
|
||
target_id = block["task_id"]
|
||
try:
|
||
target = self.get_task(target_id)
|
||
except Exception as exc:
|
||
target = None
|
||
reports.append("%s任务 %s：读取失败（%s），已跳过。"
|
||
% (kind, target_id, exc))
|
||
rejected += 1
|
||
continue
|
||
relation = "other"
|
||
if target is not None:
|
||
try:
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
relation = _marker_ancestry(
|
||
conn, task, target)
|
||
finally:
|
||
conn.close()
|
||
except Exception:
|
||
relation = "other"
|
||
gate_kind = "cancel" if kind == "取消" else "retry"
|
||
ok, reason = _marker_gate(target, task, relation,
|
||
gate_kind)
|
||
if not ok:
|
||
reports.append("%s任务 %s：拒绝（%s）。"
|
||
% (kind, target_id, reason))
|
||
rejected += 1
|
||
continue
|
||
auth_ok, auth_reason = self._marker_auth(task, action)
|
||
if auth_ok is False:
|
||
reports.append("%s任务 %s：拒绝（发送方创建人无"
|
||
" %s 权限）。"
|
||
% (kind, target_id, action))
|
||
rejected += 1
|
||
continue
|
||
if auth_ok is None and relation != "descendant":
|
||
reports.append("%s任务 %s：拒绝（%s，仅派生链内"
|
||
"可用）。"
|
||
% (kind, target_id, auth_reason))
|
||
rejected += 1
|
||
continue
|
||
try:
|
||
if kind == "取消":
|
||
done = self.cancel(target_id)
|
||
if done:
|
||
reports.append("已取消任务 %s，状态为"
|
||
" interrupted。" % target_id)
|
||
else:
|
||
reports.append("取消任务 %s：未执行（目标"
|
||
"状态已变或不存在）。"
|
||
% target_id)
|
||
rejected += 1
|
||
else:
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
cur = conn.execute(
|
||
"UPDATE tasks SET status = 'queued',"
|
||
" result_text = NULL, error_text = NULL,"
|
||
" actual_model = NULL, updated_at = ?"
|
||
" WHERE id = ? AND status IN"
|
||
" ('failed', 'interrupted',"
|
||
" 'merge-failed')",
|
||
(time.time(), target_id))
|
||
conn.commit()
|
||
done = cur.rowcount == 1
|
||
finally:
|
||
conn.close()
|
||
if done:
|
||
self._audit(task["project_id"],
|
||
"worker_marker_retry",
|
||
target_id)
|
||
reports.append("已重试任务 %s，状态回"
|
||
" queued，上一轮结果已清空。"
|
||
% target_id)
|
||
else:
|
||
reports.append("重试任务 %s：未执行（目标"
|
||
"状态已变）。" % target_id)
|
||
rejected += 1
|
||
except Exception as exc:
|
||
reports.append("%s任务 %s：执行失败（%s）。"
|
||
% (kind, target_id, exc))
|
||
rejected += 1
|
||
for poll in polls:
|
||
try:
|
||
reports.append(self._poll_task_snapshot(task, poll))
|
||
except Exception as exc:
|
||
reports.append("轮询任务 %s 失败：%s"
|
||
% (poll["task_id"], exc))
|
||
for wait in waits:
|
||
try:
|
||
reports.extend(self._wait_task_targets(task, wait))
|
||
except Exception as exc:
|
||
reports.append("等待任务失败：%s" % exc)
|
||
if reports:
|
||
body = _marker_header() + "\n" + "\n".join(reports)
|
||
self._queue_self_followup(task, body)
|
||
detail = ("%s: spawned=%d rejected=%d polls=%d waits=%d "
|
||
"cancels=%d retries=%d"
|
||
% (task["id"], created, rejected, len(polls),
|
||
len(waits), len(cancels), len(retries)))
|
||
self._audit(task["project_id"], "worker_marker_tools", detail)
|
||
|
||
def _deliver_peer_messages(self, task, result):
|
||
"""投递本轮 agent 输出里的互发消息标记块（见 peer_messages）。
|
||
|
||
只投递 completed 轮次的块：失败 / 超时 / 取消 / 配额中断
|
||
的轮次不投递；任务执行期间被取消时也不投递。投递本身
|
||
不中断任何任务：对方运行中/排队中排成后续消息（轮次边
|
||
界执行），对方终态则续跑复活；拒绝（自己、跨项目、
|
||
goal 目标、已合并、复活位被占等）只记审计，不炸 run_one。
|
||
投递不查发送方配额（发送本身不花钱），目标任务执行时
|
||
的配额门照常生效——互发消息不构成配额绕过。"""
|
||
if not result or result.get("status") != "completed":
|
||
return
|
||
if self._was_cancelled(task["id"]):
|
||
return
|
||
messages = result.get("task_messages") or []
|
||
if not messages:
|
||
return
|
||
delivered = 0
|
||
rejected = []
|
||
for msg in messages:
|
||
try:
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
status, detail = _deliver_peer_message(
|
||
conn, task["id"], msg["to_task_id"],
|
||
msg["body"], msg["mode"])
|
||
finally:
|
||
conn.close()
|
||
except Exception as exc:
|
||
status, detail = "rejected", "delivery error: %s" % exc
|
||
if status in ("delivered", "revived"):
|
||
delivered += 1
|
||
else:
|
||
rejected.append("%s: %s" % (msg["to_task_id"], detail))
|
||
detail = "%s: delivered=%d rejected=%d" % (
|
||
task["id"], delivered, len(rejected))
|
||
if rejected:
|
||
detail += " (" + "; ".join(rejected) + ")"
|
||
self._audit(task["project_id"], "worker_peer_messages", detail)
|
||
|
||
def _docker_call_with_quota_poll(self, task, fn, quota_check=None):
|
||
"""Docker 同步调用包一层配额轮询（G8），与 popen 路径同款。
|
||
|
||
fn 在后台线程里跑（DockerExecutor 的调用本身是阻塞的
|
||
docker run，无法像 popen 那样在主循环里轮询）；主循环
|
||
每 quota_poll_interval_seconds（缺省 30 秒）调一次
|
||
quota_check，确认超限时按任务容器名真杀容器，fn 返回
|
||
后把结果状态改写成内部标记 'quota_exceeded'，调用方
|
||
（run_one）据此走统一的配额中断收尾：状态 interrupted、
|
||
已花成本照记、续跑消息保留。quota_check 为 None 时
|
||
直接调 fn，不起线程。quota_check 本身抛异常时按
|
||
fail-open 处理（只记 warning，不杀容器），只有确认
|
||
超限才杀；取消时 cancel() 已按容器名杀过容器，线程
|
||
随之结束，轮询循环不再多杀一次。fn 抛异常时在主线程
|
||
原样重抛，由 _run_attempt 外层的 except 写成 failed，
|
||
与不包轮询时的行为一致。注意：轮询查的是已记账用量
|
||
（与 popen 路径同款），在跑回合的 token 只在回合结束
|
||
才回传，所以掐断粒度仍是轮询间隔，不是 token 级。
|
||
"""
|
||
if quota_check is None:
|
||
return fn()
|
||
box = {}
|
||
|
||
def _target():
|
||
try:
|
||
box["result"] = fn()
|
||
except Exception as exc: # noqa: BLE001 -- 主线程重抛
|
||
box["error"] = exc
|
||
|
||
thread = threading.Thread(target=_target, daemon=True)
|
||
thread.start()
|
||
interval = getattr(self, "quota_poll_interval_seconds", 30)
|
||
if interval is None:
|
||
interval = 30
|
||
last_check = 0.0
|
||
quota_hit = False
|
||
while thread.is_alive():
|
||
if task["id"] in self._cancel_requested or self._loop_stop:
|
||
break
|
||
now = time.time()
|
||
if now - last_check >= interval:
|
||
last_check = now
|
||
try:
|
||
over = bool(quota_check())
|
||
except Exception as exc:
|
||
try:
|
||
self.logger.warning(
|
||
"docker quota poll failed: %s", exc)
|
||
except Exception:
|
||
pass
|
||
over = False
|
||
if over:
|
||
quota_hit = True
|
||
container = None
|
||
try:
|
||
container = self.executor.container_name(task)
|
||
except Exception:
|
||
container = None
|
||
if container is not None and hasattr(
|
||
self.executor, "kill_container"):
|
||
try:
|
||
self.executor.kill_container(container)
|
||
except Exception:
|
||
pass
|
||
break
|
||
thread.join(timeout=0.05)
|
||
thread.join()
|
||
if "error" in box:
|
||
raise box["error"]
|
||
result = box.get("result")
|
||
if quota_hit and isinstance(result, dict):
|
||
result = dict(result, status="quota_exceeded")
|
||
return result
|
||
|
||
def _run_attempt(self, task, quota_check=None):
|
||
"""单次执行尝试（不含认领、key 门、重试、成本与最终审计）。
|
||
|
||
返回 (result, error)：result 是执行器结果 dict，error 是
|
||
runner 异常时的文本（无异常时 None）。Docker 路径下把
|
||
worker 级 task_timeout 与执行器级 docker_timeout 取较小
|
||
值作为本次 docker run 超时（docker_timeout 未设时取
|
||
task_timeout）；Local popen 路径下把 task_timeout 作为
|
||
轮询截止时间；Local 同步路径无法中途抢占，超时只能靠
|
||
执行器自身 timeout（见模块 docstring 的诚实边界）。
|
||
调用方负责：取消优先、超时转 failed、重试、成本与审计。
|
||
"""
|
||
task_id = task["id"]
|
||
result = None
|
||
error = None
|
||
# goal 模式：goal_mode=1（新语义，prompt 即目标）或带
|
||
# verify_command（旧语义）时走 host.py goal 循环，不走
|
||
# 普通 dispatch。goal 循环内部已多轮重试，worker 级
|
||
# 的已存在续跑（run）对它没有意义（goal 没有 run 语义）。
|
||
goal_mode = bool(task.get("goal_mode")
|
||
or task.get("verify_command"))
|
||
# 附件 seed：dispatch/goal 首轮才用；run 续跑不传（文件已
|
||
# 在 worktree 里）。cloud 执行器没有 worktree 可 seed，
|
||
# 创建时本应已拒绝，这里是兜底防线（直接失败，不静默跑）。
|
||
seed_host_dir = self._task_seed_dir(task)
|
||
if seed_host_dir and isinstance(self.executor, CloudExecutor):
|
||
return ({"executor": "cloud", "task_id": task_id,
|
||
"status": "failed", "stdout": "",
|
||
"returncode": None, "host_status": None,
|
||
"stderr": ""},
|
||
"cloud 执行器不支持附件")
|
||
# Docker 同步路径：dispatch 阻塞在 docker run 里时，取消
|
||
# 线程靠容器名杀容器（见 cancel()）。容器名按任务派生，
|
||
# 与执行器本身无关，所以这里只记名不碰执行器状态。
|
||
if hasattr(self.executor, "container_name"):
|
||
try:
|
||
self._running_set(
|
||
task_id,
|
||
container=self.executor.container_name(task))
|
||
except Exception:
|
||
pass
|
||
# 长任务不限时：任务带 no_time_limit=1 时，本次 worker
|
||
# 级与 Docker 级超时一律按 None 算（不限时），不改全局
|
||
# task_timeout（其它任务仍走默认 1800 秒）；host 单轮
|
||
# turn_timeout（默认 600 秒）不在 worker 侧传参，仍是
|
||
# 单轮护栏——不限的是任务总墙钟与总轮次，不是单轮无限。
|
||
_no_time_limit = bool(task.get("no_time_limit"))
|
||
_effective_task_timeout = None if _no_time_limit \
|
||
else self.task_timeout
|
||
# Docker 超时取较小值：两个超时都不应静默延长另一个。
|
||
# 注意绝不能改共享的 executor.timeout（并发下两个任务会
|
||
# 互相覆盖对方的超时）；超时只作为本次调用的参数传下去。
|
||
effective_docker_timeout = None
|
||
is_docker = isinstance(self.executor, DockerExecutor)
|
||
if is_docker and _effective_task_timeout is not None:
|
||
if self.executor.timeout is None:
|
||
effective_docker_timeout = _effective_task_timeout
|
||
else:
|
||
try:
|
||
effective_docker_timeout = min(
|
||
float(_effective_task_timeout),
|
||
float(self.executor.timeout))
|
||
except (TypeError, ValueError):
|
||
effective_docker_timeout = _effective_task_timeout
|
||
# P0 安全批后：Docker 同步路径起取消看门狗线程
|
||
# （阻塞在 docker run 里时，库状态变 interrupted 就
|
||
# 按容器名真杀；非 Docker 路径不启动，避免空线程）。
|
||
_watchdog_stop = None
|
||
_watchdog_thread = None
|
||
_quota_watchdog_thread = None
|
||
# P0 三件其一：goal 任务起断点镜像线程（任何执行器）；
|
||
# 非 goal 任务没有断点语义，不起。终局镜像在 finally 补。
|
||
_mirror_stop = None
|
||
_mirror_thread = None
|
||
if goal_mode:
|
||
try:
|
||
_mirror_stop = threading.Event()
|
||
_mirror_thread = threading.Thread(
|
||
target=self._checkpoint_mirror_loop,
|
||
args=(task, _mirror_stop), daemon=True)
|
||
_mirror_thread.start()
|
||
except Exception:
|
||
_mirror_stop = None
|
||
_mirror_thread = None
|
||
if is_docker and hasattr(self.executor, "kill_container"):
|
||
try:
|
||
_container_name = self.executor.container_name(task)
|
||
except Exception:
|
||
_container_name = None
|
||
if _container_name:
|
||
_watchdog_stop = threading.Event()
|
||
_watchdog_thread = threading.Thread(
|
||
target=self._docker_cancel_watchdog,
|
||
args=(task_id, _container_name, _watchdog_stop),
|
||
daemon=True)
|
||
_watchdog_thread.start()
|
||
# P1 配额看门狗：只在同步 Docker 路径起（进程
|
||
# 模式的 _execute_via_process 已每 30 秒轮询
|
||
# quota_check，不重复起）。quota_check 为 None
|
||
# 时不起（无配额检查可做，不空转线程）。
|
||
if quota_check is not None and not self._use_popen():
|
||
_quota_watchdog_thread = threading.Thread(
|
||
target=self._docker_quota_watchdog,
|
||
args=(task_id, _container_name,
|
||
_watchdog_stop, quota_check),
|
||
daemon=True)
|
||
_quota_watchdog_thread.start()
|
||
# 启动前分段日志（2026-10-29 任务 A）：认领后 61 秒中断，
|
||
# 但阻塞的 docker run 中途没有任何标记、容器 --rm 后输出
|
||
# 也没了，事后完全无法定位卡在哪一段。派发开始/结束进
|
||
# 审计（PG 落盘，容器没了也丢不了），下次卡死时最后一个
|
||
# 标记直接说明卡在哪一段；kind/容器名/超时不含 prompt/key。
|
||
_exec_kind = ("goal-resume-breakpoint"
|
||
if task.get("_goal_resume_breakpoint")
|
||
else "goal-resume" if task.get("_resume")
|
||
else "goal" if goal_mode else "dispatch")
|
||
_exec_container = None
|
||
if hasattr(self.executor, "container_name"):
|
||
try:
|
||
_exec_container = self.executor.container_name(task)
|
||
except Exception:
|
||
_exec_container = None
|
||
_exec_t0 = time.time()
|
||
_exec_error = None
|
||
_exec_detail = "task_id=%s kind=%s executor=%s timeout=%s" % (
|
||
task_id, _exec_kind, self._executor_name(),
|
||
"None" if _effective_task_timeout is None
|
||
else _effective_task_timeout)
|
||
if _exec_container:
|
||
_exec_detail += " container=%s" % _exec_container
|
||
self._audit(task.get("project_id"), "worker_exec_start",
|
||
_exec_detail)
|
||
log_event(self.logger, "worker_exec_start", task_id=task_id,
|
||
message="worker exec start: %s" % _exec_detail)
|
||
try:
|
||
gateway_provider, gateway_env = self._ensure_gateway(task)
|
||
if self._use_popen():
|
||
if goal_mode and task.get("_goal_resume_breakpoint"):
|
||
# G10 断点：无 --goal-text，不开新循环。
|
||
popen_kind = "goal-resume-breakpoint"
|
||
elif goal_mode and task.get("_resume"):
|
||
popen_kind = "goal-resume"
|
||
elif goal_mode:
|
||
popen_kind = "goal"
|
||
else:
|
||
popen_kind = "dispatch"
|
||
result = self._execute_via_process(
|
||
task, kind=popen_kind,
|
||
deadline_seconds=_effective_task_timeout,
|
||
seed_host_dir=seed_host_dir,
|
||
quota_check=quota_check)
|
||
combined = (result.get("stdout", "") or "") + \
|
||
(result.get("stderr", "") or "")
|
||
if not goal_mode and result["status"] not in (
|
||
"completed", "cancelled", "timeout",
|
||
"quota_exceeded") and \
|
||
_ALREADY_EXISTS_MARKER in combined:
|
||
result = self._execute_via_process(
|
||
task, kind="run",
|
||
deadline_seconds=_effective_task_timeout,
|
||
quota_check=quota_check)
|
||
else:
|
||
if is_docker:
|
||
(ok, tenant_mounts, tenant_network, proxy_env,
|
||
prep_err) = self._docker_task_context(task)
|
||
if not ok:
|
||
# 提前返回会跳过下面的 finally（清容器
|
||
# 标记），这里先手动清掉，否则 _running
|
||
# 里永远残留该任务，心跳与退出清理
|
||
# 都会把它当在跑任务。
|
||
self._running_clear(task_id)
|
||
return ({"executor": "docker",
|
||
"task_id": task_id,
|
||
"status": "failed", "stdout": "",
|
||
"returncode": None, "host_status": None,
|
||
"stderr": ""},
|
||
prep_err)
|
||
if goal_mode:
|
||
if task.get("_goal_resume_breakpoint"):
|
||
# G10 断点：Docker 路用无 --goal-text 的
|
||
# goal-resume-breakpoint 命令。
|
||
result = self._docker_call_with_quota_poll(
|
||
task,
|
||
lambda: self.executor.goal_resume_breakpoint_task(
|
||
task,
|
||
timeout=effective_docker_timeout,
|
||
extra_mounts=tenant_mounts,
|
||
network=tenant_network,
|
||
extra_env=_merge_env(proxy_env,
|
||
gateway_env),
|
||
model_provider=gateway_provider),
|
||
quota_check)
|
||
elif task.get("_resume"):
|
||
# P1 断点续跑传 None（不传 --goal-text，
|
||
# host 沿用原 goal+verify）；开新目标
|
||
# 才传本轮 prompt 当新 goal_text。
|
||
_resume_text = None if task.get(
|
||
"_resume_kind") == "breakpoint" \
|
||
else task["prompt"]
|
||
result = self._docker_call_with_quota_poll(
|
||
task,
|
||
lambda: self.executor.goal_resume_task(
|
||
task, _resume_text,
|
||
timeout=effective_docker_timeout,
|
||
extra_mounts=tenant_mounts,
|
||
network=tenant_network,
|
||
extra_env=_merge_env(proxy_env,
|
||
gateway_env),
|
||
model_provider=gateway_provider),
|
||
quota_check)
|
||
else:
|
||
result = self._docker_call_with_quota_poll(
|
||
task,
|
||
lambda: self.executor.goal_task(
|
||
task, timeout=effective_docker_timeout,
|
||
extra_mounts=tenant_mounts,
|
||
network=tenant_network,
|
||
extra_env=_merge_env(proxy_env,
|
||
gateway_env),
|
||
seed_host_dir=seed_host_dir,
|
||
model_provider=gateway_provider),
|
||
quota_check)
|
||
else:
|
||
result = self._docker_call_with_quota_poll(
|
||
task,
|
||
lambda: self.executor.dispatch_task(
|
||
task, timeout=effective_docker_timeout,
|
||
extra_mounts=tenant_mounts,
|
||
network=tenant_network,
|
||
extra_env=_merge_env(proxy_env,
|
||
gateway_env),
|
||
seed_host_dir=seed_host_dir,
|
||
model_provider=gateway_provider),
|
||
quota_check)
|
||
else:
|
||
# seed 参数只传给 LocalExecutor：CloudExecutor
|
||
# 基类与测试假执行器的 goal_task/dispatch_task
|
||
# 只有 (task) 一个参数，多传 seed_dir 会直接
|
||
# TypeError（2026-10-27 实测炸掉 3 个旧测试）；
|
||
# Docker 路径上面已有 is_docker 守卫，不走这里。
|
||
# 网关任务在非 Local 执行器上无法注入 key 与
|
||
# provider，必须 loud fail，不静默跑错网关。
|
||
if gateway_provider and not isinstance(
|
||
self.executor, LocalExecutor):
|
||
raise RuntimeError(
|
||
"sub2api 网关不支持该执行器：%s"
|
||
% type(self.executor).__name__)
|
||
if goal_mode:
|
||
|
||
if task.get("_goal_resume_breakpoint") and isinstance(
|
||
self.executor, LocalExecutor):
|
||
result = self.executor.goal_resume_breakpoint_task(
|
||
task, model_provider=gateway_provider)
|
||
elif task.get("_goal_resume_breakpoint"):
|
||
# 假执行器/CloudExecutor 没有断点方法时
|
||
# 绝不回落成 goal（会撞已存在）或
|
||
# goal-resume --goal-text（会开新循环）：
|
||
# 明确失败，等支持断点的执行器再跑。
|
||
result = {"executor": self._executor_name(),
|
||
"task_id": task_id,
|
||
"status": "failed", "stdout": "",
|
||
"returncode": None,
|
||
"host_status": None, "stderr": "",
|
||
"goal_met": None,
|
||
"goal_error": "executor 不支持"
|
||
" goal 断点续跑"}
|
||
error = "executor 不支持 goal 断点续跑"
|
||
elif task.get("_resume") and isinstance(
|
||
self.executor, LocalExecutor):
|
||
_resume_text = None if task.get(
|
||
"_resume_kind") == "breakpoint" \
|
||
else task["prompt"]
|
||
result = self.executor.goal_resume_task(
|
||
task, _resume_text,
|
||
extra_env=gateway_env,
|
||
model_provider=gateway_provider)
|
||
elif isinstance(self.executor, LocalExecutor):
|
||
result = self.executor.goal_task(
|
||
task, seed_dir=seed_host_dir,
|
||
extra_env=gateway_env,
|
||
model_provider=gateway_provider)
|
||
else:
|
||
result = self.executor.goal_task(task)
|
||
else:
|
||
if isinstance(self.executor, LocalExecutor):
|
||
result = self.executor.dispatch_task(
|
||
task, seed_dir=seed_host_dir,
|
||
extra_env=gateway_env,
|
||
model_provider=gateway_provider)
|
||
else:
|
||
result = self.executor.dispatch_task(task)
|
||
combined = (result.get("stdout", "") or "") + \
|
||
(result.get("stderr", "") or "")
|
||
if not goal_mode and result["status"] not in (
|
||
"completed", "quota_exceeded") and \
|
||
_ALREADY_EXISTS_MARKER in combined:
|
||
if is_docker:
|
||
result = self._docker_call_with_quota_poll(
|
||
task,
|
||
lambda: self.executor.run_task(
|
||
task, timeout=effective_docker_timeout,
|
||
extra_mounts=tenant_mounts
|
||
if self.tenant_isolation else None,
|
||
network=tenant_network
|
||
if self.tenant_isolation else None,
|
||
extra_env=_merge_env(
|
||
proxy_env if self.egress_proxy_port
|
||
else None, gateway_env),
|
||
model_provider=gateway_provider),
|
||
quota_check)
|
||
else:
|
||
if gateway_provider and not isinstance(
|
||
self.executor, LocalExecutor):
|
||
raise RuntimeError(
|
||
"sub2api 网关不支持该执行器：%s"
|
||
% type(self.executor).__name__)
|
||
result = self.executor.run_task(
|
||
task, extra_env=gateway_env,
|
||
model_provider=gateway_provider) \
|
||
if isinstance(self.executor, LocalExecutor) \
|
||
else self.executor.run_task(task)
|
||
except Exception as exc: # runner 异常也如实写回 failed
|
||
result = {"executor": "local", "task_id": task_id,
|
||
"status": "failed", "stdout": "", "returncode": None,
|
||
"host_status": None, "stderr": ""}
|
||
error = str(exc)
|
||
_exec_error = exc
|
||
finally:
|
||
if _watchdog_stop is not None:
|
||
_watchdog_stop.set()
|
||
if _watchdog_thread is not None:
|
||
_watchdog_thread.join(timeout=2)
|
||
if _quota_watchdog_thread is not None:
|
||
_quota_watchdog_thread.join(timeout=2)
|
||
if _mirror_stop is not None:
|
||
_mirror_stop.set()
|
||
if _mirror_thread is not None:
|
||
_mirror_thread.join(timeout=2)
|
||
# 终局镜像：执行结束时的断点（最后一轮）必须落库，
|
||
# 否则中断后库断点比实际少一轮；失败不影响结果写回。
|
||
try:
|
||
self._mirror_checkpoint_once(task)
|
||
except Exception:
|
||
pass
|
||
# 同步路径执行结束：清掉取消用的容器标记（取消路径里
|
||
# 若已清过，这里再清一次无害）。进程模式的 proc 由
|
||
# _execute_via_process 自己的 finally 清，这里只清
|
||
# 容器项：_running_clear 整个删掉该任务项会连 proc
|
||
# 也一起丢，所以这里只在 proc 为空时清容器标记——
|
||
# 实际做法是：如果该任务项里没有 proc，就整项清除；
|
||
# 有 proc（进程模式）时不动它，等它自己的 finally。
|
||
entry = self._running_snapshot().get(task_id) or {}
|
||
if entry.get("proc") is None:
|
||
self._running_clear(task_id)
|
||
# 启动前分段日志（续）：派发结束标记与失败输出尾巴。结束
|
||
# 标记与开始标记配对：只有开始没有结束 = 卡在执行器调用
|
||
# 里；失败/中断时的容器输出尾巴记进 worker 日志——任务 A
|
||
# 当时这部分输出随 --rm 消失，是事后无法定位的另一半。
|
||
_exec_elapsed_ms = int((time.time() - _exec_t0) * 1000)
|
||
if _exec_error is not None:
|
||
self._audit(task.get("project_id"), "worker_exec_error",
|
||
"task_id=%s elapsed_ms=%d error=%s"
|
||
% (task_id, _exec_elapsed_ms,
|
||
_exec_error.__class__.__name__))
|
||
log_event(self.logger, "worker_exec_error", task_id=task_id,
|
||
message="worker exec error: elapsed_ms=%d error=%s"
|
||
% (_exec_elapsed_ms,
|
||
_exec_error.__class__.__name__))
|
||
else:
|
||
self._audit(task.get("project_id"), "worker_exec_done",
|
||
"task_id=%s elapsed_ms=%d status=%s returncode=%s"
|
||
% (task_id, _exec_elapsed_ms, result.get("status"),
|
||
result.get("returncode")))
|
||
log_event(self.logger, "worker_exec_done", task_id=task_id,
|
||
status=result.get("status"),
|
||
message="worker exec done: elapsed_ms=%d status=%s"
|
||
% (_exec_elapsed_ms, result.get("status")))
|
||
_exec_out = result.get("stdout", "") or ""
|
||
if result.get("status") in ("failed", "interrupted") and _exec_out:
|
||
self.logger.info(
|
||
"worker_exec_stdout_tail task_id=%s tail=%s",
|
||
task_id, _exec_out[-2000:])
|
||
# Docker 超时标记：执行器已在超时时 kill+rm 容器并返回
|
||
# failed（stdout 含 timeout），这里把状态统一成 timeout，
|
||
# 调用方据此置 failed 并记 worker_timeout 审计。
|
||
if result is not None and result.get("status") == "failed" and \
|
||
"docker run timeout" in (result.get("stdout", "") or ""):
|
||
result = dict(result, status="timeout")
|
||
# P1 配额看门狗命中改写：同步路径被看门狗杀容器后，
|
||
# 执行器只返回普通 failed，run_one 无法区分“模型失败”
|
||
# 与“配额杀容器”——命中时改写成 quota_exceeded，run_one
|
||
# 走统一配额中断收尾（interrupted + 配额原因 + 成本照记，
|
||
# 不重试、不续跑下一轮）。改写前先取走命中标记，避免
|
||
# 旧命中污染同一任务的后续尝试；超时标记优先于配额
|
||
# 改写的情况不存在（看门狗杀容器时执行器不会同时报超时）。
|
||
if result is not None and self._take_quota_hit(task_id):
|
||
result = dict(result, status="quota_exceeded")
|
||
if error is None:
|
||
error = ("quota_exceeded: Docker 配额看门狗已杀容器"
|
||
"（配额中断，下一轮不再续跑）")
|
||
return result, error
|
||
|
||
def _attempt_error(self, result, error, token):
|
||
"""一次尝试失败的可读原因（供 fallback 两次原因拼接用）。
|
||
|
||
超时用统一的超时文本；runner 异常用异常文本；同步路径的
|
||
stdout 是 host.py 的 JSON 时取它的 text 字段（比整段 JSON
|
||
可读），执行器已提取出 text 时直接用该文本；最后退到
|
||
host_status token 或通用文本。绝不返回
|
||
任何密钥（stdout 里本就不应含密钥，text 字段同样不含）。
|
||
"""
|
||
if result is not None and result.get("status") == "timeout":
|
||
return "task timeout after %s seconds" % self.task_timeout
|
||
if error:
|
||
return error
|
||
stdout = (result or {}).get("stdout", "") or ""
|
||
try:
|
||
parsed = json.loads(stdout)
|
||
except (TypeError, ValueError):
|
||
parsed = None
|
||
if isinstance(parsed, dict) and parsed.get("text"):
|
||
return str(parsed["text"])
|
||
# dispatch 输出现在带 error 字段（host.py 从 thread_history
|
||
# 库里捞回的失败原因，如 404）；text 为空时优先用它，
|
||
# 否则两次失败的拼接里只剩 host_status=404 这种代号。
|
||
if isinstance(parsed, dict) and parsed.get("error"):
|
||
err = parsed["error"]
|
||
if isinstance(err, dict):
|
||
err = err.get("message") or str(err)
|
||
return str(err)[:1000]
|
||
# host.py 崩溃时 stdout 常为空、真正的报错在 stderr（例如
|
||
# “缺少 --repo”、Read-only file system）；只用 stdout 时
|
||
# 最终错误会被写成通用的 host execution failed，排障无从
|
||
# 下手，所以 stdout 为空时取 stderr 尾部（实际报错行在尾）。
|
||
stderr = (result or {}).get("stderr", "") or ""
|
||
if not stdout.strip() and stderr.strip():
|
||
return stderr.strip()[-1000:]
|
||
# 执行器现在返回的是解析出的 text（不再是整段 JSON）：
|
||
# JSON 解析失败且 stdout 非空时，stdout 本身就是报错
|
||
# 文本，绝不能丢掉它退到 host_status=404 这种代号
|
||
# （2026-10-30 全量因此丢了 "primary 404 boom"）。
|
||
if stdout.strip():
|
||
return stdout.strip()[:1000]
|
||
if token is not None:
|
||
return "host_status=%s" % token
|
||
return stdout.strip() or "host execution failed"
|
||
|
||
def _fallback_token(self, result, error):
|
||
"""一次失败尝试的可 fallback 信号。
|
||
|
||
host_status 的真实含义是 host.py 回合状态（completed /
|
||
failed / timeout / interrupted），不是 HTTP 状态码——
|
||
真实世界里 host.py 从不返回 status=\"404\"，只有单测里的
|
||
假 runner 才会这么写。所以这里不能把 host_status 原样
|
||
当 token：host_status 本身已是可 fallback 码（404/429/
|
||
5xx/timeout，含历史假 runner 形态）时直接返回；否则
|
||
（最常见：host_status=\"failed\"）从回合 JSON 的 text/error
|
||
字段、stdout、stderr、runner 异常里找显式 HTTP 状态码
|
||
或“model not found”类文本（按 404 计），找到且可
|
||
fallback 才返回；找不到显式信号时返回 None，绝不凭
|
||
“host execution failed”这类通用文本猜测，否则任何执行
|
||
器故障都会被误当成模型故障换模型烧额度。thread/start
|
||
崩溃（host.py 非 0 退出、不打印 JSON）时 host_status
|
||
为 None，同样走这条提取路径。
|
||
"""
|
||
host_status = (result or {}).get("host_status")
|
||
if host_status is not None and is_fallbackable(host_status):
|
||
return host_status
|
||
combined_parts = [
|
||
(result or {}).get("stdout", "") or "",
|
||
(result or {}).get("stderr", "") or "",
|
||
error or "",
|
||
]
|
||
combined = " ".join(combined_parts)
|
||
if not combined:
|
||
return None
|
||
lowered = combined.lower()
|
||
if "model not found" in lowered or "unknown model" in lowered \
|
||
or "no such model" in lowered:
|
||
return "404"
|
||
match = re.search(
|
||
r"(?:http/|status[\"':\s]{0,10}|code[\"':\s]{0,10}|error[\"':\s]{0,10})"
|
||
r"(404|429|500|502|503|504)\b",
|
||
combined, re.IGNORECASE)
|
||
if match:
|
||
return match.group(1)
|
||
match = re.search(r"\b(404|429|500|502|503|504)\b", combined)
|
||
if match:
|
||
return match.group(1)
|
||
return None
|
||
|
||
# -- 协调员 ---------------------------------------------------------
|
||
def _coord_sync_children(self, task, parts):
|
||
"""按分工建/复用子任务，返回 [{name, id}]。
|
||
|
||
已有子任务按 coordinator_part（分工名）匹配：已完成的直接
|
||
复用；失败/中断的回 queued 重跑（旧输出清掉）；未终态的
|
||
不动（正被别的 worker 跑或排队）；缺失的才新建。重跑协调
|
||
员因此不会重复建子任务。子任务是普通任务：执行器过滤、
|
||
配额门、key 门都在子任务认领时照常生效。
|
||
"""
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
_coord_ensure_columns(conn)
|
||
try:
|
||
conn.commit()
|
||
except Exception:
|
||
pass
|
||
rows = conn.execute(
|
||
"SELECT id, coordinator_part, status FROM tasks"
|
||
" WHERE parent_task_id = ?",
|
||
(task["id"],)).fetchall()
|
||
finally:
|
||
conn.close()
|
||
existing = {}
|
||
for row in rows:
|
||
try:
|
||
existing[row["coordinator_part"]] = {
|
||
"id": row["id"], "status": row["status"]}
|
||
except (KeyError, IndexError, TypeError):
|
||
continue
|
||
now = time.time()
|
||
children = []
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
for part in parts:
|
||
name = part["name"]
|
||
prev = existing.get(name)
|
||
if prev is not None:
|
||
if prev["status"] in TERMINAL_STATUSES and \
|
||
prev["status"] != "completed":
|
||
conn.execute(
|
||
"UPDATE tasks SET status = 'queued',"
|
||
" result_text = NULL, error_text = NULL,"
|
||
" updated_at = ? WHERE id = ?",
|
||
(now, prev["id"]))
|
||
self._audit(
|
||
task["project_id"],
|
||
"coordinator_child_requeued",
|
||
"%s -> %s" % (name, prev["id"]))
|
||
children.append({"name": name, "id": prev["id"]})
|
||
continue
|
||
# 自动化入闸（创建时门）：子任务认领时虽有配额门，
|
||
# 但不建任务才是扇出不烧钱的硬保证。父任务创建人
|
||
# 超配额时跳过建这个子任务、记审计，父任务本身不
|
||
# 失败（协调员按缺失子任务的既有语义等超时）；同
|
||
# 一 worker 进程内同一分工只记一次审计，避免每轮
|
||
# 重调都刷屏。查库失败时 _quota_exceeded 同样返回
|
||
# 非空串（fail-closed），一并跳过。
|
||
gate = self._quota_exceeded(
|
||
{"created_by": task["created_by"]}, None)
|
||
if gate is not None:
|
||
key = (task["id"], name)
|
||
if key not in self._coord_quota_skipped:
|
||
self._coord_quota_skipped.add(key)
|
||
self._audit(
|
||
task["project_id"],
|
||
"coordinator_child_quota_skipped",
|
||
"%s: %s" % (name, gate))
|
||
continue
|
||
child_id = uuid.uuid4().hex
|
||
conn.execute(
|
||
"INSERT INTO tasks(id, project_id, prompt, status,"
|
||
" idempotency_key, depends_on, priority, created_by,"
|
||
" created_at, updated_at, last_heartbeat, executor,"
|
||
" model, effort, environment_id, parent_task_id,"
|
||
" coordinator_part, child_model, child_effort)"
|
||
" VALUES(?, ?, ?, 'queued', NULL, '[]', ?, ?, ?, ?,"
|
||
" NULL, ?, ?, ?, ?, ?, ?, ?, ?)",
|
||
(child_id, task["project_id"], part["prompt"],
|
||
task.get("priority", 0) or 0, task["created_by"],
|
||
now, now,
|
||
part.get("executor") or task.get("executor"),
|
||
# P1：分工未指定模型/强度时继承父任务（旧版
|
||
# 直接写 part 的 None，子任务会落到 worker
|
||
# 默认模型/强度，与父任务的显式选择脱节，
|
||
# 既改成本也改行为，且排查时看不出继承关系）。
|
||
# 派生默认优先于父任务自身：分工显式写 >
|
||
# 协调员的 child_model / child_effort >
|
||
# 协调员自身的 model / effort。
|
||
part.get("model") or task.get("child_model")
|
||
or task.get("model"),
|
||
part.get("effort") or task.get("child_effort")
|
||
or task.get("effort"),
|
||
task.get("environment_id"), task["id"], name,
|
||
task.get("child_model"),
|
||
task.get("child_effort")))
|
||
self._audit(
|
||
task["project_id"], "coordinator_child_created",
|
||
"%s -> %s" % (name, child_id))
|
||
children.append({"name": name, "id": child_id})
|
||
conn.commit()
|
||
finally:
|
||
conn.close()
|
||
return children
|
||
|
||
def _coord_child_rows(self, child_ids):
|
||
"""子任务当前行：{id: {status, result_text, error_text}}。
|
||
|
||
已删或查不到的子任务按 interrupted 终态合成一行（error_text
|
||
注明行已消失）：等齐判断里缺失不再等到超时，合流报告里
|
||
该子任务标 interrupted 且整体不为 completed；孤儿标记不
|
||
会对不存在的行再 UPDATE（rowcount 本来就是 0）。行消失
|
||
不是完成，绝不能静默当 completed 计。
|
||
"""
|
||
rows = {}
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
for cid in child_ids:
|
||
row = conn.execute(
|
||
"SELECT id, status, result_text, error_text FROM tasks"
|
||
" WHERE id = ?", (cid,)).fetchone()
|
||
if row is None:
|
||
rows[cid] = {
|
||
"status": "interrupted",
|
||
"result_text": None,
|
||
"error_text": "子任务行已消失（已删除）"}
|
||
continue
|
||
try:
|
||
rows[row["id"]] = {
|
||
"status": row["status"],
|
||
"result_text": row["result_text"],
|
||
"error_text": row["error_text"]}
|
||
except (KeyError, IndexError, TypeError):
|
||
continue
|
||
finally:
|
||
conn.close()
|
||
return rows
|
||
|
||
def _coord_mark_orphans(self, task, child_ids):
|
||
"""把仍未终态的子任务标 interrupted 并记审计（P1 孤儿收口）。
|
||
|
||
只对 CAS 改到行的子任务记审计（rowcount==1）：已被别的
|
||
worker 终态化的子任务不动，避免覆盖它的结果。返回被标
|
||
记的子任务 id 列表。正在跑的子任务被标 interrupted 后，
|
||
跑它的 worker 会在轮询里看到库状态并杀进程/容器（popen
|
||
每 1 秒查库、Docker 取消看门狗每 1 秒查库），不会无声
|
||
继续烧钱；若那个 worker 也已死，子任务至少已是终态，
|
||
不会再被认领烧钱。
|
||
"""
|
||
marked = []
|
||
rows = self._coord_child_rows(child_ids)
|
||
pending = _coord_nonterminal([
|
||
{"id": cid, "status": (rows.get(cid) or {}).get("status")}
|
||
for cid in child_ids])
|
||
if not pending:
|
||
return marked
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
for ch in pending:
|
||
cur = conn.execute(
|
||
"UPDATE tasks SET status = 'interrupted',"
|
||
" interrupted_reason = 'coordinator_orphan',"
|
||
" updated_at = ? WHERE id = ? AND status NOT IN"
|
||
" ('completed', 'failed', 'interrupted', 'merged',"
|
||
" 'merge-failed')",
|
||
(time.time(), ch["id"]))
|
||
if cur.rowcount == 1:
|
||
marked.append(ch["id"])
|
||
self._audit(
|
||
task["project_id"], "coordinator_child_orphaned",
|
||
"%s -> %s (parent %s)"
|
||
% (ch["id"], "interrupted", task["id"]))
|
||
conn.commit()
|
||
finally:
|
||
conn.close()
|
||
return marked
|
||
|
||
def _reap_kill_requests(self, sweep_threshold=300):
|
||
"""跨 worker 真 kill 的收殓器（run loop 每轮调一次）。
|
||
|
||
分工（与 owner 活着时的 1 秒看门狗互补，绝不抢杀）：
|
||
owner 活着时，它的 _docker_cancel_watchdog 在任务变
|
||
interrupted 后 1 秒内按容器名真杀，本函数不碰；只有
|
||
owner 已死（任务心跳停滞超过 sweep_threshold，与陈旧
|
||
回收同一阈值、同一口径 COALESCE(last_heartbeat,
|
||
updated_at)）时，本函数才动手——因为这时已没有任何
|
||
进程会来杀这个容器，不杀就是一直烧钱。
|
||
|
||
判定链（任一不满足即跳过，请求留待下轮；任务行失踪
|
||
或任务已不在 interrupted 时请求直接终结，避免复活的
|
||
任务（同 id 重回 queued）被旧请求误杀）：
|
||
1. 请求 status 仍为 pending；
|
||
2. 任务存在且 status == 'interrupted'；
|
||
3. 任务心跳停滞超过 sweep_threshold；
|
||
4. 本机 executor 有 container_name/kill_container
|
||
（docker 执行器才有；local 执行器无容器名，跳过）。
|
||
杀本身按任务派生容器名（executor.container_name，
|
||
codex-<安全字符任务 id>），绝不按前缀扫、绝不杀自由
|
||
传入的名字；本机无此容器时 kill_container 返回 False
|
||
也不报错，请求照常终结。终结时写 kill_request_reaped
|
||
审计；单条请求异常只记日志、不炸整轮。
|
||
返回终结的请求 id 列表。
|
||
"""
|
||
done = []
|
||
try:
|
||
conn = _db_connect(self.db_path)
|
||
except Exception as exc:
|
||
self.logger.warning("kill reap: db connect failed: %s", exc)
|
||
return done
|
||
try:
|
||
try:
|
||
conn.execute(
|
||
"CREATE TABLE IF NOT EXISTS kill_requests("
|
||
"id TEXT PRIMARY KEY, "
|
||
"task_id TEXT, "
|
||
"requested_by TEXT, "
|
||
"status TEXT, "
|
||
"requested_at DOUBLE PRECISION, "
|
||
"done_at DOUBLE PRECISION)")
|
||
conn.commit()
|
||
except Exception:
|
||
pass
|
||
try:
|
||
reqs = conn.execute(
|
||
"SELECT id, task_id FROM kill_requests"
|
||
" WHERE status = 'pending'").fetchall()
|
||
except Exception as exc:
|
||
self.logger.warning("kill reap: list failed: %s", exc)
|
||
return done
|
||
now = time.time()
|
||
kill = getattr(self.executor, "kill_container", None)
|
||
name_of = getattr(self.executor, "container_name", None)
|
||
for req in reqs:
|
||
req_id = req["id"]
|
||
task_id = req["task_id"]
|
||
try:
|
||
row = conn.execute(
|
||
"SELECT status, project_id, last_heartbeat,"
|
||
" updated_at FROM tasks WHERE id = ?",
|
||
(task_id,)).fetchone()
|
||
if row is None:
|
||
# 任务行失踪：请求已无意义，直接终结。
|
||
conn.execute(
|
||
"UPDATE kill_requests SET status = 'done',"
|
||
" done_at = ? WHERE id = ?",
|
||
(now, req_id))
|
||
conn.commit()
|
||
done.append(req_id)
|
||
continue
|
||
if row["status"] != "interrupted":
|
||
# 任务已不在 interrupted（重回 queued 等）：
|
||
# 旧请求直接终结，绝不能杀新跑。
|
||
conn.execute(
|
||
"UPDATE kill_requests SET status = 'done',"
|
||
" done_at = ? WHERE id = ?",
|
||
(now, req_id))
|
||
conn.commit()
|
||
done.append(req_id)
|
||
self._audit(row["project_id"],
|
||
"kill_request_stale",
|
||
"%s not_interrupted" % task_id)
|
||
continue
|
||
hb = row["last_heartbeat"]
|
||
if hb is None:
|
||
hb = row["updated_at"]
|
||
if hb is not None and hb >= now - sweep_threshold:
|
||
# owner 还活着：留待下轮，绝不抢杀。
|
||
continue
|
||
# owner 已死：按任务派生容器名在本机真杀。
|
||
# 容器名只由任务 id 派生，不接受外部输入；
|
||
# 本机无此容器时 kill 返回 False 也不报错。
|
||
if kill is not None and name_of is not None:
|
||
try:
|
||
kill({"id": task_id})
|
||
except Exception:
|
||
pass
|
||
conn.execute(
|
||
"UPDATE kill_requests SET status = 'done',"
|
||
" done_at = ? WHERE id = ?",
|
||
(now, req_id))
|
||
conn.commit()
|
||
done.append(req_id)
|
||
self._audit(row["project_id"], "kill_request_reaped",
|
||
task_id)
|
||
except Exception as exc:
|
||
self.logger.warning(
|
||
"kill reap: request %s failed: %s", req_id, exc)
|
||
try:
|
||
conn.rollback()
|
||
except Exception:
|
||
pass
|
||
continue
|
||
finally:
|
||
conn.close()
|
||
return done
|
||
|
||
def sweep_orphaned_coordinator_children(self):
|
||
"""常驻循环每轮的孤儿扫尾（P1）：父已终态或父行失踪、子
|
||
仍未终态时，把子标 interrupted 并记审计。
|
||
|
||
父仍在 running/queued 时绝不动子（协调员等齐期间子任务
|
||
未终态是正常状态）。父行失踪（被删）同样算孤儿：没有
|
||
父会再来合流，子任务继续跑只是烧钱。返回被标记的
|
||
(parent_id, child_id) 对列表；查库失败时返回 [] 且只记
|
||
日志，不炸循环。
|
||
"""
|
||
marked = []
|
||
try:
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
rows = conn.execute(
|
||
"SELECT id, parent_task_id, project_id FROM tasks"
|
||
" WHERE parent_task_id IS NOT NULL AND status NOT IN"
|
||
" ('completed', 'failed', 'interrupted', 'merged',"
|
||
" 'merge-failed')").fetchall()
|
||
parents = {}
|
||
for row in rows:
|
||
try:
|
||
pid = row["parent_task_id"]
|
||
except (KeyError, IndexError, TypeError):
|
||
continue
|
||
if pid not in parents:
|
||
prow = conn.execute(
|
||
"SELECT status FROM tasks WHERE id = ?",
|
||
(pid,)).fetchone()
|
||
try:
|
||
parents[pid] = prow["status"] \
|
||
if prow is not None else None
|
||
except (KeyError, IndexError, TypeError):
|
||
parents[pid] = None
|
||
for row in rows:
|
||
try:
|
||
cid = row["id"]
|
||
pid = row["parent_task_id"]
|
||
proj = row["project_id"]
|
||
except (KeyError, IndexError, TypeError):
|
||
continue
|
||
pstatus = parents.get(pid)
|
||
if pstatus is not None and \
|
||
pstatus not in TERMINAL_STATUSES:
|
||
continue
|
||
cur = conn.execute(
|
||
"UPDATE tasks SET status = 'interrupted',"
|
||
" interrupted_reason = 'coordinator_orphan',"
|
||
" updated_at = ? WHERE id = ? AND status NOT IN"
|
||
" ('completed', 'failed', 'interrupted', 'merged',"
|
||
" 'merge-failed')",
|
||
(time.time(), cid))
|
||
if cur.rowcount == 1:
|
||
marked.append((pid, cid))
|
||
self._audit(
|
||
proj, "coordinator_child_orphaned",
|
||
"%s -> interrupted (parent %s, parent"
|
||
" status %s)" % (cid, pid, pstatus))
|
||
conn.commit()
|
||
finally:
|
||
conn.close()
|
||
except Exception as exc:
|
||
log_event(self.logger, "worker_orphan_sweep_failed",
|
||
message="orphan sweep failed: %s" % exc)
|
||
return marked
|
||
|
||
def _run_coordinator(self, task, started_at, poll_interval=None,
|
||
timeout=None):
|
||
"""协调员执行：建子任务 → 等齐 → 合流汇总写回。
|
||
|
||
占一个 worker 槽位直到子任务全部终态：等齐期间每轮刷心跳
|
||
（sweep 不会误回收）；子任务本身由各 worker 并行认领。
|
||
占位由调用方 run_one 的 finally 在终局后释放，本函数
|
||
不重复清（旧版直接 return 导致占位永不释放，见上）。
|
||
取消与超时都会尽力取消未终态的子任务，不留孤儿继续烧
|
||
模型费用。返回 run_one 同款结果 dict（无模型花费，
|
||
cost_record_id 为 None）。
|
||
"""
|
||
task_id = task["id"]
|
||
project_id = task["project_id"]
|
||
parts = task.get("coordinator_parts") or []
|
||
if poll_interval is None:
|
||
poll_interval = _COORD_POLL_INTERVAL
|
||
if timeout is None:
|
||
timeout = _COORD_TIMEOUT
|
||
children = self._coord_sync_children(task, parts)
|
||
child_ids = [c["id"] for c in children]
|
||
deadline = time.time() + timeout
|
||
cancelled = False
|
||
timed_out = False
|
||
while True:
|
||
try:
|
||
self.heartbeat(task_id)
|
||
except Exception:
|
||
pass
|
||
# 行消失防线：协调员本体行被删时 _was_cancelled 查不到
|
||
# 行会返回 False（get_task 为 None），不能因此等到
|
||
# deadline——行没了即按取消处理，立即跳出放槽；终局
|
||
# CAS 写不回去，最终状态按 interrupted 计（见循环后
|
||
# final_written 分支），不写回、不抛错、不当完成。
|
||
if self.get_task(task_id) is None:
|
||
cancelled = True
|
||
break
|
||
if self._was_cancelled(task_id):
|
||
cancelled = True
|
||
break
|
||
# 等齐判断：_coord_child_rows 已把行消失的子任务合成
|
||
# interrupted 终态，缺失不再拖住循环等到超时。
|
||
rows = self._coord_child_rows(child_ids)
|
||
if all((rows.get(cid) or {}).get("status") in TERMINAL_STATUSES
|
||
for cid in child_ids):
|
||
break
|
||
if time.time() >= deadline:
|
||
timed_out = True
|
||
break
|
||
time.sleep(poll_interval)
|
||
orphaned = []
|
||
if cancelled or timed_out:
|
||
rows = self._coord_child_rows(child_ids)
|
||
for cid in child_ids:
|
||
if (rows.get(cid) or {}).get("status") \
|
||
not in TERMINAL_STATUSES:
|
||
try:
|
||
self.cancel(cid)
|
||
except Exception:
|
||
pass
|
||
# P1：cancel() 对已终态/不存在的子任务返回 False，
|
||
# 跨 worker 在跑的子任务也可能因竞态仍未终态——
|
||
# cancel 之后仍未终态的一律直接标 interrupted 并记
|
||
# 审计（_coord_mark_orphans），绝不留孤儿无声烧钱；
|
||
# 标记结果写进汇总报告，调用方在详情里看得到。
|
||
try:
|
||
orphaned = self._coord_mark_orphans(task, child_ids)
|
||
except Exception:
|
||
orphaned = []
|
||
rows = self._coord_child_rows(child_ids)
|
||
merged = []
|
||
for ch in children:
|
||
row = rows.get(ch["id"]) or {}
|
||
merged.append({
|
||
"name": ch["name"], "id": ch["id"],
|
||
"status": row.get("status") or "?",
|
||
"result_text": row.get("result_text"),
|
||
"error_text": row.get("error_text")})
|
||
report, overall = _coord_build_report(merged)
|
||
if orphaned:
|
||
report += ("\n\n孤儿子任务已标 interrupted：%s"
|
||
% "、".join(orphaned))
|
||
if cancelled:
|
||
final = "interrupted"
|
||
error = "cancelled"
|
||
elif timed_out:
|
||
final = "failed"
|
||
pending = [m["name"] for m in merged
|
||
if m["status"] not in TERMINAL_STATUSES]
|
||
error = "coordinator timeout: %s 未完成" % "、".join(pending)
|
||
report = error + "\n\n" + report
|
||
else:
|
||
final = overall
|
||
error = None
|
||
if final != "completed":
|
||
failed = [m["name"] for m in merged
|
||
if m["status"] != "completed"]
|
||
error = "coordinator: %s 未完成" % "、".join(failed)
|
||
final_written = self._set_status_if_running(task_id, final)
|
||
if not final_written:
|
||
final = "interrupted"
|
||
finished_at = time.time()
|
||
# P0 安全批后：CAS 没改到行时不写回（见终局路径注释）。
|
||
if final_written:
|
||
self._write_execution_result(
|
||
task_id, result_text=report, error_text=error,
|
||
actual_model=None, started_at=started_at,
|
||
finished_at=finished_at, only_if_status=final)
|
||
self._cleanup_task_seed(task)
|
||
if final_written:
|
||
self._audit(project_id,
|
||
"worker_completed" if final == "completed"
|
||
else ("worker_interrupted"
|
||
if final == "interrupted"
|
||
else "worker_failed"),
|
||
task_id)
|
||
log_event(self.logger,
|
||
"worker_completed" if final == "completed"
|
||
else ("worker_interrupted"
|
||
if final == "interrupted"
|
||
else "worker_failed"),
|
||
task_id=task_id, project_id=project_id, status=final)
|
||
if final in ("completed", "failed"):
|
||
self._notify_task_finished(task_id, project_id, final,
|
||
error=error)
|
||
else:
|
||
self._audit(project_id, "worker_interrupted", task_id)
|
||
# P0 三件其三：时间线终局步骤（best-effort）。
|
||
try:
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
self._ensure_result_columns(conn)
|
||
self._record_step(conn, task_id, "final",
|
||
"终态 %s" % final)
|
||
conn.commit()
|
||
finally:
|
||
conn.close()
|
||
except Exception:
|
||
pass
|
||
host_name = self.executor._host_name(task) \
|
||
if hasattr(self.executor, "_host_name") else task_id
|
||
return {"dispatched": True, "task_id": task_id,
|
||
"host_name": host_name, "status": final,
|
||
"executor": self._executor_name(), "stdout": report,
|
||
"error": error, "events": [], "cost_record_id": None}
|
||
|
||
def _record_usage_cost(self, task, actual_model, usage_total,
|
||
stdout, fallback_count):
|
||
"""记一条任务成本并累加 worker 级用量（调用方须持 _db_lock）。
|
||
|
||
P1 成本修齐：旧版 max_cost_usd 参数存在但 usage_cost_usd
|
||
从未累加——_budget_exceeded 的成本门永远是 0，等于没设。
|
||
本函数是唯一记账点（终局块与取消分支共用）：actual 行
|
||
按价目表算 usd_micros（无价模型为 None），有价行才累加
|
||
usage_cost_usd（无价行不折算成 0 美元，仍受无价上限管）；
|
||
无用量时回落字符估算记 estimated，绝不冒充 actual。
|
||
cache_write_input_tokens 不进这里：它只在 usage dict 里
|
||
做观测（见 cost.py 的不计费声明），不加进 total、不用
|
||
于定价。返回 cost record id。
|
||
"""
|
||
task_id = task["id"]
|
||
if usage_total and (usage_total.get("total_tokens") or 0) > 0:
|
||
micros = price_usd_micros(
|
||
actual_model, usage_total.get("input_tokens", 0),
|
||
usage_total.get("cached_input_tokens", 0),
|
||
usage_total.get("output_tokens", 0))
|
||
cost_id = self.costs.record(
|
||
task_id, task["project_id"], actual_model,
|
||
input_tokens=usage_total.get("input_tokens", 0),
|
||
output_tokens=usage_total.get("output_tokens", 0),
|
||
cached_input_tokens=usage_total.get(
|
||
"cached_input_tokens", 0),
|
||
usd_micros=micros,
|
||
source="actual", fallback_count=fallback_count)
|
||
else:
|
||
micros = None
|
||
total = estimate_tokens(task.get("prompt") or "", stdout)
|
||
cost_id = self.costs.record(
|
||
task_id, task["project_id"], actual_model,
|
||
input_tokens=total, output_tokens=0, usd_micros=None,
|
||
source="estimated", fallback_count=fallback_count)
|
||
if micros is not None:
|
||
# max_cost_usd 不再是死参数：只有有价花费才累加，
|
||
# _budget_exceeded 据此在下一认领前挡住超支 worker。
|
||
self.usage_cost_usd += micros / 1000000.0
|
||
return cost_id
|
||
|
||
def run_one(self, task=None):
|
||
"""认领并执行一个任务，返回结果 dict（见模块 docstring）。
|
||
|
||
未派发时返回 {"dispatched": False, "reason": ...} 且不动任务；
|
||
派发后（即使缺 key 失败）返回 dispatched=True 与最终状态。
|
||
"""
|
||
if self._budget_exceeded():
|
||
return {"dispatched": False, "reason": "budget_exceeded"}
|
||
if task is None:
|
||
task = self.pick_next()
|
||
if task is None:
|
||
return {"dispatched": False, "reason": "no_ready_task"}
|
||
task_id = task["id"]
|
||
if not self._claim(task_id, executor=self._executor_name()):
|
||
return {"dispatched": False, "reason": "already_claimed",
|
||
"task_id": task_id}
|
||
# P0 三件其三：时间线 claim 步骤（best-effort，写失败
|
||
# 不影响执行；step_no 由 _record_step 自增）。
|
||
try:
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
self._ensure_result_columns(conn)
|
||
self._record_step(conn, task_id, "claim", "worker 认领")
|
||
conn.commit()
|
||
finally:
|
||
conn.close()
|
||
except Exception:
|
||
pass
|
||
task = self.get_task(task_id) or task
|
||
# G10：goal 断点续跑标记——认领后立即清掉库里标记（本轮
|
||
# 已认领为 running，不会重复触发；本轮若再被杀回
|
||
# interrupted，用户需重新调 /goal-resume 才会再置标记，
|
||
# 不会因旧标记残留而误把一次普通重跑当成断点续跑）。
|
||
# 断点优先于续跑消息：本轮不把 prompt 换成 resume_message，
|
||
# 也不消费它（局部变量置 None，终局清理不删库里那条），
|
||
# 避免断点续跑把一条待续跑消息悄悄吃掉。
|
||
if task.get("goal_resume_pending"):
|
||
task = dict(task)
|
||
task["_goal_resume_breakpoint"] = True
|
||
# P0 三件其一：断点丢失重建——host-state 里条目已丢
|
||
# 但库里有断点时，先按库断点拼最小条目写回，再由
|
||
# 正常 goal-resume 续跑；库里也没有断点时返回 False。
|
||
# 从未启动回落（2026-10-29 实测任务 A）：重建失败
|
||
# 且四者全无执行痕迹时，goal-resume 注定被 host
|
||
# 拒绝 → failed，改走全新 goal（见
|
||
# _resolve_breakpoint_resume），不造假断点。
|
||
task = self._resolve_breakpoint_resume(task)
|
||
try:
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
conn.execute(
|
||
"UPDATE tasks SET goal_resume_pending = 0"
|
||
" WHERE id = ?", (task_id,))
|
||
conn.commit()
|
||
finally:
|
||
conn.close()
|
||
except Exception:
|
||
pass
|
||
resume_message = None
|
||
else:
|
||
resume_message = task.get("resume_message")
|
||
# 续跑消息：旧任务继续会话时本轮要说的话——本轮 prompt 换
|
||
# 成它（只换内存里的副本，原 prompt 不动）；_resume 标记
|
||
# 让 _run_attempt 在 goal 模式下走 goal-resume 而不是 goal
|
||
# （goal 会撞已存在，只有 goal-resume 能在旧线程上开新循环）。
|
||
# P1 goal-resume 分流（断点 vs 开新目标，三处一致的 worker 侧）：
|
||
# - 旧验收形态（verify_command 非空）：一律断点续跑——
|
||
# 绝不能把续跑消息当新 goal_text 传给 host.py，否则
|
||
# host 的 goal-resume --goal-text 分支会用 verify=None
|
||
# 开新循环，把原 verify_command 覆盖掉；断点调用不传
|
||
# --goal-text，host 沿用原 goal + 原 verify + 原轮次，
|
||
# 从 goal_rounds_done 的下一轮续跑。续跑消息本轮不
|
||
# 当 prompt 用（断点语义就是续原目标），跑完照常清掉。
|
||
# - goal_mode=1（无验收）：开新目标——续跑消息就是新
|
||
# goal_text（服务端 /continue 带 goal_mode=1 时已把
|
||
# task.goal 写成该消息，这里与之对齐）。
|
||
# - 非 goal 任务：沿用旧行为（prompt 换成续跑消息，走
|
||
# 已存在续跑 run 路径）。
|
||
if resume_message:
|
||
task = dict(task)
|
||
if task.get("verify_command"):
|
||
task["_resume"] = True
|
||
task["_resume_kind"] = "breakpoint"
|
||
elif task.get("goal_mode"):
|
||
task["prompt"] = resume_message
|
||
task["_resume"] = True
|
||
task["_resume_kind"] = "new-goal"
|
||
else:
|
||
task["prompt"] = resume_message
|
||
task["_resume"] = True
|
||
# 自动记忆注入（官方 use_memories）：首轮才注入，见
|
||
# _apply_auto_memory_injection 的跳过规则；只改内存副本。
|
||
task = self._apply_auto_memory_injection(task)
|
||
self._audit(task["project_id"], "worker_claim",
|
||
"%s executor=%s" % (task_id, self._executor_name()))
|
||
# log_event 只接受 task_id/project_id/status/message 四个
|
||
# 字段，executor 只进审计 detail，不进日志调用（否则
|
||
# TypeError 会炸掉整个 run_one）。
|
||
log_event(self.logger, "worker_claim", task_id=task_id,
|
||
project_id=task["project_id"], status="running")
|
||
started_at = time.time()
|
||
|
||
# 缺 key 门：认领之后、调 host.py 之前；缺 key 直接 failed，
|
||
# 不调 runner（假 CLI 单测可断言 runner 未被调用）。
|
||
if not self.key_available():
|
||
error = _missing_key_error(self.env_key)
|
||
self._set_status(task_id, "failed")
|
||
self._write_execution_result(
|
||
task_id, error_text=error, actual_model=self.model,
|
||
started_at=started_at, finished_at=time.time(),
|
||
only_if_status="failed")
|
||
# 缺 key 门提前返回：同样要清暂存，否则与仓库解析
|
||
# 失败一路同款残留（见 _cleanup_task_seed）。
|
||
self._cleanup_task_seed(task)
|
||
self._audit(task["project_id"], "worker_missing_key", error)
|
||
log_event(self.logger, "worker_missing_key", task_id=task_id,
|
||
project_id=task["project_id"], status="failed")
|
||
self._notify_task_finished(
|
||
task_id, task["project_id"], "failed", error=error)
|
||
# 认领占位在此释放：缺 key 门在执行器起进程之前返回，
|
||
# 不清占位会把槽位漏到 worker 退出（见 _claim 文档）。
|
||
self._running_clear(task_id)
|
||
return {"dispatched": True, "task_id": task_id,
|
||
"host_name": self.executor._host_name(task)
|
||
if hasattr(self.executor, "_host_name") else task_id,
|
||
"status": "failed", "executor": self._executor_name(),
|
||
"stdout": "", "error": error, "events": [],
|
||
"cost_record_id": None}
|
||
|
||
# 项目仓库解析：认领 + 缺 key 门之后、真正执行之前。
|
||
# repo_url 为空时不动（默认 repo）；解析失败时任务直接 failed，
|
||
# 错误文本写清是仓库解析失败，绝不标 completed，也不在此重试。
|
||
repo_error = self._apply_project_repo(task)
|
||
if repo_error is not None:
|
||
self._set_status(task_id, "failed")
|
||
self._write_execution_result(
|
||
task_id, error_text=repo_error, actual_model=self.model,
|
||
started_at=started_at, finished_at=time.time(),
|
||
only_if_status="failed")
|
||
# 仓库解析失败提前返回：够不到正常终局后的清理，
|
||
# 这里必须先清暂存，否则 bundle 目录永久残留。
|
||
self._cleanup_task_seed(task)
|
||
self._audit(task["project_id"], "worker_repo_failed", repo_error)
|
||
log_event(self.logger, "worker_repo_failed", task_id=task_id,
|
||
project_id=task["project_id"], status="failed")
|
||
self._notify_task_finished(
|
||
task_id, task["project_id"], "failed", error=repo_error)
|
||
# 同缺 key 门：仓库解析失败在起进程之前返回，先清占位。
|
||
self._running_clear(task_id)
|
||
return {"dispatched": True, "task_id": task_id,
|
||
"host_name": self.executor._host_name(task)
|
||
if hasattr(self.executor, "_host_name") else task_id,
|
||
"status": "failed", "executor": self._executor_name(),
|
||
"stdout": "", "error": repo_error, "events": [],
|
||
"cost_record_id": None}
|
||
|
||
# 配额认领门：排队期间配额被花完（或被调低）时，任务不能
|
||
# 再开跑——直接标 interrupted + 配额原因，不记成本（还没花
|
||
# 钱），续跑消息不消费，顶上来后重试/续跑可再跑。与 key
|
||
# 门、仓库门同款提前返回形状，区别是状态为 interrupted
|
||
#（可续跑）而非 failed。G7 分层：认领门用 closed——
|
||
# 查询异常时同样标 interrupted（quota_check_unavailable，
|
||
# 可重试/续跑），绝不在“不知道超没超”时开跑；轮询与
|
||
# 轮次边界仍用默认 open，查询抖动不杀在跑任务。
|
||
quota_error = self._quota_exceeded(task, None, on_error="closed")
|
||
if quota_error is not None:
|
||
self._set_status(task_id, "interrupted")
|
||
# 配额中断不自动恢复：要等配额恢复/调高，自动
|
||
# 回 queued 只会让 autoscaler 按队列深度白拉 worker。
|
||
self._mark_interrupted_reason(task_id, "quota")
|
||
self._write_execution_result(
|
||
task_id, error_text=quota_error, actual_model=self.model,
|
||
started_at=started_at, finished_at=time.time(),
|
||
only_if_status="interrupted")
|
||
self._audit(task["project_id"], "worker_quota_stop",
|
||
"%s: %s" % (task_id, quota_error))
|
||
log_event(self.logger, "worker_quota_stop", task_id=task_id,
|
||
project_id=task["project_id"],
|
||
status="interrupted")
|
||
# 同缺 key 门：配额门在起进程之前返回，先清占位。
|
||
self._running_clear(task_id)
|
||
return {"dispatched": True, "task_id": task_id,
|
||
"host_name": self.executor._host_name(task)
|
||
if hasattr(self.executor, "_host_name") else task_id,
|
||
"status": "interrupted",
|
||
"executor": self._executor_name(),
|
||
"stdout": "", "error": quota_error, "events": [],
|
||
"cost_record_id": None}
|
||
|
||
# P1 goal 轮次上限门（执行侧防线）：旧验收形态的
|
||
# goal_max_rounds 超过 GOAL_MAX_ROUNDS_CAP 时直接 failed，
|
||
# 不进 host.py goal 循环烧轮次；goal_mode=1 的 0（模型宣告
|
||
# 才结束）豁免轮次上限（上界是配额），但 0 配上旧验收命令
|
||
# 时同样 failed（服务端建任务时 0 本就 400，老行若有 0
|
||
# 验收形态属于脏数据，不按不限轮次跑）。建任务时的 400
|
||
# 校验在 server.py（另一路文件），这里是执行侧第二道门，
|
||
# 错误文本写明上限与“请新建合规任务”，不静默截断轮次
|
||
# （截断会改任务语义，调用方看不出被改过）。
|
||
_goal_rounds = task.get("goal_max_rounds")
|
||
if (task.get("verify_command") or task.get("goal_mode")) \
|
||
and _goal_rounds is not None:
|
||
_goal_cap_error = None
|
||
try:
|
||
_goal_rounds_int = int(_goal_rounds)
|
||
except (TypeError, ValueError):
|
||
_goal_rounds_int = None
|
||
if _goal_rounds_int is not None \
|
||
and _goal_rounds_int > GOAL_MAX_ROUNDS_CAP:
|
||
_goal_cap_error = (
|
||
"goal_max_rounds 超过上限 %d 轮（当前 %d），"
|
||
"请新建轮次合规的任务" % (
|
||
GOAL_MAX_ROUNDS_CAP, _goal_rounds_int))
|
||
elif _goal_rounds_int == 0 and task.get("verify_command") \
|
||
and not task.get("goal_mode"):
|
||
_goal_cap_error = (
|
||
"旧验收形态 goal_max_rounds=0 表示不限轮次，"
|
||
"已禁用（上限 %d 轮），请新建轮次合规的任务"
|
||
% GOAL_MAX_ROUNDS_CAP)
|
||
if _goal_cap_error is not None:
|
||
self._set_status(task_id, "failed")
|
||
self._write_execution_result(
|
||
task_id, error_text=_goal_cap_error,
|
||
actual_model=self.model,
|
||
started_at=started_at, finished_at=time.time(),
|
||
only_if_status="failed")
|
||
self._audit(task["project_id"], "worker_goal_cap_rejected",
|
||
"%s: %s" % (task_id, _goal_cap_error))
|
||
log_event(self.logger, "worker_goal_cap_rejected",
|
||
task_id=task_id, project_id=task["project_id"],
|
||
status="failed")
|
||
self._running_clear(task_id)
|
||
return {"dispatched": True, "task_id": task_id,
|
||
"host_name": self.executor._host_name(task)
|
||
if hasattr(self.executor, "_host_name") else task_id,
|
||
"status": "failed",
|
||
"executor": self._executor_name(),
|
||
"stdout": "", "error": _goal_cap_error, "events": [],
|
||
"cost_record_id": None}
|
||
|
||
# 协调员分支：不跑模型回合，改为按分工建并行子任务、等齐、
|
||
# 合流汇总。已过 key 门与配额门；仓库门已对协调员任务本身
|
||
# 跑过（项目仓库解析失败会在前面直接 failed，不会建一堆
|
||
# 注定失败的子任务）；子任务各自再过 key/配额/仓库门。
|
||
# 槽位清理：普通任务在 _run_attempt 的 finally 里清
|
||
# _running 占位，协调员不走 _run_attempt——旧版这里直接
|
||
# return _run_coordinator，协调员完成后续占位永不释放，
|
||
# 同进程 worker 的有效容量会逐个协调员递减直至再也
|
||
# 认领不了（2026-10-28 40 人与 18 人旧轮的第 4 槽/C-6
|
||
# 未认领即此漏）。finally 清槽，异常路径同样不漏。
|
||
if task.get("coordinator_parts"):
|
||
try:
|
||
return self._run_coordinator(task, started_at)
|
||
finally:
|
||
self._running_clear(task_id)
|
||
|
||
events = []
|
||
fallback_count = 0
|
||
result = None
|
||
error = None
|
||
retry_count = 0
|
||
# 最后真正跑过的模型：主模型起手，fallback 后换成 fallback 模型。
|
||
actual_model = self.model
|
||
fallback_model = None
|
||
first_error = None
|
||
# 模型优先级：任务级 > worker 主模型 > 配置文件（host.py
|
||
# 再兜底）。此前这里无条件写成 worker 主模型，导致控制
|
||
# 台选的模型被悄悄覆盖（任务级模型实际不生效）；修掉。
|
||
# fallback 时下面再把 task["model"] 换成 fallback 模型。
|
||
# 网关门：sub2api 时先给属主开通网关账号（幂等）并解析默认
|
||
# 模型；失败直接 failed，不进执行器（避免跑错网关或拿着
|
||
# xcmapi 模型名去撞 sub2api）。xcmapi 下 _ensure_gateway
|
||
# 直接返回 (None, None)，不做任何事。注意：这里改的是
|
||
# 原 task dict，随后 task = dict(task) 会原样拷走。
|
||
try:
|
||
gateway_provider, gateway_env = self._ensure_gateway(task)
|
||
if not task.get("model") and gateway_provider == "sub2api":
|
||
task["model"] = self._sub2api_default_model(gateway_env)
|
||
except Exception as exc:
|
||
error = "网关准备失败: %s" % exc
|
||
self._set_status(task_id, "failed")
|
||
self._write_execution_result(
|
||
task_id, error_text=error, actual_model=self.model,
|
||
started_at=started_at, finished_at=time.time(),
|
||
only_if_status="failed")
|
||
self._audit(task["project_id"], "worker_gateway_failed",
|
||
"%s: %s" % (task_id, error))
|
||
log_event(self.logger, "worker_gateway_failed",
|
||
task_id=task_id, project_id=task["project_id"],
|
||
status="failed")
|
||
self._notify_task_finished(
|
||
task_id, task["project_id"], "failed", error=error)
|
||
return {"dispatched": True, "task_id": task_id,
|
||
"host_name": self.executor._host_name(task)
|
||
if hasattr(self.executor, "_host_name") else task_id,
|
||
"status": "failed", "executor": self._executor_name(),
|
||
"stdout": "", "error": error, "events": [],
|
||
"cost_record_id": None}
|
||
|
||
task = dict(task)
|
||
if not task.get("model"):
|
||
task["model"] = self.model
|
||
actual_model = task["model"]
|
||
# 有限重试与真实 fallback 的优先级：取消不重试；一次尝试失败
|
||
# 且可 fallback、且显式配了 fallback_models 时，优先真换模型
|
||
# 再跑一次（最多 1 次，同一任务、同一 worktree/分支，dispatch
|
||
# 报“任务已存在”时 _run_attempt 内部走现有的 run 续跑路径），
|
||
# 第二次的结果就是最终结果，不再做同模型重试；未配
|
||
# fallback_models 时保持老行为（router 给出下一模型时只记事件
|
||
# 不重试）；其它失败（含 worker 超时）按 max_retries 做同模型
|
||
# 重试，退避等待后重跑一次 _run_attempt。
|
||
# 各次尝试的真实用量都要算进成本（失败尝试花的钱也是钱）：
|
||
# 主尝试与 fallback 尝试的结果都先累加，最后只记一条汇总。
|
||
usage_total = None
|
||
# 配额中断状态：任一边界（重试/fallback/后续消息/轮询掐
|
||
# 断）命中后置位，跳过后续轮次，走统一的配额中断收尾
|
||
#（interrupted + 配额原因 + 已花成本照记 + 续跑消息保留）。
|
||
quota_stop = False
|
||
quota_error = None
|
||
|
||
def _quota_hit():
|
||
# 闭包读当前值：usage_total 与 task 在重试/fallback
|
||
# 时会被重绑，轮询掐断要按最新值判断。
|
||
# P1 分流：执行中轮询只认真正的 quota_exceeded，
|
||
# quota_check_failed（查库失败）不杀当前轮——查询
|
||
# 抖动不应杀掉已在跑的回合；但它在重试/fallback/
|
||
# 后续消息等新回合边界仍是拒绝（那些边界直接调
|
||
# _quota_exceeded，不走这个过滤闭包）。
|
||
err = self._quota_exceeded(task, usage_total)
|
||
return err is not None and \
|
||
not err.startswith("quota_check_failed:")
|
||
|
||
# 分支任务：先 fork 再跑首轮（见 _run_fork）；首轮失败
|
||
# 不自动重试/fallback，失败后走重试入口重跑首轮（已存在
|
||
# 续跑），不重复 fork。range(0) 让下面的尝试循环跳过，
|
||
# 直接进统一的收尾（超时/取消/成本/审计/通知）逻辑。
|
||
if task.get("forked_from"):
|
||
result, error = self._run_fork(task, quota_check=_quota_hit)
|
||
usage_total = sum_usage(usage_total,
|
||
(result or {}).get("usage"))
|
||
self._deliver_peer_messages(task, result)
|
||
self._deliver_marker_tools(task, result)
|
||
if (result or {}).get("status") == "quota_exceeded":
|
||
quota_error = self._quota_exceeded(task, usage_total) or \
|
||
"quota_exceeded: 本月配额已用完"
|
||
quota_stop = True
|
||
for attempt in range(0 if task.get("forked_from")
|
||
else self.max_retries + 1):
|
||
if quota_stop:
|
||
break
|
||
if attempt > 0:
|
||
# 重试边界：上一轮已花钱，跑下一轮前再查一次。
|
||
quota_error = self._quota_exceeded(task, usage_total)
|
||
if quota_error is not None:
|
||
quota_stop = True
|
||
break
|
||
result, error = self._run_attempt(task, quota_check=_quota_hit)
|
||
usage_total = sum_usage(usage_total, (result or {}).get("usage"))
|
||
self._deliver_peer_messages(task, result)
|
||
self._deliver_marker_tools(task, result)
|
||
if (result or {}).get("status") == "quota_exceeded":
|
||
# 轮询掐断：进程已被杀，配额原因以现查为准。
|
||
quota_error = self._quota_exceeded(task, usage_total) or \
|
||
"quota_exceeded: 本月配额已用完"
|
||
quota_stop = True
|
||
break
|
||
if self._was_cancelled(task_id) or \
|
||
result.get("status") == "cancelled":
|
||
break
|
||
if task.get("verify_command") or task.get("goal_mode"):
|
||
# goal 模式只跑一次：循环内部已多轮重试（turn→验测→
|
||
# 喂回，或新形态的模型自决多轮），worker 级重试/fallback
|
||
# 不再叠加——同任务重跑会撞 host 任务名已存在，第二
|
||
# 次尝试必失败且污染错误文本。
|
||
# P1 唯一例外：goal 首轮撞“已存在”时（重试入口
|
||
# 重排同一 goal 任务的典型死法），不再直接 failed，
|
||
# 而是走一次断点续跑（goal-resume 不传 --goal-text，
|
||
# 沿用原 goal+verify，从断点续跑）；断点续跑的
|
||
# 结果即本轮结果（成功则完成，仍失败则以续跑的
|
||
# 错误为准，不再拼“已存在”）。已是续跑的任务
|
||
# （_resume）不再二次回落，避免断点循环。
|
||
# 分支任务（forked_from）不走这里：fork 首轮的
|
||
# 已存在语义是 dispatch 续跑，已在 _run_attempt
|
||
# 内处理，不叠加 goal 断点。
|
||
if not task.get("_resume") and not task.get("forked_from"):
|
||
_combined = ((result or {}).get("stdout", "") or "") \
|
||
+ ((result or {}).get("stderr", "") or "")
|
||
_goal_err = (result or {}).get("goal_error") or ""
|
||
if _ALREADY_EXISTS_MARKER in _combined or \
|
||
_ALREADY_EXISTS_MARKER in _goal_err or \
|
||
_ALREADY_EXISTS_MARKER in (error or ""):
|
||
self._audit(task["project_id"],
|
||
"worker_goal_resume_fallback",
|
||
"%s: goal 已存在，走断点续跑" % task_id)
|
||
_resume_task = dict(task)
|
||
_resume_task["_resume"] = True
|
||
_resume_task["_resume_kind"] = "breakpoint"
|
||
result, error = self._run_attempt(
|
||
_resume_task, quota_check=_quota_hit)
|
||
usage_total = sum_usage(
|
||
usage_total, (result or {}).get("usage"))
|
||
self._deliver_peer_messages(task, result)
|
||
self._deliver_marker_tools(task, result)
|
||
if (result or {}).get("status") == "quota_exceeded":
|
||
quota_error = self._quota_exceeded(
|
||
task, usage_total) or \
|
||
"quota_exceeded: 本月配额已用完"
|
||
quota_stop = True
|
||
break
|
||
if result.get("status") == "completed":
|
||
break
|
||
host_status_try = self._fallback_token(result, error)
|
||
if host_status_try is None and \
|
||
result.get("status") == "timeout":
|
||
# 进程/容器超时路径的 host_status 为空，超时本身就是
|
||
# 可 fallback 的失败 token（models.FALLBACKABLE 含它）。
|
||
host_status_try = "timeout"
|
||
if fallback_model is None and self.fallback_models \
|
||
and host_status_try is not None \
|
||
and is_fallbackable(host_status_try):
|
||
fb = self.fallback_models[0]
|
||
record = dict(task)
|
||
record["events"] = events
|
||
# record_fallback 本身会 append 到 events，
|
||
# 这里不要再 append 一次，否则事件会重复。
|
||
record_fallback(
|
||
record, self.model, fb, str(host_status_try))
|
||
fallback_count = 1
|
||
self._audit(task["project_id"], "worker_fallback",
|
||
"%s -> %s" % (self.model, fb))
|
||
first_error = self._attempt_error(
|
||
result, error, host_status_try)
|
||
fallback_model = fb
|
||
actual_model = fb
|
||
task = dict(task)
|
||
task["model"] = fb
|
||
# fallback 边界：换模型再跑一轮前先查配额，超限
|
||
# 时不烧第二份钱，直接走配额中断收尾。
|
||
quota_error = self._quota_exceeded(task, usage_total)
|
||
if quota_error is not None:
|
||
quota_stop = True
|
||
break
|
||
result, error = self._run_attempt(task, quota_check=_quota_hit)
|
||
usage_total = sum_usage(usage_total,
|
||
(result or {}).get("usage"))
|
||
self._deliver_peer_messages(task, result)
|
||
self._deliver_marker_tools(task, result)
|
||
if (result or {}).get("status") == "quota_exceeded":
|
||
quota_error = self._quota_exceeded(task, usage_total) or \
|
||
"quota_exceeded: 本月配额已用完"
|
||
quota_stop = True
|
||
break
|
||
if self.router is not None and host_status_try is not None \
|
||
and is_fallbackable(host_status_try) and \
|
||
self.router.next_after(
|
||
self.model, host_status_try) is not None:
|
||
break
|
||
if attempt < self.max_retries:
|
||
retry_count += 1
|
||
self._audit(
|
||
task["project_id"], "worker_retry",
|
||
"%s: attempt %d failed, retry %d/%d"
|
||
% (task_id, attempt + 1, retry_count,
|
||
self.max_retries))
|
||
if self.retry_backoff_seconds:
|
||
time.sleep(self.retry_backoff_seconds)
|
||
continue
|
||
break
|
||
# 超时转 failed：超时是失败，不是取消；最终状态必须是
|
||
# VALID_STATUSES 里的 failed，绝不能把 "timeout" 写进库。
|
||
if result is not None and result.get("status") == "timeout":
|
||
result = dict(result, status="failed")
|
||
error = "task timeout after %s seconds" % self.task_timeout
|
||
self._audit(task["project_id"], "worker_timeout", task_id)
|
||
|
||
# 后续消息（排队/直接发送）：首轮 completed 且非 goal 时，
|
||
# 按 direct 优先、队内先来先服务逐条再跑一轮（同一 thread，
|
||
# 历史与 worktree 改动都在）。直接发送在 POST 时已进链路，
|
||
# 这里只负责把轮次跑出来。一条失败即停：任务记 failed，
|
||
# 剩余 pending 留给重试入口（重试走已存在续跑，跑完继续
|
||
# 排空剩余消息）；取消时该条回 pending，任务走中断收尾。
|
||
if (result is not None and result.get("status") == "completed"
|
||
and not quota_stop
|
||
and not task.get("verify_command")
|
||
and not task.get("goal_mode")):
|
||
while True:
|
||
fu = self._claim_next_followup(task_id)
|
||
if fu is None:
|
||
break
|
||
# 后续消息执行时查配额：超限时该条回 pending（留
|
||
# 给重试入口），任务走配额中断收尾，不再排空剩余；
|
||
# 建消息时不查（排队本身不花钱，顶上来后可跑）。
|
||
quota_error = self._quota_exceeded(task, usage_total)
|
||
if quota_error is not None:
|
||
self._restore_followup(fu["id"])
|
||
quota_stop = True
|
||
break
|
||
fu_task = dict(task)
|
||
# 任务间消息的接收方提示：带发送方 id，接收方
|
||
# agent 凭此知道回给哪个任务；用户后续消息无前缀。
|
||
fu_prompt = fu["body"]
|
||
if fu.get("sender_task_id"):
|
||
fu_prompt = _peer_prefix(fu["sender_task_id"]) + fu_prompt
|
||
fu_task["prompt"] = fu_prompt
|
||
fu_result, fu_error = self._run_attempt(
|
||
fu_task, quota_check=_quota_hit)
|
||
usage_total = sum_usage(
|
||
usage_total, (fu_result or {}).get("usage"))
|
||
self._deliver_peer_messages(task, fu_result)
|
||
self._deliver_marker_tools(task, fu_result)
|
||
if (fu_result or {}).get("status") == "quota_exceeded":
|
||
self._restore_followup(fu["id"])
|
||
quota_error = self._quota_exceeded(task, usage_total) or \
|
||
"quota_exceeded: 本月配额已用完"
|
||
quota_stop = True
|
||
break
|
||
if self._was_cancelled(task_id) or \
|
||
(fu_result or {}).get("status") == "cancelled":
|
||
self._restore_followup(fu["id"])
|
||
result = fu_result
|
||
break
|
||
if (fu_result or {}).get("status") == "timeout":
|
||
fu_result = dict(fu_result, status="failed")
|
||
fu_error = ("task timeout after %s seconds"
|
||
% self.task_timeout)
|
||
if (fu_result or {}).get("status") == "completed":
|
||
self._finish_followup(fu["id"], "done")
|
||
result = fu_result
|
||
error = None
|
||
continue
|
||
self._finish_followup(fu["id"], "failed")
|
||
result = fu_result
|
||
error = fu_error or self._attempt_error(
|
||
fu_result, None, fu_result.get("host_status"))
|
||
break
|
||
|
||
# 取消优先：执行期间被 cancel 的任务保持 interrupted，
|
||
# 绝不用执行结果覆盖它（见模块 docstring 的取消说明）。
|
||
if self._was_cancelled(task_id) or result.get("status") == "cancelled":
|
||
self._set_status(task_id, "interrupted")
|
||
self._write_execution_result(
|
||
task_id, result_text=(result or {}).get("stdout", ""),
|
||
error_text="cancelled", actual_model=actual_model,
|
||
started_at=started_at, finished_at=time.time(),
|
||
turn_id=(result or {}).get("turn_id"),
|
||
root_turn_id=(result or {}).get("root_turn_id"),
|
||
only_if_status="interrupted",
|
||
result_full=(result or {}).get("stdout_full"))
|
||
# P1：取消分支先记一条成本再返回（与配额中断收尾
|
||
# 同款记账点）。旧版这里直接返回 cost_record_id=None，
|
||
# 被取消前已花的模型费用就从账上丢了；取消不等于没
|
||
# 花钱。记账失败不挡取消返回（costs.record 异常时
|
||
# cost_id 置 None，任务状态仍是 interrupted）。
|
||
try:
|
||
with self._db_lock:
|
||
_cancel_cost_id = self._record_usage_cost(
|
||
task, actual_model, usage_total,
|
||
(result or {}).get("stdout", ""), fallback_count)
|
||
except Exception:
|
||
_cancel_cost_id = None
|
||
log_event(self.logger, "worker_cancelled", task_id=task_id,
|
||
project_id=task["project_id"], status="interrupted")
|
||
return {"dispatched": True, "task_id": task_id,
|
||
"host_name": self.executor._host_name(task)
|
||
if hasattr(self.executor, "_host_name") else task_id,
|
||
"status": "interrupted", "executor": self._executor_name(),
|
||
"stdout": (result or {}).get("stdout", ""),
|
||
"error": "cancelled", "events": [],
|
||
"cost_record_id": _cancel_cost_id}
|
||
|
||
stdout = result.get("stdout", "") or ""
|
||
host_status = result.get("host_status")
|
||
final = result["status"]
|
||
if quota_stop:
|
||
# 配额中断收尾：状态一律 interrupted + 配额原因，不跑
|
||
# 失败原因重算（否则 fallback 拼接会把配额原因盖掉），
|
||
# 已花成本仍按 usage_total 照记（下面成本块），续跑
|
||
# 消息保留（下面清消息时跳过），不发终态推送（与取消
|
||
# 同款：interrupted 不在推送事件集里）。quota_exceeded
|
||
# 这个内部标记绝不能原样写进 tasks.status。
|
||
final = "interrupted"
|
||
error = quota_error
|
||
if final != "completed" and not quota_stop:
|
||
if fallback_model is not None:
|
||
# 真 fallback 后第二次也失败：错误文本必须同时写清
|
||
# 两次原因，不能只留第二次的（超时转换已在前面把
|
||
# error 置成超时文本，那时 second_error 就是它）。
|
||
second_error = error or self._attempt_error(
|
||
result, None, host_status)
|
||
error = ("primary model %s failed: %s; "
|
||
"fallback model %s failed: %s"
|
||
% (self.model, first_error,
|
||
fallback_model, second_error))
|
||
else:
|
||
if result.get("goal_error"):
|
||
error = result["goal_error"]
|
||
if error is None:
|
||
stderr = (result or {}).get("stderr", "") or ""
|
||
error = stdout.strip() or \
|
||
(stderr.strip()[-1000:] if stderr.strip() else "") \
|
||
or "host execution failed"
|
||
if retry_count:
|
||
error = "[retries=%d] %s" % (retry_count, error)
|
||
# 未配 fallback_models 时的老行为：只记事件、不重试。
|
||
# 注意用提取出的 token（真实回合失败时 host_status
|
||
# 只是 "failed"，真正的 404 藏在回合 JSON 里），
|
||
# 不能直接用原始 host_status 判断。
|
||
_token = self._fallback_token(result, error)
|
||
if self.router is not None and _token is not None \
|
||
and is_fallbackable(_token):
|
||
nxt = self.router.next_after(self.model, _token)
|
||
if nxt is not None:
|
||
record = dict(task)
|
||
record["events"] = events
|
||
# record_fallback 本身会 append 到 events，
|
||
# 这里不要再 append 一次，否则事件会重复。
|
||
record_fallback(
|
||
record, self.model, nxt, str(_token))
|
||
fallback_count = 1
|
||
self._audit(task["project_id"], "worker_fallback",
|
||
"%s -> %s" % (self.model, nxt))
|
||
final_written = self._set_status_if_running(task_id, final)
|
||
if quota_stop and final_written and final == "interrupted":
|
||
self._mark_interrupted_reason(task_id, "quota")
|
||
finished_at = time.time()
|
||
# P0 安全批后：CAS 没改到行（任务已被取消/回收）时
|
||
# 绝不写回结果，否则迟到的旧进程会覆盖新一轮的结果
|
||
# 文本；CAS 成功时写回也要带状态守卫。
|
||
if final_written:
|
||
self._write_execution_result(
|
||
task_id, result_text=stdout, error_text=error,
|
||
actual_model=actual_model, started_at=started_at,
|
||
finished_at=finished_at,
|
||
turn_id=(result or {}).get("turn_id"),
|
||
root_turn_id=(result or {}).get("root_turn_id"),
|
||
only_if_status=final,
|
||
result_full=(result or {}).get("stdout_full"))
|
||
# 附件暂存清理：任务终局后删 bundle 目录（重试走已存在
|
||
# 续跑，文件已在 worktree 里，不再需要暂存）。
|
||
self._cleanup_task_seed(task)
|
||
with self._db_lock:
|
||
self.usage_turns += 1
|
||
# 成本记账走 _record_usage_cost（终局与取消分支共用
|
||
# 的唯一记账点）：actual/estimated 分流、usage_cost_usd
|
||
# 累加都在里面；cost 记录共用一条连接，并发下必须
|
||
# 持锁写，否则两个线程共用同一 sqlite 连接会直接报错。
|
||
cost_id = self._record_usage_cost(
|
||
task, actual_model, usage_total, stdout, fallback_count)
|
||
if final_written:
|
||
# 审计名须与终态一致：interrupted 任务记 worker_failed
|
||
# 会把“可续跑的中断”误读成失败，误导后续排查。
|
||
# 终态审计细节（2026-10-29 任务 A 教训）：interrupted /
|
||
# failed 时只记 task_id，事后完全无法区分是取消、
|
||
# 配额、host 返回 interrupted 还是其它失败——且自动
|
||
# 续跑会把 error_text 覆盖掉，第一轮的原因就此丢失。
|
||
# 因此非完成终态的审计 detail 带上 error（截 500 字、
|
||
# 去换行，不含 prompt/key），完成态仍只记 task_id。
|
||
_final_detail = task_id
|
||
if final in ("interrupted", "failed") and error:
|
||
_final_detail = "%s: %s" % (
|
||
task_id, str(error)[:500].replace("\n", " "))
|
||
self._audit(task["project_id"],
|
||
"worker_quota_stop" if quota_stop else
|
||
("worker_completed" if final == "completed"
|
||
else "worker_interrupted"
|
||
if final == "interrupted"
|
||
else "worker_failed"), _final_detail)
|
||
log_event(self.logger,
|
||
"worker_quota_stop" if quota_stop else
|
||
("worker_completed" if final == "completed"
|
||
else "worker_interrupted"
|
||
if final == "interrupted"
|
||
else "worker_failed"),
|
||
task_id=task_id, project_id=task["project_id"],
|
||
status=final)
|
||
# P0 三件其三：时间线终局步骤（best-effort）。
|
||
try:
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
self._ensure_result_columns(conn)
|
||
self._record_step(conn, task_id, "final",
|
||
"终态 %s" % final)
|
||
conn.commit()
|
||
finally:
|
||
conn.close()
|
||
except Exception:
|
||
pass
|
||
if result.get("goal_met") is False:
|
||
# goal 未达只发 goal-unmet，不再重复发 failed：
|
||
# 同一结局两条通知会刷屏，digest 偏好下两者又都是
|
||
# 即时事件，更不能双发。
|
||
self._notify_task_finished(
|
||
task_id, task["project_id"], "goal-unmet", error=error)
|
||
elif final in ("completed", "failed"):
|
||
self._notify_task_finished(
|
||
task_id, task["project_id"], final, error=error)
|
||
else:
|
||
# 已被取消/回收：DB 状态以 interrupted 为准，不再
|
||
# 补一条 worker_completed/failed 审计误导排查。
|
||
final = "interrupted"
|
||
self._audit(task["project_id"], "worker_interrupted", task_id)
|
||
if resume_message and not quota_stop:
|
||
# 本轮已跑完（无论成败）：续跑消息已消费，清掉；进程
|
||
# 中途被杀时清不到，下次认领会重跑同一条消息——可接受
|
||
# 的续跑语义，不会静默丢消息。清失败不挡结果返回。
|
||
# 配额中断时不消费：顶上来后续跑还要说这句话。
|
||
try:
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
conn.execute(
|
||
"UPDATE tasks SET resume_message = NULL"
|
||
" WHERE id = ?", (task_id,))
|
||
conn.commit()
|
||
finally:
|
||
conn.close()
|
||
except Exception:
|
||
pass
|
||
return {"dispatched": True, "task_id": task_id,
|
||
"host_name": self.executor._host_name(task)
|
||
if hasattr(self.executor, "_host_name") else task_id,
|
||
"status": final, "executor": self._executor_name(),
|
||
"stdout": stdout, "error": error, "events": events,
|
||
"actual_model": actual_model,
|
||
"cost_record_id": cost_id}
|
||
|
||
def _run_pr_jobs(self):
|
||
"""跑本主机的 pr 任务（HA 阶段 1），返回处理结果列表。
|
||
|
||
只有 local worker 跑：host-worktrees/ 与 tenants/ 在主机上
|
||
都可见，容器内 worker 看不到 host-worktrees/。每轮把本主机
|
||
的 queued 任务按 CAS 认领跑完（pr 任务稀少，串行即可）；
|
||
execute_task_pr 内部已写回任务 pr_url/pr_number、记审计、
|
||
发 review-requested/merge-failed，这里只写回任务终态。
|
||
跳过（PrSkipped，如任务已不是 merged）与落盘失败都记
|
||
failed + error，任务本身保持 merged、可重调 /pr 再排队。
|
||
P1 非 local 通道：旧版非 local worker 直接返回 []，
|
||
排队的 pr 任务看起来像“没人处理”且无任何痕迹——
|
||
静默未完成。现在改为显式 needs_local_worker：只读
|
||
peek 本主机 queued 的 pr 任务，逐条把行状态持久化
|
||
为 needs_local_worker（CAS，只改仍是 queued 的行，
|
||
已被 local worker 认领的不覆盖），并记审计 + 日志
|
||
返回 needs_local_worker 条目。任务本身保持 merged，
|
||
pr 任务保持可被 local worker 认领（claim_pr_job
|
||
同时认 queued 与 needs_local_worker），绝不把它标
|
||
failed（failed 会让调用方以为 PR 已失败，且任务仍
|
||
可重调 /pr 的语义会被混淆）。签名无效时同样不碰
|
||
任务（返回 []，只记日志），fail-closed。
|
||
"""
|
||
if not self._code_signature_valid():
|
||
log_event(self.logger, "worker_pr_refused_code_signature",
|
||
message="code signature missing or changed,"
|
||
" refusing pr jobs")
|
||
return []
|
||
if self._executor_name() != "local":
|
||
results = []
|
||
try:
|
||
from cloud.pr_exec import ensure_pr_jobs_table
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
ensure_pr_jobs_table(conn)
|
||
rows = conn.execute(
|
||
"SELECT id, task_id, project_id FROM pr_jobs"
|
||
" WHERE host = ? AND status = 'queued'",
|
||
(self.host_name,)).fetchall()
|
||
finally:
|
||
conn.close()
|
||
for row in rows:
|
||
try:
|
||
job_id = row["id"]
|
||
task_id = row["task_id"]
|
||
project_id = row["project_id"]
|
||
except (KeyError, IndexError, TypeError):
|
||
continue
|
||
reason = ("PR 执行需要 local worker（host-worktrees"
|
||
" 只在主机上可见），本 worker 为 %s，"
|
||
"任务保持可认领等 local worker 认领"
|
||
% self._executor_name())
|
||
marked = False
|
||
try:
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
marked = mark_pr_job_needs_local_worker(
|
||
conn, job_id, reason)
|
||
finally:
|
||
conn.close()
|
||
except Exception as exc:
|
||
log_event(self.logger, "pr_job_mark_failed",
|
||
message="pr job mark failed: %s" % exc)
|
||
continue
|
||
if not marked:
|
||
# 已被 local worker 认领或已终态：不覆盖、不重复记。
|
||
continue
|
||
self._audit(project_id, "pr_job_needs_local_worker",
|
||
"%s: %s" % (job_id, reason))
|
||
log_event(self.logger, "pr_job_needs_local_worker",
|
||
task_id=task_id, status="needs_local_worker",
|
||
message=reason)
|
||
results.append({"pr_job_id": job_id,
|
||
"task_id": task_id,
|
||
"status": "needs_local_worker",
|
||
"error": reason})
|
||
except Exception as exc:
|
||
log_event(self.logger, "pr_job_peek_failed",
|
||
message="pr job peek failed: %s" % exc)
|
||
return results
|
||
results = []
|
||
while True:
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
job = claim_pr_job(conn, self.host_name)
|
||
finally:
|
||
conn.close()
|
||
if job is None:
|
||
break
|
||
job_id = job["id"]
|
||
task_id = job["task_id"]
|
||
try:
|
||
# G13：审计操作人用 job 的费用归属人（请求 /pr 的
|
||
# 用户），老行没有 created_by 时回落 "system"。
|
||
result = execute_task_pr(
|
||
self.db_path, task_id,
|
||
job.get("created_by") or "system")
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
finish_pr_job(conn, job_id, "completed",
|
||
result["pr_url"], result["pr_number"])
|
||
finally:
|
||
conn.close()
|
||
results.append({"pr_job_id": job_id, "task_id": task_id,
|
||
"status": "completed",
|
||
"pr_url": result["pr_url"]})
|
||
log_event(self.logger, "pr_job_completed",
|
||
task_id=task_id, status="completed")
|
||
except PrSkipped as exc:
|
||
err = "skipped: %s" % exc.reason
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
finish_pr_job(conn, job_id, "failed", error=err)
|
||
finally:
|
||
conn.close()
|
||
results.append({"pr_job_id": job_id, "task_id": task_id,
|
||
"status": "failed", "error": err})
|
||
except PrStepError as exc:
|
||
err = "%s: %s" % (exc.step, exc.message)
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
finish_pr_job(conn, job_id, "failed", error=err)
|
||
finally:
|
||
conn.close()
|
||
results.append({"pr_job_id": job_id, "task_id": task_id,
|
||
"status": "failed", "error": err})
|
||
except Exception as exc: # noqa: BLE001 -- 任务失败不能炸循环
|
||
err = str(exc)
|
||
conn = _db_connect(self.db_path)
|
||
try:
|
||
finish_pr_job(conn, job_id, "failed", error=err)
|
||
finally:
|
||
conn.close()
|
||
results.append({"pr_job_id": job_id, "task_id": task_id,
|
||
"status": "failed", "error": err})
|
||
return results
|
||
|
||
def run_all(self, max_tasks=None):
|
||
"""循环认领执行，直到无就绪任务、预算超限或达到 max_tasks。
|
||
|
||
每轮都重新读库，所以上一轮刚 completed 的任务会解锁它的
|
||
依赖任务；每轮至少推进一个任务到终态，不会空转死循环。
|
||
|
||
capacity<=1 时走旧串行循环（行为与原来完全一致）。
|
||
capacity>1 时用线程池并行跑：同一 worker 内最多
|
||
capacity 个任务同时执行（capacity 从此是真实并发上限，
|
||
不再只是注册表展示）。认领本身是 CAS，两个线程同时
|
||
盯上同一任务时只有一个能拿到它，另一个的 run_one 返回
|
||
already_claimed，不计入结果、不死循环。依赖解锁靠
|
||
“等一个在跑任务完成后再认领”：pending 非空但暂无就绪
|
||
任务时，等一个完成再继续，而不是直接退出。
|
||
停机信号（_loop_stop/stop_event）只阻止新认领，不会
|
||
丢弃已提交的任务；排空等待由 run_loop 的 finally 负责。
|
||
"""
|
||
# 环境准备先跑：每轮先把 creating 的环境准备好（clone +
|
||
# checkout 分支），再认领普通任务；一条环境失败只标它
|
||
# 自己 failed，不挡任务——准备函数内部已隔离异常，这里
|
||
# 再包一层，防 DB 结构异常（如老库缺表）带走整轮任务。
|
||
try:
|
||
self._prepare_pending_environments()
|
||
except Exception as exc:
|
||
log_event(self.logger, "worker_env_prepare_failed",
|
||
message="worker env prepare failed: %s" % exc)
|
||
# pr 任务先跑（HA 阶段 1）：每轮先把本主机的 queued pr 任务
|
||
# 跑完再认领普通任务；pr_jobs_only 时只跑 pr 任务。
|
||
self._run_pr_jobs()
|
||
if self.pr_jobs_only:
|
||
return []
|
||
if self.capacity <= 1:
|
||
results = []
|
||
while max_tasks is None or len(results) < max_tasks:
|
||
result = self.run_one()
|
||
if not result["dispatched"]:
|
||
break
|
||
results.append(result)
|
||
return results
|
||
# capacity>1：手动线程池（不用 concurrent.futures）。
|
||
# 原因：本仓库有个单测把 cloud/ 插进 sys.path 后
|
||
# `import queue` 会解析到 cloud/queue.py 而不是标准库，
|
||
# concurrent.futures 内部依赖标准库 queue.SimpleQueue，
|
||
# 在该单测之后导入它会直接 AttributeError；手动用
|
||
# threading.Condition + 共享计数实现线程池，避开这个坑。
|
||
# 池线程数按队列深度自适应：每轮开始先数就绪任务，
|
||
# 线程数 = min(capacity, 就绪数)——队列只有 1 个任务时
|
||
# 只开 1 个线程，不为空转线程池买单；队列深时开满
|
||
# capacity。就绪数是快照，任务完成解锁依赖时，池线程
|
||
# 靠“无就绪但仍有在跑任务则等待重试”继续接新任务，
|
||
# 不会因为初始快照小就漏派。
|
||
# 池线程数 = capacity；每个池线程循环认领-执行，认领
|
||
# 本身是 CAS，两个线程盯上同一任务时只有一个能拿到。
|
||
# 依赖解锁：某线程暂无就绪任务时，若仍有线程在跑就等
|
||
# 条件变量（跑完的线程会 notify，可能解锁了依赖），
|
||
# 全部线程都空闲且无就绪任务时才退出。
|
||
# max_tasks 约束：已派发结果数 + 正在跑的任务数达到
|
||
# max_tasks 就不再认领，避免线程池提前提交导致超跑。
|
||
# 停机信号：只阻止新认领，不丢弃已在跑的任务；run_all
|
||
# 在停机时立刻返回（后台线程继续跑），run_loop 的
|
||
# finally 做带超时的排空，超时后才杀残留。
|
||
state_lock = threading.Condition()
|
||
shared = {"dispatched": 0, "active": 0, "stopped": False}
|
||
results = []
|
||
results_lock = threading.Lock()
|
||
|
||
def _pool_thread():
|
||
while True:
|
||
with state_lock:
|
||
if shared["stopped"] or self._loop_stop:
|
||
return
|
||
if max_tasks is not None and \
|
||
shared["dispatched"] + shared["active"] >= max_tasks:
|
||
return
|
||
try:
|
||
task = self.pick_next()
|
||
except Exception as exc:
|
||
# 认领读库失败（如缺表、锁冲突）不炸池线程：
|
||
# 等一轮再试，与 run_loop 的容错一致。
|
||
log_event(self.logger, "worker_pool_pick_error",
|
||
message="worker pool pick error: %s" % exc)
|
||
state_lock.wait(timeout=0.2)
|
||
continue
|
||
if task is None:
|
||
if shared["active"] > 0:
|
||
state_lock.wait(timeout=0.2)
|
||
continue
|
||
return
|
||
shared["active"] += 1
|
||
result = {"dispatched": False}
|
||
try:
|
||
result = self.run_one(task)
|
||
except Exception as exc:
|
||
# 单个任务的意外异常绝不能拖垮整个池（串行
|
||
# 模式下 run_loop 也是这么容错的）；该任务已
|
||
# 认领为 running，之后由自动陈旧回收处理。
|
||
log_event(self.logger, "worker_pool_task_error",
|
||
message="worker pool task error: %s" % exc)
|
||
finally:
|
||
with state_lock:
|
||
shared["active"] -= 1
|
||
if result["dispatched"]:
|
||
shared["dispatched"] += 1
|
||
state_lock.notify_all()
|
||
if result["dispatched"]:
|
||
with results_lock:
|
||
results.append(result)
|
||
|
||
# 池线程数恒为 capacity，绝不按本轮开始时的就绪快照缩减。
|
||
# 快照缩减是 PM 等待死锁的根因：本轮开始时若只有 1 个就绪
|
||
# 任务（如刚认领的 PM），只开 1 个线程；该线程跑 PM 模型回合
|
||
# 后进入标记等待（最长 3600 秒），期间 PM 新建/重试的子任务
|
||
# 入队，却再无空闲线程认领它——run_loop 又被本轮 run_all 的
|
||
# join 挡住、开不了新一轮；若此时另一 worker 已缩容，就只剩
|
||
# 一个被等待占住的 worker，子任务等满等待超时也无人跑。
|
||
# 恒开 capacity 个线程没有空转代价：空闲线程在 task 为 None
|
||
# 且 active 为 0 时立刻退出，有在跑任务时才 0.2 秒轮询认领，
|
||
# 正好接住轮中新建的任务。
|
||
threads = [threading.Thread(target=_pool_thread, daemon=True)
|
||
for _ in range(self.capacity)]
|
||
for t in threads:
|
||
t.start()
|
||
for t in threads:
|
||
# 正常情况等全部池线程结束；停机信号来时不等了，
|
||
# 立刻返回给 run_loop 做排空（后台线程是 daemon，
|
||
# 且 run_loop finally 会杀残留，不会悬着进程）。
|
||
while t.is_alive():
|
||
t.join(timeout=0.2)
|
||
if self._loop_stop:
|
||
break
|
||
if self._loop_stop:
|
||
break
|
||
return results
|
||
|
||
def _process_push_due(self, now=None):
|
||
"""G11/G12 后台投递：每轮循环调用，绝不中断在跑任务。
|
||
|
||
outbox 每轮都试到期的行（重试节奏由 next_attempt_at
|
||
控，默认 60 秒退避、最多 3 次，到上限后删行并记日志，
|
||
不丢任务状态）；摘要只在距上次发送满 digest_interval
|
||
后才发一次（默认每小时），避免每轮都把摘要提前发掉。
|
||
全部异常只记日志，不抛、不炸循环。force 语义由调用方
|
||
通过把 _last_digest_sent 置 0 实现（单测用）。
|
||
"""
|
||
now = time.time() if now is None else now
|
||
try:
|
||
from cloud.push import process_outbox_due, send_digest_due
|
||
out = process_outbox_due(
|
||
self.db_path, now=now,
|
||
max_attempts=self.outbox_max_attempts,
|
||
backoff_seconds=self.outbox_backoff_seconds)
|
||
for failed in out.get("failed", []):
|
||
log_event(self.logger, "worker_push_outbox_dropped",
|
||
message="push outbox dropped after retries: %s"
|
||
% failed)
|
||
if now - self._last_digest_sent >= \
|
||
self.digest_interval_seconds:
|
||
self._last_digest_sent = now
|
||
digest = send_digest_due(self.db_path, now=now)
|
||
for err in digest.get("errors", []):
|
||
log_event(self.logger, "worker_push_digest_failed",
|
||
message="push digest send failed: %s" % err)
|
||
except Exception as exc:
|
||
log_event(self.logger, "worker_push_due_failed",
|
||
message="push due processing failed: %s" % exc)
|
||
|
||
def run_loop(self, interval=300, max_seconds=None, stop_event=None,
|
||
drain_timeout=60, sweep_threshold=300):
|
||
"""常驻循环：每 interval 秒跑一轮 run_all，空闲也不退出。
|
||
|
||
退出条件：stop_event 被置位、_loop_stop 被信号处理置位、
|
||
或 max_seconds 到期。退出时先优雅排空：等正在跑的任务
|
||
自然结束（最多 drain_timeout 秒），超时后才杀残留进程组/
|
||
容器并标 interrupted——以前是退出时立刻杀，部署重启会
|
||
无谓中断一个 16 秒就能跑完的任务。
|
||
|
||
运行中保障（与并发一起上的两件套）：
|
||
- 心跳线程：每 30 秒给所有正在跑的任务刷 last_heartbeat，
|
||
否则长任务（>300 秒）在跑也会被陈旧回收误杀。
|
||
- 自动陈旧回收：每轮结束调一次 sweep_stale_tasks（阈值
|
||
sweep_threshold，默认 300 秒），把死 worker 遗留的
|
||
running 任务标 interrupted 并审计；活任务因心跳新鲜
|
||
绝不会被扫到。回收失败只记日志，不炸循环。
|
||
"""
|
||
# 常驻启动清理兜底：清掉上一轮残留的 codex- 容器，
|
||
# 失败只记日志（见 cleanup_stale_containers），不炸循环。
|
||
self.cleanup_stale_containers()
|
||
# 运维心跳：启动时先写一次，之后每轮结束再写，server 的
|
||
# /health 据此算 worker_ok（见 server.py 的健康检查）。
|
||
self._write_heartbeat()
|
||
# 多 worker 池注册：启动时注册一次，之后每轮结束写注册表
|
||
# 心跳（见本轮结束处）；注册失败只记日志，不炸循环。
|
||
self.register()
|
||
started = time.time()
|
||
all_results = []
|
||
# 心跳线程：daemon，run_loop 退出即停，不阻塞进程退出。
|
||
hb_stop = threading.Event()
|
||
|
||
def _hb_loop():
|
||
while not hb_stop.is_set() and not self._loop_stop:
|
||
if stop_event is not None and stop_event.is_set():
|
||
break
|
||
self._heartbeat_running_tasks()
|
||
# 注册表心跳也每 30 秒刷一次：长任务跑满一轮时
|
||
# run_all 久久不返回，只靠轮次结束的心跳会让
|
||
# /workers 的 running 数长期停在旧值；这里刷的
|
||
# running 含启动中占位，与认领门口径一致。
|
||
self._registry_heartbeat()
|
||
hb_stop.wait(30)
|
||
|
||
hb_thread = threading.Thread(target=_hb_loop, daemon=True)
|
||
hb_thread.start()
|
||
try:
|
||
while not self._loop_stop:
|
||
if stop_event is not None and stop_event.is_set():
|
||
break
|
||
if max_seconds is not None and \
|
||
time.time() - started >= max_seconds:
|
||
break
|
||
# 代码变化自退：部署换目录后，旧代码 worker 若还抢
|
||
# 任务会污染生产（2026-10-27 实弹：05:51 的旧
|
||
# docker worker 成孤儿后继续用旧代码抢任务、按老路
|
||
# 失败且不删暂存）。检查点在轮次边界，绝不中断在跑
|
||
# 任务；退出码 75（EX_TEMPFAIL，失败类）：unit 托管
|
||
# 的由 systemd on-failure 重拉新版，autoscaler 托管
|
||
# 的由 reconcile 重拉，孤儿直接消失。单次 run_all
|
||
# 不走这里（进程本来就退出）。P1 fail-closed：
|
||
# 签名缺失同样自退（旧版只在“变了”时退，取不到
|
||
# 签名时继续跑是 fail-open）；每次认领前的签名门
|
||
# 在 _claim 里（池线程不走轮次边界也拦得住）。
|
||
if self._code_changed() or \
|
||
not self._code_signature_valid():
|
||
log_event(self.logger, "worker_code_changed_exit",
|
||
message="worker code changed, exiting "
|
||
"for restart")
|
||
raise SystemExit(75)
|
||
try:
|
||
results = self.run_all()
|
||
except Exception as exc:
|
||
# 常驻循环一轮炸了不带走整个 worker：记结构化日志，
|
||
# 睡完本轮间隔再试下一轮（缺表、锁冲突都属此类）。
|
||
log_event(self.logger, "worker_loop_error",
|
||
message="worker_loop_error: %s" % exc)
|
||
results = []
|
||
all_results.extend(results)
|
||
# G11/G12 后台投递：摘要（每小时）与 outbox 重试，
|
||
# 轮次边界做，不中断在跑任务，失败只记日志。
|
||
self._process_push_due()
|
||
# 自动记忆提取：同款轮次边界后台扫，失败只记日志
|
||
# （process_auto_memories 内部已兜底，不炸循环）。
|
||
self.process_auto_memories()
|
||
# 自动陈旧回收：死 worker 的 running 任务在这里被
|
||
# 标 interrupted；活任务心跳新鲜，不会被扫到。
|
||
if sweep_threshold is not None:
|
||
try:
|
||
from cloud.server import sweep_stale_tasks
|
||
sweep_stale_tasks(self.db_path, sweep_threshold)
|
||
except Exception as exc:
|
||
log_event(self.logger, "worker_sweep_failed",
|
||
message="worker sweep failed: %s" % exc)
|
||
# P1 孤儿扫尾（接在陈旧回收之后）：协调员本身
|
||
# 被回收标 interrupted 后，它的未终态子任务
|
||
# 在这里被统一标 interrupted；协调员仍在跑时
|
||
# 本函数不动子任务，见 sweep 方法 docstring。
|
||
try:
|
||
self.sweep_orphaned_coordinator_children()
|
||
except Exception as exc:
|
||
log_event(self.logger, "worker_orphan_sweep_failed",
|
||
message="orphan sweep failed: %s" % exc)
|
||
# 跨 worker 真 kill 的收殓器：放在 sweep 块外、每轮
|
||
# 都跑（旧 PR 只在 sweep 块内跑，sweep 关掉时请
|
||
# 求会积压；本函数自带心跳停滞判定，sweep 关掉
|
||
# 也不抢杀活 owner）。陈旧回收刚把死 worker 的
|
||
# running 任务标 interrupted 并写了 kill 请求，
|
||
# 这里按请求杀本机遗留容器；owner 活着的任务心
|
||
# 跳新鲜，收殓器绝不抢杀。
|
||
try:
|
||
self._reap_kill_requests(
|
||
sweep_threshold
|
||
if sweep_threshold is not None else 300)
|
||
except Exception as exc:
|
||
log_event(self.logger, "worker_reap_failed",
|
||
message="worker reap failed: %s" % exc)
|
||
# 长任务自动恢复：接在 sweep 之后——刚被 sweep
|
||
# 标 interrupted 的系统中断 goal 任务，本轮即可
|
||
# 回 queued、下一轮认领即断点续跑；用户暂停、
|
||
# 配额、协调员孤儿一律不接（见方法 docstring）。
|
||
# 放在 sweep 块外：sweep 关掉时，已由其它路径
|
||
# （如 worker 退出）标 interrupted 的任务仍要恢复。
|
||
self.auto_resume_interrupted()
|
||
# 每轮结束写心跳：哪怕本轮一个任务都没派，worker 活着
|
||
# 这件事本身也要被 server 看到（文件心跳 + 注册表心跳）。
|
||
self._write_heartbeat()
|
||
self._registry_heartbeat()
|
||
# 按 0.2 秒切片睡，期间可响应 stop_event/_loop_stop。
|
||
deadline = time.time() + interval
|
||
while time.time() < deadline:
|
||
if self._loop_stop:
|
||
break
|
||
if stop_event is not None and stop_event.is_set():
|
||
break
|
||
if max_seconds is not None and \
|
||
time.time() - started >= max_seconds:
|
||
break
|
||
time.sleep(0.2)
|
||
finally:
|
||
hb_stop.set()
|
||
# 等心跳线程彻底退出再返回：否则它手头一次未写完的
|
||
# 注册表心跳可能在调用方清理 DB 目录时重建 control.db，
|
||
# 导致 tearDown rmtree 报 Directory not empty（全量套件
|
||
# 里 test_code_changed_exits_loop_before_claiming 偶发
|
||
# error 即此竞态）。daemon 线程 + set() 会唤醒 wait，
|
||
# 正常情况下 join 瞬间返回；5 秒超时兜底，避免写挂时
|
||
# 拖住退出。
|
||
hb_thread.join(timeout=5)
|
||
# 优雅排空：先等正在跑的任务自然结束，超时后才杀。
|
||
# 串行模式下 run_all 返回时任务已跑完，这里立刻通过；
|
||
# 并行模式下停机信号退出时后台线程可能还在跑，先等
|
||
# drain_timeout 秒，超时后按下面杀残留。
|
||
if drain_timeout is not None:
|
||
self._wait_for_running(timeout=drain_timeout)
|
||
for task_id, entry in self._running_snapshot().items():
|
||
if entry.get("proc") is not None:
|
||
self._kill_proc_group(entry["proc"])
|
||
try:
|
||
entry["proc"].wait(timeout=3)
|
||
except Exception:
|
||
pass
|
||
if entry.get("container") is not None and \
|
||
hasattr(self.executor, "kill_container"):
|
||
try:
|
||
self.executor.kill_container(entry["container"])
|
||
except Exception:
|
||
pass
|
||
self._running_clear(task_id)
|
||
self._set_status(task_id, "interrupted")
|
||
# worker 退出时的系统中断，可自动恢复。
|
||
self._mark_interrupted_reason(task_id, "worker_shutdown")
|
||
# 审计：此前这条路径只改状态不记审计，中断只能靠
|
||
# 重启时间 + 后续 auto_resumed 推断（2026-10-29
|
||
# 实弹）；记一条 worker_shutdown_interrupted，
|
||
# detail 为任务 id，与 sweep_interrupted 同口径。
|
||
try:
|
||
_conn = _db_connect(self.db_path)
|
||
try:
|
||
_row = _conn.execute(
|
||
"SELECT project_id FROM tasks WHERE id = ?",
|
||
(task_id,)).fetchone()
|
||
finally:
|
||
_conn.close()
|
||
_pid = _row[0] if _row is not None else ""
|
||
if _pid is None:
|
||
_pid = ""
|
||
except Exception:
|
||
_pid = ""
|
||
self._audit(_pid, "worker_shutdown_interrupted",
|
||
task_id)
|
||
return all_results
|
||
|
||
|
||
def _build_cli_executor(args, host_py):
|
||
"""CLI 的执行器构造：local 时返回 None（Worker 默认 LocalExecutor），
|
||
docker 时返回 DockerExecutor（env_key 名用 Worker 的解析规则，
|
||
值本身绝不从这里传——DockerExecutor 只查存在性，-e 只传名），
|
||
cloud 时返回 CloudExecutor（通用 HTTP runner，未配 endpoint
|
||
时执行即明确失败，绝不降级）。"""
|
||
if getattr(args, "executor", "local") == "cloud":
|
||
return CloudExecutor()
|
||
if getattr(args, "executor", "local") != "docker":
|
||
return None
|
||
return DockerExecutor(
|
||
host_py, config_path=args.config, image=args.docker_image,
|
||
cpus=args.docker_cpus, memory=args.docker_memory,
|
||
user=args.docker_user, worktrees_root=args.docker_worktrees_root,
|
||
env_key=resolve_env_key(args.env_key, args.codex_config),
|
||
timeout=args.docker_timeout,
|
||
extra_mounts=args.docker_extra_mounts,
|
||
security_opts=args.docker_security_opt,
|
||
pids_limit=args.docker_pids_limit,
|
||
runtime=getattr(args, "docker_runtime", None))
|
||
|
||
|
||
def main(argv=None):
|
||
"""worker CLI：默认单次 run_all；--loop 进常驻循环。
|
||
|
||
只打印任务结果摘要（状态、host 名、错误文本），绝不打印
|
||
任何环境变量的值。
|
||
"""
|
||
parser = argparse.ArgumentParser(description="cloud worker")
|
||
parser.add_argument("--db", default="cloud-control-plane.db")
|
||
parser.add_argument("--host-py", default=None,
|
||
help="host.py 路径，缺省为 cloud/ 上一级的 host.py")
|
||
parser.add_argument("--config", default=None)
|
||
parser.add_argument("--repo", default=None)
|
||
parser.add_argument("--heartbeat-file", default=None,
|
||
help="运维心跳文件路径（默认：环境变量 "
|
||
"CLOUD_WORKER_HEARTBEAT_FILE，未设则为 "
|
||
"./worker-heartbeat.json；主机部署时用此"
|
||
"参数指到 /var/lib/codex-app-server/ 下，"
|
||
"与 server 的 health 读取路径保持一致）")
|
||
parser.add_argument("--repos-dir", default="./repos",
|
||
help="项目仓库检出根目录（默认 ./repos）；项目有"
|
||
" repo_url 时 worker 在此 clone/fetch 各项目"
|
||
"仓库；Docker 执行器下它必须放在"
|
||
" --docker-extra-mounts 已挂载的目录下，"
|
||
"否则容器内 task repo 路径不存在")
|
||
parser.add_argument("--tenant-isolation", action="store_true",
|
||
help="多租户隔离（仅 Docker 执行器）：每个项目用"
|
||
"独立的 tenants_root/<项目>/work、codex-home、"
|
||
"test-repo 副本与项目独立 Docker 网络，容器"
|
||
"内路径不变，宿主挂载源按项目分开")
|
||
parser.add_argument("--tenants-root",
|
||
default="/var/lib/codex-app-server/tenants")
|
||
parser.add_argument("--tenant-template-home",
|
||
default="/var/lib/codex-app-server/codex-home",
|
||
help="租户 codex-home 播种模板：只取其中的"
|
||
"config.toml，绝不复制 thread_history")
|
||
parser.add_argument("--tenant-template-repo",
|
||
default="/var/lib/codex-app-server/test-repo",
|
||
help="租户 test-repo 播种模板（整目录复制）")
|
||
parser.add_argument("--env-key", default=None)
|
||
parser.add_argument("--codex-config", default=None,
|
||
help="含 env_key 行的 codex 配置文件（只读名字）")
|
||
parser.add_argument("--model", default=None)
|
||
parser.add_argument("--fallback-models", default=None,
|
||
help="真实 fallback 换模型列表，逗号分隔"
|
||
"（默认空 = 关闭，只记事件不换模型；"
|
||
"配了时主模型可 fallback 失败后最多换"
|
||
"列表里的第一个模型再跑一次，401/400"
|
||
"绝不 fallback，取消绝不 fallback）")
|
||
parser.add_argument("--max-turns", type=int, default=None)
|
||
parser.add_argument("--ignore-host-budget", dest="ignore_host_budget",
|
||
action="store_true", default=True,
|
||
help="G9（默认开）：worker 路径忽略 host 的 "
|
||
"max_daily_turns 预算，云端 per-user USD "
|
||
"配额 + worker 门才是口径")
|
||
parser.add_argument("--no-ignore-host-budget",
|
||
dest="ignore_host_budget", action="store_false",
|
||
help="G9：关闭忽略，worker 调 host.py 时 host "
|
||
"的 max_daily_turns 预算照常生效（双重门）")
|
||
parser.add_argument("--max-tasks", type=int, default=None)
|
||
parser.add_argument("--executor", default="local",
|
||
choices=["local", "docker", "cloud"],
|
||
help="执行器：local 调 host.py 子进程；docker"
|
||
" 调 DockerExecutor（无 docker 时任务"
|
||
"明确失败，不静默降级）；cloud 调通用"
|
||
" HTTP runner（需 CLOUD_RUNNER_ENDPOINT）")
|
||
parser.add_argument("--worker-id", default=None,
|
||
help="多 worker 池身份（默认：主机名-进程号；"
|
||
"生产部署应配稳定 id，否则重启后旧行变 stale）")
|
||
parser.add_argument("--worker-capacity", type=int, default=None,
|
||
help="worker 并发上限：同一 worker 内最多同时跑"
|
||
"几个任务（未给时读环境变量"
|
||
" CLOUD_WORKER_CAPACITY，未设则默认 1，"
|
||
"即旧串行行为；>1 时用线程池并行，认领"
|
||
"仍是 CAS，成本写库持锁）")
|
||
parser.add_argument("--host-name", default=None,
|
||
help="本 worker 所在主机名（HA 阶段 1）：认领任务"
|
||
"时写进 tasks.worktree_host，/pr 据此判断"
|
||
"worktree 在哪台主机；容器内 worker 必须由"
|
||
"autoscaler 显式传宿主名，不能用容器名")
|
||
parser.add_argument("--pr-jobs-only", action="store_true",
|
||
help="只跑 pr 任务（HA 阶段 1 的跨主机 /pr 演练"
|
||
"与专用 pr-runner 用）：不认领普通任务")
|
||
parser.add_argument("--docker-image", default="codex-app-server:latest")
|
||
parser.add_argument("--docker-cpus", default="2")
|
||
parser.add_argument("--docker-memory", default="2g")
|
||
parser.add_argument("--docker-user", default="1000:1000")
|
||
parser.add_argument("--docker-worktrees-root", default=None)
|
||
parser.add_argument("--docker-timeout", type=float, default=None)
|
||
parser.add_argument("--docker-extra-mounts", default=None,
|
||
help="docker 额外挂载，逗号分隔 host:container[:ro|rw]，"
|
||
"用于容器内 host.py 写 work/、host-state.json，"
|
||
"以及挂载原仓库和 codex-home（同路径挂载，"
|
||
"容器内配置无需改路径）")
|
||
parser.add_argument("--docker-security-opt", action="append",
|
||
default=None,
|
||
help="docker --security-opt，可重复（如"
|
||
" apparmor=docker-codex；no-new-privileges"
|
||
" 已硬编码，重复传也不重复）")
|
||
parser.add_argument("--docker-pids-limit", type=int, default=512,
|
||
help="docker --pids-limit（默认 512，防 fork 炸弹；"
|
||
"codex 工具链正常用量远低于此）")
|
||
parser.add_argument("--docker-runtime", default=None,
|
||
help="docker --runtime（不传 = docker 默认 runc；"
|
||
"传 runsc = 任务容器走 gVisor 沙箱，需 daemon"
|
||
"已注册该 runtime，否则起容器失败）")
|
||
parser.add_argument("--egress-proxy-port", type=int, default=0,
|
||
help="egress 过滤代理端口（0 = 不启用；生产配"
|
||
" 8888）：启用后 Docker 任务容器的 HTTP(S)"
|
||
" 出口全部走宿主该端口的代理，代理只放行"
|
||
"白名单域名；网关查不到时任务直接 failed，"
|
||
"绝不无代理运行")
|
||
parser.add_argument("--claim-executor-only", action="store_true",
|
||
help="严格 executor 认领：只认领 executor 恰好"
|
||
"等于本 worker 类型的任务，绝不认领"
|
||
" executor 为 NULL 的任务。宿主级沙箱"
|
||
" worker 必须开，否则它会抢走本该进"
|
||
" Docker 容器的 NULL 任务")
|
||
parser.add_argument("--websearch-mcp", action="store_true",
|
||
help="租户 codex-home 自动播种网页搜索 MCP："
|
||
"_tenant_dirs 在播种 config.toml 后追加"
|
||
" [mcp_servers.websearch] 并拷入"
|
||
" websearch.py / websearch_mcp.py；"
|
||
"默认关，老租户不受影响")
|
||
parser.add_argument("--seed-skills", action="store_true",
|
||
help="租户 codex-home 自动播种对齐技能包："
|
||
"_tenant_dirs 把 cloud/skills/ 的 SKILL.md"
|
||
" 拷进租户 codex-home/skills/；"
|
||
"默认关，老租户不受影响；租户改过的"
|
||
"技能文件绝不覆盖")
|
||
parser.add_argument("--imagegen-mcp", action="store_true",
|
||
help="租户 codex-home 自动播种图片生成 MCP："
|
||
"_tenant_dirs 追加 [mcp_servers.imagegen] 并拷入"
|
||
" imagegen.py / imagegen_mcp.py，_prepare_tenant"
|
||
" 写 [mcp_servers.imagegen.env]（系统级"
|
||
" IMAGE_GEN_API_KEY + 代理 + NO_PROXY）；"
|
||
"默认关，老租户不受影响；未配 key 时工具"
|
||
"明确报未配置，不回退任务自己的网关 key")
|
||
parser.add_argument("--task-timeout", type=float, default=1800,
|
||
help="每任务截止时间（秒，默认 1800）；0 表示不限时"
|
||
"（与 host --turn-timeout 0 同约定，worker 级"
|
||
"不设截止，但 Docker 路径仍受 --docker-timeout"
|
||
"约束）；超时后真杀容器/进程组、任务置 failed；"
|
||
"与 --docker-timeout（单次 docker run 超时）取"
|
||
"较小值，与 --max-seconds（run_loop 整体墙钟上限）不同")
|
||
parser.add_argument("--max-retries", type=int, default=1,
|
||
help="模型任务失败重试次数（默认 1，0 表示不重试；"
|
||
"取消与已记 fallback 的失败不重试）")
|
||
parser.add_argument("--retry-backoff-seconds", type=float, default=5.0,
|
||
help="重试前等待秒数（默认 5）")
|
||
parser.add_argument("--auto-resume-max", type=int, default=50,
|
||
help="长任务自动恢复上限（次数，非时间，默认 50）："
|
||
"同一 goal 任务被系统中断后自动续跑超过该次数"
|
||
"即停住等人工，防确定性失败无限烧额度")
|
||
parser.add_argument("--loop", action="store_true",
|
||
help="常驻循环，每 --interval 秒跑一轮")
|
||
parser.add_argument("--interval", type=int, default=300)
|
||
parser.add_argument("--drain-timeout", type=float, default=60,
|
||
help="停机优雅排空等待上限（秒）：SIGTERM/SIGINT"
|
||
"后先等在跑任务自然结束，超时后才杀残留"
|
||
"并标 interrupted；默认 60 秒（systemd"
|
||
"默认 90 秒停机时限内能完成）")
|
||
parser.add_argument("--sweep-threshold", type=float, default=300,
|
||
help="自动陈旧回收阈值（秒）：running 任务心跳"
|
||
"超过该秒数未更新即被回收；活任务每 30"
|
||
"秒刷心跳，不会被误杀")
|
||
parser.add_argument("--max-seconds", type=float, default=None)
|
||
args = parser.parse_args(argv)
|
||
setup_logging()
|
||
host_py = args.host_py
|
||
if host_py is None:
|
||
here = os.path.dirname(os.path.abspath(__file__))
|
||
host_py = os.path.join(os.path.dirname(here), "host.py")
|
||
worker = Worker(args.db, host_py, config_path=args.config,
|
||
repo=args.repo, env_key=args.env_key,
|
||
codex_config_path=args.codex_config, model=args.model,
|
||
fallback_models=(
|
||
[m.strip() for m in
|
||
args.fallback_models.split(",")]
|
||
if args.fallback_models else []),
|
||
max_turns=args.max_turns,
|
||
ignore_host_budget=args.ignore_host_budget,
|
||
executor=_build_cli_executor(args, host_py),
|
||
task_timeout=args.task_timeout,
|
||
max_retries=args.max_retries,
|
||
retry_backoff_seconds=args.retry_backoff_seconds,
|
||
auto_resume_max=args.auto_resume_max,
|
||
repos_dir=args.repos_dir,
|
||
heartbeat_file=args.heartbeat_file,
|
||
worker_id=args.worker_id,
|
||
capacity=args.worker_capacity,
|
||
host_name=args.host_name,
|
||
pr_jobs_only=args.pr_jobs_only,
|
||
tenant_isolation=args.tenant_isolation,
|
||
tenants_root=args.tenants_root,
|
||
tenant_template_home=args.tenant_template_home,
|
||
tenant_template_repo=args.tenant_template_repo,
|
||
egress_proxy_port=args.egress_proxy_port,
|
||
claim_executor_only=args.claim_executor_only,
|
||
websearch_mcp=args.websearch_mcp,
|
||
seed_skills=args.seed_skills,
|
||
imagegen_mcp=args.imagegen_mcp)
|
||
# CLI 启动清理兜底（常驻与单次都做）：清掉上一轮残留容器。
|
||
worker.cleanup_stale_containers()
|
||
# CLI 起 worker 前先建表/迁移：全新库直接起循环不应因缺表而炸。
|
||
try:
|
||
from cloud.server import init_db as _init_db
|
||
_init_db(args.db)
|
||
except Exception as exc:
|
||
print("worker db 初始化失败：%s" % exc, file=sys.stderr)
|
||
raise SystemExit(1)
|
||
if args.loop:
|
||
def _on_signal(signum, frame):
|
||
worker._loop_stop = True
|
||
signal.signal(signal.SIGTERM, _on_signal)
|
||
signal.signal(signal.SIGINT, _on_signal)
|
||
try:
|
||
if args.loop:
|
||
results = worker.run_loop(interval=args.interval,
|
||
max_seconds=args.max_seconds,
|
||
drain_timeout=args.drain_timeout,
|
||
sweep_threshold=args.sweep_threshold)
|
||
else:
|
||
results = worker.run_all(max_tasks=args.max_tasks)
|
||
finally:
|
||
worker.close()
|
||
print(json.dumps(
|
||
[{"task_id": r.get("task_id"), "status": r.get("status"),
|
||
"reason": r.get("reason"), "error": r.get("error")}
|
||
for r in results],
|
||
indent=2, ensure_ascii=False))
|
||
|
||
|
||
if __name__ == "__main__":
|
||
main()
|