host
f73c3f7a05
此前 /pr 的子进程调用没有超时：远端通路卡死时 handler 线程永久挂住，客户端断开也收不回来。_run 真实路径一律 带 timeout（默认 120 秒，CLOUD_PR_SUBPROCESS_TIMEOUT 可改，非法值回落默认），超时杀子进程、返回 124，调用方 按原有失败路径处理（push/pr_create/commit 抛 PrStepError， 探针按跳过处理）。App 路径的 GitHub API 调用已有 30 秒 超时，本次不动。测试：真实 sleep 超时 2 个、环境变量 2 项、超时映射 push/commit 2 个
383 lines
17 KiB
Python
383 lines
17 KiB
Python
"""云端自动 PR：默认关闭、显式触发、缺条件时跳过不失败。
|
||
|
||
零第三方依赖，只用 Python 标准库；不改 host.py 的任务逻辑，
|
||
只复用它的规则：host 任务名 = cloud 任务 id 经安全字符过滤
|
||
（见 cloud/sandbox.py LocalExecutor._host_name），分支名是
|
||
host/<host 任务名>，worktree 在 <仓库根>/work/host-worktrees/
|
||
<host 任务名>。
|
||
|
||
触发规则（按顺序，任一步跳过即返回，不推、不改任务状态）：
|
||
1. 任务所属项目的 auto_pr 不为 1 -> pr_skipped, auto_pr_disabled
|
||
2. 本地没有对应 worktree -> pr_skipped, no_worktree
|
||
3. worktree 里没有 git remote origin -> pr_skipped, no_remote
|
||
4. 本机没有 gh 二进制 -> pr_skipped, no_gh
|
||
5. gh 未登录（gh auth status 非零） -> pr_skipped,
|
||
gh_not_authenticated
|
||
只有以上全过才提交 worktree 改动、push 分支 + gh pr create；
|
||
push 或 pr create 失败时抛 PrStepError（调用方据此返回 502），
|
||
任务状态保持 merged 不变，绝不把任务标 failed。
|
||
|
||
提交规则（create_task_pr 在全部跳过检查通过后、push 之前做）：
|
||
worktree 必须正好在分支 host/<任务名> 上，否则直接失败，
|
||
绝不在别的分支上提交；先清掉 worktree 内的 __pycache__ /
|
||
*.pyc（与 host.py merge 同规则，只动 worktree，不动主仓），
|
||
再 git add -A + commit；nothing to commit 时直接继续。
|
||
身份用环境里的 git 身份（与 host.py merge 一致），缺身份时
|
||
回退到 cloud/cloud@local 再试一次。
|
||
|
||
顺序约束（与云端 merge 接口配套）：任务 completed -> 调
|
||
POST /tasks/{id}/merge 改状态 -> 调 POST /tasks/{id}/pr 推分支
|
||
建 PR -> 再在 GitHub 上合 PR 或本地 host.py merge。本地
|
||
host.py merge 成功后会删分支和 worktree，此后调 PR 会跳过
|
||
no_worktree，所以 PR 必须在本地 git merge 之前做。
|
||
|
||
worker 不自动推：自动 PR 只由显式调 POST /tasks/{id}/pr
|
||
触发。默认自动推会把测试任务也推到远端，本模块不做。
|
||
|
||
本模块不存、不打任何密钥：gh 的登录凭证由 gh 自己管
|
||
（gh auth login），push/pr 的命令与错误文本里不出现任何
|
||
token；错误文本只含 git/gh 的原文摘要，不含远端 URL 里的
|
||
userinfo（若 remote URL 含 userinfo，摘要里把它抹掉）。
|
||
|
||
App-token 路径（cloud/github_app.py 的 resolve_app_token 供
|
||
token）：调用方传 app_token 时，跳过 gh 二进制与 gh 登录
|
||
检查（App 路径不需要 gh）；push 走 git credential helper，
|
||
token 只放进子进程环境变量 GIT_TOKEN，不进命令行参数、不
|
||
进文件，helper 命令里只有变量名；建 PR 走 GitHub REST API
|
||
（先读仓库 default_branch 再建，base 不猜 main）。没传
|
||
app_token 时走原来的 gh 路径，两条路不互相干扰；App 没装
|
||
在该仓库时调用方不传 token、自动回落 PAT。
|
||
|
||
超时：本模块所有 git/gh 子进程都有超时（默认 120 秒，
|
||
环境变量 CLOUD_PR_SUBPROCESS_TIMEOUT 可改，非法值回落
|
||
默认）。此前没有超时：远端通路卡死时（主机到 GitHub 的
|
||
间歇性劣化曾让连接挂 120 秒以上）handler 线程会永久挂住，
|
||
客户端断开也收不回来。超时只保证 handler 线程在时限内
|
||
结束并返回 502（任务保持 merged、可安全重调），不能把
|
||
结果送给已经断开的客户端；超时发生在 push 时远端引用
|
||
未必已更新（git push 按引用原子更新，传一半被杀不会
|
||
留下半个分支），重调安全。超时落在就绪探针（remote
|
||
查询、gh auth）上时按原有跳过 reason 返回（no_remote /
|
||
gh_not_authenticated），不单独区分——本地 git 命令正常
|
||
在 1 秒内完成，探针超时只意味着 git 本身已 wedged。
|
||
"""
|
||
|
||
import os
|
||
import re
|
||
import shutil
|
||
import subprocess
|
||
|
||
_SKIP_AUTO_PR = "auto_pr_disabled"
|
||
_SKIP_NO_WORKTREE = "no_worktree"
|
||
_SKIP_NO_REMOTE = "no_remote"
|
||
_SKIP_NO_GH = "no_gh"
|
||
_SKIP_NOT_AUTHED = "gh_not_authenticated"
|
||
|
||
# 子进程超时（秒）：远端通路卡死时 handler 线程的上限。
|
||
# 正常 git 本地操作 < 1 秒、健康远端 push 几秒，120 秒只在
|
||
# 通路劣化时才触发（此前实测主机到 GitHub 的挂起超过 120 秒）。
|
||
_DEFAULT_SUBPROCESS_TIMEOUT = 120
|
||
|
||
|
||
def _pr_subprocess_timeout():
|
||
"""超时取值：环境变量 CLOUD_PR_SUBPROCESS_TIMEOUT（整数秒），
|
||
未设、非整数或 <= 0 时回落默认。"""
|
||
raw = os.environ.get("CLOUD_PR_SUBPROCESS_TIMEOUT")
|
||
if raw is None:
|
||
return _DEFAULT_SUBPROCESS_TIMEOUT
|
||
try:
|
||
value = int(raw)
|
||
except (TypeError, ValueError):
|
||
return _DEFAULT_SUBPROCESS_TIMEOUT
|
||
if value <= 0:
|
||
return _DEFAULT_SUBPROCESS_TIMEOUT
|
||
return value
|
||
|
||
|
||
class PrSkipped(Exception):
|
||
"""跳过：reason 见上列常量，调用方返回 200 + pr_skipped。"""
|
||
|
||
def __init__(self, reason):
|
||
super().__init__(reason)
|
||
self.reason = reason
|
||
|
||
|
||
class PrStepError(Exception):
|
||
"""push / pr create 哪一步失败：step 为 push 或 pr_create。"""
|
||
|
||
def __init__(self, step, message):
|
||
super().__init__(message)
|
||
self.step = step
|
||
self.message = message
|
||
|
||
|
||
def host_name_for_task(task_id):
|
||
"""cloud 任务 id -> host 任务名（与 sandbox._host_name 同规则）。"""
|
||
return re.sub(r"[^A-Za-z0-9_.-]", "-", str(task_id))
|
||
|
||
|
||
def default_worktrees_root():
|
||
"""默认 worktree 根：<仓库根>/work/host-worktrees。"""
|
||
here = os.path.dirname(os.path.abspath(__file__))
|
||
return os.path.join(os.path.dirname(here), "work", "host-worktrees")
|
||
|
||
|
||
def resolve_worktrees_root(override=None):
|
||
"""worktree 根优先级：显式参数 > 环境变量 > 默认。"""
|
||
if override:
|
||
return override
|
||
return os.environ.get("CLOUD_HOST_WORKTREES_DIR") or \
|
||
default_worktrees_root()
|
||
|
||
|
||
def _scrub_url(text):
|
||
"""抹掉 URL 里的 userinfo（https://user:pass@host -> https://host），
|
||
防止错误文本把凭证带出去。"""
|
||
if not text:
|
||
return text
|
||
return re.sub(r"(https?://)[^/@\s]+@", r"\1", text)
|
||
|
||
|
||
def _run(cmd, cwd=None, runner=None, env=None, timeout=None):
|
||
"""命令唯一入口：runner 供单测注入，返回带 returncode/stdout/
|
||
stderr 的对象；真实路径走 subprocess。env 只在真实路径生效
|
||
（与进程环境合并），注入的 runner 不收 env——token 绝不进
|
||
假 runner 记录的命令，也不进真实路径的命令行参数。
|
||
超时：真实路径一律带 timeout（默认见 _pr_subprocess_timeout），
|
||
超时杀掉子进程并返回 returncode=124 的结果，调用方按原有
|
||
失败路径处理（push/pr_create/commit 抛 PrStepError，探针
|
||
按跳过处理）；注入的 runner 不收 timeout，单测用返回值
|
||
模拟超时映射，不模拟真实挂起。"""
|
||
if runner is not None:
|
||
return runner(cmd, cwd)
|
||
if timeout is None:
|
||
timeout = _pr_subprocess_timeout()
|
||
full_env = None
|
||
if env:
|
||
full_env = dict(os.environ)
|
||
full_env.update(env)
|
||
try:
|
||
return subprocess.run(cmd, cwd=cwd, capture_output=True, text=True,
|
||
env=full_env, timeout=timeout)
|
||
except subprocess.TimeoutExpired:
|
||
return subprocess.CompletedProcess(
|
||
cmd, 124, "",
|
||
"command timed out after %s seconds: %s"
|
||
% (timeout, " ".join(cmd)))
|
||
|
||
|
||
_GITHUB_REMOTE_RE = re.compile(
|
||
r"^(?:https?://(?:[^/@\s]+@)?github\.com/"
|
||
r"|git@github\.com:|ssh://git@github\.com/)"
|
||
r"([^/\s]+)/([^/\s]+?)(?:\.git)?/?$")
|
||
|
||
|
||
def parse_git_remote_owner_repo(url):
|
||
"""解析 git remote URL -> (owner, repo)，非 github.com 仓库
|
||
或解析失败返回 None（调用方据此回落 PAT，不报错）。"""
|
||
if not url or not isinstance(url, str):
|
||
return None
|
||
m = _GITHUB_REMOTE_RE.match(url.strip())
|
||
if not m:
|
||
return None
|
||
return m.group(1), m.group(2)
|
||
|
||
|
||
def _default_github_api(method, url, token, payload):
|
||
"""默认 GitHub API 客户端：复用 github_app 的 urllib 实现
|
||
（返回 (status, dict)；懒 import，避免模块导入环）。"""
|
||
from cloud.github_app import _github_api_request
|
||
return _github_api_request(method, url, token, payload, None)
|
||
|
||
|
||
def _create_pr_via_api(owner, repo, branch, title, body, token,
|
||
github_api):
|
||
"""用 installation token 走 REST API 建 PR：先读仓库
|
||
default_branch 当 base（不猜 main），再 POST /pulls。
|
||
成功返回 {pr_url, pr_number, branch}，失败抛 PrStepError。"""
|
||
api = github_api or _default_github_api
|
||
status, repo_info = api(
|
||
"GET", "https://api.github.com/repos/%s/%s" % (owner, repo),
|
||
token, None)
|
||
if status != 200:
|
||
raise PrStepError(
|
||
"pr_create",
|
||
"GitHub API 读仓库失败：%s" % (repo_info.get("message"),))
|
||
base = repo_info.get("default_branch") or "main"
|
||
status, data = api(
|
||
"POST", "https://api.github.com/repos/%s/%s/pulls" % (owner, repo),
|
||
token, {"title": title, "head": branch, "base": base,
|
||
"body": body})
|
||
if status not in (200, 201) or not data.get("html_url"):
|
||
raise PrStepError(
|
||
"pr_create",
|
||
"GitHub API 建 PR 失败：%s" % (data.get("message"),))
|
||
return {"pr_url": data["html_url"], "pr_number": data.get("number"),
|
||
"branch": branch}
|
||
|
||
|
||
def _git_remote_url(worktree_dir, git_runner=None):
|
||
r = _run(["git", "-C", worktree_dir, "remote", "get-url", "origin"],
|
||
cwd=worktree_dir, runner=git_runner)
|
||
url = (getattr(r, "stdout", "") or "").strip()
|
||
if getattr(r, "returncode", 1) != 0 or not url:
|
||
return None
|
||
return url
|
||
|
||
|
||
def check_pr_ready(worktree_dir, git_runner=None, gh_which=None,
|
||
gh_runner=None, app_token=None):
|
||
"""按顺序做跳过检查，全过返回 (worktree_dir, branch)；任一步
|
||
不过抛 PrSkipped。branch 由调用方按 host 任务名算好传进来
|
||
的是 worktree 检查——本函数只查 worktree 与 remote/gh/登录，
|
||
分支名由 create_task_pr 组装。传了 app_token 时跳过 gh
|
||
二进制与 gh 登录检查（App 路径不需要 gh）。"""
|
||
if not os.path.isdir(worktree_dir):
|
||
raise PrSkipped(_SKIP_NO_WORKTREE)
|
||
if _git_remote_url(worktree_dir, git_runner) is None:
|
||
raise PrSkipped(_SKIP_NO_REMOTE)
|
||
if app_token:
|
||
return True
|
||
gh_bin = gh_which if gh_which is not None else shutil.which("gh")
|
||
if not gh_bin:
|
||
raise PrSkipped(_SKIP_NO_GH)
|
||
r = _run(["gh", "auth", "status"], cwd=worktree_dir, runner=gh_runner)
|
||
if getattr(r, "returncode", 1) != 0:
|
||
raise PrSkipped(_SKIP_NOT_AUTHED)
|
||
return True
|
||
|
||
|
||
def _commit_worktree_changes(worktree_dir, branch, task_id,
|
||
git_runner=None):
|
||
"""push 之前把 worktree 的改动提交到分支上。
|
||
|
||
顺序：先确认 worktree 正好在目标分支上（不在就直接失败，
|
||
绝不在别的分支上提交，commit 到 main 里就污染主分支了）；
|
||
再清掉 worktree 内的 __pycache__ / *.pyc（可再生的缓存，
|
||
与 host.py merge 同规则）；再 git add -A + commit。
|
||
nothing to commit 时直接返回；缺 git 身份时回退到
|
||
cloud/cloud@local 再试一次；其它失败抛 PrStepError(commit)。
|
||
"""
|
||
cur = _run(["git", "-C", worktree_dir, "branch", "--show-current"],
|
||
cwd=worktree_dir, runner=git_runner)
|
||
if (getattr(cur, "stdout", "") or "").strip() != branch:
|
||
raise PrStepError(
|
||
"commit", "worktree 不在分支 %s 上，拒绝提交" % branch)
|
||
for root, dirs, files in os.walk(worktree_dir):
|
||
if ".git" in dirs:
|
||
dirs.remove(".git")
|
||
if "__pycache__" in dirs:
|
||
shutil.rmtree(os.path.join(root, "__pycache__"),
|
||
ignore_errors=True)
|
||
dirs.remove("__pycache__")
|
||
for fn in files:
|
||
if fn.endswith(".pyc"):
|
||
try:
|
||
os.remove(os.path.join(root, fn))
|
||
except OSError:
|
||
pass
|
||
add = _run(["git", "-C", worktree_dir, "add", "-A"],
|
||
cwd=worktree_dir, runner=git_runner)
|
||
if getattr(add, "returncode", 1) != 0:
|
||
raise PrStepError(
|
||
"commit", "git add 失败：%s" % (
|
||
(getattr(add, "stderr", "") or "")
|
||
or (getattr(add, "stdout", "") or "")))
|
||
msg = "cloud: %s" % task_id
|
||
c = _run(["git", "-C", worktree_dir, "commit", "-m", msg],
|
||
cwd=worktree_dir, runner=git_runner)
|
||
if getattr(c, "returncode", 1) == 0:
|
||
return
|
||
out = ((getattr(c, "stdout", "") or "")
|
||
+ (getattr(c, "stderr", "") or ""))
|
||
if "nothing to commit" in out:
|
||
return
|
||
if ("tell me who you are" in out or "user.name" in out
|
||
or "user.email" in out):
|
||
c2 = _run(["git", "-C", worktree_dir,
|
||
"-c", "user.name=cloud", "-c", "user.email=cloud@local",
|
||
"commit", "-m", msg],
|
||
cwd=worktree_dir, runner=git_runner)
|
||
if getattr(c2, "returncode", 1) == 0:
|
||
return
|
||
out2 = ((getattr(c2, "stdout", "") or "")
|
||
+ (getattr(c2, "stderr", "") or ""))
|
||
if "nothing to commit" in out2:
|
||
return
|
||
raise PrStepError(
|
||
"commit", "git commit 失败（已试 cloud 身份）：%s" % out2[:300])
|
||
raise PrStepError("commit", "git commit 失败：%s" % out[:300])
|
||
|
||
|
||
def create_task_pr(task_id, auto_pr, worktrees_root=None,
|
||
git_runner=None, gh_runner=None, gh_which=None,
|
||
app_token=None, github_api=None):
|
||
"""给一个已 merged 的云端任务建 PR。
|
||
|
||
auto_pr 不为 1 时直接抛 PrSkipped(auto_pr_disabled)，调用方
|
||
在查 worktree 之前就应先查项目 auto_pr——本函数也再查一次，
|
||
防止直接调用者漏查。成功返回 {pr_url, pr_number, branch}。
|
||
全部跳过检查通过后，先把 worktree 改动提交到分支再 push，
|
||
否则未提交的改动进不了 PR。
|
||
传了 app_token 时走 App 路径：push 用 credential helper
|
||
（token 只进子进程环境，不进参数），建 PR 走 GitHub API；
|
||
不传时走原来的 gh 路径。
|
||
"""
|
||
if auto_pr != 1:
|
||
raise PrSkipped(_SKIP_AUTO_PR)
|
||
host_name = host_name_for_task(task_id)
|
||
worktree_dir = os.path.join(
|
||
resolve_worktrees_root(worktrees_root), host_name)
|
||
branch = "host/" + host_name
|
||
check_pr_ready(worktree_dir, git_runner=git_runner,
|
||
gh_which=gh_which, gh_runner=gh_runner,
|
||
app_token=app_token)
|
||
_commit_worktree_changes(worktree_dir, branch, task_id,
|
||
git_runner=git_runner)
|
||
title = "cloud: %s" % task_id
|
||
body = "cloud 任务 %s 的自动 PR。" % task_id
|
||
if app_token:
|
||
remote_url = _git_remote_url(worktree_dir, git_runner)
|
||
owner_repo = parse_git_remote_owner_repo(remote_url)
|
||
if owner_repo is None:
|
||
raise PrStepError(
|
||
"push",
|
||
"worktree remote 不是 github.com 仓库，App token 无法"
|
||
" push：%s" % _scrub_url(remote_url))
|
||
push = _run(
|
||
["git", "-C", worktree_dir, "-c",
|
||
"credential.helper=!f() { echo username=x-access-token;"
|
||
" echo password=$GIT_TOKEN; }; f",
|
||
"push", "origin", branch],
|
||
cwd=worktree_dir, runner=git_runner,
|
||
env={"GIT_TOKEN": app_token})
|
||
if getattr(push, "returncode", 1) != 0:
|
||
raise PrStepError(
|
||
"push", "git push origin %s 失败：%s" % (
|
||
branch, _scrub_url(
|
||
(getattr(push, "stderr", "") or "")
|
||
or (getattr(push, "stdout", "") or ""))))
|
||
return _create_pr_via_api(owner_repo[0], owner_repo[1], branch,
|
||
title, body, app_token, github_api)
|
||
push = _run(["git", "-C", worktree_dir, "push", "origin", branch],
|
||
cwd=worktree_dir, runner=git_runner)
|
||
if getattr(push, "returncode", 1) != 0:
|
||
raise PrStepError(
|
||
"push", "git push origin %s 失败：%s" % (
|
||
branch, _scrub_url(
|
||
(getattr(push, "stderr", "") or "")
|
||
or (getattr(push, "stdout", "") or ""))))
|
||
r = _run(["gh", "pr", "create", "--title", title,
|
||
"--body", body,
|
||
"--head", branch],
|
||
cwd=worktree_dir, runner=gh_runner)
|
||
out = (getattr(r, "stdout", "") or "").strip()
|
||
if getattr(r, "returncode", 1) != 0 or not out:
|
||
raise PrStepError(
|
||
"pr_create", "gh pr create 失败：%s" % _scrub_url(
|
||
(getattr(r, "stderr", "") or "") or out))
|
||
pr_url = out.splitlines()[0].strip()
|
||
m = re.search(r"/(\d+)\s*$", pr_url)
|
||
pr_number = int(m.group(1)) if m else None
|
||
return {"pr_url": pr_url, "pr_number": pr_number, "branch": branch}
|