基底:#251 头 174b631(堆叠于 #242 feat-ci-expand 之上), 基线 origin/main aebc6238;本轨不改 #251 已定语义、不动 workflow 门槛、不删检查、不加 skip。 lint(ruff E,F@120):109 -> 10 - F401 43 -> 0:删未用导入;try/except 双绑定的名字出现 “打地鼠”(删 except 分支后 try 分支又被标出),两分支 均已核对正文无引用后清掉;MCP 三模块的 Error 名不删, 改走 importlib 属性绑定(见下),两路径绑定都保留。 - E702 40 -> 0:全在 host.py main() 的 argparse 段,纯分行, tokenize 按分号 token 拆(含 2377 行多行语句只断首个分号); 一次 5 空格缩进失误已回滚重做并以 ast + 全量 host 测试验证。 - E501 13 -> 0:host.py 5 项随 E702 分行连带清掉;mcp_server 3 项把描述拆成相邻字面量(AST 比对 24 条工具描述与原值 完全一致,注意 ruff 按东亚宽度计长,CJK 算 2);test_host 5 项断开 git commit 参数表。 - E402 8 -> 10:生产侧 2 项清掉(db.py _re、github_app _base64 移到顶部,之前无引用、无重名);剩余 10 项全是 tests/ 的 sys.path 前置 import(6 项旧 + 4 个新测试文件 沿用同一既定模式),移上去会破坏单文件直跑,有意保留。 - F841 4 -> 0:只去绑定、不删调用(run_task、DB 入队、 tarfile 上下文、git 提交),各有注释与原测试/新测试覆盖; host dispatch_from_branch 新增真实 git 仓库回归测试。 - F811 1 -> 0:server 本地 _row_get 与 peer_messages.row_get 逐行相同,删本地定义,全部调用点解析到同一 import; 新增 test_server_row_get(同一对象 + 三路行为)。 mypy(生产范围):24 -> 0,零新增 type: ignore - no-redef 9 -> 0:MCP 三模块 try/except 两次 from-import 改 importlib.import_module + 属性绑定(cloud 优先、flat copy 回落、内层 ImportError 同样回落,语义与原来一致; flat-copy 子进程测试证实回落后 __module__ 为顶层名); server _row_get 随 F811 消除。 - arg-type 10 -> 0:harness2_tasks shared 改 TypedDict 逐键精确类型;CodexProbeAdapter 补 no-op validate_tree (_adapter_validate 本就跳过缺失方法,行为一致)。 - var-annotated 3 -> 0:sub2api _SHARED_ADMIN/_login_slots、 verify-multiuser RESULTS 加精确注解。 - assignment/misc 2 -> 0:marker_tools ZoneInfo 先声明 Any 再 try import(草稿先验 mypy 通过)。 bandit:总数 286 不变,High 仍 4 项,全在 host.py,本轨不改 (属第 3 项白名单轨范围),只做 triage: - 1319 run_verify:无插值,整串验测命令,管道/&& 须 shell 语义(新增 test_host_run_verify 断言管道与 && 真实生效)。 - 1497 notify:模板来自算子配置,三处插值全部 shlex.quote, 证据为既有 test_notify_failure_quotes_shell_metacharacters (; touch 与 $(touch ) 不执行)。 - 2099/2103 goal verify:无插值,任务配置验测串,同 run_verify。 - 无新增 High;pip-audit 0 漏洞。 覆盖率(同口径三套件 coverage 计量):75.96% -> 77.87% (22392 行/缺 5385 -> 22437 行/缺 4963,显示 76% -> 78%) - client.py 0% -> 90%(pipe 假子进程 + main 分支) - run_parallel.py 0% -> 97%、migrate 包装 0% -> 95% - playwright_mcp 37% -> 62%、imagegen_mcp 47% -> 79%、 websearch_mcp 51% -> 86%、deploy.py 56% -> 77% - verify-multiuser 0% -> 27%(仅 check/req helper;main() 是主机侧真实多用户实弹验证,按门禁不建生产任务、 不以 mock 冒充,仍为缺口) - host.py 55% -> 57%(文件 2020 -> 2060 行系 E702 分行 的计量口径变化,非代码膨胀) - 未到 85%:门槛保持 75% 不动,不删低覆盖模块出统计口径; 缺口主力仍是 server.py / worker.py / host.py 大文件, 下一阶段须针对性深测,非本轨快赢可及。 验证:三套件 1576 -> 1629 项全过(tests 127、cloud/tests 1449 skipped=4、deploy/tests 53);ruff E9 通过;mypy 0; bandit High 4(同上);pip-audit 0 漏洞;试跑产物已删, 提交后工作树干净。
397 lines
14 KiB
Python
397 lines
14 KiB
Python
"""外部集成 webhook 与推送通知模块(云端架构实施)。
|
||
|
||
零第三方依赖,只用 Python 标准库。
|
||
|
||
本模块只做两件事:
|
||
|
||
1. 入站 webhook 的签名校验与事件解析(GitHub、Slack)。
|
||
2. 出站状态通知的占位分发(webhook + Slack)。
|
||
|
||
边界(不要超出本模块宣称的范围):
|
||
|
||
- Web Push / APNs / FCM 明确是后续需要证书的一步,本版只有
|
||
webhook + Slack 占位,不要编造已接通手机推送。
|
||
- 密钥走环境变量是调用方的事:本模块的校验函数只接受参数,
|
||
不自己读生产密钥,也不把任何密钥写入文件。
|
||
- 本模块不联网、不调用真实 GitHub/Slack、不跑真实模型回合。
|
||
"""
|
||
|
||
import hashlib
|
||
import hmac
|
||
import time
|
||
|
||
from cloud.db import connect as _db_connect
|
||
|
||
__all__ = [
|
||
"verify_github_signature",
|
||
"is_trusted_fix_review_author",
|
||
"parse_github_pr_comment",
|
||
"parse_github_check_run",
|
||
"WebhookDeliveryStore",
|
||
"verify_slack_signature",
|
||
"parse_slack_command",
|
||
"SlackUserMap",
|
||
"ALLOWED_NOTIFICATION_EVENTS",
|
||
"NotificationService",
|
||
]
|
||
|
||
|
||
def verify_github_signature(secret, payload_bytes, signature_header):
|
||
"""校验 GitHub webhook 签名。
|
||
|
||
signature_header 形如 ``"sha256=<hex>"``,用
|
||
``hmac.new(secret.encode(), payload_bytes, hashlib.sha256).hexdigest()``
|
||
比对,比较用 ``hmac.compare_digest``。缺 secret 或缺 header 返回 False。
|
||
"""
|
||
if not secret or not signature_header:
|
||
return False
|
||
if isinstance(payload_bytes, str):
|
||
payload_bytes = payload_bytes.encode("utf-8")
|
||
if not isinstance(payload_bytes, (bytes, bytearray)):
|
||
return False
|
||
if not isinstance(signature_header, str):
|
||
return False
|
||
try:
|
||
expected = "sha256=" + hmac.new(
|
||
secret.encode("utf-8"), bytes(payload_bytes), hashlib.sha256
|
||
).hexdigest()
|
||
except (AttributeError, TypeError, ValueError):
|
||
return False
|
||
return hmac.compare_digest(expected, signature_header)
|
||
|
||
|
||
def verify_github_signature_any(secrets, payload_bytes, signature_header):
|
||
"""多 secret 验签:任一 secret 对上即 True,全不对或无 secret
|
||
即 False。过渡期手动 webhook(env secret)与 App webhook
|
||
(库里 secret)共存时用;App 装好、手动 webhook 删掉后,
|
||
env secret 自然不再被命中,不用改代码。"""
|
||
if not secrets:
|
||
return False
|
||
for secret in secrets:
|
||
if secret and verify_github_signature(
|
||
secret, payload_bytes, signature_header):
|
||
return True
|
||
return False
|
||
|
||
|
||
def parse_github_pr_comment(payload):
|
||
"""解析 GitHub issue_comment 事件,返回 fix-review 触发点或 None。
|
||
|
||
这是 PR 评论 → fix-review 的触发点,真正执行 fix-review 由控制面调
|
||
host.py 完成,本函数只解析不执行。
|
||
|
||
只有当 action == "created" 且 comment body 里含 "/fix-review" 时返回::
|
||
|
||
{"type": "fix_review", "repo": ..., "pr_number": ...,
|
||
"comment": ..., "author": ...}
|
||
|
||
其中 repo 取 payload repository full_name,pr_number 取 issue number,
|
||
comment 取 body,author 取 comment user login。其他情况返回 None。
|
||
"""
|
||
if not isinstance(payload, dict):
|
||
return None
|
||
if payload.get("action") != "created":
|
||
return None
|
||
comment = payload.get("comment")
|
||
if not isinstance(comment, dict):
|
||
return None
|
||
body = comment.get("body")
|
||
if not isinstance(body, str) or "/fix-review" not in body:
|
||
return None
|
||
repository = payload.get("repository") or {}
|
||
issue = payload.get("issue") or {}
|
||
user = comment.get("user") or {}
|
||
return {
|
||
"type": "fix_review",
|
||
"repo": repository.get("full_name"),
|
||
"pr_number": issue.get("number"),
|
||
"comment": body,
|
||
"author": user.get("login"),
|
||
}
|
||
|
||
|
||
# GitHub 在 issue_comment / pull_request_review_comment 事件里
|
||
# 自带 author_association(由 GitHub 设定,不是评论者自填):
|
||
# OWNER / MEMBER / COLLABORATOR 是仓库可信身份;CONTRIBUTOR
|
||
# 及以下不是。PR/issue 作者本人评论自己的 PR 也可信(作者
|
||
# 只能在自己的 PR 上触发,不构成跨仓库免费入口)。
|
||
_TRUSTED_FIX_REVIEW_ASSOCIATIONS = frozenset(
|
||
{"OWNER", "MEMBER", "COLLABORATOR"})
|
||
|
||
|
||
def is_trusted_fix_review_author(payload):
|
||
"""P0 安全批:/fix-review 评论者是否可信。
|
||
|
||
只认两条,任一成立即 True,其余一律 False(fail-closed):
|
||
1. comment.user.author_association 在可信集里;
|
||
2. 评论者 login 等于 issue 作者或 pull_request 作者。
|
||
association 缺失且作者比对不上时返回 False——此时不要
|
||
用 App token 现查协作者(本函数不联网),直接拒单,
|
||
绝不因“查不到”就放行。这是 GitHub 事件字段校验,不
|
||
是实时协作者 API 查询,部署文档里必须如实写清这一点。
|
||
"""
|
||
if not isinstance(payload, dict):
|
||
return False
|
||
comment = payload.get("comment")
|
||
if not isinstance(comment, dict):
|
||
return False
|
||
user = comment.get("user") or {}
|
||
author = user.get("login")
|
||
if not author:
|
||
return False
|
||
if user.get("author_association") in _TRUSTED_FIX_REVIEW_ASSOCIATIONS:
|
||
return True
|
||
issue = payload.get("issue") or {}
|
||
issue_user = issue.get("user") or {}
|
||
if author == issue_user.get("login"):
|
||
return True
|
||
pull = payload.get("pull_request") or {}
|
||
pull_user = pull.get("user") or {}
|
||
if author == pull_user.get("login"):
|
||
return True
|
||
return False
|
||
|
||
|
||
def parse_github_check_run(payload):
|
||
"""解析 GitHub check_run / check_suite 事件,返回结构化结果或 None。
|
||
|
||
只在结论已定的时候返回:check_run / check_suite 的 conclusion
|
||
为 "success" 或 "failure" 时返回::
|
||
|
||
{"type": "check_run", "repo": ..., "head_sha": ...,
|
||
"conclusion": ..., "check_name": ...}
|
||
|
||
其中 repo 取 repository full_name,head_sha 取 check_run /
|
||
check_suite 的 head_sha,check_name 取 check_run 的 name
|
||
(check_suite 没有单项名时为 None)。结论为 None / 进行中
|
||
(如 queued / in_progress / neutral / cancelled 等)时返回 None,
|
||
调用方不应据此建任务或改任务状态——CI 状态回填由 pr-checks
|
||
主动拉取,webhook 只做通知解析。
|
||
"""
|
||
if not isinstance(payload, dict):
|
||
return None
|
||
for key in ("check_run", "check_suite"):
|
||
run = payload.get(key)
|
||
if not isinstance(run, dict):
|
||
continue
|
||
conclusion = run.get("conclusion")
|
||
if conclusion not in ("success", "failure"):
|
||
return None
|
||
repository = payload.get("repository") or {}
|
||
return {
|
||
"type": "check_run",
|
||
"repo": repository.get("full_name"),
|
||
"head_sha": run.get("head_sha"),
|
||
"conclusion": conclusion,
|
||
"check_name": run.get("name"),
|
||
}
|
||
return None
|
||
|
||
|
||
class WebhookDeliveryStore:
|
||
"""GitHub webhook delivery 去重(持久化,不只放内存)。
|
||
|
||
表 webhook_deliveries:delivery_id(主键,X-GitHub-Delivery 头)、
|
||
event、repo、created_at(epoch 秒)。claim() 第一次返回 True,
|
||
同一 delivery_id 第二次返回 False,调用方应直接回 200 且标记
|
||
duplicate,不重复建任务、不重复记审计。INSERT 冲突时先按
|
||
IntegrityError 处理,再回查一次确认(PG 适配层抛的完整性错误
|
||
类型与 SQLite 不同,这里按“插入失败后回查存在即重复”处理,
|
||
不依赖具体异常类名)。
|
||
"""
|
||
|
||
def __init__(self, db_path):
|
||
self.db_path = db_path
|
||
self._conn = _db_connect(db_path)
|
||
self._conn.execute(
|
||
"CREATE TABLE IF NOT EXISTS webhook_deliveries("
|
||
"delivery_id TEXT PRIMARY KEY, "
|
||
"event TEXT, "
|
||
"repo TEXT, "
|
||
"created_at REAL)"
|
||
)
|
||
self._conn.commit()
|
||
|
||
def claim(self, delivery_id, event=None, repo=None):
|
||
"""认领一次 delivery:首次 True,重复 False。空 id 不认领。"""
|
||
if not delivery_id:
|
||
return True
|
||
row = self._conn.execute(
|
||
"SELECT delivery_id FROM webhook_deliveries WHERE delivery_id = ?",
|
||
(delivery_id,),
|
||
).fetchone()
|
||
if row is not None:
|
||
return False
|
||
try:
|
||
self._conn.execute(
|
||
"INSERT INTO webhook_deliveries"
|
||
"(delivery_id, event, repo, created_at)"
|
||
" VALUES(?, ?, ?, ?)",
|
||
(delivery_id, event, repo, time.time()),
|
||
)
|
||
self._conn.commit()
|
||
except Exception:
|
||
try:
|
||
self._conn.rollback()
|
||
except Exception:
|
||
pass
|
||
row = self._conn.execute(
|
||
"SELECT delivery_id FROM webhook_deliveries"
|
||
" WHERE delivery_id = ?",
|
||
(delivery_id,),
|
||
).fetchone()
|
||
if row is not None:
|
||
return False
|
||
raise
|
||
return True
|
||
|
||
def close(self):
|
||
"""关闭数据库连接(可选,单测/退出时用)。"""
|
||
self._conn.close()
|
||
|
||
|
||
def verify_slack_signature(signing_secret, timestamp, body, signature_header):
|
||
"""校验 Slack 请求签名(防重放)。
|
||
|
||
Slack 规则:basestring = ``f"v0:{timestamp}:{body}"``,hmac sha256,
|
||
header 形如 ``"v0=<hex>"``。时间戳与当前时间差超过 300 秒返回 False
|
||
(防重放);缺参返回 False。
|
||
"""
|
||
if not signing_secret or timestamp is None or body is None:
|
||
return False
|
||
if not signature_header:
|
||
return False
|
||
try:
|
||
ts_int = int(timestamp)
|
||
except (TypeError, ValueError):
|
||
return False
|
||
if abs(time.time() - ts_int) > 300:
|
||
return False
|
||
try:
|
||
if isinstance(body, (bytes, bytearray)):
|
||
basestring = b"v0:" + str(ts_int).encode("utf-8") + b":" + bytes(body)
|
||
expected = "v0=" + hmac.new(
|
||
signing_secret.encode("utf-8"), basestring, hashlib.sha256
|
||
).hexdigest()
|
||
else:
|
||
basestring = f"v0:{timestamp}:{body}"
|
||
expected = "v0=" + hmac.new(
|
||
signing_secret.encode("utf-8"),
|
||
basestring.encode("utf-8"),
|
||
hashlib.sha256,
|
||
).hexdigest()
|
||
except (AttributeError, TypeError, ValueError):
|
||
return False
|
||
if not isinstance(signature_header, str):
|
||
return False
|
||
return hmac.compare_digest(expected, signature_header)
|
||
|
||
|
||
def parse_slack_command(text):
|
||
"""解析 Slack 斜杠命令。
|
||
|
||
- ``"/codex 做一个加法函数"`` ->
|
||
``{"command": "codex", "prompt": "做一个加法函数"}``
|
||
- ``"/codex status <任务名>"`` ->
|
||
``{"command": "status", "task": "<任务名>"}``
|
||
- 空或非 ``/codex`` 开头 -> ``{"command": None, "prompt": text}``
|
||
"""
|
||
if not text or not isinstance(text, str):
|
||
return {"command": None, "prompt": text}
|
||
stripped = text.strip()
|
||
if stripped == "/codex" or stripped.startswith("/codex "):
|
||
rest = stripped[len("/codex"):].strip()
|
||
if rest == "status" or rest.startswith("status "):
|
||
task = rest[len("status"):].strip()
|
||
return {"command": "status", "task": task}
|
||
return {"command": "codex", "prompt": rest}
|
||
return {"command": None, "prompt": text}
|
||
|
||
|
||
class SlackUserMap:
|
||
"""Slack 用户 ID 到内部 user_id 的映射(SQLite)。"""
|
||
|
||
def __init__(self, db_path):
|
||
self.db_path = db_path
|
||
self._conn = _db_connect(db_path)
|
||
self._conn.execute(
|
||
"CREATE TABLE IF NOT EXISTS slack_users ("
|
||
"slack_id TEXT PRIMARY KEY, "
|
||
"user_id TEXT, "
|
||
"created_at REAL)"
|
||
)
|
||
self._conn.commit()
|
||
|
||
def map(self, slack_id, user_id):
|
||
"""把 slack_id 映射到 user_id(已存在则覆盖)。"""
|
||
self._conn.execute(
|
||
"INSERT OR REPLACE INTO slack_users "
|
||
"(slack_id, user_id, created_at) VALUES (?, ?, ?)",
|
||
(slack_id, user_id, time.time()),
|
||
)
|
||
self._conn.commit()
|
||
|
||
def lookup(self, slack_id):
|
||
"""返回 slack_id 对应的 user_id,未映射返回 None。"""
|
||
row = self._conn.execute(
|
||
"SELECT user_id FROM slack_users WHERE slack_id = ?",
|
||
(slack_id,),
|
||
).fetchone()
|
||
return row[0] if row else None
|
||
|
||
def close(self):
|
||
"""关闭数据库连接(可选,单测/退出时用)。"""
|
||
self._conn.close()
|
||
|
||
|
||
# 只发状态变化的事件集合:进行中的状态(如 running)不触发通知。
|
||
ALLOWED_NOTIFICATION_EVENTS = {
|
||
"completed",
|
||
"failed",
|
||
"goal-unmet",
|
||
"merge-failed",
|
||
"review-requested",
|
||
}
|
||
|
||
|
||
class NotificationService:
|
||
"""状态通知分发(webhook + Slack 占位)。
|
||
|
||
Web Push / APNs / FCM 明确是后续需要证书的一步,本版只有 webhook +
|
||
Slack 占位,不要编造已接通手机推送。
|
||
|
||
sender 都是可调用对象,单测可注入假 sender,默认 None 时不发、
|
||
不报错。sender 被调用时收到一个 dict payload::
|
||
|
||
{"event_type": ..., "task_id": ..., "project_id": ..., "detail": ...}
|
||
"""
|
||
|
||
def __init__(self, webhook_sender=None, slack_sender=None):
|
||
self.webhook_sender = webhook_sender
|
||
self.slack_sender = slack_sender
|
||
|
||
def notify(self, event_type, task_id, project_id, detail=""):
|
||
"""发送一条状态通知。
|
||
|
||
event_type 不在 ALLOWED_NOTIFICATION_EVENTS 里时返回
|
||
``{"sent": False, "reason": "event not allowed"}`` 且不调用 sender;
|
||
否则依次调 webhook_sender 和 slack_sender(不为 None 的才调),
|
||
返回 ``{"sent": True, "channels": [...已调的通道名...]}``。
|
||
"""
|
||
if event_type not in ALLOWED_NOTIFICATION_EVENTS:
|
||
return {"sent": False, "reason": "event not allowed"}
|
||
payload = {
|
||
"event_type": event_type,
|
||
"task_id": task_id,
|
||
"project_id": project_id,
|
||
"detail": detail,
|
||
}
|
||
channels = []
|
||
if self.webhook_sender is not None:
|
||
self.webhook_sender(payload)
|
||
channels.append("webhook")
|
||
if self.slack_sender is not None:
|
||
self.slack_sender(payload)
|
||
channels.append("slack")
|
||
return {"sent": True, "channels": channels}
|