推送真发送:webhook/slack 内置真实发送端 + worker 终态自动通知

此前订阅与分发框架都在,但分发端点传 sender=None,
webhook/slack 永远报无 sender、从未真发过。本轮:
- push:send_webhook/send_slack(标准库 urllib POST,
  单次 10 秒超时,不重试,失败记 errors);sender=None
  时用内置端,显式 sender 优先;webpush/apns/fcm 仍报
  缺凭证(原因写明:webpush 需浏览器端三字段订阅与标准库
  没有的 P-256 ECDH + AES-GCM;apns 需 HTTP/2;fcm 需
  Firebase 服务账号,三者均无现成接收方,不造假发送)
- worker:run_one 终态(completed/failed,缺 key 与仓库
  失败两条早退路径也在内)后 best-effort 调分发,通知
  失败只记日志、绝不影响任务状态;取消(interrupted)
  不在事件集合里,不通知
- 测试:真实发送端 6 个(本地回环服务器)、worker 终态
  通知 2 个,旧无 sender 断言按新语义改写
This commit is contained in:
host committed 2026-10-26 17:31:30 +08:00
1 parent b7dace55b9
commit fb3a311da8
7 files changed
+429 -29

No files matched your search

+19 -7
View File
@@ -222,12 +222,24 @@ webhook 实测)。
### 推送凭证清单 ### 推送凭证清单
- webhook / slack:需要可注入 sender(worker 侧),HTTP 层没有 - webhook / slack:已有内置真实发送端(标准库 urllib POST,
sender 时明确报错,不静默成功。 单次 10 秒超时,不重试,失败记入分发结果的 errors):
- webpush:需要 VAPID 密钥对(私钥绝不进仓库)。 webhook 投 JSON(event_type、project_id、task_id、payload),
- apns:需要 APNs 证书。 slack 投 Slack incoming webhook 格式的一行文本(任务 id +
- fcm:需要 FCM key。 事件 + 项目,失败附 error 摘要)。任务终态(completed /
以上三者目前只有接口 + 缺凭证报错,没有真实发送端。 failed)由 worker 在 run_one 的终态写回后自动通知订阅者;
手动分发走 `POST /admin/push/dispatch`,同样经内置发送端。
显式注入的 sender 优先于内置端。
- webpush:需要 VAPID 密钥对(私钥绝不进仓库),且需要
浏览器端订阅(endpoint + p256dh + auth,现有订阅模型只有
一个 target 文本字段,装不下)与标准库没有的 P-256 ECDH +
AES-GCM——只有接口 + 缺凭证报错,没有真实发送端。
- apns:需要 APNs 证书与标准库没有的 HTTP/2——只有接口 +
缺凭证报错。
- fcm:需要 FCM key 与用户另建的 Firebase 服务账号——只有
接口 + 缺凭证报错。
以上三者目前只有接口 + 缺凭证报错,没有真实发送端,也
不编造推送。
### SQLite 并发上限 ### SQLite 并发上限
@@ -374,7 +386,7 @@ psycopg,写明“需安装 psycopg”),绝不静默回落到 SQLite,
| `POST /projects/{id}/push/subscriptions` | 该项目 `create_task` | 记订阅 body `{channel, target, preference?}`,channel 为 webhook/slack/webpush/apns/fcm,preference 为 instant/digest/silent,返回 `201` | | `POST /projects/{id}/push/subscriptions` | 该项目 `create_task` | 记订阅 body `{channel, target, preference?}`,channel 为 webhook/slack/webpush/apns/fcm,preference 为 instant/digest/silent,返回 `201` |
| `GET /projects/{id}/push/subscriptions` | 该项目 `read` | 只列调用者自己的订阅(不泄露他人 target) | | `GET /projects/{id}/push/subscriptions` | 该项目 `read` | 只列调用者自己的订阅(不泄露他人 target) |
| `DELETE /projects/{id}/push/subscriptions/{sid}` | 该项目 `create_task` | 只能删自己的订阅,项目 owner 可删任意;不存在 `404`,他人订阅 `403` | | `DELETE /projects/{id}/push/subscriptions/{sid}` | 该项目 `create_task` | 只能删自己的订阅,项目 owner 可删任意;不存在 `404`,他人订阅 `403` |
| `POST /admin/push/dispatch` | 该项目 `read` | body `{event_type, project_id, payload?}`;只发状态变化事件(completed/failed/goal-unmet/merge-failed/review-requested);HTTP 层无注入 sender:webhook/slack 报“无 sender”,webpush/apns/fcm 报缺凭证,都不静默成功;真实发送端由 worker 注入 sender 后调 `push.dispatch_event` | | `POST /admin/push/dispatch` | 该项目 `read` | body `{event_type, project_id, payload?}`;只发状态变化事件(completed/failed/goal-unmet/merge-failed/review-requested);sender=None 时用内置真实发送端(webhook/slack 真 POST,webpush/apns/fcm 报缺凭证),都不静默成功;worker 在任务终态(completed/failed)时自动调分发,不依赖手动调本端点 |
| `POST /webhooks/github` | GitHub 签名 | 见“webhook 端点” | | `POST /webhooks/github` | GitHub 签名 | 见“webhook 端点” |
| `POST /webhooks/slack` | Slack 签名 | 见“webhook 端点” | | `POST /webhooks/slack` | Slack 签名 | 见“webhook 端点” |
+121 -11
View File
@@ -17,21 +17,36 @@
``digest`` 偏好下本次不立即发,返回结果里明确列入 ``digest`` 偏好下本次不立即发,返回结果里明确列入
``skipped_for_digest``(待摘要列表),不是静默丢弃。 ``skipped_for_digest``(待摘要列表),不是静默丢弃。
- 发送端可注入:``webhook`` / ``slack`` 走可注入的 ``sender``。 - 发送端可注入:``webhook`` / ``slack`` 走可注入的 ``sender``。
``sender`` 为 ``None`` 时对这两个通道也返回“无 sender”错误, ``sender`` 为 ``None`` 时用内置真实发送端(见 :func:`send_webhook` /
不静默成功。``sender`` 可以是可调用对象(收到一个 dict,见 :func:`send_slack`,标准库 urllib POST,单次 10 秒超时,不
重试,失败记入 errors);显式传入的 sender 优先于内置端。
``sender`` 可以是可调用对象(收到一个 dict,见
:func:`dispatch_event`),也可以是按通道分的 dict :func:`dispatch_event`),也可以是按通道分的 dict
(``{"webhook": callable, "slack": callable}``),或带 ``send`` (``{"webhook": callable, "slack": callable}``),或带 ``send``
方法的对象。 方法的对象。显式 sender 里没有某通道时,该通道仍报
“无 sender”,不静默成功。
- ``webpush`` / ``apns`` / ``fcm`` 只有接口,没有真实发送端: - ``webpush`` / ``apns`` / ``fcm`` 只有接口,没有真实发送端:
一律明确报错“缺凭证(VAPID / APNs 证书 / FCM key)”,不编造 一律明确报错“缺凭证(VAPID / APNs 证书 / FCM key)”,不编造
真实推送、不生成或提交任何私钥到仓库、不向真实推送服务发 真实推送、不生成或提交任何私钥到仓库、不向真实推送服务发
任何请求。真实凭证与真实发送端是后续待接线的工作,见模块 任何请求。原因不是“没空写”:webpush 需要浏览器端订阅
末尾的缺口说明。 (endpoint + p256dh + auth 三字段,现有订阅模型只有一个
- 本模块不联网、不调用真实推送服务、不跑真实模型回合。 target 文本字段,装不下)与标准库没有的 P-256 ECDH +
密钥、token、证书一律不由本模块读取、写入或返回。 AES-GCM;apns 需要标准库没有的 HTTP/2;fcm 需要用户
另建 Firebase 项目与服务账号——三者都没有现成接收方,
本模块不为此造一套半吊子加密或假发送。真实凭证与真实
发送端是后续待接线的工作,见模块末尾的缺口说明。
- 本模块不跑真实模型回合。webhook/slack 的真实发送只 POST
到订阅者自己填的 target URL(创建订阅已要求项目
create_task 权限;能建订阅的成员本来就能经任务执行代码,
因此不对 target 做内网屏蔽——屏蔽既拦不住提权,也会误伤
本地接收器)。密钥、token、证书一律不由本模块读取、写入
或返回。
""" """
import json
import sqlite3 import sqlite3
import urllib.error
import urllib.request
from cloud.db import connect as _db_connect from cloud.db import connect as _db_connect
from cloud.db import PgConnection as _PgConnection from cloud.db import PgConnection as _PgConnection
import time import time
@@ -44,10 +59,13 @@ __all__ = [
"ALLOWED_PUSH_EVENTS", "ALLOWED_PUSH_EVENTS",
"FAILURE_EVENTS", "FAILURE_EVENTS",
"MISSING_CREDENTIAL_REASONS", "MISSING_CREDENTIAL_REASONS",
"DEFAULT_SENDERS",
"init_push_db", "init_push_db",
"create_subscription", "create_subscription",
"delete_subscription", "delete_subscription",
"list_subscriptions", "list_subscriptions",
"send_webhook",
"send_slack",
"dispatch_event", "dispatch_event",
] ]
@@ -254,6 +272,91 @@ def _resolve_channel_sender(sender, channel):
return send if callable(send) else None return send if callable(send) else None
# 真实发送端:单次投递超时(秒),不重试——失败记入 errors,
# 下一次事件自然重发,不在这里循环撞墙。
_PUSH_SEND_TIMEOUT = 10
def _post_json(url, payload):
"""POST JSON 到 url:返回 (ok, detail),2xx 为 ok。
HTTP 状态错误与网络异常一律返回 (False, reason),不抛
(调用方据此记 errors);只有调用方传了非法 URL 时
Request 构造本身可能抛 ValueError,由发送端先校验 scheme
挡住,不走到这里。
"""
data = json.dumps(payload).encode("utf-8")
request = urllib.request.Request(
url, data=data, method="POST",
headers={"Content-Type": "application/json",
"User-Agent": "codex-app-server"})
try:
with urllib.request.urlopen(
request, timeout=_PUSH_SEND_TIMEOUT) as response:
response.read()
return True, "HTTP %s" % response.status
except urllib.error.HTTPError as e:
return False, "HTTP %s" % e.code
except Exception as exc: # noqa: BLE001 -- 网络异常转 reason,不抛
return False, "%s: %s" % (type(exc).__name__, exc)
def _require_http_url(target, channel):
"""发送端 target 校验:只接受 http(s) URL,其他一律 ValueError
(dispatch 记入 errors,不静默成功、不乱 POST)。"""
if not isinstance(target, str) or not (
target.startswith("http://") or target.startswith("https://")):
raise ValueError(
"%s target must be an http(s) URL" % channel)
return target
def send_webhook(message):
"""webhook 通道真实发送:POST JSON 到订阅的 target URL。
body 固定为 {event_type, project_id, task_id, payload},
task_id 从 payload 里取(没有则为 None)。非 2xx 抛
RuntimeError(dispatch 记入 errors,不算 sent)。
"""
target = _require_http_url(message["target"], "webhook")
payload = message["payload"] if isinstance(
message["payload"], dict) else {}
ok, detail = _post_json(target, {
"event_type": message["event_type"],
"project_id": message["project_id"],
"task_id": payload.get("task_id"),
"payload": message["payload"],
})
if not ok:
raise RuntimeError("webhook delivery failed: %s" % detail)
def send_slack(message):
"""slack 通道真实发送:POST Slack incoming webhook 格式
({"text": ...})到 target URL(Slack webhook 地址本身
就是 http(s) URL,与 webhook 通道同一投递路径)。
文本一行讲清事件:任务 id + 事件类型 + 项目,失败类
事件附 error 摘要(截断 500 字)。非 2xx 抛 RuntimeError。
"""
target = _require_http_url(message["target"], "slack")
payload = message["payload"] if isinstance(
message["payload"], dict) else {}
text = "[codex] task %s %s (project %s)" % (
payload.get("task_id", "?"), message["event_type"],
message["project_id"])
if payload.get("error"):
text += "\nerror: %s" % str(payload["error"])[:500]
ok, detail = _post_json(target, {"text": text})
if not ok:
raise RuntimeError("slack delivery failed: %s" % detail)
# sender=None 时的内置真实发送端:只覆盖 webhook/slack;
# webpush/apns/fcm 不在此列,仍走缺凭证报错。
DEFAULT_SENDERS = {"webhook": send_webhook, "slack": send_slack}
def dispatch_event(db, event_type, project_id, payload, sender=None): def dispatch_event(db, event_type, project_id, payload, sender=None):
"""分发一次状态变化事件,只发本项目的订阅。 """分发一次状态变化事件,只发本项目的订阅。
@@ -265,9 +368,10 @@ def dispatch_event(db, event_type, project_id, payload, sender=None):
project_id: 只查该项目的订阅,跨项目不混。 project_id: 只查该项目的订阅,跨项目不混。
payload: 原事件载荷(dict 或其他),原样放入发送给 sender payload: 原事件载荷(dict 或其他),原样放入发送给 sender
的消息 dict 的 ``payload`` 字段。 的消息 dict 的 ``payload`` 字段。
sender: 可注入发送端(见模块 docstring)。webhook/slack sender: 可注入发送端(见模块 docstring);为 None 时用
通道在 sender 为 None 时返回“无 sender”错误,不静默成功; 内置真实发送端(webhook/slack 真 POST,webpush/apns/fcm
webpush/apns/fcm 通道不走 sender,一律返回缺凭证错误。 仍报缺凭证);显式传入的 sender 优先于内置端,显式
sender 里没有某通道时该通道报“无 sender”,不静默成功。
返回 dict,键固定为:: 返回 dict,键固定为::
@@ -314,6 +418,11 @@ def dispatch_event(db, event_type, project_id, payload, sender=None):
}] }]
return result return result
if sender is None:
# 内置真实发送端:webhook/slack 真 POST;显式 sender
# 传入时不走这里(注入优先),webpush/apns/fcm 永远
# 不走 sender,仍在循环里报缺凭证。
sender = DEFAULT_SENDERS
subscriptions = list_subscriptions(db, project_id) subscriptions = list_subscriptions(db, project_id)
is_failure = event_type in FAILURE_EVENTS is_failure = event_type in FAILURE_EVENTS
for sub in subscriptions: for sub in subscriptions:
@@ -336,7 +445,8 @@ def dispatch_event(db, event_type, project_id, payload, sender=None):
result["errors"].append( result["errors"].append(
dict(base, reason=MISSING_CREDENTIAL_REASONS[sub["channel"]])) dict(base, reason=MISSING_CREDENTIAL_REASONS[sub["channel"]]))
continue continue
# webhook/slack:必须有可注入 sender,否则明确报错。 # webhook/slack:sender=None 时已换成内置真实发送端;
# 显式 sender 里没有该通道时明确报错,不静默成功。
channel_sender = _resolve_channel_sender(sender, sub["channel"]) channel_sender = _resolve_channel_sender(sender, sub["channel"])
if channel_sender is None: if channel_sender is None:
result["errors"].append( result["errors"].append(
+3 -3
View File
@@ -2428,9 +2428,9 @@ class CloudControlPlaneHandler(BaseHTTPRequestHandler):
if not auth.check_permission(user_id, project_id, "read"): if not auth.check_permission(user_id, project_id, "read"):
self._send_json(403, {"error": "forbidden"}) self._send_json(403, {"error": "forbidden"})
return return
# HTTP 层无可注入 sender:webhook/slack 会报“无 sender”, # 分发:sender=None 时用 push 的内置真实发送端
# webpush/apns/fcm 会报缺凭证——都不静默成功。真实发送端 # (webhook/slack 真 POST,webpush/apns/fcm 仍报缺凭证);
# 由 worker 注入 sender 后调 push.dispatch_event。 # 显式注入的 sender 优先于内置端(见 push.dispatch_event)。
result = _push_dispatch( result = _push_dispatch(
self.server.db_path, event_type, project_id, self.server.db_path, event_type, project_id,
body.get("payload") or {}, sender=None) body.get("payload") or {}, sender=None)
+149 -5
View File
@@ -1,14 +1,17 @@
"""push.py 单测:订阅增删查、偏好过滤、分发错误不静默丢。 """push.py 单测:订阅增删查、偏好过滤、分发错误不静默丢。
只用标准库,不联网、不向真实推送服务发任何请求:所有发送都用 不连外网、不向真实推送服务发任何请求:注入 sender 的测试
假 sender(可调用对象或按通道分的 dict)注入;webpush/apns/fcm 用假 sender;真实发送端的测试用本地回环 HTTP 服务器
只断言缺凭证报错,不调用任何 sender。 (127.0.0.1 临时端口)收 POST,断言方法、头与 body。
本文件不含任何真实密钥、token、证书或私钥。 本文件不含任何真实密钥、token、证书或私钥。
""" """
import http.server
import json
import os import os
import sqlite3 import sqlite3
import sys import sys
import threading
import unittest import unittest
from pathlib import Path from pathlib import Path
@@ -22,6 +25,7 @@ try:
delete_subscription, delete_subscription,
dispatch_event, dispatch_event,
list_subscriptions, list_subscriptions,
send_webhook,
) )
except ImportError: # discover -s cloud/tests 时把项目根目录补进 sys.path except ImportError: # discover -s cloud/tests 时把项目根目录补进 sys.path
sys.path.insert(0, str(Path(__file__).resolve().parents[2])) sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
@@ -34,6 +38,7 @@ except ImportError: # discover -s cloud/tests 时把项目根目录补进 sys.p
delete_subscription, delete_subscription,
dispatch_event, dispatch_event,
list_subscriptions, list_subscriptions,
send_webhook,
) )
@@ -184,10 +189,13 @@ class TestDispatchPreferences(PushTestCase):
class TestDispatchSendersAndCredentials(PushTestCase): class TestDispatchSendersAndCredentials(PushTestCase):
def test_webhook_without_sender_returns_no_sender_error_not_silent_ok(self): def test_explicit_sender_missing_channel_returns_no_sender_error_not_silent_ok(self):
# sender=None 时内置真实发送端会真 POST(见 TestRealSenders),
# “无 sender”只发生在显式 sender 里没有该通道时,仍不静默成功。
self._sub(channel="webhook") self._sub(channel="webhook")
result = dispatch_event( result = dispatch_event(
self.conn, "completed", "proj-a", {"task_id": "t-1"}, sender=None) self.conn, "completed", "proj-a", {"task_id": "t-1"},
sender={})
self.assertEqual(result["sent"], []) self.assertEqual(result["sent"], [])
self.assertEqual(len(result["errors"]), 1) self.assertEqual(len(result["errors"]), 1)
self.assertIn("no sender", result["errors"][0]["reason"]) self.assertIn("no sender", result["errors"][0]["reason"])
@@ -300,5 +308,141 @@ class TestDispatchIsolation(PushTestCase):
len(list_subscriptions(self.conn, "proj-b", "user-1")), 1) len(list_subscriptions(self.conn, "proj-b", "user-1")), 1)
class _RecordingHandler(http.server.BaseHTTPRequestHandler):
"""本地回环接收器:记录每次 POST 的头与 body,按类属性回状态。"""
requests = []
status = 200
def do_POST(self):
length = int(self.headers.get("Content-Length") or 0)
body = self.rfile.read(length) if length else b""
_RecordingHandler.requests.append({
"path": self.path,
"content_type": self.headers.get("Content-Type"),
"body": body,
})
self.send_response(_RecordingHandler.status)
self.send_header("Content-Length", "2")
self.end_headers()
self.wfile.write(b"{}")
def log_message(self, *args):
pass
class TestRealSenders(PushTestCase):
"""内置真实发送端:对本地回环服务器真 POST,断言方法/头/body;
不连外网。"""
def setUp(self):
super().setUp()
_RecordingHandler.requests = []
_RecordingHandler.status = 200
self.server = http.server.ThreadingHTTPServer(
("127.0.0.1", 0), _RecordingHandler)
self.port = self.server.server_address[1]
self.thread = threading.Thread(
target=self.server.serve_forever, daemon=True)
self.thread.start()
def tearDown(self):
self.server.shutdown()
self.server.server_close()
self.thread.join(timeout=5)
super().tearDown()
def _url(self, path="/hook"):
return "http://127.0.0.1:%d%s" % (self.port, path)
def test_webhook_posts_json_via_default_sender(self):
create_subscription(
self.conn, "proj-a", "user-1", "webhook", self._url(),
"instant")
result = dispatch_event(
self.conn, "completed", "proj-a", {"task_id": "t-1"},
sender=None)
self.assertEqual(len(result["sent"]), 1)
self.assertEqual(result["errors"], [])
self.assertEqual(len(_RecordingHandler.requests), 1)
req = _RecordingHandler.requests[0]
self.assertEqual(req["content_type"], "application/json")
body = json.loads(req["body"].decode("utf-8"))
self.assertEqual(body["event_type"], "completed")
self.assertEqual(body["project_id"], "proj-a")
self.assertEqual(body["task_id"], "t-1")
self.assertEqual(body["payload"], {"task_id": "t-1"})
def test_webhook_non_2xx_is_error_not_sent(self):
_RecordingHandler.status = 500
create_subscription(
self.conn, "proj-a", "user-1", "webhook", self._url(),
"instant")
result = dispatch_event(
self.conn, "failed", "proj-a", {"task_id": "t-1"},
sender=None)
self.assertEqual(result["sent"], [])
self.assertEqual(len(result["errors"]), 1)
self.assertIn("delivery failed", result["errors"][0]["reason"])
self.assertEqual(len(_RecordingHandler.requests), 1)
def test_webhook_non_http_target_is_error(self):
create_subscription(
self.conn, "proj-a", "user-1", "webhook",
"ftp://example.invalid/hook", "instant")
with self.assertRaises(ValueError):
send_webhook({
"event_type": "completed", "project_id": "proj-a",
"payload": {"task_id": "t-1"},
"channel": "webhook",
"target": "ftp://example.invalid/hook",
"user_id": "user-1", "subscription_id": "s-1"})
result = dispatch_event(
self.conn, "completed", "proj-a", {"task_id": "t-1"},
sender=None)
self.assertEqual(result["sent"], [])
self.assertIn("http(s) URL", result["errors"][0]["reason"])
def test_slack_posts_text_with_task_and_error(self):
create_subscription(
self.conn, "proj-a", "user-1", "slack", self._url(),
"instant")
result = dispatch_event(
self.conn, "failed", "proj-a",
{"task_id": "t-9", "error": "boom happened"},
sender=None)
self.assertEqual(len(result["sent"]), 1)
body = json.loads(_RecordingHandler.requests[0]["body"].decode())
self.assertIn("t-9", body["text"])
self.assertIn("failed", body["text"])
self.assertIn("boom happened", body["text"])
def test_webpush_still_missing_credential_with_default_sender(self):
create_subscription(
self.conn, "proj-a", "user-1", "webpush",
"https://example.invalid/endpoint", "instant")
result = dispatch_event(
self.conn, "failed", "proj-a", {"task_id": "t-1"},
sender=None)
self.assertEqual(result["sent"], [])
self.assertIn("missing credential", result["errors"][0]["reason"])
self.assertEqual(_RecordingHandler.requests, [])
def test_explicit_sender_wins_over_default(self):
create_subscription(
self.conn, "proj-a", "user-1", "webhook", self._url(),
"instant")
calls = []
def fake(message):
calls.append(message)
result = dispatch_event(
self.conn, "completed", "proj-a", {"task_id": "t-1"},
sender={"webhook": fake})
self.assertEqual(len(calls), 1)
self.assertEqual(len(result["sent"]), 1)
self.assertEqual(_RecordingHandler.requests, [])
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+5 -3
View File
@@ -126,7 +126,7 @@ class OAuthPushEndpointsTestCase(unittest.TestCase):
pid = self.project["id"] pid = self.project["id"]
status, sub = self._req( status, sub = self._req(
"POST", "/projects/%s/push/subscriptions" % pid, "POST", "/projects/%s/push/subscriptions" % pid,
{"channel": "webhook", "target": "https://example.invalid/hook", {"channel": "webhook", "target": "http://127.0.0.1:1/hook",
"preference": "instant"}, "preference": "instant"},
self.owner_key) self.owner_key)
self.assertEqual(status, 201, sub) self.assertEqual(status, 201, sub)
@@ -135,15 +135,17 @@ class OAuthPushEndpointsTestCase(unittest.TestCase):
key=self.owner_key) key=self.owner_key)
self.assertEqual(status, 200) self.assertEqual(status, 200)
self.assertEqual(len(body["subscriptions"]), 1) self.assertEqual(len(body["subscriptions"]), 1)
# HTTP 层无注入 sender:分发应报“无 sender”,不静默成功。 # 内置真实发送端会真 POST:目标不可达时分发应报投递失败
# (不是“无 sender”,也不是静默成功),sent 为空。
status, body = self._req( status, body = self._req(
"POST", "/admin/push/dispatch", "POST", "/admin/push/dispatch",
{"event_type": "failed", "project_id": pid, {"event_type": "failed", "project_id": pid,
"payload": {"task_id": "t1"}}, "payload": {"task_id": "t1"}},
self.owner_key) self.owner_key)
self.assertEqual(status, 200) self.assertEqual(status, 200)
self.assertEqual(body["sent"], [])
self.assertEqual(len(body["errors"]), 1) self.assertEqual(len(body["errors"]), 1)
self.assertIn("no sender", body["errors"][0]["reason"]) self.assertIn("delivery failed", body["errors"][0]["reason"])
# outsider 不能往别人项目里订阅/分发。 # outsider 不能往别人项目里订阅/分发。
status, _ = self._req( status, _ = self._req(
"POST", "/projects/%s/push/subscriptions" % pid, "POST", "/projects/%s/push/subscriptions" % pid,
+97
View File
@@ -126,6 +126,103 @@ class WorkerTestCase(unittest.TestCase):
# B 的依赖未完成,始终不可派,保持 queued。 # B 的依赖未完成,始终不可派,保持 queued。
self.assertEqual(_status(self.db_path, "B"), "queued") self.assertEqual(_status(self.db_path, "B"), "queued")
# -- 终态推送 --------------------------------------------------------
def test_notify_on_completed_and_failed(self):
"""任务终态自动通知订阅:completed 与 failed 各发一次真 POST
到本地回环 webhook,body 含事件类型与任务 id;取消
(interrupted)不在事件集合里,不通知(见下个测试)。"""
import http.server
import threading
requests = []
class Handler(http.server.BaseHTTPRequestHandler):
def do_POST(self):
length = int(self.headers.get("Content-Length") or 0)
body = self.rfile.read(length) if length else b""
requests.append(json.loads(body.decode("utf-8")))
self.send_response(200)
self.send_header("Content-Length", "2")
self.end_headers()
self.wfile.write(b"{}")
def log_message(self, *args):
pass
server = http.server.ThreadingHTTPServer(
("127.0.0.1", 0), Handler)
thread = threading.Thread(
target=server.serve_forever, daemon=True)
thread.start()
try:
from cloud.push import create_subscription
url = "http://127.0.0.1:%d/hook" % server.server_address[1]
create_subscription(
self.db_path, "p1", "user-1", "webhook", url, "instant")
_insert_task(self.db_path, "A")
_insert_task(self.db_path, "B")
def failing(cmd):
if "dispatch" in cmd and cmd[cmd.index("--name") + 1] == "B":
return SimpleNamespace(returncode=1, stdout="boom",
stderr="")
return _completed_runner(cmd)
worker = self._worker(runner=failing)
results = worker.run_all()
self.assertEqual(
sorted(r["status"] for r in results),
["completed", "failed"])
by_task = {b["task_id"]: b for b in requests}
self.assertEqual(set(by_task), {"A", "B"})
self.assertEqual(by_task["A"]["event_type"], "completed")
self.assertEqual(by_task["B"]["event_type"], "failed")
self.assertIn("boom", json.dumps(by_task["B"]["payload"]))
finally:
server.shutdown()
server.server_close()
thread.join(timeout=5)
def test_cancel_does_not_notify(self):
"""取消只标 interrupted,不在推送事件集合里,不发通知。"""
import http.server
import threading
requests = []
class Handler(http.server.BaseHTTPRequestHandler):
def do_POST(self):
length = int(self.headers.get("Content-Length") or 0)
self.rfile.read(length) if length else b""
requests.append(True)
self.send_response(200)
self.send_header("Content-Length", "2")
self.end_headers()
self.wfile.write(b"{}")
def log_message(self, *args):
pass
server = http.server.ThreadingHTTPServer(
("127.0.0.1", 0), Handler)
thread = threading.Thread(
target=server.serve_forever, daemon=True)
thread.start()
try:
from cloud.push import create_subscription
url = "http://127.0.0.1:%d/hook" % server.server_address[1]
create_subscription(
self.db_path, "p1", "user-1", "webhook", url, "instant")
_insert_task(self.db_path, "A")
worker = self._worker(runner=_completed_runner)
self.assertTrue(worker.cancel("A"))
self.assertEqual(worker.run_all(), [])
self.assertEqual(requests, [])
finally:
server.shutdown()
server.server_close()
thread.join(timeout=5)
# -- 取消 ------------------------------------------------------------ # -- 取消 ------------------------------------------------------------
def test_cancel_queued_task_marks_interrupted(self): def test_cancel_queued_task_marks_interrupted(self):
_insert_task(self.db_path, "A") _insert_task(self.db_path, "A")
+35
View File
@@ -640,6 +640,34 @@ class Worker:
except Exception: except Exception:
pass pass
def _notify_task_finished(self, task_id, project_id, event_type,
error=None):
"""任务终态推送:best-effort,绝不因通知失败影响任务状态。
只发 completed/failed(push.ALLOWED_PUSH_EVENTS 的子集;
interrupted 不在事件集合里,取消不通知)。无订阅时
dispatch 直接返回空 sent,不联网。同步发送、单次 10 秒
超时:慢订阅会拖住 worker 循环最多几秒,换取投递结果
可观测(失败记日志,不静默丢);后台队列是后续优化,
不在本轮。
"""
try:
from cloud.push import dispatch_event
result = dispatch_event(
self.db_path, event_type, project_id,
{"task_id": task_id, "status": event_type,
"executor": self._executor_name(), "error": error})
if result.get("errors"):
log_event(self.logger, "worker_notify_failed",
task_id=task_id, project_id=project_id,
status=event_type,
message="notify errors: %s" % (result["errors"],))
except Exception as exc: # noqa: BLE001 -- 通知失败绝不炸任务
log_event(self.logger, "worker_notify_failed",
task_id=task_id, project_id=project_id,
status=event_type,
message="notify exception: %s" % exc)
# -- 认领 ---------------------------------------------------------- # -- 认领 ----------------------------------------------------------
def _budget_exceeded(self): def _budget_exceeded(self):
return budget_exceeded(self.usage_turns, self.max_turns, return budget_exceeded(self.usage_turns, self.max_turns,
@@ -1346,6 +1374,8 @@ class Worker:
self._audit(task["project_id"], "worker_missing_key", error) self._audit(task["project_id"], "worker_missing_key", error)
log_event(self.logger, "worker_missing_key", task_id=task_id, log_event(self.logger, "worker_missing_key", task_id=task_id,
project_id=task["project_id"], status="failed") project_id=task["project_id"], status="failed")
self._notify_task_finished(
task_id, task["project_id"], "failed", error=error)
return {"dispatched": True, "task_id": task_id, return {"dispatched": True, "task_id": task_id,
"host_name": self.executor._host_name(task) "host_name": self.executor._host_name(task)
if hasattr(self.executor, "_host_name") else task_id, if hasattr(self.executor, "_host_name") else task_id,
@@ -1365,6 +1395,8 @@ class Worker:
self._audit(task["project_id"], "worker_repo_failed", repo_error) self._audit(task["project_id"], "worker_repo_failed", repo_error)
log_event(self.logger, "worker_repo_failed", task_id=task_id, log_event(self.logger, "worker_repo_failed", task_id=task_id,
project_id=task["project_id"], status="failed") project_id=task["project_id"], status="failed")
self._notify_task_finished(
task_id, task["project_id"], "failed", error=repo_error)
return {"dispatched": True, "task_id": task_id, return {"dispatched": True, "task_id": task_id,
"host_name": self.executor._host_name(task) "host_name": self.executor._host_name(task)
if hasattr(self.executor, "_host_name") else task_id, if hasattr(self.executor, "_host_name") else task_id,
@@ -1537,6 +1569,9 @@ class Worker:
else "worker_failed", else "worker_failed",
task_id=task_id, project_id=task["project_id"], task_id=task_id, project_id=task["project_id"],
status=final) status=final)
if final in ("completed", "failed"):
self._notify_task_finished(
task_id, task["project_id"], final, error=error)
else: else:
# 已被取消/回收:DB 状态以 interrupted 为准,不再 # 已被取消/回收:DB 状态以 interrupted 为准,不再
# 补一条 worker_completed/failed 审计误导排查。 # 补一条 worker_completed/failed 审计误导排查。