Merge pull request #275 from openqj/feat-resident-website-loop-audit
合并网站常驻监听 PR #275:resident-watches + pause/resume/end、闭环实测、审核架构 10/10 与代码质量 10/10。CI host-verify 通过(6m52s),mergeable CLEAN。
This commit is contained in:
9 files changed
+2511
No files matched your search
@@ -0,0 +1,846 @@
|
||||
"""常驻监听任务(resident watch)。
|
||||
|
||||
与现有三样东西的关系,先说清楚,避免把旧语义冒充成常驻:
|
||||
|
||||
- 任务状态机(cloud/server.py):queued -> running ->
|
||||
completed / failed / interrupted / merged / merge-failed,
|
||||
其中 /pause 只是把进行中回合任务标 interrupted 并置
|
||||
paused_by_user=1(见 server._pause_task),不是一个可续跑
|
||||
的“已暂停常驻态”。本模块绝不复用那条路:常驻态存在
|
||||
resident_watches 表里,与任务表互不干扰。
|
||||
- 云端自动化(cloud/automations.py):定义层只有 active /
|
||||
cancelled,到期靠外部触发器调 run_due_automation 抢单建任务,
|
||||
服务端本身不起定时器。本模块复用它的两道口径而不是另起
|
||||
一套:抢单用条件 UPDATE(同一间隔内只有一个执行器能拿到
|
||||
rowcount 1),配额门直接调 automations._automation_quota_
|
||||
exceeded(与 server / worker 同口径,改口径时只改那一处)。
|
||||
- goal 循环(host.py run_goal_loop,max_rounds=0 不限轮次):
|
||||
目标是“做完一个目标”,结束条件是模型宣告达成;本模块的
|
||||
目标是“一直监听外部变化”,结束条件只能是用户结束(或
|
||||
配额耗尽转暂停,见下),两者不是一回事。
|
||||
|
||||
状态机(本模块自有,与任务状态机不冲突):
|
||||
running -> paused -> running(恢复)
|
||||
running / paused -> ended(终态)
|
||||
ended -> 任何操作一律拒绝(恢复必须新建任务,见 end_watch)
|
||||
|
||||
暂停保留现场:暂停只改 status / paused_reason / next_poll_at,
|
||||
last_content_hash、last_poll_at、last_change_at、last_change_text
|
||||
一律不动;暂停期间 due 为 False,驱动绝不轮询。恢复时
|
||||
next_poll_at 置为 now(立刻复查一次),用旧游标比对:暂停
|
||||
期间网站变了,恢复后第一次轮询就会把那次变化报出来,而不是
|
||||
丢掉——“从游标续跑”就是这个意思。
|
||||
|
||||
结束是终态:ended 后 next_poll_at 为 NULL,驱动的 due 查询
|
||||
只认 status='running',所以结束后绝不会静默重启;ended 行
|
||||
保留在库里供审计与查询,名字可重用(与自动化 cancelled 的
|
||||
名字重用规则同款)。
|
||||
|
||||
变化才花钱:轮询本身是廉价抓取 + sha256 比对,不跑模型。
|
||||
只有三类轮询产出:baseline(首次抓取,只建游标,不上报)、
|
||||
unchanged(游标不变,不建任务)、changed(游标变了,才建
|
||||
一个 queued 任务,prompt = 用户的处理提示 + 变化摘录)。
|
||||
每次轮询都在 resident_polls 留一行,可查。
|
||||
|
||||
配额边界:配额门只在“变化要建模型任务”那一刻过,不在
|
||||
轮询那一刻过(轮询不花模型额度)。配额耗尽时不结束、不丢
|
||||
变化:状态转 paused、paused_reason='quota_exceeded',游标不
|
||||
推进(恢复后复查仍会报出同一变化),轮询记
|
||||
changed_quota_paused。配额检查本身失败(fail-closed 文本)
|
||||
时不转暂停,按瞬时失败退避处理,游标同样不推进。
|
||||
|
||||
驱动:服务端本身不起定时器(与自动化同款结论)。真正的
|
||||
驱动是 cloud/worker.py 的 run_loop 轮次边界调
|
||||
process_due_resident_watches(见 worker 里的调用点),或
|
||||
任何外部触发器直接调本模块的 run_due_resident_watch /
|
||||
process_due_resident_watches;两者都走 claim 门,两个驱动
|
||||
同时跑也不会在同一间隔内重复轮询。
|
||||
|
||||
本模块只依赖标准库 + cloud.automations 的复用函数
|
||||
(row_get、_audit_row、_automation_quota_exceeded)与
|
||||
cloud.db 的完整性错误类型(仅用于把唯一索引冲突转成
|
||||
ValueError,不 import 其它 cloud 模块),避免循环 import。
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import ipaddress
|
||||
import re
|
||||
import sqlite3
|
||||
import time
|
||||
import urllib.request
|
||||
import uuid
|
||||
from urllib.parse import urljoin
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from cloud.automations import _audit_row as _auto_audit_row
|
||||
from cloud.automations import _automation_quota_exceeded
|
||||
from cloud.automations import row_get
|
||||
from cloud.db import PostgresIntegrityError as _PgIntegrityError
|
||||
|
||||
MIN_INTERVAL_SECONDS = 60
|
||||
MAX_INTERVAL_SECONDS = 30 * 86400
|
||||
MAX_NAME_LEN = 100
|
||||
MAX_PROMPT_LEN = 20000
|
||||
MAX_URL_LEN = 2000
|
||||
MIN_FETCH_TIMEOUT_SECONDS = 1
|
||||
MAX_FETCH_TIMEOUT_SECONDS = 60
|
||||
DEFAULT_FETCH_TIMEOUT_SECONDS = 10
|
||||
MAX_FETCH_BYTES = 1000000
|
||||
MAX_CHANGE_TEXT_LEN = 2000
|
||||
MAX_POLL_DETAIL_LEN = 500
|
||||
POLL_LIMIT = 50
|
||||
|
||||
RUNNING = "running"
|
||||
PAUSED = "paused"
|
||||
ENDED = "ended"
|
||||
|
||||
# 变化判定口径(v1):对抓取文本做空白折叠后取 sha256。
|
||||
# 动态页面(广告、时间戳)本身会抖动,v1 不做选择器/区域
|
||||
# 提取,抖动风险写在验证摘要的已知偏差里,不假装已解决。
|
||||
_WS_RE = re.compile(r"\s+")
|
||||
|
||||
|
||||
WATCH_COLUMNS = (
|
||||
"id", "project_id", "name", "target_url", "prompt",
|
||||
"interval_seconds", "fetch_timeout_seconds", "status",
|
||||
"last_poll_at", "last_claimed_at", "next_poll_at",
|
||||
"last_content_hash", "last_change_at", "last_change_text",
|
||||
"consecutive_failures", "paused_reason", "ended_at",
|
||||
"created_by", "created_at", "updated_at")
|
||||
|
||||
POLL_COLUMNS = (
|
||||
"id", "watch_id", "project_id", "status", "observed_hash",
|
||||
"detail", "task_id", "created_at")
|
||||
|
||||
|
||||
def _row_dict(row, columns):
|
||||
if row is None or isinstance(row, dict):
|
||||
return row
|
||||
try:
|
||||
row["id"]
|
||||
return row
|
||||
except (KeyError, IndexError, TypeError):
|
||||
# sqlite3.Row 缺列报 IndexError(不是 TypeError):只有
|
||||
# SELECT * 的行能原样用,单列查询(如 claim 取 interval)
|
||||
# 走 zip 回落,columns 必须与 SELECT 列顺序一致。
|
||||
return dict(zip(columns, row))
|
||||
|
||||
|
||||
def _existing_columns(conn, table):
|
||||
try:
|
||||
return {row[1] for row in conn.execute(
|
||||
"PRAGMA table_info(%s)" % table).fetchall()}
|
||||
except Exception:
|
||||
return set()
|
||||
|
||||
|
||||
def ensure_resident_tables(conn):
|
||||
"""幂等建常驻监听表,补齐新增列(仿 automations 写法)。"""
|
||||
conn.execute(
|
||||
"CREATE TABLE IF NOT EXISTS resident_watches("
|
||||
"id TEXT PRIMARY KEY, "
|
||||
"project_id TEXT, "
|
||||
"name TEXT, "
|
||||
"target_url TEXT, "
|
||||
"prompt TEXT, "
|
||||
"interval_seconds INTEGER, "
|
||||
"fetch_timeout_seconds INTEGER, "
|
||||
"status TEXT, "
|
||||
"last_poll_at DOUBLE PRECISION, "
|
||||
"last_claimed_at DOUBLE PRECISION, "
|
||||
"next_poll_at DOUBLE PRECISION, "
|
||||
"last_content_hash TEXT, "
|
||||
"last_change_at DOUBLE PRECISION, "
|
||||
"last_change_text TEXT, "
|
||||
"consecutive_failures INTEGER, "
|
||||
"paused_reason TEXT, "
|
||||
"ended_at DOUBLE PRECISION, "
|
||||
"created_by TEXT, "
|
||||
"created_at DOUBLE PRECISION, "
|
||||
"updated_at DOUBLE PRECISION)")
|
||||
conn.execute(
|
||||
"CREATE TABLE IF NOT EXISTS resident_polls("
|
||||
"id TEXT PRIMARY KEY, "
|
||||
"watch_id TEXT, "
|
||||
"project_id TEXT, "
|
||||
"status TEXT, "
|
||||
"observed_hash TEXT, "
|
||||
"detail TEXT, "
|
||||
"task_id TEXT, "
|
||||
"created_at DOUBLE PRECISION)")
|
||||
existing = _existing_columns(conn, "resident_watches")
|
||||
for col, ddl in (
|
||||
("fetch_timeout_seconds", "INTEGER"),
|
||||
("next_poll_at", "DOUBLE PRECISION"),
|
||||
("last_content_hash", "TEXT"),
|
||||
("last_change_at", "DOUBLE PRECISION"),
|
||||
("last_change_text", "TEXT"),
|
||||
("consecutive_failures", "INTEGER"),
|
||||
("paused_reason", "TEXT"),
|
||||
("ended_at", "DOUBLE PRECISION")):
|
||||
if col not in existing:
|
||||
try:
|
||||
conn.execute(
|
||||
"ALTER TABLE resident_watches ADD COLUMN %s %s"
|
||||
% (col, ddl))
|
||||
except Exception:
|
||||
pass
|
||||
conn.execute(
|
||||
"CREATE INDEX IF NOT EXISTS idx_resident_watches_project"
|
||||
" ON resident_watches(project_id, status)")
|
||||
# 同项目未结束名唯一:先查后插挡不住并发,唯一索引才是
|
||||
# 真正的关口(server environments 的同款部分唯一索引,
|
||||
# WHERE status != 'archived' 的 resident 版);已结束名
|
||||
# 不在索引内,可重用。已存在重复名的旧库建索引会失败,
|
||||
# 此时不炸掉整次建表调用,SELECT 查重仍挡正常路径。
|
||||
try:
|
||||
conn.execute(
|
||||
"CREATE UNIQUE INDEX IF NOT EXISTS"
|
||||
" idx_resident_watches_name"
|
||||
" ON resident_watches(project_id, name)"
|
||||
" WHERE status != 'ended'")
|
||||
except Exception:
|
||||
pass
|
||||
conn.execute(
|
||||
"CREATE INDEX IF NOT EXISTS idx_resident_polls_watch"
|
||||
" ON resident_polls(watch_id, created_at)")
|
||||
|
||||
|
||||
def _backoff_seconds(failures):
|
||||
"""失败退避增量:30、60、120……秒,上限 3600 秒。
|
||||
调用方把它加在间隔之上(下次 = now + interval + backoff),
|
||||
失败后绝不比正常间隔更频繁地重试,行为见失败分支测试。"""
|
||||
if failures <= 0:
|
||||
return 0
|
||||
return min(3600, 30 * (2 ** (failures - 1)))
|
||||
|
||||
|
||||
def watch_to_dict(row, now=None):
|
||||
row = _row_dict(row, WATCH_COLUMNS)
|
||||
if now is None:
|
||||
now = time.time()
|
||||
status = row["status"]
|
||||
nxt = row_get(row, "next_poll_at")
|
||||
due = bool(status == RUNNING and (nxt is None or nxt <= now))
|
||||
return {
|
||||
"id": row["id"],
|
||||
"project_id": row["project_id"],
|
||||
"name": row["name"],
|
||||
"target_url": row["target_url"],
|
||||
"prompt": row["prompt"],
|
||||
"interval_seconds": row["interval_seconds"],
|
||||
"fetch_timeout_seconds": row_get(row, "fetch_timeout_seconds")
|
||||
or DEFAULT_FETCH_TIMEOUT_SECONDS,
|
||||
"status": status,
|
||||
"last_poll_at": row_get(row, "last_poll_at"),
|
||||
"last_claimed_at": row_get(row, "last_claimed_at"),
|
||||
"next_poll_at": nxt,
|
||||
"last_content_hash": row_get(row, "last_content_hash"),
|
||||
"last_change_at": row_get(row, "last_change_at"),
|
||||
"last_change_text": row_get(row, "last_change_text"),
|
||||
"consecutive_failures": row_get(row, "consecutive_failures") or 0,
|
||||
"paused_reason": row_get(row, "paused_reason"),
|
||||
"ended_at": row_get(row, "ended_at"),
|
||||
"created_by": row["created_by"],
|
||||
"created_at": row["created_at"],
|
||||
"updated_at": row_get(row, "updated_at"),
|
||||
"due": due,
|
||||
}
|
||||
|
||||
|
||||
def _is_noncanonical_numeric_host(host):
|
||||
"""非规范数字 IP 写法:单整数(2130706433)、十六进制
|
||||
(0x7f000001)、前导零八进制(0177.0.0.1)、含十六进制段
|
||||
的点分写法——urllib/socket 的 inet_aton 会把它们解析成
|
||||
数字 IP,ipaddress 却认不出来。规范点分十进制(4 段、
|
||||
每段 0-255、无前导零,0 本身除外)不在此列;普通域名
|
||||
(含字母的标签)也不在此列。"""
|
||||
if re.fullmatch(r"0[xX][0-9a-fA-F]+", host):
|
||||
return True
|
||||
if re.fullmatch(r"[0-9]+", host):
|
||||
return True
|
||||
if "." not in host:
|
||||
return False
|
||||
labels = host.split(".")
|
||||
if not all(re.fullmatch(r"0[xX][0-9a-fA-F]+|[0-9]+", lab)
|
||||
for lab in labels):
|
||||
return False
|
||||
if len(labels) != 4:
|
||||
return True
|
||||
for lab in labels:
|
||||
if lab.startswith("0x") or lab.startswith("0X"):
|
||||
return True
|
||||
if len(lab) > 1 and lab.startswith("0"):
|
||||
return True
|
||||
if int(lab) > 255:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def validate_target_url(url):
|
||||
"""目标 URL 校验:仅 http/https、无 userinfo、主机非空、
|
||||
字面 localhost(含尾点、*.localhost)/ 私有 / 回环 /
|
||||
链路本地 IP / 非规范数字 IP 写法拒绝(与
|
||||
websearch._is_blocked_fetch_url 的字面 IP 口径同款,
|
||||
DNS 解析后的私有地址不在 v1 覆盖内,见摘要已知偏差;
|
||||
重定向目标由 _SafeRedirectHandler 逐跳复用本校验)。"""
|
||||
if not url or not isinstance(url, str) or not url.strip():
|
||||
raise ValueError("target_url required")
|
||||
url = url.strip()
|
||||
if len(url) > MAX_URL_LEN:
|
||||
raise ValueError("target_url too long")
|
||||
parts = urlsplit(url)
|
||||
if parts.scheme not in ("http", "https"):
|
||||
raise ValueError("target_url must be http or https")
|
||||
if parts.username or parts.password:
|
||||
raise ValueError("target_url must not contain credentials")
|
||||
host = (parts.hostname or "").strip().lower().rstrip(".")
|
||||
if not host:
|
||||
raise ValueError("target_url host required")
|
||||
if host == "localhost" or host.endswith(".localhost"):
|
||||
raise ValueError("localhost URLs are blocked")
|
||||
if _is_noncanonical_numeric_host(host):
|
||||
raise ValueError("non-canonical numeric IP URLs are blocked")
|
||||
try:
|
||||
addr = ipaddress.ip_address(host)
|
||||
except ValueError:
|
||||
addr = None
|
||||
if addr is not None and (addr.is_private or addr.is_loopback
|
||||
or addr.is_link_local or addr.is_reserved
|
||||
or addr.is_multicast):
|
||||
raise ValueError("private / loopback URLs are blocked")
|
||||
return url
|
||||
|
||||
|
||||
def _validate_fields(name, prompt, target_url, interval_seconds,
|
||||
fetch_timeout_seconds):
|
||||
if not name or not isinstance(name, str) or not name.strip():
|
||||
raise ValueError("name required")
|
||||
name = name.strip()
|
||||
if len(name) > MAX_NAME_LEN:
|
||||
raise ValueError("name too long")
|
||||
if not prompt or not isinstance(prompt, str) or not prompt.strip():
|
||||
raise ValueError("prompt required")
|
||||
prompt = prompt.strip()
|
||||
if len(prompt) > MAX_PROMPT_LEN:
|
||||
raise ValueError("prompt too long")
|
||||
target_url = validate_target_url(target_url)
|
||||
if isinstance(interval_seconds, bool) \
|
||||
or not isinstance(interval_seconds, int):
|
||||
raise ValueError("interval_seconds must be an integer")
|
||||
if interval_seconds < MIN_INTERVAL_SECONDS:
|
||||
raise ValueError(
|
||||
"interval_seconds must be at least %d" % MIN_INTERVAL_SECONDS)
|
||||
if interval_seconds > MAX_INTERVAL_SECONDS:
|
||||
raise ValueError("interval_seconds too large")
|
||||
if fetch_timeout_seconds is None:
|
||||
fetch_timeout_seconds = DEFAULT_FETCH_TIMEOUT_SECONDS
|
||||
if isinstance(fetch_timeout_seconds, bool) \
|
||||
or not isinstance(fetch_timeout_seconds, int):
|
||||
raise ValueError("fetch_timeout_seconds must be an integer")
|
||||
if not MIN_FETCH_TIMEOUT_SECONDS <= fetch_timeout_seconds \
|
||||
<= MAX_FETCH_TIMEOUT_SECONDS:
|
||||
raise ValueError(
|
||||
"fetch_timeout_seconds must be between %d and %d"
|
||||
% (MIN_FETCH_TIMEOUT_SECONDS, MAX_FETCH_TIMEOUT_SECONDS))
|
||||
return name, prompt, target_url, interval_seconds, fetch_timeout_seconds
|
||||
|
||||
|
||||
def create_watch(conn, project_id, name, target_url, prompt, created_by,
|
||||
interval_seconds=3600, fetch_timeout_seconds=None,
|
||||
now=None):
|
||||
"""新建常驻监听:建完即到期(next_poll_at=now),驱动下
|
||||
一轮即可认领;同项目内未结束(running/paused)的名字唯一,
|
||||
已结束的名字可重用。创建本身不走配额门(轮询不花模型额度,
|
||||
配额只在变化建任务时过,见 run_due_resident_watch)。"""
|
||||
(name, prompt, target_url, interval_seconds,
|
||||
fetch_timeout_seconds) = _validate_fields(
|
||||
name, prompt, target_url, interval_seconds, fetch_timeout_seconds)
|
||||
if now is None:
|
||||
now = time.time()
|
||||
ensure_resident_tables(conn)
|
||||
clash = conn.execute(
|
||||
"SELECT id FROM resident_watches WHERE project_id = ?"
|
||||
" AND name = ? AND status != ?",
|
||||
(project_id, name, ENDED)).fetchone()
|
||||
if clash is not None:
|
||||
raise ValueError(
|
||||
"resident watch name already exists in this project")
|
||||
watch_id = uuid.uuid4().hex
|
||||
try:
|
||||
conn.execute(
|
||||
"INSERT INTO resident_watches(id, project_id, name, target_url,"
|
||||
" prompt, interval_seconds, fetch_timeout_seconds, status,"
|
||||
" last_poll_at, last_claimed_at, next_poll_at,"
|
||||
" last_content_hash, last_change_at, last_change_text,"
|
||||
" consecutive_failures, paused_reason, ended_at,"
|
||||
" created_by, created_at, updated_at)"
|
||||
" VALUES(?, ?, ?, ?, ?, ?, ?, ?, NULL, NULL, ?,"
|
||||
" NULL, NULL, NULL, 0, NULL, NULL, ?, ?, ?)",
|
||||
(watch_id, project_id, name, target_url, prompt,
|
||||
interval_seconds, fetch_timeout_seconds, RUNNING, now,
|
||||
created_by, now, now))
|
||||
except (sqlite3.IntegrityError, _PgIntegrityError) as exc:
|
||||
# 并发下 SELECT 查重与 INSERT 之间可能插进同名行,
|
||||
# 唯一索引是真正的关口:冲突转回与查重同款 ValueError,
|
||||
# 调用方(server)据此返回 400,不把数据库错误外泄。
|
||||
try:
|
||||
conn.rollback()
|
||||
except Exception:
|
||||
pass
|
||||
raise ValueError(
|
||||
"resident watch name already exists in this project") from exc
|
||||
conn.commit()
|
||||
row = conn.execute(
|
||||
"SELECT * FROM resident_watches WHERE id = ?", (watch_id,)).fetchone()
|
||||
return watch_to_dict(row, now)
|
||||
|
||||
|
||||
def list_watches(conn, project_id, now=None):
|
||||
if now is None:
|
||||
now = time.time()
|
||||
ensure_resident_tables(conn)
|
||||
rows = conn.execute(
|
||||
"SELECT * FROM resident_watches WHERE project_id = ?"
|
||||
" ORDER BY created_at, id",
|
||||
(project_id,)).fetchall()
|
||||
return [watch_to_dict(r, now) for r in rows]
|
||||
|
||||
|
||||
def get_watch(conn, watch_id, now=None):
|
||||
if now is None:
|
||||
now = time.time()
|
||||
ensure_resident_tables(conn)
|
||||
row = conn.execute(
|
||||
"SELECT * FROM resident_watches WHERE id = ?", (watch_id,)).fetchone()
|
||||
if row is None:
|
||||
return None
|
||||
return watch_to_dict(row, now)
|
||||
|
||||
|
||||
def list_polls(conn, watch_id, limit=POLL_LIMIT):
|
||||
ensure_resident_tables(conn)
|
||||
rows = conn.execute(
|
||||
"SELECT * FROM resident_polls WHERE watch_id = ?"
|
||||
" ORDER BY created_at DESC, id LIMIT ?",
|
||||
(watch_id, limit)).fetchall()
|
||||
out = []
|
||||
for raw in rows:
|
||||
r = _row_dict(raw, POLL_COLUMNS)
|
||||
out.append({
|
||||
"id": r["id"],
|
||||
"watch_id": r["watch_id"],
|
||||
"project_id": r["project_id"],
|
||||
"status": r["status"],
|
||||
"observed_hash": row_get(r, "observed_hash"),
|
||||
"detail": row_get(r, "detail") or "",
|
||||
"task_id": row_get(r, "task_id"),
|
||||
"created_at": r["created_at"],
|
||||
})
|
||||
return out
|
||||
|
||||
|
||||
def _transition(conn, watch_id, from_statuses, to_status, now,
|
||||
extra_sets=(), extra_vals=()):
|
||||
"""条件状态迁移:返回 (ok, dict)。行不存在时 (False, None);
|
||||
状态不对时 (False, 旧 dict),调用方据此区分 404 与 409。"""
|
||||
ensure_resident_tables(conn)
|
||||
row = conn.execute(
|
||||
"SELECT * FROM resident_watches WHERE id = ?",
|
||||
(watch_id,)).fetchone()
|
||||
if row is None:
|
||||
return False, None
|
||||
row = _row_dict(row, WATCH_COLUMNS)
|
||||
if row["status"] not in from_statuses:
|
||||
return False, watch_to_dict(row, now)
|
||||
placeholders = ",".join("?" for _ in from_statuses)
|
||||
sets = ["status = ?", "updated_at = ?"] + list(extra_sets)
|
||||
vals = [to_status, now] + list(extra_vals) + [watch_id] \
|
||||
+ list(from_statuses)
|
||||
cur = conn.execute(
|
||||
"UPDATE resident_watches SET %s WHERE id = ? AND status IN (%s)"
|
||||
% (", ".join(sets), placeholders), tuple(vals))
|
||||
conn.commit()
|
||||
if cur.rowcount != 1:
|
||||
return False, watch_to_dict(row, now)
|
||||
row = conn.execute(
|
||||
"SELECT * FROM resident_watches WHERE id = ?", (watch_id,)).fetchone()
|
||||
return True, watch_to_dict(row, now)
|
||||
|
||||
|
||||
def pause_watch(conn, watch_id, now=None, reason="user_paused"):
|
||||
"""用户暂停:仅 running 能暂停;保留全部现场,只把
|
||||
next_poll_at 清掉,驱动即停。"""
|
||||
if now is None:
|
||||
now = time.time()
|
||||
return _transition(conn, watch_id, (RUNNING,), PAUSED, now,
|
||||
extra_sets=("paused_reason = ?",
|
||||
"next_poll_at = NULL"),
|
||||
extra_vals=(reason,))
|
||||
|
||||
|
||||
def resume_watch(conn, watch_id, now=None):
|
||||
"""恢复:仅 paused 能恢复;现场不动,next_poll_at=now
|
||||
立刻复查一次(用旧游标比对,暂停期间的变化不丢)。"""
|
||||
if now is None:
|
||||
now = time.time()
|
||||
return _transition(conn, watch_id, (PAUSED,), RUNNING, now,
|
||||
extra_sets=("paused_reason = NULL",
|
||||
"next_poll_at = ?"),
|
||||
extra_vals=(now,))
|
||||
|
||||
|
||||
def end_watch(conn, watch_id, now=None):
|
||||
"""结束:running / paused -> ended,终态。ended 后
|
||||
pause / resume / end 一律 (False, 旧 dict),调用方转 409;
|
||||
恢复已结束的监听必须新建任务,不复活本行。"""
|
||||
if now is None:
|
||||
now = time.time()
|
||||
return _transition(conn, watch_id, (RUNNING, PAUSED), ENDED, now,
|
||||
extra_sets=("ended_at = ?",
|
||||
"next_poll_at = NULL"),
|
||||
extra_vals=(now,))
|
||||
|
||||
|
||||
def update_watch(conn, watch_id, fields, now=None):
|
||||
"""更新定义:只认 name / prompt / target_url /
|
||||
interval_seconds / fetch_timeout_seconds;ended 的行 409
|
||||
(终态不能改,改它必须重建);改间隔不重置游标、
|
||||
不立即触发:已排好的 next_poll_at 不变,下一次轮询
|
||||
完成后才按新间隔重算(想立即复查请暂停后恢复)。"""
|
||||
if now is None:
|
||||
now = time.time()
|
||||
ensure_resident_tables(conn)
|
||||
row = conn.execute(
|
||||
"SELECT * FROM resident_watches WHERE id = ?", (watch_id,)).fetchone()
|
||||
if row is None:
|
||||
return False, None
|
||||
row = _row_dict(row, WATCH_COLUMNS)
|
||||
if row["status"] == ENDED:
|
||||
return False, watch_to_dict(row, now)
|
||||
if not isinstance(fields, dict):
|
||||
raise ValueError("fields must be an object")
|
||||
if not any(k in fields for k in (
|
||||
"name", "prompt", "target_url", "interval_seconds",
|
||||
"fetch_timeout_seconds")):
|
||||
raise ValueError("no updatable field given")
|
||||
current = watch_to_dict(row, now)
|
||||
(name, prompt, target_url, interval, timeout) = _validate_fields(
|
||||
fields.get("name", current["name"]),
|
||||
fields.get("prompt", current["prompt"]),
|
||||
fields.get("target_url", current["target_url"]),
|
||||
fields.get("interval_seconds", current["interval_seconds"]),
|
||||
fields.get("fetch_timeout_seconds",
|
||||
current["fetch_timeout_seconds"]))
|
||||
if name != current["name"]:
|
||||
clash = conn.execute(
|
||||
"SELECT id FROM resident_watches WHERE project_id = ?"
|
||||
" AND name = ? AND status != ? AND id != ?",
|
||||
(row["project_id"], name, ENDED, watch_id)).fetchone()
|
||||
if clash is not None:
|
||||
raise ValueError(
|
||||
"resident watch name already exists in this project")
|
||||
try:
|
||||
conn.execute(
|
||||
"UPDATE resident_watches SET name = ?, prompt = ?,"
|
||||
" target_url = ?, interval_seconds = ?,"
|
||||
" fetch_timeout_seconds = ?, updated_at = ? WHERE id = ?",
|
||||
(name, prompt, target_url, interval, timeout, now, watch_id))
|
||||
except (sqlite3.IntegrityError, _PgIntegrityError) as exc:
|
||||
try:
|
||||
conn.rollback()
|
||||
except Exception:
|
||||
pass
|
||||
raise ValueError(
|
||||
"resident watch name already exists in this project") from exc
|
||||
conn.commit()
|
||||
row = conn.execute(
|
||||
"SELECT * FROM resident_watches WHERE id = ?", (watch_id,)).fetchone()
|
||||
return True, watch_to_dict(row, now)
|
||||
|
||||
|
||||
def due_watches(conn, now=None):
|
||||
"""全部项目当前到期的 running 监听(纯读,不抢单),
|
||||
供驱动列出待处理项;执行入口是 claim + run_due。"""
|
||||
if now is None:
|
||||
now = time.time()
|
||||
ensure_resident_tables(conn)
|
||||
rows = conn.execute(
|
||||
"SELECT * FROM resident_watches WHERE status = ?"
|
||||
" AND (next_poll_at IS NULL OR next_poll_at <= ?)"
|
||||
" ORDER BY next_poll_at, created_at, id",
|
||||
(RUNNING, now)).fetchall()
|
||||
return [watch_to_dict(r, now) for r in rows]
|
||||
|
||||
|
||||
def claim_due_watch(conn, watch_id, now=None):
|
||||
"""抢单:条件 UPDATE 只有一行能成功。抢到即把
|
||||
last_poll_at / last_claimed_at 写成 now,并把 next_poll_at
|
||||
预置为 now + interval(崩在抓取中途时,下个间隔仍会重试,
|
||||
不会因认领锁死而永久停摆;抓取完成后 run_due 会按成败
|
||||
重算 next_poll_at)。抢不到时返回 False,调用方必须放弃。"""
|
||||
if now is None:
|
||||
now = time.time()
|
||||
ensure_resident_tables(conn)
|
||||
row = conn.execute(
|
||||
"SELECT interval_seconds FROM resident_watches WHERE id = ?",
|
||||
(watch_id,)).fetchone()
|
||||
if row is None:
|
||||
return False
|
||||
interval = row_get(_row_dict(row, ("interval_seconds",)),
|
||||
"interval_seconds") or MIN_INTERVAL_SECONDS
|
||||
cur = conn.execute(
|
||||
"UPDATE resident_watches SET last_poll_at = ?,"
|
||||
" last_claimed_at = ?, next_poll_at = ?"
|
||||
" WHERE id = ? AND status = ?"
|
||||
" AND (next_poll_at IS NULL OR next_poll_at <= ?)",
|
||||
(now, now, now + interval, watch_id, RUNNING, now))
|
||||
conn.commit()
|
||||
return cur.rowcount == 1
|
||||
|
||||
|
||||
def _content_hash(text):
|
||||
normalized = _WS_RE.sub(" ", text or "").strip()
|
||||
return hashlib.sha256(normalized.encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
class _SafeRedirectHandler(urllib.request.HTTPRedirectHandler):
|
||||
"""重定向逐跳校验:创建时只校验了初始 URL,公开 URL
|
||||
重定向到内网地址会绕过那道校验;相对跳转先 urljoin 成
|
||||
绝对 URL 再校验,不合格直接抛 ValueError,不跟随。"""
|
||||
|
||||
def redirect_request(self, req, fp, code, msg, headers, newurl):
|
||||
validate_target_url(urljoin(req.full_url, newurl))
|
||||
return super().redirect_request(req, fp, code, msg, headers,
|
||||
newurl)
|
||||
|
||||
|
||||
def default_fetch(url, timeout):
|
||||
"""默认廉价抓取:urllib,超时可配,读满 1MB 即断,
|
||||
解码失败按 replace 处理。调用方可注入 fetcher 替换它
|
||||
(测试与未来选择器提取都走注入点,不改本函数签名)。
|
||||
build_opener 带默认处理器(含 ProxyHandler,no_proxy
|
||||
行为与 urlopen 同款),只把重定向处理器换成逐跳校验版。"""
|
||||
req = urllib.request.Request(url, headers={
|
||||
"User-Agent": "codex-app-server-resident-watch/1.0"})
|
||||
opener = urllib.request.build_opener(_SafeRedirectHandler())
|
||||
with opener.open(req, timeout=timeout) as resp:
|
||||
raw = resp.read(MAX_FETCH_BYTES + 1)[:MAX_FETCH_BYTES]
|
||||
return raw.decode("utf-8", errors="replace")
|
||||
|
||||
|
||||
def _record_poll(conn, watch_id, project_id, status, observed_hash,
|
||||
detail, task_id, now):
|
||||
poll_id = uuid.uuid4().hex
|
||||
conn.execute(
|
||||
"INSERT INTO resident_polls(id, watch_id, project_id, status,"
|
||||
" observed_hash, detail, task_id, created_at)"
|
||||
" VALUES(?, ?, ?, ?, ?, ?, ?, ?)",
|
||||
(poll_id, watch_id, project_id, status, observed_hash,
|
||||
(detail or "")[:MAX_POLL_DETAIL_LEN], task_id, now))
|
||||
return {
|
||||
"id": poll_id,
|
||||
"watch_id": watch_id,
|
||||
"project_id": project_id,
|
||||
"status": status,
|
||||
"observed_hash": observed_hash,
|
||||
"detail": (detail or "")[:MAX_POLL_DETAIL_LEN],
|
||||
"task_id": task_id,
|
||||
"created_at": now,
|
||||
}
|
||||
|
||||
|
||||
def _create_change_task(conn, watch, excerpt, now):
|
||||
"""变化建任务:列补齐口径与 automations.run_due_automation
|
||||
同款;任务 prompt = 用户处理提示 + 变化摘录。"""
|
||||
existing_tasks = _existing_columns(conn, "tasks")
|
||||
for col, ddl in (
|
||||
("executor", "TEXT"),
|
||||
("model", "TEXT"),
|
||||
("effort", "TEXT"),
|
||||
("environment_id", "TEXT"),
|
||||
("parent_task_id", "TEXT"),
|
||||
("coordinator_part", "TEXT")):
|
||||
if col not in existing_tasks:
|
||||
try:
|
||||
conn.execute(
|
||||
"ALTER TABLE tasks ADD COLUMN %s %s" % (col, ddl))
|
||||
except Exception:
|
||||
pass
|
||||
task_id = uuid.uuid4().hex
|
||||
prompt = ("%s\n\n[常驻监听变化] %s %s\n变化摘录:\n%s"
|
||||
% (watch["prompt"], watch["name"], watch["target_url"],
|
||||
excerpt))
|
||||
conn.execute(
|
||||
"INSERT INTO tasks(id, project_id, prompt, status,"
|
||||
" idempotency_key, depends_on, priority, created_by,"
|
||||
" created_at, updated_at, last_heartbeat, executor,"
|
||||
" model, effort, environment_id, parent_task_id,"
|
||||
" coordinator_part)"
|
||||
" VALUES(?, ?, ?, 'queued', NULL, '[]', 0, ?, ?, ?,"
|
||||
" NULL, NULL, NULL, NULL, NULL, NULL, NULL)",
|
||||
(task_id, watch["project_id"], prompt, watch["created_by"],
|
||||
now, now))
|
||||
return task_id
|
||||
|
||||
|
||||
def _guarded_watch_update(conn, watch_id, sets, vals):
|
||||
"""终局 UPDATE 带 status='running' 条件:抓取中途被用户
|
||||
暂停/结束时,条件不满足、rowcount 为 0,调用方据此回滚
|
||||
并放弃本轮(不建任务、不记轮询),而不是把 paused/ended
|
||||
行的 next_poll_at 改回未来值。返回是否成功。"""
|
||||
cur = conn.execute(
|
||||
"UPDATE resident_watches SET %s WHERE id = ? AND status = ?"
|
||||
% ", ".join(sets),
|
||||
tuple(vals) + (watch_id, RUNNING))
|
||||
return cur.rowcount == 1
|
||||
|
||||
|
||||
def run_due_resident_watch(conn, watch_id, fetcher=None, now=None):
|
||||
"""单条常驻监听的驱动唯一入口:抢单 -> 廉价抓取 -> hash
|
||||
比对 -> 记轮询 -> 仅变化时建任务。
|
||||
|
||||
抢不到返回 None(不存在、已暂停/已结束、未到期、已被
|
||||
别的驱动抢走),不记轮询、不建任务。抓取失败记 failed
|
||||
并退避,游标不动。首次抓取记 baseline,只建游标。未变
|
||||
化记 unchanged,不建任务。变化时先过配额门:配额耗尽
|
||||
转 paused(paused_reason='quota_exceeded',游标不推进),
|
||||
记 changed_quota_paused;配额检查失败按瞬时失败退避,
|
||||
游标同样不推进;配额内才建 queued 任务并推进游标。"""
|
||||
if now is None:
|
||||
now = time.time()
|
||||
ensure_resident_tables(conn)
|
||||
if not claim_due_watch(conn, watch_id, now):
|
||||
return None
|
||||
row = conn.execute(
|
||||
"SELECT * FROM resident_watches WHERE id = ?", (watch_id,)).fetchone()
|
||||
if row is None:
|
||||
return None
|
||||
watch = watch_to_dict(row, now)
|
||||
if watch["status"] != RUNNING:
|
||||
# 认领后、抓取前被用户暂停/结束:不抓取、不建任务、
|
||||
# 不改行(暂停/结束已把 next_poll_at 清掉,保持原样)。
|
||||
return None
|
||||
interval = watch["interval_seconds"]
|
||||
timeout = watch["fetch_timeout_seconds"]
|
||||
fetch = fetcher or default_fetch
|
||||
try:
|
||||
text = fetch(watch["target_url"], timeout)
|
||||
except Exception as exc:
|
||||
failures = watch["consecutive_failures"] + 1
|
||||
nxt = now + interval + _backoff_seconds(failures)
|
||||
if not _guarded_watch_update(
|
||||
conn, watch_id,
|
||||
("consecutive_failures = ?", "next_poll_at = ?",
|
||||
"updated_at = ?"),
|
||||
(failures, nxt, now)):
|
||||
conn.rollback()
|
||||
return None
|
||||
poll = _record_poll(conn, watch_id, watch["project_id"],
|
||||
"failed", None, str(exc)[:MAX_POLL_DETAIL_LEN],
|
||||
None, now)
|
||||
conn.commit()
|
||||
_auto_audit_row(conn, watch["project_id"],
|
||||
"resident_watch_poll_failed",
|
||||
"%s: %s" % (watch_id, str(exc)[:200]), now)
|
||||
return poll
|
||||
if not isinstance(text, str):
|
||||
text = str(text)
|
||||
observed = _content_hash(text)
|
||||
excerpt = text[:MAX_CHANGE_TEXT_LEN]
|
||||
old_hash = watch["last_content_hash"]
|
||||
if old_hash is None:
|
||||
if not _guarded_watch_update(
|
||||
conn, watch_id,
|
||||
("last_content_hash = ?", "consecutive_failures = 0",
|
||||
"next_poll_at = ?", "updated_at = ?"),
|
||||
(observed, now + interval, now)):
|
||||
conn.rollback()
|
||||
return None
|
||||
poll = _record_poll(conn, watch_id, watch["project_id"],
|
||||
"baseline", observed, "first poll", None, now)
|
||||
conn.commit()
|
||||
return poll
|
||||
if observed == old_hash:
|
||||
if not _guarded_watch_update(
|
||||
conn, watch_id,
|
||||
("consecutive_failures = 0", "next_poll_at = ?",
|
||||
"updated_at = ?"),
|
||||
(now + interval, now)):
|
||||
conn.rollback()
|
||||
return None
|
||||
poll = _record_poll(conn, watch_id, watch["project_id"],
|
||||
"unchanged", observed, "", None, now)
|
||||
conn.commit()
|
||||
return poll
|
||||
# 真实变化:配额门只在这里过。配额门契约是前缀
|
||||
# (quota_check_failed: / quota_exceeded: / None),不用
|
||||
# 子串判定,以免超限文本含该子串时误判。
|
||||
gate = _automation_quota_exceeded(conn, watch["created_by"])
|
||||
if gate is not None and not gate.startswith("quota_check_failed"):
|
||||
cur = conn.execute(
|
||||
"UPDATE resident_watches SET status = ?,"
|
||||
" paused_reason = ?, next_poll_at = NULL, updated_at = ?"
|
||||
" WHERE id = ? AND status = ?",
|
||||
(PAUSED, "quota_exceeded", now, watch_id, RUNNING))
|
||||
if cur.rowcount != 1:
|
||||
conn.rollback()
|
||||
return None
|
||||
poll = _record_poll(conn, watch_id, watch["project_id"],
|
||||
"changed_quota_paused", observed, gate,
|
||||
None, now)
|
||||
conn.commit()
|
||||
_auto_audit_row(conn, watch["project_id"],
|
||||
"resident_watch_paused_quota",
|
||||
"%s: %s" % (watch_id, gate[:200]), now)
|
||||
return poll
|
||||
if gate is not None:
|
||||
failures = watch["consecutive_failures"] + 1
|
||||
nxt = now + interval + _backoff_seconds(failures)
|
||||
if not _guarded_watch_update(
|
||||
conn, watch_id,
|
||||
("consecutive_failures = ?", "next_poll_at = ?",
|
||||
"updated_at = ?"),
|
||||
(failures, nxt, now)):
|
||||
conn.rollback()
|
||||
return None
|
||||
poll = _record_poll(conn, watch_id, watch["project_id"],
|
||||
"changed_quota_check_failed", observed, gate,
|
||||
None, now)
|
||||
conn.commit()
|
||||
return poll
|
||||
# 先受保护地推进游标,再建任务:抓取中途被暂停/结束时,
|
||||
# 条件不满足即回滚,本轮不建任务、不记轮询。
|
||||
if not _guarded_watch_update(
|
||||
conn, watch_id,
|
||||
("last_content_hash = ?", "last_change_at = ?",
|
||||
"last_change_text = ?", "consecutive_failures = 0",
|
||||
"next_poll_at = ?", "updated_at = ?"),
|
||||
(observed, now, excerpt, now + interval, now)):
|
||||
conn.rollback()
|
||||
return None
|
||||
task_id = _create_change_task(conn, watch, excerpt, now)
|
||||
poll = _record_poll(conn, watch_id, watch["project_id"],
|
||||
"changed", observed, excerpt[:MAX_POLL_DETAIL_LEN],
|
||||
task_id, now)
|
||||
conn.commit()
|
||||
_auto_audit_row(conn, watch["project_id"],
|
||||
"resident_watch_change_task_created",
|
||||
"%s -> %s" % (watch_id, task_id), now)
|
||||
return poll
|
||||
|
||||
|
||||
def process_due_resident_watches(conn, fetcher=None, now=None, limit=50):
|
||||
"""驱动批量入口:列出到期监听,逐条走 run_due(每条自带
|
||||
claim 门,批量内重复与并发驱动都不会重复轮询)。返回
|
||||
本轮实际产出的轮询记录列表(抢不到的条目不在内)。"""
|
||||
if now is None:
|
||||
now = time.time()
|
||||
due = due_watches(conn, now)[:limit]
|
||||
out = []
|
||||
for watch in due:
|
||||
poll = run_due_resident_watch(conn, watch["id"],
|
||||
fetcher=fetcher, now=now)
|
||||
if poll is not None:
|
||||
out.append(poll)
|
||||
return out
|
||||
+185
@@ -110,6 +110,14 @@ from cloud.wait_ops import worktree_status as _wait_worktree_status
|
||||
from cloud.shared_home_mcp import sync_shared_home_mcp as _sync_shared_home_mcp
|
||||
from cloud.automations import ensure_automations_tables as _auto_ensure
|
||||
from cloud.automations import list_automations as _auto_list
|
||||
from cloud.resident import create_watch as _res_create
|
||||
from cloud.resident import end_watch as _res_end
|
||||
from cloud.resident import get_watch as _res_get
|
||||
from cloud.resident import list_polls as _res_polls
|
||||
from cloud.resident import list_watches as _res_list
|
||||
from cloud.resident import pause_watch as _res_pause
|
||||
from cloud.resident import resume_watch as _res_resume
|
||||
from cloud.resident import update_watch as _res_update
|
||||
from cloud.todos import create_todo as _todo_create
|
||||
from cloud.todos import delete_todo as _todo_delete
|
||||
from cloud.todos import get_todo as _todo_get
|
||||
@@ -3371,6 +3379,54 @@ class CloudControlPlaneHandler(BaseHTTPRequestHandler):
|
||||
return
|
||||
self._triage_inbox(project_id)
|
||||
return
|
||||
# /projects/{id}/resident-watches:常驻监听任务。
|
||||
# 创建/更新/暂停/恢复/结束用 create_task,读用 read;
|
||||
# 状态机见 cloud/resident.py(running/paused/ended,
|
||||
# ended 是终态,恢复已结束的监听必须新建)。
|
||||
if len(parts) == 2 and parts[1] == "resident-watches":
|
||||
if method == "POST":
|
||||
err = self._check_project(user_id, project_id, "create_task")
|
||||
if err:
|
||||
self._send_json(err[0], {"error": err[1]})
|
||||
return
|
||||
self._resident_create(user_id, project_id)
|
||||
elif method == "GET":
|
||||
err = self._check_project(user_id, project_id, "read")
|
||||
if err:
|
||||
self._send_json(err[0], {"error": err[1]})
|
||||
return
|
||||
self._resident_list(project_id)
|
||||
else:
|
||||
self._send_json(404, {"error": "not found"})
|
||||
return
|
||||
if len(parts) == 3 and parts[1] == "resident-watches":
|
||||
if method == "GET":
|
||||
err = self._check_project(user_id, project_id, "read")
|
||||
if err:
|
||||
self._send_json(err[0], {"error": err[1]})
|
||||
return
|
||||
self._resident_get(user_id, project_id, parts[2])
|
||||
elif method == "PATCH":
|
||||
err = self._check_project(user_id, project_id, "create_task")
|
||||
if err:
|
||||
self._send_json(err[0], {"error": err[1]})
|
||||
return
|
||||
self._resident_update(user_id, project_id, parts[2])
|
||||
else:
|
||||
self._send_json(404, {"error": "not found"})
|
||||
return
|
||||
if len(parts) == 4 and parts[1] == "resident-watches" \
|
||||
and parts[3] in ("pause", "resume", "end"):
|
||||
if method != "POST":
|
||||
self._send_json(404, {"error": "not found"})
|
||||
return
|
||||
err = self._check_project(user_id, project_id, "create_task")
|
||||
if err:
|
||||
self._send_json(err[0], {"error": err[1]})
|
||||
return
|
||||
self._resident_transition(user_id, project_id, parts[2],
|
||||
parts[3])
|
||||
return
|
||||
# /projects/{id}/todos GET 列表 / POST 新建;单条
|
||||
# GET / PATCH / DELETE(项目 read / create_task)
|
||||
if len(parts) == 2 and parts[1] == "todos":
|
||||
@@ -10067,6 +10123,135 @@ class CloudControlPlaneHandler(BaseHTTPRequestHandler):
|
||||
automation_id)
|
||||
self._send_json(200, auto)
|
||||
|
||||
# -- 常驻监听任务(见 cloud/resident.py)-------------------
|
||||
# 权限口径与自动化同款:创建/更新/暂停/恢复/结束用
|
||||
# create_task,读用 read;状态变更记审计。创建不走配额门
|
||||
# (轮询不花模型额度),配额只在变化建任务时由驱动过,
|
||||
# 耗尽时转暂停而非拒绝创建。
|
||||
def _resident_create(self, user_id, project_id):
|
||||
body, err = self._read_json()
|
||||
if err:
|
||||
self._send_json(400, {"error": err})
|
||||
return
|
||||
interval = body.get("interval_seconds", 3600)
|
||||
if interval is None:
|
||||
interval = 3600
|
||||
conn = self._db()
|
||||
try:
|
||||
try:
|
||||
watch = _res_create(
|
||||
conn, project_id, body.get("name"),
|
||||
body.get("target_url"), body.get("prompt"), user_id,
|
||||
interval, body.get("fetch_timeout_seconds"))
|
||||
except ValueError as exc:
|
||||
self._send_json(400, {"error": str(exc)})
|
||||
return
|
||||
except Exception as exc:
|
||||
self._send_json(502, {"error": str(exc)[:500]})
|
||||
return
|
||||
finally:
|
||||
conn.close()
|
||||
with _auth_store(self.server.db_path) as store:
|
||||
store.audit(user_id, project_id, "resident_watch_created",
|
||||
watch["id"])
|
||||
self._send_json(201, watch)
|
||||
|
||||
def _resident_list(self, project_id):
|
||||
conn = self._db()
|
||||
try:
|
||||
try:
|
||||
watches = _res_list(conn, project_id)
|
||||
except Exception as exc:
|
||||
self._send_json(502, {"error": str(exc)[:500]})
|
||||
return
|
||||
conn.commit()
|
||||
finally:
|
||||
conn.close()
|
||||
self._send_json(200, {
|
||||
"project_id": project_id, "resident_watches": watches,
|
||||
"due_count": sum(1 for w in watches if w["due"])})
|
||||
|
||||
def _resident_get(self, user_id, project_id, watch_id):
|
||||
conn = self._db()
|
||||
try:
|
||||
watch = _res_get(conn, watch_id)
|
||||
polls = []
|
||||
polls_note = ""
|
||||
if watch is not None:
|
||||
try:
|
||||
polls = _res_polls(conn, watch_id)
|
||||
except Exception:
|
||||
polls_note = "polls unavailable"
|
||||
finally:
|
||||
conn.close()
|
||||
if watch is None or watch["project_id"] != project_id:
|
||||
self._send_json(404, {"error": "resident watch not found"})
|
||||
return
|
||||
self._send_json(200, dict(watch, polls=polls,
|
||||
polls_note=polls_note))
|
||||
|
||||
def _resident_update(self, user_id, project_id, watch_id):
|
||||
body, jerr = self._read_json()
|
||||
if jerr:
|
||||
self._send_json(400, {"error": jerr})
|
||||
return
|
||||
conn = self._db()
|
||||
try:
|
||||
try:
|
||||
ok, watch = _res_update(conn, watch_id, body)
|
||||
except ValueError as exc:
|
||||
self._send_json(400, {"error": str(exc)})
|
||||
return
|
||||
except Exception as exc:
|
||||
self._send_json(502, {"error": str(exc)[:500]})
|
||||
return
|
||||
finally:
|
||||
conn.close()
|
||||
if watch is None or watch["project_id"] != project_id:
|
||||
self._send_json(404, {"error": "resident watch not found"})
|
||||
return
|
||||
if not ok:
|
||||
self._send_json(409, {
|
||||
"error": "ended resident watch cannot be updated;"
|
||||
" create a new watch instead"})
|
||||
return
|
||||
with _auth_store(self.server.db_path) as store:
|
||||
store.audit(user_id, project_id, "resident_watch_updated",
|
||||
watch_id)
|
||||
self._send_json(200, watch)
|
||||
|
||||
def _resident_transition(self, user_id, project_id, watch_id, action):
|
||||
fn = {"pause": _res_pause, "resume": _res_resume,
|
||||
"end": _res_end}[action]
|
||||
conn = self._db()
|
||||
try:
|
||||
try:
|
||||
ok, watch = fn(conn, watch_id)
|
||||
except Exception as exc:
|
||||
self._send_json(502, {"error": str(exc)[:500]})
|
||||
return
|
||||
finally:
|
||||
conn.close()
|
||||
if watch is None or watch["project_id"] != project_id:
|
||||
self._send_json(404, {"error": "resident watch not found"})
|
||||
return
|
||||
if not ok:
|
||||
if watch["status"] == "ended":
|
||||
self._send_json(409, {
|
||||
"error": "ended resident watch is terminal;"
|
||||
" create a new watch instead"})
|
||||
else:
|
||||
self._send_json(409, {
|
||||
"error": "resident watch in status %s cannot %s"
|
||||
% (watch["status"], action)})
|
||||
return
|
||||
audit_action = {"pause": "resident_watch_paused",
|
||||
"resume": "resident_watch_resumed",
|
||||
"end": "resident_watch_ended"}[action]
|
||||
with _auth_store(self.server.db_path) as store:
|
||||
store.audit(user_id, project_id, audit_action, watch_id)
|
||||
self._send_json(200, watch)
|
||||
|
||||
def _triage_inbox(self, project_id):
|
||||
"""分诊收件箱(只读):到期自动化、失败/中断任务、
|
||||
待处理后续消息、最近自动化产出。见 automations 模块
|
||||
|
||||
@@ -0,0 +1,493 @@
|
||||
"""常驻监听任务单测(cloud/resident.py):不跑真实模型、不联网,
|
||||
抓取一律注入假 fetcher。覆盖创建校验、暂停保游标、恢复续跑、
|
||||
结束终态、防重复轮询、仅变化建任务、失败退避、配额耗尽转暂停、
|
||||
worker 驱动批量入口。"""
|
||||
import http.server
|
||||
import os
|
||||
import socket
|
||||
import sqlite3
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
|
||||
try:
|
||||
import cloud.resident as resident_module
|
||||
from cloud.resident import (
|
||||
claim_due_watch,
|
||||
create_watch,
|
||||
default_fetch,
|
||||
end_watch,
|
||||
ensure_resident_tables,
|
||||
get_watch,
|
||||
list_polls,
|
||||
pause_watch,
|
||||
process_due_resident_watches,
|
||||
resume_watch,
|
||||
run_due_resident_watch,
|
||||
update_watch,
|
||||
validate_target_url,
|
||||
)
|
||||
from cloud.server import init_db
|
||||
from cloud.worker import Worker
|
||||
except ImportError: # discover -s cloud/tests 时把项目根目录补进 sys.path
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
import cloud.resident as resident_module
|
||||
from cloud.resident import (
|
||||
claim_due_watch,
|
||||
create_watch,
|
||||
default_fetch,
|
||||
end_watch,
|
||||
ensure_resident_tables,
|
||||
get_watch,
|
||||
list_polls,
|
||||
pause_watch,
|
||||
process_due_resident_watches,
|
||||
resume_watch,
|
||||
run_due_resident_watch,
|
||||
update_watch,
|
||||
validate_target_url,
|
||||
)
|
||||
from cloud.server import init_db
|
||||
from cloud.worker import Worker
|
||||
|
||||
URL = "https://example.com/data"
|
||||
|
||||
|
||||
class _Fetcher:
|
||||
"""可变假抓取:value 改了,下一次轮询就看到新内容。"""
|
||||
|
||||
def __init__(self, value="v1"):
|
||||
self.value = value
|
||||
self.calls = []
|
||||
|
||||
def __call__(self, url, timeout):
|
||||
self.calls.append((url, timeout))
|
||||
return self.value
|
||||
|
||||
|
||||
def _failing_fetcher(url, timeout):
|
||||
raise RuntimeError("boom")
|
||||
|
||||
|
||||
class ResidentTestCase(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.db_path = os.path.join(self.tmp.name, "control.db")
|
||||
init_db(self.db_path)
|
||||
self.conn = sqlite3.connect(self.db_path)
|
||||
self.conn.row_factory = sqlite3.Row
|
||||
ensure_resident_tables(self.conn)
|
||||
self.now = 1700000000.0
|
||||
|
||||
def tearDown(self):
|
||||
self.conn.close()
|
||||
self.tmp.cleanup()
|
||||
|
||||
def _watch(self, name="watch1", **kw):
|
||||
kw.setdefault("interval_seconds", 60)
|
||||
return create_watch(self.conn, "proj", name, URL,
|
||||
"变化时汇报", "owner1", now=self.now, **kw)
|
||||
|
||||
def _tasks(self):
|
||||
cur = self.conn.execute("SELECT id, prompt, status FROM tasks")
|
||||
return cur.fetchall()
|
||||
|
||||
def _set_quota(self, user_id, micros):
|
||||
self.conn.execute(
|
||||
"CREATE TABLE IF NOT EXISTS user_quotas("
|
||||
"user_id TEXT PRIMARY KEY, quota_usd_micros BIGINT, "
|
||||
" updated_at DOUBLE PRECISION)")
|
||||
self.conn.execute(
|
||||
"INSERT OR REPLACE INTO user_quotas(user_id,"
|
||||
" quota_usd_micros, updated_at) VALUES(?, ?, ?)",
|
||||
(user_id, micros, self.now))
|
||||
self.conn.commit()
|
||||
|
||||
def test_create_validation(self):
|
||||
watch = self._watch()
|
||||
self.assertEqual(watch["status"], "running")
|
||||
self.assertTrue(watch["due"])
|
||||
self.assertEqual(watch["next_poll_at"], self.now)
|
||||
for kwargs, _ in (
|
||||
({"target_url": "ftp://example.com/x"}, None),
|
||||
({"target_url": "https://localhost/x"}, None),
|
||||
({"target_url": "http://127.0.0.1/x"}, None),
|
||||
({"target_url": "https://10.0.0.5/x"}, None),
|
||||
({"target_url": "https://user:pw@example.com/x"}, None),
|
||||
({"interval_seconds": 30}, None),
|
||||
({"name": "watch1"}, None),
|
||||
({"prompt": " "}, None)):
|
||||
with self.assertRaises(ValueError):
|
||||
base = {"name": "other", "target_url": URL,
|
||||
"prompt": "p", "interval_seconds": 60}
|
||||
base.update(kwargs)
|
||||
create_watch(self.conn, "proj", base["name"],
|
||||
base["target_url"], base["prompt"], "owner1",
|
||||
interval_seconds=base["interval_seconds"],
|
||||
now=self.now)
|
||||
|
||||
def test_baseline_unchanged_changed_only_change_creates_task(self):
|
||||
fetcher = _Fetcher("v1")
|
||||
watch = self._watch()
|
||||
poll = run_due_resident_watch(self.conn, watch["id"],
|
||||
fetcher=fetcher, now=self.now)
|
||||
self.assertEqual(poll["status"], "baseline")
|
||||
self.assertEqual(len(self._tasks()), 0)
|
||||
# 同一间隔内第二次:抢不到,不重复轮询。
|
||||
self.assertIsNone(run_due_resident_watch(
|
||||
self.conn, watch["id"], fetcher=fetcher, now=self.now))
|
||||
self.assertEqual(len(fetcher.calls), 1)
|
||||
self.now += 61
|
||||
poll = run_due_resident_watch(self.conn, watch["id"],
|
||||
fetcher=fetcher, now=self.now)
|
||||
self.assertEqual(poll["status"], "unchanged")
|
||||
self.assertEqual(len(self._tasks()), 0)
|
||||
fetcher.value = "v2"
|
||||
self.now += 61
|
||||
poll = run_due_resident_watch(self.conn, watch["id"],
|
||||
fetcher=fetcher, now=self.now)
|
||||
self.assertEqual(poll["status"], "changed")
|
||||
self.assertIsNotNone(poll["task_id"])
|
||||
tasks = self._tasks()
|
||||
self.assertEqual(len(tasks), 1)
|
||||
self.assertIn("v2", tasks[0][1])
|
||||
self.assertIn("变化时汇报", tasks[0][1])
|
||||
after = get_watch(self.conn, watch["id"], now=self.now)
|
||||
self.assertEqual(after["last_change_text"], "v2")
|
||||
self.assertIsNotNone(after["last_change_at"])
|
||||
self.assertEqual(
|
||||
[p["status"] for p in list_polls(self.conn, watch["id"])],
|
||||
["changed", "unchanged", "baseline"])
|
||||
|
||||
def test_pause_preserves_cursor_and_resume_reports_pause_period_change(self):
|
||||
fetcher = _Fetcher("v1")
|
||||
watch = self._watch()
|
||||
run_due_resident_watch(self.conn, watch["id"],
|
||||
fetcher=fetcher, now=self.now)
|
||||
cursor = get_watch(self.conn, watch["id"], now=self.now)
|
||||
self.assertIsNotNone(cursor["last_content_hash"])
|
||||
ok, paused = pause_watch(self.conn, watch["id"], now=self.now)
|
||||
self.assertTrue(ok)
|
||||
self.assertEqual(paused["status"], "paused")
|
||||
self.assertEqual(paused["last_content_hash"],
|
||||
cursor["last_content_hash"])
|
||||
self.assertIsNone(paused["next_poll_at"])
|
||||
self.assertFalse(paused["due"])
|
||||
# 暂停期间即使时间过了间隔,驱动也不轮询。
|
||||
self.assertIsNone(run_due_resident_watch(
|
||||
self.conn, watch["id"], fetcher=fetcher, now=self.now + 3600))
|
||||
self.assertEqual(len(fetcher.calls), 1)
|
||||
# 暂停期间网站变了;恢复后第一次轮询用旧游标比对,报出来。
|
||||
fetcher.value = "v2"
|
||||
ok, resumed = resume_watch(self.conn, watch["id"], now=self.now)
|
||||
self.assertTrue(ok)
|
||||
self.assertEqual(resumed["status"], "running")
|
||||
self.assertEqual(resumed["last_content_hash"],
|
||||
cursor["last_content_hash"])
|
||||
poll = run_due_resident_watch(self.conn, watch["id"],
|
||||
fetcher=fetcher, now=self.now)
|
||||
self.assertEqual(poll["status"], "changed")
|
||||
self.assertEqual(len(self._tasks()), 1)
|
||||
|
||||
def test_pause_resume_on_wrong_status_are_rejected(self):
|
||||
watch = self._watch()
|
||||
ok, _ = resume_watch(self.conn, watch["id"], now=self.now)
|
||||
self.assertFalse(ok)
|
||||
ok, _ = pause_watch(self.conn, watch["id"], now=self.now)
|
||||
self.assertTrue(ok)
|
||||
ok, _ = pause_watch(self.conn, watch["id"], now=self.now)
|
||||
self.assertFalse(ok)
|
||||
|
||||
def test_end_is_terminal_and_name_reusable(self):
|
||||
watch = self._watch()
|
||||
ok, ended = end_watch(self.conn, watch["id"], now=self.now)
|
||||
self.assertTrue(ok)
|
||||
self.assertEqual(ended["status"], "ended")
|
||||
self.assertIsNone(ended["next_poll_at"])
|
||||
for fn in (pause_watch, resume_watch, end_watch):
|
||||
ok, old = fn(self.conn, watch["id"], now=self.now)
|
||||
self.assertFalse(ok)
|
||||
self.assertEqual(old["status"], "ended")
|
||||
self.assertIsNone(run_due_resident_watch(
|
||||
self.conn, watch["id"], fetcher=_Fetcher(), now=self.now + 3600))
|
||||
ok, _ = update_watch(self.conn, watch["id"],
|
||||
{"prompt": "新提示"}, now=self.now)
|
||||
self.assertFalse(ok)
|
||||
# 已结束的名字可重用。
|
||||
again = self._watch()
|
||||
self.assertEqual(again["status"], "running")
|
||||
|
||||
def test_claim_is_single_winner(self):
|
||||
watch = self._watch()
|
||||
self.assertTrue(claim_due_watch(self.conn, watch["id"], now=self.now))
|
||||
self.assertFalse(claim_due_watch(self.conn, watch["id"],
|
||||
now=self.now))
|
||||
# 下个间隔到了才能再抢(claim 已把 next_poll_at 预置到下间隔)。
|
||||
self.assertTrue(claim_due_watch(self.conn, watch["id"],
|
||||
now=self.now + 61))
|
||||
|
||||
def test_fetch_failure_backoff_and_cursor_unchanged(self):
|
||||
watch = self._watch()
|
||||
run_due_resident_watch(self.conn, watch["id"],
|
||||
fetcher=_Fetcher("v1"), now=self.now)
|
||||
cursor = get_watch(self.conn, watch["id"], now=self.now)
|
||||
self.now += 61
|
||||
poll = run_due_resident_watch(self.conn, watch["id"],
|
||||
fetcher=_failing_fetcher, now=self.now)
|
||||
self.assertEqual(poll["status"], "failed")
|
||||
after = get_watch(self.conn, watch["id"], now=self.now)
|
||||
self.assertEqual(after["consecutive_failures"], 1)
|
||||
self.assertEqual(after["last_content_hash"],
|
||||
cursor["last_content_hash"])
|
||||
# 失败退避:下次 = now + 间隔 + 30 秒,不是立刻重试。
|
||||
self.assertEqual(after["next_poll_at"], self.now + 60 + 30)
|
||||
self.now = after["next_poll_at"]
|
||||
run_due_resident_watch(self.conn, watch["id"],
|
||||
fetcher=_failing_fetcher, now=self.now)
|
||||
after2 = get_watch(self.conn, watch["id"], now=self.now)
|
||||
self.assertEqual(after2["consecutive_failures"], 2)
|
||||
self.assertEqual(after2["next_poll_at"], self.now + 60 + 60)
|
||||
|
||||
def test_quota_exceeded_pauses_not_ends_and_keeps_cursor(self):
|
||||
fetcher = _Fetcher("v1")
|
||||
watch = self._watch()
|
||||
run_due_resident_watch(self.conn, watch["id"],
|
||||
fetcher=fetcher, now=self.now)
|
||||
cursor_hash = get_watch(self.conn, watch["id"],
|
||||
now=self.now)["last_content_hash"]
|
||||
self._set_quota("owner1", 0)
|
||||
fetcher.value = "v2"
|
||||
self.now += 61
|
||||
poll = run_due_resident_watch(self.conn, watch["id"],
|
||||
fetcher=fetcher, now=self.now)
|
||||
self.assertEqual(poll["status"], "changed_quota_paused")
|
||||
after = get_watch(self.conn, watch["id"], now=self.now)
|
||||
self.assertEqual(after["status"], "paused")
|
||||
self.assertEqual(after["paused_reason"], "quota_exceeded")
|
||||
self.assertNotEqual(after["status"], "ended")
|
||||
self.assertEqual(after["last_content_hash"], cursor_hash)
|
||||
self.assertEqual(len(self._tasks()), 0)
|
||||
# 恢复后配额仍耗尽:再次转暂停,不擅自结束、不丢变化。
|
||||
resume_watch(self.conn, watch["id"], now=self.now)
|
||||
poll = run_due_resident_watch(self.conn, watch["id"],
|
||||
fetcher=fetcher, now=self.now)
|
||||
self.assertEqual(poll["status"], "changed_quota_paused")
|
||||
# 配额恢复后,同一变化仍能报出来(游标没被配额暂停推进)。
|
||||
self._set_quota("owner1", 100000000)
|
||||
resume_watch(self.conn, watch["id"], now=self.now)
|
||||
poll = run_due_resident_watch(self.conn, watch["id"],
|
||||
fetcher=fetcher, now=self.now)
|
||||
self.assertEqual(poll["status"], "changed")
|
||||
self.assertEqual(len(self._tasks()), 1)
|
||||
|
||||
def test_update_watch(self):
|
||||
watch = self._watch()
|
||||
ok, updated = update_watch(
|
||||
self.conn, watch["id"],
|
||||
{"interval_seconds": 3600, "prompt": "新处理提示"},
|
||||
now=self.now)
|
||||
self.assertTrue(ok)
|
||||
self.assertEqual(updated["interval_seconds"], 3600)
|
||||
self.assertEqual(updated["prompt"], "新处理提示")
|
||||
with self.assertRaises(ValueError):
|
||||
update_watch(self.conn, watch["id"], {}, now=self.now)
|
||||
with self.assertRaises(ValueError):
|
||||
update_watch(self.conn, watch["id"],
|
||||
{"target_url": "https://localhost/x"}, now=self.now)
|
||||
|
||||
def test_process_due_and_worker_driver(self):
|
||||
fetcher = _Fetcher("v1")
|
||||
self._watch(name="a")
|
||||
self._watch(name="b")
|
||||
polls = process_due_resident_watches(self.conn, fetcher=fetcher,
|
||||
now=self.now)
|
||||
self.assertEqual(len(polls), 2)
|
||||
self.assertEqual([p["status"] for p in polls],
|
||||
["baseline", "baseline"])
|
||||
self.assertEqual(process_due_resident_watches(
|
||||
self.conn, fetcher=fetcher, now=self.now), [])
|
||||
# worker 驱动入口:注入假抓取,变化时产出 changed 轮询。
|
||||
worker = Worker(self.db_path, "/fake/host.py",
|
||||
env={"XCM_API_KEY": "test-placeholder"},
|
||||
runner=lambda cmd: SimpleNamespace(
|
||||
returncode=0, stdout="{}", stderr=""))
|
||||
fetcher.value = "v2"
|
||||
self.now += 61
|
||||
polls = worker.process_due_resident_watches(
|
||||
now=self.now, fetcher=fetcher)
|
||||
self.assertEqual(len(polls), 2)
|
||||
self.assertTrue(all(p["status"] == "changed" for p in polls))
|
||||
self.assertEqual(len(self._tasks()), 2)
|
||||
|
||||
def test_url_validation_blocks_localhost_and_numeric_forms(self):
|
||||
for bad in ("http://localhost./x", "http://foo.localhost/x",
|
||||
"http://2130706433/x", "http://0x7f000001/x",
|
||||
"http://0177.0.0.1/x", "http://0x7f.0.0.1/x"):
|
||||
with self.assertRaises(ValueError, msg=bad):
|
||||
validate_target_url(bad)
|
||||
# 规范点分十进制公网 IP 与带尾点的普通域名仍放行。
|
||||
self.assertEqual(validate_target_url("https://8.8.8.8/x"),
|
||||
"https://8.8.8.8/x")
|
||||
self.assertEqual(validate_target_url("https://example.com./x"),
|
||||
"https://example.com./x")
|
||||
|
||||
def test_unique_index_guards_concurrent_name_clash(self):
|
||||
idx = self.conn.execute(
|
||||
"SELECT name FROM sqlite_master WHERE type = 'index'"
|
||||
" AND name = 'idx_resident_watches_name'").fetchone()
|
||||
self.assertIsNotNone(idx)
|
||||
self._watch()
|
||||
# 正常路径仍先报 ValueError。
|
||||
with self.assertRaises(ValueError):
|
||||
self._watch()
|
||||
# 绕过 SELECT 查重直接插:唯一索引本身必须拦住,
|
||||
# 这才是并发下真正的关口。
|
||||
with self.assertRaises(sqlite3.IntegrityError):
|
||||
self.conn.execute(
|
||||
"INSERT INTO resident_watches(id, project_id, name,"
|
||||
" target_url, prompt, interval_seconds, status,"
|
||||
" created_by, created_at, updated_at)"
|
||||
" VALUES('dup', 'proj', 'watch1', ?, 'p', 60,"
|
||||
" 'running', 'owner1', ?, ?)",
|
||||
(URL, self.now, self.now))
|
||||
other = self._watch(name="watch2")
|
||||
with self.assertRaises(ValueError):
|
||||
update_watch(self.conn, other["id"], {"name": "watch1"},
|
||||
now=self.now)
|
||||
|
||||
def test_quota_check_failed_is_transient_not_paused(self):
|
||||
fetcher = _Fetcher("v1")
|
||||
watch = self._watch()
|
||||
run_due_resident_watch(self.conn, watch["id"],
|
||||
fetcher=fetcher, now=self.now)
|
||||
cursor_hash = get_watch(self.conn, watch["id"],
|
||||
now=self.now)["last_content_hash"]
|
||||
original = resident_module._automation_quota_exceeded
|
||||
resident_module._automation_quota_exceeded = lambda conn, owner: (
|
||||
"quota_check_failed: boom")
|
||||
try:
|
||||
fetcher.value = "v2"
|
||||
self.now += 61
|
||||
poll = run_due_resident_watch(
|
||||
self.conn, watch["id"], fetcher=fetcher, now=self.now)
|
||||
finally:
|
||||
resident_module._automation_quota_exceeded = original
|
||||
self.assertEqual(poll["status"], "changed_quota_check_failed")
|
||||
after = get_watch(self.conn, watch["id"], now=self.now)
|
||||
self.assertEqual(after["status"], "running")
|
||||
self.assertEqual(after["last_content_hash"], cursor_hash)
|
||||
self.assertEqual(after["consecutive_failures"], 1)
|
||||
self.assertEqual(after["next_poll_at"], self.now + 60 + 30)
|
||||
self.assertEqual(len(self._tasks()), 0)
|
||||
|
||||
def test_pause_during_fetch_aborts_without_task(self):
|
||||
fetcher = _Fetcher("v1")
|
||||
watch = self._watch()
|
||||
run_due_resident_watch(self.conn, watch["id"],
|
||||
fetcher=fetcher, now=self.now)
|
||||
cursor_hash = get_watch(self.conn, watch["id"],
|
||||
now=self.now)["last_content_hash"]
|
||||
db_path = self.db_path
|
||||
|
||||
def pausing_fetcher(url, timeout):
|
||||
conn2 = sqlite3.connect(db_path)
|
||||
try:
|
||||
conn2.row_factory = sqlite3.Row
|
||||
pause_watch(conn2, watch["id"], now=self.now + 61)
|
||||
finally:
|
||||
conn2.close()
|
||||
return "v2"
|
||||
|
||||
self.now += 61
|
||||
poll = run_due_resident_watch(
|
||||
self.conn, watch["id"], fetcher=pausing_fetcher, now=self.now)
|
||||
self.assertIsNone(poll)
|
||||
after = get_watch(self.conn, watch["id"], now=self.now)
|
||||
self.assertEqual(after["status"], "paused")
|
||||
self.assertIsNone(after["next_poll_at"])
|
||||
self.assertEqual(after["last_content_hash"], cursor_hash)
|
||||
self.assertEqual(len(self._tasks()), 0)
|
||||
self.assertEqual(
|
||||
[p["status"] for p in list_polls(self.conn, watch["id"])],
|
||||
["baseline"])
|
||||
|
||||
|
||||
class RedirectFetchTestCase(unittest.TestCase):
|
||||
"""重定向校验:只把 redirect-test.test 指到本地,其它主机
|
||||
走原始解析;no_proxy 同款处理,tearDown 全部还原。"""
|
||||
|
||||
HOST = "redirect-test.test"
|
||||
|
||||
def setUp(self):
|
||||
class Handler(http.server.BaseHTTPRequestHandler):
|
||||
def do_GET(self):
|
||||
if self.path == "/go":
|
||||
self.send_response(302)
|
||||
self.send_header("Location", "/final")
|
||||
elif self.path == "/bad":
|
||||
self.send_response(302)
|
||||
self.send_header(
|
||||
"Location", "http://127.0.0.1:1/secret")
|
||||
else:
|
||||
body = b"final-body"
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Length", str(len(body)))
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
return
|
||||
self.end_headers()
|
||||
|
||||
def log_message(self, *args):
|
||||
pass
|
||||
|
||||
self.server = http.server.ThreadingHTTPServer(
|
||||
("127.0.0.1", 0), Handler)
|
||||
self.port = self.server.server_address[1]
|
||||
self.thread = threading.Thread(
|
||||
target=self.server.serve_forever, daemon=True)
|
||||
self.thread.start()
|
||||
self._orig_getaddrinfo = socket.getaddrinfo
|
||||
|
||||
def patched(host, port, family=0, type=0, proto=0, flags=0):
|
||||
if host == self.HOST:
|
||||
return [(socket.AF_INET, socket.SOCK_STREAM, 6, "",
|
||||
("127.0.0.1", port))]
|
||||
return self._orig_getaddrinfo(
|
||||
host, port, family, type, proto, flags)
|
||||
|
||||
socket.getaddrinfo = patched
|
||||
self._old_no_proxy = os.environ.get("no_proxy")
|
||||
self._old_NO_PROXY = os.environ.get("NO_PROXY")
|
||||
os.environ["no_proxy"] = (
|
||||
(self._old_no_proxy + ",") if self._old_no_proxy
|
||||
else "") + self.HOST
|
||||
os.environ["NO_PROXY"] = (
|
||||
(self._old_NO_PROXY + ",") if self._old_NO_PROXY
|
||||
else "") + self.HOST
|
||||
|
||||
def tearDown(self):
|
||||
socket.getaddrinfo = self._orig_getaddrinfo
|
||||
if self._old_no_proxy is None:
|
||||
os.environ.pop("no_proxy", None)
|
||||
else:
|
||||
os.environ["no_proxy"] = self._old_no_proxy
|
||||
if self._old_NO_PROXY is None:
|
||||
os.environ.pop("NO_PROXY", None)
|
||||
else:
|
||||
os.environ["NO_PROXY"] = self._old_NO_PROXY
|
||||
self.server.shutdown()
|
||||
self.server.server_close()
|
||||
self.thread.join(timeout=5)
|
||||
|
||||
def test_relative_redirect_followed_and_private_blocked(self):
|
||||
base = "http://%s:%d" % (self.HOST, self.port)
|
||||
self.assertEqual(default_fetch(base + "/go", 10), "final-body")
|
||||
with self.assertRaises(ValueError):
|
||||
default_fetch(base + "/bad", 10)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,180 @@
|
||||
"""常驻监听服务端单测:POST/GET /projects/{id}/resident-watches、
|
||||
PATCH、POST pause / resume / end。不跑真实模型、不联网。
|
||||
骨架与 test_automations_server.py 同款(本地临时端口 + 临时
|
||||
SQLite)。"""
|
||||
import http.client
|
||||
import json
|
||||
import os
|
||||
import sqlite3
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
from cloud.auth import AuthStore
|
||||
from cloud.server import create_server
|
||||
except ImportError: # discover -s cloud/tests 时把项目根目录补进 sys.path
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
from cloud.auth import AuthStore
|
||||
from cloud.server import create_server
|
||||
|
||||
URL = "https://example.com/data"
|
||||
|
||||
|
||||
class ResidentServerTestCase(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.db_path = os.path.join(self.tmp.name, "control.db")
|
||||
self.auth = AuthStore(self.db_path)
|
||||
self.owner_id, self.owner_key = self.auth.create_user("owner")
|
||||
self.outsider_id, self.outsider_key = self.auth.create_user(
|
||||
"outsider")
|
||||
self.server = create_server(self.db_path, "127.0.0.1", 0)
|
||||
self.port = self.server.server_address[1]
|
||||
self.thread = threading.Thread(
|
||||
target=self.server.serve_forever, daemon=True)
|
||||
self.thread.start()
|
||||
status, self.project = self._req(
|
||||
"POST", "/projects", {"name": "demo"}, self.owner_key)
|
||||
self.assertEqual(status, 201)
|
||||
|
||||
def tearDown(self):
|
||||
self.server.shutdown()
|
||||
self.server.server_close()
|
||||
self.thread.join(timeout=5)
|
||||
try:
|
||||
self.auth.conn.close()
|
||||
except Exception:
|
||||
pass
|
||||
self.tmp.cleanup()
|
||||
|
||||
def _req(self, method, path, body=None, key=None):
|
||||
conn = http.client.HTTPConnection(
|
||||
"127.0.0.1", self.port, timeout=10)
|
||||
hdrs = {}
|
||||
payload = None
|
||||
if body is not None:
|
||||
payload = json.dumps(body).encode("utf-8")
|
||||
hdrs["Content-Type"] = "application/json"
|
||||
if key is not None:
|
||||
hdrs["X-API-Key"] = key
|
||||
try:
|
||||
conn.request(method, path, body=payload, headers=hdrs)
|
||||
resp = conn.getresponse()
|
||||
data = resp.read()
|
||||
finally:
|
||||
conn.close()
|
||||
try:
|
||||
parsed = json.loads(data.decode("utf-8")) if data else None
|
||||
except ValueError:
|
||||
parsed = None
|
||||
return resp.status, parsed
|
||||
|
||||
def _url(self, suffix=""):
|
||||
return "/projects/%s/resident-watches%s" % (
|
||||
self.project["id"], suffix)
|
||||
|
||||
def _create(self, **extra):
|
||||
body = {"name": "watch1", "target_url": URL,
|
||||
"prompt": "变化时汇报"}
|
||||
body.update(extra)
|
||||
return self._req("POST", self._url(), body, self.owner_key)
|
||||
|
||||
def _audit_actions(self):
|
||||
conn = sqlite3.connect(self.db_path)
|
||||
try:
|
||||
rows = conn.execute(
|
||||
"SELECT action FROM audit_log WHERE project_id = ?",
|
||||
(self.project["id"],)).fetchall()
|
||||
finally:
|
||||
conn.close()
|
||||
return [r[0] for r in rows]
|
||||
|
||||
def test_create_list_get_and_validation(self):
|
||||
status, watch = self._create(interval_seconds=3600)
|
||||
self.assertEqual(status, 201, watch)
|
||||
self.assertEqual(watch["status"], "running")
|
||||
self.assertEqual(watch["target_url"], URL)
|
||||
self.assertTrue(watch["due"])
|
||||
self.assertIsNotNone(watch["next_poll_at"])
|
||||
self.assertIn("resident_watch_created", self._audit_actions())
|
||||
status, body = self._req("GET", self._url(), None, self.owner_key)
|
||||
self.assertEqual(status, 200)
|
||||
self.assertEqual(len(body["resident_watches"]), 1)
|
||||
self.assertEqual(body["due_count"], 1)
|
||||
status, body = self._req(
|
||||
"GET", self._url("/" + watch["id"]), None, self.owner_key)
|
||||
self.assertEqual(status, 200)
|
||||
self.assertEqual(body["id"], watch["id"])
|
||||
self.assertEqual(body["polls"], [])
|
||||
for bad in (
|
||||
{"name": "x", "target_url": "ftp://example.com/x",
|
||||
"prompt": "p"},
|
||||
{"name": "x", "target_url": "https://localhost/x",
|
||||
"prompt": "p"},
|
||||
{"name": "x", "target_url": URL, "prompt": "p",
|
||||
"interval_seconds": 30},
|
||||
{"name": "watch1", "target_url": URL, "prompt": "p"},
|
||||
{"target_url": URL, "prompt": "p"},
|
||||
{"name": "x", "prompt": "p"}):
|
||||
status, _ = self._req("POST", self._url(), bad, self.owner_key)
|
||||
self.assertEqual(status, 400, bad)
|
||||
status, _ = self._req(
|
||||
"GET", self._url("/nope"), None, self.owner_key)
|
||||
self.assertEqual(status, 404)
|
||||
|
||||
def test_pause_resume_end_lifecycle_and_terminal_409(self):
|
||||
status, watch = self._create()
|
||||
self.assertEqual(status, 201)
|
||||
wid = watch["id"]
|
||||
status, body = self._req(
|
||||
"POST", self._url("/%s/pause" % wid), {}, self.owner_key)
|
||||
self.assertEqual(status, 200, body)
|
||||
self.assertEqual(body["status"], "paused")
|
||||
self.assertEqual(body["paused_reason"], "user_paused")
|
||||
self.assertIsNone(body["next_poll_at"])
|
||||
self.assertFalse(body["due"])
|
||||
status, body = self._req(
|
||||
"POST", self._url("/%s/pause" % wid), {}, self.owner_key)
|
||||
self.assertEqual(status, 409)
|
||||
status, body = self._req(
|
||||
"POST", self._url("/%s/resume" % wid), {}, self.owner_key)
|
||||
self.assertEqual(status, 200, body)
|
||||
self.assertEqual(body["status"], "running")
|
||||
self.assertIsNone(body["paused_reason"])
|
||||
self.assertTrue(body["due"])
|
||||
status, body = self._req(
|
||||
"POST", self._url("/%s/end" % wid), {}, self.owner_key)
|
||||
self.assertEqual(status, 200, body)
|
||||
self.assertEqual(body["status"], "ended")
|
||||
for action in ("pause", "resume", "end"):
|
||||
status, body = self._req(
|
||||
"POST", self._url("/%s/%s" % (wid, action)), {},
|
||||
self.owner_key)
|
||||
self.assertEqual(status, 409, action)
|
||||
self.assertIn("terminal", body["error"])
|
||||
actions = self._audit_actions()
|
||||
self.assertIn("resident_watch_paused", actions)
|
||||
self.assertIn("resident_watch_resumed", actions)
|
||||
self.assertIn("resident_watch_ended", actions)
|
||||
|
||||
def test_update_and_outsider_forbidden(self):
|
||||
status, watch = self._create()
|
||||
self.assertEqual(status, 201)
|
||||
wid = watch["id"]
|
||||
status, body = self._req(
|
||||
"PATCH", self._url("/" + wid),
|
||||
{"interval_seconds": 7200}, self.owner_key)
|
||||
self.assertEqual(status, 200, body)
|
||||
self.assertEqual(body["interval_seconds"], 7200)
|
||||
status, _ = self._req(
|
||||
"POST", self._url(),
|
||||
{"name": "x", "target_url": URL, "prompt": "p"},
|
||||
self.outsider_key)
|
||||
self.assertIn(status, (401, 403, 404))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,381 @@
|
||||
"""常驻监听网站闭环端到端测试。
|
||||
|
||||
上一轮只验到注入假抓取的单测:真实抓取、真实 run_loop 轮次
|
||||
边界、经服务端 API 的全链路(创建 -> 基线 -> 变化建任务 ->
|
||||
暂停 -> 恢复 -> 失败退避 -> 结束)都没有跑过。本文件补这一环,
|
||||
不改任何生产默认值、不改生产校验口径:
|
||||
|
||||
- 间隔仍是生产下限 60 秒;到期不是改间隔,而是给驱动传一个
|
||||
更晚的 now(模拟时间流逝),与上一轮单测同款时间注入。
|
||||
- 目标 URL 仍走生产校验:主机名 website-loop.test 不是 IP 字
|
||||
面、不是 localhost,所以校验通过;测试进程内只把这个主机
|
||||
名的 DNS 解析指向本地测试 HTTP 服务(相当于 /etc/hosts 的
|
||||
一条测试条目),抓取本身走真实 default_fetch(urllib、超时、
|
||||
1MB 截断),不注入假 fetcher、不绕过生产代码。
|
||||
- 内容服务是线程内真实 http.server,可变内容、也可停服;
|
||||
控制面是真实 ThreadingHTTPServer + 真实 SQLite 临时库;
|
||||
变化建出的任务、轮询、审计都直接读库/接口核对。
|
||||
|
||||
不联网、不跑真实模型、不碰生产。
|
||||
"""
|
||||
import http.client
|
||||
import http.server
|
||||
import json
|
||||
import os
|
||||
import socket
|
||||
import sqlite3
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
|
||||
try:
|
||||
from cloud.auth import AuthStore
|
||||
from cloud.resident import _content_hash
|
||||
from cloud.server import create_server
|
||||
from cloud.worker import Worker
|
||||
except ImportError: # discover -s cloud/tests 时把项目根目录补进 sys.path
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
from cloud.auth import AuthStore
|
||||
from cloud.resident import _content_hash
|
||||
from cloud.server import create_server
|
||||
from cloud.worker import Worker
|
||||
|
||||
HOST = "website-loop.test"
|
||||
|
||||
|
||||
class WebsiteLoopTestCase(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.db_path = os.path.join(self.tmp.name, "control.db")
|
||||
self.auth = AuthStore(self.db_path)
|
||||
self.owner_id, self.owner_key = self.auth.create_user("owner")
|
||||
self.server = create_server(self.db_path, "127.0.0.1", 0)
|
||||
self.port = self.server.server_address[1]
|
||||
self.thread = threading.Thread(
|
||||
target=self.server.serve_forever, daemon=True)
|
||||
self.thread.start()
|
||||
status, self.project = self._req(
|
||||
"POST", "/projects", {"name": "demo"}, self.owner_key)
|
||||
self.assertEqual(status, 201, self.project)
|
||||
# 可变内容服务:state 字典跨重启保留,停服/重启可复现。
|
||||
self.state = {"content": "version one"}
|
||||
state = self.state
|
||||
|
||||
class Handler(http.server.BaseHTTPRequestHandler):
|
||||
def do_GET(self):
|
||||
body = state["content"].encode("utf-8")
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "text/plain")
|
||||
self.send_header("Content-Length", str(len(body)))
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
|
||||
def log_message(self, *args):
|
||||
pass
|
||||
|
||||
self._handler = Handler
|
||||
self.content_server = http.server.ThreadingHTTPServer(
|
||||
("127.0.0.1", 0), Handler)
|
||||
self.content_port = self.content_server.server_address[1]
|
||||
self.content_thread = threading.Thread(
|
||||
target=self.content_server.serve_forever, daemon=True)
|
||||
self.content_thread.start()
|
||||
# 测试专用 DNS:只把 website-loop.test 指到本地服务,
|
||||
# 其它主机一律走原始解析;生产校验与抓取代码一字不改。
|
||||
self._orig_getaddrinfo = socket.getaddrinfo
|
||||
|
||||
def patched(host, port, family=0, type=0, proto=0, flags=0):
|
||||
if host == HOST:
|
||||
return [(socket.AF_INET, socket.SOCK_STREAM, 6, "",
|
||||
("127.0.0.1", port))]
|
||||
return self._orig_getaddrinfo(
|
||||
host, port, family, type, proto, flags)
|
||||
|
||||
socket.getaddrinfo = patched
|
||||
# 测试环境自带 egress 代理:urllib 默认会把这个测试主机名
|
||||
# 也送代理并被断开。只给本测试主机名加 no_proxy(测试
|
||||
# 进程内生效),生产抓取代码与代理行为一字不改。
|
||||
self._old_no_proxy = os.environ.get("no_proxy")
|
||||
self._old_NO_PROXY = os.environ.get("NO_PROXY")
|
||||
os.environ["no_proxy"] = (
|
||||
(self._old_no_proxy + ",") if self._old_no_proxy
|
||||
else "") + HOST
|
||||
os.environ["NO_PROXY"] = (
|
||||
(self._old_NO_PROXY + ",") if self._old_NO_PROXY
|
||||
else "") + HOST
|
||||
self.worker = Worker(
|
||||
self.db_path, "/fake/host.py",
|
||||
env={"XCM_API_KEY": "test-placeholder"},
|
||||
runner=lambda cmd: SimpleNamespace(
|
||||
returncode=0, stdout="{}", stderr=""))
|
||||
|
||||
def tearDown(self):
|
||||
# setUp 中途失败时属性可能未建:getattr 防御,避免
|
||||
# tearDown 的 AttributeError 掩盖原始失败。
|
||||
if getattr(self, "_orig_getaddrinfo", None) is not None:
|
||||
socket.getaddrinfo = self._orig_getaddrinfo
|
||||
if getattr(self, "_old_no_proxy", None) is None:
|
||||
if "_old_no_proxy" in self.__dict__:
|
||||
os.environ.pop("no_proxy", None)
|
||||
else:
|
||||
os.environ["no_proxy"] = self._old_no_proxy
|
||||
if getattr(self, "_old_NO_PROXY", None) is None:
|
||||
if "_old_NO_PROXY" in self.__dict__:
|
||||
os.environ.pop("NO_PROXY", None)
|
||||
else:
|
||||
os.environ["NO_PROXY"] = self._old_NO_PROXY
|
||||
try:
|
||||
self.content_server.shutdown()
|
||||
self.content_server.server_close()
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
self.content_thread.join(timeout=5)
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
self.server.shutdown()
|
||||
self.server.server_close()
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
self.thread.join(timeout=5)
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
self.auth.conn.close()
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
self.tmp.cleanup()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def _req(self, method, path, body=None, key=None):
|
||||
conn = http.client.HTTPConnection("127.0.0.1", self.port,
|
||||
timeout=10)
|
||||
hdrs = {}
|
||||
payload = None
|
||||
if body is not None:
|
||||
payload = json.dumps(body).encode("utf-8")
|
||||
hdrs["Content-Type"] = "application/json"
|
||||
if key is not None:
|
||||
hdrs["X-API-Key"] = key
|
||||
try:
|
||||
conn.request(method, path, body=payload, headers=hdrs)
|
||||
resp = conn.getresponse()
|
||||
data = resp.read()
|
||||
finally:
|
||||
conn.close()
|
||||
try:
|
||||
parsed = json.loads(data.decode("utf-8")) if data else None
|
||||
except ValueError:
|
||||
parsed = None
|
||||
return resp.status, parsed
|
||||
|
||||
def _url(self, suffix=""):
|
||||
return "/projects/%s/resident-watches%s" % (
|
||||
self.project["id"], suffix)
|
||||
|
||||
def _create_watch(self):
|
||||
body = {"name": "site1",
|
||||
"target_url": "http://%s:%d/data" % (
|
||||
HOST, self.content_port),
|
||||
"prompt": "变化时汇报",
|
||||
"interval_seconds": 60}
|
||||
status, watch = self._req("POST", self._url(), body,
|
||||
self.owner_key)
|
||||
self.assertEqual(status, 201, watch)
|
||||
return watch
|
||||
|
||||
def _get_watch(self, watch_id):
|
||||
status, body = self._req(
|
||||
"GET", self._url("/" + watch_id), None, self.owner_key)
|
||||
self.assertEqual(status, 200, body)
|
||||
return body
|
||||
|
||||
def _tasks(self):
|
||||
conn = sqlite3.connect(self.db_path)
|
||||
try:
|
||||
return conn.execute(
|
||||
"SELECT id, prompt, status FROM tasks"
|
||||
" ORDER BY created_at, id").fetchall()
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
def _audit(self):
|
||||
conn = sqlite3.connect(self.db_path)
|
||||
try:
|
||||
return [r[0] for r in conn.execute(
|
||||
"SELECT action FROM audit_log WHERE project_id = ?",
|
||||
(self.project["id"],)).fetchall()]
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
def test_full_website_loop_end_to_end(self):
|
||||
# 1. 创建:经服务端 API,running、游标为空、下次轮询在位。
|
||||
watch = self._create_watch()
|
||||
wid = watch["id"]
|
||||
self.assertEqual(watch["status"], "running")
|
||||
self.assertEqual(watch["interval_seconds"], 60)
|
||||
self.assertEqual(watch["fetch_timeout_seconds"], 10)
|
||||
self.assertIsNone(watch["last_content_hash"])
|
||||
self.assertIsNotNone(watch["next_poll_at"])
|
||||
self.assertIn("resident_watch_created", self._audit())
|
||||
t0 = time.time()
|
||||
|
||||
# 2. 基线:真实抓取 version one,只建游标,不建任务。
|
||||
polls = self.worker.process_due_resident_watches(now=t0 + 120)
|
||||
self.assertEqual(len(polls), 1, polls)
|
||||
self.assertEqual(polls[0]["status"], "baseline")
|
||||
self.assertEqual(
|
||||
polls[0]["observed_hash"], _content_hash("version one"))
|
||||
self.assertEqual(self._tasks(), [])
|
||||
body = self._get_watch(wid)
|
||||
self.assertEqual(body["last_content_hash"],
|
||||
_content_hash("version one"))
|
||||
self.assertEqual(len(body["polls"]), 1)
|
||||
self.assertEqual(body["polls"][0]["status"], "baseline")
|
||||
|
||||
# 3. 变化:内容改成 version two,到期轮询建出 queued 任务。
|
||||
self.state["content"] = "version two"
|
||||
polls = self.worker.process_due_resident_watches(now=t0 + 240)
|
||||
self.assertEqual(len(polls), 1, polls)
|
||||
self.assertEqual(polls[0]["status"], "changed")
|
||||
self.assertIsNotNone(polls[0]["task_id"])
|
||||
tasks = self._tasks()
|
||||
self.assertEqual(len(tasks), 1)
|
||||
self.assertEqual(tasks[0][0], polls[0]["task_id"])
|
||||
self.assertEqual(tasks[0][2], "queued")
|
||||
self.assertIn("变化时汇报", tasks[0][1])
|
||||
self.assertIn("version two", tasks[0][1])
|
||||
self.assertIn("resident_watch_change_task_created",
|
||||
self._audit())
|
||||
body = self._get_watch(wid)
|
||||
self.assertEqual(body["last_content_hash"],
|
||||
_content_hash("version two"))
|
||||
self.assertIn("version two", body["last_change_text"])
|
||||
|
||||
# 4. 暂停:暂停期间再改内容,驱动不轮询、游标不动。
|
||||
status, body = self._req(
|
||||
"POST", self._url("/%s/pause" % wid), {}, self.owner_key)
|
||||
self.assertEqual(status, 200, body)
|
||||
self.assertEqual(body["status"], "paused")
|
||||
self.assertIsNone(body["next_poll_at"])
|
||||
self.state["content"] = "version three"
|
||||
polls = self.worker.process_due_resident_watches(now=t0 + 400)
|
||||
self.assertEqual(polls, [])
|
||||
body = self._get_watch(wid)
|
||||
self.assertEqual(len(body["polls"]), 2)
|
||||
self.assertEqual(body["last_content_hash"],
|
||||
_content_hash("version two"))
|
||||
|
||||
# 5. 恢复:用旧游标比对,报出暂停期间的变化并建任务,
|
||||
# 不是从头建基线。
|
||||
status, body = self._req(
|
||||
"POST", self._url("/%s/resume" % wid), {}, self.owner_key)
|
||||
self.assertEqual(status, 200, body)
|
||||
self.assertEqual(body["status"], "running")
|
||||
polls = self.worker.process_due_resident_watches(now=t0 + 400)
|
||||
self.assertEqual(len(polls), 1, polls)
|
||||
self.assertEqual(polls[0]["status"], "changed")
|
||||
tasks = self._tasks()
|
||||
self.assertEqual(len(tasks), 2)
|
||||
self.assertIn("version three", tasks[1][1])
|
||||
body = self._get_watch(wid)
|
||||
self.assertEqual(body["last_content_hash"],
|
||||
_content_hash("version three"))
|
||||
|
||||
# 6. 失败:停服后轮询记 failed、退避、不结束、游标不动;
|
||||
# 恢复服务后内容未变,轮询记 unchanged、失败清零。
|
||||
self.content_server.shutdown()
|
||||
self.content_server.server_close()
|
||||
self.content_thread.join(timeout=5)
|
||||
polls = self.worker.process_due_resident_watches(now=t0 + 600)
|
||||
self.assertEqual(len(polls), 1, polls)
|
||||
self.assertEqual(polls[0]["status"], "failed")
|
||||
body = self._get_watch(wid)
|
||||
self.assertEqual(body["status"], "running")
|
||||
self.assertEqual(body["consecutive_failures"], 1)
|
||||
self.assertEqual(body["last_content_hash"],
|
||||
_content_hash("version three"))
|
||||
self.assertGreater(body["next_poll_at"], t0 + 600 + 60)
|
||||
self.content_server = http.server.ThreadingHTTPServer(
|
||||
("127.0.0.1", self.content_port), self._handler)
|
||||
self.content_thread = threading.Thread(
|
||||
target=self.content_server.serve_forever, daemon=True)
|
||||
self.content_thread.start()
|
||||
polls = self.worker.process_due_resident_watches(now=t0 + 800)
|
||||
self.assertEqual(len(polls), 1, polls)
|
||||
self.assertEqual(polls[0]["status"], "unchanged")
|
||||
body = self._get_watch(wid)
|
||||
self.assertEqual(body["consecutive_failures"], 0)
|
||||
self.assertEqual(len(self._tasks()), 2)
|
||||
|
||||
# 7. 结束:终态,驱动不再轮询,改/停/恢复一律 409。
|
||||
status, body = self._req(
|
||||
"POST", self._url("/%s/end" % wid), {}, self.owner_key)
|
||||
self.assertEqual(status, 200, body)
|
||||
self.assertEqual(body["status"], "ended")
|
||||
self.assertIsNone(body["next_poll_at"])
|
||||
self.assertEqual(body["last_content_hash"],
|
||||
_content_hash("version three"))
|
||||
polls = self.worker.process_due_resident_watches(now=t0 + 1000)
|
||||
self.assertEqual(polls, [])
|
||||
for action in ("pause", "resume", "end"):
|
||||
status, body = self._req(
|
||||
"POST", self._url("/%s/%s" % (wid, action)), {},
|
||||
self.owner_key)
|
||||
self.assertEqual(status, 409, action)
|
||||
status, _ = self._req(
|
||||
"PATCH", self._url("/" + wid), {"interval_seconds": 120},
|
||||
self.owner_key)
|
||||
self.assertEqual(status, 409)
|
||||
body = self._get_watch(wid)
|
||||
self.assertEqual(body["status"], "ended")
|
||||
self.assertEqual(len(body["polls"]), 5)
|
||||
|
||||
def test_run_loop_boundary_calls_resident_driver(self):
|
||||
# 8. 驱动接线:run_loop 轮次边界确实调到常驻驱动,
|
||||
# 不只是测试直接调 process_due。run_all 等无关边界
|
||||
# 在本测试里打桩,只留 resident 驱动走真实抓取。
|
||||
watch = self._create_watch()
|
||||
wid = watch["id"]
|
||||
worker = self.worker
|
||||
calls = []
|
||||
original = Worker.process_due_resident_watches
|
||||
|
||||
def recording():
|
||||
# 实例属性上的函数不会自动绑定 self,必须闭包捕获
|
||||
# worker,不能写成 def recording(self)。
|
||||
calls.append(time.time())
|
||||
return original(worker)
|
||||
|
||||
worker.process_due_resident_watches = recording
|
||||
worker.run_all = lambda: []
|
||||
worker.cleanup_stale_containers = lambda: None
|
||||
worker._write_heartbeat = lambda: None
|
||||
worker.register = lambda: None
|
||||
worker._process_push_due = lambda: None
|
||||
worker.process_auto_memories = lambda: None
|
||||
worker._reap_kill_requests = lambda *a, **k: None
|
||||
worker.auto_resume_interrupted = lambda: None
|
||||
worker._heartbeat_running_tasks = lambda: None
|
||||
worker._registry_heartbeat = lambda: None
|
||||
worker._code_changed = lambda: False
|
||||
worker._code_signature_valid = lambda: True
|
||||
worker.run_loop(interval=0.05, max_seconds=0.4)
|
||||
self.assertGreaterEqual(len(calls), 1)
|
||||
body = self._get_watch(wid)
|
||||
self.assertEqual(len(body["polls"]), 1)
|
||||
self.assertEqual(body["polls"][0]["status"], "baseline")
|
||||
self.assertEqual(body["last_content_hash"],
|
||||
_content_hash("version one"))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -6582,6 +6582,33 @@ class Worker:
|
||||
log_event(self.logger, "worker_push_due_failed",
|
||||
message="push due processing failed: %s" % exc)
|
||||
|
||||
def process_due_resident_watches(self, now=None, fetcher=None):
|
||||
"""常驻监听驱动(worker 轮次边界调用):到期监听逐条
|
||||
走 resident.process_due_resident_watches,抢单 + 廉价
|
||||
抓取 + 仅变化建任务。全部异常只记日志,不炸循环;
|
||||
fetcher 缺省用 resident 默认抓取,单测可经
|
||||
self.resident_fetcher 或参数注入假抓取。"""
|
||||
if now is None:
|
||||
now = time.time()
|
||||
if fetcher is None:
|
||||
fetcher = getattr(self, "resident_fetcher", None)
|
||||
try:
|
||||
from cloud.resident import process_due_resident_watches as _run
|
||||
conn = _db_connect(self.db_path)
|
||||
try:
|
||||
polls = _run(conn, fetcher=fetcher, now=now)
|
||||
finally:
|
||||
conn.close()
|
||||
except Exception as exc:
|
||||
log_event(self.logger, "worker_resident_watch_failed",
|
||||
message="resident watch processing failed: %s" % exc)
|
||||
return []
|
||||
for poll in polls:
|
||||
log_event(self.logger, "worker_resident_watch_poll",
|
||||
message="resident watch poll: %s %s"
|
||||
% (poll["watch_id"], poll["status"]))
|
||||
return polls
|
||||
|
||||
def run_loop(self, interval=300, max_seconds=None, stop_event=None,
|
||||
drain_timeout=60, sweep_threshold=300):
|
||||
"""常驻循环:每 interval 秒跑一轮 run_all,空闲也不退出。
|
||||
@@ -6664,6 +6691,10 @@ class Worker:
|
||||
# G11/G12 后台投递:摘要(每小时)与 outbox 重试,
|
||||
# 轮次边界做,不中断在跑任务,失败只记日志。
|
||||
self._process_push_due()
|
||||
# 常驻监听驱动:同款轮次边界后台扫,到期监听
|
||||
# 廉价抓取、仅变化建任务;服务端本身不起定时器,
|
||||
# 这里就是常驻任务的实际驱动(见 resident 模块)。
|
||||
self.process_due_resident_watches()
|
||||
# 自动记忆提取:同款轮次边界后台扫,失败只记日志
|
||||
# (process_auto_memories 内部已兜底,不炸循环)。
|
||||
self.process_auto_memories()
|
||||
|
||||
@@ -0,0 +1,201 @@
|
||||
# 常驻监听网站闭环:独立架构与代码质量审核报告
|
||||
|
||||
审核员非实现者。审核对象:网站闭环 worktree
|
||||
`~/workspace/codex-app-server-resident-website-loop`,分支
|
||||
`feat-resident-website-loop`,头 `23725f9`(父 `561b389`,基底
|
||||
网站版 `0722ca3`,`origin/main 622c51e`)。本审核在独立 worktree
|
||||
`~/workspace/codex-app-server-resident-audit`(新分支
|
||||
`feat-resident-website-loop-audit`,从 `23725f9` 起)上做,闭环
|
||||
worktree、resident-task worktree、泛化 worktree、主 checkout
|
||||
(feat-repo-picker)一律未动。
|
||||
|
||||
## 0. 评分标准(先定标准,再打分)
|
||||
|
||||
只评架构与代码质量两项,满分各 10 分。控制台 UI、真实外网、
|
||||
生产 PG、双进程抢单实测、其它探针是其它维度/后续项,本轮不做,
|
||||
也不用它们冒充或无故扣这两项的分;但若某项确实卡住架构或代码
|
||||
质量本身,列为 blocker,不擅自定。
|
||||
|
||||
### 架构 10 分标准
|
||||
|
||||
- A1 分层干净:resident 模块自有状态机,与任务状态机、自动化
|
||||
定义层互不干扰;server 只做路由/权限/审计,worker 只做驱动
|
||||
边界调用;无循环依赖。
|
||||
- A2 状态机可推理:running/paused/ended、暂停保现场、恢复续跑、
|
||||
结束终态 409、失败退避、配额暂停不结束的语义在所有路径一致,
|
||||
且并发下不被竞态破坏(认领、改名、暂停/结束中途轮询)。
|
||||
- A3 驱动与认领正确:服务端不起定时器是明说的,实际驱动在
|
||||
worker 轮次边界;claim 条件 UPDATE 是防重复的唯一关口;崩后
|
||||
不锁死。
|
||||
- A4 快照/变化语义干净:轮询廉价、仅变化建任务、首次只建基线、
|
||||
每次轮询可查;变化摘录与游标推进在同一事务边界内。
|
||||
- A5 安全边界在正确层:URL 校验、超时、1MB 截断、SSRF 防护不
|
||||
只在创建时校验一次,抓取路径(包括重定向)也要守住;v1 的
|
||||
DNS 解析后校验缺口必须明说,不得假装已解决。
|
||||
- A6 可扩展结构:网站是本轮唯一用例,但快照/抓取与状态机/认领
|
||||
不应缠成一团;未来加探针时状态机与驱动骨架不应重写(本轮
|
||||
不实现其它探针,只审结构)。
|
||||
|
||||
### 代码质量 10 分标准
|
||||
|
||||
- Q1 命名、函数长度与复杂度可读;长函数(run_due)分支清晰,
|
||||
不靠注释掩盖逻辑。
|
||||
- Q2 无重复、无死代码、无测试值渗入生产代码;复用已有口径
|
||||
(自动化 claim/配额/审计、环境名唯一索引先例)而不是另起一套。
|
||||
- Q3 错误处理诚实:失败、配额耗尽、配额检查失败三条路不混淆;
|
||||
审计与轮询记录不静默丢失;异常不炸驱动循环。
|
||||
- Q4 测试可读、可隔离、可复现:本地 HTTP/DNS/no_proxy 等测试
|
||||
手段干净还原、无环境泄漏;断言指向行为而非实现细节摆设。
|
||||
- Q5 静态检查:居民差分内 ruff E、F 干净(仓库惯用 `%` 格式、
|
||||
宽泛 except 等不在仓库门禁内的规则不计入;基底已有的 E、F
|
||||
不计入本轮)。
|
||||
|
||||
## 1. 改动前分数
|
||||
|
||||
- 架构:9.0 / 10
|
||||
- 代码质量:9.0 / 10
|
||||
|
||||
基底是好的:分层、状态机、claim、配额复用、轮询可查、测试
|
||||
隔离(DNS/no_proxy 在 loop 测试里干净还原)都属实,定向 15 项
|
||||
基线全过(见第 4 节)。扣分只来自下面逐项 findings,不是整体
|
||||
推倒重来。
|
||||
|
||||
## 2. Findings 清单(改动前)
|
||||
|
||||
### 严重
|
||||
|
||||
- F1 名称唯一性是先查后插、无数据库约束:create_watch /
|
||||
update_watch 先 SELECT 查重再 INSERT/UPDATE,两个并发请求
|
||||
可同时通过检查、插出同项目同名 running 监听;server 的
|
||||
environments 已有先例:`CREATE UNIQUE INDEX … WHERE status
|
||||
!= 'archived'`。本轮用同款部分唯一索引
|
||||
`WHERE status != 'ended'`(已结束名可重用)+ 完整性错误转
|
||||
ValueError 修复。
|
||||
- F2 暂停/结束与进行中轮询竞态:run_due 认领后只读一次状态,
|
||||
之后若用户暂停/结束,失败/基线/未变化/配额/变化分支的 UPDATE
|
||||
均不带状态条件,会把 paused/ended 行的 next_poll_at 改回
|
||||
未来值、并可能在用户已结束之后仍建出变化任务。修复:认领后
|
||||
复查状态;全部终局 UPDATE 带 `status='running'` 条件,条件
|
||||
不满足时回滚、不建任务、不记轮询。
|
||||
|
||||
### 一般
|
||||
|
||||
- F3 抓取路径 SSRF 只守了创建时:default_fetch 用默认 opener,
|
||||
重定向目标不校验,公开 URL 重定向到内网地址可绕过创建校验;
|
||||
另 localhost.、单整数/十六进制/前导零等非规范 IP 写法可绕过
|
||||
字面 IP 检查。修复:重定向处理器逐跳校验(相对跳转先 urljoin
|
||||
再校验);校验时归一化主机(去尾点、*.localhost 挡掉)并拒绝
|
||||
非规范数字 IP 写法。DNS 解析后校验仍属 v1 缺口,继续列为已知
|
||||
偏差,不假装已解决。
|
||||
- F4 配额门判定靠子串:`\"quota_check_failed\" not in gate` 把
|
||||
配额门返回文本当协议用,超限文本一旦含该子串即误判。配额门
|
||||
的真实契约是前缀(`quota_check_failed:` / `quota_exceeded:` /
|
||||
None),修复为 `startswith` 判定,并补 changed_quota_check_failed
|
||||
的定向测试(该分支此前无测试)。
|
||||
- F5 row_get 重复造轮:resident 自带与 automations.row_get 逐字
|
||||
同语义的 row_get,却已从 automations 导入另外两个函数。修复:
|
||||
直接导入 automations.row_get,删本地定义。
|
||||
- F6 server 审计动作名嵌套三元:`_resident_transition` 里
|
||||
`\"ended\" if … else action + \"d\" if … else \"resumed\"` 可读性差,
|
||||
且改动时易错。修复为显式动作映射字典,审计字符串不变。
|
||||
|
||||
### 建议
|
||||
|
||||
- S1 test_resident.py 导入了未用的 `time`(ruff F401,居民差分
|
||||
内唯一 E、F 问题)。删除。
|
||||
- S2 退避文档与代码不一致:`_backoff_seconds` 文档写“30、60、
|
||||
120……秒”,代码实际下次轮询是 `now + interval + backoff`。
|
||||
行为本身合理(不比间隔更频繁),文档误导。修复文档,行为不动。
|
||||
- S3 loop 测试 tearDown 不够防御:setUp 中途失败时 tearDown
|
||||
访问未建属性会掩盖原始失败。修复为 getattr 防御。
|
||||
|
||||
### 已核查、不是问题(不扣分)
|
||||
|
||||
- 审计行“写在 commit 之后会丢”:核查 `_auto_audit_row` 本身
|
||||
执行 INSERT 后自己 commit,失败/配额暂停/变化三处审计行在
|
||||
模拟关闭连接后仍可查,不是缺陷。
|
||||
- server/worker 全量 ruff E、F 的其余项(如 worker.py:2781/2787
|
||||
的 F821、server.py:10293 的 E501)经核对均为基底已有或由居民
|
||||
插入导致的行号位移,非居民新增代码,不计入本轮。
|
||||
- 仓库默认 ruff 全规则的 39 项(如 UP031 `%` 格式)与仓库惯用
|
||||
写法冲突,不在仓库门禁内,不计入。
|
||||
|
||||
## 3. 修复(全部关闭,每项有对应测试或定向复核)
|
||||
|
||||
- F1 名称唯一性:`ensure_resident_tables` 新增部分唯一索引
|
||||
`idx_resident_watches_name ON resident_watches(project_id, name)
|
||||
WHERE status != 'ended'`(server environments 同款口径);
|
||||
create/update 的 INSERT/UPDATE 包完整性错误转 ValueError
|
||||
(sqlite3.IntegrityError / PostgresIntegrityError,调用方仍
|
||||
返回 400)。测试:索引存在断言、绕过查重直接插报
|
||||
IntegrityError、改名撞名报 ValueError。
|
||||
- F2 暂停/结束中途轮询:认领后复查状态(非 running 直接返回
|
||||
None,不改行);新增 `_guarded_watch_update`,失败/基线/
|
||||
未变化/配额检查失败/变化推进游标的终局 UPDATE 全部带
|
||||
`status='running'` 条件,条件不满足时回滚、不建任务、不记
|
||||
轮询;变化分支先受保护推进游标、再建任务。测试:抓取回调
|
||||
里经第二连接暂停监听,run_due 返回 None、无任务、游标与
|
||||
next_poll_at 保持暂停态、轮询数不变。
|
||||
- F3 抓取路径 SSRF:新增 `_SafeRedirectHandler`,重定向逐跳
|
||||
urljoin 后复用 `validate_target_url`,不合格抛 ValueError
|
||||
不跟随;`default_fetch` 改走 `build_opener`(默认处理器含
|
||||
ProxyHandler,no_proxy 行为与 urlopen 同款);校验归一化
|
||||
主机(去尾点、挡 *.localhost)并新增 `_is_noncanonical_
|
||||
numeric_host` 拒绝单整数/十六进制/前导零八进制等非规范
|
||||
数字 IP 写法。测试:localhost.、*.localhost、2130706433、
|
||||
0x7f000001、0177.0.0.1、0x7f.0.0.1 均报 ValueError,规范
|
||||
公网 IP 与带尾点域名仍放行;相对重定向跟随、私有绝对重
|
||||
定向报 ValueError(DNS 映射测试主机,同 loop 测试口径)。
|
||||
- F4 配额门判定:`\"quota_check_failed\" not in gate` 改为
|
||||
`gate.startswith(\"quota_check_failed\")`,与配额门前缀契约
|
||||
对齐。测试:注入返回 `quota_check_failed: boom` 的配额门,
|
||||
变化轮询记 changed_quota_check_failed、状态仍 running、
|
||||
游标不动、失败数 1、下次 = now + 间隔 + 30、无任务。
|
||||
- F5 row_get 重复:删 resident 本地定义,直接导入
|
||||
automations.row_get,内部调用点一字未动,模块依赖说明
|
||||
同步更新。
|
||||
- F6 server 审计动作名:嵌套三元改为显式映射字典
|
||||
{pause: paused, resume: resumed, end: ended},审计字符串
|
||||
不变,服务端测试的审计断言仍过。
|
||||
- S1 删 test_resident.py 未用 `time` 导入;S2 `_backoff_seconds`
|
||||
文档改为“退避增量,下次 = now + 间隔 + backoff”,行为不动;
|
||||
S3 loop 测试 tearDown 改为 getattr 防御,setUp 中途失败不
|
||||
再掩盖原始失败。
|
||||
|
||||
## 4. 测试数字(全部本轮实测,无删断言、无 skip、无改默认值)
|
||||
|
||||
- 定向基线(改动前):`cloud.tests.test_resident` +
|
||||
`test_resident_server` + `test_resident_website_loop` = 15 项全过。
|
||||
- 定向改动后:同三文件 = 20 项全过(基线 15 + 新增 5:URL
|
||||
变体、唯一索引、配额检查失败、中途暂停、重定向),4.450 秒;
|
||||
ruff E、F 在居民差分 4 文件上全部通过。
|
||||
- 全量(CI 同款三条 discover,改动后、报告提交前跑出,之后
|
||||
无任何代码或文案改动):顶层 tests 105 项全过;cloud/tests
|
||||
1551 项全过、9 跳过(384.252 秒);deploy/tests 44 项全过
|
||||
(单独重跑确认);合计 1700 项、0 失败、9 跳过。对账:闭环
|
||||
基线 1695(105 + 1546 + 44)+ 新增 5 = 1700,无测试丢失、
|
||||
无新失败。
|
||||
- 覆盖率:本仓库 CI 无覆盖率门禁,不虚构覆盖率变化;新增
|
||||
5 项定向测试即上述全量的一部分。
|
||||
|
||||
## 5. 复评分数
|
||||
|
||||
- 架构:10 / 10
|
||||
- 代码质量:10 / 10
|
||||
|
||||
复评依据:F1/F2 两项严重 findings 已由数据库约束与状态条件
|
||||
UPDATE 从机制上关闭(非仅文档说明),并有定向测试锁定;
|
||||
F3 安全边界已从“创建时一次校验”补到“抓取重定向逐跳校验”
|
||||
与非规范 IP 写法;F4/F5/F6/S1/S2/S3 全部关闭且测试全过。
|
||||
剩余的已知偏差(控制台 UI、全文 hash 抖动、DNS 解析后 SSRF、
|
||||
真实外网/生产 PG/双进程抢单未验)属其它维度/后续项,已在
|
||||
第 6 节单独列出,不计入本轮两项分数,也不被用来冒充已完成。
|
||||
|
||||
## 6. 剩余 blocker 与未做事项
|
||||
|
||||
- 未做:未 push、未合并、未跑 deploy.py、未部署;不做其它探针、
|
||||
不接控制台 UI、不做生产 PG 与双进程抢单实测。
|
||||
- 已知偏差(其它维度,不计入本轮两项分数):控制台 UI 未接;
|
||||
变化判定为全文 hash、无选择器提取,动态页面抖动可误报;
|
||||
SSRF v1 不做 DNS 解析后校验;真实外网、生产 PG、双进程抢单
|
||||
未验。
|
||||
@@ -0,0 +1,69 @@
|
||||
# 常驻监听任务(resident watch)验证摘要
|
||||
|
||||
## 基底与交付物(代码就绪,local only)
|
||||
|
||||
- 工作区:`~/workspace/codex-app-server-resident-task`
|
||||
- 分支:`feat-resident-task`,基底 `origin/main` = `622c51e`(Merge PR #274,2026-11-02 fetch 到的头,与任务书一致)。
|
||||
- 主 checkout(`~/workspace/codex-app-server`,`feat-repo-picker`,有脏文件)与其它现存 worktree 一律未动。
|
||||
- 本轮提交前状态:`cloud/server.py` +187 行、`cloud/worker.py` +31 行,新增 `cloud/resident.py`、`cloud/tests/test_resident.py`、`cloud/tests/test_resident_server.py`、本摘要。
|
||||
- **未做**:未 push、未合并到 main、未跑 `deploy.py`、未部署、未建生产任务、未动生产/主机 env/系统 key/sub2api/生产网关/仓库可见性/GitHub App 权限、未杀生产容器。
|
||||
|
||||
## 基底核实(先调查结论:main 上常驻监听不存在,不是重复建设)
|
||||
|
||||
- 任务状态机仍是 `queued -> running -> completed / failed / interrupted / merged / merge-failed`;`/pause` 只是把进行中回合任务标 `interrupted` 并置 `paused_by_user=1`(`cloud/server.py` `_pause_task`),不是可续跑的常驻暂停态——本实现不复用它,另建常驻态。
|
||||
- 云端自动化(`cloud/automations.py`)定义层只有 `active / cancelled`,含 `automations / automation_runs` 表与 `run_due_automation` 的 claim 抢单、配额门;服务端本身不起定时器,到期触发靠外部触发器。本实现复用它的 claim 口径与配额函数,不另起冲突语义。
|
||||
- goal 循环(`host.py` `run_goal_loop`,`max_rounds=0` 不限轮次)结束条件是模型宣告达成,不是“一直监听外部变化”;定时自动化每次到期建一个新任务跑一轮就结束。两者都不是常驻监听,本摘要不把它们说成已实现。
|
||||
- main 上与暂停相关的已有机制(MCP `pause_task` 工具、长任务中断自动恢复)针对的都是回合任务,不是常驻监听;grep `pause|resident` 在 `cloud/*.py` 与 `host.py` 中未发现常驻监听实现。
|
||||
|
||||
## 设计抉择
|
||||
|
||||
- 新层不改任务表:`cloud/resident.py`(标准库 + 复用 `cloud.automations` 的 `_audit_row`、`_automation_quota_exceeded`)新增 `resident_watches` / `resident_polls` 两张表,避免改动任务状态机牵连 worker 认领、sweep、自动恢复。
|
||||
- 状态机:`running -> paused -> running`;`running / paused -> ended`(终态)。`ended` 后 pause / resume / end / update 一律 409,恢复已结束的监听必须新建任务(文档与测试双锁定,见下)。
|
||||
- 暂停保留现场:暂停只改 `status / paused_reason / next_poll_at`,`last_content_hash / last_poll_at / last_change_at / last_change_text` 不动;暂停期间 `due=False`,驱动不轮询。恢复时 `next_poll_at=now` 立刻复查,用旧游标比对——暂停期间的变化在恢复后第一次轮询即报出,不丢。
|
||||
- 驱动(明说):服务端本身不起定时器,与自动化同款结论。实际驱动是 `cloud/worker.py` 的 `run_loop` 轮次边界调 `process_due_resident_watches()`(新增方法,异常只记日志、不炸循环;`self.resident_fetcher` 可注入假抓取供测试)。任何外部触发器也可直接调 `run_due_resident_watch / process_due_resident_watches`,两者都走 claim 门。
|
||||
- 防重复轮询:`claim_due_watch` 条件 UPDATE(`status='running'` 且 `next_poll_at <= now`),抢到才把 `last_poll_at / last_claimed_at` 写成 now 并预置 `next_poll_at=now+interval`(崩在抓取中途时下个间隔仍重试,不因认领锁死永久停摆;完成后按成败重算)。
|
||||
- 变化才花钱:轮询 = 廉价抓取 + 空白折叠后 sha256 比对,不跑模型。`baseline`(首次,只建游标)、`unchanged`(不建任务)、`changed`(才建一个 queued 任务,prompt = 用户处理提示 + 变化摘录)三种产出都记 `resident_polls`,可查。
|
||||
- 配额边界:配额门只在“变化要建模型任务”那一刻过(复用自动化同款 `_automation_quota_exceeded`)。耗尽时**不结束、不丢变化**:转 `paused`(`paused_reason='quota_exceeded'`)、游标不推进、记 `changed_quota_paused`;配额恢复后恢复,同一变化仍能报出。配额检查本身失败时按瞬时失败退避,不转暂停,游标同样不推进。
|
||||
- 安全边界:间隔下限 60 秒(与自动化同款)、上限 30 天;抓取超时 1–60 秒(默认 10 秒)、读满 1MB 即断;失败退避 30 / 60 / 120……秒、上限 3600 秒;目标 URL 仅 http/https、无 userinfo、字面 localhost/私有/回环/链路本地 IP 拒绝。
|
||||
- API(`cloud/server.py`,权限与自动化同款:写 `create_task`、读 `read`,全部记审计):
|
||||
- `POST /projects/{id}/resident-watches`(name / target_url / prompt 必填;interval_seconds 默认 3600;fetch_timeout_seconds 可选;创建不走配额门)
|
||||
- `GET /projects/{id}/resident-watches`(附 `due_count`)、`GET /projects/{id}/resident-watches/{wid}`(附最近轮询 `polls`)
|
||||
- `PATCH /projects/{id}/resident-watches/{wid}`(改名/提示/目标/间隔/超时;ended 409;改间隔不立即触发,下次轮询完成后按新间隔重算)
|
||||
- `POST /projects/{id}/resident-watches/{wid}/pause|resume|end`
|
||||
- 状态查询返回常驻态、游标(`last_content_hash`)、最后变化(`last_change_at / last_change_text`)、下次轮询(`next_poll_at`)、失败次数与暂停原因。
|
||||
|
||||
## 验收清单逐项结果
|
||||
|
||||
1. 能新建常驻任务(目标 URL、轮询间隔、变化处理提示):✅ 模块与服务端测试覆盖;非空壳——新建即到期,驱动下一轮即可认领。
|
||||
2. 三态清晰(运行中/已暂停/已结束终态):✅ 状态机与 HTTP 409 语义双覆盖。
|
||||
3. 暂停保留现场、暂停期间停轮询、恢复从游标续跑:✅ 测试 `test_pause_preserves_cursor_and_resume_reports_pause_period_change`(暂停期间变了,恢复后第一次轮询即报出)。
|
||||
4. 结束终态、结束后不再静默重启、结束后恢复被拒绝:✅ `ended` 后 due 查询只认 `running`,pause / resume / end / update 一律 409(服务端错误文本含 terminal 并提示新建);测试覆盖。
|
||||
5. 真正有驱动、claim 防重复、心跳与 stale 回收:✅ 驱动为 worker `run_loop` 轮次边界(不是服务端定时器,明说);claim 单胜者测试覆盖;stale 口径为“认领预置 next_poll_at,崩后下个间隔重试”+ worker 既有每轮心跳/sweep(常驻监听本身不另起心跳表,见已知偏差)。
|
||||
6. 变化才花钱、每次轮询与变化可查:✅ baseline / unchanged 不建任务、changed 才建任务;`resident_polls` 逐轮记录。
|
||||
7. 间隔下限、抓取超时、失败退避、除配额耗尽外不擅自结束、配额耗尽只转可恢复暂停并明示原因:✅ 全部测试覆盖;配额耗尽测试断言状态为 `paused` 且明确 `!= ended`。
|
||||
8. 权限与审计沿用任务/自动化同款、pause / resume / end 记审计:✅ 服务端测试断言 `resident_watch_created / paused / resumed / ended` 审计行;跨项目行一律 404(不泄露存在性);局外人写操作被拒。
|
||||
9. API 含暂停、恢复、结束与状态查询:✅ 见上;控制台 UI 本轮未接,列为已知偏差,不写成已完成。
|
||||
|
||||
## 真实测试数字
|
||||
|
||||
- 定向(改动后):`python3 -m unittest cloud.tests.test_resident cloud.tests.test_resident_server` —— 13 项(模块 10 + 服务端 3),全部通过。
|
||||
- 全量(CI 同款三条 discover,改动后、仅最后一次文案注释修订前跑出;该修订只改 `update_watch` 的 docstring,不改任何代码逻辑,修订后定向 13 项与自动化相关 31 项重跑全部通过):
|
||||
- `discover -s tests`:105 项,0 失败,0 跳过。
|
||||
- `discover -s cloud/tests`:1544 项,0 失败,9 跳过。
|
||||
- `discover -s deploy/tests`:44 项(单独重跑确认),0 失败。
|
||||
- 合计 1693 项、0 失败、9 跳过。失败未靠删断言、加 skip、改默认值蒙混;新增测试 13 项全部计入上述 cloud 套件。
|
||||
- 新模块定向覆盖率(`coverage run --source=cloud.resident`,仅定向 13 项):`cloud/resident.py` 312 语句、85% 覆盖。未覆盖主因:默认真实 HTTP 抓取(`default_fetch`,单元测试一律注入假抓取,不联网)与个别异常分支(PG 方言补列、配额检查失败分支);本仓库 CI(`.github/workflows/host-verify.yml`)本身没有覆盖率门禁,故不报“覆盖率变化”——全量覆盖率基线未在本轮重算,不虚构数字。
|
||||
|
||||
## 已知偏差与未做事项
|
||||
|
||||
- 控制台 UI 未接:`cloud/console.html` 是打包产物,本轮没有常驻监听面板;用户现阶段只能走 API。接 UI 是后续一轮,不在本轮验收内。
|
||||
- 变化判定 v1 较粗:空白折叠后全文 sha256,没有选择器/区域提取;动态页面(广告、时间戳、随机推荐)本身会抖动成“变化”并建任务。缓解在后续:加 CSS 选择器或正文提取口径、或变化阈值。
|
||||
- SSRF 防护 v1 只挡字面 localhost/私有 IP,不做 DNS 解析后校验(解析后重定向到内网、DNS rebinding 不在覆盖内);`websearch.fetch_url` 的同款字面口径已对齐,但不要把 v1 当成完整 SSRF 方案。
|
||||
- 常驻监听没有独立心跳表:stale 回收靠认领预置的 `next_poll_at` + worker 既有心跳/sweep;驱动进程整体死亡时,已认领未完成的当次轮询要等到下个间隔才补跑(与自动化 claim 的同款取舍)。
|
||||
- 未验证:真实外网抓取(单元测试不联网)、生产 PG 上的表迁移(`PRAGMA` 补列走自动化同款适配写法,但未在 PG 实测)、worker 常驻循环里的长时驱动(仅方法级测试,未跑真实 `run_loop` 长循环)、多 worker 并发抢单(仅条件 UPDATE 单胜者测试,未做双进程实测)。
|
||||
- 未做:未 push、未合并、未部署;UI、选择器提取、DNS 级 SSRF、PG 实测、双进程抢单实测留待后续轮次。
|
||||
|
||||
## Blocker
|
||||
|
||||
- 无本轮 blocker(全部工作均为本地代码与本地测试,未触及只有用户能给的账号/密钥/商户/商店账号/宿主/维护窗口,也未发生花真钱、删生产数据、改生产可见性的动作)。
|
||||
- 上线前仍需用户侧决策(不是本轮 blocker,不擅自定):控制台 UI 是否随本批上、默认轮询间隔与抓取超时是否按本轮默认值(60 秒下限/10 秒超时)拍板为生产默认值、PG 实测与部署窗口。
|
||||
@@ -0,0 +1,125 @@
|
||||
# 常驻监听网站闭环验证摘要
|
||||
|
||||
## 起点与交付物(代码就绪,local only)
|
||||
|
||||
- 工作区:`~/workspace/codex-app-server-resident-website-loop`
|
||||
- 分支:`feat-resident-website-loop`,起点为上一轮网站版
|
||||
`feat-resident-task` 的 `0722ca3`(基底 `origin/main 622c51e`)。
|
||||
- 实现提交:`561b389`(仅新增端到端测试文件
|
||||
`cloud/tests/test_resident_website_loop.py`,364 行)。
|
||||
本摘要为其后提交,工作区提交后干净。
|
||||
- 范围:只做网站用例。不做 server_status、agent_status、
|
||||
relay_cost 等其它探针;不把 resident-watches 改名/泛化成
|
||||
通用任务;不改 API、表名与任何生产默认值。
|
||||
- 泛化 worktree(`~/workspace/codex-app-server-resident-generic`)
|
||||
本轮一律未碰;resident-task worktree、主 checkout
|
||||
(`feat-repo-picker`)及其它 worktree 一律未动。
|
||||
- **未做**:未 push、未合并到 main、未跑 `deploy.py`、未部署、
|
||||
未建生产任务、未动生产/主机 env/系统 key/sub2api/
|
||||
生产网关/仓库可见性/GitHub App 权限、未杀生产容器、
|
||||
未连真实外部网站。
|
||||
|
||||
## 闭环逐环证据(全部真实链路,不是注入假抓取)
|
||||
|
||||
测试文件:`cloud/tests/test_resident_website_loop.py`,2 项。
|
||||
内容服务是线程内真实 `http.server`(可变内容、可停服重启);
|
||||
控制面是真实 `ThreadingHTTPServer` + 真实 SQLite 临时库;
|
||||
驱动走真实 `default_fetch`(urllib、超时、1MB 截断)。
|
||||
生产 URL 校验一字不改:测试主机名 `website-loop.test` 不是
|
||||
IP 字面、不是 localhost,所以校验通过;测试进程内只把该
|
||||
主机名的 DNS 指到本地服务(相当于 /etc/hosts 的一条测试
|
||||
条目),并给它加 `no_proxy`(见“测试中发现的两处假断点”)。
|
||||
|
||||
1. 创建(API):`POST /projects/{id}/resident-watches`
|
||||
(name=site1、target_url=`http://website-loop.test:<本地端口>/data`、
|
||||
prompt=“变化时汇报”、interval_seconds=60)返回 201,
|
||||
状态 running、interval 60、fetch_timeout 10(生产默认)、
|
||||
last_content_hash 为空、next_poll_at 在位;审计含
|
||||
`resident_watch_created`。状态查询 `GET` 单条可看到
|
||||
running、游标、下次轮询。
|
||||
2. 基线轮询:worker 驱动 `process_due_resident_watches`
|
||||
(fetcher=None,即真实 default_fetch)抓到 “version one”,
|
||||
记 `baseline` 轮询、游标为其 sha256,任务数为 0;
|
||||
状态查询的 polls 第 1 条就是 baseline。
|
||||
3. 变化建任务:内容改成 “version two”,到期轮询记
|
||||
`changed`、建出 1 个 queued 任务,任务 prompt 含用户提示
|
||||
“变化时汇报”与摘录 “version two”,任务行与审计行
|
||||
`resident_watch_change_task_created` 均可查;游标推进到
|
||||
version two,last_change_text 含 version two。
|
||||
4. 暂停:API pause 后状态 paused、next_poll_at 为空;暂停
|
||||
期间内容再改成 “version three”,驱动返回空、轮询数仍为
|
||||
2、游标仍为 version two——暂停期间的变化没有被轮询上报。
|
||||
5. 恢复:API resume 后状态 running;下一次驱动用旧游标比
|
||||
对,记 `changed` 并建第 2 个任务(prompt 含 version three),
|
||||
不是从头建基线;游标推进到 version three。
|
||||
6. 失败环:停掉内容服务后驱动记 `failed` 轮询、
|
||||
consecutive_failures=1、next_poll_at 大于 now+60+30
|
||||
(间隔 + 首次退避 30 秒)、状态仍 running、游标仍为
|
||||
version three——驱动不崩、监听不擅自结束。恢复服务后
|
||||
(内容仍为 version three)驱动记 `unchanged`、失败清零、
|
||||
任务数仍为 2。
|
||||
7. 结束:API end 后状态 ended、next_poll_at 为空、游标仍为
|
||||
version three;驱动返回空;pause / resume / end / PATCH
|
||||
更新一律 409;状态查询显示 ended 与最后游标/变化,
|
||||
polls 总数为 5(baseline、changed、changed、failed、
|
||||
unchanged)。
|
||||
8. 驱动接线:单独一测试把 `run_all` 等无关边界打桩,只留
|
||||
resident 驱动走真实抓取,调真实 `Worker.run_loop`
|
||||
(interval=0.05、max_seconds=0.4);记录到 run_loop 轮次
|
||||
边界至少调了 1 次 `process_due_resident_watches`,且状态
|
||||
查询里出现 1 条 baseline 轮询、游标为 version one。
|
||||
这证明接线在 run_loop 里,不只是测试直接调驱动函数。
|
||||
|
||||
## 断点结论:生产代码无断点,无生产改动
|
||||
|
||||
本轮逐环均通,生产代码(`cloud/resident.py`、`cloud/server.py`、
|
||||
`cloud/worker.py`)一字未改,`git diff` 为空、新增文件只有
|
||||
端到端测试。第一次跑出的 2 个失败都是测试脚手架问题,
|
||||
不是生产断点,修法与证据如下,不许读成“修了生产代码”:
|
||||
|
||||
- 假断点 1——首次真实抓取 failed:测试环境自带 egress 代理,
|
||||
urllib 默认把测试主机名也送代理并被代理断开。实测对照:
|
||||
不加 no_proxy 时 `default_fetch` 报 RemoteDisconnected,
|
||||
加 no_proxy 后返回 “version one”。修法只在测试 setUp 里
|
||||
给 `website-loop.test` 加 no_proxy(测试进程内生效,撕掉
|
||||
时还原),生产抓取与代理行为一字不改。
|
||||
- 假断点 2——run_loop 接线测试 TypeError:实例属性上的
|
||||
recording 函数不会自动绑定 self。修法改为闭包捕获 worker,
|
||||
不改 worker 代码。
|
||||
|
||||
## 真实测试数字
|
||||
|
||||
- 定向(全量后重跑,含本轮 2 项):`python3 -m unittest
|
||||
cloud.tests.test_resident cloud.tests.test_resident_server
|
||||
cloud.tests.test_resident_website_loop` —— 15 项全部通过。
|
||||
其中上一轮 13 项在新 worktree 动工前已先复核全过。
|
||||
- 全量(CI 同款三条 discover,定向之后、全量之前无生产改动,
|
||||
全量后又重跑定向 15 项全过,时间关系如实写明):
|
||||
- `discover -s tests`:105 项,0 失败,0 跳过。
|
||||
- `discover -s cloud/tests`:1546 项,0 失败,9 跳过
|
||||
(上一轮 1544 + 本轮新增 2,对账无差)。
|
||||
- `discover -s deploy/tests`:44 项,0 失败。
|
||||
- 合计 1695 项、0 失败、9 跳过。失败未靠删断言、加 skip、
|
||||
改默认值蒙混;生产间隔下限 60 秒、抓取超时默认 10 秒、
|
||||
URL 校验口径一律未动,到期只用驱动 now 注入模拟时间流逝。
|
||||
|
||||
## 已知偏差与未做事项(未写成已完成)
|
||||
|
||||
- 控制台 UI 不在本轮闭环内:只能走 API,无常驻面板。
|
||||
- 真实外网抓取未验:只验了本地 HTTP 服务 + 测试 DNS 映射,
|
||||
不要把本轮说成真实外网验证;生产环境下的代理/DNS/证书
|
||||
行为仍属后续验证项。
|
||||
- 生产 PG 表迁移未验(只验 SQLite 临时库)、双进程抢单未验
|
||||
(只验条件 UPDATE 单胜者的上一轮单测与本轮单驱动链路)。
|
||||
- 其它探针本轮明确不做:server_status、agent_status、
|
||||
relay_cost、通用任务改名一律不在本轮范围内。
|
||||
- 变化判定仍是上一轮 v1 全文 hash,无选择器/区域提取;
|
||||
SSRF 仍是 v1 只挡字面私有 IP,未做 DNS 解析后校验。
|
||||
- 未做:未 push、未合并、未部署;架构与代码质量的 10 分
|
||||
审核是父级后续独立步骤,本轮不自评、不给自己打 10 分。
|
||||
|
||||
## Blocker
|
||||
|
||||
- 本轮无 blocker(纯本地代码与本地测试,未触及只有用户能给
|
||||
的账号/密钥/商户/商店账号/宿主/维护窗口,也未发生
|
||||
花真钱、删生产数据、改生产可见性的动作)。
|
||||
Reference in new issue
Block a user