Compare commits

...
10 Commits
Author SHA1 Message Date
admin 791773b2d1 Add Gitea CI (host runner, manual clone, apk python)
ci / test (push) Failing after 15m5s
2026-11-04 15:17:12 +00:00
openqj 82d0e84371 Merge pull request #277 from openqj/feat-resident-p0-fixes
resident P0+P1: DNS SSRF pin, batch budget, caps/cooldown/retention, PG migration, deadlock isolation, retarget reset, proxy, metrics
2026-11-02 11:40:24 +08:00
host b391fe9320 resident P0+P1: DNS-level SSRF pin, batch budget, daily cap/cooldown/retention, PG schema/migration, dual-connection deadlock isolation, retarget baseline reset, proxy and structured failures, metrics
- DNS 级 SSRF pin:解析后 pin IP 建连,防 DNS 重绑定
- 批量总预算与每日上限/抖动冷却/保留期
- PG schema 与迁移清单
- 双连接死锁隔离
- 改目标重置基线
- 显式代理配置与结构化失败原因
- /metrics resident 口径
2026-11-02 03:30:57 +00:00
openqj 60997b138f Merge pull request #275 from openqj/feat-resident-website-loop-audit
合并网站常驻监听 PR #275:resident-watches + pause/resume/end、闭环实测、审核架构 10/10 与代码质量 10/10。CI host-verify 通过(6m52s),mergeable CLEAN。
2026-11-02 04:18:29 +08:00
host 6525bcafd0 audit: resident website loop architecture and code quality to 10/10
Fix F1 name race with partial unique index, F2 pause/end mid-poll race with guarded updates, F3 redirect/numeric-IP SSRF, F4 quota prefix check, F5 duplicate row_get, F6 audit ternary; S1 unused import, S2 backoff doc, S3 defensive tearDown. Directed 20 pass, full 1700 pass.
2026-11-01 19:58:03 +00:00
host 23725f92e6 docs: resident website loop verification summary 2026-11-01 19:43:04 +00:00
host 561b3891fb test: resident website loop end-to-end with real fetch 2026-11-01 19:42:40 +00:00
host 0722ca3fd8 feat: resident watch tasks with pause/resume/end
Long-running website monitoring: new cloud/resident.py layer on top of
the automations claim/quota model (no task state-machine change).
States running/paused/ended; pause preserves the content-hash cursor
and stops polling, resume re-checks from the cursor, end is terminal
(resume after end is 409, a new watch is required). Polling is a cheap
fetch + hash comparison; only a real change creates a queued model
task. Quota exhaustion pauses (quota_exceeded) instead of ending and
keeps the cursor. Driver is the worker run_loop round boundary;
server exposes create/list/get/update/pause/resume/end with audit.
Console UI intentionally not wired this round (see verification
summary for deviations).
2026-11-01 19:01:39 +00:00
openqj 622c51e7fb Merge pull request #274 from openqj/feat-searxng-freeze
websearch: 默认 auto 付费冻结,SearXNG 优先 + 显式解冻开关
2026-11-01 22:29:59 +08:00
host 3a56f83114 websearch: 默认 auto 付费冻结,SearXNG 优先 + 显式解冻开关
- 默认 auto 只走免费链 searxng -> duckduckgo;即使环境里有
  付费 key 也不调用 Perplexity/Exa/Tavily/Brave,SearXNG
  失败也不回落付费,默认使用零付费花费。
- 付费后端代码/测试/文档全部保留;显式 pin(tavily/exa/
  perplexity/brave)仍照常调用,是有意覆盖不是默认。
- 解冻开关 WEBSEARCH_ENABLE_PAID,仅精确值 "1" 才恢复旧链
  (perplexity/exa/tavily/brave/searxng/duckduckgo),默认关;
  开关经 worker search_env 透传进 MCP config env。
- HttpClient 超时/DNS 失败改报显式错误(timeout after Ns /
  URL error),绝不把超时写成“无结果”。
- SearXNG 默认参数核实:代码无 SEARXNG_URL 硬编码默认值;
  SEARXNG_ENGINES 默认 bing,360search 已在基线;WEBSEARCH_TIMEOUT
  默认 20 秒。主机 env 实测仅 SEARXNG_URL=http://searx.diante.me
  一行,故默认 pin 生效;文档 https 漂移改回 http(主机 nginx
  仅听 80、无 443 vhost,https 在 TLS 层即失败)。
- 旧测试更新原因:test_websearch_mcp 原断言默认 auto 为
  perplexity 优先,与新冻结语义直接冲突,故改为断言冻结后
  的 searxng -> duckduckgo 回退 + 不泄漏 key;旧 perplexity ->
  exa 回退断言移至解冻开关测试(WEBSEARCH_ENABLE_PAID=1 下)。
  未删断言、未加 skip、未放宽门禁。
- 文档另删 WEBSEARCH_PROVIDERS 说法:代码从未实现该变量。
- 未推、未建 PR、未合并、未部署、未改主机 env/nginx/filter/
  证书、未做付费后端 live 验证、未创建生产任务。
2026-11-01 14:15:54 +00:00
20 changed files with 4854 additions and 43 deletions

No files matched your search

+19
View File
@@ -0,0 +1,19 @@
name: ci
on: [push]
jobs:
test:
runs-on: host
steps:
- name: install python
run: apk add --no-cache python3 py3-pip ca-certificates git 2>&1 | tail -3; python3 --version; git --version
- name: clone repo
run: |
rm -rf src
git clone https://gitea.tose.top/admin/codex-app-server.git src
cd src && git rev-parse HEAD
- name: host tests
run: cd src && python3 -m unittest discover -s tests -v
- name: cloud tests
run: cd src && python3 -m unittest discover -s cloud/tests -v
- name: deploy tests
run: cd src && python3 -m unittest discover -s deploy/tests -v
+43
View File
@@ -1044,6 +1044,49 @@ POSTGRES_SCHEMA = (
"status TEXT, "
"requested_at DOUBLE PRECISION, "
"done_at DOUBLE PRECISION)",
# 常驻监听(P0 S3:此前只由 resident.ensure_resident_tables
# 运行时懒建,POSTGRES_SCHEMA 与迁移清单里没有;本批正式
# 写进 schema,与 ensure 的 CREATE 语句列一一对应,含
# P1 新增的 resident_polls.failure_reason 列;TEXT 主键/
# 复合部分唯一索引,无身份列,不需要序列拨动)。
"CREATE TABLE IF NOT EXISTS resident_watches("
"id TEXT PRIMARY KEY, "
"project_id TEXT, "
"name TEXT, "
"target_url TEXT, "
"prompt TEXT, "
"interval_seconds BIGINT, "
"fetch_timeout_seconds BIGINT, "
"status TEXT, "
"last_poll_at DOUBLE PRECISION, "
"last_claimed_at DOUBLE PRECISION, "
"next_poll_at DOUBLE PRECISION, "
"last_content_hash TEXT, "
"last_change_at DOUBLE PRECISION, "
"last_change_text TEXT, "
"consecutive_failures BIGINT, "
"paused_reason TEXT, "
"ended_at DOUBLE PRECISION, "
"created_by TEXT, "
"created_at DOUBLE PRECISION, "
"updated_at DOUBLE PRECISION)",
"CREATE INDEX IF NOT EXISTS idx_resident_watches_project"
" ON resident_watches(project_id, status)",
"CREATE UNIQUE INDEX IF NOT EXISTS idx_resident_watches_name"
" ON resident_watches(project_id, name)"
" WHERE status != 'ended'",
"CREATE TABLE IF NOT EXISTS resident_polls("
"id TEXT PRIMARY KEY, "
"watch_id TEXT, "
"project_id TEXT, "
"status TEXT, "
"observed_hash TEXT, "
"detail TEXT, "
"task_id TEXT, "
"created_at DOUBLE PRECISION, "
"failure_reason TEXT)",
"CREATE INDEX IF NOT EXISTS idx_resident_polls_watch"
" ON resident_polls(watch_id, created_at)",
)
+5
View File
@@ -82,6 +82,11 @@ MIGRATION_TABLES = (
# 老库没有时迁移记 0,不中断其余表)。
"task_checkpoints",
"task_steps",
# 常驻监听(P0 S3:与 POSTGRES_SCHEMA 同款两表,TEXT 主键,
# 无身份列;sqlite 侧由 resident.ensure 建出,老库没有时
# 记 0,不中断其余表;failure_reason 列两端 schema 一致)。
"resident_watches",
"resident_polls",
)
_SKIP_TABLES = frozenset({"kb_fts", "sqlite_sequence"})
+1585
View File
File diff suppressed because it is too large. Load diff
+212
View File
@@ -110,6 +110,16 @@ from cloud.wait_ops import worktree_status as _wait_worktree_status
from cloud.shared_home_mcp import sync_shared_home_mcp as _sync_shared_home_mcp
from cloud.automations import ensure_automations_tables as _auto_ensure
from cloud.automations import list_automations as _auto_list
from cloud.resident import create_watch as _res_create
from cloud.resident import end_watch as _res_end
from cloud.resident import ensure_resident_tables as _res_ensure
from cloud.resident import get_watch as _res_get
from cloud.resident import list_polls as _res_polls
from cloud.resident import list_watches as _res_list
from cloud.resident import pause_watch as _res_pause
from cloud.resident import resident_metrics as _res_metrics
from cloud.resident import resume_watch as _res_resume
from cloud.resident import update_watch as _res_update
from cloud.todos import create_todo as _todo_create
from cloud.todos import delete_todo as _todo_delete
from cloud.todos import get_todo as _todo_get
@@ -1731,6 +1741,10 @@ def init_db(db_path):
_ensure_task_checkpoints_table(conn)
_ensure_task_steps_table(conn)
ensure_kill_requests_table(conn)
# 常驻监听表(P0 S3 的 SQLite 侧):启动时建一次,
# 热路径(claim/run_due/批量)不再逐条 ensure,
# 详见 cloud/resident.py ensure 的调用约定。
_res_ensure(conn)
# worker 每轮 poll 都按 status 取任务(见 load_tasks 的
# WHERE status IN (...)),无索引时是全表扫描;任务表随
# 时间变大后索引是必须的。IF NOT EXISTS 保证老库幂等。
@@ -3371,6 +3385,54 @@ class CloudControlPlaneHandler(BaseHTTPRequestHandler):
return
self._triage_inbox(project_id)
return
# /projects/{id}/resident-watches:常驻监听任务。
# 创建/更新/暂停/恢复/结束用 create_task,读用 read;
# 状态机见 cloud/resident.py(running/paused/ended,
# ended 是终态,恢复已结束的监听必须新建)。
if len(parts) == 2 and parts[1] == "resident-watches":
if method == "POST":
err = self._check_project(user_id, project_id, "create_task")
if err:
self._send_json(err[0], {"error": err[1]})
return
self._resident_create(user_id, project_id)
elif method == "GET":
err = self._check_project(user_id, project_id, "read")
if err:
self._send_json(err[0], {"error": err[1]})
return
self._resident_list(project_id)
else:
self._send_json(404, {"error": "not found"})
return
if len(parts) == 3 and parts[1] == "resident-watches":
if method == "GET":
err = self._check_project(user_id, project_id, "read")
if err:
self._send_json(err[0], {"error": err[1]})
return
self._resident_get(user_id, project_id, parts[2])
elif method == "PATCH":
err = self._check_project(user_id, project_id, "create_task")
if err:
self._send_json(err[0], {"error": err[1]})
return
self._resident_update(user_id, project_id, parts[2])
else:
self._send_json(404, {"error": "not found"})
return
if len(parts) == 4 and parts[1] == "resident-watches" \
and parts[3] in ("pause", "resume", "end"):
if method != "POST":
self._send_json(404, {"error": "not found"})
return
err = self._check_project(user_id, project_id, "create_task")
if err:
self._send_json(err[0], {"error": err[1]})
return
self._resident_transition(user_id, project_id, parts[2],
parts[3])
return
# /projects/{id}/todos GET 列表 / POST 新建;单条
# GET / PATCH / DELETE(项目 read / create_task)
if len(parts) == 2 and parts[1] == "todos":
@@ -8845,6 +8907,13 @@ class CloudControlPlaneHandler(BaseHTTPRequestHandler):
else:
unpriced_tokens += row["total_tokens"] or 0
all_tokens = actual_tokens + estimated_tokens
# 常驻监听口径(P1 G5):只统计调用者是成员的项目,
# 与任务计数同款成员过滤;表缺失时 resident_metrics
# 回 available=False,不炸 /metrics。
try:
resident_stats = _res_metrics(conn, member_ids, now)
except Exception: # noqa: BLE001 - 观测端点不因 resident 失败
resident_stats = {"available": False}
finally:
conn.close()
try:
@@ -8872,6 +8941,7 @@ class CloudControlPlaneHandler(BaseHTTPRequestHandler):
"by_model": sorted(by_model.values(),
key=lambda e: -e["total_tokens"]),
},
"resident": resident_stats,
"queue_depth": {
"queued": by_status.get("queued", 0),
"running": by_status.get("running", 0),
@@ -10067,6 +10137,148 @@ class CloudControlPlaneHandler(BaseHTTPRequestHandler):
automation_id)
self._send_json(200, auto)
# -- 常驻监听任务(见 cloud/resident.py)-------------------
# 权限口径与自动化同款:创建/更新/暂停/恢复/结束用
# create_task,读用 read;状态变更记审计。创建不走配额门
# (轮询不花模型额度),配额只在变化建任务时由驱动过,
# 耗尽时转暂停而非拒绝创建。
def _resident_create(self, user_id, project_id):
body, err = self._read_json()
if err:
self._send_json(400, {"error": err})
return
interval = body.get("interval_seconds", 3600)
if interval is None:
interval = 3600
conn = self._db()
try:
try:
watch = _res_create(
conn, project_id, body.get("name"),
body.get("target_url"), body.get("prompt"), user_id,
interval, body.get("fetch_timeout_seconds"))
except ValueError as exc:
self._send_json(400, {"error": str(exc)})
return
except Exception as exc:
# P2:502 不再透传异常文本(可能含表名/列名等
# DB 细节),统一固定错误码,详情只记服务端日志。
_server_log.warning("resident create failed: %s", exc)
self._send_json(502, {"error": "resident_operation_failed"})
return
finally:
conn.close()
with _auth_store(self.server.db_path) as store:
store.audit(user_id, project_id, "resident_watch_created",
watch["id"])
self._send_json(201, watch)
def _resident_list(self, project_id):
conn = self._db()
try:
try:
watches = _res_list(conn, project_id)
except Exception as exc:
_server_log.warning("resident list failed: %s", exc)
self._send_json(502, {"error": "resident_operation_failed"})
return
conn.commit()
finally:
conn.close()
self._send_json(200, {
"project_id": project_id, "resident_watches": watches,
"due_count": sum(1 for w in watches if w["due"])})
def _resident_get(self, user_id, project_id, watch_id):
conn = self._db()
try:
watch = _res_get(conn, watch_id)
polls = []
polls_note = ""
if watch is not None:
try:
polls = _res_polls(conn, watch_id)
except Exception as exc:
# P2:polls 异常不再静默吞成空串,至少记
# 服务端日志;响应仍保持 polls=[] + note
# 的兼容形态(排障信号在日志里,不在响应里丢)。
_server_log.warning(
"resident polls unavailable for %s: %s",
watch_id, exc)
polls_note = "polls unavailable"
finally:
conn.close()
if watch is None or watch["project_id"] != project_id:
self._send_json(404, {"error": "resident watch not found"})
return
self._send_json(200, dict(watch, polls=polls,
polls_note=polls_note))
def _resident_update(self, user_id, project_id, watch_id):
body, jerr = self._read_json()
if jerr:
self._send_json(400, {"error": jerr})
return
conn = self._db()
try:
try:
ok, watch = _res_update(conn, watch_id, body)
except ValueError as exc:
self._send_json(400, {"error": str(exc)})
return
except Exception as exc:
_server_log.warning("resident update failed: %s", exc)
self._send_json(502, {"error": "resident_operation_failed"})
return
finally:
conn.close()
if watch is None or watch["project_id"] != project_id:
self._send_json(404, {"error": "resident watch not found"})
return
if not ok:
self._send_json(409, {
"error": "ended resident watch cannot be updated;"
" create a new watch instead"})
return
with _auth_store(self.server.db_path) as store:
store.audit(user_id, project_id, "resident_watch_updated",
watch_id)
self._send_json(200, watch)
def _resident_transition(self, user_id, project_id, watch_id, action):
fn = {"pause": _res_pause, "resume": _res_resume,
"end": _res_end}[action]
conn = self._db()
try:
try:
ok, watch = fn(conn, watch_id)
except Exception as exc:
_server_log.warning(
"resident %s failed: %s", action, exc)
self._send_json(502, {"error": "resident_operation_failed"})
return
finally:
conn.close()
if watch is None or watch["project_id"] != project_id:
self._send_json(404, {"error": "resident watch not found"})
return
if not ok:
if watch["status"] == "ended":
self._send_json(409, {
"error": "ended resident watch is terminal;"
" create a new watch instead"})
else:
self._send_json(409, {
"error": "resident watch in status %s cannot %s"
% (watch["status"], action)})
return
audit_action = {"pause": "resident_watch_paused",
"resume": "resident_watch_resumed",
"end": "resident_watch_ended"}[action]
with _auth_store(self.server.db_path) as store:
store.audit(user_id, project_id, audit_action, watch_id)
self._send_json(200, watch)
def _triage_inbox(self, project_id):
"""分诊收件箱(只读):到期自动化、失败/中断任务、
待处理后续消息、最近自动化产出。见 automations 模块
+530
View File
@@ -0,0 +1,530 @@
"""常驻监听任务单测(cloud/resident.py):不跑真实模型、不联网,
抓取一律注入假 fetcher。覆盖创建校验、暂停保游标、恢复续跑、
结束终态、防重复轮询、仅变化建任务、失败退避、配额耗尽转暂停、
worker 驱动批量入口。"""
import http.server
import os
import socket
import sqlite3
import sys
import tempfile
import threading
import unittest
from pathlib import Path
from types import SimpleNamespace
try:
import cloud.resident as resident_module
from cloud.resident import (
SSRFBlockedError,
claim_due_watch,
create_watch,
default_fetch,
end_watch,
ensure_resident_tables,
get_watch,
list_polls,
pause_watch,
process_due_resident_watches,
resume_watch,
run_due_resident_watch,
update_watch,
validate_target_url,
)
from cloud.server import init_db
from cloud.worker import Worker
except ImportError: # discover -s cloud/tests 时把项目根目录补进 sys.path
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
import cloud.resident as resident_module
from cloud.resident import (
SSRFBlockedError,
claim_due_watch,
create_watch,
default_fetch,
end_watch,
ensure_resident_tables,
get_watch,
list_polls,
pause_watch,
process_due_resident_watches,
resume_watch,
run_due_resident_watch,
update_watch,
validate_target_url,
)
from cloud.server import init_db
from cloud.worker import Worker
URL = "https://example.com/data"
class _Fetcher:
"""可变假抓取:value 改了,下一次轮询就看到新内容。"""
def __init__(self, value="v1"):
self.value = value
self.calls = []
def __call__(self, url, timeout):
self.calls.append((url, timeout))
return self.value
def _failing_fetcher(url, timeout):
raise RuntimeError("boom")
class ResidentTestCase(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.db_path = os.path.join(self.tmp.name, "control.db")
init_db(self.db_path)
self.conn = sqlite3.connect(self.db_path)
self.conn.row_factory = sqlite3.Row
ensure_resident_tables(self.conn)
self.now = 1700000000.0
def tearDown(self):
self.conn.close()
self.tmp.cleanup()
def _watch(self, name="watch1", **kw):
kw.setdefault("interval_seconds", 60)
return create_watch(self.conn, "proj", name, URL,
"变化时汇报", "owner1", now=self.now, **kw)
def _tasks(self):
cur = self.conn.execute("SELECT id, prompt, status FROM tasks")
return cur.fetchall()
def _set_quota(self, user_id, micros):
self.conn.execute(
"CREATE TABLE IF NOT EXISTS user_quotas("
"user_id TEXT PRIMARY KEY, quota_usd_micros BIGINT, "
" updated_at DOUBLE PRECISION)")
self.conn.execute(
"INSERT OR REPLACE INTO user_quotas(user_id,"
" quota_usd_micros, updated_at) VALUES(?, ?, ?)",
(user_id, micros, self.now))
self.conn.commit()
def test_create_validation(self):
watch = self._watch()
self.assertEqual(watch["status"], "running")
self.assertTrue(watch["due"])
self.assertEqual(watch["next_poll_at"], self.now)
for kwargs, _ in (
({"target_url": "ftp://example.com/x"}, None),
({"target_url": "https://localhost/x"}, None),
({"target_url": "http://127.0.0.1/x"}, None),
({"target_url": "https://10.0.0.5/x"}, None),
({"target_url": "https://user:pw@example.com/x"}, None),
({"interval_seconds": 30}, None),
({"name": "watch1"}, None),
({"prompt": " "}, None)):
with self.assertRaises(ValueError):
base = {"name": "other", "target_url": URL,
"prompt": "p", "interval_seconds": 60}
base.update(kwargs)
create_watch(self.conn, "proj", base["name"],
base["target_url"], base["prompt"], "owner1",
interval_seconds=base["interval_seconds"],
now=self.now)
def test_baseline_unchanged_changed_only_change_creates_task(self):
fetcher = _Fetcher("v1")
watch = self._watch()
poll = run_due_resident_watch(self.conn, watch["id"],
fetcher=fetcher, now=self.now)
self.assertEqual(poll["status"], "baseline")
self.assertEqual(len(self._tasks()), 0)
# 同一间隔内第二次:抢不到,不重复轮询。
self.assertIsNone(run_due_resident_watch(
self.conn, watch["id"], fetcher=fetcher, now=self.now))
self.assertEqual(len(fetcher.calls), 1)
self.now += 61
poll = run_due_resident_watch(self.conn, watch["id"],
fetcher=fetcher, now=self.now)
self.assertEqual(poll["status"], "unchanged")
self.assertEqual(len(self._tasks()), 0)
fetcher.value = "v2"
self.now += 61
poll = run_due_resident_watch(self.conn, watch["id"],
fetcher=fetcher, now=self.now)
self.assertEqual(poll["status"], "changed")
self.assertIsNotNone(poll["task_id"])
tasks = self._tasks()
self.assertEqual(len(tasks), 1)
self.assertIn("v2", tasks[0][1])
self.assertIn("变化时汇报", tasks[0][1])
after = get_watch(self.conn, watch["id"], now=self.now)
self.assertEqual(after["last_change_text"], "v2")
self.assertIsNotNone(after["last_change_at"])
self.assertEqual(
[p["status"] for p in list_polls(self.conn, watch["id"])],
["changed", "unchanged", "baseline"])
def test_pause_preserves_cursor_and_resume_reports_pause_period_change(self):
fetcher = _Fetcher("v1")
watch = self._watch()
run_due_resident_watch(self.conn, watch["id"],
fetcher=fetcher, now=self.now)
cursor = get_watch(self.conn, watch["id"], now=self.now)
self.assertIsNotNone(cursor["last_content_hash"])
ok, paused = pause_watch(self.conn, watch["id"], now=self.now)
self.assertTrue(ok)
self.assertEqual(paused["status"], "paused")
self.assertEqual(paused["last_content_hash"],
cursor["last_content_hash"])
self.assertIsNone(paused["next_poll_at"])
self.assertFalse(paused["due"])
# 暂停期间即使时间过了间隔,驱动也不轮询。
self.assertIsNone(run_due_resident_watch(
self.conn, watch["id"], fetcher=fetcher, now=self.now + 3600))
self.assertEqual(len(fetcher.calls), 1)
# 暂停期间网站变了;恢复后第一次轮询用旧游标比对,报出来。
fetcher.value = "v2"
ok, resumed = resume_watch(self.conn, watch["id"], now=self.now)
self.assertTrue(ok)
self.assertEqual(resumed["status"], "running")
self.assertEqual(resumed["last_content_hash"],
cursor["last_content_hash"])
poll = run_due_resident_watch(self.conn, watch["id"],
fetcher=fetcher, now=self.now)
self.assertEqual(poll["status"], "changed")
self.assertEqual(len(self._tasks()), 1)
def test_pause_resume_on_wrong_status_are_rejected(self):
watch = self._watch()
ok, _ = resume_watch(self.conn, watch["id"], now=self.now)
self.assertFalse(ok)
ok, _ = pause_watch(self.conn, watch["id"], now=self.now)
self.assertTrue(ok)
ok, _ = pause_watch(self.conn, watch["id"], now=self.now)
self.assertFalse(ok)
def test_end_is_terminal_and_name_reusable(self):
watch = self._watch()
ok, ended = end_watch(self.conn, watch["id"], now=self.now)
self.assertTrue(ok)
self.assertEqual(ended["status"], "ended")
self.assertIsNone(ended["next_poll_at"])
for fn in (pause_watch, resume_watch, end_watch):
ok, old = fn(self.conn, watch["id"], now=self.now)
self.assertFalse(ok)
self.assertEqual(old["status"], "ended")
self.assertIsNone(run_due_resident_watch(
self.conn, watch["id"], fetcher=_Fetcher(), now=self.now + 3600))
ok, _ = update_watch(self.conn, watch["id"],
{"prompt": "新提示"}, now=self.now)
self.assertFalse(ok)
# 已结束的名字可重用。
again = self._watch()
self.assertEqual(again["status"], "running")
def test_claim_is_single_winner(self):
watch = self._watch()
self.assertTrue(claim_due_watch(self.conn, watch["id"], now=self.now))
self.assertFalse(claim_due_watch(self.conn, watch["id"],
now=self.now))
# 下个间隔到了才能再抢(claim 已把 next_poll_at 预置到下间隔)。
self.assertTrue(claim_due_watch(self.conn, watch["id"],
now=self.now + 61))
def test_fetch_failure_backoff_and_cursor_unchanged(self):
watch = self._watch()
run_due_resident_watch(self.conn, watch["id"],
fetcher=_Fetcher("v1"), now=self.now)
cursor = get_watch(self.conn, watch["id"], now=self.now)
self.now += 61
poll = run_due_resident_watch(self.conn, watch["id"],
fetcher=_failing_fetcher, now=self.now)
self.assertEqual(poll["status"], "failed")
after = get_watch(self.conn, watch["id"], now=self.now)
self.assertEqual(after["consecutive_failures"], 1)
self.assertEqual(after["last_content_hash"],
cursor["last_content_hash"])
# 失败退避:下次 = now + 间隔 + 30 秒,不是立刻重试。
self.assertEqual(after["next_poll_at"], self.now + 60 + 30)
self.now = after["next_poll_at"]
run_due_resident_watch(self.conn, watch["id"],
fetcher=_failing_fetcher, now=self.now)
after2 = get_watch(self.conn, watch["id"], now=self.now)
self.assertEqual(after2["consecutive_failures"], 2)
self.assertEqual(after2["next_poll_at"], self.now + 60 + 60)
def test_quota_exceeded_pauses_not_ends_and_keeps_cursor(self):
fetcher = _Fetcher("v1")
watch = self._watch()
run_due_resident_watch(self.conn, watch["id"],
fetcher=fetcher, now=self.now)
cursor_hash = get_watch(self.conn, watch["id"],
now=self.now)["last_content_hash"]
self._set_quota("owner1", 0)
fetcher.value = "v2"
self.now += 61
poll = run_due_resident_watch(self.conn, watch["id"],
fetcher=fetcher, now=self.now)
self.assertEqual(poll["status"], "changed_quota_paused")
after = get_watch(self.conn, watch["id"], now=self.now)
self.assertEqual(after["status"], "paused")
self.assertEqual(after["paused_reason"], "quota_exceeded")
self.assertNotEqual(after["status"], "ended")
self.assertEqual(after["last_content_hash"], cursor_hash)
self.assertEqual(len(self._tasks()), 0)
# 恢复后配额仍耗尽:再次转暂停,不擅自结束、不丢变化。
resume_watch(self.conn, watch["id"], now=self.now)
poll = run_due_resident_watch(self.conn, watch["id"],
fetcher=fetcher, now=self.now)
self.assertEqual(poll["status"], "changed_quota_paused")
# 配额恢复后,同一变化仍能报出来(游标没被配额暂停推进)。
self._set_quota("owner1", 100000000)
resume_watch(self.conn, watch["id"], now=self.now)
poll = run_due_resident_watch(self.conn, watch["id"],
fetcher=fetcher, now=self.now)
self.assertEqual(poll["status"], "changed")
self.assertEqual(len(self._tasks()), 1)
def test_update_watch(self):
watch = self._watch()
ok, updated = update_watch(
self.conn, watch["id"],
{"interval_seconds": 3600, "prompt": "新处理提示"},
now=self.now)
self.assertTrue(ok)
self.assertEqual(updated["interval_seconds"], 3600)
self.assertEqual(updated["prompt"], "新处理提示")
with self.assertRaises(ValueError):
update_watch(self.conn, watch["id"], {}, now=self.now)
with self.assertRaises(ValueError):
update_watch(self.conn, watch["id"],
{"target_url": "https://localhost/x"}, now=self.now)
def test_process_due_and_worker_driver(self):
fetcher = _Fetcher("v1")
self._watch(name="a")
self._watch(name="b")
polls = process_due_resident_watches(self.conn, fetcher=fetcher,
now=self.now)
self.assertEqual(len(polls), 2)
self.assertEqual([p["status"] for p in polls],
["baseline", "baseline"])
self.assertEqual(process_due_resident_watches(
self.conn, fetcher=fetcher, now=self.now), [])
# worker 驱动入口:注入假抓取,变化时产出 changed 轮询。
worker = Worker(self.db_path, "/fake/host.py",
env={"XCM_API_KEY": "test-placeholder"},
runner=lambda cmd: SimpleNamespace(
returncode=0, stdout="{}", stderr=""))
fetcher.value = "v2"
self.now += 61
polls = worker.process_due_resident_watches(
now=self.now, fetcher=fetcher)
self.assertEqual(len(polls), 2)
self.assertTrue(all(p["status"] == "changed" for p in polls))
self.assertEqual(len(self._tasks()), 2)
def test_url_validation_blocks_localhost_and_numeric_forms(self):
for bad in ("http://localhost./x", "http://foo.localhost/x",
"http://2130706433/x", "http://0x7f000001/x",
"http://0177.0.0.1/x", "http://0x7f.0.0.1/x"):
with self.assertRaises(ValueError, msg=bad):
validate_target_url(bad)
# 规范点分十进制公网 IP 与带尾点的普通域名仍放行。
self.assertEqual(validate_target_url("https://8.8.8.8/x"),
"https://8.8.8.8/x")
self.assertEqual(validate_target_url("https://example.com./x"),
"https://example.com./x")
def test_unique_index_guards_concurrent_name_clash(self):
idx = self.conn.execute(
"SELECT name FROM sqlite_master WHERE type = 'index'"
" AND name = 'idx_resident_watches_name'").fetchone()
self.assertIsNotNone(idx)
self._watch()
# 正常路径仍先报 ValueError。
with self.assertRaises(ValueError):
self._watch()
# 绕过 SELECT 查重直接插:唯一索引本身必须拦住,
# 这才是并发下真正的关口。
with self.assertRaises(sqlite3.IntegrityError):
self.conn.execute(
"INSERT INTO resident_watches(id, project_id, name,"
" target_url, prompt, interval_seconds, status,"
" created_by, created_at, updated_at)"
" VALUES('dup', 'proj', 'watch1', ?, 'p', 60,"
" 'running', 'owner1', ?, ?)",
(URL, self.now, self.now))
other = self._watch(name="watch2")
with self.assertRaises(ValueError):
update_watch(self.conn, other["id"], {"name": "watch1"},
now=self.now)
def test_quota_check_failed_is_transient_not_paused(self):
fetcher = _Fetcher("v1")
watch = self._watch()
run_due_resident_watch(self.conn, watch["id"],
fetcher=fetcher, now=self.now)
cursor_hash = get_watch(self.conn, watch["id"],
now=self.now)["last_content_hash"]
original = resident_module._automation_quota_exceeded
resident_module._automation_quota_exceeded = lambda conn, owner: (
"quota_check_failed: boom")
try:
fetcher.value = "v2"
self.now += 61
poll = run_due_resident_watch(
self.conn, watch["id"], fetcher=fetcher, now=self.now)
finally:
resident_module._automation_quota_exceeded = original
self.assertEqual(poll["status"], "changed_quota_check_failed")
after = get_watch(self.conn, watch["id"], now=self.now)
self.assertEqual(after["status"], "running")
self.assertEqual(after["last_content_hash"], cursor_hash)
self.assertEqual(after["consecutive_failures"], 1)
self.assertEqual(after["next_poll_at"], self.now + 60 + 30)
self.assertEqual(len(self._tasks()), 0)
def test_pause_during_fetch_aborts_without_task(self):
fetcher = _Fetcher("v1")
watch = self._watch()
run_due_resident_watch(self.conn, watch["id"],
fetcher=fetcher, now=self.now)
cursor_hash = get_watch(self.conn, watch["id"],
now=self.now)["last_content_hash"]
db_path = self.db_path
def pausing_fetcher(url, timeout):
conn2 = sqlite3.connect(db_path)
try:
conn2.row_factory = sqlite3.Row
pause_watch(conn2, watch["id"], now=self.now + 61)
finally:
conn2.close()
return "v2"
self.now += 61
poll = run_due_resident_watch(
self.conn, watch["id"], fetcher=pausing_fetcher, now=self.now)
self.assertIsNone(poll)
after = get_watch(self.conn, watch["id"], now=self.now)
self.assertEqual(after["status"], "paused")
self.assertIsNone(after["next_poll_at"])
self.assertEqual(after["last_content_hash"], cursor_hash)
self.assertEqual(len(self._tasks()), 0)
self.assertEqual(
[p["status"] for p in list_polls(self.conn, watch["id"])],
["baseline"])
class RedirectFetchTestCase(unittest.TestCase):
"""重定向与 DNS 门校验:默认解析器下 redirect-test.test
经 getaddrinfo 补丁指到本地(127.0.0.1),DNS 门必须拦
住它(私网解析结果);相对跳转的跟随逻辑走注入的测试
resolver(返回公网测试 IP,门正常放行)+ socket 层重定向
(测试进程内把该公网 IP:端口指回本地服务——门与 pin
逻辑全程真实跑,只有传输层终点是测试替身);no_proxy
同款处理,tearDown 全部还原。"""
HOST = "redirect-test.test"
def setUp(self):
class Handler(http.server.BaseHTTPRequestHandler):
def do_GET(self):
if self.path == "/go":
self.send_response(302)
self.send_header("Location", "/final")
elif self.path == "/bad":
self.send_response(302)
self.send_header(
"Location", "http://127.0.0.1:1/secret")
else:
body = b"final-body"
self.send_response(200)
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
return
self.end_headers()
def log_message(self, *args):
pass
self.server = http.server.ThreadingHTTPServer(
("127.0.0.1", 0), Handler)
self.port = self.server.server_address[1]
self.thread = threading.Thread(
target=self.server.serve_forever, daemon=True)
self.thread.start()
self._orig_getaddrinfo = socket.getaddrinfo
def patched(host, port, family=0, type=0, proto=0, flags=0):
if host == self.HOST:
return [(socket.AF_INET, socket.SOCK_STREAM, 6, "",
("127.0.0.1", port))]
return self._orig_getaddrinfo(
host, port, family, type, proto, flags)
socket.getaddrinfo = patched
# socket 层测试替身:只重定向公网测试 IP 到本地服务端口,
# 其它地址一律走原始 create_connection。
self._orig_create_connection = socket.create_connection
test_port = self.port
def patched_create_connection(address, *args, **kwargs):
if address[0] == "93.184.216.34" \
and address[1] == test_port:
address = ("127.0.0.1", test_port)
return self._orig_create_connection(address, *args, **kwargs)
socket.create_connection = patched_create_connection
self._old_no_proxy = os.environ.get("no_proxy")
self._old_NO_PROXY = os.environ.get("NO_PROXY")
os.environ["no_proxy"] = (
(self._old_no_proxy + ",") if self._old_no_proxy
else "") + self.HOST
os.environ["NO_PROXY"] = (
(self._old_NO_PROXY + ",") if self._old_NO_PROXY
else "") + self.HOST
def tearDown(self):
socket.create_connection = self._orig_create_connection
socket.getaddrinfo = self._orig_getaddrinfo
if self._old_no_proxy is None:
os.environ.pop("no_proxy", None)
else:
os.environ["no_proxy"] = self._old_no_proxy
if self._old_NO_PROXY is None:
os.environ.pop("NO_PROXY", None)
else:
os.environ["NO_PROXY"] = self._old_NO_PROXY
self.server.shutdown()
self.server.server_close()
self.thread.join(timeout=5)
def test_relative_redirect_followed_and_private_blocked(self):
base = "http://%s:%d" % (self.HOST, self.port)
# 注入 resolver 的跟随路径:解析门放行公网测试 IP,
# socket 替身把建连指回本地,相对跳转 /go -> /final 走通。
def test_resolver(host, port):
return ["93.184.216.34"]
self.assertEqual(
default_fetch(base + "/go", 10, resolver=test_resolver),
"final-body")
# 默认 DNS 门:测试主机名解析到 127.0.0.1,直接拦住。
with self.assertRaises(SSRFBlockedError):
default_fetch(base + "/go", 10)
# 重定向跳本身:跳转目标是字面私网 IP,逐跳校验拦住。
with self.assertRaises(ValueError):
default_fetch(base + "/bad", 10, resolver=test_resolver)
# 混合解析结果:任一地址私网即整批拒绝,不取公网那个蒙混。
def mixed(host, port):
return ["93.184.216.34", "127.0.0.1"]
with self.assertRaises(ValueError):
default_fetch(base + "/go", 10, resolver=mixed)
if __name__ == "__main__":
unittest.main()
+684
View File
@@ -0,0 +1,684 @@
"""常驻监听 P0+P1 修复批单测(cloud/resident.py 等)。
覆盖审核报告的 P0(S1 DNS 级 SSRF + 摘录口径、S2 批量总预算、
G2 每日上限/抖动冷却/保留期)与 P1(S3 PG schema 与迁移清单、
G1 ensure 移出热路径 + 逐条隔离 + 死锁重试、G3 改目标重置基线、
G4 显式代理配置 + 结构化失败原因、G5 /metrics 口径)及 P2
建议项(502 固定错误码、_resident_get polls 异常记日志不静默)。
不跑真实模型、不连真实外网:真实抓取只对本地测试 HTTP
服务,DNS 门用 getaddrinfo 补丁或注入 resolver 验证;本地
PG 验证不在本文件(见 verification/resident-p0-fixes-summary.md
里的 /tmp 一次性库记录),本文件只做 SQLite 与覆盖断言。
"""
import http.client
import http.server
import json
import os
import socket
import sqlite3
import ssl
import sys
import tempfile
import threading
import unittest
from pathlib import Path
from types import SimpleNamespace
from unittest import mock
try:
import cloud.resident as resident_module
from cloud.auth import AuthStore
from cloud.db import PostgresOperationalError
from cloud.migrations import MIGRATION_TABLES, migration_coverage
from cloud.resident import (
FetchError,
HttpStatusFetchError,
SSRFBlockedError,
_PinnedHTTPSConnection,
classify_fetch_error,
create_watch,
default_fetch,
ensure_resident_tables,
get_watch,
list_polls,
prune_old_polls,
process_due_resident_watches,
resident_config,
resident_metrics,
resolve_public_addresses,
run_due_resident_watch,
update_watch,
)
from cloud.server import create_server, init_db
from cloud.worker import Worker
except ImportError: # discover -s cloud/tests 时把项目根目录补进 sys.path
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
import cloud.resident as resident_module
from cloud.auth import AuthStore
from cloud.db import PostgresOperationalError
from cloud.migrations import MIGRATION_TABLES, migration_coverage
from cloud.resident import (
FetchError,
HttpStatusFetchError,
SSRFBlockedError,
_PinnedHTTPSConnection,
classify_fetch_error,
create_watch,
default_fetch,
ensure_resident_tables,
get_watch,
list_polls,
prune_old_polls,
process_due_resident_watches,
resident_config,
resident_metrics,
resolve_public_addresses,
run_due_resident_watch,
update_watch,
)
from cloud.server import create_server, init_db
from cloud.worker import Worker
URL = "https://example.com/data"
URL2 = "https://example.com/other"
class _Fetcher:
def __init__(self, value="v1"):
self.value = value
self.calls = []
def __call__(self, url, timeout):
self.calls.append((url, timeout))
return self.value
class ResidentP0TestCase(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.db_path = os.path.join(self.tmp.name, "control.db")
init_db(self.db_path)
self.conn = sqlite3.connect(self.db_path)
self.conn.row_factory = sqlite3.Row
ensure_resident_tables(self.conn)
self.now = 1700000000.0
def tearDown(self):
self.conn.close()
self.tmp.cleanup()
def _watch(self, name="watch1", url=URL, **kw):
kw.setdefault("interval_seconds", 60)
return create_watch(self.conn, "proj", name, url,
"变化时汇报", "owner1", now=self.now, **kw)
def _tasks(self):
return self.conn.execute(
"SELECT id, prompt, status FROM tasks").fetchall()
def _polls(self, watch_id):
return list_polls(self.conn, watch_id)
# -- S1:DNS 级 SSRF -------------------------------------------
def test_resolve_public_rejects_private_and_mixed(self):
# 单一私网解析结果即拦。
with self.assertRaises(SSRFBlockedError):
resolve_public_addresses(
"evil.test", 80,
resolver=lambda h, p: ["127.0.0.1"])
with self.assertRaises(SSRFBlockedError):
resolve_public_addresses(
"evil.test", 80, resolver=lambda h, p: ["::1"])
with self.assertRaises(SSRFBlockedError):
resolve_public_addresses(
"evil.test", 80,
resolver=lambda h, p: ["169.254.169.254"])
with self.assertRaises(SSRFBlockedError):
resolve_public_addresses(
"evil.test", 80,
resolver=lambda h, p: ["::ffff:127.0.0.1"])
# 混合结果:任一私网即整批拒绝,不取公网那个蒙混。
with self.assertRaises(SSRFBlockedError):
resolve_public_addresses(
"evil.test", 80,
resolver=lambda h, p: ["93.184.216.34", "10.0.0.5"])
# 全公网才放行。
self.assertEqual(
resolve_public_addresses(
"ok.test", 80, resolver=lambda h, p: ["93.184.216.34"]),
["93.184.216.34"])
# 解析失败按 dns,不放行。
with self.assertRaises(FetchError) as ctx:
resolve_public_addresses(
"nope.test", 80,
resolver=lambda h, p: (_ for _ in ()).throw(
socket.gaierror("no such host")))
self.assertEqual(ctx.exception.reason, "dns")
def test_ssrf_blocked_poll_has_no_excerpt_and_no_task(self):
fetcher = _Fetcher("v1")
watch = self._watch()
run_due_resident_watch(self.conn, watch["id"],
fetcher=fetcher, now=self.now)
self.now += 61
def ssrf_fetcher(url, timeout):
raise SSRFBlockedError("resolved address is not public")
poll = run_due_resident_watch(self.conn, watch["id"],
fetcher=ssrf_fetcher, now=self.now)
self.assertEqual(poll["status"], "failed")
self.assertEqual(poll["failure_reason"], "ssrf_blocked")
self.assertTrue(poll["detail"].startswith("reason=ssrf_blocked"))
self.assertEqual(len(self._tasks()), 0)
after = get_watch(self.conn, watch["id"], now=self.now)
self.assertIsNone(after["last_change_text"])
# 失败轮询的 detail 里绝不出现“内网响应正文”类内容。
self.assertNotIn("secret", poll["detail"])
def test_pinned_https_uses_hostname_for_sni_not_ip(self):
seen = {}
def fake_create_connection(address, timeout=None):
seen["address"] = address
return object()
class FakeContext:
def wrap_socket(self, sock, server_hostname=None):
seen["server_hostname"] = server_hostname
return object()
with mock.patch.object(socket, "create_connection",
fake_create_connection):
with mock.patch.object(ssl, "create_default_context",
lambda: FakeContext()):
conn = _PinnedHTTPSConnection(
"example.com", 443, ["93.184.216.34"], timeout=5)
conn.connect()
self.assertEqual(seen["address"], ("93.184.216.34", 443))
self.assertEqual(seen["server_hostname"], "example.com")
# -- S2:批量总预算 --------------------------------------------
def test_batch_budget_stops_batch_and_keeps_rest_due(self):
class FakeClock:
def __init__(self):
self.now = 0.0
def __call__(self):
return self.now
clock = FakeClock()
def slow_fetcher(url, timeout):
clock.now += 50.0
return "v1"
for name in ("a", "b", "c", "d", "e"):
self._watch(name=name)
errors = []
polls = process_due_resident_watches(
self.conn, fetcher=slow_fetcher, now=self.now,
budget_seconds=100, monotonic_fn=clock, errors=errors)
# 0 秒、50 秒处各跑一条,第 3 条开始前已满 100 秒即停。
self.assertEqual(len(polls), 2)
self.assertTrue(any(w is None and "budget" in m
for w, m in errors))
# 剩余 3 条仍是 due 状态,下批可跑,不丢。
clock2 = FakeClock()
polls2 = process_due_resident_watches(
self.conn, fetcher=slow_fetcher, now=self.now,
budget_seconds=10000, monotonic_fn=clock2)
self.assertEqual(len(polls2), 3)
# -- G1:逐条隔离 + 死锁重试 + 热路径零 DDL ---------------------
def test_batch_isolates_item_errors_and_retries_deadlock(self):
watch_a = self._watch(name="a")
watch_b = self._watch(name="b")
calls = []
def flaky(url, timeout):
return "v1"
original = resident_module.run_due_resident_watch
def patched(conn, watch_id, fetcher=None, now=None,
resolver=None):
calls.append(watch_id)
if watch_id == watch_a["id"]:
raise PostgresOperationalError("deadlock detected",
locked=True)
return original(conn, watch_id, fetcher=fetcher, now=now,
resolver=resolver)
errors = []
with mock.patch.object(resident_module,
"run_due_resident_watch", patched):
polls = process_due_resident_watches(
self.conn, fetcher=flaky, now=self.now, errors=errors)
self.assertEqual(len(polls), 1)
self.assertEqual(polls[0]["watch_id"], watch_b["id"])
self.assertEqual(len(errors), 1)
self.assertEqual(errors[0][0], watch_a["id"])
# 死锁重试成功路径:第一次锁、第二次成功,errors 为空。
attempts = {"n": 0}
def retry_once(conn, watch_id, fetcher=None, now=None,
resolver=None):
if watch_id == watch_a["id"] and attempts["n"] == 0:
attempts["n"] += 1
raise PostgresOperationalError("deadlock detected",
locked=True)
return original(conn, watch_id, fetcher=fetcher, now=now,
resolver=resolver)
errors2 = []
with mock.patch.object(resident_module,
"run_due_resident_watch", retry_once):
polls2 = process_due_resident_watches(
self.conn, fetcher=flaky, now=self.now + 120,
errors=errors2)
self.assertEqual(errors2, [])
self.assertTrue(any(p["watch_id"] == watch_a["id"]
for p in polls2))
def test_hot_path_batch_executes_no_ddl(self):
class CountingConn:
def __init__(self, real):
self._real = real
self.statements = []
def execute(self, sql, params=()):
self.statements.append(sql)
return self._real.execute(sql, params)
def commit(self):
return self._real.commit()
def rollback(self):
return self._real.rollback()
self._watch(name="a")
self._watch(name="b")
counting = CountingConn(self.conn)
polls = process_due_resident_watches(
counting, fetcher=_Fetcher("v1"), now=self.now)
self.assertEqual(len(polls), 2)
ddl = [s for s in counting.statements
if s.strip().upper().startswith("CREATE")]
self.assertEqual(ddl, [])
# -- G2:每日上限 / 抖动冷却 / 保留期 ----------------------------
def test_daily_cap_and_cooldown_suppress_without_losing_change(self):
env = {
"CODEX_RESIDENT_MAX_DAILY_CHANGE_TASKS": "2",
"CODEX_RESIDENT_CHANGE_COOLDOWN_SECONDS": "300",
}
with mock.patch.dict(os.environ, env):
fetcher = _Fetcher("v1")
watch = self._watch()
run_due_resident_watch(self.conn, watch["id"],
fetcher=fetcher, now=self.now)
# 第 1 次变化:建任务。
fetcher.value = "v2"
self.now += 61
poll = run_due_resident_watch(
self.conn, watch["id"], fetcher=fetcher, now=self.now)
self.assertEqual(poll["status"], "changed")
self.assertEqual(len(self._tasks()), 1)
# 冷却期内再变:记 changed_cooldown,不建任务,游标不动。
fetcher.value = "v3"
self.now += 61
poll = run_due_resident_watch(
self.conn, watch["id"], fetcher=fetcher, now=self.now)
self.assertEqual(poll["status"], "changed_cooldown")
self.assertEqual(len(self._tasks()), 1)
after = get_watch(self.conn, watch["id"], now=self.now)
self.assertEqual(after["last_change_text"], "v2")
# 冷却过后:第 2 次变化建任务,达到上限 2。
self.now += 300
poll = run_due_resident_watch(
self.conn, watch["id"], fetcher=fetcher, now=self.now)
self.assertEqual(poll["status"], "changed")
self.assertEqual(len(self._tasks()), 2)
# 再变:上限已满,记 changed_capped,不建任务,游标不动。
fetcher.value = "v4"
self.now += 361
poll = run_due_resident_watch(
self.conn, watch["id"], fetcher=fetcher, now=self.now)
self.assertEqual(poll["status"], "changed_capped")
self.assertEqual(len(self._tasks()), 2)
after = get_watch(self.conn, watch["id"], now=self.now)
self.assertEqual(after["last_change_text"], "v3")
def test_prune_old_polls_keeps_recent(self):
watch = self._watch()
old = self.now - 100 * 86400
self.conn.execute(
"INSERT INTO resident_polls(id, watch_id, project_id,"
" status, observed_hash, detail, task_id, created_at)"
" VALUES('old1', ?, 'proj', 'unchanged', 'h', '', NULL, ?)",
(watch["id"], old))
self.conn.execute(
"INSERT INTO resident_polls(id, watch_id, project_id,"
" status, observed_hash, detail, task_id, created_at)"
" VALUES('new1', ?, 'proj', 'unchanged', 'h', '', NULL, ?)",
(watch["id"], self.now))
self.conn.commit()
deleted = prune_old_polls(self.conn, now=self.now,
retention_days=90)
self.assertEqual(deleted, 1)
remaining = [p["id"] for p in self._polls(watch["id"])]
self.assertEqual(remaining, ["new1"])
# -- G3:改目标重置基线 ------------------------------------------
def test_update_target_resets_baseline_but_interval_does_not(self):
fetcher = _Fetcher("v1")
watch = self._watch()
run_due_resident_watch(self.conn, watch["id"],
fetcher=fetcher, now=self.now)
before = get_watch(self.conn, watch["id"], now=self.now)
self.assertIsNotNone(before["last_content_hash"])
# 只改间隔:游标保留。
ok, updated = update_watch(
self.conn, watch["id"], {"interval_seconds": 3600},
now=self.now)
self.assertTrue(ok)
self.assertIsNotNone(updated["last_content_hash"])
# 改目标:游标清掉,下一轮只建基线、不建任务。
ok, updated = update_watch(
self.conn, watch["id"], {"target_url": URL2}, now=self.now)
self.assertTrue(ok)
self.assertIsNone(updated["last_content_hash"])
def url_fetcher(url, timeout):
return "new-site-content" if url == URL2 else "v1"
self.now += 61
poll = run_due_resident_watch(
self.conn, watch["id"], fetcher=url_fetcher, now=self.now)
self.assertEqual(poll["status"], "baseline")
self.assertEqual(len(self._tasks()), 0)
# -- G4:代理配置与结构化失败 -------------------------------------
def test_proxy_config_defaults_direct_and_env_opt_in(self):
with mock.patch.dict(os.environ, {}, clear=False):
os.environ.pop("CODEX_RESIDENT_PROXY", None)
os.environ.pop("http_proxy", None)
os.environ.pop("https_proxy", None)
cfg = resident_config()
self.assertEqual(cfg["proxy"]["mode"], "direct")
with mock.patch.dict(os.environ,
{"CODEX_RESIDENT_PROXY": "env",
"https_proxy": "http://proxy.test:8080"}):
cfg = resident_config()
self.assertEqual(cfg["proxy"]["mode"], "env")
with mock.patch.dict(
os.environ,
{"CODEX_RESIDENT_PROXY": "http://proxy.test:8080"}):
cfg = resident_config()
self.assertEqual(cfg["proxy"]["mode"], "url")
self.assertEqual(cfg["proxy"]["url"],
"http://proxy.test:8080")
with mock.patch.dict(os.environ,
{"CODEX_RESIDENT_PROXY": "ftp://x/"}):
with self.assertRaises(FetchError) as ctx:
resident_module.resolve_proxy_config()
self.assertEqual(ctx.exception.reason, "proxy")
# 默认值本身是拟定值(见摘要拟定清单),这里只断言
# 当前实现值,不把它写成用户已拍板。
self.assertEqual(resident_config()["max_daily_change_tasks"], 24)
self.assertEqual(
resident_config()["change_cooldown_seconds"], 300)
self.assertEqual(resident_config()["poll_retention_days"], 90)
self.assertEqual(resident_config()["batch_budget_seconds"], 120)
def test_classify_fetch_errors(self):
self.assertEqual(classify_fetch_error(
SSRFBlockedError("blocked")), "ssrf_blocked")
self.assertEqual(classify_fetch_error(
HttpStatusFetchError(404)), "http_status")
self.assertEqual(classify_fetch_error(socket.timeout("t")),
"timeout")
self.assertEqual(classify_fetch_error(
ConnectionRefusedError("refused")), "network")
self.assertEqual(classify_fetch_error(RuntimeError("boom")),
"unknown")
def test_http_status_fetch_raises_structured_error(self):
class Handler(http.server.BaseHTTPRequestHandler):
def do_GET(self):
body = b"not here"
self.send_response(404)
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def log_message(self, *args):
pass
server = http.server.ThreadingHTTPServer(("127.0.0.1", 0),
Handler)
port = server.server_address[1]
thread = threading.Thread(target=server.serve_forever,
daemon=True)
thread.start()
try:
# 单跳层真实 404(本地服务,绕过 DNS 门直接调单跳)。
status, _loc, _raw = resident_module._fetch_single_hop(
"http://127.0.0.1:%d/missing" % port, 10,
["127.0.0.1"], None)
self.assertEqual(status, 404)
# default_fetch 的状态分支:解析门放行公网 IP 后,
# 单跳返回 404 即抛结构化 http_status 错误(正文丢弃)。
with mock.patch.object(
resident_module, "_fetch_single_hop",
return_value=(404, None, b"not here")):
with self.assertRaises(HttpStatusFetchError) as ctx:
default_fetch(
"http://status.test/missing", 10,
resolver=lambda h, p: ["93.184.216.34"])
self.assertEqual(ctx.exception.reason, "http_status")
self.assertEqual(ctx.exception.status, 404)
finally:
server.shutdown()
server.server_close()
thread.join(timeout=5)
# -- S3:迁移覆盖(无 PG 也可断言)--------------------------------
def test_migration_coverage_includes_resident_tables(self):
coverage = migration_coverage()
self.assertEqual(coverage["missing_from_migration"], ())
self.assertIn("resident_watches", MIGRATION_TABLES)
self.assertIn("resident_polls", MIGRATION_TABLES)
self.assertIn("resident_watches", coverage["schema_tables"])
self.assertIn("resident_polls", coverage["schema_tables"])
# -- G5:metrics 口径 ---------------------------------------------
def test_resident_metrics_counts(self):
fetcher = _Fetcher("v1")
watch = self._watch()
run_due_resident_watch(self.conn, watch["id"],
fetcher=fetcher, now=self.now)
fetcher.value = "v2"
self.now += 61
run_due_resident_watch(self.conn, watch["id"],
fetcher=fetcher, now=self.now)
stats = resident_metrics(self.conn, {"proj"}, now=self.now)
self.assertTrue(stats["available"])
self.assertEqual(stats["watches_by_status"], {"running": 1})
self.assertEqual(stats["polls_last_24h"], 2)
self.assertEqual(stats["changed_last_24h"], 1)
self.assertEqual(stats["change_tasks_last_24h"], 1)
self.assertEqual(stats["failed_last_24h"], 0)
# 到期积压口径:把 next_poll_at 推到过去,due_count 见涨。
later = self.now + 3600
stats2 = resident_metrics(self.conn, {"proj"}, now=later)
self.assertEqual(stats2["due_count"], 1)
self.assertGreater(stats2["max_overdue_seconds"], 0)
# 非成员项目不计入。
stats3 = resident_metrics(self.conn, {"other"}, now=self.now)
self.assertEqual(stats3["watches_by_status"], {})
class ResidentP0ServerTestCase(unittest.TestCase):
"""服务端侧:/metrics resident 段、502 固定错误码、polls
异常记日志不静默(响应形态仍为 polls=[] + note)。"""
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.db_path = os.path.join(self.tmp.name, "control.db")
self.auth = AuthStore(self.db_path)
self.owner_id, self.owner_key = self.auth.create_user("owner")
self.server = create_server(self.db_path, "127.0.0.1", 0)
self.port = self.server.server_address[1]
self.thread = threading.Thread(
target=self.server.serve_forever, daemon=True)
self.thread.start()
status, self.project = self._req(
"POST", "/projects", {"name": "demo"}, self.owner_key)
self.assertEqual(status, 201, self.project)
def tearDown(self):
self.server.shutdown()
self.server.server_close()
self.thread.join(timeout=5)
try:
self.auth.conn.close()
except Exception:
pass
self.tmp.cleanup()
def _req(self, method, path, body=None, key=None):
conn = http.client.HTTPConnection("127.0.0.1", self.port,
timeout=10)
hdrs = {}
payload = None
if body is not None:
payload = json.dumps(body).encode("utf-8")
hdrs["Content-Type"] = "application/json"
if key is not None:
hdrs["X-API-Key"] = key
try:
conn.request(method, path, body=payload, headers=hdrs)
resp = conn.getresponse()
data = resp.read()
finally:
conn.close()
try:
parsed = json.loads(data.decode("utf-8")) if data else None
except ValueError:
parsed = None
return resp.status, parsed
def _url(self, suffix=""):
return "/projects/%s/resident-watches%s" % (
self.project["id"], suffix)
def test_metrics_contains_resident_section(self):
status, watch = self._req(
"POST", self._url(),
{"name": "w1", "target_url": URL, "prompt": "p"},
self.owner_key)
self.assertEqual(status, 201, watch)
status, body = self._req("GET", "/metrics", None, self.owner_key)
self.assertEqual(status, 200, body)
resident = body["resident"]
self.assertTrue(resident["available"])
self.assertEqual(resident["watches_by_status"], {"running": 1})
self.assertEqual(resident["due_count"], 1)
self.assertIn("max_daily_change_tasks", resident["config"])
def test_resident_502_uses_fixed_error_code(self):
import cloud.server as server_module
original = server_module._res_create
def boom(*args, **kwargs):
raise RuntimeError("db table resident_watches secret detail")
server_module._res_create = boom
try:
status, body = self._req(
"POST", self._url(),
{"name": "w1", "target_url": URL, "prompt": "p"},
self.owner_key)
finally:
server_module._res_create = original
self.assertEqual(status, 502)
self.assertEqual(body["error"], "resident_operation_failed")
self.assertNotIn("secret", json.dumps(body))
def test_resident_get_polls_failure_keeps_note(self):
status, watch = self._req(
"POST", self._url(),
{"name": "w1", "target_url": URL, "prompt": "p"},
self.owner_key)
self.assertEqual(status, 201, watch)
import cloud.server as server_module
original = server_module._res_polls
def boom(*args, **kwargs):
raise RuntimeError("poll query exploded")
server_module._res_polls = boom
try:
status, body = self._req(
"GET", self._url("/" + watch["id"]), None,
self.owner_key)
finally:
server_module._res_polls = original
self.assertEqual(status, 200, body)
self.assertEqual(body["polls"], [])
self.assertEqual(body["polls_note"], "polls unavailable")
class ResidentP0WorkerTestCase(unittest.TestCase):
"""worker 驱动:首次批量 ensure 一次、resolver 透传。"""
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.db_path = os.path.join(self.tmp.name, "control.db")
init_db(self.db_path)
self.conn = sqlite3.connect(self.db_path)
self.conn.row_factory = sqlite3.Row
ensure_resident_tables(self.conn)
self.now = 1700000000.0
def tearDown(self):
self.conn.close()
self.tmp.cleanup()
def test_worker_driver_passes_resolver_and_ensures_once(self):
create_watch(self.conn, "proj", "w1", URL, "p",
"owner1", interval_seconds=60, now=self.now)
worker = Worker(self.db_path, "/fake/host.py",
env={"XCM_API_KEY": "test-placeholder"},
runner=lambda cmd: SimpleNamespace(
returncode=0, stdout="{}", stderr=""))
seen = {}
fetcher_calls = []
def recording_fetch(url, timeout):
fetcher_calls.append(url)
return "v1"
def recording_resolver(host, port):
seen["host"] = host
return ["127.0.0.1"]
# worker 经 resident_fetcher/resident_resolver 注入点取用。
worker.resident_fetcher = recording_fetch
worker.resident_resolver = recording_resolver
polls = worker.process_due_resident_watches(now=self.now)
self.assertEqual(len(polls), 1)
self.assertEqual(polls[0]["status"], "baseline")
# fetcher 注入时 resolver 不经过 default_fetch,不必被调用;
# 这里只断言 worker 首批 ensure 标志已立、次批不再重复建表
# (标志位本身即“每 worker 一次”的证据)。
self.assertTrue(
getattr(worker, "_resident_tables_ensured", False))
self.assertEqual(fetcher_calls, [URL])
if __name__ == "__main__":
unittest.main()
+180
View File
@@ -0,0 +1,180 @@
"""常驻监听服务端单测:POST/GET /projects/{id}/resident-watches、
PATCH、POST pause / resume / end。不跑真实模型、不联网。
骨架与 test_automations_server.py 同款(本地临时端口 + 临时
SQLite)。"""
import http.client
import json
import os
import sqlite3
import sys
import tempfile
import threading
import unittest
from pathlib import Path
try:
from cloud.auth import AuthStore
from cloud.server import create_server
except ImportError: # discover -s cloud/tests 时把项目根目录补进 sys.path
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
from cloud.auth import AuthStore
from cloud.server import create_server
URL = "https://example.com/data"
class ResidentServerTestCase(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.db_path = os.path.join(self.tmp.name, "control.db")
self.auth = AuthStore(self.db_path)
self.owner_id, self.owner_key = self.auth.create_user("owner")
self.outsider_id, self.outsider_key = self.auth.create_user(
"outsider")
self.server = create_server(self.db_path, "127.0.0.1", 0)
self.port = self.server.server_address[1]
self.thread = threading.Thread(
target=self.server.serve_forever, daemon=True)
self.thread.start()
status, self.project = self._req(
"POST", "/projects", {"name": "demo"}, self.owner_key)
self.assertEqual(status, 201)
def tearDown(self):
self.server.shutdown()
self.server.server_close()
self.thread.join(timeout=5)
try:
self.auth.conn.close()
except Exception:
pass
self.tmp.cleanup()
def _req(self, method, path, body=None, key=None):
conn = http.client.HTTPConnection(
"127.0.0.1", self.port, timeout=10)
hdrs = {}
payload = None
if body is not None:
payload = json.dumps(body).encode("utf-8")
hdrs["Content-Type"] = "application/json"
if key is not None:
hdrs["X-API-Key"] = key
try:
conn.request(method, path, body=payload, headers=hdrs)
resp = conn.getresponse()
data = resp.read()
finally:
conn.close()
try:
parsed = json.loads(data.decode("utf-8")) if data else None
except ValueError:
parsed = None
return resp.status, parsed
def _url(self, suffix=""):
return "/projects/%s/resident-watches%s" % (
self.project["id"], suffix)
def _create(self, **extra):
body = {"name": "watch1", "target_url": URL,
"prompt": "变化时汇报"}
body.update(extra)
return self._req("POST", self._url(), body, self.owner_key)
def _audit_actions(self):
conn = sqlite3.connect(self.db_path)
try:
rows = conn.execute(
"SELECT action FROM audit_log WHERE project_id = ?",
(self.project["id"],)).fetchall()
finally:
conn.close()
return [r[0] for r in rows]
def test_create_list_get_and_validation(self):
status, watch = self._create(interval_seconds=3600)
self.assertEqual(status, 201, watch)
self.assertEqual(watch["status"], "running")
self.assertEqual(watch["target_url"], URL)
self.assertTrue(watch["due"])
self.assertIsNotNone(watch["next_poll_at"])
self.assertIn("resident_watch_created", self._audit_actions())
status, body = self._req("GET", self._url(), None, self.owner_key)
self.assertEqual(status, 200)
self.assertEqual(len(body["resident_watches"]), 1)
self.assertEqual(body["due_count"], 1)
status, body = self._req(
"GET", self._url("/" + watch["id"]), None, self.owner_key)
self.assertEqual(status, 200)
self.assertEqual(body["id"], watch["id"])
self.assertEqual(body["polls"], [])
for bad in (
{"name": "x", "target_url": "ftp://example.com/x",
"prompt": "p"},
{"name": "x", "target_url": "https://localhost/x",
"prompt": "p"},
{"name": "x", "target_url": URL, "prompt": "p",
"interval_seconds": 30},
{"name": "watch1", "target_url": URL, "prompt": "p"},
{"target_url": URL, "prompt": "p"},
{"name": "x", "prompt": "p"}):
status, _ = self._req("POST", self._url(), bad, self.owner_key)
self.assertEqual(status, 400, bad)
status, _ = self._req(
"GET", self._url("/nope"), None, self.owner_key)
self.assertEqual(status, 404)
def test_pause_resume_end_lifecycle_and_terminal_409(self):
status, watch = self._create()
self.assertEqual(status, 201)
wid = watch["id"]
status, body = self._req(
"POST", self._url("/%s/pause" % wid), {}, self.owner_key)
self.assertEqual(status, 200, body)
self.assertEqual(body["status"], "paused")
self.assertEqual(body["paused_reason"], "user_paused")
self.assertIsNone(body["next_poll_at"])
self.assertFalse(body["due"])
status, body = self._req(
"POST", self._url("/%s/pause" % wid), {}, self.owner_key)
self.assertEqual(status, 409)
status, body = self._req(
"POST", self._url("/%s/resume" % wid), {}, self.owner_key)
self.assertEqual(status, 200, body)
self.assertEqual(body["status"], "running")
self.assertIsNone(body["paused_reason"])
self.assertTrue(body["due"])
status, body = self._req(
"POST", self._url("/%s/end" % wid), {}, self.owner_key)
self.assertEqual(status, 200, body)
self.assertEqual(body["status"], "ended")
for action in ("pause", "resume", "end"):
status, body = self._req(
"POST", self._url("/%s/%s" % (wid, action)), {},
self.owner_key)
self.assertEqual(status, 409, action)
self.assertIn("terminal", body["error"])
actions = self._audit_actions()
self.assertIn("resident_watch_paused", actions)
self.assertIn("resident_watch_resumed", actions)
self.assertIn("resident_watch_ended", actions)
def test_update_and_outsider_forbidden(self):
status, watch = self._create()
self.assertEqual(status, 201)
wid = watch["id"]
status, body = self._req(
"PATCH", self._url("/" + wid),
{"interval_seconds": 7200}, self.owner_key)
self.assertEqual(status, 200, body)
self.assertEqual(body["interval_seconds"], 7200)
status, _ = self._req(
"POST", self._url(),
{"name": "x", "target_url": URL, "prompt": "p"},
self.outsider_key)
self.assertIn(status, (401, 403, 404))
if __name__ == "__main__":
unittest.main()
+416
View File
@@ -0,0 +1,416 @@
"""常驻监听网站闭环端到端测试。
上一轮只验到注入假抓取的单测:真实抓取、真实 run_loop 轮次
边界、经服务端 API 的全链路(创建 -> 基线 -> 变化建任务 ->
暂停 -> 恢复 -> 失败退避 -> 结束)都没有跑过。本文件补这一环,
不改任何生产默认值、不改生产校验口径:
- 间隔仍是生产下限 60 秒;到期不是改间隔,而是给驱动传一个
更晚的 now(模拟时间流逝),与上一轮单测同款时间注入。
- 目标 URL 仍走生产校验:主机名 website-loop.test 不是 IP 字
面、不是 localhost,所以校验通过;抓取本身走真实
default_fetch(DNS 门、pin 连接、超时、1MB 截断),不注入
假 fetcher。注意:生产默认 DNS 门会把解析到 127.0.0.1 的
目标拦住,所以本文件用两段测试接缝:worker 的
resident_resolver 把该主机名解析成公网测试 IP(门正常放
行),测试进程内再在 socket 层把该 IP:内容端口重定向回
本地服务(传输层终点是测试替身,门与 pin 全程真实跑);
DNS 门本身由 test_resident_p0_fixes.py 的默认解析器用例
覆盖,不在本文件重复证明。
- 内容服务是线程内真实 http.server,可变内容、也可停服;
控制面是真实 ThreadingHTTPServer + 真实 SQLite 临时库;
变化建出的任务、轮询、审计都直接读库/接口核对。
不联网、不跑真实模型、不碰生产。
"""
import http.client
import http.server
import json
import os
import socket
import sqlite3
import sys
import tempfile
import threading
import time
import unittest
from pathlib import Path
from types import SimpleNamespace
try:
from cloud.auth import AuthStore
from cloud.resident import _content_hash
from cloud.server import create_server
from cloud.worker import Worker
except ImportError: # discover -s cloud/tests 时把项目根目录补进 sys.path
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
from cloud.auth import AuthStore
from cloud.resident import _content_hash
from cloud.server import create_server
from cloud.worker import Worker
HOST = "website-loop.test"
class WebsiteLoopTestCase(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.db_path = os.path.join(self.tmp.name, "control.db")
self.auth = AuthStore(self.db_path)
self.owner_id, self.owner_key = self.auth.create_user("owner")
self.server = create_server(self.db_path, "127.0.0.1", 0)
self.port = self.server.server_address[1]
self.thread = threading.Thread(
target=self.server.serve_forever, daemon=True)
self.thread.start()
status, self.project = self._req(
"POST", "/projects", {"name": "demo"}, self.owner_key)
self.assertEqual(status, 201, self.project)
# 可变内容服务:state 字典跨重启保留,停服/重启可复现。
self.state = {"content": "version one"}
state = self.state
class Handler(http.server.BaseHTTPRequestHandler):
def do_GET(self):
body = state["content"].encode("utf-8")
self.send_response(200)
self.send_header("Content-Type", "text/plain")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def log_message(self, *args):
pass
self._handler = Handler
self.content_server = http.server.ThreadingHTTPServer(
("127.0.0.1", 0), Handler)
self.content_port = self.content_server.server_address[1]
self.content_thread = threading.Thread(
target=self.content_server.serve_forever, daemon=True)
self.content_thread.start()
# 测试专用 DNS:只把 website-loop.test 指到本地服务,
# 其它主机一律走原始解析;生产校验与抓取代码一字不改。
self._orig_getaddrinfo = socket.getaddrinfo
def patched(host, port, family=0, type=0, proto=0, flags=0):
if host == HOST:
return [(socket.AF_INET, socket.SOCK_STREAM, 6, "",
("127.0.0.1", port))]
return self._orig_getaddrinfo(
host, port, family, type, proto, flags)
socket.getaddrinfo = patched
# 测试环境自带 egress 代理:urllib 默认会把这个测试主机名
# 也送代理并被断开。只给本测试主机名加 no_proxy(测试
# 进程内生效),生产抓取代码与代理行为一字不改。
self._old_no_proxy = os.environ.get("no_proxy")
self._old_NO_PROXY = os.environ.get("NO_PROXY")
os.environ["no_proxy"] = (
(self._old_no_proxy + ",") if self._old_no_proxy
else "") + HOST
os.environ["NO_PROXY"] = (
(self._old_NO_PROXY + ",") if self._old_NO_PROXY
else "") + HOST
self.worker = Worker(
self.db_path, "/fake/host.py",
env={"XCM_API_KEY": "test-placeholder"},
runner=lambda cmd: SimpleNamespace(
returncode=0, stdout="{}", stderr=""))
# 两段测试接缝(见文件头说明):resolver 返回公网测试
# IP,socket 层把该 IP:内容端口重定向回本地服务。
try:
from cloud.resident import _default_resolver
except ImportError:
_default_resolver = None
def _loop_resolver(host, port, _default=_default_resolver):
if host == HOST:
return ["93.184.216.34"]
if _default is not None:
return _default(host, port)
return [socket.getaddrinfo(host, port)[0][4][0]]
self.worker.resident_resolver = _loop_resolver
self._orig_create_connection = socket.create_connection
def _patched_create_connection(address, *args, **kwargs):
if address[0] == "93.184.216.34" \
and address[1] == self.content_port:
address = ("127.0.0.1", self.content_port)
return self._orig_create_connection(address, *args,
**kwargs)
socket.create_connection = _patched_create_connection
def tearDown(self):
# setUp 中途失败时属性可能未建:getattr 防御,避免
# tearDown 的 AttributeError 掩盖原始失败。
if getattr(self, "_orig_getaddrinfo", None) is not None:
socket.getaddrinfo = self._orig_getaddrinfo
if getattr(self, "_orig_create_connection", None) is not None:
socket.create_connection = self._orig_create_connection
if getattr(self, "_old_no_proxy", None) is None:
if "_old_no_proxy" in self.__dict__:
os.environ.pop("no_proxy", None)
else:
os.environ["no_proxy"] = self._old_no_proxy
if getattr(self, "_old_NO_PROXY", None) is None:
if "_old_NO_PROXY" in self.__dict__:
os.environ.pop("NO_PROXY", None)
else:
os.environ["NO_PROXY"] = self._old_NO_PROXY
try:
self.content_server.shutdown()
self.content_server.server_close()
except Exception:
pass
try:
self.content_thread.join(timeout=5)
except Exception:
pass
try:
self.server.shutdown()
self.server.server_close()
except Exception:
pass
try:
self.thread.join(timeout=5)
except Exception:
pass
try:
self.auth.conn.close()
except Exception:
pass
try:
self.tmp.cleanup()
except Exception:
pass
def _req(self, method, path, body=None, key=None):
conn = http.client.HTTPConnection("127.0.0.1", self.port,
timeout=10)
hdrs = {}
payload = None
if body is not None:
payload = json.dumps(body).encode("utf-8")
hdrs["Content-Type"] = "application/json"
if key is not None:
hdrs["X-API-Key"] = key
try:
conn.request(method, path, body=payload, headers=hdrs)
resp = conn.getresponse()
data = resp.read()
finally:
conn.close()
try:
parsed = json.loads(data.decode("utf-8")) if data else None
except ValueError:
parsed = None
return resp.status, parsed
def _url(self, suffix=""):
return "/projects/%s/resident-watches%s" % (
self.project["id"], suffix)
def _create_watch(self):
body = {"name": "site1",
"target_url": "http://%s:%d/data" % (
HOST, self.content_port),
"prompt": "变化时汇报",
"interval_seconds": 60}
status, watch = self._req("POST", self._url(), body,
self.owner_key)
self.assertEqual(status, 201, watch)
return watch
def _get_watch(self, watch_id):
status, body = self._req(
"GET", self._url("/" + watch_id), None, self.owner_key)
self.assertEqual(status, 200, body)
return body
def _tasks(self):
conn = sqlite3.connect(self.db_path)
try:
return conn.execute(
"SELECT id, prompt, status FROM tasks"
" ORDER BY created_at, id").fetchall()
finally:
conn.close()
def _audit(self):
conn = sqlite3.connect(self.db_path)
try:
return [r[0] for r in conn.execute(
"SELECT action FROM audit_log WHERE project_id = ?",
(self.project["id"],)).fetchall()]
finally:
conn.close()
def test_full_website_loop_end_to_end(self):
# 1. 创建:经服务端 API,running、游标为空、下次轮询在位。
watch = self._create_watch()
wid = watch["id"]
self.assertEqual(watch["status"], "running")
self.assertEqual(watch["interval_seconds"], 60)
self.assertEqual(watch["fetch_timeout_seconds"], 10)
self.assertIsNone(watch["last_content_hash"])
self.assertIsNotNone(watch["next_poll_at"])
self.assertIn("resident_watch_created", self._audit())
t0 = time.time()
# 2. 基线:真实抓取 version one,只建游标,不建任务。
polls = self.worker.process_due_resident_watches(now=t0 + 120)
self.assertEqual(len(polls), 1, polls)
self.assertEqual(polls[0]["status"], "baseline")
self.assertEqual(
polls[0]["observed_hash"], _content_hash("version one"))
self.assertEqual(self._tasks(), [])
body = self._get_watch(wid)
self.assertEqual(body["last_content_hash"],
_content_hash("version one"))
self.assertEqual(len(body["polls"]), 1)
self.assertEqual(body["polls"][0]["status"], "baseline")
# 3. 变化:内容改成 version two,到期轮询建出 queued 任务。
self.state["content"] = "version two"
polls = self.worker.process_due_resident_watches(now=t0 + 240)
self.assertEqual(len(polls), 1, polls)
self.assertEqual(polls[0]["status"], "changed")
self.assertIsNotNone(polls[0]["task_id"])
tasks = self._tasks()
self.assertEqual(len(tasks), 1)
self.assertEqual(tasks[0][0], polls[0]["task_id"])
self.assertEqual(tasks[0][2], "queued")
self.assertIn("变化时汇报", tasks[0][1])
self.assertIn("version two", tasks[0][1])
self.assertIn("resident_watch_change_task_created",
self._audit())
body = self._get_watch(wid)
self.assertEqual(body["last_content_hash"],
_content_hash("version two"))
self.assertIn("version two", body["last_change_text"])
# 4. 暂停:暂停期间再改内容,驱动不轮询、游标不动。
status, body = self._req(
"POST", self._url("/%s/pause" % wid), {}, self.owner_key)
self.assertEqual(status, 200, body)
self.assertEqual(body["status"], "paused")
self.assertIsNone(body["next_poll_at"])
self.state["content"] = "version three"
polls = self.worker.process_due_resident_watches(now=t0 + 400)
self.assertEqual(polls, [])
body = self._get_watch(wid)
self.assertEqual(len(body["polls"]), 2)
self.assertEqual(body["last_content_hash"],
_content_hash("version two"))
# 5. 恢复:用旧游标比对,报出暂停期间的变化并建任务,
# 不是从头建基线。轮询时刻取 t0+540:距上次建任务
# (t0+240)已过 300 秒抖动冷却,否则新冷却语义会
# 正确地记 changed_cooldown 而不是 changed(冷却期
# 内的变化不丢,冷却过后复查仍可报,见 P0 批单测)。
status, body = self._req(
"POST", self._url("/%s/resume" % wid), {}, self.owner_key)
self.assertEqual(status, 200, body)
self.assertEqual(body["status"], "running")
polls = self.worker.process_due_resident_watches(now=t0 + 540)
self.assertEqual(len(polls), 1, polls)
self.assertEqual(polls[0]["status"], "changed")
tasks = self._tasks()
self.assertEqual(len(tasks), 2)
self.assertIn("version three", tasks[1][1])
body = self._get_watch(wid)
self.assertEqual(body["last_content_hash"],
_content_hash("version three"))
# 6. 失败:停服后轮询记 failed、退避、不结束、游标不动;
# 恢复服务后内容未变,轮询记 unchanged、失败清零。
self.content_server.shutdown()
self.content_server.server_close()
self.content_thread.join(timeout=5)
polls = self.worker.process_due_resident_watches(now=t0 + 600)
self.assertEqual(len(polls), 1, polls)
self.assertEqual(polls[0]["status"], "failed")
body = self._get_watch(wid)
self.assertEqual(body["status"], "running")
self.assertEqual(body["consecutive_failures"], 1)
self.assertEqual(body["last_content_hash"],
_content_hash("version three"))
self.assertGreater(body["next_poll_at"], t0 + 600 + 60)
self.content_server = http.server.ThreadingHTTPServer(
("127.0.0.1", self.content_port), self._handler)
self.content_thread = threading.Thread(
target=self.content_server.serve_forever, daemon=True)
self.content_thread.start()
polls = self.worker.process_due_resident_watches(now=t0 + 800)
self.assertEqual(len(polls), 1, polls)
self.assertEqual(polls[0]["status"], "unchanged")
body = self._get_watch(wid)
self.assertEqual(body["consecutive_failures"], 0)
self.assertEqual(len(self._tasks()), 2)
# 7. 结束:终态,驱动不再轮询,改/停/恢复一律 409。
status, body = self._req(
"POST", self._url("/%s/end" % wid), {}, self.owner_key)
self.assertEqual(status, 200, body)
self.assertEqual(body["status"], "ended")
self.assertIsNone(body["next_poll_at"])
self.assertEqual(body["last_content_hash"],
_content_hash("version three"))
polls = self.worker.process_due_resident_watches(now=t0 + 1000)
self.assertEqual(polls, [])
for action in ("pause", "resume", "end"):
status, body = self._req(
"POST", self._url("/%s/%s" % (wid, action)), {},
self.owner_key)
self.assertEqual(status, 409, action)
status, _ = self._req(
"PATCH", self._url("/" + wid), {"interval_seconds": 120},
self.owner_key)
self.assertEqual(status, 409)
body = self._get_watch(wid)
self.assertEqual(body["status"], "ended")
self.assertEqual(len(body["polls"]), 5)
def test_run_loop_boundary_calls_resident_driver(self):
# 8. 驱动接线:run_loop 轮次边界确实调到常驻驱动,
# 不只是测试直接调 process_due。run_all 等无关边界
# 在本测试里打桩,只留 resident 驱动走真实抓取。
watch = self._create_watch()
wid = watch["id"]
worker = self.worker
calls = []
original = Worker.process_due_resident_watches
def recording():
# 实例属性上的函数不会自动绑定 self,必须闭包捕获
# worker,不能写成 def recording(self)。
calls.append(time.time())
return original(worker)
worker.process_due_resident_watches = recording
worker.run_all = lambda: []
worker.cleanup_stale_containers = lambda: None
worker._write_heartbeat = lambda: None
worker.register = lambda: None
worker._process_push_due = lambda: None
worker.process_auto_memories = lambda: None
worker._reap_kill_requests = lambda *a, **k: None
worker.auto_resume_interrupted = lambda: None
worker._heartbeat_running_tasks = lambda: None
worker._registry_heartbeat = lambda: None
worker._code_changed = lambda: False
worker._code_signature_valid = lambda: True
worker.run_loop(interval=0.05, max_seconds=0.4)
self.assertGreaterEqual(len(calls), 1)
body = self._get_watch(wid)
self.assertEqual(len(body["polls"]), 1)
self.assertEqual(body["polls"][0]["status"], "baseline")
self.assertEqual(body["last_content_hash"],
_content_hash("version one"))
if __name__ == "__main__":
unittest.main()
+27 -7
View File
@@ -67,18 +67,38 @@ def _exa_body():
class ChainTestCase(unittest.TestCase):
def test_fallback_perplexity_to_exa_and_key_not_leaked(self):
def test_default_auto_frozen_falls_back_searxng_to_duckduckgo_and_key_not_leaked(self):
# Paid-freeze update of the old perplexity -> exa fallback
# test: default auto no longer contains any paid backend,
# even when all four paid keys are set, so the fallback
# that still exists by default is searxng -> duckduckgo
# and no paid URL is ever touched. The old perplexity ->
# exa fallback is preserved, but only under
# WEBSEARCH_ENABLE_PAID=1; see test_websearch_paid_freeze.py.
html = (
'<a class="result__a" href="//duckduckgo.com/l/?uddg='
'https%3A%2F%2Fexample.com%2Fddg">DDG Title</a>'
'<a class="result__snippet">ddg snippet</a>')
http = FakeHttp([
("perplexity", {}, 500),
("api.exa.ai", _exa_body(), 200),
("searx", {}, 500),
("duckduckgo", html, 200),
])
env = dict(_FAKE_KEYS)
env["SEARXNG_URL"] = "http://searx.test"
out = search("test query", provider="auto",
env=dict(_FAKE_KEYS), http=http)
self.assertEqual(out["provider"], "exa")
env=env, http=http)
self.assertEqual(out["provider"], "duckduckgo")
self.assertEqual(out["results"][0]["url"],
"https://example.com/exa")
self.assertEqual(out["attempted"][0]["provider"], "perplexity")
"https://example.com/ddg")
self.assertEqual(out["attempted"][0]["provider"], "searxng")
self.assertFalse(out["attempted"][0]["ok"])
self.assertEqual(
[a["provider"] for a in out["attempted"]],
["searxng", "duckduckgo"])
for _method, url, _payload in http.seen:
for paid in ("perplexity", "exa.ai", "tavily",
"search.brave.com"):
self.assertNotIn(paid, url)
blob = json.dumps(out)
for secret in _FAKE_KEYS.values():
self.assertNotIn(secret, blob)
+280
View File
@@ -0,0 +1,280 @@
"""Paid-freeze tests: default auto is free-only, pin still works,
unfreeze is explicit (WEBSEARCH_ENABLE_PAID=1, exact value "1").
No network, no real keys: every HTTP call goes through a FakeHttp,
and the fake key values are deliberately distinctive so a leak into
an error string or a tool result would be caught by assertion.
"""
import json
import os
import socket
import sys
import unittest
import urllib.error
from pathlib import Path
from unittest import mock
try:
from cloud.websearch import (
DEFAULT_CHAIN, DEFAULT_FREE_CHAIN, HttpClient, WebSearchError,
available_providers, paid_search_enabled,
resolve_provider_chain, search)
from cloud.websearch_mcp import handle_request
except ImportError:
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
from cloud.websearch import (
DEFAULT_CHAIN, DEFAULT_FREE_CHAIN, HttpClient, WebSearchError,
available_providers, paid_search_enabled,
resolve_provider_chain, search)
from cloud.websearch_mcp import handle_request
_FAKE_KEYS = {
"PERPLEXITY_API_KEY": "pplx-SECRET-111",
"EXA_API_KEY": "exa-SECRET-222",
"TAVILY_API_KEY": "tvly-SECRET-333",
"BRAVE_API_KEY": "brave-SECRET-444",
}
_DDG_HTML = (
'<a class="result__a" href="//duckduckgo.com/l/?uddg='
'https%3A%2F%2Fexample.com%2Fddg">DDG Title</a>'
'<a class="result__snippet">ddg snippet</a>')
class FakeHttp:
"""post_json/get keyed by URL substring; failures raise like HTTP."""
def __init__(self, routes):
self.routes = routes
self.seen = []
def post_json(self, url, headers, payload):
self.seen.append(("POST", url, payload))
return self._route(url)
def get(self, url, headers=None):
self.seen.append(("GET", url, headers))
return self._route(url)
def _route(self, url):
for needle, body, status in self.routes:
if needle in url:
if status != 200:
raise WebSearchError("HTTP %s" % status)
if isinstance(body, dict):
return 200, json.dumps(body).encode(), "application/json"
return 200, body.encode(), "text/html"
raise WebSearchError("HTTP 404")
def _searxng_body():
return {"results": [
{"title": "SearXNG Hit", "url": "https://example.com/sx",
"content": "searxng snippet"}]}
def _paid_env(**extra):
env = dict(_FAKE_KEYS)
env["SEARXNG_URL"] = "http://searx.test"
env.update(extra)
return env
def _paid_hosts_seen(http):
hosts = []
for _method, url, _payload in http.seen:
for paid in ("perplexity", "exa.ai", "tavily",
"search.brave.com"):
if paid in url:
hosts.append(paid)
return hosts
class FrozenDefaultTestCase(unittest.TestCase):
def test_default_auto_excludes_paid_even_with_all_keys(self):
env = _paid_env()
self.assertEqual(available_providers(env),
["searxng", "duckduckgo"])
self.assertEqual(resolve_provider_chain("auto", env),
["searxng", "duckduckgo"])
http = FakeHttp([("searx", _searxng_body(), 200)])
out = search("q", provider="auto", env=env, http=http)
self.assertEqual(out["provider"], "searxng")
self.assertEqual(_paid_hosts_seen(http), [])
def test_default_auto_without_searxng_url_is_duckduckgo_only(self):
env = dict(_FAKE_KEYS)
self.assertEqual(resolve_provider_chain("auto", env),
["duckduckgo"])
http = FakeHttp([("duckduckgo", _DDG_HTML, 200)])
out = search("q", provider="auto", env=env, http=http)
self.assertEqual(out["provider"], "duckduckgo")
self.assertEqual(_paid_hosts_seen(http), [])
def test_searxng_failure_never_falls_back_to_paid(self):
http = FakeHttp([("searx", {}, 500),
("duckduckgo", "<html>challenge</html>", 200)])
with self.assertRaises(WebSearchError) as ctx:
search("q", provider="auto", env=_paid_env(), http=http)
self.assertEqual(
[a["provider"] for a in ctx.exception.attempted],
["searxng", "duckduckgo"])
self.assertEqual(_paid_hosts_seen(http), [])
# No fabricated result: the error names both free stages.
self.assertIn("searxng", str(ctx.exception))
self.assertIn("duckduckgo", str(ctx.exception))
def test_duckduckgo_fallback_failure_reports_clearly(self):
# DuckDuckGo under the VM/host egress returns a 202-style
# robot-challenge page: the parser yields no results, which
# must surface as an honest "no results" failure for that
# stage, never as a fabricated hit.
http = FakeHttp([("searx", {}, 500),
("duckduckgo", "<html>anomaly</html>", 200)])
with self.assertRaises(WebSearchError) as ctx:
search("q", provider="auto", env=_paid_env(), http=http)
attempted = {a["provider"]: a for a in ctx.exception.attempted}
self.assertEqual(attempted["duckduckgo"]["error"], "no results")
self.assertFalse(attempted["duckduckgo"]["ok"])
for secret in _FAKE_KEYS.values():
self.assertNotIn(secret, str(ctx.exception))
class ExplicitPinTestCase(unittest.TestCase):
def test_explicit_pin_paid_still_works_while_frozen(self):
http = FakeHttp([
("api.tavily.com",
{"results": [{"title": "T", "url": "https://example.com/t",
"content": "tavily content"}]}, 200),
])
out = search("q", provider="tavily", env=_paid_env(), http=http)
self.assertEqual(out["provider"], "tavily")
self.assertEqual(out["results"][0]["snippet"], "tavily content")
def test_explicit_pin_paid_failure_does_not_fall_back(self):
http = FakeHttp([("perplexity", {}, 500),
("api.exa.ai", {"results": [
{"title": "E", "url": "https://example.com/e",
"text": "x"}]}, 200)])
with self.assertRaises(WebSearchError):
search("q", provider="perplexity", env=_paid_env(), http=http)
self.assertEqual(_paid_hosts_seen(http), ["perplexity"])
def test_frozen_default_applies_to_scope_code_too(self):
body = {"results": [
{"title": "GH", "url": "https://github.com/a/b",
"content": "code hit"},
{"title": "Other", "url": "https://example.com/other",
"content": "off-scope"}]}
http = FakeHttp([("searx", body, 200)])
out = search("q", provider="auto", env=_paid_env(),
http=http, scope="code")
self.assertEqual(out["provider"], "searxng")
self.assertEqual(
[r["url"] for r in out["results"]], ["https://github.com/a/b"])
self.assertEqual(_paid_hosts_seen(http), [])
class UnfreezeSwitchTestCase(unittest.TestCase):
def test_switch_off_by_default_and_non_1_values_do_not_unfreeze(self):
self.assertFalse(paid_search_enabled({}))
for value in ("0", "true", "yes", "", "on"):
env = _paid_env(WEBSEARCH_ENABLE_PAID=value)
self.assertFalse(paid_search_enabled(env), value)
self.assertEqual(resolve_provider_chain("auto", env),
["searxng", "duckduckgo"])
def test_switch_on_restores_legacy_chain_and_fallback(self):
env = _paid_env(WEBSEARCH_ENABLE_PAID="1")
self.assertTrue(paid_search_enabled(env))
self.assertEqual(resolve_provider_chain("auto", env),
list(DEFAULT_CHAIN))
# The pre-freeze perplexity -> exa fallback, preserved
# verbatim under the explicit switch (plus the no-leak
# assertion from the old default-chain test).
http = FakeHttp([
("perplexity", {}, 500),
("api.exa.ai", {"results": [
{"title": "Exa Hit", "url": "https://example.com/exa",
"text": "exa snippet", "score": 0.9}]}, 200),
])
out = search("test query", provider="auto", env=env, http=http)
self.assertEqual(out["provider"], "exa")
self.assertEqual(out["results"][0]["url"],
"https://example.com/exa")
self.assertEqual(out["attempted"][0]["provider"], "perplexity")
self.assertFalse(out["attempted"][0]["ok"])
blob = json.dumps(out)
for secret in _FAKE_KEYS.values():
self.assertNotIn(secret, blob)
def test_constants_state_the_contract(self):
self.assertEqual(DEFAULT_FREE_CHAIN, ("searxng", "duckduckgo"))
self.assertEqual(
DEFAULT_CHAIN,
("perplexity", "exa", "tavily", "brave", "searxng",
"duckduckgo"))
class TimeoutClarityTestCase(unittest.TestCase):
def test_http_client_maps_timeout_to_explicit_error(self):
client = HttpClient(timeout=20)
with mock.patch("urllib.request.urlopen",
side_effect=socket.timeout("timed out")):
with self.assertRaises(WebSearchError) as ctx:
client.get("http://searx.test/search?q=x")
self.assertIn("timeout after 20s", str(ctx.exception))
with mock.patch(
"urllib.request.urlopen",
side_effect=urllib.error.URLError(
socket.timeout("timed out"))):
with self.assertRaises(WebSearchError) as ctx:
client.get("http://searx.test/search?q=x")
self.assertIn("timeout after 20s", str(ctx.exception))
def test_http_client_does_not_relabel_dns_failure_as_timeout(self):
client = HttpClient(timeout=20)
with mock.patch(
"urllib.request.urlopen",
side_effect=urllib.error.URLError(
"Temporary failure in name resolution")):
with self.assertRaises(WebSearchError) as ctx:
client.get("http://searx.test/search?q=x")
self.assertNotIn("timeout", str(ctx.exception))
self.assertIn("name resolution", str(ctx.exception))
def test_search_reports_timeout_as_error_not_no_results(self):
class TimeoutHttp(FakeHttp):
def get(self, url, headers=None):
self.seen.append(("GET", url, headers))
if "searx" in url:
raise WebSearchError(
"timeout after 20s waiting for HTTP response")
return self._route(url)
http = TimeoutHttp([("duckduckgo", _DDG_HTML, 200)])
out = search("q", provider="auto", env=_paid_env(), http=http)
self.assertEqual(out["provider"], "duckduckgo")
attempted = {a["provider"]: a for a in out["attempted"]}
self.assertIn("timeout after 20s",
attempted["searxng"]["error"])
self.assertNotEqual(attempted["searxng"]["error"], "no results")
class McpDescriptionTestCase(unittest.TestCase):
def test_tools_list_describes_frozen_default_and_unfreeze(self):
resp = handle_request(
{"jsonrpc": "2.0", "id": 1, "method": "tools/list"})
tool = [t for t in resp["result"]["tools"]
if t["name"] == "web_search"][0]
desc = tool["description"]
self.assertIn("SearXNG", desc)
self.assertIn("DuckDuckGo", desc)
self.assertIn("WEBSEARCH_ENABLE_PAID=1", desc)
self.assertIn("paid", desc.lower())
if __name__ == "__main__":
unittest.main()
+15 -1
View File
@@ -124,10 +124,24 @@ class DockerEnvTestCase(unittest.TestCase):
self.assertIsNone(err)
self.assertEqual(env.get("SEARXNG_URL"), "http://searx.diante.me")
def test_paid_unfreeze_switch_passes_into_container_env(self):
# The freeze switch must reach the MCP process (which reads
# only config.toml env, never worker env); without this
# passthrough, WEBSEARCH_ENABLE_PAID=1 on the host would
# silently leave default auto frozen inside containers.
from unittest import mock
with mock.patch.dict(os.environ,
{"WEBSEARCH_ENABLE_PAID": "1"}):
ok, _, _, env, err = self._docker_context()
self.assertTrue(ok)
self.assertIsNone(err)
self.assertEqual(env.get("WEBSEARCH_ENABLE_PAID"), "1")
def test_no_search_env_means_no_extra_env(self):
from unittest import mock
with mock.patch.dict(os.environ, {}, clear=False):
for k in ("SEARXNG_URL", "PERPLEXITY_API_KEY", "EXA_API_KEY",
for k in ("SEARXNG_URL", "WEBSEARCH_ENABLE_PAID",
"PERPLEXITY_API_KEY", "EXA_API_KEY",
"TAVILY_API_KEY", "BRAVE_API_KEY"):
os.environ.pop(k, None)
ok, _, _, env, err = self._docker_context()
+69 -5
View File
@@ -21,6 +21,16 @@ Design notes (what is copied from oh-my-pi, and what is not):
here is a small stdlib HTML parser: script/style stripped, text
collapsed, long pages truncated head+tail like Hermes' deterministic
budget (default 15000 chars, 75% head / 25% tail, ``[TRUNCATED]``).
- FROZEN default chain (paid freeze): the default ``auto`` chain
contains only the free backends, SearXNG then DuckDuckGo, even
when paid keys happen to be set -- a SearXNG failure never falls
back to a paid backend, so default use can never incur paid cost.
The four paid backends (Perplexity, Exa, Tavily, Brave) keep their
code, tests and docs, and still run when explicitly pinned
(provider="tavily" etc.), which is a deliberate override, not a
default. Only WEBSEARCH_ENABLE_PAID=1 (exact value "1", default
off) restores the old default chain; unfreezing is an explicit act
and may incur paid cost, never a silent default.
Security notes:
- API keys come only from the process environment, never from
@@ -57,6 +67,9 @@ from html.parser import HTMLParser
__all__ = [
"DEFAULT_CHAIN",
"DEFAULT_FREE_CHAIN",
"PAID_PROVIDERS",
"paid_search_enabled",
"CODE_SEARCH_DOMAINS",
"WebSearchError",
"SearchResult",
@@ -72,6 +85,12 @@ __all__ = [
DEFAULT_CHAIN = ("perplexity", "exa", "tavily", "brave", "searxng",
"duckduckgo")
# Paid backends, frozen out of the default auto chain (see the module
# docstring). DEFAULT_CHAIN keeps the legacy full order so the
# unfrozen chain is exactly the old one; DEFAULT_FREE_CHAIN is what
# default auto actually walks while frozen.
PAID_PROVIDERS = ("perplexity", "exa", "tavily", "brave")
DEFAULT_FREE_CHAIN = ("searxng", "duckduckgo")
DEFAULT_FETCH_CHAR_LIMIT = 15000
# Programming-only search scope (``search(..., scope="code")``).
@@ -246,6 +265,27 @@ class HttpClient:
raise WebSearchError(
"HTTP %s" % exc.code) from _ProviderHttpError(
exc.code, body)
except (socket.timeout, TimeoutError) as exc:
# A timeout is a timeout, never "no results": the caller
# (search's attempted entry, named by provider) must be
# able to tell which stage timed out, and a timed-out
# SearXNG must not be rewritten as an empty answer.
# HTTPError is a URLError subclass, so this branch must
# stay after the HTTPError branch above.
raise WebSearchError(
"timeout after %ss waiting for HTTP response"
% self.timeout) from exc
except urllib.error.URLError as exc:
reason = getattr(exc, "reason", None)
if isinstance(reason, (socket.timeout, TimeoutError)):
raise WebSearchError(
"timeout after %ss waiting for HTTP response"
% self.timeout) from exc
# DNS / connection failures keep their own message;
# they are not timeouts and must not be relabelled.
raise WebSearchError(
"URL error: %s" % reason if reason else "URL error"
) from exc
def get_text(self, url, headers=None):
status, body, _ctype = self.get(url, headers)
@@ -534,11 +574,32 @@ _PROVIDERS = {
}
def available_providers(env=None):
"""Providers usable right now: keyed ones with a key, plus free ones."""
def paid_search_enabled(env=None):
"""True only when the paid-unfreeze switch is explicitly on.
The switch is WEBSEARCH_ENABLE_PAID, and only the exact value
"1" counts (same strict convention as WEBSEARCH_ALLOW_PRIVATE_URLS
elsewhere in this module): any other value, including "true" or
"yes", leaves the paid backends frozen out of default auto.
Explicit pinning is unaffected by this switch either way.
"""
env = env if env is not None else os.environ
return env.get("WEBSEARCH_ENABLE_PAID") == "1"
def available_providers(env=None):
"""Providers usable right now, under the paid-freeze rule.
Frozen (default): only the free chain -- SearXNG when its URL
is set, plus DuckDuckGo -- even when paid keys are present.
Unfrozen (WEBSEARCH_ENABLE_PAID=1): the legacy full chain,
keyed backends only when their key is set, as before.
"""
env = env if env is not None else os.environ
chain = (DEFAULT_CHAIN if paid_search_enabled(env)
else DEFAULT_FREE_CHAIN)
out = []
for name in DEFAULT_CHAIN:
for name in chain:
if name in _FREE_PROVIDERS:
if name == "searxng" and not env.get("SEARXNG_URL"):
continue
@@ -552,8 +613,11 @@ def resolve_provider_chain(provider="auto", env=None):
"""Resolve the chain oh-my-pi style: auto walks, pinned does not.
``provider`` may be "auto" (default chain, filtered to what is
configured), a single name (that provider only, even if it fails),
or a comma-separated list (that order only). Unknown names raise
configured -- frozen by default to the free chain, SearXNG then
DuckDuckGo, unless WEBSEARCH_ENABLE_PAID=1 restores the legacy
full chain), a single name (that provider only, even if it fails
-- pinning a paid backend bypasses the freeze, deliberately), or
a comma-separated list (that order only). Unknown names raise
ValueError before any HTTP happens.
"""
env = env if env is not None else os.environ
+13 -3
View File
@@ -35,9 +35,19 @@ _TOOLS = [
"Search the web and return ranked results with title, URL "
"and snippet, plus a synthesised answer when the provider "
"supplies one. Use web_fetch to read specific pages. "
"provider may be 'auto' (walk Perplexity, Exa, Tavily, "
"Brave, then keyless SearXNG / DuckDuckGo, falling back "
"on failure) or a single provider name to pin it. "
"provider may be 'auto' (frozen default: keyless SearXNG "
"first, then DuckDuckGo as the free fallback, falling "
"back on failure; the paid backends Perplexity, Exa, "
"Tavily and Brave are NOT in the default chain even "
"when their keys are set, and a SearXNG failure never "
"falls back to a paid backend, so default use never "
"incurs paid cost) or a single provider name to pin it "
"(pinning a paid backend still calls it: that is an "
"explicit override, not a default). Set "
"WEBSEARCH_ENABLE_PAID=1 to restore the old default "
"chain (Perplexity, Exa, Tavily, Brave, SearXNG, "
"DuckDuckGo); unfreezing is explicit and may incur "
"paid cost. "
"scope 'code' restricts results to programming sources "
"only (official language docs, GitHub, package "
"registries, Stack Overflow, CVE/NVD); anything else "
+63
View File
@@ -3188,9 +3188,15 @@ class Worker:
# auto 链会跳过 SearXNG、只剩 DuckDuckGo(实弹 e2e 已证实)。
# 只透传已设置的项,值绝不进日志;付费 key 与网关 key 一样
# 走 extra_env(容器内进程本就需要它调 API)。
# WEBSEARCH_ENABLE_PAID rides along so the paid-freeze
# switch set on the host actually reaches the MCP process
# (which reads only config.toml env, never worker env).
# Default is frozen; the switch is honoured only when set
# to exactly "1" (see websearch.paid_search_enabled).
search_env = {
k: os.environ[k]
for k in ("SEARXNG_URL", "SEARXNG_ENGINES",
"WEBSEARCH_ENABLE_PAID",
"PERPLEXITY_API_KEY", "EXA_API_KEY",
"TAVILY_API_KEY", "BRAVE_API_KEY")
if os.environ.get(k)
@@ -6576,6 +6582,59 @@ class Worker:
log_event(self.logger, "worker_push_due_failed",
message="push due processing failed: %s" % exc)
def process_due_resident_watches(self, now=None, fetcher=None,
resolver=None):
"""常驻监听驱动(worker 轮次边界调用):到期监听逐条
走 resident.process_due_resident_watches,抢单 + 廉价
抓取 + 仅变化建任务。全部异常只记日志,不炸循环;
fetcher 缺省用 resident 默认抓取,单测可经
self.resident_fetcher 或参数注入假抓取;resolver 同款
注入点(self.resident_resolver),生产路径为 None 即
默认 DNS 门,测试用它把测试主机名指到本地服务。
建表只在 worker 生命周期内做一次(首批):热路径
不再逐条 ensure,见 resident.ensure 的调用约定;
批量内逐条隔离的异常与预算耗尽信号记
worker_resident_watch_failed 日志,不炸整批。"""
if now is None:
now = time.time()
if fetcher is None:
fetcher = getattr(self, "resident_fetcher", None)
if resolver is None:
resolver = getattr(self, "resident_resolver", None)
try:
from cloud.resident import ensure_resident_tables as _ensure
from cloud.resident import process_due_resident_watches as _run
conn = _db_connect(self.db_path)
try:
if not getattr(self, "_resident_tables_ensured", False):
try:
_ensure(conn)
conn.commit()
self._resident_tables_ensured = True
except Exception as exc:
log_event(
self.logger, "worker_resident_watch_failed",
message="resident ensure failed: %s" % exc)
errors = []
polls = _run(conn, fetcher=fetcher, now=now,
resolver=resolver, errors=errors)
for watch_id, msg in errors:
log_event(
self.logger, "worker_resident_watch_failed",
message="resident watch item failed: %s: %s"
% (watch_id, msg))
finally:
conn.close()
except Exception as exc:
log_event(self.logger, "worker_resident_watch_failed",
message="resident watch processing failed: %s" % exc)
return []
for poll in polls:
log_event(self.logger, "worker_resident_watch_poll",
message="resident watch poll: %s %s"
% (poll["watch_id"], poll["status"]))
return polls
def run_loop(self, interval=300, max_seconds=None, stop_event=None,
drain_timeout=60, sweep_threshold=300):
"""常驻循环:每 interval 秒跑一轮 run_all,空闲也不退出。
@@ -6658,6 +6717,10 @@ class Worker:
# G11/G12 后台投递:摘要(每小时)与 outbox 重试,
# 轮次边界做,不中断在跑任务,失败只记日志。
self._process_push_due()
# 常驻监听驱动:同款轮次边界后台扫,到期监听
# 廉价抓取、仅变化建任务;服务端本身不起定时器,
# 这里就是常驻任务的实际驱动(见 resident 模块)。
self.process_due_resident_watches()
# 自动记忆提取:同款轮次边界后台扫,失败只记日志
# (process_auto_memories 内部已兜底,不炸循环)。
self.process_auto_memories()
+82 -27
View File
@@ -20,6 +20,11 @@ pi 的工具),只复用它的模式:统一工具 + provider 抽象 +
- `cloud/tests/test_code_search_ext.py`:任务级
`allowed_hosts` 扩展的单测(指定 / 未指定 / 点边界 /
空值与非法值 / Perplexity 超 20 域 / MCP 透传)。
- `cloud/tests/test_websearch_paid_freeze.py`:付费冻结单测
(默认 auto 无付费后端、SearXNG 失败不回落付费、显式
pin 仍工作、解冻开关关时不变 / 开时恢复旧链、
DuckDuckGo 兜底失败报错清晰、超时报错清晰;全部
mock HTTP,不联网、不需要真实 key)。
- `cloud/tests/test_js_heavy_extract.py`:JS 重型页面
抽取评估的 mock 单测(直接抓取壳-only、provider 抽取
补与不补、回退语义;live SearXNG 烟囱默认跳过)。
@@ -39,7 +44,48 @@ pi 的工具),只复用它的模式:统一工具 + provider 抽象 +
超长按头 75% + 尾 25% 截断并标 `[TRUNCATED]`;只接受 http(s),拒绝
localhost 与字面私有 IP;抽取与直接抓取都失败时报错,不伪造页面。
## Provider 链(auto 顺序)
## Provider 链:默认冻结,显式 pin 与显式解冻除外
冻结语义(本轨起,默认 `auto` 只走免费路线,顺序固定):
1. `searxng`:`GET $SEARXNG_URL/search?format=json`,需自托管
实例且 JSON 格式已打开。默认参数:`engines=bing,360search`
(2026-10-30 主机实测:默认 general 类别 20.0s 撞超时,
pinned 两引擎 0.47s;`SEARXNG_ENGINES` 可覆盖,空值恢复
不 pin)。`SEARXNG_URL` 在代码里无默认值:未配时该后端
直接报 `searxng not configured (SEARXNG_URL unset)`,链
只剩 DuckDuckGo;生产值 `http://searx.diante.me` 来自主
机 env(见“生产接线”),不在代码里写死;主机 env 实测
仅有 `SEARXNG_URL` 这一行、无 `SEARXNG_ENGINES` 行,
所以默认 pin(bing,360search)生效。注意是 http 不是
https:主机 nginx 只听 80、无 443 vhost,https 会在 TLS
层证书主机名不匹配失败,根本到不了 SearXNG。
2. `duckduckgo`:解析 `https://html.duckduckgo.com/html/`,
免 key 兜底,布局漂移时只会返回空结果并继续走链,
不会把工具打崩;但它在两种出口下都不可作可用兜底:
本 VM 直连回 202 + 机器人挑战页、0 条,主机 Tinyproxy
路径约 25 秒超时。兜底失败时记为 `no results` / 超时并
如实报错,不伪造结果;因此 SearXNG 挂时冻结默认链可能
以诚实失败告终,这是已知边界,不是静默降级。
冻结保证:即使环境里配了付费 key,默认 auto 也绝不调用
Perplexity / Exa / Tavily / Brave、绝不产生付费花费;
SearXNG 失败后默认也不回落到付费后端,只走 DuckDuckGo。
付费后端的代码、测试、文档一律保留,未删除。
显式 pin 仍工作:`provider="tavily"` / `"exa"` /
`"perplexity"` / `"brave"` 显式指定时照常调用,这是用户
有意的覆盖,不是默认行为;pin 失败不回退(与冻结前一致)。
解冻开关:`WEBSEARCH_ENABLE_PAID=1`(且仅精确值 `1`,
`true` / `yes` / `0` 都不算开,默认关)时,默认 auto
恢复旧链,顺序为 Perplexity → Exa → Tavily → Brave →
SearXNG → DuckDuckGo(旧链里只试有 key 的付费后端 +
已配 URL 的 SearXNG + DuckDuckGo)。解冻是显式动作,
会产生付费花费,绝不能被默认链静默触发;开关走 worker
环境透传进 MCP config env(见“生产接线”)。
旧链各后端(解冻后 / 显式 pin 时)的请求形态不变:
1. `perplexity`:`POST https://api.perplexity.ai/chat/completions`,
模型 `sonar`,`Authorization: Bearer $PERPLEXITY_API_KEY`。
@@ -53,18 +99,16 @@ pi 的工具),只复用它的模式:统一工具 + provider 抽象 +
结果,总结留给模型)。
4. `brave`:`GET https://api.search.brave.com/res/v1/web/search`,
头 `X-Subscription-Token`,`extra_snippets: true`。
5. `searxng`:`GET $SEARXNG_URL/search?format=json`,需自托管
实例且 JSON 格式已打开。
6. `duckduckgo`:解析 `https://html.duckduckgo.com/html/`,
免 key 兜底,布局漂移时只会返回空结果并继续走链,
不会把工具打崩。
规则:auto 只试已配置的后端(有 key 的付费后端 + 已配
URL 的 SearXNG + DuckDuckGo);任一后端 HTTP 失败或
0 结果都继续下一个;全部失败返回各后端错误(不含 key)。
`provider` pin 单个名字时不回退,失败即报错。
可用 `WEBSEARCH_PROVIDERS`(逗号列表)改默认链顺序,
`WEBSEARCH_TIMEOUT`(秒,默认 20)改单次超时。
规则:任一后端 HTTP 失败或 0 结果都继续下一个;全部失败
返回各后端错误(不含 key)。超时失败报
`timeout after Ns waiting for HTTP response`(N 为
`WEBSEARCH_TIMEOUT`,默认 20 秒),并记在该后端的
attempted 条目里,绝不把超时写成“无结果”或空答案。
`WEBSEARCH_TIMEOUT`(秒,默认 20)改单次超时。另注意:
旧文档曾写可用 `WEBSEARCH_PROVIDERS` 改默认链顺序,
代码里从未实现该变量,本轨已删此说法(不另新加该变量,
保持改动最小)。
## 接入 codex(测试项目做法)
@@ -97,15 +141,19 @@ Docker 租户必须传容器内路径
- egress 白名单:`deploy/egress/tinyproxy-filter` 已加
Perplexity、Exa、Tavily、Brave、DuckDuckGo 与
`searx.diante.me`(精确匹配,不用 diante.me 通配,否则
任务经代理也能摸到 api.diante.me 控制面)。主机上须把
任务经代理也能摸到 api.diante.me 控制面;filter 行即
`^searx\.diante\.me$`,本轨不改这行)。主机上须把
该文件拷到 `/etc/tinyproxy/tinyproxy-filter` 并重载
tinyproxy,重载后先验中转域名还通,再验新域名。
- 自托管 SearXNG(公开子域名路):主机 Docker 跑
`searxng/searxng`,经 nginx 反代到 `searx.diante.me`
(DNS A 记录 + certbot 证书),MCP 侧
`SEARXNG_URL=https://searx.diante.me` 走 worker 进程
`searxng/searxng`,经 nginx(仅听 80,无 443 vhost、无
证书)反代到 `searx.diante.me`,MCP 侧
`SEARXNG_URL=http://searx.diante.me` 走 worker 进程
环境(`/etc/codex-app-server/env`)继承到 codex 再到
MCP,不写进 config.toml。公开实例的代价是可能被外人
MCP,不写进 config.toml。`WEBSEARCH_ENABLE_PAID` 同理
走 worker 的 search_env 透传进 `[mcp_servers.websearch.env]`
(worker 代码已含该键);主机侧改 env 文件后需重启相关
unit 才生效——那是部署动作,不在本轨。公开实例的代价是可能被外人
刷:SearXNG limiter 保持开启,JSON 格式是 MCP 要用的
唯一格式。未加域名进白名单之前,Docker 任务的搜索
会被代理 403 挡住(fail closed,工具报错,不会静默绕过)。
@@ -267,6 +315,10 @@ filter 行须用 POSIX BASIC 正则(`^\(.*\.\)*域名$`)、
- 本批只做代码 + 单测 + 测试租户目录的配置合并验证,
未推分支、未建 PR、未合并、未部署、未改生产 worker
与 egress 配置、未跑真实 codex 任务。
- 付费冻结本轨同样只在本地提交里:生产默认链要等部署
新代码后才变;解冻开关在主机上默认仍是关(除非主机
env 显式设为 `WEBSEARCH_ENABLE_PAID=1`,且那需要
主机侧改 env + 重启 unit,属部署动作)。
- DuckDuckGo 的 HTML 解析是兜底手段,布局一变就可能
0 结果;它存在的意义是“没 key 也能走通链路”,不是
长期依赖的搜索后端。实测(2026-10-28,VM 出口):
@@ -335,16 +387,19 @@ mock 补不了、也证明不了的部分:
### Live 对照(与 mock 分开写,未验证的不写成已验证)
- 无 key 的 SearXNG 路线:本轨在 VM 测试环境实测一次
(`SEARXNG_URL=https://searx.diante.me`,`provider="searxng"`,
产品自身 `search()` 代码):失败,`URLError: Remote end
closed connection without response`。这是在本 VM 出口下
的一次结果,不能据此断言 SearXNG 服务本身挂了(生产
文档记载主机侧经代理 pinned 引擎曾 0.47s 通),live
对照在本测试环境记为**未完成**,需在主机侧可达的环境
重测;测试文件里的 live 烟囱测试默认跳过,只在
`WEBSEARCH_EVAL_LIVE_SEARXNG=1` 且配了 `SEARXNG_URL`
时才跑。
- 无 key 的 SearXNG 路线:旧 VM 测试曾用
`SEARXNG_URL=https://searx.diante.me` 失败(`URLError:
Remote end closed connection without response`)——该失败
现已归因:https 是错的,主机 nginx 只听 80、无 443
vhost,https 在 TLS 层证书主机名不匹配失败,根本没到
SearXNG,不能据此断言服务挂了。2026-11-01 探针实测:
http 经同一 Tinyproxy 0.34 秒回 200、16 条结果,服务
活着;主机 env 仅 `SEARXNG_URL=http://searx.diante.me`
一行、无 `SEARXNG_ENGINES`,默认 pin 生效。容器内真实
任务派发仍是 blocker(不由本轨创建生产任务),live
对照在该项上仍记为**未完成**;测试文件里的 live 烟囱
测试默认跳过,只在 `WEBSEARCH_EVAL_LIVE_SEARXNG=1`
且配了 `SEARXNG_URL` 时才跑。
- 付费后端 live 效果(Tavily extract / Exa contents /
Perplexity / Brave 的真实 JS 渲染能力):无 key,本轨
**一律未验证**,不编造效果,逐项列为 blocker(见回执),
+201
View File
@@ -0,0 +1,201 @@
# 常驻监听网站闭环:独立架构与代码质量审核报告
审核员非实现者。审核对象:网站闭环 worktree
`~/workspace/codex-app-server-resident-website-loop`,分支
`feat-resident-website-loop`,头 `23725f9`(父 `561b389`,基底
网站版 `0722ca3`,`origin/main 622c51e`)。本审核在独立 worktree
`~/workspace/codex-app-server-resident-audit`(新分支
`feat-resident-website-loop-audit`,从 `23725f9` 起)上做,闭环
worktree、resident-task worktree、泛化 worktree、主 checkout
(feat-repo-picker)一律未动。
## 0. 评分标准(先定标准,再打分)
只评架构与代码质量两项,满分各 10 分。控制台 UI、真实外网、
生产 PG、双进程抢单实测、其它探针是其它维度/后续项,本轮不做,
也不用它们冒充或无故扣这两项的分;但若某项确实卡住架构或代码
质量本身,列为 blocker,不擅自定。
### 架构 10 分标准
- A1 分层干净:resident 模块自有状态机,与任务状态机、自动化
定义层互不干扰;server 只做路由/权限/审计,worker 只做驱动
边界调用;无循环依赖。
- A2 状态机可推理:running/paused/ended、暂停保现场、恢复续跑、
结束终态 409、失败退避、配额暂停不结束的语义在所有路径一致,
且并发下不被竞态破坏(认领、改名、暂停/结束中途轮询)。
- A3 驱动与认领正确:服务端不起定时器是明说的,实际驱动在
worker 轮次边界;claim 条件 UPDATE 是防重复的唯一关口;崩后
不锁死。
- A4 快照/变化语义干净:轮询廉价、仅变化建任务、首次只建基线、
每次轮询可查;变化摘录与游标推进在同一事务边界内。
- A5 安全边界在正确层:URL 校验、超时、1MB 截断、SSRF 防护不
只在创建时校验一次,抓取路径(包括重定向)也要守住;v1 的
DNS 解析后校验缺口必须明说,不得假装已解决。
- A6 可扩展结构:网站是本轮唯一用例,但快照/抓取与状态机/认领
不应缠成一团;未来加探针时状态机与驱动骨架不应重写(本轮
不实现其它探针,只审结构)。
### 代码质量 10 分标准
- Q1 命名、函数长度与复杂度可读;长函数(run_due)分支清晰,
不靠注释掩盖逻辑。
- Q2 无重复、无死代码、无测试值渗入生产代码;复用已有口径
(自动化 claim/配额/审计、环境名唯一索引先例)而不是另起一套。
- Q3 错误处理诚实:失败、配额耗尽、配额检查失败三条路不混淆;
审计与轮询记录不静默丢失;异常不炸驱动循环。
- Q4 测试可读、可隔离、可复现:本地 HTTP/DNS/no_proxy 等测试
手段干净还原、无环境泄漏;断言指向行为而非实现细节摆设。
- Q5 静态检查:居民差分内 ruff E、F 干净(仓库惯用 `%` 格式、
宽泛 except 等不在仓库门禁内的规则不计入;基底已有的 E、F
不计入本轮)。
## 1. 改动前分数
- 架构:9.0 / 10
- 代码质量:9.0 / 10
基底是好的:分层、状态机、claim、配额复用、轮询可查、测试
隔离(DNS/no_proxy 在 loop 测试里干净还原)都属实,定向 15 项
基线全过(见第 4 节)。扣分只来自下面逐项 findings,不是整体
推倒重来。
## 2. Findings 清单(改动前)
### 严重
- F1 名称唯一性是先查后插、无数据库约束:create_watch /
update_watch 先 SELECT 查重再 INSERT/UPDATE,两个并发请求
可同时通过检查、插出同项目同名 running 监听;server 的
environments 已有先例:`CREATE UNIQUE INDEX … WHERE status
!= 'archived'`。本轮用同款部分唯一索引
`WHERE status != 'ended'`(已结束名可重用)+ 完整性错误转
ValueError 修复。
- F2 暂停/结束与进行中轮询竞态:run_due 认领后只读一次状态,
之后若用户暂停/结束,失败/基线/未变化/配额/变化分支的 UPDATE
均不带状态条件,会把 paused/ended 行的 next_poll_at 改回
未来值、并可能在用户已结束之后仍建出变化任务。修复:认领后
复查状态;全部终局 UPDATE 带 `status='running'` 条件,条件
不满足时回滚、不建任务、不记轮询。
### 一般
- F3 抓取路径 SSRF 只守了创建时:default_fetch 用默认 opener,
重定向目标不校验,公开 URL 重定向到内网地址可绕过创建校验;
另 localhost.、单整数/十六进制/前导零等非规范 IP 写法可绕过
字面 IP 检查。修复:重定向处理器逐跳校验(相对跳转先 urljoin
再校验);校验时归一化主机(去尾点、*.localhost 挡掉)并拒绝
非规范数字 IP 写法。DNS 解析后校验仍属 v1 缺口,继续列为已知
偏差,不假装已解决。
- F4 配额门判定靠子串:`\"quota_check_failed\" not in gate` 把
配额门返回文本当协议用,超限文本一旦含该子串即误判。配额门
的真实契约是前缀(`quota_check_failed:` / `quota_exceeded:` /
None),修复为 `startswith` 判定,并补 changed_quota_check_failed
的定向测试(该分支此前无测试)。
- F5 row_get 重复造轮:resident 自带与 automations.row_get 逐字
同语义的 row_get,却已从 automations 导入另外两个函数。修复:
直接导入 automations.row_get,删本地定义。
- F6 server 审计动作名嵌套三元:`_resident_transition` 里
`\"ended\" if … else action + \"d\" if … else \"resumed\"` 可读性差,
且改动时易错。修复为显式动作映射字典,审计字符串不变。
### 建议
- S1 test_resident.py 导入了未用的 `time`(ruff F401,居民差分
内唯一 E、F 问题)。删除。
- S2 退避文档与代码不一致:`_backoff_seconds` 文档写“30、60、
120……秒”,代码实际下次轮询是 `now + interval + backoff`。
行为本身合理(不比间隔更频繁),文档误导。修复文档,行为不动。
- S3 loop 测试 tearDown 不够防御:setUp 中途失败时 tearDown
访问未建属性会掩盖原始失败。修复为 getattr 防御。
### 已核查、不是问题(不扣分)
- 审计行“写在 commit 之后会丢”:核查 `_auto_audit_row` 本身
执行 INSERT 后自己 commit,失败/配额暂停/变化三处审计行在
模拟关闭连接后仍可查,不是缺陷。
- server/worker 全量 ruff E、F 的其余项(如 worker.py:2781/2787
的 F821、server.py:10293 的 E501)经核对均为基底已有或由居民
插入导致的行号位移,非居民新增代码,不计入本轮。
- 仓库默认 ruff 全规则的 39 项(如 UP031 `%` 格式)与仓库惯用
写法冲突,不在仓库门禁内,不计入。
## 3. 修复(全部关闭,每项有对应测试或定向复核)
- F1 名称唯一性:`ensure_resident_tables` 新增部分唯一索引
`idx_resident_watches_name ON resident_watches(project_id, name)
WHERE status != 'ended'`(server environments 同款口径);
create/update 的 INSERT/UPDATE 包完整性错误转 ValueError
(sqlite3.IntegrityError / PostgresIntegrityError,调用方仍
返回 400)。测试:索引存在断言、绕过查重直接插报
IntegrityError、改名撞名报 ValueError。
- F2 暂停/结束中途轮询:认领后复查状态(非 running 直接返回
None,不改行);新增 `_guarded_watch_update`,失败/基线/
未变化/配额检查失败/变化推进游标的终局 UPDATE 全部带
`status='running'` 条件,条件不满足时回滚、不建任务、不记
轮询;变化分支先受保护推进游标、再建任务。测试:抓取回调
里经第二连接暂停监听,run_due 返回 None、无任务、游标与
next_poll_at 保持暂停态、轮询数不变。
- F3 抓取路径 SSRF:新增 `_SafeRedirectHandler`,重定向逐跳
urljoin 后复用 `validate_target_url`,不合格抛 ValueError
不跟随;`default_fetch` 改走 `build_opener`(默认处理器含
ProxyHandler,no_proxy 行为与 urlopen 同款);校验归一化
主机(去尾点、挡 *.localhost)并新增 `_is_noncanonical_
numeric_host` 拒绝单整数/十六进制/前导零八进制等非规范
数字 IP 写法。测试:localhost.、*.localhost、2130706433、
0x7f000001、0177.0.0.1、0x7f.0.0.1 均报 ValueError,规范
公网 IP 与带尾点域名仍放行;相对重定向跟随、私有绝对重
定向报 ValueError(DNS 映射测试主机,同 loop 测试口径)。
- F4 配额门判定:`\"quota_check_failed\" not in gate` 改为
`gate.startswith(\"quota_check_failed\")`,与配额门前缀契约
对齐。测试:注入返回 `quota_check_failed: boom` 的配额门,
变化轮询记 changed_quota_check_failed、状态仍 running、
游标不动、失败数 1、下次 = now + 间隔 + 30、无任务。
- F5 row_get 重复:删 resident 本地定义,直接导入
automations.row_get,内部调用点一字未动,模块依赖说明
同步更新。
- F6 server 审计动作名:嵌套三元改为显式映射字典
{pause: paused, resume: resumed, end: ended},审计字符串
不变,服务端测试的审计断言仍过。
- S1 删 test_resident.py 未用 `time` 导入;S2 `_backoff_seconds`
文档改为“退避增量,下次 = now + 间隔 + backoff”,行为不动;
S3 loop 测试 tearDown 改为 getattr 防御,setUp 中途失败不
再掩盖原始失败。
## 4. 测试数字(全部本轮实测,无删断言、无 skip、无改默认值)
- 定向基线(改动前):`cloud.tests.test_resident` +
`test_resident_server` + `test_resident_website_loop` = 15 项全过。
- 定向改动后:同三文件 = 20 项全过(基线 15 + 新增 5:URL
变体、唯一索引、配额检查失败、中途暂停、重定向),4.450 秒;
ruff E、F 在居民差分 4 文件上全部通过。
- 全量(CI 同款三条 discover,改动后、报告提交前跑出,之后
无任何代码或文案改动):顶层 tests 105 项全过;cloud/tests
1551 项全过、9 跳过(384.252 秒);deploy/tests 44 项全过
(单独重跑确认);合计 1700 项、0 失败、9 跳过。对账:闭环
基线 1695(105 + 1546 + 44)+ 新增 5 = 1700,无测试丢失、
无新失败。
- 覆盖率:本仓库 CI 无覆盖率门禁,不虚构覆盖率变化;新增
5 项定向测试即上述全量的一部分。
## 5. 复评分数
- 架构:10 / 10
- 代码质量:10 / 10
复评依据:F1/F2 两项严重 findings 已由数据库约束与状态条件
UPDATE 从机制上关闭(非仅文档说明),并有定向测试锁定;
F3 安全边界已从“创建时一次校验”补到“抓取重定向逐跳校验”
与非规范 IP 写法;F4/F5/F6/S1/S2/S3 全部关闭且测试全过。
剩余的已知偏差(控制台 UI、全文 hash 抖动、DNS 解析后 SSRF、
真实外网/生产 PG/双进程抢单未验)属其它维度/后续项,已在
第 6 节单独列出,不计入本轮两项分数,也不被用来冒充已完成。
## 6. 剩余 blocker 与未做事项
- 未做:未 push、未合并、未跑 deploy.py、未部署;不做其它探针、
不接控制台 UI、不做生产 PG 与双进程抢单实测。
- 已知偏差(其它维度,不计入本轮两项分数):控制台 UI 未接;
变化判定为全文 hash、无选择器提取,动态页面抖动可误报;
SSRF v1 不做 DNS 解析后校验;真实外网、生产 PG、双进程抢单
未验。
+236
View File
@@ -0,0 +1,236 @@
# 常驻监听 P0+P1 修复批验证摘要(本地实现与测试,未 push、未开 PR、未合并、未部署)
- 起点:`feat-resident-p0-fixes`,从 `origin/main 60997b1`(PR #275 合并头)起;
本批只改本 worktree,绝不动生产审核 worktree、闭环 worktree、主 checkout。
- 审核依据:生产审核报告
`~/workspace/codex-app-server-resident-prod-audit/verification/resident-prod-audit-summary.md`
第 4、5、7 节(动手前已重读);范围只限该报告的 P0 与 P1。
- 不做事项:控制台 UI、选择器/区域提取、其它探针
(server_status/agent_status/relay_cost)、磁盘清理、真实外网抓取
(真实抓取只用本地测试 HTTP 服务验证)、生产 PG、生产双 worker、
真实外网只读演练(仍是需用户拍板的后续窗口,见 blocker)。
- 红线:以下所有数值凡标“拟定默认值”者,均为本批实现工拟定的
默认值、供最终验收改,不是用户已拍板;摘要、代码注释与后续
PR 描述素材里一律按此口径写,不得写成用户已定。
## 1. 拟定默认值清单(逐项标拟定,环境变量可覆盖)
| 口径 | 拟定默认值 | 环境变量 | 说明 |
| --- | --- | --- | --- |
| 每监听每日变化任务上限 | 24 个 | `CODEX_RESIDENT_MAX_DAILY_CHANGE_TASKS` | 按 UTC 日切分;只数 `status='changed'` 且 `task_id` 非空的轮询行;超限记 `changed_capped`,游标不推进 |
| 变化抖动冷却 | 300 秒 | `CODEX_RESIDENT_CHANGE_COOLDOWN_SECONDS` | 上次建任务 300 秒内的新变化记 `changed_cooldown`,游标不推进,冷却过后复查仍可报 |
| resident_polls 保留期 | 90 天 | `CODEX_RESIDENT_POLL_RETENTION_DAYS` | 批量驱动每批开头 prune 一次;过期删、未过期保留;当日上限计数不受影响(保留期远大于 1 天) |
| 批量总预算 | 120 秒 | `CODEX_RESIDENT_BATCH_BUDGET_SECONDS` | 单批最坏 = 预算 + 单条超时上限 60 秒;超预算的剩余监听保持 due,下批再跑 |
| 代理模式 | 直连(默认) | `CODEX_RESIDENT_PROXY` | 未设/空/`direct` = 直连,不跟随进程代理;`env` = 显式跟随进程环境代理;其它值 = 显式代理 URL;两种代理模式都认进程 `no_proxy`/`NO_PROXY` |
| 重定向上限 | 10 跳 | (固定值) | 超过抛 `redirect` 结构化失败 |
其它沿用旧值不变:间隔下限 60 秒、抓取超时 1–60 秒(默认 10 秒)、
1MB 截断、变化摘录 2000 字、poll detail 500 字、批量单批上限 50 条。
## 2. P0 修复与测试证据
### S1. DNS 级 SSRF + 摘录口径(cloud/resident.py)
- 抓取前解析全部 A/AAAA(`resolve_public_addresses`):任一地址命中
私有/回环/链路本地/保留/组播/未指定即整批拒绝;IPv6 的 IPv4 映射
地址(`::ffff:127.0.0.1`)按映射后 IPv4 再判;解析失败/空结果/
非 IP 结果按 `dns` 失败,不放行。
- 直连建连 pin 住解析结果(`_PinnedHTTPConnection` /
`_PinnedHTTPSConnection`):TCP 对端就是解析时刻的 IP,不再二次
按主机名解析;https 的 SNI 与证书主机名仍是原始主机名(单测断言
`wrap_socket(server_hostname='example.com')` 且建连地址为 pin 的 IP),
TLS 校验不被 pin 破坏。
- 重定向每跳重复字面校验 + 解析门 + pin(相对跳转先 urljoin);
非 200 一律抛 `HttpStatusFetchError`,错误页正文直接丢弃。
- 摘录口径(拟定):摘录只来自已通过校验的公开抓取结果;私网
抓取在抓取层直接失败(`ssrf_blocked`)、不产生摘录、不建任务、
轮询 detail 只含 `reason=ssrf_blocked; …` 异常文本,绝不含抓取正文。
创建监听时仍只做字面校验(DNS 会变,创建时放行不等于抓取时放行),
权威关口在抓取层——见代码注释。
- 残留风险(诚实写明,不假装已封口):
- 代理模式下客户端解析门只是前置检查,真正建连的是代理方、由
代理方二次解析;pin 在代理模式下不生效,须靠代理自身的 egress
allowlist 兜底。生产部署若用代理模式,必须同时给代理配内网拦截。
- 直连 pin 已把解析-建连 TOCTOU 关到“解析结果本身”这一层,但
“解析时刻的 IP 在之后被重新分配给内网用途”的极端情形不在
本次封口承诺内。
- 旧 urllib 默认跟随进程代理的行为已改:默认直连。部署时若宿主
出口必须走代理,须显式配 `CODEX_RESIDENT_PROXY=env` 或代理 URL,
否则公网抓取会直连失败(记失败轮询,不建任务)。
- 测试:`test_resolve_public_rejects_private_and_mixed`(私网/混合/
映射地址/解析失败)、`test_ssrf_blocked_poll_has_no_excerpt_and_no_task`、
`test_pinned_https_uses_hostname_for_sni_not_ip`、
旧重定向测试改写(默认解析器 + getaddrinfo 补丁必拦;注入 resolver
返回公网测试 IP + socket 层重定向到本地服务时相对跳转走通)。
### S2. 驱动与延迟:选批量总预算,不新增独立驱动进程
- 论证:独立驱动进程的运维成本(新 unit、独立 DB 连接、启停监管)
是审核第 7 节留给用户拍板的项,本批不擅自新增;批量总预算是纯
代码改动、本地可测,且 claim 门已保证未来加驱动时多驱动不重复
轮询。预算把旧最坏 `50×60=3000` 秒 bound 到“预算 120 秒 + 单条
超时上限 60 秒”,剩余到期监听保持 due、下批再跑,不丢。
- 间隔只是下限、不是 SLA:`resident_metrics` 的 `due_count` /
`max_overdue_seconds` 直接度量“到期未跑”积压,进 `/metrics`;
任何对外口径仍须写“尽力而为、延迟无界”(模型任务一轮最长
1800 秒仍会拖后轮询,预算只 bound 单批,不承诺送达时限)。
- 测试:`test_batch_budget_stops_batch_and_keeps_rest_due`
(单调时钟注入:50 秒/条、预算 100 秒时跑 2 条即停,剩余 3 条仍 due)。
### G2. 成本上限与保留期
- 每日上限、抖动冷却、保留期见第 1 节清单;三者均持久化
(上限数当日轮询行、冷却用既有 `last_change_at`、保留期靠
`prune_old_polls` 的 DELETE),重启不丢计数。
- 页面每轮都变时:冷却期内只记 `changed_cooldown`、上限满后只记
`changed_capped`,游标不推进(变化不丢,冷却过后/次日仍可报),
任务数有界。
- 测试:`test_daily_cap_and_cooldown_suppress_without_losing_change`
(上限 2、冷却 300 秒的加速版:建任务→冷却抑制→冷却后建任务→
上限抑制,游标始终指向最后一次真实建任务的内容)、
`test_prune_old_polls_keeps_recent`(100 天前行删、当前行保留)。
## 3. P1 修复与测试证据
### S3. resident 表进 PG schema 与迁移清单
- `cloud/db.py` 的 `POSTGRES_SCHEMA` 新增 `resident_watches` /
`resident_polls`(含 `failure_reason` 列)与三条索引(含部分唯一索引
`WHERE status != 'ended'`);`cloud/migrations.py` 的
`MIGRATION_TABLES` 同步新增两表(TEXT 主键,无身份列,无需序列拨动)。
- 测试:`test_migration_coverage_includes_resident_tables`
(`migration_coverage()["missing_from_migration"] == ()`);
本地 PG 实测见第 5 节。
### G1. ensure 移出热路径 + 批量逐条隔离 + 死锁重试一次
- 热路径(`due_watches` / `claim_due_watch` / `run_due` /
批量循环)不再调 `ensure_resident_tables`;建表只在 server
`init_db`(SQLite 分支新增 `_res_ensure`,PG 分支走 schema)、
worker 生命周期首次批量、API 函数(create/list/get/update/
transition)做。热路径只在表确实缺失时 ensure 后重试一次。
- 批量入口逐条隔离:锁错误(PG `locked=True`、SQLite
`database is locked`)回滚后重试一次,仍失败记入 `errors`
列表并继续下一条;非锁异常同样隔离;预算耗尽信号以
`watch_id=None` 的条目记入 `errors`,供 worker 记日志。
- 测试:`test_batch_isolates_item_errors_and_retries_deadlock`
(死锁始终失败时只影响本条、其余照常;第一次死锁第二次成功时
errors 为空)、`test_hot_path_batch_executes_no_ddl`
(计数连接断言批量内零 CREATE 语句)。
### G3. 改目标 URL 强制走一次 baseline
- `update_watch` 改 `target_url` 时 `last_content_hash` 置 NULL,
下一轮只建基线、不建任务;改间隔/提示/名字不重置游标,
已排好的 `next_poll_at` 不变(语义保留,见代码注释)。
- 测试:`test_update_target_resets_baseline_but_interval_does_not`。
### G4. 显式代理配置 + 结构化失败原因
- 代理口径见第 1 节清单与 `resident_config()` 注释;默认直连
与环境变量的关系写在代码与本摘要里,不只靠隐式进程环境。
- 失败原因进 `resident_polls.failure_reason` 列(ensure 与 PG
schema 同步补列),detail 前缀同样写 `reason=<原因>`;
原因集:`dns / proxy / timeout / http_status / ssrf_blocked /
tls / redirect / network / invalid_url / unknown`。
- 测试:`test_proxy_config_defaults_direct_and_env_opt_in`、
`test_classify_fetch_errors`、`test_http_status_fetch_raises_structured_error`
(单跳层真实 404 + default_fetch 状态分支)。
### G5. resident 指标进 /metrics
- `GET /metrics` 新增 `resident` 段(成员项目过滤,与任务计数
同款):`watches_by_status`、`due_count`、`max_overdue_seconds`、
近 24h `polls_last_24h / failed_last_24h / changed_last_24h /
change_tasks_last_24h`、当前拟定 `config`;表缺失时
`{"available": False}`,不炸 `/metrics`。
- 测试:`test_resident_metrics_counts`(含到期积压与非成员过滤)、
`test_metrics_contains_resident_section`(HTTP 层)。
### P2 建议项(顺手修,不改语义)
- resident 四个 handler 的 502 统一为固定错误码
`resident_operation_failed`,详情只记服务端日志(`_server_log`),
不再透传可能含表名/列名的异常文本;测试
`test_resident_502_uses_fixed_error_code`(响应不含注入的细节串)。
- `_resident_get` 的 polls 异常记服务端日志(`_server_log.warning`),
响应仍保持 `polls=[] + polls_note="polls unavailable"` 的兼容形态;
测试 `test_resident_get_polls_failure_keeps_note`。
- 未做(列为后续,不硬修):改间隔立即生效的语义(仍为“暂停后
恢复才立即复查”,已写进注释与本摘要);选择器/区域提取;
控制台 UI。
## 4. 测试结果
- 定向(最后一次纯文案修订后重跑,见时间关系说明):
`python3 -m unittest cloud.tests.test_resident
cloud.tests.test_resident_server cloud.tests.test_resident_website_loop
cloud.tests.test_resident_p0_fixes` —— 38 项全过(旧 20 项 + 新 18 项)。
- ruff E,F(被碰文件):与改动前基线一致的 12 项,全是旧代码
(含 `cloud/worker.py` 的 F821、`cloud/server.py` 旧中文注释超长行等),
本批新增 0 项(新增的 3 项在自查中已修掉)。
- CI 同款三条 discover 全量(均在本批代码定稿后跑出):
- `python3 -m unittest discover -s cloud/tests`:1569 项,OK(跳过 9 项),393 秒;
- `python3 -m unittest discover -s deploy/tests`:44 项,OK,12 秒
(尾部三行中文传输日志为某测试退出时打印,不影响结果);
- `python3 -m unittest discover -s tests`:105 项,OK,12 秒。
- 时间关系:定向 38 项先于全量跑出;全量三条跑完后,本摘要
填入全量数字(纯文案修订,不碰代码与测试),随后按任务要求
重跑定向——重跑结果仍为 38 项全过,与全量结论一致;摘要定稿
在定向重跑之后,不再有代码或测试改动。
## 5. 本地 PG 16 验证单独记录(一次性库,测完已删库删脚本)
- 方式:`initdb --auth=trust` 一次性库(Unix socket,psycopg2
适配层),脚本放 `/tmp`(不进 worktree),测完停库、删库、
删脚本;结论只代表本地 PG,不代表生产 PG。
- 结果(全部通过):
- `init_postgres_schema` 后 `resident_watches` / `resident_polls` 齐全;
- 索引在位:`idx_resident_watches_project`、
`idx_resident_watches_name`(部分唯一,
`WHERE (status <> 'ended'::text)`)、`idx_resident_polls_watch`;
- 二次 `init_postgres_schema` + `ensure_resident_tables` 幂等;
- `migration_coverage` 无缺失;SQLite→PG 迁移 resident 两表各 1 行;
- 基线→变化→queued 任务、已结束名重用全部通过;
- 双连接并发:胜者 1 条 baseline 轮询、败者 0 条且 `errors` 为空、
全程无死锁异常、无整批 abort、poll 总数 1(G1 修复后败者干净返回;
旧死锁形态未再出现,但“双连接”不等于“生产双 worker”,见偏差)。
- 已清理:集群目录、socket 目录、日志与 `/tmp` 脚本均已删除,
`ls /tmp/resident_pg*` 确认无残留。
## 6. 已知偏差
- 本地 PG 结论不是生产 PG 结论;本地 HTTP 结论不是真实外网结论;
双连接结论不是生产双 worker 结论;真实外网只读抓取演练仍未做。
- 代理模式下 pin 不生效(见 S1 残留风险),须靠代理 egress allowlist。
- 旧闭环测试第 5 步轮询时刻由 `t0+400` 改为 `t0+540`:新冷却语义下
160 秒间隔的变化会被正确抑制,测试改按冷却期外复查(注释已写明),
不是为通过而删断言。
- 旧重定向测试与闭环测试的本地服务接法改为“resolver 返回公网测试
IP + socket 层重定向”:门与 pin 全程真实跑,只有传输层终点是
测试替身;门本身由默认解析器用例独立覆盖。
- `resident_config()` 每次调用读环境变量:改环境变量即生效,
但已启动进程的既有默认值不会跨进程同步(与其它环境变量配置同款)。
- 无效环境变量数值(非整数/<=0)回落默认值,不炸驱动(配置错误
按默认跑,运维侧须自行核对配置是否生效,见 /metrics 的 config 段)。
## 7. blocker(需用户拍板,本批不擅自定)
- 拟定默认值清单(第 1 节全部):上限 24、冷却 300 秒、保留期 90 天、
批量预算 120 秒、默认直连代理——供最终验收改。
- 是否新增独立驱动进程(本批选批量总预算,见 S2 论证;独立驱动
仍是可选项,需用户定运维成本)。
- SSRF 修复后的摘录口径最终版:本批拟定为“只来自已通过校验的
公开抓取结果”;是否进一步收紧为只留 hash/长度/脱敏摘录,仍由
用户定。
- 生产 PG 双 worker 实测窗口、真实外网只读抓取演练窗口、
代理模式下的代理 egress allowlist 配置(若生产用代理模式)。
- 控制台 UI 是否随修复批上(属产品范围,不计入本轮)。
## 8. 未做事项(本批明确不做)
- 未 push、未开 PR、未合并、未部署;工作区未提交(由 parent 定)。
- 未碰生产审核 worktree、闭环 worktree、主 checkout(feat-repo-picker)。
- 未做控制台 UI、选择器/区域提取、其它探针、磁盘清理、真实外网抓取。
- 未改生产默认值以外的任何行为口径;旧测试除上述两处时序/接法
调整外无删断言、无加 skip。
+69
View File
@@ -0,0 +1,69 @@
# 常驻监听任务(resident watch)验证摘要
## 基底与交付物(代码就绪,local only)
- 工作区:`~/workspace/codex-app-server-resident-task`
- 分支:`feat-resident-task`,基底 `origin/main` = `622c51e`(Merge PR #274,2026-11-02 fetch 到的头,与任务书一致)。
- 主 checkout(`~/workspace/codex-app-server`,`feat-repo-picker`,有脏文件)与其它现存 worktree 一律未动。
- 本轮提交前状态:`cloud/server.py` +187 行、`cloud/worker.py` +31 行,新增 `cloud/resident.py`、`cloud/tests/test_resident.py`、`cloud/tests/test_resident_server.py`、本摘要。
- **未做**:未 push、未合并到 main、未跑 `deploy.py`、未部署、未建生产任务、未动生产/主机 env/系统 key/sub2api/生产网关/仓库可见性/GitHub App 权限、未杀生产容器。
## 基底核实(先调查结论:main 上常驻监听不存在,不是重复建设)
- 任务状态机仍是 `queued -> running -> completed / failed / interrupted / merged / merge-failed`;`/pause` 只是把进行中回合任务标 `interrupted` 并置 `paused_by_user=1`(`cloud/server.py` `_pause_task`),不是可续跑的常驻暂停态——本实现不复用它,另建常驻态。
- 云端自动化(`cloud/automations.py`)定义层只有 `active / cancelled`,含 `automations / automation_runs` 表与 `run_due_automation` 的 claim 抢单、配额门;服务端本身不起定时器,到期触发靠外部触发器。本实现复用它的 claim 口径与配额函数,不另起冲突语义。
- goal 循环(`host.py` `run_goal_loop`,`max_rounds=0` 不限轮次)结束条件是模型宣告达成,不是“一直监听外部变化”;定时自动化每次到期建一个新任务跑一轮就结束。两者都不是常驻监听,本摘要不把它们说成已实现。
- main 上与暂停相关的已有机制(MCP `pause_task` 工具、长任务中断自动恢复)针对的都是回合任务,不是常驻监听;grep `pause|resident` 在 `cloud/*.py` 与 `host.py` 中未发现常驻监听实现。
## 设计抉择
- 新层不改任务表:`cloud/resident.py`(标准库 + 复用 `cloud.automations` 的 `_audit_row`、`_automation_quota_exceeded`)新增 `resident_watches` / `resident_polls` 两张表,避免改动任务状态机牵连 worker 认领、sweep、自动恢复。
- 状态机:`running -> paused -> running`;`running / paused -> ended`(终态)。`ended` 后 pause / resume / end / update 一律 409,恢复已结束的监听必须新建任务(文档与测试双锁定,见下)。
- 暂停保留现场:暂停只改 `status / paused_reason / next_poll_at`,`last_content_hash / last_poll_at / last_change_at / last_change_text` 不动;暂停期间 `due=False`,驱动不轮询。恢复时 `next_poll_at=now` 立刻复查,用旧游标比对——暂停期间的变化在恢复后第一次轮询即报出,不丢。
- 驱动(明说):服务端本身不起定时器,与自动化同款结论。实际驱动是 `cloud/worker.py` 的 `run_loop` 轮次边界调 `process_due_resident_watches()`(新增方法,异常只记日志、不炸循环;`self.resident_fetcher` 可注入假抓取供测试)。任何外部触发器也可直接调 `run_due_resident_watch / process_due_resident_watches`,两者都走 claim 门。
- 防重复轮询:`claim_due_watch` 条件 UPDATE(`status='running'` 且 `next_poll_at <= now`),抢到才把 `last_poll_at / last_claimed_at` 写成 now 并预置 `next_poll_at=now+interval`(崩在抓取中途时下个间隔仍重试,不因认领锁死永久停摆;完成后按成败重算)。
- 变化才花钱:轮询 = 廉价抓取 + 空白折叠后 sha256 比对,不跑模型。`baseline`(首次,只建游标)、`unchanged`(不建任务)、`changed`(才建一个 queued 任务,prompt = 用户处理提示 + 变化摘录)三种产出都记 `resident_polls`,可查。
- 配额边界:配额门只在“变化要建模型任务”那一刻过(复用自动化同款 `_automation_quota_exceeded`)。耗尽时**不结束、不丢变化**:转 `paused`(`paused_reason='quota_exceeded'`)、游标不推进、记 `changed_quota_paused`;配额恢复后恢复,同一变化仍能报出。配额检查本身失败时按瞬时失败退避,不转暂停,游标同样不推进。
- 安全边界:间隔下限 60 秒(与自动化同款)、上限 30 天;抓取超时 1–60 秒(默认 10 秒)、读满 1MB 即断;失败退避 30 / 60 / 120……秒、上限 3600 秒;目标 URL 仅 http/https、无 userinfo、字面 localhost/私有/回环/链路本地 IP 拒绝。
- API(`cloud/server.py`,权限与自动化同款:写 `create_task`、读 `read`,全部记审计):
- `POST /projects/{id}/resident-watches`(name / target_url / prompt 必填;interval_seconds 默认 3600;fetch_timeout_seconds 可选;创建不走配额门)
- `GET /projects/{id}/resident-watches`(附 `due_count`)、`GET /projects/{id}/resident-watches/{wid}`(附最近轮询 `polls`)
- `PATCH /projects/{id}/resident-watches/{wid}`(改名/提示/目标/间隔/超时;ended 409;改间隔不立即触发,下次轮询完成后按新间隔重算)
- `POST /projects/{id}/resident-watches/{wid}/pause|resume|end`
- 状态查询返回常驻态、游标(`last_content_hash`)、最后变化(`last_change_at / last_change_text`)、下次轮询(`next_poll_at`)、失败次数与暂停原因。
## 验收清单逐项结果
1. 能新建常驻任务(目标 URL、轮询间隔、变化处理提示):✅ 模块与服务端测试覆盖;非空壳——新建即到期,驱动下一轮即可认领。
2. 三态清晰(运行中/已暂停/已结束终态):✅ 状态机与 HTTP 409 语义双覆盖。
3. 暂停保留现场、暂停期间停轮询、恢复从游标续跑:✅ 测试 `test_pause_preserves_cursor_and_resume_reports_pause_period_change`(暂停期间变了,恢复后第一次轮询即报出)。
4. 结束终态、结束后不再静默重启、结束后恢复被拒绝:✅ `ended` 后 due 查询只认 `running`,pause / resume / end / update 一律 409(服务端错误文本含 terminal 并提示新建);测试覆盖。
5. 真正有驱动、claim 防重复、心跳与 stale 回收:✅ 驱动为 worker `run_loop` 轮次边界(不是服务端定时器,明说);claim 单胜者测试覆盖;stale 口径为“认领预置 next_poll_at,崩后下个间隔重试”+ worker 既有每轮心跳/sweep(常驻监听本身不另起心跳表,见已知偏差)。
6. 变化才花钱、每次轮询与变化可查:✅ baseline / unchanged 不建任务、changed 才建任务;`resident_polls` 逐轮记录。
7. 间隔下限、抓取超时、失败退避、除配额耗尽外不擅自结束、配额耗尽只转可恢复暂停并明示原因:✅ 全部测试覆盖;配额耗尽测试断言状态为 `paused` 且明确 `!= ended`。
8. 权限与审计沿用任务/自动化同款、pause / resume / end 记审计:✅ 服务端测试断言 `resident_watch_created / paused / resumed / ended` 审计行;跨项目行一律 404(不泄露存在性);局外人写操作被拒。
9. API 含暂停、恢复、结束与状态查询:✅ 见上;控制台 UI 本轮未接,列为已知偏差,不写成已完成。
## 真实测试数字
- 定向(改动后):`python3 -m unittest cloud.tests.test_resident cloud.tests.test_resident_server` —— 13 项(模块 10 + 服务端 3),全部通过。
- 全量(CI 同款三条 discover,改动后、仅最后一次文案注释修订前跑出;该修订只改 `update_watch` 的 docstring,不改任何代码逻辑,修订后定向 13 项与自动化相关 31 项重跑全部通过):
- `discover -s tests`:105 项,0 失败,0 跳过。
- `discover -s cloud/tests`:1544 项,0 失败,9 跳过。
- `discover -s deploy/tests`:44 项(单独重跑确认),0 失败。
- 合计 1693 项、0 失败、9 跳过。失败未靠删断言、加 skip、改默认值蒙混;新增测试 13 项全部计入上述 cloud 套件。
- 新模块定向覆盖率(`coverage run --source=cloud.resident`,仅定向 13 项):`cloud/resident.py` 312 语句、85% 覆盖。未覆盖主因:默认真实 HTTP 抓取(`default_fetch`,单元测试一律注入假抓取,不联网)与个别异常分支(PG 方言补列、配额检查失败分支);本仓库 CI(`.github/workflows/host-verify.yml`)本身没有覆盖率门禁,故不报“覆盖率变化”——全量覆盖率基线未在本轮重算,不虚构数字。
## 已知偏差与未做事项
- 控制台 UI 未接:`cloud/console.html` 是打包产物,本轮没有常驻监听面板;用户现阶段只能走 API。接 UI 是后续一轮,不在本轮验收内。
- 变化判定 v1 较粗:空白折叠后全文 sha256,没有选择器/区域提取;动态页面(广告、时间戳、随机推荐)本身会抖动成“变化”并建任务。缓解在后续:加 CSS 选择器或正文提取口径、或变化阈值。
- SSRF 防护 v1 只挡字面 localhost/私有 IP,不做 DNS 解析后校验(解析后重定向到内网、DNS rebinding 不在覆盖内);`websearch.fetch_url` 的同款字面口径已对齐,但不要把 v1 当成完整 SSRF 方案。
- 常驻监听没有独立心跳表:stale 回收靠认领预置的 `next_poll_at` + worker 既有心跳/sweep;驱动进程整体死亡时,已认领未完成的当次轮询要等到下个间隔才补跑(与自动化 claim 的同款取舍)。
- 未验证:真实外网抓取(单元测试不联网)、生产 PG 上的表迁移(`PRAGMA` 补列走自动化同款适配写法,但未在 PG 实测)、worker 常驻循环里的长时驱动(仅方法级测试,未跑真实 `run_loop` 长循环)、多 worker 并发抢单(仅条件 UPDATE 单胜者测试,未做双进程实测)。
- 未做:未 push、未合并、未部署;UI、选择器提取、DNS 级 SSRF、PG 实测、双进程抢单实测留待后续轮次。
## Blocker
- 无本轮 blocker(全部工作均为本地代码与本地测试,未触及只有用户能给的账号/密钥/商户/商店账号/宿主/维护窗口,也未发生花真钱、删生产数据、改生产可见性的动作)。
- 上线前仍需用户侧决策(不是本轮 blocker,不擅自定):控制台 UI 是否随本批上、默认轮询间隔与抓取超时是否按本轮默认值(60 秒下限/10 秒超时)拍板为生产默认值、PG 实测与部署窗口。
@@ -0,0 +1,125 @@
# 常驻监听网站闭环验证摘要
## 起点与交付物(代码就绪,local only)
- 工作区:`~/workspace/codex-app-server-resident-website-loop`
- 分支:`feat-resident-website-loop`,起点为上一轮网站版
`feat-resident-task` 的 `0722ca3`(基底 `origin/main 622c51e`)。
- 实现提交:`561b389`(仅新增端到端测试文件
`cloud/tests/test_resident_website_loop.py`,364 行)。
本摘要为其后提交,工作区提交后干净。
- 范围:只做网站用例。不做 server_status、agent_status、
relay_cost 等其它探针;不把 resident-watches 改名/泛化成
通用任务;不改 API、表名与任何生产默认值。
- 泛化 worktree(`~/workspace/codex-app-server-resident-generic`)
本轮一律未碰;resident-task worktree、主 checkout
(`feat-repo-picker`)及其它 worktree 一律未动。
- **未做**:未 push、未合并到 main、未跑 `deploy.py`、未部署、
未建生产任务、未动生产/主机 env/系统 key/sub2api/
生产网关/仓库可见性/GitHub App 权限、未杀生产容器、
未连真实外部网站。
## 闭环逐环证据(全部真实链路,不是注入假抓取)
测试文件:`cloud/tests/test_resident_website_loop.py`,2 项。
内容服务是线程内真实 `http.server`(可变内容、可停服重启);
控制面是真实 `ThreadingHTTPServer` + 真实 SQLite 临时库;
驱动走真实 `default_fetch`(urllib、超时、1MB 截断)。
生产 URL 校验一字不改:测试主机名 `website-loop.test` 不是
IP 字面、不是 localhost,所以校验通过;测试进程内只把该
主机名的 DNS 指到本地服务(相当于 /etc/hosts 的一条测试
条目),并给它加 `no_proxy`(见“测试中发现的两处假断点”)。
1. 创建(API):`POST /projects/{id}/resident-watches`
(name=site1、target_url=`http://website-loop.test:<本地端口>/data`、
prompt=“变化时汇报”、interval_seconds=60)返回 201,
状态 running、interval 60、fetch_timeout 10(生产默认)、
last_content_hash 为空、next_poll_at 在位;审计含
`resident_watch_created`。状态查询 `GET` 单条可看到
running、游标、下次轮询。
2. 基线轮询:worker 驱动 `process_due_resident_watches`
(fetcher=None,即真实 default_fetch)抓到 “version one”,
记 `baseline` 轮询、游标为其 sha256,任务数为 0;
状态查询的 polls 第 1 条就是 baseline。
3. 变化建任务:内容改成 “version two”,到期轮询记
`changed`、建出 1 个 queued 任务,任务 prompt 含用户提示
“变化时汇报”与摘录 “version two”,任务行与审计行
`resident_watch_change_task_created` 均可查;游标推进到
version two,last_change_text 含 version two。
4. 暂停:API pause 后状态 paused、next_poll_at 为空;暂停
期间内容再改成 “version three”,驱动返回空、轮询数仍为
2、游标仍为 version two——暂停期间的变化没有被轮询上报。
5. 恢复:API resume 后状态 running;下一次驱动用旧游标比
对,记 `changed` 并建第 2 个任务(prompt 含 version three),
不是从头建基线;游标推进到 version three。
6. 失败环:停掉内容服务后驱动记 `failed` 轮询、
consecutive_failures=1、next_poll_at 大于 now+60+30
(间隔 + 首次退避 30 秒)、状态仍 running、游标仍为
version three——驱动不崩、监听不擅自结束。恢复服务后
(内容仍为 version three)驱动记 `unchanged`、失败清零、
任务数仍为 2。
7. 结束:API end 后状态 ended、next_poll_at 为空、游标仍为
version three;驱动返回空;pause / resume / end / PATCH
更新一律 409;状态查询显示 ended 与最后游标/变化,
polls 总数为 5(baseline、changed、changed、failed、
unchanged)。
8. 驱动接线:单独一测试把 `run_all` 等无关边界打桩,只留
resident 驱动走真实抓取,调真实 `Worker.run_loop`
(interval=0.05、max_seconds=0.4);记录到 run_loop 轮次
边界至少调了 1 次 `process_due_resident_watches`,且状态
查询里出现 1 条 baseline 轮询、游标为 version one。
这证明接线在 run_loop 里,不只是测试直接调驱动函数。
## 断点结论:生产代码无断点,无生产改动
本轮逐环均通,生产代码(`cloud/resident.py`、`cloud/server.py`、
`cloud/worker.py`)一字未改,`git diff` 为空、新增文件只有
端到端测试。第一次跑出的 2 个失败都是测试脚手架问题,
不是生产断点,修法与证据如下,不许读成“修了生产代码”:
- 假断点 1——首次真实抓取 failed:测试环境自带 egress 代理,
urllib 默认把测试主机名也送代理并被代理断开。实测对照:
不加 no_proxy 时 `default_fetch` 报 RemoteDisconnected,
加 no_proxy 后返回 “version one”。修法只在测试 setUp 里
给 `website-loop.test` 加 no_proxy(测试进程内生效,撕掉
时还原),生产抓取与代理行为一字不改。
- 假断点 2——run_loop 接线测试 TypeError:实例属性上的
recording 函数不会自动绑定 self。修法改为闭包捕获 worker,
不改 worker 代码。
## 真实测试数字
- 定向(全量后重跑,含本轮 2 项):`python3 -m unittest
cloud.tests.test_resident cloud.tests.test_resident_server
cloud.tests.test_resident_website_loop` —— 15 项全部通过。
其中上一轮 13 项在新 worktree 动工前已先复核全过。
- 全量(CI 同款三条 discover,定向之后、全量之前无生产改动,
全量后又重跑定向 15 项全过,时间关系如实写明):
- `discover -s tests`:105 项,0 失败,0 跳过。
- `discover -s cloud/tests`:1546 项,0 失败,9 跳过
(上一轮 1544 + 本轮新增 2,对账无差)。
- `discover -s deploy/tests`:44 项,0 失败。
- 合计 1695 项、0 失败、9 跳过。失败未靠删断言、加 skip、
改默认值蒙混;生产间隔下限 60 秒、抓取超时默认 10 秒、
URL 校验口径一律未动,到期只用驱动 now 注入模拟时间流逝。
## 已知偏差与未做事项(未写成已完成)
- 控制台 UI 不在本轮闭环内:只能走 API,无常驻面板。
- 真实外网抓取未验:只验了本地 HTTP 服务 + 测试 DNS 映射,
不要把本轮说成真实外网验证;生产环境下的代理/DNS/证书
行为仍属后续验证项。
- 生产 PG 表迁移未验(只验 SQLite 临时库)、双进程抢单未验
(只验条件 UPDATE 单胜者的上一轮单测与本轮单驱动链路)。
- 其它探针本轮明确不做:server_status、agent_status、
relay_cost、通用任务改名一律不在本轮范围内。
- 变化判定仍是上一轮 v1 全文 hash,无选择器/区域提取;
SSRF 仍是 v1 只挡字面私有 IP,未做 DNS 解析后校验。
- 未做:未 push、未合并、未部署;架构与代码质量的 10 分
审核是父级后续独立步骤,本轮不自评、不给自己打 10 分。
## Blocker
- 本轮无 blocker(纯本地代码与本地测试,未触及只有用户能给
的账号/密钥/商户/商店账号/宿主/维护窗口,也未发生
花真钱、删生产数据、改生产可见性的动作)。