websearch: task-level allowed_hosts extension for scope=code + JS-heavy extract evaluation (mock-first)

- search(..., allowed_hosts=[...]) extends (never replaces) the 15-domain
  code list for that call only; scope=general + allowed_hosts raises,
  empty/illegal values raise before any HTTP, dot-boundary matching,
  extension values never in error strings or attempted entries.
- Native-filter parity: Tavily/Exa get the effective list, Perplexity
  skips its native filter past its 20-domain cap (post-filter stays
  authoritative), Brave/SearXNG/DuckDuckGo rely on post-filtering.
- MCP web_search gains allowed_hosts passthrough (only when supplied,
  so older search_fn fakes keep working).
- JS-heavy evaluation documented in docs/web-search-mcp.md, mock-first:
  direct fetch is shell-only by construction; provider extract can
  supply rendered text in mock only; live SearXNG in this VM env
  failed (remote closed connection) and paid-backend live effects are
  unvalidated blockers, not claimed.
- Tests: test_code_search_ext (9) + test_js_heavy_extract (5 + 1 live
  skipped); existing websearch/code-search suites still green.

(cherry picked from commit 9c26bbe848438e1f34bcb000700aeb4c284f4f23)

cloud/tests/test_code_search_ext.py

@@ -0,0 +1,220 @@
+"""Task-level code-scope extension (search(..., allowed_hosts=[...])).
+
+All HTTP is mocked; no network, no real keys. Fake key values are
+distinctive so a leak into an error string or result is caught by
+assertion. Extension values are the caller's own input echoed in
+its own result only -- no test here writes them to a log, and the
+code under test never formats them into error strings.
+"""
+
+import json
+import sys
+import unittest
+from pathlib import Path
+
+try:
+    from cloud import websearch as _ws
+    from cloud.websearch import (
+        CODE_SEARCH_DOMAINS, WebSearchError, _is_code_domain,
+        _normalize_allowed_hosts, search)
+    from cloud.websearch_mcp import handle_request
+except ImportError:
+    sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
+    from cloud import websearch as _ws
+    from cloud.websearch import (
+        CODE_SEARCH_DOMAINS, WebSearchError, _is_code_domain,
+        _normalize_allowed_hosts, search)
+    from cloud.websearch_mcp import handle_request
+
+_FAKE_KEYS = {
+    "PERPLEXITY_API_KEY": "pplx-SECRET-111",
+    "EXA_API_KEY": "exa-SECRET-222",
+    "TAVILY_API_KEY": "tvly-SECRET-333",
+    "BRAVE_API_KEY": "brave-SECRET-444",
+}
+
+_EXTRA = "docs.example-project.dev"
+
+
+class FakeHttp:
+    """post_json/get keyed by URL substring, in route order."""
+
+    def __init__(self, routes):
+        self.routes = routes
+        self.seen = []
+
+    def post_json(self, url, headers, payload):
+        self.seen.append(("POST", url, payload))
+        return self._route(url)
+
+    def get(self, url, headers=None):
+        self.seen.append(("GET", url, headers))
+        return self._route(url)
+
+    def _route(self, url):
+        for needle, body, status in self.routes:
+            if needle in url:
+                if status != 200:
+                    raise WebSearchError("HTTP %s" % status)
+                if isinstance(body, dict):
+                    return 200, json.dumps(body).encode(), \
+                        "application/json"
+                return 200, body.encode(), "text/html"
+        raise WebSearchError("HTTP 404")
+
+
+def _tavily_body(urls):
+    return {"results": [{"title": "T %s" % u, "url": u,
+                         "content": "snippet %s" % u} for u in urls]}
+
+
+class ExtensionTestCase(unittest.TestCase):
+    def test_specified_extra_domain_returned_and_native_param_sent(self):
+        http = FakeHttp([
+            ("api.tavily.com", _tavily_body([
+                "https://docs.example-project.dev/guide",
+                "https://github.com/example/project",
+                "https://random-blog.example.com/post",
+            ]), 200),
+        ])
+        out = search("project guide", provider="tavily", scope="code",
+                     allowed_hosts=[_EXTRA],
+                     env=dict(_FAKE_KEYS), http=http)
+        payload = http.seen[0][2]
+        self.assertIn(_EXTRA, payload["include_domains"])
+        self.assertEqual(out["code_domains"],
+                         list(CODE_SEARCH_DOMAINS) + [_EXTRA])
+        self.assertEqual(
+            [r["url"] for r in out["results"]],
+            ["https://docs.example-project.dev/guide",
+             "https://github.com/example/project"])
+        self.assertEqual(out["attempted"][0]["domain_filtered_out"], 1)
+        for secret in _FAKE_KEYS.values():
+            self.assertNotIn(secret, json.dumps(out))
+
+    def test_unspecified_extra_domain_still_filtered_out(self):
+        http = FakeHttp([
+            ("api.tavily.com", _tavily_body([
+                "https://docs.example-project.dev/guide",
+                "https://github.com/example/project",
+            ]), 200),
+        ])
+        out = search("project guide", provider="tavily", scope="code",
+                     env=dict(_FAKE_KEYS), http=http)
+        self.assertEqual(out["code_domains"], list(CODE_SEARCH_DOMAINS))
+        self.assertEqual([r["url"] for r in out["results"]],
+                         ["https://github.com/example/project"])
+
+    def test_dot_boundary_positive_and_negative(self):
+        domains = list(CODE_SEARCH_DOMAINS) + [_EXTRA]
+        self.assertTrue(_is_code_domain(
+            "https://api.docs.example-project.dev/v1", domains))
+        self.assertTrue(_is_code_domain(
+            "https://docs.example-project.dev/", domains))
+        self.assertFalse(_is_code_domain(
+            "https://evildocs.example-project.dev/", domains))
+        self.assertFalse(_is_code_domain(
+            "https://docs.example-project.dev.evil.com/", domains))
+        # Same boundary rule for a builtin domain, unchanged.
+        self.assertFalse(_is_code_domain(
+            "https://evilgithub.com/x", domains))
+        self.assertTrue(_is_code_domain(
+            "https://gist.github.com/x", domains))
+
+    def test_builtin_entries_are_deduped_not_errors(self):
+        self.assertEqual(
+            _normalize_allowed_hosts(["github.com", _EXTRA]), [_EXTRA])
+        self.assertEqual(
+            _normalize_allowed_hosts(["Docs.Example-Project.Dev."]),
+            ["docs.example-project.dev"])
+
+    def test_empty_and_illegal_values_raise_before_http(self):
+        bad_values = [
+            [], [""], ["   "], ["*.example.com"],
+            ["http://example.com"], ["https://example.com/x"],
+            ["example.com:8080"], ["example.com/path"],
+            [".example.com"], ["example.com..dev"], ["exa mple.com"],
+            ["user@example.com"], [123], [None], ["github.com"],
+        ]
+        for bad in bad_values:
+            http = FakeHttp([("api.tavily.com", _tavily_body([]), 200)])
+            with self.assertRaises(ValueError, msg=repr(bad)):
+                search("q", provider="tavily", scope="code",
+                       allowed_hosts=bad, env=dict(_FAKE_KEYS),
+                       http=http)
+            self.assertEqual(http.seen, [],
+                             "invalid %r must fail before any HTTP"
+                             % (bad,))
+        # A bare string is iterable but is not a list of hostnames.
+        with self.assertRaises(ValueError):
+            search("q", provider="tavily", scope="code",
+                   allowed_hosts="example.com",
+                   env=dict(_FAKE_KEYS), http=FakeHttp([]))
+
+    def test_allowed_hosts_with_general_scope_raises_not_ignored(self):
+        http = FakeHttp([("api.tavily.com", _tavily_body(
+            ["https://docs.example-project.dev/guide"]), 200)])
+        with self.assertRaises(ValueError):
+            search("q", provider="tavily", scope="general",
+                   allowed_hosts=[_EXTRA],
+                   env=dict(_FAKE_KEYS), http=http)
+        self.assertEqual(http.seen, [])
+
+    def test_perplexity_over_cap_skips_native_filter_keeps_postfilter(self):
+        extras = ["extra%d.example.com" % i for i in range(6)]
+        self.assertGreater(len(CODE_SEARCH_DOMAINS) + len(extras), 20)
+        http = FakeHttp([
+            ("api.perplexity.ai",
+             {"choices": [{"message": {"content": "answer"}}],
+              "search_results": [
+                  {"title": "P",
+                   "url": "https://extra0.example.com/doc",
+                   "snippet": "extra"},
+                  {"title": "Ad", "url": "https://shop.example.com/x",
+                   "snippet": "ad"}]}, 200),
+        ])
+        out = search("q", provider="perplexity", scope="code",
+                     allowed_hosts=extras,
+                     env=dict(_FAKE_KEYS), http=http)
+        payload = http.seen[0][2]
+        self.assertNotIn("search_domain_filter", payload)
+        self.assertEqual([r["url"] for r in out["results"]],
+                         ["https://extra0.example.com/doc"])
+        self.assertEqual(len(out["code_domains"]), 21)
+
+
+class McpPassthroughTestCase(unittest.TestCase):
+    def test_mcp_forwards_allowed_hosts_when_supplied(self):
+        captured = {}
+
+        def fake_search(query, limit=5, provider="auto", scope="general",
+                        allowed_hosts=None):
+            captured["allowed_hosts"] = allowed_hosts
+            return {"provider": "fake", "results": [], "answer": None,
+                    "attempted": [], "scope": scope}
+
+        resp = handle_request(
+            {"jsonrpc": "2.0", "id": 1, "method": "tools/call",
+             "params": {"name": "web_search",
+                        "arguments": {"query": "hi", "scope": "code",
+                                      "allowed_hosts": [_EXTRA]}}},
+            search_fn=fake_search)
+        self.assertFalse(resp["result"]["isError"])
+        self.assertEqual(captured["allowed_hosts"], [_EXTRA])
+
+    def test_mcp_without_allowed_hosts_keeps_old_fake_working(self):
+        def fake_search(query, limit=5, provider="auto",
+                        scope="general"):
+            return {"provider": "fake", "results": [], "answer": None,
+                    "attempted": [], "scope": scope}
+
+        resp = handle_request(
+            {"jsonrpc": "2.0", "id": 2, "method": "tools/call",
+             "params": {"name": "web_search",
+                        "arguments": {"query": "hi"}}},
+            search_fn=fake_search)
+        self.assertFalse(resp["result"]["isError"])
+
+
+if __name__ == "__main__":
+    unittest.main()

cloud/tests/test_js_heavy_extract.py

@@ -0,0 +1,155 @@
+"""JS-heavy page extraction evaluation, mock-first.
+
+All HTTP is mocked; no network, no real keys. The three page shapes
+below are the mock stand-ins for real JS-heavy pages (a static shell
+whose body is rendered by JS, a shell whose pagination is JS-loaded,
+and a shell whose content lives only in a <script> blob):
+
+- direct fetch (fetch_url) parses static HTML only: <script> and
+  <style> contents are stripped, so anything JS would have rendered
+  is absent from the result -- this is by construction, not a bug;
+- a provider extract stage (fetch_via_provider, mocked here) can
+  return the rendered text when the provider actually renders, and
+  an empty extraction is a failure, never a fabricated page.
+
+Live对照: only the keyless SearXNG route is ever used live, and
+only when a test environment actually provides SEARXNG_URL; this
+file never calls a paid backend live. Set
+WEBSEARCH_EVAL_LIVE_SEARXNG=1 with SEARXNG_URL in the environment
+to enable the live smoke test, otherwise it is skipped.
+"""
+
+import json
+import os
+import sys
+import unittest
+from pathlib import Path
+
+try:
+    from cloud.websearch import (
+        WebSearchError, fetch_best, fetch_url, fetch_via_provider)
+except ImportError:
+    sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
+    from cloud.websearch import (
+        WebSearchError, fetch_best, fetch_url, fetch_via_provider)
+
+_FAKE_KEYS = {
+    "TAVILY_API_KEY": "tvly-SECRET-333",
+    "EXA_API_KEY": "exa-SECRET-222",
+}
+
+# Shape 1: shell HTML only; the article body is JS-rendered.
+_SHELL_HTML = """<html><head><title>Guide</title></head><body>
+<div id="app">Loading…</div>
+<script src="/app.js"></script>
+<script>window.__DATA__ = {"body": "RENDERED-BODY-PARAGRAPH"};</script>
+<a href="https://example.com/guide/page2">Next (static link)</a>
+</body></html>"""
+
+# Shape 2: paginated list; page 2 items arrive only via JS fetch.
+_PAGED_SHELL_HTML = """<html><head><title>List</title></head><body>
+<ul id="items"><li>ITEM-PAGE-1</li></ul>
+<button id="more">Load more</button>
+<script>fetch("/api/items?page=2").then(r => r.json());</script>
+</body></html>"""
+
+
+class FakeHttp:
+    """get/post_json keyed by URL substring, in route order."""
+
+    def __init__(self, routes):
+        self.routes = routes
+        self.seen = []
+
+    def post_json(self, url, headers, payload):
+        self.seen.append(("POST", url, payload))
+        return self._route(url)
+
+    def get(self, url, headers=None):
+        self.seen.append(("GET", url, headers))
+        return self._route(url)
+
+    def _route(self, url):
+        for needle, body, status in self.routes:
+            if needle in url:
+                if status != 200:
+                    raise WebSearchError("HTTP %s" % status)
+                if isinstance(body, dict):
+                    return 200, json.dumps(body).encode(), \
+                        "application/json"
+                return 200, body.encode(), "text/html"
+        raise WebSearchError("HTTP 404")
+
+
+class JsHeavyDirectFetchTestCase(unittest.TestCase):
+    def test_direct_fetch_gets_shell_only_not_js_rendered_body(self):
+        http = FakeHttp([("example.com/guide", _SHELL_HTML, 200)])
+        out = fetch_url("https://example.com/guide", http=http, env={})
+        # Static shell survives: title + placeholder + static link.
+        self.assertEqual(out["title"], "Guide")
+        self.assertIn("Loading", out["text"])
+        self.assertIn("https://example.com/guide/page2", out["links"])
+        # JS-rendered body does not: <script> contents are stripped,
+        # so neither the rendered paragraph nor the data blob leaks
+        # in as text.
+        self.assertNotIn("RENDERED-BODY-PARAGRAPH", out["text"])
+        self.assertNotIn("window.__DATA__", out["text"])
+
+    def test_direct_fetch_gets_page1_only_not_js_paginated_items(self):
+        http = FakeHttp([("example.com/list", _PAGED_SHELL_HTML, 200)])
+        out = fetch_url("https://example.com/list", http=http, env={})
+        self.assertIn("ITEM-PAGE-1", out["text"])
+        self.assertNotIn("ITEM-PAGE-2", out["text"])
+        self.assertNotIn("/api/items?page=2", out["text"])
+
+
+class JsHeavyProviderExtractTestCase(unittest.TestCase):
+    def test_mock_provider_extract_can_supply_rendered_text(self):
+        # Mock stand-in for "provider renders JS": Tavily extract
+        # returns the full rendered article text.
+        http = FakeHttp([
+            ("api.tavily.com/extract",
+             {"results": [{"raw_content": "RENDERED-BODY-PARAGRAPH "
+                                          "plus ITEM-PAGE-2"}]}, 200),
+        ])
+        out = fetch_via_provider("https://example.com/guide",
+                                 env=dict(_FAKE_KEYS), http=http)
+        self.assertEqual(out["source"], "tavily-extract")
+        self.assertIn("RENDERED-BODY-PARAGRAPH", out["text"])
+
+    def test_mock_provider_extract_empty_is_failure_not_fabrication(self):
+        http = FakeHttp([
+            ("api.tavily.com/extract", {"results": []}, 200),
+            ("api.exa.ai/contents", {"results": []}, 200),
+        ])
+        with self.assertRaises(WebSearchError):
+            fetch_via_provider("https://example.com/guide",
+                               env=dict(_FAKE_KEYS), http=http)
+
+    def test_fetch_best_falls_back_to_shell_when_extract_fails(self):
+        http = FakeHttp([
+            ("api.tavily.com/extract", {"results": []}, 200),
+            ("api.exa.ai/contents", {"results": []}, 200),
+            ("example.com/guide", _SHELL_HTML, 200),
+        ])
+        out = fetch_best("https://example.com/guide",
+                         env=dict(_FAKE_KEYS), http=http)
+        self.assertEqual(out["source"], "direct-fetch")
+        self.assertNotIn("RENDERED-BODY-PARAGRAPH", out["text"])
+
+
+@unittest.skipUnless(
+    os.environ.get("WEBSEARCH_EVAL_LIVE_SEARXNG") == "1"
+    and os.environ.get("SEARXNG_URL"),
+    "live SearXNG对照 only when the test env provides SEARXNG_URL")
+class LiveSearxngSmokeTestCase(unittest.TestCase):
+    def test_live_searxng_chain_answers_without_paid_keys(self):
+        from cloud.websearch import search
+        out = search("python pathlib documentation", provider="searxng",
+                     scope="general",
+                     env={"SEARXNG_URL": os.environ["SEARXNG_URL"]})
+        self.assertTrue(out["results"])
+
+
+if __name__ == "__main__":
+    unittest.main()

cloud/websearch.py

@@ -96,19 +96,83 @@ CODE_SEARCH_DOMAINS = (
 )
 
 
-def _is_code_domain(url):
-    """True when url's host is in (or under) CODE_SEARCH_DOMAINS."""
+def _is_code_domain(url, domains=None):
+    """True when url's host is in (or under) ``domains``.
+
+    Defaults to CODE_SEARCH_DOMAINS. Matching is dot-boundary:
+    the host must equal a listed domain or end with "." + domain,
+    so evilgithub.com never matches github.com and
+    github.com.evil.com never matches either.
+    """
+    if domains is None:
+        domains = CODE_SEARCH_DOMAINS
     try:
         host = (urllib.parse.urlparse(url).hostname or "").lower()
     except Exception:
         return False
     host = host.rstrip(".")
-    for domain in CODE_SEARCH_DOMAINS:
+    for domain in domains:
         if host == domain or host.endswith("." + domain):
             return True
     return False
 
 
+def _normalize_allowed_hosts(allowed_hosts):
+    """Validate a task-level code-scope extension list.
+
+    Returns the normalised extra domains (lowercased, trailing dot
+    stripped, builtin duplicates dropped, order preserved). Anything
+    else raises ValueError before any HTTP happens -- never silently
+    degrading to the builtin list or to an unscoped search:
+
+    - None is not valid here; callers pass it through only by
+      leaving ``search(..., allowed_hosts=None)`` unset.
+    - a bare string is rejected (it is iterable, so accepting it
+      would silently treat "example.com" as a list of characters);
+    - an empty list is rejected (empty means "no extension", which
+      is spelled by omitting the argument, not by passing []);
+    - each entry must be a bare hostname: no scheme, port, path,
+      userinfo, wildcard, whitespace, leading dot or empty label.
+    Extension values are never written to logs: this function only
+    validates, and search() never formats the list into an error
+    string or an attempted entry (the effective list is echoed
+    back to the caller in its own result's ``code_domains`` only).
+    """
+    if isinstance(allowed_hosts, str) or not isinstance(
+            allowed_hosts, (list, tuple)):
+        raise ValueError(
+            "allowed_hosts must be a list of hostnames, not %r"
+            % (type(allowed_hosts).__name__,))
+    if not allowed_hosts:
+        raise ValueError(
+            "allowed_hosts must be non-empty; omit the argument "
+            "for no extension")
+    seen = set()
+    out = []
+    for entry in allowed_hosts:
+        if not isinstance(entry, str):
+            raise ValueError(
+                "allowed_hosts entries must be strings, not %r"
+                % (type(entry).__name__,))
+        host = entry.strip().lower().rstrip(".")
+        if (not host or host.startswith(".") or ".." in host
+                or any(c in host for c in "/:?#@[] \t")
+                or "*" in host or host in CODE_SEARCH_DOMAINS):
+            if host in CODE_SEARCH_DOMAINS:
+                continue
+            raise ValueError(
+                "invalid allowed_hosts entry: %r (expected a bare "
+                "hostname such as 'example.com')" % (entry,))
+        if host not in seen:
+            seen.add(host)
+            out.append(host)
+    if not out:
+        raise ValueError(
+            "allowed_hosts adds no new domain beyond the builtin "
+            "list; omit the argument for no extension")
+    return out
+
+
 def _normalize_scope(scope):
     """Map scope to "general" (default) or "code"; reject anything else.
 
@@ -243,11 +307,19 @@ def _provider_perplexity(query, limit, env, http, domains=None):
     # shape instead of degrading to a link list.
     # Native domain filter: top-level search_domain_filter (Sonar chat
     # completions; allowlist mode, max 20 domains -- CODE_SEARCH_DOMAINS
-    # has 15, so it fits whole).
+    # has 15, so the builtin list fits whole; see the over-cap
+    # handling below for task-level extensions).
     payload = {"model": "sonar",
                "messages": [{"role": "user", "content": query}]}
     if domains:
-        payload["search_domain_filter"] = list(domains)
+        # search_domain_filter caps at 20 domains. The builtin list
+        # is 15, but a task-level allowed_hosts extension can push
+        # the effective list past 20; sending an over-cap filter
+        # would fail the provider call, so in that case the native
+        # filter is skipped and search()'s post-filter -- which
+        # always applies -- remains the authoritative scope.
+        if len(domains) <= 20:
+            payload["search_domain_filter"] = list(domains)
     _status, body, _ctype = http.post_json(
         "https://api.perplexity.ai/chat/completions",
         _default_headers(env, "perplexity"),
@@ -502,7 +574,7 @@ def resolve_provider_chain(provider="auto", env=None):
 
 
 def search(query, limit=5, provider="auto", env=None, http=None,
-           scope="general"):
+           scope="general", allowed_hosts=None):
     """Search the web. Returns {provider, results, answer, attempted,
     scope} (plus code_domains when scope="code").
 
@@ -517,6 +589,20 @@ def search(query, limit=5, provider="auto", env=None, http=None,
     Default scope="general" behaves exactly as before (no domain
     filter, no post-filtering).
 
+    ``allowed_hosts`` is a task-level, explicit extension of the
+    code list only -- it is NOT a general "search any site" switch.
+    It is honoured only with scope="code"; passing it with
+    scope="general" raises ValueError instead of being silently
+    ignored, and omitting it leaves behaviour exactly as before
+    (builtin 15 domains only). When given, the effective list is
+    builtin + validated extras, and the same native-filter /
+    post-filter split applies: native-capable backends receive the
+    effective list (Perplexity skips its native filter when the
+    effective list exceeds its 20-domain cap, relying on the
+    post-filter), and backends without a native filter are scoped
+    by post-filtering alone. Extension values are never formatted
+    into error strings or attempted entries.
+
     Raises WebSearchError (with .attempted) when every provider in
     the chain fails or returns no results. Error strings never contain
     API keys; see the module docstring.
@@ -524,7 +610,17 @@ def search(query, limit=5, provider="auto", env=None, http=None,
     if not query or not str(query).strip():
         raise ValueError("query must be a non-empty string")
     scope = _normalize_scope(scope)
-    domains = list(CODE_SEARCH_DOMAINS) if scope == "code" else None
+    if allowed_hosts is not None and scope != "code":
+        raise ValueError(
+            "allowed_hosts is only valid with scope=\"code\" "
+            "(got scope=%r); it is an extension of the code-domain "
+            "list, not a general search override" % (scope,))
+    if scope == "code":
+        domains = list(CODE_SEARCH_DOMAINS)
+        if allowed_hosts is not None:
+            domains.extend(_normalize_allowed_hosts(allowed_hosts))
+    else:
+        domains = None
     env = env if env is not None else os.environ
     http = http or HttpClient(
         timeout=int(env.get("WEBSEARCH_TIMEOUT", "20") or 20))
@@ -549,7 +645,8 @@ def search(query, limit=5, provider="auto", env=None, http=None,
                               "error": "no results"})
             continue
         if scope == "code":
-            kept = [r for r in results if _is_code_domain(r.get("url"))]
+            kept = [r for r in results
+                    if _is_code_domain(r.get("url"), domains)]
             if not kept:
                 attempted.append(
                     {"provider": name, "ok": False,

cloud/websearch_mcp.py

@@ -42,7 +42,12 @@ _TOOLS = [
             "only (official language docs, GitHub, package "
             "registries, Stack Overflow, CVE/NVD); anything else "
             "is filtered out, never returned. Default scope "
-            "'general' is unfiltered. This tool does not do "
+            "'general' is unfiltered. With scope 'code' only, "
+            "allowed_hosts may name extra project domains for "
+            "this call (explicit allowlist extension, subdomains "
+            "included, dot-boundary matched); it is not an "
+            "arbitrary-site search, and it is rejected with "
+            "scope 'general'. This tool does not do "
             "shopping, social, places, travel, general news, "
             "weather, finance, or browser checkout."
         ),
@@ -61,6 +66,21 @@ _TOOLS = [
                                          "search. code: programming "
                                          "sources only, see tool "
                                          "description."},
+                "allowed_hosts": {"type": "array",
+                                  "items": {"type": "string"},
+                                  "description": "Only with scope "
+                                                 "'code': extra "
+                                                 "domains for this "
+                                                 "call only, on top "
+                                                 "of the builtin "
+                                                 "programming list. "
+                                                 "Omit for no "
+                                                 "extension; empty, "
+                                                 "wildcard, or "
+                                                 "non-hostname "
+                                                 "values are "
+                                                 "rejected, never "
+                                                 "silently ignored."},
             },
             "required": ["query"],
         },
@@ -139,11 +159,19 @@ def handle_request(request, search_fn=None, fetch_fn=None):
         args = params.get("arguments") or {}
         try:
             if name == "web_search":
+                # allowed_hosts is passed through only when the
+                # caller supplied it, so older search_fn fakes
+                # without that parameter keep working.
+                search_kwargs = {}
+                if "allowed_hosts" in args:
+                    search_kwargs["allowed_hosts"] = \
+                        args.get("allowed_hosts")
                 result = search_fn(
                     args.get("query"),
                     limit=args.get("limit", 5),
                     provider=args.get("provider", "auto"),
-                    scope=args.get("scope", "general"))
+                    scope=args.get("scope", "general"),
+                    **search_kwargs)
             elif name == "web_fetch":
                 result = fetch_fn(
                     args.get("url"),

docs/web-search-mcp.md

@@ -17,6 +17,12 @@ pi 的工具），只复用它的模式：统一工具 + provider 抽象 +
   mock HTTP，不联网、不需要真实 key。
 - `cloud/tests/test_code_search.py`：编程作用域与抽取回退链
   的单测，同样全部 mock HTTP、不联网、不需要真实 key。
+- `cloud/tests/test_code_search_ext.py`：任务级
+  `allowed_hosts` 扩展的单测（指定 / 未指定 / 点边界 /
+  空值与非法值 / Perplexity 超 20 域 / MCP 透传）。
+- `cloud/tests/test_js_heavy_extract.py`：JS 重型页面
+  抽取评估的 mock 单测（直接抓取壳-only、provider 抽取
+  补与不补、回退语义；live SearXNG 烟囱默认跳过）。
 
 ## 工具
 
@@ -163,6 +169,50 @@ changelog / release notes 只有落在上述域名时才在
   results`），链继续走下一个后端；全部后端都空才
   报错。
 
+### 清单外项目官网域：任务级显式扩展 `allowed_hosts`
+
+这不是“任意站搜索”，仍是显式白名单机制，只是白名单
+可以由调用方在当次调用里显式加域。选型是任务级参数
+`(a)`，不是配置级扩展清单 `(b)`，理由：改动最小
+（只动 `search()` 签名与 MCP 透传，不动 worker、
+unit、env、播种与主机配置）；按调用隔离（某任务要
+某项目官网，不会把该域放宽给所有任务——配置级清单
+一旦写入就是全局放宽，安全姿态更差）；用完即弃
+（不持久化，下次调用不带就不生效）。
+
+用法：`search(..., scope="code",
+allowed_hosts=["docs.example-project.dev"])`，MCP
+侧 `web_search` 的 `allowed_hosts` 数组字段同名透传。
+规则（全部由 `cloud/websearch.py` 的
+`_normalize_allowed_hosts` 在发任何 HTTP 之前强制）：
+
+- 只在 `scope="code"` 下生效；`scope="general"` 下传
+  `allowed_hosts` 直接报 `ValueError`，绝不静默忽略；
+  不传时行为与本版之前完全一致（仍是 15 域）。
+- 生效清单 = 内置 15 域 + 本次显式 extras（去重、保序）；
+  返回结果的 `code_domains` 即为该生效清单。
+- 匹配仍是点边界：子域名算（`api.docs.example-project.dev`
+  算），`evildocs.example-project.dev` 与
+  `docs.example-project.dev.evil.com` 都不算。
+- 空值 / 非法值报错不静默退化：`[]`、裸字符串
+  （可迭代但不是域名列表）、`*` 通配、带 scheme /
+  端口 / 路径 / userinfo 的值、前导点、空标签、非字符串
+  元素、以及全部等于内置域（即“扩展了个寂寞”）都报
+  `ValueError`，且此时尚未发任何 HTTP。
+- 扩展值绝不写进日志：错误串与 `attempted` 条目里不
+  出现扩展值；扩展值只出现在调用方自己那次结果的
+  `code_domains` 回显里（调用方自己指定的值回显给
+  自己，不是日志）。
+- 与原生后端限定方式的关系，和现有逻辑一致：原生
+  支持的后端（Tavily / Exa 的 `include_domains`）直接传
+  生效清单；Perplexity 的 `search_domain_filter` 上限
+  20 个域，内置 15 + 扩展超 20 时跳过其原生过滤、只靠
+  事后过滤（事后过滤永远是权威，语义不变）；原生不
+  支持的后端（Brave、SearXNG、DuckDuckGo）本来就不传
+  参数，只靠 `search()` 返回后的统一事后过滤。
+- `web_fetch` 不受 `allowed_hosts` 影响：fetch 本来就是
+  按指定 URL 抓取，没有域名白名单可扩展。
+
 ### 抓取回退链与 403 语义（web_fetch）
 
 `web_fetch` 实际调 `fetch_best`，顺序固定：
@@ -230,5 +280,73 @@ filter 行须用 POSIX BASIC 正则（`^\(.*\.\)*域名$`）、
   主机 unit / filter / env 接线、GitHub API token
   （本设计走网页搜索后端限定域名，不调 GitHub API，
   所以不需要 token；真要 GitHub 代码内容搜索 API 是
-  另一功能，不在本轨）、清单外项目官网域、老租户
-  脚本升级。
+  另一功能，不在本轨）、老租户脚本升级。清单外项目
+  官网域本轨已做（见上节 `allowed_hosts`，本地提交、
+  未推未部署），仍缺真实 key 的 live 验证。
+
+## JS 重型站点抽取效果评估（本轨新增，mock 为主）
+
+评估对象是已有抽取链，不改默认抽取顺序、不因评估换
+后端：`fetch_best` 固定为 Tavily extract → Exa contents
+→ 直接抓取。结论只写在这里，不改代码行为。
+测试文件：`cloud/tests/test_js_heavy_extract.py`（全部
+mock HTTP，只有一个显式开关的 live 烟囱测试，默认跳过）。
+
+### 阶段 1：直接抓取能拿到什么、拿不到什么
+
+直接抓取是标准库 HTML 解析：`<script>` / `<style>` /
+`<noscript>` 内容一律剥离，只留静态文本与绝对 http(s)
+链接（相对链接不进 `links` 列表）。所以：
+
+- 能拿到：静态壳里的标题、占位文本（如 "Loading…"）、
+  静态正文、绝对链接；首屏静态内容本身就是正文的页面
+  不受影响。
+- 拿不到：正文靠 JS 渲染的段落（mock 壳里
+  `RENDERED-BODY-PARAGRAPH` 只存在于 `<script>` 数据块
+  中，直接抓取结果里没有它，也不把数据块原文泄漏进
+  文本）；分页靠 JS 加载的后续项（mock 里第 2 页
+  `ITEM-PAGE-2` 只经 `/api/items?page=2` 到达，直接抓取
+  只有第 1 页静态项，也拿不到该 API 地址本身）。
+- 为什么：解析器没有 JS 引擎，也从不执行脚本或追
+  JS 发起的请求；这是结构性缺失，不是解析器 bug，
+  调解析参数补不回来。直接抓取在 JS 重型站上只能
+  给出“壳 + 首屏静态部分”，不能谎称是全文。
+
+### 阶段 2：provider 抽取在 mock 层面能补什么、补不了什么
+
+mock 层面（伪造的 Tavily extract 响应返回完整渲染文本）
+证明链路本身能补：`fetch_via_provider` 返回
+`source=tavily-extract` 的完整文本；抽取与直接都失败时
+`fetch_best` 回退到直接抓取的壳（`source=direct-fetch`），
+仍拿不到渲染正文——回退保证的是“不断链”，不是“补全文”。
+
+mock 补不了、也证明不了的部分：
+
+- 真实 provider 是否真的渲染该站的 JS（取决于各家
+  extract 实现是否跑无头浏览器、等多久、反爬策略），
+  mock 响应是“我假设它渲染了”，不是实测证据。
+- JS 分页 / 交互（点“加载更多”、登录后内容、滚动加载）
+  provider 抽取一般只取单 URL 的渲染结果，不会替调用方
+  翻页或交互；多页内容仍需逐 URL fetch。
+- 抽取返回空时是失败，不是空页：`fetch_via_provider`
+  空结果报 `WebSearchError`，不伪造页面；但 `fetch_best`
+  随后回退直接抓取会返回壳文本——调用方须以 `source`
+  字段分辨“渲染全文”与“壳”，不能只看 `text` 非空就当全文。
+
+### Live 对照（与 mock 分开写，未验证的不写成已验证）
+
+- 无 key 的 SearXNG 路线：本轨在 VM 测试环境实测一次
+  （`SEARXNG_URL=https://searx.diante.me`，`provider="searxng"`，
+  产品自身 `search()` 代码）：失败，`URLError: Remote end
+  closed connection without response`。这是在本 VM 出口下
+  的一次结果，不能据此断言 SearXNG 服务本身挂了（生产
+  文档记载主机侧经代理 pinned 引擎曾 0.47s 通），live
+  对照在本测试环境记为**未完成**，需在主机侧可达的环境
+  重测；测试文件里的 live 烟囱测试默认跳过，只在
+  `WEBSEARCH_EVAL_LIVE_SEARXNG=1` 且配了 `SEARXNG_URL`
+  时才跑。
+- 付费后端 live 效果（Tavily extract / Exa contents /
+  Perplexity / Brave 的真实 JS 渲染能力）：无 key，本轨
+  **一律未验证**，不编造效果，逐项列为 blocker（见回执），
+  在拿到 key 之前，文档里关于 provider 渲染能力的任何
+  表述都只是 mock 层面的链路结论。