websearch: task-level allowed_hosts extension for scope=code + JS-heavy extract evaluation (mock-first)

- search(..., allowed_hosts=[...]) extends (never replaces) the 15-domain
  code list for that call only; scope=general + allowed_hosts raises,
  empty/illegal values raise before any HTTP, dot-boundary matching,
  extension values never in error strings or attempted entries.
- Native-filter parity: Tavily/Exa get the effective list, Perplexity
  skips its native filter past its 20-domain cap (post-filter stays
  authoritative), Brave/SearXNG/DuckDuckGo rely on post-filtering.
- MCP web_search gains allowed_hosts passthrough (only when supplied,
  so older search_fn fakes keep working).
- JS-heavy evaluation documented in docs/web-search-mcp.md, mock-first:
  direct fetch is shell-only by construction; provider extract can
  supply rendered text in mock only; live SearXNG in this VM env
  failed (remote closed connection) and paid-backend live effects are
  unvalidated blockers, not claimed.
- Tests: test_code_search_ext (9) + test_js_heavy_extract (5 + 1 live
  skipped); existing websearch/code-search suites still green.

(cherry picked from commit 9c26bbe848438e1f34bcb000700aeb4c284f4f23)
This commit is contained in:
host committed 2026-11-01 09:06:28 +00:00
1 parent 125b436ab3
commit e9fa43dd8b
5 files changed
+630 -12

No files matched your search

+30 -2
View File
@@ -42,7 +42,12 @@ _TOOLS = [
"only (official language docs, GitHub, package "
"registries, Stack Overflow, CVE/NVD); anything else "
"is filtered out, never returned. Default scope "
"'general' is unfiltered. This tool does not do "
"'general' is unfiltered. With scope 'code' only, "
"allowed_hosts may name extra project domains for "
"this call (explicit allowlist extension, subdomains "
"included, dot-boundary matched); it is not an "
"arbitrary-site search, and it is rejected with "
"scope 'general'. This tool does not do "
"shopping, social, places, travel, general news, "
"weather, finance, or browser checkout."
),
@@ -61,6 +66,21 @@ _TOOLS = [
"search. code: programming "
"sources only, see tool "
"description."},
"allowed_hosts": {"type": "array",
"items": {"type": "string"},
"description": "Only with scope "
"'code': extra "
"domains for this "
"call only, on top "
"of the builtin "
"programming list. "
"Omit for no "
"extension; empty, "
"wildcard, or "
"non-hostname "
"values are "
"rejected, never "
"silently ignored."},
},
"required": ["query"],
},
@@ -139,11 +159,19 @@ def handle_request(request, search_fn=None, fetch_fn=None):
args = params.get("arguments") or {}
try:
if name == "web_search":
# allowed_hosts is passed through only when the
# caller supplied it, so older search_fn fakes
# without that parameter keep working.
search_kwargs = {}
if "allowed_hosts" in args:
search_kwargs["allowed_hosts"] = \
args.get("allowed_hosts")
result = search_fn(
args.get("query"),
limit=args.get("limit", 5),
provider=args.get("provider", "auto"),
scope=args.get("scope", "general"))
scope=args.get("scope", "general"),
**search_kwargs)
elif name == "web_fetch":
result = fetch_fn(
args.get("url"),