websearch: task-level allowed_hosts extension for scope=code + JS-heavy extract evaluation (mock-first)
- search(..., allowed_hosts=[...]) extends (never replaces) the 15-domain code list for that call only; scope=general + allowed_hosts raises, empty/illegal values raise before any HTTP, dot-boundary matching, extension values never in error strings or attempted entries. - Native-filter parity: Tavily/Exa get the effective list, Perplexity skips its native filter past its 20-domain cap (post-filter stays authoritative), Brave/SearXNG/DuckDuckGo rely on post-filtering. - MCP web_search gains allowed_hosts passthrough (only when supplied, so older search_fn fakes keep working). - JS-heavy evaluation documented in docs/web-search-mcp.md, mock-first: direct fetch is shell-only by construction; provider extract can supply rendered text in mock only; live SearXNG in this VM env failed (remote closed connection) and paid-backend live effects are unvalidated blockers, not claimed. - Tests: test_code_search_ext (9) + test_js_heavy_extract (5 + 1 live skipped); existing websearch/code-search suites still green. (cherry picked from commit 9c26bbe848438e1f34bcb000700aeb4c284f4f23)
This commit is contained in:
1 parent
125b436ab3
commit
e9fa43dd8b
5 files changed
+630
-12
No files matched your search
+30
-2
@@ -42,7 +42,12 @@ _TOOLS = [
|
||||
"only (official language docs, GitHub, package "
|
||||
"registries, Stack Overflow, CVE/NVD); anything else "
|
||||
"is filtered out, never returned. Default scope "
|
||||
"'general' is unfiltered. This tool does not do "
|
||||
"'general' is unfiltered. With scope 'code' only, "
|
||||
"allowed_hosts may name extra project domains for "
|
||||
"this call (explicit allowlist extension, subdomains "
|
||||
"included, dot-boundary matched); it is not an "
|
||||
"arbitrary-site search, and it is rejected with "
|
||||
"scope 'general'. This tool does not do "
|
||||
"shopping, social, places, travel, general news, "
|
||||
"weather, finance, or browser checkout."
|
||||
),
|
||||
@@ -61,6 +66,21 @@ _TOOLS = [
|
||||
"search. code: programming "
|
||||
"sources only, see tool "
|
||||
"description."},
|
||||
"allowed_hosts": {"type": "array",
|
||||
"items": {"type": "string"},
|
||||
"description": "Only with scope "
|
||||
"'code': extra "
|
||||
"domains for this "
|
||||
"call only, on top "
|
||||
"of the builtin "
|
||||
"programming list. "
|
||||
"Omit for no "
|
||||
"extension; empty, "
|
||||
"wildcard, or "
|
||||
"non-hostname "
|
||||
"values are "
|
||||
"rejected, never "
|
||||
"silently ignored."},
|
||||
},
|
||||
"required": ["query"],
|
||||
},
|
||||
@@ -139,11 +159,19 @@ def handle_request(request, search_fn=None, fetch_fn=None):
|
||||
args = params.get("arguments") or {}
|
||||
try:
|
||||
if name == "web_search":
|
||||
# allowed_hosts is passed through only when the
|
||||
# caller supplied it, so older search_fn fakes
|
||||
# without that parameter keep working.
|
||||
search_kwargs = {}
|
||||
if "allowed_hosts" in args:
|
||||
search_kwargs["allowed_hosts"] = \
|
||||
args.get("allowed_hosts")
|
||||
result = search_fn(
|
||||
args.get("query"),
|
||||
limit=args.get("limit", 5),
|
||||
provider=args.get("provider", "auto"),
|
||||
scope=args.get("scope", "general"))
|
||||
scope=args.get("scope", "general"),
|
||||
**search_kwargs)
|
||||
elif name == "web_fetch":
|
||||
result = fetch_fn(
|
||||
args.get("url"),
|
||||
|
||||
Reference in new issue
Block a user