host
621c08d10f
B1: gateway capability check now triggers on walking the sub2api gateway, not on the gateway having filled a model; task-level explicit models are audited (worker_capability_gateway_explicit) but never blocked, preserving explicit priority; gateway-default models still fail-fast under active. I1: dry-run adds a gateway-model section, sourced only from CLOUD_SUB2API_DEFAULT_MODEL (never queries the relay); an active tier whose requires the gateway model fails or cannot verify blocks, shadow only warns. I2: shadow review gate now counts completed samples (completed >= 20), same basis for sample_shortfall; the endpoint attaches per-tier capability status (read-only policy load) and flags capability-failing tiers in the recommendation. I3: configured-but-broken routing policy logs a warning in load_routing_policy (unconfigured stays silent); the worker sets routing_policy_load_failed and audits worker_routing_policy_load_failed per task. I4: resume preamble wraps prior output/error as data with an explicit do-not-execute constraint; position verified in run_one (tree -> preamble -> project -> original) and the docs corrected from the false 'closest to the original prompt' claim. I5: decided model is written back with a CAS UPDATE (model IS NULL OR model=''), so later attempts reuse it; explicit models are never overwritten. I6: agent_* columns added to worker._ensure_result_columns and db.init_postgres_schema backfill; migrations backfills them for tasks before insert (sender_task_id pattern); column-level tests for agent_tool_approvals and task_attempts. I7: coordinator requeue adds AND status=<snapshot> (rowcount 0 skips requeue/audit) and filters paused_by_user / user_cancel / user_pause children, which are never revived or cleared. I8: explicit-model tasks (including coordinator children that inherited a parent model) audit a source=explicit worker_routing row; evaluate_shadow counts it as per-tier explicit_n only, never routed_n, never a dedup slot. M1/M2/M3/M5/M6/M8/M10: dedup slot moved after tier validation, shadow sort doc fixed, truncated recommendation notes truncation, seed cleanup on gateway/quota/goal-cap early returns, tools-only coordinator definitions rejected, preamble goal branch marked effectively unreachable, DESIGN.md drift corrected (outside repo). M4/M7/M9: documentation/new-field only (pass_rate_fully_measured, whitespace-model shadow-drop note, bound-note race note); old field semantics unchanged.
406 lines
18 KiB
Python
406 lines
18 KiB
Python
"""Routing policy dry-run (v0d): validate before applying, never apply.
|
||
|
||
A routing policy takes effect the moment a worker loads it from the
|
||
host env -- there is no CI gate for an env change. A typo in a tier
|
||
mode silently drops that tier, a bad model name is never checked
|
||
against a whitelist (models are not whitelisted by design), and a
|
||
policy file that fails to parse is silently ignored by the loader
|
||
(fail-closed to the worker default, but also silently). This module
|
||
surfaces all of that BEFORE the policy is applied, and it never
|
||
applies anything itself: it only reports what WOULD happen if the
|
||
given policy document were loaded.
|
||
|
||
What it reports, per tier (plan / goal / default):
|
||
* the effective model and mode, and whether the mode came from an
|
||
explicit tier mode or from the global mode,
|
||
* tiers that were dropped, and why (invalid tier mode, no model,
|
||
unrecognised entry) -- a dropped tier is a BLOCKER, not a warning
|
||
to skim past, because applying the policy would silently not route
|
||
that tier,
|
||
* tiers absent from the document (worker default, not routed),
|
||
* unknown tier names in the document (ignored by the parser),
|
||
* whether any tier would actually activate on next dispatch,
|
||
* capability checks (v8): per tier, whether its model meets
|
||
its requires against the policy's model_capabilities
|
||
directory -- an active tier that fails or cannot be verified
|
||
is a BLOCKER (it would block dispatch at the worker
|
||
capability gate); a shadow/off tier with the same problem
|
||
is a warning only.
|
||
* gateway model check (Audit fix 3, I1): the same requires,
|
||
checked against the sub2api gateway model -- the model
|
||
production actually dispatches, which is NOT the tier
|
||
model. The gateway model comes only from env
|
||
(CLOUD_SUB2API_DEFAULT_MODEL); this module never calls
|
||
the relay to list models. An active tier whose requires
|
||
the gateway model fails or cannot verify is a BLOCKER;
|
||
a shadow tier is a warning only. "运营方须先把网关模型
|
||
写进能力目录" is now a checkable rule, not a comment.
|
||
|
||
What it does NOT do, honestly:
|
||
* It does NOT validate model names against a provider or relay
|
||
catalog. Task models are deliberately not whitelisted in this
|
||
codebase (the relay catalog changes), so a model typo here is
|
||
reported as "unverified, not whitelisted" only if the caller
|
||
passes a known-model set; without one, model names are reported
|
||
as-is and the caller must check them against the relay catalog
|
||
themselves. This module never invents a catalog.
|
||
* It does NOT check shadow sample counts. That is the shadow
|
||
report's job (GET /routing-shadow); a dry-run that said "ready
|
||
to activate" without sample data would be fabrication.
|
||
* It does NOT change host env, worker state, or any file. The CLI
|
||
exits 1 when the policy is invalid or any tier was dropped or
|
||
unknown, so a deploy script can gate on it; warnings alone
|
||
(no version, fallback_models present) do not block, but are
|
||
always printed.
|
||
|
||
Loading order, mirrored from load_routing_policy in models.py:
|
||
file (CLOUD_ROUTING_POLICY_FILE) first, then JSON
|
||
(CLOUD_ROUTING_POLICY_JSON) overrides whole tier entries (not
|
||
per-field: a JSON tier entry replaces the file's tier entry
|
||
entirely), then CLOUD_ROUTING_MODE overrides the GLOBAL mode
|
||
only -- it never overrides an explicit tier mode. Failed file/JSON
|
||
sources, which the loader silently ignores, are reported here as
|
||
warnings instead of being swallowed.
|
||
"""
|
||
|
||
import argparse
|
||
import json
|
||
import os
|
||
import sys
|
||
|
||
try:
|
||
from cloud.models import ROUTING_MODES, ROUTING_TIERS
|
||
from cloud.models import parse_routing_policy
|
||
from cloud.models import parse_tier_requires
|
||
from cloud.models import check_model_capabilities
|
||
except ImportError: # run as a script from the repo root
|
||
sys.path.insert(0, os.path.join(
|
||
os.path.dirname(os.path.abspath(__file__)), ".."))
|
||
from cloud.models import ROUTING_MODES, ROUTING_TIERS
|
||
from cloud.models import parse_routing_policy
|
||
from cloud.models import parse_tier_requires
|
||
from cloud.models import check_model_capabilities
|
||
|
||
|
||
def _tier_dropped_reason(raw_entry):
|
||
if isinstance(raw_entry, dict):
|
||
if "mode" in raw_entry and \
|
||
raw_entry.get("mode") not in ROUTING_MODES:
|
||
return "invalid tier mode %r" % (raw_entry.get("mode"),)
|
||
if "requires" in raw_entry and \
|
||
parse_tier_requires(raw_entry.get("requires")) is None:
|
||
return "invalid tier requires %r" % (raw_entry.get("requires"),)
|
||
return "no usable model"
|
||
if isinstance(raw_entry, str):
|
||
return "empty model" if not raw_entry.strip() else "unknown"
|
||
return "unrecognised tier entry (expected string or dict)"
|
||
|
||
|
||
def dry_run_policy_document(data, env_mode_override=None,
|
||
gateway_model=None):
|
||
"""Dry-run one merged policy document (pure, no IO).
|
||
|
||
``data`` is the merged document as parse_routing_policy would
|
||
receive it. ``env_mode_override`` is the raw CLOUD_ROUTING_MODE
|
||
value (or None): a valid value overrides the global mode only,
|
||
an invalid value is ignored with a warning. ``gateway_model``
|
||
is the sub2api gateway model from env only (or None when
|
||
unset); it is never fetched from the relay here.
|
||
"""
|
||
warnings = []
|
||
tiers_report = {}
|
||
dropped = []
|
||
unknown = []
|
||
_gw = {"model": gateway_model,
|
||
"source": ("CLOUD_SUB2API_DEFAULT_MODEL"
|
||
if gateway_model else "env_unset"),
|
||
"tiers": {}}
|
||
if not isinstance(data, dict):
|
||
return {
|
||
"valid": False, "blocked": True, "version": None,
|
||
"global_mode": None, "global_mode_source": None,
|
||
"tiers": tiers_report, "dropped_tiers": dropped,
|
||
"unknown_tiers": unknown,
|
||
"warnings": ["policy document is not an object; "
|
||
"the loader would ignore it entirely"],
|
||
"would_activate": [], "would_shadow": [], "would_off": [],
|
||
"gateway": _gw,
|
||
}
|
||
raw_tiers = data.get("tiers")
|
||
if isinstance(raw_tiers, dict):
|
||
for name in sorted(raw_tiers):
|
||
if name not in ROUTING_TIERS:
|
||
unknown.append(name)
|
||
warnings.append(
|
||
"unknown tier %r is ignored by the parser" % name)
|
||
policy = parse_routing_policy(data)
|
||
if policy is None:
|
||
warnings.append(
|
||
"policy is unusable (no usable tiers, or global mode "
|
||
"outside off/shadow/active); the loader would ignore it "
|
||
"entirely and workers would keep the worker default")
|
||
return {
|
||
"valid": False, "blocked": True, "version": None,
|
||
"global_mode": None, "global_mode_source": None,
|
||
"tiers": tiers_report, "dropped_tiers": dropped,
|
||
"unknown_tiers": unknown, "warnings": warnings,
|
||
"would_activate": [], "would_shadow": [], "would_off": [],
|
||
"gateway": _gw,
|
||
}
|
||
global_mode = policy.mode
|
||
global_source = "policy"
|
||
if env_mode_override:
|
||
if env_mode_override in ROUTING_MODES:
|
||
global_mode = env_mode_override
|
||
global_source = "env_override"
|
||
warnings.append(
|
||
"CLOUD_ROUTING_MODE overrides the global mode to %r, "
|
||
"but does NOT override explicit tier modes" %
|
||
env_mode_override)
|
||
else:
|
||
warnings.append(
|
||
"CLOUD_ROUTING_MODE %r is invalid and ignored; "
|
||
"the global mode from the policy stands" %
|
||
env_mode_override)
|
||
# Apply the env global override to the effective per-tier modes
|
||
# without mutating the parsed policy: a tier with an explicit
|
||
# mode keeps it, a tier without one follows the overridden global.
|
||
effective_policy_mode = global_mode
|
||
cap_blocked = False
|
||
for tier in ROUTING_TIERS:
|
||
raw_entry = raw_tiers.get(tier) if isinstance(raw_tiers,
|
||
dict) else None
|
||
if tier not in policy.tiers:
|
||
if raw_entry is not None:
|
||
reason = _tier_dropped_reason(raw_entry)
|
||
dropped.append(tier)
|
||
warnings.append(
|
||
"tier %r was dropped: %s; applying this policy "
|
||
"would silently not route %r tasks" %
|
||
(tier, reason, tier))
|
||
tiers_report[tier] = {
|
||
"model": None, "mode": None,
|
||
"mode_source": None, "status": "dropped",
|
||
"dropped_reason": reason,
|
||
}
|
||
else:
|
||
tiers_report[tier] = {
|
||
"model": None, "mode": None,
|
||
"mode_source": None, "status": "absent",
|
||
"dropped_reason": None,
|
||
}
|
||
continue
|
||
if tier in policy.tier_modes:
|
||
mode = policy.tier_modes[tier]
|
||
mode_source = "tier_mode"
|
||
else:
|
||
mode = effective_policy_mode
|
||
mode_source = "global_mode"
|
||
status = "active" if mode == "active" else (
|
||
"shadow" if mode == "shadow" else "off")
|
||
# Capability check (v8): the same check decide_routing
|
||
# runs, reported before activation. "not_applicable"
|
||
# when the tier carries no requirements; "unknown" when
|
||
# requirements exist but the model (or a needed field)
|
||
# is absent from the directory.
|
||
requires = policy.tier_requirements.get(tier) or {}
|
||
if not requires:
|
||
cap = {"required": None, "missing": [], "unknown": False,
|
||
"status": "not_applicable"}
|
||
else:
|
||
cap_ok, cap_missing, cap_unknown = check_model_capabilities(
|
||
policy.model_capabilities, policy.tiers[tier], requires)
|
||
cap = {"required": requires, "missing": cap_missing,
|
||
"unknown": cap_unknown,
|
||
"status": "ok" if cap_ok else (
|
||
"unknown" if cap_unknown else "fail")}
|
||
tiers_report[tier] = {
|
||
"model": policy.tiers[tier], "mode": mode,
|
||
"mode_source": mode_source, "status": status,
|
||
"dropped_reason": None, "capabilities": cap,
|
||
}
|
||
if cap["status"] in ("fail", "unknown"):
|
||
note = ("tier %r model %r %s capability: %s" % (
|
||
tier, policy.tiers[tier],
|
||
"cannot verify" if cap["status"] == "unknown"
|
||
else "lacks",
|
||
", ".join(cap["missing"])))
|
||
if status == "active":
|
||
# An active tier that fails/unknowns its capability
|
||
# check would block dispatch (worker capability
|
||
# gate), so it blocks the dry-run too.
|
||
cap_blocked = True
|
||
warnings.append(note + "; applying this policy "
|
||
"would block %r tasks at dispatch" % tier)
|
||
else:
|
||
warnings.append(note + "; %s mode does not block "
|
||
"dispatch" % status)
|
||
# Audit fix 3（I1）：网关模型检查——生产实际派发的
|
||
# 是网关模型，不是档模型；档模型干跑通过不代表
|
||
# 网关模型能过。只查 active/shadow 档（worker 的
|
||
# 网关检查只在这两种模式下跑，off 档不查）；网关
|
||
# 模型未在 env 里设定时记 unknown（live 回落会去
|
||
# 继电器查首个模型，干跑绝不联网，所以只能是
|
||
# unknown，不能假装 ok）。
|
||
_gw_blocked = False
|
||
for tier in ROUTING_TIERS:
|
||
_tstatus = tiers_report[tier]["status"]
|
||
if _tstatus not in ("active", "shadow"):
|
||
_gw["tiers"][tier] = {
|
||
"status": "not_checked", "missing": [],
|
||
"unknown": False, "tier_mode": _tstatus}
|
||
continue
|
||
_req = policy.tier_requirements.get(tier) or {}
|
||
if not _req:
|
||
_gw["tiers"][tier] = {
|
||
"status": "not_applicable", "missing": [],
|
||
"unknown": False, "tier_mode": _tstatus}
|
||
continue
|
||
if gateway_model is None:
|
||
_gw["tiers"][tier] = {
|
||
"status": "unknown", "missing": [],
|
||
"unknown": True, "tier_mode": _tstatus}
|
||
_note = ("tier %r requires %s but the gateway model"
|
||
" is not set in CLOUD_SUB2API_DEFAULT_MODEL"
|
||
" (and this dry-run never queries the relay)"
|
||
" -- cannot verify it against the capability"
|
||
" directory" % (tier, sorted(_req)))
|
||
else:
|
||
_ok, _missing, _unknown = check_model_capabilities(
|
||
policy.model_capabilities, gateway_model, _req)
|
||
_gw["tiers"][tier] = {
|
||
"status": "ok" if _ok else (
|
||
"unknown" if _unknown else "fail"),
|
||
"missing": _missing, "unknown": _unknown,
|
||
"tier_mode": _tstatus}
|
||
_note = ("tier %r gateway model %r %s capability:"
|
||
" %s" % (tier, gateway_model,
|
||
"cannot verify" if _unknown else "lacks",
|
||
", ".join(_missing)))
|
||
if _gw["tiers"][tier]["status"] in ("fail", "unknown"):
|
||
if _tstatus == "active":
|
||
_gw_blocked = True
|
||
warnings.append(_note + "; applying this policy "
|
||
"would block %r tasks at the gateway"
|
||
" gate" % tier)
|
||
else:
|
||
warnings.append(_note + "; shadow mode does not"
|
||
" block dispatch")
|
||
would_activate = [t for t in ROUTING_TIERS
|
||
if tiers_report[t]["status"] == "active"]
|
||
would_shadow = [t for t in ROUTING_TIERS
|
||
if tiers_report[t]["status"] == "shadow"]
|
||
would_off = [t for t in ROUTING_TIERS
|
||
if tiers_report[t]["status"] == "off"]
|
||
if policy.version is None:
|
||
warnings.append(
|
||
"no policy version: policy changes will be hard to "
|
||
"attribute in shadow reports (decisions record "
|
||
"policy_version)")
|
||
if data.get("fallback_models") is not None:
|
||
warnings.append(
|
||
"fallback_models in the policy are parsed but NOT "
|
||
"applied; only worker CLI --fallback-models enables a "
|
||
"real fallback retry")
|
||
if would_activate:
|
||
warnings.append(
|
||
"activation warning: tier(s) %s would actually route "
|
||
"tasks on next worker dispatch if this policy were "
|
||
"applied; this dry-run does not activate anything" %
|
||
", ".join(would_activate))
|
||
blocked = bool(dropped or unknown or cap_blocked or _gw_blocked)
|
||
return {
|
||
"valid": True, "blocked": blocked,
|
||
"version": policy.version, "global_mode": global_mode,
|
||
"global_mode_source": global_source,
|
||
"tiers": tiers_report, "dropped_tiers": dropped,
|
||
"unknown_tiers": unknown, "warnings": warnings,
|
||
"would_activate": would_activate,
|
||
"would_shadow": would_shadow, "would_off": would_off,
|
||
"gateway": _gw,
|
||
}
|
||
|
||
|
||
def dry_run_from_env(env=None):
|
||
"""Dry-run from an env mapping, mirroring load_routing_policy.
|
||
|
||
Returns the report dict. File/JSON failures that the loader
|
||
silently ignores are reported as warnings here.
|
||
"""
|
||
env = env if env is not None else os.environ
|
||
warnings = []
|
||
merged = None
|
||
path = env.get("CLOUD_ROUTING_POLICY_FILE") or ""
|
||
if path:
|
||
try:
|
||
with open(path, encoding="utf-8") as fh:
|
||
merged = json.load(fh)
|
||
except (OSError, ValueError) as exc:
|
||
warnings.append(
|
||
"policy file %r could not be read/parsed (%s); "
|
||
"the loader would ignore it" % (path, exc))
|
||
merged = None
|
||
raw = env.get("CLOUD_ROUTING_POLICY_JSON") or ""
|
||
if raw:
|
||
try:
|
||
override = json.loads(raw)
|
||
except ValueError as exc:
|
||
warnings.append(
|
||
"CLOUD_ROUTING_POLICY_JSON could not be parsed "
|
||
"(%s); the loader would ignore it" % exc)
|
||
override = None
|
||
if isinstance(override, dict):
|
||
if not isinstance(merged, dict):
|
||
merged = override
|
||
else:
|
||
merged = dict(merged)
|
||
merged.update(override)
|
||
if isinstance(merged.get("tiers"), dict) and \
|
||
isinstance(override.get("tiers"), dict):
|
||
tiers = dict(merged["tiers"])
|
||
tiers.update(override["tiers"])
|
||
merged["tiers"] = tiers
|
||
_gw_model = (env.get("CLOUD_SUB2API_DEFAULT_MODEL") or "").strip()
|
||
report = dry_run_policy_document(
|
||
merged, env_mode_override=env.get("CLOUD_ROUTING_MODE") or None,
|
||
gateway_model=_gw_model or None)
|
||
# Source-failure warnings belong before the document warnings,
|
||
# and they do not change blocked (a failed source alone is not
|
||
# a dropped tier; the document report already covers validity).
|
||
report["warnings"] = warnings + report["warnings"]
|
||
return report
|
||
|
||
|
||
def main(argv=None):
|
||
parser = argparse.ArgumentParser(
|
||
description="Dry-run a routing policy (never applies it).")
|
||
parser.add_argument("--policy-file", default=None,
|
||
help="Policy JSON file (same as "
|
||
"CLOUD_ROUTING_POLICY_FILE)")
|
||
parser.add_argument("--policy-json", default=None,
|
||
help="Policy JSON string (same as "
|
||
"CLOUD_ROUTING_POLICY_JSON)")
|
||
parser.add_argument("--mode", default=None,
|
||
help="Global mode override (same as "
|
||
"CLOUD_ROUTING_MODE)")
|
||
parser.add_argument("--gateway-model", default=None,
|
||
help="Gateway model (same as "
|
||
"CLOUD_SUB2API_DEFAULT_MODEL)")
|
||
args = parser.parse_args(argv)
|
||
env = {}
|
||
if args.policy_file:
|
||
env["CLOUD_ROUTING_POLICY_FILE"] = args.policy_file
|
||
if args.policy_json:
|
||
env["CLOUD_ROUTING_POLICY_JSON"] = args.policy_json
|
||
if args.mode:
|
||
env["CLOUD_ROUTING_MODE"] = args.mode
|
||
if args.gateway_model:
|
||
env["CLOUD_SUB2API_DEFAULT_MODEL"] = args.gateway_model
|
||
report = dry_run_from_env(env)
|
||
print(json.dumps(report, indent=2, sort_keys=True))
|
||
return 1 if (not report["valid"] or report["blocked"]) else 0
|
||
|
||
|
||
if __name__ == "__main__":
|
||
raise SystemExit(main())
|