fix: audit batch 2 visibility (I-1 shadow dropped rows, I-2 partial policy failure, I-5 failure policy table)
- I-1: evaluate_shadow counts dropped rows by cause (unparseable / unknown_tier / explicit / duplicate) with the identity audit_rows_total == routed_task_n + dropped_total; the shadow endpoint passes created_at/id through and reports audit totals next to sample counts. - I-2 (option b): parse logs a warning naming each tier dropped for an invalid mode/requires; load attaches partial_failures to the policy; the worker records a task-level worker_routing_policy_load_failed audit for partial failures (same action as full failure, detail says failure=partial); the shadow report carries the policy fingerprint, dropped tiers and partial failures (no task context on that path, so no task-level audit there). - I-5: coordinator gate audit lookup failure now raises _CoordGateLookupFailed and the merge converts it to escalate (no requeue, no round consumed); persisted-model lookup failure and required-tools check failure each write a named audit row before falling through to dispatch/allow; audit-loop evaluate failure stays escalate (regression test). - Test fixtures: coordinator suites build audit_log via AuthStore as production's first real audit does, since their stubbed _audit never creates the table.
This commit is contained in:
1 parent
44e0f4f1d6
commit
a7f77c8a98
7 files changed
+648
-40
No files matched your search
+36
-14
@@ -379,7 +379,16 @@ class RoutingPolicy:
|
||||
|
||||
def __init__(self, version=None, mode="shadow", tiers=None,
|
||||
fallback_models=None, tier_modes=None,
|
||||
model_capabilities=None, tier_requirements=None):
|
||||
model_capabilities=None, tier_requirements=None,
|
||||
dropped_tiers=None, partial_failures=None):
|
||||
# Audit batch 2 (I-2): tiers the parser dropped for an
|
||||
# invalid mode/requires, and source-level failures that
|
||||
# still left a usable policy. Both default to [] so
|
||||
# directly-constructed policies never raise AttributeError
|
||||
# on these fields; parse/load fill them in. They are
|
||||
# visibility only: they never change routing decisions.
|
||||
self.dropped_tiers = list(dropped_tiers or [])
|
||||
self.partial_failures = list(partial_failures or [])
|
||||
self.version = version if isinstance(version, str) \
|
||||
and version.strip() else None
|
||||
self.mode = mode if mode in ROUTING_MODES else "off"
|
||||
@@ -474,6 +483,12 @@ def parse_routing_policy(data):
|
||||
# accidentally activate, and never kills the other tiers.
|
||||
raw_tiers = data.get("tiers") if isinstance(data.get("tiers"),
|
||||
dict) else {}
|
||||
# Audit batch 2 (I-2): record every tier the fail-closed pops
|
||||
# actually remove, with a warning naming the tier. A pop that
|
||||
# finds the tier already absent (e.g. _clean_tiers dropped it
|
||||
# for an empty model) is not a second drop and is not logged:
|
||||
# the warning must name a tier this pop really removed.
|
||||
dropped_tiers = []
|
||||
tier_modes = {}
|
||||
for tier, entry in raw_tiers.items():
|
||||
if not isinstance(entry, dict) or "mode" not in entry:
|
||||
@@ -481,8 +496,12 @@ def parse_routing_policy(data):
|
||||
tier_mode = entry.get("mode")
|
||||
if tier_mode in ROUTING_MODES:
|
||||
tier_modes[tier] = tier_mode
|
||||
else:
|
||||
elif tier in tiers:
|
||||
tiers.pop(tier, None)
|
||||
dropped_tiers.append(tier)
|
||||
_logger.warning(
|
||||
"routing policy tier %r dropped: invalid mode %r",
|
||||
tier, tier_mode)
|
||||
if not tiers:
|
||||
return None
|
||||
# Per-tier requirements (v8): a tier entry dict may carry
|
||||
@@ -496,7 +515,12 @@ def parse_routing_policy(data):
|
||||
continue
|
||||
cleaned = parse_tier_requires(entry.get("requires"))
|
||||
if cleaned is None:
|
||||
tiers.pop(tier, None)
|
||||
if tier in tiers:
|
||||
tiers.pop(tier, None)
|
||||
dropped_tiers.append(tier)
|
||||
_logger.warning(
|
||||
"routing policy tier %r dropped: invalid requires",
|
||||
tier)
|
||||
elif cleaned:
|
||||
tier_requirements[tier] = cleaned
|
||||
if not tiers:
|
||||
@@ -510,7 +534,8 @@ def parse_routing_policy(data):
|
||||
tier_modes=tier_modes,
|
||||
model_capabilities=data.get(
|
||||
"model_capabilities"),
|
||||
tier_requirements=tier_requirements)
|
||||
tier_requirements=tier_requirements,
|
||||
dropped_tiers=dropped_tiers)
|
||||
|
||||
|
||||
def load_routing_policy(env=None):
|
||||
@@ -573,17 +598,14 @@ def load_routing_policy(env=None):
|
||||
" worker default", "; ".join(failures))
|
||||
return None
|
||||
if failures:
|
||||
# 9.5-code fail-open 清单:部分失败是 fail-open——
|
||||
# Audit batch 2 (I-2): 部分失败仍是 fail-open——
|
||||
# 可用来源继续生效、失败来源被丢弃,派发不炸。
|
||||
# 可见性只有这条警告日志:load_routing_policy 仍
|
||||
# 返回一个 policy,worker 的
|
||||
# routing_policy_load_failed 标志只在全失败时
|
||||
# 置位,所以部分失败不会产生任务级审计;运营方
|
||||
# 只能从 worker 日志看到它,不能从审计里查到。
|
||||
# 这是已知缺口(非静默,但不如全失败可见),
|
||||
# 本轨不改返回类型:生产调用点只有 2 处(server
|
||||
# 影子端点经 _load_routing_policy、worker 派发),
|
||||
# 真正的成本是同步改调用方与测试,而不是“牵动全部”。
|
||||
# 但可见性不再只有这条警告日志:失败清单挂在返回
|
||||
# 的 policy 上(partial_failures),worker 派发时
|
||||
# 据此记任务级审计(与全失败同款 action),影子
|
||||
# 报告里也附指纹与丢弃档;返回类型本身不变,
|
||||
# 仍是 RoutingPolicy 或 None。
|
||||
policy.partial_failures = list(failures)
|
||||
_logger.warning(
|
||||
"routing policy partially failed (%s); continuing with"
|
||||
" the usable source", "; ".join(failures))
|
||||
|
||||
Reference in new issue
Block a user