fix: audit batch 2 visibility (I-1 shadow dropped rows, I-2 partial policy failure, I-5 failure policy table)
- I-1: evaluate_shadow counts dropped rows by cause (unparseable / unknown_tier / explicit / duplicate) with the identity audit_rows_total == routed_task_n + dropped_total; the shadow endpoint passes created_at/id through and reports audit totals next to sample counts. - I-2 (option b): parse logs a warning naming each tier dropped for an invalid mode/requires; load attaches partial_failures to the policy; the worker records a task-level worker_routing_policy_load_failed audit for partial failures (same action as full failure, detail says failure=partial); the shadow report carries the policy fingerprint, dropped tiers and partial failures (no task context on that path, so no task-level audit there). - I-5: coordinator gate audit lookup failure now raises _CoordGateLookupFailed and the merge converts it to escalate (no requeue, no round consumed); persisted-model lookup failure and required-tools check failure each write a named audit row before falling through to dispatch/allow; audit-loop evaluate failure stays escalate (regression test). - Test fixtures: coordinator suites build audit_log via AuthStore as production's first real audit does, since their stubbed _audit never creates the table.
This commit is contained in:
1 parent
44e0f4f1d6
commit
a7f77c8a98
7 files changed
+648
-40
No files matched your search
+34
-7
@@ -68,13 +68,11 @@ def parse_worker_routing_detail(detail):
|
||||
|
||||
Old format (no suggested/chosen) parses with both models None.
|
||||
A ``-`` placeholder also means None, never a model named "-".
|
||||
9.5-code fail-open 清单:解析失败的行返回 None 后在
|
||||
evaluate_shadow 里直接跳过、不计数、不报错——样本无
|
||||
声消失(例如模型名含空白会使整行匹配失败,见 M7)。
|
||||
这是已知缺口:改返回类型只需同步评估函数的唯一生产
|
||||
调用方(server 影子端点)与测试,本轨只在此处写明,
|
||||
不改语义;运营方无法从报告里看到丢了多少行,只能靠
|
||||
审计行总数与样本数对不上来发现。
|
||||
Audit batch 2 (I-1): 解析语义本身不变(失败仍返回 None),
|
||||
但 evaluate_shadow 不再静默跳过:不可解析的行计入
|
||||
dropped_rows["unparseable"],报告里可见(例如模型名含
|
||||
空白会使整行匹配失败,见 M7——行不再无声消失,只是
|
||||
不计为样本)。
|
||||
"""
|
||||
if not isinstance(detail, str):
|
||||
return None
|
||||
@@ -144,10 +142,23 @@ def evaluate_shadow(audit_rows, tasks_by_id, costs_by_task,
|
||||
"unpriced_tokens": int}. ``explicit_n``: number of tasks
|
||||
with a creation-time model in the same project (counted
|
||||
separately, never as routed samples).
|
||||
Audit batch 2 (I-1): every input row lands in exactly one
|
||||
bucket — a routed sample, or ``dropped_rows``. Dropped
|
||||
buckets are the real skip reasons in the loop below:
|
||||
``unparseable`` (detail does not match the format),
|
||||
``unknown_tier`` (tier is not a known tier), ``explicit``
|
||||
(source=explicit rows, also counted per-tier as explicit_n,
|
||||
never as samples), ``duplicate`` (a later row for an
|
||||
already-counted task). Identity: ``audit_rows_total ==
|
||||
routed_task_n + dropped_total`` — the caller can reconcile
|
||||
the report against the audit rows it fetched instead of
|
||||
guessing whether samples were lost.
|
||||
"""
|
||||
tiers = {tier: _empty_tier() for tier in TIERS}
|
||||
seen = set()
|
||||
routed_task_ids = set()
|
||||
dropped_rows = {"unparseable": 0, "unknown_tier": 0,
|
||||
"explicit": 0, "duplicate": 0}
|
||||
rows = list(audit_rows or [])
|
||||
if any(isinstance(r, dict) and r.get("created_at") is not None
|
||||
for r in rows):
|
||||
@@ -169,6 +180,7 @@ def evaluate_shadow(audit_rows, tasks_by_id, costs_by_task,
|
||||
parsed = parse_worker_routing_detail(
|
||||
(row or {}).get("detail"))
|
||||
if parsed is None:
|
||||
dropped_rows["unparseable"] += 1
|
||||
continue
|
||||
task_id = parsed["task_id"]
|
||||
tier = parsed["tier"]
|
||||
@@ -178,14 +190,23 @@ def evaluate_shadow(audit_rows, tasks_by_id, costs_by_task,
|
||||
# occupy the dedup slot (Audit fix 3, M1): a task
|
||||
# whose first row is an unknown tier must not lose
|
||||
# its later known-tier row to dedup.
|
||||
# Audit batch 2 (I-1): the row is counted as dropped,
|
||||
# not silently skipped.
|
||||
dropped_rows["unknown_tier"] += 1
|
||||
continue
|
||||
if parsed["source"] == "explicit":
|
||||
# Audit fix 3(I8):显式行只计 per-tier
|
||||
# explicit_n:不计 routed_n、不占 seen(同一任务
|
||||
# 日后若出现路由行仍可计为样本),不破坏去重。
|
||||
# Audit batch 2 (I-1): explicit rows are also
|
||||
# counted in dropped_rows so the row identity
|
||||
# reconciles; they are not "lost" — they are
|
||||
# counted separately by design.
|
||||
tiers[tier]["explicit_n"] += 1
|
||||
dropped_rows["explicit"] += 1
|
||||
continue
|
||||
if task_id in seen:
|
||||
dropped_rows["duplicate"] += 1
|
||||
continue
|
||||
seen.add(task_id)
|
||||
bucket = tiers[tier]
|
||||
@@ -250,6 +271,12 @@ def evaluate_shadow(audit_rows, tasks_by_id, costs_by_task,
|
||||
"tiers": tiers,
|
||||
"explicit_n": int(explicit_n or 0),
|
||||
"routed_task_n": len(routed_task_ids),
|
||||
# Audit batch 2 (I-1): row reconciliation. Every input
|
||||
# row is either a routed sample or in dropped_rows, so
|
||||
# audit_rows_total == routed_task_n + dropped_total.
|
||||
"audit_rows_total": len(rows),
|
||||
"dropped_rows": dropped_rows,
|
||||
"dropped_total": sum(dropped_rows.values()),
|
||||
"counterfactual": "not_measured",
|
||||
"counterfactual_note": (
|
||||
"Shadow suggested models never ran, so this report cannot "
|
||||
|
||||
Reference in new issue
Block a user