audit: unify audit-of-audit sampling rule (gap 2)

Single sampling rule; track 3 (#244) is authoritative for WHO is
mandatory, track 5 (#238) for HOW the rest are drawn:

- Mandatory (1.0): golden-task records + rebuild / refreeze /
  escalation records + third-failure near-complete still-failed
  cases (mandatory_triggers, full list per case). Track 3's old
  deterministic every-10th ordinary rule is removed (it drew zero
  from batches under 10 ordinary records and was predictable);
  track 5's old narrow mandatory list is expanded with golden /
  refreeze / escalate. blocked_quota stays non-mandatory.
- Random tiers: normal 0.10, critical-path 0.50, seed required
  and recorded, ceil so a non-empty group is never rounded to
  zero. 0.10/0.50 are TEST DEFAULTS, not approved production
  values; production rates remain a user decision.
- One pass, no double counting: mandatory records leave the
  random pools before drawing; a multi-trigger record (e.g.
  golden + rebuild) is sampled once with all triggers listed,
  and its correction follows the (original, corrected) verdict
  pair, never the trigger list.
- cloud/audit_pushback.py::select_audit_sample is now only a
  record-shaped adapter over cloud/audit_sampling.py::select_sample
  (seed required, retired reason names critical_path-for-kinds
  and periodic_sample removed); AUDIT_SAMPLE_EVERY_NTH deleted.
- Correction: original never mutated, correction appended,
  corrector != signer enforced in audit_sampling when both names
  are supplied; gate effect pinned (correction to pass never
  unblocks by itself -- schedule/cancel-and-rerun, blocked until
  a new passing re-audit; correction to fail re-blocks at once).
  Track 3 / track 5 retrigger vocabularies documented as synonyms.
- Skill text (tightening 3 item 2 + correction paragraph)
  rewritten to the unified rule; new test file pins the
  boundaries (mandatory union, dedup, ceil incl. the 9-record
  case, seed reproducibility, golden-set construction under both
  sample vocabularies, adapter order/non-mutation, correction
  preservation on both tracks).
This commit is contained in:
host committed 2026-11-03 02:49:25 +00:00
1 parent 8006a6fcba
commit a433460343
5 files changed
+546 -60

No files matched your search

+77 -25
View File
@@ -212,9 +212,10 @@ Tightening 3 -- the audit itself is audited. (i) Independent
re-runs use exact artifacts (#211 triple-hash), never reports
alone. (ii) Golden tasks (human-approved only; example/sample
files are rejected by ``validate_golden_task_set``) and
rebuild/refreeze/escalation records are always sampled, and
ordinary records are sampled every ``AUDIT_SAMPLE_EVERY_NTH``
records, deterministically (``select_audit_sample``). (iii) G6
rebuild/refreeze/escalation records are always sampled, and the
rest are drawn by seeded random sampling (``select_audit_sample``,
which only adapts ``cloud/audit_sampling.py::select_sample`` --
the unified sampling rule lives there, not here). (iii) G6
user acceptance is the final human gate and is never ticked by
automation (``check_g6_guard``). A sampled audit found wrong is
corrected by appending a correction record that preserves the
@@ -282,10 +283,15 @@ MAX_TARGETED_FIX_ROUNDS_PER_CONTRACT_VERSION = 1
# items, with no core-structure item and no broken passed item,
# may qualify as "nearly done". More than this is not near-complete.
NEAR_COMPLETE_MAX_FAILED_ITEMS = 2
# Audit-of-audit sampling (tightening 3): golden tasks and
# rebuild/refreeze/escalation records are always sampled; ordinary
# records are sampled deterministically every Nth record.
AUDIT_SAMPLE_EVERY_NTH = 10
# Audit-of-audit sampling (tightening 3): the unified rule lives in
# cloud/audit_sampling.py (mandatory = golden tasks + rebuild /
# refreeze / escalation + third-failure near-complete still failed,
# all at 1.0; the rest by seeded random draw, normal 0.10 /
# critical-path 0.50, ceil so a small group is never rounded to
# zero). The old deterministic "every 10th record" rule is
# superseded and removed: it drew zero from batches under 10
# ordinary records. The 0.10 / 0.50 ratios are TEST DEFAULTS, not
# approved production values; production rates are a user decision.
# Gate checks, in the order they are reported. The first nine are the
# hard trigger list from the audit-hardening brief; the tenth is the
@@ -1151,27 +1157,66 @@ def validate_golden_task_set(tasks):
return (not reasons, reasons)
def select_audit_sample(records, golden_task_ids):
def select_audit_sample(records, golden_task_ids, seed=None,
normal_ratio=None, critical_ratio=None):
"""Select audit records for audit-of-audit sampling (tightening 3).
Always sampled: golden tasks, and rebuild / refreeze /
escalation records (the critical paths). Ordinary records are
sampled deterministically every AUDIT_SAMPLE_EVERY_NTH record
(by position, so the sample is reproducible and checkable).
Returns a list of ``{"record": ..., "reason": ...}``.
ADAPTER, not a second rule: this function translates records
into ``cloud/audit_sampling.py`` cases, runs the unified
``select_sample`` there, and translates the plan back into
record order. The unified rule is: mandatory (1.0) = golden
tasks + rebuild / refreeze / escalation records + third
failure root-caused near-complete whose continuation round
still failed; the rest by seeded random draw (normal default
0.10, critical-path default 0.50, ceil so a non-empty group
is never rounded to zero). ``seed`` is REQUIRED -- a missing
seed raises ValueError, because an unseeded draw cannot be
reproduced on review. Selection is a single pass: a mandatory
record is removed from the random pools before drawing, so it
consumes no random quota, and a multi-trigger record is
returned ONCE with all its triggers in ``reasons`` (``reason``
is the first trigger, in golden / rebuild / third-failure /
refreeze / escalate order).
Retired reason names: track 3's old ``critical_path`` label
for mandatory kinds and ``periodic_sample`` label for every
10th record no longer exist -- mandatory kinds carry their own
trigger names, and random draws carry ``random_critical_path``
/ ``random_normal``. Input records are never mutated.
Returns a list of ``{"record": ..., "reason": ...,
"reasons": [...]}`` in original record order.
"""
golden = set(golden_task_ids or [])
from cloud.audit_sampling import select_sample as _select_sample
records = list(records or [])
cases = []
for index, record in enumerate(records):
case = dict(record)
# Unique per-record id for the plan; the golden check uses
# task_id, so sharing a task_id across records is fine.
case["id"] = "record-%d" % index
cases.append(case)
kwargs = {}
if normal_ratio is not None:
kwargs["normal_ratio"] = normal_ratio
if critical_ratio is not None:
kwargs["critical_ratio"] = critical_ratio
plan = _select_sample(cases, seed, golden_task_ids=golden_task_ids,
**kwargs)
reasons_by_id = {}
for entry in plan["mandatory"]:
reasons_by_id[entry["case_id"]] = list(entry["triggers"])
for case_id in plan["random_critical_path"]:
reasons_by_id[case_id] = ["random_critical_path"]
for case_id in plan["random_normal"]:
reasons_by_id[case_id] = ["random_normal"]
selected = []
for index, record in enumerate(records or []):
reason = None
if record.get("task_id") in golden:
reason = "golden_task"
elif record.get("kind") in ("rebuild", "refreeze", "escalate"):
reason = "critical_path"
elif (index + 1) % AUDIT_SAMPLE_EVERY_NTH == 0:
reason = "periodic_sample"
if reason:
selected.append({"record": record, "reason": reason})
for index, record in enumerate(records):
reasons = reasons_by_id.get("record-%d" % index)
if reasons:
selected.append({"record": record, "reason": reasons[0],
"reasons": list(reasons)})
return selected
@@ -1188,7 +1233,14 @@ def build_audit_correction(original_entry, corrected_verdict, corrector,
(should have blocked) re-triggers pushback and re-blocks;
a wrong block (should have passed) schedules a re-audit --
it does not jump straight to merge, because the correction
itself is a review, not a full audit.
itself is a review, not a full audit. This is the SAME pair
rule as ``cloud/audit_sampling.py::handle_sampling_result``,
only the vocabulary differs (there: wrong pass ->
``reopen_audit_and_pushback``, wrong block ->
``cancel_pushback_and_rerun_audit`` where "cancel" means
cancel-and-rerun, never cancel-and-merge); the gate effect is
identical and is pinned by the wiring track (#247): a
correction to ``pass`` never unblocks by itself.
"""
errors = []
signer = original_entry.get("signer")
+166 -24
View File
@@ -11,9 +11,31 @@ pushback/rebuild counters, or the golden-set approval flow; it only
defines which audit cases a human must re-examine, and what happens
when that re-examination finds the audit wrong.
UNIFIED SAMPLING RULE (single rule; this module is canonical for
selection, ``cloud/audit_pushback.py::select_audit_sample`` is only
its record-shaped adapter -- there is no second rule):
The scope of "mandatory" comes from track 3 (#244): golden-task
records and rebuild / refreeze / escalation records. The drawing
mechanism comes from track 5 (#238): seeded random draws with ceil
rounding. Track 3's old deterministic "every 10th record" rule is
superseded and removed: a batch of fewer than 10 ordinary records
drew zero under it, and a position-known draw is predictable. The
rates below are TEST DEFAULTS (``DEFAULT_*``), not approved
production values; the production rates are still a user decision.
Mandatory 1.0 is a rule, not a tunable rate.
Sampling rules (all testable, none left to "regular review"):
1. Mandatory sampling (ratio 1.0, never skipped):
1. Mandatory sampling (ratio 1.0, never skipped). A case is
mandatory when ANY of these triggers holds (trigger order for
display and for the legacy single-trigger helper is golden,
rebuild, third-failure, refreeze, escalate):
- ``golden_task``: the case's task is in the human-approved
golden set (``task_id`` when present, else ``id``). A golden
task's records are mandatory whatever their kind -- an
ordinary pass verdict on a golden task is sampled, not just
its failures.
- ``rebuild``: any case whose audit decision was rebuild-from-
scratch (推倒重做). A rebuild destroys the old implementation
in the workspace, so its verdict must always be re-examined.
@@ -22,18 +44,63 @@ Sampling rules (all testable, none left to "regular review"):
was given one targeted continuation round, and still failed.
This is the exact boundary where the "one more round vs
rebuild" judgement is most error-prone, so it is always sampled.
- ``refreeze``: a contract/design re-freeze record.
- ``escalate``: an escalation record.
A near-complete third failure whose continuation round SUCCEEDED
is not mandatory (the judgement proved itself), and a
near-complete first/second failure is not mandatory either (the
threshold is the third failure, per the rebuild rule).
threshold is the third failure, per the rebuild rule). A
``blocked_quota`` record is NOT mandatory under this rule (it
is not in track 3's mandatory kind list); it is sampled only
through the random tiers below.
2. Random sampling of the rest, with the seed recorded:
- normal audit cases: default ratio 0.10;
- critical-path cases (``critical_path=True``): default ratio
0.50 -- critical paths get a higher rate, not a free pass.
The seed is part of the plan: a sample selected without a
recorded seed is not reproducible and therefore not auditable.
Selection is deterministic given (cases, ratios, seed).
- critical-path cases (``critical_path=True`` and NOT already
mandatory): default ratio 0.50 -- critical paths get a higher
rate, not a free pass. (Track 3's old ``critical_path``
reason label for mandatory kinds is retired: mandatory kinds
now carry their own trigger names, and ``critical_path``
means only this 0.50 random tier.)
Mandatory cases are removed from the random pools BEFORE
drawing, so they consume no random quota. The seed is part of
the plan: a sample selected without a recorded seed is not
reproducible and therefore not auditable. Selection is
deterministic given (cases, ratios, seed).
4. One pass, no double counting. Selection partitions every case
exactly once: mandatory (all its triggers listed, one sample,
one sampling entry) or random-critical or random-normal. A
record that is both a golden task AND a rebuild is sampled
ONCE, with ``triggers == ["golden_task", "rebuild"]``; its
correction, if any, is a single record whose retrigger follows
the (original, corrected) verdict pair, not the trigger list --
triggers decide selection only, never the correction action.
Two different records (e.g. a triage record and the later
rebuild record for the same task) are two samples, not a
double count: deduplication is per record, not per task.
5. Golden-set construction. The caller-supplied golden id set
must be built from human-approved, non-sample tasks only:
``build_golden_task_ids`` applies both vocabularies -- a task
counts only with ``human_approved_by`` set AND none of
``is_example``, ``sample=True``, or an id starting with
``sample-``. Track 3's per-task approver rule and track 5's
sample-exclusion rule are both preserved; neither alone is
sufficient.
6. Correction gate effect (shared with track 3 / wiring #247).
A sampling correction NEVER unblocks a merge by itself: a
correction to ``pass`` (a wrong block) schedules a re-audit
and the gate stays blocked until a new passing re-audit
conclusion exists; a correction to ``fail`` (a wrong pass)
re-triggers pushback and re-blocks at once. Vocabulary
synonyms for the same pair actions: track 3
``retrigger_pushback_and_reblock`` == track 5
``reopen_audit_and_pushback`` (wrong pass), and track 3
``schedule_reaudit`` == track 5
``cancel_pushback_and_rerun_audit`` (wrong block, where
"cancel" means cancel-and-rerun, never cancel-and-merge).
3. Golden-set approval is separate from sampling and is NOT
satisfied by sample files: ``filter_golden_tasks`` drops any task
@@ -81,33 +148,88 @@ _RETRIGGER = {
}
def mandatory_trigger(case):
"""Return the mandatory trigger name for ``case``, or None.
def mandatory_triggers(case, golden_task_ids=None):
"""Return ALL mandatory trigger names for ``case`` (possibly []).
``case`` fields used: ``decision``, ``failure_count``,
``root_cause`` (``"structural"`` / ``"near_complete"`` / other),
Trigger order: golden_task, rebuild,
third_failure_near_complete_still_failed, refreeze, escalate.
``case`` fields used: ``task_id`` (else ``id``) for the golden
check, ``decision`` / ``kind`` for rebuild / refreeze /
escalate, ``failure_count``, ``root_cause``
(``"structural"`` / ``"near_complete"`` / other),
``continuation_failed`` (bool).
"""
if case.get("decision") == "rebuild":
return "rebuild"
triggers = []
golden = set(golden_task_ids or [])
task_id = case.get("task_id") if case.get("task_id") is not None \
else case.get("id")
if task_id is not None and task_id in golden:
triggers.append("golden_task")
if case.get("decision") == "rebuild" or case.get("kind") == "rebuild":
triggers.append("rebuild")
if (case.get("failure_count", 0) >= 3
and case.get("root_cause") == "near_complete"
and case.get("continuation_failed")):
return "third_failure_near_complete_still_failed"
return None
triggers.append("third_failure_near_complete_still_failed")
if case.get("decision") == "refreeze" or case.get("kind") == "refreeze":
triggers.append("refreeze")
if case.get("decision") == "escalate" or case.get("kind") == "escalate":
triggers.append("escalate")
return triggers
def mandatory_trigger(case, golden_task_ids=None):
"""Return the first mandatory trigger name for ``case``, or None.
Legacy single-trigger view of ``mandatory_triggers``; new code
must use ``mandatory_triggers`` so a multi-trigger case is not
silently reported under one reason only.
"""
triggers = mandatory_triggers(case, golden_task_ids)
return triggers[0] if triggers else None
def build_golden_task_ids(tasks):
"""Build the golden task id list from candidate tasks.
A task counts only when it has a human approver
(``human_approved_by``) AND is not a sample/example under any
vocabulary: ``is_example``, ``sample=True``, or an id starting
with ``sample-``. Returns ids in input order, samples dropped
silently here (count them with ``filter_golden_tasks`` when a
dropped count must be reported).
"""
ids = []
for task in tasks or []:
if not isinstance(task, dict):
continue
task_id = task.get("id", "")
if task.get("is_example") or task.get("sample"):
continue
if str(task_id).startswith("sample-"):
continue
if not task.get("human_approved_by"):
continue
ids.append(task_id)
return ids
def select_sample(cases, seed, normal_ratio=DEFAULT_NORMAL_RATIO,
critical_ratio=DEFAULT_CRITICAL_PATH_RATIO):
critical_ratio=DEFAULT_CRITICAL_PATH_RATIO,
golden_task_ids=None):
"""Select the audit sample. Returns a plan dict.
Mandatory cases are always included. The remaining cases are
grouped into critical-path and normal, each shuffled with
Single pass, no double counting: mandatory cases (all their
triggers listed) are always included and removed from the
random pools before drawing. The remaining cases are grouped
into critical-path and normal, each shuffled with
``random.Random(seed + ":" + group)`` and truncated to
``ceil(ratio * group_size)`` (a group of 1 with any positive
ratio therefore contributes 1 -- a small group is never rounded
down to zero and silently unsampled). ``seed`` is required: an
unseeded selection cannot be reproduced on review.
``golden_task_ids`` is the human-approved golden set (see
``build_golden_task_ids``); without it no case is golden.
"""
if seed is None or seed == "":
raise ValueError("sampling seed is required: an unseeded "
@@ -116,9 +238,9 @@ def select_sample(cases, seed, normal_ratio=DEFAULT_NORMAL_RATIO,
critical = []
normal = []
for case in cases:
trigger = mandatory_trigger(case)
if trigger is not None:
mandatory.append((case, trigger))
triggers = mandatory_triggers(case, golden_task_ids)
if triggers:
mandatory.append((case, triggers))
elif case.get("critical_path"):
critical.append(case)
else:
@@ -138,7 +260,9 @@ def select_sample(cases, seed, normal_ratio=DEFAULT_NORMAL_RATIO,
"ratios": {"normal": normal_ratio,
"critical_path": critical_ratio,
"mandatory": 1.0},
"mandatory": [{"case_id": c.get("id"), "trigger": t}
"golden_task_ids": sorted(golden_task_ids or []),
"mandatory": [{"case_id": c.get("id"), "trigger": t[0],
"triggers": list(t)}
for c, t in mandatory],
"random_critical_path": [c.get("id") for c in sampled_critical],
"random_normal": [c.get("id") for c in sampled_normal],
@@ -203,7 +327,8 @@ def golden_approval_checklist(tasks, approver=None, approved_at=None,
"dropped_samples": filtered["dropped_samples"]}
def handle_sampling_result(case, sampling_verdict, corrected_verdict=None):
def handle_sampling_result(case, sampling_verdict, corrected_verdict=None,
corrector=None, original_signer=None):
"""Turn one human sampling result into a record entry + actions.
``sampling_verdict`` is ``"correct"`` or ``"incorrect"``.
@@ -211,13 +336,30 @@ def handle_sampling_result(case, sampling_verdict, corrected_verdict=None):
should have been issued) must be supplied; the retrigger follows
the ORIGINAL decision via ``_RETRIGGER``, or escalates when the
original decision is unknown. Returns the entry dict; the caller
appends it with ``append_audit_record``.
appends it with ``append_audit_record``. This function never
mutates ``case``: the original conclusion is preserved and the
correction lives only in the returned entry.
Corrector separation (track 3 rule, enforced here when both
names are supplied): ``corrector`` must differ from
``original_signer``. Gate effect (shared with track 3 / wiring
#247, stated here so the retrigger names cannot be misread):
a correction never unblocks a merge by itself -- a correction
to ``pass`` schedules a re-audit (``cancel_pushback_and_rerun_audit``
== track 3 ``schedule_reaudit``) and the gate stays blocked
until a new passing re-audit exists; a correction to ``fail``
re-triggers pushback and re-blocks at once
(``reopen_audit_and_pushback`` == track 3
``retrigger_pushback_and_reblock``).
"""
case_id = case.get("id")
original = case.get("decision")
if sampling_verdict not in ("correct", "incorrect"):
raise ValueError("sampling_verdict must be 'correct' or "
"'incorrect', got %r" % (sampling_verdict,))
if (corrector is not None and original_signer is not None
and corrector == original_signer):
raise ValueError("corrector must differ from original signer")
if sampling_verdict == "correct":
return {"case_id": case_id, "original_decision": original,
"sampling_verdict": "correct",
+34 -6
View File
@@ -923,11 +923,29 @@ tests/ # QA 线程专属,含跨端一致性用例
- 审计兜底三件套,缺一不可,全部可测试或可核查:
① 独立重跑的确切产物测试,不是只看报告:复用三方哈希模式
(核心报告、产物清单、重算 hash 三方一致),产物不一致即停。
② 金任务集与关键路径的人工审定样本,定期抽查审计结论本身:
金任务集必须人工审定(每项有 `human_approved_by`),示例文件
不算金任务;抽查比例写死:金任务与推倒/重新冻结/升级记录
100% 抽查,普通记录每 10 条确定性抽 1 条(按序号、可复现,
不是随机抽);抽查结果处理:抽查发现判错即走纠错路径。
② 审计结论本身的抽查,唯一口径(第 2 项补缺口对齐后,
代码 `cloud/audit_sampling.py`、本段、测试三处一致,
不再有两套口径):必抽范围以第 3 轨为准、抽取机制以
第 5 轨为准。第 3 轨原文“普通记录每 10 条确定性抽 1 条
(不是随机抽)”已废止——不足 10 条的普通批次按它抽到
0 条,且序号抽取可被预测;第 5 轨原文必抽窄清单已
扩充——金任务、重新冻结、升级记录补进必抽。
必抽(比例 1.0,规则不是可调比例):金任务记录(任务在
人工审定金任务集内,不论该记录本身是何种结论)、推倒、
重新冻结、升级记录,以及第三次失败根因判为接近完成、
续轮仍 fail 的案例。金任务集必须人工审定(每项有
`human_approved_by`),且样本/示例一律排除(`is_example`
或 `sample=True` 或 id 以 `sample-` 开头,任一即排除)。
随机抽查(其余记录,须带 seed,seed 记入抽查计划,
同 seed 同批同结果、可事后复现):普通记录默认 0.10、
关键路径记录(`critical_path=True` 的非必抽记录)默认
0.50;每组 ceil 取整,非空组至少抽 1 条,绝不因四舍五入
抽到 0。0.10 / 0.50 只是测试默认值,不是已拍板的生产值,
生产抽查比例仍待用户拍板。单次抽取、不双重计入:必抽
记录先划出,不进随机池、不占随机名额;一记录多触发
(如既是金任务又是推倒)只抽一次、理由全列、纠错只记
一条;纠错动作按(原结论,纠正结论)对查表,与触发理由
无关。抽查发现判错即走纠错路径(见下)。
③ G6 用户验收这个最后人工关口不能被自动化绕过去:审计升级、
推回、推倒、抽查纠错都不能替用户勾验收,验收清单最后只能用户勾。
- 审计判错的纠错路径:事后抽查发现审计结论判错(该拦没拦、或
@@ -936,7 +954,17 @@ tests/ # QA 线程专属,含跨端一致性用例
原记录不动,追加一条纠错记录(引用原记录 hash、写清原结论、
纠正后结论、纠错人与原因);该拦没拦的纠正 → 重新触发推回并
恢复阻塞;不该拦拦了的纠正 → 安排重新审计(不是直接放行合并,
因为纠错本身只是复核、不是完整审计);纠错人必须与原签字人不同。
因为纠错本身只是复核、不是完整审计),门禁在新通过重审结论
出现前继续阻塞;纠错人必须与原签字人不同。动作名对照(同
一对动作、两轨词汇):错放行 → 第 3 轨
`retrigger_pushback_and_reblock` = 第 5 轨
`reopen_audit_and_pushback`;错拦截 → 第 3 轨
`schedule_reaudit` = 第 5 轨 `cancel_pushback_and_rerun_audit`
(其中 cancel 指取消并重审,绝不是取消即放行);错推倒 →
停推倒并从 git 历史恢复旧实现;原结论未知 → 升级人工。
已废止的理由名:`critical_path`(曾指必抽种类,现只指
0.50 随机档)、`periodic_sample`(曾指每 10 条抽取,已随
确定性抽取一并废止)。
## 诚实边界:原生客户端(v0.5 后置占位版)
+16 -5
View File
@@ -784,18 +784,29 @@ class AuditOfAuditTestCase(unittest.TestCase):
valid3, _r3 = validate_golden_task_set([{"id": "gold-2"}])
self.assertFalse(valid3)
def test_sampling_covers_golden_critical_and_periodic(self):
def test_sampling_covers_golden_mandatory_and_random(self):
# Unified rule (gap 2): mandatory kinds carry their own
# trigger names (no "critical_path" label for kinds, no
# "periodic_sample"); ordinary records are drawn by seeded
# random sampling with ceil, so 9 ordinary records still
# contribute exactly ceil(0.10 * 9) == 1 -- the old
# deterministic every-10th rule drew zero from this batch.
records = [{"task_id": "gold-1", "kind": "pushback"}]
records += [{"task_id": "t-%d" % i, "kind": "pushback"}
for i in range(9)]
records.append({"task_id": "t-re", "kind": "rebuild"})
selected = select_audit_sample(records, {"gold-1"})
selected = select_audit_sample(records, {"gold-1"}, seed="s")
reasons = {entry["record"]["task_id"]: entry["reason"]
for entry in selected}
self.assertEqual(reasons["gold-1"], "golden_task")
self.assertEqual(reasons["t-re"], "critical_path")
# 10th record (index 9) is the periodic sample.
self.assertIn("t-8", reasons)
self.assertEqual(reasons["t-re"], "rebuild")
ordinary = [t for t in reasons if t.startswith("t-")
and t != "t-re"]
self.assertEqual(len(ordinary), 1)
self.assertNotIn("periodic_sample", set(reasons.values()))
self.assertNotIn("critical_path", set(reasons.values()))
with self.assertRaises(ValueError):
select_audit_sample(records, {"gold-1"})
def test_correction_preserves_original_and_restores_flow(self):
original = {"task_id": "t-1", "round": 0, "verdict": "pass",
@@ -0,0 +1,253 @@
"""Tests for the UNIFIED audit-of-audit sampling rule (gap 2).
This file pins the alignment of track 3 (#244) and track 5 (#238)
only; it does not re-test either track's other semantics.
Unified rule under test (see ``cloud/audit_sampling.py``):
- Mandatory (1.0) = golden-task records + rebuild / refreeze /
escalation records + third-failure near-complete still-failed
cases. Scope from #244, expanded with #238's third-failure
trigger; #244's deterministic every-10th ordinary rule is gone.
- The rest by seeded random draw: normal 0.10, critical-path
0.50, ceil so a non-empty group is never rounded to zero.
The 0.10 / 0.50 values are TEST DEFAULTS, not approved
production rates.
- One pass, no double counting: mandatory records leave the
random pools before drawing; a multi-trigger record is sampled
once with all its triggers listed; correction follows the
(original, corrected) verdict pair, not the trigger list.
- Correction preserves the original: the original record is never
mutated, the correction is a separate appended record, and the
corrector must differ from the original signer.
"""
import copy
import os
import sys
import unittest
sys.path.insert(0, os.path.dirname(os.path.dirname(
os.path.dirname(os.path.abspath(__file__)))))
from cloud.audit_pushback import ( # noqa: E402
build_audit_correction, select_audit_sample)
from cloud.audit_sampling import ( # noqa: E402
append_audit_record, build_golden_task_ids, handle_sampling_result,
mandatory_trigger, mandatory_triggers, select_sample)
def _case(case_id, **kw):
case = {"id": case_id}
case.update(kw)
return case
class TestMandatoryUnion(unittest.TestCase):
def test_all_mandatory_triggers_sampled(self):
cases = [
_case("g-push", task_id="gold-1", kind="pushback"),
_case("r", decision="rebuild"),
_case("rf", kind="refreeze"),
_case("esc", kind="escalate"),
_case("third", decision="fail_pushback", failure_count=3,
root_cause="near_complete", continuation_failed=True),
_case("n0", decision="pass"),
]
plan = select_sample(cases, seed="s", golden_task_ids={"gold-1"})
by_id = {e["case_id"]: e["triggers"] for e in plan["mandatory"]}
self.assertEqual(by_id["g-push"], ["golden_task"])
self.assertEqual(by_id["r"], ["rebuild"])
self.assertEqual(by_id["rf"], ["refreeze"])
self.assertEqual(by_id["esc"], ["escalate"])
self.assertEqual(by_id["third"],
["third_failure_near_complete_still_failed"])
self.assertIn("n0", plan["random_normal"])
self.assertEqual(len(plan["sampled_ids"]),
len(set(plan["sampled_ids"])))
def test_blocked_quota_is_not_mandatory(self):
# Track 3's mandatory kind list is rebuild/refreeze/escalate
# only; a quota-blocked record reaches the random tiers.
case = _case("q", kind="blocked_quota")
self.assertEqual(mandatory_triggers(case), [])
plan = select_sample([case], seed="s")
self.assertEqual(plan["mandatory"], [])
self.assertEqual(plan["random_normal"], ["q"])
def test_golden_task_ordinary_record_is_mandatory(self):
# Golden membership is per task, not per record kind: even
# a plain pass record of a golden task is mandatory.
case = _case("g-pass", task_id="gold-1", kind="pushback",
decision="pass")
plan = select_sample([case], seed="s",
golden_task_ids={"gold-1"})
self.assertEqual(plan["mandatory"][0]["triggers"],
["golden_task"])
class TestNoDoubleCounting(unittest.TestCase):
def test_golden_rebuild_sampled_once_with_all_triggers(self):
cases = [_case("g-re", task_id="gold-1", kind="rebuild")]
cases += [_case("n%d" % i, decision="pass") for i in range(20)]
plan = select_sample(cases, seed="s", golden_task_ids={"gold-1"})
self.assertEqual(plan["sampled_ids"].count("g-re"), 1)
self.assertEqual(plan["mandatory"][0]["triggers"],
["golden_task", "rebuild"])
# The mandatory record consumed no random quota: the draw
# is still ceil(0.10 * 20) == 2 over the ordinary pool.
self.assertEqual(len(plan["random_normal"]), 2)
self.assertNotIn("g-re", plan["random_normal"])
# Legacy single-trigger view reports the first trigger.
self.assertEqual(
mandatory_trigger(cases[0], {"gold-1"}), "golden_task")
def test_triage_and_later_rebuild_are_two_records_not_double(self):
# Deduplication is per record: a triage record and the
# later rebuild record for the same task are two samples.
cases = [
_case("triage", decision="fail_pushback", failure_count=3,
root_cause="near_complete", continuation_failed=True),
_case("rebuild", decision="rebuild"),
]
plan = select_sample(cases, seed="s")
self.assertEqual(sorted(plan["sampled_ids"]),
["rebuild", "triage"])
class TestRandomTiersAndCeil(unittest.TestCase):
def test_ratios_and_seed_reproducibility(self):
cases = [_case("n%d" % i, decision="pass") for i in range(20)]
cases += [_case("c%d" % i, decision="pass", critical_path=True)
for i in range(10)]
plan = select_sample(cases, seed="seed-1")
self.assertEqual(len(plan["random_normal"]), 2)
self.assertEqual(len(plan["random_critical_path"]), 5)
again = select_sample(cases, seed="seed-1")
self.assertEqual(plan["sampled_ids"], again["sampled_ids"])
def test_small_groups_never_round_to_zero(self):
self.assertEqual(len(select_sample(
[_case("only", decision="pass")], seed="s")["random_normal"]),
1)
# 9 ordinary records: the old every-10th rule drew ZERO
# here; ceil(0.10 * 9) == 1 under the unified rule.
self.assertEqual(len(select_sample(
[_case("n%d" % i, decision="pass") for i in range(9)],
seed="s")["random_normal"]), 1)
self.assertEqual(len(select_sample(
[_case("n%d" % i, decision="pass") for i in range(19)],
seed="s")["random_normal"]), 2)
self.assertEqual(len(select_sample(
[_case("c0", decision="pass", critical_path=True)],
seed="s")["random_critical_path"]), 1)
self.assertEqual(len(select_sample(
[_case("c%d" % i, decision="pass", critical_path=True)
for i in range(3)], seed="s")["random_critical_path"]), 2)
def test_missing_seed_rejected(self):
with self.assertRaises(ValueError):
select_sample([_case("n", decision="pass")], seed="")
class TestGoldenSetConstruction(unittest.TestCase):
def test_build_golden_ids_applies_both_vocabularies(self):
tasks = [
{"id": "gold-1", "human_approved_by": "user"},
{"id": "sample-2", "human_approved_by": "user"},
{"id": "gold-2", "sample": True, "human_approved_by": "user"},
{"id": "gold-3", "is_example": True,
"human_approved_by": "user"},
{"id": "gold-4"},
]
self.assertEqual(build_golden_task_ids(tasks), ["gold-1"])
class TestAdapter(unittest.TestCase):
def test_adapter_returns_unified_selection_in_record_order(self):
records = [
{"task_id": "gold-1", "kind": "pushback"},
{"task_id": "gold-1", "kind": "fix"},
{"task_id": "gold-1", "kind": "rebuild"},
{"task_id": "t-rf", "kind": "refreeze"},
{"task_id": "t-esc", "kind": "escalate"},
]
records += [{"task_id": "t-%d" % i, "kind": "pushback"}
for i in range(9)]
before = copy.deepcopy(records)
golden = set(build_golden_task_ids(
[{"id": "gold-1", "human_approved_by": "user"}]))
selected = select_audit_sample(records, golden, seed="s")
by_record = {id(e["record"]): e for e in selected}
# Every golden record is mandatory, whatever its kind; the
# golden rebuild record carries both triggers, once.
self.assertEqual(by_record[id(records[0])]["reasons"],
["golden_task"])
self.assertEqual(by_record[id(records[1])]["reasons"],
["golden_task"])
self.assertEqual(by_record[id(records[2])]["reasons"],
["golden_task", "rebuild"])
self.assertEqual(by_record[id(records[3])]["reason"], "refreeze")
self.assertEqual(by_record[id(records[4])]["reason"], "escalate")
# 9 ordinary records contribute ceil(0.10 * 9) == 1.
ordinary = [e for e in selected
if e["record"]["task_id"].startswith("t-")
and e["record"]["task_id"] not in ("t-rf", "t-esc")]
self.assertEqual(len(ordinary), 1)
self.assertEqual(ordinary[0]["reason"], "random_normal")
# Selected entries keep original record order.
order = [records.index(e["record"]) for e in selected]
self.assertEqual(order, sorted(order))
# Input records are never mutated by selection.
self.assertEqual(records, before)
with self.assertRaises(ValueError):
select_audit_sample(records, golden)
class TestCorrectionPreservesOriginal(unittest.TestCase):
def test_track3_correction_appends_without_mutating(self):
original = {"task_id": "t-1", "round": 0, "verdict": "pass",
"signer": "auditor-a", "entry_hash": "h" * 64}
before = dict(original)
correction, errors = build_audit_correction(
original, "fail", "auditor-b", "sample re-run found ci red")
self.assertEqual(errors, [])
self.assertEqual(original, before)
self.assertEqual(correction["original_entry_hash"], "h" * 64)
self.assertEqual(correction["restored_action"],
"retrigger_pushback_and_reblock")
log = [original, correction]
self.assertEqual(log[0]["verdict"], "pass")
_bad, bad_errors = build_audit_correction(
original, "fail", "auditor-a", "same signer")
self.assertTrue(bad_errors)
def test_track5_correction_appends_without_mutating(self):
case = {"id": "a", "decision": "pass"}
before = dict(case)
entry = handle_sampling_result(
case, "incorrect", corrected_verdict="fail_pushback",
corrector="auditor-b", original_signer="auditor-a")
# Track 5 vocabulary for the same wrong-pass pair as track
# 3's retrigger_pushback_and_reblock (synonyms, see the
# unified rule in cloud/audit_sampling.py).
self.assertEqual(entry["retrigger"], "reopen_audit_and_pushback")
self.assertEqual(case, before)
log = []
log2 = append_audit_record(log, entry)
self.assertEqual(log, [])
self.assertEqual(len(log2), 1)
with self.assertRaises(ValueError):
handle_sampling_result(
case, "incorrect", corrected_verdict="fail_pushback",
corrector="auditor-a", original_signer="auditor-a")
def test_correct_verdict_only_records(self):
entry = handle_sampling_result(
_case("a", decision="pass"), "correct")
self.assertIsNone(entry["corrected_decision"])
self.assertIsNone(entry["retrigger"])
if __name__ == "__main__":
unittest.main()