audit: unify audit-of-audit sampling rule (gap 2)
Single sampling rule; track 3 (#244) is authoritative for WHO is mandatory, track 5 (#238) for HOW the rest are drawn: - Mandatory (1.0): golden-task records + rebuild / refreeze / escalation records + third-failure near-complete still-failed cases (mandatory_triggers, full list per case). Track 3's old deterministic every-10th ordinary rule is removed (it drew zero from batches under 10 ordinary records and was predictable); track 5's old narrow mandatory list is expanded with golden / refreeze / escalate. blocked_quota stays non-mandatory. - Random tiers: normal 0.10, critical-path 0.50, seed required and recorded, ceil so a non-empty group is never rounded to zero. 0.10/0.50 are TEST DEFAULTS, not approved production values; production rates remain a user decision. - One pass, no double counting: mandatory records leave the random pools before drawing; a multi-trigger record (e.g. golden + rebuild) is sampled once with all triggers listed, and its correction follows the (original, corrected) verdict pair, never the trigger list. - cloud/audit_pushback.py::select_audit_sample is now only a record-shaped adapter over cloud/audit_sampling.py::select_sample (seed required, retired reason names critical_path-for-kinds and periodic_sample removed); AUDIT_SAMPLE_EVERY_NTH deleted. - Correction: original never mutated, correction appended, corrector != signer enforced in audit_sampling when both names are supplied; gate effect pinned (correction to pass never unblocks by itself -- schedule/cancel-and-rerun, blocked until a new passing re-audit; correction to fail re-blocks at once). Track 3 / track 5 retrigger vocabularies documented as synonyms. - Skill text (tightening 3 item 2 + correction paragraph) rewritten to the unified rule; new test file pins the boundaries (mandatory union, dedup, ceil incl. the 9-record case, seed reproducibility, golden-set construction under both sample vocabularies, adapter order/non-mutation, correction preservation on both tracks).
This commit is contained in:
1 parent
f83bd79525
commit
6dcd965542
5 files changed
+546
-60
No files matched your search
@@ -784,18 +784,29 @@ class AuditOfAuditTestCase(unittest.TestCase):
|
||||
valid3, _r3 = validate_golden_task_set([{"id": "gold-2"}])
|
||||
self.assertFalse(valid3)
|
||||
|
||||
def test_sampling_covers_golden_critical_and_periodic(self):
|
||||
def test_sampling_covers_golden_mandatory_and_random(self):
|
||||
# Unified rule (gap 2): mandatory kinds carry their own
|
||||
# trigger names (no "critical_path" label for kinds, no
|
||||
# "periodic_sample"); ordinary records are drawn by seeded
|
||||
# random sampling with ceil, so 9 ordinary records still
|
||||
# contribute exactly ceil(0.10 * 9) == 1 -- the old
|
||||
# deterministic every-10th rule drew zero from this batch.
|
||||
records = [{"task_id": "gold-1", "kind": "pushback"}]
|
||||
records += [{"task_id": "t-%d" % i, "kind": "pushback"}
|
||||
for i in range(9)]
|
||||
records.append({"task_id": "t-re", "kind": "rebuild"})
|
||||
selected = select_audit_sample(records, {"gold-1"})
|
||||
selected = select_audit_sample(records, {"gold-1"}, seed="s")
|
||||
reasons = {entry["record"]["task_id"]: entry["reason"]
|
||||
for entry in selected}
|
||||
self.assertEqual(reasons["gold-1"], "golden_task")
|
||||
self.assertEqual(reasons["t-re"], "critical_path")
|
||||
# 10th record (index 9) is the periodic sample.
|
||||
self.assertIn("t-8", reasons)
|
||||
self.assertEqual(reasons["t-re"], "rebuild")
|
||||
ordinary = [t for t in reasons if t.startswith("t-")
|
||||
and t != "t-re"]
|
||||
self.assertEqual(len(ordinary), 1)
|
||||
self.assertNotIn("periodic_sample", set(reasons.values()))
|
||||
self.assertNotIn("critical_path", set(reasons.values()))
|
||||
with self.assertRaises(ValueError):
|
||||
select_audit_sample(records, {"gold-1"})
|
||||
|
||||
def test_correction_preserves_original_and_restores_flow(self):
|
||||
original = {"task_id": "t-1", "round": 0, "verdict": "pass",
|
||||
|
||||
Reference in new issue
Block a user