audit: lifecycle hook writes the attempt ledger (gap 3 follow-up)

- cloud/attempt_hook.py (new): canonical policy -- one counted
  dispatch = one row, is_defect always False from the worker;
  human/first rounds and human breakpoint resumes count;
  platform auto-resume reruns, coordinator fan-out, and
  pre-execution gate failures do not; transient retries inside
  one dispatch are not new attempts. Platform origin survives
  via the task_attempt_origin_markers table (same-transaction
  write on requeue, queued_at generation guard on consume).
  Defects are explicit-recorder only; corrections are
  append-only (record_correction, last-wins).
- cloud/worker.py: Worker(attempt_ledger_hook=False) + CLI
  --attempt-ledger-hook (default OFF = gap-3 behaviour exactly);
  marker write in auto_resume_interrupted, marker consume +
  classify in run_one, one ledger write at each terminal point
  (cancel branch, normal terminal block); all hook writes
  best-effort, never fail the task or block a requeue.
- cloud/attempt_ledger.py: corrects_attempt_no column (new DDL
  supersedes gap 3's pre-merge DDL) + record_correction +
  correction-aware aggregation; read path uses SELECT * so
  gap-3-era tables still read; cost code untouched.
- cloud/tests/test_attempt_hook.py: 32 tests (policy, markers,
  corrections, worker integration, mixed endpoint).
- docs/arch-conformance-metrics.md: policy, switch, PG manual
  steps (new PG_DDL + PG_MARKER_DDL), remaining user decisions.
This commit is contained in:
host committed 2026-11-03 02:49:26 +00:00
1 parent 436a92ba1d
commit dc556b0c19
5 files changed
+1522 -38

No files matched your search

+192 -25
View File
@@ -41,7 +41,22 @@ Semantics (pinned by tests, not left to each caller):
``compute_metrics`` carries per task) is the category of its
FIRST defect row in attempt order. Later defect rows still count
toward ``defects``; only the category label is first-wins, so it
is deterministic.
is deterministic. Corrections (below) re-evaluate this label:
the first defect row AFTER corrections is applied wins.
- A defect judgement can be CORRECTED, never rewritten:
``record_correction`` appends a new row whose
``corrects_attempt_no`` points at the attempt it corrects; the
old row stays untouched, and aggregation applies corrections in
attempt order with last-wins (so a wrongly marked defect can be
unmarked, and a defect missed at record time can be added after
review). A correction row is never itself counted as an
attempt, cannot change an attempt's outcome (only its defect
judgement), and cannot target an interrupted row, another
correction row, or a nonexistent attempt -- all raise
``ValueError`` and write nothing. Who may correct is a policy
question answered in ``cloud/attempt_hook.py`` (explicit
recorder only, maintainer bar); the worker hook never writes
corrections and never marks a defect.
- Old tasks have no ledger rows, and that is NOT backfilled: the
aggregate simply has no entry for them, the endpoint leaves their
``attempts``/``defects`` as ``None``, and ``compute_metrics``
@@ -75,16 +90,32 @@ Persistence notes:
used for missing ``cost_records``.
- The read path (``GET /metrics-report``) never creates the table:
a read-only endpoint must not mutate schema. Table creation
happens only on the write path, inside ``record_attempt``.
- No lifecycle hook writes to this ledger yet. Hooking a writer
into the worker's retry/resume loops would risk exactly the
misclassification this module refuses to make (platform
auto-resume counted as an attempt, transient retry counted as a
defect). The writer -- which lifecycle event counts as an
attempt, and who judges ``is_defect`` -- is a separate decision,
listed as an open item; until it lands, the ledger fills only
through explicit ``record_attempt`` calls, and production rates
stay honestly unknown rather than auto-filled.
happens only on the write path, inside ``record_attempt`` /
``record_correction``.
- Lifecycle hook (this track, see ``cloud/attempt_hook.py``):
the worker writes exactly one row per COUNTED dispatch (see
that module's policy for which dispatches count: human/first
rounds and human breakpoint resumes count; platform auto-resume
reruns, coordinator fan-out, and pre-execution gate failures do
not). The hook is OFF by default (``attempt_ledger_hook=False``):
with it off, this module's behaviour is exactly audit gap 3 --
the ledger fills only through explicit ``record_attempt`` /
``record_correction`` calls, and production rates stay honestly
unknown until the hook is deliberately enabled. The worker hook
always records ``is_defect=False``: it never judges a defect,
so enabling it can never turn a transient retry into a defect.
- Schema evolution note (pre-production, this track):
``corrects_attempt_no`` was added to the DDL after audit gap 3
but BEFORE either DDL reached production (gap 3 is still an
OPEN, unmerged PR), so no production table needs migration:
fresh tables get the column from the DDL below. A database
created from gap 3's DDL alone (temporary/test databases only)
gains the column with one statement --
``ALTER TABLE task_attempts ADD COLUMN corrects_attempt_no
INTEGER`` -- which is also the documented step if gap 3's DDL
was ever run on production Postgres before this track lands.
The read path uses ``SELECT *`` deliberately so a table without
the column still reads (its rows aggregate as plain attempts).
This module imports no ``cloud.*`` module and opens no database
connection itself: callers pass a connection in, so the layer rules
@@ -102,6 +133,11 @@ OUTCOMES = frozenset({"success", "failure", "interrupted"})
# One DDL serves both backends: every type and the composite key are
# valid in SQLite and in Postgres, and there is no AUTOINCREMENT or
# identity column for either backend's sequence handling to chase.
# ``corrects_attempt_no`` (NULL = a real attempt row; non-NULL = a
# correction row for that attempt_no, never counted as an attempt)
# was added by the lifecycle-hook track; it SUPERSEDES gap 3's DDL
# for any production manual step (gap 3 never merged, so no
# production table exists yet -- see the module docstring).
SQLITE_DDL = (
"CREATE TABLE IF NOT EXISTS task_attempts("
"task_id TEXT NOT NULL, "
@@ -110,6 +146,7 @@ SQLITE_DDL = (
"is_defect INTEGER NOT NULL DEFAULT 0, "
"defect_category TEXT, "
"created_at DOUBLE PRECISION, "
"corrects_attempt_no INTEGER, "
"PRIMARY KEY (task_id, attempt_no))"
)
@@ -117,8 +154,19 @@ SQLITE_DDL = (
# table exists before expecting real rates from a Postgres-backed
# endpoint. It is intentionally NOT part of POSTGRES_SCHEMA (see
# the module docstring: migrations coverage must not widen silently).
# This is the NEW DDL (with ``corrects_attempt_no``): if gap 3's
# older DDL was ever run first, follow it with
# ``ALTER TABLE task_attempts ADD COLUMN corrects_attempt_no
# INTEGER`` and verify before expecting corrections to work.
PG_DDL = SQLITE_DDL
# One-line upgrade for a table created from gap 3's DDL (no
# ``corrects_attempt_no`` column yet). Valid in both backends.
OLD_TABLE_ALTER = (
"ALTER TABLE task_attempts "
"ADD COLUMN corrects_attempt_no INTEGER"
)
def ensure_task_attempts_table(conn):
"""Create the ledger table if absent. Idempotent; does not
@@ -153,8 +201,9 @@ def record_attempt(conn, task_id, outcome, is_defect=False,
attempt_no = int(row[0]) + 1
conn.execute(
"INSERT INTO task_attempts(task_id, attempt_no, outcome, "
"is_defect, defect_category, created_at) "
"VALUES(?, ?, ?, ?, ?, ?)",
"is_defect, defect_category, created_at, "
"corrects_attempt_no) "
"VALUES(?, ?, ?, ?, ?, ?, NULL)",
(task_id, attempt_no, outcome, 1 if is_defect else 0,
defect_category,
time.time() if created_at is None else created_at),
@@ -163,13 +212,87 @@ def record_attempt(conn, task_id, outcome, is_defect=False,
return attempt_no
def record_correction(conn, task_id, attempt_no, is_defect,
defect_category=None, created_at=None):
"""Append a CORRECTION row for an already-recorded attempt.
The correction does not rewrite history: the target row stays
exactly as recorded, and this new row (its own ``MAX + 1``
``attempt_no``) carries the corrected defect judgement in
``is_defect`` / ``defect_category`` plus
``corrects_attempt_no = attempt_no``. Aggregation applies
corrections in attempt order, last-wins, and never counts a
correction row as an attempt.
Only the defect judgement is correctable: the correction row
copies the target's outcome, so it cannot turn a success into
a failure or vice versa. Validates before writing: missing
``task_id``, a nonexistent target, a target that is itself a
correction row, a target whose outcome is ``interrupted``
(interrupted rows are never defects, so there is no defect
judgement to correct), and a defect category without a defect
flag all raise ``ValueError`` and write nothing. On a table
created from gap 3's DDL (no ``corrects_attempt_no`` column),
the INSERT raises the backend's OperationalError loudly --
run ``OLD_TABLE_ALTER`` first (see the module docstring);
a correction must never silently degrade into a plain row.
Returns the correction row's own ``attempt_no``.
"""
if not task_id:
raise ValueError("task_id is required")
if defect_category is not None and not is_defect:
raise ValueError("defect_category requires is_defect=True")
ensure_task_attempts_table(conn)
target = conn.execute(
"SELECT outcome, corrects_attempt_no FROM task_attempts "
"WHERE task_id = ? AND attempt_no = ?",
(task_id, attempt_no),
).fetchone()
if target is None:
raise ValueError(
"no such attempt to correct: %s #%s" % (task_id, attempt_no))
if target["corrects_attempt_no"] is not None:
raise ValueError(
"cannot correct a correction row: %s #%s"
% (task_id, attempt_no))
if target["outcome"] == "interrupted":
raise ValueError(
"an interrupted attempt has no defect judgement to "
"correct: %s #%s" % (task_id, attempt_no))
row = conn.execute(
"SELECT COALESCE(MAX(attempt_no), 0) FROM task_attempts "
"WHERE task_id = ?",
(task_id,),
).fetchone()
correction_no = int(row[0]) + 1
conn.execute(
"INSERT INTO task_attempts(task_id, attempt_no, outcome, "
"is_defect, defect_category, created_at, "
"corrects_attempt_no) "
"VALUES(?, ?, ?, ?, ?, ?, ?)",
(task_id, correction_no, target["outcome"],
1 if is_defect else 0, defect_category,
time.time() if created_at is None else created_at,
attempt_no),
)
conn.commit()
return correction_no
def read_attempt_rows(conn):
"""All ledger rows in attempt order. Raises the backend's
OperationalError when the table does not exist yet; the
endpoint caller treats that as "no ledger", not as a failure."""
endpoint caller treats that as "no ledger", not as a failure.
``SELECT *`` is deliberate: a table created from gap 3's DDL
has no ``corrects_attempt_no`` column, and naming that column
here would turn those readable rows into an endpoint error.
With ``*`` the rows still read, and ``aggregate_attempts``
treats a missing ``corrects_attempt_no`` as a plain attempt
row (see ``_field_optional``).
"""
return conn.execute(
"SELECT task_id, attempt_no, outcome, is_defect, "
"defect_category FROM task_attempts "
"SELECT * FROM task_attempts "
"ORDER BY task_id, attempt_no").fetchall()
@@ -180,6 +303,19 @@ def _field(row, name):
return row[name]
def _field_optional(row, name):
"""Like ``_field`` but missing keys mean "absent", not an
error. Used ONLY for ``corrects_attempt_no``: rows built by
gap-3-era readers (plain dicts, or tables without the column)
carry no such key, and they are plain attempt rows by
definition -- failing loudly there would break every gap-3
aggregate test for no benefit."""
try:
return row[name]
except (KeyError, IndexError, TypeError):
return None
def aggregate_attempts(rows):
"""Pure aggregation: ledger rows -> ``{task_id: record}`` where
``record`` is the ``attempts``/``defects``/``defect_category``
@@ -187,24 +323,55 @@ def aggregate_attempts(rows):
Interrupted rows are excluded from both counts, and a task
with only interrupted rows is omitted entirely (no counted
attempt => its rates stay unknown, never 0).
attempt => its rates stay unknown, never 0). Correction rows
(``corrects_attempt_no`` set) are excluded from the attempt
count and instead OVERWRITE the defect judgement of their
target attempt, in attempt order, last-wins: a task's
``defects`` is the number of its counted attempts whose
judgement AFTER corrections is a defect, and its
``defect_category`` is the category of its FIRST defect
attempt in attempt order AFTER corrections.
"""
per_task = {}
ordered = sorted(
rows,
key=lambda r: (_field(r, "task_id"), _field(r, "attempt_no")))
# Per task: attempt_no -> [is_defect, defect_category], in
# attempt order (insertion order = attempt_no order, since the
# input was sorted). Corrections mutate the target entry; they
# never create an entry of their own.
judged = {}
for row in ordered:
task_id = _field(row, "task_id")
corrects = _field_optional(row, "corrects_attempt_no")
if corrects is not None:
entries = judged.get(task_id)
if entries is not None and corrects in entries:
entries[corrects] = [
bool(_field(row, "is_defect")),
_field(row, "defect_category")]
continue
if _field(row, "outcome") == "interrupted":
continue
entry = per_task.setdefault(
task_id, {"attempts": 0, "defects": 0,
"defect_category": None})
entry["attempts"] += 1
if _field(row, "is_defect"):
entry["defects"] += 1
if entry["defect_category"] is None:
entry["defect_category"] = _field(row, "defect_category")
entries = judged.setdefault(task_id, {})
entries[_field(row, "attempt_no")] = [
bool(_field(row, "is_defect")),
_field(row, "defect_category")]
for task_id, entries in judged.items():
attempts = len(entries)
if attempts == 0:
continue
defects = 0
defect_category = None
for _no in sorted(entries):
is_defect, category = entries[_no]
if is_defect:
defects += 1
if defect_category is None:
defect_category = category
per_task[task_id] = {
"attempts": attempts, "defects": defects,
"defect_category": defect_category}
return per_task