audit: lifecycle hook writes the attempt ledger (gap 3 follow-up)
- cloud/attempt_hook.py (new): canonical policy -- one counted dispatch = one row, is_defect always False from the worker; human/first rounds and human breakpoint resumes count; platform auto-resume reruns, coordinator fan-out, and pre-execution gate failures do not; transient retries inside one dispatch are not new attempts. Platform origin survives via the task_attempt_origin_markers table (same-transaction write on requeue, queued_at generation guard on consume). Defects are explicit-recorder only; corrections are append-only (record_correction, last-wins). - cloud/worker.py: Worker(attempt_ledger_hook=False) + CLI --attempt-ledger-hook (default OFF = gap-3 behaviour exactly); marker write in auto_resume_interrupted, marker consume + classify in run_one, one ledger write at each terminal point (cancel branch, normal terminal block); all hook writes best-effort, never fail the task or block a requeue. - cloud/attempt_ledger.py: corrects_attempt_no column (new DDL supersedes gap 3's pre-merge DDL) + record_correction + correction-aware aggregation; read path uses SELECT * so gap-3-era tables still read; cost code untouched. - cloud/tests/test_attempt_hook.py: 32 tests (policy, markers, corrections, worker integration, mixed endpoint). - docs/arch-conformance-metrics.md: policy, switch, PG manual steps (new PG_DDL + PG_MARKER_DDL), remaining user decisions.
This commit is contained in:
1 parent
436a92ba1d
commit
dc556b0c19
5 files changed
+1522
-38
No files matched your search
+192
-25
@@ -41,7 +41,22 @@ Semantics (pinned by tests, not left to each caller):
|
||||
``compute_metrics`` carries per task) is the category of its
|
||||
FIRST defect row in attempt order. Later defect rows still count
|
||||
toward ``defects``; only the category label is first-wins, so it
|
||||
is deterministic.
|
||||
is deterministic. Corrections (below) re-evaluate this label:
|
||||
the first defect row AFTER corrections is applied wins.
|
||||
- A defect judgement can be CORRECTED, never rewritten:
|
||||
``record_correction`` appends a new row whose
|
||||
``corrects_attempt_no`` points at the attempt it corrects; the
|
||||
old row stays untouched, and aggregation applies corrections in
|
||||
attempt order with last-wins (so a wrongly marked defect can be
|
||||
unmarked, and a defect missed at record time can be added after
|
||||
review). A correction row is never itself counted as an
|
||||
attempt, cannot change an attempt's outcome (only its defect
|
||||
judgement), and cannot target an interrupted row, another
|
||||
correction row, or a nonexistent attempt -- all raise
|
||||
``ValueError`` and write nothing. Who may correct is a policy
|
||||
question answered in ``cloud/attempt_hook.py`` (explicit
|
||||
recorder only, maintainer bar); the worker hook never writes
|
||||
corrections and never marks a defect.
|
||||
- Old tasks have no ledger rows, and that is NOT backfilled: the
|
||||
aggregate simply has no entry for them, the endpoint leaves their
|
||||
``attempts``/``defects`` as ``None``, and ``compute_metrics``
|
||||
@@ -75,16 +90,32 @@ Persistence notes:
|
||||
used for missing ``cost_records``.
|
||||
- The read path (``GET /metrics-report``) never creates the table:
|
||||
a read-only endpoint must not mutate schema. Table creation
|
||||
happens only on the write path, inside ``record_attempt``.
|
||||
- No lifecycle hook writes to this ledger yet. Hooking a writer
|
||||
into the worker's retry/resume loops would risk exactly the
|
||||
misclassification this module refuses to make (platform
|
||||
auto-resume counted as an attempt, transient retry counted as a
|
||||
defect). The writer -- which lifecycle event counts as an
|
||||
attempt, and who judges ``is_defect`` -- is a separate decision,
|
||||
listed as an open item; until it lands, the ledger fills only
|
||||
through explicit ``record_attempt`` calls, and production rates
|
||||
stay honestly unknown rather than auto-filled.
|
||||
happens only on the write path, inside ``record_attempt`` /
|
||||
``record_correction``.
|
||||
- Lifecycle hook (this track, see ``cloud/attempt_hook.py``):
|
||||
the worker writes exactly one row per COUNTED dispatch (see
|
||||
that module's policy for which dispatches count: human/first
|
||||
rounds and human breakpoint resumes count; platform auto-resume
|
||||
reruns, coordinator fan-out, and pre-execution gate failures do
|
||||
not). The hook is OFF by default (``attempt_ledger_hook=False``):
|
||||
with it off, this module's behaviour is exactly audit gap 3 --
|
||||
the ledger fills only through explicit ``record_attempt`` /
|
||||
``record_correction`` calls, and production rates stay honestly
|
||||
unknown until the hook is deliberately enabled. The worker hook
|
||||
always records ``is_defect=False``: it never judges a defect,
|
||||
so enabling it can never turn a transient retry into a defect.
|
||||
- Schema evolution note (pre-production, this track):
|
||||
``corrects_attempt_no`` was added to the DDL after audit gap 3
|
||||
but BEFORE either DDL reached production (gap 3 is still an
|
||||
OPEN, unmerged PR), so no production table needs migration:
|
||||
fresh tables get the column from the DDL below. A database
|
||||
created from gap 3's DDL alone (temporary/test databases only)
|
||||
gains the column with one statement --
|
||||
``ALTER TABLE task_attempts ADD COLUMN corrects_attempt_no
|
||||
INTEGER`` -- which is also the documented step if gap 3's DDL
|
||||
was ever run on production Postgres before this track lands.
|
||||
The read path uses ``SELECT *`` deliberately so a table without
|
||||
the column still reads (its rows aggregate as plain attempts).
|
||||
|
||||
This module imports no ``cloud.*`` module and opens no database
|
||||
connection itself: callers pass a connection in, so the layer rules
|
||||
@@ -102,6 +133,11 @@ OUTCOMES = frozenset({"success", "failure", "interrupted"})
|
||||
# One DDL serves both backends: every type and the composite key are
|
||||
# valid in SQLite and in Postgres, and there is no AUTOINCREMENT or
|
||||
# identity column for either backend's sequence handling to chase.
|
||||
# ``corrects_attempt_no`` (NULL = a real attempt row; non-NULL = a
|
||||
# correction row for that attempt_no, never counted as an attempt)
|
||||
# was added by the lifecycle-hook track; it SUPERSEDES gap 3's DDL
|
||||
# for any production manual step (gap 3 never merged, so no
|
||||
# production table exists yet -- see the module docstring).
|
||||
SQLITE_DDL = (
|
||||
"CREATE TABLE IF NOT EXISTS task_attempts("
|
||||
"task_id TEXT NOT NULL, "
|
||||
@@ -110,6 +146,7 @@ SQLITE_DDL = (
|
||||
"is_defect INTEGER NOT NULL DEFAULT 0, "
|
||||
"defect_category TEXT, "
|
||||
"created_at DOUBLE PRECISION, "
|
||||
"corrects_attempt_no INTEGER, "
|
||||
"PRIMARY KEY (task_id, attempt_no))"
|
||||
)
|
||||
|
||||
@@ -117,8 +154,19 @@ SQLITE_DDL = (
|
||||
# table exists before expecting real rates from a Postgres-backed
|
||||
# endpoint. It is intentionally NOT part of POSTGRES_SCHEMA (see
|
||||
# the module docstring: migrations coverage must not widen silently).
|
||||
# This is the NEW DDL (with ``corrects_attempt_no``): if gap 3's
|
||||
# older DDL was ever run first, follow it with
|
||||
# ``ALTER TABLE task_attempts ADD COLUMN corrects_attempt_no
|
||||
# INTEGER`` and verify before expecting corrections to work.
|
||||
PG_DDL = SQLITE_DDL
|
||||
|
||||
# One-line upgrade for a table created from gap 3's DDL (no
|
||||
# ``corrects_attempt_no`` column yet). Valid in both backends.
|
||||
OLD_TABLE_ALTER = (
|
||||
"ALTER TABLE task_attempts "
|
||||
"ADD COLUMN corrects_attempt_no INTEGER"
|
||||
)
|
||||
|
||||
|
||||
def ensure_task_attempts_table(conn):
|
||||
"""Create the ledger table if absent. Idempotent; does not
|
||||
@@ -153,8 +201,9 @@ def record_attempt(conn, task_id, outcome, is_defect=False,
|
||||
attempt_no = int(row[0]) + 1
|
||||
conn.execute(
|
||||
"INSERT INTO task_attempts(task_id, attempt_no, outcome, "
|
||||
"is_defect, defect_category, created_at) "
|
||||
"VALUES(?, ?, ?, ?, ?, ?)",
|
||||
"is_defect, defect_category, created_at, "
|
||||
"corrects_attempt_no) "
|
||||
"VALUES(?, ?, ?, ?, ?, ?, NULL)",
|
||||
(task_id, attempt_no, outcome, 1 if is_defect else 0,
|
||||
defect_category,
|
||||
time.time() if created_at is None else created_at),
|
||||
@@ -163,13 +212,87 @@ def record_attempt(conn, task_id, outcome, is_defect=False,
|
||||
return attempt_no
|
||||
|
||||
|
||||
def record_correction(conn, task_id, attempt_no, is_defect,
|
||||
defect_category=None, created_at=None):
|
||||
"""Append a CORRECTION row for an already-recorded attempt.
|
||||
|
||||
The correction does not rewrite history: the target row stays
|
||||
exactly as recorded, and this new row (its own ``MAX + 1``
|
||||
``attempt_no``) carries the corrected defect judgement in
|
||||
``is_defect`` / ``defect_category`` plus
|
||||
``corrects_attempt_no = attempt_no``. Aggregation applies
|
||||
corrections in attempt order, last-wins, and never counts a
|
||||
correction row as an attempt.
|
||||
|
||||
Only the defect judgement is correctable: the correction row
|
||||
copies the target's outcome, so it cannot turn a success into
|
||||
a failure or vice versa. Validates before writing: missing
|
||||
``task_id``, a nonexistent target, a target that is itself a
|
||||
correction row, a target whose outcome is ``interrupted``
|
||||
(interrupted rows are never defects, so there is no defect
|
||||
judgement to correct), and a defect category without a defect
|
||||
flag all raise ``ValueError`` and write nothing. On a table
|
||||
created from gap 3's DDL (no ``corrects_attempt_no`` column),
|
||||
the INSERT raises the backend's OperationalError loudly --
|
||||
run ``OLD_TABLE_ALTER`` first (see the module docstring);
|
||||
a correction must never silently degrade into a plain row.
|
||||
Returns the correction row's own ``attempt_no``.
|
||||
"""
|
||||
if not task_id:
|
||||
raise ValueError("task_id is required")
|
||||
if defect_category is not None and not is_defect:
|
||||
raise ValueError("defect_category requires is_defect=True")
|
||||
ensure_task_attempts_table(conn)
|
||||
target = conn.execute(
|
||||
"SELECT outcome, corrects_attempt_no FROM task_attempts "
|
||||
"WHERE task_id = ? AND attempt_no = ?",
|
||||
(task_id, attempt_no),
|
||||
).fetchone()
|
||||
if target is None:
|
||||
raise ValueError(
|
||||
"no such attempt to correct: %s #%s" % (task_id, attempt_no))
|
||||
if target["corrects_attempt_no"] is not None:
|
||||
raise ValueError(
|
||||
"cannot correct a correction row: %s #%s"
|
||||
% (task_id, attempt_no))
|
||||
if target["outcome"] == "interrupted":
|
||||
raise ValueError(
|
||||
"an interrupted attempt has no defect judgement to "
|
||||
"correct: %s #%s" % (task_id, attempt_no))
|
||||
row = conn.execute(
|
||||
"SELECT COALESCE(MAX(attempt_no), 0) FROM task_attempts "
|
||||
"WHERE task_id = ?",
|
||||
(task_id,),
|
||||
).fetchone()
|
||||
correction_no = int(row[0]) + 1
|
||||
conn.execute(
|
||||
"INSERT INTO task_attempts(task_id, attempt_no, outcome, "
|
||||
"is_defect, defect_category, created_at, "
|
||||
"corrects_attempt_no) "
|
||||
"VALUES(?, ?, ?, ?, ?, ?, ?)",
|
||||
(task_id, correction_no, target["outcome"],
|
||||
1 if is_defect else 0, defect_category,
|
||||
time.time() if created_at is None else created_at,
|
||||
attempt_no),
|
||||
)
|
||||
conn.commit()
|
||||
return correction_no
|
||||
|
||||
|
||||
def read_attempt_rows(conn):
|
||||
"""All ledger rows in attempt order. Raises the backend's
|
||||
OperationalError when the table does not exist yet; the
|
||||
endpoint caller treats that as "no ledger", not as a failure."""
|
||||
endpoint caller treats that as "no ledger", not as a failure.
|
||||
|
||||
``SELECT *`` is deliberate: a table created from gap 3's DDL
|
||||
has no ``corrects_attempt_no`` column, and naming that column
|
||||
here would turn those readable rows into an endpoint error.
|
||||
With ``*`` the rows still read, and ``aggregate_attempts``
|
||||
treats a missing ``corrects_attempt_no`` as a plain attempt
|
||||
row (see ``_field_optional``).
|
||||
"""
|
||||
return conn.execute(
|
||||
"SELECT task_id, attempt_no, outcome, is_defect, "
|
||||
"defect_category FROM task_attempts "
|
||||
"SELECT * FROM task_attempts "
|
||||
"ORDER BY task_id, attempt_no").fetchall()
|
||||
|
||||
|
||||
@@ -180,6 +303,19 @@ def _field(row, name):
|
||||
return row[name]
|
||||
|
||||
|
||||
def _field_optional(row, name):
|
||||
"""Like ``_field`` but missing keys mean "absent", not an
|
||||
error. Used ONLY for ``corrects_attempt_no``: rows built by
|
||||
gap-3-era readers (plain dicts, or tables without the column)
|
||||
carry no such key, and they are plain attempt rows by
|
||||
definition -- failing loudly there would break every gap-3
|
||||
aggregate test for no benefit."""
|
||||
try:
|
||||
return row[name]
|
||||
except (KeyError, IndexError, TypeError):
|
||||
return None
|
||||
|
||||
|
||||
def aggregate_attempts(rows):
|
||||
"""Pure aggregation: ledger rows -> ``{task_id: record}`` where
|
||||
``record`` is the ``attempts``/``defects``/``defect_category``
|
||||
@@ -187,24 +323,55 @@ def aggregate_attempts(rows):
|
||||
|
||||
Interrupted rows are excluded from both counts, and a task
|
||||
with only interrupted rows is omitted entirely (no counted
|
||||
attempt => its rates stay unknown, never 0).
|
||||
attempt => its rates stay unknown, never 0). Correction rows
|
||||
(``corrects_attempt_no`` set) are excluded from the attempt
|
||||
count and instead OVERWRITE the defect judgement of their
|
||||
target attempt, in attempt order, last-wins: a task's
|
||||
``defects`` is the number of its counted attempts whose
|
||||
judgement AFTER corrections is a defect, and its
|
||||
``defect_category`` is the category of its FIRST defect
|
||||
attempt in attempt order AFTER corrections.
|
||||
"""
|
||||
per_task = {}
|
||||
ordered = sorted(
|
||||
rows,
|
||||
key=lambda r: (_field(r, "task_id"), _field(r, "attempt_no")))
|
||||
# Per task: attempt_no -> [is_defect, defect_category], in
|
||||
# attempt order (insertion order = attempt_no order, since the
|
||||
# input was sorted). Corrections mutate the target entry; they
|
||||
# never create an entry of their own.
|
||||
judged = {}
|
||||
for row in ordered:
|
||||
task_id = _field(row, "task_id")
|
||||
corrects = _field_optional(row, "corrects_attempt_no")
|
||||
if corrects is not None:
|
||||
entries = judged.get(task_id)
|
||||
if entries is not None and corrects in entries:
|
||||
entries[corrects] = [
|
||||
bool(_field(row, "is_defect")),
|
||||
_field(row, "defect_category")]
|
||||
continue
|
||||
if _field(row, "outcome") == "interrupted":
|
||||
continue
|
||||
entry = per_task.setdefault(
|
||||
task_id, {"attempts": 0, "defects": 0,
|
||||
"defect_category": None})
|
||||
entry["attempts"] += 1
|
||||
if _field(row, "is_defect"):
|
||||
entry["defects"] += 1
|
||||
if entry["defect_category"] is None:
|
||||
entry["defect_category"] = _field(row, "defect_category")
|
||||
entries = judged.setdefault(task_id, {})
|
||||
entries[_field(row, "attempt_no")] = [
|
||||
bool(_field(row, "is_defect")),
|
||||
_field(row, "defect_category")]
|
||||
for task_id, entries in judged.items():
|
||||
attempts = len(entries)
|
||||
if attempts == 0:
|
||||
continue
|
||||
defects = 0
|
||||
defect_category = None
|
||||
for _no in sorted(entries):
|
||||
is_defect, category = entries[_no]
|
||||
if is_defect:
|
||||
defects += 1
|
||||
if defect_category is None:
|
||||
defect_category = category
|
||||
per_task[task_id] = {
|
||||
"attempts": attempts, "defects": defects,
|
||||
"defect_category": defect_category}
|
||||
return per_task
|
||||
|
||||
|
||||
|
||||
Reference in new issue
Block a user