"""Task-attempt ledger for the step-5 metrics loop (audit gap 3). Choice made here, with the trade-off stated rather than hidden: - This module adds a SEPARATE ``task_attempts`` table; it does NOT add ``attempts``/``defects`` columns to ``tasks``. Columns would be simpler to query (one row per task, no join), but they collapse history into two counters: a task's per-attempt story (which attempt failed, with what defect category, in what order) would be unrecoverable, and widening the hot ``tasks`` table on production Postgres is a heavier migration than adding a new, empty table. The ledger costs one aggregate query at report time, and that query is exactly what ``aggregate_attempts`` exists to pin down. - The ledger is append-only in intent: ``record_attempt`` only ever inserts a new ``(task_id, attempt_no)`` row (``attempt_no`` is ``MAX + 1`` per task, never reused). There is no update path, so a recorded attempt cannot be silently rewritten into a first-pass success after the fact. Semantics (pinned by tests, not left to each caller): - One row = one real attempt at the task. ``outcome`` is one of ``success`` / ``failure`` / ``interrupted``. - ``interrupted`` rows are kept for audit but NEVER count toward ``attempts`` or ``defects``: a platform interruption that auto-resumes as a fresh run is a platform event, not an agent failure, and the step-5 rule that ``interrupted`` is not a terminal task status would be meaningless if the interruption still inflated the attempt count. A task whose ledger holds only interrupted rows has NO counted attempt, so it is omitted from the aggregate entirely -- the endpoint then still reports its rates as unknown (``None``), never as 0. - ``is_defect`` is set explicitly by the recorder, never inferred from ``outcome``: a failed attempt is not automatically a defect (a retry can fail for a transient reason without being a defect round), and a defect round is a judgement the recording caller makes. ``defect_category`` requires ``is_defect``; an interrupted row can never be a defect. Both contradictions raise ``ValueError`` instead of being stored as ambiguous data. - ``defect_category`` for a task (the single category that ``compute_metrics`` carries per task) is the category of its FIRST defect row in attempt order. Later defect rows still count toward ``defects``; only the category label is first-wins, so it is deterministic. Corrections (below) re-evaluate this label: the first defect row AFTER corrections is applied wins. - A defect judgement can be CORRECTED, never rewritten: ``record_correction`` appends a new row whose ``corrects_attempt_no`` points at the attempt it corrects; the old row stays untouched, and aggregation applies corrections in attempt order with last-wins (so a wrongly marked defect can be unmarked, and a defect missed at record time can be added after review). A correction row is never itself counted as an attempt, cannot change an attempt's outcome (only its defect judgement), and cannot target an interrupted row, another correction row, or a nonexistent attempt -- all raise ``ValueError`` and write nothing. Who may correct is a policy question answered in ``cloud/attempt_hook.py`` (explicit recorder only, maintainer bar); the worker hook never writes corrections and never marks a defect. - Old tasks have no ledger rows, and that is NOT backfilled: the aggregate simply has no entry for them, the endpoint leaves their ``attempts``/``defects`` as ``None``, and ``compute_metrics`` excludes them from rate denominators (they are neither counted as first-pass nor as defect-free). Fabricating a row for an old task (e.g. ``attempts=1, defects=0`` from "it completed") would be inventing history. Persistence notes: - SQLite: ``ensure_task_attempts_table`` runs ``CREATE TABLE IF NOT EXISTS`` -- idempotent, safe to call on every write, and a no-op on a database that already has the table (including one that already has rows). - Postgres: the production table is NOT created by ``init_postgres_schema`` (deliberately: that schema is covered by the migrations test asserting ``MIGRATION_TABLES`` equals the schema set, and this ledger must not silently widen that set). ``PG_DDL`` below is the exact statement to run manually on production Postgres and verify (``\\d task_attempts`` or the ``information_schema`` equivalent) before any Postgres-backed endpoint is expected to return real rates. The DDL uses only Postgres-valid types (``TEXT``/``INTEGER``/``DOUBLE PRECISION``), a composite primary key, and no ``AUTOINCREMENT``/identity column, so it needs no sequence handling; the parameterized ``INSERT`` uses ``?`` placeholders, which the ``cloud.db`` Postgres adapter translates to ``%s`` (see the adapter test in ``test_attempt_ledger.py``). Until that manual step runs, the endpoint treats a missing table exactly like "no ledger rows": unknown rates, not an error -- the same fail-soft shape already used for missing ``cost_records``. - The read path (``GET /metrics-report``) never creates the table: a read-only endpoint must not mutate schema. Table creation happens only on the write path, inside ``record_attempt`` / ``record_correction``. - Lifecycle hook (this track, see ``cloud/attempt_hook.py``): the worker writes exactly one row per COUNTED dispatch (see that module's policy for which dispatches count: human/first rounds and human breakpoint resumes count; platform auto-resume reruns, coordinator fan-out, and pre-execution gate failures do not). The hook is OFF by default (``attempt_ledger_hook=False``): with it off, this module's behaviour is exactly audit gap 3 -- the ledger fills only through explicit ``record_attempt`` / ``record_correction`` calls, and production rates stay honestly unknown until the hook is deliberately enabled. The worker hook always records ``is_defect=False``: it never judges a defect, so enabling it can never turn a transient retry into a defect. - Schema evolution note (pre-production, this track): ``corrects_attempt_no`` was added to the DDL after audit gap 3 but BEFORE either DDL reached production (gap 3 is still an OPEN, unmerged PR), so no production table needs migration: fresh tables get the column from the DDL below. A database created from gap 3's DDL alone (temporary/test databases only) gains the column with one statement -- ``ALTER TABLE task_attempts ADD COLUMN corrects_attempt_no INTEGER`` -- which is also the documented step if gap 3's DDL was ever run on production Postgres before this track lands. The read path uses ``SELECT *`` deliberately so a table without the column still reads (its rows aggregate as plain attempts). This module imports no ``cloud.*`` module and opens no database connection itself: callers pass a connection in, so the layer rules (``arch_conformance``: no direct driver connection outside ``cloud.db``) are untouched -- there is no ``sqlite3.connect`` and no ``psycopg*`` import here at all. """ import time TABLE = "task_attempts" OUTCOMES = frozenset({"success", "failure", "interrupted"}) # One DDL serves both backends: every type and the composite key are # valid in SQLite and in Postgres, and there is no AUTOINCREMENT or # identity column for either backend's sequence handling to chase. # ``corrects_attempt_no`` (NULL = a real attempt row; non-NULL = a # correction row for that attempt_no, never counted as an attempt) # was added by the lifecycle-hook track; it SUPERSEDES gap 3's DDL # for any production manual step (gap 3 never merged, so no # production table exists yet -- see the module docstring). SQLITE_DDL = ( "CREATE TABLE IF NOT EXISTS task_attempts(" "task_id TEXT NOT NULL, " "attempt_no INTEGER NOT NULL, " "outcome TEXT NOT NULL, " "is_defect INTEGER NOT NULL DEFAULT 0, " "defect_category TEXT, " "created_at DOUBLE PRECISION, " "corrects_attempt_no INTEGER, " "PRIMARY KEY (task_id, attempt_no))" ) # Production Postgres: run this verbatim by hand and verify the # table exists before expecting real rates from a Postgres-backed # endpoint. It is intentionally NOT part of POSTGRES_SCHEMA (see # the module docstring: migrations coverage must not widen silently). # This is the NEW DDL (with ``corrects_attempt_no``): if gap 3's # older DDL was ever run first, follow it with # ``ALTER TABLE task_attempts ADD COLUMN corrects_attempt_no # INTEGER`` and verify before expecting corrections to work. PG_DDL = SQLITE_DDL # One-line upgrade for a table created from gap 3's DDL (no # ``corrects_attempt_no`` column yet). Valid in both backends. OLD_TABLE_ALTER = ( "ALTER TABLE task_attempts " "ADD COLUMN corrects_attempt_no INTEGER" ) def ensure_task_attempts_table(conn): """Create the ledger table if absent. Idempotent; does not touch existing rows. The caller owns the commit (``record_attempt`` commits; a bare ensure inside a larger transaction does not).""" conn.execute(SQLITE_DDL) def record_attempt(conn, task_id, outcome, is_defect=False, defect_category=None, created_at=None): """Append one attempt row for ``task_id``. Returns ``attempt_no``. Validates before writing: unknown outcome, a defect flag on an interrupted row, and a defect category without a defect flag all raise ``ValueError`` and write nothing. """ if not task_id: raise ValueError("task_id is required") if outcome not in OUTCOMES: raise ValueError("outcome must be one of %s, got %r" % (sorted(OUTCOMES), outcome)) if outcome == "interrupted" and is_defect: raise ValueError("an interrupted attempt is never a defect") if defect_category is not None and not is_defect: raise ValueError("defect_category requires is_defect=True") ensure_task_attempts_table(conn) row = conn.execute( "SELECT COALESCE(MAX(attempt_no), 0) FROM task_attempts " "WHERE task_id = ?", (task_id,), ).fetchone() attempt_no = int(row[0]) + 1 conn.execute( "INSERT INTO task_attempts(task_id, attempt_no, outcome, " "is_defect, defect_category, created_at, " "corrects_attempt_no) " "VALUES(?, ?, ?, ?, ?, ?, NULL)", (task_id, attempt_no, outcome, 1 if is_defect else 0, defect_category, time.time() if created_at is None else created_at), ) conn.commit() return attempt_no def record_correction(conn, task_id, attempt_no, is_defect, defect_category=None, created_at=None): """Append a CORRECTION row for an already-recorded attempt. The correction does not rewrite history: the target row stays exactly as recorded, and this new row (its own ``MAX + 1`` ``attempt_no``) carries the corrected defect judgement in ``is_defect`` / ``defect_category`` plus ``corrects_attempt_no = attempt_no``. Aggregation applies corrections in attempt order, last-wins, and never counts a correction row as an attempt. Only the defect judgement is correctable: the correction row copies the target's outcome, so it cannot turn a success into a failure or vice versa. Validates before writing: missing ``task_id``, a nonexistent target, a target that is itself a correction row, a target whose outcome is ``interrupted`` (interrupted rows are never defects, so there is no defect judgement to correct), and a defect category without a defect flag all raise ``ValueError`` and write nothing. On a table created from gap 3's DDL (no ``corrects_attempt_no`` column), the INSERT raises the backend's OperationalError loudly -- run ``OLD_TABLE_ALTER`` first (see the module docstring); a correction must never silently degrade into a plain row. Returns the correction row's own ``attempt_no``. """ if not task_id: raise ValueError("task_id is required") if defect_category is not None and not is_defect: raise ValueError("defect_category requires is_defect=True") ensure_task_attempts_table(conn) target = conn.execute( "SELECT outcome, corrects_attempt_no FROM task_attempts " "WHERE task_id = ? AND attempt_no = ?", (task_id, attempt_no), ).fetchone() if target is None: raise ValueError( "no such attempt to correct: %s #%s" % (task_id, attempt_no)) if target["corrects_attempt_no"] is not None: raise ValueError( "cannot correct a correction row: %s #%s" % (task_id, attempt_no)) if target["outcome"] == "interrupted": raise ValueError( "an interrupted attempt has no defect judgement to " "correct: %s #%s" % (task_id, attempt_no)) row = conn.execute( "SELECT COALESCE(MAX(attempt_no), 0) FROM task_attempts " "WHERE task_id = ?", (task_id,), ).fetchone() correction_no = int(row[0]) + 1 conn.execute( "INSERT INTO task_attempts(task_id, attempt_no, outcome, " "is_defect, defect_category, created_at, " "corrects_attempt_no) " "VALUES(?, ?, ?, ?, ?, ?, ?)", (task_id, correction_no, target["outcome"], 1 if is_defect else 0, defect_category, time.time() if created_at is None else created_at, attempt_no), ) conn.commit() return correction_no def read_attempt_rows(conn): """All ledger rows in attempt order. Raises the backend's OperationalError when the table does not exist yet; the endpoint caller treats that as "no ledger", not as a failure. ``SELECT *`` is deliberate: a table created from gap 3's DDL has no ``corrects_attempt_no`` column, and naming that column here would turn those readable rows into an endpoint error. With ``*`` the rows still read, and ``aggregate_attempts`` treats a missing ``corrects_attempt_no`` as a plain attempt row (see ``_field_optional``). """ return conn.execute( "SELECT * FROM task_attempts " "ORDER BY task_id, attempt_no").fetchall() def _field(row, name): # sqlite3.Row, dict, and the Postgres adapter's rows all support # key access; a plain tuple row is not accepted (callers pass # rows, not raw tuples, so a wrong shape fails loudly here). return row[name] def _field_optional(row, name): """Like ``_field`` but missing keys mean "absent", not an error. Used ONLY for ``corrects_attempt_no``: rows built by gap-3-era readers (plain dicts, or tables without the column) carry no such key, and they are plain attempt rows by definition -- failing loudly there would break every gap-3 aggregate test for no benefit.""" try: return row[name] except (KeyError, IndexError, TypeError): return None def aggregate_attempts(rows): """Pure aggregation: ledger rows -> ``{task_id: record}`` where ``record`` is the ``attempts``/``defects``/``defect_category`` dict that ``compute_metrics`` consumes. Interrupted rows are excluded from both counts, and a task with only interrupted rows is omitted entirely (no counted attempt => its rates stay unknown, never 0). Correction rows (``corrects_attempt_no`` set) are excluded from the attempt count and instead OVERWRITE the defect judgement of their target attempt, in attempt order, last-wins: a task's ``defects`` is the number of its counted attempts whose judgement AFTER corrections is a defect, and its ``defect_category`` is the category of its FIRST defect attempt in attempt order AFTER corrections. """ per_task = {} ordered = sorted( rows, key=lambda r: (_field(r, "task_id"), _field(r, "attempt_no"))) # Per task: attempt_no -> [is_defect, defect_category], in # attempt order (insertion order = attempt_no order, since the # input was sorted). Corrections mutate the target entry; they # never create an entry of their own. judged = {} for row in ordered: task_id = _field(row, "task_id") corrects = _field_optional(row, "corrects_attempt_no") if corrects is not None: entries = judged.get(task_id) if entries is not None and corrects in entries: entries[corrects] = [ bool(_field(row, "is_defect")), _field(row, "defect_category")] continue if _field(row, "outcome") == "interrupted": continue entries = judged.setdefault(task_id, {}) entries[_field(row, "attempt_no")] = [ bool(_field(row, "is_defect")), _field(row, "defect_category")] for task_id, entries in judged.items(): attempts = len(entries) if attempts == 0: continue defects = 0 defect_category = None for _no in sorted(entries): is_defect, category = entries[_no] if is_defect: defects += 1 if defect_category is None: defect_category = category per_task[task_id] = { "attempts": attempts, "defects": defects, "defect_category": defect_category} return per_task def apply_to_records(records, aggregates): """Return a new record list with ledger values merged in. Records for tasks with no aggregate entry keep whatever ``attempts``/``defects`` they already carried (the endpoint passes ``None`` there, i.e. old tasks stay unknown). Input records are not modified. """ merged = [] for record in records: out = dict(record) agg = aggregates.get(record.get("id")) if agg is not None: out["attempts"] = agg["attempts"] out["defects"] = agg["defects"] out["defect_category"] = agg["defect_category"] merged.append(out) return merged