codex-app-server/cloud/attempt_ledger.py

"""Task-attempt ledger for the step-5 metrics loop (audit gap 3).

Choice made here, with the trade-off stated rather than hidden:

- This module adds a SEPARATE ``task_attempts`` table; it does NOT
  add ``attempts``/``defects`` columns to ``tasks``. Columns would be
  simpler to query (one row per task, no join), but they collapse
  history into two counters: a task's per-attempt story (which
  attempt failed, with what defect category, in what order) would be
  unrecoverable, and widening the hot ``tasks`` table on production
  Postgres is a heavier migration than adding a new, empty table.
  The ledger costs one aggregate query at report time, and that
  query is exactly what ``aggregate_attempts`` exists to pin down.
- The ledger is append-only in intent: ``record_attempt`` only ever
  inserts a new ``(task_id, attempt_no)`` row (``attempt_no`` is
  ``MAX + 1`` per task, never reused). There is no update path, so a
  recorded attempt cannot be silently rewritten into a first-pass
  success after the fact.

Semantics (pinned by tests, not left to each caller):

- One row = one real attempt at the task. ``outcome`` is one of
  ``success`` / ``failure`` / ``interrupted``.
- ``interrupted`` rows are kept for audit but NEVER count toward
  ``attempts`` or ``defects``: a platform interruption that
  auto-resumes as a fresh run is a platform event, not an agent
  failure, and the step-5 rule that ``interrupted`` is not a
  terminal task status would be meaningless if the interruption
  still inflated the attempt count. A task whose ledger holds only
  interrupted rows has NO counted attempt, so it is omitted from
  the aggregate entirely -- the endpoint then still reports its
  rates as unknown (``None``), never as 0.
- ``is_defect`` is set explicitly by the recorder, never inferred
  from ``outcome``: a failed attempt is not automatically a defect
  (a retry can fail for a transient reason without being a
  defect round), and a defect round is a judgement the recording
  caller makes. ``defect_category`` requires ``is_defect``; an
  interrupted row can never be a defect. Both contradictions raise
  ``ValueError`` instead of being stored as ambiguous data.
- ``defect_category`` for a task (the single category that
  ``compute_metrics`` carries per task) is the category of its
  FIRST defect row in attempt order. Later defect rows still count
  toward ``defects``; only the category label is first-wins, so it
  is deterministic. Corrections (below) re-evaluate this label:
  the first defect row AFTER corrections is applied wins.
- A defect judgement can be CORRECTED, never rewritten:
  ``record_correction`` appends a new row whose
  ``corrects_attempt_no`` points at the attempt it corrects; the
  old row stays untouched, and aggregation applies corrections in
  attempt order with last-wins (so a wrongly marked defect can be
  unmarked, and a defect missed at record time can be added after
  review). A correction row is never itself counted as an
  attempt, cannot change an attempt's outcome (only its defect
  judgement), and cannot target an interrupted row, another
  correction row, or a nonexistent attempt -- all raise
  ``ValueError`` and write nothing. Who may correct is a policy
  question answered in ``cloud/attempt_hook.py`` (explicit
  recorder only, maintainer bar); the worker hook never writes
  corrections and never marks a defect.
- Old tasks have no ledger rows, and that is NOT backfilled: the
  aggregate simply has no entry for them, the endpoint leaves their
  ``attempts``/``defects`` as ``None``, and ``compute_metrics``
  excludes them from rate denominators (they are neither counted
  as first-pass nor as defect-free). Fabricating a row for an old
  task (e.g. ``attempts=1, defects=0`` from "it completed") would
  be inventing history.

Persistence notes:

- SQLite: ``ensure_task_attempts_table`` runs
  ``CREATE TABLE IF NOT EXISTS`` -- idempotent, safe to call on
  every write, and a no-op on a database that already has the
  table (including one that already has rows).
- Postgres: the production table is NOT created by
  ``init_postgres_schema`` (deliberately: that schema is covered by
  the migrations test asserting ``MIGRATION_TABLES`` equals the
  schema set, and this ledger must not silently widen that set).
  ``PG_DDL`` below is the exact statement to run manually on
  production Postgres and verify (``\\d task_attempts`` or the
  ``information_schema`` equivalent) before any Postgres-backed
  endpoint is expected to return real rates. The DDL uses only
  Postgres-valid types (``TEXT``/``INTEGER``/``DOUBLE PRECISION``),
  a composite primary key, and no ``AUTOINCREMENT``/identity
  column, so it needs no sequence handling; the parameterized
  ``INSERT`` uses ``?`` placeholders, which the ``cloud.db``
  Postgres adapter translates to ``%s`` (see the adapter test in
  ``test_attempt_ledger.py``). Until that manual step runs, the
  endpoint treats a missing table exactly like "no ledger rows":
  unknown rates, not an error -- the same fail-soft shape already
  used for missing ``cost_records``.
- The read path (``GET /metrics-report``) never creates the table:
  a read-only endpoint must not mutate schema. Table creation
  happens only on the write path, inside ``record_attempt`` /
  ``record_correction``.
- Lifecycle hook (this track, see ``cloud/attempt_hook.py``):
  the worker writes exactly one row per COUNTED dispatch (see
  that module's policy for which dispatches count: human/first
  rounds and human breakpoint resumes count; platform auto-resume
  reruns, coordinator fan-out, and pre-execution gate failures do
  not). The hook is OFF by default (``attempt_ledger_hook=False``):
  with it off, this module's behaviour is exactly audit gap 3 --
  the ledger fills only through explicit ``record_attempt`` /
  ``record_correction`` calls, and production rates stay honestly
  unknown until the hook is deliberately enabled. The worker hook
  always records ``is_defect=False``: it never judges a defect,
  so enabling it can never turn a transient retry into a defect.
- Schema evolution note (pre-production, this track):
  ``corrects_attempt_no`` was added to the DDL after audit gap 3
  but BEFORE either DDL reached production (gap 3 is still an
  OPEN, unmerged PR), so no production table needs migration:
  fresh tables get the column from the DDL below. A database
  created from gap 3's DDL alone (temporary/test databases only)
  gains the column with one statement --
  ``ALTER TABLE task_attempts ADD COLUMN corrects_attempt_no
  INTEGER`` -- which is also the documented step if gap 3's DDL
  was ever run on production Postgres before this track lands.
  The read path uses ``SELECT *`` deliberately so a table without
  the column still reads (its rows aggregate as plain attempts).

This module imports no ``cloud.*`` module and opens no database
connection itself: callers pass a connection in, so the layer rules
(``arch_conformance``: no direct driver connection outside
``cloud.db``) are untouched -- there is no ``sqlite3.connect`` and
no ``psycopg*`` import here at all.
"""

import time

TABLE = "task_attempts"

OUTCOMES = frozenset({"success", "failure", "interrupted"})

# One DDL serves both backends: every type and the composite key are
# valid in SQLite and in Postgres, and there is no AUTOINCREMENT or
# identity column for either backend's sequence handling to chase.
# ``corrects_attempt_no`` (NULL = a real attempt row; non-NULL = a
# correction row for that attempt_no, never counted as an attempt)
# was added by the lifecycle-hook track; it SUPERSEDES gap 3's DDL
# for any production manual step (gap 3 never merged, so no
# production table exists yet -- see the module docstring).
SQLITE_DDL = (
    "CREATE TABLE IF NOT EXISTS task_attempts("
    "task_id TEXT NOT NULL, "
    "attempt_no INTEGER NOT NULL, "
    "outcome TEXT NOT NULL, "
    "is_defect INTEGER NOT NULL DEFAULT 0, "
    "defect_category TEXT, "
    "created_at DOUBLE PRECISION, "
    "corrects_attempt_no INTEGER, "
    "PRIMARY KEY (task_id, attempt_no))"
)

# Production Postgres: run this verbatim by hand and verify the
# table exists before expecting real rates from a Postgres-backed
# endpoint. It is intentionally NOT part of POSTGRES_SCHEMA (see
# the module docstring: migrations coverage must not widen silently).
# This is the NEW DDL (with ``corrects_attempt_no``): if gap 3's
# older DDL was ever run first, follow it with
# ``ALTER TABLE task_attempts ADD COLUMN corrects_attempt_no
# INTEGER`` and verify before expecting corrections to work.
PG_DDL = SQLITE_DDL

# One-line upgrade for a table created from gap 3's DDL (no
# ``corrects_attempt_no`` column yet). Valid in both backends.
OLD_TABLE_ALTER = (
    "ALTER TABLE task_attempts "
    "ADD COLUMN corrects_attempt_no INTEGER"
)


def ensure_task_attempts_table(conn):
    """Create the ledger table if absent. Idempotent; does not
    touch existing rows. The caller owns the commit (``record_attempt``
    commits; a bare ensure inside a larger transaction does not)."""
    conn.execute(SQLITE_DDL)


def record_attempt(conn, task_id, outcome, is_defect=False,
                   defect_category=None, created_at=None):
    """Append one attempt row for ``task_id``. Returns ``attempt_no``.

    Validates before writing: unknown outcome, a defect flag on an
    interrupted row, and a defect category without a defect flag
    all raise ``ValueError`` and write nothing.
    """
    if not task_id:
        raise ValueError("task_id is required")
    if outcome not in OUTCOMES:
        raise ValueError("outcome must be one of %s, got %r"
                         % (sorted(OUTCOMES), outcome))
    if outcome == "interrupted" and is_defect:
        raise ValueError("an interrupted attempt is never a defect")
    if defect_category is not None and not is_defect:
        raise ValueError("defect_category requires is_defect=True")
    ensure_task_attempts_table(conn)
    row = conn.execute(
        "SELECT COALESCE(MAX(attempt_no), 0) FROM task_attempts "
        "WHERE task_id = ?",
        (task_id,),
    ).fetchone()
    attempt_no = int(row[0]) + 1
    conn.execute(
        "INSERT INTO task_attempts(task_id, attempt_no, outcome, "
        "is_defect, defect_category, created_at, "
        "corrects_attempt_no) "
        "VALUES(?, ?, ?, ?, ?, ?, NULL)",
        (task_id, attempt_no, outcome, 1 if is_defect else 0,
         defect_category,
         time.time() if created_at is None else created_at),
    )
    conn.commit()
    return attempt_no


def record_correction(conn, task_id, attempt_no, is_defect,
                      defect_category=None, created_at=None):
    """Append a CORRECTION row for an already-recorded attempt.

    The correction does not rewrite history: the target row stays
    exactly as recorded, and this new row (its own ``MAX + 1``
    ``attempt_no``) carries the corrected defect judgement in
    ``is_defect`` / ``defect_category`` plus
    ``corrects_attempt_no = attempt_no``. Aggregation applies
    corrections in attempt order, last-wins, and never counts a
    correction row as an attempt.

    Only the defect judgement is correctable: the correction row
    copies the target's outcome, so it cannot turn a success into
    a failure or vice versa. Validates before writing: missing
    ``task_id``, a nonexistent target, a target that is itself a
    correction row, a target whose outcome is ``interrupted``
    (interrupted rows are never defects, so there is no defect
    judgement to correct), and a defect category without a defect
    flag all raise ``ValueError`` and write nothing. On a table
    created from gap 3's DDL (no ``corrects_attempt_no`` column),
    the INSERT raises the backend's OperationalError loudly --
    run ``OLD_TABLE_ALTER`` first (see the module docstring);
    a correction must never silently degrade into a plain row.
    Returns the correction row's own ``attempt_no``.
    """
    if not task_id:
        raise ValueError("task_id is required")
    if defect_category is not None and not is_defect:
        raise ValueError("defect_category requires is_defect=True")
    ensure_task_attempts_table(conn)
    target = conn.execute(
        "SELECT outcome, corrects_attempt_no FROM task_attempts "
        "WHERE task_id = ? AND attempt_no = ?",
        (task_id, attempt_no),
    ).fetchone()
    if target is None:
        raise ValueError(
            "no such attempt to correct: %s #%s" % (task_id, attempt_no))
    if target["corrects_attempt_no"] is not None:
        raise ValueError(
            "cannot correct a correction row: %s #%s"
            % (task_id, attempt_no))
    if target["outcome"] == "interrupted":
        raise ValueError(
            "an interrupted attempt has no defect judgement to "
            "correct: %s #%s" % (task_id, attempt_no))
    row = conn.execute(
        "SELECT COALESCE(MAX(attempt_no), 0) FROM task_attempts "
        "WHERE task_id = ?",
        (task_id,),
    ).fetchone()
    correction_no = int(row[0]) + 1
    conn.execute(
        "INSERT INTO task_attempts(task_id, attempt_no, outcome, "
        "is_defect, defect_category, created_at, "
        "corrects_attempt_no) "
        "VALUES(?, ?, ?, ?, ?, ?, ?)",
        (task_id, correction_no, target["outcome"],
         1 if is_defect else 0, defect_category,
         time.time() if created_at is None else created_at,
         attempt_no),
    )
    conn.commit()
    return correction_no


def read_attempt_rows(conn):
    """All ledger rows in attempt order. Raises the backend's
    OperationalError when the table does not exist yet; the
    endpoint caller treats that as "no ledger", not as a failure.

    ``SELECT *`` is deliberate: a table created from gap 3's DDL
    has no ``corrects_attempt_no`` column, and naming that column
    here would turn those readable rows into an endpoint error.
    With ``*`` the rows still read, and ``aggregate_attempts``
    treats a missing ``corrects_attempt_no`` as a plain attempt
    row (see ``_field_optional``).
    """
    return conn.execute(
        "SELECT * FROM task_attempts "
        "ORDER BY task_id, attempt_no").fetchall()


def _field(row, name):
    # sqlite3.Row, dict, and the Postgres adapter's rows all support
    # key access; a plain tuple row is not accepted (callers pass
    # rows, not raw tuples, so a wrong shape fails loudly here).
    return row[name]


def _field_optional(row, name):
    """Like ``_field`` but missing keys mean "absent", not an
    error. Used ONLY for ``corrects_attempt_no``: rows built by
    gap-3-era readers (plain dicts, or tables without the column)
    carry no such key, and they are plain attempt rows by
    definition -- failing loudly there would break every gap-3
    aggregate test for no benefit."""
    try:
        return row[name]
    except (KeyError, IndexError, TypeError):
        return None


def aggregate_attempts(rows):
    """Pure aggregation: ledger rows -> ``{task_id: record}`` where
    ``record`` is the ``attempts``/``defects``/``defect_category``
    dict that ``compute_metrics`` consumes.

    Interrupted rows are excluded from both counts, and a task
    with only interrupted rows is omitted entirely (no counted
    attempt => its rates stay unknown, never 0). Correction rows
    (``corrects_attempt_no`` set) are excluded from the attempt
    count and instead OVERWRITE the defect judgement of their
    target attempt, in attempt order, last-wins: a task's
    ``defects`` is the number of its counted attempts whose
    judgement AFTER corrections is a defect, and its
    ``defect_category`` is the category of its FIRST defect
    attempt in attempt order AFTER corrections.
    """
    per_task = {}
    ordered = sorted(
        rows,
        key=lambda r: (_field(r, "task_id"), _field(r, "attempt_no")))
    # Per task: attempt_no -> [is_defect, defect_category], in
    # attempt order (insertion order = attempt_no order, since the
    # input was sorted). Corrections mutate the target entry; they
    # never create an entry of their own.
    judged = {}
    for row in ordered:
        task_id = _field(row, "task_id")
        corrects = _field_optional(row, "corrects_attempt_no")
        if corrects is not None:
            entries = judged.get(task_id)
            if entries is not None and corrects in entries:
                entries[corrects] = [
                    bool(_field(row, "is_defect")),
                    _field(row, "defect_category")]
            continue
        if _field(row, "outcome") == "interrupted":
            continue
        entries = judged.setdefault(task_id, {})
        entries[_field(row, "attempt_no")] = [
            bool(_field(row, "is_defect")),
            _field(row, "defect_category")]
    for task_id, entries in judged.items():
        attempts = len(entries)
        if attempts == 0:
            continue
        defects = 0
        defect_category = None
        for _no in sorted(entries):
            is_defect, category = entries[_no]
            if is_defect:
                defects += 1
                if defect_category is None:
                    defect_category = category
        per_task[task_id] = {
            "attempts": attempts, "defects": defects,
            "defect_category": defect_category}
    return per_task


def apply_to_records(records, aggregates):
    """Return a new record list with ledger values merged in.

    Records for tasks with no aggregate entry keep whatever
    ``attempts``/``defects`` they already carried (the endpoint
    passes ``None`` there, i.e. old tasks stay unknown). Input
    records are not modified.
    """
    merged = []
    for record in records:
        out = dict(record)
        agg = aggregates.get(record.get("id"))
        if agg is not None:
            out["attempts"] = agg["attempts"]
            out["defects"] = agg["defects"]
            out["defect_category"] = agg["defect_category"]
        merged.append(out)
    return merged