437 lines
19 KiB
Python
437 lines
19 KiB
Python
"""Task-attempt ledger for the step-5 metrics loop (audit gap 3).
|
||
|
||
Choice made here, with the trade-off stated rather than hidden:
|
||
|
||
- This module adds a SEPARATE ``task_attempts`` table; it does NOT
|
||
add ``attempts``/``defects`` columns to ``tasks``. Columns would be
|
||
simpler to query (one row per task, no join), but they collapse
|
||
history into two counters: a task's per-attempt story (which
|
||
attempt failed, with what defect category, in what order) would be
|
||
unrecoverable, and widening the hot ``tasks`` table on production
|
||
Postgres is a heavier migration than adding a new, empty table.
|
||
The ledger costs one aggregate query at report time, and that
|
||
query is exactly what ``aggregate_attempts`` exists to pin down.
|
||
- The ledger is append-only in intent: ``record_attempt`` only ever
|
||
inserts a new ``(task_id, attempt_no)`` row (``attempt_no`` is
|
||
``MAX + 1`` per task, never reused). There is no update path, so a
|
||
recorded attempt cannot be silently rewritten into a first-pass
|
||
success after the fact.
|
||
|
||
Semantics (pinned by tests, not left to each caller):
|
||
|
||
- One row = one real attempt at the task. ``outcome`` is one of
|
||
``success`` / ``failure`` / ``interrupted``.
|
||
- ``interrupted`` rows are kept for audit but NEVER count toward
|
||
``attempts`` or ``defects``: a platform interruption that
|
||
auto-resumes as a fresh run is a platform event, not an agent
|
||
failure, and the step-5 rule that ``interrupted`` is not a
|
||
terminal task status would be meaningless if the interruption
|
||
still inflated the attempt count. A task whose ledger holds only
|
||
interrupted rows has NO counted attempt, so it is omitted from
|
||
the aggregate entirely -- the endpoint then still reports its
|
||
rates as unknown (``None``), never as 0.
|
||
- ``is_defect`` is set explicitly by the recorder, never inferred
|
||
from ``outcome``: a failed attempt is not automatically a defect
|
||
(a retry can fail for a transient reason without being a
|
||
defect round), and a defect round is a judgement the recording
|
||
caller makes. ``defect_category`` requires ``is_defect``; an
|
||
interrupted row can never be a defect. Both contradictions raise
|
||
``ValueError`` instead of being stored as ambiguous data.
|
||
- ``defect_category`` for a task (the single category that
|
||
``compute_metrics`` carries per task) is the category of its
|
||
FIRST defect row in attempt order. Later defect rows still count
|
||
toward ``defects``; only the category label is first-wins, so it
|
||
is deterministic. Corrections (below) re-evaluate this label:
|
||
the first defect row AFTER corrections is applied wins.
|
||
- A defect judgement can be CORRECTED, never rewritten:
|
||
``record_correction`` appends a new row whose
|
||
``corrects_attempt_no`` points at the attempt it corrects; the
|
||
old row stays untouched, and aggregation applies corrections in
|
||
attempt order with last-wins (so a wrongly marked defect can be
|
||
unmarked, and a defect missed at record time can be added after
|
||
review). A correction row is never itself counted as an
|
||
attempt, cannot change an attempt's outcome (only its defect
|
||
judgement), and cannot target an interrupted row, another
|
||
correction row, or a nonexistent attempt -- all raise
|
||
``ValueError`` and write nothing. Who may correct is a policy
|
||
question answered in ``cloud/attempt_hook.py`` (explicit
|
||
recorder only, maintainer bar); the worker hook never writes
|
||
corrections and never marks a defect.
|
||
- Old tasks have no ledger rows, and that is NOT backfilled: the
|
||
aggregate simply has no entry for them, the endpoint leaves their
|
||
``attempts``/``defects`` as ``None``, and ``compute_metrics``
|
||
excludes them from rate denominators (they are neither counted
|
||
as first-pass nor as defect-free). Fabricating a row for an old
|
||
task (e.g. ``attempts=1, defects=0`` from "it completed") would
|
||
be inventing history.
|
||
|
||
Persistence notes:
|
||
|
||
- SQLite: ``ensure_task_attempts_table`` runs
|
||
``CREATE TABLE IF NOT EXISTS`` -- idempotent, safe to call on
|
||
every write, and a no-op on a database that already has the
|
||
table (including one that already has rows).
|
||
- Postgres: the production table is NOT created by
|
||
``init_postgres_schema`` (deliberately: that schema is covered by
|
||
the migrations test asserting ``MIGRATION_TABLES`` equals the
|
||
schema set, and this ledger must not silently widen that set).
|
||
``PG_DDL`` below is the exact statement to run manually on
|
||
production Postgres and verify (``\\d task_attempts`` or the
|
||
``information_schema`` equivalent) before any Postgres-backed
|
||
endpoint is expected to return real rates. The DDL uses only
|
||
Postgres-valid types (``TEXT``/``INTEGER``/``DOUBLE PRECISION``),
|
||
a composite primary key, and no ``AUTOINCREMENT``/identity
|
||
column, so it needs no sequence handling; the parameterized
|
||
``INSERT`` uses ``?`` placeholders, which the ``cloud.db``
|
||
Postgres adapter translates to ``%s`` (see the adapter test in
|
||
``test_attempt_ledger.py``). Until that manual step runs, the
|
||
endpoint treats a missing table exactly like "no ledger rows":
|
||
unknown rates, not an error -- the same fail-soft shape already
|
||
used for missing ``cost_records``.
|
||
- The read path (``GET /metrics-report``) never creates the table:
|
||
a read-only endpoint must not mutate schema. Table creation
|
||
happens only on the write path, inside ``record_attempt`` /
|
||
``record_correction``.
|
||
- Lifecycle hook (this track, see ``cloud/attempt_hook.py``):
|
||
the worker writes exactly one row per COUNTED dispatch (see
|
||
that module's policy for which dispatches count: human/first
|
||
rounds and human breakpoint resumes count; platform auto-resume
|
||
reruns, coordinator fan-out, and pre-execution gate failures do
|
||
not). The hook is OFF by default (``attempt_ledger_hook=False``):
|
||
with it off, this module's behaviour is exactly audit gap 3 --
|
||
the ledger fills only through explicit ``record_attempt`` /
|
||
``record_correction`` calls, and production rates stay honestly
|
||
unknown until the hook is deliberately enabled. The worker hook
|
||
always records ``is_defect=False``: it never judges a defect,
|
||
so enabling it can never turn a transient retry into a defect.
|
||
- Schema evolution note (pre-production, this track):
|
||
``corrects_attempt_no`` was added to the DDL after audit gap 3
|
||
but BEFORE either DDL reached production (gap 3 is still an
|
||
OPEN, unmerged PR), so no production table needs migration:
|
||
fresh tables get the column from the DDL below. A database
|
||
created from gap 3's DDL alone (temporary/test databases only)
|
||
gains the column with one statement --
|
||
``ALTER TABLE task_attempts ADD COLUMN corrects_attempt_no
|
||
INTEGER`` -- which is also the documented step if gap 3's DDL
|
||
was ever run on production Postgres before this track lands.
|
||
The read path uses ``SELECT *`` deliberately so a table without
|
||
the column still reads (its rows aggregate as plain attempts).
|
||
|
||
This module imports no ``cloud.*`` module and opens no database
|
||
connection itself: callers pass a connection in, so the layer rules
|
||
(``arch_conformance``: no direct driver connection outside
|
||
``cloud.db``) are untouched -- there is no ``sqlite3.connect`` and
|
||
no ``psycopg*`` import here at all.
|
||
"""
|
||
|
||
import time
|
||
|
||
TABLE = "task_attempts"
|
||
|
||
OUTCOMES = frozenset({"success", "failure", "interrupted"})
|
||
|
||
# One DDL serves both backends: every type and the composite key are
|
||
# valid in SQLite and in Postgres, and there is no AUTOINCREMENT or
|
||
# identity column for either backend's sequence handling to chase.
|
||
# ``corrects_attempt_no`` (NULL = a real attempt row; non-NULL = a
|
||
# correction row for that attempt_no, never counted as an attempt)
|
||
# was added by the lifecycle-hook track; it SUPERSEDES gap 3's DDL
|
||
# for any production manual step (gap 3 never merged, so no
|
||
# production table exists yet -- see the module docstring).
|
||
SQLITE_DDL = (
|
||
"CREATE TABLE IF NOT EXISTS task_attempts("
|
||
"task_id TEXT NOT NULL, "
|
||
"attempt_no INTEGER NOT NULL, "
|
||
"outcome TEXT NOT NULL, "
|
||
"is_defect INTEGER NOT NULL DEFAULT 0, "
|
||
"defect_category TEXT, "
|
||
"created_at DOUBLE PRECISION, "
|
||
"corrects_attempt_no INTEGER, "
|
||
"PRIMARY KEY (task_id, attempt_no))"
|
||
)
|
||
|
||
# Production Postgres: run this verbatim by hand and verify the
|
||
# table exists before expecting real rates from a Postgres-backed
|
||
# endpoint. It is intentionally NOT part of POSTGRES_SCHEMA (see
|
||
# the module docstring: migrations coverage must not widen silently).
|
||
# This is the NEW DDL (with ``corrects_attempt_no``): if gap 3's
|
||
# older DDL was ever run first, follow it with
|
||
# ``ALTER TABLE task_attempts ADD COLUMN corrects_attempt_no
|
||
# INTEGER`` and verify before expecting corrections to work.
|
||
PG_DDL = SQLITE_DDL
|
||
|
||
# One-line upgrade for a table created from gap 3's DDL (no
|
||
# ``corrects_attempt_no`` column yet). Valid in both backends.
|
||
OLD_TABLE_ALTER = (
|
||
"ALTER TABLE task_attempts "
|
||
"ADD COLUMN corrects_attempt_no INTEGER"
|
||
)
|
||
|
||
|
||
def _existing_columns(conn, table):
|
||
"""已有列集合（automations.py 同款口径：SQLite 走 PRAGMA，
|
||
其它后端异常时回空集、后续 ALTER 逐条 try/except 兜底）。"""
|
||
try:
|
||
return {row[1] for row in conn.execute(
|
||
"PRAGMA table_info(%s)" % table).fetchall()}
|
||
except Exception:
|
||
return set()
|
||
|
||
|
||
def ensure_task_attempts_table(conn):
|
||
"""Create the ledger table if absent. Idempotent; does not
|
||
touch existing rows. The caller owns the commit (``record_attempt``
|
||
commits; a bare ensure inside a larger transaction does not).
|
||
|
||
演练注 FLAG（合入语义取舍点，主结论见演练报告）：
|
||
main 侧 init_db/worker 用 cloud.attempts.py 的旧 DDL 建同名
|
||
task_attempts 表（kind/interrupted_reason/checkpoint_round/
|
||
result_text/error_text，无 outcome 系列）；本模块的 CREATE
|
||
TABLE IF NOT EXISTS 在旧表已存在时是空操作，之后按台账
|
||
列写行即报 no column named outcome。本解法把表升级为两
|
||
组列的并集：旧表先建时补台账 4 列，台账表先建时补旧 5
|
||
列（新增列一律可空/带默认值，绝不对已有行加 NOT NULL，
|
||
否则旧行会挡住 ALTER）。两边读路径均按列名取数，互不
|
||
干扰；但这只是演练期保守解法——真合前须用户定：并集表、
|
||
台账改名、或统一 schema 三选一，不可悄悄定为并集表。
|
||
"""
|
||
conn.execute(SQLITE_DDL)
|
||
existing = _existing_columns(conn, "task_attempts")
|
||
for col, ddl in (
|
||
("outcome", "TEXT"),
|
||
("is_defect", "INTEGER DEFAULT 0"),
|
||
("defect_category", "TEXT"),
|
||
("corrects_attempt_no", "INTEGER"),
|
||
("kind", "TEXT"),
|
||
("interrupted_reason", "TEXT"),
|
||
("checkpoint_round", "INTEGER"),
|
||
("result_text", "TEXT"),
|
||
("error_text", "TEXT")):
|
||
if col not in existing:
|
||
try:
|
||
conn.execute(
|
||
"ALTER TABLE task_attempts ADD COLUMN %s %s"
|
||
% (col, ddl))
|
||
except Exception:
|
||
pass
|
||
|
||
|
||
def record_attempt(conn, task_id, outcome, is_defect=False,
|
||
defect_category=None, created_at=None):
|
||
"""Append one attempt row for ``task_id``. Returns ``attempt_no``.
|
||
|
||
Validates before writing: unknown outcome, a defect flag on an
|
||
interrupted row, and a defect category without a defect flag
|
||
all raise ``ValueError`` and write nothing.
|
||
"""
|
||
if not task_id:
|
||
raise ValueError("task_id is required")
|
||
if outcome not in OUTCOMES:
|
||
raise ValueError("outcome must be one of %s, got %r"
|
||
% (sorted(OUTCOMES), outcome))
|
||
if outcome == "interrupted" and is_defect:
|
||
raise ValueError("an interrupted attempt is never a defect")
|
||
if defect_category is not None and not is_defect:
|
||
raise ValueError("defect_category requires is_defect=True")
|
||
ensure_task_attempts_table(conn)
|
||
row = conn.execute(
|
||
"SELECT COALESCE(MAX(attempt_no), 0) FROM task_attempts "
|
||
"WHERE task_id = ?",
|
||
(task_id,),
|
||
).fetchone()
|
||
attempt_no = int(row[0]) + 1
|
||
conn.execute(
|
||
"INSERT INTO task_attempts(task_id, attempt_no, outcome, "
|
||
"is_defect, defect_category, created_at, "
|
||
"corrects_attempt_no) "
|
||
"VALUES(?, ?, ?, ?, ?, ?, NULL)",
|
||
(task_id, attempt_no, outcome, 1 if is_defect else 0,
|
||
defect_category,
|
||
time.time() if created_at is None else created_at),
|
||
)
|
||
conn.commit()
|
||
return attempt_no
|
||
|
||
|
||
def record_correction(conn, task_id, attempt_no, is_defect,
|
||
defect_category=None, created_at=None):
|
||
"""Append a CORRECTION row for an already-recorded attempt.
|
||
|
||
The correction does not rewrite history: the target row stays
|
||
exactly as recorded, and this new row (its own ``MAX + 1``
|
||
``attempt_no``) carries the corrected defect judgement in
|
||
``is_defect`` / ``defect_category`` plus
|
||
``corrects_attempt_no = attempt_no``. Aggregation applies
|
||
corrections in attempt order, last-wins, and never counts a
|
||
correction row as an attempt.
|
||
|
||
Only the defect judgement is correctable: the correction row
|
||
copies the target's outcome, so it cannot turn a success into
|
||
a failure or vice versa. Validates before writing: missing
|
||
``task_id``, a nonexistent target, a target that is itself a
|
||
correction row, a target whose outcome is ``interrupted``
|
||
(interrupted rows are never defects, so there is no defect
|
||
judgement to correct), and a defect category without a defect
|
||
flag all raise ``ValueError`` and write nothing. On a table
|
||
created from gap 3's DDL (no ``corrects_attempt_no`` column),
|
||
the INSERT raises the backend's OperationalError loudly --
|
||
run ``OLD_TABLE_ALTER`` first (see the module docstring);
|
||
a correction must never silently degrade into a plain row.
|
||
Returns the correction row's own ``attempt_no``.
|
||
"""
|
||
if not task_id:
|
||
raise ValueError("task_id is required")
|
||
if defect_category is not None and not is_defect:
|
||
raise ValueError("defect_category requires is_defect=True")
|
||
ensure_task_attempts_table(conn)
|
||
target = conn.execute(
|
||
"SELECT outcome, corrects_attempt_no FROM task_attempts "
|
||
"WHERE task_id = ? AND attempt_no = ?",
|
||
(task_id, attempt_no),
|
||
).fetchone()
|
||
if target is None:
|
||
raise ValueError(
|
||
"no such attempt to correct: %s #%s" % (task_id, attempt_no))
|
||
if target["corrects_attempt_no"] is not None:
|
||
raise ValueError(
|
||
"cannot correct a correction row: %s #%s"
|
||
% (task_id, attempt_no))
|
||
if target["outcome"] == "interrupted":
|
||
raise ValueError(
|
||
"an interrupted attempt has no defect judgement to "
|
||
"correct: %s #%s" % (task_id, attempt_no))
|
||
row = conn.execute(
|
||
"SELECT COALESCE(MAX(attempt_no), 0) FROM task_attempts "
|
||
"WHERE task_id = ?",
|
||
(task_id,),
|
||
).fetchone()
|
||
correction_no = int(row[0]) + 1
|
||
conn.execute(
|
||
"INSERT INTO task_attempts(task_id, attempt_no, outcome, "
|
||
"is_defect, defect_category, created_at, "
|
||
"corrects_attempt_no) "
|
||
"VALUES(?, ?, ?, ?, ?, ?, ?)",
|
||
(task_id, correction_no, target["outcome"],
|
||
1 if is_defect else 0, defect_category,
|
||
time.time() if created_at is None else created_at,
|
||
attempt_no),
|
||
)
|
||
conn.commit()
|
||
return correction_no
|
||
|
||
|
||
def read_attempt_rows(conn):
|
||
"""All ledger rows in attempt order. Raises the backend's
|
||
OperationalError when the table does not exist yet; the
|
||
endpoint caller treats that as "no ledger", not as a failure.
|
||
|
||
``SELECT *`` is deliberate: a table created from gap 3's DDL
|
||
has no ``corrects_attempt_no`` column, and naming that column
|
||
here would turn those readable rows into an endpoint error.
|
||
With ``*`` the rows still read, and ``aggregate_attempts``
|
||
treats a missing ``corrects_attempt_no`` as a plain attempt
|
||
row (see ``_field_optional``).
|
||
"""
|
||
return conn.execute(
|
||
"SELECT * FROM task_attempts "
|
||
"ORDER BY task_id, attempt_no").fetchall()
|
||
|
||
|
||
def _field(row, name):
|
||
# sqlite3.Row, dict, and the Postgres adapter's rows all support
|
||
# key access; a plain tuple row is not accepted (callers pass
|
||
# rows, not raw tuples, so a wrong shape fails loudly here).
|
||
return row[name]
|
||
|
||
|
||
def _field_optional(row, name):
|
||
"""Like ``_field`` but missing keys mean "absent", not an
|
||
error. Used ONLY for ``corrects_attempt_no``: rows built by
|
||
gap-3-era readers (plain dicts, or tables without the column)
|
||
carry no such key, and they are plain attempt rows by
|
||
definition -- failing loudly there would break every gap-3
|
||
aggregate test for no benefit."""
|
||
try:
|
||
return row[name]
|
||
except (KeyError, IndexError, TypeError):
|
||
return None
|
||
|
||
|
||
def aggregate_attempts(rows):
|
||
"""Pure aggregation: ledger rows -> ``{task_id: record}`` where
|
||
``record`` is the ``attempts``/``defects``/``defect_category``
|
||
dict that ``compute_metrics`` consumes.
|
||
|
||
Interrupted rows are excluded from both counts, and a task
|
||
with only interrupted rows is omitted entirely (no counted
|
||
attempt => its rates stay unknown, never 0). Correction rows
|
||
(``corrects_attempt_no`` set) are excluded from the attempt
|
||
count and instead OVERWRITE the defect judgement of their
|
||
target attempt, in attempt order, last-wins: a task's
|
||
``defects`` is the number of its counted attempts whose
|
||
judgement AFTER corrections is a defect, and its
|
||
``defect_category`` is the category of its FIRST defect
|
||
attempt in attempt order AFTER corrections.
|
||
"""
|
||
per_task = {}
|
||
ordered = sorted(
|
||
rows,
|
||
key=lambda r: (_field(r, "task_id"), _field(r, "attempt_no")))
|
||
# Per task: attempt_no -> [is_defect, defect_category], in
|
||
# attempt order (insertion order = attempt_no order, since the
|
||
# input was sorted). Corrections mutate the target entry; they
|
||
# never create an entry of their own.
|
||
judged = {}
|
||
for row in ordered:
|
||
task_id = _field(row, "task_id")
|
||
corrects = _field_optional(row, "corrects_attempt_no")
|
||
if corrects is not None:
|
||
entries = judged.get(task_id)
|
||
if entries is not None and corrects in entries:
|
||
entries[corrects] = [
|
||
bool(_field(row, "is_defect")),
|
||
_field(row, "defect_category")]
|
||
continue
|
||
if _field(row, "outcome") == "interrupted":
|
||
continue
|
||
entries = judged.setdefault(task_id, {})
|
||
entries[_field(row, "attempt_no")] = [
|
||
bool(_field(row, "is_defect")),
|
||
_field(row, "defect_category")]
|
||
for task_id, entries in judged.items():
|
||
attempts = len(entries)
|
||
if attempts == 0:
|
||
continue
|
||
defects = 0
|
||
defect_category = None
|
||
for _no in sorted(entries):
|
||
is_defect, category = entries[_no]
|
||
if is_defect:
|
||
defects += 1
|
||
if defect_category is None:
|
||
defect_category = category
|
||
per_task[task_id] = {
|
||
"attempts": attempts, "defects": defects,
|
||
"defect_category": defect_category}
|
||
return per_task
|
||
|
||
|
||
def apply_to_records(records, aggregates):
|
||
"""Return a new record list with ledger values merged in.
|
||
|
||
Records for tasks with no aggregate entry keep whatever
|
||
``attempts``/``defects`` they already carried (the endpoint
|
||
passes ``None`` there, i.e. old tasks stay unknown). Input
|
||
records are not modified.
|
||
"""
|
||
merged = []
|
||
for record in records:
|
||
out = dict(record)
|
||
agg = aggregates.get(record.get("id"))
|
||
if agg is not None:
|
||
out["attempts"] = agg["attempts"]
|
||
out["defects"] = agg["defects"]
|
||
out["defect_category"] = agg["defect_category"]
|
||
merged.append(out)
|
||
return merged
|