audit: lifecycle hook writes the attempt ledger (gap 3 follow-up)

- cloud/attempt_hook.py (new): canonical policy -- one counted
  dispatch = one row, is_defect always False from the worker;
  human/first rounds and human breakpoint resumes count;
  platform auto-resume reruns, coordinator fan-out, and
  pre-execution gate failures do not; transient retries inside
  one dispatch are not new attempts. Platform origin survives
  via the task_attempt_origin_markers table (same-transaction
  write on requeue, queued_at generation guard on consume).
  Defects are explicit-recorder only; corrections are
  append-only (record_correction, last-wins).
- cloud/worker.py: Worker(attempt_ledger_hook=False) + CLI
  --attempt-ledger-hook (default OFF = gap-3 behaviour exactly);
  marker write in auto_resume_interrupted, marker consume +
  classify in run_one, one ledger write at each terminal point
  (cancel branch, normal terminal block); all hook writes
  best-effort, never fail the task or block a requeue.
- cloud/attempt_ledger.py: corrects_attempt_no column (new DDL
  supersedes gap 3's pre-merge DDL) + record_correction +
  correction-aware aggregation; read path uses SELECT * so
  gap-3-era tables still read; cost code untouched.
- cloud/tests/test_attempt_hook.py: 32 tests (policy, markers,
  corrections, worker integration, mixed endpoint).
- docs/arch-conformance-metrics.md: policy, switch, PG manual
  steps (new PG_DDL + PG_MARKER_DDL), remaining user decisions.
This commit is contained in:
host committed 2026-11-03 02:49:26 +00:00
1 parent 436a92ba1d
commit dc556b0c19
5 files changed
+1522 -38

No files matched your search

+358
View File
@@ -0,0 +1,358 @@
"""Lifecycle hook policy for the task-attempt ledger (audit gap 3
follow-up: "which dispatch counts as an attempt, and who judges
``is_defect``").
This module is the CANONICAL policy text. The same policy is
restated (shorter) in ``docs/arch-conformance-metrics.md`` and
pinned by ``cloud/tests/test_attempt_hook.py``; if the three ever
disagree, this module wins and the other two must be fixed.
The problem it solves: audit gap 3 built the ledger but
deliberately left the writer unconnected, because a naive writer
-- one row per worker execution -- misclassifies exactly twice:
it counts a platform auto-resume as an attempt, and it counts a
transient retry as a defect. This module fixes the writer's
trigger, not the ledger's math: ``aggregate_attempts`` is
untouched by the hook (its only change in this track is the
correction-row rule in ``attempt_ledger.py``), and the cost
dual-column code (``cloud.cost``) is untouched entirely.
POLICY (each rule is enforced in code below, not just prose):
1. One counted dispatch = one row, at most. A dispatch that the
hook counts writes EXACTLY ONE ledger row, when its outcome is
known (terminal ``completed`` / ``failed`` / ``interrupted``,
or the goal-conversion case in ``worker.run_one``), with
``is_defect=False`` always (rule 6). A dispatch that the hook
does not count writes NO row at all -- not even an
``interrupted`` row -- so an uncounted dispatch can never
create the ledger table as a side effect.
2. What counts as one attempt -- a logical agent round, not a
worker execution:
- First dispatch of a task: counts (``first_or_human_retry``).
It is the task's first agent round.
- Human ``/retry`` of a failed/interrupted/merge-failed task:
counts (same kind). A retry is a human-initiated new round:
new model work, new result, new cost. The server endpoint
leaves no marker behind, so the worker cannot distinguish a
retry from a first dispatch -- and does not need to, because
both count.
- Human ``/continue`` with a new message: counts
(``human_continue``). The message starts a new round on the
same session thread. Non-goal tasks and goal tasks opening a
NEW goal loop (``_resume_kind == "new-goal"``) are both here.
- Human breakpoint resume (``/goal-resume``, or ``/continue``
on an old-verify goal task, i.e. ``_resume_kind ==
"breakpoint"``): counts (``breakpoint_resume``) -- but as
the SAME logical round's row, not as an extra attempt. The
fragment that was interrupted wrote only an excluded
``interrupted`` row (gap 3 rule), so the resume's terminal
row is the round's single counted row. A resume that ends
interrupted again writes only another excluded row: the
round is still pending, and still contributes zero counted
attempts. Rationale: one goal loop = one attempt, however
many executions the loop took; the interruption history
stays visible in the ``interrupted`` rows and in
``auto_resume_count`` / audit / task_steps, not in the
attempt count. EXCEPTION (ordering, rule 4): when the
resume was itself platform-triggered, the platform marker
wins and the dispatch does NOT count -- a platform goal
auto-resume and a human goal-resume are the same loop
continuation, but only the human one is attributed to the
agent. That asymmetry is deliberate and uniform with the
platform rule below; flipping it is the listed user
decision, not a silent default.
- Coordinator parent dispatch: does NOT count
(``coordinator_orchestration``). The parent runs no model
round (``cost_record_id`` is None) and its terminal status
is an aggregate of its children; each child records its own
row through its own dispatch. Counting the parent would
double-count orchestration as an agent attempt.
- Platform auto-resume dispatch (worker
``auto_resume_interrupted`` requeue, goal or non-goal):
does NOT count (``platform_auto_resume``). This is the
default the gap-3 track demanded, and it is UNIFORM: every
platform requeue is excluded, whether it continues a goal
loop or fresh-runs a non-goal task. Rationale: the platform
killed the previous execution and re-ran it without any
human or coordinator decision; attributing that execution
to the agent would punish platform instability in an
agent-quality rate. TRADE-OFF, STATED NOT HIDDEN: a task
whose terminal outcome came ONLY from platform dispatches
stays unknown in the ledger (no counted row), so it is
excluded from rate denominators -- the rates describe
tasks whose terminal outcome came from a human/first
round. The platform reruns themselves are NOT lost: they
are recorded in ``tasks.auto_resume_count``, the
``worker_auto_resumed`` audit events, and the
``task_steps`` ``auto_resume`` rows. The rejected
alternative was a separately marked ledger row that is
excluded from denominators: same denominator outcome, but
a second row kind, a new column, and aggregate exclusion
logic, for co-location only. If a future policy wants
platform completions counted as loop completions, the flip
is one line in ``classify_dispatch`` (platform + breakpoint
-> counted) plus tests -- it is listed as a user decision
in the track summary, not silently decided here.
- Pre-execution gate failures: do NOT count. A dispatch that
never reached the model -- missing key, repository
resolution failure, quota gate, goal-rounds cap, gateway
preparation failure -- is an admission rejection, not an
attempt. It writes no row (and no cost record either, which
is the existing behaviour); it stays visible in audit
(``worker_missing_key`` / ``worker_repo_failed`` /
``worker_quota_stop`` / ``worker_goal_cap_rejected`` /
``worker_gateway_failed``).
- Dispatches that never ran (budget gate, no ready task,
claim lost to another worker): do NOT count and are not
even classified -- ``run_one`` returns before the hook
runs, and no marker is consumed (see rule 4).
3. Transient retries inside one dispatch are NOT new attempts.
The worker's ``max_retries`` loop, its one real fallback
re-run, its goal "already exists" breakpoint fallback, and
its follow-up drain all run inside a single ``run_one``
dispatch, share one cost aggregate, and produce ONE ledger
row: the dispatch's terminal outcome. An inner failure that
is later retried successfully leaves no failure row behind;
an inner failure that exhausts retries leaves one failure
row, not N. This is enforced by placement (the hook is
called once per dispatch, at ``run_one``'s terminal points)
rather than by counting executions.
4. Platform identification -- the marker mechanism. The worker
that requeues a task is not necessarily the worker that later
claims it, so the origin must survive in the database, not in
memory. ``auto_resume_interrupted`` writes one marker row
(``task_attempt_origin_markers``) in the SAME transaction as
the requeue, with ``queued_at`` equal to the ``updated_at``
the requeue writes. ``run_one`` captures the queued task's
``updated_at`` BEFORE claiming (claim overwrites it), and
after a successful claim consumes the marker ONLY if
``queued_at`` still matches. Any intervening human action
(cancel, retry, continue) rewrites ``updated_at``, so a stale
marker can never misclassify a later human round: on
mismatch the marker is deleted as stale and the dispatch is
classified by its own fields. A claim that loses the race
consumes nothing, so the winning worker still sees its
marker. Marker writes and reads happen ONLY when the hook is
enabled; with the hook off, ``auto_resume_interrupted``
writes no marker and ``run_one`` reads none, so off behaviour
is byte-identical to gap 3 (no new table, no new row).
Marker-table creation is a write-path-only step, like the
ledger table itself; the read path (``GET /metrics-report``)
never touches it.
5. Switch. ``Worker(attempt_ledger_hook=False)`` (CLI
``--attempt-ledger-hook``), default OFF. OFF means: no ledger
row, no marker row, no marker-table creation, no ledger-table
creation from the worker -- exactly gap 3's "explicit calls
only" behaviour, including old tasks still returning ``None``
(unknown, not 0) and the read endpoint never creating a
table. ON means rules 1-4 apply. Enabling the hook
mid-flight has one documented edge: a task requeued by
auto-resume WHILE the hook was off carries no marker, so its
next dispatch counts as a human/first round (one extra
counted row for that in-flight task, once). That is the
fail-safe direction (a human round is the common case), and
it self-heals: every later requeue writes a marker.
6. ``is_defect`` -- who judges, and how a wrong judgement is
fixed. The worker hook NEVER judges a defect: it always
records ``is_defect=False``. A failed dispatch is not a
defect (transient failures, quota stops, and cancelled rounds
all fail without being defects), and the worker cannot tell
a transient failure from a defect round at runtime -- so it
must not guess. The ONLY defect judgements in the ledger are
explicit ones, made at record time by an explicit recorder:
a human maintainer (or a coordinator review) calling
``record_attempt`` / ``record_correction`` directly with
``is_defect=True`` and a category. The bar for that authority
is maintainer-level (``approve_merge``, the same bar as
``/retry``: both spend model money and move decision-grade
rates). A defect judgement made in error is corrected ONLY by
appending a correction row (``record_correction``): the old
row is never updated or deleted, corrections apply in
attempt order with last-wins, and a correction can only
change the defect judgement, never the outcome. There is NO
HTTP endpoint for marking or correcting defects in this
track -- the function is the interface, and an endpoint (with
the ``approve_merge`` permission check) is follow-up work,
listed as unconnected in the track summary. Until it exists,
defect marks come only from explicit, reviewed calls, never
from automation.
7. Failure discipline. Every hook write is best-effort: a ledger
or marker failure is logged (``worker_attempt_ledger_failed``
/ ``worker_attempt_marker_failed``) and NEVER fails the task,
blocks a requeue, or changes a task's status. The one
exception is validation inside ``record_attempt`` /
``record_correction`` (bad outcome, defect on an interrupted
row, correcting a nonexistent attempt): those raise
``ValueError`` BEFORE writing, by design, so ambiguous data
is never stored -- but the worker hook only ever passes
validated values (a terminal outcome, ``is_defect=False``),
so it cannot hit that path in normal operation.
8. Postgres. The marker table follows the ledger table's manual
discipline: it is NOT created by ``init_postgres_schema``,
and ``PG_MARKER_DDL`` below is the exact statement to run by
hand on production Postgres (and verify) before enabling the
hook against a Postgres-backed control plane. Until that
step runs, marker writes fail (best-effort, rule 7) and
platform dispatches degrade to counted -- the same
fail-safe direction as rule 5's mid-flight edge, and the
reason the PG manual step is a precondition for enabling
the hook in production, not an afterthought. Real Postgres
verification was NOT done on this track (no local Postgres):
PG evidence here is adapter-level only (``?`` placeholders
translate to ``%s`` through ``cloud.db``, DDL uses only
Postgres-valid types, no autoincrement/identity column),
exactly like gap 3's PG evidence, and labelled as such.
This module imports ``cloud.attempt_ledger`` only, opens no
database connection itself (callers pass a connection in), and
contains no ``sqlite3.connect`` / ``psycopg*`` import, so the
``arch_conformance`` layer rules are untouched.
"""
from cloud.attempt_ledger import record_attempt
MARKER_TABLE = "task_attempt_origin_markers"
# One DDL serves both backends: TEXT / DOUBLE PRECISION / composite
# primary key, no AUTOINCREMENT or identity column -- the same
# discipline as the ledger DDL. ``queued_at`` is the ``updated_at``
# the requeue wrote; it is the generation guard, not a timestamp
# for display.
MARKER_DDL = (
"CREATE TABLE IF NOT EXISTS task_attempt_origin_markers("
"task_id TEXT NOT NULL PRIMARY KEY, "
"origin TEXT NOT NULL, "
"queued_at DOUBLE PRECISION NOT NULL)"
)
# Production Postgres manual step (see rule 8). Intentionally NOT
# part of POSTGRES_SCHEMA, same reason as the ledger PG_DDL.
PG_MARKER_DDL = MARKER_DDL
PLATFORM_ORIGIN = "platform_auto_resume"
# Decision kinds returned by classify_dispatch. ``counted`` is the
# only field the worker acts on; ``kind`` is the audit label.
KINDS = frozenset({
"coordinator_orchestration",
"platform_auto_resume",
"breakpoint_resume",
"human_continue",
"first_or_human_retry",
})
def classify_dispatch(task, platform_marker):
"""Pure policy: task fields + marker -> decision dict.
``task`` is the dispatch's task dict (post-claim, post
breakpoint resolution: ``_goal_resume_breakpoint`` is True
only when a breakpoint resume actually applies, False when
resolution fell back to a fresh goal run). ``platform_marker``
is True when this dispatch consumed a platform auto-resume
marker for the queued generation (rule 4), else False.
Returns ``{"counted": bool, "kind": str}``. Order matters
and is part of the policy: coordinator fan-out never counts
even with a marker; a platform marker beats every other
signal (uniform platform exclusion, rule 2); a breakpoint
resume beats a continue message (a stale resume_message on
a breakpoint task is not a new round); anything else with a
resume message is a human continue; the rest is a first
dispatch or a human retry, which count identically.
"""
task = task or {}
if task.get("coordinator_parts"):
return {"counted": False,
"kind": "coordinator_orchestration"}
if platform_marker:
return {"counted": False, "kind": "platform_auto_resume"}
if task.get("_goal_resume_breakpoint") is True or (
task.get("_resume_kind") == "breakpoint"):
return {"counted": True, "kind": "breakpoint_resume"}
if task.get("resume_message"):
return {"counted": True, "kind": "human_continue"}
return {"counted": True, "kind": "first_or_human_retry"}
def ensure_marker_table(conn):
"""Create the marker table if absent. Idempotent; the caller
owns the commit (``mark_platform_rerun`` commits via the
caller's requeue transaction in the worker; a bare ensure
inside a larger transaction does not commit)."""
conn.execute(MARKER_DDL)
def mark_platform_rerun(conn, task_id, queued_at):
"""Record that ``task_id`` was requeued by platform auto-resume
at ``queued_at`` (the requeue's ``updated_at``). Overwrites
any marker for the task: a requeue supersedes any older
generation. Callers must call this inside the requeue
transaction (the worker does: same connection, same commit
as the status change), so a marker can never exist without
its queued state, or vice versa. Best-effort at the call
site: on failure the caller logs and proceeds with the
requeue (rule 7) -- a missing marker degrades to counted,
never to a blocked requeue."""
if not task_id:
raise ValueError("task_id is required")
ensure_marker_table(conn)
conn.execute(
"DELETE FROM task_attempt_origin_markers WHERE task_id = ?",
(task_id,),
)
conn.execute(
"INSERT INTO task_attempt_origin_markers(task_id, origin, "
"queued_at) VALUES(?, ?, ?)",
(task_id, PLATFORM_ORIGIN, queued_at),
)
def consume_platform_marker(conn, task_id, queued_at):
"""Consume the platform marker for ``task_id`` iff its
``queued_at`` still matches this dispatch's queued
``updated_at``. Returns True on a match (this dispatch IS
the platform rerun). On mismatch the marker belongs to an
older generation (a human action rewrote ``updated_at``
since) and is deleted as stale; returns False. When
``queued_at`` is None the generation cannot be verified, so
nothing is consumed or deleted and False is returned (the
dispatch then counts -- the rule-5 fail-safe direction).
A missing marker table raises the backend's OperationalError
(loud, not silent): the worker call site treats that as
"no marker" and logs it, it never fails the dispatch."""
if not task_id or queued_at is None:
return False
cur = conn.execute(
"DELETE FROM task_attempt_origin_markers "
"WHERE task_id = ? AND queued_at = ?",
(task_id, queued_at),
)
if cur.rowcount == 1:
return True
conn.execute(
"DELETE FROM task_attempt_origin_markers WHERE task_id = ?",
(task_id,),
)
return False
def record_dispatch_result(conn, task_id, outcome):
"""Write the ledger row for one COUNTED dispatch. The worker
calls this only when ``classify_dispatch`` said counted, and
only with a terminal outcome (``success`` / ``failure`` /
``interrupted``). ``is_defect`` is always False here (rule
6): the worker never judges a defect. Returns the new
``attempt_no``. The counted filter lives in the caller
(``classify_dispatch``); this function takes the outcome
only, so a miscounted call site cannot smuggle a defect
judgement in through here either."""
return record_attempt(conn, task_id, outcome, is_defect=False)
+192 -25
View File
@@ -41,7 +41,22 @@ Semantics (pinned by tests, not left to each caller):
``compute_metrics`` carries per task) is the category of its
FIRST defect row in attempt order. Later defect rows still count
toward ``defects``; only the category label is first-wins, so it
is deterministic.
is deterministic. Corrections (below) re-evaluate this label:
the first defect row AFTER corrections is applied wins.
- A defect judgement can be CORRECTED, never rewritten:
``record_correction`` appends a new row whose
``corrects_attempt_no`` points at the attempt it corrects; the
old row stays untouched, and aggregation applies corrections in
attempt order with last-wins (so a wrongly marked defect can be
unmarked, and a defect missed at record time can be added after
review). A correction row is never itself counted as an
attempt, cannot change an attempt's outcome (only its defect
judgement), and cannot target an interrupted row, another
correction row, or a nonexistent attempt -- all raise
``ValueError`` and write nothing. Who may correct is a policy
question answered in ``cloud/attempt_hook.py`` (explicit
recorder only, maintainer bar); the worker hook never writes
corrections and never marks a defect.
- Old tasks have no ledger rows, and that is NOT backfilled: the
aggregate simply has no entry for them, the endpoint leaves their
``attempts``/``defects`` as ``None``, and ``compute_metrics``
@@ -75,16 +90,32 @@ Persistence notes:
used for missing ``cost_records``.
- The read path (``GET /metrics-report``) never creates the table:
a read-only endpoint must not mutate schema. Table creation
happens only on the write path, inside ``record_attempt``.
- No lifecycle hook writes to this ledger yet. Hooking a writer
into the worker's retry/resume loops would risk exactly the
misclassification this module refuses to make (platform
auto-resume counted as an attempt, transient retry counted as a
defect). The writer -- which lifecycle event counts as an
attempt, and who judges ``is_defect`` -- is a separate decision,
listed as an open item; until it lands, the ledger fills only
through explicit ``record_attempt`` calls, and production rates
stay honestly unknown rather than auto-filled.
happens only on the write path, inside ``record_attempt`` /
``record_correction``.
- Lifecycle hook (this track, see ``cloud/attempt_hook.py``):
the worker writes exactly one row per COUNTED dispatch (see
that module's policy for which dispatches count: human/first
rounds and human breakpoint resumes count; platform auto-resume
reruns, coordinator fan-out, and pre-execution gate failures do
not). The hook is OFF by default (``attempt_ledger_hook=False``):
with it off, this module's behaviour is exactly audit gap 3 --
the ledger fills only through explicit ``record_attempt`` /
``record_correction`` calls, and production rates stay honestly
unknown until the hook is deliberately enabled. The worker hook
always records ``is_defect=False``: it never judges a defect,
so enabling it can never turn a transient retry into a defect.
- Schema evolution note (pre-production, this track):
``corrects_attempt_no`` was added to the DDL after audit gap 3
but BEFORE either DDL reached production (gap 3 is still an
OPEN, unmerged PR), so no production table needs migration:
fresh tables get the column from the DDL below. A database
created from gap 3's DDL alone (temporary/test databases only)
gains the column with one statement --
``ALTER TABLE task_attempts ADD COLUMN corrects_attempt_no
INTEGER`` -- which is also the documented step if gap 3's DDL
was ever run on production Postgres before this track lands.
The read path uses ``SELECT *`` deliberately so a table without
the column still reads (its rows aggregate as plain attempts).
This module imports no ``cloud.*`` module and opens no database
connection itself: callers pass a connection in, so the layer rules
@@ -102,6 +133,11 @@ OUTCOMES = frozenset({"success", "failure", "interrupted"})
# One DDL serves both backends: every type and the composite key are
# valid in SQLite and in Postgres, and there is no AUTOINCREMENT or
# identity column for either backend's sequence handling to chase.
# ``corrects_attempt_no`` (NULL = a real attempt row; non-NULL = a
# correction row for that attempt_no, never counted as an attempt)
# was added by the lifecycle-hook track; it SUPERSEDES gap 3's DDL
# for any production manual step (gap 3 never merged, so no
# production table exists yet -- see the module docstring).
SQLITE_DDL = (
"CREATE TABLE IF NOT EXISTS task_attempts("
"task_id TEXT NOT NULL, "
@@ -110,6 +146,7 @@ SQLITE_DDL = (
"is_defect INTEGER NOT NULL DEFAULT 0, "
"defect_category TEXT, "
"created_at DOUBLE PRECISION, "
"corrects_attempt_no INTEGER, "
"PRIMARY KEY (task_id, attempt_no))"
)
@@ -117,8 +154,19 @@ SQLITE_DDL = (
# table exists before expecting real rates from a Postgres-backed
# endpoint. It is intentionally NOT part of POSTGRES_SCHEMA (see
# the module docstring: migrations coverage must not widen silently).
# This is the NEW DDL (with ``corrects_attempt_no``): if gap 3's
# older DDL was ever run first, follow it with
# ``ALTER TABLE task_attempts ADD COLUMN corrects_attempt_no
# INTEGER`` and verify before expecting corrections to work.
PG_DDL = SQLITE_DDL
# One-line upgrade for a table created from gap 3's DDL (no
# ``corrects_attempt_no`` column yet). Valid in both backends.
OLD_TABLE_ALTER = (
"ALTER TABLE task_attempts "
"ADD COLUMN corrects_attempt_no INTEGER"
)
def ensure_task_attempts_table(conn):
"""Create the ledger table if absent. Idempotent; does not
@@ -153,8 +201,9 @@ def record_attempt(conn, task_id, outcome, is_defect=False,
attempt_no = int(row[0]) + 1
conn.execute(
"INSERT INTO task_attempts(task_id, attempt_no, outcome, "
"is_defect, defect_category, created_at) "
"VALUES(?, ?, ?, ?, ?, ?)",
"is_defect, defect_category, created_at, "
"corrects_attempt_no) "
"VALUES(?, ?, ?, ?, ?, ?, NULL)",
(task_id, attempt_no, outcome, 1 if is_defect else 0,
defect_category,
time.time() if created_at is None else created_at),
@@ -163,13 +212,87 @@ def record_attempt(conn, task_id, outcome, is_defect=False,
return attempt_no
def record_correction(conn, task_id, attempt_no, is_defect,
defect_category=None, created_at=None):
"""Append a CORRECTION row for an already-recorded attempt.
The correction does not rewrite history: the target row stays
exactly as recorded, and this new row (its own ``MAX + 1``
``attempt_no``) carries the corrected defect judgement in
``is_defect`` / ``defect_category`` plus
``corrects_attempt_no = attempt_no``. Aggregation applies
corrections in attempt order, last-wins, and never counts a
correction row as an attempt.
Only the defect judgement is correctable: the correction row
copies the target's outcome, so it cannot turn a success into
a failure or vice versa. Validates before writing: missing
``task_id``, a nonexistent target, a target that is itself a
correction row, a target whose outcome is ``interrupted``
(interrupted rows are never defects, so there is no defect
judgement to correct), and a defect category without a defect
flag all raise ``ValueError`` and write nothing. On a table
created from gap 3's DDL (no ``corrects_attempt_no`` column),
the INSERT raises the backend's OperationalError loudly --
run ``OLD_TABLE_ALTER`` first (see the module docstring);
a correction must never silently degrade into a plain row.
Returns the correction row's own ``attempt_no``.
"""
if not task_id:
raise ValueError("task_id is required")
if defect_category is not None and not is_defect:
raise ValueError("defect_category requires is_defect=True")
ensure_task_attempts_table(conn)
target = conn.execute(
"SELECT outcome, corrects_attempt_no FROM task_attempts "
"WHERE task_id = ? AND attempt_no = ?",
(task_id, attempt_no),
).fetchone()
if target is None:
raise ValueError(
"no such attempt to correct: %s #%s" % (task_id, attempt_no))
if target["corrects_attempt_no"] is not None:
raise ValueError(
"cannot correct a correction row: %s #%s"
% (task_id, attempt_no))
if target["outcome"] == "interrupted":
raise ValueError(
"an interrupted attempt has no defect judgement to "
"correct: %s #%s" % (task_id, attempt_no))
row = conn.execute(
"SELECT COALESCE(MAX(attempt_no), 0) FROM task_attempts "
"WHERE task_id = ?",
(task_id,),
).fetchone()
correction_no = int(row[0]) + 1
conn.execute(
"INSERT INTO task_attempts(task_id, attempt_no, outcome, "
"is_defect, defect_category, created_at, "
"corrects_attempt_no) "
"VALUES(?, ?, ?, ?, ?, ?, ?)",
(task_id, correction_no, target["outcome"],
1 if is_defect else 0, defect_category,
time.time() if created_at is None else created_at,
attempt_no),
)
conn.commit()
return correction_no
def read_attempt_rows(conn):
"""All ledger rows in attempt order. Raises the backend's
OperationalError when the table does not exist yet; the
endpoint caller treats that as "no ledger", not as a failure."""
endpoint caller treats that as "no ledger", not as a failure.
``SELECT *`` is deliberate: a table created from gap 3's DDL
has no ``corrects_attempt_no`` column, and naming that column
here would turn those readable rows into an endpoint error.
With ``*`` the rows still read, and ``aggregate_attempts``
treats a missing ``corrects_attempt_no`` as a plain attempt
row (see ``_field_optional``).
"""
return conn.execute(
"SELECT task_id, attempt_no, outcome, is_defect, "
"defect_category FROM task_attempts "
"SELECT * FROM task_attempts "
"ORDER BY task_id, attempt_no").fetchall()
@@ -180,6 +303,19 @@ def _field(row, name):
return row[name]
def _field_optional(row, name):
"""Like ``_field`` but missing keys mean "absent", not an
error. Used ONLY for ``corrects_attempt_no``: rows built by
gap-3-era readers (plain dicts, or tables without the column)
carry no such key, and they are plain attempt rows by
definition -- failing loudly there would break every gap-3
aggregate test for no benefit."""
try:
return row[name]
except (KeyError, IndexError, TypeError):
return None
def aggregate_attempts(rows):
"""Pure aggregation: ledger rows -> ``{task_id: record}`` where
``record`` is the ``attempts``/``defects``/``defect_category``
@@ -187,24 +323,55 @@ def aggregate_attempts(rows):
Interrupted rows are excluded from both counts, and a task
with only interrupted rows is omitted entirely (no counted
attempt => its rates stay unknown, never 0).
attempt => its rates stay unknown, never 0). Correction rows
(``corrects_attempt_no`` set) are excluded from the attempt
count and instead OVERWRITE the defect judgement of their
target attempt, in attempt order, last-wins: a task's
``defects`` is the number of its counted attempts whose
judgement AFTER corrections is a defect, and its
``defect_category`` is the category of its FIRST defect
attempt in attempt order AFTER corrections.
"""
per_task = {}
ordered = sorted(
rows,
key=lambda r: (_field(r, "task_id"), _field(r, "attempt_no")))
# Per task: attempt_no -> [is_defect, defect_category], in
# attempt order (insertion order = attempt_no order, since the
# input was sorted). Corrections mutate the target entry; they
# never create an entry of their own.
judged = {}
for row in ordered:
task_id = _field(row, "task_id")
corrects = _field_optional(row, "corrects_attempt_no")
if corrects is not None:
entries = judged.get(task_id)
if entries is not None and corrects in entries:
entries[corrects] = [
bool(_field(row, "is_defect")),
_field(row, "defect_category")]
continue
if _field(row, "outcome") == "interrupted":
continue
entry = per_task.setdefault(
task_id, {"attempts": 0, "defects": 0,
"defect_category": None})
entry["attempts"] += 1
if _field(row, "is_defect"):
entry["defects"] += 1
if entry["defect_category"] is None:
entry["defect_category"] = _field(row, "defect_category")
entries = judged.setdefault(task_id, {})
entries[_field(row, "attempt_no")] = [
bool(_field(row, "is_defect")),
_field(row, "defect_category")]
for task_id, entries in judged.items():
attempts = len(entries)
if attempts == 0:
continue
defects = 0
defect_category = None
for _no in sorted(entries):
is_defect, category = entries[_no]
if is_defect:
defects += 1
if defect_category is None:
defect_category = category
per_task[task_id] = {
"attempts": attempts, "defects": defects,
"defect_category": defect_category}
return per_task
+729
View File
@@ -0,0 +1,729 @@
"""Tests for the task-attempt lifecycle hook (audit gap 3
follow-up).
Only the standard library; temporary SQLite databases, a fake
CLI runner, and a temporary HTTP server for the endpoint test.
No real model, no network, no secrets, no real Postgres (the
Postgres claims are adapter-level only, labelled as such).
Pinned policy (see ``cloud/attempt_hook.py`` -- the module is
canonical, these tests pin its observable behaviour):
- classify: coordinator never counts; a platform marker beats
every other signal; a breakpoint resume counts as the same
logical round's row (not an extra attempt); a continue message
counts as a new round; first dispatch / human retry count.
- markers: consume only matches the queued generation; a stale
marker is deleted, never misapplied; an unverifiable
generation consumes nothing.
- corrections: append-only, last-wins, never counted as an
attempt, cannot change an outcome, cannot target interrupted
rows / correction rows / nonexistent attempts.
- worker: hook OFF writes nothing and creates no table (gap-3
behaviour); hook ON writes exactly one row per counted
dispatch (a transient retry inside one dispatch is still one
row); a platform rerun writes no row; a gate failure writes
no row; a ledger write failure never fails the task.
- endpoint: with ledger tasks + old tasks + a platform-completed
task + an interrupted task side by side, terminal denominators
contain only known tasks, interrupted status is non-terminal,
interrupted rows count for nothing, and unknown rates are
None + *_known False, never 0; the cost dual-column discipline
is unchanged (priced partial shown, total None when incomplete,
unpriced never folded to 0).
"""
import http.client
import json
import os
import sqlite3
import sys
import tempfile
import threading
import time
import unittest
from pathlib import Path
from types import SimpleNamespace
try:
from cloud.attempt_hook import (
MARKER_DDL,
PG_MARKER_DDL,
classify_dispatch,
consume_platform_marker,
ensure_marker_table,
mark_platform_rerun,
)
from cloud.attempt_ledger import (
OLD_TABLE_ALTER,
PG_DDL,
aggregate_attempts,
apply_to_records,
read_attempt_rows,
record_attempt,
record_correction,
)
from cloud.db import _convert_placeholders, _translate_sql
from cloud.metrics import compute_metrics
except ImportError:
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
from cloud.attempt_hook import (
MARKER_DDL,
PG_MARKER_DDL,
classify_dispatch,
consume_platform_marker,
ensure_marker_table,
mark_platform_rerun,
)
from cloud.attempt_ledger import (
OLD_TABLE_ALTER,
PG_DDL,
aggregate_attempts,
apply_to_records,
read_attempt_rows,
record_attempt,
record_correction,
)
from cloud.db import _convert_placeholders, _translate_sql
from cloud.metrics import compute_metrics
def _sqlite_conn(path):
from cloud.db import connect as db_connect
return db_connect(path)
def _insert_task(db_path, task_id, prompt="do work", status="queued",
project_id="p1", **extra):
now = time.time()
cols = ["id", "project_id", "prompt", "status",
"idempotency_key", "depends_on", "priority", "created_by",
"created_at", "updated_at", "last_heartbeat"]
vals = [task_id, project_id, prompt, status, None, "[]", 0,
"tester", now, now, None]
for key, value in extra.items():
cols.append(key)
vals.append(value)
conn = sqlite3.connect(db_path)
try:
conn.execute(
"INSERT INTO tasks(%s) VALUES(%s)"
% (", ".join(cols), ", ".join("?" * len(cols))),
vals)
conn.commit()
finally:
conn.close()
def _task_status(db_path, task_id):
conn = sqlite3.connect(db_path)
try:
row = conn.execute(
"SELECT status FROM tasks WHERE id = ?",
(task_id,)).fetchone()
finally:
conn.close()
return row[0] if row else None
def _ledger_rows(db_path):
conn = sqlite3.connect(db_path)
try:
rows = conn.execute(
"SELECT task_id, attempt_no, outcome, is_defect, "
"defect_category, corrects_attempt_no FROM task_attempts "
"ORDER BY task_id, attempt_no").fetchall()
except sqlite3.OperationalError:
return []
finally:
conn.close()
return [tuple(r) for r in rows]
def _table_exists(db_path, table):
conn = sqlite3.connect(db_path)
try:
row = conn.execute(
"SELECT COUNT(*) FROM sqlite_master WHERE type='table' "
"AND name=?", (table,)).fetchone()
finally:
conn.close()
return row[0] == 1
def _completed_runner(cmd):
"""Fake CLI: dispatch/run always succeed, host.py-style JSON."""
if "dispatch" in cmd:
name = cmd[cmd.index("--name") + 1]
else:
name = cmd[cmd.index("--task") + 1]
return SimpleNamespace(
returncode=0,
stdout=json.dumps({"task": name, "status": "completed",
"text": "done"}),
stderr="")
class TestClassifyDispatch(unittest.TestCase):
def test_coordinator_beats_marker(self):
decision = classify_dispatch(
{"coordinator_parts": [{"name": "a"}]}, True)
self.assertFalse(decision["counted"])
self.assertEqual(decision["kind"], "coordinator_orchestration")
def test_platform_marker_beats_breakpoint(self):
# A platform goal auto-resume is a breakpoint continuation
# AND a platform rerun: uniform platform exclusion wins.
decision = classify_dispatch(
{"_goal_resume_breakpoint": True}, True)
self.assertFalse(decision["counted"])
self.assertEqual(decision["kind"], "platform_auto_resume")
def test_human_breakpoint_resume_counts(self):
decision = classify_dispatch(
{"_goal_resume_breakpoint": True}, False)
self.assertTrue(decision["counted"])
self.assertEqual(decision["kind"], "breakpoint_resume")
def test_resume_kind_breakpoint_counts_despite_message(self):
# Old-verify goal + /continue: the message is NOT a new
# goal, the loop continues from its breakpoint.
decision = classify_dispatch(
{"resume_message": "keep going",
"_resume_kind": "breakpoint"}, False)
self.assertTrue(decision["counted"])
self.assertEqual(decision["kind"], "breakpoint_resume")
def test_breakpoint_fallback_is_not_a_resume(self):
# Resolution fell back to a fresh goal run: the pending
# flag alone must not classify it as a resume.
decision = classify_dispatch(
{"goal_resume_pending": 1,
"_goal_resume_breakpoint": False}, False)
self.assertTrue(decision["counted"])
self.assertEqual(decision["kind"], "first_or_human_retry")
def test_new_goal_continue_counts_as_new_round(self):
decision = classify_dispatch(
{"resume_message": "new goal",
"_resume_kind": "new-goal"}, False)
self.assertTrue(decision["counted"])
self.assertEqual(decision["kind"], "human_continue")
def test_plain_continue_counts(self):
decision = classify_dispatch(
{"resume_message": "hello"}, False)
self.assertTrue(decision["counted"])
self.assertEqual(decision["kind"], "human_continue")
def test_first_dispatch_counts(self):
decision = classify_dispatch({}, False)
self.assertTrue(decision["counted"])
self.assertEqual(decision["kind"], "first_or_human_retry")
class TestMarkerStore(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.db_path = os.path.join(self.tmp.name, "control.db")
self.conn = _sqlite_conn(self.db_path)
def tearDown(self):
self.conn.close()
self.tmp.cleanup()
def _markers(self):
try:
return self.conn.execute(
"SELECT task_id, origin, queued_at FROM "
"task_attempt_origin_markers").fetchall()
except Exception:
return []
def test_mark_then_consume_matching(self):
mark_platform_rerun(self.conn, "t-1", 1234.5)
self.conn.commit()
self.assertTrue(consume_platform_marker(self.conn, "t-1", 1234.5))
self.conn.commit()
self.assertEqual(self._markers(), [])
# Consuming twice never matches twice.
self.assertFalse(
consume_platform_marker(self.conn, "t-1", 1234.5))
def test_mismatched_marker_is_stale_and_deleted(self):
mark_platform_rerun(self.conn, "t-1", 111.0)
self.conn.commit()
self.assertFalse(
consume_platform_marker(self.conn, "t-1", 222.0))
self.conn.commit()
self.assertEqual(self._markers(), [])
def test_none_generation_consumes_nothing(self):
mark_platform_rerun(self.conn, "t-1", 111.0)
self.conn.commit()
self.assertFalse(
consume_platform_marker(self.conn, "t-1", None))
# The marker survives for a later verifiable consume.
self.assertTrue(
consume_platform_marker(self.conn, "t-1", 111.0))
def test_requeue_overwrites_older_marker(self):
mark_platform_rerun(self.conn, "t-1", 111.0)
mark_platform_rerun(self.conn, "t-1", 222.0)
self.conn.commit()
self.assertTrue(
consume_platform_marker(self.conn, "t-1", 222.0))
def test_mark_requires_task_id(self):
with self.assertRaises(ValueError):
mark_platform_rerun(self.conn, "", 1.0)
def test_marker_pg_adapter_evidence(self):
# Adapter-level only: no real Postgres on this machine.
self.assertEqual(PG_MARKER_DDL, MARKER_DDL)
translated, _kind = _translate_sql(MARKER_DDL)
self.assertIn(
"CREATE TABLE IF NOT EXISTS task_attempt_origin_markers",
translated)
self.assertNotIn("AUTOINCREMENT", translated)
self.assertNotIn("GENERATED", translated)
self.assertIn("PRIMARY KEY", translated)
insert = ("INSERT INTO task_attempt_origin_markers(task_id, "
"origin, queued_at) VALUES(?, ?, ?)")
self.assertEqual(_convert_placeholders(insert).count("%s"), 3)
self.assertNotIn("?", _convert_placeholders(insert))
class TestCorrectionRows(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.db_path = os.path.join(self.tmp.name, "control.db")
self.conn = _sqlite_conn(self.db_path)
def tearDown(self):
self.conn.close()
self.tmp.cleanup()
def _agg(self, task_id):
rows = read_attempt_rows(self.conn)
agg = aggregate_attempts(rows)
return agg.get(task_id)
def test_correction_unmarks_a_defect(self):
record_attempt(self.conn, "t-1", "failure", is_defect=True,
defect_category="contract")
record_correction(self.conn, "t-1", 1, is_defect=False)
agg = self._agg("t-1")
self.assertEqual(agg["attempts"], 1)
self.assertEqual(agg["defects"], 0)
self.assertIsNone(agg["defect_category"])
def test_correction_marks_a_missed_defect(self):
record_attempt(self.conn, "t-1", "failure")
record_correction(self.conn, "t-1", 1, is_defect=True,
defect_category="directory")
agg = self._agg("t-1")
self.assertEqual(agg["attempts"], 1)
self.assertEqual(agg["defects"], 1)
self.assertEqual(agg["defect_category"], "directory")
def test_correction_is_never_counted_as_attempt(self):
record_attempt(self.conn, "t-1", "failure", is_defect=True,
defect_category="contract")
record_attempt(self.conn, "t-1", "success")
record_correction(self.conn, "t-1", 1, is_defect=False)
agg = self._agg("t-1")
self.assertEqual(agg["attempts"], 2)
self.assertEqual(agg["defects"], 0)
def test_last_correction_wins(self):
record_attempt(self.conn, "t-1", "failure", is_defect=True,
defect_category="contract")
record_correction(self.conn, "t-1", 1, is_defect=False)
record_correction(self.conn, "t-1", 1, is_defect=True,
defect_category="directory")
agg = self._agg("t-1")
self.assertEqual(agg["defects"], 1)
self.assertEqual(agg["defect_category"], "directory")
def test_first_defect_category_reevaluated_after_correction(self):
record_attempt(self.conn, "t-1", "failure", is_defect=True,
defect_category="contract")
record_attempt(self.conn, "t-1", "failure", is_defect=True,
defect_category="directory")
record_correction(self.conn, "t-1", 1, is_defect=False)
agg = self._agg("t-1")
self.assertEqual(agg["defects"], 1)
self.assertEqual(agg["defect_category"], "directory")
def test_success_row_can_be_judged_defective_on_review(self):
# Defect judgement is independent of outcome: a shipped
# success can still be judged defective after review.
record_attempt(self.conn, "t-1", "success")
record_correction(self.conn, "t-1", 1, is_defect=True,
defect_category="contract")
agg = self._agg("t-1")
self.assertEqual(agg["attempts"], 1)
self.assertEqual(agg["defects"], 1)
def test_correction_targets_rejected(self):
record_attempt(self.conn, "t-1", "failure")
record_attempt(self.conn, "t-1", "interrupted")
with self.assertRaises(ValueError):
record_correction(self.conn, "t-1", 99, is_defect=True)
with self.assertRaises(ValueError):
# Attempt 2 is the interrupted row: no defect
# judgement exists to correct.
record_correction(self.conn, "t-1", 2, is_defect=True)
with self.assertRaises(ValueError):
# Category without a defect flag.
record_correction(self.conn, "t-1", 1, is_defect=False,
defect_category="contract")
# The two rejected writes above left exactly the two
# original rows; the category rejection wrote nothing.
self.assertEqual(len(_ledger_rows_for(self.conn)), 2)
corr = record_correction(self.conn, "t-1", 1, is_defect=True,
defect_category="contract")
with self.assertRaises(ValueError):
# A correction row is not itself correctable.
record_correction(self.conn, "t-1", corr, is_defect=False)
def test_old_gap3_table_reads_and_upgrades(self):
# A table built from gap 3's DDL (no corrects column):
# old rows still read as plain attempts...
gap3_ddl = (
"CREATE TABLE IF NOT EXISTS task_attempts("
"task_id TEXT NOT NULL, "
"attempt_no INTEGER NOT NULL, "
"outcome TEXT NOT NULL, "
"is_defect INTEGER NOT NULL DEFAULT 0, "
"defect_category TEXT, "
"created_at DOUBLE PRECISION, "
"PRIMARY KEY (task_id, attempt_no))")
self.conn.execute(gap3_ddl)
self.conn.execute(
"INSERT INTO task_attempts(task_id, attempt_no, outcome, "
"is_defect, defect_category, created_at) "
"VALUES('t-old', 1, 'success', 0, NULL, 1.0)")
self.conn.commit()
agg = aggregate_attempts(read_attempt_rows(self.conn))
self.assertEqual(agg["t-old"]["attempts"], 1)
# ...and the documented one-line ALTER enables
# corrections on that table.
self.conn.execute(OLD_TABLE_ALTER)
self.conn.commit()
record_correction(self.conn, "t-old", 1, is_defect=True,
defect_category="contract")
self.assertEqual(self._agg("t-old")["defects"], 1)
def test_correction_pg_adapter_evidence(self):
translated, _kind = _translate_sql(PG_DDL)
self.assertIn("corrects_attempt_no INTEGER", translated)
self.assertIn("PRIMARY KEY (task_id, attempt_no)", translated)
self.assertNotIn("AUTOINCREMENT", translated)
self.assertNotIn("GENERATED", translated)
insert = (
"INSERT INTO task_attempts(task_id, attempt_no, outcome, "
"is_defect, defect_category, created_at, "
"corrects_attempt_no) VALUES(?, ?, ?, ?, ?, ?, ?)")
self.assertEqual(_convert_placeholders(insert).count("%s"), 7)
self.assertNotIn("?", _convert_placeholders(insert))
def _ledger_rows_for(conn):
return conn.execute(
"SELECT * FROM task_attempts ORDER BY task_id, attempt_no"
).fetchall()
class TestWorkerHook(unittest.TestCase):
def setUp(self):
self._old_push_allow = os.environ.get(
"CODEX_PUSH_ALLOW_PRIVATE_TARGETS")
os.environ["CODEX_PUSH_ALLOW_PRIVATE_TARGETS"] = "1"
self.tmp = tempfile.TemporaryDirectory()
self.db_path = os.path.join(self.tmp.name, "control.db")
from cloud.server import init_db
init_db(self.db_path)
self.env = {"XCM_API_KEY": "test-placeholder"}
def tearDown(self):
if getattr(self, "_old_push_allow", None) is None:
os.environ.pop("CODEX_PUSH_ALLOW_PRIVATE_TARGETS", None)
else:
os.environ["CODEX_PUSH_ALLOW_PRIVATE_TARGETS"] = (
self._old_push_allow)
self.tmp.cleanup()
def _worker(self, runner=None, **kwargs):
from cloud.worker import Worker
kwargs.setdefault("env", self.env)
kwargs.setdefault("retry_backoff_seconds", 0)
return Worker(self.db_path, "/fake/host.py",
runner=runner or _completed_runner, **kwargs)
def test_hook_off_writes_nothing_and_creates_no_table(self):
_insert_task(self.db_path, "A")
worker = self._worker()
self.assertFalse(worker.attempt_ledger_hook)
result = worker.run_one()
self.assertEqual(result["status"], "completed")
self.assertFalse(_table_exists(self.db_path, "task_attempts"))
self.assertFalse(
_table_exists(self.db_path, "task_attempt_origin_markers"))
def test_hook_on_first_run_writes_one_success_row(self):
_insert_task(self.db_path, "A")
worker = self._worker(attempt_ledger_hook=True)
result = worker.run_one()
self.assertEqual(result["status"], "completed")
rows = _ledger_rows(self.db_path)
self.assertEqual(len(rows), 1)
self.assertEqual(rows[0][:5], ("A", 1, "success", 0, None))
self.assertIsNone(rows[0][5])
def test_hook_on_transient_retry_is_still_one_row(self):
calls = {"n": 0}
def flaky(cmd):
calls["n"] += 1
if calls["n"] == 1:
return SimpleNamespace(returncode=1, stdout="boom",
stderr="")
return _completed_runner(cmd)
_insert_task(self.db_path, "A")
worker = self._worker(runner=flaky, attempt_ledger_hook=True,
max_retries=1)
result = worker.run_one()
self.assertEqual(result["status"], "completed")
rows = _ledger_rows(self.db_path)
self.assertEqual(len(rows), 1)
self.assertEqual(rows[0][2], "success")
def test_hook_on_human_retry_writes_second_row(self):
_insert_task(self.db_path, "A")
worker = self._worker(attempt_ledger_hook=True)
self.assertEqual(worker.run_one()["status"], "completed")
# Simulate the server /retry endpoint: back to queued,
# results cleared, NO platform marker written.
conn = sqlite3.connect(self.db_path)
try:
conn.execute(
"UPDATE tasks SET status='queued', result_text=NULL, "
"error_text=NULL, actual_model=NULL WHERE id='A'")
conn.commit()
finally:
conn.close()
worker2 = self._worker(attempt_ledger_hook=True)
self.assertEqual(worker2.run_one()["status"], "completed")
rows = _ledger_rows(self.db_path)
self.assertEqual(len(rows), 2)
self.assertEqual([r[1] for r in rows], [1, 2])
self.assertEqual([r[2] for r in rows],
["success", "success"])
def test_hook_on_platform_rerun_writes_no_row(self):
from cloud.attempt_ledger import record_attempt
from cloud.db import connect as db_connect
# Previous dispatch ended interrupted (its row is
# count-neutral by the gap-3 rule).
_insert_task(self.db_path, "A", status="interrupted",
interrupted_reason="worker_shutdown",
paused_by_user=0, auto_resume_count=0)
conn = db_connect(self.db_path)
try:
record_attempt(conn, "A", "interrupted")
finally:
conn.close()
worker = self._worker(attempt_ledger_hook=True)
resumed = worker.auto_resume_interrupted()
self.assertEqual(resumed, ["A"])
self.assertEqual(_task_status(self.db_path, "A"), "queued")
result = worker.run_one()
self.assertEqual(result["status"], "completed")
rows = _ledger_rows(self.db_path)
# Still exactly the one interrupted row: the platform
# rerun added no counted row, and its marker is gone.
self.assertEqual(len(rows), 1)
self.assertEqual(rows[0][2], "interrupted")
conn = sqlite3.connect(self.db_path)
try:
markers = conn.execute(
"SELECT COUNT(*) FROM task_attempt_origin_markers"
).fetchone()[0]
finally:
conn.close()
self.assertEqual(markers, 0)
def test_hook_on_gate_failure_writes_no_row(self):
_insert_task(self.db_path, "A")
worker = self._worker(attempt_ledger_hook=True, env={})
result = worker.run_one()
self.assertEqual(result["status"], "failed")
self.assertFalse(_table_exists(self.db_path, "task_attempts"))
def test_hook_on_cancelled_dispatch_writes_interrupted_row(self):
# Established pattern (test_worker
# test_cancel_during_dispatch_keeps_interrupted): the fake
# runner cancels the task mid-dispatch, then reports
# completed; run_one must keep interrupted, and the hook
# must write exactly one interrupted row for it.
worker_ref = {}
def cancelling(cmd):
worker_ref["w"].cancel("A")
return _completed_runner(cmd)
_insert_task(self.db_path, "A")
worker = self._worker(runner=cancelling,
attempt_ledger_hook=True)
worker_ref["w"] = worker
result = worker.run_one()
self.assertEqual(result["status"], "interrupted")
self.assertEqual(_task_status(self.db_path, "A"), "interrupted")
rows = _ledger_rows(self.db_path)
self.assertEqual(len(rows), 1)
self.assertEqual(rows[0][2], "interrupted")
# Interrupted rows never enter the aggregate.
records = apply_to_records(
[{"id": "A", "status": "interrupted",
"attempts": None, "defects": None}],
aggregate_attempts(read_attempt_rows(
_sqlite_conn(self.db_path))))
metrics = compute_metrics(records)
self.assertEqual(metrics["terminal_tasks"], 0)
self.assertIsNone(metrics["first_pass_rate"])
self.assertFalse(metrics["first_pass_known"])
def test_hook_write_failure_never_fails_task(self):
import cloud.worker as worker_mod
_insert_task(self.db_path, "A")
worker = self._worker(attempt_ledger_hook=True)
original = worker_mod._attempt_record
def boom(conn, task_id, outcome):
raise RuntimeError("ledger down")
worker_mod._attempt_record = boom
try:
result = worker.run_one()
finally:
worker_mod._attempt_record = original
self.assertEqual(result["status"], "completed")
self.assertEqual(_task_status(self.db_path, "A"), "completed")
class MetricsHookEndpointTestCase(unittest.TestCase):
"""Mixed ledger/old/platform/interrupted tasks on the real
endpoint: terminal denominators contain only known tasks."""
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.db_path = os.path.join(self.tmp.name, "control.db")
from cloud.auth import AuthStore
from cloud.cost import CostStore
from cloud.server import create_server
self.auth = AuthStore(self.db_path)
self.owner_id, self.owner_key = self.auth.create_user("owner")
self.server = create_server(self.db_path, "127.0.0.1", 0)
self.port = self.server.server_address[1]
self.thread = threading.Thread(
target=self.server.serve_forever, daemon=True)
self.thread.start()
status, self.project = self._req(
"POST", "/projects", {"name": "demo"}, self.owner_key)
self.assertEqual(status, 201)
self.costs = CostStore(self.db_path)
def tearDown(self):
self.server.shutdown()
self.server.server_close()
self.thread.join(timeout=5)
try:
self.auth.conn.close()
except Exception:
pass
self.tmp.cleanup()
def _req(self, method, path, body=None, key=None):
conn = http.client.HTTPConnection("127.0.0.1", self.port,
timeout=10)
hdrs = {}
payload = None
if body is not None:
payload = json.dumps(body).encode("utf-8")
hdrs["Content-Type"] = "application/json"
if key is not None:
hdrs["X-API-Key"] = key
try:
conn.request(method, path, body=payload, headers=hdrs)
resp = conn.getresponse()
data = resp.read()
finally:
conn.close()
try:
parsed = json.loads(data.decode("utf-8")) if data else None
except ValueError:
parsed = None
return resp.status, parsed
def test_mixed_tasks_known_only_denominators(self):
from cloud.db import connect as db_connect
pid = self.project["id"]
conn = db_connect(self.db_path)
try:
for tid, status in (
("t-a", "completed"), ("t-b", "merged"),
("t-old", "completed"), ("t-plat", "completed"),
("t-int", "interrupted")):
conn.execute(
"INSERT INTO tasks(id, project_id, status) "
"VALUES(?, ?, ?)", (tid, pid, status))
conn.commit()
# t-a: one clean round -> first-pass candidate.
record_attempt(conn, "t-a", "success")
# t-b: defect round then success -> rework + defect.
record_attempt(conn, "t-b", "failure", is_defect=True,
defect_category="contract")
record_attempt(conn, "t-b", "success")
# t-plat: completed ONLY via a platform rerun: its
# ledger holds an interrupted row and nothing else,
# so it stays unknown (never 0, never first-pass).
record_attempt(conn, "t-plat", "interrupted")
# t-int: interrupted STATUS with a failure row: the
# status alone keeps it out of terminal tasks.
record_attempt(conn, "t-int", "failure", is_defect=True,
defect_category="contract")
finally:
conn.close()
# Priced cost only for t-a: the dual-column discipline
# must survive the ledger (total None when incomplete,
# priced partial still shown, unpriced never folded to 0).
self.costs.record("t-a", pid, "gpt-6-sol",
input_tokens=1000, output_tokens=100,
usd_micros=5000, source="actual")
status, report = self._req(
"GET", "/metrics-report", key=self.owner_key)
self.assertEqual(status, 200)
metrics = report["metrics"]
# Terminal tasks: t-a, t-b, t-old, t-plat (t-int is not
# terminal despite its ledger row).
self.assertEqual(metrics["terminal_tasks"], 4)
# Known-only denominators: t-a + t-b (2 tasks). If the
# old or platform task were wrongly counted as
# first-pass / defect-free, these would be 3/4 and 1/4.
self.assertAlmostEqual(metrics["first_pass_rate"], 0.5)
self.assertTrue(metrics["first_pass_known"])
self.assertAlmostEqual(metrics["rework_rate"], 0.5)
self.assertAlmostEqual(metrics["defect_rate"], 0.5)
self.assertTrue(metrics["defect_known"])
self.assertEqual(metrics["defect_by_category"],
{"contract": 1})
self.assertIsNone(metrics["total_usd_micros"])
self.assertEqual(metrics["priced_usd_micros_partial"], 5000)
self.assertFalse(metrics["cost_complete"])
if __name__ == "__main__":
unittest.main()
+149
View File
@@ -116,6 +116,19 @@ Docker 下 repos_dir 必须放在 extra-mounts 已挂载的目录下,
推送到远端不在本轮范围:任务在本地检出上执行、本地 worktree
合并仍按 host.py 现有流程,git push 需要仓库写凭证,控制面
不存、不推。
任务尝试台账写入钩子(见 cloud/attempt_hook.py 的完整政策,
默认关,Worker 参数 attempt_ledger_hook / CLI
--attempt-ledger-hook):开时 run_one 给每个计数的派发写且
只写一行台账(终态 completed -> success、failed -> failure、
interrupted -> interrupted,is_defect 恒 False);同一派发内
的 max_retries 重试、fallback 换模型重跑、goal 已存在断点回
落、后续消息排空都不另起行。平台自动续跑派发(认领时消费
到 auto_resume_interrupted 在同一事务里写的标记)不写行;
协调员父派发不写行(子任务各自写行);缺 key / 仓库 / 配额 /
轮次上限 / 网关等执行前门失败不写行(准入拒绝不是尝试)。
关时 worker 不写任何台账行、不写标记、不建表,与审计加固
补缺口第 3 项的显式填充行为完全一致。
"""
import argparse
@@ -134,6 +147,10 @@ import time
import uuid
from types import SimpleNamespace
from cloud.attempt_hook import classify_dispatch as _attempt_classify
from cloud.attempt_hook import consume_platform_marker as _attempt_consume
from cloud.attempt_hook import mark_platform_rerun as _attempt_mark
from cloud.attempt_hook import record_dispatch_result as _attempt_record
from cloud.auth import AuthStore
from cloud.coordinator import build_report as _coord_build_report
from cloud.coordinator import DEFAULT_POLL_INTERVAL as _COORD_POLL_INTERVAL
@@ -478,6 +495,7 @@ class Worker:
outbox_max_attempts=3,
quota_poll_interval_seconds=30,
auto_resume_max=50,
attempt_ledger_hook=False,
ignore_host_budget=True,
worker_address=None):
self.db_path = str(db_path)
@@ -583,6 +601,13 @@ class Worker:
# 防确定性失败(每次续跑立刻又中断)无限烧额度;
# 正常部署重启/偶发中断远达不到 50 次。
self.auto_resume_max = auto_resume_max
# 任务尝试台账写入钩子(见 cloud/attempt_hook.py 的完整
# 政策):默认关。关时 worker 不写任何台账行、不写平台
# 标记、不建台账/标记表,行为与审计加固补缺口第 3 项
# (显式 record_attempt 填充)完全一致;开时 run_one
# 按政策给计数的派发写且只写一行(is_defect 恒 False,
# worker 永不判定缺陷),平台自动续跑派发不写行。
self.attempt_ledger_hook = bool(attempt_ledger_hook)
# 项目仓库检出根目录:CLI --repos-dir 可调(默认 ./repos);
# Docker 执行器下它必须放在 extra-mounts 已挂载的目录下,
# 否则容器内 task["repo"] 路径不存在——见 cloud/README.md。
@@ -1124,6 +1149,23 @@ class Worker:
resumed.append(task_id)
self._audit(project_id, "worker_auto_resumed",
task_id)
# 台账钩子(仅开时):同一事务里写平台
# 标记,queued_at 取本次重排队写的
# updated_at(即 now),下次认领时凭它
# 识别这就是平台续跑派发(见
# attempt_hook.py 规则 4)。标记写失败
# 绝不挡重排队:只记日志、本次重排队照
# 常提交,派发退化为计数( fail-safe
# 方向,见规则 5 的 mid-flight 说明)。
if self.attempt_ledger_hook:
try:
_attempt_mark(conn, task_id, now)
except Exception as exc:
log_event(
self.logger,
"worker_attempt_marker_failed",
message="attempt marker failed: %s"
% exc)
if is_goal:
summary = ("自动续跑(无进展计数 %d,"
"下次退避 %d 秒)"
@@ -5608,6 +5650,26 @@ class Worker:
self.usage_cost_usd += micros / 1000000.0
return cost_id
def _record_ledger_attempt(self, task_id, outcome, counted):
"""台账钩子唯一写行入口(best-effort,见 attempt_hook.py
规则 7):关时或不计数时直接返回,不碰数据库、不建
表;开且计数时写一行(is_defect 恒 False)。写失败
只记日志,绝不影响任务状态与返回值。调用方须只在
run_one 的终局点(取消分支、正常终局块)调用,门失败
与协调员分支故意不调。"""
if not self.attempt_ledger_hook or not counted:
return None
try:
conn = _db_connect(self.db_path)
try:
return _attempt_record(conn, task_id, outcome)
finally:
conn.close()
except Exception as exc:
log_event(self.logger, "worker_attempt_ledger_failed",
message="attempt ledger write failed: %s" % exc)
return None
def run_one(self, task=None):
"""认领并执行一个任务,返回结果 dict(见模块 docstring)。
@@ -5621,6 +5683,25 @@ class Worker:
if task is None:
return {"dispatched": False, "reason": "no_ready_task"}
task_id = task["id"]
# 台账钩子:认领前抓排队态的 updated_at(认领会改写
# 它)。仅开时用:凭它在认领后消费平台标记(见
# attempt_hook.py 规则 4);任务字典里没有时(调用方
# 自建字典)认领前直读一次库,读不到则为 None(标记
# 无法验证 → 派发按计数处理,fail-safe 方向)。
_ledger_queued_at = task.get("updated_at")
if self.attempt_ledger_hook and _ledger_queued_at is None:
try:
_qconn = _db_connect(self.db_path)
try:
_qrow = _qconn.execute(
"SELECT updated_at FROM tasks WHERE id = ?",
(task_id,)).fetchone()
if _qrow is not None:
_ledger_queued_at = _qrow[0]
finally:
_qconn.close()
except Exception:
_ledger_queued_at = None
if not self._claim(task_id, executor=self._executor_name()):
return {"dispatched": False, "reason": "already_claimed",
"task_id": task_id}
@@ -5637,6 +5718,28 @@ class Worker:
except Exception:
pass
task = self.get_task(task_id) or task
# 台账钩子:认领成功后立即消费平台标记(仅开时)。
# 认领已赢才消费:认领失败的分支在上面已返回,标记
# 留给赢的 worker;queued_at 不匹配时消费函数删掉
# 陈旧标记并返回 False(见 attempt_hook.py 规则 4)。
# 标记表不存在等错误只记日志、不挡派发(派发退化为
# 计数,fail-safe 方向)。
_ledger_platform = False
if self.attempt_ledger_hook:
try:
_mconn = _db_connect(self.db_path)
try:
_ledger_platform = _attempt_consume(
_mconn, task_id, _ledger_queued_at)
_mconn.commit()
finally:
_mconn.close()
except Exception as exc:
log_event(self.logger,
"worker_attempt_marker_failed",
message="attempt marker consume failed: %s"
% exc)
_ledger_platform = False
# G10:goal 断点续跑标记——认领后立即清掉库里标记(本轮
# 已认领为 running,不会重复触发;本轮若再被杀回
# interrupted,用户需重新调 /goal-resume 才会再置标记,
@@ -5698,6 +5801,19 @@ class Worker:
else:
task["prompt"] = resume_message
task["_resume"] = True
# 台账钩子:分类必须放在断点解析与续跑消息处理之后,
# 因为分类读的是解析后的字段:_goal_resume_breakpoint
# 为 False 表示断点解析已回落成全新 goal(那就是一轮
# 新尝试,不是断点续跑);_resume_kind == "breakpoint"
# 表示旧验收 goal 的续跑消息沿用原循环(同轮续跑)。
# 关时 _ledger_counted 恒 False,后面所有写行点都
# 是空操作;门失败提前返回(缺 key / 仓库 / 配额 /
# 轮次上限 / 网关)故意不写行(见 attempt_hook.py
# 规则 2:准入拒绝不是尝试),协调员分支同理。
_ledger_decision = _attempt_classify(
task, _ledger_platform) if self.attempt_ledger_hook \
else {"counted": False, "kind": "hook_disabled"}
_ledger_counted = bool(_ledger_decision["counted"])
# 自动记忆注入(官方 use_memories):首轮才注入,见
# _apply_auto_memory_injection 的跳过规则;只改内存副本。
task = self._apply_auto_memory_injection(task)
@@ -5872,6 +5988,11 @@ class Worker:
# 认领不了(2026-10-28 40 人与 18 人旧轮的第 4 槽/C-6
# 未认领即此漏)。finally 清槽,异常路径同样不漏。
if task.get("coordinator_parts"):
# 台账钩子:协调员父派发不写行(attempt_hook.py
# 规则 2:父任务只做扇出聚合、无模型回合,子任务
# 各自经自己的 run_one 写行)。这里直接返回,刻
# 意绕过下面的终局写行点;_ledger_counted 对协调
# 员恒 False, defensive 双保险。
try:
return self._run_coordinator(task, started_at)
finally:
@@ -6208,6 +6329,10 @@ class Worker:
_cancel_cost_id = None
log_event(self.logger, "worker_cancelled", task_id=task_id,
project_id=task["project_id"], status="interrupted")
# 台账钩子:取消派发写 interrupted 行(不计分母,
# 只留档);不计数派发(平台/协调员)时为空操作。
self._record_ledger_attempt(task_id, "interrupted",
_ledger_counted)
return {"dispatched": True, "task_id": task_id,
"host_name": self.executor._host_name(task)
if hasattr(self.executor, "_host_name") else task_id,
@@ -6360,6 +6485,21 @@ class Worker:
# 补一条 worker_completed/failed 审计误导排查。
final = "interrupted"
self._audit(task["project_id"], "worker_interrupted", task_id)
# 台账钩子:本派发唯一正常写行点(取消分支已单独
# 写,协调员与门失败分支不写)。终态映射:completed
# -> success,failed -> failure,interrupted ->
# interrupted;/goal 转换时 final 已被改写成 queued,
# 但主轮已完成,记 success;final 落在三个终态之外
# 时不写(不猜测 outcome,宁缺勿错)。
if goal_converted:
self._record_ledger_attempt(task_id, "success",
_ledger_counted)
elif final in ("completed", "failed", "interrupted"):
self._record_ledger_attempt(
task_id,
{"completed": "success", "failed": "failure",
"interrupted": "interrupted"}[final],
_ledger_counted)
if resume_message and not quota_stop:
# 本轮已跑完(无论成败):续跑消息已消费,清掉;进程
# 中途被杀时清不到,下次认领会重跑同一条消息——可接受
@@ -7130,6 +7270,14 @@ def main(argv=None):
help="长任务自动恢复上限(次数,非时间,默认 50):"
"同一 goal 任务被系统中断后自动续跑超过该次数"
"即停住等人工,防确定性失败无限烧额度")
parser.add_argument("--attempt-ledger-hook", action="store_true",
help="任务尝试台账写入钩子(默认关):开时 worker"
" 按 cloud/attempt_hook.py 的政策给计数的派发"
" 写且只写一行台账(is_defect 恒 False),平台"
" 自动续跑派发不写行;关时行为与审计加固补缺口"
" 第 3 项完全一致(只经显式 record_attempt 填充)。"
" 生产 PG 上启用前须先手动执行台账 PG_DDL 与"
" 标记表 PG_MARKER_DDL 并验证")
parser.add_argument("--loop", action="store_true",
help="常驻循环,每 --interval 秒跑一轮")
parser.add_argument("--interval", type=int, default=300)
@@ -7164,6 +7312,7 @@ def main(argv=None):
max_retries=args.max_retries,
retry_backoff_seconds=args.retry_backoff_seconds,
auto_resume_max=args.auto_resume_max,
attempt_ledger_hook=args.attempt_ledger_hook,
repos_dir=args.repos_dir,
heartbeat_file=args.heartbeat_file,
worker_id=args.worker_id,