audit: task-attempt ledger feeds real first-pass/defect/rework rates (gap 3)

- cloud/attempt_ledger.py: separate task_attempts table (not tasks
  columns) with per-attempt rows, MAX+1 attempt_no, append-only
  intent; interrupted rows kept but never counted; is_defect is
  explicit, never inferred from failure; first defect category
  wins; no-ledger tasks stay unknown (None), never backfilled.
  SQLite DDL is CREATE TABLE IF NOT EXISTS (idempotent); PG_DDL
  is the verbatim manual step for production Postgres, kept out
  of POSTGRES_SCHEMA on purpose (migrations coverage untouched).
- cloud/server.py GET /metrics-report: aggregates the ledger
  into compute_metrics records; missing ledger table degrades to
  unknown rates, not an error. Cost dual-column code untouched.
- cloud/metrics.py: docstring only (ledger is now the source);
  compute_metrics logic and all #238 semantics unchanged.
- Tests: 16 ledger tests (store/aggregate/metrics/PG adapter
  evidence) + 3 endpoint tests (real rates, old task beside
  ledger task, interrupted status non-terminal).
This commit is contained in:
host committed 2026-11-04 20:56:06 +00:00
1 parent 8a4e80c20d
commit d303dedf32
6 files changed
+639 -22

No files matched your search

+228
View File
@@ -0,0 +1,228 @@
"""Task-attempt ledger for the step-5 metrics loop (audit gap 3).
Choice made here, with the trade-off stated rather than hidden:
- This module adds a SEPARATE ``task_attempts`` table; it does NOT
add ``attempts``/``defects`` columns to ``tasks``. Columns would be
simpler to query (one row per task, no join), but they collapse
history into two counters: a task's per-attempt story (which
attempt failed, with what defect category, in what order) would be
unrecoverable, and widening the hot ``tasks`` table on production
Postgres is a heavier migration than adding a new, empty table.
The ledger costs one aggregate query at report time, and that
query is exactly what ``aggregate_attempts`` exists to pin down.
- The ledger is append-only in intent: ``record_attempt`` only ever
inserts a new ``(task_id, attempt_no)`` row (``attempt_no`` is
``MAX + 1`` per task, never reused). There is no update path, so a
recorded attempt cannot be silently rewritten into a first-pass
success after the fact.
Semantics (pinned by tests, not left to each caller):
- One row = one real attempt at the task. ``outcome`` is one of
``success`` / ``failure`` / ``interrupted``.
- ``interrupted`` rows are kept for audit but NEVER count toward
``attempts`` or ``defects``: a platform interruption that
auto-resumes as a fresh run is a platform event, not an agent
failure, and the step-5 rule that ``interrupted`` is not a
terminal task status would be meaningless if the interruption
still inflated the attempt count. A task whose ledger holds only
interrupted rows has NO counted attempt, so it is omitted from
the aggregate entirely -- the endpoint then still reports its
rates as unknown (``None``), never as 0.
- ``is_defect`` is set explicitly by the recorder, never inferred
from ``outcome``: a failed attempt is not automatically a defect
(a retry can fail for a transient reason without being a
defect round), and a defect round is a judgement the recording
caller makes. ``defect_category`` requires ``is_defect``; an
interrupted row can never be a defect. Both contradictions raise
``ValueError`` instead of being stored as ambiguous data.
- ``defect_category`` for a task (the single category that
``compute_metrics`` carries per task) is the category of its
FIRST defect row in attempt order. Later defect rows still count
toward ``defects``; only the category label is first-wins, so it
is deterministic.
- Old tasks have no ledger rows, and that is NOT backfilled: the
aggregate simply has no entry for them, the endpoint leaves their
``attempts``/``defects`` as ``None``, and ``compute_metrics``
excludes them from rate denominators (they are neither counted
as first-pass nor as defect-free). Fabricating a row for an old
task (e.g. ``attempts=1, defects=0`` from "it completed") would
be inventing history.
Persistence notes:
- SQLite: ``ensure_task_attempts_table`` runs
``CREATE TABLE IF NOT EXISTS`` -- idempotent, safe to call on
every write, and a no-op on a database that already has the
table (including one that already has rows).
- Postgres: the production table is NOT created by
``init_postgres_schema`` (deliberately: that schema is covered by
the migrations test asserting ``MIGRATION_TABLES`` equals the
schema set, and this ledger must not silently widen that set).
``PG_DDL`` below is the exact statement to run manually on
production Postgres and verify (``\\d task_attempts`` or the
``information_schema`` equivalent) before any Postgres-backed
endpoint is expected to return real rates. The DDL uses only
Postgres-valid types (``TEXT``/``INTEGER``/``DOUBLE PRECISION``),
a composite primary key, and no ``AUTOINCREMENT``/identity
column, so it needs no sequence handling; the parameterized
``INSERT`` uses ``?`` placeholders, which the ``cloud.db``
Postgres adapter translates to ``%s`` (see the adapter test in
``test_attempt_ledger.py``). Until that manual step runs, the
endpoint treats a missing table exactly like "no ledger rows":
unknown rates, not an error -- the same fail-soft shape already
used for missing ``cost_records``.
- The read path (``GET /metrics-report``) never creates the table:
a read-only endpoint must not mutate schema. Table creation
happens only on the write path, inside ``record_attempt``.
- No lifecycle hook writes to this ledger yet. Hooking a writer
into the worker's retry/resume loops would risk exactly the
misclassification this module refuses to make (platform
auto-resume counted as an attempt, transient retry counted as a
defect). The writer -- which lifecycle event counts as an
attempt, and who judges ``is_defect`` -- is a separate decision,
listed as an open item; until it lands, the ledger fills only
through explicit ``record_attempt`` calls, and production rates
stay honestly unknown rather than auto-filled.
This module imports no ``cloud.*`` module and opens no database
connection itself: callers pass a connection in, so the layer rules
(``arch_conformance``: no direct driver connection outside
``cloud.db``) are untouched -- there is no ``sqlite3.connect`` and
no ``psycopg*`` import here at all.
"""
import time
TABLE = "task_attempts"
OUTCOMES = frozenset({"success", "failure", "interrupted"})
# One DDL serves both backends: every type and the composite key are
# valid in SQLite and in Postgres, and there is no AUTOINCREMENT or
# identity column for either backend's sequence handling to chase.
SQLITE_DDL = (
"CREATE TABLE IF NOT EXISTS task_attempts("
"task_id TEXT NOT NULL, "
"attempt_no INTEGER NOT NULL, "
"outcome TEXT NOT NULL, "
"is_defect INTEGER NOT NULL DEFAULT 0, "
"defect_category TEXT, "
"created_at DOUBLE PRECISION, "
"PRIMARY KEY (task_id, attempt_no))"
)
# Production Postgres: run this verbatim by hand and verify the
# table exists before expecting real rates from a Postgres-backed
# endpoint. It is intentionally NOT part of POSTGRES_SCHEMA (see
# the module docstring: migrations coverage must not widen silently).
PG_DDL = SQLITE_DDL
def ensure_task_attempts_table(conn):
"""Create the ledger table if absent. Idempotent; does not
touch existing rows. The caller owns the commit (``record_attempt``
commits; a bare ensure inside a larger transaction does not)."""
conn.execute(SQLITE_DDL)
def record_attempt(conn, task_id, outcome, is_defect=False,
defect_category=None, created_at=None):
"""Append one attempt row for ``task_id``. Returns ``attempt_no``.
Validates before writing: unknown outcome, a defect flag on an
interrupted row, and a defect category without a defect flag
all raise ``ValueError`` and write nothing.
"""
if not task_id:
raise ValueError("task_id is required")
if outcome not in OUTCOMES:
raise ValueError("outcome must be one of %s, got %r"
% (sorted(OUTCOMES), outcome))
if outcome == "interrupted" and is_defect:
raise ValueError("an interrupted attempt is never a defect")
if defect_category is not None and not is_defect:
raise ValueError("defect_category requires is_defect=True")
ensure_task_attempts_table(conn)
row = conn.execute(
"SELECT COALESCE(MAX(attempt_no), 0) FROM task_attempts "
"WHERE task_id = ?",
(task_id,),
).fetchone()
attempt_no = int(row[0]) + 1
conn.execute(
"INSERT INTO task_attempts(task_id, attempt_no, outcome, "
"is_defect, defect_category, created_at) "
"VALUES(?, ?, ?, ?, ?, ?)",
(task_id, attempt_no, outcome, 1 if is_defect else 0,
defect_category,
time.time() if created_at is None else created_at),
)
conn.commit()
return attempt_no
def read_attempt_rows(conn):
"""All ledger rows in attempt order. Raises the backend's
OperationalError when the table does not exist yet; the
endpoint caller treats that as "no ledger", not as a failure."""
return conn.execute(
"SELECT task_id, attempt_no, outcome, is_defect, "
"defect_category FROM task_attempts "
"ORDER BY task_id, attempt_no").fetchall()
def _field(row, name):
# sqlite3.Row, dict, and the Postgres adapter's rows all support
# key access; a plain tuple row is not accepted (callers pass
# rows, not raw tuples, so a wrong shape fails loudly here).
return row[name]
def aggregate_attempts(rows):
"""Pure aggregation: ledger rows -> ``{task_id: record}`` where
``record`` is the ``attempts``/``defects``/``defect_category``
dict that ``compute_metrics`` consumes.
Interrupted rows are excluded from both counts, and a task
with only interrupted rows is omitted entirely (no counted
attempt => its rates stay unknown, never 0).
"""
per_task = {}
ordered = sorted(
rows,
key=lambda r: (_field(r, "task_id"), _field(r, "attempt_no")))
for row in ordered:
task_id = _field(row, "task_id")
if _field(row, "outcome") == "interrupted":
continue
entry = per_task.setdefault(
task_id, {"attempts": 0, "defects": 0,
"defect_category": None})
entry["attempts"] += 1
if _field(row, "is_defect"):
entry["defects"] += 1
if entry["defect_category"] is None:
entry["defect_category"] = _field(row, "defect_category")
return per_task
def apply_to_records(records, aggregates):
"""Return a new record list with ledger values merged in.
Records for tasks with no aggregate entry keep whatever
``attempts``/``defects`` they already carried (the endpoint
passes ``None`` there, i.e. old tasks stay unknown). Input
records are not modified.
"""
merged = []
for record in records:
out = dict(record)
agg = aggregates.get(record.get("id"))
if agg is not None:
out["attempts"] = agg["attempts"]
out["defects"] = agg["defects"]
out["defect_category"] = agg["defect_category"]
merged.append(out)
return merged
+9 -8
View File
@@ -4,8 +4,8 @@ Zero third-party dependencies; standard library only. This module
imports no ``cloud.*`` module: it computes over plain record dicts so
the collection logic is testable without a database, and so the
report shape is identical whether the records came from the control
plane (``server._metrics_report``), a test fixture, or a future
attempt ledger.
plane (``server._metrics_report`` reading the attempt ledger), a
test fixture, or a hand-built record list.
Definitions (pinned here, not left to each caller):
@@ -17,12 +17,13 @@ Definitions (pinned here, not left to each caller):
- First-pass green: terminal task whose ``attempts`` == 1 and whose
status is ``completed`` / ``merged``.
- Defect: a task with ``defects`` > 0, where ``defects`` is the count
of failed CI/audit rounds or follow-up rounds for that task. The
current tasks schema has NO attempts/defects column, so a caller
that cannot supply them must pass ``None`` -- and the rates then
come back as ``None`` with ``*_known`` False. An unknown rate is
never reported as 0: 0 would claim "no defects", which is a
decision-grade lie.
of failed CI/audit rounds or follow-up rounds for that task. Since
audit gap 3 these numbers come from the ``task_attempts`` ledger
(``cloud/attempt_ledger.py``: one row per real attempt, aggregated
by the endpoint); a task with no ledger rows must still pass
``None`` -- and the rates then come back as ``None`` with
``*_known`` False. An unknown rate is never reported as 0: 0
would claim "no defects", which is a decision-grade lie.
- Rework: terminal task with ``attempts`` > 1. Same None discipline.
- Per-task cost reuses the #212 dual-column discipline from
``cloud.cost``: priced USD micros and unpriced actual tokens are
+19 -6
View File
@@ -169,6 +169,9 @@ from cloud.approvals import ensure_approvals_table as _appr_ensure
from cloud.approvals import expire_overdue as _appr_expire
from cloud.approvals import get_approval as _appr_get
from cloud.approvals import list_approvals as _appr_list
from cloud.attempt_ledger import aggregate_attempts as _aggregate_attempts
from cloud.attempt_ledger import apply_to_records as _apply_attempts
from cloud.attempt_ledger import read_attempt_rows as _read_attempt_rows
from cloud.auto_memory import AutoMemoryStore
from cloud.auto_memory import process_due_tasks as _auto_process_due
from cloud.identity import IdentityStore
@@ -10129,12 +10132,14 @@ pid 须为整数、address 须为字符串,否则 400(注册表里的
"""指标闭环报告:首过绿灯率 / 缺陷率 / 返工率 / 每任务成本。
权限与 /metrics 同款:只统计调用者是成员的项目。
诚实边界:当前 tasks 表没有 attempts/defects 列,所以
attempts/defects 一律传 None,首过/缺陷/返工率回 None
(未知,不报 0);金任务集未经人工审定,报告恒为
pending_human_approval、decision_ready False。等任务尝试
台账落库、金任务人工审定后,这两处再接真实输入——
在此之前绝不把样本集或未知率包装成可决策数字。
台账(audit gap 3,见 ``cloud/attempt_ledger.py``):任务的
attempts/defects 来自 ``task_attempts`` 台账的聚合,不再
恒为 None;台账里只有中断行的任务、以及台账里根本没有
行的老任务,仍传 None——未知率回 None(未知,不报 0),
老任务绝不被算成首过或无缺陷。台账表尚不存在(老库、
或生产 PG 未手动执行 ``PG_DDL``)时与无台账行同等处理,
不因此让端点报错;金任务集未经人工审定,报告恒为
pending_human_approval、decision_ready False。
成本走 #212 双列口径:有价合计与无价 actual token 分开,
总额在成本不完整时为 None,不把无价折成 0。
"""
@@ -10153,6 +10158,14 @@ pid 须为整数、address 须为字符串,否则 400(注册表里的
"attempts": None, "defects": None}
for row in rows
if row["project_id"] in member_ids]
try:
aggregates = _aggregate_attempts(
_read_attempt_rows(conn))
except (sqlite3.OperationalError, _PgOperationalError):
# No ledger table yet (old DB, or production PG
# before the manual PG_DDL step): same as no rows.
aggregates = {}
records = _apply_attempts(records, aggregates)
try:
cost_rows = conn.execute(
"SELECT task_id, project_id, source, total_tokens,"
+245
View File
@@ -0,0 +1,245 @@
"""Tests for the task-attempt ledger (audit gap 3).
Only the standard library; a temporary SQLite DB for the store
tests, plain dicts for the pure aggregation. No real model, no
network, no secrets, no real Postgres (the Postgres claim is
checked at the adapter level only, and labelled as such).
Pinned boundaries:
- interrupted rows never inflate attempts/defects, and a task with
only interrupted rows stays unknown (omitted, not 0);
- old tasks with no ledger rows stay unknown (None + *_known False),
never counted as first-pass or defect-free;
- interrupted task STATUS is never terminal, even with ledger rows;
- the dual-column cost discipline is unchanged by the ledger.
"""
import os
import sys
import unittest
sys.path.insert(0, os.path.dirname(os.path.dirname(
os.path.dirname(os.path.abspath(__file__)))))
from cloud.attempt_ledger import ( # noqa: E402
PG_DDL,
aggregate_attempts,
apply_to_records,
ensure_task_attempts_table,
read_attempt_rows,
record_attempt,
)
from cloud.db import _convert_placeholders, _translate_sql # noqa: E402
from cloud.metrics import compute_metrics # noqa: E402
def _sqlite_conn(path):
from cloud.db import connect as db_connect
return db_connect(path)
class TestLedgerStore(unittest.TestCase):
def setUp(self):
import tempfile
self.tmp = tempfile.TemporaryDirectory()
self.db_path = os.path.join(self.tmp.name, "control.db")
self.conn = _sqlite_conn(self.db_path)
def tearDown(self):
self.conn.close()
self.tmp.cleanup()
def _count(self):
# Validation in record_attempt runs before ensure, so a
# rejected write may leave the table uncreated: that is
# still "writes nothing", counted as 0 here.
try:
return self.conn.execute(
"SELECT COUNT(*) FROM task_attempts").fetchone()[0]
except Exception: # noqa: BLE001 -- missing table == 0 rows
return 0
def test_ensure_is_idempotent_and_preserves_rows(self):
ensure_task_attempts_table(self.conn)
record_attempt(self.conn, "t-1", "failure", is_defect=True,
defect_category="contract")
ensure_task_attempts_table(self.conn)
ensure_task_attempts_table(self.conn)
self.assertEqual(self._count(), 1)
def test_attempt_numbers_are_per_task_and_sequential(self):
self.assertEqual(
record_attempt(self.conn, "t-1", "failure", is_defect=True,
defect_category="contract"), 1)
self.assertEqual(
record_attempt(self.conn, "t-1", "success"), 2)
self.assertEqual(
record_attempt(self.conn, "t-2", "success"), 1)
rows = {(r["task_id"], r["attempt_no"])
for r in read_attempt_rows(self.conn)}
self.assertEqual(rows, {("t-1", 1), ("t-1", 2), ("t-2", 1)})
def test_unknown_outcome_rejected_and_writes_nothing(self):
with self.assertRaises(ValueError):
record_attempt(self.conn, "t-1", "maybe")
# The table was ensured but no row was written.
self.assertEqual(self._count(), 0)
def test_interrupted_row_can_never_be_a_defect(self):
with self.assertRaises(ValueError):
record_attempt(self.conn, "t-1", "interrupted",
is_defect=True)
self.assertEqual(self._count(), 0)
def test_defect_category_without_defect_flag_rejected(self):
with self.assertRaises(ValueError):
record_attempt(self.conn, "t-1", "failure",
defect_category="contract")
self.assertEqual(self._count(), 0)
class TestAggregate(unittest.TestCase):
def _row(self, task_id, attempt_no, outcome, is_defect=0,
category=None):
return {"task_id": task_id, "attempt_no": attempt_no,
"outcome": outcome, "is_defect": is_defect,
"defect_category": category}
def test_interrupted_rows_excluded_from_both_counts(self):
agg = aggregate_attempts([
self._row("t-1", 1, "interrupted"),
self._row("t-1", 2, "failure", 1, "contract"),
self._row("t-1", 3, "success"),
])
self.assertEqual(agg["t-1"]["attempts"], 2)
self.assertEqual(agg["t-1"]["defects"], 1)
self.assertEqual(agg["t-1"]["defect_category"], "contract")
def test_only_interrupted_task_is_omitted_not_zero(self):
agg = aggregate_attempts([
self._row("t-1", 1, "interrupted"),
])
self.assertNotIn("t-1", agg)
def test_first_defect_category_wins_regardless_of_row_order(self):
agg = aggregate_attempts([
self._row("t-1", 2, "failure", 1, "directory"),
self._row("t-1", 1, "failure", 1, "contract"),
])
self.assertEqual(agg["t-1"]["defects"], 2)
self.assertEqual(agg["t-1"]["defect_category"], "contract")
def test_failed_attempt_is_not_automatically_a_defect(self):
agg = aggregate_attempts([
self._row("t-1", 1, "failure"),
self._row("t-1", 2, "success"),
])
self.assertEqual(agg["t-1"]["attempts"], 2)
self.assertEqual(agg["t-1"]["defects"], 0)
self.assertIsNone(agg["t-1"]["defect_category"])
def test_apply_does_not_mutate_input_and_keeps_old_unknown(self):
records = [{"id": "old", "status": "completed",
"attempts": None, "defects": None},
{"id": "new", "status": "completed",
"attempts": None, "defects": None}]
merged = apply_to_records(
records, {"new": {"attempts": 1, "defects": 0,
"defect_category": None}})
self.assertIsNone(records[1]["attempts"]) # input untouched
self.assertIsNone(merged[0]["attempts"]) # old stays unknown
self.assertEqual(merged[1]["attempts"], 1)
self.assertEqual(merged[1]["defects"], 0)
class TestMetricsFromLedger(unittest.TestCase):
def test_real_rates_from_ledger(self):
records = apply_to_records([
{"id": "a", "status": "completed",
"attempts": None, "defects": None},
{"id": "b", "status": "merged",
"attempts": None, "defects": None},
{"id": "c", "status": "failed",
"attempts": None, "defects": None},
], {
"a": {"attempts": 1, "defects": 0,
"defect_category": None},
"b": {"attempts": 3, "defects": 2,
"defect_category": "contract"},
"c": {"attempts": 2, "defects": 1,
"defect_category": "contract"},
})
m = compute_metrics(records)
self.assertEqual(m["terminal_tasks"], 3)
self.assertAlmostEqual(m["first_pass_rate"], 1 / 3)
self.assertAlmostEqual(m["rework_rate"], 2 / 3)
self.assertAlmostEqual(m["defect_rate"], 2 / 3)
self.assertEqual(m["defect_by_category"], {"contract": 2})
self.assertTrue(m["first_pass_known"])
self.assertTrue(m["defect_known"])
def test_old_tasks_without_rows_stay_unknown_not_zero(self):
records = [{"id": "old", "status": "completed",
"attempts": None, "defects": None}]
m = compute_metrics(records)
self.assertEqual(m["terminal_tasks"], 1)
self.assertIsNone(m["first_pass_rate"])
self.assertIsNone(m["rework_rate"])
self.assertIsNone(m["defect_rate"])
self.assertFalse(m["first_pass_known"])
self.assertFalse(m["defect_known"])
self.assertEqual(m["defect_tasks"], 0) # 0 known defective,
# not a claim of "no defects": defect_known is False above.
def test_interrupted_status_never_terminal_even_with_rows(self):
records = apply_to_records([
{"id": "i", "status": "interrupted",
"attempts": None, "defects": None},
], {"i": {"attempts": 1, "defects": 0,
"defect_category": None}})
m = compute_metrics(records)
self.assertEqual(m["terminal_tasks"], 0)
self.assertIsNone(m["first_pass_rate"])
self.assertFalse(m["first_pass_known"])
def test_ledger_does_not_change_dual_column_cost(self):
records = apply_to_records([
{"id": "a", "status": "completed",
"attempts": None, "defects": None},
], {"a": {"attempts": 1, "defects": 0,
"defect_category": None}})
costs = {"a": {"priced_usd_micros": 5000,
"unpriced_actual_tokens": 700,
"has_rows": True}}
m = compute_metrics(records, costs)
self.assertAlmostEqual(m["first_pass_rate"], 1.0)
self.assertIsNone(m["total_usd_micros"])
self.assertEqual(m["priced_usd_micros_partial"], 5000)
self.assertEqual(
m["per_task_cost"][0]["unpriced_actual_tokens"], 700)
class TestPostgresAdapterEvidence(unittest.TestCase):
"""Adapter-level evidence only -- no real Postgres here (see
the module docstring: production PG still needs the manual
PG_DDL step and a live verification)."""
def test_pg_ddl_needs_no_sequence_or_identity_handling(self):
translated, _kind = _translate_sql(PG_DDL)
self.assertIn("CREATE TABLE IF NOT EXISTS task_attempts",
translated)
self.assertNotIn("AUTOINCREMENT", translated)
self.assertNotIn("GENERATED", translated)
self.assertIn("PRIMARY KEY (task_id, attempt_no)", translated)
def test_insert_placeholders_translate_to_percent_s(self):
sql = ("INSERT INTO task_attempts(task_id, attempt_no, "
"outcome, is_defect, defect_category, created_at) "
"VALUES(?, ?, ?, ?, ?, ?)")
translated = _convert_placeholders(sql)
self.assertEqual(translated.count("%s"), 6)
self.assertNotIn("?", translated)
if __name__ == "__main__":
unittest.main()
+85 -4
View File
@@ -4,8 +4,11 @@ Only the standard library http.client; a temporary SQLite DB and a
temporary port; no real model, no network, no secrets.
Covers: unauthenticated -> 401, empty project -> report shape with
unknown rates (None, not 0) and pending golden set, and a completed
task with a priced cost row -> terminal count and dual-column cost.
unknown rates (None, not 0) and pending golden set, a completed
task with a priced cost row -> terminal count and dual-column cost,
and (audit gap 3) ledger-backed real rates, old tasks staying
unknown beside ledger tasks, and interrupted status staying
non-terminal even with ledger rows.
"""
import http.client
@@ -109,12 +112,90 @@ class MetricsReportEndpointTestCase(unittest.TestCase):
"GET", "/metrics-report", key=self.owner_key)
self.assertEqual(status, 200)
self.assertEqual(report["metrics"]["terminal_tasks"], 1)
# Attempts are not in the schema: first-pass stays unknown,
# it must not be fabricated as 0 or 1 from status alone.
# No ledger rows for this task (and no ledger table yet):
# first-pass stays unknown, it must not be fabricated as
# 0 or 1 from status alone.
self.assertIsNone(report["metrics"]["first_pass_rate"])
self.assertEqual(report["metrics"]["total_usd_micros"], 5000)
self.assertTrue(report["metrics"]["cost_complete"])
def test_ledger_task_gives_real_rates(self):
from cloud.attempt_ledger import record_attempt
from cloud.db import connect as db_connect
conn = db_connect(self.db_path)
try:
conn.execute(
"INSERT INTO tasks(id, project_id, status) VALUES(?, ?, ?)",
("t-led", self.project["id"], "completed"))
conn.commit()
record_attempt(conn, "t-led", "success")
finally:
conn.close()
status, report = self._req(
"GET", "/metrics-report", key=self.owner_key)
self.assertEqual(status, 200)
metrics = report["metrics"]
self.assertEqual(metrics["terminal_tasks"], 1)
# A ledger row exists, so 0 defects here is a real 0, not
# a fabricated one: defect_known is True alongside 0.0.
self.assertAlmostEqual(metrics["first_pass_rate"], 1.0)
self.assertTrue(metrics["first_pass_known"])
self.assertAlmostEqual(metrics["defect_rate"], 0.0)
self.assertTrue(metrics["defect_known"])
self.assertAlmostEqual(metrics["rework_rate"], 0.0)
def test_old_task_without_ledger_does_not_dilute_real_rate(self):
from cloud.attempt_ledger import record_attempt
from cloud.db import connect as db_connect
conn = db_connect(self.db_path)
try:
conn.execute(
"INSERT INTO tasks(id, project_id, status) VALUES(?, ?, ?)",
("t-old", self.project["id"], "completed"))
conn.execute(
"INSERT INTO tasks(id, project_id, status) VALUES(?, ?, ?)",
("t-new", self.project["id"], "merged"))
conn.commit()
record_attempt(conn, "t-new", "failure", is_defect=True,
defect_category="contract")
record_attempt(conn, "t-new", "success")
finally:
conn.close()
status, report = self._req(
"GET", "/metrics-report", key=self.owner_key)
self.assertEqual(status, 200)
metrics = report["metrics"]
self.assertEqual(metrics["terminal_tasks"], 2)
# The old task is excluded from the denominator (known-only
# denominator, as in compute_metrics); it is neither counted
# as first-pass nor as defect-free.
self.assertAlmostEqual(metrics["first_pass_rate"], 0.0)
self.assertAlmostEqual(metrics["rework_rate"], 1.0)
self.assertAlmostEqual(metrics["defect_rate"], 1.0)
self.assertEqual(metrics["defect_by_category"],
{"contract": 1})
def test_interrupted_status_excluded_even_with_ledger_rows(self):
from cloud.attempt_ledger import record_attempt
from cloud.db import connect as db_connect
conn = db_connect(self.db_path)
try:
conn.execute(
"INSERT INTO tasks(id, project_id, status) VALUES(?, ?, ?)",
("t-int", self.project["id"], "interrupted"))
conn.commit()
record_attempt(conn, "t-int", "failure", is_defect=True,
defect_category="contract")
finally:
conn.close()
status, report = self._req(
"GET", "/metrics-report", key=self.owner_key)
self.assertEqual(status, 200)
metrics = report["metrics"]
self.assertEqual(metrics["terminal_tasks"], 0)
self.assertIsNone(metrics["first_pass_rate"])
self.assertFalse(metrics["first_pass_known"])
if __name__ == "__main__":
unittest.main()