FLAG-1 unified task_attempts schema (single canonical DDL + 3-shape migration)
User decision 2026-11-05: unified schema, rejecting the rehearsal runtime column-union workaround and a ledger rename. One definition (cloud.attempt_ledger.TASK_ATTEMPTS_DDL, 12 columns) shared by cloud.attempts and cloud.db.POSTGRES_SCHEMA; ensure builds the unified table fresh and migrates old resume, gap-3 ledger, and rehearsal shapes in place with rows intact. aggregate counts ledger rows only (outcome NULL = resume row, never counted). SQLite verified by 8 new tests; Postgres real-DB upgrade path explicitly NOT verified (see batch summary). Local commit only: no merge, no push, no deploy.
This commit is contained in:
1 parent
6d009af48f
commit
6a70775ad9
6 files changed
+467
-113
No files matched your search
+172
-73
@@ -67,10 +67,50 @@ Semantics (pinned by tests, not left to each caller):
|
||||
|
||||
Persistence notes:
|
||||
|
||||
- One table, one definition. ``task_attempts`` is defined ONCE
|
||||
below (``TASK_ATTEMPTS_DDL``, aliased as ``SQLITE_DDL`` /
|
||||
``PG_DDL``); ``cloud.attempts`` (the resume-history writer) and
|
||||
``cloud.db.POSTGRES_SCHEMA`` (the fresh-Postgres schema) both
|
||||
import this same string instead of carrying their own copy.
|
||||
The two historical shapes -- main's resume DDL (``kind`` /
|
||||
``interrupted_reason`` / ``checkpoint_round`` / ``result_text``
|
||||
/ ``error_text``) and gap 3's ledger DDL (``outcome`` /
|
||||
``is_defect`` / ``defect_category`` / ``corrects_attempt_no``)
|
||||
-- are superseded by this unified definition, which carries
|
||||
both column groups. The shared columns (``task_id``,
|
||||
``attempt_no``, ``created_at``) had identical meaning and type
|
||||
on both sides, so there is no semantic conflict to arbitrate:
|
||||
the union of the column sets IS the unified schema, with one
|
||||
deliberate relaxation (see below).
|
||||
- Row kinds in one table. A row is a ledger row iff ``outcome``
|
||||
is non-NULL (written by this module, outcome validated in
|
||||
Python before the insert); a row with ``outcome`` NULL is a
|
||||
resume-history row (written by ``cloud.attempts``). The two
|
||||
writers insert disjoint column groups and read by column name,
|
||||
so they share the table without overwriting each other's
|
||||
fields. ``aggregate_attempts`` counts ONLY ledger rows: a
|
||||
resume row (outcome NULL) is skipped, never counted as a
|
||||
defect-free first pass -- otherwise a task that was merely
|
||||
auto-resumed would silently gain a counted attempt.
|
||||
- The one deliberate relaxation: ``outcome`` is nullable in the
|
||||
unified DDL even though the ledger writer always supplies it.
|
||||
A ``NOT NULL`` on ``outcome`` would make the resume writer's
|
||||
insert (which has no outcome to report) fail on a fresh table,
|
||||
so the constraint lives in Python validation here, not in the
|
||||
DDL. ``is_defect`` keeps ``NOT NULL DEFAULT 0`` so a resume
|
||||
row that omits it still reads as "not a defect judgement".
|
||||
- SQLite: ``ensure_task_attempts_table`` runs
|
||||
``CREATE TABLE IF NOT EXISTS`` -- idempotent, safe to call on
|
||||
every write, and a no-op on a database that already has the
|
||||
table (including one that already has rows).
|
||||
``CREATE TABLE IF NOT EXISTS`` with the unified DDL, then
|
||||
migrates a pre-existing legacy table (old resume shape, gap-3
|
||||
ledger shape, or the rehearsal-era 12-column shape) by adding
|
||||
whichever unified columns it lacks. Added columns are always
|
||||
nullable or defaulted, so existing rows are never blocked;
|
||||
SQLite ``ALTER TABLE`` cannot tighten an existing column's
|
||||
nullability, so a migrated table keeps its looser column
|
||||
types -- the column SET and the data are unified, the
|
||||
``NOT NULL`` spelling on fresh tables is not retrofitted.
|
||||
This is a migration to the single canonical definition, not a
|
||||
second definition assembled at runtime.
|
||||
- Postgres: the production table is NOT created by
|
||||
``init_postgres_schema`` (deliberately: that schema is covered by
|
||||
the migrations test asserting ``MIGRATION_TABLES`` equals the
|
||||
@@ -87,7 +127,10 @@ Persistence notes:
|
||||
``test_attempt_ledger.py``). Until that manual step runs, the
|
||||
endpoint treats a missing table exactly like "no ledger rows":
|
||||
unknown rates, not an error -- the same fail-soft shape already
|
||||
used for missing ``cost_records``.
|
||||
used for missing ``cost_records``. The Postgres upgrade path
|
||||
for a legacy-shaped table (fresh ``PG_DDL`` vs. adding the
|
||||
missing unified columns) has NOT been verified against a real
|
||||
Postgres instance in this batch -- see the batch summary.
|
||||
- The read path (``GET /metrics-report``) never creates the table:
|
||||
a read-only endpoint must not mutate schema. Table creation
|
||||
happens only on the write path, inside ``record_attempt`` /
|
||||
@@ -104,18 +147,15 @@ Persistence notes:
|
||||
unknown until the hook is deliberately enabled. The worker hook
|
||||
always records ``is_defect=False``: it never judges a defect,
|
||||
so enabling it can never turn a transient retry into a defect.
|
||||
- Schema evolution note (pre-production, this track):
|
||||
``corrects_attempt_no`` was added to the DDL after audit gap 3
|
||||
but BEFORE either DDL reached production (gap 3 is still an
|
||||
OPEN, unmerged PR), so no production table needs migration:
|
||||
fresh tables get the column from the DDL below. A database
|
||||
created from gap 3's DDL alone (temporary/test databases only)
|
||||
gains the column with one statement --
|
||||
``ALTER TABLE task_attempts ADD COLUMN corrects_attempt_no
|
||||
INTEGER`` -- which is also the documented step if gap 3's DDL
|
||||
was ever run on production Postgres before this track lands.
|
||||
The read path uses ``SELECT *`` deliberately so a table without
|
||||
the column still reads (its rows aggregate as plain attempts).
|
||||
- Schema evolution note (unified schema, this batch):
|
||||
the old resume DDL, the gap-3 ledger DDL, and the rehearsal
|
||||
12-column shape are all legacy shapes of the SAME table.
|
||||
``ensure_task_attempts_table`` migrates any of them to the
|
||||
unified column set in place (add missing columns only, never
|
||||
drop or rewrite rows); the read path uses ``SELECT *``
|
||||
deliberately so rows of any shape still read, and
|
||||
``aggregate_attempts`` treats a row without an outcome as a
|
||||
resume row, not as a ledger attempt (see above).
|
||||
|
||||
This module imports no ``cloud.*`` module and opens no database
|
||||
connection itself: callers pass a connection in, so the layer rules
|
||||
@@ -130,38 +170,92 @@ TABLE = "task_attempts"
|
||||
|
||||
OUTCOMES = frozenset({"success", "failure", "interrupted"})
|
||||
|
||||
# One DDL serves both backends: every type and the composite key are
|
||||
# valid in SQLite and in Postgres, and there is no AUTOINCREMENT or
|
||||
# identity column for either backend's sequence handling to chase.
|
||||
# ``corrects_attempt_no`` (NULL = a real attempt row; non-NULL = a
|
||||
# correction row for that attempt_no, never counted as an attempt)
|
||||
# was added by the lifecycle-hook track; it SUPERSEDES gap 3's DDL
|
||||
# for any production manual step (gap 3 never merged, so no
|
||||
# production table exists yet -- see the module docstring).
|
||||
SQLITE_DDL = (
|
||||
# THE single canonical definition of task_attempts (unified
|
||||
# schema, FLAG-1 decision 2026-11-05: "unified schema", rejecting
|
||||
# both the rehearsal-era runtime column-union workaround and a
|
||||
# ledger rename). One DDL serves both backends: every type and
|
||||
# the composite key are valid in SQLite and in Postgres, and
|
||||
# there is no AUTOINCREMENT or identity column for either
|
||||
# backend's sequence handling to chase.
|
||||
#
|
||||
# Column groups, in order (each group's internal order preserves
|
||||
# its historical DDL order):
|
||||
# - shared: task_id, attempt_no (composite PK), created_at last;
|
||||
# - ledger (gap 3 + lifecycle hook): outcome, is_defect,
|
||||
# defect_category, corrects_attempt_no (NULL = a real attempt
|
||||
# row; non-NULL = a correction row for that attempt_no, never
|
||||
# counted as an attempt);
|
||||
# - resume (main v6): kind, interrupted_reason,
|
||||
# checkpoint_round, result_text, error_text.
|
||||
# ``outcome`` is deliberately nullable (see the module docstring:
|
||||
# the resume writer has no outcome to report, so a NOT NULL here
|
||||
# would break its insert; the ledger writer validates outcome in
|
||||
# Python instead). ``cloud.attempts.CREATE_TABLE_SQL`` and the
|
||||
# ``task_attempts`` entry of ``cloud.db.POSTGRES_SCHEMA`` are
|
||||
# this same string object, imported -- never a second copy.
|
||||
TASK_ATTEMPTS_DDL = (
|
||||
"CREATE TABLE IF NOT EXISTS task_attempts("
|
||||
"task_id TEXT NOT NULL, "
|
||||
"attempt_no INTEGER NOT NULL, "
|
||||
"outcome TEXT NOT NULL, "
|
||||
"outcome TEXT, "
|
||||
"is_defect INTEGER NOT NULL DEFAULT 0, "
|
||||
"defect_category TEXT, "
|
||||
"created_at DOUBLE PRECISION, "
|
||||
"corrects_attempt_no INTEGER, "
|
||||
"kind TEXT, "
|
||||
"interrupted_reason TEXT, "
|
||||
"checkpoint_round INTEGER, "
|
||||
"result_text TEXT, "
|
||||
"error_text TEXT, "
|
||||
"created_at DOUBLE PRECISION, "
|
||||
"PRIMARY KEY (task_id, attempt_no))"
|
||||
)
|
||||
|
||||
SQLITE_DDL = TASK_ATTEMPTS_DDL
|
||||
|
||||
# Production Postgres: run this verbatim by hand and verify the
|
||||
# table exists before expecting real rates from a Postgres-backed
|
||||
# endpoint. It is intentionally NOT part of POSTGRES_SCHEMA (see
|
||||
# the module docstring: migrations coverage must not widen silently).
|
||||
# This is the NEW DDL (with ``corrects_attempt_no``): if gap 3's
|
||||
# older DDL was ever run first, follow it with
|
||||
# ``ALTER TABLE task_attempts ADD COLUMN corrects_attempt_no
|
||||
# INTEGER`` and verify before expecting corrections to work.
|
||||
PG_DDL = SQLITE_DDL
|
||||
# For a legacy-shaped table already on Postgres, add whichever
|
||||
# unified columns are missing (see UNIFIED_COLUMN_TYPES);
|
||||
# that Postgres upgrade path is NOT yet verified against a real
|
||||
# Postgres instance -- see the batch summary.
|
||||
PG_DDL = TASK_ATTEMPTS_DDL
|
||||
|
||||
# One-line upgrade for a table created from gap 3's DDL (no
|
||||
# ``corrects_attempt_no`` column yet). Valid in both backends.
|
||||
# Unified column set, in DDL order. The migration in
|
||||
# ``ensure_task_attempts_table`` adds exactly these columns when
|
||||
# missing; the ALTER types below are the migration spellings
|
||||
# (nullable/defaulted, so existing rows are never blocked --
|
||||
# SQLite ALTER cannot add a NOT NULL column without a default
|
||||
# to a table that already has rows, and cannot tighten an
|
||||
# existing column at all).
|
||||
UNIFIED_COLUMNS = (
|
||||
"task_id", "attempt_no", "outcome", "is_defect",
|
||||
"defect_category", "corrects_attempt_no", "kind",
|
||||
"interrupted_reason", "checkpoint_round", "result_text",
|
||||
"error_text", "created_at",
|
||||
)
|
||||
|
||||
UNIFIED_COLUMN_TYPES = {
|
||||
"task_id": "TEXT",
|
||||
"attempt_no": "INTEGER",
|
||||
"outcome": "TEXT",
|
||||
"is_defect": "INTEGER DEFAULT 0",
|
||||
"defect_category": "TEXT",
|
||||
"corrects_attempt_no": "INTEGER",
|
||||
"kind": "TEXT",
|
||||
"interrupted_reason": "TEXT",
|
||||
"checkpoint_round": "INTEGER",
|
||||
"result_text": "TEXT",
|
||||
"error_text": "TEXT",
|
||||
"created_at": "DOUBLE PRECISION",
|
||||
}
|
||||
|
||||
# Legacy one-line upgrade for a table created from gap 3's DDL
|
||||
# (no ``corrects_attempt_no`` column yet). Retained for
|
||||
# backward compatibility with existing callers/tests; it is a
|
||||
# single step of the general migration above, not a second
|
||||
# schema definition. Valid in both backends.
|
||||
OLD_TABLE_ALTER = (
|
||||
"ALTER TABLE task_attempts "
|
||||
"ADD COLUMN corrects_attempt_no INTEGER"
|
||||
@@ -179,39 +273,34 @@ def _existing_columns(conn, table):
|
||||
|
||||
|
||||
def ensure_task_attempts_table(conn):
|
||||
"""Create the ledger table if absent. Idempotent; does not
|
||||
touch existing rows. The caller owns the commit (``record_attempt``
|
||||
"""Create the unified table if absent; migrate a legacy
|
||||
table to it if present. Idempotent; does not touch existing
|
||||
rows. The caller owns the commit (``record_attempt``
|
||||
commits; a bare ensure inside a larger transaction does not).
|
||||
|
||||
演练注 FLAG(合入语义取舍点,主结论见演练报告):
|
||||
main 侧 init_db/worker 用 cloud.attempts.py 的旧 DDL 建同名
|
||||
task_attempts 表(kind/interrupted_reason/checkpoint_round/
|
||||
result_text/error_text,无 outcome 系列);本模块的 CREATE
|
||||
TABLE IF NOT EXISTS 在旧表已存在时是空操作,之后按台账
|
||||
列写行即报 no column named outcome。本解法把表升级为两
|
||||
组列的并集:旧表先建时补台账 4 列,台账表先建时补旧 5
|
||||
列(新增列一律可空/带默认值,绝不对已有行加 NOT NULL,
|
||||
否则旧行会挡住 ALTER)。两边读路径均按列名取数,互不
|
||||
干扰;但这只是演练期保守解法——真合前须用户定:并集表、
|
||||
台账改名、或统一 schema 三选一,不可悄悄定为并集表。
|
||||
Unified schema (FLAG-1, user decision 2026-11-05): there is
|
||||
exactly one ``task_attempts`` definition
|
||||
(``TASK_ATTEMPTS_DDL``). A fresh database gets it verbatim
|
||||
from ``CREATE TABLE IF NOT EXISTS``. A database whose table
|
||||
was built from any legacy shape -- the old resume DDL, the
|
||||
gap-3 ledger DDL, or the rehearsal-era 12-column shape --
|
||||
is migrated in place by adding whichever unified columns it
|
||||
lacks (``UNIFIED_COLUMN_TYPES``); added columns are always
|
||||
nullable or defaulted, so existing rows are never blocked,
|
||||
and no row is dropped or rewritten. This replaces the
|
||||
rehearsal-era runtime column-completion workaround: the
|
||||
target here is the single canonical definition, and
|
||||
``cloud.attempts`` / ``cloud.db`` share that same string
|
||||
instead of each carrying a DDL of their own.
|
||||
"""
|
||||
conn.execute(SQLITE_DDL)
|
||||
conn.execute(TASK_ATTEMPTS_DDL)
|
||||
existing = _existing_columns(conn, "task_attempts")
|
||||
for col, ddl in (
|
||||
("outcome", "TEXT"),
|
||||
("is_defect", "INTEGER DEFAULT 0"),
|
||||
("defect_category", "TEXT"),
|
||||
("corrects_attempt_no", "INTEGER"),
|
||||
("kind", "TEXT"),
|
||||
("interrupted_reason", "TEXT"),
|
||||
("checkpoint_round", "INTEGER"),
|
||||
("result_text", "TEXT"),
|
||||
("error_text", "TEXT")):
|
||||
for col in UNIFIED_COLUMNS:
|
||||
if col not in existing:
|
||||
try:
|
||||
conn.execute(
|
||||
"ALTER TABLE task_attempts ADD COLUMN %s %s"
|
||||
% (col, ddl))
|
||||
% (col, UNIFIED_COLUMN_TYPES[col]))
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
@@ -272,11 +361,11 @@ def record_correction(conn, task_id, attempt_no, is_defect,
|
||||
correction row, a target whose outcome is ``interrupted``
|
||||
(interrupted rows are never defects, so there is no defect
|
||||
judgement to correct), and a defect category without a defect
|
||||
flag all raise ``ValueError`` and write nothing. On a table
|
||||
created from gap 3's DDL (no ``corrects_attempt_no`` column),
|
||||
the INSERT raises the backend's OperationalError loudly --
|
||||
run ``OLD_TABLE_ALTER`` first (see the module docstring);
|
||||
a correction must never silently degrade into a plain row.
|
||||
flag all raise ``ValueError`` and write nothing. A table
|
||||
created from a legacy DDL is migrated by ``ensure`` first
|
||||
(above), so the ``corrects_attempt_no`` column is present
|
||||
before the INSERT runs; a correction must never silently
|
||||
degrade into a plain row.
|
||||
Returns the correction row's own ``attempt_no``.
|
||||
"""
|
||||
if not task_id:
|
||||
@@ -325,12 +414,13 @@ def read_attempt_rows(conn):
|
||||
OperationalError when the table does not exist yet; the
|
||||
endpoint caller treats that as "no ledger", not as a failure.
|
||||
|
||||
``SELECT *`` is deliberate: a table created from gap 3's DDL
|
||||
has no ``corrects_attempt_no`` column, and naming that column
|
||||
here would turn those readable rows into an endpoint error.
|
||||
``SELECT *`` is deliberate: a legacy-shaped table may lack
|
||||
columns the unified DDL has, and naming a missing column
|
||||
here would turn readable rows into an endpoint error.
|
||||
With ``*`` the rows still read, and ``aggregate_attempts``
|
||||
treats a missing ``corrects_attempt_no`` as a plain attempt
|
||||
row (see ``_field_optional``).
|
||||
row (see ``_field_optional``) and a missing/NULL ``outcome``
|
||||
as a resume row, not a ledger attempt.
|
||||
"""
|
||||
return conn.execute(
|
||||
"SELECT * FROM task_attempts "
|
||||
@@ -364,10 +454,14 @@ def aggregate_attempts(rows):
|
||||
|
||||
Interrupted rows are excluded from both counts, and a task
|
||||
with only interrupted rows is omitted entirely (no counted
|
||||
attempt => its rates stay unknown, never 0). Correction rows
|
||||
(``corrects_attempt_no`` set) are excluded from the attempt
|
||||
count and instead OVERWRITE the defect judgement of their
|
||||
target attempt, in attempt order, last-wins: a task's
|
||||
attempt => its rates stay unknown, never 0). Resume rows
|
||||
(``outcome`` NULL -- written by ``cloud.attempts``, which
|
||||
records an interrupted attempt's history, not a ledger
|
||||
outcome) are likewise excluded: they are not ledger attempts
|
||||
and must never inflate a task's counted attempts. Correction
|
||||
rows (``corrects_attempt_no`` set) are excluded from the
|
||||
attempt count and instead OVERWRITE the defect judgement of
|
||||
their target attempt, in attempt order, last-wins: a task's
|
||||
``defects`` is the number of its counted attempts whose
|
||||
judgement AFTER corrections is a defect, and its
|
||||
``defect_category`` is the category of its FIRST defect
|
||||
@@ -392,7 +486,12 @@ def aggregate_attempts(rows):
|
||||
bool(_field(row, "is_defect")),
|
||||
_field(row, "defect_category")]
|
||||
continue
|
||||
if _field(row, "outcome") == "interrupted":
|
||||
outcome = _field_optional(row, "outcome")
|
||||
if outcome is None:
|
||||
# Resume-history row (cloud.attempts): no ledger
|
||||
# outcome, so not a counted ledger attempt.
|
||||
continue
|
||||
if outcome == "interrupted":
|
||||
continue
|
||||
entries = judged.setdefault(task_id, {})
|
||||
entries[_field(row, "attempt_no")] = [
|
||||
|
||||
+18
-14
@@ -31,7 +31,13 @@ host.py、不改 codex 线程恢复:
|
||||
result_text / error_text 各截断 2000 字。记账与回 queued
|
||||
在同一事务里,记账失败不单独炸恢复循环(外层已兜底),
|
||||
但同一 (task_id, attempt_no) 重复记时 INSERT OR IGNORE
|
||||
保留第一次,不覆盖。
|
||||
保留第一次,不覆盖。表定义是统一 schema(FLAG-1,
|
||||
2026-11-05):本模块不自带一份 DDL,CREATE_TABLE_SQL
|
||||
与建表/迁移均取自 ``cloud.attempt_ledger`` 的
|
||||
``TASK_ATTEMPTS_DDL`` / ``ensure_task_attempts_table``,
|
||||
与台账写路径共用同一定义;本模块写的行 outcome 为
|
||||
NULL(恢复历史行,非台账计数行,见 attempt_ledger 对
|
||||
aggregate 的口径)。
|
||||
② 重跑前言:非 goal 平台全新重排的任务在认领时,worker
|
||||
在内存副本的 prompt 前贴一段上次尝试说明(原 prompt
|
||||
不落库,与项目/待办/记忆注入同款只改内存副本),内容
|
||||
@@ -63,25 +69,23 @@ host.py、不改 codex 线程恢复:
|
||||
|
||||
import time
|
||||
|
||||
from cloud.attempt_ledger import TASK_ATTEMPTS_DDL
|
||||
from cloud.attempt_ledger import ensure_task_attempts_table as _ensure_unified
|
||||
|
||||
ATTEMPT_TEXT_CHARS = 2000
|
||||
PREAMBLE_TEXT_CHARS = 1000
|
||||
|
||||
CREATE_TABLE_SQL = (
|
||||
"CREATE TABLE IF NOT EXISTS task_attempts("
|
||||
"task_id TEXT, "
|
||||
"attempt_no INTEGER, "
|
||||
"kind TEXT, "
|
||||
"interrupted_reason TEXT, "
|
||||
"checkpoint_round INTEGER, "
|
||||
"result_text TEXT, "
|
||||
"error_text TEXT, "
|
||||
"created_at DOUBLE PRECISION, "
|
||||
"PRIMARY KEY(task_id, attempt_no))")
|
||||
# 统一 schema:与台账共用同一定义,绝不在此另写一份 DDL
|
||||
# (历史教训:两份 DDL 同名异构,CREATE TABLE IF NOT EXISTS
|
||||
# 使后建的一方成空操作、写行报缺列)。别名保留,供旧调用
|
||||
# 方按名取定义;建表与老表迁移一律走台账的 ensure。
|
||||
CREATE_TABLE_SQL = TASK_ATTEMPTS_DDL
|
||||
|
||||
|
||||
def ensure_attempts_table(conn):
|
||||
"""幂等建尝试账本表(server/worker 两支共用同一 DDL)。"""
|
||||
conn.execute(CREATE_TABLE_SQL)
|
||||
"""幂等建尝试账本表(server/worker 两支共用同一 DDL):
|
||||
新库建统一表,老库迁移到统一列集(见 attempt_ledger)。"""
|
||||
_ensure_unified(conn)
|
||||
|
||||
|
||||
def _trunc(text, limit):
|
||||
|
||||
+11
-12
@@ -65,6 +65,12 @@ import time
|
||||
from abc import ABC, abstractmethod
|
||||
from contextlib import contextmanager
|
||||
|
||||
# 统一 schema:task_attempts 的 PG 新库定义与台账定义是同一
|
||||
# 字符串(cloud.attempt_ledger.TASK_ATTEMPTS_DDL),不在此另
|
||||
# 写一份。attempt_ledger 只依赖标准库 time,与本模块无循环
|
||||
# 导入;层规则上它也不属于 orchestration 模块集合。
|
||||
from cloud.attempt_ledger import TASK_ATTEMPTS_DDL as _TASK_ATTEMPTS_DDL
|
||||
|
||||
# 与 server.py 旧 _connect 的 timeout=30s 保持一致的连接级等待;
|
||||
# PRAGMA busy_timeout 是语句级等待,两者不冲突,后者以毫秒为单位
|
||||
# 在每个连接上单独设置(不随库文件持久化,见 connect 的注释)。
|
||||
@@ -876,18 +882,11 @@ POSTGRES_SCHEMA = (
|
||||
"created_at DOUBLE PRECISION)",
|
||||
"CREATE INDEX IF NOT EXISTS idx_task_steps_task"
|
||||
" ON task_steps(task_id, step_no)",
|
||||
# 尝试账本(v6,见 cloud/attempts.py):(task_id, attempt_no)
|
||||
# 唯一;PG 新库一次建齐,老库走 server/worker ensure。
|
||||
"CREATE TABLE IF NOT EXISTS task_attempts("
|
||||
"task_id TEXT, "
|
||||
"attempt_no INTEGER, "
|
||||
"kind TEXT, "
|
||||
"interrupted_reason TEXT, "
|
||||
"checkpoint_round INTEGER, "
|
||||
"result_text TEXT, "
|
||||
"error_text TEXT, "
|
||||
"created_at DOUBLE PRECISION, "
|
||||
"PRIMARY KEY(task_id, attempt_no))",
|
||||
# 尝试账本(统一 schema,见 cloud/attempt_ledger.py):
|
||||
# (task_id, attempt_no) 唯一;PG 新库一次建齐,老库走
|
||||
# server/worker ensure 迁移到同一列集。此处是导入的统
|
||||
# 一定义本身,不是第二份 DDL。
|
||||
_TASK_ATTEMPTS_DDL,
|
||||
# 审计循环状态(audit loop):task_id 单行主键;PG 新库一次
|
||||
# 建齐,老库走 server/worker ensure(见 cloud/audit_loop.py)。
|
||||
"CREATE TABLE IF NOT EXISTS task_audit_loop_state("
|
||||
|
||||
+17
-12
@@ -21,9 +21,10 @@
|
||||
报告:表齐但钩子关 ≠ 就绪,钩子开但表不齐 ≠ 就绪。
|
||||
3. DDL 清单不另造:清单直接引用 attempt_ledger.PG_DDL
|
||||
与 attempt_hook.PG_MARKER_DDL 两个常量(本模块不
|
||||
复制 DDL 文本,复制就会漂移)。另含 gap 3 旧表升级句
|
||||
attempt_ledger.OLD_TABLE_ALTER(仅当旧 DDL 建过表时
|
||||
需要;gap 3 未合并,生产尚无旧表,属防御项)。
|
||||
复制 DDL 文本,复制就会漂移)。老形态表升级走
|
||||
attempt_ledger.ensure_task_attempts_table 的统一列补齐
|
||||
(仅当老形态建过表时需要;属防御项,PG 真实库升级
|
||||
路径未实证)。
|
||||
4. 校验函数 verify_ledger_ready(conn):只读探针——查两
|
||||
张表的存在与必需列,不建表、不写行(读路径绝不改
|
||||
schema,与 metrics-report 端点的只读纪律一致)。
|
||||
@@ -42,11 +43,14 @@ from cloud.attempt_hook import PG_MARKER_DDL as _PG_MARKER_DDL
|
||||
from cloud.attempt_ledger import OLD_TABLE_ALTER as _OLD_TABLE_ALTER
|
||||
from cloud.attempt_ledger import PG_DDL as _LEDGER_PG_DDL
|
||||
|
||||
# 必需列:缺任一列即视为表不齐(旧 DDL 建的表缺
|
||||
# corrects_attempt_no 时走 OLD_TABLE_ALTER 补列)。
|
||||
# 必需列:统一 schema 的全部 12 列;缺任一列即视为表
|
||||
# 不齐(老形态建的表由 ensure_task_attempts_table 补齐
|
||||
# 缺失列后复核,不再只补 corrects_attempt_no 一列)。
|
||||
LEDGER_REQUIRED_COLUMNS = (
|
||||
"task_id", "attempt_no", "outcome", "is_defect",
|
||||
"defect_category", "created_at", "corrects_attempt_no",
|
||||
"defect_category", "corrects_attempt_no", "kind",
|
||||
"interrupted_reason", "checkpoint_round", "result_text",
|
||||
"error_text", "created_at",
|
||||
)
|
||||
MARKER_REQUIRED_COLUMNS = ("task_id", "origin", "queued_at")
|
||||
|
||||
@@ -72,12 +76,13 @@ LEDGER_DDL_CHECKLIST = (
|
||||
},
|
||||
{
|
||||
"step": 3,
|
||||
"action": "仅当表由 gap 3 旧 DDL 建过时:执行"
|
||||
" attempt_ledger.OLD_TABLE_ALTER 补"
|
||||
" corrects_attempt_no 列(gap 3 未合并,"
|
||||
"生产正常路径跳过此步)",
|
||||
"ddl_source": "cloud.attempt_ledger.OLD_TABLE_ALTER",
|
||||
"verify": "task_attempts 含 corrects_attempt_no 列",
|
||||
"action": "仅当表已由老形态(旧恢复 DDL、gap 3 台账 DDL"
|
||||
" 或演练形态)建过时:经 attempt_ledger."
|
||||
"ensure_task_attempts_table 补齐缺失的统一列"
|
||||
"(SQLite 已实测;PG 真实库升级路径未实证,"
|
||||
"生产 PG 上执行前须先在真实 PG 上验证)",
|
||||
"ddl_source": "cloud.attempt_ledger.TASK_ATTEMPTS_DDL",
|
||||
"verify": "task_attempts 含统一 schema 全部 12 列",
|
||||
},
|
||||
{
|
||||
"step": 4,
|
||||
|
||||
@@ -0,0 +1,236 @@
|
||||
"""Unified task_attempts schema (FLAG-1, user decision 2026-11-05).
|
||||
|
||||
Pins the single-definition rule and the three-shape migration:
|
||||
|
||||
- one DDL: attempt_ledger.TASK_ATTEMPTS_DDL is the same string
|
||||
used by cloud.attempts.CREATE_TABLE_SQL and by the
|
||||
task_attempts entry of cloud.db.POSTGRES_SCHEMA;
|
||||
- fresh database: ensure builds exactly the 12 unified columns;
|
||||
- old resume shape, gap-3 ledger shape, and the rehearsal-era
|
||||
12-column shape all migrate in place with rows intact, and
|
||||
both writers (resume + ledger) work on the migrated table;
|
||||
- aggregation counts ledger rows only: a resume row (outcome
|
||||
NULL) never becomes a counted, defect-free attempt.
|
||||
|
||||
SQLite only, run live. The Postgres upgrade path is NOT proven
|
||||
here (adapter-level DDL text check only, same as the other
|
||||
ledger tests); the batch summary states that explicitly.
|
||||
"""
|
||||
|
||||
import os
|
||||
import sqlite3
|
||||
import sys
|
||||
import unittest
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(
|
||||
os.path.dirname(os.path.abspath(__file__)))))
|
||||
|
||||
from cloud.attempt_ledger import ( # noqa: E402
|
||||
PG_DDL,
|
||||
SQLITE_DDL,
|
||||
TASK_ATTEMPTS_DDL,
|
||||
UNIFIED_COLUMNS,
|
||||
aggregate_attempts,
|
||||
ensure_task_attempts_table,
|
||||
read_attempt_rows,
|
||||
record_attempt as ledger_record,
|
||||
record_correction as ledger_correct,
|
||||
)
|
||||
from cloud.attempts import ( # noqa: E402
|
||||
CREATE_TABLE_SQL,
|
||||
ensure_attempts_table,
|
||||
latest_attempt,
|
||||
record_attempt as resume_record,
|
||||
)
|
||||
from cloud.db import POSTGRES_SCHEMA # noqa: E402
|
||||
from cloud.db import _translate_sql # noqa: E402
|
||||
|
||||
OLD_DDL = (
|
||||
"CREATE TABLE task_attempts("
|
||||
"task_id TEXT, attempt_no INTEGER, kind TEXT, "
|
||||
"interrupted_reason TEXT, checkpoint_round INTEGER, "
|
||||
"result_text TEXT, error_text TEXT, "
|
||||
"created_at DOUBLE PRECISION, "
|
||||
"PRIMARY KEY(task_id, attempt_no))")
|
||||
|
||||
GAP3_DDL = (
|
||||
"CREATE TABLE task_attempts("
|
||||
"task_id TEXT NOT NULL, attempt_no INTEGER NOT NULL, "
|
||||
"outcome TEXT NOT NULL, is_defect INTEGER NOT NULL DEFAULT 0, "
|
||||
"defect_category TEXT, created_at DOUBLE PRECISION, "
|
||||
"PRIMARY KEY (task_id, attempt_no))")
|
||||
|
||||
# Rehearsal-era shape: same 12 column names as unified, but the
|
||||
# looser rehearsal types (outcome nullable, is_defect without
|
||||
# NOT NULL, PK columns without NOT NULL spelling).
|
||||
REHEARSAL_DDL = (
|
||||
"CREATE TABLE task_attempts("
|
||||
"task_id TEXT, attempt_no INTEGER, outcome TEXT, "
|
||||
"is_defect INTEGER DEFAULT 0, defect_category TEXT, "
|
||||
"corrects_attempt_no INTEGER, kind TEXT, "
|
||||
"interrupted_reason TEXT, checkpoint_round INTEGER, "
|
||||
"result_text TEXT, error_text TEXT, "
|
||||
"created_at DOUBLE PRECISION, "
|
||||
"PRIMARY KEY(task_id, attempt_no))")
|
||||
|
||||
|
||||
def _columns(conn):
|
||||
return [row[1] for row in conn.execute(
|
||||
"PRAGMA table_info(task_attempts)").fetchall()]
|
||||
|
||||
|
||||
def _row_by_key(conn, task_id, attempt_no):
|
||||
row = conn.execute(
|
||||
"SELECT * FROM task_attempts WHERE task_id = ? "
|
||||
"AND attempt_no = ?", (task_id, attempt_no)).fetchone()
|
||||
cols = [d[0] for d in conn.execute(
|
||||
"SELECT * FROM task_attempts LIMIT 0").description]
|
||||
return dict(zip(cols, row)) if row is not None else None
|
||||
|
||||
|
||||
class TestSingleDefinition(unittest.TestCase):
|
||||
def test_one_ddl_string_everywhere(self):
|
||||
self.assertEqual(SQLITE_DDL, TASK_ATTEMPTS_DDL)
|
||||
self.assertEqual(PG_DDL, TASK_ATTEMPTS_DDL)
|
||||
self.assertEqual(CREATE_TABLE_SQL, TASK_ATTEMPTS_DDL)
|
||||
schema_statements = [s for s in POSTGRES_SCHEMA
|
||||
if "task_attempts" in s]
|
||||
self.assertEqual(len(schema_statements), 1)
|
||||
self.assertEqual(schema_statements[0], TASK_ATTEMPTS_DDL)
|
||||
|
||||
def test_unified_column_set(self):
|
||||
self.assertEqual(tuple(UNIFIED_COLUMNS), (
|
||||
"task_id", "attempt_no", "outcome", "is_defect",
|
||||
"defect_category", "corrects_attempt_no", "kind",
|
||||
"interrupted_reason", "checkpoint_round", "result_text",
|
||||
"error_text", "created_at"))
|
||||
|
||||
def test_pg_ddl_text_is_static_only(self):
|
||||
translated, _kind = _translate_sql(PG_DDL)
|
||||
for col in UNIFIED_COLUMNS:
|
||||
self.assertIn(col, translated)
|
||||
self.assertIn("PRIMARY KEY (task_id, attempt_no)", translated)
|
||||
self.assertNotIn("AUTOINCREMENT", translated)
|
||||
self.assertNotIn("GENERATED", translated)
|
||||
|
||||
|
||||
class TestFreshAndMigrations(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.conn = sqlite3.connect(":memory:")
|
||||
self.conn.row_factory = sqlite3.Row
|
||||
|
||||
def tearDown(self):
|
||||
self.conn.close()
|
||||
|
||||
def test_fresh_build_has_exact_unified_columns(self):
|
||||
ensure_task_attempts_table(self.conn)
|
||||
self.assertEqual(_columns(self.conn), list(UNIFIED_COLUMNS))
|
||||
# Idempotent, rows preserved.
|
||||
ledger_record(self.conn, "t-1", "success")
|
||||
ensure_task_attempts_table(self.conn)
|
||||
ensure_attempts_table(self.conn)
|
||||
self.assertEqual(_columns(self.conn), list(UNIFIED_COLUMNS))
|
||||
self.assertEqual(self.conn.execute(
|
||||
"SELECT COUNT(*) FROM task_attempts").fetchone()[0], 1)
|
||||
|
||||
def test_old_shape_migrates_with_rows_intact(self):
|
||||
self.conn.execute(OLD_DDL)
|
||||
self.conn.execute(
|
||||
"INSERT INTO task_attempts(task_id, attempt_no, kind, "
|
||||
"interrupted_reason, checkpoint_round, result_text, "
|
||||
"error_text, created_at) VALUES(?, ?, ?, ?, ?, ?, ?, ?)",
|
||||
("t-old", 1, "platform_fresh", "worker_shutdown", -1,
|
||||
"partial output", "killed", 1.0))
|
||||
self.conn.commit()
|
||||
before = _row_by_key(self.conn, "t-old", 1)
|
||||
ensure_task_attempts_table(self.conn)
|
||||
# Migrated tables keep ALTER-append order, not fresh DDL
|
||||
# order; every read path is by column name, so the set
|
||||
# is the invariant, not the order (fresh tables pin order).
|
||||
self.assertEqual(set(_columns(self.conn)),
|
||||
set(UNIFIED_COLUMNS))
|
||||
after = _row_by_key(self.conn, "t-old", 1)
|
||||
for col in ("task_id", "attempt_no", "kind",
|
||||
"interrupted_reason", "checkpoint_round",
|
||||
"result_text", "error_text", "created_at"):
|
||||
self.assertEqual(after[col], before[col], col)
|
||||
self.assertIsNone(after["outcome"])
|
||||
# Both writers now work on the migrated table.
|
||||
resume_record(self.conn, "t-old", 2, "platform_fresh",
|
||||
"worker_shutdown", -1, "second", None)
|
||||
ledger_record(self.conn, "t-led", "failure", is_defect=True,
|
||||
defect_category="contract")
|
||||
self.conn.commit()
|
||||
latest = latest_attempt(self.conn, "t-old")
|
||||
self.assertEqual(latest["attempt_no"], 2)
|
||||
self.assertEqual(latest["kind"], "platform_fresh")
|
||||
# Aggregation counts the ledger row only, never the
|
||||
# resume rows (outcome NULL).
|
||||
agg = aggregate_attempts(read_attempt_rows(self.conn))
|
||||
self.assertEqual(agg, {"t-led": {
|
||||
"attempts": 1, "defects": 1,
|
||||
"defect_category": "contract"}})
|
||||
|
||||
def test_gap3_shape_migrates_and_corrections_work(self):
|
||||
self.conn.execute(GAP3_DDL)
|
||||
self.conn.execute(
|
||||
"INSERT INTO task_attempts(task_id, attempt_no, outcome, "
|
||||
"is_defect, defect_category, created_at) "
|
||||
"VALUES('t-gap', 1, 'success', 0, NULL, 1.0)")
|
||||
self.conn.commit()
|
||||
ensure_task_attempts_table(self.conn)
|
||||
self.assertEqual(set(_columns(self.conn)),
|
||||
set(UNIFIED_COLUMNS))
|
||||
row = _row_by_key(self.conn, "t-gap", 1)
|
||||
self.assertEqual(row["outcome"], "success")
|
||||
ledger_correct(self.conn, "t-gap", 1, is_defect=True,
|
||||
defect_category="contract")
|
||||
agg = aggregate_attempts(read_attempt_rows(self.conn))
|
||||
self.assertEqual(agg["t-gap"]["defects"], 1)
|
||||
|
||||
def test_rehearsal_shape_migrates_with_rows_intact(self):
|
||||
self.conn.execute(REHEARSAL_DDL)
|
||||
self.conn.execute(
|
||||
"INSERT INTO task_attempts(task_id, attempt_no, outcome, "
|
||||
"is_defect, defect_category, corrects_attempt_no, kind, "
|
||||
"interrupted_reason, checkpoint_round, result_text, "
|
||||
"error_text, created_at) VALUES(?, ?, ?, ?, ?, ?, ?, ?, "
|
||||
"?, ?, ?, ?)",
|
||||
("t-reh", 1, "failure", 1, "contract", None,
|
||||
"platform_fresh", "worker_shutdown", -1, "out", "err",
|
||||
2.0))
|
||||
self.conn.commit()
|
||||
before = _row_by_key(self.conn, "t-reh", 1)
|
||||
ensure_task_attempts_table(self.conn)
|
||||
self.assertEqual(_columns(self.conn), list(UNIFIED_COLUMNS))
|
||||
after = _row_by_key(self.conn, "t-reh", 1)
|
||||
self.assertEqual(after, before)
|
||||
# Both writers still work after the migration.
|
||||
resume_record(self.conn, "t-reh", 2, "goal_breakpoint",
|
||||
"paused", 3, "r", "e")
|
||||
ledger_record(self.conn, "t-reh2", "success")
|
||||
agg = aggregate_attempts(read_attempt_rows(self.conn))
|
||||
self.assertEqual(agg["t-reh"]["defects"], 1)
|
||||
self.assertEqual(agg["t-reh2"]["attempts"], 1)
|
||||
|
||||
def test_resume_rows_never_count_in_aggregation(self):
|
||||
# Pure aggregation: a NULL-outcome row is a resume row.
|
||||
agg = aggregate_attempts([
|
||||
{"task_id": "t-1", "attempt_no": 1, "outcome": None,
|
||||
"is_defect": 0, "defect_category": None,
|
||||
"corrects_attempt_no": None},
|
||||
{"task_id": "t-1", "attempt_no": 2, "outcome": "success",
|
||||
"is_defect": 0, "defect_category": None,
|
||||
"corrects_attempt_no": None},
|
||||
])
|
||||
self.assertEqual(agg["t-1"]["attempts"], 1)
|
||||
agg_only_resume = aggregate_attempts([
|
||||
{"task_id": "t-2", "attempt_no": 1, "outcome": None,
|
||||
"is_defect": 0, "defect_category": None,
|
||||
"corrects_attempt_no": None},
|
||||
])
|
||||
self.assertNotIn("t-2", agg_only_resume)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -223,9 +223,14 @@ class LedgerOnboardingGapTestCase(unittest.TestCase):
|
||||
finally:
|
||||
conn.close()
|
||||
self.assertFalse(out["tables_ready"])
|
||||
# Unified schema: the required set is all 12 columns, so a
|
||||
# gap-3-shaped table is missing corrects_attempt_no AND
|
||||
# the five resume columns, not just the one column.
|
||||
self.assertEqual(
|
||||
out["missing"],
|
||||
["task_attempts missing columns: corrects_attempt_no"])
|
||||
["task_attempts missing columns: corrects_attempt_no, "
|
||||
"kind, interrupted_reason, checkpoint_round, "
|
||||
"result_text, error_text"])
|
||||
|
||||
def test_verify_marker_missing_column(self):
|
||||
conn = self._sqlite(
|
||||
@@ -237,9 +242,15 @@ class LedgerOnboardingGapTestCase(unittest.TestCase):
|
||||
finally:
|
||||
conn.close()
|
||||
self.assertFalse(out["tables_ready"])
|
||||
# Unified schema: the gap-3-shaped ledger table is also
|
||||
# missing the five resume columns, reported before the
|
||||
# marker table's own missing column.
|
||||
self.assertEqual(
|
||||
out["missing"],
|
||||
["task_attempt_origin_markers missing columns: queued_at"])
|
||||
["task_attempts missing columns: kind, "
|
||||
"interrupted_reason, checkpoint_round, result_text, "
|
||||
"error_text",
|
||||
"task_attempt_origin_markers missing columns: queued_at"])
|
||||
|
||||
|
||||
class LedgerOutboxGapTestCase(unittest.TestCase):
|
||||
|
||||
Reference in new issue
Block a user