Commit Graph
3 Commits
Author SHA1 Message Date
host 8b2686548c fix: serialize concurrent PG schema init with advisory lock
ci / test (push) Successful in 23m44s
Startup race: concurrent init_db (cloud-server + worker) ran
CREATE TABLE / CREATE INDEX IF NOT EXISTS against a DB where the
object did not exist yet; the check-then-create is not atomic, so
both sessions created it and one died with UniqueViolation on
pg_type_typname_nsp_index (tables) or pg_class_relname_nsp_index
(indexes). Hit on the 37ba687 and 7983567 deploys (7983567 added
idx_tasks_parent); systemd restart healed it every time.

Fix (advisory lock, not catch-duplicate_object): init_postgres_schema
takes pg_advisory_xact_lock (key 20261108, released at its commit),
and server.init_db takes the session-level lock for the whole PG
branch so the _ensure_* objects outside POSTGRES_SCHEMA
(task_continue_messages, user_gateway_keys, task_attempts,
task_audit_loop_state) are covered too. Same idiom as the existing
RATE_LIMIT_SWEEP_ADVISORY_KEY sweep lock; key is distinct.

Tests: two new real-PG regression tests (skipped without
CLOUD_TEST_PG_DSN), 4 processes racing on an empty schema, for
init_postgres_schema and for full init_db; both fail on the
unfixed code and pass with the fix.
2026-11-08 05:49:52 +00:00
host f16941347f fix: transplant PG init backfill rollback scope + audit-loop state upsert registration (code-to10 v2, from 854f810)
- db: translate INSERT OR IGNORE ... SELECT to ON CONFLICT DO NOTHING;
  register task_audit_loop_state (task_id) in _OR_REPLACE_PK (direct port).
- server: _backfill_default_quotas uses SAVEPOINT containment, no longer
  commits the caller's transaction (direct port; callers are init_db only).
- worker: NOT ported — integration _coord_child_rows has no column-fallback
  retry loop, so the InFailedSqlTransaction cascade site does not exist here.
- tests: rewritten for the integration base (no cloud.audit_loop /
  _coord_audit_loop_merge here); real-PG cases skip without DSN/driver.

(cherry picked from commit 79e6843610fcd275af874e7b4afc57af7ffa579e)
2026-11-04 21:03:32 +00:00
host b0347b374d fix: PG init backfill rollback scope + audit-loop state upsert registration
- db: translate INSERT OR IGNORE ... SELECT to ON CONFLICT DO NOTHING
  (VALUES-only regex left SELECT form untranslated -> PG syntax error);
  register task_audit_loop_state (task_id) in _OR_REPLACE_PK so PG
  INSERT OR REPLACE becomes a real upsert instead of DO NOTHING.
- server: _backfill_default_quotas uses SAVEPOINT containment and no
  longer commits the caller's transaction, so a backfill failure cannot
  roll back init_db's migrate ALTERs.
- worker: _coord_child_rows rolls back before column-fallback retries so
  a missing-column failure on PG does not cascade into
  InFailedSqlTransaction.
- tests: test_pg_init_fix.py (translation, SQLite backfill/upsert,
  real-PG fresh-init + merge + upsert + backfill containment, skipped
  without CLOUD_TEST_PG_DSN/driver).
2026-11-04 18:02:22 +00:00