fix: serialize concurrent PG schema init with advisory lock
ci / test (push) Successful in 23m44s

Startup race: concurrent init_db (cloud-server + worker) ran
CREATE TABLE / CREATE INDEX IF NOT EXISTS against a DB where the
object did not exist yet; the check-then-create is not atomic, so
both sessions created it and one died with UniqueViolation on
pg_type_typname_nsp_index (tables) or pg_class_relname_nsp_index
(indexes). Hit on the 37ba687 and 7983567 deploys (7983567 added
idx_tasks_parent); systemd restart healed it every time.

Fix (advisory lock, not catch-duplicate_object): init_postgres_schema
takes pg_advisory_xact_lock (key 20261108, released at its commit),
and server.init_db takes the session-level lock for the whole PG
branch so the _ensure_* objects outside POSTGRES_SCHEMA
(task_continue_messages, user_gateway_keys, task_attempts,
task_audit_loop_state) are covered too. Same idiom as the existing
RATE_LIMIT_SWEEP_ADVISORY_KEY sweep lock; key is distinct.

Tests: two new real-PG regression tests (skipped without
CLOUD_TEST_PG_DSN), 4 processes racing on an empty schema, for
init_postgres_schema and for full init_db; both fail on the
unfixed code and pass with the fix.
This commit is contained in:
host committed 2026-11-08 05:49:52 +00:00
1 parent 7983567491
commit 8b2686548c
4 files changed
+238 -16

No files matched your search

+55
View File
@@ -1165,6 +1165,49 @@ POSTGRES_SCHEMA = (
)
# 启动期 schema 初始化 advisory lock key(Postgres):
# 并发启动(cloud-server 与 worker 同时 init_db)时,CREATE TABLE /
# CREATE INDEX 的 IF NOT EXISTS 检查与建对象不是原子的——两个会话
# 都查到“不存在”后同时建,会撞 pg_type_typname_nsp_index(建表)
# 或 pg_class_relname_nsp_index(建索引)的 UniqueViolation,启动
# 即崩、靠 systemd 重启才自愈(37ba687 与 7983567 两次部署均撞上;
# 7983567 新增的 idx_tasks_parent 是当时的新对象)。
# 持锁串行化整个初始化:init_postgres_schema 内持事务级锁(随本
# 函数末尾的 commit 释放),server.init_db 另持会话级锁覆盖其后
# 的 _ensure_*(task_continue_messages / user_gateway_keys /
# task_attempts / task_audit_loop_state 不在 POSTGRES_SCHEMA 里,同样
# 会撞)。key 须全项目唯一:限流清理另有
# RATE_LIMIT_SWEEP_ADVISORY_KEY = 20261027(cloud/server.py),
# 两处绝不能撞 key。
SCHEMA_INIT_ADVISORY_KEY = 20261108
def acquire_schema_init_lock(conn):
"""取 schema 初始化会话级 advisory 锁(阻塞等,不 try)。
调用方须在初始化全部完成后调 release_schema_init_lock 释放;
连接关闭时也会自动释放。初始化必须做完不能跳过,所以这里用
阻塞取锁,不用 pg_try_advisory_lock。
"""
conn.execute(
"SELECT pg_advisory_lock(?)",
(SCHEMA_INIT_ADVISORY_KEY,))
def release_schema_init_lock(conn):
"""放 schema 初始化会话级 advisory 锁(best-effort)。
事务已 abort 时 SELECT 本身也会失败——吞掉,连接关闭时锁
同样释放,绝不能让放锁失败再挡住启动。
"""
try:
conn.execute(
"SELECT pg_advisory_unlock(?)",
(SCHEMA_INIT_ADVISORY_KEY,))
except Exception:
pass
def init_postgres_schema(conn):
"""在给定适配连接上建出全部 Postgres 表(幂等),并提交。
@@ -1178,6 +1221,15 @@ def init_postgres_schema(conn):
库是 no-op。server.init_db 的 Postgres 分支还会再跑一次
migrate_projects_table,两次补列互不冲突(第二次是 no-op)。
"""
# 事务级 advisory 锁:并发 init_postgres_schema 串行执行,
# 随本函数末尾的 commit 释放;独立调用方(如 migrations)
# 同样受保护。server.init_db 另持会话级锁覆盖 _ensure_*。
# 不取结果行:pg_advisory_xact_lock 返回 void,且假连接
# (test_p2_migrations / test_audit_batch1 的 FakeCursor)
# 没有 fetchone——execute 本身已阻塞到取锁成功。
conn.execute(
"SELECT pg_advisory_xact_lock(?)",
(SCHEMA_INIT_ADVISORY_KEY,))
for statement in POSTGRES_SCHEMA:
conn.execute(statement)
cols = {row[1] for row in conn.execute(
@@ -1379,6 +1431,9 @@ __all__ = [
"PgCursor",
"PgRow",
"POSTGRES_SCHEMA",
"SCHEMA_INIT_ADVISORY_KEY",
"acquire_schema_init_lock",
"release_schema_init_lock",
"init_postgres_schema",
"create_database",
"database_from_env",