Startup race: concurrent init_db (cloud-server + worker) ran CREATE TABLE / CREATE INDEX IF NOT EXISTS against a DB where the object did not exist yet; the check-then-create is not atomic, so both sessions created it and one died with UniqueViolation on pg_type_typname_nsp_index (tables) or pg_class_relname_nsp_index (indexes). Hit on the37ba687and7983567deploys (7983567added idx_tasks_parent); systemd restart healed it every time. Fix (advisory lock, not catch-duplicate_object): init_postgres_schema takes pg_advisory_xact_lock (key 20261108, released at its commit), and server.init_db takes the session-level lock for the whole PG branch so the _ensure_* objects outside POSTGRES_SCHEMA (task_continue_messages, user_gateway_keys, task_attempts, task_audit_loop_state) are covered too. Same idiom as the existing RATE_LIMIT_SWEEP_ADVISORY_KEY sweep lock; key is distinct. Tests: two new real-PG regression tests (skipped without CLOUD_TEST_PG_DSN), 4 processes racing on an empty schema, for init_postgres_schema and for full init_db; both fail on the unfixed code and pass with the fix.
This commit is contained in:
1 parent
7983567491
commit
8b2686548c
4 files changed
+238
-16
No files matched your search
+55
@@ -1165,6 +1165,49 @@ POSTGRES_SCHEMA = (
|
||||
)
|
||||
|
||||
|
||||
# 启动期 schema 初始化 advisory lock key(Postgres):
|
||||
# 并发启动(cloud-server 与 worker 同时 init_db)时,CREATE TABLE /
|
||||
# CREATE INDEX 的 IF NOT EXISTS 检查与建对象不是原子的——两个会话
|
||||
# 都查到“不存在”后同时建,会撞 pg_type_typname_nsp_index(建表)
|
||||
# 或 pg_class_relname_nsp_index(建索引)的 UniqueViolation,启动
|
||||
# 即崩、靠 systemd 重启才自愈(37ba687 与 7983567 两次部署均撞上;
|
||||
# 7983567 新增的 idx_tasks_parent 是当时的新对象)。
|
||||
# 持锁串行化整个初始化:init_postgres_schema 内持事务级锁(随本
|
||||
# 函数末尾的 commit 释放),server.init_db 另持会话级锁覆盖其后
|
||||
# 的 _ensure_*(task_continue_messages / user_gateway_keys /
|
||||
# task_attempts / task_audit_loop_state 不在 POSTGRES_SCHEMA 里,同样
|
||||
# 会撞)。key 须全项目唯一:限流清理另有
|
||||
# RATE_LIMIT_SWEEP_ADVISORY_KEY = 20261027(cloud/server.py),
|
||||
# 两处绝不能撞 key。
|
||||
SCHEMA_INIT_ADVISORY_KEY = 20261108
|
||||
|
||||
|
||||
def acquire_schema_init_lock(conn):
|
||||
"""取 schema 初始化会话级 advisory 锁(阻塞等,不 try)。
|
||||
|
||||
调用方须在初始化全部完成后调 release_schema_init_lock 释放;
|
||||
连接关闭时也会自动释放。初始化必须做完不能跳过,所以这里用
|
||||
阻塞取锁,不用 pg_try_advisory_lock。
|
||||
"""
|
||||
conn.execute(
|
||||
"SELECT pg_advisory_lock(?)",
|
||||
(SCHEMA_INIT_ADVISORY_KEY,))
|
||||
|
||||
|
||||
def release_schema_init_lock(conn):
|
||||
"""放 schema 初始化会话级 advisory 锁(best-effort)。
|
||||
|
||||
事务已 abort 时 SELECT 本身也会失败——吞掉,连接关闭时锁
|
||||
同样释放,绝不能让放锁失败再挡住启动。
|
||||
"""
|
||||
try:
|
||||
conn.execute(
|
||||
"SELECT pg_advisory_unlock(?)",
|
||||
(SCHEMA_INIT_ADVISORY_KEY,))
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def init_postgres_schema(conn):
|
||||
"""在给定适配连接上建出全部 Postgres 表(幂等),并提交。
|
||||
|
||||
@@ -1178,6 +1221,15 @@ def init_postgres_schema(conn):
|
||||
库是 no-op。server.init_db 的 Postgres 分支还会再跑一次
|
||||
migrate_projects_table,两次补列互不冲突(第二次是 no-op)。
|
||||
"""
|
||||
# 事务级 advisory 锁:并发 init_postgres_schema 串行执行,
|
||||
# 随本函数末尾的 commit 释放;独立调用方(如 migrations)
|
||||
# 同样受保护。server.init_db 另持会话级锁覆盖 _ensure_*。
|
||||
# 不取结果行:pg_advisory_xact_lock 返回 void,且假连接
|
||||
# (test_p2_migrations / test_audit_batch1 的 FakeCursor)
|
||||
# 没有 fetchone——execute 本身已阻塞到取锁成功。
|
||||
conn.execute(
|
||||
"SELECT pg_advisory_xact_lock(?)",
|
||||
(SCHEMA_INIT_ADVISORY_KEY,))
|
||||
for statement in POSTGRES_SCHEMA:
|
||||
conn.execute(statement)
|
||||
cols = {row[1] for row in conn.execute(
|
||||
@@ -1379,6 +1431,9 @@ __all__ = [
|
||||
"PgCursor",
|
||||
"PgRow",
|
||||
"POSTGRES_SCHEMA",
|
||||
"SCHEMA_INIT_ADVISORY_KEY",
|
||||
"acquire_schema_init_lock",
|
||||
"release_schema_init_lock",
|
||||
"init_postgres_schema",
|
||||
"create_database",
|
||||
"database_from_env",
|
||||
|
||||
Reference in new issue
Block a user