Postgres real backend: psycopg adapter, schema, migration

Replace the PostgresDatabase skeleton with a real implementation:
psycopg (v3) first, psycopg2 second, lazy load, missing driver
still raises PostgresDriverMissingError with no silent SQLite
fallback. A sqlite3-style adapter (PgConnection/PgCursor/PgRow)
translates ? placeholders to %s outside string literals, maps
UNIQUE/FK violations, lock errors and connection failures to
explicit Postgres error types, and handles INSERT OR REPLACE,
PRAGMA table_info, AUTOINCREMENT and FTS5 refusal. POSTGRES_SCHEMA
covers all business tables with tasks result columns; server
init_db uses it under CLOUD_DB_BACKEND=postgres, SQLite path
unchanged. migrate_sqlite_to_postgres.py copies rows with
ON CONFLICT DO NOTHING and reseeds identity sequences.
Tests: cloud 239 (235 pass, 4 skipped without a real server),
host 27 pass; real-server tests skip unless CLOUD_TEST_PG_DSN
is set.
This commit is contained in:
host committed 2026-10-26 03:23:06 +08:00
1 parent a19e77ee44
commit dc3f572b2b
10 files changed
+1542 -66

No files matched your search

+35 -11
View File
@@ -29,7 +29,7 @@ audit_log 三张表;本 server 用 projects/tasks 两张表,表名不冲突
启动:
python3 -m cloud.server [--host 127.0.0.1] [--port 8765] [--db <路径>]
后端由 CLOUD_DB_BACKEND 选(sqlite 默认;postgres 目前只是骨架,
后端由 CLOUD_DB_BACKEND 选(sqlite 默认;postgres 已接通,
驱动缺失时启动即明确报错,见 cloud/db.py)。
测试用 create_server(db_path, host, port) 只返回 ThreadingHTTPServer
@@ -50,6 +50,9 @@ from cloud.auth import AuthStore
from cloud.cost import CostStore
from cloud.db import connect as _db_connect
from cloud.db import create_database as _create_database
from cloud.db import init_postgres_schema as _init_postgres_schema
from cloud.db import PostgresIntegrityError as _PgIntegrityError
from cloud.db import PostgresOperationalError as _PgOperationalError
from cloud.kb import KBStore
from cloud.memory import MemoryStore
from cloud.oauth import (
@@ -79,10 +82,14 @@ DEFAULT_SWEEP_THRESHOLD_SECONDS = 300
def _connect(db_path):
"""每个请求新建一个连接;走 cloud.db 的 WAL + busy_timeout。
"""每个请求新建一个连接;后端按 CLOUD_DB_BACKEND 分发。
单机 SQLite 的上限仍在:WAL + 有限重试只能缓解多线程写冲突,
高并发要换 Postgres(见 cloud/README.md)。
cloud.db.connect() 本身已按环境变量分发:sqlite 时是
WAL + busy_timeout 的 SQLite 连接;postgres 时是 psycopg
适配连接(db_path 被忽略,连接参数只走 CLOUD_DB_*
环境变量,驱动缺失时明确报错,绝不回落)。因此 server
自身的表与 AuthStore / MemoryStore 等沿用 connect() 的
调用点在 PG 模式下同样走 Postgres,无需逐一改写。
"""
return _db_connect(str(db_path))
@@ -111,10 +118,26 @@ def migrate_tasks_table(conn):
"ALTER TABLE tasks ADD COLUMN %s %s" % (name, coltype))
def _is_postgres_backend():
backend = os.environ.get("CLOUD_DB_BACKEND") or "sqlite"
return str(backend).strip().lower() in ("postgres", "postgresql")
def init_db(db_path):
"""创建 server 自己的 projects/tasks 表(AuthStore 的表由其自建)。"""
"""创建 server 自己的 projects/tasks 表(AuthStore 的表由其自建)。
Postgres 模式:建 cloud.db.POSTGRES_SCHEMA 的全部表(其中
projects/tasks 已含 migrate_tasks_table 的 6 列),再跑一次
migrate_tasks_table——PRAGMA 翻译后在 PG 上是 no-op,老库
迁移仍走同一函数。SQLite 模式:原行为不变。
"""
conn = _connect(db_path)
try:
if _is_postgres_backend():
_init_postgres_schema(conn)
migrate_tasks_table(conn)
conn.commit()
return
conn.execute(
"CREATE TABLE IF NOT EXISTS projects("
"id TEXT PRIMARY KEY, "
@@ -411,7 +434,7 @@ class CloudControlPlaneHandler(BaseHTTPRequestHandler):
"owner_id": user_id,
"created_at": now,
}
except sqlite3.IntegrityError:
except (sqlite3.IntegrityError, _PgIntegrityError):
self._send_json(409, {"error": "project exists"})
return
finally:
@@ -889,7 +912,7 @@ class CloudControlPlaneHandler(BaseHTTPRequestHandler):
json.dumps(depends_on), priority, user_id, now, now),
)
conn.commit()
except sqlite3.IntegrityError:
except (sqlite3.IntegrityError, _PgIntegrityError):
# 并发下同幂等键抢建:回读已有任务,不建重复。
row = conn.execute(
"SELECT * FROM tasks WHERE project_id = ?"
@@ -1116,7 +1139,7 @@ class CloudControlPlaneHandler(BaseHTTPRequestHandler):
"SELECT project_id, source, total_tokens, usd_micros,"
" fallback_count FROM cost_records"
).fetchall()
except sqlite3.OperationalError:
except (sqlite3.OperationalError, _PgOperationalError):
cost_rows = []
actual_tokens = estimated_tokens = 0
actual_usd = estimated_usd = 0
@@ -1325,11 +1348,12 @@ def create_server(db_path, host="127.0.0.1", port=8765, backend=None):
backend 为 None 时读环境变量 CLOUD_DB_BACKEND,未设则默认
sqlite(旧行为不变)。选 postgres 时在建表之前就调用工厂:
驱动缺失则初始化即抛明确错误(文案含 psycopg),启动即失败,
绝不在运行时中途才炸、也绝不静默回落到 SQLite;Postgres
目前只是骨架,即使驱动已装也尚未接通,见 cloud/db.py 与 README。
绝不在运行时中途才炸、也绝不静默回落到 SQLite;驱动已装
且能连上服务器时 Postgres 即为可用后端(适配层见
cloud/db.py,部署步骤见 cloud/README.md)。
"""
db_path = str(db_path)
# 后端校验必须在 init_db 之前:postgres 骨架不能先建 SQLite 表
# 后端校验必须在 init_db 之前:postgres 模式不能先建 SQLite 表
# 再失败,那会假装 sqlite 已可用。
_create_database(backend=backend, db_path=db_path)
init_db(db_path)