Deploy: deploy.sh builds tarball, backs up current code (keep 3), deploys, restarts, health-gates 60s (ok/backend=postgres/db_ok), auto-rolls back on failure; rollback.sh restores latest backup. Rollback is code-only, DB migrations are additive-only so old code can still read new columns. Backup: restore_postgres refuses to restore into the production DB name unless explicitly allowed (drill never allows it); restore-drill.sh backs up fresh, restores to a test DB, compares tasks/projects/users/audit row counts, drops the test DB. Monitoring: worker writes heartbeat file every loop (atomic replace, failure only logged); GET /health adds worker_last_seen_seconds_ago + worker_ok (180s default, env tunable, still HTTP 200 when worker is stale); monitor.sh checks server/worker/health/pg_isready/docker-postgres/disk/ app dir as one JSON line, exit non-zero on any fail, plus systemd timer every 5 minutes. Push alerting to chat still needs a separate channel decision (SSH key or webhook URL). Multi-user: verify-multiuser.py creates test users/projects in memory-key only, checks RBAC via real HTTP API (owner/ maintainer/contributor/viewer, cross-project isolation, merge 200/403/409, no-key 401), test tasks carry a never- satisfiable dependency so the production worker never claims them, cleanup deletes only multiuser-test- rows (audit log is append-only and keeps test entries by design). Tests: host 27 pass, cloud 298 pass (8 skipped are environment-dependent real-Postgres tests).
261 lines
12 KiB
Python
Executable File
261 lines
12 KiB
Python
Executable File
#!/usr/bin/env python3
|
||
"""多用户真实验证:主机侧以 codexsvc 身份运行,直接对生产库 + 本地 HTTP API 验证。
|
||
|
||
用法(主机上):
|
||
python3 /opt/codex-app-server/deploy/verify-multiuser.py [--base URL] [--db PATH]
|
||
默认 --base http://127.0.0.1:8765,--db /var/lib/codex-app-server/control.db
|
||
(Postgres 后端时 db 参数只作占位,连接参数走 CLOUD_DB_* 环境变量,
|
||
与 server/worker 一致;SQLite 后端时就是库文件路径)。
|
||
|
||
安全与防误伤(写进逻辑里,不是只写在注释里):
|
||
- 测试用户、项目、任务全部用 multiuser-test- 前缀(项目 id 固定为
|
||
multiuser-test-p1 / multiuser-test-p2,用户按用户名 multiuser-test-*
|
||
查找),清理时只删这些前缀的行,绝不碰其它生产数据。
|
||
- 测试任务创建时都带一个永不存在的依赖 multiuser-test-never-completes:
|
||
worker 只派依赖全部 completed 的 queued 任务(见 cloud/queue.py 的
|
||
ready_tasks),所以这些测试任务永远不会被生产 worker 认领去跑模型——
|
||
不花模型钱、不改任何仓库。验证完任务会被删掉,依赖本身只是一个
|
||
字符串,不对应任何真实任务行。
|
||
- API key 只存在内存里:创建用户时 AuthStore 返回的明文 key 只保存在
|
||
本进程的变量里,用完即弃;脚本全程只打印每项 PASS/FAIL 与总数,
|
||
绝不打印任何 API key;任务 prompt、用户 id 等内容也不打印,只打印
|
||
检查名与结果。
|
||
- 审计日志是 append-only 的:验证过程产生的审计条目(建项目、设成员、
|
||
建任务、merge 等)会留在 audit_log 里,脚本不删审计表——这是故意的,
|
||
审计就该留痕;README 里也写清了这一点。
|
||
- 清理失败时脚本退出非零并列出残留测试 id(不含 key),不静默假装干净。
|
||
- 本脚本不跑真实模型、不推代码、不改生产项目与任务,只动 multiuser-test-*
|
||
的行;跑之前不需要停 worker(见上,测试任务不会被认领)。
|
||
|
||
检查项(预期值写死在断言里,实际不符即 FAIL):
|
||
1. owner 建 P1、P2 成功(201)
|
||
2. owner 建任务成功(201,contributor 建任务也成功)
|
||
3. viewer 建任务 403;非成员(viewer 对 P2)建任务 403
|
||
4. viewer 读 P1 任务成功(200)
|
||
5. viewer 读 P2 任务 403(跨项目隔离:P1 成员不是 P2 成员)
|
||
6. maintainer 对 completed 测试任务调 merge 成功(200,状态变 merged)
|
||
7. contributor 对 completed 测试任务调 merge 403
|
||
8. maintainer 对 queued 测试任务调 merge 409(状态不对)
|
||
9. 无 key 调 API 401
|
||
"""
|
||
|
||
import argparse
|
||
import json
|
||
import os
|
||
import sys
|
||
import urllib.error
|
||
import urllib.request
|
||
|
||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||
|
||
from cloud.auth import AuthStore # noqa: E402
|
||
from cloud.db import connect as db_connect # noqa: E402
|
||
|
||
PREFIX = "multiuser-test-"
|
||
P1 = PREFIX + "p1"
|
||
P2 = PREFIX + "p2"
|
||
NEVER_DEP = PREFIX + "never-completes"
|
||
|
||
RESULTS = []
|
||
|
||
|
||
def check(name, ok):
|
||
RESULTS.append((name, bool(ok)))
|
||
print("%s %s" % ("PASS" if ok else "FAIL", name))
|
||
|
||
|
||
def req(base, method, path, key=None, body=None):
|
||
data = json.dumps(body).encode("utf-8") if body is not None else None
|
||
r = urllib.request.Request(base + path, data=data, method=method)
|
||
if key is not None:
|
||
r.add_header("X-API-Key", key)
|
||
if body is not None:
|
||
r.add_header("Content-Type", "application/json")
|
||
try:
|
||
with urllib.request.urlopen(r, timeout=15) as resp:
|
||
raw = resp.read()
|
||
return resp.status, (json.loads(raw) if raw else {})
|
||
except urllib.error.HTTPError as e:
|
||
raw = e.read()
|
||
try:
|
||
payload = json.loads(raw) if raw else {}
|
||
except ValueError:
|
||
payload = {}
|
||
return e.code, payload
|
||
except Exception as e:
|
||
return None, {"error": str(e)}
|
||
|
||
|
||
def main():
|
||
ap = argparse.ArgumentParser(description="multiuser verification")
|
||
ap.add_argument("--base", default="http://127.0.0.1:8765")
|
||
ap.add_argument("--db", default="/var/lib/codex-app-server/control.db")
|
||
args = ap.parse_args()
|
||
base = args.base.rstrip("/")
|
||
|
||
auth = AuthStore(args.db)
|
||
created_task_ids = []
|
||
created_user_ids = []
|
||
try:
|
||
# -- 预清理:上一轮残留的测试行(可重复跑)-------------------
|
||
conn = db_connect(args.db)
|
||
try:
|
||
conn.execute(
|
||
"DELETE FROM tasks WHERE project_id LIKE ?",
|
||
(PREFIX + "%",))
|
||
conn.execute(
|
||
"DELETE FROM project_members WHERE project_id LIKE ?",
|
||
(PREFIX + "%",))
|
||
conn.execute(
|
||
"DELETE FROM projects WHERE id LIKE ?", (PREFIX + "%",))
|
||
conn.execute(
|
||
"DELETE FROM users WHERE username LIKE ?", (PREFIX + "%",))
|
||
conn.commit()
|
||
finally:
|
||
conn.close()
|
||
|
||
# -- 建 4 个测试用户(key 只在内存里)------------------------
|
||
users = {}
|
||
for role in ("owner", "maintainer", "contributor", "viewer"):
|
||
uid, key = auth.create_user(PREFIX + role)
|
||
users[role] = (uid, key)
|
||
created_user_ids.append(uid)
|
||
|
||
# -- 建 2 个测试项目(owner 建,P2 只有 owner 是成员)--------
|
||
s, p1 = req(base, "POST", "/projects",
|
||
users["owner"][1], {"id": P1, "name": P1})
|
||
check("owner 创建项目 P1(201)", s == 201)
|
||
s, p2 = req(base, "POST", "/projects",
|
||
users["owner"][1], {"id": P2, "name": P2})
|
||
check("owner 创建项目 P2(201)", s == 201)
|
||
if s != 201 or p1.get("id") != P1:
|
||
print("项目创建失败,无法继续验证")
|
||
return 1
|
||
for role in ("maintainer", "contributor", "viewer"):
|
||
auth.set_member(P1, users["owner"][0], users[role][0], role)
|
||
|
||
# -- 建任务权限 ----------------------------------------------
|
||
# 全部测试任务带永不满足的依赖,生产 worker 绝不会认领(见模块注释)。
|
||
s, t_owner = req(base, "POST", "/tasks", users["owner"][1], {
|
||
"project_id": P1, "prompt": "multiuser test task",
|
||
"depends_on": [NEVER_DEP]})
|
||
check("owner 在 P1 建任务(201)", s == 201)
|
||
if s == 201:
|
||
created_task_ids.append(t_owner["id"])
|
||
s, t_contrib = req(base, "POST", "/tasks",
|
||
users["contributor"][1], {
|
||
"project_id": P1, "prompt": "multiuser test task",
|
||
"depends_on": [NEVER_DEP]})
|
||
check("contributor 在 P1 建任务(201)", s == 201)
|
||
if s == 201:
|
||
created_task_ids.append(t_contrib["id"])
|
||
s, _ = req(base, "POST", "/tasks", users["viewer"][1], {
|
||
"project_id": P1, "prompt": "multiuser test task"})
|
||
check("viewer 在 P1 建任务(403)", s == 403)
|
||
s, _ = req(base, "POST", "/tasks", users["viewer"][1], {
|
||
"project_id": P2, "prompt": "multiuser test task"})
|
||
check("非成员 viewer 在 P2 建任务(403)", s == 403)
|
||
|
||
# -- 读任务权限与跨项目隔离 ----------------------------------
|
||
if created_task_ids:
|
||
tid = created_task_ids[0]
|
||
s, _ = req(base, "GET", "/tasks/" + tid, users["viewer"][1])
|
||
check("viewer 读 P1 任务(200)", s == 200)
|
||
s, t_p2 = req(base, "POST", "/tasks", users["owner"][1], {
|
||
"project_id": P2, "prompt": "multiuser test task",
|
||
"depends_on": [NEVER_DEP]})
|
||
check("owner 在 P2 建任务(201)", s == 201)
|
||
if s == 201:
|
||
created_task_ids.append(t_p2["id"])
|
||
s, _ = req(base, "GET", "/tasks/" + t_p2["id"],
|
||
users["viewer"][1])
|
||
check("P1 成员 viewer 读 P2 任务(403,跨项目隔离)", s == 403)
|
||
|
||
# -- merge 权限(需先把测试任务直接置 completed,不跑模型)--
|
||
conn = db_connect(args.db)
|
||
try:
|
||
for tid in created_task_ids:
|
||
conn.execute(
|
||
"UPDATE tasks SET status = 'completed' "
|
||
"WHERE id = ?", (tid,))
|
||
conn.commit()
|
||
finally:
|
||
conn.close()
|
||
if len(created_task_ids) >= 1:
|
||
s, merged = req(base, "POST",
|
||
"/tasks/" + created_task_ids[0] + "/merge",
|
||
users["maintainer"][1])
|
||
check("maintainer 对 completed 任务调 merge(200 且变 merged)",
|
||
s == 200 and merged.get("status") == "merged")
|
||
if len(created_task_ids) >= 2:
|
||
s, _ = req(base, "POST",
|
||
"/tasks/" + created_task_ids[1] + "/merge",
|
||
users["contributor"][1])
|
||
check("contributor 对 completed 任务调 merge(403)", s == 403)
|
||
# queued 任务调 merge 应为 409:再建一个(不置 completed)。
|
||
s, t_queued = req(base, "POST", "/tasks", users["owner"][1], {
|
||
"project_id": P1, "prompt": "multiuser test task",
|
||
"depends_on": [NEVER_DEP]})
|
||
if s == 201:
|
||
created_task_ids.append(t_queued["id"])
|
||
s, _ = req(base, "POST",
|
||
"/tasks/" + t_queued["id"] + "/merge",
|
||
users["maintainer"][1])
|
||
check("maintainer 对 queued 任务调 merge(409)", s == 409)
|
||
|
||
# -- 无 key ---------------------------------------------------
|
||
s, _ = req(base, "GET", "/tasks/" + created_task_ids[0]
|
||
if created_task_ids else "/tasks/none")
|
||
check("无 key 调 API(401)", s == 401)
|
||
finally:
|
||
# -- 清理:只删 multiuser-test-* 的行,审计表不动 ------------
|
||
leftover = []
|
||
try:
|
||
conn = db_connect(args.db)
|
||
try:
|
||
for tid in created_task_ids:
|
||
conn.execute("DELETE FROM tasks WHERE id = ?", (tid,))
|
||
conn.execute(
|
||
"DELETE FROM tasks WHERE project_id LIKE ?",
|
||
(PREFIX + "%",))
|
||
conn.execute(
|
||
"DELETE FROM project_members WHERE project_id LIKE ?",
|
||
(PREFIX + "%",))
|
||
conn.execute(
|
||
"DELETE FROM projects WHERE id LIKE ?", (PREFIX + "%",))
|
||
conn.execute(
|
||
"DELETE FROM users WHERE username LIKE ?",
|
||
(PREFIX + "%",))
|
||
conn.commit()
|
||
# 校验残留(只查测试前缀的行,不打印 key 或内容)。
|
||
for table, col in (("tasks", "project_id"),
|
||
("projects", "id"),
|
||
("users", "username")):
|
||
row = conn.execute(
|
||
"SELECT COUNT(*) FROM %s WHERE %s LIKE ?"
|
||
% (table, col), (PREFIX + "%",)).fetchone()
|
||
if row and row[0]:
|
||
leftover.append("%s:%d" % (table, row[0]))
|
||
finally:
|
||
conn.close()
|
||
except Exception as e:
|
||
leftover.append("cleanup-error:%s" % e)
|
||
try:
|
||
auth.conn.close()
|
||
except Exception:
|
||
pass
|
||
if leftover:
|
||
print("清理未完成,残留测试行:%s" % ", ".join(leftover))
|
||
print("注意:审计日志里的测试条目是故意留下的,不算残留。")
|
||
|
||
passed = sum(1 for _, ok in RESULTS if ok)
|
||
total = len(RESULTS)
|
||
print("多用户验证:%d/%d 通过" % (passed, total))
|
||
if leftover:
|
||
return 1
|
||
return 0 if passed == total else 1
|
||
|
||
|
||
if __name__ == "__main__":
|
||
sys.exit(main())
|