Commit Graph
10 Commits
Author SHA1 Message Date
openqjandhost fc8e337d0f HA 阶段 1:worktree_host + 跨主机 /pr 队列化 (#50)
* 部署自动化:唯一发布链路 deploy.py + host_deploy.sh,sha/commit 双重验真,失败回滚

旧链路三次栽过的坑:通用 tar 路径导致旧包误部署、传输不验 sha、
主机脚本多份漂移、health 通过不等于新代码上线。本单把四点变硬门禁:
- 产物唯一命名 codex-app-server-<commit>.tar.gz + .sha256,远端唯一路径
- git archive 产物 + DEPLOYED_COMMIT 清单,默认拒绝脏工作区
- 传输链 scp -> scp_run -> chunk 分片,每路传完远端验 sha,不符删掉重来
- 主机侧唯一脚本:先验 sha/清单,staging 解包 + 编译检查,换目录上线,
  work/ 原样保留,health + 上线 commit 不符即回滚并退出非零
- deploy.py 成功后独立 ssh 再读一次 live 清单二次核对
- CI:host-verify 加 deploy/tests;release.yml 只在手动/tag + production
  environment 审批时部署,主干推送绝不自动部署生产
- deploy.sh 标记为旧链路弃用
- health 的 commit 字段已在父提交 HA 阶段 0 里随 server.py 落地,
  本分支整体包含它;commit 为空时只靠文件核对,门禁同样成立
测试:host 32、cloud 487(4 跳过)、deploy 9,全部通过;
host_deploy.sh 仿真覆盖成功/坏 sha/清单不符/health commit 不符回滚。
生产实机部署未做,等确认后再用新链路推。

* HA stage 1: worktree_host + queued cross-host PR jobs

---------

Co-authored-by: host <host@test.local>
2026-10-27 00:05:18 +08:00
host 63823fc2e3 逐任务真实成本:上游用量事件 + 价目表计价
此前每任务只记一条 estimated(字符数/4,金额为空),注释称
host 协议不暴露真实 token 数。本轮实测推翻了该前提:
- account/usage/read 在中继上是死的(要求 ChatGPT 账号登录),
  不是来源,已在 cost.py docstring 写明死胡同
- app-server 每上游补全发 thread/tokenUsage/updated,last 是该次
  补全的用量;中继 5 调用回合实测 last 之和等于最后 total
  (input/output/cached/reasoning 四路都对得上)

实现:host.py run_turn 从事件窗口按 threadId+turnId 汇总用量,
重试/goal 各轮累加,dispatch/run JSON 与 goal 用量行透传;
sandbox 三路解析(dispatch JSON 解析改用完整 stdout,顺带修
了长 text 下 failed 被误判 completed 的潜 bug);worker 每任务
记一条 actual(重试/fallback 用量累加进同一条),无用量时回落
estimated;cost.py 价目表(默认 OpenAI 挂牌价,env 可覆盖,
无价模型 usd 记 None、报表进 unpriced_actual_tokens 而非 0 美元),
cost_records 加 cached_input_tokens 列(SQLite/PG 两支迁移);
报表与按天行新增 cached_input_tokens/unpriced_actual_tokens。
测试:host 32、cloud 483 全过。
2026-10-26 18:41:27 +08:00
host 5ee9d291dd 云端 goal/merge/review 三事件:goal 循环 + PR 事件通知
此前 goal-unmet/merge-failed/review-requested 只有允许列表,
云端没有生产者(概念只在本地 host.py)。本轮给三者接真生产者:
- goal:POST /tasks 接受 goal/verify_command/goal_max_rounds
  (迁移补列,SQLite 与 Postgres 两支);带 verify_command 的
  任务走 host.py goal 循环(Local/Docker 执行器 goal_task,
  turn→验测→喂回重试,缺省 3 轮);达标 completed,未达标
  failed 且只发 goal-unmet、不重复发 failed;goal 模式下 worker
  级重试/fallback 关闭,同任务 /retry 会撞 host 任务名已存在
  (已在文档写明,须新建任务);CloudExecutor 无 goal 概念,
  如实失败不降级
- review-requested:POST /tasks/{id}/pr 建 PR 成功时发
  (payload 含 pr_url/pr_number)
- merge-failed:PR 落盘(push/建 PR)失败 502 时发(payload
  含 step/error),任务保持 merged、可重调 /pr;跳过时不发
测试:host 29、cloud 469 全过。
2026-10-26 18:11:12 +08:00
host 9fa80bef89 修 GitHub App 安装在 Postgres 上崩:适配层译 INSERT OR IGNORE,安装后增减仓库同步 2026-10-26 16:11:29 +08:00
host 26d9edac33 Perf: light worker poll query + tasks(status) index
Worker polls the DB every interval (production 30s) and every
poll ran SELECT * FROM tasks: full rows including result_text
and error_text (up to 4000 chars each) for every task ever,
growing with history. load_tasks now selects only the columns
pick_next actually needs (id/project/prompt/status/depends/
priority/timestamps/executor) and only queued/completed/merged
rows; running/failed rows are neither ready nor completed, so
excluding them cannot change pick_next. get_task still does
SELECT * for the single claimed task. Plus CREATE INDEX IF NOT
EXISTS idx_tasks_status on tasks(status) in both SQLite
init_db and the Postgres schema, so the poll filter stops being
a full table scan on old databases too. Test: load_tasks returns
only queued+completed out of queued/completed/running/failed.
2026-10-26 06:19:01 +08:00
host 306cdf0d0c Cloud dispatch pool + auto PR (opt-in, skip clearly)
Worker pool: workers table + register/heartbeat (heartbeat is
registration, no 404), GET /workers with alive (120s default),
POST /workers/heartbeat (X-API-Key, missing worker_id 400).
Tasks gain executor (local/docker/cloud, None = any worker);
worker claim filters by executor in pick_next and in the CAS
WHERE clause, CAS semantics unchanged; CloudExecutor _name
now correctly returns cloud, not local.
CloudExecutor: generic HTTP runner interface (endpoint from
explicit arg or CLOUD_RUNNER_ENDPOINT, auth header name only,
token value only from env into the request header, never into
logs/body/errors); no endpoint configured = explicit failure,
never silent downgrade. Vendor adapters (ECS/Cloud Run) are
out of scope, documented as such.
Auto PR: projects.auto_pr (0/1, create-time only, no PATCH
route exists); POST /tasks/{id}/pr (maintainer+ via
approve_merge) only for merged tasks (else 409); skip reasons
in order: auto_pr_disabled, no_worktree, no_remote, no_gh,
gh_not_authenticated — all return 200 with pr_skipped, task
stays merged, no push attempted; push/pr-create failure
returns 502 with step, task stays merged; success writes
pr_url/pr_number and audits pr_created. No default auto-push
from the worker. Remote URLs in errors are scrubbed of
userinfo; gh credentials are managed by gh itself, never
stored by the control plane.
Tests: host 27 pass, cloud 323 pass (8 skipped are
environment-dependent real-Postgres tests).
2026-10-26 04:57:37 +08:00
host bed3fc0dff Project repo_url: worker can run tasks from any git repo
Projects gain repo_url (validated: https:// or git@ only, no
userinfo/password, no file:// or local paths; old DBs get the
column via idempotent ALTER in both SQLite and Postgres init).
Worker resolves a project's repo_url to a local checkout under
--repos-dir (default ./repos) before dispatch: clone if absent,
verify remote origin matches and fetch if present, never force
reset/clean, never push (push needs write credentials the
control plane does not store). Resolve failure fails the task
with a clear error, never completed. Docker workers need
repos_dir under a same-path extra mount (e.g.
/var/lib/codex-app-server/repos).
Also: driver-missing test is now mocked instead of depending
on the host really having no psycopg driver installed.
Deploy package: Caddyfile template exposing only /webhooks/*
and /oauth/* (tasks/projects/admin stay 403) plus a
webhook-secret generator script that never prints the secret.
Tests: host 27 pass, cloud 286 pass (8 skipped are
environment-dependent real-Postgres tests).
2026-10-26 04:16:11 +08:00
host dc3f572b2b Postgres real backend: psycopg adapter, schema, migration
Replace the PostgresDatabase skeleton with a real implementation:
psycopg (v3) first, psycopg2 second, lazy load, missing driver
still raises PostgresDriverMissingError with no silent SQLite
fallback. A sqlite3-style adapter (PgConnection/PgCursor/PgRow)
translates ? placeholders to %s outside string literals, maps
UNIQUE/FK violations, lock errors and connection failures to
explicit Postgres error types, and handles INSERT OR REPLACE,
PRAGMA table_info, AUTOINCREMENT and FTS5 refusal. POSTGRES_SCHEMA
covers all business tables with tasks result columns; server
init_db uses it under CLOUD_DB_BACKEND=postgres, SQLite path
unchanged. migrate_sqlite_to_postgres.py copies rows with
ON CONFLICT DO NOTHING and reseeds identity sequences.
Tests: cloud 239 (235 pass, 4 skipped without a real server),
host 27 pass; real-server tests skip unless CLOUD_TEST_PG_DSN
is set.
2026-10-26 03:23:06 +08:00
host 07ef869267 cloud: worker dispatch, DB abstraction, packaging 0.1.0, repo CI workflow 2026-10-26 01:44:15 +08:00
host 076ba01c74 cloud: fill gaps (memory/KB/cost/models endpoints, WAL concurrency, OAuth scaffold, push subscriptions) 2026-10-26 01:32:57 +08:00