Stacked on v0.3 + v0.4 (HEAD d6522ba feat-full-auto): v0.3/v0.4
content preserved, not reverted. Adds G7 post-release ops and
observation (monitoring, errors, cost/quota, feedback, hotfix loop,
rollback triggers, ops manual) and G8 iteration and scaling (v1.1
handover list, contract v1->v2 protocol, new-end onboarding,
scale/performance, deprecation), with G6->G7 and G7->G8 gates,
DAG, gate table, coordinator derivation, G0-G8 todos, checklist,
09-ops and 10-iteration role templates, and ops-manual / v1.1
handover templates in both body and appendix. Settles web+React+
multi-viewport focus, cross-viewport E2E, web single-end release
with iOS/Android native postposed, open-source-first, and the
TypeScript full-stack default (user-request-only language change;
codex-app-server itself stays Python). Pending #199 and #205
merge, then one rebase round. PR only, do not merge or deploy.
- P1-1 directory-isolation executable check: git diff --name-only
against merge-base, output recorded in merge record
- P1-2 derivation checklist table (8 required items, missing one blocks dispatch)
- P1-3 QA split into single-client review and cross-client E2E with
fixed deliverables and agent-test attribution
- P1-4 sub-task slim-pack trimming rules (listed chapters only, never full copies)
- P2: inter-thread Q&A default policy, docs/decisions.md append-only,
per-client onboarding docs (web first at G4a), v1.1 handover checklist
- Body and appendix kept in sync; new lightweight keyword test
- worker._tenant_dirs: playwright refresh log used undefined 'task';
use the method argument project_id (ruff F821, real bug: NameError
aborted tenant seeding when a playwright script was refreshed).
Regression: cloud/tests/test_worker_playwright_seed_f821.py (2 tests;
pre-fix run reproduces NameError at worker.py:2780).
- harness2.assign_ids: hashlib.sha1(..., usedforsecurity=False) -- the
digest is only a deterministic shuffle seed, not a security use;
digest value unchanged, bandit B324 no longer fires (High 5 -> 4).
- harness2.select_final: rename ambiguous loop var 'l' to 'lab'
(ruff E741, behaviour-invariant).
- Deliberately NOT changed: host.py shell=True x4 (see PR body -- all
four are intentional shell commands, changing semantics is unsafe),
remaining ruff 109 / mypy 24, workflow thresholds (75% coverage
gate unchanged; no gate was relaxed or removed).
Phase 1 of audit-hardening CI expansion (baseline origin/main aebc623):
- lint: ruff E9 hard gate; full E,F@120 report-only (baseline 111 items)
- typecheck: mypy default mode, production scope, report-only (baseline 24)
- security: bandit production scope, report-only (baseline 287, High 5)
- depscan: pip-audit on requirements-ci.txt, hard gate (0 vulns; repo has
no production dependency manifest, see summary)
- coverage: production 58 files, fail-under=75 (baseline measured 76%)
- release build: E9 + pip-audit hard gates before packaging; full reports
stay in host-verify to avoid drift
No production code changed; CI billing failure mode unchanged.
Replace bare open().read()/write() in tests with Path.read_text/
write_text (or json.loads over read_text); the five remaining
open(...).close() empty-file creations become Path.write_text('').
No assertion, predicate, or production code changed; no behaviour
change beyond closing the handle deterministically.
Long-function splits and server.py module split intentionally not
included: the longest functions (_dispatch, _create_task,
_dispatch_project, ~375-395 lines in a 10430-line server.py) are
central routing/creation paths, and splitting them is not low-risk;
see PR body.
Goal tasks pushed back by the audit loop lost the packet preamble on
the next round: claim-time _apply_audit_loop_packet pasted it into the
in-memory prompt/goal, but the next round collided with the existing
host task name, fell back to breakpoint goal-resume (no --goal-text),
and the host reused the stored goal, discarding the preamble.
Fix (minimal, direction (c) trimmed): host goal-resume gains
--resume-note, accepted only by the breakpoint branch and prepended
to the first turn prompt of this resume only -- never written back
to the stored goal, never touching rounds or verification. The worker
stashes the same preamble as the memory-only _audit_loop_resume_note
at claim time and forwards it on breakpoint resume (Local/Docker
goal_resume_task and the popen goal-resume command, each guarded to
ignore the note when goal_text is set for a new-goal resume).
Accounting untouched: audit 3-round cap, goal_max_rounds/quota, and
DB goal columns are unchanged; non-goal pushback is unchanged.
Tests: cloud/tests/test_audit_n12.py (9 tests) -- host first-turn-only
note + stored-goal/rounds, worker exists-fallback end-to-end with
--resume-note, non-goal regression, goal-column accounting, and
Local/Docker/popen command shapes. Full cloud suite: 1760 OK
(skipped=11), baseline 1751 + 9 new; tests/test_host.py: 62 OK.
- I-3: new models.evaluate_capability returning a CapabilityVerdict
(ok/missing/unknown from check, known/unknown_missing from split,
status/verb derivations); decide_routing, worker gateway check,
worker persisted recheck, dry-run tier+gateway checks and server
shadow endpoint all call it. Blocking rules stay at the call
sites (gateway has no worker-default fallback, recheck blocks
only on known, dry-run/shadow never block); all user-visible
audit details, dry-run notes and shadow notes are byte-identical.
- I-4: new models.merge_policy_documents shared by
load_routing_policy and dry_run_from_env; the old tiers-update
block is kept verbatim (it was a no-op after update(): JSON
tiers replace the whole map, file tiers absent from JSON are
dropped -- preserved, not fixed, as that would be a behaviour
change). dropped_tiers/partial_failures remain load-time
records, not merge products.
- Tests: test_audit_batch3.py, 13 tests (verdict matrix + parity
with check/split, decide/dry-run wording, merge boundaries +
300 seeded random pairs + loader/dry-run agreement).
Full suite: 1694 OK (skipped=11), baseline 1681 + 13, 0 failures.
- I-1: evaluate_shadow counts dropped rows by cause (unparseable /
unknown_tier / explicit / duplicate) with the identity
audit_rows_total == routed_task_n + dropped_total; the shadow
endpoint passes created_at/id through and reports audit totals
next to sample counts.
- I-2 (option b): parse logs a warning naming each tier dropped for
an invalid mode/requires; load attaches partial_failures to the
policy; the worker records a task-level
worker_routing_policy_load_failed audit for partial failures
(same action as full failure, detail says failure=partial);
the shadow report carries the policy fingerprint, dropped tiers
and partial failures (no task context on that path, so no
task-level audit there).
- I-5: coordinator gate audit lookup failure now raises
_CoordGateLookupFailed and the merge converts it to escalate
(no requeue, no round consumed); persisted-model lookup failure
and required-tools check failure each write a named audit row
before falling through to dispatch/allow; audit-loop evaluate
failure stays escalate (regression test).
- Test fixtures: coordinator suites build audit_log via AuthStore
as production's first real audit does, since their stubbed
_audit never creates the table.
- B-A: migrations.py tasks backfill loop now types audit_loop as
INTEGER (other 6 columns stay TEXT), matching db.py backfill,
server/worker ensure lists; TEXT readback '0' is truthy and
would silently enable the audit loop for audit_loop=0 rows.
- B-C: gateway preparation failure early-return now calls
_running_clear(task_id) next to _cleanup_task_seed, matching
missing-key/repo/capability/tool gates; previously the capacity
slot leaked until worker exit (default capacity 1 blocks reclaim).
- Tests: cloud/tests/test_audit_batch1.py (3 tests): migration
ALTER types, INTEGER 0 readback not enabled via real worker
row mapping, gateway failure clears slot and capacity reclaim.
_claim doc 'three gates' list left unchanged (full early-return
gate enumeration not verified this batch; residual wording).
- No other audit items touched (I-1..I-14, N-1..N-12).
- M1: test_goal_branch_via_convert_queued_then_claim actually calls
server._goal_convert_queued on a queued auto-resumed task before
claiming, instead of only inserting the post-conversion state.
- M5: test_worker_routing_audit_rows_carry_no_fingerprint drives a
real run_one with an active policy and asserts worker_routing
audit rows exist and carry no fingerprint (negative control
verified: injecting fingerprint into the detail fails it).
- M2: 'all callers' wording tightened to the real counts
(load_routing_policy: 2 production call sites; evaluate_shadow: 1).
- M4: content_fingerprint docstring now states the fingerprint covers
the full normalized payload including fallback_models, which the
router deliberately does not apply (fail-safe direction).
No behaviour change; M3 DESIGN.md edits are outside the repo.
- db: translate INSERT OR IGNORE ... SELECT to ON CONFLICT DO NOTHING
(VALUES-only regex left SELECT form untranslated -> PG syntax error);
register task_audit_loop_state (task_id) in _OR_REPLACE_PK so PG
INSERT OR REPLACE becomes a real upsert instead of DO NOTHING.
- server: _backfill_default_quotas uses SAVEPOINT containment and no
longer commits the caller's transaction, so a backfill failure cannot
roll back init_db's migrate ALTERs.
- worker: _coord_child_rows rolls back before column-fallback retries so
a missing-column failure on PG does not cascade into
InFailedSqlTransaction.
- tests: test_pg_init_fix.py (translation, SQLite backfill/upsert,
real-PG fresh-init + merge + upsert + backfill containment, skipped
without CLOUD_TEST_PG_DSN/driver).
Parent audit loop sits after all children are aggregated and
before the parent final state; the single-task _audit_loop_wire
is never applied to parents or children. Parent gates are
per-child gates child:<name> (completed only); execution
failures map to one deterministic finding per child; subjective
findings are split off before evaluate and only audited as
needs_review. Gate-rejected children (capability/tool/approval
audit rows, or distinctive error markers) bypass evaluate and
escalate directly without consuming a round; tolerated, paused
and quota-missing parts are excluded, never findings. Rounds
are counted only in the parent task_audit_loop_state; on
pushback the packet is stored on the parent and split per failed
child into that child's audit_loop_packet column, consumed once
by the existing packet-injection path at next claim; pass and
escalate clear parent and child packets. POST /tasks now allows
audit_loop on coordinator parents (the old 400 rationale is
gone); children still take no external switch.
Audit-loop wiring, worker completion path only: after the final
write-back (and only when the final is completed/failed, not quota,
not cancelled, not goal-converted, and the task carries audit_loop),
collect this attempt's deterministic gate results (execution,
goal_verify only when host reported goal_met, capability /
required_tools / gateway as passed-or-inapplicable markers, since a
pre-dispatch gate rejection early-returns before any runner round
and requeueing it would deterministically re-fail the same gate)
and call audit_loop_evaluate. Pushback stores the packet JSON in
tasks.audit_loop_packet and CAS-requeues the task; the next claim
prepends the packet preamble to prompt (and goal) with a data-only
separator, DB prompt untouched. Pass clears the packet. Escalate
sets failed (VALID_STATUSES has no needs_review, never invented)
with an escalation error and clears the packet; subjective
needs_review leaves the outcome untouched and only audits.
New tasks.audit_loop / audit_loop_packet columns follow the
attempts DDL pattern in worker ensure, server migrate, PG schema,
PG backfill, migrations backfill, and POST /tasks accepts
audit_loop (True/1 only, coordinator tasks rejected like v9).
Coordinator parent tasks return via _run_coordinator before the
attempt loop and coordinator children never inherit audit_loop,
so coordinator wiring is left as legacy, unchanged.
Audit loop engine (cloud/audit_loop.py): Finding/PushbackPacket/LoopState pure functions; deterministic gates only, subjective -> needs_review; packet completeness enforced; standards-weakening -> escalate; regression and missing-gate checks enforce full re-run; MAX_PUSHBACKS=3, round>=3 -> escalate. Persistence in new single-row task_audit_loop_state (attempts DDL pattern: module CREATE_TABLE_SQL + db.py POSTGRES_SCHEMA + migrations list + server init + worker ensure); task_attempts NOT reused (attempt_no PK collision) and task_followups NOT reused (user messages). Worker.audit_loop_evaluate is the explicit opt-in hook (task audit_loop field only, no env switch, never called by run_one); escalate audits worker_audit_loop_escalated; corrupted/missing count fail-closes to escalate.
Parse: agent_tools/tool_approvals column parsing unified into agent_def shared helpers (parse None = undeclared; server keeps None, worker maps to []/{} at _task_from_row, dispatch behaviour unchanged). Goal: _apply_resume_preamble goal branch proven reachable via _goal_convert_queued on a queued auto-resumed task, kept with a test and corrected docs (not dead code). Fail-open: audit write failure and preamble lookup failure now log warnings (best-effort unchanged); partial policy failure and shadow parse drops documented as known gaps. M7: shadow endpoint explicit_capability_failures_truncated test. Fingerprint: RoutingPolicy.content_fingerprint (canonical JSON sha256, version excluded) surfaced only in dry-run output, never in audit rows. M1 wording: legacy gap stated as fixed stock persisting per-task until completion, not converging.
N1-1: writeback scope verified too broad (any chosen, incl. shadow
applied=False, worker_default and worker_explicit) and narrowed to
source=policy + applied only; writeback now records a
worker_model_persisted audit row as the identity marker.
N1: persisted models are rechecked against current tier requires
before dispatch on later attempts (fixes4 split known/unknown):
active + known-missing fails fast (worker_capability_recheck_
rejected, no runner), unknown-only and shadow only audit.
User-explicit, worker-CLI-explicit and gateway-filled tasks skip
the recheck; legacy models are recognised by a policy applied
worker_routing row. Recheck never rewrites tasks.model and never
re-decides.
Explicit gateway capability now splits known/unknown: known-missing
explicit models fail-fast under active (same shape as the gateway
default gate), unknown explicit models stay audit-only, shadow never
blocks; explicit priority now protects which model the user chose,
not that the chosen model must run. Shadow report reads
worker_capability_gateway_explicit rows as explicit_capability_failures
with a field note distinguishing the three explicit counts.
N2: coordinator requeue CAS rowcount=0 branch covered via a proxy
connection test (no refactor needed). N3: shadow endpoint test on
the create_server + real HTTP skeleton (capability, truncated, and
explicit-failure fields). N4: creation-layer rejection tests for
tools/instructions/tool_approvals on goal/coordinator tasks.
N5: model writeback failure logs a warning (best-effort kept).
N1: writeback capability-recheck trade-off documented in comment
only; recheck itself is a later batch.
B1: gateway capability check now triggers on walking the sub2api
gateway, not on the gateway having filled a model; task-level
explicit models are audited (worker_capability_gateway_explicit)
but never blocked, preserving explicit priority; gateway-default
models still fail-fast under active.
I1: dry-run adds a gateway-model section, sourced only from
CLOUD_SUB2API_DEFAULT_MODEL (never queries the relay); an active
tier whose requires the gateway model fails or cannot verify
blocks, shadow only warns.
I2: shadow review gate now counts completed samples
(completed >= 20), same basis for sample_shortfall; the endpoint
attaches per-tier capability status (read-only policy load) and
flags capability-failing tiers in the recommendation.
I3: configured-but-broken routing policy logs a warning in
load_routing_policy (unconfigured stays silent); the worker sets
routing_policy_load_failed and audits
worker_routing_policy_load_failed per task.
I4: resume preamble wraps prior output/error as data with an
explicit do-not-execute constraint; position verified in run_one
(tree -> preamble -> project -> original) and the docs corrected
from the false 'closest to the original prompt' claim.
I5: decided model is written back with a CAS UPDATE
(model IS NULL OR model=''), so later attempts reuse it;
explicit models are never overwritten.
I6: agent_* columns added to worker._ensure_result_columns and
db.init_postgres_schema backfill; migrations backfills them for
tasks before insert (sender_task_id pattern); column-level tests
for agent_tool_approvals and task_attempts.
I7: coordinator requeue adds AND status=<snapshot> (rowcount 0
skips requeue/audit) and filters paused_by_user / user_cancel /
user_pause children, which are never revived or cleared.
I8: explicit-model tasks (including coordinator children that
inherited a parent model) audit a source=explicit worker_routing
row; evaluate_shadow counts it as per-tier explicit_n only,
never routed_n, never a dedup slot.
M1/M2/M3/M5/M6/M8/M10: dedup slot moved after tier validation,
shadow sort doc fixed, truncated recommendation notes truncation,
seed cleanup on gateway/quota/goal-cap early returns, tools-only
coordinator definitions rejected, preamble goal branch marked
effectively unreachable, DESIGN.md drift corrected (outside repo).
M4/M7/M9: documentation/new-field only (pass_rate_fully_measured,
whitespace-model shadow-drop note, bound-note race note); old
field semantics unchanged.
- RoutingPolicy.fallback_models: move the cleaning loop into
__init__ (it was unreachable after tier_mode's return, so the
attribute never existed); still parsed but not applied.
- Capability fallback: RoutingDecision gains capability_fallback;
the worker audits worker_capability_fallback when it fires, so
a tier-model rejection followed by a worker-default fallback
is no longer invisible (source stays worker_default, no
model_routing event, so shadow samples are unaffected).
- Registry: resolve_registry_agent checks skills directly
against the loaded definition instead of re-parsing a
hand-built dict, so future fields cannot be silently dropped
the way tool_approvals was in v9.
- Shadow: evaluate_shadow sorts rows earliest-first by
created_at (id tiebreak) when timestamps are present, so
dedup no longer depends on caller row order; the server
path (which strips created_at) is unchanged.
- Gateway: when sub2api fills the default model, the worker
now checks it against the tier's capability requirements
with the same mode semantics as v8 -- shadow only audits,
active fails fast with worker_capability_gate_rejected and
never calls the runner. No worker-default fallback here:
the gateway model is the task's gateway model, falling back
to a worker default would run against the wrong gateway.
Agent definitions gain tool_approvals: known tool -> ask/auto/deny.
Unknown tool/policy is a 400, and required+deny is a 400
contradiction. The declaration is persisted (tasks.agent_tool_
approvals, PG + migrate), echoed in GET and the creation response
(via to_summary), and resolved for registry definitions -- fixing
a latent drop where resolve_registry_agent re-parsed without new
fields. At dispatch the worker audits the declaration and fails
fast on a defensive required+deny re-check (v3 tools-gate shape,
runner never called); goal/coordinator tasks reject the field at
creation like instructions. effective_tool_approvals maps a
declaration + host global approval to the actual host answer,
with ask_honoured always False: host answers elicitation with
one global switch and carries no tool identity, and the approvals
queue is record-only with no runtime ask channel, so per-tool
interception and credential scope are explicitly NOT claimed.
Policy carries an operator-declared model_capabilities directory
and per-tier requires (tool_calling/reasoning/vision booleans,
min_context_tokens). decide_routing checks the policy tier model
in active mode: capable applies, incapable falls back to a capable
worker default, and if both fail or are unknown it returns
capability_blocked, which the worker turns into a failed task
(v3 tools-gate shape, runner never called). Shadow mode only
annotates the reason, never blocks. Explicit task and worker CLI
models bypass the check by design. Invalid requires drops only
that tier (fail-closed, like invalid tier mode). Dry-run reports
per-tier capability status and blocks on active tiers that fail
or cannot be verified.
Production code had no golden tasks, scorer, or rubric
(harness2.py states this explicitly); routing shadow gives
cost/status only and admits counterfactual not_measured,
so 'is the suggested model actually better' had no
mechanism. This track adds a pure module + CLI that scores
already-completed task records against human-curated golden
tasks and compares two runs, without running tasks, spending
model cost, touching the DB, or activating anything.
Golden tasks: JSON {id, prompt, tier, checks}; only four
deterministic check types (contains, not_contains,
status_completed, max_tokens, max_cost_usd, the last in
cost.py microdollar units); invalid definitions fail loudly.
Scoring: per-task score = passed / measured checks;
cost checks with missing/unpriced data are unknown, excluded
from the denominator and reported separately (never counted
as pass or fail); empty result_text is a string-check fail,
not unknown. all_pass means 'all measured checks pass' and
is reported alongside fully_measured_n so a pass rate is
never misread as 'all checks passed'. Comparison: per-run
stats, paired delta over golden tasks present in BOTH runs
(unpaired listed separately, missing listed, unmatched
records excluded with a warning), duplicate task_id rejected.
Paired n < 20 (same threshold as shadow) => ready_for_
decision false, no significance or p-values computed.
Deliberately NOT included: a task runner (golden tasks
still need a separate run-and-record step or manual task
creation), an LLM judge or quality score, and any routing
activation suggestion. A contains check can be gamed by
echoing, so golden sets must be human-curated with checks
that require actually doing the work; the example file in
eval/ is format-only, not a decision-ready set. Tests:
test_eval_loop.py (8 tests).
Foundry 'platform restarts with original input' gap, after
reading the existing mechanisms: creation idempotency
(UNIQUE project_id+idempotency_key), goal checkpoints
(task_checkpoints per round, same-thread breakpoint resume
with no-progress fuse), merge/cost/notification CAS, and
coordinator child reuse already exist. The uncovered segment
is non-goal platform-interrupted tasks: auto_resume requeues
them fresh in the same worktree after clearing result/error/
interrupted_reason, so the new run gets only the original
prompt and cannot tell what the previous attempt had done.
This track adds, in worker/server layer only (no host.py or
codex thread change): task_attempts ledger, recorded by
auto_resume_interrupted in the same transaction before
clearing (attempt_no = old auto_resume_count + 1, kind
goal_breakpoint|platform_fresh, interrupted_reason,
checkpoint_round, result/error truncated to 2000 chars,
INSERT OR IGNORE keeps the first record on duplicate);
resume preamble injected in-memory only at claim for
platform_fresh reruns (previous attempt facts, truncated to
1000 chars, plus check-worktree-first and verify-before-
recreating-external-artifacts instructions; skipped for
coordinator/_resume/breakpoint tasks and when no record
exists, never fabricated); attempts surfaced read-only in
GET /tasks/{id}/timeline under existing project-read
permission. Fresh PG schema, migrations table list, and
server/worker ensure paths share one DDL.
Deliberately NOT claimed: the preamble is an instruction,
not a hard guarantee like creation idempotency or merge
CAS; the ledger does not track worktree files or external
artifacts (no worktree snapshot, executor-dependent paths;
hard dedup for those still rests on their existing gates);
manual retry paths do not go through auto_resume, so they
get no ledger record or preamble. Tests:
test_attempts.py (7 tests).
Coordinator parts gain three enforced bounds, validated in
validate_parts and carried through the coordinator_parts JSON
(no schema change): timeout_seconds (1..86400, per-attempt
wall clock from coordinator start; the coordinator poll loop
cancels only that child with the same cancel mechanism as
the global timeout, other parts keep waiting, audit
coordinator_child_bound_exceeded), on_failure fail|tolerate
(tolerated failures do not fail the coordinator, are named
in the report head, and their failed children are not
requeued on coordinator re-run while interrupted children
still are), and result_chars (0..2000, default 500, enforced
in build_report; the 4000-char report writeback cap still
applies). evaluate_outcome is the single outcome rule used
by both build_report and the worker error text, so tolerated
failures never leak into the coordinator failure reason.
Unknown part names fail closed.
Deliberately NOT added, after reading the execution layer:
per-part token/cost budget (cost is recorded only on
completion, so a running child's spend is unobservable and
creator quota is monthly, not per part), per-part round cap
(normal tasks have no per-task round cap in the worker),
and write-scope isolation (sandbox is per-project worktree,
a scope string in a prompt would be an instruction, not a
guarantee). Tests: test_delegation_bounds.py (8 tests).
resolve_registry_agent in cloud/agent_def.py: agent_name may be
<name> or <name>@<version>, agent_version may also be given
separately; both given must agree or 400. Unpinned loads
<name>.json (default); pinned loads <name>.<version>.json and
requires the file's own version field to match, with no silent
fallback to the default file. Version strings are restricted
to alphanumerics/._- (no path separators, no ..); the
agents-dir traversal guard is a second line of defence.
Server: agent_version is only for agent_name (inline agent
with agent_version is 400); stored agent_name is the base
name, not the @ form. Snapshot fixture:
agents/research-brief.v1.json (same content as the default
v1 file). Tests: test_agent_version.py (7 tests).
New module cloud/routing_dryrun.py: dry-run a merged policy
document or an env mapping (file first, JSON overrides whole
tier entries, CLOUD_ROUTING_MODE overrides global mode only,
never explicit tier modes), reporting per-tier effective
model/mode/mode-source, dropped tiers with reasons, absent and
unknown tiers, and would_activate/would_shadow/would_off.
Dropped or unknown tiers block (CLI exit 1); warnings alone
(no version, fallback_models parsed but not applied, failed
file/JSON sources that the loader silently ignores) do not
block but are always printed. Model names are NOT validated
against a catalog (deliberately not whitelisted) and sample
counts are NOT checked here (that is the shadow report's job);
the CLI never writes env, files, or worker state.
Tests: test_routing_dryrun.py (8 tests).
RoutingPolicy gains tier_modes: a tier entry may be a string
(model, mode inherits global) or {model, mode}. The explicit
tier mode overrides the global mode in both directions, so plan
can be active while goal/default stay shadow, and one tier can
be rolled back to off without touching the others. An invalid
tier mode drops that tier entirely (fail-closed: a typo must
never accidentally activate a tier, and never kills other
tiers). Tier-off tasks return worker_default and are not
audited as policy/shadow, so they never inflate shadow sample
counts. Explicit task and worker-CLI models still win over any
tier mode.
Tests: test_routing_tier.py (5 tests).
New pure module cloud/routing_shadow.py: parse worker_routing
audit details (old format without suggested model parses with
suggested unknown, never guessed), evaluate per-tier routed
samples with one-task-one-sample dedup, explicit tasks counted
separately, status under the chosen model, token/USD with
unpriced tokens kept apart, and a min-20 gate per tier. The
report explicitly marks the counterfactual not_measured:
shadow suggested models never ran, so no suggested-vs-chosen
cost/quality comparison is possible. Worker audit detail now
includes suggested/chosen for future rows. GET /routing-shadow
serves the report (project read permission, truncated flag
over 10000 audit rows, cost table ensured so a fresh DB does
not 500).
Tests: test_routing_shadow.py (5 tests).
host