host
362dffba4f
Production code had no golden tasks, scorer, or rubric
(harness2.py states this explicitly); routing shadow gives
cost/status only and admits counterfactual not_measured,
so 'is the suggested model actually better' had no
mechanism. This track adds a pure module + CLI that scores
already-completed task records against human-curated golden
tasks and compares two runs, without running tasks, spending
model cost, touching the DB, or activating anything.
Golden tasks: JSON {id, prompt, tier, checks}; only four
deterministic check types (contains, not_contains,
status_completed, max_tokens, max_cost_usd, the last in
cost.py microdollar units); invalid definitions fail loudly.
Scoring: per-task score = passed / measured checks;
cost checks with missing/unpriced data are unknown, excluded
from the denominator and reported separately (never counted
as pass or fail); empty result_text is a string-check fail,
not unknown. all_pass means 'all measured checks pass' and
is reported alongside fully_measured_n so a pass rate is
never misread as 'all checks passed'. Comparison: per-run
stats, paired delta over golden tasks present in BOTH runs
(unpaired listed separately, missing listed, unmatched
records excluded with a warning), duplicate task_id rejected.
Paired n < 20 (same threshold as shadow) => ready_for_
decision false, no significance or p-values computed.
Deliberately NOT included: a task runner (golden tasks
still need a separate run-and-record step or manual task
creation), an LLM judge or quality score, and any routing
activation suggestion. A contains check can be gamed by
echoing, so golden sets must be human-curated with checks
that require actually doing the work; the example file in
eval/ is format-only, not a decision-ready set. Tests:
test_eval_loop.py (8 tests).
23 lines
1.2 KiB
Markdown
23 lines
1.2 KiB
Markdown
# eval/ — 评估闭环的金任务示例（格式说明，不是已审定的金任务集）
|
||
|
||
`golden-tasks.example.json` 只演示 `cloud/eval_loop.py` 接受的格式：
|
||
每项 `{id, prompt, tier, checks}`，检查只认 contains / not_contains /
|
||
status_completed / max_tokens / max_cost_usd 五种确定性检查。
|
||
|
||
它不是一份可直接用来做激活决定的金任务集：里面的检查很弱
|
||
（例如只要求出现一个数值或一个函数名），容易被回显骗过。
|
||
真实金任务集必须由人逐条审定：检查要尽量要求“必须做对才过”
|
||
（算出的具体结果、必须经计算才得到的输出），而不是只要求
|
||
出现关键词；需要人工判断质量的任务不要硬塞进金任务集。
|
||
|
||
用法（只评分已完成任务的记录，不跑任务、不花模型费）：
|
||
|
||
python3 -m cloud.eval_loop --golden eval/golden-tasks.example.json \
|
||
--run-a run-a.json --run-b run-b.json \
|
||
--label-a model-a --label-b model-b
|
||
|
||
run 文件是记录列表，每条 `{task_id, golden_id, model, status,
|
||
result_text, total_tokens, usd_micros, unpriced_tokens}`，
|
||
其中 usd_micros 与 cloud/cost.py 同款微美元单位；成本数据
|
||
缺失时相关检查计为 unknown，不计入分母。
|