feat: routing shadow evaluation v0b (read-only, never auto-activates)
New pure module cloud/routing_shadow.py: parse worker_routing audit details (old format without suggested model parses with suggested unknown, never guessed), evaluate per-tier routed samples with one-task-one-sample dedup, explicit tasks counted separately, status under the chosen model, token/USD with unpriced tokens kept apart, and a min-20 gate per tier. The report explicitly marks the counterfactual not_measured: shadow suggested models never ran, so no suggested-vs-chosen cost/quality comparison is possible. Worker audit detail now includes suggested/chosen for future rows. GET /routing-shadow serves the report (project read permission, truncated flag over 10000 audit rows, cost table ensured so a fresh DB does not 500). Tests: test_routing_shadow.py (5 tests).
This commit is contained in:
1 parent
fd37505c48
commit
bac2a39377
4 files changed
+481
-2
No files matched your search
+9
-2
@@ -5920,10 +5920,17 @@ class Worker:
|
||||
# 不需要再用一条事件重复说明。
|
||||
if decision.source in ("policy", "shadow"):
|
||||
record_routing({"id": task_id, "events": events}, decision)
|
||||
# 影子评估(v0b)需要建议模型:旧格式只有
|
||||
# tier/source/applied,评估时建议模型只能记
|
||||
# unknown;新格式补 suggested/chosen,旧解析
|
||||
# 器仍能读(见 cloud/routing_shadow.py)。
|
||||
self._audit(task["project_id"], "worker_routing",
|
||||
"%s tier=%s source=%s applied=%s" % (
|
||||
"%s tier=%s source=%s applied=%s"
|
||||
" suggested=%s chosen=%s" % (
|
||||
task_id, decision.tier, decision.source,
|
||||
decision.applied))
|
||||
decision.applied,
|
||||
decision.suggested_model or "-",
|
||||
decision.chosen_model or "-"))
|
||||
task["model"] = decision.chosen_model or self.model
|
||||
# agent 指令通道(v2):local/docker 由 sandbox 的
|
||||
# --developer-instructions 派发,prompt 保持干净;通用
|
||||
|
||||
Reference in new issue
Block a user