fix: 9.5-code reachable items (parse unification, goal branch, fail-open logs, M7 test, fingerprint)
Parse: agent_tools/tool_approvals column parsing unified into agent_def shared helpers (parse None = undeclared; server keeps None, worker maps to []/{} at _task_from_row, dispatch behaviour unchanged). Goal: _apply_resume_preamble goal branch proven reachable via _goal_convert_queued on a queued auto-resumed task, kept with a test and corrected docs (not dead code). Fail-open: audit write failure and preamble lookup failure now log warnings (best-effort unchanged); partial policy failure and shadow parse drops documented as known gaps. M7: shadow endpoint explicit_capability_failures_truncated test. Fingerprint: RoutingPolicy.content_fingerprint (canonical JSON sha256, version excluded) surfaced only in dry-run output, never in audit rows. M1 wording: legacy gap stated as fixed stock persisting per-task until completion, not converging.
This commit is contained in:
1 parent
d95d678379
commit
1b8b6e525d
7 files changed
+498
-57
No files matched your search
@@ -200,6 +200,53 @@ class AgentDefinition:
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def parse_agent_tools_column(raw):
|
||||||
|
"""Parse one tasks.agent_tools column value (the single shared
|
||||||
|
parser for server and worker; 9.5-code unification).
|
||||||
|
|
||||||
|
Returns a list copy, or None when the value is missing, is not
|
||||||
|
a JSON list, or is otherwise dirty. None is the parse-layer
|
||||||
|
default and means "undeclared", not "no required tools": the
|
||||||
|
server keeps None in task serialization (an old row with a
|
||||||
|
dirty value serializes exactly as before), while the worker
|
||||||
|
maps None to [] at the _task_from_row boundary, because
|
||||||
|
dispatch treats "no declaration" and "no required tools"
|
||||||
|
identically (both skip the v3 gate, and every worker use site
|
||||||
|
already reads ``task.get("agent_tools") or []``). Unifying the
|
||||||
|
parse logic therefore changes no observable behaviour on
|
||||||
|
either side; the old server/worker defaults (None vs []) were
|
||||||
|
only different spellings of the same dispatch outcome.
|
||||||
|
"""
|
||||||
|
if raw is None:
|
||||||
|
return None
|
||||||
|
if isinstance(raw, list):
|
||||||
|
return list(raw)
|
||||||
|
try:
|
||||||
|
parsed = json.loads(raw)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return None
|
||||||
|
return list(parsed) if isinstance(parsed, list) else None
|
||||||
|
|
||||||
|
|
||||||
|
def parse_agent_tool_approvals_column(raw):
|
||||||
|
"""Parse one tasks.agent_tool_approvals column value (the
|
||||||
|
single shared parser for server and worker; same None
|
||||||
|
convention as parse_agent_tools_column: None means
|
||||||
|
"undeclared", the server keeps it, the worker maps it to {}
|
||||||
|
at the _task_from_row boundary because every worker use site
|
||||||
|
already reads ``task.get("agent_tool_approvals") or {}``).
|
||||||
|
"""
|
||||||
|
if raw is None:
|
||||||
|
return None
|
||||||
|
if isinstance(raw, dict):
|
||||||
|
return dict(raw)
|
||||||
|
try:
|
||||||
|
parsed = json.loads(raw)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return None
|
||||||
|
return dict(parsed) if isinstance(parsed, dict) else None
|
||||||
|
|
||||||
|
|
||||||
def effective_tool_approvals(tool_approvals, host_approval):
|
def effective_tool_approvals(tool_approvals, host_approval):
|
||||||
"""What host.py will actually answer per declared tool, today.
|
"""What host.py will actually answer per declared tool, today.
|
||||||
|
|
||||||
|
|||||||
@@ -414,6 +414,37 @@ class RoutingPolicy:
|
|||||||
"""Effective mode for one tier: explicit tier mode, else global."""
|
"""Effective mode for one tier: explicit tier mode, else global."""
|
||||||
return self.tier_modes.get(tier, self.mode)
|
return self.tier_modes.get(tier, self.mode)
|
||||||
|
|
||||||
|
def content_fingerprint(self):
|
||||||
|
"""Content fingerprint of the parsed policy (9.5-code).
|
||||||
|
|
||||||
|
sha256 of the canonical JSON (sorted keys, tight
|
||||||
|
separators) over the fields that decide routing and
|
||||||
|
capability checks: mode, tiers, tier_modes,
|
||||||
|
tier_requirements, model_capabilities, fallback_models.
|
||||||
|
The operator-declared ``version`` string is deliberately
|
||||||
|
NOT part of the fingerprint: it is hand-filled and may
|
||||||
|
stay unchanged while requires change, so it cannot be
|
||||||
|
trusted to detect a change. The fingerprint is only for
|
||||||
|
dry-run output and debugging; it is never written into
|
||||||
|
worker_routing audit rows, so the shadow parser (which
|
||||||
|
splits audit detail on whitespace) is untouched.
|
||||||
|
Two policies that parse to the same effective fields
|
||||||
|
share a fingerprint even if their raw documents differ
|
||||||
|
in key order or in dropped/invalid tiers.
|
||||||
|
"""
|
||||||
|
import hashlib
|
||||||
|
payload = {
|
||||||
|
"mode": self.mode,
|
||||||
|
"tiers": self.tiers,
|
||||||
|
"tier_modes": self.tier_modes,
|
||||||
|
"tier_requirements": self.tier_requirements,
|
||||||
|
"model_capabilities": self.model_capabilities,
|
||||||
|
"fallback_models": self.fallback_models,
|
||||||
|
}
|
||||||
|
canonical = json.dumps(payload, sort_keys=True,
|
||||||
|
separators=(",", ":"))
|
||||||
|
return hashlib.sha256(canonical.encode("utf-8")).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
def parse_routing_policy(data):
|
def parse_routing_policy(data):
|
||||||
"""Parse one policy document, or return None when it is unusable.
|
"""Parse one policy document, or return None when it is unusable.
|
||||||
@@ -538,6 +569,15 @@ def load_routing_policy(env=None):
|
|||||||
" worker default", "; ".join(failures))
|
" worker default", "; ".join(failures))
|
||||||
return None
|
return None
|
||||||
if failures:
|
if failures:
|
||||||
|
# 9.5-code fail-open 清单:部分失败是 fail-open——
|
||||||
|
# 可用来源继续生效、失败来源被丢弃,派发不炸。
|
||||||
|
# 可见性只有这条警告日志:load_routing_policy 仍
|
||||||
|
# 返回一个 policy,worker 的
|
||||||
|
# routing_policy_load_failed 标志只在全失败时
|
||||||
|
# 置位,所以部分失败不会产生任务级审计;运营方
|
||||||
|
# 只能从 worker 日志看到它,不能从审计里查到。
|
||||||
|
# 这是已知缺口(非静默,但不如全失败可见),
|
||||||
|
# 改动返回类型会牵动全部调用方,本轨不改。
|
||||||
_logger.warning(
|
_logger.warning(
|
||||||
"routing policy partially failed (%s); continuing with"
|
"routing policy partially failed (%s); continuing with"
|
||||||
" the usable source", "; ".join(failures))
|
" the usable source", "; ".join(failures))
|
||||||
|
|||||||
@@ -118,6 +118,7 @@ def dry_run_policy_document(data, env_mode_override=None,
|
|||||||
if not isinstance(data, dict):
|
if not isinstance(data, dict):
|
||||||
return {
|
return {
|
||||||
"valid": False, "blocked": True, "version": None,
|
"valid": False, "blocked": True, "version": None,
|
||||||
|
"content_fingerprint": None,
|
||||||
"global_mode": None, "global_mode_source": None,
|
"global_mode": None, "global_mode_source": None,
|
||||||
"tiers": tiers_report, "dropped_tiers": dropped,
|
"tiers": tiers_report, "dropped_tiers": dropped,
|
||||||
"unknown_tiers": unknown,
|
"unknown_tiers": unknown,
|
||||||
@@ -141,6 +142,7 @@ def dry_run_policy_document(data, env_mode_override=None,
|
|||||||
"entirely and workers would keep the worker default")
|
"entirely and workers would keep the worker default")
|
||||||
return {
|
return {
|
||||||
"valid": False, "blocked": True, "version": None,
|
"valid": False, "blocked": True, "version": None,
|
||||||
|
"content_fingerprint": None,
|
||||||
"global_mode": None, "global_mode_source": None,
|
"global_mode": None, "global_mode_source": None,
|
||||||
"tiers": tiers_report, "dropped_tiers": dropped,
|
"tiers": tiers_report, "dropped_tiers": dropped,
|
||||||
"unknown_tiers": unknown, "warnings": warnings,
|
"unknown_tiers": unknown, "warnings": warnings,
|
||||||
@@ -311,7 +313,9 @@ def dry_run_policy_document(data, env_mode_override=None,
|
|||||||
blocked = bool(dropped or unknown or cap_blocked or _gw_blocked)
|
blocked = bool(dropped or unknown or cap_blocked or _gw_blocked)
|
||||||
return {
|
return {
|
||||||
"valid": True, "blocked": blocked,
|
"valid": True, "blocked": blocked,
|
||||||
"version": policy.version, "global_mode": global_mode,
|
"version": policy.version,
|
||||||
|
"content_fingerprint": policy.content_fingerprint(),
|
||||||
|
"global_mode": global_mode,
|
||||||
"global_mode_source": global_source,
|
"global_mode_source": global_source,
|
||||||
"tiers": tiers_report, "dropped_tiers": dropped,
|
"tiers": tiers_report, "dropped_tiers": dropped,
|
||||||
"unknown_tiers": unknown, "warnings": warnings,
|
"unknown_tiers": unknown, "warnings": warnings,
|
||||||
|
|||||||
@@ -68,6 +68,12 @@ def parse_worker_routing_detail(detail):
|
|||||||
|
|
||||||
Old format (no suggested/chosen) parses with both models None.
|
Old format (no suggested/chosen) parses with both models None.
|
||||||
A ``-`` placeholder also means None, never a model named "-".
|
A ``-`` placeholder also means None, never a model named "-".
|
||||||
|
9.5-code fail-open 清单:解析失败的行返回 None 后在
|
||||||
|
evaluate_shadow 里直接跳过、不计数、不报错——样本无
|
||||||
|
声消失(例如模型名含空白会使整行匹配失败,见 M7)。
|
||||||
|
这是已知缺口:改返回类型会牵动评估函数与端点,本轨
|
||||||
|
只在此处写明,不改语义;运营方无法从报告里看到丢
|
||||||
|
了多少行,只能靠审计行总数与样本数对不上来发现。
|
||||||
"""
|
"""
|
||||||
if not isinstance(detail, str):
|
if not isinstance(detail, str):
|
||||||
return None
|
return None
|
||||||
|
|||||||
+11
-22
@@ -60,6 +60,8 @@ from urllib.parse import parse_qs, quote, urlparse
|
|||||||
from cloud.agent_def import default_agents_dir as _agent_default_dir
|
from cloud.agent_def import default_agents_dir as _agent_default_dir
|
||||||
from cloud.agent_def import list_known_skills as _agent_known_skills
|
from cloud.agent_def import list_known_skills as _agent_known_skills
|
||||||
from cloud.agent_def import parse_agent_definition as _parse_agent_definition
|
from cloud.agent_def import parse_agent_definition as _parse_agent_definition
|
||||||
|
from cloud.agent_def import parse_agent_tool_approvals_column as _parse_tool_approvals_col
|
||||||
|
from cloud.agent_def import parse_agent_tools_column as _parse_tools_col
|
||||||
from cloud.agent_def import resolve_registry_agent as _resolve_agent
|
from cloud.agent_def import resolve_registry_agent as _resolve_agent
|
||||||
from cloud.attempts import ensure_attempts_table as _ensure_attempts
|
from cloud.attempts import ensure_attempts_table as _ensure_attempts
|
||||||
from cloud.auth import ROLES as _MEMBER_ROLES
|
from cloud.auth import ROLES as _MEMBER_ROLES
|
||||||
@@ -1798,32 +1800,19 @@ def _row_get(row, name):
|
|||||||
|
|
||||||
|
|
||||||
def _agent_tools_from_row(row):
|
def _agent_tools_from_row(row):
|
||||||
"""agent_tools 列的容错解析:JSON 数组,非数组/脏数据为 None。"""
|
"""agent_tools 列的容错解析:唯一实现在
|
||||||
raw = _row_get(row, "agent_tools")
|
agent_def.parse_agent_tools_column(9.5-code 统一);本
|
||||||
if raw is None:
|
函数只取列值并转发,缺列/脏数据为 None(= 未声明),
|
||||||
return None
|
序列化口径与统一前逐字一致。"""
|
||||||
if isinstance(raw, list):
|
return _parse_tools_col(_row_get(row, "agent_tools"))
|
||||||
return list(raw)
|
|
||||||
try:
|
|
||||||
parsed = json.loads(raw)
|
|
||||||
except (TypeError, ValueError):
|
|
||||||
return None
|
|
||||||
return list(parsed) if isinstance(parsed, list) else None
|
|
||||||
|
|
||||||
|
|
||||||
def _agent_tool_approvals_from_row(row):
|
def _agent_tool_approvals_from_row(row):
|
||||||
"""agent_tool_approvals 列的容错解析:JSON 对象,非对象/脏
|
"""agent_tool_approvals 列的容错解析:唯一实现在
|
||||||
|
agent_def.parse_agent_tool_approvals_column;缺列/脏
|
||||||
数据为 None(NULL = 未声明,与 v3 tools 同款语义)。"""
|
数据为 None(NULL = 未声明,与 v3 tools 同款语义)。"""
|
||||||
raw = _row_get(row, "agent_tool_approvals")
|
return _parse_tool_approvals_col(
|
||||||
if raw is None:
|
_row_get(row, "agent_tool_approvals"))
|
||||||
return None
|
|
||||||
if isinstance(raw, dict):
|
|
||||||
return dict(raw)
|
|
||||||
try:
|
|
||||||
parsed = json.loads(raw)
|
|
||||||
except (TypeError, ValueError):
|
|
||||||
return None
|
|
||||||
return dict(parsed) if isinstance(parsed, dict) else None
|
|
||||||
|
|
||||||
|
|
||||||
def _task_to_dict(row, attachments=None, total_tokens=None):
|
def _task_to_dict(row, attachments=None, total_tokens=None):
|
||||||
|
|||||||
@@ -0,0 +1,321 @@
|
|||||||
|
"""9.5-code track: code-reachable items only.
|
||||||
|
|
||||||
|
- Parse unification: agent_tools / tool_approvals column parsing
|
||||||
|
lives only in agent_def.py; server keeps None (undeclared),
|
||||||
|
worker maps None to [] / {} at the _task_from_row boundary.
|
||||||
|
- Goal branch of _apply_resume_preamble is reachable (via
|
||||||
|
_goal_convert_queued on a queued auto-resumed task), so it is
|
||||||
|
kept and tested, not deleted as dead code.
|
||||||
|
- Fail-open inventory: audit write failure and preamble lookup
|
||||||
|
failure log a warning without breaking dispatch.
|
||||||
|
- M7: shadow endpoint explicit_capability_failures_truncated flag.
|
||||||
|
- Policy content fingerprint: stable for the same document,
|
||||||
|
changes when requires change, never touches audit rows.
|
||||||
|
|
||||||
|
Run: python3 -m unittest discover -s cloud/tests -p "test_95_code.py" -v
|
||||||
|
No real models, no network, no real gateway, no real PG.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import http.client
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sqlite3
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import threading
|
||||||
|
import time
|
||||||
|
import unittest
|
||||||
|
from types import SimpleNamespace
|
||||||
|
from unittest import mock
|
||||||
|
|
||||||
|
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", ".."))
|
||||||
|
|
||||||
|
import cloud.worker as worker_module
|
||||||
|
from cloud.agent_def import (
|
||||||
|
parse_agent_tool_approvals_column,
|
||||||
|
parse_agent_tools_column,
|
||||||
|
)
|
||||||
|
from cloud.attempts import ensure_attempts_table, record_attempt
|
||||||
|
from cloud.auth import AuthStore
|
||||||
|
from cloud.models import parse_routing_policy
|
||||||
|
from cloud.routing_dryrun import dry_run_policy_document
|
||||||
|
from cloud.server import (
|
||||||
|
_agent_tool_approvals_from_row,
|
||||||
|
_agent_tools_from_row,
|
||||||
|
create_server,
|
||||||
|
init_db,
|
||||||
|
)
|
||||||
|
from cloud.worker import (
|
||||||
|
Worker,
|
||||||
|
_agent_tool_approvals_from_col,
|
||||||
|
_agent_tools_from_col,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class ParseUnificationTestCase(unittest.TestCase):
|
||||||
|
def test_shared_parser_is_single_source(self):
|
||||||
|
# Missing / dirty / wrong-type all parse to None.
|
||||||
|
for raw in (None, "not json", '{"a": 1}', '["a", 1]', 42):
|
||||||
|
if raw == '["a", 1]':
|
||||||
|
# A JSON list parses even if items are not strings;
|
||||||
|
# the parser only guarantees list shape, not items.
|
||||||
|
self.assertEqual(parse_agent_tools_column(raw),
|
||||||
|
["a", 1])
|
||||||
|
continue
|
||||||
|
self.assertIsNone(parse_agent_tools_column(raw))
|
||||||
|
for raw in (None, "not json", '["a"]', '{"a": 1}', 42):
|
||||||
|
if raw == '{"a": 1}':
|
||||||
|
self.assertEqual(
|
||||||
|
parse_agent_tool_approvals_column(raw), {"a": 1})
|
||||||
|
continue
|
||||||
|
self.assertIsNone(parse_agent_tool_approvals_column(raw))
|
||||||
|
# Valid values are copied, not aliased.
|
||||||
|
src = ["websearch"]
|
||||||
|
self.assertEqual(parse_agent_tools_column(src), src)
|
||||||
|
self.assertIsNot(parse_agent_tools_column(src), src)
|
||||||
|
src = {"websearch": "ask"}
|
||||||
|
self.assertEqual(parse_agent_tool_approvals_column(src), src)
|
||||||
|
self.assertIsNot(parse_agent_tool_approvals_column(src), src)
|
||||||
|
|
||||||
|
def test_server_keeps_none_worker_maps_to_empty(self):
|
||||||
|
# Server serialization: missing/dirty stays None (undeclared).
|
||||||
|
self.assertIsNone(_agent_tools_from_row({}))
|
||||||
|
self.assertIsNone(_agent_tools_from_row({"agent_tools": "junk"}))
|
||||||
|
self.assertEqual(
|
||||||
|
_agent_tools_from_row({"agent_tools": '["websearch"]'}),
|
||||||
|
["websearch"])
|
||||||
|
self.assertIsNone(_agent_tool_approvals_from_row({}))
|
||||||
|
self.assertIsNone(
|
||||||
|
_agent_tool_approvals_from_row(
|
||||||
|
{"agent_tool_approvals": "junk"}))
|
||||||
|
# Worker dispatch: same parse, None mapped to [] / {}.
|
||||||
|
self.assertEqual(_agent_tools_from_col(None), [])
|
||||||
|
self.assertEqual(_agent_tools_from_col("junk"), [])
|
||||||
|
self.assertEqual(_agent_tools_from_col('["websearch"]'),
|
||||||
|
["websearch"])
|
||||||
|
self.assertEqual(_agent_tool_approvals_from_col(None), {})
|
||||||
|
self.assertEqual(_agent_tool_approvals_from_col("junk"), {})
|
||||||
|
self.assertEqual(
|
||||||
|
_agent_tool_approvals_from_col('{"websearch": "ask"}'),
|
||||||
|
{"websearch": "ask"})
|
||||||
|
|
||||||
|
|
||||||
|
def _insert_task(db_path, task_id, prompt="做活", status="queued",
|
||||||
|
project_id="p1", **cols):
|
||||||
|
now = time.time()
|
||||||
|
conn = sqlite3.connect(db_path)
|
||||||
|
try:
|
||||||
|
base = {"id": task_id, "project_id": project_id,
|
||||||
|
"prompt": prompt, "status": status,
|
||||||
|
"idempotency_key": None, "depends_on": "[]",
|
||||||
|
"priority": 0, "created_by": "tester",
|
||||||
|
"created_at": now, "updated_at": now,
|
||||||
|
"last_heartbeat": None}
|
||||||
|
base.update(cols)
|
||||||
|
keys = list(base.keys())
|
||||||
|
conn.execute(
|
||||||
|
"INSERT INTO tasks(%s) VALUES(%s)"
|
||||||
|
% (", ".join(keys), ", ".join("?" * len(keys))),
|
||||||
|
[base[k] for k in keys])
|
||||||
|
conn.commit()
|
||||||
|
finally:
|
||||||
|
conn.close()
|
||||||
|
|
||||||
|
|
||||||
|
class WorkerCase(unittest.TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
self.tmp = tempfile.TemporaryDirectory()
|
||||||
|
self.db_path = os.path.join(self.tmp.name, "control.db")
|
||||||
|
init_db(self.db_path)
|
||||||
|
self.env = {"XCM_API_KEY": "test-placeholder"}
|
||||||
|
|
||||||
|
def tearDown(self):
|
||||||
|
self.tmp.cleanup()
|
||||||
|
|
||||||
|
def _worker(self):
|
||||||
|
def _runner(cmd):
|
||||||
|
return SimpleNamespace(returncode=0, stdout="{}",
|
||||||
|
stderr="")
|
||||||
|
return Worker(self.db_path, "/fake/host.py",
|
||||||
|
runner=_runner, env=self.env)
|
||||||
|
|
||||||
|
|
||||||
|
class GoalBranchTestCase(WorkerCase):
|
||||||
|
def test_goal_branch_reached_for_converted_goal_task(self):
|
||||||
|
# Reachability proof (9.5-code): a non-goal task auto-resumed
|
||||||
|
# as platform_fresh (count=1, queued) that is then converted
|
||||||
|
# to a goal while still queued reaches the goal branch:
|
||||||
|
# no _resume (resume_message was cleared by auto_resume),
|
||||||
|
# no _goal_resume_breakpoint, count>0, latest attempt fresh.
|
||||||
|
_insert_task(self.db_path, "t1", prompt="做活",
|
||||||
|
goal="做目标", goal_mode=1, auto_resume_count=1)
|
||||||
|
conn = sqlite3.connect(self.db_path)
|
||||||
|
try:
|
||||||
|
ensure_attempts_table(conn)
|
||||||
|
record_attempt(conn, "t1", 1, "platform_fresh",
|
||||||
|
"worker_shutdown", -1,
|
||||||
|
"做到一半:已建 a.py", "被平台杀掉")
|
||||||
|
conn.commit()
|
||||||
|
finally:
|
||||||
|
conn.close()
|
||||||
|
worker = self._worker()
|
||||||
|
task = worker.get_task("t1")
|
||||||
|
injected = worker._apply_resume_preamble(task)
|
||||||
|
self.assertIn("上次尝试说明", injected["goal"])
|
||||||
|
self.assertTrue(injected["goal"].endswith("做目标"))
|
||||||
|
self.assertIn("上次尝试说明", injected["prompt"])
|
||||||
|
self.assertTrue(injected["prompt"].endswith("做活"))
|
||||||
|
# Memory only: the stored goal is untouched.
|
||||||
|
conn = sqlite3.connect(self.db_path)
|
||||||
|
try:
|
||||||
|
row = conn.execute("SELECT goal FROM tasks WHERE id = ?",
|
||||||
|
("t1",)).fetchone()
|
||||||
|
finally:
|
||||||
|
conn.close()
|
||||||
|
self.assertEqual(row[0], "做目标")
|
||||||
|
|
||||||
|
|
||||||
|
class FailOpenLogTestCase(WorkerCase):
|
||||||
|
def test_audit_failure_logs_without_breaking_dispatch(self):
|
||||||
|
worker = self._worker()
|
||||||
|
calls = []
|
||||||
|
with mock.patch.object(
|
||||||
|
worker_module, "AuthStore",
|
||||||
|
side_effect=RuntimeError("db down")), \
|
||||||
|
mock.patch.object(
|
||||||
|
worker_module, "log_event",
|
||||||
|
side_effect=lambda *a, **k: calls.append(a[1])):
|
||||||
|
# Must not raise: audit is best-effort by design.
|
||||||
|
worker._audit("p1", "worker_claim", "t1")
|
||||||
|
self.assertEqual(calls, ["worker_audit_failed"])
|
||||||
|
|
||||||
|
def test_preamble_lookup_failure_logs_and_returns_task(self):
|
||||||
|
worker = self._worker()
|
||||||
|
task = {"id": "t1", "project_id": "p1", "prompt": "做活",
|
||||||
|
"auto_resume_count": 1}
|
||||||
|
calls = []
|
||||||
|
with mock.patch.object(
|
||||||
|
worker_module, "_db_connect",
|
||||||
|
side_effect=RuntimeError("db down")), \
|
||||||
|
mock.patch.object(
|
||||||
|
worker_module, "log_event",
|
||||||
|
side_effect=lambda *a, **k: calls.append(a[1])):
|
||||||
|
out = worker._apply_resume_preamble(task)
|
||||||
|
self.assertIs(out["prompt"], "做活")
|
||||||
|
self.assertEqual(
|
||||||
|
calls, ["worker_resume_preamble_lookup_failed"])
|
||||||
|
|
||||||
|
|
||||||
|
class FingerprintTestCase(unittest.TestCase):
|
||||||
|
def test_stable_and_sensitive_to_requires(self):
|
||||||
|
doc = {"mode": "shadow",
|
||||||
|
"tiers": {"default": "m1"},
|
||||||
|
"model_capabilities": {"m1": {"tool_calling": True}}}
|
||||||
|
a = parse_routing_policy(doc)
|
||||||
|
b = parse_routing_policy(
|
||||||
|
{"model_capabilities": {"m1": {"tool_calling": True}},
|
||||||
|
"tiers": {"default": "m1"}, "mode": "shadow"})
|
||||||
|
self.assertEqual(a.content_fingerprint(),
|
||||||
|
b.content_fingerprint())
|
||||||
|
c = parse_routing_policy({
|
||||||
|
"mode": "shadow",
|
||||||
|
"tiers": {"default": {"model": "m1",
|
||||||
|
"requires": {"tool_calling": True}}},
|
||||||
|
"model_capabilities": {"m1": {"tool_calling": True}}})
|
||||||
|
self.assertNotEqual(a.content_fingerprint(),
|
||||||
|
c.content_fingerprint())
|
||||||
|
# Operator version string does not affect the fingerprint.
|
||||||
|
d = parse_routing_policy(dict(doc, version="v99"))
|
||||||
|
self.assertEqual(a.content_fingerprint(),
|
||||||
|
d.content_fingerprint())
|
||||||
|
# Dry-run surfaces it; audit rows never carry it.
|
||||||
|
report = dry_run_policy_document(doc)
|
||||||
|
self.assertEqual(report["content_fingerprint"],
|
||||||
|
a.content_fingerprint())
|
||||||
|
|
||||||
|
|
||||||
|
class _HttpServerCase(unittest.TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
self._old = {k: os.environ.get(k) for k in (
|
||||||
|
"CLOUD_ROUTING_POLICY_JSON", "CLOUD_ROUTING_POLICY_FILE")}
|
||||||
|
os.environ.pop("CLOUD_ROUTING_POLICY_JSON", None)
|
||||||
|
os.environ.pop("CLOUD_ROUTING_POLICY_FILE", None)
|
||||||
|
self.tmp = tempfile.TemporaryDirectory()
|
||||||
|
self.db_path = os.path.join(self.tmp.name, "control.db")
|
||||||
|
self.auth = AuthStore(self.db_path)
|
||||||
|
self.owner_id, self.owner_key = self.auth.create_user("owner")
|
||||||
|
self.server = create_server(self.db_path, "127.0.0.1", 0)
|
||||||
|
self.port = self.server.server_address[1]
|
||||||
|
self.thread = threading.Thread(
|
||||||
|
target=self.server.serve_forever, daemon=True)
|
||||||
|
self.thread.start()
|
||||||
|
|
||||||
|
def tearDown(self):
|
||||||
|
self.server.shutdown()
|
||||||
|
self.server.server_close()
|
||||||
|
self.thread.join(timeout=5)
|
||||||
|
for k, v in self._old.items():
|
||||||
|
if v is None:
|
||||||
|
os.environ.pop(k, None)
|
||||||
|
else:
|
||||||
|
os.environ[k] = v
|
||||||
|
try:
|
||||||
|
self.auth.conn.close()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
self.tmp.cleanup()
|
||||||
|
|
||||||
|
def _req(self, method, path, body=None, key=None):
|
||||||
|
conn = http.client.HTTPConnection(
|
||||||
|
"127.0.0.1", self.port, timeout=10)
|
||||||
|
hdrs = {}
|
||||||
|
payload = None
|
||||||
|
if body is not None:
|
||||||
|
payload = json.dumps(body).encode("utf-8")
|
||||||
|
hdrs["Content-Type"] = "application/json"
|
||||||
|
if key is not None:
|
||||||
|
hdrs["X-API-Key"] = key
|
||||||
|
try:
|
||||||
|
conn.request(method, path, body=payload, headers=hdrs)
|
||||||
|
resp = conn.getresponse()
|
||||||
|
data = resp.read()
|
||||||
|
finally:
|
||||||
|
conn.close()
|
||||||
|
try:
|
||||||
|
parsed = json.loads(data.decode("utf-8")) if data else None
|
||||||
|
except ValueError:
|
||||||
|
parsed = None
|
||||||
|
return resp.status, parsed
|
||||||
|
|
||||||
|
def _project(self, name="p"):
|
||||||
|
status, project = self._req(
|
||||||
|
"POST", "/projects", {"name": name, "auto_pr": 1},
|
||||||
|
self.owner_key)
|
||||||
|
self.assertEqual(status, 201, project)
|
||||||
|
return project["id"]
|
||||||
|
|
||||||
|
|
||||||
|
class ExplicitTruncationTestCase(_HttpServerCase):
|
||||||
|
def test_explicit_failures_truncated_flag(self):
|
||||||
|
pid = self._project()
|
||||||
|
conn = sqlite3.connect(self.db_path)
|
||||||
|
try:
|
||||||
|
conn.executemany(
|
||||||
|
"INSERT INTO audit_log(user_id, project_id, action,"
|
||||||
|
" detail, created_at) VALUES(?, ?, ?, ?, ?)",
|
||||||
|
[("tester", pid,
|
||||||
|
"worker_capability_gateway_explicit",
|
||||||
|
"E1 model=m1 tier=default mode=active"
|
||||||
|
" missing=tool_calling unknown=False blocked=True",
|
||||||
|
time.time())] * 10001)
|
||||||
|
conn.commit()
|
||||||
|
finally:
|
||||||
|
conn.close()
|
||||||
|
status, report = self._req(
|
||||||
|
"GET", "/routing-shadow?project_id=" + pid,
|
||||||
|
key=self.owner_key)
|
||||||
|
self.assertEqual(status, 200, report)
|
||||||
|
self.assertTrue(
|
||||||
|
report["explicit_capability_failures_truncated"])
|
||||||
|
self.assertEqual(
|
||||||
|
len(report["explicit_capability_failures"]), 10000)
|
||||||
+68
-34
@@ -137,6 +137,8 @@ from types import SimpleNamespace
|
|||||||
from cloud.agent_def import AgentDefinition as _AgentDefinition
|
from cloud.agent_def import AgentDefinition as _AgentDefinition
|
||||||
from cloud.agent_def import format_instructions_header as _agent_header
|
from cloud.agent_def import format_instructions_header as _agent_header
|
||||||
from cloud.agent_def import missing_required_tools as _missing_tools
|
from cloud.agent_def import missing_required_tools as _missing_tools
|
||||||
|
from cloud.agent_def import parse_agent_tool_approvals_column as _parse_tool_approvals_col
|
||||||
|
from cloud.agent_def import parse_agent_tools_column as _parse_tools_col
|
||||||
from cloud.attempts import build_resume_preamble as _resume_preamble
|
from cloud.attempts import build_resume_preamble as _resume_preamble
|
||||||
from cloud.attempts import ensure_attempts_table as _ensure_attempts
|
from cloud.attempts import ensure_attempts_table as _ensure_attempts
|
||||||
from cloud.attempts import latest_attempt as _latest_attempt
|
from cloud.attempts import latest_attempt as _latest_attempt
|
||||||
@@ -362,30 +364,22 @@ def _resolve_model(model=None, config_path=None):
|
|||||||
|
|
||||||
|
|
||||||
def _agent_tools_from_col(raw):
|
def _agent_tools_from_col(raw):
|
||||||
"""agent_tools 列的容错解析:JSON 数组,非数组/脏数据为 []。"""
|
"""agent_tools 列的容错解析:唯一实现在
|
||||||
if raw is None:
|
agent_def.parse_agent_tools_column(9.5-code 统一);本
|
||||||
return []
|
函数只把解析层的 None(= 未声明)映射为派发默认值
|
||||||
if isinstance(raw, list):
|
[]——worker 每个使用点本来就读 ``or []``,所以映射后
|
||||||
return list(raw)
|
的派发行为与统一前逐字一致,缺列/脏数据不炸认领。"""
|
||||||
try:
|
parsed = _parse_tools_col(raw)
|
||||||
parsed = json.loads(raw)
|
return parsed if parsed is not None else []
|
||||||
except (TypeError, ValueError):
|
|
||||||
return []
|
|
||||||
return list(parsed) if isinstance(parsed, list) else []
|
|
||||||
|
|
||||||
|
|
||||||
def _agent_tool_approvals_from_col(raw):
|
def _agent_tool_approvals_from_col(raw):
|
||||||
"""agent_tool_approvals 列的容错解析:JSON 对象,非对象/
|
"""agent_tool_approvals 列的容错解析:唯一实现在
|
||||||
脏数据为 {}(= 未声明),不炸认领。"""
|
agent_def.parse_agent_tool_approvals_column;解析层的
|
||||||
if raw is None:
|
None(= 未声明)映射为派发默认值 {},理由同上(每个
|
||||||
return {}
|
使用点本来就读 ``or {}``),派发行为与统一前一致。"""
|
||||||
if isinstance(raw, dict):
|
parsed = _parse_tool_approvals_col(raw)
|
||||||
return dict(raw)
|
return parsed if parsed is not None else {}
|
||||||
try:
|
|
||||||
parsed = json.loads(raw)
|
|
||||||
except (TypeError, ValueError):
|
|
||||||
return {}
|
|
||||||
return dict(parsed) if isinstance(parsed, dict) else {}
|
|
||||||
|
|
||||||
|
|
||||||
def _task_from_row(row):
|
def _task_from_row(row):
|
||||||
@@ -1772,7 +1766,14 @@ class Worker:
|
|||||||
conn.close()
|
conn.close()
|
||||||
|
|
||||||
def _audit(self, project_id, action, detail=""):
|
def _audit(self, project_id, action, detail=""):
|
||||||
"""best-effort 审计:失败不影响任务状态。"""
|
"""best-effort 审计:失败不影响任务状态(9.5-code
|
||||||
|
fail-open 清单第 1 项)。
|
||||||
|
|
||||||
|
此处是 fail-open:审计写失败时派发照常进行,运营
|
||||||
|
方只能从这条警告日志看到审计丢了——审计失败绝不
|
||||||
|
炸派发(语义不变),但失败本身不再静默。日志调用
|
||||||
|
本身也加 guard,日志失败时仍不炸派发。
|
||||||
|
"""
|
||||||
try:
|
try:
|
||||||
store = AuthStore(self.db_path)
|
store = AuthStore(self.db_path)
|
||||||
try:
|
try:
|
||||||
@@ -1782,8 +1783,14 @@ class Worker:
|
|||||||
store.conn.close()
|
store.conn.close()
|
||||||
except Exception:
|
except Exception:
|
||||||
pass
|
pass
|
||||||
except Exception:
|
except Exception as exc:
|
||||||
pass
|
try:
|
||||||
|
log_event(self.logger, "worker_audit_failed",
|
||||||
|
project_id=project_id, status="warning",
|
||||||
|
message="audit write failed (%s): %s"
|
||||||
|
% (action, exc))
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
def _notify_task_finished(self, task_id, project_id, event_type,
|
def _notify_task_finished(self, task_id, project_id, event_type,
|
||||||
error=None):
|
error=None):
|
||||||
@@ -2013,8 +2020,8 @@ class Worker:
|
|||||||
是模型最先看到的一段之一;位置是刻意的前置(重
|
是模型最先看到的一段之一;位置是刻意的前置(重
|
||||||
排说明应当先被看到),不是紧贴原 prompt。goal
|
排说明应当先被看到),不是紧贴原 prompt。goal
|
||||||
任务若意外走到这里也同步贴 goal(执行器取 goal
|
任务若意外走到这里也同步贴 goal(执行器取 goal
|
||||||
优先于 prompt);该分支正常流程不可达(见下),
|
优先于 prompt);goal 分支在正常流程可达(见下),
|
||||||
只是防御性保留。查账本失败时原样返回,不炸认领。
|
不是死代码,绝不能删。查账本失败时原样返回,不炸认领。
|
||||||
"""
|
"""
|
||||||
if task.get("coordinator_parts"):
|
if task.get("coordinator_parts"):
|
||||||
return task
|
return task
|
||||||
@@ -2028,7 +2035,20 @@ class Worker:
|
|||||||
attempt = _latest_attempt(conn, task["id"])
|
attempt = _latest_attempt(conn, task["id"])
|
||||||
finally:
|
finally:
|
||||||
conn.close()
|
conn.close()
|
||||||
except Exception:
|
except Exception as exc:
|
||||||
|
# 9.5-code fail-open 清单:账本查不到时不贴前言、
|
||||||
|
# 不炸认领(语义不变),但失败记一条警告日志,
|
||||||
|
# 否则“前言丢了”在运行期不可见。
|
||||||
|
try:
|
||||||
|
log_event(self.logger,
|
||||||
|
"worker_resume_preamble_lookup_failed",
|
||||||
|
task_id=task["id"],
|
||||||
|
project_id=task.get("project_id"),
|
||||||
|
status="warning",
|
||||||
|
message="resume preamble lookup failed: %s"
|
||||||
|
% exc)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
return task
|
return task
|
||||||
preamble = _resume_preamble(attempt)
|
preamble = _resume_preamble(attempt)
|
||||||
if not preamble:
|
if not preamble:
|
||||||
@@ -2037,13 +2057,19 @@ class Worker:
|
|||||||
new_task["prompt"] = preamble + "\n\n" + (
|
new_task["prompt"] = preamble + "\n\n" + (
|
||||||
task.get("prompt") or "")
|
task.get("prompt") or "")
|
||||||
if task.get("goal_mode") or task.get("verify_command"):
|
if task.get("goal_mode") or task.get("verify_command"):
|
||||||
# Audit fix 3(M8):此分支正常流程不可达——
|
# 9.5-code 已证实可达(非 M8 所说的防御性保留):
|
||||||
# build_resume_preamble 只对 platform_fresh 出
|
# platform_fresh 账本只由 auto_resume 给非 goal
|
||||||
# 前言,而 auto_resume 把 goal 任务一律归为
|
# 任务记(_record_attempt 唯一调用点),但同一
|
||||||
# goal_breakpoint;且 goal 断点任务在函数入口
|
# 任务随后可在 queued 状态下被
|
||||||
# 已被 _goal_resume_breakpoint 挡回。保留不删:
|
# server._goal_convert_queued 原地转成 goal
|
||||||
# 万一未来有 platform_fresh 的 goal 尝试,goal
|
# (goal_mode=1,CAS 只改 queued 行,不动
|
||||||
# 字段不同步贴前言会静默丢说明。
|
# auto_resume_count 与账本);认领时它既无
|
||||||
|
# _resume(auto_resume 已清 resume_message),
|
||||||
|
# 也无 _goal_resume_breakpoint,于是走到这里。
|
||||||
|
# 删掉本分支会让这类转 goal 任务静默丢前言,
|
||||||
|
# 故保留;复活路径(_goal_revive_terminal)因
|
||||||
|
# resume_message 置 _resume 而在入口被挡,不
|
||||||
|
# 走本分支,二者不要混淆。
|
||||||
new_task["goal"] = preamble + "\n\n" + (
|
new_task["goal"] = preamble + "\n\n" + (
|
||||||
task.get("goal") or task.get("prompt") or "")
|
task.get("goal") or task.get("prompt") or "")
|
||||||
new_task["_resume_preamble_injected"] = True
|
new_task["_resume_preamble_injected"] = True
|
||||||
@@ -5835,6 +5861,14 @@ class Worker:
|
|||||||
也就是写回的来源。
|
也就是写回的来源。
|
||||||
查找失败(DB 异常)时返回 False 并记警告日志:复核查
|
查找失败(DB 异常)时返回 False 并记警告日志:复核查
|
||||||
不到依据就跳过,而不是拦任务。
|
不到依据就跳过,而不是拦任务。
|
||||||
|
|
||||||
|
残留缺口措辞(9.5-code,M1):fixes3 时期由 shadow /
|
||||||
|
worker_default 决策写回的旧模型既无 persisted 行又
|
||||||
|
无 policy 行,启发式认不出来——这部分存量是固定
|
||||||
|
的、逐任务持续直至完成,不是“会收敛”(收敛一词
|
||||||
|
会误读为随时间自动消失;实际上每个存量任务在完
|
||||||
|
成前每次续跑都不经复核)。新任务不再产生这类写
|
||||||
|
回,所以缺口不扩大,但存量本身不会被重新识别。
|
||||||
"""
|
"""
|
||||||
try:
|
try:
|
||||||
conn = _db_connect(self.db_path)
|
conn = _db_connect(self.db_path)
|
||||||
|
|||||||
Reference in new issue
Block a user