feat: architecture conformance checks in CI and metrics feedback loop

- cloud/arch_conformance.py: contract-version (+1 only, approved
  change record), forbidden-directory (own dir only, .. and symlink
  escapes resolved), forbidden-import layer bans audited against the
  baseline (foundation/execution/driver, server sqlite3 Row adapters
  deliberately allowed), G2 frozen-stack swap guard; skill-project
  one-call check.
- cloud/metrics.py: first-pass/defect/rework rates over terminal
  tasks, unknown rates as None (never 0), #212 dual-column per-task
  cost (total None when incomplete), pure feedback rules
  (tighten/route-with-extra-audit/recalibrate, auto_apply False),
  golden-set gate: unapproved sample set is never decision-ready.
- GET /metrics-report (authenticated, member-scoped, same as
  /metrics): attempts/defects are None until an attempt ledger
  exists, so rates stay unknown rather than fabricated.
- Tests: 29 arch + 15 metrics + 3 endpoint, each check with
  positive and negative cases; repo-level scan pins zero layer
  violations on the baseline so a future violation fails CI via the
  existing host-verify discover run (no workflow change).
This commit is contained in:
host committed 2026-11-03 02:49:25 +00:00
1 parent 382ec6d62e
commit dba6536ac2
7 files changed
+1201

No files matched your search

+299
View File
@@ -0,0 +1,299 @@
"""Architecture-conformance checks for the control plane and for
skill-driven projects (software-engineering skill, G2 frozen artefacts).
Zero third-party dependencies; standard library only. This module
imports no ``cloud.*`` module, so it can never violate the layer bans
it enforces, and it can be reused verbatim against a skill project's
own directory tree (see ``check_skill_project``).
The four checks, each with a stated rule and a stated reason:
1. Contract version (``check_contract_version``). Rule mirrors the
skill's contract-freeze change protocol: after freeze, a version
change is allowed only with a coordinator-approved change record
that names backward compatibility, the affected clients, and (for
a breaking change) a regression task per affected client; the
version must advance by exactly one. Reason: a silent contract
drift is how parallel client threads diverge; the protocol is the
only sanctioned path, so the check enforces the protocol instead
of merely documenting it.
2. Forbidden directories (``is_within`` / ``check_task_writes``).
Rule reuses the two directory definitions that already exist and
invents no third one: on the control plane a task's own directory
is its tenant work directory (``worker._tenant_dirs`` ``work``);
in a skill project a client thread's own directory is its
``apps/<client>/`` tree. Any write outside the own directory --
including ``..`` escapes and symlink escapes, both resolved before
comparison -- is a violation. Reason: cross-directory writes are
how one tenant/client corrupts another's worktree.
3. Forbidden imports (``check_layer_imports``). Bans were chosen by
auditing the current tree, not by wishful layering; every ban
below passes on the baseline, and the repo-level test pins that:
- Foundation modules (``db``, ``models``, ``queue``,
``logging_config``) must not import orchestration/service
modules (``server``, ``worker``, ``pr_exec``, ``webhooks``,
``autoscaler``, ``cost``). Reason: the foundation is imported
by everything; an upward import is a cycle waiting to happen
and drags service state into pure helpers.
- The execution module (``sandbox``) must not import persistence
or orchestration (``db``, ``server``, ``worker``, ``cost``,
``auth``, ``pr``, ``pr_exec``, ``webhooks``). Reason: executors
must stay swappable and testable with fake runners; decisions
about mounts and networks arrive as arguments, not by reading
the database.
- No cloud module except the driver-adjacent ones (``db``,
``backup``, ``migrations``) may open a database connection
directly (``sqlite3.connect`` or any ``psycopg*`` import).
Service modules must obtain connections through
``cloud.db.connect`` or a Store. Reason: the backend switch
(sqlite/postgres) lives in exactly one place; a direct driver
call bypasses it. Deliberately NOT banned: ``server.py``'s use
of ``sqlite3.Row`` / ``sqlite3.IntegrityError`` as type adapters
on connections obtained from ``cloud.db`` -- that is adaptation,
not a bypass, and banning it would fail the baseline for no
safety gain. The test asserts the narrow form (no ``connect``
call, no driver import) rather than the broad form.
Only static ``ast`` imports are seen; dynamic imports
(``importlib``) are out of scope and are noted, not claimed.
4. Frozen stack (``check_frozen_stack``). Rule mirrors the skill's
G2 rule "frozen stack must not be swapped mid-flight": a client
whose stack entry changes, or disappears, after freeze fails
unless an approved change record covers it; adding a brand-new
client (no frozen entry) is an addition, not a swap, and passes.
Reason: the shared core is built against the frozen table; a
mid-flight swap silently invalidates it.
All checkers return plain data (dicts/lists), never raise for a
violation, so CI can report every violation in one run instead of
stopping at the first.
"""
import ast
import os
from pathlib import Path
FOUNDATION_MODULES = frozenset({"db", "models", "queue", "logging_config"})
ORCHESTRATION_MODULES = frozenset(
{"server", "worker", "pr_exec", "webhooks", "autoscaler", "cost"})
EXECUTION_MODULES = frozenset({"sandbox"})
EXECUTION_BANNED = frozenset(
{"db", "server", "worker", "cost", "auth", "pr", "pr_exec", "webhooks"})
DRIVER_ADJACENT_MODULES = frozenset({"db", "backup", "migrations"})
# ---------------------------------------------------------------- contract
def parse_version(text):
"""Parse ``v1`` / ``1`` to int; anything else is None (unparseable
is a failure, never silently treated as the frozen version)."""
if text is None:
return None
text = str(text).strip()
if text.startswith("v"):
text = text[1:]
if text.isdigit():
return int(text)
return None
def _change_record_problems(change, breaking):
"""Missing pieces of a contract-change record, per the protocol."""
problems = []
if not isinstance(change, dict):
return ["change record missing (frozen contract changed without "
"a coordinator-approved change record)"]
if not change.get("approved"):
problems.append("change not coordinator-approved")
if change.get("backward_compatible") is None:
problems.append("backward compatibility not declared")
affected = change.get("affected_clients")
if not affected or not isinstance(affected, list):
problems.append("affected clients not listed")
if breaking and not change.get("regression_task"):
problems.append("breaking change without a regression task")
return problems
def check_contract_version(frozen, current, change=None):
"""Return ``{"ok": bool, "reasons": [...]}``.
Equal versions pass regardless of ``change``. A different version
passes only when it is exactly frozen + 1 and ``change`` satisfies
the protocol; a breaking change is one where
``backward_compatible`` is False.
"""
frozen_n = parse_version(frozen)
current_n = parse_version(current)
if frozen_n is None or current_n is None:
return {"ok": False,
"reasons": ["unparseable contract version: frozen=%r "
"current=%r" % (frozen, current)]}
if current_n == frozen_n:
return {"ok": True, "reasons": []}
reasons = []
if current_n != frozen_n + 1:
reasons.append("contract version must advance by exactly one "
"(frozen v%d -> v%d, not v%d)"
% (frozen_n, frozen_n + 1, current_n))
breaking = isinstance(change, dict) and \
change.get("backward_compatible") is False
reasons.extend(_change_record_problems(change, breaking))
return {"ok": not reasons, "reasons": reasons}
# ------------------------------------------------------------- frozen stack
def check_frozen_stack(frozen, current, change=None):
"""``frozen``/``current`` map client -> stack string.
Changed or removed entries fail without an approved change record
(same shape as the contract-change record, minus the version
rules); added clients pass as additions.
"""
frozen = frozen or {}
current = current or {}
reasons = []
swapped = sorted(c for c in frozen
if c not in current or current[c] != frozen[c])
if not swapped:
return {"ok": True, "reasons": [], "swapped": []}
problems = _change_record_problems(change, breaking=False)
if problems:
reasons.append("frozen stack changed for %s without an approved "
"change record: %s" % (swapped, "; ".join(problems)))
return {"ok": not reasons, "reasons": reasons, "swapped": swapped}
# ------------------------------------------------------------ directories
def is_within(root, candidate):
"""True iff ``candidate`` resolves inside ``root`` (symlinks and
``..`` resolved first, so both escape routes are caught)."""
try:
root_p = Path(root).resolve()
cand_p = Path(candidate).resolve()
except (OSError, RuntimeError):
return False
return cand_p == root_p or root_p in cand_p.parents
def check_task_writes(own_dir, paths):
"""Return the subset of ``paths`` written outside ``own_dir``."""
return [p for p in paths if not is_within(own_dir, p)]
# ----------------------------------------------------------------- imports
def scan_imports(source):
"""Static import scan of one module's source.
Returns ``{"cloud": set, "driver_connect": bool, "driver_import": bool}``:
``cloud`` holds the ``cloud.<name>`` top-level names imported;
``driver_connect`` is True on any ``sqlite3.connect`` call;
``driver_import`` is True on any ``psycopg*`` import. A bare
``import sqlite3`` (Row/exception adapters) sets neither flag.
"""
cloud_imports = set()
driver_connect = False
driver_import = False
try:
tree = ast.parse(source)
except SyntaxError:
return {"cloud": cloud_imports,
"driver_connect": driver_connect,
"driver_import": driver_import,
"parse_error": True}
for node in ast.walk(tree):
if isinstance(node, ast.Import):
for alias in node.names:
if alias.name == "sqlite3":
continue
if alias.name.startswith("psycopg"):
driver_import = True
if alias.name == "cloud" or alias.name.startswith("cloud."):
parts = alias.name.split(".")
if len(parts) > 1:
cloud_imports.add(parts[1])
elif isinstance(node, ast.ImportFrom):
if node.module and node.module.startswith("psycopg"):
driver_import = True
if node.module == "cloud" or (node.module or "").startswith("cloud."):
if node.module == "cloud":
for alias in node.names:
cloud_imports.add(alias.name)
else:
cloud_imports.add(node.module.split(".")[1])
elif isinstance(node, ast.Call):
func = node.func
if isinstance(func, ast.Attribute) and func.attr == "connect":
value = func.value
if isinstance(value, ast.Name) and value.id == "sqlite3":
driver_connect = True
return {"cloud": cloud_imports,
"driver_connect": driver_connect,
"driver_import": driver_import}
def check_layer_imports(sources):
"""``sources`` maps module name -> source text. Returns a list of
violation dicts; empty means the layer rules hold."""
violations = []
for module, source in sorted(sources.items()):
scanned = scan_imports(source)
if scanned.get("parse_error"):
violations.append({"module": module, "rule": "parse",
"detail": "source does not parse; "
"imports cannot be verified"})
continue
if module in FOUNDATION_MODULES:
bad = scanned["cloud"] & ORCHESTRATION_MODULES
if bad:
violations.append({"module": module,
"rule": "foundation_must_not_import_orchestration",
"detail": sorted(bad)})
if module in EXECUTION_MODULES:
bad = scanned["cloud"] & EXECUTION_BANNED
if bad:
violations.append({"module": module,
"rule": "execution_must_not_import_persistence_or_orchestration",
"detail": sorted(bad)})
if module not in DRIVER_ADJACENT_MODULES:
if scanned["driver_connect"] or scanned["driver_import"]:
violations.append({"module": module,
"rule": "service_must_not_open_driver_connection_directly",
"detail": "use cloud.db.connect or a Store"})
return violations
def scan_cloud_package(repo_root):
"""Run ``check_layer_imports`` over every ``cloud/*.py`` in a repo
checkout (tests and caches excluded). This is the repo-level test
that makes a layer violation fail CI."""
cloud_dir = Path(repo_root) / "cloud"
sources = {}
if cloud_dir.is_dir():
for path in sorted(cloud_dir.glob("*.py")):
sources[path.stem] = path.read_text(encoding="utf-8")
return check_layer_imports(sources)
def check_skill_project(project_root, frozen_version, current_version,
frozen_stack, current_stack, own_dir, writes,
change=None):
"""One-call conformance for a skill project: contract version,
frozen stack, and directory writes together. Returns a dict with
one entry per check plus an overall ``ok``."""
contract = check_contract_version(frozen_version, current_version,
change=change)
stack = check_frozen_stack(frozen_stack, current_stack, change=change)
outside = check_task_writes(own_dir, writes)
directory = {"ok": not outside,
"reasons": (["writes outside own directory: %s" % outside]
if outside else [])}
return {"ok": contract["ok"] and stack["ok"] and directory["ok"],
"contract": contract, "stack": stack, "directory": directory}
+253
View File
@@ -0,0 +1,253 @@
"""Metrics collection and feedback loop for agent-task quality.
Zero third-party dependencies; standard library only. This module
imports no ``cloud.*`` module: it computes over plain record dicts so
the collection logic is testable without a database, and so the
report shape is identical whether the records came from the control
plane (``server._metrics_report``), a test fixture, or a future
attempt ledger.
Definitions (pinned here, not left to each caller):
- A task counts toward a rate only when it reached a terminal status
(``completed`` / ``merged`` / ``failed`` / ``merge-failed``).
``interrupted`` is deliberately NOT terminal: platform-interrupted
tasks auto-resume as fresh runs, so counting them as failures
would punish a platform event, not the agent.
- First-pass green: terminal task whose ``attempts`` == 1 and whose
status is ``completed`` / ``merged``.
- Defect: a task with ``defects`` > 0, where ``defects`` is the count
of failed CI/audit rounds or follow-up rounds for that task. The
current tasks schema has NO attempts/defects column, so a caller
that cannot supply them must pass ``None`` -- and the rates then
come back as ``None`` with ``*_known`` False. An unknown rate is
never reported as 0: 0 would claim "no defects", which is a
decision-grade lie.
- Rework: terminal task with ``attempts`` > 1. Same None discipline.
- Per-task cost reuses the #212 dual-column discipline from
``cloud.cost``: priced USD micros and unpriced actual tokens are
reported in separate columns. ``total_usd_micros`` is ``None``
whenever any task's cost is incomplete (no cost rows, or unpriced
actual tokens > 0); the priced partial sum is still shown, labelled
``priced_usd_micros_partial``, so nobody mistakes it for a total.
Unpriced cost is never folded into 0.
Feedback (``evaluate_feedback``) is deliberately a pure function
returning action records, not an auto-applier: an action says WHAT
should change and WHY, with its evidence attached; applying it to
routing or to the audit standards is a separate, approved step
(see ``docs/arch-conformance-metrics.md``). The rules are testable:
- defect rate for a category above threshold -> ``tighten_check``
for that category's check;
- first-pass rate below its floor -> ``route_with_extra_audit``;
- a check whose violations are overturned on review above
threshold -> ``recalibrate_check`` (review the threshold, do NOT
auto-apply a new one -- a check that fires wrongly must not
silently widen itself).
Thresholds passed in are labelled by source in the report. The
defaults in ``DEFAULT_THRESHOLDS`` are starting values for tests,
NOT human-approved production thresholds: the report marks them
``thresholds_source: "default_unapproved"`` until a human approves a
set, and a report built on unapproved thresholds is still usable
for collection -- it is just not decision-grade on its own.
Golden tasks: a report is ``decision_ready`` only when a golden set
has been human-approved (``approved_by`` + ``approved_at`` present
and at least one golden task listed). Until then the golden section
reads ``status: "pending_human_approval"`` and the report says so
plainly; the sample-only set that exists today must never be
presented as decision-ready. This module never invents golden tasks.
"""
import json
import time
TERMINAL_STATUSES = frozenset(
{"completed", "merged", "failed", "merge-failed"})
SUCCESS_STATUSES = frozenset({"completed", "merged"})
DEFAULT_THRESHOLDS = {
"defect_rate": 0.20,
"first_pass_rate_min": 0.50,
"check_overturn_rate": 0.30,
}
def _rate(numerator, denominator):
if denominator is None or denominator == 0:
return None
return numerator / denominator
def compute_metrics(records, costs=None):
"""``records``: list of dicts with keys ``id``, ``status`` and
optional ``attempts`` (int), ``defects`` (int),
``defect_category`` (str). ``costs``: dict task_id -> dict with
``priced_usd_micros`` (int|None), ``unpriced_actual_tokens`` (int),
``has_rows`` (bool). Returns the metrics dict."""
costs = costs or {}
terminal = [r for r in records if r.get("status") in TERMINAL_STATUSES]
known_attempts = [r for r in terminal if r.get("attempts") is not None]
first_pass = [r for r in known_attempts
if r["attempts"] == 1
and r.get("status") in SUCCESS_STATUSES]
rework = [r for r in known_attempts if r["attempts"] > 1]
known_defects = [r for r in terminal if r.get("defects") is not None]
defective = [r for r in known_defects if r["defects"] > 0]
defect_by_category = {}
for r in defective:
cat = r.get("defect_category") or "uncategorized"
defect_by_category[cat] = defect_by_category.get(cat, 0) + 1
attempts_total = sum(r["attempts"] for r in known_attempts)
per_task_cost = []
priced_partial = 0
cost_complete = True
for r in terminal:
c = costs.get(r["id"]) or {}
priced = c.get("priced_usd_micros")
unpriced = int(c.get("unpriced_actual_tokens") or 0)
has_rows = bool(c.get("has_rows"))
complete = has_rows and unpriced == 0 and priced is not None
if not complete:
cost_complete = False
if priced is not None:
priced_partial += int(priced)
per_task_cost.append({
"task_id": r["id"],
"priced_usd_micros": priced,
"unpriced_actual_tokens": unpriced,
"cost_complete": complete,
})
if not terminal:
cost_complete = False
return {
"tasks_total": len(records),
"terminal_tasks": len(terminal),
"first_pass_tasks": len(first_pass),
"first_pass_rate": _rate(len(first_pass), len(known_attempts)),
"first_pass_known": bool(known_attempts),
"rework_tasks": len(rework),
"rework_rate": _rate(len(rework), len(known_attempts)),
"rework_known": bool(known_attempts),
"avg_attempts": (attempts_total / len(known_attempts)
if known_attempts else None),
"defect_tasks": len(defective),
"defect_rate": _rate(len(defective), len(known_defects)),
"defect_known": bool(known_defects),
"defect_by_category": defect_by_category,
"per_task_cost": per_task_cost,
# Dual-column total: None unless every terminal task's cost
# is complete; the partial sum is shown separately, labelled.
"priced_usd_micros_partial": priced_partial,
"total_usd_micros": priced_partial if cost_complete else None,
"cost_complete": cost_complete,
}
def evaluate_feedback(metrics, thresholds=None, check_stats=None):
"""Testable feedback rules. Returns a list of action dicts; each
carries its evidence so a reviewer can audit why it fired."""
thresholds = thresholds or DEFAULT_THRESHOLDS
actions = []
defect_rate = metrics.get("defect_rate")
if metrics.get("defect_known") and defect_rate is not None:
limit = thresholds.get("defect_rate")
if limit is not None and defect_rate > limit:
for cat, count in sorted(
metrics.get("defect_by_category", {}).items()):
actions.append({
"kind": "tighten_check",
"target": cat,
"reason": "defect_rate %.3f > threshold %.3f "
"(%d defective tasks in category)"
% (defect_rate, limit, count),
"evidence": {"defect_rate": defect_rate,
"threshold": limit,
"category_tasks": count},
"auto_apply": False,
})
first_pass = metrics.get("first_pass_rate")
floor = thresholds.get("first_pass_rate_min")
if first_pass is not None and floor is not None and first_pass < floor:
actions.append({
"kind": "route_with_extra_audit",
"target": "all",
"reason": "first_pass_rate %.3f < floor %.3f"
% (first_pass, floor),
"evidence": {"first_pass_rate": first_pass, "floor": floor},
"auto_apply": False,
})
overturn_limit = thresholds.get("check_overturn_rate")
for stat in check_stats or []:
violations = stat.get("violations") or 0
overturned = stat.get("overturned") or 0
if violations and overturn_limit is not None:
rate = overturned / violations
if rate > overturn_limit:
actions.append({
"kind": "recalibrate_check",
"target": stat.get("check_id"),
"reason": "overturn_rate %.3f > threshold %.3f "
"(%d/%d violations overturned on review)"
% (rate, overturn_limit, overturned,
violations),
"evidence": {"overturn_rate": rate,
"threshold": overturn_limit,
"violations": violations,
"overturned": overturned},
"auto_apply": False,
})
return actions
def build_report(metrics, golden=None, thresholds=None,
thresholds_source="default_unapproved",
check_stats=None):
"""Assemble the JSON report document.
``golden``: None, or a dict with ``tasks`` (list of task ids),
``approved_by`` and ``approved_at``. Only a fully approved golden
set makes the report decision-ready; anything else is pending.
"""
golden = golden or {}
tasks = golden.get("tasks") or []
approved = bool(tasks) and bool(golden.get("approved_by")) \
and bool(golden.get("approved_at"))
return {
"generated_at": time.time(),
"golden_set": {
"status": "approved" if approved else "pending_human_approval",
"tasks": list(tasks),
"approved_by": golden.get("approved_by"),
"approved_at": golden.get("approved_at"),
},
"decision_ready": approved,
"decision_note": ("golden set human-approved; metrics may inform "
"decisions" if approved else
"golden set NOT human-approved: this report "
"collects metrics only and must not be used "
"as a decision basis"),
"metrics": metrics,
"thresholds": dict(thresholds or DEFAULT_THRESHOLDS),
"thresholds_source": thresholds_source,
"feedback_actions": evaluate_feedback(
metrics, thresholds=thresholds, check_stats=check_stats),
# How the loop closes (see docs/arch-conformance-metrics.md):
# feedback actions are consumed by (a) routing -- tasks in a
# flagged category are routed with an extra audit gate, and
# (b) audit standards -- a tighten/recalibrate action opens a
# standards-review item; the standards document itself changes
# only through the contract-change protocol
# (arch_conformance.check_contract_version) with coordinator
# approval, never by auto-applying an action record.
"feedback_consumers": ["routing", "audit_standards"],
}
def render_report(report):
"""Canonical JSON rendering of a report document."""
return json.dumps(report, indent=2, sort_keys=True, default=str)
+56
View File
@@ -158,6 +158,8 @@ from cloud.auto_memory import process_due_tasks as _auto_process_due
from cloud.identity import IdentityStore
from cloud.kb import KBStore
from cloud.memory import MemoryStore
from cloud.metrics import build_report as _build_metrics_report
from cloud.metrics import compute_metrics as _compute_metrics
from cloud.pr import PrSkipped, PrStepError
from cloud.uploads import (
BUNDLE_TTL_SECONDS,
@@ -3008,6 +3010,8 @@ class CloudControlPlaneHandler(BaseHTTPRequestHandler):
self._workers_heartbeat(user_id)
elif path == "/metrics" and method == "GET":
self._metrics(user_id)
elif path == "/metrics-report" and method == "GET":
self._metrics_report(user_id)
elif path == "/models" and method == "GET":
self._models_overview(user_id)
elif path == "/relay/models" and method == "GET":
@@ -9629,6 +9633,58 @@ pid 须为整数、address 须为字符串,否则 400(注册表里的
},
})
def _metrics_report(self, user_id):
"""指标闭环报告:首过绿灯率 / 缺陷率 / 返工率 / 每任务成本。
权限与 /metrics 同款:只统计调用者是成员的项目。
诚实边界:当前 tasks 表没有 attempts/defects 列,所以
attempts/defects 一律传 None,首过/缺陷/返工率回 None
(未知,不报 0);金任务集未经人工审定,报告恒为
pending_human_approval、decision_ready False。等任务尝试
台账落库、金任务人工审定后,这两处再接真实输入——
在此之前绝不把样本集或未知率包装成可决策数字。
成本走 #212 双列口径:有价合计与无价 actual token 分开,
总额在成本不完整时为 None,不把无价折成 0。
"""
conn = self._db()
try:
member_ids = {
row["project_id"]
for row in conn.execute(
"SELECT project_id FROM project_members WHERE user_id = ?",
(user_id,),
).fetchall()
}
rows = conn.execute(
"SELECT id, project_id, status FROM tasks").fetchall()
records = [{"id": row["id"], "status": row["status"],
"attempts": None, "defects": None}
for row in rows
if row["project_id"] in member_ids]
try:
cost_rows = conn.execute(
"SELECT task_id, project_id, source, total_tokens,"
" usd_micros FROM cost_records").fetchall()
except (sqlite3.OperationalError, _PgOperationalError):
cost_rows = []
costs = {}
for row in cost_rows:
if row["project_id"] not in member_ids:
continue
entry = costs.setdefault(row["task_id"], {
"priced_usd_micros": 0, "unpriced_actual_tokens": 0,
"has_rows": True})
if row["source"] == "actual" \
and row["usd_micros"] is None:
entry["unpriced_actual_tokens"] += \
row["total_tokens"] or 0
elif row["usd_micros"] is not None:
entry["priced_usd_micros"] += row["usd_micros"]
finally:
conn.close()
metrics = _compute_metrics(records, costs)
self._send_json(200, _build_metrics_report(metrics))
# -- OAuth(pre-auth,靠 state + secret,不走 X-API-Key)-----------
def _oauth_provider(self, path):
parts = [p for p in path.split("/") if p]
+221
View File
@@ -0,0 +1,221 @@
"""Tests for cloud.arch_conformance: architecture consistency checks.
Every check has a positive and a negative case; the repo-level test
pins that the current tree passes the layer bans, so a future
violation fails CI (the existing host-verify workflow runs
``discover -s cloud/tests``; a failure here is a CI failure).
Only the standard library; no real model, no network, no secrets.
"""
import os
import sys
import tempfile
import unittest
from pathlib import Path
sys.path.insert(0, os.path.dirname(os.path.dirname(
os.path.dirname(os.path.abspath(__file__)))))
from cloud.arch_conformance import ( # noqa: E402
check_contract_version, check_frozen_stack, check_layer_imports,
check_skill_project, check_task_writes, is_within,
scan_cloud_package, scan_imports)
class TestContractVersion(unittest.TestCase):
def test_equal_versions_pass(self):
self.assertTrue(check_contract_version("v1", "v1")["ok"])
def test_approved_step_bump_passes(self):
change = {"approved": True, "backward_compatible": True,
"affected_clients": ["web"]}
self.assertTrue(
check_contract_version("v1", "v2", change=change)["ok"])
def test_breaking_bump_needs_regression_task(self):
change = {"approved": True, "backward_compatible": False,
"affected_clients": ["web"]}
verdict = check_contract_version("v1", "v2", change=change)
self.assertFalse(verdict["ok"])
self.assertTrue(any("regression" in r for r in verdict["reasons"]))
def test_breaking_bump_with_regression_task_passes(self):
change = {"approved": True, "backward_compatible": False,
"affected_clients": ["web"], "regression_task": "t-1"}
self.assertTrue(
check_contract_version("v1", "v2", change=change)["ok"])
def test_unapproved_bump_fails(self):
change = {"approved": False, "backward_compatible": True,
"affected_clients": ["web"]}
self.assertFalse(
check_contract_version("v1", "v2", change=change)["ok"])
def test_bump_without_change_record_fails(self):
self.assertFalse(check_contract_version("v1", "v2")["ok"])
def test_skip_bump_fails_even_when_approved(self):
change = {"approved": True, "backward_compatible": True,
"affected_clients": ["web"]}
verdict = check_contract_version("v1", "v3", change=change)
self.assertFalse(verdict["ok"])
def test_unparseable_version_fails(self):
self.assertFalse(check_contract_version("v1", "draft")["ok"])
class TestFrozenStack(unittest.TestCase):
def test_unchanged_stack_passes(self):
frozen = {"web": "React", "ios": "SwiftUI"}
self.assertTrue(check_frozen_stack(frozen, dict(frozen))["ok"])
def test_new_client_is_addition_not_swap(self):
frozen = {"web": "React"}
current = {"web": "React", "ios": "SwiftUI"}
self.assertTrue(check_frozen_stack(frozen, current)["ok"])
def test_swap_without_approval_fails(self):
verdict = check_frozen_stack({"web": "React"}, {"web": "Vue"})
self.assertFalse(verdict["ok"])
self.assertEqual(verdict["swapped"], ["web"])
def test_removed_client_without_approval_fails(self):
self.assertFalse(
check_frozen_stack({"web": "React", "ios": "SwiftUI"},
{"web": "React"})["ok"])
def test_swap_with_approved_change_passes(self):
change = {"approved": True, "backward_compatible": True,
"affected_clients": ["web"]}
self.assertTrue(
check_frozen_stack({"web": "React"}, {"web": "Vue"},
change=change)["ok"])
class TestDirectoryGuard(unittest.TestCase):
def test_inside_passes_and_outside_fails(self):
with tempfile.TemporaryDirectory() as tmp:
own = os.path.join(tmp, "work")
os.makedirs(own)
self.assertTrue(is_within(own, os.path.join(own, "a.txt")))
self.assertTrue(is_within(own, own))
self.assertFalse(is_within(own, os.path.join(tmp, "other")))
def test_dotdot_escape_fails(self):
with tempfile.TemporaryDirectory() as tmp:
own = os.path.join(tmp, "work")
os.makedirs(own)
escape = os.path.join(own, "..", "secret.txt")
self.assertFalse(is_within(own, escape))
self.assertEqual(check_task_writes(own, [escape]), [escape])
def test_symlink_escape_fails(self):
with tempfile.TemporaryDirectory() as tmp:
own = os.path.join(tmp, "work")
outside = os.path.join(tmp, "outside")
os.makedirs(own)
os.makedirs(outside)
link = os.path.join(own, "link")
try:
os.symlink(outside, link)
except OSError:
self.skipTest("symlinks unavailable")
self.assertFalse(is_within(own, link))
def test_mixed_writes_report_only_violations(self):
with tempfile.TemporaryDirectory() as tmp:
own = os.path.join(tmp, "work")
os.makedirs(own)
good = os.path.join(own, "ok.txt")
bad = os.path.join(tmp, "bad.txt")
self.assertEqual(check_task_writes(own, [good, bad]), [bad])
class TestLayerImports(unittest.TestCase):
def test_foundation_importing_server_fails(self):
verdict = check_layer_imports(
{"db": "from cloud.server import foo\n"})
self.assertEqual(len(verdict), 1)
self.assertEqual(verdict[0]["rule"],
"foundation_must_not_import_orchestration")
def test_foundation_importing_store_passes(self):
# db importing nothing, models importing nothing: the baseline
# shape. A foundation module importing another foundation
# module is not an upward import.
self.assertEqual(check_layer_imports(
{"models": "import os\n", "queue": "from cloud.models import x\n"}),
[])
def test_sandbox_importing_db_fails(self):
verdict = check_layer_imports(
{"sandbox": "from cloud.db import connect\n"})
self.assertEqual(len(verdict), 1)
def test_sandbox_stdlib_only_passes(self):
self.assertEqual(check_layer_imports(
{"sandbox": "import os\nimport subprocess\n"}), [])
def test_service_opening_sqlite_directly_fails(self):
verdict = check_layer_imports(
{"server": "import sqlite3\nconn = sqlite3.connect('x.db')\n"})
self.assertTrue(any(v["rule"] == "service_must_not_open_driver_"
"connection_directly" for v in verdict))
def test_service_sqlite_row_adapter_passes(self):
# Deliberate narrow ban: Row/exception adapters on a
# cloud.db connection are adaptation, not a bypass.
self.assertEqual(check_layer_imports(
{"server": "import sqlite3\nconn.row_factory = sqlite3.Row\n"
"try:\n pass\nexcept sqlite3.IntegrityError:\n"
" pass\n"}), [])
def test_service_importing_psycopg_fails(self):
verdict = check_layer_imports(
{"worker": "import psycopg\n"})
self.assertTrue(any(v["rule"] == "service_must_not_open_driver_"
"connection_directly" for v in verdict))
def test_driver_adjacent_modules_exempt(self):
self.assertEqual(check_layer_imports(
{"backup": "import sqlite3\nsqlite3.connect('x.db')\n",
"migrations": "import sqlite3\nsqlite3.connect('y.db')\n"}), [])
def test_scan_imports_sees_from_cloud_form(self):
scanned = scan_imports("from cloud.worker import Worker\n")
self.assertIn("worker", scanned["cloud"])
class TestRepoLevelScan(unittest.TestCase):
def test_current_tree_has_zero_layer_violations(self):
repo_root = Path(__file__).resolve().parents[2]
violations = scan_cloud_package(repo_root)
self.assertEqual(violations, [])
class TestSkillProjectFixture(unittest.TestCase):
def test_frozen_project_passes_all_checks(self):
with tempfile.TemporaryDirectory() as tmp:
own = os.path.join(tmp, "apps", "web")
os.makedirs(own)
verdict = check_skill_project(
tmp, "v1", "v1", {"web": "React"}, {"web": "React"},
own, [os.path.join(own, "App.tsx")])
self.assertTrue(verdict["ok"])
def test_drifting_project_fails_each_changed_check(self):
with tempfile.TemporaryDirectory() as tmp:
own = os.path.join(tmp, "apps", "web")
os.makedirs(own)
verdict = check_skill_project(
tmp, "v1", "v2", {"web": "React"}, {"web": "Vue"},
own, [os.path.join(tmp, "apps", "ios", "leak.txt")])
self.assertFalse(verdict["ok"])
self.assertFalse(verdict["contract"]["ok"])
self.assertFalse(verdict["stack"]["ok"])
self.assertFalse(verdict["directory"]["ok"])
if __name__ == "__main__":
unittest.main()
+186
View File
@@ -0,0 +1,186 @@
"""Tests for cloud.metrics: collection, dual-column cost, feedback.
Every metric has a positive case and an unknown/edge case; unknown
is never reported as zero, and an unapproved golden set never makes
a report decision-ready.
Only the standard library; no real model, no network, no secrets.
"""
import os
import sys
import unittest
sys.path.insert(0, os.path.dirname(os.path.dirname(
os.path.dirname(os.path.abspath(__file__)))))
from cloud.arch_conformance import check_contract_version # noqa: E402
from cloud.metrics import ( # noqa: E402
build_report, compute_metrics, evaluate_feedback)
def _record(task_id, status, attempts=None, defects=None, category=None):
return {"id": task_id, "status": status, "attempts": attempts,
"defects": defects, "defect_category": category}
class TestComputeMetrics(unittest.TestCase):
def test_first_pass_rework_defect_rates(self):
records = [
_record("a", "completed", attempts=1, defects=0),
_record("b", "merged", attempts=3, defects=2,
category="contract"),
_record("c", "failed", attempts=2, defects=1,
category="contract"),
_record("d", "queued", attempts=1), # not terminal: excluded
_record("e", "interrupted", attempts=1), # not terminal
]
m = compute_metrics(records)
self.assertEqual(m["terminal_tasks"], 3)
self.assertAlmostEqual(m["first_pass_rate"], 1 / 3)
self.assertAlmostEqual(m["rework_rate"], 2 / 3)
self.assertAlmostEqual(m["defect_rate"], 2 / 3)
self.assertEqual(m["defect_by_category"], {"contract": 2})
def test_unknown_attempts_and_defects_are_none_not_zero(self):
records = [_record("a", "completed"),
_record("b", "failed")]
m = compute_metrics(records)
self.assertIsNone(m["first_pass_rate"])
self.assertIsNone(m["rework_rate"])
self.assertIsNone(m["defect_rate"])
self.assertFalse(m["first_pass_known"])
self.assertFalse(m["rework_known"])
self.assertFalse(m["defect_known"])
def test_empty_set_rates_are_none(self):
m = compute_metrics([])
self.assertEqual(m["terminal_tasks"], 0)
self.assertIsNone(m["first_pass_rate"])
self.assertIsNone(m["defect_rate"])
self.assertIsNone(m["total_usd_micros"])
self.assertFalse(m["cost_complete"])
class TestDualColumnCost(unittest.TestCase):
def test_complete_cost_totals(self):
records = [_record("a", "completed", attempts=1)]
costs = {"a": {"priced_usd_micros": 5000,
"unpriced_actual_tokens": 0, "has_rows": True}}
m = compute_metrics(records, costs)
self.assertEqual(m["total_usd_micros"], 5000)
self.assertTrue(m["cost_complete"])
def test_unpriced_cost_never_folds_into_zero_or_total(self):
records = [_record("a", "completed", attempts=1)]
costs = {"a": {"priced_usd_micros": 5000,
"unpriced_actual_tokens": 700, "has_rows": True}}
m = compute_metrics(records, costs)
self.assertIsNone(m["total_usd_micros"])
self.assertEqual(m["priced_usd_micros_partial"], 5000)
self.assertFalse(m["cost_complete"])
self.assertEqual(m["per_task_cost"][0]["unpriced_actual_tokens"],
700)
def test_missing_cost_rows_make_total_unknown(self):
records = [_record("a", "completed", attempts=1)]
m = compute_metrics(records, costs={})
self.assertIsNone(m["total_usd_micros"])
self.assertFalse(m["cost_complete"])
class TestFeedbackRules(unittest.TestCase):
def test_defect_breach_tightens_check(self):
m = compute_metrics([
_record("a", "completed", attempts=1),
_record("b", "failed", attempts=1, defects=1,
category="contract"),
])
actions = evaluate_feedback(
m, thresholds={"defect_rate": 0.20,
"first_pass_rate_min": 0.50,
"check_overturn_rate": 0.30})
tighten = [a for a in actions if a["kind"] == "tighten_check"]
self.assertEqual(len(tighten), 1)
self.assertEqual(tighten[0]["target"], "contract")
self.assertFalse(tighten[0]["auto_apply"])
def test_no_breach_no_action(self):
m = compute_metrics([_record("a", "completed", attempts=1)])
actions = evaluate_feedback(
m, thresholds={"defect_rate": 0.20,
"first_pass_rate_min": 0.50,
"check_overturn_rate": 0.30})
self.assertEqual(actions, [])
def test_low_first_pass_routes_with_extra_audit(self):
m = compute_metrics([
_record("a", "completed", attempts=3),
_record("b", "failed", attempts=2),
])
actions = evaluate_feedback(
m, thresholds={"defect_rate": 0.20,
"first_pass_rate_min": 0.50,
"check_overturn_rate": 0.30})
self.assertTrue(any(a["kind"] == "route_with_extra_audit"
for a in actions))
def test_high_overturn_recalibrates_without_auto_apply(self):
m = compute_metrics([_record("a", "completed", attempts=1)])
actions = evaluate_feedback(
m, thresholds={"defect_rate": 0.20,
"first_pass_rate_min": 0.50,
"check_overturn_rate": 0.30},
check_stats=[{"check_id": "forbidden-dir", "violations": 10,
"overturned": 5}])
recal = [a for a in actions if a["kind"] == "recalibrate_check"]
self.assertEqual(len(recal), 1)
self.assertFalse(recal[0]["auto_apply"])
def test_low_overturn_no_recalibration(self):
m = compute_metrics([_record("a", "completed", attempts=1)])
actions = evaluate_feedback(
m, check_stats=[{"check_id": "forbidden-dir",
"violations": 10, "overturned": 1}])
self.assertEqual(
[a for a in actions if a["kind"] == "recalibrate_check"], [])
class TestGoldenSetAndStandardsLoop(unittest.TestCase):
def test_unapproved_golden_set_is_not_decision_ready(self):
m = compute_metrics([_record("a", "completed", attempts=1)])
report = build_report(m, golden={"tasks": ["a", "b"]})
self.assertFalse(report["decision_ready"])
self.assertEqual(report["golden_set"]["status"],
"pending_human_approval")
def test_approved_golden_set_is_decision_ready(self):
m = compute_metrics([_record("a", "completed", attempts=1)])
report = build_report(
m, golden={"tasks": ["a"], "approved_by": "human",
"approved_at": "2026-10-31"},
thresholds_source="human_approved")
self.assertTrue(report["decision_ready"])
self.assertEqual(report["thresholds_source"], "human_approved")
def test_sample_only_set_must_not_be_presented_as_golden(self):
# The set that exists today is sample-only: no approval fields
# exist for it, so any report built from it stays pending.
m = compute_metrics([_record("a", "completed", attempts=1)])
report = build_report(m, golden={"tasks": ["sample-1"]})
self.assertFalse(report["decision_ready"])
self.assertIn("NOT human-approved", report["decision_note"])
def test_standards_change_goes_through_contract_protocol(self):
# The loop closes through the same protocol: an audit-standard
# version bump without approval fails; with approval it passes.
self.assertFalse(
check_contract_version("v1", "v2")["ok"])
self.assertTrue(check_contract_version(
"v1", "v2",
change={"approved": True, "backward_compatible": True,
"affected_clients": ["all"]})["ok"])
if __name__ == "__main__":
unittest.main()
+120
View File
@@ -0,0 +1,120 @@
"""Endpoint test for GET /metrics-report.
Only the standard library http.client; a temporary SQLite DB and a
temporary port; no real model, no network, no secrets.
Covers: unauthenticated -> 401, empty project -> report shape with
unknown rates (None, not 0) and pending golden set, and a completed
task with a priced cost row -> terminal count and dual-column cost.
"""
import http.client
import json
import os
import sys
import tempfile
import threading
import unittest
from pathlib import Path
try:
from cloud.auth import AuthStore
from cloud.cost import CostStore
from cloud.server import create_server
except ImportError:
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
from cloud.auth import AuthStore
from cloud.cost import CostStore
from cloud.server import create_server
class MetricsReportEndpointTestCase(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.db_path = os.path.join(self.tmp.name, "control.db")
self.auth = AuthStore(self.db_path)
self.owner_id, self.owner_key = self.auth.create_user("owner")
self.server = create_server(self.db_path, "127.0.0.1", 0)
self.port = self.server.server_address[1]
self.thread = threading.Thread(
target=self.server.serve_forever, daemon=True)
self.thread.start()
status, self.project = self._req(
"POST", "/projects", {"name": "demo"}, self.owner_key)
self.assertEqual(status, 201)
def tearDown(self):
self.server.shutdown()
self.server.server_close()
self.thread.join(timeout=5)
try:
self.auth.conn.close()
except Exception:
pass
self.tmp.cleanup()
def _req(self, method, path, body=None, key=None):
conn = http.client.HTTPConnection("127.0.0.1", self.port, timeout=10)
hdrs = {}
payload = None
if body is not None:
payload = json.dumps(body).encode("utf-8")
hdrs["Content-Type"] = "application/json"
if key is not None:
hdrs["X-API-Key"] = key
try:
conn.request(method, path, body=payload, headers=hdrs)
resp = conn.getresponse()
data = resp.read()
finally:
conn.close()
try:
parsed = json.loads(data.decode("utf-8")) if data else None
except ValueError:
parsed = None
return resp.status, parsed
def test_unauthenticated_is_401(self):
status, _ = self._req("GET", "/metrics-report")
self.assertEqual(status, 401)
def test_empty_project_report_is_pending_and_unknown(self):
status, report = self._req(
"GET", "/metrics-report", key=self.owner_key)
self.assertEqual(status, 200)
self.assertFalse(report["decision_ready"])
self.assertEqual(report["golden_set"]["status"],
"pending_human_approval")
self.assertIsNone(report["metrics"]["first_pass_rate"])
self.assertIsNone(report["metrics"]["defect_rate"])
self.assertIsNone(report["metrics"]["total_usd_micros"])
self.assertEqual(report["thresholds_source"],
"default_unapproved")
def test_completed_task_with_priced_cost_row(self):
from cloud.db import connect as db_connect
conn = db_connect(self.db_path)
try:
conn.execute(
"INSERT INTO tasks(id, project_id, status) VALUES(?, ?, ?)",
("t-1", self.project["id"], "completed"))
conn.commit()
finally:
conn.close()
store = CostStore(self.db_path)
store.record("t-1", self.project["id"], "gpt-6-sol",
input_tokens=1000, output_tokens=100,
usd_micros=5000, source="actual")
status, report = self._req(
"GET", "/metrics-report", key=self.owner_key)
self.assertEqual(status, 200)
self.assertEqual(report["metrics"]["terminal_tasks"], 1)
# Attempts are not in the schema: first-pass stays unknown,
# it must not be fabricated as 0 or 1 from status alone.
self.assertIsNone(report["metrics"]["first_pass_rate"])
self.assertEqual(report["metrics"]["total_usd_micros"], 5000)
self.assertTrue(report["metrics"]["cost_complete"])
if __name__ == "__main__":
unittest.main()
+66
View File
@@ -0,0 +1,66 @@
# 架构一致性进 CI 与指标闭环(审计加固第 5 步)
实现:`cloud/arch_conformance.py`(纯检查,标准库)、`cloud/metrics.py`
(纯计算,标准库)、端点 `GET /metrics-report`(`cloud/server.py`,
与 `/metrics` 同款认证与成员范围)。测试失败即 CI 失败:现有
`host-verify` 工作流已跑 `discover -s cloud/tests`,本轨未改任何
workflow——第 2 轨拥有 lint/类型/安扫,本轨不碰。
## 架构一致性检查(规则、理由、测试)
| 检查 | 规则 | 理由 | 测试 |
|---|---|---|---|
| 合同版本 | 冻结后版本只能 +1,且须有协调员批准的变更记录(声明向后兼容、列出受影响客户端;破坏性变更须有回归任务) | 契约静默漂移是并行线程分叉的根源;协议是唯一合法路径,检查执行协议而非只记录协议 | `test_arch_conformance.py::TestContractVersion`(正向:等值/批准+1/破坏性+回归任务;负向:未批准、无记录、跳号、不可解析) |
| 禁目录 | 任务只能写自有目录;`..` 与符号链接逃逸在解析后判定 | 跨目录写会 corrupt 他人 worktree | `TestDirectoryGuard`(正向:目录内;负向:`..`、symlink、混合写只报违规项)。口径复用已有定义,不另造:控制面 = 租户 work 目录(`worker._tenant_dirs` 的 `work`),技能项目 = `apps/<client>/` |
| 禁 import | 基础层(db/models/queue/logging_config)不 import 编排层;执行层(sandbox)不 import 持久化/编排层;除 db/backup/migrations 外任何 cloud 模块不得直接 `sqlite3.connect` 或 import `psycopg*`(服务层走 `cloud.db.connect` 或 Store) | 基础层上引是循环依赖前兆;执行器须可换、可假 runner 测试;驱动直连绕过 sqlite/postgres 唯一分发点 | `TestLayerImports`(每项正向+负向)+ 仓库级 `TestRepoLevelScan`(当前主线零违规,未来违规即 CI 失败)。有意不禁:`server.py` 的 `sqlite3.Row`/异常类型适配(那是在 `cloud.db` 连接上的适配,不是绕过;禁它会在基线误报) |
| 技术栈冻结 | G2 冻结表变更/删除须有批准的变更记录;新增客户端是新增不是换栈,通过 | 共享核心按冻结表构建,中途换栈使其静默失效 | `TestFrozenStack`(正向:不变/新增;负向:换栈/删除无批准) |
技能项目一键版:`check_skill_project` 把合同版本、冻结栈、目录写
三项合在一起验,`TestSkillProjectFixture` 用合成项目目录正向/负向
各验一次。静态 AST 扫描看不到 `importlib` 动态导入——这是已知
边界,不声称覆盖。
## 指标采集与报告
端点:`GET /metrics-report`(需 X-API-Key,成员范围)。报告格式
见 `cloud/metrics.py::build_report`:`golden_set`、`decision_ready`、
`metrics`、`thresholds` + `thresholds_source`、`feedback_actions`、
`feedback_consumers`。
指标口径(`compute_metrics`):
- 首过绿灯率 = 首过成功任务 / 已知 attempts 的终态任务;缺陷率 =
有缺陷任务 / 已知 defects 的终态任务;返工率 = attempts>1 任务 /
已知 attempts 的终态任务。终态 = completed/merged/failed/
merge-failed;`interrupted` 不算终态(平台中断自动重跑,算失败
是罚平台事件)。
- 每任务成本复用 #212 双列口径:有价 USD micros 与无价 actual
token 分列;成本不完整(无成本行或有无价 token)时
`total_usd_micros` 为 `None`,有价部分只以
`priced_usd_micros_partial` 出现——无价永不记 0、永不折进总额。
- 未知率回 `None` + `*_known: false`,绝不报 0。
当前诚实边界(端点实测):tasks 表无 attempts/defects 列,端点
传 `None`,首过/缺陷/返工率恒为 `None`;金任务集未经人工审定,
报告恒为 `pending_human_approval`、`decision_ready: false`。接真实
输入需两项前置(见待审定清单),本轨不编造。
## 回流与审计标准迭代(可测试规则,非口号)
`evaluate_feedback` 是纯函数,只产出 `auto_apply: false` 的行动
记录,不自动改路由、不自动改标准;行动由两处消费:
- 路由:`tighten_check`(某类缺陷率 > 阈值)→ 该类任务路由时加
一道审计门;`route_with_extra_audit`(首过率 < 下限)→ 全量加
审计门。消费动作本身须经协调员/人工确认后才生效。
- 审计标准:`tighten_check` / `recalibrate_check`(某检查的违规被
复核推翻率 > 阈值)各开一条标准复核项;标准文档本身的版本
变更走同一合同变更协议(`check_contract_version`,批准 +1),
测试 `TestGoldenSetAndStandardsLoop` 钉住“无批准改标准失败、
有批准通过”。误报复核规则:推翻率高只触发“复核阈值”行动,
绝不让检查自动放宽自己。
默认阈值(`DEFAULT_THRESHOLDS`:缺陷率 0.20、首过下限 0.50、
推翻率 0.30)只是测试用初值,报告标注
`thresholds_source: "default_unapproved"`;生产阈值须人工拍板后
以 `thresholds_source: "human_approved"` 传入,见待审定清单。