feat: architecture conformance checks in CI and metrics feedback loop
- cloud/arch_conformance.py: contract-version (+1 only, approved change record), forbidden-directory (own dir only, .. and symlink escapes resolved), forbidden-import layer bans audited against the baseline (foundation/execution/driver, server sqlite3 Row adapters deliberately allowed), G2 frozen-stack swap guard; skill-project one-call check. - cloud/metrics.py: first-pass/defect/rework rates over terminal tasks, unknown rates as None (never 0), #212 dual-column per-task cost (total None when incomplete), pure feedback rules (tighten/route-with-extra-audit/recalibrate, auto_apply False), golden-set gate: unapproved sample set is never decision-ready. - GET /metrics-report (authenticated, member-scoped, same as /metrics): attempts/defects are None until an attempt ledger exists, so rates stay unknown rather than fabricated. - Tests: 29 arch + 15 metrics + 3 endpoint, each check with positive and negative cases; repo-level scan pins zero layer violations on the baseline so a future violation fails CI via the existing host-verify discover run (no workflow change).
This commit is contained in:
1 parent
382ec6d62e
commit
dba6536ac2
7 files changed
+1201
No files matched your search
@@ -0,0 +1,299 @@
|
||||
"""Architecture-conformance checks for the control plane and for
|
||||
skill-driven projects (software-engineering skill, G2 frozen artefacts).
|
||||
|
||||
Zero third-party dependencies; standard library only. This module
|
||||
imports no ``cloud.*`` module, so it can never violate the layer bans
|
||||
it enforces, and it can be reused verbatim against a skill project's
|
||||
own directory tree (see ``check_skill_project``).
|
||||
|
||||
The four checks, each with a stated rule and a stated reason:
|
||||
|
||||
1. Contract version (``check_contract_version``). Rule mirrors the
|
||||
skill's contract-freeze change protocol: after freeze, a version
|
||||
change is allowed only with a coordinator-approved change record
|
||||
that names backward compatibility, the affected clients, and (for
|
||||
a breaking change) a regression task per affected client; the
|
||||
version must advance by exactly one. Reason: a silent contract
|
||||
drift is how parallel client threads diverge; the protocol is the
|
||||
only sanctioned path, so the check enforces the protocol instead
|
||||
of merely documenting it.
|
||||
|
||||
2. Forbidden directories (``is_within`` / ``check_task_writes``).
|
||||
Rule reuses the two directory definitions that already exist and
|
||||
invents no third one: on the control plane a task's own directory
|
||||
is its tenant work directory (``worker._tenant_dirs`` ``work``);
|
||||
in a skill project a client thread's own directory is its
|
||||
``apps/<client>/`` tree. Any write outside the own directory --
|
||||
including ``..`` escapes and symlink escapes, both resolved before
|
||||
comparison -- is a violation. Reason: cross-directory writes are
|
||||
how one tenant/client corrupts another's worktree.
|
||||
|
||||
3. Forbidden imports (``check_layer_imports``). Bans were chosen by
|
||||
auditing the current tree, not by wishful layering; every ban
|
||||
below passes on the baseline, and the repo-level test pins that:
|
||||
|
||||
- Foundation modules (``db``, ``models``, ``queue``,
|
||||
``logging_config``) must not import orchestration/service
|
||||
modules (``server``, ``worker``, ``pr_exec``, ``webhooks``,
|
||||
``autoscaler``, ``cost``). Reason: the foundation is imported
|
||||
by everything; an upward import is a cycle waiting to happen
|
||||
and drags service state into pure helpers.
|
||||
- The execution module (``sandbox``) must not import persistence
|
||||
or orchestration (``db``, ``server``, ``worker``, ``cost``,
|
||||
``auth``, ``pr``, ``pr_exec``, ``webhooks``). Reason: executors
|
||||
must stay swappable and testable with fake runners; decisions
|
||||
about mounts and networks arrive as arguments, not by reading
|
||||
the database.
|
||||
- No cloud module except the driver-adjacent ones (``db``,
|
||||
``backup``, ``migrations``) may open a database connection
|
||||
directly (``sqlite3.connect`` or any ``psycopg*`` import).
|
||||
Service modules must obtain connections through
|
||||
``cloud.db.connect`` or a Store. Reason: the backend switch
|
||||
(sqlite/postgres) lives in exactly one place; a direct driver
|
||||
call bypasses it. Deliberately NOT banned: ``server.py``'s use
|
||||
of ``sqlite3.Row`` / ``sqlite3.IntegrityError`` as type adapters
|
||||
on connections obtained from ``cloud.db`` -- that is adaptation,
|
||||
not a bypass, and banning it would fail the baseline for no
|
||||
safety gain. The test asserts the narrow form (no ``connect``
|
||||
call, no driver import) rather than the broad form.
|
||||
|
||||
Only static ``ast`` imports are seen; dynamic imports
|
||||
(``importlib``) are out of scope and are noted, not claimed.
|
||||
|
||||
4. Frozen stack (``check_frozen_stack``). Rule mirrors the skill's
|
||||
G2 rule "frozen stack must not be swapped mid-flight": a client
|
||||
whose stack entry changes, or disappears, after freeze fails
|
||||
unless an approved change record covers it; adding a brand-new
|
||||
client (no frozen entry) is an addition, not a swap, and passes.
|
||||
Reason: the shared core is built against the frozen table; a
|
||||
mid-flight swap silently invalidates it.
|
||||
|
||||
All checkers return plain data (dicts/lists), never raise for a
|
||||
violation, so CI can report every violation in one run instead of
|
||||
stopping at the first.
|
||||
"""
|
||||
|
||||
import ast
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
FOUNDATION_MODULES = frozenset({"db", "models", "queue", "logging_config"})
|
||||
ORCHESTRATION_MODULES = frozenset(
|
||||
{"server", "worker", "pr_exec", "webhooks", "autoscaler", "cost"})
|
||||
EXECUTION_MODULES = frozenset({"sandbox"})
|
||||
EXECUTION_BANNED = frozenset(
|
||||
{"db", "server", "worker", "cost", "auth", "pr", "pr_exec", "webhooks"})
|
||||
DRIVER_ADJACENT_MODULES = frozenset({"db", "backup", "migrations"})
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- contract
|
||||
|
||||
def parse_version(text):
|
||||
"""Parse ``v1`` / ``1`` to int; anything else is None (unparseable
|
||||
is a failure, never silently treated as the frozen version)."""
|
||||
if text is None:
|
||||
return None
|
||||
text = str(text).strip()
|
||||
if text.startswith("v"):
|
||||
text = text[1:]
|
||||
if text.isdigit():
|
||||
return int(text)
|
||||
return None
|
||||
|
||||
|
||||
def _change_record_problems(change, breaking):
|
||||
"""Missing pieces of a contract-change record, per the protocol."""
|
||||
problems = []
|
||||
if not isinstance(change, dict):
|
||||
return ["change record missing (frozen contract changed without "
|
||||
"a coordinator-approved change record)"]
|
||||
if not change.get("approved"):
|
||||
problems.append("change not coordinator-approved")
|
||||
if change.get("backward_compatible") is None:
|
||||
problems.append("backward compatibility not declared")
|
||||
affected = change.get("affected_clients")
|
||||
if not affected or not isinstance(affected, list):
|
||||
problems.append("affected clients not listed")
|
||||
if breaking and not change.get("regression_task"):
|
||||
problems.append("breaking change without a regression task")
|
||||
return problems
|
||||
|
||||
|
||||
def check_contract_version(frozen, current, change=None):
|
||||
"""Return ``{"ok": bool, "reasons": [...]}``.
|
||||
|
||||
Equal versions pass regardless of ``change``. A different version
|
||||
passes only when it is exactly frozen + 1 and ``change`` satisfies
|
||||
the protocol; a breaking change is one where
|
||||
``backward_compatible`` is False.
|
||||
"""
|
||||
frozen_n = parse_version(frozen)
|
||||
current_n = parse_version(current)
|
||||
if frozen_n is None or current_n is None:
|
||||
return {"ok": False,
|
||||
"reasons": ["unparseable contract version: frozen=%r "
|
||||
"current=%r" % (frozen, current)]}
|
||||
if current_n == frozen_n:
|
||||
return {"ok": True, "reasons": []}
|
||||
reasons = []
|
||||
if current_n != frozen_n + 1:
|
||||
reasons.append("contract version must advance by exactly one "
|
||||
"(frozen v%d -> v%d, not v%d)"
|
||||
% (frozen_n, frozen_n + 1, current_n))
|
||||
breaking = isinstance(change, dict) and \
|
||||
change.get("backward_compatible") is False
|
||||
reasons.extend(_change_record_problems(change, breaking))
|
||||
return {"ok": not reasons, "reasons": reasons}
|
||||
|
||||
|
||||
# ------------------------------------------------------------- frozen stack
|
||||
|
||||
def check_frozen_stack(frozen, current, change=None):
|
||||
"""``frozen``/``current`` map client -> stack string.
|
||||
|
||||
Changed or removed entries fail without an approved change record
|
||||
(same shape as the contract-change record, minus the version
|
||||
rules); added clients pass as additions.
|
||||
"""
|
||||
frozen = frozen or {}
|
||||
current = current or {}
|
||||
reasons = []
|
||||
swapped = sorted(c for c in frozen
|
||||
if c not in current or current[c] != frozen[c])
|
||||
if not swapped:
|
||||
return {"ok": True, "reasons": [], "swapped": []}
|
||||
problems = _change_record_problems(change, breaking=False)
|
||||
if problems:
|
||||
reasons.append("frozen stack changed for %s without an approved "
|
||||
"change record: %s" % (swapped, "; ".join(problems)))
|
||||
return {"ok": not reasons, "reasons": reasons, "swapped": swapped}
|
||||
|
||||
|
||||
# ------------------------------------------------------------ directories
|
||||
|
||||
def is_within(root, candidate):
|
||||
"""True iff ``candidate`` resolves inside ``root`` (symlinks and
|
||||
``..`` resolved first, so both escape routes are caught)."""
|
||||
try:
|
||||
root_p = Path(root).resolve()
|
||||
cand_p = Path(candidate).resolve()
|
||||
except (OSError, RuntimeError):
|
||||
return False
|
||||
return cand_p == root_p or root_p in cand_p.parents
|
||||
|
||||
|
||||
def check_task_writes(own_dir, paths):
|
||||
"""Return the subset of ``paths`` written outside ``own_dir``."""
|
||||
return [p for p in paths if not is_within(own_dir, p)]
|
||||
|
||||
|
||||
# ----------------------------------------------------------------- imports
|
||||
|
||||
def scan_imports(source):
|
||||
"""Static import scan of one module's source.
|
||||
|
||||
Returns ``{"cloud": set, "driver_connect": bool, "driver_import": bool}``:
|
||||
``cloud`` holds the ``cloud.<name>`` top-level names imported;
|
||||
``driver_connect`` is True on any ``sqlite3.connect`` call;
|
||||
``driver_import`` is True on any ``psycopg*`` import. A bare
|
||||
``import sqlite3`` (Row/exception adapters) sets neither flag.
|
||||
"""
|
||||
cloud_imports = set()
|
||||
driver_connect = False
|
||||
driver_import = False
|
||||
try:
|
||||
tree = ast.parse(source)
|
||||
except SyntaxError:
|
||||
return {"cloud": cloud_imports,
|
||||
"driver_connect": driver_connect,
|
||||
"driver_import": driver_import,
|
||||
"parse_error": True}
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, ast.Import):
|
||||
for alias in node.names:
|
||||
if alias.name == "sqlite3":
|
||||
continue
|
||||
if alias.name.startswith("psycopg"):
|
||||
driver_import = True
|
||||
if alias.name == "cloud" or alias.name.startswith("cloud."):
|
||||
parts = alias.name.split(".")
|
||||
if len(parts) > 1:
|
||||
cloud_imports.add(parts[1])
|
||||
elif isinstance(node, ast.ImportFrom):
|
||||
if node.module and node.module.startswith("psycopg"):
|
||||
driver_import = True
|
||||
if node.module == "cloud" or (node.module or "").startswith("cloud."):
|
||||
if node.module == "cloud":
|
||||
for alias in node.names:
|
||||
cloud_imports.add(alias.name)
|
||||
else:
|
||||
cloud_imports.add(node.module.split(".")[1])
|
||||
elif isinstance(node, ast.Call):
|
||||
func = node.func
|
||||
if isinstance(func, ast.Attribute) and func.attr == "connect":
|
||||
value = func.value
|
||||
if isinstance(value, ast.Name) and value.id == "sqlite3":
|
||||
driver_connect = True
|
||||
return {"cloud": cloud_imports,
|
||||
"driver_connect": driver_connect,
|
||||
"driver_import": driver_import}
|
||||
|
||||
|
||||
def check_layer_imports(sources):
|
||||
"""``sources`` maps module name -> source text. Returns a list of
|
||||
violation dicts; empty means the layer rules hold."""
|
||||
violations = []
|
||||
for module, source in sorted(sources.items()):
|
||||
scanned = scan_imports(source)
|
||||
if scanned.get("parse_error"):
|
||||
violations.append({"module": module, "rule": "parse",
|
||||
"detail": "source does not parse; "
|
||||
"imports cannot be verified"})
|
||||
continue
|
||||
if module in FOUNDATION_MODULES:
|
||||
bad = scanned["cloud"] & ORCHESTRATION_MODULES
|
||||
if bad:
|
||||
violations.append({"module": module,
|
||||
"rule": "foundation_must_not_import_orchestration",
|
||||
"detail": sorted(bad)})
|
||||
if module in EXECUTION_MODULES:
|
||||
bad = scanned["cloud"] & EXECUTION_BANNED
|
||||
if bad:
|
||||
violations.append({"module": module,
|
||||
"rule": "execution_must_not_import_persistence_or_orchestration",
|
||||
"detail": sorted(bad)})
|
||||
if module not in DRIVER_ADJACENT_MODULES:
|
||||
if scanned["driver_connect"] or scanned["driver_import"]:
|
||||
violations.append({"module": module,
|
||||
"rule": "service_must_not_open_driver_connection_directly",
|
||||
"detail": "use cloud.db.connect or a Store"})
|
||||
return violations
|
||||
|
||||
|
||||
def scan_cloud_package(repo_root):
|
||||
"""Run ``check_layer_imports`` over every ``cloud/*.py`` in a repo
|
||||
checkout (tests and caches excluded). This is the repo-level test
|
||||
that makes a layer violation fail CI."""
|
||||
cloud_dir = Path(repo_root) / "cloud"
|
||||
sources = {}
|
||||
if cloud_dir.is_dir():
|
||||
for path in sorted(cloud_dir.glob("*.py")):
|
||||
sources[path.stem] = path.read_text(encoding="utf-8")
|
||||
return check_layer_imports(sources)
|
||||
|
||||
|
||||
def check_skill_project(project_root, frozen_version, current_version,
|
||||
frozen_stack, current_stack, own_dir, writes,
|
||||
change=None):
|
||||
"""One-call conformance for a skill project: contract version,
|
||||
frozen stack, and directory writes together. Returns a dict with
|
||||
one entry per check plus an overall ``ok``."""
|
||||
contract = check_contract_version(frozen_version, current_version,
|
||||
change=change)
|
||||
stack = check_frozen_stack(frozen_stack, current_stack, change=change)
|
||||
outside = check_task_writes(own_dir, writes)
|
||||
directory = {"ok": not outside,
|
||||
"reasons": (["writes outside own directory: %s" % outside]
|
||||
if outside else [])}
|
||||
return {"ok": contract["ok"] and stack["ok"] and directory["ok"],
|
||||
"contract": contract, "stack": stack, "directory": directory}
|
||||
@@ -0,0 +1,253 @@
|
||||
"""Metrics collection and feedback loop for agent-task quality.
|
||||
|
||||
Zero third-party dependencies; standard library only. This module
|
||||
imports no ``cloud.*`` module: it computes over plain record dicts so
|
||||
the collection logic is testable without a database, and so the
|
||||
report shape is identical whether the records came from the control
|
||||
plane (``server._metrics_report``), a test fixture, or a future
|
||||
attempt ledger.
|
||||
|
||||
Definitions (pinned here, not left to each caller):
|
||||
|
||||
- A task counts toward a rate only when it reached a terminal status
|
||||
(``completed`` / ``merged`` / ``failed`` / ``merge-failed``).
|
||||
``interrupted`` is deliberately NOT terminal: platform-interrupted
|
||||
tasks auto-resume as fresh runs, so counting them as failures
|
||||
would punish a platform event, not the agent.
|
||||
- First-pass green: terminal task whose ``attempts`` == 1 and whose
|
||||
status is ``completed`` / ``merged``.
|
||||
- Defect: a task with ``defects`` > 0, where ``defects`` is the count
|
||||
of failed CI/audit rounds or follow-up rounds for that task. The
|
||||
current tasks schema has NO attempts/defects column, so a caller
|
||||
that cannot supply them must pass ``None`` -- and the rates then
|
||||
come back as ``None`` with ``*_known`` False. An unknown rate is
|
||||
never reported as 0: 0 would claim "no defects", which is a
|
||||
decision-grade lie.
|
||||
- Rework: terminal task with ``attempts`` > 1. Same None discipline.
|
||||
- Per-task cost reuses the #212 dual-column discipline from
|
||||
``cloud.cost``: priced USD micros and unpriced actual tokens are
|
||||
reported in separate columns. ``total_usd_micros`` is ``None``
|
||||
whenever any task's cost is incomplete (no cost rows, or unpriced
|
||||
actual tokens > 0); the priced partial sum is still shown, labelled
|
||||
``priced_usd_micros_partial``, so nobody mistakes it for a total.
|
||||
Unpriced cost is never folded into 0.
|
||||
|
||||
Feedback (``evaluate_feedback``) is deliberately a pure function
|
||||
returning action records, not an auto-applier: an action says WHAT
|
||||
should change and WHY, with its evidence attached; applying it to
|
||||
routing or to the audit standards is a separate, approved step
|
||||
(see ``docs/arch-conformance-metrics.md``). The rules are testable:
|
||||
|
||||
- defect rate for a category above threshold -> ``tighten_check``
|
||||
for that category's check;
|
||||
- first-pass rate below its floor -> ``route_with_extra_audit``;
|
||||
- a check whose violations are overturned on review above
|
||||
threshold -> ``recalibrate_check`` (review the threshold, do NOT
|
||||
auto-apply a new one -- a check that fires wrongly must not
|
||||
silently widen itself).
|
||||
|
||||
Thresholds passed in are labelled by source in the report. The
|
||||
defaults in ``DEFAULT_THRESHOLDS`` are starting values for tests,
|
||||
NOT human-approved production thresholds: the report marks them
|
||||
``thresholds_source: "default_unapproved"`` until a human approves a
|
||||
set, and a report built on unapproved thresholds is still usable
|
||||
for collection -- it is just not decision-grade on its own.
|
||||
|
||||
Golden tasks: a report is ``decision_ready`` only when a golden set
|
||||
has been human-approved (``approved_by`` + ``approved_at`` present
|
||||
and at least one golden task listed). Until then the golden section
|
||||
reads ``status: "pending_human_approval"`` and the report says so
|
||||
plainly; the sample-only set that exists today must never be
|
||||
presented as decision-ready. This module never invents golden tasks.
|
||||
"""
|
||||
|
||||
import json
|
||||
import time
|
||||
|
||||
TERMINAL_STATUSES = frozenset(
|
||||
{"completed", "merged", "failed", "merge-failed"})
|
||||
SUCCESS_STATUSES = frozenset({"completed", "merged"})
|
||||
|
||||
DEFAULT_THRESHOLDS = {
|
||||
"defect_rate": 0.20,
|
||||
"first_pass_rate_min": 0.50,
|
||||
"check_overturn_rate": 0.30,
|
||||
}
|
||||
|
||||
|
||||
def _rate(numerator, denominator):
|
||||
if denominator is None or denominator == 0:
|
||||
return None
|
||||
return numerator / denominator
|
||||
|
||||
|
||||
def compute_metrics(records, costs=None):
|
||||
"""``records``: list of dicts with keys ``id``, ``status`` and
|
||||
optional ``attempts`` (int), ``defects`` (int),
|
||||
``defect_category`` (str). ``costs``: dict task_id -> dict with
|
||||
``priced_usd_micros`` (int|None), ``unpriced_actual_tokens`` (int),
|
||||
``has_rows`` (bool). Returns the metrics dict."""
|
||||
costs = costs or {}
|
||||
terminal = [r for r in records if r.get("status") in TERMINAL_STATUSES]
|
||||
known_attempts = [r for r in terminal if r.get("attempts") is not None]
|
||||
first_pass = [r for r in known_attempts
|
||||
if r["attempts"] == 1
|
||||
and r.get("status") in SUCCESS_STATUSES]
|
||||
rework = [r for r in known_attempts if r["attempts"] > 1]
|
||||
known_defects = [r for r in terminal if r.get("defects") is not None]
|
||||
defective = [r for r in known_defects if r["defects"] > 0]
|
||||
defect_by_category = {}
|
||||
for r in defective:
|
||||
cat = r.get("defect_category") or "uncategorized"
|
||||
defect_by_category[cat] = defect_by_category.get(cat, 0) + 1
|
||||
attempts_total = sum(r["attempts"] for r in known_attempts)
|
||||
|
||||
per_task_cost = []
|
||||
priced_partial = 0
|
||||
cost_complete = True
|
||||
for r in terminal:
|
||||
c = costs.get(r["id"]) or {}
|
||||
priced = c.get("priced_usd_micros")
|
||||
unpriced = int(c.get("unpriced_actual_tokens") or 0)
|
||||
has_rows = bool(c.get("has_rows"))
|
||||
complete = has_rows and unpriced == 0 and priced is not None
|
||||
if not complete:
|
||||
cost_complete = False
|
||||
if priced is not None:
|
||||
priced_partial += int(priced)
|
||||
per_task_cost.append({
|
||||
"task_id": r["id"],
|
||||
"priced_usd_micros": priced,
|
||||
"unpriced_actual_tokens": unpriced,
|
||||
"cost_complete": complete,
|
||||
})
|
||||
if not terminal:
|
||||
cost_complete = False
|
||||
|
||||
return {
|
||||
"tasks_total": len(records),
|
||||
"terminal_tasks": len(terminal),
|
||||
"first_pass_tasks": len(first_pass),
|
||||
"first_pass_rate": _rate(len(first_pass), len(known_attempts)),
|
||||
"first_pass_known": bool(known_attempts),
|
||||
"rework_tasks": len(rework),
|
||||
"rework_rate": _rate(len(rework), len(known_attempts)),
|
||||
"rework_known": bool(known_attempts),
|
||||
"avg_attempts": (attempts_total / len(known_attempts)
|
||||
if known_attempts else None),
|
||||
"defect_tasks": len(defective),
|
||||
"defect_rate": _rate(len(defective), len(known_defects)),
|
||||
"defect_known": bool(known_defects),
|
||||
"defect_by_category": defect_by_category,
|
||||
"per_task_cost": per_task_cost,
|
||||
# Dual-column total: None unless every terminal task's cost
|
||||
# is complete; the partial sum is shown separately, labelled.
|
||||
"priced_usd_micros_partial": priced_partial,
|
||||
"total_usd_micros": priced_partial if cost_complete else None,
|
||||
"cost_complete": cost_complete,
|
||||
}
|
||||
|
||||
|
||||
def evaluate_feedback(metrics, thresholds=None, check_stats=None):
|
||||
"""Testable feedback rules. Returns a list of action dicts; each
|
||||
carries its evidence so a reviewer can audit why it fired."""
|
||||
thresholds = thresholds or DEFAULT_THRESHOLDS
|
||||
actions = []
|
||||
defect_rate = metrics.get("defect_rate")
|
||||
if metrics.get("defect_known") and defect_rate is not None:
|
||||
limit = thresholds.get("defect_rate")
|
||||
if limit is not None and defect_rate > limit:
|
||||
for cat, count in sorted(
|
||||
metrics.get("defect_by_category", {}).items()):
|
||||
actions.append({
|
||||
"kind": "tighten_check",
|
||||
"target": cat,
|
||||
"reason": "defect_rate %.3f > threshold %.3f "
|
||||
"(%d defective tasks in category)"
|
||||
% (defect_rate, limit, count),
|
||||
"evidence": {"defect_rate": defect_rate,
|
||||
"threshold": limit,
|
||||
"category_tasks": count},
|
||||
"auto_apply": False,
|
||||
})
|
||||
first_pass = metrics.get("first_pass_rate")
|
||||
floor = thresholds.get("first_pass_rate_min")
|
||||
if first_pass is not None and floor is not None and first_pass < floor:
|
||||
actions.append({
|
||||
"kind": "route_with_extra_audit",
|
||||
"target": "all",
|
||||
"reason": "first_pass_rate %.3f < floor %.3f"
|
||||
% (first_pass, floor),
|
||||
"evidence": {"first_pass_rate": first_pass, "floor": floor},
|
||||
"auto_apply": False,
|
||||
})
|
||||
overturn_limit = thresholds.get("check_overturn_rate")
|
||||
for stat in check_stats or []:
|
||||
violations = stat.get("violations") or 0
|
||||
overturned = stat.get("overturned") or 0
|
||||
if violations and overturn_limit is not None:
|
||||
rate = overturned / violations
|
||||
if rate > overturn_limit:
|
||||
actions.append({
|
||||
"kind": "recalibrate_check",
|
||||
"target": stat.get("check_id"),
|
||||
"reason": "overturn_rate %.3f > threshold %.3f "
|
||||
"(%d/%d violations overturned on review)"
|
||||
% (rate, overturn_limit, overturned,
|
||||
violations),
|
||||
"evidence": {"overturn_rate": rate,
|
||||
"threshold": overturn_limit,
|
||||
"violations": violations,
|
||||
"overturned": overturned},
|
||||
"auto_apply": False,
|
||||
})
|
||||
return actions
|
||||
|
||||
|
||||
def build_report(metrics, golden=None, thresholds=None,
|
||||
thresholds_source="default_unapproved",
|
||||
check_stats=None):
|
||||
"""Assemble the JSON report document.
|
||||
|
||||
``golden``: None, or a dict with ``tasks`` (list of task ids),
|
||||
``approved_by`` and ``approved_at``. Only a fully approved golden
|
||||
set makes the report decision-ready; anything else is pending.
|
||||
"""
|
||||
golden = golden or {}
|
||||
tasks = golden.get("tasks") or []
|
||||
approved = bool(tasks) and bool(golden.get("approved_by")) \
|
||||
and bool(golden.get("approved_at"))
|
||||
return {
|
||||
"generated_at": time.time(),
|
||||
"golden_set": {
|
||||
"status": "approved" if approved else "pending_human_approval",
|
||||
"tasks": list(tasks),
|
||||
"approved_by": golden.get("approved_by"),
|
||||
"approved_at": golden.get("approved_at"),
|
||||
},
|
||||
"decision_ready": approved,
|
||||
"decision_note": ("golden set human-approved; metrics may inform "
|
||||
"decisions" if approved else
|
||||
"golden set NOT human-approved: this report "
|
||||
"collects metrics only and must not be used "
|
||||
"as a decision basis"),
|
||||
"metrics": metrics,
|
||||
"thresholds": dict(thresholds or DEFAULT_THRESHOLDS),
|
||||
"thresholds_source": thresholds_source,
|
||||
"feedback_actions": evaluate_feedback(
|
||||
metrics, thresholds=thresholds, check_stats=check_stats),
|
||||
# How the loop closes (see docs/arch-conformance-metrics.md):
|
||||
# feedback actions are consumed by (a) routing -- tasks in a
|
||||
# flagged category are routed with an extra audit gate, and
|
||||
# (b) audit standards -- a tighten/recalibrate action opens a
|
||||
# standards-review item; the standards document itself changes
|
||||
# only through the contract-change protocol
|
||||
# (arch_conformance.check_contract_version) with coordinator
|
||||
# approval, never by auto-applying an action record.
|
||||
"feedback_consumers": ["routing", "audit_standards"],
|
||||
}
|
||||
|
||||
|
||||
def render_report(report):
|
||||
"""Canonical JSON rendering of a report document."""
|
||||
return json.dumps(report, indent=2, sort_keys=True, default=str)
|
||||
@@ -158,6 +158,8 @@ from cloud.auto_memory import process_due_tasks as _auto_process_due
|
||||
from cloud.identity import IdentityStore
|
||||
from cloud.kb import KBStore
|
||||
from cloud.memory import MemoryStore
|
||||
from cloud.metrics import build_report as _build_metrics_report
|
||||
from cloud.metrics import compute_metrics as _compute_metrics
|
||||
from cloud.pr import PrSkipped, PrStepError
|
||||
from cloud.uploads import (
|
||||
BUNDLE_TTL_SECONDS,
|
||||
@@ -3008,6 +3010,8 @@ class CloudControlPlaneHandler(BaseHTTPRequestHandler):
|
||||
self._workers_heartbeat(user_id)
|
||||
elif path == "/metrics" and method == "GET":
|
||||
self._metrics(user_id)
|
||||
elif path == "/metrics-report" and method == "GET":
|
||||
self._metrics_report(user_id)
|
||||
elif path == "/models" and method == "GET":
|
||||
self._models_overview(user_id)
|
||||
elif path == "/relay/models" and method == "GET":
|
||||
@@ -9629,6 +9633,58 @@ pid 须为整数、address 须为字符串,否则 400(注册表里的
|
||||
},
|
||||
})
|
||||
|
||||
def _metrics_report(self, user_id):
|
||||
"""指标闭环报告:首过绿灯率 / 缺陷率 / 返工率 / 每任务成本。
|
||||
|
||||
权限与 /metrics 同款:只统计调用者是成员的项目。
|
||||
诚实边界:当前 tasks 表没有 attempts/defects 列,所以
|
||||
attempts/defects 一律传 None,首过/缺陷/返工率回 None
|
||||
(未知,不报 0);金任务集未经人工审定,报告恒为
|
||||
pending_human_approval、decision_ready False。等任务尝试
|
||||
台账落库、金任务人工审定后,这两处再接真实输入——
|
||||
在此之前绝不把样本集或未知率包装成可决策数字。
|
||||
成本走 #212 双列口径:有价合计与无价 actual token 分开,
|
||||
总额在成本不完整时为 None,不把无价折成 0。
|
||||
"""
|
||||
conn = self._db()
|
||||
try:
|
||||
member_ids = {
|
||||
row["project_id"]
|
||||
for row in conn.execute(
|
||||
"SELECT project_id FROM project_members WHERE user_id = ?",
|
||||
(user_id,),
|
||||
).fetchall()
|
||||
}
|
||||
rows = conn.execute(
|
||||
"SELECT id, project_id, status FROM tasks").fetchall()
|
||||
records = [{"id": row["id"], "status": row["status"],
|
||||
"attempts": None, "defects": None}
|
||||
for row in rows
|
||||
if row["project_id"] in member_ids]
|
||||
try:
|
||||
cost_rows = conn.execute(
|
||||
"SELECT task_id, project_id, source, total_tokens,"
|
||||
" usd_micros FROM cost_records").fetchall()
|
||||
except (sqlite3.OperationalError, _PgOperationalError):
|
||||
cost_rows = []
|
||||
costs = {}
|
||||
for row in cost_rows:
|
||||
if row["project_id"] not in member_ids:
|
||||
continue
|
||||
entry = costs.setdefault(row["task_id"], {
|
||||
"priced_usd_micros": 0, "unpriced_actual_tokens": 0,
|
||||
"has_rows": True})
|
||||
if row["source"] == "actual" \
|
||||
and row["usd_micros"] is None:
|
||||
entry["unpriced_actual_tokens"] += \
|
||||
row["total_tokens"] or 0
|
||||
elif row["usd_micros"] is not None:
|
||||
entry["priced_usd_micros"] += row["usd_micros"]
|
||||
finally:
|
||||
conn.close()
|
||||
metrics = _compute_metrics(records, costs)
|
||||
self._send_json(200, _build_metrics_report(metrics))
|
||||
|
||||
# -- OAuth(pre-auth,靠 state + secret,不走 X-API-Key)-----------
|
||||
def _oauth_provider(self, path):
|
||||
parts = [p for p in path.split("/") if p]
|
||||
|
||||
@@ -0,0 +1,221 @@
|
||||
"""Tests for cloud.arch_conformance: architecture consistency checks.
|
||||
|
||||
Every check has a positive and a negative case; the repo-level test
|
||||
pins that the current tree passes the layer bans, so a future
|
||||
violation fails CI (the existing host-verify workflow runs
|
||||
``discover -s cloud/tests``; a failure here is a CI failure).
|
||||
|
||||
Only the standard library; no real model, no network, no secrets.
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(
|
||||
os.path.dirname(os.path.abspath(__file__)))))
|
||||
|
||||
from cloud.arch_conformance import ( # noqa: E402
|
||||
check_contract_version, check_frozen_stack, check_layer_imports,
|
||||
check_skill_project, check_task_writes, is_within,
|
||||
scan_cloud_package, scan_imports)
|
||||
|
||||
|
||||
class TestContractVersion(unittest.TestCase):
|
||||
def test_equal_versions_pass(self):
|
||||
self.assertTrue(check_contract_version("v1", "v1")["ok"])
|
||||
|
||||
def test_approved_step_bump_passes(self):
|
||||
change = {"approved": True, "backward_compatible": True,
|
||||
"affected_clients": ["web"]}
|
||||
self.assertTrue(
|
||||
check_contract_version("v1", "v2", change=change)["ok"])
|
||||
|
||||
def test_breaking_bump_needs_regression_task(self):
|
||||
change = {"approved": True, "backward_compatible": False,
|
||||
"affected_clients": ["web"]}
|
||||
verdict = check_contract_version("v1", "v2", change=change)
|
||||
self.assertFalse(verdict["ok"])
|
||||
self.assertTrue(any("regression" in r for r in verdict["reasons"]))
|
||||
|
||||
def test_breaking_bump_with_regression_task_passes(self):
|
||||
change = {"approved": True, "backward_compatible": False,
|
||||
"affected_clients": ["web"], "regression_task": "t-1"}
|
||||
self.assertTrue(
|
||||
check_contract_version("v1", "v2", change=change)["ok"])
|
||||
|
||||
def test_unapproved_bump_fails(self):
|
||||
change = {"approved": False, "backward_compatible": True,
|
||||
"affected_clients": ["web"]}
|
||||
self.assertFalse(
|
||||
check_contract_version("v1", "v2", change=change)["ok"])
|
||||
|
||||
def test_bump_without_change_record_fails(self):
|
||||
self.assertFalse(check_contract_version("v1", "v2")["ok"])
|
||||
|
||||
def test_skip_bump_fails_even_when_approved(self):
|
||||
change = {"approved": True, "backward_compatible": True,
|
||||
"affected_clients": ["web"]}
|
||||
verdict = check_contract_version("v1", "v3", change=change)
|
||||
self.assertFalse(verdict["ok"])
|
||||
|
||||
def test_unparseable_version_fails(self):
|
||||
self.assertFalse(check_contract_version("v1", "draft")["ok"])
|
||||
|
||||
|
||||
class TestFrozenStack(unittest.TestCase):
|
||||
def test_unchanged_stack_passes(self):
|
||||
frozen = {"web": "React", "ios": "SwiftUI"}
|
||||
self.assertTrue(check_frozen_stack(frozen, dict(frozen))["ok"])
|
||||
|
||||
def test_new_client_is_addition_not_swap(self):
|
||||
frozen = {"web": "React"}
|
||||
current = {"web": "React", "ios": "SwiftUI"}
|
||||
self.assertTrue(check_frozen_stack(frozen, current)["ok"])
|
||||
|
||||
def test_swap_without_approval_fails(self):
|
||||
verdict = check_frozen_stack({"web": "React"}, {"web": "Vue"})
|
||||
self.assertFalse(verdict["ok"])
|
||||
self.assertEqual(verdict["swapped"], ["web"])
|
||||
|
||||
def test_removed_client_without_approval_fails(self):
|
||||
self.assertFalse(
|
||||
check_frozen_stack({"web": "React", "ios": "SwiftUI"},
|
||||
{"web": "React"})["ok"])
|
||||
|
||||
def test_swap_with_approved_change_passes(self):
|
||||
change = {"approved": True, "backward_compatible": True,
|
||||
"affected_clients": ["web"]}
|
||||
self.assertTrue(
|
||||
check_frozen_stack({"web": "React"}, {"web": "Vue"},
|
||||
change=change)["ok"])
|
||||
|
||||
|
||||
class TestDirectoryGuard(unittest.TestCase):
|
||||
def test_inside_passes_and_outside_fails(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
own = os.path.join(tmp, "work")
|
||||
os.makedirs(own)
|
||||
self.assertTrue(is_within(own, os.path.join(own, "a.txt")))
|
||||
self.assertTrue(is_within(own, own))
|
||||
self.assertFalse(is_within(own, os.path.join(tmp, "other")))
|
||||
|
||||
def test_dotdot_escape_fails(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
own = os.path.join(tmp, "work")
|
||||
os.makedirs(own)
|
||||
escape = os.path.join(own, "..", "secret.txt")
|
||||
self.assertFalse(is_within(own, escape))
|
||||
self.assertEqual(check_task_writes(own, [escape]), [escape])
|
||||
|
||||
def test_symlink_escape_fails(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
own = os.path.join(tmp, "work")
|
||||
outside = os.path.join(tmp, "outside")
|
||||
os.makedirs(own)
|
||||
os.makedirs(outside)
|
||||
link = os.path.join(own, "link")
|
||||
try:
|
||||
os.symlink(outside, link)
|
||||
except OSError:
|
||||
self.skipTest("symlinks unavailable")
|
||||
self.assertFalse(is_within(own, link))
|
||||
|
||||
def test_mixed_writes_report_only_violations(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
own = os.path.join(tmp, "work")
|
||||
os.makedirs(own)
|
||||
good = os.path.join(own, "ok.txt")
|
||||
bad = os.path.join(tmp, "bad.txt")
|
||||
self.assertEqual(check_task_writes(own, [good, bad]), [bad])
|
||||
|
||||
|
||||
class TestLayerImports(unittest.TestCase):
|
||||
def test_foundation_importing_server_fails(self):
|
||||
verdict = check_layer_imports(
|
||||
{"db": "from cloud.server import foo\n"})
|
||||
self.assertEqual(len(verdict), 1)
|
||||
self.assertEqual(verdict[0]["rule"],
|
||||
"foundation_must_not_import_orchestration")
|
||||
|
||||
def test_foundation_importing_store_passes(self):
|
||||
# db importing nothing, models importing nothing: the baseline
|
||||
# shape. A foundation module importing another foundation
|
||||
# module is not an upward import.
|
||||
self.assertEqual(check_layer_imports(
|
||||
{"models": "import os\n", "queue": "from cloud.models import x\n"}),
|
||||
[])
|
||||
|
||||
def test_sandbox_importing_db_fails(self):
|
||||
verdict = check_layer_imports(
|
||||
{"sandbox": "from cloud.db import connect\n"})
|
||||
self.assertEqual(len(verdict), 1)
|
||||
|
||||
def test_sandbox_stdlib_only_passes(self):
|
||||
self.assertEqual(check_layer_imports(
|
||||
{"sandbox": "import os\nimport subprocess\n"}), [])
|
||||
|
||||
def test_service_opening_sqlite_directly_fails(self):
|
||||
verdict = check_layer_imports(
|
||||
{"server": "import sqlite3\nconn = sqlite3.connect('x.db')\n"})
|
||||
self.assertTrue(any(v["rule"] == "service_must_not_open_driver_"
|
||||
"connection_directly" for v in verdict))
|
||||
|
||||
def test_service_sqlite_row_adapter_passes(self):
|
||||
# Deliberate narrow ban: Row/exception adapters on a
|
||||
# cloud.db connection are adaptation, not a bypass.
|
||||
self.assertEqual(check_layer_imports(
|
||||
{"server": "import sqlite3\nconn.row_factory = sqlite3.Row\n"
|
||||
"try:\n pass\nexcept sqlite3.IntegrityError:\n"
|
||||
" pass\n"}), [])
|
||||
|
||||
def test_service_importing_psycopg_fails(self):
|
||||
verdict = check_layer_imports(
|
||||
{"worker": "import psycopg\n"})
|
||||
self.assertTrue(any(v["rule"] == "service_must_not_open_driver_"
|
||||
"connection_directly" for v in verdict))
|
||||
|
||||
def test_driver_adjacent_modules_exempt(self):
|
||||
self.assertEqual(check_layer_imports(
|
||||
{"backup": "import sqlite3\nsqlite3.connect('x.db')\n",
|
||||
"migrations": "import sqlite3\nsqlite3.connect('y.db')\n"}), [])
|
||||
|
||||
def test_scan_imports_sees_from_cloud_form(self):
|
||||
scanned = scan_imports("from cloud.worker import Worker\n")
|
||||
self.assertIn("worker", scanned["cloud"])
|
||||
|
||||
|
||||
class TestRepoLevelScan(unittest.TestCase):
|
||||
def test_current_tree_has_zero_layer_violations(self):
|
||||
repo_root = Path(__file__).resolve().parents[2]
|
||||
violations = scan_cloud_package(repo_root)
|
||||
self.assertEqual(violations, [])
|
||||
|
||||
|
||||
class TestSkillProjectFixture(unittest.TestCase):
|
||||
def test_frozen_project_passes_all_checks(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
own = os.path.join(tmp, "apps", "web")
|
||||
os.makedirs(own)
|
||||
verdict = check_skill_project(
|
||||
tmp, "v1", "v1", {"web": "React"}, {"web": "React"},
|
||||
own, [os.path.join(own, "App.tsx")])
|
||||
self.assertTrue(verdict["ok"])
|
||||
|
||||
def test_drifting_project_fails_each_changed_check(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
own = os.path.join(tmp, "apps", "web")
|
||||
os.makedirs(own)
|
||||
verdict = check_skill_project(
|
||||
tmp, "v1", "v2", {"web": "React"}, {"web": "Vue"},
|
||||
own, [os.path.join(tmp, "apps", "ios", "leak.txt")])
|
||||
self.assertFalse(verdict["ok"])
|
||||
self.assertFalse(verdict["contract"]["ok"])
|
||||
self.assertFalse(verdict["stack"]["ok"])
|
||||
self.assertFalse(verdict["directory"]["ok"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,186 @@
|
||||
"""Tests for cloud.metrics: collection, dual-column cost, feedback.
|
||||
|
||||
Every metric has a positive case and an unknown/edge case; unknown
|
||||
is never reported as zero, and an unapproved golden set never makes
|
||||
a report decision-ready.
|
||||
|
||||
Only the standard library; no real model, no network, no secrets.
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import unittest
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(
|
||||
os.path.dirname(os.path.abspath(__file__)))))
|
||||
|
||||
from cloud.arch_conformance import check_contract_version # noqa: E402
|
||||
from cloud.metrics import ( # noqa: E402
|
||||
build_report, compute_metrics, evaluate_feedback)
|
||||
|
||||
|
||||
def _record(task_id, status, attempts=None, defects=None, category=None):
|
||||
return {"id": task_id, "status": status, "attempts": attempts,
|
||||
"defects": defects, "defect_category": category}
|
||||
|
||||
|
||||
class TestComputeMetrics(unittest.TestCase):
|
||||
def test_first_pass_rework_defect_rates(self):
|
||||
records = [
|
||||
_record("a", "completed", attempts=1, defects=0),
|
||||
_record("b", "merged", attempts=3, defects=2,
|
||||
category="contract"),
|
||||
_record("c", "failed", attempts=2, defects=1,
|
||||
category="contract"),
|
||||
_record("d", "queued", attempts=1), # not terminal: excluded
|
||||
_record("e", "interrupted", attempts=1), # not terminal
|
||||
]
|
||||
m = compute_metrics(records)
|
||||
self.assertEqual(m["terminal_tasks"], 3)
|
||||
self.assertAlmostEqual(m["first_pass_rate"], 1 / 3)
|
||||
self.assertAlmostEqual(m["rework_rate"], 2 / 3)
|
||||
self.assertAlmostEqual(m["defect_rate"], 2 / 3)
|
||||
self.assertEqual(m["defect_by_category"], {"contract": 2})
|
||||
|
||||
def test_unknown_attempts_and_defects_are_none_not_zero(self):
|
||||
records = [_record("a", "completed"),
|
||||
_record("b", "failed")]
|
||||
m = compute_metrics(records)
|
||||
self.assertIsNone(m["first_pass_rate"])
|
||||
self.assertIsNone(m["rework_rate"])
|
||||
self.assertIsNone(m["defect_rate"])
|
||||
self.assertFalse(m["first_pass_known"])
|
||||
self.assertFalse(m["rework_known"])
|
||||
self.assertFalse(m["defect_known"])
|
||||
|
||||
def test_empty_set_rates_are_none(self):
|
||||
m = compute_metrics([])
|
||||
self.assertEqual(m["terminal_tasks"], 0)
|
||||
self.assertIsNone(m["first_pass_rate"])
|
||||
self.assertIsNone(m["defect_rate"])
|
||||
self.assertIsNone(m["total_usd_micros"])
|
||||
self.assertFalse(m["cost_complete"])
|
||||
|
||||
|
||||
class TestDualColumnCost(unittest.TestCase):
|
||||
def test_complete_cost_totals(self):
|
||||
records = [_record("a", "completed", attempts=1)]
|
||||
costs = {"a": {"priced_usd_micros": 5000,
|
||||
"unpriced_actual_tokens": 0, "has_rows": True}}
|
||||
m = compute_metrics(records, costs)
|
||||
self.assertEqual(m["total_usd_micros"], 5000)
|
||||
self.assertTrue(m["cost_complete"])
|
||||
|
||||
def test_unpriced_cost_never_folds_into_zero_or_total(self):
|
||||
records = [_record("a", "completed", attempts=1)]
|
||||
costs = {"a": {"priced_usd_micros": 5000,
|
||||
"unpriced_actual_tokens": 700, "has_rows": True}}
|
||||
m = compute_metrics(records, costs)
|
||||
self.assertIsNone(m["total_usd_micros"])
|
||||
self.assertEqual(m["priced_usd_micros_partial"], 5000)
|
||||
self.assertFalse(m["cost_complete"])
|
||||
self.assertEqual(m["per_task_cost"][0]["unpriced_actual_tokens"],
|
||||
700)
|
||||
|
||||
def test_missing_cost_rows_make_total_unknown(self):
|
||||
records = [_record("a", "completed", attempts=1)]
|
||||
m = compute_metrics(records, costs={})
|
||||
self.assertIsNone(m["total_usd_micros"])
|
||||
self.assertFalse(m["cost_complete"])
|
||||
|
||||
|
||||
class TestFeedbackRules(unittest.TestCase):
|
||||
def test_defect_breach_tightens_check(self):
|
||||
m = compute_metrics([
|
||||
_record("a", "completed", attempts=1),
|
||||
_record("b", "failed", attempts=1, defects=1,
|
||||
category="contract"),
|
||||
])
|
||||
actions = evaluate_feedback(
|
||||
m, thresholds={"defect_rate": 0.20,
|
||||
"first_pass_rate_min": 0.50,
|
||||
"check_overturn_rate": 0.30})
|
||||
tighten = [a for a in actions if a["kind"] == "tighten_check"]
|
||||
self.assertEqual(len(tighten), 1)
|
||||
self.assertEqual(tighten[0]["target"], "contract")
|
||||
self.assertFalse(tighten[0]["auto_apply"])
|
||||
|
||||
def test_no_breach_no_action(self):
|
||||
m = compute_metrics([_record("a", "completed", attempts=1)])
|
||||
actions = evaluate_feedback(
|
||||
m, thresholds={"defect_rate": 0.20,
|
||||
"first_pass_rate_min": 0.50,
|
||||
"check_overturn_rate": 0.30})
|
||||
self.assertEqual(actions, [])
|
||||
|
||||
def test_low_first_pass_routes_with_extra_audit(self):
|
||||
m = compute_metrics([
|
||||
_record("a", "completed", attempts=3),
|
||||
_record("b", "failed", attempts=2),
|
||||
])
|
||||
actions = evaluate_feedback(
|
||||
m, thresholds={"defect_rate": 0.20,
|
||||
"first_pass_rate_min": 0.50,
|
||||
"check_overturn_rate": 0.30})
|
||||
self.assertTrue(any(a["kind"] == "route_with_extra_audit"
|
||||
for a in actions))
|
||||
|
||||
def test_high_overturn_recalibrates_without_auto_apply(self):
|
||||
m = compute_metrics([_record("a", "completed", attempts=1)])
|
||||
actions = evaluate_feedback(
|
||||
m, thresholds={"defect_rate": 0.20,
|
||||
"first_pass_rate_min": 0.50,
|
||||
"check_overturn_rate": 0.30},
|
||||
check_stats=[{"check_id": "forbidden-dir", "violations": 10,
|
||||
"overturned": 5}])
|
||||
recal = [a for a in actions if a["kind"] == "recalibrate_check"]
|
||||
self.assertEqual(len(recal), 1)
|
||||
self.assertFalse(recal[0]["auto_apply"])
|
||||
|
||||
def test_low_overturn_no_recalibration(self):
|
||||
m = compute_metrics([_record("a", "completed", attempts=1)])
|
||||
actions = evaluate_feedback(
|
||||
m, check_stats=[{"check_id": "forbidden-dir",
|
||||
"violations": 10, "overturned": 1}])
|
||||
self.assertEqual(
|
||||
[a for a in actions if a["kind"] == "recalibrate_check"], [])
|
||||
|
||||
|
||||
class TestGoldenSetAndStandardsLoop(unittest.TestCase):
|
||||
def test_unapproved_golden_set_is_not_decision_ready(self):
|
||||
m = compute_metrics([_record("a", "completed", attempts=1)])
|
||||
report = build_report(m, golden={"tasks": ["a", "b"]})
|
||||
self.assertFalse(report["decision_ready"])
|
||||
self.assertEqual(report["golden_set"]["status"],
|
||||
"pending_human_approval")
|
||||
|
||||
def test_approved_golden_set_is_decision_ready(self):
|
||||
m = compute_metrics([_record("a", "completed", attempts=1)])
|
||||
report = build_report(
|
||||
m, golden={"tasks": ["a"], "approved_by": "human",
|
||||
"approved_at": "2026-10-31"},
|
||||
thresholds_source="human_approved")
|
||||
self.assertTrue(report["decision_ready"])
|
||||
self.assertEqual(report["thresholds_source"], "human_approved")
|
||||
|
||||
def test_sample_only_set_must_not_be_presented_as_golden(self):
|
||||
# The set that exists today is sample-only: no approval fields
|
||||
# exist for it, so any report built from it stays pending.
|
||||
m = compute_metrics([_record("a", "completed", attempts=1)])
|
||||
report = build_report(m, golden={"tasks": ["sample-1"]})
|
||||
self.assertFalse(report["decision_ready"])
|
||||
self.assertIn("NOT human-approved", report["decision_note"])
|
||||
|
||||
def test_standards_change_goes_through_contract_protocol(self):
|
||||
# The loop closes through the same protocol: an audit-standard
|
||||
# version bump without approval fails; with approval it passes.
|
||||
self.assertFalse(
|
||||
check_contract_version("v1", "v2")["ok"])
|
||||
self.assertTrue(check_contract_version(
|
||||
"v1", "v2",
|
||||
change={"approved": True, "backward_compatible": True,
|
||||
"affected_clients": ["all"]})["ok"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,120 @@
|
||||
"""Endpoint test for GET /metrics-report.
|
||||
|
||||
Only the standard library http.client; a temporary SQLite DB and a
|
||||
temporary port; no real model, no network, no secrets.
|
||||
|
||||
Covers: unauthenticated -> 401, empty project -> report shape with
|
||||
unknown rates (None, not 0) and pending golden set, and a completed
|
||||
task with a priced cost row -> terminal count and dual-column cost.
|
||||
"""
|
||||
|
||||
import http.client
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
from cloud.auth import AuthStore
|
||||
from cloud.cost import CostStore
|
||||
from cloud.server import create_server
|
||||
except ImportError:
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
from cloud.auth import AuthStore
|
||||
from cloud.cost import CostStore
|
||||
from cloud.server import create_server
|
||||
|
||||
|
||||
class MetricsReportEndpointTestCase(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.db_path = os.path.join(self.tmp.name, "control.db")
|
||||
self.auth = AuthStore(self.db_path)
|
||||
self.owner_id, self.owner_key = self.auth.create_user("owner")
|
||||
self.server = create_server(self.db_path, "127.0.0.1", 0)
|
||||
self.port = self.server.server_address[1]
|
||||
self.thread = threading.Thread(
|
||||
target=self.server.serve_forever, daemon=True)
|
||||
self.thread.start()
|
||||
status, self.project = self._req(
|
||||
"POST", "/projects", {"name": "demo"}, self.owner_key)
|
||||
self.assertEqual(status, 201)
|
||||
|
||||
def tearDown(self):
|
||||
self.server.shutdown()
|
||||
self.server.server_close()
|
||||
self.thread.join(timeout=5)
|
||||
try:
|
||||
self.auth.conn.close()
|
||||
except Exception:
|
||||
pass
|
||||
self.tmp.cleanup()
|
||||
|
||||
def _req(self, method, path, body=None, key=None):
|
||||
conn = http.client.HTTPConnection("127.0.0.1", self.port, timeout=10)
|
||||
hdrs = {}
|
||||
payload = None
|
||||
if body is not None:
|
||||
payload = json.dumps(body).encode("utf-8")
|
||||
hdrs["Content-Type"] = "application/json"
|
||||
if key is not None:
|
||||
hdrs["X-API-Key"] = key
|
||||
try:
|
||||
conn.request(method, path, body=payload, headers=hdrs)
|
||||
resp = conn.getresponse()
|
||||
data = resp.read()
|
||||
finally:
|
||||
conn.close()
|
||||
try:
|
||||
parsed = json.loads(data.decode("utf-8")) if data else None
|
||||
except ValueError:
|
||||
parsed = None
|
||||
return resp.status, parsed
|
||||
|
||||
def test_unauthenticated_is_401(self):
|
||||
status, _ = self._req("GET", "/metrics-report")
|
||||
self.assertEqual(status, 401)
|
||||
|
||||
def test_empty_project_report_is_pending_and_unknown(self):
|
||||
status, report = self._req(
|
||||
"GET", "/metrics-report", key=self.owner_key)
|
||||
self.assertEqual(status, 200)
|
||||
self.assertFalse(report["decision_ready"])
|
||||
self.assertEqual(report["golden_set"]["status"],
|
||||
"pending_human_approval")
|
||||
self.assertIsNone(report["metrics"]["first_pass_rate"])
|
||||
self.assertIsNone(report["metrics"]["defect_rate"])
|
||||
self.assertIsNone(report["metrics"]["total_usd_micros"])
|
||||
self.assertEqual(report["thresholds_source"],
|
||||
"default_unapproved")
|
||||
|
||||
def test_completed_task_with_priced_cost_row(self):
|
||||
from cloud.db import connect as db_connect
|
||||
conn = db_connect(self.db_path)
|
||||
try:
|
||||
conn.execute(
|
||||
"INSERT INTO tasks(id, project_id, status) VALUES(?, ?, ?)",
|
||||
("t-1", self.project["id"], "completed"))
|
||||
conn.commit()
|
||||
finally:
|
||||
conn.close()
|
||||
store = CostStore(self.db_path)
|
||||
store.record("t-1", self.project["id"], "gpt-6-sol",
|
||||
input_tokens=1000, output_tokens=100,
|
||||
usd_micros=5000, source="actual")
|
||||
status, report = self._req(
|
||||
"GET", "/metrics-report", key=self.owner_key)
|
||||
self.assertEqual(status, 200)
|
||||
self.assertEqual(report["metrics"]["terminal_tasks"], 1)
|
||||
# Attempts are not in the schema: first-pass stays unknown,
|
||||
# it must not be fabricated as 0 or 1 from status alone.
|
||||
self.assertIsNone(report["metrics"]["first_pass_rate"])
|
||||
self.assertEqual(report["metrics"]["total_usd_micros"], 5000)
|
||||
self.assertTrue(report["metrics"]["cost_complete"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,66 @@
|
||||
# 架构一致性进 CI 与指标闭环(审计加固第 5 步)
|
||||
|
||||
实现:`cloud/arch_conformance.py`(纯检查,标准库)、`cloud/metrics.py`
|
||||
(纯计算,标准库)、端点 `GET /metrics-report`(`cloud/server.py`,
|
||||
与 `/metrics` 同款认证与成员范围)。测试失败即 CI 失败:现有
|
||||
`host-verify` 工作流已跑 `discover -s cloud/tests`,本轨未改任何
|
||||
workflow——第 2 轨拥有 lint/类型/安扫,本轨不碰。
|
||||
|
||||
## 架构一致性检查(规则、理由、测试)
|
||||
|
||||
| 检查 | 规则 | 理由 | 测试 |
|
||||
|---|---|---|---|
|
||||
| 合同版本 | 冻结后版本只能 +1,且须有协调员批准的变更记录(声明向后兼容、列出受影响客户端;破坏性变更须有回归任务) | 契约静默漂移是并行线程分叉的根源;协议是唯一合法路径,检查执行协议而非只记录协议 | `test_arch_conformance.py::TestContractVersion`(正向:等值/批准+1/破坏性+回归任务;负向:未批准、无记录、跳号、不可解析) |
|
||||
| 禁目录 | 任务只能写自有目录;`..` 与符号链接逃逸在解析后判定 | 跨目录写会 corrupt 他人 worktree | `TestDirectoryGuard`(正向:目录内;负向:`..`、symlink、混合写只报违规项)。口径复用已有定义,不另造:控制面 = 租户 work 目录(`worker._tenant_dirs` 的 `work`),技能项目 = `apps/<client>/` |
|
||||
| 禁 import | 基础层(db/models/queue/logging_config)不 import 编排层;执行层(sandbox)不 import 持久化/编排层;除 db/backup/migrations 外任何 cloud 模块不得直接 `sqlite3.connect` 或 import `psycopg*`(服务层走 `cloud.db.connect` 或 Store) | 基础层上引是循环依赖前兆;执行器须可换、可假 runner 测试;驱动直连绕过 sqlite/postgres 唯一分发点 | `TestLayerImports`(每项正向+负向)+ 仓库级 `TestRepoLevelScan`(当前主线零违规,未来违规即 CI 失败)。有意不禁:`server.py` 的 `sqlite3.Row`/异常类型适配(那是在 `cloud.db` 连接上的适配,不是绕过;禁它会在基线误报) |
|
||||
| 技术栈冻结 | G2 冻结表变更/删除须有批准的变更记录;新增客户端是新增不是换栈,通过 | 共享核心按冻结表构建,中途换栈使其静默失效 | `TestFrozenStack`(正向:不变/新增;负向:换栈/删除无批准) |
|
||||
|
||||
技能项目一键版:`check_skill_project` 把合同版本、冻结栈、目录写
|
||||
三项合在一起验,`TestSkillProjectFixture` 用合成项目目录正向/负向
|
||||
各验一次。静态 AST 扫描看不到 `importlib` 动态导入——这是已知
|
||||
边界,不声称覆盖。
|
||||
|
||||
## 指标采集与报告
|
||||
|
||||
端点:`GET /metrics-report`(需 X-API-Key,成员范围)。报告格式
|
||||
见 `cloud/metrics.py::build_report`:`golden_set`、`decision_ready`、
|
||||
`metrics`、`thresholds` + `thresholds_source`、`feedback_actions`、
|
||||
`feedback_consumers`。
|
||||
|
||||
指标口径(`compute_metrics`):
|
||||
|
||||
- 首过绿灯率 = 首过成功任务 / 已知 attempts 的终态任务;缺陷率 =
|
||||
有缺陷任务 / 已知 defects 的终态任务;返工率 = attempts>1 任务 /
|
||||
已知 attempts 的终态任务。终态 = completed/merged/failed/
|
||||
merge-failed;`interrupted` 不算终态(平台中断自动重跑,算失败
|
||||
是罚平台事件)。
|
||||
- 每任务成本复用 #212 双列口径:有价 USD micros 与无价 actual
|
||||
token 分列;成本不完整(无成本行或有无价 token)时
|
||||
`total_usd_micros` 为 `None`,有价部分只以
|
||||
`priced_usd_micros_partial` 出现——无价永不记 0、永不折进总额。
|
||||
- 未知率回 `None` + `*_known: false`,绝不报 0。
|
||||
|
||||
当前诚实边界(端点实测):tasks 表无 attempts/defects 列,端点
|
||||
传 `None`,首过/缺陷/返工率恒为 `None`;金任务集未经人工审定,
|
||||
报告恒为 `pending_human_approval`、`decision_ready: false`。接真实
|
||||
输入需两项前置(见待审定清单),本轨不编造。
|
||||
|
||||
## 回流与审计标准迭代(可测试规则,非口号)
|
||||
|
||||
`evaluate_feedback` 是纯函数,只产出 `auto_apply: false` 的行动
|
||||
记录,不自动改路由、不自动改标准;行动由两处消费:
|
||||
|
||||
- 路由:`tighten_check`(某类缺陷率 > 阈值)→ 该类任务路由时加
|
||||
一道审计门;`route_with_extra_audit`(首过率 < 下限)→ 全量加
|
||||
审计门。消费动作本身须经协调员/人工确认后才生效。
|
||||
- 审计标准:`tighten_check` / `recalibrate_check`(某检查的违规被
|
||||
复核推翻率 > 阈值)各开一条标准复核项;标准文档本身的版本
|
||||
变更走同一合同变更协议(`check_contract_version`,批准 +1),
|
||||
测试 `TestGoldenSetAndStandardsLoop` 钉住“无批准改标准失败、
|
||||
有批准通过”。误报复核规则:推翻率高只触发“复核阈值”行动,
|
||||
绝不让检查自动放宽自己。
|
||||
|
||||
默认阈值(`DEFAULT_THRESHOLDS`:缺陷率 0.20、首过下限 0.50、
|
||||
推翻率 0.30)只是测试用初值,报告标注
|
||||
`thresholds_source: "default_unapproved"`;生产阈值须人工拍板后
|
||||
以 `thresholds_source: "human_approved"` 传入,见待审定清单。
|
||||
Reference in new issue
Block a user