feat(chv): 阶段 4 截图三级管线——a11y 角色解析 + diff_shot 逐像素裁剪 + verify_shot 全屏校验

- snapshot 纯字符串解析 a11y 角色计数/interactive/字符数/hash
  (不跑 JS、不取原始 DOM,禁 op 保持拒绝);status 带计数器
  与 verify_share
- diff_shot:PNG 全帧 Pillow 逐像素比对(单通道阈值 10),
  无变化 changed:false size 0;有变化 bbox 外扩 8px、钳制视口、
  宽高下限 32px,再 clip 截 JPEG q70;Pillow 缺失如实报错并
  指引回落 screenshot 全帧;open 不清基线(跨页局部差分靠它)
- verify_shot:全帧 JPEG q70,只用于检查点
- build.sh 与 playwright 同批 pip 装 pillow(只加 pillow);
  buildinfo BASE_CONTENTS 登记 pillow(pip 名覆盖)
- 单测 cloud/tests/test_chv_opt_stage4.py:31 项(bbox 纯函数、
  a11y 解析、计数器、Pillow 缺失回落、禁 op、lockfile 往返)
This commit is contained in:
host committed 2026-11-08 09:45:00 +00:00
1 parent 5dc75128f8
commit f723eb8a88
4 files changed
+893 -6

No files matched your search

+323 -5
View File
@@ -24,8 +24,32 @@
executable_path,不走 channel/env)。
* 截图:JPEG quality 70,viewport 宽封顶 1280(上下文创建时
定死 1280x800);diff_only 时与上一帧比对,字节相同即不落盘、
返回 changed=false(整帧相同跳过;变化区裁剪是阶段 4 的事,
这里不冒充已做)。
返回 changed=false(整帧相同跳过)。
* 阶段 4(feat-chv-opt-landing):截图三级管线。
一级 a11y+DOM 主通道:snapshot 在 aria_snapshot 字符串基础
上纯字符串解析出角色计数(button/link/textbox/heading 等)
与 interactive 总数、a11y 字符数与 a11y_hash——不跑 JS、不取
原始 DOM、不新增 evaluate/content 通路,禁 op 保持拒绝。
二级差分截图(op `diff_shot`):先截 PNG 全帧,用 Pillow
解码与上一帧逐像素比对——像素判变阈值 DIFF_PIXEL_THRESHOLD
=10(单通道差值 >10 才算变,滤掉抗锯齿 1 位抖动);无变化
返回 changed:false size 0;有变化算出变化区 bbox(外扩
margin 8px、钳制到视口内、宽高下限 32px),再用
page.screenshot(type=jpeg, quality=70, clip=bbox) 只截变
化区。Pillow 缺失时 diff_shot 如实报错并指引回落全帧
screenshot,绝不静默假装裁剪。full_size 是本帧 PNG 经
Pillow 以 JPEG q70 编码得到的全帧字节数(与浏览器直出
JPEG q70 同参数、字节数可能有数百字节差异,仅作对照)。
三级全屏校验(op `verify_shot`):全帧 JPEG q70,专用于
少数校验场景。事件驱动口径:日常只在动作后调 diff_shot
(变化才出图),verify_shot 只在检查点调用。计数器
shots_full/shots_diff/shots_verify/shots_skipped 由
status 返回,verify_share = shots_verify /
(shots_full+shots_diff+shots_verify),分母为 0 时记 0.0。
计数口径:shots_full = screenshot 落盘数;shots_diff =
diff_shot 返回 changed:true 的数;shots_verify =
verify_shot 成功数;shots_skipped = screenshot diff_only
跳过数 + diff_shot changed:false 数。
对齐 Muse P2 的不变口径:子代理只向单元返回页面无障碍快照,
不见原始 DOM,不能在页面上下文跑 JS,不暴露 evaluate/content,
@@ -35,10 +59,19 @@ page.accessibility,改用 page.aria_snapshot()(返回 YAML 式
操作(Unix socket /run/chv/browser.sock,JSON 行):
open {url, session?} -> 导航并返回 {title, url, a11y, truncated}
snapshot {session?} -> 当前页的 {title, url, a11y, truncated}
snapshot {session?} -> 当前页的 {title, url, a11y, truncated,
a11y_roles, a11y_interactive,
a11y_chars, a11y_hash}
screenshot {session?, diff_only?} -> JPEG 落盘 /run/chv/shots/<uuid>.jpg,
返回 {path, size, format, quality,
width, changed, sha256}
diff_shot {session?} -> PNG 全帧逐像素比对;无变化
{changed:false, size:0};有变化只截变
化区 JPEG,返回 {changed:true, bbox,
size, full_size, path, sha256}
verify_shot {session?} -> 全帧 JPEG q70(只用于检查点),
返回 {path, size, format, quality,
width, sha256}
close {session?} -> 关该会话的页(浏览器进程继续跑)
sessions {} -> 各会话 {session, url, title}
status {} -> {engine, profile, profile_on_data_disk,
@@ -95,6 +128,20 @@ MAX_REQUEST_BYTES = 65536
MAX_A11Y_NODES = 200
MAX_A11Y_CHARS = 65536
NAV_TIMEOUT_MS = 30000
# 阶段 4 差分截图口径(写死在代码里,驱动与报告按此口径实测):
# 像素判变阈值:单通道差值 >10 才算变(滤抗锯齿 1 位抖动);
# bbox 外扩 margin 8px;变化区宽高下限 32px(不足则绕中心扩到
# 32px 再钳制到视口内)。
DIFF_PIXEL_THRESHOLD = 10
DIFF_MARGIN_PX = 8
DIFF_MIN_SIZE_PX = 32
# a11y 角色里算 interactive 的集合(aria_snapshot 字符串解析用;
# searchbox 是 textbox 的搜索变体,同样算可交互)。
INTERACTIVE_ROLES = frozenset([
"button", "link", "textbox", "searchbox", "checkbox", "radio",
"combobox", "listbox", "menuitem", "tab", "switch", "slider",
"spinbutton",
])
# 拒绝的 URL 主机:localhost 与私有地址字面量(主机名判定)。
_PRIVATE_HOST_RE = re.compile(
@@ -215,6 +262,137 @@ def truncate_a11y(node, max_nodes=MAX_A11Y_NODES,
return tree, state["truncated"]
def parse_a11y(a11y):
"""解析无障碍快照出角色计数(纯函数,可单测)。
输入是 aria_snapshot 字符串(主路)或旧版 dict 树(回落)。
字符串按行解析:行形如 `- link "More information..." [ref=e3]`,
嵌套行带前导空格;角色取 `- ` 后的第一个 token。dict 树则
递归数每个节点的 role 字段。返回 (roles, interactive, chars,
a11y_hash):roles 是 {角色: 个数},interactive 是
INTERACTIVE_ROLES 内的角色总数,chars 是快照字符数,
a11y_hash 是快照规范文本的 sha256(字符串用原文,dict 用
sort_keys 的 JSON)。不跑 JS、不取原始 DOM。"""
roles = {}
if isinstance(a11y, str):
text = a11y
for line in text.splitlines():
m = re.match(r"^\s*-\s+([A-Za-z][\w-]*)", line)
if m:
role = m.group(1).lower()
roles[role] = roles.get(role, 0) + 1
elif isinstance(a11y, dict):
text = json.dumps(a11y, ensure_ascii=False, sort_keys=True)
def walk(n):
if not isinstance(n, dict):
return
role = n.get("role")
if isinstance(role, str) and role:
key = role.lower()
roles[key] = roles.get(key, 0) + 1
for value in n.values():
if isinstance(value, list):
for child in value:
walk(child)
elif isinstance(value, dict):
walk(value)
walk(a11y)
else:
text = ""
interactive = sum(count for role, count in roles.items()
if role in INTERACTIVE_ROLES)
digest = hashlib.sha256(text.encode()).hexdigest()
return roles, interactive, len(text), digest
def expand_bbox(x0, y0, x1, y1, width, height,
margin=DIFF_MARGIN_PX, min_size=DIFF_MIN_SIZE_PX):
"""变化区 bbox 外扩 + 钳制 + 最小尺寸(纯函数,可单测)。
输入是像素差分的包围盒(左闭右开 [x0,y0,x1,y1))。先外扩
margin,再钳制到 [0,0,width,height] 内;宽/高不足 min_size
时绕中心扩到 min_size 再钳制(图本身比 min_size 还小时取
整幅宽/高)。返回 Playwright clip 形 {"x","y","width",
"height"}(均为 int)。"""
x0 = max(0, x0 - margin)
y0 = max(0, y0 - margin)
x1 = min(width, x1 + margin)
y1 = min(height, y1 + margin)
# 宽/高不足 min_size 时绕中心扩;一侧贴边扩不动时向另一侧
# 补足(只有图本身比 min_size 还窄/矮时才取整幅宽/高)。
for axis in ("x", "y"):
lo, hi, size = (x0, x1, width) if axis == "x" \
else (y0, y1, height)
if hi - lo < min_size:
need = min_size - (hi - lo)
lo = max(0, lo - need // 2)
hi = min(size, hi + (need - need // 2))
if hi - lo < min_size:
if lo == 0:
hi = min(size, lo + min_size)
elif hi == size:
lo = max(0, hi - min_size)
if hi - lo < min_size:
lo, hi = 0, size
if axis == "x":
x0, x1 = lo, hi
else:
y0, y1 = lo, hi
return {"x": int(x0), "y": int(y0),
"width": int(x1 - x0), "height": int(y1 - y0)}
def _load_pillow():
"""懒加载 Pillow(diff_shot 专用);缺失时报明错并指引回落
全帧 screenshot,绝不静默假装裁剪。返回 (Image, ImageChops)。"""
try:
from PIL import Image, ImageChops
except ImportError as e:
raise RuntimeError(
"diff_shot 需要 Pillow(%s):镜像里 pip 未装 pillow,"
"无法逐像素比对与裁剪;回落:用 screenshot 全帧 JPEG "
"q70(op screenshot,不带 diff_only)" % e)
return Image, ImageChops
def compute_diff_bbox(prev_png, cur_png,
margin=DIFF_MARGIN_PX,
min_size=DIFF_MIN_SIZE_PX,
pillow=None):
"""两帧 PNG 的逐像素比对(纯函数,可单测,需 Pillow)。
返回 (changed, bbox):无变化时 (False, None);有变化时
(True, clip dict)。像素判变:逐通道差值 > DIFF_PIXEL_THRESHOLD
才算变(见模块 docstring)。两帧尺寸不同时视为整帧变化,
bbox 即当前帧全幅。pillow 供单测注入 (Image, ImageChops),
正常运行时用 _load_pillow()。"""
Image, ImageChops = pillow or _load_pillow()
import io
prev = Image.open(io.BytesIO(prev_png)).convert("RGB")
cur = Image.open(io.BytesIO(cur_png)).convert("RGB")
width, height = cur.size
if prev.size != cur.size:
return True, {"x": 0, "y": 0, "width": width,
"height": height}
diff = ImageChops.difference(prev, cur)
# 逐通道阈值掩码:任一通道差值 >10 即变(convert("L") 会按
# 亮度加权、漏掉单通道小变化,故三通道分别取阈值再取并)。
masks = []
for channel in diff.split():
masks.append(channel.point(
lambda p: 255 if p > DIFF_PIXEL_THRESHOLD else 0))
mask = masks[0]
for extra in masks[1:]:
mask = ImageChops.lighter(mask, extra)
box = mask.getbbox()
if box is None:
return False, None
return True, expand_bbox(box[0], box[1], box[2], box[3],
width, height, margin, min_size)
def _load_playwright():
"""懒加载 playwright;缺失时报明错(不静默假装有浏览器)。
返回 sync_playwright() 的上下文管理器实例(须调用,
@@ -278,6 +456,10 @@ class BrowserAgent:
self._engine = "shell"
self._mounted_full = False
self._last_shot = {}
# 阶段 4:上一帧 PNG(diff_shot 的比对基线)与截图计数器。
self._last_png = {}
self._counters = {"shots_full": 0, "shots_diff": 0,
"shots_verify": 0, "shots_skipped": 0}
# -- 浏览器生命周期 ----------------------------------------------
def _close_context(self):
@@ -394,6 +576,11 @@ class BrowserAgent:
pass
page = self._context.new_page()
self._pages[session] = page
# 注意:open 不清 diff 基线——diff_shot 永远与“上一帧”
# 逐像素比对,不管上一帧是哪一页;导航后调 diff_shot
# 得到的正是新旧两页渲染的真实差异区(跨页局部变化
# 如搜索词 Cat->Dog 就靠这个)。只有 close(会话结束)
# 才清基线,下次 open 后的首次 diff_shot 为全幅首帧。
page.goto(url, timeout=NAV_TIMEOUT_MS)
page.wait_for_load_state("domcontentloaded",
timeout=NAV_TIMEOUT_MS)
@@ -431,8 +618,12 @@ class BrowserAgent:
"aria_snapshot/accessibility)"}
except Exception as e:
return {"ok": False, "error": "取快照失败:%s" % e}
# 阶段 4 一级通道:纯字符串解析 enrich(不跑 JS、不取 DOM)。
roles, interactive, chars, digest = parse_a11y(tree)
return {"ok": True, "title": title, "url": url,
"a11y": tree, "truncated": truncated,
"a11y_roles": roles, "a11y_interactive": interactive,
"a11y_chars": chars, "a11y_hash": digest,
"session": session}
def screenshot(self, session=None, diff_only=False):
@@ -453,7 +644,8 @@ class BrowserAgent:
digest = hashlib.sha256(data).hexdigest()
if diff_only and self._last_shot.get(session) == digest:
# 整帧与上一帧字节相同:跳过落盘(阶段 4 的变化区
# 裁剪不在本阶段做,这里只做整帧相同跳过)。
# 裁剪走 diff_shot op,这里只做整帧相同跳过)。
self._counters["shots_skipped"] += 1
return {"ok": True, "changed": False, "size": 0,
"path": None, "format": "jpeg",
"quality": JPEG_QUALITY,
@@ -467,11 +659,124 @@ class BrowserAgent:
except OSError as e:
return {"ok": False, "error": "截图落盘失败:%s" % e}
self._last_shot[session] = digest
self._counters["shots_full"] += 1
return {"ok": True, "path": path, "size": len(data),
"format": "jpeg", "quality": JPEG_QUALITY,
"width": VIEWPORT_WIDTH, "changed": True,
"sha256": digest, "session": session}
def diff_shot(self, session=None):
"""二级差分截图(阶段 4):PNG 全帧逐像素比对,只截变化区。
无变化:{changed:false, size:0}(计 shots_skipped)。
有变化:clip=bbox 的 JPEG q70 落盘,返回 {changed:true,
bbox, size, full_size, path, sha256}(计 shots_diff)。
首帧(本会话无上一帧)视为整帧变化,bbox 为全视口。
Pillow 缺失时 ok:false 并指引回落 screenshot 全帧,
不计入计数器、不落盘、不假装裁剪。"""
session = session or DEFAULT_SESSION
page = self._page(session)
if page is None:
return {"ok": False,
"error": "会话 %s 还没有打开的页面(先 open)"
% session}
try:
png = page.screenshot(type="png")
except Exception as e:
return {"ok": False, "error": "截图失败:%s" % e}
if not isinstance(png, (bytes, bytearray)):
return {"ok": False, "error": "截图未返回字节数据"}
png = bytes(png)
try:
Image, _ImageChops = _load_pillow()
except RuntimeError as e:
return {"ok": False, "error": str(e),
"fallback": "screenshot"}
prev = self._last_png.get(session)
if prev is None:
changed, bbox = True, {"x": 0, "y": 0,
"width": VIEWPORT_WIDTH,
"height": VIEWPORT_HEIGHT}
else:
try:
changed, bbox = compute_diff_bbox(prev, png)
except Exception as e:
return {"ok": False,
"error": "差分比对失败:%s" % e}
# full_size:本帧 PNG 经 Pillow 以 JPEG q70 编码的全帧
# 字节数(与浏览器直出同参数,仅作对照,不直接等同)。
import io
try:
buf = io.BytesIO()
Image.open(io.BytesIO(png)).convert("RGB").save(
buf, "JPEG", quality=JPEG_QUALITY)
full_size = len(buf.getvalue())
except Exception:
full_size = len(png)
self._last_png[session] = png
if not changed:
self._counters["shots_skipped"] += 1
return {"ok": True, "changed": False, "size": 0,
"path": None, "bbox": None,
"full_size": full_size, "format": "jpeg",
"quality": JPEG_QUALITY, "session": session}
try:
clip = page.screenshot(type="jpeg",
quality=JPEG_QUALITY, clip=bbox)
except Exception as e:
return {"ok": False, "error": "变化区截图失败:%s" % e}
if not isinstance(clip, (bytes, bytearray)):
return {"ok": False, "error": "变化区截图未返回字节数据"}
clip = bytes(clip)
try:
os.makedirs(self._shots_dir, exist_ok=True)
path = os.path.join(self._shots_dir, "%s.jpg"
% uuid.uuid4().hex)
with open(path, "wb") as fh:
fh.write(clip)
except OSError as e:
return {"ok": False, "error": "截图落盘失败:%s" % e}
self._counters["shots_diff"] += 1
return {"ok": True, "changed": True, "bbox": bbox,
"path": path, "size": len(clip),
"full_size": full_size, "format": "jpeg",
"quality": JPEG_QUALITY,
"sha256": hashlib.sha256(clip).hexdigest(),
"session": session}
def verify_shot(self, session=None):
"""三级全屏校验(阶段 4):全帧 JPEG q70,只用于检查点。
计 shots_verify。日常动作后用 diff_shot,检查点才调它。"""
session = session or DEFAULT_SESSION
page = self._page(session)
if page is None:
return {"ok": False,
"error": "会话 %s 还没有打开的页面(先 open)"
% session}
try:
data = page.screenshot(type="jpeg",
quality=JPEG_QUALITY)
except Exception as e:
return {"ok": False, "error": "截图失败:%s" % e}
if not isinstance(data, (bytes, bytearray)):
return {"ok": False, "error": "截图未返回字节数据"}
data = bytes(data)
try:
os.makedirs(self._shots_dir, exist_ok=True)
path = os.path.join(self._shots_dir, "%s.jpg"
% uuid.uuid4().hex)
with open(path, "wb") as fh:
fh.write(data)
except OSError as e:
return {"ok": False, "error": "截图落盘失败:%s" % e}
self._last_shot[session] = hashlib.sha256(data).hexdigest()
self._counters["shots_verify"] += 1
return {"ok": True, "path": path, "size": len(data),
"format": "jpeg", "quality": JPEG_QUALITY,
"width": VIEWPORT_WIDTH, "changed": True,
"sha256": self._last_shot[session],
"session": session}
def close(self, session=None):
session = session or DEFAULT_SESSION
page = self._pages.pop(session, None)
@@ -481,6 +786,7 @@ class BrowserAgent:
except Exception:
pass
self._last_shot.pop(session, None)
self._last_png.pop(session, None)
return {"ok": True, "closed": True, "session": session}
def sessions(self):
@@ -499,6 +805,11 @@ class BrowserAgent:
# -- 服务化:状态 / CDP / 引擎 --------------------------------------
def status(self):
total = (self._counters["shots_full"]
+ self._counters["shots_diff"]
+ self._counters["shots_verify"])
verify_share = (self._counters["shots_verify"] / total
if total else 0.0)
return {"ok": True, "engine": self._engine,
"profile": self._profile_dir,
"profile_on_data_disk": self._profile_on_data_disk,
@@ -509,7 +820,9 @@ class BrowserAgent:
"mounted_full": self._mounted_full,
"full_present": full_chromium_present(
self._full_browsers_dir),
"full_browsers_dir": self._full_browsers_dir}
"full_browsers_dir": self._full_browsers_dir,
"counters": dict(self._counters),
"verify_share": verify_share}
def cdp(self):
ok, err = self._ensure_browser()
@@ -607,6 +920,10 @@ def handle_request(agent, req):
if op == "screenshot":
return agent.screenshot(req.get("session"),
bool(req.get("diff_only")))
if op == "diff_shot":
return agent.diff_shot(req.get("session"))
if op == "verify_shot":
return agent.verify_shot(req.get("session"))
if op == "close":
return agent.close(req.get("session"))
if op == "sessions":
@@ -623,6 +940,7 @@ def handle_request(agent, req):
return agent.unmount_full()
return {"ok": False,
"error": "未知操作:%r(白名单:open/snapshot/screenshot/"
"diff_shot/verify_shot/"
"close/sessions/status/cdp/engine/mount_full/"
"unmount_full/ping)" % (op,)}