feat(vom): expose record-safe semantic observations - #100
Conversation
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.
Suppressed comments (3)
apps/extension/src/tools/capture-vom-observation.ts:121
- The function currently always returns
captured: captured.nodes. IfincludeCapturedstays optional/opt-in, the return value should conditionally include it, and whenredactValuesis enabled it should scrubformValueto avoid leaking user-entered data.
return {
text: rendered.text,
refs: rendered.refs,
truncated: rendered.truncated,
captured: captured.nodes,
packages/vom/src/render.ts:637
redactValuescurrently masksnode.valuefor all nodes, not just fillable form controls. SinceVomOptions.redactValuesis documented as masking “form values”, enabling it could also hide useful non-form values (e.g., sliders/progress values) that are not sensitive.
const value =
node.sensitive || redactValues
? rawValue !== undefined
? SENSITIVE_MASK
: undefined
apps/extension/src/tools/capture-vom-observation.ts:4
- This file introduces a runtime circular import:
observation.tsimportscaptureVomObservationfrom this module, while this module importsbuildVomScenefromobservation.ts. Circular module dependencies are fragile (especially if any bundling/transpilation outputs CJS) and make it easy to accidentally introduce initialization-order bugs later.
import { type RenderedRef, renderVom, type VomOptions } from "@browser-skill/vom";
import type { CdpAxNode } from "./observation";
import { buildVomScene } from "./observation";
import type { CdpRunner } from "./shared";
84f37ab to
b1e28a1
Compare
Share the VOM capture pipeline with recording, enrich rendered refs, and support value redaction without regressing cooperative cancellation.
1d81da0 to
3df2738
Compare
|
整体方向是没有问题:抽取 captureVomObservation 供录制流程复用,同时增加值脱敏和更丰富的元素引用信息。 一个比较重要的问题是:返回对象并不真正满足 record-safe。 建议在合并前完成以下修改: 如果必须返回 frameDocuments,在 redactValues: true 时需要同时清除 DOM 和 AX 数据中的原始 value 字段。 增加一点回归测试:对完整 result 执行 JSON.stringify,确认邮箱、密码、OTP、信用卡号等测试值均不存在。 因为 PR 的核心目标之一就是提供 record-safe 的语义观察能力,所以这个问题属于合并阻塞项。修复并补充测试后,就可以merge了 @shnpd |
Return only frame identity and node geometry for matching so a serialized capture result cannot leak form values.
背景
录制功能需要保存页面的语义观察,以便将操作步骤与对应页面元素准确关联。现有 VOM 捕获流程内嵌在
snapshot/observe中,无法直接被录制模块复用,同时录制结果也需要避免包含用户输入等敏感数据。主要改动
本 PR 抽取了共享的
captureVomObservation,统一处理 Accessibility Tree 获取、页面信息捕获、VOM Scene 构建和语义文本渲染。现有snapshot/observe已迁移到该接口,并保留原有的取消、失败降级、Overlay 排除及 Conditional Surface 等行为。VOM 渲染新增了统一隐藏表单值的能力。启用后会保留字段的填写状态,但不会输出邮箱、手机号、密码等真实内容。密码、验证码和信用卡相关字段也会被自动识别为敏感字段。
此外,渲染后的元素引用新增了角色、名称、上下文和文本行号,方便录制模块将用户操作与 VOM 中对应的语义元素关联。
兼容性
redactValues默认关闭,不影响现有调用方snapshot/observe的协议返回结构保持不变ref和backendNodeId测试
补充了表单值脱敏、敏感字段识别、引用语义信息和文本行号相关测试,并调整现有测试以兼容扩展后的引用结构。