Skip to content

feat(vom): expose record-safe semantic observations - #100

Draft
shnpd wants to merge 6 commits into
mainfrom
feat/record-vom-observation
Draft

feat(vom): expose record-safe semantic observations#100
shnpd wants to merge 6 commits into
mainfrom
feat/record-vom-observation

Conversation

@shnpd

@shnpd shnpd commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator

背景

录制功能需要保存页面的语义观察,以便将操作步骤与对应页面元素准确关联。现有 VOM 捕获流程内嵌在 snapshot / observe 中,无法直接被录制模块复用,同时录制结果也需要避免包含用户输入等敏感数据。

主要改动

本 PR 抽取了共享的 captureVomObservation,统一处理 Accessibility Tree 获取、页面信息捕获、VOM Scene 构建和语义文本渲染。现有 snapshot / observe 已迁移到该接口,并保留原有的取消、失败降级、Overlay 排除及 Conditional Surface 等行为。

VOM 渲染新增了统一隐藏表单值的能力。启用后会保留字段的填写状态,但不会输出邮箱、手机号、密码等真实内容。密码、验证码和信用卡相关字段也会被自动识别为敏感字段。

此外,渲染后的元素引用新增了角色、名称、上下文和文本行号,方便录制模块将用户操作与 VOM 中对应的语义元素关联。

兼容性

  • redactValues 默认关闭,不影响现有调用方
  • snapshot / observe 的协议返回结构保持不变
  • 元素引用保留原有的 refbackendNodeId
  • 本 PR 仅提供录制所需的共享能力,尚未直接接入录制流程

测试

补充了表单值脱敏、敏感字段识别、引用语义信息和文本行号相关测试,并调整现有测试以兼容扩展后的引用结构。

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Suppressed comments (3)

apps/extension/src/tools/capture-vom-observation.ts:121

  • The function currently always returns captured: captured.nodes. If includeCaptured stays optional/opt-in, the return value should conditionally include it, and when redactValues is enabled it should scrub formValue to avoid leaking user-entered data.
  return {
    text: rendered.text,
    refs: rendered.refs,
    truncated: rendered.truncated,
    captured: captured.nodes,

packages/vom/src/render.ts:637

  • redactValues currently masks node.value for all nodes, not just fillable form controls. Since VomOptions.redactValues is documented as masking “form values”, enabling it could also hide useful non-form values (e.g., sliders/progress values) that are not sensitive.
  const value =
    node.sensitive || redactValues
      ? rawValue !== undefined
        ? SENSITIVE_MASK
        : undefined

apps/extension/src/tools/capture-vom-observation.ts:4

  • This file introduces a runtime circular import: observation.ts imports captureVomObservation from this module, while this module imports buildVomScene from observation.ts. Circular module dependencies are fragile (especially if any bundling/transpilation outputs CJS) and make it easy to accidentally introduce initialization-order bugs later.
import { type RenderedRef, renderVom, type VomOptions } from "@browser-skill/vom";
import type { CdpAxNode } from "./observation";
import { buildVomScene } from "./observation";
import type { CdpRunner } from "./shared";

Comment thread apps/extension/src/tools/capture-vom-observation.ts Outdated
@shnpd
shnpd marked this pull request as draft August 18, 2026 08:58
@Ljy-0827
Ljy-0827 force-pushed the feat/record-vom-observation branch from 84f37ab to b1e28a1 Compare August 19, 2026 03:14
shnpd and others added 3 commits August 19, 2026 15:05
Share the VOM capture pipeline with recording, enrich rendered refs, and support value redaction without regressing cooperative cancellation.
@Ljy-0827
Ljy-0827 force-pushed the feat/record-vom-observation branch from 1d81da0 to 3df2738 Compare August 19, 2026 07:05
@Ljy-0827
Ljy-0827 marked this pull request as ready for review August 19, 2026 07:11
@iuyo5678

Copy link
Copy Markdown
Collaborator

整体方向是没有问题:抽取 captureVomObservation 供录制流程复用,同时增加值脱敏和更丰富的元素引用信息。

一个比较重要的问题是:返回对象并不真正满足 record-safe。
redactValues: true 目前只会脱敏 renderVom 生成的 text,例如把输入值替换成 •••;
但 captureVomObservation 返回的 frameDocuments 仍然包含原始 DOM/AX 数据,其中可能存在:
formValue
formDefaultValue
attrs.value
AX node 的 value
就是:文本虽然已经脱敏,但只要agent侧或者应用侧记录日志、序列化或持久化完整返回对象,邮箱、手机号、验证码、信用卡号等用户输入仍可能被保存下来。

建议在合并前完成以下修改:
最好不要从公开的 record-safe API 返回完整原始 frameDocuments,只返回录制匹配所需的最小字段,例如 frameId、backendNodeId、tag、rect 和 target 信息。

如果必须返回 frameDocuments,在 redactValues: true 时需要同时清除 DOM 和 AX 数据中的原始 value 字段。

增加一点回归测试:对完整 result 执行 JSON.stringify,确认邮箱、密码、OTP、信用卡号等测试值均不存在。

因为 PR 的核心目标之一就是提供 record-safe 的语义观察能力,所以这个问题属于合并阻塞项。修复并补充测试后,就可以merge了 @shnpd

Return only frame identity and node geometry for matching so a serialized capture result cannot leak form values.
@shnpd
shnpd marked this pull request as draft August 21, 2026 02:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants