Skip to content

Add bounded Browser page runtime and agent control - #1869

Open
brsbl wants to merge 12 commits into
mainfrom
bb/browser-page-runtime-final-thr_p9qmfkggwm
Open

Add bounded Browser page runtime and agent control#1869
brsbl wants to merge 12 commits into
mainfrom
bb/browser-page-runtime-final-thr_p9qmfkggwm

Conversation

@brsbl

@brsbl brsbl commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • add a general, bounded Browser-page script runtime targeting an explicit BB Browser tab
  • run page scripts in an isolated world by default, with JSON-only inputs/results, cancellation, time and byte limits, and navigation invalidation
  • broker exact client/window/tab requests between server-side agent tools and the selected desktop Browser view
  • expose audited agent operations for tab discovery, DOM snapshots, click, type, keyboard, scroll, navigation, screenshots, and bounded ephemeral scripts
  • preserve the plugin-owned experimental_browserAction entry point without retaining the discarded Browser-specific inspection schema
  • keep the host external-link action rightmost and move excess plugin Browser actions into a keyboard-accessible overflow menu
  • publish the additive Browser plugin surface as SDK 0.4.9 now that 0.4.8 is released

Security and lifecycle

  • Browser scripts receive no Node, Electron, or BB app-shell objects
  • target selection is explicit and epoch-bound; main-frame navigation, tab detach, window close, disconnect, timeout, and caller abort invalidate in-flight work
  • AbortSignal remains renderer-local; cancellation crosses Electron IPC only as a bounded { tabId, requestId } message
  • concurrent calls use independent request IDs and cleanup paths
  • custom scripts remain inside the same bounded runtime and normal native agent-tool permission/audit flow
  • a persisted manifest generated by the real 0.4.8 scaffold proves prior-release plugins continue loading under 0.4.9

Visual evidence

The runtime comparison uses the same 1280×900 desktop dev-app window, Browser runtime visual QA thread, Browser Core Test Lab fixture, Team members route, localhost URL, Browser panel width, and device scale. Before is the exact merge base fa9a15ba; after is the exact PR head dd66e0893.

Browser action: before

State: the localhost Team members page is open as a complete Browser page. The full http://127.0.0.1:43173/ URL is visible, and the merge base has no plugin Browser action.

Before: complete localhost Team members page without a plugin Browser action

Browser action: after

Action: select Inspect this Browser page in the Browser toolbar.

Result: the same complete localhost page remains visible while a compact, non-modal result reports the live heading, control counts, input count, and navigation epoch for that exact tab revision.

After: complete localhost page with the settled non-modal Browser runtime result

Overflow interaction sequence

These two captures are both from exact head dd66e0893 in the same 1280×900 window. The Browser panel is deliberately constrained so only actions 2–4 fit inline.

Step 1 — trigger in context: the overflow trigger appears immediately before the host external-link action, which remains rightmost. No hover-only state is required.

Overflow trigger in the constrained Browser toolbar

Step 2 — open menu: focus the overflow trigger and press Enter. The menu opens with ordered actions 5–8 while the trigger and external-link action remain visible in context.

Keyboard-open Browser action overflow menu

Validation

  • exact desktop Browser action returned Team members, 3 buttons, 1 input, and navigation epoch 1
  • complete localhost page, tab title, and full URL were visually verified before and after the action
  • constrained toolbar ordering, keyboard Enter activation, Escape dismissal, focus behavior, and five repeated open/close cycles passed
  • exact agent bridge passed tab discovery, interactive snapshot, isolated custom script, type, click, scroll, and screenshot; readback was { filter: "Dara", activity: "Invite member clicked through the Browser runtime.", scrollY: 420 } with a 778×1336 capture
  • focused app, desktop, server, desktop-contract, plugin SDK, template, and compatibility suites passed
  • affected-package typecheck passed 10/10 tasks; git diff --check passed

Supersedes the Browser runtime portion of #1642 and #1643. Does not merge or ship the Browser Context consumer.

BB-Thread-ID: thr_p9qmfkggwm

AGENT GENERATED: by GPT-5.6-Sol

brsbl added 2 commits August 19, 2026 12:27
…me-final-thr_p9qmfkggwm

# Conflicts:
#	apps/app/src/components/secondary-panel/BrowserTabContent.tsx
#	apps/app/src/lib/plugin-slots.ts
#	apps/desktop/src/desktop-browser-ipc.ts
#	apps/desktop/src/desktop-browser-view.ts
#	apps/desktop/src/preload.ts
#	apps/desktop/test/desktop-browser-main-ipc.test.ts
#	apps/desktop/test/preload-browser-api.test.ts
#	apps/server/src/ws/client-protocol.ts
#	apps/server/src/ws/hub.ts
#	packages/domain/src/change-kinds.ts
#	packages/templates/src/generated/plugin-sdk-dts.generated.ts
@brsbl
brsbl marked this pull request as ready for review August 19, 2026 19:41
@bb-slop-cop

bb-slop-cop Bot commented Aug 19, 2026

Copy link
Copy Markdown

🚨 SLOP COP 🚨 · review

I am SlopCop. I started parallel security, code quality, architecture, and performance reviews. I will also test the Browser workflow where possible. I will post one final review after I check the results.

Comment thread apps/app/src/lib/browser-control-client.ts
Comment thread apps/app/src/lib/browser-control-client.ts
Comment thread apps/desktop/src/desktop-browser-view.ts Outdated
Comment thread apps/desktop/src/desktop-browser-page-runtime.ts Outdated
Comment thread apps/app/src/lib/browser-control-client.ts
Comment thread apps/server/src/ws/client-protocol.ts

@bb-slop-cop bb-slop-cop Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚨 SLOP COP 🚨 · review

Plain English summary: This PR lets approved tools control one exact Browser tab. They can inspect, click, type, scroll, navigate, run bounded code, and capture images.

The server, web app, and desktop shell match each request to one page revision. Plugins also gain a compact Browser toolbar action.

I found six issues:

  • High: One script timeout can terminate unrelated scripts and page code in the same tab.
  • High: Normal navigation leaves a short window where a script can run against the page that the user is leaving.
  • Medium: A large DOM snapshot can stop the page renderer and delay cancellation.
  • Medium: Screenshot cancellation does not stop the native capture task.
  • Medium: Backspace, Delete, and arrow-key actions report success without native browser behavior.
  • Medium: Deep recursive response data can escape the invalid-message guard and leave the WebSocket open.

I found no earlier Browser runtime that this change should reuse. A tab operation manager can centralize navigation state, cancellation, and hard limits.

Validation passed for all six affected package typechecks. The full desktop suite passed 231 tests, and the two separate test runs passed.

The native end-to-end test loaded Example Domain. The page runtime returned its heading, and image capture returned the same navigation epoch.

I left line comments with the trigger and a testable fix for each issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant