Skip to content

feat: add optional Cua Driver computer backend - #65

Open
f-trycua wants to merge 2 commits into
CopilotKit:mainfrom
f-trycua:feat/cua-driver-computer-use
Open

feat: add optional Cua Driver computer backend#65
f-trycua wants to merge 2 commits into
CopilotKit:mainfrom
f-trycua:feat/cua-driver-computer-use

Conversation

@f-trycua

Copy link
Copy Markdown

What this changes

Adds an opt-in COMPUTER_BACKEND=cua-driver implementation behind the existing computer-service contract. Playwright remains the default.

The Cua Driver backend:

  • uses semantic browser snapshots and refs for read, click, type, and key actions;
  • preserves the existing HTTP, WebSocket, token, takeover, shell, and per-Bot APIs;
  • keeps isolated browser profiles under PROFILES_DIR/cua-driver, separate from Playwright profiles;
  • uses native window screenshots and native input for takeover actions; and
  • runs in both the standalone agent-computer image and the all-in-one production image.

The production packaging starts Xvfb and Openbox only for the Cua backend. It keeps Chromium and the display under pwuser, waits for graceful browser/profile shutdown before stopping X11, and builds both container paths in CI.

Where it runs

  • New state that outlives a request? No new distributed request state. The backend uses the existing per-Bot computer lifecycle. Durable browser data lives in the mounted profile root under PROFILES_DIR/cua-driver; live Cua sessions remain process-local, like the existing live Playwright browser.
  • What happens on the second replica? Multi-replica behavior is unchanged. Each agent-computer replica owns the browsers and profiles assigned to that deployment instance; the supervisor's per-Bot container model remains the isolation path.
  • Anything serialised? No new cross-process serialization or database writes. The manager deduplicates concurrent starts for one Bot inside its process, and each isolated named profile remains owned by that computer instance.
  • Anything fanned out to a browser? The existing gateway and WebSocket screen path are unchanged. Frames and takeover input still use the current computer-service endpoints.
  • New listener, port, or schedule? None. The computer service remains on port 4100 inside the deployment boundary, and the all-in-one image still exposes only port 3001.

Boundary and audit

  • Every acting call still goes through the existing gateway: resolve, decide, audit, then act.
  • Backend refusals and failures return through the existing computer endpoints, so the gateway retains the same audit path.
  • The backend does not add client-trusted identity or policy inputs. Direct computer-service requests still require COMPUTER_TOKEN.

Changelog

  • No changelog entry: deployment behavior is unchanged unless an operator explicitly sets COMPUTER_BACKEND=cua-driver. The new configuration and operational differences are documented in the README and deployment, configuration, and architecture guides.

Known limitations

  • Linux semantic clicks use a background-safe synthetic DOM event. Controls that require a trusted browser gesture may reject it.
  • Cua profiles do not apply OpenBot's per-Bot egress proxy settings yet; the API reports egress: null for this backend.
  • Ref-scoped key actions support Enter and printable characters. Arrows and editing keys use the native window route without a ref.

Proof

Candidate: 53e5a8868ee85ea5131ea89d6353c1685b849eab

  • Formatting and lint passed. Lint reported only pre-existing warnings.
  • Workspace and agent-computer typechecks passed.
  • Production build passed.
  • agent-computer: 99 tests passed.
  • Current-main database suite: 752 tests passed, with five deployment-smoke skips.
  • Both agent-computer/Dockerfile and the root production Dockerfile built successfully from the candidate.
  • Standalone Cua smoke: navigation, semantic read/snapshot refs, isolated profile, and native PNG screenshot passed.
  • Standalone default smoke: Playwright navigation passed, with no Xvfb or Openbox process started.
  • All-in-one Cua smoke: Xvfb, Openbox, Bun, and Chromium ran as pwuser; navigation and native screenshot passed.
  • Graceful shutdown smoke: docker stop --timeout 20 logged SIGTERM: closing the browser so its profile is flushed before X11 exited.
  • Claude Code Fable reviewed the implementation and final integration delta. All must-fix findings were closed, and its final verdict was READY TO SUBMIT.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant