mac-use

A self-learning macOS automation agent. Drive your Mac with natural language — it sees the screen via the accessibility tree + screenshots, acts through a native Swift sidecar, learns reusable skills from successful tasks (self-healing on replay), and remembers your preferences across sessions.

One engine: every role runs on a single pi-coding-agent session, so the whole run is one persisted timeline. Built on pi-mono (pi-ai / pi-agent-core / pi-coding-agent / pi-tui), runs on Bun.

Roles (Tab to switch)

A role is just a tool profile + confirm policy applied to the one shared session — switching never starts a new agent, and the conversation stays a single continuous timeline.

• Manager (default) — all tools (Mac control + coding). Generalist; plans and does the task.
• Operator — Mac only: the 10 sidecar tools (click/type/key/screenshot/read_ui/scroll/…). Built-in coding tools off.
• Coder — code only: read/edit files and run shell commands. Mac tools off.
• Terminator — all tools, fully autonomous (skips confirmation).

The 2 memory tools (memory_add / memory_search) stay available in every role.

A confirmation gate blocks dangerous shell commands (rm -rf, sudo, git push, curl | sh, …) for every role except Terminator.

Memory

Long-term memory lives in ~/.mac-use/memory/:

• memory.db — SQLite source of truth (via Bun's built-in bun:sqlite, zero dependency).
• MEMORY.md — a generated, human-readable view you can open and inspect.

The agent sees only memory_add / memory_search. The background pipeline runs at the task boundary: relevant memories are injected into the prompt at task start, and durable facts / preferences are extracted (via the LLM) and stored after the task finishes.

Setup

Quick start (one-shot)

./install.sh

The script is idempotent — safe to re-run. It detects / installs Xcode Command Line Tools (for swiftc), installs Bun if missing, runs bun install, builds the Swift sidecar into bin/sidecar, and writes a .env template. Two things it can't script are called out at the end: filling in your real API key in .env, and granting Accessibility + Screen Recording to the host app in System Settings (then restart the host app).

Manual steps

bun install
bun run sidecar:build      # compiles sidecar/main.swift -> bin/sidecar
bun link                   # registers the global `mac-use` command (optional)

Note on pi-mono dependencies

The four @earendil-works/* packages are pulled from the npm registry, not a local monorepo. If you ever see a Cannot find module "@earendil-works/..." error after pulling, just run bun install again — that's almost always enough.

^0.75.5 is pinned to the API surface that this codebase was developed and tested against. Newer 0.7x versions (latest is 0.79.0) may have breaking changes; bump deliberately after testing, not by accident.

Create .env:

MAC_USE_PROVIDER=deepseek
MAC_USE_MODEL=deepseek-v4-flash
DEES_API_KEY=sk-...

Any pi-ai provider works — change .env, code stays the same. .env is read from the install dir or ~/.mac-use/.env, so the global command works from any directory.

macOS permissions: grant Accessibility and Screen Recording to the host app (Terminal / Claude).

Run

mac-use                # global command — 4-role TUI (default Manager), new persisted session
mac-use --continue     # resume the most recent session and keep going
bun run src/tui.ts     # same TUI without the global command (or `bun start`)
bun run src/cli.ts     # plain REPL, no TUI (also supports --continue)

Inside the TUI: type a task, Tab cycles roles, dangerous shell steps prompt y/n. Commands: /help, /agent <name>, /skills, /replay <name>, /session, /sessions, exit.

Sessions persist to ~/.mac-use/sessions/<timestamp>_<id>.jsonl.

Layout

sidecar/main.swift   native layer (10 commands), built to bin/sidecar
src/config.ts        .env loading + pi-ai provider abstraction
src/sidecar.ts       JSON-over-stdio bridge to the sidecar
src/tools.ts         10 Mac-control (sidecar) tools
src/memory.ts        SQLite memory + 2 tools + extraction + MEMORY.md export
src/roles.ts         role registry (tool profile + confirm policy) + state
src/confirm.ts       confirmation gate + dangerous-command matcher
src/session.ts       the single pi-coding-agent session (buildSession)
src/home.ts          ~/.mac-use paths (sessions, memory)
src/skills.ts        SKILL.md storage
src/recorder.ts      learning (auto-record skills after success)
src/healer.ts        healing (replay + auto-fix a skill)
src/tui.ts           pi-tui terminal interface
src/cli.ts           plain REPL
src/bin.ts           global `mac-use` entry
src/test-*.ts        tests

See progress.md for the full build log (stages 0–11).