Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 6 additions & 17 deletions .claude/skills/project-guardrails/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,26 +23,15 @@ one, stop and ask the user first. This is the fast "don't" list; AGENTS.md's

- **No streaming STT.** The AssemblyAI Sync API returns the full transcript in
one response. Overlay goes "Transcribing…" → full text.
- **No separate LLM cleanup pass.** Cleanup rides in the dictation request's
server-side `llm` block (`TranscriptionSteering`). No LLM Gateway client, no
- **No separate LLM cleanup pass.** Cleanup rides in the Sync STT request's
`config.prompt` (`TranscriptionPrompt`). No LLM Gateway client, no
`StylerProtocol`, no post-transcription styling stage.
Comment on lines +26 to 28
- **No local models / model downloads.** Transcription is a remote AssemblyAI
call. No on-device ASR/LLM, no model cache, no download UI.
- **Never send `config.prompt`.** It takes a _description of the audio_, not
instructions, and a custom value replaces the service's managed default
including its language steering. Vocabulary goes in `keyterms_prompt`, the
text before the cursor in `conversation_context`. An imperative like
"Transcribe speech into markdown." in `prompt` is a measured no-op — that is
exactly why those fields exist.
- **Never send anything about the frontmost app.** `AppKindPriming` recognized
the app's bundle ID as a kind (terminal, code editor, Slack, Obsidian) and
sent a formatting clause as `llm.instruction`; the whole path was removed.
The bundle ID isn't captured, and `llm` always goes out empty so the
service's default cleanup rewrite applies everywhere.
- Don't reintroduce a "remove filler words (um, uh, like)" directive — the STT
prompt doesn't act on it; it was deliberately dropped, and disfluency removal
is the LLM rewrite's job. Same for a language directive: pinning to English
hurt non-English speech, so language is left to the model's own detection.
- Don't reintroduce a "remove filler words (um, uh, like)" directive in the
prompt — `universal-3-5-pro` ignores it; it was deliberately dropped. Same for
a language directive: pinning the prompt to English hurt non-English speech, so
language is left to the model's own detection.
- **Injection is always a clipboard paste** (save → write → ⌘V → settle →
restore), degrading to "left it on the clipboard" when the target is lost. No
keystroke-by-keystroke typing path, no length threshold.
Expand Down
131 changes: 53 additions & 78 deletions AGENTS.md

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions App/Blurt/Blurt/Info.plist
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,9 @@
<key>CFBundlePackageType</key>
<string>APPL</string>
<key>CFBundleShortVersionString</key>
<string>0.1.36</string>
<string>0.1.34</string>
<key>CFBundleVersion</key>
<string>37</string>
<string>35</string>
<key>LSApplicationCategoryType</key>
<string>public.app-category.productivity</string>
<key>LSMinimumSystemVersion</key>
Expand Down
4 changes: 2 additions & 2 deletions App/Blurt/Blurt/Wizard/Steps/KeyTermsStepView.swift
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@ import SwiftUI

/// The "Key Terms" section of the Settings window: a free-text
/// area where the user lists comma-separated domain words (names, jargon, product
/// names). These ride on every request as its `keyterms_prompt` vocabulary list
/// (see `KeyTermsStore` / `TranscriptionSteering.build`), so the model favors those
/// names). These are folded into every transcription's prompt as spelling priming
/// (see `KeyTermsStore` / `TranscriptionPrompt.build`), so the model favors those
/// spellings. Optional — it never gates setup; an empty list just sends no terms.
struct KeyTermsStepView: View {
/// Stored in UserDefaults so multiple settings windows/readers see edits live.
Expand Down
4 changes: 2 additions & 2 deletions App/Blurt/project.yml
Original file line number Diff line number Diff line change
Expand Up @@ -91,8 +91,8 @@ targets:
properties:
CFBundleName: Blurt
CFBundleDisplayName: Blurt
CFBundleVersion: "37"
CFBundleShortVersionString: "0.1.36"
CFBundleVersion: "35"
CFBundleShortVersionString: "0.1.34"
LSMinimumSystemVersion: "15.0"
LSApplicationCategoryType: public.app-category.productivity
NSMicrophoneUsageDescription: Blurt uses your microphone to transcribe your dictation with AssemblyAI.
Expand Down
10 changes: 5 additions & 5 deletions BLURTENGINE.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ press() ──▶ MicCapture.start() release() ──▶ MicCapture.s
Key properties of the design, which your integration can rely on:

- **One request per utterance, no streaming.** The dictation API returns the complete transcript — and its LLM-rewritten form — in the response body: no upload step, no job polling, no incremental deltas, no second request for the cleanup. `TranscriberProtocol.transcribe` is a single `async throws -> String`. UIs should show a "transcribing…" state and then the whole result; there is nothing to stream.
- **Cleanup happens server-side.** The request's `llm` block asks the service for a cleanup rewrite (remove disfluencies, fix punctuation), applied to the verbatim transcript inside the same call; it always goes out empty, which selects the service's own default cleanup instruction. Recognition is primed separately by `conversation_context` and `keyterms_prompt` (built by `TranscriptionSteering` from the captured context); `config.prompt` is never sent, and neither is anything describing the destination app. The engine pastes `llm_response`, falling back to the verbatim `text` when the best-effort rewrite failed (`llm_error`) — a degradation, never a user-facing error. There is no client-side LLM pass, no styling stage, and deliberately no hook for one.
- **Cleanup happens server-side.** The request's empty `llm` block asks the service for its default cleanup rewrite (remove disfluencies, fix punctuation), applied to the verbatim transcript inside the same call; the per-utterance `config.prompt` (built by `TranscriptionPrompt` from the captured context) primes the _transcription_. The engine pastes `llm_response`, falling back to the verbatim `text` when the best-effort rewrite failed (`llm_error`) — a degradation, never a user-facing error. There is no client-side LLM pass, no styling stage, and deliberately no hook for one.
- **Latency is pre-paid where possible.** `press()` fires a detached `warmUp()` at the transcriber (pre-opening the HTTPS connection while the user speaks, ~170 ms saved cold) and kicks off the cross-process accessibility read of the focused field without awaiting it — the read is then consumed at transcribe time with a bounded wait (`DictationSession.contextWaitBudget`, 500 ms), so an unresponsive frontmost app costs the transcript its priming, never a multi-second stall — and never delays the recording indicator. On the way out, `release()` flips the phase to `.transcribing` _before_ reading the recorded audio back, so a host's stop cue fires at key-up rather than after the disk read.
- **A held trigger auto-releases.** `DictationSession` stops recording after `maxRecordingSeconds` (default `SyncSTTLimits.autoReleaseSeconds`, 115 s) so audio never exceeds what the endpoint accepts, and transcribes what it has. Clips shorter than `SyncSTTLimits.minPCMBytes` (~100 ms of audio — an accidental tap) are dropped as a silent no-op rather than sent to earn a 400.

Expand Down Expand Up @@ -152,15 +152,15 @@ The session calls `setTargetApp` at press time with the app that was frontmost w

Recognition quality comes from per-utterance priming, assembled automatically inside `press()` — hosts don't call these APIs directly, but should know what's collected:

- **`TranscriptionContext`** carries the frontmost app name, window title, focused-field label, the text before the caret, the selected text (which a paste will replace), and the user's key terms. It's captured via Accessibility at press time (skipped in secure fields), off the hot path — and consumed at transcribe time with a bounded wait (`DictationSession.contextWaitBudget`), so a hung read is abandoned rather than stalling the transcript. Only the prior text and key terms reach the request; the prior text and window title also steer the injector's paste separator, and nothing else is consumed.
- **`TranscriptionSteering.build(context:)`** renders that into the request's two customization fields: the prior-cursor text as the single `conversation_context` turn (clipped to 4096 chars keeping the tail, since the words nearest the cursor carry the continuity) and the user's key terms as `keyterms_prompt` (whole terms fitted to 2048 chars total). **`config.prompt` is never sent**: it takes a description of the audio rather than instructions, and a custom value replaces the service's managed default including its language steering, which is why the key terms ride in their own field. Nothing else is rendered — no app/field names, no selected text (the paste replaces it), and no standing annotation-suppression clause ("Transcribe without speaker labels, …" is part of the service's own default). Empty fields are omitted rather than sent as `[]`. Three further deliberate omissions: no language directive, no "remove filler words" clause (not something the STT prompt acts on — a no-op), and nothing identifying the destination app — an earlier `AppKindPriming` sent a bundle-ID-derived formatting clause as `llm.instruction`, and that whole path was removed. Don't reintroduce any of them.
- **`TranscriptionContext`** carries the frontmost app name, window title, focused-field label, the text before the caret, the selected text (which a paste will replace), and the user's key terms. It's captured via Accessibility at press time (skipped in secure fields), off the hot path — and consumed at transcribe time with a bounded wait (`DictationSession.contextWaitBudget`), so a hung read is abandoned rather than stalling the transcript.
- **`TranscriptionPrompt.build(context:)`** renders that into the dictation request's `config.prompt` (transcription steering only — the cleanup rewrite is the separate `llm` block), opening with the fixed `baseInstruction` ("Transcribe without speaker labels, audio event descriptions, or emotion markers.") and staying under the API's 4096-character cap. An empty context yields `nil`, which omits the field so the server applies its own default. Two deliberate omissions, both regression-tested: no language directive (pinning to English hurt non-English speech) and no "remove filler words" clause (not in the model's trained instruction set — a no-op). Don't reintroduce either.
- **`KeyTermsStore`** persists the user's domain vocabulary (names, jargon) in `UserDefaults`; `DictationSession` re-reads it at every press via its `keyTermsProvider` closure, so Settings edits apply to the next utterance without rebuilding the session. Pass your own provider to source terms from elsewhere.

For key storage, compose against **`APIKeyGateway`** — the injectable `current` / `save(_:)` / `hasKey` seam over the key store. `ProductionAPIKeyStore` forwards to the Keychain-backed `APIKeyStore`; `InMemoryAPIKeyStore` is a ready-made in-memory conformance for tests and harnesses (Blurt's XCUITest runs use it so the real Keychain item is never touched, and its `hasKey` backs the session's `readinessCheck`). For a settings UI, **`APIKeySubmission`** wraps the gateway with the validate-then-save flow (`submit(_:)` → valid / invalid / unreachable / saveFailed, via `APIKeyValidator`): it saves only a key AssemblyAI actively accepts, so an unverified key never persists. Two projections keep the surrounding UI out of your views: `Outcome.failureReport` classifies a failure as `.inline(message:)` (recoverable — show it beside the field) or `.alert(title:message:)` (a Keychain fault retyping can't fix), and **`APIKeyDisplay.resolve(key:)`** renders the stored key for an account row — masked tail, status and VoiceOver wording, and the connect-vs-rotate control titles. The mask reveals only the last `revealedTailLength` characters and, below `minimumLengthToMask`, none at all, so a short key can't be shown whole.

Setup gating has a projection too: **`SetupReadiness.isReady(permissions:hasAPIKey:)`** is the "fully configured" rule (deliberately excluding the trigger key, which has a default), `SetupReadiness.pollInterval(isReady:)` is the permission-poll cadence (brisk during setup, coasting once ready), and `PermissionStatus.lostGrant(since:)` detects a permission revoked out from under a configured app.

Each completed dictation is appended to **`DictationLog`** (a local JSONL history at `~/Library/Logs/Blurt/dictations.jsonl` — `DictationLog.defaultURL`, or `defaultDisplayPath` for the home-abbreviated form to show in UI) — each entry is the transcript plus the exact steering fields sent (`conversation_context`, `keyterms_prompt`), never the context that wasn't sent — but only while developer mode is switched on. **`DeveloperModeStore`** persists that opt-in in `UserDefaults` (`BlurtDeveloperMode`, off by default); with it off, nothing is written to disk. Blurt surfaces the switch (and the log path) in the Settings window's Developer section.
Each completed dictation is appended to **`DictationLog`** (a local JSONL history at `~/Library/Logs/Blurt/dictations.jsonl` — `DictationLog.defaultURL`, or `defaultDisplayPath` for the home-abbreviated form to show in UI) with its context snapshot — but only while developer mode is switched on. **`DeveloperModeStore`** persists that opt-in in `UserDefaults` (`BlurtDeveloperMode`, off by default); with it off, nothing is written to disk. Blurt surfaces the switch (and the log path) in the Settings window's Developer section.

## Hotkey building blocks

Expand Down Expand Up @@ -195,7 +195,7 @@ Run `swift test` for the engine suites (`--filter DictationSessionTests` for one
Each of these was tried the other way and reverted; the longer stories are in [AGENTS.md](AGENTS.md) and the source comments:

- **No external SPM dependencies in the engine.** Foundation/Security/AVFoundation only.
- **No streaming STT, no local models, no client-side LLM cleanup pass.** One dictation request per utterance is the architecture; the cleanup rewrite is server-side (the request's `llm` block), and request steering belongs in `TranscriptionSteering`.
- **No streaming STT, no local models, no client-side LLM cleanup pass.** One dictation request per utterance is the architecture; the cleanup rewrite is server-side (the request's `llm` block), and transcription steering belongs in `TranscriptionPrompt`.
- **No `AVAudioEngine`/`installTap` capture path.** Fresh `AVAudioRecorder` per session, resolved at record time.
- **Paste is always clipboard-based** (save → write → ⌘V → settle → restore), with the copied-to-clipboard degradation for lost targets.
- **No English-pinning or filler-word clauses in the prompt.**
Expand Down
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,14 +166,14 @@ Sources/BlurtEngine/ Swift 6 package owning the pipeline — no external dep
Audio/ MicCapture: fresh AVAudioRecorder per session, 16 kHz mono PCM,
live level meter; DX7/Juno-106 sound packs
STT/ AssemblyAITranscriber: one POST to dictation.assemblyai.com/transcribe
(STT + LLM rewrite) + TranscriptionSteering context/keyterms
(STT + LLM rewrite) + TranscriptionPrompt contextual priming
Pipeline/ DictationSession actor: press/release/cancel commands, phase
stream, auto-release before the API's recording cap
Hotkey/ DictationKeyGate/Router: pure, unit-tested state machine for the
lone-modifier trigger (tap vs hold vs combo)
Injection/ KeyInjector: save clipboard → paste via synthesized ⌘V → restore
FocusCapture/ Accessibility reads of the focused app/window/field feeding the
request's steering fields, the log, and paste separators
FocusCapture/ Accessibility reads of the focused app/window/field that prime
the transcription prompt
Config/, Update/ Keychain API-key store, key terms, download-only release check

App/Blurt/ AppKit/SwiftUI shell (Xcode project generated by XcodeGen)
Expand Down
8 changes: 4 additions & 4 deletions Sources/BlurtEngine/Config/KeyTermsStore.swift
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
import Foundation

/// Storage for the user's dictation "key terms" — a comma-separated list of
/// domain words (names, jargon, product names) sent as the dictation request's
/// `keyterms_prompt` vocabulary list, so the model is more likely to spell
/// them correctly (see `TranscriptionSteering.build`).
/// domain words (names, jargon, product names) that get folded into the dictation
/// request `prompt` as vocabulary priming, so the model is more likely to spell
/// them correctly (see `TranscriptionPrompt.build`).
///
/// Unlike the API key these aren't secret, so they live in `UserDefaults` rather
/// than the Keychain. The transcription pipeline reads the parsed list via
Expand Down Expand Up @@ -37,7 +37,7 @@ public enum KeyTermsStore {
}

/// Pure parse of a comma-separated string into a clean term list. Exposed so
/// `TranscriptionSteering` and tests can reuse the exact same rules.
/// `TranscriptionPrompt` and tests can reuse the exact same rules.
public static func parse(_ text: String?) -> [String] {
guard let text else { return [] }
var seen = Set<String>()
Expand Down
24 changes: 13 additions & 11 deletions Sources/BlurtEngine/FocusCapture/FocusCapture.swift
Original file line number Diff line number Diff line change
Expand Up @@ -10,25 +10,27 @@ enum FocusCapture {
@MainActor
static func captureFrontmost() -> CapturedFocus? {
guard let app = NSWorkspace.shared.frontmostApplication else { return nil }
return CapturedFocus(pid: app.processIdentifier, processName: app.localizedName)
return CapturedFocus(
pid: app.processIdentifier,
processName: app.localizedName
)
}

static func runningApp(for captured: CapturedFocus) -> NSRunningApplication? {
NSRunningApplication(processIdentifier: captured.pid)
}

/// Accessibility-derived focus context read from the system-wide focused UI
/// element at dictation start (see `TranscriptionContext` for what each
/// signal feeds). Every field is best-effort: any signal that can't be read
/// is `nil`, and a fully-empty result simply means less context, never an
/// error.
/// Accessibility-derived priming read from the system-wide focused UI element
/// at dictation start (see `TranscriptionContext`). Every field is
/// best-effort: any signal that can't be read is `nil`, and a fully-empty
/// result simply means less context, never an error.
struct FocusedFieldContext: Sendable {
/// Text immediately preceding the insertion point.
/// Text immediately preceding the insertion point ("prior chunk context").
let priorText: String?
/// The text currently selected in the focused field — the dictation will
/// replace it.
/// replace it, so it primes the model on what the utterance is about.
let selectedText: String?
/// The focused window's title (in a code editor it names the open file).
/// The focused window's title a dense topic hint.
let windowTitle: String?
/// A short label for the focused field ("To", "Search", "Message").
let fieldLabel: String?
Expand All @@ -47,8 +49,8 @@ enum FocusCapture {
///
/// Secure text fields (password inputs) are detected by role **or** subrole and
/// never have their contents read, so a typed password — selected or not — can't
/// leak into the dictation log or the injector. The check fails closed: an
/// unreadable role is treated as secure, since it can't be shown not to be.
/// leak into the STT prompt. The check fails closed: an unreadable role is
/// treated as secure, since it can't be shown not to be.
///
/// Deliberately `nonisolated`: each read below is a synchronous cross-process
/// IPC round trip into the frontmost app, and an unresponsive app blocks the
Expand Down
Loading