From 39ad6de5a0afc4e66f03420646bfe3e79992414f Mon Sep 17 00:00:00 2001 From: Alex Kroman Date: Wed, 29 Jul 2026 17:12:24 -0400 Subject: [PATCH] Revert the app to v0.1.34 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rolls the tree back to e08a9ce (v0.1.34) by reverting everything merged since, newest first: - #101 chore: bump to v0.1.36 - #100 feat(stt): stop sending anything about the running app to the LLM - #99 chore: bump to v0.1.35 - #98 Add app-kind guidance to transcription prompts #98 and #100 partly cancelled — #100 removed the app-kind clause #98 added — but #98 carried much more than that, and all of it is undone here. Restored as a result: - `config.prompt` is sent again, built by `TranscriptionPrompt` from the focused app, window, and field plus the user's key terms and the text around the cursor. - `TranscriptionSteering` is gone, and with it the split into `conversation_context`, `keyterms_prompt`, and `llm.instruction`. Key terms are back to a `Keywords: a, b, c.` clause inside the prompt; prior-cursor text is contextual priming in the prompt rather than a conversation turn. - `DictationLog` records the assembled prompt again, alongside the frontmost app's bundle ID. - `AppKindPriming` returns with #98 and is not re-removed: #100's removal was reverted too, so the bundle-ID → app-kind table is live again. Docs (AGENTS.md, BLURTENGINE.md, README.md, the project-guardrails skill) go back to describing that design. `CFBundleShortVersionString` is 0.1.34. `git diff e08a9ce` is empty, so this is an exact rollback rather than an approximation. Verified: swift test (424 tests, 67 suites) and the Debug app build both pass. scripts/check.sh as a whole does not pass on this machine — it aborts in the XCUITest suite with the "harness window was not presented" failures that predate all of this work. Co-Authored-By: Claude Opus 5 (1M context) --- .claude/skills/project-guardrails/SKILL.md | 23 +-- AGENTS.md | 131 +++++++--------- App/Blurt/Blurt/Info.plist | 4 +- .../Blurt/Wizard/Steps/KeyTermsStepView.swift | 4 +- App/Blurt/project.yml | 4 +- BLURTENGINE.md | 10 +- README.md | 6 +- .../BlurtEngine/Config/KeyTermsStore.swift | 8 +- .../FocusCapture/FocusCapture.swift | 24 +-- .../BlurtEngine/Pipeline/DictationLog.swift | 63 +++----- .../Pipeline/DictationSession+Pipeline.swift | 3 +- .../STT/AssemblyAITranscriber.swift | 72 +++------ .../STT/TranscriptionContext.swift | 49 +++--- .../BlurtEngine/STT/TranscriptionPrompt.swift | 130 ++++++++++++++++ .../STT/TranscriptionSteering.swift | 109 ------------- .../AssemblyAITranscriberTests.swift | 66 +++----- .../BlurtEngineTests/DictationLogTests.swift | 115 +++++--------- .../TranscriptionContextTests.swift | 43 ++--- .../TranscriptionPromptTests.swift | 147 ++++++++++++++++++ .../TranscriptionSteeringTests.swift | 138 ---------------- 20 files changed, 506 insertions(+), 643 deletions(-) create mode 100644 Sources/BlurtEngine/STT/TranscriptionPrompt.swift delete mode 100644 Sources/BlurtEngine/STT/TranscriptionSteering.swift create mode 100644 Tests/BlurtEngineTests/TranscriptionPromptTests.swift delete mode 100644 Tests/BlurtEngineTests/TranscriptionSteeringTests.swift diff --git a/.claude/skills/project-guardrails/SKILL.md b/.claude/skills/project-guardrails/SKILL.md index 8cecfd9..8dca901 100644 --- a/.claude/skills/project-guardrails/SKILL.md +++ b/.claude/skills/project-guardrails/SKILL.md @@ -23,26 +23,15 @@ one, stop and ask the user first. This is the fast "don't" list; AGENTS.md's - **No streaming STT.** The AssemblyAI Sync API returns the full transcript in one response. Overlay goes "Transcribing…" → full text. -- **No separate LLM cleanup pass.** Cleanup rides in the dictation request's - server-side `llm` block (`TranscriptionSteering`). No LLM Gateway client, no +- **No separate LLM cleanup pass.** Cleanup rides in the Sync STT request's + `config.prompt` (`TranscriptionPrompt`). No LLM Gateway client, no `StylerProtocol`, no post-transcription styling stage. - **No local models / model downloads.** Transcription is a remote AssemblyAI call. No on-device ASR/LLM, no model cache, no download UI. -- **Never send `config.prompt`.** It takes a _description of the audio_, not - instructions, and a custom value replaces the service's managed default - including its language steering. Vocabulary goes in `keyterms_prompt`, the - text before the cursor in `conversation_context`. An imperative like - "Transcribe speech into markdown." in `prompt` is a measured no-op — that is - exactly why those fields exist. -- **Never send anything about the frontmost app.** `AppKindPriming` recognized - the app's bundle ID as a kind (terminal, code editor, Slack, Obsidian) and - sent a formatting clause as `llm.instruction`; the whole path was removed. - The bundle ID isn't captured, and `llm` always goes out empty so the - service's default cleanup rewrite applies everywhere. -- Don't reintroduce a "remove filler words (um, uh, like)" directive — the STT - prompt doesn't act on it; it was deliberately dropped, and disfluency removal - is the LLM rewrite's job. Same for a language directive: pinning to English - hurt non-English speech, so language is left to the model's own detection. +- Don't reintroduce a "remove filler words (um, uh, like)" directive in the + prompt — `universal-3-5-pro` ignores it; it was deliberately dropped. Same for + a language directive: pinning the prompt to English hurt non-English speech, so + language is left to the model's own detection. - **Injection is always a clipboard paste** (save → write → ⌘V → settle → restore), degrading to "left it on the clipboard" when the target is lost. No keystroke-by-keystroke typing path, no length threshold. diff --git a/AGENTS.md b/AGENTS.md index 93672bf..dc64a03 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -7,10 +7,11 @@ the Claude-Code-specific tooling under `.claude/` (hooks, skills, subagents). Blurt is a macOS dictation app powered by [AssemblyAI](https://www.assemblyai.com). Tap or hold a trigger key, speak, and polished text is pasted into the focused app. Transcription is **one remote -AssemblyAI dictation API call**: per-utterance steering rides along with the request — the text before -the cursor as `conversation_context` and the user's key terms as `keyterms_prompt` — and the same -request asks the service for its server-side LLM cleanup rewrite (`config.llm`), so the text that -comes back is already polished. The user supplies their own API key. +AssemblyAI dictation API call**: a per-utterance `prompt` (a transcription directive plus contextual +priming built from the focused app/window/field and the user's key terms) rides along with the +request, and the same request asks the service for its server-side LLM cleanup rewrite +(`config.llm`), so the text that comes back is already polished. The user supplies their own API +key. Four reflexes before you touch anything: @@ -52,7 +53,7 @@ Sources/BlurtEngine/ the engine (dependency-free Swift package) Injection/ KeyInjector (clipboard paste), SystemClipboard Permissions/ PermissionsChecker (mic + Accessibility) Pipeline/ DictationSession (actor) + phases, UI projections, geometry, log - STT/ AssemblyAITranscriber, TranscriptionSteering/Context, SyncSTTLimits + STT/ AssemblyAITranscriber, TranscriptionPrompt/Context, SyncSTTLimits Update/ UpdateChecker (download-only) + the launch-check policy App/Blurt/ project.yml XcodeGen source of truth — Blurt.xcodeproj is GENERATED @@ -158,27 +159,23 @@ In Claude Code on the web, a `SessionStart` hook installs the portable linters a Each was tried the other way and reverted. If a task seems to require one, stop and ask first. (`.claude/skills/project-guardrails` is the compressed version of this list.) -| Don't | Because | -| --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Add an external SPM dependency to the engine | Dependency-free by rule (biggest supply-chain risk); a `check.sh` guard fails on `.package(` in `Package.swift` or a `url:`/`github:` package in `project.yml`. Extend `BlurtEngine` instead. | -| Use `AVAudioEngine` / `installTap` for capture | A long-lived engine bound its input graph to one device and went stale on a mic↔built-in switch — `-10868` (`kAudioUnitErr_FormatNotSupported`) or all-zero buffers. `MicCapture` uses a fresh `AVAudioRecorder` per session. | -| Add streaming STT | The dictation API returns the full (already rewritten) text in one response; the overlay shows "Transcribing…" then the full text. | -| Add a client-side LLM cleanup pass | Cleanup is the dictation API's server-side rewrite, requested by the `llm` block on the same `/transcribe` call. No LLM Gateway client, no `StylerProtocol`, no styling stage, no second request — request steering belongs in `TranscriptionSteering`. | -| Add local models or model downloads | Transcription is a remote AssemblyAI call: no on-device ASR/LLM, no model cache, no download UI. | -| Send `config.prompt` at all | The field takes a _description of the audio_, not instructions, and a custom value replaces the service's managed default **including its language steering**. Vocabulary goes in `keyterms_prompt`, preceding text in `conversation_context`. | -| Pin the prompt to English | Hurt non-English transcription; language is left to the model's own detection — and a custom prompt is what drops the managed default's language steering in the first place. | -| Put formatting instructions in `config.prompt` | Reshaping output is not something the STT prompt acts on, so _"Transcribe speech into markdown."_ was a measured no-op. | -| Send anything about the frontmost app to the service | Removed on purpose. `AppKindPriming` recognized the app's bundle ID as a kind and sent a formatting clause as `llm.instruction`; the bundle ID is no longer captured, and `llm` always goes out empty so the service's default cleanup applies everywhere. | -| Pack key terms into the prompt as `Keywords: a, b, c.` | The API documents `keyterms_prompt` for exactly this, and warns against packing keyword lists into the prompt. | -| Add a "remove filler words (um, uh, like)" clause | Not something the STT prompt acts on — a no-op, deliberately dropped; disfluency removal is the server-side LLM rewrite's job. | -| Add a keystroke-typing paste path or a length threshold | Injection is **always** clipboard paste (save → write → ⌘V → settle → restore), with the copied-to-clipboard degradation when the target is lost. | -| Add `LSUIElement` or a menu-bar-**only** mode | Blurt is a Dock app first. The `MenuBarExtra` status item is convenience layered on the Dock icon; the notch can hide a status item, so nothing may depend on it. A menu-bar-only variant was reverted twice. | -| Add a `KeyboardShortcuts` package or a key+modifier chord | The trigger is a single lone modifier, home-grown (`CGEventTap` + `DictationKeyGate`), and swallows nothing. | -| Add a self-replacing install or background auto-updater | Updates are download-only; `mxcl/AppUpdater` and its in-place updater were removed. The once-a-day launch _check_ (`AutomaticUpdateCheck`) is fine; installing for the user, or polling, is not. Extend `UpdateCheckModel`. | -| Hand-edit `Blurt.xcodeproj/project.pbxproj` | Generated from `project.yml`; `check.sh`'s drift check fails on any manual edit (a Claude PreToolUse hook also blocks it). | -| Redirect the post-build install away from `/Applications` | TCC won't register apps in DerivedData/`/tmp`, so permission toggles never appear. | -| Touch the real Keychain in tests | `APIKeyStore` is the production item — a test that writes it triggers Keychain prompts and corrupts the real item's ACL. Use an isolated service (see `KeychainStoreTests`) or `InMemoryAPIKeyStore`. | -| Add backwards-compat shims for removed types | Deleted types stay deleted — no deprecated re-exports. | +| Don't | Because | +| --------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Add an external SPM dependency to the engine | Dependency-free by rule (biggest supply-chain risk); a `check.sh` guard fails on `.package(` in `Package.swift` or a `url:`/`github:` package in `project.yml`. Extend `BlurtEngine` instead. | +| Use `AVAudioEngine` / `installTap` for capture | A long-lived engine bound its input graph to one device and went stale on a mic↔built-in switch — `-10868` (`kAudioUnitErr_FormatNotSupported`) or all-zero buffers. `MicCapture` uses a fresh `AVAudioRecorder` per session. | +| Add streaming STT | The dictation API returns the full (already rewritten) text in one response; the overlay shows "Transcribing…" then the full text. | +| Add a client-side LLM cleanup pass | Cleanup is the dictation API's server-side rewrite, requested by the `llm` block on the same `/transcribe` call. No LLM Gateway client, no `StylerProtocol`, no styling stage, no second request — transcription steering belongs in `TranscriptionPrompt`. | +| Add local models or model downloads | Transcription is a remote AssemblyAI call: no on-device ASR/LLM, no model cache, no download UI. | +| Pin the prompt to English | Hurt non-English transcription; language is left to the model's own detection. | +| Add a "remove filler words (um, uh, like)" clause | Not in the STT model's trained instruction set — a no-op, deliberately dropped; disfluency removal is the server-side LLM rewrite's job. | +| Add a keystroke-typing paste path or a length threshold | Injection is **always** clipboard paste (save → write → ⌘V → settle → restore), with the copied-to-clipboard degradation when the target is lost. | +| Add `LSUIElement` or a menu-bar-**only** mode | Blurt is a Dock app first. The `MenuBarExtra` status item is convenience layered on the Dock icon; the notch can hide a status item, so nothing may depend on it. A menu-bar-only variant was reverted twice. | +| Add a `KeyboardShortcuts` package or a key+modifier chord | The trigger is a single lone modifier, home-grown (`CGEventTap` + `DictationKeyGate`), and swallows nothing. | +| Add a self-replacing install or background auto-updater | Updates are download-only; `mxcl/AppUpdater` and its in-place updater were removed. The once-a-day launch _check_ (`AutomaticUpdateCheck`) is fine; installing for the user, or polling, is not. Extend `UpdateCheckModel`. | +| Hand-edit `Blurt.xcodeproj/project.pbxproj` | Generated from `project.yml`; `check.sh`'s drift check fails on any manual edit (a Claude PreToolUse hook also blocks it). | +| Redirect the post-build install away from `/Applications` | TCC won't register apps in DerivedData/`/tmp`, so permission toggles never appear. | +| Touch the real Keychain in tests | `APIKeyStore` is the production item — a test that writes it triggers Keychain prompts and corrupts the real item's ACL. Use an isolated service (see `KeychainStoreTests`) or `InMemoryAPIKeyStore`. | +| Add backwards-compat shims for removed types | Deleted types stay deleted — no deprecated re-exports. | Release-side invariants (hardened runtime and a secure timestamp on every nested mach-o and embedded framework, or notarization rejects the build; roll-forward-only for a bad release) live in @@ -245,13 +242,11 @@ the seam they inject. Implements `TranscriberProtocol` against AssemblyAI's **dictation** API: a single `POST https://dictation.assemblyai.com/transcribe` with the captured audio as a raw S16LE PCM blob -in the `audio` multipart part plus a JSON `config` part (`sample_rate`, `channels`, and the steering -fields built per utterance by `TranscriptionSteering`: `conversation_context`, `keyterms_prompt`, and -the `llm` block). No model header — the service pins the STT model server-side. **`prompt` is never -sent** — see [Transcription steering](#transcription-steering). The `llm` block asks the service to -run a cleanup rewrite (remove disfluencies, fix punctuation) over the verbatim transcript, all inside -the same request; it always goes out empty, which selects the service's own default cleanup -instruction. The response carries both `text` (verbatim) and +in the `audio` multipart part plus a JSON `config` part (`sample_rate`, `channels`, `prompt`, and an +empty `llm` block). No model header — the service pins the STT model server-side. The `prompt` +(built per utterance by `TranscriptionPrompt`) steers _transcription_; the `llm` block asks the +service to run its default LLM cleanup rewrite (remove disfluencies, fix punctuation) over the +verbatim transcript, all inside the same request. The response carries both `text` (verbatim) and `llm_response` (the rewrite); the transcriber returns the rewrite and falls back to `text` when `llm_response` is null — the rewrite is best-effort (5 s server-side budget), so a rewrite failure (`llm_error`) is a logged degradation, never a user-facing error. @@ -378,49 +373,30 @@ the `@AppStorage(TriggerKeyStore.defaultsKey)` + `TriggerKey.fromPersisted` pair restating that pairing per view. The unset default belongs to `fromPersisted` (an absent keycode maps to right ⌘), so views must not re-declare `TriggerKey.rightCommand.rawValue` themselves. -## Transcription steering - -`Sources/BlurtEngine/STT/TranscriptionSteering.swift` renders the captured context into the two -request-customization fields, each with one job. It's unit-tested in -`Tests/BlurtEngineTests/TranscriptionSteeringTests.swift`. - -| Field | Carries | Cap | -| ---------------------- | ----------------------------------- | ----------------------- | -| `conversation_context` | prior-cursor text, as a single turn | 4096 chars, clip head | -| `keyterms_prompt` | the user's key terms, verbatim | 2048 chars, whole terms | - -**`config.prompt` is never sent, and that is the whole point of this design.** The field takes a -_description of the audio_ ("Cardiology consultation about chest pain symptoms."), not instructions — -transcription behavior is optimized out of the box — so an imperative like -_"Transcribe speech into markdown."_ was aimed at a field that doesn't act on instructions and was a -no-op. Vocabulary rides in `keyterms_prompt` rather than being packed into the prompt as a -`Keywords: a, b, c.` clause. Sending no prompt also keeps the service's managed default, which a -custom prompt replaces wholesale — **including its language steering**, which is the mechanism behind -the older finding that pinning the prompt to English hurt non-English speech. - -**Nothing describing the destination app is sent.** An earlier design recognized the frontmost app's -bundle ID as a _kind_ (terminal, code editor, Slack, Obsidian) and sent a matching formatting clause -as `llm.instruction`; that whole path — `AppKindPriming`, `TranscriptionContext.bundleID`, the -`CapturedFocus` bundle-ID capture, and the log's `llm_instruction` column — was removed. The `llm` -block now always goes out empty, so the service's own default cleanup rewrite runs for every -utterance regardless of where the text is going. - -The remaining focus context is **not** sent either: app and field names render nowhere (real-world -logs showed them crowding the request — VS Code, for one, parks a screen-reader help announcement in -the focused field's description), the window title is read only to anchor the injector's paste -separator, and selected text is never priming because the paste replaces it. All of it is still -captured — the injector's separator logic consumes the prior text and window title — and reading -stays privacy-guarded: prior and selected text are skipped in secure fields, detected by AX role -**or** subrole and failing closed when the role can't be read, so a password is never read out of the -field at all. That guard, not the steering builder, is what keeps a password out of -`conversation_context` and off the dictation log. - -`build(context:)` returns `.empty` when there's no usable context (or when nothing renders — an -unrecognized app with no prior text and no key terms), and every empty field is **omitted** from the -JSON rather than sent as `[]` or `null`, so the service applies its own defaults. Three omissions are -deliberate — no annotation-suppression clause (_"Transcribe without speaker labels, …"_ is already -part of the dictation service's own default prompt), no language directive, and no filler-word clause; -see [Settled decisions](#settled-decisions--dont-reintroduce-these). +## Transcription prompt + +`Sources/BlurtEngine/STT/TranscriptionPrompt.swift` builds the instruction passed as the dictation +request's `config.prompt` — it steers the _transcription_, not the LLM rewrite (that's the request's +separate `llm` block). It's unit-tested in `Tests/BlurtEngineTests/TranscriptionPromptTests.swift`. + +Every built prompt opens with the fixed `baseInstruction` — _"Transcribe without speaker labels, +audio event descriptions, or emotion markers."_ — a negative-exclusion clause that suppresses the +annotation markers (`[Speaker]`, `[door creaks]`, `[laughing]`) the model would otherwise paste into +the user's text. + +`build(context:)` wraps that pivot in _contextual priming_: prior-cursor text; the selected text (the +highlighted run the dictation will replace, so the model is primed on what's being rewritten — read +via `kAXSelectedTextAttribute`, skipped in secure fields, detected by AX role **or** subrole and +failing closed when the role can't be read, so a password can't reach the prompt); a topic hint from +the window title; a destination sentence from the app/field; and inline keyword boosting from the +user's key terms. It's phrased per AssemblyAI's Universal-3 Pro prompting guidance +(positive/authoritative wording, no "Don't"/"Avoid"/"Never") and stays under the dictation API's +documented 4096-character cap on `config.prompt` (`characterCap`). + +`build(context:)` returns `nil` when there's no usable context, and passing `prompt: nil` to the +transcriber omits the field so the server applies its own default. Two omissions are deliberate and +regression-tested — no language directive and no filler-word clause; see +[Settled decisions](#settled-decisions--dont-reintroduce-these). ## Settings, persistence, and cues @@ -450,8 +426,7 @@ pure edge detector deciding when the chimes fire; the AppKit `CueSoundPlayer` ju resolves. History: **`RecentDictations`** is an in-memory, newest-first ring shown in the ready window (never -written to disk). **`DictationLog`** appends each completed dictation — the transcript plus the exact -steering fields sent (`conversation_context` and `keyterms_prompt`, under the wire's own names) — to +written to disk). **`DictationLog`** appends each completed dictation with its context snapshot to `~/Library/Logs/Blurt/dictations.jsonl` (`DictationLog.defaultURL`, or `defaultDisplayPath` for the home-abbreviated form to show in UI — derived next to the URL so the label can't drift from the write target) — but **only** while developer mode is on; with it off, nothing is written. The Settings diff --git a/App/Blurt/Blurt/Info.plist b/App/Blurt/Blurt/Info.plist index 5a22f54..2d0a224 100644 --- a/App/Blurt/Blurt/Info.plist +++ b/App/Blurt/Blurt/Info.plist @@ -17,9 +17,9 @@ CFBundlePackageType APPL CFBundleShortVersionString - 0.1.36 + 0.1.34 CFBundleVersion - 37 + 35 LSApplicationCategoryType public.app-category.productivity LSMinimumSystemVersion diff --git a/App/Blurt/Blurt/Wizard/Steps/KeyTermsStepView.swift b/App/Blurt/Blurt/Wizard/Steps/KeyTermsStepView.swift index 871f018..bd6027d 100644 --- a/App/Blurt/Blurt/Wizard/Steps/KeyTermsStepView.swift +++ b/App/Blurt/Blurt/Wizard/Steps/KeyTermsStepView.swift @@ -3,8 +3,8 @@ import SwiftUI /// The "Key Terms" section of the Settings window: a free-text /// area where the user lists comma-separated domain words (names, jargon, product -/// names). These ride on every request as its `keyterms_prompt` vocabulary list -/// (see `KeyTermsStore` / `TranscriptionSteering.build`), so the model favors those +/// names). These are folded into every transcription's prompt as spelling priming +/// (see `KeyTermsStore` / `TranscriptionPrompt.build`), so the model favors those /// spellings. Optional — it never gates setup; an empty list just sends no terms. struct KeyTermsStepView: View { /// Stored in UserDefaults so multiple settings windows/readers see edits live. diff --git a/App/Blurt/project.yml b/App/Blurt/project.yml index 8f7db85..6f297b8 100644 --- a/App/Blurt/project.yml +++ b/App/Blurt/project.yml @@ -91,8 +91,8 @@ targets: properties: CFBundleName: Blurt CFBundleDisplayName: Blurt - CFBundleVersion: "37" - CFBundleShortVersionString: "0.1.36" + CFBundleVersion: "35" + CFBundleShortVersionString: "0.1.34" LSMinimumSystemVersion: "15.0" LSApplicationCategoryType: public.app-category.productivity NSMicrophoneUsageDescription: Blurt uses your microphone to transcribe your dictation with AssemblyAI. diff --git a/BLURTENGINE.md b/BLURTENGINE.md index 26d39e1..d0c2f9c 100644 --- a/BLURTENGINE.md +++ b/BLURTENGINE.md @@ -59,7 +59,7 @@ press() ──▶ MicCapture.start() release() ──▶ MicCapture.s Key properties of the design, which your integration can rely on: - **One request per utterance, no streaming.** The dictation API returns the complete transcript — and its LLM-rewritten form — in the response body: no upload step, no job polling, no incremental deltas, no second request for the cleanup. `TranscriberProtocol.transcribe` is a single `async throws -> String`. UIs should show a "transcribing…" state and then the whole result; there is nothing to stream. -- **Cleanup happens server-side.** The request's `llm` block asks the service for a cleanup rewrite (remove disfluencies, fix punctuation), applied to the verbatim transcript inside the same call; it always goes out empty, which selects the service's own default cleanup instruction. Recognition is primed separately by `conversation_context` and `keyterms_prompt` (built by `TranscriptionSteering` from the captured context); `config.prompt` is never sent, and neither is anything describing the destination app. The engine pastes `llm_response`, falling back to the verbatim `text` when the best-effort rewrite failed (`llm_error`) — a degradation, never a user-facing error. There is no client-side LLM pass, no styling stage, and deliberately no hook for one. +- **Cleanup happens server-side.** The request's empty `llm` block asks the service for its default cleanup rewrite (remove disfluencies, fix punctuation), applied to the verbatim transcript inside the same call; the per-utterance `config.prompt` (built by `TranscriptionPrompt` from the captured context) primes the _transcription_. The engine pastes `llm_response`, falling back to the verbatim `text` when the best-effort rewrite failed (`llm_error`) — a degradation, never a user-facing error. There is no client-side LLM pass, no styling stage, and deliberately no hook for one. - **Latency is pre-paid where possible.** `press()` fires a detached `warmUp()` at the transcriber (pre-opening the HTTPS connection while the user speaks, ~170 ms saved cold) and kicks off the cross-process accessibility read of the focused field without awaiting it — the read is then consumed at transcribe time with a bounded wait (`DictationSession.contextWaitBudget`, 500 ms), so an unresponsive frontmost app costs the transcript its priming, never a multi-second stall — and never delays the recording indicator. On the way out, `release()` flips the phase to `.transcribing` _before_ reading the recorded audio back, so a host's stop cue fires at key-up rather than after the disk read. - **A held trigger auto-releases.** `DictationSession` stops recording after `maxRecordingSeconds` (default `SyncSTTLimits.autoReleaseSeconds`, 115 s) so audio never exceeds what the endpoint accepts, and transcribes what it has. Clips shorter than `SyncSTTLimits.minPCMBytes` (~100 ms of audio — an accidental tap) are dropped as a silent no-op rather than sent to earn a 400. @@ -152,15 +152,15 @@ The session calls `setTargetApp` at press time with the app that was frontmost w Recognition quality comes from per-utterance priming, assembled automatically inside `press()` — hosts don't call these APIs directly, but should know what's collected: -- **`TranscriptionContext`** carries the frontmost app name, window title, focused-field label, the text before the caret, the selected text (which a paste will replace), and the user's key terms. It's captured via Accessibility at press time (skipped in secure fields), off the hot path — and consumed at transcribe time with a bounded wait (`DictationSession.contextWaitBudget`), so a hung read is abandoned rather than stalling the transcript. Only the prior text and key terms reach the request; the prior text and window title also steer the injector's paste separator, and nothing else is consumed. -- **`TranscriptionSteering.build(context:)`** renders that into the request's two customization fields: the prior-cursor text as the single `conversation_context` turn (clipped to 4096 chars keeping the tail, since the words nearest the cursor carry the continuity) and the user's key terms as `keyterms_prompt` (whole terms fitted to 2048 chars total). **`config.prompt` is never sent**: it takes a description of the audio rather than instructions, and a custom value replaces the service's managed default including its language steering, which is why the key terms ride in their own field. Nothing else is rendered — no app/field names, no selected text (the paste replaces it), and no standing annotation-suppression clause ("Transcribe without speaker labels, …" is part of the service's own default). Empty fields are omitted rather than sent as `[]`. Three further deliberate omissions: no language directive, no "remove filler words" clause (not something the STT prompt acts on — a no-op), and nothing identifying the destination app — an earlier `AppKindPriming` sent a bundle-ID-derived formatting clause as `llm.instruction`, and that whole path was removed. Don't reintroduce any of them. +- **`TranscriptionContext`** carries the frontmost app name, window title, focused-field label, the text before the caret, the selected text (which a paste will replace), and the user's key terms. It's captured via Accessibility at press time (skipped in secure fields), off the hot path — and consumed at transcribe time with a bounded wait (`DictationSession.contextWaitBudget`), so a hung read is abandoned rather than stalling the transcript. +- **`TranscriptionPrompt.build(context:)`** renders that into the dictation request's `config.prompt` (transcription steering only — the cleanup rewrite is the separate `llm` block), opening with the fixed `baseInstruction` ("Transcribe without speaker labels, audio event descriptions, or emotion markers.") and staying under the API's 4096-character cap. An empty context yields `nil`, which omits the field so the server applies its own default. Two deliberate omissions, both regression-tested: no language directive (pinning to English hurt non-English speech) and no "remove filler words" clause (not in the model's trained instruction set — a no-op). Don't reintroduce either. - **`KeyTermsStore`** persists the user's domain vocabulary (names, jargon) in `UserDefaults`; `DictationSession` re-reads it at every press via its `keyTermsProvider` closure, so Settings edits apply to the next utterance without rebuilding the session. Pass your own provider to source terms from elsewhere. For key storage, compose against **`APIKeyGateway`** — the injectable `current` / `save(_:)` / `hasKey` seam over the key store. `ProductionAPIKeyStore` forwards to the Keychain-backed `APIKeyStore`; `InMemoryAPIKeyStore` is a ready-made in-memory conformance for tests and harnesses (Blurt's XCUITest runs use it so the real Keychain item is never touched, and its `hasKey` backs the session's `readinessCheck`). For a settings UI, **`APIKeySubmission`** wraps the gateway with the validate-then-save flow (`submit(_:)` → valid / invalid / unreachable / saveFailed, via `APIKeyValidator`): it saves only a key AssemblyAI actively accepts, so an unverified key never persists. Two projections keep the surrounding UI out of your views: `Outcome.failureReport` classifies a failure as `.inline(message:)` (recoverable — show it beside the field) or `.alert(title:message:)` (a Keychain fault retyping can't fix), and **`APIKeyDisplay.resolve(key:)`** renders the stored key for an account row — masked tail, status and VoiceOver wording, and the connect-vs-rotate control titles. The mask reveals only the last `revealedTailLength` characters and, below `minimumLengthToMask`, none at all, so a short key can't be shown whole. Setup gating has a projection too: **`SetupReadiness.isReady(permissions:hasAPIKey:)`** is the "fully configured" rule (deliberately excluding the trigger key, which has a default), `SetupReadiness.pollInterval(isReady:)` is the permission-poll cadence (brisk during setup, coasting once ready), and `PermissionStatus.lostGrant(since:)` detects a permission revoked out from under a configured app. -Each completed dictation is appended to **`DictationLog`** (a local JSONL history at `~/Library/Logs/Blurt/dictations.jsonl` — `DictationLog.defaultURL`, or `defaultDisplayPath` for the home-abbreviated form to show in UI) — each entry is the transcript plus the exact steering fields sent (`conversation_context`, `keyterms_prompt`), never the context that wasn't sent — but only while developer mode is switched on. **`DeveloperModeStore`** persists that opt-in in `UserDefaults` (`BlurtDeveloperMode`, off by default); with it off, nothing is written to disk. Blurt surfaces the switch (and the log path) in the Settings window's Developer section. +Each completed dictation is appended to **`DictationLog`** (a local JSONL history at `~/Library/Logs/Blurt/dictations.jsonl` — `DictationLog.defaultURL`, or `defaultDisplayPath` for the home-abbreviated form to show in UI) with its context snapshot — but only while developer mode is switched on. **`DeveloperModeStore`** persists that opt-in in `UserDefaults` (`BlurtDeveloperMode`, off by default); with it off, nothing is written to disk. Blurt surfaces the switch (and the log path) in the Settings window's Developer section. ## Hotkey building blocks @@ -195,7 +195,7 @@ Run `swift test` for the engine suites (`--filter DictationSessionTests` for one Each of these was tried the other way and reverted; the longer stories are in [AGENTS.md](AGENTS.md) and the source comments: - **No external SPM dependencies in the engine.** Foundation/Security/AVFoundation only. -- **No streaming STT, no local models, no client-side LLM cleanup pass.** One dictation request per utterance is the architecture; the cleanup rewrite is server-side (the request's `llm` block), and request steering belongs in `TranscriptionSteering`. +- **No streaming STT, no local models, no client-side LLM cleanup pass.** One dictation request per utterance is the architecture; the cleanup rewrite is server-side (the request's `llm` block), and transcription steering belongs in `TranscriptionPrompt`. - **No `AVAudioEngine`/`installTap` capture path.** Fresh `AVAudioRecorder` per session, resolved at record time. - **Paste is always clipboard-based** (save → write → ⌘V → settle → restore), with the copied-to-clipboard degradation for lost targets. - **No English-pinning or filler-word clauses in the prompt.** diff --git a/README.md b/README.md index f229f0b..9075c6c 100644 --- a/README.md +++ b/README.md @@ -166,14 +166,14 @@ Sources/BlurtEngine/ Swift 6 package owning the pipeline — no external dep Audio/ MicCapture: fresh AVAudioRecorder per session, 16 kHz mono PCM, live level meter; DX7/Juno-106 sound packs STT/ AssemblyAITranscriber: one POST to dictation.assemblyai.com/transcribe - (STT + LLM rewrite) + TranscriptionSteering context/keyterms + (STT + LLM rewrite) + TranscriptionPrompt contextual priming Pipeline/ DictationSession actor: press/release/cancel commands, phase stream, auto-release before the API's recording cap Hotkey/ DictationKeyGate/Router: pure, unit-tested state machine for the lone-modifier trigger (tap vs hold vs combo) Injection/ KeyInjector: save clipboard → paste via synthesized ⌘V → restore - FocusCapture/ Accessibility reads of the focused app/window/field feeding the - request's steering fields, the log, and paste separators + FocusCapture/ Accessibility reads of the focused app/window/field that prime + the transcription prompt Config/, Update/ Keychain API-key store, key terms, download-only release check App/Blurt/ AppKit/SwiftUI shell (Xcode project generated by XcodeGen) diff --git a/Sources/BlurtEngine/Config/KeyTermsStore.swift b/Sources/BlurtEngine/Config/KeyTermsStore.swift index c96488f..de9e223 100644 --- a/Sources/BlurtEngine/Config/KeyTermsStore.swift +++ b/Sources/BlurtEngine/Config/KeyTermsStore.swift @@ -1,9 +1,9 @@ import Foundation /// Storage for the user's dictation "key terms" — a comma-separated list of -/// domain words (names, jargon, product names) sent as the dictation request's -/// `keyterms_prompt` vocabulary list, so the model is more likely to spell -/// them correctly (see `TranscriptionSteering.build`). +/// domain words (names, jargon, product names) that get folded into the dictation +/// request `prompt` as vocabulary priming, so the model is more likely to spell +/// them correctly (see `TranscriptionPrompt.build`). /// /// Unlike the API key these aren't secret, so they live in `UserDefaults` rather /// than the Keychain. The transcription pipeline reads the parsed list via @@ -37,7 +37,7 @@ public enum KeyTermsStore { } /// Pure parse of a comma-separated string into a clean term list. Exposed so - /// `TranscriptionSteering` and tests can reuse the exact same rules. + /// `TranscriptionPrompt` and tests can reuse the exact same rules. public static func parse(_ text: String?) -> [String] { guard let text else { return [] } var seen = Set() diff --git a/Sources/BlurtEngine/FocusCapture/FocusCapture.swift b/Sources/BlurtEngine/FocusCapture/FocusCapture.swift index 4724e42..86d9da4 100644 --- a/Sources/BlurtEngine/FocusCapture/FocusCapture.swift +++ b/Sources/BlurtEngine/FocusCapture/FocusCapture.swift @@ -10,25 +10,27 @@ enum FocusCapture { @MainActor static func captureFrontmost() -> CapturedFocus? { guard let app = NSWorkspace.shared.frontmostApplication else { return nil } - return CapturedFocus(pid: app.processIdentifier, processName: app.localizedName) + return CapturedFocus( + pid: app.processIdentifier, + processName: app.localizedName + ) } static func runningApp(for captured: CapturedFocus) -> NSRunningApplication? { NSRunningApplication(processIdentifier: captured.pid) } - /// Accessibility-derived focus context read from the system-wide focused UI - /// element at dictation start (see `TranscriptionContext` for what each - /// signal feeds). Every field is best-effort: any signal that can't be read - /// is `nil`, and a fully-empty result simply means less context, never an - /// error. + /// Accessibility-derived priming read from the system-wide focused UI element + /// at dictation start (see `TranscriptionContext`). Every field is + /// best-effort: any signal that can't be read is `nil`, and a fully-empty + /// result simply means less context, never an error. struct FocusedFieldContext: Sendable { - /// Text immediately preceding the insertion point. + /// Text immediately preceding the insertion point ("prior chunk context"). let priorText: String? /// The text currently selected in the focused field — the dictation will - /// replace it. + /// replace it, so it primes the model on what the utterance is about. let selectedText: String? - /// The focused window's title (in a code editor it names the open file). + /// The focused window's title — a dense topic hint. let windowTitle: String? /// A short label for the focused field ("To", "Search", "Message"). let fieldLabel: String? @@ -47,8 +49,8 @@ enum FocusCapture { /// /// Secure text fields (password inputs) are detected by role **or** subrole and /// never have their contents read, so a typed password — selected or not — can't - /// leak into the dictation log or the injector. The check fails closed: an - /// unreadable role is treated as secure, since it can't be shown not to be. + /// leak into the STT prompt. The check fails closed: an unreadable role is + /// treated as secure, since it can't be shown not to be. /// /// Deliberately `nonisolated`: each read below is a synchronous cross-process /// IPC round trip into the frontmost app, and an unresponsive app blocks the diff --git a/Sources/BlurtEngine/Pipeline/DictationLog.swift b/Sources/BlurtEngine/Pipeline/DictationLog.swift index 8441771..15c87f5 100644 --- a/Sources/BlurtEngine/Pipeline/DictationLog.swift +++ b/Sources/BlurtEngine/Pipeline/DictationLog.swift @@ -1,54 +1,31 @@ import Foundation -/// Append-only JSONL log of completed dictations — the transcript that came -/// back and every request field that was sent to get it — at +/// Append-only JSONL log of completed transcripts at /// `~/Library/Logs/Blurt/dictations.jsonl`. Used to build a real-world /// corpus for prompt iteration. Written only while developer mode is switched /// on (`DeveloperModeStore` — the Settings window's Developer section, which /// also displays this path), so a user who never opts in has no dictation /// text on disk. public enum DictationLog { - /// One logged dictation: what came back (`transcript`), when, and exactly what - /// was sent to steer it. Fields carry the request's own wire names so a log - /// line reads as the request it describes. - /// - /// Only what was *sent* is recorded. The captured-but-unsent focus context — - /// app and field names, the window title, selected text — deliberately stays - /// off disk. Prior-cursor text is on the sent side of that line now that it - /// rides as `conversation_context`; what keeps a password out of it is - /// `FocusCapture`, which skips prior and selected text in secure fields - /// entirely (failing closed when the AX role can't be read), so it never - /// reaches a context in the first place. struct Entry: Encodable { let transcript: String let ts: String - /// The steering fields sent to AssemblyAI for this utterance. Built here - /// from `context` (rather than threaded through from the transcriber) so the - /// log always reflects what was actually sent, even for calls that construct - /// an entry directly from a context. - let conversationContext: [String] - let keytermsPrompt: [String] - - enum CodingKeys: String, CodingKey { - case transcript - case ts - case conversationContext = "conversation_context" - case keytermsPrompt = "keyterms_prompt" - } - - /// Mirrors `DictationConfig.encode(to:)`: an empty array omits its field, so - /// a line states only what the request actually carried. - func encode(to encoder: Encoder) throws { - var container = encoder.container(keyedBy: CodingKeys.self) - try container.encode(transcript, forKey: .transcript) - try container.encode(ts, forKey: .ts) - if !conversationContext.isEmpty { - try container.encode(conversationContext, forKey: .conversationContext) - } - if !keytermsPrompt.isEmpty { - try container.encode(keytermsPrompt, forKey: .keytermsPrompt) - } - } + /// Focused-app topic hint sent as context, when one was captured. + let app: String? + /// Focused-window title sent as a topic hint, when one was captured. + let window: String? + /// Focused-field label sent as context, when one was captured. + let field: String? + /// Text-before-cursor "prior chunk context" sent, when any was captured. + /// Lets you verify accessibility-tree prior-text reading actually fired. + let prior: String? + /// Selected text sent as context (the dictation replaced it), when any. + let selected: String? + /// The fully-assembled `config.prompt` sent to AssemblyAI for this + /// utterance. Built here from `context` (rather than threaded through from + /// the transcriber) so the log always reflects what was actually sent, + /// even for calls that construct an entry directly from a context. + let prompt: String? } /// Where the log lives. Public so the Settings window's Developer section @@ -111,11 +88,11 @@ public enum DictationLog { static func write( transcript: String, context: TranscriptionContext? = nil, to url: URL, now: Date ) { - let steering = TranscriptionSteering.build(context: context) let entry = Entry( transcript: transcript, ts: now.formatted(timestampFormat), - conversationContext: steering.conversationContext, - keytermsPrompt: steering.keyterms) + app: context?.appName, window: context?.windowTitle, field: context?.fieldLabel, + prior: context?.priorText, selected: context?.selectedText, + prompt: TranscriptionPrompt.build(context: context)) guard var line = try? makeEncoder().encode(entry) else { return } line.append(0x0A) // '\n' diff --git a/Sources/BlurtEngine/Pipeline/DictationSession+Pipeline.swift b/Sources/BlurtEngine/Pipeline/DictationSession+Pipeline.swift index 63005f8..eb830eb 100644 --- a/Sources/BlurtEngine/Pipeline/DictationSession+Pipeline.swift +++ b/Sources/BlurtEngine/Pipeline/DictationSession+Pipeline.swift @@ -39,7 +39,8 @@ extension DictationSession { // cancelled pipeline clear a *newer* press's stream: `cancel()` detaches this // task while it's parked in `firstValue`, a fresh `press()` installs its own // `contextStream`, and this task's resumption then nils that one out — so - // dictation #2 transcribes with `context: nil`, silently losing its priming. + // dictation #2 transcribes with `context: nil`, losing not just its priming but + // `baseInstruction`, and `[Speaker]`-style markers can reach the pasted text. // The window is microseconds, but the invariant is now local instead of // depending on scheduling. let stream = contextStream diff --git a/Sources/BlurtEngine/STT/AssemblyAITranscriber.swift b/Sources/BlurtEngine/STT/AssemblyAITranscriber.swift index 5fb1faf..dcd2353 100644 --- a/Sources/BlurtEngine/STT/AssemblyAITranscriber.swift +++ b/Sources/BlurtEngine/STT/AssemblyAITranscriber.swift @@ -51,8 +51,8 @@ public struct AssemblyAITranscriber: TranscriberProtocol { guard let apiKey = apiKeyProvider(), !apiKey.isEmpty else { throw BlurtError.apiKeyMissing } - let steering = TranscriptionSteering.build(context: context) - let config = try makeConfigData(sampleRate: sampleRate, steering: steering) + let prompt = TranscriptionPrompt.build(context: context) + let config = try makeConfigData(sampleRate: sampleRate, prompt: prompt) let boundary = "blurt-\(UUID().uuidString)" var request = URLRequest(url: baseURL.appendingPathComponent("transcribe")) @@ -105,20 +105,18 @@ public struct AssemblyAITranscriber: TranscriberProtocol { "warm-up connect \(elapsedMs, format: .fixed(precision: 0), privacy: .public)ms") } - /// Builds the JSON `config` part sent alongside the audio. Each steering field - /// is included only when it carries something — an empty array omits the field - /// rather than stating an empty value, so the service applies its own default. - /// The `llm` block always rides along — see `DictationConfig.llm`. Internal so - /// tests can assert the steering wiring without inspecting the multipart upload - /// body (which `URLProtocol` mocks can't observe reliably for `upload(from:)`). - func makeConfigData(sampleRate: Int, steering: TranscriptionSteering.Fields) throws -> Data { + /// Builds the JSON `config` part sent alongside the audio. The context + /// `prompt` is included only when non-empty; a nil or blank prompt omits the + /// field so the server applies its default prompt. The `llm` block always + /// rides along — see `DictationConfig.llm`. Internal so tests can assert the + /// prompt wiring without inspecting the multipart upload body (which + /// `URLProtocol` mocks can't observe reliably for `upload(from:)`). + func makeConfigData(sampleRate: Int, prompt: String?) throws -> Data { try JSONEncoder().encode( DictationConfig( sampleRate: sampleRate, channels: 1, - conversationContext: steering.conversationContext, - keytermsPrompt: steering.keyterms, - llm: LLMRewrite() + prompt: prompt.trimmedNonEmpty() ) ) } @@ -196,55 +194,27 @@ public struct AssemblyAITranscriber: TranscriberProtocol { // MARK: - Wire types - /// The JSON `config` part. Note the absence of `prompt`: that field takes a - /// description of the audio rather than instructions, and a custom value - /// replaces the service's managed default (language steering included), so - /// Blurt sends none — see `TranscriptionSteering` for the full reasoning. private struct DictationConfig: Encodable { let sampleRate: Int let channels: Int - /// Turns preceding this utterance (Blurt sends at most one: the text before - /// the insertion point). Encoded only when non-nil — the synthesized - /// `encode` uses `encodeIfPresent` for optionals — so an utterance with no - /// prior text omits the field instead of sending `[]`. - let conversationContext: [String] - /// The user's key terms as the explicit vocabulary list, omitted when empty. - let keytermsPrompt: [String] - /// The rewrite request. Always present so the service runs the rewrite at - /// all, and always empty so the instruction it runs is the service's own - /// default cleanup — Blurt sends nothing describing the destination app. - let llm: LLMRewrite + /// Custom transcription instruction. Encoded only when non-nil (the + /// synthesized `encode` uses `encodeIfPresent` for optionals), so omitting + /// it falls back to the server's default prompt. Steers *transcription*; + /// the cleanup rewrite is the `llm` block's job. + let prompt: String? + /// The rewrite request. An empty object selects the service's default + /// cleanup instruction; per the API's `instruction`-mode rules, output + /// format and don't-answer-the-text safeguards are enforced server-side, + /// so nothing rides along here. + let llm = LLMRewrite() enum CodingKeys: String, CodingKey { case sampleRate = "sample_rate" case channels - case conversationContext = "conversation_context" - case keytermsPrompt = "keyterms_prompt" + case prompt case llm } - - /// Hand-written so an empty array *omits* its field rather than encoding - /// `[]`. The distinction is on the wire: `"keyterms_prompt": []` states an - /// empty vocabulary, while omission leaves the service to its own handling. - /// (Optional arrays would express this too, but read as "maybe no list" - /// where the truth is "a list, possibly empty".) - func encode(to encoder: Encoder) throws { - var container = encoder.container(keyedBy: CodingKeys.self) - try container.encode(sampleRate, forKey: .sampleRate) - try container.encode(channels, forKey: .channels) - if !conversationContext.isEmpty { - try container.encode(conversationContext, forKey: .conversationContext) - } - if !keytermsPrompt.isEmpty { - try container.encode(keytermsPrompt, forKey: .keytermsPrompt) - } - try container.encode(llm, forKey: .llm) - } } - /// Encodes as `{}`: asking for the rewrite without steering it. An - /// `instruction` here once carried a formatting clause derived from the - /// frontmost app's bundle ID; that was removed, and the service's default - /// cleanup instruction is what runs now. private struct LLMRewrite: Encodable {} private struct DictationResponse: Decodable { diff --git a/Sources/BlurtEngine/STT/TranscriptionContext.swift b/Sources/BlurtEngine/STT/TranscriptionContext.swift index 331ec29..5a5fc3e 100644 --- a/Sources/BlurtEngine/STT/TranscriptionContext.swift +++ b/Sources/BlurtEngine/STT/TranscriptionContext.swift @@ -1,42 +1,39 @@ -/// Per-utterance snapshot of where the dictation is going, gathered at -/// dictation start from the focused app and field. `TranscriptionSteering.build` -/// renders only the parts the request uses — the prior text (as -/// `conversation_context`) and the key terms (as `keyterms_prompt`). Nothing -/// identifying the destination app is sent. The prior text and window title -/// also steer the injector's paste separator; nothing else is consumed, and the -/// rest of the context is neither sent nor logged. +/// Per-utterance context the STT model is trained to use as *contextual* +/// priming — it improves recognition accuracy (vocabulary, continuity, +/// capitalization) without changing the output format. Gathered at dictation +/// start from the focused app and the text preceding the cursor, then rendered +/// into the request `prompt` by `TranscriptionPrompt.build`. /// -/// Every focus field is optional and best-effort: whatever couldn't be read is -/// `nil`, and an entirely empty context customizes nothing (the server applies -/// its own defaults). +/// Both fields are optional: whichever is available is used, and an entirely +/// empty context yields no prompt (the server applies its own default). public struct TranscriptionContext: Sendable, Equatable { - /// The frontmost application's display name (e.g. "Slack", "Xcode"). Never - /// sent. + /// The frontmost application's display name (e.g. "Slack", "Xcode"), passed + /// as a domain/topic hint so the model expects that app's vocabulary. public let appName: String? - /// The focused window's title (e.g. "main.py — blurt", a document name, a - /// Slack channel). Never sent. Read only to anchor the injector's same-window - /// separator fallback. + /// The focused window's title (e.g. "Re: Q3 pricing — Gmail", a document + /// name, a Slack channel). The densest topic hint available — usually packed + /// with the proper nouns and domain vocabulary the model would otherwise guess. public let windowTitle: String? /// A short label for the focused field (placeholder/title/role, e.g. "To", - /// "Subject", "Search", "Message"). Never sent. + /// "Subject", "Search", "Message"), passed so the model knows what *kind* of + /// text is expected — an email address, a search query, and prose should be + /// transcribed differently. public let fieldLabel: String? - /// Text immediately preceding the insertion point in the focused field. Sent - /// as the single `conversation_context` turn, so the model knows what the - /// utterance continues; it also drives the injector's leading-separator - /// decision. Skipped entirely in secure fields by `FocusCapture`, so a - /// password never reaches this field. + /// Text immediately preceding the insertion point in the focused field, + /// passed as "prior chunk context" so the transcript continues naturally. public let priorText: String? - /// The text currently selected in the focused field, when any. Never sent: the - /// paste replaces it, so priming the model with it would condition the - /// transcription on text that is on its way out. + /// The text currently selected in the focused field, when any. Dictating with + /// a selection replaces it (the paste overwrites the highlighted range), so + /// this is passed as priming for what the utterance is about — the vocabulary + /// and topic of the text being rewritten. public let selectedText: String? - /// User-configured domain vocabulary (names, jargon, product names), sent as - /// `keyterms_prompt` so the model favors these spellings. Unlike the other + /// User-configured domain vocabulary (names, jargon, product names) carried as + /// spelling priming so the model favors these spellings. Unlike the other /// fields this isn't per-utterance focus state — it's the same list every time, /// sourced from `KeyTermsStore`. public let keyTerms: [String] diff --git a/Sources/BlurtEngine/STT/TranscriptionPrompt.swift b/Sources/BlurtEngine/STT/TranscriptionPrompt.swift new file mode 100644 index 0000000..9339c54 --- /dev/null +++ b/Sources/BlurtEngine/STT/TranscriptionPrompt.swift @@ -0,0 +1,130 @@ +/// Builds the instruction sent to the dictation API as the `prompt` +/// field of the request `config` (see `AssemblyAITranscriber`). The STT model +/// prepends this to its own system prompt. +/// +/// Every built prompt opens with a fixed `baseInstruction` — a plain-text +/// exclusion clause (see below) — and wraps it in +/// *contextual* priming: a topic hint built from the window title, a +/// destination sentence built from the focused app and field label, "prior +/// chunk context" (the text preceding the cursor), the selected text (which the +/// dictation replaces), and keyword boosting, all of which the model is +/// mid-trained to use for better recognition accuracy. +/// +/// On the directives in `baseInstruction`: a "remove filler words"-style +/// *content* reshaping is **not** in the model's trained instruction set, so it +/// is a no-op and is deliberately omitted (see the project memory note). A +/// language directive is likewise omitted — pinning the prompt to English hurt +/// non-English transcription, so language is left to the model's own detection. +/// The negative feature *exclusion* ("without speaker labels, …") is a trained +/// instruction-following type, so it does take effect — the exclusion +/// suppresses the annotation markers the model would otherwise emit (`[Speaker]`, +/// `[door creaks]`, `[laughing]`, …), which in a dictation product would be +/// pasted into the user's text as literal tokens. The list is trimmed to the +/// three annotation types a dictation user could plausibly trigger; the rarer +/// types (unclear-speech, censor, foreign-language, lyrics) are left out to keep +/// the negative clause short, matching the doc's brief negative examples. +/// +/// Output follows the trained format with `baseInstruction` as its pivot: the +/// prior-chunk context, the topic hint, and the destination sentence precede it +/// as the `{context}. {baseInstruction}` shape, and keyword boosting trails it +/// inline as `Keywords: a, b, c.` (per the mid-training instruction-type +/// reference). It stays under the API's `characterCap`: the contextual +/// blocks are clipped upstream in `FocusCapture`, and the key-terms clause is +/// fitted to the remaining budget here. Exercised by +/// `Tests/BlurtEngineTests/TranscriptionPromptTests.swift`. +enum TranscriptionPrompt { + /// The standing dictation instruction prepended to the model's system prompt + /// on every built prompt. A negative-exclusion clause (§5/§6) naming the + /// annotation feature types the model is trained to emit, so it suppresses + /// them. No language directive: pinning the prompt to English degraded + /// transcription for non-English speech, so the model is left to detect the + /// spoken language itself. + static let baseInstruction = + "Transcribe without speaker labels, audio event descriptions, or emotion markers." + + /// Hard cap the dictation API places on `config.prompt` ("max 4096 chars"); + /// a longer prompt risks failing the whole request, so `build` must never + /// exceed it. The contextual blocks are all clipped upstream in + /// `FocusCapture`; the user's key terms are the one unbounded input, so + /// `build` fits them to whatever budget remains. + static let characterCap = 4096 + + /// Renders `context` into a transcription prompt, or `nil` when there is no usable + /// context (the server then applies its own default prompt). + static func build(context: TranscriptionContext?) -> String? { + // `isEmpty` is the context type's own "no usable content" rule — the same + // predicate `DictationSession.performPress` gates on before yielding a + // context. Asking it here (rather than re-deriving the field-by-field test) + // keeps a newly added context signal from being silently dropped. + guard let context, !context.isEmpty else { return nil } + let prior = context.priorText.trimmedNonEmpty() ?? "" + let selected = context.selectedText.trimmedNonEmpty() ?? "" + let app = context.appName.trimmedNonEmpty() ?? "" + let window = context.windowTitle.trimmedNonEmpty() ?? "" + let field = context.fieldLabel.trimmedNonEmpty() ?? "" + let keyTerms = context.keyTerms + + // `baseInstruction` is the pivot of the trained format. Contextual priming + // sits *before* it; keyword boosting trails *after* it. The leading blocks, + // separated by blank lines, precede it: + // 1. the prior-chunk block (`Previous transcript:\n…`, its own paragraph), + // 2. the selected-text block (`Selected text:\n…`, what the dictation + // replaces — primes vocabulary/topic of the text being rewritten), + // 3. the location clause (topic hint + destination sentence). + var blocks: [String] = [] + if !prior.isEmpty { + blocks.append("Previous transcript:\n\(prior)") + } + if !selected.isEmpty { + blocks.append("Selected text:\n\(selected)") + } + let location = locationClause(app: app, window: window, field: field) + // The topic hint and `baseInstruction` share one line as the trained + // `{context}. {baseInstruction}` shape; with no topic it's the bare base. + let instruction = location.isEmpty ? baseInstruction : "\(location) \(baseInstruction)" + blocks.append(instruction) + var prompt = blocks.joined(separator: "\n\n") + if !keyTerms.isEmpty { + // Spelling priming: the user's domain vocabulary, boosted via the trained + // inline `Keywords: a, b, c.` form (Section 2.3) trailing the marker so the + // model favors these exact spellings for names/jargon it would guess at. + // The terms list is the one input with no upstream length cap, so include + // only as many whole terms as `characterCap` leaves room for, so a huge + // Settings list can't crowd out the instruction itself or balloon every + // request. + var included: [String] = [] + var remaining = characterCap - prompt.count - " Keywords: .".count + for term in keyTerms { + let cost = term.count + (included.isEmpty ? 0 : ", ".count) + guard cost <= remaining else { break } + included.append(term) + remaining -= cost + } + if !included.isEmpty { + prompt += " Keywords: \(included.joined(separator: ", "))." + } + } + return prompt + } + + /// The "where am I typing" priming clause, assembled from whichever of the + /// app / window / field signals are present (empty when none are). Two trained + /// shapes joined by a space: a topic hint built from the window title (the + /// richest vocabulary signal — `This is about "…".`, mid-training §2.1) leads, + /// and a destination sentence built from the app/field (`Dictated into …`) + /// trails it. Each sentence ends with a period so the clause joins cleanly + /// before `baseInstruction`. + private static func locationClause(app: String, window: String, field: String) -> String { + let topic = window.isEmpty ? "" : "This is about \"\(window)\"." + + let destination: String + switch (app.isEmpty, field.isEmpty) { + case (false, false): destination = "Dictated into \(app), in the \"\(field)\" field." + case (false, true): destination = "Dictated into \(app)." + case (true, false): destination = "Dictated in the \"\(field)\" field." + case (true, true): destination = "" + } + + return [topic, destination].filter { !$0.isEmpty }.joined(separator: " ") + } +} diff --git a/Sources/BlurtEngine/STT/TranscriptionSteering.swift b/Sources/BlurtEngine/STT/TranscriptionSteering.swift deleted file mode 100644 index f809f14..0000000 --- a/Sources/BlurtEngine/STT/TranscriptionSteering.swift +++ /dev/null @@ -1,109 +0,0 @@ -/// Renders a `TranscriptionContext` into the two request-customization fields -/// the dictation API accepts, each of which has one job (see -/// `AssemblyAITranscriber` for the wire encoding): -/// -/// - `conversation_context` — the text immediately before the insertion point, -/// as a single turn. This is real left-context: it tells the model what the -/// utterance is continuing, which is what fixes mid-sentence casing and -/// proper-noun consistency. -/// - `keyterms_prompt` — the user's key terms, verbatim, as the explicit -/// vocabulary list the field is for. -/// -/// **`config.prompt` is deliberately never sent.** The field takes a -/// *description of the audio* ("Cardiology consultation about chest pain -/// symptoms."), not instructions — transcription behavior is optimized out of -/// the box, so an imperative like "Transcribe speech into markdown." was aimed -/// at a field that does not act on instructions. That is also why the key terms -/// ride in `keyterms_prompt` rather than being packed into the prompt as a -/// `Keywords:` clause. Sending no prompt keeps the service's managed default — -/// a custom prompt replaces it wholesale, including its language steering, -/// which is the mechanism behind the earlier finding that pinning the prompt to -/// English hurt non-English speech. -/// -/// **Nothing describing the destination app is sent either.** An earlier -/// version recognized the frontmost app's bundle ID as a *kind* (terminal, code -/// editor, Slack, Obsidian) and sent a matching formatting clause as -/// `llm.instruction`. That is gone: the `llm` block now carries no instruction, -/// so the service's own default cleanup rewrite applies to every utterance -/// regardless of where the text is going. -/// -/// Blurt has no earlier turns to send: `conversation_context` carries exactly -/// one entry, the prior-cursor text, or none. -/// -/// The rest of the captured context is not sent at all. The app name and field -/// label render nowhere (real-world logs showed them crowding the request — VS -/// Code parks a screen-reader help announcement in the focused field's -/// description), and the window title is read only to anchor the injector's -/// paste separator. Selected text is never priming: the paste replaces it, so -/// conditioning the model on it would prime for text on its way out. -/// -/// Exercised by `Tests/BlurtEngineTests/TranscriptionSteeringTests.swift`. -enum TranscriptionSteering { - /// What one utterance sends beyond the audio and its geometry. Built here so - /// the transcriber and the dictation log describe the same request rather than - /// each deriving it. - struct Fields: Sendable, Equatable { - /// Turns preceding this utterance, oldest first. At most one entry (the - /// prior-cursor text). - let conversationContext: [String] - /// Explicit vocabulary to bias recognition toward, in the user's own - /// spelling and capitalization. - let keyterms: [String] - - /// Nothing to customize — every field omitted, so the service applies its - /// managed default prompt and its default cleanup rewrite. Also the value - /// to compare against for "does this utterance customize anything?" — - /// `Fields` is `Equatable`, so no separate emptiness predicate exists to - /// drift from the fields themselves. - static let empty = Fields(conversationContext: [], keyterms: []) - } - - /// Cap the dictation API documents for `conversation_context`: 4096 characters - /// across all turns. Over-cap context is trimmed rather than rejected, so this - /// is a quality guard, not a request-failure guard — but the clip must keep the - /// *tail*, since the words nearest the insertion point are the ones carrying - /// continuity. `FocusCapture` already caps prior text far below this - /// (`maxPriorChars`, 320); the guard is here so a hand-built or future-widened - /// context can't silently exceed the field. - static let conversationContextCharacterCap = 4096 - - /// Cap the dictation API documents for `keyterms_prompt`: 2048 characters - /// totalled across every term. Key terms are the one input with no upstream - /// length limit — a Settings list of any size — so `build` includes only as - /// many whole leading terms as fit. - static let keytermsCharacterCap = 2048 - - /// Renders `context` into the fields to send. An absent or unusable context - /// yields `.empty`, which sends no customization at all. - static func build(context: TranscriptionContext?) -> Fields { - // `isEmpty` is the context type's own "no usable content" rule — the same - // predicate `DictationSession.performPress` gates on before yielding a - // context. Asking it here (rather than re-deriving the field-by-field test) - // keeps a newly added context signal from being silently dropped. - guard let context, !context.isEmpty else { return .empty } - return Fields( - conversationContext: priorTurn(of: context).map { [$0] } ?? [], - keyterms: fittedKeyterms(context.keyTerms)) - } - - /// The prior-cursor text as one conversation turn, clipped to the field's cap - /// from the front so the text nearest the insertion point survives. - private static func priorTurn(of context: TranscriptionContext) -> String? { - guard let prior = context.priorText.trimmedNonEmpty() else { return nil } - guard prior.count > conversationContextCharacterCap else { return prior } - return String(prior.suffix(conversationContextCharacterCap)) - } - - /// As many whole leading terms as the total-length cap allows. Whole terms - /// only: half a proper noun biases the model toward a spelling nobody wants. - private static func fittedKeyterms(_ terms: [String]) -> [String] { - var included: [String] = [] - var remaining = keytermsCharacterCap - for term in terms { - guard term.count <= remaining else { break } - included.append(term) - remaining -= term.count - } - return included - } -} diff --git a/Tests/BlurtEngineTests/AssemblyAITranscriberTests.swift b/Tests/BlurtEngineTests/AssemblyAITranscriberTests.swift index 5f641c2..2da4581 100644 --- a/Tests/BlurtEngineTests/AssemblyAITranscriberTests.swift +++ b/Tests/BlurtEngineTests/AssemblyAITranscriberTests.swift @@ -52,14 +52,14 @@ struct HTTPClientTests { #expect(result == expected) } - @Test("transcriber succeeds with a real context (builds and sends steering fields)") + @Test("transcriber succeeds with a real context (builds and sends a prompt)") func transcribeWithContext() async throws { let transport = FakeHTTPTransport { request in guard request.url?.path.hasSuffix("/transcribe") == true else { return (404, Data()) } return (200, json(["text": "hello world"])) } - // A non-empty context exercises the TranscriptionSteering.build path inside + // A non-empty context exercises the TranscriptionPrompt.build path inside // transcribe() that the nil-context happy path skips. The fake can't observe // the multipart upload body, so this asserts the request still round-trips // cleanly rather than the wire contents (covered directly by makeConfigData). @@ -154,59 +154,29 @@ struct HTTPClientTests { } } - @Test("config part carries each steering field under its documented wire name") - func configIncludesSteeringFields() throws { - let object = try configObject( - steering: TranscriptionSteering.Fields( - conversationContext: ["$ git status"], keyterms: ["Blurt", "AssemblyAI"])) - #expect(object["conversation_context"] as? [String] == ["$ git status"]) - #expect(object["keyterms_prompt"] as? [String] == ["Blurt", "AssemblyAI"]) + @Test("config part carries the built context prompt") + func configIncludesPrompt() throws { + let object = try configObject(prompt: "CONTEXT. Transcribe.") + #expect(object["prompt"] as? String == "CONTEXT. Transcribe.") #expect(object["sample_rate"] as? Int == 16_000) // The capture path is mono by construction; the declared geometry must agree. #expect(object["channels"] as? Int == 1) } - @Test("config part never sends a prompt — the managed default is what steers transcription") - func configNeverSendsPrompt() throws { - // A custom `config.prompt` replaces the service's managed default *and* its - // language steering, and the field wants a description of the audio rather - // than instructions — which is why vocabulary rides in `keyterms_prompt`. - // Sending no prompt keeps the managed default, so this pins the field's - // absence for every steering shape. - let shapes: [TranscriptionSteering.Fields] = [ - .empty, - TranscriptionSteering.Fields(conversationContext: ["Hi Sam,"], keyterms: ["Blurt"]), - ] - for steering in shapes { - #expect(try configObject(steering: steering).keys.contains("prompt") == false) - } - } - - @Test("config part always requests the unsteered default cleanup rewrite") - func configRequestsDefaultRewrite() throws { - // `llm` must be present and empty for every utterance: present so the service + @Test("config part always requests the default cleanup rewrite", arguments: ["CONTEXT. Transcribe.", nil]) + func configRequestsDefaultRewrite(prompt: String?) throws { + // `llm` must be present and empty on every request: present so the service // runs the rewrite at all, empty so the server-owned default cleanup // instruction (and its guardrails) applies rather than a client-side copy. - // Empty also means nothing about the destination app rides here, which is - // what an earlier `instruction` carried. `isEmpty == true` covers presence - // too — it is false for a missing `llm`. - let shapes: [TranscriptionSteering.Fields] = [ - .empty, - TranscriptionSteering.Fields(conversationContext: ["Hi Sam,"], keyterms: ["Blurt"]), - ] - for steering in shapes { - #expect((try configObject(steering: steering)["llm"] as? [String: Any])?.isEmpty == true) - } + // `isEmpty == true` also covers presence — it is false for a missing `llm`. + #expect((try configObject(prompt: prompt)["llm"] as? [String: Any])?.isEmpty == true) } - @Test("config part omits the context and keyterms fields when they are empty") - func configOmitsEmptySteeringFields() throws { - // An empty array is not the same as an absent field: sending - // `"keyterms_prompt": []` states an empty vocabulary where omission lets the - // service apply its own handling. - let object = try configObject(steering: .empty) - #expect(object.keys.contains("conversation_context") == false) - #expect(object.keys.contains("keyterms_prompt") == false) + @Test( + "config part omits the prompt field when there is no usable context", + arguments: [nil, " \n"]) + func configOmitsPrompt(prompt: String?) throws { + #expect(try configObject(prompt: prompt).keys.contains("prompt") == false) } @Test("the multipart body frames the audio and config parts the dictation API expects") @@ -336,9 +306,9 @@ struct HTTPClientTests { /// config assertion below wants, since `makeConfigData` returns raw JSON. /// A part that isn't a JSON object at all fails here rather than turning every /// downstream assertion into a silent nil-compare. - private func configObject(steering: TranscriptionSteering.Fields) throws -> [String: Any] { + private func configObject(prompt: String?) throws -> [String: Any] { let config = try makeTranscriber(apiKey: "test-key") - .makeConfigData(sampleRate: 16_000, steering: steering) + .makeConfigData(sampleRate: 16_000, prompt: prompt) return try #require(JSONSerialization.jsonObject(with: config) as? [String: Any]) } diff --git a/Tests/BlurtEngineTests/DictationLogTests.swift b/Tests/BlurtEngineTests/DictationLogTests.swift index 84187ad..163fabb 100644 --- a/Tests/BlurtEngineTests/DictationLogTests.swift +++ b/Tests/BlurtEngineTests/DictationLogTests.swift @@ -8,26 +8,15 @@ private struct DecodedEntry: Decodable { let ts: String } -/// Decodes the steering fields so tests can assert the exact request -/// customization that was sent is what lands on disk, under the same wire names -/// the request uses. -private struct DecodedSteering: Decodable { - let conversationContext: [String] - let keytermsPrompt: [String] - - enum CodingKeys: String, CodingKey { - case conversationContext = "conversation_context" - case keytermsPrompt = "keyterms_prompt" - } - - // A missing array decodes to empty rather than nil: whether a field was - // *omitted* is asserted against the raw line, so nothing here needs to tell - // absent from empty. - init(from decoder: Decoder) throws { - let container = try decoder.container(keyedBy: CodingKeys.self) - conversationContext = try container.decodeIfPresent([String].self, forKey: .conversationContext) ?? [] - keytermsPrompt = try container.decodeIfPresent([String].self, forKey: .keytermsPrompt) ?? [] - } +/// Decodes the optional focus-context fields so tests can assert they're +/// threaded from the `TranscriptionContext` onto disk. +private struct DecodedContext: Decodable { + let app: String? + let window: String? + let field: String? + let prior: String? + let selected: String? + let prompt: String? } /// Each test gets a fresh empty file in a unique temp directory so the host's real @@ -103,57 +92,50 @@ struct DictationLogTests { #expect(transcript < ts) } - @Test("logs only what was sent — context captured but never sent stays off disk") - func logsOnlyWhatWasSent() { + @Test("threads focus context (incl. selected text) onto disk") + func logsContext() { let url = makeTempLogURL() let context = TranscriptionContext( - appName: "Obsidian", windowTitle: "Grocery list", - fieldLabel: "text entry area", priorText: "- milk", selectedText: "- bread") + appName: "Mail", windowTitle: "Re: Q3 pricing", fieldLabel: "Body", + priorText: "Hi Sam,", selectedText: "the old plan") DictationLog.write(transcript: "p", context: context, to: url, now: Date()) let line = readLog(url).split(separator: "\n").first.map(String.init) ?? "" - // Everything the request carried is recorded, under the wire's own names… - let decoded = try? JSONDecoder().decode(DecodedSteering.self, from: Data(line.utf8)) - #expect(decoded?.conversationContext == ["- milk"]) - // …and none of the captured-but-unsent context is. Values, not just keys: - // the entry must carry no trace of what stayed on the machine. Two are - // load-bearing: selected text, which the paste replaces so it is never sent, - // and anything naming the destination app, which is never sent either. - for unsent in ["Obsidian", "Grocery list", "text entry area", "- bread"] { - #expect(!line.contains(unsent)) - } + let decoded = try? JSONDecoder().decode(DecodedContext.self, from: Data(line.utf8)) + #expect(decoded?.app == "Mail") + #expect(decoded?.window == "Re: Q3 pricing") + #expect(decoded?.field == "Body") + #expect(decoded?.prior == "Hi Sam,") + #expect(decoded?.selected == "the old plan") + } + + @Test("omits the selected field when nothing is selected") + func omitsSelectedWhenAbsent() { + let url = makeTempLogURL() + DictationLog.write(transcript: "p", context: nil, to: url, now: Date()) + let line = readLog(url).split(separator: "\n").first.map(String.init) ?? "" + // `Encodable` synthesis uses `encodeIfPresent`, so a nil field is absent + // rather than `"selected":null`. + #expect(!line.contains("selected")) } - @Test("logs the same steering fields the transcriber sends") - func logsAssembledSteering() { + @Test("logs the same assembled prompt the transcriber sends") + func logsAssembledPrompt() { let url = makeTempLogURL() let context = TranscriptionContext( - appName: "Obsidian", windowTitle: "Grocery list", - fieldLabel: "text entry area", priorText: "- milk", keyTerms: ["Blurt"]) + appName: "Mail", windowTitle: "Re: Q3 pricing", fieldLabel: "Body", + priorText: "Hi Sam,", selectedText: "the old plan") DictationLog.write(transcript: "p", context: context, to: url, now: Date()) let line = readLog(url).split(separator: "\n").first.map(String.init) ?? "" - let decoded = try? JSONDecoder().decode(DecodedSteering.self, from: Data(line.utf8)) - let sent = TranscriptionSteering.build(context: context) - // The log is the corpus prompt iteration reads, so it has to agree with the - // builder field-for-field rather than approximately. - #expect(decoded?.conversationContext == sent.conversationContext) - #expect(decoded?.keytermsPrompt == sent.keyterms) - #expect(decoded?.conversationContext == ["- milk"]) - #expect(decoded?.keytermsPrompt == ["Blurt"]) + let decoded = try? JSONDecoder().decode(DecodedContext.self, from: Data(line.utf8)) + #expect(decoded?.prompt == TranscriptionPrompt.build(context: context)) } - @Test("omits every steering field when there is no context to build one") - func omitsSteeringWhenNoContext() { + @Test("omits the prompt field when there is no context to build one") + func omitsPromptWhenNoContext() { let url = makeTempLogURL() DictationLog.write(transcript: "p", context: nil, to: url, now: Date()) let line = readLog(url).split(separator: "\n").first.map(String.init) ?? "" - // Absent, not `null` and not `[]` — the entry should read as "nothing was - // customized", matching the request, which omits these fields too. `prompt` - // and `llm_instruction` are never written for any context, so their absence - // here doubles as a pin against reintroducing either. #expect(!line.contains("\"prompt\"")) - #expect(!line.contains("\"llm_instruction\"")) - #expect(!line.contains("\"conversation_context\"")) - #expect(!line.contains("\"keyterms_prompt\"")) } @Test("survives unicode in transcript field") @@ -196,20 +178,11 @@ struct DictationLogGateTests { @Test("the gate is checked before the context is touched, for both settings") func gateAppliesToContextualEntries() { - // The pipeline always passes the captured context — the input the logged - // steering fields are built from. Off must persist nothing at all; on - // persists what was sent, and still never the context that wasn't (the - // window title and field label stay off disk either way). - // - // Prior-cursor text *is* sent now (as `conversation_context`), so it is on - // disk for a user who opted in. What keeps a password out of it is upstream, - // in `FocusCapture`: prior and selected text are skipped entirely in secure - // fields, failing closed when the AX role can't be read. That guard is - // covered in `FocusCaptureTests`; this suite can only see contexts that - // already cleared it. + // The pipeline always passes the captured context, which is the part carrying + // prior text and the assembled prompt. Off must persist none of it. let context = TranscriptionContext( - appName: "Terminal", windowTitle: "Vault", fieldLabel: "Command", - priorText: "$ git", selectedText: nil) + appName: "1Password", windowTitle: "Vault", fieldLabel: "Password", + priorText: "hunter2", selectedText: nil) let offURL = makeTempLogURL() DictationLog.append( transcript: "p", context: context, @@ -222,11 +195,7 @@ struct DictationLogGateTests { DictationLog.queue.sync {} #expect(!FileManager.default.fileExists(atPath: offURL.path)) - let logged = readLog(onURL) - #expect(logged.contains("$ git")) - #expect(!logged.contains("Terminal")) - #expect(!logged.contains("Vault")) - #expect(!logged.contains("Command")) + #expect(readLog(onURL).contains("hunter2")) } } diff --git a/Tests/BlurtEngineTests/TranscriptionContextTests.swift b/Tests/BlurtEngineTests/TranscriptionContextTests.swift index 3550178..deef980 100644 --- a/Tests/BlurtEngineTests/TranscriptionContextTests.swift +++ b/Tests/BlurtEngineTests/TranscriptionContextTests.swift @@ -3,11 +3,9 @@ import Testing @testable import BlurtEngine /// `TranscriptionContext.isEmpty` is the gate `FocusCapture`/`DictationSession` -/// use to decide whether a context is worth carrying at all (steering AND log). -/// The agreement with `TranscriptionSteering.build` is one-directional: -/// `isEmpty == true` must always correspond to empty steering fields, while a -/// non-empty context may still steer nothing — its signals can be carry-only -/// (app name, window title, field label, or selected text). +/// use to decide whether a context is worth sending as priming. It mirrors the +/// emptiness logic in `TranscriptionPrompt.build`, so the two must agree: +/// `isEmpty == true` should always correspond to `build` returning `nil`. @Suite("TranscriptionContext") struct TranscriptionContextTests { @Test("both fields nil is empty") @@ -40,46 +38,31 @@ struct TranscriptionContextTests { #expect(TranscriptionContext(appName: nil, priorText: nil, selectedText: " \n").isEmpty) } - @Test("key terms alone make it non-empty (and produce keyterms)") + @Test("key terms alone make it non-empty (and produce a prompt)") func keyTermsPresent() { let context = TranscriptionContext(appName: nil, priorText: nil, keyTerms: ["Blurt"]) #expect(!context.isEmpty) - #expect(TranscriptionSteering.build(context: context).keyterms == ["Blurt"]) + #expect(TranscriptionPrompt.build(context: context) != nil) } - @Test("an empty context always corresponds to empty steering fields") - func agreesWithSteeringBuild() { + @Test("emptiness agrees with TranscriptionPrompt.build returning nil") + func agreesWithPromptBuild() { let empties = [ TranscriptionContext(appName: nil, priorText: nil), TranscriptionContext(appName: " ", priorText: "\n"), ] for context in empties { #expect(context.isEmpty) - #expect(TranscriptionSteering.build(context: context) == .empty) + #expect(TranscriptionPrompt.build(context: context) == nil) } - // The two renderable signals (key terms, prior text) each steer a field… - let renderable = [ - TranscriptionContext(appName: nil, priorText: nil, keyTerms: ["Blurt"]), - TranscriptionContext(appName: nil, priorText: "Hi Sam,"), + let nonEmpties = [ + TranscriptionContext(appName: "Mail", priorText: nil), + TranscriptionContext(appName: nil, priorText: nil, selectedText: "selected"), ] - for context in renderable { + for context in nonEmpties { #expect(!context.isEmpty) - #expect(TranscriptionSteering.build(context: context) != .empty) - } - - // …while carry-only signals make the context non-empty (worth carrying for - // the injector) yet steer nothing. Two are load-bearing: selected text, - // because the paste replaces it so it is never priming; and everything - // naming the destination app, which is deliberately not sent at all. - let carryOnly = [ - TranscriptionContext(appName: "Mail", priorText: nil, selectedText: "sel"), - TranscriptionContext(appName: "Code", windowTitle: "main.py — blurt", priorText: nil), - TranscriptionContext(appName: "Slack", fieldLabel: "Message", priorText: nil), - ] - for context in carryOnly { - #expect(!context.isEmpty) - #expect(TranscriptionSteering.build(context: context) == .empty) + #expect(TranscriptionPrompt.build(context: context) != nil) } } diff --git a/Tests/BlurtEngineTests/TranscriptionPromptTests.swift b/Tests/BlurtEngineTests/TranscriptionPromptTests.swift new file mode 100644 index 0000000..260465a --- /dev/null +++ b/Tests/BlurtEngineTests/TranscriptionPromptTests.swift @@ -0,0 +1,147 @@ +import Testing + +@testable import BlurtEngine + +@Suite("TranscriptionPrompt") +struct TranscriptionPromptTests { + /// The standing plain-text exclusion clause that every built prompt carries + /// (see `TranscriptionPrompt.baseInstruction`). Kept here as the single source + /// of truth so the expectations below read clearly. + static let base = + "Transcribe without speaker labels, audio event descriptions, or emotion markers." + + /// One `build(context:)` → prompt expectation. Parameterizing these (rather + /// than a `@Test` apiece) keeps the whole context→prompt contract in one + /// readable table and gives per-case failure output. + struct Case: Sendable, CustomTestStringConvertible { + let name: String + let context: TranscriptionContext? + let expected: String? + var testDescription: String { name } + } + + static let cases: [Case] = [ + Case(name: "nil context → no prompt (server default)", context: nil, expected: nil), + Case( + name: "empty context → no prompt", + context: TranscriptionContext(appName: nil, priorText: nil), expected: nil), + Case( + name: "whitespace-only context → no prompt", + context: TranscriptionContext(appName: " ", priorText: "\n"), expected: nil), + Case( + name: "app only → destination sentence", + context: TranscriptionContext(appName: "Slack", priorText: nil), + expected: "Dictated into Slack. \(base)"), + Case( + name: "prior only → Previous transcript framing", + context: TranscriptionContext(appName: nil, priorText: "and then the build finished"), + expected: "Previous transcript:\nand then the build finished\n\n\(base)"), + Case( + name: "app + window → topic hint leads, destination trails", + context: TranscriptionContext(appName: "Mail", windowTitle: "Re: Q3 pricing", priorText: nil), + expected: "This is about \"Re: Q3 pricing\". Dictated into Mail. \(base)"), + Case( + name: "window only → bare topic hint", + context: TranscriptionContext(appName: nil, windowTitle: "Untitled.txt", priorText: nil), + expected: "This is about \"Untitled.txt\". \(base)"), + Case( + name: "field only → destination sentence", + context: TranscriptionContext(appName: nil, fieldLabel: "Search", priorText: nil), + expected: "Dictated in the \"Search\" field. \(base)"), + Case( + name: "app + field without window → destination names both", + context: TranscriptionContext(appName: "Slack", fieldLabel: "Message", priorText: nil), + expected: "Dictated into Slack, in the \"Message\" field. \(base)"), + Case( + name: "all four signals combine", + context: TranscriptionContext( + appName: "Slack", windowTitle: "#eng-backend", fieldLabel: "Message", priorText: "thanks for"), + expected: + "Previous transcript:\nthanks for\n\nThis is about \"#eng-backend\". Dictated into Slack, in the \"Message\" field. \(base)" + ), + Case( + name: "prior + app combine", + context: TranscriptionContext(appName: "Mail", priorText: "Dear Sam,"), + expected: "Previous transcript:\nDear Sam,\n\nDictated into Mail. \(base)"), + Case( + name: "prior + app are trimmed", + context: TranscriptionContext(appName: " Notes ", priorText: " hello "), + expected: "Previous transcript:\nhello\n\nDictated into Notes. \(base)"), + Case( + name: "selected only → Selected text framing", + context: TranscriptionContext(appName: nil, priorText: nil, selectedText: "the quarterly numbers"), + expected: "Selected text:\nthe quarterly numbers\n\n\(base)"), + Case( + name: "selected follows prior as its own block", + context: TranscriptionContext(appName: nil, priorText: "as we discussed,", selectedText: "the old plan"), + expected: "Previous transcript:\nas we discussed,\n\nSelected text:\nthe old plan\n\n\(base)"), + Case( + name: "selected + location + prior combine", + context: TranscriptionContext( + appName: "Mail", windowTitle: "Re: Q3 pricing", fieldLabel: "Body", + priorText: "Hi Sam,", selectedText: "let's push the date"), + expected: + "Previous transcript:\nHi Sam,\n\nSelected text:\nlet's push the date\n\nThis is about \"Re: Q3 pricing\". Dictated into Mail, in the \"Body\" field. \(base)" + ), + Case( + name: "blank selected adds no block", + context: TranscriptionContext(appName: "Notes", priorText: nil, selectedText: " \n"), + expected: "Dictated into Notes. \(base)"), + Case( + name: "selected sits between prior and keyword boost", + context: TranscriptionContext( + appName: "Slack", priorText: "thanks for", selectedText: "the draft", keyTerms: ["Blurt"]), + expected: + "Previous transcript:\nthanks for\n\nSelected text:\nthe draft\n\nDictated into Slack. \(base) Keywords: Blurt." + ), + Case( + name: "key terms only → inline keyword boost", + context: TranscriptionContext(appName: nil, priorText: nil, keyTerms: ["AssemblyAI", "Kubernetes"]), + expected: "\(base) Keywords: AssemblyAI, Kubernetes."), + Case( + name: "key terms trail base alongside focus context", + context: TranscriptionContext(appName: "Slack", priorText: nil, keyTerms: ["Blurt"]), + expected: "Dictated into Slack. \(base) Keywords: Blurt."), + Case( + name: "empty key terms add no clause", + context: TranscriptionContext(appName: "Notes", priorText: nil, keyTerms: []), + expected: "Dictated into Notes. \(base)"), + ] + + @Test("build maps focus context to the transcription prompt", arguments: cases) + func build(_ c: Case) { + #expect(TranscriptionPrompt.build(context: c.context) == c.expected) + } + + @Test("built prompt fits within the API's 4096-character cap for capped prior text") + func withinCap() { + let longPrior = String(repeating: "word ", count: 200) + let prompt = TranscriptionPrompt.build( + context: TranscriptionContext(appName: "Xcode", priorText: longPrior)) + #expect((prompt?.count ?? 0) <= TranscriptionPrompt.characterCap) + } + + @Test("the keyword clause is omitted entirely when not even the first term fits") + func keyTermsOmittedWhenNoneFit() { + // A single term longer than the whole cap leaves no budget for even one + // keyword: the clause (and its "Keywords:" scaffolding) must be dropped + // whole, not emitted empty or dangling. + let huge = String(repeating: "k", count: TranscriptionPrompt.characterCap) + let prompt = TranscriptionPrompt.build( + context: TranscriptionContext(appName: "Xcode", priorText: nil, keyTerms: [huge])) + #expect(prompt == "Dictated into Xcode. \(Self.base)") + } + + @Test("an oversized key-terms list is fitted to the cap, keeping whole leading terms") + func keyTermsFittedToCap() throws { + // Key terms are the one input with no upstream length cap; a huge Settings + // list must not push the prompt over the API cap (which fails the request). + let terms = (0..<2000).map { "term\($0)" } + let prompt = TranscriptionPrompt.build( + context: TranscriptionContext(appName: "Xcode", priorText: nil, keyTerms: terms)) + let built = try #require(prompt) + #expect(built.count <= TranscriptionPrompt.characterCap) + #expect(built.contains(" Keywords: term0, term1")) + #expect(built.hasSuffix(".")) + } +} diff --git a/Tests/BlurtEngineTests/TranscriptionSteeringTests.swift b/Tests/BlurtEngineTests/TranscriptionSteeringTests.swift deleted file mode 100644 index 850b4ce..0000000 --- a/Tests/BlurtEngineTests/TranscriptionSteeringTests.swift +++ /dev/null @@ -1,138 +0,0 @@ -import Testing - -@testable import BlurtEngine - -@Suite("TranscriptionSteering") -struct TranscriptionSteeringTests { - /// One `build(context:)` → steering-fields expectation. Parameterizing these - /// (rather than a `@Test` apiece) keeps the whole context→wire contract in one - /// readable table and gives per-case failure output. - /// - /// Each recognized signal has exactly one home, and the table pins both - /// directions — what renders and what must never leak into the wrong field: - /// prior-cursor text → `conversation_context`, key terms → `keyterms_prompt`. - /// Every other focus signal renders nowhere. That includes anything naming the - /// destination app: the table keeps a case per app family that once earned a - /// formatting clause, each pinned to `.empty`, so reintroducing app-kind - /// steering fails here. - struct Case: Sendable, CustomTestStringConvertible { - let name: String - let context: TranscriptionContext? - let expected: TranscriptionSteering.Fields - var testDescription: String { name } - } - - static let cases: [Case] = [ - Case(name: "nil context → nothing to steer with", context: nil, expected: .empty), - Case( - name: "empty context → nothing to steer with", - context: TranscriptionContext(appName: nil, priorText: nil), expected: .empty), - Case( - name: "whitespace-only context → nothing to steer with", - context: TranscriptionContext(appName: " ", priorText: "\n"), expected: .empty), - Case( - name: "app name and field label render nowhere", - context: TranscriptionContext( - appName: "Mail", windowTitle: "Re: Q3 pricing", fieldLabel: "Body", - priorText: nil, selectedText: "the old plan"), - expected: .empty), - Case( - name: "prior-cursor text becomes the single conversation-context turn", - context: TranscriptionContext(appName: "Mail", priorText: "Hi Sam, thanks for"), - expected: TranscriptionSteering.Fields( - conversationContext: ["Hi Sam, thanks for"], keyterms: [])), - Case( - name: "prior text is trimmed of surrounding whitespace", - context: TranscriptionContext(appName: nil, priorText: " Hi Sam,\n "), - expected: TranscriptionSteering.Fields(conversationContext: ["Hi Sam,"], keyterms: [])), - Case( - name: "key terms become keyterms_prompt, not a prompt clause", - context: TranscriptionContext(appName: nil, priorText: nil, keyTerms: ["AssemblyAI", "Kubernetes"]), - expected: TranscriptionSteering.Fields( - conversationContext: [], keyterms: ["AssemblyAI", "Kubernetes"])), - Case( - name: "a terminal steers nothing", - context: TranscriptionContext(appName: "Terminal", windowTitle: "zsh — 80×24", priorText: nil), - expected: .empty), - Case( - name: "a code editor steers nothing, not even the open file's language", - context: TranscriptionContext(appName: "Code", windowTitle: "main.py — blurt", priorText: nil), - expected: .empty), - Case( - name: "Slack steers nothing", - context: TranscriptionContext(appName: "Slack", fieldLabel: "Message", priorText: nil), - expected: .empty), - Case( - name: "Obsidian steers nothing", - context: TranscriptionContext( - appName: "Obsidian", windowTitle: "Grocery list - Cowork - Obsidian 1.12.7", - fieldLabel: "text entry area", priorText: nil), - expected: .empty), - Case( - name: "both fields populate independently, and the app still renders nowhere", - context: TranscriptionContext( - appName: "Terminal", windowTitle: "zsh — 80×24", - priorText: "$ git status", selectedText: "modified: README.md", keyTerms: ["Blurt"]), - expected: TranscriptionSteering.Fields( - conversationContext: ["$ git status"], keyterms: ["Blurt"])), - ] - - @Test("build maps focus context to the dictation steering fields", arguments: cases) - func build(_ c: Case) { - #expect(TranscriptionSteering.build(context: c.context) == c.expected) - } - - // MARK: - Selected text never becomes context - - @Test("selected text stays out of conversation context — the paste replaces it") - func selectedTextIsNotContext() { - // Selected text is about to be *overwritten* by the paste, so priming the - // model with it would condition the transcription on text that is on its way - // out. Only the text before the insertion point is real left-context. - let fields = TranscriptionSteering.build( - context: TranscriptionContext( - appName: nil, priorText: "keep this", selectedText: "REPLACED")) - #expect(fields.conversationContext == ["keep this"]) - } - - // MARK: - Documented caps - - @Test("an oversized key-terms list is fitted to the keyterms cap, keeping whole leading terms") - func keyTermsFittedToCap() { - // Key terms are the one unbounded input (a Settings list of any length), and - // the field's cap is the total across all terms — so a huge list must be cut - // to whole terms rather than pushing the request over the documented limit. - let terms = (0..<2000).map { "term\($0)" } - let fields = TranscriptionSteering.build( - context: TranscriptionContext(appName: nil, priorText: nil, keyTerms: terms)) - #expect(fields.keyterms.first == "term0") - #expect(fields.keyterms.count < terms.count) - #expect(fields.keyterms.reduce(0) { $0 + $1.count } <= TranscriptionSteering.keytermsCharacterCap) - } - - @Test("a single key term larger than the cap drops the field entirely") - func keyTermsOmittedWhenNoneFit() { - let huge = String(repeating: "k", count: TranscriptionSteering.keytermsCharacterCap + 1) - let fields = TranscriptionSteering.build( - context: TranscriptionContext(appName: nil, priorText: nil, keyTerms: [huge])) - #expect(fields.keyterms.isEmpty) - } - - @Test("over-long prior text is clipped to the cap, keeping the text nearest the cursor") - func priorTextClippedToCapKeepingTail() { - // The words immediately before the insertion point are the ones that carry - // continuity, so a clip must drop the *head* — the opposite of how the key - // terms list is fitted. (FocusCapture already caps prior text far below this; - // the guard is here so a hand-built or future-widened context can't exceed - // the field's documented limit.) - let long = - String(repeating: "a", count: 100) - + String(repeating: "b", count: TranscriptionSteering.conversationContextCharacterCap) - let fields = TranscriptionSteering.build( - context: TranscriptionContext(appName: nil, priorText: long)) - let turn = fields.conversationContext.first ?? "" - #expect(turn.count == TranscriptionSteering.conversationContextCharacterCap) - #expect(turn.hasSuffix("b")) - #expect(!turn.contains("a")) - } -}