diff --git a/.claude/skills/project-guardrails/SKILL.md b/.claude/skills/project-guardrails/SKILL.md index 98293f77..aa0b015e 100644 --- a/.claude/skills/project-guardrails/SKILL.md +++ b/.claude/skills/project-guardrails/SKILL.md @@ -83,9 +83,16 @@ genuinely correct, and reaching for it means it's time to stop and ask. entry the user opens and asks for a task, past dictations included. The third is the Rewrite Selection key's (`Rewrite/SelectionRewriter.swift`, through `GatewayConnection.send`): highlighted text rewritten as a spoken - instruction asks, and pasted over the selection — the one gateway output - that is pasted, owner-directed (2026-11-03). It never touches a dictation's - words; the instruction it is given is a hand-back press's, never pasted. + instruction asks, and pasted over the selection — owner-directed + (2026-11-03). It never touches a dictation's words; the instruction it is + given is a hand-back press's, never pasted. Its sibling, the Compose From + Instructions key's (`Rewrite/MessageComposer.swift`, also through + `GatewayConnection.send`), writes the text a hand-back press's spoken + instructions describe ("a Slack reply to Dan saying…") for the app in front, + pasted at the cursor — owner-directed (2026-11-06). Those two are the only + gateway outputs that are pasted. Neither is a door back to a cleanup pass: + the words they are given are instructions to work _from_, never the text + pasted tidied up. The fourth is the Control Your Mac key's (`Command/VoiceCommandPlanner.swift`, through `GatewayConnection.send`): a spoken command turned into steps from a closed list (`VoiceCommandStep`) that Blurt carries out itself — the one diff --git a/AGENTS.md b/AGENTS.md index 7592513c..7a792c97 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -88,7 +88,8 @@ Sources/BlurtEngine/ the engine (dependency-free Swift package) Permissions/ PermissionStatus + SetupReadiness (every platform); PermissionsChecker (mic + Accessibility), PermissionsReset, SigningIdentity (macOS only) - Rewrite/ SelectionRewriter (the Rewrite Selection key's Gemma rewrite) + Rewrite/ SelectionRewriter (the Rewrite Selection key's Gemma rewrite), + MessageComposer (Compose From Instructions' Gemma writing) Command/ the Control Your Mac key: VoiceCommandStep (the closed step list + its JSON parse, KeyChord, NameMatch, DestructiveGuard), VoiceCommandPlanner (the Gemma plan), SystemVoiceCommandPerformer (macOS only: NSWorkspace, @@ -423,34 +424,34 @@ a rule that outlives its row keeps firing and keeps citing this table while enfo project has already reversed. Reworded the row? Update the anchor. Reversed the decision? Delete the rule along with the row. -| Don't | Because | -| ------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Add an external SPM dependency to the engine | Dependency-free by rule (biggest supply-chain risk); a `check.sh` guard fails on `.package(` in `Package.swift` or a `url:`/`github:` package in `project.yml`. Extend `BlurtEngine` instead. | -| Use `AVAudioEngine` / `installTap` for capture | A long-lived engine bound its input graph to one device and went stale on a mic↔built-in switch — `-10868` (`kAudioUnitErr_FormatNotSupported`) or all-zero buffers. `MicCapture` builds a fresh `AVCaptureSession` recorder per capture (owner-directed move from `AVAudioRecorder`, 2026-08-25). | -| Pre-open the mic to shave bring-up latency (a warm/prepared recorder) | Measured against `kAudioDevicePropertyDeviceIsRunningSomewhere`: neither building an `AVCaptureSession` nor the retired `AVAudioRecorder.prepareToRecord()` opens the device, so neither pre-pays the 180–600 ms route activation `record()` costs. A warm-recorder lifecycle bought ~15 ms and cost a device-identity check, a pin check, an expiry and a bring-up flag. `MicCapture.warmUp()` is stateless: it absorbs the process's first-touch cost and holds nothing. | -| Add streaming STT | The dictation API returns the full (already rewritten) text in one response; the overlay shows "Transcribing…" then the full text. Dictation never streams. The one WebSocket is the main window's **Record** button (`Sources/AssemblyAI/Realtime/RealtimeTranscriber.swift`, the one file the invariant gate exempts): a meeting or call streamed to the realtime API so its words, partials included, appear as they're spoken. It never carries a dictation. Read-aloud's **TTS** is AssemblyAI's streaming HTTP API, a response body read as it arrives. | -| Call the dictation API in a way its docs don't describe | **The reference is the source of truth.** Every path, header, multipart part, `config` key, response key and error key Blurt sends or reads is one it documents. The route has been renamed repeatedly and still answers to names the docs have dropped — `prompt`, `keyterms`, `word_boost`, `language_code`, `llm: null` — so a 200 is not evidence a field is supported, only that it has not been removed yet; treat anything absent from the docs as deprecated. Undocumented behaviour carries no compatibility promise, so a feature resting on it can stop working silently: that is why `llm: null` (the only rewrite off switch there is) and the `message` error field were dropped on 2026-09-11, along with `warmUp()`'s GET at the bare host root in favour of the documented `GET /warm`. Running tighter than a documented cap is fine — `STTPrompt.characterCap` does. Measuring is still how a claim gets settled, and the docs have been wrong twice (see `DictationWireTypes` on unknown fields, `STTPrompt` on the 6000 cap), but a measurement licenses _distrusting_ a documented field, never _sending_ an undocumented one. | -| Keep a copy of the recording for anything but the over-cap fallback | `MicCapture.stop()` answers a byte count, not the audio (owner-directed, 2026-09-08): accumulating the blob in the capture lock cost a second `memcpy` of every byte and held a duplicate until release, and nothing read it back. The one copy that exists is the transcriber's (`relay`, in `AssemblyAITranscriber+LiveCapFallback.swift`), kept on the upload's task to back the documented `/v1/transcribe` fallback when the streaming route refuses a recording past its 120 s cap (`413 audio_too_large`): the 2026-09-08 rule's "no retry-from-buffer path", reversed for that one case on 2026-10-29. Don't read it back for a retry, a replay, a cache or a second request of any other kind, and don't move it into `MicCapture`. `check-invariants.sh` keeps the buffered route's path inside that one file. | -| Add a client-side LLM cleanup pass | Cleanup is the dictation API's server-side rewrite, requested by `config.llm_instruction` on the same `/v1/transcribe/live` call. No LLM Gateway client, no `StylerProtocol`, no styling stage, no second request — transcription steering belongs in `STTPrompt`. The one gateway client allowed is the read-aloud feature's, confined to `Sources/BlurtEngine/TTS/ReadAloudLLM.swift` (the invariant gate exempts that file and the file-transcript one below): the skip-jargon listening rewrite of a selection, and the spoken answer to a request about one, neither of which the TTS API has a server-side equivalent for. It never cleans up a dictation; the only transcript it sees is a hand-back press's request (`.handedBack`), which is never pasted. The second is file transcription's (`Sources/BlurtEngine/STT/TranscriptTransform.swift`, also exempt): stock transforms — summary, key points, to-do list and the like — over a saved transcript, which the async API has no rewrite for and which can run past the dictation request's limits. It is never in the dictation path and nothing it writes is pasted: it runs only on a saved history entry the user opens and asks for a task — a recording, a file, a read-aloud clip, or a past dictation — after the fact. The third is the Rewrite Selection key's (`Sources/BlurtEngine/Rewrite/SelectionRewriter.swift`, on Gemma through `GatewayConnection.send`, so the gateway URL stays in one file): it rewrites text the user highlighted as a spoken instruction asks, and is the one client whose output is pasted — over that selection, owner-directed (2026-11-03). Its only transcript is the instruction (`.handedBack`), never pasted itself; it is not a door back to a cleanup pass. The fourth is the Control Your Mac key's (`Sources/BlurtEngine/Command/VoiceCommandPlanner.swift`, Gemma through `GatewayConnection.send` too): it turns a spoken command (`.handedBack`) and a snapshot of the screen read from the Accessibility tree — never a screenshot (`ScreenContext`: the frontmost app, its window's title, and up to 150 numbered actionable elements by label, never a field's contents or a password field) — into steps from a closed list (`VoiceCommandStep`: open, hide or quit an app, open an `http`/`https`/`mailto` link, press a shortcut, type, a media key, press a named button or choose a menu item through Accessibility, press, focus, select, adjust or open the context menu of an element the snapshot numbered, scroll, and move, resize, minimize or full-screen the frontmost window) that Blurt carries out itself (`SystemVoiceCommandPerformer`) — the one gateway output acted on, owner-directed (2026-11-03). Every gateway client binds its key and transport the same way: a `GatewayConnection`, with `GatewayClient`'s one shared `init(apiKeyProvider:transport:)` (`ReadAloudLLM.swift`, where the gateway URL stays). The reply is data, never script: don't widen it to AppleScript, a shell, or any URL scheme, and its typed step still pastes through the session's injector. The destructive-action ban is code as well as prompt (`DestructiveGuard`: ⌘+delete chords, ⌘⇧Q and ⌥⌘⇧Q (Log Out), and buttons or menu items that delete, trash, erase, shut down, restart or log out). | -| Add local models or model downloads | Transcription is a remote AssemblyAI call: no on-device ASR/LLM, no model cache, no download UI. | -| Pin transcription to English, or set a language at all | Hurt non-English transcription; language is left to the model's own detection. **No `config.language_codes`** either — note the plural, which is the documented spelling, though the undocumented singular `language_code` is accepted too (a 200, not the unknown-key 400 a bogus name earns), so both are live and both are pinned absent. Detection was measured to work with neither it nor a prompt set (es/fr/de/ja clips each transcribed in their own language against the live endpoint, rewrite included), and re-measured 2026-09-11: a Spanish clip came back as Spanish with no field, with `language_codes: ["es"]`, **and** with `language_codes: ["en"]` — asking for English did not force English, so the reference's `["en"]` default does not describe this route and omission is not a bet on a default. Setting one would only take working detection away. `KeytermsWireTests` asserts the absence of both spellings. | -| Send `config.prompt` alongside `config.stt_prompt` | They are the same field under two names — the docs give `stt_prompt` as the field and note it is "Also accepted as `prompt`" — and a request carrying both is rejected before the audio is read (`provide only one of stt_prompt or prompt; they are the same field`). So this is never a compatible addition: it is a 400 on _every_ dictation. `stt_prompt` is the documented spelling and the only one to send; `prompt` is the legacy one, and the route's own inner validator still answers under it (see `STTPrompt.characterCap`), which is a reason to treat it as deprecated rather than interchangeable. | -| Bring back `config.conversation_context` | Replaced by `config.stt_prompt` (`STTPrompt`), which carries the same two signals as one string. Both fields still work and can ride the same request, so re-adding the turn list sends the prior text twice rather than failing. `prompt` is `stt_prompt`'s own alias — a request carrying both is a 400 (`they are the same field`) — and a custom prompt replaces the service's managed default, which is the trade this swap accepted. | -| Widen the request's context past history + the prior chunk | `STTPrompt.text` reads exactly two fields of `TranscriptionContext` — `recentTranscripts` and `priorText`. The app name, window title, field label and selected text are captured for the paste path and the developer-mode log and stay on the machine; the app/window/field hints and the `Selected text:` block were removed, not gated. Don't add one back, and don't route that context onto the request by another path. The user's key terms are the exception that proves the rule: they _are_ sent, as the request's own `keyterms_prompt` list (`KeyTerms`) — never folded back into the prompt. | -| Add a "remove filler words (um, uh, like)" clause | Not in the STT model's trained instruction set — a no-op, deliberately dropped; disfluency removal is the server-side LLM rewrite's job. | -| Add a keystroke-typing paste path or a length threshold | Injection is **always** clipboard paste (save → write → ⌘V → settle → restore), with the copied-to-clipboard degradation when the target is lost. | -| Add `LSUIElement` or a menu-bar-**only** mode | Blurt is a Dock app first. The `MenuBarExtra` status item is convenience layered on the Dock icon; the notch can hide a status item, so nothing may depend on it. A menu-bar-only variant was reverted twice. | -| Drop or hide pasting an API key in favour of Sign in with AssemblyAI | Both stay. A team rolling Blurt out shares one key rather than adding every member, engineer or not, to its AssemblyAI account, and an earlier build that hid the key row from new installs sent such teams back to old releases. `CredentialRows.resolve` offers a key whenever the user isn't signed in: "Use API Key…" beside the filled Sign In… in the `.choice` row, or the key row once one is stored. | -| Add a `KeyboardShortcuts` package or a key+modifier chord | The trigger is a single lone modifier, home-grown (`CGEventTap` + `DictationKeyGate`), and swallows nothing. | -| Hand-roll an updater beside Sparkle, or start it in a dev build | Updates go through Sparkle (`UpdaterModel`, app-side; EdDSA-signed appcast published per release). The download-only GitHub-API checker it replaced and the earlier `mxcl/AppUpdater` are gone — don't add a second update path, and keep update code out of the engine. Only the shipping bundle id starts the updater: a dev build must never replace itself with the release. | -| Hand-edit `Blurt.xcodeproj/project.pbxproj` | Generated from `project.yml`; `check.sh`'s drift check fails on any manual edit (a Claude PreToolUse hook also blocks it). | -| Redirect the post-build install away from `/Applications` | TCC won't register apps in DerivedData/`/tmp`, so permission toggles never appear. | -| Touch the real Keychain in tests | `APIKeyStore` is the production item — a test that writes it triggers Keychain prompts and corrupts the real item's ACL. Use an isolated service (see `KeychainStoreTests`) or `InMemoryAPIKeyStore`. | -| Add backwards-compat shims for removed types | Deleted types stay deleted — no deprecated re-exports. | -| Give the keyboard extension a microphone or any audio type | iOS lets no keyboard use the microphone, and the design keeps every audio type out of it: the app listens and transcribes over its always-on capture, the keyboard is a remote control that inserts the words (`KeyboardRelayInjector`). An `AVFoundation` import in `App/BlurtiOS/BlurtKeyboard`, `BlurtDesign` or `BlurtiOSCore` (which the keyboard links) is that decision being reversed by accident. | -| Move the iPhone's logic back into its UI targets, or views into `BlurtiOSCore` | The iPhone code splits the way the Mac's does: `App/BlurtiOS/BlurtiOSCore` is the logic (the App Group contract, `KeyboardModel`, `VoiceState`, `KeyboardInteraction`, term packs, the relay injector) — no SwiftUI, no views, no audio, `package` access, a static framework both targets link — and it alone carries the 88% coverage gate (`scripts/ios-test.sh`), as the engine alone does on the Mac. Logic in a view file escapes that gate; SwiftUI in the core drags the views' colours and tokens into a module the keyboard's memory budget pays for. A UI hook the model needs becomes a shell-side extension (`KeyboardModel+Palette.swift`). | -| Deep-link into Settings › General › Keyboard (`App-Prefs:`, `prefs:`) | Private URLs: App Review rejects them (2.5.1) and they break between iOS releases. `UIApplication.openSettingsURLString` — Blurt's own page, whose Keyboards row iOS adds for any app with a keyboard — is the only Settings link; the setup step names the whole path (Apps › Blurt › Keyboards) because some builds (the iOS 26 simulator) open Settings at its top level instead. | -| Bundle the brand's fonts in the iPhone app | Modern Gothic Mono, UN 11ST and Oceanic Text were bundled and removed: their licence covers the web, not an app in a public MIT repo. `BlurtType`'s roles (`mono`, `heading`, `body`) map to the system's monospaced, serif and default faces; a face comes back only with written app-embedding rights, through the same roles. | +| Don't | Because | +| ------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Add an external SPM dependency to the engine | Dependency-free by rule (biggest supply-chain risk); a `check.sh` guard fails on `.package(` in `Package.swift` or a `url:`/`github:` package in `project.yml`. Extend `BlurtEngine` instead. | +| Use `AVAudioEngine` / `installTap` for capture | A long-lived engine bound its input graph to one device and went stale on a mic↔built-in switch — `-10868` (`kAudioUnitErr_FormatNotSupported`) or all-zero buffers. `MicCapture` builds a fresh `AVCaptureSession` recorder per capture (owner-directed move from `AVAudioRecorder`, 2026-08-25). | +| Pre-open the mic to shave bring-up latency (a warm/prepared recorder) | Measured against `kAudioDevicePropertyDeviceIsRunningSomewhere`: neither building an `AVCaptureSession` nor the retired `AVAudioRecorder.prepareToRecord()` opens the device, so neither pre-pays the 180–600 ms route activation `record()` costs. A warm-recorder lifecycle bought ~15 ms and cost a device-identity check, a pin check, an expiry and a bring-up flag. `MicCapture.warmUp()` is stateless: it absorbs the process's first-touch cost and holds nothing. | +| Add streaming STT | The dictation API returns the full (already rewritten) text in one response; the overlay shows "Transcribing…" then the full text. Dictation never streams. The one WebSocket is the main window's **Record** button (`Sources/AssemblyAI/Realtime/RealtimeTranscriber.swift`, the one file the invariant gate exempts): a meeting or call streamed to the realtime API so its words, partials included, appear as they're spoken. It never carries a dictation. Read-aloud's **TTS** is AssemblyAI's streaming HTTP API, a response body read as it arrives. | +| Call the dictation API in a way its docs don't describe | **The reference is the source of truth.** Every path, header, multipart part, `config` key, response key and error key Blurt sends or reads is one it documents. The route has been renamed repeatedly and still answers to names the docs have dropped — `prompt`, `keyterms`, `word_boost`, `language_code`, `llm: null` — so a 200 is not evidence a field is supported, only that it has not been removed yet; treat anything absent from the docs as deprecated. Undocumented behaviour carries no compatibility promise, so a feature resting on it can stop working silently: that is why `llm: null` (the only rewrite off switch there is) and the `message` error field were dropped on 2026-09-11, along with `warmUp()`'s GET at the bare host root in favour of the documented `GET /warm`. Running tighter than a documented cap is fine — `STTPrompt.characterCap` does. Measuring is still how a claim gets settled, and the docs have been wrong twice (see `DictationWireTypes` on unknown fields, `STTPrompt` on the 6000 cap), but a measurement licenses _distrusting_ a documented field, never _sending_ an undocumented one. | +| Keep a copy of the recording for anything but the over-cap fallback | `MicCapture.stop()` answers a byte count, not the audio (owner-directed, 2026-09-08): accumulating the blob in the capture lock cost a second `memcpy` of every byte and held a duplicate until release, and nothing read it back. The one copy that exists is the transcriber's (`relay`, in `AssemblyAITranscriber+LiveCapFallback.swift`), kept on the upload's task to back the documented `/v1/transcribe` fallback when the streaming route refuses a recording past its 120 s cap (`413 audio_too_large`): the 2026-09-08 rule's "no retry-from-buffer path", reversed for that one case on 2026-10-29. Don't read it back for a retry, a replay, a cache or a second request of any other kind, and don't move it into `MicCapture`. `check-invariants.sh` keeps the buffered route's path inside that one file. | +| Add a client-side LLM cleanup pass | Cleanup is the dictation API's server-side rewrite, requested by `config.llm_instruction` on the same `/v1/transcribe/live` call. No LLM Gateway client, no `StylerProtocol`, no styling stage, no second request — transcription steering belongs in `STTPrompt`. The one gateway client allowed is the read-aloud feature's, confined to `Sources/BlurtEngine/TTS/ReadAloudLLM.swift` (the invariant gate exempts that file and the file-transcript one below): the skip-jargon listening rewrite of a selection, and the spoken answer to a request about one, neither of which the TTS API has a server-side equivalent for. It never cleans up a dictation; the only transcript it sees is a hand-back press's request (`.handedBack`), which is never pasted. The second is file transcription's (`Sources/BlurtEngine/STT/TranscriptTransform.swift`, also exempt): stock transforms — summary, key points, to-do list and the like — over a saved transcript, which the async API has no rewrite for and which can run past the dictation request's limits. It is never in the dictation path and nothing it writes is pasted: it runs only on a saved history entry the user opens and asks for a task — a recording, a file, a read-aloud clip, or a past dictation — after the fact. The third is the Rewrite Selection key's (`Sources/BlurtEngine/Rewrite/SelectionRewriter.swift`, on Gemma through `GatewayConnection.send`, so the gateway URL stays in one file): it rewrites text the user highlighted as a spoken instruction asks, pasted over that selection, owner-directed (2026-11-03). Its only transcript is the instruction (`.handedBack`), never pasted itself; it is not a door back to a cleanup pass. Its sibling is the Compose From Instructions key's (`Sources/BlurtEngine/Rewrite/MessageComposer.swift`, Gemma through `GatewayConnection.send` too): it writes the text a hand-back press's spoken instructions describe, for the app in front (its name only), pasted at the cursor, owner-directed (2026-11-06). Those two are the only clients whose output is pasted, and neither is a cleanup pass: the transcript each gets is instructions to work from, never the text that is pasted. The fourth is the Control Your Mac key's (`Sources/BlurtEngine/Command/VoiceCommandPlanner.swift`, Gemma through `GatewayConnection.send` too): it turns a spoken command (`.handedBack`) and a snapshot of the screen read from the Accessibility tree — never a screenshot (`ScreenContext`: the frontmost app, its window's title, and up to 150 numbered actionable elements by label, never a field's contents or a password field) — into steps from a closed list (`VoiceCommandStep`: open, hide or quit an app, open an `http`/`https`/`mailto` link, press a shortcut, type, a media key, press a named button or choose a menu item through Accessibility, press, focus, select, adjust or open the context menu of an element the snapshot numbered, scroll, and move, resize, minimize or full-screen the frontmost window) that Blurt carries out itself (`SystemVoiceCommandPerformer`) — the one gateway output acted on, owner-directed (2026-11-03). Every gateway client binds its key and transport the same way: a `GatewayConnection`, with `GatewayClient`'s one shared `init(apiKeyProvider:transport:)` (`ReadAloudLLM.swift`, where the gateway URL stays). The reply is data, never script: don't widen it to AppleScript, a shell, or any URL scheme, and its typed step still pastes through the session's injector. The destructive-action ban is code as well as prompt (`DestructiveGuard`: ⌘+delete chords, ⌘⇧Q and ⌥⌘⇧Q (Log Out), and buttons or menu items that delete, trash, erase, shut down, restart or log out). | +| Add local models or model downloads | Transcription is a remote AssemblyAI call: no on-device ASR/LLM, no model cache, no download UI. | +| Pin transcription to English, or set a language at all | Hurt non-English transcription; language is left to the model's own detection. **No `config.language_codes`** either — note the plural, which is the documented spelling, though the undocumented singular `language_code` is accepted too (a 200, not the unknown-key 400 a bogus name earns), so both are live and both are pinned absent. Detection was measured to work with neither it nor a prompt set (es/fr/de/ja clips each transcribed in their own language against the live endpoint, rewrite included), and re-measured 2026-09-11: a Spanish clip came back as Spanish with no field, with `language_codes: ["es"]`, **and** with `language_codes: ["en"]` — asking for English did not force English, so the reference's `["en"]` default does not describe this route and omission is not a bet on a default. Setting one would only take working detection away. `KeytermsWireTests` asserts the absence of both spellings. | +| Send `config.prompt` alongside `config.stt_prompt` | They are the same field under two names — the docs give `stt_prompt` as the field and note it is "Also accepted as `prompt`" — and a request carrying both is rejected before the audio is read (`provide only one of stt_prompt or prompt; they are the same field`). So this is never a compatible addition: it is a 400 on _every_ dictation. `stt_prompt` is the documented spelling and the only one to send; `prompt` is the legacy one, and the route's own inner validator still answers under it (see `STTPrompt.characterCap`), which is a reason to treat it as deprecated rather than interchangeable. | +| Bring back `config.conversation_context` | Replaced by `config.stt_prompt` (`STTPrompt`), which carries the same two signals as one string. Both fields still work and can ride the same request, so re-adding the turn list sends the prior text twice rather than failing. `prompt` is `stt_prompt`'s own alias — a request carrying both is a 400 (`they are the same field`) — and a custom prompt replaces the service's managed default, which is the trade this swap accepted. | +| Widen the request's context past history + the prior chunk | `STTPrompt.text` reads exactly two fields of `TranscriptionContext` — `recentTranscripts` and `priorText`. The app name, window title, field label and selected text are captured for the paste path and the developer-mode log and stay on the machine; the app/window/field hints and the `Selected text:` block were removed, not gated. Don't add one back, and don't route that context onto the request by another path. The user's key terms are the exception that proves the rule: they _are_ sent, as the request's own `keyterms_prompt` list (`KeyTerms`) — never folded back into the prompt. | +| Add a "remove filler words (um, uh, like)" clause | Not in the STT model's trained instruction set — a no-op, deliberately dropped; disfluency removal is the server-side LLM rewrite's job. | +| Add a keystroke-typing paste path or a length threshold | Injection is **always** clipboard paste (save → write → ⌘V → settle → restore), with the copied-to-clipboard degradation when the target is lost. | +| Add `LSUIElement` or a menu-bar-**only** mode | Blurt is a Dock app first. The `MenuBarExtra` status item is convenience layered on the Dock icon; the notch can hide a status item, so nothing may depend on it. A menu-bar-only variant was reverted twice. | +| Drop or hide pasting an API key in favour of Sign in with AssemblyAI | Both stay. A team rolling Blurt out shares one key rather than adding every member, engineer or not, to its AssemblyAI account, and an earlier build that hid the key row from new installs sent such teams back to old releases. `CredentialRows.resolve` offers a key whenever the user isn't signed in: "Use API Key…" beside the filled Sign In… in the `.choice` row, or the key row once one is stored. | +| Add a `KeyboardShortcuts` package or a key+modifier chord | The trigger is a single lone modifier, home-grown (`CGEventTap` + `DictationKeyGate`), and swallows nothing. | +| Hand-roll an updater beside Sparkle, or start it in a dev build | Updates go through Sparkle (`UpdaterModel`, app-side; EdDSA-signed appcast published per release). The download-only GitHub-API checker it replaced and the earlier `mxcl/AppUpdater` are gone — don't add a second update path, and keep update code out of the engine. Only the shipping bundle id starts the updater: a dev build must never replace itself with the release. | +| Hand-edit `Blurt.xcodeproj/project.pbxproj` | Generated from `project.yml`; `check.sh`'s drift check fails on any manual edit (a Claude PreToolUse hook also blocks it). | +| Redirect the post-build install away from `/Applications` | TCC won't register apps in DerivedData/`/tmp`, so permission toggles never appear. | +| Touch the real Keychain in tests | `APIKeyStore` is the production item — a test that writes it triggers Keychain prompts and corrupts the real item's ACL. Use an isolated service (see `KeychainStoreTests`) or `InMemoryAPIKeyStore`. | +| Add backwards-compat shims for removed types | Deleted types stay deleted — no deprecated re-exports. | +| Give the keyboard extension a microphone or any audio type | iOS lets no keyboard use the microphone, and the design keeps every audio type out of it: the app listens and transcribes over its always-on capture, the keyboard is a remote control that inserts the words (`KeyboardRelayInjector`). An `AVFoundation` import in `App/BlurtiOS/BlurtKeyboard`, `BlurtDesign` or `BlurtiOSCore` (which the keyboard links) is that decision being reversed by accident. | +| Move the iPhone's logic back into its UI targets, or views into `BlurtiOSCore` | The iPhone code splits the way the Mac's does: `App/BlurtiOS/BlurtiOSCore` is the logic (the App Group contract, `KeyboardModel`, `VoiceState`, `KeyboardInteraction`, term packs, the relay injector) — no SwiftUI, no views, no audio, `package` access, a static framework both targets link — and it alone carries the 88% coverage gate (`scripts/ios-test.sh`), as the engine alone does on the Mac. Logic in a view file escapes that gate; SwiftUI in the core drags the views' colours and tokens into a module the keyboard's memory budget pays for. A UI hook the model needs becomes a shell-side extension (`KeyboardModel+Palette.swift`). | +| Deep-link into Settings › General › Keyboard (`App-Prefs:`, `prefs:`) | Private URLs: App Review rejects them (2.5.1) and they break between iOS releases. `UIApplication.openSettingsURLString` — Blurt's own page, whose Keyboards row iOS adds for any app with a keyboard — is the only Settings link; the setup step names the whole path (Apps › Blurt › Keyboards) because some builds (the iOS 26 simulator) open Settings at its top level instead. | +| Bundle the brand's fonts in the iPhone app | Modern Gothic Mono, UN 11ST and Oceanic Text were bundled and removed: their licence covers the web, not an app in a public MIT repo. `BlurtType`'s roles (`mono`, `heading`, `body`) map to the system's monospaced, serif and default faces; a face comes back only with written app-embedding rights, through the same roles. | **Platform scope.** Two rows above are the Mac shell's, not the iPhone's, and the iPhone app diverges from them on purpose: "no warm recorder" — on iOS the app keeps the microphone open for a listening window (`App/BlurtiOS/BlurtiOS/Sources/AudioSource.swift`), because iOS refuses to open it from the background and the keyboard, which is where a dictation starts, is in the background of whatever app the user is typing in; and "injection is always a clipboard paste" — on iOS the keyboard inserts through its `textDocumentProxy`, and the clipboard is only the degradation when no keyboard is there to take the words. Neither changes the engine: `MicCapture` and `KeyInjector` stay as they are and are macOS-only; the iOS app supplies its own `MicCaptureProtocol` and `InjectorProtocol` conformances. `App/BlurtiOS/DESIGN.md` is the iPhone's own list. @@ -892,7 +893,7 @@ lands), and `keyTap.refreshBinding()` after the user picks a different trigger k ## Hotkey The triggers are **lone modifier keys**, each set to dictate, read the selection aloud, ask about it, -rewrite it, or nothing, and each tapped, held, or either (see Keys below) — implemented in-house. Four pieces, +rewrite it, write from spoken instructions, or nothing, and each tapped, held, or either (see Keys below) — implemented in-house. Four pieces, three of them pure engine logic: - **`TriggerKey`** (`Hotkey/TriggerKey.swift`) — enum of the curated lone momentary modifiers usable @@ -961,7 +962,7 @@ through `@AppStorage` and builds the snapshot from them. The unset defaults belo **Keys.** Settings' Keys tab, after General (`KeysSettingsTab`), has one row per action (`TriggerKeyChoice` minus Off) — dictate (with the main window's style, no custom style, or each profile; the styles disabled while enhanced transcripts are off), Read Selection Aloud, Ask About -Selection, Rewrite Selection, Control Your Mac — each with a pop-up choosing its key or None +Selection, Rewrite Selection, Compose From Instructions, Control Your Mac — each with a pop-up choosing its key or None (`TriggerKeyStyleStore.assign(_:to:)`: a key does one thing, so choosing one in use moves it, and the Dictate row's key becomes the bound key), then one Activation pop-up for every key. Each key's start/stop cue voice is the Sounds tab's, after Keys (`SoundsSettingsTab`): one row per key that isn't Off. `TriggerKeySoundStore` @@ -1031,6 +1032,21 @@ it stops a read. A selection past `SelectionRewriter.maxCharacters` (20,000) fai rewriting a part, since the paste replaces all of it, and a reply cut off at the token cap is never pasted. No key rewrites by default; rewrites aren't kept in the history. +**Compose** is rewrite with no selection: a compose key's press skips the Accessibility read, as a +command's does (`.selected` from the key-down, still proving itself by a tap or a hold before the +mic opens), then records spoken instructions — "ok, a Slack reply to Dan: we should have a feature +to just speak instructions, keep it casual" — with `pressHandingBack`. The transcript handed back +goes to `MessageComposing` (`MessageComposer` in production: one chat completion on Gemma, the +rewrite's model) with the name of the app the hand-back press pinned as its paste target +(`DictationSession.pinnedAppName` — not whatever is in front once it's transcribed, so switching +apps meanwhile can't get an email pasted into Slack; never the window title or any field's text), so the instruction can fit the writing to it — a chat message for Slack, an email +for Mail. It is pasted the way a rewrite is (`replaceSelection(with:)`: the hand-back press's +target, no separator, over whatever is selected, else at the cursor), wears the pill as +`OverlayUIState.composing` ("Writing…") and then the same notices, and shares the router's job +lifecycle, so any press abandons it before it pastes. A reply cut off at the token cap +(`MessageComposer.maxTokens`) is never pasted. No key composes by default; compositions aren't kept +in the history. + **Control Your Mac** rides the same hand-back, with no selection: a command key's press skips the Accessibility read (it is `.selected` from the key-down) but still proves itself by a tap or a hold before the mic opens, then records a spoken command with `pressHandingBack`. The transcript and a @@ -1144,7 +1160,7 @@ item that deletes, trashes, erases, shuts down, restarts or logs out once matche prompt's ban, in code. A `type` step is pasted by `DictationSession.typeText(_:)` through `InjectorProtocol.insertAtFocus`, which pastes where focus is now — an earlier step may have brought another app forward — without touching the press's pinned -target, so it can't race the next press pinning its own. Rewrite and command share one job lifecycle +target, so it can't race the next press pinning its own. Rewrite, compose and command share one job lifecycle in the router (`job`, `jobPill`, `.stopJob`, `Mode.working`, `processingState`), and the state machine itself emits `.prepareCommand` the moment a command press proves itself. Every Accessibility action on an element goes through `performAction(_:on:)`, which runs the destructive guard first. The pill reads `OverlayUIState.commanding` ("Working…") diff --git a/App/Blurt/Blurt/AppCoordinator.swift b/App/Blurt/Blurt/AppCoordinator.swift index 322d25fb..871304fa 100644 --- a/App/Blurt/Blurt/AppCoordinator.swift +++ b/App/Blurt/Blurt/AppCoordinator.swift @@ -165,7 +165,7 @@ final class AppCoordinator { let router = triggerKeySeams.map { SelectionSpeechRouter( - session: session, speaker: $0.speaker, rewriter: $0.rewriter, planner: $0.planner, + session: session, speaker: $0.speaker, rewriter: $0.rewriter, composer: $0.composer, planner: $0.planner, performer: $0.performer, readSelection: $0.readSelection) } ?? SelectionSpeechRouter( diff --git a/App/Blurt/Blurt/AppDelegate.swift b/App/Blurt/Blurt/AppDelegate.swift index 114ac3d1..3ed8ed5d 100644 --- a/App/Blurt/Blurt/AppDelegate.swift +++ b/App/Blurt/Blurt/AppDelegate.swift @@ -122,8 +122,9 @@ final class AppDelegate: NSObject, NSApplicationDelegate { triggerKeys: TriggerKeySeams( speaker: UITestReadAloudSpeaker(), readSelection: { [selection = UITestIdentifiers.cannedSelection] in selection }, - // Rewrite and command keys offline too: no gateway, nothing acted on. - rewriter: UITestRewriter(), planner: UITestPlanner(), performer: UITestPerformer()), + // Rewrite, compose and command keys offline too: no gateway, nothing acted on. + rewriter: UITestRewriter(), composer: UITestComposer(), planner: UITestPlanner(), + performer: UITestPerformer()), // A fresh history per run — never the user's saved transcripts — // seeded with known entries when the suite asks, and the canned // transport behind transcription and the LLM. diff --git a/App/Blurt/Blurt/Overlay/OverlayView.swift b/App/Blurt/Blurt/Overlay/OverlayView.swift index fba2c644..545441eb 100644 --- a/App/Blurt/Blurt/Overlay/OverlayView.swift +++ b/App/Blurt/Blurt/Overlay/OverlayView.swift @@ -126,6 +126,9 @@ struct OverlayView: View { case .rewriting: waitingContent("Rewriting") .transition(.opacity) + case .composing: + waitingContent("Writing") + .transition(.opacity) case .commanding: waitingContent("Working") .transition(.opacity) diff --git a/App/Blurt/Blurt/SoundPackCatalog+ReadAloud.swift b/App/Blurt/Blurt/SoundPackCatalog+ReadAloud.swift index 1a76ca1b..f4cfab8a 100644 --- a/App/Blurt/Blurt/SoundPackCatalog+ReadAloud.swift +++ b/App/Blurt/Blurt/SoundPackCatalog+ReadAloud.swift @@ -19,12 +19,12 @@ extension SoundPackStore { /// The setting a key with no voice of its own follows: its action's. A read /// key sounds like a read and a command key like a command, and any other - /// key — dictate, ask, rewrite, each a recording — like a dictation. + /// key — dictate, ask, rewrite, compose, each a recording — like a dictation. static func fallback(for action: TriggerKeyAction) -> SoundPackStore { switch action { case .readAloud: .readAloud case .command: .command - case .off, .dictate, .ask, .rewrite: .dictation + case .off, .dictate, .ask, .rewrite, .compose: .dictation } } } diff --git a/App/Blurt/Blurt/TriggerKeySeams.swift b/App/Blurt/Blurt/TriggerKeySeams.swift index d54388cb..b6d3730f 100644 --- a/App/Blurt/Blurt/TriggerKeySeams.swift +++ b/App/Blurt/Blurt/TriggerKeySeams.swift @@ -1,13 +1,15 @@ import BlurtEngine /// Stand-ins for the trigger keys' own jobs — read-aloud's speaker and -/// selection read, Rewrite Selection's rewriter, Control Your Mac's planner +/// selection read, Rewrite Selection's rewriter, Compose From Instructions' +/// composer, Control Your Mac's planner /// and performer — so a host can run them with no network, audio or /// Accessibility (Blurt's UI tests). struct TriggerKeySeams { let speaker: any ReadAloudSpeaker let readSelection: @Sendable () async -> String? let rewriter: any SelectionRewriting + let composer: any MessageComposing let planner: any VoiceCommandPlanning let performer: any VoiceCommandPerforming } diff --git a/App/Blurt/Blurt/UITestCommandDoubles.swift b/App/Blurt/Blurt/UITestCommandDoubles.swift index b92e7c31..7ba1f7ff 100644 --- a/App/Blurt/Blurt/UITestCommandDoubles.swift +++ b/App/Blurt/Blurt/UITestCommandDoubles.swift @@ -2,7 +2,7 @@ import BlurtEngine import Foundation - // The rewrite and command keys' stand-ins under UI testing, beside the + // The rewrite, compose and command keys' stand-ins under UI testing, beside the // pipeline doubles in UITestSupport.swift: no gateway, nothing acted on. /// Stub rewriter: the rewrite a spoken instruction asks for, as a marked @@ -13,6 +13,14 @@ } } + /// Stub composer: the instructions as a marked echo, so a test sees them + /// reach it and the result pasted. + nonisolated struct UITestComposer: MessageComposing { + func compose(_ instructions: String, for app: String?) async throws -> String { + "Composed (\(instructions))" + } + } + /// Stub planner: a few canned commands, each a fixed plan, so the router's /// whole command path runs with no gateway. "Open X" opens X; "type X" /// types it; "mute" mutes; anything else is a command Blurt can't do. diff --git a/App/Blurt/Blurt/UITestSupport.swift b/App/Blurt/Blurt/UITestSupport.swift index aa032631..546a5fac 100644 --- a/App/Blurt/Blurt/UITestSupport.swift +++ b/App/Blurt/Blurt/UITestSupport.swift @@ -290,6 +290,8 @@ .accessibilityIdentifier(UITestIdentifiers.harnessSetCommand) Button("Rewrite") { coordinator?.setKeyChoiceForTesting(.rewrite, for: state.selectedKey) } .accessibilityIdentifier(UITestIdentifiers.harnessSetRewrite) + Button("Compose") { coordinator?.setKeyChoiceForTesting(.compose, for: state.selectedKey) } + .accessibilityIdentifier(UITestIdentifiers.harnessSetCompose) Button("Tap Mode") { coordinator?.setActivationForTesting(.tap) } .accessibilityIdentifier(UITestIdentifiers.harnessSetTapMode) } diff --git a/App/Blurt/Blurt/Wizard/KeyCap.swift b/App/Blurt/Blurt/Wizard/KeyCap.swift index 052d2d49..bf8e6da3 100644 --- a/App/Blurt/Blurt/Wizard/KeyCap.swift +++ b/App/Blurt/Blurt/Wizard/KeyCap.swift @@ -43,6 +43,7 @@ extension TriggerKeyAction { case .readAloud: LibraryKind.readAloud.systemImage case .ask: LibraryKind.ask.systemImage case .rewrite: "pencil.line" + case .compose: "square.and.pencil" case .command: "command" case .dictate, .off: "keyboard" } diff --git a/App/Blurt/Blurt/Wizard/KeysSettingsTab.swift b/App/Blurt/Blurt/Wizard/KeysSettingsTab.swift index 35d1dac4..dbffb9e3 100644 --- a/App/Blurt/Blurt/Wizard/KeysSettingsTab.swift +++ b/App/Blurt/Blurt/Wizard/KeysSettingsTab.swift @@ -62,6 +62,7 @@ struct KeysSettingsTab: View { ActionRow(choice: .readAloud, title: KeyActionName.readAloud), ActionRow(choice: .ask, title: KeyActionName.ask), ActionRow(choice: .rewrite, title: KeyActionName.rewrite), + ActionRow(choice: .compose, title: KeyActionName.compose), ActionRow(choice: .command, title: KeyActionName.command), ] } diff --git a/App/Blurt/Blurt/Wizard/ReadyView.swift b/App/Blurt/Blurt/Wizard/ReadyView.swift index 14edb8af..4c8fa7b4 100644 --- a/App/Blurt/Blurt/Wizard/ReadyView.swift +++ b/App/Blurt/Blurt/Wizard/ReadyView.swift @@ -205,6 +205,7 @@ struct ReadyView: View { case .readAloud: LibraryKind.readAloud.name case .ask: LibraryKind.ask.name case .rewrite: KeyActionName.rewrite + case .compose: KeyActionName.compose case .command: KeyActionName.command case .dictate(TriggerKeyChoice.mainWindowStyle), .off: "Dictate" case .dictate(let style): diff --git a/App/Blurt/Blurt/Wizard/SoundsSettingsTab.swift b/App/Blurt/Blurt/Wizard/SoundsSettingsTab.swift index 960c409f..44ee6abb 100644 --- a/App/Blurt/Blurt/Wizard/SoundsSettingsTab.swift +++ b/App/Blurt/Blurt/Wizard/SoundsSettingsTab.swift @@ -48,6 +48,7 @@ struct SoundsSettingsTab: View { case .readAloud: KeyActionName.readAloud case .ask: KeyActionName.ask case .rewrite: KeyActionName.rewrite + case .compose: KeyActionName.compose case .command: KeyActionName.command } } diff --git a/App/Blurt/BlurtUITests/CommandPipelineUITests.swift b/App/Blurt/BlurtUITests/CommandPipelineUITests.swift index 44bb6b69..9909a9e5 100644 --- a/App/Blurt/BlurtUITests/CommandPipelineUITests.swift +++ b/App/Blurt/BlurtUITests/CommandPipelineUITests.swift @@ -116,4 +116,14 @@ final class CommandPipelineUITests: BlurtUITestCase { waitForText( pastedLabel(in: harness), containing: "Rewritten (Make it shorter)", timeout: 15, "The rewrite should paste") } + + /// A compose key: no selection needed; what the spoken instructions describe is pasted at the cursor. + func testComposeKeyPastesTheComposition() { + let harness = readyHarness() + bind("left ⌃", with: UITestIdentifiers.harnessSetCompose, in: harness) + command("Tell Dan we should ship it", in: harness) + waitForText( + pastedLabel(in: harness), containing: "Composed (Tell Dan we should ship it)", timeout: 15, + "The composition should paste") + } } diff --git a/App/Blurt/BlurtUITests/README.md b/App/Blurt/BlurtUITests/README.md index bb313dfd..022b52c0 100644 --- a/App/Blurt/BlurtUITests/README.md +++ b/App/Blurt/BlurtUITests/README.md @@ -18,12 +18,13 @@ observable UI rather than mocking views. `DictationSession` press/release — via both the real key tap and the direct Start/Stop seam — and asserts the live status walks to `recording` and back to `idle`, including two back-to-back runs to prove the pipeline re-arms. -- **Control Your Mac and Rewrite Selection** — `CommandPipelineUITests`: a command key held +- **Control Your Mac, Rewrite Selection and Compose From Instructions** — `CommandPipelineUITests`: a command key held through the real key tap, recorded and transcribed by the session, planned and carried out by the router with the stub planner and performer (`UITestCommandDoubles.swift` — no gateway, nothing acted on), and the pill: Working for as long as the command runs, then Done; a typed step pasted through the session; a complaint noted, never planned; overheard speech refused; a command it can't - do failing with the key still working after; and a rewrite pasted over the canned selection. + do failing with the key still working after; a rewrite pasted over the canned selection; and a + composition pasted from spoken instructions, no selection needed. - **Pasting into an app** — `DictationPipelineUITests`: the stub injector records what it would paste, with the harness window standing in for the target app. diff --git a/App/Blurt/BlurtUITests/SettingsUITests.swift b/App/Blurt/BlurtUITests/SettingsUITests.swift index cfe71bdf..3bf59d36 100644 --- a/App/Blurt/BlurtUITests/SettingsUITests.swift +++ b/App/Blurt/BlurtUITests/SettingsUITests.swift @@ -247,7 +247,7 @@ final class SettingsUITests: BlurtUITestCase { let actions = [ "Dictate", "Dictate: No custom style", KeyActionName.readAloud, KeyActionName.ask, - KeyActionName.rewrite, KeyActionName.command, + KeyActionName.rewrite, KeyActionName.compose, KeyActionName.command, ] for action in actions { XCTAssertTrue( diff --git a/App/Blurt/Shared/KeyActionName.swift b/App/Blurt/Shared/KeyActionName.swift index d04116c6..dd67051c 100644 --- a/App/Blurt/Shared/KeyActionName.swift +++ b/App/Blurt/Shared/KeyActionName.swift @@ -1,4 +1,4 @@ -/// The names the Keys tab gives its read, ask, rewrite and command actions. The tab's menu, the +/// The names the Keys tab gives its read, ask, rewrite, compose and command actions. The tab's menu, the /// ready screen's key rows, the Read Aloud tab's footers and the UI tests all /// quote them, so they're spelled once — here in `Shared/`, where the UI tests /// can reach them too. @@ -6,5 +6,6 @@ enum KeyActionName { static let readAloud = "Read Selection Aloud" static let ask = "Ask About Selection" static let rewrite = "Rewrite Selection" + static let compose = "Compose From Instructions" static let command = "Control Your Mac" } diff --git a/App/Blurt/Shared/UITestIdentifiers.swift b/App/Blurt/Shared/UITestIdentifiers.swift index 99d21a1b..dd1997e5 100644 --- a/App/Blurt/Shared/UITestIdentifiers.swift +++ b/App/Blurt/Shared/UITestIdentifiers.swift @@ -90,6 +90,7 @@ enum UITestIdentifiers { static func keySoundPicker(_ keyCode: Int) -> String { "settings.keySound.picker.\(keyCode)" } static let harnessSetCommand = "harness.key.setCommand" static let harnessSetRewrite = "harness.key.setRewrite" + static let harnessSetCompose = "harness.key.setCompose" /// What the stub planner was asked to plan, and what the stub performer did. static let plannedLabel = "harness.planned" static let performedLabel = "harness.performed" diff --git a/App/BlurtiOS/BlurtiOSCore/VoiceState.swift b/App/BlurtiOS/BlurtiOSCore/VoiceState.swift index c3372ddf..3cb47ed0 100644 --- a/App/BlurtiOS/BlurtiOSCore/VoiceState.swift +++ b/App/BlurtiOS/BlurtiOSCore/VoiceState.swift @@ -122,8 +122,8 @@ extension PhaseSnapshot.State { nonisolated package init(_ overlay: OverlayUIState) { self = switch overlay { - // Reading aloud, rewriting and commands are the Mac's alone, so the iPhone never sees them. - case .idle, .speaking, .rewriting, .commanding, .done, .noted: .idle + // Reading aloud, rewriting, composing and commands are the Mac's alone, so the iPhone never sees them. + case .idle, .speaking, .rewriting, .composing, .commanding, .done, .noted: .idle case .connecting: .connecting case .recording: .recording case .processing: .processing @@ -140,8 +140,8 @@ extension PhaseSnapshot { nonisolated package static func message(for overlay: OverlayUIState) -> String? { switch overlay { case .error(let message, _): message - case .idle, .connecting, .recording, .processing, .speaking, .rewriting, .commanding, .pasted, .noTarget, - .done, .noted: + case .idle, .connecting, .recording, .processing, .speaking, .rewriting, .composing, .commanding, .pasted, + .noTarget, .done, .noted: nil } } diff --git a/Sources/BlurtEngine/Hotkey/SelectionSpeechRouter.swift b/Sources/BlurtEngine/Hotkey/SelectionSpeechRouter.swift index 24fde015..c922d915 100644 --- a/Sources/BlurtEngine/Hotkey/SelectionSpeechRouter.swift +++ b/Sources/BlurtEngine/Hotkey/SelectionSpeechRouter.swift @@ -6,7 +6,9 @@ import Foundation /// set to read aloud reads the selected text; a press of a key set to ask /// records a spoken request about it and reads the answer; a press of a key /// set to rewrite records a spoken instruction and pastes the selection, - /// rewritten that way, over it; a press of a key set to command records a + /// rewritten that way, over it; a press of a key set to compose records + /// spoken instructions and pastes what they describe at the cursor; a press + /// of a key set to command records a /// spoken command and carries it out; a dictating key's /// commands go straight to `DictationSession.submit` (see `TriggerKeyBindings` /// for which key does what). @@ -26,6 +28,7 @@ import Foundation private let session: DictationSession private let speaker: any ReadAloudSpeaker private let rewriter: any SelectionRewriting + private let composer: any MessageComposing private let planner: any VoiceCommandPlanning private let performer: any VoiceCommandPerforming private let readSelection: @Sendable () async -> String? @@ -35,8 +38,8 @@ import Foundation private let clock: any Clock private var routing = SelectionSpeechRouting() private var speech: Task? - /// The job in flight — a rewrite until it pastes, a command until its - /// last step. Cancelled by any press. + /// The job in flight — a rewrite or a composition until it pastes, a + /// command until its last step. Cancelled by any press. private var job: Task? /// The screen read a command's release started (`.readScreen`), running /// while the command transcribes; `.runCommand` plans with its result. @@ -63,7 +66,7 @@ import Foundation /// A read, answer or voice sample failed (it's already logged): the app /// beeps, since the pill just goes away. public var speechFailed: (any Error) -> Void = { _ in } - /// What the pill shows for a job: `.rewriting` or `.commanding` as it + /// What the pill shows for a job: `.rewriting`, `.composing` or `.commanding` as it /// starts, then its outcome — Pasted (Done, for a command), Copied, or the /// error — or nil when a press abandoned it, and the pill goes back to /// what the session is doing. @@ -81,7 +84,7 @@ import Foundation session: session, speaker: SelectionSpeaker(clips: clips), readSelection: SelectionSpeaker.focusedSelection) } - /// The seams: what speaks, what rewrites, what plans and carries out a + /// The seams: what speaks, what rewrites, what composes, what plans and carries out a /// command, how the selection is read, where the /// settings each press reads come from, and the clock its timers run on. /// Public so a host can run the whole read-aloud path offline (Blurt's UI @@ -89,6 +92,7 @@ import Foundation public init( session: DictationSession, speaker: any ReadAloudSpeaker, rewriter: any SelectionRewriting = SelectionRewriter(), + composer: any MessageComposing = MessageComposer(), planner: any VoiceCommandPlanning = VoiceCommandPlanner(), performer: any VoiceCommandPerforming = SystemVoiceCommandPerformer(), readSelection: @escaping @Sendable () async -> String?, defaults: UserDefaults = .standard, @@ -97,6 +101,7 @@ import Foundation self.session = session self.speaker = speaker self.rewriter = rewriter + self.composer = composer self.planner = planner self.performer = performer self.readSelection = readSelection @@ -163,13 +168,13 @@ import Foundation case .startHoldTimer: startHoldTimer() case .syncGateAfterSession: syncGateAfterSession() case .resetGate: resetGate() - case .speak, .answer, .stopSpeech, .rewrite, .runCommand, .stopJob: performTask(action) + case .speak, .answer, .stopSpeech, .rewrite, .compose, .runCommand, .stopJob: performTask(action) } } } /// The actions that start or stop a task of this router's own: a read, an - /// answer, a rewrite, a command. + /// answer, a rewrite, a composition, a command. private func performTask(_ action: SelectionSpeechRouting.Action) { switch action { case .speak(let text): @@ -190,11 +195,12 @@ import Foundation // by this read's late end. endSpeech() case .rewrite(let request, let selection): rewrite(selection, as: request) + case .compose(let request): compose(request) case .runCommand(let command): run(command) case .stopJob: // A stopped command's own end never runs, so stop its screen read and - // undo its preparation here — before it ran, too; for a rewrite - // there's neither, and this does nothing. + // undo its preparation here — before it ran, too; for a rewrite or a + // composition there's neither, and this does nothing. screenRead?.cancel() screenRead = nil Task { [performer] in await performer.commandEnded() } @@ -227,17 +233,37 @@ import Foundation /// rewrite this router stopped reports nothing more: `.stopJob` already /// ended it. private func rewrite(_ selection: String, as request: String) { - jobPill(.rewriting) - job = Task { [weak self, rewriter, session] in + paste(as: .rewriting, named: "rewrite") { [rewriter] in try await rewriter.rewrite(selection, as: request) } + } + + /// Writes what the spoken `instructions` describe and pastes it at the + /// cursor as a rewrite is pasted (`replaceSelection`: the hand-back + /// press's target, no separating space). Written for that same target + /// ("Slack" gets a chat message, "Mail" an email) — the app the press + /// pinned, not whatever the user switched to while it transcribed. + private func compose(_ instructions: String) { + paste(as: .composing, named: "compose") { [composer, session] in + try await composer.compose(instructions, for: await session.pinnedAppName) + } + } + + /// One rewrite or composition as the router's job: `pill` while `write` + /// runs, then its text pasted through the session's injector and the + /// outcome on the pill. + private func paste( + as pill: OverlayUIState, named name: String, _ write: @escaping @Sendable () async throws -> String + ) { + jobPill(pill) + job = Task { [weak self, session] in let outcome: OverlayUIState do { - let text = try await rewriter.rewrite(selection, as: request) + let text = try await write() try Task.checkCancellation() outcome = try await session.replaceSelection(with: text).overlayState } catch { // A cancelled request answers `URLError.cancelled`: abandoned, not failed. if Self.isCancellation(error) { return } - Self.logger.error("rewrite failed: \(error.localizedDescription, privacy: .public)") + Self.logger.error("\(name, privacy: .public) failed: \(error.localizedDescription, privacy: .public)") outcome = Self.failure(error) } guard !Task.isCancelled, let self else { return } @@ -286,13 +312,13 @@ import Foundation } } - /// Whether a read, answer, rewrite or command is wearing the pill — what + /// Whether a read, answer, rewrite, composition or command is wearing the pill — what /// `PipelinePhase.overlayState(whileSpeaking:)` asks, so a session phase /// arriving under any of them can't cover it. public var ownsPill: Bool { activeGeneration != nil || job != nil } /// What the session's transcription shows as when it's a job's spoken - /// words — "Rewriting", "Working" — or nil for a dictation's "Transcribing": + /// words — "Rewriting", "Writing", "Working" — or nil for a dictation's "Transcribing": /// what `PipelinePhase.overlayState(whileSpeaking:processingAs:)` asks. public var processingState: OverlayUIState? { routing.processingState } diff --git a/Sources/BlurtEngine/Hotkey/SelectionSpeechRouting+Types.swift b/Sources/BlurtEngine/Hotkey/SelectionSpeechRouting+Types.swift index d5df00bd..ad79a5da 100644 --- a/Sources/BlurtEngine/Hotkey/SelectionSpeechRouting+Types.swift +++ b/Sources/BlurtEngine/Hotkey/SelectionSpeechRouting+Types.swift @@ -15,6 +15,8 @@ extension SelectionSpeechRouting { case answer(request: String, selection: String) /// Rewrite `selection` as the spoken `request` asks, and paste it over the selection. case rewrite(request: String, selection: String) + /// Write what the spoken `request` describes, and paste it at the cursor. + case compose(request: String) case stopSpeech /// Carry out the spoken `command`. case runCommand(String) @@ -24,8 +26,8 @@ extension SelectionSpeechRouting { /// A command's recording ended: read the screen while it transcribes, for /// `.runCommand` to plan with. case readScreen - /// Abandon the job in flight — a rewrite before it pastes, a command - /// before its next step. + /// Abandon the job in flight — a rewrite or a composition before it + /// pastes, a command before its next step. case stopJob /// A dictation ended without a key event: `DictationKeyTap.syncAfterTerminalPhase`. case syncGateAfterSession @@ -36,7 +38,7 @@ extension SelectionSpeechRouting { } enum Intent: Sendable, Equatable { - case read, ask, rewrite, command + case read, ask, rewrite, compose, command /// Whether the press records a spoken request (`.asking`) rather than reading. var records: Bool { self != .read } @@ -68,7 +70,7 @@ extension SelectionSpeechRouting { /// was let go of before the mic ever opened). case transcribing(Intent, String, started: Bool) case speaking - /// A rewrite or a command is under way: the router's one job. Like + /// A rewrite, a composition or a command is under way: the router's one job. Like /// `.speaking`, the gate is free under it, and any press abandons it. case working } diff --git a/Sources/BlurtEngine/Hotkey/SelectionSpeechRouting.swift b/Sources/BlurtEngine/Hotkey/SelectionSpeechRouting.swift index 14d2fb80..ef92d0f0 100644 --- a/Sources/BlurtEngine/Hotkey/SelectionSpeechRouting.swift +++ b/Sources/BlurtEngine/Hotkey/SelectionSpeechRouting.swift @@ -3,7 +3,9 @@ /// dictation session; a read-aloud key reads the selection aloud; an ask key /// records a spoken request about the selection and reads the answer; a rewrite /// key records a spoken instruction and pastes the selection, rewritten, over -/// it; a command key records a spoken command and carries it out. It is a +/// it; a compose key records spoken instructions and pastes what they +/// describe at the cursor; a command key records a spoken command and +/// carries it out. It is a /// pure state machine. The app's `SelectionSpeechRouter` carries out the /// actions it returns (the AX read, the hold timer, the speech task, the gate /// resets), the same split as `DictationKeyRouter` under `DictationKeyTap`. @@ -26,7 +28,9 @@ /// transcript is the instruction the selection is rewritten by. /// - A command records the same way, and its transcript is carried out. It /// needs no selection, so it skips the read: the press is `.selected` from -/// the key-down, and still waits to prove itself before the mic opens. +/// the key-down, and still waits to prove itself before the mic opens. A +/// compose press does the same, and its transcript is the instructions for +/// what to write. /// /// The Accessibility read is in flight while those signals land, so a press /// first waits in `.deciding`, noting them, and acts once the read returns. @@ -57,12 +61,14 @@ struct SelectionSpeechRouting: Sendable { /// What the pill shows while the session records or transcribes a job's /// spoken words, in place of "Transcribing": "Rewriting" for a rewrite's - /// instruction, "Working" for a command. Nil for anything else. + /// instruction, "Writing" for a composition's, "Working" for a command. + /// Nil for anything else. var processingState: OverlayUIState? { switch mode { case .asking(let intent, _, _, _), .transcribing(let intent, _, _): switch intent { case .rewrite: .rewriting + case .compose: .composing case .command: .commanding case .read, .ask: nil } @@ -81,9 +87,9 @@ struct SelectionSpeechRouting: Sendable { // Off only reaches here when Settings turned the key off mid-press: // dictate, as it was bound to. case .dictate, .off: return [.forward(.press(key: key))] - case .command: + case .command, .compose: // Nothing to read: the press only has to prove itself. - mode = .selected(.command, "") + mode = .selected(Self.intent(action), "") return holds ? [.startHoldTimer] : [] case .readAloud, .ask, .rewrite: mode = .deciding(Self.intent(action), trigger: nil, ended: false) @@ -94,8 +100,8 @@ struct SelectionSpeechRouting: Sendable { // answer, or a tapped read, which frees the gate as it starts. return interrupt(with: .stopSpeech, key: key, action: action) case .working: - // Any press abandons a rewrite before it pastes, or a command before - // its next step, as it stops a read: + // Any press abandons a rewrite or a composition before it pastes, or a + // command before its next step, as it stops a read: // a paste landing in the middle of whatever the press starts would be // the worse surprise. The gate is free here too. return interrupt(with: .stopJob, key: key, action: action) @@ -126,7 +132,7 @@ struct SelectionSpeechRouting: Sendable { mode = .passthrough switch action { case .dictate, .off: return [stop, .forward(.press(key: key))] - case .readAloud, .ask, .rewrite, .command: return [stop, .resetGate] + case .readAloud, .ask, .rewrite, .compose, .command: return [stop, .resetGate] } } @@ -276,7 +282,7 @@ struct SelectionSpeechRouting: Sendable { // every other key — dictation's among them — dead until it ended. A // held read keeps it, so letting go stops the read. return trigger == .tap ? [.speak(selection), .resetGate] : [.speak(selection)] - case .ask, .rewrite, .command: + case .ask, .rewrite, .compose, .command: mode = .asking(intent, selection, started: !sessionBusy, byHold: trigger == .hold) // A command's screen is prepared now (`.prepareCommand`), while it's // spoken — and before the session is free to record it, if it isn't yet. @@ -289,6 +295,7 @@ struct SelectionSpeechRouting: Sendable { switch action { case .ask: .ask case .rewrite: .rewrite + case .compose: .compose case .command: .command case .readAloud, .dictate, .off: .read } @@ -304,7 +311,7 @@ struct SelectionSpeechRouting: Sendable { return [.resetGate] } - /// The job — a rewrite or a command — finished or failed. As with + /// The job — a rewrite, a composition or a command — finished or failed. As with /// `speechFinished`, the caller must not report a job it stopped itself. mutating func jobFinished() -> [Action] { guard mode == .working else { return [] } @@ -353,6 +360,9 @@ struct SelectionSpeechRouting: Sendable { case .rewrite: mode = .working return [.rewrite(request: request, selection: selection)] + reset + case .compose: + mode = .working + return [.compose(request: request)] + reset case .command: mode = .working // Handed back by the recording cap, the key still down, it skipped diff --git a/Sources/BlurtEngine/Hotkey/TriggerActivation.swift b/Sources/BlurtEngine/Hotkey/TriggerActivation.swift index 8fb9f959..a6fa9ab4 100644 --- a/Sources/BlurtEngine/Hotkey/TriggerActivation.swift +++ b/Sources/BlurtEngine/Hotkey/TriggerActivation.swift @@ -42,6 +42,7 @@ public enum TriggerActivation: String, CaseIterable, Sendable { case .readAloud: "listen" case .ask: "ask" case .rewrite: "rewrite" + case .compose: "give instructions" case .command: "give a command" case .dictate, .off: "talk" } diff --git a/Sources/BlurtEngine/Hotkey/TriggerKeyBindings.swift b/Sources/BlurtEngine/Hotkey/TriggerKeyBindings.swift index 2477bc64..3732eb0e 100644 --- a/Sources/BlurtEngine/Hotkey/TriggerKeyBindings.swift +++ b/Sources/BlurtEngine/Hotkey/TriggerKeyBindings.swift @@ -12,6 +12,10 @@ public enum TriggerKeyAction: Sendable, Hashable { /// Records a spoken instruction ("make it more formal") and pastes the /// selected text, rewritten that way, over it. case rewrite + /// Records spoken instructions for something to write ("a Slack reply to + /// Dan saying we should…") and pastes what they describe at the cursor + /// (`MessageComposer`). Needs no selection. + case compose /// Records a spoken command ("open Slack and start a new message") and /// carries it out (`VoiceCommandPlanner`, `VoiceCommandPerforming`). case command @@ -29,6 +33,7 @@ public enum TriggerKeyChoice: Hashable, Sendable { case readAloud case ask case rewrite + case compose case command /// The style id for "use the main window's selection". Not a UUID, and not @@ -44,18 +49,20 @@ public enum TriggerKeyChoice: Hashable, Sendable { case .readAloud: .readAloud case .ask: .ask case .rewrite: .rewrite + case .compose: .compose case .command: .command } } /// The stored form. Any string that isn't one of the action markers is a - /// style id — the encoding from before read, ask, rewrite and command existed. + /// style id — the encoding from before read, ask, rewrite, compose and command existed. init(raw: String) { switch raw { case "": self = .off case "read-aloud": self = .readAloud case "ask": self = .ask case "rewrite": self = .rewrite + case "compose": self = .compose case "command": self = .command default: self = .dictate(style: raw) } @@ -68,6 +75,7 @@ public enum TriggerKeyChoice: Hashable, Sendable { case .readAloud: "read-aloud" case .ask: "ask" case .rewrite: "rewrite" + case .compose: "compose" case .command: "command" } } diff --git a/Sources/BlurtEngine/Pipeline/DictationLog+Commands.swift b/Sources/BlurtEngine/Pipeline/DictationLog+Commands.swift index 5c70a22d..f42c133f 100644 --- a/Sources/BlurtEngine/Pipeline/DictationLog+Commands.swift +++ b/Sources/BlurtEngine/Pipeline/DictationLog+Commands.swift @@ -173,7 +173,7 @@ extension DictationLog { case .pasted: "pasted" case .noTarget: "copied" case .noted: "noted" - case .idle, .connecting, .recording, .processing, .speaking, .rewriting, .commanding, .error: "failed" + case .idle, .connecting, .recording, .processing, .speaking, .rewriting, .composing, .commanding, .error: "failed" } } } diff --git a/Sources/BlurtEngine/Pipeline/DictationSession+Pipeline.swift b/Sources/BlurtEngine/Pipeline/DictationSession+Pipeline.swift index 27d72e4e..7c6799dc 100644 --- a/Sources/BlurtEngine/Pipeline/DictationSession+Pipeline.swift +++ b/Sources/BlurtEngine/Pipeline/DictationSession+Pipeline.swift @@ -227,7 +227,9 @@ extension DictationSession { } /// Pastes `text` over the selection in the app the last hand-back press - /// captured: the Rewrite Selection key's rewrite. Through this session's + /// captured: the Rewrite Selection key's rewrite, and the Compose From + /// Instructions key's writing (over whatever is selected, else at the + /// cursor). Through this session's /// injector, so it queues behind any dictation paste still settling rather /// than racing it on the pasteboard. No prior text and no window title, so /// no separating space is added: the paste replaces the selection rather diff --git a/Sources/BlurtEngine/Pipeline/DictationSession+Press.swift b/Sources/BlurtEngine/Pipeline/DictationSession+Press.swift index 0dda9a27..7b2b5a5b 100644 --- a/Sources/BlurtEngine/Pipeline/DictationSession+Press.swift +++ b/Sources/BlurtEngine/Pipeline/DictationSession+Press.swift @@ -156,6 +156,7 @@ extension DictationSession { let captureFrontmost = seams.captureFrontmost let captured = await captureFrontmost() await injector.setTarget(captured) + pinnedAppName = captured?.processName // Key terms are read synchronously at press (cheap UserDefaults read), so // each dictation observably re-reads Settings edits at press time. let keyTerms = keyTermsProvider() diff --git a/Sources/BlurtEngine/Pipeline/DictationSession.swift b/Sources/BlurtEngine/Pipeline/DictationSession.swift index 3fce4cfc..0eaa5fad 100644 --- a/Sources/BlurtEngine/Pipeline/DictationSession.swift +++ b/Sources/BlurtEngine/Pipeline/DictationSession.swift @@ -161,6 +161,11 @@ public actor DictationSession { /// The style this press resolved (`pressStyleProvider`), named on its Recent row. var pressStyle = PressStyle.none + /// The name of the app the last press pinned as its paste target + /// (`injector.setTarget`): what a composition is written for, since it is + /// pasted there — never the app in front by the time it is written. + public internal(set) var pinnedAppName: String? + /// The production entry point: the real log, and the real focus capture unless the host /// supplies its own (`hostFocusCapture`: the iOS app, whose keyboard knows the text before /// the cursor). Delegates to the seam-carrying initializer below, which can't be public. diff --git a/Sources/BlurtEngine/Pipeline/OverlayUIState.swift b/Sources/BlurtEngine/Pipeline/OverlayUIState.swift index 73cd347d..c4a1f913 100644 --- a/Sources/BlurtEngine/Pipeline/OverlayUIState.swift +++ b/Sources/BlurtEngine/Pipeline/OverlayUIState.swift @@ -37,6 +37,10 @@ public enum OverlayUIState: Equatable, Sendable { /// phase either: the app shows it from the spoken instruction's hand-back /// until the rewrite pastes, then the notice a dictation ends with. case rewriting + /// Writing what spoken instructions describe (the Compose From Instructions + /// key). Not a pipeline phase: the app shows it from the instructions' + /// hand-back until the writing pastes, then the notice a dictation ends with. + case composing /// Carrying out a spoken command (the Control Your Mac key): planning it, /// then its steps. Not a pipeline phase: the app shows it from the /// command's hand-back until the last step is done, then `.done` or the error. @@ -80,6 +84,7 @@ public enum OverlayUIState: Equatable, Sendable { case .recording: "Recording." case .speaking: "Reading aloud." case .rewriting: "Rewriting the selection." + case .composing: "Writing your message." case .commanding: "Carrying out your command." case .processing: "Processing." case .error(let message, _): message @@ -103,7 +108,7 @@ public enum OverlayUIState: Equatable, Sendable { // A sentence, not a word: long enough to read the device name. case .error(_, _?): 3.0 case .error, .noTarget: 1.6 - case .idle, .connecting, .recording, .processing, .speaking, .rewriting, .commanding: nil + case .idle, .connecting, .recording, .processing, .speaking, .rewriting, .composing, .commanding: nil } } @@ -166,7 +171,8 @@ extension PipelinePhase { /// /// `processingAs`: what this phase's transcription shows as when it's a /// job's spoken words rather than a dictation — `.rewriting` for a Rewrite - /// Selection press's instruction, `.commanding` for a command — so the pill + /// Selection press's instruction, `.composing` for a compose press's, + /// `.commanding` for a command — so the pill /// reads as one step from release to the job's end. public func overlayState(whileSpeaking speaking: Bool, processingAs: OverlayUIState? = nil) -> OverlayUIState? { if speaking && !isCapturing { return nil } diff --git a/Sources/BlurtEngine/Rewrite/MessageComposer.swift b/Sources/BlurtEngine/Rewrite/MessageComposer.swift new file mode 100644 index 00000000..e8df20d4 --- /dev/null +++ b/Sources/BlurtEngine/Rewrite/MessageComposer.swift @@ -0,0 +1,78 @@ +/// Writes what spoken instructions describe ("ok, a Slack reply to Dan: we +/// should have a feature to just speak instructions, keep it casual"), for the +/// Compose From Instructions key (`SelectionSpeechRouting`). Public so a host +/// can stand in its own (the engine's tests and the app's UI tests do). +public protocol MessageComposing: Sendable { + /// The text `instructions` describe, ready to paste into `app` — the + /// frontmost app's name, nil when it isn't known. + func compose(_ instructions: String, for app: String?) async throws -> String +} + +/// The production `MessageComposing`: one `POST /v1/chat/completions` to +/// AssemblyAI's LLM Gateway, through `GatewayConnection.send`. +/// +/// **The rewrite key's sibling, and like it pasted.** It never sees a +/// dictation's words as text to clean up: the transcript it gets is a +/// hand-back press's spoken instructions (`.handedBack`), which it writes +/// *from*, never pastes. Dictation cleanup stays the dictation API's +/// server-side rewrite (AGENTS.md's settled decisions). +public struct MessageComposer: MessageComposing, GatewayClient { + /// The rewrite key's model (`SelectionRewriter.model`): writing, where the + /// larger model earns its latency. + static let model = SelectionRewriter.model + /// Room for a long email or a page of notes. A reply that hits it is cut + /// short and never pasted. + static let maxTokens = 4_000 + /// The rewrite key's: the same model writing, in one unstreamed reply. + static let timeout = SelectionRewriter.timeout + + static let instruction = """ + Someone has said out loud what they want written, and your reply is pasted where their \ + cursor is, as if they had typed it themselves. What they said is instructions about the text — \ + who it is for, what it should say, how it should sound — not the text itself. Write the text \ + they describe. + + Never include the instructions themselves ("ok, I'm writing a Slack reply", "make it \ + friendly"). Where they spell out what to say, keep their points and their wording close; \ + where they only sketch it, write it out. Write as them, in the first person, to whoever they \ + are writing to. + + Follow any tone, length, format or language they ask for. Otherwise keep it short and \ + natural, in the language they spoke. Fit it to the app it is going into, named in when \ + known: a chat app such as Slack, Microsoft Teams, Discord or Messages gets a conversational \ + message with no subject line, greeting block or sign-off (in Slack, bold is *single \ + asterisks*); an email gets a greeting and a sign-off; anything else gets plain prose. Use a \ + list only when the content is one, and no Markdown headings. + + The instructions were transcribed from speech and may contain small transcription mistakes. \ + Go with their most likely meaning rather than asking a question back. Never invent names, \ + dates, numbers or commitments they did not give, and never leave a placeholder such as \ + [Name]: write around what is missing. + + Reply with only the text to paste: no preamble such as "Sure" or "Here is", no notes, and no \ + quotation marks around it. + """ + + /// The gateway call, key and transport bound in. + private let gateway: GatewayConnection + + public init(gateway: GatewayConnection) { + self.gateway = gateway + } + + public func compose(_ instructions: String, for app: String?) async throws -> String { + let reply = try await gateway.send( + ReadAloudLLM.chat( + system: Self.instruction, user: Self.prompt(instructions, for: app), maxTokens: Self.maxTokens, + model: Self.model), timeout: Self.timeout) + guard !reply.truncated else { throw ReadAloudLLMError.truncated } + return reply.text + } + + /// The user turn: the app, when known, then the instructions. + static func prompt(_ instructions: String, for app: String?) -> String { + let spoken = "\n\(instructions)\n" + guard let app, !app.isEmpty else { return spoken } + return "\(app)\n\n\(spoken)" + } +} diff --git a/Tests/BlurtEngineTests/MessageComposerTests.swift b/Tests/BlurtEngineTests/MessageComposerTests.swift new file mode 100644 index 00000000..b9644274 --- /dev/null +++ b/Tests/BlurtEngineTests/MessageComposerTests.swift @@ -0,0 +1,58 @@ +import Foundation +import Testing + +@testable import AssemblyAI +@testable import BlurtEngine + +@Suite("MessageComposer") +struct MessageComposerTests { + private func composer(_ transport: any HTTPTransport) -> MessageComposer { + MessageComposer(apiKeyProvider: { "raw-key" }, transport: transport) + } + + @Test("posts one chat completion to Gemma: the instruction as system, the app and instructions tagged as user") + func request() async throws { + let sent = ValueBox(nil) + let transport = FakeHTTPTransport { request in + sent.value = request + return (200, completion("Hey Dan, we should have a way to just speak instructions.")) + } + let text = try await composer(transport).compose( + "ok I'm writing a Slack reply, tell Dan we should have a feature to just speak instructions", for: "Slack") + #expect(text == "Hey Dan, we should have a way to just speak instructions.") + + let request = try #require(sent.value) + #expect(request.url == ReadAloudLLM.endpoint) + #expect(request.timeoutInterval == MessageComposer.timeout) + let body = try #require(request.httpBody) + let object = try #require(try JSONSerialization.jsonObject(with: body) as? [String: Any]) + #expect(object["model"] as? String == "gemma-4-31b") + #expect(object["max_tokens"] as? Int == MessageComposer.maxTokens) + let messages = try #require(object["messages"] as? [[String: String]]) + #expect( + messages == [ + ["role": "system", "content": MessageComposer.instruction], + [ + "role": "user", + "content": + "Slack\n\n\n" + + "ok I'm writing a Slack reply, tell Dan we should have a feature to just speak instructions\n" + + "", + ], + ]) + } + + @Test("with no app known, the user turn is the instructions alone") + func noApp() { + #expect(MessageComposer.prompt("say hi", for: nil) == "\nsay hi\n") + #expect(MessageComposer.prompt("say hi", for: "") == "\nsay hi\n") + } + + @Test("a composition cut off at the token cap is never handed back to paste") + func truncated() async { + let transport = FakeHTTPTransport { _ in (200, completion("Hey Dan, we", finishReason: "length")) } + await #expect(throws: ReadAloudLLMError.truncated) { + try await composer(transport).compose("tell Dan hi", for: "Slack") + } + } +} diff --git a/Tests/BlurtEngineTests/SelectionSpeechRouterComposeTests.swift b/Tests/BlurtEngineTests/SelectionSpeechRouterComposeTests.swift new file mode 100644 index 00000000..6caad9b0 --- /dev/null +++ b/Tests/BlurtEngineTests/SelectionSpeechRouterComposeTests.swift @@ -0,0 +1,85 @@ +import Foundation +import Testing + +@testable import BlurtEngine + +/// The router carrying out a Compose From Instructions press end to end: the +/// spoken instructions recorded by the session, the composition for the app +/// in front, and the paste through the session's own injector. +@MainActor +@Suite("SelectionSpeechRouter · Compose") +struct SelectionSpeechRouterComposeTests { + private let defaults = freshDefaults() + + /// Runs one held compose press of right ⌥, the session transcribing the + /// instructions as "Tell Dan we should ship it.", the press pinning Slack. + private func compose( + _ composer: StubComposer, injectorError: (any Error & Sendable)? = nil + ) async throws -> Rig { + let fixture = makeSession( + mode: .transcript("Tell Dan we should ship it."), frontmost: CapturedFocus(pid: 42, processName: "Slack")) + await fixture.injector.setError(injectorError) + let router = SelectionSpeechRouter( + session: fixture.session, speaker: GatedSpeaker(), composer: composer, + // A compose press never reads the selection: this would fail it if it did. + readSelection: { + Issue.record("a compose press read the selection") + return nil + }, defaults: defaults) + TriggerKeyStyleStore(defaults: defaults).setChoice(.compose, for: .rightOption) + let events = JobEvents(watching: router) + let phases = try await speakHeldPress(router, on: fixture) + try await waitUntil { !composer.calls.isEmpty } + return Rig(router: router, events: events, injector: fixture.injector, phases: phases) + } + + /// A compose press's router, what it reported, and the injector it pasted through. + @MainActor + private struct Rig { + let router: SelectionSpeechRouter + let events: JobEvents + let injector: StubInjector + let phases: Task + + /// What the pill showed after "Writing", once the composition has ended. + func outcomes() async throws -> [OverlayUIState?] { + try await events.outcomes(after: .composing, stopping: phases) + } + } + + @Test("the spoken instructions are written for the app the press pinned and pasted there with no separator") + func pastesComposition() async throws { + let composer = StubComposer(reply: "Hey Dan, let's ship it.") + let rig = try await compose(composer) + let injector = rig.injector + #expect(try await rig.outcomes() == [.pasted]) + #expect(composer.calls == [ComposeCall(instructions: "Tell Dan we should ship it.", app: "Slack")]) + #expect(await injector.inserted == ["Hey Dan, let's ship it."]) + #expect(await injector.insertedPrior == [nil]) + #expect(await injector.insertedWindowTitles == [nil]) + } + + @Test("with nothing editable focused, the composition is a quiet Copied") + func noTargetIsQuiet() async throws { + let rig = try await compose(StubComposer(reply: "Hey Dan"), injectorError: BlurtError.noEditableTarget) + #expect(try await rig.outcomes() == [.noTarget]) + } + + @Test("a failed composition pastes nothing and shows the error") + func failureShowsError() async throws { + let rig = try await compose(StubComposer(error: ReadAloudLLMError.truncated)) + #expect(try await rig.outcomes() == [.error(message: "The LLM Gateway's rewrite was cut short.")]) + #expect(await rig.injector.inserted.isEmpty) + } + + @Test("a press while the model is still writing abandons it: nothing pasted, one end, no late one") + func pressAbandons() async throws { + let rig = try await compose(StubComposer(hangs: true)) + rig.router.press(key: .rightCommand) + #expect(rig.events.pills == [.composing, nil]) + try await Task.sleep(for: .milliseconds(100)) + rig.phases.cancel() + #expect(rig.events.pills == [.composing, nil]) + #expect(await rig.injector.inserted.isEmpty) + } +} diff --git a/Tests/BlurtEngineTests/SelectionSpeechRouterRewriteTests.swift b/Tests/BlurtEngineTests/SelectionSpeechRouterRewriteTests.swift index 9d680b75..c14a4484 100644 --- a/Tests/BlurtEngineTests/SelectionSpeechRouterRewriteTests.swift +++ b/Tests/BlurtEngineTests/SelectionSpeechRouterRewriteTests.swift @@ -1,5 +1,4 @@ import Foundation -import Synchronization import Testing @testable import BlurtEngine @@ -50,7 +49,7 @@ struct SelectionSpeechRouterRewriteTests { let rig = try await rewrite(rewriter) let injector = rig.injector #expect(try await rig.outcomes() == [.pasted]) - #expect(rewriter.calls == [StubRewriter.Call(selection: "hey all", request: "Make it more formal.")]) + #expect(rewriter.calls == [RewriteCall(selection: "hey all", request: "Make it more formal.")]) #expect(await injector.inserted == ["Dear all,"]) // Nothing for the separator to go on: the paste replaces, never follows. #expect(await injector.insertedPrior == [nil]) @@ -90,32 +89,3 @@ struct SelectionSpeechRouterRewriteTests { #expect(await rig.injector.inserted.isEmpty) } } - -/// A rewriter that records each call and answers with `reply`, or throws -/// `error` — or, `hangs`, writes until cancelled. -final class StubRewriter: SelectionRewriting { - struct Call: Equatable { - let selection: String - let request: String - } - - private let reply: String - private let error: (any Error & Sendable)? - private let hangs: Bool - private let recorded = Mutex<[Call]>([]) - - init(reply: String = "", error: (any Error & Sendable)? = nil, hangs: Bool = false) { - self.reply = reply - self.error = error - self.hangs = hangs - } - - var calls: [Call] { recorded.withLock { $0 } } - - func rewrite(_ selection: String, as request: String) async throws -> String { - recorded.withLock { $0.append(Call(selection: selection, request: request)) } - if hangs { try await Task.sleep(for: .seconds(60)) } - if let error { throw error } - return reply - } -} diff --git a/Tests/BlurtEngineTests/SelectionSpeechRoutingComposeTests.swift b/Tests/BlurtEngineTests/SelectionSpeechRoutingComposeTests.swift new file mode 100644 index 00000000..15cbac58 --- /dev/null +++ b/Tests/BlurtEngineTests/SelectionSpeechRoutingComposeTests.swift @@ -0,0 +1,81 @@ +import Testing + +@testable import BlurtEngine + +@Suite("SelectionSpeechRouting · Compose") +struct SelectionSpeechRoutingComposeTests { + /// A compose press: right ⌥, holdable unless told otherwise. + private func compose(_ routing: inout SelectionSpeechRouting, holds: Bool = true) -> [SelectionSpeechRouting.Action] { + routing.press(key: .rightOption, action: .compose, holds: holds) + } + + /// A held compose press, its instructions recorded and let go. + private func spoken(_ routing: inout SelectionSpeechRouting) { + _ = compose(&routing) + _ = routing.holdElapsed() + _ = routing.submit(.release) + } + + @Test("a compose press reads no selection, and records once it proves itself by a tap or a hold") + func recordsWithoutSelection() { + var tapped = SelectionSpeechRouting() + #expect(compose(&tapped, holds: false).isEmpty) + #expect(tapped.keyLatched() == [.forward(.pressHandingBack(key: nil))]) + + var held = SelectionSpeechRouting() + #expect(compose(&held) == [.startHoldTimer]) + #expect(held.holdElapsed() == [.forward(.pressHandingBack(key: nil))]) + } + + @Test("let go or a combo before the press proves itself: nothing recorded") + func unprovenPressDoesNothing() { + var routing = SelectionSpeechRouting() + _ = compose(&routing) + #expect(routing.submit(.cancel).isEmpty) + #expect(routing.holdElapsed().isEmpty) + } + + @Test("the instructions handed back are composed, with no command preparation, and the end frees the gate") + func composes() { + var routing = SelectionSpeechRouting() + spoken(&routing) + #expect(routing.sessionPhaseChanged(.handedBack("Tell Dan hi.")) == [.compose(request: "Tell Dan hi.")]) + #expect(routing.speechFinished().isEmpty) + #expect(routing.jobFinished() == [.resetGate]) + } + + @Test("a compose shows as Writing from the press until it's handed back") + func processingState() { + var routing = SelectionSpeechRouting() + _ = compose(&routing) + _ = routing.holdElapsed() + #expect(routing.processingState == .composing) + _ = routing.submit(.release) + #expect(routing.processingState == .composing) + _ = routing.sessionPhaseChanged(.handedBack("Tell Dan hi.")) + #expect(routing.processingState == nil) + } + + @Test("a press abandons a composition before it pastes: the dictation key then dictates, any other is cleared") + func pressAbandons() { + let presses: [(press: (TriggerKey, TriggerKeyAction), expected: [SelectionSpeechRouting.Action])] = [ + ((.rightCommand, .dictate), [.stopJob, .forward(.press(key: .rightCommand))]), + ((.rightOption, .compose), [.stopJob, .resetGate]), + ] + for ((key, action), expected) in presses { + var routing = SelectionSpeechRouting() + spoken(&routing) + _ = routing.sessionPhaseChanged(.handedBack("Tell Dan hi.")) + #expect(routing.press(key: key, action: action, holds: true) == expected) + // The router stopped it, so its late end is nothing. + #expect(routing.jobFinished().isEmpty) + } + } + + @Test("nothing said: the press ends like a dictation, with no job to stop") + func nothingSaid() { + var routing = SelectionSpeechRouting() + spoken(&routing) + #expect(routing.sessionPhaseChanged(.idle) == [.syncGateAfterSession]) + } +} diff --git a/Tests/BlurtEngineTests/Stubs/DictationSessionTestSupport.swift b/Tests/BlurtEngineTests/Stubs/DictationSessionTestSupport.swift index 1ef83afe..7430ff9e 100644 --- a/Tests/BlurtEngineTests/Stubs/DictationSessionTestSupport.swift +++ b/Tests/BlurtEngineTests/Stubs/DictationSessionTestSupport.swift @@ -245,3 +245,53 @@ final class JobEvents { return Array(pills.dropFirst()) } } + +/// A rewriter or composer that records each call and answers with `reply`, +/// or throws `error` — or, `hangs`, writes until cancelled. Shared by the +/// rewrite and compose router suites (`StubRewriter`, `StubComposer`). +final class StubWriter: Sendable { + private let reply: String + private let error: (any Error & Sendable)? + private let hangs: Bool + private let recorded = Mutex<[Call]>([]) + + init(reply: String = "", error: (any Error & Sendable)? = nil, hangs: Bool = false) { + self.reply = reply + self.error = error + self.hangs = hangs + } + + var calls: [Call] { recorded.withLock { $0 } } + + fileprivate func write(_ call: Call) async throws -> String { + recorded.withLock { $0.append(call) } + if hangs { try await Task.sleep(for: .seconds(60)) } + if let error { throw error } + return reply + } +} + +struct RewriteCall: Equatable, Sendable { + let selection: String + let request: String +} + +struct ComposeCall: Equatable, Sendable { + let instructions: String + let app: String? +} + +typealias StubRewriter = StubWriter +typealias StubComposer = StubWriter + +extension StubWriter: SelectionRewriting where Call == RewriteCall { + func rewrite(_ selection: String, as request: String) async throws -> String { + try await write(RewriteCall(selection: selection, request: request)) + } +} + +extension StubWriter: MessageComposing where Call == ComposeCall { + func compose(_ instructions: String, for app: String?) async throws -> String { + try await write(ComposeCall(instructions: instructions, app: app)) + } +} diff --git a/Tests/BlurtEngineTests/TriggerKeyStyleTests.swift b/Tests/BlurtEngineTests/TriggerKeyStyleTests.swift index 6dc6b675..c4b48882 100644 --- a/Tests/BlurtEngineTests/TriggerKeyStyleTests.swift +++ b/Tests/BlurtEngineTests/TriggerKeyStyleTests.swift @@ -55,7 +55,8 @@ struct TriggerKeyStyleTests { @Test("choices keep the stored form styles had before read and ask") func rawForm() { for choice in [ - TriggerKeyChoice.off, .readAloud, .ask, .rewrite, .command, .dictate, .dictate(style: UUID().uuidString), + TriggerKeyChoice.off, .readAloud, .ask, .rewrite, .compose, .command, .dictate, + .dictate(style: UUID().uuidString), ] { #expect(TriggerKeyChoice(raw: choice.raw) == choice) } diff --git a/scripts/check-invariants.sh b/scripts/check-invariants.sh index 11582423..d1977300 100755 --- a/scripts/check-invariants.sh +++ b/scripts/check-invariants.sh @@ -101,8 +101,9 @@ NOT_TRANSCRIPT_LLM=":!Sources/BlurtEngine/STT/TranscriptTransform.swift" # The Rewrite Selection key's client (Rewrite/SelectionRewriter.swift) needs no # carve-out: it sends through GatewayConnection.send (in ReadAloudLLM.swift), so the gateway URL this rule # greps for stays in ReadAloudLLM.swift. Keep it that way rather than adding a -# third exemption. The same goes for the Control Your Mac key's planner -# (Command/VoiceCommandPlanner.swift). +# third exemption. The same goes for the Compose From Instructions key's +# composer (Rewrite/MessageComposer.swift) and the Control Your Mac key's +# planner (Command/VoiceCommandPlanner.swift). # The socket rule's one carve-out: the main window's Record button, which # streams a meeting or call to AssemblyAI's realtime API so its words appear as # they're spoken. It never carries a dictation — dictation stays one HTTP @@ -154,7 +155,7 @@ SCOPES=( ADVICE=( "MicCapture builds a fresh AVCaptureSession recorder per capture — a long-lived engine goes stale on a device switch" "the dictation API returns the full text in one response; there is no streaming dictation, and the one socket is the Record button's live transcript (RealtimeTranscriber.swift; read-aloud's TTS is a streaming HTTP response)" - "cleanup is the API's server-side rewrite via config.llm_instruction on the same /v1/transcribe/live call (the gateway clients are read-aloud's, ReadAloudLLM.swift, file transcripts', TranscriptTransform.swift, the rewrite key's, SelectionRewriter.swift, and the command key's, VoiceCommandPlanner.swift, both sending through GatewayConnection in ReadAloudLLM.swift)" + "cleanup is the API's server-side rewrite via config.llm_instruction on the same /v1/transcribe/live call (the gateway clients are read-aloud's, ReadAloudLLM.swift, file transcripts', TranscriptTransform.swift, the rewrite key's, SelectionRewriter.swift, the compose key's, MessageComposer.swift, and the command key's, VoiceCommandPlanner.swift, all three sending through GatewayConnection in ReadAloudLLM.swift)" "transcription is a remote AssemblyAI call — no on-device ASR/LLM, no model cache" "leave language to the model's own detection; setting the field takes that away" "config.conversation_context was replaced by config.stt_prompt (STTPrompt)"