Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .env.example
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# TranscriptLab Nova private MCP example configuration
# TranscriptLab Nova private service example configuration

MCP_CURSOR_KEY_FILE=/path/to/transcriptlab-mcp/cursor-integrity-key

# Optional hosted speech-to-text
Transcription__OpenRouter__ApiKey=YOUR_OPENROUTER_API_KEY
12 changes: 12 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -231,6 +231,18 @@ Uses the supported OpenVINO Whisper sidecar for Intel GPU acceleration. The back

OpenVINO Whisper models are stored under `data/models/openvino-genai/` and can be downloaded on first use or through `POST /api/settings/models/manage`.

#### OpenRouter (hosted speech-to-text)

OpenRouter is available as a first-class remote engine when a server-side API key is configured. TranscriptLab discovers the current speech-to-text model catalog at runtime, stores the selected model with each project, and keeps remote models out of the local Model Manager.

Configure the key with an environment variable so it is never committed:

```bash
export Transcription__OpenRouter__ApiKey=YOUR_OPENROUTER_API_KEY
```

Optional settings can override `Transcription__OpenRouter__BaseUrl`, `FallbackModels`, and `TimeoutSeconds`. `BaseUrl` must be an absolute HTTPS URL. Prepared audio is uploaded to OpenRouter's hosted `/api/v1/audio/transcriptions` endpoint.

Configuration for all WhisperNet engines in `appsettings.json`:
```json
{
Expand Down
2 changes: 2 additions & 0 deletions class-transcriber-backend-prd.md
Original file line number Diff line number Diff line change
Expand Up @@ -342,6 +342,7 @@ Current backend extension points may additionally expose:
- `OpenVinoWhisperSidecar`
- `OnnxWhisper`
- `OpenAiCompatible`
- `OpenRouter`

Implementation note:
- `SherpaOnnx` may run on a local .NET runtime path or isolated helper worker as long as it stays behind the transcription engine abstraction.
Expand All @@ -350,6 +351,7 @@ Implementation note:
- `OpenVinoWhisperSidecar` runs through a long-lived Python FastAPI sidecar with an OpenAI-compatible API. The sidecar manages its own model downloads. The C# engine uses `ISpeechToTextClient` (Microsoft.Extensions.AI experimental) internally. It is the recommended OpenVINO engine for deployments with local GPU hardware.
- `OnnxWhisper` is a reserved placeholder for a future native .NET ONNX Whisper engine. It reports unavailable and must not be used in production.
- `OpenAiCompatible` proxies transcription to any configured OpenAI-compatible API. It must not appear in the engine selector when `BaseUrl` is not configured.
- `OpenRouter` sends prepared audio to OpenRouter's hosted speech-to-text endpoint and discovers transcription-capable models dynamically. It must not appear in the engine selector when its server-side API key is not configured, must use the model stored in each project's settings, and must remain outside the local filesystem Model Manager.

## Suggested model values for MVP
- `tiny`
Expand Down
3 changes: 2 additions & 1 deletion class-transcriber-backend-tech-stack-requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,8 +64,9 @@
- **SherpaOnnx** via the official local **.NET runtime/package** is approved behind the engine abstraction; running it through an isolated helper worker process is allowed when needed for cancellation or runtime isolation
- **Whisper.net** managed library with **Whisper.net.Runtime** (CPU), **Whisper.net.Runtime.Cuda** (NVIDIA GPU), and **Whisper.net.Runtime.CoreML** (native macOS Apple Silicon) runtimes are approved behind the engine abstraction, but CPU, CUDA, and CoreML execution must run through isolated helper worker processes because Whisper.net runtime loading is process-global. CoreML is only approved for native macOS ARM64 runs; Docker/Linux on macOS should be treated as CPU-only unless it calls a native host sidecar.
- A separate **Python FastAPI sidecar** backed by **openvino-genai**, **fastapi**, and **uvicorn** is approved for the `OpenVinoWhisperSidecar` engine; the sidecar runs as a long-lived localhost HTTP server managed by the API process (spawned lazily on first use, killed on shutdown), caches loaded Whisper models in memory between jobs, and avoids the native library version conflict between the .NET `Whisper.net.Runtime.OpenVino` binding and newer OpenVINO Python package installs; the sidecar exposes an OpenAI-compatible `/v1/audio/transcriptions` endpoint plus a model management API with SSE-streamed download progress; it manages its own model downloads independently of the C# download infrastructure; the C# engine communicates with it via `ISpeechToTextClient` from `Microsoft.Extensions.AI.Abstractions`
- **Microsoft.Extensions.AI.Abstractions** is approved as an internal calling abstraction for HTTP-based transcription engines (`OpenVinoWhisperSidecar`, `OpenAiCompatible`); it is used as an implementation detail within the engine and must not replace `IRegisteredTranscriptionEngine` as the public engine contract; the `MEAI001` experimental diagnostic should be suppressed project-wide via `<NoWarn>` in the `.csproj` file when the package is added
- **Microsoft.Extensions.AI.Abstractions** is approved as an internal calling abstraction for HTTP-based transcription engines (`OpenVinoWhisperSidecar`, `OpenAiCompatible`, `OpenRouter`); it is used as an implementation detail within the engine and must not replace `IRegisteredTranscriptionEngine` as the public engine contract; the `MEAI001` experimental diagnostic should be suppressed project-wide via `<NoWarn>` in the `.csproj` file when the package is added
- A generic **`OpenAiCompatible`** proxy engine is approved that forwards transcription requests to any OpenAI-compatible `/v1/audio/transcriptions` endpoint; it shares HTTP multipart construction code with the `OpenVinoWhisperSidecar` client and is hidden from the engine selector when `BaseUrl` is not configured
- A first-class **`OpenRouter`** engine is approved for hosted speech-to-text through `/api/v1/audio/transcriptions`; it reuses the shared OpenAI-compatible multipart helper, authenticates only with a server-side API key, discovers models with `GET /api/v1/models?output_modalities=transcription`, uses the project-selected model, retries with the default JSON response when a provider rejects `verbose_json`, and stays outside local model download/install management
- An **`OnnxWhisper`** engine placeholder value is approved in the `TranscriptionEngine` enum; do not add `Microsoft.ML.OnnxRuntime` or any ONNX inference package until the full implementation is planned
- SSE (Server-Sent Events) streaming is the approved pattern for long-running sidecar model download progress; the C# caller must consume the SSE stream until `status=complete` or `status=error`
- Keep engine-specific logic behind a dedicated transcription service and engine interface
Expand Down
1 change: 1 addition & 0 deletions class-transcriber-frontend-prd.md
Original file line number Diff line number Diff line change
Expand Up @@ -533,6 +533,7 @@ The app must expose a global settings page for future uploads.
- upload modal should start from global defaults but allow override per batch
- batch and retry flows should allow diarization to be enabled or disabled per request
- engine selectors in settings, upload, retry, diagnostics, and model management should surface runtime-available engines from the backend, including Intel GPU options such as `OpenVinoWhisperSidecar` and native Apple Silicon options such as `WhisperNetCoreML` when those runtimes are installed
- when `OpenRouter` is available, settings, upload, and retry selectors should show its backend-discovered transcription models and clearly disclose that audio is sent to a remote provider; OpenRouter models must not appear in the local filesystem Model Manager
- the settings page should also expose a model manager below the defaults form in a vertical stack layout
- the model manager should show known engine/model combinations, local install state, install path, and the latest probe result
- installed models should be probed on page load so runtime problems are visible without queueing an upload
Expand Down
2 changes: 2 additions & 0 deletions class-transcriber-shared-api-contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -248,6 +248,7 @@ WhisperNetCoreML
OpenVinoWhisperSidecar
OnnxWhisper
OpenAiCompatible
OpenRouter
```

Implementation note:
Expand All @@ -260,6 +261,7 @@ Implementation note:
- `OpenVinoWhisperSidecar` uses a long-lived Python FastAPI sidecar backed by `openvino_genai`. The sidecar exposes an OpenAI-compatible `/v1/audio/transcriptions` endpoint and a model management API. It caches loaded Whisper pipelines in memory between jobs and manages its own model downloads internally. The C# engine communicates with it via `ISpeechToTextClient` (Microsoft.Extensions.AI experimental). It is the recommended OpenVINO engine for deployments with a local GPU.
- `OnnxWhisper` is a reserved placeholder for a future native .NET ONNX Whisper engine using `Microsoft.ML.OnnxRuntime`. It is not yet implemented and reports unavailable in all current releases.
- `OpenAiCompatible` is a generic proxy engine that forwards transcription to any external service that exposes an OpenAI-compatible `/v1/audio/transcriptions` endpoint (e.g., the local `OpenVinoWhisperSidecar`, Whisper.cpp server, Ollama). It is hidden from the engine selector when `BaseUrl` is not configured.
- `OpenRouter` is a first-class hosted speech-to-text engine using OpenRouter's `/api/v1/audio/transcriptions` endpoint. It is hidden from the engine selector until a server-side API key is configured. Its transcription-capable models are discovered through `/api/v1/models?output_modalities=transcription`, the selected model is stored in each project's existing `model` field, and its remote models are not included in the local Model Manager catalog.
- `WhisperNet`, `WhisperNetCuda`, and `WhisperNetCoreML` use shared ggml model files and support auto-download. `WhisperNetCoreML` also requires the matching CoreML encoder `.mlmodelc` package beside the ggml model.
- `OpenVinoWhisperSidecar` uses curated pre-exported model directories under `models/openvino-genai/` and supports managed download/redownload/probe from the settings model manager. Download is proxied to the sidecar's own model management API.

Expand Down
Loading
Loading