# QVAC Documentation — full text Contents of this corpus: - 55 pages. - Every page of the collections that publish no versions. - @qvac/sdk v0.21, the current line of /sdk. Other lines are not included here: v0.20 → /sdk/v0.20/llms-full.txt, v0.19 → /sdk/v0.19/llms-full.txt, v0.18 → /sdk/v0.18/llms-full.txt - @qvac/cli v0.15, the current line of /cli. Other lines are not included here: v0.14 → /cli/v0.14/llms-full.txt, v0.13 → /cli/v0.13/llms-full.txt, v0.12 → /cli/v0.12/llms-full.txt - 1 release-notes page is withheld. Fetch one as its own page, from the line index that lists it. - How to pick a line: /resources/docs-for-ai-agents --- title: "QVAC by Tether: the Infinite Stable Intelligence Platform" canonical: https://docs.qvac.tether.io/ecosystem/ collection: "Ecosystem" --- # QVAC by Tether: the Infinite Stable Intelligence Platform (/ecosystem) import { MessagesSquare, Hash, Languages, Mic, Music, Speech, Volume2, ScanText, Image as ImageIcon, Video, Rotate3d, GalleryHorizontal, FlaskConical, ScanSearch, Shapes, Eye, Brain, DoorOpen, Server, MonitorPlay } from 'lucide-react' import { SiKotlin, SiPython, SiTypescript } from '@icons-pack/react-simple-icons' ## Why QVAC? QVAC is Tether's answer to centralized AI by ensuring AI is not tied to massive data centers in the hands of a few *but is free to run on everyone's devices, without a central point of failure or arbitrary censorship*. ## Features ## AI capabilities Text generation}> LLM inference for text generation and chat via Fabric LLM. Text embeddings}> Vector embedding generation for semantic search, clustering, and retrieval, via Fabric LLM. RAG}> Out-of-the-box retrieval-augmented generation workflow. Fine-tuning}> Adapting LLMs to domain-specific tasks via LoRA. Multimodal}> LLM inference over text, images, and other media within a single conversation context. Image generation}> Text-to-image and image-to-image generation via a customized Diffusion backend. World simulation}> Turn a first-frame image and navigation controls into a streamed, navigable video via an interactive world model. Music generation}> Generate music from text prompts and lyrics via ACE-Step or MiniMax-Music3. Transcription}> Automatic speech recognition (ASR) via a customized Whisper backend or NVIDIA Parakeet. Text-to-Speech}> Speech synthesis (TTS) via a customized GGML backend. Voice assistant}> Real-time voice pipeline: transcription, text generation, and speech synthesis in one loop. Translation}> Text-to-text neural machine translation (NMT), via Fabric LLM and Bergamot. BCI}> Brain–computer interface transcription via a customized Whisper backend. VLA}> Vision-language-action for robot control via a customized GGML backend. OCR}> Optical character recognition for extracting text from images via ONNX Runtime. Image classification}> Classify images into labels with confidence scores via a customized GGML backend. ## System overview System overview diagram System overview diagram *The SDK is the main entry point for using QVAC*. It is type-safe and exposes all QVAC capabilities through a unified interface. Two clients share that interface — JS/TS (`@qvac/sdk`, running on Node.js, [Bare](https://bare.pears.com), and [Expo](https://expo.dev)) and Python (`tetherto-qvac-sdk`, asyncio-native) — and both call into the same underlying runtime, so capabilities and behavior are identical either way. Additionally, QVAC provides a CLI with development tools and an HTTP server that wraps QVAC and exposes an [**OpenAI-compatible API**](https://ecosystem.openai.com/docs/api-reference). *You can use the HTTP server as a local model provider to connect any AI tool, harness, or system.* Finally, QVAC also encompasses [**QV.AC**](https://qv.ac), a desktop and mobile assistant app, to empower users and showcase QVAC capabilities, as well as [**research initiatives**](https://huggingface.co/qvac) to advance the state of the art in local AI. ## Next steps Choose how you want to start with QVAC: } title="Get started"> Learn the essentials for using the SDK: system requirements, compatibility matrix, setup basics, and core usage flows. } title="JS/TS SDK"> Install `@qvac/sdk`, run the quickstart, and target Node.js, Bare, or Expo. } title="Python SDK"> Install `tetherto-qvac-sdk`, run the quickstart, and drive the same worker from Python. } title="See QVAC in practice"> Try QV.AC, our flagship desktop and mobile assistant app built with QVAC. } title="Use QVAC as a local model provider"> Run the QVAC HTTP server and connect it to any OpenAI-compatible tool, harness, or system. --- title: "Addons" canonical: https://docs.qvac.tether.io/ecosystem/addons/ collection: "Ecosystem" --- # Addons (/ecosystem/addons) ## Overview Each addon is a JavaScript [Bare module](https://bare.pears.com) that adds support for one or more AI tasks to QVAC ecosystem. *If you use the SDK, you **don’t need** to use addons directly.* The SDK is the canonical way to build with QVAC: it unifies all addon capabilities under a single interface, enables QVAC to run on JS runtimes beyond Bare, and provides out-of-the-box P2P connectivity. ## Compatibility matrix All addons support the following platforms: | Platform | Min Version | Architecture | GPU API/Backend | Notes | | -------- | ----------- | ------------ | ---------------------------- | --------------------------------------------------------------------------- | | macOS | 14.0+ | arm64 | Metal | Arch x64 supports CPU inference only; Intel iGPU acceleration not supported | | iOS | 17.0+ | arm64 | Metal | Requires Expo | | Linux | Ubuntu 22+ | arm64, x64 | Vulkan | Vulkan runtime required | | Android | 12+ | arm64 | Vulkan, OpenCL (Adreno 700+) | Requires Expo | | Windows | 10+ | x64 | Vulkan | Vulkan-capable GPU + vendor drivers required | ## Catalog The following table lists all available addons for QVAC ecosystem: | Package | AI Tasks | Description | inference engine | Stability | | -------------------------------------------------------------------------------- | ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | ------------ | | [`@qvac/llm-llamacpp`](/ecosystem/addons/llm-llamacpp) | Completion, multimodal | LLM inference for text generation and chat with support to images, and other media within a single conversation context. | [`qvac-fabric-llm.cpp`](https://github.com/tetherto/qvac-fabric-llm.cpp) | stable | | [`@qvac/embed-llamacpp`](/ecosystem/addons/embed-llamacpp) | Text embeddings, RAG | Vector embedding generation for semantic search, clustering, and retrieval that seamlessly suppports retrieval-augmented generation workflow. | `qvac-fabric-llm.cpp` | stable | | [`@qvac/translation-nmtcpp`](/ecosystem/addons/translation-nmtcpp) | Translation | Text-to-text neural machine translation (NMT). | `qvac-fabric-llm.cpp` and [Bergamot](https://browser.mt) | stable | | [`@qvac/transcription-whispercpp`](/ecosystem/addons/transcription-whispercpp) | Transcription | Automatic speech recognition (ASR) for speech-to-text. | [`qvac-fabric-speech.cpp`](https://github.com/tetherto/qvac-fabric-speech.cpp) | stable | | [`@qvac/transcription-parakeet`](/ecosystem/addons/transcription-parakeet) | Transcription | Automatic speech recognition (ASR) for speech-to-text with speaker diarization. | [NVIDIA Parakeet](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2) via [ONNX Runtime](https://onnxruntime.ai) | stable | | [`@qvac/tts-ggml`](/ecosystem/addons/tts-ggml) | Text-to-speech | Speech synthesis for text-to-speech (TTS). | [`qvac-tts.cpp`](https://github.com/tetherto/qvac-fabric-speech.cpp/tree/master/engines/tts) (GGML) | stable | | [`@qvac/audiogen-ggml`](/ecosystem/addons/audiogen-ggml) | Music generation | Generate music from text, lyrics, and musical controls. | [ACE-Step 1.5](https://github.com/ace-step/ACE-Step-1.5) (GGML) | experimental | | [`@qvac/ocr-ggml`](https://github.com/tetherto/qvac/tree/main/packages/ocr-ggml) | OCR | Optical character recognition (OCR) for extracting text from images. | GGML (EasyOCR and docTR pipelines on GGUF weights) | stable | | [`@qvac/diffusion-cpp`](/ecosystem/addons/diffusion-cpp) | Image generation | Text-to-image generation from text prompts. | [`qvac-ext-stable-diffusion.cpp`](https://github.com/tetherto/qvac-ext-stable-diffusion.cpp) | stable | --- title: "@qvac/audiogen-ggml" canonical: https://docs.qvac.tether.io/ecosystem/addons/audiogen-ggml/ collection: "Ecosystem" --- # @qvac/audiogen-ggml (/ecosystem/addons/audiogen-ggml) ## Overview [`@qvac/audiogen-ggml`](https://github.com/tetherto/qvac/tree/main/packages/audiogen-ggml) is a Bare native addon for generating music from captions, lyrics, and musical controls, optionally conditioned on a reference recording or re-rendering an existing song as a cover. It runs ACE-Step 1.5 locally with CPU inference or optional Metal/Vulkan acceleration (desktop, Android, and iOS prebuilds) and returns interleaved stereo Int16 PCM. SDK consumers should use [`audioGen()` through `@qvac/sdk`](/sdk/ai-capabilities/music-generation). Use this package directly only when building against the addon-level Bare API. ## Pipeline Generation runs through four persistent model stages: 1. A Qwen3 text encoder for captions and lyrics. 2. An ACE-Step language model for song structure and musical controls. 3. A DiT model for latent audio generation. 4. A VAE decoder for the final waveform. The models are loaded once and reused between runs. ## Models | Stage | File | | -------------- | -------------------------------- | | Text encoder | `Qwen3-Embedding-0.6B-Q8_0.gguf` | | Language model | `acestep-5Hz-lm-0.6B-Q8_0.gguf` | | Turbo DiT, Q4 | `acestep-v15-turbo-Q4_K_M.gguf` | | Turbo DiT, Q8 | `acestep-v15-turbo-Q8_0.gguf` | | SFT DiT, Q8 | `acestep-v15-sft-Q8_0.gguf` | | VAE | `vae-BF16.gguf` | Choose one DiT variant and combine it with the fixed text encoder, language model, and VAE. ## Installation ```bash npm install @qvac/audiogen-ggml ``` The package requires a supported native prebuild and the four GGUF files. Models are not bundled with the addon. ## Minimal Bare usage ```js const { AudioGen } = require("@qvac/audiogen-ggml"); const model = new AudioGen({ files: { modelDir: "/path/to/acestep/models", ditVariant: "turbo-q4", }, config: { useGPU: true }, }); await model.load(); const response = await model.run("lo-fi hip hop with mellow piano", { lyrics: "[Instrumental]", seed: 42, }); for await (const update of response.iterate()) { if (update.outputArray) { // Interleaved Int16 PCM at update.sampleRate. } } const stats = await response.await(); await model.destroy(); ``` `destroy()` is terminal: create a new `AudioGen` instance instead of calling `load()` again on a destroyed one. ## Reference and cover audio `run()` also accepts `referenceAudio` (timbre conditioning) and, with `taskType: "cover-nofsq"`, `sourceAudio` plus `audioCoverStrength` / `coverNoiseStrength`. Both PCM inputs are `Float32Array` values holding finite, normalized samples in interleaved stereo order at 48 kHz; the addon does not resample or convert channels. Through the SDK, `audioGen()` accepts file paths (decoded for you) or raw PCM buffers — see [Music generation](/sdk/ai-capabilities/music-generation#reference-audio-and-covers). The addon's ordered `edit()` pipeline (Flow-Edit and Repaint) is exposed by the SDK as [`audioEdit()`](/sdk/ai-capabilities/music-generation#edit-an-existing-recording), and its reverse `understand()` pipeline as [`audioUnderstand()`](/sdk/ai-capabilities/music-generation#understand-a-recording). For runtime controls, LM sampling and DCW parameters, supported encoders, cancellation, and complete examples, see the [package README](https://github.com/tetherto/qvac/tree/main/packages/audiogen-ggml#readme). --- title: "@qvac/diffusion-cpp" canonical: https://docs.qvac.tether.io/ecosystem/addons/diffusion-cpp/ collection: "Ecosystem" --- # @qvac/diffusion-cpp (/ecosystem/addons/diffusion-cpp) ## Overview [Bare module](https://bare.pears.com) that adds support for text-to-image generation in QVAC using [`qvac-ext-stable-diffusion.cpp`](https://github.com/tetherto/qvac-ext-stable-diffusion.cpp) as the inference engine. ## Models Supports FLUX.2-klein, SD1.x, SD2.x, SDXL, and SD3 model families. ### FLUX.2-klein Three separate components are required: * **Diffusion model** (`flux-2-klein-4b-Q8_0.gguf`) — the main image transformer. * **Text encoder** (`Qwen3-4B-Q4_K_M.gguf`) — Qwen3 4B in standard GGML Q4\_K\_M format. * **VAE** (`flux2-vae.safetensors`) — standard safetensors format, compatible as-is. ### Model file reference — FLUX.2-klein 4B | Role | File | Source | | --------------- | --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | Diffusion model | `flux-2-klein-4b-Q8_0.gguf` | [leejet/FLUX.2-klein-4B-GGUF](https://huggingface.co/leejet/FLUX.2-klein-4B-GGUF) | | Text encoder | `Qwen3-4B-Q4_K_M.gguf` | [unsloth/Qwen3-4B-GGUF](https://huggingface.co/unsloth/Qwen3-4B-GGUF) | | VAE | `flux2-vae.safetensors` | [black-forest-labs/FLUX.2-klein-4B](https://huggingface.co/black-forest-labs/FLUX.2-klein-4B/blob/main/vae/diffusion_pytorch_model.safetensors) | ### Stable Diffusion * **Stable Diffusion 1.x / 2.x** — all-in-one checkpoint as a single `*.gguf` file * **Stable Diffusion XL** — all-in-one `*.gguf` or split CLIP encoders * **Stable Diffusion 3** — safetensors with separate CLIP encoders ## Requirements * Memory: 16 GB unified memory on Apple Silicon, or 8 GB VRAM on GPU. * Bare `>= v1.24` ## Installation ```bash npm i @qvac/diffusion-cpp ``` ## Quickstart If you don't have Bare runtime, install it: ```bash npm i -g bare ``` Create a new project: ```bash mkdir qvac-diffusion-quickstart cd qvac-diffusion-quickstart npm init -y ``` Install dependencies: ```bash npm i @qvac/diffusion-cpp bare-path bare-process bare-fs ``` Download the FLUX.2 \[klein] 4B model files (\~6.8 GB total): ```bash mkdir -p models curl -L -C - -o models/flux-2-klein-4b-Q8_0.gguf \ https://huggingface.co/leejet/FLUX.2-klein-4B-GGUF/resolve/main/flux-2-klein-4b-Q8_0.gguf curl -L -C - -o models/Qwen3-4B-Q4_K_M.gguf \ https://huggingface.co/unsloth/Qwen3-4B-GGUF/resolve/main/Qwen3-4B-Q4_K_M.gguf curl -L -C - -o models/flux2-vae.safetensors \ https://huggingface.co/black-forest-labs/FLUX.2-klein-4B/resolve/main/vae/diffusion_pytorch_model.safetensors ``` Create `index.js`: ```js title="index.js" lineNumbers 'use strict' const path = require('bare-path') const fs = require('bare-fs') const process = require('bare-process') const ImgStableDiffusion = require('@qvac/diffusion-cpp') async function main () { const MODELS_DIR = path.resolve(__dirname, './models') const args = { logger: console, files: { model: path.join(MODELS_DIR, 'flux-2-klein-4b-Q8_0.gguf'), llm: path.join(MODELS_DIR, 'Qwen3-4B-Q4_K_M.gguf'), vae: path.join(MODELS_DIR, 'flux2-vae.safetensors') }, config: { threads: 8 } } const model = new ImgStableDiffusion(args) await model.load() try { const images = [] const response = await model.run({ prompt: 'a majestic red fox in a snowy forest, golden light, photorealistic', steps: 20, width: 512, height: 512, guidance: 3.5, seed: 42 }) await response .onUpdate(data => { if (data instanceof Uint8Array) { images.push(data) } else if (typeof data === 'string') { try { const tick = JSON.parse(data) if ('step' in tick) process.stdout.write(`\rStep ${tick.step}/${tick.total}`) } catch (_) {} } }) .await() console.log('\n') if (images.length > 0) { fs.writeFileSync('output.png', images[0]) console.log('Saved → output.png') } } catch (error) { console.error('Error occurred:', error.message || error) } finally { await model.unload() } } main().catch(error => { console.error('Fatal error:', error.message) process.exit(1) }) ``` Run `index.js`: ```bash bare index.js ``` ## Usage ### 1. Import the model class ```js const ImgStableDiffusion = require('@qvac/diffusion-cpp') ``` ### 2. Create the `args` object ```js const path = require('bare-path') const MODELS_DIR = path.resolve(__dirname, './models') const args = { logger: console, files: { model: path.join(MODELS_DIR, 'flux-2-klein-4b-Q8_0.gguf'), llm: path.join(MODELS_DIR, 'Qwen3-4B-Q4_K_M.gguf'), vae: path.join(MODELS_DIR, 'flux2-vae.safetensors') }, config: { threads: 8 }, opts: { stats: true } } ``` | Property | Required | Description | | -------------- | -------- | ------------------------------------------------------------------------------------------------ | | `files` | ✅ | Object of absolute paths to model files (see below) | | `files.model` | ✅ | Absolute path to diffusion model file (diffusion-only GGUF for FLUX.2; all-in-one for SD1.x/2.x) | | `files.clipL` | — | Absolute path to separate CLIP-L text encoder (SD3) | | `files.clipG` | — | Absolute path to separate CLIP-G text encoder (SDXL / SD3) | | `files.t5Xxl` | — | Absolute path to separate T5-XXL text encoder (SD3) | | `files.llm` | — | Absolute path to Qwen3 LLM text encoder (FLUX.2 \[klein]) | | `files.vae` | — | Absolute path to separate VAE file | | `files.esrgan` | — | Absolute path to ESRGAN upscaler model for post-generation upscale | | `config` | — | Native backend configuration object (see next section) | | `logger` | — | Logger instance for JS wrapper logs (e.g. `console`) | | `opts` | — | Additional options (e.g. `{ stats: true }`) | Native C++ logs are process-global. Configure native log routing once with `require('@qvac/diffusion-cpp/addonLogging').setLogger(...)`. ### 3. Configure the native backend (`args.config`) `config` is a field on the `args` object built in step 2 — there is no separate constructor argument. The native backend reads it during `load()`. ```js args.config = { threads: 8 // CPU threads for tensor operations (Metal handles GPU automatically) } ``` Config values are coerced to strings internally. Generation parameters (prompt, steps, seed, etc.) are JSON-serialized with their native types preserved. | Parameter | Type | Default | Description | | -------------------- | ----------------------------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | `threads` | number | auto | Number of CPU threads for model loading and CPU ops | | `type` | `'f32'` \| `'f16'` \| `'q4_0'` \| `'q8_0'` \| … | auto | Override weight quantisation type | | `rng` | `'cpu'` \| `'cuda'` \| `'std_default'` | `'cuda'` | RNG backend (`'cuda'` = philox RNG — not GPU-specific despite the name; recommended) | | `backend` | string | auto | Graph execution backend, globally or per module (for example `cuda0` or `diffusion=vulkan0,te=cpu`) | | `params_backend` | string | auto | Parameter residency, including `diffusion=cpu` or `diffusion=disk`. A `disk` value disables eager loading for the whole context — see the note below | | `max_vram` | number \| string | `0` | VRAM budget in GiB for graph-cut execution; supports per-device values such as `cuda0=6,vulkan0=4` | | `stream_layers` | `true` \| `false` | `false` | Stream diffusion layers from CPU RAM when graph cutting is active and diffusion parameters are CPU-backed | | `flash_attn` | `true` \| `false` | `false` | Enable flash attention (reduces memory) | | `upscaler_tile_size` | number | `128` | ESRGAN upscaler tile size | `clip_on_cpu`, `vae_on_cpu` and `control_net_cpu` were removed and now fail the load. If the option was `false`, remove it and change nothing else — it had no effect, and there is no replacement. If it was enabled, keep the previous parameter placement with `params_backend: 'te=cpu'` in place of `clip_on_cpu` and `params_backend: 'vae=cpu'` in place of `vae_on_cpu`. To move graph execution instead, use `backend` with the same module name. `control_net_cpu` has no parameter-residency equivalent — use `backend: 'controlnet=cpu'`. A `disk` value in `params_backend` turns off eager loading for the **entire** context, not just the module you name. The engine releases those weights at each phase boundary and re-reads them from the model file on every job, so the load cost is paid per generation rather than once. ### 4. Create a model instance ```js const model = new ImgStableDiffusion(args) ``` The constructor takes a single object containing `files`, `config`, `logger`, and `opts`. It stores configuration only — no memory is allocated yet. ### 5. Load the Model ```js await model.load() ``` This creates the native `sd_ctx_t` and loads all weights into memory. It can take 10–30 seconds depending on disk speed and model size. All model files must be passed as absolute paths via the `files` object. ### 6. Run Inference The primary API. Returns a `QvacResponse` that streams step-progress ticks and the final PNG: ```js const images = [] const response = await model.run({ prompt: 'a majestic red fox in a snowy forest, golden light, photorealistic', steps: 20, width: 512, height: 512, guidance: 3.5, seed: 42 }) await response .onUpdate(data => { if (data instanceof Uint8Array) { images.push(data) } else if (typeof data === 'string') { try { const tick = JSON.parse(data) if ('step' in tick) process.stdout.write(`\rStep ${tick.step}/${tick.total}`) } catch (_) {} } }) .await() require('bare-fs').writeFileSync('output.png', images[0]) ``` **Generation parameters:** | Parameter | Type | Default | Description | | ----------------- | --------------------------------- | ------- | ---------------------------------------------------------------------------------- | | `prompt` | string | — | Text prompt | | `negative_prompt` | string | `''` | Things to avoid in the output | | `width` | number | `512` | Output width in pixels (multiple of 8) | | `height` | number | `512` | Output height in pixels (multiple of 8) | | `steps` | number | `20` | Number of diffusion steps | | `guidance` | number | `3.5` | Distilled guidance scale (FLUX.2) | | `cfg_scale` | number | `7.0` | Classifier-free guidance scale (SD1.x / SD2.x) | | `sampling_method` | string | auto | Sampler name; auto-selects `euler` for FLUX.2, `euler_a` for SD1.x | | `scheduler` | string | auto | Scheduler; auto-selected per model family | | `seed` | number | `-1` | Random seed (-1 for random) | | `batch_count` | number | `1` | Number of images to generate | | `vae_tiling` | boolean | `false` | Enable VAE tiling (required for large images on 16 GB) | | `cache_preset` | string | — | Step-caching preset: `slow`, `medium`, `fast`, `ultra` | | `upscale` | boolean \| `{ repeats?: number }` | `false` | Post-generation ESRGAN upscale. Requires `files.esrgan`; `repeats` defaults to `1` | Do not set `sampling_method: 'euler_a'` for FLUX.2 models — it will produce random noise. Leave the field unset to let the library auto-select `euler` for flow-matching models. ### 7. Release Resources ```js await model.unload() ``` `unload()` calls `free_sd_ctx` which releases all GPU and CPU memory. The JS object can be safely garbage collected afterwards. ## More resources [Package at npm](https://www.npmjs.com/package/@qvac/diffusion-cpp) --- title: "@qvac/embed-llamacpp" canonical: https://docs.qvac.tether.io/ecosystem/addons/embed-llamacpp/ collection: "Ecosystem" --- # @qvac/embed-llamacpp (/ecosystem/addons/embed-llamacpp) ## Overview [Bare module](https://bare.pears.com) that adds support for text embeddings and RAG in QVAC using [`qvac-fabric-llm.cpp`](https://github.com/tetherto/qvac-fabric-llm.cpp) as the inference engine. ## Models You can load any [`llama.cpp`](https://github.com/ggml-org/llama.cpp)-compatible embeddings model. Model file format: `*.gguf`. ## Requirement Bare `>= v1.24` ## Installation ```bash npm i @qvac/embed-llamacpp ``` `@qvac/embed-llamacpp` links the shared `@qvac/fabric` runtime. `@qvac/fabric` ships its native runtime and ggml backends for each desktop host through a version-locked, `os`/`cpu` filtered optional dependency: | Host | Package | | ------------------- | --------------------------- | | linux-x64 (glibc) | `@qvac/fabric-linux-x64` | | linux-arm64 (glibc) | `@qvac/fabric-linux-arm64` | | darwin-arm64 | `@qvac/fabric-darwin-arm64` | | darwin-x64 | `@qvac/fabric-darwin-x64` | | win32-x64 | `@qvac/fabric-win32-x64` | Do not depend on the desktop platform packages directly. Supported installers are npm 7+, pnpm, bun, and Yarn Berry; Yarn v1 and `--omit=optional` installs skip the platform package and fail at require time with an error naming it. Mobile targets are cross-built, so no install host ever matches their `os`, and optional-dependency filtering can never select them. Mobile applications must declare `@qvac/fabric` and the target's fabric platform package as direct dependencies at the same exact version, one that satisfies the `@qvac/fabric` range `@qvac/embed-llamacpp` declares: | Target | Package | | ------------------------- | ---------------------------- | | android-arm64 | `@qvac/fabric-android-arm64` | | ios (device + simulators) | `@qvac/fabric-ios` | ```json { "dependencies": { "@qvac/embed-llamacpp": "x.y.z", "@qvac/fabric": "a.b.c", "@qvac/fabric-android-arm64": "a.b.c" } } ``` ## Quickstart If you don't have Bare runtime, install it: ```bash npm i -g bare ``` Create a new project: ```bash mkdir qvac-embed-quickstart cd qvac-embed-quickstart npm init -y ``` Install dependencies: ```bash npm i @qvac/embed-llamacpp bare-path ``` Download a compatible model: ```bash curl -L --create-dirs -o models/gte-large_fp16.gguf \ https://huggingface.co/ChristianAzinn/gte-large-gguf/resolve/main/gte-large_fp16.gguf ``` Create `index.js`: ```js title="index.js" lineNumbers 'use strict' const path = require('bare-path') const GGMLBert = require('@qvac/embed-llamacpp') async function main () { const modelName = 'gte-large_fp16.gguf' const dirPath = path.resolve('./models') const modelPath = path.join(dirPath, modelName) // 1. Configuring model settings const model = new GGMLBert({ files: { model: [modelPath] }, config: { device: 'gpu', gpu_layers: '25' }, logger: console, opts: { stats: true } }) // 2. Loading model await model.load() try { // 3. Generating embeddings const query = 'Hello, can you suggest a game I can play with my 1 year old daughter?' const response = await model.run(query) const embeddings = await response.await() console.log('Embeddings shape:', embeddings.length, 'x', embeddings[0].length) console.log('First few values of first embedding:') console.log(embeddings[0].slice(0, 5)) } catch (error) { const errorMessage = error?.message || error?.toString() || String(error) console.error('Error occurred:', errorMessage) console.error('Error details:', error) } finally { // 4. Cleaning up resources await model.unload() } } main().catch(console.error) ``` Run `index.js`: ```bash bare index.js ``` ## Usage ### 1. Import the Model Class ```js const GGMLBert = require('@qvac/embed-llamacpp') ``` ### 2. Create Local Model Paths The addon reads GGUF files directly from disk. Download the model, then pass absolute local paths to `files.model`. ```js const path = require('bare-path') const dirPath = path.resolve('./models') const modelName = 'gte-large_fp16.gguf' const modelPath = path.join(dirPath, modelName) ``` ### 3. Create the `args` obj ```js const args = { files: { model: [modelPath] }, config: { device: 'gpu', gpu_layers: '25' }, logger: console, opts: { stats: true } } ``` The `args` obj contains the following properties: * `files.model`: An array of absolute paths to the model file(s) on disk. For sharded models, provide every shard in order. * `config`: A dictionary of hyper-parameters used to tweak the behaviour of the model. * `logger`: This property is used to create a `QvacLogger` instance, which handles all logging functionality. * `opts.stats`: This flag determines whether to calculate inference stats. ### 4. Create `config` The `config` obj consists of a set of hyper-parameters which can be used to tweak the behaviour of the model.\ *All parameters must be strings.* ```js // an example of possible configuration const config = { device: 'gpu', gpu_layers: '99', batch_size: '1024', ctx_size: '512' } ``` | Parameter | Range / Type | Default | Description | | --------------- | -------------------------------------------------- | ------------- | ------------------------------------------------------------------------------------- | | device | `"gpu"` or `"cpu"` | `"gpu"` | Device to run inference on | | gpu\_layers | integer | 0 | Number of model layers to offload to GPU | | batch\_size | integer | 2048 | Tokens processed per batch | | ctx\_size | 0 – model-dependent | model default | Runtime context window in tokens | | pooling | `"none"`, `"mean"`, `"cls"`, `"last"`, or `"rank"` | model default | Pooling type for embeddings | | attention | `"causal"` or `"non-causal"` | model default | Attention type for embeddings | | embd\_normalize | integer | 2 | Embedding normalization (-1=none, 0=max abs int16, 1=taxicab, 2=euclidean, >2=p-norm) | | flash\_attn | `"on"`, `"off"`, or `"auto"` | `"auto"` | Enable/disable flash attention | | main-gpu | integer, `"integrated"`, or `"dedicated"` | — | GPU selection for multi-GPU systems | | verbosity | 0 – 3 (0=ERROR, 1=WARNING, 2=INFO, 3=DEBUG) | 0 | Logging verbosity level | #### IGPU/GPU selection logic: | Scenario | main-gpu not specified | main-gpu: `"dedicated"` | main-gpu: `"integrated"` | | ------------------------------ | --------------------------------- | ----------------------- | ------------------------ | | Devices considered | All GPUs (dedicated + integrated) | Only dedicated GPUs | Only integrated GPUs | | System with iGPU only | ✅ Uses iGPU | ❌ Falls back to CPU | ✅ Uses iGPU | | System with dedicated GPU only | ✅ Uses dedicated GPU | ✅ Uses dedicated GPU | ❌ Falls back to CPU | | System with both | ✅ Uses dedicated GPU (preferred) | ✅ Uses dedicated GPU | ✅ Uses integrated GPU | ### 5. Instantiate the model ```js const model = new GGMLBert(args) ``` ### 6. Load the model ```js await model.load() ``` `load()` reads the file(s) listed in `files.model` directly from disk and activates the model. The caller is responsible for ensuring the files already exist at those paths. ### 7. Generate embeddings for input sequence The model outputs a vector for the input sequence. ```js const query = 'Hello, can you suggest a game I can play with my 1 year old daughter?' const response = await model.run(query) const embeddings = await response.await() ``` ### 8. Release Resources Unload the model when finished: ```javascript try { await model.unload() } catch (error) { console.error('Failed to unload model:', error) } ``` ## More resources [Package at npm](https://www.npmjs.com/package/@qvac/embed-llamacpp) --- title: "@qvac/llm-llamacpp" canonical: https://docs.qvac.tether.io/ecosystem/addons/llm-llamacpp/ collection: "Ecosystem" --- # @qvac/llm-llamacpp (/ecosystem/addons/llm-llamacpp) ## Overview [Bare module](https://bare.pears.com) that adds support for text completion and multimodal prompts in QVAC using [`qvac-fabric-llm.cpp`](https://github.com/tetherto/qvac-fabric-llm.cpp) as the inference engine. ## Models You can load any [`llama.cpp`](https://github.com/ggml-org/llama.cpp)-compatible text-generation/chat model. Model file format: `*.gguf`. ## Requirement Bare `>= v1.24` ## Installation ```bash npm i @qvac/llm-llamacpp ``` `@qvac/llm-llamacpp` links the shared `@qvac/fabric` runtime. `@qvac/fabric` ships its native runtime and ggml backends for each desktop host through a version-locked, `os`/`cpu` filtered optional dependency: | Host | Package | | ------------------- | --------------------------- | | linux-x64 (glibc) | `@qvac/fabric-linux-x64` | | linux-arm64 (glibc) | `@qvac/fabric-linux-arm64` | | darwin-arm64 | `@qvac/fabric-darwin-arm64` | | darwin-x64 | `@qvac/fabric-darwin-x64` | | win32-x64 | `@qvac/fabric-win32-x64` | Do not depend on the desktop platform packages directly. Supported installers are npm 7+, pnpm, bun, and Yarn Berry; Yarn v1 and `--omit=optional` installs skip the platform package and fail at require time with an error naming it. Mobile targets are cross-built, so no install host ever matches their `os`, and optional-dependency filtering can never select them. Mobile applications must declare `@qvac/fabric` and the target's fabric platform package as direct dependencies at the same exact version, one that satisfies the `@qvac/fabric` range `@qvac/llm-llamacpp` declares: | Target | Package | | ------------------------- | ---------------------------- | | android-arm64 | `@qvac/fabric-android-arm64` | | ios (device + simulators) | `@qvac/fabric-ios` | ```json { "dependencies": { "@qvac/llm-llamacpp": "x.y.z", "@qvac/fabric": "a.b.c", "@qvac/fabric-android-arm64": "a.b.c" } } ``` ## Quickstart If you don't have Bare runtime, install it: ```bash npm i -g bare ``` Create a new project: ```bash mkdir qvac-llm-quickstart cd qvac-llm-quickstart npm init -y ``` Install dependencies: ```bash npm i @qvac/llm-llamacpp bare-path bare-process ``` Download a compatible model: ```bash curl -L --create-dirs -o models/Llama-3.2-1B-Instruct-Q4_0.gguf \ https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF/resolve/main/Llama-3.2-1B-Instruct-Q4_0.gguf ``` Create `index.js`: ```js title="index.js" lineNumbers 'use strict' const LlmLlamacpp = require('@qvac/llm-llamacpp') const path = require('bare-path') const process = require('bare-process') async function main () { const modelName = 'Llama-3.2-1B-Instruct-Q4_0.gguf' const dirPath = path.resolve('./models') const modelPath = path.join(dirPath, modelName) // 1. Configuring model settings const config = { device: 'gpu', gpu_layers: '999', ctx_size: '1024' } // 2. Loading model const model = new LlmLlamacpp({ files: { model: [modelPath] }, config, logger: console, opts: { stats: true } }) await model.load() try { // 3. Running inference with conversation prompt const prompt = [ { role: 'system', content: 'You are a helpful, respectful and honest assistant.' }, { role: 'user', content: 'what is bitcoin?' }, { role: 'assistant', content: "It's a digital currency." }, { role: 'user', content: 'Can you elaborate on the previous topic?' } ] const response = await model.run(prompt) let fullResponse = '' await response .onUpdate(data => { process.stdout.write(data) fullResponse += data }) .await() console.log('\n') console.log('Full response:\n', fullResponse) console.log(`Inference stats: ${JSON.stringify(response.stats)}`) } catch (error) { const errorMessage = error?.message || error?.toString() || String(error) console.error('Error occurred:', errorMessage) console.error('Error details:', error) } finally { // 4. Cleaning up resources await model.unload() } } main().catch(error => { console.error('Fatal error in main function:', { error: error.message, stack: error.stack, timestamp: new Date().toISOString() }) process.exit(1) }) ``` Run `index.js`: ```bash bare index.js ``` ## Usage ### 1. Import the Model Class ```js const LlmLlamacpp = require('@qvac/llm-llamacpp') ``` ### 2. Create Local Model Paths The addon reads GGUF files directly from disk. Download the model, then pass absolute local paths to `files.model`. ```js const path = require('bare-path') const dirPath = path.resolve('./models') const modelName = 'Llama-3.2-1B-Instruct-Q4_0.gguf' const modelPath = path.join(dirPath, modelName) ``` ### 3. Create the `args` obj ```js // a minimal config; see step 4 for all available options const config = { gpu_layers: '99', ctx_size: '1024', device: 'cpu' } const args = { files: { model: [modelPath], // projectionModel: path.join(dirPath, 'mmproj-SmolVLM2-500M-Video-Instruct-Q8_0.gguf') // for multimodal support pass the projection model path }, config, opts: { stats: true }, logger: console } ``` The `args` obj contains the following properties: * `files.model`: Required. An array of absolute paths to the GGUF model file(s) to load. For sharded models, provide every shard and companion file in order. * `files.projectionModel`: Optional. Absolute path to the projection model file. This is required for multimodal support. * `config`: The model configuration object. * `logger`: This property is used to create a `QvacLogger` instance, which handles all logging functionality. * `opts.stats`: This flag determines whether to calculate inference stats. ### 4. Create the `config` obj The `config` obj consists of a set of hyper-parameters which can be used to tweak the behaviour of the model.\ *All parameters must be strings.* ```js // an example of possible configuration const config = { gpu_layers: '99', // number of model layers offloaded to GPU. ctx_size: '1024', // context length device: 'cpu', // must be specified: 'gpu' or 'cpu' else it will throw an error load_mode: 'none' // read fully into memory: no mmap, mlock or direct I/O } ``` | Parameter | Range / Type | Default | Description | | ------------------ | --------------------------------------------------------- | ---------------------------- | -------------------------------------------------- | | device | `"gpu"` or `"cpu"` | — (required) | Device to run inference on | | gpu\_layers | integer | 0 | Number of model layers to offload to GPU | | ctx\_size | 0 – model-dependent | 4096 (0 = loaded from model) | Context window size | | lora | string | — | Path to LoRA adapter file | | temp | 0.00 – 2.00 | 0.8 | Sampling temperature | | top\_p | 0 – 1 | 0.9 | Top-p (nucleus) sampling | | top\_k | 0 – 128 | 40 | Top-k sampling | | predict | integer (-1 = infinity) | -1 | Maximum tokens to predict | | seed | integer | -1 (random) | Random seed for sampling | | load\_mode | `"none"`, `"mmap"`, `"mlock"`, `"mmap+mlock"`, or `"dio"` | `"mmap"` | Select the model loading mode | | reverse\_prompt | string (comma-separated) | — | Stop generation when these strings are encountered | | repeat\_penalty | float | 1.1 | Repetition penalty | | presence\_penalty | float | 0 | Presence penalty for sampling | | frequency\_penalty | float | 0 | Frequency penalty for sampling | | tools | `"true"` or `"false"` | `"false"` | Enable tool calling with jinja templating | | verbosity | 0 – 3 (0=ERROR, 1=WARNING, 2=INFO, 3=DEBUG) | 0 | Logging verbosity level | | main-gpu | integer, `"integrated"`, or `"dedicated"` | — | GPU selection for multi-GPU systems | #### IGPU/GPU selection logic: | Scenario | main-gpu not specified | main-gpu: `"dedicated"` | main-gpu: `"integrated"` | | ------------------------------ | --------------------------------- | ----------------------- | ------------------------ | | Devices considered | All GPUs (dedicated + integrated) | Only dedicated GPUs | Only integrated GPUs | | System with iGPU only | ✅ Uses iGPU | ❌ Falls back to CPU | ✅ Uses iGPU | | System with dedicated GPU only | ✅ Uses dedicated GPU | ✅ Uses dedicated GPU | ❌ Falls back to CPU | | System with both | ✅ Uses dedicated GPU (preferred) | ✅ Uses dedicated GPU | ✅ Uses integrated GPU | ### 5. Create Model Instance ```js const model = new LlmLlamacpp(args) ``` ### 6. Load Model ```js await model.load() ``` Loads the model file(s) passed in `files.model` and activates the native addon. If a projection model was provided (`files.projectionModel`), it is loaded as part of the same step. ### 7. Run Inference Pass an array of messages (following the chat completion format) to the `run` method. Process the generated tokens asynchronously: ```javascript try { const messages = [ { role: 'system', content: 'You are a helpful assistant.' }, { role: 'user', content: 'What is the capital of France?' } ] const response = await model.run(messages) const buffer = [] // Option 1: Process streamed output using async iterator for await (const token of response.iterate()) { process.stdout.write(token) // Write token directly to output buffer.push(token) } // Option 2: Process streamed output using callback await response.onUpdate(token => ).await() console.log('\n--- Full Response ---\n', buffer.join('')) } catch (error) { console.error('Inference failed:', error) } ``` ### 8. Release Resources Unload the model when finished: ```javascript try { await model.unload() } catch (error) { console.error('Failed to unload model:', error) } ``` ## More resources [Package at npm](https://www.npmjs.com/package/@qvac/llm-llamacpp) --- title: "@qvac/transcription-parakeet" canonical: https://docs.qvac.tether.io/ecosystem/addons/transcription-parakeet/ collection: "Ecosystem" --- # @qvac/transcription-parakeet (/ecosystem/addons/transcription-parakeet) ## Overview [Bare module](https://bare.pears.com) that adds support for transcription in QVAC using [NVIDIA Parakeet](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2) ASR models via [ONNX Runtime](https://onnxruntime.ai) as the inference engine. Parakeet supports multiple model variants: * **TDT** — multilingual transcription (\~25 languages) with automatic language detection. * **CTC** — english-only, fast transcription with punctuation and capitalization. * **EOU** — real-time streaming with end-of-utterance detection (optimized for low latency). * **Sortformer** — speaker diarization (up to 4 speakers). ## Models Parakeet uses multiple model files depending on the variant: **TDT (multilingual):** * Encoder ONNX model * Encoder data file * Decoder ONNX model * Vocabulary file * Preprocessor ONNX model **CTC (English-only):** * Model ONNX file * Model data file * Tokenizer file **Sortformer (diarization):** * Single ONNX model file **EOU (streaming):** * Encoder ONNX model * Decoder ONNX model * Tokenizer file Model files are available from Hugging Face: * **CTC:** [parakeet-ctc-0.6b-ONNX](https://huggingface.co/onnx-community/parakeet-ctc-0.6b-ONNX/tree/main/onnx) * **TDT:** [parakeet-tdt-0.6b-v3-onnx](https://huggingface.co/istupakov/parakeet-tdt-0.6b-v3-onnx) * **EOU:** [parakeet-rs realtime\_eou\_120m-v1](https://huggingface.co/nvidia/parakeet_realtime_eou_120m-v1) * **Sortformer:** [parakeet-rs sortformer](https://huggingface.co/altunenes/parakeet-rs) ## Requirement Bare `>= v1.20` ## Installation ```bash npm i @qvac/transcription-parakeet ``` ## Quickstart If you don't have Bare runtime, install it: ```bash npm i -g bare ``` Create a new project: ```bash mkdir qvac-parakeet-quickstart cd qvac-parakeet-quickstart npm init -y ``` Install dependencies: ```bash npm i @qvac/dl-filesystem @qvac/transcription-parakeet bare-path bare-process ``` Download the TDT model files and place them in `models/parakeet-tdt-0.6b-v3-onnx/`: * `encoder.onnx` * `encoder.onnx_data` * `decoder.onnx` * `vocab.txt` * `preprocessor.onnx` Download from [Hugging Face](https://huggingface.co/istupakov/parakeet-tdt-0.6b-v3-onnx). Create `index.js`: ```js title="index.js" lineNumbers 'use strict' const path = require('bare-path') const process = require('bare-process') const binding = require('@qvac/transcription-parakeet/binding') const { ParakeetInterface } = require('@qvac/transcription-parakeet/parakeet') async function main () { const modelPath = path.join('.', 'models', 'parakeet-tdt-0.6b-v3-onnx') const audioPath = path.join('.', 'my-audio.wav') const config = { modelPath, modelType: 'tdt', maxThreads: 4, useGPU: false } const transcriptions = [] const outputCallback = (handle, event, data, error) => { if (event === 'transcription' && data && data.text) { transcriptions.push(data.text) } } const parakeet = new ParakeetInterface(binding, config, outputCallback) await parakeet.loadWeights() await parakeet.activate() const fs = require('bare-fs') const audioBuffer = fs.readFileSync(audioPath) const audioData = audioBuffer.subarray(44) // Skip WAV header await parakeet.append({ type: 'audio', data: audioData.buffer }) await parakeet.append({ type: 'end of job' }) // Wait briefly for processing await new Promise(resolve => setTimeout(resolve, 5000)) console.log('=== TRANSCRIPTION ===') console.log(transcriptions.join(' ')) console.log('=====================') await parakeet.destroyInstance() } main().catch(err => { console.error(err) process.exit(1) }) ``` Run `index.js`: ```bash bare index.js ``` ## Usage ### 1. Choose a Data Loader First, select and instantiate a data loader that provides access to model files: ```javascript // Option A: Filesystem Data Loader - for local model files const FilesystemDL = require('@qvac/dl-filesystem') const fsDL = new FilesystemDL({ dirPath: './path/to/model/files' }) // Option B: Hyperdrive Data Loader - for peer-to-peer distributed models const HyperDriveDL = require('@qvac/dl-hyperdrive') const hdDL = new HyperDriveDL({ key: 'hd://', store: corestore }) ``` ### 2. Configure Parakeet Parameters The addon accepts the following configuration: | Key | Type | Description | | ------------ | ------- | -------------------------------------------- | | `modelPath` | string | Path to the model directory | | `modelType` | string | `'ctc'`, `'tdt'`, `'eou'`, or `'sortformer'` | | `maxThreads` | number | Maximum CPU threads to use | | `useGPU` | boolean | Enable GPU acceleration | | `language` | string | Language code or `'auto'` (TDT only) | ### 3. Configuration Example ```javascript // TDT (multilingual, recommended) const config = { modelPath: './models/parakeet-tdt-0.6b-v3-onnx', modelType: 'tdt', maxThreads: 4, useGPU: false, language: 'auto' } // CTC (English-only, fastest) const ctcConfig = { modelPath: './models/parakeet-ctc-0.6b-ONNX', modelType: 'ctc', maxThreads: 4, useGPU: false } // Sortformer (speaker diarization) const sortformerConfig = { modelPath: './models/sortformer', modelType: 'sortformer', maxThreads: 4, useGPU: false } ``` ### 4. Create Model Instance ```javascript const binding = require('@qvac/transcription-parakeet/binding') const { ParakeetInterface } = require('@qvac/transcription-parakeet/parakeet') const outputCallback = (handle, event, data, error) => { if (event === 'transcription' && data && data.text) { console.log('Transcription:', data.text) } } const parakeet = new ParakeetInterface(binding, config, outputCallback) ``` ### 5. Load Model Load model weights and activate the inference engine: ```javascript try { await parakeet.loadWeights() await parakeet.activate() } catch (error) { console.error('Failed to load model:', error) } ``` ### 6. Run Transcription Pass audio data to the model for transcription: ```javascript try { const fs = require('bare-fs') const audioBuffer = fs.readFileSync('path/to/your/audio.wav') const audioData = audioBuffer.subarray(44) // Skip WAV header for raw PCM await parakeet.append({ type: 'audio', data: audioData.buffer }) await parakeet.append({ type: 'end of job' }) } catch (error) { console.error('Transcription failed:', error) } ``` **Output Callback Events:** * **`transcription`** — partial or complete transcription result (`data.text`, `data.confidence`, `data.isFinal`) * **`progress`** — processing progress (`data.percent`, `data.timeElapsed`) * **`diarization`** — speaker identification (`data.speakerId`, `data.startTime`, `data.endTime`) * **`complete`** — job completed successfully * **`error`** — error occurred (`error` string) ### 7. Release Resources Always destroy the instance when finished to free memory and resources: ```javascript try { await parakeet.destroyInstance() } catch (error) { console.error('Failed to destroy instance:', error) } ``` ## More resources [Package at npm](https://www.npmjs.com/package/@qvac/transcription-parakeet) --- title: "@qvac/transcription-whispercpp" canonical: https://docs.qvac.tether.io/ecosystem/addons/transcription-whispercpp/ collection: "Ecosystem" --- # @qvac/transcription-whispercpp (/ecosystem/addons/transcription-whispercpp) ## Overview [Bare module](https://bare.pears.com) that adds support for transcription in QVAC using [`qvac-fabric-speech.cpp`](https://github.com/tetherto/qvac-fabric-speech.cpp) as the inference engine. ## Models You should load two models: * a [`whisper.cpp`](https://github.com/ggml-org/whisper.cpp)-compatible model for transcription. Model file format: `*.bin`; and * a VAD model (e.g., Silero) converted to GGML. Model file format: `*.bin` *(optional, recommended)*. ## Requirement Bare `>= v1.24` ## Installation ```bash npm i @qvac/transcription-whispercpp ``` ## Quickstart If you don't have Bare runtime, install it: ```bash npm i -g bare ``` Create a new project: ```bash mkdir qvac-transcription-quickstart cd qvac-transcription-quickstart npm init -y ``` Install dependencies: ```bash npm i @qvac/transcription-whispercpp bare-fs bare-process ``` Download models and place them in `models/`: * A Whisper model (e.g., `ggml-tiny.bin`) from [Hugging Face](https://huggingface.co/ggerganov/whisper.cpp/tree/main) * *(Optional)* A Silero VAD model (`ggml-silero-v5.1.2.bin`) Create `index.js`: ```js title="index.js" lineNumbers 'use strict' const fs = require('bare-fs') const path = require('bare-path') const process = require('bare-process') const TranscriptionWhispercpp = require('@qvac/transcription-whispercpp') async function main () { const modelsDir = './models' const audioFilePath = './my-audio.raw' // 1. Constructor arguments — point directly at local model files const constructorArgs = { files: { model: path.join(modelsDir, 'ggml-tiny.bin'), vadModel: path.join(modelsDir, 'ggml-silero-v5.1.2.bin') }, opts: { stats: true } } // 2. Configuration object const config = { whisperConfig: { audio_format: 's16le', vad_params: { threshold: 0.35, min_speech_duration_ms: 200, min_silence_duration_ms: 150, max_speech_duration_s: 30, speech_pad_ms: 600, samples_overlap: 0.3 }, language: '' } } // 3. Loading model const model = new TranscriptionWhispercpp(constructorArgs, config) await model.load() // 4. Running transcription const bitRate = 128000 const bytesPerSecond = bitRate / 8 const audioStream = fs.createReadStream(audioFilePath, { highWaterMark: bytesPerSecond }) const response = await model.run(audioStream) const full = [] response.onUpdate((outputArr) => { const items = Array.isArray(outputArr) ? outputArr : [outputArr] const last = items[items.length - 1] if (last && last.text) console.log('[onUpdate]', last.start, '→', last.end, last.text) }) for await (const output of response.iterate()) { const items = Array.isArray(output) ? output : [output] full.push(...items) } if (full.length) { const text = full.map(s => s.text).join(' ').trim() console.log('\n=== TRANSCRIPTION ===') console.log(text) console.log('=====================\n') } else { console.log('No transcription output received.') } // 5. Cleaning up resources await model.destroy() } main().catch(err => { console.error(err) process.exit(1) }) ``` Run `index.js`: ```bash bare index.js ``` ## Usage ### 1. Provide Model Files The addon loads model weights directly from local file paths. Make sure the whisper model (and optional VAD model) already exist on disk, then pass their paths via the `files` field of the constructor arguments: ```javascript const path = require('bare-path') const constructorArgs = { files: { model: path.join('./models', 'ggml-tiny.bin'), // whisper model weights vadModel: path.join('./models', 'ggml-silero-v5.1.2.bin') // optional VAD model } } ``` #### Fetching model files from the QVAC model registry If you don't want to stage files manually, use [`@qvac/registry-client`](https://www.npmjs.com/package/@qvac/registry-client) to download them to disk first, then pass the resulting paths to the constructor: ```javascript const path = require('bare-path') const { QVACRegistryClient } = require('@qvac/registry-client') async function ensureModels (outputDir) { const client = new QVACRegistryClient() await client.ready() try { const modelPath = path.join(outputDir, 'ggml-tiny.bin') await client.downloadModel( 'qvac_models_compiled/whisper//ggml-tiny.bin', // registry path 's3', // registry source { outputFile: modelPath, timeout: 600000 } ) return { model: modelPath } } finally { await client.close() } } ``` ### 2. Configure Transcription Parameters Most users interact with the addon exclusively through `index.js`. From that entrypoint we surface a small, safe subset of options; everything else keeps whisper.cpp defaults. #### What index.js accepts | Section | Key | Description | | --------------- | --------------------------------- | ---------------------------------------------------------------------------------------------------------- | | `contextParams` | `model` | Absolute or relative path to the `.bin` whisper model | | | | *(all other context keys keep their defaults because changing them forces a full reload, see below)* | | `whisperConfig` | *(any `whisper_full_params` key)* | Forwarded untouched. We surface convenience defaults in `index.js`, but every whisper.cpp flag is accepted | | `miscConfig` | `caption_enabled` | Formats segments with `<\|start\|>..<\|end\|>` markers | #### Context keys that force a full reload Internally `WhisperModel::configContextIsChanged()` watches `model`, `use_gpu`, `flash_attn` and `gpu_device`. If any of these change we must: 1. Call `unload()` (destroys the current `whisper_context` and `whisper_state`). 2. Recreate the context via `whisper_init_from_file_with_params`. 3. Warm up the model again before the next job. Depending on model size this can take several seconds. Everything else in `whisperConfig`—language, temperatures, VAD settings, etc.—is applied in place and does **not** trigger a reload. If you are seeing unexpected pauses, double-check that you are not mutating these four context keys between jobs. #### Advanced configuration Need more than the handful of options exposed in `index.js`? The upstream whisper.cpp documentation lists every flag available through `whisper_full_params`. Rather than duplicating that matrix here, refer to: * The official parameter reference: [`whisper_full_params`](https://github.com/ggerganov/whisper.cpp/blob/master/examples/stream/stream.cpp#L30-L96) * Our longer examples for concrete shapes: * `examples/example.audio-ctx-chunking.js` (shows `offset_ms`, `duration_ms`, `audio_ctx`, and reload loops) * `examples/example.live-transcription.js` (shows streaming chunks into a single job) Those scripts stay in sync with the codebase and are the best place to copy from when you need the raw addon surface. ### 3. Configuration Example Quick JS-level configuration (what you typically pass to `new TranscriptionWhispercpp(...)`): ```javascript const config = { contextParams: { model: './models/ggml-tiny.bin' }, whisperConfig: { language: 'en', duration_ms: 0, temperature: 0.0, suppress_nst: true, n_threads: 0, vad_model_path: './models/ggml-silero-v5.1.2.bin', vadParams: { threshold: 0.6, min_speech_duration_ms: 250, min_silence_duration_ms: 200 } }, miscConfig: { caption_enabled: false } } ``` Between this minimal configuration and the example scripts you should have everything needed, whether you are wiring the addon by hand or just instantiating `TranscriptionWhispercpp`. **Available Whisper Models:** * `ggml-tiny.bin` - Smallest, fastest (39MB) * `ggml-base.bin` - Balanced size/accuracy (142MB) * `ggml-small.bin` - Better accuracy (466MB) * `ggml-medium.bin` - High accuracy (1.5GB) * `ggml-large.bin` - Best accuracy (3.1GB) **VAD Model:** * `ggml-silero-v5.1.2.bin` - Silero VAD model for voice activity detection Ensure the model files exist on disk before constructing the model. ### 4. Create Model Instance Import the specific Whisper model class based on the installed package and instantiate it: ```javascript const TranscriptionWhispercpp = require('@qvac/transcription-whispercpp') const model = new TranscriptionWhispercpp(constructorArgs, config) ``` Note : This import changes depending on the package installed. ### 5. Load Model Load the model weights and initialize the inference engine. Optionally provide a callback for progress updates: ```javascript try { // Basic usage await model.load() // Advanced usage with progress tracking await model.load( false, // reserved flag, kept for forwarding compatibility (progress) => console.log(`Loading: ${progress.overallProgress}% complete`) ) } catch (error) { console.error('Failed to load model:', error) } ``` **Progress Callback Data** The progress callback receives an object with the following properties: | Property | Type | Description | | --------------------- | ------ | -------------------------------------- | | `action` | string | Current operation being performed | | `totalSize` | number | Total bytes to be loaded | | `totalFiles` | number | Total number of files to process | | `filesProcessed` | number | Number of files completed so far | | `currentFile` | string | Name of file currently being processed | | `currentFileProgress` | string | Percentage progress on current file | | `overallProgress` | string | Overall loading progress percentage | ### 6. Run Transcription Pass an audio stream (e.g., from `bare-fs.createReadStream`) to the `run` method. Process the transcription results asynchronously. There are two ways to receive transcription results: #### Option 1: Real-time Streaming with `onUpdate()` The `onUpdate()` callback receives each transcription segment **in real-time** as whisper.cpp generates them during processing. This is ideal for live transcription display or progressive updates. ```javascript try { const audioStream = fs.createReadStream('path/to/your/audio.ogg', { highWaterMark: 16000 // Adjust based on bitrate (e.g., 128000 / 8) }) const response = await model.run(audioStream) // Receive segments as they are transcribed (real-time streaming) await response .onUpdate(segment => { console.log('New segment transcribed:', segment) // Each segment arrives immediately after whisper.cpp processes it }) .await() // Wait for transcription to complete console.log('Transcription finished!') } catch (error) { console.error('Transcription failed:', error) } ``` #### Option 2: Complete Result with `iterate()` The `iterate()` method returns all transcription segments **after the entire transcription completes**. This is useful when you need the full result before processing. ```javascript try { const audioStream = fs.createReadStream('path/to/your/audio.ogg', { highWaterMark: 16000 }) const response = await model.run(audioStream) // Wait for complete transcription, then iterate over all segments for await (const transcriptionChunk of response.iterate()) { console.log('Transcription chunk:', transcriptionChunk) } console.log('Transcription finished!') } catch (error) { console.error('Transcription failed:', error) } ``` **Key Differences:** * **`onUpdate()`**: Real-time streaming - segments arrive as they are generated by whisper.cpp's `new_segment_callback` * **`iterate()`**: Batch processing - all segments available after transcription completes #### Chunking long recordings with reload() `examples/example.audio-ctx-chunking.js` shows the production pattern: reuse a model instance, call `reload()` with `{ offset_ms, duration_ms, audio_ctx }` per chunk (first chunk uses `audio_ctx = 0`, subsequent ones clamp to \~1500), then run the full audio stream. The matching integration test (`test/integration/audio-ctx-chunking.test.js`) exercises exactly the same flow. #### Live streaming a single job `examples/example.live-transcription.js` feeds tiny PCM buffers into a pushable `Readable`, keeps a single `model.run(...)` open, and relies on `onUpdate()` for incremental text. `test/integration/live-stream-simulation.test.js` covers both the streaming case and a segmented loop without any `reload()` calls. ### 7. Release Resources Always unload the model when finished to free up memory and resources: ```javascript try { await model.unload() } catch (error) { console.error('Failed to unload model:', error) } ``` ## Decoder + VAD + Whisper Integration AddOn This package combines audio decoding, optional VAD trimming, and Whisper transcription into a single `TranscriptionFfmpegAddon`. It automatically: 1. Decodes or ingests raw PCM/encoded audio 2. (Optionally) applies Silero VAD to drop non-speech 3. Feeds speech segments to Whisper for transcription The principles are the same than for the single Whisper addon but with some differences in the configuration interface. ### Usage Import `TranscriptionFfmpegAddon` from the `transcription-ffmpeg.js` module: ```javascript const TranscriptionFfmpegAddon = require('@qvac/transcription-whispercpp/transcription-ffmpeg') ``` ### Configuration When you instantiate `TranscriptionFfmpegAddon`, pass: * `files.model`: path to the whisper model file (with optional `files.vadModel`) * `params.decoder.audioFormat`: one of * `'decoded'` (raw PCM input - for pre-decoded audio files) * `'encoded'` | `'s16le'` | `'f32le'` | `'mp3'` | `'wav'` | `'m4a'` (for encoded audio files) * `params.decoder.streamIndex`: stream index of the media file (default: 0) * `params.decoder.inputBitrate`: bitrate of the media file in bps (used to calculate buffer size) ### Usage Example See `examples/example.ffmpeg.js` for a full working script that demonstrates the FFmpeg decoder + Whisper transcription pipeline with encoded audio files (MP3, etc.). ### Additional Features * **Progress Tracking:** Monitor loading progress with callbacks * **Performance Stats:** Measure inference time with the `stats` option ## More resources [Package at npm](https://www.npmjs.com/package/@qvac/transcription-whispercpp) --- title: "@qvac/translation-nmtcpp" canonical: https://docs.qvac.tether.io/ecosystem/addons/translation-nmtcpp/ collection: "Ecosystem" --- # @qvac/translation-nmtcpp (/ecosystem/addons/translation-nmtcpp) > **Migration Note (v1.0.0+):** Opus/Marian model support has been removed. Only IndicTrans2 and Bergamot backends are supported. If you were using Opus models, migrate to Bergamot for European language pairs. ## Overview [Bare module](https://bare.pears.com) that adds support for translation in QVAC using either [`qvac-fabric-llm.cpp`](https://github.com/tetherto/qvac-fabric-llm.cpp) or [Bergamot](https://browser.mt) as the inference engine. ## Models You should load a model compatible with your chosen inference engine: * `qvac-fabric-llm.cpp` (default): IndicTrans2, converted to GGML. Model file format: `*.bin`. * Bergamot: Bergamot model bundle. Required files: model `*.bin` + `vocab*.spm`. ## Requirement Bare `>= v1.24` ## Installation ```bash npm i @qvac/translation-nmtcpp ``` `@qvac/translation-nmtcpp` links the shared `@qvac/fabric` runtime for both engines. `@qvac/fabric` ships its native runtime and ggml backends for each desktop host through a version-locked, `os`/`cpu` filtered optional dependency: | Host | Package | | ------------------- | --------------------------- | | linux-x64 (glibc) | `@qvac/fabric-linux-x64` | | linux-arm64 (glibc) | `@qvac/fabric-linux-arm64` | | darwin-arm64 | `@qvac/fabric-darwin-arm64` | | darwin-x64 | `@qvac/fabric-darwin-x64` | | win32-x64 | `@qvac/fabric-win32-x64` | Do not depend on the desktop platform packages directly. Supported installers are npm 7+, pnpm, bun, and Yarn Berry; Yarn v1 and `--omit=optional` installs skip the platform package and fail at require time with an error naming it. Mobile targets are cross-built, so no install host ever matches their `os`, and optional-dependency filtering can never select them. Mobile applications must declare `@qvac/fabric` and the target's fabric platform package as direct dependencies at the same exact version, one that satisfies the `@qvac/fabric` range `@qvac/translation-nmtcpp` declares: | Target | Package | | ------------------------- | ---------------------------- | | android-arm64 | `@qvac/fabric-android-arm64` | | ios (device + simulators) | `@qvac/fabric-ios` | ```json { "dependencies": { "@qvac/translation-nmtcpp": "x.y.z", "@qvac/fabric": "a.b.c", "@qvac/fabric-android-arm64": "a.b.c" } } ``` ## Quickstart If you don't have Bare runtime, install it: ```bash npm i -g bare ``` Create a new project: ```bash mkdir qvac-translation-quickstart cd qvac-translation-quickstart npm init -y ``` Install dependencies: ```bash npm i @qvac/translation-nmtcpp ``` Create `example.js`: ```js title="example.js" lineNumbers'use strict' /** * Quickstart Example — Bergamot Backend * * This example demonstrates translation using the Bergamot backend * with local model files or auto-download via Firefox CDN (English to Italian). * * Usage: * bare example.js * BERGAMOT_MODEL_PATH=/path/to/bergamot/model bare example.js * * Enable verbose C++ logging: * VERBOSE=1 bare example.js */ const TranslationNmtcpp = require('@qvac/translation-nmtcpp') const { ensureBergamotModelFiles, getBergamotFileNames } = require('@qvac/translation-nmtcpp/lib/bergamot-model-fetcher') const path = require('bare-path') const process = require('bare-process') // ============================================================ // LOGGING CONFIGURATION // Set VERBOSE=1 environment variable to enable C++ debug logs // ============================================================ const VERBOSE = process.env.VERBOSE === '1' || process.env.VERBOSE === 'true' const logger = VERBOSE ? { info: (msg) => console.log('[C++ INFO]', msg), warn: (msg) => console.warn('[C++ WARN]', msg), error: (msg) => console.error('[C++ ERROR]', msg), debug: (msg) => console.log('[C++ DEBUG]', msg) } : null // null = suppress all C++ logs const text = 'Machine translation has revolutionized how we communicate across language barriers in the modern digital world.' async function testBergamot () { console.log('\n=== Testing Bergamot Backend ===\n') const srcLang = 'en' const dstLang = 'it' // Use local model path if provided, otherwise auto-download const bergamotPath = process.env.BERGAMOT_MODEL_PATH || './model/bergamot/enit' // Ensure model files are present (downloads from Firefox CDN if not) const modelDir = await ensureBergamotModelFiles(srcLang, dstLang, bergamotPath) console.log('Model directory:', modelDir) const fileNames = getBergamotFileNames(srcLang, dstLang) console.log('Loading model...') // Create the model with resolved file paths const model = new TranslationNmtcpp({ files: { model: path.join(modelDir, fileNames.modelName), srcVocab: path.join(modelDir, fileNames.srcVocabName), dstVocab: path.join(modelDir, fileNames.dstVocabName) }, params: { mode: 'full', dstLang, srcLang }, config: { modelType: TranslationNmtcpp.ModelTypes.Bergamot }, logger // Pass the logger }) // Load model await model.load() console.log('Model loaded successfully!') try { console.log('Running translation...') console.log('Input text:', text) // Run the Model const response = await model.run(text) await response .onUpdate(data => { console.log('Translation output:', data) }) .await() console.log('Bergamot translation finished!') } finally { console.log('Unloading model...') await model.unload() console.log('Done!') } } async function main () { try { await testBergamot() console.log('\n=== All Tests Completed Successfully! ===\n') } catch (error) { console.error('Test failed:', error) throw error } } main() ``` Run `example.js`: ```bash bare example.js ``` ## Usage The library provides a straightforward and intuitive workflow for translating text. Irrespective of the chosen model, the workflow remains the same: ### 1. Obtain the model files The model class is **files-based**: you pass it the resolved on-disk paths of the model weights (and, for Bergamot, the vocab files). The package ships fetcher helpers that download the files on demand and return the directory they were written to: * **IndicTrans2** — `ensureIndicTransModelFile()` downloads the GGML model from the **QVAC model registry** (via `@qvac/registry-client`). * **Bergamot** — `ensureBergamotModelFiles()` downloads the model + vocab bundle from Mozilla's **Firefox Remote Settings CDN**. Both helpers are idempotent: if a valid file already exists at the destination, they return immediately without re-downloading. You can also point the model class at files you have downloaded yourself — the fetchers are a convenience, not a requirement. ```javascript // IndicTrans2 — fetch from the QVAC model registry const { ensureIndicTransModelFile, getIndicTransFileName } = require('@qvac/translation-nmtcpp/lib/indictrans-model-fetcher') const path = require('bare-path') const modelPath = path.join('./model/indictrans', getIndicTransFileName()) await ensureIndicTransModelFile(modelPath) // downloads if not already present ``` ```javascript // Bergamot — fetch from the Firefox Remote Settings CDN const { ensureBergamotModelFiles, getBergamotFileNames } = require('@qvac/translation-nmtcpp/lib/bergamot-model-fetcher') const modelDir = await ensureBergamotModelFiles('en', 'it', './model/bergamot/enit') const fileNames = getBergamotFileNames('en', 'it') // { modelName, srcVocabName, dstVocabName } ``` ### 2. Create the `args` object The model is constructed from a single options object: `files` (resolved paths from Step 1), `params` (languages and mode), `config` (model type and decoding options), and an optional `logger`. The shape of `files` varies slightly depending on which backend you're using. #### IndicTrans2 For Indic language translations (English ↔ Hindi, Bengali, Tamil, etc.) IndicTrans2 needs only the model weights file: ```javascript const args = { files: { model: modelPath // resolved path from Step 1 }, params: { mode: 'full', srcLang: 'eng_Latn', // Source language (ISO 15924 code) dstLang: 'hin_Deva' // Target language (ISO 15924 code) }, config: { modelType: TranslationNmtcpp.ModelTypes.IndicTrans } } ``` **Key Parameters:** | Parameter | Description | Example | | ------------- | ----------------------------------------------------------------- | ------------------------------------------------------------------- | | `srcLang` | Source language (ISO 15924) | `'eng_Latn'`, `'hin_Deva'`, `'ben_Beng'` | | `dstLang` | Target language (ISO 15924) | `'eng_Latn'`, `'hin_Deva'`, `'tam_Taml'` | | `files.model` | Path to the model weights file | `'./model/indictrans/ggml-indictrans2-en-indic-dist-200M-q4_0.bin'` | | `modelType` | **Required in config**: `TranslationNmtcpp.ModelTypes.IndicTrans` | - | **IndicTrans2 model naming pattern:** * `ggml-indictrans2-{direction}-{size}.bin` for q0f32 quantization * `ggml-indictrans2-{direction}-{size}-q0f16.bin` for q0f16 quantization * `ggml-indictrans2-{direction}-{size}-q4_0.bin` for q4\_0 quantization Where `direction` is `en-indic`, `indic-en`, or `indic-indic`, and `size` is `dist-200M`, `dist-320M`, or `1B`. #### Bergamot Bergamot needs the model weights plus a source and target vocabulary file. Use the paths returned by `ensureBergamotModelFiles()` / `getBergamotFileNames()`, or point at files you downloaded yourself: ```javascript const path = require('bare-path') const args = { files: { model: path.join(modelDir, fileNames.modelName), srcVocab: path.join(modelDir, fileNames.srcVocabName), dstVocab: path.join(modelDir, fileNames.dstVocabName) }, params: { mode: 'full', srcLang: 'en', // Source language (ISO 639-1 code) dstLang: 'it' // Target language (ISO 639-1 code) }, config: { modelType: TranslationNmtcpp.ModelTypes.Bergamot } } ``` **Bergamot Model Files by Language Pair:** | Language Pair | Model File | Vocab File(s) | | ------------- | ------------------------------- | ---------------------------------------- | | en→it | `model.enit.intgemm.alphas.bin` | `vocab.enit.spm` | | it→en | `model.iten.intgemm.alphas.bin` | `vocab.iten.spm` | | en→es | `model.enes.intgemm.alphas.bin` | `vocab.enes.spm` | | es→en | `model.esen.intgemm.alphas.bin` | `vocab.esen.spm` | | en→fr | `model.enfr.intgemm.alphas.bin` | `vocab.enfr.spm` | | fr→en | `model.fren.intgemm.alphas.bin` | (see Firefox Translations models) | | en→de | `model.ende.intgemm.alphas.bin` | `vocab.ende.spm` | | en→ru | `model.enru.intgemm.alphas.bin` | `vocab.enru.spm` | | ru→en | `model.ruen.intgemm.alphas.bin` | `vocab.ruen.spm` | | en→zh | `model.enzh.intgemm.alphas.bin` | `srcvocab.enzh.spm`, `trgvocab.enzh.spm` | | zh→en | `model.zhen.intgemm.alphas.bin` | `vocab.zhen.spm` | | en→ja | `model.enja.intgemm.alphas.bin` | `srcvocab.enja.spm`, `trgvocab.enja.spm` | | ja→en | `model.jaen.intgemm.alphas.bin` | `vocab.jaen.spm` | `getBergamotFileNames(srcLang, dstLang)` returns the correct `modelName`, `srcVocabName`, and `dstVocabName` for each pair (including the separate source/target vocabs used by CJK languages), so you normally don't need to hard-code these. **Key Parameters:** | Parameter | Description | Example | | ---------------- | --------------------------------------------------------------- | ------------------------------------------------------- | | `srcLang` | Source language (ISO 639-1) | `'en'`, `'es'`, `'de'` | | `dstLang` | Target language (ISO 639-1) | `'it'`, `'fr'`, `'de'` | | `files.model` | Path to the model weights file | `'./model/bergamot/enit/model.enit.intgemm.alphas.bin'` | | `files.srcVocab` | Path to the source vocabulary file | `'./model/bergamot/enit/vocab.enit.spm'` | | `files.dstVocab` | Path to the target vocabulary file | `'./model/bergamot/enit/vocab.enit.spm'` | | `modelType` | **Required in config**: `TranslationNmtcpp.ModelTypes.Bergamot` | - | **Bergamot model file naming convention:** * `model.{srctgt}.intgemm.alphas.bin` - Model weights (e.g., `model.enit.intgemm.alphas.bin`) * `vocab.{srctgt}.spm` - Shared vocabulary for most language pairs * `srcvocab.{srctgt}.spm` + `trgvocab.{srctgt}.spm` - Separate vocabs for CJK languages (zh, ja) Use a **unique directory per model** to avoid file conflicts when using multiple models: * `./model/indictrans` for IndicTrans English→Hindi * `./model/bergamot/enit` for Bergamot English→Italian The list of supported languages for the `srcLang` and `dstLang` parameters differ by model type. ### 3. Create the `config` object The `config` object contains two types of parameters: 1. **Model-specific parameters** (required for some backends) 2. **Generation/decoding parameters** (optional, controls output quality) #### Model-Specific Parameters | Parameter | IndicTrans2 | Bergamot | | ------------------ | ------------ | --------------------------------------------- | | `config.modelType` | **Required** | **Required** | | `files.srcVocab` | Not needed | **Required** (passed via `files`, see Step 2) | | `files.dstVocab` | Not needed | **Required** (passed via `files`, see Step 2) | #### Generation/Decoding Parameters (IndicTrans Only) These parameters control how the model generates output. **Note:** Full parameter support is only available for IndicTrans2 models. Bergamot has limited parameter support. ```javascript // Generation parameters for IndicTrans2 const generationParams = { beamsize: 4, // Beam search width (>=1). 1 disables beam search lengthpenalty: 0.6, // Length normalization strength (>=0) maxlength: 128, // Maximum generated tokens (>0) repetitionpenalty: 1.2, // Penalize previously generated tokens (0..2) norepeatngramsize: 2, // Disallow repeating n-grams of this size (0..10) temperature: 0.8, // Sampling temperature [0..2] topk: 40, // Keep top-K logits [0..vocab_size] topp: 0.9 // Nucleus sampling threshold (0 < p <= 1) } ``` ### 4. Create Model Instance Import `TranslationNmtcpp` and create an instance from the single options object built in Step 2. Decoding options from Step 3 are merged into `args.config`: ```javascript const TranslationNmtcpp = require('@qvac/translation-nmtcpp') ``` #### IndicTrans2 ```javascript // IndicTrans - must specify modelType + generation parameters const model = new TranslationNmtcpp({ ...args, // files + params from Step 2 config: { modelType: TranslationNmtcpp.ModelTypes.IndicTrans, ...generationParams, // Spread generation params from Step 3 maxlength: 256 // Override for longer outputs } }) ``` #### Bergamot ```javascript // Bergamot - vocab files are passed via `files` (see Step 2); limited generation params support const model = new TranslationNmtcpp({ ...args, // files (model + srcVocab + dstVocab) + params from Step 2 config: { modelType: TranslationNmtcpp.ModelTypes.Bergamot, beamsize: 4 // Only beamsize supported for Bergamot } }) ``` **Available Model Types:** ```javascript TranslationNmtcpp.ModelTypes = { IndicTrans: 'IndicTrans', // Indic language models Bergamot: 'Bergamot' // Firefox Translations models } ``` ### 5. Load Model ```javascript try { // Basic usage await model.load() } catch (error) { console.error('Failed to load model:', error) } ``` ### 6. Run the Model We can perform inference on the input text using the `run()` method. This method returns a [`QVACResponse`](#glossary) object. ```javascript try { // Execute translation on input text const response = await model.run('Hello world! Welcome to the internet of peers!') // Process streamed output using callback await response .onUpdate(outputChunk => { // Handle each new piece of translated text console.log(outputChunk) }) .await() // Wait for translation to complete // Access performance statistics (if enabled with opts.stats). The counters // accumulate over the life of the loaded model; difference consecutive // readings for per-run figures. if (response.stats) { console.log('Total translation time so far:', response.stats.totalTime, 's') } } catch (error) { console.error('Translation failed:', error) } ``` ### 7. Batch Translation (Bergamot Only) For translating multiple texts efficiently, use the `runBatch()` method instead of calling `run()` multiple times. `runBatch()` is only available with the **Bergamot backend**. IndicTrans2 models should use sequential `run()` calls. ```javascript // Array of texts to translate (English) const textsToTranslate = [ 'Hello world!', 'How are you today?', 'Machine translation has revolutionized communication.' ] try { // Batch translation - returns array of translated strings const translations = await model.runBatch(textsToTranslate) // Output each translation translations.forEach((translatedText, index) => { console.log(`Original: ${textsToTranslate[index]}`) console.log(`Translated: ${translatedText}\n`) }) } catch (error) { console.error('Batch translation failed:', error) } ``` **`runBatch()` vs `run()`:** | Method | Input | Output | Backend Support | | ----------------- | ---------------- | ----------------------------- | -------------------------- | | `run(text)` | Single string | `QVACResponse` with streaming | All (IndicTrans, Bergamot) | | `runBatch(texts)` | Array of strings | Array of strings | **Bergamot only** | `runBatch()` is significantly faster when translating multiple texts as it processes them in a single batch operation. ### 8. Unload the Model ```javascript // Always unload the model when finished to free memory try { await model.unload() } catch (error) { console.error('Failed to unload model:', error) } ``` ## Supported Languages ### IndicTrans2 Models (QVAC registry) IndicTrans2 supports translation between English and 22 Indic languages. The following directions are available via the QVAC model registry: | Direction | Available | Sizes | | --------------- | --------- | -------- | | English → Indic | Yes | 200M, 1B | | Indic → English | Yes | 200M, 1B | | Indic → Indic | Yes | 320M, 1B | **Supported Indic Languages:**
Assamese (asm_Beng) Kashmiri (Arabic) (kas_Arab) Punjabi (pan_Guru)
Bengali (ben_Beng) Kashmiri (Devanagari) (kas_Deva) Sanskrit (san_Deva)
Bodo (brx_Deva) Maithili (mai_Deva) Santali (sat_Olck)
Dogri (doi_Deva) Malayalam (mal_Mlym) Sindhi (Arabic) (snd_Arab)
English (eng_Latn) Marathi (mar_Deva) Sindhi (Devanagari) (snd_Deva)
Konkani (gom_Deva) Manipuri (Bengali) (mni_Beng) Tamil (tam_Taml)
Gujarati (guj_Gujr) Manipuri (Meitei) (mni_Mtei) Telugu (tel_Telu)
Hindi (hin_Deva) Nepali (npi_Deva) Urdu (urd_Arab)
Kannada (kan_Knda) Odia (ory_Orya)
### Bergamot Models (Firefox Translations) **Language pairs available via the Firefox Remote Settings CDN:** | Language | Code | en→X | X→en | | ---------- | ---- | ---- | ---- | | Arabic | ar | Yes | Yes | | Czech | cs | Yes | Yes | | Spanish | es | Yes | Yes | | French | fr | Yes | Yes | | Italian | it | Yes | Yes | | Japanese | ja | Yes | Yes | | Portuguese | pt | Yes | Yes | | Russian | ru | Yes | Yes | | Chinese | zh | Yes | Yes | The Bergamot backend supports all language pairs available in [Firefox Translations](https://github.com/mozilla/firefox-translations-models). See the Firefox Translations models repository for the complete and up-to-date list of supported language pairs. ## ModelClasses and Packages ### ModelClass The main class exported by this library is `TranslationNmtcpp`, which supports multiple translation backends: ```javascript const TranslationNmtcpp = require('@qvac/translation-nmtcpp') // Available model types TranslationNmtcpp.ModelTypes = { IndicTrans: 'IndicTrans', // For Indic language translations Bergamot: 'Bergamot' // For Bergamot/Firefox translations } ``` ### Available Packages #### Main Package | Package | Description | Backends | Languages | | -------------------------- | ------------------------ | -------------------- | ----------------------------------------------- | | `@qvac/translation-nmtcpp` | Main translation package | Bergamot, IndicTrans | See [Supported Languages](#supported-languages) | The main package supports both backends and all their respective languages. See [Supported Languages](#supported-languages) for the complete list. ## Logging The library supports configurable logging for both JavaScript and C++ (native) components. By default, C++ logs are suppressed for cleaner output. ### Enabling C++ Logs To enable verbose C++ logging, pass a `logger` object in the options: ```javascript // Enable C++ logging const logger = { info: (msg) => console.log('[C++ INFO]', msg), warn: (msg) => console.warn('[C++ WARN]', msg), error: (msg) => console.error('[C++ ERROR]', msg), debug: (msg) => console.log('[C++ DEBUG]', msg) } const args = { files: { model: modelPath, srcVocab, dstVocab }, // resolved paths (see Step 1) params: { mode: 'full', srcLang: 'en', dstLang: 'it' }, config: { modelType: TranslationNmtcpp.ModelTypes.Bergamot }, logger // Pass logger to enable C++ logs } ``` ### Disabling C++ Logs To suppress all C++ logs, either omit the `logger` parameter or set it to `null`: ```javascript const args = { files: { model: modelPath, srcVocab, dstVocab }, // resolved paths (see Step 1) params: { mode: 'full', srcLang: 'en', dstLang: 'it' }, config: { modelType: TranslationNmtcpp.ModelTypes.Bergamot } // No logger = suppress C++ logs } ``` ### Using Environment Variables (Recommended for Examples) All examples support the `VERBOSE` environment variable: ```bash # Run with C++ logging disabled (default) bare examples/quickstart.js # Run with C++ logging enabled VERBOSE=1 bare examples/quickstart.js ``` ### Log Levels The C++ backend supports these log levels (mapped from native priority): | Priority | Level | Description | | -------- | ------- | ---------------------- | | 0 | `error` | Critical errors | | 1 | `warn` | Warnings | | 2 | `info` | Informational messages | | 3 | `debug` | Debug/trace messages | ## More resources [Package at npm](https://www.npmjs.com/package/@qvac/translation-nmtcpp) --- title: "@qvac/tts-ggml" canonical: https://docs.qvac.tether.io/ecosystem/addons/tts-ggml/ collection: "Ecosystem" --- # @qvac/tts-ggml (/ecosystem/addons/tts-ggml) ## Overview [Bare module](https://bare.pears.com) that adds support for text-to-speech in QVAC, backed by the [`qvac-tts.cpp`](https://github.com/tetherto/qvac-fabric-speech.cpp/tree/master/engines/tts) GGML library. It runs in-process with a persistent native engine — the GGUFs, the S3Gen preload, the ggml backend, and any voice-conditioning tensors are loaded once and reused across every synthesis call. GPU acceleration (Metal on macOS/iOS, CUDA on NVIDIA Linux x64, Vulkan / OpenCL on Linux/Windows/Android) is **opt-in** via `config: { useGPU: true }`; the default is CPU. ## Models Six engine families are wrapped — **Chatterbox**, **Supertonic**, **Parler-TTS**, **CosyVoice3**, **Audio8**, and **MOSS** — each with its own model layout under `models/`: | Model | GGUF files | Languages / notes | | ----------------------- | ------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------- | | Chatterbox Turbo | `chatterbox-t3-turbo.gguf`, `chatterbox-s3gen.gguf` | English; voice cloning | | Chatterbox multilingual | `chatterbox-t3-mtl.gguf`, `chatterbox-s3gen-mtl.gguf` | 23 languages (see below); voice cloning | | Supertonic | `supertonic.gguf` | English; voice baked in | | Supertonic 2 | `supertonic2.gguf` | Multilingual: `en`, `ko`, `es`, `pt`, `fr` | | Supertonic 3 | `supertonic3.gguf` | Multilingual: 31 languages plus the language-agnostic `na` code (see below) | | Parler-TTS Mini v1 | `parler-tts-mini-v1.gguf` | English; description-conditioned voices | | CosyVoice3 | `cosyvoice3/` dir (`cosyvoice3-{llm,flow,hift}-*.gguf` + `voice.gguf` + tokenizer files) | Instruct-conditioned; strong on Chinese + 17 dialects; native 24 kHz | | Audio8 | `audio8-lm-q8_0.gguf` + `audio8-codec-decoder-q8_0.gguf` (+ `audio8-codec-encoder-q8_0.gguf` to clone) | Multilingual (language inferred from the text); voice cloning; native 44.1 kHz | | MOSS-TTS v1.5 | `moss-tts-delay-f16.gguf` + `moss-codec-decoder-f16.gguf` (+ `moss-codec-encoder-f16.gguf` to clone) | Multilingual (`language` hint); directable pauses, duration and pronunciation; voice cloning; native chunk streaming; native 24 kHz; 8B, desktop only | | MOSS-TTSD | `moss-ttsd-f16.gguf` + `moss-codec-decoder-f16.gguf` + `moss-codec-encoder-f16.gguf` | Multi-speaker dialogue, one reference recording per speaker; native 24 kHz; 8B, desktop only | **Chatterbox multilingual (23 languages):** Arabic (`ar`), Danish (`da`), German (`de`), Greek (`el`), English (`en`), Spanish (`es`), Finnish (`fi`), French (`fr`), Hebrew (`he`), Hindi (`hi`), Italian (`it`), Japanese (`ja`), Korean (`ko`), Malay (`ms`), Dutch (`nl`), Norwegian (`no`), Polish (`pl`), Portuguese (`pt`), Russian (`ru`), Swedish (`sv`), Swahili (`sw`), Turkish (`tr`), Chinese (`zh`). **Supertonic 3 (31 languages + `na`):** `ar`, `bg`, `hr`, `cs`, `da`, `nl`, `en`, `et`, `fi`, `fr`, `de`, `el`, `hi`, `hu`, `id`, `it`, `ja`, `ko`, `lv`, `lt`, `pl`, `pt`, `ro`, `ru`, `sk`, `sl`, `es`, `sv`, `tr`, `uk`, `vi`. Pass `language: 'na'` when the input language is unknown. Point the addon at a custom location via `files.modelDir` (engine auto-detected from the GGUF filenames present), or pass explicit `files.t3Model` + `files.s3genModel` (Chatterbox), `files.supertonicModel` (Supertonic), `files.parlerModel` (Parler-TTS), `files.cosyvoiceModelDir` (CosyVoice3), `files.audio8Lm` + `files.audio8CodecDecoder` (+ `files.audio8CodecEncoder` for voice cloning) (Audio8), or `files.mossBackbone` + `files.mossCodecDecoder` (+ `files.mossCodecEncoder` for voice cloning and dialogue) (MOSS). Use `npm run download-models:registry` to acquire registry-published Chatterbox, Supertonic, Parler, and Audio8 models. CosyVoice3 and MOSS must currently be staged from local converted artifacts. ## Requirement Bare $\geq$ v1.19 ## Installation ```bash npm i @qvac/tts-ggml ``` `@qvac/tts-ggml` is a meta package that ships the JavaScript wrapper only. The native prebuild for each desktop host installs through a version-locked, `os`/`cpu` filtered optional dependency: | Host | Package | | ------------------- | ----------------------------- | | linux-x64 (glibc) | `@qvac/tts-ggml-linux-x64` | | linux-arm64 (glibc) | `@qvac/tts-ggml-linux-arm64` | | darwin-arm64 | `@qvac/tts-ggml-darwin-arm64` | | darwin-x64 | `@qvac/tts-ggml-darwin-x64` | | win32-x64 | `@qvac/tts-ggml-win32-x64` | Do not depend on the desktop platform packages directly. Supported installers are npm 7+, pnpm, bun, and Yarn Berry; Yarn v1 and `--omit=optional` installs skip the platform package and fail at require time with an error naming it. A locally built `prebuilds/` directory in the package root always takes precedence, and `require('@qvac/tts-ggml').resolveBackendsDir()` locates the directory holding the host's binaries and dynamically loaded ggml backends. Unsupported targets require an explicit source build; installation does not automatically compile a local addon. Mobile targets are cross-built, so no install host ever matches their `os`, and optional-dependency filtering can never select them. Mobile applications must declare the target's platform package as a direct dependency, pinned to the exact `@qvac/tts-ggml` version: | Target | Package | | ------------------------- | ------------------------------ | | android-arm64 | `@qvac/tts-ggml-android-arm64` | | ios (device + simulators) | `@qvac/tts-ggml-ios` | ```json { "dependencies": { "@qvac/tts-ggml": "x.y.z", "@qvac/tts-ggml-android-arm64": "x.y.z" } } ``` The Linux x64 prebuild bundles the CUDA and Vulkan backends as runtime-loaded modules (CUDA is also available as an opt-in `ENABLE_CUDA=ON` source build on Linux arm64 and Windows x64). With `useGPU: true` the engine prefers CUDA when the NVIDIA driver and the CUDA 13 runtime libraries (`cudart`, `cuBLAS`) resolve at load time; otherwise the CUDA module is skipped and it falls back to Vulkan or CPU. Where more than one backend is usable, `TTS_CPP_GPU_BACKEND` (`cuda` | `vulkan` | `metal` | `opencl`) pins the choice — the SDK worker inherits the host process environment, so exporting it before starting the SDK is enough. ## Quickstart If you don't have Bare runtime, install it: ```bash npm i -g bare ``` Create a new project: ```bash mkdir qvac-tts-quickstart cd qvac-tts-quickstart npm init -y ``` Install dependencies: ```bash npm i @qvac/tts-ggml bare-fs bare-path ``` Place the Chatterbox GGUF files into `models/`: `chatterbox-t3-turbo.gguf` and `chatterbox-s3gen.gguf`. Optionally place a mono reference WAV (≥ 5 s of clean speech) at `./reference.wav` for voice cloning. Create `index.js` : ```js title="index.js" lineNumbers "use strict"; const fs = require("bare-fs"); const TTSGgml = require("@qvac/tts-ggml"); const SAMPLE_RATE = 24000; async function main() { const model = new TTSGgml({ files: { modelDir: "./models" }, // contains chatterbox-{t3-turbo,s3gen}.gguf referenceAudio: "./reference.wav", // optional voice cloning config: { language: "en" }, opts: { stats: true }, }); try { console.log("Loading Chatterbox TTS model..."); await model.load(); console.log("Model loaded."); const textToSynthesize = "Hello world! This is a test of the Chatterbox TTS system."; console.log(`Running TTS on: "${textToSynthesize}"`); const response = await model.run({ input: textToSynthesize, type: "text", }); let pcm = []; await response .onUpdate((data) => { if (data && data.outputArray) pcm = pcm.concat(Array.from(data.outputArray)); }) .await(); console.log("TTS finished!"); if (response.stats) { console.log(`Inference stats: ${JSON.stringify(response.stats)}`); } console.log(`Generated ${pcm.length} audio samples at ${SAMPLE_RATE}Hz`); } catch (err) { console.error("Error during TTS processing:", err); } finally { console.log("Unloading model..."); await model.unload(); console.log("Model unloaded."); } } main().catch(console.error); ``` Run `index.js`: ```bash bare index.js ``` ## Usage ### 1. Import the Model Class ```js const TTSGgml = require("@qvac/tts-ggml"); ``` ### 2. Create the Model Instance ```js const model = new TTSGgml({ files: { modelDir: "./models" }, referenceAudio: "./voices/me.wav", config: { language: "en", useGPU: false }, opts: { stats: true }, }); ``` The most common constructor options: | Option | Type | Default | Description | | ----------------------------------------------- | ---------------- | --------------- | ------------------------------------------------------------------------------------------------------------- | | `files.modelDir` | string | — | Directory containing the two GGUFs (engine auto-detected) | | `files.t3Model` / `files.s3genModel` | string | — | Override `modelDir` for the Chatterbox T3 / S3Gen GGUF | | `files.supertonicModel` | string | — | Supertonic GGUF path | | `files.parlerModel` | string | — | Parler GGUF path (mini / large / indic variant) | | `files.cosyvoiceModelDir` | string | — | CosyVoice3 model directory (`cosyvoice3-{llm,flow,hift}-*.gguf` + `voice.gguf` + tokenizer files) | | `files.audio8Lm` / `files.audio8CodecDecoder` | string | — | Audio8 LM / codec decoder GGUFs (override `modelDir`) | | `files.audio8CodecEncoder` | string | — | Audio8 codec encoder — only needed to clone a voice | | `files.mossBackbone` / `files.mossCodecDecoder` | string | — | MOSS backbone (`moss-tts-delay-*` or `moss-ttsd-*`) / codec decoder GGUFs (override `modelDir`) | | `files.mossCodecEncoder` | string | — | MOSS codec encoder — only needed to clone a voice or for `dialogueReferences` | | `referenceAudio` | string | — | Mono WAV for voice cloning (Chatterbox: ≥ 5 s; Audio8: also needs `referenceText`; MOSS: 24 kHz) | | `dialogueReferences` | string\[] | — | MOSS-TTSD: one 24 kHz WAV per speaker (`[S1]`, `[S2]`, …); the text opens with their transcripts | | `durationTokens` | number | 0 | MOSS: target length in codec frames (12.5 per second); `0` keeps it free | | `referenceText` | string | — | Audio8-only: what `referenceAudio` says, verbatim; required with a reference | | `voiceDir` | string | — | Pre-baked voice profile directory | | `emotion` / `pace` | string | — | Cross-engine conditioning — Parler + CosyVoice3 emotions; Parler / CosyVoice3 / Supertonic pace | | `instruct` | object \| string | — | CosyVoice3-only: `dialect` / `volume` / `style` object (precedence in that order) or a raw instruction string | | `temperature` / `topK` / `topP` | number | engine defaults | Parler + Audio8 sampling knobs (Audio8: temp 0.7 / top-k 50 / top-p 0.9) | | `greedy` | boolean | `false` | Audio8-only: take the argmax; ignores `temperature` / `topK` / `topP` | | `streamChunkTokens` | number | 0 | `> 0` enables native chunk streaming (25 tokens ≈ 1 s of audio) | | `cfmSteps` | number | 2 | `1` halves CFM cost for faster synthesis | | `config.language` | string | `'en'` | Language code; multilingual models accept `es/fr/de/pt/it/zh/ja/ko/...` | | `config.useGPU` | boolean | `false` | Route through Metal / CUDA / Vulkan / OpenCL if available | | `config.outputSampleRate` | number | 24000 | Resample the native 24 kHz output | | `opts.stats` | boolean | `false` | Populate `response.stats` with RTF, backend info, etc. | See the [package README](https://github.com/tetherto/qvac/tree/main/packages/tts-ggml) for the full option set (GPU/backend, KV-cache, and Android-specific options). ### 3. Load the Model ```js await model.load(); ``` `load()` constructs the native engine — it loads T3, preloads S3Gen, and bakes voice conditioning. Subsequent `run()` calls reuse all of it. ### 4. Run TTS Synthesis Pass the text to synthesize to the `run` method and process the generated audio output asynchronously: ```javascript try { const textToSynthesize = "Hello world! This is a test of the TTS system."; let audioSamples = []; const response = await model.run({ input: textToSynthesize, type: "text", }); await response .onUpdate((data) => { if (data && data.outputArray) { audioSamples = audioSamples.concat(Array.from(data.outputArray)); } }) .await(); console.log(`Total audio samples generated: ${audioSamples.length}`); // audioSamples now contains the complete audio as PCM data (16-bit, 24 kHz, mono) if (response.stats) { console.log(`Inference stats: ${JSON.stringify(response.stats)}`); } } catch (error) { console.error("TTS synthesis failed:", error); } ``` ### 5. Release Resources Unload the model when finished: ```javascript try { await model.unload(); } catch (error) { console.error("Failed to unload model:", error); } ``` ## Streaming ### Sentence streaming — `runStreaming(asyncIterable)` Use when your text arrives as discrete sentences (e.g. buffered LLM output) and you want the audio to flow sentence-by-sentence. One `onUpdate` event per input yield: ```js async function* sentencesOverTime() { yield "First sentence."; await new Promise((r) => setTimeout(r, 200)); yield "The second arrives shortly after."; } const response = await model.runStreaming(sentencesOverTime()); await response .onUpdate((data) => { // data.outputArray — Int16 PCM for this sentence's audio // data.chunkIndex — 0-based index of the yielded sentence // data.sentenceChunk — the sentence text that produced this audio }) .await(); ``` `runStreaming(textStream, options)` accepts a string, string array, iterable, or async iterable. Async iterables default to `accumulateSentences: true`; strings, arrays, and synchronous iterables default to one synthesis job per item. Set `accumulateSentences` explicitly to override that behavior. Select `sentenceDelimiterPreset: "latin"`, `"multilingual"`, or `"cjk"`, or provide a `sentenceDelimiter` regular expression. `maxBufferScalars` forces a flush at the buffer limit and `flushAfterMs` flushes incomplete text after the configured delay. Parler description fields and Audio8 `referenceAudio` / `referenceText` can also be fixed for the full streaming response. ### Chunk streaming — `streamChunkTokens` Use when you want the fastest possible first-audio-out **within a single utterance**. The C++ engine splits each synthesis into chunks of `streamChunkTokens` speech tokens and emits audio per chunk: ```js const model = new TTSGgml({ files: { modelDir: "./models" }, streamChunkTokens: 25, // ~1 s of audio per chunk streamFirstChunkTokens: 10, // smaller first chunk = faster first-audio-out cfmSteps: 1, config: { language: "en" }, }); await model.load(); const response = await model.run({ input: "A long sentence produces many chunks...", }); await response .onUpdate((data) => { if (data && data.outputArray) playPcmChunk(data.outputArray); }) .await(); ``` ## Voice cloning Pass a mono WAV with ≥ 5 s of clean speech. The engine does the loudness normalisation, resampling, and all conditioning natively at `load()` time: ```js const model = new TTSGgml({ files: { modelDir: "./models" }, referenceAudio: "./voices/me.wav", config: { language: "en" }, }); ``` Alternatively point at a pre-baked profile directory via `voiceDir`. When both are supplied, missing tensors in `voiceDir` are backfilled from `referenceAudio`. **Audio8** clones from the recording plus **what is said in it**: supply `referenceAudio` and `referenceText` (the verbatim transcript) together with `files.audio8CodecEncoder`, which encodes the reference to codes in-process — no enrolment step, no voice profile. The reference can also be switched per call. **CosyVoice3** does not take a reference — it always speaks with its baked default voice. ## Speech enhancement (LavaSR) Opt-in neural post-processing that bandwidth-extends the synthesized audio to **48 kHz** with a synthesised high band, using the LavaSR Vocos enhancer run on the CPU/GGML path. It is fully backward compatible — provide no enhancer GGUF and nothing changes. Enhancement is enabled simply by supplying the enhancer GGUF; there is no separate on/off flag. ```js const model = new TTSGgml({ engine: TTSGgml.ENGINE_SUPERTONIC, // Providing the enhancer GGUF is what turns enhancement on: files: { supertonicModel, lavasrEnhancer: "models/lavasr/lavasr-enhancer.gguf", }, config: { language: "en" }, }); // The output callback now reports 48000: // response.onUpdate(d => ) ``` The GGUF path may instead be given as an `enhancer: { type: 'lavasr', enhancerPath }` block. * Works for Chatterbox, Supertonic, Parler, and CosyVoice3 (Audio8 is not supported) — on the batch path, sentence-level streaming, and native chunk streaming where the selected engine supports it. * For native chunk streaming the enhancer runs over a sliding window with look-ahead + crossfade, so each emitted chunk is bandwidth-extended seam-free; this adds \~0.34 s of look-ahead latency. * The enhancer always runs at 48 kHz internally. By default the emitted audio is 48 kHz; set `config.outputSampleRate` to resample the enhanced output (`sampleRate` reports the actual rate). ### Denoiser LavaSR's first stage — the UL-UNAS **denoiser** that cleans the signal before the enhancer bandwidth-extends it — is enabled the same way, via `files.lavasrDenoiser` (or a `denoiser: { type: 'lavasr', denoiserPath }` block), and runs before the enhancer on the batch path: ```js const model = new TTSGgml({ engine: TTSGgml.ENGINE_SUPERTONIC, files: { supertonicModel, lavasrDenoiser: "models/lavasr/lavasr-denoiser.gguf", // cleaned first… lavasrEnhancer: "models/lavasr/lavasr-enhancer.gguf", // …then upsampled }, config: { language: "en" }, }); ``` * The denoiser forward runs at 16 kHz internally (resampled in/out), so it is **rate-preserving** — the emitted audio keeps the engine's sample rate. With no denoiser path the output is unchanged. * Denoiser + Chatterbox native chunk streaming (`streamChunkTokens > 0`) is rejected up front; use batch synthesis, or drop the denoiser for streaming. ## Output Format Audio is received via the `onUpdate` callback of the response object as raw PCM samples. ```javascript response.onUpdate((data) => { data.outputArray; // Int16Array — mono PCM data.sampleRate; // actual output sample rate data.chunkIndex; // present on sentence-streaming events only data.sentenceChunk; // present on sentence-streaming events only }); ``` When synthesis completes and `opts: { stats: true }` was set, `response.stats` reports performance: ```javascript response.stats.totalTime; // seconds response.stats.realTimeFactor; // synthesis time / audio duration; < 1 means streaming is possible response.stats.audioDurationMs; response.stats.totalSamples; response.stats.tokensPerSecond; response.stats.backendDevice; // 0 CPU, 1 GPU response.stats.backendId; // 0 CPU, 1 Metal, 2 CUDA, 3 Vulkan, 4 OpenCL, 99 other response.stats.enhancerBackendDevice; // -1 absent, 0 CPU, 1 GPU response.stats.enhancerBackendId; ``` **Audio Format Specifications:** * **Sample Rate:** engine-native by default (24 kHz for Chatterbox/CosyVoice3/MOSS, 44.1 kHz for Supertonic/Parler/Audio8, 48 kHz with the enhancer), configurable via `config.outputSampleRate` (MOSS emits 24 kHz only) * **Format:** 16-bit signed PCM, mono channel * **Data Type:** Int16Array containing raw audio samples ## Public helpers and errors Import `splitTtsText` from `@qvac/tts-ggml/text-chunker` and `accumulateTextStream` from `@qvac/tts-ggml/text-stream-accumulator`. The legacy `@qvac/tts-ggml/lib/textStreamAccumulator.js` path remains supported. `QvacErrorAddonTTSGgml` and `ERR_CODES` are exported from `@qvac/tts-ggml`. Error codes are: | Code | Name | | ----: | ---------------------- | | 13001 | `FAILED_TO_ACTIVATE` | | 13002 | `FAILED_TO_APPEND` | | 13003 | `FAILED_TO_GET_STATUS` | | 13004 | `FAILED_TO_PAUSE` | | 13005 | `FAILED_TO_CANCEL` | | 13006 | `FAILED_TO_DESTROY` | | 13007 | `FAILED_TO_UNLOAD` | | 13008 | `FAILED_TO_LOAD` | | 13009 | `FAILED_TO_RELOAD` | | 13010 | `FAILED_TO_STOP` | | 13011 | `JOB_ALREADY_RUNNING` | ## More resources [Package at npm](https://www.npmjs.com/package/@qvac/tts-ggml) --- title: "Resources" canonical: https://docs.qvac.tether.io/resources/ collection: "Resources" --- # Resources (/resources) ## Overview This collection holds the material that helps you build with QVAC without documenting a release of it. Nothing here is versioned: a tutorial on packaging an Electron app and a fix for a missing system library do not expire with a patch, so they live here in one copy rather than in each documentation line. It holds the tutorials for building on a specific platform, the troubleshooting guide for when something does not start, and the procedure a coding agent follows to read the documentation matching the release a project has installed. It also indexes what lives outside this site: recipes published on the main website, and sample projects published as source code. Example apps and code samples for the QVAC SDK. Step-by-step recipes for building with local AI. --- title: "Docs for AI agents" canonical: https://docs.qvac.tether.io/resources/docs-for-ai-agents/ collection: "Resources" --- # Docs for AI agents (/resources/docs-for-ai-agents) ## Overview QVAC docs offers AI agents the following: * `llms.txt` — an index of what is published and where to fetch it * `llms-full.txt` — the full text of the documentation in one fetch * Ask AI chat — an assistant that answers from these pages * Markdown negotiation — any page as plain Markdown * MCP server — these docs as a tool your editor can call The SDK and the CLI publish one copy of their documentation per release, which changes how you use most of the above. Read [Versioning](#versioning) before you rely on an answer about either. ## `llms.txt` An index. It tells you what is published and where to fetch it, so you can read one thing instead of everything. There is one at each of three levels: | Fetch | To get | | ----------------------------------------- | ------------------------------------------------------------------------- | | `/llms.txt` | Every collection on the site, and where to resolve the versioned ones | | `/{collection}/llms.txt` | Every published line of one collection, with each line's index and corpus | | `/{collection}/v{major}.{minor}/llms.txt` | Every page of one line | Start at `/llms.txt`. It is small, and it names where to go next. For a versioned collection, `/{collection}/versions.json` carries what that collection's `llms.txt` carries, as JSON. Fetch that one if you are parsing rather than reading. ## `llms-full.txt` The full text of many pages in one fetch. Use it when you need broad context and do not yet know which page holds the answer. Use a single page when you do. The same three levels: | Fetch | To get | | ---------------------------------------------- | ------------------------------------------------------------------------- | | `/llms-full.txt` | The unversioned collections, plus the current line of every versioned one | | `/{collection}/llms-full.txt` | The current line of that collection | | `/{collection}/v{major}.{minor}/llms-full.txt` | That line | Every corpus opens with a header naming what is in it, how many pages it holds, and where to go for what it leaves out. Read the header — it is how you find out you fetched the wrong scope. Release notes are withheld from every corpus. Fetch one as its own page, from that line's index. If you are answering about a specific release, fetch that line's corpus. The site corpus carries only current lines. ## Ask AI chat An assistant that answers from this documentation and cites the pages it used. Use it when you do not know the term to search for. Press `Cmd + I` (`Ctrl + I` on Windows and Linux), or click **Ask AI** in the header. It searches the line you are reading, the unversioned collections, and the current line of every other versioned collection. It will not answer from a line you are not in, so open a page of the line you care about first. ## Markdown negotiation Every page is also published as Markdown, without the site around it. Use it to hand a page to a model. There are two ways to get it: * **Append `.md` to the path** — `/sdk/js-ts-sdk` becomes `/sdk/js-ts-sdk.md`. * **Send `Accept: text/markdown`** — any page URL redirects to its Markdown. Use this when you cannot rewrite the URL. Each Markdown page opens with front matter naming its collection, the package that collection tracks, its line, and whether that line is the current one. Check `line` and `current_line` to confirm you have the release you meant. Cite the URL without `.md`. That is the one a person can open. On a page you are reading, **Copy page** copies its Markdown, and the button beside it opens the page in ChatGPT or Claude. ## MCP server A Model Context Protocol server, so an MCP-capable editor or agent can search these docs as a tool instead of fetching URLs. Add this endpoint to your client: ``` https://mcp.inkeep.com/tetherio/mcp ``` The **For AI** menu in the header copies that address for you. ## Versioning The SDK and the CLI each publish one complete copy of their documentation per release — a **documentation line**. A line is self-contained, so reading the wrong one gives you answers that are internally consistent and wrong for the project in front of you. ### Resolve the line before you read 1. **Read the installed version from the project**, not from this site: `package.json` and the lockfile for `@qvac/sdk` or `@qvac/cli`. Keep the major and minor, drop the patch — `0.16.3` is `v0.16`. 2. **Fetch `/{collection}/versions.json`.** It names every published line and marks which one is current. 3. **Take the line whose `version` matches.** Read its `index` for a page list, or its `corpus` for the whole line in one fetch. If you are a person rather than an agent, skip all three: use the version switcher at the top of the sidebar. ### The current line carries no version in its URL The current line is served at `/sdk/…`; every older line carries its own segment, `/sdk/v0.19/…`. Cite the URL of the line you actually read. ### When no line matches Only the most recent releases stay published, so a project pinned further back will find no line of its own. * Read the **nearest older line** — the highest published version below the one installed. * **Say which line you read and which release the project is on**, so the reader can judge the gap. * Never fall forward to a newer line. It describes behaviour the installed release may not have. ### Never mix lines Do not combine text from two lines of the same collection in one answer. Their APIs differ; that is why they are separate. Combining collections is fine. The SDK and the CLI are versioned independently — resolve each at its own line. --- title: "Troubleshooting" canonical: https://docs.qvac.tether.io/resources/troubleshooting/ collection: "Resources" --- # Troubleshooting (/resources/troubleshooting) ## CLI error: command not found ### Situation You ran a `qvac` command (e.g., `qvac doctor`) and the shell responded with: ``` zsh: command not found: qvac ``` (On bash: `bash: qvac: command not found`.) ### Cause The `qvac` binary is shipped by the `@qvac/cli` npm package. It is only added to your shell `PATH` when `@qvac/cli` is installed globally. Without a global install, the binary is not discoverable by name from your shell. ### Solution Install `@qvac/cli` globally: ```bash npm install -g @qvac/cli ``` Then re-run your command. For example: ```bash qvac doctor ``` **Tip:** If you cannot install the CLI globally, you can run it with `npx` instead: ```bash npx --package "@qvac/cli" qvac doctor ``` See [CLI → Usage](/cli#usage) for the full setup, including installing `@qvac/sdk` in your project. ## Startup failure: missing libatomic.so.1 ### Situation On a minimal Linux image, worker startup fails and the error cause includes: ```text libatomic.so.1: cannot open shared object file: No such file or directory ``` The top-level SDK error may report an RPC initialization timeout. Check `error.cause` for the underlying loader error. ### Cause The Linux `rocksdb-native` prebuilds used by the SDK's storage dependencies require `libatomic.so.1`. Minimal images such as `node:22-slim` may omit this library. The dependency is needed even for CPU-only inference, before a model is loaded. ### Solution Install `libatomic1` in the Debian or Ubuntu environment that runs the worker: ```bash sudo apt update sudo apt install -y libatomic1 ``` Omit `sudo` when running as root inside a container. Restart the application after installing the library. See [system requirements](/sdk/system-requirements#linux) and [issue #4290](https://github.com/tetherto/qvac/issues/4290). ## Startup crash: requested module does not provide a default export ### Situation You ran `qvac bundle sdk`, started your app, and then the worklet crashed with: ``` SyntaxError: The requested module '@qvac/sdk/tts-ggml/plugin' does not provide an export named 'default' ``` ### Cause This usually means there is version skew between `@qvac/cli` and `@qvac/sdk`. An older CLI version may generate a bundle using outdated built-in plugin metadata, while a newer SDK version exports the renamed plugin only through its current named export shape. Starting with `@qvac/cli` 0.6.0, `qvac bundle sdk` delegates to `@qvac/sdk/commands`, which keeps the bundling logic aligned with the SDK. ### Solution Upgrade `@qvac/cli`, then rebuild the SDK bundle: ```bash npm install -g @qvac/cli@latest qvac bundle sdk ``` If you do not install the CLI globally, run the latest version with `npx` instead. ## SDK fails to load on GCC-10-era embedded Linux ### Situation You are running the QVAC SDK on a Yocto-based or other embedded Linux image with GCC 10 era `libstdc++` (GLIBCXX version 3.4.28 or lower). The SDK fails at module load time with an error such as: ``` Error: /usr/lib/aarch64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found ``` ### Cause `rocksdb-native`, a transitive dependency reached through `corestore` and `hypercore-storage`, requires `GLIBCXX_3.4.30`. Its Linux ARM64 prebuilds are compiled against Ubuntu 22.04, and every version within the SDK's current semver range carries this floor. Pinning an older version does not help. ### Solution Switch the embedded image toolchain from GCC to clang with static `libc++` linkage. This removes the versioned `libstdc++` symbol dependency entirely. The QVAC-owned inference addons (such as `@qvac/translation-nmtcpp`) already use this approach and carry no `libstdc++` requirement. GCC-10-era embedded Linux images are not a currently supported target. See [GitHub issue #3365](https://github.com/tetherto/qvac/issues/3365) for the full diagnosis and audit. --- title: "Build an Electron app" canonical: https://docs.qvac.tether.io/resources/tutorials/electron/ collection: "Resources" --- # Build an Electron app (/resources/tutorials/electron) ## What we'll build We'll build an LLM chat desktop application using the following stack: * **Electron** as the desktop runtime; * **React** for the user interface; * **Tailwind CSS** for styling; and * **QVAC** to run LLM inference locally. By the end of this tutorial, we'll have built the following:
## Prerequisites * Node.js `>= v22.17` * npm `>= v10.9` * Linux/macOS (Windows with small adjustments) Some commands are Bash‑specific. On Windows, use PowerShell/WSL or adapt them. ## Step 1: set up an Electron project We'll use [electron-vite](https://electron-vite.org) to scaffold a full Electron + React + TypeScript project in a single command. Create a new project: ```bash npm create @quick-start/electron@latest llm-desktop-app -- --template react-ts ``` When prompted, answer **No** to both "Add Electron updater plugin?" and "Enable Electron download mirror proxy?". Install dependencies and configure the dev script: ```bash cd llm-desktop-app npm install npm pkg set scripts.dev="electron-vite dev -- --no-sandbox" ``` The `--no-sandbox` flag disables the Chromium sandbox, which is required on Linux when the SUID helper is not configured. Start the app: ```bash npm run dev ``` Confirm that an Electron window opens and renders the default electron-vite page:
Electron window
The scaffold gives us a working Electron app with React, TypeScript, Vite HMR, and a preload script — all pre-configured. The key files we'll modify are: | File | Purpose | | -------------------------- | ------------------------------------ | | `src/main/index.ts` | Electron main process | | `src/preload/index.ts` | IPC bridge between main and renderer | | `src/preload/index.d.ts` | Type declarations for the bridge | | `src/renderer/src/App.tsx` | React root component | | `electron.vite.config.ts` | Vite config for all three processes | ## Step 2: create the chat UI In this step, we'll add Tailwind CSS and replace the scaffold's default UI with a chat interface. The chat will use stub responses for now — we'll wire it to QVAC in the next step. Install Tailwind CSS: ```bash npm i tailwindcss @tailwindcss/vite ``` Add the Tailwind plugin to the renderer in `electron.vite.config.ts`: ```ts title="electron.vite.config.ts" lineNumbers import { resolve } from 'path' import { defineConfig } from 'electron-vite' import react from '@vitejs/plugin-react' import tailwindcss from '@tailwindcss/vite' // [!code ++] export default defineConfig({ main: {}, preload: {}, renderer: { resolve: { alias: { '@renderer': resolve('src/renderer/src') } }, plugins: [react()] // [!code --] plugins: [react(), tailwindcss()] // [!code ++] } }) ``` This tells Vite to process Tailwind utility classes in the renderer. Replace the entire contents of `src/renderer/src/assets/main.css` with the following code: ```css title="src/renderer/src/assets/main.css" @import 'tailwindcss'; ``` This replaces the scaffold's default styles with Tailwind's base layer. Replace the entire contents of `src/renderer/src/main.tsx` with the following code: ```ts title="src/renderer/src/main.tsx" lineNumbers import './assets/main.css' import { createRoot } from 'react-dom/client' import App from './App' createRoot(document.getElementById('root')!).render() ``` This removes React Strict Mode, which avoids double-invoked effects in development. That matters when we load a model in `useEffect`. Replace the entire contents of `src/renderer/src/App.tsx` with the following code: ```ts title="src/renderer/src/App.tsx" lineNumbers import { useState } from 'react' type Message = { role: 'user' | 'assistant'; content: string } function App(): React.JSX.Element { const [loading, setLoading] = useState(false) const [messages, setMessages] = useState([]) const [input, setInput] = useState('') const handleSend = (): void => { if (!input.trim()) return setMessages(prev => [ ...prev, { role: 'user', content: input }, { role: 'assistant', content: 'Stub response from the assistant.' } ]) setInput('') } return (

LLM Desktop App

{loading ? 'Loading model…' : 'Ready'}
{loading ? (
) : ( messages.map((msg, i) => (
{msg.content}
)) )}