Introduction

Overview

Install the JS/TS or Python clients in your project. Then, load models and use them to perform AI inference locally, or delegate inference to peers using the built-in P2P capability.

The SDK has two layers:

• Worker — the runtime that actually runs the models. Today the worker ships as the @qvac/sdk npm package.
• Clients — per-language APIs that drive the worker. Two clients ship today: JS/TS (@qvac/sdk on npm) and Python (tetherto-qvac-sdk on PyPI). Same worker, same generated contract, same capabilities.

Pick the client that matches your stack:

Description

The SDK is cross-platform, type-safe, and pluggable, exposing all QVAC capabilities through a unified interface.

Key features

• Cross-platform: portable code across Linux, macOS, and Windows (JS/TS on Node.js / Bare runtime, Python on CPython); Android and iOS via Expo (TS only).
• Pluggable: build lean apps by including only what you need, and extend the SDK with custom plugins.
• Type-safe: typed APIs in both JavaScript and Python (Pydantic models generated from the same contract).
• Unified interface: multiple AI tasks, one client package per language.

Functionalities

AI tasks

• Text generation: LLM inference for text generation and chat via qvac-fabric-llm.cpp.
• Text embeddings: vector embedding generation for semantic search, clustering, and retrieval, via qvac-fabric-llm.cpp.
• RAG: out-of-the-box retrieval-augmented generation workflow.
• Fine-tuning: adapting LLMs to domain-specific tasks via LoRA.
• Multimodal: LLM inference over text, images, and other media within a single conversation context.
• Batch processing: run multiple LLM prompts concurrently through a single loaded model in one call.
• Image generation: text-to-image and image-to-image generation via a customized Diffusion engine.
• Video generation: text-to-video and image-to-video generation via a customized Diffusion engine.
• Music generation: generate music from text, lyrics, and musical controls via ACE-Step.
• Transcription: automatic speech recognition (ASR) for speech-to-text via a customized Whisper engine or NVIDIA Parakeet.
• Text-to-Speech: speech synthesis for text-to-speech (TTS) via a customized GGML backend.
• Voice assistant: real-time voice conversation pipeline chaining transcription, text generation, and text-to-speech.
• Translation: text-to-text neural machine translation (NMT), via qvac-fabric-llm.cpp and Bergamot.
• BCI: brain–computer interface (BCI) transcription that decodes multi-channel neural signals into text, via a customized Whisper backend.
• VLA: vision-language-action that turns camera frames, robot state, and natural-language instruction into action chunks for robot control, via a customized GGML backend.
• OCR: optical character recognition (OCR) for extracting text from images via ONNX runtime.
• Image classification: assigning class labels with confidence scores to images, via a customized GGML backend.

P2P capabilities

• Delegated inference: delegate inference to peers via the Holepunch stack, enabling resource sharing.
• Fetch models: download AI models from peers via the distributed model registry.
• Blind relays: connect peers across NATs/firewalls by routing traffic through relay nodes.

Utilities

• Logging: visibility into what's happening during loading, inference, and other operations.
• Profiler: measure and export timing metrics across model loading, inference, and P2P delegation.
• Download Lifecycle: pause and resume model downloads.
• Runtime lifecycle: suspend and resume the SDK runtime (e.g., on app background/foreground) and query lifecycle state.
• Cancellation: cancel any in-flight inference, model load, or download by requestId, or broad-cancel by modelId for unload/shutdown.
• Sharded models: download a model that is sharded into multiple parts.

Flow

Before you can use a model, you need to load it from some location into memory. The flow for performing AI inference is:

1. Load one model into the SDK. You can load multiple models simultaneously by repeating the call.
2. Perform AI tasks by calling the appropriate SDK functions — e.g., a completion.
3. When you are done with a model, unload it to release computer resources.
4. Finally, close the SDK instance.

The concrete API names differ by client — in JS/TS it is loadModel() / completion() / unloadModel() / close() (see API reference); in Python it is load_model / completion / unload_model and the Client() context manager (see Python SDK).

Models

Each AI task works with different model families, and among the supported ones, you can choose which to use and how to obtain them. Model loading manages the download and caching of models (one or multiple files), and their loading from disk into memory, preparing them for use.

Models can be loaded from three different locations:

• Local filesystem, by providing a path
• HTTP server, by providing an HTTP URL
• Our distributed model registry

SDK packages do not ship with built-in models, but their APIs expose constants representing preconfigured models (e.g., LLAMA_3_2_1B_INST_Q4_0). Each constant maps a model already published to our model registry. When loading a model, you can provide one of these constants instead of a location, making model retrieval transparent.

Model registry index

See the index of models available in our distributed model registry.

For more on querying the model registry, see modelRegistryList(), modelRegistrySearch(), and modelRegistryGetModel() (JS/TS), or model_registry_list / _search / _get_model (Python — see Python SDK — Running examples for a runnable script).

For more on loading models in JS/TS, see loadModel() at @qvac/sdk API reference.

Configuration

Configuration

Shared options and configuration schema.

Plugin system

API reference

How it works

How it works

Understand what happens under the hood when you use QVAC SDK in your application

Other resources

• SDK landing page
• @qvac/sdk on npm
• tetherto-qvac-sdk on PyPI