New: QVAC SDK now supports Kotlin. Build local AI into your JVM and Android apps.
QVAC Logo
SDKConfigurationPlugin system
v0.24, the current release

Plugin system

Enable and disable built-in AI capabilities, and add new ones via custom plugins.

Overview

Each QVAC AI capability maps to a built-in plugin in the SDK. This lets you enable only what you need for your project and reduce your application's final bundle size.

In addition, you can add custom plugins — both your own and community ones — that extend QVAC's capabilities. In both cases, you'll use the QVAC configuration file and then bundle either with QVAC CLI or programmatically via @qvac/sdk/commands.

Built-in plugins

Catalog

All the built-in plugins you can select, along with the AI tasks that depend on each one:

PluginUse in qvac.config.*AI tasks that require it
LLM@qvac/sdk/llamacpp-completion/pluginText generation; multimodal; RAG ; Fine-tuning
Embeddings@qvac/sdk/llamacpp-embedding/pluginText embeddings; RAG
ASR with customized Whisper engine@qvac/sdk/whispercpp-transcription/pluginTranscription
ASR with Parakeet@qvac/sdk/parakeet-transcription/pluginTranscription
BCI transcription@qvac/sdk/bci-whispercpp-transcription/pluginTranscription
BCI transcription@qvac/sdk/bci-whispercpp-transcription/pluginTranscription
NMT@qvac/sdk/nmtcpp-translation/pluginTranslation
TTS@qvac/sdk/onnx-tts/pluginText-to-Speech
OCR@qvac/sdk/ggml-ocr/pluginOCR
Diffusion@qvac/sdk/sdcpp-generation/pluginImage generation
AudioGen@qvac/sdk/audiogen-ggml/pluginMusic generation

Enabling

In your qvac.config.*, add the built-in plugins you’ll need in your project. For example:

qvac.config.json
{
  "plugins": [
    "@qvac/sdk/llamacpp-completion/plugin",
    "@qvac/sdk/ggml-ocr/plugin"
  ]
}

When developing for desktop environment, use the QVAC CLI to (re)bundle the SDK only with the selected plugins:

qvac bundle sdk

Only the selected plugins are included in your bundle, significantly reducing your application size. If plugins is omitted or an empty array, it bundles all built-in plugins by default.

Tips:

Custom plugins

Custom plugins are consumed as npm packages. Install the package, enable it in your qvac.config.* file like any built-in plugin, then import and use its API alongside the SDK's regular API.

Enabling

Install the plugin package in your project. For example:

npm i qvac-echo-plugin

In your qvac.config.*, add its <package_name>/plugin specifier alongside any built-in plugins. For example:

qvac.config.json
{
  "plugins": [
    "@qvac/sdk/llamacpp-completion/plugin",
    "qvac-echo-plugin/plugin"
  ]
}

When developing for desktop environment, use the SDK CLI to (re)bundle the SDK with all listed plugins:

qvac bundle sdk

Tip: when adding one or more custom plugins, you must also add all the built-in plugins you will need to use.

Usage

Custom plugins are consumed as npm packages. Install the package, enable it in your qvac.config.* file like any built-in plugin, then import and use its API alongside the SDK’s regular API.

import { loadModel, unloadModel } from "@qvac/sdk";

// 1. Import the API you need from the custom plugin package:
import { echo, echoStream } from "qvac-echo-plugin";

// 2. Load the model(s) required by the custom plugin:
const modelId = await loadModel({
  modelSrc: "/path/to/echo-model.bin",
  modelType: "echo",
});

// 3. Call functions exposed by the custom plugin API:
const result = await echo({ modelId, message: "Hello, plugin system!" });
console.log(result);

for await (const char of echoStream({ modelId, message: "Streaming test!" })) {
  process.stdout.write(char);
}

await unloadModel({ modelId });

The Python and Kotlin SDKs expose plugin invocation through the same worker contract:

plugins.py
"""Python port of packages/sdk/examples/plugins.ts (plugin invocation).

The low-level plugin API. A model's `modelType` selects a plugin; each plugin
exposes named `handler`s with their own request/response schemas.
`invoke_plugin(t, model_id, handler, params)` runs one handler and returns its
result; `invoke_plugin_stream(...)` yields streamed chunks.

This example uses the `custom-echo-plugin` fixture (modelType `echo`), whose
handlers are `echo` ({message} -> {message}) and `echoStream` ({message} ->
{chunk} chunks). To run it you need a worker bundled with that plugin; point
the loader at it via argv, or adapt the handler/params to your own plugin.
(A production model that exposes handlers is VLA — see vla.py, which drives the
`vlaRun` / `vlaHparams` handlers through typed wrappers.)

RUN: python examples/plugins.py [model-src]
"""

from __future__ import annotations

import asyncio
import sys

from tetherto.qvac_sdk import (
    Client,
    invoke_plugin,
    invoke_plugin_stream,
    load_model,
    unload_model,
)


def print_progress(p) -> None:
    """Print model download progress; pass as `on_progress=` to `load_model`."""
    line = (
        f"▸ Downloading {p.percentage:.0f}% "
        f"({p.downloaded / 1e6:.1f}/{p.total / 1e6:.1f} MB)"
    )
    print(line, end="\r" if sys.stderr.isatty() else "\n", file=sys.stderr)
    if p.percentage >= 100:
        print(file=sys.stderr)


async def main() -> int:
    if len(sys.argv) < 2:
        print(
            "Usage: python examples/plugins.py <model-src>\n"
            "  <model-src> must resolve to a model whose plugin exposes the "
            "'echo'/'echoStream' handlers (the custom-echo-plugin fixture).",
            file=sys.stderr,
        )
        return 1
    model_src = sys.argv[1]

    async with Client() as client:
        t = client.transport
        try:
            print("▸ Loading plugin-backed model...")
            model_id = await load_model(
                t,
                model_src=model_src,
                model_type="echo",
                on_progress=print_progress,
            )
            print(f"▸ Model loaded: {model_id}")

            print("\n▸ 1. invoke_plugin (request/reply handler 'echo')")
            result = await invoke_plugin(
                t, model_id, "echo", params={"message": "hello from the python sdk"}
            )
            print(f"▸ echo -> {result}")

            print("\n▸ 2. invoke_plugin_stream (streaming handler 'echoStream')")
            print("▸ chunks: ", end="")
            async for chunk in invoke_plugin_stream(
                t, model_id, "echoStream", params={"message": "streamed message"}
            ):
                sys.stdout.write(str(getattr(chunk, "chunk", chunk)))
                sys.stdout.flush()
            print()

            await unload_model(t, model_id)
            print("▸ Model unloaded")
        except Exception as error:
            print(f"✖ {error}", file=sys.stderr)
            return 1
    return 0


if __name__ == "__main__":
    sys.exit(asyncio.run(main()))

Runtime registration on Bare

The plugins array in qvac.config.* is bundle-time configuration — it controls which addons get packed into your worker. That is separate from runtime registration, which determines the plugins live in the worker process when an SDK call runs.

On Node.js and Expo the SDK starts a worker that registers the plugins in its worker entry: the ones plugins lists once a bundle step has run, or every built-in plugin when none has. You never register manually. In-process Bare uses @qvac/inference with no spawned worker — register the plugins you use before the first call:

import { plugins } from "@qvac/inference";
import { llmPlugin } from "@qvac/inference/llamacpp-completion/plugin";

const sdk = plugins([llmPlugin]); // or registerPlugin(llmPlugin) from "@qvac/inference/plugins"

Calls made before any plugin is registered raise PluginsNotRegisteredError.

In-process Bare uses @qvac/inference. @qvac/bare-sdk is deprecated; last release is 0.18.2.

Notes

  • In qvac.config.*, if plugins is omitted or set to an empty array, the SDK bundles all built-in plugins. If plugins is set, it bundles only the listed plugins.
  • Each plugin maps to one QVAC addon.
  • Write a custom plugin.
  • Programmatic bundling: you can also prepare and validate the SDK bundle programmatically via API from a Node script. For example:
import { bundleSdk, formatVerifyBundleResult } from "@qvac/sdk/commands";

const { verification } = await bundleSdk({
  projectRoot: process.cwd(),
  configPath: "./qvac.config.json",
  target: "bare-sidecar",
  quiet: true,
});

console.log(formatVerifyBundleResult(verification));
  • Bundle output: bundleSdk writes the worker entry to qvac/worker.entry.mjs and replaces qvac/worker/ with bare-stow's harness package: the bundle, the small module that starts it (index.js) and its package.json. target picks where the worker runs: bare-sidecar (the default, for Node.js and Electron), react-native (Expo), or pear-runtime (Pear). The react-native target also writes Bare Kit and the native addons the worker loads into android/qvac-bare-kit and ios/QvacBareKit (iOS on macOS only); pass link: false to skip that, or a list of hosts to write only theirs. bundleSdk runs verifyBundle on the new bundle before it replaces anything, returns what it found as verification, and fails with BUNDLE_VERIFICATION_FAILED on its errors; an addon built for another Bare ABI is an error, and a package that only declares a newer engines.bare is a warning. A bundle that fails leaves the previous qvac/worker/ and native modules in place. The harness package.json records the SDK version that bundled it, and a Node.js app warns at startup when the installed SDK differs, so bundle again after upgrading @qvac/sdk. A Node.js app that has not bundled runs qvac/worker.entry.mjs, or the SDK's own worker with every built-in plugin, without a bundle.

  • Addon platform packages: QVAC ships part of its binaries in separate packages, one per platform, for the speech addons (tts-ggml, asr-ggml, audiogen-ggml) and for @qvac/fabric, the runtime the other native addons share. They must be installed at an exact version. The package for the machine you bundle on installs with the addon; Android, iOS and other desktop hosts need theirs added. Without them the bundle cannot be built, and bundleSdk fails with HOST_PREBUILDS_MISSING, naming each package and version to add. Pass installMissingPrebuilds: true to bundleSdk to add the missing Android and iOS ones to package.json with the project's package manager (npm 7+, pnpm, bun, or Yarn Berry) before bundling; another desktop host's packages are added by hand. The package manager is read from the project's packageManager field, lockfile, or installed node_modules; a workspace member uses its workspace root's. If none of those identify it, nothing is installed and the error lists the dependencies to add. It is off by default, so a plain bundleSdk call never changes package.json or node_modules. ensureHostPrebuilds({ projectRoot, hosts, addons }) runs the install step on its own.

  • Package managers that do not hoist: the bundle imports bare-stow-runtime, and the module that starts it imports bare-stow-host, plus bare-sidecar for the bare-sidecar target or react-native-bare-kit for react-native, by name from your project. npm and bun's default linker place them in your project's node_modules with the SDK. With pnpm or bun's isolated linker, add them to your project's dependencies, for example pnpm add bare-stow-runtime bare-stow-host bare-sidecar; otherwise bundleSdk fails with BUNDLE_FAILED, naming the module it cannot find. Yarn needs nodeLinker: node-modules in .yarnrc.yml, because the bundler reads packages from node_modules. A Node.js app that runs without a bundle needs none of them.

On this page

Ask anything about QVAC.