---
title: "Plugin system"
canonical: https://docs.qvac.tether.io/sdk/v0.23/configuration/plugins/
collection: "SDK"
package: "@qvac/sdk"
line: v0.23
current_line: false
---

# Plugin system (/sdk/v0.23/configuration/plugins)



## Overview

Each QVAC AI capability maps to a **built-in plugin** in the SDK. This lets you enable only what you need for your project and reduce your application's final bundle size.

In addition, you can add **custom plugins** — both your own and community ones — that extend QVAC's capabilities. In both cases, you'll use the QVAC configuration file and then bundle either with [QVAC CLI](/cli) or [programmatically via `@qvac/sdk/commands`](#notes).

## Built-in plugins

### Catalog

All the built-in plugins you can select, along with the AI tasks that depend on each one:

| Plugin                             | Use in `qvac.config.*`                          | AI tasks that require it                                                                                                                                                                                          |
| ---------------------------------- | ----------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| LLM                                | `@qvac/sdk/llamacpp-completion/plugin`          | [Text generation](/sdk/v0.23/ai-capabilities/text-generation); [multimodal](/sdk/v0.23/ai-capabilities/multimodal); [RAG](/sdk/v0.23/ai-capabilities/rag) ; [Fine-tuning](/sdk/v0.23/ai-capabilities/fine-tuning) |
| Embeddings                         | `@qvac/sdk/llamacpp-embedding/plugin`           | [Text embeddings](/sdk/v0.23/ai-capabilities/text-embeddings); [RAG](/sdk/v0.23/ai-capabilities/rag)                                                                                                              |
| ASR with customized Whisper engine | `@qvac/sdk/whispercpp-transcription/plugin`     | [Transcription](/sdk/v0.23/ai-capabilities/transcription)                                                                                                                                                         |
| ASR with Parakeet                  | `@qvac/sdk/parakeet-transcription/plugin`       | [Transcription](/sdk/v0.23/ai-capabilities/transcription)                                                                                                                                                         |
| BCI transcription                  | `@qvac/sdk/bci-whispercpp-transcription/plugin` | [Transcription](/sdk/v0.23/ai-capabilities/transcription#bci-neural-signal-transcription)                                                                                                                         |
| BCI transcription                  | `@qvac/sdk/bci-whispercpp-transcription/plugin` | [Transcription](/sdk/v0.23/ai-capabilities/transcription#bci-neural-signal-transcription)                                                                                                                         |
| NMT                                | `@qvac/sdk/nmtcpp-translation/plugin`           | [Translation](/sdk/v0.23/ai-capabilities/translation)                                                                                                                                                             |
| TTS                                | `@qvac/sdk/onnx-tts/plugin`                     | [Text-to-Speech](/sdk/v0.23/ai-capabilities/text-to-speech)                                                                                                                                                       |
| OCR                                | `@qvac/sdk/ggml-ocr/plugin`                     | [OCR](/sdk/v0.23/ai-capabilities/ocr)                                                                                                                                                                             |
| Diffusion                          | `@qvac/sdk/sdcpp-generation/plugin`             | [Image generation](/sdk/v0.23/ai-capabilities/image-generation)                                                                                                                                                   |
| AudioGen                           | `@qvac/sdk/audiogen-ggml/plugin`                | [Music generation](/sdk/v0.23/ai-capabilities/music-generation)                                                                                                                                                   |

### Enabling

<Steps>
  <Step>
    [In your `qvac.config.*`, add the built-in plugins you’ll need in your project](/sdk/v0.23/configuration). For example:

    ```json title="qvac.config.json"
    {
      "plugins": [
        "@qvac/sdk/llamacpp-completion/plugin",
        "@qvac/sdk/ggml-ocr/plugin"
      ]
    }
    ```
  </Step>

  <Step>
    <Tabs>
      <Tab value="desktop" label="Node.js/Bare" default>
        When developing for desktop environment, use the QVAC CLI to (re)bundle the SDK only with the selected plugins:

        ```bash
        qvac bundle sdk
        ```
      </Tab>

      <Tab value="mobile" label="Expo">
        [When developing for mobile environment, use the Expo CLI to prebuild your project only with the selected plugins:](/sdk/v0.23/js-ts-sdk#expo)

        ```bash
        npx expo prebuild
        ```
      </Tab>
    </Tabs>
  </Step>
</Steps>

Only the selected plugins are included in your bundle, significantly reducing your application size. If `plugins` is omitted or an empty array, it bundles all built-in plugins by default.

<Callout type="success">
  **Tips:**

  * You **do not need** the CLI (`@qvac/cli`) to bundle the SDK. See [Notes](#notes) to learn how to do it programmatically via `@qvac/sdk/commands`.
  * See [QVAC CLI](/cli) to learn how to install and use it.
  * When developing an Electron app, you can bundle the SDK during packaging via Electron Forge. See [Tutorials → Electron → Step 4: package for distribution](/resources/tutorials/electron#step-4-package-for-distribution).
</Callout>

## Custom plugins

Custom plugins are consumed as npm packages. Install the package, enable it in your `qvac.config.*` file like any built-in plugin, then import and use its API alongside the SDK's regular API.

### Enabling

<Steps>
  <Step>
    Install the plugin package in your project. For example:

    ```bash
    npm i qvac-echo-plugin
    ```
  </Step>

  <Step>
    [In your `qvac.config.*`, add its `<package_name>/plugin` specifier alongside any built-in plugins](/sdk/v0.23/configuration). For example:

    ```json title="qvac.config.json"
    {
      "plugins": [
        "@qvac/sdk/llamacpp-completion/plugin",
        "qvac-echo-plugin/plugin" // [!code highlight]
      ]
    }
    ```
  </Step>

  <Step>
    <Tabs>
      <Tab value="desktop" label="Node.js/Bare" default>
        When developing for desktop environment, use the SDK CLI to (re)bundle the SDK with all listed plugins:

        ```bash
        qvac bundle sdk
        ```
      </Tab>

      <Tab value="mobile" label="Expo">
        [When developing for mobile environment, use the Expo CLI to prebuild your project with all listed plugins:](/sdk/v0.23/js-ts-sdk#expo)

        ```bash
        npx expo prebuild
        ```
      </Tab>
    </Tabs>
  </Step>
</Steps>

<Callout type="success">
  **Tip:** when adding one or more custom plugins, you **must** also add **all** the built-in plugins you will need to use.
</Callout>

### Usage

Custom plugins are consumed as npm packages. Install the package, enable it in your `qvac.config.*` file like any built-in plugin, then import and use its API alongside the SDK’s regular API.

```ts
import { loadModel, unloadModel } from "@qvac/sdk";

// 1. Import the API you need from the custom plugin package:
import { echo, echoStream } from "qvac-echo-plugin";

// 2. Load the model(s) required by the custom plugin:
const modelId = await loadModel({
  modelSrc: "/path/to/echo-model.bin",
  modelType: "echo",
});

// 3. Call functions exposed by the custom plugin API:
const result = await echo({ modelId, message: "Hello, plugin system!" });
console.log(result);

for await (const char of echoStream({ modelId, message: "Streaming test!" })) {
  process.stdout.write(char);
}

await unloadModel({ modelId });
```

The Python and Kotlin SDKs expose plugin invocation through the same worker contract:

<Tabs>
  <Tab value="python" label="Python" default>
    <WrapCode>
      ```python file=<rootDir>/packages/sdk-python/examples/plugins.py title="plugins.py" lineNumbers
      """Python port of packages/sdk/examples/plugins.ts (plugin invocation).

      The low-level plugin API. A model's `modelType` selects a plugin; each plugin
      exposes named `handler`s with their own request/response schemas.
      `invoke_plugin(t, model_id, handler, params)` runs one handler and returns its
      result; `invoke_plugin_stream(...)` yields streamed chunks.

      This example uses the `custom-echo-plugin` fixture (modelType `echo`), whose
      handlers are `echo` ({message} -> {message}) and `echoStream` ({message} ->
      {chunk} chunks). To run it you need a worker bundled with that plugin; point
      the loader at it via argv, or adapt the handler/params to your own plugin.
      (A production model that exposes handlers is VLA — see vla.py, which drives the
      `vlaRun` / `vlaHparams` handlers through typed wrappers.)

      RUN: python examples/plugins.py [model-src]
      """

      from __future__ import annotations

      import asyncio
      import sys

      from tetherto.qvac_sdk import (
          Client,
          invoke_plugin,
          invoke_plugin_stream,
          load_model,
          unload_model,
      )


      def print_progress(p) -> None:
          """Print model download progress; pass as `on_progress=` to `load_model`."""
          line = (
              f"▸ Downloading {p.percentage:.0f}% "
              f"({p.downloaded / 1e6:.1f}/{p.total / 1e6:.1f} MB)"
          )
          print(line, end="\r" if sys.stderr.isatty() else "\n", file=sys.stderr)
          if p.percentage >= 100:
              print(file=sys.stderr)


      async def main() -> int:
          if len(sys.argv) < 2:
              print(
                  "Usage: python examples/plugins.py <model-src>\n"
                  "  <model-src> must resolve to a model whose plugin exposes the "
                  "'echo'/'echoStream' handlers (the custom-echo-plugin fixture).",
                  file=sys.stderr,
              )
              return 1
          model_src = sys.argv[1]

          async with Client() as client:
              t = client.transport
              try:
                  print("▸ Loading plugin-backed model...")
                  model_id = await load_model(
                      t,
                      model_src=model_src,
                      model_type="echo",
                      on_progress=print_progress,
                  )
                  print(f"▸ Model loaded: {model_id}")

                  print("\n▸ 1. invoke_plugin (request/reply handler 'echo')")
                  result = await invoke_plugin(
                      t, model_id, "echo", params={"message": "hello from the python sdk"}
                  )
                  print(f"▸ echo -> {result}")

                  print("\n▸ 2. invoke_plugin_stream (streaming handler 'echoStream')")
                  print("▸ chunks: ", end="")
                  async for chunk in invoke_plugin_stream(
                      t, model_id, "echoStream", params={"message": "streamed message"}
                  ):
                      sys.stdout.write(str(getattr(chunk, "chunk", chunk)))
                      sys.stdout.flush()
                  print()

                  await unload_model(t, model_id)
                  print("▸ Model unloaded")
              except Exception as error:
                  print(f"✖ {error}", file=sys.stderr)
                  return 1
          return 0


      if __name__ == "__main__":
          sys.exit(asyncio.run(main()))
      ```
    </WrapCode>
  </Tab>

  <Tab value="kotlin" label="Kotlin">
    <WrapCode>
      ```kotlin file=<rootDir>/packages/sdk-kotlin/examples/src/Plugins.kt title="Plugins.kt" lineNumbers
      // Kotlin port of packages/sdk/examples/plugins.ts (plugin invocation).
      //
      // The low-level plugin API. A model's `modelType` selects a plugin; each plugin
      // exposes named `handler`s with their own request/response schemas.
      // `client.plugins.invoke()` runs one handler and returns its result;
      // `client.plugins.stream()` returns a `Flow` of streamed chunks.
      //
      // This example uses the `custom-echo-plugin` fixture (modelType `echo`), whose
      // handlers are `echo` ({message} -> {message}) and `echoStream` ({message} ->
      // {chunk} chunks). To run it you need a worker bundled with that plugin; point
      // the loader at it via args, or adapt the handler/params to your own plugin.
      // (A production model that exposes handlers is VLA — see Vla.kt, which drives
      // the `vlaRun` / `vlaHparams` handlers through the typed `client.vla` wrapper.)
      //
      // Plugin payloads are plugin-defined, so `params` and `result` stay
      // `JsonElement`. The stream flow carries every wire frame, including the
      // closing `done` frame, so it is filtered out here.
      //
      // RUN: ./gradlew -p examples run -Pexample=Plugins --args="<model-src>"

      import io.tether.qvac.sdk.QvacClient
      import io.tether.qvac.sdk.QvacProgressEvent
      import io.tether.qvac.sdk.generated.schema.LoadModelResponse
      import io.tether.qvac.sdk.generated.schema.ModelProgressResponse
      import io.tether.qvac.sdk.generated.schema.PluginInvokeRequest
      import io.tether.qvac.sdk.generated.schema.PluginInvokeStreamRequest
      import io.tether.qvac.sdk.jvm.JvmBareKitTransport
      import io.tether.qvac.sdk.models
      import io.tether.qvac.sdk.plugins
      import kotlin.system.exitProcess
      import kotlinx.coroutines.flow.filter
      import kotlinx.coroutines.runBlocking
      import kotlinx.serialization.json.JsonObject
      import kotlinx.serialization.json.JsonPrimitive
      import kotlinx.serialization.json.buildJsonObject
      import kotlinx.serialization.json.put

      fun main(args: Array<String>): Unit = runBlocking {
          if (args.isEmpty()) {
              System.err.println(
                  "Usage: ./gradlew -p examples run -Pexample=Plugins --args=\"<model-src>\"\n" +
                      "  <model-src> must resolve to a model whose plugin exposes the " +
                      "'echo'/'echoStream' handlers (the custom-echo-plugin fixture).",
              )
              exitProcess(1)
          }
          val modelSrc = args[0]

          val client = QvacClient(JvmBareKitTransport.connect())
          try {
              println("▸ Loading plugin-backed model...")
              val modelId = client.loadWithProgress(modelSrc, modelType = "echo")
              println("▸ Model loaded: $modelId")

              println("\n▸ 1. plugins.invoke (request/reply handler 'echo')")
              val response = client.plugins.invoke(
                  PluginInvokeRequest(
                      modelId = modelId,
                      handler = "echo",
                      params = buildJsonObject { put("message", "hello from the kotlin sdk") },
                  ),
              )
              println("▸ echo -> ${response.result}")

              println("\n▸ 2. plugins.stream (streaming handler 'echoStream')")
              print("▸ chunks: ")
              client.plugins.stream(
                  PluginInvokeStreamRequest(
                      modelId = modelId,
                      handler = "echoStream",
                      params = buildJsonObject { put("message", "streamed message") },
                  ),
              ).filter { it.done != true }.collect { frame ->
                  val chunk = (frame.result as? JsonObject)?.get("chunk") ?: frame.result
                  print((chunk as? JsonPrimitive)?.content ?: chunk.toString())
                  System.out.flush()
              }
              println()

              client.models.unload(modelId)
              println("▸ Model unloaded")
          } finally {
              client.close()
          }
      }

      /** Load [source] as [modelType], printing download progress, and return its model id. */
      private suspend fun QvacClient.loadWithProgress(source: String, modelType: String): String {
          var result: LoadModelResponse? = null
          models.loadWithProgress(source, modelType).collect { event ->
              when (event) {
                  is QvacProgressEvent.Progress -> printProgress(event.value)
                  is QvacProgressEvent.Result -> result = event.value
              }
          }
          val loaded = checkNotNull(result) { "loadModel ended without a result" }
          return loaded.modelId ?: error(loaded.error ?: "loadModel failed")
      }

      private fun printProgress(p: ModelProgressResponse) {
          val line = "▸ Downloading %.0f%% (%.1f/%.1f MB)".format(p.percentage, p.downloaded / 1e6, p.total / 1e6)
          if (System.console() != null) System.err.print("\r$line") else System.err.println(line)
          if (p.percentage >= 100) System.err.println()
      }
      ```
    </WrapCode>
  </Tab>
</Tabs>

## Runtime registration on Bare

The `plugins` array in `qvac.config.*` is **bundle-time** configuration — it controls which addons get packed into your worker. That is separate from **runtime registration**, which determines the plugins live in the worker process when an SDK call runs.

On Node.js and Expo the SDK spawns a worker that auto-registers the full built-in set, so you never register manually. In-process Bare uses `@qvac/inference` with no spawned worker — register the plugins you use before the first call:

```js
import { plugins } from "@qvac/inference";
import { llmPlugin } from "@qvac/inference/llamacpp-completion/plugin";

const sdk = plugins([llmPlugin]); // or registerPlugin(llmPlugin) from "@qvac/inference/plugins"
```

Calls made before any plugin is registered raise `PluginsNotRegisteredError`.

<Callout type="info">
  In-process Bare uses [`@qvac/inference`](https://github.com/tetherto/qvac/tree/main/packages/inference). `@qvac/bare-sdk` is deprecated; last release is 0.18.2.
</Callout>

## Notes

* In `qvac.config.*`, if `plugins` is omitted or set to an empty array, the SDK bundles all built-in plugins. If `plugins` is set, it bundles only the listed plugins.
* [Each plugin maps to one QVAC addon](https://github.com/tetherto/qvac/tree/main/packages).
* [Write a custom plugin](/sdk/v0.23/configuration/plugins/write-custom-plugin).
* **Programmatic bundling:** you can also prepare and validate the SDK bundle programmatically via API from a Node script. For example:

```ts
import { bundleSdk, verifyBundle } from "@qvac/sdk/commands";

await bundleSdk({
  projectRoot: process.cwd(),
  configPath: "./qvac.config.json",
  quiet: true,
});

const result = await verifyBundle({
  projectRoot: process.cwd(),
  addonsSource: "./qvac/worker.bundle.js",
  hosts: ["android-arm64", "ios-arm64"],
});
```

* **Addon platform packages:** QVAC ships part of its Android and iOS binaries in separate packages, for the speech addons (`tts-ggml`, `asr-ggml`, `audiogen-ggml`) and for `@qvac/fabric`, the runtime the other native addons share. They must be installed at an exact version. Pass `installMissingPrebuilds: true` to `bundleSdk` to add the missing ones to `package.json` with the project's package manager (npm 7+, pnpm, bun, or Yarn Berry) before bundling. Without them the bundle cannot be built, and `bundleSdk` fails with `HOST_PREBUILDS_MISSING`, naming each package and version to add. The package manager is read from the project's `packageManager` field, lockfile, or installed `node_modules`; a workspace member uses its workspace root's. If none of those identify it, nothing is installed and the error lists the dependencies to add. It is off by default, so a plain `bundleSdk` call never changes `package.json` or `node_modules`. `ensureHostPrebuilds({ projectRoot, hosts, addons })` runs the install step on its own.
