New: QVAC SDK now supports Kotlin. Build local AI into your JVM and Android apps.
QVAC Logo
SDKText generation
v0.24, the current release

Text generation

LLM inference for text generation and chat — i.e., use a large language model to generate text output based on input prompts and context.

Overview

Text generation uses qvac-fabric-llm.cpp as inference engine. Load any supported model using modelType: "llm". Then, provide array history as input where each element is an object with properties:

  • role: string; can be either "user" or "assistant"
  • content: string

role: "user" indicates that content is a previous prompt. role: "assistant" indicates that content is a previous inference (LLM output).

Output is generated based on the full sequence of messages provided in history.

Functions

Use the following sequence of function calls:

  1. loadModel()
  2. completion()
  3. unloadModel()

For how to use each function, see SDK — API reference.

Models

You can load any llama.cpp-compatible text-generation/chat model. Model file format: *.gguf.

  • If the model is sharded across multiple files (a multi-file bundle), see Sharded models.
  • To adapt a model to domain-specific tasks, see Fine-tuning.
  • For models available as constants, see SDK — Models.

Features

  • Event stream: completion() exposes a single ordered events async iterable plus an aggregated final promise. Events are discriminated by type — contentDelta, thinkingDelta, toolCall, toolError, completionStats, completionDone, rawDelta. The terminal completionDone event carries a stopReason (e.g. "eos", "length", "cancelled").
  • Thinking content: models that emit <think> blocks surface them as dedicated thinkingDelta events (enable with captureThinking: true), so consumers don't have to parse tags from raw text.
  • Tool calls: the model emits structured tool calls as toolCall events ordered alongside content and thinking in the same stream.
  • Tool choice: generationParams.tool_choice takes "auto" (default), "required", "none" or the name of a declared tool. On llama.cpp-backed models the sampler is constrained by the chat template's tool-call grammar, so under "required" the model cannot finish without a call, though a long text reply can use up the tokens before it reaches one, and a name restricts the call to that tool; a request whose choice cannot be applied fails before generating.
  • Tool parameters: a tool's parameters is a JSON Schema object and is passed to the model as declared, including nested properties, items, nullable types (["string", "null"]) and anyOf. Zod parameters are converted with z.toJSONSchema, so nested Zod objects and arrays keep their structure.
  • Tool errors: a tool-call region the model emitted that fails to parse or validate surfaces as a toolError event and in final.toolErrors, so an empty toolCalls can be told apart from a malformed call.
  • Past tool calls in history: an assistant message carries the calls it made in toolCalls (id, name, arguments), and a tool message names the call it answers with toolCallId and toolName. llama.cpp-backed models render them with their own chat template instead of reading them as text in content.
  • Parallel tool calls: one response may carry several tool calls on llama.cpp-backed models. generationParams.parallel_tool_calls defaults to true; set it to false to get at most one.
  • MCP: plug MCP servers into completion() so the model can use external tools (e.g., web search) via the same tool-call mechanism.
  • Raw output: with emitRawDeltas: true, every raw model token is also emitted as a rawDelta event in parallel to the structured events — useful for debugging or full-fidelity logging.
  • KV cache: cache and reuse the model's key/value attention state to speed up follow-up turns in long conversations. The kvCache parameter sets the cache key — pass a string to manage a session manually, or true to let the SDK auto-generate one.
  • Multimodal: attach images (and other media) to prompts so the model reasons over text and images together in the same conversation — see Multimodal.
  • Multi-job continuous batching: load with modelConfig.parallel >= 2 (default 1) to run several completion() calls in parallel on one loaded model. Up to parallel prompts finish in roughly the time of the slowest one (extras queue), and each call has its own requestId so you can cancel one without touching the others — see Concurrent completions.
  • Batch processing: run multiple prompts through a single loaded model in one call with batchCompletion() — optimizes resources and reduces total run time. See Batch processing.
  • Speculative decoding: load with modelConfig.spec_type to draft several tokens per step and verify them in one decode, with multi-token-prediction (MTP) layers, the model's own or an external head, or with a separate DFlash draft model. Output keeps the distribution of plain decoding. See Speculative decoding.
  • AMD NPU: on Linux x64 with an AMD XDNA2 NPU, load with modelConfig.device: "npu" to run the ops the NPU supports on it, with the CPU running the rest; the LLM layers leave the GPU free for another model. See AMD NPU.

Examples

Usage

The canonical way to consume completion() is the events async iterable plus the aggregated final promise. The following script shows how to handle each event type and read the aggregated result:

completion-events.js
/**
 * Event-driven completion — demonstrates the unified `CompletionEvent` stream.
 *
 * `completion()` returns a `CompletionRun` with two primary surfaces:
 *
 *  - `events`  — an `AsyncIterable<CompletionEvent>` of ordered, typed events
 *                (`contentDelta`, `thinkingDelta`, `toolCall`, `toolError`,
 *                 `completionStats`, `completionDone`, `rawDelta`).
 *  - `final`   — a `Promise<CompletionFinal>` that resolves once the stream
 *                ends, providing aggregated `contentText`, `thinkingText`,
 *                `toolCalls`, `stats`, and `raw.fullText`.
 *
 * Set `captureThinking: true` to attempt best-effort `<think>` block parsing
 * into dedicated `thinkingDelta` events. `final.raw.fullText` keeps the exact
 * model output.
 */
import { completion, loadModel, unloadModel, QWEN3_600M_INST_Q4 } from '@qvac/sdk';
try {
    const modelId = await loadModel({
        modelSrc: QWEN3_600M_INST_Q4,
        modelConfig: { ctx_size: 4096 },
        onProgress: (p) => {
            const mb = (n) => (n / 1e6).toFixed(1);
            const line = `▸ Downloading ${p.percentage.toFixed(0)}% (${mb(p.downloaded)}/${mb(p.total)} MB)`;
            process.stderr.write(process.stderr.isTTY ? `\r${line}` : `${line}\n`);
            if (p.percentage >= 100)
                process.stderr.write('\n');
        }
    });
    console.log(`▸ Model loaded: ${modelId}`);
    const result = completion({
        modelId,
        history: [{ role: 'user', content: 'Explain quantum computing in 2 sentences' }],
        stream: true,
        captureThinking: true
    });
    for await (const event of result.events) {
        handleEvent(event);
    }
    const final = await result.final;
    console.log('\n▸ Final result');
    console.log(`▸ Content: ${final.contentText}`);
    if (final.thinkingText) {
        console.log(`▸ Thinking: ${final.thinkingText}`);
    }
    if (final.stats) {
        console.log(`▸ ${final.stats.tokensPerSecond?.toFixed(1)} tok/s`);
    }
    if (final.toolCalls.length > 0) {
        console.log(`▸ Tool calls: ${final.toolCalls.map((c) => c.name).join(', ')}`);
    }
    if (final.stopReason) {
        console.log(`▸ Stop reason: ${final.stopReason}`);
    }
    console.log(`▸ Raw output length: ${final.raw.fullText.length} chars`);
    await unloadModel({ modelId, clearStorage: false });
}
catch (error) {
    console.error('✖', error);
    process.exit(1);
}
function handleEvent(event) {
    switch (event.type) {
        case 'contentDelta':
            process.stdout.write(event.text);
            break;
        case 'thinkingDelta':
            process.stdout.write(`\x1b[2m${event.text}\x1b[0m`);
            break;
        case 'toolCall':
            console.log(`\n▸ Tool: ${event.call.name}(${JSON.stringify(event.call.arguments)})`);
            break;
        case 'toolError':
            console.log(`\n✖ Tool error [${event.error.code}]: ${event.error.message}`);
            break;
        case 'completionStats':
            console.log(`\n▸ ${event.stats.tokensPerSecond?.toFixed(1)} tok/s`);
            break;
        case 'completionDone':
            if (event.stopReason === 'error' && 'error' in event) {
                console.log(`\n✖ ${event.error.message}`);
            }
            break;
        case 'rawDelta':
            break;
    }
}

The examples below (Tool call, MCP, KV cache) still consume result.tokenStream and result.toolCallStream, which are convenience wrappers around the canonical events / final stream shown above. Both APIs are supported; new code should prefer events / final.

Concurrent completions

Use this when you have more than one prompt in flight against the same loaded model at the same time — for example, serving several users, or fanning out related sub-tasks — and want them to finish together instead of one after another.

Load the model with modelConfig.parallel >= 2, then call completion() several times without awaiting between calls. The SDK runs up to parallel of them together on the model, so up to parallel prompts finish in roughly the time of the slowest one instead of the sum of all of them; beyond that they queue. Each call has its own requestId; cancel({ requestId }) stops just that call and its peers keep going — useful for a per-request stop button, rather than killing the whole model.

The default is 1 (one request at a time), so you have to set parallel explicitly to enable batching. Also raise ctx_size with it: the model's context window is split evenly across the parallel slots, so parallel: 4 on a ctx_size: 4096 model gives each request only ~1024 tokens. Size ctx_size for parallel × per-request context.

completion() and batchCompletion() share the model's parallel slots. Each call is one request — a big batchCompletion({ prompts: N }) never blocks more than one concurrent completion() from starting — but every prompt in the batch takes one decode slot, so a large batch fills all parallel slots itself and any concurrent completion() you launch will only start decoding once the batch has room. If you already have every prompt ready up front, batchCompletion() is often simpler; use concurrent completion() when prompts arrive over time or you want independent streams and cancellation per call.

The following script loads a parallel: 4 model, fires four completions at once, and then cancels one of two long runs to show its peer keeps decoding:

multi-job-completion.js
/**
 * Multi-job continuous batching — concurrent completions on one loaded LLM.
 *
 * Unlike `batchCompletion()` (which bundles many prompts into ONE call), this
 * shows independent `completion()` calls fired concurrently against the SAME
 * loaded model. When the model is loaded with `modelConfig.parallel >= 2`, the
 * SDK admits up to `parallel` completions at once and the llama.cpp addon
 * decodes their sequences together — so a burst of prompts finishes in roughly
 * the time of the slowest one, not the sum of all of them.
 *
 * Each `completion()` returns an independent `CompletionRun`:
 *  - `events`     — an `AsyncIterable` of streamed events for that call.
 *  - `final`      — a `Promise<CompletionFinal>` with THIS call's own
 *                   `contentText` and `stats` (its own tokens/sec, TTFT).
 *                   `stats.avgConcurrentSeq` stays model-level (how busy the
 *                   shared backend was).
 *  - `requestId`  — cancel just this call with `cancel({ requestId })`; peers
 *                   keep decoding.
 *
 * Completions and batches share the model's `parallel` admission cap: with
 * `parallel` in flight, admitting one more waits FIFO until a slot frees.
 *
 * Run from packages/sdk:
 *   npx tsx examples/multi-job-completion.ts
 */
import { completion, cancel, loadModel, unloadModel, InferenceCancelledError, LLAMA_3_2_1B_INST_Q4_0 } from '@qvac/sdk';
const PROMPTS = [
    { id: 'cherry', ask: 'Reply with only the word CHERRY.' },
    { id: 'banana', ask: 'Reply with only the word BANANA.' },
    { id: 'grape', ask: 'Reply with only the word GRAPE.' },
    { id: 'lemon', ask: 'Reply with only the word LEMON.' }
];
try {
    // `parallel: 4` opens the concurrent decode slots continuous batching needs.
    const modelId = await loadModel({
        modelSrc: LLAMA_3_2_1B_INST_Q4_0,
        modelType: 'llm',
        modelConfig: { ctx_size: 4096, parallel: 4 },
        onProgress: (p) => {
            const mb = (n) => (n / 1e6).toFixed(1);
            const line = `▸ Downloading ${p.percentage.toFixed(0)}% (${mb(p.downloaded)}/${mb(p.total)} MB)`;
            process.stderr.write(process.stderr.isTTY ? `\r${line}` : `${line}\n`);
            if (p.percentage >= 100)
                process.stderr.write('\n');
        }
    });
    console.log(`▸ Model loaded: ${modelId}`);
    // ---- Concurrent completions. Fire every call before awaiting any, so all
    // four are admitted and decode together (not one-after-another). ----
    console.log('\n▸ Firing 4 completions concurrently:');
    const started = Date.now();
    const runs = PROMPTS.map(({ ask }) => completion({
        modelId,
        history: [{ role: 'user', content: ask }],
        generationParams: { temp: 0, seed: 42, predict: 16 },
        stream: true
    }));
    const outputs = await Promise.all(runs.map(async (run, i) => {
        const final = await run.final;
        return {
            id: PROMPTS[i].id,
            text: final.contentText.replace(/\s+/g, ' ').trim(),
            stats: final.stats
        };
    }));
    const elapsed = Date.now() - started;
    for (const { id, text, stats } of outputs) {
        const tps = stats?.tokensPerSecond?.toFixed(1) ?? 'n/a';
        console.log(`  ▸ ${id}: "${text}"  (own ${tps} tok/s)`);
    }
    // Report the engine's own measure of overlap rather than inferring it from
    // wall-clock: avgConcurrentSeq > 1 means the addon decoded sequences together.
    const peakConcurrent = Math.max(...outputs.map((o) => o.stats?.avgConcurrentSeq ?? 0));
    console.log(`\n▸ All 4 finished in ${elapsed} ms; engine avg concurrent sequences: ${peakConcurrent.toFixed(1)} (> 1 = decoded together).`);
    // ---- Per-request cancel isolation. Fire two long runs; cancel only the
    // first by its requestId. The second must keep decoding past the cancel. ----
    console.log('\n▸ Per-request cancel: cancelling one of two in-flight completions:');
    const doomed = completion({
        modelId,
        history: [{ role: 'user', content: 'Write a long story about an otter.' }],
        generationParams: { temp: 0, seed: 1, predict: 256 },
        stream: true
    });
    const survivor = completion({
        modelId,
        history: [{ role: 'user', content: 'Write a long story about a melon.' }],
        generationParams: { temp: 0, seed: 42, predict: 256 },
        stream: true
    });
    // Consume the survivor's stream directly so we can count tokens produced AFTER
    // the cancel is acknowledged — a short survivor could otherwise finish before
    // the cancel lands and pass even under a model-wide cancel.
    const survivorTokens = survivor.events[Symbol.asyncIterator]();
    // Wait until BOTH runs are actually streaming before cancelling, so the cancel
    // hits a genuinely in-flight peer, not a pre-admission drop.
    await Promise.all([doomed.events[Symbol.asyncIterator]().next(), survivorTokens.next()]);
    await cancel({ requestId: doomed.requestId });
    let doomedCancelled = false;
    try {
        await doomed.final;
    }
    catch (err) {
        doomedCancelled = err instanceof InferenceCancelledError;
    }
    // Drain what the survivor emits from here on: a per-request cancel leaves it
    // decoding to completion, a model-wide cancel would have stopped it.
    let tokensAfterCancel = 0;
    for (let next = await survivorTokens.next(); !next.done; next = await survivorTokens.next()) {
        tokensAfterCancel++;
    }
    const survivorText = (await survivor.final).contentText.replace(/\s+/g, ' ').trim();
    console.log(`  ▸ cancelled run -> ${doomedCancelled ? 'cancelled' : 'NOT cancelled'}`);
    console.log(`  ▸ peer kept decoding -> +${tokensAfterCancel} tokens after the cancel ack`);
    await unloadModel({ modelId, clearStorage: false });
    // A zero exit must prove concurrent scheduling, typed cancellation, and survivor
    // isolation: the first burst reported multiple resident sequences, the doomed
    // run cancelled, and its peer produced tokens AFTER the cancel landed.
    if (peakConcurrent <= 1 || !doomedCancelled || tokensAfterCancel === 0) {
        console.error(`✖ multi-job behavior not demonstrated (peakConcurrent=${peakConcurrent.toFixed(2)}, doomedCancelled=${doomedCancelled}, tokensAfterCancel=${tokensAfterCancel}, survivorText="${survivorText}")`);
        process.exit(1);
    }
    process.exit(0);
}
catch (error) {
    console.error('✖', error);
    process.exit(1);
}

Tool call

JS and TS examples here and in Deferred tool loading import their tool schemas and a mock executor from a sibling shared module. Save it next to the example — as shared.js for JS, shared.ts for TS.

shared.js
import { z } from 'zod';
export const weatherSchema = z.object({
    city: z.string().describe('City name'),
    country: z.string().describe('Country code').optional()
});
export const horoscopeSchema = z.object({
    sign: z.string().describe('An astrological sign like Taurus or Aquarius')
});
export const toolSchemas = {
    get_weather: weatherSchema,
    get_horoscope: horoscopeSchema
};
export const tools = [
    {
        name: 'get_weather',
        description: 'Get current weather for a city',
        parameters: weatherSchema
    },
    {
        name: 'get_horoscope',
        description: "Get today's horoscope for an astrological sign",
        parameters: horoscopeSchema
    }
];
export function mockExecute(name, args) {
    if (name === 'get_weather') {
        const a = args;
        return JSON.stringify({
            city: a.city,
            country: a.country ?? 'unknown',
            temperature: '22°C',
            condition: 'Partly cloudy'
        });
    }
    if (name === 'get_horoscope') {
        return JSON.stringify({
            sign: args.sign,
            horoscope: 'Today is a great day for new beginnings.'
        });
    }
    return JSON.stringify({ error: `Unknown tool: ${name}` });
}

The following script shows how to provide tool definitions to completion(), consume the streaming output, and read the parsed tool calls.

completion-tool-call.js
import { completion, loadModel, unloadModel, QWEN3_1_7B_INST_Q4 } from '@qvac/sdk';
import { tools, toolSchemas, mockExecute } from './shared.js';
try {
    const modelId = await loadModel({
        modelSrc: QWEN3_1_7B_INST_Q4,
        modelConfig: {
            ctx_size: 4096,
            tools: true
        },
        onProgress: (p) => {
            const mb = (n) => (n / 1e6).toFixed(1);
            const line = `▸ Downloading ${p.percentage.toFixed(0)}% (${mb(p.downloaded)}/${mb(p.total)} MB)`;
            process.stderr.write(process.stderr.isTTY ? `\r${line}` : `${line}\n`);
            if (p.percentage >= 100)
                process.stderr.write('\n');
        }
    });
    console.log(`▸ Model loaded successfully! Model ID: ${modelId}`);
    const history = [
        {
            role: 'system',
            content: 'You are a helpful assistant that can use tools to get the weather and horoscope.'
        },
        {
            role: 'user',
            content: "What's the weather in Tokyo and my horoscope for Aquarius?"
        }
    ];
    console.log('\n▸ AI Response:');
    console.log('▸ (Streaming with tool definitions in prompt)\n');
    const result = completion({ modelId, history, stream: true, tools });
    const tokensTask = (async () => {
        for await (const token of result.tokenStream) {
            process.stdout.write(token);
        }
    })();
    const toolsTask = (async () => {
        for await (const evt of result.toolCallStream) {
            console.log(`\n\n▸ Tool Call Detected: ${evt.call.name}(${JSON.stringify(evt.call.arguments)})`);
            console.log(`▸ ID: ${evt.call.id}`);
        }
    })();
    await Promise.all([tokensTask, toolsTask]);
    const stats = await result.stats;
    const toolCalls = await result.toolCalls;
    console.log('\n\n▸ Parsed Tool Calls:');
    if (toolCalls.length > 0) {
        for (const call of toolCalls) {
            console.log(`▸ ${call.name}(${JSON.stringify(call.arguments)})`);
            const schema = toolSchemas[call.name];
            if (schema) {
                const validated = schema.safeParse(call.arguments);
                if (validated.success) {
                    console.log(`▸ Arguments validated with Zod`);
                }
                else {
                    console.log(`✖ Validation failed:`, validated.error);
                }
            }
        }
    }
    else {
        console.log('▸ No tool calls detected in response');
    }
    console.log('\n▸ Performance Stats:', stats);
    if (toolCalls.length > 0) {
        console.log('\n\n▸ Simulating Tool Execution...');
        const toolResults = toolCalls.map((call) => {
            const result = mockExecute(call.name, call.arguments);
            console.log(`▸ ${call.name}: ${result}`);
            return { toolCallId: call.id, result };
        });
        history.push({
            role: 'assistant',
            content: await result.text
        });
        for (const toolResult of toolResults) {
            history.push({
                role: 'tool',
                content: toolResult.result
            });
        }
        console.log('\n\n▸ Follow-up Response with Tool Results:');
        const followUpResult = completion({
            modelId,
            history,
            stream: true,
            tools
        });
        for await (const token of followUpResult.tokenStream) {
            process.stdout.write(token);
        }
        const followUpStats = await followUpResult.stats;
        console.log('\n\n▸ Follow-up Stats:', followUpStats);
    }
    console.log('\n\n▸ Completed!');
    await unloadModel({ modelId, clearStorage: false });
}
catch (error) {
    console.error('✖', error);
    process.exit(1);
}

Worker-orchestrated tool loop

Python and Kotlin also expose an advanced worker-orchestrated path, where the worker runs the whole multi-turn tool loop and calls back into your local handlers. Use this only when you need the worker to own the tool loop rather than managing each turn in application code:

completion_orchestrate.py
"""Worker-orchestrated tool loop — mirrors the SDK's server-side tool loop
(beyond the client-side loop in completion_tools.py).

`completion_orchestrate()` hands the multi-turn tool loop to the WORKER: when
the model asks for a tool, the worker emits a callback frame, this client runs
the registered `handler`, writes the result back, and generation continues —
no manual history stitching. Every tool needs a `handler` (it runs code on
this machine, so only a local worker may orchestrate).

Advanced path: `completion_orchestrate` lives in `tetherto.qvac_sdk._completion` rather than
the flat `tetherto.qvac_sdk` surface. The public tool API is `completion(tools=...)` (the
client-side loop in completion_tools.py), matching the JS SDK; worker
orchestration has no JS client wrapper yet.

RUN: python examples/completion_orchestrate.py
"""

from __future__ import annotations

import asyncio
import sys

from tetherto.qvac_sdk import Client, load_model, unload_model
from tetherto.qvac_sdk._completion import completion_orchestrate
from tetherto.qvac_sdk.models import QWEN3_1_7B_INST_Q4


def print_progress(p) -> None:
    """Print model download progress; pass as `on_progress=` to `load_model`."""
    line = (
        f"▸ Downloading {p.percentage:.0f}% "
        f"({p.downloaded / 1e6:.1f}/{p.total / 1e6:.1f} MB)"
    )
    print(line, end="\r" if sys.stderr.isatty() else "\n", file=sys.stderr)
    if p.percentage >= 100:
        print(file=sys.stderr)


async def get_weather(arguments) -> str:
    return f"The weather in {arguments.get('location')} is 22°C and sunny."


async def get_horoscope(arguments) -> str:
    return f"{arguments.get('sign')}: today favours careful validation."


TOOLS = [
    {
        "name": "get_weather",
        "description": "Get the current weather for a location",
        "parameters": {
            "type": "object",
            "properties": {"location": {"type": "string"}},
            "required": ["location"],
        },
        "handler": get_weather,
    },
    {
        "name": "get_horoscope",
        "description": "Get the horoscope for a star sign",
        "parameters": {
            "type": "object",
            "properties": {"sign": {"type": "string"}},
            "required": ["sign"],
        },
        "handler": get_horoscope,
    },
]


async def main() -> int:
    async with Client() as client:
        t = client.transport
        try:
            model_id = await load_model(
                t,
                model_src=QWEN3_1_7B_INST_Q4,
                model_config={"ctx_size": 4096, "tools": True},
                on_progress=print_progress,
            )
            print(f"▸ Model loaded: {model_id}\n")

            run = completion_orchestrate(
                t,
                model_id=model_id,
                history=[
                    {
                        "role": "user",
                        "content": "What's the weather in Tokyo and my horoscope for Aquarius?",
                    }
                ],
                tools=TOOLS,
                max_tool_turns=4,
            )

            async for event in run.events:
                if event.type == "contentDelta":
                    sys.stdout.write(event.text)
                    sys.stdout.flush()

            final = await run.final
            print("\n\n▸ Final answer:", final.content_text)

            await unload_model(t, model_id)
        except Exception as error:
            print(f"✖ {error}", file=sys.stderr)
            return 1
    return 0


if __name__ == "__main__":
    sys.exit(asyncio.run(main()))

Deferred tool loading

A large tool set costs prompt tokens on every conversation and makes the model's choice harder. Setting deferLoading: true on a tool keeps it registered but leaves its parameter schema out of the prompt: the model sees only its name and description, in a compact catalog carried by a built-in tool_search tool, and loads the schema when it needs it.

The JavaScript and TypeScript examples below import the same shared module shown under Tool call; save it alongside this example too.

completion-deferred-tools.js
// Deferred tool loading: a large tool set stays out of the prompt until the
// model asks for it. Only `get_weather` and the built-in `tool_search` (whose
// description carries a name-and-description catalog of the rest) are in the
// initial prompt; a deferred schema is declared in the prompt once the model
// has searched for it.
import { TOOL_SEARCH_NAME, completion, executeToolSearch, loadModel, unloadModel, QWEN3_1_7B_INST_Q4 } from '@qvac/sdk';
import { z } from 'zod';
import { weatherSchema, horoscopeSchema, mockExecute } from './shared.js';
const tools = [
    {
        name: 'get_weather',
        description: 'Get current weather for a city',
        parameters: weatherSchema
    },
    {
        name: 'get_horoscope',
        description: "Get today's horoscope for an astrological sign",
        parameters: horoscopeSchema,
        deferLoading: true,
        group: 'astrology'
    },
    {
        name: 'get_moon_phase',
        description: 'Get the current phase of the moon',
        parameters: z.object({ date: z.string().describe('ISO date').optional() }),
        deferLoading: true,
        group: 'astrology'
    }
];
try {
    const modelId = await loadModel({
        modelSrc: QWEN3_1_7B_INST_Q4,
        modelConfig: { ctx_size: 4096, tools: true }
    });
    console.log(`▸ Model loaded. Model ID: ${modelId}`);
    // One search loads at most two definitions, one unless the model asks for
    // more. Keep these the same for the whole conversation.
    const params = { modelId, tools, toolSearch: { defaultLimit: 1, maxLimit: 2 } };
    const history = [
        {
            role: 'system',
            content: 'You are a helpful assistant. Search for a tool before calling it.'
        },
        { role: 'user', content: "What's my horoscope for Aquarius?" }
    ];
    console.log('\n▸ Turn 1 — only get_weather and tool_search are in the prompt\n');
    const first = completion({ ...params, history, stream: true });
    for await (const token of first.tokenStream)
        process.stdout.write(token);
    const calls = await first.toolCalls;
    const search = calls.find((call) => call.name === TOOL_SEARCH_NAME);
    if (!search) {
        console.log('\n\n▸ The model answered without searching; nothing more to show.');
    }
    else {
        console.log(`\n\n▸ The model searched: ${JSON.stringify(search.arguments)}`);
        // Running the search is the SDK's job, not a tool handler's. The result is
        // an ordinary tool message naming what it loaded: keeping it in the history
        // is what declares those definitions on the next turn. Passing the same
        // params applies the limits the model was told.
        history.push({ role: 'assistant', content: await first.text });
        history.push({
            role: 'tool',
            content: executeToolSearch(params, search.arguments, history)
        });
        console.log('▸ Definitions loaded. Turn 2 — the model can now call them directly.\n');
        const second = completion({ ...params, history, stream: true });
        for await (const token of second.tokenStream)
            process.stdout.write(token);
        for (const call of await second.toolCalls) {
            console.log(`\n\n▸ Tool call: ${call.name}(${JSON.stringify(call.arguments)})`);
            console.log(`▸ Result: ${mockExecute(call.name, call.arguments)}`);
        }
    }
    console.log('\n\n▸ Completed!');
    await unloadModel({ modelId, clearStorage: false });
}
catch (error) {
    console.error('✖', error);
    process.exit(1);
}

The Python and Kotlin examples take the worker-orchestrated loop while the others take the client-side one. Deferring a tool works the same in both languages — the worker builds the catalog and reads back what the conversation has loaded — but running tool_search from client code needs the executeToolSearch helper, which only the JS SDK exports. On the orchestrated path the worker runs the search itself.

The model calls tool_search({ query }) with a capability or an exact tool name; the SDK searches the registered tools outside inference and returns a tool result naming the matches, the next turn declares their definitions in the prompt alongside the other tools, and the model calls them by name.

Notes:

  • Tools without deferLoading behave exactly as before.

  • tool_search is a reserved name and is added only when something defers.

  • A search ranks the deferred tools by where the query matches: an exact name first, then words in the name, then in the description, then in the group. An exact name loads that tool alone. Any other query loads the best match and every match at least half as strong, up to the search limit. A word that only appears in another tool's description or group does not load it beside a stronger match, while tools that match equally well load together.

  • Two limits control how many tool definitions one tool_search call loads. The model can ask for a number itself, with the call's limit argument.

    • defaultLimit (3 unless you set it) is how many load when the model does not ask for a number.
    • maxLimit (8 unless you set it) is the most one call can load. If the model asks for more, it gets maxLimit.

    For example, with the defaults a search for github loads up to 3 matching tools, and a search that asks for limit: 20 still loads at most 8.

    Set them with toolSearch: { defaultLimit, maxLimit } on completion(), or per prompt on batchCompletion(); Python takes tool_search= and Kotlin QvacCompletionOptions(toolSearch = QvacToolSearchLimits(...)). A defaultLimit above maxLimit, or a limit under 1, is rejected.

  • Set the limits once per conversation. The model reads them in the tool_search definition near the start of the prompt, so changing them re-processes the prompt from there.

  • When you run tool_search yourself, pass executeToolSearch the same params you gave completion() (executeToolSearch(params, call.arguments, history)) so it applies the limits the model was told. Given only a tool list, it applies the defaults. With mcp, pass await resolveToolSearchSource(params) instead, once per conversation: it lists the MCP tools the way completion() does.

  • group is the heading a tool is listed under in the catalog — a skill or an MCP server name.

  • To defer an MCP server's tools, set it on the client entry: mcp: [{ client, deferLoading: true, group: 'github' }].

  • Loading a definition twice declares it once; the search result says the tool is already available.

  • Give tools distinct names. A deferred tool the prompt could not declare is left out of the catalog and cannot be loaded: an empty name, auto, none or required, or a name equal to an always-loaded tool's, tool_search or an earlier deferred tool's once every run of characters outside A-Z, a-z, 0-9 and - is read as - (get_weather and get-weather).

  • completion() stays stateless. The loaded set is read back from the conversation, so a reopened chat given the same tools and the same history has the same tools loaded, and concurrent chats cannot mix.

  • generationParams.tool_choice cannot name a deferred tool — its schema is not in the prompt, so the call cannot be constrained. Force "tool_search" instead.

  • The tool block sits near the start of the prompt, so the turn after a load re-processes most or all of the prompt instead of reusing the cached prefix.

  • On the worker-orchestrated loop the SDK runs tool_search itself; your handlers are never called for it.

MCP

You create and manage the MCP client, connect it to one or more MCP servers, and pass it to completion(). The following script shows how to attach an MCP client to completion() so the model can call a web search tool and then continue with the results:

completion-mcp.js
/**
 * MCP DuckDuckGo Search Example
 *
 * A web search example using DuckDuckGo - no API key required!
 * The server provides tools to search the web and get answers.
 *
 * Prerequisites:
 * - Install MCP SDK: npm install @modelcontextprotocol/sdk
 *
 * Run with: npx tsx examples/mcp-websearch.ts
 */
import { completion, loadModel, unloadModel, QWEN3_1_7B_INST_Q4 } from '@qvac/sdk';
// MCP SDK is a user-installed optional dependency
// Install with: npm install @modelcontextprotocol/sdk
import { Client } from '@modelcontextprotocol/sdk/client/index.js';
import { StdioClientTransport } from '@modelcontextprotocol/sdk/client/stdio.js';
function parseSearchResults(mcpResult) {
    try {
        const result = mcpResult;
        // Extract text content from MCP response
        const textContent = result.content?.find((c) => c.type === 'text');
        if (!textContent?.text) {
            return JSON.stringify(mcpResult);
        }
        // Parse the JSON array of search results
        const rawResults = JSON.parse(textContent.text);
        // Extract just the useful fields (title, url, snippet)
        const cleanResults = rawResults.slice(0, 5).map((r) => ({
            title: r.title ?? 'Unknown',
            url: r.url ?? '',
            snippet: r.snippet ?? ''
        }));
        // Format as concise text for LLM
        return cleanResults
            .map((r, i) => `[${i + 1}] ${r.title}\n    URL: ${r.url}\n    ${r.snippet}`)
            .join('\n\n');
    }
    catch {
        // If parsing fails, return a truncated version
        const str = typeof mcpResult === 'string' ? mcpResult : JSON.stringify(mcpResult);
        return str.slice(0, 2000);
    }
}
let mcpClient = null;
try {
    console.log('▸ MCP DuckDuckGo Search Example\n');
    // ============================================================
    // STEP 1: Connect to DuckDuckGo MCP server
    // ============================================================
    console.log('▸ Starting DuckDuckGo MCP server...');
    mcpClient = new Client({
        name: 'qvac-ddg-example',
        version: '1.0.0'
    });
    const transport = new StdioClientTransport({
        command: 'npx',
        args: ['-y', '@oevortex/ddg_search']
    });
    await mcpClient.connect(transport);
    console.log('▸ MCP server connected\n');
    // ============================================================
    // STEP 2: Load model
    // ============================================================
    console.log('▸ Loading model...');
    const modelId = await loadModel({
        modelSrc: QWEN3_1_7B_INST_Q4,
        modelConfig: {
            ctx_size: 4096,
            tools: true
        },
        onProgress: (p) => {
            const mb = (n) => (n / 1e6).toFixed(1);
            const line = `▸ Downloading ${p.percentage.toFixed(0)}% (${mb(p.downloaded)}/${mb(p.total)} MB)`;
            process.stderr.write(process.stderr.isTTY ? `\r${line}` : `${line}\n`);
            if (p.percentage >= 100)
                process.stderr.write('\n');
        }
    });
    console.log(`\n▸ Model loaded\n`);
    // ============================================================
    // STEP 3: Ask AI to search the web (with MCP client)
    // ============================================================
    const history = [
        {
            role: 'system',
            content: `You are a helpful assistant with access to web search.
Use the search tool when you need current information.
Always cite your sources with the URL.`
        },
        {
            role: 'user',
            content: 'What is the current weather in New York City?'
        }
    ];
    console.log('▸ Asking AI to search the web...\n');
    console.log('▸ AI Response:');
    // Pass MCP client directly to completion - tools are adapted internally!
    const result = completion({
        modelId,
        history,
        stream: true,
        mcp: [{ client: mcpClient, includeResources: false }]
    });
    for await (const token of result.tokenStream) {
        process.stdout.write(token);
    }
    const toolCalls = await result.toolCalls;
    console.log('\n');
    // ============================================================
    // STEP 4: Execute tool calls using call() - automatic MCP routing!
    // ============================================================
    if (toolCalls.length > 0) {
        console.log('▸ Executing search...\n');
        const toolResults = [];
        for (const toolCall of toolCalls) {
            console.log(`▸ ${toolCall.name}(${JSON.stringify(toolCall.arguments)})`);
            if (!toolCall.invoke) {
                console.log(`▸ No handler found for tool "${toolCall.name}"`);
                continue;
            }
            // Use invoke() - automatically routes to the correct MCP client!
            const mcpResult = await toolCall.invoke();
            // Parse and clean up the search results
            const cleanResult = parseSearchResults(mcpResult);
            console.log(`▸ Got search results:`);
            console.log(cleanResult
                .split('\n')
                .map((l) => `      ${l}`)
                .join('\n'));
            console.log();
            toolResults.push({ id: toolCall.id, result: cleanResult });
        }
        // ============================================================
        // STEP 5: Continue with search results
        // ============================================================
        console.log('▸ Getting AI response with search results...\n');
        history.push({
            role: 'assistant',
            content: await result.text
        });
        for (const tr of toolResults) {
            history.push({
                role: 'tool',
                content: tr.result
            });
        }
        console.log('▸ Final Response:');
        const finalResult = completion({
            modelId,
            history,
            stream: true,
            mcp: [{ client: mcpClient, includeResources: false }]
        });
        for await (const token of finalResult.tokenStream) {
            process.stdout.write(token);
        }
        console.log('\n');
    }
    // ============================================================
    // Cleanup
    // ============================================================
    console.log('▸ Cleaning up...');
    await unloadModel({ modelId, clearStorage: false });
    console.log('▸ Done\n');
    console.log('▸ Example completed!');
    process.exit(0);
}
catch (error) {
    console.error('✖', error);
    process.exit(1);
}
finally {
    if (mcpClient) {
        try {
            await mcpClient.close();
        }
        catch {
            // Ignore close errors
        }
    }
}

MCP is available in the JS/TS client only. The integration lives in the client, not in the worker, so the Python and Kotlin clients cannot reach it.

Count tokens before generation

Use calcTokens() with a loaded language model to estimate prompt size without starting a generation job:

const tokens = await calcTokens({
  modelId,
  history: [{ role: 'system', content: 'You are a helpful assistant.' }],
  text: 'Summarize this conversation.'
})

When neither history nor enabled tools are supplied, text is counted as raw text. Otherwise, the SDK renders the prompt with the loaded model's chat template, adds the model's configured system prompt when the history has none, and appends text as the final user turn. Load the model with modelConfig: { tools: true } for tool definitions to be counted; otherwise they are ignored and text-only input remains raw. Pass the same toolSearch limits to calcTokens() and completion() when using deferred tools. Media content and attachments are not supported. The call returns a number and can run while the model is generating.

The count uses every supplied tool and the reasoning setting from model load. If completion() uses a named generationParams.tool_choice or overrides generationParams.reasoning_budget for one request, that request can render a different prompt and report a different promptTokens value.

KV cache

The kvCache parameter on completion() both enables caching and sets the cache key. It accepts:

  • A non-empty string (e.g. "user-123-session-a") — use this key for a caller-managed cache. You control the session identity, so you can reuse the same key across turns and later clean it up with deleteCache({ kvCacheKey }).
  • true — SDK-managed auto cache. The SDK generates the key internally from the conversation history. Convenient for single-session use; you don't get a stable key for per-session cleanup, but you can reclaim them in bulk with deleteCache({ auto: true }).
  • false or undefined — no persistent cache key (default). Keyless requests reuse a shared prefix only when the model loads with cache_share_across_keys: true.

SDK-managed auto caches are removed after 24 hours idle or under disk quota pressure. The quota is 512 MiB on React Native and 4 GiB on other runtimes. deleteCache({ auto: true }) reclaims them on demand instead of waiting for either bound, skipping any cache a request is still using. Caller-managed string-key caches are not subject to the automatic retention policy and are not touched by that call — clean those up with deleteCache({ kvCacheKey }).

On a cached turn, stats.promptTokens counts only the prompt tokens the model processed on that turn. Tokens reused from the cache are not included, so a warm turn reports far fewer than the full prompt. stats.cacheTokens is the number of tokens held in the cache after the turn. Before v0.22, promptTokens counted the whole prompt.

Prompt-prefix sharing across conversations is opt-in. Set cache_share_across_keys: true in the model's modelConfig to let a decoder-only request start from the longest text or image prefix held by another conversation on the same loaded model. This also applies to requests without kvCache; their state stays in memory but is never written to an SDK cache file. With the default false, a keyless request starts cold and resets afterwards, while a keyed request reuses only its own state. Keep the default on multi-tenant hosts: enabling sharing can expose prompt-prefix overlap through stats.promptTokens, stats.cacheTokens and timing.

There is no separate kvCacheKey completion parameter — when you want a caller-managed cache, pass the key string directly as kvCache:

completion({
  modelId,
  history,
  stream: true,
  kvCache: "user-123-session-a",
});

On hybrid and recurrent models (Qwen3.5, Jamba, DeepSeek V4) and on sliding-window models (Gemma 3/4, gpt-oss), a turn under a caller-managed kvCache key whose history diverges from the cache, such as the next turn under a thinking template or a regenerate, continues from a checkpoint instead of reprocessing the whole conversation. Each checkpoint costs host RAM, up to hundreds of MiB on larger models, which model fit checks do not count. Tune them at load time in modelConfig:

  • cache_checkpoints: checkpoints kept per sequence (0–1024, default 1). Editing the user message k-th from the end without reprocessing needs k + 1; 0 turns checkpoints off.
  • cache_checkpoints_max_bytes: byte budget for them per sequence, 0 (default) for unlimited.
  • cache_checkpoint_storage: 'memory' (default) or 'disk'. With 'disk', set cache_checkpoint_dir to a writable directory; on mobile, use the app's cache directory.
  • cache_ram_mib: host-RAM budget, in MiB, for cached conversations that are not running, so switching back to a caller-managed kvCache key skips reading its cache file. Default 0 (off).
  • cache_share_across_keys: whether requests may reuse a prefix held by another conversation. Default false; set true to opt in to model-wide sharing.

Start a chat from a primed cache

In-memory reuse only helps while the base conversation is still loaded. To start a chat from a caller-managed cache that may no longer be in memory, such as a system prompt and tools primed with prefill in an earlier session, pass its key as seedFrom:

// The model is loaded with `cache_share_across_keys: true` in its modelConfig.
// Prime the base once.
await completion({ modelId, history: systemAndTools, stream: false, kvCache: "base", prefill: true }).final;

// Each new chat starts from it and processes only its own message.
completion({
  modelId,
  history: [...systemAndTools, { role: "user", content: "First question" }],
  stream: true,
  kvCache: "chat-2",
  seedFrom: "base",
});
  • seedFrom needs a string kvCache other than itself; it has no meaning for the auto cache.
  • The model reads the seedFrom cache and reuses the prefix it shares with the prompt, so a history that does not continue the base reuses less, down to nothing on hybrid and recurrent models. The source is never written; the chat's own key is.
  • It applies only while the chat's kvCache key has no cache. Once the chat has one, later turns ignore it.
  • The call is refused when no cache named seedFrom exists for the model and system prompt, or when the model was not loaded with cache_share_across_keys: true, which is off by default.
  • The source must have been saved by a model loaded with the same parallel value.

Prepare the system prompt and tools ahead of time

Use prefill: true to evaluate a prompt before the first question arrives. It requires kvCache: true or a nonempty named key and finishes with stats, no answer and no tool calls. Await the priming run before sending the question:

const history = [{ role: "system", content: "You are a helpful assistant." }];
await completion({ modelId, history, tools, stream: false, kvCache: "chat", prefill: true }).final;
const answer = await completion({
  modelId,
  history: [...history, { role: "user", content: "What is the weather in Lugano?" }],
  tools,
  stream: false,
  kvCache: "chat",
}).final;

Load the model with modelConfig: { tools: true } when passing tools. Resend the same system prompt and tools on the question so their cached prefix can be reused. A prefill rejects forced generationParams.tool_choice values ("required" or a named tool, including "tool_search"); use the default "auto" or "none" instead. A prefill also needs at least one message: a system message in the history, or modelConfig.system_prompt set at load. Tools alone are rejected. Auto-cache priming saves the complete prompt history without appending an empty assistant message.

With kvCache: true, the first question moves the cache to the conversation's new history, so the prime serves one conversation. Use a named key for reuse across conversations, or prime each conversation separately.

Affected templates require @qvac/llm-llamacpp 0.64.1 or later. Full reuse is template-dependent; arbitrary custom templates may fall back to normal rendering. Cache priming is available through the library API, not the CLI OpenAI-compatible HTTP endpoints or model-load configuration.

Compare cached and uncached turns

The following script enables kvCache: true to speed up follow-up turns, and then compares it with kvCache: false on the same history:

completion-kv-cache.js
import { completion, LLAMA_3_2_1B_INST_Q4_0, loadModel, unloadModel, VERBOSITY } from '@qvac/sdk';
try {
    // Load the model
    const modelId = await loadModel({
        modelSrc: LLAMA_3_2_1B_INST_Q4_0,
        modelConfig: {
            device: 'gpu',
            ctx_size: 2048,
            verbosity: VERBOSITY.ERROR
        }
    });
    console.log('▸ Testing KV Cache functionality...\n');
    // First conversation with auto-keyed cache enabled
    console.log('▸ First conversation (building cache for the next turn):');
    const history1 = [{ role: 'user', content: 'What is the capital of France?' }];
    const result1 = completion({
        modelId,
        history: history1,
        stream: true,
        kvCache: true
    }); // kvCache = true
    for await (const token of result1.tokenStream) {
        process.stdout.write(token);
    }
    const final1 = await result1.final;
    const stats1 = final1.stats;
    console.log(`\n▸ First completion stats: ${JSON.stringify(stats1)}\n`);
    // Continue conversation (should reuse the completed first-turn cache).
    console.log('▸ Continuing conversation (reusing previous turn cache):');
    const history2 = [
        { role: 'user', content: 'What is the capital of France?' },
        {
            role: 'assistant',
            content: final1.cacheableAssistantContent ?? final1.contentText
        },
        { role: 'user', content: 'What about Germany?' }
    ];
    // Auto-keyed caching should:
    // 1. Find the cache saved after turn 1 under [user, assistant]
    // 2. Load that cache and process only the new "What about Germany?" user turn
    // 3. Save the updated cache and rename it to include the new assistant response
    const result2 = completion({
        modelId,
        history: history2,
        stream: true,
        kvCache: true
    }); // kvCache = true
    for await (const token of result2.tokenStream) {
        process.stdout.write(token);
    }
    const stats2 = await result2.stats;
    console.log(`\n▸ Second completion stats: ${JSON.stringify(stats2)}\n`);
    // Compare with non-cached version
    console.log('▸ Same conversation without cache:');
    const result3 = completion({
        modelId,
        history: history2,
        stream: true,
        kvCache: false
    }); // kvCache = false
    for await (const token of result3.tokenStream) {
        process.stdout.write(token);
    }
    const stats3 = await result3.stats;
    console.log(`\n▸ Non-cached completion stats: ${JSON.stringify(stats3)}\n`);
    console.log('▸ KV Cache test completed!');
    await unloadModel({ modelId, clearStorage: false });
}
catch (error) {
    console.error('✖', error);
    process.exit(1);
}

Speculative decoding

Speculative decoding drafts several tokens per step and has the model verify the whole draft in one decode. The model checks every drafted token, so output keeps the distribution of plain decoding. Enable it at load time with modelConfig.spec_type:

  • "draft-mtp" — drafts with multi-token-prediction (MTP) layers: the model's own, for example QWEN3_5_0_8B_MTP_Q4_K_M (also 2B, 4B and 9B) or QWEN3_8_27B_MULTIMODAL_UD_Q4_K_XL, or an external MTP head given as modelConfig.draftModelSrc, for example QWEN3_5_0_8B_MTP_Q4_K_M for QWEN3_5_0_8B_MULTIMODAL_Q4_K_M, or a Gemma 4 assistant for its Gemma 4 model. The head, when given, otherwise the model, must have MTP layers, and a head must match the model (hidden width and vocabulary), or the load fails. A head is placed like the model, and model fit checks count its memory, except a Gemma 4 assistant's.
  • "draft-dflash" — drafts with a separate DFlash model, given as modelConfig.draftModelSrc, which it requires: for example QWEN3_5_4B_DFLASH_Q8_0 with QWEN3_5_4B_MULTIMODAL_Q4_K_M. The draft model is placed like the model, and model fit checks do not count its memory, so leave room for it.
  • "none" (default) — one token per step.

spec_draft_n_max (1–64, default 3) caps the draft length and is lowered at load, with a warning, to fit the batch. spec_draft_n_min (0–64, default 0) drops shorter drafts, and spec_draft_p_min (0–1, default 0) is the minimum draft-head probability for a token to extend the draft. These three require spec_type "draft-mtp" or "draft-dflash". Speculative decoding works with multimodal prompts, parallel and kvCache.

When it is on, stats.draftTokens counts the drafted tokens and stats.draftAcceptedTokens the ones the model kept. For batchCompletion() they are batch-level, on run.stats. The gain depends on the device: with several concurrent sequences, the larger verification batches can cost more than they save.

const modelId = await loadModel({
  modelSrc: QWEN3_5_4B_MULTIMODAL_Q4_K_M,
  modelType: "llm",
  modelConfig: {
    spec_type: "draft-dflash",
    draftModelSrc: QWEN3_5_4B_DFLASH_Q8_0,
  },
});

AMD NPU

On Linux x64 with an AMD XDNA2 NPU (Strix Point, Strix Halo or Krackan), load with modelConfig.device: "npu" to run the ops the NPU supports on it, with the CPU running the rest. The NPU needs XRT 2.25 or newer and read/write access to its /dev/accel node, and is never selected automatically.

  • Without a listed NPU, device: "npu" falls back to the GPU, then the CPU. A listed NPU that fails to open runs the load on the CPU.
  • The LLM layers leave the GPU free for another model. A vision model's projector runs on a GPU beside the NPU by default, and mmproj-use-gpu: false keeps it on the CPU.
  • stats.backendDevice reports "npu" when the NPU is selected.

Only a model with a 1024-wide hidden size, such as Qwen3.5-0.8B, decodes on the NPU, and only with the weight types its fused decode layer takes. Those depend on how the GGUF was quantized, not on its quantization name. Of the registry builds:

  • QWEN3_5_0_8B_MULTIMODAL_Q4_K_M runs.
  • QWEN3_5_0_8B_MULTIMODAL_Q6_K and QWEN3_5_0_8B_MULTIMODAL_Q8_0 fail at the first decode.
  • With QWEN3_5_2B_MULTIMODAL_Q4_K_M, QWEN3_5_4B_MULTIMODAL_Q4_K_M and QWEN3_5_9B_MULTIMODAL_Q4_K_M, only prefill runs on the NPU, and decode runs on the CPU.
const modelId = await loadModel({
  modelSrc: QWEN3_5_0_8B_MULTIMODAL_Q4_K_M,
  modelType: "llm",
  modelConfig: { device: "npu" },
});

Tip: all examples throughout this documentation are self-contained and runnable. For instructions on how to run them, see the JS/TS quickstart, the Python quickstart, or the Kotlin quickstart.

On this page

Ask anything about QVAC.