---
title: "Text generation"
canonical: https://docs.qvac.tether.io/sdk/v0.23/ai-capabilities/text-generation/
collection: "SDK"
package: "@qvac/sdk"
line: v0.23
current_line: false
---

# Text generation (/sdk/v0.23/ai-capabilities/text-generation)



## Overview

Text generation uses [`qvac-fabric-llm.cpp`](https://github.com/tetherto/qvac-fabric-llm.cpp) as inference engine. Load any supported model using `modelType: "llm"`. Then, provide array `history` as input where each element is an object with properties:

* `role: string`; can be either `"user"` or `"assistant"`
* `content: string`

`role: "user"` indicates that `content` is a previous prompt.  `role: "assistant"` indicates that `content` is a previous inference (LLM output).

Output is generated based on the full sequence of messages provided in `history`.

## Functions

Use the following sequence of function calls:

1. [`loadModel()`](/sdk/v0.23/reference/api#loadmodel)
2. [`completion()`](/sdk/v0.23/reference/api#completion)
3. [`unloadModel()`](/sdk/v0.23/reference/api#unloadmodel)

For how to use each function, see [SDK — API reference](/sdk/v0.23/reference/api/).

## Models

You can load any [`llama.cpp`](https://github.com/ggml-org/llama.cpp)-compatible text-generation/chat model. Model file format: `*.gguf`.

* If the model is sharded across multiple files (a multi-file bundle), see [Sharded models](/sdk/v0.23/models/sharded-models).
* To adapt a model to domain-specific tasks, see [Fine-tuning](/sdk/v0.23/ai-capabilities/fine-tuning).
* For models available as constants, see [SDK — Models](/sdk/v0.23/#models).

## Features

* Event stream: `completion()` exposes a single ordered `events` async iterable plus an aggregated `final` promise. Events are discriminated by `type` — `contentDelta`, `thinkingDelta`, `toolCall`, `toolError`, `completionStats`, `completionDone`, `rawDelta`. The terminal `completionDone` event carries a `stopReason` (e.g. `"eos"`, `"length"`, `"cancelled"`).
* Thinking content: models that emit `<think>` blocks surface them as dedicated `thinkingDelta` events (enable with `captureThinking: true`), so consumers don't have to parse tags from raw text.
* Tool calls: the model emits structured tool calls as `toolCall` events ordered alongside content and thinking in the same stream.
* Tool choice: `generationParams.tool_choice` takes `"auto"` (default), `"required"`, `"none"` or the name of a declared tool. On llama.cpp-backed models the sampler is constrained by the chat template's tool-call grammar, so `"required"` guarantees a call and a name restricts the call to that tool; a request that cannot honour the choice fails instead of answering in prose.
* Tool parameters: a tool's `parameters` is a JSON Schema object and is passed to the model as declared, including nested `properties`, `items`, nullable types (`["string", "null"]`) and `anyOf`. Zod `parameters` are converted with `z.toJSONSchema`, so nested Zod objects and arrays keep their structure.
* Tool errors: a tool-call region the model emitted that fails to parse or validate surfaces as a `toolError` event and in `final.toolErrors`, so an empty `toolCalls` can be told apart from a malformed call.
* Past tool calls in history: an `assistant` message carries the calls it made in `toolCalls` (`id`, `name`, `arguments`), and a `tool` message names the call it answers with `toolCallId` and `toolName`. llama.cpp-backed models render them with their own chat template instead of reading them as text in `content`.
* Parallel tool calls: one response may carry several tool calls on llama.cpp-backed models. `generationParams.parallel_tool_calls` defaults to `true`; set it to `false` to get at most one.
* MCP: plug MCP servers into `completion()` so the model can use external tools (e.g., web search) via the same tool-call mechanism.
* Raw output: with `emitRawDeltas: true`, every raw model token is also emitted as a `rawDelta` event in parallel to the structured events — useful for debugging or full-fidelity logging.
* KV cache: cache and reuse the model's key/value attention state to speed up follow-up turns in long conversations. The `kvCache` parameter sets the cache key — pass a string to manage a session manually, or `true` to let the SDK auto-generate one.
* Multimodal: attach images (and other media) to prompts so the model reasons over text and images together in the same conversation — see [Multimodal](/sdk/v0.23/ai-capabilities/multimodal).
* Multi-job continuous batching: load with `modelConfig.parallel >= 2` (default `1`) to run several `completion()` calls in parallel on one loaded model. Up to `parallel` prompts finish in roughly the time of the slowest one (extras queue), and each call has its own `requestId` so you can cancel one without touching the others — see [Concurrent completions](#concurrent-completions).
* Batch processing: run multiple prompts through a single loaded model in one call with `batchCompletion()` — optimizes resources and reduces total run time. See [Batch processing](/sdk/v0.23/ai-capabilities/batch-processing).
* Speculative decoding: load with `modelConfig.spec_type` to draft several tokens per step and verify them in one decode, with multi-token-prediction (MTP) layers, the model's own or an external head, or with a separate DFlash draft model. Output keeps the distribution of plain decoding. See [Speculative decoding](#speculative-decoding).

## Examples

### Usage

The canonical way to consume `completion()` is the `events` async iterable plus the aggregated `final` promise. The following script shows how to handle each event type and read the aggregated result:

<Tabs>
  <Tab value="js" label="JavaScript" default>
    <WrapCode>
      ```js file=<rootDir>/packages/sdk/dist/examples/completion-events.js title="completion-events.js" lineNumbers
      /**
       * Event-driven completion — demonstrates the unified `CompletionEvent` stream.
       *
       * `completion()` returns a `CompletionRun` with two primary surfaces:
       *
       *  - `events`  — an `AsyncIterable<CompletionEvent>` of ordered, typed events
       *                (`contentDelta`, `thinkingDelta`, `toolCall`, `toolError`,
       *                 `completionStats`, `completionDone`, `rawDelta`).
       *  - `final`   — a `Promise<CompletionFinal>` that resolves once the stream
       *                ends, providing aggregated `contentText`, `thinkingText`,
       *                `toolCalls`, `stats`, and `raw.fullText`.
       *
       * Set `captureThinking: true` to attempt best-effort `<think>` block parsing
       * into dedicated `thinkingDelta` events. `final.raw.fullText` keeps the exact
       * model output.
       */
      import { completion, loadModel, unloadModel, QWEN3_600M_INST_Q4 } from '@qvac/sdk';
      try {
          const modelId = await loadModel({
              modelSrc: QWEN3_600M_INST_Q4,
              modelConfig: { ctx_size: 4096 },
              onProgress: (p) => {
                  const mb = (n) => (n / 1e6).toFixed(1);
                  const line = `▸ Downloading ${p.percentage.toFixed(0)}% (${mb(p.downloaded)}/${mb(p.total)} MB)`;
                  process.stderr.write(process.stderr.isTTY ? `\r${line}` : `${line}\n`);
                  if (p.percentage >= 100)
                      process.stderr.write('\n');
              }
          });
          console.log(`▸ Model loaded: ${modelId}`);
          const result = completion({
              modelId,
              history: [{ role: 'user', content: 'Explain quantum computing in 2 sentences' }],
              stream: true,
              captureThinking: true
          });
          for await (const event of result.events) {
              handleEvent(event);
          }
          const final = await result.final;
          console.log('\n▸ Final result');
          console.log(`▸ Content: ${final.contentText}`);
          if (final.thinkingText) {
              console.log(`▸ Thinking: ${final.thinkingText}`);
          }
          if (final.stats) {
              console.log(`▸ ${final.stats.tokensPerSecond?.toFixed(1)} tok/s`);
          }
          if (final.toolCalls.length > 0) {
              console.log(`▸ Tool calls: ${final.toolCalls.map((c) => c.name).join(', ')}`);
          }
          if (final.stopReason) {
              console.log(`▸ Stop reason: ${final.stopReason}`);
          }
          console.log(`▸ Raw output length: ${final.raw.fullText.length} chars`);
          await unloadModel({ modelId, clearStorage: false });
      }
      catch (error) {
          console.error('✖', error);
          process.exit(1);
      }
      function handleEvent(event) {
          switch (event.type) {
              case 'contentDelta':
                  process.stdout.write(event.text);
                  break;
              case 'thinkingDelta':
                  process.stdout.write(`\x1b[2m${event.text}\x1b[0m`);
                  break;
              case 'toolCall':
                  console.log(`\n▸ Tool: ${event.call.name}(${JSON.stringify(event.call.arguments)})`);
                  break;
              case 'toolError':
                  console.log(`\n✖ Tool error [${event.error.code}]: ${event.error.message}`);
                  break;
              case 'completionStats':
                  console.log(`\n▸ ${event.stats.tokensPerSecond?.toFixed(1)} tok/s`);
                  break;
              case 'completionDone':
                  if (event.stopReason === 'error' && 'error' in event) {
                      console.log(`\n✖ ${event.error.message}`);
                  }
                  break;
              case 'rawDelta':
                  break;
          }
      }
      ```
    </WrapCode>
  </Tab>

  <Tab value="ts" label="TypeScript">
    <WrapCode>
      ```ts file=<rootDir>/packages/sdk/examples/completion-events.ts title="completion-events.ts" lineNumbers
      /**
       * Event-driven completion — demonstrates the unified `CompletionEvent` stream.
       *
       * `completion()` returns a `CompletionRun` with two primary surfaces:
       *
       *  - `events`  — an `AsyncIterable<CompletionEvent>` of ordered, typed events
       *                (`contentDelta`, `thinkingDelta`, `toolCall`, `toolError`,
       *                 `completionStats`, `completionDone`, `rawDelta`).
       *  - `final`   — a `Promise<CompletionFinal>` that resolves once the stream
       *                ends, providing aggregated `contentText`, `thinkingText`,
       *                `toolCalls`, `stats`, and `raw.fullText`.
       *
       * Set `captureThinking: true` to attempt best-effort `<think>` block parsing
       * into dedicated `thinkingDelta` events. `final.raw.fullText` keeps the exact
       * model output.
       */

      import {
        completion,
        loadModel,
        unloadModel,
        QWEN3_600M_INST_Q4,
        type CompletionEvent
      } from '@qvac/sdk'

      try {
        const modelId = await loadModel({
          modelSrc: QWEN3_600M_INST_Q4,
          modelConfig: { ctx_size: 4096 },
          onProgress: (p) => {
            const mb = (n: number) => (n / 1e6).toFixed(1)
            const line = `▸ Downloading ${p.percentage.toFixed(0)}% (${mb(p.downloaded)}/${mb(p.total)} MB)`
            process.stderr.write(process.stderr.isTTY ? `\r${line}` : `${line}\n`)
            if (p.percentage >= 100) process.stderr.write('\n')
          }
        })
        console.log(`▸ Model loaded: ${modelId}`)

        const result = completion({
          modelId,
          history: [{ role: 'user', content: 'Explain quantum computing in 2 sentences' }],
          stream: true,
          captureThinking: true
        })

        for await (const event of result.events) {
          handleEvent(event)
        }

        const final = await result.final

        console.log('\n▸ Final result')
        console.log(`▸ Content: ${final.contentText}`)
        if (final.thinkingText) {
          console.log(`▸ Thinking: ${final.thinkingText}`)
        }
        if (final.stats) {
          console.log(`▸ ${final.stats.tokensPerSecond?.toFixed(1)} tok/s`)
        }
        if (final.toolCalls.length > 0) {
          console.log(`▸ Tool calls: ${final.toolCalls.map((c) => c.name).join(', ')}`)
        }
        if (final.stopReason) {
          console.log(`▸ Stop reason: ${final.stopReason}`)
        }
        console.log(`▸ Raw output length: ${final.raw.fullText.length} chars`)

        await unloadModel({ modelId, clearStorage: false })
      } catch (error) {
        console.error('✖', error)
        process.exit(1)
      }

      function handleEvent(event: CompletionEvent) {
        switch (event.type) {
          case 'contentDelta':
            process.stdout.write(event.text)
            break
          case 'thinkingDelta':
            process.stdout.write(`\x1b[2m${event.text}\x1b[0m`)
            break
          case 'toolCall':
            console.log(`\n▸ Tool: ${event.call.name}(${JSON.stringify(event.call.arguments)})`)
            break
          case 'toolError':
            console.log(`\n✖ Tool error [${event.error.code}]: ${event.error.message}`)
            break
          case 'completionStats':
            console.log(`\n▸ ${event.stats.tokensPerSecond?.toFixed(1)} tok/s`)
            break
          case 'completionDone':
            if (event.stopReason === 'error' && 'error' in event) {
              console.log(`\n✖ ${event.error.message}`)
            }
            break
          case 'rawDelta':
            break
        }
      }
      ```
    </WrapCode>
  </Tab>

  <Tab value="python" label="Python">
    <WrapCode>
      ```python file=<rootDir>/packages/sdk-python/examples/completion_events.py title="completion_events.py" lineNumbers
      """Python port of packages/sdk/examples/completion-events.ts.

      The unified typed event stream. `completion()` returns a `CompletionRun` with:

        - `events` — an async iterator of ordered, typed events (`contentDelta`,
          `thinkingDelta`, `toolCall`, `toolError`, `completionStats`,
          `completionDone`, `rawDelta`).
        - `final` — an awaitable `CompletionFinal` with aggregated `content_text`,
          `thinking_text`, `tool_calls`, `stats`, and `raw_full_text`.

      Set `capture_thinking=True` for best-effort `<think>` parsing into
      `thinkingDelta` events.

      RUN: python examples/completion_events.py
      """

      from __future__ import annotations

      import asyncio
      import sys

      from tetherto.qvac_sdk import Client, completion, load_model, unload_model
      from tetherto.qvac_sdk.models import QWEN3_600M_INST_Q4

      DIM = "\x1b[2m"
      RESET = "\x1b[0m"


      def print_progress(p) -> None:
          """Print model download progress; pass as `on_progress=` to `load_model`."""
          line = (
              f"▸ Downloading {p.percentage:.0f}% "
              f"({p.downloaded / 1e6:.1f}/{p.total / 1e6:.1f} MB)"
          )
          print(line, end="\r" if sys.stderr.isatty() else "\n", file=sys.stderr)
          if p.percentage >= 100:
              print(file=sys.stderr)


      def handle_event(event) -> None:
          if event.type == "contentDelta":
              sys.stdout.write(event.text)
              sys.stdout.flush()
          elif event.type == "thinkingDelta":
              sys.stdout.write(f"{DIM}{event.text}{RESET}")
              sys.stdout.flush()
          elif event.type == "toolCall":
              print(f"\n▸ Tool: {event.call.name}({event.call.arguments})")
          elif event.type == "toolError":
              print(f"\n✖ Tool error [{event.error.code}]: {event.error.message}")
          elif event.type == "completionStats":
              tps = event.stats.tokens_per_second
              print(f"\n▸ {tps:.1f} tok/s" if tps is not None else "")
          elif event.type == "completionDone":
              if getattr(event, "stop_reason", None) == "error" and getattr(
                  event, "error", None
              ):
                  print(f"\n✖ {event.error.message}")


      async def main() -> int:
          async with Client() as client:
              t = client.transport
              try:
                  model_id = await load_model(
                      t,
                      model_src=QWEN3_600M_INST_Q4,
                      model_config={"ctx_size": 4096},
                      on_progress=print_progress,
                  )
                  print(f"▸ Model loaded: {model_id}")

                  run = completion(
                      t,
                      model_id=model_id,
                      history=[
                          {
                              "role": "user",
                              "content": "Explain quantum computing in 2 sentences",
                          }
                      ],
                      capture_thinking=True,
                  )

                  async for event in run.events:
                      handle_event(event)

                  final = await run.final

                  print("\n▸ Final result")
                  print(f"▸ Content: {final.content_text}")
                  if final.thinking_text:
                      print(f"▸ Thinking: {final.thinking_text}")
                  if final.stats and final.stats.tokens_per_second is not None:
                      print(f"▸ {final.stats.tokens_per_second:.1f} tok/s")
                  if final.tool_calls:
                      print(f"▸ Tool calls: {', '.join(c.name for c in final.tool_calls)}")
                  if final.stop_reason:
                      print(f"▸ Stop reason: {final.stop_reason}")
                  print(f"▸ Raw output length: {len(final.raw_full_text or '')} chars")

                  await unload_model(t, model_id)
              except Exception as error:
                  print(f"✖ {error}", file=sys.stderr)
                  return 1
          return 0


      if __name__ == "__main__":
          sys.exit(asyncio.run(main()))
      ```
    </WrapCode>
  </Tab>

  <Tab value="kotlin" label="Kotlin">
    <WrapCode>
      ```kotlin file=<rootDir>/packages/sdk-kotlin/examples/src/CompletionEvents.kt title="CompletionEvents.kt" lineNumbers
      // Kotlin port of packages/sdk/examples/completion-events.ts.
      //
      // The unified typed event stream. `client.completion.run()` returns a
      // `QvacCompletionRun` with:
      //
      //   - `events` — a `Flow` of ordered, typed `QvacCompletionEvent`s
      //     (`ContentDelta`, `ThinkingDelta`, `ToolCallEvent`, `ToolError`, `Stats`,
      //     `Done`, `RawDelta`).
      //   - `final` — a `Deferred<QvacCompletionFinal>` with the aggregated `text`,
      //     `thinking`, `toolCalls`, `stats`, and `rawFullText`.
      //
      // `QvacCompletionEvent` is a sealed interface, so the `when` below is checked
      // for exhaustiveness at compile time. Set `captureThinking = true` for
      // best-effort `<think>` parsing into `ThinkingDelta` events.
      //
      // RUN: ./gradlew -p examples run -Pexample=CompletionEvents

      import io.tether.qvac.sdk.QvacClient
      import io.tether.qvac.sdk.QvacCompletionEvent
      import io.tether.qvac.sdk.QvacCompletionOptions
      import io.tether.qvac.sdk.QvacMessage
      import io.tether.qvac.sdk.QvacProgressEvent
      import io.tether.qvac.sdk.completion
      import io.tether.qvac.sdk.generated.ModelConstant
      import io.tether.qvac.sdk.generated.Models
      import io.tether.qvac.sdk.generated.schema.LoadModelResponse
      import io.tether.qvac.sdk.generated.schema.ModelProgressResponse
      import io.tether.qvac.sdk.jvm.JvmBareKitTransport
      import io.tether.qvac.sdk.models
      import io.tether.qvac.sdk.run
      import kotlinx.coroutines.runBlocking
      import kotlinx.serialization.json.JsonObject
      import kotlinx.serialization.json.buildJsonObject
      import kotlinx.serialization.json.put

      private const val DIM = "\u001b[2m"
      private const val RESET = "\u001b[0m"

      fun main(): Unit = runBlocking {
          val client = QvacClient(JvmBareKitTransport.connect())
          try {
              val modelId = client.loadWithProgress(
                  Models.QWEN3_600M_INST_Q4,
                  modelConfig = buildJsonObject { put("ctx_size", 4096) },
              )
              println("▸ Model loaded: $modelId")

              val run = client.completion.run(
                  modelId = modelId,
                  history = listOf(QvacMessage.user("Explain quantum computing in 2 sentences")),
                  options = QvacCompletionOptions(captureThinking = true),
              )

              run.events.collect(::handleEvent)

              val final = run.final.await()

              println("\n▸ Final result")
              println("▸ Content: ${final.text}")
              if (!final.thinking.isNullOrEmpty()) println("▸ Thinking: ${final.thinking}")
              final.stats?.tokensPerSecond?.let { println("▸ %.1f tok/s".format(it)) }
              if (final.toolCalls.isNotEmpty()) {
                  println("▸ Tool calls: ${final.toolCalls.joinToString { it.name }}")
              }
              final.stopReason?.let { println("▸ Stop reason: $it") }
              println("▸ Raw output length: ${final.rawFullText.length} chars")

              client.models.unload(modelId, clearStorage = false)
          } finally {
              client.close()
          }
      }

      private fun handleEvent(event: QvacCompletionEvent) {
          when (event) {
              is QvacCompletionEvent.ContentDelta -> print(event.text)
              is QvacCompletionEvent.ThinkingDelta -> print("$DIM${event.text}$RESET")
              is QvacCompletionEvent.ToolCallEvent -> println("\n▸ Tool: ${event.call.name}(${event.call.arguments})")
              is QvacCompletionEvent.ToolError -> println("\n✖ Tool error [${event.code}]: ${event.message}")
              is QvacCompletionEvent.Stats -> event.value.tokensPerSecond?.let { println("\n▸ %.1f tok/s".format(it)) }
              is QvacCompletionEvent.Done -> if (event.stopReason == "error" && event.error != null) {
                  println("\n✖ ${event.error}")
              }
              is QvacCompletionEvent.RawDelta, is QvacCompletionEvent.Unknown -> Unit
          }
      }

      /** Load [model], printing download progress, and return its model id. */
      private suspend fun QvacClient.loadWithProgress(model: ModelConstant, modelConfig: JsonObject? = null): String {
          var result: LoadModelResponse? = null
          models.loadWithProgress(model.src, model.engine, model.name, modelConfig = modelConfig).collect { event ->
              when (event) {
                  is QvacProgressEvent.Progress -> printProgress(event.value)
                  is QvacProgressEvent.Result -> result = event.value
              }
          }
          val loaded = checkNotNull(result) { "loadModel ended without a result" }
          return loaded.modelId ?: error(loaded.error ?: "loadModel failed")
      }

      private fun printProgress(p: ModelProgressResponse) {
          val line = "▸ Downloading %.0f%% (%.1f/%.1f MB)".format(p.percentage, p.downloaded / 1e6, p.total / 1e6)
          if (System.console() != null) System.err.print("\r$line") else System.err.println(line)
          if (p.percentage >= 100) System.err.println()
      }
      ```
    </WrapCode>
  </Tab>
</Tabs>

<Callout type="info">
  The examples below (`Tool call`, `MCP`, `KV cache`) still consume `result.tokenStream` and `result.toolCallStream`, which are convenience wrappers around the canonical `events` / `final` stream shown above. Both APIs are supported; new code should prefer `events` / `final`.
</Callout>

### Concurrent completions

Use this when you have more than one prompt in flight against the same loaded model at the same time — for example, serving several users, or fanning out related sub-tasks — and want them to finish together instead of one after another.

Load the model with `modelConfig.parallel >= 2`, then call `completion()` several times without awaiting between calls. The SDK runs up to `parallel` of them together on the model, so **up to `parallel` prompts finish in roughly the time of the slowest one** instead of the sum of all of them; beyond that they queue. Each call has its own `requestId`; `cancel({ requestId })` stops just that call and its peers keep going — useful for a per-request stop button, rather than killing the whole model.

The default is `1` (one request at a time), so you have to set `parallel` explicitly to enable batching. Also raise `ctx_size` with it: the model's context window is split evenly across the `parallel` slots, so `parallel: 4` on a `ctx_size: 4096` model gives each request only \~1024 tokens. Size `ctx_size` for `parallel × per-request context`.

`completion()` and `batchCompletion()` share the model's `parallel` slots. Each call is one request — a big `batchCompletion({ prompts: N })` never blocks more than one concurrent `completion()` from starting — but every prompt in the batch takes one decode slot, so a large batch fills all `parallel` slots itself and any concurrent `completion()` you launch will only start decoding once the batch has room. If you already have every prompt ready up front, [`batchCompletion()`](/sdk/v0.23/ai-capabilities/batch-processing) is often simpler; use concurrent `completion()` when prompts arrive over time or you want independent streams and cancellation per call.

The following script loads a `parallel: 4` model, fires four completions at once, and then cancels one of two long runs to show its peer keeps decoding:

<Tabs>
  <Tab value="js" label="JavaScript" default>
    <WrapCode>
      ```js file=<rootDir>/packages/sdk/dist/examples/multi-job-completion.js title="multi-job-completion.js" lineNumbers
      /**
       * Multi-job continuous batching — concurrent completions on one loaded LLM.
       *
       * Unlike `batchCompletion()` (which bundles many prompts into ONE call), this
       * shows independent `completion()` calls fired concurrently against the SAME
       * loaded model. When the model is loaded with `modelConfig.parallel >= 2`, the
       * SDK admits up to `parallel` completions at once and the llama.cpp addon
       * decodes their sequences together — so a burst of prompts finishes in roughly
       * the time of the slowest one, not the sum of all of them.
       *
       * Each `completion()` returns an independent `CompletionRun`:
       *  - `events`     — an `AsyncIterable` of streamed events for that call.
       *  - `final`      — a `Promise<CompletionFinal>` with THIS call's own
       *                   `contentText` and `stats` (its own tokens/sec, TTFT).
       *                   `stats.avgConcurrentSeq` stays model-level (how busy the
       *                   shared backend was).
       *  - `requestId`  — cancel just this call with `cancel({ requestId })`; peers
       *                   keep decoding.
       *
       * Completions and batches share the model's `parallel` admission cap: with
       * `parallel` in flight, admitting one more waits FIFO until a slot frees.
       *
       * Run from packages/sdk:
       *   npx tsx examples/multi-job-completion.ts
       */
      import { completion, cancel, loadModel, unloadModel, InferenceCancelledError, LLAMA_3_2_1B_INST_Q4_0 } from '@qvac/sdk';
      const PROMPTS = [
          { id: 'cherry', ask: 'Reply with only the word CHERRY.' },
          { id: 'banana', ask: 'Reply with only the word BANANA.' },
          { id: 'grape', ask: 'Reply with only the word GRAPE.' },
          { id: 'lemon', ask: 'Reply with only the word LEMON.' }
      ];
      try {
          // `parallel: 4` opens the concurrent decode slots continuous batching needs.
          const modelId = await loadModel({
              modelSrc: LLAMA_3_2_1B_INST_Q4_0,
              modelType: 'llm',
              modelConfig: { ctx_size: 4096, parallel: 4 },
              onProgress: (p) => {
                  const mb = (n) => (n / 1e6).toFixed(1);
                  const line = `▸ Downloading ${p.percentage.toFixed(0)}% (${mb(p.downloaded)}/${mb(p.total)} MB)`;
                  process.stderr.write(process.stderr.isTTY ? `\r${line}` : `${line}\n`);
                  if (p.percentage >= 100)
                      process.stderr.write('\n');
              }
          });
          console.log(`▸ Model loaded: ${modelId}`);
          // ---- Concurrent completions. Fire every call before awaiting any, so all
          // four are admitted and decode together (not one-after-another). ----
          console.log('\n▸ Firing 4 completions concurrently:');
          const started = Date.now();
          const runs = PROMPTS.map(({ ask }) => completion({
              modelId,
              history: [{ role: 'user', content: ask }],
              generationParams: { temp: 0, seed: 42, predict: 16 },
              stream: true
          }));
          const outputs = await Promise.all(runs.map(async (run, i) => {
              const final = await run.final;
              return {
                  id: PROMPTS[i].id,
                  text: final.contentText.replace(/\s+/g, ' ').trim(),
                  stats: final.stats
              };
          }));
          const elapsed = Date.now() - started;
          for (const { id, text, stats } of outputs) {
              const tps = stats?.tokensPerSecond?.toFixed(1) ?? 'n/a';
              console.log(`  ▸ ${id}: "${text}"  (own ${tps} tok/s)`);
          }
          // Report the engine's own measure of overlap rather than inferring it from
          // wall-clock: avgConcurrentSeq > 1 means the addon decoded sequences together.
          const peakConcurrent = Math.max(...outputs.map((o) => o.stats?.avgConcurrentSeq ?? 0));
          console.log(`\n▸ All 4 finished in ${elapsed} ms; engine avg concurrent sequences: ${peakConcurrent.toFixed(1)} (> 1 = decoded together).`);
          // ---- Per-request cancel isolation. Fire two long runs; cancel only the
          // first by its requestId. The second must keep decoding past the cancel. ----
          console.log('\n▸ Per-request cancel: cancelling one of two in-flight completions:');
          const doomed = completion({
              modelId,
              history: [{ role: 'user', content: 'Write a long story about an otter.' }],
              generationParams: { temp: 0, seed: 1, predict: 256 },
              stream: true
          });
          const survivor = completion({
              modelId,
              history: [{ role: 'user', content: 'Write a long story about a melon.' }],
              generationParams: { temp: 0, seed: 42, predict: 256 },
              stream: true
          });
          // Consume the survivor's stream directly so we can count tokens produced AFTER
          // the cancel is acknowledged — a short survivor could otherwise finish before
          // the cancel lands and pass even under a model-wide cancel.
          const survivorTokens = survivor.events[Symbol.asyncIterator]();
          // Wait until BOTH runs are actually streaming before cancelling, so the cancel
          // hits a genuinely in-flight peer, not a pre-admission drop.
          await Promise.all([doomed.events[Symbol.asyncIterator]().next(), survivorTokens.next()]);
          await cancel({ requestId: doomed.requestId });
          let doomedCancelled = false;
          try {
              await doomed.final;
          }
          catch (err) {
              doomedCancelled = err instanceof InferenceCancelledError;
          }
          // Drain what the survivor emits from here on: a per-request cancel leaves it
          // decoding to completion, a model-wide cancel would have stopped it.
          let tokensAfterCancel = 0;
          for (let next = await survivorTokens.next(); !next.done; next = await survivorTokens.next()) {
              tokensAfterCancel++;
          }
          const survivorText = (await survivor.final).contentText.replace(/\s+/g, ' ').trim();
          console.log(`  ▸ cancelled run -> ${doomedCancelled ? 'cancelled' : 'NOT cancelled'}`);
          console.log(`  ▸ peer kept decoding -> +${tokensAfterCancel} tokens after the cancel ack`);
          await unloadModel({ modelId, clearStorage: false });
          // A zero exit must prove concurrent scheduling, typed cancellation, and survivor
          // isolation: the first burst reported multiple resident sequences, the doomed
          // run cancelled, and its peer produced tokens AFTER the cancel landed.
          if (peakConcurrent <= 1 || !doomedCancelled || tokensAfterCancel === 0) {
              console.error(`✖ multi-job behavior not demonstrated (peakConcurrent=${peakConcurrent.toFixed(2)}, doomedCancelled=${doomedCancelled}, tokensAfterCancel=${tokensAfterCancel}, survivorText="${survivorText}")`);
              process.exit(1);
          }
          process.exit(0);
      }
      catch (error) {
          console.error('✖', error);
          process.exit(1);
      }
      ```
    </WrapCode>
  </Tab>

  <Tab value="ts" label="TypeScript">
    <WrapCode>
      ```ts file=<rootDir>/packages/sdk/examples/multi-job-completion.ts title="multi-job-completion.ts" lineNumbers
      /**
       * Multi-job continuous batching — concurrent completions on one loaded LLM.
       *
       * Unlike `batchCompletion()` (which bundles many prompts into ONE call), this
       * shows independent `completion()` calls fired concurrently against the SAME
       * loaded model. When the model is loaded with `modelConfig.parallel >= 2`, the
       * SDK admits up to `parallel` completions at once and the llama.cpp addon
       * decodes their sequences together — so a burst of prompts finishes in roughly
       * the time of the slowest one, not the sum of all of them.
       *
       * Each `completion()` returns an independent `CompletionRun`:
       *  - `events`     — an `AsyncIterable` of streamed events for that call.
       *  - `final`      — a `Promise<CompletionFinal>` with THIS call's own
       *                   `contentText` and `stats` (its own tokens/sec, TTFT).
       *                   `stats.avgConcurrentSeq` stays model-level (how busy the
       *                   shared backend was).
       *  - `requestId`  — cancel just this call with `cancel({ requestId })`; peers
       *                   keep decoding.
       *
       * Completions and batches share the model's `parallel` admission cap: with
       * `parallel` in flight, admitting one more waits FIFO until a slot frees.
       *
       * Run from packages/sdk:
       *   npx tsx examples/multi-job-completion.ts
       */

      import {
        completion,
        cancel,
        loadModel,
        unloadModel,
        InferenceCancelledError,
        LLAMA_3_2_1B_INST_Q4_0
      } from '@qvac/sdk'

      const PROMPTS = [
        { id: 'cherry', ask: 'Reply with only the word CHERRY.' },
        { id: 'banana', ask: 'Reply with only the word BANANA.' },
        { id: 'grape', ask: 'Reply with only the word GRAPE.' },
        { id: 'lemon', ask: 'Reply with only the word LEMON.' }
      ]

      try {
        // `parallel: 4` opens the concurrent decode slots continuous batching needs.
        const modelId = await loadModel({
          modelSrc: LLAMA_3_2_1B_INST_Q4_0,
          modelType: 'llm',
          modelConfig: { ctx_size: 4096, parallel: 4 },
          onProgress: (p) => {
            const mb = (n: number) => (n / 1e6).toFixed(1)
            const line = `▸ Downloading ${p.percentage.toFixed(0)}% (${mb(p.downloaded)}/${mb(p.total)} MB)`
            process.stderr.write(process.stderr.isTTY ? `\r${line}` : `${line}\n`)
            if (p.percentage >= 100) process.stderr.write('\n')
          }
        })
        console.log(`▸ Model loaded: ${modelId}`)

        // ---- Concurrent completions. Fire every call before awaiting any, so all
        // four are admitted and decode together (not one-after-another). ----
        console.log('\n▸ Firing 4 completions concurrently:')
        const started = Date.now()
        const runs = PROMPTS.map(({ ask }) =>
          completion({
            modelId,
            history: [{ role: 'user', content: ask }],
            generationParams: { temp: 0, seed: 42, predict: 16 },
            stream: true
          })
        )

        const outputs = await Promise.all(
          runs.map(async (run, i) => {
            const final = await run.final
            return {
              id: PROMPTS[i]!.id,
              text: final.contentText.replace(/\s+/g, ' ').trim(),
              stats: final.stats
            }
          })
        )
        const elapsed = Date.now() - started

        for (const { id, text, stats } of outputs) {
          const tps = stats?.tokensPerSecond?.toFixed(1) ?? 'n/a'
          console.log(`  ▸ ${id}: "${text}"  (own ${tps} tok/s)`)
        }
        // Report the engine's own measure of overlap rather than inferring it from
        // wall-clock: avgConcurrentSeq > 1 means the addon decoded sequences together.
        const peakConcurrent = Math.max(...outputs.map((o) => o.stats?.avgConcurrentSeq ?? 0))
        console.log(
          `\n▸ All 4 finished in ${elapsed} ms; engine avg concurrent sequences: ${peakConcurrent.toFixed(1)} (> 1 = decoded together).`
        )

        // ---- Per-request cancel isolation. Fire two long runs; cancel only the
        // first by its requestId. The second must keep decoding past the cancel. ----
        console.log('\n▸ Per-request cancel: cancelling one of two in-flight completions:')
        const doomed = completion({
          modelId,
          history: [{ role: 'user', content: 'Write a long story about an otter.' }],
          generationParams: { temp: 0, seed: 1, predict: 256 },
          stream: true
        })
        const survivor = completion({
          modelId,
          history: [{ role: 'user', content: 'Write a long story about a melon.' }],
          generationParams: { temp: 0, seed: 42, predict: 256 },
          stream: true
        })

        // Consume the survivor's stream directly so we can count tokens produced AFTER
        // the cancel is acknowledged — a short survivor could otherwise finish before
        // the cancel lands and pass even under a model-wide cancel.
        const survivorTokens = survivor.events[Symbol.asyncIterator]()

        // Wait until BOTH runs are actually streaming before cancelling, so the cancel
        // hits a genuinely in-flight peer, not a pre-admission drop.
        await Promise.all([doomed.events[Symbol.asyncIterator]().next(), survivorTokens.next()])

        await cancel({ requestId: doomed.requestId })

        let doomedCancelled = false
        try {
          await doomed.final
        } catch (err) {
          doomedCancelled = err instanceof InferenceCancelledError
        }

        // Drain what the survivor emits from here on: a per-request cancel leaves it
        // decoding to completion, a model-wide cancel would have stopped it.
        let tokensAfterCancel = 0
        for (let next = await survivorTokens.next(); !next.done; next = await survivorTokens.next()) {
          tokensAfterCancel++
        }
        const survivorText = (await survivor.final).contentText.replace(/\s+/g, ' ').trim()
        console.log(`  ▸ cancelled run -> ${doomedCancelled ? 'cancelled' : 'NOT cancelled'}`)
        console.log(`  ▸ peer kept decoding -> +${tokensAfterCancel} tokens after the cancel ack`)

        await unloadModel({ modelId, clearStorage: false })

        // A zero exit must prove concurrent scheduling, typed cancellation, and survivor
        // isolation: the first burst reported multiple resident sequences, the doomed
        // run cancelled, and its peer produced tokens AFTER the cancel landed.
        if (peakConcurrent <= 1 || !doomedCancelled || tokensAfterCancel === 0) {
          console.error(
            `✖ multi-job behavior not demonstrated (peakConcurrent=${peakConcurrent.toFixed(2)}, doomedCancelled=${doomedCancelled}, tokensAfterCancel=${tokensAfterCancel}, survivorText="${survivorText}")`
          )
          process.exit(1)
        }
        process.exit(0)
      } catch (error) {
        console.error('✖', error)
        process.exit(1)
      }
      ```
    </WrapCode>
  </Tab>

  <Tab value="python" label="Python">
    <WrapCode>
      ```python file=<rootDir>/packages/sdk-python/examples/multi_job_completion.py title="multi_job_completion.py" lineNumbers
      """Python port of packages/sdk/examples/multi-job-completion.ts.

      Multi-job continuous batching — concurrent completions on one loaded LLM.

      Unlike `batch_completion_stream` (which bundles many prompts into ONE call),
      this shows independent `completion()` calls fired concurrently against the SAME
      loaded model. When the model is loaded with `model_config["parallel"] >= 2`, the
      SDK admits up to `parallel` completions at once and the llama.cpp addon decodes
      their sequences together — so a burst of prompts finishes in roughly the time
      of the slowest one, not the sum of all of them.

      Each `completion()` returns an independent `CompletionRun`:
       - `events`     — an async iterator of streamed events for that call.
       - `final`      — an awaitable `CompletionFinal` with THIS call's own
                        `content_text` and `stats` (its own tokens/sec, TTFT).
                        `stats.avg_concurrent_seq` stays model-level (how busy the
                        shared backend was).
       - `request_id` — cancel just this call with `cancel(request_id=...)`; peers
                        keep decoding.

      Completions and batches share the model's `parallel` admission cap: with
      `parallel` in flight, admitting one more waits FIFO until a slot frees.

      RUN: python examples/multi_job_completion.py
      """

      from __future__ import annotations

      import asyncio
      import sys
      import time

      from tetherto.qvac_sdk import (
          Client,
          InferenceCancelledError,
          cancel,
          completion,
          load_model,
          unload_model,
      )
      from tetherto.qvac_sdk.models import LLAMA_3_2_1B_INST_Q4_0

      PROMPTS = [
          ("cherry", "Reply with only the word CHERRY."),
          ("banana", "Reply with only the word BANANA."),
          ("grape", "Reply with only the word GRAPE."),
          ("lemon", "Reply with only the word LEMON."),
      ]


      def print_progress(p) -> None:
          """Print model download progress; pass as `on_progress=` to `load_model`."""
          line = (
              f"▸ Downloading {p.percentage:.0f}% "
              f"({p.downloaded / 1e6:.1f}/{p.total / 1e6:.1f} MB)"
          )
          print(line, end="\r" if sys.stderr.isatty() else "\n", file=sys.stderr)
          if p.percentage >= 100:
              print(file=sys.stderr)


      async def main() -> int:
          async with Client() as client:
              t = client.transport
              try:
                  # `parallel: 4` opens the concurrent decode slots continuous
                  # batching needs.
                  model_id = await load_model(
                      t,
                      model_src=LLAMA_3_2_1B_INST_Q4_0,
                      model_config={"ctx_size": 4096, "parallel": 4},
                      on_progress=print_progress,
                  )
                  print(f"▸ Model loaded: {model_id}")

                  # ---- Concurrent completions. Fire every call before awaiting any,
                  # so all four are admitted and decode together (not
                  # one-after-another). ----
                  print("\n▸ Firing 4 completions concurrently:")
                  started = time.monotonic()
                  runs = [
                      completion(
                          t,
                          model_id=model_id,
                          history=[{"role": "user", "content": ask}],
                          generation_params={"temp": 0, "seed": 42, "predict": 16},
                      )
                      for _name, ask in PROMPTS
                  ]

                  finals = await asyncio.gather(*(run.final for run in runs))
                  elapsed_ms = (time.monotonic() - started) * 1000

                  outputs = [
                      (name, " ".join(final.content_text.split()), final.stats)
                      for (name, _ask), final in zip(PROMPTS, finals)
                  ]

                  for name, text, stats in outputs:
                      tokens_per_second = getattr(stats, "tokens_per_second", None)
                      tps = (
                          f"{tokens_per_second:.1f}"
                          if tokens_per_second is not None
                          else "n/a"
                      )
                      print(f'  ▸ {name}: "{text}"  (own {tps} tok/s)')

                  # Report the engine's own measure of overlap rather than inferring it
                  # from wall-clock: avg_concurrent_seq > 1 means the addon decoded
                  # sequences together.
                  peak_concurrent = max(
                      getattr(stats, "avg_concurrent_seq", None) or 0
                      for _name, _text, stats in outputs
                  )
                  print(
                      f"\n▸ All 4 finished in {elapsed_ms:.0f} ms; engine avg concurrent "
                      f"sequences: {peak_concurrent:.1f} (> 1 = decoded together)."
                  )

                  # ---- Per-request cancel isolation. Fire two long runs; cancel only
                  # the first by its request_id. The second must keep decoding past
                  # the cancel. ----
                  print(
                      "\n▸ Per-request cancel: cancelling one of two in-flight completions:"
                  )
                  doomed = completion(
                      t,
                      model_id=model_id,
                      history=[
                          {"role": "user", "content": "Write a long story about an otter."}
                      ],
                      generation_params={"temp": 0, "seed": 1, "predict": 256},
                  )
                  survivor = completion(
                      t,
                      model_id=model_id,
                      history=[
                          {"role": "user", "content": "Write a long story about a melon."}
                      ],
                      generation_params={"temp": 0, "seed": 42, "predict": 256},
                  )

                  # Bind each stream once: `run.events` returns a NEW iterator per
                  # access and every one of them drains the same queue, so reading the
                  # property twice would split a run's events between two iterators.
                  # Holding the survivor's iterator also lets us count what it emits
                  # AFTER the cancel is acknowledged — a short survivor could
                  # otherwise finish before the cancel lands and pass even under a
                  # model-wide cancel.
                  doomed_events = doomed.events
                  survivor_events = survivor.events

                  # Wait until BOTH runs are actually streaming before cancelling, so
                  # the cancel hits a genuinely in-flight peer, not a pre-admission
                  # drop.
                  await asyncio.gather(anext(doomed_events), anext(survivor_events))

                  await cancel(t, request_id=doomed.request_id)

                  doomed_cancelled = False
                  try:
                      await doomed.final
                  except Exception as error:
                      doomed_cancelled = isinstance(error, InferenceCancelledError)

                  # Drain what the survivor emits from here on: a per-request cancel
                  # leaves it decoding to completion, a model-wide cancel would have
                  # stopped it.
                  tokens_after_cancel = 0
                  async for _event in survivor_events:
                      tokens_after_cancel += 1

                  survivor_final = await survivor.final
                  survivor_text = " ".join(survivor_final.content_text.split())
                  state = "cancelled" if doomed_cancelled else "NOT cancelled"
                  print(f"  ▸ cancelled run -> {state}")
                  print(
                      f"  ▸ peer kept decoding -> +{tokens_after_cancel} tokens after "
                      f"the cancel ack"
                  )

                  await unload_model(t, model_id)

                  # A zero exit must prove concurrent scheduling, typed cancellation,
                  # and survivor isolation: the first burst reported multiple resident
                  # sequences, the doomed run cancelled, and its peer produced tokens
                  # AFTER the cancel landed.
                  if (
                      peak_concurrent <= 1
                      or not doomed_cancelled
                      or tokens_after_cancel == 0
                  ):
                      print(
                          f"✖ multi-job behavior not demonstrated "
                          f"(peak_concurrent={peak_concurrent:.2f}, "
                          f"doomed_cancelled={doomed_cancelled}, "
                          f"tokens_after_cancel={tokens_after_cancel}, "
                          f'survivor_text="{survivor_text}")',
                          file=sys.stderr,
                      )
                      return 1
              except Exception as error:
                  print(f"✖ {error}", file=sys.stderr)
                  return 1
          return 0


      if __name__ == "__main__":
          sys.exit(asyncio.run(main()))
      ```
    </WrapCode>
  </Tab>

  <Tab value="kotlin" label="Kotlin">
    <WrapCode>
      ```kotlin file=<rootDir>/packages/sdk-kotlin/examples/src/MultiJobCompletion.kt title="MultiJobCompletion.kt" lineNumbers
      // Kotlin port of packages/sdk/examples/multi-job-completion.ts.
      //
      // Multi-job continuous batching — concurrent completions on one loaded LLM.
      //
      // Unlike `client.completion.batch()` (which bundles many prompts into ONE
      // call), this shows independent `completion.run()` calls fired concurrently
      // against the SAME loaded model. When the model is loaded with
      // `parallel >= 2`, the SDK admits up to `parallel` completions at once and the
      // llama.cpp addon decodes their sequences together — so a burst of prompts
      // finishes in roughly the time of the slowest one, not the sum of all of them.
      //
      // Each `completion.run()` returns an independent `QvacCompletionRun`:
      //
      //   - `events`    — a `Flow` of streamed events for that call.
      //   - `final`     — a `Deferred<QvacCompletionFinal>` with THIS call's own
      //                   `text` and `stats` (its own tokens/sec, TTFT).
      //                   `stats.averageConcurrentSequences` stays model-level (how
      //                   busy the shared backend was).
      //   - `cancel()`  — cancels just this call by its `requestId`; peers keep
      //                   decoding.
      //
      // `run()` is not a suspend function: it starts the request and returns at once,
      // so firing every call before awaiting any is a plain `map`, and `awaitAll()`
      // joins the `Deferred` results. A cancelled run's `final` fails with the typed
      // `QvacCompletionCancelledException`, which carries the partial result.
      //
      // Completions and batches share the model's `parallel` admission cap: with
      // `parallel` in flight, admitting one more waits FIFO until a slot frees.
      //
      // RUN: ./gradlew -p examples run -Pexample=MultiJobCompletion

      import io.tether.qvac.sdk.QvacClient
      import io.tether.qvac.sdk.QvacCompletionCancelledException
      import io.tether.qvac.sdk.QvacCompletionOptions
      import io.tether.qvac.sdk.QvacGenerationOptions
      import io.tether.qvac.sdk.QvacMessage
      import io.tether.qvac.sdk.QvacProgressEvent
      import io.tether.qvac.sdk.completion
      import io.tether.qvac.sdk.generated.ModelConstant
      import io.tether.qvac.sdk.generated.Models
      import io.tether.qvac.sdk.generated.schema.LoadModelResponse
      import io.tether.qvac.sdk.generated.schema.ModelProgressResponse
      import io.tether.qvac.sdk.jvm.JvmBareKitTransport
      import io.tether.qvac.sdk.models
      import io.tether.qvac.sdk.run
      import kotlin.time.TimeSource
      import kotlinx.coroutines.async
      import kotlinx.coroutines.awaitAll
      import kotlinx.coroutines.flow.first
      import kotlinx.coroutines.runBlocking
      import kotlinx.serialization.json.JsonObject
      import kotlinx.serialization.json.buildJsonObject
      import kotlinx.serialization.json.put

      private val PROMPTS = listOf(
          "cherry" to "Reply with only the word CHERRY.",
          "banana" to "Reply with only the word BANANA.",
          "grape" to "Reply with only the word GRAPE.",
          "lemon" to "Reply with only the word LEMON.",
      )

      fun main(): Unit = runBlocking {
          val client = QvacClient(JvmBareKitTransport.connect())
          try {
              // `parallel: 4` opens the concurrent decode slots continuous batching needs.
              val modelId = client.loadWithProgress(
                  Models.LLAMA_3_2_1B_INST_Q4_0,
                  modelConfig = buildJsonObject {
                      put("ctx_size", 4096)
                      put("parallel", 4)
                  },
              )
              println("▸ Model loaded: $modelId")

              // ---- Concurrent completions. Fire every call before awaiting any, so
              // all four are admitted and decode together (not one-after-another). ----
              println("\n▸ Firing 4 completions concurrently:")
              val started = TimeSource.Monotonic.markNow()
              val runs = PROMPTS.map { (_, ask) ->
                  client.completion.run(
                      modelId = modelId,
                      history = listOf(QvacMessage.user(ask)),
                      options = QvacCompletionOptions(
                          generation = QvacGenerationOptions(temperature = 0.0, seed = 42, predict = 16),
                      ),
                  )
              }

              val finals = runs.map { it.final }.awaitAll()
              val elapsedMs = started.elapsedNow().inWholeMilliseconds

              PROMPTS.zip(finals).forEach { (prompt, final) ->
                  val tps = final.stats?.tokensPerSecond?.let { "%.1f".format(it) } ?: "n/a"
                  println("  ▸ ${prompt.first}: \"${final.text.collapseWhitespace()}\"  (own $tps tok/s)")
              }

              // Report the engine's own measure of overlap rather than inferring it
              // from wall-clock: averageConcurrentSequences > 1 means the addon
              // decoded sequences together.
              val peakConcurrent = finals.maxOf { it.stats?.averageConcurrentSequences ?: 0.0 }
              println(
                  "\n▸ All 4 finished in $elapsedMs ms; engine avg concurrent " +
                      "sequences: %.1f (> 1 = decoded together).".format(peakConcurrent),
              )

              // ---- Per-request cancel isolation. Fire two long runs; cancel only the
              // first by its requestId. The second must keep decoding past the cancel. ----
              println("\n▸ Per-request cancel: cancelling one of two in-flight completions:")
              val doomed = client.completion.run(
                  modelId = modelId,
                  history = listOf(QvacMessage.user("Write a long story about an otter.")),
                  options = QvacCompletionOptions(
                      generation = QvacGenerationOptions(temperature = 0.0, seed = 1, predict = 256),
                  ),
              )
              val survivor = client.completion.run(
                  modelId = modelId,
                  history = listOf(QvacMessage.user("Write a long story about a melon.")),
                  options = QvacCompletionOptions(
                      generation = QvacGenerationOptions(temperature = 0.0, seed = 42, predict = 256),
                  ),
              )

              // Wait until BOTH runs are actually streaming before cancelling, so the
              // cancel hits a genuinely in-flight peer, not a pre-admission drop.
              // `events` is backed by one channel per run: every collector drains the
              // same queue, so `first()` takes one event and leaves the rest for the
              // later collection below.
              listOf(async { doomed.events.first() }, async { survivor.events.first() }).awaitAll()

              doomed.cancel()

              val doomedCancelled = try {
                  doomed.final.await()
                  false
              } catch (_: QvacCompletionCancelledException) {
                  true
              }

              // Drain what the survivor emits from here on: a per-request cancel
              // leaves it decoding to completion, a model-wide cancel would have
              // stopped it.
              var tokensAfterCancel = 0
              survivor.events.collect { tokensAfterCancel++ }

              val survivorText = survivor.final.await().text.collapseWhitespace()
              println("  ▸ cancelled run -> ${if (doomedCancelled) "cancelled" else "NOT cancelled"}")
              println("  ▸ peer kept decoding -> +$tokensAfterCancel tokens after the cancel ack")

              client.models.unload(modelId)

              // A clean exit must prove concurrent scheduling, typed cancellation, and
              // survivor isolation: the first burst reported multiple resident
              // sequences, the doomed run cancelled, and its peer produced tokens
              // AFTER the cancel landed.
              check(peakConcurrent > 1 && doomedCancelled && tokensAfterCancel > 0) {
                  "✖ multi-job behavior not demonstrated (peakConcurrent=%.2f, ".format(peakConcurrent) +
                      "doomedCancelled=$doomedCancelled, tokensAfterCancel=$tokensAfterCancel, " +
                      "survivorText=\"$survivorText\")"
              }
          } finally {
              client.close()
          }
      }

      private fun String.collapseWhitespace(): String = trim().split(Regex("\\s+")).joinToString(" ")

      /** Load [model], printing download progress, and return its model id. */
      private suspend fun QvacClient.loadWithProgress(model: ModelConstant, modelConfig: JsonObject? = null): String {
          var result: LoadModelResponse? = null
          models.loadWithProgress(model.src, model.engine, model.name, modelConfig = modelConfig).collect { event ->
              when (event) {
                  is QvacProgressEvent.Progress -> printProgress(event.value)
                  is QvacProgressEvent.Result -> result = event.value
              }
          }
          val loaded = checkNotNull(result) { "loadModel ended without a result" }
          return loaded.modelId ?: error(loaded.error ?: "loadModel failed")
      }

      private fun printProgress(p: ModelProgressResponse) {
          val line = "▸ Downloading %.0f%% (%.1f/%.1f MB)".format(p.percentage, p.downloaded / 1e6, p.total / 1e6)
          if (System.console() != null) System.err.print("\r$line") else System.err.println(line)
          if (p.percentage >= 100) System.err.println()
      }
      ```
    </WrapCode>
  </Tab>
</Tabs>

### Tool call

JS and TS examples here and in [Deferred tool loading](#deferred-tool-loading) import their tool schemas and a mock executor from a sibling `shared` module. Save it next to the example — as `shared.js` for JS, `shared.ts` for TS.

<Tabs>
  <Tab value="js" label="JavaScript" default>
    <WrapCode>
      ```js file=<rootDir>/packages/sdk/dist/examples/tools/shared.js title="shared.js" lineNumbers
      import { z } from 'zod';
      export const weatherSchema = z.object({
          city: z.string().describe('City name'),
          country: z.string().describe('Country code').optional()
      });
      export const horoscopeSchema = z.object({
          sign: z.string().describe('An astrological sign like Taurus or Aquarius')
      });
      export const toolSchemas = {
          get_weather: weatherSchema,
          get_horoscope: horoscopeSchema
      };
      export const tools = [
          {
              name: 'get_weather',
              description: 'Get current weather for a city',
              parameters: weatherSchema
          },
          {
              name: 'get_horoscope',
              description: "Get today's horoscope for an astrological sign",
              parameters: horoscopeSchema
          }
      ];
      export function mockExecute(name, args) {
          if (name === 'get_weather') {
              const a = args;
              return JSON.stringify({
                  city: a.city,
                  country: a.country ?? 'unknown',
                  temperature: '22°C',
                  condition: 'Partly cloudy'
              });
          }
          if (name === 'get_horoscope') {
              return JSON.stringify({
                  sign: args.sign,
                  horoscope: 'Today is a great day for new beginnings.'
              });
          }
          return JSON.stringify({ error: `Unknown tool: ${name}` });
      }
      ```
    </WrapCode>
  </Tab>

  <Tab value="ts" label="TypeScript">
    <WrapCode>
      ```ts file=<rootDir>/packages/sdk/examples/tools/shared.ts title="shared.ts" lineNumbers
      import { z } from 'zod'
      import type { ToolInput } from '@qvac/sdk'

      export const weatherSchema = z.object({
        city: z.string().describe('City name'),
        country: z.string().describe('Country code').optional()
      })

      export const horoscopeSchema = z.object({
        sign: z.string().describe('An astrological sign like Taurus or Aquarius')
      })

      export const toolSchemas = {
        get_weather: weatherSchema,
        get_horoscope: horoscopeSchema
      } as const

      export const tools: ToolInput[] = [
        {
          name: 'get_weather',
          description: 'Get current weather for a city',
          parameters: weatherSchema
        },
        {
          name: 'get_horoscope',
          description: "Get today's horoscope for an astrological sign",
          parameters: horoscopeSchema
        }
      ]

      export function mockExecute(name: string, args: Record<string, unknown>): string {
        if (name === 'get_weather') {
          const a = args as { city: string; country?: string }
          return JSON.stringify({
            city: a.city,
            country: a.country ?? 'unknown',
            temperature: '22°C',
            condition: 'Partly cloudy'
          })
        }
        if (name === 'get_horoscope') {
          return JSON.stringify({
            sign: (args as { sign: string }).sign,
            horoscope: 'Today is a great day for new beginnings.'
          })
        }
        return JSON.stringify({ error: `Unknown tool: ${name}` })
      }
      ```
    </WrapCode>
  </Tab>
</Tabs>

The following script shows how to provide tool definitions to `completion()`, consume the streaming output, and read the parsed tool calls.

<Tabs>
  <Tab value="js" label="JavaScript" default>
    <WrapCode>
      ```js file=<rootDir>/packages/sdk/dist/examples/tools/llamacpp-native-tools.js title="completion-tool-call.js" lineNumbers
      import { completion, loadModel, unloadModel, QWEN3_1_7B_INST_Q4 } from '@qvac/sdk';
      import { tools, toolSchemas, mockExecute } from './shared.js';
      try {
          const modelId = await loadModel({
              modelSrc: QWEN3_1_7B_INST_Q4,
              modelConfig: {
                  ctx_size: 4096,
                  tools: true
              },
              onProgress: (p) => {
                  const mb = (n) => (n / 1e6).toFixed(1);
                  const line = `▸ Downloading ${p.percentage.toFixed(0)}% (${mb(p.downloaded)}/${mb(p.total)} MB)`;
                  process.stderr.write(process.stderr.isTTY ? `\r${line}` : `${line}\n`);
                  if (p.percentage >= 100)
                      process.stderr.write('\n');
              }
          });
          console.log(`▸ Model loaded successfully! Model ID: ${modelId}`);
          const history = [
              {
                  role: 'system',
                  content: 'You are a helpful assistant that can use tools to get the weather and horoscope.'
              },
              {
                  role: 'user',
                  content: "What's the weather in Tokyo and my horoscope for Aquarius?"
              }
          ];
          console.log('\n▸ AI Response:');
          console.log('▸ (Streaming with tool definitions in prompt)\n');
          const result = completion({ modelId, history, stream: true, tools });
          const tokensTask = (async () => {
              for await (const token of result.tokenStream) {
                  process.stdout.write(token);
              }
          })();
          const toolsTask = (async () => {
              for await (const evt of result.toolCallStream) {
                  console.log(`\n\n▸ Tool Call Detected: ${evt.call.name}(${JSON.stringify(evt.call.arguments)})`);
                  console.log(`▸ ID: ${evt.call.id}`);
              }
          })();
          await Promise.all([tokensTask, toolsTask]);
          const stats = await result.stats;
          const toolCalls = await result.toolCalls;
          console.log('\n\n▸ Parsed Tool Calls:');
          if (toolCalls.length > 0) {
              for (const call of toolCalls) {
                  console.log(`▸ ${call.name}(${JSON.stringify(call.arguments)})`);
                  const schema = toolSchemas[call.name];
                  if (schema) {
                      const validated = schema.safeParse(call.arguments);
                      if (validated.success) {
                          console.log(`▸ Arguments validated with Zod`);
                      }
                      else {
                          console.log(`✖ Validation failed:`, validated.error);
                      }
                  }
              }
          }
          else {
              console.log('▸ No tool calls detected in response');
          }
          console.log('\n▸ Performance Stats:', stats);
          if (toolCalls.length > 0) {
              console.log('\n\n▸ Simulating Tool Execution...');
              const toolResults = toolCalls.map((call) => {
                  const result = mockExecute(call.name, call.arguments);
                  console.log(`▸ ${call.name}: ${result}`);
                  return { toolCallId: call.id, result };
              });
              history.push({
                  role: 'assistant',
                  content: await result.text
              });
              for (const toolResult of toolResults) {
                  history.push({
                      role: 'tool',
                      content: toolResult.result
                  });
              }
              console.log('\n\n▸ Follow-up Response with Tool Results:');
              const followUpResult = completion({
                  modelId,
                  history,
                  stream: true,
                  tools
              });
              for await (const token of followUpResult.tokenStream) {
                  process.stdout.write(token);
              }
              const followUpStats = await followUpResult.stats;
              console.log('\n\n▸ Follow-up Stats:', followUpStats);
          }
          console.log('\n\n▸ Completed!');
          await unloadModel({ modelId, clearStorage: false });
      }
      catch (error) {
          console.error('✖', error);
          process.exit(1);
      }
      ```
    </WrapCode>
  </Tab>

  <Tab value="ts" label="TypeScript">
    <WrapCode>
      ```ts file=<rootDir>/packages/sdk/examples/tools/llamacpp-native-tools.ts title="completion-tool-call.ts" lineNumbers
      import {
        completion,
        loadModel,
        unloadModel,
        type ToolCall,
        type CompletionStats,
        QWEN3_1_7B_INST_Q4
      } from '@qvac/sdk'
      import { tools, toolSchemas, mockExecute } from './shared.js'

      try {
        const modelId = await loadModel({
          modelSrc: QWEN3_1_7B_INST_Q4,
          modelConfig: {
            ctx_size: 4096,
            tools: true
          },
          onProgress: (p) => {
            const mb = (n: number) => (n / 1e6).toFixed(1)
            const line = `▸ Downloading ${p.percentage.toFixed(0)}% (${mb(p.downloaded)}/${mb(p.total)} MB)`
            process.stderr.write(process.stderr.isTTY ? `\r${line}` : `${line}\n`)
            if (p.percentage >= 100) process.stderr.write('\n')
          }
        })
        console.log(`▸ Model loaded successfully! Model ID: ${modelId}`)

        const history = [
          {
            role: 'system',
            content: 'You are a helpful assistant that can use tools to get the weather and horoscope.'
          },
          {
            role: 'user',
            content: "What's the weather in Tokyo and my horoscope for Aquarius?"
          }
        ]

        console.log('\n▸ AI Response:')
        console.log('▸ (Streaming with tool definitions in prompt)\n')

        const result = completion({ modelId, history, stream: true, tools })

        const tokensTask = (async () => {
          for await (const token of result.tokenStream) {
            process.stdout.write(token)
          }
        })()

        const toolsTask = (async () => {
          for await (const evt of result.toolCallStream) {
            console.log(
              `\n\n▸ Tool Call Detected: ${evt.call.name}(${JSON.stringify(evt.call.arguments)})`
            )
            console.log(`▸ ID: ${evt.call.id}`)
          }
        })()

        await Promise.all([tokensTask, toolsTask])

        const stats: CompletionStats | undefined = await result.stats
        const toolCalls: ToolCall[] = await result.toolCalls

        console.log('\n\n▸ Parsed Tool Calls:')
        if (toolCalls.length > 0) {
          for (const call of toolCalls) {
            console.log(`▸ ${call.name}(${JSON.stringify(call.arguments)})`)

            const schema = toolSchemas[call.name as keyof typeof toolSchemas]
            if (schema) {
              const validated = schema.safeParse(call.arguments)
              if (validated.success) {
                console.log(`▸ Arguments validated with Zod`)
              } else {
                console.log(`✖ Validation failed:`, validated.error)
              }
            }
          }
        } else {
          console.log('▸ No tool calls detected in response')
        }

        console.log('\n▸ Performance Stats:', stats)

        if (toolCalls.length > 0) {
          console.log('\n\n▸ Simulating Tool Execution...')

          const toolResults = toolCalls.map((call) => {
            const result = mockExecute(call.name, call.arguments)
            console.log(`▸ ${call.name}: ${result}`)
            return { toolCallId: call.id, result }
          })

          history.push({
            role: 'assistant',
            content: await result.text
          })

          for (const toolResult of toolResults) {
            history.push({
              role: 'tool',
              content: toolResult.result
            })
          }

          console.log('\n\n▸ Follow-up Response with Tool Results:')
          const followUpResult = completion({
            modelId,
            history,
            stream: true,
            tools
          })

          for await (const token of followUpResult.tokenStream) {
            process.stdout.write(token)
          }

          const followUpStats = await followUpResult.stats
          console.log('\n\n▸ Follow-up Stats:', followUpStats)
        }

        console.log('\n\n▸ Completed!')
        await unloadModel({ modelId, clearStorage: false })
      } catch (error) {
        console.error('✖', error)
        process.exit(1)
      }
      ```
    </WrapCode>
  </Tab>

  <Tab value="python" label="Python">
    <WrapCode>
      ```python file=<rootDir>/packages/sdk-python/examples/completion_tools.py title="completion_tools.py" lineNumbers
      """Python port of packages/sdk/examples/tools/llamacpp-native-tools.ts.

      The client-side tool loop: run a completion with tool definitions, read back
      the model's `tool_calls`, execute them yourself, append the results to the
      history, and run a follow-up completion. (For the WORKER to run this loop for
      you, see completion_orchestrate.py.)

      RUN: python examples/completion_tools.py
      """

      from __future__ import annotations

      import asyncio
      import json
      import sys

      from tetherto.qvac_sdk import Client, completion, load_model, unload_model
      from tetherto.qvac_sdk.models import QWEN3_1_7B_INST_Q4


      def print_progress(p) -> None:
          """Print model download progress; pass as `on_progress=` to `load_model`."""
          line = (
              f"▸ Downloading {p.percentage:.0f}% "
              f"({p.downloaded / 1e6:.1f}/{p.total / 1e6:.1f} MB)"
          )
          print(line, end="\r" if sys.stderr.isatty() else "\n", file=sys.stderr)
          if p.percentage >= 100:
              print(file=sys.stderr)


      TOOLS = [
          {
              "name": "get_weather",
              "description": "Get the current weather for a location",
              "parameters": {
                  "type": "object",
                  "properties": {"location": {"type": "string"}},
                  "required": ["location"],
              },
          },
          {
              "name": "get_horoscope",
              "description": "Get the horoscope for a star sign",
              "parameters": {
                  "type": "object",
                  "properties": {"sign": {"type": "string"}},
                  "required": ["sign"],
              },
          },
      ]


      def mock_execute(name, arguments) -> str:
          if name == "get_weather":
              return f"The weather in {arguments.get('location')} is 22°C and sunny."
          if name == "get_horoscope":
              return f"{arguments.get('sign')}: today favours careful validation."
          return "unknown tool"


      async def main() -> int:
          async with Client() as client:
              t = client.transport
              try:
                  model_id = await load_model(
                      t,
                      model_src=QWEN3_1_7B_INST_Q4,
                      model_config={"ctx_size": 4096, "tools": True},
                      on_progress=print_progress,
                  )
                  print(f"▸ Model loaded: {model_id}")

                  history = [
                      {
                          "role": "system",
                          "content": "You are a helpful assistant that can use tools for weather and horoscopes.",
                      },
                      {
                          "role": "user",
                          "content": "What's the weather in Tokyo and my horoscope for Aquarius?",
                      },
                  ]

                  print("\n▸ AI Response (streaming with tool definitions):\n")
                  run = completion(t, model_id=model_id, history=history, tools=TOOLS)
                  async for event in run.events:
                      if event.type == "contentDelta":
                          sys.stdout.write(event.text)
                          sys.stdout.flush()
                      elif event.type == "toolCall":
                          print(
                              f"\n\n▸ Tool Call: {event.call.name}({json.dumps(dict(event.call.arguments.root))})"
                          )

                  final = await run.final
                  print("\n\n▸ Parsed Tool Calls:")
                  if not final.tool_calls:
                      print("▸ No tool calls detected")
                  for call in final.tool_calls:
                      print(f"▸ {call.name}({json.dumps(call.arguments)})")

                  if final.tool_calls:
                      print("\n▸ Simulating Tool Execution...")
                      history.append({"role": "assistant", "content": final.content_text})
                      for call in final.tool_calls:
                          result = mock_execute(call.name, call.arguments)
                          print(f"▸ {call.name}: {result}")
                          history.append({"role": "tool", "content": result})

                      print("\n▸ Follow-up Response with Tool Results:\n")
                      follow_up = completion(
                          t, model_id=model_id, history=history, tools=TOOLS
                      )
                      async for event in follow_up.events:
                          if event.type == "contentDelta":
                              sys.stdout.write(event.text)
                              sys.stdout.flush()
                      print()

                  print("\n▸ Completed!")
                  await unload_model(t, model_id)
              except Exception as error:
                  print(f"✖ {error}", file=sys.stderr)
                  return 1
          return 0


      if __name__ == "__main__":
          sys.exit(asyncio.run(main()))
      ```
    </WrapCode>
  </Tab>

  <Tab value="kotlin" label="Kotlin">
    <WrapCode>
      ```kotlin file=<rootDir>/packages/sdk-kotlin/examples/src/CompletionTools.kt title="CompletionTools.kt" lineNumbers
      // Kotlin port of packages/sdk/examples/tools/llamacpp-native-tools.ts.
      //
      // The client-side tool loop: run a completion with tool definitions, read back
      // the model's tool calls, execute them yourself, append the results to the
      // history, and run a follow-up completion. (For the WORKER to run this loop for
      // you, see CompletionOrchestrate.kt.)
      //
      // Tools are typed `QvacTool` values: flat parameters use `QvacToolParameter`
      // with the `QvacToolParameterType` enum, and the SDK builds the JSON Schema
      // `parameters` object sent on the wire. Tool calls arrive as the
      // `QvacCompletionEvent.ToolCallEvent` arm of the sealed event type.
      //
      // RUN: ./gradlew -p examples run -Pexample=CompletionTools

      import io.tether.qvac.sdk.QvacClient
      import io.tether.qvac.sdk.QvacCompletionEvent
      import io.tether.qvac.sdk.QvacMessage
      import io.tether.qvac.sdk.QvacProgressEvent
      import io.tether.qvac.sdk.QvacTool
      import io.tether.qvac.sdk.QvacToolParameter
      import io.tether.qvac.sdk.QvacToolParameterType
      import io.tether.qvac.sdk.completion
      import io.tether.qvac.sdk.generated.ModelConstant
      import io.tether.qvac.sdk.generated.Models
      import io.tether.qvac.sdk.generated.schema.LoadModelResponse
      import io.tether.qvac.sdk.generated.schema.ModelProgressResponse
      import io.tether.qvac.sdk.jvm.JvmBareKitTransport
      import io.tether.qvac.sdk.models
      import io.tether.qvac.sdk.run
      import kotlinx.coroutines.runBlocking
      import kotlinx.serialization.json.JsonObject
      import kotlinx.serialization.json.buildJsonObject
      import kotlinx.serialization.json.jsonPrimitive
      import kotlinx.serialization.json.put

      private val TOOLS = listOf(
          QvacTool(
              name = "get_weather",
              description = "Get the current weather for a location",
              parameters = mapOf("location" to QvacToolParameter(QvacToolParameterType.STRING)),
              required = setOf("location"),
          ),
          QvacTool(
              name = "get_horoscope",
              description = "Get the horoscope for a star sign",
              parameters = mapOf("sign" to QvacToolParameter(QvacToolParameterType.STRING)),
              required = setOf("sign"),
          ),
      )

      private fun mockExecute(name: String, arguments: JsonObject): String = when (name) {
          "get_weather" -> "The weather in ${arguments.string("location")} is 22°C and sunny."
          "get_horoscope" -> "${arguments.string("sign")}: today favours careful validation."
          else -> "unknown tool"
      }

      private fun JsonObject.string(key: String): String? = get(key)?.jsonPrimitive?.content

      fun main(): Unit = runBlocking {
          val client = QvacClient(JvmBareKitTransport.connect())
          try {
              val modelId = client.loadWithProgress(
                  Models.QWEN3_1_7B_INST_Q4,
                  modelConfig = buildJsonObject {
                      put("ctx_size", 4096)
                      put("tools", true)
                  },
              )
              println("▸ Model loaded: $modelId")

              val history = mutableListOf(
                  QvacMessage.system("You are a helpful assistant that can use tools for weather and horoscopes."),
                  QvacMessage.user("What's the weather in Tokyo and my horoscope for Aquarius?"),
              )

              println("\n▸ AI Response (streaming with tool definitions):\n")
              val run = client.completion.run(modelId = modelId, history = history, tools = TOOLS)
              run.events.collect { event ->
                  when (event) {
                      is QvacCompletionEvent.ContentDelta -> print(event.text)
                      is QvacCompletionEvent.ToolCallEvent ->
                          println("\n\n▸ Tool Call: ${event.call.name}(${event.call.arguments})")
                      else -> Unit
                  }
              }

              val final = run.final.await()
              println("\n\n▸ Parsed Tool Calls:")
              if (final.toolCalls.isEmpty()) println("▸ No tool calls detected")
              for (call in final.toolCalls) println("▸ ${call.name}(${call.arguments})")

              if (final.toolCalls.isNotEmpty()) {
                  println("\n▸ Simulating Tool Execution...")
                  history += QvacMessage.assistant(final.text, toolCalls = final.toolCalls)
                  for (call in final.toolCalls) {
                      val result = mockExecute(call.name, call.arguments)
                      println("▸ ${call.name}: $result")
                      history += QvacMessage.tool(result, toolCallId = call.id, toolName = call.name)
                  }

                  println("\n▸ Follow-up Response with Tool Results:\n")
                  val followUp = client.completion.run(modelId = modelId, history = history, tools = TOOLS)
                  followUp.tokens.collect { print(it) }
                  println()
              }

              println("\n▸ Completed!")
              client.models.unload(modelId)
          } finally {
              client.close()
          }
      }

      /** Load [model], printing download progress, and return its model id. */
      private suspend fun QvacClient.loadWithProgress(model: ModelConstant, modelConfig: JsonObject? = null): String {
          var result: LoadModelResponse? = null
          models.loadWithProgress(model.src, model.engine, model.name, modelConfig = modelConfig).collect { event ->
              when (event) {
                  is QvacProgressEvent.Progress -> printProgress(event.value)
                  is QvacProgressEvent.Result -> result = event.value
              }
          }
          val loaded = checkNotNull(result) { "loadModel ended without a result" }
          return loaded.modelId ?: error(loaded.error ?: "loadModel failed")
      }

      private fun printProgress(p: ModelProgressResponse) {
          val line = "▸ Downloading %.0f%% (%.1f/%.1f MB)".format(p.percentage, p.downloaded / 1e6, p.total / 1e6)
          if (System.console() != null) System.err.print("\r$line") else System.err.println(line)
          if (p.percentage >= 100) System.err.println()
      }
      ```
    </WrapCode>
  </Tab>
</Tabs>

### Worker-orchestrated tool loop

Python and Kotlin also expose an advanced worker-orchestrated path, where the worker runs
the whole multi-turn tool loop and calls back into your local handlers. Use this
only when you need the worker to own the tool loop rather than managing each
turn in application code:

<Tabs>
  <Tab value="python" label="Python" default>
    <WrapCode>
      ```python file=<rootDir>/packages/sdk-python/examples/completion_orchestrate.py title="completion_orchestrate.py" lineNumbers
      """Worker-orchestrated tool loop — mirrors the SDK's server-side tool loop
      (beyond the client-side loop in completion_tools.py).

      `completion_orchestrate()` hands the multi-turn tool loop to the WORKER: when
      the model asks for a tool, the worker emits a callback frame, this client runs
      the registered `handler`, writes the result back, and generation continues —
      no manual history stitching. Every tool needs a `handler` (it runs code on
      this machine, so only a local worker may orchestrate).

      Advanced path: `completion_orchestrate` lives in `tetherto.qvac_sdk._completion` rather than
      the flat `tetherto.qvac_sdk` surface. The public tool API is `completion(tools=...)` (the
      client-side loop in completion_tools.py), matching the JS SDK; worker
      orchestration has no JS client wrapper yet.

      RUN: python examples/completion_orchestrate.py
      """

      from __future__ import annotations

      import asyncio
      import sys

      from tetherto.qvac_sdk import Client, load_model, unload_model
      from tetherto.qvac_sdk._completion import completion_orchestrate
      from tetherto.qvac_sdk.models import QWEN3_1_7B_INST_Q4


      def print_progress(p) -> None:
          """Print model download progress; pass as `on_progress=` to `load_model`."""
          line = (
              f"▸ Downloading {p.percentage:.0f}% "
              f"({p.downloaded / 1e6:.1f}/{p.total / 1e6:.1f} MB)"
          )
          print(line, end="\r" if sys.stderr.isatty() else "\n", file=sys.stderr)
          if p.percentage >= 100:
              print(file=sys.stderr)


      async def get_weather(arguments) -> str:
          return f"The weather in {arguments.get('location')} is 22°C and sunny."


      async def get_horoscope(arguments) -> str:
          return f"{arguments.get('sign')}: today favours careful validation."


      TOOLS = [
          {
              "name": "get_weather",
              "description": "Get the current weather for a location",
              "parameters": {
                  "type": "object",
                  "properties": {"location": {"type": "string"}},
                  "required": ["location"],
              },
              "handler": get_weather,
          },
          {
              "name": "get_horoscope",
              "description": "Get the horoscope for a star sign",
              "parameters": {
                  "type": "object",
                  "properties": {"sign": {"type": "string"}},
                  "required": ["sign"],
              },
              "handler": get_horoscope,
          },
      ]


      async def main() -> int:
          async with Client() as client:
              t = client.transport
              try:
                  model_id = await load_model(
                      t,
                      model_src=QWEN3_1_7B_INST_Q4,
                      model_config={"ctx_size": 4096, "tools": True},
                      on_progress=print_progress,
                  )
                  print(f"▸ Model loaded: {model_id}\n")

                  run = completion_orchestrate(
                      t,
                      model_id=model_id,
                      history=[
                          {
                              "role": "user",
                              "content": "What's the weather in Tokyo and my horoscope for Aquarius?",
                          }
                      ],
                      tools=TOOLS,
                      max_tool_turns=4,
                  )

                  async for event in run.events:
                      if event.type == "contentDelta":
                          sys.stdout.write(event.text)
                          sys.stdout.flush()

                  final = await run.final
                  print("\n\n▸ Final answer:", final.content_text)

                  await unload_model(t, model_id)
              except Exception as error:
                  print(f"✖ {error}", file=sys.stderr)
                  return 1
          return 0


      if __name__ == "__main__":
          sys.exit(asyncio.run(main()))
      ```
    </WrapCode>
  </Tab>

  <Tab value="kotlin" label="Kotlin">
    <WrapCode>
      ```kotlin file=<rootDir>/packages/sdk-kotlin/examples/src/CompletionOrchestrate.kt title="CompletionOrchestrate.kt" lineNumbers
      // Kotlin port of packages/sdk-python/examples/completion_orchestrate.py (no TS
      // counterpart: the JS SDK has no client wrapper for worker orchestration yet).
      // It is the server-side counterpart of the client-side loop in CompletionTools.kt.
      //
      // `client.completion.orchestrate()` hands the multi-turn tool loop to the
      // WORKER: when the model asks for a tool, the worker emits a callback frame,
      // this client runs the tool's `handler`, writes the result back up the duplex
      // request stream, and generation continues — no manual history stitching.
      // Every tool needs a `handler` (it runs code on this machine, so only a local
      // worker may orchestrate).
      //
      // Unlike Python, where it lives under a private module, `orchestrate()` is part
      // of the public Kotlin completion API. Handlers are `suspend` functions that
      // return a `JsonElement`; the worker stringifies it into the `tool` message.
      //
      // RUN: ./gradlew -p examples run -Pexample=CompletionOrchestrate

      import io.tether.qvac.sdk.QvacClient
      import io.tether.qvac.sdk.QvacMessage
      import io.tether.qvac.sdk.QvacProgressEvent
      import io.tether.qvac.sdk.QvacTool
      import io.tether.qvac.sdk.QvacToolParameter
      import io.tether.qvac.sdk.QvacToolParameterType
      import io.tether.qvac.sdk.completion
      import io.tether.qvac.sdk.generated.ModelConstant
      import io.tether.qvac.sdk.generated.Models
      import io.tether.qvac.sdk.generated.schema.LoadModelResponse
      import io.tether.qvac.sdk.generated.schema.ModelProgressResponse
      import io.tether.qvac.sdk.jvm.JvmBareKitTransport
      import io.tether.qvac.sdk.models
      import io.tether.qvac.sdk.orchestrate
      import kotlinx.coroutines.runBlocking
      import kotlinx.serialization.json.JsonObject
      import kotlinx.serialization.json.JsonPrimitive
      import kotlinx.serialization.json.buildJsonObject
      import kotlinx.serialization.json.jsonPrimitive
      import kotlinx.serialization.json.put

      private val TOOLS = listOf(
          QvacTool(
              name = "get_weather",
              description = "Get the current weather for a location",
              parameters = mapOf("location" to QvacToolParameter(QvacToolParameterType.STRING)),
              required = setOf("location"),
              handler = { arguments ->
                  JsonPrimitive("The weather in ${arguments.string("location")} is 22°C and sunny.")
              },
          ),
          QvacTool(
              name = "get_horoscope",
              description = "Get the horoscope for a star sign",
              parameters = mapOf("sign" to QvacToolParameter(QvacToolParameterType.STRING)),
              required = setOf("sign"),
              handler = { arguments ->
                  JsonPrimitive("${arguments.string("sign")}: today favours careful validation.")
              },
          ),
      )

      private fun JsonObject.string(key: String): String? = get(key)?.jsonPrimitive?.content

      fun main(): Unit = runBlocking {
          val client = QvacClient(JvmBareKitTransport.connect())
          try {
              val modelId = client.loadWithProgress(
                  Models.QWEN3_1_7B_INST_Q4,
                  modelConfig = buildJsonObject {
                      put("ctx_size", 4096)
                      put("tools", true)
                  },
              )
              println("▸ Model loaded: $modelId\n")

              val run = client.completion.orchestrate(
                  modelId = modelId,
                  history = listOf(QvacMessage.user("What's the weather in Tokyo and my horoscope for Aquarius?")),
                  tools = TOOLS,
                  maxToolTurns = 4,
              )

              run.tokens.collect { print(it) }

              val final = run.final.await()
              println("\n\n▸ Final answer: ${final.text}")

              client.models.unload(modelId)
          } finally {
              client.close()
          }
      }

      /** Load [model], printing download progress, and return its model id. */
      private suspend fun QvacClient.loadWithProgress(model: ModelConstant, modelConfig: JsonObject? = null): String {
          var result: LoadModelResponse? = null
          models.loadWithProgress(model.src, model.engine, model.name, modelConfig = modelConfig).collect { event ->
              when (event) {
                  is QvacProgressEvent.Progress -> printProgress(event.value)
                  is QvacProgressEvent.Result -> result = event.value
              }
          }
          val loaded = checkNotNull(result) { "loadModel ended without a result" }
          return loaded.modelId ?: error(loaded.error ?: "loadModel failed")
      }

      private fun printProgress(p: ModelProgressResponse) {
          val line = "▸ Downloading %.0f%% (%.1f/%.1f MB)".format(p.percentage, p.downloaded / 1e6, p.total / 1e6)
          if (System.console() != null) System.err.print("\r$line") else System.err.println(line)
          if (p.percentage >= 100) System.err.println()
      }
      ```
    </WrapCode>
  </Tab>
</Tabs>

### Deferred tool loading

A large tool set costs prompt tokens on every conversation and makes the model's
choice harder. Setting `deferLoading: true` on a tool keeps it registered but
leaves its parameter schema out of the prompt: the model sees only its name and
description, in a compact catalog carried by a built-in `tool_search` tool, and
loads the schema when it needs it.

The JavaScript and TypeScript examples below import the same `shared` module shown under [Tool call](#tool-call); save it alongside this example too.

<Tabs>
  <Tab value="js" label="JavaScript" default>
    <WrapCode>
      ```js file=<rootDir>/packages/sdk/dist/examples/tools/llamacpp-deferred-tools.js title="completion-deferred-tools.js" lineNumbers
      // Deferred tool loading: a large tool set stays out of the prompt until the
      // model asks for it. Only `get_weather` and the built-in `tool_search` (whose
      // description carries a name-and-description catalog of the rest) are in the
      // initial prompt; a deferred schema is declared in the prompt once the model
      // has searched for it.
      import { TOOL_SEARCH_NAME, completion, executeToolSearch, loadModel, unloadModel, QWEN3_1_7B_INST_Q4 } from '@qvac/sdk';
      import { z } from 'zod';
      import { weatherSchema, horoscopeSchema, mockExecute } from './shared.js';
      const tools = [
          {
              name: 'get_weather',
              description: 'Get current weather for a city',
              parameters: weatherSchema
          },
          {
              name: 'get_horoscope',
              description: "Get today's horoscope for an astrological sign",
              parameters: horoscopeSchema,
              deferLoading: true,
              group: 'astrology'
          },
          {
              name: 'get_moon_phase',
              description: 'Get the current phase of the moon',
              parameters: z.object({ date: z.string().describe('ISO date').optional() }),
              deferLoading: true,
              group: 'astrology'
          }
      ];
      try {
          const modelId = await loadModel({
              modelSrc: QWEN3_1_7B_INST_Q4,
              modelConfig: { ctx_size: 4096, tools: true }
          });
          console.log(`▸ Model loaded. Model ID: ${modelId}`);
          // One search loads at most two definitions, one unless the model asks for
          // more. Keep these the same for the whole conversation.
          const params = { modelId, tools, toolSearch: { defaultLimit: 1, maxLimit: 2 } };
          const history = [
              {
                  role: 'system',
                  content: 'You are a helpful assistant. Search for a tool before calling it.'
              },
              { role: 'user', content: "What's my horoscope for Aquarius?" }
          ];
          console.log('\n▸ Turn 1 — only get_weather and tool_search are in the prompt\n');
          const first = completion({ ...params, history, stream: true });
          for await (const token of first.tokenStream)
              process.stdout.write(token);
          const calls = await first.toolCalls;
          const search = calls.find((call) => call.name === TOOL_SEARCH_NAME);
          if (!search) {
              console.log('\n\n▸ The model answered without searching; nothing more to show.');
          }
          else {
              console.log(`\n\n▸ The model searched: ${JSON.stringify(search.arguments)}`);
              // Running the search is the SDK's job, not a tool handler's. The result is
              // an ordinary tool message naming what it loaded: keeping it in the history
              // is what declares those definitions on the next turn. Passing the same
              // params applies the limits the model was told.
              history.push({ role: 'assistant', content: await first.text });
              history.push({
                  role: 'tool',
                  content: executeToolSearch(params, search.arguments, history)
              });
              console.log('▸ Definitions loaded. Turn 2 — the model can now call them directly.\n');
              const second = completion({ ...params, history, stream: true });
              for await (const token of second.tokenStream)
                  process.stdout.write(token);
              for (const call of await second.toolCalls) {
                  console.log(`\n\n▸ Tool call: ${call.name}(${JSON.stringify(call.arguments)})`);
                  console.log(`▸ Result: ${mockExecute(call.name, call.arguments)}`);
              }
          }
          console.log('\n\n▸ Completed!');
          await unloadModel({ modelId, clearStorage: false });
      }
      catch (error) {
          console.error('✖', error);
          process.exit(1);
      }
      ```
    </WrapCode>
  </Tab>

  <Tab value="ts" label="TypeScript">
    <WrapCode>
      ```ts file=<rootDir>/packages/sdk/examples/tools/llamacpp-deferred-tools.ts title="completion-deferred-tools.ts" lineNumbers
      // Deferred tool loading: a large tool set stays out of the prompt until the
      // model asks for it. Only `get_weather` and the built-in `tool_search` (whose
      // description carries a name-and-description catalog of the rest) are in the
      // initial prompt; a deferred schema is declared in the prompt once the model
      // has searched for it.
      import {
        TOOL_SEARCH_NAME,
        completion,
        executeToolSearch,
        loadModel,
        unloadModel,
        type ToolInput,
        QWEN3_1_7B_INST_Q4
      } from '@qvac/sdk'
      import { z } from 'zod'
      import { weatherSchema, horoscopeSchema, mockExecute } from './shared.js'

      const tools: ToolInput[] = [
        {
          name: 'get_weather',
          description: 'Get current weather for a city',
          parameters: weatherSchema
        },
        {
          name: 'get_horoscope',
          description: "Get today's horoscope for an astrological sign",
          parameters: horoscopeSchema,
          deferLoading: true,
          group: 'astrology'
        },
        {
          name: 'get_moon_phase',
          description: 'Get the current phase of the moon',
          parameters: z.object({ date: z.string().describe('ISO date').optional() }),
          deferLoading: true,
          group: 'astrology'
        }
      ]

      try {
        const modelId = await loadModel({
          modelSrc: QWEN3_1_7B_INST_Q4,
          modelConfig: { ctx_size: 4096, tools: true }
        })
        console.log(`▸ Model loaded. Model ID: ${modelId}`)

        // One search loads at most two definitions, one unless the model asks for
        // more. Keep these the same for the whole conversation.
        const params = { modelId, tools, toolSearch: { defaultLimit: 1, maxLimit: 2 } }

        const history = [
          {
            role: 'system',
            content: 'You are a helpful assistant. Search for a tool before calling it.'
          },
          { role: 'user', content: "What's my horoscope for Aquarius?" }
        ]

        console.log('\n▸ Turn 1 — only get_weather and tool_search are in the prompt\n')
        const first = completion({ ...params, history, stream: true })
        for await (const token of first.tokenStream) process.stdout.write(token)

        const calls = await first.toolCalls
        const search = calls.find((call) => call.name === TOOL_SEARCH_NAME)
        if (!search) {
          console.log('\n\n▸ The model answered without searching; nothing more to show.')
        } else {
          console.log(`\n\n▸ The model searched: ${JSON.stringify(search.arguments)}`)

          // Running the search is the SDK's job, not a tool handler's. The result is
          // an ordinary tool message naming what it loaded: keeping it in the history
          // is what declares those definitions on the next turn. Passing the same
          // params applies the limits the model was told.
          history.push({ role: 'assistant', content: await first.text })
          history.push({
            role: 'tool',
            content: executeToolSearch(params, search.arguments, history)
          })

          console.log('▸ Definitions loaded. Turn 2 — the model can now call them directly.\n')
          const second = completion({ ...params, history, stream: true })
          for await (const token of second.tokenStream) process.stdout.write(token)

          for (const call of await second.toolCalls) {
            console.log(`\n\n▸ Tool call: ${call.name}(${JSON.stringify(call.arguments)})`)
            console.log(`▸ Result: ${mockExecute(call.name, call.arguments)}`)
          }
        }

        console.log('\n\n▸ Completed!')
        await unloadModel({ modelId, clearStorage: false })
      } catch (error) {
        console.error('✖', error)
        process.exit(1)
      }
      ```
    </WrapCode>
  </Tab>

  <Tab value="python" label="Python">
    <WrapCode>
      ```python file=<rootDir>/packages/sdk-python/examples/completion_deferred_tools.py title="completion_deferred_tools.py" lineNumbers
      """Deferred tool loading — Python port of
      packages/sdk/examples/tools/llamacpp-deferred-tools.ts.

      A large tool set stays out of the prompt until the model asks for it. Only
      `get_weather` and the built-in `tool_search` (whose description carries a
      name-and-description catalog of the rest) are in the initial prompt; a
      deferred schema is declared in the prompt once the model has searched for it.

      Runs through the worker-orchestrated loop rather than the client-side loop of
      completion_tools.py: executing `tool_search` from the client needs the
      `executeToolSearch` helper, which only the JS SDK exports. On this path the
      worker runs the search itself and calls back only for the real tools.

      RUN: python examples/completion_deferred_tools.py
      """

      from __future__ import annotations

      import asyncio
      import json
      import sys

      from tetherto.qvac_sdk import Client, load_model, unload_model
      from tetherto.qvac_sdk._completion import completion_orchestrate
      from tetherto.qvac_sdk.models import QWEN3_1_7B_INST_Q4


      def print_progress(p) -> None:
          """Print model download progress; pass as `on_progress=` to `load_model`."""
          line = (
              f"▸ Downloading {p.percentage:.0f}% "
              f"({p.downloaded / 1e6:.1f}/{p.total / 1e6:.1f} MB)"
          )
          print(line, end="\r" if sys.stderr.isatty() else "\n", file=sys.stderr)
          if p.percentage >= 100:
              print(file=sys.stderr)


      async def get_weather(arguments) -> str:
          return json.dumps(
              {
                  "city": arguments.get("city"),
                  "country": arguments.get("country", "unknown"),
                  "temperature": "22°C",
                  "condition": "Partly cloudy",
              }
          )


      async def get_horoscope(arguments) -> str:
          return json.dumps(
              {
                  "sign": arguments.get("sign"),
                  "horoscope": "Today is a great day for new beginnings.",
              }
          )


      async def get_moon_phase(arguments) -> str:
          return json.dumps(
              {"date": arguments.get("date", "today"), "phase": "Waxing gibbous"}
          )


      # `defer_loading` is the snake_case alias of the wire field `deferLoading`; the
      # SDK maps it before sending. `group` is the heading the tool is listed under
      # in the catalog, and is ignored for tools that are not deferred.
      TOOLS = [
          {
              "name": "get_weather",
              "description": "Get current weather for a city",
              "parameters": {
                  "type": "object",
                  "properties": {
                      "city": {"type": "string", "description": "City name"},
                      "country": {"type": "string", "description": "Country code"},
                  },
                  "required": ["city"],
              },
              "handler": get_weather,
          },
          {
              "name": "get_horoscope",
              "description": "Get today's horoscope for an astrological sign",
              "parameters": {
                  "type": "object",
                  "properties": {
                      "sign": {
                          "type": "string",
                          "description": "An astrological sign like Taurus or Aquarius",
                      }
                  },
                  "required": ["sign"],
              },
              "handler": get_horoscope,
              "defer_loading": True,
              "group": "astrology",
          },
          {
              "name": "get_moon_phase",
              "description": "Get the current phase of the moon",
              "parameters": {
                  "type": "object",
                  "properties": {"date": {"type": "string", "description": "ISO date"}},
              },
              "handler": get_moon_phase,
              "defer_loading": True,
              "group": "astrology",
          },
      ]


      async def main() -> int:
          async with Client() as client:
              t = client.transport
              try:
                  model_id = await load_model(
                      t,
                      model_src=QWEN3_1_7B_INST_Q4,
                      model_config={"ctx_size": 4096, "tools": True},
                      on_progress=print_progress,
                  )
                  print(f"▸ Model loaded: {model_id}")
                  print("\n▸ Only get_weather and tool_search are in the prompt\n")

                  run = completion_orchestrate(
                      t,
                      model_id=model_id,
                      history=[
                          {
                              "role": "system",
                              "content": "You are a helpful assistant. Search for a tool before calling it.",
                          },
                          {"role": "user", "content": "What's my horoscope for Aquarius?"},
                      ],
                      tools=TOOLS,
                      # One search loads at most two definitions, one unless the
                      # model asks for more.
                      tool_search={"default_limit": 1, "max_limit": 2},
                      max_tool_turns=4,
                  )

                  async for event in run.events:
                      if event.type == "contentDelta":
                          sys.stdout.write(event.text)
                          sys.stdout.flush()
                      elif event.type == "toolCall":
                          arguments = json.dumps(dict(event.call.arguments.root))
                          print(f"\n\n▸ Tool call: {event.call.name}({arguments})\n")

                  final = await run.final
                  print(f"\n\n▸ Final answer: {final.content_text}")

                  print("\n▸ Completed!")
                  await unload_model(t, model_id)
              except Exception as error:
                  print(f"✖ {error}", file=sys.stderr)
                  return 1
          return 0


      if __name__ == "__main__":
          sys.exit(asyncio.run(main()))
      ```
    </WrapCode>
  </Tab>

  <Tab value="kotlin" label="Kotlin">
    <WrapCode>
      ```kotlin file=<rootDir>/packages/sdk-kotlin/examples/src/CompletionDeferredTools.kt title="CompletionDeferredTools.kt" lineNumbers
      // Kotlin port of packages/sdk/examples/tools/llamacpp-deferred-tools.ts.
      //
      // Deferred tool loading. A large tool set stays out of the prompt until the
      // model asks for it. Only `get_weather` and the built-in `tool_search` (whose
      // description carries a name-and-description catalog of the rest) are in the
      // initial prompt; a deferred schema is declared in the prompt once the model
      // has searched for it.
      //
      // Runs through the worker-orchestrated loop rather than the client-side loop of
      // CompletionTools.kt: executing `tool_search` from the client needs the
      // `executeToolSearch` helper, which only the JS SDK exports. On this path the
      // worker runs the search itself and calls back only for the real tools.
      //
      // `QvacTool` (used by `client.completion.orchestrate()`) has no `deferLoading`
      // or `group` field, so this example drives the generated typed operation
      // directly: `client.completionOrchestrate(request, input)` takes a
      // `CompletionOrchestrateRequest` whose `CompletionStreamRequestToolsItem`
      // carries both fields, and returns a `Flow` of typed frames. The duplex `input`
      // is a `Flow<ByteArray>` fed from a `Channel`: each `toolCallback` frame is
      // answered with one newline-delimited `{callId, result}` line, and the worker
      // resumes generation. `group` is the heading the tool is listed under in the
      // catalog, and is ignored for tools that are not deferred. The `tool_search`
      // limits go in the request's typed `toolSearch` field
      // (`CompletionStreamRequestToolSearch`).
      //
      // RUN: ./gradlew -p examples run -Pexample=CompletionDeferredTools

      import io.tether.qvac.sdk.QvacClient
      import io.tether.qvac.sdk.QvacProgressEvent
      import io.tether.qvac.sdk.completionOrchestrate
      import io.tether.qvac.sdk.generated.ModelConstant
      import io.tether.qvac.sdk.generated.Models
      import io.tether.qvac.sdk.generated.schema.CompletionOrchestrateRequest
      import io.tether.qvac.sdk.generated.schema.CompletionStreamRequestHistoryItem
      import io.tether.qvac.sdk.generated.schema.CompletionStreamRequestHistoryItemContent
      import io.tether.qvac.sdk.generated.schema.CompletionStreamRequestToolSearch
      import io.tether.qvac.sdk.generated.schema.CompletionStreamRequestToolsItem
      import io.tether.qvac.sdk.generated.schema.CompletionStreamResponseEventsItem
      import io.tether.qvac.sdk.generated.schema.LoadModelResponse
      import io.tether.qvac.sdk.generated.schema.ModelProgressResponse
      import io.tether.qvac.sdk.jvm.JvmBareKitTransport
      import io.tether.qvac.sdk.models
      import kotlinx.coroutines.channels.Channel
      import kotlinx.coroutines.flow.receiveAsFlow
      import kotlinx.coroutines.runBlocking
      import kotlinx.serialization.json.JsonElement
      import kotlinx.serialization.json.JsonObject
      import kotlinx.serialization.json.add
      import kotlinx.serialization.json.buildJsonObject
      import kotlinx.serialization.json.jsonPrimitive
      import kotlinx.serialization.json.put
      import kotlinx.serialization.json.putJsonArray
      import kotlinx.serialization.json.putJsonObject

      private val HANDLERS: Map<String, (JsonObject) -> JsonElement> = mapOf(
          "get_weather" to { arguments ->
              buildJsonObject {
                  put("city", arguments.string("city"))
                  put("country", arguments.string("country") ?: "unknown")
                  put("temperature", "22°C")
                  put("condition", "Partly cloudy")
              }
          },
          "get_horoscope" to { arguments ->
              buildJsonObject {
                  put("sign", arguments.string("sign"))
                  put("horoscope", "Today is a great day for new beginnings.")
              }
          },
          "get_moon_phase" to { arguments ->
              buildJsonObject {
                  put("date", arguments.string("date") ?: "today")
                  put("phase", "Waxing gibbous")
              }
          },
      )

      private val TOOLS = listOf(
          CompletionStreamRequestToolsItem(
              name = "get_weather",
              description = "Get current weather for a city",
              parameters = objectSchema(
                  "city" to "City name",
                  "country" to "Country code",
                  required = listOf("city"),
              ),
          ),
          CompletionStreamRequestToolsItem(
              name = "get_horoscope",
              description = "Get today's horoscope for an astrological sign",
              parameters = objectSchema(
                  "sign" to "An astrological sign like Taurus or Aquarius",
                  required = listOf("sign"),
              ),
              deferLoading = true,
              group = "astrology",
          ),
          CompletionStreamRequestToolsItem(
              name = "get_moon_phase",
              description = "Get the current phase of the moon",
              parameters = objectSchema("date" to "ISO date"),
              deferLoading = true,
              group = "astrology",
          ),
      )

      fun main(): Unit = runBlocking {
          val client = QvacClient(JvmBareKitTransport.connect())
          try {
              val modelId = client.loadWithProgress(
                  Models.QWEN3_1_7B_INST_Q4,
                  modelConfig = buildJsonObject {
                      put("ctx_size", 4096)
                      put("tools", true)
                  },
              )
              println("▸ Model loaded: $modelId")
              println("\n▸ Only get_weather and tool_search are in the prompt\n")

              val request = CompletionOrchestrateRequest(
                  modelId = modelId,
                  history = listOf(
                      CompletionStreamRequestHistoryItem(
                          role = "system",
                          content = CompletionStreamRequestHistoryItemContent.Variant1(
                              "You are a helpful assistant. Search for a tool before calling it.",
                          ),
                      ),
                      CompletionStreamRequestHistoryItem(
                          role = "user",
                          content = CompletionStreamRequestHistoryItemContent.Variant1("What's my horoscope for Aquarius?"),
                      ),
                  ),
                  tools = TOOLS,
                  // One search loads at most two definitions, one unless the
                  // model asks for more.
                  toolSearch = CompletionStreamRequestToolSearch(defaultLimit = 1, maxLimit = 2),
                  stream = true,
                  maxToolTurns = 4,
              )

              // The final answer is the text of the last turn.
              val upstream = Channel<ByteArray>(Channel.UNLIMITED)
              var turn: Long? = null
              val answer = StringBuilder()
              try {
                  client.completionOrchestrate(request, upstream.receiveAsFlow()).collect { frame ->
                      if (frame.turn != null && frame.turn != turn) {
                          turn = frame.turn
                          answer.clear()
                      }
                      for (event in frame.events.orEmpty()) {
                          when (event) {
                              is CompletionStreamResponseEventsItem.ContentDelta -> {
                                  print(event.value.text)
                                  answer.append(event.value.text)
                              }
                              is CompletionStreamResponseEventsItem.ToolCall -> {
                                  val call = event.value.call
                                  println("\n\n▸ Tool call: ${call.name}(${JsonObject(call.arguments)})\n")
                              }
                              is CompletionStreamResponseEventsItem.CompletionDoneError -> error(event.value.error.message)
                              else -> Unit
                          }
                      }
                      frame.toolCallback?.let { callback ->
                          val handler = HANDLERS.getValue(callback.name)
                          val reply = buildJsonObject {
                              put("callId", callback.callId)
                              put("result", handler(JsonObject(callback.arguments)))
                          }
                          upstream.send("$reply\n".encodeToByteArray())
                      }
                  }
              } finally {
                  upstream.close()
              }
              println("\n\n▸ Final answer: $answer")

              println("\n▸ Completed!")
              client.models.unload(modelId)
          } finally {
              client.close()
          }
      }

      private fun JsonObject.string(key: String): String? = get(key)?.jsonPrimitive?.content

      /** A JSON Schema object of string properties, each with a description. */
      private fun objectSchema(vararg properties: Pair<String, String>, required: List<String> = emptyList()) =
          buildJsonObject {
              put("type", "object")
              putJsonObject("properties") {
                  for ((name, description) in properties) {
                      putJsonObject(name) {
                          put("type", "string")
                          put("description", description)
                      }
                  }
              }
              if (required.isNotEmpty()) putJsonArray("required") { required.forEach { add(it) } }
          }

      /** Load [model], printing download progress, and return its model id. */
      private suspend fun QvacClient.loadWithProgress(model: ModelConstant, modelConfig: JsonObject? = null): String {
          var result: LoadModelResponse? = null
          models.loadWithProgress(model.src, model.engine, model.name, modelConfig = modelConfig).collect { event ->
              when (event) {
                  is QvacProgressEvent.Progress -> printProgress(event.value)
                  is QvacProgressEvent.Result -> result = event.value
              }
          }
          val loaded = checkNotNull(result) { "loadModel ended without a result" }
          return loaded.modelId ?: error(loaded.error ?: "loadModel failed")
      }

      private fun printProgress(p: ModelProgressResponse) {
          val line = "▸ Downloading %.0f%% (%.1f/%.1f MB)".format(p.percentage, p.downloaded / 1e6, p.total / 1e6)
          if (System.console() != null) System.err.print("\r$line") else System.err.println(line)
          if (p.percentage >= 100) System.err.println()
      }
      ```
    </WrapCode>
  </Tab>
</Tabs>

The Python and Kotlin examples take the [worker-orchestrated loop](#worker-orchestrated-tool-loop) while the others take the client-side one. Deferring a tool works the same in both languages — the worker builds the catalog and reads back what the conversation has loaded — but running `tool_search` from client code needs the `executeToolSearch` helper, which only the JS SDK exports. On the orchestrated path the worker runs the search itself.

The model calls `tool_search({ query })` with a capability or an exact tool
name; the SDK searches the registered tools outside inference and returns a
tool result naming the matches, the next turn declares their definitions in the
prompt alongside the other tools, and the model calls them by name.

Notes:

* Tools without `deferLoading` behave exactly as before.
* `tool_search` is a reserved name and is added only when something defers.
* `group` is the heading a tool is listed under in the catalog — a skill or an
  MCP server name.
* To defer an MCP server's tools, set it on the client entry:
  `mcp: [{ client, deferLoading: true, group: 'github' }]`.
* Loading a definition twice declares it once; the search result says the tool
  is already available.
* Give tools distinct names. A deferred tool the prompt could not declare is
  left out of the catalog and cannot be loaded: an empty name, `auto`, `none`
  or `required`, or a name equal to an always-loaded tool's, `tool_search` or
  an earlier deferred tool's once every run of characters outside `A-Z`, `a-z`,
  `0-9` and `-` is read as `-` (`get_weather` and `get-weather`).
* `completion()` stays stateless. The loaded set is read back from the
  conversation, so a reopened chat given the same tools and the same history has
  the same tools loaded, and concurrent chats cannot mix.
* `generationParams.tool_choice` cannot name a deferred tool — its schema is not
  in the prompt, so the call cannot be constrained. Force `"tool_search"`
  instead.
* The tool block sits near the start of the prompt, so the turn after a load
  re-processes most or all of the prompt instead of reusing the cached prefix.
* On the worker-orchestrated loop the SDK runs `tool_search` itself; your
  handlers are never called for it.

### MCP

You create and manage the MCP client, connect it to one or more MCP servers, and pass it to `completion()`. The following script shows how to attach an MCP client to `completion()` so the model can call a web search tool and then continue with the results:

<Tabs>
  <Tab value="js" label="JavaScript" default>
    <WrapCode>
      ```js file=<rootDir>/packages/sdk/dist/examples/mcp-websearch.js title="completion-mcp.js" lineNumbers
      /**
       * MCP DuckDuckGo Search Example
       *
       * A web search example using DuckDuckGo - no API key required!
       * The server provides tools to search the web and get answers.
       *
       * Prerequisites:
       * - Install MCP SDK: npm install @modelcontextprotocol/sdk
       *
       * Run with: npx tsx examples/mcp-websearch.ts
       */
      import { completion, loadModel, unloadModel, QWEN3_1_7B_INST_Q4 } from '@qvac/sdk';
      // MCP SDK is a user-installed optional dependency
      // Install with: npm install @modelcontextprotocol/sdk
      import { Client } from '@modelcontextprotocol/sdk/client/index.js';
      import { StdioClientTransport } from '@modelcontextprotocol/sdk/client/stdio.js';
      function parseSearchResults(mcpResult) {
          try {
              const result = mcpResult;
              // Extract text content from MCP response
              const textContent = result.content?.find((c) => c.type === 'text');
              if (!textContent?.text) {
                  return JSON.stringify(mcpResult);
              }
              // Parse the JSON array of search results
              const rawResults = JSON.parse(textContent.text);
              // Extract just the useful fields (title, url, snippet)
              const cleanResults = rawResults.slice(0, 5).map((r) => ({
                  title: r.title ?? 'Unknown',
                  url: r.url ?? '',
                  snippet: r.snippet ?? ''
              }));
              // Format as concise text for LLM
              return cleanResults
                  .map((r, i) => `[${i + 1}] ${r.title}\n    URL: ${r.url}\n    ${r.snippet}`)
                  .join('\n\n');
          }
          catch {
              // If parsing fails, return a truncated version
              const str = typeof mcpResult === 'string' ? mcpResult : JSON.stringify(mcpResult);
              return str.slice(0, 2000);
          }
      }
      let mcpClient = null;
      try {
          console.log('▸ MCP DuckDuckGo Search Example\n');
          // ============================================================
          // STEP 1: Connect to DuckDuckGo MCP server
          // ============================================================
          console.log('▸ Starting DuckDuckGo MCP server...');
          mcpClient = new Client({
              name: 'qvac-ddg-example',
              version: '1.0.0'
          });
          const transport = new StdioClientTransport({
              command: 'npx',
              args: ['-y', '@oevortex/ddg_search']
          });
          await mcpClient.connect(transport);
          console.log('▸ MCP server connected\n');
          // ============================================================
          // STEP 2: Load model
          // ============================================================
          console.log('▸ Loading model...');
          const modelId = await loadModel({
              modelSrc: QWEN3_1_7B_INST_Q4,
              modelConfig: {
                  ctx_size: 4096,
                  tools: true
              },
              onProgress: (p) => {
                  const mb = (n) => (n / 1e6).toFixed(1);
                  const line = `▸ Downloading ${p.percentage.toFixed(0)}% (${mb(p.downloaded)}/${mb(p.total)} MB)`;
                  process.stderr.write(process.stderr.isTTY ? `\r${line}` : `${line}\n`);
                  if (p.percentage >= 100)
                      process.stderr.write('\n');
              }
          });
          console.log(`\n▸ Model loaded\n`);
          // ============================================================
          // STEP 3: Ask AI to search the web (with MCP client)
          // ============================================================
          const history = [
              {
                  role: 'system',
                  content: `You are a helpful assistant with access to web search.
      Use the search tool when you need current information.
      Always cite your sources with the URL.`
              },
              {
                  role: 'user',
                  content: 'What is the current weather in New York City?'
              }
          ];
          console.log('▸ Asking AI to search the web...\n');
          console.log('▸ AI Response:');
          // Pass MCP client directly to completion - tools are adapted internally!
          const result = completion({
              modelId,
              history,
              stream: true,
              mcp: [{ client: mcpClient, includeResources: false }]
          });
          for await (const token of result.tokenStream) {
              process.stdout.write(token);
          }
          const toolCalls = await result.toolCalls;
          console.log('\n');
          // ============================================================
          // STEP 4: Execute tool calls using call() - automatic MCP routing!
          // ============================================================
          if (toolCalls.length > 0) {
              console.log('▸ Executing search...\n');
              const toolResults = [];
              for (const toolCall of toolCalls) {
                  console.log(`▸ ${toolCall.name}(${JSON.stringify(toolCall.arguments)})`);
                  if (!toolCall.invoke) {
                      console.log(`▸ No handler found for tool "${toolCall.name}"`);
                      continue;
                  }
                  // Use invoke() - automatically routes to the correct MCP client!
                  const mcpResult = await toolCall.invoke();
                  // Parse and clean up the search results
                  const cleanResult = parseSearchResults(mcpResult);
                  console.log(`▸ Got search results:`);
                  console.log(cleanResult
                      .split('\n')
                      .map((l) => `      ${l}`)
                      .join('\n'));
                  console.log();
                  toolResults.push({ id: toolCall.id, result: cleanResult });
              }
              // ============================================================
              // STEP 5: Continue with search results
              // ============================================================
              console.log('▸ Getting AI response with search results...\n');
              history.push({
                  role: 'assistant',
                  content: await result.text
              });
              for (const tr of toolResults) {
                  history.push({
                      role: 'tool',
                      content: tr.result
                  });
              }
              console.log('▸ Final Response:');
              const finalResult = completion({
                  modelId,
                  history,
                  stream: true,
                  mcp: [{ client: mcpClient, includeResources: false }]
              });
              for await (const token of finalResult.tokenStream) {
                  process.stdout.write(token);
              }
              console.log('\n');
          }
          // ============================================================
          // Cleanup
          // ============================================================
          console.log('▸ Cleaning up...');
          await unloadModel({ modelId, clearStorage: false });
          console.log('▸ Done\n');
          console.log('▸ Example completed!');
          process.exit(0);
      }
      catch (error) {
          console.error('✖', error);
          process.exit(1);
      }
      finally {
          if (mcpClient) {
              try {
                  await mcpClient.close();
              }
              catch {
                  // Ignore close errors
              }
          }
      }
      ```
    </WrapCode>
  </Tab>

  <Tab value="ts" label="TypeScript">
    <WrapCode>
      ```ts file=<rootDir>/packages/sdk/examples/mcp-websearch.ts title="completion-mcp.ts" lineNumbers
      /**
       * MCP DuckDuckGo Search Example
       *
       * A web search example using DuckDuckGo - no API key required!
       * The server provides tools to search the web and get answers.
       *
       * Prerequisites:
       * - Install MCP SDK: npm install @modelcontextprotocol/sdk
       *
       * Run with: npx tsx examples/mcp-websearch.ts
       */

      import { completion, loadModel, unloadModel, QWEN3_1_7B_INST_Q4 } from '@qvac/sdk'

      // MCP SDK is a user-installed optional dependency
      // Install with: npm install @modelcontextprotocol/sdk
      import { Client } from '@modelcontextprotocol/sdk/client/index.js'
      import { StdioClientTransport } from '@modelcontextprotocol/sdk/client/stdio.js'

      // ============================================================
      // Helper: Parse MCP search results into clean format for LLM
      // ============================================================
      type SearchResult = {
        title: string
        url: string
        snippet: string
      }

      type McpContent = {
        type: string
        text?: string
      }

      type McpToolResult = {
        content: McpContent[]
      }

      type RawSearchResult = {
        title?: string
        url?: string
        snippet?: string
        Description?: string
      }

      function parseSearchResults(mcpResult: unknown): string {
        try {
          const result = mcpResult as McpToolResult

          // Extract text content from MCP response
          const textContent = result.content?.find((c) => c.type === 'text')
          if (!textContent?.text) {
            return JSON.stringify(mcpResult)
          }

          // Parse the JSON array of search results
          const rawResults = JSON.parse(textContent.text) as RawSearchResult[]

          // Extract just the useful fields (title, url, snippet)
          const cleanResults: SearchResult[] = rawResults.slice(0, 5).map((r) => ({
            title: r.title ?? 'Unknown',
            url: r.url ?? '',
            snippet: r.snippet ?? ''
          }))

          // Format as concise text for LLM
          return cleanResults
            .map((r, i) => `[${i + 1}] ${r.title}\n    URL: ${r.url}\n    ${r.snippet}`)
            .join('\n\n')
        } catch {
          // If parsing fails, return a truncated version
          const str = typeof mcpResult === 'string' ? mcpResult : JSON.stringify(mcpResult)
          return str.slice(0, 2000)
        }
      }

      let mcpClient: Client | null = null

      try {
        console.log('▸ MCP DuckDuckGo Search Example\n')

        // ============================================================
        // STEP 1: Connect to DuckDuckGo MCP server
        // ============================================================
        console.log('▸ Starting DuckDuckGo MCP server...')

        mcpClient = new Client({
          name: 'qvac-ddg-example',
          version: '1.0.0'
        })

        const transport = new StdioClientTransport({
          command: 'npx',
          args: ['-y', '@oevortex/ddg_search']
        })

        await mcpClient.connect(transport)
        console.log('▸ MCP server connected\n')

        // ============================================================
        // STEP 2: Load model
        // ============================================================
        console.log('▸ Loading model...')
        const modelId = await loadModel({
          modelSrc: QWEN3_1_7B_INST_Q4,
          modelConfig: {
            ctx_size: 4096,
            tools: true
          },
          onProgress: (p) => {
            const mb = (n: number) => (n / 1e6).toFixed(1)
            const line = `▸ Downloading ${p.percentage.toFixed(0)}% (${mb(p.downloaded)}/${mb(p.total)} MB)`
            process.stderr.write(process.stderr.isTTY ? `\r${line}` : `${line}\n`)
            if (p.percentage >= 100) process.stderr.write('\n')
          }
        })
        console.log(`\n▸ Model loaded\n`)

        // ============================================================
        // STEP 3: Ask AI to search the web (with MCP client)
        // ============================================================
        const history = [
          {
            role: 'system',
            content: `You are a helpful assistant with access to web search.
      Use the search tool when you need current information.
      Always cite your sources with the URL.`
          },
          {
            role: 'user',
            content: 'What is the current weather in New York City?'
          }
        ]

        console.log('▸ Asking AI to search the web...\n')
        console.log('▸ AI Response:')

        // Pass MCP client directly to completion - tools are adapted internally!
        const result = completion({
          modelId,
          history,
          stream: true,
          mcp: [{ client: mcpClient, includeResources: false }]
        })

        for await (const token of result.tokenStream) {
          process.stdout.write(token)
        }

        const toolCalls = await result.toolCalls
        console.log('\n')

        // ============================================================
        // STEP 4: Execute tool calls using call() - automatic MCP routing!
        // ============================================================
        if (toolCalls.length > 0) {
          console.log('▸ Executing search...\n')

          const toolResults: Array<{ id: string; result: string }> = []

          for (const toolCall of toolCalls) {
            console.log(`▸ ${toolCall.name}(${JSON.stringify(toolCall.arguments)})`)

            if (!toolCall.invoke) {
              console.log(`▸ No handler found for tool "${toolCall.name}"`)
              continue
            }

            // Use invoke() - automatically routes to the correct MCP client!
            const mcpResult = await toolCall.invoke()

            // Parse and clean up the search results
            const cleanResult = parseSearchResults(mcpResult)

            console.log(`▸ Got search results:`)
            console.log(
              cleanResult
                .split('\n')
                .map((l) => `      ${l}`)
                .join('\n')
            )
            console.log()

            toolResults.push({ id: toolCall.id, result: cleanResult })
          }

          // ============================================================
          // STEP 5: Continue with search results
          // ============================================================
          console.log('▸ Getting AI response with search results...\n')

          history.push({
            role: 'assistant',
            content: await result.text
          })

          for (const tr of toolResults) {
            history.push({
              role: 'tool',
              content: tr.result
            })
          }

          console.log('▸ Final Response:')
          const finalResult = completion({
            modelId,
            history,
            stream: true,
            mcp: [{ client: mcpClient, includeResources: false }]
          })

          for await (const token of finalResult.tokenStream) {
            process.stdout.write(token)
          }
          console.log('\n')
        }

        // ============================================================
        // Cleanup
        // ============================================================
        console.log('▸ Cleaning up...')
        await unloadModel({ modelId, clearStorage: false })
        console.log('▸ Done\n')

        console.log('▸ Example completed!')
        process.exit(0)
      } catch (error) {
        console.error('✖', error)
        process.exit(1)
      } finally {
        if (mcpClient) {
          try {
            await mcpClient.close()
          } catch {
            // Ignore close errors
          }
        }
      }
      ```
    </WrapCode>
  </Tab>
</Tabs>

<Callout type="warn">
  MCP is available in the JS/TS client only. The integration lives in the client, not in the worker, so the Python and Kotlin clients cannot reach it.
</Callout>

### KV cache

The `kvCache` parameter on `completion()` both enables caching and sets the cache key. It accepts:

* A **non-empty string** (e.g. `"user-123-session-a"`) — use this key for a caller-managed cache. You control the session identity, so you can reuse the same key across turns and later clean it up with [`deleteCache({ kvCacheKey })`](/sdk/v0.23/reference/api#deletecache).
* `true` — SDK-managed auto cache. The SDK generates the key internally from the conversation history. Convenient for single-session use; you don't get a stable key for per-session cleanup, but you can reclaim them in bulk with [`deleteCache({ auto: true })`](/sdk/v0.23/reference/api#deletecache).
* `false` or `undefined` — no caching (default).

SDK-managed auto caches are removed after 24 hours idle or under disk quota pressure. The quota is 512 MiB on React Native and 4 GiB on other runtimes. [`deleteCache({ auto: true })`](/sdk/v0.23/reference/api#deletecache) reclaims them on demand instead of waiting for either bound, skipping any cache a request is still using. Caller-managed string-key caches are not subject to the automatic retention policy and are not touched by that call — clean those up with [`deleteCache({ kvCacheKey })`](/sdk/v0.23/reference/api#deletecache).

On a cached turn, `stats.promptTokens` counts only the prompt tokens the model processed on that turn. Tokens reused from the cache are not included, so a warm turn reports far fewer than the full prompt. `stats.cacheTokens` is the number of tokens held in the cache after the turn. Before v0.22, `promptTokens` counted the whole prompt.

There is no separate `kvCacheKey` completion parameter — when you want a caller-managed cache, pass the key string directly as `kvCache`:

```js
completion({
  modelId,
  history,
  stream: true,
  kvCache: "user-123-session-a",
});
```

On hybrid and recurrent models (Qwen3.5, Jamba, DeepSeek V4) and on sliding-window models (Gemma 3/4, gpt-oss), a turn under a caller-managed `kvCache` key whose history diverges from the cache, such as the next turn under a thinking template or a regenerate, continues from a checkpoint instead of reprocessing the whole conversation. Each checkpoint costs host RAM, up to hundreds of MiB on larger models, which model fit checks do not count. Tune them at load time in `modelConfig`:

* `cache_checkpoints`: checkpoints kept per sequence (0–1024, default 1). Editing the user message k-th from the end without reprocessing needs `k + 1`; `0` turns checkpoints off.
* `cache_checkpoints_max_bytes`: byte budget for them per sequence, `0` (default) for unlimited.
* `cache_checkpoint_storage`: `'memory'` (default) or `'disk'`. With `'disk'`, set `cache_checkpoint_dir` to a writable directory; on mobile, use the app's cache directory.
* `cache_ram_mib`: host-RAM budget, in MiB, for cached conversations that are not running, so switching back to a caller-managed `kvCache` key skips reading its cache file. Default `0` (off).

#### Prepare the system prompt and tools ahead of time

Use `prefill: true` to evaluate a prompt before the first question arrives. It requires `kvCache: true` or a nonempty named key and finishes with stats, no answer and no tool calls. Await the priming run before sending the question:

```typescript
const history = [{ role: "system", content: "You are a helpful assistant." }];
await completion({ modelId, history, tools, stream: false, kvCache: "chat", prefill: true }).final;
const answer = await completion({
  modelId,
  history: [...history, { role: "user", content: "What is the weather in Lugano?" }],
  tools,
  stream: false,
  kvCache: "chat",
}).final;
```

Load the model with `modelConfig: { tools: true }` when passing tools. Resend the same system prompt and tools on the question so their cached prefix can be reused. A prefill rejects forced `generationParams.tool_choice` values (`"required"` or a named tool, including `"tool_search"`); use the default `"auto"` or `"none"` instead. Auto-cache priming saves the complete prompt history without appending an empty assistant message.

With `kvCache: true`, the first question moves the cache to the conversation's new history, so the prime serves one conversation. Use a named key for reuse across conversations, or prime each conversation separately.

Affected templates require `@qvac/llm-llamacpp` 0.64.1 or later. Full reuse is template-dependent; arbitrary custom templates may fall back to normal rendering. Cache priming is available through the library API, not the CLI OpenAI-compatible HTTP endpoints or model-load configuration.

#### Compare cached and uncached turns

The following script enables `kvCache: true` to speed up follow-up turns, and then compares it with `kvCache: false` on the same history:

<Tabs>
  <Tab value="js" label="JavaScript" default>
    <WrapCode>
      ```js file=<rootDir>/packages/sdk/dist/examples/kv-cache-example.js title="completion-kv-cache.js" lineNumbers
      import { completion, LLAMA_3_2_1B_INST_Q4_0, loadModel, unloadModel, VERBOSITY } from '@qvac/sdk';
      try {
          // Load the model
          const modelId = await loadModel({
              modelSrc: LLAMA_3_2_1B_INST_Q4_0,
              modelConfig: {
                  device: 'gpu',
                  ctx_size: 2048,
                  verbosity: VERBOSITY.ERROR
              }
          });
          console.log('▸ Testing KV Cache functionality...\n');
          // First conversation with auto-keyed cache enabled
          console.log('▸ First conversation (building cache for the next turn):');
          const history1 = [{ role: 'user', content: 'What is the capital of France?' }];
          const result1 = completion({
              modelId,
              history: history1,
              stream: true,
              kvCache: true
          }); // kvCache = true
          for await (const token of result1.tokenStream) {
              process.stdout.write(token);
          }
          const final1 = await result1.final;
          const stats1 = final1.stats;
          console.log(`\n▸ First completion stats: ${JSON.stringify(stats1)}\n`);
          // Continue conversation (should reuse the completed first-turn cache).
          console.log('▸ Continuing conversation (reusing previous turn cache):');
          const history2 = [
              { role: 'user', content: 'What is the capital of France?' },
              {
                  role: 'assistant',
                  content: final1.cacheableAssistantContent ?? final1.contentText
              },
              { role: 'user', content: 'What about Germany?' }
          ];
          // Auto-keyed caching should:
          // 1. Find the cache saved after turn 1 under [user, assistant]
          // 2. Load that cache and process only the new "What about Germany?" user turn
          // 3. Save the updated cache and rename it to include the new assistant response
          const result2 = completion({
              modelId,
              history: history2,
              stream: true,
              kvCache: true
          }); // kvCache = true
          for await (const token of result2.tokenStream) {
              process.stdout.write(token);
          }
          const stats2 = await result2.stats;
          console.log(`\n▸ Second completion stats: ${JSON.stringify(stats2)}\n`);
          // Compare with non-cached version
          console.log('▸ Same conversation without cache:');
          const result3 = completion({
              modelId,
              history: history2,
              stream: true,
              kvCache: false
          }); // kvCache = false
          for await (const token of result3.tokenStream) {
              process.stdout.write(token);
          }
          const stats3 = await result3.stats;
          console.log(`\n▸ Non-cached completion stats: ${JSON.stringify(stats3)}\n`);
          console.log('▸ KV Cache test completed!');
          await unloadModel({ modelId, clearStorage: false });
      }
      catch (error) {
          console.error('✖', error);
          process.exit(1);
      }
      ```
    </WrapCode>
  </Tab>

  <Tab value="ts" label="TypeScript">
    <WrapCode>
      ```ts file=<rootDir>/packages/sdk/examples/kv-cache-example.ts title="completion-kv-cache.ts" lineNumbers
      import { completion, LLAMA_3_2_1B_INST_Q4_0, loadModel, unloadModel, VERBOSITY } from '@qvac/sdk'

      try {
        // Load the model
        const modelId = await loadModel({
          modelSrc: LLAMA_3_2_1B_INST_Q4_0,
          modelConfig: {
            device: 'gpu',
            ctx_size: 2048,
            verbosity: VERBOSITY.ERROR
          }
        })

        console.log('▸ Testing KV Cache functionality...\n')

        // First conversation with auto-keyed cache enabled
        console.log('▸ First conversation (building cache for the next turn):')
        const history1 = [{ role: 'user', content: 'What is the capital of France?' }]

        const result1 = completion({
          modelId,
          history: history1,
          stream: true,
          kvCache: true
        }) // kvCache = true

        for await (const token of result1.tokenStream) {
          process.stdout.write(token)
        }

        const final1 = await result1.final
        const stats1 = final1.stats
        console.log(`\n▸ First completion stats: ${JSON.stringify(stats1)}\n`)

        // Continue conversation (should reuse the completed first-turn cache).
        console.log('▸ Continuing conversation (reusing previous turn cache):')
        const history2 = [
          { role: 'user', content: 'What is the capital of France?' },
          {
            role: 'assistant',
            content: final1.cacheableAssistantContent ?? final1.contentText
          },
          { role: 'user', content: 'What about Germany?' }
        ]

        // Auto-keyed caching should:
        // 1. Find the cache saved after turn 1 under [user, assistant]
        // 2. Load that cache and process only the new "What about Germany?" user turn
        // 3. Save the updated cache and rename it to include the new assistant response
        const result2 = completion({
          modelId,
          history: history2,
          stream: true,
          kvCache: true
        }) // kvCache = true

        for await (const token of result2.tokenStream) {
          process.stdout.write(token)
        }

        const stats2 = await result2.stats
        console.log(`\n▸ Second completion stats: ${JSON.stringify(stats2)}\n`)

        // Compare with non-cached version
        console.log('▸ Same conversation without cache:')
        const result3 = completion({
          modelId,
          history: history2,
          stream: true,
          kvCache: false
        }) // kvCache = false

        for await (const token of result3.tokenStream) {
          process.stdout.write(token)
        }

        const stats3 = await result3.stats
        console.log(`\n▸ Non-cached completion stats: ${JSON.stringify(stats3)}\n`)

        console.log('▸ KV Cache test completed!')

        await unloadModel({ modelId, clearStorage: false })
      } catch (error) {
        console.error('✖', error)
        process.exit(1)
      }
      ```
    </WrapCode>
  </Tab>

  <Tab value="python" label="Python">
    <WrapCode>
      ```python file=<rootDir>/packages/sdk-python/examples/kv_cache.py title="completion_kv_cache.py" lineNumbers
      """Python port of packages/sdk/examples/kv-cache-example.ts.

      `kv_cache=True` asks the SDK to key the cache off the conversation itself, so
      a follow-up turn reprocesses only the new message. The three completions below
      show the effect: the first builds the cache, the second reuses it, and the
      third replays the same history with `kv_cache=False` to compare.

      Read `promptTokens` to see it work — on a warm turn it counts only the tokens
      the model actually processed, not the whole prompt. `cacheTokens` is what the
      cache holds after the turn.

      RUN: python examples/kv_cache.py
      """

      from __future__ import annotations

      import asyncio
      import sys

      from tetherto.qvac_sdk import Client, completion, load_model, unload_model
      from tetherto.qvac_sdk.models import LLAMA_3_2_1B_INST_Q4_0


      async def stream_turn(run, label):
          """Print the tokens as they arrive, then the turn's stats."""
          async for event in run.events:
              if event.type == "contentDelta":
                  sys.stdout.write(event.text)
                  sys.stdout.flush()
          final = await run.final
          # Dumped by alias so the keys read like the JS stats object.
          stats = (
              final.stats.model_dump_json(by_alias=True, exclude_none=True)
              if final.stats
              else "{}"
          )
          print(f"\n▸ {label} stats: {stats}\n")
          return final


      async def main() -> int:
          async with Client() as client:
              t = client.transport
              try:
                  model_id = await load_model(
                      t,
                      model_src=LLAMA_3_2_1B_INST_Q4_0,
                      model_config={
                          "device": "gpu",
                          "ctx_size": 2048,
                          "verbosity": 0,  # 0=ERROR, 1=WARN, 2=INFO, 3=DEBUG
                      },
                  )

                  print("▸ Testing KV Cache functionality...\n")

                  print("▸ First conversation (building cache for the next turn):")
                  history1 = [{"role": "user", "content": "What is the capital of France?"}]
                  run1 = completion(
                      t, model_id=model_id, history=history1, stream=True, kv_cache=True
                  )
                  final1 = await stream_turn(run1, "First completion")

                  # Continue conversation (should reuse the completed first-turn cache).
                  print("▸ Continuing conversation (reusing previous turn cache):")
                  history2 = [
                      {"role": "user", "content": "What is the capital of France?"},
                      {
                          "role": "assistant",
                          # The cacheable form drops any thinking block, so the
                          # history sent back matches what the cache was keyed on.
                          "content": final1.cacheable_assistant_content
                          or final1.content_text,
                      },
                      {"role": "user", "content": "What about Germany?"},
                  ]

                  # Auto-keyed caching should:
                  # 1. Find the cache saved after turn 1 under [user, assistant]
                  # 2. Load that cache and process only the new "What about Germany?" user turn
                  # 3. Save the updated cache and rename it to include the new assistant response
                  run2 = completion(
                      t, model_id=model_id, history=history2, stream=True, kv_cache=True
                  )
                  await stream_turn(run2, "Second completion")

                  # Compare with non-cached version
                  print("▸ Same conversation without cache:")
                  run3 = completion(
                      t, model_id=model_id, history=history2, stream=True, kv_cache=False
                  )
                  await stream_turn(run3, "Non-cached completion")

                  print("▸ KV Cache test completed!")

                  await unload_model(t, model_id)
              except Exception as error:
                  print(f"✖ {error}", file=sys.stderr)
                  return 1
          return 0


      if __name__ == "__main__":
          sys.exit(asyncio.run(main()))
      ```
    </WrapCode>
  </Tab>

  <Tab value="kotlin" label="Kotlin">
    <WrapCode>
      ```kotlin file=<rootDir>/packages/sdk-kotlin/examples/src/KvCache.kt title="KvCache.kt" lineNumbers
      // Kotlin port of packages/sdk/examples/kv-cache-example.ts.
      //
      // `QvacKvCache.Enabled(true)` asks the SDK to key the cache off the
      // conversation itself, so a follow-up turn reprocesses only the new message.
      // The three completions below show the effect: the first builds the cache, the
      // second reuses it, and the third replays the same history with
      // `QvacKvCache.Enabled(false)` to compare.
      //
      // Read `promptTokens` to see it work — on a warm turn it counts only the tokens
      // the model actually processed, not the whole prompt. `cacheTokens` is what the
      // cache holds after the turn.
      //
      // `QvacKvCache` is a sealed type: `Enabled(true/false)` for the auto-keyed
      // cache, or `Key("name")` for a named one, so an invalid mix cannot be built.
      // `QvacCompletionStats.raw` keeps the stats object exactly as the worker sent
      // it, so the printed keys read like the JS stats object.
      //
      // RUN: ./gradlew -p examples run -Pexample=KvCache

      import io.tether.qvac.sdk.QvacClient
      import io.tether.qvac.sdk.QvacCompletionFinal
      import io.tether.qvac.sdk.QvacCompletionOptions
      import io.tether.qvac.sdk.QvacCompletionRun
      import io.tether.qvac.sdk.QvacKvCache
      import io.tether.qvac.sdk.QvacMessage
      import io.tether.qvac.sdk.completion
      import io.tether.qvac.sdk.generated.Models
      import io.tether.qvac.sdk.jvm.JvmBareKitTransport
      import io.tether.qvac.sdk.models
      import io.tether.qvac.sdk.run
      import kotlinx.coroutines.runBlocking
      import kotlinx.serialization.json.buildJsonObject
      import kotlinx.serialization.json.put

      fun main(): Unit = runBlocking {
          val client = QvacClient(JvmBareKitTransport.connect())
          try {
              val model = Models.LLAMA_3_2_1B_INST_Q4_0
              val loaded = client.models.load(
                  source = model.src,
                  modelType = model.engine,
                  modelName = model.name,
                  modelConfig = buildJsonObject {
                      put("device", "gpu")
                      put("ctx_size", 2048)
                      put("verbosity", 0) // 0=ERROR, 1=WARN, 2=INFO, 3=DEBUG
                  },
              )
              val modelId = loaded.modelId ?: error(loaded.error ?: "loadModel failed")

              println("▸ Testing KV Cache functionality...\n")

              println("▸ First conversation (building cache for the next turn):")
              val history1 = listOf(QvacMessage.user("What is the capital of France?"))
              val run1 = client.completion.run(
                  modelId = modelId,
                  history = history1,
                  options = QvacCompletionOptions(stream = true, kvCache = QvacKvCache.Enabled(true)),
              )
              val final1 = streamTurn(run1, "First completion")

              // Continue conversation (should reuse the completed first-turn cache).
              println("▸ Continuing conversation (reusing previous turn cache):")
              val history2 = listOf(
                  QvacMessage.user("What is the capital of France?"),
                  // The cacheable form drops any thinking block, so the history sent
                  // back matches what the cache was keyed on.
                  QvacMessage.assistant(final1.cacheableAssistantContent.orEmpty().ifEmpty { final1.text }),
                  QvacMessage.user("What about Germany?"),
              )

              // Auto-keyed caching should:
              // 1. Find the cache saved after turn 1 under [user, assistant]
              // 2. Load that cache and process only the new "What about Germany?" user turn
              // 3. Save the updated cache and rename it to include the new assistant response
              val run2 = client.completion.run(
                  modelId = modelId,
                  history = history2,
                  options = QvacCompletionOptions(stream = true, kvCache = QvacKvCache.Enabled(true)),
              )
              streamTurn(run2, "Second completion")

              // Compare with non-cached version
              println("▸ Same conversation without cache:")
              val run3 = client.completion.run(
                  modelId = modelId,
                  history = history2,
                  options = QvacCompletionOptions(stream = true, kvCache = QvacKvCache.Enabled(false)),
              )
              streamTurn(run3, "Non-cached completion")

              println("▸ KV Cache test completed!")

              client.models.unload(modelId)
          } finally {
              client.close()
          }
      }

      /** Print the tokens as they arrive, then the turn's stats. */
      private suspend fun streamTurn(run: QvacCompletionRun, label: String): QvacCompletionFinal {
          run.tokens.collect { print(it) }
          val final = run.final.await()
          println("\n▸ $label stats: ${final.stats?.raw ?: "{}"}\n")
          return final
      }
      ```
    </WrapCode>
  </Tab>
</Tabs>

### Speculative decoding

Speculative decoding drafts several tokens per step and has the model verify the whole draft in one decode. The model checks every drafted token, so output keeps the distribution of plain decoding. Enable it at load time with `modelConfig.spec_type`:

* `"draft-mtp"` — drafts with multi-token-prediction (MTP) layers: the model's own, for example `QWEN3_5_0_8B_MTP_Q4_K_M` (also 2B, 4B and 9B) or `QWEN3_8_27B_MULTIMODAL_UD_Q4_K_XL`, or an external MTP head given as `modelConfig.draftModelSrc`, for example `QWEN3_5_0_8B_MTP_Q4_K_M` for `QWEN3_5_0_8B_MULTIMODAL_Q4_K_M`, or a Gemma 4 assistant for its Gemma 4 model. The head, when given, otherwise the model, must have MTP layers, and a head must match the model (hidden width and vocabulary), or the load fails. A head is placed like the model, and model fit checks count its memory, except a Gemma 4 assistant's.
* `"draft-dflash"` — drafts with a separate DFlash model, given as `modelConfig.draftModelSrc`, which it requires: for example `QWEN3_5_4B_DFLASH_Q8_0` with `QWEN3_5_4B_MULTIMODAL_Q4_K_M`. The draft model is placed like the model, and model fit checks do not count its memory, so leave room for it.
* `"none"` (default) — one token per step.

`spec_draft_n_max` (1–64, default 3) caps the draft length and is lowered at load, with a warning, to fit the batch. `spec_draft_n_min` (0–64, default 0) drops shorter drafts, and `spec_draft_p_min` (0–1, default 0) is the minimum draft-head probability for a token to extend the draft. These three require `spec_type` `"draft-mtp"` or `"draft-dflash"`. Speculative decoding works with multimodal prompts, `parallel` and `kvCache`.

When it is on, `stats.draftTokens` counts the drafted tokens and `stats.draftAcceptedTokens` the ones the model kept. For [`batchCompletion()`](/sdk/v0.23/reference/api#batchcompletion) they are batch-level, on `run.stats`. The gain depends on the device: with several concurrent sequences, the larger verification batches can cost more than they save.

```js
const modelId = await loadModel({
  modelSrc: QWEN3_5_4B_MULTIMODAL_Q4_K_M,
  modelType: "llm",
  modelConfig: {
    spec_type: "draft-dflash",
    draftModelSrc: QWEN3_5_4B_DFLASH_Q8_0,
  },
});
```

<Callout type="success">
  **Tip:** all examples throughout this documentation are self-contained and runnable. For instructions on how to run them, see the [JS/TS quickstart](/sdk/v0.23/js-ts-sdk#quickstart), the [Python quickstart](/sdk/v0.23/python-sdk#quickstart), or the [Kotlin quickstart](/sdk/v0.23/kotlin-sdk#quickstart).
</Callout>
