# Batch processing (/ai-capabilities/batch-processing)



## Overview

**What it does:** `batchCompletion()` runs **multiple prompts through a single loaded model in one call**, instead of firing a separate [`completion()`](/reference/api#completion) call per prompt.

**When to use it:** you have several prompts to run against the same model and want them handled together as one job.

**Why use it:**

* **Optimizes resources:** all prompts share one loaded model — you don't load the model multiple times to run prompts side by side.
* **Lower total run time:** the prompts run together rather than one after another, so the whole set finishes faster than the equivalent sequence of `completion()` calls.

To use it, load the model with `modelConfig.parallel >= 2` (this is what lets it handle more than one prompt at a time), then pass an array of `prompts` — each with its own `history` and, optionally, `generationParams`, `responseFormat`, `tools`, `mcp`, and multimodal `attachments`.

<Callout type="info">
  Batch processing is specific to **LLM text generation**. It does not apply to other capabilities such as embeddings or transcription.
</Callout>

## Functions

Use the following sequence of function calls:

1. [`loadModel()`](/reference/api#loadmodel) — load with `modelConfig.parallel >= 2`.
2. [`batchCompletion()`](/reference/api#batchcompletion)
3. [`unloadModel()`](/reference/api#unloadmodel)

For how to use each function, see [SDK — API reference](/reference/api/).

## Models

You can load any [`llama.cpp`](https://github.com/ggml-org/llama.cpp)-compatible text-generation/chat model (same models as [Text generation](/ai-capabilities/text-generation)). Model file format: `*.gguf`.

* Per-prompt `tools`/`mcp` require a tool-capable model loaded with `modelConfig.tools: true`.
* Per-prompt `attachments` require a multimodal-capable model plus its matching `projectionModelSrc` — see [Multimodal](/ai-capabilities/multimodal).
* For models available as constants, see [SDK — Models](/introduction#models).

## Features

`batchCompletion()` returns a `BatchCompletionRun` that exposes several ways to consume the batch:

* **Merged event stream:** `run.events` is a single async iterable yielding `{ id, event }` interleaved across all prompts, where `event` is the same discriminated `CompletionEvent` (`contentDelta`, `thinkingDelta`, `toolCall`, `completionStats`, `completionDone`, etc.) used by `completion()`.
* **Per-prompt streams:** `run.byId(id)` returns `{ events, final }` scoped to one prompt — its own event iterable plus an aggregated `final` promise.
* **Assigned ids:** `run.ids` resolves to the addon-assigned ids in prompt order.
* **Ordered results:** `run.results` resolves to `BatchCompletionResult[]` in prompt order. It is **all-or-nothing for stream-level failures** — if the batch handler throws (e.g. a `ContextOverflowError`), the whole run rejects and no per-prompt finals can be recovered. Graceful terminal events such as cancellation still settle each `byId(id).final` according to that id's terminal state.
* **Batch-level stats:** `run.stats` resolves to a single `CompletionStats | undefined` aggregated across the whole batch (e.g. total tokens), rather than per prompt.
* **Per-prompt tools and MCP:** each prompt can carry its own `tools` and/or `mcp` clients, resolved client-side (including MCP tool discovery) before submission. Tool calls are routed back to the correct prompt.
* **Cancellable:** like any other request — see [Cancellation](/runtime/cancellation).

<Callout type="info">
  Validation rules: prompt `id`s must be unique within a batch, and a prompt's `responseFormat` (`json_object` / `json_schema`) cannot be combined with `tools` or `mcp` (tools already constrain output via their parameter schema).
</Callout>

## Example

The following script loads a model with `parallel: 4`, submits several prompts in one `batchCompletion()` call, consumes the merged event stream, and reads the ordered results and batch-level stats:

<Tabs>
  <Tab value="js" label="JavaScript" default>
    <WrapCode>
      ```js file=<rootDir>/packages/sdk/dist/examples/batch-completion.js title="batch-completion.js" lineNumbers
      /**
       * Batch completion — continuous batching over a single loaded LLM.
       *
       * `batchCompletion()` submits multiple prompts to one loaded model in a single
       * call so the llama.cpp addon can decode the active sequences together. Load the
       * model with `modelConfig.parallel >= 2` to make the concurrent slots available.
       *
       * The returned `BatchCompletionRun` exposes:
       *  - `events`     — a merged `AsyncIterable<{ id, event }>` across all prompts.
       *  - `byId(id)`   — per-prompt `events` and a `final` aggregation for one prompt.
       *  - `ids`        — a `Promise<string[]>` of addon-assigned ids in prompt order.
       *  - `results`    — a `Promise<BatchCompletionResult[]>` in prompt order. It is
       *                   ALL-OR-NOTHING for stream-level failures: if the batch
       *                   handler throws (e.g. a context overflow) the whole run
       *                   rejects and no per-prompt finals can be recovered. Graceful
       *                   terminal events such as cancellation still settle each
       *                   `byId(id).final` according to that id's terminal state.
       *  - `stats`      — a `Promise<CompletionStats | undefined>`. Stats are
       *                   BATCH-LEVEL: the addon aggregates decode metrics across the
       *                   whole batch (e.g. `avgConcurrentSeq`, total tokens), so they
       *                   surface once here rather than per prompt. Per-prompt
       *                   `final.stats` is intentionally left undefined.
       *
       * Each prompt may additionally carry (not shown here to keep the run fast/cheap):
       *  - `tools` / `mcp`  — per-prompt tools; handlers attach only to tool calls
       *                       produced by that prompt. Requires a tool-capable model
       *                       loaded with `modelConfig.tools: true` (e.g. Qwen). See
       *                       the `tools/` examples for the tool-calling surface.
       *  - `attachments`    — the same multimodal attachments `completion()` supports,
       *                       when the loaded model has a projection model.
       *  - `responseFormat` — `json_object` / `json_schema` (cannot be combined with
       *                       `tools` or `mcp`, which already constrain output).
       *
       * Run from packages/sdk:
       *   bun run examples/batch-completion.ts
       */
      import { batchCompletion, loadModel, unloadModel, LLAMA_3_2_1B_INST_Q4_0 } from '@qvac/sdk';
      try {
          // `parallel: 4` opens the concurrent decode slots continuous batching needs.
          const modelId = await loadModel({
              modelSrc: LLAMA_3_2_1B_INST_Q4_0,
              modelType: 'llm',
              modelConfig: { ctx_size: 4096, parallel: 4 },
              onProgress: (p) => {
                  const mb = (n) => (n / 1e6).toFixed(1);
                  const line = `▸ Downloading ${p.percentage.toFixed(0)}% (${mb(p.downloaded)}/${mb(p.total)} MB)`;
                  process.stderr.write(process.stderr.isTTY ? `\r${line}` : `${line}\n`);
                  if (p.percentage >= 100)
                      process.stderr.write('\n');
              }
          });
          console.log(`▸ Model loaded: ${modelId}`);
          const run = batchCompletion({
              modelId,
              prompts: [
                  {
                      id: 'cherry',
                      history: [{ role: 'user', content: 'Reply with only the word CHERRY.' }],
                      generationParams: { temp: 0, seed: 42, predict: 16 }
                  },
                  {
                      id: 'banana',
                      history: [{ role: 'user', content: 'Reply with only the word BANANA.' }],
                      generationParams: { temp: 0, seed: 42, predict: 16 }
                  },
                  {
                      id: 'grape',
                      history: [{ role: 'user', content: 'Reply with only the word GRAPE.' }],
                      generationParams: { temp: 0, seed: 42, predict: 16 }
                  }
              ]
          });
          console.log('\n▸ Merged event stream (interleaved across prompts):');
          for await (const { id, event } of run.events) {
              if (event.type === 'contentDelta') {
                  process.stdout.write(`[${id}] ${event.text}`);
              }
          }
          // `results` is ordered by prompt and all-or-nothing on stream-level failure.
          const results = await run.results;
          console.log('\n\n▸ Ordered results:');
          for (const { id, final } of results) {
              console.log(`  ▸ ${id}: ${final.contentText.replace(/\s+/g, ' ').trim()}`);
          }
          // Per-prompt aggregation is also available directly via `byId(id)`.
          const ids = await run.ids;
          const firstFinal = await run.byId(ids[0]).final;
          console.log(`\n▸ byId("${ids[0]}").final -> ${firstFinal.contentText.trim()}`);
          // Batch-level stats (a single CompletionStats for the whole run, or undefined).
          const stats = await run.stats;
          if (stats) {
              console.log(`\n▸ Batch stats: ${stats.tokensPerSecond?.toFixed(1)} tok/s across the batch`);
          }
          await unloadModel({ modelId, clearStorage: false });
          process.exit(0);
      }
      catch (error) {
          console.error('✖', error);
          process.exit(1);
      }
      ```
    </WrapCode>
  </Tab>

  <Tab value="ts" label="TypeScript">
    <WrapCode>
      ```ts file=<rootDir>/packages/sdk/examples/batch-completion.ts title="batch-completion.ts" lineNumbers
      /**
       * Batch completion — continuous batching over a single loaded LLM.
       *
       * `batchCompletion()` submits multiple prompts to one loaded model in a single
       * call so the llama.cpp addon can decode the active sequences together. Load the
       * model with `modelConfig.parallel >= 2` to make the concurrent slots available.
       *
       * The returned `BatchCompletionRun` exposes:
       *  - `events`     — a merged `AsyncIterable<{ id, event }>` across all prompts.
       *  - `byId(id)`   — per-prompt `events` and a `final` aggregation for one prompt.
       *  - `ids`        — a `Promise<string[]>` of addon-assigned ids in prompt order.
       *  - `results`    — a `Promise<BatchCompletionResult[]>` in prompt order. It is
       *                   ALL-OR-NOTHING for stream-level failures: if the batch
       *                   handler throws (e.g. a context overflow) the whole run
       *                   rejects and no per-prompt finals can be recovered. Graceful
       *                   terminal events such as cancellation still settle each
       *                   `byId(id).final` according to that id's terminal state.
       *  - `stats`      — a `Promise<CompletionStats | undefined>`. Stats are
       *                   BATCH-LEVEL: the addon aggregates decode metrics across the
       *                   whole batch (e.g. `avgConcurrentSeq`, total tokens), so they
       *                   surface once here rather than per prompt. Per-prompt
       *                   `final.stats` is intentionally left undefined.
       *
       * Each prompt may additionally carry (not shown here to keep the run fast/cheap):
       *  - `tools` / `mcp`  — per-prompt tools; handlers attach only to tool calls
       *                       produced by that prompt. Requires a tool-capable model
       *                       loaded with `modelConfig.tools: true` (e.g. Qwen). See
       *                       the `tools/` examples for the tool-calling surface.
       *  - `attachments`    — the same multimodal attachments `completion()` supports,
       *                       when the loaded model has a projection model.
       *  - `responseFormat` — `json_object` / `json_schema` (cannot be combined with
       *                       `tools` or `mcp`, which already constrain output).
       *
       * Run from packages/sdk:
       *   bun run examples/batch-completion.ts
       */

      import { batchCompletion, loadModel, unloadModel, LLAMA_3_2_1B_INST_Q4_0 } from '@qvac/sdk'

      try {
        // `parallel: 4` opens the concurrent decode slots continuous batching needs.
        const modelId = await loadModel({
          modelSrc: LLAMA_3_2_1B_INST_Q4_0,
          modelType: 'llm',
          modelConfig: { ctx_size: 4096, parallel: 4 },
          onProgress: (p) => {
            const mb = (n: number) => (n / 1e6).toFixed(1)
            const line = `▸ Downloading ${p.percentage.toFixed(0)}% (${mb(p.downloaded)}/${mb(p.total)} MB)`
            process.stderr.write(process.stderr.isTTY ? `\r${line}` : `${line}\n`)
            if (p.percentage >= 100) process.stderr.write('\n')
          }
        })
        console.log(`▸ Model loaded: ${modelId}`)

        const run = batchCompletion({
          modelId,
          prompts: [
            {
              id: 'cherry',
              history: [{ role: 'user', content: 'Reply with only the word CHERRY.' }],
              generationParams: { temp: 0, seed: 42, predict: 16 }
            },
            {
              id: 'banana',
              history: [{ role: 'user', content: 'Reply with only the word BANANA.' }],
              generationParams: { temp: 0, seed: 42, predict: 16 }
            },
            {
              id: 'grape',
              history: [{ role: 'user', content: 'Reply with only the word GRAPE.' }],
              generationParams: { temp: 0, seed: 42, predict: 16 }
            }
          ]
        })

        console.log('\n▸ Merged event stream (interleaved across prompts):')
        for await (const { id, event } of run.events) {
          if (event.type === 'contentDelta') {
            process.stdout.write(`[${id}] ${event.text}`)
          }
        }

        // `results` is ordered by prompt and all-or-nothing on stream-level failure.
        const results = await run.results
        console.log('\n\n▸ Ordered results:')
        for (const { id, final } of results) {
          console.log(`  ▸ ${id}: ${final.contentText.replace(/\s+/g, ' ').trim()}`)
        }

        // Per-prompt aggregation is also available directly via `byId(id)`.
        const ids = await run.ids
        const firstFinal = await run.byId(ids[0]!).final
        console.log(`\n▸ byId("${ids[0]}").final -> ${firstFinal.contentText.trim()}`)

        // Batch-level stats (a single CompletionStats for the whole run, or undefined).
        const stats = await run.stats
        if (stats) {
          console.log(`\n▸ Batch stats: ${stats.tokensPerSecond?.toFixed(1)} tok/s across the batch`)
        }

        await unloadModel({ modelId, clearStorage: false })
        process.exit(0)
      } catch (error) {
        console.error('✖', error)
        process.exit(1)
      }
      ```
    </WrapCode>
  </Tab>
</Tabs>

<Callout type="success">
  **Tip:** all examples throughout this documentation are self-contained and runnable. For instructions on how to run them, see [SDK quickstart](/quickstart).
</Callout>