---
title: "Multimodal"
canonical: https://docs.qvac.tether.io/sdk/v0.23/ai-capabilities/multimodal/
collection: "SDK"
package: "@qvac/sdk"
line: v0.23
current_line: false
---

# Multimodal (/sdk/v0.23/ai-capabilities/multimodal)



## Overview

[Text generation](/sdk/v0.23/ai-capabilities/text-generation) supports multimodal prompts. Multimodal lets you attach media files to inputs for [`completion()`](/sdk/v0.23/reference/api#completion). You can include multiple attachments in the same request (e.g., to compare two images).

Compared to text-only completion inference, the key differences are:

* You must load a multimodal-capable LLM **and** its matching `projectionModelSrc` via [`loadModel()`](/sdk/v0.23/reference/api#loadmodel).
* Your `history` messages can include `attachments: [{ path: "/path/to/image.jpg" }]` (the file must exist on disk).
* Images larger than 50 megapixels (width × height) are rejected before decoding. To accept larger images, raise the limit at load time with `modelConfig['image-max-megapixels']` (a positive integer, in millions of pixels).
* Aside from attachments, you still call `completion({ modelId, history, stream })` the same way and consume the same streaming output.

## Functions

Use the following sequence of function calls:

1. [`loadModel()`](/sdk/v0.23/reference/api#loadmodel)
2. [`completion()`](/sdk/v0.23/reference/api#completion)
3. [`unloadModel()`](/sdk/v0.23/reference/api#unloadmodel)

For how to use each function, see [SDK — API reference](/sdk/v0.23/reference/api/).

## Models

You should load two models:

* a `llama.cpp`-compatible multimodal-capable LLM. Model file format: `*.gguf`; and
* a matching projection model (`mmproj-*.gguf`). Model file format: `*.gguf`.

Recommended pairs:

* VisionPsy Nano — one weights + `mmproj` pair per variant (choose the pair for the variant you want):
  * Flash: `VISIONPSY_NANO_460M_MULTIMODAL_Q8_0` (or `..._Q4_K_M`) + `MMPROJ_VISIONPSY_NANO_460M_MULTIMODAL_Q8_0`.
  * Base: `VISIONPSY_NANO_460M_MULTIMODAL_Q8_0_1` (or `..._Q4_K_M_1`) + `MMPROJ_VISIONPSY_NANO_460M_MULTIMODAL_Q8_0_1`.
  * Note: the un-suffixed constants are Flash; the `_1`-suffixed constants are base.
* Qwen2.5-Omni + mmproj-\* (or Qwen3-VL + mmproj-\*)
* SmolVLM2 + mmproj-\*

For models available as constants, see [SDK — Models](/sdk/v0.23/#models).

### VisionPsy Nano: base vs. Flash

VisionPsy Nano ships in two variants — base and Flash — as separate weights + `mmproj` pairs (see the constants above). Pick a variant by loading its matching pair; the flag `modelConfig.image_no_upscale` supplies the preprocessing rule the two mmprojs cannot express through their (identical) metadata, so you must set it to match the pair you loaded:

* Base pair: leave `image_no_upscale` unset (the field is optional; unset defers to the mmproj's own value).
* Flash pair: set `image_no_upscale: 'on'`.

Loading Flash weights with `image_no_upscale` unset — or base weights with `image_no_upscale: 'on'` — passes load validation (both mmprojs declare the same `preproc_image_size`), but silently uses the wrong preprocessing rule and degrades quality.

`image_no_upscale` is read only by the idefics3-style preprocessor, and it takes effect only when the loaded `mmproj` declares `clip.vision.preproc_image_size`. In practice, that means the VisionPsy pairs above — on the other recommended pairs the flag either warns and is ignored (Qwen2.5-Omni / Qwen3-VL) or is accepted but inert (SmolVLM2, whose published `mmproj` declares no `preproc_image_size`). Treat `image_no_upscale` as a VisionPsy-only key.

## Example

The following script shows an example of multimodal completion with one image (and optionally two):

<Tabs>
  <Tab value="js" label="JavaScript" default>
    <WrapCode>
      ```js file=<rootDir>/packages/sdk/dist/examples/llamacpp-multimodal.js title="multimodal.js" lineNumbers
      import { completion, loadModel, SMOLVLM2_500M_MULTIMODAL_Q8_0, MMPROJ_SMOLVLM2_500M_MULTIMODAL_Q8_0, unloadModel } from '@qvac/sdk';
      if (process.argv.length < 3) {
          console.error(`▸ Specify an image file path as the first argument and a second image file path as the second (optional) argument`);
          process.exit(1);
      }
      try {
          // const modelPath = args[modelIndex + 1]!;
          const imageFilePath = process.argv[2];
          // Load the main model with projection in a single step
          const modelId = await loadModel({
              modelSrc: SMOLVLM2_500M_MULTIMODAL_Q8_0,
              modelConfig: {
                  // An image is tiled into hundreds of context units before a single word
                  // of the prompt is counted, and the comparison below sends two at once.
                  ctx_size: 8192,
                  projectionModelSrc: MMPROJ_SMOLVLM2_500M_MULTIMODAL_Q8_0
              },
              onProgress: (p) => {
                  const mb = (n) => (n / 1e6).toFixed(1);
                  const line = `▸ Downloading ${p.percentage.toFixed(0)}% (${mb(p.downloaded)}/${mb(p.total)} MB)`;
                  process.stderr.write(process.stderr.isTTY ? `\r${line}` : `${line}\n`);
                  if (p.percentage >= 100)
                      process.stderr.write('\n');
              }
          });
          //Using one particular media
          const history = [
              {
                  role: 'user',
                  content: "What's in this image?",
                  attachments: [{ path: imageFilePath }]
              }
          ];
          const result = completion({ modelId, history, stream: true });
          for await (const token of result.tokenStream) {
              process.stdout.write(token);
          }
          const stats = await result.stats;
          console.log('\n▸ Performance Stats:', stats);
          console.log('▸ --------------------------------');
          //Using multiple media
          if (process.argv.length < 4) {
              console.log(`▸ Only one image provided, terminating`);
              process.exit(0);
          }
          const imageFilePath2 = process.argv[3];
          const history2 = [
              {
                  role: 'user',
                  content: 'Compare the two newspaper articles',
                  attachments: [{ path: imageFilePath }, { path: imageFilePath2 }]
              }
          ];
          const result2 = completion({ modelId, history: history2, stream: true });
          for await (const token of result2.tokenStream) {
              process.stdout.write(token);
          }
          const stats2 = await result2.stats;
          console.log('\n▸ Performance Stats:', stats2);
          console.log('▸ --------------------------------');
          // Ordered text and media: each image keeps its position in the turn, so the
          // text can sit next to the picture it refers to. Attachments can't do this,
          // because they are all sent before the text.
          const content = [
              { type: 'text', text: 'Image A: ' },
              { type: 'media', path: imageFilePath },
              { type: 'text', text: '\nImage B: ' },
              { type: 'media', path: imageFilePath2 },
              { type: 'text', text: '\nWhich image is a newspaper?' }
          ];
          const history3 = [{ role: 'user', content }];
          const result3 = completion({ modelId, history: history3, stream: true });
          for await (const token of result3.tokenStream) {
              process.stdout.write(token);
          }
          const stats3 = await result3.stats;
          console.log('\n▸ Performance Stats:', stats3);
          console.log('▸ --------------------------------');
          await unloadModel({ modelId, clearStorage: false });
      }
      catch (error) {
          console.error('✖', error);
          process.exit(1);
      }
      ```
    </WrapCode>
  </Tab>

  <Tab value="ts" label="TypeScript">
    <WrapCode>
      ```ts file=<rootDir>/packages/sdk/examples/llamacpp-multimodal.ts title="multimodal.ts" lineNumbers
      import {
        completion,
        loadModel,
        SMOLVLM2_500M_MULTIMODAL_Q8_0,
        MMPROJ_SMOLVLM2_500M_MULTIMODAL_Q8_0,
        unloadModel,
        type HistoryContentPart
      } from '@qvac/sdk'

      if (process.argv.length < 3) {
        console.error(
          `▸ Specify an image file path as the first argument and a second image file path as the second (optional) argument`
        )
        process.exit(1)
      }

      try {
        // const modelPath = args[modelIndex + 1]!;
        const imageFilePath = process.argv[2]!

        // Load the main model with projection in a single step
        const modelId = await loadModel({
          modelSrc: SMOLVLM2_500M_MULTIMODAL_Q8_0,
          modelConfig: {
            // An image is tiled into hundreds of context units before a single word
            // of the prompt is counted, and the comparison below sends two at once.
            ctx_size: 8192,
            projectionModelSrc: MMPROJ_SMOLVLM2_500M_MULTIMODAL_Q8_0
          },
          onProgress: (p) => {
            const mb = (n: number) => (n / 1e6).toFixed(1)
            const line = `▸ Downloading ${p.percentage.toFixed(0)}% (${mb(p.downloaded)}/${mb(p.total)} MB)`
            process.stderr.write(process.stderr.isTTY ? `\r${line}` : `${line}\n`)
            if (p.percentage >= 100) process.stderr.write('\n')
          }
        })

        //Using one particular media
        const history = [
          {
            role: 'user',
            content: "What's in this image?",
            attachments: [{ path: imageFilePath }]
          }
        ]
        const result = completion({ modelId, history, stream: true })

        for await (const token of result.tokenStream) {
          process.stdout.write(token)
        }

        const stats = await result.stats

        console.log('\n▸ Performance Stats:', stats)

        console.log('▸ --------------------------------')

        //Using multiple media
        if (process.argv.length < 4) {
          console.log(`▸ Only one image provided, terminating`)
          process.exit(0)
        }

        const imageFilePath2 = process.argv[3]!

        const history2 = [
          {
            role: 'user',
            content: 'Compare the two newspaper articles',
            attachments: [{ path: imageFilePath }, { path: imageFilePath2 }]
          }
        ]

        const result2 = completion({ modelId, history: history2, stream: true })

        for await (const token of result2.tokenStream) {
          process.stdout.write(token)
        }

        const stats2 = await result2.stats

        console.log('\n▸ Performance Stats:', stats2)

        console.log('▸ --------------------------------')

        // Ordered text and media: each image keeps its position in the turn, so the
        // text can sit next to the picture it refers to. Attachments can't do this,
        // because they are all sent before the text.
        const content: HistoryContentPart[] = [
          { type: 'text', text: 'Image A: ' },
          { type: 'media', path: imageFilePath },
          { type: 'text', text: '\nImage B: ' },
          { type: 'media', path: imageFilePath2 },
          { type: 'text', text: '\nWhich image is a newspaper?' }
        ]
        const history3 = [{ role: 'user', content }]

        const result3 = completion({ modelId, history: history3, stream: true })

        for await (const token of result3.tokenStream) {
          process.stdout.write(token)
        }

        const stats3 = await result3.stats

        console.log('\n▸ Performance Stats:', stats3)

        console.log('▸ --------------------------------')

        await unloadModel({ modelId, clearStorage: false })
      } catch (error) {
        console.error('✖', error)
        process.exit(1)
      }
      ```
    </WrapCode>
  </Tab>

  <Tab value="python" label="Python">
    <WrapCode>
      ```python file=<rootDir>/packages/sdk-python/examples/multimodal.py title="multimodal.py" lineNumbers
      """Multimodal completion — Python port of
      packages/sdk/examples/llamacpp-multimodal.ts.

      Load a multimodal LLM together with its matching projection model, then hang
      `attachments` off a user turn: one image first, then both images in a single
      request so the model can compare them.

      RUN: python examples/multimodal.py <image path> [<second image path>]
      """

      from __future__ import annotations

      import asyncio
      import sys

      from tetherto.qvac_sdk import Client, completion, load_model, unload_model
      from tetherto.qvac_sdk.models import (
          MMPROJ_SMOLVLM2_500M_MULTIMODAL_Q8_0,
          SMOLVLM2_500M_MULTIMODAL_Q8_0,
      )


      def print_progress(p) -> None:
          """Print model download progress; pass as `on_progress=` to `load_model`."""
          line = (
              f"▸ Downloading {p.percentage:.0f}% "
              f"({p.downloaded / 1e6:.1f}/{p.total / 1e6:.1f} MB)"
          )
          print(line, end="\r" if sys.stderr.isatty() else "\n", file=sys.stderr)
          if p.percentage >= 100:
              print(file=sys.stderr)


      async def main() -> int:
          if len(sys.argv) < 2:
              print(
                  "▸ Specify an image file path as the first argument and a second "
                  "image file path as the second (optional) argument",
                  file=sys.stderr,
              )
              return 1

          image_file_path = sys.argv[1]

          async with Client() as client:
              t = client.transport
              try:
                  # Load the main model with projection in a single step
                  model_id = await load_model(
                      t,
                      model_src=SMOLVLM2_500M_MULTIMODAL_Q8_0,
                      model_config={
                          # An image is tiled into hundreds of context units before a
                          # single word of the prompt is counted, and the comparison
                          # below sends two at once.
                          "ctx_size": 8192,
                          "projectionModelSrc": MMPROJ_SMOLVLM2_500M_MULTIMODAL_Q8_0,
                      },
                      on_progress=print_progress,
                  )

                  # Using one particular media
                  history = [
                      {
                          "role": "user",
                          "content": "What's in this image?",
                          "attachments": [{"path": image_file_path}],
                      }
                  ]
                  run = completion(t, model_id=model_id, history=history, stream=True)

                  async for event in run.events:
                      if event.type == "contentDelta":
                          sys.stdout.write(event.text)
                          sys.stdout.flush()

                  final = await run.final
                  print("\n▸ Performance Stats:", final.stats)
                  print("▸ --------------------------------")

                  # Using multiple media
                  if len(sys.argv) < 3:
                      print("▸ Only one image provided, terminating")
                      return 0

                  image_file_path2 = sys.argv[2]

                  history2 = [
                      {
                          "role": "user",
                          "content": "Compare the two newspaper articles",
                          "attachments": [
                              {"path": image_file_path},
                              {"path": image_file_path2},
                          ],
                      }
                  ]
                  run2 = completion(t, model_id=model_id, history=history2, stream=True)

                  async for event in run2.events:
                      if event.type == "contentDelta":
                          sys.stdout.write(event.text)
                          sys.stdout.flush()

                  final2 = await run2.final
                  print("\n▸ Performance Stats:", final2.stats)
                  print("▸ --------------------------------")

                  await unload_model(t, model_id)
              except Exception as error:
                  print(f"✖ {error}", file=sys.stderr)
                  return 1
          return 0


      if __name__ == "__main__":
          sys.exit(asyncio.run(main()))
      ```
    </WrapCode>
  </Tab>

  <Tab value="kotlin" label="Kotlin">
    <WrapCode>
      ```kotlin file=<rootDir>/packages/sdk-kotlin/examples/src/Multimodal.kt title="Multimodal.kt" lineNumbers
      // Kotlin port of packages/sdk/examples/llamacpp-multimodal.ts.
      //
      // Multimodal completion. Load a multimodal LLM together with its matching
      // projection model, then hang `attachments` off a user turn: one image first,
      // then both images in a single request so the model can compare them.
      //
      // The projection model goes in `modelConfig` as `projectionModelSrc`, the same
      // wire field as JS and Python; a `ModelConstant` is passed by its `.src`.
      // Attachments are typed `QvacAttachment` values on `QvacMessage.user()`.
      //
      // RUN: ./gradlew -p examples run -Pexample=Multimodal --args="<image path> [<second image path>]"

      import io.tether.qvac.sdk.QvacAttachment
      import io.tether.qvac.sdk.QvacClient
      import io.tether.qvac.sdk.QvacMessage
      import io.tether.qvac.sdk.QvacProgressEvent
      import io.tether.qvac.sdk.completion
      import io.tether.qvac.sdk.generated.ModelConstant
      import io.tether.qvac.sdk.generated.Models
      import io.tether.qvac.sdk.generated.schema.LoadModelResponse
      import io.tether.qvac.sdk.generated.schema.ModelProgressResponse
      import io.tether.qvac.sdk.jvm.JvmBareKitTransport
      import io.tether.qvac.sdk.models
      import io.tether.qvac.sdk.run
      import kotlin.system.exitProcess
      import kotlinx.coroutines.runBlocking
      import kotlinx.serialization.json.JsonObject
      import kotlinx.serialization.json.buildJsonObject
      import kotlinx.serialization.json.put

      fun main(args: Array<String>): Unit = runBlocking {
          if (args.isEmpty()) {
              System.err.println(
                  "▸ Specify an image file path as the first argument and a second " +
                      "image file path as the second (optional) argument",
              )
              exitProcess(1)
          }

          val imageFilePath = args[0]

          val client = QvacClient(JvmBareKitTransport.connect())
          try {
              // Load the main model with projection in a single step
              val modelId = client.loadWithProgress(
                  Models.SMOLVLM2_500M_MULTIMODAL_Q8_0,
                  modelConfig = buildJsonObject {
                      // An image is tiled into hundreds of context units before a
                      // single word of the prompt is counted, and the comparison
                      // below sends two at once.
                      put("ctx_size", 8192)
                      put("projectionModelSrc", Models.MMPROJ_SMOLVLM2_500M_MULTIMODAL_Q8_0.src)
                  },
              )

              // Using one particular media
              val history = listOf(QvacMessage.user("What's in this image?", listOf(QvacAttachment(imageFilePath))))
              val run = client.completion.run(modelId = modelId, history = history)

              run.tokens.collect { print(it) }

              val final = run.final.await()
              println("\n▸ Performance Stats: ${final.stats?.raw}")
              println("▸ --------------------------------")

              // Using multiple media
              if (args.size < 2) {
                  println("▸ Only one image provided, terminating")
                  return@runBlocking
              }

              val imageFilePath2 = args[1]

              val history2 = listOf(
                  QvacMessage.user(
                      "Compare the two newspaper articles",
                      listOf(QvacAttachment(imageFilePath), QvacAttachment(imageFilePath2)),
                  ),
              )
              val run2 = client.completion.run(modelId = modelId, history = history2)

              run2.tokens.collect { print(it) }

              val final2 = run2.final.await()
              println("\n▸ Performance Stats: ${final2.stats?.raw}")
              println("▸ --------------------------------")

              client.models.unload(modelId)
          } finally {
              client.close()
          }
      }

      /** Load [model], printing download progress, and return its model id. */
      private suspend fun QvacClient.loadWithProgress(model: ModelConstant, modelConfig: JsonObject? = null): String {
          var result: LoadModelResponse? = null
          models.loadWithProgress(model.src, model.engine, model.name, modelConfig = modelConfig).collect { event ->
              when (event) {
                  is QvacProgressEvent.Progress -> printProgress(event.value)
                  is QvacProgressEvent.Result -> result = event.value
              }
          }
          val loaded = checkNotNull(result) { "loadModel ended without a result" }
          return loaded.modelId ?: error(loaded.error ?: "loadModel failed")
      }

      private fun printProgress(p: ModelProgressResponse) {
          val line = "▸ Downloading %.0f%% (%.1f/%.1f MB)".format(p.percentage, p.downloaded / 1e6, p.total / 1e6)
          if (System.console() != null) System.err.print("\r$line") else System.err.println(line)
          if (p.percentage >= 100) System.err.println()
      }
      ```
    </WrapCode>
  </Tab>
</Tabs>

<Callout type="success">
  **Tip:** all examples throughout this documentation are self-contained and runnable. For instructions on how to run them, see the [JS/TS quickstart](/sdk/v0.23/js-ts-sdk#quickstart), the [Python quickstart](/sdk/v0.23/python-sdk#quickstart), or the [Kotlin quickstart](/sdk/v0.23/kotlin-sdk#quickstart).
</Callout>
