Multimodal

Overview

Text generation supports multimodal prompts. Multimodal lets you attach media files to inputs for completion(). You can include multiple attachments in the same request (e.g., to compare two images).

Compared to text-only completion inference, the key differences are:

• You must load a multimodal-capable LLM and its matching projectionModelSrc via loadModel().
• Your history messages can include attachments: [{ path: "/path/to/image.jpg" }] (the file must exist on disk).
• Aside from attachments, you still call completion({ modelId, history, stream }) the same way and consume the same streaming output.

Functions

Use the following sequence of function calls:

1. loadModel()
2. completion()
3. unloadModel()

For how to use each function, see SDK — API reference.

Models

You should load two models:

• a llama.cpp-compatible multimodal-capable LLM. Model file format: *.gguf; and
• a matching projection model (mmproj-*.gguf). Model file format: *.gguf.

Recommended pairs:

• VisionPsy Nano — one weights + mmproj pair per variant (choose the pair for the variant you want):
  • Flash: VISIONPSY_NANO_460M_MULTIMODAL_Q8_0 (or ..._Q4_K_M) + MMPROJ_VISIONPSY_NANO_460M_MULTIMODAL_Q8_0.
  • Base: VISIONPSY_NANO_460M_MULTIMODAL_Q8_0_1 (or ..._Q4_K_M_1) + MMPROJ_VISIONPSY_NANO_460M_MULTIMODAL_Q8_0_1.
  • Note: the un-suffixed constants are Flash; the _1-suffixed constants are base.
• Qwen2.5-Omni + mmproj-* (or Qwen3-VL + mmproj-*)
• SmolVLM2 + mmproj-*

For models available as constants, see SDK — Models.

VisionPsy Nano: base vs. Flash

VisionPsy Nano ships in two variants — base and Flash — as separate weights + mmproj pairs (see the constants above). Pick a variant by loading its matching pair; the flag modelConfig.image_no_upscale supplies the preprocessing rule the two mmprojs cannot express through their (identical) metadata, so you must set it to match the pair you loaded:

• Base pair: leave image_no_upscale unset (the field is optional; unset defers to the mmproj's own value).
• Flash pair: set image_no_upscale: 'on'.

Loading Flash weights with image_no_upscale unset — or base weights with image_no_upscale: 'on' — passes load validation (both mmprojs declare the same preproc_image_size), but silently uses the wrong preprocessing rule and degrades quality.

image_no_upscale is read only by the idefics3-style preprocessor, and it takes effect only when the loaded mmproj declares clip.vision.preproc_image_size. In practice, that means the VisionPsy pairs above — on the other recommended pairs the flag either warns and is ignored (Qwen2.5-Omni / Qwen3-VL) or is accepted but inert (SmolVLM2, whose published mmproj declares no preproc_image_size). Treat image_no_upscale as a VisionPsy-only key.

Example

The following script shows an example of multimodal completion with one image (and optionally two):

import { completion, loadModel, SMOLVLM2_500M_MULTIMODAL_Q8_0, MMPROJ_SMOLVLM2_500M_MULTIMODAL_Q8_0, unloadModel } from '@qvac/sdk';
if (process.argv.length < 3) {
    console.error(`▸ Specify an image file path as the first argument and a second image file path as the second (optional) argument`);
    process.exit(1);
}
try {
    // const modelPath = args[modelIndex + 1]!;
    const imageFilePath = process.argv[2];
    // Load the main model with projection in a single step
    const modelId = await loadModel({
        modelSrc: SMOLVLM2_500M_MULTIMODAL_Q8_0,
        modelConfig: {
            ctx_size: 1024,
            projectionModelSrc: MMPROJ_SMOLVLM2_500M_MULTIMODAL_Q8_0
        },
        onProgress: (p) => {
            const mb = (n) => (n / 1e6).toFixed(1);
            const line = `▸ Downloading ${p.percentage.toFixed(0)}% (${mb(p.downloaded)}/${mb(p.total)} MB)`;
            process.stderr.write(process.stderr.isTTY ? `\r${line}` : `${line}\n`);
            if (p.percentage >= 100)
                process.stderr.write('\n');
        }
    });
    //Using one particular media
    const history = [
        {
            role: 'user',
            content: "What's in this image?",
            attachments: [{ path: imageFilePath }]
        }
    ];
    const result = completion({ modelId, history, stream: true });
    for await (const token of result.tokenStream) {
        process.stdout.write(token);
    }
    const stats = await result.stats;
    console.log('\n▸ Performance Stats:', stats);
    console.log('▸ --------------------------------');
    //Using multiple media
    if (process.argv.length < 4) {
        console.log(`▸ Only one image provided, terminating`);
        process.exit(0);
    }
    const imageFilePath2 = process.argv[3];
    const history2 = [
        {
            role: 'user',
            content: 'Compare the two newspaper articles',
            attachments: [{ path: imageFilePath }, { path: imageFilePath2 }]
        }
    ];
    const result2 = completion({ modelId, history: history2, stream: true });
    for await (const token of result2.tokenStream) {
        process.stdout.write(token);
    }
    const stats2 = await result2.stats;
    console.log('\n▸ Performance Stats:', stats2);
    console.log('▸ --------------------------------');
    await unloadModel({ modelId, clearStorage: false });
}
catch (error) {
    console.error('✖', error);
    process.exit(1);
}