Multimodal embeddings
Embed text, images, and audio in a shared vector space for cross-media search and retrieval.
Overview
Multimodal embeddings use the Fabric LLM inference engine through @qvac/embed-llamacpp. Load EmbeddingGemma 2 using modelType: "llamacpp-embedding" to embed text, images, and audio in a shared vector space.
Pass an inputs array to embedBatch(). It returns embedding: number[][], with one 768-dimensional vector per item. Compare these vectors to retrieve images or audio using a text query.
For text-only models and the text input form, see Text embeddings.
Functions
Use the following sequence of function calls:
For how to use each function, see SDK API reference.
Models
Use EMBEDDINGGEMMA_2_Q8_0 for the model and MMPROJ_EMBEDDINGGEMMA_2_Q8_0 for its matching image and audio projector. Import both constants from @qvac/sdk, load the model with modelType: "llamacpp-embedding", and pass the projector through modelConfig.projectionModelSrc. Model and projector sources also accept local paths, HTTP(S) URLs, and registry URIs through the usual model download lifecycle.
For models available as constants, see SDK Models.
Inputs
Pass inputs to embedBatch(). Each item is a text string or an ordered array of text strings and { media: { type, value } } parts. Use type: "base64" for encoded image or audio bytes, or type: "filePath" for a path accessible to the inference worker. The result contains embedding: number[][], with one vector per input item, including a batch with only one item. Plain text still works with a loaded projector. Use embed({ modelId, input }) for one text or ordered text/media input. It returns embedding: number[]. embed({ text }) still works. embed({ inputs }) keeps its original return shape, but it is deprecated and logs a warning. Use embedBatch() instead. Choose one input field per call. Task prompts such as task: search result | query: remain text you supply.
Media requires a projector. Images include JPEG, PNG and BMP; audio supports WAV, MP3 and FLAC. WebP, AAC/M4A, Ogg and video are unsupported.
Model configuration
ctxSizesets the runtime context size in tokens. Omitted, it uses the trained context size. Set it explicitly to bound memory for long-context models.batchSizedefaults to1024. A mixed input's text and media tokens must fit together in this budget and inctxSize. A large EmbeddingGemma 2 image can take 1120 tokens; the example uses2048for both settings.imageMaxTokensandimageMinTokensset positive per-image token budgets. Omitted, they keep the projector defaults. The minimum must not exceed the maximum.mmprojNoAudio: trueskips loading the audio encoder and rejects audio input. The default isfalse.mmprojUseGpuoverrides projector placement. Omitted, it uses the addon's device policy. It is ignored when the model runs on CPU.imageMaxMegapixelslimits image size before decoding. The default is50.audioMaxSecondslimits audio duration before decoding. The default is600.
Projector options require projectionModelSrc. A memory assessment with a projector is unsupported because the fitter does not count projector memory. Each model processes one request at a time; an overlapping call rejects as busy. Await the current call before starting another. Cancellation uses the existing cancel() API and applies to the model's active job.
Example
Save the example as embed-multimodal.ts and pass an image and an audio file:
npx tsx embed-multimodal.ts image.jpg audio.wavThe following script embeds text, an image, and audio with EmbeddingGemma 2 in one shared vector space:
// Embed text, an image, and audio with EmbeddingGemma 2 in one shared vector space.
// Usage: npx tsx embed-multimodal.ts [image.jpg] [audio.wav]
// Defaults use image/test.jpg and audio/sample-16khz.wav from the source repository.
import { close, embedBatch, EMBEDDINGGEMMA_2_Q8_0, loadModel, MMPROJ_EMBEDDINGGEMMA_2_Q8_0, unloadModel } from '@qvac/sdk';
import { resolve } from 'node:path';
import { fileURLToPath } from 'node:url';
// EmbeddingGemma 2 normalizes its embeddings, so their dot product is cosine similarity.
function cosineSimilarity(vecA, vecB) {
let dotProduct = 0;
for (let i = 0; i < vecA.length; i++) {
dotProduct += vecA[i] * vecB[i];
}
return dotProduct;
}
let modelId;
try {
const [imagePath = fileURLToPath(new URL('./image/test.jpg', import.meta.url)), audioPath = fileURLToPath(new URL('./audio/sample-16khz.wav', import.meta.url))] = process.argv.slice(2);
console.error('▸ Loading EmbeddingGemma 2 and its projector');
modelId = await loadModel({
modelType: 'llamacpp-embedding',
modelSrc: EMBEDDINGGEMMA_2_Q8_0,
modelConfig: {
projectionModelSrc: MMPROJ_EMBEDDINGGEMMA_2_Q8_0,
ctxSize: 2048,
batchSize: 2048
},
onProgress: (p) => {
const mb = (n) => (n / 1e6).toFixed(1);
const line = `▸ Downloading ${p.percentage.toFixed(0)}% (${mb(p.downloaded)}/${mb(p.total)} MB)`;
process.stderr.write(process.stderr.isTTY ? `\r${line}` : `${line}\n`);
if (p.percentage >= 100)
process.stderr.write('\n');
}
});
const { embedding, stats } = await embedBatch({
modelId,
inputs: [
'task: search result | query: a newspaper report about the Titanic',
['title: none | text: ', { media: { type: 'filePath', value: resolve(imagePath) } }],
[{ media: { type: 'filePath', value: resolve(audioPath) } }]
]
});
const [textEmbedding, imageEmbedding, audioEmbedding] = embedding;
if (!textEmbedding || !imageEmbedding || !audioEmbedding) {
throw new Error('Expected text, image, and audio embeddings');
}
const labels = ['Text query', `Image: ${imagePath}`, `Audio: ${audioPath}`];
for (const [index, values] of embedding.entries()) {
console.log(labels[index]);
console.log('Embedding dimensions:', values.length);
console.log('First 10 values:', values.slice(0, 10));
}
console.log('Text/image similarity:', cosineSimilarity(textEmbedding, imageEmbedding).toFixed(4));
console.log('Text/audio similarity:', cosineSimilarity(textEmbedding, audioEmbedding).toFixed(4));
console.log('Higher scores indicate a closer match to the text query.');
if (stats)
console.log('Performance stats:', stats);
}
catch (error) {
console.error('✖', error);
process.exitCode = 1;
}
finally {
if (modelId) {
try {
await unloadModel({ modelId });
}
catch (error) {
console.error('✖ Failed to unload model:', error);
process.exitCode = 1;
}
}
try {
await close();
}
catch (error) {
console.error('✖ Failed to close SDK connection:', error);
process.exitCode = 1;
}
}The Python client supports this capability through the same worker. A dedicated Python example is not yet published. See the Python SDK for the API surface.
Tip: all examples throughout this documentation are self-contained and runnable. For instructions on how to run them, see the JS/TS quickstart, the Python quickstart, or the Kotlin quickstart.