Image embeddings
Generate image descriptors for visual place recognition and retrieval with MegaLoc
Overview
Image embeddings use a GGML inference engine through @qvac/classification-ggml. Load MegaLoc with modelSrc: MEGALOC_F16 or modelSrc: MEGALOC_MIXED. For custom weights, use modelType: "ggml-image-embedding" with a GGUF URL or local file as modelSrc.
Pass one JPEG or PNG as input: { image: Uint8Array } to embed(). Use embedBatch() with inputs: [{ image: firstBytes }, { image: secondBytes }] for multiple images. For packed RGB bytes, also supply width, height, and channels: 3. Both dimensions must be integers from 1 to 16384, and the buffer must contain exactly width * height * 3 bytes. Omit all three fields for encoded images. The addon converts the image to RGB and resizes it to 322 by 322 pixels without cropping or padding.
The result contains embedding: number[], an L2-normalized descriptor with 8448 values, and optional stats. A dot product between two descriptors gives their cosine similarity. embedBatch() returns embedding: number[][] in input order, including batches with one image. Use these descriptors to compare images of places or build an image retrieval index. Text embeddings use the same embed() and embedBatch() functions with a text model. Class labels use classify().
MegaLoc also accepts one encoded image media part per input, using a base64 source or a file path accessible to the worker. It rejects text, audio, and inputs with multiple media parts.
Functions
Use the following sequence of function calls:
For how to use each function, see SDK API reference.
Models
MegaLoc requires one GGUF file. Import MEGALOC_F16 for F16 weights, or MEGALOC_MIXED for an F16 body with a Q6_K final projection. Pass either constant as modelSrc to download the weights from the registry.
The original safetensors checkpoint cannot be loaded directly. See the addon's MegaLoc conversion guide for conversion and model provenance.
Features
Choose a device with modelConfig.backend, using its exact GGML name, such as CPU, CUDA0, Vulkan0, or MTL0. Metal and Metal0 are aliases. If omitted, the addon selects a GPU, then CPU. An explicitly requested device must be available. modelConfig.threads accepts integers from 1 to 1024. modelConfig.backendsDir optionally selects a directory of GGML backend modules.
stats exposes totalTime in milliseconds, weightBytes, cpuFallbackNodes, and backendId. Backend IDs are CPU=0, Metal=1, CUDA=2, Vulkan=3, OpenCL=4, and other=99. GPU support depends on the installed Fabric build. For a batch, totalTime sums the native times; the other fields describe the final image.
When bundling a plugin subset, include @qvac/sdk/ggml-image-embedding/plugin or PLUGIN_IMAGE_EMBEDDING. The default SDK worker includes it. The ggml-classification plugin continues to load MobileNet classifiers.
Cancellation
Calls queue FIFO per model, with one active call and at most 64 waiting. Different models run independently. The returned promise exposes requestId immediately. Use cancel({ requestId }) to cancel that call, or cancel({ modelId, kind: "imageEmbedding" }) for all matching calls.
Cancelling a queued call prevents native inference from starting. Cancelling an active call discards its result after native inference finishes and rejects with InferenceCancelledError. The next call waits until that work finishes. MegaLoc does not expose a public method to interrupt native computation.
Example
The following script compares two images using MEGALOC_F16. Pass the two JPEG or PNG paths, followed by an optional GGUF URL or local file path to use custom weights:
/**
* Compare two images using the MegaLoc F16 model.
* Pass a GGUF URL or local file after the two JPEG/PNG paths to use custom weights.
*/
import { readFile } from 'node:fs/promises';
import { loadModel, embedBatch, unloadModel, close, ModelType, MEGALOC_F16 } from '@qvac/sdk';
let modelId;
try {
const [firstPath, secondPath, modelSrc = MEGALOC_F16] = process.argv.slice(2);
if (!firstPath || !secondPath) {
throw new Error('Usage: megaloc <first.jpg> <second.jpg> [GGUF URL or path]');
}
console.log('▸ Loading MegaLoc');
modelId = await loadModel({ modelSrc, modelType: ModelType.ggmlImageEmbedding });
const { embedding, stats } = await embedBatch({
modelId,
inputs: [{ image: await readFile(firstPath) }, { image: await readFile(secondPath) }]
});
const [first, second] = embedding;
if (!first || !second)
throw new Error('Expected two image descriptors');
const similarity = first.reduce((sum, value, index) => sum + value * second[index], 0);
console.log(JSON.stringify({ similarity, dimensions: first.length, stats }, null, 2));
}
catch (error) {
console.error('✖', error instanceof Error ? error.message : String(error));
process.exitCode = 1;
}
finally {
try {
if (modelId)
await unloadModel({ modelId });
}
finally {
await close();
}
}The Python client supports this capability through the same worker, using embed() or embed_batch() with a request containing base64 image bytes. A dedicated Python example is not yet published. See the Python SDK for the API.
All examples throughout this documentation are self-contained and runnable. For instructions on how to run them, see the JS/TS quickstart, the Python quickstart, or the Kotlin quickstart.