Reranking

Overview

Reranking uses @qvac/embed-llamacpp with the Fabric LLM inference engine. Load a reranker (cross-encoder) model using modelType: 'llamacpp-rerank'. modelConfig.device defaults to 'gpu'.

rerank() scores each document against a query. Pass a non-empty query and a non-empty array of documents.

rerank() returns one score per document, in the order the documents were given. Higher is more relevant. Scores are the model's raw classifier output (a logit for most rerankers), so they can be negative and are comparable only within one model. rerank() does not sort; sort the documents by score yourself, as the example below does.

Functions

Use the following sequence of function calls:

1. loadModel()
2. rerank()
3. unloadModel()

For how to use each function, see SDK — API reference.

Models

Use a reranker GGUF, such as bge-reranker, jina-reranker or Qwen3-Reranker. The registry has no reranker constants yet, so pass a local GGUF path or a URL as modelSrc with explicit modelType: 'llamacpp-rerank'. For checkpoints split into multiple GGUF files, see Sharded models.

Rerankers and embedding models are not interchangeable: a reranker cannot be loaded for embed(), and an embedding model has no classifier head to score documents with.

The load-time batch_size sets the tokens per forward pass. Each document, paired with the query, must fit within it; the documents are packed into as few passes as fit. threads, threads-batch, gpu_layers, flash_attn, and backend control execution. The reranker fixes context size, pooling, and parallelism; these settings cannot be configured through the SDK.

Cancellation and concurrency

The promise returned by rerank() exposes requestId before you await it. Pass that ID to cancel() to interrupt the request. Its promise rejects with InferenceCancelledError.

Each loaded model runs one rerank request at a time. Concurrent calls wait in a FIFO queue with up to 64 waiting requests per model. When the queue is full, additional calls reject with RequestRejectedByPolicyError without affecting active or queued requests. Cancelling a queued call by requestId rejects it with InferenceCancelledError before native work starts and leaves the active request running. See Cancellation for broad cancellation by model ID.

Example

The following script loads a small reranker, or a GGUF path or URL passed as its first argument, scores four support articles against a question, and prints them from most to least relevant:

import { loadModel, rerank, unloadModel, close } from '@qvac/sdk';
// Any reranker (cross-encoder) GGUF, such as bge-reranker, jina-reranker or Qwen3-Reranker.
const modelSrc = process.argv[2] ??
    'https://huggingface.co/ggml-org/models/resolve/499bc8821c6b12b4e53c5bffcb21ec206f212d81/jina-reranker-v1-tiny-en/ggml-model-f16.gguf';
const query = 'How do I get a refund for a duplicate charge?';
const documents = [
    'Refunds for duplicate or failed payments are issued within 5 business days.',
    'Our office is open Monday to Friday, 9am to 5pm.',
    'To change your password, open Settings and choose Security.',
    'If you were charged twice, contact billing with both transaction IDs.'
];
let modelId;
try {
    console.error('▸ Loading reranker');
    modelId = await loadModel({ modelSrc, modelType: 'llamacpp-rerank' });
    // One score per document, in input order. Higher is more relevant.
    const scores = await rerank({ modelId, query, documents });
    const ranked = documents
        .map((document, index) => ({ score: scores[index], document }))
        .sort((a, b) => b.score - a.score);
    console.log(`Query: ${query}\n`);
    for (const { score, document } of ranked)
        console.log(`${score.toFixed(3)}  ${document}`);
}
catch (error) {
    console.error('✖', error);
    process.exitCode = 1;
}
finally {
    if (modelId)
        await unloadModel({ modelId });
    await close();
}