---
title: "Reranking"
canonical: https://docs.qvac.tether.io/sdk/ai-capabilities/reranking/
collection: "SDK"
package: "@qvac/sdk"
line: v0.23
current_line: true
---
# Reranking (/sdk/ai-capabilities/reranking)
## Overview
Reranking uses [`@qvac/embed-llamacpp`](https://github.com/tetherto/qvac/tree/main/packages/embed-llamacpp) with the Fabric LLM inference engine. Load a reranker (cross-encoder) model using `modelType: 'llamacpp-rerank'`. `modelConfig.device` defaults to `'gpu'`.
[`rerank()`](/sdk/reference/api#rerank) scores each document against a query. Pass a non-empty `query` and a non-empty array of `documents`.
`rerank()` returns one score per document, in the order the documents were given. Higher is more relevant. Scores are the model's raw classifier output (a logit for most rerankers), so they can be negative and are comparable only within one model. `rerank()` does not sort; sort the documents by score yourself, as the example below does.
## Functions
Use the following sequence of function calls:
1. [`loadModel()`](/sdk/reference/api#loadmodel)
2. [`rerank()`](/sdk/reference/api#rerank)
3. [`unloadModel()`](/sdk/reference/api#unloadmodel)
For how to use each function, see [SDK — API reference](/sdk/reference/api/).
## Models
Use a reranker GGUF, such as bge-reranker, jina-reranker or Qwen3-Reranker. The registry has no reranker constants yet, so pass a local GGUF path or a URL as `modelSrc` with explicit `modelType: 'llamacpp-rerank'`. For checkpoints split into multiple GGUF files, see [Sharded models](/sdk/models/sharded-models).
Rerankers and embedding models are not interchangeable: a reranker cannot be loaded for [`embed()`](/sdk/reference/api#embed), and an embedding model has no classifier head to score documents with.
The load-time `batch_size` sets the tokens per forward pass. Each document, paired with the query, must fit within it; the documents are packed into as few passes as fit. `threads`, `threads-batch`, `gpu_layers`, `flash_attn`, and `backend` control execution. The reranker fixes context size, pooling, and parallelism; these settings cannot be configured through the SDK.
## Cancellation and concurrency
The promise returned by [`rerank()`](/sdk/reference/api#rerank) exposes `requestId` before you await it. Pass that ID to [`cancel()`](/sdk/reference/api#cancel) to interrupt the request. Its promise rejects with `InferenceCancelledError`.
Each loaded model runs one rerank request at a time. Concurrent calls wait in a FIFO queue with up to 64 waiting requests per model. When the queue is full, additional calls reject with `RequestRejectedByPolicyError` without affecting active or queued requests. Cancelling a queued call by `requestId` rejects it with `InferenceCancelledError` before native work starts and leaves the active request running. See [Cancellation](/sdk/runtime/cancellation) for broad cancellation by model ID.
## Example
The following script loads a small reranker, or a GGUF path or URL passed as its first argument, scores four support articles against a question, and prints them from most to least relevant:
```js file=/packages/sdk/dist/examples/rerank.js title="rerank.js" lineNumbers
import { loadModel, rerank, unloadModel, close } from '@qvac/sdk';
// Any reranker (cross-encoder) GGUF, such as bge-reranker, jina-reranker or Qwen3-Reranker.
const modelSrc = process.argv[2] ??
'https://huggingface.co/ggml-org/models/resolve/499bc8821c6b12b4e53c5bffcb21ec206f212d81/jina-reranker-v1-tiny-en/ggml-model-f16.gguf';
const query = 'How do I get a refund for a duplicate charge?';
const documents = [
'Refunds for duplicate or failed payments are issued within 5 business days.',
'Our office is open Monday to Friday, 9am to 5pm.',
'To change your password, open Settings and choose Security.',
'If you were charged twice, contact billing with both transaction IDs.'
];
let modelId;
try {
console.error('▸ Loading reranker');
modelId = await loadModel({ modelSrc, modelType: 'llamacpp-rerank' });
// One score per document, in input order. Higher is more relevant.
const scores = await rerank({ modelId, query, documents });
const ranked = documents
.map((document, index) => ({ score: scores[index], document }))
.sort((a, b) => b.score - a.score);
console.log(`Query: ${query}\n`);
for (const { score, document } of ranked)
console.log(`${score.toFixed(3)} ${document}`);
}
catch (error) {
console.error('✖', error);
process.exitCode = 1;
}
finally {
if (modelId)
await unloadModel({ modelId });
await close();
}
```
```ts file=/packages/sdk/examples/rerank.ts title="rerank.ts" lineNumbers
import { loadModel, rerank, unloadModel, close } from '@qvac/sdk'
// Any reranker (cross-encoder) GGUF, such as bge-reranker, jina-reranker or Qwen3-Reranker.
const modelSrc =
process.argv[2] ??
'https://huggingface.co/ggml-org/models/resolve/499bc8821c6b12b4e53c5bffcb21ec206f212d81/jina-reranker-v1-tiny-en/ggml-model-f16.gguf'
const query = 'How do I get a refund for a duplicate charge?'
const documents = [
'Refunds for duplicate or failed payments are issued within 5 business days.',
'Our office is open Monday to Friday, 9am to 5pm.',
'To change your password, open Settings and choose Security.',
'If you were charged twice, contact billing with both transaction IDs.'
]
let modelId: string | undefined
try {
console.error('▸ Loading reranker')
modelId = await loadModel({ modelSrc, modelType: 'llamacpp-rerank' })
// One score per document, in input order. Higher is more relevant.
const scores = await rerank({ modelId, query, documents })
const ranked = documents
.map((document, index) => ({ score: scores[index]!, document }))
.sort((a, b) => b.score - a.score)
console.log(`Query: ${query}\n`)
for (const { score, document } of ranked) console.log(`${score.toFixed(3)} ${document}`)
} catch (error) {
console.error('✖', error)
process.exitCode = 1
} finally {
if (modelId) await unloadModel({ modelId })
await close()
}
```
```python file=/packages/sdk-python/examples/rerank.py title="rerank.py" lineNumbers
"""Python port of packages/sdk/examples/rerank.ts.
Loads a small reranker, or a GGUF path or URL passed as the first argument,
scores four support articles against a question, and prints them from most to
least relevant.
Python calls the generated `rerank` stub, so the request is a `RerankRequest`
and the scores are read off the response. `Client` closes the SDK on exit,
which the TypeScript example does with `close()`.
RUN: python examples/rerank.py [path-or-url-to-reranker.gguf]
"""
from __future__ import annotations
import asyncio
import sys
from tetherto.qvac_sdk import (
Client,
RerankRequest,
load_model,
rerank,
unload_model,
)
# Any reranker (cross-encoder) GGUF, such as bge-reranker, jina-reranker or
# Qwen3-Reranker.
DEFAULT_MODEL_SRC = (
"https://huggingface.co/ggml-org/models/resolve/"
"499bc8821c6b12b4e53c5bffcb21ec206f212d81/"
"jina-reranker-v1-tiny-en/ggml-model-f16.gguf"
)
QUERY = "How do I get a refund for a duplicate charge?"
DOCUMENTS = [
"Refunds for duplicate or failed payments are issued within 5 business days.",
"Our office is open Monday to Friday, 9am to 5pm.",
"To change your password, open Settings and choose Security.",
"If you were charged twice, contact billing with both transaction IDs.",
]
async def main() -> int:
model_src = sys.argv[1] if len(sys.argv) > 1 else DEFAULT_MODEL_SRC
async with Client() as client:
t = client.transport
model_id = None
try:
print("▸ Loading reranker", file=sys.stderr)
model_id = await load_model(
t, model_src=model_src, model_type="llamacpp-rerank"
)
# One score per document, in input order. Higher is more relevant.
response = await rerank(
t,
RerankRequest.model_validate(
{
"type": "rerank",
"modelId": model_id,
"query": QUERY,
"documents": DOCUMENTS,
}
),
)
ranked = sorted(
zip(response.scores, DOCUMENTS), key=lambda pair: pair[0], reverse=True
)
print(f"Query: {QUERY}\n")
for score, document in ranked:
print(f"{score:.3f} {document}")
except Exception as error:
print(f"✖ {error}", file=sys.stderr)
return 1
finally:
if model_id is not None:
await unload_model(t, model_id)
return 0
if __name__ == "__main__":
sys.exit(asyncio.run(main()))
```
**Tip:** all examples throughout this documentation are self-contained and runnable. For instructions on how to run them, see the [JS/TS quickstart](/sdk/js-ts-sdk#quickstart) or the [Python quickstart](/sdk/python-sdk#quickstart).