New: QVAC SDK now supports Kotlin. Build local AI into your JVM and Android apps.
QVAC Logo
SDKAssess model fit
v0.24, the current release

Assess model fit

Check whether a set of models fits in memory before downloading it.

Overview

assessModelFit() answers one question before anything is downloaded: do these models fit in this device's memory when loaded together? For every model it runs the engine's own fitter, against the model where it is already on disk and otherwise against the registry's weightless description of it, tens of KB, the tensor list with no data section. Each fitter reports the memory its load takes on each device. The SDK sums that memory per device and compares the totals against what is free.

It is advisory. It never downloads weights or loads a model, and it does not block loadModel(), reserve memory, pick a model for you, or make any claim about speed.

Functions

  1. assessModelFit() — describe the loads you are considering, as loadModel() takes them.
  2. downloadAsset() or loadModel() — fetch whichever model you picked.

For how to use each function, see SDK — API reference.

Parameters

assessModelFit({
  models: [
    {
      modelSrc: QWEN3_8B_INST_Q4_K_M,
      modelType: "llamacpp-completion",
      modelConfig: { ctx_size: 8192 },
    },
  ],
});

models

The models, at least one, assessed as if loaded at the same time. Each takes the same parameters as loadModel():

  • modelSrc — the model, as loadModel() takes it. A catalog constant resolves to the registry's weightless description of the model; a path already on disk is read directly.
  • modelType — optional. The engine that would run the load, inferred from modelSrc when omitted, as loadModel() infers it. Required only where the source names no engine.
  • modelConfig — optional. The config loadModel() would be given, including the companion model sources a compound load needs.

The engine's own plugin resolves that config, so the settings the fitter reads are the ones the load would run with, and the same object can be handed to loadModel() afterwards.

Two fit-only fields describe a workload the load itself does not carry: transcriptionWorkload for MOSS-Transcribe and ttsWorkload for MOSS-TTS/TTSD and MOSS-SoundEffect.

Returns

{
  verdict: "fits" | "does-not-fit" | "unknown",
  pools: [{ id, unified, usedBytes, freeBytes?, totalBytes? }],
  models: [
    {
      status: "fits" | "does-not-fit" | "unknown",
      devices: [{ id, name, unified, usedBytes, freeBytes?, totalBytes? }],
      stats: { ctxSize, gpuLayers } | {},
    },
  ],
}

verdict

The verdict for every model loaded together. For a single model it is that model's status, with the engine's own margin applied by its fitter. For several models:

VerdictMeaning
fitsEvery model's fitter reached a verdict, and on every device the summed memory is within 90% of what is free.
does-not-fitA fitter projects its load will not fit, or on some device the summed memory exceeds 90% of what is free.
unknownA model got no verdict, or a device reported no free memory.

does-not-fit wins over unknown: when one model is unknown and another overflows a device, the set does not fit.

Devices that share memory with the host, the CPU itself, an integrated GPU, Apple's Metal GPU, Android's OpenCL GPU and any device that reports no memory of its own, are counted as one pool against the host's free memory. On iOS the host's free memory is capped at the app's own limit. A discrete GPU is its own pool.

pools

The memory pools the verdict was judged on: host for the host's memory, and one pool per discrete GPU, keyed by its id. Each carries usedBytes, what the set takes from it, and freeBytes and totalBytes, the lowest figures reported for it. unified is true for the host pool.

models

One entry per model, in input order: the engine addon's own fit result.

  • status — that model's verdict alone.
  • devices — one row per device the load uses, the host last. id is the PCI bus ID where the backend reports one and the device name otherwise, so the same GPU matches across engines. name is the backend's device name, such as CPU, CUDA0 or MTL0. unified marks memory the host owns. usedBytes is what the load takes there; freeBytes and totalBytes are what the device reported.
  • stats — for a llama.cpp model with a verdict, the context size and GPU layers the load resolves to: the ones the fitter settled on when it fits, the ones the load asked for when it does not. Empty for every other model.

A model the SDK cannot size has status: "unknown", no rows and empty stats.

Errors

A load that loadModel() would refuse throws the same error: no plugin for the model type, a config the plugin refuses, or the engine fitter rejecting the load, such as a model it cannot read. A load that would start but cannot be sized is unknown.

Where verdicts are available

A verdict exists wherever the engine's own fitter runs: every platform that ships that engine's addon, mobile included, for a model on disk or with a published registry description.

The fitter covers llamacpp-completion, llamacpp-embedding, whispercpp-transcription, parakeet-transcription, moss-transcribe, bci-whispercpp-transcription, tts-ggml, audiogen-ggml, sdcpp-generation, ggml-ocr and nmtcpp-translation. The translation fitter reads the model and vocabulary files themselves, so it answers for files on disk only. OCR sizes its weights alone, leaving the memory an image needs uncounted. For sdcpp-generation the fitter covers image generation with any configured ESRGAN upscaler, video, standalone mode: 'upscale' and mode: 'world'; a world load needs a pre-built sceneSrc, and world.fitSteps sets how many walk steps the fit covers.

Everything else is unknown: onnx-tts, ggml-vla and ggml-classification, a translation model not yet downloaded, MOSS-Speech, a TTS load with a LavaSR enhancer or denoiser, a diffusion world load without a sceneSrc, a load naming an ESRGAN checkpoint not yet downloaded, a diffusion load pinned to one device, split models, multimodal completion loads naming a projection model, and any model whose registry description cannot be fetched within the fitStubBudgetMs budget.

Load-time check

Once the weights are on disk, loadModel() runs the same engine fitter against the real files and the resolved load settings before it loads, with the same calculation. The fitter runs in the SDK's worker process on every platform.

A fit that does not finish, because the fitter crashed or the process quit, is retried on the next load with the same settings. A second one in a row resolves to unknown for a while. A fitter that does not return within 30 seconds resolves to unknown and the load continues. A fitter that rejects the load is logged and resolves to unknown, so the check never fails a load on its own. Cancelling a load during its fit stops it before the model loads. Verdicts appear on the SDK log stream as [advisory-fit:…] lines.

getLoadedModelInfo() returns the outcome as fitProbe, in the same shape as an assessModelFit() result with one model: verdict, pools, and models holding the engine's own fit result.

modelFitPolicy

Pass it to loadModel() for one load, or set it in the engine config for every load.

  • log (default) — the verdict is reported and the load proceeds.
  • refuse — a load the fitter projects will not fit is rejected with MODEL_FIT_REFUSED before the addon allocates anything. Only does-not-fit refuses; unknown never does, so a load is never refused for lack of evidence.
  • off — the check does not run, and fitProbe is then absent.
await loadModel({
  modelSrc: QWEN3_8B_INST_Q4_K_M,
  modelType: 'llamacpp-completion',
  modelConfig: { ctx_size: 8192 },
  modelFitPolicy: 'refuse'
})

Examples

Assess before download

The following script asks which of five Qwen3 models fit this machine at 8192 tokens of context, printing each model's verdict against its size on disk, then the verdict for all five loaded together:

assess-model-fit.js
/**
 * Which models fit this machine, asked before downloading any weights. Each
 * engine's fitter reads the registry's weightless description of a model, so
 * nothing is fetched beyond tens of KB, loaded or reserved.
 *
 * Each model gets its own verdict and the memory it takes on each device, and
 * the set gets one verdict for all of them loaded together, with what it takes
 * from each memory pool. `unknown` is a valid answer: show it as "can't say",
 * never as "no".
 */
import { assessModelFit, QWEN3_600M_INST_Q4, QWEN3_1_7B_INST_Q4, QWEN3_4B_INST_Q4_K_M, QWEN3_8B_INST_Q4_K_M, QWEN3_8_27B_MULTIMODAL_UD_Q8_K_XL } from '@qvac/sdk';
// The fitter sizes the load as `loadModel` would run it, context included.
const CONTEXT_TOKENS = 8192;
// A ladder ending well past what a laptop has, so one screen shows every verdict.
const CANDIDATES = [
    QWEN3_600M_INST_Q4,
    QWEN3_1_7B_INST_Q4,
    QWEN3_4B_INST_Q4_K_M,
    QWEN3_8B_INST_Q4_K_M,
    QWEN3_8_27B_MULTIMODAL_UD_Q8_K_XL
];
const VERDICT_MARK = {
    fits: '✔',
    'does-not-fit': '✖',
    unknown: '?'
};
function gib(bytes) {
    return `${(bytes / 1024 ** 3).toFixed(2)} GiB`;
}
try {
    const result = await assessModelFit({
        models: CANDIDATES.map((model) => ({
            modelSrc: model,
            modelType: 'llamacpp-completion',
            modelConfig: { ctx_size: CONTEXT_TOKENS }
        }))
    });
    console.log(`▸ At ${CONTEXT_TOKENS} tokens of context, each model alone`);
    for (const [index, model] of result.models.entries()) {
        const candidate = CANDIDATES[index];
        const size = gib(candidate.expectedSize).padStart(9);
        console.log(`  ${VERDICT_MARK[model.status]} ${candidate.name.padEnd(46)} ${size} on disk`);
        for (const device of model.devices) {
            console.log(`      ${device.name.padEnd(10)} ${gib(device.usedBytes).padStart(10)}`);
        }
    }
    console.log(`\n▸ All five loaded together: ${result.verdict}`);
    for (const pool of result.pools) {
        const free = pool.freeBytes === undefined ? 'no free figure' : `of ${gib(pool.freeBytes)} free`;
        console.log(`  ${pool.id.padEnd(14)} ${gib(pool.usedBytes).padStart(10)} ${free}`);
    }
    process.exit(0);
}
catch (error) {
    console.error('✖', error);
    process.exit(1);
}

Load-time check

The following script loads gpt-oss-20B at 128k context with an f32 KV cache under modelFitPolicy: 'refuse', so a does-not-fit verdict rejects the load, and reprints the [advisory-fit:…] verdicts from the SDK log stream:

advisory-model-fit.js
// `modelFitPolicy` decides what the engine fitter's verdict does to a load:
// `log` (the default) reports it and loads anyway, `refuse` rejects a load the
// fitter projects will not fit, `off` skips the check.
import { loadModel, loggingStream, SDK_LOG_ID, GPT_OSS_20B_INST_Q4_K_M } from '@qvac/sdk';
// Verdicts go to the SDK server log stream, not stdout.
void (async () => {
    for await (const log of loggingStream({ id: SDK_LOG_ID })) {
        if (log.message.includes('[advisory-fit:'))
            console.log(`▸ ${log.message}`);
    }
})().catch(() => { });
try {
    // gpt-oss-20B fits here at the default KV cache type and does not at f32.
    await loadModel({
        modelSrc: GPT_OSS_20B_INST_Q4_K_M,
        modelConfig: {
            ctx_size: 131072,
            'cache-type-k': 'f32',
            'cache-type-v': 'f32'
        },
        modelFitPolicy: 'refuse'
    });
    console.log('▸ Loaded — the fitter projected this one to fit here');
    process.exit(0);
}
catch (error) {
    console.error('✖', error instanceof Error ? error.message : error);
    process.exit(1);
}

Tip: all examples throughout this documentation are self-contained and runnable. For instructions on how to run them, see the JS/TS quickstart, the Python quickstart, or the Kotlin quickstart.

On this page

Ask anything about QVAC.