Assess model fit

Overview

assessModelFit() answers one question before anything is downloaded: is this model likely to fit in memory? For every candidate it runs the engine's own fitter — against the artifact where it is already on disk, and otherwise against the registry's weightless description of it, tens of KB, the tensor list with no data section. It also reads the model constant metadata together with a fresh memory sample, and returns a verdict for each candidate plus one for the set.

It is advisory. It never downloads weights or loads a model, and it does not block loadModel(), reserve memory, pick a model for you, or make any claim about speed.

Functions

1. assessModelFit() — describe the loads you are considering, as loadModel() takes them.
2. downloadAsset() or loadModel() — fetch whichever candidate you picked.

For how to use each function, see SDK — API reference.

Parameters

assessModelFit({
  models: [
    {
      modelSrc: QWEN3_8B_INST_Q4_K_M,
      modelType: "llm",
      modelConfig: { ctx_size: 8192 },
    },
  ],
  execution: "sequential",
});

models

The candidates, at least one. They are assessed together, against a single budget. Each takes the same parameters as loadModel():

• modelSrc — the model, as loadModel() takes it. A catalog constant resolves to the registry's weightless description of the artifact; a path already on disk is read directly.
• modelType — optional. The engine that would run the load, inferred from modelSrc when omitted, as loadModel() infers it. Required only where the source names no engine.
• modelConfig — optional. The config loadModel() would be given, including the companion model sources a compound load needs.

The engine's own plugin resolves that config, so the settings the fitter reads are the ones the load would run with, and the same object can be handed to loadModel() afterwards. The estimator reads its workload from the same config: ctx_size for a token context, duration_ms for an audio window.

execution

Either sequential or concurrent:

• sequential: counts every model as resident but adds only the largest single working peak.
• concurrent: adds every peak.

The mode describes what you intend to do so the numbers match your plan — the SDK does not schedule, serialize, or reserve anything on the strength of it.

Defaults to sequential.

policy

How much memory to withhold from the budget as headroom, left for the rest of the system:

• interactive-v1: withholds 20% of the memory free at the time of the call, capped at 2 GiB on desktop and 1 GiB on mobile. Right now, it is the only accepted value and it applies by default, so passing it is optional.

Returns

The result answers for the whole set, and result.models repeats the same shape for each candidate. These are the fields to branch on.

verdict

evidence

What backs the verdict.

native-fit is the answer the loader itself would give on this machine. The fitter reports the bytes it would allocate, and those bytes compose: each candidate is measured on its own, and the set is summed under one budget according to execution. A refusal from the fitter stands on its own, whatever the arithmetic says, because it knows things the byte total does not carry.

computed-only is the fallback wherever the fitter reaches no verdict. A set rests on its weakest evidence, so one computed-only model makes the whole result computed-only.

Read evidence alongside the verdict, because the same unknown can mean different things:

• Under native-fit, the numbers came out too close to call.
• Under computed-only, nothing measured this load, so the answer says nothing about whether the model fits.

That difference matters once you act on the result. Hiding the likely-too-large candidates works the same way under both, since that verdict is trustworthy either way. Telling a user that a model will fit needs native-fit; the computed floor never produces likely-fits.

basis

The memory the verdict was weighed against:

• system-memory — device RAM and system-wide use. Desktop, and Android, whose low-memory killer acts system-wide.
• process-memory — the app's own ceiling. iOS, where jetsam terminates an app on its own footprint against a limit well below device RAM.
• device-memory — a discrete GPU's own memory, when the model would execute there.
• device-budget — on Windows, the GPU memory the OS grants this process, since a card's readings there are per-process rather than device-wide.

Both device bases additionally require the system-memory budget to hold, because a GPU load is paid for in system RAM too.

device

Per model: where that load resolved to execute, gpu or cpu, after the device defaults this host applies for itself. Absent for an engine that expresses no placement.

Read it alongside evidence. The llama fitters read device memory alone and decline a cpu load, so one there carries no native-fit and falls back to the computed floor. The speech and voice fitters answer for a cpu load like any other.

reasons

Every unknown names its cause here — on the model when the cause is that model, on the result when it is the machine. A verdict that fell back because the fitter produced nothing says so, and names what the fitter reported. The ones you are most likely to meet:

• The fitter could not be asked, or ran and declined — the reason names what it reported.
• The model is not a catalog constant, so there is no resource profile to read.
• The model's engine has no fitter, so nothing sizes the load beyond its weights.

Where verdicts are available

native-fit evidence — and with it any likely-fits — exists wherever the engine's own fitter runs: every platform that ships that engine's addon, mobile included, for a source on disk or with a published registry description.

The fitter covers llamacpp-completion, llamacpp-embedding, whispercpp-transcription, parakeet-transcription, bci-whispercpp-transcription, tts-ggml, audiogen-ggml, sdcpp-generation, ggml-ocr and nmtcpp-translation. The translation fitter sizes the model and vocabulary files themselves, so it answers for files on disk and never before download. Audiogen and diffusion report no byte breakdown, OCR sizes its weights alone, leaving the memory an image needs uncounted, and translation reports one allowance for the whole load. A fit from any of the four therefore falls back to the floor when assessed together with other models, while a refusal still stands. The BCI projection includes the embedder: the host RAM it keeps is counted in hostBytes, read from the bci-embedder.bin beside the model, or from the load's embedderModelSrc where it names one.

Everywhere else the assessment falls back to the computed floor: likely-too-large when the weights alone exceed the budget, unknown otherwise. That covers onnx-tts, ggml-vla and ggml-classification, a translation model not yet downloaded, MOSS-Speech, diffusion world and upscale loads, sharded models, multimodal completion loads naming a projection model, and any source whose registry description cannot be fetched.

The longer treatment, including how a GPU changes which memory the budget measures, lives in packages/sdk/docs/assess-model-fit.md.

Load-time probe

Once the weights are on disk, loadModel() runs the same engine fitter against the real file and the resolved load settings before it loads. It is the stronger evidence once you have the file. Desktop runs it in a disposable child process. Mobile, which cannot spawn one, runs it on a worker thread.

A crash, timeout, or unsupported configuration resolves to no verdict. Verdicts appear on the SDK log stream as [advisory-fit:…] lines.

getLoadedModelInfo() returns the outcome as fitProbe:

• verdict — fit, does-not-fit, or unknown. These are unhedged, unlike the verdicts above, because the probe measured this build against this file.
• engine — the engine package whose fitter ran.
• reason and message — why the probe reached that verdict.
• plan — the placement it projected, where the fitter resolved one: nCtx, nGpuLayers, nGpuDevices.
• projection — what the fitter measured, present on fit and does-not-fit alike. Engines do not all measure the same things, so every field is optional: deviceBytes and hostBytes for the peaks, weightsBytes, contextBytes and computeBytes within deviceBytes where the engine separates them, deviceFreeBytes and deviceTotalBytes, and report, the engine's own memory table.

modelFitPolicy

Pass it to loadModel() for one load, or set it in the engine config for every load.

• log (default) — the verdict is reported and the load proceeds.
• refuse — a load the fitter projects will not fit is rejected with MODEL_FIT_REFUSED before the addon allocates anything. Only does-not-fit refuses; unknown never does, so a load is never refused for lack of evidence.
• off — the probe does not run, and fitProbe is then absent.

await loadModel({
  modelSrc: QWEN3_8B_INST_Q4_K_M,
  modelType: 'llamacpp-completion',
  modelConfig: { ctx_size: 8192 },
  modelFitPolicy: 'refuse'
})

Examples

Assess before download

The following script asks which of five Qwen3 models this machine is likely to run at 8192 tokens of context, printing the budget basis, each verdict against the model's size on disk, and the reasons behind every unknown:

/**
 * Which models is this machine likely to run — asked before downloading any
 * weights. Nothing is fetched, loaded or reserved.
 *
 * The header prints the evidence the answer rests on: `system-memory` on a
 * CPU-only or integrated-GPU host, `device-memory` on a discrete card,
 * `device-budget` for the per-process allowance Windows grants on one. Which
 * of those applies depends on where the engine would put the model.
 *
 * `unknown` is a real answer, not an error: the evidence does not support a
 * call either way. Show it as "can't say", never as "no".
 */
import { assessModelFit, QWEN3_600M_INST_Q4, QWEN3_1_7B_INST_Q4, QWEN3_4B_INST_Q4_K_M, QWEN3_8B_INST_Q4_K_M, QWEN3_8_27B_MULTIMODAL_UD_Q8_K_XL } from '@qvac/sdk';
// Sizes the assessment the way it sizes the load: the estimator reads the
// context out of the config `loadModel` would be given.
const CONTEXT_TOKENS = 8192;
// A ladder ending well past what a laptop has, so one screen shows every verdict.
const CANDIDATES = [
    QWEN3_600M_INST_Q4,
    QWEN3_1_7B_INST_Q4,
    QWEN3_4B_INST_Q4_K_M,
    QWEN3_8B_INST_Q4_K_M,
    QWEN3_8_27B_MULTIMODAL_UD_Q8_K_XL
];
const VERDICT_MARK = {
    'likely-fits': '✔',
    'likely-too-large': '✖',
    unknown: '?'
};
function gib(bytes) {
    return `${(bytes / 1024 ** 3).toFixed(2)} GiB`;
}
try {
    const result = await assessModelFit({
        models: CANDIDATES.map((model) => ({
            modelSrc: model,
            modelType: 'llamacpp-completion',
            modelConfig: { ctx_size: CONTEXT_TOKENS }
        })),
        // Declared for aggregation only, not a scheduling instruction: 'sequential'
        // counts the largest operation peak, 'concurrent' counts one per model.
        execution: 'sequential',
        policy: 'interactive-v1'
    });
    console.log(`▸ Budget basis: ${result.basis}`);
    if (result.budget) {
        console.log(`    ${gib(result.budget.availableAfterReserveBytes)} budget` +
            ` (${gib(result.budget.totalBytes)} total,` +
            ` ${gib(result.budget.usedBytes)} in use,` +
            ` ${gib(result.budget.availableBytes)} free,` +
            ` ${gib(result.budget.reservedBytes)} held back)`);
    }
    // Says which device the verdicts assume, when the model would run on a GPU.
    const placement = result.assumptions.find((line) => line.includes('assumed to execute on'));
    if (placement)
        console.log(`▸ ${placement}`);
    console.log(`\n▸ At ${CONTEXT_TOKENS} tokens of context`);
    for (const [index, model] of result.models.entries()) {
        const mark = VERDICT_MARK[model.verdict];
        const size = gib(CANDIDATES[index].expectedSize).padStart(9);
        const needs = model.evidence === 'computed-only' && model.floorBytes
            ? `at least ${gib(model.floorBytes)}`
            : '';
        console.log(`  ${mark} ${model.name.padEnd(46)} ${size} on disk  ${needs}`);
        // Present on every `unknown`, and worth surfacing: it names what is missing.
        if (model.verdict === 'unknown') {
            for (const reason of model.reasons)
                console.log(`      ${reason}`);
        }
    }
    console.log(`\n▸ All five together, sequentially: ${result.verdict}`);
    console.log('▸ Advisory only — this does not gate loadModel or reserve anything.');
    process.exit(0);
}
catch (error) {
    console.error('✖', error);
    process.exit(1);
}

Load-time probe

The following script loads Qwen3.5 0.8B at 4k context, which the fitter projects to fit, and reprints the [advisory-fit:…] verdicts from the SDK log stream. Set QVAC_FIT_DEMO_ATTEMPT_OVERSIZED=1 to also load gpt-oss-20B at 128k context with an f32 KV cache, which it projects not to fit and loads anyway:

// `modelFitPolicy` decides what the engine fitter's verdict does to a load:
// `log` (the default) reports it and loads anyway, `refuse` rejects a load the
// fitter projects will not fit, `off` skips the check.
import { loadModel, loggingStream, SDK_LOG_ID, GPT_OSS_20B_INST_Q4_K_M } from '@qvac/sdk';
// Verdicts go to the SDK server log stream, not stdout.
void (async () => {
    for await (const log of loggingStream({ id: SDK_LOG_ID })) {
        if (log.message.includes('[advisory-fit:'))
            console.log(`▸ ${log.message}`);
    }
})().catch(() => { });
try {
    // gpt-oss-20B fits here at the default KV cache type and does not at f32.
    await loadModel({
        modelSrc: GPT_OSS_20B_INST_Q4_K_M,
        modelConfig: {
            ctx_size: 131072,
            'cache-type-k': 'f32',
            'cache-type-v': 'f32'
        },
        modelFitPolicy: 'refuse'
    });
    console.log('▸ Loaded — the fitter projected this one to fit here');
    process.exit(0);
}
catch (error) {
    console.error('✖', error instanceof Error ? error.message : error);
    process.exit(1);
}