Adapting LLMs to domain-specific tasks via LoRA — i.e., train small adapter weights on custom datasets to specialize model behavior.

Overview

Fine-tuning trains a LoRA (Low-Rank Adaptation) adapter on top of an LLM base model, to be used at inference time with completion.

Load any supported LLM using modelType: "llm". Then call finetune() with the dataset and training settings. Training can be done in two modes:

SFT: chat-based; enable with assistantLossOnly: true.
Causal: raw text; default (assistantLossOnly: false).

The output is a small .gguf adapter file that you can pass to completion() via modelConfig.lora.

Functions

Use the following sequence of function calls:

For how to use each function, see SDK — API reference.

Models

You can load any llama.cpp-compatible text-generation/chat model that meets the following criteria:

File format: *.gguf (or use constants)
Architecture: qwen3, gemma3, or bitnet
Quantization: F32, F16, Q4_0, Q8_0, TQ1_0, or TQ2_0

In QVAC, not every model compatible with text generation/chat is also compatible with fine-tuning. Only those that meet all the criteria above are.

For models available as constants, see SDK — Models. Note that not every model constant is fine-tunable: for example, many Qwen3-4B/8B constants are Q4_K_M. Check a model's quantization with getModelInfo() before training.

Training

SFT

Supervised fine-tuning (SFT) teaches the model how to respond to prompts. Use it for chat tuning, instruction following, or any task where you want to shape assistant responses.

Dataset format: JSONL where each line is a JSON object with a messages array. Supported roles: system, user, assistant, and tool. Example:

{"messages":[{"role":"system","content":"You are a helpful assistant."},{"role":"user","content":"What is 2+2?"},{"role":"assistant","content":"2+2 equals 4."}]}
{"messages":[{"role":"user","content":"What is the capital of France?"},{"role":"assistant","content":"The capital of France is Paris."}]}

Causal

Causal fine-tuning adapts the model to a domain by training on raw text. Use it for domain adaptation, style transfer, or tasks where you want the model to better reflect specialized vocabulary, patterns, or tone.

Dataset format: plain text file. Example:

This is sample training text.
Another paragraph of content.

Example

The example trains on a small SFT dataset with a separate eval file. Save both next to the example, as train.jsonl and eval.jsonl, then swap in your own data — the example reads whatever those two files contain.

train.jsonl

{"messages":[{"role":"system","content":"Answer in one short sentence."},{"role":"user","content":"What is the capital of Japan?"},{"role":"assistant","content":"Tokyo is the capital of Japan."}]}
{"messages":[{"role":"system","content":"Answer in one short sentence."},{"role":"user","content":"How many continents are there?"},{"role":"assistant","content":"There are seven continents."}]}
{"messages":[{"role":"user","content":"At what temperature does water boil at sea level?"},{"role":"assistant","content":"Water boils at 100 degrees Celsius at sea level."}]}
{"messages":[{"role":"user","content":"Which planet is closest to the Sun?"},{"role":"assistant","content":"Mercury is the planet closest to the Sun."}]}
{"messages":[{"role":"user","content":"What is the largest ocean on Earth?"},{"role":"assistant","content":"The Pacific is the largest ocean on Earth."}]}
{"messages":[{"role":"user","content":"How many bones are in the adult human body?"},{"role":"assistant","content":"An adult human body has 206 bones."}]}
{"messages":[{"role":"user","content":"What gas do plants absorb from the air?"},{"role":"assistant","content":"Plants absorb carbon dioxide from the air."}]}
{"messages":[{"role":"user","content":"What is the chemical symbol for iron?"},{"role":"assistant","content":"The chemical symbol for iron is Fe."}]}
{"messages":[{"role":"user","content":"Which river is the longest in Africa?"},{"role":"assistant","content":"The Nile is the longest river in Africa."}]}
{"messages":[{"role":"user","content":"How many sides does a hexagon have?"},{"role":"assistant","content":"A hexagon has six sides."}]}
{"messages":[{"role":"user","content":"What is the speed of light in a vacuum?"},{"role":"assistant","content":"Light travels at about 300,000 kilometres per second in a vacuum."}]}
{"messages":[{"role":"user","content":"Which metal is liquid at room temperature?"},{"role":"assistant","content":"Mercury is liquid at room temperature."}]}
{"messages":[{"role":"user","content":"What is the smallest prime number?"},{"role":"assistant","content":"Two is the smallest prime number."}]}
{"messages":[{"role":"user","content":"How long does the Earth take to orbit the Sun?"},{"role":"assistant","content":"The Earth takes about 365 days to orbit the Sun."}]}
{"messages":[{"role":"user","content":"What is the hardest natural substance?"},{"role":"assistant","content":"Diamond is the hardest natural substance."}]}
{"messages":[{"role":"user","content":"Which organ pumps blood through the body?"},{"role":"assistant","content":"The heart pumps blood through the body."}]}
{"messages":[{"role":"user","content":"What is the main gas in the Earth's atmosphere?"},{"role":"assistant","content":"Nitrogen is the main gas in the Earth's atmosphere."}]}
{"messages":[{"role":"user","content":"How many degrees are in a right angle?"},{"role":"assistant","content":"A right angle has 90 degrees."}]}
{"messages":[{"role":"user","content":"Which country has the largest population?"},{"role":"assistant","content":"India has the largest population."}]}
{"messages":[{"role":"user","content":"What is the freezing point of water in Fahrenheit?"},{"role":"assistant","content":"Water freezes at 32 degrees Fahrenheit."}]}
{"messages":[{"role":"user","content":"What does DNA stand for?"},{"role":"assistant","content":"DNA stands for deoxyribonucleic acid."}]}
{"messages":[{"role":"user","content":"Which mountain is the tallest above sea level?"},{"role":"assistant","content":"Mount Everest is the tallest mountain above sea level."}]}
{"messages":[{"role":"user","content":"How many players are on a football team on the pitch?"},{"role":"assistant","content":"A football team fields eleven players."}]}
{"messages":[{"role":"user","content":"What is the currency of the United Kingdom?"},{"role":"assistant","content":"The currency of the United Kingdom is the pound sterling."}]}

The following script loads an LLM, runs fine-tuning on that dataset, and optionally demonstrates pause/resume when invoked with --pause-resume. The adapter and its checkpoints land in ./finetune-output:

llamacpp-finetune.js

import path from 'node:path';
import { fileURLToPath } from 'node:url';
import { finetune, loadModel, QWEN3_600M_INST_Q4, unloadModel } from '@qvac/sdk';
const pauseResumeEnabled = process.argv.includes('--pause-resume');
// The worker resolves dataset paths against the process working directory, so
// derive them from this file's own folder instead.
const datasetDir = path.dirname(fileURLToPath(import.meta.url));
let modelId;
let exitCode = 0;
async function readProgress(handle, onTick) {
    for await (const tick of handle.progressStream) {
        const phase = tick.is_train ? 'train' : 'val';
        console.log(`▸ epoch=${tick.current_epoch + 1} step=${tick.global_steps} batch=${tick.current_batch}/${tick.total_batches} ${phase} loss=${tick.loss?.toFixed(4)} acc=${tick.accuracy?.toFixed(4)} eta=${Math.round(tick.eta_ms / 1000)}s`);
        onTick(tick.global_steps);
    }
}
try {
    modelId = await loadModel({
        modelSrc: QWEN3_600M_INST_Q4,
        modelConfig: {
            device: 'gpu',
            ctx_size: 512
        }
    });
    console.log(`▸ Model loaded with ID: ${modelId}`);
    const loadedModelId = modelId;
    const finetuneParams = {
        modelId: loadedModelId,
        options: {
            trainDatasetDir: path.join(datasetDir, 'train.jsonl'),
            validation: {
                type: 'dataset',
                path: path.join(datasetDir, 'eval.jsonl')
            },
            numberOfEpochs: 2,
            learningRate: 1e-4,
            lrMin: 1e-8,
            loraModules: 'attn_q,attn_k,attn_v,attn_o,ffn_gate,ffn_up,ffn_down',
            assistantLossOnly: true,
            checkpointSaveSteps: 2,
            checkpointSaveDir: './finetune-output/checkpoints',
            outputParametersDir: './finetune-output'
        }
    };
    const handle = finetune(finetuneParams);
    let pauseRequested = false;
    let pauseResultPromise;
    const progressTask = readProgress(handle, (globalSteps) => {
        if (pauseResumeEnabled && !pauseRequested && globalSteps >= 10) {
            pauseRequested = true;
            console.log('▸ Requesting a pause so the run can be resumed...');
            pauseResultPromise = finetune({
                modelId: loadedModelId,
                operation: 'pause'
            });
        }
    });
    const initialResult = await handle.result;
    await progressTask;
    if (pauseResultPromise) {
        await pauseResultPromise;
    }
    console.log('▸ Initial finetune result:', initialResult);
    if (pauseResumeEnabled && initialResult.status === 'PAUSED') {
        console.log('▸ Resuming from the saved checkpoint...');
        const resumedHandle = finetune({
            ...finetuneParams,
            operation: 'resume'
        });
        const resumedProgressTask = readProgress(resumedHandle, function () { });
        const resumedResult = await resumedHandle.result;
        await resumedProgressTask;
        console.log('▸ Resumed finetune result:', resumedResult);
    }
}
catch (error) {
    console.error('✖', error);
    exitCode = 1;
}
finally {
    if (modelId) {
        await unloadModel({ modelId, clearStorage: false });
    }
}
process.exit(exitCode);

Tip: all examples throughout this documentation are self-contained and runnable. For instructions on how to run them, see the JS/TS quickstart or the Python quickstart.

Fine-tuning