VLA
Vision-language-action inference — run a VLA policy that turns camera frames, robot state, and a natural-language instruction into an action chunk.
Overview
Vision-language-action (VLA) inference uses a GGML engine (@qvac/vla-ggml) to run VLA policies. Load a model using modelType: "ggml-vla". Then, feed it preprocessed camera frames, the robot's current state, and a tokenized natural-language instruction; the model returns an action chunk — chunkSize future timesteps of an actionDim-dimensional action vector — to drive the robot's actuators.
The same vla() interface drives multiple policy families that differ in (i) how many camera views they expect, (ii) how they consume the robot state, and (iii) how they consume camera frames. vlaHparams() reports these per-model traits — numCameras, stateInputMode, and imageInputMode — so you can shape your inputs accordingly without hardcoding to a single architecture.
imageInputMode is either "pixels" or "patches":
"pixels": eachimages[]entry is a3·w·hpixel plane, as prepared byvlaPreprocessImage()."patches": eachimages[]entry is a pre-patchified buffer ofvlaHparams().imagePatchElemsfloats, not a3·w·hpixel plane. A real consumer patchifies each camera frame the wayGr00tPolicydoes, and the model's learned patch-embed runs inside the addon.
vla() returns the produced action chunk together with per-stage timing stats.
Functions
Use the following sequence of function calls:
loadModel()vlaHparams()— to size your input buffersvla()unloadModel()
The SDK also exposes two helpers to prepare the wire-format tensors expected by vla():
vlaPreprocessImage(): prepares raw camera pixels into the image tensorvla()expects.vlaPadState(): zero-pads a robot-state vector tohparams.maxStateDim(continuous-state models only).
Both helpers are inlined client-side (no native binding required), so they work under Node, Bun, and Expo even without VLA prebuilds. The natural-language instruction must be tokenized on the consumer side using the model's tokenizer.
For how to use each function, see SDK — API reference.
Models
Supported model families:
- SmolVLA: single all-in-one
*.gguffile. Expects 2 camera views and a continuous robot state. Available constant:SMOLVLA_LIBERO_VISION_Q8. - π₀.₅ (pi05): single all-in-one
*.gguffile. ExpectsvlaHparams().numCamerascamera views (3 forPI05_BASE_Q_AGGRESSIVE) and a discrete robot state tokenized into the language prompt instead of a state buffer; thenoiseprior is required. Available constant:PI05_BASE_Q_AGGRESSIVE. - NVIDIA GR00T N1.7-3B (LIBERO): single all-in-one
*.gguffile converted from the LIBERO checkpoint. Expects 2 camera views and stores a single embodiment row —vlaSetEmbodiment()is rejected. ReportsimageInputMode: "patches"— each camera view is a pre-patchified buffer ofvlaHparams().imagePatchElemsfloats, sovlaPreprocessImage()does not apply. Uses a continuous robot state (pad withvlaPadState()); thenoiseprior is required (flow-matching). Available constants:GROOT_Q8_VF16(desktop) andGROOT_Q5_VF16(mobile). Preferred for LIBERO-only use. - NVIDIA GR00T N1.7-3B (base, multi-embodiment): single all-in-one
*.gguffile converted from the base checkpoint, carrying multiple embodiments — 17 rows shipped (out of up to 32), camera count per embodiment (see GR00T embodiments below). Same input contract as the LIBERO build:imageInputMode: "patches"(each camera view is a pre-patchified buffer ofvlaHparams().imagePatchElemsfloats, sovlaPreprocessImage()does not apply), continuous robot state (pad withvlaPadState()),noiseprior required (flow-matching). Available constants:GROOT_MULTI_Q8_VF16(desktop) andGROOT_MULTI_Q5_VF16(mobile).
For models available as constants, see SDK — Models.
GR00T embodiments
Multi-embodiment selection applies to the GROOT_MULTI_* constants. The LIBERO-only GROOT_Q8_VF16 / GROOT_Q5_VF16 GGUFs store a single embodiment row and reject any switch — the rest of this section is about GROOT_MULTI_*. A single GROOT_MULTI_* GGUF carries up to 32 embodiments; 17 rows ship in the current builds.
Pick one at load via modelConfig.embodiment. The selector accepts:
- a tag string (e.g.
"libero_sim"), - a numeric
cat_idin0..31, - an object
{ tag? | catId?, numCameras? }— at most one oftag/catId, plus an optionalnumCamerasoverride bounded to1..64. On the load path, an object with onlynumCamerasand notagorcatIdis valid: it keeps the GGUF's default embodiment and overrides the count. - omitted — the GGUF's default embodiment is used.
numCameras is not only an override for a rig with a different view count — it is required for an embodiment the GGUF has no count for. Only a subset of the shipped rows are self-describing in the GGUF (9 of 17 in GROOT_MULTI_*), so most rows will not load or switch without it; the runtime never guesses.
Switch at runtime with vlaSetEmbodiment({ modelId, embodiment }) — same selector shape, except a switch must name an embodiment (a tag or a catId; the bare { numCameras } form is rejected). A switch reads ~20 MB from the GGUF versus reloading the whole ~4 GB file — rejected on a single-embodiment GGUF (GROOT_Q8_VF16 / GROOT_Q5_VF16), and while a vla() response is outstanding (the queued switch throws JOB_ALREADY_RUNNING — await the in-flight vla() first).
vlaHparams() reports the resolved embodiment as selectedEmbodimentTag and selectedEmbodimentCatId, together with the active numCameras and actionDim. These fields are also present on the single-embodiment GROOT_Q* constants (they surface the baked-in row), so their presence is not a capability check — a switch may still be rejected. Rebuild your inference buffers from vlaHparams() after every vlaSetEmbodiment().
Examples
SmolVLA
The following script loads SmolVLA-LIBERO from the registry, builds synthetic inputs (zero-filled gray images + BOS-only tokens + zero state), and runs a single inference pass — printing the produced action chunk and per-stage timings:
/**
* SmolVLA (vision-language-action) example using the QVAC SDK.
*
* Loads the SmolVLA-LIBERO GGUF model, runs a single inference pass with
* synthetic inputs (zero-filled gray images + BOS-only tokens + zero state +
* zero noise), and prints the produced action chunk + per-stage timings.
*
* Usage:
* bun examples/vla-smolvla.ts [path-to-smolvla.gguf]
*
* By default the example pulls the registry-baked SmolVLA-LIBERO GGUF
* (~1.9 GB) on first run and caches it locally. Pass an absolute path on
* the command line to override and load a local GGUF instead.
*/
import { close, loadModel, SMOLVLA_LIBERO_VISION_Q8, unloadModel, vla, vlaHparams, vlaPadState, vlaPreprocessImage } from '@qvac/sdk';
const modelSrcOverride = process.argv[2];
const modelSrc = modelSrcOverride ?? SMOLVLA_LIBERO_VISION_Q8;
try {
console.log('▸ Loading SmolVLA model...');
const modelId = await loadModel({
modelSrc,
modelType: 'ggml-vla',
modelConfig: { backend: 'cpu' },
onProgress: (p) => {
const mb = (n) => (n / 1e6).toFixed(1);
const line = `▸ Downloading ${p.percentage.toFixed(0)}% (${mb(p.downloaded)}/${mb(p.total)} MB)`;
process.stderr.write(process.stderr.isTTY ? `\r${line}` : `${line}\n`);
if (p.percentage >= 100)
process.stderr.write('\n');
}
});
if (typeof modelSrc !== 'string')
process.stderr.write('\n');
console.log(`▸ Model loaded: ${modelId}`);
const { hparams, backendName } = await vlaHparams({ modelId });
console.log(`▸ Backend: ${backendName ?? '(unknown)'}`);
console.log('▸ Hparams:', hparams);
// Build synthetic inputs sized to the model's expectations. A real
// consumer would: read camera frames, tokenize the instruction with the
// SmolVLM2 tokenizer, and read the robot's current end-effector pose.
const size = hparams.visionImageSize;
const dummyPixels = new Uint8Array(size * size * 3).fill(128);
const front = vlaPreprocessImage(dummyPixels, size, size, { size });
const wrist = vlaPreprocessImage(dummyPixels, size, size, { size });
const tokens = new Int32Array(hparams.tokenizerMaxLength);
const mask = new Uint8Array(hparams.tokenizerMaxLength);
// BOS-only "instruction" for the smoke test.
tokens[0] = 1;
mask[0] = 1;
const state = vlaPadState([0, 0, 0, 0, 0, 0], hparams.maxStateDim);
const noise = new Float32Array(hparams.chunkSize * hparams.maxActionDim);
console.log('▸ Running VLA inference...');
const { actions, actionDim, chunkSize, stats } = await vla({
modelId,
images: [front, wrist],
imgWidth: size,
imgHeight: size,
state,
tokens,
mask,
noise
});
console.log(`▸ Got ${chunkSize} action steps of dim ${actionDim}.`);
console.log(Array.from(actions.subarray(0, actionDim)));
if (stats) {
console.log(`▸ Timing: vision=${stats.vision_ms?.toFixed(0)}ms ` +
`smollm2=${stats.smollm2_total_ms?.toFixed(0)}ms ` +
`ode=${stats.ode_ms?.toFixed(0)}ms ` +
`total=${stats.total_ms?.toFixed(0)}ms`);
}
await unloadModel({ modelId, clearStorage: false });
console.log('▸ Model unloaded.');
process.exit(0);
}
catch (error) {
console.error('✖', error);
await close();
process.exit(1);
}pi05
The following script loads pi05 from the registry, builds synthetic inputs (zero-filled gray images + BOS-only tokens + empty state), and runs a single inference pass — printing the produced action chunk and per-stage timings:
/**
* π₀.₅ (pi05) vision-language-action example using the QVAC SDK.
*
* Loads the Physical Intelligence π₀.₅ GGUF model, runs a single inference
* pass with synthetic inputs (zero-filled gray images + BOS-only tokens +
* seeded noise), and prints the produced action chunk + per-stage timings.
*
* π₀.₅ differs from SmolVLA in two ways the SDK surfaces via `vlaHparams()`:
* - `numCameras: 3` — it expects exactly three camera frames (not two).
* - `stateInputMode: 'discrete'` — the robot state is tokenised into the
* language prompt, so the `state` buffer is ignored. We pass an empty
* `Float32Array(0)`. π₀.₅ also requires the `noise` prior.
*
* Usage:
* bun examples/vla-pi05.ts [path-to-pi05.gguf]
*
* By default the example pulls the registry-baked π₀.₅ GGUF (~3.9 GB) on
* first run and caches it locally. Pass an absolute path on the command line
* to override and load a local GGUF instead.
*/
import { close, loadModel, PI05_BASE_Q_AGGRESSIVE, unloadModel, vla, vlaHparams, vlaPreprocessImage } from '@qvac/sdk';
const modelSrcOverride = process.argv[2];
const modelSrc = modelSrcOverride ?? PI05_BASE_Q_AGGRESSIVE;
try {
console.log('▸ Loading π₀.₅ (pi05) model...');
const modelId = await loadModel({
modelSrc,
modelType: 'ggml-vla',
modelConfig: { backend: 'cpu' },
onProgress: (p) => {
const mb = (n) => (n / 1e6).toFixed(1);
const line = `▸ Downloading ${p.percentage.toFixed(0)}% (${mb(p.downloaded)}/${mb(p.total)} MB)`;
process.stderr.write(process.stderr.isTTY ? `\r${line}` : `${line}\n`);
if (p.percentage >= 100)
process.stderr.write('\n');
}
});
if (typeof modelSrc !== 'string')
process.stderr.write('\n');
console.log(`▸ Model loaded: ${modelId}`);
const { hparams, backendName } = await vlaHparams({ modelId });
console.log(`▸ Backend: ${backendName ?? '(unknown)'}`);
console.log('▸ Hparams:', hparams);
// Build synthetic inputs sized to the model's expectations. A real
// consumer would: read camera frames, tokenize the instruction with the
// model's tokenizer, and (for π₀.₅) inline the robot state into the prompt.
const size = hparams.visionImageSize;
const numCameras = hparams.numCameras ?? 3;
const dummyPixels = new Uint8Array(size * size * 3).fill(128);
// π₀.₅ expects exactly `numCameras` frames.
const images = Array.from({ length: numCameras }, () => vlaPreprocessImage(dummyPixels, size, size, { size }));
const tokens = new Int32Array(hparams.tokenizerMaxLength);
const mask = new Uint8Array(hparams.tokenizerMaxLength);
// BOS-only "instruction" for the smoke test.
tokens[0] = 1;
mask[0] = 1;
// Discrete-state model: the state buffer is ignored (state is tokenised
// into the prompt), so pass an empty Float32Array. π₀.₅ requires `noise`.
const state = new Float32Array(0);
const noise = new Float32Array(hparams.chunkSize * hparams.maxActionDim);
console.log('▸ Running VLA inference...');
const { actions, actionDim, chunkSize, stats } = await vla({
modelId,
images,
imgWidth: size,
imgHeight: size,
state,
tokens,
mask,
noise
});
console.log(`▸ Got ${chunkSize} action steps of dim ${actionDim}.`);
console.log(Array.from(actions.subarray(0, actionDim)));
if (stats) {
console.log(`▸ Timing: vision=${stats.vision_ms?.toFixed(0)}ms ` +
`prefill=${stats.prefill_total_ms?.toFixed(0)}ms ` +
`ode=${stats.ode_ms?.toFixed(0)}ms ` +
`total=${stats.total_ms?.toFixed(0)}ms`);
}
await unloadModel({ modelId, clearStorage: false });
console.log('▸ Model unloaded.');
process.exit(0);
}
catch (error) {
console.error('✖', error);
await close();
process.exit(1);
}GR00T
The following script loads a GR00T N1.7-3B GGUF from the registry and runs a single inference pass with synthetic inputs. Unlike SmolVLA and pi05, GR00T reports imageInputMode: "patches": each camera entry is a pre-patchified buffer of hparams.imagePatchElems floats, the continuous state is padded with vlaPadState(), and the noise prior is required.
Two optional environment variables drive the multi-embodiment path exposed by GROOT_MULTI_*:
QVAC_GROOT_EMBODIMENTselects the embodiment at load — a tag string (e.g.libero_sim) or a numericcat_id.QVAC_GROOT_SWITCH_EMBODIMENTtriggers a runtimevlaSetEmbodiment()after the first inference pass; the example then rebuilds its inputs from the refreshedvlaHparams()and runs a second pass on the new embodiment.
The example defaults to GROOT_Q8_VF16 (the LIBERO single-embodiment build), on which a switch is rejected. To exercise QVAC_GROOT_SWITCH_EMBODIMENT, pass a local GROOT_MULTI_* GGUF path on the command line — the multi-embodiment constants are supported by @qvac/vla-ggml 0.17.0 and later.
/**
* NVIDIA GR00T N1.7-3B (vision-language-action) example using the QVAC SDK.
*
* Loads the GR00T-N1.7-3B LIBERO GGUF model, runs a single inference pass with
* synthetic inputs, and prints the produced action chunk + per-stage timings.
*
* GR00T differs from SmolVLA / π₀.₅ in the ways the SDK surfaces via
* `vlaHparams()`:
* - `imageInputMode: 'patches'` — each `images` entry is a pre-patchified
* buffer of `hparams.imagePatchElems` floats, NOT a `3·w·h` pixel plane.
* A real consumer patchifies each camera frame the way `Gr00tPolicy` does
* (the model's learned patch-embed runs inside the addon); here we feed
* synthetic patch buffers of the right length.
* - `stateInputMode: 'continuous'` — the robot state is projected by an
* in-model linear layer; pad it to `hparams.maxStateDim` with `vlaPadState`.
* - `noise` is REQUIRED. GR00T is a flow-matching model that does not sample
* its own prior; a missing `noise` is rejected as INVALID_INPUT.
*
* The prompt must place one contiguous run of `MERGED_TOKENS_PER_IMAGE`
* image-placeholder tokens per camera. A real consumer produces this by
* tokenising the instruction with the Qwen3-VL tokenizer, which inserts the
* image tokens; we lay it out by hand for the smoke.
*
* Usage:
* bun examples/vla-groot.ts [path-to-groot.gguf]
*
* By default the example pulls the registry-baked GR00T-LIBERO GGUF (~3.76 GB)
* on first run and caches it locally. Pass an absolute path on the command line
* to override and load a local GGUF instead.
*
* Multi-embodiment GGUFs (vla-ggml >= 0.17.0): one GGUF can carry many
* embodiments, one active at a time. Optional env vars demonstrate selection:
* QVAC_GROOT_EMBODIMENT initial selection at load — a tag
* (e.g. 'libero_sim') or a numeric cat_id
* QVAC_GROOT_SWITCH_EMBODIMENT after load, switch via vlaSetEmbodiment()
* and run inference on the new embodiment
* Both are no-ops you can omit; the GGUF's default embodiment is used. On a
* single-embodiment GGUF a switch is rejected by the addon.
*/
import { close, loadModel, GROOT_Q8_VF16, unloadModel, vla, vlaHparams, vlaSetEmbodiment, vlaPadState } from '@qvac/sdk';
// LIBERO GR00T prompt layout (Qwen3-VL tokenizer). GR00T reports
// `hparams.tokenizerMaxLength === 0` — it does not surface a tokenizer length,
// so the prompt length is fixed by the model: 2 cameras × 64 merged image
// tokens (256 patches, 2×2 merge) = 128 image tokens, plus text ≈ 150 total.
// A real consumer gets these from the tokenizer; the smoke hard-codes them.
// The prompt length follows the ACTIVE embodiment's camera count (a 4-camera
// DROID row needs 4 image-token runs, which a fixed 2-camera length cannot
// hold), so it is computed per inference, not a constant.
const IMAGE_TOKEN_ID = 151655;
const MERGED_TOKENS_PER_IMAGE = 64;
const PROMPT_TEXT_TAIL = 20;
const TEXT_TOKEN_ID = 1000;
const modelSrcOverride = process.argv[2];
const modelSrc = modelSrcOverride ?? GROOT_Q8_VF16;
// A tag string or a numeric cat_id, straight from the env. '24' is a cat_id,
// 'libero_sim' a tag.
function embodimentFromEnv(name) {
const env = process.env;
const raw = env[name];
if (raw === undefined || raw === '')
return undefined;
return /^\d+$/.test(raw) ? Number(raw) : raw;
}
const embodiment = embodimentFromEnv('QVAC_GROOT_EMBODIMENT');
const switchEmbodiment = embodimentFromEnv('QVAC_GROOT_SWITCH_EMBODIMENT');
try {
console.log('▸ Loading GR00T (N1.7-3B LIBERO) model...');
const modelId = await loadModel({
modelSrc,
modelType: 'ggml-vla',
modelConfig: { backend: 'cpu', ...(embodiment !== undefined && { embodiment }) },
onProgress: (p) => {
const mb = (n) => (n / 1e6).toFixed(1);
const line = `▸ Downloading ${p.percentage.toFixed(0)}% (${mb(p.downloaded)}/${mb(p.total)} MB)`;
process.stderr.write(process.stderr.isTTY ? `\r${line}` : `${line}\n`);
if (p.percentage >= 100)
process.stderr.write('\n');
}
});
if (typeof modelSrc !== 'string')
process.stderr.write('\n');
console.log(`▸ Model loaded: ${modelId}`);
const { hparams, backendName } = await vlaHparams({ modelId });
console.log(`▸ Backend: ${backendName ?? '(unknown)'}`);
console.log('▸ Hparams:', hparams);
if (hparams.selectedEmbodimentTag !== undefined) {
console.log(`▸ Embodiment: ${hparams.selectedEmbodimentTag} ` +
`(cat_id ${hparams.selectedEmbodimentCatId}, ${hparams.numCameras} cameras)`);
}
// Inputs are sized off the hparams, which follow the ACTIVE embodiment
// (numCameras, actionDim, ...) — rebuild them after every embodiment switch.
async function runInference(hp) {
const patchElems = hp.imagePatchElems;
if (hp.imageInputMode !== 'patches' || patchElems === undefined) {
throw new Error(`expected a patch-input model (imageInputMode 'patches' + imagePatchElems); ` +
`got imageInputMode=${hp.imageInputMode}`);
}
const numCameras = hp.numCameras ?? 2;
// Patch-input model: each camera is a pre-patchified buffer of exactly
// `imagePatchElems` floats. A real consumer patchifies each camera frame the
// way `Gr00tPolicy` does (the model's learned patch-embed runs inside the
// addon); we use small synthetic values.
const images = Array.from({ length: numCameras }, () => new Float32Array(patchElems).fill(0.02));
// Continuous-state model: pad the robot state to `maxStateDim`.
const state = vlaPadState([0, 0, 0, 0, 0, 0], hp.maxStateDim);
// GR00T requires the noise prior (flow-matching; it is not sampled in-model).
const noise = new Float32Array(hp.chunkSize * hp.maxActionDim);
// Prompt: one run of `MERGED_TOKENS_PER_IMAGE` image tokens per camera,
// each followed by a text separator, plus a short text tail. Sized off the
// active embodiment's camera count — a fixed 2-camera length would
// silently truncate the image-token runs of cameras 3+ after a switch to
// e.g. the 4-camera DROID row.
const promptLength = numCameras * (MERGED_TOKENS_PER_IMAGE + 1) + PROMPT_TEXT_TAIL;
const tokens = new Int32Array(promptLength);
let w = 0;
for (let cam = 0; cam < numCameras; cam++) {
for (let k = 0; k < MERGED_TOKENS_PER_IMAGE && w < tokens.length; k++) {
tokens[w++] = IMAGE_TOKEN_ID;
}
if (w < tokens.length)
tokens[w++] = TEXT_TOKEN_ID + cam;
}
for (; w < tokens.length; w++)
tokens[w] = TEXT_TOKEN_ID + w;
const mask = new Uint8Array(promptLength).fill(1);
console.log('▸ Running VLA inference...');
const { actions, actionDim, chunkSize, stats } = await vla({
modelId,
images,
// Patch inputs ignore imgWidth/imgHeight, but the request schema requires
// positive integers; pass the model's vision image size.
imgWidth: hp.visionImageSize,
imgHeight: hp.visionImageSize,
state,
tokens,
mask,
noise
});
console.log(`▸ Got ${chunkSize} action steps of dim ${actionDim}.`);
console.log(Array.from(actions.subarray(0, actionDim)));
if (stats) {
console.log(`▸ Timing: vision=${stats.vision_ms?.toFixed(0)}ms ` +
`ode=${stats.ode_ms?.toFixed(0)}ms ` +
`total=${stats.total_ms?.toFixed(0)}ms`);
}
}
await runInference(hparams);
if (switchEmbodiment !== undefined) {
console.log(`▸ Switching embodiment to ${JSON.stringify(switchEmbodiment)}...`);
const { hparams: refreshed } = await vlaSetEmbodiment({ modelId, embodiment: switchEmbodiment });
console.log(`▸ Embodiment: ${refreshed.selectedEmbodimentTag} ` +
`(cat_id ${refreshed.selectedEmbodimentCatId}, ${refreshed.numCameras} cameras)`);
await runInference(refreshed);
}
await unloadModel({ modelId, clearStorage: false });
console.log('▸ Model unloaded.');
process.exit(0);
}
catch (error) {
console.error('✖', error);
await close();
process.exit(1);
}Tip: all examples throughout this documentation are self-contained and runnable. For instructions on how to run them, see the JS/TS quickstart or the Python quickstart.