---
title: "VLA"
canonical: https://docs.qvac.tether.io/sdk/v0.18/ai-capabilities/vla/
collection: "SDK"
package: "@qvac/sdk"
line: v0.18
current_line: false
---
# VLA (/sdk/v0.18/ai-capabilities/vla)
## Overview
Vision-language-action (VLA) inference uses a **GGML** engine ([`@qvac/vla-ggml`](https://github.com/tetherto/qvac/tree/main/packages/vla-ggml)) to run VLA policies. Load a model using `modelType: "ggml-vla"`. Then, feed it preprocessed camera frames, the robot's current state, and a tokenized natural-language instruction; the model returns an **action chunk** — `chunkSize` future timesteps of an `actionDim`-dimensional action vector — to drive the robot's actuators.
The same `vla()` interface drives multiple policy families that differ in (i) how many camera views they expect, (ii) how they consume the robot state, and (iii) how they consume camera frames. `vlaHparams()` reports these per-model traits — `numCameras`, `stateInputMode`, and `imageInputMode` — so you can shape your inputs accordingly without hardcoding to a single architecture.
`imageInputMode` is either `"pixels"` or `"patches"`:
* `"pixels"`: each `images[]` entry is a `3·w·h` pixel plane, as prepared by `vlaPreprocessImage()`.
* `"patches"`: each `images[]` entry is a pre-patchified buffer of `vlaHparams().imagePatchElems` floats, not a `3·w·h` pixel plane. A real consumer patchifies each camera frame the way `Gr00tPolicy` does, and the model's learned patch-embed runs inside the addon.
`vla()` returns the produced action chunk together with per-stage timing stats.
## Functions
Use the following sequence of function calls:
1. [`loadModel()`](/sdk/v0.18/reference/api#loadmodel)
2. [`vlaHparams()`](/sdk/v0.18/reference/api#vlahparams) — to size your input buffers
3. [`vla()`](/sdk/v0.18/reference/api#vla)
4. [`unloadModel()`](/sdk/v0.18/reference/api#unloadmodel)
The SDK also exposes two helpers to prepare the wire-format tensors expected by `vla()`:
* `vlaPreprocessImage()`: prepares raw camera pixels into the image tensor `vla()` expects.
* `vlaPadState()`: zero-pads a robot-state vector to `hparams.maxStateDim` (continuous-state models only).
Both helpers are inlined client-side (no native binding required), so they work under Node, Bun, and Expo even without VLA prebuilds. The natural-language instruction must be tokenized on the consumer side using the model's tokenizer.
For how to use each function, see [SDK — API reference](/sdk/v0.18/reference/api/).
## Models
Supported model families:
* **[SmolVLA](https://huggingface.co/lerobot/smolvla_base)**: single all-in-one `*.gguf` file. Expects **2 camera views** and a **continuous** robot state. Available constant: `SMOLVLA_LIBERO_VISION_Q8`.
* **[π₀.₅ (pi05)](https://www.physicalintelligence.company/blog/pi05)**: single all-in-one `*.gguf` file. Expects **`vlaHparams().numCameras` camera views** (3 for `PI05_BASE_Q_AGGRESSIVE`) and a **discrete** robot state tokenized into the language prompt instead of a state buffer; the `noise` prior is required. Available constant: `PI05_BASE_Q_AGGRESSIVE`.
* **NVIDIA GR00T N1.7-3B (LIBERO)**: single all-in-one `*.gguf` file converted from the [LIBERO checkpoint](https://huggingface.co/nvidia/GR00T-N1.7-LIBERO). Expects **2 camera views** and **stores a single embodiment row** — `vlaSetEmbodiment()` is rejected. Reports `imageInputMode: "patches"` — each camera view is a pre-patchified buffer of `vlaHparams().imagePatchElems` floats, so `vlaPreprocessImage()` does not apply. Uses a **continuous** robot state (pad with `vlaPadState()`); the `noise` prior is required (flow-matching). Available constants: `GROOT_Q8_VF16` (desktop) and `GROOT_Q5_VF16` (mobile). Preferred for LIBERO-only use.
* **NVIDIA GR00T N1.7-3B (base, multi-embodiment)**: single all-in-one `*.gguf` file converted from the [base checkpoint](https://huggingface.co/nvidia/GR00T-N1.7-3B), carrying **multiple embodiments** — 17 rows shipped (out of up to 32), camera count per embodiment (see [GR00T embodiments](#gr00t-embodiments) below). Same input contract as the LIBERO build: `imageInputMode: "patches"` (each camera view is a pre-patchified buffer of `vlaHparams().imagePatchElems` floats, so `vlaPreprocessImage()` does not apply), **continuous** robot state (pad with `vlaPadState()`), `noise` prior required (flow-matching). Available constants: `GROOT_MULTI_Q8_VF16` (desktop) and `GROOT_MULTI_Q5_VF16` (mobile).
For models available as constants, see [SDK — Models](/sdk/v0.18/#models).
### GR00T embodiments
Multi-embodiment selection applies to the `GROOT_MULTI_*` constants. The LIBERO-only `GROOT_Q8_VF16` / `GROOT_Q5_VF16` GGUFs store a single embodiment row and reject any switch — the rest of this section is about `GROOT_MULTI_*`. A single `GROOT_MULTI_*` GGUF carries up to 32 embodiments; 17 rows ship in the current builds.
**Pick one at load** via `modelConfig.embodiment`. The selector accepts:
* a tag string (e.g. `"libero_sim"`),
* a numeric `cat_id` in `0..31`,
* an object `{ tag? | catId?, numCameras? }` — at most one of `tag` / `catId`, plus an optional `numCameras` override bounded to `1..64`. On the load path, an object with only `numCameras` and no `tag` or `catId` is valid: it keeps the GGUF's default embodiment and overrides the count.
* omitted — the GGUF's default embodiment is used.
**`numCameras` is not only an override for a rig with a different view count** — it is required for an embodiment the GGUF has no count for. Only a subset of the shipped rows are self-describing in the GGUF (9 of 17 in `GROOT_MULTI_*`), so most rows will not load or switch without it; the runtime never guesses.
**Switch at runtime** with `vlaSetEmbodiment({ modelId, embodiment })` — same selector shape, except a switch must name an embodiment (a `tag` or a `catId`; the bare `{ numCameras }` form is rejected). A switch reads \~20 MB from the GGUF versus reloading the whole \~4 GB file — rejected on a single-embodiment GGUF (`GROOT_Q8_VF16` / `GROOT_Q5_VF16`), and while a `vla()` response is outstanding (the queued switch throws `JOB_ALREADY_RUNNING` — await the in-flight `vla()` first).
`vlaHparams()` reports the resolved embodiment as `selectedEmbodimentTag` and `selectedEmbodimentCatId`, together with the active `numCameras` and `actionDim`. These fields are also present on the single-embodiment `GROOT_Q*` constants (they surface the baked-in row), so their presence is not a capability check — a switch may still be rejected. Rebuild your inference buffers from `vlaHparams()` after every `vlaSetEmbodiment()`.
## Examples
### SmolVLA
The following script loads SmolVLA-LIBERO from the registry, builds synthetic inputs (zero-filled gray images + BOS-only tokens + zero state), and runs a single inference pass — printing the produced action chunk and per-stage timings:
```js file=/packages/sdk/dist/examples/vla-smolvla.js title="vla-smolvla.js" lineNumbers
/**
* SmolVLA (vision-language-action) example using the QVAC SDK.
*
* Loads the SmolVLA-LIBERO GGUF model, runs a single inference pass with
* synthetic inputs (zero-filled gray images + BOS-only tokens + zero state +
* zero noise), and prints the produced action chunk + per-stage timings.
*
* Usage:
* bun examples/vla-smolvla.ts [path-to-smolvla.gguf]
*
* By default the example pulls the registry-baked SmolVLA-LIBERO GGUF
* (~1.9 GB) on first run and caches it locally. Pass an absolute path on
* the command line to override and load a local GGUF instead.
*/
import { close, loadModel, SMOLVLA_LIBERO_VISION_Q8, unloadModel, vla, vlaHparams, vlaPadState, vlaPreprocessImage } from '@qvac/sdk';
const modelSrcOverride = process.argv[2];
const modelSrc = modelSrcOverride ?? SMOLVLA_LIBERO_VISION_Q8;
try {
console.log('▸ Loading SmolVLA model...');
const modelId = await loadModel({
modelSrc,
modelType: 'ggml-vla',
modelConfig: { backend: 'cpu' },
onProgress: (p) => {
const mb = (n) => (n / 1e6).toFixed(1);
const line = `▸ Downloading ${p.percentage.toFixed(0)}% (${mb(p.downloaded)}/${mb(p.total)} MB)`;
process.stderr.write(process.stderr.isTTY ? `\r${line}` : `${line}\n`);
if (p.percentage >= 100)
process.stderr.write('\n');
}
});
if (typeof modelSrc !== 'string')
process.stderr.write('\n');
console.log(`▸ Model loaded: ${modelId}`);
const { hparams, backendName } = await vlaHparams({ modelId });
console.log(`▸ Backend: ${backendName ?? '(unknown)'}`);
console.log('▸ Hparams:', hparams);
// Build synthetic inputs sized to the model's expectations. A real
// consumer would: read camera frames, tokenize the instruction with the
// SmolVLM2 tokenizer, and read the robot's current end-effector pose.
const size = hparams.visionImageSize;
const dummyPixels = new Uint8Array(size * size * 3).fill(128);
const front = vlaPreprocessImage(dummyPixels, size, size, { size });
const wrist = vlaPreprocessImage(dummyPixels, size, size, { size });
const tokens = new Int32Array(hparams.tokenizerMaxLength);
const mask = new Uint8Array(hparams.tokenizerMaxLength);
// BOS-only "instruction" for the smoke test.
tokens[0] = 1;
mask[0] = 1;
const state = vlaPadState([0, 0, 0, 0, 0, 0], hparams.maxStateDim);
const noise = new Float32Array(hparams.chunkSize * hparams.maxActionDim);
console.log('▸ Running VLA inference...');
const { actions, actionDim, chunkSize, stats } = await vla({
modelId,
images: [front, wrist],
imgWidth: size,
imgHeight: size,
state,
tokens,
mask,
noise
});
console.log(`▸ Got ${chunkSize} action steps of dim ${actionDim}.`);
console.log(Array.from(actions.subarray(0, actionDim)));
if (stats) {
console.log(`▸ Timing: vision=${stats.vision_ms?.toFixed(0)}ms ` +
`smollm2=${stats.smollm2_total_ms?.toFixed(0)}ms ` +
`ode=${stats.ode_ms?.toFixed(0)}ms ` +
`total=${stats.total_ms?.toFixed(0)}ms`);
}
await unloadModel({ modelId, clearStorage: false });
console.log('▸ Model unloaded.');
process.exit(0);
}
catch (error) {
console.error('✖', error);
await close();
process.exit(1);
}
```
```ts file=/packages/sdk/examples/vla-smolvla.ts title="vla-smolvla.ts" lineNumbers
/**
* SmolVLA (vision-language-action) example using the QVAC SDK.
*
* Loads the SmolVLA-LIBERO GGUF model, runs a single inference pass with
* synthetic inputs (zero-filled gray images + BOS-only tokens + zero state +
* zero noise), and prints the produced action chunk + per-stage timings.
*
* Usage:
* bun examples/vla-smolvla.ts [path-to-smolvla.gguf]
*
* By default the example pulls the registry-baked SmolVLA-LIBERO GGUF
* (~1.9 GB) on first run and caches it locally. Pass an absolute path on
* the command line to override and load a local GGUF instead.
*/
import {
close,
loadModel,
SMOLVLA_LIBERO_VISION_Q8,
unloadModel,
vla,
vlaHparams,
vlaPadState,
vlaPreprocessImage
} from '@qvac/sdk'
const modelSrcOverride = process.argv[2]
const modelSrc = modelSrcOverride ?? SMOLVLA_LIBERO_VISION_Q8
try {
console.log('▸ Loading SmolVLA model...')
const modelId = await loadModel({
modelSrc,
modelType: 'ggml-vla',
modelConfig: { backend: 'cpu' },
onProgress: (p) => {
const mb = (n: number) => (n / 1e6).toFixed(1)
const line = `▸ Downloading ${p.percentage.toFixed(0)}% (${mb(p.downloaded)}/${mb(p.total)} MB)`
process.stderr.write(process.stderr.isTTY ? `\r${line}` : `${line}\n`)
if (p.percentage >= 100) process.stderr.write('\n')
}
})
if (typeof modelSrc !== 'string') process.stderr.write('\n')
console.log(`▸ Model loaded: ${modelId}`)
const { hparams, backendName } = await vlaHparams({ modelId })
console.log(`▸ Backend: ${backendName ?? '(unknown)'}`)
console.log('▸ Hparams:', hparams)
// Build synthetic inputs sized to the model's expectations. A real
// consumer would: read camera frames, tokenize the instruction with the
// SmolVLM2 tokenizer, and read the robot's current end-effector pose.
const size = hparams.visionImageSize
const dummyPixels = new Uint8Array(size * size * 3).fill(128)
const front = vlaPreprocessImage(dummyPixels, size, size, { size })
const wrist = vlaPreprocessImage(dummyPixels, size, size, { size })
const tokens = new Int32Array(hparams.tokenizerMaxLength)
const mask = new Uint8Array(hparams.tokenizerMaxLength)
// BOS-only "instruction" for the smoke test.
tokens[0] = 1
mask[0] = 1
const state = vlaPadState([0, 0, 0, 0, 0, 0], hparams.maxStateDim)
const noise = new Float32Array(hparams.chunkSize * hparams.maxActionDim)
console.log('▸ Running VLA inference...')
const { actions, actionDim, chunkSize, stats } = await vla({
modelId,
images: [front, wrist],
imgWidth: size,
imgHeight: size,
state,
tokens,
mask,
noise
})
console.log(`▸ Got ${chunkSize} action steps of dim ${actionDim}.`)
console.log(Array.from(actions.subarray(0, actionDim)))
if (stats) {
console.log(
`▸ Timing: vision=${stats.vision_ms?.toFixed(0)}ms ` +
`smollm2=${stats.smollm2_total_ms?.toFixed(0)}ms ` +
`ode=${stats.ode_ms?.toFixed(0)}ms ` +
`total=${stats.total_ms?.toFixed(0)}ms`
)
}
await unloadModel({ modelId, clearStorage: false })
console.log('▸ Model unloaded.')
process.exit(0)
} catch (error) {
console.error('✖', error)
await close()
process.exit(1)
}
```
```python file=/packages/sdk-python/examples/vla.py title="vla.py" lineNumbers
"""Python port of packages/sdk/examples/vla-smolvla.ts.
Vision-language-action inference with SmolVLA. Loads the GGUF model, reads the
model's hyperparameters, builds synthetic inputs sized to those hparams
(gray images + BOS-only tokens + zero state + zero noise), runs one inference
pass, and prints the produced action chunk.
The VLA helpers are numpy-based, so this example is guarded on numpy:
from tetherto.qvac_sdk import vla, vla_hparams, vla_preprocess_image, vla_pad_state
RUN: python examples/vla.py [path-to-smolvla.gguf]
"""
from __future__ import annotations
import asyncio
import sys
from tetherto.qvac_sdk import (
Client,
load_model,
unload_model,
vla,
vla_hparams,
vla_pad_state,
vla_preprocess_image,
)
from tetherto.qvac_sdk.models import SMOLVLA_LIBERO_VISION_Q8
def print_progress(p) -> None:
"""Print model download progress; pass as `on_progress=` to `load_model`."""
line = (
f"▸ Downloading {p.percentage:.0f}% "
f"({p.downloaded / 1e6:.1f}/{p.total / 1e6:.1f} MB)"
)
print(line, end="\r" if sys.stderr.isatty() else "\n", file=sys.stderr)
if p.percentage >= 100:
print(file=sys.stderr)
async def main() -> int:
try:
import numpy as np
except ImportError:
print("✖ This example needs numpy: pip install numpy", file=sys.stderr)
return 1
model_src = sys.argv[1] if len(sys.argv) > 1 else SMOLVLA_LIBERO_VISION_Q8
async with Client() as client:
t = client.transport
try:
print("▸ Loading SmolVLA model...")
model_id = await load_model(
t,
model_src=model_src,
model_type="ggml-vla",
model_config={"backend": "cpu"},
on_progress=print_progress,
)
print(f"▸ Model loaded: {model_id}")
hparams, backend_name = await vla_hparams(t, model_id=model_id)
print(f"▸ Backend: {backend_name or '(unknown)'}")
print(f"▸ Hparams: {hparams}")
# Synthetic inputs sized to the model's expectations. A real consumer
# would read camera frames, tokenize the instruction with the
# SmolVLM2 tokenizer, and read the robot's current end-effector pose.
size = hparams.vision_image_size
dummy_pixels = np.full((size, size, 3), 128, dtype=np.uint8)
front = vla_preprocess_image(dummy_pixels, size, size, size=size)
wrist = vla_preprocess_image(dummy_pixels, size, size, size=size)
tokens = np.zeros(hparams.tokenizer_max_length, dtype=np.int32)
mask = np.zeros(hparams.tokenizer_max_length, dtype=np.uint8)
# BOS-only "instruction" for the smoke test.
tokens[0] = 1
mask[0] = 1
state = vla_pad_state([0, 0, 0, 0, 0, 0], hparams.max_state_dim)
noise = np.zeros(
hparams.chunk_size * hparams.max_action_dim, dtype=np.float32
)
print("▸ Running VLA inference...")
result = await vla(
t,
model_id=model_id,
images=[front, wrist],
img_width=size,
img_height=size,
state=state,
tokens=tokens,
mask=mask,
noise=noise,
)
print(f"▸ Got {result.chunk_size} action steps of dim {result.action_dim}.")
print(list(result.actions[: result.action_dim]))
if result.stats:
print(f"▸ Timing: {result.stats}")
await unload_model(t, model_id)
print("▸ Model unloaded.")
except Exception as error:
print(f"✖ {error}", file=sys.stderr)
return 1
return 0
if __name__ == "__main__":
sys.exit(asyncio.run(main()))
```
### pi05
The following script loads pi05 from the registry, builds synthetic inputs (zero-filled gray images + BOS-only tokens + empty state), and runs a single inference pass — printing the produced action chunk and per-stage timings:
```js file=/packages/sdk/dist/examples/vla-pi05.js title="vla-pi05.js" lineNumbers
/**
* π₀.₅ (pi05) vision-language-action example using the QVAC SDK.
*
* Loads the Physical Intelligence π₀.₅ GGUF model, runs a single inference
* pass with synthetic inputs (zero-filled gray images + BOS-only tokens +
* seeded noise), and prints the produced action chunk + per-stage timings.
*
* π₀.₅ differs from SmolVLA in two ways the SDK surfaces via `vlaHparams()`:
* - `numCameras: 3` — it expects exactly three camera frames (not two).
* - `stateInputMode: 'discrete'` — the robot state is tokenised into the
* language prompt, so the `state` buffer is ignored. We pass an empty
* `Float32Array(0)`. π₀.₅ also requires the `noise` prior.
*
* Usage:
* bun examples/vla-pi05.ts [path-to-pi05.gguf]
*
* By default the example pulls the registry-baked π₀.₅ GGUF (~3.9 GB) on
* first run and caches it locally. Pass an absolute path on the command line
* to override and load a local GGUF instead.
*/
import { close, loadModel, PI05_BASE_Q_AGGRESSIVE, unloadModel, vla, vlaHparams, vlaPreprocessImage } from '@qvac/sdk';
const modelSrcOverride = process.argv[2];
const modelSrc = modelSrcOverride ?? PI05_BASE_Q_AGGRESSIVE;
try {
console.log('▸ Loading π₀.₅ (pi05) model...');
const modelId = await loadModel({
modelSrc,
modelType: 'ggml-vla',
modelConfig: { backend: 'cpu' },
onProgress: (p) => {
const mb = (n) => (n / 1e6).toFixed(1);
const line = `▸ Downloading ${p.percentage.toFixed(0)}% (${mb(p.downloaded)}/${mb(p.total)} MB)`;
process.stderr.write(process.stderr.isTTY ? `\r${line}` : `${line}\n`);
if (p.percentage >= 100)
process.stderr.write('\n');
}
});
if (typeof modelSrc !== 'string')
process.stderr.write('\n');
console.log(`▸ Model loaded: ${modelId}`);
const { hparams, backendName } = await vlaHparams({ modelId });
console.log(`▸ Backend: ${backendName ?? '(unknown)'}`);
console.log('▸ Hparams:', hparams);
// Build synthetic inputs sized to the model's expectations. A real
// consumer would: read camera frames, tokenize the instruction with the
// model's tokenizer, and (for π₀.₅) inline the robot state into the prompt.
const size = hparams.visionImageSize;
const numCameras = hparams.numCameras ?? 3;
const dummyPixels = new Uint8Array(size * size * 3).fill(128);
// π₀.₅ expects exactly `numCameras` frames.
const images = Array.from({ length: numCameras }, () => vlaPreprocessImage(dummyPixels, size, size, { size }));
const tokens = new Int32Array(hparams.tokenizerMaxLength);
const mask = new Uint8Array(hparams.tokenizerMaxLength);
// BOS-only "instruction" for the smoke test.
tokens[0] = 1;
mask[0] = 1;
// Discrete-state model: the state buffer is ignored (state is tokenised
// into the prompt), so pass an empty Float32Array. π₀.₅ requires `noise`.
const state = new Float32Array(0);
const noise = new Float32Array(hparams.chunkSize * hparams.maxActionDim);
console.log('▸ Running VLA inference...');
const { actions, actionDim, chunkSize, stats } = await vla({
modelId,
images,
imgWidth: size,
imgHeight: size,
state,
tokens,
mask,
noise
});
console.log(`▸ Got ${chunkSize} action steps of dim ${actionDim}.`);
console.log(Array.from(actions.subarray(0, actionDim)));
if (stats) {
console.log(`▸ Timing: vision=${stats.vision_ms?.toFixed(0)}ms ` +
`prefill=${stats.prefill_total_ms?.toFixed(0)}ms ` +
`ode=${stats.ode_ms?.toFixed(0)}ms ` +
`total=${stats.total_ms?.toFixed(0)}ms`);
}
await unloadModel({ modelId, clearStorage: false });
console.log('▸ Model unloaded.');
process.exit(0);
}
catch (error) {
console.error('✖', error);
await close();
process.exit(1);
}
```
```ts file=/packages/sdk/examples/vla-pi05.ts title="vla-pi05.ts" lineNumbers
/**
* π₀.₅ (pi05) vision-language-action example using the QVAC SDK.
*
* Loads the Physical Intelligence π₀.₅ GGUF model, runs a single inference
* pass with synthetic inputs (zero-filled gray images + BOS-only tokens +
* seeded noise), and prints the produced action chunk + per-stage timings.
*
* π₀.₅ differs from SmolVLA in two ways the SDK surfaces via `vlaHparams()`:
* - `numCameras: 3` — it expects exactly three camera frames (not two).
* - `stateInputMode: 'discrete'` — the robot state is tokenised into the
* language prompt, so the `state` buffer is ignored. We pass an empty
* `Float32Array(0)`. π₀.₅ also requires the `noise` prior.
*
* Usage:
* bun examples/vla-pi05.ts [path-to-pi05.gguf]
*
* By default the example pulls the registry-baked π₀.₅ GGUF (~3.9 GB) on
* first run and caches it locally. Pass an absolute path on the command line
* to override and load a local GGUF instead.
*/
import {
close,
loadModel,
PI05_BASE_Q_AGGRESSIVE,
unloadModel,
vla,
vlaHparams,
vlaPreprocessImage
} from '@qvac/sdk'
const modelSrcOverride = process.argv[2]
const modelSrc = modelSrcOverride ?? PI05_BASE_Q_AGGRESSIVE
try {
console.log('▸ Loading π₀.₅ (pi05) model...')
const modelId = await loadModel({
modelSrc,
modelType: 'ggml-vla',
modelConfig: { backend: 'cpu' },
onProgress: (p) => {
const mb = (n: number) => (n / 1e6).toFixed(1)
const line = `▸ Downloading ${p.percentage.toFixed(0)}% (${mb(p.downloaded)}/${mb(p.total)} MB)`
process.stderr.write(process.stderr.isTTY ? `\r${line}` : `${line}\n`)
if (p.percentage >= 100) process.stderr.write('\n')
}
})
if (typeof modelSrc !== 'string') process.stderr.write('\n')
console.log(`▸ Model loaded: ${modelId}`)
const { hparams, backendName } = await vlaHparams({ modelId })
console.log(`▸ Backend: ${backendName ?? '(unknown)'}`)
console.log('▸ Hparams:', hparams)
// Build synthetic inputs sized to the model's expectations. A real
// consumer would: read camera frames, tokenize the instruction with the
// model's tokenizer, and (for π₀.₅) inline the robot state into the prompt.
const size = hparams.visionImageSize
const numCameras = hparams.numCameras ?? 3
const dummyPixels = new Uint8Array(size * size * 3).fill(128)
// π₀.₅ expects exactly `numCameras` frames.
const images = Array.from({ length: numCameras }, () =>
vlaPreprocessImage(dummyPixels, size, size, { size })
)
const tokens = new Int32Array(hparams.tokenizerMaxLength)
const mask = new Uint8Array(hparams.tokenizerMaxLength)
// BOS-only "instruction" for the smoke test.
tokens[0] = 1
mask[0] = 1
// Discrete-state model: the state buffer is ignored (state is tokenised
// into the prompt), so pass an empty Float32Array. π₀.₅ requires `noise`.
const state = new Float32Array(0)
const noise = new Float32Array(hparams.chunkSize * hparams.maxActionDim)
console.log('▸ Running VLA inference...')
const { actions, actionDim, chunkSize, stats } = await vla({
modelId,
images,
imgWidth: size,
imgHeight: size,
state,
tokens,
mask,
noise
})
console.log(`▸ Got ${chunkSize} action steps of dim ${actionDim}.`)
console.log(Array.from(actions.subarray(0, actionDim)))
if (stats) {
console.log(
`▸ Timing: vision=${stats.vision_ms?.toFixed(0)}ms ` +
`prefill=${stats.prefill_total_ms?.toFixed(0)}ms ` +
`ode=${stats.ode_ms?.toFixed(0)}ms ` +
`total=${stats.total_ms?.toFixed(0)}ms`
)
}
await unloadModel({ modelId, clearStorage: false })
console.log('▸ Model unloaded.')
process.exit(0)
} catch (error) {
console.error('✖', error)
await close()
process.exit(1)
}
```
### GR00T
The following script loads a GR00T N1.7-3B GGUF from the registry and runs a single inference pass with synthetic inputs. Unlike SmolVLA and pi05, GR00T reports `imageInputMode: "patches"`: each camera entry is a pre-patchified buffer of `hparams.imagePatchElems` floats, the continuous state is padded with `vlaPadState()`, and the `noise` prior is required.
Two optional environment variables drive the multi-embodiment path exposed by `GROOT_MULTI_*`:
* `QVAC_GROOT_EMBODIMENT` selects the embodiment at load — a tag string (e.g. `libero_sim`) or a numeric `cat_id`.
* `QVAC_GROOT_SWITCH_EMBODIMENT` triggers a runtime `vlaSetEmbodiment()` after the first inference pass; the example then rebuilds its inputs from the refreshed `vlaHparams()` and runs a second pass on the new embodiment.
The example defaults to `GROOT_Q8_VF16` (the LIBERO single-embodiment build), on which a switch is rejected. To exercise `QVAC_GROOT_SWITCH_EMBODIMENT`, pass a local `GROOT_MULTI_*` GGUF path on the command line — the multi-embodiment constants are supported by `@qvac/vla-ggml` 0.17.0 and later.
```js file=/packages/sdk/dist/examples/vla-groot.js title="vla-groot.js" lineNumbers
/**
* NVIDIA GR00T N1.7-3B (vision-language-action) example using the QVAC SDK.
*
* Loads the GR00T-N1.7-3B LIBERO GGUF model, runs a single inference pass with
* synthetic inputs, and prints the produced action chunk + per-stage timings.
*
* GR00T differs from SmolVLA / π₀.₅ in the ways the SDK surfaces via
* `vlaHparams()`:
* - `imageInputMode: 'patches'` — each `images` entry is a pre-patchified
* buffer of `hparams.imagePatchElems` floats, NOT a `3·w·h` pixel plane.
* A real consumer patchifies each camera frame the way `Gr00tPolicy` does
* (the model's learned patch-embed runs inside the addon); here we feed
* synthetic patch buffers of the right length.
* - `stateInputMode: 'continuous'` — the robot state is projected by an
* in-model linear layer; pad it to `hparams.maxStateDim` with `vlaPadState`.
* - `noise` is REQUIRED. GR00T is a flow-matching model that does not sample
* its own prior; a missing `noise` is rejected as INVALID_INPUT.
*
* The prompt must place one contiguous run of `MERGED_TOKENS_PER_IMAGE`
* image-placeholder tokens per camera. A real consumer produces this by
* tokenising the instruction with the Qwen3-VL tokenizer, which inserts the
* image tokens; we lay it out by hand for the smoke.
*
* Usage:
* bun examples/vla-groot.ts [path-to-groot.gguf]
*
* By default the example pulls the registry-baked GR00T-LIBERO GGUF (~3.76 GB)
* on first run and caches it locally. Pass an absolute path on the command line
* to override and load a local GGUF instead.
*
* Multi-embodiment GGUFs (vla-ggml >= 0.17.0): one GGUF can carry many
* embodiments, one active at a time. Optional env vars demonstrate selection:
* QVAC_GROOT_EMBODIMENT initial selection at load — a tag
* (e.g. 'libero_sim') or a numeric cat_id
* QVAC_GROOT_SWITCH_EMBODIMENT after load, switch via vlaSetEmbodiment()
* and run inference on the new embodiment
* Both are no-ops you can omit; the GGUF's default embodiment is used. On a
* single-embodiment GGUF a switch is rejected by the addon.
*/
import { close, loadModel, GROOT_Q8_VF16, unloadModel, vla, vlaHparams, vlaSetEmbodiment, vlaPadState } from '@qvac/sdk';
// LIBERO GR00T prompt layout (Qwen3-VL tokenizer). GR00T reports
// `hparams.tokenizerMaxLength === 0` — it does not surface a tokenizer length,
// so the prompt length is fixed by the model: 2 cameras × 64 merged image
// tokens (256 patches, 2×2 merge) = 128 image tokens, plus text ≈ 150 total.
// A real consumer gets these from the tokenizer; the smoke hard-codes them.
// The prompt length follows the ACTIVE embodiment's camera count (a 4-camera
// DROID row needs 4 image-token runs, which a fixed 2-camera length cannot
// hold), so it is computed per inference, not a constant.
const IMAGE_TOKEN_ID = 151655;
const MERGED_TOKENS_PER_IMAGE = 64;
const PROMPT_TEXT_TAIL = 20;
const TEXT_TOKEN_ID = 1000;
const modelSrcOverride = process.argv[2];
const modelSrc = modelSrcOverride ?? GROOT_Q8_VF16;
// A tag string or a numeric cat_id, straight from the env. '24' is a cat_id,
// 'libero_sim' a tag.
function embodimentFromEnv(name) {
const env = process.env;
const raw = env[name];
if (raw === undefined || raw === '')
return undefined;
return /^\d+$/.test(raw) ? Number(raw) : raw;
}
const embodiment = embodimentFromEnv('QVAC_GROOT_EMBODIMENT');
const switchEmbodiment = embodimentFromEnv('QVAC_GROOT_SWITCH_EMBODIMENT');
try {
console.log('▸ Loading GR00T (N1.7-3B LIBERO) model...');
const modelId = await loadModel({
modelSrc,
modelType: 'ggml-vla',
modelConfig: { backend: 'cpu', ...(embodiment !== undefined && { embodiment }) },
onProgress: (p) => {
const mb = (n) => (n / 1e6).toFixed(1);
const line = `▸ Downloading ${p.percentage.toFixed(0)}% (${mb(p.downloaded)}/${mb(p.total)} MB)`;
process.stderr.write(process.stderr.isTTY ? `\r${line}` : `${line}\n`);
if (p.percentage >= 100)
process.stderr.write('\n');
}
});
if (typeof modelSrc !== 'string')
process.stderr.write('\n');
console.log(`▸ Model loaded: ${modelId}`);
const { hparams, backendName } = await vlaHparams({ modelId });
console.log(`▸ Backend: ${backendName ?? '(unknown)'}`);
console.log('▸ Hparams:', hparams);
if (hparams.selectedEmbodimentTag !== undefined) {
console.log(`▸ Embodiment: ${hparams.selectedEmbodimentTag} ` +
`(cat_id ${hparams.selectedEmbodimentCatId}, ${hparams.numCameras} cameras)`);
}
// Inputs are sized off the hparams, which follow the ACTIVE embodiment
// (numCameras, actionDim, ...) — rebuild them after every embodiment switch.
async function runInference(hp) {
const patchElems = hp.imagePatchElems;
if (hp.imageInputMode !== 'patches' || patchElems === undefined) {
throw new Error(`expected a patch-input model (imageInputMode 'patches' + imagePatchElems); ` +
`got imageInputMode=${hp.imageInputMode}`);
}
const numCameras = hp.numCameras ?? 2;
// Patch-input model: each camera is a pre-patchified buffer of exactly
// `imagePatchElems` floats. A real consumer patchifies each camera frame the
// way `Gr00tPolicy` does (the model's learned patch-embed runs inside the
// addon); we use small synthetic values.
const images = Array.from({ length: numCameras }, () => new Float32Array(patchElems).fill(0.02));
// Continuous-state model: pad the robot state to `maxStateDim`.
const state = vlaPadState([0, 0, 0, 0, 0, 0], hp.maxStateDim);
// GR00T requires the noise prior (flow-matching; it is not sampled in-model).
const noise = new Float32Array(hp.chunkSize * hp.maxActionDim);
// Prompt: one run of `MERGED_TOKENS_PER_IMAGE` image tokens per camera,
// each followed by a text separator, plus a short text tail. Sized off the
// active embodiment's camera count — a fixed 2-camera length would
// silently truncate the image-token runs of cameras 3+ after a switch to
// e.g. the 4-camera DROID row.
const promptLength = numCameras * (MERGED_TOKENS_PER_IMAGE + 1) + PROMPT_TEXT_TAIL;
const tokens = new Int32Array(promptLength);
let w = 0;
for (let cam = 0; cam < numCameras; cam++) {
for (let k = 0; k < MERGED_TOKENS_PER_IMAGE && w < tokens.length; k++) {
tokens[w++] = IMAGE_TOKEN_ID;
}
if (w < tokens.length)
tokens[w++] = TEXT_TOKEN_ID + cam;
}
for (; w < tokens.length; w++)
tokens[w] = TEXT_TOKEN_ID + w;
const mask = new Uint8Array(promptLength).fill(1);
console.log('▸ Running VLA inference...');
const { actions, actionDim, chunkSize, stats } = await vla({
modelId,
images,
// Patch inputs ignore imgWidth/imgHeight, but the request schema requires
// positive integers; pass the model's vision image size.
imgWidth: hp.visionImageSize,
imgHeight: hp.visionImageSize,
state,
tokens,
mask,
noise
});
console.log(`▸ Got ${chunkSize} action steps of dim ${actionDim}.`);
console.log(Array.from(actions.subarray(0, actionDim)));
if (stats) {
console.log(`▸ Timing: vision=${stats.vision_ms?.toFixed(0)}ms ` +
`ode=${stats.ode_ms?.toFixed(0)}ms ` +
`total=${stats.total_ms?.toFixed(0)}ms`);
}
}
await runInference(hparams);
if (switchEmbodiment !== undefined) {
console.log(`▸ Switching embodiment to ${JSON.stringify(switchEmbodiment)}...`);
const { hparams: refreshed } = await vlaSetEmbodiment({ modelId, embodiment: switchEmbodiment });
console.log(`▸ Embodiment: ${refreshed.selectedEmbodimentTag} ` +
`(cat_id ${refreshed.selectedEmbodimentCatId}, ${refreshed.numCameras} cameras)`);
await runInference(refreshed);
}
await unloadModel({ modelId, clearStorage: false });
console.log('▸ Model unloaded.');
process.exit(0);
}
catch (error) {
console.error('✖', error);
await close();
process.exit(1);
}
```
```ts file=/packages/sdk/examples/vla-groot.ts title="vla-groot.ts" lineNumbers
/**
* NVIDIA GR00T N1.7-3B (vision-language-action) example using the QVAC SDK.
*
* Loads the GR00T-N1.7-3B LIBERO GGUF model, runs a single inference pass with
* synthetic inputs, and prints the produced action chunk + per-stage timings.
*
* GR00T differs from SmolVLA / π₀.₅ in the ways the SDK surfaces via
* `vlaHparams()`:
* - `imageInputMode: 'patches'` — each `images` entry is a pre-patchified
* buffer of `hparams.imagePatchElems` floats, NOT a `3·w·h` pixel plane.
* A real consumer patchifies each camera frame the way `Gr00tPolicy` does
* (the model's learned patch-embed runs inside the addon); here we feed
* synthetic patch buffers of the right length.
* - `stateInputMode: 'continuous'` — the robot state is projected by an
* in-model linear layer; pad it to `hparams.maxStateDim` with `vlaPadState`.
* - `noise` is REQUIRED. GR00T is a flow-matching model that does not sample
* its own prior; a missing `noise` is rejected as INVALID_INPUT.
*
* The prompt must place one contiguous run of `MERGED_TOKENS_PER_IMAGE`
* image-placeholder tokens per camera. A real consumer produces this by
* tokenising the instruction with the Qwen3-VL tokenizer, which inserts the
* image tokens; we lay it out by hand for the smoke.
*
* Usage:
* bun examples/vla-groot.ts [path-to-groot.gguf]
*
* By default the example pulls the registry-baked GR00T-LIBERO GGUF (~3.76 GB)
* on first run and caches it locally. Pass an absolute path on the command line
* to override and load a local GGUF instead.
*
* Multi-embodiment GGUFs (vla-ggml >= 0.17.0): one GGUF can carry many
* embodiments, one active at a time. Optional env vars demonstrate selection:
* QVAC_GROOT_EMBODIMENT initial selection at load — a tag
* (e.g. 'libero_sim') or a numeric cat_id
* QVAC_GROOT_SWITCH_EMBODIMENT after load, switch via vlaSetEmbodiment()
* and run inference on the new embodiment
* Both are no-ops you can omit; the GGUF's default embodiment is used. On a
* single-embodiment GGUF a switch is rejected by the addon.
*/
import {
close,
loadModel,
GROOT_Q8_VF16,
unloadModel,
vla,
vlaHparams,
vlaSetEmbodiment,
vlaPadState,
type VlaEmbodimentSelection
} from '@qvac/sdk'
// LIBERO GR00T prompt layout (Qwen3-VL tokenizer). GR00T reports
// `hparams.tokenizerMaxLength === 0` — it does not surface a tokenizer length,
// so the prompt length is fixed by the model: 2 cameras × 64 merged image
// tokens (256 patches, 2×2 merge) = 128 image tokens, plus text ≈ 150 total.
// A real consumer gets these from the tokenizer; the smoke hard-codes them.
// The prompt length follows the ACTIVE embodiment's camera count (a 4-camera
// DROID row needs 4 image-token runs, which a fixed 2-camera length cannot
// hold), so it is computed per inference, not a constant.
const IMAGE_TOKEN_ID = 151655
const MERGED_TOKENS_PER_IMAGE = 64
const PROMPT_TEXT_TAIL = 20
const TEXT_TOKEN_ID = 1000
const modelSrcOverride = process.argv[2]
const modelSrc = modelSrcOverride ?? GROOT_Q8_VF16
// A tag string or a numeric cat_id, straight from the env. '24' is a cat_id,
// 'libero_sim' a tag.
function embodimentFromEnv(name: string): VlaEmbodimentSelection | undefined {
const env = process.env as Record
const raw = env[name]
if (raw === undefined || raw === '') return undefined
return /^\d+$/.test(raw) ? Number(raw) : raw
}
const embodiment = embodimentFromEnv('QVAC_GROOT_EMBODIMENT')
const switchEmbodiment = embodimentFromEnv('QVAC_GROOT_SWITCH_EMBODIMENT')
try {
console.log('▸ Loading GR00T (N1.7-3B LIBERO) model...')
const modelId = await loadModel({
modelSrc,
modelType: 'ggml-vla',
modelConfig: { backend: 'cpu', ...(embodiment !== undefined && { embodiment }) },
onProgress: (p) => {
const mb = (n: number) => (n / 1e6).toFixed(1)
const line = `▸ Downloading ${p.percentage.toFixed(0)}% (${mb(p.downloaded)}/${mb(p.total)} MB)`
process.stderr.write(process.stderr.isTTY ? `\r${line}` : `${line}\n`)
if (p.percentage >= 100) process.stderr.write('\n')
}
})
if (typeof modelSrc !== 'string') process.stderr.write('\n')
console.log(`▸ Model loaded: ${modelId}`)
const { hparams, backendName } = await vlaHparams({ modelId })
console.log(`▸ Backend: ${backendName ?? '(unknown)'}`)
console.log('▸ Hparams:', hparams)
if (hparams.selectedEmbodimentTag !== undefined) {
console.log(
`▸ Embodiment: ${hparams.selectedEmbodimentTag} ` +
`(cat_id ${hparams.selectedEmbodimentCatId}, ${hparams.numCameras} cameras)`
)
}
// Inputs are sized off the hparams, which follow the ACTIVE embodiment
// (numCameras, actionDim, ...) — rebuild them after every embodiment switch.
async function runInference(hp: typeof hparams) {
const patchElems = hp.imagePatchElems
if (hp.imageInputMode !== 'patches' || patchElems === undefined) {
throw new Error(
`expected a patch-input model (imageInputMode 'patches' + imagePatchElems); ` +
`got imageInputMode=${hp.imageInputMode}`
)
}
const numCameras = hp.numCameras ?? 2
// Patch-input model: each camera is a pre-patchified buffer of exactly
// `imagePatchElems` floats. A real consumer patchifies each camera frame the
// way `Gr00tPolicy` does (the model's learned patch-embed runs inside the
// addon); we use small synthetic values.
const images = Array.from({ length: numCameras }, () => new Float32Array(patchElems).fill(0.02))
// Continuous-state model: pad the robot state to `maxStateDim`.
const state = vlaPadState([0, 0, 0, 0, 0, 0], hp.maxStateDim)
// GR00T requires the noise prior (flow-matching; it is not sampled in-model).
const noise = new Float32Array(hp.chunkSize * hp.maxActionDim)
// Prompt: one run of `MERGED_TOKENS_PER_IMAGE` image tokens per camera,
// each followed by a text separator, plus a short text tail. Sized off the
// active embodiment's camera count — a fixed 2-camera length would
// silently truncate the image-token runs of cameras 3+ after a switch to
// e.g. the 4-camera DROID row.
const promptLength = numCameras * (MERGED_TOKENS_PER_IMAGE + 1) + PROMPT_TEXT_TAIL
const tokens = new Int32Array(promptLength)
let w = 0
for (let cam = 0; cam < numCameras; cam++) {
for (let k = 0; k < MERGED_TOKENS_PER_IMAGE && w < tokens.length; k++) {
tokens[w++] = IMAGE_TOKEN_ID
}
if (w < tokens.length) tokens[w++] = TEXT_TOKEN_ID + cam
}
for (; w < tokens.length; w++) tokens[w] = TEXT_TOKEN_ID + w
const mask = new Uint8Array(promptLength).fill(1)
console.log('▸ Running VLA inference...')
const { actions, actionDim, chunkSize, stats } = await vla({
modelId,
images,
// Patch inputs ignore imgWidth/imgHeight, but the request schema requires
// positive integers; pass the model's vision image size.
imgWidth: hp.visionImageSize,
imgHeight: hp.visionImageSize,
state,
tokens,
mask,
noise
})
console.log(`▸ Got ${chunkSize} action steps of dim ${actionDim}.`)
console.log(Array.from(actions.subarray(0, actionDim)))
if (stats) {
console.log(
`▸ Timing: vision=${stats.vision_ms?.toFixed(0)}ms ` +
`ode=${stats.ode_ms?.toFixed(0)}ms ` +
`total=${stats.total_ms?.toFixed(0)}ms`
)
}
}
await runInference(hparams)
if (switchEmbodiment !== undefined) {
console.log(`▸ Switching embodiment to ${JSON.stringify(switchEmbodiment)}...`)
const { hparams: refreshed } = await vlaSetEmbodiment({ modelId, embodiment: switchEmbodiment })
console.log(
`▸ Embodiment: ${refreshed.selectedEmbodimentTag} ` +
`(cat_id ${refreshed.selectedEmbodimentCatId}, ${refreshed.numCameras} cameras)`
)
await runInference(refreshed)
}
await unloadModel({ modelId, clearStorage: false })
console.log('▸ Model unloaded.')
process.exit(0)
} catch (error) {
console.error('✖', error)
await close()
process.exit(1)
}
```
**Tip:** all examples throughout this documentation are self-contained and runnable. For instructions on how to run them, see the [JS/TS quickstart](/sdk/v0.18/js-ts-sdk#quickstart) or the [Python quickstart](/sdk/v0.18/python-sdk#quickstart).