New: QVAC SDK now supports Kotlin. Build local AI into your JVM and Android apps.
QVAC Logo
EcosystemAddonsdiffusion-cpp

@qvac/diffusion-cpp

Text-to-image generation and reference image editing.

Overview

Bare module for text-to-image generation and reference image editing in QVAC, using qvac-ext-stable-diffusion.cpp as the inference engine.

Models

Supports Qwen-Image-2.1, FLUX.2-klein, SD1.x, SD2.x, SDXL, and SD3 model families.

Qwen-Image-2.1

Three separate components are required for text-to-image:

  • Diffusion model (qwen_image_2.1-Q4_K.gguf) - the main image transformer.
  • Text encoder (Qwen3VL-8B-Instruct-Q4_K_M.gguf) - the matching Qwen3-VL encoder.
  • VAE (qwen_image_2.1_vae_bf16.safetensors) - the 2.1 VAE; earlier Qwen Image and Wan VAEs are incompatible.

Reference editing also requires the vision projector (mmproj-Qwen3VL-8B-Instruct-F16.gguf) as files.llmVision, separate from the video API's clipVision.

Model file reference for Qwen-Image-2.1

RoleFileSource
Diffusion modelqwen_image_2.1-Q4_K.ggufleejet/Qwen-Image-2.1-GGUF
Text encoderQwen3VL-8B-Instruct-Q4_K_M.ggufQwen/Qwen3-VL-8B-Instruct-GGUF
VAEqwen_image_2.1_vae_bf16.safetensorsComfy-Org/Qwen-Image-2.1
Vision projector, required for reference editingmmproj-Qwen3VL-8B-Instruct-F16.ggufQwen/Qwen3-VL-8B-Instruct-GGUF

For SDK configuration and a runnable example, see Qwen-Image-2.1.

FLUX.2-klein

Three separate components are required:

  • Diffusion model (flux-2-klein-4b-Q8_0.gguf) — the main image transformer.
  • Text encoder (Qwen3-4B-Q4_K_M.gguf) — Qwen3 4B in standard GGML Q4_K_M format.
  • VAE (flux2-vae.safetensors) — standard safetensors format, compatible as-is.

Model file reference — FLUX.2-klein 4B

RoleFileSource
Diffusion modelflux-2-klein-4b-Q8_0.ggufleejet/FLUX.2-klein-4B-GGUF
Text encoderQwen3-4B-Q4_K_M.ggufunsloth/Qwen3-4B-GGUF
VAEflux2-vae.safetensorsblack-forest-labs/FLUX.2-klein-4B

Stable Diffusion

  • Stable Diffusion 1.x / 2.x — all-in-one checkpoint as a single *.gguf file
  • Stable Diffusion XL — all-in-one *.gguf or split CLIP encoders
  • Stable Diffusion 3 — safetensors with separate CLIP encoders

Requirements

  • Memory: 16 GB unified memory on Apple Silicon, or 8 GB VRAM on GPU.
  • Bare >= v1.24

Installation

npm i @qvac/diffusion-cpp

Quickstart

If you don't have Bare runtime, install it:

npm i -g bare

Create a new project:

mkdir qvac-diffusion-quickstart
cd qvac-diffusion-quickstart
npm init -y

Install dependencies:

npm i @qvac/diffusion-cpp bare-path bare-process bare-fs

Download the FLUX.2 [klein] 4B model files (~6.8 GB total):

mkdir -p models

curl -L -C - -o models/flux-2-klein-4b-Q8_0.gguf \
  https://huggingface.co/leejet/FLUX.2-klein-4B-GGUF/resolve/main/flux-2-klein-4b-Q8_0.gguf

curl -L -C - -o models/Qwen3-4B-Q4_K_M.gguf \
  https://huggingface.co/unsloth/Qwen3-4B-GGUF/resolve/main/Qwen3-4B-Q4_K_M.gguf

curl -L -C - -o models/flux2-vae.safetensors \
  https://huggingface.co/black-forest-labs/FLUX.2-klein-4B/resolve/main/vae/diffusion_pytorch_model.safetensors

Create index.js:

index.js
'use strict'

const path = require('bare-path')
const fs = require('bare-fs')
const process = require('bare-process')
const ImgStableDiffusion = require('@qvac/diffusion-cpp')

async function main () {
  const MODELS_DIR = path.resolve(__dirname, './models')

  const args = {
    logger: console,
    files: {
      model: path.join(MODELS_DIR, 'flux-2-klein-4b-Q8_0.gguf'),
      llm:   path.join(MODELS_DIR, 'Qwen3-4B-Q4_K_M.gguf'),
      vae:   path.join(MODELS_DIR, 'flux2-vae.safetensors')
    },
    config: { threads: 8 }
  }

  const model = new ImgStableDiffusion(args)
  await model.load()

  try {
    const images = []

    const response = await model.run({
      prompt: 'a majestic red fox in a snowy forest, golden light, photorealistic',
      steps: 20,
      width: 512,
      height: 512,
      guidance: 3.5,
      seed: 42
    })

    await response
      .onUpdate(data => {
        if (data instanceof Uint8Array) {
          images.push(data)
        } else if (typeof data === 'string') {
          try {
            const tick = JSON.parse(data)
            if ('step' in tick) process.stdout.write(`\rStep ${tick.step}/${tick.total}`)
          } catch (_) {}
        }
      })
      .await()

    console.log('\n')

    if (images.length > 0) {
      fs.writeFileSync('output.png', images[0])
      console.log('Saved → output.png')
    }
  } catch (error) {
    console.error('Error occurred:', error.message || error)
  } finally {
    await model.unload()
  }
}

main().catch(error => {
  console.error('Fatal error:', error.message)
  process.exit(1)
})

Run index.js:

bare index.js

Usage

1. Import the model class

const ImgStableDiffusion = require('@qvac/diffusion-cpp')

2. Create the args object

const path = require('bare-path')

const MODELS_DIR = path.resolve(__dirname, './models')
const args = {
  logger: console,
  files: {
    model: path.join(MODELS_DIR, 'flux-2-klein-4b-Q8_0.gguf'),
    llm:   path.join(MODELS_DIR, 'Qwen3-4B-Q4_K_M.gguf'),
    vae:   path.join(MODELS_DIR, 'flux2-vae.safetensors')
  },
  config: { threads: 8 },
  opts: { stats: true }
}
PropertyRequiredDescription
files✅Object of absolute paths to model files (see below)
files.model✅Absolute path to diffusion model file (diffusion-only GGUF for FLUX.2; all-in-one for SD1.x/2.x)
files.clipL—Absolute path to separate CLIP-L text encoder (SD3)
files.clipG—Absolute path to separate CLIP-G text encoder (SDXL / SD3)
files.t5Xxl—Absolute path to separate T5-XXL text encoder (SD3)
files.llm—Absolute path to the LLM text encoder, Qwen3 for FLUX.2 or Qwen3-VL for Qwen-Image-2.1
files.llmVision—Absolute path to the Qwen3-VL vision projector for Qwen-Image-2.1 reference editing
files.vae—Absolute path to separate VAE file
files.esrgan—Absolute path to ESRGAN upscaler model for post-generation upscale
config—Native backend configuration object (see next section)
logger—Logger instance for JS wrapper logs (e.g. console)
opts—Additional options (e.g. { stats: true })

Native C++ logs are process-global. Configure native log routing once with require('@qvac/diffusion-cpp/addonLogging').setLogger(...).

3. Configure the native backend (args.config)

config is a field on the args object built in step 2 — there is no separate constructor argument. The native backend reads it during load().

args.config = {
  threads: 8  // CPU threads for tensor operations (Metal handles GPU automatically)
}

Config values are coerced to strings internally. Generation parameters (prompt, steps, seed, etc.) are JSON-serialized with their native types preserved.

ParameterTypeDefaultDescription
threadsnumberautoNumber of CPU threads for model loading and CPU ops
type'f32' | 'f16' | 'q4_0' | 'q8_0' | …autoOverride weight quantisation type
rng'cpu' | 'cuda' | 'std_default''cuda'RNG backend ('cuda' = philox RNG — not GPU-specific despite the name; recommended)
backendstringautoGraph execution backend, globally or per module (for example cuda0 or diffusion=vulkan0,te=cpu)
params_backendstringautoParameter residency, including diffusion=cpu or diffusion=disk. A disk value disables eager loading for the whole context — see the note below
max_vramnumber | string0VRAM budget in GiB for graph-cut execution; supports per-device values such as cuda0=6,vulkan0=4
stream_layerstrue | falsefalseStream diffusion layers from CPU RAM when graph cutting is active and diffusion parameters are CPU-backed
flash_attntrue | falsefalseEnable flash attention (reduces memory)
upscaler_tile_sizenumber128ESRGAN upscaler tile size

clip_on_cpu, vae_on_cpu and control_net_cpu were removed and now fail the load. If the option was false, remove it and change nothing else — it had no effect, and there is no replacement. If it was enabled, keep the previous parameter placement with params_backend: 'te=cpu' in place of clip_on_cpu and params_backend: 'vae=cpu' in place of vae_on_cpu. To move graph execution instead, use backend with the same module name. control_net_cpu has no parameter-residency equivalent — use backend: 'controlnet=cpu'.

A disk value in params_backend turns off eager loading for the entire context, not just the module you name. The engine releases those weights at each phase boundary and re-reads them from the model file on every job, so the load cost is paid per generation rather than once.

4. Create a model instance

const model = new ImgStableDiffusion(args)

The constructor takes a single object containing files, config, logger, and opts. It stores configuration only — no memory is allocated yet.

5. Load the Model

await model.load()

This creates the native sd_ctx_t and loads all weights into memory. It can take 10–30 seconds depending on disk speed and model size. All model files must be passed as absolute paths via the files object.

6. Run Inference

The primary API. Returns a QvacResponse that streams step-progress ticks and the final PNG:

const images = []

const response = await model.run({
  prompt: 'a majestic red fox in a snowy forest, golden light, photorealistic',
  steps: 20,
  width: 512,
  height: 512,
  guidance: 3.5,
  seed: 42
})

await response
  .onUpdate(data => {
    if (data instanceof Uint8Array) {
      images.push(data)
    } else if (typeof data === 'string') {
      try {
        const tick = JSON.parse(data)
        if ('step' in tick) process.stdout.write(`\rStep ${tick.step}/${tick.total}`)
      } catch (_) {}
    }
  })
  .await()

require('bare-fs').writeFileSync('output.png', images[0])

Generation parameters:

ParameterTypeDefaultDescription
promptstring—Text prompt
negative_promptstring''Things to avoid in the output
widthnumber512Output width in pixels, a multiple of 8; Qwen-Image-2.1 requires a multiple of 32
heightnumber512Output height in pixels, a multiple of 8; Qwen-Image-2.1 requires a multiple of 32
stepsnumber20Number of diffusion steps
guidancenumber3.5Distilled guidance scale (FLUX.2)
cfg_scalenumber7.0Classifier-free guidance scale for SD and Qwen-Image
sampling_methodstringautoSampler name; auto-selects euler for FLUX.2, euler_a for SD1.x
schedulerstringautoScheduler; auto-selected per model family
seednumber-1Random seed (-1 for random)
batch_countnumber1Number of images to generate
vae_tilingbooleanfalseEnable VAE tiling (required for large images on 16 GB)
cache_presetstring—Step-caching preset: slow, medium, fast, ultra
init_imageUint8Array—Single PNG/JPEG reference, mutually exclusive with init_images
init_imagesUint8Array[]—Ordered references for FLUX.2 or Qwen-Image-2.1; requires a model with reference-editing support
strengthnumbermodel-dependentSD/SDXL img2img denoising strength; does not control reference fidelity for FLUX.2 or Qwen-Image-2.1
upscaleboolean | { repeats?: number }falsePost-generation ESRGAN upscale for RGB output. Requires files.esrgan; repeats defaults to 1. Qwen-Image-2.1 RGBA output rejects upscaling

Do not set sampling_method: 'euler_a' for FLUX.2 models — it will produce random noise. Leave the field unset to let the library auto-select euler for flow-matching models.

7. Release Resources

await model.unload()

unload() calls free_sd_ctx which releases all GPU and CPU memory. The JS object can be safely garbage collected afterwards.

More resources

Package at npm

On this page

Ask anything about QVAC.