Image generation
Text-to-image and image-to-image generation using a customized Diffusion engine.
Overview
Image generation runs on a customized Diffusion engine (qvac-ext-stable-diffusion.cpp). Load a supported model using modelType: "diffusion". Then, provide a text prompt describing the image to generate.
For image-to-image, pass init_image as PNG or JPEG bytes in a Uint8Array. SD models denoise the source image; reference-editing models such as Qwen-Image-2.1 generate a new target from noise conditioned on your references. Pass init_images for ordered references on supported models. The two fields are mutually exclusive. diffusion() returns PNG images as Uint8Array buffers and preserves alpha when the model produces RGBA. Use progressStream to track generation progress.
For higher-resolution outputs, you can chain a Real-ESRGAN upscaler. Either attach it as a post-processing step at load time and trigger it per generation via diffusion({ upscale }), or load an ESRGAN model standalone with modelConfig.mode: "upscale" and call upscale() on any PNG/JPEG buffer — generated or not. Both paths return PNG Uint8Array buffers and accept a repeats option that compounds the scale factor across sequential passes.
Functions
Use the following sequence of function calls:
loadModel()diffusion()orupscale()unloadModel()
For how to use each function, see SDK — API reference.
Models
Supported model families and their file layouts:
- FLUX.2-klein: split layout — diffusion model
*.gguf+ LLM text encoder*.gguf(viallmModelSrc) + VAE*.safetensors(viavaeModelSrc). - FLUX.3 Image Klein: split layout — diffusion model
*.gguf+ Qwen3-VL 4B text encoder*.gguf(viallmModelSrc) + FLUX.2 VAE*.safetensors(viavaeModelSrc). Leavepredictionunset; the engine detects FLUX.3 from the weights. QVAC provides no registry entry or model constant for these weights. - Qwen-Image-2.1 and Qwen-Image-2.1-Turbo use local diffusion weights as
modelSrc, the matching Qwen3-VL encoder asllmModelSrc, and the 2.1 VAE asvaeModelSrc. Reference editing with the GGUF encoder also needs its vision projector asprojectionModelSrc. QVAC provides no registry entry or model constant for these weights. - Ideogram 4: split layout — diffusion model
*.gguf+ unconditional diffusion model*.gguf(viauncondModelSrc) + LLM text encoder*.gguf(viallmModelSrc) + VAE*.safetensors(viavaeModelSrc). - MOSS-Image: split layout — diffusion model
*.safetensors+ Qwen3.5 LLM text encoder*.gguf(viallmModelSrc) + VAE*.safetensors(viavaeModelSrc) + the encoder's externaltokenizer.json(viatokenizerSrc). QVAC provides no registry entry or model constant for these weights. - SD1.x, SD2.x: single all-in-one
*.gguffile. No companion files needed. - SDXL, SD3: may require separate CLIP/T5 text encoder files (
clipLModelSrc,clipGModelSrc,t5XxlModelSrc) inmodelConfigdepending on the model variant. - ESRGAN: post-generation upscaler,
*.pthformat. Two ways to load:- Paired with a diffusion model — set
modelConfig.upscaler.model_srcatloadModel()time alongside the diffusionmodelSrc. - Standalone — pass the ESRGAN model as the top-level
modelSrcwithmodelConfig.mode: "upscale".
- Paired with a diffusion model — set
For models available as constants, see SDK — Models.
On upscaling: ESRGAN can be used standalone via modelConfig.mode: "upscale" + upscale(), or paired with an RGB-output diffusion model via diffusion({ upscale }). ESRGAN accepts RGB only. Qwen-Image-2.1 produces RGBA and rejects post-generation upscaling; keep upscale omitted or false. Available constants: REALESRGAN_X4PLUS_ANIME_6B, REALESRGAN_X4PLUS.
Examples
FLUX.2-klein
The following script shows text-to-image generation using FLUX.2-klein with its split-layout model (separate diffusion model, LLM text encoder, and VAE):
import { loadModel, unloadModel, diffusion, FLUX_2_KLEIN_4B_Q4_0, FLUX_2_KLEIN_4B_VAE, QWEN3_4B_Q4_K_M } from '@qvac/sdk';
import fs from 'fs';
import path from 'path';
// FLUX.2 [klein] uses a split-layout: separate diffusion model + LLM text encoder + VAE
const diffusionModelSrc = process.argv[2] || FLUX_2_KLEIN_4B_Q4_0;
const llmModelSrc = process.argv[3] || QWEN3_4B_Q4_K_M;
const vaeModelSrc = process.argv[4] || FLUX_2_KLEIN_4B_VAE;
const prompt = process.argv[5] || 'a futuristic city at sunset, photorealistic';
const outputDir = process.argv[6] || '.';
console.log('▸ Loading FLUX.2 [klein] split-layout model...');
const modelId = await loadModel({
modelSrc: diffusionModelSrc,
modelType: 'sdcpp-generation',
modelConfig: {
device: 'gpu',
threads: 4,
llmModelSrc,
vaeModelSrc
},
onProgress: (p) => {
const mb = (n) => (n / 1e6).toFixed(1);
const line = `▸ Downloading ${p.percentage.toFixed(0)}% (${mb(p.downloaded)}/${mb(p.total)} MB)`;
process.stderr.write(process.stderr.isTTY ? `\r${line}` : `${line}\n`);
if (p.percentage >= 100)
process.stderr.write('\n');
}
});
console.log(`▸ Model loaded: ${modelId}`);
console.log(`▸ Generating: "${prompt}"`);
const { progressStream, outputs, stats } = diffusion({
modelId,
prompt,
width: 512,
height: 512,
steps: 20,
guidance: 3.5,
cfg_scale: 1,
seed: -1
});
for await (const { step, totalSteps } of progressStream) {
console.log(`▸ step ${step}/${totalSteps}`);
}
const buffers = await outputs;
for (let i = 0; i < buffers.length; i++) {
const outputPath = path.join(outputDir, `flux2_${i}.png`);
fs.writeFileSync(outputPath, buffers[i]);
console.log(`▸ Saved ${outputPath}`);
}
console.log('▸ Stats:', await stats);
await unloadModel({ modelId, clearStorage: false });
console.log('▸ Done');
process.exit(0);FLUX.3 Image Klein
FLUX.3 Image Klein uses the same split layout as FLUX.2-klein: the FLUX.3 diffusion model as modelSrc, a Qwen3-VL 4B text encoder as llmModelSrc, and the FLUX.2 VAE as vaeModelSrc. The examples use the recommended 16 Euler steps at cfg_scale: 1 for the distilled model. A reference image passed through init_image, or up to 10 through init_images, switches generation to editing; no @imageN tags or prediction override are needed. More than 10 references fail in the native engine.
The FLUX.3 diffusion weights are under the FLUX Kommunity License. The example downloads pinned weights by default; the model, encoder and VAE arguments accept local paths or URLs.
The following script generates from text, or edits when reference image paths follow the model arguments. Pass empty strings to keep the default output directory and weights:
node --import tsx examples/diffusion-flux3-klein.ts
node --import tsx examples/diffusion-flux3-klein.ts "Put the cat in a garden" "" "" "" "" cat.pngimport { close, diffusion, loadModel, unloadModel } from '@qvac/sdk';
import fs from 'fs';
import path from 'path';
// FLUX.3 Image Klein text-to-image, or reference editing when reference images are passed.
// FLUX.3 pairs its diffusion model with a Qwen3-VL 4B text encoder and the FLUX.2 VAE.
// Defaults download pinned Hugging Face weights; overrides accept paths or URLs.
// FLUX.3 model terms: https://huggingface.co/black-forest-labs/flux-3-image-klein/blob/09561231f44e41381c461caf53d5452005a60ac9/LICENSE.md
const HF = 'https://huggingface.co';
const prompt = process.argv[2] || "A lovely cat holding a sign that says 'FLUX3'";
const outputDir = process.argv[3] || '.';
const diffusionModelSrc = process.argv[4] ||
`${HF}/leejet/flux-3-image-klein-GGUF/resolve/1c2ad2f9b9247ccca6506a23bdb033fdb0873321/flux3-image-klein-Q8_0.gguf`;
const llmModelSrc = process.argv[5] ||
`${HF}/Qwen/Qwen3-VL-4B-Instruct-GGUF/resolve/1cd86afb9a95c410a6038ab3b40d8b578c892266/Qwen3VL-4B-Instruct-Q4_K_M.gguf`;
const vaeModelSrc = process.argv[6] ||
`${HF}/Comfy-Org/Flux-2-Klein-4B/resolve/5b3abca5c0dc1ecd4eaf3340868d09063fc1c965/split_files/vae/flux2-vae.safetensors`;
const referencePaths = process.argv.slice(7);
const usage = 'Usage: node --import tsx examples/diffusion-flux3-klein.ts [prompt] [outputDir] [model] [llm] [vae] [...references]';
if (process.argv.includes('--help')) {
console.log(usage);
console.log('▸ Defaults download pinned Hugging Face weights. Model overrides accept paths or URLs.');
console.log('▸ Pass up to 10 reference image paths to edit them instead of generating from text.');
process.exit(0);
}
if (referencePaths.length > 10) {
console.error('✖ FLUX.3 accepts at most 10 reference images');
console.error(usage);
process.exit(1);
}
function modelSource(source) {
return /^https?:\/\//.test(source) ? source : path.resolve(source);
}
let modelId;
try {
const init_images = referencePaths.map((file) => new Uint8Array(fs.readFileSync(file)));
console.log('▸ Loading FLUX.3 Image Klein split-layout model...');
modelId = await loadModel({
modelSrc: modelSource(diffusionModelSrc),
modelType: 'sdcpp-generation',
modelConfig: {
llmModelSrc: modelSource(llmModelSrc),
vaeModelSrc: modelSource(vaeModelSrc),
flash_attn: true
},
onProgress: (progress) => {
const mb = (bytes) => (bytes / 1e6).toFixed(1);
const line = `▸ Downloading ${progress.percentage.toFixed(0)}% (${mb(progress.downloaded)}/${mb(progress.total)} MB)`;
process.stderr.write(process.stderr.isTTY ? `\r${line}` : `${line}\n`);
if (progress.percentage >= 100)
process.stderr.write('\n');
}
});
console.log(`▸ Model loaded: ${modelId}`);
console.log(`▸ Generating: "${prompt}"`);
// Recommended settings for the distilled model: 16 Euler steps at CFG 1.
const { progressStream, outputs, stats } = diffusion({
modelId,
prompt,
...(init_images.length > 0 && { init_images }),
width: 512,
height: 512,
steps: 16,
cfg_scale: 1,
sampling_method: 'euler',
seed: 42
});
for await (const { step, totalSteps } of progressStream) {
console.log(`▸ step ${step}/${totalSteps}`);
}
const buffers = await outputs;
if (buffers.length === 0)
throw new Error('FLUX.3 returned no image.');
for (let i = 0; i < buffers.length; i++) {
const outputPath = path.join(outputDir, `flux3_${i}.png`);
fs.writeFileSync(outputPath, buffers[i]);
console.log(`▸ Saved ${outputPath}`);
}
console.log('▸ Stats:', await stats);
}
catch (error) {
console.error('✖', error);
process.exitCode = 1;
}
finally {
try {
if (modelId)
await unloadModel({ modelId, clearStorage: false });
}
catch (error) {
console.error('✖ Unload failed:', error);
process.exitCode = 1;
}
try {
await close();
}
catch (error) {
console.error('✖ Cleanup failed:', error);
process.exitCode = 1;
}
}
if (!process.exitCode)
console.log('▸ Done');Qwen-Image-2.1
Qwen-Image-2.1 uses a split layout with a Qwen3-VL LLM text encoder (llmModelSrc), its vision projector (projectionModelSrc), and the 2.1 VAE (vaeModelSrc). Reference images are passed in order through init_images.
Qwen-Image-2.1-Turbo reuses the same encoder, projector and VAE. Supply Turbo diffusion weights as modelSrc. Use Euler, cfg_scale: 1, and the fixed schedule through diffusion():
diffusion({
modelId,
prompt: "A red ceramic teapot on a wooden table.",
width: 512,
height: 512,
sampling_method: "euler",
cfg_scale: 1,
custom_sigmas: [1, 0.978453, 0.95418, 0.926626, 0.89508, 0.845148, 0.704534, 0.414568, 0],
seed: 42,
});custom_sigmas includes the terminal zero and overrides scheduler and steps. The schedule above runs eight denoising steps, which progressStream reports. stats.totalSteps counts steps across all completed requests on the loaded model. Custom schedules need at least two finite, non-negative values that remain strictly decreasing at float precision and end in zero. Omitting the field preserves ordinary sampling.
For Qwen outputs at 2048×2048 or larger, pass vae_tiling: true and vae_tile_size: 64 to diffusion(). Tile sizes are in latent units. For Qwen-Image-2.1, 64 corresponds to 1024 output pixels; enabling tiling alone leaves the native default tile too large to split a 2K image.
The following script combines two references into an RGBA image, with optional Qwen-Image-2.1-Turbo sampling. Pass the two reference image paths as the first two arguments and an optional prompt as the third:
QWEN_TURBO=1 node --import tsx examples/diffusion-qwen-image-21.ts \
reference-a.png reference-b.png "" . /absolute/path/to/qwen-image-2.1-turbo.ggufThe Python example uses the same arguments and QWEN_TURBO flag. Turbo mode requires the model argument, which accepts a local path or URL. Leave QWEN_TURBO unset to run the base model with 30 steps and CFG 6.
import { close, diffusion, loadModel, unloadModel } from '@qvac/sdk';
import fs from 'fs';
import path from 'path';
// Combine two references into an RGBA image, with optional Qwen-Image-2.1-Turbo sampling.
// Set QWEN_TURBO=1 and pass Turbo weights as the model argument, as a path or URL.
// Use the pinned model revisions from the diffusion addon's Qwen-Image-2.1 downloader.
// Model terms: https://huggingface.co/Qwen/Qwen-Image-2.1/blob/d26bb61231c349cf6b7896fa83353113880e1ba3/LICENSE
const turbo = process.env['QWEN_TURBO'] === '1';
const HF = 'https://huggingface.co';
const encoderRepo = `${HF}/Qwen/Qwen3-VL-8B-Instruct-GGUF/resolve/f982a07559d4a2f6c8744d840bf6fccab30eea96`;
const referencePathA = process.argv[2];
const referencePathB = process.argv[3];
const prompt = process.argv[4] ||
'Place the subjects from both reference images side by side. Generate an RGBA image with a transparent background and no scenery.';
const outputDir = process.argv[5] || '.';
const diffusionModelSrc = process.argv[6] ||
`${HF}/leejet/Qwen-Image-2.1-GGUF/resolve/cc11433936a06e9765f7c0c0b1f0436cfd2b9856/qwen_image_2.1-Q4_K.gguf`;
const llmModelSrc = process.argv[7] || `${encoderRepo}/Qwen3VL-8B-Instruct-Q4_K_M.gguf`;
const vaeModelSrc = process.argv[8] ||
`${HF}/Comfy-Org/Qwen-Image-2.1/resolve/a8342359c61979b06a8dafb1a47991fe1f30e994/vae/qwen_image_2.1_vae_bf16.safetensors`;
const projectionModelSrc = process.argv[9] || `${encoderRepo}/mmproj-Qwen3VL-8B-Instruct-F16.gguf`;
const usage = 'Usage: node --import tsx examples/diffusion-qwen-image-21.ts <referenceA> <referenceB> [prompt] [outputDir] [model] [llm] [vae] [projector]';
if (process.argv.includes('--help')) {
console.log(usage);
console.log('▸ Defaults download pinned Hugging Face weights. Model overrides accept paths or URLs.');
console.log('▸ Turbo: set QWEN_TURBO=1 and pass your Turbo weights as the model argument.');
console.log('▸ The default prompt combines both subjects on a transparent background.');
process.exit(0);
}
if (!referencePathA || !referencePathB) {
console.error('✖ two reference image paths are required');
console.error(usage);
process.exit(1);
}
if (turbo && !process.argv[6]) {
console.error('✖ Pass the Turbo diffusion weights as the model argument, as a path or URL');
process.exit(1);
}
function modelSource(source) {
return /^https?:\/\//.test(source) ? source : path.resolve(source);
}
let modelId;
try {
const init_images = [referencePathA, referencePathB].map((file) => new Uint8Array(fs.readFileSync(file)));
console.log('▸ Loading Qwen-Image-2.1 split-layout model...');
modelId = await loadModel({
modelSrc: modelSource(diffusionModelSrc),
modelType: 'sdcpp-generation',
modelConfig: {
llmModelSrc: modelSource(llmModelSrc),
vaeModelSrc: modelSource(vaeModelSrc),
projectionModelSrc: modelSource(projectionModelSrc),
flash_attn: true
},
onProgress: (progress) => {
const mb = (bytes) => (bytes / 1e6).toFixed(1);
const line = `▸ Downloading ${progress.percentage.toFixed(0)}% (${mb(progress.downloaded)}/${mb(progress.total)} MB)`;
process.stderr.write(process.stderr.isTTY ? `\r${line}` : `${line}\n`);
if (progress.percentage >= 100)
process.stderr.write('\n');
}
});
console.log(`▸ Model loaded: ${modelId}`);
console.log(`▸ Generating from "${referencePathA}" and "${referencePathB}": "${prompt}"`);
const { progressStream, outputs, stats } = diffusion({
modelId,
prompt,
init_images,
width: 512,
height: 512,
...(!turbo && { steps: 30 }),
cfg_scale: turbo ? 1 : 6,
...(turbo && {
// Official Turbo sample_sigmas plus the native terminal zero.
custom_sigmas: [1, 0.978453, 0.95418, 0.926626, 0.89508, 0.845148, 0.704534, 0.414568, 0]
}),
sampling_method: 'euler',
seed: 42
});
for await (const { step, totalSteps } of progressStream) {
console.log(`▸ step ${step}/${totalSteps}`);
}
const buffers = await outputs;
if (buffers.length === 0)
throw new Error('Qwen-Image-2.1 returned no image.');
for (let i = 0; i < buffers.length; i++) {
const outputPath = path.join(outputDir, `qwen_image_21${turbo ? '_turbo' : ''}_${i}.png`);
fs.writeFileSync(outputPath, buffers[i]);
console.log(`▸ Saved ${outputPath}`);
}
console.log('▸ Stats:', await stats);
}
catch (error) {
console.error('✖', error);
process.exitCode = 1;
}
finally {
try {
if (modelId)
await unloadModel({ modelId, clearStorage: false });
}
catch (error) {
console.error('✖ Unload failed:', error);
process.exitCode = 1;
}
try {
await close();
}
catch (error) {
console.error('✖ Cleanup failed:', error);
process.exitCode = 1;
}
}
if (!process.exitCode)
console.log('▸ Done');Ideogram 4
Ideogram 4 uses a split layout with an extra unconditional diffusion model (uncondModelSrc) alongside the Qwen3-VL LLM text encoder (llmModelSrc) and VAE (vaeModelSrc). Unlike the other families, its prompt must be a JSON-serialized structured caption — an object with a high_level_description, a style_description, and a compositional_deconstruction that lays out each element with an explicit bounding box (bbox) and, for text elements, the literal text to render. Plain-text prompts produce degenerate or placeholder output.
Pass the structured caption serialized, not as an object: JSON.stringify(...) in JavaScript and TypeScript, or json.dumps(..., separators=(",", ":")) in Python, whose default spacing the model does not expect. The bounding boxes are expressed in the target canvas coordinate space ([x0, y0, x1, y1]), so keep them consistent with the requested width/height. Failure to abide by the JSON format specified within the examples, or adding NSFW prompts, will result in censored image output.
The following script loads Ideogram 4 in split-layout (diffusion model + unconditional model + LLM text encoder + VAE) and generates from a structured caption:
import { diffusion, loadModel, unloadModel } from '@qvac/sdk';
import fs from 'fs';
import path from 'path';
const diffusionModelSrc = process.argv[2] ||
'https://huggingface.co/leejet/ideogram-4-GGUF/resolve/main/ideogram4-Q4_0.gguf';
const uncondModelSrc = process.argv[3] ||
'https://huggingface.co/leejet/ideogram-4-GGUF/resolve/main/ideogram4_uncond-Q4_0.gguf';
const llmModelSrc = process.argv[4] ||
'https://huggingface.co/unsloth/Qwen3-VL-8B-Instruct-GGUF/resolve/main/Qwen3-VL-8B-Instruct-Q4_K_M.gguf';
const vaeModelSrc = process.argv[5] ||
'https://huggingface.co/Comfy-Org/flux2-dev/resolve/main/split_files/vae/flux2-vae.safetensors';
const outputDir = process.argv[6] || '.';
function formatMb(bytes) {
return (bytes / 1e6).toFixed(1);
}
// Ideogram 4 expects a JSON-serialized structured caption with explicit
// bounding boxes. Plain-text prompts produce degenerate or placeholder output.
const structuredPrompt = {
high_level_description: 'A bright product photo of one yellow coffee mug on a clean office desk with a tiny readable label that says IDEO.',
style_description: {
aesthetics: 'simple product photography, clean bright office desk, minimal composition',
lighting: 'soft studio light with gentle shadows',
photo: 'sharp focus, crisp readable label text',
medium: 'product photo',
color_palette: ['#FFD54A', '#FFFFFF', '#111111', '#7EC8E3']
},
compositional_deconstruction: {
canvas: 'Square canvas, upright orientation. All text is horizontal and readable.',
background: 'Clean white office desk with a soft blue background gradient.',
elements: [
{
type: 'obj',
bbox: [230, 260, 800, 740],
desc: 'Exactly one matte yellow ceramic coffee mug centered on the desk.'
},
{
type: 'text',
bbox: [510, 360, 620, 640],
text: 'IDEO',
desc: 'Small black label printed horizontally on the mug.'
}
]
}
};
if (process.argv.includes('--help')) {
console.log('Usage: tsx examples/diffusion-ideogram.ts [model] [uncond] [llm] [vae] [output-dir]');
console.log('▸ Defaults use the four Hugging Face model URLs supported by the diffusion addon.');
process.exit(0);
}
let modelId;
try {
console.log('▸ Loading Ideogram 4 split-layout model...');
modelId = await loadModel({
modelSrc: diffusionModelSrc,
modelType: 'sdcpp-generation',
modelConfig: {
device: 'gpu',
threads: 4,
diffusion_fa: true,
offload_to_cpu: true,
llmModelSrc,
vaeModelSrc,
uncondModelSrc
},
onProgress: (progress) => {
const line = `▸ Downloading ${progress.percentage.toFixed(0)}% (${formatMb(progress.downloaded)}/${formatMb(progress.total)} MB)`;
process.stderr.write(process.stderr.isTTY ? `\r${line}` : `${line}\n`);
if (progress.percentage >= 100)
process.stderr.write('\n');
}
});
console.log(`▸ Model loaded: ${modelId}`);
const { progressStream, outputs, stats } = diffusion({
modelId,
prompt: JSON.stringify(structuredPrompt),
width: 768,
height: 768,
steps: 16,
cfg_scale: 7,
seed: 42
});
for await (const { step, totalSteps } of progressStream) {
console.log(`▸ step ${step}/${totalSteps}`);
}
const buffers = await outputs;
for (let i = 0; i < buffers.length; i++) {
const outputPath = path.join(outputDir, `ideogram_${i}.png`);
fs.writeFileSync(outputPath, buffers[i]);
console.log(`▸ Saved ${outputPath}`);
}
console.log('▸ Stats:', await stats);
await unloadModel({ modelId, clearStorage: false });
modelId = undefined;
console.log('▸ Done');
process.exit(0);
}
catch (error) {
if (modelId) {
try {
await unloadModel({ modelId, clearStorage: false });
}
catch {
// Preserve the original error.
}
}
console.error('✖', error);
process.exit(1);
}The Kotlin example serializes the caption with buildJsonObject { ... }.toString(), which already produces the compact form.
MOSS-Image
MOSS-Image uses a split layout with a Qwen3.5 LLM text encoder (llmModelSrc), a VAE (vaeModelSrc), and the encoder's external tokenizer (tokenizerSrc), which is only valid in diffusion mode. It accepts plain-text prompts, and for layout control a JSON-serialized layout: a summary with an overall caption, and items that each set a label ("text" or "object"), a caption, a bounding box (bbox) and, for text, the literal text to render. Start from Euler sampling, the discrete scheduler, cfg_scale: 3 and 50 steps.
Pass the layout serialized, not as an object: JSON.stringify(...) in JavaScript and TypeScript, or json.dumps(..., ensure_ascii=False) in Python so non-ASCII text reaches the model as characters rather than \uXXXX escapes. Set bbox_coord_space: "abs_centralized" and bbox_format: "yxyx": boxes are [y_min, x_min, y_max, x_max] in pixels with the canvas centre as the origin, so keep them consistent with the requested width/height.
The following script loads MOSS-Image in split layout (diffusion model + LLM text encoder + VAE + tokenizer) and generates a poster from a JSON layout:
import { close, diffusion, loadModel, unloadModel } from '@qvac/sdk';
import fs from 'fs';
import path from 'path';
// Generate a poster from a MOSS-Image JSON layout prompt.
// Use the pinned model revisions from the diffusion addon's MOSS-Image downloader.
// Model terms (Apache-2.0): https://huggingface.co/OpenMOSS-Team/MOSS-Image-SFT
const HF = 'https://huggingface.co';
const mossRepo = `${HF}/Comfy-Org/MOSS-Image/resolve/530a4c94f92c35ef6818b1b035a983fe9dcf5f54`;
const outputDir = process.argv[2] || '.';
const diffusionModelSrc = process.argv[3] || `${mossRepo}/diffusion_models/moss_image_sft_bf16.safetensors`;
const llmModelSrc = process.argv[4] ||
`${HF}/unsloth/Qwen3.5-9B-GGUF/resolve/3885219b6810b007914f3a7950a8d1b469d598a5/Qwen3.5-9B-Q4_K_M.gguf`;
const vaeModelSrc = process.argv[5] || `${mossRepo}/vae/flux2-vae_small.safetensors`;
const tokenizerSrc = process.argv[6] ||
`${HF}/OpenMOSS-Team/MOSS-Image-SFT/resolve/f972bd4b4b9e3852133cc5da954d87bc32cd8fb9/tokenizer/tokenizer.json`;
const usage = 'Usage: node --import tsx examples/diffusion-moss-image.ts [outputDir] [model] [llm] [vae] [tokenizer]';
if (process.argv.includes('--help')) {
console.log(usage);
console.log('▸ Defaults download pinned Hugging Face weights. Model overrides accept paths or URLs.');
process.exit(0);
}
const width = 1024;
const height = 1024;
// Boxes are [y_min, x_min, y_max, x_max] in absolute pixels with the canvas
// centre as the origin, so a 1024×1024 canvas spans -512..512 on both axes.
// Keep them consistent with width and height.
const layout = {
summary: {
meta: ['poster'],
canvas: 'single',
caption: 'A minimalist travel poster of a snowy mountain at sunrise, flat illustration style.'
},
items: [
{
label: 'text',
caption: 'Bold white sans-serif title across the top of the poster.',
text: 'SUMMIT',
bbox: [-448, -352, -320, 352]
},
{
label: 'object',
caption: 'A snowy mountain peak lit by an orange sunrise.',
bbox: [-224, -320, 384, 320]
}
],
bbox_coord_space: 'abs_centralized',
bbox_format: 'yxyx'
};
function modelSource(source) {
return /^https?:\/\//.test(source) ? source : path.resolve(source);
}
let modelId;
// Progress fires many times per percent on a multi-GB download; write once per whole percent
// of each file.
const lastPercent = new Map();
try {
console.log('▸ Loading MOSS-Image...');
modelId = await loadModel({
modelSrc: modelSource(diffusionModelSrc),
modelType: 'sdcpp-generation',
modelConfig: {
llmModelSrc: modelSource(llmModelSrc),
vaeModelSrc: modelSource(vaeModelSrc),
tokenizerSrc: modelSource(tokenizerSrc),
flash_attn: true
},
onProgress: (progress) => {
const percent = Math.floor(progress.percentage);
if (lastPercent.get(progress.downloadKey) === percent)
return;
lastPercent.set(progress.downloadKey, percent);
const mb = (bytes) => (bytes / 1e6).toFixed(1);
const line = `▸ Downloading ${progress.percentage.toFixed(0)}% (${mb(progress.downloaded)}/${mb(progress.total)} MB)`;
process.stderr.write(process.stderr.isTTY ? `\r${line}` : `${line}\n`);
if (progress.percentage >= 100)
process.stderr.write('\n');
}
});
console.log(`▸ Model loaded: ${modelId}`);
const { progressStream, outputs, stats } = diffusion({
modelId,
prompt: JSON.stringify(layout),
width,
height,
steps: 50,
cfg_scale: 3,
sampling_method: 'euler',
scheduler: 'discrete',
seed: 42
});
for await (const { step, totalSteps } of progressStream) {
console.log(`▸ step ${step}/${totalSteps}`);
}
const buffers = await outputs;
if (buffers.length === 0)
throw new Error('MOSS-Image returned no image.');
for (let i = 0; i < buffers.length; i++) {
const outputPath = path.join(outputDir, `moss_image_${i}.png`);
fs.writeFileSync(outputPath, buffers[i]);
console.log(`▸ Saved ${outputPath}`);
}
console.log('▸ Stats:', await stats);
}
catch (error) {
console.error('✖', error);
process.exitCode = 1;
}
finally {
try {
if (modelId)
await unloadModel({ modelId, clearStorage: false });
}
catch (error) {
console.error('✖ Unload failed:', error);
process.exitCode = 1;
}
try {
await close();
}
catch (error) {
console.error('✖ Cleanup failed:', error);
process.exitCode = 1;
}
}
if (!process.exitCode)
console.log('▸ Done');Stable Diffusion
The following script shows a minimal text-to-image generation example using a single all-in-one SD 2.1 model:
import { loadModel, unloadModel, diffusion, SD_V2_1_1B_Q8_0 } from '@qvac/sdk';
import fs from 'fs';
// Minimal diffusion example — single GGUF model, no companion files needed.
// Works with SD 1.x / 2.x all-in-one models.
const modelSrc = process.argv[2] || SD_V2_1_1B_Q8_0;
const prompt = process.argv[3] || 'a photo of a cat sitting on a windowsill';
const modelId = await loadModel({
modelSrc,
modelType: 'sdcpp-generation',
modelConfig: { prediction: 'v' }
});
const { outputs } = diffusion({ modelId, prompt });
const buffers = await outputs;
fs.writeFileSync('output.png', buffers[0]);
console.log('▸ Saved output.png');
await unloadModel({ modelId, clearStorage: false });
process.exit(0);Image-to-image
Pass init_image to transform an existing image guided by a text prompt. Behavior depends on the model family:
- FLUX.2: in-context conditioning. Requires
prediction: "flux2_flow"inmodelConfigatloadModel()time;strengthis ignored on this path. - FLUX.3 Image Klein uses reference conditioning with automatic model detection. Leave
predictionunset;strengthdoes not apply. - Qwen-Image-2.1 uses reference conditioning with automatic model detection. Supply
projectionModelSrcat load time for editing with the GGUF encoder. Useinit_imageor orderedinit_images;strengthdoes not control reference fidelity. - SD / SDXL / SD3: SDEdit-style. Use
strengthto control how much the source is preserved (0= keep source,1= ignore source).
The following script loads FLUX.2-klein in split-layout and transforms an input image using in-context conditioning (prediction: "flux2_flow"):
import { loadModel, unloadModel, diffusion, FLUX_2_KLEIN_4B_Q4_0, FLUX_2_KLEIN_4B_VAE, QWEN3_4B_Q4_K_M } from '@qvac/sdk';
import fs from 'fs';
import path from 'path';
// img2img with FLUX.2 [klein] split-layout — uses in-context conditioning ("flux2_flow").
const inputPath = process.argv[2];
const prompt = process.argv[3] || 'oil painting style, vibrant colors';
const outputDir = process.argv[4] || '.';
const diffusionModelSrc = process.argv[5] || FLUX_2_KLEIN_4B_Q4_0;
const llmModelSrc = process.argv[6] || QWEN3_4B_Q4_K_M;
const vaeModelSrc = process.argv[7] || FLUX_2_KLEIN_4B_VAE;
if (!inputPath) {
console.error('✖ input image path is required');
console.error('Usage: node dist/examples/diffusion-flux2-klein-img2img.js <inputImage> [prompt] [outputDir] [diffusionModelSrc] [llmModelSrc] [vaeModelSrc]');
process.exit(1);
}
try {
console.log('▸ Loading FLUX.2 [klein] split-layout model...');
const modelId = await loadModel({
modelSrc: diffusionModelSrc,
modelType: 'sdcpp-generation',
modelConfig: {
device: 'gpu',
threads: 4,
llmModelSrc,
vaeModelSrc,
prediction: 'flux2_flow'
},
onProgress: (p) => {
const mb = (n) => (n / 1e6).toFixed(1);
const line = `▸ Downloading ${p.percentage.toFixed(0)}% (${mb(p.downloaded)}/${mb(p.total)} MB)`;
process.stderr.write(process.stderr.isTTY ? `\r${line}` : `${line}\n`);
if (p.percentage >= 100)
process.stderr.write('\n');
}
});
console.log(`▸ Model loaded: ${modelId}`);
const init_image = new Uint8Array(fs.readFileSync(inputPath));
console.log(`▸ Transforming "${inputPath}" with prompt: "${prompt}"`);
const { progressStream, outputs, stats } = diffusion({
modelId,
prompt,
init_image,
steps: 20,
guidance: 3.5,
cfg_scale: 1,
seed: -1
});
for await (const { step, totalSteps } of progressStream) {
console.log(`▸ step ${step}/${totalSteps}`);
}
const buffers = await outputs;
for (let i = 0; i < buffers.length; i++) {
const outputPath = path.join(outputDir, `flux2_img2img_${i}.png`);
fs.writeFileSync(outputPath, buffers[i]);
console.log(`▸ Saved ${outputPath}`);
}
console.log('▸ Stats:', await stats);
await unloadModel({ modelId, clearStorage: false });
console.log('▸ Done');
process.exit(0);
}
catch (error) {
console.error('✖', error);
process.exit(1);
}Upscaling
Post-processing
Pass upscale to diffusion() to upscale generated images in the same call. Requires modelConfig.upscaler = { type: "esrgan", model_src, tile_size? } at loadModel() time. Behavior depends on the value passed:
true(or{}/{ repeats: 1 }): single pass at the model's native scale factor (e.g.x4for RealESRGAN_x4plus).{ repeats: N }: N sequential passes — each pass multiplies the output dimensions by the model's scale factor (e.g.repeats: 2with anx4model →x16).- When
batch_count > 1, every output image is upscaled independently.
Qwen-Image-2.1 produces RGBA and rejects this option; leave upscale omitted or false.
The following script loads SD 2.1 with an ESRGAN upscaler and generates both a single-pass (x4) and a two-pass (x16) upscale from the same prompt:
import { loadModel, unloadModel, diffusion, SD_V2_1_1B_Q8_0, REALESRGAN_X4PLUS_ANIME_6B } from '@qvac/sdk';
import fs from 'fs';
import path from 'path';
// ESRGAN upscale example.
//
// Usage:
// npx tsx examples/diffusion-esrgan-upscale.ts [esrganSrc] [prompt] [outputDir]
const esrganArg = process.argv[2];
const promptArg = process.argv[3];
const outputDirArg = process.argv[4];
const esrganModelSrc = esrganArg ?? REALESRGAN_X4PLUS_ANIME_6B;
const prompt = promptArg ??
'an illustrated red fox portrait, clean line art, soft watercolor background, detailed fur, crisp eyes';
const negative_prompt = 'blurry, low quality, watermark, text';
const outputDir = outputDirArg ?? '.';
const seed = 42;
try {
console.log('▸ Loading SD 2.1 + ESRGAN upscaler...');
const modelId = await loadModel({
modelSrc: SD_V2_1_1B_Q8_0,
modelConfig: {
prediction: 'v',
upscaler: {
type: 'esrgan',
model_src: esrganModelSrc,
tile_size: 128
}
},
onProgress: (p) => {
const mb = (n) => (n / 1e6).toFixed(1);
const line = `▸ Downloading ${p.percentage.toFixed(0)}% (${mb(p.downloaded)}/${mb(p.total)} MB)`;
process.stderr.write(process.stderr.isTTY ? `\r${line}` : `${line}\n`);
if (p.percentage >= 100)
process.stderr.write('\n');
}
});
console.log(`▸ Model loaded: ${modelId}`);
// Source size is intentionally small — each ESRGAN repeat multiplies dimensions.
const baseParams = {
modelId,
prompt,
negative_prompt,
width: 128,
height: 128,
steps: 5,
cfg_scale: 7.5,
seed
};
console.log(`▸ Generating ESRGAN x4 upscale: "${prompt}"`);
const single = diffusion({ ...baseParams, upscale: true });
for await (const { step, totalSteps } of single.progressStream) {
console.log(`▸ step ${step}/${totalSteps}`);
}
const singleBuffers = await single.outputs;
for (let i = 0; i < singleBuffers.length; i++) {
const out = path.join(outputDir, `sd2_esrgan_x4_seed${seed}_${i}.png`);
fs.writeFileSync(out, singleBuffers[i]);
console.log(`▸ Saved ${out}`);
}
console.log('▸ Stats:', await single.stats);
console.log('▸ Generating ESRGAN two-pass x16 upscale...');
const twoPass = diffusion({ ...baseParams, upscale: { repeats: 2 } });
for await (const { step, totalSteps } of twoPass.progressStream) {
console.log(`▸ step ${step}/${totalSteps}`);
}
const twoPassBuffers = await twoPass.outputs;
for (let i = 0; i < twoPassBuffers.length; i++) {
const out = path.join(outputDir, `sd2_esrgan_x16_seed${seed}_${i}.png`);
fs.writeFileSync(out, twoPassBuffers[i]);
console.log(`▸ Saved ${out}`);
}
console.log('▸ Stats:', await twoPass.stats);
await unloadModel({ modelId, clearStorage: false });
console.log('▸ Done');
process.exit(0);
}
catch (error) {
console.error('✖', error);
process.exit(1);
}Standalone
The SDK also exposes upscale() — a standalone path that upscales any PNG/JPEG image without running diffusion. Load the ESRGAN model directly with modelConfig.mode: "upscale":
const modelId = await loadModel(REALESRGAN_X4PLUS_ANIME_6B, {
modelType: "diffusion",
modelConfig: { mode: "upscale", upscaler: { tile_size: 128 } },
});
const { outputs } = upscale({ modelId, image: pngBytes, repeats: 2 });
const [upscaledPng] = await outputs;tile_size defaults to 128. Increase it to reduce tile seams on large inputs, at the cost of more memory per pass.
Tip: all examples throughout this documentation are self-contained and runnable. For instructions on how to run them, see the JS/TS quickstart, the Python quickstart, or the Kotlin quickstart.