# Music generation (/ai-capabilities/music-generation)



## Overview

Music generation uses [`@qvac/audiogen-ggml`](https://github.com/tetherto/qvac/tree/main/packages/audiogen-ggml), a GGML-backed ACE-Step 1.5 engine. Load the four model stages with `modelType: "audiogen"`, then call `audioGen()` with a caption and optional lyrics or musical controls.

`audioGen()` returns immediately with:

* `requestId` — a synchronous identifier for targeted cancellation.
* `progressStream` — stage and step updates while generation runs.
* `audio` — a promise resolving to interleaved PCM, its sample rate, channel count, and `bitsPerSample`.
* `stats` — a promise resolving to timing statistics.

<Callout type="info">
  AudioGen is currently validated by the SDK end-to-end suite on desktop
  Node.js/Bare. Its four-model pipeline has a large memory and download
  footprint, so test your chosen model set and hardware before deploying to
  other runtimes.
</Callout>

## Functions

Use the following sequence:

1. [`loadModel()`](/reference/api#loadmodel)
2. [`audioGen()`](/reference/api#audiogen)
3. [`unloadModel()`](/reference/api#unloadmodel)

For complete signatures, see the [SDK API reference](/reference/api/).

## Enable the plugin

When your project selects plugins explicitly, add the AudioGen plugin:

```json title="qvac.config.json"
{
  "plugins": ["@qvac/sdk/audiogen-ggml/plugin"]
}
```

Then rebuild the SDK bundle:

```bash
qvac bundle sdk
```

If `plugins` is omitted or empty, all built-in plugins are included. See the [plugin system](/configuration/plugins) for bundle and Bare runtime registration details.

## Models

ACE-Step uses four independent GGUF files:

| Stage          | SDK model constant                   | Purpose                                |
| -------------- | ------------------------------------ | -------------------------------------- |
| Text encoder   | `AUDIOGEN_QWEN3_EMBEDDING_0_6B_Q8_0` | Encodes captions and lyrics            |
| Language model | `AUDIOGEN_ACESTEP_5HZ_LM_0_6B_Q8_0`  | Plans the song and musical structure   |
| DiT            | `AUDIOGEN_ACESTEP_V15_TURBO_Q4_K_M`  | Fast, lower-memory waveform generation |
| DiT            | `AUDIOGEN_ACESTEP_V15_TURBO_Q8_0`    | Higher-precision Turbo generation      |
| DiT            | `AUDIOGEN_ACESTEP_V15_SFT_Q8_0`      | Supervised fine-tuned generation       |
| VAE            | `AUDIOGEN_VAE_BF16`                  | Decodes the latent into PCM audio      |

<Callout type="warning">
  The first AudioGen load downloads approximately 3.3 GB across these four
  files. The SDK downloads them sequentially to avoid splitting P2P bandwidth
  across four registry streams. On slow or unstable links, set
  `registryStreamTimeoutMs: 600000` and `registryDownloadMaxRetries: 10` in
  `qvac.config.json`, then point `QVAC_CONFIG_PATH` at that file before running
  the example.
</Callout>

Unlike single-file model families, AudioGen does not use a top-level `modelSrc`. Supply every stage in `modelConfig`:

```ts
import {
  AUDIOGEN_ACESTEP_5HZ_LM_0_6B_Q8_0,
  AUDIOGEN_ACESTEP_V15_TURBO_Q4_K_M,
  AUDIOGEN_QWEN3_EMBEDDING_0_6B_Q8_0,
  AUDIOGEN_VAE_BF16,
  loadModel,
} from "@qvac/sdk";

const modelId = await loadModel({
  modelType: "audiogen",
  modelConfig: {
    textEncModelSrc: AUDIOGEN_QWEN3_EMBEDDING_0_6B_Q8_0,
    lmModelSrc: AUDIOGEN_ACESTEP_5HZ_LM_0_6B_Q8_0,
    ditModelSrc: AUDIOGEN_ACESTEP_V15_TURBO_Q4_K_M,
    vaeModelSrc: AUDIOGEN_VAE_BF16,
    useGPU: true,
  },
});
```

Advanced load-time controls are `inferenceSteps`, `shift`, `nGpuLayers`, `threads`, and `backendsDir`. Turbo and SFT variants have different tuning requirements; omit `inferenceSteps` and `shift` unless you need to override the addon's defaults.

For more about model constants and sources, see [SDK — Models](/introduction#models).

## Generate audio

`caption` is required and cannot be empty. Lyrics and musical controls are optional:

```ts
import { audioGen } from "@qvac/sdk";

const run = audioGen({
  modelId,
  caption: "Energetic cumbia with brass stabs and live percussion",
  lyrics: `[verse]
The city wakes beneath the moon

[chorus]
Dance until the morning comes`,
  vocalLanguage: "en",
  bpm: 98,
  keyscale: "A minor",
  timesignature: "4/4",
  duration: 30,
  seed: 42,
});

for await (const progress of run.progressStream) {
  console.log(`${progress.stage}: ${progress.step}/${progress.total}`);
}

const [{ pcm, sampleRate, channels, bitsPerSample }, stats] = await Promise.all([
  run.audio,
  run.stats,
]);
```

Omit `lyrics` or use `"[Instrumental]"` for instrumental output. Omitted musical controls are inferred from the caption.

`duration` is approximate. ACE-Step rounds the requested length to its latent frame grid, so the generated clip can be shorter or longer than requested. Use `stats.audioDurationMs` or calculate the duration from the returned PCM frame count and `sampleRate`; those values describe the actual output.

The SDK returns raw interleaved PCM rather than a WAV file. Use `bitsPerSample` when calculating sample counts or constructing a WAV header so consumers remain correct if the addon changes sample width. The complete example below includes a dependency-free WAV writer.

## Cancellation

The request ID is available before generation starts, so a stop button can cancel exactly that run:

```ts
import { audioGen, cancel, InferenceCancelledError } from "@qvac/sdk";

const run = audioGen({ modelId, caption: "Ambient electronic music" });
stopButton.onclick = () => cancel({ requestId: run.requestId });

try {
  await run.audio;
} catch (error) {
  if (!(error instanceof InferenceCancelledError)) throw error;
}
```

AudioGen supports hard cancellation: the addon interrupts the active ACE-Step generation. The stream terminates with `stopReason: "cancelled"`, and the `audio` and `stats` promises reject with `InferenceCancelledError`. You can also broad-cancel AudioGen work with `cancel({ modelId, kind: "audiogen" })`.

See [Runtime — Cancellation](/runtime/cancellation) for cancellation behavior shared across the SDK.

## Example

This example loads the registry-hosted ACE-Step models, reports progress, generates an instrumental, and writes the PCM output as WAV.

<Callout type="info">
  The generated Python client exposes the contract-level `audio_gen_stream()` method. Unlike the TypeScript `audioGen()` wrapper, it yields progress, base64 PCM chunks, and the terminal stats frame directly; callers decode and concatenate the chunks themselves — see the Python tab.
</Callout>

<Tabs>
  <Tab value="ts" label="TypeScript" default>
    <WrapCode>
      ```ts file=<rootDir>/packages/sdk/examples/audiogen/generate-music.ts title="generate-music.ts" lineNumbers
      import { writeFileSync } from 'node:fs'
      import {
        AUDIOGEN_ACESTEP_5HZ_LM_0_6B_Q8_0,
        AUDIOGEN_ACESTEP_V15_TURBO_Q4_K_M,
        AUDIOGEN_QWEN3_EMBEDDING_0_6B_Q8_0,
        AUDIOGEN_VAE_BF16,
        audioGen,
        loadModel,
        unloadModel,
        type ModelProgressUpdate
      } from '@qvac/sdk'

      // Usage:
      //   bun examples/audiogen/generate-music.ts "lo-fi hip hop, mellow piano" output.wav
      const caption =
        process.argv[2] ?? 'Lo-fi hip hop with mellow piano, soft drums, and a warm bass line'
      const outputPath = process.argv[3] ?? 'audiogen-output.wav'

      let modelId: string | undefined
      const lastLoggedPercentageByDownload = new Map<string, number>()
      const completedDownloads = new Set<string>()

      try {
        console.log('▸ Loading ACE-Step AudioGen models...')
        modelId = await loadModel({
          modelType: 'audiogen',
          modelConfig: {
            textEncModelSrc: AUDIOGEN_QWEN3_EMBEDDING_0_6B_Q8_0,
            lmModelSrc: AUDIOGEN_ACESTEP_5HZ_LM_0_6B_Q8_0,
            ditModelSrc: AUDIOGEN_ACESTEP_V15_TURBO_Q4_K_M,
            vaeModelSrc: AUDIOGEN_VAE_BF16,
            useGPU: true,
            inferenceSteps: 8
          },
          onProgress: (progress: ModelProgressUpdate) => {
            const mb = (bytes: number) => (bytes / 1e6).toFixed(1)
            const label = getDownloadLabel(progress.downloadKey)
            const line =
              `▸ Downloading ${label}: ${progress.percentage.toFixed(0)}% ` +
              `(${mb(progress.downloaded)}/${mb(progress.total)} MB)`

            if (process.stderr.isTTY) {
              process.stderr.write(`\r${line}`)
              if (progress.percentage >= 100 && !completedDownloads.has(progress.downloadKey)) {
                completedDownloads.add(progress.downloadKey)
                process.stderr.write('\n')
              }
              return
            }

            const percentageBucket = Math.floor(progress.percentage / 5) * 5
            const lastLogged = lastLoggedPercentageByDownload.get(progress.downloadKey)
            const isNewCompletion = progress.percentage >= 100 && lastLogged !== 100
            if (lastLogged === undefined || percentageBucket > lastLogged || isNewCompletion) {
              lastLoggedPercentageByDownload.set(progress.downloadKey, percentageBucket)
              process.stderr.write(`${line}\n`)
            }
          }
        })

        console.log(`▸ Model loaded: ${modelId}`)
        console.log(`▸ Generating: ${caption}`)

        const run = audioGen({
          modelId,
          caption,
          lyrics: '[Instrumental]',
          seed: 42,
          duration: 10
        })
        console.log(`▸ requestId: ${run.requestId}`)

        for await (const progress of run.progressStream) {
          console.log(`▸ ${progress.stage}: ${progress.step}/${progress.total}`)
        }

        const [audio, stats] = await Promise.all([run.audio, run.stats])
        const wav = createWav(audio.pcm, audio.sampleRate, audio.channels, audio.bitsPerSample)
        writeFileSync(outputPath, wav)

        const samplesPerChannel = audio.pcm.byteLength / (audio.bitsPerSample / 8) / audio.channels
        console.log(
          `▸ Generated ${samplesPerChannel} samples per channel at ` +
            `${audio.sampleRate} Hz (${audio.channels} channels)`
        )
        if (stats) console.log(`▸ Stats: ${JSON.stringify(stats)}`)
        console.log(`▸ Saved ${outputPath}`)

        await unloadModel({ modelId })
        modelId = undefined
        console.log('▸ Model unloaded')
        process.exit(0)
      } catch (error) {
        if (modelId !== undefined) {
          try {
            await unloadModel({ modelId })
          } catch {
            // Preserve the generation error as the primary failure.
          }
        }
        console.error('✖', error)
        process.exit(1)
      }

      function getDownloadLabel(downloadKey: string) {
        return downloadKey.split('/').pop() ?? downloadKey
      }

      function createWav(pcm: Uint8Array, sampleRate: number, channels: number, bitsPerSample: number) {
        const header = new ArrayBuffer(44)
        const view = new DataView(header)
        const blockAlign = channels * (bitsPerSample / 8)

        writeAscii(view, 0, 'RIFF')
        view.setUint32(4, 36 + pcm.byteLength, true)
        writeAscii(view, 8, 'WAVE')
        writeAscii(view, 12, 'fmt ')
        view.setUint32(16, 16, true)
        view.setUint16(20, 1, true)
        view.setUint16(22, channels, true)
        view.setUint32(24, sampleRate, true)
        view.setUint32(28, sampleRate * blockAlign, true)
        view.setUint16(32, blockAlign, true)
        view.setUint16(34, bitsPerSample, true)
        writeAscii(view, 36, 'data')
        view.setUint32(40, pcm.byteLength, true)

        const wav = new Uint8Array(44 + pcm.byteLength)
        wav.set(new Uint8Array(header))
        wav.set(pcm, 44)
        return wav
      }

      function writeAscii(view: DataView, offset: number, value: string) {
        for (let index = 0; index < value.length; index++) {
          view.setUint8(offset + index, value.charCodeAt(index))
        }
      }
      ```
    </WrapCode>
  </Tab>

  <Tab value="python" label="Python">
    <WrapCode>
      ```python file=<rootDir>/packages/sdk-python/examples/audiogen.py title="audiogen.py" lineNumbers
      """Python port of packages/sdk/examples/audiogen/generate-music.ts.

      Generate music with ACE-Step AudioGen. Python exposes the contract-level
      `audio_gen_stream` method, so this example decodes and joins its base64 PCM
      chunks before writing a WAV file.

      RUN:
        python examples/audiogen.py "lo-fi hip hop, mellow piano" audiogen-output.wav
      """

      from __future__ import annotations

      import argparse
      import asyncio
      import base64
      import sys
      import wave

      from _common import print_progress

      from tetherto.qvac_sdk import (
          Client,
          generate_client_request_id,
          load_model,
          unload_model,
      )
      from tetherto.qvac_sdk.methods import audio_gen_stream
      from tetherto.qvac_sdk.models import (
          AUDIOGEN_ACESTEP_5HZ_LM_0_6B_Q8_0,
          AUDIOGEN_ACESTEP_V15_TURBO_Q4_K_M,
          AUDIOGEN_QWEN3_EMBEDDING_0_6B_Q8_0,
          AUDIOGEN_VAE_BF16,
      )
      from tetherto.qvac_sdk.schemas import AudioGenStreamRequest


      def write_wav(
          pcm: bytes, sample_rate: int, channels: int, bits_per_sample: int, path: str
      ) -> None:
          with wave.open(path, "wb") as output:
              output.setnchannels(channels)
              output.setsampwidth(bits_per_sample // 8)
              output.setframerate(sample_rate)
              output.writeframes(pcm)


      async def main() -> int:
          parser = argparse.ArgumentParser(description="Generate music with AudioGen")
          parser.add_argument(
              "caption",
              nargs="?",
              default="Lo-fi hip hop with mellow piano, soft drums, and warm bass",
          )
          parser.add_argument("output", nargs="?", default="audiogen-output.wav")
          args = parser.parse_args()

          async with Client() as client:
              transport = client.transport
              model_id: str | None = None

              try:
                  print("▸ Loading ACE-Step AudioGen models...")
                  model_id = await load_model(
                      transport,
                      model_type="audiogen-ggml",
                      model_config={
                          "textEncModelSrc": AUDIOGEN_QWEN3_EMBEDDING_0_6B_Q8_0.src,
                          "lmModelSrc": AUDIOGEN_ACESTEP_5HZ_LM_0_6B_Q8_0.src,
                          "ditModelSrc": AUDIOGEN_ACESTEP_V15_TURBO_Q4_K_M.src,
                          "vaeModelSrc": AUDIOGEN_VAE_BF16.src,
                          "useGPU": True,
                          "inferenceSteps": 8,
                      },
                      on_progress=print_progress,
                  )
                  print(f"▸ Model loaded: {model_id}")

                  request_id = generate_client_request_id()
                  request = AudioGenStreamRequest(
                      model_id=model_id,
                      request_id=request_id,
                      caption=args.caption,
                      lyrics="[Instrumental]",
                      seed=42,
                      duration=10,
                  )

                  print(f"▸ requestId: {request_id}")
                  print(f"▸ Generating: {args.caption}")

                  pcm_chunks: list[bytes] = []
                  sample_rate: int | None = None
                  channels: int | None = None
                  bits_per_sample: int | None = None
                  stats = None

                  async for response in audio_gen_stream(transport, request):
                      if response.progress is not None:
                          progress = response.progress
                          print(f"▸ {progress.stage}: {progress.step}/{progress.total}")
                      if response.data is not None:
                          pcm_chunks.append(base64.b64decode(response.data))
                          sample_rate = response.sample_rate
                          channels = response.channels
                          bits_per_sample = response.bits_per_sample
                      if response.done:
                          stats = response.stats
                          if (
                              response.stop_reason is not None
                              and response.stop_reason.value == "cancelled"
                          ):
                              print("▸ Generation cancelled", file=sys.stderr)
                              return 1
                          break

                  if (
                      sample_rate is None
                      or channels is None
                      or bits_per_sample is None
                      or bits_per_sample % 8 != 0
                      or not pcm_chunks
                  ):
                      raise RuntimeError("AudioGen stream ended without audio data")

                  pcm = b"".join(pcm_chunks)
                  write_wav(pcm, sample_rate, channels, bits_per_sample, args.output)
                  samples_per_channel = len(pcm) // (bits_per_sample // 8) // channels
                  print(
                      f"▸ Generated {samples_per_channel} samples per channel at "
                      f"{sample_rate} Hz ({channels} channels)"
                  )
                  if stats is not None:
                      print(f"▸ Stats: {stats.model_dump(by_alias=True)}")
                  print(f"▸ Saved {args.output}")
              except Exception as error:
                  print(f"✖ {error}", file=sys.stderr)
                  return 1
              finally:
                  if model_id is not None:
                      try:
                          await unload_model(transport, model_id)
                          print("▸ Model unloaded")
                      except Exception as unload_error:
                          print(f"✖ Failed to unload model: {unload_error}", file=sys.stderr)

          return 0


      if __name__ == "__main__":
          sys.exit(asyncio.run(main()))
      ```
    </WrapCode>
  </Tab>
</Tabs>

<Callout type="success">
  **Tip:** see the [JS/TS quickstart](/js-ts-sdk#quickstart) or the [Python
  quickstart](/python-sdk#quickstart) for setup instructions, and apply the
  first-run download configuration above on constrained connections.
</Callout>