# World simulation (/ai-capabilities/world-simulation)



## Overview

World simulation runs on a **customized Diffusion engine** ([`qvac-ext-stable-diffusion.cpp`](https://github.com/tetherto/qvac-ext-stable-diffusion.cpp)) exposed as an interactive session powered by **ABot-World**. Load a supported model using `modelType: "sdcpp-generation"` with `modelConfig.mode: "world"`. Then call `worldCreateScene()` once to build a world from a text prompt and a first-frame image, and `worldStep()` repeatedly to walk it. Each call generates one block of video streamed frame-by-frame.

The session runs on the machine hosting the worker; there is no remote route.

<Callout type="warn">
  World simulation is hardware-intensive. The default **832×480** tier needs **≥20 GB** of video memory (a 24 GB card in practice). Pass **448×256** to use the low-VRAM tier, which runs on \~6 GB cards well below interactive frame rates.
</Callout>

## Functions

Use the following sequence of function calls:

1. [`loadModel()`](/reference/api#loadmodel)
2. [`worldCreateScene()`](/reference/api) — once per world
3. [`worldStep()`](/reference/api) — repeatedly, one call per generated block
4. [`unloadModel()`](/reference/api#unloadmodel)

For how to use each function, see [SDK — API reference](/reference/api/).

### `worldCreateScene()`

Builds the world from a text `prompt` and a first-frame `image` (`Uint8Array` of PNG or JPEG bytes, ≤ 3 MB). Optional `width` and `height` are positive multiples of **32**, at most **4096** each, and `width × height` must stay within **2,088,960** pixels (1920×1088). They default to **832×480**.

It returns `{ requestId, stats }`, plus `scene` (a `Promise<Uint8Array>` holding the scene pack) when you pass `returnPack: true`. Save that pack and load it back later with `modelConfig.sceneSrc` to walk the same world again without regenerating it. Calling `worldCreateScene()` twice on the same session replaces the world and restarts the walk.

### `worldStep()`

Generates the next block of the walk under the `keys` held for it. One call produces one block — a second call while one is in flight is rejected, so drive the next call off the previous one.

`keys` accepts three equivalent forms:

* an array of key names (`["W", "L"]`),
* a key-state object (`{ W: true, L: true }`), or
* a raw 8-bit mask.

Use **WASD** to move (forward, left, back, right) and **IJKL** to steer the camera (up, left, down, right). Pass an empty value for an idle block.

It returns `{ requestId, frameStream, progressStream, frames, stats }`:

* `frameStream` — an async iterable of decoded frames, PNG by default or JPEG when `modelConfig.world.frameJpegQuality` is set. Display each frame as it arrives.
* `progressStream` — emits one summary tick per block, **after** the frames.
* `frames` — a `Promise` resolving to the whole block once decoding is done.
* `stats` — timing and frame counts.

Cancelling with `cancel({ requestId })` is block-granular: the current block finishes internally, delivery stops, and the call rejects with `InferenceCancelledError`. A cancel that lands after the block already completed resolves normally — handle both outcomes.

## Models

ABot-World uses a split layout with four models:

* `ABOT_WORLD_0_5B_Q8_0` — the world-model transformer (DiT). Passed as `modelSrc`.
* `ABOT_WORLD_0_5B_LF_VAE` — the taehv streaming decoder used on every step. Passed as `modelConfig.taehvModelSrc`.
* `ABOT_WORLD_0_5B_LF_VAE_F16` — the full-precision VAE that encodes the first frame during `worldCreateScene`. Passed as `modelConfig.vaeModelSrc`.
* `UMT5_XXL_ENC_Q8_0` — the UMT5 text encoder for the prompt. Passed as `modelConfig.t5XxlModelSrc`.

```ts
import {
  ABOT_WORLD_0_5B_Q8_0,
  ABOT_WORLD_0_5B_LF_VAE,
  ABOT_WORLD_0_5B_LF_VAE_F16,
  UMT5_XXL_ENC_Q8_0,
  loadModel,
} from "@qvac/sdk";

const modelId = await loadModel({
  modelSrc: ABOT_WORLD_0_5B_Q8_0,
  modelType: "sdcpp-generation",
  modelConfig: {
    mode: "world",
    taehvModelSrc: ABOT_WORLD_0_5B_LF_VAE,
    t5XxlModelSrc: UMT5_XXL_ENC_Q8_0,
    vaeModelSrc: ABOT_WORLD_0_5B_LF_VAE_F16,
    world: { kvCache: true, frameJpegQuality: 85 },
  },
});
```

Set `world.numFramePerBlock` to control the block size; the SDK caps it at 64.

For models available as constants, see [SDK — Models](/introduction#models).

## Example

The following script loads an ABot-World session, creates a world from a first-frame image, walks it for several blocks under a scripted key tape, and writes each decoded frame to disk:

<Tabs>
  <Tab value="js" label="JavaScript" default>
    <WrapCode>
      ```js file=<rootDir>/packages/sdk/dist/examples/abot-world.js title="abot-world.js" lineNumbers
      import { cancel, loadModel, unloadModel, worldCreateScene, worldStep, ABOT_WORLD_0_5B_Q8_0, ABOT_WORLD_0_5B_LF_TAEHV_VAE, ABOT_WORLD_0_5B_LF_WAN_VAE, UMT5_XXL_ENC_Q8_0 } from '@qvac/sdk';
      import fs from 'fs';
      import path from 'path';
      // ABot-World: build a world from a single image, then walk it. Each step
      // generates one block of video under the keys held for it and streams the
      // decoded frames.
      //
      // Runs on the machine hosting the worker; there is no remote route. The
      // default 832x480 tier needs >= 20 GB free VRAM on a dedicated GPU (24 GB card
      // in practice); pass 448 256 to use the low-VRAM tier, which runs on ~6 GB
      // cards well below interactive frame rates.
      const firstFramePath = process.argv[2];
      const prompt = process.argv[3] || '| unknown | A realistic outdoor world scene with a navigable path.';
      const outputDir = process.argv[4] || '.';
      const blocks = Number(process.argv[5] || 8);
      const width = Number(process.argv[6] || 832);
      const height = Number(process.argv[7] || 480);
      if (!firstFramePath) {
          console.error('✖ first frame image path is required');
          console.error('Usage: bun run bare:example dist/examples/abot-world.js ' +
              '<firstFrameImage> [prompt] [outputDir] [blocks] [width] [height]');
          process.exit(1);
      }
      // A scripted walk: forward, forward while looking right, forward, idle. `keys`
      // also accepts a key-state object ({ W: true }) or a raw 8-bit mask, so a live
      // keyboard handler can pass its state straight through.
      const TAPE = [['W'], ['W', 'L'], ['W'], []];
      try {
          console.log('▸ Loading ABot-World session (DiT + taehv + scene encoders)...');
          const modelId = await loadModel({
              modelSrc: ABOT_WORLD_0_5B_Q8_0,
              modelType: 'sdcpp-generation',
              modelConfig: {
                  mode: 'world',
                  // Two different VAEs, and the names are close enough to swap by accident:
                  //   taehvModelSrc — taew2_2, the tiny STREAMING decoder. Turns each block's
                  //     latents into pixels during the walk. Used by every worldStep.
                  //   vaeModelSrc   — the full-precision Wan2.2 VAE. ENCODES the first frame
                  //     during worldCreateScene, and is not touched again after that.
                  // Swapping them loads at first and fails once the walk starts.
                  taehvModelSrc: ABOT_WORLD_0_5B_LF_TAEHV_VAE,
                  t5XxlModelSrc: UMT5_XXL_ENC_Q8_0,
                  vaeModelSrc: ABOT_WORLD_0_5B_LF_WAN_VAE,
                  world: {
                      seed: 42,
                      // ~3.7x fewer frame-passes per block. Without it, block times ramp from
                      // ~1.8 s to ~7.5 s as the recompute window fills.
                      kvCache: true,
                      // JPEG rather than PNG: a block is ~14 MB of raw pixels, and this
                      // example writes every frame to disk.
                      frameJpegQuality: 85
                  }
              }
          });
          console.log('▸ Creating the world (umT5 prompt encode + Wan2.2 VAE first-frame encode)...');
          const creation = worldCreateScene({
              modelId,
              prompt,
              image: new Uint8Array(fs.readFileSync(firstFramePath)),
              width,
              height,
              // The world is live on the session without this; ask for the bytes only
              // because this example saves them to walk the same world again later.
              returnPack: true
          });
          const scene = await creation.scene;
          const sceneStats = await creation.stats;
          const scenePath = path.join(outputDir, 'world.safetensors');
          fs.writeFileSync(scenePath, scene);
          console.log(`  scene pack written to ${scenePath} (${scene.length} bytes, ${sceneStats?.sceneCreateMs ?? '?'} ms)`);
          console.log('  reuse it later with modelConfig.sceneSrc to walk this same world again');
          console.log(`▸ Walking ${blocks} blocks...`);
          let frameNumber = 0;
          for (let block = 0; block < blocks; block++) {
              const keys = TAPE[block % TAPE.length];
              // One block at a time: a second step while this one streams is rejected,
              // so drive the next call off this one rather than firing them in parallel.
              const run = worldStep({ modelId, keys });
              for await (const frame of run.frameStream) {
                  const framePath = path.join(outputDir, `frame-${String(frameNumber++).padStart(4, '0')}.jpg`);
                  fs.writeFileSync(framePath, frame);
              }
              const stats = await run.stats;
              console.log(`  block ${block + 1}/${blocks} [${keys.join('+') || 'idle'}] — ` +
                  `${stats?.frames ?? '?'} frames in ${stats?.stepMs ?? '?'} ms`);
          }
          console.log(`✔ Wrote ${frameNumber} frames to ${outputDir}`);
          // Targeted cancellation. Cancel is block-granular: the engine has no
          // mid-block abort, so the current block finishes internally, delivery stops,
          // and the step rejects rather than returning a truncated block. A cancel that
          // lands after the block already finished legitimately succeeds instead — both
          // outcomes are correct, so handle each.
          console.log('▸ Cancelling a block mid-flight to show the contract...');
          const cancelling = worldStep({ modelId, keys: ['W'] });
          // The timer can fire after the block already finished, and cancelling a
          // request that is no longer in flight rejects. Handle it here rather than
          // leaving a floating promise: an unhandled rejection from a fire-and-forget
          // cancel would take the whole example down for the race it is demonstrating.
          const cancelTimer = setTimeout(() => {
              cancel({ requestId: cancelling.requestId }).catch(() => {
                  // The block won the race; the await below reports that outcome.
              });
          }, 200);
          try {
              await cancelling.frames;
              console.log('  the block completed before the cancel landed (the tolerated race)');
          }
          catch (error) {
              console.log(`  cancelled as expected: ${error instanceof Error ? error.message : String(error)}`);
          }
          finally {
              clearTimeout(cancelTimer);
          }
          // A cancelled step is terminal for the native session, but the SDK drops it
          // for you: another worldStep on this same modelId would transparently rebuild
          // from the promoted pack, with no unloadModel/loadModel cycle. Unloading here
          // only because the example is finished.
          await unloadModel({ modelId });
      }
      catch (error) {
          console.error('✖ World walk failed:', error instanceof Error ? error.message : error);
          process.exit(1);
      }
      process.exit(0);
      ```
    </WrapCode>
  </Tab>

  <Tab value="ts" label="TypeScript">
    <WrapCode>
      ```ts file=<rootDir>/packages/sdk/examples/abot-world.ts title="abot-world.ts" lineNumbers
      import {
        cancel,
        loadModel,
        unloadModel,
        worldCreateScene,
        worldStep,
        ABOT_WORLD_0_5B_Q8_0,
        ABOT_WORLD_0_5B_LF_TAEHV_VAE,
        ABOT_WORLD_0_5B_LF_WAN_VAE,
        UMT5_XXL_ENC_Q8_0
      } from '@qvac/sdk'
      import fs from 'fs'
      import path from 'path'

      // ABot-World: build a world from a single image, then walk it. Each step
      // generates one block of video under the keys held for it and streams the
      // decoded frames.
      //
      // Runs on the machine hosting the worker; there is no remote route. The
      // default 832x480 tier needs >= 20 GB free VRAM on a dedicated GPU (24 GB card
      // in practice); pass 448 256 to use the low-VRAM tier, which runs on ~6 GB
      // cards well below interactive frame rates.
      const firstFramePath = process.argv[2]
      const prompt =
        process.argv[3] || '| unknown | A realistic outdoor world scene with a navigable path.'
      const outputDir = process.argv[4] || '.'
      const blocks = Number(process.argv[5] || 8)
      const width = Number(process.argv[6] || 832)
      const height = Number(process.argv[7] || 480)

      if (!firstFramePath) {
        console.error('✖ first frame image path is required')
        console.error(
          'Usage: bun run bare:example dist/examples/abot-world.js ' +
            '<firstFrameImage> [prompt] [outputDir] [blocks] [width] [height]'
        )
        process.exit(1)
      }

      // A scripted walk: forward, forward while looking right, forward, idle. `keys`
      // also accepts a key-state object ({ W: true }) or a raw 8-bit mask, so a live
      // keyboard handler can pass its state straight through.
      const TAPE: string[][] = [['W'], ['W', 'L'], ['W'], []]

      try {
        console.log('▸ Loading ABot-World session (DiT + taehv + scene encoders)...')
        const modelId = await loadModel({
          modelSrc: ABOT_WORLD_0_5B_Q8_0,
          modelType: 'sdcpp-generation',
          modelConfig: {
            mode: 'world',
            // Two different VAEs, and the names are close enough to swap by accident:
            //   taehvModelSrc — taew2_2, the tiny STREAMING decoder. Turns each block's
            //     latents into pixels during the walk. Used by every worldStep.
            //   vaeModelSrc   — the full-precision Wan2.2 VAE. ENCODES the first frame
            //     during worldCreateScene, and is not touched again after that.
            // Swapping them loads at first and fails once the walk starts.
            taehvModelSrc: ABOT_WORLD_0_5B_LF_TAEHV_VAE,
            t5XxlModelSrc: UMT5_XXL_ENC_Q8_0,
            vaeModelSrc: ABOT_WORLD_0_5B_LF_WAN_VAE,
            world: {
              seed: 42,
              // ~3.7x fewer frame-passes per block. Without it, block times ramp from
              // ~1.8 s to ~7.5 s as the recompute window fills.
              kvCache: true,
              // JPEG rather than PNG: a block is ~14 MB of raw pixels, and this
              // example writes every frame to disk.
              frameJpegQuality: 85
            }
          }
        })

        console.log('▸ Creating the world (umT5 prompt encode + Wan2.2 VAE first-frame encode)...')
        const creation = worldCreateScene({
          modelId,
          prompt,
          image: new Uint8Array(fs.readFileSync(firstFramePath)),
          width,
          height,
          // The world is live on the session without this; ask for the bytes only
          // because this example saves them to walk the same world again later.
          returnPack: true
        })
        const scene = await creation.scene
        const sceneStats = await creation.stats
        const scenePath = path.join(outputDir, 'world.safetensors')
        fs.writeFileSync(scenePath, scene)
        console.log(
          `  scene pack written to ${scenePath} (${scene.length} bytes, ${sceneStats?.sceneCreateMs ?? '?'} ms)`
        )
        console.log('  reuse it later with modelConfig.sceneSrc to walk this same world again')

        console.log(`▸ Walking ${blocks} blocks...`)
        let frameNumber = 0
        for (let block = 0; block < blocks; block++) {
          const keys = TAPE[block % TAPE.length]!
          // One block at a time: a second step while this one streams is rejected,
          // so drive the next call off this one rather than firing them in parallel.
          const run = worldStep({ modelId, keys })

          for await (const frame of run.frameStream) {
            const framePath = path.join(outputDir, `frame-${String(frameNumber++).padStart(4, '0')}.jpg`)
            fs.writeFileSync(framePath, frame)
          }

          const stats = await run.stats
          console.log(
            `  block ${block + 1}/${blocks} [${keys.join('+') || 'idle'}] — ` +
              `${stats?.frames ?? '?'} frames in ${stats?.stepMs ?? '?'} ms`
          )
        }

        console.log(`✔ Wrote ${frameNumber} frames to ${outputDir}`)

        // Targeted cancellation. Cancel is block-granular: the engine has no
        // mid-block abort, so the current block finishes internally, delivery stops,
        // and the step rejects rather than returning a truncated block. A cancel that
        // lands after the block already finished legitimately succeeds instead — both
        // outcomes are correct, so handle each.
        console.log('▸ Cancelling a block mid-flight to show the contract...')
        const cancelling = worldStep({ modelId, keys: ['W'] })
        // The timer can fire after the block already finished, and cancelling a
        // request that is no longer in flight rejects. Handle it here rather than
        // leaving a floating promise: an unhandled rejection from a fire-and-forget
        // cancel would take the whole example down for the race it is demonstrating.
        const cancelTimer = setTimeout(() => {
          cancel({ requestId: cancelling.requestId }).catch(() => {
            // The block won the race; the await below reports that outcome.
          })
        }, 200)
        try {
          await cancelling.frames
          console.log('  the block completed before the cancel landed (the tolerated race)')
        } catch (error) {
          console.log(
            `  cancelled as expected: ${error instanceof Error ? error.message : String(error)}`
          )
        } finally {
          clearTimeout(cancelTimer)
        }

        // A cancelled step is terminal for the native session, but the SDK drops it
        // for you: another worldStep on this same modelId would transparently rebuild
        // from the promoted pack, with no unloadModel/loadModel cycle. Unloading here
        // only because the example is finished.
        await unloadModel({ modelId })
      } catch (error) {
        console.error('✖ World walk failed:', error instanceof Error ? error.message : error)
        process.exit(1)
      }

      process.exit(0)
      ```
    </WrapCode>
  </Tab>
</Tabs>

<Callout type="info">
  The Python client supports this capability through the same worker. A dedicated Python example is not yet published — see the [Python SDK](/python-sdk) for the API surface.
</Callout>

<Callout type="success">
  **Tip:** all examples throughout this documentation are self-contained and runnable. For instructions on how to run them, see the [JS/TS quickstart](/js-ts-sdk#quickstart) or the [Python quickstart](/python-sdk#quickstart).
</Callout>