---
title: "Sharded models"
canonical: https://docs.qvac.tether.io/sdk/v0.23/models/sharded-models/
collection: "SDK"
package: "@qvac/sdk"
line: v0.23
current_line: false
---

# Sharded models (/sdk/v0.23/models/sharded-models)



## Overview

Some models may be distributed as **multiple files** (shards) instead of a single large one. [`loadModel()`](/sdk/v0.23/reference/api#loadmodel) supports sharded models and ensures that all shards are available before loading it. For this, shard file names **must** follow the pattern: `<name>-00001-of-0000X.<ext>`.

<Callout type="info">
  **Important:** for now, sharded models are supported only for GGUF models.
</Callout>

## Supported formats

* Archives (`.tar`, `.tar.gz`, `.tgz`): HTTP or local with automatic extraction
* HTTP sharded URL: pass the download URL of any shard and the SDK will fetch the remaining shards
* Local shards: pass the path to any shard file.

## Local sharded models

All files **must** be present in the same directory:

* All numbered shard files, for example:
  * `model-00001-of-00005.gguf`
  * `model-00002-of-00005.gguf`
  * …
  * `model-00005-of-00005.gguf`
* A companion tensors manifest file:
  * `model.tensors.txt`

When using `loadModel()`, you pass the path to the first shard (e.g., `model-00001-of-00005.gguf`), and the SDK automatically detects and loads the remaining shards.

## Functions

1. [`loadModel()`](/sdk/v0.23/reference/api#loadmodel) — pass the path/URL of any shard; SDK fetches remaining shards
2. Use the model as usual ([`completion()`](/sdk/v0.23/reference/api#completion), [`embed()`](/sdk/v0.23/reference/api#embed), etc.)
3. [`unloadModel()`](/sdk/v0.23/reference/api#unloadmodel)

For how to use each function, see [SDK — API reference](/sdk/v0.23/reference/api/).

## Example

The following script shows an example of loading a sharded model with per-shard progress tracking:

<Tabs>
  <Tab value="js" label="JavaScript" default>
    <WrapCode>
      ```js file=<rootDir>/packages/sdk/dist/examples/llamacpp-sharded.js title="sharded-models.js" lineNumbers
      import { completion, loadModel, unloadModel, VERBOSITY } from '@qvac/sdk';
      // Sharded models can be loaded from:
      // 1. HTTP archives: "https://example.com/model.tar.gz"
      // 2. HTTP pattern: "https://example.com/model-00001-of-00005.gguf"
      // 3. Hyperdrive: use any sharded model source/constant, eg: LLAMA_3_2_1B_INST_Q4_0_SHARD
      // 4. Local filesystem: pass the path to the first shard file (Note: All shards must be in the same directory)
      // 5. Local archive: pass the path to the archive file (.tar, .tar.gz, .tgz)
      try {
          const modelId = await loadModel({
              modelSrc: 'https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct-GGUF/resolve/main/qwen2.5-coder-7b-instruct-q4_0-00001-of-00002.gguf',
              modelType: 'llamacpp-completion',
              modelConfig: {
                  device: 'gpu',
                  ctx_size: 2048,
                  verbosity: VERBOSITY.ERROR
              },
              onProgress: (progress) => {
                  // For sharded models, progress.shardInfo contains detailed progress for both
                  // individual shards AND overall download progress across all shards
                  if (progress.shardInfo) {
                      // For pattern-based or Hyperdrive shards
                      const { shardInfo } = progress;
                      console.log(`▸ Downloading ${shardInfo.shardName} (${shardInfo.currentShard}/${shardInfo.totalShards})\n` +
                          `   File: ${progress.percentage.toFixed(1)}% (${(progress.downloaded / 1024 / 1024).toFixed(2)}MB / ${(progress.total / 1024 / 1024).toFixed(2)}MB)\n` +
                          `   Overall: ${shardInfo.overallPercentage.toFixed(1)}% (${(shardInfo.overallDownloaded / 1024 / 1024).toFixed(2)}MB / ${(shardInfo.overallTotal / 1024 / 1024).toFixed(2)}MB)`);
                  }
                  else {
                      // For archive-based shards
                      console.log(`▸ Progress: ${progress.percentage.toFixed(1)}% ` +
                          `(${(progress.downloaded / 1024 / 1024).toFixed(2)}MB / ${(progress.total / 1024 / 1024).toFixed(2)}MB)`);
                  }
              }
          });
          const history = [
              {
                  role: 'user',
                  content: 'What are the benefits of sharding large language models? Use emojis in your response.'
              }
          ];
          const result = completion({ modelId, history, stream: true });
          console.log('\n▸ Model response:');
          for await (const token of result.tokenStream) {
              process.stdout.write(token);
          }
          const stats = await result.stats;
          console.log('\n\n▸ Performance Stats:', stats);
          await unloadModel({ modelId, clearStorage: false });
          process.exit(0);
      }
      catch (error) {
          console.error('✖', error);
          process.exit(1);
      }
      ```
    </WrapCode>
  </Tab>

  <Tab value="ts" label="TypeScript">
    <WrapCode>
      ```ts file=<rootDir>/packages/sdk/examples/llamacpp-sharded.ts title="sharded-models.ts" lineNumbers
      import { completion, loadModel, unloadModel, VERBOSITY } from '@qvac/sdk'

      // Sharded models can be loaded from:
      // 1. HTTP archives: "https://example.com/model.tar.gz"
      // 2. HTTP pattern: "https://example.com/model-00001-of-00005.gguf"
      // 3. Hyperdrive: use any sharded model source/constant, eg: LLAMA_3_2_1B_INST_Q4_0_SHARD
      // 4. Local filesystem: pass the path to the first shard file (Note: All shards must be in the same directory)
      // 5. Local archive: pass the path to the archive file (.tar, .tar.gz, .tgz)

      try {
        const modelId = await loadModel({
          modelSrc:
            'https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct-GGUF/resolve/main/qwen2.5-coder-7b-instruct-q4_0-00001-of-00002.gguf',
          modelType: 'llamacpp-completion',
          modelConfig: {
            device: 'gpu',
            ctx_size: 2048,
            verbosity: VERBOSITY.ERROR
          },
          onProgress: (progress) => {
            // For sharded models, progress.shardInfo contains detailed progress for both
            // individual shards AND overall download progress across all shards
            if (progress.shardInfo) {
              // For pattern-based or Hyperdrive shards
              const { shardInfo } = progress

              console.log(
                `▸ Downloading ${shardInfo.shardName} (${shardInfo.currentShard}/${shardInfo.totalShards})\n` +
                  `   File: ${progress.percentage.toFixed(1)}% (${(progress.downloaded / 1024 / 1024).toFixed(2)}MB / ${(progress.total / 1024 / 1024).toFixed(2)}MB)\n` +
                  `   Overall: ${shardInfo.overallPercentage.toFixed(1)}% (${(shardInfo.overallDownloaded / 1024 / 1024).toFixed(2)}MB / ${(shardInfo.overallTotal / 1024 / 1024).toFixed(2)}MB)`
              )
            } else {
              // For archive-based shards
              console.log(
                `▸ Progress: ${progress.percentage.toFixed(1)}% ` +
                  `(${(progress.downloaded / 1024 / 1024).toFixed(2)}MB / ${(progress.total / 1024 / 1024).toFixed(2)}MB)`
              )
            }
          }
        })

        const history = [
          {
            role: 'user',
            content:
              'What are the benefits of sharding large language models? Use emojis in your response.'
          }
        ]

        const result = completion({ modelId, history, stream: true })

        console.log('\n▸ Model response:')
        for await (const token of result.tokenStream) {
          process.stdout.write(token)
        }

        const stats = await result.stats
        console.log('\n\n▸ Performance Stats:', stats)

        await unloadModel({ modelId, clearStorage: false })
        process.exit(0)
      } catch (error) {
        console.error('✖', error)
        process.exit(1)
      }
      ```
    </WrapCode>
  </Tab>

  <Tab value="python" label="Python">
    <WrapCode>
      ```python file=<rootDir>/packages/sdk-python/examples/sharded_models.py title="sharded_models.py" lineNumbers
      """Python port of packages/sdk/examples/llamacpp-sharded.ts.

      Load a model distributed as multiple shards, tracking per-shard download
      progress, then stream a completion against it.

      Sharded models can be loaded from:
      1. HTTP archives: "https://example.com/model.tar.gz"
      2. HTTP pattern: "https://example.com/model-00001-of-00005.gguf"
      3. Hyperdrive: use any sharded model source/constant, eg: LLAMA_3_2_1B_INST_Q4_0_SHARD
      4. Local filesystem: pass the path to the first shard file (Note: All shards must
         be in the same directory)
      5. Local archive: pass the path to the archive file (.tar, .tar.gz, .tgz)

      RUN: python examples/sharded_models.py
      """

      from __future__ import annotations

      import asyncio
      import sys

      from tetherto.qvac_sdk import Client, completion, load_model, unload_model


      def print_progress(p) -> None:
          """Print shard download progress; pass as `on_progress=` to `load_model`."""
          # For sharded models, p.shard_info contains detailed progress for both
          # individual shards AND overall download progress across all shards
          if p.shard_info is not None:
              # For pattern-based or Hyperdrive shards. The shard counters are floats
              # on the wire, so format them as integers.
              shard_info = p.shard_info
              print(
                  f"▸ Downloading {shard_info.shard_name} "
                  f"({shard_info.current_shard:.0f}/{shard_info.total_shards:.0f})\n"
                  f"   File: {p.percentage:.1f}% "
                  f"({p.downloaded / 1024 / 1024:.2f}MB / "
                  f"{p.total / 1024 / 1024:.2f}MB)\n"
                  f"   Overall: {shard_info.overall_percentage:.1f}% "
                  f"({shard_info.overall_downloaded / 1024 / 1024:.2f}MB / "
                  f"{shard_info.overall_total / 1024 / 1024:.2f}MB)"
              )
          else:
              # For archive-based shards
              print(
                  f"▸ Progress: {p.percentage:.1f}% "
                  f"({p.downloaded / 1024 / 1024:.2f}MB / "
                  f"{p.total / 1024 / 1024:.2f}MB)"
              )


      async def main() -> int:
          async with Client() as client:
              t = client.transport
              try:
                  model_id = await load_model(
                      t,
                      model_src=(
                          "https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct-GGUF/"
                          "resolve/main/qwen2.5-coder-7b-instruct-q4_0-00001-of-00002.gguf"
                      ),
                      model_type="llamacpp-completion",
                      model_config={
                          "device": "gpu",
                          "ctx_size": 2048,
                          "verbosity": 0,  # 0=ERROR, 1=WARN, 2=INFO, 3=DEBUG
                      },
                      on_progress=print_progress,
                  )

                  run = completion(
                      t,
                      model_id=model_id,
                      history=[
                          {
                              "role": "user",
                              "content": (
                                  "What are the benefits of sharding large language "
                                  "models? Use emojis in your response."
                              ),
                          }
                      ],
                  )

                  print("\n▸ Model response:")
                  async for event in run.events:
                      if event.type == "contentDelta":
                          sys.stdout.write(event.text)
                          sys.stdout.flush()

                  final = await run.final
                  print("\n\n▸ Performance Stats:", final.stats)

                  await unload_model(t, model_id, clear_storage=False)
              except Exception as error:
                  print(f"✖ {error}", file=sys.stderr)
                  return 1
          return 0


      if __name__ == "__main__":
          sys.exit(asyncio.run(main()))
      ```
    </WrapCode>
  </Tab>

  <Tab value="kotlin" label="Kotlin">
    <WrapCode>
      ```kotlin file=<rootDir>/packages/sdk-kotlin/examples/src/ShardedModels.kt title="ShardedModels.kt" lineNumbers
      // Kotlin port of packages/sdk/examples/llamacpp-sharded.ts.
      //
      // Load a model distributed as multiple shards, tracking per-shard download
      // progress, then stream a completion against it.
      //
      // Sharded models can be loaded from:
      // 1. HTTP archives: "https://example.com/model.tar.gz"
      // 2. HTTP pattern: "https://example.com/model-00001-of-00005.gguf"
      // 3. Hyperdrive: use any sharded model source/constant, eg: Models.LLAMA_3_2_1B_INST_Q4_0_SHARD
      // 4. Local filesystem: pass the path to the first shard file (Note: All shards must
      //    be in the same directory)
      // 5. Local archive: pass the path to the archive file (.tar, .tar.gz, .tgz)
      //
      // Progress frames are the generated typed `ModelProgressResponse`; its nullable
      // `shardInfo` tells shard-pattern downloads apart from archive downloads.
      //
      // RUN: ./gradlew -p examples run -Pexample=ShardedModels

      import io.tether.qvac.sdk.QvacClient
      import io.tether.qvac.sdk.QvacCompletionEvent
      import io.tether.qvac.sdk.QvacMessage
      import io.tether.qvac.sdk.QvacProgressEvent
      import io.tether.qvac.sdk.completion
      import io.tether.qvac.sdk.generated.schema.LoadModelResponse
      import io.tether.qvac.sdk.generated.schema.ModelProgressResponse
      import io.tether.qvac.sdk.jvm.JvmBareKitTransport
      import io.tether.qvac.sdk.models
      import io.tether.qvac.sdk.run
      import kotlinx.coroutines.runBlocking
      import kotlinx.serialization.json.JsonObject
      import kotlinx.serialization.json.buildJsonObject
      import kotlinx.serialization.json.put

      private const val MODEL_SRC = "https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct-GGUF/" +
          "resolve/main/qwen2.5-coder-7b-instruct-q4_0-00001-of-00002.gguf"

      fun main(): Unit = runBlocking {
          val client = QvacClient(JvmBareKitTransport.connect())
          try {
              val modelId = client.loadWithProgress(
                  MODEL_SRC,
                  modelType = "llamacpp-completion",
                  modelConfig = buildJsonObject {
                      put("device", "gpu")
                      put("ctx_size", 2048)
                      put("verbosity", 0) // 0=ERROR, 1=WARN, 2=INFO, 3=DEBUG
                  },
              )

              val run = client.completion.run(
                  modelId = modelId,
                  history = listOf(
                      QvacMessage.user(
                          "What are the benefits of sharding large language models? Use emojis in your response.",
                      ),
                  ),
              )

              println("\n▸ Model response:")
              run.events.collect { event ->
                  if (event is QvacCompletionEvent.ContentDelta) {
                      print(event.text)
                      System.out.flush()
                  }
              }

              val final = run.final.await()
              println("\n\n▸ Performance Stats: ${final.stats?.raw}")

              client.models.unload(modelId, clearStorage = false)
          } finally {
              client.close()
          }
      }

      /** Load [source], printing shard download progress, and return its model id. */
      private suspend fun QvacClient.loadWithProgress(source: String, modelType: String, modelConfig: JsonObject): String {
          var result: LoadModelResponse? = null
          models.loadWithProgress(source, modelType, modelConfig = modelConfig).collect { event ->
              when (event) {
                  is QvacProgressEvent.Progress -> printProgress(event.value)
                  is QvacProgressEvent.Result -> result = event.value
              }
          }
          val loaded = checkNotNull(result) { "loadModel ended without a result" }
          return loaded.modelId ?: error(loaded.error ?: "loadModel failed")
      }

      private fun printProgress(p: ModelProgressResponse) {
          // For sharded models, p.shardInfo contains detailed progress for both
          // individual shards AND overall download progress across all shards
          val shardInfo = p.shardInfo
          if (shardInfo != null) {
              // For pattern-based or Hyperdrive shards. The shard counters are doubles
              // on the wire, so format them as integers.
              println(
                  "▸ Downloading %s (%.0f/%.0f)".format(shardInfo.shardName, shardInfo.currentShard, shardInfo.totalShards),
              )
              println("   File: %.1f%% (%.2fMB / %.2fMB)".format(p.percentage, mb(p.downloaded), mb(p.total)))
              println(
                  "   Overall: %.1f%% (%.2fMB / %.2fMB)".format(
                      shardInfo.overallPercentage,
                      mb(shardInfo.overallDownloaded),
                      mb(shardInfo.overallTotal),
                  ),
              )
          } else {
              // For archive-based shards
              println("▸ Progress: %.1f%% (%.2fMB / %.2fMB)".format(p.percentage, mb(p.downloaded), mb(p.total)))
          }
      }

      private fun mb(bytes: Double) = bytes / 1024 / 1024
      ```
    </WrapCode>
  </Tab>
</Tabs>

<Callout type="success">
  **Tip:** all examples throughout this documentation are self-contained and runnable. For instructions on how to run them, see the [JS/TS quickstart](/sdk/v0.23/js-ts-sdk#quickstart), the [Python quickstart](/sdk/v0.23/python-sdk#quickstart), or the [Kotlin quickstart](/sdk/v0.23/kotlin-sdk#quickstart).
</Callout>
