---
title: "Decisions"
canonical: https://docs.qvac.tether.io/sdk/v0.23/ai-capabilities/decisions/
collection: "SDK"
package: "@qvac/sdk"
line: v0.23
current_line: false
---

# Decisions (/sdk/v0.23/ai-capabilities/decisions)



## Overview

Decisions currently use [`@qvac/embed-llamacpp`](https://github.com/tetherto/qvac/tree/main/packages/embed-llamacpp) with the Fabric LLM inference engine. Load a compatible decision model using `modelType: 'llamacpp-decision'` and `modelConfig: { device: 'cpu' }` or `{ device: 'gpu' }`.

[`decide()`](/sdk/v0.23/reference/api#decide) answers typed questions about a state. Pass `state` for one result, or `states` for an array of results in input order. A state can be text, a number, a boolean, structured JSON, or a conversation array. Top-level `null` is not accepted.

The result contains `answers`, keyed by your question IDs, and `usage` with token counts and truncation information. Each answer includes confidence information. `action.actProbability` currently carries no usable signal and must not drive application actions. Decisions return one aggregate result without a token stream.

## Functions

Use the following sequence of function calls:

1. [`loadModel()`](/sdk/v0.23/reference/api#loadmodel)
2. [`decide()`](/sdk/v0.23/reference/api#decide)
3. [`unloadModel()`](/sdk/v0.23/reference/api#unloadmodel)

For how to use each function, see [SDK — API reference](/sdk/v0.23/reference/api/).

## Models

Use a compatible decision checkpoint in GGUF format. The SDK exports these registry constants from `@qvac/sdk` and `@qvac/sdk/models`:

* Laya: `LAYA_421M_F16` and `LAYA_421M_Q8_0`.
* Laya multilingual: `LAYA_MULTILINGUAL_322M_F16` and `LAYA_MULTILINGUAL_322M_Q8_0`.
* Laya typed decisions: `LAYA_TYPED_DECISIONS_421M_F16` and `LAYA_TYPED_DECISIONS_421M_Q8_0`.

Pass a registry constant as `modelSrc` to `loadModel()`. Its `llamacpp-decision` engine selects the decision plugin automatically. The decision and embedding plugins share an addon package, but their descriptors are not interchangeable. The example below loads `LAYA_MULTILINGUAL_322M_Q8_0` by default.

You can also pass a local GGUF path or another model source supported by [`loadModel()`](/sdk/v0.23/reference/api#loadmodel) with explicit `modelType: 'llamacpp-decision'`. An embedding-only GGUF cannot answer decision questions. For checkpoints split into multiple GGUF files, see [Sharded models](/sdk/v0.23/models/sharded-models).

## Questions and token budgets

Each entry in `questions` has a `type` and `instructions`:

* `choice` selects a label from `criteria`, which can be an array of labels or an object mapping labels to descriptions. The answer contains `choice` and a probability per label.
* `score` estimates an ordinal level from `criteria`, an array of descriptions ordered from lowest to highest. The answer contains the expected `score`, a `legend`, and probabilities.
* `noul` estimates the probability that a statement holds. The answer contains `noul`. Optional `criteria` describe the true and false options, and `labels` change the text shown for them.

`optionOrder` changes the order in which the model sees options. Answers still use the original option order. `maxLen` sets the token budget per sequence, and `headMaxLen` sets the budget for a question and its options. Omit them to use the checkpoint's budgets. Inspect `usage.truncated`, `usage.truncatedQuestions`, and `usage.stateTokensDropped` to detect truncation.

The load-time `batch_size` controls tokens per forward pass; each sequence must fit within it. `threads`, `threads-batch`, `gpu_layers`, and `flash_attn` control execution. The decision backend fixes context size, pooling, and parallelism; these settings cannot be configured through the SDK.

## Cancellation and concurrency

The promise returned by [`decide()`](/sdk/v0.23/reference/api#decide) exposes `requestId` before you await it. Pass that ID to [`cancel()`](/sdk/v0.23/reference/api#cancel) to interrupt the request. Its promise rejects with `InferenceCancelledError` after the native response settles.

Each loaded model runs one decision request at a time. Concurrent calls wait in a FIFO queue with up to 64 waiting requests per model. When the queue is full, additional calls reject with `RequestRejectedByPolicyError` without affecting active or queued requests. Cancelling a queued call by `requestId` rejects it with `InferenceCancelledError` before native work starts and leaves the active request running. Use `states` to evaluate several states in one request. See [Cancellation](/sdk/v0.23/runtime/cancellation) for broad cancellation by model ID.

## Example

The following script loads the default registry checkpoint, or a local GGUF path passed as its first argument, and answers choice, urgency-score, and refund-probability questions about a support ticket:

<Tabs>
  <Tab value="js" label="JavaScript" default>
    <WrapCode>
      ```js file=<rootDir>/packages/sdk/dist/examples/decisions.js title="decisions.js" lineNumbers
      import { loadModel, decide, unloadModel, close, LAYA_MULTILINGUAL_322M_Q8_0 } from '@qvac/sdk';
      const modelSrc = process.argv[2] ?? LAYA_MULTILINGUAL_322M_Q8_0;
      let modelId;
      try {
          console.error('▸ Loading decision model');
          modelId = await loadModel({
              modelSrc,
              modelType: 'llamacpp-decision',
              modelConfig: { device: 'cpu' }
          });
          const result = await decide({
              modelId,
              state: 'My payment failed twice and I was charged both times. Please refund the duplicate.',
              questions: {
                  department: {
                      type: 'choice',
                      instructions: 'Which team should handle this ticket?',
                      criteria: ['billing', 'technical']
                  },
                  urgency: {
                      type: 'score',
                      instructions: 'How urgent is this?',
                      criteria: ['not urgent', 'somewhat urgent', 'very urgent']
                  },
                  refund: { type: 'noul', instructions: 'The customer asks for a refund.' }
              }
          });
          console.log(JSON.stringify(result, null, 2));
      }
      catch (error) {
          console.error('✖', error);
          process.exitCode = 1;
      }
      finally {
          if (modelId)
              await unloadModel({ modelId });
          await close();
      }
      ```
    </WrapCode>
  </Tab>

  <Tab value="ts" label="TypeScript">
    <WrapCode>
      ```ts file=<rootDir>/packages/sdk/examples/decisions.ts title="decisions.ts" lineNumbers
      import { loadModel, decide, unloadModel, close, LAYA_MULTILINGUAL_322M_Q8_0 } from '@qvac/sdk'

      const modelSrc = process.argv[2] ?? LAYA_MULTILINGUAL_322M_Q8_0

      let modelId: string | undefined
      try {
        console.error('▸ Loading decision model')
        modelId = await loadModel({
          modelSrc,
          modelType: 'llamacpp-decision',
          modelConfig: { device: 'cpu' }
        })
        const result = await decide({
          modelId,
          state: 'My payment failed twice and I was charged both times. Please refund the duplicate.',
          questions: {
            department: {
              type: 'choice',
              instructions: 'Which team should handle this ticket?',
              criteria: ['billing', 'technical']
            },
            urgency: {
              type: 'score',
              instructions: 'How urgent is this?',
              criteria: ['not urgent', 'somewhat urgent', 'very urgent']
            },
            refund: { type: 'noul', instructions: 'The customer asks for a refund.' }
          }
        })
        console.log(JSON.stringify(result, null, 2))
      } catch (error) {
        console.error('✖', error)
        process.exitCode = 1
      } finally {
        if (modelId) await unloadModel({ modelId })
        await close()
      }
      ```
    </WrapCode>
  </Tab>

  <Tab value="python" label="Python">
    <WrapCode>
      ```python file=<rootDir>/packages/sdk-python/examples/decisions.py title="decisions.py" lineNumbers
      """Python port of packages/sdk/examples/decisions.ts.

      Loads the default registry decision checkpoint, or a local GGUF path passed as
      the first argument, and answers choice, urgency-score, and refund-probability
      questions about a support ticket.

      Python calls the generated `decide` stub, so the request is a `DecideRequest`
      and the result is dumped with its camelCase wire names, like the JS output.
      `Client` closes the SDK on exit, which the TypeScript example does with
      `close()`.

      RUN: python examples/decisions.py [path-to-decision.gguf]
      """

      from __future__ import annotations

      import asyncio
      import json
      import sys

      from tetherto.qvac_sdk import (
          Client,
          DecideRequest,
          decide,
          load_model,
          unload_model,
      )
      from tetherto.qvac_sdk.models import LAYA_MULTILINGUAL_322M_Q8_0


      async def main() -> int:
          model_src = sys.argv[1] if len(sys.argv) > 1 else LAYA_MULTILINGUAL_322M_Q8_0

          async with Client() as client:
              t = client.transport
              model_id = None
              try:
                  print("▸ Loading decision model", file=sys.stderr)
                  model_id = await load_model(
                      t,
                      model_src=model_src,
                      model_type="llamacpp-decision",
                      model_config={"device": "cpu"},
                  )
                  request = DecideRequest.model_validate(
                      {
                          "type": "decide",
                          "modelId": model_id,
                          "state": (
                              "My payment failed twice and I was charged both times. "
                              "Please refund the duplicate."
                          ),
                          "questions": {
                              "department": {
                                  "type": "choice",
                                  "instructions": "Which team should handle this ticket?",
                                  "criteria": ["billing", "technical"],
                              },
                              "urgency": {
                                  "type": "score",
                                  "instructions": "How urgent is this?",
                                  "criteria": [
                                      "not urgent",
                                      "somewhat urgent",
                                      "very urgent",
                                  ],
                              },
                              "refund": {
                                  "type": "noul",
                                  "instructions": "The customer asks for a refund.",
                              },
                          },
                      }
                  )
                  response = await decide(t, request)
                  result = response.result.model_dump(
                      mode="json", by_alias=True, exclude_none=True
                  )
                  print(json.dumps(result, indent=2))
              except Exception as error:
                  print(f"✖ {error}", file=sys.stderr)
                  return 1
              finally:
                  if model_id is not None:
                      await unload_model(t, model_id)
          return 0


      if __name__ == "__main__":
          sys.exit(asyncio.run(main()))
      ```
    </WrapCode>
  </Tab>

  <Tab value="kotlin" label="Kotlin">
    <WrapCode>
      ```kotlin file=<rootDir>/packages/sdk-kotlin/examples/src/Decisions.kt title="Decisions.kt" lineNumbers
      // Kotlin port of packages/sdk/examples/decisions.ts.
      //
      // Loads the default registry decision checkpoint, or a local GGUF path passed
      // as the first argument, and answers choice, urgency-score, and
      // refund-probability questions about a support ticket.
      //
      // Kotlin calls the generated typed `decide` operation, so the request is built
      // from the generated `DecideRequest` classes: each question is an arm of the
      // sealed `DecideSingleRequestQuestionsValue` (`Choice`, `Score`, `Noul`), and a
      // field that accepts several JSON types is a sealed union whose arms are named
      // `Variant1`, `Variant2`, ... (`Variant1` is the string arm here). A question
      // with a missing field or a wrong type is a compile error. The result is
      // serialized back with its wire names, like the JS output.
      //
      // RUN: ./gradlew -p examples run -Pexample=Decisions --args="[path-to-decision.gguf]"

      import io.tether.qvac.sdk.QvacClient
      import io.tether.qvac.sdk.decide
      import io.tether.qvac.sdk.generated.Models
      import io.tether.qvac.sdk.generated.schema.DecideRequest
      import io.tether.qvac.sdk.generated.schema.DecideSingleRequest
      import io.tether.qvac.sdk.generated.schema.DecideSingleRequestQuestionsValue
      import io.tether.qvac.sdk.generated.schema.DecideSingleRequestQuestionsValueChoice
      import io.tether.qvac.sdk.generated.schema.DecideSingleRequestQuestionsValueChoiceCriteria
      import io.tether.qvac.sdk.generated.schema.DecideSingleRequestQuestionsValueChoiceCriteriaVariant1Item
      import io.tether.qvac.sdk.generated.schema.DecideSingleRequestQuestionsValueChoiceInstructions
      import io.tether.qvac.sdk.generated.schema.DecideSingleRequestQuestionsValueNoul
      import io.tether.qvac.sdk.generated.schema.DecideSingleRequestQuestionsValueNoulInstructions
      import io.tether.qvac.sdk.generated.schema.DecideSingleRequestQuestionsValueScore
      import io.tether.qvac.sdk.generated.schema.DecideSingleRequestQuestionsValueScoreCriteriaItem
      import io.tether.qvac.sdk.generated.schema.DecideSingleRequestQuestionsValueScoreInstructions
      import io.tether.qvac.sdk.generated.schema.DecideSingleRequestState
      import io.tether.qvac.sdk.jvm.JvmBareKitTransport
      import io.tether.qvac.sdk.models
      import kotlinx.coroutines.runBlocking
      import kotlinx.serialization.json.Json
      import kotlinx.serialization.json.buildJsonObject
      import kotlinx.serialization.json.put

      private val PRETTY = Json {
          prettyPrint = true
          explicitNulls = false
      }

      fun main(args: Array<String>): Unit = runBlocking {
          val modelSrc = args.getOrNull(0) ?: Models.LAYA_MULTILINGUAL_322M_Q8_0.src

          val client = QvacClient(JvmBareKitTransport.connect())
          var modelId: String? = null
          try {
              System.err.println("▸ Loading decision model")
              val loaded = client.models.load(
                  source = modelSrc,
                  modelType = "llamacpp-decision",
                  modelConfig = buildJsonObject { put("device", "cpu") },
              )
              modelId = loaded.modelId ?: error(loaded.error ?: "loadModel failed")

              val request = DecideRequest.DecideSingleRequest(
                  DecideSingleRequest(
                      modelId = modelId,
                      state = DecideSingleRequestState.Variant1(
                          "My payment failed twice and I was charged both times. Please refund the duplicate.",
                      ),
                      questions = mapOf(
                          "department" to DecideSingleRequestQuestionsValue.Choice(
                              DecideSingleRequestQuestionsValueChoice(
                                  instructions = DecideSingleRequestQuestionsValueChoiceInstructions.Variant1(
                                      "Which team should handle this ticket?",
                                  ),
                                  criteria = DecideSingleRequestQuestionsValueChoiceCriteria.Variant1(
                                      listOf("billing", "technical")
                                          .map(DecideSingleRequestQuestionsValueChoiceCriteriaVariant1Item::Variant1),
                                  ),
                              ),
                          ),
                          "urgency" to DecideSingleRequestQuestionsValue.Score(
                              DecideSingleRequestQuestionsValueScore(
                                  instructions = DecideSingleRequestQuestionsValueScoreInstructions.Variant1(
                                      "How urgent is this?",
                                  ),
                                  criteria = listOf("not urgent", "somewhat urgent", "very urgent")
                                      .map(DecideSingleRequestQuestionsValueScoreCriteriaItem::Variant1),
                              ),
                          ),
                          "refund" to DecideSingleRequestQuestionsValue.Noul(
                              DecideSingleRequestQuestionsValueNoul(
                                  instructions = DecideSingleRequestQuestionsValueNoulInstructions.Variant1(
                                      "The customer asks for a refund.",
                                  ),
                              ),
                          ),
                      ),
                  ),
              )
              val response = client.decide(request)
              println(PRETTY.encodeToString(response.result))
          } finally {
              modelId?.let { client.models.unload(it) }
              client.close()
          }
      }
      ```
    </WrapCode>
  </Tab>
</Tabs>

<Callout type="success">
  **Tip:** all examples throughout this documentation are self-contained and runnable. For instructions on how to run them, see the [JS/TS quickstart](/sdk/v0.23/js-ts-sdk#quickstart), the [Python quickstart](/sdk/v0.23/python-sdk#quickstart), or the [Kotlin quickstart](/sdk/v0.23/kotlin-sdk#quickstart).
</Callout>
