New: QVAC SDK now supports Kotlin. Build local AI into your JVM and Android apps.
QVAC Logo
SDKSessions
v0.24, the current release

Caller sessions

Let several callers share one SDK runtime, each with its own requests, cancellation scope and model leases.

Overview

One SDK runtime can serve several callers, for example an agent harness and the skill runner it starts. Without sessions, every caller shares one request namespace: any caller can cancel another caller's request, and any caller's unloadModel() frees a model that another caller is still using.

openSession() returns a session handle with the same model-work methods as the module-level SDK. Every call made through the handle carries the session's id, so:

  • cancel() reaches only the session's own requests.
  • loadModel() takes a lease on the model. unloadModel() releases only the session's lease, and the model unloads when its last lease is released.
  • session.close() cancels the session's requests, waits for them to end, disposes the vector indexes the session opened and releases its model leases.

Calls made without a session run as the owner. Existing single-caller code needs no change.

Functions

  1. openSession({ name? }) — returns a session handle at once. The open round trip runs in the background; calls on the handle wait for it.
  2. session.close() — ends the session. It does not stop the runtime.
  3. close() — the owner's shutdown. It stops the runtime and ends every session.

For how to use each function, see SDK — API reference.

Requests and cancellation

The worker keys every request by its session and its requestId, so two sessions may use the same requestId without conflict.

  • session.cancel({ requestId }) with another session's requestId cancels nothing, and the other session's request keeps running.
  • session.cancel({ modelId }) cancels only the session's requests on that model.
  • The owner's cancel({ modelId }) cancels the requests of every session on that model.

See Cancellation for the cancel forms themselves.

Model leases

A loaded model records which sessions hold it.

  • loadModel() for a configuration that is already loaded adds a lease and returns the same modelId. Concurrent loads of one configuration share one load.
  • unloadModel() releases the caller's lease and resolves with { released, unloaded }. unloaded is true only when that release was the last lease.
  • getLoadedModelInfo() reports the model's leaseCount.
  • A reload of a loaded model's configuration, loadModel({ modelId, modelConfig }), fails with ModelLeasedError while another session holds a lease on the model.
  • The modelId stays the hash of the configuration the model was first loaded with. After a reload, a session that holds no lease and loads that first configuration gets ModelReconfiguredError instead of the reconfigured model.

Owner-only operations

suspend(), resume(), deleteCache(), startRpcServer() and stopRpcServer() change the runtime for every session, so only the owner may call them. A session handle has no method for them, and the worker refuses them from a session with OwnerOnlyError. A session cannot open another session.

Runtime loss

When the worker exits or the owner calls close(), every session ends:

  • Later calls on a session fail locally with SessionInvalidatedError.
  • Calls that name a model the session had loaded fail with ModelHandleInvalidatedError until the model is loaded again.
  • Calls in flight when the worker exits fail with WorkerCrashedError.

The SDK does not reconnect a session or restart the worker.

Example

The following script opens two sessions that share one model by lease, shows that only the session that started a completion can cancel it, and releases the leases one session at a time:

sessions.js
/**
 * Two caller sessions on one runtime.
 *
 * Each session gets its own requests and model leases:
 *
 *  - Both sessions load the same configuration and get the same `modelId`;
 *    the second load adds a lease instead of loading the model again.
 *  - A session's `cancel({ requestId })` reaches only its own requests, so
 *    the second session's cancel leaves the first session's completion
 *    running; the first session's own cancel stops it.
 *  - `unloadModel()` releases the caller's lease. The model unloads when its
 *    last lease is released, here when the second session closes.
 *
 * Only the module-level `close()` stops the runtime; it ends every session.
 */
import { close, getLoadedModelInfo, openSession, QWEN3_600M_INST_Q4 } from '@qvac/sdk';
try {
    const harness = openSession({ name: 'harness' });
    const skillRunner = openSession({ name: 'skill-runner' });
    const modelId = await harness.loadModel({
        modelSrc: QWEN3_600M_INST_Q4,
        modelConfig: { ctx_size: 4096 }
    });
    const sameModelId = await skillRunner.loadModel({
        modelSrc: QWEN3_600M_INST_Q4,
        modelConfig: { ctx_size: 4096 }
    });
    const { leaseCount } = await getLoadedModelInfo({ modelId });
    console.log(`▸ same model for both sessions: ${modelId === sameModelId}, leases: ${leaseCount}`);
    const run = harness.completion({
        modelId,
        history: [{ role: 'user', content: 'Write a long essay about the history of Rome.' }],
        stream: true
    });
    // The skill runner cannot reach the harness's request.
    await skillRunner.cancel({ requestId: run.requestId });
    let deltasAfterForeignCancel = 0;
    let stopReason;
    for await (const event of run.events) {
        if (event.type === 'contentDelta') {
            deltasAfterForeignCancel++;
            // The harness stops its own completion after a few more tokens.
            if (deltasAfterForeignCancel === 5)
                await harness.cancel({ requestId: run.requestId });
        }
        else if (event.type === 'completionDone') {
            stopReason = event.stopReason;
        }
    }
    console.log(`▸ ${deltasAfterForeignCancel} tokens arrived after skill-runner's cancel; harness's cancel ended the run: stopReason=${stopReason}`);
    const result = await harness.unloadModel({ modelId });
    console.log(`▸ harness unload: released=${result.released}, unloaded=${result.unloaded}`);
    // Closing a session releases its leases; this was the model's last one.
    await skillRunner.close();
    console.log('▸ skill-runner closed and released the last lease');
    await harness.close();
    await close();
    process.exit(0);
}
catch (error) {
    console.error('✖', error);
    process.exit(1);
}

Tip: all examples throughout this documentation are self-contained and runnable. For instructions on how to run them, see the JS/TS quickstart, the Python quickstart, or the Kotlin quickstart.

Errors

  • SessionIdConflictError — the worker already has a session with the id the client generated.
  • SessionNotOpenError — the worker received a call for a session it does not have, for example one that already closed.
  • SessionInvalidatedError — a call on a session handle after the session ended: its own close(), the owner's close() or the loss of the worker.
  • ModelHandleInvalidatedError — a call naming a model the session loaded, after the session ended. Load the model again.
  • ModelLeasedError — a reload of a model's configuration while another session holds a lease on it.
  • ModelReconfiguredError — a load, from a session that holds no lease, of a configuration whose model was since reloaded with another configuration.
  • OwnerOnlyError — a session called an owner-only operation.

On this page

Ask anything about QVAC.