Class GenAiChatModel

java.lang.Object
smile.onnx.genai.GenAiChatModel
All Implemented Interfaces:
AutoCloseable, LanguageModel
public final class GenAiChatModel extends Object implements LanguageModel, AutoCloseable
LanguageModel adapter over ONNX Runtime GenAI.

Holds a shared Model + Tokenizer and creates a per-call Generator. Streaming uses TokenizerStream and GenerationListener.onText(String). Cooperative cancel is checked between Generator.generateNextToken() steps.

Serve wires this via ChatService OGA fallback (tools I/O, optional multimodal). Continuous batching via GenAI OgaEngine is deferred.

  • Method Details

    • of

      public static GenAiChatModel of(String modelDir)
      Loads a GenAI chat model from a model directory.
      Parameters:
      modelDir - directory containing genai_config.json.
      Returns:
      chat model owned by the caller.

    • of

      public static GenAiChatModel of(Path modelDir)
      Loads a GenAI chat model from a model directory using providers from genai_config.json.
      Parameters:
      modelDir - directory containing genai_config.json.
      Returns:
      chat model owned by the caller.

    • open

      public static GenAiChatModel open(Path modelDir)
      Opens a GenAI chat model, preferring CUDA when available (see Model.open(Path)).
      Parameters:
      modelDir - directory containing genai_config.json.
      Returns:
      chat model owned by the caller.

    • open

      public static GenAiChatModel open(String modelDir)
      Opens a GenAI chat model, preferring CUDA when available.
      Parameters:
      modelDir - directory containing genai_config.json.
      Returns:
      chat model owned by the caller.

    • of

      public static GenAiChatModel of(Config config)
      Loads a GenAI chat model from a Config.
      Parameters:
      config - GenAI config.
      Returns:
      chat model owned by the caller.

    • family

      public String family()
      Description copied from interface: LanguageModel
      Returns the architecture family label.

      Examples: meta/llama3, alibaba/qwen3.5. This is not necessarily the public API model id used by serve.

      Specified by:
      family in interface LanguageModel
      Returns:
      the family label string.

    • name

      public String name()
      Description copied from interface: LanguageModel
      Returns the model instance / checkpoint name.
      Specified by:
      name in interface LanguageModel
      Returns:
      the instance name (e.g. directory or HF repo leaf name).

    • provider

      public String provider()
      Returns the GenAI execution-provider path selected at open (Model.provider()).
      Returns:
      provider id (e.g. cuda, dml, default).

    • maxSeqLen

      public int maxSeqLen()
      Description copied from interface: LanguageModel
      Returns the configured maximum sequence length (max-model-len / context length).
      Specified by:
      maxSeqLen in interface LanguageModel
      Returns:
      the maximum number of tokens allowed for prompt plus completion.

    • encodeChat

      public int[] encodeChat(Message... dialog)
      Description copied from interface: LanguageModel
      Encodes a dialog with the model chat template, leaving the assistant turn open for completion.
      Specified by:
      encodeChat in interface LanguageModel
      Parameters:
      dialog - ordered conversation turns (system / user / assistant).
      Returns:
      prompt token ids ready for LanguageModel.generate(int[], int, double, double, boolean, long, GenerationListener).

    • encodeChat

      public int[] encodeChat(Message[] dialog, ChatOptions options)
      Description copied from interface: LanguageModel
      Encodes a dialog with optional tool-calling options.

      Default ignores options and delegates to LanguageModel.encodeChat(Message...).

      Specified by:
      encodeChat in interface LanguageModel
      Parameters:
      dialog - ordered conversation turns.
      options - chat options; may be null.
      Returns:
      prompt token ids.

    • generate

      public ChatCompletion generate(int[] prompt, int maxGenLen, double temperature, double topp, boolean logprobs, long seed, GenerationListener listener, BooleanSupplier cancelRequested)
      Description copied from interface: LanguageModel
      Generates a completion from an already-tokenized prompt.
      Specified by:
      generate in interface LanguageModel
      Parameters:
      prompt - prompt token id sequence.
      maxGenLen - maximum number of new tokens to generate (not including the prompt itself).
      temperature - sampling temperature; higher values increase randomness.
      topp - nucleus-sampling top-p threshold in (0, 1].
      logprobs - true to include per-token log-probabilities in the result.
      seed - optional RNG seed for deterministic sampling; 0 means non-deterministic.
      listener - optional progress callback; may be null.
      cancelRequested - when non-null and returns true, generation stops between decode steps with CancellationException.
      Returns:
      the completion for prompt.

    • chat

      public ChatCompletion chat(Message[] dialog, int maxGenLen, double temperature, double topp, boolean logprobs, long seed, GenerationListener listener, BooleanSupplier cancelRequested)
      Description copied from interface: LanguageModel
      Generates an assistant reply for a dialog.

      Equivalent to LanguageModel.encodeChat(Message...) followed by LanguageModel.generate(int[], int, double, double, boolean, long, GenerationListener).

      Specified by:
      chat in interface LanguageModel
      Parameters:
      dialog - ordered conversation turns.
      maxGenLen - maximum number of new tokens to generate.
      temperature - sampling temperature; higher values increase randomness.
      topp - nucleus-sampling top-p threshold in (0, 1].
      logprobs - true to include per-token log-probabilities in the result.
      seed - optional RNG seed for deterministic sampling; 0 means non-deterministic.
      listener - optional progress callback; may be null.
      cancelRequested - cooperative cancel flag; may be null.
      Returns:
      the completion for dialog.

    • close

      public void close()
      Specified by:
      close in interface AutoCloseable