Interface LanguageModel

All Known Implementing Classes:
GenAiChatModel, Llama, MiniMax, Qwen
public interface LanguageModel
Common façade for chat-capable decoder LLMs (Llama, Qwen, …).

Implementations load a checkpoint, expose chat-template encoding, and run autoregressive generation for a single prompt. Concurrent multi-request scheduling belongs in InferenceEngine, not on this interface.

Streaming and metrics use a single optional GenerationListener. Transport-specific types such as SubmissionPublisher belong at the serve boundary (see GenerationListeners.toPublisher(SubmissionPublisher)).

Cooperative cancel: pass a non-null cancelRequested supplier; the implementation checks it between decode steps and throws CancellationException when true (KV is still unbound in finally).

  • Method Summary

    Modifier and Type
    Method
    Description
    default ChatCompletion
    chat(Message[] dialog, int maxGenLen, double temperature, double topp, boolean logprobs, long seed, GenerationListener listener)
    Generates an assistant reply for a dialog (no cancel).
    ChatCompletion
    chat(Message[] dialog, int maxGenLen, double temperature, double topp, boolean logprobs, long seed, GenerationListener listener, BooleanSupplier cancelRequested)
    Generates an assistant reply for a dialog.
    int[]
    encodeChat(Message... dialog)
    Encodes a dialog with the model chat template, leaving the assistant turn open for completion.
    default int[]
    encodeChat(Message[] dialog, ChatOptions options)
    Encodes a dialog with optional tool-calling options.
    String
    family()
    Returns the architecture family label.
    default ChatCompletion
    generate(int[] prompt, int maxGenLen, double temperature, double topp, boolean logprobs, long seed, GenerationListener listener)
    Generates a completion from an already-tokenized prompt (no cancel).
    ChatCompletion
    generate(int[] prompt, int maxGenLen, double temperature, double topp, boolean logprobs, long seed, GenerationListener listener, BooleanSupplier cancelRequested)
    Generates a completion from an already-tokenized prompt.
    int
    maxSeqLen()
    Returns the configured maximum sequence length (max-model-len / context length).
    String
    name()
    Returns the model instance / checkpoint name.

  • Method Details

    • family

      String family()
      Returns the architecture family label.

      Examples: meta/llama3, alibaba/qwen3.5. This is not necessarily the public API model id used by serve.

      Returns:
      the family label string.

    • name

      String name()
      Returns the model instance / checkpoint name.
      Returns:
      the instance name (e.g. directory or HF repo leaf name).

    • maxSeqLen

      int maxSeqLen()
      Returns the configured maximum sequence length (max-model-len / context length).
      Returns:
      the maximum number of tokens allowed for prompt plus completion.

    • encodeChat

      int[] encodeChat(Message... dialog)
      Encodes a dialog with the model chat template, leaving the assistant turn open for completion.
      Parameters:
      dialog - ordered conversation turns (system / user / assistant).
      Returns:
      prompt token ids ready for generate(int[], int, double, double, boolean, long, GenerationListener).

    • encodeChat

      default int[] encodeChat(Message[] dialog, ChatOptions options)
      Encodes a dialog with optional tool-calling options.

      Default ignores options and delegates to encodeChat(Message...).

      Parameters:
      dialog - ordered conversation turns.
      options - chat options; may be null.
      Returns:
      prompt token ids.

    • generate

      default ChatCompletion generate(int[] prompt, int maxGenLen, double temperature, double topp, boolean logprobs, long seed, GenerationListener listener)
      Generates a completion from an already-tokenized prompt (no cancel).
      Parameters:
      prompt - prompt token id sequence.
      maxGenLen - maximum number of new tokens to generate.
      temperature - sampling temperature.
      topp - nucleus-sampling top-p threshold.
      logprobs - whether to include per-token log-probabilities.
      seed - RNG seed; 0 means non-deterministic.
      listener - optional progress callback; may be null.
      Returns:
      the completion for prompt.
      See Also:

    • generate

      ChatCompletion generate(int[] prompt, int maxGenLen, double temperature, double topp, boolean logprobs, long seed, GenerationListener listener, BooleanSupplier cancelRequested)
      Generates a completion from an already-tokenized prompt.
      Parameters:
      prompt - prompt token id sequence.
      maxGenLen - maximum number of new tokens to generate (not including the prompt itself).
      temperature - sampling temperature; higher values increase randomness.
      topp - nucleus-sampling top-p threshold in (0, 1].
      logprobs - true to include per-token log-probabilities in the result.
      seed - optional RNG seed for deterministic sampling; 0 means non-deterministic.
      listener - optional progress callback; may be null.
      cancelRequested - when non-null and returns true, generation stops between decode steps with CancellationException.
      Returns:
      the completion for prompt.

    • chat

      default ChatCompletion chat(Message[] dialog, int maxGenLen, double temperature, double topp, boolean logprobs, long seed, GenerationListener listener)
      Generates an assistant reply for a dialog (no cancel).
      Parameters:
      dialog - chat messages.
      maxGenLen - maximum number of new tokens to generate.
      temperature - sampling temperature.
      topp - nucleus-sampling top-p threshold.
      logprobs - whether to include per-token log-probabilities.
      seed - RNG seed; 0 means non-deterministic.
      listener - optional progress callback; may be null.
      Returns:
      the chat completion.
      See Also:

    • chat

      ChatCompletion chat(Message[] dialog, int maxGenLen, double temperature, double topp, boolean logprobs, long seed, GenerationListener listener, BooleanSupplier cancelRequested)
      Generates an assistant reply for a dialog.

      Equivalent to encodeChat(Message...) followed by generate(int[], int, double, double, boolean, long, GenerationListener).

      Parameters:
      dialog - ordered conversation turns.
      maxGenLen - maximum number of new tokens to generate.
      temperature - sampling temperature; higher values increase randomness.
      topp - nucleus-sampling top-p threshold in (0, 1].
      logprobs - true to include per-token log-probabilities in the result.
      seed - optional RNG seed for deterministic sampling; 0 means non-deterministic.
      listener - optional progress callback; may be null.
      cancelRequested - cooperative cancel flag; may be null.
      Returns:
      the completion for dialog.