Class Llama

java.lang.Object
smile.llm.model.llama.Llama
All Implemented Interfaces:
ModelExecutor, LanguageModel
public class Llama extends Object implements LanguageModel, ModelExecutor
LLaMA model specification.

  • Constructor Summary

    Constructors
    Constructor
    Description
    Llama(String name, LlamaModel model, Tokenizer tokenizer, LlamaModelArgs params)
    Constructor.

  • Method Summary

    Modifier and Type
    Method
    Description
    int
    bind(int[] prompt, int totalCapacity)
    Binds one request into the KV pool (multi-request safe).
    static Llama
    build(String checkpointDir, String tokenizerPath, int maxBatchSize, int maxSeqLen, byte deviceId)
    Builds a Llama instance by initializing and loading a model checkpoint.
    static Llama
    build(String checkpointDir, String tokenizerPath, int maxBatchSize, int maxSeqLen, byte deviceId, double memFractionStatic)
    Builds a Llama instance by initializing and loading a model checkpoint.
    static Llama
    build(String checkpointDir, String tokenizerPath, int maxBatchSize, int maxSeqLen, byte deviceId, double memFractionStatic, String kvCacheDtype)
    Builds a Llama instance by initializing and loading a model checkpoint.
    static Llama
    build(String checkpointDir, String tokenizerPath, int maxBatchSize, int maxSeqLen, byte deviceId, double memFractionStatic, String kvCacheDtype, int pageSize)
    Builds a Llama instance by initializing and loading a model checkpoint.
    static Llama
    build(String checkpointDir, String tokenizerPath, int maxBatchSize, int maxSeqLen, byte deviceId, double memFractionStatic, String kvCacheDtype, int pageSize, int modelLoaderThreads)
    Builds a Llama instance by initializing and loading a model checkpoint.
    ChatCompletion
    chat(Message[] dialog, int maxGenLen, double temperature, double topp, boolean logprobs, long seed, GenerationListener listener, BooleanSupplier cancelRequested)
    Generates an assistant response for a conversational dialog.
    ChatCompletion
    complete(String prompt, int maxGenLen, double temperature, double topp, boolean logprobs, long seed, GenerationListener listener)
    Performs text completion for a prompt.
    String
    decode(int[] tokens)
    Decodes token ids to UTF-8 text.
    Tensor
    decodeStep(int[] requestIds, int[] lastTokens, int[] positions)
    One decode step for an active set (same or mixed positions; engine may cohort by position).
    int[]
    encodeChat(Message... dialog)
    Encodes a dialog with the model chat template, leaving the assistant turn open for completion.
    void
    evict(int requestId)
    Instant Eviction without radix insert.
    String
    family()
    Returns the architecture family label.
    void
    finish(int requestId, int[] sequenceTokens)
    Inserts into the radix tree (when enabled) and unbinds the request.
    ChatCompletion
    generate(int[] prompt, int maxGenLen, double temperature, double topp, boolean logprobs, long seed, GenerationListener listener, BooleanSupplier cancelRequested)
    Generates a text completion from a tokenized prompt using nucleus sampling.
    KvCachePool
    kvCachePool()
    Primary KV cache pool (rank 0 for TP).
    int
    maxSeqLen()
    Returns the configured maximum sequence length (max-model-len / context length).
    LanguageModel
    model()
    Returns the underlying language-model façade.
    String
    name()
    Returns the model instance name.
    int
    padToken()
    Returns the pad token id.
    LlamaModelArgs
    params()
    Returns the transformer hyperparameters loaded from the checkpoint.
    static ScalarType
    parseDtypeName(String name)
    Parses a human-readable floating-point dtype name.
    Tensor
    prefill(int requestId, int[] prompt, int prefixLen)
    Prefills the unbound suffix of a prompt (activates this request alone).
    Tensor
    prefillChunk(int requestId, int[] prompt, int from, int to)
    Prefills prompt[from, to) into the KV cache (exclusive activate).
    int
    prefixLen(int requestId)
    Page-aligned matched prefix length for a bound request.
    void
    setPrefixReuseEnabled(boolean enabled)
    Enables or disables radix prefix reuse for all KV pools on this model.
    int[]
    stopTokens()
    Returns stop / EOS token ids.
    String
    toString()
     
    String
    tryDecode(int[] tokens, boolean skipSpecial)
    Attempts a streaming decode; returns empty on incomplete UTF-8.

    Methods inherited from class Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

    Methods inherited from interface LanguageModel

    chat, encodeChat, generate

    Methods inherited from interface ModelExecutor

    idleDecodeGraphPrefetch, prefillMultimodal, speculateStep, speculateStep, supportsStepApi, warmPrefix

  • Constructor Details

    • Llama

      public Llama(String name, LlamaModel model, Tokenizer tokenizer, LlamaModelArgs params)
      Constructor.
      Parameters:
      name - the model name.
      model - the transformer model.
      tokenizer - the tokenizer.
      params - the model hyperparameters.

  • Method Details

    • toString

      public String toString()
      Overrides:
      toString in class Object

    • family

      public String family()
      Description copied from interface: LanguageModel
      Returns the architecture family label.

      Examples: meta/llama3, alibaba/qwen3.5. This is not necessarily the public API model id used by serve.

      Specified by:
      family in interface LanguageModel
      Returns:
      the family label string.

    • setPrefixReuseEnabled

      public void setPrefixReuseEnabled(boolean enabled)
      Enables or disables radix prefix reuse for all KV pools on this model.
      Parameters:
      enabled - true to match/insert prefixes across requests.

    • params

      public LlamaModelArgs params()
      Returns the transformer hyperparameters loaded from the checkpoint.
      Returns:
      model args from config.json / params.json.

    • name

      public String name()
      Returns the model instance name.
      Specified by:
      name in interface LanguageModel
      Returns:
      the model instance name.

    • maxSeqLen

      public int maxSeqLen()
      Description copied from interface: LanguageModel
      Returns the configured maximum sequence length (max-model-len / context length).
      Specified by:
      maxSeqLen in interface LanguageModel
      Specified by:
      maxSeqLen in interface ModelExecutor
      Returns:
      the maximum number of tokens allowed for prompt plus completion.

    • encodeChat

      public int[] encodeChat(Message... dialog)
      Description copied from interface: LanguageModel
      Encodes a dialog with the model chat template, leaving the assistant turn open for completion.
      Specified by:
      encodeChat in interface LanguageModel
      Parameters:
      dialog - ordered conversation turns (system / user / assistant).
      Returns:
      prompt token ids ready for LanguageModel.generate(int[], int, double, double, boolean, long, GenerationListener).

    • build

      public static Llama build(String checkpointDir, String tokenizerPath, int maxBatchSize, int maxSeqLen, byte deviceId) throws IOException
      Builds a Llama instance by initializing and loading a model checkpoint.

      Supports two on-disk layouts:

      • Meta — params.json plus consolidated.*.pt shards.
      • HuggingFace — config.json plus *.safetensors (optionally indexed by model.safetensors.index.json).
      Parameters:
      checkpointDir - the directory path of checkpoint files.
      tokenizerPath - the path of tokenizer model file.
      maxBatchSize - the maximum batch size for inference.
      maxSeqLen - the maximum sequence length for input text.
      deviceId - the optional CUDA device ID. If negative, don't use CUDA.
      Returns:
      an instance of Llama model.
      Throws:
      IOException - if fail to open model checkpoint.

    • build

      public static Llama build(String checkpointDir, String tokenizerPath, int maxBatchSize, int maxSeqLen, byte deviceId, double memFractionStatic) throws IOException
      Builds a Llama instance by initializing and loading a model checkpoint.

      When memFractionStatic > 0, a KvCachePool is allocated after weight loading with SGLang --mem-fraction-static semantics: y is a fraction of total GPU memory for the static region (weights + KV); see smile.chat.mem-fraction-static in smile-serve.

      Parameters:
      checkpointDir - the directory path of checkpoint files.
      tokenizerPath - the path of tokenizer model file.
      maxBatchSize - the maximum batch size for inference.
      maxSeqLen - the maximum sequence length for input text.
      deviceId - the optional CUDA device ID. If negative, don't use CUDA.
      memFractionStatic - static-region fraction of total GPU memory for weights + KV; <= 0 keeps the default test-sized pool.
      Returns:
      an instance of Llama model.
      Throws:
      IOException - if fail to open model checkpoint.

    • build

      public static Llama build(String checkpointDir, String tokenizerPath, int maxBatchSize, int maxSeqLen, byte deviceId, double memFractionStatic, String kvCacheDtype) throws IOException
      Builds a Llama instance by initializing and loading a model checkpoint.

      When memFractionStatic > 0, a KvCachePool is allocated after weight loading with SGLang --mem-fraction-static semantics: y is a fraction of total GPU memory for the static region (weights + KV); see smile.chat.mem-fraction-static in smile-serve.

      Parameters:
      checkpointDir - the directory path of checkpoint files.
      tokenizerPath - the path of tokenizer model file.
      maxBatchSize - the maximum batch size for inference.
      maxSeqLen - the maximum sequence length for input text.
      deviceId - the optional CUDA device ID. If negative, don't use CUDA.
      memFractionStatic - static-region fraction of total GPU memory for weights + KV; <= 0 keeps the default test-sized pool.
      kvCacheDtype - optional KV-cache element dtype name (e.g. bfloat16, float16); null/blank uses torch_dtype from config.json, then the CUDA compute dtype.
      Returns:
      an instance of Llama model.
      Throws:
      IOException - if fail to open model checkpoint.

    • build

      public static Llama build(String checkpointDir, String tokenizerPath, int maxBatchSize, int maxSeqLen, byte deviceId, double memFractionStatic, String kvCacheDtype, int pageSize) throws IOException
      Builds a Llama instance by initializing and loading a model checkpoint.

      When memFractionStatic > 0, a KvCachePool is allocated after weight loading with SGLang --mem-fraction-static semantics: y is a fraction of total GPU memory for the static region (weights + KV); see smile.chat.mem-fraction-static in smile-serve.

      Parameters:
      checkpointDir - the directory path of checkpoint files.
      tokenizerPath - the path of tokenizer model file.
      maxBatchSize - the maximum batch size for inference.
      maxSeqLen - the maximum sequence length for input text.
      deviceId - the optional CUDA device ID. If negative, don't use CUDA.
      memFractionStatic - static-region fraction of total GPU memory for weights + KV; <= 0 keeps the default test-sized pool.
      kvCacheDtype - optional KV-cache element dtype name (e.g. bfloat16, float16); null/blank uses torch_dtype from config.json, then the CUDA compute dtype.
      pageSize - tokens per radix / KV pool page (>= 1).
      Returns:
      an instance of Llama model.
      Throws:
      IOException - if fail to open model checkpoint.

    • build

      public static Llama build(String checkpointDir, String tokenizerPath, int maxBatchSize, int maxSeqLen, byte deviceId, double memFractionStatic, String kvCacheDtype, int pageSize, int modelLoaderThreads) throws IOException
      Builds a Llama instance by initializing and loading a model checkpoint.

      When memFractionStatic > 0, a KvCachePool is allocated after weight loading with SGLang --mem-fraction-static semantics: y is a fraction of total GPU memory for the static region (weights + KV); see smile.chat.mem-fraction-static in smile-serve.

      Parameters:
      checkpointDir - the directory path of checkpoint files.
      tokenizerPath - the path of tokenizer model file.
      maxBatchSize - the maximum batch size for inference.
      maxSeqLen - the maximum sequence length for input text.
      deviceId - the optional CUDA device ID. If negative, don't use CUDA.
      memFractionStatic - static-region fraction of total GPU memory for weights + KV; <= 0 keeps the default test-sized pool.
      kvCacheDtype - optional KV-cache element dtype name (e.g. bfloat16, float16); null/blank uses torch_dtype from config.json, then the CUDA compute dtype.
      pageSize - tokens per radix / KV pool page (>= 1).
      modelLoaderThreads - safetensors loader threads; 0 = auto (SafeTensorsLoaderThreads.resolve(int, int)).
      Returns:
      an instance of Llama model.
      Throws:
      IOException - if fail to open model checkpoint.

    • parseDtypeName

      public static ScalarType parseDtypeName(String name)
      Parses a human-readable floating-point dtype name.
      Parameters:
      name - dtype name (case-insensitive), e.g. bfloat16, float16.
      Returns:
      the corresponding ScalarType.

    • generate

      public ChatCompletion generate(int[] prompt, int maxGenLen, double temperature, double topp, boolean logprobs, long seed, GenerationListener listener, BooleanSupplier cancelRequested)
      Generates a text completion from a tokenized prompt using nucleus sampling.
      Specified by:
      generate in interface LanguageModel
      Parameters:
      prompt - tokenized prompt.
      maxGenLen - maximum number of new tokens to generate.
      temperature - temperature value for controlling randomness in sampling.
      topp - top-p probability threshold for nucleus sampling.
      logprobs - flag indicating whether to compute token log probabilities.
      seed - optional RNG seed to sample deterministically; 0 is non-deterministic.
      listener - optional generation progress callback.
      cancelRequested - when non-null and returns true, generation stops between decode steps with CancellationException.
      Returns:
      the generated text completion.

    • complete

      public ChatCompletion complete(String prompt, int maxGenLen, double temperature, double topp, boolean logprobs, long seed, GenerationListener listener)
      Performs text completion for a prompt.
      Parameters:
      prompt - text prompt.
      maxGenLen - maximum number of new tokens to generate.
      temperature - temperature value for controlling randomness in sampling.
      topp - top-p probability threshold for nucleus sampling.
      logprobs - flag indicating whether to compute token log probabilities.
      seed - optional RNG seed to sample deterministically.
      listener - optional generation progress callback.
      Returns:
      the generated text completion.

    • chat

      public ChatCompletion chat(Message[] dialog, int maxGenLen, double temperature, double topp, boolean logprobs, long seed, GenerationListener listener, BooleanSupplier cancelRequested)
      Generates an assistant response for a conversational dialog.
      Specified by:
      chat in interface LanguageModel
      Parameters:
      dialog - ordered conversation turns.
      maxGenLen - maximum number of new tokens to generate.
      temperature - temperature value for controlling randomness in sampling.
      topp - top-p probability threshold for nucleus sampling.
      logprobs - flag indicating whether to compute token log probabilities.
      seed - optional RNG seed to sample deterministically.
      listener - optional generation progress callback.
      cancelRequested - cooperative cancel flag; may be null.
      Returns:
      the generated chat response.

    • model

      public LanguageModel model()
      Description copied from interface: ModelExecutor
      Returns the underlying language-model façade.
      Specified by:
      model in interface ModelExecutor
      Returns:
      language model.

    • kvCachePool

      public KvCachePool kvCachePool()
      Description copied from interface: ModelExecutor
      Primary KV cache pool (rank 0 for TP). May be null only for models without full-attention layers.
      Specified by:
      kvCachePool in interface ModelExecutor
      Returns:
      KV cache pool, or null.

    • padToken

      public int padToken()
      Description copied from interface: ModelExecutor
      Returns the pad token id.
      Specified by:
      padToken in interface ModelExecutor
      Returns:
      pad token id.

    • stopTokens

      public int[] stopTokens()
      Description copied from interface: ModelExecutor
      Returns stop / EOS token ids.
      Specified by:
      stopTokens in interface ModelExecutor
      Returns:
      stop token ids.

    • decode

      public String decode(int[] tokens)
      Description copied from interface: ModelExecutor
      Decodes token ids to UTF-8 text.
      Specified by:
      decode in interface ModelExecutor
      Parameters:
      tokens - token ids.
      Returns:
      decoded text.

    • tryDecode

      public String tryDecode(int[] tokens, boolean skipSpecial) throws CharacterCodingException
      Description copied from interface: ModelExecutor
      Attempts a streaming decode; returns empty on incomplete UTF-8.
      Specified by:
      tryDecode in interface ModelExecutor
      Parameters:
      tokens - token ids.
      skipSpecial - whether to skip special tokens.
      Returns:
      decoded text, or empty on incomplete UTF-8.
      Throws:
      CharacterCodingException - when the sequence is incomplete.

    • bind

      public int bind(int[] prompt, int totalCapacity)
      Description copied from interface: ModelExecutor
      Binds one request into the KV pool (multi-request safe).
      Specified by:
      bind in interface ModelExecutor
      Parameters:
      prompt - prompt token ids.
      totalCapacity - desired slots (prompt + generation).
      Returns:
      request id (> 0).

    • prefixLen

      public int prefixLen(int requestId)
      Description copied from interface: ModelExecutor
      Page-aligned matched prefix length for a bound request.
      Specified by:
      prefixLen in interface ModelExecutor
      Parameters:
      requestId - id from ModelExecutor.bind(int[], int).
      Returns:
      prefix length (>= 0).

    • prefill

      public Tensor prefill(int requestId, int[] prompt, int prefixLen)
      Description copied from interface: ModelExecutor
      Prefills the unbound suffix of a prompt (activates this request alone).
      Specified by:
      prefill in interface ModelExecutor
      Parameters:
      requestId - request id.
      prompt - full prompt tokens.
      prefixLen - cached prefix to skip (from ModelExecutor.prefixLen(int)).
      Returns:
      last-position logits shaped [1, vocab] (caller owns).

    • prefillChunk

      public Tensor prefillChunk(int requestId, int[] prompt, int from, int to)
      Prefills prompt[from, to) into the KV cache (exclusive activate). Returns last-token logits only when to == prompt.length; otherwise null (intermediate chunk).
      Specified by:
      prefillChunk in interface ModelExecutor
      Parameters:
      requestId - KV request id.
      prompt - full prompt tokens.
      from - start position (usually matched prefix, possibly trimmed).
      to - exclusive end (<= prompt.length).
      Returns:
      logits [1, vocab] or null.

    • decodeStep

      public Tensor decodeStep(int[] requestIds, int[] lastTokens, int[] positions)
      Description copied from interface: ModelExecutor
      One decode step for an active set (same or mixed positions; engine may cohort by position). Tokens are [B] last tokens; positions are the write index for each row.
      Specified by:
      decodeStep in interface ModelExecutor
      Parameters:
      requestIds - active KV request ids (order = batch).
      lastTokens - token id per request.
      positions - absolute position per request.
      Returns:
      logits [B, vocab] (caller owns).

    • finish

      public void finish(int requestId, int[] sequenceTokens)
      Description copied from interface: ModelExecutor
      Inserts into the radix tree (when enabled) and unbinds the request.
      Specified by:
      finish in interface ModelExecutor
      Parameters:
      requestId - request id.
      sequenceTokens - prompt + completion (no pad).

    • evict

      public void evict(int requestId)
      Description copied from interface: ModelExecutor
      Instant Eviction without radix insert.
      Specified by:
      evict in interface ModelExecutor
      Parameters:
      requestId - request id.