Class Qwen

java.lang.Object
smile.llm.model.qwen.Qwen
All Implemented Interfaces:
AutoCloseable, ModelExecutor, LanguageModel
public class Qwen extends Object implements LanguageModel, AutoCloseable, ModelExecutor
Qwen3.5 hybrid text model (Gated DeltaNet + gated full attention).

  • Constructor Details

    • Qwen

      public Qwen(String name, QwenModel model, Tokenizer tokenizer, QwenModelArgs params)
      Constructor.
      Parameters:
      name - model instance / checkpoint name.
      model - decoder module (single-device).
      tokenizer - chat / completion tokenizer.
      params - hyperparameters from the checkpoint.

    • Qwen

      public Qwen(String name, QwenModel[] models, TensorParallelGroup tpGroup, Tokenizer tokenizer, QwenModelArgs params)
      Tensor-parallel constructor.
      Parameters:
      name - model instance / checkpoint name.
      models - one decoder shard per TP rank.
      tpGroup - tensor-parallel group, or null when models.length == 1.
      tokenizer - chat / completion tokenizer.
      params - hyperparameters from the checkpoint.

    • Qwen

      public Qwen(String name, QwenModel[] models, TensorParallelGroup tpGroup, Tokenizer tokenizer, QwenModelArgs params, QwenVisionArgs visionArgs, QwenVlProcessor vlProcessor)
      Multimodal constructor.
      Parameters:
      name - model instance / checkpoint name.
      models - one decoder shard per TP rank.
      tpGroup - tensor-parallel group, or null.
      tokenizer - chat tokenizer.
      params - text hyperparameters.
      visionArgs - vision hyperparameters, or null.
      vlProcessor - multimodal processor, or null.

  • Method Details

    • isMultimodal

      public boolean isMultimodal()
      Returns whether a vision tower is loaded.
      Returns:
      true when a vision tower is loaded.

    • vlProcessor

      public QwenVlProcessor vlProcessor()
      Returns the VL processor.
      Returns:
      VL processor, or null for text-only.

    • visionArgs

      public QwenVisionArgs visionArgs()
      Returns the vision args.
      Returns:
      vision args, or null.

    • close

      public void close()
      Specified by:
      close in interface AutoCloseable

    • toString

      public String toString()
      Overrides:
      toString in class Object

    • family

      public String family()
      Description copied from interface: LanguageModel
      Returns the architecture family label.

      Examples: meta/llama3, alibaba/qwen3.5. This is not necessarily the public API model id used by serve.

      Specified by:
      family in interface LanguageModel
      Returns:
      the family label string.

    • setPrefixReplayEnabled

      public void setPrefixReplayEnabled(boolean enabled)
      Enables Phase-1 hybrid prefix replay: radix KV hits are allowed and warmPrefix(int, int[], int) rebuilds DeltaNet state over the matched prefix.
      Parameters:
      enabled - true to allow safe hybrid prefix reuse.

    • isPrefixReplayEnabled

      public boolean isPrefixReplayEnabled()
      Returns whether hybrid DeltaNet warm-prefix replay is enabled.
      Returns:
      whether hybrid DeltaNet warm-prefix replay is enabled.

    • setSpeculativeEnabled

      public void setSpeculativeEnabled(boolean enabled)
      Enables or disables native MTP speculative decoding when MTP weights are loaded.
      Parameters:
      enabled - true to use draft/verify (default when MTP is present).

    • isSpeculativeEnabled

      public boolean isSpeculativeEnabled()
      Returns whether MTP speculation is enabled.
      Returns:
      whether MTP speculation is enabled.

    • setNumSpeculativeTokens

      public void setNumSpeculativeTokens(int n)
      Sets draft depth override (<= 0 restores the HF/config default).
      Parameters:
      n - draft token count, capped at QwenModelArgs.MAX_SPECULATIVE_TOKENS.

    • numSpeculativeTokens

      public int numSpeculativeTokens()
      Returns the resolved draft depth when speculation is active.
      Returns:
      resolved draft depth when speculation is active, else 0.

    • speculativeAcceptRate

      public double speculativeAcceptRate()
      Draft-token accept rate over completed speculative rounds (accepted / proposed), or 0 when none yet.
      Returns:
      speculative accept rate.

    • speculativeMeanAcceptedDepth

      public double speculativeMeanAcceptedDepth()
      Mean accepted draft depth per speculative round (0 when none).
      Returns:
      mean accepted draft depth.

    • speculativeMeanTargetForwardsPerRound

      public double speculativeMeanTargetForwardsPerRound()
      Mean target verify forwards per speculative round (window verify → 1).
      Returns:
      average, or 0 when no rounds yet.

    • speculativeMeanRoundTimingMs

      public double[] speculativeMeanRoundTimingMs()
      Mean online-round wall-clock breakdown in milliseconds: {draftMs, verifyMs, replayMs, bookkeepingMs}, or all zero before any online round.
      Returns:
      mean per-round timing breakdown in milliseconds.

    • resetSpeculativeMetrics

      public void resetSpeculativeMetrics()
      Resets speculative accept-rate and round-timing counters.

    • setPrefixReuseEnabled

      public void setPrefixReuseEnabled(boolean enabled)
      Enables or disables radix prefix reuse on every TP rank's KV pool.
      Parameters:
      enabled - true to match/insert prefixes across requests.

    • name

      public String name()
      Description copied from interface: LanguageModel
      Returns the model instance / checkpoint name.
      Specified by:
      name in interface LanguageModel
      Returns:
      the instance name (e.g. directory or HF repo leaf name).

    • maxSeqLen

      public int maxSeqLen()
      Description copied from interface: LanguageModel
      Returns the configured maximum sequence length (max-model-len / context length).
      Specified by:
      maxSeqLen in interface LanguageModel
      Specified by:
      maxSeqLen in interface ModelExecutor
      Returns:
      the maximum number of tokens allowed for prompt plus completion.

    • encodeChat

      public int[] encodeChat(Message... dialog)
      Description copied from interface: LanguageModel
      Encodes a dialog with the model chat template, leaving the assistant turn open for completion.
      Specified by:
      encodeChat in interface LanguageModel
      Parameters:
      dialog - ordered conversation turns (system / user / assistant).
      Returns:
      prompt token ids ready for LanguageModel.generate(int[], int, double, double, boolean, long, GenerationListener).

    • encodeChat

      public int[] encodeChat(Message[] dialog, ChatOptions options)
      Description copied from interface: LanguageModel
      Encodes a dialog with optional tool-calling options.

      Default ignores options and delegates to LanguageModel.encodeChat(Message...).

      Specified by:
      encodeChat in interface LanguageModel
      Parameters:
      dialog - ordered conversation turns.
      options - chat options; may be null.
      Returns:
      prompt token ids.

    • processMultimodal

      public QwenVlProcessor.ProcessedMultimodal processMultimodal(Message... dialog) throws IOException
      Processes a multimodal dialog (images/video) into tokens + vision tensors.
      Parameters:
      dialog - chat turns.
      Returns:
      processed multimodal input.
      Throws:
      IOException - if media cannot be loaded.

    • processMultimodal

      public QwenVlProcessor.ProcessedMultimodal processMultimodal(Message[] dialog, ChatOptions options) throws IOException
      Preprocesses a multimodal dialog with chat options (thinking mode, reasoning effort).
      Parameters:
      dialog - the messages.
      options - chat options; may be null.
      Returns:
      the processed prompt and vision tensors.
      Throws:
      IOException - if media cannot be read.

    • params

      public QwenModelArgs params()
      Hyperparameters from the checkpoint.
      Returns:
      model args.

    • build

      public static Qwen build(String checkpointDir, int maxBatchSize, int maxSeqLen, byte deviceId) throws IOException
      Builds a Qwen instance from a HuggingFace checkpoint directory.
      Parameters:
      checkpointDir - directory containing config.json and weights.
      maxBatchSize - maximum batch size for inference.
      maxSeqLen - maximum sequence length; <= 0 uses the config value.
      deviceId - CUDA device id, or negative for CPU.
      Returns:
      a loaded Qwen model.
      Throws:
      IOException - if the checkpoint cannot be read.

    • build

      public static Qwen build(String checkpointDir, int maxBatchSize, int maxSeqLen, byte deviceId, double memFractionStatic, String kvCacheDtype) throws IOException
      Builds a Qwen instance from a HuggingFace checkpoint directory.
      Parameters:
      checkpointDir - directory containing config.json and weights.
      maxBatchSize - maximum batch size for inference.
      maxSeqLen - maximum sequence length; <= 0 uses the config value.
      deviceId - CUDA device id, or negative for CPU.
      memFractionStatic - static-region fraction of total GPU memory (SGLang-style); <=0 keeps test sizing.
      kvCacheDtype - optional KV dtype override.
      Returns:
      a loaded Qwen model.
      Throws:
      IOException - if the checkpoint cannot be read.

    • build

      public static Qwen build(String checkpointDir, int maxBatchSize, int maxSeqLen, byte deviceId, double memFractionStatic, String kvCacheDtype, ParallelConfig parallel) throws IOException
      Builds a Qwen instance with optional tensor parallelism.
      Parameters:
      checkpointDir - directory containing config.json and weights.
      maxBatchSize - maximum batch size for inference.
      maxSeqLen - maximum sequence length; <= 0 uses the config value.
      deviceId - CUDA device id, or negative for CPU.
      memFractionStatic - static-region fraction of total GPU memory (SGLang-style); <=0 keeps test sizing.
      kvCacheDtype - optional KV dtype override.
      parallel - ParallelConfig.tensorParallel(byte...) for multi-GPU; ppSize must be 1.
      Returns:
      a loaded Qwen model.
      Throws:
      IOException - if the checkpoint cannot be read.

    • build

      public static Qwen build(String checkpointDir, int maxBatchSize, int maxSeqLen, byte deviceId, double memFractionStatic, String kvCacheDtype, int pageSize, ParallelConfig parallel) throws IOException
      Builds a Qwen instance with optional tensor parallelism and KV page size.
      Parameters:
      checkpointDir - directory containing config.json and weights.
      maxBatchSize - maximum batch size for inference.
      maxSeqLen - maximum sequence length; <= 0 uses the config value.
      deviceId - CUDA device id, or negative for CPU.
      memFractionStatic - static-region fraction of total GPU memory (SGLang-style); <=0 keeps test sizing.
      kvCacheDtype - optional KV dtype override.
      pageSize - tokens per radix / KV pool page (>= 1).
      parallel - ParallelConfig.tensorParallel(byte...) for multi-GPU; ppSize must be 1.
      Returns:
      a loaded Qwen model.
      Throws:
      IOException - if the checkpoint cannot be read.

    • build

      public static Qwen build(String checkpointDir, int maxBatchSize, int maxSeqLen, byte deviceId, double memFractionStatic, String kvCacheDtype, int pageSize, ParallelConfig parallel, int modelLoaderThreads) throws IOException
      Builds a Qwen instance with optional tensor parallelism, KV page size, and safetensors loader concurrency.
      Parameters:
      checkpointDir - directory containing config.json and weights.
      maxBatchSize - maximum batch size for inference.
      maxSeqLen - maximum sequence length; <= 0 uses the config value.
      deviceId - CUDA device id, or negative for CPU.
      memFractionStatic - static-region fraction of total GPU memory (SGLang-style); <=0 keeps test sizing.
      kvCacheDtype - optional KV dtype override.
      pageSize - tokens per radix / KV pool page (>= 1).
      parallel - ParallelConfig.tensorParallel(byte...) for multi-GPU; ppSize must be 1.
      modelLoaderThreads - safetensors loader threads; 0 = auto (SafeTensorsLoaderThreads.resolve(int, int)).
      Returns:
      a loaded Qwen model.
      Throws:
      IOException - if the checkpoint cannot be read.

    • build

      public static Qwen build(String checkpointDir, int maxBatchSize, int maxSeqLen, byte deviceId, double memFractionStatic, String kvCacheDtype, int pageSize, ParallelConfig parallel, int modelLoaderThreads, int speculativeRows, int speculativeDepth) throws IOException
      Like the overload above, additionally reserving DeltaNet checkpoint memory for batched MTP speculation before the KV pool is sized, so the KV budget accounts for it and a large speculating cohort never has to fall back to plain decode for lack of memory.
      Parameters:
      checkpointDir - directory containing config.json and weights.
      maxBatchSize - maximum batch size for inference.
      maxSeqLen - maximum sequence length; <= 0 uses the config value.
      deviceId - CUDA device id, or negative for CPU.
      memFractionStatic - static-region fraction of total GPU memory (SGLang-style); <=0 keeps test sizing.
      kvCacheDtype - optional KV dtype override.
      pageSize - tokens per radix / KV pool page (>= 1).
      parallel - ParallelConfig.tensorParallel(byte...) for multi-GPU; ppSize must be 1.
      modelLoaderThreads - safetensors loader threads; 0 = auto (SafeTensorsLoaderThreads.resolve(int, int)).
      speculativeRows - rows (concurrently speculating requests) to provision; 0 reserves nothing.
      speculativeDepth - draft depth to provision for; <= 0 uses the model default.
      Returns:
      a loaded Qwen model.
      Throws:
      IOException - if the checkpoint cannot be read.

    • generate

      public ChatCompletion generate(int[] prompt, int maxGenLen, double temperature, double topp, boolean logprobs, long seed, GenerationListener listener, BooleanSupplier cancelRequested)
      Description copied from interface: LanguageModel
      Generates a completion from an already-tokenized prompt.
      Specified by:
      generate in interface LanguageModel
      Parameters:
      prompt - prompt token id sequence.
      maxGenLen - maximum number of new tokens to generate (not including the prompt itself).
      temperature - sampling temperature; higher values increase randomness.
      topp - nucleus-sampling top-p threshold in (0, 1].
      logprobs - true to include per-token log-probabilities in the result.
      seed - optional RNG seed for deterministic sampling; 0 means non-deterministic.
      listener - optional progress callback; may be null.
      cancelRequested - when non-null and returns true, generation stops between decode steps with CancellationException.
      Returns:
      the completion for prompt.

    • complete

      public ChatCompletion complete(String prompt, int maxGenLen, double temperature, double topp, boolean logprobs, long seed, GenerationListener listener)
      Performs text completion for a prompt.
      Parameters:
      prompt - text prompt.
      maxGenLen - maximum number of new tokens to generate.
      temperature - temperature value for controlling randomness in sampling.
      topp - top-p probability threshold for nucleus sampling.
      logprobs - flag indicating whether to compute token log probabilities.
      seed - optional RNG seed to sample deterministically.
      listener - optional generation progress callback.
      Returns:
      the generated text completion.

    • chat

      public ChatCompletion chat(Message[] dialog, int maxGenLen, double temperature, double topp, boolean logprobs, long seed, GenerationListener listener, BooleanSupplier cancelRequested)
      Description copied from interface: LanguageModel
      Generates an assistant reply for a dialog.

      Equivalent to LanguageModel.encodeChat(Message...) followed by LanguageModel.generate(int[], int, double, double, boolean, long, GenerationListener).

      Specified by:
      chat in interface LanguageModel
      Parameters:
      dialog - ordered conversation turns.
      maxGenLen - maximum number of new tokens to generate.
      temperature - sampling temperature; higher values increase randomness.
      topp - nucleus-sampling top-p threshold in (0, 1].
      logprobs - true to include per-token log-probabilities in the result.
      seed - optional RNG seed for deterministic sampling; 0 means non-deterministic.
      listener - optional progress callback; may be null.
      cancelRequested - cooperative cancel flag; may be null.
      Returns:
      the completion for dialog.

    • model

      public LanguageModel model()
      Description copied from interface: ModelExecutor
      Returns the underlying language-model façade.
      Specified by:
      model in interface ModelExecutor
      Returns:
      language model.

    • kvCachePool

      public KvCachePool kvCachePool()
      Description copied from interface: ModelExecutor
      Primary KV cache pool (rank 0 for TP). May be null only for models without full-attention layers.
      Specified by:
      kvCachePool in interface ModelExecutor
      Returns:
      KV cache pool, or null.

    • padToken

      public int padToken()
      Description copied from interface: ModelExecutor
      Returns the pad token id.
      Specified by:
      padToken in interface ModelExecutor
      Returns:
      pad token id.

    • stopTokens

      public int[] stopTokens()
      Description copied from interface: ModelExecutor
      Returns stop / EOS token ids.
      Specified by:
      stopTokens in interface ModelExecutor
      Returns:
      stop token ids.

    • decode

      public String decode(int[] tokens)
      Description copied from interface: ModelExecutor
      Decodes token ids to UTF-8 text.
      Specified by:
      decode in interface ModelExecutor
      Parameters:
      tokens - token ids.
      Returns:
      decoded text.

    • tryDecode

      public String tryDecode(int[] tokens, boolean skipSpecial) throws CharacterCodingException
      Description copied from interface: ModelExecutor
      Attempts a streaming decode; returns empty on incomplete UTF-8.
      Specified by:
      tryDecode in interface ModelExecutor
      Parameters:
      tokens - token ids.
      skipSpecial - whether to skip special tokens.
      Returns:
      decoded text, or empty on incomplete UTF-8.
      Throws:
      CharacterCodingException - when the sequence is incomplete.

    • bind

      public int bind(int[] prompt, int totalCapacity)
      Description copied from interface: ModelExecutor
      Binds one request into the KV pool (multi-request safe).
      Specified by:
      bind in interface ModelExecutor
      Parameters:
      prompt - prompt token ids.
      totalCapacity - desired slots (prompt + generation).
      Returns:
      request id (> 0).

    • prefixLen

      public int prefixLen(int requestId)
      Description copied from interface: ModelExecutor
      Page-aligned matched prefix length for a bound request.
      Specified by:
      prefixLen in interface ModelExecutor
      Parameters:
      requestId - id from ModelExecutor.bind(int[], int).
      Returns:
      prefix length (>= 0).

    • prefillMultimodal

      public Tensor prefillMultimodal(int requestId, MultimodalPrompt promptArg, int from, int to)
      Description copied from interface: ModelExecutor
      Prefills a multimodal prompt using vision embeds + interleaved mRoPE. Default throws; Qwen and MiniMax override.
      Specified by:
      prefillMultimodal in interface ModelExecutor
      Parameters:
      requestId - request id.
      promptArg - preprocess result.
      from - inclusive start (normally 0; prefix reuse disabled for VL).
      to - exclusive end.
      Returns:
      logits when to == inputIds.length, else null.

    • prefill

      public Tensor prefill(int requestId, int[] prompt, int prefixLen)
      Description copied from interface: ModelExecutor
      Prefills the unbound suffix of a prompt (activates this request alone).
      Specified by:
      prefill in interface ModelExecutor
      Parameters:
      requestId - request id.
      prompt - full prompt tokens.
      prefixLen - cached prefix to skip (from ModelExecutor.prefixLen(int)).
      Returns:
      last-position logits shaped [1, vocab] (caller owns).

    • warmPrefix

      public void warmPrefix(int requestId, int[] prompt, int prefixLen)
      Phase-1 hybrid prefix hit: run a full hybrid forward over prompt[0, prefixLen) so DeltaNet recurrent/conv state matches a cold prefill. Shared radix KV pages are rewritten with equivalent values.

      Skipping full-attention residuals here would be incorrect — DeltaNet inputs depend on the residual stream after attention blocks.

      Specified by:
      warmPrefix in interface ModelExecutor
      Parameters:
      requestId - request id from ModelExecutor.bind(int[], int).
      prompt - full prompt tokens.
      prefixLen - page-aligned matched prefix (>= 0).

    • prefillChunk

      public Tensor prefillChunk(int requestId, int[] prompt, int from, int to)
      Description copied from interface: ModelExecutor
      Prefills prompt[from, to) (chunked Continuous Batching). Returns last-token logits only when to == prompt.length.
      Specified by:
      prefillChunk in interface ModelExecutor
      Parameters:
      requestId - request id.
      prompt - full prompt tokens.
      from - inclusive start position in the prompt.
      to - exclusive end position.
      Returns:
      logits [1, vocab] when the prompt is fully prefilled; otherwise null.

    • speculateStep

      public int[][] speculateStep(int[] requestIds, int[] lastTokens, int[] positions, int numDrafts, double[] temperatures, double[] topps)
      Description copied from interface: ModelExecutor
      Same as ModelExecutor.speculateStep(int[], int[], int[], int, double, double) with per-request sampling parameters, so a cohort whose requests sample differently is still verified in one batched forward. The default delegates to the scalar overload when every request shares the same parameters and otherwise runs the requests one at a time.
      Specified by:
      speculateStep in interface ModelExecutor
      Parameters:
      requestIds - active KV request ids (order = batch).
      lastTokens - last committed token per request.
      positions - absolute position of lastTokens[i] (write of next is +1).
      numDrafts - draft depth (same for every request in this call).
      temperatures - per-request sampling temperature (verify).
      topps - per-request nucleus top-p (verify).
      Returns:
      accepted token ids per request (each length >= 1; caller owns arrays).

    • speculateStep

      public int[][] speculateStep(int[] requestIds, int[] lastTokens, int[] positions, int numDrafts, double temperature, double topp)
      Description copied from interface: ModelExecutor
      One speculative MTP round for a uniform draft depth.

      Default throws; Qwen overrides when MTP weights are loaded.

      Specified by:
      speculateStep in interface ModelExecutor
      Parameters:
      requestIds - active KV request ids (order = batch).
      lastTokens - last committed token per request.
      positions - absolute position of lastTokens[i] (write of next is +1).
      numDrafts - draft depth (same for every request in this call).
      temperature - sampling temperature (verify).
      topp - nucleus top-p (verify).
      Returns:
      accepted token ids per request (each length >= 1; caller owns arrays).

    • decodeStep

      public Tensor decodeStep(int[] requestIds, int[] lastTokens, int[] positions)
      Description copied from interface: ModelExecutor
      One decode step for an active set (same or mixed positions; engine may cohort by position). Tokens are [B] last tokens; positions are the write index for each row.
      Specified by:
      decodeStep in interface ModelExecutor
      Parameters:
      requestIds - active KV request ids (order = batch).
      lastTokens - token id per request.
      positions - absolute position per request.
      Returns:
      logits [B, vocab] (caller owns).

    • finish

      public void finish(int requestId, int[] sequenceTokens)
      Description copied from interface: ModelExecutor
      Inserts into the radix tree (when enabled) and unbinds the request.
      Specified by:
      finish in interface ModelExecutor
      Parameters:
      requestId - request id.
      sequenceTokens - prompt + completion (no pad).

    • evict

      public void evict(int requestId)
      Description copied from interface: ModelExecutor
      Instant Eviction without radix insert.
      Specified by:
      evict in interface ModelExecutor
      Parameters:
      requestId - request id.

    • idleDecodeGraphPrefetch

      public void idleDecodeGraphPrefetch()
      Description copied from interface: ModelExecutor
      Advances pending decode-graph prefetch when the scheduler is idle but KV remains bound (e.g. between continuous-batching waves).
      Specified by:
      idleDecodeGraphPrefetch in interface ModelExecutor