Interface ModelExecutor

All Known Implementing Classes:
Llama, Qwen
public interface ModelExecutor
Low-level execution surface used by InferenceEngine.

Continuous batching uses bind(int[], int)/prefill(int, int[], int)/decodeStep(int[], int[], int[]) instead of monolithic LanguageModel.generate(int[], int, double, double, boolean, long, GenerationListener). Offline callers may still use LanguageModel.generate(int[], int, double, double, boolean, long, GenerationListener).

  • Method Summary

    Modifier and Type
    Method
    Description
    int
    bind(int[] prompt, int totalCapacity)
    Binds one request into the KV pool (multi-request safe).
    String
    decode(int[] tokens)
    Decodes token ids to UTF-8 text.
    Tensor
    decodeStep(int[] requestIds, int[] lastTokens, int[] positions)
    One decode step for an active set (same or mixed positions; engine may cohort by position).
    void
    evict(int requestId)
    Instant Eviction without radix insert.
    void
    finish(int requestId, int[] sequenceTokens)
    Inserts into the radix tree (when enabled) and unbinds the request.
    default void
    idleDecodeGraphPrefetch()
    Advances pending decode-graph prefetch when the scheduler is idle but KV remains bound (e.g.
    KvCachePool
    kvCachePool()
    Primary KV cache pool (rank 0 for TP).
    default int
    maxSeqLen()
    Max sequence length for this model (prompt + completion).
    LanguageModel
    model()
    Returns the underlying language-model façade.
    int
    padToken()
    Returns the pad token id.
    Tensor
    prefill(int requestId, int[] prompt, int prefixLen)
    Prefills the unbound suffix of a prompt (activates this request alone).
    default Tensor
    prefillChunk(int requestId, int[] prompt, int from, int to)
    Prefills prompt[from, to) (chunked Continuous Batching).
    default Tensor
    prefillMultimodal(int requestId, QwenVlProcessor.ProcessedMultimodal multimodal, int from, int to)
    Prefills a multimodal prompt using vision embeds + interleaved mRoPE.
    int
    prefixLen(int requestId)
    Page-aligned matched prefix length for a bound request.
    default int[][]
    speculateStep(int[] requestIds, int[] lastTokens, int[] positions, int numDrafts, double[] temperatures, double[] topps)
    Same as speculateStep(int[], int[], int[], int, double, double) with per-request sampling parameters, so a cohort whose requests sample differently is still verified in one batched forward.
    default int[][]
    speculateStep(int[] requestIds, int[] lastTokens, int[] positions, int numDrafts, double temperature, double topp)
    One speculative MTP round for a uniform draft depth.
    int[]
    stopTokens()
    Returns stop / EOS token ids.
    default boolean
    supportsStepApi()
    When false, InferenceEngine uses serial LanguageModel.generate(int[], int, double, double, boolean, long, GenerationListener) (test stubs without tensors / KV).
    String
    tryDecode(int[] tokens, boolean skipSpecial)
    Attempts a streaming decode; returns empty on incomplete UTF-8.
    default void
    warmPrefix(int requestId, int[] prompt, int prefixLen)
    After a radix KV prefix hit of length prefixLen, restores any non-KV state required before suffix prefill may start at prefixLen.

  • Method Details

    • model

      LanguageModel model()
      Returns the underlying language-model façade.
      Returns:
      language model.

    • kvCachePool

      KvCachePool kvCachePool()
      Primary KV cache pool (rank 0 for TP). May be null only for models without full-attention layers.
      Returns:
      KV cache pool, or null.

    • padToken

      int padToken()
      Returns the pad token id.
      Returns:
      pad token id.

    • stopTokens

      int[] stopTokens()
      Returns stop / EOS token ids.
      Returns:
      stop token ids.

    • decode

      String decode(int[] tokens)
      Decodes token ids to UTF-8 text.
      Parameters:
      tokens - token ids.
      Returns:
      decoded text.

    • tryDecode

      String tryDecode(int[] tokens, boolean skipSpecial) throws CharacterCodingException
      Attempts a streaming decode; returns empty on incomplete UTF-8.
      Parameters:
      tokens - token ids.
      skipSpecial - whether to skip special tokens.
      Returns:
      decoded text, or empty on incomplete UTF-8.
      Throws:
      CharacterCodingException - when the sequence is incomplete.

    • supportsStepApi

      default boolean supportsStepApi()
      When false, InferenceEngine uses serial LanguageModel.generate(int[], int, double, double, boolean, long, GenerationListener) (test stubs without tensors / KV).
      Returns:
      true when bind/prefill/decodeStep are supported.

    • bind

      int bind(int[] prompt, int totalCapacity)
      Binds one request into the KV pool (multi-request safe).
      Parameters:
      prompt - prompt token ids.
      totalCapacity - desired slots (prompt + generation).
      Returns:
      request id (> 0).

    • prefixLen

      int prefixLen(int requestId)
      Page-aligned matched prefix length for a bound request.
      Parameters:
      requestId - id from bind(int[], int).
      Returns:
      prefix length (>= 0).

    • prefill

      Tensor prefill(int requestId, int[] prompt, int prefixLen)
      Prefills the unbound suffix of a prompt (activates this request alone).
      Parameters:
      requestId - request id.
      prompt - full prompt tokens.
      prefixLen - cached prefix to skip (from prefixLen(int)).
      Returns:
      last-position logits shaped [1, vocab] (caller owns).

    • prefillChunk

      default Tensor prefillChunk(int requestId, int[] prompt, int from, int to)
      Prefills prompt[from, to) (chunked Continuous Batching). Returns last-token logits only when to == prompt.length.
      Parameters:
      requestId - request id.
      prompt - full prompt tokens.
      from - inclusive start position in the prompt.
      to - exclusive end position.
      Returns:
      logits [1, vocab] when the prompt is fully prefilled; otherwise null.

    • warmPrefix

      default void warmPrefix(int requestId, int[] prompt, int prefixLen)
      After a radix KV prefix hit of length prefixLen, restores any non-KV state required before suffix prefill may start at prefixLen.

      Default is a no-op (pure Transformer KV reuse). Hybrid models that keep DeltaNet (or similar) state must replay or restore that state here when prefix reuse is enabled.

      Parameters:
      requestId - request id from bind(int[], int).
      prompt - full prompt tokens.
      prefixLen - page-aligned matched prefix (>= 0).

    • prefillMultimodal

      default Tensor prefillMultimodal(int requestId, QwenVlProcessor.ProcessedMultimodal multimodal, int from, int to)
      Prefills a multimodal prompt using vision embeds + interleaved mRoPE. Default throws; Qwen overrides.
      Parameters:
      requestId - request id.
      multimodal - preprocess result.
      from - inclusive start (normally 0; prefix reuse disabled for VL).
      to - exclusive end.
      Returns:
      logits when to == inputIds.length, else null.

    • decodeStep

      Tensor decodeStep(int[] requestIds, int[] lastTokens, int[] positions)
      One decode step for an active set (same or mixed positions; engine may cohort by position). Tokens are [B] last tokens; positions are the write index for each row.
      Parameters:
      requestIds - active KV request ids (order = batch).
      lastTokens - token id per request.
      positions - absolute position per request.
      Returns:
      logits [B, vocab] (caller owns).

    • speculateStep

      default int[][] speculateStep(int[] requestIds, int[] lastTokens, int[] positions, int numDrafts, double temperature, double topp)
      One speculative MTP round for a uniform draft depth.

      Default throws; Qwen overrides when MTP weights are loaded.

      Parameters:
      requestIds - active KV request ids (order = batch).
      lastTokens - last committed token per request.
      positions - absolute position of lastTokens[i] (write of next is +1).
      numDrafts - draft depth (same for every request in this call).
      temperature - sampling temperature (verify).
      topp - nucleus top-p (verify).
      Returns:
      accepted token ids per request (each length >= 1; caller owns arrays).

    • speculateStep

      default int[][] speculateStep(int[] requestIds, int[] lastTokens, int[] positions, int numDrafts, double[] temperatures, double[] topps)
      Same as speculateStep(int[], int[], int[], int, double, double) with per-request sampling parameters, so a cohort whose requests sample differently is still verified in one batched forward. The default delegates to the scalar overload when every request shares the same parameters and otherwise runs the requests one at a time.
      Parameters:
      requestIds - active KV request ids (order = batch).
      lastTokens - last committed token per request.
      positions - absolute position of lastTokens[i] (write of next is +1).
      numDrafts - draft depth (same for every request in this call).
      temperatures - per-request sampling temperature (verify).
      topps - per-request nucleus top-p (verify).
      Returns:
      accepted token ids per request (each length >= 1; caller owns arrays).

    • idleDecodeGraphPrefetch

      default void idleDecodeGraphPrefetch()
      Advances pending decode-graph prefetch when the scheduler is idle but KV remains bound (e.g. between continuous-batching waves).

    • finish

      void finish(int requestId, int[] sequenceTokens)
      Inserts into the radix tree (when enabled) and unbinds the request.
      Parameters:
      requestId - request id.
      sequenceTokens - prompt + completion (no pad).

    • evict

      void evict(int requestId)
      Instant Eviction without radix insert.
      Parameters:
      requestId - request id.

    • maxSeqLen

      default int maxSeqLen()
      Max sequence length for this model (prompt + completion).
      Returns:
      max seq len.