Class QwenBlock

java.lang.Object
smile.llm.model.qwen.QwenBlock
public class QwenBlock extends Object
One hybrid decoder block: pre-norm → mixer (DeltaNet or gated attn) → residual → post-norm → SwiGLU FFN → residual.

  • Constructor Details

    • QwenBlock

      public QwenBlock(int layerId, QwenModelArgs args, DeltaNetStatePool statePool)
      Constructor.
      Parameters:
      layerId - stack layer index.
      args - model args.
      statePool - DeltaNet state pool (linear layers; may be null).

    • QwenBlock

      public QwenBlock(int layerId, QwenModelArgs args, DeltaNetStatePool statePool, TensorShardSpec shard, TensorParallelGroup tpGroup)
      Tensor-parallel constructor.
      Parameters:
      layerId - stack layer index.
      args - model args.
      statePool - DeltaNet state pool (linear layers; may be null).
      shard - local head / FFN shard description, or null for full width.
      tpGroup - tensor-parallel group, or null for single-device.

  • Method Details

    • layerId

      public int layerId()
      Returns the stack layer index.
      Returns:
      stack layer index.

    • layerType

      public String layerType()
      Returns the layer type string.
      Returns:
      full_attention or linear_attention.

    • selfAttn

      public GatedAttention selfAttn()
      Returns the gated full-attention mixer.
      Returns:
      gated full attention, or null for DeltaNet layers.

    • linearAttn

      public GatedDeltaNet linearAttn()
      Returns the DeltaNet mixer.
      Returns:
      DeltaNet mixer, or null for full-attention layers.

    • feedForward

      public FeedForward feedForward()
      Returns the SwiGLU feed-forward.
      Returns:
      SwiGLU feed-forward.

    • forward

      public Tensor forward(Tensor x, int startPos, Tensor cos, Tensor sin, Tensor mask)
      Forward pass.
      Parameters:
      x - hidden states.
      startPos - cache start position (full-attn).
      cos - partial RoPE cosines for this window (full-attn).
      sin - partial RoPE sines for this window (full-attn).
      mask - causal attention mask (full-attn).
      Returns:
      block output.

    • forward

      public Tensor forward(Tensor x, int[] positions, Tensor cos, Tensor sin, Tensor mask)
      Forward with per-row cache write positions (full-attn layers).
      Parameters:
      x - hidden states.
      positions - absolute write position per batch row.
      cos - partial RoPE cosines.
      sin - partial RoPE sines.
      mask - causal mask, or null.
      Returns:
      block output.