Record Class QwenModelArgs

java.lang.Object

java.lang.Record

smile.llm.model.qwen.QwenModelArgs

Record Components:

dim - token embedding / hidden size.

numLayers - number of hybrid decoder blocks.

numHeads - number of query attention heads (full-attn layers).

numKvHeads - number of key/value heads (full-attn layers).

headDim - attention head dimension (may differ from dim/numHeads).

vocabSize - vocabulary size.

intermediateSize - SwiGLU FFN hidden size.

normEps - RMSNorm epsilon.

ropeTheta - RoPE base theta.

partialRotaryFactor - fraction of headDim rotated by RoPE.

linearConvKernelDim - Gated DeltaNet causal conv kernel size.

linearKeyHeadDim - DeltaNet key head dimension.

linearValueHeadDim - DeltaNet value head dimension.

linearNumKeyHeads - DeltaNet key head count.

linearNumValueHeads - DeltaNet value head count.

layerTypes - per-layer mixer type (linear_attention or full_attention).

maxBatchSize - maximum inference batch size.

maxSeqLen - maximum sequence length.

mtpNumHiddenLayers - native MTP draft layers (0 = none).

defaultNumSpeculativeTokens - default draft depth when speculation is enabled.

public record QwenModelArgs(int dim, int numLayers, int numHeads, int numKvHeads, int headDim, int vocabSize, int intermediateSize, double normEps, double ropeTheta, double partialRotaryFactor, int linearConvKernelDim, int linearKeyHeadDim, int linearValueHeadDim, int linearNumKeyHeads, int linearNumValueHeads, String[] layerTypes, int maxBatchSize, int maxSeqLen, int mtpNumHiddenLayers, int defaultNumSpeculativeTokens)
extends Record

Qwen3.5 dense hybrid text-tower hyperparameters.

Field Summary

Fields

Modifier and Type	Field	Description
static final String	FULL_ATTENTION	Layer type for gated full softmax attention.
static final String	LINEAR_ATTENTION	Layer type for Gated DeltaNet (linear attention).
static final int	MAX_SPECULATIVE_TOKENS	Maximum draft tokens per speculative round (block size 8).

Constructor Summary

Constructors

Constructor	Description
QwenModelArgs()	Tiny defaults suitable for unit tests.
QwenModelArgs(int dim, int numLayers, int numHeads, int numKvHeads, int headDim, int vocabSize, int intermediateSize, double normEps, double ropeTheta, double partialRotaryFactor, int linearConvKernelDim, int linearKeyHeadDim, int linearValueHeadDim, int linearNumKeyHeads, int linearNumValueHeads, String[] layerTypes, int maxBatchSize, int maxSeqLen)	Constructor without MTP fields (mtpNumHiddenLayers = 0).
QwenModelArgs(int dim, int numLayers, int numHeads, int numKvHeads, int headDim, int vocabSize, int intermediateSize, double normEps, double ropeTheta, double partialRotaryFactor, int linearConvKernelDim, int linearKeyHeadDim, int linearValueHeadDim, int linearNumKeyHeads, int linearNumValueHeads, String[] layerTypes, int maxBatchSize, int maxSeqLen, int mtpNumHiddenLayers, int defaultNumSpeculativeTokens)	Creates an instance of a QwenModelArgs record class.

Method Summary

Modifier and Type	Method	Description
static String[]	defaultLayerTypes(int numLayers, int interval)	Builds the default 3:1 hybrid layer type pattern.
int	defaultNumSpeculativeTokens()	Returns the value of the defaultNumSpeculativeTokens record component.
int	dim()	Returns the value of the dim record component.
final boolean	equals(Object o)	Indicates whether some other object is "equal to" this one.
static QwenModelArgs	fromHuggingFace(String path, int maxBatchSize, int maxSeqLen)	Loads hyperparameters from a HuggingFace config.json.
static QwenModelArgs	fromTextConfig(tools.jackson.databind.JsonNode text, int maxBatchSize, int maxSeqLen)	Parses a text-config JSON object into model args.
int	fullAttentionLayerIndex(int stackLayerId)	Maps a stack layer id to the KV-cache pool layer index, or -1 when the stack layer is linear attention.
final int	hashCode()	Returns a hash code value for this object.
boolean	hasMtp()	Returns whether a native MTP head is configured.
int	headDim()	Returns the value of the headDim record component.
int	intermediateSize()	Returns the value of the intermediateSize record component.
KvCacheLayout	kvCacheLayout()	Returns a KV cache layout covering only full-attention layers.
KvCacheLayout	kvCacheLayout(TensorShardSpec shard)	KV layout for a tensor-parallel shard (local KV head count).
String[]	layerTypes()	Returns the value of the layerTypes record component.
int	linearAttentionLayerIndex(int stackLayerId)	Maps a stack layer id to the DeltaNet state-pool layer index, or -1 when the stack layer is full attention.
int	linearConvDim()	DeltaNet fused QKV channel count (2*key_dim + value_dim).
int	linearConvDim(TensorShardSpec shard)	Local DeltaNet conv channel count for a TP shard.
int	linearConvKernelDim()	Returns the value of the linearConvKernelDim record component.
int	linearKeyHeadDim()	Returns the value of the linearKeyHeadDim record component.
int	linearNumKeyHeads()	Returns the value of the linearNumKeyHeads record component.
int	linearNumValueHeads()	Returns the value of the linearNumValueHeads record component.
int	linearValueHeadDim()	Returns the value of the linearValueHeadDim record component.
int	maxBatchSize()	Returns the value of the maxBatchSize record component.
int	maxSeqLen()	Returns the value of the maxSeqLen record component.
int	mtpNumHiddenLayers()	Returns the value of the mtpNumHiddenLayers record component.
double	normEps()	Returns the value of the normEps record component.
int	numFullAttentionLayers()	Returns the number of full-attention layers.
int	numHeads()	Returns the value of the numHeads record component.
int	numKvHeads()	Returns the value of the numKvHeads record component.
int	numLayers()	Returns the value of the numLayers record component.
int	numLinearAttentionLayers()	Returns the number of linear-attention (DeltaNet) layers.
double	partialRotaryFactor()	Returns the value of the partialRotaryFactor record component.
int	resolveNumSpeculativeTokens(int requested)	Resolves draft depth for a request, capped at MAX_SPECULATIVE_TOKENS.
double	ropeTheta()	Returns the value of the ropeTheta record component.
int	rotaryDim()	Returns the rotary dimension round(headDim * partialRotaryFactor).
final String	toString()	Returns a string representation of this record class.
int	vocabSize()	Returns the value of the vocabSize record component.

Methods inherited from class Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Details

LINEAR_ATTENTION

public static final String LINEAR_ATTENTION

Layer type for Gated DeltaNet (linear attention).

See Also:

• Constant Field Values

FULL_ATTENTION

public static final String FULL_ATTENTION

Layer type for gated full softmax attention.

See Also:

• Constant Field Values

MAX_SPECULATIVE_TOKENS

public static final int MAX_SPECULATIVE_TOKENS

Maximum draft tokens per speculative round (block size 8).

See Also:

• Constant Field Values

Constructor Details

QwenModelArgs

public QwenModelArgs()

Tiny defaults suitable for unit tests.

QwenModelArgs

public QwenModelArgs(int dim,
 int numLayers,
 int numHeads,
 int numKvHeads,
 int headDim,
 int vocabSize,
 int intermediateSize,
 double normEps,
 double ropeTheta,
 double partialRotaryFactor,
 int linearConvKernelDim,
 int linearKeyHeadDim,
 int linearValueHeadDim,
 int linearNumKeyHeads,
 int linearNumValueHeads,
 String[] layerTypes,
 int maxBatchSize,
 int maxSeqLen)

Constructor without MTP fields (mtpNumHiddenLayers = 0).

Parameters:

dim - model hidden size.

numLayers - decoder layer count.

numHeads - query head count.

numKvHeads - key/value head count.

headDim - per-head dimension.

vocabSize - vocabulary size.

intermediateSize - FFN intermediate size.

normEps - RMSNorm epsilon.

ropeTheta - RoPE base theta.

partialRotaryFactor - partial RoPE fraction.

linearConvKernelDim - DeltaNet conv kernel size.

linearKeyHeadDim - DeltaNet key head dim.

linearValueHeadDim - DeltaNet value head dim.

linearNumKeyHeads - DeltaNet key head count.

linearNumValueHeads - DeltaNet value head count.

layerTypes - per-layer attention type strings.

maxBatchSize - max concurrent batch size.

maxSeqLen - max sequence length.

QwenModelArgs

public QwenModelArgs(int dim,
 int numLayers,
 int numHeads,
 int numKvHeads,
 int headDim,
 int vocabSize,
 int intermediateSize,
 double normEps,
 double ropeTheta,
 double partialRotaryFactor,
 int linearConvKernelDim,
 int linearKeyHeadDim,
 int linearValueHeadDim,
 int linearNumKeyHeads,
 int linearNumValueHeads,
 String[] layerTypes,
 int maxBatchSize,
 int maxSeqLen,
 int mtpNumHiddenLayers,
 int defaultNumSpeculativeTokens)

Creates an instance of a QwenModelArgs record class.

Parameters:

dim - the value for the dim record component

numLayers - the value for the numLayers record component

numHeads - the value for the numHeads record component

numKvHeads - the value for the numKvHeads record component

headDim - the value for the headDim record component

vocabSize - the value for the vocabSize record component

intermediateSize - the value for the intermediateSize record component

normEps - the value for the normEps record component

ropeTheta - the value for the ropeTheta record component

partialRotaryFactor - the value for the partialRotaryFactor record component

linearConvKernelDim - the value for the linearConvKernelDim record component

linearKeyHeadDim - the value for the linearKeyHeadDim record component

linearValueHeadDim - the value for the linearValueHeadDim record component

linearNumKeyHeads - the value for the linearNumKeyHeads record component

linearNumValueHeads - the value for the linearNumValueHeads record component

layerTypes - the value for the layerTypes record component

maxBatchSize - the value for the maxBatchSize record component

maxSeqLen - the value for the maxSeqLen record component

mtpNumHiddenLayers - the value for the mtpNumHiddenLayers record component

defaultNumSpeculativeTokens - the value for the defaultNumSpeculativeTokens record component

Method Details

hasMtp

public boolean hasMtp()

Returns whether a native MTP head is configured.

Returns:

true when a native MTP head is configured.

resolveNumSpeculativeTokens

public int resolveNumSpeculativeTokens(int requested)

Resolves draft depth for a request, capped at MAX_SPECULATIVE_TOKENS.

Parameters:

requested - <= 0 uses defaultNumSpeculativeTokens().

Returns:

draft count in [0, MAX_SPECULATIVE_TOKENS].

rotaryDim

public int rotaryDim()

Returns the rotary dimension round(headDim * partialRotaryFactor).

Returns:

rotary dim (must be even for complex RoPE).

numFullAttentionLayers

public int numFullAttentionLayers()

Returns the number of full-attention layers.

Returns:

full-attention layer count.

numLinearAttentionLayers

public int numLinearAttentionLayers()

Returns the number of linear-attention (DeltaNet) layers.

Returns:

linear-attention layer count.

fullAttentionLayerIndex

public int fullAttentionLayerIndex(int stackLayerId)

Maps a stack layer id to the KV-cache pool layer index, or -1 when the stack layer is linear attention.

Parameters:

stackLayerId - zero-based decoder layer index.

Returns:

full-attention ordinal, or -1.

linearAttentionLayerIndex

public int linearAttentionLayerIndex(int stackLayerId)

Maps a stack layer id to the DeltaNet state-pool layer index, or -1 when the stack layer is full attention.

Parameters:

stackLayerId - zero-based decoder layer index.

Returns:

linear-attention ordinal, or -1.

kvCacheLayout

public KvCacheLayout kvCacheLayout()

Returns a KV cache layout covering only full-attention layers.

Returns:

cache layout for KvCachePool.

kvCacheLayout

public KvCacheLayout kvCacheLayout(TensorShardSpec shard)

KV layout for a tensor-parallel shard (local KV head count).

Parameters:

shard - TP shard description, or null for the full layout.

Returns:

cache layout for KvCachePool.

linearConvDim

public int linearConvDim(TensorShardSpec shard)

Local DeltaNet conv channel count for a TP shard.

Parameters:

shard - TP shard description, or null for the full width.

Returns:

conv channel count.

linearConvDim

public int linearConvDim()

DeltaNet fused QKV channel count (2*key_dim + value_dim).

Returns:

conv channel count.

fromHuggingFace

public static QwenModelArgs fromHuggingFace(String path,
 int maxBatchSize,
 int maxSeqLen)
                                     throws IOException

Loads hyperparameters from a HuggingFace config.json. Accepts either a top-level text config (qwen3_5_text) or a multimodal wrapper with nested text_config.

Parameters:

path - path to config.json.

maxBatchSize - maximum batch size.

maxSeqLen - maximum sequence length; <= 0 derives from max_position_embeddings in the config.

Returns:

model args.

Throws:

IOException - if the file cannot be read.

fromTextConfig

public static QwenModelArgs fromTextConfig(tools.jackson.databind.JsonNode text,
 int maxBatchSize,
 int maxSeqLen)

Parses a text-config JSON object into model args.

Parameters:

text - text config node.

maxBatchSize - maximum batch size.

maxSeqLen - maximum sequence length; <= 0 derives from max_position_embeddings in the config.

Returns:

model args.

defaultLayerTypes

public static String[] defaultLayerTypes(int numLayers,
 int interval)

Builds the default 3:1 hybrid layer type pattern.

Parameters:

numLayers - number of layers.

interval - full-attention interval (every N-th layer is full).

Returns:

layer type array.

toString

public final String toString()

Returns a string representation of this record class. The representation contains the name of the class, followed by the name and value of each of the record components.

Specified by:

toString in class Record

Returns:

a string representation of this object

hashCode

public final int hashCode()

Returns a hash code value for this object. The value is derived from the hash code of each of the record components.

Specified by:

hashCode in class Record

Returns:

a hash code value for this object

equals

public final boolean equals(Object o)

Indicates whether some other object is "equal to" this one. The objects are equal if the other object is of the same class and if all the record components are equal. Reference components are compared with Objects::equals(Object,Object); primitive components are compared with the compare method from their corresponding wrapper classes.

Specified by:

equals in class Record

Parameters:

o - the object with which to compare

Returns:

true if this object is the same as the o argument; false otherwise.

dim

public int dim()

Returns the value of the dim record component.

Returns:

the value of the dim record component

numLayers

public int numLayers()

Returns the value of the numLayers record component.

Returns:

the value of the numLayers record component

numHeads

public int numHeads()

Returns the value of the numHeads record component.

Returns:

the value of the numHeads record component

numKvHeads

public int numKvHeads()

Returns the value of the numKvHeads record component.

Returns:

the value of the numKvHeads record component

headDim

public int headDim()

Returns the value of the headDim record component.

Returns:

the value of the headDim record component

vocabSize

public int vocabSize()

Returns the value of the vocabSize record component.

Returns:

the value of the vocabSize record component

intermediateSize

public int intermediateSize()

Returns the value of the intermediateSize record component.

Returns:

the value of the intermediateSize record component

normEps

public double normEps()

Returns the value of the normEps record component.

Returns:

the value of the normEps record component

ropeTheta

public double ropeTheta()

Returns the value of the ropeTheta record component.

Returns:

the value of the ropeTheta record component

partialRotaryFactor

public double partialRotaryFactor()

Returns the value of the partialRotaryFactor record component.

Returns:

the value of the partialRotaryFactor record component

linearConvKernelDim

public int linearConvKernelDim()

Returns the value of the linearConvKernelDim record component.

Returns:

the value of the linearConvKernelDim record component

linearKeyHeadDim

public int linearKeyHeadDim()

Returns the value of the linearKeyHeadDim record component.

Returns:

the value of the linearKeyHeadDim record component

linearValueHeadDim

public int linearValueHeadDim()

Returns the value of the linearValueHeadDim record component.

Returns:

the value of the linearValueHeadDim record component

linearNumKeyHeads

public int linearNumKeyHeads()

Returns the value of the linearNumKeyHeads record component.

Returns:

the value of the linearNumKeyHeads record component

linearNumValueHeads

public int linearNumValueHeads()

Returns the value of the linearNumValueHeads record component.

Returns:

the value of the linearNumValueHeads record component

layerTypes

public String[] layerTypes()

Returns the value of the layerTypes record component.

Returns:

the value of the layerTypes record component

maxBatchSize

public int maxBatchSize()

Returns the value of the maxBatchSize record component.

Returns:

the value of the maxBatchSize record component

maxSeqLen

public int maxSeqLen()

Returns the value of the maxSeqLen record component.

Returns:

the value of the maxSeqLen record component

mtpNumHiddenLayers

public int mtpNumHiddenLayers()

Returns the value of the mtpNumHiddenLayers record component.

Returns:

the value of the mtpNumHiddenLayers record component

defaultNumSpeculativeTokens

public int defaultNumSpeculativeTokens()

Returns the value of the defaultNumSpeculativeTokens record component.

Returns:

the value of the defaultNumSpeculativeTokens record component