Package smile.llm.model.qwen

package smile.llm.model.qwen

Alibaba Qwen3 model family. The Qwen3 family architecture uses a flexible design combining dense and sparse Mixture-of-Experts (MoE) layouts, a unified dual "thinking/non-thinking" mode, and hybrid linear-attention blocks (such as Gated DeltaNet mixed with full attention) scaling from 0.6B to over 2T parameters. Hybrid Attention Core: Integrates efficient linear attention (Gated DeltaNet) for most layers to eliminate explosive KV-cache memory growth, while strategically retaining full attention layers (e.g., a 3:1 ratio) to preserve broad context recall. Dense & Sparse MoE Options: Ships with both dense parameter setups and ultra-sparse Mixture-of-Experts variants where only a small active fraction of total parameters fire per token. Unified Thinking Modes: Built-in dynamic framework handles rapid conversational responses as well as deep, multi-step explicit reasoning without switching models. Native Multimodality: Early-fusion architectures designed natively from the base level to process cross-lingual text, vision (images/video), and audio tokens.

Class	Description
DeltaNetStatePool	Per-request DeltaNet recurrent and causal-conv state for linear-attention layers.
GatedAttention	Qwen3.5 gated full attention (GQA + partial RoPE + sigmoid output gate).
GatedDeltaNet	Qwen3.5 Gated DeltaNet linear-attention mixer (reference / torch path).
InterleavedMRope	Interleaved multimodal RoPE for Qwen3.8 / Qwen3.5 VL text towers.
InterleavedMRope.MropePositions	mRoPE position planes for one sequence.
MtpAnchorPool	Per-request MTP draft anchor (post-final-norm hidden feeding the MTP head's first draft step), keyed by request id.
PartialRotaryEncoding	Partial rotary positional encoding for Qwen3.5 gated attention.
PartialRotaryEncoding.CosSin	Precomputed cos/sin tables for HF-style partial RoPE.
Qwen	Qwen3.5 hybrid text model (Gated DeltaNet + gated full attention).
QwenBlock	One hybrid decoder block: pre-norm → mixer (DeltaNet or gated attn) → residual → post-norm → SwiGLU FFN → residual.
QwenModel	Qwen3.5 hybrid text model: embeddings, hybrid blocks, final norm, LM head.
QwenModelArgs	Qwen3.5 dense hybrid text-tower hyperparameters.
QwenMtp	Qwen3.5/3.8 native multi-token prediction (MTP) draft head.
QwenRMSNorm	Qwen3.5 RMSNorm: rms_norm(x) * (1 + weight) with weight initialized to zeros.
QwenRMSNormGated	Gated RMSNorm used inside Gated DeltaNet: weight * rms_norm(x) * silu(gate) with weight initialized to ones.
QwenVisionArgs	Qwen3.8 / Qwen3.5 native vision-tower hyperparameters plus multimodal token ids.
QwenVisionTower	Qwen3.8 / Qwen3.5 vision tower (model.visual.*).
QwenVlProcessor	HuggingFace-compatible image/video preprocessor for Qwen3.8 native VLM.
QwenVlProcessor.ProcessedMultimodal	Result of multimodal preprocessing for one prompt.
QwenWeightShard	Applies TP weight slices for Qwen module parameter names.
Tokenizer	Qwen3.5 chat tokenizer (byte-level BPE via Tiktoken).
VisionLayerNorm	Standard LayerNorm with learnable weight and bias (ViT / merger).