Package smile.llm.model.llama

package smile.llm.model.llama

Meta Llama 3 model family. Meta Llama 3 uses a standard decoder-only dense transformer architecture. Key design features include Grouped-Query Attention (GQA) for faster inference, a large 128k token vocabulary tokenizer, Rotary Position Embeddings (RoPE), and training scales spanning dense models from 1B up to a 405B parameter flagship. Trained with sequence lengths up to 8k tokens initially, scaling to 128k in later 3.1 updates, utilizing explicit masks to block cross-document attention.

Class	Description
GroupedQueryAttention	Grouped Query Attention (GQA).
Llama	LLaMA model specification.
LlamaBlock	A Llama dense-decoder block: grouped-query attention followed by a SwiGLU feed-forward network, each with a residual connection and RMSNorm.
LlamaModel	Llama dense decoder: token embeddings, stacked LlamaBlocks, final RMSNorm, and the output projection.
LlamaModelArgs	Llama / Meta dense-decoder hyperparameters.
Tokenizer	Custom tokenizer for Llama 3 models.