Package smile.llm.transformer

package smile.llm.transformer
A Transformer is a deep learning architecture, serves as the foundational engine for modern Large Language Models (LLMs).

1. Tokenization and Embeddings

Raw text is first broken down into "tokens" (often subwords), which are then translated into high-dimensional embedding vectors.

2. Positional Encoding

Because the transformer processes all tokens in parallel, it needs a way to know the sequence of the words. Positional encoding solves this by adding unique, position-dependent signals to token embeddings.

3. Self-Attention Mechanism

This is the heart of the Transformer, which dynamically weighs the importance of different elements in a sequence. By computing how every token relates to all other tokens, it assigns context-aware representations for every token.

4. Feed-Forward Networks and Stacking

After passing through self-attention layers, the vector representations go through a Feed-Forward Neural Network (MLP).

Multiple layers of these transformer blocks are stacked on top of each other, allowing the LLM to build a progressively deeper, more abstract understanding of the text. Depending on the task, Transformers are typically structured in one of three ways:

Encoder-Only
Models like BERT process input text to understand its meaning, intent, or sentiment. They are ideal for text classification and extractive search.
Decoder-Only
Models like GPT specialize in generating new text token by token based on previous tokens. They are the driving force behind generative AI.
Encoder-Decoder
The original architecture combined both; the encoder processes the input sequence (e.g., in English) and the decoder generates the output sequence (e.g., translating it into French).