Package smile.llm

package smile.llm

Large language models.

Related Packages

Package	Description
smile.llm.attention	Multi-head and paged attention kernels and backends for LLM inference.
smile.llm.cache	In LLM serving, Radix KV cache management uses a Radix Tree (compressed prefix tree) to organize and reuse Key-Value (KV) cache activations across multiple requests.
smile.llm.checkpoint	Model checkpoint and Safetensors shard loading utilities.
smile.llm.cluster
smile.llm.engine	Continuous-batching LLM inference engine, speculative decoding, sampling, and CUDA graph execution.
smile.llm.parallel	Tensor / pipeline / data-parallel primitives for LLM inference.
smile.llm.quant	Weight quantization backends: FP8 / NVFP4 (primary) and Marlin (Ampere failover).
smile.llm.template	Chat templates for encoding conversation dialogs and tool definitions into prompts.
smile.llm.tokenizer	LLM Tokenization.
smile.llm.tool	Tool-call parsers and post-processors for structured function calling in LLM responses.
smile.llm.transformer	A Transformer is a deep learning architecture, serves as the foundational engine for modern Large Language Models (LLMs).

Class	Description
AudioUrlPart	Audio content referencing an HTTP(S) or data: URL.
ChatCompletion	Chat completion.
ChatOptions	Per-request chat options for tool calling.
ContentPart	One piece of multimodal chat content (OpenAI-style).
FinishReason	The reasons that the chat completions finish.
FunctionCall	Function invocation payload inside a ToolCall.
FunctionDefinition	OpenAI-style function definition inside a ToolDefinition.
GenerationListener	Callback for autoregressive generation progress on a single request.
GenerationListeners	Helpers for composing and adapting GenerationListeners.
ImageUrlPart	Image content referencing an HTTP(S) or data: URL.
LanguageModel	Common façade for chat-capable decoder LLMs (Llama, Qwen, …).
Message	Dialog message with OpenAI-style multimodal content parts and optional tool fields.
Role	The role of message speaker in a dialog.
TextPart	Plain-text content part.
ToolCall	OpenAI-compatible tool call emitted by the assistant.
ToolChoice	OpenAI tool_choice semantics.
ToolChoice.Auto	Model may call tools or answer normally.
ToolChoice.Named	Force a specific function by name.
ToolChoice.None	Do not inject tools; disable tool-call parsing.
ToolChoice.Required	Model should call at least one tool.
ToolDefinition	OpenAI-compatible tool definition (type: "function").
VideoUrlPart	Video content referencing an HTTP(S) or file URL.