Package smile.llm.cache
package smile.llm.cache
In LLM serving, Radix KV cache management uses a Radix Tree (compressed
prefix tree) to organize and reuse Key-Value (KV) cache activations across
multiple requests. Instead of recomputing shared system prompts or
conversation histories, the inference engine finds the longest matching
prefix and only computes the diverging suffix, saving enormous amounts
of GPU memory and compute cycles.
Every node in the radix tree represents a specific token prefix. If 1,000 requests are all prepended with the same 512-token system prompt, the KV state for that prompt is computed only once and shared across all requests.
When conversation paths or agent workflows diverge, the tree branches. This allows the server to retain the shared history (e.g., turns 1 through 9) while only caching or computing the new, unique turn for each specific request.
Incoming prompts are matched against paths in the tree, allowing for instantaneous reuse of existing K and V values before the decoding phase even starts.
-
ClassDescriptionFlashInfer CSR page-table metadata for a bound KV request window.Result returned by
RadixCache.insert(int[], Tensor).Thrown when a request attempts to use more KV slots than were reserved at bind time.Family-agnostic layout used to size aKvCachePool.Physical KV-cache memory pool backed by aRadixCachefor prefix-sharing across requests.Result of SGLang-style static KV budget arithmetic (testable without CUDA).Result returned byRadixCache.matchPrefix(int[]).Radix tree-based KV cache for LLM inference serving.A node in the radix tree used for KV cache management.