Record Class KvCacheLayout

java.lang.Object
java.lang.Record
smile.llm.cache.KvCacheLayout
Record Components:
numLayers - number of transformer layers that store KV in the pool.
numKvHeads - number of key/value heads.
headDim - dimension of each attention head.
maxBatchSize - maximum concurrent batch size (used for CPU / test sizing).
maxSeqLen - maximum sequence length (used for CPU / test sizing and minima).
public record KvCacheLayout(int numLayers, int numKvHeads, int headDim, int maxBatchSize, int maxSeqLen) extends Record
Family-agnostic layout used to size a KvCachePool.

  • Constructor Summary

    Constructors
    Constructor
    Description
    KvCacheLayout(int numLayers, int numKvHeads, int headDim, int maxBatchSize, int maxSeqLen)
    Creates an instance of a KvCacheLayout record class.

  • Method Summary

    Modifier and Type
    Method
    Description
    final boolean
    equals(Object o)
    Indicates whether some other object is "equal to" this one.
    final int
    hashCode()
    Returns a hash code value for this object.
    int
    headDim()
    Returns the value of the headDim record component.
    int
    maxBatchSize()
    Returns the value of the maxBatchSize record component.
    int
    maxSeqLen()
    Returns the value of the maxSeqLen record component.
    int
    numKvHeads()
    Returns the value of the numKvHeads record component.
    int
    numLayers()
    Returns the value of the numLayers record component.
    static KvCacheLayout
    of(int numLayers, int dim, int numHeads, Integer numKvHeads, int maxBatchSize, int maxSeqLen)
    Builds a layout from common dense-decoder dimensions.
    final String
    toString()
    Returns a string representation of this record class.

    Methods inherited from class Object

    clone, finalize, getClass, notify, notifyAll, wait, wait, wait

  • Constructor Details

    • KvCacheLayout

      public KvCacheLayout(int numLayers, int numKvHeads, int headDim, int maxBatchSize, int maxSeqLen)
      Creates an instance of a KvCacheLayout record class.
      Parameters:
      numLayers - the value for the numLayers record component
      numKvHeads - the value for the numKvHeads record component
      headDim - the value for the headDim record component
      maxBatchSize - the value for the maxBatchSize record component
      maxSeqLen - the value for the maxSeqLen record component

  • Method Details

    • of

      public static KvCacheLayout of(int numLayers, int dim, int numHeads, Integer numKvHeads, int maxBatchSize, int maxSeqLen)
      Builds a layout from common dense-decoder dimensions.
      Parameters:
      numLayers - layer count.
      dim - token embedding dimension.
      numHeads - query head count.
      numKvHeads - key/value head count, or null to use numHeads.
      maxBatchSize - max batch size.
      maxSeqLen - max sequence length.
      Returns:
      the cache layout.

    • toString

      public final String toString()
      Returns a string representation of this record class. The representation contains the name of the class, followed by the name and value of each of the record components.
      Specified by:
      toString in class Record
      Returns:
      a string representation of this object

    • hashCode

      public final int hashCode()
      Returns a hash code value for this object. The value is derived from the hash code of each of the record components.
      Specified by:
      hashCode in class Record
      Returns:
      a hash code value for this object

    • equals

      public final boolean equals(Object o)
      Indicates whether some other object is "equal to" this one. The objects are equal if the other object is of the same class and if all the record components are equal. All components in this record class are compared with the compare method from their corresponding wrapper classes.
      Specified by:
      equals in class Record
      Parameters:
      o - the object with which to compare
      Returns:
      true if this object is the same as the o argument; false otherwise.

    • numLayers

      public int numLayers()
      Returns the value of the numLayers record component.
      Returns:
      the value of the numLayers record component

    • numKvHeads

      public int numKvHeads()
      Returns the value of the numKvHeads record component.
      Returns:
      the value of the numKvHeads record component

    • headDim

      public int headDim()
      Returns the value of the headDim record component.
      Returns:
      the value of the headDim record component

    • maxBatchSize

      public int maxBatchSize()
      Returns the value of the maxBatchSize record component.
      Returns:
      the value of the maxBatchSize record component

    • maxSeqLen

      public int maxSeqLen()
      Returns the value of the maxSeqLen record component.
      Returns:
      the value of the maxSeqLen record component