Record Class MiniMaxModelArgs

java.lang.Object
java.lang.Record
smile.llm.model.minimax.MiniMaxModelArgs
Record Components:
dim - token embedding / hidden size.
numLayers - number of decoder blocks.
numHeads - number of query heads.
numKvHeads - number of key/value heads.
headDim - attention head dimension.
vocabSize - vocabulary size.
normEps - RMSNorm epsilon (Gemma-style 1 + weight scale).
ropeTheta - RoPE base theta.
rotaryDim - number of rotated head dimensions (partial RoPE).
denseIntermediateSize - FFN width of the dense (non-MoE) layers.
expertIntermediateSize - FFN width of one routed expert.
sharedIntermediateSize - FFN width of the shared expert.
numExperts - routed experts per MoE layer.
numExpertsPerTok - experts selected per token.
numSharedExperts - shared experts per MoE layer.
routedScalingFactor - multiplier applied to the renormalized routing weights.
swigluAlpha - swigluoai sigmoid slope.
swigluLimit - swigluoai clamp.
moeLayers - per-layer flag: true for MoE, false for dense FFN.
sparseLayers - per-layer flag: true for MSA, false for dense GQA.
indexDim - MSA index head dimension.
numIndexHeads - MSA index-query heads (equals numKvHeads; one per GQA group).
topkBlocks - MSA blocks selected per query and index head.
blockSize - MSA block size in tokens (equals the KV page size).
initBlocks - leading blocks always selected.
localBlocks - most recent blocks always selected.
numMtpModules - native multi-token-prediction modules (0 = none).
maxBatchSize - maximum inference batch size.
maxSeqLen - maximum sequence length.
weightMode - how linear weights are stored and multiplied (chosen by the loader, not read from config.json).
public record MiniMaxModelArgs(int dim, int numLayers, int numHeads, int numKvHeads, int headDim, int vocabSize, double normEps, double ropeTheta, int rotaryDim, int denseIntermediateSize, int expertIntermediateSize, int sharedIntermediateSize, int numExperts, int numExpertsPerTok, int numSharedExperts, double routedScalingFactor, double swigluAlpha, double swigluLimit, boolean[] moeLayers, boolean[] sparseLayers, int indexDim, int numIndexHeads, int topkBlocks, int blockSize, int initBlocks, int localBlocks, int numMtpModules, int maxBatchSize, int maxSeqLen, MiniMaxWeightMode weightMode) extends Record
MiniMax-M3 text-tower hyperparameters: GQA with MiniMax Sparse Attention (MSA) and a sigmoid-routed mixture of experts.

  • Constructor Details

    • MiniMaxModelArgs

      public MiniMaxModelArgs(int dim, int numLayers, int numHeads, int numKvHeads, int headDim, int vocabSize, double normEps, double ropeTheta, int rotaryDim, int denseIntermediateSize, int expertIntermediateSize, int sharedIntermediateSize, int numExperts, int numExpertsPerTok, int numSharedExperts, double routedScalingFactor, double swigluAlpha, double swigluLimit, boolean[] moeLayers, boolean[] sparseLayers, int indexDim, int numIndexHeads, int topkBlocks, int blockSize, int initBlocks, int localBlocks, int numMtpModules, int maxBatchSize, int maxSeqLen, MiniMaxWeightMode weightMode)
      Creates an instance of a MiniMaxModelArgs record class.
      Parameters:
      dim - the value for the dim record component
      numLayers - the value for the numLayers record component
      numHeads - the value for the numHeads record component
      numKvHeads - the value for the numKvHeads record component
      headDim - the value for the headDim record component
      vocabSize - the value for the vocabSize record component
      normEps - the value for the normEps record component
      ropeTheta - the value for the ropeTheta record component
      rotaryDim - the value for the rotaryDim record component
      denseIntermediateSize - the value for the denseIntermediateSize record component
      expertIntermediateSize - the value for the expertIntermediateSize record component
      sharedIntermediateSize - the value for the sharedIntermediateSize record component
      numExperts - the value for the numExperts record component
      numExpertsPerTok - the value for the numExpertsPerTok record component
      numSharedExperts - the value for the numSharedExperts record component
      routedScalingFactor - the value for the routedScalingFactor record component
      swigluAlpha - the value for the swigluAlpha record component
      swigluLimit - the value for the swigluLimit record component
      moeLayers - the value for the moeLayers record component
      sparseLayers - the value for the sparseLayers record component
      indexDim - the value for the indexDim record component
      numIndexHeads - the value for the numIndexHeads record component
      topkBlocks - the value for the topkBlocks record component
      blockSize - the value for the blockSize record component
      initBlocks - the value for the initBlocks record component
      localBlocks - the value for the localBlocks record component
      numMtpModules - the value for the numMtpModules record component
      maxBatchSize - the value for the maxBatchSize record component
      maxSeqLen - the value for the maxSeqLen record component
      weightMode - the value for the weightMode record component

  • Method Details

    • isMoe

      public boolean isMoe(int layerId)
      Returns whether a layer uses a mixture of experts.
      Parameters:
      layerId - zero-based decoder layer index.
      Returns:
      true for MoE, false for the dense FFN.

    • isSparse

      public boolean isSparse(int layerId)
      Returns whether a layer uses MiniMax Sparse Attention.
      Parameters:
      layerId - zero-based decoder layer index.
      Returns:
      true for MSA, false for dense GQA.

    • numSparseLayers

      public int numSparseLayers()
      Returns the number of MSA layers (each also keeps an index-key cache).
      Returns:
      MSA layer count.

    • numMoeLayers

      public int numMoeLayers()
      Returns the number of MoE layers.
      Returns:
      MoE layer count.

    • numIndexKeyHeads

      public int numIndexKeyHeads()
      Returns the number of index-key heads per token. The checkpoint's index_k_proj has width indexDim, i.e. one key head shared by all numIndexHeads index-query heads.
      Returns:
      index-key heads.

    • withWeightMode

      public MiniMaxModelArgs withWeightMode(MiniMaxWeightMode mode)
      Returns a copy with another weight mode.
      Parameters:
      mode - how linear weights are stored and multiplied.
      Returns:
      the new args.

    • hasMtp

      public boolean hasMtp()
      Returns whether native MTP modules are configured.
      Returns:
      true when MTP modules exist.

    • kvCacheLayout

      public KvCacheLayout kvCacheLayout()
      Returns the KV cache layout. Every layer, sparse or dense, keeps a full K/V cache.
      Returns:
      cache layout for KvCachePool.

    • kvCacheLayout

      public KvCacheLayout kvCacheLayout(TensorShardSpec shard)
      KV layout for a tensor-parallel shard (local KV head count).
      Parameters:
      shard - TP shard description, or null for the full layout.
      Returns:
      cache layout for KvCachePool.

    • fromHuggingFace

      public static MiniMaxModelArgs fromHuggingFace(String path, int maxBatchSize, int maxSeqLen) throws IOException
      Loads hyperparameters from a HuggingFace config.json. Accepts the multimodal wrapper (minimax_m3_vl, nested text_config) or a bare text config.
      Parameters:
      path - path to config.json.
      maxBatchSize - maximum batch size.
      maxSeqLen - maximum sequence length; <= 0 uses max_position_embeddings.
      Returns:
      model args.
      Throws:
      IOException - if the file cannot be read.

    • fromTextConfig

      public static MiniMaxModelArgs fromTextConfig(tools.jackson.databind.JsonNode text, int maxBatchSize, int maxSeqLen)
      Parses a text-config JSON object into model args.
      Parameters:
      text - text config node.
      maxBatchSize - maximum batch size.
      maxSeqLen - maximum sequence length; <= 0 uses max_position_embeddings.
      Returns:
      model args.

    • toString

      public final String toString()
      Returns a string representation of this record class. The representation contains the name of the class, followed by the name and value of each of the record components.
      Specified by:
      toString in class Record
      Returns:
      a string representation of this object

    • hashCode

      public final int hashCode()
      Returns a hash code value for this object. The value is derived from the hash code of each of the record components.
      Specified by:
      hashCode in class Record
      Returns:
      a hash code value for this object

    • equals

      public final boolean equals(Object o)
      Indicates whether some other object is "equal to" this one. The objects are equal if the other object is of the same class and if all the record components are equal. Reference components are compared with Objects::equals(Object,Object); primitive components are compared with the compare method from their corresponding wrapper classes.
      Specified by:
      equals in class Record
      Parameters:
      o - the object with which to compare
      Returns:
      true if this object is the same as the o argument; false otherwise.

    • dim

      public int dim()
      Returns the value of the dim record component.
      Returns:
      the value of the dim record component

    • numLayers

      public int numLayers()
      Returns the value of the numLayers record component.
      Returns:
      the value of the numLayers record component

    • numHeads

      public int numHeads()
      Returns the value of the numHeads record component.
      Returns:
      the value of the numHeads record component

    • numKvHeads

      public int numKvHeads()
      Returns the value of the numKvHeads record component.
      Returns:
      the value of the numKvHeads record component

    • headDim

      public int headDim()
      Returns the value of the headDim record component.
      Returns:
      the value of the headDim record component

    • vocabSize

      public int vocabSize()
      Returns the value of the vocabSize record component.
      Returns:
      the value of the vocabSize record component

    • normEps

      public double normEps()
      Returns the value of the normEps record component.
      Returns:
      the value of the normEps record component

    • ropeTheta

      public double ropeTheta()
      Returns the value of the ropeTheta record component.
      Returns:
      the value of the ropeTheta record component

    • rotaryDim

      public int rotaryDim()
      Returns the value of the rotaryDim record component.
      Returns:
      the value of the rotaryDim record component

    • denseIntermediateSize

      public int denseIntermediateSize()
      Returns the value of the denseIntermediateSize record component.
      Returns:
      the value of the denseIntermediateSize record component

    • expertIntermediateSize

      public int expertIntermediateSize()
      Returns the value of the expertIntermediateSize record component.
      Returns:
      the value of the expertIntermediateSize record component

    • sharedIntermediateSize

      public int sharedIntermediateSize()
      Returns the value of the sharedIntermediateSize record component.
      Returns:
      the value of the sharedIntermediateSize record component

    • numExperts

      public int numExperts()
      Returns the value of the numExperts record component.
      Returns:
      the value of the numExperts record component

    • numExpertsPerTok

      public int numExpertsPerTok()
      Returns the value of the numExpertsPerTok record component.
      Returns:
      the value of the numExpertsPerTok record component

    • numSharedExperts

      public int numSharedExperts()
      Returns the value of the numSharedExperts record component.
      Returns:
      the value of the numSharedExperts record component

    • routedScalingFactor

      public double routedScalingFactor()
      Returns the value of the routedScalingFactor record component.
      Returns:
      the value of the routedScalingFactor record component

    • swigluAlpha

      public double swigluAlpha()
      Returns the value of the swigluAlpha record component.
      Returns:
      the value of the swigluAlpha record component

    • swigluLimit

      public double swigluLimit()
      Returns the value of the swigluLimit record component.
      Returns:
      the value of the swigluLimit record component

    • moeLayers

      public boolean[] moeLayers()
      Returns the value of the moeLayers record component.
      Returns:
      the value of the moeLayers record component

    • sparseLayers

      public boolean[] sparseLayers()
      Returns the value of the sparseLayers record component.
      Returns:
      the value of the sparseLayers record component

    • indexDim

      public int indexDim()
      Returns the value of the indexDim record component.
      Returns:
      the value of the indexDim record component

    • numIndexHeads

      public int numIndexHeads()
      Returns the value of the numIndexHeads record component.
      Returns:
      the value of the numIndexHeads record component

    • topkBlocks

      public int topkBlocks()
      Returns the value of the topkBlocks record component.
      Returns:
      the value of the topkBlocks record component

    • blockSize

      public int blockSize()
      Returns the value of the blockSize record component.
      Returns:
      the value of the blockSize record component

    • initBlocks

      public int initBlocks()
      Returns the value of the initBlocks record component.
      Returns:
      the value of the initBlocks record component

    • localBlocks

      public int localBlocks()
      Returns the value of the localBlocks record component.
      Returns:
      the value of the localBlocks record component

    • numMtpModules

      public int numMtpModules()
      Returns the value of the numMtpModules record component.
      Returns:
      the value of the numMtpModules record component

    • maxBatchSize

      public int maxBatchSize()
      Returns the value of the maxBatchSize record component.
      Returns:
      the value of the maxBatchSize record component

    • maxSeqLen

      public int maxSeqLen()
      Returns the value of the maxSeqLen record component.
      Returns:
      the value of the maxSeqLen record component

    • weightMode

      public MiniMaxWeightMode weightMode()
      Returns the value of the weightMode record component.
      Returns:
      the value of the weightMode record component