Class MiniMaxVisionTower

java.lang.Object

smile.deep.layer.LayerBlock

smile.llm.model.minimax.MiniMaxVisionTower

All Implemented Interfaces:

Function<Tensor,Tensor>, Layer

public class MiniMaxVisionTower
extends LayerBlock

MiniMax-M3 vision encoder: a CLIP-style ViT (Conv3d patch embedding, pre-LayerNorm, 32 pre-norm blocks with a partial 3D RoPE over t/h/w, no post-LayerNorm), followed by the projector MLP and the 2x2 patch-merge MLP that map patches to language-model embeddings.

Parameter names match the checkpoint once the vision_tower. prefix is dropped (vision_model.encoder.layers.N.self_attn.q_proj.weight, multi_modal_projector.linear_1.bias, ...), so load(File, Device, ScalarType) can hand tensors straight to loadStateDict.

Input is the processor's patch matrix: one row of C * T * P * P values per patch, patches ordered by (time group, merge block row, merge block column, row in block, column in block). The Conv3d is applied as the equivalent linear map.

Field Summary

Fields inherited from class LayerBlock

device, dtype, module

Constructor Summary

Constructors

Constructor	Description
MiniMaxVisionTower(MiniMaxVisionArgs args)	Constructs the module graph (on CPU, float32).

Method Summary

Modifier and Type	Method	Description
MiniMaxVisionArgs	args()	Returns the vision hyperparameters.
Tensor	forward(Tensor input)	Forward propagation (or forward pass) through the layer.
Tensor	forward(Tensor pixelValues, int[][] gridThw)	Encodes patches into language-model embeddings.
static MiniMaxVisionTower	load(File dir, Device device, ScalarType dtype)	Loads the vision weights of a checkpoint directory into a new tower on device.

Methods inherited from class LayerBlock

add, add, device, dtype, eval, isTraining, load, loadStateDict, module, save, setRequiresGrad, to, to, toString, train

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface Function

andThen, compose

Methods inherited from interface Layer

apply, name

Constructor Details

MiniMaxVisionTower

public MiniMaxVisionTower(MiniMaxVisionArgs args)

Constructs the module graph (on CPU, float32).

Parameters:

args - hyperparameters.

Method Details

args

public MiniMaxVisionArgs args()

Returns the vision hyperparameters.

Returns:

the hyperparameters.

forward

public Tensor forward(Tensor input)

Description copied from interface: Layer

Forward propagation (or forward pass) through the layer.

Parameters:

input - the input tensor.

Returns:

the output tensor.

forward

public Tensor forward(Tensor pixelValues,
 int[][] gridThw)

Encodes patches into language-model embeddings.

Parameters:

pixelValues - [N, patchDim] patches in the compute dtype, on this module's device.

gridThw - one (t, h, w) patch grid per image or video, in patch units; the rows of pixelValues are the grids' patches back to back.

Returns:

[N / merge^2, textHidden] embeddings, one row per merged token (the caller owns it).

load

public static MiniMaxVisionTower load(File dir,
 Device device,
 ScalarType dtype)
                               throws IOException

Loads the vision weights of a checkpoint directory into a new tower on device.

Parameters:

dir - directory with config.json and either model.safetensors.index.json or *.safetensors files holding the vision tensors.

device - target device.

dtype - compute dtype (the checkpoint stores float32 patch embeddings and bfloat16 for the rest).

Returns:

the loaded tower, or null when the checkpoint has no vision_config.

Throws:

IOException - if tensors are missing or unreadable.