Class QwenVlProcessor

java.lang.Object

smile.llm.model.qwen.QwenVlProcessor

public class QwenVlProcessor
extends Object

HuggingFace-compatible image/video preprocessor for Qwen3.8 native VLM.

Produces packed patches, grid metadata, expanded chat token ids, and interleaved mRoPE position planes.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static final record	QwenVlProcessor.ProcessedMultimodal	Result of multimodal preprocessing for one prompt.

Constructor Summary

Constructors

Constructor	Description
QwenVlProcessor(QwenVisionArgs visionArgs, Tokenizer tokenizer, int patchSize, int temporalPatchSize, int mergeSize, int minPixels, int maxPixels, double[] imageMean, double[] imageStd, double defaultVideoFps, int videoMaxPixels)	Creates a VL processor for chat tokenization and vision preprocessing.

Method Summary

Modifier and Type	Method	Description
static QwenVlProcessor	fromCheckpoint(String checkpointDir, QwenVisionArgs visionArgs, Tokenizer tokenizer)	Loads processor settings from a checkpoint directory.
QwenVlProcessor.ProcessedMultimodal	process(Message... dialog)	Processes a chat dialog into tokens, vision tensors, and mRoPE planes.

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

QwenVlProcessor

public QwenVlProcessor(QwenVisionArgs visionArgs,
 Tokenizer tokenizer,
 int patchSize,
 int temporalPatchSize,
 int mergeSize,
 int minPixels,
 int maxPixels,
 double[] imageMean,
 double[] imageStd,
 double defaultVideoFps,
 int videoMaxPixels)

Creates a VL processor for chat tokenization and vision preprocessing.

Parameters:

visionArgs - vision / token id config.

tokenizer - chat tokenizer.

patchSize - spatial patch size.

temporalPatchSize - temporal patch size.

mergeSize - spatial merge size.

minPixels - smart-resize min pixels.

maxPixels - smart-resize max pixels.

imageMean - normalize mean (RGB).

imageStd - normalize std (RGB).

defaultVideoFps - default video sampling fps.

videoMaxPixels - video smart-resize max pixels.

Method Details

fromCheckpoint

public static QwenVlProcessor fromCheckpoint(String checkpointDir,
 QwenVisionArgs visionArgs,
 Tokenizer tokenizer)
                                      throws IOException

Loads processor settings from a checkpoint directory.

Parameters:

checkpointDir - model directory.

visionArgs - vision args from config.json.

tokenizer - tokenizer.

Returns:

processor.

Throws:

IOException - if configs cannot be read.

process

public QwenVlProcessor.ProcessedMultimodal process(Message... dialog)
                                            throws IOException

Processes a chat dialog into tokens, vision tensors, and mRoPE planes.

Parameters:

dialog - conversation turns.

Returns:

processed multimodal batch (single sequence).

Throws:

IOException - if media cannot be loaded.