Class MultiModalProcessor

java.lang.Object

smile.onnx.genai.MultiModalProcessor

All Implemented Interfaces:

AutoCloseable

public final class MultiModalProcessor
extends Object
implements AutoCloseable

Converts text / images / audios into NamedTensors for multimodal models.

Method Summary

Modifier and Type	Method	Description
void	close()
TokenizerStream	createStream()	Creates a tokenizer stream from this processor.
String	decode(int[] tokens)	Decodes token ids using the processor's tokenizer.
static MultiModalProcessor	of(Model model)	Creates a processor for the given model.
NamedTensors	processAudios(String prompt, Audios audios)	Processes a prompt and optional audios into model inputs.
NamedTensors	processImages(String prompt, Images images)	Processes a prompt and optional images into model inputs.
NamedTensors	processImagesAndAudios(String prompt, Images images, Audios audios)	Processes a prompt with optional images and audios.

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

of

public static MultiModalProcessor of(Model model)

Creates a processor for the given model.

Parameters:

model - multimodal GenAI model.

Returns:

processor owned by the caller.

processImages

public NamedTensors processImages(String prompt,
 Images images)

Processes a prompt and optional images into model inputs.

Parameters:

prompt - text prompt.

images - images, or null.

Returns:

named tensors owned by the caller.

processAudios

public NamedTensors processAudios(String prompt,
 Audios audios)

Processes a prompt and optional audios into model inputs.

Parameters:

prompt - text prompt.

audios - audios, or null.

Returns:

named tensors owned by the caller.

processImagesAndAudios

public NamedTensors processImagesAndAudios(String prompt,
 Images images,
 Audios audios)

Processes a prompt with optional images and audios.

Parameters:

prompt - text prompt.

images - images, or null.

audios - audios, or null.

Returns:

named tensors owned by the caller.

decode

public String decode(int[] tokens)

Decodes token ids using the processor's tokenizer.

Parameters:

tokens - token ids.

Returns:

decoded text.

createStream

public TokenizerStream createStream()

Creates a tokenizer stream from this processor.

Returns:

stream owned by the caller.

close

public void close()

Specified by:

close in interface AutoCloseable