Class Tokenizer

java.lang.Object

smile.onnx.genai.Tokenizer

All Implemented Interfaces:

AutoCloseable

public final class Tokenizer
extends Object
implements AutoCloseable

Converts between text and GenAI token ids, including chat templates.

Method Summary

Modifier and Type	Method	Description
String	applyChatTemplate(String templateStr, String messagesJson, String toolsJson, boolean addGenerationPrompt)	Applies the model's chat template.
int	bosTokenId()	Returns the BOS token id.
void	close()
TokenizerStream	createStream()	Creates a streaming decoder for use with Generator.
String	decode(int[] tokens)	Decodes token ids to text.
String[]	decodeBatch(Sequences sequences)	Decodes each sequence in a Sequences to text.
Sequences	encode(String text)	Encodes a single string into a Sequences with one sequence.
Sequences	encodeBatch(String[] texts)	Encodes multiple strings into one Sequences (one entry per string).
int[]	encodeToArray(String text)	Encodes text and returns the first sequence as an int[].
static Tokenizer	of(String modelDir)	Creates a tokenizer from a model directory path.
static Tokenizer	of(Path modelDir)	Creates a tokenizer from a model directory path.
static Tokenizer	of(Config config)	Creates a tokenizer from a config.
static Tokenizer	of(Model model)	Creates a tokenizer from a loaded model.
int	padTokenId()	Returns the PAD token id.
int	toTokenId(String token)	Converts a string to a single token id.
void	updateOptions(Map<String,String> options)	Updates tokenizer options (key/value string pairs).

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

of

public static Tokenizer of(Model model)

Creates a tokenizer from a loaded model.

Parameters:

model - the model.

Returns:

a new tokenizer owned by the caller.

of

public static Tokenizer of(Config config)

Creates a tokenizer from a config.

Parameters:

config - the config.

Returns:

a new tokenizer owned by the caller.

of

public static Tokenizer of(String modelDir)

Creates a tokenizer from a model directory path.

Parameters:

modelDir - GenAI model directory.

Returns:

a new tokenizer owned by the caller.

of

public static Tokenizer of(Path modelDir)

Creates a tokenizer from a model directory path.

Parameters:

modelDir - GenAI model directory.

Returns:

a new tokenizer owned by the caller.

encode

public Sequences encode(String text)

Encodes a single string into a Sequences with one sequence.

Parameters:

text - text to encode.

Returns:

sequences owned by the caller.

encodeBatch

public Sequences encodeBatch(String[] texts)

Encodes multiple strings into one Sequences (one entry per string).

Parameters:

texts - texts to encode.

Returns:

sequences owned by the caller.

encodeToArray

public int[] encodeToArray(String text)

Encodes text and returns the first sequence as an int[].

Parameters:

text - text to encode.

Returns:

token ids.

decode

public String decode(int[] tokens)

Decodes token ids to text.

Parameters:

tokens - token ids.

Returns:

decoded text.

decodeBatch

public String[] decodeBatch(Sequences sequences)

Decodes each sequence in a Sequences to text.

Parameters:

sequences - encoded sequences.

Returns:

one string per sequence.

applyChatTemplate

public String applyChatTemplate(String templateStr,
 String messagesJson,
 String toolsJson,
 boolean addGenerationPrompt)

Applies the model's chat template.

Parameters:

templateStr - template string, or null to use the model default.

messagesJson - messages as a JSON array string.

toolsJson - tools JSON, or null when unused.

addGenerationPrompt - whether to append the assistant generation prompt.

Returns:

formatted prompt text.

updateOptions

public void updateOptions(Map<String,String> options)

Updates tokenizer options (key/value string pairs).

Parameters:

options - option map; no-op when null or empty.

createStream

public TokenizerStream createStream()

Creates a streaming decoder for use with Generator.

Returns:

a new stream owned by the caller.

bosTokenId

public int bosTokenId()

Returns the BOS token id.

Returns:

BOS token id.

padTokenId

public int padTokenId()

Returns the PAD token id.

Returns:

PAD token id.

toTokenId

public int toTokenId(String token)

Converts a string to a single token id.

Parameters:

token - token string.

Returns:

token id.

close

public void close()

Specified by:

close in interface AutoCloseable