Class Tokenizer

java.lang.Object

smile.llm.tokenizer.Tiktoken

smile.llm.model.qwen.Tokenizer

All Implemented Interfaces:

Tokenizer

public class Tokenizer
extends Tiktoken

Qwen3.5 chat tokenizer (byte-level BPE via Tiktoken).

Loads from a checkpoint directory. Prefers HuggingFace tokenizer.json (includes added_tokens with the real <|im_start|> / <|im_end|> ids) over vocab.json alone.

Field Summary

Fields inherited from class Tiktoken

ranks, specialTokens

Constructor Summary

Constructors

Constructor	Description
Tokenizer(Map<Bytes,Integer> ranks)	Constructor.
Tokenizer(Map<Bytes,Integer> ranks, String bos, String eos, String... specialTokens)	Constructor.

Method Summary

Modifier and Type	Method	Description
int[]	encodeDialog(Message... dialog)	Encodes a chat dialog in the Qwen chat-template format and leaves the assistant header open for completion.
int[]	encodeDialog(Message[] dialog, ChatOptions options)	Encodes a chat dialog with optional tool-calling options.
static Map<Bytes,Integer>	loadTokenizerJson(Path tokenizerJson)	Extracts vocab from HuggingFace tokenizer.json (model.vocab plus added_tokens).
static Tokenizer	of(String checkpointDir)	Loads a tokenizer from a HuggingFace checkpoint directory.
int	pad()	Padding token id (<|endoftext|>).
void	requireChatSpecialsInVocab(int vocabSize)	Ensures chat control tokens lie in [0, vocabSize).
int[]	stopTokens()	Stop token ids used during generation.

Methods inherited from class Tiktoken

allowSpecialTokens, decode, encode, encode, isSpecialTokenAllowed, load, size, specialToken, tokenize, tryDecode, tryDecode

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

Tokenizer

public Tokenizer(Map<Bytes,Integer> ranks)

Constructor.

Parameters:

ranks - token byte sequence → id map.

Tokenizer

public Tokenizer(Map<Bytes,Integer> ranks,
 String bos,
 String eos,
 String... specialTokens)

Constructor.

Parameters:

ranks - token → id map.

bos - beginning-of-sequence token.

eos - end-of-sequence token.

specialTokens - additional special tokens.

Method Details

pad

public int pad()

Padding token id (<|endoftext|>).

Returns:

pad token id.

stopTokens

public int[] stopTokens()

Stop token ids used during generation.

Returns:

stop token id array.

encodeDialog

public int[] encodeDialog(Message... dialog)

Encodes a chat dialog in the Qwen chat-template format and leaves the assistant header open for completion.

Parameters:

dialog - conversation turns.

Returns:

token ids.

encodeDialog

public int[] encodeDialog(Message[] dialog,
 ChatOptions options)

Encodes a chat dialog with optional tool-calling options.

Parameters:

dialog - conversation turns.

options - chat options; when tools are present (or the dialog already contains tool turns), uses Qwen3ChatTemplate.

Returns:

token ids.

of

public static Tokenizer of(String checkpointDir)
                    throws IOException

Loads a tokenizer from a HuggingFace checkpoint directory.

Order matters: tokenizer.json is preferred because it carries added_tokens with the true chat special ids. Loading vocab.json alone and then appending specials at maxId+1 remaps <|im_start|> past vocab_size and crashes embedding gather on GPU.

Parameters:

checkpointDir - model directory.

Returns:

tokenizer instance.

Throws:

IOException - if no supported tokenizer files are found.

loadTokenizerJson

public static Map<Bytes,Integer> loadTokenizerJson(Path tokenizerJson)
                                            throws IOException

Extracts vocab from HuggingFace tokenizer.json (model.vocab plus added_tokens).

Parameters:

tokenizerJson - path to tokenizer.json.

Returns:

token byte sequence → id map (GPT-2 keys converted to raw bytes).

Throws:

IOException - if the file is missing or not a valid tokenizer.json.

requireChatSpecialsInVocab

public void requireChatSpecialsInVocab(int vocabSize)

Ensures chat control tokens lie in [0, vocabSize). Call after construction against the model config vocab size.

Parameters:

vocabSize - embedding / lm_head vocabulary size from model config.