Class HuggingFaceBpeVocab

java.lang.Object

smile.llm.tokenizer.HuggingFaceBpeVocab

public final class HuggingFaceBpeVocab
extends Object

Loads HuggingFace tokenizer.json BPE vocab into Tiktoken-style byte ranks.

Method Summary

Modifier and Type	Method	Description
static Map<Bytes,Integer>	loadTokenizerJson(Path tokenizerJson)	Extracts vocab from HuggingFace tokenizer.json (model.vocab plus added_tokens).
static void	mergeAddedTokens(Map<Bytes,Integer> ranks, tools.jackson.databind.JsonNode root)	Merges added_tokens entries into ranks (content → id).

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

loadTokenizerJson

public static Map<Bytes,Integer> loadTokenizerJson(Path tokenizerJson)
                                            throws IOException

Extracts vocab from HuggingFace tokenizer.json (model.vocab plus added_tokens).

Parameters:

tokenizerJson - path to tokenizer.json.

Returns:

token byte sequence → id map (GPT-2 keys converted to raw bytes).

Throws:

IOException - if the file is missing or not a valid tokenizer.json.

mergeAddedTokens

public static void mergeAddedTokens(Map<Bytes,Integer> ranks,
 tools.jackson.databind.JsonNode root)

Merges added_tokens entries into ranks (content → id).

Parameters:

ranks - mutable BPE rank map.

root - tokenizer JSON root containing added_tokens.