Class Gpt2ByteMap

java.lang.Object

smile.llm.tokenizer.Gpt2ByteMap

public final class Gpt2ByteMap
extends Object

GPT-2 / HuggingFace byte-level BPE mapping between raw bytes and the printable Unicode characters used as tokenizer.json vocab keys.

HF stores each vocab entry as a string of these mapped characters. Tiktoken-style BPE operates on raw UTF-8 bytes, so keys must be converted with vocabTokenToBytes(String) when loading an HF vocab.

See Also:

• GPT-2 encoder

Method Summary

Modifier and Type	Method	Description
static char	byteToUnicode(int unsignedByte)	Maps a raw byte to its GPT-2 vocab character (for tests / debugging).
static Bytes	vocabTokenToBytes(String token)	Converts an HF vocab token string (GPT-2 unicode-mapped) into the raw byte sequence Tiktoken BPE expects.

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

vocabTokenToBytes

public static Bytes vocabTokenToBytes(String token)

Converts an HF vocab token string (GPT-2 unicode-mapped) into the raw byte sequence Tiktoken BPE expects. Falls back to UTF-8 of token if any character is outside the GPT-2 map (e.g. unusual added tokens).

Parameters:

token - vocab key from tokenizer.json / vocab.json.

Returns:

raw bytes for the rank map.

byteToUnicode

public static char byteToUnicode(int unsignedByte)

Maps a raw byte to its GPT-2 vocab character (for tests / debugging).

Parameters:

unsignedByte - byte value in [0, 255].

Returns:

the GPT-2 unicode code unit for that byte.