TokenizerStream

java.lang.Object

smile.onnx.genai.TokenizerStream

All Implemented Interfaces:: AutoCloseable

public final class TokenizerStream extends Object implements AutoCloseable

Incremental token → text decoder used while streaming generation.

Method Summary

Modifier and Type

Method

Description

void

close()

String

decode(int token)

Decodes one newly generated token into a text chunk (may be empty while multi-byte / multi-token pieces accumulate).

Methods inherited from class Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details
- decode
  
  public String decode(int token)
  
  Decodes one newly generated token into a text chunk (may be empty while multi-byte / multi-token pieces accumulate).
  The native buffer is owned by the stream until the next decode call or close(); it must not be freed with OgaDestroyString.
  
  Parameters:
  
  token - token id.
  
  Returns:
  
  decoded chunk; never null.
- close
  
  public void close()
  
  Specified by:
  
  close in interface AutoCloseable