Class Generator

java.lang.Object
smile.onnx.genai.Generator
All Implemented Interfaces:
AutoCloseable, Iterable<Integer>
public final class Generator extends Object implements AutoCloseable, Iterable<Integer>
Autoregressive token generator. Canonical loop: 
generator.appendTokens(promptTokens);
try (var stream = tokenizer.createStream()) {
    while (!generator.isDone()) {
        generator.generateNextToken();
        String chunk = stream.decode(generator.getLastToken(0));
    }
}

Also implements IterableInteger> so callers can write for (int token : generator).

  • Method Details

    • of

      public static Generator of(Model model, GeneratorParams params)
      Creates a generator from a model and search parameters.
      Parameters:
      model - the model (must outlive this generator).
      params - search parameters.
      Returns:
      a new generator owned by the caller.

    • isDone

      public boolean isDone()
      Returns whether generation has finished.
      Returns:
      true when no more tokens will be produced.

    • appendTokens

      public void appendTokens(int[] tokens)
      Appends prompt token ids.
      Parameters:
      tokens - prompt tokens.

    • appendSequences

      public void appendSequences(Sequences sequences)
      Appends encoded sequences as the prompt.
      Parameters:
      sequences - token sequences (typically from Tokenizer.encode(String)).

    • setModelInput

      public void setModelInput(String name, Tensor tensor)
      Sets a single named model input tensor.
      Parameters:
      name - input name.
      tensor - input tensor.

    • setInputs

      public void setInputs(NamedTensors namedTensors)
      Sets multimodal / multi-input tensors (e.g. from MultiModalProcessor).
      Parameters:
      namedTensors - named inputs.

    • generateNextToken

      public void generateNextToken()
      Generates the next token (computes logits and samples).

    • tokenCount

      public long tokenCount()
      Returns the current token count in the generator state.
      Returns:
      token count.

    • rewindTo

      public void rewindTo(long newLength)
      Rewinds generation state to the given length.
      Parameters:
      newLength - desired length in tokens after rewind.

    • getSequence

      public int[] getSequence(long index)
      Returns a copy of the full token sequence at index.
      Parameters:
      index - sequence index (usually 0).
      Returns:
      token ids including the prompt.

    • getLastToken

      public int getLastToken(long index)
      Returns the last token in the sequence at index.
      Parameters:
      index - sequence index (usually 0).
      Returns:
      last token id.

    • setActiveAdapter

      public void setActiveAdapter(Adapters adapters, String adapterName)
      Activates a previously loaded LoRA adapter for this generator.
      Parameters:
      adapters - adapter container.
      adapterName - adapter name passed to Adapters.load(String, String).

    • getInput

      public Tensor getInput(String name)
      Returns a copy of a named model input as a Tensor.
      Parameters:
      name - input name.
      Returns:
      tensor owned by the caller.

    • getOutput

      public Tensor getOutput(String name)
      Returns a copy of a named model output as a Tensor.
      Parameters:
      name - output name.
      Returns:
      tensor owned by the caller.

    • iterator

      public Iterator<Integer> iterator()
      Specified by:
      iterator in interface Iterable<Integer>

    • close

      public void close()
      Specified by:
      close in interface AutoCloseable