Package smile.onnx.genai

package smile.onnx.genai

Idiomatic Java API for ONNX Runtime GenAI.

This package wraps the GenAI C API through Panama FFM bindings in smile.onnx.genai.foreign. Handles implement AutoCloseable and throw unchecked GenAIException on native errors.

Layers

• Core handles — Config, Model, Tokenizer, Generator, …
• Convenience — SimpleGenAI for prompt → stream → text.
• Serve seam — GenAiChatModel implements LanguageModel; smile-serve uses it as OGA fallback (allowlist + optional Olive) when Torch CUDA/builtin does not apply.
• Allowlist / Olive target — GenAISupportedModels, GenAI.resolveOliveTarget().

Quick start

try (var genai = SimpleGenAI.of("models/phi-3-mini-cpu")) {
    try (var params = genai.createGeneratorParams()) {
        params.setSearchOption("max_length", 256)
              .setSearchOption("temperature", 0.8);
        String reply = genai.generate(params, "Hello!", System.out::print);
    }
}

Providers

Model.open(String) cascades CUDA → RyzenAI → OpenVINO NPU → QNN → DirectML (Windows) → CPU when SMILE_ONNX_GENAI_PROVIDER is auto (default). Each accelerator requires matching EP natives. Classical Vitis AI for general ONNX is SessionOptions.appendVitisAiExecutionProvider().

Native libraries

Both onnxruntime and onnxruntime-genai must be on the OS library search path. Use --enable-native-access=ALL-UNNAMED.

OgaEngine continuous batching is not wrapped yet; map it to smile.llm.engine.ModelExecutor in a later phase.

Related Packages

Package	Description
smile.onnx	Java API for the ONNX Runtime inference engine.
smile.onnx.genai.foreign
smile.onnx.foreign

Class	Description
Adapters	Container for LoRA adapters that can be activated on a Generator.
Audios	Loaded audios for multimodal GenAI models.
Config	GenAI model configuration: execution providers and overlay JSON.
GenAI	Process-wide ONNX Runtime GenAI controls: library availability, telemetry, logging helpers, and shutdown.
GenAiChatModel	LanguageModel adapter over ONNX Runtime GenAI.
GenAIException	Unchecked exception thrown when an ONNX Runtime GenAI operation fails.
GenAIOliveTarget	Olive optimize target derived from the GenAI EP cascade.
GenAISupportedModels	Allowlist of architectures supported by ONNX Runtime GenAI for chat.
GenAISupportedModels.Family	Chat-capable (and Whisper) families tracked for Olive / OGA conversion.
Generator	Autoregressive token generator.
GeneratorParams	Search and sampling options for a Generator.
Images	Loaded images for multimodal GenAI models.
Model	An ONNX Runtime GenAI model.
MultiModalProcessor	Converts text / images / audios into NamedTensors for multimodal models.
NamedTensors	Named collection of tensors matching model input names.
Sequences	A collection of token-id sequences (batch of prompts / responses).
SimpleGenAI	Convenience façade for single-prompt text generation with an optional streaming callback.
Tensor	Thin GenAI tensor wrapper for Generator.setModelInput(String, Tensor) / multimodal inputs.
Tensor.ElementType	Element types matching OgaElementType.
Tokenizer	Converts between text and GenAI token ids, including chat templates.
TokenizerStream	Incremental token → text decoder used while streaming generation.