ONNX GenAI

Overview

The smile.onnx.genai package provides idiomatic Java wrappers over the ONNX Runtime GenAI C API, built on Panama FFM bindings in smile.onnx.genai.foreign.

GenAI has no one-shot generate(): the loop is AppendTokens → while (!IsDone) GenerateNextToken. This package mirrors that contract and adds a LanguageModel adapter for smile-serve wiring.

Classical ONNX inference (InferenceSession) is covered on ONNX. Chat HTTP serving is on SMILE Serve.

Supported Models

The GenAI runtime loads any model directory that carries a genai_config.json and dispatches on the architecture recorded there. The families below match the upstream ONNX Runtime GenAI supported-model list and model builder; the checkpoint listed for each is a known-good, GenAI-ready Hub repository. SMILE's own native smile.llm implementations cover Llama-3.1 and Qwen3.5/3.8 separately — everything else runs through this package.

Vision-language and audio models use the Images / Audios / MultiModalProcessor path; the Serve chat endpoint is text-only, with multimodal media materialized to temp files first. File layouts and provider folders vary per repo — Model.open / GenAiChatModel.open auto-detect the nested package that matches your execution provider.

Prerequisites

Download GenAI binaries from the onnxruntime-genai releases and ORT from the onnxruntime releases.

Native libraries

Both shared libraries must be loadable. With pip (onnxruntime-genai-cuda / onnxruntime-genai), typical Windows locations are:

Use environment variables (recommended). GenAI preloads these DLLs by absolute path before FFM lookup, which also avoids an older C:\Windows\System32\onnxruntime.dll winning the Windows search order:

$ort = "$env:LOCALAPPDATA\Python\pythoncore-3.14-64\Lib\site-packages\onnxruntime\capi"
$genai = "$env:LOCALAPPDATA\Python\pythoncore-3.14-64\Lib\site-packages\onnxruntime_genai"
$model = "C:\path\to\phi3-mini-4k-instruct\cpu_and_mobile\cpu-int4-rtn-block-32-acc-level-4"

$env:ONNXRUNTIME_NATIVE_PATH = $ort
$env:ONNXRUNTIME_GENAI_NATIVE_PATH = $genai
$env:SMILE_ONNX_GENAI_MODEL = $model
# optional: $env:SMILE_ONNX_GENAI_PROVIDER = "auto"   # or "cuda" / "cpu"

Model.open / SimpleGenAI.open / GenAiChatModel.open run an accelerator cascade when preference is auto: CUDA → RyzenAI → OpenVINO NPU → QNN → DirectML (Windows) → CPU. Each step is attempted only when matching EP natives are present; Java failures fall through. On non-Windows, DirectML is skipped automatically. Classical Vitis AI (general ONNX on AMD NPU) is configured on smile.onnx.SessionOptions, not via this GenAI preference. Model.of always uses genai_config.json providers unchanged.

Check availability:

if (!GenAI.available()) {
    // natives missing — skip or fail fast
}
GenAI.init(); // throws GenAIException with install hints if load fails

Package Map

Quick Start

import smile.onnx.genai.*;

try (var genai = SimpleGenAI.of("models/phi-3-mini-4k-instruct-cpu")) {
    try (var params = genai.createGeneratorParams()) {
        params.setSearchOption("max_length", 256)
              .setSearchOption("temperature", 0.8);
        String reply = genai.generate(params, "Write a haiku about Java.", System.out::print);
    }
}

Canonical low-level loop:

try (var model = Model.of(modelDir);
     var tokenizer = model.createTokenizer();
     var params = model.createGeneratorParams();
     var generator = Generator.of(model, params);
     var stream = tokenizer.createStream()) {
    params.setSearchOption("max_length", 128);
    generator.appendTokens(tokenizer.encodeToArray(prompt));
    while (!generator.isDone()) {
        generator.generateNextToken();
        System.out.print(stream.decode(generator.getLastToken(0)));
    }
}

Chat / Serve Seam (GenAiChatModel)

try (var chat = GenAiChatModel.of("models/phi-3-mini-4k-instruct-cpu")) {
    var completion = chat.chat(
            new Message[]{ new Message(Role.user, "Hello!") },
            64, 0.7, 0.9, false, 0,
            new GenerationListener() {
                @Override public void onText(String chunk) {
                    System.out.print(chunk);
                }
            },
            null);
}

GenAiChatModel:

• family() → "onnx/genai"
• Builds GenAI chat-template JSON from smile.llm.Message[] (including tools)
• Streams via GenerationListener.onText; strips <think>…</think> and reports onThinkingTokens while streaming
• Honors BooleanSupplier cancelRequested between tokens
• Multimodal: local image/audio file paths through MultiModalProcessor
• Final content is sanitized with AssistantTextSanitizer (also via serve’s ToolCallPostProcessor for structured tool_calls)

smile-serve OGA fallback

ChatService keeps Torch for CUDA + builtin Llama/Qwen. Otherwise it tries ORT GenAI:

1. Local / HF snapshot with genai_config.json (repo root or nested under provider folders such as cuda/cuda-int4-rtn-block-32/, as in microsoft/Phi-4-mini-instruct-onnx) → GenAiChatModel.open. Nested packages are picked to match GenAI.resolveOliveTarget() (CUDA / DML / CPU).
2. Else a prior Olive cache hit under {SMILE_CACHE}/olive/... → open
3. Else chat stays unavailable (HTTP 503)

Serve does not run Olive at startup. olive optimize (int4/GPTQ) can take hours; convert offline with smile.chat.Olive.resolveOrConvert / the olive_cli.py bootstrap, then restart serve (same HF id finds the cache, or point smile.chat.model at the GenAI output directory).

Serve config (smile.chat.oga.*): enabled, precision (auto = cascade default; Olive optimize clamps unsupported values such as fp8 → int4), cache-dir, olive-command, optional device / provider. Offline Olive optimize --device/--provider follow GenAI.resolveOliveTarget() (same EP cascade as Model.open, honor SMILE_ONNX_GENAI_PROVIDER); DirectML is remapped to CPU for the Olive CLI because optimize does not list DmlExecutionProvider. If Olive fails while registering an unused ORT EP (common on Windows when the CUDA wheel ships onnxruntime_providers_tensorrt.dll but TensorRT/nvinfer is not installed), the offline launcher uses a Python bootstrap that only registers EPs Olive explicitly requested, and may still retry with CPUExecutionProvider. Override the interpreter with SMILE_OLIVE_PYTHON. Conversion uses --exporter model_builder (GenAI-ready output). Runtime load still uses GenAiChatModel.open / the GenAI EP cascade.

Olive’s Python env needs a full toolchain for optimize (especially int4/GPTQ calibration). Typical install:

pip install "olive-ai[gpu]" datasets
# or CPU: pip install olive-ai datasets

Missing datasets fails mid-run with ModuleNotFoundError: No module named 'datasets'.

Tools I/O (Phase 1): OpenAI tools reach the GenAI chat template; completions are post-processed to structured tool_calls (JsonToolCallParser / Qwen3 XML). No agent loop / tool execution.

Multimodal (Phase 2): serve materializes data-URL / HTTP media to temp files (GenAiMediaMaterializer) before GenAiChatModel multimodal chat.

Continuous batching / GenAI OgaEngine remains out of scope.

CUDA / NPU Providers

Model.open cascade (SMILE_ONNX_GENAI_PROVIDER=auto):

try (var model = Model.open(modelDir)) {
    // model.provider() is cuda | ryzenai | openvino | qnn | dml | default
}

AMD Ryzen AI (LLMs): hybrid (NPU+iGPU) vs NPU-only is which model package you load (AMD HF hybrid vs NPU collections). Needs onnxruntime-genai-directml-ryzenai (or Ryzen AI MSI), NPU drivers, and a GPU driver for hybrid.

Windows DirectML: general GPU via onnxruntime_providers_dml (iGPU/dGPU). Tried on auto after NPU candidates and before CPU; inactive off Windows.

AMD Vitis AI (general ONNX on NPU): use smile.onnx.SessionOptions.appendVitisAiExecutionProvider() — not this GenAI cascade.

Out of scope for GenAI cascade: ROCm/MIGraphX, NvTensorRtRtx (TensorRT-RTX).

Or set GenAI providers explicitly:

try (var config = Config.of(modelDir)) {
    config.clearProviders()
          .appendProvider("cuda")
          .setProviderOption("cuda", "device_id", "0");
    try (var model = Model.of(config)) {
        // ...
    }
}

Tests

Unit tests under deep/src/test/java/smile/onnx/genai/:

• GenAIExceptionTest / GenAIProviderPreferenceTest — no natives required
• GenAINativeTest — needs natives; model tests need SMILE_ONNX_GENAI_MODEL

GitHub Actions (.github/workflows/ci.yml) installs CPU onnxruntime / onnxruntime-genai, sets ONNXRUNTIME_NATIVE_PATH / ONNXRUNTIME_GENAI_NATIVE_PATH, and forces SMILE_ONNX_GENAI_PROVIDER=cpu (runners have no GPU). Without SMILE_ONNX_GENAI_MODEL, model cases are skipped; sequencesRoundTrip still runs when natives load.

Set the environment variables in the shell, then run tests. Both sbt and Gradle fork the test JVM and inherit these variables. Tests call Model.open, so they pick up CUDA when the GenAI CUDA EP works and otherwise use CPU:

$env:ONNXRUNTIME_NATIVE_PATH = $ort
$env:ONNXRUNTIME_GENAI_NATIVE_PATH = $genai
$env:SMILE_ONNX_GENAI_MODEL = $model
# $env:SMILE_ONNX_GENAI_PROVIDER = "auto"  # default; use "cpu" to force CPU

# If you change SMILE_* after a prior Gradle run, avoid a stale configuration-cache
# entry that pinned the old env:
#   ./gradlew --no-configuration-cache :deep:test --tests "smile.onnx.genai.*"
sbt "deep/testOnly smile.onnx.genai.*"
# or
./gradlew :deep:test --tests "smile.onnx.genai.*"

Do not pass native/model settings as sbt "-D…" or sbt "-J-D…" — those do not configure the forked test JVM reliably (and -J-D fails under sbt 2).

Without SMILE_ONNX_GENAI_MODEL, only library-level checks run. Point it at a real GenAI model directory (e.g. Phi-3 CPU) for full generate smokes. Tiny onnxruntime-genai fixtures under test/models/ (Identity dummy_*.onnx) only support load / tokenize / create-generator, not a full decode loop.

Shutdown

Call GenAI.shutdown() only after every GenAI AutoCloseable is closed. The next GenAI API call re-initializes the library.

See Also

• ONNX — classical InferenceSession / execution providers
• Large Language Models — LanguageModel API and LibTorch path
• SMILE Serve — chat HTTP API and OGA fallback