Deep Learning

Deep learning is based on artificial neural networks with representation learning. The adjective “deep” refers to stacking many layers so that each stage transforms the input into a slightly more abstract representation. Unlike classical pipelines that rely on hand-crafted features, deep models discover useful features from data — though architecture choices such as depth, width, and regularization still matter.

SMILE’s smile-core module already ships a tabular MLP for classification and regression. The smile-deep module goes further: an idiomatic Java API over LibTorch (CPU, CUDA, and MPS), computer-vision models and transforms (smile.vision), and ONNX Runtime inference (smile.onnx). Large language models and ONNX GenAI are covered separately on llm.html.

Prerequisites

Runtime requirements for smile-deep:

Java 25+: Panama FFM is used for native interop; use a JDK that matches the project’s Java toolchain.
smile_torch / LibTorch: The native smile_torch shared library and its LibTorch dependencies must be on the platform loader path: PATH (Windows), LD_LIBRARY_PATH (Linux), or DYLD_LIBRARY_PATH (macOS).
ONNX Runtime: For smile.onnx, place onnxruntime.dll / libonnxruntime.so / libonnxruntime.dylib on the same search path (ORT releases).
--enable-native-access=ALL-UNNAMED: Required when launching outside Gradle or SMILE Studio so FFM can load native code.
--add-opens=java.base/java.nio=ALL-UNNAMED: Optional but recommended if you hit buffer-interop errors; SMILE’s Gradle tests set this flag.

// Example JVM flags for a custom launcher
// java --enable-native-access=ALL-UNNAMED \
//      --add-opens=java.base/java.nio=ALL-UNNAMED \
//      -jar myapp.jar

Module Structure

The deep-learning surface (excluding LLMs and GenAI) is organized as follows:

smile.deep
├── tensor/        Tensor, Index, Device, DeviceType, ScalarType, Layout
├── layer/         Layer interface and built-in layers
├── activation/    ActivationFunction and activation modules
├── metric/        Accuracy, Precision, Recall, F1Score, Averaging
├── Loss.java      Static factories for standard losses
├── Optimizer.java SGD, Adam, AdamW, RMSprop
├── Model.java     Trainable model wrapper over a LayerBlock
├── Dataset.java   Dataset interface + factories
├── DatasetImpl.java, SampleBatch.java
└── CUDA.java      GPU info helpers

smile.torch
├── smile_torch_h.java  FFM downcalls for the C ABI
└── Native.java         Cleaner / error-handling over raw FFM bindings

smile.vision
├── transform/     Transform, ImageClassification pipeline
├── layer/         MBConv, FusedMBConv, Conv2dNormActivation, …
├── EfficientNet.java   EfficientNet-V2 + pretrained factories
├── VisionModel.java    LayerBlock coupled with a Transform
├── ImageDataset.java   Folder-per-class dataset with prefetch
└── ImageNet.java       1000-class labels / folders / target maps

smile.onnx
├── foreign/       Panama FFM bindings to the ONNX Runtime C API
├── InferenceSession.java  Load and run ONNX models
├── OrtValue.java          Tensor / sequence / map containers
├── SessionOptions.java    Threads, graph opts, execution providers
├── RunOptions.java        Per-run logging and cancellation
├── Environment.java       Shared OrtEnv / thread pools
└── ModelMetadata.java, NodeInfo.java, TensorInfo.java, …

The native bridge lives under deep/src/main/cpp and exposes a compact C ABI (smile_torch) over LibTorch. ONNX Runtime is linked separately through smile.onnx.foreign.

Tensors

Tensor is a multidimensional array backed by a native LibTorch tensor. It implements AutoCloseable — always close tensors (or use a scope) when they are no longer needed to avoid native memory leaks.

Factory Methods

import smile.deep.tensor.Tensor;

// Zeros / ones
Tensor z = Tensor.zeros(3, 4);       // shape [3,4], float32
Tensor o = Tensor.ones(2, 3);

// Random
Tensor r  = Tensor.rand(5, 5);       // uniform [0,1)
Tensor rn = Tensor.randn(5, 5);      // standard normal

// From Java arrays
float[] data = {1f, 2f, 3f, 4f};
Tensor t = Tensor.of(data, 2, 2);    // shape [2,2]

long[] ldata = {0L, 1L, 2L};
Tensor li = Tensor.of(ldata, 3);     // Int64 tensor

// Arange and eye
Tensor ar = Tensor.arange(0, 10, 1); // [0,1,...,9]
Tensor eye = Tensor.eye(4);

Indexing

smile.deep.tensor.Index provides Python-style index objects for slicing and advanced indexing.

import smile.deep.tensor.Index;
import smile.deep.tensor.Tensor;

Tensor t = Tensor.rand(4, 4);

Tensor col1 = t.get(Index.Colon, Index.of(1));   // all rows, column 1
Tensor row2 = t.get(Index.of(2));                // row 2
Tensor sub  = t.get(Index.Slice(1, 3));          // rows 1–2
Tensor last = t.get(Index.Ellipsis, Index.of(3));
Tensor newDim = t.get(Index.None, Index.of(0));  // insert batch dim

int[] rows = {0, 2};
Tensor rowIdx = Tensor.of(rows, 2);
Tensor subset = t.get(rowIdx);                   // rows 0 and 2

Arithmetic & Math

import smile.deep.tensor.Device;
import smile.deep.tensor.ScalarType;
import smile.deep.tensor.Tensor;

Tensor a = Tensor.ones(3);
Tensor b = Tensor.ones(3).mul(2.0);

// Non-mutating
Tensor sum  = a.add(b);
Tensor diff = a.sub(1.0f);
Tensor prod = a.mul(3.0);
Tensor quot = a.div(2.0);

// In-place (trailing underscore)
a.add_(1.0);
a.sub_(0.5f);
a.mul_(2.0);
a.exp_();
a.fill_(0.0f);

// Reductions and shape
double s = a.sum().doubleValue();
Tensor argmax = a.argmax(0, false);
Tensor topk2  = a.topk(2, 0, true, true)._1();

long[] shape = a.shape();
int rank = a.dim();
Tensor flat = a.view(-1);
Tensor t2d = flat.reshape(3, 1);
Tensor tr = t2d.t().contiguous();

// Casting / device move
Tensor fp16 = a.to(ScalarType.Float16);
Tensor onCuda = a.to(Device.CUDA());

AutoScope & noGradGuard

Use AutoScope with Tensor.push / Tensor.pop to batch-free many tensors. Call promoteToParent() on values that must outlive the current scope. For inference, Tensor.noGradGuard() disables gradient tracking and reduces memory use.

import smile.deep.tensor.Tensor;
import smile.util.AutoScope;

try (var scope = new AutoScope()) {
    Tensor.push(scope);
    Tensor result = Tensor.randn(8, 8).mul(2.0);
    result.promoteToParent(); // survive pop of this scope
    Tensor.pop();             // closes other tracked tensors
}

try (var guard = Tensor.noGradGuard()) {
    // Tensor output = model.forward(input);
    // no gradient graph is built
}

Options: dtype & device

Set global defaults with Tensor.setDefaultOptions, or pass a per-call Tensor.Options. There is no Device.setDefaultDevice() — prefer Device.preferredDevice() and apply it via options or model.to(device).

import smile.deep.tensor.Device;
import smile.deep.tensor.ScalarType;
import smile.deep.tensor.Tensor;
import smile.deep.tensor.Tensor.Options;

Device device = Device.preferredDevice(); // CUDA if available, else CPU

Tensor.setDefaultOptions(new Options()
        .dtype(ScalarType.Float32)
        .device(device));

// Per-tensor override
Tensor t = Tensor.ones(new Options().dtype(ScalarType.Float64), 3, 3);

// BF16 performance check (Ampere or newer)
boolean bf16 = Tensor.isBF16Supported();

Layers

All layers implement Layer, which extends Function<Tensor, Tensor> and exposes forward, a native module handle, a name, and to(Device).

// Layer interface (sketch)
// public interface Layer extends Function<Tensor, Tensor> {
//     Tensor forward(Tensor input);
//     MemorySegment module();
//     String name();
//     Layer to(Device device);
// }

Dense Shortcuts

Layer factories combine a linear layer with an activation (and optional dropout) into a SequentialBlock.

import smile.deep.layer.Layer;
import smile.deep.layer.LinearLayer;
import smile.deep.layer.SequentialBlock;

LinearLayer fc = Layer.linear(128, 64);           // no activation
SequentialBlock r  = Layer.relu(128, 64);        // Linear + ReLU
SequentialBlock rd = Layer.relu(128, 64, 0.2);  // + Dropout(0.2)
SequentialBlock g  = Layer.gelu(128, 64);
SequentialBlock s  = Layer.silu(128, 64);
SequentialBlock t  = Layer.tanh(128, 64);
SequentialBlock sg = Layer.sigmoid(128, 64);
SequentialBlock ls = Layer.logSoftmax(128, 64);
SequentialBlock lk = Layer.leaky(128, 64, 0.01);

Convolution & Pooling

import smile.deep.layer.Layer;

// Simple conv: in, out, kernel
var c1 = Layer.conv2d(3, 32, 3);

// Full control: kernel, stride, padding, dilation, groups, bias, mode
var c2 = Layer.conv2d(3, 32, 3, 1, 1, 1, 1, true, "zeros");

// String padding ("valid" or "same")
var c3 = Layer.conv2d(3, 32, 3, 1, "same", 1, 1, true, "zeros");

var mp = Layer.maxPool2d(2);
var ap = Layer.avgPool2d(2);
var aa = Layer.adaptiveAvgPool2d(1);  // global 1×1

Norms, Dropout & Embedding

import smile.deep.layer.Layer;

var bn1 = Layer.batchNorm1d(64);
var bn2 = Layer.batchNorm2d(32);
var gn  = Layer.groupNorm(4, 32);
var rms = Layer.rmsNorm(64);

var drop = Layer.dropout(0.3);
var emb  = Layer.embedding(50000, 256);
var emb2 = Layer.embedding(50000, 256, 1.0);  // scale alpha

SequentialBlock

import smile.deep.layer.Layer;
import smile.deep.layer.SequentialBlock;
import smile.deep.tensor.Tensor;

SequentialBlock mlp = new SequentialBlock(
    Layer.relu(784, 256),
    Layer.relu(256, 128),
    Layer.logSoftmax(128, 10)
);

// Tensor output = mlp.forward(input);

SequentialBlock seq = new SequentialBlock();
seq.add(Layer.linear(64, 32));
seq.add(Layer.relu(32, 10));

Activations

Activations implement ActivationFunction (which extends Layer). Use them standalone or inside a SequentialBlock.

Class	Activation
`ReLU`	max(0, x)
`LeakyReLU`	max(αx, x)
`GELU`	Gaussian-error linear unit
`SiLU`	x·σ(x) (Swish)
`Tanh`	tanh(x)
`Sigmoid`	σ(x)
`Softmax`	softmax along last dim
`LogSoftmax`	log-softmax
`LogSigmoid`	log(σ(x))
`GLU`	Gated linear unit
`HardShrink`	x if \|x\| > λ else 0
`SoftShrink`	sign(x)·max(0,\|x\|−λ)
`TanhShrink`	x − tanh(x)

import smile.deep.activation.GELU;
import smile.deep.activation.ReLU;
import smile.deep.tensor.Tensor;

Tensor x = Tensor.randn(8, 16);
ReLU relu = new ReLU(true);   // inplace
Tensor y = relu.forward(x);

GELU gelu = new GELU();
Tensor z = gelu.forward(x);

Loss & Optimizers

Loss Factories

Loss is a BiFunction<Tensor, Tensor, Tensor>. Standard PyTorch-style losses are available as static factories.

import smile.deep.Loss;
import smile.deep.tensor.Tensor;

Loss l1   = Loss.l1();              // MAE
Loss mse  = Loss.mse();
Loss bce  = Loss.bce();             // expects sigmoid input
Loss bceL = Loss.bceWithLogits();
Loss ce   = Loss.crossEntropy();    // classification logits
Loss nll  = Loss.nll();             // expects log-softmax
Loss sl1  = Loss.smoothL1();
Loss hub  = Loss.huber(0.5);
Loss kl   = Loss.kl();
Loss hinge = Loss.hingeEmbedding();

// Tensor lossTensor = ce.apply(logits, labels);
// double lossVal = lossTensor.doubleValue();

// Three-argument helpers
// Tensor mr = Loss.marginRanking(input1, input2, target);
// Tensor tm = Loss.tripleMarginRanking(anchor, positive, negative);

Optimizers

Create an optimizer from a Model and learning rate. Each step is reset() → loss.backward() → step(). Note the capital A in Optimizer.Adam.

import smile.deep.Model;
import smile.deep.Optimizer;
import smile.deep.layer.Layer;
import smile.deep.layer.SequentialBlock;

Model model = new Model(new SequentialBlock(
    Layer.relu(784, 128),
    Layer.logSoftmax(128, 10)));

Optimizer sgd   = Optimizer.SGD(model, 0.01);
Optimizer adam  = Optimizer.Adam(model, 1e-3);
Optimizer adamW = Optimizer.AdamW(model, 1e-3);
Optimizer rms   = Optimizer.RMSprop(model, 1e-3);

// Per step inside a training loop:
// optimizer.reset();
// loss.backward();
// optimizer.step();

Model API

Compose a Model from a custom LayerBlock when you need architectures beyond a plain sequential stack. Register submodules with add(name, layer) so parameters are tracked for the optimizer.

import smile.deep.Model;
import smile.deep.layer.Conv2dLayer;
import smile.deep.layer.Layer;
import smile.deep.layer.LayerBlock;
import smile.deep.layer.LinearLayer;
import smile.deep.tensor.Tensor;

LayerBlock net = new LayerBlock("MyCNN") {
    private final Conv2dLayer conv1 = Layer.conv2d(1, 32, 3);
    private final LinearLayer fc = Layer.linear(32 * 13 * 13, 10);

    {
        add("conv1", conv1);
        add("fc", fc);
    }

    @Override
    public Tensor forward(Tensor input) {
        Tensor h = conv1.forward(input);
        h = h.relu_();
        h = Layer.maxPool2d(2).forward(h);
        h = h.view(h.size(0), -1);
        return fc.forward(h);
    }
};

Model model = new Model(net);

Model.train runs the full epoch loop: shuffle via the training Dataset iterator (each pass reshuffles), optimizer reset / forward / backward / step, optional validation metrics, and optional checkpoint path.

import smile.deep.Loss;
import smile.deep.Model;
import smile.deep.Optimizer;
import smile.deep.metric.Accuracy;

// model.train(
//     10,                 // epochs
//     optimizer,
//     Loss.crossEntropy(),
//     trainDataset,
//     testDataset,        // may be null
//     null,               // checkpoint path
//     new Accuracy()      // Metric...
// );

Metrics & Data

Metrics

Metrics implement update, compute, reset, and name. For multi-class scores, pass an Averaging strategy (Macro, Micro, Weighted).

Class	Description
`Accuracy`	# correct / total
`Precision`	TP / (TP + FP)
`Recall`	TP / (TP + FN)
`F1Score`	Harmonic mean of precision and recall

import smile.deep.metric.Accuracy;
import smile.deep.metric.Averaging;
import smile.deep.metric.F1Score;
import smile.deep.metric.Precision;
import smile.deep.metric.Recall;

Accuracy acc = new Accuracy();
Precision mp = new Precision(Averaging.Macro);
Recall mr = new Recall(Averaging.Micro);
F1Score wf1 = new F1Score(Averaging.Weighted);
F1Score binF1 = new F1Score();  // binary, threshold 0.5

// acc.update(output, target);
// double result = acc.compute();
// acc.reset();

Datasets & Sampling

Build mini-batch datasets from arrays with Dataset.of, or load IDX MNIST with Dataset.mnist. Dataset itself is Iterable<SampleBatch> — each iterator reshuffles. Every SampleBatch exposes data() and target() tensors.

import smile.deep.Dataset;
import smile.deep.SampleBatch;

float[][] features = /* ... */;
int[] labels = /* ... */;
Dataset ds = Dataset.of(features, labels, 64);
// also: Dataset.of(double[][], int[], batch)
// MNIST IDX: Dataset.mnist("data/mnist", true, 64);

for (SampleBatch batch : ds) {
    var x = batch.data();
    var y = batch.target();
    // train step...
    batch.close();
}

CUDA

smile.deep.CUDA reports device availability and memory. Use Device.CUDA() or Device.CUDA(index) to place tensors and models. Tensor.isBF16Supported() reports whether BF16 is performant on the current GPU (Ampere or newer).

import smile.deep.CUDA;
import smile.deep.tensor.Device;
import smile.deep.tensor.Tensor;

boolean available = CUDA.isAvailable();
long count = CUDA.deviceCount();
long total = CUDA.totalMemory(0);
long free  = CUDA.freeMemory(0);
long[] stats = CUDA.allocatorStats(0); // {allocated, reserved}

Device gpu = CUDA.device();       // Device.CUDA()
Device gpu1 = CUDA.device((byte) 1);

boolean bf16 = Tensor.isBF16Supported();

Computer Vision

The smile.vision package builds image-classification pipelines on the deep layer stack: transforms, folder datasets, ImageNet label maps, and EfficientNet-V2.

Transforms

import java.awt.image.BufferedImage;
import java.io.File;
import javax.imageio.ImageIO;
import smile.deep.tensor.Tensor;
import smile.vision.transform.ImageClassification;
import smile.vision.transform.Transform;

// ImageNet-style: resize shorter side → 384, center-crop 384×384, normalize
Transform t = Transform.classification(384, 384);

Transform custom = new ImageClassification(
        224, 256,
        new float[]{0.5f, 0.5f, 0.5f},
        new float[]{0.5f, 0.5f, 0.5f},
        java.awt.Image.SCALE_SMOOTH);

BufferedImage img = ImageIO.read(new File("cat.jpg"));
try (Tensor batch = t.forward(img)) {
    // batch shape: [1, 3, 384, 384]
}

float[] mean = Transform.DEFAULT_MEAN; // {0.485f, 0.456f, 0.406f}
float[] std  = Transform.DEFAULT_STD;  // {0.229f, 0.224f, 0.225f}

ImageDataset

ImageDataset reads a folder-per-class tree and prefetches batches on a background thread. The constructor is ImageDataset(String root, int batch, Transform transform, ToIntFunction<String> targetTransform).

import smile.vision.ImageDataset;
import smile.vision.ImageNet;
import smile.vision.transform.Transform;

Transform t = Transform.classification(224, 224);

// root/
//   dog/dog001.jpg
//   cat/cat001.jpg
ImageDataset ds = new ImageDataset(
        "data/train",
        32,
        t,
        ImageNet.folder2Target);

for (var batch : ds) {
    var images = batch.data();   // [32, 3, 224, 224]
    var labels = batch.target(); // [32]
}

ImageNet Labels

ImageNet is an interface of static fields — there is no ImageNet.INSTANCE. Use ImageNet.labels[i], ImageNet.folders, ImageNet.folder2Target, and ImageNet.label2Target.

import smile.vision.ImageNet;

String label = ImageNet.labels[282];
String folder = ImageNet.folders[282];
int id = ImageNet.folder2Target.applyAsInt("n02124075");
int fromLabel = ImageNet.label2Target.applyAsInt("tiger cat");

EfficientNet-V2

Factory	Variant	Input size	Parameters
`EfficientNet.V2S()`	EfficientNet-V2-S	384 × 384	~21 M
`EfficientNet.V2M()`	EfficientNet-V2-M	480 × 480	~54 M
`EfficientNet.V2L()`	EfficientNet-V2-L	480 × 480	~119 M

VisionModel.forward(BufferedImage...) applies the model’s associated transform automatically. output.topk(5) returns a Tuple2<Tensor, Tensor> of values and indices. Custom architectures can be built with MBConvConfig stages.

import java.io.File;
import javax.imageio.ImageIO;
import smile.deep.tensor.Device;
import smile.deep.tensor.Tensor;
import smile.vision.EfficientNet;
import smile.vision.ImageNet;
import smile.vision.layer.MBConvConfig;

var model = EfficientNet.V2S();
// or EfficientNet.V2S("checkpoints/efficientnet_v2_s.pt");
model.to(Device.preferredDevice());
model.eval();

var img = ImageIO.read(new File("dog.jpg"));
try (var guard = Tensor.noGradGuard();
     var output = model.forward(img)) {
    var top5 = output.topk(5);
    try (var indices = top5._2().to(Device.CPU())) {
        System.out.println(ImageNet.labels[indices.getInt(0, 0)]);
    }
}

// Custom stages (sketch)
MBConvConfig[] config = {
    MBConvConfig.FusedMBConv(1, 3, 1, 24, 24, 2),
    MBConvConfig.MBConv(4, 3, 2, 24, 48, 4),
};
// new EfficientNet(config, 0.2, 0.2, 1000, 1280, null);

End-to-End Examples

Gentle MNIST MLP

Pick a device with Device.preferredDevice(), set tensor defaults via Tensor.setDefaultOptions (there is no setDefaultDevice), wrap arrays with Dataset.of, and train with NLL loss over a log-softmax head.

import java.util.Map;
import org.apache.commons.csv.CSVFormat;
import smile.deep.*;
import smile.deep.layer.*;
import smile.deep.metric.*;
import smile.deep.tensor.*;
import smile.io.Read;

Device device = Device.preferredDevice();
Tensor.setDefaultOptions(new Tensor.Options().device(device));

Model net = new Model(new SequentialBlock(
        Layer.relu(784, 64, 0.5),
        Layer.relu(64, 32),
        Layer.logSoftmax(32, 10)));

System.out.println(net);
net.to(device);

CSVFormat format = CSVFormat.Builder.create().setDelimiter(' ').build();
double[][] x = Read.csv("data/mnist/mnist2500_X.txt", format).toArray();
int[] y = Read.csv("data/mnist/mnist2500_labels.txt", format)
        .column(0).toIntArray();
Dataset dataset = Dataset.of(x, y, 64);

Optimizer optimizer = Optimizer.SGD(net, 0.01);
Loss loss = Loss.nll();
net.train(100, optimizer, loss, dataset);

try (var guard = Tensor.noGradGuard()) {
    Map<String, Double> metrics = net.eval(dataset,
            new Accuracy(),
            new Precision(Averaging.Micro),
            new Precision(Averaging.Macro),
            new Recall(Averaging.Weighted));
    for (var entry : metrics.entrySet()) {
        System.out.format("Training %s = %.2f%%%n",
                entry.getKey(), 100 * entry.getValue());
    }
}

Manual Training Loop

When you need full control, drive epochs yourself with optimizer.reset(), backward(), and step(). Alternatively, call Model.train as in the LeNet example below.

import smile.deep.*;
import smile.deep.layer.*;
import smile.deep.tensor.Tensor;

SequentialBlock mlp = new SequentialBlock(
    Layer.relu(784, 256),
    Layer.relu(256, 128),
    Layer.logSoftmax(128, 10));
Model model = new Model(mlp);

Optimizer optimizer = Optimizer.Adam(model, 1e-3);
Loss criterion = Loss.nll();

// Dataset trainDataset = Dataset.of(x, y, 64);
for (int epoch = 0; epoch < 5; epoch++) {
    for (SampleBatch batch : trainDataset) {
        optimizer.reset();
        Tensor logp = model.forward(batch.data());
        Tensor loss = criterion.apply(logp, batch.target());
        loss.backward();
        optimizer.step();
        batch.close();
    }
}

LeNet with Model.train

A convolutional LeNet on MNIST. Load IDX files with Dataset.mnist(path, trainMode, batch) (shape [N, 1, 28, 28]), then call the high-level Model.train API.

import smile.deep.*;
import smile.deep.activation.LogSoftmax;
import smile.deep.activation.ReLU;
import smile.deep.layer.*;
import smile.deep.metric.Accuracy;
import smile.deep.tensor.Tensor;

Dataset mnistTrainDataset = Dataset.mnist("data/mnist", true, 64);
Dataset mnistTestDataset = Dataset.mnist("data/mnist", false, 64);

LayerBlock net = new LayerBlock("LeNet") {
    private final Conv2dLayer conv1 = Layer.conv2d(1, 6, 5);
    private final Conv2dLayer conv2 = Layer.conv2d(6, 16, 5);
    private final LinearLayer fc1 = Layer.linear(16 * 4 * 4, 120);
    private final LinearLayer fc2 = Layer.linear(120, 84);
    private final LinearLayer fc3 = Layer.linear(84, 10);
    private final MaxPool2dLayer pool = Layer.maxPool2d(2);

    {
        add("conv1", conv1); add("conv2", conv2);
        add("fc1", fc1); add("fc2", fc2); add("fc3", fc3);
    }

    @Override
    public Tensor forward(Tensor input) {
        Tensor x = pool.forward(
                new ReLU(true).forward(conv1.forward(input)));
        x = pool.forward(new ReLU(true).forward(conv2.forward(x)));
        x = x.view(x.size(0), -1);
        x = new ReLU(true).forward(fc1.forward(x));
        x = new ReLU(true).forward(fc2.forward(x));
        return new LogSoftmax().forward(fc3.forward(x));
    }
};

Model lenet = new Model(net);
lenet.train(
    10,
    Optimizer.SGD(lenet, 0.01, 0.9, 0.0, 0.0, false),
    Loss.nll(),
    mnistTrainDataset,
    mnistTestDataset,
    null,
    new Accuracy());

EfficientNet Pretrained Inference

Download weights such as efficientnet_v2_s.pt into model/EfficientNet/ (or pass a custom path). The first CUDA inference is often slower because the driver context and allocator warm up.

import java.io.File;
import javax.imageio.ImageIO;
import smile.deep.tensor.Device;
import smile.deep.tensor.Tensor;
import smile.vision.EfficientNet;
import smile.vision.ImageNet;

Device device = Device.preferredDevice();
var model = EfficientNet.V2S();
model.to(device);
model.eval();

var lenna = ImageIO.read(new File("data/image/Lenna.png"));
var panda = ImageIO.read(new File("data/image/panda.jpg"));

try (var guard = Tensor.noGradGuard()) {
    long start = System.nanoTime();
    var output = model.forward(panda);
    System.out.println("1st run: "
            + (System.nanoTime() - start) / 1_000_000 + "ms");

    start = System.nanoTime();
    output = model.forward(lenna, panda);
    System.out.println("2nd run: "
            + (System.nanoTime() - start) / 1_000_000 + "ms");

    var topk = output.topk(5);
    topk._2().to(Device.CPU());
    String[] images = {"Lenna", "Panda"};
    for (int i = 0; i < 2; i++) {
        System.out.println("======== " + images[i] + " ========");
        for (int j = 0; j < 5; j++) {
            System.out.println(
                    ImageNet.labels[topk._2().getInt(i, j)]);
        }
    }
}

ONNX Runtime

smile.onnx wraps the ONNX Runtime C API through Panama FFM so you can load models exported from PyTorch, TensorFlow, scikit-learn, XGBoost, and more. Generative ONNX (GenAI) is documented on llm.html, not here.

Overview

┌──────────────────────────────────────┐
│           Your Application           │
├──────────────────────────────────────┤
│              smile.onnx              │
│  InferenceSession  OrtValue  ...     │
├──────────────────────────────────────┤
│   smile.onnx.foreign  (Panama FFM)   │
├──────────────────────────────────────┤
│    onnxruntime shared library        │
└──────────────────────────────────────┘

Prerequisites

Requirement	Notes
Java 25+	Same FFM requirement as smile-deep
ONNX Runtime 1.18+	Shared library on PATH / LD_LIBRARY_PATH / DYLD_LIBRARY_PATH
JVM flag	`--enable-native-access=ALL-UNNAMED`

Package Structure

Class / Enum	Role
`InferenceSession`	Loads a model and runs inference
`OrtValue`	Tensor / sequence / map container
`SessionOptions`	Threads, graph opts, EPs
`RunOptions`	Per-run log tag, severity, cancel
`Environment`	Shared OrtEnv / thread pools
`ModelMetadata`	Producer, version, custom metadata
`NodeInfo`	Name + type for one I/O node
`TensorInfo`	Element type + shape
`ElementType`	ONNX tensor element types
`OnnxType`	TENSOR, SEQUENCE, MAP, …
`GraphOptimizationLevel`	Optimization depth
`ExecutionMode`	SEQUENTIAL or PARALLEL
`LoggingLevel`	ORT logging severity
`OnnxException`	Runtime exception on ORT errors

Quick Start

import java.util.Map;
import smile.onnx.InferenceSession;
import smile.onnx.OrtValue;

try (var session = InferenceSession.create("resnet50.onnx")) {
    session.inputNames().forEach(System.out::println);

    float[] pixels = /* NCHW float32, e.g. 1×3×224×224 */;
    long[] shape = {1, 3, 224, 224};

    try (OrtValue input = OrtValue.fromFloatArray(pixels, shape)) {
        OrtValue[] outputs = session.run(Map.of("data", input));
        float[] scores = outputs[0].toFloatArray();
        // argmax(scores) → predicted class
        for (OrtValue v : outputs) {
            v.close();
        }
    }
}

InferenceSession

Create sessions from a file path or byte array, optionally with SessionOptions. run accepts a Map<String, OrtValue> and is thread-safe for concurrent calls on the same session.

import smile.onnx.*;

InferenceSession session = InferenceSession.create("model.onnx");

try (var opts = new SessionOptions()) {
    opts.setIntraOpNumThreads(4)
        .setGraphOptimizationLevel(GraphOptimizationLevel.ENABLE_ALL);
    session = InferenceSession.create("model.onnx", opts);
}

// byte[] modelBytes = Files.readAllBytes(Path.of("model.onnx"));
// InferenceSession.fromBytes = InferenceSession.create(modelBytes);

// OrtValue[] all = session.run(inputs);
// OrtValue[] some = session.run(inputs, new String[]{"output_0"});
// try (var runOpts = new RunOptions()) {
//     runOpts.setLogTag("my-run");
//     session.run(inputs, outputNames, runOpts);
// }

OrtValue

import smile.onnx.OrtValue;

OrtValue v = OrtValue.fromFloatArray(
        new float[]{1f, 2f, 3f, 4f}, new long[]{2, 2});
OrtValue d = OrtValue.fromDoubleArray(
        new double[]{1.0, 2.0}, new long[]{2});
OrtValue i = OrtValue.fromIntArray(
        new int[]{10, 20}, new long[]{2});
OrtValue l = OrtValue.fromLongArray(
        new long[]{100L, 200L}, new long[]{2});

float[] floats = v.toFloatArray();
var ti = v.tensorInfo();
// ti.elementType(), ti.shape(), ti.rank(), ti.isDynamic()
v.close();

SessionOptions & Environment

import smile.onnx.*;

try (var opts = new SessionOptions()) {
    opts.setIntraOpNumThreads(4)
        .setInterOpNumThreads(2)
        .setGraphOptimizationLevel(GraphOptimizationLevel.ENABLE_ALL)
        .setExecutionMode(ExecutionMode.SEQUENTIAL)
        .enableCpuMemArena()
        .setLogId("my-session")
        .setLogSeverityLevel(LoggingLevel.WARNING)
        .appendCudaExecutionProvider(0);

    try (var session = InferenceSession.create("model.onnx", opts)) {
        // ...
    }
}

try (var env = new Environment(LoggingLevel.WARNING, "my-app");
     var s1 = env.createSession("model_a.onnx");
     var s2 = env.createSession("model_b.onnx")) {
    System.out.println(Environment.buildInfo());
    System.out.println(Environment.availableProviders());
}

Execution Providers

Append providers on SessionOptions before creating the session. If a provider is missing from the ORT build, creation fails — check Environment.availableProviders() first.

Provider helper	Typical use
`appendCudaExecutionProvider(deviceId)`	NVIDIA GPU inference
`appendTensorRTExecutionProvider(deviceId)`	TensorRT-accelerated GPU (when built into ORT)
CPU (default)	Always available; no append needed

Resource Management

InferenceSession, OrtValue, SessionOptions, RunOptions, and Environment all implement AutoCloseable. Prefer try-with-resources. Output arrays from session.run must be closed explicitly (the session does not own them).

Tutorial: ResNet-50

Load an image as NCHW float32 (resize, ImageNet mean/std), run the session, then softmax / argmax. Preprocessing details vary by export; keep the tensor layout consistent with the model’s input name and shape.

import java.util.Map;
import smile.onnx.*;

float[] pixels = preprocessImage("cat.jpg", 224, 224); // your NCHW code
long[] shape = {1, 3, 224, 224};

try (var session = InferenceSession.create("resnet50.onnx");
     OrtValue input = OrtValue.fromFloatArray(pixels, shape)) {
    System.out.println(session.inputInfos().getFirst());
    String name = session.inputNames().getFirst();
    OrtValue[] outputs = session.run(Map.of(name, input));
    try {
        float[] logits = outputs[0].toFloatArray();
        // softmax + argmax → class id
    } finally {
        for (OrtValue v : outputs) v.close();
    }
}

Tutorial: Shared Environment

import smile.onnx.*;

try (var env = new Environment(LoggingLevel.WARNING, "model-server");
     var opts = new SessionOptions()) {
    opts.setGraphOptimizationLevel(GraphOptimizationLevel.ENABLE_ALL)
        .setIntraOpNumThreads(4);

    try (InferenceSession a = env.createSession("detector.onnx", opts);
         InferenceSession b = env.createSession("classifier.onnx", opts)) {
        // Both sessions share OrtEnv thread pools
        System.out.println(Environment.availableProviders());
    }
}

Tutorial: CUDA Execution Provider

import java.util.List;
import java.util.Map;
import smile.onnx.*;

List<String> providers = Environment.availableProviders();

try (var opts = new SessionOptions()) {
    if (providers.contains("CUDAExecutionProvider")) {
        opts.appendCudaExecutionProvider(0);
    }
    opts.setGraphOptimizationLevel(GraphOptimizationLevel.ENABLE_ALL);

    try (var session = InferenceSession.create("resnet50.onnx", opts);
         OrtValue input = OrtValue.fromFloatArray(
                 new float[3 * 224 * 224], new long[]{1, 3, 224, 224})) {
        OrtValue[] outputs = session.run(
                Map.of(session.inputNames().getFirst(), input));
        for (OrtValue v : outputs) v.close();
    }
}

API Quick Reference

InferenceSession	Description
`create(path)`	Load from file (default options)
`create(path, opts)`	Load from file with options
`create(bytes)` / `create(bytes, opts)`	Load from byte array
`run(Map)`	Run all outputs
`run(Map, String[])`	Selected outputs
`run(Map, String[], RunOptions)`	With per-call options
`inputNames()` / `outputNames()`	I/O name lists
`inputInfos()` / `outputInfos()`	`List<NodeInfo>`
`metadata()`	`ModelMetadata`
`close()`	Release native session

OrtValue	Description
`fromFloatArray` / `fromDoubleArray`	FLOAT / DOUBLE tensors
`fromIntArray` / `fromLongArray`	INT32 / INT64
`fromByteArray` / `fromBooleanArray`	INT8 / BOOL
`toFloatArray()` …	Extract typed data
`tensorInfo()`	Shape + element type
`close()`	Release native value

SessionOptions	Description
`setIntraOpNumThreads` / `setInterOpNumThreads`	Thread pools
`setGraphOptimizationLevel`	DISABLE_ALL … ENABLE_ALL
`setExecutionMode`	SEQUENTIAL or PARALLEL
`appendCudaExecutionProvider`	Add CUDA EP
`enableProfiling` / `disableProfiling`	Chrome-trace profiling
`setOptimizedModelFilePath`	Persist optimized graph

For chat / generative ONNX workflows, see Large Language Models.

← Regression Time Series →