Hyperparameter Optimization

A hyperparameter is chosen before training, unlike learned parameters such as weights, split thresholds, and centroids. Model hyperparameters determine model capacity or shape (for example tree depth or network topology); algorithm hyperparameters govern optimization, such as learning rate and mini-batch size.

smile.hpo.Hyperparameters is a lightweight fluent builder for these search spaces. Its grid and random strategies yield ordinary java.util.Properties, which can be passed directly to a model overload such as fit(formula, data, props).

Registering Parameters

Every overloaded add(name, ...) returns the same builder, so fixed values, discrete choices, and ranges can be chained. Names must be the property keys recognized by the target model.

add(name, value)
Fix an int, double, or String in every generated configuration.
add(name, values)
Choose from a non-empty int[], double[], or String[].
add(name, int start, int end)
Integer range with automatic step max(1, (end - start) / 10).
add(name, int start, int end, int step)
Integer range with an explicit positive grid step.
add(name, double start, double end[, double step])
Floating-point range with automatic or explicit grid step.
import smile.hpo.Hyperparameters;

var hp = new Hyperparameters()
    .add("smile.random_forest.trees", 100)
    .add("smile.random_forest.mtry", new int[] {2, 3, 4})
    .add("smile.random_forest.max_nodes", 100, 300, 50)
    .add("smile.svm.C", new double[] {0.01, 0.1, 1.0, 10.0});

For grid search, ranges include stepped values from start up to the largest value not greater than end. The step applies only to the grid. Random search ignores it and samples ranged integers and doubles uniformly from [start, end).

hp.grid() is a finite stream containing the Cartesian product of all registered choices. Count a grid before fitting and call grid() again afterward because count() consumes the stream. A few wide dimensions can create a combinatorial explosion.

import smile.classification.RandomForest;
import smile.datasets.ImageSegmentation;
import smile.hpo.Hyperparameters;
import smile.validation.metric.Accuracy;

var segment = new ImageSegmentation();
var hp = new Hyperparameters()
    .add("smile.random_forest.trees", 100)
    .add("smile.random_forest.mtry", new int[] {2, 3, 4})
    .add("smile.random_forest.max_nodes", 100, 300, 50);

System.out.println("Configurations: " + hp.grid().count());
var best = new double[] {0.0};
var bestProps = new java.util.Properties[1];
hp.grid().forEach(props -> {
    var model = RandomForest.fit(segment.formula(), segment.train(), props);
    double accuracy = Accuracy.of(segment.testy(), model.predict(segment.test()));
    if (accuracy > best[0]) {
        best[0] = accuracy;
        bestProps[0] = props;
    }
});
System.out.printf("Best accuracy: %.2f%%%n", 100.0 * best[0]);

Random search lets the evaluation budget stay independent of the full Cartesian product. random() is intentionally unbounded: always add limit(n), or use the finite convenience method random(n). Never call collect(), count(), or forEach() directly on the unbounded stream.

import smile.classification.RandomForest;
import smile.datasets.ImageSegmentation;
import smile.hpo.Hyperparameters;
import smile.validation.metric.Accuracy;

var segment = new ImageSegmentation();
var hp = new Hyperparameters()
    .add("smile.random_forest.trees", new int[] {100, 200, 500})
    .add("smile.random_forest.mtry", 2, 18)
    .add("smile.random_forest.max_nodes", 50, 500);

hp.random().limit(20).forEach(props -> {
    var model = RandomForest.fit(segment.formula(), segment.train(), props);
    double accuracy = Accuracy.of(segment.testy(), model.predict(segment.test()));
    System.out.printf("%.4f  %s%n", accuracy, props);
});
// Equivalent finite stream: hp.random(20)

Managing a Search Space

Adding the same name again overwrites its previous definition. remove(name) deletes one parameter, clear() deletes all parameters, and size() reports the current number. Both removal methods return the builder for chaining.

import smile.hpo.Hyperparameters;

var hp = new Hyperparameters()
    .add("smile.random_forest.trees", 100)
    .add("smile.random_forest.mtry", new int[] {2, 3});
hp.add("smile.random_forest.trees", new int[] {100, 200, 500});
System.out.println(hp.size()); // 2
hp.remove("smile.random_forest.mtry");
hp.clear();

Model Integration

Each model documents the property keys accepted by its fit(..., Properties) overload. The same builder can therefore tune any model without a model-specific search API. This short Gradient Tree Boost example uses the smile.gradient_boost.* keys.

import smile.classification.GradientTreeBoost;
import smile.datasets.Iris;
import smile.hpo.Hyperparameters;

var iris = new Iris();
var hp = new Hyperparameters()
    .add("smile.gradient_boost.trees", new int[] {100, 200})
    .add("smile.gradient_boost.max_depth", new int[] {4, 6, 8})
    .add("smile.gradient_boost.shrinkage", new double[] {0.05, 0.1, 0.2})
    .add("smile.gradient_boost.sampling_rate", new double[] {0.7, 0.8, 1.0});

hp.random(12).forEach(props ->
    GradientTreeBoost.fit(iris.formula(), iris.data(), props));

Cross-Validation Inside the Loop

A held-out test set is useful for a final unbiased estimate, but small training sets benefit from cross-validation during tuning. Evaluate each configuration with CrossValidation.classification and choose by the average fold metric; keep the final test set outside this loop.

import java.util.Properties;
import smile.classification.RandomForest;
import smile.datasets.Iris;
import smile.hpo.Hyperparameters;
import smile.validation.CrossValidation;

var iris = new Iris();
var hp = new Hyperparameters()
    .add("smile.random_forest.trees", new int[] {100, 200})
    .add("smile.random_forest.max_nodes", new int[] {10, 20, 40});
double best = 0.0;
Properties bestProps = null;

for (var props : (Iterable<Properties>) hp.grid()::iterator) {
    var cv = CrossValidation.classification(5, iris.formula(), iris.data(),
        (formula, data) -> RandomForest.fit(formula, data, props));
    double accuracy = cv.avg().accuracy();
    if (accuracy > best) { best = accuracy; bestProps = props; }
}
System.out.printf("Best CV accuracy: %.4f%n", best);

Bayesian Optimization

When evaluations are expensive, Bayesian optimization builds a Gaussian-process surrogate and chooses the next trial with an acquisition function. Pass a search space and an objective that maps each Properties configuration to a scalar score. Set maximize = false to minimize.

import smile.hpo.Hyperparameters;

var hp = new Hyperparameters()
    .add("x1", -5.0, 5.0)
    .add("x2", -5.0, 5.0);

var result = hp.bayes(props -> {
    double x1 = Double.parseDouble(props.getProperty("x1"));
    double x2 = Double.parseDouble(props.getProperty("x2"));
    return x1 * x1 + x2 * x2;
}, 10, false);

System.out.println(result);
System.out.println("Best value: " + result.bestValue());

Best Practices

  1. Start with a budgeted random search, then narrow a small grid around the promising region.
  2. Count grid configurations first, and always limit random streams with limit(n) or random(n).
  3. Represent multiplicative scales explicitly, for example new double[] {0.001, 0.01, 0.1, 1, 10, 100} for regularization C or learning rate.
  4. Use parallel() only after checking the trainer. Models such as random forest already parallelize internally; nested parallelism can oversubscribe resources or deadlock.
  5. Persist the winning Properties so training and deployment use the exact same configuration.
import java.io.FileOutputStream;
import java.util.Properties;

Properties bestProps = /* result of the search */ new Properties();
try (var out = new FileOutputStream("best.properties")) {
    bestProps.store(out, "Best hyperparameters");
}