Class SparkHPO

java.lang.Object

smile.spark.SparkHPO

public final class SparkHPO
extends Object

Distributed hyperparameter optimization (HPO) across an Apache Spark cluster.

The training data is broadcast once to executors, and hyperparameter configurations are evaluated in parallel using either k-fold cross validation or a held-out test split.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static interface	SparkHPO.ClassificationTrainer<T, M extends Classifier<T>>	Functional interface for training a parameterized classifier on array samples.
static interface	SparkHPO.DataFrameClassificationTrainer<M extends DataFrameClassifier>	Functional interface for training a parameterized DataFrame classifier.
static interface	SparkHPO.DataFrameRegressionTrainer<M extends DataFrameRegression>	Functional interface for training a parameterized DataFrame regression model.
static interface	SparkHPO.RegressionTrainer<T, M extends Regression<T>>	Functional interface for training a parameterized regression model on array samples.

Method Summary

Modifier and Type	Method	Description
static <M extends DataFrameClassifier> List<ClassificationValidations<M>>	classification(org.apache.spark.sql.SparkSession spark, int k, Formula formula, DataFrame data, List<Properties> configurations, SparkHPO.DataFrameClassificationTrainer<M> trainer)	Distributed cross-validation for classification on DataFrame.
static <T, M extends Classifier<T>> List<ClassificationValidations<M>>	classification(org.apache.spark.sql.SparkSession spark, int k, T[] x, int[] y, List<Properties> configurations, SparkHPO.ClassificationTrainer<T,M> trainer)	Distributed cross-validation for classification on array data.
static <M extends DataFrameClassifier> List<ClassificationValidation<M>>	classification(org.apache.spark.sql.SparkSession spark, Formula formula, DataFrame train, DataFrame test, List<Properties> configurations, SparkHPO.DataFrameClassificationTrainer<M> trainer)	Distributed train/test validation for classification on DataFrame.
static <T, M extends Classifier<T>> List<ClassificationValidation<M>>	classification(org.apache.spark.sql.SparkSession spark, T[] x, int[] y, T[] testx, int[] testy, List<Properties> configurations, SparkHPO.ClassificationTrainer<T,M> trainer)	Distributed train/test validation for classification on array data.
static <M extends DataFrameRegression> List<RegressionValidations<M>>	regression(org.apache.spark.sql.SparkSession spark, int k, Formula formula, DataFrame data, List<Properties> configurations, SparkHPO.DataFrameRegressionTrainer<M> trainer)	Distributed cross-validation for regression on DataFrame.
static <T, M extends Regression<T>> List<RegressionValidations<M>>	regression(org.apache.spark.sql.SparkSession spark, int k, T[] x, double[] y, List<Properties> configurations, SparkHPO.RegressionTrainer<T,M> trainer)	Distributed cross-validation for regression on array data.
static <M extends DataFrameRegression> List<RegressionValidation<M>>	regression(org.apache.spark.sql.SparkSession spark, Formula formula, DataFrame train, DataFrame test, List<Properties> configurations, SparkHPO.DataFrameRegressionTrainer<M> trainer)	Distributed train/test validation for regression on DataFrame.
static <T, M extends Regression<T>> List<RegressionValidation<M>>	regression(org.apache.spark.sql.SparkSession spark, T[] x, double[] y, T[] testx, double[] testy, List<Properties> configurations, SparkHPO.RegressionTrainer<T,M> trainer)	Distributed train/test validation for regression on array data.

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

classification

public static <T, M extends Classifier<T>>
List<ClassificationValidations<M>> classification(org.apache.spark.sql.SparkSession spark,
 int k,
 T[] x,
 int[] y,
 List<Properties> configurations,
 SparkHPO.ClassificationTrainer<T,M> trainer)

Distributed cross-validation for classification on array data.

Type Parameters:

T - the sample type.

M - the model type.

Parameters:

spark - the Spark session.

k - the number of folds.

x - the sample array.

y - the label array.

configurations - the list of hyperparameter configurations.

trainer - the trainer function.

Returns:

cross-validation results per configuration.

classification

public static <M extends DataFrameClassifier>
List<ClassificationValidations<M>> classification(org.apache.spark.sql.SparkSession spark,
 int k,
 Formula formula,
 DataFrame data,
 List<Properties> configurations,
 SparkHPO.DataFrameClassificationTrainer<M> trainer)

Distributed cross-validation for classification on DataFrame.

Type Parameters:

M - the model type.

Parameters:

spark - the Spark session.

k - the number of folds.

formula - the model formula.

data - the DataFrame.

configurations - the list of hyperparameter configurations.

trainer - the trainer function.

Returns:

cross-validation results per configuration.

classification

public static <T, M extends Classifier<T>>
List<ClassificationValidation<M>> classification(org.apache.spark.sql.SparkSession spark,
 T[] x,
 int[] y,
 T[] testx,
 int[] testy,
 List<Properties> configurations,
 SparkHPO.ClassificationTrainer<T,M> trainer)

Distributed train/test validation for classification on array data.

Type Parameters:

T - the sample type.

M - the model type.

Parameters:

spark - the Spark session.

x - training samples.

y - training labels.

testx - test samples.

testy - test labels.

configurations - the list of hyperparameter configurations.

trainer - the trainer function.

Returns:

validation results per configuration.

classification

public static <M extends DataFrameClassifier>
List<ClassificationValidation<M>> classification(org.apache.spark.sql.SparkSession spark,
 Formula formula,
 DataFrame train,
 DataFrame test,
 List<Properties> configurations,
 SparkHPO.DataFrameClassificationTrainer<M> trainer)

Distributed train/test validation for classification on DataFrame.

Type Parameters:

M - the model type.

Parameters:

spark - the Spark session.

formula - the model formula.

train - the training DataFrame.

test - the validation DataFrame.

configurations - the list of hyperparameter configurations.

trainer - the trainer function.

Returns:

validation results per configuration.

regression

public static <T, M extends Regression<T>>
List<RegressionValidations<M>> regression(org.apache.spark.sql.SparkSession spark,
 int k,
 T[] x,
 double[] y,
 List<Properties> configurations,
 SparkHPO.RegressionTrainer<T,M> trainer)

Distributed cross-validation for regression on array data.

Type Parameters:

T - the sample type.

M - the model type.

Parameters:

spark - the Spark session.

k - the number of folds.

x - the sample array.

y - the continuous target values.

configurations - the list of hyperparameter configurations.

trainer - the trainer function.

Returns:

cross-validation results per configuration.

regression

public static <M extends DataFrameRegression>
List<RegressionValidations<M>> regression(org.apache.spark.sql.SparkSession spark,
 int k,
 Formula formula,
 DataFrame data,
 List<Properties> configurations,
 SparkHPO.DataFrameRegressionTrainer<M> trainer)

Distributed cross-validation for regression on DataFrame.

Type Parameters:

M - the model type.

Parameters:

spark - the Spark session.

k - the number of folds.

formula - the model formula.

data - the DataFrame.

configurations - the list of hyperparameter configurations.

trainer - the trainer function.

Returns:

cross-validation results per configuration.

regression

public static <T, M extends Regression<T>>
List<RegressionValidation<M>> regression(org.apache.spark.sql.SparkSession spark,
 T[] x,
 double[] y,
 T[] testx,
 double[] testy,
 List<Properties> configurations,
 SparkHPO.RegressionTrainer<T,M> trainer)

Distributed train/test validation for regression on array data.

Type Parameters:

T - the sample type.

M - the model type.

Parameters:

spark - the Spark session.

x - training samples.

y - training continuous target values.

testx - test samples.

testy - test targets.

configurations - the list of hyperparameter configurations.

trainer - the trainer function.

Returns:

validation results per configuration.

regression

public static <M extends DataFrameRegression>
List<RegressionValidation<M>> regression(org.apache.spark.sql.SparkSession spark,
 Formula formula,
 DataFrame train,
 DataFrame test,
 List<Properties> configurations,
 SparkHPO.DataFrameRegressionTrainer<M> trainer)

Distributed train/test validation for regression on DataFrame.

Type Parameters:

M - the model type.

Parameters:

spark - the Spark session.

formula - the model formula.

train - the training DataFrame.

test - the validation DataFrame.

configurations - the list of hyperparameter configurations.

trainer - the trainer function.

Returns:

validation results per configuration.