Package smile.spark

package smile.spark
Integration between SMILE machine learning and Apache Spark.

This package bridges the local, in-memory SMILE data model with the distributed Spark SQL and Spark ML abstractions. It is organized around four capabilities:

  • DataFrame conversion — move data between a Spark SQL Dataset<Row> and a SMILE DataFrame in either direction, so data prepared in Spark can be fed to SMILE algorithms and SMILE results can be pushed back to Spark.
  • Spark ML pipeline integration — drop any SMILE classifier or regression model into a Spark ML Pipeline through the Estimator/Model pairs in this package.
  • Distributed hyperparameter optimization — evaluate a list of hyperparameter configurations in parallel across Spark executors.
  • Type mapping — translate Spark SQL types to and from SMILE data types, including nullability, arrays, structs, and Spark ML vectors.

Key classes

SparkDataFrames
Converts a Spark Dataset<Row> into a SMILE DataFrame, either by collecting it to the driver or by streaming it lazily as Tuples.
SmileDataFrames
Converts a SMILE DataFrame into a distributed Spark Dataset<Row>.
SparkRowTuple
Adapts a Spark Row to the SMILE Tuple interface, delegating field access to the underlying row.
DataTypeUtils
Bidirectional mapping between Spark SQL data types and SMILE data types, used to translate schemas during conversion.
SmileClassifier / SmileClassificationModel
A Spark ML Estimator/Model pair that trains and applies any SMILE Classifier from a lambda Trainer.
SmileRegressor / SmileRegressionModel
The regression counterpart, wrapping any SMILE Regression model as a Spark ML Regressor and its fitted model.
SparkHPO
Distributed cross-validation and train/test hyperparameter search for classification and regression, parallelized over Spark executors.

Quick start

// Spark DataFrame -> SMILE DataFrame
DataFrame smileDf = SparkDataFrames.toSmile(sparkDf);

// Wrap any SMILE classifier as a Spark ML estimator and fit it
SmileClassifier classifier = new SmileClassifier()
    .setTrainer((x, y) -> smile.classification.KNN.fit(x, y, 5))
    .setFeaturesCol("features")
    .setLabelCol("label");

SmileClassificationModel model = classifier.fit(trainingData);
Dataset<Row> predictions = model.transform(testData);

Dependencies

The module depends on smile-core and on the Spark libraries (spark-core, spark-sql, spark-mllib, and hadoop-common), which are provided and expected to be supplied by the Spark runtime environment. All operations require an active SparkSession.