Class SparkDataFrames

java.lang.Object
smile.spark.SparkDataFrames
public final class SparkDataFrames extends Object
Converts an Apache Spark SQL Dataset<Row> to a SMILE DataFrame.

  • Method Summary

    Modifier and Type
    Method
    Description
    static Stream<Tuple>
    stream(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> df)
    Streams rows from a Spark Dataset as SMILE Tuples without collecting the entire dataset into a single list in memory.
    static DataFrame
    toSmile(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> df)
    Converts an Apache Spark Dataset of Rows into an in-memory SMILE DataFrame.

    Methods inherited from class Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Method Details

    • toSmile

      public static DataFrame toSmile(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> df)
      Converts an Apache Spark Dataset of Rows into an in-memory SMILE DataFrame.
      Parameters:
      df - the Spark DataFrame.
      Returns:
      the SMILE DataFrame.

    • stream

      public static Stream<Tuple> stream(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> df)
      Streams rows from a Spark Dataset as SMILE Tuples without collecting the entire dataset into a single list in memory.
      Parameters:
      df - the Spark DataFrame.
      Returns:
      a stream of SMILE Tuples.