Data Processing

Machine learning is all about building models from data. However, data scientists frequently talk about models and algorithms first, which very likely generates suboptimal results. The other approach is to play with the data first. Even simple statistics and plots can help us get feelings of data and problems, which more likely lead us to better modelling.

Features

A feature is an individual measurable property of an observation — also called an explanatory variable, predictor, or regressor. Choosing informative, discriminating features matters as much as choosing the algorithm. In SMILE, tabular features live in a DataFrame (or a plain double[][] / vector). For modeling, the Formula API selects and encodes columns; reusable numeric transforms such as standardization live under smile.feature.transform (see the Tutorial).

Feature engineering still needs domain knowledge: combine automated transforms with what you know about the problem. This page focuses on holding and shaping data; the Tutorial continues into Formula-based modeling.

Data Type

Statistically, attributes are qualitative (categories) or quantitative (counts and measurements), and may be measured on a nominal, ordinal, interval, or ratio scale. SMILE maps that vocabulary onto a typed schema:

DataTypes / StructType:

Physical column types (DoubleType, IntType, StringType, date/time, arrays, …). A DataFrame’s schema() is a StructType of StructFields.

Measure:

How values should be interpreted for learning and plotting. NominalScale and OrdinalScale attach category labels to integral codes; numeric columns are typically continuous (interval/ratio).

ValueVector.nominal / ordinal:

Build factorized categorical columns efficiently. Plain ValueVector.of(String…) keeps a StringVector for text-oriented work.

Many algorithms expect numeric matrices; trees and Formula-based workflows can consume nominal fields more directly. Dates are first-class for indexing and plots; strings are useful for NLP after further encoding. When you need an explicit CSV schema, you will see StructType and NominalScale again in the CSV section.

DataFrame

While some SMILE algorithms take simple double[] as input, we often use the encapsulation class DataFrame. DataFrame is a two-dimensional data structure like a table with rows and columns. Each column is a ValueVector that is a one-dimensional labeled abstraction to store a sequence of values having the same type. Columns in a DataFrame may have different data types. There are concrete subclasses of ValueVector for each primitive data type and generic object types. Creating a ValueVector by passing a list of values.

import java.time.*
import smile.data.*
import smile.data.vector.*

ValueVector.of("A", 1.0, 2.0, 3.0)
ValueVector.of("B", Instant.now())
ValueVector.nominal("C", "test", "train", "test", "train")
ValueVector.of("D",
    "this is a string vector",
    "Nominal/ordinal vectors store data as integers internally")
ObjectVector.of("E", Index.range(0, 4).toArray(), new int[]{3, 3, 3, 3})

A[1, 2, 3]

B[2025-01-18T23:08:54.375073600Z]

C[test, train, test, train]

D[this is a string vector, Nominal/ordinal vectors store data as integers internally]

E[[0, 1, 2, 3], [3, 3, 3, 3]]

Note that the nominal and ordinal methods factorize string values and store them as integral values internally, which are more efficient, compact, and friendly for machine learning algorithm. In contrast, the ValueVector.of(String...) method returns a StringVector that store string values as is, which is useful for text processing.

Creation

For illustration, create a DataFrame from a 2-dimensional array (default column names V1, V2, …) or from typed columns. Call schema() to inspect field names, types, and nullability.

Java 
var df = DataFrame.of(MathEx.randn(6, 4))
var mixed = new DataFrame(
        ValueVector.of("A", 1.0),
        ValueVector.of("B", LocalDate.parse("2013-01-02")),
        ValueVector.of("C", "foo"),
        ObjectVector.of("D", Index.range(0, 4).toArray()),
        ObjectVector.of("E", new int[]{3, 3, 3, 3})
)
mixed.schema()

+---+---------+---------+---------+---------+
|   |       V1|       V2|       V3|       V4|
+---+---------+---------+---------+---------+
|  0|-0.201469| 0.970363| 2.726932|-0.146014|
|  1| 1.872161| 0.495932| 0.553859|-0.028237|
|  2|-0.504866|-0.179409| 0.201377| 0.281267|
|  3| 0.894446| 0.791521| 0.053346| 0.213519|
|  4| 0.200011|-0.203736|-0.349196|-1.193759|
|  5|  1.52529|-1.407597|  1.16758| -1.78291|
+---+---------+---------+---------+---------+

+---+---+----------+---+------------+------------+
|   |  A|         B|  C|           D|           E|
+---+---+----------+---+------------+------------+
|  0|  1|2013-01-02|foo|[0, 1, 2, 3]|[3, 3, 3, 3]|
+---+---+----------+---+------------+------------+

{
  A: double NOT NULL,
  B: Date,
  C: String,
  D: int[],
  E: int[]
}

We can create a DataFrame with a collection of records or beans.

Java 
enum Gender {Male, Female}
record Person(String name, Gender gender, String state, LocalDate birthday, int age, Double salary) { }
List<Person> persons = new ArrayList<>();
persons.add(new Person("Alex", Gender.Male, "NY", LocalDate.of(1980, 10, 1), 38, 10000.));
persons.add(new Person("Bob", Gender.Male, "AZ", LocalDate.of(1995, 3, 4), 23, null));
persons.add(new Person("Jane", Gender.Female, "CA", LocalDate.of(1970, 3, 1), 48, 230000.));
persons.add(new Person("Amy", Gender.Female, "NY", LocalDate.of(2005, 12, 10), 13, null));
var df = DataFrame.of(Person.class, persons);

+---+----+------+-----+----------+---+------+
|   |name|gender|state|  birthday|age|salary|
+---+----+------+-----+----------+---+------+
|  0|Alex|  Male|   NY|1980-10-01| 38| 10000|
|  1| Bob|  Male|   AZ|1995-03-04| 23|  null|
|  2|Jane|Female|   CA|1970-03-01| 48|230000|
|  3| Amy|Female|   NY|2005-12-10| 13|  null|
+---+----+------+-----+----------+---+------+

In above example, the column 'state' is of string type. Apparently, it is a categorical variable. We can factorize such string columns to categorical values:

Java 
var df2 = df.factorize("state");
df.get(0, 2)
df2.get(0, 2)

+---+----+------+-----+----------+---+------+
|   |name|gender|state|  birthday|age|salary|
+---+----+------+-----+----------+---+------+
|  0|Alex|  Male|   NY|1980-10-01| 38| 10000|
|  1| Bob|  Male|   AZ|1995-03-04| 23|  null|
|  2|Jane|Female|   CA|1970-03-01| 48|230000|
|  3| Amy|Female|   NY|2005-12-10| 13|  null|
+---+----+------+-----+----------+---+------+

"NY"
2

On the surface, everything seems same. But the column 'state' is actually converted to integral values under the hood.

Most SMILE parsers return a DataFrame. The snippet below is a quick preview of ARFF I/O; see Data Import and Export for each format in detail.

import smile.data.*
import smile.datasets.*
import smile.io.*
var iris = Read.arff("data/weka/iris.arff")

+---+-----------+----------+-----------+----------+-----------+
|   |sepallength|sepalwidth|petallength|petalwidth|      class|
+---+-----------+----------+-----------+----------+-----------+
|  0|        5.1|       3.5|        1.4|       0.2|Iris-setosa|
|  1|        4.9|         3|        1.4|       0.2|Iris-setosa|
|  2|        4.7|       3.2|        1.3|       0.2|Iris-setosa|
|  3|        4.6|       3.1|        1.5|       0.2|Iris-setosa|
|  4|          5|       3.6|        1.4|       0.2|Iris-setosa|
|  5|        5.4|       3.9|        1.7|       0.4|Iris-setosa|
|  6|        4.6|       3.4|        1.4|       0.3|Iris-setosa|
|  7|          5|       3.4|        1.5|       0.2|Iris-setosa|
|  8|        4.4|       2.9|        1.4|       0.2|Iris-setosa|
|  9|        4.9|       3.1|        1.5|       0.1|Iris-setosa|
+---+-----------+----------+-----------+----------+-----------+
140 more rows...

The smile.datasets package also provides many public datasets as ready-made objects (no file path required).

Java 
var iris = new Iris().data() // use built-in dataset object

+---+-----------+----------+-----------+----------+-----------+
|   |sepallength|sepalwidth|petallength|petalwidth|      class|
+---+-----------+----------+-----------+----------+-----------+
|  0|        5.1|       3.5|        1.4|       0.2|Iris-setosa|
|  1|        4.9|         3|        1.4|       0.2|Iris-setosa|
|  2|        4.7|       3.2|        1.3|       0.2|Iris-setosa|
|  3|        4.6|       3.1|        1.5|       0.2|Iris-setosa|
|  4|          5|       3.6|        1.4|       0.2|Iris-setosa|
|  5|        5.4|       3.9|        1.7|       0.4|Iris-setosa|
|  6|        4.6|       3.4|        1.4|       0.3|Iris-setosa|
|  7|          5|       3.4|        1.5|       0.2|Iris-setosa|
|  8|        4.4|       2.9|        1.4|       0.2|Iris-setosa|
|  9|        4.9|       3.1|        1.5|       0.1|Iris-setosa|
+---+-----------+----------+-----------+----------+-----------+
140 more rows...

Indexing

We can set a row index to the data frames that must be of length the number of rows with no duplicates nor missing values. The row index serves as a unique identifier for each row in a DataFrame. It allows you to access rows efficiently with the loc() method. Compared with the normal ordinal index, the object-based row index often contains certain semantic meaning.

Java 
var df = DataFrame.of(MathEx.randn(6, 4))
var dates = Dates.range(LocalDate.of(2025,2,1), 6)
df = df.setIndex(dates)
df.loc(dates[1])
df.loc(dates[1], dates[2])

+----------+---------+---------+---------+---------+
|          |       V1|       V2|       V3|       V4|
+----------+---------+---------+---------+---------+
|2025-02-01|-0.201469| 0.970363| 2.726932|-0.146014|
|2025-02-02| 1.872161| 0.495932| 0.553859|-0.028237|
|2025-02-03|-0.504866|-0.179409| 0.201377| 0.281267|
|2025-02-04| 0.894446| 0.791521| 0.053346| 0.213519|
|2025-02-05| 0.200011|-0.203736|-0.349196|-1.193759|
|2025-02-06|  1.52529|-1.407597|  1.16758| -1.78291|
+----------+---------+---------+---------+---------+
{
  V1: -0.133296,
  V2: -2.461161,
  V3: 0.25011,
  V4: 1.132062
}

+----------+---------+---------+---------+--------+
|          |       V1|       V2|       V3|      V4|
+----------+---------+---------+---------+--------+
|2025-02-02|-0.133296|-2.461161|  0.25011|1.132062|
|2025-02-03|  0.25248|-0.063054|-1.128157| 0.37634|
+----------+---------+---------+---------+--------+

The row index may be an existing column, which will be removed in the resulting data frame.

Java 
var df = new DataFrame(
        ValueVector.of("A", 1.0),
        ValueVector.of("B", LocalDate.parse("2013-01-02")),
        ValueVector.of("C", "foo"),
        ObjectVector.of("D", Index.range(0, 4).toArray()),
        ObjectVector.of("E", new int[]{3, 3, 3, 3})
)
df = df.setIndex("B")

+----------+---+---+------------+------------+
|          |  A|  C|           D|           E|
+----------+---+---+------------+------------+
|2013-01-02|  1|foo|[0, 1, 2, 3]|[3, 3, 3, 3]|
+----------+---+---+------------+------------+

Viewing Data

By default, DataFrame.toString() returns a pretty print of top 10 rows. You may also use DataFrame.head() and DataFrame.tail() with a specified number to view the top and bottom rows of the frame respectively.

Java 
println(df.head(3))
println(df.tail(3))

+----------+---------+---------+---------+---------+
|          |       V1|       V2|       V3|       V4|
+----------+---------+---------+---------+---------+
|2025-02-01|-0.189794| 0.609897|-0.289189|-0.636956|
|2025-02-02|-0.133296|-2.461161|  0.25011| 1.132062|
|2025-02-03|  0.25248|-0.063054|-1.128157|  0.37634|
+----------+---------+---------+---------+---------+
3 more rows...

+----------+---------+---------+---------+--------+
|          |       V1|       V2|       V3|      V4|
+----------+---------+---------+---------+--------+
|2025-02-04|-1.391481| 1.398828| 0.294973|1.353308|
|2025-02-05| 2.812277|  0.82762|-0.294806|1.836631|
|2025-02-06| 1.091213|-0.190432| 1.963064|0.725228|
+----------+---------+---------+---------+--------+

The method describe() shows the data structure and statistic summary:

Java 
iris.describe()

+---+-----------+-----+--------------------+-----+--------+--------+---+---+---+---+---+
|   |     column| type|             measure|count|    mean|     std|min|25%|50%|75%|max|
+---+-----------+-----+--------------------+-----+--------+--------+---+---+---+---+---+
|  0|sepallength|float|                null|  150|5.843333|0.828066|4.3|5.1|5.8|6.4|7.9|
|  1| sepalwidth|float|                null|  150|   3.054|0.433594|  2|2.8|  3|3.3|4.4|
|  2|petallength|float|                null|  150|3.758667| 1.76442|  1|1.6|4.4|5.1|6.9|
|  3| petalwidth|float|                null|  150|1.198667|0.763161|0.1|0.3|1.3|1.8|2.5|
|  4|      class| byte|nominal[Iris-seto...|  150|     NaN|     NaN|  0|  0|  1|  2|  2|
+---+-----------+-----+--------------------+-----+--------+--------+---+---+---+---+---+

Selection

We can get a row with the array syntax or slice a subset of rows.

iris.get(0)
iris.get(Index.range(10, 20))

{
  sepallength: 5.1,
  sepalwidth: 3.5,
  petallength: 1.4,
  petalwidth: 0.2,
  class: Iris-setosa
}

+---+-----------+----------+-----------+----------+-----------+
|   |sepallength|sepalwidth|petallength|petalwidth|      class|
+---+-----------+----------+-----------+----------+-----------+
|  0|        5.4|       3.7|        1.5|       0.2|Iris-setosa|
|  1|        4.8|       3.4|        1.6|       0.2|Iris-setosa|
|  2|        4.8|         3|        1.4|       0.1|Iris-setosa|
|  3|        4.3|         3|        1.1|       0.1|Iris-setosa|
|  4|        5.8|         4|        1.2|       0.2|Iris-setosa|
|  5|        5.7|       4.4|        1.5|       0.4|Iris-setosa|
|  6|        5.4|       3.9|        1.3|       0.4|Iris-setosa|
|  7|        5.1|       3.5|        1.4|       0.3|Iris-setosa|
|  8|        5.7|       3.8|        1.7|       0.3|Iris-setosa|
|  9|        5.1|       3.8|        1.5|       0.3|Iris-setosa|
+---+-----------+----------+-----------+----------+-----------+

Similarly, we can refer a column by its name or select a few columns to create a new data frame.

iris.column("sepallength")
iris.select("sepallength", "sepalwidth")

sepallength[5.1, 4.9, 4.7, 4.6, 5, 5.4, 4.6, 5, 4.4, 4.9, ..., 140 more]

+---+-----------+----------+
|   |sepallength|sepalwidth|
+---+-----------+----------+
|  0|        5.1|       3.5|
|  1|        4.9|         3|
|  2|        4.7|       3.2|
|  3|        4.6|       3.1|
|  4|          5|       3.6|
|  5|        5.4|       3.9|
|  6|        4.6|       3.4|
|  7|          5|       3.4|
|  8|        4.4|       2.9|
|  9|        4.9|       3.1|
+---+-----------+----------+
140 more rows...

We can also select rows with boolean indexing. The below example uses isin() method for filtering:

Java 
iris.get(iris.column("class").isin("Iris-setosa", "Iris-virginica"))

+---+-----------+----------+-----------+----------+-----------+
|   |sepallength|sepalwidth|petallength|petalwidth|      class|
+---+-----------+----------+-----------+----------+-----------+
|  0|        5.1|       3.5|        1.4|       0.2|Iris-setosa|
|  1|        4.9|         3|        1.4|       0.2|Iris-setosa|
|  2|        4.7|       3.2|        1.3|       0.2|Iris-setosa|
|  3|        4.6|       3.1|        1.5|       0.2|Iris-setosa|
|  4|          5|       3.6|        1.4|       0.2|Iris-setosa|
|  5|        5.4|       3.9|        1.7|       0.4|Iris-setosa|
|  6|        4.6|       3.4|        1.4|       0.3|Iris-setosa|
|  7|          5|       3.4|        1.5|       0.2|Iris-setosa|
|  8|        4.4|       2.9|        1.4|       0.2|Iris-setosa|
|  9|        4.9|       3.1|        1.5|       0.1|Iris-setosa|
+---+-----------+----------+-----------+----------+-----------+
90 more rows...

Setting

Setting values by position:

Java 
iris.set(0, 0, 1.5)

Adding columns:

Java 
var df = DataFrame.of(MathEx.randn(150, 3))
iris.add(df.column("V1"), df.column("V3"))

Setting a column:

Java 
iris.set("V1", df.column("V3"))

Merge

The merge() method can combine data frames horizontally:

Java 
var df3 = iris.merge(df)

To concatenate data frames vertically, use concat():

Java 
var iris2 = iris.concat(iris)

To join two data frames on their index:

Java 
    var dates = Dates.range(LocalDate.of(2025,2,1), 6);
    var df1 = DataFrame.of(MathEx.randn(6, 4)).setIndex(dates);
    var df2 = DataFrame.of(MathEx.randn(6, 4)).setIndex(dates);
    var df = df1.join(df2);

Missing Data

For object column, null indicates missing data. For primitive columns, SMILE maintains a bit mask to indicate missing values. However, it is conventional that users use NaN to represent missing data for floating numeric types. One may use DataFrame.isNullAt(i, j) to check if a cell is null or NaN.

DataFrame.dropna() drops any rows that have null or missing value:

Java 
var df = DataFrame.of(MathEx.randn(6, 4))
df.set(0, 0, Double.NaN)
df.set(1, 3, Double.NaN)
df.dropna()

+---+---------+---------+---------+---------+
|   |       V1|       V2|       V3|       V4|
+---+---------+---------+---------+---------+
|  0|-0.504866|-0.179409| 0.201377| 0.281267|
|  1| 0.894446| 0.791521| 0.053346| 0.213519|
|  2| 0.200011|-0.203736|-0.349196|-1.193759|
|  3|  1.52529|-1.407597|  1.16758| -1.78291|
+---+---------+---------+---------+---------+

DataFrame.fillna() fills missing data in numeric columns:

Java 
df.fillna(100)

+---+---------+---------+---------+---------+
|   |       V1|       V2|       V3|       V4|
+---+---------+---------+---------+---------+
|  0|      100| 0.970363| 2.726932|-0.146014|
|  1| 1.872161| 0.495932| 0.553859|      100|
|  2|-0.504866|-0.179409| 0.201377| 0.281267|
|  3| 0.894446| 0.791521| 0.053346| 0.213519|
|  4| 0.200011|-0.203736|-0.349196|-1.193759|
|  5|  1.52529|-1.407597|  1.16758| -1.78291|
+---+---------+---------+---------+---------+

Operations

Advanced operations such as exists, forall, find, filter are also supported. In Java API, all these operations are on Stream. The corresponding methods are anyMatch, allMatch, findAny, and filter. The predicate of these functions expect a Tuple

iris.stream().anyMatch(row -> row.getDouble(0) > 4.5)
iris.stream().allMatch(row -> row.getDouble(0) < 10)
iris.stream().filter(row -> row.getByte("class") == 1).findAny()
iris.stream().filter(row -> row.getString("class").equals("Iris-versicolor")).findAny()
var stream = iris.stream().filter(row -> row.getDouble(1) > 3 && row.getByte("class") != 0)
DataFrame.of(iris.schema(), stream)

true

true

Optional[{
  sepallength: 6.2,
  sepalwidth: 2.9,
  petallength: 4.3,
  petalwidth: 1.3,
  class: Iris-versicolor
}]

Optional[{
  sepallength: 6.2,
  sepalwidth: 2.9,
  petallength: 4.3,
  petalwidth: 1.3,
  class: Iris-versicolor
}]

+-----------+----------+-----------+----------+---------------+
|sepallength|sepalwidth|petallength|petalwidth|          class|
+-----------+----------+-----------+----------+---------------+
|          7|       3.2|        4.7|       1.4|Iris-versicolor|
|        6.4|       3.2|        4.5|       1.5|Iris-versicolor|
|        6.9|       3.1|        4.9|       1.5|Iris-versicolor|
|        6.3|       3.3|        4.7|       1.6|Iris-versicolor|
|        6.7|       3.1|        4.4|       1.4|Iris-versicolor|
|          6|       3.4|        4.5|       1.6|Iris-versicolor|
|        6.7|       3.1|        4.7|       1.5|Iris-versicolor|
|        6.3|       3.3|          6|       2.5| Iris-virginica|
|        7.2|       3.6|        6.1|       2.5| Iris-virginica|
+-----------+----------+-----------+----------+---------------+
15 more rows...

For data wrangling, the most important functions of DataFrame are map and groupBy.

var x6 = iris.stream().map(row -> {
    var x = new double[6];
    for (int i = 0; i < 4; i++) x[i] = row.getDouble(i);
    x[4] = x[0] * x[1];
    x[5] = x[2] * x[3];
    return x;
})
x6.forEach(xi -> System.out.println(Arrays.toString(xi)))
[6.199999809265137, 2.9000000953674316, 4.300000190734863, 1.2999999523162842, 17.980000038146954, 5.590000042915335]
[7.300000190734863, 2.9000000953674316, 6.300000190734863, 1.7999999523162842, 21.170001249313373, 11.340000042915335]
[7.699999809265137, 3.0, 6.099999904632568, 2.299999952316284, 23.09999942779541, 14.029999489784245]
[6.699999809265137, 2.5, 5.800000190734863, 1.7999999523162842, 16.749999523162842, 10.440000066757193]
[7.199999809265137, 3.5999999046325684, 6.099999904632568, 2.5, 25.919998626709003, 15.249999761581421]
[6.5, 3.200000047683716, 5.099999904632568, 2.0, 20.800000309944153, 10.199999809265137]
[6.400000095367432, 2.700000047683716, 5.300000190734863, 1.899999976158142, 17.28000056266785, 10.070000236034389]
[5.699999809265137, 2.5999999046325684, 3.5, 1.0, 14.819998960495013, 3.5]
[4.599999904632568, 3.5999999046325684, 1.0, 0.20000000298023224, 16.55999921798707, 0.20000000298023224]
[5.400000095367432, 3.0, 4.5, 1.5, 16.200000286102295, 6.75]
[6.699999809265137, 3.0999999046325684, 4.400000095367432, 1.399999976158142, 20.76999876976015, 6.160000028610227]
[5.099999904632568, 3.799999952316284, 1.600000023841858, 0.20000000298023224, 19.379999394416814, 0.32000000953674324]
[5.599999904632568, 3.0, 4.5, 1.5, 16.799999713897705, 6.75]
[6.0, 3.4000000953674316, 4.5, 1.600000023841858, 20.40000057220459, 7.200000107288361]
[5.099999904632568, 3.299999952316284, 1.7000000476837158, 0.5, 16.82999944210053, 0.8500000238418579]
[5.5, 2.4000000953674316, 3.799999952316284, 1.100000023841858, 13.200000524520874, 4.1800000381469715]
[7.099999904632568, 3.0, 5.900000095367432, 2.0999999046325684, 21.299999713897705, 12.38999963760375]
[6.300000190734863, 3.4000000953674316, 5.599999904632568, 2.4000000953674316, 21.420001249313373, 13.440000305175772]
[5.099999904632568, 2.5, 3.0, 1.100000023841858, 12.749999761581421, 3.3000000715255737]
[6.400000095367432, 3.0999999046325684, 5.5, 1.7999999523162842, 19.839999685287466, 9.899999737739563]
[6.300000190734863, 2.9000000953674316, 5.599999904632568, 1.7999999523162842, 18.27000115394594, 10.079999561309819]
[5.5, 2.4000000953674316, 3.700000047683716, 1.0, 13.200000524520874, 3.700000047683716]
[6.5, 3.0, 5.800000190734863, 2.200000047683716, 19.5, 12.76000069618226]
[7.599999904632568, 3.0, 6.599999904632568, 2.0999999046325684, 22.799999713897705, 13.859999170303354]
[4.900000095367432, 2.5, 4.5, 1.7000000476837158, 12.250000238418579, 7.650000214576721]
[5.0, 2.299999952316284, 3.299999952316284, 1.0, 11.499999761581421, 3.299999952316284]
[5.599999904632568, 2.700000047683716, 4.199999809265137, 1.2999999523162842, 15.120000009536739, 5.45999955177308]
...

java.util.stream.ReferencePipeline$3@32eff876

The groupBy operation groups elements according to a classification function, and returning the results in a Map. The classification function maps elements to some key type K. The collector produces a map whose keys are the values resulting from applying the classification function to the input elements, and whose corresponding values are Lists containing the input elements which map to the associated key under the classification function.

iris.stream().collect(java.util.stream.Collectors.groupingBy(row -> row.getString("class")))

{Iris-versicolor=[{
  sepallength: 7,
  sepalwidth: 3.2,
  petallength: 4.7,
  petalwidth: 1.4,
  class: Iris-versicolor
}, {
  sepallength: 6.4,
  sepalwidth: 3.2,
  petallength: 4.5,
  petalwidth: 1.5,
  class: Iris-versicolor
}, {
  sepallength: 6.9,
  sepalwidth: 3.1,
  petallength: 4.9,
  petalwidth: 1.5,
  class: Iris-versicolor
}, {
  sepallength: 5.5,
  sepalwidth: 2.3,
  petallength: 4,
  petalwidth: 1.3,
  class: Iris-versicolor
}, {
  sepallength: 6.5,
  sepalwidth: 2.8,
  petallength: 4.6,
  petalwidth: 1.5,
  class: Iris-versicolor
}, {
  sepallength: 5.7,
  sepalwidth: 2.8,
  petallength: 4.5,
  petalwidth: 1.3,
  class: Iris-versicolor
},  ...  class: Iris-setosa
}, {
  sepallength: 4.6,
  sepalwidth: 3.2,
  petallength: 1.4,
  petalwidth: 0.2,
  class: Iris-setosa
}, {
  sepallength: 5.3,
  sepalwidth: 3.7,
  petallength: 1.5,
  petalwidth: 0.2,
  class: Iris-setosa
}, {
  sepallength: 5,
  sepalwidth: 3.3,
  petallength: 1.4,
  petalwidth: 0.2,
  class: Iris-setosa
}]}

SQL

While SMILE provides many imperative ways to manipulate DataFrames as shown above, it is often easier to do so with SQL.

Java 
SQL sql = new SQL();
sql.parquet("user", "data/kylo/userdata1.parquet");
sql.json("books", "data/kylo/books_array.json");
sql.csv("gdp", "data/regression/gdp.csv");
sql.csv("diabetes", "data/regression/diabetes.csv");

var tables = sql.tables();
var columns = sql.describe("user");

+----------+-------+
|TABLE_NAME|REMARKS|
+----------+-------+
|     books|   null|
|  diabetes|   null|
|       gdp|   null|
|      user|   null|
+----------+-------+

+-----------------+---------+-----------+
|      COLUMN_NAME|TYPE_NAME|IS_NULLABLE|
+-----------------+---------+-----------+
|registration_dttm|TIMESTAMP|        YES|
|               id|  INTEGER|        YES|
|       first_name|  VARCHAR|        YES|
|        last_name|  VARCHAR|        YES|
|            email|  VARCHAR|        YES|
|           gender|  VARCHAR|        YES|
|       ip_address|  VARCHAR|        YES|
|               cc|  VARCHAR|        YES|
|          country|  VARCHAR|        YES|
|        birthdate|  VARCHAR|        YES|
+-----------------+---------+-----------+
3 more rows...

In the above, we create a database and create four tables by loading parquet, json, and csv files. We also use the describe function to obtain the schema of the table user. With SQL, it is easy to filter data and the result is a DataFrame.

Java 
var user = sql.query("SELECT * FROM user WHERE country = 'China'");

+-------------------+---+----------+----------+--------------------+------+---------------+-------------------+-------+---------+---------+--------------------+--------+
|  registration_dttm| id|first_name| last_name|               email|gender|     ip_address|                 cc|country|birthdate|   salary|               title|comments|
+-------------------+---+----------+----------+--------------------+------+---------------+-------------------+-------+---------+---------+--------------------+--------+
|2016-02-03T00:36:21|  4|    Denise|     Riley|    driley3@gmpg.org|Female|  140.35.109.83|   3576031598965625|  China| 4/8/1997| 90263.05|Senior Cost Accou...|        |
|2016-02-03T18:04:34| 12|     Alice|     Berry|aberryb@wikipedia...|Female| 246.225.12.189|   4917830851454417|  China|8/12/1968| 22944.53|    Quality Engineer|        |
|2016-02-03T10:30:36| 20|   Rebecca|      Bell| rbellj@bandcamp.com|Female|172.215.104.127|                   |  China|         |137251.19|                    |        |
|2016-02-03T08:41:26| 27|     Henry|     Henry| hhenryq@godaddy.com|  Male| 191.88.236.116|4905730021217853521|  China|9/22/1995|284300.15|Nuclear Power Eng...|        |
|2016-02-03T20:46:39| 37|   Dorothy|     Gomez|dgomez10@jiathis.com|Female| 65.111.200.146| 493684876859391834|  China|         | 57194.86|                    |        |
|2016-02-03T08:34:26| 43|    Amanda|      Gray|  agray16@cdbaby.com|Female| 252.20.193.145|   3561501596653859|  China|8/28/1967|213410.26|Senior Quality En...|        |
|2016-02-03T00:05:52| 53|     Ralph|     Price|  rprice1g@tmall.com|  Male|   152.6.235.33|   4844227560658222|  China|8/26/1986| 168208.4|             Teacher|        |
|2016-02-03T16:03:13| 55|      Anna|Montgomery|amontgomery1i@goo...|Female|  80.111.141.47|   3586860392406446|  China| 9/6/1957|  92837.5|Software Test Eng...|     1E2|
|2016-02-03T00:33:25| 57|    Willie|    Palmer|wpalmer1k@t-onlin...|  Male| 164.107.46.161|   4026614769857244|  China|8/23/1986|184978.64|Environmental Spe...|        |
|2016-02-03T05:55:57| 58|    Arthur|     Berry|    aberry1l@unc.edu|  Male|    52.42.24.55|   3542761473624274|  China|         |144164.88|                    |        |
+-------------------+---+----------+----------+--------------------+------+---------------+-------------------+-------+---------+---------+--------------------+--------+
179 more rows...

Of course, join is very useful to prepare data from multiple sources. The result DataFrame may be feed to downstream machine learning algorithms.

Java 
var gdp = sql.query("SELECT * FROM user LEFT JOIN gdp ON user.country = gdp.Country");

+-------------------+---+----------+---------+--------------------+------+---------------+------------------+---------+----------+---------+--------------------+--------------------+---------+----------+-----+--------+
|  registration_dttm| id|first_name|last_name|               email|gender|     ip_address|                cc|  country| birthdate|   salary|               title|            comments|  Country|GDP Growth| Debt|Interest|
+-------------------+---+----------+---------+--------------------+------+---------------+------------------+---------+----------+---------+--------------------+--------------------+---------+----------+-----+--------+
|2016-02-03T07:55:29|  1|    Amanda|   Jordan|    ajordan0@com.com|Female|    1.197.201.2|  6759521864920116|Indonesia|  3/8/1971| 49756.53|    Internal Auditor|               1E+02|Indonesia|       6.5| 26.2|     7.7|
|2016-02-03T17:04:03|  2|    Albert|  Freeman|     afreeman1@is.gd|  Male| 218.111.175.34|                  |   Canada| 1/16/1968|150280.17|       Accountant IV|                    |   Canada|       2.5| 52.5|     9.5|
|2016-02-03T07:22:34|  6|   Kathryn|    White|  kwhite5@google.com|Female| 195.131.81.179|  3583136326049310|Indonesia| 2/25/1983| 69227.11|   Account Executive|                    |Indonesia|       6.5| 26.2|     7.7|
|2016-02-03T08:33:08|  7|    Samuel|   Holmes|sholmes6@foxnews.com|  Male| 232.234.81.197|  3582641366974690| Portugal|12/18/1987| 14247.62|Senior Financial ...|                    | Portugal|      -1.6| 92.5|     9.7|
|2016-02-03T18:29:47| 10|     Emily|  Stewart|estewart9@opensou...|Female| 143.28.251.245|  3574254110301671|  Nigeria| 1/28/1997| 27234.28|     Health Coach IV|                    |  Nigeria|       7.4|    3|     6.6|
|2016-02-03T08:53:23| 15|   Dorothy|   Hudson|dhudsone@blogger.com|Female|       8.59.7.0|  3542586858224170|    Japan|12/20/1989|157099.71|  Nurse Practicioner|        alert('hi...|    Japan|      -0.6|174.8|    15.7|
|2016-02-03T00:44:01| 16|     Bruce|   Willis|bwillisf@bluehost...|  Male|239.182.219.189|  3573030625927601|   Brazil|          |239100.65|                    |                    |   Brazil|       2.7| 52.8|    24.1|
|2016-02-03T16:44:24| 18|   Stephen|  Wallace|swallaceh@netvibe...|  Male|  152.49.213.62|  5433943468526428|  Ukraine| 1/15/1978|248877.99|Account Represent...|                    |  Ukraine|       5.2| 27.4|     5.2|
|2016-02-03T18:50:55| 23|   Gregory|   Barnes|  gbarnesm@google.ru|  Male| 220.22.114.145|  3538432455620641|  Tunisia| 1/23/1971|182233.49|Senior Sales Asso...|         사회과학원 어학연구소|  Tunisia|        -2|   44|     5.8|
|2016-02-03T08:02:34| 26|   Anthony| Lawrence|alawrencep@miitbe...|  Male| 121.211.242.99|564182969714151470|    Japan|12/10/1979|170085.81| Electrical Engineer|                    |    Japan|      -0.6|174.8|    15.7|
+-------------------+---+----------+---------+--------------------+------+---------------+------------------+---------+----------+---------+--------------------+--------------------+---------+----------+-----+--------+
990 more rows...

Sparse Dataset

The feature vectors could be very sparse. To save space, SparseDataset stores data in a list of lists (LIL) sparse matrix format. SparseDataset stores one list per row, where each entry stores a column index and value. Typically, these entries are kept sorted by column index for faster lookup.

SparseDataset is often used to construct the data matrix. Once the matrix is constructed, it is typically converted to a format, such as Harwell-Boeing column-compressed sparse matrix format, which is more efficient for matrix operations.

The class BinarySparseDataset is more efficient for binary sparse data. In BinarySparseDataset, each item is stored as an integer array, which are the indices of nonzero elements in ascending order.

Loading sparse files is covered under Data Import and Export: LibSVM, coordinate triple lists, and Harwell-Boeing.

To read and write files (CSV, Parquet, ARFF, JSON, sparse formats, and more), continue with Data Import and Export.