Feature Engineering

Feature quality often matters more than the choice of learning algorithm. SMILE’s smile.feature.* packages cover scaling, encoding, dimensionality reduction, imputation, selection, and tree-ensemble SHAP.

This page follows the same playground conventions as Classification and Regression: Java / Scala / Kotlin tabs, runnable snippets, and expected outputs from unit tests where available.

SubpackagePurpose
smile.feature.transform Scale, standardize, or row-normalize numeric columns
smile.feature.extraction PCA family, random projection, text / categorical encoders
smile.feature.imputation Fill missing values before training
smile.feature.selection Rank or search for informative features
smile.feature.importance SHAP / TreeSHAP explanations

Column-wise transformers return InvertibleColumnTransform (extends Transform) and chain with Transform.pipeline(...). Deep manifold methods (UMAP, t-SNE, LLE, …) live on the Manifold Learning page; here we keep PCA / KernelPCA / GHA / RandomProjection as feature extractors.

Feature Transformation

Package smile.feature.transform: five column-wise invertible scalers plus a stateless row-wise Normalizer. Fit on training data, then apply to train/test (and invert when you need the original scale).

Scaler

Min–max scaling maps each column to [0, 1] using training min/max. Out-of-range values are clamped, so invert is lossy outside the training range. Prefer when algorithms want bounded inputs and outliers are mild.

import smile.datasets.Iris
import smile.feature.transform.Scaler

var iris = new Iris()
var scaler = Scaler.fit(iris.data())
var scaled = scaler.apply(iris.data())
var restored = scaler.invert(scaled)
scaled.getDouble(0, 0)

first petal length scaled ∈ [0, 1]

WinsorScaler

Outlier-robust min–max: default 5th–95th percentile bounds (via IQAgent), then scale to [0, 1]. Use when extreme tails would crush the range of ordinary values.

import smile.datasets.Iris
import smile.feature.transform.WinsorScaler

var iris = new Iris()
var t = WinsorScaler.fit(iris.data())                 // 5%–95%
var t2 = WinsorScaler.fit(iris.data(), 0.01, 0.99)    // custom
t.apply(iris.data())

MaxAbsScaler

Divide by max(|x|) → [−1, 1] without centering, so zeros stay zero. Good for sparse matrices and sparse-kernel SVM.

import smile.datasets.Iris
import smile.feature.transform.MaxAbsScaler

var t = MaxAbsScaler.fit(new Iris().data())
t.apply(new Iris().data())

Standardizer

Z-score: (x − μ) / σ with sample standard deviation. Prefer for distance-based and linear models when features are roughly Gaussian. Not robust to outliers — use RobustStandardizer then.

import smile.datasets.Iris
import smile.feature.transform.Standardizer

var t = Standardizer.fit(new Iris().data())
var z = t.apply(new Iris().data())
z.mean()   // ≈ 0 per column

RobustStandardizer

Median / IQR standardization. Same use cases as Standardizer when outliers inflate σ.

import smile.datasets.Iris
import smile.feature.transform.RobustStandardizer

RobustStandardizer.fit(new Iris().data()).apply(new Iris().data())

Normalizer

Row-wise unit norm (Norm.L1, L2, or L_INF). Stateless — no fit. Use for TF vectors and cosine-similarity settings.

import smile.datasets.Iris
import smile.feature.transform.Normalizer

var iris = new Iris()
var l2 = new Normalizer(Normalizer.Norm.L2, iris.data().names())
l2.apply(iris.data())

Transform Pipelines

Chain with Transform.pipeline(...) when stages are already fitted, or Transform.fit(data, trainer1, trainer2, …) when each stage must see the previous stage’s output. See also base/DATA_TRANSFORMS.md.

import smile.data.transform.Transform
import smile.datasets.Iris
import smile.feature.transform.MaxAbsScaler
import smile.feature.transform.Standardizer

var df = new Iris().data()
// Fit each stage on the previous stage's output:
Transform pipe = Transform.fit(df, Standardizer::fit, MaxAbsScaler::fit)
// Or compose already-fitted transforms: Transform.pipeline(t1, t2)
pipe.apply(df)

Feature Extraction

Package smile.feature.extraction: linear and nonlinear projections plus encoders that turn text / categoricals into numeric vectors.

Principal Component Analysis

Orthogonal projection onto directions of maximum variance. PCA.fit uses the covariance matrix; PCA.cor uses correlations when scales differ. Keep top components by count or cumulative variance.

import smile.datasets.PenDigits
import smile.feature.extraction.PCA
import smile.plot.swing.ScatterPlot
import smile.plot.swing.ScreePlot

var pen = new PenDigits()
double[][] x = pen.x()
int[] y = pen.y()

var pca = PCA.fit(x)
double[] vp = pca.varianceProportion().toArray(new double[0])
new ScreePlot(vp).figure().show()              // scree plot below

double[][] x3 = pca.getProjection(3).apply(x)
ScatterPlot.of(x3, y, '.').figure().show()     // 3D projection below

(opens scree plot, then 3D PenDigits projection)

The scree plot graphs variance proportion against component number. Look for an “elbow” where remaining components are small and roughly equal, then project into that many dimensions (here, three).

PCA scree plot of PenDigits
Scree plot of PenDigits eigenvalues — look for an elbow to choose how many components to keep.
PCA projection of PenDigits
PenDigits projected onto the first three principal components.

Probabilistic PCA

Latent variable model y ∼ W·x + μ + ε with isotropic noise. Fit with a fixed latent dimension k.

import smile.datasets.PenDigits
import smile.feature.extraction.ProbabilisticPCA

var pen = new PenDigits()
var ppca = ProbabilisticPCA.fit(pen.x(), 3)
ppca.variance()
ppca.apply(pen.x())
Probabilistic PCA on PenDigits
Probabilistic PCA latent projection of PenDigits.

Kernel PCA

Nonlinear PCA via a Mercer kernel. Options live on smile.manifold.KPCA.Options (component count and eigenvalue threshold).

Hyperparameters are fields of the nested KPCA.Options record. Pass an Options instance to fit, or a java.util.Properties map using the keys below (Options.toProperties() / Options.of(props) round-trip). Used by KernelPCA.fit(data, kernel, options).

d
smile.kpca.d
Number of principal components to keep.
threshold
smile.kpca.threshold
Drop components with eigenvalues below this threshold.
import smile.datasets.PenDigits
import smile.feature.extraction.KernelPCA
import smile.manifold.KPCA
import smile.math.kernel.GaussianKernel

var features = new PenDigits().data().drop("class")
var kpca = KernelPCA.fit(features, new GaussianKernel(45),
    new KPCA.Options(3))
kpca.apply(features)
Kernel PCA on PenDigits
Gaussian-kernel PCA projection of PenDigits.

Generalized Hebbian Algorithm

Online PCA for streaming / large data: update with centered samples, then apply. Requires a learning-rate TimeFunction.

import smile.datasets.PenDigits
import smile.feature.extraction.GHA
import smile.math.MathEx
import smile.util.function.TimeFunction

MathEx.setSeed(19650218)
double[][] x = new PenDigits().x()
var gha = new GHA(x[0].length, 3, TimeFunction.constant(0.00001))
for (double[] xi : x) gha.update(xi)
gha.apply(x)
GHA projection of PenDigits
Online GHA principal components on PenDigits.

Random Projection

Johnson–Lindenstrauss random maps — no training data. Dense Gaussian or sparse ternary projections preserve distances approximately.

import smile.feature.extraction.RandomProjection
import smile.math.MathEx

MathEx.setSeed(19650218)
var rp = RandomProjection.of(1000, 50)       // dense
var rps = RandomProjection.sparse(1000, 50)  // sparse
double[] y = rp.apply(new double[1000])
y.length

50

BagOfWords

Text column → dense count (or binary presence) vector over a fixed vocabulary. Prefer when word order does not matter.

import smile.feature.extraction.BagOfWords

String[] vocabulary = {"smile", "feature", "learning"}
var bow = new BagOfWords(s -> s.toLowerCase().split("\\s+"), vocabulary)
// binary presence: new BagOfWords(tokenizer, vocabulary, true)
bow.apply("SMILE feature learning")

BinaryEncoder, SparseEncoder, HashEncoder

BinaryEncoder one-hots categoricals to sparse int[] indices. SparseEncoder mixes numeric values with one-hot categoricals into a SparseArray. HashEncoder is the hashing trick for open vocabularies (optional alternating signs).

import smile.datasets.Default
import smile.feature.extraction.BinaryEncoder
import smile.feature.extraction.HashEncoder
import smile.feature.extraction.SparseEncoder

var data = new Default().data()
var binary = new BinaryEncoder(data.schema(), "student")
int[] idx = binary.apply(data.get(0))

var sparse = new SparseEncoder(data.schema())
var sa = sparse.apply(data.get(0))

var hash = new HashEncoder(s -> s.toLowerCase().split("\\s+"), 1 << 16)
hash.apply("student yes balance high")

DateFeature

Calendar / time fields come from Formula terms date(...) / DateFeature in smile.data.formula — not a separate smile.feature class. See Formula docs for year, month, dayOfWeek, etc.

import smile.data.formula.DateFeature
import smile.data.formula.Formula
import static smile.data.formula.Terms.date

// Extract calendar fields from a timestamp column "ts"
Formula formula = Formula.of("y",
    date("ts", DateFeature.YEAR, DateFeature.MONTH, DateFeature.DAY_OF_WEEK))

Missing Value Imputation

Real datasets often contain missing entries from nonresponse, dropout, sensor failure, or researcher error. Before imputing, ask why values are missing — the mechanism drives which methods are safe.

MCAR (missing completely at random): missingness is unrelated to any variable; complete-case analysis is unbiased but wasteful. MAR / covariate-dependent: missingness depends on observed variables, not on the missing values themselves — most modern imputers target this ignorable case. Non-ignorable (MNAR): missingness depends on the unobserved value (e.g. high incomes withheld); simple imputation can be badly biased and needs specialized modeling.

Imputation is not the only option: EM can estimate models with missingness directly, and some learners tolerate missing inputs when absence is structural rather than measurement noise. When you do impute, treat filled values as uncertain relative to truly observed ones.

Package smile.feature.imputation. DataFrame null and matrix NaN count as missing. Imputers implement Transform; check rows with SimpleImputer.hasMissing(tuple).

SimpleImputer

Column-wise constant fill — a fast baseline when missingness is roughly MCAR. fit(df) uses the median for numeric columns, the mode for boolean/nominal columns, and empty string for text. fit(df, lower, upper) imputes numerics with the mean of values between those percentiles (e.g. 0.0, 1.0 for the full mean). Or pass a custom Map of column → constant to the constructor.

import java.util.Map
import smile.datasets.Iris
import smile.feature.imputation.SimpleImputer

var df = new Iris().data()
var median = SimpleImputer.fit(df)                 // default: median
var mean = SimpleImputer.fit(df, 0.0, 1.0)         // mean in [0,1] percentiles
var custom = new SimpleImputer(Map.of("sepallength", 5.0))
median.apply(df)

KNNImputer

For a row with a missing attribute i, find k neighbors that have i observed and are closest on the other complete attributes (e.g. Euclidean), then fill with their (distance-weighted) average on i. Prefer when nearby rows share local structure. Constructor takes the reference frame + k (optional distance / columns); no separate training step beyond storing neighbors.

import smile.datasets.Iris
import smile.feature.imputation.KNNImputer

var df = new Iris().data()
var imputer = new KNNImputer(df, 5)  // Euclidean on all columns
imputer.apply(df)

KMedoidsImputer

k-medoids clustering (k-means with a real data point as the center). Missing entries in a row are copied from the nearest medoid. Fit with KMedoidsImputer.fit(df, distance, k) — useful for mixed-type data when a missing-tolerant distance is available.

import smile.datasets.Iris
import smile.feature.imputation.KMedoidsImputer
import smile.math.MathEx

var df = new Iris().data()
var imputer = KMedoidsImputer.fit(df,
    (x, y) -> MathEx.distanceWithMissingValues(x.toArray(), y.toArray()), 3)
imputer.apply(df)

SVDImputer

Low-rank matrix completion via SVD. Given A = U Σ VT, estimate missing entry (i, j) by regressing row i on the top k right singular vectors (excluding column j), then reconstructing from that linear combination. Because SVD needs a complete matrix, missing cells are first filled (e.g. column means); EM then iterates estimate → re-factorize until the change falls below a threshold.

Unlike the other imputers, SVDImputer.impute works on double[][] with NaN markers (not a DataFrame Transform). Best when numeric features are strongly correlated / approximately low-rank.

import smile.feature.imputation.SVDImputer
import smile.math.MathEx

MathEx.setSeed(19650218)
double[][] X = ... // may contain NaN
double[][] filled = SVDImputer.impute(X, 5, 100)

Feature Selection

Package smile.feature.selection: fast univariate filters plus GAFE as a wrapper search.

Hughes effect. With fixed sample size, adding irrelevant dimensions eventually hurts predictive accuracy (curse of dimensionality). Prefer ranking/selection or projection before distance-based models.

SignalNoiseRatio

Binary classification filter: |μ₁ − μ₂| / (σ₁ + σ₂) per feature (Golub-style). Higher is better.

import java.util.Comparator
import smile.datasets.BreastCancer
import smile.feature.selection.SignalNoiseRatio

var data = new BreastCancer().data()
var scores = SignalNoiseRatio.fit(data, "diagnosis")
java.util.Arrays.stream(scores)
    .sorted(Comparator.reverseOrder())
    .limit(5)
    .forEach(System.out::println)

SumSquaresRatio

Multi-class BSS/WSS ratio per feature. Higher values mean better class separability.

import smile.datasets.Iris
import smile.feature.selection.SumSquaresRatio

var scores = SumSquaresRatio.fit(new Iris().data(), "class")
java.util.Arrays.sort(scores)
for (var s : scores) System.out.println(s)
Iris petal width vs length
Iris petal measurements separate classes well — high BSS/WSS features.

FRegression

Univariate F-tests for regression responses (Pearson for numeric predictors; ANOVA for categoricals). Filter by pvalue().

import smile.datasets.Abalone
import smile.feature.selection.FRegression

var scores = FRegression.fit(new Abalone().train(), "rings")
java.util.Arrays.stream(scores)
    .filter(r -> r.pvalue() < 0.05)
    .map(FRegression::feature)
    .forEach(System.out::println)

InformationValue

Binary IV / WoE scoring (credit-style). IV < 0.02 useless; 0.1–0.3 medium; > 0.5 suspicious. InformationValue.toTransform(ivs) builds a WoE ColumnTransform.

import java.util.Comparator
import smile.datasets.Default
import smile.feature.selection.InformationValue

var data = new Default().data()
var ivs = InformationValue.fit(data, "default", 10)
java.util.Arrays.sort(ivs, Comparator.reverseOrder())
var woe = InformationValue.toTransform(ivs)
woe.apply(data)

GAFE

Genetic-algorithm wrapper search. Use GAFE.fitness(...) with a train/test metric and trainer, then apply(generations, population, length, fitness).

import smile.classification.DecisionTree
import smile.datasets.ImageSegmentation
import smile.feature.selection.GAFE
import smile.validation.metric.Accuracy

var segment = new ImageSegmentation()
var gafe = new GAFE()
var bits = gafe.apply(50, 10, segment.train().ncol() - 1,
    GAFE.fitness("class", segment.train(), segment.test(),
        new Accuracy(), DecisionTree::fit))
bits[bits.length - 1]  // elitism: inspect last generation

last-generation BitString (fitness + selected bits)

Feature Importance

Ensemble impurity importance is a fast global ranking from tree models. TreeSHAP gives exact local Shapley attributions for tree ensembles (RandomForest, GradientTreeBoost, AdaBoost, DecisionTree).

Ensemble Impurity Importance

import smile.classification.RandomForest
import smile.datasets.Iris

var iris = new Iris()
var rf = RandomForest.fit(iris.formula(), iris.data())
rf.importance()

TreeSHAP

Exact SHAP for trees via TreeSHAP. For classification, phi.length = p · k (features × classes); for regression, phi.length = p. shap(DataFrame) / shap(Stream) return mean absolute SHAP — a global importance proxy.

import smile.classification.RandomForest
import smile.datasets.Iris

var iris = new Iris()
var rf = RandomForest.fit(iris.formula(), iris.data())
double[] local = rf.shap(iris.data().get(0))   // signed, one row
double[] global = rf.shap(iris.data())         // mean |SHAP|
global

Choosing the Right Technique

SituationRecommended
Bounded inputs, mild outliersScaler
Bounded inputs, heavy outliersWinsorScaler
Sparse features, no centeringMaxAbsScaler
Gaussian-like, distance modelsStandardizer
Outliers + distance modelsRobustStandardizer
Row direction / cosine similarityNormalizer
Linear DR, moderate sizePCA / PCA.cor
Probabilistic latent modelProbabilisticPCA
Nonlinear structureKernelPCA
Streaming / very largeGHA
Fast approximate DRRandomProjection
Text bag-of-wordsBagOfWords / HashEncoder
Categorical → sparse indicesBinaryEncoder / SparseEncoder
MCAR baseline imputationSimpleImputer
Local structure imputationKNNImputer
Clustered mixed typesKMedoidsImputer
Low-rank numeric matrixSVDImputer
Multi-class filterSumSquaresRatio
Binary filterSignalNoiseRatio
Regression filterFRegression
Binary WoE / creditInformationValue
Wrapper searchGAFE
Tree ensemble explanationTreeSHAP (model.shap)