Package-level declarations
Model validation and selection.
Model validation is the task of confirming that the outputs of a statistical model are acceptable with respect to the real data-generating process. A model can be validated only relative to some application area. A model that is valid for one application might be invalid for some other applications.
Types
Bootstrap validation runner.
Cross-validation runner.
Leave-one-out cross validation runner.
One-shot train/test evaluation runner.
Properties
Functions
Adjusted Rand Index assumes the generalized hyper-geometric distribution as the model of randomness. The adjusted Rand index has the maximum value 1, and its expected value is 0 in the case of random clusters. A larger adjusted Rand index means a higher agreement between two partitions. The adjusted Rand index is recommended for measuring agreement even when the partitions compared have different numbers of clusters.
The area under the curve (AUC). When using normalized units, the area under the curve is equal to the probability that a classifier will rank a randomly chosen positive instance higher than a randomly chosen negative one (assuming 'positive' ranks higher than 'negative').
Cross entropy generalizes the log loss metric to multiclass problems.
The F-score (or F-measure) considers both the precision and the recall of the test to compute the score. The precision p is the number of correct positive results divided by the number of all positive results, and the recall r is the number of correct positive results divided by the number of positive results that should have been returned.
Log loss is an evaluation metric for binary classifiers and it is sometimes the optimization objective as well in case of logistic regression and neural networks. Log Loss takes into account the uncertainty of the prediction based on how much it varies from the actual label. This provides a more nuanced view of the performance of the model. In general, minimizing Log Loss gives greater accuracy for the classifier. However, it is susceptible in case of imbalanced data.
Mean absolute deviation error.
MCC is a correlation coefficient between prediction and actual values. It is considered as a balanced measure for binary classification, even in unbalanced data sets. It varies between -1 and +1. 1 when there is perfect agreement between ground truth and prediction, -1 when there is a perfect disagreement between ground truth and predictions. MCC of 0 means the model is not better than random.
Mean squared error.
R2 coefficient of determination measures how well the regression line approximates the real data points. An R2 of 1.0 indicates that the regression line perfectly fits the data.
Rand index is defined as the number of pairs of objects that are either in the same group or in different groups in both partitions divided by the total number of pairs of objects. The Rand index lies between 0 and 1. When two partitions agree perfectly, the Rand index achieves the maximum value 1. A problem with Rand index is that the expected value of the Rand index between two random partitions is not a constant. This problem is corrected by the adjusted Rand index.
Root mean squared error.
Residual sum of squares.
Sensitivity or true positive rate (TPR) (also called hit rate, recall) is a statistical measure of the performance of a binary classification test. Sensitivity is the proportion of actual positives which are correctly identified as such.
Specificity or True Negative Rate is a statistical measure of the performance of a binary classification test. Specificity measures the proportion of negatives which are correctly identified.