LIBLINEAR -- A Library for Large Linear Classification
LIBLINEAR -- A Library for Large Linear Classification
LIBLINEAR is a simple package for solving large-scale regularized linear classification, regression and outlier detection. It currently supports
To get started, please read the Quick Start'' section first. For developers, please check the Library Usage'' section to learn
how to integrate LIBLINEAR in your software.
There are some large data for which with/without nonlinear mappings gives similar performances. Without using kernels, one can efficiently train a much larger set via linear classification/regression. These data usually have a large number of features. Document classification is an example.
Warning: While generally liblinear is very fast, its default solver may be slow under certain situations (e.g., data not scaled or C is large). See Appendix B of our SVM guide about how to handle such cases. http://www.csie.ntu.edu.tw/~cjlin/papers/guide/guide.pdf
Warning: If you are a beginner and your data sets are not large, you should consider LIBSVM first.
LIBSVM page: http://www.csie.ntu.edu.tw/~cjlin/libsvm
See the section ``Installation'' for installing LIBLINEAR.
After installation, there are programs train' and predict' for
training and testing, respectively.
About the data format, please check the README file of LIBSVM. Note that feature index must start from 1 (but not 0).
A sample classification data included in this package is `heart_scale'.
Type train heart_scale', and the program will read the training data and output the model file heart_scale.model'. If you have a test
set called heart_scale.t, then type predict heart_scale.t heart_scale.model output' to see the prediction accuracy. The output'
file contains the predicted class labels.
For more information about train' and predict', see the sections
train' Usage and predict' Usage.
To obtain good performances, sometimes one needs to scale the
data. Please check the program svm-scale' of LIBSVM. For large and sparse data, use -l 0' to keep the sparsity.
On Unix systems, type make' to build the train', predict', and svm-scale' programs. Run them without arguments to show the usages.
On other systems, consult `Makefile' to build them (e.g., see 'Building Windows binaries' in this file).
This software uses some level-1 BLAS subroutines. The needed functions are included in this package. If a BLAS library is available on your machine, you may use it by modifying the Makefile: Unmark the following line
#LIBS = -lblasand mark
LIBS = blas/blas.aThe tool `svm-scale', borrowed from LIBSVM, is for scaling input data file.
Usage: train [options] training_set_file [model_file] options: -s type : set type of solver (default 1) for multi-class classification 0 -- L2-regularized logistic regression (primal) 1 -- L2-regularized L2-loss support vector classification (dual) 2 -- L2-regularized L2-loss support vector classification (primal) 3 -- L2-regularized L1-loss support vector classification (dual) 4 -- support vector classification by Crammer and Singer 5 -- L1-regularized L2-loss support vector classification 6 -- L1-regularized logistic regression 7 -- L2-regularized logistic regression (dual) for regression 11 -- L2-regularized L2-loss support vector regression (primal) 12 -- L2-regularized L2-loss support vector regression (dual) 13 -- L2-regularized L1-loss support vector regression (dual) for outlier detection 21 -- one-class support vector machine (dual) -c cost : set the parameter C (default 1) -p epsilon : set the epsilon in loss function of epsilon-SVR (default 0.1) -n nu : set the parameter nu of one-class SVM (default 0.5) -e epsilon : set tolerance of termination criterion -s 0 and 2 |f'(w)|2 = 0, instance x becomes [x; bias]; if = 0, w becomes [w; w{n+1}] and x becomes [x; bias]. For example, L2-regularized logistic regression (-s 0) becomes
min_w w^Tw/2 + (w_{n+1})^2/2 + C \sum log(1 + exp(-y_i [w; w_{n+1}]^T[x_i; bias]))
Some may prefer not having (w_{n+1})^2/2 (i.e., bias variable not regularized). For primal solvers (-s 0, 2, 5, 6, 11), we provide an option -R to remove (w_{n+1})^2/2. However, -R is generally not needed as for most data with/without (w_{n+1})^2/2 give similar performances.
The primal-dual relationship implies that -s 1 and -s 2 give the same model, -s 0 and -s 7 give the same, and -s 11 and -s 12 give the same.
We implement 1-vs-the rest multi-class strategy for classification. In training i vs. non_i, their C parameters are (weight from -wi)C and C, respectively. If there are only two classes, we train only one model. Thus weight1C vs. weight2*C is used. See examples below.
We also implement multi-class SVM by Crammer and Singer (-s 4):
min_{w_m, \xi_i} 0.5 \sum_m ||w_m||^2 + C \sum_i \xi_i s.t. w^T_{y_i} x_i - w^T_m x_i >= \e^m_i - \xi_i \forall m,i
where e^m_i = 0 if y_i = m, e^m_i = 1 if y_i != m,
Here we solve the dual problem:
min_{\alpha} 0.5 \sum_m ||w_m(\alpha)||^2 + \sum_i \sum_m e^m_i alpha^m_i s.t. \alpha^m_i train data_file
Train linear SVM with L2-loss function.
train -s 0 data_file
Train a logistic regression model.
train -s 21 -n 0.1 data_file
Train a linear one-class SVM which selects roughly 10% data as outliers.
train -v 5 -e 0.001 data_file
Do five-fold cross-validation using L2-loss SVM. Use a smaller stopping tolerance 0.001 than the default 0.1 if you want more accurate solutions.
train -C data_file ... Best C = 0.000488281 CV accuracy = 83.3333% train -c 0.000488281 data_file
Conduct cross validation many times by L2-loss SVM and find the parameter C which achieves the best cross validation accuracy. Then use the selected C to train the data for getting a model.
train -C -s 0 -v 3 -c 0.5 -e 0.0001 data_file
For parameter selection by -C, users can specify other solvers (currently -s 0, -s 2 and -s 11 are supported) and different number of CV folds. Further, users can use the -c option to specify the smallest C value of the search range. This option is useful when users want to rerun the parameter selection procedure from a specified C under a different setting, such as a stricter stopping tolerance -e 0.0001 in the above example. Similarly, for -s 11, users can use the -p option to specify the maximal p value of the search range.
train -c 10 -w1 2 -w2 5 -w3 2 four_class_data_file
Train four classifiers: positive negative Cp Cn class 1 class 2,3,4. 20 10 class 2 class 1,3,4. 50 10 class 3 class 1,2,4. 20 10 class 4 class 1,2,3. 10 10
train -c 10 -w3 1 -w2 5 two_class_data_file
If there are only two classes, we train ONE model. The C values for the two classes are 10 and 50.
predict -b 1 test_file data_file.model output_file
Output probability estimates (for logistic regression only).
These functions and structures are declared in the header file linear.h'. You can see train.c' and predict.c' for examples showing how to use them. We define LIBLINEAR_VERSION and declare extern int liblinear_version; '
in linear.h, so you can check the version number.
Function: model* train(const struct problem *prob, const struct parameter *param);
This function constructs and returns a linear classification or regression model according to the given training data and parameters.
struct problem describes the problem:
struct problem
{
int l, n;
double *y;
struct feature_node **x;
double bias;
}; where l' is the number of training data. If bias >= 0, we assume that one additional feature is added to the end of each data instance. n' is the number of feature (including the bias feature
if bias >= 0). y' is an array containing the target values. (integers in classification, real numbers in regression) And x' is an array
of pointers, each of which points to a sparse representation (array
of feature_node) of one training vector.
For example, if we have the following training data:
LABEL ATTR1 ATTR2 ATTR3 ATTR4 ATTR5
1 0 0.1 0.2 0 0 2 0 0.1 0.3 -1.2 0 1 0.4 0 0 0 0 2 0 0.1 0 1.4 0.5 3 -0.1 -0.2 0.1 1.1 0.1
and bias = 1, then the components of problem are:
l = 5 n = 6
y -> 1 2 1 2 3
x -> [ ] -> (2,0.1) (3,0.2) (6,1) (-1,?) [ ] -> (2,0.1) (3,0.3) (4,-1.2) (6,1) (-1,?) [ ] -> (1,0.4) (6,1) (-1,?) [ ] -> (2,0.1) (4,1.4) (5,0.5) (6,1) (-1,?) [ ] -> (1,-0.1) (2,-0.2) (3,0.1) (4,1.1) (5,0.1) (6,1) (-1,?)
struct parameter describes the parameters of a linear classification or regression model:
struct parameter
{
int solver_type;
/* these are for training only */
double eps; /* stopping tolerance */
double C;
double nu; /* one-class SVM only */
int nr_weight;
int *weight_label;
double* weight;
double p;
double *init_sol;
int regularize_bias;
bool w_recalc; /* for -s 1, 3; may be extended to -s 12, 13, 21 */
};solver_type can be one of L2R_LR, L2R_L2LOSS_SVC_DUAL, L2R_L2LOSS_SVC, L2R_L1LOSS_SVC_DUAL, MCSVM_CS, L1R_L2LOSS_SVC, L1R_LR, L2R_LR_DUAL, L2R_L2LOSS_SVR, L2R_L2LOSS_SVR_DUAL, L2R_L1LOSS_SVR_DUAL, ONECLASS_SVM. for classification L2R_LR L2-regularized logistic regression (primal) L2R_L2LOSS_SVC_DUAL L2-regularized L2-loss support vector classification (dual) L2R_L2LOSS_SVC L2-regularized L2-loss support vector classification (primal) L2R_L1LOSS_SVC_DUAL L2-regularized L1-loss support vector classification (dual) MCSVM_CS support vector classification by Crammer and Singer L1R_L2LOSS_SVC L1-regularized L2-loss support vector classification L1R_LR L1-regularized logistic regression L2R_LR_DUAL L2-regularized logistic regression (dual) for regression L2R_L2LOSS_SVR L2-regularized L2-loss support vector regression (primal) L2R_L2LOSS_SVR_DUAL L2-regularized L2-loss support vector regression (dual) L2R_L1LOSS_SVR_DUAL L2-regularized L1-loss support vector regression (dual) for outlier detection ONECLASS_SVM one-class support vector machine (dual)
C is the cost of constraints violation. p is the sensitiveness of loss of support vector regression. nu in ONECLASS_SVM approximates the fraction of data as outliers. eps is the stopping criterion.
nr_weight, weight_label, and weight are used to change the penalty for some classes (If the weight for a class is not changed, it is set to 1). This is useful for training classifier using unbalanced input data or with asymmetr
No open issues yet, or sync has not completed.