Contents

Logistic Regression

Logistic regression is a method for modeling the relationships between one or more explanatory variables and a categorical variable by expressing the posterior statistical distribution of the categorical variable via linear functions on observed data. If the categorical variable is binary, taking only two values, “0” and “1”, the logistic regression is simple, otherwise, it is multinomial.

Details

Given n feature vectors of n p-dimensional feature vectors a vector of class labels \(y = (y_1,\ldots{},y_n )\), where \(y_i \in \{0, 1, \ldots, K-1\}\) and \(K\) is the number of classes, describes the class to which the feature vector \(x_i\) belongs, the problem is to train a logistic regression model.

The logistic regression model is the set of vectors \(\beta =\left\{ {\beta }_{0}=\left({\beta }_{00}\dots {\beta }_{0p}\right), {..\beta }_{K-1}=\left({\beta }_{K-10}\dots {\beta }_{K-1p}\right)\right\}\) that gives the posterior probability

\[{P\left\{y=k|x\right\}= p}_{k}\left(x, \beta \right)=\mathrm{ }\frac{{e}^{{f}_{k}\left(x, \beta \right)}}{\sum _{i=0}^{K-1}{e}^{{f}_{i}\left(x, \beta \right)}}, \text{where} {f}_{k}\left(x, \beta \right)= {\beta }_{k0}+ \sum _{j=1}^{p}{\beta }_{kj}*{x}_{j}\]

for a given feature vector \(x = (x_1, \ldots, x_p)\) and class label \(y \in \{0, 1, \ldots, K - 1\}\) for each \(k = 0, \ldots, K-1\). See [Hastie2009].

If the categorical variable is binary, the model is defined as a single vector \({\beta }_{0}=\left({\beta }_{00}\dots {\beta }_{0p}\right)\) that determines the posterior probability

\[ \begin{align}\begin{aligned}P\left\{y=1|x\right\}= \mathrm{\sigma }\left(x,\mathrm{ }\mathrm{\beta }\right)=\frac{1}{1+ {e}^{-f\left(x, \beta \right)}}\\P\left\{y=0|x\right\}=1- P\left\{y=1|x\right\}\end{aligned}\end{align} \]

Training Stage

Training procedure is an iterative algorithm which minimizes objective function

\[L\left(\beta \right)=-\frac{1}{n}\sum _{i=1}^{n}\mathrm{log}{p}_{{y}_{i}}\left({x}_{i},\mathrm{ }\mathrm{\beta }\right)+\mathrm{ }{\lambda }_{1}\sum _{k=0}^{K-1}{\|{\beta }_{k}\|}_{L1}+ {\lambda }_{2}\sum _{k=0}^{K-1}{\|{\beta }_{k}\|}_{L2}\]

where the first term is the negative log-likelihood of conditional \(Y\) given \(X\), and the latter terms are regularization ones that penalize the complexity of the model (large \(\beta\) values), \(\lambda_1\) and \(\lambda_2\) are non-negative regularization parameters applied to L1 and L2 norm of vectors in \(\beta\).

For more details, see [Hastie2009], [Bishop2006].

For the objective function minimization the library supports the iterative algorithms defined by the interface of daal::algorithms::iterative_solver. See Iterative Solver.

Prediction Stage

Given logistic regression model and vectors \(x_1, \ldots, x_r\), the problem is to calculate the responses for those vectors, and their probabilities and logarithms of probabilities if required. The computation is based on formula (1) in multinomial case and on formula (2) in binary case.

Usage of Training Alternative

To build a Logistic Regression model using methods of the Model Builder class of Logistic Regression, complete the following steps:

  • Create a Logistic Regression model builder using a constructor with the required number of responses and features.

  • Use the setBeta method to add the set of pre-calculated coefficients to the model. Specify random access iterators to the first and the last element of the set of coefficients [ISO/IEC 14882:2011 §24.2.7]_.

    Note

    If your set of coefficients does not contain an intercept, interceptFlag is automatically set to False, and to True, otherwise.

  • Use the getModel method to get the trained Logistic Regression model.

  • Use the getStatus method to check the status of the model building process. If DAAL_NOTHROW_EXCEPTIONS macros is defined, the status report contains the list of errors that describe the problems API encountered (in case of API runtime failure).

Note

If after calling the getModel method you use the setBeta method to update coefficients, the initial model will be automatically updated with the new \(\beta\) coefficients.

Examples

Batch Processing

Logistic regression algorithm follows the general workflow described in Classification Usage Model.

Training

For a description of the input and output, refer to Classification Usage Model.

In addition to the parameters of classifier described in Classification Usage Model, the logistic regression batch training algorithm has the following parameters:

Training Parameters for Logistic Regression (Batch Processing)

Parameter

Default Value

Description

algorithmFPType

float

The floating-point type that the algorithm uses for intermediate computations. Can be float or double.

method

defaultDense

The computation method used by the logistic regression. The only training method supported so far is the default dense method.

nClasses

Not applicable

The number of classes. A required parameter.

interceptFlag

True

A flag that indicates a need to compute \(\theta_j\)

penaltyL1

\(0\)

L1 regularization coefficient

Note

L1 regularization is not supported on GPU.

penaltyL2

\(0\)

L2 regularization coefficient

optimizationSolver

SGD solver

All iterative solvers are available as optimization procedures to use at the training stage:

Prediction

For a description of the input, refer to Classification Usage Model.

At the prediction stage logistic regression batch algorithm has the following parameters:

Prediction Parameters for Logistic Regression (Batch Processing)

Parameter

Default Value

Description

algorithmFPType

float

The floating-point type that the algorithm uses for intermediate computations. Can be float or double.

method

defaultDense

The computation method used by logistic regression. The only prediction method supported so far is the default dense method.

nClasses

Not applicable

The number of classes. A required parameter.

resultsToCompute

computeClassesLabels

The 64-bit integer flag that specifies which extra characteristics of the logistic regression to compute.

Provide one of the following values to request a single characteristic or use bitwise OR to request a combination of the characteristics:

  • computeClassesLabels for prediction

  • computeClassesProbabilities for probabilities

  • computeClassesLogProbabilities for logProbabilities

Output

In addition to classifier output, logistic regression prediction calculates the result described below. Pass the Result ID as a parameter to the methods that access the results of your algorithm.

Prediction Output for Logistic Regression (Batch Processing)

Result ID

Result

probabilities

A numeric table of size \(n \times nClasses\) containing probabilities of classes computed when computeClassesProbabilities option is enabled.

logProbabilities

A numeric table of size \(n \times nClasses\) containing logarithms of classes’ probabilities computed when computeClassesLogProbabilities option is enabled.

Note

Note that:

  • If resultsToCompute does not contain computeClassesLabels, the prediction table is NULL.

  • If resultsToCompute does not contain computeClassesProbabilities, the probabilities table is NULL.

  • If resultsToCompute does not contain computeClassesLogProbabilities, the logProbabilities table is NULL.

  • By default, each numeric table of this result is an object of the HomogenNumericTable class, but you can define the result as an object of any class derived from NumericTable except for PackedSymmetricMatrix and PackedTriangularMatrix.

Examples

previous

Distributed Processing: Prediction of Ratings

next

Naïve Bayes Classifier