LASSO

LASSO is a technique that is applied to regression algorithms. It places a parametrizable upper bound on the sum of the coefficients that are each multiplied with the predictor variables, which prevents some coefficients emerging from the training phase that are many times greater than the others. This has two useful effects:

• regularization: it prevents individual predictor variables from having a disproportionately large effect on the result, which can often result from overlearning / overfitting;
• dimensionality reduction: in a situation where there are a large number of predictor variables, using LASSO with a sufficiently low upper bound parameter will lead to some predictor variables emerging with a zero coefficient, which effectively removes them from the model.

Although it is a useful technique in many situations, LASSO has the following disadvantages. If they are relevant to a use case, elastic net should be considered as an alternative:

• Where predictor variables are highly correlated, LASSO tends to select one variable and reject the others;
• Where there are a large number of predictor variables but only a small number of examples, LASSO is mathematically constrained to never select more predictor variables than the number of examples.

Least angle regression (LARS) can be regarded as a method with which to perform LASSO. The basic idea is similar to the now obsolete stepwise regression, but LARS produces more stable and usable results. Rather than adding and removing whole variables from the model, LARS adjusts the contributions of variables to the model by multiplying them with fractions. The procedure starts by finding the predictor variable that most closely correlates with the dependent variable. The contribution fraction for this predictor variable is gradually increased until the effect of the remaining fraction that has not yet been added is equal to the effect of the next best-correlated predictor variable. The procedure is then repeated for this second predictor variable, then for the third predictor variable, and so on.