Machine learning 3-ridge regression, Lasso, variable selection technique

One . Ridge return

1.1 What is ridge regression

Ridge regression is a regression method of biased estimation specially used for collinear data analysis , It is actually an improved least square method , But it gives up the unbiasedness of the least squares , Loss of some information , Give up part of the accuracy to seek a regression equation that is less effective but more in line with the reality .

Here we introduce the regression coefficient formula of downhill regression ,B(k)=(X’X+kI)-1X’Y As an estimate of the regression coefficient , This value is more stable than the least square estimate . call B(k) Ridge estimation of regression coefficient . obviously , When k=0 when , be B(k) It becomes the least square estimation ; And when k→∞ when ,B(k) It tends to 0. therefore ,k The value should not be too large , We're going to let k Lower value .

1.2 Ridge trace figure

When there is no singularity , The ridge trace should be stable and gradually tend to 0
Observe the ridge estimation through the ridge trace map , You can determine which variables should be eliminated

1.3 Properties of ridge regression estimation

1.4 Ridge trace analysis

1.5 General selection principle of ridge parameters

choice k（ or lambda） value , To cause to
（1） The ridge estimation of each regression coefficient is basically stable ;
（2） The regression coefficient with unreasonable sign when using least square estimation , The sign of its ridge estimation becomes reasonable ;
（3） There is no absolute value of the regression coefficient that is not in line with the actual meaning ;
（4） The sum of squared residuals does not increase much .

1.6 Variance expansion factor method

1.7 use R Language carries on ridge regression

Code :

library(MASS)
longley
summary(fm1 <- lm(Employed ~ ., data = longley))

names(longley)[1] <- "y"
lm.ridge(y ~ ., longley)
plot(lm.ridge(y ~ ., longley, lambda = seq(0,0.1,0.001)))

select(lm.ridge(y ~ ., longley, lambda = seq(0,0.1,0.001)))

Two . Lasso

1.1 Lasso summary

The problems of ridge regression :

1. There are too many calculation methods for ridge parameters , The difference is too big
2. The variables were screened according to ridge trace map , Too random
3. Ridge regression regression regression model （ If there is no variable filter ） Include all variables

LASSO

1. Tibshirani(1996) Put forward Lasso(The Least Absolute Shrinkage and Selectionatoroperator) Algorithm

2. By constructing a first-order penalty function, a refined model is obtained ; By finalizing some indicators （ Variable ） The coefficient is zero （ The ridge regression estimation coefficient is equal to 0 There is little chance of death , Make it difficult to filter variables ）, Strong explanatory power

3. Good at processing data with multicollinearity , Not the same as ridge regression is biased estimation

1.2 Why? LASSO Can filter variables directly

1.3 LASSO vs Ridge return

1.4 A more general model

1.5 Elastic net

Zouand Hastie (2005) Put forward elasticnet

Reference resources :

1. http://www.dataguru.cn/article-4063-1.html
2. https://zhuanlan.zhihu.com/p/426162272