Chapter III summary of linear model

Catalog

The third chapter   Summary of linear models

 1. The basic form of linear model

Scalar form ：

  Vector form ：

  Generalized linear model ：

  2. Linear regression

Univariate linear regression （ Only one feature x）：

Multiple linear regression ：

3. Log probability regression -- Also known as logistic regression , To solve the problem of classification

Logarithmic probability function （ For the rate function ）：

  Logarithmic probability ：

  Loss function ：

Information entropy -- Measure the uncertainty of things ：

  Relative entropy ：

Cross entropy ：

4. Linear discriminant analysis （LDA）

Generalized eigenvalue ;

Generalized Rayleigh quotient ：

5. take LDA Extended to multi classification tasks , The basic idea of multi classification learning is to disassemble the multi classification task into several binary classification tasks .

 6. Category imbalance

---

The third chapter   Summary of linear models

Linear model is a function of prediction learned in the form of linear combination . The main content of this chapter is the basic form of linear model 、 Linear regression 、 Log probability regression 、 Linear discriminant analysis 、 Multi category learning 、 Category imbalance .

 1. The basic form of linear model

Scalar form ：

  Vector form ：

  Generalized linear model ：

  2. Linear regression

Univariate linear regression （ Only one feature x）：

Multiple linear regression ：

  Be careful ：1. The loss function is obtained by least square method or maximum likelihood estimation ;2. The problem becomes to find the maximum value of convex function ;3. A positive semidefinite of a Hessian matrix is a convex function ;4. Matrix derivation is generally the denominator layout , The default matrix is column vector ;5. When X The dimension of is larger than the number of samples , Can cause w More solutions , Analogy linear equation has too many dependent variables , There will be multiple solutions .

3. Log probability regression -- Also known as logistic regression , be used for Solve the problem of classification

Logarithmic probability function （ For the rate function ）：

  Logarithmic probability ：

  Loss function ：

  The loss function can also be obtained according to information theory ：

Information entropy -- Measure the uncertainty of things ：

  Relative entropy ：

Cross entropy ：

  Cross entropy is used to measure the difference between two distributions , Typically, the difference between the ideal distribution and the predicted distribution .

4. Linear discriminant analysis （LDA）

The basic idea ： There is a straight line , Make the projection of similar samples as close as possible , The projection of heterogeneous samples should be as far away as possible .

LDA It is also often regarded as a classical supervised dimensionality reduction technology .

solve w The process uses generalized eigenvalues and generalized Rayleigh quotients .

Generalized eigenvalue ;

by A be relative to B Generalized eigenvalue of ,x by A be relative to B Is a generalized eigenvalue Eigenvector of .

Generalized Rayleigh quotient ：

5. take LDA Extended to multi classification tasks , The basic idea of multi classification learning is to disassemble the multi classification task into several binary classification tasks .

 6. Category imbalance

The number of training samples in different categories varies greatly .

Rebalancing strategy ：

Undersampling ： Remove some counterexamples , Make positive , The number of counterexamples is close to , Representative algorithm EasyEnsemble;

Oversampling ： Add some positive examples , Make the number of positive and negative examples close to , Representative algorithm SMOTE;

Threshold shift ： Re scaling is embedded in the decision-making process of classifier prediction .