Cross validation (CV) learning notes

1. Cross validation can be used to evaluate the performance of machine learning training model , Parameter optimization can also be carried out .
2. Common methods of dividing data sets ： Directly divide the sample data into training and verification data sets . shortcoming ： There is no cross method , The validation data set has no contribution to the training of the model .
3. Common cross validation methods ：
   1. k-flod cv：
      1. The sample data is divided into k Group , One set at a time as a validation data set , The rest k-1 Group as training data set . Then we get k A training model , take k The mean value of the validation accuracy of the models is used as the performance index of the model
      2. advantage ： All samples will be used for model training , The evaluation result is credible .
   2. leave-one-out cv: Let the original data set contain n Samples , Select one sample at a time as the validation data set , rest n-1 Samples as training data set , Will have a n A training model , take n The average validation accuracy of the training models is the performance index of the model .
      1. advantage ： ditto
      2. shortcoming ： There are many models that need training , And the training data set is large , High calculation cost
4. In order to further improve the performance of the model in predicting unknown data , Different parameter settings need to be optimized and compared , This process is called model selection . For a particular problem , The process of adjusting parameters to find the optimal super parameters .
5. Judge the training condition of the model according to the deviation and variance ：
   1. Deviation describes the difference between the predicted value and the real value
   2. Variance describes the variation range of the predicted value , The degree of dispersion , The greater the variance , The more scattered the distribution of the prediction result data .
   3. High deviation is under fitting , High variance is over fitting . Because deviation refers to how much data we ignore , Variance refers to the dependence of the model on data
   4. High variance ： The model changes significantly according to the training data set
   5. Validation sets can prevent over fitting .
6. Set up the pre-test evaluation model , And make improvements before the real test , This prediction trial is called a verification set .
7. Evaluate the degree of data fitting , Use the cost function J=aJtrain( Training set error )+bJcv（ Cross validation set error ）
8. Regularization term ：
   1. Generally, it is a monotone increasing function of model complexity , The more complex the model , The larger the value of the regularization term , For example, the regularization term can be the norm of the model parameter vector .
   2. From the perspective of Bayesian estimation , The regularization term corresponds to the prior probability of the model
   3. L1、L2 Regularization can be understood as the introduction of a priori distribution into the model ,L1 Regularization introduces Laplace distribution ,L2 Regularization introduces Gaussian distribution .
      1. Laplace is distributed in 0 Highlight near value , And Gaussian distribution in 0 The distribution around the value is flat , The distribution on both sides is sparse . Correspondingly （ In fact, it is against , Because the training process is to minimize the loss ）,L1 Regularization tends to sparse models ,L2 Regularization imposes heavy penalties on parameters with high weights .
   4. The regularization term corresponds to the prior information in the posterior probability estimation , The loss function corresponds to the likelihood function , The product of the two yields the Bayesian maximum a posteriori probability .
   5. Logarithm of Bayesian posterior probability can be transformed into loss function + Regularization term .
   6. maximum likelihood ： The multiplication of all sample probabilities maximizes
9. Select the training method according to the data set ：
   1. When the given data is sufficient , Cut the data into training sets （ Training models ）, Verification set （ Model selection ）, Test set （ Model to evaluate ）. Select the model with the minimum prediction error in the verification set
   2. When the data set is insufficient , Use cross validation （ Reuse data ）