ROC and AUC details of the recommended system

Preface

This is definitely a heavyweight content , It is also a very basic thing . For the model , Different thresholds have different generalization capabilities , Now , If you want to compare the generalization ability of the two models , The level of this threshold will also be affected , Therefore, we need to be able to comprehensively consider the generalization performance of the model under all thresholds , This can also make the model adapt to different tasks , How should we evaluate the model at this time ？ ROC Curve is a common tool .

Here and what we said earlier PR The curve is similar （mAP）, But before PR The horizontal and vertical coordinates in the curve are precision and recall . But here it is replaced by the real case rate （TPR） And false positive rate （FPR）, These two are also calculated according to the confusion matrix , If you don't understand it here, you can go to my previous article Target detection index mAP Detailed explanation .

1、PR Curves and ROC The difference between curves

First of all ROC Definition of curve , Let's take a look at the picture below ：

  True case rate TPR: This is recall , In the samples with positive real situation , The model predicts a positive proportion , The formula

  Positive case rate FPR: This is the sample with negative real situation , The model predicts a positive proportion , The formula

These two are ROC The horizontal and vertical coordinates of the curve , So why don't you use it in the recommendation system PR Curve ？

This is because in the recommendation system, it is easy to have imbalance between positive and negative samples , comparison PR curve ,ROC A characteristic of curve is , When the distribution of positive and negative samples changes , ROC The shape of the curve can remain basically the same , and P-R The shape of the curve usually changes dramatically , This feature makes ROC The curve can minimize the interference caused by different test sets , More objectively measure the performance of the model itself . In many practical problems , The number of positive and negative samples is often uneven , For example, in computing advertising , Positive samples are often negative samples 1/10000, If you choose a different test set ,P-R The curve changes a lot , and ROC The curve can more stably reflect the quality of the model , That's why ROC The reason why curves are widely used . Of course , choice PR Curve or ROC The curve should still depend on practical problems , If you want to see more of the performance of the model on a specific data set , PR The curve can reflect the performance more intuitively . In the anti fraud scenario , Assume that the normal user is a positive class （ Design proportion 99.9%）, Fraudulent users are negative （ Design proportion 0.1%）. If accuracy evaluation is used , Then all users can be predicted as positive classes to obtain 99.9% The accuracy of . This is not a good prediction , Because of fraud, all users failed to find . Use AUC assessment , Now FPR=1,TPR=1, Corresponding AUC=0.5 .AUC Successfully pointed out that this is not a good prediction result .

2、AUC Physical meaning and calculation

AUC The affirmation represented is ROC The area under the curve , But its real meaning is to randomly give a positive sample and a negative sample , Positive sample prediction The probability that the score is greater than the predicted score of the negative sample .AUC The bigger it is , It means that the probability of the model prediction sample being a positive sample is greater than that of the model prediction sample being a negative sample , The measurement of sample prediction sorting quality mentioned in the book , It is closely related to the sorting error . Such as ：AUC=0.8 Express ： Given a positive sample and a negative sample , stay 80% Under the circumstances , The probability that the model predicts positive samples is greater than that of negative samples .（ The probability of ranking positive samples in front of negative samples ）. So where the positive and negative samples are unbalanced AUC Still apply .

among AUC The benefits are as follows ：

• Use accuracy , Recall rate and F1 The threshold value needs to be set when evaluating the model , The size of the threshold will affect the generalization ability of the model , Use AUC You can evaluate the performance of the model without setting a threshold ;
• AUC Calculation is mainly related to sorting , it Sensitive to sorting , Less sensitive to prediction scores （ Sorting results are more important , No matter what the predicted value is , As long as the prediction probability of positive cases is greater than that of negative cases , Multiply the prediction probability by 1.2,AUC The value remains the same ）
• under certain conditions ,RankBoost The global function of algorithm optimization is AUC, have access to RankBoost Algorithm optimization AUC An approximation of or to AUC Local optimization .（paper）;xgboost Provides direct optimization AUC The function of , The objective function is set to ：objective = ‘rank:pairwise’.
• When the distribution of positive and negative samples changes ,P-R The shape of the curve will generally change violently , and ROC The shape of the curve can remain basically the same . This makes ROC The curve can minimize the interference caused by different test sets , More objectively measure the performance of the model itself .

Reference resources ： Tumbling cockroach