Common evaluation indexes of classification model -- confusion matrix and ROC curve

Confusion matrix

Quantitative accuracy

• Overall classification accuracy (Overall Accuracy)
  The number of samples with correct classification / The total number of samples
  OA=（e+f+g）/(a+b+c+d+e+f+g+h+i)
• Kappa coefficient
  P₀=OA
  P_e=∑( Real sample number × Number of classified samples )/ The total number of samples ²
  for example ：P_e=[(a+b+c)×(a+d+g)+(b+e+h)×(d+e+f)+(g+h+i)×(c+f+i)] /(a+b+c+d+e+f+g+h+i)²
  Kappa=(P₀－P_e )/(1-P_e)
• User accuracy (User accuracy )
  From the perspective of prediction , The number of correctly predicted samples accounts for the total number of predicted samples in this category
  • Category 1_{user_accuracy}=a/(a+d+g)
  • Category 2_{user_accuracy}=e/(b+e+h)
  • Category 3_{user_accuracy}=i/(c+f+i)
• Producer’s accuracy (Producer accuracy)
  From the perspective of samples , The number of correctly predicted samples accounts for the total number of samples in this category
  • Category 1_{Prod.accuracy}=a/(a+b+c)
  • Category 2_{Prod.accuracy}=e/(d+e+f)
  • Category 3_{Prod.accuracy}=i/(g+h+i)

ROC curve

roc The curve is mainly used to evaluate the accuracy of the binary classification model

a:TP（ True positive ）
c:FP（ False positive ）
b:FN（ False positive ）
d:TN（ True negative ）

• True inspection rate ( sensitivity , Hit rate )
  TPR=a/(a+b)
• False detection rate ( Specificity , False alarm rate )
  FPR=c/(c+d)

ROC curve The horizontal axis of the coordinate is FPR, The vertical axis of the coordinate is TPR,ROC The curve does not describe FPR And TPR The functional relationship between

ROC The curve does not describe FPR And TPR The functional relationship between ,FPR And TPR There is no functional relationship between ,FPR And TPR yes threshold τ Function of ,τ Describes the percentage of samples tested positive , When τ=0 when , All samples were identified as negative , here FPR( Specificity 、 False alarm rate 、 False detection rate ) by 0(FPR The smaller the better. ), but TPR( sensitivity 、 shooting 、 True inspection rate ) by 0(TPR The bigger the better ), When τ=1 All samples were identified as positive , here TPR=1,FPR=1. Therefore, it is necessary to select an appropriate τ Value makes TPR The bigger the better ,FPR The smaller the better. .

Area under curve (Area Under Curve,AUC)
AUC be equal to roc The area of the Encircled City along the horizontal axis of the curve and coordinate , yes roc Integral of the curve ,AUC The larger the number, the better