Focal and global knowledge distillation for detectors

cvpr2021
Zhendong Yang*1,2 Zhe Li2 Xiaohu Jiang1 Y uan Gong1
Zehuan Y uan2 Danpei Zhao3 Chun Y uan†1
1Tsinghua Shenzhen International Graduate School 2ByteDance Inc
3BeiHang University

0. Abstract

The author thinks that in the field of target detection , The effect of knowledge distillation is poor , The reason is that there is a big gap between the characteristics of teachers and students in the background and prospects . If the distillation temperature is the same , It will absorb a lot of noise .
Global and local distillation , Local distillation will separate the foreground and background （ Semantic segmentation ？）, To make students pay attention to the essence of teachers . Global distillation reconstructs the relationship between different pixels , Fill in the global information .

1. Model structure

1.1 Local distillation

It's simple ,r by gt_box, Only this one is 1.
Because the front and back scenes are not balanced , Yes mask Improved ：

Then we add space and channel attention ：

The loss consists of two parts , Feature loss
Yes and attention loss ：

1.2 Global distillation