(note) learning rate setting of optimizer Adam

Record the problems you see ：Adam Learning rate setting

Commonly used neural network optimizer Adam The adaptive learning rate of is not really adaptive .

From a statistical point of view ,Adam The adaptive principle of is also to modify the gradient according to statistics , But it is still inseparable from the previously set learning rate . If the learning rate is set too large , Will cause the model to diverge , Cause slow convergence or fall into local minimum , Because too much learning rate will skip the optimal solution or suboptimal solution in the optimization process . According to experience , At the beginning, it will not set too much learning rate , Instead, you need to choose according to different tasks . Usually, the default learning rate is 1e-3.

At the same time, the loss function of neural network is basically not convex function , Gradient descent method, these optimization methods are mainly aimed at convex functions , Therefore, in terms of optimization, the learning rate of deep learning will be much smaller than that of traditional machine learning . If the learning rate is set too high ,Adam The optimizer will only correct the gradient , It will not affect the initial learning rate , At this time, the model Loss There will be large fluctuations , This means that there is no way for the model to converge .

1. be relative to SGD Come on ,Adam Of lr More unified .

That's different task You can use the same lr To adjust , More versatility ;

1. learning rate decay Very important .

Even in accordance with paper How it works ,lr It can automatically learn without adjustment , However, after a drop, the efficiency is still significantly improved ;

1. lr Of decay The impact is far less than SGD.

Generally speaking SGD stay CV There are two problems lr falling , Each promotion is considerable , however Adam After the first, the subsequent impact is minimal . This is even in multi-scale In the long-term training ,Adam Not so good SGD;

Reference：

https://www.zhihu.com/question/387050717