Pytoch learning notes -- Summary of common functions 3

1--torch.optim.SGD() Function extension

import torch

LEARNING_RATE = 0.01  #  Gradient descent learning rate 
MOMENTUM = 0.9  #  Impulse size 
WEIGHT_DECAY = 0.0005 #  Weight attenuation coefficient 

optimizer = torch.optim.SGD(
    net.parameters(),
    lr = LEARNING_RATE,
    momentum = MOMENTUM,
    weight_decay = WEIGHT_DECAY,
    nesterov = True
    )

Parameter interpretation ：lr It means the learning rate ;momentum Represents impulse factor ;weight_decay Represents the weight attenuation coefficient （ Will use L2 The regularization ）;nesterov Said the use of Nesterov impulse ;

Conventional gradient descent algorithm ：​​​​​​​​​​​​​

l It means the learning rate ; J(θ) The loss function ;▽ Indicates gradient ;

belt momentum Gradient descent algorithm ： 

 m Represents impulse factor ,l It means the learning rate ;

be based on Nesterov impulse Gradient descent algorithm ：

belt weight_decay Gradient descent algorithm ：

The main function is the loss function increase L2 The regularization , It is strongly recommended that Reference link 1 understand L2 The role of regularization , That is, how to avoid over fitting , Weight attenuation through Reference link 2​​​​​​​ understand .

2--torch.manual_seed() Functions and torch.cuda.manual_seed() function

torch.manual_seed() function ： by CPU Set seeds , Ensure that the random number generated by each experiment is fixed , That is, the initialization is the same ;

torch.cuda.manual_seed() function ： by At present GPU Set seeds , The functions and torch.manual_seed() function identical ;

torch.cuda.manual_seed_all() function ： by all GPU Set seeds .

In the neural network , Parameters are initialized randomly by default . Different initialization parameters often lead to different results , When we get good results, we usually hope that this result can be repeated . stay pytorch in , By setting the random number seed, ensure that the initialization operation is the same every time the code runs , Thus in the same algorithm or neural network program , Make sure the result of the operation is the same . Reference link 1​​​​​​​ Reference link 2