Development history of convolutional neural network (part)

The development of convolution neural network

LeNet

Two layers of convolution + Two pooling layers （ The sample layer ）+ Two fully connected layers

AlexNet

（2012）

Bigger and deeper

Relu、Dropout、 Maximum pool layer 、 Data to enhance

VGG

（2015）

VGG block ： Use a lot of 3x3（ Generally, no matter how big, the effect will be bad ） Stack the convolution layers of and add a pool layer to make VGG block , Then use these VGG Blocks form a network

NIN

（2014）

NiN The full connection layer has been completely eliminated .（ NiN The block starts with an ordinary convolution , Then there are two 1×1 The convolution of layer . these two items. 1×1 The convolution layer acts as a band ReLU Activate the pixel by pixel full connection layer of the function . The convolution window shape of the first layer is usually set by the user . The subsequent convolution window shape is fixed to 1×1.( Nonlinearity is added to each pixel )）

NiN Use one NiN block , The number of output channels is equal to the number of label categories . Put one last Global average pooling layer （global average pooling layer）, Generate a logarithmic probability （logits）.NiN One advantage of the design is , It significantly reduces the number of parameters required by the model （ It's not easy to over fit , Fewer parameters . Improve generalization ）. However , In practice , This design sometimes increases the time of training the model .（ Convergence is too slow ）

** thought ：** Calculate an average value for all pixels of the characteristic map of each output channel , After global average pooling, we get a dimension == Number of categories Of Eigenvector , Then enter it directly into softmax layer ;

** effect ：** Instead of Fully connected layer , Images of any size are acceptable

1） It can better correspond the category to the characteristic diagram of the last convolution layer （ Each channel corresponds to a category , In this way, each feature graph can be regarded as the category confidence graph corresponding to the category ）

2） Reduce the amount of parameters , The global average pooling layer has no parameters , Prevents over fitting in this layer

3） Integrate global spatial information , For input pictures spatial translation More robust

（softmax Written in train In the function （loss function ）, Not on the network .【train_ch6 Inside 】）

GoogLeNet

（2014）

Increase the depth and width of the network while reducing parameters .

Simultaneous convolution repolymerization on multiple sizes . It is convolution on multiple scales at the same time , It can extract features of different scales .

Five paragraph , Nine inception block , It is the first network to reach hundreds of times .

Batch normalization

It works on ：

​ Full connection layer and convolution layer output , Before activating the function ;

​ Full connection layer and convolution layer input .

For the full connectivity layer , Acting on the feature dimension ;

For the convolution layer , Acting on the channel dimension .

Batch normalization fixes the mean and variance in a small batch

, Then learn the appropriate offset and zoom .

It can accelerate the convergence speed , But generally, the accuracy of the model is not changed .

ResNet

（2015）

[ Multiply large numbers by decimals , It could be a decimal ; Large number plus decimal , The result is still a large number ]

When the number of layers is too deep , The training effect of neural network becomes worse .

A network without residuals has information loss . and resnet No at all .