Deep learning neural network

（ Ancient notes , Carry it up and leave a souvenir , When I was studying, I read many big guys' blogs ）
Preface ： I learned various neural networks some time ago , Make a summary today . So that I can review myself later ！

One ．RNN- Cyclic neural network

1. principle ： according to “ Human cognition is based on past experience and memory ” This view is put forward .RNN It's called a cyclic neural network , That is, the current output of a sequence is also related to the previous output . The concrete manifestation is that the network will memorize the previous information and apply it to the calculation of the current output , That is, nodes between hidden layers are no longer connected, but connected , And the input of the hidden layer includes not only the output of the input layer, but also the output of the previous hidden layer . For example, when dealing with movie reviews , You cannot process one at a time , Instead, all comments should be transformed into a large vector , Another one-time treatment .

2. structure ：

Each box can be regarded as a unit , Each unit does the same thing , Explain... In one sentence RNN Namely , Reuse of a unit structure .

3. application ： natural language processing （NLP）, The text generated （ Machines write novels ）, Language models, etc .

4. A brief example ：

（1） Generate Shakespeare's anthology ： We make use of RNN Cyclic neural network , Generate new text .

The code is as follows ：<1> Define model parameters

<2> Recovery model

<3> result ：


（2） distinguish mnist Data sets ：MNIST Datasets are a classic introduction to deep learning demo, It is from 6 Million training pictures and 1 Ten thousand test pictures , Every picture is 28*28 size （ Here's the picture ）, And it's all black and white （ The black here is a 0-1 Floating point number , The darker the black, the closer the value 1）, These pictures are collected by different people from handwritten 0 To 9 The number of .

stay tensorflow Already embedded in mnist Data sets , as follows ：

The code is as follows ：

There are ：

Two ．CNN- Convolutional neural networks

principle ： In the neural network , Each neuron in each layer is connected to each neuron in the next layer , This connection is called full connection （Full Connected）. Take image recognition as an example , Input is every pixel , Then the relationship between two pixels ( No matter how far apart ), All by neurons on the next layer " Calculation " 了 . This method of full connection is too " stupid " 了 , Because image recognition first has to find the image of each part " edge " and " outline ", and " edge " and " outline " It is only related to the adjacent pixels . At this time, convolutional neural network (CNN) That comes in handy , Convolutional neural network can be simply understood as , With a filter (Filter) Between adjacent pixels " outline " To filter out .
structure ：

The picture shows a complete depth CNN Model , The first box is the input layer , The convolution layer and pooling layer are immediately followed , These two floors are CNN Unique . Convolution layer + The pool layer can appear many times in the hidden layer , In the figure above, we draw twice . Several convolutions + After pooling, there is a full connection layer . Picture is the working principle of convolution layer ：

Experiment 1 ：

1. Mission requirements

mnist Handwriting recognition is a classic introductory classification task . Given 28x28x1 The input image of , Output 0-9 A total of ten categories of classification results . This implementation is divided into two parts ：（1）. Using traditional fully connected neural network to realize classification ,（2） Using convolutional neural networks （CNN） Implementation classification ！

2. Using tools

Python,pycharm

Tensorflow 1.1.4

3. frame

（ One ） Traditional neural networks

Input layer ： Stretched handwritten text image , Dimension for [-1,28x28].
Fully connected layer ：500 Nodes , Dimension for [-1,500].
Output layer ： Classification of output , Dimension for [-1,10].
（ Two ） Convolutional neural networks

1. Input layer ： Handwritten text image , Dimension for [-1,28,28,1].

2. Convolution layer 1：filter Of shape by 5x5x32,strides by 1,padding by “SAME”. The dimension after convolution is [-1,28,28,32].

3. Pooling layer 2：max-pooling,ksize by 2x2, In steps of 2. The dimension after pooling is [-1,14,14,32].

4. Convolution layer 3：filter Of shape by 5x5x64,strides by 1,padding by “SAME”. The dimension after convolution is [-1,14,14,64].

5. Pooling layer 4：max-pooling,ksize by 2x2, In steps of 2. The dimension after pooling is [-1,7,7,64].

6. After pooling, the results unfold ： Pool layer 4 feature map an ,[-1,7,7,64] => [-1,3136].

7. Fully connected layer 5： Input dimensions [-1,3136], Output dimension [-1,512].

8. Fully connected layer 6： Input dimensions [-1,512], Output dimension [-1,10], the softmax after , Get the results of classification .

4． Code ：

（1） Defining parameters ：

（2） Define initialization function

（3） Define the first convolution layer + Pooling layer ：

（4） Define the second convolution layer + Pooling layer ：

Pooling results unfold ：

（5） Running results ：

<1> Traditional neural networks

<2> Convolutional neural networks -CNN

You can see CNN Training results of , More accurate .

Experiment two ：

1.（1） summary ： Using convolution network to complete an image classification function ;

（2） After training , The model is kept in model in , It can be classified online ;

（3） stay cnn.py The code includes two functions: training and testing , By modifying the train=true Or is it false To train or test .

2. Data preparation ：

The pictures of the tutorial are from Cifar Get from dataset ,download_cifar.py from Keras Self contained Cifar Part of the data set is obtained Cifar Data sets , And convert it to jpg picture .

The default from the Cifar Data set selected 3 Class picture , Every kind 50 Pictures , Namely

0 => The plane

1 => automobile

2 => bird

The pictures are all on data In the folder , according to label_id.jpg Name it , for example 2_111.jpg The representative picture category is 2（ bird ）,id by 111.

3. Code and its comments ：

（1） Import required libraries

（2） data fetch ：

（3） Definition placeholder( Containers ）

In addition to image data and Label,Dropout The rate should also be placeholder in , Because different settings are needed in the training phase and the testing phase Dropout rate

（4） Define convolutions ：

（5） Define the fully connected layer ：

（5） Define the loss function and optimizer ：

（6） Training ：

（7） test ：

（8） result ：