Up sampling method (deconvolution, interpolation, anti pooling)

Catalog

1. deconvolution

（1） Normal convolution

（2） deconvolution

2. Anti pooling

3. Interpolation method  

（1） Nearest neighbor interpolation

 （2） linear interpolation

---

1. deconvolution

deconvolution ： Deconvolution can be understood as Inverse operation of convolution operation , Deconvolution Union Cannot restore to the original image , Because convolution will Loss of value , It only reverses the steps in the convolution process once , So it can also be called Transposition convolution .

Deconvolution formula ：

If there's a hole ：

that , How does deconvolution work ？

First, let's take a look at how normal convolution is calculated .

（1） Normal convolution

Suppose the input is 4*4,kernel_size by 3*3,padding by 0,stride by 1

We all know that the operation of convolution is the convolution kernel on the characteristic graph Sliding calculation （ Here's the picture ）, The process is Multiply and then sum Of , That's in the underlying code 4*4 How is the characteristic graph of 3*3 Convolution kernel of for multiplication ？

 

Look at the conclusion first. ：

Images ：        Convolution kernel ：        Output ：

Look at the process again ：

• in the light of Single sheet Characteristics of figure , Input 4*4 The size of will be flattened （flatten） become 16*1 Sequence ,16 Is the number of pixels of the input feature map ,1 Is the number of pictures , A single picture .
• take 3*3 The convolution kernel of 0 operation , Become the same size as the input 4*4 Matrix , Re flatten as 1*16, The known output is 2*2 size , Therefore, the convolution kernel is 4*16 Matrix . among 4 Is the number of pixels of the output feature map ,16 Is the number of pixels of the input feature map .
• take 4*16 The convolution kernel of and 16*1 Matrix multiplication is performed on the characteristic graph of , You get the output 4*1 Sequence , Proceed again reshape operation , To get the final 2*2 Characteristic graph .

（2） deconvolution

Suppose the image size is 2*2, Convolution kernels for 3*3, padding = 0, stride = 1

deconvolution （ Transposition convolution ） How is it calculated at the bottom ？ In fact, the same principle

 

Look at the conclusion first. ：

Images ：        Convolution kernel ：        Output ：

Look at the process again ：

• Enter 2*2 The picture is pulled into a one-dimensional vector 4*1
• Convolution kernel consists of 3*3 Turn into 2*2, Perform the culling operation ,16 Is the number of pixel values of the output image ,4 Is the number of pixels of the input feature map
• Multiply the convolution kernel by the image to get a 16 One dimensional vector of pixel values 16*1, Proceed again reshape operation , To 4*4 Characteristic graph

From the above we can see that , The convolution kernel of the two operations is transposed in shape （ Different weights ）, Therefore, the image is convoluted , Then perform transpose convolution , The two images are not equal . Therefore, the operation is irreversible .

2. Anti pooling

As the name suggests, anti pooling is pooling Reverse operation .

For the anti pooling operation of maximum pooling ： When maximizing pooling , The Internet will Record the index position of the maximum value , Here's the picture max-pooling In operation, in the Yellow window （ On the left ） Of 0.8 Is the maximum , The network will remember the index location , Like the one on the right 4×4 The marks in the grid are shown , In this way, during reverse pooling , You can put the value in the position of the corresponding maximum index , Make up for the rest 0, Here's the picture uppooling Operate the Yellow window （ Right picture ） Lieutenant general 1.3 Put it at the position of the largest index recorded before , Make up for the rest 0.

For example, in the code of maximum pooling return_indices It means whether to save the index of the maximum value , In case of reverse pooling ： 

  Anti maximum pooling code ,indices Index the location for the saved maximum

 

For the anti pooling operation of average pooling ： There is no need to record the maximum value index like maximum pooling , In reverse pooling , just Fill each value in the corresponding window . As shown in the figure below ：

3. Interpolation method  

Then interpolation , There are many ways of interpolation , This paper introduces nearest neighbor interpolation and bilinear interpolation .

• Calculation effect ： Nearest neighbor interpolation algorithm < Bilinear interpolation  
• Calculation speed ： Nearest neighbor interpolation algorithm > Bilinear interpolation

（1） Nearest neighbor interpolation

         The simplest interpolation method , There is no need to calculate , In the four adjacent pixels of the pixel to be calculated , Assign the gray level of the adjacent pixel nearest to the pixel to be obtained to the pixel to be obtained .（ shortcoming ： The output image will be jagged ）

        set up i+u,j+v（i,j As a positive integer ,u,v Is greater than zero and less than 1 Decimals of , The same below ） Is the pixel coordinates to be obtained , Then the gray value of the pixel to be calculated f（i+u,j+v）

The formula is as follows ：src Is the original coordinate ,dst Is the coordinate to be found

Pictured ： The image is from the original 3×3 Turn into 4×4 Size image

srcWidth by 3;dstWidth by 4

To find the first point of the graph （0,0） For example ：

srcX = （0*（3/4）,0*（3/4））=（0,0）= 234

It is concluded that the （0,0） It is the coordinates of the original drawing , Then the value of the point to be calculated is replaced by the value corresponding to the position of the original figure

Waiting for the figure （3,0） Point for example ：

srcX = （（3*（3/4）,（0*（3/4））=（3*0.75,0）=（2.25）=（2,0）=89

Find the nearest coordinate by rounding  

 （2） linear interpolation

① Single linear interpolation

 

Let's take a look at how to calculate it according to an example .

take 2*2 The picture of is expanded to 4*4 Pictures of the ：

Calculation ① To ⑫ Value

The formula ：, Or the above two formulas can , Calculation y Value

notes ：xi It's coordinates ,yi by xi The corresponding value

①：x = 2,  x0 = 1, x1 = 4, y0 = 4, y1 = 6

②：x = 3, x0 = 1, x1 = 4, y0 = 4, y1 = 6

③：x = 2, x0 = 1, x1 = 4, y0 = 4, y1 = 10

④：x = 3, x0 = 1, x1 = 4, y0 = 4, y1 = 10

⑤：x = 2, x0 = 1, x1 = 4, y0 = 6, y1 = 8

② Bilinear interpolation

         double linear Interpolation is also called first-order interpolation , It needs three times of interpolation to get the final result , It is an improvement of nearest neighbor interpolation , First, first order linear interpolation is applied to the two horizontal directions , Then the first-order linear interpolation is carried out in the vertical direction . 

 

Reference resources

Overview of up sampling methods ： linear interpolation , Transposition convolution , To pool