Context encoders: feature learning by painting paper notes

Computer Vision and Pattern Recognition Nov 2016

Abbreviation of the network CE, The overall architecture of the model is an encoder - decoder .

Model structures,

Network structure

• Encoder Composed of convolution （ from AlexNet Separated from ）, Used to extract multiple feature map.$227\times 227$ Code to $6\times 6\times 256$.
• The full connection layer is used in multiple feature map To send messages （ Light use Encoder The convolution layer of cannot connect all positions in a characteristic graph ）.

In order to reduce the amount of calculation , This network uses the channel full connection layer . That is, for multiple characteristic graphs , Not all of them are connected by one brain , Instead, each feature graph is connected separately .
In this case , If there is m individual $n\times n$ Characteristic graph , Then the parameter quantity will change from the original $m^2{n}^4$ Reduce to $m{n}^4$.

• Decoder It is mainly composed of deconvolution and upper sampling layer （ Coincide with Encoder contrary ）, Is used to feature map enlarge To original size .

Loss function

It's made up of two parts . Use only L2 Loss function doesn't work , Because it does not have translation invariance , It makes it easier for the generated pixel values to be biased towards the average . So the author adds here adversarial loss.

• reconstruction loss (L2)
  $${\mathcal{L}}_{rec}(x)=\Vert \hat{M}\odot (x-F((1-\hat{M})\odot x)){\Vert }_2^2$$
  effect ： Capture the overall structure of the repair area and its consistency with the surrounding visible area .

$M$ yes mask,x It's the input image ,$\odot$ Is the product of elements ,F yes context encoder The result .

• adversarial loss（ suffer GAN inspire ）
  effect ： Make the prediction of the repaired area look more realistic .

GAN It is usually composed of generator and discriminator , This article makes the previous Encoder-Decoder The result is a generator , Therefore, it is only necessary to define an additional discriminator .

$$\begin{array}{rl}{\mathcal{L}}_{adv}=\underset{D}{\max }& {\mathbb{E}}_{x\in \mathcal{X}}[\log (D(x))+\log (1-D(F((1-\hat{M})\odot x)))]\end{array}$$
During training , function F And D A combination of SGD（ Stochastic gradient descent ） To optimize .

Joint loss function

Is to add the two loss functions after weighting .
$$\mathcal{L}={\lambda }_{rec}{\mathcal{L}}_{rec}+{\lambda }_{adv}{\mathcal{L}}_{adv}.$$

appendix

About L2 The loss cannot be directly used , original text .