Interpretation of video generation paper of fed shot video to video (neurips 2019)

The paper : 《Few-shot Video-to-Video Synthesis
Ting-Chun》
github: https://github.com/NVlabs/few-shot-vid2vid

solve the problem

Even though vid2vid（ See last article Video-to-Video Interpretation of the thesis ） Significant progress has been made , But there are two main limitations ;
1、 A lot of data is needed . Training requires a large amount of target human body or target scene data ;
2、 The generalization ability of the model is limited . Can only generate training concentrated in the human body , For people who have never seen human body, their generalization ability is poor ;
To solve the above problems , The author puts forward few-shot vid2vid, Several examples of target domain are used in reasoning , There is no target or scene video before learning and generating . By using attention The network weight generation module of the mechanism improves the generalization ability of the model ;

Algorithm

vid2vid programme

vid2vid Structure is shown in figure 2a, The formula is as follows 1,

Generate the image ${\tilde{X}}_t$, Based on the past $\tau +1$ Split graph $s_{t-\tau }^t$ And the past $\tau$ A generation diagram ${\tilde{X}}_{t-\tau }^{t-1}$. The general design of the generator is as follows 2,

${\tilde{m}}_t,{\tilde{w}}_{t-1},{\tilde{h}}_t$ The learning process is like 3,4,5,

${\tilde{m}}_t$ For occluded areas ,${\tilde{w}}_{t-1}$ Is optical flow ,${\tilde{h}}_t$ To generate an intermediate image ;

Few-shot vid2vid

however vid2vid Method cannot generate video without domain , For example, you can only generate people who appear in the training set . In order to make $F$ Suitable for domains that have never been seen , Add additional input , therefore $F$ There are two inputs ：K Sample images of target domains $e_1,e_2,...,e_K$, And the corresponding semantic map $s_{e_1},s_{e_2},...,s_{e_K}$, Such as the type 6
The model allows $F$ Use the input sample mode to generate a video , Therefore, the author uses the network weight generation module $E$ Extract the sample mode , Calculate network parameters by using sample modal information ${\theta }_H$, Such as the type 7.

$E$ Not used ${\theta }_M$ And ${\theta }_W$, Because ${\theta }_M$ And ${\theta }_W$ Generate images based on the past , Itself has been shared across domains ;
$E$ Generate weights only for spatially adjusted branches , The design has two advantages ：
1、 Reduce $E$ Generate parameter quantity , Avoid overfitting ;
2、 Avoid samples and output images shotcut, Because the generated parameters are only used for the space adjustment module

Network weight generation module

$E$ Extract appearance mode by learning parameters , Introduce video generation branch ;
Consider entering only 1 A sample （k=1） And several sample cases （k>1）;
When the input example is 1, namely k=1 when ,（image animation Mission ）$E$ Decouple into two subnetworks ： Feature extractor $E_F$ And multi-layer perceptron $E_P$.
Feature extractor $E_F$ Including several convolution layers , Extract the appearance representation $q$, Applied to multi-layer perceptron $E_P$ Generate weights ${\theta }_H$, For intermediate image generation network $H$,$H$ Yes $L$ layer ,$E$ There are also $L$ layer , Pictured 2c Shown ;

When the input samples are more than 1 when , namely K>1, to want to $E$ The appearance mode of any number of samples can be extracted , Because different input samples have different angle correlations , So the author designed attention Mechanism $E_A$, Aggregate the extracted appearance modes ;
$E_A$ Apply to each sample semantic diagram $s_{e_k}$, Get the key vector ${\alpha }_k\in R^{C\times N}$, And the current input semantic map $s_t$, Get the key vector ${\alpha }_t\in R^{C\times N}$, And then pass ${\alpha }_k=({\alpha }_k{)}^T⨂{\alpha }_t$, obtain attention The weight ${\alpha }_k\in R^{N\times N}$, Applied to appearance characterization $q$. Pictured 2b, In the generation phase attention map Help capture relevant body parts , Pictured 7c.

To reduce the burden of image generation network , Sure wrap Examples , And generate an intermediate image ${\tilde{h}}_t^{\prime }$ combination , Such as the type 11,

Estimate additional optical flow ${\tilde{w}}_{e_t}$ And mask ${\tilde{m}}_{e_t}$, When entering multiple samples ,$e_1$ by attention weight ${\alpha }_k$ The sample with the greatest similarity with the current frame in ; It was found that , When the background area remains unchanged , To generate pose To be helpful to ;

experiment

The author compares different methods to achieve the generalization ability of the target domain , As shown in the table 1;
Encoder Indicates that the coding sample is a sub style vector ;
ConcatStyle Indicates that the coding sample is a sub style vector , Compare it with the input segmentation graph concat, Generate enhanced split input ;
AdaIN stay H After each spatial modulation module, add AdaIN Normalized layer .
The implementation details are shown in the figure 9 Shown ,

chart 3 Produce different results for the human body

chart 4 Compare for various methods
chart 5 Generate results for street scenes
chart 6 Generate results for the face

chart 7a Compare the influence of different training sample numbers on the results , chart 7b Compare the impact of different sample numbers on the results , chart 7c visualization attention map result ;

Conclusion

few-shot vid2vid It can generate videos that have never been seen before , Due to the dynamic weight generation mechanism based on the sample image .
But there are still two limitations ：
1、 When there is a great difference between the test domain and the training domain , The generation effect is relatively poor ;
2、 When segmentation and key point estimation are inaccurate , The generation effect is relatively poor ;