Deep learning series 47: Super sub model real esrgan

1. Model is introduced

1.1 Degenerate model

First, the training data uses 2 individual first-order：

In the last step, ringing and overshoot are added ：

1.2 SRCNN

take CNN The first article in the field of hyperfractionation ：

The experimental setup of convolution kernel and channel number in this paper is ：

1.3 SRGAN

SRGAN take GAN Introduction of hyperdomain , Used to solve the following problems ：
1） High frequency details (high-frequency details) The loss of , The overall image is too smooth / Fuzzy ;
2） Inconsistent with human visual perception , The accuracy of the hyperspectral image does not match people's expectations （ People may be more concerned about the future , And the requirement for background clarity is not high ）.

Propose the following improvements ：

• new backbone：SRResNet;
• GAN-based network And New loss function ：
• adversarial loss： Enhance the sense of reality (photo-realistic natural images);
• content loss： obtain HR image And the perceptual similarity of the generated image (perceptual similarity), Not just pixel level similarity (pixel similarity); In other words, the similarity of feature space is not the similarity of pixel space .
• Use subjective assessment , More emphasis on human perception .

The model structure is as follows ,Generator Network is SRResNet, The paper uses 16 individual residual blocks;Discriminator The Internet is 8 Sub convolution operation （4 The next step is 2）+2 Sub full connection layer VGG The Internet .

1.4 ESRGAN

enhanced SRGAN, It mainly solves the problems of blurred details and artifacts .

• SRResNet Improvement of network structure ：
  1） remove BN, It is helpful to remove artifacts , Improve the generalization ability ;
  2） Use Residual-in-Residual Dense Block (RRDB) As a basic building block , Stronger and easier to train ;
• GAN-based Network The improvement of the loss function ： Use RaGAN (Relativistic average GAN) Relative loss function in , Improve the relative authenticity of the image to restore more texture details ;
• Improvement of perceptual loss function ： Use VGG The reconstruction loss is calculated by the eigenvalue before the activation layer , Improved brightness consistency and texture recovery .
  

2. Quick start

2.1 Various resources

The green version of exe See... For documentation github, Support windows,linux,mac and NCNN
Online version ：https://huggingface.co/spaces/akhaliq/Real-ESRGAN
Usage method ：./realesrgan-ncnn-vulkan.exe -i Second dimensional picture .jpg -o Picture of two spiny newts .png -n realesrgan-x4plus-anime
The parameters are as follows ：

Usage: realesrgan-ncnn-vulkan.exe -i infile -o outfile [options]...
  -h                   show this help
  -i input-path        input image path (jpg/png/webp) or directory
  -o output-path       output image path (jpg/png/webp) or directory
  -s scale             upscale ratio (can be 2, 3, 4. default=4)
  -t tile-size         tile size (>=32/0=auto, default=0) can be 0,0,0 for multi-gpu
  -m model-path        folder path to the pre-trained models. default=models
  -n model-name        model name (default=realesr-animevideov3, can be realesr-animevideov3 | realesrgan-x4plus | realesrgan-x4plus-anime | realesrnet-x4plus)
  -g gpu-id            gpu device to use (default=auto) can be 0,1,2 for multi-gpu
  -j load:proc:save    thread count for load/proc/save (default=1:2:2) can be 1:2,2,2:2 for multi-gpu
  -x                   enable tta mode"
  -f format            output image format (jpg/png/webp, default=ext/png)
  -v                   verbose output

Existing models ：
realesrgan-x4plus（ Default ） Clear effect , Prefer brain tonic ;
reaesrnet-x4plus（ The effect is vague , Prefer to smear ）
realesrgan-x4plus-anime（ For animation illustration image optimization , There's a smaller volume ）
realesr-animevideov3 ( For animation video )
This is the plan for the future ：

2.2 github Previous source code

git clone https://github.com/xinntao/Real-ESRGAN.git
cd Real-ESRGAN
#  install  basicsr - https://github.com/xinntao/BasicSR
#  We use BasicSR To train and infer 
pip install basicsr
# facexlib and gfpgan Is used to enhance the face 
pip install facexlib
pip install gfpgan
pip install -r requirements.txt
python setup.py develop
wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.0/RealESRGAN_x4plus.pth -P experiments/pretrained_models

And then execute ：

python inference_realesrgan.py -n RealESRGAN_x4plus -i inputs --face_enhance

2.3 Training and fine tuning

Reference resources https://github.com/xinntao/Real-ESRGAN/blob/master/docs/Training_CN.md
Fine tuning , You can use the degradation model that comes with the program , You can also provide your own data pairs .