All kinds of ice! Use paddegan of the propeller to realize makeup migration

Project links ： Versatile ice ！ be based on PaddleGAN Realize makeup migration

1 Project background

I recently read an article about using ai Blog post for Bingbing makeup , The article uses SCGAN framework , But directly use SCGN The effect of extracting makeup is not very good , The solution of the article is to cut out the face from the picture, then segment the facial features, and finally use SCGN Make up online , The final effect is also very good . So I had an idea —— use PaddleGAN Of PSGAN Model To realize the makeup migration of Bingbing , as for PSGAN What are the advantages? Let's take a look at PSGAN Principle analysis of .

2 PSGAN principle

• PSGAN: Pose and Expression Robust Spatial-Aware GAN for Customizable Makeup Transfer

PSGAN The task of the model is makeup migration , That is, the makeup on any reference image is transferred to the source image without makeup . Many people need this technology to beautify their image . Recently, most of the makeup migration methods are based on the generation of anti network （GAN）. They usually use CycleGAN Framework , And training on two datasets , That is, no makeup image and makeup image . however , There is a limitation in the existing methods ： Only works well in front face images , There is no special module for processing the attitude and expression difference between the source image and the reference image .PSGAN It is a brand-new attitude robust perceptible space generation confrontation network .PSGAN It is divided into three parts ： Makeup refining network （MDNet）、 Pay attention to the deformation of makeup （AMM） Modules and makeup removal - Then make up the Internet （DRNet）. These three new modules enable PSGAN With the above perfect makeup migration model should have the ability .

3 use PaddleGAN Training

The development kit of fighter generated countermeasure network –PaddleGAN, Provide developers with classic and cutting-edge high-performance implementations of generative warfare Networks , And support developers to quickly build 、 Train and deploy to generate counterwork network , For academic purposes 、 Entertainment and industrial applications .

• PaddleGAN The official link

3.1 install PaddleGAN

#  To install paddlegan Go to the root , So first switch directories 
%cd /home/aistudio/
#  Use git clone addleGAN Warehouse to current directory 
!git clone https://gitee.com/paddlepaddle/PaddleGAN.git
#  After downloading, switch to the directory PaddleGAN Of 
%cd /home/aistudio/PaddleGAN

/home/aistudio

3.2 Install third party libraries

When downloading a third-party library ,pip Many downloads are from foreign sources , So we can change to the domestic source, and the download speed will be faster , Here I provide three sources .

• tsinghua ：https://pypi.tuna.tsinghua.edu.cn/simple
• Alibaba cloud ：http://mirrors.aliyun.com/pypi/simple/
• University of science and technology of China https://pypi.mirrors.ustc.edu.cn/simple/

#  Switching path 
%cd /home/aistudio/PaddleGAN/
#  Install the required third-party libraries 
!pip install -r requirements.txt
!pip install boost -i https://mirrors.aliyun.com/pypi/simple/
!pip install CMake -i https://mirrors.aliyun.com/pypi/simple/ 
!pip install dlib -i https://mirrors.aliyun.com/pypi/simple/  
#  The following prompt indicates that the installation is successful 
# Successfully built dlib
# Installing collected packages: dlib
# Successfully installed dlib-19.22.0

3.3 Cut the face

Be careful ： Actually in psgan Image processing has been done in , But when I was training, my face was not in the middle , So you can run the following code to crop face and change image size .（ It can also not run ）

I'm going to use it directly here dlib, Cut out the face area

#  Cut the face 
import cv2
import dlib

#  Path before and after clipping 
cut_before_img="/home/aistudio/work/ps_source.png"
cut_after_img="/home/aistudio/work/ps_source_face.png"
img = cv2.imread(cut_before_img)
height, width = img.shape[:2]

face_detector = dlib.get_frontal_face_detector()
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
faces = face_detector(gray, 1)

def get_boundingbox(face, width, height, scale=1.6, minsize=None):
    """ Expects a dlib face to generate a quadratic bounding box. :param face: dlib face class :param width: frame width :param height: frame height :param scale: bounding box size multiplier to get a bigger face region :param minsize: set minimum bounding box size :return: x, y, bounding_box_size in opencv form """
    x1 = face.left()
    y1 = face.top()
    x2 = face.right()
    y2 = face.bottom()
    size_bb = int(max(x2 - x1, y2 - y1) * scale)
    if minsize:
        if size_bb < minsize:
            size_bb = minsize
    center_x, center_y = (x1 + x2) // 2, (y1 + y2) // 2

    # Check for out of bounds, x-y top left corner
    x1 = max(int(center_x - size_bb // 2), 0)
    y1 = max(int(center_y - size_bb // 2), 0)
    # Check for too big bb size for given x, y
    size_bb = min(width - x1, size_bb)
    size_bb = min(height - y1, size_bb)

    return x1, y1, size_bb


if len(faces):
    face = faces[0]

x,y,size = get_boundingbox(face, width, height)
cropped_face = img[y-50:y+size,x:x+size]
cv2.imwrite(cut_after_img, cropped_face)

True

#  Change image size 
import cv2
im1 = cv2.imread(cut_after_img)
im2 = cv2.resize(im1,(im1.shape[1],im1.shape[1]),)  #  Resize the picture 
cv2.imwrite(cut_after_img,im2)

True

#  Crop contrast display 
import cv2
from matplotlib import pyplot as plt
%matplotlib inline
#  Path before clipping 
before = cv2.imread(cut_before_img)
after = cv2.imread(cut_after_img)

plt.figure(figsize=(10, 10))
plt.subplot(1, 2, 1)
plt.title('cut-before')
plt.imshow(before[:, :, ::-1])

plt.subplot(1, 2, 2)
plt.title('cut-after')
plt.imshow(after[:, :, ::-1])


plt.show()

3.4 Make up migration

Run the following command , You can complete makeup migration , After the program runs successfully , The image file after makeup migration will be generated in the current folder . The original pictures and references are provided in this project for display , The specific order is as follows ：

python tools/psgan_infer.py \  
  --config-file configs/makeup.yaml \
  --model_path /your/model/path \
  --source_path  docs/imgs/ps_source.png  \
  --reference_dir docs/imgs/ref \
  --evaluate-only

Parameter description :

• config-file: PSGAN Network to parameter profile , The format is yaml

• model_path: The path to save the network weight file after the training （ You can leave it blank , Can also be Click to download The weight ）

• source_path: Full path of original image file without makeup , contain Picture file name

• reference_dir: Make up reference image file path , It doesn't contain Picture file name

• The results are automatically saved to /home/aistudio/PaddleGAN/output/transfered_ref_ps_ref.png

#  Makeup migration code 
%cd /home/aistudio/PaddleGAN
!python tools/psgan_infer.py \
--config-file configs/makeup.yaml \
--source_path  ~/work/ps_source_face.png   \
--reference_dir   ~/work/ref \
--evaluate-only
#  The following prompt indicates that the operation is successful 
# Transfered image output/transfered_ref_ps_ref.png has been saved!
# done!!!

3.5 Results show

import cv2
from matplotlib import pyplot as plt
%matplotlib inline
ps_source = cv2.imread('/home/aistudio/work/ps_source_face.png')
ps_ref = cv2.imread('/home/aistudio/work/ref/ps_ref.png') #  This is the path of makeup pictures 
transfered_ref_ps_ref = cv2.imread('/home/aistudio/PaddleGAN/output/transfered_ref_ps_ref.png')

plt.figure(figsize=(10, 10))
plt.subplot(1, 3, 1)
plt.title('Origanl')
plt.imshow(ps_source[:, :, ::-1])
plt.xticks([])
plt.yticks([])

plt.subplot(1, 3, 2)
plt.title('MarkUp')
plt.imshow(ps_ref[:, :, ::-1])
plt.xticks([])
plt.yticks([])

plt.subplot(1, 3, 3)
plt.title('After')
plt.imshow(transfered_ref_ps_ref[:, :, ::-1])
plt.xticks([])
plt.yticks([])

#  preservation 
save_path='/home/aistudio/work/output.png'
plt.savefig(save_path)

plt.show()

4 summary

Looking at the final training effect, I feel pretty good . I've seen it here. Don't you try it yourself , There may be unexpected effects yo

5 Reference material

• PSGAN: Pose and Expression Robust Spatial-Aware GAN for Customizable Makeup Transfer
• PaddleGAN The official link
• be based on PaddleGAN Of PSGAN Model one click makeup migration