Internship report 1 - face 3D reconstruction method

The main problem of face 3D reconstruction

1. Detail reconstruction
2. Texture reconstruction
3. Joint reconstruction of texture and shape

At present, due to the lack of effective information of a single picture , It is difficult to reconstruct the details of a picture from a single picture , It needs to cooperate with a priori algorithm or neural network to further process the image .
And , Today, with the increasing meaning of privacy protection ,RGBD The lack of image data also hinders the training of machine learning .

Study through thesis , Found a model ,DF2NET, Can use a single image to reconstruct face data .

Reconstruction of details

In the reconstruction of details , The most commonly used solution is 3DMM Model , however 3DMM The model needs to use hundreds of dimensional data to represent the data of tens of thousands of vertices , It will make it difficult to deal with the details in the reconstruction process , stay 2D To 3D There will be distortion in the conversion of , And it requires a lot of computing overhead , Data alignment is required during initialization , It is also a time-consuming process .
however 3DMM It can predict a three-dimensional figure according to any two-dimensional image , This is its advantage , Just rely on 3DMM Unable to deal with the details of the face , It needs to be used with other models .

also SfS Model , Through the transformation of light, the light and dark details of the image are transformed , So as to analyze the shape of the image , However, this model will cause the image to only judge the shape when changing the light and shade , It is difficult to judge the concave convex nature of the image , As a result, other technologies need to be used to restore during reconstruction , For example, get the depth of the image and UV Space to restore , As shown in the example below ,SfS There is no good way to analyze concavity .

Reconstruction of texture

In the reconstruction of texture , Mainly used

• A model-based approach
• Image based approach

The efficiency of model-based method is relatively low ,3DMM Model is the model-based method used , Before he uses a single picture like 3 When transforming a dimensional image , Limited by linear subspace , It will cause the loss of texture details , The texture restored by computational learning algorithm will lose its authenticity , And the shape needs to be considered 、 Reflectivity 、 Lighting parameters of the picture , Further setting of parameters is required .

The image-based method can capture the texture of any image , There is no need to consider the shape of the image 、 Reflectivity picture and illumination parameters , But it may be limited by the sharpness of the image, resulting in no good way to restore the texture .

Restoration steps of image-based method ：

1. Obtain images and feature points from multiple angles of the face
2. Use curves to mark facial features
3. Generate 3D template face
4. Through the given characteristic point in UV Space to fit the face

Above picture ,a Several angles and feature points of the image are given ,B Marked the characteristics of the face ,C given 3 Dimensional face model ,D It is given that the number of characteristic points is 13 The model fitting of 3D face reconstruction ,E The number of characteristic points is given in 119 The fitting of the 3D face reconstruction model .

Face reconstruction based on texture shape

Let's start with DF2NET Method , The model diagram of this method is as follows ：

He used D-Net、F-Net、Fr-Net Three networks are superimposed to process a single image .

1.D-Net： adopt 2D Picture get depth map
2.F-Net： Input 2D Image and depth map , Output refined depth map （ Restore high frequency details ）
3.Fr-Net： Restore details from images with different sharpness .（ Weight the shadow ）

github Link to ：https://github.com/xiaoxingzeng/DF2Net

because macos I won't support it cuda, So you need to torch All of the cuda The function is changed to cpu To deal with , Of course, you can also try to use it mac os Supported by mps To deal with .

By running demo Program , And in the code img_list Make changes （ This one in the code is wrong , Incomplete documents , Need to change ）, To be able to get one tensor Format data output .
And in img Get the generated pictures under the directory .

But at present, due to limited technical capacity , Didn't get 3 The generated image of dimension .

Here the image is tested ：
The input image ：

Output image ：

Image cropping ：

The current analysis results in demo The program only supports face recognition and clipping .
But look closely at the output image , The resolution of the output image is compressed , Lost some pixels , The image effect is not very good , Guess the face information of 3D reconstruction will not be ideal .

The scheme given in this paper is to restore the depth map of the image , And pass by D-Net and Fr-Net Further detailed treatment .

But the problem of test effect may be caused by the low resolution of the image , It can be considered to cooperate with the super-resolution scheme to restore the details of the face .

Next, I tested Facial_Details_Synthesis-master project .
This is a code base for synthesizing facial details from a single input image .

github link ：https://github.com/apchenstu/Facial_Details_Synthesis/tree/def9bfe044790d771d15dc4a62bb47b78d7d6153

This repository is provided by 5 It consists of three independent parts ：DFDN、emotionNet、landmarkDetector、proxyEstimator and faceRender.

This method adopts the way of antagonizing the network to recover the details of the face .
This method takes into account the normal 、 light 、 shadow 、 Albedo 、 mood 、 expression 、 posture 、 Facial texture , The facial expression of human face is processed a priori , Facial details are reconstructed by judging users' emotions through the expression database .
If there are losses in the reconstruction process , Then recover facial details through training .

The model used here is BFM Model , Links to models ：https://faces.dmi.unibas.ch/bfm/bfm2017.html

At present, the method is tested in eos-py There are some problems in the installation of the module , It's not settled yet , The initial suspicion is that Cmake and visual stdio No binding , Lead to eos-py 0.16.1 Version of the module cannot be installed . Trying to test and repair the code .