Coloring old photos - deoldify get started quickly

Qiyuan

Recently, my family turned out some old black-and-white photos , Want me to help color old photos , So I am gayhub I found DeOldify This project .
Project address ：https://github.com/jantic/DeOldify
The author in DeepAI I have provided Demo, You can upload photos directly to experience .
https://deepai.org/machine-learning-model/colorizer

Here are some examples provided by the author , A sneak peek ：

principle

I can't find the corresponding paper for this project , I guess the author didn't send the paper , Because in Readme There is a subtitle in Stuff That Should Probably Be In A Paper.

According to the author's readme , The bottom layer of the project still uses the famous generation confrontation network (Gan), But he adopted a new GAN Training methods , be called NoGAN( At first glance, the name thought he was useless GAN).
generator ( Responsible for model coloring ) He uses classic ”U-net“, At the same time, it adopts pre training for the generator .U-net The backbone network of is also used ResNet, Later, the running code will be automatically downloaded online ResNet Model parameter file .

The following figure for U-net structure , Picture source [1]:

Practice

Before starting , First, learn about the three versions of the pre training model provided by the author .
namely Artistic、Stable、Video
The original description is as follows ：

Artistic - This model achieves the highest quality results in image coloration, in terms of interesting details and vibrance. The most notable drawback however is that it’s a bit of a pain to fiddle around with to get the best results (you have to adjust the rendering resolution or render_factor to achieve this). Additionally, the model does not do as well as stable in a few key common scenarios- nature scenes and portraits. The model uses a resnet34 backbone on a UNet with an emphasis on depth of layers on the decoder side. This model was trained with 5 critic pretrain/GAN cycle repeats via NoGAN, in addition to the initial generator/critic pretrain/GAN NoGAN training, at 192px. This adds up to a total of 32% of Imagenet data trained once (12.5 hours of direct GAN training).

Stable - This model achieves the best results with landscapes and portraits. Notably, it produces less “zombies”- where faces or limbs stay gray rather than being colored in properly. It generally has less weird miscolorations than artistic, but it’s also less colorful in general. This model uses a resnet101 backbone on a UNet with an emphasis on width of layers on the decoder side. This model was trained with 3 critic pretrain/GAN cycle repeats via NoGAN, in addition to the initial generator/critic pretrain/GAN NoGAN training, at 192px. This adds up to a total of 7% of Imagenet data trained once (3 hours of direct GAN training).

Video - This model is optimized for smooth, consistent and flicker-free video. This would definitely be the least colorful of the three models, but it’s honestly not too far off from “stable”. The model is the same as “stable” in terms of architecture, but differs in training. It’s trained for a mere 2.2% of Imagenet data once at 192px, using only the initial generator/critic pretrain/GAN NoGAN training (1 hour of direct GAN training).

Simple understanding , That is to say Artistic The model tends to be artistic , Coloring will tend to be rich and bold ,Stable The model is stable , Coloring is usually realistic , Suitable for scenery and portrait . Both of them are for image coloring , For video coloring, you need to use the third Vidio Model , It solves the flicker problem of video coloring .

Now let's officially enter the code operation link , My system environment is Windows10, Although the environment used by the author is Linux, And say no windows Environmental Science , But I still found something in the code to windows Interface called by the user .

First clone the above code warehouse , Create a new one models The folder is used to store the pre training weights of the three models .
The author of this file provided the download address of the online disk , You can also use the download address provided by me , Download faster ：
https://pan.baidu.com/s/1Wgib3b8o5F5W2dUnYrY_zw?pwd=8888

after , Create a new call file in the root directory main.py：

import os
from deoldify.visualize import get_artistic_video_colorizer, get_stable_image_colorizer, get_artistic_image_colorizer
import warnings

warnings.filterwarnings("ignore")


def colorizer(input_path, output_dir):
    output_path = os.path.join(output_dir, os.path.basename(input_path))
    image_colorizer = get_stable_image_colorizer()  # stable Stable version 
    # image_colorizer = get_artistic_image_colorizer() # artistic  Art Edition 
    result = image_colorizer.get_transformed_image(input_path, render_factor=30, post_process=True, watermarked=True)

    if result is not None:
        result.save(output_path, quality=95)
        result.close()
        
if __name__ == '__main__':
    colorizer("./data/image/yanb_gray.jpg", "./data/result/")

The interface file provided by the author is placed in visualize.py In this file ,get_artistic_image_colorizer You can call artistic Model ,get_stable_image_colorizer() You can call stable Model .
In order to intuitively feel the coloring effect of this model , I downloaded. Biddle Photos of the , First, gray it ：

from PIL import Image

image_raw = Image.open("data/image/yanbi.jpg")
image_black_white = image_raw.convert('L')
image_black_white.save('data/image/yanbi_gray.jpg')

Then input it into the model for coloring , The effect is shown below ：

It turned out to be very Amazing ah ！ After coloring , I found that Bi Dao's clothes changed color , Because the model cannot infer the original color information according to the gray details . Art model (Artistic) The color on the model is significantly more stable than that on the model (Stable) The color is more “ Glamorous ” some . The main thing is , Bi Dao after color remodeling , The skin turns white ！

Return to the right topic , Next, I wanted to test the video recovery effect again , Because the author also provides colorize_from_file_name This interface can be used to color video , But I found some problems in the input path . According to the warehouse issue[2], The author explains that this may be Linux and windows The path of is incompatible , I can only try again later .

Code backup

in addition , I found that the author of this repository is still maintaining updates , In order to ensure the stability of subsequent results , I will back up the current code , At the same time, it is inconvenient to go gayhub Our readers can download ：
https://pan.baidu.com/s/1k4Dp8Sanz4FzbR6jtoTZ6g?pwd=8888

References

[1]https://www.likecs.com/show-205090690.html
[2]https://zhuanlan.zhihu.com/p/55050741