Dark king ｜ analysis of zego low illumination image enhancement technology

In low light night , The picture captured by the camera is usually dim , The definition of the picture is much lower than that of the naked eye . With the development of real-time audio and video application technology , We have seen various video enhancement technologies for image quality enhancement , that

Is there a technology , It can make the video look clearer than the actual situation or close to the actual situation under low light conditions ？

 

Bedroom walls and lights , Before and after low illumination enhancement

Application scenarios of low illumination image enhancement technology

Low illumination image enhancement refers to image quality enhancement of images taken under dark light conditions , Make it clear or close to the naked eye . We summarized some strong demand scenarios related to low illumination , As follows ：

1、 Improve the visibility of night inspection or monitoring    

At night , The video quality and clarity of surveillance cameras or car safety cameras are crucial .

 

Wuzhen archway at night , Contrast before and after low illumination enhancement  

 

The purpose of setting surveillance cameras in public places is to monitor and record . However , Night video recording is mostly dark and unclear in the case of insufficient light , It cannot be used as a clue or evidence in criminal cases to provide clear and strong support .

The video recording of car safety camera is very important for the handling of traffic accidents , The clearer the video, the more details , Help the traffic police deal with it better and faster .

2、 For real-time streaming media or video chat    

When it comes to internet entertainment and social scenes , We hope to see each other's faces clearly even in dark light .

 

This is a real event that happened to our customers . This customer is a large-scale online dating platform , There is a user who doesn't like turning on the light in her bedroom, plus her dark skin , As a result, her netizens couldn't see her face in this dark environment . Our customers maintain a high degree of respect for users , So I hope to improve the user experience without turning on the light or insufficient light .

3、 be used for  AI  Image enhancement technology of content recognition    

our  Avatar  The technology can model the character in real time according to the input image . Face recognition in this technology 、 Many video related algorithms, such as human key recognition, have high requirements for the quality of the input image . The algorithm can correctly recognize the position of face and body depends on whether the input image is clear enough .

When the input image is dark or blurred , Recognizing face and body contours is very challenging , So the final output effect will be affected .

Traditional practices and disadvantages

In order to meet the needs of users , Solve these problems of low video quality under insufficient light conditions , Through self research  AI  Algorithm enhances low illumination video .

1、 Use traditional algorithms to enhance video brightness    

The main function of low illumination enhancement is to improve the brightness of the picture , Also consider the following limitations ：

 

• Picture overexposure
  ： High brightness pixels should not be excessively highlighted

 

• Suppress noise ：
  Image noise should not be amplified

 

• The outline is clear
  ： Effective information should be retained to the greatest extent

 

It is difficult for traditional algorithms to consider the above requirements at the same time . Depth learning method can also be used for low illumination image enhancement , But it's in memory 、CPU and GPU There is a certain consumption , Its disadvantages are as follows ：

 

• The model and inference base are large

 

• The amount of calculation is large

 

• Power consumption

 

Community corridors at night , Before and after low illumination enhancement

 

2、 Common problems of traditional algorithms    

The traditional algorithm enhances the low illumination of video at the same time , All contents in the screen are processed according to the same rules , It is easy to cause the gray scale of the bright area in the picture to be overstretched , Make the picture appear overexposed .

 

The model of deep learning method is generally large , Large amount of calculation , High power consumption , It is difficult to perform real-time processing on mobile terminals or edge devices , The coverage models are extremely limited .

 

ZEGO Low illumination image enhancement , better

1、ZEGO Low illuminance , Make up for the shortcomings of traditional algorithms    

In order to solve the problem of traditional low illumination image enhancement , We have made the following improvements ：

 

• First of all, not all video pixels should be highlighted
  , We should avoid over highlighting pixels that are already bright enough . Some video pixels are low brightness , It needs to be brightened to different brightness accordingly .ZEGO  The low illumination algorithm scans the video frame by frame , Classify pixels with different brightness .

 

•  
  The deep learning model will not be loaded in the actual running scenario
  . contrary , We only load  2D/3D LUT（ Also known as lookup table ）. We trained the deep learning model offline , And extract  2D/3D-LUTs, Finally, it is implanted into our algorithm , Therefore, while achieving the same effect as the deep learning model , It avoids some disadvantages of itself .

 

Wuzhen waterway under the sunset , Before and after low illumination enhancement

 

2、ZEGO Advantages of low illumination image enhancement technology    

• FHD  real-time video  

FHD  Video refers to high-definition video with high frame rate , It can give users a better viewing experience . Because of the deep learning algorithm training  2D/3D-LUT, The algorithm can guarantee the quality of enhancement while processing in real time .

 

• Comfortable and natural visual experience

We believe that  " Less is more  " Idea . We deal with areas of different brightness to varying degrees , Avoid over exposure , Make the video look natural 、 comfortable . When we detect that the video is bright enough , We don't even do anything .

 

• Comprehensive coverage of mobile devices , Including low-end equipment

Benefit from offline deep learning methods and other innovative measures ,ZEGO  Low illumination technology even in low-end or old smart phones （ Like millet  2S  or  iPhone 4S） And Internet of things devices can also have excellent performance .

 

• Robustness under extreme lighting conditions

ZEGO  Low illumination technology has been tested in a large number of extreme lighting situations , Able to work well in extremely dark or bright conditions . We also conducted a comparative test with other friendly solutions , It is found that the latter will appear dazzle or flicker under extreme lighting conditions .

ZEGO Research and development ideas of low illumination image enhancement technology

ZEGO  That structure technology has an ai r & D team specializing in video enhancement , And has systematically established a set of complex artificial intelligence image quality enhancement algorithm . The team has been deeply involved in image processing technology for many years , Constantly surpass the existing technical means , Even in extremely low illumination scenes , It can also provide amazing picture quality , At the same time, it can be used on edge devices with high resolution 、 hfr 、 Run in real time .

1、 Lightweight deep learning model    

Although deep learning technology can play a great role in enhancing low light video , But it is difficult to run in real time on low-end models .

 

We use model training to get  2D/3D LUT  And load the look-up table in the running phase for low illumination enhancement . Based on this lookup table , The actual computing power and memory consumption of the algorithm are very small , So that the low-end models can also carry out real-time image quality enhancement .

2、 Brighten appropriately to different levels    

The algorithm will scan the whole image and evaluate the brightness level of different areas . We only brighten the areas with low illumination and effective information, and we won't deal with those areas that are already bright enough .

3、 Steadily increase the amount of light to avoid flickering  

In order to give users a better use experience , We will dynamically adjust the enhancement level of low illumination in a short period of time , So that it can smoothly transition to the target brightness , Avoid flickering of the picture caused by flickering of the ambient light .

 

If the transition of enhancement effect is not considered , When the ambient light changes frequently, it will cause the algorithm to switch the enhancement mode back and forth, so that the picture flashes . The algorithm will gradually brighten in a certain period of time , Ensure smooth transition of brightening effect , Make the naked eye look natural and comfortable . The idea behind all these efforts is people-oriented .

4、 Avoid detecting light changes between frames    

In practice , The light may be flickering , The illumination between two consecutive frames may be quite different . We will not detect the illumination change between two adjacent frames , Avoid additional computational costs .

 

Detecting light changes will consume some computing resources , This will slow down processing and introduce additional delays . besides , The detection algorithm is very complex , Its judgment may be wrong . Considering these limitations , We decided to avoid detecting light changes between frames .

5、 Conservative processing of image noise    

Image noise is a classical problem in image enhancement . There are various algorithms for processing image noise on the market , After comprehensive experiments and comparisons, we have summarized the following principles ：

 

• When we brighten the pixel brightness of the effective area ,
  It is necessary to try to avoid magnifying image noise
  . We design complex algorithms to detect image noise , And deal with them in different ways .

 

• Try not to suppress image noise
  , Because this is a heavy computing task , It will also slow down the processing speed . Besides , The treatment result is likely to be unsatisfactory .

 

• If necessary ,
  Take some conservative measures to suppress image noise
  . for example , To suppress one pixel in the noise , You can replace it with the average of the illuminance of the surrounding pixels .

summary

 ZEGO  That is to say, the technology of low illumination processing can

Intelligently perceive the brightness of the environment

, and

Intelligently adjust the brightness of the image to meet the comfort level of human eyes

, All with

Meet the user's perception as the standard

Intelligent adjustment of dark scenes .

 

future , That is, the technology video algorithm team will continue to deepen user needs , Committed to providing users with a more comfortable visual experience ！