Target segmentation for 10000 frames of video, less than 1.4GB of video memory, open source code | ECCV 2022

bright and quick From the Aofei temple
qubits | official account QbitAI

Why , How good Fujiwara Qianhua , All of a sudden “ High temperature red version ”？

This big purple hand , Is mieba alive ？？

If you think the above effects are just coloring the object later , That was really AI Cheated .

These strange colors , In fact, it is the representation of video object segmentation .

but u1s1, This effect is really indistinguishable for a time .

Whether it's cute girl's flying hair ：

Or a towel that changes its shape 、 Objects block back and forth ：

AI The segmentation of the target can be called perfect , It seems that the color “ weld ” Go up .

It's not just high-precision segmentation of targets , This method can also handle more than 10000 frame In the video .

And the segmentation effect is always maintained at the same level , The second half of the video is still silky and fine .

What's more surprising is , This method is right GPU Not very demanding .

The researchers said that during the experiment , This method consumes GPU Memory never exceeds 1.4GB.

Need to know , Current similar methods based on attention mechanism , You can't even process more than... On ordinary consumer graphics cards 1 Minute video .

This is the University of Illinois Urbana - A long video target segmentation method recently proposed by scholars at the University of champagne XMem.

At present has been ECCV 2022 receive , The code is open source, too .

Such a silky effect , still Reddit Attract many netizens to watch , Heat up to 800+.

Netizens are joking ：

Why paint your hands purple ？

Who knows if mieba has a hobby in computer vision ？

Imitate human memory

At present, there are many video object segmentation methods , But they are either slow to process , Or yes GPU Demand is high , Or the accuracy is not high enough .

And the method proposed in this paper , It can be said that the above three aspects are taken into account .

It can not only quickly segment long videos , The number of frames can reach 20FPS, At the same time, in ordinary GPU I can finish it .

What's special about it is , It is inspired by human memory patterns .

1968 year , Psychologists Atkinson and schifflin proposed Multiple storage model （Atkinson-Shiffrin memory model）.

The model says , Human memory can be divided into 3 Patterns ： Instantaneous memory 、 Short term memory and long term memory .

Refer to the above mode , Researchers AI The framework is also divided into 3 Memory mode . Namely ：

• Instant memory updated in time

• High resolution working memory

• Dense long-term memory .

among , The transient memory is updated every frame , To record the image information in the picture .

Working memory collects picture information from transient memory , The update frequency is every r Frame once .

When the working memory is saturated , It will be compressed and transferred to long-term memory .

When the long-term memory is saturated , Will forget outdated features over time ; Generally speaking, this will be saturated after processing thousands of frames .

thus ,GPU Memory will not be insufficient due to the passage of time .

Usually , Segmentation of the video target will give the image of the first frame and the target object mask , Then the model will track the relevant targets , Generate corresponding masks for subsequent frames .

The specific term ,XMem The process of processing a single frame is as follows ：

Whole AI Frame by 3 An end-to-end convolution network .

One Query encoder （Query encoder） Used to track, extract and query specific image features .

One decoder （Decoder） Responsible for obtaining the output of the memory reading step , To generate an object mask .

One Value encoder （Value encoder） You can combine the image with the mask of the target , So as to extract new memory characteristic values .

The characteristic value extracted by the final value encoder will be added to the working memory .

From the experimental results , This method is applied to short video and long video , It's all done SOTA.

When processing long videos , As the number of frames increases ,XMem The performance of has not decreased .

Research team

One of the authors is Chinese Ho Kei (Rex) Cheng.

He graduated from Hong Kong University of science and Technology , At the University of Illinois, Urbana - A doctoral degree at the University of champagne .

The research direction is computer vision .

Many of his papers have been CVPR、NeurIPS、ECCV Wait for the top to receive .

Another author is Alexander G. Schwing.

He is now at the University of Illinois, Urbana - Assistant professor at the University of champagne , He graduated from the Federal Institute of technology in Zurich .

His research interests are machine learning and computer vision .

Address of thesis ：
https://arxiv.org/abs/2207.07115

GitHub：
https://github.com/hkchengrex/XMem