Targeted, real-time audio and video optimization in remote control

Click on One click subscription 《 Yunjian coffee 》 special column , Get the official recommended boutique content , Learn technology without getting lost ！

5G Remote control scene , Delay of real-time audio and video transmission 、 Caton rate and anti weak network have very high requirements , This article will introduce how to combine 5G Network features , Joint optimization in real-time audio and video communication link , Meet the remote control needs of industry scenarios , Reduce picture delay .

In the last article , We introduced the technical points of remote control . Start with this chapter , The author will introduce the application and optimization of the three remote control technologies in turn . This paper will start with real-time audio and video communication technology , It is mainly used to solve the real-time transmission of the picture and sound of the controlled equipment or the surrounding environment of the vehicle to the remote control terminal in remote control , It is convenient for the remote driver or operator to clearly understand the surrounding conditions of the controlled equipment , So as to carry out targeted manipulation . For example, the front and side rear images of the vehicle moving forward , The grab arm pictures in the process of excavator operation need to be transmitted remotely through real-time audio and video technology .

In order to ensure the real-time and smooth control , Compared to the transmission of sound , Remote control mainly has very high requirements for picture transmission , For picture delay 、 This is especially true for core indicators such as Caton rate and anti weak network ability . Take the low-speed remote driving scenario as an example , The delay needs to be less than 200ms, Try to get close to 100ms, The Caton rate should preferably be less than one thousandth 2, Able to resist extreme conditions and average RTT Network fluctuation and with considerable delay 20%-30% About packet loss rate . The requirements of these indicators are often significantly higher than previous teleconferences 、 live broadcast 、 Monitoring and other application scenarios , For real-time audio and video technology , Reduce delay , And reduce the Caton rate 、 Improving the ability to resist weak networks is often contradictory , So this is a very big challenge .

Comparison of indicators between remote control and other application scenarios

Follow the map , Explore the key points of optimization

The following figure is a schematic diagram of a typical video transmission link , It is mainly collected by 、 code 、 send out 、 transmission 、 receive 、 decode 、 Rendering and other main modules .

Schematic diagram of typical video transmission link

collection ： The original image frame data is collected from the camera

code ： Encode the acquired original image frame

send out ： Package and send the encoded video frame

transmission ： Transfer the packaged data from the network

receive ： Receive the packed data and recover the video frame

decode ： Decode the video frame to recover the original image frame data

Rendering ： Render the original image frame data and output it to the screen

In real-time audio and video communication , It is mainly responsible for resisting network fluctuation 、 The responsibility for reducing the Caton rate is the in the receiving module jitterbuffer, At the same time, it is also one of the main contributors of time delay . In different projects jitterbuffer The implementation of is slightly different , But basically there will be out of order 、 Frame detection 、 Frame buffer and other functions .jitterbuffer It is mainly responsible for correctly receiving video frames and caching them properly , After confirming that the decodeable conditions are met , According to the estimated inter frame delay （ Time difference between two frames - Time difference between two frames ） After smoothing , Sent to the subsequent decoding and rendering module . In this way, even if the network fluctuates , because jitterbuffer Smoothing of , Adjacent video frames can still be rendered close to the desired time interval , So as to play smoothly . Usually in response to packet loss 、 Out of sequence and delay jitter , The Internet RTT And the greater the delay jitter , Needed jitterbuffer The greater the , At this time, due to the increase of cache , The video delay will increase accordingly . This is the root cause of the contradiction between the three indicators .

In addition to the receiving module , Let's look at other modules . With the increasing computing power of chips , code 、 decode 、 The delay of rendering and other modules has been very small , Basically all in 10ms within , It can even do 5ms about , There is little room for optimization , It has little impact on the three core indicators . Time delay of acquisition and transmission module , Mainly affected by external objective conditions , The former depends on the camera , The latter depends on the network . The sending module will lose packets for data transmission 、 Time delay and jitter , Receive effect from influence . Therefore, in order to achieve the three core indicators , The sending and receiving modules need to be optimized . Through the optimization of transmission module , On the basis of ensuring the Caton rate and the ability to resist weak networks , Minimize the number of receivers jitterbuffer Size , So as to reduce the delay .

have a definite object in view , Design optimization

For the joint optimization of transmitting and receiving modules , The implementation of different projects is not consistent , The complexity and effect are also quite different . The following is the implementation diagram of a more complex sending and receiving module in the real-time audio and video communication architecture . The real-time audio and video communication in Tencent's remote control products also uses such a composition .

Schematic diagram of transmitting and receiving module

The sending module is mainly composed of packet protocol 、 Congestion control 、 Send window 、 Error code, etc , In order to improve the transmission efficiency and anti weak network ability , Packet protocols are usually based on standards RTP agreement , The underlying the UDP agreement . Congestion control is mainly to estimate the network state , For sending pacing Make recommendations on window and bit rate . Error code , Mainly to resist RTP Packet loss , Improve forward error correction capability , In this way, partial packet loss can be recovered by error decoding without relying on retransmission .

In the receiving module jitterbuffer Out of order cache involved 、 Frame detection cache and out of frame cache , And unpacking 、 Error decoding 、 Link state estimation, feedback and other modules ; Link state estimation feedback , It is mainly used to estimate the packet loss of the link 、 Delay and delay jitter , Used to direct jitterbuffer Size design , It also provides reference for sender congestion control .

above-mentioned , The purpose of optimization is to reduce jitterbuffer size , The inter frame delay fluctuation is the main factor affecting jitterbuffer The core factor of size . In addition to the fluctuations of the network itself , Packet loss retransmission is the main contributor to the peak of delay fluctuation . Therefore, the first consideration of joint optimization of transmission and reception is to reduce packet loss and retransmission . Tencent is mainly in congestion control 、 Error coding , in the light of 5G The remote control scene has been better optimized , The probability of packet loss retransmission is reduced .

Congestion control ： At present, the common congestion control methods for real-time audio and video are , The better ones are BBR、GCC etc. .

BBR It is mainly based on the delay bandwidth product of the network , The maximum bandwidth and minimum delay of the network are detected respectively , It is considered that the product of the two is the maximum data capacity that can be carried on the network , Its advantage is that it can resist random network delay and packet loss fluctuation noise , The disadvantage is that the throughput will be reduced when the minimum delay is measured , For sudden network deterioration , It takes longer to reduce to the actual bandwidth . and BBR Originally not designed for video transmission , The application experience in real-time audio and video is limited .

GCC It is based on both delay congestion control and packet loss congestion control , And take the minimum of both . In delay congestion control , In order to smooth the influence of network fluctuation noise on delay gradient estimation ,GCC Kalman filter is used for processing .GCC The advantage is that both delay and packet loss can be considered , And have good practical application experience .

Error code ： In network transmission , The packet loss model can be understood as a deleted channel , Packets are randomly deleted during transmission . Therefore, forward error correction coding suitable for deleting channels can be used （FEC）, Recover packet loss by increasing the number of redundancy during packet transmission . Consider error correction performance and computational complexity , Linear block codes are mainly used for audio and video transmission , Commonly used XOR codes 、RS Code, etc . because FEC In design , Mainly for random errors , In a short coding length （ Number of coded packets ） Next , This method can resist a certain degree of random packet loss . But for the sudden packet loss caused by congestion or network quality degradation , Short encoding length still cannot resist , At this time, the tradition will increase the time interval between packets , And increase the coding length , To resist sudden packet loss .

be based on 5G Optimization and enhancement of air interface network

stay 5G In the remote control scene , Network time delay 5G The delay and fluctuation of air interface account for a large proportion , and 5G The network model of air interface is different from that of traditional router . Traditional routing is mainly congestion and packet loss , Without retransmission ; 5G There are both air interface error packet loss and congestion packet loss , Bring a certain retransmission ; The increase of traditional routing delay is mainly caused by congestion , 5G Air interface resource scheduling cycle , There will also be a certain degree of delay fluctuation , Especially for uplink data transmission .5G The bandwidth of the air interface is related to the signal-to-noise ratio and the air interface load , Will change over time ; The bandwidth of traditional routing is relatively fixed , Mainly affected by network load .

Router and 5G Comparison of air interface network characteristics

Optimization of congestion control ： It can be seen that 5G There are great differences between air interface network and traditional routing , In the face of delay jitter caused by resource scheduling cycle and bandwidth fluctuation caused by signal quality , BBR The applicability of congestion control is limited . in consideration of 5G The signal quality in the air interface will lead to great changes in the network bandwidth , So you can GCC Time delay 、 Based on packet loss congestion control , Add congestion control based on air interface signal to interference noise ratio and network load estimation , So for 5G The change of air interface network has faster response speed . At the same time, you can modify GCC Kalman filter algorithm used in time delay gradient estimation , Better smooth the delay gradient jitter caused by resource scheduling cycle .

Optimization of error coding ： be based on 5G Characteristics of air interface network , It can be seen that 5G The air port is retransmitted by itself , The probability of packet loss is less , Short coding length can be used to resist random packet loss . 5G The sudden packet loss in the air interface is often caused by 5G The sudden decline of air interface signal quality leads to , This period of deep fading is usually related to mobility , The faster you move , Shorter period , When moving at low speed 10ms about . The traditional way of simply introducing longer packet spacing and increasing coding length can not effectively deal with , And it will increase the amount of data sent , This leads to packet loss and deterioration . Combined with congestion control estimation based on air interface signal-to-interference noise ratio , This sudden packet loss can be predicted immediately , By reducing the bit rate , Extend sending time , Without increasing the coding length , Reduce the probability of burst packet loss . At the same time, packet interleaving can be introduced , Interleave the codes , Resist sudden packet loss to a certain extent .

Overall ,5G Remote control scene requires very high delay of audio and video , Although through combination 5G Network features , Some joint optimization has been carried out in sending and receiving , It can meet the remote control requirements of some medium and low speed industry scenarios , But far from the ideal of the industry 100ms Indicators still have some challenges , Especially in the cross regional remote control scene . In the future, we need to introduce more joint optimization methods combined with the network , In addition, it can also be considered in camera acquisition 、 More mining on coding , To maximize end-to-end results .

Mao Junling's previous wonderful articles are recommended ： Distance between people and equipment ,5G How to complete the remote control in the era

《 Yunjian coffee 》 It's Tencent cloud plus community boutique content column . Cloud recommendation officer specially invites industry leaders , Focus on the landing of cutting-edge technology and theoretical practice , Continue to interpret the hot technologies in the cloud era for you 、 Explore new opportunities for industry development . One click subscription , We will regularly push premium content for you .