Application of hard coding and streaming integration scheme based on spice protocol in cloud games

background

With virtualization technologies such as simulators , Development of containerization and other technologies , In the androyun game / In the cloud mobile phone scene , More and smaller granularity can be virtualized on the service host side  Android  example . The core technology is the graphics virtualization technology , How to make the most of the host side  GPU  Resources to render and encode , Do not consider the use of soft knitting  CPU  Resources are encoded for rendering because of the delay caused by efficiency .

Linux  Graphic stack

Let's first look at a more general  linux  Graphic stack ：

• X  agreement
  ： Earlier agreements ,X server  Direct management  GPU  Internal  framebuffer  and  X Client  Submission of orders , adopt  XClient（Xlib  or  XCB） towards  Xserver（Xorg） Submit relevant commands to realize , And there are many extension protocols , But the drawback requires an extra  Windows Manager  To handle multiple applications . It has been  Wayland  This extended protocol replaces ,composer  Process input , window , Composite display and other functions .

• GLX
  ： Because it is used for indirect rendering , Did two jobs ：1） take  OpenGL  and  X window API  binding  2） adopt  X server  forward  GL  Call to . In essence  X  The protocol .

• FB driver
  ： Historical legacy display subsystem , Provides  Framebuffer  obtain , Image manipulation primitive , Power management and other functions .

• OpenGL
  ： A unified  3D  Graphic rendering  API  Interface , Major manufacturers （Intel、 Nvidia、AMD、Qualcomm  etc. ） All supported interfaces , The mainstream implementation is open source  mesa.Mesa 3D  Is its most popular open source implementation , It is worth noting that  Mesa  Not only support  OpenGL, And support  Vulkan, Direct 3D  Wait for the rendering  API.

• DRM
  ：Direct Rendering Manager,  At present, the mainstream  GPU  Display subsystem , User mode use  libDRM  Of  DRM API  To operate  DRM  equipment , Yes  GPU  adopt  ioctl  And other standard file operations , Realization : framebuffer  management .  User mode Abstract rendering capability ： Such as  Buffer Object  management ,GPU  Operation order submission, etc , General and specific  driver  relevant . Virtual Driver  Support ： contain  vmwgfx（VMware  Bridging equipment ） and  virgl（Virto  Bridging equipment ） Prime Zero-copy memory,buffer  Through user mode  fd  The file descriptor represents the... In the actual video memory  DMA buffer, adopt  Prime API  export  FD, Can be in  IPC  Passed between .

contain  Wayland  All of the more mainstream image stacks have become extremely complex ：

The image data stream of each application is complex , But the general streamline is ： application （ Show  Client）->（ Show  Server）->OpenGL/EGL->Mesa 3D->libDRM->（ kernel ）DRM->GPU  drive .

Android  Graphic stack

With  SurfaceFlinger  At the core , Maintained all  app  Overlapping coverage of windows ：

• OpenGL ES：2D/3D  Rendering path , Use  drm  All functions for painting and rendering .

• Gralloc FB:  Use  drm  by  app  Provide functions such as video memory management .

• HWComposer:  call  openGL  Window Composition  RGB  perhaps  YUV, Realize screen drawing .

comprehensive  Linux  Graphics stack and  Android  The graphics stack can be found to be based on  drm  Realization , The core idea of implementing hard coding scheme is that both rendering and coding use the host side  GPU, And render and encode  Zero-copy, So there are two kinds of technologies ：

• virto-gpu  Technology will  OpenGLES  Command export （virgl） Then call the host side in turn  virglrenderer, Translate it back to  OpenGL  and  GLSL, Then call the host's  OpenGL, This part of the technical representative is  Qemu  programme . Use fake abstractions  GPU. In the abstraction layer  GPU  Layer to intercept , And call the host side  GPU.

2.  Direct export  DRM  Handle , utilize  DRM  Of  Zero-copy  Features for rendering and coding , Rendering and coding pass  IPC  Technology transfer  fd, This part represents  AiC（Android in Container） Containerization Technology .

3.  The above two types of technologies are finally  drm  Images  buffer, Therefore, the capital can pass  IPC  Technology passes between the rendering process and the encoding process  IPC  Pass on . The rendering process is usually in the container / In the simulator , The encoding process is usually outside the container .

Multimedia codec related

The above graphics stack involves display and rendering , In the cloud game scene , We also need to consider the technical stack of coding design . In terms of coding ：

• stay  Linux  in  ffmepg  perhaps  gstreamer  And other standard multimedia frameworks have encapsulated the application interface , The hardware is connected to provide encoding and decoding, such as  CUDA NVEnc  Interface or  VAAPI  Interface .

• stay  Android  Use in  OMX  As its multimedia framework ,MediaCodec  Drive docking  vendor  Driver to realize hard coding and decoding capability .

If you want to  Android  Hardware codec is used in , Or implement a set of  OMX  To  ffmpeg  Transformation translation of , Or the manufacturer can connect and realize  OMX  Of  vendor  drive , Otherwise, it is difficult to  Android（ In a container or simulator ） Internal hardware coding . A more reasonable way is to export  libdrm  Of  FD, Rendering and encoding in different processes , Code selection in  host  Call in  ffmepg  perhaps  vender  The coding  API  Encoding , And then complete the whole streaming .

programme

Hard coding currently focuses on processing the transfer of images between processes  prime FD, And the corresponding audio , two-way  input  Communication etc. . Adopt a unified  Spice  Agreement or modification  Spice  Agreement unification  Android  Virtualization and containerization consolidation solutions .

spice  agreement

SPICE,Simple Protocol for Independent Computing Environment, yes  Redhat  The company's open source remote desktop virtualization protocol , Designed to provide a business level remote desktop experience .Spice  The protocol has the following advantages ：

• Open source ： Easy to expand and customize ;

• Cross platform ：Windows/Linux/Mac OS  Full platform compatibility ;

• Support external devices ： Except common  USB  Out of equipment , Devices such as printers and scanners can also be used remotely ;

• Rich media support ： Including video 、 Audio 、 Images ;

• Smaller bandwidth footprint ：Spice  Built in image compression algorithm , Effectively reduce the bandwidth occupation during data transmission ;

• More secure data transmission ：Spice  have access to  OpenSSL  Encrypted transmission of data .

summary

There are four parts ： agreement 、 Client side 、 Server side and virtual machine side . among ：

• agreement ： It is the client side 、 The rules to be followed when the server side and the virtual machine side interact ;

• client ： Responsible for receiving and converting virtual machine data , And sending user input data to the virtual machine , So that users can interact with virtual machines ;

• Server side ： Integrated into the  Hypervisor  An internal user layer component , bring  Hypervisor（ Such as  QEMU） Support  Spice  agreement ;

• Virtual machine side ： Refers to all required components deployed inside the virtual machine , Such as  QXL  drive 、Spice Agent  etc. .

Image stream

The figure above shows the whole image from  Guest OS  To the client image transmission path , among ：

• QEMU: Virtual machine environment , Currently in use

• Guest OS: The operating system running in the virtual machine

• Client OS: Running on the  host  Side application

• GDI/X：Graphics Device Interface, Image engine , Display interface provided by image stack （ Such as  mesa）

• QXL： Device drivers , Provides a set of dynamic   The device needs the client's  QXL  Drive to play its full role . however , When there is no drive , The standard  VGA  It can also support the device . This mode also shows the boot phase of virtual machine startup .QXL  The device passes through the command and pointer ring , Display interrupt , Pointer event ,I/O  Port to interact with the driver .

QXL  Other features of the device include ：

• Initialize and map devices  ROM,RAM  and  VRAM  To physical memory

• mapping  I/O  port , Deal with reading and writing to manage ： Area update , command , Pointer notification ,IRQ  to update , Mode setting , Device reset , Keep a log, etc .

• Ring - Initializing and maintaining command and pointer rings , Get commands and pointer commands from the ring , Wait for a notice . Maintain the resource ring .

• Use  QXLWorker  Interface with the corresponding  red worker  signal communication , This is  red dispatcher  Implemented in , It translates device calls into messages written to  red worker  passageway , Or from  red worker  Read messages from the channel .

• register  QXL  Interface enable  worker  Be able to communicate with equipment . This interface includes  PCI  Information and functions （ Such as attaching to one  worker, Get the display and pointer commands from the ring , Display and pointer notifications , Mode change notification, etc ）.

• Define support  QXL  Mode and change the current mode （ Such as  VGA： All listeners reflect a single device ）

• Deal with in  VGA  Initialization shown in mode , to update , Change size and refresh .

VDagent  Command flow

Spice Server

Spice  Agreement transformation

Spice Client  received  Spice Server  From the end  main,display,playback  Wait for the passage :

• Show channels in the default  display  On the call  GTK widget  Related components draw images on the screen .

• obtain  playback  Audio data , adopt  gstreamer  Of  pipeline, call  alsa  Play the audio .

• Other mouse and keyboard processing , Do nothing about .

To suit our streaming transformation, the following ：

Major changes to the original agreement ：

• Display Channel  passageway , This part is getting  FD  after , Originally painted on  GTK  The process of  HwFrame  Adapter module , convert to  RTC  The data source required for encoding （YUV  perhaps  RGBA）.

• Main Channel  Channel increase  VDAgent  Type add custom type transfer  RTC  Remote call instruction , Reverse encapsulation will  RTC  The events and  DataChannel  Pass to  GameService（ Game management service process ）.

Spice  Agreement capture

Can pass  socat  Wait for tool agents  domain socket  To analyze  spice  agreement , For a complete  Spice  Protocol interaction process , adopt  TCP dump  Grab  wireshark  The log is as follows ：

Through the first  main channel  Establishing a connection , authentication , Then set up  Spice Display, Spice PLAYBACK, SPICE RECORD, SPICE INPUT  Equal channel , Finally, a specific message is sent through each channel .

Here we focus on the following :

• Main Channel  Of  VDAgent  passageway , stay  CLIPBOARD  and  FILE_XFER  Add  VD_AGENT_VENDOR_DATA, For remote  gRPC  call , receive  android  Side package data .

2. Display Channel

• GL_SCANOUT_UNIX,  Screen initialization / Message after resolution change , Generally used during initialization .

• GL_DRAW_DONE, This message is delivered when the screen content changes , It can be thought of as every Android screen .

3. Playback Channel,android  Sound message of the system , Such as volume change , Sound start and stop, etc .

QEMU  programme

QEMU  The scheme can directly reuse the  qemu+kvm  programme , In addition to exporting different for different hardware  fd  outside .

AiC  Containers

Compared to virtualization , What makes containerization special is  qemu  Has integrated  Spice Server  Components , If the virtualized container needs to be encoded outside the container, it needs to export audio , Video and control channels , Then implement a set of similar  Spice  The architecture of the protocol , For uniform compatibility, the forwarding module in the figure below is added  Adapter/SpiceServer, take  IPC  The channel is forwarded to again  Spice  passageway , Realization  QEMU/AiC  Unification of schemes .

Scheme parameters

In the whole integration solution , The following factors and parameters need to be considered ：

• Simulator or container environment . If you export  fd  The simulator and container solutions are consistent , No need to distinguish , Otherwise, you need to distinguish by environment variables or by passing in startup parameters .

• DRM device  Appoint . In carrying  GTK  You need to specify in the version of  Display  Of  device, remove  Xorg  After dependency, specify  Render node.

• Encoding video card hardware designation , Because different hardware codes are different , In the coding module, the current hardware information is required to determine the coding mode .

Graphics export

Export from virtual machine or container , There are two types of graphics memory  fd:

KHR_STREAM

Render to the host side  surface,suface  export  EGLSTREAM, adopt

eglCreateStreamKHR

and

eglGetStreamFileDescriptorKHR

Export the corresponding  EGLStreamKHR  File descriptor , Apply to  NVIDIA  The graphics card . The consumer side passes

eglStreamConsumerAcquireKHR

Export the corresponding  stream, But coding cannot be used directly  stream  type , CUDA  Provides  OpenGL  And  CUDA  interoperability  API, take  texture  perhaps  render buffer  binding  CUDA  After resources ,

CuGraphicsSubResourceGetMappedArray

Mapping out  CUarray  Pointer for encoder .

MESA_IMAGE

Render to the host side  texture,texture  Export to  DMA buffer, adopt

eglExportDMABUFImageQueryMESA

,

eglExportDMABUFImageMESA

export , Apply to  AMD/Intel  The graphics card . The consumer side creates  EGLImage,  And bind the  2D  texture , Put this texture's  textureID  To the encoder  VAAPI  Encode through this encoder .

Optimization and evolution

Code refactoring

As the types of supported schemes increase , The whole project is gradually difficult to maintain under the condition of meeting the functions , adopt  C++ Rewrite the modules , take  HwFrame  Module abstraction , On the log /SRE  Partition and reconstruction of each module , Modularize the project .

remove  Xorg

Xorg  As  X(11) In the agreement  Server  The implementation of the ,Spice Client  By calling  GTK API  End to do  client, There are drawbacks ：

• By default, the exported  fd, adopt  GTK widget  Draw in default  Display  On , But this step is not required in the actual streaming process .

• Deploy  Xorg  It also adds complexity .

You need to combine dependencies with  GTK  Component removal of , Reduce component dependency complexity and performance consumption .

To be specific ：

• Display channel  dependent  GTK widget  Dependency removal ,.

• Replace the original  Display, Yes  Nvidia,
  getPlatformDisplayEXT(EGL_PLATFORM_DEVICE_EXT, (EGLNativeDisplayType)dev[num], NULL)
  export .

• Replace the original  Display, Yes  VAAPI  Of  AMD  perhaps  Intel  The graphics card , Because of the  mesa  Graphic stack ,
  getPlatformDisplayEXT(EGL_PLATFORM_GBM_MESA
  export , One thing to note is that  VAAPI  Interface , Will be used for initialization  Display  Switch to  DRM  export .

#if ENABLE_GTK
int vaapi_init() {
 x11_display = XOpenDisplay(g_getenv("DISPLAY"));
 va_display = vaGetDisplay(x11_display);
#else
int vaapi_init(int drm_fd) {
 va_display = (uint64_t)vaGetDisplayDRM(drm_fd);
 g_message("drm_fd:%d va_dpy:%p", drm_fd, va_display);

#endif
 int major_ver, minor_ver;
 va_status = vaInitialize(va_display, &major_ver, &minor_ver);
 return 0;
}


remove  gstreamer

The audio  pipeline  Used  gstreamer, This part of the dependency can be removed .

Graphic conversion optimization

The overall idea is the image  Zero-copy, Reduce to  CPU  And  GPU  Copy between and convert between graphic formats .

The encoding card supports

Support the host to pass  PCIE  Insert an external hardware coding card for hardware coding .

Mixed hardware encoding support

The overall idea is to use  host  Rendering capabilities , Will render the  RGBA  perhaps  YUV  Export to encoding card , Maximize the use of rendering resources , Work to increase the number of concurrent paths .

Self upgrade

adopt  Host Gameservice  Process self upgrades firmware , Do not rely on overall deployment  pod  Node image update , It can be upgraded flexibly .

Monitoring and control  SRE

Manage system indicators and monitor performance , perfect  SRE  And other monitoring systems , The governance process crashes , stuck , Internal leakage detection, etc .

other

The implementation and online deployment of the video stream hard coding scheme of the whole cloud game cannot be separated from cross departmental cooperation , Once again, I would like to thank the internal brother teams, such as the basic system department, who worked together to design the whole scheme from the beginning to the launch  STE, multimedia  RTC  Etc , Through team collaboration, the whole scheme is promoted to go online and the subsequent online continuous optimization and governance .

About us

As a byte beating video medium stage Department , The video architecture supports the on-demand of byte based products 、 live broadcast 、 real-time communication 、 picture 、 Cloud games 、 Multimedia business development , The goal is to become the industry leader in multimedia solutions , Build the ultimate video technology / Product service experience ！

Video architecture - The equipment and service team focuses on multimedia +IoT/5G  Related fields , Incubate new scenarios and core technologies that enable business , Create the ultimate 、 Technical solutions combining software and hardware , Online cloud game 、 Cloud phone 、 Internet of vision 、 Multi screen interaction and other services , It supports Tiktok 、 Watermelon and many other internal products , It is also provided by volcanic engines  toB  service . Welcome more students to join us , Build the industry's top video innovation technology , contact

[email protected]

  Indicate the “ Equipment and service direction ”.

Reference resources

• https://en.wikipedia.org/wiki/Direct_Rendering_Manager

• https://en.wikipedia.org/wiki/Direct_Rendering_Infrastructure

• https://en.wikipedia.org/wiki/Mesa_(computer_graphics)

• https://source.android.com/devices/graphics

• https://www.opengl.org/

• https://www.mesa3d.org/

• https://www.spice-space.org/

• https://www.qemu.org/

• https://docs.nvidia.com/cuda/cuda-driver-api/index.html

​