How to build a liquid cooling data center is supported by blue ocean brain liquid cooling technology

With cloud computing 、 big data 、 Development and application of artificial intelligence and other technologies ,“ Count from east to west ” The implementation of the policy and the continuous expansion of the scale of the data center , The overall energy consumption of the data center is increasing . The traditional heat dissipation method can no longer meet the development needs of the current high-density data center . The emergence of liquid cooling technology , It improves the traditional heat dissipation form , And with high reliability 、 Low energy consumption and other advantages , Gradually applied to the data center .

1  The development stage of Data Center

since 20 century 60 s , The development of data center is divided into three stages :

One 、 Early stage

1996 year , The United States Exodus company ( Specializing in the construction of computer room facilities and bandwidth services ) First put forward “IDC”.

Two 、 The second stage

as time goes on , Data center hosting services began to be refined , From full server hosting to website hosting , The virtual hosting service appears .

3、 ... and 、 Cloud computing stage

Through virtualization technology and container technology , Completely realize the pooling of computing power resources of data center servers . be-all CPU、 Memory 、 Hard disk and other resources are managed by more powerful virtualization software , Then distribute it to users .

2  The structure of the data center

In terms of hardware type , The data center is similar to the internal computer room we have seen before , Just the specifications 、 Grade 、 A higher level of Management .

Now , With the change of national policy , domestic CPU The rise occupies an increasing share . domestic CPU use ARM framework , Higher cost performance , A lower cost .

On the whole , The hardware of the data center is divided into two categories , I.e. main equipment and supporting equipment .

One 、 Main equipment

The most basic main equipment in the data center is the server . Like desktop computers , contain CPU、 Memory 、 a main board 、 Hard disk 、 The graphics card (GPU)、 Power supply, etc .

Blue ocean brain server

Server rack

A common standard rack , The height dimension is usually 42U.1U be equal to 4.445cm, The frame width is 600mm or 800mm. There are many rack depths , if there be 600mm、800mm、900mm、1000mm、1200mm etc. . Generally speaking ,IT equipment ( The server ) The rack is deep (1100mm or 1200mm), Communication equipment is shallow (600mm).

IT Comparison of rack depth between equipment and communication equipment

In the rack IT Devices except servers , There are also professional storage devices such as disk arrays . With the development of big data , The amount of data generated by human beings is increasing rapidly every year , The number of storage devices and performance requirements have increased .

as everyone knows , The mainstream computer storage hard disk is divided into HDD and SSD.HDD It's a traditional mechanical hard disk , and SSD Is the growing popularity of solid state drives .SSD It belongs to semiconductor memory , Due to the fast storage speed and small volume , So it's very popular . But the price is more expensive . For data centers ,HDD It is still the mainstream choice of cost performance . at present SSD It is mainly used for high-end customers and high-performance business .

Speaking of switches , We have to mention a noun —— Overhead type .TOR Switch is a common term in the field of data center . seeing the name of a thing one thinks of its function , It's the switch on the top of the rack . This kind of switch is the lowest network switching equipment in the data center , Be responsible for connecting the servers in this rack to the upper switches .

TOR Switch location

actually , Rack switches are not necessarily placed on the top of the rack . It can be located on the top of the rack , It can also be located in the middle or bottom of the rack （ Usually on the top , Because it is conducive to internal wiring ）.

Further up the rack is a row of racks and n Row rack . Connecting these racks to servers requires network technology in the data center （ The most popular data center networking architecture is leaf ridge network （Spine-Leaf））.

Now , In order to transmit data with high bandwidth , Optical fiber is widely used in data centers instead of network cable . So optical fiber 、 Optical module and optical communication equipment (OTN etc. ) Become an important part of the data center .

Distributed deployment is now popular , There is a lot of data flow between data centers , High bandwidth requirements .

Optical communication backbone network equipment

Two 、 Supporting equipment

There are many types of supporting equipment and facilities on the ground floor , It mainly includes power supply and distribution system and cooling system , Fire fighting system 、 Monitoring system and building management system .

• Power supply and distribution system

  Power supply is the basis for the normal operation of the data center . The main function of the power distribution equipment in the data center is to turn on the electric energy / Turn off 、 Control and protection . The main power distribution equipment is the power distribution cabinet . The data center distribution cabinet is divided into medium voltage distribution cabinet and low voltage distribution cabinet .

  

  Schematic diagram of power supply and distribution in data center

  Except for power distribution cabinet , To ensure normal power supply in case of emergency , The data center will also be equipped with a large number of UPS( Uninterruptible power supply ) Even diesel generator sets . Data centers generally provide 48V DC and 220V Alternating current (ac) ( adopt AC-DC Conversion and DC-AC Inverse conversion ). DC is now becoming more and more data centers ( Such as Google) The choice of , Because the DC loss is smaller , Higher utilization of electric energy , In line with the development trend of high computing power and high energy consumption in data centers .

• Cooling system

  At present, data center refrigeration mainly includes two ways ： Air and liquid cooling .

  The wind is cold

  Air cooling generally adopts air-cooled air conditioning system . Like household air conditioning , The air-cooled air conditioner in the data center is also divided into indoor unit and outdoor unit . relatively speaking , Mature technology , Simple structure , Easy to maintain .

  Liquid cooling

  Liquid cooling , Use liquid as refrigerant to cool and dissipate heat . The thermal conductivity of liquid is that of air 25 times . In the same volume , The heat carried away by liquid is close to that of air 3000 times . In terms of noise , At the same heat dissipation level , The noise of liquid cooling is lower than that of air cooling 20-35 Decibel . In terms of energy consumption , Liquid cooling is less than air cooling 30%-50%. At present, liquid cooling technology is generally favored by the industry , But it's still in the exploratory stage . in general , The market prospect of liquid cooling is very broad , According to statistics, the market scale exceeds 100 billion .

  

  Blue ocean brain liquid cooling technology

  About refrigeration and heat dissipation, it is worth mentioning that , Cabinet pool level 、 Row level 、 Near end refrigeration methods such as cabinet level are emerging , Become the mainstream choice for new data centers . Traditional refrigeration is room level , The whole machine room is air-conditioned . The cooling path in this way is too long , Efficiency is too low , Unable to meet the heat dissipation requirements of high-power equipment , Energy consumption is also high . Cabinet pool level 、 Row level 、 The cabinet level is based on a cabinet pool 、 A row of cabinets or a single cabinet as the center for heat dissipation design .

  

  Cabinet row level heat dissipation , Take a row of cabinets as the object to design the air duct . The airflow path is significantly shortened , High heat dissipation efficiency . In addition to power distribution and cooling , The data center also has some equipment and facilities related to management and operation , For example, moving ring monitoring system 、 Building automatic control system 、 Fire fighting system, etc .

3  Development analysis of liquid cooling technology in data center

Data center plays an important role in the development of information society and digital economy , Provide calculations 、 Storage 、 Network capabilities 、 Social support 、 video 、 Normal operation of online shopping and other applications . As shown in the figure , In the energy consumption structure of the data center, the energy consumption of the cooling system accounts for up to 40%, Become an important factor for energy efficiency optimization in data centers .

With the state's attention to new infrastructure , The data center has ushered in a new wave of development , With the development of digital society , The data center will also play an increasingly important role , Promote data processing 、 Storage 、 Construction and development of transmission and other related equipment .

In terms of development , Liquid cooling technology has a long application history in aerospace and other fields , In the data center industry, the development is relatively slow . Some experts put forward the dual channel system of liquid cooling combined with air cooling technology , And test the experiment PUE Result reaches 1.1 To 1.2 Between ; Based on the sample data of liquid cooling heat dissipation in multiple cities , Build the energy consumption calculation model of refrigeration and heat dissipation system in different urban data centers , Effectively reduce the energy consumption of the data center system . Based on the energy-saving analysis of the data center , Adopt heat recovery technology , While reducing energy consumption in the data center , Improve the efficiency of heat recovery . Beyond seas , A large number of researchers have done a lot of research on reducing the loss of Data Center , There are many application cases , Such as IBM The company's SuperMUC Make full use of the refrigeration advantages of liquid cooling technology , Greatly reduce the loss rate of the data center .

Aiming at the technical application and development of data center liquid cooling technology in different industries , The following is mainly from the submerged 、 Cold plate 、 The development of three liquid cooling technologies of spray type is analyzed , The characteristics of the three liquid cooling technologies are systematically compared , The different characteristics of liquid cooling technology and the types of applicable application scenarios are summarized , It provides a reference for the scene application and technology development of liquid cooling technology in data center .

Analysis of liquid cooling technology

Liquid cooling technology is the technology of heat dissipation for electronic equipment , It is mainly different from the commonly used air cooling technology . Current liquid cooling technologies include submerged 、 Cold plate 、 Spray type, etc .

One 、 Submerged liquid cooling technology

Immersion liquid cooling technology through immersion heating devices , Make the device in direct contact with the liquid , Then heat exchange . According to whether there is phase change in the medium, it can be divided into two types: single-phase immersion and phase change immersion .

Submerged liquid cooling technology is mainly composed of coolant 、 Cavity structure 、 Heat exchange module and related connecting pipes and other facilities . among , The coolant is the heat exchange medium of the data center , High insulation 、 Low viscosity and super compatibility , It is the main medium of submerged liquid cooling technology . The surface heat dissipation performance of heating devices such as motherboard chip affects the heat transfer intensity of medium boiling to a great extent , If the chip surface is smooth , The heat transfer coefficient of the liquid on the heat exchange surface is high , So as to improve the heat emission rate . The way of installing heat dissipation cover is often used in the market , Increase the heat dissipation area of the chip , So as to improve the heat dissipation performance of the data center , Reduce the loss rate . Cooling module is the main heat dissipation module of submerged liquid cooling technology , Including liquid gas heat exchanger 、 Circulating pump and other accessories and facilities . The deployment of cooling module can affect the condensation heat transfer efficiency of coolant , Change the cooling limit of submerged system , This reduces the heat dissipation efficiency of the data center , Most of its modules are suitable for high performance 、 High heat flow calculation scenario , Like the dawn of Zhongke E Level 1 supercomputing equipment , Completely get rid of the restriction of air cooling and heat dissipation , Remove the air-cooled radiator , To configure 512 Compute nodes , The peak power consumption of the system is 249 kW,PUE reach 1.04, Effectively solve the problem of refrigeration and heat dissipation caused by high-density Computing .

Two 、 Cold plate liquid cooling analysis

The cold plate liquid cooling system is mainly composed of heat exchange and cold plate 、 Heat exchange unit and circulating pipeline 、 Cold source and other components . Cold plate liquid cooling transfers heat to the cooling liquid in the circulating pipeline , The heat generated by the server is taken away by the refrigeration characteristics of the liquid itself , Improve the cooling efficiency of the cold plate , Significantly reduce the energy consumption of the data center .

The heat exchange cooling plate passes through the air 、 Forced convection of water or other cooling medium in the channel , Take away the thermal energy in the server , It can effectively reduce the cost of data center PUE value . Cold plate materials are generally composed of materials with high thermal conductivity , Make the cold plate surface close to isothermal , Take away a lot of concentrated heat . And the application of cold plate reduces the pollution of cooling medium to electronic components , Improve the service life of data center facilities . The cooling capacity distribution unit is mainly vertical and horizontal , The low-temperature coolant is evenly distributed to each liquid separation unit through vertical and horizontal liquid separation units , Prevent the chip heat from being effectively dissipated , Reduce hidden dangers caused by node temperature rise , Improve the computing power of the system . The circulating pipeline is connected to the heat exchange and cooling plate 、 Important components of heat exchange unit and other related devices , It is generally divided into direct connection type and loop type , Mostly according to the different needs of data center construction , Select the appropriate pipeline for connection . The cold source acts as the final outlet to take away the heat generated by the data center , It is the main part of the cooling system . The low-temperature cooling water provided by outdoor cooling equipment is mostly used as the cold source in the heat exchange unit . The cold source is in indirect contact with the heated coolant , Cool down by taking away the heat of the coolant , Allow the coolant to enter the chip module at a low temperature , Then enter the heat dissipation cycle . therefore , Whether the selection of cold source is reasonable or not , It will directly affect the use efficiency and operation economy of the whole system , For example, Alibaba Qiandao Lake data center makes use of the natural resources of the surrounding environment , Adopt technologies such as lake water natural cooling system , send PUE reach 1.28, Effectively save the construction cost of Data Center .

3、 ... and 、 Spray liquid cooling analysis

Spray liquid cooling generally does not require significant changes to the infrastructure of the data center , Just adjust the spray head in the spray module , Realize the accurate correspondence between the spraying equipment and the server . Compared with traditional air-cooled servers , It can effectively reduce the heat generated by the device , To control the temperature of the data center . Spray liquid cooling server is the main internal device of the whole spray system , With its high performance 、 Low energy consumption 、 Features of highly reliable operation , It can effectively reduce the energy consumption of the data center . At present , Most advanced spray liquid cooling servers use chip level spray liquid cooling technology , The heat generated by the components is taken away by the coolant , After the coolant temperature rises , Then enter the heat exchange device of the cooling system for heat exchange .

Four 、 Comparison of liquid cooling technology

Yes, cold plate 、 Submerged 、 Spray liquid cooling technology , The details are shown in the table . It can be seen from the table that , Cold plate liquid cooling technology in maintainability 、 Space utilization 、 It has strong application advantages in compatibility ; But in terms of cost , Due to its custom cold plate device , The cost of technology application is relatively high . The spray liquid cooling technology is through the transformation of the old server and cabinet form , Greatly reduce the construction cost of data center infrastructure . The submerged technology is compared with the other two technologies , Although the maintainability and compatibility of the device are poor , However, it has good performance in space utilization and recyclability , Reduce energy consumption in the data center .

Analysis on the development of liquid cooling industry

One 、 Analysis of industry application problems

Liquid cooling technology has significant advantages in heat dissipation of data center industry , But in the data center industry, large-scale applications , We still need to improve the innovation and applicability of Technology , The following industry development problems need to be solved .

1、 Applicability of liquid cooling

The application of every emerging technology , Manufacturers are required to conduct long-term experiments , To ensure technical stability . And small-scale fixed-point experiments need to be carried out before mass application , Improve the practicability of liquid cooling . At present , Liquid cooling technology should further strengthen small-scale fixed-point experiments , And strive to apply it to the data center industry in batches .

2、 Reconfigurability of liquid cooling architecture

The deployment environment of liquid cooling mode is quite different from that of traditional computer room , And the deployment architecture also needs to be rebuilt . therefore , Using liquid cooling technology not only faces high deployment costs , It also increases the time cost of operation and maintenance personnel , Significantly increase the operation and maintenance cost of the data center , therefore , Further work is needed in the architecture 、 Optimize operation and maintenance .

3、 Standardization of liquid cooling design

Liquid cooling lacks mature industry norms in the process of data center construction and testing , Most enterprises carry out R & D according to their own demand characteristics , Failure to reach a unified standard in the industry . If there is no standardized and unified management , It will lead to the non-standard development of liquid cooling technology , Once a destructive failure occurs , Will completely crack down on the development of liquid cooling technology . meanwhile , Non standard products are also not conducive to cost and application , Therefore, we must speed up the standardization process .

Two 、 Industry development suggestions

Application of liquid cooling technology in data center industry , The process of gradual development of , Must rely on manufacturers 、 Promotion of users and industry organizations , In order to get a substantial improvement in the future development , Specific suggestions are as follows .

1、 Improve the universality of liquid cooling technology materials , Reduce construction costs .

For example, in the cold plate liquid cooling technology , There are many specifications for cold plate , This leads to the high cost of cold plate technology . If the raw materials and accessories for liquid cooling application are deeply studied , Reduce the cost of manufacturing liquid cooling raw materials and accessories , It will greatly improve the development speed of liquid cooling data center .

2、 Improve the operation and maintenance efficiency of liquid cooling technology .

The operation and maintenance mode of liquid cooling technology is quite different from that of traditional air cooling technology , Thus increasing the work cost of operation and maintenance personnel . therefore , All manufacturers and equipment manufacturers who need to produce accessories provide installation 、 Specification of commissioning and operation and maintenance . Through a standardized process , Not only ensure the stability of its application , And greatly improve the work efficiency of operation and maintenance personnel .

3、 Strengthen the verification of liquid cooling technology , Ensure reliable operation of the technology .

Reliability implementation of liquid cooling technology , It needs to rely on the equipment supplier 、 Service provider 、 Users and other enterprises . From energy saving 、 Reliability and applicability , Promote the operation of the pilot demonstration of liquid cooling technology , And on the basis of strengthening verification , Mass promotion in the data center industry .

4  Blue ocean brain liquid cooling solution

Blue ocean brain liquid cooling technology uses liquid as a heat transfer medium , Achieve the purpose of reducing the temperature of the equipment . The liquid is directed directly to the heat source to take away the heat , It does not need to be cooled indirectly by air like air cooling . For high density 、 Large scale 、 The data center with high heat dissipation demand has obvious advantages .      

Liquid cooled data center solution

For applications that require maximum performance 、 Density and efficiency of large system customers , Supercomputers provide liquid cooled cabinets , Supports direct liquid cooling of all components in a compact blade configuration .

The architecture contains many supporting high wattage CPU and GPU（ exceed 500W） The innovative function of , Greatly reduce interconnect wiring requirements , Reduce operating costs . Liquid cooled infrastructure also leads to a more compact system structure , Reduce the use of expensive optical interconnection cables .

Besides , Its infrastructure is carefully designed , Support multiprocessor architecture and accelerator options , At least in the next decade with the next generation CPU、GPU Keep forward compatibility with interconnection technology .

Cabinet architecture

                                              

The calculation of liquid cooling cabinet and the basic components of switch include ：

• Server cabinet ： One cabinet can hold 8 Chassis , Most support 64 A computing blade and 512 A processor .

• Computing blades ： The blade is a module in the chassis , Include CPU、 Structural connections 、 pcb , And cooling and power components . Pre install four double socket nodes . Insert the chassis from the front of the cabinet .

• Switch chassis ： Can accommodate 8 Switch chassis , Most support 64 A switch . Insert the chassis from the rear of the cabinet .

• Switch blade ： Including optical switch , Circuit board , Used to connect all components and power supplies required for computing blade and cooling .

Each cabinet contains 8 Server chassis and 8 Switch chassis , Connect directly from the switch to the blade , No wiring required . Switches and switch ports are directly liquid cooled . Place the blade vertically , The switch is placed horizontally , The switch can be directly connected to the blade .

The pre installed liquid cooled computing blade is equipped with four dual processor servers . All components are directly liquid cooled . The enclosure is not configured with a fan .

network

• 64  Oral exchange , Provide   12.8 Tb/s bandwidth

• Each switch chassis supports up to 8 A switch blade , Each computing blade supports up to 8 Fiber optic connections

• Each chassis 2 A switch , Used to direct to 32 Compute nodes （8 A computing blade ） Single point connection （ Each node has a network connection ）

• Each cabinet has 16 A switch , Used to direct to 256 Compute nodes （64 A computing blade ） Single point connection （ Each node has a network connection ）

• Between any endpoints , most 3 Secondary connection

Power Supply

The liquid cooling cabinet can support up to 300 Kilowatt power .

cooling

The cooling of the cabinet and all components is accomplished by circulating liquid throughout the computing facility . Cooling distribution unit （CDU） The hot water is discharged by cooling the liquid through the heat exchanger in the data center .

The whole cooling circuit is a source of CDU The closed loop . One CDU It can support up to four liquid cooling cabinets .CDU Keep the coolant at a certain temperature , And transfer the heat to the water in the data center through the heat transfer mechanism .

The liquid cooling passes through a series of flow pipes to each blade and component in the liquid cooler , These flow pipes distribute the coolant from the main pipe to each blade and switch , Then return the hot liquid to the pipeline . Cooling pipe equipped with computing blade in front of the cabinet , Cooling pipe with switch at the back . The connection between the computing blade and the switch blade is fast and leak free , The blade server can be removed for maintenance without shutting down the whole system . The cold plate is directly removed CPU Heat in . When NIC The mezzanine card is located in CPU Above , from CPU Cold plate cooling . The power and cooling budget of liquid cooled supercomputer are much smaller than that of air-cooled device .

5  How to build a data center ？

Before introducing the construction process , Let's first learn about the service providers of the data center .

Data center service provider

Although the data center is the infrastructure , But not entirely state-owned . It can also be private , Built and operated by private enterprises . But according to national laws and regulations , You must have an Internet data center operation license , To provide data center services . Generally speaking , Data center service providers are divided into three types of companies .

One 、 Telecom operators , Such as China Mobile 、 China Telecom 、 China Unicom .

Large state-owned enterprises , Strong capital , Strong talent , Abundant resources , Experienced in infrastructure construction , It is the leading force in the domestic data center market . Take China Telecom for example , As the largest IDC Service provider , They have 794 Data centers ,50 More than 10000 racks , The largest number , The most widespread .

Two 、 Cloud computing service providers , Like Ali 、 tencent 、 HUAWEI, etc. .

The provider itself carries out cloud computing business , Therefore, a large number of data centers are built by ourselves or jointly . Very professional in the construction and operation of data center , Strong technical strength . Take Ali for example , In the world 25 Hundreds of cloud data centers are deployed in different regions , In Zhangbei 、 Heyuan 、 nantong 、 Ulanchab 、 Hangzhou and other places have multiple super data centers , Hundreds of billions of investment .

3、 ... and 、 Third party service providers specializing in data centers , Such as universal data 、 Century interconnection 、 Halo new network 、 Baoxin software 、 CICC data 、 Aofei data 、 Data port, etc .

By building a data center or renting an operator's Data Center , Provide services to customers , It is not limited by the network and region of a single telecom operator , It can provide relatively balanced network exports . meanwhile , More flexible service , Be good at providing customized services according to the needs of customers .

According to different sales methods , The third-party data center is divided into retail data center and wholesale data center . For small and medium-sized Internet retail companies 、 General enterprises and other customers , Provide relatively standardized server hosting services and network bandwidth services . Wholesale data center , For large Internet companies 、 Cloud computing vendors or telecom operators , Provide customized services , Generally, the module is the minimum rental unit .

Construction of data center

The construction of data center is a highly professional project , Strong resource integration capability is required . In chronological order , The construction of data center inevitably includes the following series of professional projects :

• Site Selection

• land 、 Electric power 、 water 、 Energy assessment 、 Environmental assessment, etc

• feasibility study

• Project approval

• Scheme design and construction drawing design

• Project construction general contracting bidding procurement 、 And supervision bidding procurement

• Division of subcontracting boundary of professional subcontracting 、 Engineering subcontractor procurement 、 Procurement of main equipment

• Civil engineering

• Mechanical and electrical installation

• Equipment commissioning

• Acceptance delivery

In the data center location stage , Each data center service provider will according to its own needs and external conditions ( temperature 、 geological 、 Natural disasters 、 Power resources 、 Energy consumption index ) Plan the location of the data center .

The design of the data center is very complex , There are many factors to consider , A large number of professional design tools will also be used .

The planning idea of Data Center

The design includes not only the overall conceptual design of the building and the park , It also includes the detailed design of each subsystem of the data center . You need to output the design scheme and construction drawings of each detail , Facilitate subsequent construction . After the design is finished , Enter the real construction stage . The construction process will not be introduced one by one . It's nothing more than civil construction 、 Installation of supporting equipment and facilities 、 Installation and commissioning of main equipment and facilities, etc .

Every detail of the construction process , Countries have corresponding standards to regulate and restrict . Regular construction units can basically meet the requirements .

Several common construction steps

6  summary

Data center is an important information implementation in the digital age and an important carrier of computing power , Directly determine the digital competitiveness of a country . stay “ Count from east to west ” Driven by strategy , Domestic data centers will usher in another wave of development boom . According to statistics ,2022 China's data center business market will reach 3200.5 One hundred million yuan , The annual compound growth rate is 27.0%.2025 year , Domestic data center IT The investment scale is expected to reach 7070.9 One hundred million yuan . In addition to the rapid increase in quantity , The data center is developing towards green and intelligent , Actively introduce AI Artificial intelligence , Improve energy efficiency , Reduce operational complexity . Whether there will be new morphological changes in the data center in the future , We'll see .

Blue ocean brain liquid cooling server is a one-stop server for the construction of digital center , With super computing power density 、 High energy efficiency ratio 、 High network bandwidth 、 Easy to expand 、 Easy to manage and other advantages , Meet the deployment of enterprise computer room 、 Large scale data center cluster deployment and artificial intelligence computing cluster deployment .