Embracing cloud Nativity: Practice of Jiangsu Mobile order center

In recent days, , The products of Jiangsu Mobile order center constructed by Haojing technology successfully passed the ICT Institute  “ Cloud native business application maturity ” Excellent （L4） Certification assessment for , It has become the first business application that has passed the certification in China .

The following is an excerpt from 《 Cloud native Capability Maturity Model   The first 2 part ： Business applications 》：

-- Cloud native ： Cloud nativity is an idea for cloud application design , The best practice path to make full use of cloud efficiency , Help enterprises build flexible and reliable systems 、 loose coupling 、 Easy to manage 、 Observable application system , Improve delivery efficiency , Reduce the complexity of operation and maintenance .

-- Cloud native business application maturity model ： From the enterprise business application infrastructure domain 、 Application R & D domain and service governance domain 、 Twenty process areas comprehensively evaluate the flexibility of enterprise business application 、 High availability 、 Self healing 、 Cloud native capability maturity level in five dimensions, including observability and automation .

This also marks that the cloud native business application architecture of the product has reached the domestic leading level , Today, we share with you the order center products developed by Haojing technology based on cloud native , Understand the current situation , How can the order center solve the contradiction between high availability and rapid business development .

Construction background ： Business development changes rapidly

With the mobile Internet entering 5G Time ,O2O、 seckill 、 Group purchase 、 Live delivery 、 New retail and other new business models are emerging , The operator market has also expanded from the traditional individual family users to the diversified businesses of government and enterprise users , The integration between traditional business and new business will be closer , The order mode puts forward new support requirements .

This trend of innovation and integration development , It brings new challenges to the order service of operators ：

• With the expansion of emerging businesses such as government and enterprises , Orders are growing rapidly , Original application + The database architecture is under great pressure , One side , The application of chimney type is difficult to expand laterally , On the other hand , New data components need to be introduced to buffer database pressure .

• seckill 、 Emerging businesses such as live broadcasting and goods delivery have brought about a sudden increase in order volume in a short time , How to deal with the requirements for host resources in a short period of time and reduce the expansion cost as much as possible ？ We need to consider whether we can make a trade-off between our own services and external requests when the system is overloaded , Ensure the stability of core services , Realize the resilience of the system .

• In order to seize the new market , Demand lead times are getting shorter , Versions are launched more frequently , The traditional R & D delivery mode of manual workshop can no longer meet the business delivery requirements , Need to be in DevOps Introduce the R & D efficiency tool system under the concept .

The order center system is becoming larger and larger , It involves complex technical components and architectures , Especially in the process of supporting government and enterprise businesses , Nonstandard 、 Long process and multi system collaboration scenarios are very common , Once the order center itself or the surrounding system is abnormal , There is a chain reaction of various components , Generate a large number of alarms , How to quickly locate the root cause of the problem , How to realize system self-healing 、 Realize no perception at the business level , How to present the whole system in a visual way 、 Realize digital twinning .

Problem solving ： Embrace cloud native

To solve these problems , The products of Haojing technology order center focus on the comprehensive cloud 、 Microservices 、DevOps、 Intelligent operation and maintenance , Driven by technological development IT The support system provides innovative value for the business , Fully implement the cloud native architecture from four aspects ：

Full cloud ：

The application completes the containerization transformation , Cloud based PaaS platform , Adapt to the base of rock foundation , Using cloud native technology components .

Microservices ：

Domain Driven Design Guides microservice transformation , The order center is divided into several micro service lines , structure “ tube 、 Diagnosis 、 cure ” Micro service governance system , Provide visual service orchestration capabilities .

DevOps：

Build a tool system integrating research and operation , Realize the agile R & D of the order center 、 Fast delivery 、 Shorten the iteration period , Respond quickly to customer needs .

Intelligent operation and maintenance ：

Realize multi-dimensional monitoring and alarm , Monitoring linkage and unified display of business and system , Fast location and recovery of support failure , Realize normalized fault attack and defense drill .

Cloud native overall architecture

Build a tool integrating research and operation to realize agile research and development 、 Fast iteration . Through the micro service splitting of the order center, the cloud native architecture and the gray-scale publishing mechanism , Upgrade or failure does not affect the business , Quickly find problems through multidimensional monitoring , Quickly locate the problem through the call chain , Through unified service governance and automated container scheduling, the system can achieve flexible scalability , Ensure that the order center

7*24

Hours high availability .

The order center can be divided into infrastructure layer from bottom to top 、 The data layer 、 Application server 、 Application client , There is also a unified monitoring and operation center ：

• Infrastructure layer ： The order center runs on a virtualized infrastructure , platform PaaS Components running on virtual machines , Business applications run on containers .

• The data layer ： A large number of order centers use cloud native PaaS Components , Include redis、MQ、ElasticSearch etc. , Reduce the pressure on the database while ensuring the high availability of the data layer .

• Application server ： Based on microservice service, several microservice lines have been built , The unified service governance of multiple micro service lines is realized by constructing the micro service gateway of the order center , Ensure high availability of the order center .

• Application client ： Multi channel and multi type client access requests , Through the first Nginx Then enter the order center . The order center deploys the production center 、 Gray scale center and disaster recovery center ,Nginx It is mainly used for switching between multiple centers .

• Unified monitoring of operation and maintenance ： Build an integrated research and operation tool to realize agile research and development 、 Fast iteration , Realize multi-dimensional unified monitoring and display of business and system through unified log and operation and maintenance monitoring , The problem can be quickly located through the call chain .

Cloud native application practice

Cloud native practice  1： Cloud native PAAS platform

Bottom PaaS All platforms use cloud native components , Such as Redis、MQ、ES etc. , Make sure the bottom layer PaaS High availability of components .

Use Redis Cache configuration data and hotspot instance data . Load the physical library query results into the cache when the order is queried for the first time , In transit orders are stored in the hotspot cache cluster , The completion is stored in the normal cache cluster , Subsequent order queries start from the cache , In this way, the access pressure of the database can be greatly reduced .

Synchronize the index fields of strong search requirements in the order and its extended information to ElasticSearch after , You can separate the query service , Thus, without affecting the operation of the order process , Improve order query efficiency , The order query performance ranges from 600 pen /s Upgrade to 3000 pen /s.

Use MQ Make order receipt , Realize peak cutting and valley filling through message service , Improve order receipt efficiency ; utilize MQ Realize asynchronous interface log consumption ; Realize cache synchronization among order centers through message service .

Cloud native practice  2： Micro service

According to the domain driven design （DDD） methodology , The order center is divided into several loosely coupled micro service lines according to business areas , Attend to each one's own duties , Reduce the impact of correlation , Shorten the iteration period . The database is also split accordingly , Reduce the impact of the fault . As shown in the figure below , The order center is split into group number cards 、 Provincial number card 、 Broadband 、 Gathering customers 、 Electronic protocol 、 Dispatching and other micro service lines , Each microservice line has its own database , Thus, application isolation and data isolation between micro service lines are realized , Flexible expansion on demand .

Cloud native practice  3： Micro service governance

The order center uses the microservice gateway to implement service governance , Use Spring Cloud Gateway frame , In addition, authentication service and micro service management and control platform SCP(Sentinel) A complete set of , Realize unified security authentication for multiple micro service lines in the order center 、 Business diversion 、 Service tracking 、 Monitoring diagnosis 、 Current limiting 、 Fuse 、 Degradation and other service governance capabilities , Automated traffic management using policy configuration , Realize the manageability of services 、 Observable and automated operation and maintenance .

Key capabilities of microservice governance ：

• Service failure ： The service of the order center can be configured , When the number of call failures in the downstream service unit time reaches the threshold , Prevent the whole system from chain reaction or even avalanche , Temporarily stop the call to the service .

• Service restriction ： When an entry requests  QPS  When the service configuration threshold is exceeded , For example, when high concurrent order query requests occur instantaneously , The order query service can be configured to restrict query requests from certain channels , To ensure that this service is not overwhelmed .

• service degradation ： As shown in the figure above , When the service pressure of the order center reaches a certain level , The order center is configured with a service degradation policy , Downgrade highly tolerant Services , For example, the query service is degraded , To ensure the stability of core services such as order creation service .

Cloud native practice  4： Integration of research and operation

introduce DevOps Integrated research and operation tool , Implement requirements management 、 Iteration plan 、 Source control 、 Source code compilation package 、 The static test 、 Create a mirror image 、 automated testing 、 Continuous release deployment 、 Full stack monitoring 、 The R & D of automatic operation and maintenance is the tool support of the whole assembly line of operation and maintenance , Realize agile R & D of business system 、 Fast release online 、 Shorten the iteration period , Respond quickly to customer needs .

Cloud native practice  5： How to live / Gray switch

Before production on the order center version, the verification must be completed in the gray center . The version is released in the gray center , Adjust routing policy , Switch part of the traffic to the gray center , function 2 It will be officially released to the production center after it is verified that there is no problem .

How to live / Key capability of gray switching ：

• Grayscale routing switching ： After the new version is released in the gray Center , Adjust routing policy , Switch part of the traffic to the gray center , function 2 It will be officially released to Nanjing Center only after it is verified that there is no problem .

• Multi center cache synchronization ： During the release , The cached data between the two centers needs to be synchronized , adopt MQ The mutual notification mechanism loads the changed data into the cache of each center , And through the database log 、 Additional means such as message monitoring ensure the correctness and consistency of multi center cache synchronization , The seamless switching between production center and gray center is realized .

• Data layer version control ： There are version number fields in the database tables , It can identify the data involved in the old and new versions , It can be used for problem location and version rollback , Realize the gray-scale publishing of the data layer .

Cloud native practice  6： Multidimensional monitoring and operation and maintenance

Through system monitoring 、 Unified collection and analysis of logs , Realize the visualization and automatic alarm of multi-dimensional monitoring of the system . Quickly locate the problem through the call chain . It can automatically trigger the elastic expansion of the application container according to the resources, business indicators or fault conditions , Ensure business 7*24 Hours of uninterrupted operation .

Multidimensional monitoring, operation and maintenance key capabilities ：

• Realize log monitoring and call chain output through unified log collection , Quickly locate the problem through the call chain .

• Have comprehensive monitoring, display and analysis of business and system , Realize the monitoring linkage between business and system , It is convenient to quickly locate the cause of abnormality .

• Support the closed-loop management of alarms ： Support alarm collection 、 Inform the relevant responsible person 、 Alarm timeout reminder 、 Problem handling and recording 、 Alarm recovery confirmation 、 Alarm off .

• Operation and maintenance personnel use systematic operation and maintenance tools , Can be found in 5 Quickly find the root cause of the problem and deal with it within minutes , Greatly reduce the time required for overall fault repair .

Cloud native practice  7： Normalized fault injection

The order center system is becoming larger and larger , It is difficult to assess the impact of upgrades or potential failures . Through normalized active fault injection , Verify the fault tolerance of the system architecture , Identify potential problems ahead of time , Iteratively improve the architecture and operation and maintenance methods , Ultimately, business resilience .

Order center solutions ： Simulate the sudden high concurrent order or query and accidental failure of the order center , The following fault drill scheme has been customized ：（ The following examples are common drill scenarios ）

CPU Full load ：

The scene that ： Simulate some nodes cpu Full load , Lead to cpu The average utilization rate remains high

The effect of the drill ： The system monitors cpu After high load , Automatic horizontal capacity expansion node , until cpu The load remains at normal level

Service delay ：

The scene that ： Simulate the service delay of some nodes , This leads to high overall service delay

The effect of the drill ： The system automatically expands the nodes horizontally , Until the delay falls back to the normal level

Pod End ：

The scene that ： Simulate the service termination caused by the failure of some nodes , Actually stop some nodes

The effect of the drill ： The system finds that some nodes are down , Automatically pull up new Pod node

The value of cloud native construction

1、 By introducing DevOps, Realize the agile research and development of order system 、 Fast release online 、 Shorten the iteration period , Respond quickly to customer needs .

Realize agile R & D of business system , In particular, the rapid support of the diversified business of government and enterprise leaders , The average online cycle of demand is shortened 50%.

• Early verification of version through online gray-scale release , Seamless switching between production and grayscale is realized .

• Through the implementation of the version of the rapid release online 、 Shorten the iteration period , Launch one version a week .

• Monthly R & D capacity increases 50%.

2、 Cloud native technology reduces database pressure while achieving high performance , adopt redis Cache configuration data and hotspot instance data , By using ES Speed up order query , By using MQ Improve order receipt efficiency 、 Realize multi center cache synchronization .

• Order creation performance improved to 5 Ten thousand brush / minute .

• The order query performance has been improved to 3000 pen / second .

3、 Meet the second kill 、 Live broadcasting and other emerging businesses as well as government and enterprise businesses 2C The demand of the sudden increase in the peak of the order volume , Ensure that the order center is highly available in real time .

The order center has load balancing capability , Horizontal expansion and contraction capacity including overload , Ensure the reliability of the application itself .

• Realize fault isolation and recovery ： Deal with failure , Through active control of channel access points , Connect transactions to a functioning access point , Complete isolation of faults . Through the automatic restart and capacity expansion and contraction of the fault point, the fault recovery can be realized .

• When the system pressure reaches a certain level , It can effectively realize current limiting and high-risk fusing / Degraded processing .

4、 Realize multi-level and three-dimensional monitoring and visual display , Improve operation and maintenance efficiency through the linkage of business and system monitoring , Quickly locate the problem through the call chain , Support the closed-loop management of alarms , It can automatically trigger the elastic expansion of the application container according to the resources, business indicators or fault conditions , The observability and automatic operation and maintenance of the system are realized .