Wonderful! The description of meituan Octo distributed service management system is too clear

OCTO Meituan is a distributed service communication framework and service governance system with hundreds of billions of calls , Service registration can be realized 、 Service auto discovery 、 Service management 、 Fault tolerant processing 、 Data visualization 、 Service monitoring alarm 、 Service groups, etc . This article summarizes OCTO Framework principle 、Java Application integration method 、 With the use of its console .

1 summary

OCTO yes octopus( octopus ) Abbreviation . It is meituan's company level infrastructure , Provide a unified high-performance service communication framework for all businesses of the company , Make the business have good service operation ability , Easy service registration 、 Service auto discovery 、 Load balancing 、 Fault tolerance 、 Grayscale Publishing 、 Data visualization 、 Monitoring alarm and other functions , Improve the efficiency of service opening 、 Availability and service operation and maintenance efficiency .

【 In particular 】OCTO It is a special system within meituan , Not open source , The system cannot be built externally . 【 Significance of this paper 】OCTO It is a heavyweight service governance system in China , At present, the amount of calls is 100 billion , By combing the architecture principle and application method of the system , It can help us deepen our understanding of distributed services .

1.1 Position in meituan's technical architecture

1.2 features

• Naming service - Service registration ; Service auto discovery

• Service management - Service status monitoring ; Service startup 、 stop it ; Service load balancing

• Fault tolerant processing - Shield abnormal services in real time , Automatically allocate request traffic

• Traffic distribution - Grayscale Publishing 、 Node dynamic traffic allocation and other scenarios

• Data visualization - Service call statistics report analysis , Provide clear data chart display , Clear understanding of dependencies between services

• Service group - It supports dynamic automatic grouping of services and user-defined grouping in different scenarios , Solve the problem of cross machine room call penetration in multi machine room scenario 、Sandbox Other questions

• Service monitoring alarm - Support multiple indicators of service and interface level 、 Multi dimensional monitoring , Support a variety of alarm methods

• Unified configuration management - Support unified management of service configuration , Flexible setting of differences between different environments , Support for historical versions , Real time distribution after configuration item change

• Distributed service tracking - Easy diagnostic service access is slow 、 Abnormal jitter and other problems

• Overload protection - Flexibly define quotas for service consumers , When the regulating amount exceeds the maximum threshold , Based on different service consumers QoS distinguish , Trigger overload protection .

• Service access control

1.3 Environmental Division

On the service provider's environmental branch line (IDC) And offline ( Office cloud ) Two systems , Offline system is a simulation of online system . There are... In each system test/staging/prod Three environments .

1.4 Calling process

• Parties (provider/consumer) stay OCTO Register your own appkey, such as appkey-provider/appkey-consumer

• provider stay OCTO Registration service on ( Marked as appkey-provider), The same appkey There are deployments in all three environments ;

• Suppose that staging Environmental Science Of consumer stay OCTO Request service on ( Mark yourself appkey-consumer, The goal is appkey-provider)

• OCTO Inquire about staging Environmental appkey-provider List of services for , And send it to consumer

• consumer adopt mtthrift visit IP:PORT service

1.5 understand appkey

With Nginx To understand by reference . For the traditional configuration , The mapping relationship between the domain name and the physical server is determined by Ngnix maintain , The increase or decrease of physical server needs to be adjusted by operation and maintenance personnel , Cannot dynamically complete ：

about appkey Configuration mode , Added appkey First floor ：

• The domain name and appkey The mapping relationship between Nginx To configure , There is no need to adjust in the future ;

• and appkey Mapping relationship with physical server , It can be adjusted dynamically .

Thrift Empathy , For client requests (appkey:port),Thrift Server adopt appkey Find the physical server (IP:port).

2 The overall architecture

2.1 MTransport( Service communication framework )

　　MTthrift Is based on Thrift（ from Facebook The source is Apache Thrift ） Secondary development , Is a distributed service communication framework , Committed to providing high performance and transparency RPC Remote service invocation scenarios , yes OCTO The core framework of the service governance program , Every day for 4000+ Services provide 2000 Billion + Secondary traffic support , It is widely used in all business lines of new Meida .

　　MTransport Is a multilingual service communication framework , It shields the implementation details of the underlying high-performance network communication , So as to realize simple and efficient service development .MTransport Support Thrift／HTTP／pigeon Such agreement . among Thrift Include MTthrift（Java）、PThrift（PHP）、CThrift（C/C++）、Turbo Thrift（NodeJS） etc. ,Thrift Support code implementation in different languages , Maintain the consistency of communication protocols , Support service registration 、 Service auto discovery 、 Distributed service call tracking, etc .HTTP Currently supported JAVA、NodeJs as well as C++.

　　MTthrift Provide service template management , Efficient tools such as code generation engine .

2.2 HLB( Elastic load balancer )

HLB yes Hardware Load Balance Abbreviation . all HTTP request / The response traffic will pass through the system , similar amazon elb.

2.3 SG_agent( Service governance agent )

SG yes Service Governance Abbreviation . SG_agent Deployed in each service node （ Service providers and consumers ）, Through and with MNS communicate , Service delivery registration / Find out 、 Configuration update 、 Access control 、 Quota restrictions and other functions , And report the call statistics to the performance monitoring platform .

2.4 MNS( Meituan naming service )

MNS yes Meituan Naming Service Abbreviation . MNS Is the service registry routing center , be based on ZooKeeper structure , Provide robust and reliable naming service management components for various distributed services of the company , Quickly realize service registration 、 route 、 Service auto discovery .

Summary of main services 、 node IP/Port、 Node weight 、 Storage of quota and other information / visit , And service health status detection, etc .

1. reliability ： (1) Uniformity ： No matter which server node is connected to the cluster , All presented are consistent data views . (2) Atomicity ： The update of the node is either successful , Or failure . (3) High availability ： stay 2n+1 In a cluster of machines , Even if n Machine failed , It still does not affect the high availability of the cluster .

2. De centralization ： Central MNS It mainly provides service registration 、 Find out 、 Routing policy and other functions , Other services are mainly provided by the services at each service node SG_agent Provide .

2.5 Data-center( Service data center )

Collect all access data of the company OCTO Report log data of business , Provide system performance index for each business line 、 health 、 Basic alarm, etc

2.6 Scanner( Health check system )

Scan the health status of each service , When not available from MNS Middle elimination .

2.7 MCC( Meituan configuration center )

MCC yes Meituan Config Center Abbreviation . Unified configuration center , Provide unified configuration management services , Realize the separation of configuration and code 、 The configuration information is updated in real time 、 High availability 、 version control , Improve service development efficiency , Lower maintenance cost . The principle is that the JSON The configuration file in format is stored in ZooKeeper Under the table of contents , When the user MSGP When changing the configuration , from MSGP notice SG_agent Pull data , take zk The configuration data in is landed in the specified directory of the local machine .

2.8 MSGP( Meituan service management platform )

MSGP yes Meituan Service Governance PlatForm Abbreviation . The goal is ： Provide registration services for various services of the company 、 government 、 The diagnosis , To configure , One stop management platform with quota and other functions .

3 Access method

Use thrift Provided @ThriftService、@ThriftMethod、@ThriftStruct、@ThriftField Etc , Notes on ordinary Java class , Make it a thrift Data model of （model） And service interface (service). Its use mode is similar to Dubbo Very similar ： Service providers and consumers are defined based on a common set of interfaces .

Based on SpringBoot establish interface/provider/consumer Three modules . Run on this machine provider, You can successfully register to dev Environmental OCTO On . Run on this machine consumer, You can successfully consume provider Services provided . This example has run through the meituan intranet .

3.1 Common interface service-interface

Introduce dependency package , And define interfaces DemoThriftService. The parameters used StudentParam / GenderEnum Must be marked with relevant notes .

3.1.1 pom.xml

<dependency>
    <groupId>com.meituan.service.mobile</groupId>
    <artifactId>mtthrift</artifactId>
    <version>1.8.5</version>
</dependency>
<dependency>
    <groupId>com.meituan.mtrace</groupId>
    <artifactId>mtrace</artifactId>
    <version>1.1.14</version>
</dependency>

3.1.2 StudentParam.java

/**
 * @author: kefeng.wang
 * @date: 2018-06-29 16:34
 * @description:  Students define ( As input parameter )
 **/
@ThriftStruct
public class StudentParam {
    private Integer id;
    private String name;

    @ThriftConstructor
    public StudentParam(Integer id, String name) {
        this.id = id;
        this.name = name;
    }

    @ThriftField
    public Integer getId() {
        return id;
    }

    @ThriftField(1)
    public void setId(Integer id) {
        this.id = id;
    }

    @ThriftField
    public String getName() {
        return name;
    }

    @ThriftField(2)
    public void setName(String name) {
        this.name = name;
    }
}

3.1.3 GenderEnum.java

/**
 * @author: kefeng.wang
 * @date: 2018-06-29 16:41
 * @description:  Gender definition ( As an output parameter )
 **/
@ThriftEnum
public enum GenderEnum {
    GENDER_MALE(1, "male", " men "),
    GENDER_FEMALE(2, "female", " women "),
    GENDER_UNKNOWN(0, "unknown", " Unknown Gender ");

    private Integer id;
    private String value;
    private String desc;

    GenderEnum(Integer id, String value, String desc) {
        this.id = id;
        this.value = value;
        this.desc = desc;
    }

    // @ThriftEnumValue
    public Integer getId() {
        return id;
    }

    public void setId(Integer id) {
        this.id = id;
    }

    public String getValue() {
        return value;
    }

    public void setValue(String value) {
        this.value = value;
    }

    public String getDesc() {
        return desc;
    }

    public void setDesc(String desc) {
        this.desc = desc;
    }
}

3.1.4 DemoThriftService.java

/**
 * @author: kefeng.wang
 * @date: 2018-06-29 16:19
 * @description: Thrift  Interface definition 
 **/
@ThriftService
public interface DemoThriftService {
    @ThriftMethod
    String getVersion() throws TException;

    @ThriftMethod
    StudentParam getGenderStudent(GenderEnum gender) throws TException;
}

3.2 Service providers service-provider

Introduce dependency package ：service-interface It is the public interface just defined ,hystrix For fault tolerance . In this module , First, implement the common interface , Then define the release related configuration , Run again ServiceProviderApplication Start the service provider .

3.2.1 pom.xml

 

<dependency>
    <groupId>com.meituan</groupId>
    <artifactId>service-interface</artifactId>
    <version>1.0.0</version>
</dependency>
<dependency>
    <groupId>com.netflix.hystrix</groupId>
    <artifactId>hystrix-javanica</artifactId>
    <version>1.5.12</version>
</dependency>

3.2.2 DemoThriftServiceImpl.java

/**
 * @author: kefeng.wang
 * @date: 2018-06-29 17:29
 * @description: Thrift  Interface implementation ( Service providers )
 **/
public class DemoThriftServiceImpl implements DemoThriftService {
    @Override
    public String getVersion() throws TException {
        return "1.0.0";
    }

    @Override
    @HystrixCommand
    public StudentParam getGenderStudent(GenderEnum gender) throws TException {
        return new StudentParam(1, " Zhang San ");
    }
}

3.2.3 DemoServiceProviderConfig.java

/**
 * @author: kefeng.wang
 * @date: 2018-06-29 17:45
 * @description: Thrift  Release ( Service providers )
 **/
@Configuration
public class DemoServiceProviderConfig {
    @Resource(name = "serviceProcessor")
    private DemoThriftService serviceProcessor;

    @Bean(name = "serviceProcessor")
    public DemoThriftService getDemoThriftService() {
        return new DemoThriftServiceImpl();
    }

    @Bean(name = "serverPublisher", initMethod = "publish", destroyMethod = "destroy")
    public ThriftServerPublisher getThriftServerPublisher() {
        ThriftServerPublisher serverPublisher = new ThriftServerPublisher();
        serverPublisher.setServiceInterface(DemoThriftService.class); // [MUST]  Interface class 
        serverPublisher.setServiceImpl(serviceProcessor); // [MUST]  Implementation class 
        serverPublisher.setAppKey(APPKEY_TEST_SERVER); // [MUST]  Service providers  appkey
        serverPublisher.setPort(9001); // [MUST]  The service provider listens on the port 
        return serverPublisher;
    }
}

3.2.4 ServiceProviderApplication.java

/**
 * @author: kefeng.wang
 * @date: 2018-06-29 17:50
 * @description:  start-up ( Service providers )
 **/
@SpringBootApplication
public class ServiceProviderApplication {
    public static void main(String[] args) {
        SpringApplication.run(ServiceProviderApplication.class, args);
    }
}

3.3 Serving consumers service-consumer

Introduce dependency package ：service-interface It is the public interface just defined . In this module , First, specify the service provider 、 Consumption options , Then use the common interface definition Controller To call , Run again ServiceConsumerApplication Startup service consumer . Start browser access http://localhost:8080/demo, Call successfully .

3.3.1 pom.xml

<dependency>
    <groupId>com.meituan</groupId>
    <artifactId>service-interface</artifactId>
    <version>1.0.0</version>
</dependency>

3.3.2 DemoServiceConsumerConfig.java

/**
 * @author: kefeng.wang
 * @date: 2018-06-29 18:01
 * @description: Thrift  consumer 
 **/
@Configuration
public class DemoServiceConsumerConfig {
    @Bean(name = "thriftPoolConfig")
    public MTThriftPoolConfig getMTThriftPoolConfig() {
        MTThriftPoolConfig thriftPoolConfig = new MTThriftPoolConfig();
        thriftPoolConfig.setMaxActive(100);
        thriftPoolConfig.setMaxIdle(20);
        thriftPoolConfig.setMinIdle(5);
        thriftPoolConfig.setMaxWait(3000);
        thriftPoolConfig.setTestOnBorrow(true);
        thriftPoolConfig.setTestOnReturn(false);
        thriftPoolConfig.setTestWhileIdle(false);
        return thriftPoolConfig;
    }

    @Bean(name = "demoThriftService", destroyMethod = "destroy")
    public ThriftClientProxy getThriftClientProxy(MTThriftPoolConfig thriftPoolConfig) {
        ThriftClientProxy thriftClientProxy = new ThriftClientProxy();
        thriftClientProxy.setMtThriftPoolConfig(thriftPoolConfig); // [ Optional ] To configure 
        thriftClientProxy.setServiceInterface(DemoThriftService.class); // [MUST] Interface class 
        thriftClientProxy.setAppKey(APPKEY_TEST_CLIENT); // [MUST] Serving consumers  appkey
        thriftClientProxy.setRemoteAppkey(APPKEY_TEST_SERVER); // [MUST] Service providers  appkey
        thriftClientProxy.setRemoteServerPort(9001); // [ Commonly used ] Service providers  port
        thriftClientProxy.setTimeout(30000); // [ Commonly used ] Call timeout 
        return thriftClientProxy;
    }
}

3.3.3 DemoConsumerController.java

/**
 * @author: kefeng.wang
 * @date: 2018-06-29 18:10
 * @description: Thrift  The external entrance to the demonstration 
 **/
@RestController
public class DemoConsumerController {
    private static final Logger logger = LoggerFactory.getLogger(DemoConsumerController.class);

    @Resource
    private DemoThriftService demoThriftService;

    @GetMapping("/demo")
    public StudentParam demo() {
        try {
            return demoThriftService.getGenderStudent(GenderEnum.GENDER_MALE);
        } catch (TException e) {
            logger.warn(e.getMessage(), e);
        }

        return null;
    }
}

3.3.4 ServiceConsumerApplication.java

/**
 * @author: kefeng.wang
 * @date: 2018-06-29 17:50
 * @description:  start-up ( Serving consumers )
 **/
@SpringBootApplication
public class ServiceConsumerApplication {
    public static void main(String[] args) {
        SpringApplication.run(ServiceConsumerApplication.class, args);
    }
}

4 Publishing platform (plus)

MtThrift Is in Thrift Customized modifications have been made on , Enables the code to be in Plus After release, it can be OCTO Platform discovery processing .

5 Service governance platform (MSGP)

test / staging / prod Each environment has its own WEB Management platform ( Within the company or through VPN Ability to visit ).

For the sake of information security , Relevant screenshots are not provided one by one . Common functions include ：

• Service details / Service providers ： List current appkey Each host （ Host name /IP/PORT）, You can add or delete hosts , Adjustable weight , You can enable or disable ;

• Service details / Serving consumers ： View the current by time period appkey The consumer and its consumer provider host 、 Call volume, etc ;

• Service operations / Service group ： Same center priority can be set 、 The same machine room is preferred ;

• Data analysis / Source analysis ： Count the current by time period appkey Call of upstream service （ Adjust dosage 、QPS、 Time consuming, etc ）;

• Data analysis / Analysis of whereabouts ： Count the current by time period appkey Invocation of downstream services （ Adjust dosage 、QPS、 Time consuming, etc ）;

• Data analysis / Host analysis ： Count the current by time period appkey Call status of each host （ Adjust dosage 、QPS、 Time consuming, etc ）.

Thank you for your reading , If you have better ideas for this article , Welcome to leave a message in the comment area ！