Detailed explanation of MQ message oriented middleware theory

One . Basic theory of message oriented middleware

1. summary

Message queuing has gradually become an enterprise IT The core means of system internal communication . It has low coupling 、 Reliable delivery 、 radio broadcast 、 flow control 、 Finally, a series of functions such as consistency , Become asynchronous RPC One of the main means of . common mq Yes ：RocketMQ、RabbitMQ、ActiveMQ、Redis、Kafka、ZeroMQ etc. .

2. Message middleware consists of

• Broker： Message server , As server Provide message core services
• Producer： Message producer , The sponsor of the business , Responsible for the transmission of production messages to broker
• Consumer： Message consumer , The processor of the business , In charge of from broker Get the message and do business logic processing
• Topic： The theme , Unified message gathering place under publish and subscribe mode , Different producers turn to topic Send a message , from MQ The server distributes to different subscribers , Broadcast messages
• Queue： queue ,PTP In mode , Specific producers to specific queue Send a message , Consumers subscribe to specific queue Finish receiving the specified message
• Message： Message body , A packet encoded in a fixed format defined by different communication protocols , To encapsulate business data , Realize the transmission of messages

3. Message distribution strategy

1） Publish subscribe

Producers release information , All consumers who have subscribed to me can receive . Similar to live , As long as you subscribe to the live broadcast , You can watch the live broadcast when it's time .

Pub/Sub Publish subscribe （ radio broadcast ）： Use topic As a communication carrier

explain ：
Message producer （ Release ） Publish a message to topic in , There are multiple message consumers at the same time （ subscribe ） Consume the news . It's different from point-to-point , Publish to topic Will be consumed by all subscribers .

queue Load balancing is realized , take producer The production message is sent to the message queue , Consumption by multiple consumers . But a message can only be accepted by one consumer , When no consumers are available , This message will be saved until a consumer is available .
topic Implemented publishing and subscription , When you post a message , All subscriptions to this topic All of our services can get the news , So from 1 To N Every subscriber can get a copy of a message .

2） Poll distribution

The producer sends multiple messages , Consumer servers will each consume the same number , There will be no deviation due to the advantages and disadvantages of consumer servers .

3） Fair distribution

The producer sends multiple messages , Consumers will consume different numbers of messages because of the advantages and disadvantages of the server , The better the server , The more you spend .
Compare polling distributions , There will be consumers 2 High server performance , Will consume more news , That is, the principle that those who can do more work .

4） retransmission

Producer sends message , The message distribution mechanism sends messages to the consumer server 1, If the consumer server 1 Hang up , It will be sent to the consumer server 2, And so on .

5） Message pull

After the producer message is sent , Consumers pull passively .

4. Advantages of message middleware

1） system decoupling

There is no direct call relationship between interactive systems , Just by message transmission , So the system is not very invasive , Low coupling .

2） Improve system response time

For example, the original set of logic , Completing the payment may involve modifying the order status first 、 Calculate member points 、 Only by notifying the logistics and distribution can the logic be completed ; adopt MQ Architecture design , It will make the emergency important （ Need immediate response ） The business is put into the calling method , Use message queuing with low response requirements , Put it in MQ In line , For consumers to deal with .

3） Provide services for big data processing architecture

Through message as integration , In the context of big data , Message queuing is also integrated with the real-time processing architecture , Provide performance support for data processing .

4）Java Message service ——JMS

Java Message service （Java Message Service,JMS） The API is a Java About message-oriented middleware in the platform （MOM） Of API, Used between two applications , Or send a message in a distributed system , Asynchronous communication .
JMS Medium P2P and Pub/Sub Message schema ： Point to point （point to point, queue） And publish subscriptions （publish/subscribe,topic） It was originally made by JMS Defined . The main difference between the two modes or the problem to be solved is whether the messages sent to the queue can be consumed repeatedly ( Subscribe more ).

5. Message middleware application scenario

1） asynchronous communication

Some businesses don't want or need to process messages immediately . Message queues provide an asynchronous processing mechanism , Allows the user to queue a message , But not immediately . Put as many messages as you want into the queue , Then deal with them as needed .

2） decoupling

Reduce the strong dependence between projects , Adapt to heterogeneous systems . Predict what needs the project will encounter in the future at the beginning of the project , It's extremely difficult . The message system inserts an implicit 、 Data based interface layer , Both processes need to implement this interface , When the application changes , You can expand or modify the processing on both sides independently , Just make sure they adhere to the same interface constraints .

3） redundancy

In some cases , The process of processing data can fail . Unless the data is persisted , Otherwise, it will cause loss . Message queues persist data until they are fully processed , In this way, the risk of data loss is avoided . Used by many message queues ” Insert - obtain - Delete ” In the paradigm , Before deleting a message from the queue , Requires that your processing system explicitly indicate that the message has been processed , This ensures that your data is stored safely until you have finished using it .

4） Extensibility

Because message queues decouple your processing , So it is easy to increase the frequency of message enqueue and processing , Just add another process . No need to change code 、 There is no need to adjust the parameters . It is convenient for distributed expansion .

5） Overload protection

In the case of a dramatic increase in traffic , Applications still need to work , But such burst traffic can't be extracted and predicted ; If we think that we can deal with this kind of instantaneous peak visit as the standard to put resources on standby at any time is undoubtedly a huge waste . Using message queues enables critical components to withstand sudden access pressures , It won't crash completely because of a sudden overload of requests .

6） Recoverability

When a part of the system fails , It doesn't affect the whole system . Message queues reduce the degree of coupling between processes , So even if a process that processes the message fails , Messages enqueued can still be processed after the system recovers .

7） Sequence assurance

In most usage scenarios , The order in which the data is processed is important . Most message queues are sorted anyway , And it guarantees that the data will be processed in a particular order .

8） buffer

In any important system , There will be elements that require different processing times . Message queuing helps the most efficient execution of tasks through a buffer layer , This buffer helps to control and optimize the speed of data flow through the system . To adjust the system response time .

9） Data flow processing

Massive data streams generated by distributed systems , Such as ： Business log 、 Monitoring data 、 User behavior, etc , Collect and summarize these data streams in real time or in batches , Then big data analysis is the necessary technology of the Internet , It is the best choice to complete this kind of data collection through message queuing .

6. Common protocols for message middleware

1）AMQP agreement

AMQP namely Advanced Message Queuing Protocol, An application-level standard advanced message queue protocol that provides a unified message service , Is an open standard for application layer protocols , Designed for message-oriented middleware . The client and message middleware based on this protocol can deliver messages , Not by the client / Different middleware products , Restrictions on different development languages and other conditions .
advantage ： reliable 、 Universal
Personal understanding ： In fact, it is based on tcp/ip Protocol extension , Added some security 、 Some operations of stability , Than http/https A simpler and more efficient protocol , And it is a long connection protocol .

2）MQTT agreement

MQTT（Message Queuing Telemetry Transport, Message queuing telemetry transmission ） yes IBM Developed an instant messaging protocol , It may become an important part of the Internet of things . The protocol supports all platforms , It can connect almost all online items with the outside world , Used as sensors and actuators （ Such as through Twitter Let the house network ） Communication protocol of .
advantage ： Simple format 、 Small bandwidth 、 Mobile communication 、PUSH、 Embedded system

3）STOMP agreement

STOMP（Streaming Text Orientated Message Protocol） Is the streaming text oriented message protocol , It's for MOM(Message Oriented Middleware, Message Oriented Middleware ) Designed simple text protocol .STOMP Provide an interoperable connection format , Allow clients with any STOMP The message broker （Broker） Interact .
advantage ： Command mode （ Not topic\queue Pattern ）

4）XMPP agreement

XMPP（ Extensible message processing field protocol ,Extensible Messaging and Presence Protocol） It's based on extensible markup language （XML） The agreement , More for instant messaging （IM） And online live detection . For quasi instant operation between servers . The core is based on XML streaming , This protocol may eventually allow Internet users to send instant messages to anyone else on the Internet , Even if its operating system and browser are different .
advantage ： General public 、 Strong compatibility 、 Scalable 、 High safety , but XML The coding format takes up a large amount of bandwidth

4） The other is based on TCP/IP Custom protocol

Some special frames （ Such as ：redis、kafka、zeroMq etc. ） Not strictly following according to one's own needs MQ standard , It's based on TCP\IP Self encapsulating a set of agreements , Through the network socket Interface for transmission , Realized MQ The function of .

Two . Common message middleware MQ Introduce

1.RocketMQ

Alibaba is an open source distributed 、 Message middleware of queue model , Original name Metaq,3.0 The version name is changed to RocketMQ, It's Ali's reference kafka Use of design ideas java A set of implementations mq. At the same time, the Department of Ali will have a number of internal mq product （Notify、metaq） Integration , Maintain only core functions , Remove all other runtime dependencies , Make sure the core functions are the simplest , On this basis, we will cooperate with other open-source products mentioned above to realize different scenarios mq The architecture of , At present, it is mainly used in order trading system .

It has the following characteristics ：

• Can guarantee strict message order
• Provides filtering for messages
• Provides rich message pull patterns
• Efficient subscriber level scalability
• Real time message subscription mechanism
• 100 million message stacking capacity

2.RabbitMQ

Use Erlang Write an open source message queue , It supports a lot of agreements ：AMQP,XMPP, SMTP,STOMP, Exactly so , It's very heavyweight , More suitable for enterprise level development . At the same time Broker framework , The core idea is that producers don't send messages directly to queues , Messages are queued in the central queue when they are sent to the client . Routing (Routing), Load balancing (Load balance)、 Data persistence is well supported . More for enterprise level ESB Integrate .

3. ActiveMQ

Apache Next sub project . Use Java Fully support JMS1.1 and J2EE 1.4 canonical JMS Provider Realization , A small amount of code can efficiently implement advanced application scenarios . Pluggable transport protocol support , such as ：in-VM, TCP, SSL, NIO, UDP, multicast, JGroups and JXTA transports.RabbitMQ、ZeroMQ、ActiveMQ All support common multi language clients C++、Java、.Net,、Python、 Php、 Ruby etc. .

4. Redis

Use C Language development Key-Value Of NoSQL database , Development and maintenance are very active , Although it is a Key-Value Database storage system , But it supports MQ function , So it can be used as a lightweight queue service . about RabbitMQ and Redis The entry and exit operations of , Each executive 100 Ten thousand times , Every time 10 Ten thousand times record the execution time . The test data is divided into 128Bytes、512Bytes、1K and 10K Four different sizes of data . Experiments show that ： When you join the team , When the data is small Redis The performance is higher than RabbitMQ, And if the data size exceeds 10K,Redis It's too slow to bear ; When the team , No matter the size of the data ,Redis All showed very good performance , and RabbitMQ The out of team performance is much lower than Redis.

5. Kafka

Apache Next sub project , Use scala Implementation of a high-performance distributed Publish/Subscribe Message queuing system , It has the following characteristics ：

• Fast persistence ： Through the disk sequential read-write and zero copy mechanism , Can be in O(1) Message persistence under the system overhead of ;
• High throughput ： It can be achieved on a common server 10W/s The throughput rate of ;
• High accumulation ： Support topic The next consumer is offline for a long time , There's a lot of news piling up ;
• A completely distributed system ：Broker、Producer、Consumer All native automatic support distributed , rely on zookeeper Automatic load balancing ;
• Support Hadoop Data is loaded in parallel ： For things like Hadoop The same log data and offline analysis system , But it also requires the limitation of real-time processing , This is a viable solution .

6. ZeroMQ

The fastest message queuing system , Dedicated to high throughput / Low latency scenario development , It is often used in financial applications , Focus on real-time data communication scenarios .ZMQ Can achieve RabbitMQ Not good at advanced / Complex queues , But developers need to assemble multiple technical frameworks themselves , High development cost . therefore ZeroMQ With a unique non middleware pattern , More like a socket library, You don't need to install and run a message server or middleware , Because your app itself uses ZeroMQ API Complete the role of logical Services . however ZeroMQ Only non persistent queues , If down machine , Data will be lost . Such as ：Twitter Of Storm Use in ZeroMQ As the transmission of data stream .

ZeroMQ Sockets are not related to the transport layer ：ZeroMQ Socket defines a unified... For all transport layer protocols API Interface . The default support In process (inproc) , Interprocess (IPC) , multicast ,TCP agreement , Switching between different protocols simply changes the prefix of the connection string . You can switch from local communication between processes to distributed communication at least at any time TCP signal communication .ZeroMQ In the back processing connection establishment , Disconnect and reconnect logic .

characteristic ：

• Lockless queue model ： For cross thread interactions （ The client and session） Data exchange channel between pipe, The queue algorithm without lock is adopted CAS; stay pipe There are asynchronous events registered at both ends of , Reading or writing messages to pipe When the , Will automatically trigger read and write events .
• Algorithms for batch processing ： For bulk messages , We optimized the adaptability , Can receive and send messages in batches .
• Thread binding under multi-core , need not CPU Switch ： Different from the traditional multithreading mode , Semaphore or critical area ,zeroMQ Make full use of the advantages of multi-core , Each core binding runs a worker thread , Avoid multithreading between CPU Switching overhead .

Figure comparison ：

Later, specific project applications will be provided ,...........

It's not easy to create , It's time for dinner , Scan all the codes , Blogger real name authentication , Absolute truth .