Message queue table structure for storing message data

【 Design thinking 】

• One queue corresponds to one table , A queue only processes one business message , Therefore, the unified message data format reduces the complexity of message processing for producers and consumers , And simple 、 Efficient 、 Easy maintenance 、 Easy to expand ;

• Creating a queue will redundancy the same table structure on each database , The table name contains the queue name ,SDK  Polling writes and reads messages ;

• Multiple message queuing patterns need to be considered , For this system, we only consider two queue modes , That is, work queue mode and publish subscribe mode .

Work queue mode ： The message consumer client allows one or more , But each consumer takes turns getting messages from the queue , Each message is read and consumed only once ;

Publish subscribe mode ： The message consumer client allows one or more , Each message needs to be consumed once by each client ;

• You need to consider the reliability of client consumption messages , It is divided into client confirmation and unconfirmed , Corresponding to different scenarios with different messaging reliability requirements ;

• news ID  Is a key element of message queuing , You need to include meaningful information , Such as ： Creation time , Serial number 、 producer 、 Business type, etc , To ensure rough order and improve index efficiency , It is convenient for fault diagnosis and subsequent data repair and processing in case of message delivery failure , such as ： retransmission 、 Rereading or business data correction, etc .

• You need to record where consumers read messages and log them , So that the last message read from ID  Continue reading unread messages at , Consumers can be configured to read one or more messages at a time .

【 Overall architecture diagram 】

【 Table structure design description 】

news ID Format

explain ：

It can support each message server to generate at most for each sub service in a single second 2 Of 20 Power messages , It supports millions of messages per second ID Generate , Sufficient to meet the needs of business scenarios .

<

queue_name

>_msg_tbl：

explain ：

1）msg_create_time + serial_number As a composite index , You can query messages according to time and serial number .

2） Primary key ID  It is ordered and contains meaningful information , So primary key ID It can also perform range search and precise search .

SELECT * FROM user_registration_msg_tbl WHERE id > 18446744073709551611 LIMIT 1;

The query performance of this statement can reach range, Therefore, the performance is guaranteed .

<queue_name>

_consumption_position_tbl

explain ：

1）consumer_id  When a message queue is created, it is generated according to the message queue pattern , The format is the same as that of the message ID  Agreement , Only the scene bit values are different . The same consumer_id May be held by one or more consumers .

2） When multiple clients get messages, the server side will use lock Control message reading , Each message read will last_read_msg_id to update , The next client consumer will continue to the next msg_id Continue reading messages .

<queue_name>_consumption_log_tbl

explain ：

1） Each client will write a message consumption log before reading the message and returning , And set the status ：0-

The client has read but not confirmed ;

2） If the message client has enabled the message acknowledgement mode , After the client processes the message and executes the corresponding business logic , Send a confirmation message to the server , And update the log status to  1  or  2;

3）  If the timeout is to receive the confirmation message, automatic re queuing or manual processing of unacknowledged data can be realized .