Sorting out the ideas of data processing received by TCP server, and the note of select: invalid argument error

Every time in the implementation tcp On the server side , Always thinking ： When processing the message details of the received client , Always fall into a little misunderstanding ,

Plus analyzing various business codes of the company's predecessors , Always slightly biased , Here do simple tcp Processing idea of correlation flow after receiving .

0： summary

Processing tcp When receiving ：

1：tcp Is reliable , The kernel for each tcp Client connections are allocated a send buffer and a receive buffer .

2： Based on the first point ：

====》 For each connection , It can be reliable , Receive data in sequence （ That is, put it into the corresponding receiving buffer ）.

====》 The buffer for each connection is independent , There will be no bunching .

====》 Store in buffer , It's in the form of flow , Unable to recognize the boundaries of multiple packages , Business layer adaptation is required .

3： Based on the second point ：

====》 We need to be in Business layer adaptation , Identify a complete package , There are generally two options ：（Length+Data）, ( Specific head or tail marks ), My understanding of other relevant specific agreements is probably the same

====》 As server side , We need to process multiple receive buffers at the same time （epoll/select management ）, At the same time, we should consider how to read a complete package （ How long data is read each time , Are you sure you can read a complete package ？）

4： Based on the third point ： How to ensure that an event triggers , The contents of the buffer can be read correctly .

===》 For example, if it is epoll ET Pattern , You should cycle through all the data （ There may be sticking phenomenon , Need to deal with ）

===》 If it is epoll LT perhaps select Pattern , Simple code , however , If tcp The bottom layer of the kernel has been unpacked , Send it several times , There may be a half bag phenomenon , as well as recv How to define this length ？

===》 In view of the above ：（ The best thing I can think about is ： Read a fixed length ,while Read into the application layer buffer （ We should maintain an application layer buffer per connection ） in , Then unpack the buffer ）

5： Based on the fourth point , What if some code is not temporarily stored in the application layer buffer ？（ Just consider Length+data The pattern of ）

See some code , It's a specific business , Every time io Events trigger , Receive a specific length first , Reading actual data , This can determine a complete package , To do business processing .

I will think about whether there will be defects in this treatment , The easiest thing to think about is

​ 1： An event triggers the processing of a packet , Is there any data that has not been processed , There is still data in the receive buffer of the kernel , such as epoll Of ET Mode scenario , But it looks like select and epoll Of LT The impact is not big

​ 2： If the business layer does not do relevant processing , The possible scenario is tcp Unpack the bottom , such First read the specific self fetching length , When reading the actual length data again , There may be a problem ？ The next bag hasn't arrived .

===》 however , Final thinking , If The business layer has handled , Guarantee tcp The bottom layer will not be unpacked , It should also be feasible if we do not use the application layer buffer .

===》 namely , The business layer has guaranteed that each reception is in a specific format （length+data） Complete package , Receive a specific byte header first each time , Receive actual length data after parsing After that, complete the package .

===》 Consider the event triggering feature , Ensure that the treatment is perfect , We epoll ET When dealing with , You should cycle through all the packages at once , and epoll LT as well as select In the mode of , It seems to have little effect , All can be taken correctly .

1：tcp Do something about recv When the processing of

In the actual business , We usually cooperate select perhaps epoll Manage the server-side related connections , When receiving messages from clients , There are some concerns ：

1.1：tcp Is reliable streaming , And actually the server side is Each client maintains a receive buffer and a send buffer .

First of all ：tcp The reception of the underlying stream can ensure .（ Has its own buffer , And reliable , The order ）

For each client connection , It can ensure reliable reception in sequence , Put it into its corresponding buffer .

===》 I have been falling into a misunderstanding , If you rely on tcp The underlying unpacking logic , You may receive other packages in the middle of receiving multiple packages , This is actually a misunderstanding , It is impossible to string packages .

===》 Server pair tcp Each connection has a transmit buffer and a receive buffer , For a connection, add tcp Reliable transmission of ,tcp The reception of the bottom layer can be guaranteed .

second ： Need custom agreement , Handle the buffer stream correctly .

===》tcp Although reliable , But it is received in the form of stream , Cannot know the boundaries of multiple packages .

===》 In order to correctly identify each complete package , To handle correctly （ Identify a complete package （tcp recv Is to get data from the buffer , First of all ： The buffer may contain multiple packets （ Sticky package ） second ： Probably recv Read half a bag , perhaps tcp Unpack the bottom , The second bag hasn't come yet ））

===》 So we need to make specific restrictions on the business layer , Be sure to recognize a package ： such as length+data, such as Add the header or tail of the protocol with a specific identifier

Third ： On the basis of specific agreements , How to ensure that multiple buffers are read correctly .

===》 If it is epoll Of ET Edge trigger mode , You have to use it. while Cycle through multiple reads

===》 If it is epoll Of LT Horizontal trigger mode , perhaps select, It can be triggered next time , However, you can also cycle through reading and processing to improve efficiency .

Fourth ： How to ensure the integrity of the package , A complete package to do the corresponding business processing .

===》 According to the application layer protocol , You can read the data first , Put in cache , Then the relevant data in the cache is parsed according to the protocol to process the complete package . （ Corresponding , The application layer buffer should also be a buffer corresponding to a connection ）

？？ There is no problem with using buffers , But for specific agreements , such as length+data The way , Read specific bytes first , Parse the actual length , Can you receive specific actual data ？

===》 My personal understanding is that it is possible to guarantee a certain business level ,

===》tcp If you send an oversized packet , Will unpack , After this scenario event is triggered , Read the actual length according to specific bytes , There will be problems if the next bag doesn't arrive late .

===》 But if tcp Our business level promised not to return tcp unpacking , We have controlled the size of the contract , I think it is feasible .

2： Pay attention to the details ： Network byte order , The alignment of the structure

If implemented in code tcp When designing the protocol related to sending and receiving , Need to pay attention to some details ：

1： If the agreement is structured , it is to be noted that Structure byte alignment size , It will affect the resolution of the receiving end .

2： Generally greater than 2byte Bytes of , Finally, according to Specific functions Conduct relevant Conversion between host byte order and network byte order

#include <arpa/inet.h> 
uint32_t htonl(uint32_t hostlong);        //32 Bit host byte order to network byte order  Host to Network Long 4 byte 
uint16_t htons(uint16_t hostlong);        // 16 bit host byte order is converted to network byte order  Host to Network Short 2 byte 

uint32_t ntohl(uint32_t hostlong)         //32 Bit network byte order to host byte order  Network to Host Long
uint16_t ntohs(uint16_t hostlong)         //16 Bit network byte order to host byte order  Network to Host Short

3： Code testing select: Invalid argument

The stack memory definition structure variable is best cleared .

In a simple demo When testing , The environment cannot be started , Report errors select: Invalid argument

After Baidu combination test , When found select Last parameter struct timeval tv; Set up questions

===》1： Refer to baidu , There are similar problems because tv.tv_usec Overvalued .

===》2： However, this value is not set too large in my code , But it is not cleared during initialization .

	struct timeval tv;
	memset(&tv, 0, sizeof(struct timeval));  // Be careful   Here, clean up the memory after initialization 
	fd_set rset;
	int maxfd = m_listenfd + 1;
	while(m_running)
	{
    
		tv.tv_sec = 30;
		//tv.tv_usec = 0; // Because of the stack memory used   If the memory here is relatively large, it will lead to select Invalid argument

        rset = m_allset;
        ret = select(maxfd, &rset, (fd_set *)0,(fd_set *)0, (struct timeval *)&tv);
        ...
    }

===》 in addition , I heard from my colleagues tcp Buffer overflow problem , Personal understanding is tcp There will be no so-called overflow problem in the receive buffer , The sending buffer should have similar problems because of the sending frequency .