MySQL invalid conn troubleshooting

Service usage golang , The client library is go-mysql-driver , The system test environment is frequent but not always reported invalid conn error , But actually take sql The execution is normal .

Cause analysis

Because the connection is invalid , First, consider that the client uses expired connections .

mysql Both the server and the client can configure the maximum life cycle of their connections .

If the maximum life cycle of the connection configured by the client is greater than that of the server , And the client has no mechanism for reconnecting invalid connections , It will cause the connection of the server to expire , The client uses an expired connection , triggering invalid conn.

Inquire about mysql Maximum idle life cycle of the connection s In units of .

Client maximum lifecycle configuration

mysql server The maximum life cycle of is greater than golang The maximum connection life cycle of the client , So eliminate this reason .

There was a stalemate for a moment , Why on earth is the connection invalid . Direct packet capture analysis .

The expert system finds abnormal packages

take pcap For documents wireshark open , Use expert system to analyze the caught package as a whole .

See there are 8 An unusual RST package , The normal connection switch will not have RST Produced , Focus on the analysis RST Cause of occurrence .

Find the problem

See the specific abnormal place ,mysql The server is replying to a client ACK After the news , The client waited 10s Yes mysql The server initiated Request Quit(mysql agreement ) Command to close the connection .

See the details before and after the abnormality .

Choose... From the menu bar Statistics->Flow Graph, You can open the data flow diagram window .

You can see the process as follows ：

1,11:55:49.05 On the client side to mysql launch Request Execute Statement perform sql The order of ,

2,mysql recovery Ack

3, however mysql The execution result is not returned to the client , So the client waited 10s Then send the command to close the connection .

4,mysql stay 11:55:59.27 Only then returned the execution result .

5, But at this time, the client has closed the connection , Sending data to a closed connection triggers RST The signal , So the client responds to the server RST

See here , The task that can be clearly performed by the server sql It's overtime , Or the server timed out when it returned the results .

In the further , Is it the alarm caused by timeout ？

You can see that when reading buffer data , It is possible to report ErrInvalidConn

Keep looking at mc.buf.readNext, Timeout code found

Because it's a database , First I thought of disk , Back again top,iostat ,iotop analysis .

This discovery is abnormal .

Disk found io The usage rate will soar occasionally , therefore mysql The execution rate is affected .

The main items that occupy disk utilization are video transcoding , The transcoding time is not fixed , So when transcoding , On the same host mysql Be affected by , Caused a timeout , Cause the application layer to report invalid conn error .

Confirmed , The transcoding function can be temporarily turned off in the test environment , After closing , No more alarms appear .

Under normal circumstances , When you see mysql Return timeout yes , You should check the system immediately io Wait for the hardware index , Because this is the test environment , So I went to see it go-mysql-client Library code , Further confirmation .

The monitoring system of testing environment is not perfect , There is no monitoring interface , Otherwise, you should be able to see the disk at a glance io abnormal .

Secondly, it is late to return to the application layer to see the code , Or was invalid conn Confused , In fact, if go-mysql The error reported by the library as a timeout error may be more in line with this scenario .

Actually, from the beginning , Just used it. top and iostat Analyze the system disk occupancy , However, the indicators were not abnormal during that period , Add the reported error information is invalid conn This led me to first locate the problem as a problem with the network connection , And capture and analyze .

See this accidental mistake , Systematic indicators should be continuously observed , This time I just watched it for more than ten seconds and then turned to analyze other factors , It really shouldn't be .

The test environment starts the slow log

set global slow_query_log=1

Check the on Status