Analysis and solution of connection failure caused by MySQL using replicationconnection

MySQL Database read write separation , It is one of the common means to improve service quality , And for the technical solution , There are many mature open source frameworks or solutions , for example ：sharding-jdbc、spring Medium AbstractRoutingDatasource、MySQL-Router etc. , and mysql-jdbc Medium ReplicationConnection Can also support . This article will not do too much analysis on the technology selection of read-write separation , Just exploring in use druid As a data source 、 combination ReplicationConnection When performing read / write separation , Cause of connection failure , And find a simple and effective solution .

The problem background

For historical reasons , Some services have connection failure exceptions , Key error reports are as follows ：

It is not difficult to see from the log , This is because the connection has not been connected to for a long time MySQL Server side interaction , The server has closed the connection , Typical connection failure scenarios .

The main configurations involved are as follows ：

jdbc To configure

jdbc:mysql:replication://master_host:port,slave_host:port/database_name

druid To configure

testWhileIdle=true( namely , Idle connection checking is enabled );timeBetweenEvictionRunsMillis=6000L( namely , For the scenario of getting connections , If a connection is idle for more than 1 minute , Will be checked , If the connection is invalid , Will be discarded and retrieved ).

attach ：DruidDataSource.getConnectionDirect in , The processing logic is as follows ：

if (testWhileIdle) {
 final DruidConnectionHolder holder = poolableConnection.holder;
 long currentTimeMillis = System.currentTimeMillis();
 long lastActiveTimeMillis = holder.lastActiveTimeMillis;
 long lastExecTimeMillis = holder.lastExecTimeMillis;
 long lastKeepTimeMillis = holder.lastKeepTimeMillis;

 if (checkExecuteTime
 && lastExecTimeMillis != lastActiveTimeMillis) {
 lastActiveTimeMillis = lastExecTimeMillis;
 }

 if (lastKeepTimeMillis > lastActiveTimeMillis) {
 lastActiveTimeMillis = lastKeepTimeMillis;
 }

 long idleMillis = currentTimeMillis - lastActiveTimeMillis;

 long timeBetweenEvictionRunsMillis = this.timeBetweenEvictionRunsMillis;

 if (timeBetweenEvictionRunsMillis <= 0) {
 timeBetweenEvictionRunsMillis = DEFAULT_TIME_BETWEEN_EVICTION_RUNS_MILLIS;
 }

 if (idleMillis >= timeBetweenEvictionRunsMillis
 || idleMillis < 0 // unexcepted branch
 ) {
 boolean validate = testConnectionInternal(poolableConnection.holder, poolableConnection.conn);
 if (!validate) {
 if (LOG.isDebugEnabled()) {
 LOG.debug(&quot;skip not validate connection.&quot;);
 }

 discardConnection(poolableConnection.holder);
 continue;
 }
 }
}


mysql Timeout parameter configuration

wait_timeout=3600(3600 second , namely ： If a connection has not interacted with the server for more than an hour , The connection will be made by the server kill). Obvious , Based on the above configuration , Understand as usual , Not supposed to show up “The last packet successfully received from server was xxx,xxx,xxx milliseconds ago” The problem of .（ Of course , At that time, manual intervention was also ruled out kill The possibility of dropping the database connection ）.

When “ Ought to be right ” Our experience doesn't explain the problem , It is often necessary to get out of the shackles of experience that may float on the surface , To get to the bottom . that , What is the real reason for this problem ？

The essential reason

When using druid Manage data sources , combination mysql-jdbc Born in the Central Plains ReplicationConnection When performing read / write separation ,ReplicationConnection Actually exists in the proxy object master and slaves Two sets of connections ,druid When doing connection detection , Only... Of these can be detected master Connect , If a slave The connection has not been used for a long time , Will cause connection failure problems .

Cause analysis

mysql-jdbc in , The database driver handles the connection

combination com.mysql.jdbc.Driver Source code , It's not hard to see. mysql-jdbc The main process of obtaining connections in is as follows ：

In the “jdbc:mysql:replication://” At the beginning of the configuration jdbc-url, adopt mysql-jdbc Get the connection to , It's actually a ReplicationConnection Proxy object of , By default ,“jdbc:mysql:replication://” The first after host and port Corresponding master Connect , The subsequent host and port Corresponding slaves Connect , For the existence of multiple slave Configuration scenario , By default, a random policy is used for load balancing .

ReplicationConnection Proxy object , Use JDK Generated by dynamic proxy , among InvocationHandler The concrete realization of , yes ReplicationConnectionProxy, The key codes are as follows ：

public static ReplicationConnection createProxyInstance(List<String> masterHostList, Properties masterProperties, List<String> slaveHostList,
 Properties slaveProperties) throws SQLException {
 ReplicationConnectionProxy connProxy = new ReplicationConnectionProxy(masterHostList, masterProperties, slaveHostList, slaveProperties);
 return (ReplicationConnection) java.lang.reflect.Proxy.newProxyInstance(ReplicationConnection.class.getClassLoader(), INTERFACES_TO_PROXY, connProxy);
 }


ReplicationConnectionProxy An important part of

About database connection broker ,ReplicationConnectionProxy The main components of the are shown in the figure below ：

ReplicationConnectionProxy There is masterConnection and slavesConnection Two actual connection objects ,currentConnetion（ Current connection ） You can switch to mastetConnection perhaps slavesConnection, The switching mode can be set readOnly Realization . In business logic , This is also the core of the separation of reading and writing , Simply speaking ： Use ReplicationConnection When performing read / write separation , Just make one “ Set up connection Of readOnly Attribute ”aop that will do . be based on ReplicationConnectionProxy, Obtained from business logic Connection Proxy object , What is the main logic of database access ？

ReplicationConnection Proxy object processing

For business logic , Acquired Connection example , yes ReplicationConnection Proxy object , The proxy object passes ReplicationConnectionProxy and ReplicationMySQLConnection Cooperate with each other to complete the processing of database access , among ReplicationConnectionProxy In the realization of  InvocationHandler At the same time , It also plays the role of connection management , The core logic is as follows ：

about prepareStatement And so on ,ConnectionMySQConnection Get the current connection for processing （ This is the key point of common read-write separation processing ）; here , Focus on pingInternal Method , The processing method is also to obtain the current connection , And then execute pingInternal Logic .

about ping() This particular logic , The description in the figure is relatively simple , But the meaning of the subject remains the same , namely ： Yes master The connection and sleves All connections must be made ping() To deal with .

In the figure ,pingInternal The process and druid Of MySQ Connection check related , and ping Special treatment of , It is also the key to solve the problem .

druid Data source pair MySQ Inspection of connections

druid Chinese vs MySQL The default implementation class for connection checking is MySqlValidConnectionChecker, The core logic is as follows ：

public boolean isValidConnection(Connection conn, String validateQuery, int validationQueryTimeout) throws Exception {
 if (conn.isClosed()) {
 return false;
 }

 if (usePingMethod) {
 if (conn instanceof DruidPooledConnection) {
 conn = ((DruidPooledConnection) conn).getConnection();
 }

 if (conn instanceof ConnectionProxy) {
 conn = ((ConnectionProxy) conn).getRawObject();
 }

 if (clazz.isAssignableFrom(conn.getClass())) {
 if (validationQueryTimeout <= 0) {
 validationQueryTimeout = DEFAULT_VALIDATION_QUERY_TIMEOUT;
 }

 try {
 ping.invoke(conn, true, validationQueryTimeout * 1000);
 } catch (InvocationTargetException e) {
 Throwable cause = e.getCause();
 if (cause instanceof SQLException) {
 throw (SQLException) cause;
 }
 throw e;
 }
 return true;
 }
 }

 String query = validateQuery;
 if (validateQuery == null || validateQuery.isEmpty()) {
 query = DEFAULT_VALIDATION_QUERY;
 }

 Statement stmt = null;
 ResultSet rs = null;
 try {
 stmt = conn.createStatement();
 if (validationQueryTimeout > 0) {
 stmt.setQueryTimeout(validationQueryTimeout);
 }
 rs = stmt.executeQuery(query);
 return true;
 } finally {
 JdbcUtils.close(rs);
 JdbcUtils.close(stmt);
 }

}


Corresponding to... Used in the service mysql-jdbc(5.1.45 edition ), When not set “druid.mysql.usePingMethod” System properties , Default usePingMethod by true, as follows ：

public MySqlValidConnectionChecker(){
try {
 clazz = Utils.loadClass("com.mysql.jdbc.MySQLConnection");
 if (clazz == null) {
 clazz = Utils.loadClass("com.mysql.cj.jdbc.ConnectionImpl");
 }

 if (clazz != null) {
 ping = clazz.getMethod("pingInternal", boolean.class, int.class);
 }

 if (ping != null) {
 usePingMethod = true;
 }
 } catch (Exception e) {
 LOG.warn("Cannot resolve com.mysql.jdbc.Connection.ping method. Will use 'SELECT 1' instead.", e);
 }

 configFromProperties(System.getProperties());
}

@Override
public void configFromProperties(Properties properties) {
 String property = properties.getProperty("druid.mysql.usePingMethod");
 if ("true".equals(property)) {
 setUsePingMethod(true);
 } else if ("false".equals(property)) {
 setUsePingMethod(false);
 }
}


meanwhile , It can be seen that MySqlValidConnectionChecker Medium ping The method uses MySQLConnection Medium pingInternal Method , And the method , Combine face to face ReplicationConnection Analysis of , When calling pingInternal when , Just check the current connection . The time to perform a connection verification is through DrduiDatasource When getting a connection , At this time... Is not set readOnly attribute , Check the connection of the , Is really just ReplicationConnectionProxy Medium master Connect .

Besides , If you pass “druid.mysql.usePingMethod” Property settings usePingMeghod by false, In fact, it will also lead to connection failure , because ： When passed valideQuery( for example “select 1”) When performing connection verification , It will arrive ReplicationConnection Common query logic in , At this time, the corresponding connection is still master Connect .

One question outside the question

：ping Why does the method use “pingInternal”, It's not conventional ping？ reason ：pingInternal Control parameters such as timeout time are reserved .

Solution

Adjust the dependent version

Used in services mysql-jdbc Version is 5.1.45,druid Version is 1.1.20. Through the understanding of other high version dependencies , The problem still exists .

Modify the read / write separation implementation

The workload of modification mainly lies in data source configuration and aop adjustment , But it requires a certain overall regression verification cost , In view of the general importance of the services involved in this issue , No major adjustment for the time being .

expand mysql-jdbc drive

Based on the original ReplicationConnection The function of , expand pingInternal Adjust to normal ping, Integrate the existing Driver Expand new Driver. The scheme is feasible , But the cost of revision is not small .

be based on druid, expand MySQL Connection check

To solve problems simply and efficiently , Selective expansion MySqlValidConnectionChecker, And in druid Add the corresponding configuration to the data source . Expand as follows ：

public class MySqlReplicationCompatibleValidConnectionChecker extends MySqlValidConnectionChecker {


 private static final Log LOG = LogFactory.getLog(MySqlValidConnectionChecker.class);
 /**
 * 
 */
 private static final long serialVersionUID = 1L;

 @Override
 public boolean isValidConnection(Connection conn, String validateQuery, int validationQueryTimeout) throws Exception {

 if (conn.isClosed()) {
 return false;
 }

 if (conn instanceof DruidPooledConnection) {
 conn = ((DruidPooledConnection) conn).getConnection();
 }

 if (conn instanceof ConnectionProxy) {
 conn = ((ConnectionProxy) conn).getRawObject();
 }

 if (conn instanceof ReplicationConnection) {

 try {
 ((ReplicationConnection) conn).ping();
 LOG.info("validate connection success: connection=" + conn.toString());
 return true;
 } catch (SQLException e) {
 LOG.error("validate connection error: connection=" + conn.toString(), e);
 throw e;
 }

 }

 return super.isValidConnection(conn, validateQuery, validationQueryTimeout);
 }

}


ReplicatoinConnection.ping() In the implementation logic of , For all master and slaves The connection goes on ping operation , In the end, each ping Operations will be called to LoadBalancedConnectionProxy.doPing To deal with , And here , It can be configured in the database url Set in loadBalancePingTimeout Property setting timeout .

Turn to the technology learning and exchange platform of R & D center and industry partners , Regularly share front-line practical experience and cutting-edge technical topics in the industry . Official account 「 Transfer technology 」, Various dry goods practices , Welcome to share ~