[MySQL] character set utf8mb4 cannot store the record of expression stepping on the pit

The phenomenon

The character set on the field has priority over the character set of the table , The character set of the table takes precedence over the character set of the database , Theoretically, as long as the character set of the table is utf8mb4 Can store expressions , Is that really the case ？

MySQL The character set of the data table has been set to utf8mb4, But by JDBC Write... To the database 4 Bytes of emoji Expression times wrong , But through direct use SQL Statement inserts this... On the command line 4 Bytes of emoji The expression was successful .

Examples are as follows ：

Table structure ：

CREATE TABLE `user_info` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `name` varchar(11) NOT NULL,
  `age` int(4) DEFAULT NULL
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

JDBC write in ：

User user = new User();
user.setName("\uD83D\uDC8B");
user.setAge(18);
userMapper.insertUser(user);

The error results are as follows ：

org.springframework.jdbc.UncategorizedSQLException: 
### Error updating database. Cause: java.sql.SQLException: Incorrect string value: '\xF0\x9F\x92\x8B' for column 'name' at row 1
### The error may involve com.wakzz.database.persistence.UserMapper.insertUser-Inline
### The error occurred while setting parameters
### SQL: insert into user_info (name, age) values (?, ? )
### Cause: java.sql.SQLException: Incorrect string value: '\xF0\x9F\x92\x8B' for column 'name' at row 1
; uncategorized SQLException; SQL state [HY000]; error code [1366]; Incorrect string value: '\xF0\x9F\x92\x8B' for column 'name' at row 1; nested exception is java.sql.SQLException: Incorrect string value: '\xF0\x9F\x92\x8B' for column 'name' at row 1
	at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:89)
	at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81)
	at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81)
	at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:73)
	at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:446)
	at com.sun.proxy.$Proxy81.insert(Unknown Source)
	at org.mybatis.spring.SqlSessionTemplate.insert(SqlSessionTemplate.java:278)
	at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:58)
	at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:59)
	at com.sun.proxy.$Proxy92.insertUser(Unknown Source)

Command line write succeeded ：

mysql> insert into user_info (name,age) values ('',18);
Query OK, 1 row affected

The reason for the error

reason ：JDBC It will automatically detect MySQL Server side character_set_server Value , Automatic execution SET NAMES Command sets the character set encoding of the entire connection , Its purpose is to automatically detect the character set encoding configuration of the server and reduce JDBC The character set encoding configuration of the client . If MySQL Server side character_set_server The value of is utf8, that JDBC The encoding of the connected character set will be set to utf8, In this way, even if the character set of the table is utf8mb4 It's also impossible to store expressions .

Official statement ：https://dev.mysql.com/doc/relnotes/connector-j/5.1/en/news-5-1-13.html

see JDBC Source code discovery ：

// realJavaEncoding by url It is specified in characterEncoding Value 
if (realJavaEncoding.equalsIgnoreCase("UTF-8") || realJavaEncoding.equalsIgnoreCase("UTF8")) {
    
	// charset names are case-sensitive

	//  take MySQL Server side character_set_server
	boolean useutf8mb4 = CharsetMapping.UTF8MB4_INDEXES.contains(this.session.getServerDefaultCollationIndex());

	if (!this.useOldUTF8Behavior.getValue()) {
    
		if (dontCheckServerMatch || !this.session.characterSetNamesMatches("utf8") || (!this.session.characterSetNamesMatches("utf8mb4"))) {
    
			//  perform set names xxx
			execSQL(null, "SET NAMES " + (useutf8mb4 ? "utf8mb4" : "utf8"), -1, null, false, this.database, null, false);
			this.session.getServerVariables().put("character_set_client", useutf8mb4 ? "utf8mb4" : "utf8");
			this.session.getServerVariables().put("character_set_connection", useutf8mb4 ? "utf8mb4" : "utf8");
		}
	} else {
    
		execSQL(null, "SET NAMES latin1", -1, null, false, this.database, null, false);
		this.session.getServerVariables().put("character_set_client", "latin1");
		this.session.getServerVariables().put("character_set_connection", "latin1");
	}

	this.characterEncoding.setValue(realJavaEncoding);
}

In obtaining mysql After the server parameter of , Parse character set encoding ：

• When character_set_server by utf8 when , perform SET NAMES utf8
• When character_set_server by utf8mb4 when , perform SET NAMES utf8mb4

Test on the command line SET NAMES It is found that even if the character set of the database table is utf8mb4 when , If implemented SET NAMES utf8 It can also lead to 4 Byte character writing mysql Failure . Successfully reproduced JDBC write in emoji Write exception problem .

mysql> SET NAMES utf8;
Query OK, 0 rows affected

mysql> insert into user_info (name,age) values ('',18);
1366 - Incorrect string value: '\xF0\x9F\x92\x8B' for column 'name' at row 1

mysql> SET NAMES utf8mb4;
Query OK, 0 rows affected

mysql> insert into user_info (name,age) values ('',18);
Query OK, 1 row affected

terms of settlement

modify character_set_server by utf8mb4

modify mysql The configuration file my.cnf, Add the following configuration ：

character_set_server = utf8mb4

Need to restart the database instance .

Character set before modification

mysql> show variables like "%char%";
+--------------------------+----------------------------------------+
| Variable_name            | Value                                  |
+--------------------------+----------------------------------------+
| character_set_client     | utf8mb4                                |
| character_set_connection | utf8mb4                                |
| character_set_database   | latin1                                 |
| character_set_filesystem | binary                                 |
| character_set_results    | utf8mb4                                |
| character_set_server     | utf8                                   |
| character_set_system     | utf8                                   |
| character_sets_dir       | /usr/soft/mysql-5.6.31/share/charsets/ |
+--------------------------+----------------------------------------+
8 rows in set

mysql> SHOW VARIABLES LIKE 'collation%';
+----------------------+--------------------+
| Variable_name        | Value              |
+----------------------+--------------------+
| collation_connection | utf8mb4_general_ci |
| collation_database   | latin1_swedish_ci  |
| collation_server     | utf8_general_ci    |
+----------------------+--------------------+
3 rows in set

Modified character set

mysql> show variables like "%char%";
+--------------------------+----------------------------------------+
| Variable_name            | Value                                  |
+--------------------------+----------------------------------------+
| character_set_client     | utf8mb4                                |
| character_set_connection | utf8mb4                                |
| character_set_database   | latin1                                 |
| character_set_filesystem | binary                                 |
| character_set_results    | utf8mb4                                |
| character_set_server     | utf8mb4                                |
| character_set_system     | utf8                                   |
| character_sets_dir       | /usr/soft/mysql-5.6.31/share/charsets/ |
+--------------------------+----------------------------------------+
8 rows in set

mysql> SHOW VARIABLES LIKE 'collation%';
+----------------------+--------------------+
| Variable_name        | Value              |
+----------------------+--------------------+
| collation_connection | utf8mb4_general_ci |
| collation_database   | latin1_swedish_ci  |
| collation_server     | utf8mb4_general_ci |
+----------------------+--------------------+
3 rows in set

Manually set the code of the database connection to utf8mb4

JDBC Set the code of the connection ：

Connection conn = DriverManager.getConnection(url, userName, password);
conn.prepareStatement("set names utf8mb4").executeQuery();

If it is Spring project , It can be downloaded from ThreadLocal Get the connection ：

ConnectionHolder connectionHolder = (ConnectionHolder) TransactionSynchronizationManager.getResource(dataSource);
Connection connection = connectionHolder.getConnection();
try {
    
	PreparedStatement preparedStatement = connection.prepareStatement("SET NAMES utf8mb4");
	preparedStatement.executeQuery();
} catch (SQLException e) {
    
	e.printStackTrace();
}

among dataSource Can pass Spring Inject in ：

private DataSource dataSource;

Pay attention to using this method , The transaction needs to be opened , Otherwise, from ThreadLocal Can't get the connection .

Finally, pay to modify the database 、 surface 、 Character character set SQL：

--  Modify the character set of the database 
alter database DBNAME DEFAULT CHARACTER SET utf8mb4;

--  Modify the character set of the table , After changing the table character set, it is only valid for the new fields 
alter table tbl_name convert to character set character_name ;

--  Modify the character set of the field 
alter table tbl_name change col1 col1 varchar(20) CHARACTER SET utf8mb4;

Encode the string and store , Take out decoding

Encode the string when saving into the database ：

String encode = URLEncoder.encode(" Just 🧑", StandardCharsets.UTF_8.name());

Decode the string when fetching from the database ：

URLDecoder.decode(encode, StandardCharsets.UTF_8.name());