Cloud MySQL importing cloud data warehouse PostgreSQL best practices

explain

This article describes problems and solutions based on Tencent cloud Cloud data warehouse PostgreSQL（CDWPG）.

In addition, use ：

Tencent cloud Cloud database MySQL（TencentDB for MySQL,CDB）

Tencent cloud Data transmission service （Data Transmission Service,DTS）

background

Help users easily complete database migration to the cloud without stopping service , The real-time synchronization channel is used to easily build a highly available data storage disaster capacity architecture , Feed commercial data mining through data subscriptions 、 Scenario requirements such as business asynchronous decoupling .

structure Mysql Basic data

Create test libraries and basic tables

MySQL [(none)]> CREATE DATABASE dts_demo;
Query OK, 1 row affected (0.00 sec)

MySQL [(none)]> USE dts_demo;
Database changed
MySQL [dts_demo]> CREATE TABLE `user_info` (
    ->   `id` int(11) NOT NULL AUTO_INCREMENT,
    ->   `c_user_id` varchar(36) NOT NULL DEFAULT '',
    ->   `c_name` varchar(22) NOT NULL DEFAULT '',
    ->   `c_province_id` int(11) NOT NULL,
    ->   `c_city_id` int(11) NOT NULL,
    ->   `create_time` datetime NOT NULL,
    ->   PRIMARY KEY (`id`),
    ->   KEY `idx_user_id` (`c_user_id`)
    -> ) ENGINE=InnoDB;
Query OK, 0 rows affected (0.01 sec)

Build test data

Here we quickly build a test data , It is used to simulate the actual production in the middle order 100 A data sheet of Wan , Reference resources Fast build Mysql Million level test data .

Create an account dedicated to data synchronization

establish Mysql Sync account

According to Tencent cloud DTS Requirements of official documents , It needs to be on the source side MySQL Create a migration account in the instance , The required account permissions are as follows ：

MySQL [dts_demo]> GRANT SHOW VIEW,PROCESS,RELOAD,LOCK TABLES,REPLICATION CLIENT,REPLICATION SLAVE,SELECT ON *.* TO 'dts_user'@'%' IDENTIFIED BY 'dts_admin';
Query OK, 0 rows affected, 1 warning (0.00 sec)

MySQL [dts_demo]> GRANT ALL PRIVILEGES ON `__tencentdb__`.* TO 'dts_user'@'%';
Query OK, 0 rows affected (0.00 sec)

MySQL [dts_demo]> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

Here a super user is used to create a mysql Ordinary users "dts_user", Set the password to "dts_admin", Allow all host sources to log in , And given the corresponding authority .

among "__tencentdb__" yes DTS Users needed , This authorization cannot be omitted , Finally, don't forget to refresh permissions .

establish CDWPG Synchronize account and target database

According to Tencent cloud DTS Requirements of official documents , Target end CDWPG Create a migration account in the instance , The required account permissions are as follows ：

[[email protected] ~]# psql -d postgres -U cdwadmin -h 10.0.5.18 -p 5436 
Password for user cdwadmin: 
psql (9.5.25, server 9.4.24)
Type "help" for help.

postgres=> CREATE USER dts_user WITH PASSWORD 'dts_admin';
CREATE ROLE
postgres=> CREATE DATABASE dts_demo;
CREATE DATABASE
postgres=> \c dts_demo
You are now connected to database "dts_demo" as user "cdwadmin".
dts_demo=> ALTER database dts_demo SET search_path TO dts_demo;
ALTER DATABASE
dts_demo=> GRANT ALL PRIVILEGES ON DATABASE dts_demo TO dts_user;
GRANT
dts_demo=> GRANT Delete, Truncate, Insert, References, Select, Update, TRIGGER ON all tables in schema public TO dts_user;
GRANT

Here a super user is used to create a mysql Ordinary users "dts_user", Set the password to "dts_admin", And given the corresponding authority . In addition, due to Mysql and CDWPG The relationship is Mysql Of DB Corresponding CDWPG Of SCHEMA, So here is to create the target database first "dts_demo" And default the database to SCHEMA Set to "dts_demo".

establish CDWPG Target table

Field type comparison table

establish CDWPG The table needs to be based on Postgresql To match the field type of the source table , Otherwise, the field type does not correspond .

Mysql Field type	Postgresql Field type	Do you support
INTMDEIUMINT	INTEGER	Support
SMALLINT/TINYINT	SMALLINT	Support
BIGINT	BIGINT	Support
TINYINT UNSIGNED	SMALLINT	Support
SMALLINT UNSIGNED	INTEGER	Support
MEDIUMINT UNSIGNED	INTEGER	Support
INT UNSIGNED	BIGINT	Support
BIGINT UNSIGNED	NUMERIC	Support
BOOLEAN	BOOLEAN	I won't support it
FLOAT(UNSIGNED)	REAL	Support
DOUBLE(UNSIGNED)	DOUBLE PRECISION	Support
DECIMAL	NUMERIC	Support
CHAR	CHARACTER	Support
VARCHAR	TEXT VARCHAR	Support
DATE	DATE	Support
TIME	time without time zone	Support
DATETIME	timestamp without time zone	Support
TIMESTAMP	timestamp without time zone	Support
LONGTEXTMEDIUMTEXT/TINYTEXT/TEXT	TEXT	Support
LONGTEXTMEDIUMTEXT/TINYTEXT/TEXT	JSON( Not recommended )	Only strings that are json type
BLOB/MEDIUMBLOB/TINYBLOB/BINARYNARBINARYALONGBLOB	BYTEA	Support
ENUM	CHAR perhaps VARCHAR	Support
YEAR	SMALLINT	Support

Create the target table

Use ordinary users "dts_user" establish SCHEMA, And create the target table "user_info".

[[email protected] ~]# psql -d dts_demo -U dts_user -h 10.0.5.18 -p 5436 
Password for user dts_user: 
psql (9.5.25, server 9.4.24)
Type "help" for help.

dts_demo=> CREATE SCHEMA dts_demo;
CREATE SCHEMA
dts_demo=> CREATE TABLE user_info (
dts_demo(>     id integer NOT NULL,
dts_demo(>     c_user_id varchar NOT NULL,
dts_demo(>     c_name varchar NOT NULL,
dts_demo(>     c_province_id integer NOT NULL,
dts_demo(>     c_city_id integer NOT NULL,
dts_demo(>     create_time timestamp without time zone NOT NULL
dts_demo(> );
CREATE TABLE
dts_demo=> \d
            List of relations
  Schema  |   Name    | Type  |  Owner   
----------+-----------+-------+----------
 dts_demo | user_info | table | dts_user
(1 row)

establish DTS Data synchronization task

Particular attention

• cloud Mysql、DTS Data synchronization products 、CDWPG Cloud data warehouse , These three instances need to be purchased at the same time VPC Next , Otherwise, the network won't work , Unable to synchronize data .
• Configure cloud database MySQL To CDWPG Data synchronization task , Before the task starts , Pre inspection is required , The main inspection contents and points are as follows ：

Inspection content	checkpoint
Verify the target database schema and table Whether there is	schema and table You have to create... In advance , If not created well , May be an error
Verify whether the current user has the permission of the target data table	For the table to be synchronized , First, judge whether the current user belongs to the table owner（owner Have all permissions ）, If not , View information_schema.table_privilege Authorization information in the table , It must be ensured that ：Delete、Truncate、Insert、References、Select、Update、TRIGGER Authorization rights for , Otherwise, an error will be reported
Verify whether the disk space on the target side is sufficient	Compare the available space of the target library with the space required by the source side
Verify the source side database permissions	Check whether the source instance has permission ：Reload、LockTable、ReplClient、ReplSlave、Select、REPLICATION CLIENT
Verify the source end MySQL connect_timeout Parameters	check MySQL On the side connect_timeout Is the parameter less than 10, If it is less than, an error will be reported
Verify the source and target database connections	check MySQL and CDWPG Whether it can be connected correctly
Verify the source database version	MySQL The version must be MySQL 5.6 or MySQL 5.7
Verify the source side optimization parameters	innodb_stats_on_metadata The indicator needs to be closed
Verify the source end binlog Parameters	binlog_format Must be ROW;binlog_row_image Must be FULL;log_bin Must be ON;gtid_mode Must be ON
Verify the primary key constraint	The table to be synchronized on the source side must have a primary key
Verify the source database code	Source side must be utf8 or utf8mb4
check MySQL Whether the table name case configuration is correct	check lower_case_table_names Is the parameter 0, If 0 The configuration is incorrect
check MySQL Whether the database table name and column name contain "	CDWPG I won't support it " As a name

Create a data synchronization task

Configure synchronization tasks

test Mysql Connectivity

test CDWPG Connectivity

You can see ,CDWPG Connectivity test failed . This is because DTS The data synchronization tool is also an access side , He's right CDWPG Access also needs to be in CDWPG The client authorizes the corresponding white list access rights . Here we need to move CDWPG Cluster instance page , Click on " To configure " label , Create a new access white list , Which requires authorization IP The white list is shown in the above figure IP paragraph ：

The white list configuration is completed

Try again to test connectivity

thus , The connectivity test is complete .

Here, simply select the library table that needs to be synchronized .

In the verification test , Meet a Mysql Parameter problem , This needs to be moved Mysql Instance end , Modify the corresponding parameters .

Check the test again .

All the tests passed , Data synchronization is ready .

Data synchronization

The first step of the synchronization task is to export the source data to DTS.

The second step of the synchronization task is to import data into the target table .

dts_demo=> SELECT COUNT(1) FROM user_info;
 count  
--------
 206000
(1 row)

dts_demo=> SELECT COUNT(1) FROM user_info;
 count  
--------
 601500
(1 row)

dts_demo=> SELECT COUNT(1) FROM user_info;
  count  
---------
 1000000
(1 row)

dts_demo=> SELECT * FROM user_info ORDER BY id LIMIT 20;
 id |              c_user_id               |        c_name        | c_province_id | c_city_id |     create_time     
----+--------------------------------------+----------------------+---------------+-----------+---------------------
  1 | 1afd2630-88bc-11eb-9c30-0c42a125994e | oxlXASuDAQhIAEmDVAZ4 |             8 |        33 | 2022-03-19 22:05:05
  2 | 1afd300e-88bc-11eb-9c30-0c42a125994e | Nj27hTrqAwIQUPiO0qXo |           727 |        95 | 2028-03-19 22:05:05
  3 | 1afd4041-88bc-11eb-9c30-0c42a125994e | J9rzo41MCC2dM5Whp4Zy |           482 |        22 | 2026-03-19 22:05:05
  4 | 1afd4562-88bc-11eb-9c30-0c42a125994e | RX3eSuFHkqXmNJ8hSoas |           517 |        67 | 2023-03-19 22:05:05
  5 | 1afd4a49-88bc-11eb-9c30-0c42a125994e | YcVRm6gPdssI6cxUMZs9 |            54 |        31 | 2023-03-19 22:05:05
  6 | 1afd4ebd-88bc-11eb-9c30-0c42a125994e | ydfrgRm1VlPX8FLFSeo5 |           968 |         3 | 2027-03-19 22:05:05
  7 | 1afd530c-88bc-11eb-9c30-0c42a125994e | rsMpwgyPk0TiBXO2AFr3 |           585 |        25 | 2027-03-19 22:05:05
  8 | 1afd574a-88bc-11eb-9c30-0c42a125994e | H5aqu0qT4xgB06i1341J |           293 |        73 | 2027-03-19 22:05:05
  9 | 1afd5cf9-88bc-11eb-9c30-0c42a125994e | Y10PZgc4AzTDjxyY5ke0 |            31 |        60 | 2025-03-19 22:05:05
 10 | 1afd61a8-88bc-11eb-9c30-0c42a125994e | 761DXGqU7GUjHpKns2E0 |           732 |        12 | 2022-03-19 22:05:05
 11 | 1afd662c-88bc-11eb-9c30-0c42a125994e | AVIBJG21NLi00PX8HS7O |           384 |        97 | 2022-03-19 22:05:05
 12 | 1afd6ace-88bc-11eb-9c30-0c42a125994e | RK0E38ooDO0r1CSn6dz6 |           474 |        53 | 2022-03-19 22:05:05
 13 | 1afd6f01-88bc-11eb-9c30-0c42a125994e | pNCyKUaVYVyQqowgB3kl |           370 |        31 | 2028-03-19 22:05:05
 14 | 1afd7332-88bc-11eb-9c30-0c42a125994e | CvwX2bCq4VhshQeuy9Yf |           960 |        55 | 2024-03-19 22:05:05
 15 | 1afd775f-88bc-11eb-9c30-0c42a125994e | 3YzKT2oEXGmAIDRdo9on |           383 |        26 | 2024-03-19 22:05:05
 16 | 1afd7bcf-88bc-11eb-9c30-0c42a125994e | j8zjGigivtHUhwDq2OK9 |           172 |        90 | 2025-03-19 22:05:05
 17 | 1afd800c-88bc-11eb-9c30-0c42a125994e | 9pqJfSuEE8AlMKdHHeTD |           130 |        24 | 2025-03-19 22:05:05
 18 | 1afd842c-88bc-11eb-9c30-0c42a125994e | 0DZUqdFwtEGifda3AA4p |           480 |        67 | 2028-03-19 22:05:05
 19 | 1afd886b-88bc-11eb-9c30-0c42a125994e | 6SRyZ7v0mCP981zBaSIL |           374 |         5 | 2022-03-19 22:05:05
 20 | 1afd8c9f-88bc-11eb-9c30-0c42a125994e | jKFUparzjJAyRrv4DMST |           530 |        43 | 2024-03-19 22:05:05
(20 rows)

In the process of synchronization , You can see that the amount of data is increasing , Finally, I simply checked the data volume and records .

Take another look at the data synchronization task , Discovery has been completed , It is very convenient to use as a whole .

More options

Use py-mysql2pgsql Sync Mysql Data to Greenplum

Use rds_dbsync Sync Mysql Data to Greenplum