Jincang KFS data cascade scenario deployment

KFS Data cascade scenario deployment

The data cascading scenario is that the source side data is sent to the intermediate side database , Then it is sent from the intermediate end database to the target end database .KFS Data cascade synchronization is realized through multiple one-to-one data links .

One 、 environmental information

1、 Software version

KFS edition :Kingbase FlySync V001R006C003B20220107

Source end ：oracle 11.2.0.4.0

The middle end ：KingbaseES V008R006C003B0071

Target end ：Mysql 8.0.26

2、 Deployment Topology

KFS Centralized deployment with the database , Deploy on the same server .

3、IP Address

Source end (oracle):192.168.100.111

The middle end (kes):192.168.100.114

Target end (mysql):192.168.100.112

Two 、 Source end (oracle) KFS Deploy

Source end oracle use redo Mode analysis .

( One )、 Environment configuration

1、 Create installation users

groupadd flysync

useradd flysync -g flysync -G mysql

passwd flysync

2、 Upload installation packages and license File to server /home/flysync Under the table of contents , And extract the

tar -xzvf KingbaseFlySync-V001R006C003B20220107-replicator.tar.gz

3、 Configuration will hosts file , Add both source and target servers

vi /etc/hosts

192.168.100.111 oracle

192.168.100.114 kes

192.168.100.112 mysql

192.168.100.113 sqlserver

4、 To configure /etc/security/limits.conf file 【 Optional 】

vi /etc/security/limits.conf

flysync - nofile 65535

flysync - nproc 8096

mssql - nofile 65535

mssql - nproc 8096

5、 Turn on the time synchronization service 【 Optional 】

yum install ntp

systemctl start ntpd

systemctl enable ntpd

6、 stay /etc/sysctl.conf To configure swappiness Parameters 【 Optional 】

vi /etc/sysctl.conf

vm.swappiness = 10

sysctl -p

7、 Check the dependent software jdk and ruby【 must 】

--jdk

java -version

Version less than 1.8 Of , have access to yum install 1.8 edition

yum install java-1.8.0-openjdk.x86_64

You can also download oracle Of 1.8 Version of JDK Manually upload and install packages

--ruby

have access to yum Way to install

yum install ruby

You can also use KFS The console server comes with ruby Package replacement , Position in ：/opt/KFS/console/media/rbenv,

Extract the package to /usr/local, Then configure /etc/profile In the document $PATH environment variable , take ruby Of bin Add directory to

vi /etc/profile

export PATH=$PATH:/usr/local/ruby/bin

When the configuration is complete , Use source Command application

source /etc/profile

（ Two ）、 Database configuration

1、 Configure database time format ：

ALTER SYSTEM SET NLS_DATE_FORMAT='YYYY-MM-DD' SCOPE=SPFILE;

2、 Create connected users and grant relevant permissions ：

create user flysync identified by 123456;

alter user flysync default tablespace users;

alter user flysync quota unlimited on users;

GRANT CONNECT, RESOURCE TO FLYSYNC;

GRANT EXECUTE_CATALOG_ROLE TO FLYSYNC;

GRANT CREATE SESSION TO FLYSYNC;

GRANT SELECT ANY TRANSACTION TO FLYSYNC;

GRANT SELECT ANY TABLE TO FLYSYNC;

GRANT UNLIMITED TABLESPACE TO FLYSYNC;

GRANT DBA TO FLYSYNC;

3、 Turn on Oracle Archive log of database （ Must be turned on ） 了 , The specific steps are as follows ：

(1). View the current archive mode

SQL>select  log_mode  from  v$database;

(2). Create archive Directory

[[email protected] oracle]$ mkdir -p $ORACLE_BASE/archivelog

[[email protected] oracle]$ sqlplus / as sysdba

SQL> alter system set log_archive_dest_1="location=/data/oracle/archivelog" scope=both;

(3). Restart the database to mount state

SQL>shutdown immediate

SQL>startup mount

(4). Change the database to archive mode

SQL>alter database archivelog;

(5). Open database

SQL>alter database open;

(6). Check whether the archive mode is on

SQL>select  log_mode  from  v$database;

4、 Open completion log ：

(1). Check the status of the completion log

SQL> SELECT supplemental_log_data_min, supplemental_log_data_pk,supplemental_log_data_all FROM v$database;

SUPPLEME SUP SUP

-------- --- ---

NO NO NO

(2). Open completion log ( It is suggested that MOUNT Execution in mode )

SQL> ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;

SQL> ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (PRIMARY KEY) COLUMNS;

SQL> ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;

(3). Check the status of the completion log again

SQL> SELECT supplemental_log_data_min, supplemental_log_data_pk, supplemental_log_data_all FROM v$database;

SUPPLEME SUP SUP

-------- --- ---

YES      YES YES

must 3 Items are YES

5、 Switch log files

SQL> ALTER SYSTEM SWITCH LOGFILE;

（ 3、 ... and ）、KFS install

1、 To configure flysync.ini file

Configure... In the home directory of the installation user flysync.ini file

vi  /home/flysync/flysync.ini

Enter the following ：

[defaults]

user=flysync

install-directory=/home/flysync/kfsrep

profile-script=~/.bash_profile

rmi-port=11000

[oracle11g]

role=master

master=oracle

members=oracle

kufl-port=3112

replication-host=192.168.100.111

replication-port=1521

# The user name to connect to the database must be in upper case

replication-user=FLYSYNC

replication-password=123456

datasource-type=oracle

oracle-extractor-method=redo

datasource-oracle-service=orcl

svc-extractor-filters=dropstatementdata,ignoreddl

property=replicator.extractor.dbms.tablePatterns=FLYSYNC.*,TEST.*

property=replicator.extractor.dbms.keepMixDML=false

property=replicator.filter.ignoreddl.ignore=CREATE;ALTER;DROP;TURNCATE

2、 install

Switch to... Under the installation package directory tools subdirectories

cd /home/flysync/KingbaseFlySync-V001R006C003B20220107-replicator/tools

./fspm install

3、 take license Copy the file to the installation directory

cp license_8703_0.dat /home/flysync/kfsrep/license.dat

4、 start-up KFS

replicator start

see KFS state

replicator status

View service status

fsrepctl status

see KUFL list

kufl list

3、 ... and 、 The middle end (KES v8r6)KFS Deploy

( One )、 Environment configuration

1、 Create installation users

groupadd flysync

useradd flysync -g flysync -G mysql

passwd flysync

2、 Upload installation packages and license File to server /home/flysync Under the table of contents , And extract the

tar -xzvf KingbaseFlySync-V001R006C003B20220107-replicator.tar.gz

3、 Configuration will hosts file , Add both source and target servers

vi /etc/hosts

192.168.100.111 oracle

192.168.100.114 kes

192.168.100.112 mysql

192.168.100.113 sqlserver

4、 To configure /etc/security/limits.conf file 【 Optional 】

vi /etc/security/limits.conf

flysync - nofile 65535

flysync - nproc 8096

mssql - nofile 65535

mssql - nproc 8096

5、 Turn on the time synchronization service 【 Optional 】

yum install ntp

systemctl start ntpd

systemctl enable ntpd

6、 stay /etc/sysctl.conf To configure swappiness Parameters 【 Optional 】

vi /etc/sysctl.conf

vm.swappiness = 10

sysctl -p

7、 Check the dependent software jdk and ruby【 must 】

--jdk

java -version

Version less than 1.8 Of , have access to yum install 1.8 edition

yum install java-1.8.0-openjdk.x86_64

You can also download oracle Of 1.8 Version of JDK Manually upload and install packages

--ruby

have access to yum Way to install

yum install ruby

You can also use KFS The console server comes with ruby Package replacement , Position in ：/opt/KFS/console/media/rbenv,

Extract the package to /usr/local, Then configure /etc/profile In the document $PATH environment variable , take ruby Of bin Add directory to

vi /etc/profile

export PATH=$PATH:/usr/local/ruby/bin

When the configuration is complete , Use source Command application

source /etc/profile

( Two )、 Database configuration

  Create a connection database account and authorize ：

ksql>CREATE USER FLYSYNC SUPERUSER PASSWORD '123456';

（ 3、 ... and ）、KFS install

1、 Check the database decoderbufs Is the plug-in installed

---- Check V8 Install under directory /lib Catalog , Is there a decoderbufs.so and kfs_current_query.so package , If there is no , take KingbaseFlySync Included with the installation package decoderbuf.so and kfs_current_query.so Copied to the Kingbase Installation directory /lib Under the table of contents .

---- Check V8 Install under directory /share/extension Catalog , Is there a decoderbufs.control file , If there is no , take KingbaseFlySync Included with the installation package decoderbuf.control Copied to the Kingbase Installation directory /share/extension Under the table of contents

ls   /opt/Kingbase/ES/V8/Server/lib/decoderbufs.so

ls   /opt/Kingbase/ES/V8/Server/lib/kfs_current_query.so

ls   /opt/Kingbase/ES/V8/Server/share/extension/decoderbufs.control

without , decompression KFS The compressed package included in the installation package , Copy these files to the appropriate directory

cd  /home/flysync/KingbaseFlySync-V001R006C003B20220107-replicator/extensions/decoderbufs

cp kfs_decoderbufs_lib_linux64_v8r3.zip  /home/kingbase

cd /home/kingbase

unzip kfs_decoderbufs_lib_linux64_v8r3.zip

cp ./lib/decoderbufs.so /opt/Kingbase/ES/V8/Server/lib

cp ./lib/kfs_current_query.so /opt/Kingbase/ES/V8/Server/lib

cp ./share/extension/decoderbufs.control  /opt/Kingbase/ES/V8/Server/share/extension

chown kingbase:kingbase /opt/Kingbase/ES/V8/Server/lib/decoderbufs.so

chown kingbase:kingbase /opt/Kingbase/ES/V8/Server/lib/kfs_current_query.so

chown kingbase:kingbase /opt/Kingbase/ES/V8/Server/share/extension/decoderbufs.control

2、 Edit installation directory data/sys_hba.conf file , Add the following configuration , Enable the user to have copy permission :

vi /opt/Kingbase/data/sys_hba.conf

host replication all 127.0.0.1/32    md5

host replication all 0.0.0.0/0       md5

3、 Edit installation directory data/kingbase.conf file , Modify the following configuration ：

vi /opt/Kingbase/data/kingbase.conf

max_wal_senders=4          # Number of log sending processes , Number of databases *2, The minimum is 4

wal_keep_segments=4        # Number of logs retained , According to disk space settings , The bigger the better  

wal_level=logical          # The level of logging , It has to be for logical       

max_replication_slots=4    # Number of copy slots , Number of databases *2, The minimum is 4

Once the configuration is complete , Restart the database to make the configuration effective .

sys_ctl -D /opt/Kingbase/data restart

4、 To configure flysync.ini

Middle end flysync.ini The file configures two services , A target side service , Corresponding to the source end oracle11g; A source side service , Corresponding to the target end mysql8

[defaults]

install-directory=/home/flysync/kfsrep

profile-script=~/.bash_profile

rmi-port=11000

# Target side services

[oracle11g_kingbase8]

role=slave

master=oracle

master-kufl-port=3112

members=kes8

kufl-port=3113

replication-host=192.168.100.114

replication-port=54321

replication-user=flysync

replication-password=123456

datasource-type=kingbase

datasource-version=8

kingbase-dbname=TEST

svc-parallelization-type=none

svc-remote-filters=casetransform,rename

property=replicator.filter.casetransform.to_upper_case=false

property = replicator.filter.rename.definitionsFile=/home/flysync/kfsrep/filters-config/oracle11g_kingbase8_rename.csv

property=replicator.applier.dbms.optimizeRowEvents=true

property=replicator.applier.dbms.maxRowBatchSize=5000

# Source side service

[kingbase8]

role=master

master=kes8

members=kes8

kufl-port=3112

replication-host=192.168.100.114

replication-port=54321

replication-user=FLYSYNC

replication-password=123456

datasource-type=kingbase

datasource-version=8

kingbase-extractor-method=xlogical

kingbase-dbname=TEST

svc-extractor-filters=replicate

property=replicator.filter.replicate.do=PUBLIC.*,flysync_kingbase8.*

property=replicator.filter.replicate.ignore=PUBLIC.T1

# Large transaction splitting parameters , arrive 4000 Line start split transaction , The default is 500

property=replicator.extractor.dbms.minRowsPerBlock=4000

# Sync DDL Statement required , Supported can be configured DDL type

property=replicator.extractor.dbms.ddlListFile=/home/flysync/kfsrep/filters-config/ddl_support_list.csv

# Starting synchronization DDL Effective after ,create table as Type statements , At the same time produce DDL and DML, Whether to keep DML,true For the sake of reservation ,false In order not to keep , The default is true.

property=replicator.extractor.dbms.keepMixDML=true

5、 from KFS Under the installation package directory extensions/jdbc Copy jdbc package

V8R3（ Not included ） The version above , Use kingbase8-8.6.0.jar

V8R3( contain ) The following versions , Use kingbase8-8.2.0.jar

cd /home/flysync/KingbaseFlySync-V001R006C003B20220107-replicator/extensions/jdbc

cp kingbase8-8.6.0.jar /home/flysync/KingbaseFlySync-V001R006C003B20220107-replicator/flysync-replicator/lib

6、 install

cd /home/flysync/KingbaseFlySync-V001R006C003B20220107-replicator/tools

./fspm install

7、 Reply to license File to installation directory

cp license.dat /home/flysync/kesrep/license.dat

8、 modify rename file

Modify the target side service oracle11g_kingbase8 Of rename file

vi /home/flysync/kfsrep/filters-config/oracle11g_kingbase8_rename.csv

FLYSYNC,trep_commit_seqno,*,flysync_oracle11g_kingbase8,-,-

FLYSYNC,consistency,*,flysync_oracle11g_kingbase8,-,-

FLYSYNC,heartbeat,*,flysync_oracle11g_kingbase8,-,-

FLYSYNC,trep_shard,*,flysync_oracle11g_kingbase8,-,-

FLYSYNC,trep_shard_channel,*,flysync_oracle11g_kingbase8,-,-

test,*,*,public,-,-

9、 Start and initialize

/home/flysync/kesrep/flysync/cluster-home/bin/startall

source ~/.bash_profile

see KFS state

replicator status

View service status

fsrepctl status

see KUFL list

kufl list

Four 、 Target end (mysql8)KFS Deploy

( One )、 Environment configuration

1、 Create installation users

groupadd flysync

useradd flysync -g flysync -G mysql

passwd flysync

2、 Upload installation packages and license File to server /home/flysync Under the table of contents , And extract the

tar -xzvf KingbaseFlySync-V001R006C003B20220107-replicator.tar.gz

3、 Configuration will hosts file , Add both source and target servers

vi /etc/hosts

192.168.100.111 oracle

192.168.100.114 kes

192.168.100.112 mysql

192.168.100.113 sqlserver

4、 To configure /etc/security/limits.conf file 【 Optional 】

vi /etc/security/limits.conf

flysync - nofile 65535

flysync - nproc 8096

mssql - nofile 65535

mssql - nproc 8096

5、 Turn on the time synchronization service 【 Optional 】

yum install ntp

systemctl start ntpd

systemctl enable ntpd

6、 stay /etc/sysctl.conf To configure swappiness Parameters 【 Optional 】

vi /etc/sysctl.conf

vm.swappiness = 10

sysctl -p

7、 Check the dependent software jdk and ruby【 must 】

--jdk

java -version

Version less than 1.8 Of , have access to yum install 1.8 edition

yum install java-1.8.0-openjdk.x86_64

You can also download oracle Of 1.8 Version of JDK Manually upload and install packages

--ruby

have access to yum Way to install

yum install ruby

You can also use KFS The console server comes with ruby Package replacement , Position in ：/opt/KFS/console/media/rbenv,

Extract the package to /usr/local, Then configure /etc/profile In the document $PATH environment variable , take ruby Of bin Add directory to

vi /etc/profile

export PATH=$PATH:/usr/local/ruby/bin

When the configuration is complete , Use source Command application

source /etc/profile

（ Two ）、 Database configuration

1、mysql System parameter configuration

vi /etc/my.cnf

[mysqld]

local-infile

character-set-server = utf8

collation-server = utf8_general_ci

default-time-zone = '+08:00'

Restart the database after saving

systemctl restart mysqld

2、 Connection account configuration

Set up an account and give corresponding permissions

mysql>CREATE USER [email protected]'%' IDENTIFIED BY '123456';

mysql>GRANT ALL ON *.* TO [email protected]'%' WITH GRANT OPTION;

( 3、 ... and )、KFS install

1、 To configure flysync.ini

[defaults]

install-directory=/home/flysync/kfsrep

profile-script=~/.bash_profile

rmi-port=11000

[kingbase8mysql8]

skip_validation_check=MySQLDumpCheck,MySQLPermissionsCheck

role=slave

master=kes8

master-kufl-port=3112

members=mysql

kufl-port=3112

replication-host=192.168.100.112

replication-port=3306

replication-user=flysync

replication-password=123456

datasource-type=mysql

svc-remote-filters=casetransform,rename

property=replicator.filter.casetransform.to_upper_case=false

replicator.filter.rename.definitionsFile=/home/flysync/kfsrep/filters-config/kingbase8mysql8_rename.csv

2、 install

cd /home/flysync/KingbaseFlySync-V001R006C003B20220107-replicator/tools

./fspm install

3、 Reply to license File to installation directory

cp license.dat /home/flysync/kesrep/license.dat

4、 modify rename file

vi /home/flysync/kfsrep/filters-config/kingbase8mysql8_rename.csv

flysync_kingbase8,*,*,flysync_kingbase8mysql8,-,-

public,*,*,test,-,-

5、 Start and initialize

/home/flysync/kesrep/flysync/cluster-home/bin/startall

source ~/.bash_profile

see KFS state

replicator status

View service status

fsrepctl status

see KUFL list

kufl list

5、 ... and 、 Data relocation

( One )、 Table structure migration

Table structure migration uses the extremely fast migration mode , Execute on the intermediate end and the target end , All data sources point to the source side database .

1. The middle end (oracle11g_kingbase8)

ddlscan -target.service oracle11g_kingbase8 -source.user FLYSYNC -source.pass 123456 -source.db orcl -source.dbtype oracle -source.host 192.168.100.111 -source.port 1521 -source.schema TEST  -target.db test -mgType 0

1. Target end (kingbase8mysql8)

ddlscan -target.service kingbase8mysql8 -source.user FLYSYNC -source.pass 123456 -source.db orcl -source.dbtype oracle -source.host 192.168.100.111 -source.port 1521 -source.schema TEST  -target.db test -mgType 0

( Two )、 Stock data migration

Data cascading scenario , Data migration can use special patterns ( Source end The assembly line moves smoothly ）, Execute... At the source , In this way, the source side can generate the stock data KUFL Log synchronization to the middle end , The middle end can automatically synchronize data to the target end . With this mode, data migration can be performed only once to automatically migrate the data to the middle end and the target end .

Special mode relocation requires source side 、 Intermediate end and target end kfs Service is online state .

Source side execution ：

loader -source.service oracle11g -source.user FLYSYNC -source.pass 123456 -source.db orcl -source.dbtype oracle -source.host 192.168.100.111 -source.port 1521 -source.schema TEST -source.rmiHost 192.168.100.111 -source.rmiPort 11000 -mgType 2 -clean

During the relocation of special mode , Do not modify the service status manually . If the service status is abnormal due to external reasons , Need to

Service amended to online And then move again .