Pg-pool-ii read / write separation experience

• PG-Pool-II function
• PG-Pool-II install
  • PG-Pool-II Configuration and use
• PG-Pool-II Read-write separation test
• PG-Pool-II The performance and SQL test
  • Read only test ：
  • Read and write test ：
• Preliminary conclusion

PG-Pool-II function

• Connection pool

PG It's a multiprocess structure , Connection pooling can reduce the problem of connection vacancy , Improve resource utilization .

• Master slave copy + online recovery+auto failover

adopt pgpool management Master-slave relationship , Realization PG Master-slave high availability .

• Load balancing

When there are multiple slave Libraries , You can automatically load requests from different nodes through weight settings , Optimize performance .

• Connections limit

Set the connection number control , Avoid excessive connections that may cause access errors , When the number of connections is exceeded, go online , Queue for subsequent access .

• High availability

adopt pgpool Management ability , Can achieve its own high availability , Avoid performance problems with connection pools .

• The query cache

Cache in memory SQL result . If one is the same SQL, You can return a value from the cache , Improve query performance .

PG-Pool-II install

Resources to prepare ：

CVM 1 platform ,centos 7.2 64 position , To configure 32C64GB.

Cloud database PostgreSQL, Main instance 1c2GB,200GB disk , Read only instance 2 individual ,1c2GB,200GB disk , Combine into a read-only instance group .

1. Install according to the operating system pg The official yum Source .yum install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdgredhat-repo-latest.noarch.rpm
2. install pgpool-II Installation package , What we use here is 4.1.4 Example , You can choose the 4.2.4 edition . yum install -y pgpool-II-11 yum install -y pgpool-II-11-extensions yum install -y pgpool-II-11-devel
3. View folder ：/etc/pgpool-II-11/, Whether there is a file .

PG-Pool-II Configuration and use

The main configuration files are as follows ：

/etc/pgpool-II-11/pgpool.conf  , Master profile

/etc/pgpool-II-11/pool_hba.conf, The connection profile is similar to pg_hba.conf

/etc/pgpool-II-11/pool_passwd, Access user and password files , It mainly configures the access user name and password , Authentication for connecting to the database .

/etc/pgpool-II-11/pgpool.conf The main configuration information is as follows ：

# Basic connection information

# - pgpool Connection Settings -

listen_addresses ='*'

port =9999

socket_dir='/var/run/pgpool-II-11'

# - Manage interface configuration information , No configuration -

pcp_listen_addresses =''

pcp_port =5432

pcp_socket_dir ='/var/run/pgpool-II-11'

# - Database master node connection information configuration , important

backend_hostname0 ='172.27.32.11'

backend_port0 =5432

backend_weight0 =1

backend_data_directory0 ='/data/pgsql11/master'

backend_flag0 ='ALWAYS_MASTER'

backend_application_name0 ='master'

# Connection information of read-only instance or read-only instance group , To configure PG Read only group VIP

backend_hostname1 ='172.27.32.12'

backend_port1 =5432

backend_weight1 =1

backend_data_directory1 ='/data/pgsql11/ro'

backend_flag1 ='DISALLOW_TO_FAILOVER'

backend_application_name1 ='ro'

# - Authentication login information -

enable_pool_hba = on

pool_passwd ='pool_passwd'

# Relevant document information

pid_file_name ='/var/run/pgpool-II-11/pgpool.pid'

# Copying model

replication_mode = off

#load balance Whether the mode is on

load_balance_mode = on

#  Whether the master-slave mode is enabled （ important , Read write separation related ）

master_slave_mode = on

master_slave_sub_mode ='stream'

# Whether the health check is on , It is mainly used for fail over function

health_check_period =0

health_check_timeout =20

health_check_user ='nobody'

health_check_password =''

health_check_database =''

# Log related configuration

log_destination ='syslog'

log_connections = on

log_hostname = on

log_statement = on

log_per_node_statement = on

pool_hba.conf File configuration , Be similar to pg_hba.conf file ：

Add the following contents to the end of the file ：host    all all 0.0.0.0/0 md5

pool_passwd User name and password information , This file can be generated by executing the following statement ：

pg_md5 --md5auth --username=pgpool [email protected]

After configuration , Execute the service startup statement to start pg_pool.

systemctl start pgpool-II-11.service

After starting the service , It can be done by systemctl status pgpool-II-11.service See the startup of the service .

PG-Pool-II Read-write separation test

1. adopt pgbench adopt pgpool Perform basic data irrigation on the database . /usr/pgsql-11/bin/pgbench -U haha -p 9999 -h 172.27.32.12 test -i -s 10
2. Create a new one sql File to test vi test.sql

\set aid random_gaussian(1, :range,10.0)

\set bid random(1,1* :scale)

\set tid random(1,10* :scale)

\set delta random(-5000,5000)

SELECT abalance FROM pgbench_accounts WHERE aid = :aid;

SELECT tbalance FROM pgbench_tellers WHERE tid = :tid;

SELECT bid,aid,delta,mtime FROM pgbench_history WHERE tid = :tid;

UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid; 

SELECT abalance FROM pgbench_accounts WHERE aid = :aid; 

UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid; 

UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid; 

INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);

1. adopt pgpool 2 Perform database read / write data test ： /usr/pgsql-11/bin/pgbench -U haha -p 9999 -h 172.27.32.9 test -v -r -P 1 -f ./test.sql -c 300 -j 300 -T 120 -D scale=10000 -D range=100000000
2. Because in the above configuration file , We turned on logging SQL sentence , Therefore, you can view what has been executed by different nodes through logs SQL. cat /var/log/messages |grep "DB node id: 1" cat /var/log/messages |grep "DB node id: 0"
3. among node ID by 1 is We backend_hostname1 Configured node 1, For our read-only instance group address .
4. Where nodes ID by 0 And ours backend_hostname0 Configured nodes 0, For our master node .
5. You can see that both read and write statements are distributed to different nodes . The read-write separation function is successfully realized .

cat /var/log/messages |grep"DB node id: 0"

Aug3022:24:49VM-32-9-centos pgpool[3968]: [25447-1]2021-08-3022:24:49: pid3968: LOG:  DB node id:0backend pid:33553statement: BEGIN;

Aug3022:24:49VM-32-9-centos pgpool[5126]: [25656-1]2021-08-3022:24:49: pid5126: LOG:  DB node id:0backend pid:33531statement: UPDATE pgbench_tellers SET tbalance = tbalance + -651WHERE tid =42973;

Aug3022:24:49VM-32-9-centos pgpool[3974]: [25559-1]2021-08-3022:24:49: pid3974: LOG:  DB node id:0backend pid:33551statement: END;

……

cat /var/log/messages |grep"DB node id: 1"

Sep 112:03:25VM-32-9-centos pgpool[20505]: [26250-1]2021-09-0112:03:25: pid20505: LOG:  DB node id:1backend pid:20410statement: SELECT abalance FROM pgbench_accounts WHERE aid =45648796;

Sep 112:03:25VM-32-9-centos pgpool[20520]: [27210-1]2021-09-0112:03:25: pid20520: LOG:  DB node id:1backend pid:20411statement: SELECT abalance FROM pgbench_accounts WHERE aid =56012538;

Sep 112:03:25VM-32-9-centos pgpool[20522]: [28362-1]2021-09-0112:03:25: pid20522: LOG:  DB node id:1backend pid:20401statement: SELECT abalance FROM pgbench_accounts WHERE aid =53695003;

……