Infrastructure splitting of service splitting

Infrastructure unbundling of services

background : For historical reasons , In the early stage, multiple services share one rds Instance and a redis example , In actual use, it is often caused by a service exception rds or redis Abnormal load , This will affect other services and cause avalanches . Therefore, basic resources are split to isolate risks .

Background : Due to historical reasons, multiple services share one RDS instance and one REDIS instance in the early stage. In actual use, it is common to cause RDS or REDIS load abnormalities due to one service exception, thus affecting other services and causing avalanches. Therefore, base resource splitting is performed to isolate risks.

This split is based on AWS platform The split is based on AWS

Create a read-only copy of the original instance

Create a read-only copy instance of the original instance

principle principle

aws console-> rds console-> databases -> select rds-> action-> creat read replica

take k8s On pod The number of copies is adjusted to 0

Set the number of pod replicas on K8S to 0

kubectl edit deployments test -n test

set spec.replicas: 0

Verify whether the number of rows of the read-only instance and the original instance record is the same

Verify that the read-only instance and the original instance record the same number of rows

Not recommended for use schema Statistics Error It is not recommended to use schema statistics with errors information_schema.tables about InnoDB surface ,table_rows Row counts are only approximate estimates , inaccurate . information_schema.tables For InnoDB tables, table_rows rows count is only approximate and not accurate.

Use the following sql Generate the number of record rows of all tables under the query library ：

Use the following SQL to generate the number of record rows for all tables under the query library:

select concat(
    'select  count(*) from ',
    TABLE_SCHEMA, 
    '.',
    TABLE_name,
    ' union all'
) from information_schema.tables 
where TABLE_SCHEMA='test_db';

Generate a query similar to the following sql, Query the original instance and the read-only instance generate query SQL similar to the following for the original and read-only instances

select count(*) from table1
union all
select count(*) from table2
union all
select count(*) from table3
...

Will read only rds The instance is promoted to normal rds example

Promote a read-only RDS instance to a normal RDS instance

When the read-only instance is consistent with the source instance, promote the read-only instance to a normal instance Promote a read-only instance to a normal instance after it is consistent with the source instance

aws console-> rds console-> databases -> select rds-> action-> promote

establish redis Backup , And restore a new redis

Create a Redis backup and restore a new Redis

aws console->redis->action->backup

aws console->redis->resotre

Modify the configuration center connection information

Example Modify the connection information of the configuration center

datasource.host

redis.host

...

take k8s pod Number of replicas number before recovery

Restore the number of k8S pod copies to previous number

kubectl edit deployments test -n test

set spec.replicas: x

take aws Disassembled resources rds redis Import to existing terraform in

Import the resource RDS Redis from AWS into the existing Terraform

Refer to the following Refer to the following

Terraform Reverse export

summary

to summarize

This split can guarantee data 0 Loss , For k8s pod Copy count adjustment , The split services will be unavailable for some time according to the actual situation , It is recommended to do this when the service traffic is low

This split can ensure zero data loss. Because the number of K8S POD copies is adjusted, the split service may be unavailable for some time according to the actual situation. You are advised to perform this operation when the service traffic is low