Practice of business level disaster recovery switching drill

Business disaster recovery is the most complex scenario among all disaster recovery scenarios , Business applications involved 、 middleware 、 Database and underlying computing 、 Storage 、 Network and other resources . In terms of cloud business disaster recovery, the whole disaster recovery covers IaaS、PaaS、SaaS layer . After the disaster recovery plan is confirmed and implemented , It is necessary to carry out disaster recovery switching drill . The following mainly introduces the process and specific operation details of the disaster recovery switching drill .

Disaster recovery switching scheme diagram

One 、 Ensure that all relevant application level middleware components in the standby zone are deployed .

First, assume that the entire disaster recovery switching scenario is conducted in Guangzhou District 4 and District 6 , The fourth district is the main district 、 Zone 6 is the standby zone . Set up an application deployment environment in zone 6 without starting the business . Then according to the four zones and six zones 1 Than 1 To deploy mysql、kafka、ES、redis、MQ And so on .

Two 、 simulation 4 Zone fault

Batch stop with script 4 Application level middleware components of the zone , If necessary , You can stop the network . After observation , Send the whole 4 Zone related applications , The middleware components are all stopped , also 4 The network from District 6 to District 6 is also blocked .

3、 ... and 、 Perform the switching steps of the entire business disaster recovery

1、 Application inlet traffic switching . On the disaster recovery switching platform , Point the system client traffic to zone 6 .

matters needing attention ：① Ensure that the traffic switching takes effect ;② Observe whether the overall system flow is affected after switching .

2、 Database component switching .

Specific steps ：

① The old master database in zone 4 is set to read-only status

② Check that the synchronization is complete , Set to read / write status ;

③ datax modify , All four areas will be job Set to not running state , Open the disaster recovery area job;

④ Backup scripts enable checking ;

⑤ perform ip Switch .

Precautions for disaster recovery ：

① You need to ensure that the four zone master database is set to read-only status , Ensure data consistency .

② Pay attention to whether the data synchronization is normal .

3、 Middleware component domain name switching

Specific steps ：

① kafka The traffic of the component is switched to the six zone node . After cutting , Message read from the client / Write Whether to go to zone 6 normally .

② redis Switch to the Sixth District of Guangzhou , Perform application request domain name switching . Take the initiative failover. After switching , Observe whether the switching is normal , Whether the flow is normal .

③ MQ Switch to the Sixth District of Guangzhou , You need to turn off before switching 4 The node of the zone .

④ Spare area cancel start-up , Synchronize the data in the database to ES.

Four 、 Perform the development platform switching steps

Introduce the traffic on the customer side into the standby area 6 From the middle of the district , Observe whether the entire link request is normal , Whether each middleware operates correctly , Whether data synchronization ensures consistency .