Rescue system -- the application of read-write separation

【 Introduce 】

I am a test engineer , It is mainly responsible for the interface and the pressure measurement with convenient performance , Currently working for a medical data company , Since we are doing medical data , So the main business of major companies is medical software .

At the end of last year, the company developed a （ Shenzhen nucleic acid sampling and registration service system ）, Due to the urgent need to deploy , The whole project development 、 The test went online 10 A few days , With the gradual control of the epidemic , So the hospital that this system uses （ user ） Not often . Until this year's 6 month , Due to Guangzhou 、 Shenzhen , The epidemic began to appear , Sampling points are set in all areas of Shenzhen , Nucleic acid sampling , There is a big problem in the system , The system crash caused that the sampling staff could not normally use the applet to register data , The awkward situation of manual registration can only be temporarily adopted .

【 System introduction 】

《 Shenzhen nucleic acid sampling and registration service system 》 The system uses a front-end and back-end separation architecture , The specific technical implementation will not be introduced in detail （ Actually, I don't quite understand ）, The system is deployed in XX center ,5 Application servers （ Do load balancing ）,3 Database servers （ Master-slave ）,2 platform nginx The server （ Master-slave ）

Just talk about the business logic of the system ：

Web backstage （ A hospital or institution ）—— to grant authorization —— Applet （ Sampling staff ）（ When we make free nucleic acid, we are asked to show deep I He scans your QR code ）—— Then scan a tube bar code （ Put the sample tube where we were stabbed by a cotton swab ）—— database ——Web backstage （ A hospital or institution ） View the sampling data of the authorized staff , And use data .

【 events 】

Due to the emergence of several new crown patients in Shenzhen , The government launched the first round of city wide personnel screening , So there was a lot of data that day , In the evening, the database reaches 100 Ten thousand data , Hospitals and institutions in various regions are checking the data , Leading data , Found very slow , After a while, the applet can't store data , The customer immediately called us to check the cause , And fix the problem , It's already night 11 O 'clock , After several hours of troubleshooting , Discovery is caused by too much query data , Optimize SQL after , Obviously improved a lot , However, the large amount of data is not very good .

The specific reason —— Dozens of hospitals are checking the data , Derived data , Waiting all the time , Then the database server keeps CPU Full load , Because it is not load balancing , Cause database crash , Applet , An error is reported when saving data , Finally, the sampling staff manually registered for several hours , It greatly increases the workload of others , After all, I have tested the query interface , But only when the amount of data is very small , be ashamed ！

【 Solution 】

The next day, discuss the solution , The main reason lies in the database , Then work hard on it , After all, it's free Mysql database , Say no good cheap goods , I guess so , So I dare not use it rashly 3 Database servers are used as load balancers to solve the problem . Finally, each recipient decides to use （ Read / write separation ） To solve this problem , The specific logic is , hold 3 All database servers are used , One for applet to write data , One for Web Background query data , Leading data , The other one is used for synchronous data processing .

Add description

In this way, the data volume is large , Background query data 、 The database server crash caused by importing data will not affect the use of applet , Plus the engineering staff 、DB Cooperation of data engineers , Deal with problems arising from data import in a timely manner .

after 1 Days of modification procedures , test , Finally, update to the production environment , For a week , Millions of data every day , No more crashes , However, there is another problem during this period ,Mysql Lock table appears , But after receiving the message ,10 It will be solved in minutes .

The government asked for multiple rounds of investigation , At the same time, it will also open the function of independent code scanning and registration , Therefore, nucleic acid will be made more frequently in the city , More data , Worry about whether the program will have another accident , Affect efficiency , Therefore, cooperate with Tencent to increase server resource allocation on Tencent cloud , Redeploy a set of environment to meet the needs of the people of the whole city , The database will be switched to Oracle, Read write separation is also used in Tencent cloud environment , The following is the topology of the deployment environment ：

Add description

【 expect 】

Do you guys have any better solutions ？ Welcome to discuss .