This is Wukong's first 152 Original articles

Official website ：www.passjava.cn

Hello , I'm Wukong .

Last time our project did not put MySQL Is the high availability deployment ready ,MySQL Dual master mode + Keepalived, To ensure high availability . Simply put, there are two MySQL Master node , There are two Keepalived Installed on the host computer to monitor MySQL The state of , Once problems are found , Just restart MySQL, And the client will automatically connect to another computer MySQL.

For details, please see this article written by Wukong ： actual combat MySQL High availability Architecture

This is an accident we encountered in the project , Let's go over it .

The contents of this article are as follows ：

The scene of the accident

  • Environmental Science ： Test environment
  • Time ： In the morning 10:30
  • Feedback personnel ： Test group , There's a frying pan , After preliminary investigation by R & D colleagues , It is found that there may be a database problem .

Then start looking for reasons . Because this cluster environment is deployed by me , So if I came to check, I was familiar with it .

System deployment diagram

First, let's talk about the deployment diagram of the system , So that you can understand .

Two databases are deployed in node55 and node56 Node , They are in a master-slave relationship with each other , So it is called double master .

There are two Keepalived Deployed in node55 and node56 above , Separately monitor MySQL Container state .

Reasons for error reporting and solutions

  • ① My first thought was , Not having Keepalived To ensure high availability , Even if MySQL Hang up , It can also be done through Keepalived To restart automatically . Even if one fails to restart , There is another one that can be used ？
  • ② Then go to the server and have a look MySQL The state of the container . To MySQL On two servers of , Let's take a look at MySQL Container state ,docker ps command , Found two MySQL Containers are not in the list , This means that the container is not functioning properly .

  • ③ It's impossible , I installed Keepalived High availability components , Don't Keepalived I've also hung up ？
  • ④ Check the wave quickly Keepalived, Found two Keepalived It works . View by executing the command ：systemctl status keepalived

  • ⑤ what ,Keepalived It's normal , Keepalived It will restart every few seconds MySQL, Maybe I didn't see it in that short free time MySQL Container start up ？ Execute another command ,docker ps -a, List the status of all containers . You can see MySQL Started and exited , explain MySQL It's really rebooting .

  • ⑥ That means Keepalived Although it was restarted MySQL Containers , however MySQL I have a problem with myself , that Keepalived There is no way to improve the high availability of .
  • ⑦ Then how to fix ？ Just look at MySQL Report something wrong . Execute the command to view the container log .docker logs < Containers id>. Find the recent log ：

  • ⑧ Tips mysql-bin.index file does not exist , This file is configured for master-slave synchronization , stay my.cnf In the configuration .

After this configuration , Then, when performing master-slave synchronization , Will be in var/lib/mysql/log Multiple... Are generated under the directory mysql-bin.xxx The file of . One more mysql-bin.index Index file , It will mark now binlog Where are the log files recorded .

mysql-bin.index The contents of the document are as follows ：

/var/lib/mysql/log/mysql-bin.000001

This mysql-bin.000001 The document is still numbered , There's still a pit here , I'll talk about it later .

⑨ The error message indicates that there is a lack of mysql-bin.index, Let's check it out , Not really ！ No matter how the file disappeared , Get this log Create the folder first , then mysql It will automatically generate this file for us .

Solution ： Execute the following command to create a folder and add permissions .

mkdir logchmod 777 log -R

⑩ This is available on both servers log After the directory ,Keepalived Also help us restart automatically MySQL Containers , Then visit one of the nodes node56 Of MySQL The state of , Why , It's the wrong report .

Last_IO_Error: Got fatal error 1236 from master when reading data from binary log: 'Could not find first log file name in binary log index file'

You can see several key messages ：

  • Slave_IO_Running: NO, Currently synchronized I/O The thread is not running , This I/O Threads are from the library , It will request the master database binlog, And will get binlog Write local relay-log ( relay logs ) In file . Is not running , It means that the slave database synchronization is not running normally .
  • Master_Log_File: mysql-bin.000014, This indicates that the currently synchronized log file is 000014, We saw the node before node56 On mysql.index It says 000001, This 000014 Not at all index In the document , So it will report an error .

This involves the principle of master-slave synchronization , Previous picture ：

Two threads are generated from the library , One I/O Threads , One SQL Threads ;

I/O The thread will request the main library binlog Log files , And will get binlog Log files Write local relay-log ( relay logs ) In file ;

The main library will generate a dump Threads , Used to give to the slave I/O Thread transfer binlog;

SQL Threads , Will read relay log Log in file , And resolved into SQL Statements are executed one by one .

That right , We re specify which log file to synchronize , And the location of synchronization .

Solution ：

Look at the main library node55 Log file status on .

Write down these two messages ：File=mysql-bin.00001,Position=117748.（ There's also a hole here ： First lock the watch , Look at these two values , After starting synchronization from the library , Unlock the table ）.

The specific orders are as follows ：

FLUSH TABLES WITH READ LOCK;

SHOW MASTER STATUS

UNLOCK TABLES

Then from the library node56 Reassign the synchronized log file and location on the ：

#  Stop synchronizing from the Library STOP SLAVE;#  Set sync file and location CHANGE MASTER TO MASTER_HOST='10.2.1.55',MASTER_PORT=3306,MASTER_USER='vagrant',MASTER_PASSWORD='vagrant',MASTER_LOG_FILE='mysql-bin.000001',MASTER_LOG_POS=117748;#  Turn on synchronization START SLAVE;

Check again and no error will be reported ,I/O The thread also runs ,

Insert picture description here

And then node55 As a slave Library ,node56 As the master library , Also perform the above steps , The status display is normal , And then use navicat Connect the tool to the database , It's all normal , Under the feedback of the test group , Fix up the work .

I seem to have forgotten a question , Why? log The folder was wiped out ？？

Why there's a problem ？

Then I asked if anyone had deleted this at that time /var/lib/mysql/log Catalog , No one will delete this directory casually .

But found log The parent directory of /var/lib/mysql There are many other folders , such as xxcloud, xxcenter etc. . This is the name of several databases in our project , Just in the folder of this directory , Will be shown in navicat On , It's one-to-one , As shown in the figure below . It also shows log database .

Is there anyone from navicat I got rid of it log database ？ Very likely ！

Sure enough , A colleague was in the process of migration and upgrading , Found this log There is no database in the old system , So I cleaned it up , This is equivalent to log The database is down , At the same time, I will also put log The folder is gone . Okay , Finally, it's all over the place ！ In fact, I didn't consider this in the early stage log A problem with the catalog . you 're right , This is my pot ~

improvement

Actually, when you synchronize the database , This should not be used to override synchronization , Single database synchronization can be adopted , Will not kill log Database . however , This log It's a little strange to put the database here , Can you not show up here ？

We just specify this log The directory is not in /var/lib/mysql It's just in the catalog .

Dongge suggested ：log Files and databases data File isolation ：

datadir = /var/lib/mysql/data

log_bin = /var/lib/mysql/log

Another question , Is our high availability really high ？

At least they didn't call the police in time ,MySQL Database hung , I don't know , They are all feedback from students through tests .

Can you feel it in time MySQL Unusual ？

It can be used here Keepalived The function of sending e-mail , Or through the log alarm system . This is what needs to be improved in the future .

  • END -

once MySQL Accidents caused by misoperation ,「 High availability 」 I can't stand it anymore ！ More articles about

  1. MySQL Data recovery after misoperation （update,delete Forget to add where Conditions ）【 turn 】

    In the daily maintenance of database , Developers are the biggest headache , A lot of times, it's because of SQL sentence There is a problem with the writing, which leads to a problem with the server , Lead to exhaustion of resources . The most dangerous operation is doing DML Forget to add where Conditions , Results in a full table update , This is for operation and maintenance or ...

  2. MySQL How to recover data quickly after misoperation

    Basically every programmer who works with a database ( Of course, it could be your colleagues ) There's a problem ,MySQL How to quickly roll back after misoperation ? such as ,delete A watch , Forget the restrictions , The whole table is gone . If this is still the core business data of online environment , That's a big deal ...

  3. MySQL5.7 below , Caused by misoperation drop table db1.tb1; The recovery method of this method ：

    MySQL5.7 below , Caused by misoperation drop table db1.tb1; The recovery method of this method : 0. Stop business data writing .[iptables Prohibition ] 1. Pull the latest full file from the backup server , Restore to a temporary server ...

  4. MySQL After misoperation and deletion , How to recover data ？

    MySQL After misoperation and deletion , How to recover data ? Log in to the database mysql> select * from abc.stad;+----+-----------+| id | name |+----+----- ...

  5. MySQL How to recover data quickly after misoperation ~！~！~

    Basically every programmer who works with a database ( Of course, it could be your colleagues ) There's a problem ,MySQL How to quickly roll back after misoperation ? such as ,delete A watch , Forget the restrictions , The whole table is gone . If this is still the core business data of online environment , That's a big deal ...

  6. MySQL How to quickly roll back after misoperation ( turn )

    In this paper, from http://www.cnblogs.com/dfcao/p/6147970.html#undefined Thanks to the author Basically every programmer who works with a database ( Of course, it could be your colleagues ) There's a problem ,My ...

  7. MySQL How to recover data quickly after misoperation ？

    Abstract : utilize binlog Flash back misoperation data . Basically every programmer who works with a database ( Of course, it could be your colleagues ) There's a problem ,MySQL How to quickly roll back after misoperation ? such as ,delete A watch , Forget the restrictions , The whole table is gone . If ...

  8. Solution - problem 001： The physical machine 、 Virtual machine, etc Linux operating system /usr/bin Misoperation of directory permissions , Cause unable to switch root

    Introduction : Usually, the operation and maintenance personnel will misoperate some directory permissions , Cause some problems , So how to recover ? problem : The physical machine . Virtual machine, etc Linux operating system /usr/bin Misoperation of directory permissions , Cause unable to switch root? Experimental environment : ip Address Whether the directory is ...

  9. MySQL Data recovery after misoperation （update,delete Forget to add where Conditions ）

    In the daily maintenance of database , Developers are the biggest headache , A lot of times, it's because of SQL There is a problem with the writing of the statement, which leads to a problem with the server , Lead to exhaustion of resources . The most dangerous operation is doing DML Forget to add where Conditions , Results in a full table update , This is for operation and maintenance or D ...

  10. binlog2sql Realization MySQL Recovery of misoperation

    about MySQL The recovery of deleted data caused by misoperation in the database , Can be used based on MySQL in binlog Do something similar to flashback or generating reverse operations SQL Statement to implement , yes MySQL A very useful function in . The principle is not difficult to understand , be based on MySQL Of ...

Random recommendation

  1. Linux Lower installation Nginx1.9.3

    operating system :CentOS6.5 64bit Nginx: 1.9.3 1. download Nginx [[email protected] softs]# wget http://nginx.org/downloa ...

  2. understand ip And port

    understand IP And port IP The address is a rule , Now we use IPv4, Since 4 individual 0-255 Between the numbers make up , When it's stored inside the computer, it just needs 4 Only bytes . In the computer ,IP The address is assigned to the network card , Each Nic has a unique IP Address , If ...

  3. centos7 Add the function of graphical interface

    General installation centos The minimum version has no graphical interface by default You need to install it yourself sudo  yum groupinstall "GNOME Desktop" "Graphical Adm ...

  4. Remove WebCakeDesktop

    WebCakeDesktop.Updater.exe It's the advertising process , Please refer to http://malwaretips.com/blogs/webcake-desktop-updater-exe-rem ...

  5. MVC+EF Update the database

    Use the migration function provided by the code first to ensure the automatic matching between the model and the database , In the library package manager, execute the following commands in turn :1. Enable migration :Enable-Migrations -ContextTypeName MvcMovie.M ...

  6. form The form has no refresh submission file （iframe）

    Let's start with a piece of code (PHP Example ) 1. Form code (form.html): <iframe name="testIframeName" style="display:none ...

  7. xcode archive export ipa Time re signature

    Tossed around all night with another signature from xcode archive export ipa, Finally, I found that there was no import p12 The reason for the document .... By the way, record the reference posts : http://stackoverflow.com/quest ...

  8. Super tough PHP Code editor PHPstorm And configuration

    How to download and install Official website :http://www.jetbrains.com/phpstorm/, What this article shows is 5.0 Version to , therefore , If your software version is too high , It may be wrong , therefore , If there are questions , Please leave a message here , As ...

  9. html Realization Right click is not allowed on the page Prohibit copying Do not drag pictures Copy and cut are forbidden

    as everyone knows , The general method of shielding is to use JS To write a script , But you can also directly modify the properties of the web page to block the right button Prohibit copying . Do not right click oncontextmenu="return false" prohibit ...

  10. process , Threads ,GIL,Python Multithreading , What the hell are producer consumer models

    1. Basic knowledge of operating system , process , Threads CPU It's the core of the computer , All the computing tasks : The operating system is the manager of the computer , It is responsible for task scheduling . Allocation and management of resources , Command the whole computer hardware : So how does the operating system schedule tasks ? ...